{"report": "In our prior work, we have found that technological innovation involves not only creating new ideas but also translating those ideas into a new product or service. Innovation, and the research driving it, is inherently risky because the likelihood that research can be translated into a product or service and the ultimate value of that product or service are unknown. The Department of Commerce’s National Institute of Standards and Technology describes the path from innovation to commercialization as comprised of three overarching stages: inventing, transitioning to making, and selling. (See fig. 1 for a description of the path from innovation to commercialization.) FDA and USDA have responsibility for overseeing the safety of the food supply. In general, FDA is responsible for ensuring the safety of virtually all domestic and imported food products except those regulated by USDA. USDA is responsible for ensuring the safety of meat, poultry, processed egg products, and catfish. FDA and USDA cooperate with states, tribes, and local food safety and public health agencies to carry out their federal responsibilities. FDA and USDA carry out their responsibilities in part through inspections of facilities where food is produced. The frequency of inspections the agencies conduct varies, as follows: FDA. FDA’s authority requires a risk-based approach, in which inspection rates vary depending on the level of risk associated with a food product. FDA conducts risk-based inspections of high-risk and non-high-risk food facilities. For example, the FDA Food Safety Modernization Act, signed into law in 2011, specified that FDA had to inspect all high-risk domestic facilities at least every 3 years. USDA. Depending on the type of facility, USDA conducts inspections at least once per operating shift or maintains a constant presence. Specifically, USDA conducts carcass-by-carcass inspection at all federally inspected meat and poultry slaughter facilities and verifies that these establishments follow all food safety and humane handling requirements. At facilities that process meat and poultry products, USDA conducts inspections at least once per production shift, following the agency’s longstanding interpretation of its statutes requiring it to do so. Among other things, the Federal Food, Drug, and Cosmetic Act requires that food additives be approved by FDA before they can be lawfully used in foods. Substances added to food are considered unsafe unless the agency establishes that the use of the food additive, under specific conditions for use, will be safe, or unless the substance is generally recognized as safe (GRAS) under the conditions of its intended use among qualified experts. As we reported in 2010, the Federal Food, Drug, and Cosmetic Act exempts GRAS substances from the act’s general requirement that companies obtain FDA approval before marketing food containing a new additive. GRAS substances include hundreds of spices and artificial flavors, emulsifiers and binders, vitamins and minerals, and preservatives that manufacturers add to enhance a food’s taste, texture, nutritional content, or shelf life. The GRAS exemption allows companies, without notice to or approval from FDA, to determine whether there is enough support to claim a substance is GRAS. For a company to claim a substance is GRAS, it must conclude that there is common knowledge about the safety of the substance among experts qualified by scientific training and experience to evaluate its safety. In addition, as part of their oversight of the food supply, FDA and USDA oversee food labeling of the products under their respective jurisdictions. USDA, by statute, is charged with assuring that products under its jurisdiction, including meat, poultry, and catfish, in interstate or foreign commerce are properly marked, labeled, and packaged. USDA develops and applies the labeling requirements for these products, and food manufacturers are responsible for complying with the USDA labeling rules and adhering to the process maintained by USDA for the evaluation and approval of these product labels. Consistent with its statutes, USDA requires preapproval of all labels before manufacturers can market their products. The Federal Food, Drug, and Cosmetic Act prohibits the misbranding of food, which includes food labeling that is false or misleading. Consistent with its statutes, FDA ensures that foods within its jurisdiction are not misbranded by focusing on the labels of products already in the market. FDA establishes regulations for the enforcement of these provisions and issues guidance. Food manufacturers are responsible for compliance with misbranding provisions in the Federal Food, Drug, and Cosmetic Act and its implementing regulations. From time to time, new technologies, such as those used to make cell- cultured meat, generate challenges for FDA’s and USDA’s regulatory structure. Other examples of new food technologies to which federal agencies have needed to adapt include the genetic modification of plants and irradiation of foods. In the case of genetically modified plants, there are no specific regulations addressing products resulting from the manipulation of the genetic material of living seeds. However, under FDA policy, new genetically engineered crop varieties are treated like other foods (including their conventional counterparts) under the Federal Food Drug and Cosmetic Act and may not contain either unapproved food additives or contaminants that would adulterate the food. In 1995, FDA established a voluntary pre-market consultation process through which companies are encouraged to notify the agency before marketing a food produced from a genetically modified crop and voluntarily submit a summary of the developer-performed safety assessment. FDA evaluates the safety assessment for any issues that need to be addressed and works with the developer to resolve those issues. In the case of irradiated foods, companies seeking approval for a source of radiation used to treat a food may submit a food additive petition to FDA demonstrating the safety of the proposed use. FDA grants approval only after agency scientists have determined that the proposed use is safe, then the process can be employed commercially. General information about the process of making cell-cultured meat is available, but specific information about the technology being used and the eventual commercial production methods as well as the final products is not yet known. While firms may vary in how they make cell-cultured meat, the general process they use can be described in five phases. However, the technology and methods to commercially produce cell- cultured meat are still in development, and producers, regulators, and consumers do not yet have clarity on what these will entail. The composition of the final product is also not yet known. The general process for making cell-cultured meat contains five phases: biopsy, cell banking, growth, harvest, and food processing. (See fig. 2.) The five-phase process is generally as follows: 1. Biopsy. A biopsy is taken by collecting rice-sized tissue samples from an animal, such as livestock, chicken, or fish. During this and subsequent phases, specific laboratory sanitation procedures are followed, and antibiotics may be used in order to avoid or minimize contamination from bacteria. Growth Media According to researchers and representatives from cell-cultured meat firms, the growth media for cell-cultured meat often contains fetal bovine serum, which is obtained from blood drawn from a bovine fetus at slaughter. However, researchers and representatives from cell-culturing firms we spoke with said they are working to develop growth media that do not contain fetal bovine serum. Representatives from some of these firms also told us that the composition of the growth media, including the exact ingredients and their proportions, can vary based on the specific needs of the cells and the variety of serum used. For example, cell-cultured seafood may have different growth media and environmental requirements than cell-cultured livestock and poultry. 2. Cell banking. Biopsied cells with the most desirable traits are selected and either used immediately for cell growth or frozen to create a cell bank for later use. These desirable traits can be obtained by either selecting existing cells or using genetic engineering methods to insert, delete, or edit the DNA to target desired traits in cells. Examples of desirable traits may include cells that divide quickly, cells that divide a greater number of times, cells that result in a reduced cholesterol or fat content or other desirable nutritional traits, or cells that are more resilient to environmental factors, such as temperature, than other cells. According to agency officials and representatives from cell-cultured meat firms, this phase represents an important opportunity to ensure that the source cells used to initiate commercial production are free of pathogens or other contaminants. 3. Growth. During the cell growth phase, cells are placed in a bioreactor and begin to divide and differentiate. A bioreactor is a container that creates an environment that can sustain the growth of cells and includes the ability to control factors such as temperature, pH, and oxygen and carbon dioxide concentrations. Bioreactors can vary in size, including microwave-sized and refrigerator-sized units, but could be as large as 20 to 30 feet tall in commercial production. Bioreactors contain a growth medium, which may include ingredients such as glucose, amino acids, hormones and other growth factors, and other basic nutrients that cells need to consume in order to thrive. In addition to the medium needed for growth, the cells may need to be attached to a structure, referred to as a scaffold, to properly develop into cell-cultured meat. 4. Harvest. Once the cells have divided to form a sufficiently large amount of cell-cultured meat, producers remove—or harvest—it from the growth medium and bioreactor. If a scaffold was used to provide a structure for cells to grow on, then the cell-cultured meat would either be separated from the scaffold during harvesting or left attached to an edible scaffold. 5. Food processing. The harvested cell-cultured meat is then prepared into a product such as meatballs or chicken nuggets. In the future, products similar to intact cuts of meat such as steak or chicken breast may be produced. The technology to produce cell-cultured meat at a commercial scale is still in development, and information about the methods to be used for commercial production and the composition of the final product are not yet known. In the continuum of moving a technology from innovation to commercialization, cell-cultured meat firms are in the middle stage of building and testing their prototypes, based on our discussions with representatives from these firms. Consequently, they have not finalized aspects of the technology and eventual commercial production methods to be used or the composition of the final product. As a result, certain information is not yet available to stakeholders—including cell-cultured meat firms themselves, regulators, and the public—about specific aspects of the technology and commercial production methods that will be used, such as the composition of the growth medium and of the final products. In addition to technology development, the scarcity of publicly available research on cell-cultured meat production limits information available to agency officials and the public. Each cell-cultured meat firm is developing detailed information on its own eventual commercial production methods for making cell-cultured meat. However, the firms, similar to other technology start-ups, are reluctant to disclose intellectual property and business-sensitive information due to concerns about competition. For example, one firm told us that they can reverse engineer parts of another company’s commercial production method by seeing pictures of the equipment the other company is using. In addition, cell-cultured meat firms compete with other firms for funding from sources such as venture capitalists, foreign governments, and conventional meat companies. This competition for funding contributes to firms being reluctant to share information they consider important intellectual property, such as parts of their production processes. As a result, agency officials and other stakeholders told us that they must largely rely on whatever information the cell-cultured meat firms are willing to provide to understand details of the companies’ prototype processes and products. This limitation can affect agencies’ ability to make regulatory and other decisions. Specifically, FDA and USDA officials said they have limited information on cell-cultured meat production methods and products and need more in order to regulate this new food. One USDA official explained that the agency cannot establish labeling requirements if the agency does not know the nutritional profile of the final product. For example, if the scaffold on which the cell-cultured meat is grown is not edible, the agencies may require firms to disclose certain aspects of their commercial production methods, such as how they removed the cell- cultured meat from the scaffold. However, if the scaffold is edible, it will affect the final composition of the product, which may require different labeling than a product that was developed without edible scaffolding. This lack of information results in unanswered questions about cell- cultured meat as it relates to the eventual technology and commercial production methods to be used and the composition of the final products. Among other things, this lack of information creates challenges for industry and federal regulatory agencies as cell-cultured meat nears commercialization. The sources we reviewed and stakeholders we talked to identified a number of open questions, including the following: Tissue collection. How often will producers need to collect biopsy samples from animals, and what animals will be used? Some stakeholders have stated concerns about whether, and how, regulators will ensure that biopsies are collected from healthy animals. For example, one cell-cultured meat firm stated that tissue samples would be taken from slaughtered donor animals that met federal standards for conventional processing at the time of slaughter. However, USDA and FDA have not indicated whether they would require cell-cultured meat firms to do so. Additionally, representatives from cell-cultured meat firms stated that they did not yet know how frequently they would need to collect biopsies from animals for commercial-level production. Additionally, according to researchers, there are too many unknowns to accurately estimate how much cell- cultured meat could be produced from a single biopsy of animal tissue. Genetic engineering. Will commercial production methods involve genetic engineering? Some stakeholders expressed concern that the use of genetic engineering in cell-cultured meat production could cause the product to experience a lengthy wait for regulatory approval, similar to that for genetically engineered salmon, which took approximately 20 years. One representative from a cell-cultured meat firm noted that uncertainty about pending government regulations could negatively affect firms’ ability to attract and retain investors. Representatives from some firms said understanding what regulatory requirements will look like might influence which scientific pathways they pursue as they continue to develop their commercial production methods. According to FDA officials and representatives from one cell-cultured meat firm, it is likely that some firms will use genetic engineering in their commercial cell-cultured meat production methods. However, representatives from two other cell-cultured meat firms told us they were undecided as to whether they would use genetic engineering in their commercial production methods. Antibiotics. Will antibiotics be used to make cell-cultured meat, and will residues be present in the final product? According to agency officials, the presence of antibiotics in commercial production and the potential for residues in the resulting product would represent a significant potential concern for food safety and public health. Officials stated that they would not expect antibiotics to be used past the cell- banking phase. Representatives from cell-cultured meat firms we spoke to differed on whether they planned to use antibiotics in their commercial production process, but they had not finalized their decisions. According to one firm, if antibiotics are used, the use would be limited both in quantity and duration. Growth medium. What type of growth medium will producers use, and how might variations in the media affect the final product? According to agency officials and other stakeholders, the ingredients used in the growth medium could affect the end product’s composition and raise potential safety concerns. For example, FDA officials stated that residual growth factors, such as hormones, in the final product would be something they would likely evaluate in premarket consultations. However, representatives from cell-cultured meat firms stated that their firms have not finalized the medium they plan to use. In addition, the formulation of the medium firms use could be an important piece of intellectual property or confidential business information. Scaffold. What type of scaffold will producers use, if any, and will it be edible or inedible? The use of edible or food-grade scaffolds, where they are used, will affect the composition of the product and may need to be evaluated by federal agencies for safety. According to USDA officials, the composition of edible scaffolding may also create labeling and jurisdictional concerns. For example, USDA officials stated that the addition of edible scaffolding may require significant additional aspects of production to be subject to USDA jurisdiction. Additionally, researchers have commented that a chemical separation technique needed to separate some inedible scaffolds may also need to be evaluated for potential safety concerns. Point of harvest. How will FDA and USDA define the point of harvest? The point of harvest is the point at which FDA will transfer oversight responsibilities, including inspections, to USDA. Stakeholders have raised concerns that not having a clear definition of the point of harvest could lead to challenges such as overlapping inspection requirements or a gap in inspection. Representatives from several cell-cultured meat firms we spoke to in the spring of 2019 said it was ambiguous how FDA and USDA intended to define the point of harvest. These representatives also said it is unclear how often each agency plans to conduct inspections during the phases for which it is responsible. Agency officials stated that they are working to develop a detailed process for the transfer of jurisdiction, including defining the point of harvest. Scaling up production. How will firms scale up production to commercial levels? One 2018 study conducted by researchers in the United Kingdom stated that to produce one pound of cell-cultured meat, firms would need bioreactors at least 2 1/2 times larger than what is currently available. Similarly, a senior FDA official stated that the capacity of existing production equipment is a challenge for firms seeking to produce cell-cultured meat products at a commercial scale. As a result, the firms themselves may have to develop the equipment or custom order such equipment. Representatives from one cell- cultured meat firm told us that they are interacting with equipment providers to identify commercial-scale production equipment. Production cost. How will firms sell their product at a price point that is both profitable to the firms and affordable to the consumer? Some studies and stakeholders we interviewed, including representatives from cell-cultured meat firms, said that the high production cost of cell- cultured meat is a key industry challenge. For example, in the last two years, one firm reported that it cost $600 to make a cell-cultured meat hamburger patty and reported that it cost about $1,200 to produce a single cell-cultured meatball. One of the biggest cost drivers in the production of cell-cultured meat is the growth medium, according to some studies and some cell-cultured meat firms. To address issues of cost and scale, some firms may develop their own, less expensive growth media. Safety considerations. Are potential safety hazards in commercial production methods for cell-cultured meat different from those for conventional meat, and how will eventual commercial production methods affect the overall safety of the product? According to agency officials, cell-cultured meat may present different safety challenges compared to conventional meat. For example, according to agency officials, residues and constituents in harvested cell-cultured meat would be expected to be different from those in conventional meat, depending on the details of the production process. Representatives from one cell-cultured meat firm told us that they likely will use food processing techniques similar to those used for conventional meat, abide by similar health and safety standards, and possibly share food processing facilities. However, because specific information about commercial production methods and final products is not yet known, it is unclear whether cell-cultured meat produced on a commercial scale will pose any hazards not present in conventional meat. Product composition. What will be the composition of any eventual products? Agency officials told us that without knowing the composition of a cell-cultured meat product, it is impossible to predict how food safety and labeling requirements will apply. According to representatives from some cell-cultured meat firms, initial cell-cultured meat products most likely will not be composed entirely of cell- cultured meat but, rather, a mixture of cell-cultured meat and other ingredients such as binding, flavoring ingredients, and plant-based materials used in conventional food products. Some firms have developed prototypes of cell-cultured meat products as part of their research and development. In April 2019, representatives from one firm told us that their prototype included about 90 percent plant-based ingredients and 10 percent cell-cultured meat. However, representatives from cell-cultured meat firms stated that they aim to produce products that contain more cell-cultured meat than other ingredients. For example, some cell-cultured meat firms have stated that a long-term goal is to commercially produce cell-cultured meat products that are similar to intact cuts of meat, such as steaks. As of December 2019, these firms had not provided regulators with specific information detailing the composition of their cell-cultured meat prototypes, according to FDA and USDA officials. Environmental, animal welfare, and health impacts. How will cell- cultured meat impact the environment, animal welfare, or human health, if at all? Cell-cultured meat firms and researchers have made various claims about the potential environmental, animal welfare, and health advantages of cell-cultured meat over conventionally produced meat. For example, some cell-cultured meat firms have claimed that cell-cultured meat production would use less water and emit less greenhouse gases than conventional meat production. Some cell- cultured meat firms have also claimed that cell-cultured meat will improve animal welfare because slaughter will be unnecessary. Additionally, some stakeholders stated that because there is less opportunity for contamination from animal feces—a potential source of contamination for conventional meat—cell-cultured meat would be less likely than conventional meat to contain foodborne pathogens. However, there are disagreements regarding the accuracy of these claims. Stakeholders told us that until commercial production methods and final products are established, these claims about impacts on the environment, animal welfare, and human health will remain unsubstantiated. Timeline to market. When will cell-cultured meat products reach consumers? As of December 2019, no cell-cultured meat products were available for retail sale in the United States. Stakeholders give varying estimates for when cell-cultured meat may be commercially available. Some estimates suggest that firms may be able to commercially produce some form of cell-cultured meat product as soon as 2020, while others estimate that such products may not be available for 2 to 4 years. Labeling. How will cell-cultured meat be labeled? Labeling was an area of concern for representatives from both conventional and cell- cultured meat firms who explained that the specific terminology, such as “clean meat” or “lab-grown meat,” can sometimes reflect bias for, or against, certain products, potentially affecting consumer acceptance of these products. Additionally, stakeholders, as well as agency officials, have emphasized the importance of labeling to ensure consumers have accurate information about what they are buying. For example, in February 2018 the United States Cattlemen’s Association submitted a petition to USDA requesting that the agency limit the term “beef” to products “born, raised, and harvested in a traditional manner” and “meat” to mean the “tissue or flesh of animals that have been harvested in the traditional manner.” USDA received over 6,000 comments on the petition, and the agency had not responded to the petition as of December 2019. However, according to agency officials, USDA has committed to a public process, likely rulemaking, for the development of labeling requirements for cell- cultured meat and poultry. In addition, in recent years, a number of states have passed laws that could affect the labeling of cell-cultured meat when it comes to market. For example, in 2018, Missouri enacted a law to prohibit plant-based products and cell-cultured meat from being labeled as “meat.” Consumer Acceptance How will consumers respond to cell-cultured meat? It remains unclear whether consumers will embrace and purchase cell-cultured meat products. Stakeholders we interviewed and studies we reviewed cited consumer acceptance as a challenge for commercializing cell-cultured meat. One study noted that consumers have both positive and negative views toward cell-cultured meat, which could impact their willingness to purchase and consume such products. FDA and USDA have established multiple mechanisms to collaborate on regulatory oversight of cell-cultured meat. Specifically, the agencies have collaborated through a joint public meeting, an interagency agreement, and three working groups. However, the interagency agreement and working groups, which are ongoing mechanisms, do not fully incorporate leading practices for interagency collaboration. In addition, FDA and USDA have not documented which agency will oversee cell-cultured seafood not covered by the interagency agreement. In 2018, FDA and USDA began taking steps to collaborate on the regulatory oversight of cell-cultured meat through several mechanisms: a joint public meeting, an interagency agreement, and three working groups. The agencies held the joint meeting in October 2018 to discuss the use of cell-culture technology to develop products derived from livestock and poultry, and topics included potential hazards, oversight considerations, and labeling. As part of this meeting, FDA and USDA held an open public comment period from September through December 2018, gathered 315 written comments, and offered interested parties the opportunity to offer comments in person. The agencies received public comments from members of the public, as well as from representatives from cell-cultured meat and conventional meat industries, food and consumer safety groups, animal welfare groups, and environmental organizations, among others. The written comments the agencies received focused on such topics as environmental considerations, labeling, potential health and safety implications, and potential regulatory and inspection processes. Stakeholders also presented multiple perspectives on these issues at the meeting. For example, stakeholders expressed different views as to whether cell-cultured meat should be regulated as a food additive, considered a GRAS substance, or whether new regulations were needed. In March 2019, FDA and USDA issued a formal interagency agreement that describes the intended roles and responsibilities of each agency in overseeing cell-cultured meat. The agreement establishes the following: Oversight. FDA will oversee the early phases of growing cell-cultured meat through the point of harvest. During harvest, FDA will work with USDA to transfer regulatory oversight to USDA. USDA will then assume oversight of cell-cultured meat through the food processing phase, including labeling, as shown in figure 3. Types of meat covered. The agreement covers cell-cultured meat derived from species overseen by USDA, such as livestock, poultry, and catfish. Future actions. The agreement also details future actions the agencies plan to take, such as developing a more detailed regulatory framework or standard operating procedures and developing joint principles for product labeling. Reviewing and updating the agreement. The agreement states that the agencies have the ability to modify it as needed and will review the agreement every 3 years to determine whether they should modify or terminate it. In June 2019, FDA and USDA created three working groups to carry out the terms of the interagency agreement. The working groups are comprised of FDA and USDA officials and operate independently, though some individuals are members of multiple groups. The groups are as follows: Pre-market assessment working group. Led by FDA, this group was created to clarify the process FDA will use for pre-market reviews of cell-cultured meat. Labeling working group. Led by USDA, this group will focus on developing joint principles for product labeling and claims. Transfer of jurisdiction working group. Co-led by FDA and USDA, this group will develop procedures for the transfer of inspection at harvest, among other things. According to agency officials, the working groups are still in the initial phases of development, though some have progressed further than others. For example, as of December 2019, the pre-market assessment and labeling groups had met and begun to address various areas, while the transfer of jurisdiction working group was still in discussions to outline the roles, responsibilities, and outcomes for the group and had not held a formal meeting. FDA and USDA could more fully incorporate leading practices for collaboration in their interagency agreement and working groups. We have previously reported that interagency mechanisms or strategies to coordinate programs that address crosscutting issues may reduce potentially duplicative, overlapping, and fragmented efforts. In addition, while collaborative mechanisms may differ in complexity and scope, they all benefit from certain leading practices, which raise issues to consider when implementing these mechanisms. We compared the agencies’ interagency agreement and working groups with the seven leading practices to enhance and sustain interagency collaboration that we previously identified. These leading practices, and examples of the associated issues to consider, are as follows: Defining outcomes and monitoring accountability. Is there a way to track and monitor progress toward short-term and long-term outcomes? Do participating agencies have collaboration-related competencies or performance standards against which individual performance can be evaluated? Bridging organizational cultures. What are the commonalities between the participating agencies’ missions and cultures, and what are some potential challenges? Have participating agencies developed ways for operating across agency boundaries? Have participating agencies agreed on common terminology and definitions? Identifying and sustaining leadership. How will leadership be sustained over the long term? If leadership is shared, have roles and responsibilities been clearly identified and agreed upon? Clarifying roles and responsibilities. Have participating agencies clarified roles and responsibilities? Have participating agencies articulated and agreed to a process for making and enforcing decisions? Including relevant participants. Have all relevant participants been included? Do participants have appropriate knowledge, skills, and abilities to contribute? Identifying and leveraging resources. How will the collaborative mechanism be funded and staffed? Developing and updating written guidance and agreements. If appropriate, have the participating agencies documented their agreement regarding how they will collaborate? (A written document can incorporate agreements reached in any or all of the following areas: leadership, accountability, roles and responsibilities, and resources.) Have participating agencies developed ways to continually update or monitor written agreements? See appendix II for a full list of the associated issues to consider for each leading practice. We found that the interagency agreement for oversight of cell-cultured meat partially incorporates all seven leading practices for collaboration. For example: Defining outcomes and monitoring accountability. The interagency agreement partially incorporates the leading practice of defining outcomes and monitoring progress toward these outcomes. Specifically, the agreement identifies broad outcomes such as the development of labeling principles. However, the agreement does not describe how the agencies will track and monitor progress toward outcomes. Identifying and sustaining leadership. The agreement partially incorporates the leading practice of clarifying leadership structures. For example, it assigns each agency as the lead, or designates shared leadership, for different phases of the cell-cultured meat production process. However, the interagency agreement does not identify how the agencies will sustain leadership over the long term, including through succession planning. We have previously reported that given the importance of leadership to any collaborative effort, transitions and inconsistent leadership can weaken the effectiveness of any collaborative mechanism. Developing and updating written guidance and agreements. The agreement partially incorporates the leading practice of documenting how the agencies will collaborate. For example, the agreement includes a method for updating the document by including a provision that requires a review of the document every 3 years. This is consistent with our leading collaboration practice to continually update or monitor written agreements. However, the interagency agreement does not document how the agencies will track and monitor progress toward short-term and long-term outcomes. Table 1 provides more detail about the agencies’ incorporation of these leading collaboration practices in their interagency agreement. FDA and USDA officials told us that the interagency agreement was intended to be an initial, general outline for their collaboration. They also said that as the technology to produce cell-cultured meat develops and they implement the agreement, including developing the content of a regulatory program, they will consider incorporating leading practices for interagency collaboration. For example: Clarifying roles and responsibilities. FDA and USDA officials said in December 2019 that through the working groups the agencies would continue to explore and define the specific details of how they will manage their shared oversight responsibility. Including relevant participants. FDA officials said in December 2019 that the agency would like to engage many more stakeholders as it continues to develop its oversight of cell-cultured meat. Identifying and leveraging resources. As of December 2019, the pre-market assessment working group and the labeling working group were working to identify any human resources, physical, or financial resources they might need, according to FDA and USDA officials. The federal food safety system is on our High Risk List due to concerns about fragmentation, which we have reported has caused inconsistent oversight, ineffective coordination, and inefficient use of resources. As the agencies continue to collaborate on their shared oversight of cell- cultured meat, by more fully incorporating all seven leading practices for collaboration into their interagency agreement, they will be better positioned to address potential fragmentation in their efforts to ensure the safety of the food supply as cell-cultured meat products near commercialization and entry into the marketplace. We found that the pre-market assessment, labeling, and transfer of jurisdiction working groups that FDA and USDA created to carry out the terms of the interagency agreement either partially incorporate or do not incorporate the seven leading practices for interagency collaboration. Specifically, all three working groups have partially incorporated three of the seven leading practices for collaboration, but none of the working groups have incorporated the four remaining leading practices. For example: Defining outcomes and monitoring accountability. The working groups have all defined and agreed upon their general purposes. However, FDA and USDA have not established methods, such as milestones and metrics, to evaluate the progress of any of the working groups. For example, FDA officials said in December 2019 that their next steps are to conduct a general and qualitative risk assessment of animal cell culture food technology to systematically identify particular areas of interest from a food safety perspective and prepare detailed procedural guidelines for cell-cultured meat firms to follow. However, the officials did not have time frames or a method to evaluate progress towards completing these actions. Including relevant participants. While the working groups have included relevant FDA and USDA officials, none of the groups have included state or tribal officials in initial discussions and planning. According to the state officials we spoke with, being excluded from these federal-level discussions may hinder their ability to align their safety and labeling requirements, among other things, with federal standards. Developing and updating written guidance and agreements. None of the working groups have documented how they will collaborate. For example, the working groups have not documented leadership, accountability, roles and responsibilities, or resources needed for working groups. Table 2 provides more detail about FDA and USDA’s incorporation of leading collaboration practices in the three working groups. In December 2019, FDA and USDA officials said that as they continued to stand up these working groups, they were considering leading practices for collaboration. For example: Defining outcomes and monitoring accountability. FDA and USDA officials said they were considering means to monitor, evaluate, or report on the results of the pre-market assessment working group. Including relevant participants. FDA and USDA officials said that they were working to determine what knowledge participants in the pre-market assessment working group and the labeling working group needed to perform the work of the working group. Developing and updating written guidance and agreements. FDA and USDA officials said they were considering documenting how they will collaborate in the pre-market assessment working group, including potentially creating a charter for the working group. We have previously reported that fragmentation has caused inconsistent oversight and inefficient use of resources in the federal food safety oversight system. The agencies’ 2019 agreement to share oversight of cell-cultured meat creates a new relationship between FDA and USDA, since the agencies will oversee different stages of the production of the same food and hand off oversight at a certain point in that production. These factors contribute to an already complicated system in which the two agencies must coordinate on food safety oversight. In this context, some industry representatives and other stakeholders have expressed concerns about potential fragmentation or overlap in oversight of cell-cultured meat, such as could occur during the harvest phase of cell-cultured meat production when FDA hands off its oversight to USDA. Additionally, representatives from one cell-cultured meat firm stated that avoiding overlap in federal oversight whenever possible was important to them. For example, representatives from one firm pointed to inspection, record-keeping requirements, and regulations as potential areas at risk of overlap. They stated that potential overlap would add unnecessary, burdensome requirements and create an uneven playing field with the conventional meat industry. By more fully incorporating all seven leading practices for interagency collaboration early in the development of the three working groups, FDA and USDA could proactively minimize potential fragmentation and overlap in their oversight of cell-cultured meat, ensure consistency and efficient use of resources, and provide clarity to key stakeholders. While FDA and USDA officials told us they have decided who will oversee cell-cultured seafood, they have not formally announced or documented this decision, and some stakeholders have reported confusion or ambiguity about which agency will oversee cell-cultured seafood other than catfish. Specifically, FDA and USDA’s interagency agreement regarding cell-cultured meat states that it covers all cell-cultured meat derived from USDA-amenable species required to bear a USDA mark of inspection, which in the agreement includes livestock, poultry, and catfish. However, the agreement does not mention cell-cultured meat made from the cells of other fish, such as tuna and shellfish. FDA and USDA officials told us that FDA will have sole oversight responsibility for cell-cultured seafood other than catfish. According to FDA officials, they have verbally communicated this decision in various meetings with stakeholders. However, FDA and USDA officials told us that formally documenting FDA’s sole oversight of most cell- cultured seafood in their interagency agreement was unnecessary because FDA currently oversees most conventional seafood. According to cell-cultured meat firms, some firms are working on developing cell- cultured versions of seafood, such as bluefin tuna. However, stakeholders from two cell-cultured meat firms, including representatives of a cell- cultured seafood firm we spoke with in April 2019, stated that they did not know who in the federal government would oversee cell-cultured seafood. Representatives from one cell-cultured seafood firm said that not being able to rule out oversight by USDA prevented them from making key decisions regarding what direction to pursue in developing their commercial production method. While FDA and USDA officials told us they had agreed that FDA would oversee cell-cultured seafood other than catfish, as of December 2019, the agencies had not formally announced or documented this agreement. Developing and updating written guidance and agreements is a leading practice for collaboration, as we have previously reported. In addition, standards for internal control in the federal government state that agency management should externally communicate the necessary quality information to achieve its objectives and should select appropriate methods of communication, such as a written document or a face-to-face meeting. Management should also periodically evaluate the entity’s methods of communication so that the organization has the appropriate tools to communicate quality information throughout and outside of the entity on a timely basis. While FDA and USDA officials have informally communicated to some stakeholders that FDA will have sole oversight of most cell-cultured seafood, FDA has not communicated this information formally or in a method readily available to all relevant stakeholders, such as in their interagency agreement or other publicly available written document. FDA and USDA officials told us that they wanted to communicate this information through outreach to individual firms, but FDA or USDA officials said they did not think that revising their interagency agreement was necessary. By taking steps to document which agency will oversee cell-cultured seafood other than catfish, FDA and USDA will better ensure the public, including key stakeholders such as cell-cultured meat firms, have clarity about the agencies’ oversight responsibilities in this area. Cell-cultured meat is a new food product that raises many questions. FDA and USDA’s shared oversight of cell-cultured meat poses various challenges for these agencies, as well as stakeholders such as industry. Compounding this challenge is that specific information about key aspects of cell-cultured meat, such as the technology and production methods to be used as well as the composition of the products, is not yet known. FDA and USDA have taken steps to collaborate on their shared regulatory oversight of cell-cultured meat, including establishing an interagency agreement and three working groups. However, the interagency agreement only partially incorporates the seven leading collaboration practices that can enhance and sustain agencies’ collaborative efforts, and the working groups either partially incorporate or do not incorporate these leading practices, which has raised concerns about potential fragmentation or overlap in oversight. By more fully incorporating all seven leading practices for collaboration into their interagency agreement, FDA and USDA could build on their existing efforts and be better positioned to sustain and enhance their collaborative efforts. Moreover, by more fully incorporating all seven leading practices for interagency collaboration early in the development of the working groups, FDA and USDA could proactively minimize potential fragmentation and overlap in their oversight of cell-cultured meat and ensure they are utilizing resources efficiently or effectively. Furthermore, the interagency agreement states that it covers USDA- amenable species required to bear a USDA mark of inspection, which in the agreement includes livestock, poultry, and catfish but does not include cell-cultured seafood other than catfish. FDA and USDA officials told us they have decided FDA will oversee most cell-cultured seafood, but the agencies have not formally documented this decision. By taking steps to document in their interagency agreement, or other publicly available document, which agency will oversee cell-cultured seafood other than catfish, FDA and USDA could better ensure that members of the public and other key stakeholders such as cell-cultured meat firms have clarity about the agencies’ oversight responsibilities in this area. We are making a total of six recommendations, three to FDA and three to USDA: The Commissioner of the Food and Drug Administration, in coordination with the Secretary of Agriculture, should more fully incorporate the seven leading practices for effective collaboration in the agencies’ interagency agreement for the joint oversight of cell-cultured meat. (Recommendation 1) The Secretary of Agriculture, in coordination with the Commissioner of the Food and Drug Administration, should more fully incorporate the seven leading practices for effective collaboration in the agencies’ interagency agreement for the joint oversight of cell-cultured meat. (Recommendation 2) As the three cell-cultured meat working groups move forward, the Commissioner of the Food and Drug Administration, in coordination with the Secretary of Agriculture, should more fully incorporate the seven leading practices for effective collaboration, such as identifying specific outcomes and a way to monitor and evaluate progress toward outcomes. (Recommendation 3) As the three cell-cultured meat working groups move forward, the Secretary of Agriculture, in coordination with the Commissioner of the Food and Drug Administration, should more fully incorporate the seven leading practices for effective collaboration, such as identifying specific outcomes and a way to monitor and evaluate progress toward outcomes. (Recommendation 4) The Commissioner of the Food and Drug Administration, in coordination with the Secretary of Agriculture, should clearly document in their interagency agreement, or other publicly available document, which agency will oversee cell-cultured seafood other than catfish. (Recommendation 5) The Secretary of Agriculture, in coordination with the Commissioner of the Food and Drug Administration, should clearly document in their interagency agreement, or other publicly available document, which agency will oversee cell-cultured seafood other than catfish. (Recommendation 6) We provided a draft of this report to the Department of Health and Human Services’ (HHS) Food and Drug Administration (FDA) and the U.S. Department of Agriculture (USDA) for review and comment. In FDA’s comments, reproduced in appendix III, the agency stated that it values GAO’s recognition of the importance of collaborative mechanisms that facilitate coordination and affirmed its commitment to coordinate closely with USDA to ensure the regulatory framework for cell-cultured meat is clear and transparent to stakeholders. In USDA’s comments, reproduced in appendix IV, the department stated that the report put too much focus on best practices for interagency collaboration and not enough emphasis on industry’s role in providing the agencies with the information they need to move their processes forward to effectively regulate cell-cultured meat. USDA stated that it is difficult to review a developing technology and its future regulatory oversight when so little detailed information about the technology is known. We agree that the technology to produce cell-cultured meat is still in development and that information about the commercial production methods and composition of the final product are not yet known, as we state in our report. We also acknowledge in our report that having limited information can affect the agencies’ ability to make regulatory and other decisions. We recognize that cell-cultured meat is a new food product that raises many new questions and that specific information about key aspects of cell-cultured meat is not yet known. In light of this challenging context, it is all the more important that FDA and USDA more fully incorporate leading practices for collaboration into their joint efforts in order to ensure they are in the best possible position to oversee this new food product. FDA concurred with two recommendations and partially concurred with one. USDA also concurred with two recommendations and partially concurred with one. Specifically, both agencies agreed with our recommendations regarding (1) more fully incorporating the seven leading practices for effective collaboration in the three cell-cultured meat working groups as they move forward and (2) clearly documenting which agency will oversee cell-cultured seafood other than catfish. FDA and USDA partially concurred with our recommendation, directed to each agency, to more fully incorporate the seven leading practices for effective collaboration into the agencies’ interagency agreement for the joint oversight of cell-cultured meat. FDA stated that it concurred with the intent of incorporating the seven leading practices into the interagency agreement, and both agencies said that they are open to incorporating the practices into their development of the structure for joint oversight of cell-cultured meat. However, the agencies stated that they did not agree to revise the agreement at this time. FDA and USDA stated that the agreement is a general framework and that incorporating the leading practices would constitute an inappropriate level of detail. Instead, the agencies stated that they believe it would be most valuable to incorporate the leading practices into a more detailed joint framework or standard operating procedure they plan to issue. We appreciate the agencies’ willingness to incorporate the leading practices for effective collaboration into their efforts. The March 2019 interagency agreement states that the agencies have the ability to modify it as needed and will review the agreement every 3 years to determine whether they should modify or terminate it. Therefore, the agencies are due to revisit the agreement in March 2022, if not sooner. Regarding the agencies’ concern that incorporating the leading practices in the interagency agreement would add an inappropriate level of detail, we note that, as we state in our report, the existing agreement already partially incorporates each of the seven leading practices. We continue to believe that FDA and USDA should more fully incorporate the seven leading practices for effective collaboration into their interagency agreement for the joint oversight of cell-cultured meat. Developing a more detailed joint framework or standard operating procedure in accordance with the existing interagency agreement that incorporates those leading practices would meet the intent of our recommendation to improve the effectiveness of the agencies’ collaboration. FDA and USDA also provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, the Secretary of Agriculture, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions regarding this report, please contact me at (202) 512-3841 or morriss@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Our report (1) describes what is known about methods for commercially producing cell-cultured meat and (2) examines the extent to which the Food and Drug Administration (FDA) and U.S. Department of Agriculture (USDA) are collaborating to provide regulatory oversight of cell-cultured meat. For both objectives, we conducted a literature review of journal and media articles from 2016 through 2019 to inform our understanding of cell- cultured meat, as well as regulatory activity related to cell-cultured meat in the United States and in other countries. Specifically, we conducted a review of scholarly and trade news from 2016 through July 2019 for specific terms related to cell-cultured meat and regulatory approaches. We conducted searches in more than 30 different academic and trade databases—such as SCOPUS, Foodline, and ProQuest’s Environmental Science Collection—and identified studies relevant to our research objectives. In addition to these formal literature searches, we also asked agency officials and stakeholders to refer us to research articles and publications on cell-cultured meat. We also reviewed documentation from FDA and USDA, including the 2019 interagency agreement, existing memoranda of understanding between the two agencies, Federal Register notices about relevant public meetings, and press releases. We also reviewed documentation such as letters to regulators, presentation slides, and information on organizations’ websites from the cell-cultured meat industry, conventional meat industry, and consumer safety groups, among others. We also interviewed officials from FDA and USDA and representatives of stakeholders from the cell-cultured meat industry and industry associations, conventional meat firms and industry associations, academia, food and consumer safety groups, and state and tribal public health associations, among others. We identified stakeholders to interview through consultation with agency officials and nonfederal stakeholders and through our review of literature. We conducted 17 interviews with representatives or researchers from: six cell-cultured meat firms or industry associations, four conventional meat firms or industry associations, two food and consumer safety groups, one state and tribal public health association, and one food law policy firm. Because this is a nongeneralizable sample, the results of these interviews do not represent the views of all stakeholders involved in or with an interest in the cell-cultured or conventional meat industries or federal regulation of cell-cultured meat. However, they illustrate the range of perspectives on these topics. We also attended public meetings and conferences and conducted site visits to several locations. Specifically, we attended FDA and USDA’s public meeting in October 2018 and four conferences in 2019 that included content pertaining to food safety or cell-cultured meat. We conducted site visits to two conventional meat-processing facilities in Georgia, three cell-cultured meat firms in California, an academic cell- culturing laboratory in California, and a medical cell-culturing facility in Maryland. We identified facilities and laboratories to visit through our literature review, online research, and the assistance of agency officials and stakeholders, such as representatives from the cell-cultured meat and conventional meat industry. To describe what is known about the process for producing cell-cultured meat and potential commercial production methods, we also reviewed two sets of public comments submitted to FDA and USDA in association with the two 2018 public meetings pertaining to cell-cultured meat. These meetings were “Foods Produced Using Animal Cell Culture Technology” in July 2018 and “Use of Cell Culture Technology to Develop Products Derived from Livestock and Poultry” in October 2018. Public comments were submitted by members of the public; representatives from cell- cultured meat firms and industry associations, conventional meat companies and industry associations, food and consumer safety groups, and animal welfare groups; and environmental organizations, among others. We reviewed and analyzed all comments submitted to (1) FDA related to the July 2018 meeting and (2) FDA and USDA related to the October 2018 meeting. We also attended the October 2018 meeting and listened to agency officials’ presentations and oral remarks made by stakeholders and members of the public. We shared our description of the process for making cell-cultured meat, and associated questions, with representatives from three cell-cultured meat firms and academic researchers at two universities for their technical review and incorporated revisions as appropriate. To examine the extent to which FDA and USDA are coordinating to provide regulatory oversight of cell-cultured meat, we identified actions they took to coordinate from July 2018 through April 2020. To identify these actions, we interviewed agency officials, emailed agency officials written questions, reviewed agency documentation and public announcements, and attended public events such as the October 2018 public meeting. We compared the agencies’ interagency agreement and working groups with seven leading practices to enhance and sustain interagency collaboration. Specifically, two independent GAO reviewers assessed the degree to which agencies’ actions incorporated these leading practices. A description of these leading practices and the associated issues to consider is in appendix II. We also assessed the agencies’ actions against standards for internal control in the federal government, including standards related to communicating quality information. In this report, and in our past work, we define collaboration as any joint activity that is intended to produce more public value than could be produced when organizations act alone. We use the terms “coordination” and “collaboration” interchangeably in this report. For the purposes of our report, we define cell-cultured meat as food derived from animal cells that were grown in a controlled environment outside of the animal. We define cell-cultured seafood as a subcategory of cell-cultured meat. When referencing conventional meat, we are referring to food produced from the traditional method of slaughtering an animal, such as a cow, hog, chicken, or fish. When referencing seafood, we are referring to shellfish, sea fish, and freshwater fish served as food. We conducted this performance audit from October 2018 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Key Issues to Consider for Implementing Interagency Collaborative Mechanisms Issues to consider Have short-term and long-term outcomes been clearly defined? Is there a way to track and monitor progress toward the short-term and long-term outcomes? Do participating agencies have collaboration-related competencies or performance standards against which individual performance can be evaluated? Do participating agencies have the means to recognize and reward accomplishments related to collaboration? What are the missions and organizational cultures of the participating agencies? What are the commonalities between the participating agencies’ missions and cultures and what are some potential challenges? Have participating agencies developed ways for operating across agency boundaries? Have participating agencies agreed on common terminology and definitions? Has a lead agency or individual been identified? If leadership will be shared between one or more agencies, have roles and responsibilities been clearly identified and agreed upon? How will leadership be sustained over the long term? Have participating agencies clarified the roles and responsibilities of the participants? Have participating agencies articulated and agreed to a process for making and enforcing decisions? Have all relevant participants been included? Do the participants have: Full knowledge of the relevant resources in their agency? The ability to commit these resources? The ability to regularly attend activities of the collaborative mechanism? The appropriate knowledge, skills, and abilities to contribute? Developing and updating written guidance and agreements How will the collaborative mechanism be funded? If interagency funding is needed, is it permitted? If interagency funding is needed and permitted, is there a means to track funds in a standardized manner? How will the collaborative mechanism be staffed? Are there incentives available to encourage staff or agencies to participate? If relevant, do agencies have compatible technological systems? Have participating agencies developed online tools or other resources that facilitate joint interactions? If appropriate, have the participating agencies documented their agreement regarding how they will be collaborating? A written document can incorporate agreements reached in any or all of the following areas: Leadership Accountability Roles and responsibilities Resources Have participating agencies developed ways to continually update or monitor written agreements? Steve D. Morris, (202) 512-3841 or morriss@gao.gov In addition to the contact named above, Nico Sloss (Assistant Director), Angela Miles (Analyst-in-Charge), Sahar Angadjivand, Tim Bober, Kevin Bray, Colleen Candrl, Pin En Annie Chou, Tara Congdon, Heather Dowey, Kim Gianopoulos, Gina Hoover, Hayden Huang, Robert Lepzler, Serena Lo, David Lysy, Marc Meyer, Michael Polak, Danny Royer, Sara Sullivan, and Sarah Veale made key contributions to this report.", "summary": "Multiple firms have produced cell-cultured meat as part of their research and development. These products appear likely to become available to consumers in coming years. FDA and USDA are the primary agencies responsible for overseeing the safety of the nation's food supply. However, some stakeholders have expressed concern about the agencies' oversight of cell-cultured meat amidst a fragmented federal food safety oversight system. GAO was asked to review federal oversight of cell-cultured meat. This report (1) describes what is known about methods for commercially producing cell-cultured meat, and (2) examines the extent to which FDA and USDA are collaborating to provide regulatory oversight of cell-cultured meat. GAO conducted a literature review; reviewed documentation from FDA, USDA, and stakeholder groups; analyzed public comments submitted to the agencies; compared agency efforts with leading practices for interagency collaboration; and conducted site visits to selected cell-cultured meat firms. General information about the process of making cell-cultured meat—food products grown from the cells of livestock, poultry, and seafood—is available. However, no company is commercially producing cell-cultured meat. Specific information about the technology being used, eventual commercial production methods, and composition of the final products is not yet known. The general process contains five phases: biopsy, cell banking, growth, harvest, and food processing (see figure). The technology and methods to be used for commercial production are still in development, and producers, regulators, and consumers do not have clarity about many specifics about the process and final product. For example, it is unclear whether production methods and products will use or contain genetically-engineered cells or medications such as antibiotics. The Food and Drug Administration (FDA) and U.S. Department of Agriculture (USDA) have begun collaborating on regulatory oversight of cell-cultured meat. For example, in 2019, the agencies signed an interagency agreement and created three working groups to carry out the terms of the agreement. However, the agreement and working groups could more fully incorporate practices to enhance and sustain collaboration, such as defining outcomes. For example, the agreement identifies the development of labeling principles as an outcome, but does not describe how the agencies will track and monitor progress toward this outcome, and the working groups identify a lead agency but not members' roles. Also, agency officials said they decided FDA would oversee cell-cultured seafood other than catfish, but they have not formally announced or documented this decision. Developing and updating written guidance and agreements is also a leading practice for interagency collaboration. By fully incorporating leading practices into their efforts to collaborate, the agencies could minimize potential overlap and fragmentation, use resources in a more efficient manner, and better ensure the public and other key stakeholders have clarity about the agencies' oversight responsibilities. GAO recommends that FDA and USDA more fully incorporate leading practices for effective collaboration in the agencies' interagency agreement. FDA and USDA partially concurred and indicated a willingness to incorporate these practices in a more detailed agreement, which would also meet the intent of the recommendations. The agencies concurred with the four other recommendations.", "document_type": "gao"}
{"report": "A variety of federal laws, regulations, and policies establish requirements and guidance for EPA to follow when appointing members to serve on advisory committees. For example, one purpose of FACA is to ensure that uniform procedures govern the establishment and operation of advisory committees. Also under FACA, an agency establishing an advisory committee must, among other things, require the committee’s membership to be balanced in terms of the points of view represented and the functions to be performed by the committee. In addition, federal ethics regulations establish when and how federal officials should review financial disclosure forms to identify and prevent conflicts of interest prohibited by federal law for any prospective committee members required to file these forms in connection with their appointments to advisory committees. GSA has provided additional guidance regarding the implementation of ethics requirements under FACA. Various EPA offices and officials are responsible for helping the agency follow these requirements. For example, EPA’s Federal Advisory Committee Management Division—which has overall responsibility for committee management and ensuring that EPA’s advisory committees comply with FACA—developed the Federal Advisory Committee Handbook to clarify roles and responsibilities for complying with relevant requirements. The handbook was written primarily for EPA employees assigned as designated federal officers for committees. These officers are responsible for the day-to-day management of advisory committees and play a central role in identifying and recommending candidates who can help the committees meet their goals. EPA employees assigned as designated federal officers also are responsible for maintaining committee records. According to EPA’s Federal Advisory Committee Handbook, one of the primary reasons that Congress passed FACA was to ensure public access to the records and documents of advisory committees, and that this fosters greater transparency and accountability of agencies’ use of advisory committees. EPA’s Ethics Office is responsible for helping the agency follow federal ethics requirements. Housed within the agency’s Office of General Counsel in headquarters, the Ethics Office oversees all aspects of the agency’s ethics program, including financial disclosure reporting. The Designated Agency Ethics Official coordinates and manages the program. The Designated Agency Ethics Official delegates authority to more than 100 deputy ethics officials located throughout the agency— including in headquarters and regional offices—to carry out most elements of EPA’s ethics program. For example, deputy ethics officials are to review financial disclosure reports for prospective committee members to identify and prevent conflicts of interest. Deputy assistant administrators, deputy regional administrators, office directors, and other EPA managers may be appointed to serve as deputy ethics officials for their offices as ancillary duties to their other responsibilities. EPA can establish two kinds of advisory committees—non-discretionary and discretionary committees. The agency establishes non- discretionary committees when required to by statute or directed to by the President. For example, the Clean Air Act requires EPA to establish an advisory committee to, among other things, help EPA review standards for national ambient air quality every 5 years. EPA also can establish discretionary committees at the Administrator’s direction if, for example, these committees provide an important and unique perspective on EPA programs or operations. An example of a discretionary committee is the Pesticide Program Dialogue Committee, which was formed to help EPA perform its duties under the Federal Insecticide, Fungicide and Rodenticide Act and related laws. See appendix II for a list of EPA’s 22 advisory committees as of March 31, 2018. EPA must approve the establishment of any subcommittees formed to assist committees with their work. EPA also can appoint different types of members to its advisory committees, depending on the needs of its committees and other considerations. For instance, EPA may appoint a committee member as a federal government employee under an appropriate hiring authority. If EPA expects a federal employee to serve no more than 130 days in any 365-day period, guidance from the U.S. Office of Government Ethics (OGE), which oversees the executive branch’s ethics program, states that the employee should be designated as a special government employee (SGE). If EPA decides not to appoint the committee member as a federal employee, that committee member would be a non-employee representative. EPA decides whether to appoint committee members as federal employees. To help federal agencies such as EPA determine whether to designate committee members as SGEs or representatives, OGE has developed guidance on factors to consider when agencies make these determinations. For example, OGE guidance states that SGEs are expected to provide independent expert advice and provide their best judgment free from conflicts of interest. They are generally subject to federal ethics regulations placed on other federal employees—including the requirement to file financial disclosure forms. In addition, OGE guidance states that representatives serve as the voice of groups or entities with a financial or other stake in a particular matter before an advisory committee. Federal ethics regulations generally do not apply to representative members on FACA committees. GSA has certain government-wide responsibilities for implementing FACA, including maintaining the government-wide FACA database that tracks certain characteristics of advisory committees. Specifically, FACA requires GSA to comprehensively review the activities and responsibilities of each advisory committee annually, including the committees for which EPA officials are responsible. In turn, GSA requires federal agencies responsible for advisory committees to enter data about those committees into the database. GSA and the responsible agency (e.g., EPA) review the data on a fiscal year basis for accuracy and completeness. These reviews are typically completed by February or March of the following year. GSA’s database is accessible by the general public. It includes data on committee members and committee activities from more than 50 agencies going back to 1997. The information on EPA committees includes: whether a committee member is designated as an SGE or representative; the occupation or affiliation of a committee member; state or other geographic information associated with a committee member’s occupation or affiliation; the appointment’s start and end date for each committee member; and the dates that committees held meetings. Based on our review of EPA’s Federal Advisory Committee Handbook, the agency’s established process for appointing advisory committee members includes three main phases. These phases are soliciting nominations, evaluating candidates, and obtaining approvals from relevant EPA offices, such as the Federal Advisory Committee Management Division, before the Administrator or Deputy Administrator makes final appointment decisions. As shown in figure 1, each of the three main phases in EPA’s process involves several smaller steps. Unless noted otherwise, explanations of these steps can be found in the handbook, which documents the agency’s established process. Soliciting nominations involves six basic steps, which are carried out by a committee’s designated federal officer. The steps are as follows: Develop selection criteria. This step involves identifying the specific perspectives or points of view that should be represented by members on the committee, such as specific scientific perspectives or understandings of environmental justice. This step applies to both discretionary and non-discretionary committees. In addition, federal laws establish membership requirements for the agency’s non- discretionary committees that designated federal officers must consider when developing selection criteria. For example, the Clean Air Act requires EPA to appoint seven members—including at least one member of the National Academy of Sciences, one physician, and one person representing state air-pollution control agencies—to an independent scientific advisory committee, known as CASAC. The selection criteria developed in this step should be reflected in the notice soliciting nominations. Develop an outreach plan. This plan should: (1) describe in detail how committees intend to solicit a diverse set of nominees and (2) discuss the specific forms of solicitation. For example, one outreach plan we reviewed specified that EPA staff would solicit nominations from the American Academy of Pediatrics, American Chemical Society, and other organizations that can help EPA review the quality, relevance, and performance of its research programs. Develop membership balance plans for discretionary committees. GSA guidance states that membership balance plans for discretionary committees should describe the process used to ensure that committee membership is balanced in terms of the points of view represented and functions to be performed by the committee. For example, one membership balance plan we reviewed stated that EPA staff would consider candidates from farm worker organizations; pesticide industry and trade associations; state, local and tribal governments; and public health and other organizations. According to that membership balance plan, EPA staff also would consider prospective committee members’ geographic location to help achieve balanced membership. Solicit nominations. During this step, the designated federal officer can solicit nominations via Federal Register notices and other means, such as emails to professional associations and specific EPA email distribution lists. In response to these notices, organizations can nominate individuals, or individuals can nominate themselves or other individuals. Contact nominees after receiving nominations. During this step, the designated federal officer confirms nominees’ qualifications and experience as well as their interest in and availability to serve on the committee. Assess the diversity of the pool of nominees and conduct additional outreach, if needed, to increase the diversity of the pool. EPA’s Federal Advisory Committee Handbook provides illustrative examples of how to follow this step. In one example, the handbook explains that a committee needs a representative from local government. For the past several years, the position has been filled by someone from an affluent suburban county. To increase diversity, the handbook recommends that the designated federal officer broaden outreach to other parts of the country, especially local governments that serve low-income, rural, urban, medically underserved, or vulnerable populations. Evaluating candidates similarly involves several steps. The committee’s designated federal officer is primarily responsible for taking these steps for his or her assigned committee. In addition, a deputy ethics official is to review financial disclosure forms for any prospective members who are required to file these forms. In general, the steps for evaluating candidates are as follows: Evaluate candidates against selection criteria. During this step, the designated federal officer identifies the specific point of view that each candidate would bring to the committee—as well as each candidate’s ability to meet the selection criteria after interviewing candidates and reviewing their curriculum vitae, publications, and other relevant information. EPA’s Federal Advisory Committee Handbook notes that having the best people who represent key interests and balanced viewpoints enables the committee to provide EPA with recommendations that the agency can rely on as collective advice representing diverse stakeholder views. Identifying the best candidates may involve reviewing many more nominees than can be appointed. For example, EPA received approximately 100 nominations for 18 positions on the Science Advisory Committee on Chemicals in fiscal year 2017. Prepare a draft membership grid document with staff- recommended candidates and alternates. After evaluating individual candidates, the handbook directs the designated federal officer to recommend at least one primary and alternate candidate for each point of view and consolidate his or her short-list of recommended candidates into a draft membership grid document. The handbook indicates that this is a key step in the agency’s appointment process. It is intended to help designated federal officers identify gaps as they seek to meet FACA requirements for balanced committee membership. The handbook also directs the designated federal officer to submit the draft membership grid to EPA’s Federal Advisory Committee Management Division, EPA’s Office of General Counsel, and the Assistant Administrator for review and approval before submitting final recommendations to the Administrator. Therefore, the draft membership grid, which documents EPA staff’s rationale for recommending specific candidates, is intended to serve as the basis for discussions with EPA management as final decisions about the committee’s composition are made, according to EPA’s Federal Advisory Committee Handbook. Recommending at least one alternate for each point of view is intended to provide the EPA Administrator or Deputy Administrator—who officially selects committee members based on staff recommendations—with flexibility in appointing members, according to the handbook. Review financial disclosure forms for conformance with applicable conflict-of-interest statutes, regulations issued by OGE including any supplemental agency requirements, and other federal ethics rules, which state, among other things, that: SGEs appointed to serve on federal advisory committees generally must file financial disclosure forms within 30 days of assuming their new positions and either before providing advice to the agency or before the first committee meeting if they are eligible to file confidentially. The designated ethics official from each executive branch agency generally is to review financial disclosure reports within 60 days after receiving them and is to certify by signature and date that the filer is in compliance with federal ethics rules, and this official generally may delegate this responsibility. Obtaining approvals involves several steps and numerous EPA officials. The steps for obtaining approvals generally are as follows: EPA’s Federal Advisory Committee Management Division reviews the proposed membership for balance. EPA guidance states that designated federal officers are to obtain written concurrence from the division before preparing the final membership package for the Administrator to sign. EPA’s Office of General Counsel conducts a legal review of the proposed membership. EPA guidance states that designated federal officers are to obtain written concurrence from the Office of General Counsel prior to appointment. Assistant Administrator or Regional Administrator approves the list of recommended candidates that will be presented to the Administrator’s office. Administrator or Deputy Administrator makes final appointment decisions and signs appointment letters. From fiscal year 2017 through the first two quarters of fiscal year 2018, EPA generally followed its established process for most advisory committees; however, in fiscal year 2018, EPA did not follow a key step in its process for appointing 20 committee members to the SAB and CASAC. SAB is the agency’s largest committee and CASAC is responsible for, among other things, reviewing national ambient air-quality standards. In addition, when reviewing the step in EPA’s appointment process related specifically to financial disclosure reporting, we found that EPA did not consistently ensure that SGEs appointed to advisory committees met federal financial disclosure requirements. Our review of agency documents that supported appointment decisions for the 17 committees that appointed or reappointed committee members from fiscal year 2017 through the first two quarters of fiscal year 2018 found that EPA generally followed its process for most committees. All 14 of the discretionary committees that appointed or reappointed members during this time period developed membership balance plans, as required by GSA’s FACA regulations. In addition, 15 committees followed the step in EPA’s appointment process related to draft membership grid documents. That is, 20 of the 22 appointment packets we reviewed had draft membership grid documents reflecting EPA staff input on the best qualified and most appropriate candidates for achieving balanced committee membership. Additionally, 21 of the 22 appointment packets we reviewed contained documentation showing that EPA’s Office of General Counsel reviewed the proposed membership prior to appointment, as recommended by EPA’s Federal Advisory Committee Handbook. Figure 2 shows EPA’s established process and the steps we reviewed. For additional information about the extent to which EPA followed its process for appointing committee members, see appendix III. However, EPA did not follow a key step in its established process for appointing 20 members in fiscal year 2018 to the SAB and CASAC, which advise the agency on environmental regulatory matters, among other things. Specifically, the fiscal year 2018 appointment packets for the SAB and CASAC did not include draft membership grid documents reflecting EPA staff rationales for recommending the candidates EPA’s staff deem best qualified and most appropriate for achieving balanced committee membership. EPA officials told us in March 2019 that they did not prepare draft membership grids, as recommended by EPA’s Federal Advisory Committee Handbook, because EPA management requested a series of briefings instead. EPA officials also told us that during these briefings, EPA staff presented options for management to consider that reflected staff evaluations and summaries of public comments on candidates. EPA management then decided whom to appoint after reviewing the entire list of personnel nominated for membership—not a short-list of staff-recommended candidates, as called for by EPA’s handbook. During previous appointment cycles, EPA documents indicate and officials told us that EPA followed its established process when appointing committee members to SAB and CASAC. Specifically, documents from SAB’s and CASAC’s fiscal year 2017 appointment cycles indicate that both committees prepared draft membership grids in fiscal year 2017 in accordance with EPA’s established process. In addition, SAB and CASAC staff we interviewed told us that the process they used for filling vacancies prior to the fiscal year 2018 appointments involved vetting candidates before documenting in draft membership grids the candidates they deemed best qualified and most appropriate for achieving balanced committees. EPA officials stated that the briefing process they used in fiscal year 2018 was considered better than the use of draft membership grids, as it allowed EPA management to have in-depth discussions with SAB staff, resulting in better knowledge and a greater understanding of the SAB’s and CASAC’s membership needs. In written comments on the draft report, EPA stated that the vetting of candidates for SAB and CASAC occurred in a different manner than in previous years with a process more robust than membership grids. In addition, EPA stated that the public comment process was more robust, going beyond what was prescribed in the traditional membership process. There may be benefits to such discussions and solicitation of input. However, under EPA’s established process, agency staff are to document in draft membership grids and include in appointment packets their rationales for recommending the candidates they deem best qualified and most appropriate for achieving balanced committees. EPA developed guidance to implement FACA, one purpose of which is to encourage the establishment of uniform committee appointment and administration procedures. In written comments on the draft report, EPA noted that agency staff documented evaluations of advisory committee candidates in briefing documents. However, EPA did not provide these documents along with its comments. Moreover, neither these evaluations nor summaries of public comments were included in the packets that EPA’s Federal Advisory Committee Handbook indicates are to contain committee appointment information, impeding EPA’s ability to ensure that it consistently meets—across all of its advisory committees—FACA’s purpose of encouraging uniform committee appointment procedures. In addition, Federal Standards for Internal Control call for management to design control activities to achieve objectives and respond to risks, such as by clearly documenting all transactions and other significant events in a manner that allows the documentation to be readily available for examination. By directing officials responsible for appointing committee members to follow a key step in EPA’s appointment process—developing draft membership grids to document staff rationales for proposed membership—the agency would also have better assurance that it could show how it made appointment decisions to achieve the best qualified and most appropriate candidates for balanced membership. When reviewing the steps in EPA’s appointment process related specifically to financial disclosure reporting, we found that from fiscal year 2017 through the first two quarters of fiscal year 2018, EPA did not consistently ensure that 74 SGEs appointed or reappointed to serve on EPA advisory committees met federal financial-disclosure requirements. Of the 74 disclosure forms we reviewed, an ethics official signed and dated that the filer was in compliance with federal ethics rules for 77 percent, or 57 of the forms. However, for about 23 percent, or 17 of the 74 financial disclosure forms we reviewed, an ethics official had not signed and dated that the filer was in compliance with federal ethics rules. In addition, for about 57 percent, or 42 of the 74 forms we reviewed, we were unable to determine whether an ethics official had reviewed the financial disclosure forms within 60 days after they were filed because the forms did not indicate when EPA had received them. Table 1 illustrates the extent to which EPA took steps to ensure compliance with federal financial-disclosure-reporting requirements relevant to SGEs during this time period. In 2017, OGE found similar weaknesses in EPA’s ethics program. For example, when OGE reviewed a sample of EPA advisory committees’ ethics documents from 2015, it found that none of the financial disclosure forms for one committee had been reviewed—or signed and dated—by an ethics official to indicate that filers were in conformance with federal ethics rules. For two other committees, OGE found that EPA had not received in 2015 certain financial-disclosure forms that were due that year. We also found that EPA’s Ethics Office had not periodically evaluated, through audits or spot-checks, the quality of financial disclosure reviews conducted by its deputy ethics officials for SGEs appointed to advisory committees, as part of the periodic review of its ethics program called for by OGE regulations. An official we interviewed from EPA’s Ethics Office told us that the office did not have the staffing levels necessary to audit or spot-check financial disclosure reviews for SGEs. In addition, in a June 2018 correspondence to OGE about OGE’s review of EPA’s ethics program, EPA’s Designated Agency Ethics Official stated that EPA’s Ethics Office had fewer than three full-time equivalent positions at times during 2017. The correspondence also stated that the agency’s Office of General Counsel is committed to doubling the Ethics Office’s staffing levels in the future to increase oversight of its deputy ethics officials. Federal regulations and guidance specify that EPA has certain oversight responsibilities for its programs—including its ethics program. For example, OGE regulations: state that designated agency ethics officials, acting directly or through other officials, are responsible for carrying out effective financial disclosure programs by, among other things, using information in financial disclosure reports to prevent and resolve potential conflicts of interest; specify actions the official must take if the reviewing official concludes that information disclosed in the report may reveal a violation of applicable laws and regulations; and state that designated agency ethics officials are responsible for periodically evaluating their agencies’ ethics programs. Standards for Internal Control in the Federal Government also states that management should design control activities to achieve objectives and respond to risks, such as by comparing actual performance to planned or expected results and analyzing significant differences. Because EPA had not periodically evaluated through audits or spot- checks the quality of financial disclosure reviews for SGEs appointed to advisory committees, the agency was not well positioned to compare the program’s actual performance with planned results or address instances of noncompliance with federal ethics requirements. Until EPA’s Ethics Office, as part of its periodic review of its ethics program, evaluates—for example, through audits or spot-checks—the quality of financial disclosure reviews conducted for SGEs appointed to EPA advisory committees, it will not have reasonable assurance that it is addressing noncompliance with federal ethics requirements and preventing conflicts of interest among SGEs appointed to EPA advisory committees. EPA officials acknowledged that taking this additional oversight measure could enhance the agency’s ethics program. Of the four characteristics we reviewed—committee composition, regional affiliation, membership turnover, and number of committee meetings— one or more of the first three characteristics changed notably for four of 18 of EPA’s advisory committees after January 2017. There were no notable changes in the four characteristics we reviewed for the other 14 committees for which we reviewed at least one of the characteristics. The committee composition, regional affiliation, or membership turnover of four of EPA’s advisory committees changed notably after January 2017 compared to the period after January 2009. There was no notable change in the fourth characteristic we reviewed—that is, the number of meetings committees held. Each change identified as notable had at least a 20 percentage point difference in the change to the characteristic after January 2017 compared to the period after January 2009. See appendix I for additional information about our methodology. There was a notable decrease in the percentage of members affiliated with academic institutions on the SAB and EPA Board of Scientific Counselors (BOSC) committees after January 2017 compared to the period after January 2009. Our analysis shows that the percentage of committee members with an academic affiliation serving on the SAB decreased by 27 percentage points, or from 77 percent (36 of 47 members) on January 19, 2017, to 50 percent (22 of 44 members) about 15 months later on March 31, 2018. There was little change in the period after January 2009, when the percentage of academic members serving on the SAB remained stable at 83 percent (33 of 40 members) on January 19, 2009, and 82 percent (32 of 39 members) about 15 months later on March 31, 2010. Regarding 2013, academic members serving on the SAB decreased from 82 percent (40 of 49 members) on January 20, 2013 to 73 percent (37 of 51 members) about 15 months later. In addition to academic members, other members serving on the SAB are (1) affiliated with government (federal, local, state, or tribal) or with industry or non-government organizations (NGO); (2) are consultants; or (3) are others we could not assign to one of the above categories. See figure 3. BOSC also experienced a notable decrease in the percentage of members with an academic affiliation serving on the committee after January 2017 compared to the period after January 2009. Our analysis shows that the percentage of committee members with an academic affiliation serving on BOSC decreased by 45 percentage points, or from 65 percent (11 of 17 members) on January 19, 2017, to 20 percent (3 of 15 members) about 15 months later on March 31, 2018. There was little change in the percentage of academic members serving on BOSC after either January 2009 or January 2013. The percentage of members with an academic affiliation serving on BOSC was 55 percent (6 of 11 members) on January 19, 2009, and 56 percent (5 of 9 members) about 15 months later on March 31, 2010. Seven of 12 members were affiliated with academic institutions on January 20, 2013, and 5 of 9 members were similarly affiliated about 15 months later. See table 2. The regional affiliation of SAB committee members also changed notably after January 2017 compared to the period after January 2009. Our analysis shows that members affiliated with the southern region—which spans from Texas to Delaware—increased by about 25 percentage points, or from 28 percent (13 of 47 members) on January 19, 2017, to 52 percent (23 of 44 members) about 15 months later on March 31, 2018. There was little change in the period after January 2009, when the percentage of members affiliated with the southern region increased from 30 percent (12 of 40 members) on January 19, 2009, to 33 percent (13 of 39 members) about 15 months later on March 31, 2009. Regarding 2013, members affiliated with the southern region decreased from 33 percent (16 of 49 members) on January 20, 2013, to 27 percent (14 of 51 members) about 15 months later. Figure 4 shows the regional affiliation of SAB members using U.S. Census regions after January 2017 and January 2009. There was also a notable change in the number of members who left three committees after January 2017 compared to the number of members who left those committees after January 2009. Our analysis shows that of the members serving on January 19, 2017, 71 percent (12 of 17 members) of BOSC, 62 percent (23 of 37 members) of the Clean Air Act Advisory Committee, and 63 percent (25 of 40 members) of the Pesticide Program Dialogue Committee were no longer serving about 15 months later on March 31, 2018. There was little change in the period after January 2009, when 18 percent (2 of 11 members) of the members of BOSC and 3 percent (one of 35 members) of the members serving on the Clean Air Act Advisory Committee on January 19, 2009, were no longer serving on the committees about 15 months later on March 31, 2010. All of the members serving on the Pesticide Program Dialogue Committee (34 members) on January 19, 2009, were also serving about 15 months later on March 31, 2010. Regarding 2013, 25 percent (3 of 12 members) serving on BOSC on January 20, 2013, were not serving about 15 months later. All members serving on the other two committees on January 20, 2013, were also serving about 15 months later. In most instances, the four characteristics that we analyzed—committee composition, regional affiliation, membership turnover, and number of committee meetings held—did not change notably for the committees we reviewed from January 2017 to about 15 months later compared to the same time frame after January 2009. In many of these instances, the characteristics we analyzed had changed, but these changes were not large enough to be considered notable based on the approach we used to identify notable changes. Other than the SAB and BOSC, there were no notable changes after January 2017 in the composition of the five committees for which we analyzed this characteristic. We analyzed the committee composition of the three other committees combined because they did not have enough members to make individual analysis meaningful. Our analysis shows that the largest change after January 2017 that we did not identify as notable also occurred with BOSC. The percentage of members serving on BOSC with a government affiliation increased by 22 percentage points, or from 18 percent (3 of 17 members) on January 19, 2017, to 40 percent (6 of 15 members) about 15 months later on March 31, 2018. This compares to 2009 when the percentage of members serving on BOSC with a government affiliation remained at zero percent on January 19, 2009, (11 members) and about 15 months later on March 31, 2010, (9 members). Other than the SAB, there were no notable changes after January 2017 in the regional affiliation of members of the 10 committees for which we analyzed this characteristic. In addition to the SAB, we analyzed the regional affiliation of three other committees individually and the remaining six committees combined. The largest change in regional affiliation after January 2017 that we did not identify as notable also occurred with the SAB. Members affiliated with the northeast region decreased by more than 14 percentage points, or from 28 percent (13 of 47 members) on January 19, 2017, to 14 percent (6 of 44 members) about 15 months later on March 31, 2018. This compares to 2009 when the percentage of members affiliated with the northeast region stayed about the same, changing from 20 percent (8 of 40 members) on January 19, 2009, to 18 percent (7 of 39 members) about 15 months later on March 31, 2010. Other than BOSC, the Clean Air Act Advisory Committee, and the Pesticide Program Dialogue Committee, there were no notable changes after January 2017 to membership turnover for the 14 committees for which we analyzed this characteristic. In addition to these three committees, we analyzed the membership turnover of six other committees individually and the remaining five committees combined. Our analysis shows that the largest change in membership turnover after January 2017 that we did not identify as notable occurred with the SAB. Of the members serving on this committee on January 19, 2017, 45 percent (21 of 47 members) were no longer serving about 15 months later on March 31, 2018. This compares to 2009 when 35 percent (14 of 40 members) serving on January 19, 2009, were not serving about 15 months later on March 31, 2010. There was no notable change in the percentage decrease of meetings held before and after January 2017 compared to a similar time frame before and after January 2009. We analyzed the number of meetings held by 18 committees. Our analysis shows that for the 18 committees combined, the number of meetings decreased by 40 percent (from 90 to 54 meetings) from the approximately 15 month period before January 2017 to the approximately 15 month period after January 2017. This compares to a 27 percent decrease in meetings (from 164 to 120 meetings) from the approximately 15-month period before January 2009 to the approximately 15-month period after January 2009. Overall, there was a decrease in the number of meetings from before January 2009 to after January 2017. The number of meetings held by the 18 committees combined decreased 67 percent (from 164 to 54 meetings) from the approximately 15-month period before January 2009 to the approximately 15-month period after January 2017. Figure 5 illustrates the decrease in the number of meetings held during this time frame. The figure shows the number of meetings held by SAB separately because of the relatively large number of meetings that it held relative to the other committees. EPA’s federal advisory committees play an important role in advising the agency. EPA generally followed its established process for 15 of the 17 advisory committees that appointed or reappointed committee members during the time period we reviewed. However, EPA did not follow a key step in its process for appointing 20 members to two committees that advise the agency on environmental regulatory matters, among other things. The agency did not prepare draft membership grids with staff rationales for proposed membership, the documents intended to reflect EPA staff input on the best qualified and most appropriate candidates for achieving balanced committee membership before appointing these members. EPA officials told us in March 2019 that they did not prepare draft membership grids, as recommended by EPA’s Federal Advisory Committee Handbook, because EPA management requested a series of briefings instead. There may be benefits to following different procedures; however, under EPA’s established process, agency staff are to document in draft membership grids and include in appointment packets their rationales for recommending the candidates they deem best qualified and most appropriate for achieving balanced committees. By directing officials responsible for appointing committee members to prepare draft membership grids and include them in appointment packets for all committees, the agency would have better assurance that it could show how it made appointment decisions to achieve the best qualified and most appropriate candidates for balanced committee membership. EPA also did not consistently ensure that committee members appointed as SGEs met federal ethics requirements, and as part of its periodic review of its ethics program, EPA did not evaluate through audits or spot- checks the quality of financial disclosure reviews conducted by deputy ethics officials for these committee members. Until EPA’s Ethics Office periodically evaluates—for example, through audits or spot-checks—the quality of financial disclosure reviews conducted for SGEs appointed to EPA advisory committees, it will not have reasonable assurance that it will address noncompliance with federal ethics requirements and prevent conflicts of interest among SGEs appointed to EPA advisory committees. We are making the following two recommendations to EPA: The EPA Administrator should direct EPA officials responsible for appointing advisory committee members to follow a key step in its appointment process—developing and including draft membership grids in appointment packets with staff rationales for proposed membership— for all committees. (Recommendation 1) EPA’s Designated Agency Ethics Official should direct EPA’s Ethics Office, as part of its periodic review of EPA’s ethics program, to evaluate—for example, through audits or spot-checks—the quality of financial disclosure reviews for special government employees appointed to EPA advisory committees. (Recommendation 2) We provided a draft of this report to EPA for review and comment. In its written comments, reproduced in appendix IV, EPA disagreed with a key finding related to the first recommendation, with how we conducted some of our data analyses, and with some of the data points we presented. EPA agreed with the findings and conclusions related to the second recommendation. EPA also provided other comments, which we incorporated as appropriate. EPA stated that it believed a key finding related to the draft report’s first recommendation—that EPA follow, for all committees, the key step in its appointment process related to developing draft membership grids—was in error and should be removed from the final version of the report. EPA also stated that it followed all membership steps outlined in agency guidance with the exception of two committees, SAB and CASAC, who substituted the development of a membership grid with what the agency states was a more rigorous examination of the candidates (a series of briefings with senior management discussing the strengths and weaknesses of potential candidates). EPA stated that this is within the discretion of the EPA Administrator and that the vetting of candidates for SAB and CASAC occurred in a different manner than in previous years with a process more robust than membership grids. In addition, EPA stated that the public comment process was more robust, going beyond what was prescribed in the traditional membership process. According to EPA, for SAB and CASAC, the public was offered additional opportunity to provide input on all nominated candidates under consideration. We agree that conducting such briefings is within the discretion of the EPA Administrator, and we did not assess the outcomes of the membership appointment process. However, it remains that for SAB and CASAC, EPA did not follow a key step in its established appointment process—as documented in its agency-wide handbook—in which agency staff are to document in draft membership grids their rationales for recommending the candidates they deem best qualified and most appropriate for achieving balanced committees. While there may be benefits to following any number of alternative processes for appointing committee members, as EPA stated in its Federal Advisory Committee Advisory Handbook, EPA developed the handbook to help agency officials comply with FACA requirements. For these two advisory committees, EPA did not follow its established committee appointment process, impeding EPA’s ability to ensure that it consistently meets— across all of its advisory committees—FACA’s purpose of encouraging uniform committee appointment procedures. Furthermore, EPA did not provide documentation of the “more rigorous examination” of candidates it conducted in briefings. In its written comments, EPA stated that the SAB Staff Office documented staff evaluations in briefing documents and that we did not request such documents. However, we requested all appointment packets for the 17 committees that appointed or reappointed committee members from fiscal year 2017 through the first two quarters of fiscal year 2018. These appointment packets were to contain the documents used by EPA management to make appointment and reappointment decisions. EPA did not include the briefing documents in their packets for the SAB or CASAC, impeding EPA’s ability to ensure that it consistently meets— across all of its advisory committees—FACA’s purpose of encouraging uniform committee appointment procedures. Nor did the agency provide any such documentation in subsequent discussions about the extent to which the agency followed its established process. Our most recent meeting with EPA took place on March 19, 2019. As appropriate, we modified the report to further clarify our specific finding. Moreover, EPA disagreed with how we conducted some of our data analyses and with some of the data points we presented. We took numerous steps to ensure the accuracy of the data points presented in this report. In some instances, we identified missing or inconsistent data and shared this information with EPA officials. EPA provided some corrected data for members with missing or inconsistent appointment- date data from October 1, 2015 to March 31, 2018. We also asked EPA staff to confirm that the data had been updated in the FACA database, discussing the data with individual EPA staff members, conducting logic tests and spot-checking the data to identify errors and inconsistencies, and providing EPA with an opportunity to review and correct in writing the data presented prior to preparing our draft report. Also, in its written comments, EPA stated that we did not review data for BOSC subcommittees. Our methodology focused on the composition of committees and not their subcommittees. We continue to believe that the methodology we employed to analyze data was appropriate. We outline our rationale in appendix I, which includes the steps we took to ensure data reliability. For these reasons, we do not plan to make any further changes based on the additional data EPA provided. Lastly, EPA did not dispute our findings and conclusions related to the second recommendation that the agency evaluate, for example, through audits or spot checks, the quality of financial disclosure reviews for special government employees appointed to EPA advisory committees. EPA noted that at the time of our audit, its Ethics Office was understaffed. In its written comments, EPA said that it has now resolved these staffing issues and is engaged in a full and thorough review of all employees’ (including special government employees serving on federal advisory committees) ethics forms to ensure they meet all ethics requirements. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Administrator of the U.S. Environmental Protection Agency, the Administrator of the U.S. General Services Administration, and the Director of the U.S. Office of Government Ethics. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made major contributions to this report are listed in appendix V. To describe the U.S. Environmental Protection Agency’s (EPA) established process for appointing members to serve on EPA advisory committees, we identified and reviewed the federal laws, regulations, and policies that are relevant to EPA’s process for appointing advisory committee members. To ensure that we correctly identified all relevant laws, regulations, and guidance, we consulted with: (1) the Committee Management Secretariat at the U.S. General Services Administration (GSA), which issues regulations and guidance for Federal Advisory Committee Act (FACA) committees government-wide; (2) the U.S. Office of Government Ethics, which develops ethics-related regulations for executive branch employees; and (3) EPA. Examples of EPA guidance that we reviewed include EPA’s Federal Advisory Committee Handbook, Strengthening and Improving Membership on EPA Federal Advisory Committees, and EPA Ethics Advisory 2008-02. To evaluate the extent to which EPA followed its established process for appointing members from fiscal year 2017 through the first two quarters of fiscal year 2018, we reviewed pertinent documentation from the 17 committees that appointed or reappointed advisory committee members during this time frame. The remaining committees did not appoint any committee members during the time frame we reviewed. For the above- mentioned 17 committees, we reviewed all advisory committee appointment packets—each of which can contain appointment documents for numerous appointees or reappointees—produced during this time. We also reviewed the first section (Section 1: Identifying Information and Record of Agency Review) of the Confidential Financial Disclosure Form for EPA Special Government Employees (EPA Form 3110-48) for 74 individuals who were required to submit them to EPA to determine if they met federal financial-disclosure-reporting requirements. We reviewed all 74 of the forms provided by the 8 committees that appointed or reappointed special government employees (SGE) to serve on a committee from fiscal year 2017 through the first two quarters of fiscal year 2018. Additionally, we interviewed EPA officials involved with appointing committee members to understand the steps these officials took. We then compared the steps they described taking with selected steps in EPA’s established process for appointing members to evaluate the extent to which the agency followed its process. We focused on steps in the appointment process that were to be documented in the appointment packets, which EPA used to support appointment decisions. Specifically, we reviewed those aspects of the process for which EPA had documentary evidence, and we evaluated the implementation of ethics oversight requirements that are relevant to EPA’s committee-member appointment process. To determine whether the agency followed selected steps in its established process, two senior analysts reviewed the appointment packets. Specifically, one senior analyst conducted the primary analysis for about half of the 22 appointment packets we received, while the other conducted the primary analysis for the remaining packets. Afterwards, each analyst reviewed the other’s conclusions and noted agreement or disagreement based on the evidence provided. In some cases, discussion was necessary to resolve differences of opinion between the two analysts. Those discussions were documented. If additional documentation was necessary to resolve differences of opinion, we obtained additional information from the agency. The two analysts reached agreement on all of the packets. To describe how, if at all, selected characteristics of EPA’s advisory committees changed after January 2017, we analyzed information from the FACA database, a publically-available database maintained by GSA. The database contains information about FACA advisory committees that agencies, including EPA, are required to provide. The initial scope of our review was the 22 committees in existence on March 31, 2018. Of these 22 committees, we excluded from all of our analyses the four committees that were established after November 2007 because this is the earliest date of one of our analyses. We also excluded four other committees from the three analyses that rely on member appointment start and end dates (committee composition, membership turnover, and regional affiliation) because of missing or inconsistent data. Additionally, we excluded some other committees from some of our analyses because of other types of data reliability issues or because of the nature of the characteristic. To assess the reliability of the committee data, we reviewed database technical documentation and interviewed GSA and EPA officials to identify any potential issues with our planned analysis of the data, among other things, and determined that overall the data were sufficiently reliable for conducting analysis to describe changes in selected member and committee characteristics for our selected time periods. We discuss additional steps we took to assess the reliability of the data and data reliability issues with the FACA database at the end of this appendix. Additionally, appendix II identifies which committees we excluded from which analyses and the reasons why. Primarily using information available in the FACA database, we compared changes in four committee characteristics across committees and changes in presidential administrations. Specifically, we measured the characteristics before and after January 20, 2017, and compared them to similar periods before and after January 20, 2009. Additionally, we also compared the characteristics to those before and after January 21, 2013, to provide context to our findings and identify any patterns over time in the data. The four characteristics we measured and compared across committees and changes in presidential administrations were: Number of committee meetings For the first two characteristics, we compared across committees the percentage of members in the characteristics’ categories on either January 19, 2017, or January 19, 2009, to a day about 15 months later (either March 31, 2010, or March 31, 2018). For membership turnover, we compared across committees the percentage of members on either January 19, 2017, or January 19, 2009, who left a committee by about 15 months later (either March 31, 2010, or March 31, 2018). We chose March 31, 2018, to allow for a period of time after January 2017 for changes to occur in committee characteristics, and the fiscal year 2018 data file we received from GSA was updated as of March 31, 2018. For the fourth characteristic, we compared across committees the number of meetings held in the 15 months before January 20, 2009 and January 20, 2017, to a similar period after those dates (November 12, 2007, to March 31, 2010, or November 12, 2015, to March 31, 2018). To identify changes to a characteristic that were notable, we used the following methodology. First we identified any changes after January 2017 that were large relative to other changes to that characteristic after January 2017. If we identified a relatively large change, we then compared it to changes to the characteristic after January 2009 to assess whether it was large relative to those changes. If it was, we would identify the change as notable. The committees we analyzed individually had at least 10 members (or 10 meetings) in the relevant time periods being measured, with the exception of two committees which had nine members on March 31, 2010. We analyzed the other committees combined since relatively small changes in counts would have a relatively large impact on percentages. We measured the committee composition of 5 of 18 committees. We excluded 4 of the 18 committees because of data reliability issues and 9 committees because they were not staffed primarily with SGEs. We limited the committee composition analysis to SGEs because SGEs are expected to provide their best judgement free from conflicts of interest, rather than represent a particular viewpoint. We analyzed two of the five committees individually and the other three committees combined. To measure the composition of the five committees, we first categorized each member’s occupation from the “occupation/affiliation” field in the FACA database into one of six categories. The categories were: non-government organization (NGO); or other. To assign the categories, one GAO analyst reviewed the occupation/affiliation data for each member and assigned one of five categories (academic, consultant, government, industry, or NGO) to each member. In instances where it was unclear what category to assign, the analyst conducted online searches regarding the occupation/affiliation information to identify the type of entity and assign a category. We assigned the category “other” in 30 instances where the member was affiliated with more than one of the other categories, not affiliated with any of the other categories (for example, retired), or for which the FACA database did not provide sufficient information to assign one of the other categories. A second analyst reviewed the reasonableness of the categories assigned by the first analyst—including the additional research. The two analysts reached consensus on the categories for each member. We then applied the methodology described above to identify notable changes in committee composition after January 2017. We measured the regional affiliation of 10 of 18 committees. We excluded 8 committees because of data reliability issues. We analyzed 4 of the 10 committees individually and the other 6 committees combined. To measure the regional affiliation of the 10 committees, we assigned one of four U.S. Census regions (as defined by the U.S. Census Bureau) to each committee member based on data in the “occupation/affiliation” field in the FACA database for that member—in most instances, state information is included in this field. We then applied the methodology described above to identify notable changes in regional affiliation to the period after January 2017. The regions were: Western. We measured membership turnover in 14 of 18 committees. We excluded 4 committees because of data reliability issues. We analyzed 9 of the committees individually and the other 5 committees combined. To measure membership turnover of the 14 committees, we used date fields indicating when committee members began and ended their terms to determine the percentages of members on a committee on January 19, 2017, and January 19, 2009, who were not members about 15 months later. We then applied the methodology described above to identify notable changes in membership turnover after January 2017. We measured the change in the number of meetings for 18 committees. We analyzed two of the committees individually and the other 16 committees combined. To measure this characteristic, we used data on the date that meetings were held (we used the date that the meeting began if it was a multi-day meeting). We then applied the methodology described above to identify notable changes in the number of meetings after January 2017. We assessed the reliability of the data provided to us by GSA and took certain steps to prepare the data for analysis. GSA provided us with data files downloaded to Excel from its FACA database from October 1, 2005, to March 31, 2018, for our analysis. GSA maintains the FACA database on a fiscal year basis. During the fiscal year, staff in each agency, including EPA, are to enter data to reflect any changes about the agency’s FACA committees. At the end of each fiscal year, GSA is to perform, in conjunction with each agency, an annual comprehensive review of the data entered into the database by the agency for that fiscal year. According to GSA officials, these reviews constitute the agency’s main process for ensuring the reliability of the database. Once the review is complete, the data are locked down, meaning they can no longer be changed. We received data through the 2017 fiscal year after GSA completed the 2017 review. Because this latest GSA review was the end of fiscal year 2017 and we wanted to include data into 2018, we requested that EPA update the database to March 31, 2018, for each committee for certain data fields relevant to our analyses. We asked that for each committee, the EPA staff member responsible for entering a committee’s data in the FACA database provide confirmation to us that the data had been updated through March 31, 2018. After we received confirmation that data for the 22 committees in existence on March 31, 2018, had been updated, GSA staff provided us the data update for EPA committees from October 1, 2017, through March 31, 2018. To further assess the reliability of these data, we reviewed the database’s technical documentation and interviewed GSA and EPA officials to identify any potential issues with our planned analysis of the data. We conducted logic tests and spot-checked the data to identify errors and inconsistences. For example, we scanned committee member’s names to identify potential duplicates of the same person in the same committee and made corrections where appropriate. If a person served on more than one committee, we included that person separately for each committee on which he or she served. For each member, we also checked the appointment start and end dates indicated in each fiscal year for inconsistencies across fiscal years. In some instances, we identified missing or inconsistent data in these dates and shared this information with EPA officials. EPA was able to provide some corrected data for members with missing or inconsistent appointment-date data from October 1, 2015, to March 31, 2018. We excluded from our analyses four committees for which over 30 percent of members had appointment date issues we were not able to resolve, as well as individual members with unresolved date issues for the committees we included in the analysis. We also checked the 2018 data that GSA provided to us against the data posted to EPA’s website. We determined that overall the data were sufficiently reliable for conducting analysis to describe changes in selected member and committee characteristics for our selected time periods. Finally, we took steps to structure the data provided by GSA in the format needed for our analyses. Specifically, because GSA maintains its data on a fiscal year basis, the data we received from GSA contained a separate row in the database for each committee member for each fiscal year that he or she was a member. To facilitate our analyses, we transposed the dataset so there was one row for each member (for each committee, if a member was in more than one committee) that contained the data from all of the fiscal year records for that member. We conducted this performance audit from October 2017 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 3 provides information about each of the 22 advisory committees managed by the U.S. Environmental Protection Agency (EPA) as of March 31, 2018. For each of these committees, the table also identifies whether we included it in one or more of our analyses. If we excluded a committee from certain analyses, we also explain why. Table 4 summarizes the number of advisory-committee appointment packets for which the U.S. Environmental Protection Agency (EPA) did or did not follow the steps we evaluated for appointing members to serve on EPA advisory committees. In addition to the individuals named above, Joseph Thompson (Assistant Director), John Delicath, Charles Egan, Chad Gorman, Richard Johnson, Yvonne Jones, Mary Koenen, James Lager, Amber Sinclair, and Kiki Theodoropoulos made important contributions to this report.", "summary": "Federal advisory committees provide advice to federal agencies on many topics. As of March 31, 2018, EPA managed 22 such committees. They advise the agency on such issues as developing regulations and managing research programs. Questions have been raised about EPA's process for appointing committee members after recent policy changes affecting who serves on the advisory committees. GAO was asked to review issues related to how EPA appoints advisory committee members. This report examines: (1) EPA's process for appointing advisory committee members, (2) the extent to which EPA followed its process for selecting members from October 2016 through March 2018, and (3) how, if at all, selected characteristics of EPA advisory committees changed after January 2017. GAO reviewed relevant federal laws, regulations, and guidance; reviewed documents from committees that appointed members over this period; analyzed information from the GSA's FACA database; and interviewed agency officials. Based on GAO's review of U.S. Environmental Protection Agency's (EPA) guidance, the agency's established process for appointing advisory committee members involves three main phases: soliciting nominations, evaluating candidates, and obtaining approvals. Each phase involves several steps. For example, a key step for evaluating candidates involves EPA staff's preparing documents that reflect staff recommendations on the best qualified and most appropriate candidates for achieving balanced committee membership, according to EPA guidance. EPA generally followed its established process for most of its 22 advisory committees; however, in fiscal year 2018, EPA did not follow a key step for appointing 20 committee members to two committees GAO reviewed: the EPA Science Advisory Board and Clean Air Scientific Advisory Committee, which advise the agency on environmental regulatory matters, among other things. The 2018 appointment packets for these two committees did not contain documents reflecting EPA staff rationales for proposed membership, as called for by EPA's established process. EPA developed guidance to implement the Federal Advisory Committee Act (FACA). By directing officials responsible for appointing committee members to follow a key step in its process to document staff rationales for proposed membership, the agency would have better assurance that it will (1) consistently meet FACA's purpose of encouraging uniform appointment procedures and (2) show how it made appointment decisions to achieve the best qualified and most appropriate candidates for balanced committee membership. EPA also did not consistently ensure that members appointed as special government employees (SGE)—who are expected to provide their best judgment free from conflicts of interest and are required by federal regulations to disclose their financial interests—met federal ethics requirements. For about 23 percent, or 17 of the 74 financial disclosure forms GAO reviewed, an ethics official had not signed and dated that the SGE filing the form was in compliance with federal ethics rules. EPA also did not periodically review its ethics program, as called for by federal regulations, such as through audits or spot-checks, to evaluate the quality of financial disclosure reviews for SGEs. Until EPA's Ethics Office evaluates the quality of financial disclosure reviews of SGEs as part of its periodic review of its ethics program, it will not have reasonable assurance that it will address noncompliance with federal ethics requirements and prevent conflicts of interest on its advisory committees. Based on GAO's review of the U.S. General Services Administration's (GSA) FACA database, there were notable changes to selected characteristics of EPA advisory committees (i.e. at least a 20 percentage point difference in the change to a characteristic after January 2017 compared to the period after January 2009). Of the four characteristics GAO reviewed—committee composition, regional affiliation, membership turnover, and number of meetings committees held—one or more of the first three changed notably for four of 18 EPA advisory committees after January 2017. GAO is recommending that EPA direct (1) officials responsible for appointing committee members to follow a key step in its appointment process to document staff rationales for proposed membership and (2) EPA's Ethics Office to evaluate the quality of financial disclosure reviews of SGEs appointed to advisory committees. EPA disagreed with the first and agreed with the second recommendation. GAO continues to believe that both are valid, as discussed in the report.", "document_type": "gao"}
{"report": "According to the National Research Council, although the exact details cannot be predicted with certainty, climate change poses serious risks to many of the physical and ecological systems on which society depends. Moreover, according to key scientific assessments, the effects and costs of extreme weather events such as floods and droughts will increase in significance as what are considered rare events become more common and intense because of climate change. According to the National Academies of Sciences, Engineering, and Medicine, extreme weather events are directly traceable to loss of life, rising food and energy prices, increasing costs of disaster relief and insurance, fluctuations in property values, and concerns about national security. Table 1 shows seven effects commonly associated with climate change that DOD has documented. According to a 2010 National Research Council report on making informed decisions about climate change and our October 2009 report on climate change adaptation, most decision makers need a basic set of information to understand and make choices about how to adapt to the effects of climate change. This set of information includes information and analysis about observed climate conditions, information about observed climate effects and vulnerabilities, and projections of what climate change might mean for the local area. In November 2015, we found that in order for climate information to be useful, it must be tailored to meet the needs of each decision maker, such as an engineer responsible for building a bridge in a specific location, a county planner responsible for managing development over a larger region, or a federal official managing a national-scale program. Agencies across the federal government collect and manage many types of climate information, including observational records from satellites and weather monitoring stations on temperature and precipitation, among other things; projections from complex climate models; and tools to make this information more meaningful to decision makers. For example, the Fourth National Climate Assessment, completed in November 2018 by the U.S. Global Change Research Program, references various sources of climate information, including projected temperature and precipitation data. Likewise, in 2016, a multi-agency group led by the Strategic Environmental Research and Development Program (SERDP) developed a report and accompanying database of future sea level projections and extreme water levels, which as of May 2019 contained sea level change projections for 1,813 DOD sites worldwide. Climate projections are typically a range of possible future scenarios for particular time frames. Multiple future scenarios allow for planners and engineers to see a range of possible conditions that could occur at various points in time. For example, a planner or engineer could consider four different future scenarios occurring over the course of 20, 40, or 60 years or over the service life of the project being designed. Figure 1 shows an example of sea level change projections provided by the National Oceanic and Atmospheric Administration (NOAA). Specifically, the chart shows historical mean sea levels and multiple scenarios of projected relative sea level rise in Norfolk, Virginia. The chart shows the historical annual mean sea level from 1960 to 2018 through the bold black line. The projections use 2000 as a starting point, and so overlap with the historical data. Relative sea level rise takes into account changes in land levels—in the Norfolk area the land is generally subsiding over time. Each scenario is based on different assumptions about future greenhouse gas emissions, according to an official from NOAA’s National Ocean Service. Planners and engineers can use the multiple scenarios to evaluate when potential effects could occur and determine their risk tolerances to inform their planning or design choices. Figure 2 similarly shows the same historical mean sea levels at Norfolk, Virginia, as well as the very likely range of projections of future relative sea levels, according to the National Ocean Service. This chart shows the range of possibilities considered very likely—those between the low and intermediate scenarios in figure 1—according to an official from NOAA’s National Ocean Service. Master planning for military installations involves the evaluation of factors affecting the present and future physical development and operation of a military installation. DOD requires all installations to develop master plans. DOD’s instruction on real property management states that plans must be based on a strategic assessment of the operational mission and expected use of the installation. The plans must cover at least a 10-year period and be updated every 5 years, or more often if necessary. The plans must include lists, by year, of all construction projects, major repair and sustainment projects, and restoration and modernization projects needed within the time period covered by the plan. Individual DOD facilities projects within installations must be designed in accordance with DOD’s facilities design standards, which are defined in the Unified Facilities Criteria. Unified Facilities Criteria are technical manuals and specifications used for planning, design, construction, maintenance, and operations of all DOD facilities projects. The U.S. Army Corps of Engineers, Naval Facilities Engineering Command, and the Air Force Civil Engineer Center are responsible for administering and updating the Unified Facilities Criteria. The Unified Facilities Criteria include a core group of 27 standards that apply to building systems found in most DOD facility construction projects, and include standards such as architecture, roofing, and civil engineering. Engineers and planners apply the criteria that are most appropriate for their individual facilities projects to their project proposals and designs. Table 2 shows excerpts from requirements and guidance to project designers in the Unified Facilities Criteria relevant to the consideration of climate. Table 2. Excerpts from Unified Facilities Criteria Requirements and Guidance on Consideration of Climate Excerpt consider site-specific, long-term, climate change impacts such as drought, flood, wind, and wildfire risks. Knowing the probable wind speed and direction in a particular month can be helpful in construction and mission planning as well as in designing structures that experience severe wind-driven rain or drifting snow. Pumps, piping, and equipment must be protected from the weather. In cold climates pumps and piping must be protected from freezing temperatures. The pump station building must comply with 1-200-01 , be constructed of noncombustible materials and meet applicable building standoff distances. In new construction, the roof system selection is an integral part of the overall building design and must take into account interior building usage and climate. For example, the building can be designed to prevent outward moisture drive, support heavy roof systems (such as garden roofs or paver systems), or sloped for the desired durability (life cycle cost benefit) and aesthetic considerations. Building shape, orientation, and design must utilize the site seasonal environmental factors to minimize annual facility energy use and to optimize daylighting. Coordinate building and glazing orientation and architectural shading with seasonal solar angles and prevailing winds to enhance energy performance of the building within the site-specific micro climate. Streets, paved parking lots, roofs, and other impermeable surfaces allow no infiltration of runoff and provide little resistance to flow. Runoff draining from these surfaces can be highly concentrated and move at a velocity greater than runoff flowing over an unpaved surface. Soils must be protected from this erosive force, particularly at the edges of impermeable surfaces and soils. 11988 directs all Federal agencies to avoid floodplain development wherever there is a practicable alternative. When development within the floodplain is considered, evaluate alternative site locations to avoid or minimize adverse impacts to the floodplain. When mission needs require siting a building within or partially within the 100-year floodplain, indicate…the base flood elevation…and the minimum design flood elevation…. Extreme weather and climate change effects can damage infrastructure, requiring repairs and resulting in budgetary risks (i.e., costs) to DOD. While no individual weather event can be definitively linked to climate change, particular weather events can demonstrate the vulnerability of military facilities. For example, in October 2018, Hurricane Michael devastated Tyndall Air Force Base in Florida, shutting down most base operations until December; causing severe damage to the flight line, drone runway, and other base facilities including family housing; and destroying the base’s marina. The Air Force estimates that repairs at the base will cost about $3 billion and take 5 or more years to complete. Camp Lejeune and Marine Corps Air Stations Cherry Point and New River in North Carolina sustained heavy damage to facilities, housing, and training locations from Hurricane Florence in September 2018. The Marine Corps estimates that the recovery from the hurricane damage will cost about $3.6 billion and take years to complete. In 2014, we reported that more frequent and more severe extreme weather events and climate change effects may result in increased fiscal exposure for DOD. In the same report, officials provided examples of costs associated with extreme weather and climate change effects at DOD facilities. For example, officials from a Navy shipyard we visited stated that the catastrophic damage that could result from the flooding of a submarine in dry dock could cause substantial repair costs. In 2017, we found that DOD installations overseas face operational and budgetary risks posed by weather events and climate change effects at the military services’ installations in each of DOD’s geographic combatant commands. We recommended that the Secretaries of the Army, Navy, and Air Force work with the Office of the Secretary of Defense to issue a requirement to their installations to systematically track the costs associated with extreme weather events and climate change effects. DOD did not concur with this recommendation. In its response, DOD stated that tracking impacts and costs associated with extreme weather is important, but that the science of attributing these events to a changing climate is not supported by previous GAO reports. DOD also stated that associating a single event with climate change is difficult and does not warrant the time and money expended in doing so. However, as we stated in our response to DOD’s comments, installations generally have the capability to track the costs associated with extreme weather events, which are projected to become more frequent and intense as a result of climate change. There is substantial budgetary risk resulting from weather effects associated with climate change, and these types of repairs are neither budgeted for nor clearly represented in the federal budget process. As of April 2019, the military departments have not implemented this recommendation. Fifteen of the 23 installations we visited or contacted had integrated some considerations of extreme weather or climate change effects into their plans. For example, Langley Air Force Base, Virginia, partnered with the City of Hampton, Virginia, to study the effects of sea level rise. A 2018 addendum to the installation’s 2010 joint land use study with the City of Hampton outlined climate vulnerabilities and identified recommendations for actions to increase installation resilience. Separately, after sustaining damage from Hurricane Isabel in 2003, the installation required all new development to be constructed to a minimum elevation of 10.5 feet above sea level, higher than the flooding associated with the hurricane and one foot higher than the flooding anticipated from a storm with a 1-in-500 chance of occurring in any given year. As DOD noted in its January 2019 report to Congress on climate-related vulnerabilities, Joint Base Langley-Eustis, of which Langley Air Force Base is a part, has experienced 14 inches in relative sea level rise since 1930, due in part to land subsidence, and has experienced more frequent and severe flooding as a result. The 611th Civil Engineer Squadron, based at Joint Base Elmendorf- Richardson in Alaska, partnered with the University of Alaska, Anchorage, to develop site-specific predictive models of coastal erosion for two radar sites on the North Slope of Alaska. The squadron plans to use this information in the future to develop possible alternative facilities projects to address the erosion risks. Squadron officials told us they consulted with the military users of the radars to determine the length of time to plan for their continued use and that they intend to use this information to develop plans to address this coastal erosion. The North Slope radar sites are experiencing greater than anticipated coastal erosion rates, which have begun to threaten the infrastructure supporting the sites. Fort Irwin, California, in response to severe flash flooding in 2013 that caused loss of power and significant damage to base infrastructure, worked with the U.S. Army Corps of Engineers to develop a plan to improve stormwater drainage. The 2014 plan recommended a series of infrastructure projects, some of which Fort Irwin has implemented; others remain to be implemented, depending on the availability of funding. Figure 2 depicts flooding damage in 2013 at Fort Irwin and a stormwater diversion channel subsequently built by the installation. The flash flooding on the installation caused damage to roads and other facilities throughout the installation, according to officials. The installation subsequently raised berms and built other structures, such as the diversion channel shown in figure 3, to divert stormwater from installation facilities. Marine Corps Recruit Depot Parris Island, South Carolina, reported that the installation plans to award a contract to study sea level rise at the installation and incorporate the results into the next iteration of its master plan. The installation stated that incorporating the study’s results is included in the scope of work for the contract that has been awarded for the master plan update. Naval Station Norfolk, Virginia, noted in its 2017 master plan that climate change and sea level rise are expected to exacerbate effects to the installation from tidal flooding and storm surge, increasing risks to installation assets and capabilities. The plan established a goal of identifying measures that could minimize the effect of sea level rise on the installation. With the majority of the installation near mean sea level, Naval Station Norfolk is vulnerable to frequent flooding that is disruptive to operations. Figure 4 depicts flooding at Naval Station Norfolk. Installation officials told us that such floods can interfere with traffic on base, thus reducing the ability of those working on the installation to transit within, to, and from the base. Naval Base San Diego, California, noted in its most recent master plan that local climate change effects include water and energy shortages, loss of beaches and coastal property, and higher average temperatures, among others. The plan also stated that Naval Base San Diego should be funded to conduct a study to determine installation-specific effects of sea level rise. Navy Region Southwest subsequently partnered with the Port of San Diego to study local effects of sea level rise, which installation officials said will help them understand the effects of sea level rise on the base. Camp Lejeune, North Carolina, participated in a study of the effects of sea level rise on the installation and on certain other DOD installations in North Carolina and Florida. An installation official stated that installation officials have used the results of the study to make planning decisions, in particular by feeding the study data into the installation’s mapping of potential flood zones. The 10-year study, which concluded in 2017, was funded by SERDP and was based at Camp Lejeune to, among other things, understand the effects of climate change at Camp Lejeune. Camp Lejeune officials and one of the scientists involved in the study told us that installation officials have used the study’s results to make decisions about where to site buildings so as to take into account the possible future condition of marshes on the base. However, 8 of the 23 installations we visited or contacted had not integrated considerations of extreme weather or climate change effects into their master plans or related installation planning documents. For example, Joint Base Pearl Harbor Hickam, Hawaii, did not consider extreme weather and climate change effects in its most recent master plan, although it is located in an area that has been subject to tropical storms and where, according to projections in the DOD database of sea level change scenarios, further sea level rise is anticipated. Specifically, under the highest scenario in the database, sea level at Naval Station Pearl Harbor, part of the joint base, could rise more than 3 feet by 2065. The lowest elevation point on the base is 0.6 feet below sea level. The installation stated that it plans to incorporate the effects of climate change into the next update to its facilities master plan. Pearl Harbor Naval Shipyard, Hawaii, did not consider extreme weather or climate change effects in its most recent master plan, although it is co-located with Joint Base Pearl Harbor Hickam and therefore shares the same weather and climate conditions noted previously. Fort Wainwright, Alaska, officials told us they had not considered climate change as part of the installation’s master planning. Officials noted that the majority of the base is on thaw-stable permafrost that would be unlikely to be significantly affected by rising temperatures, but some areas of the base are on less stable permafrost. DOD noted in its January 2019 report to Congress that thawing permafrost can decrease the structural stability of buildings and other infrastructure that is built on it. Camp Pendleton, California, officials told us that although they are aware of a variety of climate-related challenges to their installation and have taken or plan to take some steps to address them, an example of which we discuss later in this report, the installation has not yet considered extreme weather and climate change effects in its master plan. The officials stated that they are still planning based on historical conditions rather than considering possible future conditions. DOD’s Unified Facilities Criteria standard specific to master planning states that where changing external conditions affect planning decisions, master planners should seek to understand, monitor, and adapt to these changes, including changes in climatic conditions such as temperature, rainfall patterns, storm frequency and intensity, and water levels. DOD’s directive on climate change adaptation further states that military departments should integrate climate change considerations into their plans. The directive also states that the Assistant Secretary of Defense for Energy, Installations, and Environment should consider climate change adaptation and resilience in the installation planning process, including the effects of climate change on both built and natural infrastructure. Our findings based on the 23 installations we reviewed for this report are consistent with our prior reports on extreme weather and climate change effects at military installations. Specifically, installations have not consistently integrated these considerations into their master plans or related installation planning documents. In May 2014, we reported that some domestic installations had integrated considerations of changing climatic conditions into their installation planning documents, but DOD had not provided key information—such as how to use climate change projections—to help ensure that efficient and consistent actions would be taken across installations. We recommended that DOD further clarify the planning actions that should be taken in installation master plans to account for climate change, to include further information about changes in applicable building codes and design standards that account for potential climate change effects and further information about potential projected climate change effects on individual installations. However, as of January 2019, DOD had not fully implemented this recommendation. For example, as we discuss later in this report, DOD’s updates to its facilities design standards lacked guidance on the use of climate projections. DOD also had not provided information on a range of potential effects of climate change on individual installations. DOD has taken some positive steps in this area, such as making available to the military services a database of sea level change scenarios for 1,774 DOD sites worldwide. However, DOD has not provided other specific types of climate projections, which we discuss in more depth later in this report. Moreover, in November 2017 we reported that about a third of the installations in our sample of overseas installations had integrated climate change adaptation into their installation plans, but the lack of key guidance and updated design standards to reflect climate change concerns hampered their ability to consistently incorporate climate change adaptation into their plans. We recommended, among other things, that the military departments integrate climate change data and projections into DOD’s facilities criteria and periodically revise those standards based on any new projections, as appropriate. DOD partially concurred, and as of January 2019, an official from the Office of the Assistant Secretary of Defense for Sustainment stated that the office was continuing to work with the military departments to evaluate how to effectively translate the latest climate data into a form usable by installation planners and facilities project designers. Based on our findings for this review, we continue to believe that DOD should take all necessary steps to implement these recommendations. While 15 of the 23 installations we visited or contacted had integrated some consideration of extreme weather or climate change effects into their planning documents, only two of these installations had taken steps to fully assess the weather and climate risks to the installation or develop plans to address identified risks. DOD has taken some broad actions to assess risk to installations from extreme weather and climate change effects. For example, in January 2018, DOD issued a report to Congress on the results of its survey of installations on the extent to which they faced a variety of extreme weather or climate effects. However, the survey responses constituted a preliminary assessment and were based on installations’ reporting of negative effects they had already experienced from extreme weather effects, rather than assessments of all future vulnerabilities based on climate projections. DOD noted that the information in the survey responses is highly qualitative and is best used as an initial indicator of where a more in-depth assessment may be warranted. However, except for two of the installations in our sample, the installations’ master plans and related installation planning documents did not (1) identify a range of possible extreme weather events and climate change effects that could affect the installation, (2) assess the likelihood of each event occurring and the possible effect on the installation, and (3) identify potential responses to these events. For example, Naval Air Station Key West, Florida, included discussion of the effects of sea level rise and storm surge on the installation in its master plan, as well as steps it could take to mitigate these effects. However, although the installation experienced drought conditions rated severe in 2011 and extreme in 2015, its master plan does not discuss effects on the installation of drought, which, according to a DOD report to Congress, can pose significant risks to an installation, including implications for base infrastructure. All of the Air Force installations in our sample rated their degree of vulnerability to a range of climatic conditions—such as flood, temperature rise, and precipitation pattern changes—in their master plans, thereby identifying a range of possible climate events and the likelihood of each event. However, of those installations that identified a range of possible extreme weather and climate change effects that could affect the installation, most did not consistently identify potential responses to these events. The two exceptions—Eglin Air Force Base, Florida, and Joint Base Langley-Eustis, Virginia—took the additional step of identifying possible actions to address these climate events. For example, Eglin Air Force Base rated itself as having a high vulnerability to storm surge, but a low vulnerability from rising temperatures, and identified steps the installation could take in facilities planning and design to mitigate the identified risks. The DOD directive on climate adaptation states that military departments should assess and manage risks to both built and natural infrastructure, including changes as appropriate to installation master planning, and should assess, incorporate, and manage the risks and effects of altered operating environments on capabilities and capacity, including basing. Moreover, Standards for Internal Control in the Federal Government states that management should identify, analyze, and respond to risks related to achieving defined objectives. Risk assessment is the identification and analysis of risks related to achieving defined objectives in order to form a basis for designing responses to these risks. Our prior work has shown that assessing risks includes assessing both the likelihood of an event occurring and the effect the event would have. Agency leaders and subject matter experts should assess each risk by assigning the likelihood of the event’s occurrence and the potential effect if the event occurs. Despite a DOD directive requiring that the military departments assess and manage risks to both built and natural infrastructure, DOD has not required in the Unified Facilities Criteria standard that guides master planning that installations assess risks posed by extreme weather and climate change effects as part of their master plans or develop plans to address identified risks. Officials in the Office of the Assistant Secretary of Defense for Sustainment acknowledged that the Unified Facilities Criteria standard on master planning does not explicitly require a risk assessment specifically for extreme weather or climate change as part of the master planning process. Because installations have not consistently assessed the risks from extreme weather and climate change effects as part of their master plans or identified potential responses to identified risks, they may formulate plans and make planning decisions without consideration of those risks. By assessing and developing actions to address these risks in their master plans, installations could better anticipate exposure of the facilities to greater than anticipated damage or degradation as a result of extreme weather events or climate change effects. Eight of the 23 installations we visited or contacted, as well as the Air Force unit responsible for the North Slope radar facilities, had made some use of climate projections to incorporate consideration of extreme weather and climate change effects into their master plans or related installation planning documents. For example, as noted previously, the 611th Civil Engineer Squadron was developing its own site-specific projections of coastal erosion affecting the North Slope radar sites in Alaska, and Norfolk Naval Shipyard considered local sea level rise projections in a study on mitigating flooding at its docks. However, officials from 11 of the 23 installations in our sample—including some from installations that had made some use of climate projections—cited the need for additional guidance from DOD or their military department headquarters on which projections to use in planning or on how to use them. This is consistent with our prior findings on DOD’s installation-level efforts to increase climate resilience. Our May 2014 report noted that installation officials told us they did not have the installation-level climate data from their military departments or from other DOD sources that they would need to understand the potential effects of climate change on their installations. We recommended, among other things, that DOD provide further direction on planning actions to account for climate change, including information about changes in applicable building codes and design standards and the projected effects of climate change on individual installations. DOD concurred but as of January 2019 had not fully implemented this recommendation, as noted previously. In December 2018, an official in the Office of the Assistant Secretary of Defense for Sustainment stated that DOD plans to develop a policy on the use of sea level rise projections by some time in 2019 and eventually to incorporate these projections into the Unified Facilities Criteria. However, DOD has no current time table for incorporating guidance on the use of other types of climate projections into its Unified Facilities Criteria. The official stated that the department is working toward eventually incorporating the use of other types of climate projections into guidance but that these types of projections would have to be vetted by DOD subject matter experts and approved prior to adoption. DOD intends to move in this direction, according to the official, but DOD has not yet developed a defined process for evaluating and incorporating the use of additional climate projections into guidance. Our prior work has found that using the best available climate information, including forward-looking projections, can help an organization to manage climate-related risks. Until November 2018, DOD’s Unified Facilities Criteria on master planning stated that changes in climate conditions are to be determined from reliable and authorized sources of existing data but that to anticipate conditions during the design life of existing or planned new facilities and infrastructure, installations could also consider climate projections from reliable and authorized sources, such as, among others, the U.S. Global Change Research Office and the National Climate Assessment. In November 2018, in response to a statutory requirement in the John S. McCain National Defense Authorization Act for Fiscal Year 2019, DOD updated the Unified Facilities Criteria on master planning to specify that climate projections from reliable and authorized sources, such the U.S. Global Change Research Office and the National Climate Assessment, shall be considered and incorporated into military construction designs and modifications. DOD guidance states that the Assistant Secretary of Defense for Energy, Installations, and Environment provides guidance and direction on relevant technologies, engineering standards, tools, development and use of scenarios, and other approaches to enable prudent climate change adaptation and resilience. The guidance also states that military departments are to leverage authoritative environmental prediction sources for appropriate data and analysis products to assess the effects of weather and climate. Installations have not consistently used climate projections in their master plans because DOD has not provided detailed guidance on how to do so. Simply updating the language of the Unified Facilities Criteria on master planning in November 2018 to require the use of climate projections does not provide guidance to installations on how to use climate projections, such as what future time periods to consider and how to incorporate projections involving multiple future scenarios, nor does it identify the specific types of projections to use. The absence of guidance has hindered the ability of some installations to effectively apply the best available climate projections to their installation master planning. If they do not use climate projections in their master plans, installations risk failing to plan for changing climate and weather conditions and, as a result, could expose their facilities to greater risk of damage or degradation from extreme weather events and climate change effects. Incorporating such data into planning would help installation master planners better anticipate changing climate and weather conditions and increase the effectiveness of the installation’s long-term investments in its facilities. Eleven of the 23 installations we visited or contacted had designed or constructed one or more individual facilities projects to increase the resilience of the facilities themselves, or to increase the resilience of the installation more broadly, to extreme weather and climate change effects. For example, Joint Base Langley-Eustis, Virginia. In 2018, officials designed a project to build a maintenance hangar with a special foundation that would elevate the floor to 10 feet above the average high-water level at the project site and protect it against coastal storm flooding. Joint Base Langley-Eustis has experienced severe flooding in the past because of its low-lying geographical elevations in the Chesapeake Bay. The installation stated in its draft encroachment management action plan that the effects of climate change may exacerbate flooding issues through sea level rise or the increasing frequency and severity of storms. Norfolk Naval Shipyard, Virginia. In 2018, shipyard officials designed a project to increase the installation’s resilience to storm-induced flooding, including building a floodwall to protect the dry docks that are used to perform maintenance on ships and submarines. Norfolk Naval Shipyard experiences extreme high tides three to five times a year on average and a significant hurricane on average once a year, according to an installation presentation, and flooding has been increasing over time in the area as relative sea levels have risen. The floodwall will enclose the dry docks, providing protection to critical assets and electrical utilities while they are in dry dock, among other things. Figure 5 depicts a flooded dry dock at Norfolk Naval Shipyard, Virginia. Installation officials told us that flooding into dry docks poses risks to the ships being serviced there and to the performance of the base’s mission of servicing and maintaining Navy ships and submarines. Camp Pendleton, California. In 2018, as part of a project to construct a new aircraft landing zone, officials included protection of the nearby coastline, which had been rapidly eroding from the impact of ocean waves and rain storms. According to officials, the erosion has accelerated in recent years and has threatened not only landing zones along the coast, but also beaches that are used for amphibious assault training. Figure 6 depicts coastal erosion near a landing zone at Camp Pendleton, California. According to officials, the erosion leading to the gulley shown in the photograph has accelerated in recent years and advances further inland every year; it is now within feet of the landing zone. The officials told us that the erosion can threaten the function of the landing zone if it reaches that site. Fort Shafter, Hawaii. In 2016, officials constructed flood mitigation structures, including a flood control levee, to protect maintenance facilities being built in a flood zone. At the time, there were no adequate permanent maintenance facilities for units stationed at the base, and the only available land big enough to support the proposed maintenance facilities was located within a flood zone. Despite limited efforts to increase the resilience of facilities to extreme weather and climate change effects, officials from 17 of the military installations in our sample said that their individual facilities project designs generally did not consider climate projections. Of the installations that stated that they considered climate projections in facilities project designs, one military installation said it uses a study on sea level rise at the installation as a tool that incorporates forward-looking projections, and another installation said it uses a NOAA web-based tool, Sea Level Rise Viewer, for graphical representations of projected sea level rise. One installation noted that it had considered sea level rise projections in a pier design, which we discuss further below. A fourth installation said it plans to use a draft Navy study on the vulnerability of coastal Navy installations to sea level rise to inform an upcoming facilities project design. However, another installation said it has used energy consumption projections, which are not climate projections, and another installation cited a Navy climate adaptation handbook, which does not include climate projections for individual Navy installations. Moreover, over the course of our review of 23 installations, we were able to identify only one project as having a design informed by climate projections. Specifically, in 2018, officials from Naval Base San Diego, California, designed a project to demolish and replace an existing pier. The project’s design was informed by the expectation of sea level rise over the 75-year lifespan of the pier. An installation official told us that the consideration of rising sea levels was not part of the original project proposal, but when a contractor provided the sea level rise projections, installation officials decided to raise the pier by one foot. Figure 7 depicts a notional example of a pier—not specific to San Diego or any other particular location—raised to account for sea level rise. The Unified Facilities Criteria on piers and wharves states that the bottom elevation of the deck slab should be kept at least one foot above the extreme high water level. In this notional example, the pier is raised to account for an anticipated one-foot sea level rise, so that the bottom of the deck slab remains one foot above the extreme high water level, as shown in the figure. DOD guidance requires the military departments to assess and manage risks to both built and natural infrastructure, including making changes, as appropriate, to design and construction standards. The guidance also requires the military departments to leverage authoritative environmental prediction sources for appropriate data and analysis products to assess weather and climate effects. However, DOD’s Unified Facilities Criteria pertaining to project design, with the exception of the standard on high performance and sustainable building requirements, do not require consideration of climate projections as part of facilities project designs. The Unified Facilities Criteria standard on high performance and sustainable building requirements requires engineers to provide building design solutions that are responsive to any government-provided projections of climate change and determination of acceptable risk. We analyzed 27 core Unified Facilities Criteria, as well as 3 other Unified Facilities Criteria, Installation Master Planning, Design: Engineering Weather Data, DOD Building Code (General Building Requirements), and one facility criteria standard on Navy and Marine Corps Design Procedures. Our analysis showed that as of March 2019 these criteria, other than the Unified Facilities Criteria standard on installation master planning, do not identify authoritative sources of climate projections for use in facilities project designs. The Unified Facilities Criteria standard on installation master planning states that climate projections from the U.S. Global Change Research Program and the National Climate Assessment as well as the National Academy of Sciences shall be considered and incorporated into military construction designs and modifications. However, an official in the Office of the Assistant Secretary of Defense for Sustainment acknowledged that this requirement in the standard on installation master planning is not sufficient on its own to apply to all facility project designs. Additionally, the standard on installation master planning does not identify the specific types of climate projections to use or how to locate them. Our analysis showed that the Unified Facilities Criteria do not provide guidance on how to incorporate projections into facilities project designs, such as how to use projections involving multiple future scenarios and what future time periods to consider. We found that while some Unified Facilities Criteria direct project designers to climate data, these are historical climate data rather than projections. For example, the following standards do not direct project designers to sources of climate projections: 2015) (change 1, Feb. 1, 2016). This guidance directs project designers to use long-term rainfall records, such as those from regional weather stations, and directs engineers toward a table that provides rainfall data for selected locations. However, information included in the guidance is historical and does not include or refer to projections. Unified Facilities Criteria 3-400-02, Design: Engineering Weather Data (Sept. 20, 2018). This guidance directs project designers toward instructions for accessing climate data for use in designing facilities and in mission planning. However, the guidance does not discuss the use of or specifically reference climate projections. Unified Facilities Criteria 3-201-01, Civil Engineering (Apr. 1, 2018) (change 1, Mar. 19, 2019). This guidance requires project designers to plan for flood hazard areas and, if the project is constructed within the 100-year floodplain, requires that the project design document include flood mitigation measures as part of the project’s scope of work. However, the guidance does not include or reference projections that would help engineers design for various potential flooding scenarios. As previously noted, in response to a statutory requirement, DOD updated its Unified Facilities Criteria on master planning in November 2018 to require installations to consider and incorporate reliable and authorized sources of data on changing environmental conditions. However, simply including this language does not provide guidance to installations on what sources of climate projections to consider and how to use them in designing facilities projects, such as what future time periods to consider and how to incorporate projections involving multiple future scenarios. In addition, the Unified Facilities Criteria standard on master planning provides requirements and guidance for installation master planning but not for the design of individual facilities projects. An official of the Office of the Assistant Secretary of Defense for Sustainment stated that his office plans to develop a policy on the use of sea level rise projections by some time in 2019 and eventually to incorporate guidance on how to use sea level rise projections into the Unified Facilities Criteria or other guidance. This official added that there is currently no defined DOD process for vetting authoritative sources of climate projections, but that DOD plans to continue vetting sources for possible use, as appropriate. Furthermore, officials of 10 of the 23 military installations we reviewed stated that in order to incorporate such projections into project designs, they would need additional guidance from DOD or their military departments identifying authoritative sources of such projections or how to use climate projections that involve multiple future scenarios and different time periods. Ultimately, installations that do not consider climate projections in the design of their facilities projects may be investing in facilities projects without considering potential risks, such as potential future damage and degradation, which are associated with additional costs and reductions in capability. If DOD does not provide guidance on the use of climate projections in facilities designs, including what sources of climate projections to use, how to use projections involving multiple future scenarios, and what future time periods to consider, installation project designers will continue to lack direction on how to use climate projections. Further, if DOD does not update the Unified Facilities Criteria to require installations to consider climate projections in project designs and incorporate the department’s guidance on how to use climate projections in project designs, installation project designers may continue to exclude consideration of climate projections from facilities project designs. Considering climate projections in facilities projects would help DOD to reduce the climate-related risks to its facilities investments. DOD has a global real estate portfolio that supports the department’s global workforce and its readiness to execute its national security missions. The department has repeatedly acknowledged the threats of extreme weather and climate change effects to its installations, and as we have previously reported, has begun taking steps to increase the resilience of its infrastructure to these threats. We found that 15 of the 23 the installations we visited or contacted had considered some type of extreme weather or climate change effects in their plans, a positive step toward increasing resilience to these climate risks. However, not all had done so and most of the installations we visited or contacted did not fully assess the risks associated with extreme weather and climate change effects—including the likelihood of the threat, potential effects on the installation, and possible responses to mitigate such effects. Likewise, many of the installations did not consider climate projections in planning. Without fully assessing the risks of extreme weather and climate change effects, and without considering climate projections as part of the planning process, installations may make planning decisions that do not fully anticipate future climate conditions. By seeking to anticipate future climate conditions, DOD may be able to reduce climate-related risks to its facilities and the corresponding budgetary risks. Eleven of the 23 installations we visited or contacted had designed or implemented one or more construction projects that incorporated resilience to extreme weather or climate change effects. These projects illustrate some of the steps that can be taken to increase an installation’s resilience to climate risks. However, most of the installations had not considered climate projections in project design. Considering climate projections in facilities projects would help DOD to reduce the climate- related risks to its facilities investments. By updating its facilities project design standards to require installations to consider climate projections in project designs, identifying authoritative sources of climate projections, and providing guidance on how to use climate projections, DOD can aid installations to better position themselves to be resilient to the risks of extreme weather and climate change effects. We are making eight recommendations, including two to DOD and two to each of the military departments. Specifically, The Secretary of the Army should ensure that the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers works with the Assistant Secretary of Defense for Sustainment; the Chief of Civil Engineers and Commander, Naval Facilities Engineering Command; and the Director of the Air Force Civil Engineer Center to update the Unified Facilities Criteria standard on installation master planning to require that master plans include (1) an assessment of the risks from extreme weather and climate change effects that are specific to the installation and (2) plans to address those risks as appropriate. (Recommendation 1) The Secretary of the Navy should ensure that the Chief of Civil Engineers and Commander, Naval Facilities Engineering Command works with the Assistant Secretary of Defense for Sustainment, the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers, and the Director of the Air Force Civil Engineer Center to update the Unified Facilities Criteria standard on installation master planning to require that master plans include (1) an assessment of the risks from extreme weather and climate change effects that are specific to the installation and (2) plans to address those risks as appropriate. (Recommendation 2) The Secretary of the Air Force should ensure that the Director of the Air Force Civil Engineer Center works with the Assistant Secretary of Defense for Sustainment; the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers; and the Chief of Civil Engineers and Commander, Naval Facilities Engineering Command to update the Unified Facilities Criteria standard on installation master planning to require that master plans include (1) an assessment of the risks from extreme weather and climate change effects that are specific to the installation and (2) plans to address those risks as appropriate. (Recommendation 3) The Secretary of Defense should issue guidance on incorporating climate projections into installation master planning, including—at a minimum— what sources of climate projections to use, how to use projections involving multiple future scenarios, and what future time periods to consider. (Recommendation 4) The Secretary of Defense should issue guidance on incorporating climate projections into facilities project designs, including—at a minimum—what sources of climate projections to use, how to use projections involving multiple future scenarios, and what future time periods to consider. (Recommendation 5) The Secretary of the Army should ensure that the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers works with the Assistant Secretary of Defense for Sustainment; the Chief of Civil Engineers and Commander, Naval Facilities Engineering Command; and the Director of the Air Force Civil Engineer Center to update relevant Unified Facilities Criteria to require that installations consider climate projections in designing facilities projects and incorporate, as appropriate, DOD guidance on the use of climate projections in facilities project designs—including identification of authoritative sources of such projections, use of projections involving multiple future scenarios, and what future time periods to consider. (Recommendation 6) The Secretary of the Navy should ensure that the Chief of Civil Engineers and Commander, Naval Facilities Engineering Command works with the Assistant Secretary of Defense for Sustainment, the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers, and the Director of the Air Force Civil Engineer Center to update relevant Unified Facilities Criteria to require that installations consider climate projections in designing facilities projects and incorporate, as appropriate, DOD guidance on the use of climate projections in facilities project designs— including identification of authoritative sources of such projections, use of projections involving multiple future scenarios, and what future time periods to consider. (Recommendation 7) The Secretary of the Air Force should ensure that the Director of the Air Force Civil Engineer Center works with the Assistant Secretary of Defense for Sustainment; the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers; and the Chief of Civil Engineers and Commander, Naval Facilities Engineering Command to update relevant Unified Facilities Criteria to require that installations consider climate projections in designing facilities projects and incorporate, as appropriate, DOD guidance on the use of climate projections in facilities project designs—including identification of authoritative sources of such projections, use of projections involving multiple future scenarios, and what future time periods to consider. (Recommendation 8) We provided a draft of this report for review and comment to DOD and NOAA. In written comments, DOD concurred with all eight of our recommendations and identified actions it plans to take to address two of them. DOD’s comments are reprinted in their entirety in appendix II. DOD also provided technical comments, which we incorporated as appropriate. NOAA did not provide any comments on the draft. We are sending copies of this report to the appropriate congressional addressees; the Secretary of Defense; the Secretaries of the Departments of the Army, Navy, and Air Force; and the Secretary of Commerce (for NOAA). In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Diana Maurer at (202) 512-9627 or at maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Senate Report 115-130, accompanying a bill for fiscal year 2018 appropriations for military construction, the Department of Veterans Affairs, and related agencies, cited concerns with the frequency and costs of extreme weather events and the potential effects of climate change, and included a provision for us to review the Department of Defense’s (DOD) progress in developing a means to account for potentially damaging weather in its project designs. In response to this provision, we examined the extent to which DOD has taken steps to incorporate resilience to extreme weather and climate change effects into (1) installation master plans and related planning documents, and (2) individual installation facilities projects. For both of our objectives, we visited or requested information from a sample of domestic military installations. We focused on domestic installations because our November 2017 report focused on foreign installations. To develop this sample, we selected installations in the continental United States, Alaska, Hawaii, and U.S. territories that had identified one or more climate-related vulnerabilities, based on their past experiences, in a DOD-administered survey of climate vulnerabilities, or installations that were referenced in a prior GAO report on weather and climate risks at DOD installations. In addition to these criteria, we selected sites that represented both a diversity in types of climate vulnerabilities and geographic diversity among the military services, as well as installations involved in any climate change-related pilot studies. From these criteria, we developed a non-generalizable sample of 23 installations. We also included in the sample one Air Force unit (not an installation) with responsibilities for particular facilities of interest in Alaska, because these facilities presented a climatic vulnerability (accelerating coastal erosion) that was not necessarily included elsewhere in the sample. We visited 10 of these installations, as well as the Air Force unit in Alaska, in person. Within the sample, we selected installations to visit based on geographic diversity and installations in proximity to each other, allowing us to visit multiple installations on each trip. For the remaining 13 installations, we developed and administered a questionnaire and document request. We received responses from 12 of these installations. One installation—Camp Lejeune—sustained significant damage from Hurricane Florence in September 2018, and to minimize the burden on installation officials’ time to respond, we met with them by phone. Results from our nongeneralizable sample cannot be used to make inferences about all DOD locations. However, the information from these installations provides valuable insights. We asked similar questions to installations on our site visits and in the questionnaires, and we collected similar documents—such as installation master plans and individual facilities project documents— allowing us to report on similar information, such as the extent to which extreme weather and climate change considerations were integrated into installation master plans and individual facilities projects. For objective one, we reviewed DOD policies, guidance, and standards related to increasing climate resilience and conducting installation master planning. These documents included, among others, DOD Directive 4715.21, which establishes policy and assigns responsibilities for DOD to assess and manage risks associated with climate change; DOD’s Unified Facilities Criteria standard on installation master planning, which establishes the requirements for installation master plans; and a memorandum from the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics on floodplain management on DOD installations. We interviewed officials in the Office of the Assistant Secretary of Defense for Sustainment and the Strategic Environmental Research and Development Program. We also interviewed officials in each of the military departments, including officials involved with installation policy, as well as officials from the engineering organizations of each military department and officials in the National Oceanic and Atmospheric Administration to discuss climate science and the data potentially available for planners to use. We reviewed documents from each of the 23 installations and the one Air Force unit in our sample, including master plans, and used interviews with installation officials and questionnaires received from installations to determine the extent to which the installations had incorporated consideration of extreme weather and climate change effects into their installation plans. We compared DOD’s actions to take steps in installation planning to increase resilience to extreme weather and climate change effects with DOD guidance on climate change adaptation and resilience, Unified Facilities Criteria standards, federal internal control standards, and best practices for enterprise risk management. For objective two, we reviewed DOD guidance, including DOD Directive 4715.21, requiring DOD components to integrate climate change considerations into DOD plans. We also reviewed DOD’s facilities project design standards—the Unified Facilities Criteria—to determine the extent to which installations incorporated requirements for climate resilience and to identify any required or recommended climate data sources for facilities project design. Specifically, we reviewed the 27 core Unified Facilities Criteria standards, as well as 3 other Unified Facilities Criteria standards outside of the core 27—because of their broad relevance to project design—and one facility criteria on Navy and Marine Corps design procedures. Additionally, we performed a content analysis of these criteria for references to climate, weather, environment, and any climate data to be used as a basis for facilities design. We also identified any required or recommended climate data sources or tools for facilities design by searching for references, web links, or tables related to climate data within the criteria. Where climate data sources were identified, we reviewed them to determine the extent to which the sources and tools involved historical data or climate projections that anticipate future climate conditions. We interviewed officials from the U.S. Army Corps of Engineers, Naval Facilities Engineering Command, and the Air Force Civil Engineer Center to understand the extent to which the Unified Facilities Criteria include guidance or data sources for adapting DOD facilities to extreme weather and climate change effects. In addition, we used interviews with installation officials and questionnaires we received from installations to determine the extent to which the installations had planned or executed any military construction or sustainment, restoration, and modernization facilities projects since 2013 that included any elements for building resilience to extreme weather or climate change effects. We then reviewed project documentation for proposed or approved facilities projects to identify the resilience measures taken. We also observed some facilities-related climate resilience measures adopted by these installations. In addition, we interviewed officials from the Office of the Assistant Secretary of Defense for Sustainment to determine what plans, if any, the office had to update Unified Facilities Criteria with climate resilience requirements. We also interviewed officials from the Office of the Assistant Secretary of the Army for Installations, Energy and Environment; the Office of the Assistant Secretary of the Navy for Energy, Installations and Environment; and the Office of the Assistant Secretary of the Air Force, Installations, Environment and Energy to identify any actions, policies, or processes related to adapting facilities to extreme weather and climate change effects. Moreover, we interviewed officials from the American Society of Civil Engineers to understand what efforts, if any, had been made to incorporate climate projections into industry standards. Finally, we compared the extent to which DOD took steps in its facilities projects and its project design standards to increase resilience with DOD guidance on climate change resilience. Table 3 lists the locations we visited or contacted during this review, including the installations receiving our questionnaire. Diana Maurer at (202) 512-9627 or maurerd@gao.gov. In addition to the contact named above, Brian J. Lepore (Director, retired), Kristy Williams (Assistant Director), Michael Armes, Kendall Childers, Simon Hirschfeld, Joanne Landesman, Amie Lesser, Grace Meany, Shahrzad Nikoo, Samantha Piercy, Monica Savoy, Benjamin Sclafani, Joseph Dean Thompson, and Jack Wang made key contributions to this report. High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas, GAO-19-157SP. Washington, D.C.: March 6, 2019. Climate Change: Analysis of Reported Federal Funding. GAO-18-223. Washington, D.C.: April 30, 2018. Climate Change Adaptation: DOD Needs to Better Incorporate Adaptation into Planning and Collaboration at Overseas Installations. GAO-18-206. Washington, D.C.: November 13, 2017. Climate Change: Information on Potential Economic Effects Could Help Guide Federal Efforts to Reduce Fiscal Exposure. GAO-17-720. Washington, D.C.: September 28, 2017. High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others. GAO-17-317. Washington, D.C.: February 15, 2017. Climate Change: Improved Federal Coordination Could Facilitate Use of Forward-Looking Climate Information in Design Standards, Building Codes, and Certifications. GAO-17-3. Washington, D.C.: November 30, 2016. Defense Infrastructure: DOD Efforts to Prevent and Mitigate Encroachment at Its Installations. GAO-17-86. Washington, D.C.: November 14, 2016. Climate Information: A National System Could Help Federal, State, Local, and Private Sector Decision Makers Use Climate Information. GAO-16-37. Washington, D.C.: November 23, 2015. High-Risk Series: An Update. GAO-15-290. Washington, D.C.: February 11, 2015. Budget Issues: Opportunities to Reduce Federal Fiscal Exposures Through Greater Resilience to Climate Change and Extreme Weather. GAO-14-504T. Washington, D.C.: July 29, 2014. Climate Change Adaptation: DOD Can Improve Infrastructure Planning and Processes to Better Account for Potential Impacts. GAO-14-446. Washington, D.C.: May 30, 2014. Extreme Weather Events: Limiting Federal Fiscal Exposure and Increasing the Nation’s Resilience. GAO-14-364T. Washington, D.C.: February 12, 2014. Climate Change: Energy Infrastructure Risks and Adaptation Efforts. GAO-14-74. Washington, D.C.: January 31, 2014. Climate Change: Federal Efforts Under Way to Assess Water Infrastructure Vulnerabilities and Address Adaptation Challenges. GAO-14-23. Washington, D.C.: November 14, 2013. Climate Change: State Should Further Improve Its Reporting on Financial Support to Developing Countries to Meet Future Requirements and Guidelines. GAO-13-829. Washington, D.C.: September 19, 2013. Climate Change: Various Adaptation Efforts Are Under Way at Key Natural Resource Management Agencies. GAO-13-253. Washington, D.C.: May 31, 2013. Climate Change: Future Federal Adaptation Efforts Could Better Support Local Infrastructure Decision Makers. GAO-13-242. Washington, D.C.: April 12, 2013. High-Risk Series: An Update. GAO-13-283. Washington, D.C.: February 14, 2013. International Climate Change Assessments: Federal Agencies Should Improve Reporting and Oversight of U.S. Funding. GAO-12-43. Washington, D.C.: November 17, 2011. Climate Change Adaptation: Federal Efforts to Provide Information Could Help Government Decision Making. GAO-12-238T. Washington, D.C.: November 16, 2011.", "summary": "DOD manages a global real-estate portfolio with an almost $1.2 trillion estimated replacement value. Since 2010, DOD has identified climate change as a threat to its operations and installations. In January 2019, DOD stated that the effects of a changing climate are a national security issue with potential impacts to the department's missions, operational plans, and installations. GAO was asked to assess DOD's progress in developing a means to account for potentially damaging weather in its facilities project designs. GAO examined the extent to which DOD has taken steps to incorporate resilience to extreme weather and climate change effects into (1) selected installation master plans and related planning documents, and (2) selected individual installation facilities projects. GAO reviewed DOD documents related to increasing climate resilience, conducting installation master planning, and designing facilities projects. GAO visited or contacted a non-generalizable sample of 23 installations that had been associated with one or more climate vulnerabilities. Department of Defense (DOD) installations have not consistently assessed risks from extreme weather and climate change effects or consistently used projections to anticipate future climate conditions. For example, DOD's 2018 preliminary assessment of extreme weather and climate effects at installations was based on the installations' reported past experiences with extreme weather rather than an analysis of future vulnerabilities based on climate projections. Fifteen of the 23 installations GAO visited or contacted had considered some extreme weather and climate change effects in their plans as required by DOD guidance, but 8 had not. For example, Fort Irwin, California, worked with the U.S. Army Corps of Engineers to improve stormwater drainage after intense flash flooding caused significant damage to base infrastructure. By contrast, Joint Base Pearl Harbor-Hickam, Hawaii, did not include such considerations in its plans, although it is located in an area subject to tropical storms and where further sea level rise is anticipated. GAO also found that most of the installations had not used climate projections, because they lack guidance on how to incorporate projections into their master plans. Not assessing risks or using climate projections in installation planning may expose DOD facilities to greater-than-anticipated damage or degradation as a result of extreme weather or climate-related effects. Eleven of the 23 installations we reviewed had designed one or more individual facilities projects to increase the resilience of the facilities to extreme weather and climate change effects. However, project designs generally did not consider climate projections, according to installation officials. These officials told us that DOD lacks guidance on how to use climate projections that involve multiple future scenarios and different time periods. Until DOD updates its facilities design standards to require installations to consider climate projections in project designs, identify authoritative sources for them to use, and provide guidance on how to use projections, installation project designers may continue to exclude consideration of climate projections from facilities project designs, potentially making investments that are planned without consideration of climate-related risks. GAO is making eight recommendations, including that the military departments work together to update master planning criteria to require an assessment of extreme weather and climate change risks and to incorporate DOD guidance on the use of climate projections into facilities design standards. GAO also recommends that DOD issue guidance on incorporating climate projections into installation master planning and facilities project designs. DOD concurred with all eight of GAO's recommendations.", "document_type": "gao"}
{"report": "Nursing homes are required to keep residents safe from harm, but when abuse is alleged, a combination of federal, state, and local agencies—as well as the nursing homes themselves—play a role in investigating. Federal laws establish minimum requirements nursing homes must meet to participate in the Medicare and Medicaid programs, including standards for the quality of care. These standards cover a variety of categories, such as resident rights, quality of care, and quality of life. In 2016, CMS finalized a comprehensive update to its nursing home standards to reflect new requirements and align requirements with current clinical practices, among other things. The changes were implemented in three phases, starting November 28, 2016. The federal government and the states share oversight responsibility for the nation’s nursing homes, with specific activities occurring at the national, regional, and state levels. CMS central office. At the national level, the CMS central office oversees the federal standards nursing homes must meet to participate in the Medicare and Medicaid programs. Primarily through its State Operations Manual, the office establishes the responsibilities of CMS’s regional offices and state survey agencies in ensuring that federal quality standards for nursing homes are met. CMS regional offices. CMS’s 10 regional offices oversee state activities and report back to the CMS central office the results of their efforts. Specifically, regional offices use the State Performance Standards System to evaluate state surveyors’ performance on factors such as the frequency and quality of state surveys. State survey agencies. Under agreement with CMS, a state survey agency in each state assesses whether nursing homes meet CMS’s standards, allowing them to participate in the Medicare and Medicaid programs. State survey agencies assess nursing homes using (1) recurring standard surveys and (2) as-needed investigations. Standard surveys. State survey agencies are required by federal law to perform unannounced, on-site standard surveys of every nursing home receiving Medicare or Medicaid payment at least every 15 months, with a statewide average frequency of every 12 months. These surveys are a comprehensive assessment designed to determine whether nursing homes are complying with Medicare and Medicaid quality standards. Investigations. In addition to standard surveys, state survey agencies are required by federal law to investigate (1) complaints submitted by residents, family members, friends, physicians, and nursing home staff; and (2) “facility-reported incidents,” including incidents involving abuse of residents, that are self-reported by the nursing homes. State survey agencies review the information provided through these complaints and incidents and determine if an on-site investigation is required. During this unannounced investigation, the state surveyors assess available evidence to determine whether the allegation can be substantiated. These investigations offer the state survey agency the opportunity to identify and correct care problems in a more timely manner than through the standard surveys. If a surveyor determines that a nursing home violated a federal standard during a survey or investigation, then a deficiency code specific to that standard is cited. For instance, one deficiency code for abuse of residents encompasses mental/verbal, sexual, or physical abuse; while a few additional deficiency codes encompass abuse-related issues, such as a failure by the nursing home to train staff on issues related to abuse. Cited deficiencies are then classified into categories according to scope (the number of residents potentially affected) and severity (the potential for or occurrence of harm to residents). (See table 1.) State survey agencies are required to enter data about deficiencies into CMS’s survey database. For most deficiencies, the nursing home is required to prepare a plan of correction, and, depending on the scope and severity of the deficiency, surveyors may re-visit the facility to ensure that the nursing home has implemented its plan and corrected the deficiency. In any instances where surveyors substantiate the occurrence of resident abuse, the state survey agency is required to refer the case to three entities: 1) local law enforcement; 2) the MFCU, if appropriate; and 3) the state’s nurse aide registry or other applicable professional licensure authority. When nursing homes are cited with deficiencies, federal enforcement actions—or penalties—can be imposed to encourage homes to make corrections. In general, enforcement actions: (1) may be initially recommended by the state survey agency, (2) are transferred to the CMS regional office for review, (3) are imposed by the same CMS regional office, and (4) are implemented—that is, put into effect. Depending on the scope and severity of the deficiency cited, the CMS regional office may impose certain enforcement actions so that they are implemented immediately. However, for other enforcement actions, the regional office may provide the nursing home with an opportunity to correct the deficiencies, which, if corrected before the scheduled effective date, can result in the penalty not being implemented. Penalties include directed in- service training, fines known as civil money penalties, denial of payment, and termination from the Medicare and Medicaid programs, among others. (See fig. 1.) When a nursing home becomes aware of an incident of alleged resident abuse, the home must: immediately report the allegation to the state survey agency and then conduct an investigation of the alleged incident. Specifically, the process is as follows: The nursing home must immediately report alleged abuse to the state survey agency. After notifying the state survey agency, the nursing home is also required to conduct its own investigation and submit its findings in a written report to the state survey agency within 5 working days of the incident. Depending on the severity of the circumstances, the state survey agency may visit the nursing home to investigate the incident or wait until the nursing home submits its report. Depending on the content of the report, the state survey agency may request the home conduct additional work or the state survey agency may investigate further on its own. If the state survey agency opts not to investigate further, it may still review the manner in which the home conducted its investigation during the state survey agency’s next scheduled standard survey. If a state survey agency determines that a nurse aide is responsible for abuse, the agency must add this finding to the state’s nurse aide registry—a registry that each state is required to maintain that lists all individuals who have satisfactorily completed approved nurse aide training and a competency evaluation program in that state. Nursing homes are prohibited from employing a nurse aide with a finding of abuse on the nurse aide registry. Further, if there is a reasonable suspicion that a crime has occurred that results in serious bodily injury, federal law requires certain covered individuals at the nursing home to immediately report to law enforcement in addition to the state survey agency. Before employing a nurse aide, nursing homes are required to check each relevant state’s registry to verify that the nurse aide has passed a competency evaluation. All nursing homes must also verify with the relevant state board of licensing the professional credentials of the licensed personnel, such as registered nurses, whom they hire. In addition to state survey agencies, there are other state and local agencies that may be involved in investigating abuse in nursing homes. These other state and local agencies that investigate abuse in nursing homes are generally focused on the different aspects of the specific alleged abuse incident, in contrast to the state survey agency, which focuses on the safety of individual residents, as well as on the facility’s policies and procedures for preventing and effectively addressing abuse. These other state and local agencies include: Adult Protective Services. In some states, Adult Protective Services’ investigators are trained to provide protection and intervention for older adults in nursing homes and can play a valuable role in helping to protect residents from abuse. Ombudsmen. Long-term care ombudsmen, who serve as advocates for nursing home residents, may also investigate abuse complaints made by or on behalf of residents. Local law enforcement. Law enforcement may also play a role in investigating alleged nursing home resident abuse. Specifically, local police departments may learn of suspected instances of resident abuse and conduct criminal investigations. MFCU. The state MFCUs typically learn of abuse allegations through referrals from state survey agencies, which CMS requires if abuse is substantiated. If, after investigating an allegation, the MFCU decides that there is sufficient evidence to press criminal charges, it may prosecute the case itself or refer the matter to the state’s attorney general or a local prosecutor. Our analysis of CMS data found that from 2013 through 2017, abuse deficiencies cited in nursing homes became more frequent, with the largest increase in severe cases. While abuse deficiencies are relatively rare—they comprise less than 1 percent of the total deficiencies in each of the years we examined—they became more common over the 5-year period. Specifically, the number of abuse deficiencies cited more than doubled—from 430 in 2013 to 875 in 2017 (a 103.5 percent increase). This trend for the abuse deficiencies is in contrast to the trend for all deficiencies, which decreased about 1 percent between 2013 and 2017. At the state level, 32 states had more abuse deficiencies cited in 2017 than 2013, six states had a consistent number, and the remaining 13 had fewer. (See app. III for additional data on abuse deficiencies by state.) Furthermore, abuse deficiencies cited in 2017 were more likely to be categorized at the highest levels of severity—deficiencies causing actual harm to residents or putting residents in immediate jeopardy—than they were in 2013. Specifically, 42.6 percent of the 875 abuse deficiencies were categorized as causing actual harm or posing immediate jeopardy to residents in 2017, compared to 31.9 percent of the 430 abuse deficiencies in 2013. (See fig. 2.) In examining the types of survey or investigations conducted to identify abuse deficiencies, we found that, from 2013 to 2017, the majority (about two-thirds in each year) were identified through either a complaint investigation or facility-reported incident investigation. In contrast, for all types of deficiencies, we found the inverse—the vast majority were identified through a standard survey. This demonstrates the unique and significant role that complaint and facility-reported incident investigations have in identifying abuse deficiencies, because they allow for the identification and correction of abuse in a more timely manner than a standard survey. In fact, for the deficiencies for which we were able to identify the source, the percentage of abuse deficiencies identified through facility-reported incident investigations increased from 42.3 percent of the 430 abuse deficiencies in 2013 to 47.4 percent of the 875 abuse deficiencies in 2017. Conversely, for all types of deficiencies, a very small percentage resulted from facility-reported incident investigations—about 5 percent or less each year. (See fig. 3.) We found that enforcement actions—or penalties—were imposed and implemented by CMS infrequently each year in response to abuse deficiencies, and that fines were the most common type of implemented penalty. Specifically, for each year from 2013 through 2017, we found that about one-third of abuse deficiencies had an enforcement action imposed but not implemented, and less than 8 percent of abuse deficiencies had enforcement actions that were implemented against the nursing home. This was fairly consistent over the 5-year period. For example, in 2017, of the 875 abuse deficiencies cited, 275 (31.4 percent) resulted in enforcement actions that were imposed but not implemented and 65 (7.4 percent) had enforcement actions that were implemented against the nursing home. Furthermore, for abuse deficiencies cited at the most severe levels—that is, those causing actual harm or immediate jeopardy to residents—a smaller percentage of the deficiencies had an enforcement action imposed but not implemented compared to all abuse deficiencies, but a larger percentage were implemented. For example, in 2017, 373 of the 875 abuse deficiencies were cited at the most severe levels; of those, 81 (21.7 percent) resulted in enforcement actions that were imposed but not implemented, and 51 (13.7 percent) were implemented against the nursing home. Regardless of the severity, the predominant reason that CMS did not implement imposed enforcement actions was because the nursing home came into compliance prior to the implementation date of the penalty. For implemented enforcement actions, fines—known as civil money penalties—were overwhelmingly the most common type of penalty implemented against nursing homes with abuse deficiencies, increasing from 69.6 percent of the 23 abuse deficiencies with implemented enforcement actions in 2013 to 83.1 percent of the 65 in 2017. Denial of payments for new Medicare and Medicaid admissions—another financial penalty—was the second most common type of implemented enforcement action, but decreased from 34.8 percent in 2013 to 13.8 percent in 2017. Mandatory termination is the most severe enforcement action as it ends all payments for Medicare and Medicaid residents; it is implemented very rarely, with only one abuse deficiency resulting in mandatory termination of the nursing home across all 5 years. (See fig. 4.) In addition, we found the number of nursing homes with abuse deficiencies also more than doubled over the 5-year period. In 2013, 394 nursing homes (2.7 percent of all surveyed nursing homes) had at least one abuse deficiency compared to 821 nursing homes (5.6 percent of all surveyed nursing homes) in 2017. A nursing home may have more than one abuse deficiency cited in a single year, such as from a standard survey early in the year and then a complaint investigation later in the year. We found that in 2013, of the 394 nursing homes that had a total of 430 abuse deficiencies cited, 85 of the homes had two or more abuse deficiencies that year. In 2017, of the 821 nursing homes that had 875 total abuse deficiencies cited, 155 had two or more that year. Further, across the 5-year period, we found that a small proportion of all nursing homes with abuse deficiencies had them in multiple consecutive years. Specifically, across all years, 2,214 total unique nursing homes (13.6 percent of all surveyed nursing homes) had at least one abuse deficiency. A small portion of these nursing homes had at least one abuse deficiency in multiple consecutive years, indicating potential patterns in abuse at these nursing homes. Specifically, 185 of the 2,214 nursing homes with abuse deficiencies over the 5-year period—8.4 percent—had an abuse deficiency in any 2 consecutive years. In addition, 25 of the nursing homes—1.1 percent—had an abuse deficiency in 3 or more consecutive years. (See fig. 5.) Finally, we analyzed a selection of characteristics, including ownership type and bed size, for these nursing homes that had abuse deficiencies cited in multiple years and compared them to homes that had abuse deficiencies cited in a single year and surveyed homes that did not have any abuse deficiencies. We found that the nursing homes differed. For example, while for-profit organizations—the largest ownership group accounting for 67.9 percent of all surveyed nursing homes—owned 66.9 percent of nursing homes without any abuse deficiencies cited over the 5- year period, they accounted for 78.6 percent of nursing homes that had abuse deficiencies cited in 2 or more years. In addition, nursing homes designated as Special Focus Facilities—a CMS program that provides increased oversight to homes with consistent poor performance— constituted 2.5 percent of all surveyed nursing homes compared to 1.9 percent of nursing homes without abuse deficiencies and 10.1 percent of nursing homes with abuse deficiencies cited in 2 or more years. (See table 2.) Our analysis of a representative sample of CMS narrative descriptions— written by state surveyors—associated with abuse deficiencies cited in 2016 and 2017 found that physical and mental/verbal abuse occurred most often in nursing homes, followed by sexual abuse. Further, staff were more often the perpetrators of the deficiencies cited as abuse than were residents or others. (See fig. 6.) Physical abuse, which CMS defines as hitting, slapping, punching, biting and kicking residents, was present in about 46 percent (+/- 5 percent) of the abuse deficiency narratives. Mental/verbal abuse, which CMS defines as verbal or nonverbal conduct that can cause a resident to experience humiliation and fear, among other things, was present in about 44 percent (+/- 5 percent) of the abuse deficiency narratives. Sexual abuse, which CMS defines as non-consensual sexual contact with a resident, was present in about 18 percent (+/- 5 percent) of the abuse deficiency narratives. Staff, which includes those working in any part of the nursing home, were perpetrators in 58 percent (+/- 5 percent) of abuse deficiency narratives, followed by resident perpetrators (30 percent +/- 5 percent) and other types of perpetrators (2 percent +/- 5 percent). Other types of perpetrators can include family members of residents or other visitors. Further, our analysis of the narratives found that sexual abuse perpetrated by residents (39 percent) occurred more frequently within our sample than sexual abuse perpetrated by staff (10 percent) or others (17 percent). When staff were the perpetrators of abuse, we found within our sample that mental/verbal abuse was the most common type of abuse (60 percent), while physical abuse was most common in situations where residents (59 percent) or others (67 percent) were the perpetrators. For examples of the different types of abuse and perpetrators from our analysis, see table 3 below. Within our sample of narratives, mental/verbal abuse was less likely to be categorized by surveyors as severe compared to physical and sexual abuse. Specifically, we found in our sample that the proportion of mental/verbal abuse (30 percent) categorized by state surveyors as severe—defined as actual harm or immediate jeopardy—was smaller than the proportion of physical (40 percent) and sexual abuse (58 percent) categorized as severe. In addition, we found that most of the mental/verbal (88 percent), physical (91 percent), and sexual abuse (77 percent) narratives in our sample were categorized by surveyors as “isolated” in scope. Stakeholder groups in most of the five states we interviewed—including state survey agencies, Adult Protective Services, law enforcement, MFCUs, ombudsmen, and nursing home administrators and clinical staff—identified risk factors for abuse in nursing homes that included resident characteristics, such as residents with infrequent visitors, and nursing home staffing characteristics, such as insufficient staffing levels. (See table 4 for a description of these risk factors.) Officials we interviewed from national organizations with knowledge of abuse in nursing homes also noted some of these same risk factors. Resident characteristics. Stakeholders in each of our five selected states noted that residents who do not have frequent visitors, are cognitively impaired, or mixed with widely different age groups may be at an increased risk for abuse. Residents who do not have frequent visitors. Stakeholders in four of the five states said that residents without regular visitors, such as family, may be at an increased risk for abuse because regular visitors could notice and report potential warning signs of abuse, such as changes in their behavior or physical appearance. Residents who are cognitively impaired. Stakeholders in each of the five states said that cognitively impaired residents may be especially vulnerable to abuse because they often cannot speak or may have difficulty recalling recent events, and they are therefore less likely to be able to remember or describe what happened. In addition to noting that cognitively impaired residents may be at an increased risk of abuse, some stakeholders said that some cognitively impaired residents may be more likely to be perpetrators of abuse as their condition can have behavioral symptoms, such as physical aggressiveness. Residents mixed with widely different age groups. Stakeholders in four of the five states also noted that elderly nursing home residents who are mixed with widely differing age groups, such as young adults with mental illness, may be at a higher risk for incidents of abuse due to the different characteristics of these groups. Combining these two populations, which have differing needs, can also be challenging for staff. For example, staff may have more experience caring for elderly residents with complex needs, such as dementia, and they may not have the necessary skills or training to care for needs of younger residents, who require other types of complex care. This can create a stressful environment for staff, which is a risk factor for staff as potential perpetrators of abuse. Two stakeholders noted that younger residents who may have mental illness can have conflicts with older and frailer residents, potentially leading to abusive incidents between residents. Nursing home staffing characteristics. Stakeholders we interviewed in each of our five selected states noted that nursing homes with insufficient staffing, inadequate staff training, and inadequate staff screening may be at risk for abuse. Nursing homes with insufficient staff. Stakeholders in each of the five states said that nursing homes with insufficient staff may be at risk for abuse because there may not be enough staff attending to the needs of residents. Stakeholders noted that nursing homes have faced challenges hiring and retaining qualified staff and that, as a result, existing staff can feel overworked, stressed, or exhausted, which can lead to abusive behaviors. Staffing issues are not just risk factors for staff as perpetrators of abuse, but they can also limit a staff member’s ability to identify and report abuse. For example, insufficient staffing may mean that there are not enough available staff to notice signs of abuse in a timely fashion, such as noticing a resident’s bruises before they heal. Nursing homes with inadequate staff training on abuse. Inadequate staff training on abuse was noted by stakeholders we interviewed in four of the five states as a risk factor for abuse because; for example, staff may not know how to diffuse challenging situations with residents and identify and report abuse. As previously noted, recognizing abuse can be challenging and, even when abuse is identified, it is often not reported. Officials from all of the nursing homes that we visited said that they provide training to their staff on abuse, including on defining abuse, identifying or detecting different types of abuse, and reporting abuse. Staff members we spoke with at one nursing home said that, not only are they trained to look for physical signs of abuse, such as bruising, but they are also trained to observe changes in behavior that may be warning signs for abuse, such as a resident suddenly withdrawing from group activities. Staff at another nursing home said that they are also taught to ask another staff member for assistance when they are feeling frustrated or stressed by caring for a particular resident. In contrast, staff at another nursing home noted the challenges of not having these types of resources and said they are needed at their facility. Nursing homes with inadequate staff screening. Stakeholders in three of our five states said that inadequate staff screening can be a risk factor for abuse. Some stakeholders said that a thorough background screening can be time consuming. Further, because staff screening through background checks and the nurse aide registry is not coordinated across the country, there are gaps that could enable individuals who committed crimes in one state to obtain employment at a nursing home in another state, a concern that we previously reported. Staff from a nursing home we visited said the prevention of abuse “starts with hiring the right staff” and noted the importance of conducting background checks and checking references for prospective employees. The key challenges for abuse investigations most frequently identified by stakeholder groups in the five states we reviewed were underreporting of abuse, cognitive impairment of victims, lack of cooperation from nursing homes, and lack of agency coordination. (See table 5 for a description of these challenges.) Officials we interviewed from national organizations with knowledge of abuse in nursing homes also noted some of these same challenges. Underreporting of abuse. Stakeholders in each of the five states in our review noted that abuse in nursing homes may be underreported because residents or their families feel uncomfortable or fear retaliation from nursing home staff. For example, residents who were sexually abused may feel ashamed or embarrassed to report these incidents. In addition, residents may fear retaliation by the nursing home staff on whom they depend, which might include substandard care, exclusion from activities, or even eviction from the home. A fear of retaliation can also extend to nursing home staff, who may witness abuse by another staff member, but may be afraid to report it out of fear that they will lose their jobs or that they will face retaliation from co-workers. This underreporting creates challenges for investigators, who are unable to investigate if they do not know that abuse has occurred. Cognitive impairment of victims. Stakeholders in each of the five states in our review said that victims with cognitive impairment may not be able to give statements regarding the abuse or may not be considered reliable witnesses. For example, residents with dementia may not be able to remember the details of an abusive incident, and their memory of the details may deteriorate over the course of an investigation. Or, residents with dementia may report abuse that stems from traumatic memories from an incident that occurred earlier in their lives. One stakeholder said this can be a challenge for investigations because they do not know how much they can rely on a cognitively impaired resident’s statement, making it difficult for them to corroborate an abuse allegation. However, one stakeholder noted that, while it can be difficult to interview abuse victims with cognitive impairment, it is important to treat their allegations seriously and with credibility. One law enforcement stakeholder noted that interviews with these victims require special training. Lack of cooperation from some nursing homes. Stakeholders in each of the five states in our review said that some nursing homes may withhold, alter, or make it difficult for investigatory agencies to gain access to necessary, timely, or accurate information about alleged abuse. This may be, for example, because they may fear adverse publicity, litigation, or penalties from the state or CMS. In addition, as noted previously, nursing home staff may be fearful of losing their jobs. Stakeholders said that nursing home staff who witnessed abuse may be intentionally vague when interviewed by investigators; for example, by saying they cannot recall an incident. Some stakeholders also noted that nursing homes may delay investigators’ access to patient records, or they may even alter patient records in order to fill in information that should have been documented but was not at the time of the incident. One stakeholder we interviewed noted that the problem is not necessarily widespread—that some nursing homes are open about sharing information while others can be more difficult. Another stakeholder noted that a nursing home’s cooperation can sometimes depend on the seriousness of the allegation. Lack of agency coordination. Stakeholders in three of the five states in our review said that having multiple agencies involved in investigations, such as the state survey agency, law enforcement, the ombudsman, and, in some states, Adult Protective Services, can create challenges, including coordinating investigations and notifying one another about investigation outcomes. One stakeholder said they sometimes begin an investigation without realizing another investigatory agency has already started its own investigation. Further, stakeholders in two of the five states in our review said that CMS does not allow state survey agencies to share important investigatory information with law enforcement. (We discuss this issue in more detail later in this report.) We found that CMS: (1) cannot readily access data on the type of abuse or type of perpetrator, (2) has not provided guidance on what information nursing homes should include in facility-reported incidents, and (3) has numerous gaps in its referral process that can result in delayed and missed referrals to other entities. Together, these gaps affect critical points in CMS’s oversight of abuse in nursing homes including the prevention, identification, and timely investigation of abuse. CMS cannot readily access information on abuse or perpetrator type in its datasets and, as a result, lacks key information critical to understanding and appropriately addressing nursing home abuse with its oversight. Specifically, in two of CMS’s datasets—complaints/facility-reported incidents and deficiencies—agency officials told us they do not require the state survey agencies to record abuse and perpetrator type. As a result, we found that CMS’s data do not readily support CMS’s understanding of the types of abuse and perpetrators that are most prevalent in nursing homes. CMS officials told us they believe that the majority of abuse is committed by nursing home residents, and that physical and sexual abuse were the most common types; officials said they based this current understanding of abuse and perpetrator types on professional experience, literature, and ad hoc analyses of deficiency narrative descriptions. However, our review of a representative sample of abuse deficiency narratives from 2016 and 2017 found that staff were more often the perpetrators of deficiencies cited as abuse than residents or others, and that physical and mental/verbal abuse occurred most often in nursing homes, followed by sexual abuse. CMS officials noted that some incidents resulting from resident altercations—particularly those that do not show a willful intent to harm—may not have been cited as an abuse deficiency by some state survey agencies and may have been cited as other deficiencies not specified as abuse. This may have contributed to the difference between CMS’s understanding of the prevalence of resident to resident abuse and what their abuse deficiency data show. If CMS required information on abuse and perpetrator type to be recorded, the agency would have a better understanding of abuse in nursing homes. However, CMS officials told us they do not currently require the state survey agencies to specify abuse and perpetrator type because they consider the surveyor’s job to be identification and documentation of noncompliance. Additionally, CMS officials told us they have not conducted a systematic review to gather information on abuse and perpetrator type. This is inconsistent with federal internal control standards directing management to use quality information to achieve program objectives. Without the systematic collection and monitoring of specific abuse and perpetrator data, CMS lacks key information and, therefore, cannot take actions—such as tailoring prevention and investigation activities—to address the most prevalent types of abuse or perpetrators. All of the state survey agencies we spoke to told us that facility-reported incidents can lack key information that can cause potential delays in abuse investigations. Specifically, officials from each of the five state survey agencies told us that the facility-reported incidents they receive from nursing homes can lack key information that affects their ability to effectively triage incidents and determine whether an investigation should occur and how soon. Two state survey agencies we spoke with said they sometimes have to conduct significant follow-up with the nursing homes to obtain the information they need to prioritize the incident for investigation—follow-up that delays and potentially negatively affects investigations. For example, one state survey agency told us that a facility reported abuse involving two residents but did not initially report that the residents were injured, and that the facility did not file an addendum to the facility-reported incident to indicate resident injury. As a result of this incomplete information, the state survey agency did not properly prioritize this incident response. Despite federal law requiring nursing homes to self-report allegations of abuse, and covered individuals to report reasonable suspicions of crimes against residents, CMS has not provided guidance on what information should be included in these reports. Our review of CMS’s State Operations Manual found that CMS does not have guidance related to the information that nursing homes or covered individuals should report to the state survey agencies or local law enforcement; in contrast, it does contain guidance on the type of information members of the public should include in a complaint about nursing home quality to the state survey agency—and CMS makes a standardized complaint template form available on its website. The lack of guidance on the information that state survey agencies should collect on facility-reported incidents is inconsistent with federal internal control standards directing management to use quality information to achieve program objectives. CMS could outline basic information requirements that states must include on incident forms used by nursing homes and covered individuals to ensure the state survey agency is receiving the information it needs to accurately and quickly triage these incidents. CMS officials told us in November 2018 that they have efforts underway to examine guidance related to the information state survey agencies need to appropriately triage these facility-reported incidents and are developing a facility-reported incident template. Until the guidance and template are in place, these facility-reported incidents may lack key information that can cause potential delays in abuse investigations. CMS requires state survey agencies to make referrals to law enforcement and, if appropriate, to MFCUs when abuse is substantiated; however, we found numerous gaps in CMS’s referral process that can result in delayed and missed referrals. (See table 6.) Timing of abuse referrals. We found CMS’s requirements for when state survey agencies should report abuse to law enforcement and MFCUs lag behind the federal requirements for when covered individuals should make such referrals, and, as a result, referrals may be significantly delayed. Specifically, federal law requires covered individuals to immediately report reasonable suspicions of a crime against a resident that results in serious bodily injury to law enforcement and the state survey agency. Conversely, state survey agencies do not have to report suspicions of crime identified on complaints submitted to, and surveys conducted by, the state survey agency until the abuse has been substantiated—a process that can often take weeks or months. Officials from one law enforcement agency and two MFCUs that we interviewed told us the delay in receiving referrals limits their ability to collect evidence and prosecute cases—for example, bedding associated with potential sexual abuse may have been washed and wounds may have healed. This is consistent with the findings of our 2002 report, where we recommended that CMS should ensure that state survey agencies immediately notify law enforcement or MFCUs when nursing homes report allegations of physical or sexual abuse. One state survey agency in our review established more stringent guidelines than CMS by requiring the surveyors to notify law enforcement and the MFCU promptly upon receiving a complaint of abuse. CMS officials told us their state survey agency reporting requirements are based on a March 2002 policy. This is inconsistent with standards for internal control, which state that management should communicate quality information externally so that external parties can help the entity achieve its objectives. Tracking of abuse referrals. In addition to delays in referring cases to law enforcement and MFCUs, CMS officials also told us that CMS does not conduct oversight to ensure that state survey agency referrals to law enforcement and the MFCUs are occurring as required for substantiated abuse, and, as a result, CMS cannot ensure that state survey agencies are complying with reporting obligations. For example, an official from one of the five state survey agencies we interviewed said they had never made a referral to law enforcement or the MFCU, despite having substantiated allegations of abuse. The state survey agency official told us that they do not refer cases to law enforcement, and that law enforcement referrals are the responsibility of the nursing home. This is incompatible with CMS guidelines requiring that substantiated abuse be referred to law enforcement; however, CMS officials told us that they do not track whether state survey agencies make referrals to law enforcement and the MFCUs. This is inconsistent with federal standards for internal control, which state that management should establish and operate monitoring activities to monitor the internal control system and evaluate the results. Definition of substantiated abuse. We found confusion among some state survey agencies about CMS’s definition of what it means to substantiate an allegation of abuse—a challenge because substantiation is a trigger in the investigation process, and CMS requires state survey agencies to make referrals to law enforcement and staff registries when abuse is substantiated by evidence. As a result, there is a potential for substantiated abuse to not be reported and, subsequently, not referred to law enforcement or MFCUs for criminal investigation. Two of the five state survey agencies in our review told us they believed they could not substantiate an allegation unless they could also cite a federal deficiency. This is inconsistent with CMS’s guidance, which says that state survey agencies can substantiate that an allegation occurred without citing a federal deficiency and that, subsequently, these substantiated allegations must be referred to law enforcement and staff registries. For example, according to CMS guidance, if the state survey agency investigated and found evidence that a resident was abused, but the nursing home had taken preventive actions against the deficient practice, the state survey agency would then substantiate that the abuse occurred, but not cite a deficiency. However, state survey agencies may decide not to substantiate an abuse allegation verified by evidence if they believe no deficiency should be cited, such as if the nursing home had taken preventive action against the deficient practice, which could result in that abuse going unreported and not referred to law enforcement, MFCUs, or staff registries. Because substantiation of abuse is a critical trigger in abuse investigations, confusion around its interpretation could prevent these important next steps. CMS officials told us they are aware that the state survey agencies have varying interpretations of what it means to substantiate abuse. According to federal standards for internal control, management should internally communicate quality information to achieve the entity’s objectives. Information sharing. We also found that CMS’s guidance on state survey agency referrals contained in its State Operations Manual does not specify what incident information can be shared with local law enforcement, either in response to local law enforcement’s request for information or when the state survey agency refers substantiated findings of abuse to local law enforcement. As a result, both state survey and law enforcement agencies expressed confusion and frustration about what information can be shared and said delays have occurred that can impede law enforcement investigations. Officials from two state survey agencies told us that CMS does not allow them to share any information with law enforcement without a written request. For example, officials from one state survey agency said that they cannot share the name of the resident abused or the time when the incident occurred. One state survey agency said that information sharing can be uneven, and told us that law enforcement is required to share information with the state survey agencies, but the state survey agencies do not share their investigatory information with law enforcement. Officials from another state survey agency wrote to CMS notifying CMS of a change in their state survey agency protocol that would make the referral process timelier by providing un-redacted survey records of substantiated abuse to local law enforcement. However, in CMS’s 2017 written response to the survey agency, CMS told them that all written requests for these records must continue to be forwarded to CMS for processing in accordance with the federal Privacy Act. When we asked CMS officials what information state survey agencies can share with law enforcement in a referral, CMS explained that scenarios for requesting information can vary, and that CMS does not prescribe a specific method as it depends on the needs of the investigation. This lack of guidance is inconsistent with federal standards for internal control, which state that management should internally communicate quality information to achieve the entity’s objectives. While nursing home abuse is relatively rare, our review shows that abuse deficiencies cited in nursing homes are becoming more frequent, with the largest increase in severe cases. As such, it is imperative that CMS have key information critical to understanding abuse and that the agency’s oversight of nursing homes is strong. We found weaknesses in both CMS’s understanding of abuse and in its oversight that need to be addressed. Specifically, because CMS cannot readily access information on abuse or perpetrator types in its data, it lacks key information critical to taking appropriate actions to address the most prevalent types of abuse and perpetrators. In addition, CMS has not provided guidance on what information should be included in facility-reported incidents, contributing to a lack of information for state survey agencies and, subsequently, delays in their investigations. This lack of guidance related to facility- reported incidents is important in light of our findings that abuse deficiencies are identified most commonly through facility-reported incidents. We also found other gaps in CMS’s process related to ensuring timely referrals of abuse to law enforcement, tracking abuse referrals, defining abuse substantiation, and sharing information with law enforcement. These gaps affect CMS’s oversight of abuse in nursing homes—including the prevention, identification and timely investigation of abuse—and may limit CMS’s ability to ensure that nursing homes meet federal requirements for residents to be free from abuse. We are making the following six recommendations to the administrator of CMS: Require that abuse and perpetrator type be submitted by state survey agencies in CMS’s databases for deficiency, complaint, and facility- reported incident data, and that CMS systematically assess trends in these data. (Recommendation 1) Develop and disseminate guidance—including a standardized form—to all state survey agencies on the information nursing homes and covered individuals should include on facility-reported incidents. (Recommendation 2) Require state survey agencies to immediately refer complaints and surveys to law enforcement (and, when applicable, to MFCUs) if they have a reasonable suspicion that a crime against a resident has occurred when the complaint is received. (Recommendation 3) Conduct oversight of state survey agencies to ensure referrals of complaints, surveys, and substantiated incidents with reasonable suspicion of a crime are referred to law enforcement (and, when applicable, to MFCUs) in a timely fashion. (Recommendation 4) Develop guidance for state survey agencies clarifying that allegations verified by evidence should be substantiated and reported to law enforcement and state registries in cases where citing a federal deficiency may not be appropriate. (Recommendation 5) Provide guidance on what information should be contained in the referral of abuse allegations to law enforcement. (Recommendation 6) We provided a draft of this product to HHS for review and comment. In its comments, reproduced in appendix IV, HHS concurred with our six recommendations and identified actions it is taking to implement them. Specifically, HHS said that it will: (1) look into options for requiring state survey agencies to record data on abuse and perpetrator type so that HHS may assess trends in these data; (2) develop guidance that includes a list of standardized data elements to be included when nursing homes report facility-reported incidents and guidance specific to the reporting and tracking of facility-reported incidents involving abuse; (3) require state survey agencies to immediately refer complaints to law enforcement if a reasonable suspicion of a crime against a resident has occurred and share relevant survey information; (4) consider how to implement mechanisms for tracking law enforcement referrals; (5) identify opportunities to clarify in guidance situations where citing a federal deficiency may not be appropriate, but reporting the abuse is still required; and (6) develop a list of standardized elements that should be included when reporting an abuse allegation to law enforcement. HHS also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of HHS and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This appendix describes our scope and methodology for determining the trends and types of abuse occurring in nursing homes in recent years. For this examination, we reviewed CMS guidance and analyzed data from 2013 through 2017, which represented the most recent data for a 5-year period at the time of our review. Specifically, we first reviewed the CMS State Operations Manual’s Appendix PP that was in effect during our period of review to determine which federal standards and deficiency codes were relevant to resident abuse. We focused our analysis on the deficiency code to be used by state surveyors when a nursing home fails to keep a resident free from abuse, which encompasses mental/verbal, sexual, or physical abuse. Surveyors can also use other deficiency codes for abuse-related issues, such as a failure by the nursing home to train staff on issues related to abuse, either in conjunction with an abuse deficiency or without an abuse deficiency. Since these abuse-related deficiency codes do not necessarily represent incidents of abuse, but do represent situations where a nursing home’s inadequate policies could leave residents vulnerable to abuse, we conducted a limited analysis on the trends of these deficiencies, which is described in appendix II. For our analysis, we identified abuse deficiencies cited by surveyors in all 50 states and Washington, D.C., between 2013 and 2017, using data provided by CMS from its Certification and Survey Provider Enhanced Reports system. Specifically, we calculated the number of abuse deficiencies cited each year and determined how many of these abuse deficiencies were at each level of severity—no actual harm with a potential for minimal harm, no actual harm with a potential for more than minimal harm, actual harm, and immediate jeopardy—for each year. We compared the results for abuse deficiencies with the results for all types of deficiencies in each year. To avoid over-counting deficiencies, deficiencies that were for the same violation on the same day for the same facility were counted as a single deficiency. We then tracked (1) the origin of these abuse deficiencies and (2) enforcement actions implemented against nursing homes with these abuse deficiencies. Origin of abuse deficiencies. To identify trends in the origin of those abuse deficiencies—that is, whether the deficiency originated from a standard survey, complaint investigation, or a facility-reported incident investigation—we analyzed data provided by CMS from its Automated Survey Processing Environment Complaint/Incident Tracking System. Specifically, we matched the deficiencies with the complaint/incident data using provider number, survey date, and deficiency code. We found that some deficiencies were the result of a combination of complaints, facility-reported incidents, surveys, or all three. We counted those deficiencies as originating from each relevant category. Enforcement actions. To identify trends in the enforcement actions imposed and implemented against nursing homes with abuse deficiencies, we analyzed data provided by CMS from its Automated Survey Processing Environment Enforcement Manager. Specifically, we matched the deficiencies with the enforcement data using provider number, survey date, case identification number, and deficiency code. To avoid over-counting, deficiencies that share the same code and case identification number were counted as a single deficiency. For each year, we determined how many of the abuse deficiencies resulted in enforcement actions imposed or implemented, the severity of the abuse deficiencies with enforcement actions, and the types of enforcement actions implemented. We then examined these abuse deficiencies to determine the number of nursing homes that had abuse deficiencies, as well as the number of homes with repeated abuse deficiencies cited across the 5 years and the characteristics of those homes. We also determined the proportion of surveyed nursing homes in a given year that had an abuse deficiency. Nursing homes that had repeated abuse deficiencies. Since a nursing home can have more than one abuse deficiency cited in a given year, we determined the number of surveyed nursing homes each year that had at least one abuse deficiency, both nationally and by state. For each of those nursing homes, we determined if the home had an abuse deficiency repeated in multiple years and in two or more consecutive years. Nursing home characteristics. We attempted to identify commonalities among homes with multiple years of abuse deficiencies, homes with only a single year with an abuse deficiency, and surveyed homes without any abuse deficiencies throughout the 5- year period. Specifically, we matched deficiency data to CMS’s publicly available Provider of Services files and the Nursing Home Compare Provider Information files for each nursing home; and we examined bed size, non-profit or for-profit status, Five-Star Quality Rating System overall rating, Special Focus Facility designation, and urban or rural location. Finally, because abuse and perpetrator type are not readily identifiable in CMS’s data, we identified this information by reviewing the narratives written by surveyors that describe the substantiated abuse. Specifically, we obtained 1,557 narrative descriptions written by state surveyors for abuse deficiencies cited in 2016 and 2017 provided by CMS from its Automated Survey Processing Environment database. From that universe of abuse deficiency narratives, we selected a randomly selected representative sample of 400 narratives, and each narrative was reviewed by two separate reviewers who independently analyzed the text of each narrative to determine the abuse and perpetrator type according to the definitions that CMS implemented on November 28, 2017, in its State Operations Manual. Any disagreements between the two reviewers were resolved by a third independent reviewer. (See table 7.) For those narratives where the abuse type could not reasonably be categorized under an existing CMS definition, reviewers had the option to mark narratives as “other.” Furthermore, we analyzed the scope and severity for each narrative within our sample. CMS’s abuse deficiency code also included involuntary seclusion in the time period we examined and is defined in its November 22, 2017, guidance as “separation of a resident from other residents or from her/his room or confinement to her/his room (with or without roommates) against the resident’s will, or the will of the resident representative.” Our analysis of the narrative descriptions found that 3 percent of the abuse deficiency narratives in our sample were attributable to involuntary seclusion. We were unable to categorize the abuse and perpetrator type for about 11 percent of the deficiency narratives in our sample, because we determined the narrative description did not meet CMS’s abuse definition. We assessed the reliability of each of the datasets by checking for missing values and obvious errors and discussed them with CMS officials who were knowledgeable about the data. In the course of this assessment, we found some data limitations. Specifically, CMS officials told us that some state survey agencies may not have entered all facility- reported incidents into the Automated Survey Processing Environment Complaint/Incident Tracking System, while other state survey agencies did. We also found underreporting, as noted in our 2019 report, where the Oregon state survey agency was not entering all abuse-related complaints or facility-reported incidents into this same database—a problem that could exist in other states. In addition, CMS officials told us that it is possible there are additional incidents that may not have been represented in the abuse deficiency data during the period of our review. Specifically, CMS officials noted that some incidents resulting from resident altercations—particularly those that do not show a willful intent to harm—may not be cited as an abuse deficiency by some state survey agencies. We therefore consider the number of abuse deficiencies that resulted from complaints or facility-reported incidents to be a conservative estimate. After reviewing the possible limitations of these data, we determined the data were sufficiently reliable for the purposes of this reporting objective. This appendix describes trends in abuse-related deficiencies over the 5- year period from 2013 through 2017. We reviewed Centers for Medicare & Medicaid Services (CMS) guidance that was in effect during this period of review to determine which federal standards and deficiency codes were relevant to resident abuse. For the report, we focused our analysis on the deficiency code cited when state surveyors substantiate incidents of abuse, but there are also deficiencies that surveyors can cite for abuse-related issues, such as a failure by the nursing home to train staff on issues related to abuse, either in conjunction with an abuse deficiency or without an abuse deficiency. Since these abuse-related deficiencies do not necessarily represent incidents of abuse, but do represent situations where a nursing home’s inadequate policies could leave residents vulnerable to abuse, we also conducted a limited analysis on the trends of these deficiencies. Specifically, we analyzed CMS data to identify the number of abuse-related deficiencies cited in each year in all 50 states and Washington, D.C., and determined how many were cited at each level of severity—no actual harm with a potential for minimal harm, no actual harm with a potential for more than minimal harm, actual harm, and immediate jeopardy. We also tracked the source of these abuse-related deficiencies—that is, whether the deficiency originated from a standard survey, complaint investigation, or a facility-reported incident investigation. Finally, we compared the results for abuse-related deficiencies with the results for all types of deficiencies cited by surveyors in each year. From 2013 to 2017, we found that abuse-related deficiencies became slightly more common with a resulting increase in severity. Specifically, abuse-related deficiencies increased by about 9.9 percent over the 5-year period, from 4,899 deficiencies cited in 2013 to 5,383 deficiencies cited in 2017, but peaked in 2016 with 5,687 deficiencies. This increasing trend for abuse-related deficiencies is in contrast to the slight decrease in all deficiencies cited over the same period, but not nearly as high as the 103.5 percent increase in abuse deficiencies. In addition, the proportion of abuse-related deficiencies cited at the highest levels of severity— deficiencies causing actual harm to residents or putting residents in immediate jeopardy—fluctuated throughout the 5-year period. Specifically, about 6.1 percent of the 4,899 abuse-related deficiencies in 2013, about 5.6 percent of the 5,278 abuse-related deficiencies in 2015, and about 7.8 percent of the 5,383 abuse-related deficiencies in 2017 caused actual harm or immediate jeopardy. (See fig. 7.) We also found that over half of the abuse-related deficiencies each year were cited by surveyors as a result of standard surveys, and the rest were cited by surveyors as a result of either complaint or facility-reported incident investigations. This falls between what we found for abuse deficiencies—the majority were a result of either complaint or facility- reported incident investigations—and all types of deficiencies—the vast majority were a result of standard surveys. Over the 5 years, similar to abuse deficiencies and all types of deficiencies, the percentage of abuse- related deficiencies that resulted from standard surveys decreased while the percentage that resulted from both complaint and facility-reported incident investigations increased. Specifically, over the 5-year period, the percentage of abuse-related deficiencies resulting from standard surveys decreased by about 8.8 percentage points, complaint investigations increased by about 3.6 percentage points, and facility-reported incident investigations increased by about 5.3 percentage points. (See fig. 8.) Tables 8 and 9 provide state-level data on abuse deficiencies and the nursing homes that had abuse deficiencies cited in consecutive years. John E. Dicken, (202) 512-7114 or dickenj@gao.gov. In addition to the contact named above, Karin Wallestad (Assistant Director); Sarah-Lynn McGrath and Kathryn Richter (Analysts-in-Charge); Luke Baron; Summar Corley; Zosha Kandel; and Julianne Flowers made key contributions to this report. Also contributing were Laurie Pachter, Jennifer Whitworth, and Vikki Porter.", "summary": "Nursing homes provide care to about 1.4 million nursing home residents—a vulnerable population of elderly and disabled individuals. CMS, an agency within the Department of Health and Human Services (HHS), defines standards nursing homes must meet to participate in the Medicare and Medicaid programs. GAO was asked to review abuse of residents in nursing homes. Among other objectives, this report: (1) determines the trends and types of abuse in recent years, and (2) evaluates CMS oversight intended to ensure residents are free from abuse. GAO reviewed CMS's policies, analyzed CMS data on abuse deficiencies from 2013 through 2017, the most recent data at the time of our review, and interviewed officials from CMS and state survey agencies in five states, as well as other key stakeholders in those states such as ombudsmen and law enforcement officials. The states were selected for variation in factors such as number of nursing homes and role of other state agencies in abuse investigations. The Centers for Medicare & Medicaid Services (CMS) is responsible for ensuring nursing homes meet federal quality standards, including that residents are free from abuse. CMS enters into agreements with state survey agencies to conduct surveys of the state's homes and to investigate complaints and incidents. GAO analysis of CMS data found that, while relatively rare, abuse deficiencies cited in nursing homes more than doubled, increasing from 430 in 2013 to 875 in 2017, with the largest increase in severe cases. GAO also reviewed a representative sample of abuse deficiency narratives from 2016 through 2017. Physical and mental/verbal abuse occurred most often in nursing homes, followed by sexual abuse, and staff were more often the perpetrators of the abuse deficiencies cited. CMS cannot readily access information on abuse or perpetrator type in its data and, therefore, lacks key information critical to taking appropriate actions. GAO also found gaps in CMS oversight, including: Gaps in CMS processes that can result in delayed and missed referrals. Federal law requires nursing home staff to immediately report to law enforcement and the state survey agency reasonable suspicions of a crime that results in serious bodily injury to a resident. However, there is no equivalent requirement that the state survey agency make a timely referral for complaints it receives directly or through surveys it conducts. CMS also does not conduct oversight to ensure that state survey agencies are correctly referring abuse cases to law enforcement. Insufficient information collected on facility-reported incidents. CMS has not issued guidance on what nursing homes should include when they self-report abuse incidents to the state survey agencies. Officials from all of the state survey agencies in GAO's review said the facility-reported incidents can lack information needed to prioritize investigations and may result in state survey agencies not responding as quickly as needed. GAO is making six recommendations, including that CMS: require state survey agencies to submit data on abuse and perpetrator type; require state survey agencies to immediately refer to law enforcement any suspicion of a crime; and develop guidance on what abuse information nursing homes should self-report. HHS concurred with all of GAO's recommendations and identified actions it will take to implement them.", "document_type": "gao"}
{"report": "This section provides an overview of (1) the legal framework governing mixed HLW, (2) the status of EM’s IWTU reengineering project, (3) EM’s requirements for capital asset projects and operations activities, (4) DOE’s policy for the review of projects with start-up risks, and (5) our best practices for assessing cost and schedule estimates. The treatment and disposal of mixed HLW at INL is governed by a number of federal laws that define the roles of federal agencies and states in managing mixed HLW, as well as cleanup agreements among DOE, the state of Idaho, and other parties. DOE primarily regulates radioactive components of HLW under the Atomic Energy Act of 1954, as amended, and the Nuclear Waste Policy Act of 1982, as amended. These acts define HLW as (1) the highly radioactive waste material resulting from the reprocessing of spent nuclear fuel, including liquid waste produced directly in reprocessing and any solid material derived from such liquid waste that contains fission products in sufficient concentrations, and (2) other highly radioactive material that the Nuclear Regulatory Commission determines by rule, consistent with existing law, requires permanent isolation. DOE considers calcine waste HLW because it is solidified liquid waste produced during the reprocessing of spent nuclear fuel. EM manages the SBW as mixed HLW because, according to reports from DOE and National Academies, (1) the SBW was produced in the later stages of spent nuclear fuel reprocessing, (2) the tanks in which the SBW is stored previously held HLW, (3) the SBW is stored in a location at INL where waste is managed as HLW, and (4) the waste contains hazardous chemicals subject to RCRA and EPA’s implementing regulations or authorized state programs that operate in lieu of the federal program. HLW must be disposed of in a geologic repository unless the Nuclear Regulatory Commission approves an alternative disposal site. DOE Order 435.1 and Manual 435.1-1 describe the department’s policy and requirements for managing DOE’s radioactive waste, including HLW, to ensure that it is managed in a manner that is protective of worker and public health and safety and the environment. Manual 435.1-1 also established processes to determine whether waste resulting from reprocessing spent nuclear fuel can be managed as transuranic waste or low-level waste if certain criteria are met, which is referred to as a determination that the waste is incidental to reprocessing. According to the manual, HLW is waste incidental to reprocessing if, among other things, the waste has been processed, or will be processed, to remove key radionuclides to the maximum extent technically and economically practicable. Hazardous components of mixed HLW are regulated by EPA or authorized states under RCRA. EPA’s regulations require hazardous waste to meet certain treatment standards before land disposal of the waste unless a variance is granted. The regulations specify that the treatment standard (i.e., the required method for treatment) for Idaho’s mixed HLW is vitrification—the immobilization of waste in glass. Where EPA has authorized states to implement hazardous waste management programs, those state programs operate instead of the federal program. EPA, under RCRA, has authorized the state of Idaho to administer its own hazardous waste management program. EPA has also authorized New Mexico to administer its own hazardous waste management program. Pursuant to such authorization, New Mexico’s Environment Department issues the permit for hazardous waste storage and disposal at WIPP under the New Mexico Hazardous Waste Act. As of March 2019, EM’s IWTU reengineering project was in phase two of the four-phased approach to get the facility operational, according to EM Idaho Cleanup Project officials. According to project reports, phase one focused on identifying fixes to resolve problems with the facility’s equipment and waste treatment process, for example, by performing engineering analyses and chemistry studies. Phase two has focused on implementing these fixes, for example, by modifying a piece of equipment that separates solidified waste before it is moved to storage canisters, according to the contractor’s project plan. Figure 1 summarizes the four- phased approach for the IWTU reengineering project. According to EM documents, as of February 2019 total expenditures on phases one and two were approximately $150 million, about $64 million more than original costs estimated for those two phases combined, and the project was over 1 year behind schedule. Phase two has taken longer and cost more than initially estimated because of additional problems and required modifications to the facility as the work has progressed, according to EM Idaho Cleanup Project officials. Appendix II provides information on the actual costs of phases one and two compared to estimated costs. As previously noted, EM officials with the Idaho Cleanup Project estimated in March 2019 that phase three may begin in summer 2019. Further, these officials stated that phase three will involve a 6- month outage to continue implementing changes to the facility prior to the start of a 60-day performance test using a simulated waste form. EM Idaho Cleanup Project officials stated that phase four could begin in early 2020 and that EM and Fluor Idaho had yet to determine whether an outage would need to occur before starting testing with a small amount of the SBW. EM divides its cleanup work into capital asset projects and operations activities, two types of activities governed by different applicable project management policies: Capital asset projects. DOE Order 413.3B governs EM’s program and project management activities for the acquisition of capital assets, with the stated goal of delivering fully capable projects within the planned cost, schedule, and performance baseline. The order establishes five critical decision points of project development that each end with a major approval milestone that cover the life of a project. The order specifies requirements that must be met, including developing and managing project cost and schedule estimates to move a project past each critical decision milestone. EM capital asset projects include construction projects and cleanup projects, such as soil and water remediation and facility decommissioning and demolition. Operations activities. Operations activities are recurring facility or environmental operations, as well as activities that are project-like, with defined start and end dates, according to EM policy. EM operations activities include operating waste processing facilities and the stabilization, packaging, transportation, and disposition of nuclear waste. EM manages operations activities based on requirements listed in a cleanup policy that it issued in July 2017. In February 2019, we found that EM cleanup site managers have discretion in how to classify cleanup work because DOE and EM have not established requirements on what work should be managed as an operations activity under EM’s cleanup policy or as a capital asset project under DOE Order 413.3B. Further, we found that operations activities have less stringent management requirements than capital asset projects. We recommended that EM establish requirements for classifying work as an operations activity and revise its cleanup policy to follow program and project management leading practices. DOE generally agreed with our recommendations. Beginning in January 2005, EM managed the development and construction of the IWTU facility as a capital asset project. Once EM determined that construction on the facility was complete in April 2012, the project exited the capital asset oversight process established in DOE Order 413.3B and has since been managed as an operations activity, according to EM Idaho Cleanup Project officials. DOE officials also told us that the IWTU reengineering project has been managed as an operations activity because the facility has been constructed and is now in a period of maintenance and repair. Figure 2 shows a picture of the exterior of the IWTU facility. In August 2016, DOE’s Deputy Secretary of Energy issued a memorandum establishing a new oversight requirement for selected projects for which an extended period of transition to operations is likely— the phase after construction is complete but before full operational capability is attained—called the operational release milestone. According to the memorandum, DOE created the operational release milestone in the department’s project life cycle to provide additional oversight after the completion of the project under DOE’s Order 413.3B. DOE officials from the Office of Project Management stated that the operational release milestone was largely created in response to EM’s experience with the IWTU facility not operating as expected. Under these new requirements, program offices are to provide DOE’s Project Management Risk Committee (PMRC) with regular updates on selected projects until full operational capability of each facility is attained. Specifically, program offices are required to (1) develop and execute a plan that describes how the program will reach operational capability, which is referred to as an operational release plan, and (2) provide progress updates to the PMRC on the project, as described below. Operational release plan. Officials from DOE’s Office of Project Management—which serves as the secretariat for the PMRC—stated that the purpose of the operational release plan is for the program office to describe what steps are required for the project to reach its operational capability. According to EM’s guidance, the operational release plan should present the key processes, activities, interrelationships, risks, management and oversight, decision milestones and approvals, and overall schedule to achieve operational release. Progress updates. According to the memorandum and the PMRC’s standard operating procedures, program offices are to provide the PMRC with quarterly progress updates on selected projects, including lessons learned, until full operational capability is attained. The GAO’s cost guide and schedule guide compiled best practices corresponding to the characteristics of high-quality and reliable cost and schedule estimates. According to the cost guide, a high-quality, reliable cost estimate has four characteristics: comprehensive, well-documented, accurate, and credible. A comprehensive cost estimate has enough detail to ensure that cost elements are neither omitted nor double-counted. If a cost estimate is not comprehensive (that is, complete), then it cannot fully meet the other characteristics (i.e., well-documented, accurate, or credible). In addition, according to the schedule guide, a high-quality, reliable schedule has four characteristics: comprehensive, well- constructed, controlled, and credible. A comprehensive schedule captures all government and contractor activities necessary to accomplish a project’s objectives, and a well-constructed schedule sequences all activities using the most straightforward logic possible. If a schedule is not comprehensive, with all activities accounted for, it is uncertain whether all activities are scheduled in the correct order, resources are properly allocated, missing activities will appear on the critical path, or a schedule risk analysis can account for all risk. If a schedule is not well-constructed, it will not be able to properly calculate dates and predict changes in the future, among other things. EM has not fully followed selected project management best practices for cost and schedule estimates for the IWTU reengineering project. EM generally followed best practices for a reliable EVM system to measure the performance of the reengineering project. However, in analyzing IWTU reengineering project data from March 2017 through February 2018, we found that the system is producing unreliable data, which may limit EM’s ability to measure the project’s performance. Further, EM has taken some steps toward meeting requirements under DOE’s process for monitoring projects with start-up risks. EM has not fully followed (i.e., has partially met) selected best practices in developing the cost and schedule estimates we reviewed for phases one and two of the IWTU reengineering project and future planned IWTU operations. We made the following observations based on our analysis of these cost estimating documents and a March 2018 project schedule: Comprehensive cost estimate (partially met): EM partially met best practices for a comprehensive cost estimate. According to our cost guide, a comprehensive cost estimate should reflect the project’s technical requirements and current schedule and account for all possible costs. While the cost estimate was based on documented technical information, it was not based on a standardized work breakdown structure. Without a standard, product-oriented work breakdown structure to facilitate the tracking of resource allocations and expenditures, EM may not be able to reliably estimate the cost of future similar programs. While assumptions are listed in EM’s documents describing the cost estimates, no document discusses whether the assumptions came from inputs from technical subject matter experts or whether the assumptions are associated with specific risks. Since assumptions are best guesses, best practices state that the risk associated with any of these assumptions changing need to be identified and assessed. Further, the IWTU reengineering project’s cost estimate was not complete because it did not account for all possible costs. According to our cost guide, a life cycle cost estimate provides an exhaustive and structured accounting of all resources and associated cost elements required to develop, produce, deploy, and sustain a particular program. The project’s cost estimate did not reflect all life cycle costs, in part because estimates for phases three and four of the project had not been developed at the time of our review. Best practices state that all costs be included in an estimate, even in early stages, such as at a rough order of magnitude. EM officials from the Idaho Cleanup Project said that a cost estimate was not developed for the total cost of the IWTU reengineering project because of the approach for negotiating the cost and schedule baseline prior to the start of each phase. Without developing a cost estimate for the IWTU reengineering project that is comprehensive (e.g., accounts for all possible costs), EM will not have reasonable assurance that it can successfully plan program resource requirements. Well-constructed schedule estimate (partially met): EM partially met best practices for a well-constructed schedule. According to our schedule guide, a well-constructed schedule includes activities that are logically sequenced; a valid critical path; and a reasonable amount of total float, meaning an accurate reflection of the schedule’s flexibility. EM’s March 2018 schedule had minimal sequencing issues and a continuous critical path, with the exception of an external dependency, and the critical path was free of lags and constraints. However, there were long duration activities on the critical path that should be reevaluated to determine if they can be broken into more manageable pieces. Without a valid critical path, management cannot focus on activities that will detrimentally affect the key program milestones and deliveries if they slip. Additionally, the schedule estimate included unreasonably large values of positive and negative float. According to best practices, a schedule should identify reasonable values of float so that the schedule’s flexibility can be determined to help accommodate for delays. EM officials from the Idaho Cleanup Project explained that the amount of total float was a result of the methods they used to structure the logic of the schedule estimate, which according to our best practices may have caused the schedule to be overly optimistic. According to scheduling best practices, without accurate values of total float, the schedule cannot be used to identify activities that could be permitted to slip and thus release and reallocate resources to activities that require more resources to be completed on time. Inaccurate values of total float also falsely depict true program status, which could lead to decisions that may jeopardize the program. In addition, the March 2018 schedule contained 14 activities with large amounts of negative float, meaning that these activities were behind schedule. Without fully developing a well-constructed schedule estimate for the IWTU reengineering project, EM will not have reasonable assurance that it can successfully achieve its plans to reengineer the IWTU and begin treatment of the SBW without further delays. Comprehensive schedule estimate (substantially met): EM substantially met best practices for a comprehensive schedule. According to our schedule guide, a comprehensive schedule includes all activities for both the government and its contractors to accomplish their objective, assigns resources (e.g., labor and materials) to all activities, and establishes how long each activity will take. EM’s March 2018 schedule substantially captured all activities, but it may not have been planned to the level of detail for the work necessary to accomplish a program’s objectives as defined in the program’s work breakdown structure. For example, the schedule had activities that were described as level of effort but were not assigned the level of effort activity type. Level of effort activities represent effort that has no measurable output and, according to best practices, should be clearly marked so they do not interfere with the critical path. Further, the schedule substantially met the best practice of assigning resources to all activities. For example, the schedule assigned resources to specific materials and equipment as well as to travel, training, and labor. Appendix II contains the full results of our analysis of selected best practices for the cost and schedule of the IWTU reengineering project. As previously noted, EM is managing the IWTU reengineering project as an operations activity. We reported in February 2019 that EM manages operations activities using less stringent requirements than those used for capital asset projects, posing cost and schedule risks. For example, under EM’s 2017 cleanup policy, there is no requirement for operations activities to follow best practices for cost estimates developed during contract execution. We recommended that EM review and revise its 2017 cleanup policy to include project management leading practices related to scope, cost, schedule performance, and independent reviews. DOE concurred with our recommendation and stated that EM was already in the process of reviewing its policy for necessary updates, revisions, and modifications, and that EM would consider our recommendation, as appropriate, during this process. EM officials with the Idaho Cleanup Project acknowledged that they do not have an estimate for the total cost or a completion date for the IWTU reengineering project or a schedule for when waste treatment operations will begin and be completed. An EM Idaho Cleanup Project official told us that Fluor Idaho submitted cost and schedule estimates for phases three and four of the reengineering project in January 2019 and that EM requested an independent cost estimate for this work from the Defense Contract Audit Agency, with contract negotiations between EM and Fluor Idaho for these phases estimated to begin in spring 2019. In addition, EM officials from the Idaho Cleanup Project acknowledged that a schedule for waste treatment operations at the project has not been developed. Further, these officials noted that design modifications to the IWTU are expected to reduce its operating capability, lengthening the time needed to treat the SBW. As a result, EM and Fluor Idaho plan to renegotiate the cost of their contract related to the treatment of the waste in the project, according to EM Idaho Cleanup Project officials. Specifically, because of the modifications to the project, the rate at which the SBW is treated will be slower than initially estimated, according to EM officials from the Idaho Cleanup Project. Treatment of all 900,000 gallons of the SBW was originally estimated to be completed in 10 months, but agency officials now estimate that treatment may take from 3 to 7 years— as much as eight times longer than originally planned. As previously noted, EM has already experienced approximately $64 million in added costs and, as of February 2019, a delay of over 1 year. Without fully following best practices for a comprehensive cost estimate and well- constructed schedule estimate for SBW waste treatment operations, EM cannot be assured that it has reliable cost and schedule estimates for decision-making, placing it at risk of continued cost overruns and delays in achieving its plans to reengineer the IWTU and begin treatment of the SBW. We analyzed IWTU reengineering project data for March 2017 through February 2018 from EM’s EVM system and found that while EM has followed (i.e., fully met or substantially met) some best practices for a reliable EVM system, the system is producing unreliable data. These unreliable data may limit EM’s ability to measure the project’s performance. EVM is a management tool used to measure the value of work accomplished in a given period and compare it with the planned value of work scheduled for the same period and with the actual cost of the work accomplished. EVM data can alert project managers to potential problems sooner than expenditures alone can, and EVM’s use as a management tool is considered a best practice for conducting cost and schedule performance analysis for projects, according to our cost guide. EM requires the use of an EVM system under its contract with Fluor Idaho for the Idaho Cleanup Project. Overall, we found that EM followed best practices to ensure that its EVM data for the IWTU reengineering project were (1) comprehensive and (2) used by leadership for decision-making. However, EM did not follow (i.e., partially met) best practices to ensure that the data resulting from the EVM system are reliable. Specifically: EM substantially met best practices for a comprehensive EVM system by, for example, requiring the contractor’s EVM system to comply with the guidelines established by the Earned Value Management Systems EIA-748-D Intent Guide; EM conducted a compliance review of Fluor Idaho’s EVM system in March 2017 and found some areas in need of improvement. In addition, EM has an EVM surveillance system in place under its contract with Fluor Idaho, and EM officials from the Idaho Cleanup Project stated that they review data from the EVM system each month. EM substantially met best practices ensuring that leadership uses the EVM data for decision-making. For example, Fluor Idaho updated data in its EVM system monthly during the period we reviewed, and EM reported issues in a monthly review briefing between EM and the contractor, according to EM Idaho Cleanup Project officials. Agency management also tracked the causes of cost and schedule variances in the data. However, the monthly reports did not contain all the information that best practices recommended. Specifically, the performance measurement baseline was not included in the contractor performance reports provided, so we could not determine how the performance measurement baseline changed as the project evolved. EM partially met best practices ensuring that the EVM system provides reliable data because, for instance, the system contained numerous anomalies, leading the system to produce unreliable data. Specifically, we found one or more anomalies present in all months of data reviewed, such as missing or negative values. While EM was able to explain the causes for most of these anomalies, negative values should occur rarely, if ever, in EVM reporting because they imply the undoing of previously scheduled or performed work. According to best practices, all anomalies should also be identified and the reason for each should be fully explained in EM’s monthly EVM reports. However, EM did not document the reasons for these anomalies in its monthly reports. EM officials from the Idaho Cleanup Project said that most of the anomalies in the data were due to the phase two estimate including authorized unpriced work—that is, additional work that EM agreed to let the contractor perform without first negotiating or independently verifying the costs. If errors in EVM reports are not detected, then EVM data will be skewed, resulting in bad decision-making and limiting EM’s ability to use the EVM system to measure project performance. Appendix III provides detailed information on EM’s performance on each EVM best practice. An EVM system that produces unreliable data may contribute to EM’s challenges in measuring the performance of its operations activities. Our findings in this regard are consistent with our prior reports examining EM’s use of EVM systems in other contracts. For example, in February 2019 we reviewed the use of EVM systems in the 21 contracts EM uses to execute its operations activities, including Fluor Idaho’s contract for the cleanup at INL, and found that EM has not followed best practices to ensure that these systems (1) are comprehensive, (2) provide reliable data, and (3) are used by EM leadership for decision-making. We recommended that EM update its cleanup policy to require that EVM systems be maintained and used in a way that follows EVM best practices, such as ensuring the reliability of the data in the system. Without following best practices for ensuring EVM data reliability for the IWTU reengineering project’s EVM system, EM leadership may not have access to reliable performance data with which to make informed decisions as it manages billions of dollars’ worth of cleanup work and provides information to Congress and other stakeholders on the cleanup work every year. In 2016, DOE instituted independent review requirements to monitor facilities with commissioning or start-up risks, and EM has taken some steps toward meeting those requirements for the IWTU reengineering project. As previously noted, DOE’s policy requires program offices to (1) develop and execute an operational release plan and (2) provide progress updates to the PMRC on the project each quarter. We made the following observations on EM’s actions to meet these requirements for the reengineering of the IWTU project: EM developed an operational release plan for the IWTU project in December 2016, which preceded EM’s developing guidance for these plans. We found that the operational release plan included the majority of elements that EM’s guidance later required. EM has provided five progress update briefings to the PMRC on the IWTU reengineering project, according to DOE documents, but these briefings have not occurred each quarter as required by DOE’s policy. Officials from DOE’s Office of Project Management told us that briefings generally occur when progress has been made on a project. EM’s guidance for operational release plans also states, with regard to progress update briefings, that an alternate reporting schedule may be proposed for PMRC approval. The PMRC made recommendations in three of these five briefings. For example, the PMRC recommended that EM revisit and review documents to ensure that the delegated authority is clear, current, and appropriate prior to facility start-up and the introduction of radioactive materials. According to documentation prepared following EM’s most recent briefing to the PMRC in February 2019, the PMRC recommended an update on the project in July 2019. Based on our review of EM documentation and plans, the agency does not have a strategy or timeline to address its three main challenges for disposing of the SBW or for identifying an alternative disposal pathway. EM identified WIPP as its preferred disposal site for the SBW in a 2005 Record of Decision document, but in March 2019 EM officials told us that a final decision on the disposal path for the SBW had not been made. The three main challenges EM faces in its plan to dispose of the SBW at its preferred disposal site are: (1) the permit for WIPP prohibits the SBW from being disposed of at WIPP, (2) federal law prohibits HLW from being disposed of at WIPP, and (3) there are existing capacity limitations to disposal at the WIPP facility. EM has taken some steps to address these challenges, as discussed further below. WIPP permit’s prohibition of the disposal of certain tank waste. New Mexico amended its permit for WIPP in 2004 to prohibit waste that has ever been managed as HLW, including the SBW at INL, from being disposed at WIPP unless the disposal of such waste is specifically approved through a permit modification. In 2013, DOE and its contractor responsible for operating and managing the facility filed a request with the state of New Mexico to modify the WIPP permit to remove this prohibition, which could allow the SBW to be disposed of at WIPP if EM determined that the SBW is waste incidental to reprocessing. However, the process was put on hold following the suspension of operations at WIPP in 2014, according to officials from DOE’s Carlsbad Field Office and New Mexico’s Environment Department. In April 2019, officials from New Mexico’s Environment Department said that they anticipated holding discussions with DOE and its contractor for the facility regarding the prohibition after the renewal of the WIPP permit in July 2020. However, a representative from a New Mexico environmental organization said that this proposed modification would likely face strong public opposition. This representative noted that previous DOE attempts to expand the types of waste that could be disposed of at WIPP caused significant public concern in New Mexico. Further, New Mexico Environment Department officials told us that processing permit modifications of this nature would likely require public hearings and opportunities for input and may take as long as 2 years or more to complete. Federal statutory prohibition on HLW disposal at WIPP. The Waste Isolation Pilot Plant Land Withdrawal Act prohibits disposal of HLW at WIPP. Therefore, to enable EM to dispose of the SBW at WIPP, the SBW would need to be classified as non-HLW, or the act would need to be amended to remove the prohibition. DOE has a process for determining that certain waste resulting from reprocessing spent nuclear fuel, such as the SBW and calcine waste, could be managed as either transuranic waste or low-level waste, which are not HLW. Under DOE Order 435.1 and Manual 435.1-1, DOE may determine that waste is incidental to reprocessing and therefore manage the waste as transuranic waste or low-level waste if it meets certain criteria. EM began developing documentation supporting a waste incidental to reprocessing determination for the SBW in 2001. For example, in September 2001, EM requested consultation from the Nuclear Regulatory Commission, which oversees the nuclear power industry, on a draft waste incidental to reprocessing determination so that the SBW could be managed as transuranic waste and disposed of at WIPP rather than in an HLW repository. DOE’s Authority to Determine That Certain Waste Is Not HLW In 2002, while litigation over the Department of Energy’s (DOE) authority to use DOE Order 435.1 and Manual 435.1-1 was pending, DOE sought enactment of legislation clarifying its authority to manage portions of tank waste that have low levels of radioactivity as low- level waste. In response, Congress enacted section 3116 of the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 in October 2004. Under section 3116, radioactive waste resulting from the reprocessing of spent nuclear fuel is not high- level waste (HLW) if the Secretary of Energy, in consultation with the Nuclear Regulatory Commission, determines that it meets specified conditions. These conditions include that the waste does not require disposal in a deep geologic repository and has had highly radioactive radionuclides removed to the maximum extent practical. However, section 3116 only applies to waste stored at DOE sites in Idaho and South Carolina that is not transported from those states. Therefore, DOE cannot use section 3116 to classify the sodium-bearing waste (SBW) as transuranic waste for disposal as DOE’s agreements with Idaho require the SBW to be removed from the state. However, DOE’s authority to use Order 435.1 and Manual 435.1-1 to classify the SBW and other waste from reprocessing as non-HLW was challenged in a federal lawsuit in 2001, resulting in EM suspending its development of the waste incidental to reprocessing determination. Following the dismissal of the lawsuit on procedural grounds, EM restarted the internal process for developing the waste incidental to reprocessing determination for the SBW, according to EM officials and documents. For example, EM identified the waste incidental to reprocessing determination for the SBW as a priority item for executive decision-making in a 2017 EM study on mission operations. Internal discussions about this determination continued between EM and DOE into 2018, but the waste incidental to reprocessing determination was not finalized, according to EM officials. In October 2018, EM published a notice in the Federal Register seeking public comment on its proposed interpretation of the statutory definition of HLW, which EM officials said could help the agency make a decision about the classification of the SBW. EM also published a supplemental notice in June 2019 to modify the interpretation and provide additional information to the public, such as on the role of the Nuclear Regulatory Commission and states. Table 1 presents the statutory definition, the proposed interpretation from the October 2018 Federal Register notice, and the modified interpretation from the June 2019 Federal Register notice. EM officials told us that under the new interpretation, waste would be disposed of in accordance with its characteristics (which determines risk) instead of solely based on the source of the waste (which does not determine risk). Stakeholders, including members of the public, state and local governments, tribes, and the Nuclear Regulatory Commission, expressed a range of perspectives about EM’s proposed interpretation in public comments. For example, some stakeholders submitted comments expressing concern about the Nuclear Regulatory Commission being excluded from the determination of what is HLW under the interpretation. These comments also stated that the interpretation is contrary to federal law and that the interpretation will elicit legal challenges. Other stakeholders expressed support for the interpretation in comments submitted to EM stating, for example, that the proposed interpretation could accelerate the cleanup of tank waste at DOE sites and result in cost savings. According to an EM document, potential benefits of the interpretation, if implemented, include a more risk-based approach to waste classification, which could provide a more cost-effective and timely approach to DOE’s cleanup mission. However, EM officials stated that it was premature to discuss the administrative actions, such as revising orders or regulations that would be required to implement the new interpretation. The June 2019 Federal Register notice states that DOE will consider what actions may be needed and appropriate to update applicable DOE directives, such as Order 435.1 and Manual 435.1-1, in light of this interpretation and address any revisions in future actions. EM officials also told us that they did not have a timeline for implementing the new interpretation. Further, EM officials stated that if the HLW interpretation is implemented, alternative disposal options could also be considered for the SBW, but they declined to specify what those options could be. Limitations on disposal at WIPP. Further, existing limitations in the disposal space at WIPP could affect the disposal of the SBW at the facility. We reported in September 2017 that DOE does not currently have sufficient disposal space at WIPP for the waste identified in its 2016 annual inventory report—a document that tracks waste intended to be disposed of at the facility. Specifically, DOE will need to expand the repository to accommodate this waste as well as other potential waste, such as the SBW, for which DOE has yet to determine if it meets all of WIPP’s waste acceptance criteria. In March 2019, DOE officials stated that WIPP could be expanded within the current Waste Isolation Pilot Plant Land Withdrawal Act boundary for the site to accommodate the current planned waste and additional waste inventories. Specifically, DOE officials said that mining for a new disposal panel and design work for additional disposal panels was under way, and mining of the additional panel was scheduled to commence in 2021. Further, in September 2017 we also reported that additional potential waste beyond what is captured in the inventory could exceed WIPP’s statutory capacity. However, in December 2018, New Mexico’s Environment Department approved a modification to the WIPP permit—which was requested by DOE and its contractor that operates and manages WIPP—that will change the way waste volume is calculated to exclude empty space inside waste packing. According to DOE officials, this means that additional waste can be disposed of at WIPP under the existing statutory limit. Further, DOE officials stated that the revised counting methodology will reduce an overstatement in the volume of record for emplaced waste by about 30 percent. However, in January 2019 three environmental organizations filed lawsuits challenging the modification, which the court consolidated and, in May 2019, stayed pending mediation. EM officials said that if the office is not able to dispose of the SBW at WIPP, its plan is to dispose of the SBW—once it is treated to a solid form in the IWTU—with the calcine waste in an HLW geologic repository. However, there is still no HLW disposal site in the United States. In 2008, DOE submitted a license application to the Nuclear Regulatory Commission for an HLW repository at Yucca Mountain, Nevada, about 100 miles northwest of Las Vegas. In 2010, however, DOE terminated its efforts to obtain a license for the Yucca Mountain repository. Under the 1995 settlement agreement with the state of Idaho, DOE is required to treat the SBW so that it is ready for disposal outside of the state by a target date of 2035. An EM official responsible for the disposition of the SBW at INL told us that EM has not developed a strategy, including a timeline, for addressing challenges, including the WIPP permit prohibition, the federal law prohibition, and existing capacity limitations, that could affect EM’s ability to meet this target date. According to standards for internal control, federal agency management should identify, analyze, and respond to risks related to achieving a defined objective. Until it develops such a strategy, including a timeline, to implement the actions required to achieve its preferred disposal pathway, or an alternative, for the SBW, EM will not have reasonable assurance that it can achieve its preferred plan for disposal or begin identifying an alternative. Moreover, if EM implements its new interpretation of HLW and uses this definition to classify the SBW as non- HLW, there is significant risk for extended litigation, which may delay to EM’s plans to dispose of the SBW at its preferred disposal site. EM faces challenges implementing its selected treatment technology for calcine waste and faces uncertainties with a waste disposal pathway. As a result, the agency is suspending further development of its plan to treat calcine waste for land disposal, according to EM documents and officials. EM Idaho Cleanup Project officials told us that the agency is continuing to make progress toward its milestones for calcine waste disposal by considering alternatives for processing the waste for land disposal and conducting a pilot project to remove it from the oldest storage vessel. However, EM does not have a strategy or timeline for determining its next steps for the ultimate treatment and disposal of calcine waste. Because of challenges with implementing its chosen treatment technology as well as selecting a potential waste disposal pathway, EM is suspending further development of its plan to treat calcine waste for land disposal, according to EM documents and officials. In December 2009 EM identified hot isostatic pressing as its preferred treatment technology for preparing the calcine waste for land disposal outside of Idaho. Hot isostatic pressing is a manufacturing process that applies elevated temperatures and pressurized gas to materials in a containment vessel, resulting in a ceramic waste form. EM officials from the Idaho Cleanup Project told us that while hot isostatic pressing is a technology used in other industries, such as in industrial manufacturing, it has not been used before to treat HLW. Further, hot isostatic pressing would require a variance or an EPA regulation establishing a new treatment standard prior to land disposal. According to EM Idaho Cleanup Project officials and agency documents, EM selected hot isostatic pressing as the treatment technology because EM’s analyses assumed it would result in significant cost savings for disposal at Yucca Mountain compared to other methods. In February 2011, an independent DOE review team issued a preliminary technology readiness assessment for using hot isostatic pressing for calcine waste treatment as part of DOE’s process for managing capital asset projects. The review team identified several concerns, such as whether components of the technology would be mature enough to meet EM’s planned milestones and challenges with EM’s decision to retrofit and reuse the IWTU for the calcine waste treatment mission. EM officials from the Idaho Cleanup Project said that the decision to retrofit and reuse the IWTU for the calcine waste after treating the SBW resulted from reluctance within DOE to build another “first-of-a-kind” treatment facility. However, the review team’s report stated that the decision to retrofit the facility may result in logistical and physical maintenance challenges because of space limitations and height requirements. Based on the results of an independent analysis of alternatives for calcine waste disposition, published in April 2016, EM decided to suspend developing the hot isostatic pressing technology, according to EM officials from the Idaho Cleanup Project. DOE initiated this analysis of alternatives in response to a new requirement from the Secretary of Energy and because hot isostatic pressing is not a mature technology for HLW, according to EM’s summary report for the analysis. The report identified uncertainties and challenges with the use of hot isostatic pressing when compared to other potential treatment options given, including that hot isostatic pressing is significantly different than vitrification and would require the development and acceptance of testing protocols to validate that it produces a robust waste form, hot isostatic pressing had the second greatest estimated cost (more than $2 billion) of the options assessed in the analysis of alternatives, hot isostatic pressing represented the highest operational safety risk of all of the options assessed given its use of high pressures and temperatures, and other treatment options may perform better for managing the waste because of significant advances in technology since the selection of hot isostatic pressing in 2009. The independent team performing this analysis also concluded that uncertainties regarding plans for an HLW geologic repository also affect EM’s ability to move forward with selecting a treatment technology. According to EM officials from the Idaho Cleanup Project and documents, EM’s selection of hot isostatic pressing was based on assumptions developed based on sending the waste to the Yucca Mountain disposal facility. Specifically, an important factor in the selection of hot isostatic pressing as the treatment technology was its ability to provide the lowest volume of final waste, while producing a robust waste form, which would reduce disposal costs at Yucca Mountain. As previously noted, the licensing for developing the Yucca Mountain facility was terminated in 2010. The team performing the analysis of alternatives concluded that because selecting an appropriate treatment technology greatly depends on the calcine waste’s disposal path and associated waste form performance requirements, EM should defer making a final decision on the treatment technology until the performance objectives of the disposal path are better defined. While further decisions regarding a treatment technology for the calcine waste are suspended, EM officials from the Idaho Cleanup Project said that they are taking steps to demonstrate to regulators from Idaho’s Department of Environmental Quality that they are making progress to prepare the calcine waste for disposal outside the state. Under DOE’s 1995 settlement agreement with Idaho, treatment of all calcine waste is to be completed by a target date of December 31, 2035. Further, DOE is required to meet interim milestones for the cleanup of the waste under a site treatment plan that DOE developed for the Idaho Department of Environmental Quality. EM officials from the Idaho Cleanup Project told us that they planned to work with the Idaho Department of Environmental Quality to make changes to milestones specific to calcine waste in the site treatment plan, and Idaho Department of Environmental Quality officials stated in December 2018 that preliminary discussions on this topic occurred in September 2018. Further, EM Idaho Cleanup Project officials identified actions that EM is taking at the site to study alternatives to treatment and aspects of the disposal process. EM officials from the Idaho Cleanup Project stated that with the suspension of developing hot isostatic pressing, they are studying the potential packaging of the calcine waste for disposal without additional treatment, or “direct disposal.” The analysis of alternatives report identified direct disposal as having significant cost savings over other technologies. However, the team performing the analysis of alternatives also found that this method has a high degree of regulatory uncertainty and it is not clear whether it would be accepted by stakeholders, such as state regulators and the public. EPA officials told us that if EM wanted to proceed with plans for the direct disposal of the calcine waste in a geologic repository, EM would need, among other things, to seek a no-migration variance from EPA. A petition for a no-migration variance must demonstrate, to a reasonable degree of certainty, that the hazardous components would not leak or escape once the HLW is buried underground for as long as the waste remains hazardous. EPA officials added that there is a very high bar for such variances; only one such request has been approved since 1984, and it was later rescinded. In February 2019, an EM Idaho Cleanup Project official told us that EM has met with officials from the Idaho Department of Environmental Quality and EPA to receive their preliminary input on this approach. EM Idaho Cleanup Project officials said that they are focusing in the near term on developing and testing a system to retrieve the calcine waste from its storage vessels, called bin sets. According to EM documents, retrieval of the calcine waste from the bin sets is a precursor to treating or packaging the waste for disposal, and there are several challenges to address in developing an effective retrieval system. As a result, EM directed its contractor to conduct a project to retrieve calcine waste from the oldest bin set and move it to a partially empty bin set under EM’s contract for hazardous waste cleanup at INL. The project serves to both test different forms of technologies and also to cease use of the older bin set, which does not have the same structural integrity as the other bin set because of its design, according to EM officials from the Idaho Cleanup Project and documents. The project is estimated to cost $50 million over 5 years, according to these officials. Fluor Idaho’s plan for the calcine waste retrieval project involves developing a full-scale mock-up of the retrieval process for testing in fiscal years 2019 and 2020, with the commissioning and start-up of the full-scale system and transfer of the waste to occur in fiscal year 2021. In February 2019, an EM official told us that $6 million was obligated to the pilot project in fiscal year 2019 in part because of increased costs for the IWTU reengineering project and cleanup of transuranic waste at INL. Despite these efforts, EM officials from the Idaho Cleanup Project acknowledged that the agency has no plan to issue a new Record of Decision or amend the 2010 Record of Decision selecting the treatment option for calcine waste. Although EM identified challenges with using hot isostatic pressing for the treatment of the calcine waste in its technical readiness assessment in 2011 and analysis of alternatives in 2016, an EM official told us that the agency does not have a strategy for determining its next steps in treating this waste for land disposal. According to standards for internal control, federal agency management should identify, analyze, and respond to risks related to achieving a defined objective. Without developing a strategy, including a timeline, to identify and develop a treatment approach for the calcine waste, EM does not have reasonable assurance that it will meet milestones for the completion of treatment of all calcine by a target date of December 31, 2035. EM has been working since 2005 to construct and operate the IWTU to treat the SBW and calcine waste at INL. Despite declaring construction complete in 2012 at a cost of $571 million, EM is still working to repair and reengineer the IWTU following the discovery of facility problems during testing, with expenditures surpassing $416 million. EM has made progress in identifying the engineering problems plaguing the facility and implementing technical changes and expects to complete the second of the four phases of the reengineering project in mid-2019, with its next series of system testing to begin in early 2020. However, EM has experienced significant cost increases and schedule delays in phase two of the IWTU project, and additional engineering and testing remains to be completed before beginning a multiyear effort to treat the SBW. EM’s ability to achieve the project’s estimated cost and schedule in phase two may have been hampered because EM has not fully followed best practices for ensuring that the cost estimate is complete and the schedule estimate is well-constructed. By ensuring that the cost estimate for future phases of the IWTU reengineering project and the SBW treatment operations is comprehensive (e.g., account for all possible costs), EM will have greater assurance that it can successfully plan program resource requirements. Moreover, by developing a well-constructed schedule estimate for the IWTU reengineering project and the SBW treatment operations, EM will have greater assurance that it can successfully achieve its plans to reengineer the IWTU and begin treatment of the SBW without further delays. Further, while EM is using an EVM system to measure the performance of the project and generally followed best practices for EVM systems, the system produces unreliable data. By following best practices for ensuring EVM data reliability for the IWTU reengineering project’s EVM system, EM leadership will have better access to reliable performance data as it manages billions of dollars’ worth of cleanup work and provides information to Congress and other stakeholders on the cleanup work every year. EM faces long-standing challenges to implementing its preferred alternative for disposing of the treated SBW at WIPP. Key among these challenges are provisions in federal law and the WIPP permit that prevent EM from disposing of the SBW at WIPP. EM has taken some steps toward addressing these challenges, such as seeking public comment on its new interpretation of the statutory definition of HLW that according to EM could allow the waste to be disposed of at WIPP or an alternative to an HLW geologic repository. However, EM has no strategy or timeline for making any changes to DOE policies and regulations that may be required to implement its new interpretation or for making decisions regarding disposing of the SBW. Until it develops such a strategy, including a timeline, to implement the actions required to achieve its preferred disposal pathway, or an alternative, for the SBW, EM will not have reasonable assurance that it can achieve its preferred plan for disposal or begin the process of identifying an alternative. Further, if EM implements its new interpretation of HLW and uses this definition to classify the SBW as non-HLW, there is significant risk for extended litigation, which may delay EM’s plans to dispose of the SBW at its preferred disposal site. Moreover, EM faces challenges in completing treatment of the calcine waste by a target date of December 31, 2035, in light of its decision to suspend development of the selected treatment technology, hot isostatic pressing, and the absence of an HLW geologic repository. Even though EM is studying alternatives to using hot isostatic pressing to prepare the calcine waste for disposal, it has not developed a strategy or a timeline for determining its plans for treating this waste for disposal. Without developing such a strategy, including a timeline, for the treatment and disposal of the calcine waste to ensure that EM meets the milestone for completing the treatment of the waste by December 31, 2035, EM does not have reasonable assurance that it can meet its milestones. We are making five recommendations to DOE: The Secretary of Energy should direct the Assistant Secretary of EM to develop cost estimates for the IWTU reengineering project and the SBW treatment operations that meet best practices for being comprehensive (e.g., account for all costs). (Recommendation 1) The Secretary of Energy should direct the Assistant Secretary of EM to develop schedule estimates for the IWTU reengineering project and the SBW treatment operations that meet best practices for being well- constructed. (Recommendation 2) The Secretary of Energy should direct the Assistant Secretary of EM to follow best practices for ensuring the reliability for the IWTU reengineering project’s EVM system. (Recommendation 3) The Secretary of Energy should direct the Assistant Secretary of EM to develop a strategy, including a timeline, for implementing the actions required to achieve its preferred disposal pathway, or an alternative, for the SBW. (Recommendation 4) The Secretary of Energy should direct the Assistant Secretary of EM to develop a strategy, including a timeline, to identify and develop a treatment approach for the disposal of the calcine waste to ensure that EM meets the milestone for completing the treatment of this waste by the target date of December 31, 2035. (Recommendation 5) We provided a draft of this report for review and comment to the Secretary of Energy and the Administrator of the EPA. DOE provided written comments on the draft report, which are presented in appendix IV. EPA did not provide written comments. DOE and EPA both provided technical comments that we incorporated in the report as appropriate. DOE agreed with our recommendations related to the management of the IWTU reengineering project, including developing cost and schedule estimates that meet best practices and ensuring the reliability of the EVM system for the project. Regarding the cost estimate, DOE committed to developing cost estimates that meet best practices and stated that cost estimates for phases three and four of the IWTU reengineering project have been developed and reviewed by the Defense Contract Audit Agency. For the schedule estimate, DOE stated that the schedules for phases three and four have been developed and that the inclusion of these phases in the schedule is in accordance with best practices for the well-constructed characteristic. With regard to the EVM system, DOE stated that cost and performance data will be included in the EVM system in accordance with EVM best practices once contract negotiations are completed, which the agency estimated would conclude by December 31, 2019. DOE also agreed with our recommendations to develop a strategy, including a timeline, for the disposal of the SBW and calcine waste. DOE further stated that EM is in the process of developing a site options analysis for INL and other EM sites to identify opportunities to complete cleanup work through more efficient and innovative approaches over the next decade. This analysis is expected to be completed in fiscal year 2020, according to DOE. DOE stated that EM’s HLW interpretation issued in June 2019 could potentially open new disposal pathways for some reprocessing waste, such as SBW and calcine, while noting that decisions about whether and how this interpretation will apply to existing wastes have yet to be made. In its written comments, DOE disagreed with our recommendation to seek clarification from Congress on DOE’s authority to classify the SBW as other than HLW if such clarification is necessary to avoid extended litigation. DOE stated the agency does not require additional clarification from Congress to classify reprocessing waste as other than HLW. We are deleting our recommendation but continue to believe that there is significant risk for extended litigation if EM implements its new interpretation of HLW and uses this definition to classify the SBW as non- HLW. Extended litigation may delay EM’s plans to dispose of the SBW at its preferred disposal site. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, the Administrator of the Environmental Protection Agency, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in appendix V. Our report examines (1) the extent to which the Department of Energy’s (DOE) Office of Environmental Management’s (EM) management of the Integrated Waste Treatment Unit (IWTU) reengineering project follows selected project management best practices; (2) challenges EM faces in the disposal of the sodium-bearing waste (SBW); and (3) challenges EM faces in the treatment and disposal of the calcine waste. To address these three objectives, we conducted a site visit to DOE’s Idaho National Laboratory (INL) in December 2017. During the site visit, we obtained documentation and interviewed officials from EM, which is responsible for hazardous waste cleanup at INL through its Idaho Cleanup Project. We also interviewed representatives from Fluor Idaho, LLC, which is the private contractor that manages hazardous waste cleanup at INL for EM, including the cleanup of the SBW and calcine waste. In addition, we conducted a site visit to Hazen Research, Inc., a subcontractor to Fluor Idaho, to observe pilot testing facilities for the IWTU reengineering project and discuss the status of the project with an EM official from the Idaho Cleanup Project and representatives from Hazen Research, Inc., and Fluor Idaho. To assess the extent to which EM’s management of the IWTU reengineering project meets selected project management best practices, we first identified areas deemed to be important to project management based on our previous work on DOE projects and leading practices from the Project Management Institute, which are generally recognized as leading practices for project management. Specifically, we reviewed the project management leading practices identified in the Project Management Institute’s A Guide to the Project Management Body of Knowledge—Sixth Edition. From this review, we selected project management practices related to developing cost and schedule estimates and conducting project monitoring through the use of earned value management (EVM) and independent reviews. We then conducted assessments of these best practices, as discussed below. Cost. To determine the extent to which the cost estimate for the IWTU reengineering project is reliable, we conducted an abridged analysis of the IWTU reengineering project’s cost estimate, focusing on its comprehensiveness. Typically, in analyzing a cost estimate against best practices in GAO’s Cost Estimating and Assessment Guide (cost guide), we examine four characteristics, each defined by multiple criteria: credible. For this review, we assessed the cost estimate for the IWTU reengineering project against the comprehensive characteristic, in part because EM officials told us that they had yet to develop a cost estimate for the program beyond phases one and two at the time of our review. Specifically, we reviewed the cost estimate for the operation of the IWTU and the IWTU reengineering project, which, at the time of our review, was only developed for phases one and two of the project. If a cost estimate is not comprehensive (that is, complete), then it cannot fully meet the well- documented, accurate, or credible best practice characteristics. For instance, if the cost estimate is missing some cost elements, then the documentation will be incomplete, the estimate will be inaccurate, and the result will not be credible because of the potential underestimating of costs and the absence of a full risk and uncertainty analysis. See appendix II for a summary assessment of the IWTU reengineering project’s cost estimate compared to selected best practices. Schedule. To assess EM’s schedule for the IWTU reengineering project, we conducted an abridged analysis of the IWTU reengineering project’s schedule, focusing on comprehensiveness and the degree to which it is well-constructed. Typically, in analyzing a schedule estimate against best practices in GAO’s Schedule Assessment Guide (schedule guide), we examine four characteristics, each defined by multiple criteria: controlled. For this review, we assessed the IWTU reengineering project schedule that EM provided in March 2018 against the well-constructed characteristic, in part because EM officials told us that they had yet to develop a schedule estimate for the totality of the reengineering project because of Fluor Idaho’s phased approach. If a schedule estimate is not well-constructed, it will not be able to properly calculate dates and predict changes in the future. When activities are missing logic links, the schedule will not be able to automatically transmit these delays to future activities that depend on them. When this happens, the schedule will not allow a sufficient understanding of the program as a whole, and users of the schedule will not have confidence in the dates and the critical path. In addition, we evaluated the comprehensive characteristic because it contributed to our analysis of EM’s EVM system, as described below. See appendix II for a summary assessment of the IWTU reengineering project’s schedule estimate compared to selected best practices. EVM. In addition, we analyzed EM’s use of EVM as a way to assess its monitoring of the IWTU reengineering project’s cost and schedule. EVM measures the value of work accomplished in a given period and compares it with the planned value of work scheduled for the period and with the actual cost of the work accomplished. It is an industry standard and is considered a best practice for conducting cost and schedule performance analysis for projects. Our EVM analysis focused on Fluor Idaho’s EVM data for the IWTU reengineering project contained in cost performance reports from March 2017 to February 2018 and the project schedule that EM provided in March 2018. Specifically, we compared this project documentation with EVM best practices as identified in our cost guide. Our research has identified a number of best practices that are the basis of effective EVM and should result in reliable and valid data that can be used for making informed decisions. These best practices have been collapsed into three high-level characteristics of a reliable EVM system, which are establish a comprehensive EVM system, ensure that the data resulting from the EVM system are reliable, and ensure that the program management team is using EVM data for decision-making purposes. See appendix III for our summary assessment of the IWTU reengineering project’s EVM data compared to best practices. EVM data are considered reliable if the overall assessment ratings for each of the three characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the EVM data cannot be considered reliable. Independent reviews. To assess the extent to which DOE has conducted independent reviews of the IWTU reengineering project, we examined DOE and EM policies to identify requirements for conducting reviews of operations activities. Specifically, we reviewed a 2016 DOE memorandum that established that DOE’s Project Management Risk Committee (PMRC) would provide independent review of selected projects in the operational release phase, the PMRC’s standard operating procedures, and EM’s guidance for projects in the operational release milestone. We examined documentation from the PMRC’s reviews of the IWTU reengineering project, including documentation that EM officials from the Idaho Cleanup Project prepared for these reviews and recommendations that the PMRC made to EM for the project. In addition, we spoke with officials from DOE’s Office of Project Management, which serves as the secretariat of the PMRC; EM’s Office of Acquisition & Project Management; and EM’s Idaho Cleanup Project about independent reviews of projects in the operational release phase. To examine challenges EM faces in the disposal of the SBW, we reviewed federal laws, regulations, and DOE policies on radioactive waste management, including those described in DOE Order 435.1 on radioactive waste management and its implementation manual. In addition, we examined EM’s October 2018 and June 2019 Federal Register notices, which provide DOE’s new interpretation of the statutory definition of high-level radioactive waste (HLW). We also reviewed documentation related to EM’s plans for disposing of the SBW at DOE’s Waste Isolation Pilot Plant (WIPP) in New Mexico, such as Record of Decision documents for proposed actions that require development of environmental impact statements, and the hazardous waste facility permit for WIPP that the New Mexico Environment Department issued. We interviewed DOE officials from the Office of the General Counsel; officials from EM’s Idaho Cleanup Project and Carlsbad Field Office, which is responsible for DOE’s oversight of WIPP; and officials from EM’s Office of Regulatory Compliance, Office of Nuclear Materials, and Office of Waste and Materials Management. We also interviewed officials from Idaho’s Department of Environmental Quality and New Mexico’s Environment Department, as well as representatives from two environmental advocacy groups in Idaho and New Mexico, to obtain their perspectives on the challenges facing EM’s SBW disposal efforts. To examine challenges EM faces in the treatment and disposal of the calcine waste, we reviewed federal laws, regulations, and documents that DOE and EM’s contractors for the Idaho Cleanup Project prepared related to the calcine waste cleanup mission. For example, we reviewed documents assessing treatment and disposal alternatives for calcine waste, including a 2016 analysis of alternatives report that EM prepared and a 2015 contractor-prepared report assessing the feasibility of the direct disposal of calcine waste. We interviewed officials from EM’s Idaho Cleanup Project and Office of Nuclear Materials; EM’s Chief Engineer; and representatives from EM’s contractor, Fluor Idaho, about plans for treating and disposing of the calcine waste and the retrieval pilot project. In addition, we reviewed Environmental Protection Agency (EPA) Resource Conservation and Recovery Act, as amended (RCRA) regulations, guidance, and documents concerning land disposal requirements. We also interviewed officials from EPA’s Office of Land and Emergency Management and Region 10 about EPA’s responsibilities for implementing RCRA. Lastly, we interviewed officials from the Idaho Department of Environmental Quality about how EM’s calcine waste treatment and disposal efforts address milestones in the Idaho Settlement Agreement. We conducted this performance audit from September 2017 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 2 describes the initial cost and schedule estimates for the four phases of the Integrated Waste Treatment Unit reengineering project compared to actual expenditures and schedule as of February 2019. Table 3 details our assessment of the Office of Environmental Management’s (EM) cost estimate for phases one and two of the Integrated Waste Treatment Unit (IWTU) reengineering project compared to selected best practices for cost estimating published in GAO’s Cost Estimating and Assessment Guide (cost guide). For this review, we assessed the cost estimate for the IWTU reengineering project against the comprehensive characteristic, in part because EM officials told us that they had yet to develop a cost estimate for the program beyond phases one and two, at the time of our review of these documents. We assessed the comprehensive characteristic for the IWTU reengineering cost estimate because if a cost estimate is not comprehensive—that is, complete—then it cannot fully meet the other best practice characteristics. According to our analysis, EM’s cost estimate for the IWTU reengineering project partially met best practices for a comprehensive cost estimate. Table 4 details our assessment of EM’s schedule for the IWTU reengineering project compared to selected best practices for project schedules published in GAO’s Schedule Assessment Guide (schedule guide). For this review, we assessed the schedule against the well- constructed characteristic, in part because EM officials told us that they had yet to develop a schedule for the totality of the reengineering project because of the contractor’s phased approach. We assessed the well- constructed characteristic because, among other reasons, if a schedule is not well-constructed, it will not be able to properly calculate dates and predict changes in the future. In addition, we evaluated the comprehensive characteristic as it is needed to evaluate an earned value management system. According to our assessment, EM’s schedule for the reengineering project partially met best practices related to the well- constructed characteristic and substantially met best practices related to the comprehensive characteristic. Table 5 details our assessment of March 2017 to February 2018 data from the Department of Energy’s (DOE) Office of Environmental Management’s (EM) earned value management (EVM) system for the Integrated Waste Treatment Unit (IWTU) reengineering project. EVM measures the value of work accomplished in a given period and compares it with the planned value of work scheduled for that period and with the actual cost of work accomplished. By using the metrics derived from these values to understand performance status and to estimate cost and time to complete, EVM can alert program managers to potential problems sooner than expenditures alone can. Our research has identified a number of best practices that are the basis of effective EVM and should result in reliable and valid EVM data that can be used for making informed decisions. Specifically, EM followed (i.e., substantially met) best practices to ensure that its EVM system is (1) comprehensive and (2) used by leadership for decision-making, but did not follow (i.e., partially met) best practices to ensure that the data resulting from the EVM system are reliable. In addition to the contact named above, Casey L. Brown (Assistant Director), Emily Ryan (Analyst in Charge), Juaná Collymore, Jennifer Echard, Richard P. Johnson, Jason Lee, Eli Lewine, Katrina Pekar- Carpenter, Karen Richey, Jeanette Soares, Sheryl Stein, Farrah M. Stone, Paul Sturm, and Sara Sullivan made key contributions to this report.", "summary": "Decades of defense activities at DOE's Idaho National Laboratory produced two forms of waste that EM has managed as HLW: liquid SBW and granular calcine waste. Under an agreement with the state, DOE must treat the waste to prepare it for removal from Idaho by 2035. Construction on the IWTU, EM's facility to treat such waste, was completed in 2012, but initial testing of the SBW treatment process revealed design problems. EM has since been working to reengineer the IWTU. Total project construction and reengineering expenditures have reached nearly $1 billion as of February 2019. GAO was asked to review EM's efforts to treat and dispose of the SBW and calcine waste. This report examines (1) the extent to which EM's management of the IWTU follows selected project management best practices; (2) challenges EM faces in disposing of the SBW; and (3) challenges EM faces in treating and disposing of the calcine waste. GAO reviewed agency documents and IWTU project data from March 2017 through February 2018, analyzed EM project management efforts against selected project management best practices for cost and schedule, and interviewed DOE officials. The Department of Energy's (DOE) Office of Environmental Management (EM) has not fully followed selected project management best practices in managing the reengineering of the Integrated Waste Treatment Unit (IWTU), shown in the figure, to treat 900,000 gallons of liquid sodium-bearing waste (SBW) that must be solidified for disposal. EM's cost and schedule estimates for IWTU reengineering did not fully meet selected best practices for cost (i.e., did not account for all costs) and schedule estimates (e.g., did not have a valid critical path). For example, EM did not follow best practices for a comprehensive cost estimate because EM did not include both government and contractor costs over the entire project. As of February 2019, EM has experienced approximately $64 million in added costs and a more than 1-year delay in IWTU reengineering. Without fully following best practices for cost and schedule estimates, EM is at risk of future cost overruns and delays in meeting its target disposal milestones. Based on GAO's review of EM documents, EM faces challenges with its plans for SBW disposal at its preferred disposal site, the Waste Isolation Pilot Plant (WIPP), an underground repository for waste contaminated by nuclear elements, near Carlsbad, New Mexico. These challenges include a statutory prohibition on the disposal of high-level waste (HLW) at WIPP. Further, EM does not have a strategy or timeline to address these challenges or to identify an alternative disposal pathway. Without such a strategy or timeline, EM risks not meeting its commitments with Idaho to prepare the SBW for removal from the state by 2035. EM faces challenges implementing its selected technology to further treat 1.2 million gallons of granular calcine waste and selecting a potential waste disposal pathway. For example, DOE has identified challenges with retrofitting the IWTU for calcine waste treatment. As a result, EM is deferring further development of its plans to treat the calcine waste. EM officials said that the agency is making progress toward calcine waste disposal by testing options for removing the waste from its storage bins, a precursor to treating or packaging the waste for disposal. However, EM does not have a strategy or timeline for determining its next steps for the treatment and disposal of calcine waste. Such a strategy could help EM in seeking alternatives to its selected treatment technology and provide assurance that it will meet its commitments with Idaho for removing calcine waste from the state by the end of 2035. GAO is making five recommendations, including that DOE develop a strategy for the disposal of the waste. DOE generally agreed with all of these recommendations.", "document_type": "gao"}
{"report": "Consistent with the discretion afforded by the APA, Regulations.gov and agency-specific comment websites use required and optional fields on comment forms to collect some identity information from commenters. In addition to the text of the comment, agencies may choose to collect identity information by requiring commenters to fill in other fields, such as name, address, and email address before they are able to submit a comment. Regardless of the fields required by the comment form, the selected agencies all accept anonymous comments in practice. Specifically, in the comment forms on Regulations.gov and agency- specific comment websites, a commenter can submit under a fictitious name, such as “Anonymous Anonymous,” enter a single letter in each required field, or provide a fabricated address. In each of these scenarios, as long as a character or characters are entered into the required fields, the comment will be accepted. Further, because the APA does not require agencies to authenticate submitted identity information, neither Regulations.gov nor the agency-specific comment websites contain mechanisms to check the validity of identity information that commenters submit through comment forms. Regulations.gov and agency-specific comment websites also collect some information about public users’ interaction with their websites through application event logs and proxy server logs, though the APA does not require agencies to collect or verify it as part of the rulemaking process. This information, which can include a public user’s Internet Protocol (IP) address, browser type and operating system, and the time and date of webpage visits, is collected separately from the comment submission process as part of routine information technology management for system security and performance, and cannot be reliably connected to specific comments. Seven of the 10 selected agencies have documented some internal guidance associated with the identity of commenters during the three phases of the public comment process: intake, analysis, and response to comments. However, the focus and substance of this guidance varies by agency and phase of the comment process. As shown in Table 1, for selected agencies that have guidance associated with the identity of commenters, it most frequently relates to the comment intake or response to comment phases of the public comment process. The guidance for these phases addresses activities such as managing duplicate comments (those with identical or near-identical comment text but varied identity information) or referring to commenters in a final rule. Agencies are not required by the APA to develop internal guidance associated with the public comment process generally, or identity information specifically. Within the discretion afforded by the APA, the 10 selected agencies’ treatment of identity information varies during the three phases of the public comment process. Selected agencies differ in how they treat identity information during the comment intake phase, particularly in terms of how they post duplicate comments, which can lead to identity information being inconsistently presented to public users of comment systems. Generally, officials told us that their agencies either (1) maintain all comments within the comment system, or (2) maintain some duplicate comment records outside of the comment system, for instance, in email file archives. When an agency chooses to post a sample of duplicate comments, the identity information and unique comment contents for all duplicate comments may not be present on the public website. For example, for all duplicate comments received, Securities and Exchange Commission (SEC) posts a single example for each set of duplicate comments and indicates the total number of comments received. As a result, the identity information and any unique comment content beyond the first example are not present on the public website. (See fig. 1.) Selected agencies’ treatment of identity information during the comment analysis phase also varies. Specifically, program offices with the responsibility for analyzing comments place varied importance on identity information during the analysis phase. Finally, all agencies draft a response to comments with their final rule, but the extent to which the agencies identify commenters or commenter types in their response also varies across the selected agencies. Our analysis of Regulations.gov and agency-specific comment websites shows that the varied comment posting practices of the 10 selected agencies are not always documented or clearly communicated to public users of the websites. The E-Government Act of 2002 requires that all public comments and other materials associated with a given rulemaking should be made “publicly available online to the extent practicable.” In addition to the requirements of the E-Government Act, key practices for transparently reporting open government data state that federal government websites—like those used to facilitate the public comment process—should fully describe the data that are made available to the public, including by disclosing data sources and limitations. We found that the selected agencies we reviewed do not effectively communicate the limitations and inconsistencies in how they post identity information associated with public comments. As a result, public users of the comment websites lack information related to data availability and limitations that could affect their ability to use and make informed decisions about the comment data and effectively participate in the rulemaking process themselves. Public users of Regulations.gov seeking to submit a comment are provided with a blanket disclosure statement related to how their identity information may be disclosed, and are generally directed to individual agency websites for additional detail about submitting comments. While additional information is provided in the Privacy Notice, User Notice, and Privacy Impact Assessment for Regulations.gov, public users are not provided any further detail on Regulations.gov regarding what information, including identity information, they should expect to find in the comment data. Additionally, there is not enough information to help public users determine whether all of the individual comments and associated identity information are posted. Available resources on Regulations.gov direct public users to participating agencies’ websites for additional information about agency-specific review and posting policies. Seven of the eight participating agencies’ websites direct public users back to Regulations.gov and the Federal Register, either on webpages that are about the public comment process in general, or on pages containing information about specific NPRMs. Three of these participating agencies – the Environmental Protection Agency (EPA), the Fish and Wildlife Service (FWS), and the Food and Drug Administration (FDA) – do provide public users with information beyond directing them back to Regulations.gov or the Federal Register, but only FDA provides users with details about posting practices that are not also made available on Regulations.gov. The eighth participating agency – the Employee Benefits Security Administration (EBSA) – does not direct public users back to Regulations.gov, and instead recreates all rulemaking materials for each NPRM on its own website, including individual links to each submitted comment. However, these links go directly to comment files, and do not link to Regulations.gov. While EBSA follows departmental guidance associated with posting duplicate comments, which allows some discretion in posting practices, the agency does not have a policy for how comments are posted to Regulations.gov or its own website. Further, in the examples we reviewed, the content of the NPRM-specific pages on EBSA’s website does not always match what is posted to Regulations.gov. Because participating agencies are not required to adhere to standardized posting practices, Regulations.gov directs public users to participating agency websites for additional information about posting practices and potential data limitations. However, these websites do not describe the limitations associated with the identity information contained in publicly posted comments. As allowed for under the APA, all of the participating agencies in our review vary in the way in which they post identity information associated with comments—particularly duplicate comments. However, the lack of accompanying disclosures may potentially lead users to assume, for example, that only one entity has weighed in on an issue when, actually, that comment represents 500 comments. Without better information about the posting process, the inconsistency in the way in which duplicate comments are presented to public users of Regulations.gov limits public users’ ability to explore and use the data and could lead users to draw inaccurate conclusions about the public comments that were submitted and how agencies considered them during the rulemaking process. Both nonparticipating agencies use comment systems other than Regulations.gov and follow standardized posting processes associated with public comments submitted to their respective comment systems, but SEC has not clearly communicated these practices to the public. Although it appears to users of the SEC website that the agency follows a consistent process for posting duplicate comments, at the time of our June 2019 report, this practice had not been documented or communicated to public users of its website. In contrast, FCC identifies its policies for posting comments and their associated identity information in a number of places on the FCC.gov website, and on its Electronic Comment Filing System (ECFS) web page within the general website. Regarding comments submitted to rulemaking proceedings through ECFS, public users are informed that all information submitted with comments, including identity information, will be made public. Our review of ECFS comment data did not identify discrepancies with this practice. Although the public comment process allows interested parties to state their views about prospective rules, the lack of communication with the public about the way in which agencies treat identity information during the posting process, particularly for duplicate comments, may inhibit users’ meaningful participation in the rulemaking process. While the APA does not include requirements for commenters to provide identity information, or for agency officials to include commenters identity as part of their consideration of comments, key practices for transparently reporting open government data state that federal government websites— like those used to facilitate the public comment process—should fully describe the data that are made available to the public, including by disclosing data sources and limitations. As shown in Table 2, we recommended in our June 2019 report that five of the selected agencies establish a policy for posting comments, and that eight selected agencies take action to more clearly communicate their policies for posting comments, particularly with regard to identity information and duplicate comments. These agencies generally agreed with our recommendations and identified actions they planned to take in response, such as developing policies for posting duplicate comments and communicating those in various ways to public users. Since issuing our June 2019 report, all of the agencies to which we made recommendations have provided us with additional updates. Specifically, SEC completed actions that are responsive to the recommendation we made to it. In this regard, in September 2019, SEC issued a memorandum that reflects SEC’s internal policies for posting duplicate comments and associated identity information. SEC has also communicated these policies to public users on the SEC.gov website by adding a disclaimer on the main comment posting page that describes how the agency posts comments. These measures will help public users better determine whether and how they can use the data associated with public comments. The other seven agencies have provided updates, but have not yet implemented the recommendations. In December 2019 and January 2020, the Bureau of Land Management (BLM), Consumer Financial Protection Bureau (CFPB), EPA, and FWS notified us that they are in the process of developing or updating policies for posting public comments as well as statements for their websites to communicate these policies to the public. Similarly, in January 2020, the Department of Health and Human Services (HHS) stated that the Centers for Medicare and Medicaid Services (CMS) would update its comment posting policy and communicate it on the CMS website. However, the excerpt of the policy language provided does not include information about how the agency posts duplicate comments. Further, CMS did not provide us with the finalized policy, and our review of the website does not indicate any changes have been made. HHS officials stated they would provide additional follow up actions by July 2020. In September 2019, EBSA also stated that it will develop a written policy regarding posting of comments, including duplicate comments, which will be available on its website. However, the agency did not provide evidence that a formal evaluation of its current practice of replicating rulemaking dockets had been conducted, and did not identify plans to do so. The Wage and Hour Division (WHD) indicated that it will add text to each webpage for any rulemaking that invites public comments that states any personal information included in the comments (including duplicate) will be posted to Regulations.gov without change. However, the preliminary text provided by officials in August 2019 does not explain WHD’s policy of posting duplicate comments as a group under a single document ID, and therefore does not clearly communicate the agency’s posting practices to the public. Chairman Green, Ranking Member Barr, and Members of the Subcommittee, this concludes my prepared remarks. I would be happy to answer any questions you may have at this time. For further information regarding this testimony, please contact Seto J. Bagdoyan, (202) 512-6722 or bagdoyans@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony are David Bruno (Assistant Director), Allison Gunn (Analyst in Charge), Elizabeth Kowalewski, and Roger Gildersleeve. Individuals who contributed to the report on which this testimony is based include Enyinnaya David Aja, Gretel Clarke, Lauren Kirkpatrick, James Murphy, Alexandria Palmer, Carl Ramirez, Shana Wallace, and April Yeaney. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Federal agencies publish on average 3,700 proposed rules yearly and are generally required to provide interested persons (commenters) an opportunity to comment on these rules. In recent years, some high-profile rulemakings have received extremely large numbers of comments, raising questions about how agencies manage the identity information associated with comments. While the APA does not require the disclosure of identifying information from a commenter, agencies may choose to collect this information. This testimony summarizes GAO's June 2019 report on public comment posting practices (GAO-19-483). In that report, GAO examined (1) the identity information collected by comment websites; (2) the guidance agencies have related to the identity of commenters; (3) how 10 selected agencies treat identity information; and (4) the extent to which the selected agencies clearly communicate their practices associated with identity information. The 10 agencies were selected on the basis of the volume of public comments they received on rulemakings. For this testimony, GAO obtained updates on the status of recommendations made to the selected agencies. The Administrative Procedure Act (APA) governs the process by which many federal agencies develop and issue regulations, which includes the public comment process (see figure below). In June 2019, GAO found that Regulations.gov and agency-specific comment websites collect some identity information—such as name, email, or address—from commenters who choose to provide it during the public comment process. The APA does not require commenters to disclose identity information when submitting comments. In addition, agencies have no obligation under the APA to verify the identity of such parties during the rulemaking process, and all selected agencies accept anonymous comments in practice. GAO found in the June 2019 report that seven of 10 selected agencies have some internal guidance associated with the identity of commenters, but the substance of this guidance varies. This reflects the differences in the way that the selected agencies handle commenter identity information internally. GAO also found that the selected agencies' practices for posting public comments to comment websites vary considerably, particularly for duplicate comments (identical or near-identical comment text but varied identity information). For example, one agency posts a single example of duplicate comments and indicates the total number of comments received, but only the example is available to public users of Regulations.gov. In contrast, other agencies post all comments individually. As a result, identity information submitted with comments is inconsistently presented on public websites. The APA allows agencies discretion in how they post comments, but GAO found that some of the selected agencies do not clearly communicate their practices for how comments and identity information are posted. GAO's key practices for transparently reporting government data state that federal government websites should disclose data sources and limitations to help public users make informed decisions about how to use the data. If not, public users of the comment websites could reach inaccurate conclusions about who submitted a particular comment, or how many individuals commented on an issue. In June 2019, GAO made recommendations to eight of the selected agencies regarding implementing and communicating public comment posting policies. The agencies generally agreed with the recommendations and identified actions they planned to take in response. Since the June 2019 report, one agency has implemented GAO's recommendation and seven agencies have identified additional planned actions.", "document_type": "gao"}
{"report": "To be eligible for the Medicare hospice benefit, an individual must be eligible for Medicare Part A (which covers inpatient care) and be medically certified as having a terminal illness with a life expectancy of 6 months or less if the illness runs it normal course. For individuals to receive care from a Medicare-approved hospice program, they must elect the hospice benefit by signing a statement indicating they are waiving their rights to Medicare payment for services related to curative treatment of their terminal illness. When enrolling in Medicare hospice care, beneficiaries can receive several different types of services in various settings. Most hospice beneficiaries receive hospice care in their own home, but they can also receive care in other settings, such as a nursing home, assisted living facility, hospice facility, or hospital. The Medicare hospice benefit covers a variety of services and supplies for the palliation and management of the terminal illness, including physician and nursing services, medical equipment and supplies including drugs for pain and symptom management, hospice aide and homemaker services, physical and occupational therapy, and spiritual and grief and loss counseling. A hospice interdisciplinary team (in collaboration with the beneficiary’s primary care provider, if any) works with the beneficiary, family, and caregiver(s) to develop a plan of care that addresses the physical, psychosocial, spiritual, and emotional needs of the beneficiary, family members, and caregiver(s). The hospice provider must make all services under the Medicare hospice benefit available to beneficiaries as needed, 24 hours a day, 7 days a week. Although hospice care is designed for beneficiaries with a life expectancy of 6 months or less, beneficiaries can receive hospice care beyond 6 months if they continue to meet hospice eligibility requirements. In addition, beneficiaries can disenroll from the hospice benefit at any time and re-enroll in hospice care at a later time. CMS pays hospices based on the level of hospice care provided to beneficiaries on a given day. There are four levels of hospice care, which are paid at either a daily rate or an hourly rate depending on the location and intensity of services provided. (See table 1.) Each care level has a payment rate that is adjusted for geographic differences in wages, and CMS updates these payment rates annually. The most common level of care is called routine home care (accounting for 98 percent of all Medicare hospice care in 2017), and hospices receive the routine home care payment daily rate regardless of whether beneficiaries receive any services on a given day. In addition, CMS imposes two payment limitations (referred to as caps) on Medicare payment for hospice services—one that limits a hospice’s number of inpatient days and one that limits a hospice’s total Medicare payments in a given year. In response to requirements in the Patient Protection and Affordable Care Act, CMS established the Hospice Quality Reporting Program, which currently includes two sets of data to assess the quality of hospice providers’ care; CMS publishes these data on its Hospice Compare website. Medicare hospice providers are required to submit these data to CMS for all patients regardless of payer source (e.g., Medicare, Medicaid, or private insurance). The two data sets are the following: Provider-reported quality measure data. This set of data (which CMS refers to as the Hospice Item Set) is used to calculate a hospice provider’s performance on quality measures, which include seven measures that reflect the percentage of all hospice patients’ stays where the provider completed various key care processes, such as screening patients for pain and shortness of breath. CMS also recently implemented an eighth measure, called the composite measure, which calculates the percentage of patients’ hospice stays in which the hospice provider completed all seven care process quality measures. Caregivers’ experience survey data. This set of data (referred to as the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Hospice Survey) is a national survey that captures, from the caregiver’s (family member or friend) perspective, the patient’s experience with hospice care. The survey includes questions that are used to calculate eight quality measures based on survey responses. For example, one measure scores how well the hospice communicated with the patient’s family. CMS oversees the quality of Medicare hospice care primarily through inspections—referred to as surveys—which are conducted by state survey agencies contracted by CMS or CMS-approved national private accrediting organizations. These surveys are used to determine whether the hospice is in compliance with federal health and safety requirements detailed in Medicare’s hospice conditions of participation. A hospice must be in compliance with these conditions to participate in the Medicare program. Medicare’s hospice conditions of participation include requirements related to patient care and organizational environment (e.g., the hospice must organize, manage, and administer its resources to provide necessary care). Each condition of participation is composed of standards associated with the condition, and a standard may have associated sub-components. For example, the “patient’s rights” condition includes standards such as “notice of rights and responsibilities” and “rights of the patient.” The “rights of the patient” standard includes sub- components, such as the patient has the right to receive effective pain management and symptom control. There are three main types of survey inspections—an initial certification survey when a provider first seeks to participate in Medicare; a re- certification survey to ensure ongoing compliance; and surveys to investigate complaints or incidents related to federal requirements. If a hospice is found to be out of compliance with hospice health and safety requirements during a survey, CMS cites the provider for non- compliance—referred to as a deficiency. These deficiencies are categorized at one of two levels: Condition-level deficiencies. These deficiencies are the most serious. A condition-level deficiency is one in which the provider violates one or more standards and the deficiencies are of such character as to substantially limit the provider’s capacity to furnish adequate care or which adversely affect the health and safety of patients. When a hospice provider is cited for a condition-level deficiency, CMS places the provider on a 90-day termination track (or 23 days if the situation is determined to pose “immediate jeopardy” to beneficiaries) within which the provider must correct the issue(s) and the correction must be confirmed via a follow-up survey visit. If this does not happen within 90 days of the survey date, CMS terminates the hospice’s Medicare provider agreement; termination is an enforcement remedy CMS uses to ensure compliance. Standard-level deficiencies. These deficiencies are less serious. A hospice provider that has a standard-level deficiency can be certified or re-certified only if the provider has submitted an acceptable plan of correction for achieving compliance within a reasonable period of time. According to CMS officials, standard-level deficiencies must also have follow-up to ensure correction, although the type of follow-up depends on the nature of the deficiency. If a standard-level deficiency is very minor and does not place any beneficiaries at risk, the follow-up may be handled through email or telephone instead of a follow-up visit. According to CMS officials, if a provider fails to submit or implement an acceptable plan of correction within a reasonable period of time acceptable to CMS, the provider is placed on the 90-day termination track noted above. For-profit and non-profit hospices served roughly the same percentage of the approximately 1.5 million Medicare hospice beneficiaries in 2017, even though for-profit hospices make up about two-thirds of all hospice providers. According to our analysis of CMS data, for-profit providers treated about 50 percent of those beneficiaries and non-profit providers treated about 48 percent in 2017. This distribution has been about the same in each year from 2014 through 2017. For example, for these years, the percentages of beneficiaries treated by for-profit providers ranged from 48.7 percent to 50.2 percent (see additional details in app. I, table 7). When comparing the beneficiary populations treated by for-profit and non- profit hospice providers, we found that they generally had similar demographic characteristics. We identified two primary exceptions to this general finding: (1) non-profit hospices had slightly higher percentages of white beneficiaries, and (2) for-profit hospices had a greater proportion of patients enrolled in both Medicare and Medicaid. See table 2 (for more detailed data, see app. I, table 8). While beneficiary demographic characteristics were generally similar, we found differences in beneficiary diagnoses between for-profit and non- profit hospices. Specifically, for-profit hospices had, on average, a greater percentage of patients with non-cancer diagnoses—77 percent of for-profit hospice beneficiaries compared to 69 percent of non-profit hospice beneficiaries in 2017. Our analysis found that for-profit providers received a higher proportion of Medicare hospice payments than did non-profit providers. For 2017, about $10.4 billion (58 percent) of the $17.9 billion dollars in Medicare payments were made to for-profit providers and $7.2 billion (40 percent) of payments were to non-profit providers. Our analysis found this same pattern in each year from 2014 through 2017. One reason for-profit hospices received a higher portion of Medicare hospice payments for the period we reviewed is because (as previously noted) they had, on average, a greater percentage of beneficiaries with non-cancer diagnoses, and we found non-cancer beneficiaries, on average, had longer lengths of stay. (See table 3.) Since hospices are typically paid a set amount per day of a hospice stay, longer stays generally result in higher payments. Beneficiaries with non-cancer diagnoses can often have longer lengths of stay compared to other beneficiaries because the progression of these diseases (such as dementia) can be harder to predict; this may result in beneficiaries being enrolled in hospice earlier than appropriate (meaning that their projected life expectancy may actually be longer than 6 months). For instance, one study noted that dementia beneficiaries’ decline may include periods of stabilization where their health stays the same or even improves, which differs from a constant and predictable decline in most beneficiaries with terminal cancer. There are likely other factors beyond a greater percentage of beneficiaries with non-cancer diagnoses that contributed to for-profit providers’ higher portion of Medicare hospice payments. We found that for-profit providers had, on average, longer lengths of stay for both cancer and non-cancer beneficiaries compared to non-profit providers. (See table 3.) For example, non-cancer beneficiaries at for-profit providers had an average length of stay of 108 days, while non-cancer beneficiaries at non- profit providers had an average length of stay of 67 days. This suggests other factors besides beneficiary diagnosis contributed to longer average length of stay for for-profit providers. (For more detailed beneficiary diagnosis data from 2014 to 2017, see app. I, table 9.) For-profit and non-profit hospice providers had similar scores on CMS’s current quality measures (provider-reported measures and caregivers’ experience measures assessed through a survey of the beneficiaries’ caregiver). CMS uses these measures to assess the quality of care provided by hospices. In addition to CMS’s current quality measures, researchers we interviewed noted that there are other care indicators that can also be used to assess the quality of care provided by hospices. According to CMS documents, CMS is working to account for other care indicators by developing additional quality measures. We assessed hospice providers’ performance on these indicators and found that performance varied between for-profit and non-profit hospices. Our review of CMS data found that for 2017, both for-profit and non-profit hospices, on average, had similar scores on the seven quality measures that are provider-reported and that CMS currently uses to assess the quality of hospice care. (See table 4.) For six of the seven measures, for-profit and non-profit hospices had average scores of 94.7 percent or better. We also found that for-profits and non-profits had similar scores (83.6 percent and 87.0 percent, respectively) on a new composite measure that CMS implemented in 2017. This composite measure was designed to provide a more comprehensive evaluation of the hospice’s care by determining whether the hospice provider completed all of the applicable parts of hospice care that are measured by the seven quality measures. When looking at the subset of providers with the lowest scores on the composite quality measure, we found that for-profit hospices were more often in this subset, even when accounting for differences in the number of for-profit and non-profit providers: For the composite measure, there were 329 providers (261 for-profits and 68 non-profits) in the 10th percentile of scores or lower, meaning that the providers had a composite measure score of 64.3 percent or lower. Among these providers, we found that for-profits were more likely to be within this grouping, with about 12 percent of all for-profit providers having scores in the 10th percentile or lower compared to 6 percent of all non-profit providers. We also assessed the subset of these 329 providers that had composite measure scores below 50 percent, meaning that they only completed all of CMS’s seven quality measures for half or fewer of the beneficiaries they treated. We found that 130 providers (112 for-profits and 18 non-profits) had scores below 50 percent on this measure. These providers treated over 24,000 beneficiaries. In addition to the provider-reported quality measures, CMS also uses the caregivers’ experience survey to assess quality of care. We analyzed CMS data on caregivers’ experience surveys for 2016 to 2017 and found that caregivers’ reported experience with hospice care was generally similar for both for-profits and non-profits. The survey assesses care in a number of areas, such as communication, training, and help with pain and symptoms. See table 5 (for more detailed data, see app. I, table 10). Although for-profit and non-profit providers’ average scores on the caregivers’ experience survey were generally similar, we found that for- profit providers were more often among those providers with the lowest scores on certain caregivers’ experience measures than were non-profit providers. For example, on the rating measure that asks caregivers to give an overall rating of the hospice, 290 providers (248 for-profit providers and 42 non-profits) had scores at the 10th percentile or lower, meaning that their score was 72 percent or lower. For this measure, lower scores mean that fewer caregivers provided a rating of 9 or 10 on a 10- point scale, with 10 being the highest possible rating. We found that 15 percent of for-profit providers were among providers with scores in the 10th percentile or lower compared to 4 percent of non-profit providers. We used Medicare claims data to calculate certain measures researchers told us could be indicators of quality of care in hospice settings. (As noted previously, CMS is working to account for other care indicators by developing additional quality measures.) These indicators fall into two categories: (1) the number of beneficiaries discharged prior to death (often referred to as the live discharge rate) and (2) provider visits to provide medical and emotional support to the beneficiary and caregivers near the end of a beneficiary’s life. Researchers told us that such measures can fill gaps in assessing the quality of care provided by hospices, and show greater variability across hospices than CMS’s current quality measures; as previously noted, our data analysis found that providers’ quality measure scores were generally very high. According to researchers we interviewed and studies we reviewed, some discharges from hospice care prior to death should be expected because, for example, patients change their mind about receiving hospice care or their condition improves and they are no longer eligible for hospice care. However, a high live discharge rate could in some cases be an indicator of poor quality of care provided or of provider misuse of the benefit, in that they may be enrolling beneficiaries who are not eligible for hospice. See text box. Live Discharges In some cases, a beneficiary may be discharged alive from hospice care prior to their death. This could be for reasons unrelated to the quality of care provided. For example, beneficiaries may reconsider their decision to start palliative treatment, and therefore leave hospice care to re-start curative treatments. In other instances, a live discharge may indicate quality of care issues. For example, a beneficiary may be unhappy with the quality of care she is receiving from her hospice provider and therefore she leaves that hospice provider to seek treatment from a different hospice provider. Given the various reasons for live discharges, we expect that hospices will have some live discharges, but interpret a high rate of live discharges as potentially suggestive of quality of care issues. We found that for-profits had higher rates of live discharges than non- profits, with 22.1 percent of beneficiaries served by for-profits being discharged alive compared to 12.0 percent of beneficiaries served by non-profits in 2017. This disparity remained true after accounting for whether beneficiaries had a cancer or non-cancer diagnosis. (See table 6; for more detailed data from 2014 to 2017, see app. I, table 11.) We found that 472 hospice providers (462 for-profit and 10 non-profit providers) had live discharge rates of 50 percent or more in 2017, meaning that half or more of their beneficiaries were discharged from hospice care prior to death. These providers provided care to about 6 percent of all beneficiaries discharged alive in 2017. According to researchers we interviewed and one of the studies we reviewed, provider visits near the end of a hospice beneficiary’s life are critical to providing quality care, including for emotional support and for training the beneficiary’s family members or other caregivers on the signs and process of dying. Assessing the number of visits near the end of life may provide insight into the quality of a hospice provider’s care; fewer visits in that time period could indicate poor quality of hospice care. CMS is currently developing a quality measure that assesses the frequency of provider visits at the beneficiary’s end of life. When analyzing CMS claims data, we found that for-profit and non-profit hospices, on average, provided a similar number of provider visits (such as nurse, doctor, social worker, or hospice aide visits) within the last 7 days of a beneficiary’s life. Specifically, in 2017, for-profits and non-profits both averaged about 6 provider visits within the last 7 days of life. We also looked at the average percentage of hospice beneficiaries who received different types of provider visits either within the last 3 days of life or last 7 days of life (consistent with CMS’s new quality measure) and found performance varied among for-profit and non-profit providers: 77 percent of for-profit beneficiaries and 85 percent of non-profit beneficiaries received at least one visit from registered nurses, physicians, or nurse practitioners in the last 3 days of life. 68 percent of for-profit beneficiaries and 57 percent of non-profit beneficiaries received at least two visits from medical social workers, chaplains or spiritual counselors, licensed practical nurses, or hospice aides in the last 7 days of life. We also found more for-profits than non-profits among a subset of hospices that did not provide any visits during the last 3 or 7 days of life in 2017. Specifically, our analysis shows that 83 hospice providers (80 for- profits and 3 non-profits) did not provide any visits in 2017 from registered nurses, physicians, or nurse practitioners in the beneficiaries’ last 3 days of life. This means that all of the 800 hospice beneficiaries treated by these providers did not receive these types of provider visits at the end of life. In addition, we found that 58 providers (55 for-profits and 3 non- profits) did not provide any visits from medical social workers, chaplains or spiritual counselors, licensed practical nurses, or hospice aides in the last 7 days of life in 2017; all of the 613 beneficiaries treated by these providers did not receive these specific provider visits at the end of life. In our review of CMS’s oversight of hospice providers, we found CMS does not instruct surveyors to review, prior to surveying hospice providers, providers’ performance on CMS quality measures (those based on provider-reported quality data or caregivers’ experience surveys) or other indicators of quality that could identify potential areas of concern. CMS issues guidance that surveyors use when conducting surveys to assess a hospice provider’s compliance with federal health and safety requirements. According to this guidance, surveyors are to prepare for hospice surveys by reviewing documents of record including licensure records, previous survey findings and complaints, media reports, and other publicly available information about the provider. A representative for an association representing state surveyors confirmed that this is the type of information surveyors typically review prior to a hospice provider survey. However, according to CMS officials and the surveyor association, CMS does not instruct surveyors to review other information such as providers’ performance on CMS quality measures or other indicators of quality that surveyors could use to identify potential areas of concern that they could focus on more closely during a survey. For example, it might be helpful for surveyors to know if a hospice provided no visits during beneficiaries’ last days of life. According to CMS officials, CMS does not use such information to target hospices for additional survey review. Several studies we reviewed and researchers we interviewed noted CMS could strengthen its survey process by incorporating additional information into the survey process, such as information on how hospice providers perform on CMS quality measures or other potential indicators of quality. For example, one study suggested that hospices with poor reported beneficiary experiences based on caregivers’ experience survey data could be identified for more frequent surveys and that such information could be used to identify care processes for closer review during surveys. Another study we reviewed concluded that claims- based measures could help guide surveyors to more closely review key processes of care to ensure Medicare beneficiaries receive high quality hospice care. In addition, a researcher we interviewed suggested when claims data show no visits during the last 2 days of life, the survey team could interview the deceased patients’ families to see if there was any harm done by the lack of visits at the end of life. And, in July 2019, the Department of Health and Human Services’ Office of the Inspector General (HHS OIG) reiterated recommendations from prior HHS OIG work that CMS analyze claims and deficiency data to identify specific patterns identified by the HHS OIG that could indicate potential issues— such as hospices that infrequently provide physician services—and that CMS instruct surveyors to pay special attention to these areas during surveys. In contrast to hospice surveys, home health agency surveyors utilize information in addition to survey findings and complaints to identify potential areas of concern. According to CMS officials and the surveyor association we interviewed, home health surveyors review certain CMS quality measures to focus the survey on specific areas of concern or to identify beneficiaries who experienced potential care issues for a more detailed survey review. According to CMS officials, the agency is considering making changes to the survey process but has not yet made any decisions. CMS officials told us they last updated the survey process in 2010, and since then, they have implemented quality measures for hospice providers (provider- reported measures in 2014 and caregivers’ experience survey measures in 2015). They also said that CMS is “currently monitoring the implementation of these programs and considering the potential benefit of incorporating review of the data into the survey process.” According to federal standards of internal control, agencies must identify, analyze, and respond to risks related to achieving objectives. By not utilizing additional information in the survey process that would allow it to identify providers and areas where risk of noncompliance is greatest, CMS is missing an opportunity to strengthen its ability to identify and respond to such risks and ensure the quality of care that hospice beneficiaries receive. CMS is limited to one hospice enforcement remedy—termination of the Medicare provider agreement. By law, to qualify for payment under the Medicare program, hospice providers must meet the program’s conditions of participation. If the agency finds a provider is not complying with the program’s conditions of participation, CMS may terminate the provider’s participation in the program. In the Medicare program, termination of a provider is the most significant action CMS can take to address provider non-compliance. As a result, CMS generally only terminates a hospice provider on the basis of a deficiency when the provider fails to correct a condition-level deficiency (the most severe) within the required time frame. Our review of CMS hospice survey data found termination happens rarely. Specifically, 19 hospices were involuntarily terminated from 2014 through 2017. This is less than half of 1 percent of the total number of hospices operating during this time period. In contrast to hospice care, where CMS’s enforcement authority is limited to termination, Congress has given the agency authority to impose additional enforcement remedies for other provider types. Additional statutory and regulatory penalties for home health agencies and nursing homes include civil money penalties, denial of payment for all new Medicare and Medicaid admissions, and imposition of training requirements for situations where it is determined that education will likely lead to provider compliance (referred to as directed in-service training). Such remedies, if available, could enable the agency to more effectively address a broader range of hospice risks. For example, additional remedies could be used in situations that warrant a remedy other than termination or that could further incentivize providers to comply with health and safety requirements or improve their quality of care. According to federal standards of internal control, agencies must identify, analyze, and respond to risks related to achieving objectives. Because CMS lacks the authority to establish such additional remedies, the agency’s ability to respond to risks and ensure quality of care for beneficiaries is limited. The HHS OIG and one researcher we interviewed have recommended CMS seek statutory authority to establish additional enforcement remedies for hospices, explaining that less severe remedies could help address performance problems that may not merit termination and incentivize agencies to improve quality of care. CMS agreed with this recommendation in March 2016 and stated it would consider submitting a request that would seek legislative authority to establish additional enforcement remedies through the President’s annual budget proposal to Congress. In a July 2018 HHS OIG report, the HHS OIG again recommended CMS seek this authority. CMS neither agreed nor disagreed with this recommendation and stated again that it would consider this recommendation when developing the agency’s proposals for the President’s annual budget. However, a request for such legislative authority was not included in the President’s fiscal year 2017, 2018, or 2019 budget proposals. The HHS OIG reiterated this recommendation in two July 2019 reports. Since 2000, the number of Medicare hospice beneficiaries has almost tripled to nearly 1.5 million in fiscal year 2017. In addition, the number of hospice providers has doubled. Given this growth, it is imperative that CMS’s oversight of the quality of Medicare hospice care keeps pace with changes so that the agency can ensure the health and safety of these terminally ill beneficiaries. While recent steps have been taken to strengthen CMS’s hospice quality oversight, including the requirement that hospices be re-certified every 3 years and CMS’s ongoing development of new quality measures, we identified additional opportunities to strengthen CMS’s oversight. Specifically, our review found that CMS could strengthen oversight by using additional information—based on currently available data—to identify potential quality issues that could focus and enhance the survey process. We also found that CMS’s lack of authority to establish additional enforcement remedies before termination, which CMS rarely uses, limits its ability to ensure hospice providers’ compliance with health and safety requirements and quality of care for beneficiaries. Congress should consider giving CMS authority to establish additional enforcement remedies for hospices that do not meet federal health and safety requirements. (Matter for Consideration 1) The Administrator of CMS should incorporate the use of additional information, such as quality measures or other information that could identify potential quality of care issues, into its survey process for overseeing hospice providers. (Recommendation 1) We provided a draft of this report to HHS for review and comment. HHS provided written comments, which are reprinted in appendix II. HHS concurred with our recommendation. HHS stated that it recognizes that meaningful quality measures can also serve as key indicators of provider quality and it will look into ways to incorporate the use of these data into the hospice survey process. In its comment letter, HHS also noted the importance of monitoring patient safety and quality of care to HHS’s hospice oversight efforts and the agency provided an overview of the key efforts it has in place to perform such monitoring. For example, in addition to survey and quality measure requirements, HHS requires hospices to implement a data-driven quality assessment and performance improvement program, intended to have hospices take a proactive approach in improving their performance using objective data. HHS also provided technical comments, which we incorporated into the report as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services, the CMS administrator, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Hospice team treated patient with respect Amount of emotional and religious support provided by the hospice team The patient got the help they needed for pain and symptoms Caregiver received the training they needed Caregiver rating of hospice agency on 10-point scale with 10 being the best hospice care possible Caregiver would recommend the hospice Non-profit hospice providers’ average scores Hospice team treated patient with respect Amount of emotional and religious support provided by the hospice team The patient got the help they needed for pain and symptoms Caregiver received the training they needed Caregiver rating of hospice agency on 10-point scale with 10 being the best hospice care possible Caregiver would recommend the hospice Government-owned hospice providers’ average scores Hospice team treated patient with respect Amount of emotional and religious support provided by the hospice team The patient got the help they needed for pain and symptoms Caregiver received the training they needed Caregiver rating of hospice agency on 10-point scale with 10 being the best hospice care possible Caregiver would recommend the hospice 2.5 survey within three categories (top scores, middle scores, and bottom scores). These data were not available for all hospice providers; our analysis of CMS caregivers’ experience survey quality measure data was for the 2,832 hospice providers that had data for the caregivers’ survey. In general, the top-box scores represent the percentage of caregivers that selected the response of “always” for the particular measure. For the rating measure, the top-box score represents caregivers that rated the hospice provider as a 9 or 10 on a 10-point scale with 10 being the highest rating. For the recommendation measure, the top-box score represents caregivers that responded that they “would definitely recommend the hospice provider.” In general, the middle-box scores represent the percentage of caregivers that selected the response of “usually” for the particular measure. For the rating measure, the middle-box score represents caregivers that rated the hospice provider as a 7 or 8 on a 10-point scale with 10 being the highest rating. For the recommendation measure, the middle-box score represents caregivers that responded that they “would probably recommend the hospice provider.” In addition to the contact named above, Gregory Giusto, Assistant Director; Christie Enders, Analyst-in-Charge; Todd Anderson, Leia Dickerson, Rob Dougherty, Krister Friday, Barbara Hansen, Jennifer Whitworth, and Chris Wickham made key contributions to this report.", "summary": "Since 2000, there has been substantial growth in Medicare payments for hospice services and the number of Medicare beneficiaries using hospice. This growth has been accompanied by an increase in the number of providers (primarily an increase in for-profit providers), reaching approximately 4,500 providers by 2017. GAO was asked to review aspects of Medicare's hospice program. This report, among other things, (1) compares quality scores and other potential indicators of quality for for-profit and non-profit hospices; and (2) examines opportunities for strengthening CMS's oversight of hospice providers. GAO analyzed CMS data on hospice care for 2014 through 2017—the latest years for which full-year data were available at the time of GAO's analysis—and reviewed research on hospice care. GAO interviewed CMS officials, researchers, provider associations, a survey agency association, and a non-generalizable sample of hospice providers selected in part through referrals from other stakeholders. GAO also reviewed relevant statutes, regulations, documents, and enforcement data. Medicare's hospice benefit provides palliative care to beneficiaries with terminal illnesses and a life expectancy of 6 months or less. GAO's review of 2017 data from the Centers for Medicare & Medicaid Services (CMS) found that for-profit and non-profit hospices had, on average, similar scores on CMS's current quality measures that indicate hospice performance in areas such as pain assessment and discussion of beneficiary treatment preferences. However, for-profits were more often among the subset of providers with the lowest scores on certain quality measures GAO reviewed. In addition to analyzing providers' scores on CMS quality measures, GAO analyzed provider performance on other indicators, identified by researchers, that could signal quality issues and found performance varied among for-profit and non-profit hospices. One of the other quality indicators GAO analyzed was the rate of beneficiaries discharged from hospice prior to death, which in some cases could indicate dissatisfaction with care leading to the beneficiary's decision to leave the hospice provider. In addition, GAO examined the number of provider visits to give medical and emotional support within the last few days of a beneficiary's life. With regard to these indicators, for 2017, GAO found the following, among other things: 472 hospice providers (462 for-profits and 10 non-profits) had a high rate of discharging beneficiaries prior to death (50 percent or more were discharged). According to research, a high discharge rate could, in some cases, be an indicator of poor quality of care or of provider misuse of the benefit, in that the hospice may be enrolling beneficiares who are not eligible for hospice care. 83 providers (80 for-profits and 3 non-profits) did not have hospice staff (such as nurses, physicians, or nurse practitioners) visit beneficiaries within the last 3 days of their life—a critical time in providing quality care, according to researchers GAO interviewed. CMS's oversight of the quality of care provided by hospice providers consists primarily of inspections—called surveys—of hospice providers. GAO found that, while CMS instructs surveyors to review previous survey findings and complaints, CMS does not instruct surveyors to use information on providers' performance on quality measures or other potential indicators of quality as part of the survey process. For example, CMS does not instruct surveyors to consider whether a hospice provided staff visits during beneficiaries' last week of life. According to research, this information could be used to enhance the survey process. GAO also found that CMS is limited to one enforcement option—termination of the Medicare provider agreement—which CMS uses rarely and generally only when providers fail to correct within the required time frame the most serious violations of federal health and safety requirements. According to two researchers, additional remedies, such as civil monetary penalties, could enhance CMS's oversight by addressing performance problems that do not merit termination and incentivize agencies to improve quality of care. CMS uses a range of remedies for other provider types, such as home health agencies and nursing homes, but lacks authority to impose such additional sanctions on hospices. CMS should incorporate the use of additional information that could be used to identify quality of care issues into its survey process for hospice oversight. Congress should consider giving CMS authority to establish additional enforcement remedies for hospices that do not meet federal health and safety requirements. The Department of Health and Human Services concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "Federal agencies are dependent on computerized (cyber) information systems and electronic data to carry out operations and to process, maintain, and report essential information. Cybersecurity—the security of these systems and data—is vital to public confidence. Ensuring the cybersecurity of the nation, including protecting privacy and sensitive data, and IRS’s efforts to address tax refund fraud due to identity theft are issues included in our High Risk List. IRS relies on information system security controls to protect the confidentiality, integrity, and availability of the sensitive financial and taxpayer information that resides on its systems. Federal law and guidance specify requirements for protecting federal information and systems. The Federal Information Security Modernization Act of 2014 (FISMA) is intended to provide a comprehensive framework for ensuring the effectiveness of information system security controls over information resources that support federal operations and assets. To accomplish this, FISMA requires each agency to develop, document, and implement an agency-wide information security program to provide security for the information and systems that support the operations and assets of the agency, using a risk-based approach. However, taxpayer information held by third-party providers is generally outside of these requirements, according to IRS officials. Fraudsters may target third parties, such as paid preparers and tax software providers, to steal taxpayer data—defined for our purposes as personally identifiable information and other personal, financial, or federal tax data—which can then be used to commit identity theft refund fraud or other types of financial crimes. Viewed broadly, identity theft tax refund fraud consists of two crimes: (1) stealing or compromising taxpayer data and (2) using stolen (or otherwise compromised) taxpayer data to file a fraudulent tax return and collect a fraudulent refund. Figure 1 presents an example of how this crime can work. In this example, a taxpayer may alert IRS of identity theft refund fraud. Alternatively, IRS can detect identity theft refund fraud through its automated filters that search for specific characteristics, as well as through other reviews of taxpayer returns. Third-party providers retain a large amount of electronic tax information, which makes them targets of various types of data theft incidents. Five common types of security incidents are shown in table 1. The number of electronically filed (e-filed) tax returns, and therefore the amount of electronically available data that are vulnerable to security incidents, has been increasing over the past several decades from 4.2 million in 1990 to 135.5 million in 2018. In 2018, approximately 90 percent of the 150.5 million filed individual income tax returns were filed with IRS electronically (see figure 2). Paid preparers prepared more than half of the e-filed returns in 2018. Multiple IRS offices have discrete responsibilities in overseeing how third- party providers secure taxpayer information, as depicted in figure 3. Oversight responsibilities are as follows: Stakeholder Liaison works with the paid preparer community to educate preparers about information security risks and guide them through the process of resolving security issues when security incidents are reported. This office is also the intake point for security incident information for paid preparers. Cybersecurity works to protect taxpayer information and IRS’s electronic systems, services, and data from internal and external cybersecurity threats—such as damage to computers, electronic communications systems, or information contained in those systems—by implementing security practices. Criminal Investigation (CI) reviews security incident reports to determine whether criminal action has occurred and investigates any potential criminal violations of applicable laws. It also investigates large-scale tax schemes and fraud. The Return Preparer Office is responsible for matters relating to the registration and the program compliance of tax return preparers who prepare returns for compensation. The office also engages in outreach and education programs and administers IRS’s Annual Filing Season program, a voluntary program to encourage noncredentialed preparers to participate in continuing education courses. Small Business/Self-Employed (SB/SE) Examination revenue agents visit e-file providers to ensure they are complying with the Authorized e-file Provider program’s requirements. Electronic Products and Services Support (EPSS) administers the Authorized e-file Provider program. It is also responsible for updating IRS Publications 1345 and 3112, which outline the requirements of the program. EPSS officials reported that they must coordinate with other business units to update individual references in the publications. EPSS is the intake point for security incident information for online providers and e-Services users, according to officials. Return Integrity and Compliance Services (RICS) monitors taxpayer accounts for potential fraud to protect revenue. RICS also manages the security incident data reports that are submitted by tax software providers. RICS is the intake point for security incident information for Security Summit and Identity Theft Tax Refund Fraud - Information Sharing and Analysis Center (ISAC) members, as described below, and actively monitors ISAC alerts from the online platform for new information that may not have been reported elsewhere. While the Office of Professional Responsibility (OPR) does not have oversight responsibilities over the security of tax information at third parties, it administers the regulations that govern the practice of tax professionals who interact with IRS on behalf of taxpayers, including attorneys, certified public accountants, and enrolled agents, among others. Treasury Department Circular 230, which incorporates the regulations, directed the Commissioner to establish OPR and any other offices within IRS to administer and enforce the regulations. However, Circular 230 does not include a requirement for practitioners concerning the security of taxpayer information. In recent years, IRS has taken a number of steps to help battle identity theft refund fraud. In 2015, IRS formed the Security Summit, a public-private partnership to protect the nation’s taxpayers and the tax system from identity theft refund fraud. The summit has representatives from IRS, state tax administrators, and industry partners including the software industry, tax professional associations, and payroll and tax financial product processors. IRS launched ISAC in the 2017 filing season. It aims to allow IRS, states, and tax preparation industry partners to quickly share information on identity theft refund fraud. It includes two components: an online platform controlled by IRS to communicate data on suspected fraud, and a collaborative organization governance structure comprising IRS, states, and industry. IRS uses a Rapid Response Team in partnership with states and industry members to coordinate responses to identity theft refund fraud incidents. The team aims to respond to significant threats within 24 to 72 hours of their discovery. The Rapid Response Team was deployed for six incidents in 2016, one in 2017, and was not deployed for any incidents in 2018. IRS seeks to help safeguard taxpayers’ information and the electronic filing system by prescribing requirements for various types of third-party providers through its Authorized e-file Provider program. These requirements are outlined in Revenue Procedure 2007-40 and Publication 1345, Handbook for Authorized IRS e-file Providers of Individual Income Tax Returns. IRS Revenue Procedure 2007-40 states that the security of taxpayer accounts and personal information is a top priority for the agency. Further, the Revenue Procedure states that it is the responsibility of each IRS Authorized e-file Provider to have security systems in place to prevent unauthorized access to taxpayer information by third parties. Some of the requirements included in this program are applicable to all types of Authorized e-file Providers, while others are applicable to one group or another. Businesses—including sole proprietors—that wish to e-file tax returns on behalf of clients must apply to IRS’s Authorized e-file Provider program and choose a provider type, as described in table 2. According to IRS, in 2018 there were more than 325,000 Authorized e-file Providers, some of which were paid preparers. More than 790,000 paid preparers had registered with IRS as of 2018; accordingly, not all paid preparers are Authorized e-file Providers and are therefore not covered by the requirements of the Authorized e-file Provider program. However, a business that has been approved as an electronic return originator (ERO) may employ multiple paid preparers who are not Authorized e-file Providers. Those paid preparers would be allowed to e-file returns under the supervision of their ERO employer. According to IRS Publication 3112, the activities and responsibilities for return preparation and e-filing are distinct and different from each other. Tax software providers, which IRS refers to as software developers in its Authorized e-file Provider program, develop tax return software that individuals and businesses can use to file their own returns, or that paid preparers can use when filing returns on behalf of clients. Online providers are the subset of tax software providers that allow individual taxpayers to self-prepare returns and file them with IRS. Providers that develop software for paid preparers’ use do not fall under the definition of an online provider. IRS has not fully incorporated the Federal Trade Commission (FTC) Safeguards Rule into its requirements for all provider types under the Authorized e-file Provider program. The Gramm-Leach-Bliley Act provided FTC with the authority to require that financial institutions subject to its jurisdiction ensure the security and confidentiality of customer records and nonpublic personal information; protect against any anticipated threats or hazards to the security of such records; and protect against unauthorized access to or use of such records or information which could result in substantial harm or inconvenience to any customer. FTC, in turn, issued a regulation known as the “FTC Safeguards Rule.” The FTC Safeguards Rule applies to financial institutions including third- party providers that help taxpayers file tax returns, such as paid preparers and providers of software that allows individuals to prepare their own tax returns. The FTC Safeguards Rule requires those institutions to develop, implement, and maintain a comprehensive written information security program. The program must contain administrative, technical, and physical safeguards that are appropriate to the provider’s size and complexity, the nature and scope of the provider’s activities, and the sensitivity of any customer information at issue. IRS addresses the FTC Safeguards Rule through its Revenue Procedure 2007-40. This Revenue Procedure provides the procedures for the Authorized e-file Provider program, and clearly states that violations of the provisions of the Gramm-Leach-Bliley Act and the implementing rules and regulations promulgated by FTC are considered violations of the Revenue Procedure. It also states that violations may subject an Authorized e-file Provider to penalties or sanctions, including suspension or expulsion from the Authorized e-file Provider program. However, the IRS publications that provide further information on the Authorized e-file Provider program only briefly discuss the FTC Safeguards Rule, and do not provide details on the required elements of an information security program. For example: Publication 3112, IRS e-file Application and Participation, states that providers should become familiar with the Privacy and Security Rules that implement the Gramm-Leach-Bliley Act, and with other important information regarding the safeguarding of personal information available on the FTC website. The publication does not detail each of the required elements of an information security program. Publication 1345, Handbook for Authorized IRS e-file Providers of Individual Income Tax Returns, which was updated in February 2019, notes FTC’s role in protecting taxpayer data and generally describes the requirement of implementing and maintaining a comprehensive information security program, including the requirement that administrative, technical, and physical safeguards be appropriate to the business’s size, nature and scope of its activities, and the sensitivity of the customer information. The publication does not detail each of the required elements of an information security program. We identified other IRS publications that are not exclusively related to the Authorized e-file Provider program that discuss the requirements of the FTC Safeguards Rule, as well as other information security measures that serve as leading practices for the broader population of tax professionals. For example, in 2018, IRS updated Publication 4557, Safeguarding Taxpayer Data: A Guide for Your Business. The publication aims to help tax professionals understand basic security steps, recognize signs of data theft, respond to data losses, and understand and comply with the FTC Safeguards Rule. This publication refers to the FTC rule and tax professionals’ responsibilities to create and enact security plans, and provides a checklist from FTC to help third-party providers implement the information security plans. IRS Publication 4600, Tips for Safeguarding Taxpayer Data, also discusses elements of the FTC Safeguards Rule. However, while IRS references these documents in Publications 3112 and 1345, Authorized e-file Providers are not obligated to consult or follow these documents. In addition, most paid preparers do not know about the FTC Safeguards Rule and likely do not have information security plans for their places of business, according to officials from several tax preparation industry groups. Industry group officials also told us that there are misconceptions about who should be responsible for implementing information security. For example, one industry group official said that paid preparers and EROs often think that their tax software providers will provide security services or that their computer firewall or antivirus software will be enough protection. Modifying the Authorized e-file Provider program requirements to explicitly incorporate the FTC Safeguards Rule’s elements of an information security program would be consistent with Internal Control Standards. The standards call for management to consider the external requirements—such as laws, regulations, and standards—and incorporate these requirements into an agency’s objectives when setting the standards for the compliance of other entities. IRS officials told us that they do not believe that federal law provides IRS with any authority to enforce the FTC Safeguards Rule. However, IRS has already stated in Revenue Procedure 2007-40 that compliance with the FTC Safeguards Rule is required for participation in the Authorized e- file Provider program. Modifying its requirements to explicitly state the elements of an information security program as required under the FTC Safeguards Rule would help IRS ensure that all types of Authorized e-file Providers are aware of, and comply with, the FTC Safeguards Rule, which could help them better protect taxpayers’ information. While modifying the Authorized e-file Provider program may not reach paid preparers who are not part of the Authorized e-file Provider program, it will strengthen the controls for EROs, tax software providers, and online providers. IRS’s Authorized e-file Provider program does not outline a set of minimum information security standards for systems used by paid preparers or Authorized e-file Providers. When we reviewed IRS’s publications for Authorized e-file Providers, we found that specific information security standards were outlined for online providers, but there were no specific standards for other types of Authorized e-file Providers or paid preparers. Officials from tax preparation groups we interviewed and IRS raised issues that relate to paid preparers’ system risks. First, the tax preparation industry groups that we spoke with stated that most paid preparers, especially small firms or individual preparers, did not know the steps that they should take to protect taxpayer information on their systems. IRS officials reported that paid preparers often do not know that they experienced a security incident until IRS informs them something is wrong with their filing patterns. Second, according to officials from several tax preparation industry groups, paid preparers often have several misconceptions as to what is required of them in protecting taxpayer data, causing confusion. Industry group officials we interviewed told us that IRS’s current publications are not clear about requirements versus leading practices. For example, IRS publication 4557, Safeguarding Taxpayer Data, provides paid preparers with some leading practices to protect taxpayer data, but the leading practices are not legal requirements, with the exception of the FTC Safeguards Rule. An official from the Return Preparer Office explained that imposing any standards for paid preparers, whether related to competency or information security, without explicit authority would leave IRS vulnerable to legal challenges because of a recent court case that found that IRS does not have the authority to regulate the competency of paid preparers. According to IRS’s Office of Chief Counsel, this ruling, combined with the lack of explicit statutory authority, prevents IRS from establishing system standards for paid preparers, because while 31 U.S.C. § 330 authorizes the Secretary of the Treasury to regulate the practice of practitioners before the Department of the Treasury, mere return preparation, including through systems practitioners use to prepare and transmit tax returns, is not considered practice before IRS. In contrast to paper filing of tax returns, certain security measures need to be taken for e-filing returns to protect the integrity of the e-file system; thus, IRS has implicit authority to regulate e-file providers insofar as their activities relate to electronically filing returns with IRS, according to IRS Office of Chief Counsel officials. These officials also noted that no single provision of the Internal Revenue Code provides IRS explicit authority to regulate the standards for e-file providers. Instead, Internal Revenue Code § 7803 gives the Commissioner of Internal Revenue broad authority to administer and supervise the internal revenue laws, and § 6011 authorizes IRS to require returns and regulate the form of such returns. When taken as a whole, these provisions of the Internal Revenue Code show congressional intent to provide the Secretary of the Treasury with broad authority to administer the method for, and requirements surrounding, the e-filing of federal tax returns, according to IRS officials. Nevertheless, having explicit authority to establish security standards for the systems of Authorized e-file Providers may help IRS better ensure the protection of taxpayers’ information and mitigate the risk of legal challenges to IRS’s ability to do so. IRS Office of Chief Counsel officials also noted that for several years the Department of the Treasury has sought additional authority for IRS to regulate all tax return preparers. For example, this request was included in the most recent (fiscal year 2020) Congressional Budget Justification. The justification for this additional authority specifically refers to the competency of tax return preparers, but does not mention security standards for the systems that those preparers use. Similarly, we have previously suggested that Congress consider granting IRS the authority to regulate the competency of paid preparers (that suggestion did not cover regulating the security of paid preparers’ systems). As of April 2019, Congress had not provided such authority. Without Congress providing IRS with explicit authority to regulate the security requirements for the systems of paid preparers or Authorized e- file Providers, Congress and IRS have limited assurance that the processes used by paid preparers or Authorized e-file Providers are adequately protecting taxpayers’ information against electronic data breaches and potential identity theft tax refund fraud. Having such explicit authority would enable IRS to establish minimum security requirements and help ensure improved taxpayer information security by paid preparers and Authorized e-file Providers. IRS does not have a robust set of information security requirements for all tax software providers in the Authorized e-file Provider program. Instead, IRS has limited security requirements for the subset of tax software providers designated as online providers outlined in IRS Publication 1345, as we discuss in the next section. In Publication 4164, Modernized e-File Guide for Software Developers and Transmitters, IRS also provides some information on “security directive rules of behavior for accessing IRS business systems” while transmitting returns to IRS. However, this document does not provide a specific list of controls to for these providers to follow. IRS has been working with the Security Summit to implement a subset of the NIST Special Publication 800-53 security and privacy controls for the industry members of the Security Summit, which represents a subset of all tax software providers. The Security Summit partners agreed voluntarily to implement about 140 tax-related controls over a 3-year period and provide self-assessments related to the implementation of those controls. IRS reported in October 2018 that 15 of the 21 Security Summit industry partners had voluntarily certified that they implemented the NIST controls in years 1 and 2 of the rollout schedule. IRS officials reported that they later determined three of the other 21 industry partners are financial institutions that do not handle taxpayer data; thus the standards are not applicable to them. IRS officials told us that they are actively following up with the remaining three providers to determine why they have not completed and submitted the self-assessment, and to what degree they have implemented the subset of NIST security controls. While this is an important and significant first step, the 15 industry partners in the Security Summit that are voluntarily adhering to the NIST security controls represent about a third of all of the tax software providers that IRS has approved to be a part of the Authorized e-file Provider program. According to IRS, these 15 Security Summit partners transmitted about 132.6 million (98.8 percent) of all of the electronically filed returns in 2018; the other two-thirds of tax software providers in the Authorized e-file Provider program transmitted about 1.6 million (1.2 percent) electronically filed returns. A Security Summit membership criterion states that only those providers that filed more than 50,000 returns with IRS during a filing season can be members, but not all tax software providers meet this threshold. Internal Control Standards state that managers consider external requirements when defining objectives, such as those set by standard- setting bodies designed to comply with laws, regulations or standards. Management should incorporate those requirements into its objectives and sets those requirements through the established standards of conduct, oversight structure, organizational structure and expectations of competence. By statue, NIST is responsible for developing information security standards and guidelines, including minimum requirements for federal information systems. According to Special Publication 800-53, the controls outlined provide a holistic approach to information security and risk management by providing organizations with the breadth and depth of security controls necessary to fundamentally strengthen their information systems and the environments in which those systems operate—contributing to systems that are more resilient in the face of cyber attacks and other threats. While the guidelines in this publication are applicable to all federal information systems, other organizations are encouraged to consider using the guidelines, as appropriate. The applicability of the selected NIST controls is evidenced by the adoption of those controls by the Security Summit partners. While most returns are filed through tax software providers that are voluntarily adhering to the security controls, these controls are not required and do not apply to all tax software providers. Additionally, IRS officials that are a part of the Security Summit stated that they cannot enforce the subset of NIST controls with the remaining Security Summit partners because the controls were set up in a voluntary program. IRS officials from multiple offices did not have a clear reason as to why this subset of NIST controls has not been incorporated into the requirements for the entire population of tax software providers in the Authorized e-file Provider program, even though some security standards had been incorporated into the Authorized e-file Provider program for a limited set of providers (online providers) as discussed in the next section. In addition, as previously discussed, IRS can prescribe the requirements to which Authorized e-file Providers must adhere when e-filing returns for taxpayers. Incorporating fundamental security controls into its Authorized e-file Provider program would give IRS greater assurance that tax software providers have identified and addressed information security risks consistent with professional standards. This missed opportunity to update the requirements for tax software providers by adopting the subset of NIST controls is due, in part, to IRS’s lack of a centralized leadership over the security of taxpayer information collected by paid preparers and tax software providers. As previously discussed, multiple IRS offices have discrete responsibilities for overseeing the security of taxpayer information while at third parties; however, no one office is responsible for, or has the authority to provide, the strategic vision, oversight, or coordination over all aspects. Further, while IRS offices coordinate to some extent, there is not a formalized governance structure, such as a steering committee, that would help provide this level of leadership, coordination, and collaboration to the agency. According to Internal Control Standards, an agency’s organizational structure provides management’s framework for planning, directing, and controlling operations to achieve agency objectives. Management develops an organizational structure with an understanding of overall responsibilities, and assigns these responsibilities to discrete units to enable the organization to operate in an efficient and effective manner and reliably report quality information. A sound internal control environment requires that the agency’s organizational structure clearly defines key areas of authority and responsibility, and establishes appropriate lines of reporting. Without setting and requiring the same security standards for all tax software providers, IRS does not have assurance that these providers have an equivalent level of standards in place to adequately protect taxpayer information. Further, in continuing to operate a voluntary security controls program, IRS does not have assurance that those software providers who are currently adhering to the standards will continue to do so in the future. Finally, without centralized leadership in this area, it is unclear how IRS will adapt to changing security threats in the future and ensuring those threats are mitigated. Online providers—tax software providers that allow individuals to prepare their own tax returns—have additional requirements for security and privacy that they must follow, as outlined in Publication 1345. IRS established six security, privacy, and business standards for online providers, including requirements for developing information privacy and security policies and reporting security incidents. Compliance with these six standards for online providers became mandatory on January 1, 2010; however, IRS has not substantially updated them since then (see appendix II for the text of the six security, privacy, and business standards). These additional requirements do not apply to paid preparers, EROs, or providers of tax software used by paid preparers. Without updating standards regularly, the standards can become outdated and lose their ability to protect information from known vulnerabilities as technology changes. For example, IRS’s current guidance refers to an outdated encryption standard. Specifically, IRS requires online providers to use, at minimum, Secure Sockets Layer 3.0 and Transport Layer Security (TLS) 1.0. However, NIST Special Publication 800-52 and industry leading practices recommend the use of TLS 1.1 as the minimum level of encryption due to known weaknesses of using TLS 1.0 to encrypt data in transmission. While the standard allows for use of later encryption versions, it refers to a minimum encryption standard that has known weaknesses. As a result, IRS and taxpayers have limited assurance that their taxpayer data are protected according to NIST guidelines and industry leading practices. Recommended controls outlined in NIST Special Publication 800-53 and our Fraud Risk Framework call for continuous monitoring and regular fraud risk assessments, respectively, to help determine the effectiveness of controls in a program. Internal Controls Standards also calls for management to periodically review the policies, procedures, and related activities for continued relevance and effectiveness in achieving the entity’s objectives or addressing related risks. When we asked why the six standards in Publication 1345 had not been updated since 2010, a senior Wage and Investment Division official stated that the publication is subject to an annual review by multiple IRS offices, but no office had identified the need to update the standards as part of these reviews. An Electronic Products and Support Services (EPSS) official told us that the standards were initially developed based on the latest technology at the time. However, according to this official, technology can become obsolete quickly, and adapting standards to keep pace with technological changes can require a lot of resources. Not updating the requirements for online providers again points to a missed opportunity due to IRS’s lack of a centralized leadership over the security of taxpayer information at paid preparers and tax software providers. In this case, centralized leadership may have identified the need to update the standards. Without periodically reviewing and updating the standards themselves, IRS has limited assurance that the standards have kept pace with technological changes, and therefore, that the online providers are protecting the taxpayer’s data. IRS uses a variety of outreach tools to communicate with third-party providers, such as paid preparers and tax software providers, about information security risks. IRS tries to educate these tax professionals about ways to improve information security practices and the benefits of doing so. For example, IRS informs paid preparers, tax software providers, and others about the importance of reporting security incidents in a timely manner to help ensure that action can be taken quickly to help protect their clients and avoid fraudulent returns being filed. Similarly, Stakeholder Liaison advises paid preparers about the steps to take to ensure that their systems are no longer vulnerable to compromise, according to Stakeholder Liaison officials. Below are examples of IRS’s recent communication efforts. IRS and the Security Summit collaborated on tax professional outreach campaigns. For example, in 2018, they launched the Tax Security 101 campaign, which provided tax professionals with basic information on how to protect taxpayer data. Each year, IRS sponsors nationwide tax forums largely targeted toward paid preparers such as enrolled agents, certified public accountants, and noncredentialed preparers. The 2018 forum included five seminars focused on securing taxpayer information, such as “Data Privacy and Cybersecurity for Tax Professionals” and “Data Compromises—It’s Not a Matter of ‘If’ but ‘When.’” IRS hosts webinars throughout the year to inform tax professionals and taxpayers about various topics, including information security. For instance, in October 2018, IRS hosted a webinar called “Protect Your Clients, Protect Yourself: Tax Security 101.” The webinar covered common security threats, signs of data theft, ways to report taxpayer data theft to IRS, and tax preparers’ obligations to create a written information security plan consistent with the FTC Safeguards Rule. Stakeholder Liaison has participated in over 1,000 virtual and in- person events since June 2015 where data security was a primary topic or featured message, according to Stakeholder Liaison officials. Further, the officials reported that there were over 165,000 attendees at these events. IRS uses social media outlets such as YouTube and Twitter to provide information to tax professionals. For example, in July and October 2017, IRS released two YouTube videos about information security for tax professionals titled “Why Tax Professionals Need a Security Plan” and “What to Do After a Tax Professional Data Compromise.” Similarly, IRS’s tax professional Twitter account, @IRStaxpros, releases information about information security (see figure 4). Though IRS has various ways to disseminate information to tax professionals, it faces a challenge reaching paid preparers who are not affiliated with larger industry groups or who do not visit the IRS.gov website, according to both IRS officials and industry group officials. According to Return Preparer Office officials, many paid preparers are not linked to standard tax communication channels, such as direct communications from IRS through news releases or email alerts. IRS and industry group officials told us one barrier to reaching these paid preparers is preparers’ belief that their businesses are too small to be a target for fraudsters. IRS officials recognize the challenges and said that they continue to address them by speaking with tax professionals about how to increase paid preparers’ awareness of information security risks, such as by making materials easy for preparers to read. IRS’s monitoring program is primarily focused on EROs’ adherence with multiple aspects of the Authorized e-file Provider program, such as requirements for Earned Income Tax Credit due diligence, advertising, and electronic signatures. The monitoring program also calls for monitoring of physical information security, which is not required as part of the Authorized e-file Provider program. The Internal Revenue Manual (IRM) details mechanisms and practices for monitoring Authorized e-file Providers, including EROs and online providers. As part of this monitoring, Small Business/Self-Employed (SB/SE) conducts field visits, the number of which more than doubled in the past few years, from almost 300 in 2015 to about 650 in 2018. SB/SE revenue agents visit providers to monitor their operations and to advise providers of any program violations. IRS uses monitoring visits to investigate allegations, complaints, and warnings against Authorized e-file Providers, as well as to determine general compliance with program requirements. While any provider type could undergo a monitoring visit, IRS officials informed us that they primarily conduct field monitoring visits for EROs, which are selected using risk-based criteria. According to these officials, SB/SE coordinates with other IRS offices to provide field monitoring on an as-needed referral basis for other types of Authorized e-file Providers. IRS officials reported that they were unable to confirm the specific number of recent referral monitoring visits but said there were likely fewer than five referrals in the past couple of years. However, the IRM section detailing the monitoring visits provides little direction for monitoring of information security standards from IRS Publication 1345. The IRM lists monitoring techniques for security, but they focus largely on physical security rather than cybersecurity controls for the electronic aspects of information security. For example, the IRM suggests that agents ask about access to physical files or office keys rather than about how providers send emails containing taxpayer information. According to our Fraud Risk Framework, agencies should use a risk- based approach to evaluate outcomes and adapt activities to improve fraud risk management. As fraudsters increasingly target paid preparers and tax software providers through cybersecurity attacks, risk-based monitoring and evaluation of cybersecurity controls could help IRS identify fraud risks and potential control deficiencies among third-party providers. IRS officials said that the SB/SE revenue agents who conduct monitoring visits do not have the technical expertise to effectively monitor information security or cybersecurity controls. For example, an IRS official stated that the IRM monitoring techniques ask about physical security instead of cybersecurity because revenue agents can verify whether filing cabinets are locked or whether computer passwords are visible, but they cannot verify cybersecurity controls, such as whether a provider’s information security policies are consistent with government and industry guidelines. Further, an SB/SE official said that, while SB/SE is responsible for monitoring Authorized e-file Providers, cybersecurity is not part of SB/SE’s role. However, we believe there are opportunities for revenue agents to ask basic cybersecurity questions and, at a minimum, use monitoring visits to help promote awareness of leading practices designed to help protect taxpayer information. For example, revenue agents could ask providers if they have secured their office’s wireless capabilities, use encryption for sensitive business information, have a designated official in case of a security incident, or know their assigned stakeholder liaison, among other things. Additionally, opportunities exist to leverage resources across IRS to monitor cybersecurity controls. For instance, Cybersecurity has technical expertise that SB/SE could leverage to help monitor these requirements, according to a Cybersecurity official. Without effective monitoring of information security standards or cybersecurity controls, IRS has limited assurance that EROs’ systems are adequately protecting taxpayers’ information. If these third parties do not adequately protect that information, taxpayers will face increased risk of both tax-related and non-tax-related identity theft. Improved monitoring could help IRS ensure that it is more effectively detecting and responding to changing fraud risks among providers. Additionally, updating documentation of monitoring activities, as needed, such as the IRM and internal guidance, along with staff training, would provide IRS with better assurance that the greatest risk areas are addressed appropriately. IRS conducts limited monitoring of the online provider subset of tax software providers enrolled in the Authorized e-file Provider program. However, these monitoring efforts are not part of the systematic Authorized e-file Provider monitoring program for EROs described above, nor are they documented in the IRM or relevant job aids. According to EPSS officials, IRS does not currently monitor all of the standards for online providers. IRS staff can remotely monitor three of the six security, privacy, and business standards for online providers through electronic means, according to EPSS officials (see table 3). EPSS officials stated that the other three standards cannot be monitored remotely (see appendix II for the full text of the six security, privacy, and business standards). For two of the three standards that cannot be monitored remotely, EPSS officials said it would be feasible for online providers to send the results of vulnerability scans (standard 2 in table 3) and privacy seal vendor certifications (standard 3 in table 3) to IRS for monitoring purposes. However, according to these officials, EPSS does not have dedicated staff who could review these results. Similarly, SB/SE, which conducts Authorized e-file Provider monitoring, does not have the technical expertise to review these results, as previously discussed. In addition, IRS cannot monitor the requirement to report security incidents, according to officials, because there is no way for the agency to know whether security incidents have occurred but were not reported. However, every fiscal year, IRS asks online providers to self-certify that they are meeting all six of the security, privacy, and business standards in IRS Publication 1345, according to an EPSS official. To self-certify, providers answer “yes” or “no” questions about whether they have complied with each standard. According to this official, companies generally indicate that they are meeting all of the standards. In addition to inconsistent monitoring of online provider requirements, IRS has not recently assessed the information security risks among all third- party provider types. IRS initially implemented the Authorized e-file Provider monitoring program described above only for EROs because they presented the greatest risk for fraud, according to an EPSS official. However, IRS’s monitoring practices and the associated IRM section have not been updated since 2011, and still reflect IRS’s initial assumption that EROs present the greatest risk for fraud among the different provider types. Additionally, while IRS assessed the security and privacy risks of tax software providers, the assessment did not compare these risks to those presented by EROs. In 2009, we recommended that IRS assess the extent to which the reliance on tax software creates significant risks to tax administration, including the security and privacy of taxpayer information. IRS agreed with our recommendation and in 2011 received the results of a third-party risk assessment to determine, in part, the security and privacy risks presented by large and small software providers. The assessment found that security presented the biggest overall risk among the areas reviewed—security of information, privacy of information, accuracy of returns, and reliability of systems—due, in part, to security being the least adequately controlled risk area by small software providers. This assessment was not designed to review the risks for other Authorized e-file Provider types, such as EROs. Our Fraud Risk Framework requires agencies to plan regular fraud risk assessments and suggests tailoring those assessments to the program. Effective managers plan to conduct such assessments at regular intervals and when there are changes to the program or operating environment, such as changes in technology that could result in increased security incidents. As part of a risk assessment, managers may examine the suitability of existing fraud controls. Such examination can help managers identify areas where existing control activities are not suitably designed or implemented to reduce risks to a tolerable level. By conducting a risk assessment for the Authorized e-file Provider program and identifying the provider types that present the greatest risks for fraud, IRS can better determine whether changes to the monitoring program are needed for each provider type. If the agency determines that changes are needed, updating documentation of monitoring activities— such as the IRM, internal guidance, and job aids, along with staff training—would provide IRS with better assurance that the greatest risk areas are addressed appropriately. Multiple offices within IRS use information on security incidents to track trends in fraud schemes, which helps them to protect taxpayer information and to prevent the filing of fraudulent tax returns. For example, when Stakeholder Liaison receives reports about a security incident involving a paid preparer, staff collect additional information about the incident, including the cause of the incident and whether taxpayer information was compromised. Stakeholder Liaison can analyze the data to show geographical information, like the states most affected by breaches; the paid preparer types most affected by incidents; and the method of attack of incidents; among other things, according to a Stakeholder Liaison official. This official said that Stakeholder Liaison also uses this information to produce daily management reports to keep leadership apprised of the number of incidents reported daily, as well as the cumulative number of affected preparers and taxpayers during the year and a comparison to data from the previous year. Return Integrity and Compliance Services (RICS) officials use a risk- based method to determine the necessary mitigation and treatment plans following a security incident. For example, RICS officials might assess a security incident as high risk, meaning that a taxpayer’s personal, financial, and tax data were compromised. For such an incident, RICS officials place the affected Taxpayer Identification Numbers (TIN) on Dynamic Selection Lists—lists of TINs affected in breaches and at risk of tax-related identity theft—to monitor future tax return filings for potential fraud. On the other hand, for low-risk incidents—incidents where fraudsters may have accessed information like street address or date of birth but not Social Security numbers—RICS may compare victims’ current tax returns with prior returns to look for differences that could indicate possible identity theft. According to RICS officials, the office also runs individuals’ information through fraud filters to help identify returns with a high likelihood of identity theft. Criminal Investigation’s (CI) Cybercrimes unit shares security incident information with the field offices where the incident occurred, according to CI officials. Area coordinators evaluate the incident information and determine whether a criminal case should be developed. If so, coordinators develop a fraud scheme package and provide it to the agent assigned to the case to help identify other potential incidents resulting from similar schemes, according to CI officials. IRS has primarily tracked information on security incidents in its RICS Incident Management Database since December 2016, according to RICS officials. Security incidents can be categorized in a number of ways, such as when hackers infiltrate third-party providers’ systems. Between 2017 and 2018, there was an overall decrease in the number of reported high-risk security incidents that led to confirmed identity theft victims across all types of security incidents. However, the number of reported security incidents from third-party providers increased about 50 percent during this same period, as shown in table 4. In turn, the number of taxpayers affected by the security incidents at third-party providers also increased. However, IRS does not have comprehensive information about the incidents because, in part, its reporting requirements do not apply to all third-party providers. For example, the Authorized e-file Provider program requires only online providers to report security incidents to IRS as soon as possible but no later than the next business day after confirmation of the incident. The information that online providers are to report includes details about the security incident and the affected taxpayers’ accounts. If paid preparers or EROs experience a security incident at their place of business, they are not required to report any information to IRS about the incident; instead, IRS encourages paid preparers to share security incident information with IRS through Stakeholder Liaison. Additionally, IRS cannot track incidents that third-party providers do not report, according to IRS officials. IRS officials and industry representatives stated that some third-party providers may not report security incidents for fear of punishment from IRS (e.g., penalties, sanctions, or removal from the Authorized e-file Provider program) or negative impacts to their business reputation. IRS has other voluntary reporting mechanisms for tax software providers or other members of the tax preparation industry. For example, members of the Security Summit can use a voluntary reporting mechanism to submit information to RICS. Some members of the Security Summit can use an additional voluntary reporting system in the ISAC online platform, which sends alerts about security incidents to others in the platform. IRS also recently revised some of its requirements that could affect paid preparers’ reporting of security incidents while using other IRS services. For example, in October 2018, the agency updated its user agreement for e-Services, a suite of web-based tools that allow paid preparers, among others, to complete transactions online with IRS. This update included a requirement to report any unauthorized use of the e-Services account or any other breach of security as soon as users become aware of the incident. According to Internal Control Standards, agencies should use quality information, both internal and external, to achieve objectives. For example, agencies should obtain data on a timely basis so that they can be used for effective monitoring. Additionally, recommended controls in NIST Special Publication 800-53 require reporting of suspected security incidents by federal agencies and their subordinate organizations. Though IRS conducts a yearly review of requirements for Authorized e- file Providers to find needed updates, the incident reporting requirement has not been identified as needing updates since 2010, according to a senior Wage and Investment official. This is another instance where centralized leadership could have identified a need to update the incident reporting requirements. According to an EPSS official, IRS originally applied this incident reporting requirement to only online providers because these providers stored a large amount of data and carried the highest risk of data loss. Similarly, IRS officials said the reporting requirement for online providers does not apply to providers of tax software used by paid preparers because those software providers do not collect or store taxpayer information on their systems. Instead, the taxpayer information is stored on a paid preparer’s hard drive. If a security incident occurred at the business of a paid preparer who uses tax software, then the preparer, not the tax software provider, would report that incident to IRS, according to IRS officials. While voluntary reporting mechanisms and updating of user agreements for IRS’s website are important steps, without a clear and standardized reporting requirement for all types of providers, IRS will not have assurance that third-party providers consistently report their security incidents in a timely manner. IRS needs this information to better understand the size and scope of information security incidents, which it uses to protect compromised individual taxpayer accounts and prevent identity theft refund fraud. Security incident information can be reported to IRS through various channels from the public to IRS offices, and the data are ultimately stored in the RICS Incident Management Database regardless of the office that initially received the information. Figure 5 depicts the flow of information from the public to IRS offices, as well as the flow of information between the offices and to IRS databases. While RICS has documented its information intake, tracking, and storage processes in the RICS Incident Management Plan, IRS does not have a comprehensive document that describes these processes across the different IRS offices. For example, incident information submitted to EPSS and Stakeholder Liaison eventually moves to RICS to be tracked in the Incident Management Database. Additionally, RICS officials told us that they track each of these reported incidents separately and that the main repository should not contain duplicate reports of the same incidents, though multiple databases may contain information about the same incident. RICS officials added that, before a new incident is added to the Incident Management Database, staff conduct a query in the database to ensure that the incident was not already added. However, IRS has not documented how the security incident data processes should flow, relying instead on informal communication efforts of the staff and the assumption that staff know where the data belong and will provide that information to the appropriate offices. Internal Control Standards state that management should develop and maintain documentation of its internal control system and implement control activities through policies. The standards also state that documentation of responsibilities through policies and periodic review of activities can contribute to the effectiveness of implementation. This limited nature of the documentation may be due to the newness of some of these data processes. For example, a Stakeholder Liaison official told us that the data intake process for Stakeholder Liaison and entry into the Return Preparers Database started at the beginning of 2018. Prior to that, a Stakeholder Liaison manager stored information about security incidents in an individual email account because there was no mechanism for storing the data in a systematic manner. Further, a senior Wage and Investment Division official stated that the processes to intake, store, and share the data among the different IRS offices continue to evolve, and that documents describing these practices may quickly become obsolete. While these processes may still be evolving, documenting them can help IRS combat identity theft by helping to ensure that security incidents are properly recorded and monitored in the IRS systems. Documenting the processes may also allow for more complete data, as the data would follow a specific routing and review process. This would reduce the risk of the data not following the various channels they go through now. Such documentation can also help IRS retain organizational knowledge, mitigate the risk of having that knowledge limited to a few personnel, and ensure that the agency implements these processes effectively in the future. Tens of millions of taxpayers use third-party providers, such as paid preparers or tax software providers, to comply with their federal income tax obligations. It is critical that taxpayers’ information, which includes personally identifiable and other sensitive information, be kept secure to maintain public confidence and avoid data breaches that expose that information for use by fraudsters. Identity theft is a constantly evolving crime, but IRS’s information security standards for third-party providers’ systems have not kept pace with the changing environment. One reason for this is that IRS lacks the explicit authority to require minimum standards for the systems of paid preparers and Authorized e-file Providers. Without this authority, Congress and IRS have limited assurance that the processes used to collect, store, and submit taxpayers’ returns adequately protect taxpayers’ information against electronic data breaches and potential tax refund fraud. Modifying its Authorized e-file Provider program requirements to explicitly state the elements of an information security program as required under the FTC Safeguards Rule would help IRS ensure that Authorized e-file Providers are aware of, and comply with, the rule. Doing so could also help these providers better protect taxpayers’ information. Additionally, IRS is missing an opportunity to capitalize on the achievements of Security Summit members to help ensure that tax software providers have an equivalent level of standards in place to adequately protect taxpayer information. The lack of centralized leadership at IRS with responsibility for coordinating all aspects of protecting taxpayer information held by third- party providers has enabled missed opportunities. Such designated leadership could help ensure greater collaboration between the various IRS offices that have roles to play in this area. This leadership could have also ensured that security standards for online providers in the Authorized e-file Provider program would have been updated. Instead, IRS introduced these standards in 2010 and has not subsequently updated them. Incorporating cybersecurity into its monitoring visits for EROs would provide IRS with greater assurance that EROs’ systems are adequately protecting taxpayers’ information from an increased risk of both tax- related and non-tax-related identity theft. Further, ensuring that IRS is using a risk-based approach to review all types of Authorized e-file Providers would provide assurance that the greatest risk areas of fraud are addressed appropriately. Finally, IRS’s efforts to protect taxpayer information at third-party providers would also be strengthened by greater consistency in requirements across provider types for reporting security incidents. Greater consistency would help to ensure IRS is obtaining timely and reliable information from third-party providers so IRS can better understand the size and scope of security incidents—data it uses to protect compromised individual taxpayer accounts and prevent identity theft refund fraud. Documenting the intake, storage, and sharing of the security incident data would also help IRS ensure that the security incidents are properly recorded and monitored. Congress should consider providing IRS with explicit authority to establish security requirements for the information systems of paid preparers and Authorized e-file Providers. (Matter for Consideration 1) We are making the following eight recommendations to IRS. The Commissioner of Internal Revenue should develop a governance structure or other form of centralized leadership, such as a steering committee, to coordinate all aspects of IRS’s efforts to protect taxpayer information while at third-party providers. (Recommendation 1) The Commissioner of Internal Revenue should modify the Authorized e- file Provider program’s requirements to explicitly state the required elements of an information security program as provided by the FTC Safeguards Rule. (Recommendation 2) The Commissioner of Internal Revenue should require that all tax software providers that participate in the Authorized e-file Provider program follow the subset of NIST Special Publication 800-53 controls that were agreed upon by the Security Summit participants. (Recommendation 3) The Commissioner of Internal Revenue should regularly review and update the security requirements that apply to tax software providers and other Authorized e-file Providers. (Recommendation 4) The Commissioner of Internal Revenue should update IRS’s monitoring programs for electronic return originators to include techniques to monitor basic information security and cybersecurity issues. Further, IRS should make the appropriate revisions to internal guidance, job aids, and staff training, as necessary. (Recommendation 5) The Commissioner of Internal Revenue should conduct a risk assessment to determine whether different monitoring approaches are appropriate for all of the provider types in the IRS’s Authorized e-file Provider program. If changes are needed, IRS should make appropriate revisions to the monitoring program, internal guidance, job aids, and staff training, as necessary. (Recommendation 6) The Commissioner of Internal Revenue should standardize the incident reporting requirements for all types Authorized e-file Providers. (Recommendation 7) The Commissioner of Internal Revenue should document intake, storage, and sharing of the security incident data across IRS offices. (Recommendation 8) We provided a draft of this report to the Commissioner of Internal Revenue for review and comment. In its written comments, which are summarized below and reproduced in appendix III, IRS agreed with three of the recommendations and disagreed with five of the recommendations. IRS also provided technical comments, which we incorporated as appropriate. IRS agreed with our recommendations to regularly review and update the security requirements that apply to the tax software provider and other Authorized e-file Providers; standardize the incident reporting requirements for all types of Authorized e-file Providers; and document intake, storage, and sharing of the security incident data across IRS offices. IRS did not provide additional detail on the actions it plans to take to address these recommendations. IRS disagreed with five of our recommendations, generally citing for all of them the lack of clear and explicit authority it would need to establish security requirements for the information systems of paid preparers and others who electronically file returns. For our recommendation to develop a governance structure or other form of centralized leadership, IRS stated it would require statutory authority that clearly communicates its authority to establish security requirements for the information systems of paid preparers and others who electronically file tax returns. Further, IRS stated that without such authority, implementing the recommendation would be an inefficient, ineffective, and costly use of resources. We disagree that convening a governance structure or other centralized form of leadership would require additional statutory authority or be inefficient, ineffective, and costly. As discussed in the report, IRS has seven different offices across the agency working on information security-related activities that could benefit from centralized oversight and coordination, such as updating existing standards, monitoring Authorized e-file Provider program compliance, and tracking security incident reports. We continue to believe that establishing a governance structure would help provide this level of leadership, coordination, and collaboration to IRS’s current efforts and therefore help alleviate the missed opportunities that we identified in the report, such as updating outdated security standards. Further, IRS could choose a leadership mechanism that it determines to be low cost and most efficient to gain a higher degree of coordination. Without this structure, it is unclear how IRS will adapt to changing security threats in the future and ensure those threats are mitigated. In our draft report, we made a recommendation that IRS modify the Authorized e-file Provider program to be consistent with the FTC Safeguards Rule. In its response, IRS stated that it did not have explicit authority to establish policy consistent with the FTC Safeguards Rule or enforce compliance with it. However, IRS clearly states in its Revenue Procedure 2007-40 that violations of the provisions of the Gramm-Leach- Bliley Act and the implementing rules and regulations promulgated by FTC are considered violations of the revenue procedure and may subject an Authorized e-file Provider to penalties or sanctions. Therefore, we believe IRS has already incorporated compliance with the FTC Safeguards Rule as part of its Authorized e-file Provider program. The intent of this recommendation is not to suggest that IRS develop new policies related to the elements of the Safeguards Rule. Instead, we believe IRS has the opportunity to explicitly state in its requirements for Authorized e-file Providers the elements of an information security program, as listed in the Safeguards Rule. This action will help third party providers become aware of their specific legal obligations to protect taxpayer data under the Gramm-Leach-Bliley Act. As such, we clarified text in the body of the report and the text of the recommendation to better reflect our intent. For our recommendation to require all tax software providers that participate in the Authorized e-file Provider program to follow the subset of NIST Special Publication 800-53 controls that were agreed upon by the Security Summit participants, IRS stated that it does not have the statutory authority for such a requirement. However, under its existing authority, IRS has already established some information security requirements for a portion of tax software providers—those that are online providers. IRS has the opportunity to further establish standards for all tax software providers by incorporating the subset of NIST controls into its Authorized e-file Provider program, which would capitalize on the work it has completed with the Security Summit members. We continue to believe that without setting and requiring the same security standards for all tax software providers, IRS does not have assurance that these providers have an equivalent level of standards in place to adequately protect taxpayer information. For our recommendation that IRS update its monitoring programs for electronic return originators, IRS stated it does not have the statutory authority to establish policy on information security and cybersecurity issues, nor to enforce compliance if noncompliance is observed. However, as we reported, IRS already monitors physical aspects of information security, which goes beyond existing Authorized e-file Provider program requirements. Since most individuals now file tax returns electronically, having checks for physical security without comparable checks for cybersecurity does not address current risks, as cyber criminals and fraudsters are increasingly attacking third-party providers, as IRS has noted. We believe that incorporating some basic cybersecurity monitoring into the visits would provide IRS the opportunity to help inform the most vulnerable third-party providers of additional guidance and resources. For our recommendation to conduct a risk assessment to determine whether different monitoring approaches are appropriate for all of the provider types in the Authorized e-file Provider program, IRS stated that changes to the monitoring program would not have value to the overall program performance absent statutory authority. We disagree with this conclusion. As discussed in the report, IRS does not currently systematically monitor the existing security requirements for online providers, nor does it conduct information security or cybersecurity monitoring for all types of Authorized e-file Providers. We believe that IRS could conduct a risk assessment of its current monitoring program within existing statutory authority and make necessary changes that would provide better assurance that all types of providers are receiving some level of oversight and that IRS is addressing the greatest risk areas appropriately. We are sending copies to the Chairmen and Ranking Members of other Senate and House committees and subcommittees that have appropriation, authorization, and oversight responsibilities for IRS. We are also sending copies of the report to the Commissioner of Internal Revenue and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9110 or Lucasjudyj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our objectives were to (1) assess what is known about the taxpayer information security requirements for the systems used by third-party providers, (2) describe Internal Revenue Service’s (IRS) outreach efforts to third-party providers on the requirements, (3) assess IRS’s monitoring processes for ensuring third-party providers’ compliance with the requirements, and (4) assess IRS’s requirements for third-party provider security incident reporting and how IRS uses that information. To assess what is known about the taxpayer information security requirements for the systems used by third-party providers, such as paid preparers and tax software providers, we reviewed applicable laws and regulations such as the Gramm-Leach-Bliley Act and relevant portions of the Internal Revenue Code, including 26 U.S.C. § 6011. This section of the Internal Revenue Code prescribes the filing of income tax returns, as well as the electronic filing requirements for returns prepared by paid preparers. We reviewed 26 U.S.C. §7803, which provides that the IRS Commissioner has the authority to administer and manage the execution and application of tax laws, while balancing the rights of, among other things, confidentiality and privacy of the taxpayer. We also reviewed the Federal Trade Commission’s (FTC) Safeguards Rule, which requires financial institutions, including tax return preparers, affiliates, and service providers, to ensure the security and confidentiality of customer records and information. This rule applies to those who are significantly engaged in providing financial products or services that include preparation and filing of tax returns. We reviewed IRS Revenue Procedure 2007-40, which informs Authorized e-file Providers of their obligations to IRS, taxpayers, and other participants in the Authorized e-file Provider program and outlines the rules governing filing electronically with IRS. We reviewed IRS publications describing the obligations in IRS’s Revenue Procedure 2007-40 and the requirements of the Authorized e- file Provider program, including IRS Publication 3112, IRS e-file Application and Participation, and IRS Publication 1345, Handbook for Authorized IRS e-file Providers of Individual Income Tax Returns. We assessed these documents to determine if the requirements for third- party providers were incorporating the laws and following leading practices as outlined by Standards for Internal Control in the Federal Government (Internal Control Standards) and A Framework for Managing Fraud Risk in Federal Programs (Fraud Risk Framework). The Fraud Reduction and Data Analytics Act of 2015, and Office of Management and Budget guidance implementing its provisions, affirm that agencies should adhere to the leading practices identified in our Fraud Risk Framework. We also compared the standards published in Publication 1345 for online providers to the National Institute of Standards and Technology (NIST) Special Publication 800-52: Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations to determine if the standards were following leading practices. We reviewed the subset of NIST Special Publication 800-53: Security and Privacy Controls for Federal Information Systems and Organizations controls that the Security Summit members agreed to voluntarily implement. We also reviewed other IRS publications that provide third-party providers with descriptions of leading practices in keeping taxpayer information safe, including IRS Publication 4557, Safeguarding Taxpayer Data: A Guide for Your Business; IRS Publication 4600, Tips for Safeguarding Taxpayer Data; IRS Publication 5293, Protect Your Clients; Protect Yourself: Data Security Resource Guide for Tax Professionals; and IRS Publication 5294, Protect Your Clients; Protect Yourself: Data Security Tips for Tax Professionals. In assessing these documents, we identified the extent of consistency among publications. We interviewed IRS officials who were responsible for various aspects of IRS’s security requirements for paid preparers and tax software providers. We conducted semistructured interviews with the following 10 industry groups and related organizations that represented a cross section of the tax preparation industry to determine their knowledge about existing information security requirements. American Coalition for Taxpayer Rights American Institute of Certified Public Accountants Council for Electronic Revenue Communication Advancement Electronic Tax Administration Advisory Committee Federation of Tax Administrators National Association of Tax Professionals National Society of Tax Professionals We reviewed IRS organization documents, including organizational charts and associated Internal Revenue Manual (IRM) provisions for the offices that have responsibilities for securing taxpayer information. We reviewed the stated missions of the offices of Electronic Products and Services Support (EPSS); Small Business/Self-Employed; Return Integrity and Compliance Services (RICS); Criminal Investigation (CI); Return Preparer Office; Office of Professional Responsibility; Cybersecurity; and Stakeholder Liaison. We also interviewed officials from these offices to determine how they coordinated the responsibilities for overseeing the security of taxpayer data among the offices. We compared IRS activities to the Internal Control Standards that identify controls that help an entity adapt to shifting environments, evolving demands, changing risks, and new priorities. To describe the outreach efforts IRS takes for third-party providers, we reviewed IRS outreach documents such as publications, news releases, social media posts, emails, webinars, and online education campaigns. We interviewed IRS officials and conducted semistructured interviews with 10 industry groups and related organizations to determine IRS’s communication efforts related to security standard enforcement and identify potential challenges that IRS faces in its outreach. To assess IRS’s monitoring processes for ensuring third-party providers’ compliance with information security requirements, we reviewed the agency’s monitoring procedures for the Authorized e-file Provider program per Rev. Proc. 2007- 40; IRS Publication 3112, IRS e-file Application and Participation; and IRS Publication 1345, Handbook for Authorized IRS e-file Providers of Individual Income Tax Returns. We reviewed the IRM section related to Monitoring the IRS e-file Program, monitoring checklists, and related job aides to determine the extent to which monitoring practices address security requirements in IRS Publication 1345. We assessed IRS’s monitoring efforts against our Fraud Risk Framework’s principles to combat fraud in a strategic, risk- based manner. We also interviewed the IRS officials responsible for overseeing the monitoring program. To assess IRS’s requirements for third-party provider reporting of security incidents and how IRS uses that information, we reviewed IRS guidance about security incident reporting requirements. We analyzed IRS data on the number and type of security incidents tracked in the RICS Incident Management Database from 2017 and 2018, the only data available following its creation in December 2016. We interviewed RICS officials about the quality of data in this database and determined that the data were sufficiently reliable to describe a minimum count of security incidents. Specifically, we asked about the responsibilities of officials collecting and using the data, the procedures in place to capture all reported data, and controls for ensuring the accuracy of the data and resolving any errors, among other things. We reviewed IRS guidance and program user agreements to determine security incident reporting requirements for third-party providers. We reviewed IRS process documentation and interviewed IRS officials from EPSS, RICS, CI, Return Preparer Office, Cybersecurity, and Stakeholder Liaison to determine the collection, routing, and storage processes for security incident information. We assessed IRS’s processes and documentation practices against leading practices outlined in NIST Special Publication 800-53 and Internal Control Standards. We interviewed IRS officials to identify ways that IRS uses this security incident information. We conducted semistructured interviews with the 10 industry groups and related organizations listed above to determine their knowledge about existing security incident reporting requirements. We conducted this performance audit from November 2017 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Internal Revenue Service (IRS) mandated that online providers adhere to six privacy, security, and business standards as part of the Authorized e-file Provider program, as listed in table 6. These standards have not been updated since they were developed in 2010. In addition to the contact named above, Jeff Arkin (Assistant Director), Robyn Trotter (Analyst-in-Charge), Christina Bixby, Alyssia Borsella, Mark Canter, Jehan Chase, Larry Crosland, Ann Czapiewski, James Andrew Howard, Michele Fejfar, and Robert Gebhart made key contributions to this report.", "summary": "Third-party providers, such as paid tax return preparers and tax preparation software providers, greatly impact IRS’s administration of the tax system. If these third parties do not properly secure taxpayers’ personal and financial information, taxpayers will be vulnerable to identity theft refund fraud and their sensitive personal information will be at risk of unauthorized disclosure. IRS estimates that it paid out at least $110 million in identity theft tax refund fraud during 2017, and at least $1.6 billion in identity theft tax refund fraud during 2016. GAO was asked to review IRS’s efforts to track, monitor, and deter theft of taxpayer information from third parties. Among other things, this report assesses what is known about the taxpayer information security requirements for the systems used by third-party providers, IRS’s processes for monitoring compliance with these requirements, and IRS’s requirements for third-party security incident reporting. GAO analyzed IRS’s information security requirements, standards, and guidance for third-party providers and compared them to relevant laws, regulations, and leading practices, such as NIST guidance and Standards for Internal Control in the Federal Government . GAO reviewed IRS’s monitoring procedures and its requirements and processes for third-party reporting of security incidents, and compared them to Internal Control Standards and GAO’s A Framework for Managing Fraud Risk in Federal Programs . GAO also interviewed IRS and tax industry group officials. Federal law and guidance require that the Internal Revenue Service (IRS) protect the confidentiality, integrity, and availability of the sensitive financial and taxpayer information that resides on its systems. However, taxpayer information held by third-party providers—such as paid tax return preparers and tax preparation software providers—generally falls outside of these requirements, according to IRS officials. In 2018, about 90 percent of individual taxpayers had their tax returns electronically filed by paid preparers or used tax preparation software to prepare and file their own returns. IRS seeks to help safeguard electronic tax return filing for various types of third-party providers through requirements under its Authorized e-file Provider program. However, IRS’s efforts do not provide assurance that taxpayers’ information is being adequately protected. Paid Preparers. IRS has not developed minimum information security requirements for the systems used by paid preparers or Authorized e-file Providers. According to IRS’s Office of Chief Counsel, IRS does not have the explicit authority to regulate security for these systems. Instead, the Internal Revenue Code gives IRS broad authority to administer and supervise the internal revenue laws. The Department of the Treasury has previously requested additional authority to regulate the competency of all paid preparers; GAO has also suggested that Congress consider granting IRS this authority. Congress has not yet provided such authority. Neither the Department of the Treasury request nor the GAO suggestion included granting IRS authority to regulate the security of paid preparers’ systems. Having such authority would enable IRS to establish minimum requirements. Further, having explicit authority to establish security standards for Authorized e-file Providers’ systems may help IRS better ensure the protection of taxpayers’ information. Tax Software Providers. As part of a public-private partnership between IRS and the tax preparation industry, 15 tax software providers voluntarily adhere to a set of about 140 information security controls developed using guidance from the National Institute of Standards and Technology (NIST). However, these controls are not required, and these providers represent only about one-third of all tax software providers. Additionally, IRS established six security, privacy, and business standards for providers of software that allows individuals to prepare their own tax returns (as opposed to software that paid preparers use). However, IRS has not substantially updated these standards since 2010, and they are, at least in part, outdated. For example, IRS cites an outdated encryption standard that NIST recommends not using due to its many known weaknesses. A key factor contributing to missed opportunities to address third-party cybersecurity is IRS’s lack of centralized leadership. Consequently, IRS is less able to ensure that third-party providers adequately protect taxpayers’ information, which may result in identity theft refund fraud. IRS monitors compliance with its electronic tax return filing program requirements for those paid preparers who electronically file returns; however, IRS’s monitoring has a limited focus on cybersecurity issues. For example, the monitoring techniques largely focus on physical security (e.g., locked filing cabinets) rather than verifying that preparers have an information security policy consistent with NIST-recommended controls. Without effective monitoring of cybersecurity controls, IRS has limited assurance that those paid preparers’ systems have adequate controls in place to protect clients’ data. IRS recently began collecting information on high-risk security incidents, such as hackers infiltrating third-party provider systems. Reported incidents increased from 2017 to 2018, the only years for which IRS has data. However, IRS does not have a full picture of the scope of incidents because of inconsistent reporting requirements, including no reporting requirements for paid preparers. GAO suggests that Congress consider providing IRS with explicit authority to establish security requirements for paid preparers’ and Authorized e-file Providers’ systems. GAO is also making eight recommendations, including that the Commissioner of Internal Revenue Develop a governance structure or other form of centralized leadership to coordinate all aspects of IRS’s efforts to protect taxpayer information while at third-party providers. Require all tax software providers to adhere to prescribed information security controls. Regularly review and update security standards for tax software providers. Update IRS’s monitoring programs to include basic cybersecurity issues. Standardize incident reporting requirements for all types of third-party providers. IRS agreed with three recommendations, including the above recommendations to regularly review and update security standards for tax software providers, and standardize incident reporting requirements. IRS disagreed with five recommendations—including the other three listed above—generally citing the lack of clear and explicit authority it would need to establish security requirements for the information systems of paid preparers and Authorized e-file Providers. GAO believes that IRS can implement these recommendations without additional statutory authority.", "document_type": "gao"}
{"report": "The Child Care and Development Block Grant (CCDBG) Act, as amended, is the main federal law governing state child-care programs for low-income working families. The act was reauthorized in 2014, and the reauthorization included a focus on improving the overall quality of child- care services and development of participating children. In September 2016, OCC published new rules (CCDF regulations) to provide clarity to states on how to implement this law and administer the program in a way that best meets the needs of children, child-care providers, and families. The CCDBG Act and CCDF regulations allow states flexibility in developing CCDF programs and policies that best suit the needs of children and parents within that state. According to OCC, these new rules also align child-care requirements with new Head Start regulations, including certain requirements for background checks, annual monitoring, and prelicensure inspections for some CCDF providers. OCC also added regulatory requirements for state lead agencies to describe in their State Plans effective internal controls that are in place to ensure integrity and accountability including 1. processes to ensure sound fiscal management, 2. processes to identify areas of risk, 3. processes to train child-care providers and staff of the lead agency and other agencies engaged in the administration of the CCDF about program requirements and integrity, and 4. regular evaluation of internal control activities. Lead agencies are also required to describe in their State Plans the processes that are in place to identify fraud or other program violations, and to investigate and recover fraudulent payments and to impose sanctions in response to fraud. OCC is a program office within ACF that works with the states to administer the CCDF program. OCC and states each have responsibility for overseeing and protecting the integrity of the CCDF program. Each state must develop, and submit to OCC for approval, a State Plan that identifies the purposes for which CCDF funds will be spent for a 3-year grant period and designates a lead agency responsible for administering child-care programs. To administer CCDF funds, federal law and regulations require that states report their CCDF expenditures and data on the number of children served by CCDF subsidies. The current reporting structure as described by OCC and ACF officials is shown in figure 1. To request funding from the CCDF, states submit a State Plan for administering their CCDF programs to OCC. OCC provides states with a Plan Preprint, which serves as a template and includes instructions and guidance on developing the State Plans and providing information required by law and regulations. Further, OCC has used the Plan Preprint to request additional information from the states. The Plan Preprint developed for fiscal years 2019–2021 State Plans consists of eight sections and is the first to include the new CCDF regulatory requirements, added in September 2016 as required by the 2014 reauthorization. One of the new requirements is for state lead agencies to describe in their State Plans effective internal controls that are in place to ensure integrity and accountability. In addition, OCC modified the Plan Preprint for fiscal years 2019–2021 State Plans to add the instruction requesting states to report information about the results of their program-integrity and fraud- fighting activities, in addition to providing descriptions of the activities themselves. The Secretary of Health and Human Services, through OCC, has the responsibility to approve State Plans that satisfy the requirements, and review and monitor state compliance with the approved State Plan. According to OCC officials, the Program Operations Division within OCC, in partnership with the OCC regional program unit staff (regional offices), reviews the State Plans and approves those that they determine have satisfied the requirements of the CCDBG Act and CCDF regulations. The CCDF has been designated as a high-priority program, as defined by OMB, under the Improper Payments Elimination and Recovery Improvement Act of 2012 (IPERIA), meaning that it is a program susceptible to significant improper payments. Federal statutes require federal agencies to evaluate programs for improper-payment risk and, for programs susceptible to significant improper payments, to report on actions taken to reduce improper payments. CCDF regulations implement these requirements by requiring states to calculate and report estimates of their improper payments, including proposed actions to address sources of error. These reports are developed by the states on a 3-year rotational cycle, and HHS reports the aggregate results in its Agency Financial Report. The CCDF gross improper payment estimate for fiscal year 2019 is approximately $325 million, and the estimated improper payment rate is 4.53 percent. OCC oversees states’ compliance with the prescribed procedures for estimating improper-payment error rates by approving the preliminary documents, approving any changes to the case samples, conducting the Joint Case Reviews, and reviewing and approving the final State Improper Payments Report and CAP submissions. If a state reports an error rate at or above 10 percent, it must also submit a CAP, which includes detailed descriptions of specific activities planned to reach a targeted reduction in errors. It must then submit an update on its progress and a new CAP the following year if it has not completed the proposed corrective actions or if the error rate is still at or above 10 percent. The improper-payment reporting process is illustrated in figure 2. In fiscal year 2019, OCC launched a formal Monitoring System to review a selection of states annually over the course of the 3-year State Plan period. According to OCC officials, the three main purposes of the Monitoring System are to: (1) ensure compliance with the CCDBG Act, CCDF regulations, and the approved State Plans; (2) identify state technical-assistance needs; and (3) identify promising practices to inform continuous quality improvement. The Monitoring System focuses on 11 topic areas, which include program integrity and accountability. In addition, other topic areas include disaster preparedness, consumer education, and health and safety requirements. OCC officials told us that monitoring is completed on a rolling basis, and that they plan to monitor one-third of states each fiscal year, from fiscal years 2019 to 2021. According to OCC officials, they scheduled the monitoring to ensure that a state will not be submitting an improper- payment report in the same year that it participates in the monitoring. Figure 3 provides additional details regarding the OCC Monitoring System process, which includes an on-site visit to monitored states. Fraud and “fraud risk” are distinct concepts. Fraud risk exists when individuals have an opportunity to engage in fraudulent activity, have an incentive or are under pressure to commit fraud, or are able to rationalize committing fraud. Although the occurrence of fraud indicates there is a fraud risk, a fraud risk can exist even if fraud has not yet been identified or occurred. For example, suspicious billing patterns or complexities in program design may indicate a risk of fraud even though fraud has not been identified or occurred. When fraud risks can be identified and mitigated, fraud may be less likely to occur. According to federal standards and guidance, executive-branch agency managers are responsible for managing fraud risks and implementing practices for combating those risks. Specifically, federal internal control standards state that management should consider the potential for fraud when identifying, analyzing, and responding to risks. As part of these standards, management assesses risks the entity faces from both external and internal sources. In addition, in July 2015, GAO issued the Fraud Risk Framework, which provides a comprehensive set of key components and leading practices that serve as a guide for agency managers to use when developing efforts to combat fraud in a strategic, risk-based way. The Fraud Risk Framework describes leading practices in four components, as shown in figure 4. The Fraud Reduction and Data Analytics Act of 2015, enacted in June 2016, required OMB to establish guidelines for federal agencies to create controls to identify and assess fraud risks, and design and implement antifraud control activities. The act further required OMB to incorporate the leading practices from the Fraud Risk Framework in the guidelines. In July 2016, OMB published guidance about enterprise risk management and internal controls in federal executive departments and agencies. Among other things, this guidance affirms that managers should adhere to the leading practices identified in the Fraud Risk Framework. As part of its oversight of states’ CCDF programs, OCC reviewed and approved State Plans for the current grant period (fiscal years 2019– 2021). However, OCC has not established written policies to guide staff review and approval of these State Plans, a process that occurs every 3 years. OCC’s lack of established policies limits its ability to ensure that staff follow appropriate protocols for consistency when reviewing and approving State Plans and to retain organizational knowledge in the event of staff turnover, which OCC noted as occurring during each review period. Further, OCC requested that states report information about the results of states’ program-integrity activities. However, most of the State Plans that it approved did not provide the results of states’ program- integrity activities as requested. OCC officials told us that they plan to continue to request that states report on the results of their program- integrity activities, but OCC has not identified what it considers to be “results” of program-integrity activities. Without taking additional steps to define its informational needs and encourage states to report the results of their program-integrity activities, OCC will not have this information to help determine whether states are effectively ensuring the integrity of the CCDF program. To provide oversight of states’ CCDF program-integrity activities, OCC reviewed and approved State Plans for the current grant period, covering fiscal years 2019–2021. To do so, OCC officials described to us a process that began with a high-level review of the draft State Plans submitted through an electronic system. After an initial review for completeness, OCC staff focused on the contents of the State Plans including states’ responsiveness to each requirement. For example, one requirement is to describe the processes that the state will use to identify risk in its CCDF program. OCC officials also stated that they consider clarity, consistency, and compliance when assessing State Plans. OCC officials also explained that they reviewed the responses to determine whether they were sufficiently detailed, and sought clarification from the states when necessary. OCC officials stated that, prior to the final approval of the State Plans, staff completed a validation form that consists of a table listing the State Plan subsections with checkboxes next to each subsection. Figure 5 outlines the timeline for review and approval of State Plans. OCC has developed a draft procedure for the State Plan review and approval process, but had neither finalized written policies before beginning its review of the fiscal years 2019–2021 State Plans, nor finalized written policies for future review periods that occur every 3 years. Instead, OCC officials told us that for the review and approval process completed in 2018, they provided their staff a variety of training materials and draft documents that encouraged discussion among those involved. These documents contained information and guidance on the process, such as explaining the overall operational processes for reviewing and approving State Plans and general roles and responsibilities. However, none of the documents were finalized as OCC’s written policies for staff to follow when implementing the fiscal years 2019–2021 State Plan review and approval process, or for subsequent review periods. In response to our request for finalized policies pertaining to how OCC reviewed and approved State Plans, OCC provided documents that have substantial limitations for explaining to OCC staff how they should review and approve State Plans. For example, OCC provided what it characterized as a three-page summary protocol, which, in part, contained a historical record of what occurred during the recently completed review period rather than guidance that would help OCC achieve its State Plan review objectives on a continuous basis. Specifically, the protocol describes the regular internal meetings and interactions that OCC staff had from September 2018 to December 2018. As such, the protocol does not describe the process that OCC staff should follow, or the meetings that should occur, when reviewing and approving State Plans in future years (i.e., on a continuous basis). OCC also developed in August 2018 a more-detailed draft procedure for reviewing and approving State Plans. The draft procedure contains information on the communication process between the central and regional offices, recognizes that there may be variation in internal processes among regional offices and from one review period to the next, and includes guidance on steps for resolving questions about State Plans, among other guidance. Unlike the three-page summary protocol, the draft procedure explicitly states its applicability to future review periods as well as the current State Plan review period, and therefore would have provided guidance for staff on a continuous basis had a finalized version been shared with staff and established as OCC’s written policies. However, because of the volume of work and differences in caseloads among regional offices, OCC officials stated that they did not share a finalized procedure with staff and that staff were neither expected nor required to use the draft procedure when conducting their review of State Plans for the fiscal years 2019–2021 review period. As such, this draft procedure did not represent the formal policies for staff to follow in performing their roles. In explaining why it relies on the three-page summary protocol and draft procedure rather than finalized written policies to guide its State Plan review and approval process, OCC officials stated that OCC needs flexibility in its policies during the review period. Specifically, there are staffing changes in both the central and regional offices for each State Plan review period, and having flexibility within the framework provided by the three-page summary protocol allows them to accommodate those changes. OCC officials noted that some of the processes are unique to each of the 10 regional offices because of differences in their structure, staffing, and caseloads. Likewise, OCC officials stated that the regional offices need flexibility to continuously adjust processes and timelines so that they can accommodate varying responsiveness from states, and evaluate the State Plans without undermining the flexibility afforded to states through the block grant. However, it is possible for OCC to establish written policies to guide processes that are common from one review period to the next, and across all regions, while still maintaining the necessary flexibility to accommodate staffing changes and regional differences, as it had already begun to do by developing its August 2018 draft procedure. In this regard, Standards for Internal Control in the Federal Government states that management should implement control activities through policies. In doing so, management communicates the policies to personnel so that personnel can implement the control activities for their assigned responsibilities. Further, Standards for Internal Control in the Federal Government includes minimum documentation requirements, such as that management develop and maintain documentation of its internal control system. An internal control system is a continuous built-in component of operations that provides reasonable assurance that an entity’s objectives will be achieved. Internal control is not one event, but a series of actions that occur throughout an entity’s operations. Further, internal control is recognized as an integral part of the operational processes management uses to guide its operations, and internal control is built into the entity as a part of the organizational structure to help managers achieve the entity’s objectives on an ongoing basis. As such, documentation of the internal control system should reflect a continuous, built-in component of operations rather than a historical record of a past event. Documentation also provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel. OCC’s lack of established written policies limits its ability to ensure that staff follow appropriate protocols on a continuous basis when implementing the State Plan review and approval process, and limits its ability to provide a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel. Without finalizing written policies, an effort that could include leveraging its previously developed August 2018 draft procedure, OCC risks losing that knowledge each time there are staffing changes among central and regional offices. In response to a 2016 HHS OIG report, OCC has attempted to collect information about the results of states’ program-integrity and fraud- fighting activities by adding a new instruction to the fiscal years 2019– 2021 Plan Preprint requesting states to report such information in their State Plans. Specifically, the HHS OIG recommended that collecting data on program-integrity and fraud-fighting results would be an important step in monitoring states’ efforts to safeguard the CCDF program. Additionally, OCC officials told us that obtaining information on the results of program-integrity activities is important for understanding national trends and helping to inform OCC’s technical assistance to states and ensure states’ accountability over their program-integrity activities. However, our review of 51 approved State Plans found that 43 State Plans (about 84 percent) did not report the results of program-integrity activities as requested (see fig. 6). The other eight states (about 16 percent) reported the results of program-integrity activities. State Plans must meet the requirements set forth in the law and the CCDF regulations to be approved. OCC officials told us that the State Plans were approved without the information on the results of program- integrity activities because, although there are instructions in the Plan Preprint for states to report this information, the CCDF regulations do not require it. Further, OCC officials told us that when OCC submitted the Plan Preprint to OMB for approval under the Paperwork Reduction Act, OCC had indicated that the program-integrity results would be collected on an informational basis, and states would not be required to provide this information. According to an OCC official, only portions of the Plan Preprint with instructions for states to report on the results of program- integrity activities were requested on an informational basis, and all other information in that section was required for approval of the State Plans. OCC officials also told us that OCC will continue to request that states report on the results of their program-integrity activities in the State Plans, but OCC has not defined what information it needs regarding the “results” of states’ program-integrity activities and has not communicated the need to states or its staff. OCC officials told us that they will ensure that states submit this information by providing guidance to states on the purpose of collecting this information. However, OCC was not able to provide us with a definition or examples of what it considers to be “results” of program- integrity activities that would be helpful for ensuring states’ accountability over their program-integrity activities. In addition, OCC officials said that OCC did not communicate to states that the information about the results of program-integrity activities was being requested on an informational basis only. According to OCC officials, OCC did not specifically communicate its intention to states because it wanted states to provide a response, if possible. Similarly, OCC had not developed any specific internal criteria for its staff to use when reviewing State Plans to determine whether certain responses were sufficient for their informational needs, such as to better understand national trends. OCC officials also stated that there was no internal written guidance explaining to OCC staff that such information was not required for State Plan approval. Rather, this standard was communicated to staff during weekly meetings. Standards for Internal Control in the Federal Government states that management should use quality information to achieve the entity’s objectives. In doing so, management identifies the information requirements needed and defines the information requirements at the relevant level and requisite specificity for appropriate personnel. Further, Standards for Internal Control in the Federal Government states that management should internally and externally communicate the necessary quality information to achieve the entity’s objectives. In this context, after defining its informational needs regarding the results of program-integrity activities, OCC’s internal and external communication could include communication to the states, which are requested to include this information in the State Plans, and to its staff who will be responsible for analyzing this information. Until OCC defines what information it needs regarding program-integrity activity results, it will be limited in its ability to obtain quality information. By not communicating informational needs to states and staff, OCC will continue to lack quality information about the results of states’ program-integrity efforts and will not be able to use that information to analyze national trends and help ensure states’ accountability over their program-integrity activities, as described. Since 2013, seven states with improper-payment rates at 10 percent or above have submitted 14 corrective action plans (CAP) to OCC for review. However, OCC does not have any documented criteria to guide the review of the CAPs submitted by states to ensure the proposed actions are aimed at root causes of improper payments and are effectively implemented. OCC also has not documented the procedures it uses to follow up with states subject to CAPs, but said it is planning to. Federal improper-payment statutes require federal agencies to review programs susceptible to significant improper-payment risks and develop actions to reduce improper payments. For example, the Improper Payments Elimination and Recovery Act of 2010 (IPERA) specifically requires agencies administering programs that are susceptible to significant improper payments, such as the CCDF, to report on actions the agency is taking to reduce improper payments. Because the CCDF is administered by states, this requirement is implemented in CCDF regulations by requiring states reporting improper-payment error rates at or above 10 percent to develop and implement CAPs. The OMB guidance implementing IPERA states that agencies should ensure that each corrective action is specifically aimed at a root cause of improper payments and that the actions are effectively implemented to prevent and reduce improper payments. According to this guidance, a root cause is something that would directly lead to an improper payment and, if corrected, would prevent the improper payment. In the proposed rulemaking in which OCC introduced the CAPs, OCC stated that the CAPs are intended to be comprehensive and detailed, so as to improve upon the descriptions of corrective actions already reported on a 3-year cycle, which sometimes lack detail or specificity. OCC officials told us that OCC reviewers use their CAP Review Tool to evaluate the CAPs for approval, which also lays out the protocol for conducting reviews. However, the CAP Review Tool does not require reviewers to document whether the corrective actions proposed by states are aimed at root causes of improper payments, or effectively implemented. Further, the written review procedure that accompanies the CAP Review Tool does not contain guidance for reviewers on evaluating whether corrective actions are aimed at root causes and are effectively implemented. OCC officials explained to us that, in their view, states are in the best position to identify the most-feasible approach to corrective actions based on their individual circumstances. We acknowledge that states should have flexibility to identify corrective actions based on their individual circumstances. However, according to OMB guidance, it is federal agencies that are to ensure that corrective actions are aimed at root causes of improper payments and effectively implemented. Further, in the proposed rulemaking in which OCC introduced the CAPs, OCC stated that it intended the CAPs to be used for OCC to hold states accountable as part of its compliance with IPERA. Accordingly, without providing additional guidance to its reviewers, OCC will lack assurance that states’ proposed corrective actions are aimed at root causes and effectively implemented. OCC officials also stated that the majority of the seven states subject to CAPs reduced their error rates over time, specifically to below 10 percent. OCC officials explained that this determination is based on the submission of the State Improper Payment Report for the next required reporting cycle or on states’ voluntarily conducting a review of a sample of cases and submitting the results to OCC to demonstrate they had reduced their error rate to below 10 percent. We did not independently corroborate OCC’s determination because assessing the reliability of the self-attested internal error-rate reviews conducted by certain states and reviewing this information was outside the scope of our work. However, as part of our review of the 14 CAPs that have been submitted to OCC in response to OCC’s improper-payment reviews since 2013, we found that one state was required to submit CAPs for 3 consecutive years and consistently proposed the same error-rate reduction targets, with different dates. This observation underscores the need to ensure the corrective actions a state proposes are specifically aimed at root causes of improper payments and are effectively implemented. OCC does not have guidance in place for its reviewers to determine whether the ongoing corrective actions a state proposes to reduce improper payments will be specifically aimed at root causes of improper payments and effectively implemented. This could leave the CCDF program at continued risk of improper payments. OCC does not have written policies for its CAP follow-up process or documentation that follow-up has been completed for past CAPs. OCC officials told us that they plan to develop such written policies, but officials did not specify a timeline for completion. OCC officials described their process used to monitor states while they are subject to a CAP, which includes additional contact when the same state has been subject to CAPs for consecutive years. This CAP follow-up process is illustrated in figure 7. According to OCC officials, OCC intends to develop written policies for the CAP follow-up process but did not provide a time frame for completion. This will include, at a minimum, a written protocol for the activities illustrated above, which will be included in the next revision of the instructions given to states for improper-payment reporting. According to OCC officials, each region currently has its own process for documenting discussions with CAP states. Having established written policies for the CAP follow-up process will help ensure that OCC’s oversight and monitoring of CAPs is carried out consistently. OCC officials told us that their Monitoring System, initiated in fiscal year 2019, plays a part in OCC’s role to ensure that states’ program-integrity activities are effective. According to OCC officials, OCC uses two tools as part of its Monitoring System—a Compliance Demonstration Packet and Data Collection Tool. States complete the Compliance Demonstration Packet to outline how they propose to demonstrate compliance with regulatory requirements and implementation of the approved State Plans throughout the Monitoring System’s phases. For example, to show effective internal controls are in place to ensure integrity and accountability, states may provide OCC with state or local policies and manuals (previsit phase), and may submit to interviews or provide system demonstrations (on-site visit phase). OCC staff use the Data Collection Tool to record comments about the evidence observed, and to note whether additional follow-up is needed. Both of these tools contain language indicating that the effectiveness of states’ program-integrity and fraud-fighting activities are evaluated by OCC staff. For purposes of the Monitoring System, OCC officials said that states have broad flexibility to propose, in the Compliance Demonstration Packet, what documents and evidence to provide. In addition, states have the flexibility to propose how the state will demonstrate compliance with regulatory requirements. This includes the requirement to describe in its State Plan effective program-integrity control activities, which includes fraud-fighting activities. OCC officials further told us that OCC does not collect the same set of information or evidence across the country. Rather, OCC collects state-specific information based on what each individual state proposes. For example, the Compliance Demonstration Packet allows states to propose an approach for demonstrating their compliance with the requirement to describe in their State Plans effective internal controls that are in place to ensure integrity and accountability. OCC officials said the primary purpose of the Monitoring System is to ensure that states are in compliance with CCDF regulations and implementing the State Plans as approved, rather than to make an assessment of the efficacy of the State Plans. When we asked OCC officials how they determine whether a state has provided appropriate and adequate documentation for the purposes of the Monitoring System, these officials told us that staff develop specific questions for each state and look for evidence showing that states are implementing the State Plans as approved. For example, OCC officials might look for evidence of a state’s implementation of certain program-integrity activities described in its approved State Plan to verify that the activities described are in place. OCC officials also stated that staff decide what is acceptable through consensus and attempt to build consistency through internal discussions regarding the appropriateness of the material that states provide. However, there are no specific criteria to guide OCC staff’s assessment of the effectiveness of states’ program-integrity activities during these discussions. For example, there are no specific criteria to help OCC staff assess whether states’ implemented control activities are effective at identifying areas of risk. OCC officials stated that the CCDF regulations and the approved State Plans are the most-detailed criteria that they use to assess data collected for the Monitoring System. However, neither the CCDF regulations nor the State Plans include specific criteria for assessing whether the control activities are effective. OCC is responsible for monitoring states’ compliance with the CCDF regulations, and these regulations explicitly require that states describe in their State Plans “effective internal controls that are in place to ensure integrity and accountability.” According to Standards for Internal Control in the Federal Government, an effective internal control system has a monitoring component that is effectively designed, implemented, and operating. Additionally, a leading practice of the Fraud Risk Framework is to examine the suitability of existing fraud controls. Managers who effectively implement an antifraud strategy monitor and evaluate the effectiveness of preventive activities in this strategy and take steps to help ensure external parties with responsibility over fraud control activities effectively implement those activities. Without developing and using criteria to assess whether states’ program-integrity control activities are effective, OCC cannot ensure that states’ internal controls for program integrity are effective. Likewise, without examining the suitability of, and monitoring the effectiveness of, the states’ fraud control activities, OCC will be challenged in effectively implementing an antifraud strategy to minimize the risk of fraud in the CCDF program. OCC developed the Grantee Internal Controls Self-Assessment Instrument (Self-Assessment Instrument) in 2010 and makes the technical-assistance tool available to the states through its website. In response to a 2016 HHS OIG report, ACF officials said that OCC would use the Self-Assessment Instrument to address the report’s recommendations to request that states examine the effectiveness of their program-integrity and fraud-fighting activities, and examine with states the benefits of expanding such activities. The Self-Assessment Instrument contains five sections: (1) Eligibility Determination and Review; (2) Improper Payment Case Review Process; (3) Fraud and Overpayment Prevention, Detection, and Recovery; (4) Federal Reporting; and (5) Audits and Monitoring. According to OCC officials, as of August 2019, 19 states have completed the Self-Assessment Instrument since its inception. OCC officials stated that use of the Self-Assessment Instrument is based entirely on states’ self-identified risks, and states are free to choose which, if any, of the sections to complete. OCC officials have noted benefits as a result of states completing the Self-Assessment Instrument. Specifically, OCC officials said that states have improved their implementation processes and policies, and improper-payment error rates have decreased. In addition to making the tool available to states, OCC officials told us that OCC also provides technical assistance in completing the Self-Assessment Instrument, which may include an on-site facilitated discussion. The facilitated discussion may cover areas including control activities to identify and prevent fraud, and strategies to investigate and collect improper payments. Following the on-site facilitated discussion, an OCC contractor compiles a report summarizing state-identified issues to address in states’ policies and procedures, according to one OCC official. However, OCC officials told us that states are not required to act on this report. In addition to the Self-Assessment Instrument, OCC has recently coordinated on the development of the Fraud Toolkit, which is a series of electronic spreadsheets that states can use to respond to questions about their fraud risk management activities—such as staff training, procedures for addressing suspected fraud, and program administration. The tools assign risk levels to these areas based on the state’s responses, and will also include recommended next steps for each of those areas and generate a report to summarize overall risk. For example, data from these tools would indicate whether states’ CCDF program staff are trained to identify forms, such as wage stubs or employer letters that may have been forged or altered. The data would also indicate whether the state has a fraud referral process in place to expedite investigations. OCC makes the Fraud Toolkit available for states to use upon request. However, other than making the tool available, OCC officials said that OCC does not usually have any further involvement in states’ use of the tool. OCC officials told us that they do not plan to use either the Self- Assessment Instrument or the Fraud Toolkit to collect data about states’ CCDF programs because both the Self-Assessment Instrument and the Fraud Toolkit are intended as primarily technical-assistance tools rather than monitoring tools or data-collection instruments. OCC officials also told us that, to formally collect information from states’ use of such tools, they would need to seek approval from OMB. OCC officials stated that OCC’s goal is to develop technical assistance that best meets the needs of the states, and not to impose additional reporting requirements on the states. Officials also noted a concern that states could cease to participate in or accept technical assistance if such assistance is seen as increasing reporting requirements. However, according to OCC officials, OCC has not conducted a cost-benefit analysis of collecting such information. Leading practices in the Fraud Risk Framework are to monitor and evaluate the effectiveness of preventive activities; collect and analyze data; and adapt activities to improve fraud risk management. Further, although external parties—in this case, the state lead agencies—may be responsible for specific fraud control activities, Standards for Internal Control in the Federal Government states that management should establish and operate monitoring activities to monitor the internal control system and evaluate the results. As part of these standards, management retains responsibility for monitoring the effectiveness of internal control over the assigned processes performed by external parties. Management is responsible for meeting internal control objectives, and may decide how the entity evaluates the costs versus benefits of various approaches to implementing an effective internal control system. However, cost alone is not an acceptable reason to avoid implementing internal controls, and cost-benefit considerations support management’s ability to effectively design, implement, and operate an internal control system that balances the allocation of resources and other factors relevant to achieving the entity’s objectives. By not evaluating the feasibility of collecting information from the Self-Assessment Instrument or the Fraud Toolkit— such as evaluating the feasibility of doing so during its Monitoring System process—OCC may be missing an opportunity to monitor the effectiveness of the internal control system to help states adapt control activities to improve fraud risk management. As described above, OCC has developed several program-integrity activities that could help assess and manage fraud risk if they were part of an antifraud strategy. For example, the improper-payment reporting process and Monitoring System are not specific to fraud but may generate information relevant to fraud risks. However, according to OCC officials, ACF has not completed a fraud risk assessment for the CCDF, which would provide a basis for the development of an antifraud strategy that describes the program’s approach for addressing prioritized fraud risks identified, as described in the Fraud Risk Framework. The Assess component of the Fraud Risk Framework calls for federal managers to plan regular fraud risk assessments and to assess risks to determine a fraud risk profile. Furthermore, Standards for Internal Control in the Federal Government states that management should consider the potential for fraud when identifying, analyzing, and responding to risks. Leading practices for planning fraud risk assessments include tailoring the fraud risk assessment to the program and planning to conduct the assessment at regular intervals and when there are changes to the program or operating environment. The leading practices also include identifying the tools, methods, and sources for gathering information about fraud risks and involving relevant stakeholders in the assessment process. The Fraud Risk Framework also identifies leading practices for conducting fraud risk assessments and documenting the program’s fraud risk profile, as illustrated in figure 8. As discussed in the Fraud Risk Framework, the fraud risk profile provides a basis for managers to develop and document an antifraud strategy that describes the program’s approach for addressing prioritized fraud risks identified. According to ACF, there is currently a process in place at the ACF level that will lead to the development of a Fraud Risk Assessment. Specifically, ACF is in the process of developing a Fraud Risk Assessment template, which will include a program fraud risk profile. The CCDF will be part of the pilot program for this effort. The Fraud Risk Assessment template will consider the Fraud Risk Framework as well as guidance contained in OMB Circular A-123, Management’s Responsibility for Enterprise Risk Management and Internal Control, according to OCC officials. These officials also stated that ACF will leverage its previously developed and implemented risk assessments, including the Program Risk Assessment that was completed for the CCDF between fiscal years 2011 and 2016 as part of the HHS Program Integrity Initiative. However, according to ACF, the development of a Fraud Risk Assessment template is currently on hold due to competing priorities. The ACF stated the agency expects to resume the process by December 2019, and OCC expects that the draft template will be completed by the end of the first quarter of fiscal year 2020. Because the CCDF is serving as the pilot for the new template, OCC expects that the initial assessment of the program will be complete by the end of the third quarter of fiscal year 2020. Until ACF finalizes its template and conducts a risk assessment for the CCDF, ACF will not be able to develop a fraud risk profile for the CCDF. The fraud risk profile is an essential piece of the antifraud strategy and informs the specific control activities managers design and implement. Although there is currently a process in place for ACF to develop a fraud risk assessment template, until ACF carries out the assessment of the CCDF and develops an associated fraud risk strategy, it will lack assurance that OCC’s program-integrity activities are suitable and targeted at prioritized fraud risks. Both state lead agencies and OCC play an important role in overseeing and protecting the integrity of the CCDF program. However, OCC has not finalized written policies that describes how staff should implement or document the State Plan review and approval process, which is an important part of OCC’s oversight of the CCDF program. OCC’s lack of established written policies limits its ability to ensure that staff follow appropriate protocols when implementing the State Plan review and approval process, and limits its ability to retain organizational knowledge in the event of staff turnover, which OCC noted as occurring during each review period. In addition, most of the State Plans submitted to OCC for the fiscal years 2019–2021 grant period did not contain information on the results of their states’ program-integrity activities. OCC also has not defined or communicated what it considers to be the “results” of program- integrity activities for states, which are requested to include this information in State Plans, or for its staff who will be responsible for analyzing this information. Until OCC defines its informational needs regarding program-integrity activity results and communicates this information to the states and its own staff, OCC may continue to lack quality information to help ensure states’ accountability of their program- integrity activities. Further, OCC does not have documented criteria to guide the review of the CAPs to ensure the proposed corrective actions are aimed at root causes of improper payments and are effectively implemented to prevent and reduce improper payments. Without criteria for its staff to use in reviewing the CAPs, OCC does not have assurance that the corrective actions a state proposes to reduce improper payments will be specifically aimed at root causes of improper payments and effectively implemented, leaving the CCDF program at continued risk of improper payments. OCC also does not have written policies for its CAP follow-up process or documentation that follow-up has been completed for past CAPs. In addition, OCC officials told us that they plan to develop a written protocol for this process, but did not specify a timeline for completion. Having established written policies for the CAP follow-up process will help ensure that OCC’s oversight and monitoring of CAPs is carried out consistently. OCC’s Monitoring System process does not currently contain criteria to assess the effectiveness of states’ program-integrity control activities, including fraud-fighting activities. Without developing and documenting criteria to assess whether states’ program-integrity control activities are effective, OCC cannot ensure that such program-integrity control activities are effective. In addition, OCC does not plan to collect any data from its technical-assistance tools that could potentially help it to monitor and evaluate the effectiveness of states’ program-integrity activities. However, OCC has not evaluated the benefits of using these tools to collect information on program-integrity activities against any costs of doing so— such as the cost of seeking OMB approval to do so. By not evaluating the feasibility of collecting information from technical-assistance tools to monitor the effectiveness of states’ program-integrity control activities, OCC may be missing an opportunity to help states adapt control activities to improve their fraud risk management. All of the foregoing program-integrity oversight and monitoring activities could contribute to a strategy for managing fraud risks in the CCDF. However, OCC has not completed a fraud risk assessment or risk profile for the program. Although there is currently a process in place for ACF to develop a fraud risk assessment template, until ACF completes this template and carries out the assessment of the CCDF, it will lack a robust antifraud strategy and assurance that OCC’s current program-integrity activities are suitable and targeted at prioritized risk. We are making the following nine recommendations, eight to the Director of OCC and one to the Assistant Secretary for ACF: The Director of OCC should establish internal written policies to effectively implement and document the State Plan review and approval process for future review and approval periods. (Recommendation 1) The Director of OCC should define the informational needs related to the results of program-integrity activities. (Recommendation 2) The Director of OCC should communicate externally to the states its informational needs related to the results of states’ program-integrity activities. (Recommendation 3) The Director of OCC should communicate internally to staff its informational needs related to the results of states’ program-integrity activities. (Recommendation 4) The Director of OCC should develop documented criteria to guide the review of CAPs submitted by states to ensure that proposed corrective actions are aimed at root causes of improper payments and are effectively implemented. (Recommendation 5) The Director of OCC should timely complete its effort to develop established written policies for the CAP follow-up process to ensure that OCC’s oversight and monitoring of CAPs is carried out consistently. (Recommendation 6) The Director of OCC should develop and document criteria to assess the effectiveness of states’ program-integrity control activities. (Recommendation 7) The Director of OCC should evaluate the feasibility of collecting information from the Grantee Internal Controls Self-Assessment Instrument (Self-Assessment Instrument) and Fraud Toolkit, such as during its Monitoring System process, to monitor the effectiveness of states’ program-integrity control activities. (Recommendation 8) The Assistant Secretary for ACF should ensure that ACF conducts a fraud risk assessment to provide a basis for the documentation and development of an antifraud strategy that describes the CCDF program’s approach to address prioritizing fraud risks identified. (Recommendation 9) We provided a draft of this report to HHS for review and comment. In its comments, reproduced in appendix I, HHS concurred with our recommendations. HHS also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services and appropriate congressional committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-6722 or bagdoyans@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Jonathon Oldmixon (Assistant Director), Erica Varner (Analyst in Charge), Yue Pui Chin, and Daniel Dye made key contributions to this report. Other contributors include James Ashley, Maria McMullen, George Ogilvie, and Sabrina Streagle.", "summary": "The CCDF is administered as a block grant to the states by OCC, an agency within the Department of Health and Human Services (HHS). Recent reports by the HHS Office of the Inspector General show that OCC's monitoring of CCDF state program-integrity efforts remains a challenge. CCDF has also been designated as a program susceptible to significant improper payments, as defined by the Office of Management and Budget. GAO was asked to review CCDF program-integrity efforts. This report discusses, among other things, the extent to which OCC provides oversight of (1) states' CCDF program-integrity activities, including encouraging that all requested information is included within State Plans; and (2) improper-payment risks and relevant corrective actions in states' CCDF programs. GAO analyzed 51 approved CCDF State Plans, including from the District of Columbia, for the fiscal years 2019–2021 grant period. GAO also reviewed OCC policies and procedures and compared them to relevant laws, regulations, and Standards for Internal Control in the Federal Government , and interviewed relevant federal officials. The Child Care and Development Fund (CCDF) provided states more than $8 billion in federal funds in fiscal year 2019. The Office of Child Care (OCC) oversees the integrity of the CCDF, which subsidizes child care for low-income families. A key part of OCC's oversight includes reviewing and approving State Plans. OCC requested but did not require states to describe in their State Plans the results of their program-integrity activities, which describe the processes that states use to identify fraud risk. Further, OCC has not defined or communicated what information it considers to be the “results” of program-integrity activities to the states and its own staff. Without defining and communicating its informational needs, OCC may continue to lack quality information that could help ensure states' accountability over their program-integrity activities. OCC oversees states' improper payment risks through a process that includes a requirement for states to submit corrective action plans (CAP) when they estimate their annual payment error rates are at or above 10 percent. Since 2013, seven states have submitted 14 CAPs. These CAPs describe states' proposed actions for reducing improper payments. However, OCC does not have documented criteria to guide its review and approval of the CAPs to ensure the proposed corrective actions are aimed at root causes of improper payments and are effectively implemented. Without developing this guidance, OCC does not have assurance that proposed corrective actions are specifically aimed at root causes of improper payments, leaving the CCDF program at continued risk of improper payments. GAO is making nine recommendations, including that OCC define and communicate its informational needs on the results of states' program-integrity activities, and that OCC develop criteria to guide the review of CAPs to ensure that proposed corrective actions are aimed at root causes of improper payments and are effectively implemented. HHS concurred with our recommendations and provided technical comments, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "According to the Senate Committee on Homeland Security and Governmental Affairs report concerning PMIAA, the purpose of PMIAA is to improve program and project management in certain larger federal agencies. The act includes requirements for OMB, OPM, and the 24 agencies listed in the CFO Act. PMIAA requires OMB’s Deputy Director for Management or the designee to, among other things: adopt government-wide standards, policies, and guidelines for program and project management for executive agencies; engage with the private sector to identify best practices in program and project management that would improve federal program and project management; conduct portfolio reviews of agency programs not less than annually, to assess the quality and effectiveness of program management, in coordination with Program Management Improvement Officers (PMIO); establish a 5-year strategic plan for program and project conduct portfolio reviews of programs on our High-Risk List. The two types of portfolio reviews required by PMIAA—the portfolio reviews of agency programs and the portfolio reviews of programs identified as high risk on our High-Risk List—are separate requirements. For purposes of this report, we define programs, projects, and portfolios consistent with how those terms are defined in OMB’s PMIAA strategic plan. OMB defines program as the functions or activities which agencies are authorized and funded by statute to administer and enforce. Programs typically involve broad objectives. OMB views projects as temporary efforts with defined scopes to create products or services to improve the efficient and effective implementation of programs. Because programs are comprised of projects, programs inherently address the projects subsumed within them. Consequently, our discussions of programs throughout this report also pertain to projects. Finally, OMB defines portfolios as organized groupings of programs whose coordination in implementation enables agencies to achieve their objectives. The act also established the Program Management Policy Council (PMPC), an interagency forum for improving agency practices related to program management. OMB’s Deputy Director for Management chairs the PMPC. The PMPC responsibilities include advising OMB on the development and applicability of government-wide standards for program management transparency. Furthermore, the act requires PMPC members “to discuss topics of importance to the workforce,” such as workforce development needs and major challenges across agencies in managing programs. As chair of the PMPC, OMB’s Deputy Director is required to preside at meetings, determine agendas, direct the work, and establish and direct its subgroups, as appropriate. The act requires the PMPC to meet not less than twice per fiscal year. Additionally, OPM’s Director, in consultation with OMB’s Director, is required to issue regulations that: identify key skills and competencies needed for a program and a project manager in an agency; establish a new job series, or update and improve an existing job series, for program and project management within agencies; and establish a new career path for program and project managers within an agency. Overall, OPM’s role in implementing PMIAA is to establish a new job series or update an existing job series by providing the occupational standards that agencies will need to develop a trained and competent workforce with the program and project management experience, knowledge, and expertise to solve management challenges and support agency decision-making. The act requires OPM to establish new—or revise existing—occupational standards in consultation with OMB. Occupational standards are included within OPM’s classification guidance, which is provided to agencies to assist in classifying positions. This guidance helps agencies to determine the proper occupational series, position title, and grade of each position. The act requires OMB’s Deputy Director of Management to oversee implementation of the standards, policies, and guidelines for executive agencies. OMB implemented some PMIAA requirements using existing processes put in place to implement GPRAMA. We previously reported that GPRAMA provides important tools that can help decision makers address challenges facing the federal government, such as the annual reviews of progress on agency strategic objectives conducted during strategic reviews and the implementation of federal government priority goals. Federal government priority goals, also known as cross-agency priority (CAP) goals, are written by OMB in partnership with agencies. GPRAMA requires OMB to coordinate with agencies to develop CAP goals, which are 4-year outcome-oriented goals covering a number of complex or high-risk management and mission issues. For example, OMB directed agencies to align their noninformation technology major acquisition programs with relevant strategic objectives so they could assess progress for the PMIAA required program portfolio reviews concurrent with required GPRAMA strategic reviews. GPRAMA also requires OMB to present a program inventory of all federal programs by making information available about each federal program on a website. Finally, GPRAMA required OMB to establish a number of CAP goals intended to cover areas where increased cross-agency collaboration is needed to improve progress towards shared, complex policy or management objectives across the federal government. OMB uses CAP goals to address issues outlined in the President’s Management Agenda. For example, OMB wrote a CAP goal to improve management of major acquisitions across the government which complements PMIAA and its required activities. PMIAA requires the OMB Deputy Director, as chair of the PMPC, to conduct portfolio reviews of programs from our High-Risk List. The PMPC is also required to review programs we identify as high risk and to make recommendations for actions to be taken by the Deputy Director for Management of OMB or a designee. See figure 1 below for an overview of roles and responsibilities of OMB, OPM, the PMPC, and agencies. Agencies responsible for PMIAA implementation have taken steps to complete some requirements, but actions remain to fully implement the law (see Table 1). OMB met the PMIAA requirement “to establish a five-year strategic plan for program and project management.” The plan OMB developed details three key strategies to implement PMIAA: (1) coordinated governance, (2) regular OMB and agency engagement reviews, and (3) strengthening program management capacity to build a capable program management workforce. The three strategies focus on areas such as clarifying key roles and responsibilities, identifying principles-based standards, and identifying plans for enhancing workforce capabilities. The plan describes the roles and functions of the PMIOs, the PMPC, and the requirements of the agency implementation plans. It outlines a phased approach for implementing PMIAA actions with milestones occurring throughout the 5- year period. We found that OMB followed several strategic planning leading practices in the creation of the PMIAA strategic plan. First, the plan incorporates general goals and objectives for agencies’ implementation of PMIAA with three corresponding strategies explaining OMB’s overall approach. OMB followed a second leading practice by gathering input from stakeholders. OMB staff told us they solicited input from congressional staff, and members of external organizations like the Federal Program and Project Management Community of Practice (FedPM CoP). Agencies’ staff also confirmed to us that they had input into the OMB plan. Third, OMB demonstrated interagency collaboration in its efforts to establish and lead the PMPC and its efforts to work with the FedPM CoP to address any issues identified by agencies. Finally, the plan included a timeline with quarterly milestones to track completion of PMIAA’s activities and to gauge progress toward achieving the desired results of PMIAA. PMIAA required OMB to establish standards and policies for executive agencies consistent with widely accepted standards for program and project management planning and delivery. A consistent set of government-wide program management standards and policies is important because it helps ensure that agencies utilize key program management practices to improve the outcomes of government programs. OMB published in June 2018 a set of standards for program and project management as part of OMB’s PMIAA strategic plan. OMB’s strategic plan directed agencies to apply these 15 standards to internal management processes for planning, implementing, and reviewing the performance of programs and activities. OMB staff told us they decided to develop this set of standards rather than adopt an existing set of consensus-based standards, such as the widely accepted standards for program and project management from the Project Management Institute (PMI). PMI is a not-for-profit association that provides global standards for, among other things, project and program management. The PMI standards are utilized worldwide and provide guidance on how to manage various aspects of projects, programs, and portfolios and are approved by the American National Standards Institute (ANSI). OMB staff told us that they decided not to specifically adopt the PMI standards because they wanted to allow agencies to use a range of standards that agencies had already developed and were using to manage their programs, such as standards developed in-house by NASA for their space flight programs. OMB further directed CFO Act agencies that the 15 standards and application of them should be incorporated or aligned with existing agency-specific program management policies and practices, and tailored to reflect program characteristics. OMB staff told us that they chose the approach to provide more principle-based standards, as opposed to specific standards, to be flexible enough for a range of government agencies to apply them. OMB’s standards are similar in definition to PMI standards, but they are less detailed by comparison. Our analysis of OMB’s standards shows that OMB uses similar definitions for all 10 of PMI’s program management standards and nine out of 10 of PMI’s project management standards, such as risk management and change management. However, OMB program and project management standards are less detailed when compared to PMI’s standards in the following ways: OMB standards do not provide a minimum threshold against which agencies can gauge to what extent they have met each standard. PMI’s Standard for Program Management provides the definition of a standard but also what components are required for an entity to confirm that the standard has been met. For example, meeting the program financial management standard in PMI requires a financial management plan to be developed, along with its related activities. This plan allows entities applying the standard to confirm whether they have met the standard for program financial management or not. OMB’s standards do not distinguish between how the standards apply differently to programs and projects while PMI has separate detailed standards for program management and for project management. The project management standards from PMI provide details on how the standards apply to more granular tasks, such as establishing a quality management or communication plan for a specific project. OMB’s standards do not distinguish between how the standards relate to each other during a program or project while PMI’s Standard for Program Management details how project standards help build on each other during a program. For example, a program scope management plan is needed to determine the type of schedule management planning that is necessary to accomplish the delivery of the program’s outputs and benefits. OMB provides minimal guidance on how standards apply differently across the life cycle of a program or project while PMI’s Standard for Program Management provides information detailing when a specific standard should be utilized in different ways during the life cycle of a program. For example, in the beginning of a program, risk management should be planned and an initial risk assessment created. Later, during program implementation, risk management tasks focus on monitoring, analyzing risk, and responding to risk. If the standards had the additional detail, it would be possible to determine if agencies are meeting them and properly applying them to programs and projects. Our work on the Digital Accountability and Transparency Act of 2014 (DATA Act) standards has emphasized the necessity for a governance structure with a clear set of policies and procedures for developing and maintaining standards over time that are consistent with leading practices. A governance structure is important because it helps ensure that the standards are developed, maintained, adjusted, and monitored over time. The DATA Act is similar to PMIAA because PMIAA gives OMB responsibility to develop standards for program management, and the DATA Act gives OMB and the Department of the Treasury responsibility for establishing data standards for the reporting of federal funds. These standards specify the data to be reported under the DATA Act and define and describe what is to be included in each element with the aim of ensuring that information will be consistent and comparable. Several governance models exist that could inform OMB’s efforts to help ensure that the standards are developed, maintained, adjusted, and monitored over time. These models define governance as an institutionalized system of decision rights and accountabilities for planning, overseeing, and managing standards. Many of these models promote having a common set of key practices that include establishing clear policies and procedures for developing, managing, and enforcing standards. A common set of key practices endorsed by standards setting organizations including the National Institute of Standards and Technology, ANSI, and the American Institute of Certified Public Accountants recommend that governance structures should include the key practices shown in the text box below. Key Practices for Governance Structures 1. Delineating roles and responsibilities for decision-making and accountability, including roles and responsibilities for stakeholder input on key decisions. 2. Obtaining input from stakeholders and involving them in key decisions, as appropriate. 3. Developing and approving standards. 4. Making decisions about changes to existing standards and resolving conflicts related to the application of standards. 5. Managing, controlling, monitoring, and enforcing consistent application of standards. OMB staff told us they did not have any additional documentation about the governance structure used to develop the program management standards and how OMB will further develop and maintain them. We compared available information about OMB’s governance structure for developing and maintaining program management standards to the five key practices on governance structures and found OMB’s governance structure is incomplete in each of the five key practices. OMB has not delineated roles and responsibilities for decision-making and accountability, including responsibilities for stakeholder input on key decisions. OMB’s strategic plan notes that one role of the PMPC is to help further develop the program management standards. However, OMB has not provided information on how roles and responsibilities will be assigned to continue developing standards in the future. Without clearly delineated roles and responsibilities, there is a risk of confusion which could impede action and accountability for future improvements to program management standards. Further, having clearly delineated roles and responsibilities is particularly important during periods of transition when administrations change. OMB has an incomplete plan for how it will obtain input from stakeholders and involve them in decision-making. OMB received input from stakeholders on the standards it developed in 2018, though the strategic plan states that standards will be further developed with the PMPC in the fourth quarter of fiscal year 2020. However, the strategic plan does not give details on how the PMPC and others will further develop standards. Without robust and comprehensive outreach to individuals who will use or otherwise be affected by the standards, the opportunity to learn from stakeholder experience and perspectives, or anyone who will use or otherwise be affected by the standards, may be diminished. OMB has an incomplete process for developing and approving program management standards. OMB developed and approved the existing standards by obtaining stakeholder input and releasing their approved standards in its strategic plan. However, the strategic plan does not provide documentation on how that process was structured and how it will function in the future. Thus, it is unclear how OMB plans to further develop the standards and what responsibilities and resources will be required from OMB, the PMPC, and agencies under the leadership of the agency PMIOs. OMB has not defined a process for making decisions about changes to existing standards and describing how conflicts related to the application of standards would be resolved. Therefore, it is unclear if or how the standards will be periodically reassessed and updated as circumstances change and leading practices in program and project management are identified. Also, lack of consensus on standards and conflict over how to use them can lead to weakened acceptance and inconsistent application. OMB has not defined a process for managing, controlling, monitoring, and enforcing consistent application of standards. OMB has not developed or directed any type of review or oversight process to determine the adequacy of existing or newly developed standards agency use to manage programs. Having such a process could help agencies to achieve a balance between consistent application of standards and flexible application to account for differences in programs, agency missions, and other factors. However, OMB staff told us that they consider the PMIAA program portfolio review process as a way to help monitor and enforce program standards, as they have a view into how each agency is applying standards for their particular portfolio of programs. Additionally, OMB has given agencies flexibility in using existing agency standards and flexibility to adopt or develop new ones. Without a review mechanism, OMB lacks reasonable assurance that agencies’ efforts to use existing standards or develop new ones will align with government-wide efforts to improve program and project management. Also, establishing an approach to monitoring agencies’ efforts would help identify opportunities to improve program management standards. Without having a governance structure for the program standards, the potential exists that standards will develop in an ad hoc manner, may be applied inconsistently or not at all, and may not be updated to reflect new developments in program management. Further, having a governance structure for managing efforts going forward better positions OMB to sustain progress on program standards as they change over time. PMIAA requires agencies and OMB to regularly review portfolios of programs to assess the quality and effectiveness of program management and identify opportunities for performance improvement. To conduct these portfolio reviews, OMB Circular A-11 notes that agencies and OMB are to use a set of broadly applicable program management principles, practices, and standards associated with successful program outcomes, in addition to more specific standards based on the type of program under review. As a way to help agencies acclimate to the requirements of PMIAA, OMB leveraged two components of the GPRA Modernization Act of 2010 (GPRAMA): the strategic review and a cross-agency priority (CAP) goal. OMB guidance stated that agencies’ portfolio reviews of programs would be conducted and integrated to the extent practical with strategic reviews. Furthermore, OMB staff told us that the implementation of PMIAA and the CAP goal for improving management of major acquisitions (CAP Goal 11) shared complementary goals and strategies. For example, the CAP Goal 11 action plan includes the routine monitoring of federal program management progress. Consequently, OMB staff said they decided that the first PMIAA program portfolio reviews would focus on major acquisitions. Excerpt from OMB Cross-agency Priority Goal 11 from 2018 President’s Management Agenda: Improve Management of Major Acquisitions Federal agencies will ensure that contracts supporting transformative and priority projects meet or beat delivery schedules, provide exceptional customer service, and achieve savings or cost avoidance for the taxpayer. The Challenge: Major acquisitions—which vary in size by agency but often exceed $50 million—account for approximately one-third of annual federal spend on contracts. These large contracts frequently support projects meant to transform areas of critical need. Yet major acquisitions often fail to achieve their goals because many federal managers lack the program management and acquisition skills required to successfully manage and integrate large and complex acquisitions into their projects. These short- comings are compounded by complex acquisition rules that reward compliance over creativity and results. The Strategies: Agencies will pursue three strategies: 1) strengthen program management capabilities in the acquisition workforce; 2) use modern and innovative acquisition flexibilities; and 3) track investments using portfolio, program, and project management principles. In 2018, OMB conducted a pilot project involving program portfolio review focused on noninformation technology (IT) major acquisition programs. According to OMB staff, the pilot project gave agencies the opportunity to complete “dry runs” for the PMIAA-required portfolio reviews and to provide lessons learned in anticipation of the fiscal year 2019 portfolio reviews. OMB planned for the results from the pilot to provide information for internal dialogue and decision-making about subsequent portfolio reviews. Further, according to OMB’s strategic plan, the purpose of the pilot was (1) to determine how well agency program portfolios of non-IT major acquisitions were performing throughout the life cycle of the investment using a set of standards and practices, and (2) to refine the process of coordinating program portfolio reviews as a component of OMB agency strategic reviews. For the pilot, OMB staff directed agencies to assess the cost, schedule, and performance of agency-selected acquisition portfolios. One result from the pilot was that agencies demonstrated a range of maturity in their abilities to collect data for these required program portfolio measures from their various departments and program types. OMB staff told us pilot agencies found it easier to compile data on major construction projects compared to service contracts. Consequently, an agency doing many of these projects might be more advanced than an agency for which major acquisitions focus on services. Department of Veterans Affairs (VA) staff shared their lessons learned from their participation in pilot portfolio reviews, as seen in the text box below. OMB staff said that they determined that the portfolio review process worked sufficiently well for the pilot agencies and continued their planned strategy of focusing solely on non-IT major acquisition programs for fiscal year 2019 portfolio reviews. Example of Department of Veteran Affairs (VA) Lessons Learned from Pilot Portfolio Review The VA looked at the effectiveness of portfolio management during the Office of Management and Budget noninformation technology major acquisition pilot portfolio review by focusing on the agency’s adherence to best practices in assessing project performance and progress. VA officials said this pilot informed their decision-making and was successful in the following ways: 1. The pilot helped VA determine logical ways to manage a portfolio by showing what data were helpful to make impactful decisions. 2. VA learned how best to display the data on cost, schedule, scope, and quality of outcomes on a dashboard to make it accessible and comparable across the agency. 3. VA learned that it needs to collect better quality data so that project management principles can be instituted and aligned across the agency. A well-developed and documented pilot program can help ensure that agency assessments produce information needed to make effective program and policy decisions. Such a process enhances the quality, credibility, and usefulness of evaluations in addition to helping to ensure the effective use of time and resources. We have identified five leading practices that, taken together, form a framework for effective pilot design, as seen in the text box below. OMB fulfilled the first leading practice of establishing objectives in its design of the PMIAA pilot program portfolio review. OMB’s PMIAA strategic plan and the CAP Goal 11 Action Plan stated the objectives of the pilot. In addition to the two objectives listed in the PMIAA strategic plan, the CAP Goal 11 Action Plan lists seven pilot objectives, as seen in the text box below. PMIAA Pilot Program Portfolio Review Objectives 1. Perform portfolio management preparation activities 2. Identify first portfolio of major acquisitions 3. Align portfolio with agency strategic goals 4. Collect performance data for each item in the portfolio 5. OMB officials said that they did not structure the pilot to follow the remaining four leading practices for effective pilot design. However, OMB said that it learned that the pilot agencies demonstrated several program management capabilities. They also learned that it would be important to tailor portfolio reviews to the agency and the program to account for significant differences in the types of acquisitions and the level of program management maturity. Despite identifying lessons learned from its pilot program portfolio review, in neglecting to fully follow leading practices, OMB may have missed opportunities to make additional improvements for fiscal year 2019 portfolio reviews. Going forward, as OMB expands the portfolio reviews to other types of program areas beyond non-IT major acquisitions, it has the opportunity to develop and learn from additional pilots. Although OMB staff have not yet determined if they will do additional pilots for program management in the future, they could decide to pilot the portfolio reviews of grants that they plan to initiate in fiscal year 2020. For fiscal year 2019, OMB directed all agencies to select portfolios of non-IT acquisition programs and align them with relevant strategic objectives as part of their internal agency strategic review processes. In spring 2019, OMB expected agencies to discuss one to two of these major-acquisition portfolio reviews during their strategic reviews with OMB. OMB expected agencies to track the cost, schedule, and performance of their selected major acquisition programs. However, OMB reports that not all agency program portfolio reviews were completed because OMB was behind in scheduling the reviews due to the partial government shutdown. According to documents we reviewed and what OMB staff told us, in October 2019 OMB completed agency program portfolio reviews with ten agencies: the Departments of Commerce, Homeland Security, Housing and Urban Development, Labor, and Transportation; the General Services Administration, the Social Security Administration, NASA, the National Science Foundation, and the US Agency for International Development. OMB staff also told us that they also held preparatory meetings with agencies to set expectations for future portfolio reviews. OMB reported that these one-on-one meetings were held with 12 agencies as of October 2019 to discuss their initial portfolio structures and other transformative initiatives. Portfolio reviews in 2020 are to expand in scope to include grants, and also will continue acquisition portfolio reviews as part of the agency’s routine management processes. However, OMB has not yet identified other program areas, such as research and development or benefit programs, to be included in future portfolio reviews. Standards for Internal Control in the Federal Government states that effective information and communication are vital for an entity to achieve its objectives. Specifically, management should externally communicate necessary quality information to achieve its objectives. Increasing communication to agencies about specific program areas, portfolio review procedures, and expectations beyond 2020 could help ensure continued progress to implement PMIAA more broadly. Furthermore, communicating such procedures with specific time frames could help agencies better direct their efforts to improve the portfolio review processes. GPRAMA requires OMB to make a list of all federal programs identified by agencies publicly available, on a central government-wide website. The implementation of the program inventory is a critical tool to help decision makers better identify and manage programs across the federal government. Among other things, the completion of the program inventory would provide agencies and Congress with a comprehensive list of programs, so it would be clear how many programs agencies are managing and how they relate to their strategic objectives and portfolios of programs at each agency. Having a program inventory could also help ensure a match between the number of agency programs and needed program manager resources. Agencies continue to struggle with challenges defining their programs. Officials from three of the five selected agencies we spoke with told us that they have not yet identified all of their programs and projects. In our first report on the program inventory in October 2014, we noted similar issues. For example, agencies were not using the same program definition approach across their subcomponents or offices, which limited comparability of their own programs. We made eight recommendations in that report to the Director of OMB to update relevant guidance to help develop a more coherent picture of all federal programs and to better ensure information is useful for decision makers. As of October 2019, OMB had not taken any actions in response to the eight recommendations. While OMB has provided a timetable for action in its June 2019 A-11 guidance, this does not complete the recommendation. In September 2017, we made two recommendations to OMB to make progress on the federal program inventory. First, we recommended that OMB consider using a systematic approach for the program inventory, such as the one we developed from principles of information architecture. Information architecture—a discipline focused on organizing and structuring information—offers an approach for developing a program inventory to support a variety of uses, including increased transparency for federal programs. OMB staff told us that they considered our information architecture approach and noted that a structured information architecture format is used on USASpending.gov. However, OMB staff told us they had not yet determined how the information architecture format of USASpending.gov—which is focused on spending data—could be used to meet additional information reporting requirements and our past recommendations related to the inventory. We made a second recommendation that OMB should revise and publicly issue OMB guidance—through an update to its Circular A-11, a memorandum, or other means—to provide time frames and associated milestones for implementing the federal program inventory. As mentioned above, OMB did provide a timetable but it does not have milestones. According to the timetable, beginning with the 2021 budget cycle, agencies’ program activities will be used for the inventory’s program-level reporting requirements. This will allow OMB and agencies to present program-level spending data by leveraging what is reported on USASpending.gov as required by the DATA Act. However, OMB’s guidance does not cover other inventory information reporting requirements, or the actions we recommended in October 2014. We will continue to monitor progress. We continue to believe it is important for OMB to implement our program inventory recommendations. Such an inventory could be a critical tool to help decision makers better identify and manage fragmentation, overlap, and duplication across the federal government. Additionally, fully taking action on these recommendations would assist agencies in identifying programs, better prepare for future PMIAA portfolio reviews, and help match resources to agencies’ program management needs. Further, OMB developed three different definitions for what constitutes a “program” or “program activity” that it provided to agencies in its PMIAA, GPRAMA, and DATA Act guidance, respectively. OMB developed each of these definitions independently and in response to three different statutory requirements. OMB staff told us that these three requirements differ in their legislative intent. The definitions and their associated guidance are in the table below. OMB has not reconciled these overlapping, yet divergent, definitions of what constitutes a “program” or “program activity.” According to Standards for Internal Control in the Federal Government, management should ensure that specific terms are fully and clearly set forth so they can be easily understood. Standards for Internal Control in the Federal Government also states that management should design processes that use entities’ objectives and related risks to identify information requirements needed to achieve objectives and address risks. OMB has defined what constitutes a “program” or “program activity” in PMIAA, GPRAMA, and the DATA Act each, but its three different program definitions and approaches to determining what is a “program,” could cause confusion for agencies. Agency officials from the Department of Energy told us they are already experiencing confusion over how to appropriately apply the applicable program definition to identify their programs for PMIAA. Agency officials from Treasury told us that different definitions for programs could contribute to confusion as they work to implement PMIAA within the Department. The inconsistent approaches may increase the burden on agencies as they work to identify, maintain, and report on three sets of differently defined programs. Conversely, clarifying the definitions could help agencies and OMB identify synergies across the three laws and increase transparency. For example, providing explanations of how the term “program” or “program activity” is used across the three statutory definitions and developing a crosswalk to show similarities and differences could provide more clarity for agencies. Then, spending and performance data can be aligned with agency strategic goals, which could be monitored, reviewed, and reported in a streamlined manner. OPM followed PMIAA requirements to create policy and guidance. Specifically, according to documents we reviewed, OPM (1) worked with subject matter experts to develop program and project management skills and competencies, (2) updated the program management 0340 job series and created guidance for identifying project management positions, (3) plans to release a career path for program and project managers by the end of calendar year 2019, and (4) plans to create a unique job identifier code that can be used to pinpoint program and project managers in any job series. These efforts will form the foundation needed by agencies to strengthen resource and talent management. Competency modeling. Since enactment of PMIAA, OPM identified skills and competencies which will be required for program and project managers. According to documents we reviewed, OPM met with subject matter experts and human capital staff in agencies to help identify the skills needed to develop the competency model. OPM also conducted a literature review looking at prior competency studies and industry practices to help identify and support program and project management competencies. OPM also drew from Project Management Institute resources, such as the Project Management Body of Knowledge and the Standard for Program Management, as part of identifying its competencies. The resulting competencies are in two categories: general and technical. General competencies focus on interpersonal or general on-the-job skills such as teamwork and problem solving. Technical competencies more narrowly focus on particular skills needed to run programs and projects, such as risk management and cost-benefit analysis. OPM documents stated that agencies will need to determine the applicability of these competencies to positions within their agency. Agencies must determine if staff meet the competencies, and if not, staff will have the opportunity to develop them or must move to a different job series, according to OPM staff. OPM staff also said additional competency assessment steps are needed to finalize the model. Agencies will be given time to consider the competency model. In addition, OPM will use subject matter expert panels to further develop the model, according to OPM documents we reviewed. Updated job series. To implement job series requirements in PMIAA, OPM staff conducted an occupational study and determined that pre- existing classification policy was sufficient for classifying program management work rather than creating a new job series classifying program management positions, according to OPM staff. Prior to OPM updating the program management 0340 job series for PMIAA, the classification standard was not developed, as it did not contain competencies describing what qualifications staff were required to meet as a program manager. In May 2019, OPM released the updated job series classification guidance designed to assist agencies in determining which employees fit in the job series. OPM also released guidance for classifying project managers to help agencies specifically identify project managers in any occupational job series. According to the memorandum sent by the Acting Director of OPM to agencies with the OPM classification guidance, agencies are required to implement the policy and guidance to covered positions by May 1, 2020. Career path. OPM staff told us that they have developed a career path for program and project managers that is currently in internal review. They said that the value of the updated career path is that it will highlight training and skills needed to progress in a program management career. According to the presentation given by OPM at the 2019 April PMPC meeting, the career path will contain: (1) a career progression outline for employees to move among and across jobs in program and project management, (2) help for employees and supervisors to plan and sequence appropriate career training and development for each general and technical competency, and (3) a list of common degrees and certifications completed by program and project managers, among other things. Staff told us they plan to release the program and project management career path for agency comment by the end of calendar year 2019. Job identifier for program managers and project managers. Because program and project managers are found in other job series outside the 0340 program management series, OPM is developing a job identifier code that can be attached to any job series for the purposes of identifying program and project managers. OPM staff told us that program managers classified to the 0340 series means that the position does not have a specialization. If the position requires specialized expertise, the position would be classified to a specialized occupational series but would also have a program management job identifier code. For example, since a grants managers is also a program manager, “grants manager (program management)” would be his or her official title. Project management positions will also use a job identifier to identify project managers in any occupational series. The job identifier will allow employees with a specialization to be designated program and project managers, while still maintaining their original career path. OPM staff told us they plan to complete this project in 2020. Our analysis of OPM Enterprise Human Resources Integration data shows that the 0340 job series included about 15,000 employees across all 24 CFO Act agencies in fiscal year 2018. However, OPM reported that not all employees in this job series are actually program and project managers; conversely, many program and project managers are working outside of the 0340 job series. Selected agencies reported varying degrees of difficulty identifying program and project managers. For example, NASA staff reported that they were able to identify almost all their program and project managers. In contrast, the Department of the Treasury reported that it faces challenges identifying the number of program and project managers outside of the program management job series, as this would require a resource-intensive manual effort, made more challenging by the agency’s large, complex, and decentralized structure. The Department of Energy (DOE) staff said they have not completed the count of their program managers. The Departments of Commerce and Veterans Affairs also report they do not know the number of program and project managers in their departments, respectively. The Department of Commerce staff told us that they cannot accurately identify the number of program and project managers until they can use the job identifier that they expect OPM to release in 2020. Further, Commerce officials told us they are also continuing to work to identify program managers and engaged the Project Management Institute (PMI) to request a list of those within Commerce who have the Project Management Professional (PMP) certification. PMI was able to provide Commerce details about the numbers of PMPs at Commerce, but PMI declined to share the names of those individuals with the PMP certification. In OPM’s 2018 Federal Workforce Priorities report, OPM recognizes that not all agencies have adequately analyzed workload demands, staffing levels, or current and future skills needs—all steps in workforce planning. As part of the OPM human capital framework, agencies are required to develop a human capital operating plan which is an agency’s human capital implementation document. These plans are to describe how agencies will execute the human capital strategies needed to implement the agency’s strategic plan and Annual Performance Plan (APP). Agencies are also required to include program specific strategies (e.g., hiring, closing skills gaps, etc.) in the APPs as appropriate. Effective workforce planning can help agencies focus on determining how many program and project managers they have, how many they may need, what skills gaps exist, and what training and other strategies can help address skills gaps. OPM’s workforce planning model is comprised of five steps: 1. Set strategic direction; 2. Analyze workforce, identify skills gaps, and conduct workforce 3. Develop action plan; 4. Implement action plans; and 5. Monitor, evaluate, and revise. The discussion below describes how OPM and agencies are working to strengthen the program management workforce in the context of OPM’s workforce planning model. Some activities may span more than one phase of workforce planning. Set strategic direction. The PMIAA strategic plan establishes direction for agencies to build its program management capacity and capability with its third strategy, “Strengthening Program Management Capacity to Build a Capable Program Management Workforce.” Setting strategic direction also involves linking work activities to the objectives of a strategic plan. OPM’s planned activities, such as updating the classification standards and creating a job identifier, are critical to executing this strategy so agencies can identify their workforce and build program management capacity through training, career paths, and mentorship opportunities. Analyze workforce, identify skills gaps, and conduct workforce analysis. OPM and agencies are in the early stages of identifying who their program and project managers are and what human capital strategies might be needed to address agencies’ needs. Documents we reviewed showed that OPM also worked with the Chief Human Capital Officers Council, the Chief Administrative Officers Council and others to develop competencies. These competencies provide a foundation for the subsequent assessment of program and project manager skills. Develop action plan. In their PMIAA implementation plans, some agencies have identified available training and possible recruitment and hiring strategies. In OPM’s model, agencies need to complete their workforce analysis before they can develop their action plans. Implement plan. This step is dependent on agencies developing action plans. However, OPM and agencies have already started to develop staff in the absence of plans. For example, OPM is working with agencies to identify program management training matching desired competencies to be placed in an online training repository that will be accessible to all agencies. OPM staff told us that agencies would provide the trainings from their learning management systems and offer them for interagency access. OPM is developing this training and development repository which will house agency-owned courses and also identify mentors in project and program management, according to OPM staff. OPM will house the repository on its training and development policy wiki at https://www.opm.gov/wiki/training/index.aspx. Each PMIO is to also establish a website with agency-specific program management tools and resources. Additionally, OMB recognized that the Federal Program and Project Management Community of Practice (FedPM CoP), scaled up from a community of practice housed in DOE, could be an important partner in supporting PMIAA implementation. As of April 2019, more than 1,000 managers had joined the FedPM CoP as indicated in its briefing to the PMPC. The FedPM CoP has identified several project management-related documents that are now available on the PMIAA portal. To further develop program managers, OMB is working with agencies to improve mentoring and recognition efforts. To improve mentoring government-wide, OMB reports that PMIOs will work with agency chief human capital offices to develop and implement a mentoring strategy for agency program managers. OMB also plans to take existing mentorship programs established in more functionally aligned-management fields (e.g., information technology, acquisition) and expand them to include a broader range of management career paths. To improve recognition efforts in acquisitions, the Chief Acquisition Officer Council plans to establish an annual award to recognize federal program manager excellence. Monitor, evaluate, and revise. This step cannot begin until agencies develop and implement their workforce action plans. As agencies begin to monitor their implementation of these plans, they will need to determine if any skills gaps exist in the program and project manager occupational series. OPM regulations require agencies to describe in their human capital operating plans agency-specific skills and competency gaps that must be closed through the use of agency selected human capital strategies. Agencies must also have policies and programs that monitor and address skills gaps within government wide and agency-specific mission-critical occupations. OPM has not yet determined if program and project management occupations are experiencing mission-critical skills gaps across the government, and OPM staff noted that agencies are not specifically required to report program and project manager skills gaps in their annual human capital operating plans. OMB and OPM both missed statutory deadlines to fulfill requirements in PMIAA. In June 2018, OMB issued the required PMIAA agency implementation guidance in the PMIAA strategic plan, 6 months after the statutory deadline of December 2017. According to OMB staff, this delay was due to their own research project to (1) build sufficient knowledge in program and project management; and (2) increase stakeholder support in Congress and with agencies for its approach. Specifically, OMB met with experts from PMI, academics, consulting firms, federal chief senior level officer (CXO) councils, and other agency officials to increase its own understanding of program and project management principles. OMB staff told us that they used the collected information to draft initial guidance, which they then shared with congressional stakeholders and executive branch agency officials to obtain feedback and incorporate changes. OMB staff also told us that it was a transition year from one administration to another administration, and this transition was an additional factor in delaying completion of the guidance. None of the selected agencies’ staff identified an impact from the delayed guidance. OPM officials told us they missed the statutory deadline to complete their required activities after the issuance of OMB guidance. The release of the policy and guidance was due to the partial government shutdown from December 22, 2018 to January 25, 2019, along with a 3-month delay due to OPM’s own internal review and clearance process. As a result, OPM released the key skills and competencies needed for program and project management on April 5, 2019, and the classification guidance for the program manager job series 0340 and project manager interpretative guidance on May 2, 2019. OPM officials told us that agencies have 1 year from the date of issuance to comment on any language in the guidance. None of the selected agencies’ staff identified an impact from OPM’s delays, although one agency expressed concern that the pace of their efforts to identify program and project managers is dependent on OPM completing the job identifier. Figure 2 shows the delays in releasing OMB and OPM guidance. OMB officials established the PMPC in 2018 and fulfilled requirements that it meet at least twice per year. By September 2018, the 24 CFO Act agencies had all appointed a PMIO and held three PMPC meetings, in September 2018, April 2019, and September 2019. Selected agenda items for these PMPC meetings included: status updates on OPM completing program and project manager competencies, job series, and career path; breakout sessions to discuss PMIAA implementation approaches with discussion of PMPC priorities and focus for 2020. At the April 2019 PMPC meeting, for example, staff from the Department of Veterans Affairs and the National Science Foundation shared some best practices, such as how to improve the tracking performance of portfolios, programs, and projects. According to OMB documents we reviewed, OMB plans to: convene the PMPC in the first quarter of each calendar year to prepare for upcoming OMB and agency strategic review meetings; use the PMPC meeting in the third quarter of the calendar year to review findings and outcomes from the most recent strategic review; update program and project management standards based on its findings and feedback at the PMPC meeting in the fourth quarter of 2020; use the PMPC to develop revised strategies, initiatives, and priorities to be reflected in an updated 5-year strategic plan at the PMPC meeting in the fourth quarter of 2021; and use the PMPC to focus on improving our high-risk areas at some future point. At the September 2019 PMPC meeting, OMB informed agencies of PMIAA implementation resources placed on OMB’s online portal for PMIAA and discussed OMB’s observations on portfolio reviews completed in 2019. One observation was the need to reinforce better visualization of performance data. In addition, OPM updated the PMPC on the status of its required PMIAA workforce efforts. The PMPC decided its primary focus for the year 2020 should be on the third strategy of the PMIAA strategic plan to build a capable workforce. Officials from the selected agencies that we interviewed provided us some suggestions on how OMB can improve the functionality of the PMPC. Table 3 illustrates the range of these suggestions: The PMPC met twice in 2019, as required by PMIAA, and has not established any working groups to help execute its significant responsibilities to share leading practices, develop standards, and help improve the workforce. Agencies have taken initial steps to incorporate requirements into program efforts. According to OMB guidance, agencies were to report in implementation plans how they are institutionalizing PMIAA efforts— especially PMIO responsibilities—into existing program and project management practices. OMB requested that agencies include 10 specific elements in their implementation plans, such as: identification of the agency PMIO, identification of major acquisition portfolios, and strategies and actions for enhancing training and improving recruitment and retention of program and project managers. These plans were due to OMB by November 30, 2018. We reviewed PMIAA draft implementation plans for 22 of the 24 CFO Act agencies and determined the extent to which agencies included the required elements in their plans. In its PMPC meeting in April 2019, OMB reported that a majority of agencies only partially included OMB requirements in their draft implementation plans. OMB told us they have not directed agencies to address missing requirements nor have they required agencies to finalize their draft implementation plans. They told us that they view the implementation plans as an opportunity for each agency to engage with OMB and discuss how they will implement PMIAA. OMB staff told us that their view is that if implementation plans provide value to agencies, they may stay in draft form and do not need to be final. Overall, draft implementation plans for these agencies provided some but not all information required to fully meet the directives from OMB. Our analysis of the plans shows that on average, agencies fully met six out of 10 requirements for their implementation plans. For example, almost all agencies met the requirements for identifying the PMIO (21 out of 22). However, 11 out of 22 agencies did not provide complete information on major acquisition portfolios. Table 4 shows how agencies’ implementation plans varied in meeting the requirements. Seven of 24 agencies reported in our questionnaire that they were creating either task forces or new or restructured offices to direct PMIAA implementation within their agencies. For example, DOE reported establishing a new office to support its PMIO. The Department of the Treasury and NASA reported creating an intra-agency cross-functional core team to discuss and design PMIAA implementation strategies. OPM reported establishing an enterprise program management office to drive the standardization of program and project management processes internally. Agencies selected PMIOs in existing leadership positions to leverage resources and agency processes to implement PMIAA. All agency PMIOs reported having additional leadership responsibilities beyond their PMIO roles. OMB documentation and information gathered from CFO Act agencies shows: every PMIO has at least one additional CXO role within its agency; thirty-eight percent of PMIOs have an additional performance management role; eight of 24 PMIOs have an additional budgetary role; and four of the 24 PMIOs have an explicit additional program or acquisition role. In the past, we have met with senior management officials from OMB and applicable agencies to discuss where additional management attention could be beneficial to addressing high-risk areas identified on our High- Risk List. We also reported that these trilateral meetings, which began in 2007 and pre-dated PMIAA’s 2016 enactment, have continued across administrations and have been critical for progress that has been made in addressing high-risk areas. According to PMIAA, OMB’s Deputy Director of Management is to conduct annual portfolio reviews of the most at-risk agency programs, as designated by our High-Risk List. OMB officials view the trilateral meetings as their method for holding the portfolio review meetings for high-risk areas as required under PMIAA. Our High-Risk List is comprised of programs as well as functions and operations. Consequently, in our assessment of OMB’s implementation of PMIAA, we consider programs, functions, and operations on our High-Risk List as relevant for OMB’s portfolio review of areas on our High-Risk List. OMB used three strategies intended to meet PMIAA’s high-risk requirements. OMB (1) expanded its strategic reviews in 2018 to include a review of some high-risk areas, (2) continued to use the long-standing trilateral meetings to review high-risk areas with agency leaders and with us, and (3) held ad hoc meetings with agencies outside of the strategic review and trilateral meetings. In preparation for the 2018 strategic reviews, OMB issued Memorandum M-18-15 directing agencies to provide several items in advance of their strategic review meetings with OMB. Requested items included updates from agencies on areas identified on our High-Risk List in which agencies disagreed with our recommendations or faced implementation barriers preventing progress. These materials were to be discussed during strategic review meetings. Thirteen CFO Act agencies reported submitting high-risk updates to OMB prior to these meetings, and eight agencies reported discussing their high-risk areas with OMB during the meetings. OMB guidance from June 2019, communicated in OMB’s Circular No. A- 11, did not include the statement from Memorandum M-18-15 that high- risk areas would be discussed during strategic review meetings. OMB staff felt that a broader approach could yield better results for addressing high-risk areas. Guidance in Circular No. A-11 maintained that agencies should submit updates about high-risk programs to OMB for the Deputy Director’s high-risk portfolio review, but it did not specify what should comprise agency updates about high-risk programs. Also, OMB staff told us that they requested that agencies provide topics for discussion at strategic review meetings, and that agencies could provide agenda items related to our High-Risk List. OMB staff said they addressed only a few of the high-risk issues during strategic reviews, both during the review process and the strategic review meetings. Discussions about high-risk issues during strategic review meetings generally focused on government-wide high-risk areas, if relevant, such as “Ensuring the Cybersecurity of the Nation” and “Improving the Management of Information Technology (IT) Acquisitions and Operations.” However, OMB and agencies also discussed high-risk areas in instances when agencies provided strategic review meeting agenda topics related to our High-Risk List. For example, Treasury staff told us they spoke with OMB this year about high-risk areas as part of the strategic review process. Treasury is directly responsible for the Enforcement of Tax Laws high-risk area and shares responsibility with other agencies for other high-risk areas, such as the government-wide areas on cybersecurity and strategic human capital. OMB has held a limited number of trilateral meetings with agencies and us about high-risk areas as part of the high-risk portfolio reviews. Between March 2018 and October 2019, OMB addressed the following five high-risk areas in trilateral meetings with applicable agencies and us: 2020 Decennial Census, Managing Federal Real Property, Government-wide Personnel Security Clearance Process, Ensuring the Cybersecurity of the Nation, and NASA Acquisition Management. OMB has not held meetings to address the remaining 30 high-risk areas on our High-Risk List. OMB staff told us they plan to hold additional meetings in the next year but that they are unlikely to be able to schedule all remaining meetings within our 2-year cycle for updating the High-Risk List. OMB staff said that it is sometimes challenging to coordinate and convene trilateral meetings given the high-ranking officials who must attend and finding available times across schedules. OMB also told us that they plan to meet with agencies for all high-risk areas eventually, but that they prioritize meetings aligned with our priority areas and the President’s Management Agenda. We evaluate progress made on high-risk areas every 2 years to determine if new areas should be added to our High-Risk List and if areas on the list should be removed due to progress to address the risks. Top leadership commitment is one of the five criteria we use to assess whether progress is being made to address and ultimately remove areas from our high-risk list. As we have reported in our March 2019 High-Risk Series report, leadership commitment is the critical element for initiating and sustaining progress, and leaders provide needed support and accountability for managing risks. Leadership commitment is vital if agencies are to adequately address high-risk areas, and trilateral meetings have been critical in focusing leadership attention in the past. Because OMB officials have met on only five of 35 high-risk areas, it remains to be seen if they will meet on all high-risk areas in the future. Convening the trilateral meetings on all high-risk areas in the 2-year reporting cycle, would better position OMB to enhance the leadership commitment needed to make greater progress on the remaining high-risk areas. Staff from OMB said that they sometimes have briefings related to agencies’ high-risk areas separate from the annual strategic review meetings and high-risk trilateral meetings. These meetings happen on an ad hoc basis and are typically initiated by agency officials. Officials from some of our selected agencies corroborated that the discussion at the strategic review meetings and trilateral meetings is not the full extent of OMB’s interaction with agencies about high-risk areas throughout the year. For example, VA officials said that high-risk areas are frequently agenda items in meetings with OMB. NASA officials said they spoke with OMB about NASA’s high-risk areas after submitting material as part of the strategic review process. The PMPC, chaired by the Deputy Director for Management of OMB, did not address our High-Risk List during its three meetings nor did it make recommendations to OMB about addressing high-risk areas, as required. The PMPC meetings have lasted 60 to 90 minutes each and the High-Risk List has not appeared as an item on any of the PMPC meeting agendas. OMB staff said PMPC meetings at this point in PMIAA implementation primarily act as forums in which agencies can share program management practices. Rather than focusing meeting time on high-risk areas, OMB staff asserted that the best use of the PMPC is primarily as a forum for agencies to share program and project management best practices. Consequently, the PMPC has not satisfied all PMPC requirements as delineated in PMIAA, including for high-risk areas to be addressed. OMB created a dashboard to identify measures of cost, schedule, and performance that agencies should use to track their selected non-IT major acquisition programs for the first PMIAA program portfolio review. OMB partnered with the General Services Administration to complete a prototype of a dashboard to show cost, schedule, and performance data from each program or project within a portfolio of programs. The dashboard also provides a short description of each program or project and its strategic alignment to the agency’s relevant strategic goal. Staff from OMB’s Office of Federal Procurement Policy said the dashboard could provide them with some visibility and improved transparency for major acquisitions programs. According to the PMIAA strategic plan, the dashboard would display the agency portfolio and summarize performance for each item in the portfolio, similar to the portfolio reviews of IT programs required by the Federal Information Technology Acquisition Reform Act. Initially, according to OMB, it plans to request summary information for each portfolio, and restrict the dashboard to authorized government employees. Moving forward, OMB staff said that as the portfolio management process matures, a portion of the dashboard may be available to the public, similar to the IT dashboard. OMB staff told us they are in conversation with agencies about how to overcome difficulties in collecting data for the dashboard. According to OMB, the results from the pilot portfolio review showed that agencies experienced challenges with collecting high-quality data. OMB staff said there will likely be more metrics for large construction projects because management practices for them are more mature than for other types of programs, such as services. OMB is working with agencies to see how they can retrieve cost, schedule, and performance data that could provide early warning indicators of potential problems with programs. Agencies reported in our questionnaire they are considering various ways to measure implementation of PMIAA. A little more than half of agencies responding to our PMIAA questionnaire provided ideas on how to measure implementation of PMIAA, such as tracking completion of their identified PMIAA milestones, developing their own survey as a baseline measure, or using their agency implementation plan outcomes to measure results. Six agencies’ questionnaire responses noted that they are planning to use existing metrics to assess program performance, either through internal processes or their annual strategic review process. For example, Treasury plans to focus in the near term on tracking completion of milestones of PMIAA implementation, such as major program and project alignment to department strategic objectives, development of an information-sharing site for program and project management resources, and workforce capabilities, among other things. VA anticipates developing outcome measures associated with successful program execution and is leveraging measures from existing plans, such as their Acquisition Human Capital Plan. OMB staff told us that they have no plans to identify measures to assess outcomes of PMIAA because it is too early and agencies are in the early stages of implementation. Rather than tracking anything specific, they told us that OMB looks at whether agencies’ PMIOs are engaged, if agencies are using training material and mentorship programs, the involvement of chief senior level officers, and if there is funding in the budget for program management certificate programs. However, OMB has not identified specific measures to track any of these areas. In collaboration with OMB, VA developed a program management maturity model survey identify capability gaps, obtain insights, and enable benchmarking of program management capabilities. It surveyed agencies’ level of maturity on a range of program management capabilities, such as talent management, governance, and portfolio management. Maturity assessment surveys can be useful tools for measuring progress to develop capacity in areas such as program management, according to subject matter specialists. Periodically measuring maturity can help agencies institutionalize continuous assessment and improvement. PMI also supports using such tools to identify trends that can help pinpoint actions needed and opportunities to learn from more mature organizations. We have found that ongoing performance measurement can serve as an early warning system to management and as a vehicle for improving accountability to the public. We have previously reported that providing baseline and trend data can help to assess an agency’s performance more fully because the data show progress over time and decision makers can use historical data to assess performance. As OMB and agencies move forward with PMIAA implementation, it will be critical to measure how agencies are maturing or building their capacity in the areas of program and project management. Such measures could include showing how OMB’s program management standards and principles are integrated into agencies’ programs and policies, the improvement of data quality used to track agency program outcomes in the program portfolio reviews, and improvement in program manager skills. Although not required by PMIAA, it is a good practice for OMB and agencies to consider ways to measure the effects of the act. Without establishing such measures to assess PMIAA outcomes, it will be challenging to gauge how agencies are making progress to identify trends, or to help agencies improve data quality. The program and project management standards OMB developed are less detailed than accepted standards and are missing several elements that would have made them more useful. For example, the OMB standards do not provide a minimum threshold against which agencies can gauge to what extent they have met each standard. Further, OMB’s current governance structure is insufficient for further developing and maintaining program management standards. Although OMB received input from stakeholders to develop the standards and plans to update them in partnership with the PMPC in 2020, OMB does not have a governance structure that assigns roles and responsibilities to further develop, approve, maintain, or monitor standards. Having such a governance structure for managing efforts going forward could help sustain the program standards as they change over time. OMB did not follow most leading practices for designing pilots and may have missed opportunities to make improvements for fiscal year 2019 portfolio reviews. OMB has not determined if it plans to conduct additional pilot efforts. Going forward, as OMB expands the portfolio reviews to other types of program areas beyond non-IT major acquisitions, it has the opportunity to develop and learn from additional pilots. Although OMB staff have not yet determined if they will do additional pilots for program management in the future, they could decide to pilot the portfolio reviews of grants that they plan to initiate in fiscal year 2020. OMB has not identified other program areas beyond non-IT major acquisitions and grants to be included in future portfolio reviews. Communicating to agencies about specific program areas, portfolio review procedures, time frames, and expectations beyond 2020 could help agencies better direct their efforts to improve the portfolio review processes and help ensure continued progress to implement PMIAA more broadly. As of October 2019, OMB had not taken any actions in response to the recommendations in our September 2017 report and has not yet fully established an inventory of federal programs. Such an inventory of programs could be a critical tool to help agency officials identify and manage programs across the federal government. Furthermore, if OMB were to fully implement our recommendations and complete the required inventory of federal programs, it would assist agencies to match resources to agencies’ program management needs and assist agencies in preparing for future PMIAA portfolio reviews. Furthermore, OMB provides three different definitions for a “program” in its guidance for PMIAA, GPRAMA, and the DATA Act. Having different definitions of what constitutes a program could lead to confusion among agencies. It could also cause increased burden on agencies as they work to identify, maintain, and report on three sets of differently defined programs. Meetings between OMB, relevant agencies, and us have been critical for past progress on high-risk areas. However, OMB has held these trilateral meetings to address only five of 35 high-risk areas since it began implementing PMIAA. These meetings could both demonstrate and improve the commitment of agency leadership to high-risk areas across the federal government. As we have reported, leadership commitment is a key tenet in agencies’ ability to address high-risk areas. Without convening trilateral meetings on each high-risk area, OMB might miss opportunities to make progress toward addressing high-risk areas by improving leadership commitment to addressing them. The PMPC did not address our High-Risk List during its meetings nor has it made recommendations to OMB about high-risk areas. The High-Risk List has not appeared as an item on any of the PMPC meeting agendas. OMB staff asserted that the best use of the PMPC’s limited meeting time is as a forum for agencies to share program management best practices. In choosing to focus on program management practices rather than high- risk areas, the PMPC has not satisfied all PMPC requirements as delineated in PMIAA. Having measures to assess outcomes of PMIAA, such as establishing a baseline of information on programs or collecting trend data, can help OMB ensure that it has established a framework to effectively guide and assess PMIAA’s implementation. Assessment measures would also allow OMB to better target efforts to improve project management and the capabilities of managers. We are making a total of eight recommendations to OMB. Specifically: The Deputy Director for Management of OMB, in conjunction with the PMPC, should develop program and project management standards to include (1) a minimum threshold for determining the extent to which agencies have met the standards, (2) how standards apply differently at the program and project levels, (3) how standards are interrelated to work in a synchronized way, and (4) how standards should be applied across the life cycle of a program or project. (Recommendation 1) The Deputy Director for Management of OMB, in conjunction with the PMPC, should create a governance structure to further develop and maintain program and project management standards that fully aligns with key practices for governance structures. (Recommendation 2) The Deputy Director for Management of OMB should, when expanding PMIAA to additional program types, design pilot efforts to follow leading practices so that OMB can optimize its efforts to improve and broaden portfolio reviews across a full range of program types. (Recommendation 3) The Deputy Director for Management of OMB should communicate program areas and timeframes, and expectations pertinent to annual program portfolio reviews, to be reviewed in future program portfolio reviews. (Recommendation 4) The Deputy Director for Management of OMB should clarify for agencies how the different definitions of a “program” relate to each other in OMB guidance. (Recommendation 5) The Deputy Director for Management of OMB should convene trilateral meetings between OMB, relevant agencies, and us for addressing all high-risk areas during each two-year high-risk cycle (Recommendation 6). The Deputy Director for Management of OMB, in conjunction with PMPC, should ensure PMPC meeting agendas include time for discussing high- risk areas during meetings and provide time for the PMPC to make recommendations to OMB about addressing high-risk areas. (Recommendation 7) The Deputy Director for Management of OMB, in conjunction with PMPC, should establish measures to assess outcomes of PMIAA, such as establishing a baseline of information on programs or collecting trend data. (Recommendation 8) We provided a draft of this product for comment to OMB, OPM, and the five selected agencies. OMB neither agreed nor disagreed with the recommendations and stated that it would take them into consideration when making future updates to its policies and guidance for agencies for improving program and service delivery. In addition, OMB, OPM, Commerce, NASA, Treasury, and Veterans Affairs provided technical comments which we incorporated as appropriate. Energy responded that it had no comments. We are sending copies of this report to congressional committees, the Acting Director of OMB and Director of OPM, The Secretaries of the Departments of Commerce, Energy, Treasury, and Veterans Affairs, the Administrator of NASA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or Jonesy@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. This report examines: (1) the steps taken by the Office of Management and Budget (OMB), the Office of Personnel Management (OPM), and the Chief Financial Officer Act of 1990 (CFO Act) agencies to implement the Program Management Improvement Accountability Act (PMIAA); (2) the extent to which OMB is using or planning to use portfolio reviews required in PMIAA to address issues on our High-Risk List; and (3) the extent to which OMB provided methods for agencies to assess the results of PMIAA. To examine the steps taken by OMB, OPM, and CFO Act agencies to implement PMIAA, we reviewed agency documents, designed and disseminated a questionnaire to the 24 CFO Act agencies, and analyzed their responses. We also selected five PMIAA CFO Act agencies as case studies. We reviewed documentation from OMB, including the OMB PMIAA strategic plan and actions taken, as well as Cross Agency Priority goal 11 quarterly reports, and screen shots of PMIAA documents on OMB Max portal. We interviewed OMB staff to gain insight into their approach to implementing PMIAA. To examine the OMB standards for program and project management, we used criteria from the Project Management Institute (PMI) for Standard for Program Management and the Project Management Body of Knowledge. In addition, we reviewed documentation from OPM regarding their PMIAA plans and documents for the update of the 0340 job series. We further analyzed Enterprise Human Resources Integration (EHRI) data from fiscal year 2018 from OPM to identify employees in current program management 0340 occupational series. We also interviewed OPM officials regarding their role in implementing PMIAA. We interviewed outside subject matter specialists to provide their views on federal program and project management. Specifically, we met with staff from PMI and Professor Janet Weiss from the University of Michigan—who had conducted a study on how to improve federal program management—as she had been recommended by the Congressional Research Service, OMB, and the IBM Center for the Business of Government. To examine the steps agencies had taken, we requested PMIAA implementation plans from all 24 CFO Act agencies. CFO Act agencies were to submit PMIAA implementation plans to OMB by November 30, 2018. We collected implementation plans between November 29, 2018, and April 16, 2019. We received 22 out of 24 implementation plans. We did not review plans from the Department of Health and Human Services or the Environmental Protection Agency because they had not completed their plans at the time of our review. Two analysts independently reviewed separate implementation plans. These reviews were then verified by another analyst. Implementation plans were evaluated on whether they fully met, partially met, or did not meet the 10 requirements provided in the OMB implementation guidance, such as how the major acquisition portfolios aligned to relevant strategic objectives, or whether the agency had existing training for program and project managers. We also disseminated a questionnaire to all CFO Act agencies to collect information on PMIAA implementation. This questionnaire was pre-tested by two CFO Act agencies and two members of the Federal Program and Project Management Community of Practice and revised for clarity. The questionnaire was sent to all 24 CFO Act agencies on February 4, 2019, and responses collected between February 11 and April 22, 2019. All 24 agencies responded to the questionnaire. Agency officials were asked questions on: 1. the steps their agency has taken to implement PMIAA, 2. the challenges their agency faces in implementing PMIAA, 3. efforts to address high-risk issues, and 4. plans to measure PMIAA outcomes, if any. We selected five agencies for case studies and analyzed further documentation and interviewed agency officials to provide illustrative examples of PMIAA implementation at the agency level. We assessed whether: agencies had responsibility for a program, function, or operation on our 2019 High-Risk List; OMB considered them further along in PMIAA implementation compared to other agencies; the agency reported it was selected for the OMB pilot of noninformation technology acquisition program portfolio reviews; agency officials reported actions taken to direct internal program management training or workforce development in their questionnaire responses or OMB required implementation plans; and agency officials reported any actions to implement PMIAA beyond the requirements listed in the OMB PMIAA strategic plan. To achieve of a range of PMIAA experiences, we selected five agencies that met varying numbers of the criteria. The Department of Commerce was chosen because all four selection criteria were met, the Department of Energy met three, the Department of Veterans Affairs met two, and the Department of the Treasury and the National Aeronautics and Space Administration each met one. We interviewed and reviewed documents from each of the agencies. We asked questions about steps agencies were taking and their interactions with OMB and OPM to help them implement PMIAA. We also asked these agencies to suggest any ways in which OMB and OPM could improve implementation. To assess the OMB PMIAA strategic plan, we reviewed leading practices on strategic planning from our body of work. We also considered testimonial evidence from OMB staff. Specifically, we reviewed prior reports on leading strategic planning practices and requirements for agencies to use in strategic planning. We selected relevant criteria from the Government Performance and Results Act of 1993 (GPRA) and the GPRA Modernization Act, that not only pertained to agency strategic plans, but also were relevant as for strategic planning principles. Specifically, we selected criteria from the following categories: (1) mission statement; (2) general goals and objectives; (3) strategies for accomplishing goals and objectives; (4) input from stakeholders; (5) interagency collaboration; 6) milestones and metrics to gauge progress. To determine the extent to which the leading practice was included in the strategic plan, we assessed documentary evidence from the PMIAA strategic plan and testimonial evidence from OMB staff as defined below: A practice was categorized as fully met if the evidence fulfilled all aspects of the definition. A practice was categorized as partially met if the evidence fulfilled some, but not all, aspects of the definition, or if the evidence was judged to fulfill the general meaning of the definition, while not technically meeting it fully. A practice was categorized as not met if no evidence was found relevant to the criterion, or if evidence did not fulfill any aspects of the definition. In addition, we reviewed documents from and interviewed selected agencies on what measures OMB was developing for evaluating PMIAA implementation. We also asked these agency officials what kinds of evaluative measures would be useful to monitor the successful implementation of PMIAA from their perspective. In addition, we assessed the pilot of the required PMIAA program portfolio reviews against the five leading practices we identified from our work on designing pilots. We determined that the design fully met the criteria when we saw evidence that all aspects of a leading practice were met. When we were unable to assess whether all aspects of a leading practice were met without additional information, we determined that the design partially met the criteria. Finally, when we saw no evidence of a leading practice, we determined that the criteria were not met. To examine OMB’s standards for program and project management, we selected two sets of criteria for program and project management criteria from PMI. PMI standards are generally recognized as leading practices for program and project management. To select program management standards, we identified 10 PMI program management activities. To select project management standards, we identified 10 project management knowledge areas. Further, PMI’s leading practices were selected to explain how program and project management standards apply differently, and how both set of standards relate to the lifecycle of a program or project. We then compared the definition of these 10 PMI program and 10 PMI project management standards to the definition of OMB’s initial 15 program and project standards released for PMIAA implementation. In addition, OMB’s initial standards were compared to PMI leading practices that distinguish the relationship between programs and projects and leading practices on applying standards across the life cycle of a program or project. We also applied leading practices we identified from our previous work on data governance standards to assess the governance process OMB used to develop, maintain, and monitor program management standards. Our past work identified common key practices for establishing effective data governance structures. This work selected a range of organizations, including domestic and international standards-setting organizations, industry groups or associations, and federal agencies, to ensure we had comprehensive perspectives of data governance key practices across several domains. Two analysts compared the five key practices on the data governance structures to OMB plans and documented practices. We assessed the reliability of OPM’s EHRI data through electronic testing to identify missing data, out of range values, and logical inconsistencies for employees classified as 0340s. We believe the EHRI data we used are sufficiently reliable for the purpose of this report. To examine the extent to which OMB is using or planning to use portfolio reviews to address our High-Risk-List, we reviewed documentation from OMB and 24 CFO Act Agencies. As part of our questionnaire, we asked 24 CFO Act agencies to provide any of our High-Risk List summary and detailed analyses that the agencies were required to submit to OMB as part of the 2018 strategic review process. We analyzed this information to determine the extent to which agencies provided information to OMB during their 2018 strategic review process. We also selected criteria from the Standards for Internal Control in the Federal Government on maintaining documentation of the internal control system to assess steps that OMB had taken related to its responsibilities for conducting high-risk portfolio reviews and the management of the Program Management Policy Council. Specifically, we selected information and communication which states that management should externally communicate the necessary quality information that an entity needs to achieve its objectives. We conducted this performance audit from June 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Yvonne D. Jones, (202) 512-6806, or jonesy@gao.gov. In addition to the contact named above, William Reinsberg (Assistant Director), Carole J. Cimitile (Analyst in Charge), Jacqueline Chapin, Martin J. De Alteriis, Emily Gamelin, Jaeyung Kim, Matthew L. McKnight, Robert Robinson, Dylan Stagner, Andrew J. Stephens, and John Villecco made key contributions to this report.", "summary": "PMIAA requires OMB to adopt program management standards and guidelines government-wide; OPM is to establish new—or revise existing—occupational standards for program and project management. PMIAA includes a provision for GAO, no later than 3 years after the enactment of the act, to issue a report examining the implementation and effectiveness of certain provisions of the act on federal program and project management. This report (1) describes steps taken by OMB, OPM, and agencies to implement PMIAA; (2) assesses OMB's efforts to address issues on GAO's High-Risk List using PMIAA; and (3) examines the extent to which OMB provided methods for agencies to measure and assess the results of PMIAA. GAO reviewed documents from and conducted interviews with OMB and OPM. GAO surveyed all 24 CFO Act agencies, and selected five agencies to illustrate implementation efforts. GAO also interviewed subject matter specialists from academia and the private sector regarding their views on how program and project management practices applied to PMIAA. The Office of Management and Budget (OMB) has begun to implement all requirements of the Program Management Improvement Accountabilitiy Act of 2016 (PMIAA), but further efforts are needed to fully implement the law. OMB released its 5-year strategic plan for PMIAA and developed program management standards. However, the standards are not detailed compared with accepted program and project management standards, and OMB's governance structure is insufficient for developing and maintaining these standards over time. In 2019, OMB conducted ten reviews of agency program portfolios—organized groupings of programs whose coordination in implementation enables agencies to achieve their objectives. Each review addressed one or two portfolios per agency. Further, OMB's required portfolio reviews of high-risk areas were limited to only five out of 35 areas on GAO's High-Risk List. OMB could establish measures to track agencies' progress. Although not required by PMIAA, this is a good practice for demonstrating improvement. As required by PMIAA, the Office of Personnel Management (OPM) developed competencies for program and project managers and updated the program management job series. Further, OPM is developing a career path for program and project managers by the end of 2019. OPM also plans to create a unique job identifier code in 2020 so that agencies can more completely identify their program management workforce. The Program Management Policy Council (PMPC), established by PMIAA and chaired by OMB's Deputy Director for Management, met for the first time in September 2018 and met twice in 2019 to discuss PMIAA implementation with Chief Financial Officers (CFO) Act agencies. All CFO Act agencies designated a Program Management Improvement Officer to participate in the PMPC. However, the PMPC has neither addressed GAO high-risk areas nor advised OMB on how to address high-risk areas, as required by the PMIAA. GAO is making eight recommendations that OMB further develop the standards to include more detail, create a governance structure for program management standards, hold meetings on all High-Risk List areas, and establish measures to track agencies' progress in program management. OMB neither agreed nor disagreed with the recommendations and stated that it would consider them when making future updates to its program management policies and guidance.", "document_type": "gao"}
{"report": "In our June 2019 report, we found that, while abuse deficiencies cited in nursing homes were relatively rare from 2013 through 2017, they became more frequent during that time, with the largest increase in severe cases. Specifically, abuse deficiencies comprised less than 1 percent of the total deficiencies in each of the years we examined, which is likely conservative. Abuse in nursing homes is often underreported by residents, family, staff, and the state survey agency, according to CMS officials and stakeholders we interviewed. However, abuse deficiencies more than doubled—from 430 in 2013 to 875 in 2017—over the 5-year period. (See appendix II.) In addition, abuse deficiencies cited in 2017 were more likely to be categorized at the highest levels of severity— deficiencies causing actual harm to residents or putting residents in immediate jeopardy—than they were in 2013. In light of the increased number and severity of abuse deficiencies, it is imperative that CMS have strong nursing home oversight in place to protect residents from abuse; however, we found oversight gaps that may limit the agency’s ability to do so. Specifically, we found that CMS: (1) cannot readily access data on the type of abuse or type of perpetrator, (2) has not provided guidance on what information nursing homes should include in facility-reported incidents, and (3) has numerous gaps in its referral process that can result in delayed and missed referrals to law enforcement. In our June 2019 report, we found that CMS’s data do not allow for the type of abuse or perpetrator to be readily identified by the agency. Specifically, CMS does not require the state survey agencies to record abuse and perpetrator type and, when this information is recorded, it cannot be easily analyzed by CMS. Therefore, we reviewed a representative sample of 400 CMS narrative descriptions—written by state surveyors—associated with abuse deficiencies cited in 2016 and 2017 to identify the most common types of abuse and perpetrators. From this review, we found that physical abuse (46 percent) and mental/verbal abuse (44 percent) occurred most often in nursing homes, followed by sexual abuse (18 percent). Furthermore, staff, which includes those working in any part of the nursing home, were more often the perpetrators (58 percent) of abuse in deficiency narratives, followed by resident perpetrators (30 percent) and other types of perpetrators (2 percent). (See appendix III for examples from our abuse deficiency narrative review.) CMS officials told us they have not conducted a systematic review to gather information on abuse and perpetrator type. Further, based on professional experience, literature, and ad hoc analyses of deficiency narrative descriptions, CMS officials told us they believe the majority of abuse is committed by nursing home residents and that physical and sexual abuse were the most common types. This understanding does not align with our findings on the most common types of abuse and perpetrators represented in CMS’s data on deficiencies cited as abuse. Without the systematic collection and monitoring of specific abuse and perpetrator data, CMS lacks key information and, therefore, cannot take actions—such as tailoring prevention and investigation activities—to address the most prevalent types of abuse or perpetrators. To address this, we recommended that CMS require state survey agencies to report abuse and perpetrator type in CMS’s databases for deficiency, complaint, and facility-reported incident data and that CMS systematically assess trends in these data. HHS concurred with our recommendation and stated that it plans to implement changes in response. As of November 2019, HHS had not implemented the recommendation. Despite federal law requiring nursing homes to self-report allegations of abuse and covered individuals to report reasonable suspicions of crimes against residents, in June 2019 we reported that CMS had not provided guidance to nursing homes on what information they should include in facility-reported incidents, contributing to a lack of information for state survey agencies and delays in their investigations. Specifically, officials from each of the five state survey agencies told us that the documentation they receive from nursing homes for facility-reported incidents can lack key information that affects their ability to triage incidents and determine whether an investigation should occur and, if so, how soon. For example, officials from two state survey agencies we interviewed said they sometimes have to conduct significant follow-up with the nursing homes to obtain the information they need to prioritize the incident for investigation—follow-up that delays and potentially negatively affects investigations. Incomplete incident reports from nursing homes are particularly problematic given that nearly half of abuse deficiencies cited between 2013 and 2017 were identified through facility-reported incidents, which is dramatically different than the approximately 5 percent of all types of deficiencies that were identified in this manner. Therefore, facility-reported incidents play a unique and significant role in identifying abuse deficiencies in nursing homes, making it critical that incident reports provided by nursing homes include the information necessary for state survey agencies to prioritize and investigate. To address this issue, we recommended that CMS develop and disseminate guidance— including a standardized form—to all state survey agencies on the information nursing homes and covered individuals should include on facility-reported incidents. HHS concurred with our recommendation and stated that it plans to implement changes in response. As of November 2019, HHS had not implemented the recommendation. In June 2019, we identified gaps in CMS’s process for referring incidents of abuse to law enforcement and, if appropriate, to MFCUs. These gaps may limit CMS’s ability to ensure that nursing homes meet federal requirements for residents to be free from abuse. Specifically, we identified issues related to (1) referring abuse to law enforcement in a timely manner, (2) tracking abuse referrals, (3) defining what it means to substantiate an allegation of abuse—that is, the determination by the state survey agency that evidence supports the abuse allegation, and (4) sharing information with law enforcement. We made recommendations that CMS address each of these four gaps in the referral process, and HHS concurred with each recommendation and stated that it plans to implement changes in response. As of November 2019, HHS had not implemented these recommendations. One of the gaps in CMS’s process is related to referring abuse to law enforcement in a timely manner. For example, law enforcement investigations can be significantly delayed because CMS requires a state survey agency to make referrals to law enforcement only after abuse is substantiated—a process that can often take weeks or months. Officials from one law enforcement agency and two MFCUs we interviewed told us the delay in receiving referrals limits their ability to collect evidence and prosecute cases—for example, bedding associated with potential sexual abuse may have been washed, and a victim’s wounds may have healed. As such, we recommended that CMS require state survey agencies to immediately refer to law enforcement any reasonable suspicion of a crime against a resident. HHS concurred with our recommendation and stated that it plans to implement changes in response. As of November 2019, HHS had not implemented this recommendation. In conclusion, while nursing home abuse is relatively rare, our June 2019 report shows that abuse deficiencies cited in nursing homes are becoming more frequent, with the largest increase in severe cases. It is imperative that CMS have more complete and readily available information on abuse to improve its oversight of nursing homes. It is also essential that CMS require state survey agencies to immediately report incidents to law enforcement if they have a reasonable suspicion that a crime against a resident has occurred in order to ensure a prompt investigation of these incidents. Chairman Neal, Ranking Member Brady, and Members of the Committee, this concludes GAO’s statement for the record. For further information about this statement, please contact John E. Dicken at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contact named above, key contributors to this statement were Karin Wallestad (Assistant Director), Sarah-Lynn McGrath (Analyst-in-Charge) and Summar C. Corley. Also contributing to the underlying report for this statement were Luke Baron, Julianne Flowers, Laurie Pachter, Vikki Porter, Kathryn Richter, and Jennifer Whitworth. We have issued a number of reports reviewing the health and welfare of the elderly in multiple settings. For example, since January 2015, we have issued reports on the incidence of abuse in nursing homes and what is known about the incidence of abuse in assisted living facilities. Reports often included key recommendations. (See table 1.) Appendix II: Severity of Abuse Deficiencies Cited in Nursing Homes, 2013 through 2017 CMS restructured its deficiency code system beginning on November 28, 2017. Due to these coding changes, we did not analyze CMS data cited by surveyors after the implementation of that change. Percentages may not add to 100 due to rounding. Elder Abuse: Federal Requirements for Oversight in Nursing Homes and Assisted Living Facilities Differ. GAO-19-599. Washington, D.C.: August 19, 2019. Nursing Homes: Improved Oversight Needed to Better Protect Residents from Abuse. GAO-19-433. Washington, D.C.: June 13, 2019. Elder Justice: Goals and Outcome Measures Would Provide DOJ with Clear Direction and a Means to Assess Its Efforts. GAO-19-365. Washington, D.C.: June 7, 2019. Management Report: CMS Needs to Address Gaps in Federal Oversight of Nursing Home Abuse Investigations That Persisted in Oregon for at Least 15 Years. GAO-19-313R. Washington, D.C.: April 15, 2019. Medicaid Assisted Living Services: Improved Federal Oversight of Beneficiary Health and Welfare is Needed. GAO-18-179. Washington, D.C.: January 5, 2018. Medicaid Managed Care: CMS Should Improve Oversight of Access and Quality in States’ Long-Term Services and Supports Programs. GAO-17- 632. Washington, D.C.: August 14, 2017. Medicaid Personal Care Services: CMS Could Do More to Harmonize Requirements across Programs. GAO-17-28. Washington, D.C.: November 23, 2016. Nursing Homes: Consumers Could Benefit from Improvements to the Nursing Home Compare Website and Five-Star Quality Rating System. GAO-17-61. Washington, D.C.: November 18, 2016. Elder Abuse: The Extent of Abuse by Guardians is Unknown, but Some Measures Exist to Help Protect Older Adults. GAO-17-33. Washington, D.C.: November 16, 2016. Skilled Nursing Facilities: CMS Should Improve Accessibility and Reliability of Expenditure Data. GAO-16-700. Washington, D.C.: September 7, 2016. Nursing Home Quality: CMS Should Continue to Improve Data and Oversight. GAO-16-33. Washington, D.C.: October 30, 2015. Antipsychotic Drug Use: HHS Has Initiatives to Reduce Use among Older Adults in Nursing Homes, but Should Expand Efforts to Other Settings. GAO-15-211. Washington, D.C.: January 30, 2015. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Nationwide, about 1.4 million elderly or disabled individuals receive care in more than 15,500 nursing homes. CMS, an agency within the Department of Health and Human Services (HHS), defines standards nursing homes must meet to participate in the Medicare and Medicaid programs. Nursing home residents often have physical or cognitive limitations that can leave them particularly vulnerable to abuse. Abuse of nursing home residents can occur in many forms—including physical, mental, verbal, and sexual—and can be committed by staff, residents, or others in the nursing home. Any incident of abuse is a serious occurrence and can result in potentially devastating consequences for residents, including lasting mental anguish, serious injury, or death. This statement summarizes GAO's June 2019 report, GAO-19-433 . Specifically, it describes: (1) the trends and types of abuse in recent years, and (2) CMS's oversight intended to ensure residents are free from abuse. It also includes a brief summary of findings and recommendations from this June 2019 report and prior GAO reports that examined the health and welfare of the elderly in multiple settings, and the status, as of November 2019, of HHS's efforts to implement the recommendations GAO made. The Centers for Medicare & Medicaid Services (CMS) is responsible for ensuring nursing homes meet federal quality standards, including that residents are free from abuse. CMS enters into agreements with state survey agencies to conduct surveys of the state's homes and to investigate complaints and incidents. GAO's June 2019 report found that, while abuse deficiencies cited in nursing homes were relatively rare from 2013 through 2017, they more than doubled during that time, increasing from 430 in 2013 to 875 in 2017, with the largest increase in severe cases. In light of the increased number and severity of abuse deficiencies, it is imperative that CMS have strong nursing home oversight in place to protect residents from abuse. However, GAO found oversight gaps that may limit the agency's ability to do so. Specifically, GAO found: (1) Information on abuse and perpetrator types is not readily available. CMS's data do not allow for the type of abuse or perpetrator to be readily identified by the agency. Specifically, CMS does not require the state survey agencies to record abuse and perpetrator type and, when this information is recorded, it cannot be easily analyzed by CMS. GAO made a recommendation that CMS require state survey agencies to submit data on abuse and perpetrator type and HHS concurred. As of November 2019, HHS had not implemented the recommendation. (2) Facility-reported incidents lack key information. Despite federal law requiring nursing homes to self-report allegations of abuse and covered individuals to report reasonable suspicions of crimes against residents, CMS has not provided guidance to nursing homes on what information they should include in facility-reported incidents, contributing to a lack of information for state survey agencies and delays in their investigations. GAO made a recommendation that CMS develop guidance on what abuse information nursing homes should self-report and HHS concurred. As of November 2019, HHS had not implemented the recommendation. (3) Gaps exist in the CMS process for state survey agency referrals to law enforcement. GAO found gaps in CMS's process for referring incidents of abuse to law enforcement. These gaps may limit CMS's ability to ensure that nursing homes meet federal requirements for residents to be free from abuse. Specifically, GAO identified issues related to (1) referring abuse to law enforcement in a timely manner, (2) tracking abuse referrals, (3) defining what it means to substantiate an allegation of abuse—that is, the determination by the state survey agency that evidence supports the abuse allegation, and (4) sharing information with law enforcement. GAO made four recommendations to address these gaps and HHS concurred. As of November 2019, HHS had not implemented these recommendations.", "document_type": "gao"}
{"report": "VA pays monthly disability compensation to veterans with disabling conditions caused or aggravated by their military service. The benefit is based on an average reduction in earning capacity across a group of individuals with similar physical or mental impairments. Disability compensation is generally paid according to the severity of the service- connected condition and is awarded in 10 percent increments, based on criteria in the VA Schedule for Rating Disabilities (VASRD or rating schedule). Veterans may claim more than one medical condition, and VBA assigns a rating percentage for each condition determined to be connected to the veteran’s service. For veterans with multiple service-connected conditions, VA calculates a rating (combined disability rating) using a table that applies a formula for combining multiple ratings into a single rating. The rating affects the amount of monthly compensation received by a veteran. Unlike some private-sector disability programs, the employment status, earnings, and ability to work generally are not factored into the disability rating and subsequent base payment. Moreover, unlike typical workers’ compensation programs for permanent impairments, no limits are generally placed on the length of time veterans can receive payments. Obligations for disability compensation have increased by 45 percent in the last 5 years, from about $54 billion in fiscal year 2013 to about $78 billion in fiscal year 2018. According to VA, this increase is due to several factors, including more beneficiaries (for example, as veterans of more recent conflicts leave military service and seek compensation), as well as rising average disability ratings that lead to higher average payments. VA reported that growth in the number of veterans with a service-connected condition is concentrated among those rated 50 percent or higher. VBA’s Compensation Service sets policy and oversees the process for determining eligibility for disability compensation. VBA staff in the regional offices process disability compensation claims. These claims processors include Rating Veterans Service Representatives (RVSR or rater), who decide on benefit entitlement and the rating percentage, and Veterans Service Representatives (VSR), who gather evidence needed for the raters to make their decisions and later authorize payment, if any. Claims processors use the Veterans Benefits Management System (VBMS)—an electronic, paperless system—to maintain, review, and make rating decisions for veterans’ claims. VBA’s reevaluation process determines whether veterans’ service- connected conditions may have changed, due to treatment or other factors, in the years following an initial evaluation for disability compensation. This process helps ensure that veterans’ service- connected conditions are being rated and compensated correctly. A first step in the process is deciding whether a condition may need to be reevaluated at a future date. As part of an evaluation for disability compensation, claims processors review medical evidence and consider whether to schedule a future review date (see fig.1). When the scheduled review date arrives, VBA revisits the case to determine whether a reevaluation of the disabling condition is still appropriate. This pre-exam review involves reviewing the veteran’s records to determine if the veteran is still experiencing similar symptoms. After this review, VBA may conduct, postpone, or cancel a reevaluation. If the reevaluation is conducted, a medical exam may be ordered, after which the rater will rate the condition based on exam results and other medical evidence. VBA regulations specify certain conditions that require reevaluation. In other instances, VA has discretion in whether to conduct reevaluations, determined upon review of a veteran’s medical record. For example, the medical record may suggest that a veteran with limited range of motion will be continuing physical rehabilitation and is expected to improve. Whether the reevaluation is required or discretionary, VBA’s regulations outline several exclusions that place limits on when VBA conducts reevaluations, such as if the veteran’s combined disability rating would not change as a result of a reduced evaluation for one or more conditions. Veterans may generally obtain health care through (1) VA medical facilities, (2) non-VA health care providers in the community for which VA pays (called community care), or (3) providers paid through veterans’ own health insurance. For VA medical facilities, VHA determines eligibility and priority for VA health care, enrolls veterans, and oversees 172 VA medical centers and over 1,000 outpatient facilities. In response to the Veterans’ Health Care Eligibility Reform Act of 1996, VHA developed a priority system to balance demand for health care with available resources. The system has eight priority groups, and first priority is generally given to veterans with service-connected conditions rated 50 percent or more and to veterans deemed unemployable because of service-connected conditions. Priority groups 2 and 3 include veterans with service-connected conditions rated 30 or 40 percent, or 10 or 20 percent, respectively, according to VHA. Veterans may be eligible for community care if, for example, VA does not offer the care or service the veteran requires, or when a VA medical facility is unable to provide the care or services consistent with the agency’s access standards. Before receiving health care through VA community care programs, veterans must generally obtain authorization from VA. The total number of veterans enrolled in VA’s health care system rose from 7.9 million to over 9 million from fiscal years 2006 through 2017. During that period, VHA’s budget more than doubled, from $37.8 billion to $92.3 billion, as health care costs were rising and its community care programs were expanding. In fiscal year 2017, VA obligated $13.6 billion of its budget for community care, and in fiscal year 2018, this increased to $14.9 billion. For health care services delivered outside of VHA medical facilities that are not funded by VA, veterans may use private health insurance. A 2018 VA survey of veterans enrolled in VA’s health care system found that about 28 percent reported being covered by private insurance. VBA tracks its performance in providing timely and accurate disability compensation decisions to veterans. VBA considers a decision to be timely if a veteran’s claim is processed within 125 days. As part of its quality assurance efforts, VBA conducts national and individual reviews of the accuracy of claims decisions, and periodic consistency studies to assess claims processors’ knowledge of regulations and guidance on specific claims processing issues, such as when to conduct reevaluations. At the VBA central office level, procedures call for VA to assess the accuracy of a random sample of completed claims from each regional office using its Systematic Technical Accuracy Review (STAR) method. STAR reviewers use a standardized checklist to review all actions taken in processing a claim and record any errors they find. VA reports national and regional office performance data for claim- based accuracy (based on the entirety of the claim) and issue-based accuracy (based on each of the individual medical conditions rated). In fiscal year 2018, VBA reported claim-based rating accuracy of about 90 percent and issue-based accuracy of about 95 percent. At the VBA regional office level, quality review teams conduct Individual Quality Reviews of individual claims processors’ work. For example, VA’s procedures call for reviews to be performed on five claims for every rater per month. The reviews are used to help assess individual claims processors’ performance. In addition to accuracy reviews, VBA’s national quality assurance efforts include periodic consistency studies on specific claims processing issues. These studies are intended to assess how consistently claims processors are making decisions across all regional offices by testing select claims processors on their knowledge of VBA’s regulations and procedures. Improvements in a veteran’s service-connected conditions and improvements in a veteran’s health outcomes have important differences. Federal law requires disability compensation to be based upon an average reduction in earning capacity across a group of individuals with a similar physical or mental impairment. In addition, for certain service- connected conditions such as amputations, VA evaluates the condition based on loss or loss of function of a body part or system, without considering assistive devices or prosthetics. As such, some service- connected conditions, such as hearing loss, are generally not expected to improve for purposes of disability compensation. In contrast, according to VHA research, a veteran’s use of a hearing aid is an example of a successful health outcome because this assistive technology can treat the symptoms of hearing loss and increase the functioning of a person. For health care delivered at VA medical facilities, our analysis of fiscal year 2018 VA data shows that more than half of veterans receiving disability compensation used VA health care for a service-connected condition. Specifically, we determined that about 54 percent of veterans, or about 2.6 million, who received disability compensation had at least one VA outpatient visit related to a service-connected condition. Veterans with higher combined disability ratings had more outpatient visits related to their service-connected conditions, on average. (See fig. 2.) Veterans using VA health care for service-connected conditions had an average of four such conditions, and the median age was 63. For veterans with the most prevalent service-connected conditions, in fiscal year 2018 the average number of visits ranged from about 6 to 11 (see table 1). The highest average number of visits was for veterans with service-connected post traumatic stress disorder (PTSD) and diabetes. For the same year, veterans receiving disability compensation had an average of nearly eight outpatient health care visits for service-connected conditions. In fiscal year 2018, about 13 percent of VA inpatient hospital stays for veterans receiving disability compensation were to treat a service- connected condition; about 87 percent of the stays for this population were to treat non-service-connected conditions. Nearly 2.1 million, or about 44 percent of veterans receiving disability compensation, had no VA outpatient visits or inpatient stays for their service-connected conditions. These veterans may have received treatment paid for through private insurance, from community care, or received no treatment for their service-connected conditions in fiscal year 2018. Veterans who did not use VA health care had an average of about four service-connected conditions, and the median age was 57. For community care (VA-funded health care delivered by non-VA providers), we could not determine the extent to which veterans receiving VA disability compensation used these health providers for their service- connected conditions because this area is not a focus of analysis for the program, according to VHA Office of Community Care officials. These officials told us that, other than for emergency care claims, information on service-connected conditions is not used to process authorizations and payments for the program because program eligibility is based on other factors, such as the availability of needed services. Veterans also receive health care outside of VHA facilities that is not funded by VA, such as through their private health insurance, and the number and types of these services for service-connected conditions are largely unknown. According to a statutorily mandated study of the use of VA’s health care system, these data are limited. The authors of this study recommended that VA consider expanding data collection efforts. VA has since worked with the Department of Health and Human Services’ Agency for Healthcare Research and Quality to expand its data collection regarding veterans, including veterans receiving disability compensation, specifically regarding veterans’ use of non-VA care and coordinating such care with VA providers. Data from this effort will be available beginning in fiscal year 2020, according to Agency for Healthcare Research and Quality researchers conducting the study. Health outcomes of veterans with service-connected conditions who receive VA health care services are not well understood, as they have not been specifically studied outside of veterans receiving disability compensation for PTSD. Based on a review of peer-reviewed literature and interviews with VA health research officials, we identified two studies on the health outcomes of veterans, both of which specifically focused on health outcomes for veterans receiving disability compensation for PTSD. One study published in 2011 found that receiving disability compensation benefits for PTSD was associated with clinically meaningful reductions in PTSD symptoms and reductions in poverty and homelessness. Another study published in 2017 found that 10 percent of men and 20 percent of women who applied for disability compensation for PTSD had a persistent serious mental illness, and over time, consistently reported more severe PTSD symptoms and poorer functioning in comparison to other study participants without severe mental illness. The study authors noted that serious mental illness was more prevalent in this population than in the VA health care system overall. They concluded that more information is needed about the characteristics of those receiving disability compensation to better understand their challenges and long-term outcomes. VA’s Health Services Research and Development office sponsors research on health conditions common in the veteran population, such as traumatic brain injury and Gulf War Illness, among others. According to an official from this office, data used for these studies generally do not include veterans’ receipt of disability compensation or their specific service-connected conditions. Several health care researchers within VA and a VA official we spoke with cited various reasons for limited research on health outcomes for veterans with service-connected conditions. According to these officials, a key challenge is that VBA and VHA data do not use the same identifiers for medical conditions that are needed to link the two information sources. VA health care researchers acknowledged benefits to including veterans’ VBA disability codes in their studies to analyze health information for veterans with service-connected conditions. A 2007 report on the options for improving the disability program also noted that the use of common diagnostic categories would allow VA program managers and researchers to compare populations and trends that would help in program planning and in epidemiological and health services research. However, VBA’s diagnostic codes are unique and do not allow comparisons of trends in disabilities in populations served by VHA or the Department of Defense. According to a VA health care researcher and a VHA official, also contributing to these challenges are the lack of data use agreements, which could better facilitate linking VBA and VHA administrative data for VA to further study health outcomes for this population. For example, according to a VA researcher, linking these data sources could allow researchers to investigate causal relationships between disability compensation and veterans’ health outcomes. We previously reported that such agreements can specify which data can be accessed and for what purpose, the duration of access, and requirements for safeguarding the data and ensuring confidentiality. VBA officials said that while they routinely share data with VHA for operational purposes, obtaining access to VBA data for research purposes has special requirements and is more cumbersome. Agency health care data are stored in VHA’s Corporate Data Warehouse, while benefits data are stored in VBA’s data warehouse. Both VHA and VBA officials noted that their data contain sensitive information and that access is carefully monitored. VA’s fiscal year 2018-2024 Strategic Plan includes goals and objectives for data-driven decision making, which include having comprehensive data to identify and meet veterans’ needs, as well as to understand the outcomes VA provides veterans and focus VA’s improvement efforts. In addition, we have previously reported that agencies can enhance and sustain their collaborative efforts by defining common outcomes, leveraging resources, and establishing compatible policies, among others. These practices include articulating agreements in formal documents, which can strengthen the commitment to working collaboratively, as well as establishing compatible policies and other means (including compatible standards and data systems) to operate across agency boundaries. VA has begun to consider ways to analyze health care services received by veterans with service-connected conditions. VA’s Office of Enterprise Integration (OEI) is tasked with providing analysis to inform VA decision- making, as well as to align planning and implementation across VA programs and initiatives. According to an OEI official, it plans to convene subject matter experts from VBA and VHA to determine options and pilot strategies to link available data, but has not yet determined the scope, specific activities, or timeframes for this effort. Until VA develops and implements a plan to address challenges that have hindered analysis thus far and enhance collaboration between VBA and VHA with regard to such analysis, VA will not be positioned to understand the characteristics, needs, and health outcomes of veterans with service-connected conditions, which available research suggests may be different from other veterans. VBA uses some information on conditions identified as potentially needing reevaluations; however, it is not analyzing and using trend and outcome information from completed reevaluations to inform which service-connected conditions to reevaluate in the future. Reevaluations of veterans’ service-connected conditions can serve as a proxy to gauge change, including improvement, in health. VBA assesses changes in veterans’ disabling conditions from reevaluations it conducts for various reasons, including evidence of potential improvement or when required by the rating schedule. A reevaluation showing a change in a given condition may result in one of three possible outcomes: an increase, decrease, or no change in the veteran’s associated disability rating. VBA developed a report to help identify unnecessary reevaluations, which included information on veterans’ conditions that are initially flagged by raters for potentially needing reevaluations in the future. Developed in 2017, VBA’s report identified potential reevaluations deemed unnecessary per VA’s regulations. For example, regulations state that veterans older than 55 are generally exempt from reevaluation, according to the VA Office of Inspector General (OIG). As part of this process, potential reevaluations identified as unwarranted by VBA’s report would be cancelled before their scheduled review dates arrived. This report also includes information on specific conditions identified for potential reevaluation, including the subset of conditions required by regulation to be reevaluated. For example, according to the data generated by the report in June 2019, PTSD was the most common condition identified for potential reevaluation, and of the conditions requiring reevaluation, prostate cancer was most common. However, VBA officials explained that if the report were to find that any of the cases were for veterans older than 55, the reevaluation would be deemed unwarranted and the scheduled review date for considering reevaluation would be cancelled. According to VBA officials, using this report helped VBA identify and cancel about 70,000 potential reevaluations deemed unnecessary, saving about $29 million. VBA plans to run similar reports as needed to identify more reevaluations that could be cancelled, according to officials. Additionally, VBA officials said that they have data on the specific conditions for which medical exams are ordered as part of the reevaluation process. Ordering exams for reevaluations occurs after a condition identified for potential reevaluation has been reviewed and a decision has been made to proceed with a reevaluation. In particular, VBA’s Exam Management System tracks exams ordered, including exams for reevaluations, and provides information about the associated conditions. However, this system does not provide information on the outcome of a reevaluation decision based on the information from these exams. While VBA has some insight into conditions set to be reevaluated, management lacks information on completed reevaluations, including (1) trends and comparisons of certain reevaluated conditions and (2) rating outcomes of reevaluation decisions for individual service-connected conditions. Reevaluation trends. VBA officials told us that they analyze trends on the numbers of veterans who have had reevaluations. However, they said they do not analyze reevaluation data to identify trends on whether certain conditions are frequently or infrequently reevaluated, including for conditions requiring reevaluation under VBA regulation. Further, although VBA has a mechanism to identify potential reevaluations for veterans with conditions requiring them, it is not analyzing the broader universe of veterans with these conditions, according to VBA officials. Such information could determine the extent to which conditions are being identified for reevaluation as required as well as the outcomes or results of these reevaluations. This trend information could also help VBA determine whether claims processors are conducting reevaluations as needed or required. Reevaluation outcomes. VBA officials said that they do not analyze information on the outcomes of reevaluation decisions for individual conditions (i.e., whether a reevaluation resulted in an increase, decrease, or no change to the rating of a particular condition). According to our analysis of VBA data, reevaluations rarely result in changes to veterans’ combined ratings. Specifically, from fiscal years 2013 through 2018, about 95 percent of reevaluations resulted in no changes to combined ratings for veterans, with about 3 percent resulting in an increase and less than 1 percent resulting in a decrease. Combined ratings alone do not offer insight into what impact reevaluations may have on ratings for individual conditions, including which ones are improving as a result of treatment. Most veterans have multiple conditions that contribute to a combined disability rating. VA reported that in 2018 veterans receiving disability compensation had an average of about five service-connected conditions. For those receiving reevaluations, this circumstance means that although the rating of one condition may decrease as a result of a reevaluation, the rating of another condition may increase based on the claims processor’s review of the medical evidence. As a result, the combined rating may not decrease despite a decrease in the rating of an individual condition. A recent report examining reevaluations for veterans with PTSD had similar findings. In its review of a sample of veterans, the study found that these veterans rarely saw a reduction in their individual rating for PTSD. In cases where an individual rating was reduced, most saw no reduction in their overall combined rating due to the fact that they had other conditions whose ratings increased and thereby offset any reduction. According to VBA officials, the agency does not analyze data on trends in reevaluated conditions or the outcomes of reevaluation decisions for specific conditions because management has not expressed interest in doing so. Further, officials said that these data are not stored together in the database. Although analyzing these data and developing a report on types of conditions reevaluated and their outcomes is feasible, according to officials, doing so would require additional steps, including analyzing the text of rating decisions. According to VA regulation, reevaluations are intended to verify the continued existence or the current severity of a disability. Federal standards for internal control state that management should establish and operate monitoring activities to evaluate the results of activities and ensure that objectives are met with minimum wasted resources. Moreover, they state that management should design a process that uses the entity’s objectives and related risks to identify the information requirements needed to achieve the objectives and address risks. These standards also state that management should use quality information to achieve the entity’s objectives. Identifying the extent to which VBA is meeting these program objectives and effectively managing resources is difficult without analyzing information about the outcomes of reevaluations for specific conditions. Such analysis could also identify trends indicating conditions with little or no potential for a rating change or missed opportunities to target other conditions likely to change as a result of reevaluations. In recent years, VBA has focused its procedures on reducing the number of unnecessary reevaluations and generally limiting the number of reevaluations conducted overall. Using outcome information could allow the agency to better target the agency’s resources and avoid the risk of unnecessary reevaluations and burdening veterans. Analyzing reevaluation trends and outcomes could also inform existing VBA policy. For example, VA is updating the rating schedule with current medical and earnings loss information, including adding conditions requiring reevaluations. Analyzing information on which conditions are reevaluated and identifying any trends in conditions that improve could help inform future updates to the rating schedule or improve the policies or practices for how the reevaluation process is implemented. VBA uses information to help gauge the timeliness and quality of reevaluation decisions, but has not fully used information related to the consistency of raters’ decisions to address potential training needs, among other issues. VBA tracks its performance in providing veterans with timely and accurate decisions on their disability compensation benefits, and uses such information—including information on reevaluations—to manage the claims process. VBA holds its claims processing staff accountable for their timeliness and accuracy through performance standards for regional office managers and individual claims processors. Timeliness. VBA measures and reports to Congress and the public its total number of claims awaiting completion, including those that have been backlogged (awaiting completion for more than 125 days). According to VA, at of the end of fiscal year 2018 it had about 364,000 disability compensation rating claims awaiting completion. Of this total, about 19,000 were reevaluations, of which fewer than 5 percent were in the backlog. VBA uses additional timeliness measures to hold regional offices accountable by tracking the timeliness of their work in each of five steps or cycles in the claims process, as managed under the National Work Queue (VBA’s system for distributing the claims workload). For example, in fiscal year 2018, preparing a rating decision for a reevaluation took an average of 1.76 days. Quality. VA uses national, regional office, and individual-level data from its accuracy reviews to oversee the quality of rating claims decisions, including reevaluations. Each regional office is to meet the national STAR issue-based target of 96 percent accuracy for the year. For reevaluations, VBA reported both claim-based and issue-based accuracy of about 95 percent for fiscal year 2018. According to VBA officials, in response to a recommendation in the VA OIG’s report on unwarranted reevaluations, in October 2018 VBA updated the STAR national quality review checklist with additional questions on (1) the need for a reevaluation, and (2) the timeframe for future reevaluation. At the individual claims processor level, VA measures accuracy using the results of Individual Quality Reviews as part of claims processors’ performance evaluations. For example, a rater is considered fully successful by achieving 92 to 96 percent accuracy on Individual Quality Reviews for a month, depending on the rater’s experience. In fiscal year 2018, VBA reported that for Individual Quality Reviews, claims processors had a 98.4 percent accuracy rate for reevaluations. Overall, few reevaluations are reviewed because reevaluations are a small proportion of VA’s claims workload. Specifically, of about 102,000 reevaluations completed in fiscal year 2018, about 1,500 were reviewed under STAR and about 10,000 were reviewed in Individual Quality Reviews. In addition to using accuracy information to measure regional office and individual performance, VBA holds regional offices and individual claims processors responsible for correcting their errors. According to VBA officials, the agency uses information from its quality reviews to provide additional guidance and training to regional offices. VBA discusses quality review information, including trends in claims processing errors, through newsletters and periodic conference calls with regional office managers and quality review teams. For example, VBA officials noted that they discussed reevaluation policies and guidance with regional office staff on three occasions between May 2017 and May 2018. Officials at the four regional offices we visited indicated that they disseminated information on reevaluations to claims processors. For example, one office’s quality review team provided additional training on reevaluations to members of the claims processing teams. Quality review team officials in each of the regional offices we visited told us that they disseminate and reinforce guidance to claims processors through periodic meetings, newsletters, or other mechanisms. VBA, however, has not fully used available information about quality to oversee and improve the reevaluation process. Specifically, VBA did not use the results of a study it conducted to further identify and correct gaps in raters’ knowledge of reevaluation processing guidance. This May 2018 study—part of VBA’s quality assurance efforts that include periodic consistency reviews of specific claims processing issues—assessed how consistently raters across regional offices understood VBA’s policies on ordering reevaluations (see table 2). The study team recommended VBA take two actions: 1. Consider having experienced quality review team staff at regional offices provide additional training on reevaluation guidance to raters. 2. Consider reviewing reevaluation decisions at the seven lowest-scoring offices because they were at high risk of inaccuracies. While VBA provided regional offices with results of the May 2018 consistency study, the agency did not implement either recommendation. VBA officials told us that they did not direct regional offices to provide additional training because the agency expected the offices to use the results of the consistency study to plan training on reevaluations for their staff. However, VBA officials told us that not all regional offices provided additional training on reevaluations. Quality review officials at the four offices we visited—which included two of the seven offices the study team identified for further review—told us that they did not provide additional training. Officials at two offices said they had previously provided guidance and training to claims processors on reevaluations. VA’s goals are to ensure timely and accurate claims decisions for veterans. Federal standards for internal control state that management should establish monitoring activities, evaluate the results, and remediate any deficiencies on a timely basis. Consistent with these standards, GAO has previously reported that a key use of performance information is to identify problems and take corrective actions, for example, by changing agency guidance or by providing training. By not implementing the study’s recommendations, VBA is missing an opportunity to identify problems and their root causes as a guide to corrective actions, including training or the improvement of training. Many raters who are trained to make these decisions did not perform well on the consistency study’s initial test. Exploring deficiencies associated with this poor performance could position VA to better manage the reevaluation process. In addition, resources spent in developing the study and analyzing its results were not used as effectively as they could have been. VBA has recently updated its procedures manual to clarify who can determine whether a reevaluation is needed, but has not outlined guidance for the knowledge, skills, and abilities needed to perform these tasks. As part of the reevaluation process to assess veterans’ conditions, VBA procedures require claims processors in regional offices to conduct a pre-exam review to determine whether a reevaluation is still appropriate when its scheduled review date arrives (see fig. 3). For the reevaluation process to work effectively, proper procedures must be in place to ensure that claims processors can make informed decisions on whether to reevaluate these conditions. Until its recent update, VBA’s procedures manual stated that staff deemed part of “the rating activity” (defined in the manual as staff including raters who specialize in rating claims) were the only claims processors who were permitted to conduct a pre-exam review to determine whether a reevaluation is warranted. In February 2019, VBA updated its procedures manual to clarify that raters or “locally designated claims processors” may conduct this review. Officials said that Veterans Service Representatives (VSR) may fill this role in some offices. Although VBA’s procedures permit VSRs to conduct pre-exam reviews, VSRs may not be qualified to do so, according to the OIG’s July 2018 report and VBA regional staff we interviewed in 2019. The OIG found that VSRs were ordering exams without raters’ pre-exam reviews, resulting in an estimated 15,500 unwarranted exams (about 29 percent of the cases from the study’s review period). These exams were determined to be unwarranted based primarily on exclusions identified in VA’s procedures that exempt certain veterans from reevaluation (see text box). The report found that, rather than sending claims to raters for pre-exam review, VSRs were ordering exams despite not having the proper training and experience to decide on whether a reevaluation was warranted, such as the specialized knowledge needed to review medical evidence. Officials in regional offices we visited expressed concern about VSRs performing this role. Specifically, staff in three of the four regional offices we spoke with—including raters, supervisors, quality assurance staff, and managers—told us that raters do the pre-exam review in their respective offices because they are the only staff qualified to perform this duty. For example, raters have more experience and training than VSRs in reviewing medical evidence to determine the need for a reevaluation, according to officials from one office. In contrast, supervisors we spoke with at another regional office told us they have opted to have VSRs do the pre-exam review as a way to manage the claims workload and enable raters to focus exclusively on rating claims. However, these supervisors expressed concern that VSR reviews could have a negative impact on quality. VBA officials said they have not outlined guidance for the skills needed to perform the pre-exam review. Rather, VBA officials said that they believe it is most effective to allow the regional offices, which vary widely in size and scope, to have discretion to identify staff to fill this role. Further, VBA officials told us that the recent update to the agency’s procedures did not reflect a policy change broadening which staff can do pre-exam reviews, but rather clarified existing practice under which VSRs were already permitted to perform this task. However, given the OIG findings that VSRs performing this task resulted in many unwarranted exams, defining the knowledge, skills, and abilities needed for the pre-exam review could provide assurance that staff who do so are qualified. Federal standards for internal control call for management to clearly assign responsibilities and document internal controls, including who should carry out which roles. Identifying the knowledge, skills and abilities needed by qualified staff to carry out their responsibilities can also help management ensure the entity’s objectives are met. Providing flexibility for regional offices can ease implementation and management of workloads, especially for offices with varied situations. However, providing flexibility does not preclude VBA from outlining the basic knowledge, skills, and abilities required to perform the pre-exam review. Further, in our prior work we found that VBA has faced challenges in defining roles for its staff, which has led to inconsistencies in the way regional offices operate. We have also found that ambiguous policies provided by other VA programs can pose risks to the quality of the process. Without clarifying in VBA’s procedures manual which knowledge, skills, and abilities are needed to fill roles in the reevaluation process, VBA may be at risk of having unqualified staff continue to order unwarranted reevaluations. This risk, in turn, could result in wasted resources and an undue burden on veterans. Despite recent changes to its procedures manual, VBA has not ensured that its training program reflects the knowledge, skills, and abilities needed for relevant staff to conduct pre-exam reviews. VBA oversees national training requirements, including training related to reevaluations, but defers to regional offices to manage other training needs. As entry- level staff, claims processors receive national training from VBA related to their job duties. For raters, this initial training covers reevaluations, including instruction on when and when not to schedule reevaluations, and case studies exploring how to make reevaluation decisions based on medical and other evidence, among other topics. VSRs may also receive general training on reviewing and evaluating evidence and are introduced to reevaluations as they learn about general claim development and ordering exams. In addition to initial training, claims processors must complete 40 hours of training per year consisting of 15 hours of training mandated by VBA and 25 hours determined by each regional office. VBA officials told us that regional offices vary in what training and when delivered to their staff. In addition to this general training, VBA officials told us that VA added controls to the Veterans Benefits Management System (VBMS) system to restrict claims processors’ ability to schedule potential reevaluations, which could reduce the possibility of unqualified staff ordering unwarranted exams during the pre-exam review. Specifically, these controls prevent claims processors from scheduling review dates for potential reevaluations when certain exclusions apply (such as that outlined in VA regulation exempting from reevaluation veterans with the minimum rating for a given condition). Further, claims processors have the ability to request to override the restrictions when they believe a reevaluation is warranted based on the circumstances of the case. These override requests are reviewed by quality assurance staff, who may approve or deny the requests. Although these controls may impose some limits on ordering unwarranted exams, they may not affect the ability of claims processors to order reevaluations in circumstances where these exclusions do not apply and for which they must use their discretion. For example, for veterans who have migraine headaches and who do not fit any of the exclusion criteria, no VBMS controls would restrict claims processors from ordering a reevaluation even if it is not appropriate based on the medical evidence or other circumstances of the case. For these controls and VBA’s procedures to be effective, providing proper training to claims processors making these decisions remains important. VBA officials told us that they did not update training requirements as a result of the recent update to procedures because this update did not constitute a policy change. Rather, they said they revised the procedures to align with the existing practice before the update, in which VSRs were permitted to do pre-exam reviews. Further, officials said that each regional office can designate qualified claims processors to perform the pre-exam review and provide training as necessary. VBA officials also said that they do not believe additional training is necessary for VSRs who may be performing this role because the procedure for ordering exams—a skill for which they have been trained—is the same for all types of exams, including those for reevaluations. Although VSRs receive training on the process of ordering an exam, VBA officials confirmed that VSR coursework does not specifically cover the pre-exam review in the reevaluation process. In contrast, raters receive training on the process of deciding whether a reevaluation is warranted, including reviewing medical evidence and applying exclusions in VBA’s procedures. Further, staff in three of the four regional offices we spoke with, including supervisors, quality assurance staff, and managers, said that VSRs do not have the proper training for this task. For example, they are not trained to review medical evidence to make an informed decision about whether a reevaluation is still warranted, according to officials. Similarly, the OIG found in its 2018 report, which reviewed a sample of claims from March through August 2017, that VSRs were unfamiliar with criteria used to determine whether or not an exam is necessary. Federal standards for internal control highlight the importance of training to develop the relevant knowledge, skills, and abilities needed for key roles. We also have previously identified key practices for training and development that suggest that agencies should have a strategy that includes tracking and other control mechanisms to ensure that the relevant employees receive training in line with their responsibilities. Without ensuring that training reflects the relevant knowledge, skills, and abilities needed by claims processors in VBA regional offices, VBA may find these staff continue to make uninformed and incorrect reevaluation decisions that are not aligned with VBA policy, guidance, and procedures. VA spends substantial time, effort, and billions of dollars per year providing disability compensation, health care, and other forms of assistance that promote the wellness of veterans with service-connected conditions. However, VA does not know whether these efforts improve the health of these veterans on several fronts. While we are encouraged by VA’s interest in considering ways to analyze health outcomes, VA has not yet established a plan for addressing the identified research challenges. Without a plan, VA will not be positioned to understand the characteristics, needs, and health outcomes of veterans with service- connected conditions or how disability compensation and health care work together to help them. Disability reevaluations can shed light on whether veterans’ service- connected conditions have changed. However, the agency could take additional steps to analyze outcome and other data on completed reevaluations. Importantly, tracking and analyzing trends and outcomes could shed light on an apparent contradiction: why the majority of recent reevaluations resulted in no change in veterans’ combined ratings when the regulations state that reevaluations generally should not be conducted in these cases. Without these analyses, VA may be unaware of any reevaluation trends, possible explanations for them, or need to recalibrate guidance or resources to address these issues. Reevaluations represent an investment of resources for VA and the veterans who undergo them. Insights into the effectiveness of the reevaluation process are thus critical for managing VBA’s workload and informing agency policy. Specifically, while VBA tested raters’ knowledge of reevaluation policies in its May 2018 consistency study, it missed opportunities to review reevaluation decisions in the offices at greatest risk of making incorrect decisions, as recommended in the consistency study report. Following up on the report’s findings could also provide insights into root causes of errors in reevaluation decisions, which could inform decisions about additional targeted training or improved guidance. For veterans who show health improvements, VBA’s reevaluation process can ensure they have the correct disability rating and associated benefit payment. However, VBA could better mitigate the risks of making unwarranted reevaluation decisions by clarifying guidance in its procedures manual about the knowledge, skills, and abilities regional office staff need to determine whether a reevaluation should be conducted. Moreover, defining training requirements would help ensure that claims processors who conduct reevaluations have the needed skill sets and that their decisions are aligned with VBA policy and guidance. Ultimately, by enhancing and linking existing information about service- connected conditions and health care and from the results of reevaluations, VA could better understand the health outcomes of veterans who have incurred or aggravated disabling conditions during military service. We are making the following five recommendations to VA: The Secretary of Veterans Affairs should ensure that the Office of Enterprise Integration develops a plan—including milestones and roles and responsibilities for OEI, VBA, and VHA—to address identified challenges that have hindered research on the health care outcomes for service-connected conditions of veterans receiving disability compensation. To align VA’s efforts with the goals of its 2018-2024 Strategic Plan, VA’s development of this plan should be completed and ready for implementation by June 1, 2020. (Recommendation 1) The Under Secretary for Benefits should develop and implement a periodic analysis of program management data for trends in the individual service-connected conditions being reevaluated as well as data on the outcomes of reevaluations. (Recommendation 2) The Under Secretary for Benefits should implement the two recommendations in VBA’s May 2018 consistency study to provide training on how to determine when a reevaluation is needed and review reevaluation decisions for accuracy at the lowest-scoring offices and take corrective action as needed. (Recommendation 3) The Under Secretary for Benefits should clarify guidance in its procedures manual regarding the knowledge, skills, and abilities needed to make decisions on whether to reevaluate veterans for changes in their service- connected conditions. (Recommendation 4) The Under Secretary for Benefits should align training requirements with the knowledge, skills, and abilities needed for reviewing claims to decide whether to conduct a reevaluation. (Recommendation 5) We provided a draft of this report to the Department of Veterans Affairs (VA) for review and comment. VA provided written comments that are reproduced in appendix I. VA agreed with recommendations 1 and 2, and concurred in principle with our other three recommendations. The comment letter described steps the Veterans Benefits Administration (VBA) plans to take, or is in the process of taking, to address the recommendations. However, except for recommendations 1 and 2, VA’s proposed actions would not fully address the underlying issues we identified. With regard to recommendation 1 to develop a plan to address challenges to studying health outcomes, VA stated that the Office of Enterprise Integration (OEI) will coordinate with VBA and the Veterans Health Administration (VHA) to create an operational plan that addresses challenges that have hindered research on health care outcomes for service-connected conditions of veterans receiving disability compensation. VA anticipates completing this plan by June 2020. With regard to recommendation 2 to use information on reevaluations to improve program management, VA stated that VBA plans to expand its review of existing data and reports to analyze trends regarding which service-connected conditions are identified for reevaluation, and review the outcomes or results of these reevaluations. VBA plans to develop and implement this effort by the end of June 2020. With regard to recommendation 3 to implement the recommendations from the 2018 consistency study, VA stated that VBA provided a reminder to all regional offices about the availability of training resources on how to determine when a reevaluation is needed. VA also stated that VBA conducted another consistency study on this issue in August 2019 and plans to inspect claims at the two lowest-scoring regional offices identified in that study by January 15, 2020. We are encouraged by VBA’s plans to use the results of the 2019 study by inspecting claims at the lowest- scoring offices. However, using the results of both the 2018 and 2019 studies would allow VBA to more fully identify and correct root causes of any deficiencies, such as through additional training or the improvement of training. With regard to recommendation 4 to clarify guidance regarding the specific knowledge, skills, and abilities staff need to determine when to reevaluate disability claims, VA recognized the importance of having appropriately skilled and trained employees to process reevaluations and other claims. VA stated that each regional office identifies which employees complete these reviews based on their staff expertise. Further, VA stated that its Systematic Technical Accuracy Review (STAR) results of 95 percent for reevaluations indicate that further action is not needed. We continue to believe that flexibility for regional offices can be balanced with assurance that staff with the appropriate knowledge, skills, and abilities are conducting this work across regional offices. In addition, the STAR accuracy rate provides limited information about the accuracy of decisions to reevaluate claims, as discussed below. As noted in the report, identifying the knowledge, skills and abilities needed by qualified staff to carry out their responsibilities can help management ensure the program’s objectives are met. With regard to recommendation 5 to improve training for reevaluations, VA stated that additional training on reevaluations is not needed because its STAR accuracy rate for reevaluations is 95 percent. As noted in the report, VBA’s STAR reviews a small percentage of all completed reevaluations, and errors related to improperly ordered reevaluations are not reflected in STAR accuracy scores. We believe that additional action is needed to address our recommendation by ensuring staff are trained appropriately on these procedures to correctly determine whether reevaluations are needed. This additional training or guidance is particularly needed given the results of VBA’s May 2018 and August 2019 consistency studies, the views of regional staff we talked with, and the large volume of unwarranted exams. We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, James Whitcomb (Assistant Director), Dana Hopings (Analyst-in-Charge), Rachel Pittenger, and Greg Whitney made key contributions to this report. Also contributing to this report were Steven Campbell, Debra Draper, Alex Galuten, Sarah Gilliland, Alison Grantham, Amber Gray, Gina Hoover, Aaron Karty, Diona Martyn, Mimi Nguyen, Jessica Orr, Claudine Pauselli, Almeta Spencer, Srinidhi Vijaykumar, and Erin Wurtemberger.", "summary": "VA receives billions of dollars per year to provide health care and disability compensation to promote the wellness of veterans with service-connected conditions. VA studies veterans' health through research and assesses changes in service-connected conditions through its reevaluation process. GAO was asked to review VA's efforts to study and gauge the health outcomes of veterans with service-connected conditions. This report examines the extent to which (1) veterans used VA health care services to treat service-connected conditions, and what is known about their health outcomes; (2) VA uses information on reevaluations to help manage the program; and (3) VA's procedures position it to determine when to conduct a reevaluation. GAO reviewed fiscal year 2018 VA health care data; selected studies; VA data on completed reevaluations from fiscal years 2013-2018; and relevant federal laws, regulations, and program guidance. GAO also interviewed staff at four VA regional offices (selected for variation in claims workload and location) and VA officials at the agency's central office. In fiscal year 2018, about 54 percent of veterans receiving Department of Veterans Affairs (VA) disability compensation had at least one VA outpatient visit to treat an injury or illness that VA deemed was incurred or aggravated during military service (i.e., a service-connected condition). However, the health outcomes of veterans with service-connected conditions, such as changes in the severity of symptoms or the incidence of mortality, are not well understood. Information about health outcomes is central to ensuring veterans' wellness and assessing improvement in their disability status. According to VA researchers GAO spoke with and academic studies GAO reviewed, various challenges have limited research on this population. For example, data reside in different VA systems and use different identifiers for medical conditions, hindering use of the data. While VA has begun to consider ways to analyze health outcomes, it has not yet established a plan for this effort, including the scope, specific activities, and timeframes for addressing the identified research challenges. VA does not glean information from the results of reevaluations to help manage its disability compensation program. Disability reevaluations help VA gauge whether veterans' service-connected conditions have changed, and whether disability compensation should be modified to reflect those changes (see figure). However, VA does not fully use key management information, such as: trends in how frequently certain conditions are reevaluated, including those required by VA regulations to be reevaluated; and outcomes of reevaluation decisions for individual conditions (i.e., whether conditions worsened or improved). Both trend and outcome information could help VA better target its resources toward reevaluating conditions more likely to change. VA recently updated its procedures manual to specify which staff may determine whether a veteran's condition should be reevaluated, but has not clearly defined skill sets and training needed to consistently implement these procedures. Specifically, the updated procedures do not indicate the knowledge, skills, and abilities staff need to determine when to conduct reevaluations. Further, VA has not ensured that training aligns with these needed skillsets. Without improving procedures and training, VA is at risk of conducting unnecessary reevaluations and burdening veterans. GAO is making five recommendations, including that VA develop a plan to address challenges to studying health outcomes, use information on reevaluations to improve program management, and improve procedures and training for reevaluations. VA agreed with two recommendations and agreed in principle with the other three, but its proposed actions do not fully address GAO's concerns.", "document_type": "gao"}
{"report": "Foreign students interested in studying in the United States must first be admitted to an SEVP-certified school or university before applying for a nonimmigrant visa at a U.S. embassy or consulate overseas to authorize travel to the United States. A visa holder must present himself or herself for inspection at a U.S. port of entry by an officer with DHS’s U.S. Customs and Border Protection to determine admissibility. Nonimmigrants, including foreign students, are permitted to enter the United States for an authorized period of stay. Schools seeking to enroll foreign students on F and M visas must pay an application fee and petition for SEVP certification by submitting an electronic certification petition and supporting documentation to ICE through SEVIS. Among other things, SEVIS assists ICE in tracking and providing oversight of foreign students—while they are approved to study in SEVP-certified U.S. educational institutions—and their accompanying dependents. Figure 1 outlines the steps required for schools seeking to obtain and maintain SEVP certification and the process for foreign nationals to pursue a course of study in the United States. More specifically, during the initial certification process, a school must provide ICE, specifically SEVP’s School Certification Unit (Certification Unit), with evidence of the school’s legitimacy (or bona fides) and eligibility. Such evidence includes the following: proof of any requisite licensure or approval by an appropriate state- level licensing or approving agency; proof of accreditation by an accrediting agency recognized by the Department of Education, if accreditation is required or otherwise claimed; DSO’s attestation statement that he or she is familiar, and intends to comply, with program rules and regulations for admission under, and maintenance and change of, nonimmigrant student status; and confirmation by the school that it is eligible for certification, among other things (willful misstatements in a school certification petition may constitute perjury); and DSOs’ proof of U.S. citizenship or lawful permanent residency. In addition, petitioning schools must generally submit a school catalog or written statement including certain information with respect to the qualifications of teaching staff, and attendance and grading policies, among other things. However, the requirement for a school catalog or written statement is not applicable to a public school or school system, a school accredited by a Department of Education–recognized accrediting agency, or a secondary school operated by or as part of such an accredited school. Moreover, an institution of higher education that is not a public educational institution or system, or not accredited by a recognized accrediting body, must provide evidence “in lieu of” meeting those criteria. Such evidence must show either that the school of higher learning confers recognized degrees upon its graduates or its credits have been and are unconditionally accepted by at least three public or accredited institutions of higher education. Schools nominate individuals to serve as DSOs, who act as liaisons between foreign students, the DSOs’ employing school, and federal government agencies. DSOs support school compliance with record- keeping, reporting, and other requirements, and provide recommendations to foreign students regarding the maintenance of their immigration status. In addition to entering and maintaining complete information on students in SEVIS in a timely manner, DSOs are responsible for using SEVIS to submit their school’s certification petition and update the information, as necessary. To demonstrate eligibility, DSOs must, among other things, provide to ICE statements certifying their familiarity and intent to comply with the program rules and regulations relating to the requirements for nonimmigrant students’ admission, maintenance of status, and change of status, and requirements for school approval. ICE’s regulations provide that willful misstatements in certification and recertification submissions may constitute perjury. Once ICE has received a complete petition from a school seeking SEVP certification, staff from SEVP’s Field Representative Unit are to conduct a site visit to the school, including each instructional site foreign students will attend, to interview school officials and review the facilities. After receiving all necessary evidence and a site-visit report from the field representatives, ICE staff in the Certification Unit analyze the documentation, determine the school’s eligibility, and certify those schools that they determine meet all of the program’s requirements. Further, DHS is required to conduct a review, every 2 years, of certified schools’ continued eligibility and compliance with the program’s requirements. To be eligible for recertification, an SEVP-certified school must demonstrate at the time of filing that it remains eligible for certification and has complied during its previous period of certification or recertification with record-keeping, retention, reporting, and other program requirements. During the recertification process, the Certification Unit requires schools to submit the same type of evidence that was required for certification, including, among other things, proof of state licensing and accreditation and DSO attestation statements and citizenship documentation. The Certification Unit also evaluates how the school has ensured that its foreign-student records are accurate and in compliance with statutory record-keeping requirements. However, site visits are not required for recertification. The Enhanced Border Security and Visa Entry Reform Act of 2002 states that a material failure of an SEVP-certified school to comply with the record-keeping and reporting requirements to receive foreign students shall result in the suspension for at least 1 year, or termination, of the school’s approval to receive such students. SEVP’s Analysis and Operations Center (Compliance Unit) conducts ongoing monitoring of SEVP-certified schools for compliance with these regulatory record- keeping and reporting requirements, as well as schools’ continued eligibility for certification. Under federal regulation, SEVP can deny an SEVP-certified school’s recertification petition or, subsequent to out-of- cycle review, withdraw certification if the school or its programs are no longer eligible for certification. Denial of recertification or withdrawal on notice as a result of out-of-cycle review may be for any valid and substantive reason, including failure to comply with record-keeping and reporting requirements, willful issuance by a DSO of a false statement, or not operating as a bona fide institution of learning, among other bases. According to federal standards and guidance, executive-branch agency managers are responsible for managing fraud risks and implementing practices for combating those risks. Federal internal control standards call for agency management officials to assess the internal and external risks their entities face as they seek to achieve their objectives. The standards state that, as part of this overall assessment, management should consider the potential for fraud when identifying, analyzing, and responding to risks. Risk management is a formal and disciplined practice for addressing risk and reducing it to an acceptable level. In July 2015, we issued the Fraud Risk Framework, which provides a comprehensive set of key components and leading practices that serve as a guide for agency managers to use when developing efforts to combat fraud in a strategic, risk-based way. The Fraud Risk Framework describes leading practices in four components: commit, assess, design and implement, and evaluate and adapt, as depicted in figure 2. The Fraud Reduction and Data Analytics Act of 2015, enacted in June 2016, requires the Office of Management and Budget (OMB), in consultation with the Comptroller General of the United States, to establish guidelines for federal agencies to create controls to identify and assess fraud risks and design and implement antifraud control activities. The act further requires OMB to incorporate the leading practices from the Fraud Risk Framework in the guidelines. In July 2016, OMB published guidance about enterprise risk management and internal controls in federal executive departments and agencies. Among other things, this guidance affirms that managers should adhere to the leading practices identified in the Fraud Risk Framework. Further, the act requires federal agencies to submit to Congress a progress report each year for 3 consecutive years on the implementation of the controls established under OMB guidelines, among other things. ICE developed a risk-assessment framework and other tools to assist in its efforts to manage fraud risks to SEVP. For example, in 2014, ICE began developing an SEVP Risk Assessment Model and Framework, which provides an overview of how SEVP identifies, assesses, responds to, and reports on identified internal and external risks to the program. Specifically, SEVP’s Risk Assessment Model and Framework—which was updated several times between 2014 and 2017—discusses categories of fraud risks to the program, including fraud associated with schools, DSOs, and students. Moreover, in 2014, ICE developed a Risk Assessment Tool for SEVP that uses data from SEVIS records to identify potential fraud and other noncompliance issues among certified schools. The tool prioritizes different risk indicators—such as the proportion of the school that consists of foreign students—and ranks schools by risk level. SEVP officials stated that schools identified as high risk receive additional administrative review by the Compliance Unit. According to SEVP officials and documentation we reviewed, ICE has continued to update and refine the tool since 2014 to improve its effectiveness in helping to identify program risks, including fraud risks. Through these and its oversight efforts, ICE has identified various fraud risks in SEVP; such risks may take various forms, including immigration benefit fraud, which involves the willful or knowing misrepresentation of material facts for the purpose of obtaining an immigration benefit, such as a nonimmigrant student status, without lawful entitlement. According to ICE documentation we reviewed and officials we spoke to, the fraud risks to the program generally fall into four broad categories: schools, students, DSOs, and third-party brokers, who are individuals engaged in the fee- or commission-based recruitment of foreign students, among other activities. Figure 3 illustrates the types of fraud that may occur in these four categories during different stages of a certified school’s involvement in the program, as we identified in ICE documentation and through our interviews with ICE officials. For specific examples of fraud risks that ICE has identified in SEVP, see figure 4. ICE has also taken steps since 2012 to strengthen its fraud risk- management efforts in response to our prior recommendations. For example, in our 2012 report on SEVP risks, we found that, among other things, ICE did not have a process to assess risks in SEVP and did not consistently implement existing internal controls for determining school eligibility. To address this and other findings, we made eight recommendations to enhance ICE’s ability to assess program risks, prevent and detect school certification fraud, and improve the controls over SEVP. ICE took action that addressed these eight recommendations and has developed various tools designed to strengthen its fraud risk- management efforts (see app. I). Further, ICE has taken steps to improve collaboration and coordination to enhance fraud risk management between SEVP and CTCEU, the unit within ICE responsible for managing criminal investigations. More specifically, ICE has embedded agents within SEVP’s Compliance Unit, and these agents help provide law-enforcement expertise within the unit and act as liaisons with ICE agents located in the field to provide information and support ongoing criminal investigations. According to a senior ICE official with CTCEU, the embedded agents have helped streamline processes and provide expertise to aid administrative and investigative efforts. Figure 5 shows the process for coordination between CTCEU and SEVP. Further, ICE officials with CTCEU stated they have acquired specialized software tools to manage fraud tips and to conduct open-source and related research on certified schools suspected of acting fraudulently. To help identify and prioritize leads, ICE officials stated that they use a software tool to efficiently help review and prioritize tips received through ICE’s tip line, which gathers tips from the general public on suspicious or potential criminal activity. To aid investigations of schools, ICE explored the use of another specialized software to aid the review of online social media associated with schools or individuals, among other things. In addition, changes to SEVIS have aided ICE’s efforts to manage fraud risks in the program. In 2008, ICE initiated an effort to modernize SEVIS to address identified system vulnerabilities, such as the inability to capture detailed school data that would allow the detection of patterns and anomalies that could indicate fraud. Although SEVIS modernization is not yet complete, changes made in the system have helped to improve system usability and the ability to identify suspected fraud in the program, according to program officials. For example, system edit checks implemented in 2015 and 2016 to verify user-entered names and addresses have enhanced data quality by helping to identify and prevent likely data-entry errors. SEVP officials also stated that improved data quality can help make it easier to distinguish potential fraud from unintentional data-entry errors. ICE officials we spoke to and related documentation we reviewed stated that SEVIS modernization efforts may include additional functionality, such as the ability to create person-centric records for each student. Although ICE has developed a Risk Assessment Model and Framework and taken other action to improve fraud risk management in SEVP, ICE has not fully developed and implemented a “fraud risk profile” that would help guide its efforts. According to our Fraud Risk Framework, an effective antifraud entity tailors the approach for carrying out fraud risk assessments to its programs. This approach allows an agency to, among other things, develop a fraud risk profile that identifies the inherent fraud risks affecting the program, assesses the likelihood and effect of each type of fraud risk that the determines the agency’s tolerance for certain types or levels of fraud risks in the program, examines the suitability of existing controls for each fraud risk, and documents the program’s fraud risk profile. Effective managers of fraud risks use this profile to help decide how to allocate resources to respond to fraud risks. Further, Federal Internal Control Standards require managers to respond to identified risks. Appendix III provides additional information on the key elements in the fraud risk-assessment process including the development of a fraud risk profile. Our assessment of SEVP’s Risk Assessment Model and Framework found that while it describes the program’s approach for managing fraud risks, it does not include all of the key elements of a fraud risk profile: First, SEVP’s Risk Assessment Model and Framework identifies three broad categories of inherent fraud risks that affect the program (those posed by schools, DSOs, and students), but does not include all risks that the program or its stakeholders have identified, such as the risk of third-party brokers. As noted previously, ICE agents and program officials identified brokers as a risk to the program because brokers have helped facilitate school and student fraud and misused or stolen student funds in the past. However, according to ICE officials with SEVP, SEVP’s Risk Assessment Model and Framework was not designed to define all of the risks posed to SEVP. Second, while SEVP’s Risk Assessment Model and Framework assesses the potential effect of its risk posed by students, schools, and DSOs, it does not discuss the likelihood of the risk’s occurrence. For example, the Risk Assessment Model and Framework contains a narrative outlining the potential negative consequences of each of the three broad risk categories but does not address the likelihood of those risks occurring. According to SEVP officials, SEVP’s Risk Register helps identify and determine the likelihood of identified program risks. However, our review of the Risk Register found that it is used to track program-wide risks and does not identify or discuss specific fraud risks. Further, these officials stated that many of the components in a fraud risk profile are included in SEVP’s Risk Assessment Tool, but this tool was developed to prioritize the review of SEVP-certified schools that have potential compliance issues and was not designed to address all SEVP fraud risks such as the risks posed by students or brokers. Using information on the likelihood of risk occurrence can help managers decide how to allocate resources. For example, managers can use this information to make decisions to allocate resources to addressing fraud risks that are most likely to occur or have relatively high impact. Third, SEVP’s Risk Assessment Model and Framework does not assess the agency’s tolerance for all fraud risks to the program. For example, while SEVP officials stated that students represent a significant risk to the program, they have not fully assessed the extent of risks associated with student fraud or the agencies’ tolerance for it. In October 2017, the SEVP Director stated that SEVP was just beginning to get a better understanding of student risks, but had not done an assessment of their likelihood and tolerance. However, SEVP officials acknowledged the importance of fully assessing student risks because of the challenges that can be associated with detecting, preventing, and responding to student fraud. Fourth, SEVP’s Risk Assessment Model and Framework does not examine the suitability of existing fraud controls or prioritize all residual risks that remain after inherent risks have been mitigated by existing control activities. We found that, while the Risk Assessment Model and Framework discusses different internal controls and tools used to prioritize and address risks in the school certification and recertification process, such as the Risk Assessment Tool, it does not explicitly identify any internal controls or tools used to prioritize or address student risks. In addition, the Risk Assessment Model and Framework does not identify and prioritize residual fraud risks that ICE has flagged as being vulnerabilities to the program. According to ICE agents in four field offices and officials in the Compliance Unit, limitations to SEVP’s ability to prevent some schools that present fraud risks from obtaining certification or continuing to participate in the program after fraud risks have been identified represent residual risks to the program. For example, officials in the Compliance Unit stated that certified schools that have been accredited through an accrediting body recognized by the Department of Education generally represent a lower fraud risk, but ICE has still experienced noncompliance and cases of fraud with these schools. At one point several fraud cases were tied to the same accrediting body. In another example of a potential residual risk to the program, ICE field agents stated that potentially fraudulent schools may continue to operate during criminal investigations, which can take several years to investigate and prosecute. During the investigation, schools may remain in operation and continue to enroll foreign students, provided their certification is not withdrawn through other administrative actions. As one example, ICE’s investigation into Prodee University—a case that involved hundreds of students—began in 2011, but warrants were not issued until 2015. The school continued to operate and accept foreign students during the 4-year investigation, creating residual risk to the program during these years. According to SEVP’s Director, the program has not developed a fraud risk profile that fully addresses all identified risks because the program has not yet developed the maturity needed to manage its risks in this way, but she noted that doing so could be a good next step in the process. Without a fraud risk profile consistent with leading practices—which identifies all fraud risks, discusses the likelihood of those risks, assesses the agency’s risk tolerance, and determines the suitability of related controls—ICE cannot ensure it has taken all necessary actions to address SEVP risks. ICE is exploring the use of better data analytics to help detect fraud in SEVP but has not yet employed techniques, like network analysis, to help detect and prevent fraud prior to certification. ICE officials with SEVP stated that they are exploring the use of additional data-analytics tools to help mitigate fraud in the program, including tools that can perform network analysis. However, these efforts are in their early stages and have been limited to conversations between program staff. While previously noted efforts to improve SEVIS may also include additional data analytics to mitigate fraud, these efforts have remained underway since 2008. Agency officials told us they recognize that better analytic tools can help them detect and prevent fraud in the certification process and are seeking additional resources to support this effort. According to agency documentation, SEVP awarded a contract in September 2018 to help establish a data-governance framework within SEVP. Among other things, the contract will examine the tools, skill sets, and number of people needed to support the data-related needs for SEVP, to include operational data and analytics. According to agency officials, SEVP plans to award a contract in the first quarter of fiscal year 2019 to provide better data-analytics support. Data-analytics approaches, such as network analysis, have the potential to enable ICE to identify high-risk schools prior to initial certification, thus allowing SEVP to apply increased oversight, as needed, during the adjudication process. Network analysis involves a quantitative approach for analyzing, summarizing, and graphically representing complex patterns of relationships among individuals or organizations. Such a technique is useful for identifying associations, such as between schools with current or past administrative and criminal concerns and those schools seeking certification. Information about the connections and relationships among schools—developed through network analysis—may then provide leads in reviews and investigations in the certification and recertification processes, which are important controls for preventing fraudulent schools from entering and remaining in the SEVP program. ICE field agents with two of five field offices we visited stated that it can be challenging to identify fraudulent schools as compared to legitimate ones during the initial certification of schools. For example, agents familiar with one investigation stated that after ICE began investigating a school for suspected fraud, the owner tried to establish another school, which was only identified because of a lead provided through interviews conducted during the investigation. Further, because tools such as the Risk Assessment Tool use data analytics, but rely on information collected from current SEVP-certified schools, it can be difficult to identify schools with fraud concerns before they are certified to participate in the program. Using a network approach in our analysis of 2,439 SEVP-certified schools, we identified 11 connections that could raise fraud concerns. Specifically, we conducted a network analysis utilizing both public and proprietary information associated with certified schools as of September 2017. We obtained basic information on these schools from ICE, such as school names and addresses. We also used public records associated with these schools related to businesses and people, such as past and current executives. Using this information and freely available public software, we identified relationships among certified schools that ICE had previously identified as having potential compliance or fraud concerns and other certified schools that did not have such concerns. For example, in 11 connections, we identified instances in which an executive appeared to have been employed by a school under active criminal investigation or administrative review who was either previously or later employed by a different school not under investigation or review. Moreover, for 2 of the 11 connections, we found additional derogatory information associated with executives tied to SEVP-certified schools that could raise fraud concerns. For instance, one executive had employment terminated from a previous school and was under investigation for misappropriating school funds for personal use. While these connections do not prove fraud or noncompliance, they do provide information about potential risks, which can inform the prioritization of administrative and investigative resources during certification. ICE currently has limited ability to identify associations among schools with potential fraud concerns before they are certified to participate in the program. According to our Fraud Risk Framework, federal managers should design and implement specific control activities to prevent and detect fraud. These control activities can include data analytics, among other things, and should emphasize fraud prevention to the extent possible. A network approach provides the capability to better prevent and detect fraud by identifying potentially fraudulent schools before they are certified by SEVP and by detecting associations that pose a fraud risk among those already certified. ICE has processes in place for school certification, recertification, and ongoing compliance monitoring, and has taken steps to improve school certification controls since our 2012 report. We also found that ICE followed its established procedures and specifically identified GAO’s fraudulent petitions or otherwise took appropriate steps to prevent the petitions from moving forward in the process during our three independent covert tests of SEVP internal controls over the school certification process. However, the agency continues to face long- standing delays in conducting recertification reviews every 2 years to ensure that SEVP-certified schools continue to meet program requirements—one of its important fraud risk controls. As a result of these delays, ICE has a queue of recertification petitions awaiting adjudication, which creates additional fraud risks to the program if higher-risk schools continue to operate pending recertification. However, the agency has not assessed the magnitude of these risks. ICE’s certification and recertification processes are designed to assess schools’ initial and continued eligibility to enroll foreign students and, as previously discussed, once a school is certified, ICE is to monitor its continued program eligibility. SEVP-certified schools are to undergo recertification reviews every 2 years (see fig. 6). Initial certification: As previously discussed, to be eligible for SEVP certification, a petitioning school must establish at the time of filing that it is a bona fide institution of learning or other recognized place of study that possesses the necessary facilities, personnel, and finances to conduct, and is in fact engaged in, instruction in recognized courses. SEVP officials stated that they address potential fraud risks during the initial certification process by verifying the schools’ information and documentation through web-based research and a site visit to interview the school’s DSO and observe the school’s facilities. According to SEVP officials and guidance, as of October 2016, field representatives are responsible for conducting and documenting site visits for certifications. When conducting the visits, field representatives are to gather evidence on school eligibility for certification, review the facilities, and interview personnel nominated on the petition to become DSOs. They may also report back any anomalies or areas of concerns they may notice for further vetting by the compliance unit. SEVP received approximately 2,000 certification petitions from fiscal years 2013 through 2017. See figure 7 for details on the number of approved and denied petitions during this period. ICE has implemented several controls to address fraud risks in the school certification process since our 2012 report on SEVP program risks, but long-standing delays in the recertification process create additional fraud risks. In particular, ICE strengthened its processes for verifying and monitoring schools’ accreditation and states’ licensing statuses. For example, since December 2012, SEVP adjudicators are to verify all “in lieu of” letters during the school’s initial-certification and recertification processes. In May 2015, SEVP developed a continuous process for verifying schools’ state licensing and accreditation status and updated its Adjudicator’s Manual with specific actions adjudicators must take to consistently verify evidence provided by schools, including “in lieu of” letters and states’ licensing documentation. In addition, SEVP took steps to ensure that all flight schools had the appropriate Federal Aviation Administration certification. Recertification: To be eligible for recertification, an SEVP-certified school must demonstrate at the time of filing that it remains eligible for certification and has complied during its previous period of certification or recertification with record-keeping, retention, reporting, and other program requirements. SEVP received approximately 14,000 recertification petitions from fiscal years 2013 through 2017. See figure 8 for details on the number of approved and denied petitions during this period. The recertification process is an important fraud risk control, according to ICE officials, since they may determine that some certified schools are potentially noncompliant during the recertification process. For example, SEVP denied 105 recertification petitions from fiscal year 2013 through fiscal year 2017. On the basis of our review of recertification denial data, the majority of denials were due to the school’s abandoning its petition for recertification by not responding to SEVP’s request for further information. Appendix IV provides additional details on the withdrawal and denial of certification and recertification petitions as outlined in federal statute and regulation. For the remaining schools, SEVP issued a formal recertification denial notice for a variety of reasons, including those that highlight fraud risks in the program, such as improper issuance of Forms I-20, including the issuance of forms to foreign students who will not be enrolled in or carry a full course of study; DSO conduct did not comply with program regulations; willful issuance by a DSO of a false statement; failure to timely report school or course of study information, including material changes; and failure to maintain the accreditation or licensing necessary to qualify graduates as represented in the school’s Form I-17. Ongoing compliance monitoring: The Enhanced Border Security and Visa Entry Reform Act of 2002 provides that SEVP-certified schools are to comply with record-keeping and reporting requirements to enroll nonimmigrant students. Between schools’ initial certifications and their subsequent recertification reviews, ICE uses a variety of mechanisms to monitor ongoing compliance with program requirements and mitigate fraud risks. For example: SEVP deployed its first group of field representatives in 2014. As of June 2018, ICE had 57 field representatives across 60 different geographic areas of responsibility nationwide. According to SEVP guidance, field representatives are to act as direct liaisons between SEVP and certified schools and are to try to meet with all certified schools in their territory at least once per year if the school has foreign students enrolled, or once every 2 years if no foreign students are enrolled. According to SEVP officials, the field representatives are to have a customer-service focus and assist DSOs in adhering to program requirements and, as a result, do not have law-enforcement or investigative responsibilities. However, if field representatives learn of potential fraud while visiting a school, they are to document and send this information to SEVP headquarters. All of the eight field representatives we interviewed reported that they primarily have a customer-service role but have also identified and reported suspected fraud to SEVP headquarters. For instance, one representative stated that she reported a language school because its stated level of student enrollment did not appear to correspond with the number of students in class during her visits to the school. SEVP adjudicators are to verify and adjudicate changes that occur at an SEVP-certified school that require an update to the school’s Form I-17 petition information in SEVIS. These changes include the school’s name, location, or new areas of study offered, among others. According to Certification Unit officials, adjudicators review information from both SEVP’s risk tools and field-representative school-visit reports when adjudicating updates to identify any indications of noncompliance or fraud that need to be further reviewed and researched by the Compliance Unit. Compliance Unit staff are to vet tips provided by external parties (such as DSOs from other schools) or internal stakeholders (such as field representatives or Certification Unit adjudicators) to determine whether they indicate the need to open an administrative or criminal investigation on the school. Compliance Unit staff may also identify schools for additional monitoring. The Compliance Unit is also responsible for extracting and analyzing data from SEVIS on an ongoing basis, including data related to certified schools and foreign students suspected of noncompliance and fraud, among other things. According to ICE officials, staff are responsible for researching schools with high-risk scores provided by the Risk Assessment Tool. ICE may conduct an out-of-cycle review of a school at any time to help determine whether the school is complying with its reporting and record-keeping requirements and to ensure the school’s continued eligibility for SEVP certification. ICE may initiate an out-of-cycle review as a result of receiving information regarding potential noncompliance or fraud. The out-of-cycle review process may include a review of student records, a request for the submission of documentation to verify accreditation, a request for proof of state licensure, or a request for any other required evidence that establishes a school’s continued eligibility for SEVP certification. ICE officials stated that they may, pending the result of this review, issue a remedial action plan to the school describing the areas of noncompliance, such as correcting student records, that the school is required to address to maintain its program eligibility. If, upon completion of an out-of-cycle review, SEVP determines that a certified school has failed to sustain eligibility or has failed to comply with the record-keeping, retention, reporting, and other requirements, SEVP will institute withdrawal proceedings by serving the school a notice of intent to withdraw SEVP certification. At the conclusion of withdrawal proceedings, a school found to be ineligible for continued SEVP certification as a result of an out-of-cycle review will receive a notice of withdrawal (see app. IV for additional information on the withdrawal process). ICE followed established procedures during our three covert tests of the internal controls over the SEVP school certification process by either successfully identifying GAO’s fraudulent petitions or by taking appropriate steps to prevent the petitions from moving forward in the process. Therefore, we did not identify any significant deficiencies during our testing of these controls. We submitted certification petitions and conducted other covert investigative work for three fictitious schools, all of which have differing certification requirements. Using these schools, GAO agents applied for SEVP certification. For one of the fictitious schools, we tested SEVP certification controls that require schools to submit complete documentation by submitting an application for the school that was missing several of the required documents. Consistent with its procedures, ICE flagged our petition as incomplete and sent us a notification stating that our petition was canceled because we failed to submit all supporting evidence as outlined in the regulations. For our second school, we tested SEVP controls requiring schools to schedule and complete a site visit conducted by an SEVP field representative, by submitting a completed petition, but avoiding the site visit and requesting that our paperwork move forward without it. SEVP’s field representative subsequently notified us that our petition would not move forward until a site visit was performed. For our third fictitious school, we submitted an application, and participated in a site visit with SEVP officials. We tested SEVP controls related to verifying application documentation, and whether SEVP site- visit officials followed established procedures for the site visit. The field representative toured the facilities and interviewed GAO agents posing as school officials. During its review of our petition, ICE took steps to verify our school’s information and discovered that documentation we submitted was fictitious. As a result, SEVP officials subsequently referred our school to ICE agents for further investigation, consistent with ICE policies and procedures. Upon learning that ICE followed its documented internal control processes, we concluded our covert testing. ICE faces long-standing challenges in conducting school recertification on a 2-year basis consistent with statute and regulation, which may allow potentially fraudulent schools to operate for a longer period without detection. The Enhanced Border Security and Visa Entry Reform Act of 2002 states that DHS must conduct compliance reviews every 2 years, during which ICE reviews a school’s records to verify that it continues to comply with program-eligibility requirements. ICE began the first recertification cycle in May 2010—8 years after the enactment of the statutory requirement for periodic review of SEVP-certified schools. As of March 2012—nearly 10 years after statutory enactment—ICE reported that it had recertified approximately 19 percent of certified schools. In October 2016, ICE reported that it had completed its first round of recertification (in other words, all existing certified schools had been recertified at least one time) and had used recertification to address a number of issues, including gathering missing data for some school records. ICE has continued to recertify schools. However, Certification Unit officials told us that, while recertification should be conducted every 2 years, ICE has been unable to meet a 2-year time frame for all certified schools. ICE has been extending schools’ certification expiration dates since officials began recertifying schools in 2010, according to Certification Unit officials, to provide additional time for adjudicating recertification petitions. According to ICE regulations, schools should be notified 180 days before their certification expiration date and must file a completed petition for recertification by such date, which is 2 years from the date of their previous SEVP certification or the recertification expiration date. However, as described in figure 9, SEVP has been extending schools’ certification expiration dates by 180 days beyond the 2-year mark as defined in ICE’s regulation. Under this process, schools must submit their complete petition and supporting documentation to SEVP within 180 days after the 2-year mark. Extending certification expiration dates increases the period between each recertification review, resulting in a decrease in the number of recertification reviews conducted in a given time frame, as shown in the hypothetical example of two schools in figure 10. For instance, if SEVP initially certified a school in January 2016, by providing an extension SEVP is setting the school’s certification expiration date to July 2018—2 years and 180 days after the initial certification—as opposed to 2 years after the initial certification, which would be consistent with ICE regulations. After receiving the school’s documentation, Certification Unit staff need time to review and adjudicate the petition. If this school submits a complete petition to SEVP in June 2018—1 month before its revised expiration date—SEVP staff may and do take additional time, depending on the facts and circumstances of the specific petition, beyond the revised expiration date to adjudicate the petition. SEVP officials stated that, if necessary, they can further extend the certification expiration date to accommodate the time needed for their review. For instance, SEVP may not adjudicate this school’s petition until December 2018. Once SEVP completes its adjudication in December 2018, the school’s new certification expiration date would be June 2021 (2 years and 180 days after December 2018). Thus, rather than potentially being able to complete two rounds of recertification during this 5-year period consistent with ICE regulation, SEVP would recertify the school only once. As we reported in 2012, according to SEVP officials, ICE delayed the recertification process until after SEVIS was deployed in 2003 and the program fee was increased in 2008 to support hiring additional staff. Further, with regard to resources, ICE officials stated that they are cross- training adjudicative staff across all of their program areas to help address the recertification workload, and creating regional adjudication teams with assigned territories similar to the field representatives’ territories to allow the adjudicators to work with the same schools throughout the school’s participation in the program. In addition, in February 2018, SEVP’s Director stated that ICE was expecting to hire additional adjudicators for a total of 10. In July 2018, ICE identified the need to increase initial certification fees and add a new recertification fee to, among other things, hire additional adjudicators to address longer recertification processing times. Specifically, ICE stated that, at present staffing levels, SEVP is able to process 1,939, or 44 percent, of the required annual projected 4,400 recertification cases. ICE’s actions to allocate additional resources to the recertification process are a step in the right direction toward addressing its recertification delays. However, it is unclear whether these actions alone will be adequate to address the delays. As of June 2018, ICE officials told us that there were 3,281 recertification petitions that needed to be adjudicated. As previously discussed, recertification reviews are an important fraud risk control because they are one of ICE’s primary means of reviewing each school’s data and identifying potential school noncompliance and fraud, especially since an out-of-cycle review may not be conducted for each school. As Federal Internal Control Standards state, management should: (1) establish and operate activities to monitor the internal control system and evaluate the results, and (2) identify, analyze, and respond to risks related to achieving the defined objectives. By not requiring schools to submit their petitions within the 180-day period prior to the 2-year expiration date, as required by regulation, ICE has limited assurance it is leveraging the recertification process effectively to identify and respond to potential fraud risks to the program, including those risks associated with allowing a fraudulent school to operate for a longer period. ICE’s plan to increase the number of SEVP adjudicators may help it meet the 2-year recertification requirement, but without monitoring and evaluating the efficacy of these actions, ICE will not have reasonable assurance it can effectively manage the recertification process and associated fraud risks. As previously discussed, ICE’s queue of recertification petitions awaiting adjudication creates additional fraud risks to the program if higher-risk schools continue to operate pending recertification. However, ICE has not assessed the magnitude of such risks. As of June 26, 2018, ICE had 3,281 recertification petitions in a queue for review, according to SEVP officials, petitions that ICE adjudicates in the order in which they were filed. As discussed, ICE uses a variety of mechanisms to monitor schools’ ongoing compliance with program requirements and mitigate fraud risks. In addition, ICE assesses and considers schools’ risks during the adjudication process for recertification. Specifically, according to SEVP’s recertification standard operating procedures, case analysts in the Certification Unit are to review the recertification packages once submitted to determine whether they are complete and prepare them for adjudication. Further, SEVP officials stated that the Certification Unit staff use an assessment of the school’s risk to help prioritize further analysis and review efforts. When adjudicating recertification petitions, adjudicators are to confirm that they have assessed the school’s risk and whether any identified risks have previously led to any further action, according to Certification Unit officials. If case analysts determine that compliance issues are present (e.g., the school has closed or the school has made updates to the Form I-17 that are awaiting adjudication), they are to notify their supervisors. For higher-risk schools, Certification Unit officials stated that adjudicators may request more detailed evidence from schools as part of recertification, consistent with their standard operating procedures, than they would for lower-risk schools to help make more efficient use of the resources in this unit. These processes have helped SEVP consider and address potential risks during the recertification process. However, SEVP has not determined risks posed by schools in its recertification queue and, according to Certification Unit officials, does not prioritize the review of schools’ recertification petitions in its queue based on risk. As previously noted, ICE is required to conduct periodic reviews every 2 years to determine SEVP-certified schools’ continued program eligibility and compliance. The statute governing recertification does not, by its terms, preclude ICE from considering a school’s relative risk as part of the compliance review process. However, SEVP’s Director and Certification Unit officials stated that a recertification process that prioritizes reviews based on school risk would not be particularly helpful or add value in addressing school compliance concerns because the officials already have a number of mechanisms they can use, as previously discussed, to address potential noncompliance, including conducting out-of-cycle reviews of high-risk schools. Although ICE considers schools’ risk-related information during the adjudication process and may identify noncompliant or potentially fraudulent schools through ongoing monitoring activities, ICE has not determined the extent to which there are residual fraud risks posed by schools in the recertification queue that ICE has identified as higher-risk than other schools awaiting recertification. According to GAO’s Fraud Risk Framework, managers should rank residual fraud risks in order of priority, using the likelihood and impact analysis, as well as risk tolerance, to help decide how to allocate resources to respond to residual fraud risks, all of which is documented in a fraud risk profile. As previously discussed, a fraud risk profile (1) identifies the inherent fraud risks affecting the program, (2) assesses the likelihood and effect of each type of fraud risk that it has identified, (3) determines the agency’s tolerance for certain types or levels of fraud risks in the program, and (4) examines the suitability of existing controls for each fraud risk. Given SEVP’s long- standing delays in recertifying schools, without an assessment of residual risks posed by the recertification queue—as part of its fraud risk profile, as previously noted—ICE cannot ensure that it is effectively addressing the risks posed by higher-risk schools awaiting recertification, a situation that does not help further strengthen ICE’s fraud risk-management efforts in SEVP. ICE has identified fraud risks related to DSOs and implemented controls to mitigate these risks, but weaknesses exist in four key areas: (1) verification of information provided by DSOs in support of their eligibility, (2) background checks, (3) mandatory compliance training, and (4) fraud- risk training. Prior to approval of schools’ nomination of individuals to serve as DSOs, these nominees must meet eligibility requirements and pass a criminal-background check, but weaknesses exist in both of these controls. In addition, once ICE approves prospective DSOs, it has controls for oversight and training; however, this training is not mandatory and does not address fraud risks. ICE has eligibility requirements for school employees seeking to serve as DSOs at SEVP-certified schools, as discussed earlier, but does not routinely verify DSO-submitted eligibility information in support of their immigration or citizenship status. According to ICE regulations, to be eligible to participate as a DSO, an individual must be a regularly employed member of the school administration whose office is located at the school and must meet two primary eligibility criteria. First, a DSO’s compensation may not include commissions for recruitment of foreign students. To verify that requirement, a field representative is to interview a school’s principal DSO during an initial certification site visit, and ask whether any prospective DSOs receive compensation from commissions for recruitment of foreign students. In addition, a field representative is to review the school’s website for recruitment-related activities and evaluate the DSO’s job title and position description, according to ICE officials. Second, DSOs must be U.S. citizens or lawful permanent residents, but the Certification Unit does not routinely verify the evidence provided to meet this eligibility requirement. Specifically, DSOs are to submit documentation during the school’s certification or recertification process— such as a passport, birth certificate, Permanent Resident Card or Alien Registration Receipt Card, or copy of naturalization/citizenship certificate—as evidence of their U.S. citizenship or lawful permanent resident status. The Certification Unit is to review this documentation to verify that the biographic details match the information provided on the school’s Form I-17. According to ICE officials, if the Certification Unit suspects that a prospective DSO’s documentation may not be valid, it will send the information to the Compliance Unit for additional review. However, neither the Certification Unit nor the Compliance Unit routinely verify the information reported by DSOs in support of their immigration or citizenship status because they do not have access to the type of information needed to independently verify this information for all prospective DSOs, according to ICE officials. Certification Unit officials told us that verifying information on naturalized U.S. citizens and lawful permanent residents would be beneficial. They said that they have previously asked for access to information, such as other DHS databases that contain information on naturalized U.S. citizens or lawful permanent residents, to strengthen their process for determining the eligibility of prospective DSOs. However, they have yet to receive access to this information. In addition, verifying eligibility information for U.S.-born citizens would also be valuable, but is more difficult than for naturalized U.S. citizens or lawful permanent residents, according to ICE officials. This is because ICE does not collect DSOs’ Social Security numbers— key information necessary to verify U.S. citizenship—in part because SEVIS does not have the necessary security features needed to collect and house those data, and adding those features would be costly. In June 2018, ICE management officials stated that they were reviewing databases that may be useful to verify DSOs’ self-reported eligibility information but did not provide any additional support or documentation of those plans or a time frame for completing this review. As outlined in our Fraud Risk Framework, as part of an effective antifraud strategy, managers should take steps to verify reported information, particularly self-reported data. Specifically, managers can benefit from conducting data matching to verify key information, including self-reported data and information necessary to determine eligibility, using government or third-party sources to verify data electronically. Until ICE routinely verifies the eligibility information submitted by prospective DSOs in support of their immigration or citizenship status, particularly for naturalized U.S. citizens and lawful permanent residents, ICE will not be able to ensure that it is preventing ineligible individuals, including those who represent a fraud risk, from becoming DSOs and providing them with access to SEVIS to maintain student records. ICE has taken some initial steps to strengthen the process for vetting prospective DSOs but has not implemented comprehensive background checks on DSO nominees prior to approving them to carry out the DSOs’ reporting, record-keeping, and other functions. ICE officials told us that they have been working since December 2016 to develop a plan to conduct comprehensive background checks on prospective DSOs to address past concerns about DSO vetting. Specifically, in 2011, ICE expressed concerns that DSOs, who were not required to undergo background checks, were responsible for maintaining updated information of foreign students in SEVIS. According to ICE officials, they have taken initial steps to address these concerns by implementing criminal-background checks on prospective DSOs. Specifically, in May 2017, ICE started conducting background checks on all school employees nominated to be DSOs at the time of petitioning for initial SEVP certification or whenever a school requests to add a new DSO. For these types of checks, ICE officials within CTCEU are to review the prospective DSO’s biographic information from both the Form I-17 and the proof of U.S. citizenship or immigration status documentation received by the school. After ICE officials in CTCEU complete this check, they are to forward the findings to SEVP for review. If SEVP determines that a prospective DSO is unsuitable for participation in the program, ICE officials in SEVP are to send a notice of rejection to the nominating school. From April 2017 to March 2018, ICE screened approximately 4,750 prospective DSOs and identified 68 individuals with a criminal history. ICE rejected the nomination of 15 of these prospective DSOs, because, for example, they had criminal histories that included instances of identity theft, fraud in obtaining U.S. citizenship, and conspiracy, among other crimes. ICE officials stated that certain crimes will not necessarily disqualify a candidate, such as misdemeanors, traffic- related infractions, or other lesser crimes. As of June 2018, ICE officials told us that they are developing a more- comprehensive background-check process to screen prospective DSOs against additional government data sources. Specifically, ICE officials told us that they are seeking to partner with DHS’s Transportation Security Administration (TSA) to collect biometric information (e.g., fingerprints) on prospective DSOs at TSA’s enrollment provider locations nationwide during the school certification process. ICE officials stated that they intend to provide the biometric information they collect through TSA’s enrollment provider to ICE’s Office of Professional Responsibility (OPR), and OPR officials will review such information to determine DSOs’ suitability. According to agency documentation, ICE’s OPR would vet such information against data sources to screen these individuals for prior criminal histories such as sexual misconduct, terrorist activities, and immigration violations. According to ICE officials, they also intend to use this process to periodically review the suitability of incumbent DSOs. While ICE officials have told us they intend to expand the screening of prospective DSOs, ICE does not have a documented implementation plan that outlines how the project will be executed. The Project Management Institute’s A Guide to the Project Management Body of Knowledge (PMBOK® Guide) identifies standards related to project-management processes, including the need to have documented implementation plans describing how the project will be executed, monitored, and controlled, as well as requirements and techniques for communication and establishing agreements among stakeholders. In addition, GAO’s Schedule Assessment Guide identifies best practices associated with developing and maintaining a reliable, high-quality schedule. ICE provided us with a draft of its revised background-check policy, talking points on its plans for these checks, and draft requirements it shared with TSA in December 2016. However, these documents do not provide a detailed project- implementation plan to guide ICE’s effort. As of June 2018, ICE and TSA officials have met twice in the last 2 years, and ICE officials do not have any documents or other written details on their planned coordination with TSA. SEVP’s Director acknowledged that SEVP will need to develop a project plan to help guide its coordination with TSA and ICE’s OPR. Without a documented implementation plan for this effort that outlines how the project will be executed, monitored, and controlled, ICE does not have reasonable assurance that it will be able to implement a more- comprehensive DSO background-check process. ICE has established mechanisms for monitoring SEVIS usage by approved DSOs and providing support to DSOs to help them ensure their schools comply with SEVP requirements but does not mandate training for DSOs. Once DSOs are approved by SEVP, they are authorized to make changes to student records in SEVIS and to create Forms I-20, which enable students to apply for nonimmigrant student status. To detect noncompliance and fraud that may be committed by DSOs during this process, ICE has established mechanisms to monitor information entered and identify data for computers used by DSOs through SEVIS compliance checks, among other things. For example, according to agency officials, ICE monitors DSO actions in SEVIS to help prevent noncompliance and fraud. In addition to monitoring DSOs’ use of SEVIS, ICE provides support and training to DSOs to help ensure they can effectively update and maintain student records in SEVIS and provide recommendations to students regarding the maintenance of their status, according to our review of ICE documentation and interviews with ICE and school officials. According to program rules, DSOs are responsible for understanding SEVP regulations related to the requirements for foreign students’ admission, maintenance of status, and change of status and requirements for school approval. To assist them, ICE officials and DSOs that we interviewed told us that SEVP uses its field representatives to provide DSOs with a point of contact for questions related to the program. According to SEVP’s internal guidance, field representatives are expected to visit the schools within their areas of responsibility at least once a year to provide in-person guidance and training to DSOs. DSOs at 15 of the 17 schools we visited stated that the field representatives were helpful, including with providing guidance on how to comply with SEVP rules and regulations. In addition, SEVP internal guidance encourages DSOs to take its web- based training course on the responsibilities and obligations for both DSOs and foreign students in SEVIS. However, this course is voluntary. According to ICE officials and field representatives, the extent to which DSOs take the voluntary training varies—some DSOs receive additional training beyond the voluntary SEVP training, but other DSOs do not complete any training. ICE officials noted that the voluntary online training may be perceived as cumbersome and that, since it is not required, many DSOs instead reach out to field representatives or call the SEVP Response Center to get answers to questions that are covered by existing training materials. ICE officials also stated that they do not know the extent to which DSOs have completed the online training because they do not track this information. Further, the officials acknowledged that since training is voluntary, some DSOs may not complete it before assuming their responsibilities and gaining access to SEVIS. ICE officials we interviewed told us they encounter problems with DSOs complying with record-keeping requirements; however, they believe most of these issues are a result of DSOs not understanding program rules or their own responsibilities within the program. According to agency documentation, in 2014 SEVP found that some DSOs were inconsistently reporting school information in several SEVIS data fields. In addition, SEVP’s Risk Assessment Tool includes a number of high-risk indicators that may stem from DSO record-keeping errors within SEVIS, including students listed as enrolled in an academic program not available at that school (e.g., doctoral students at schools without doctorate degrees available) and students listed as active who have long exceeded their program’s end date or authorized employment’s end date. Errors such as these make it difficult for ICE officials to know whether the information in SEVIS is inaccurate due to unintentional mistakes by the DSO or whether the school or its employees may be engaged in potential fraud. For additional examples of potential noncompliance or fraud, see the box below. Potential Designated School Official Noncompliance or Fraud Student and Exchange Visitor Program officials cited the following examples of potential noncompliance or fraud that they have encountered, among others: the reported foreign-student enrollment listed in the Student and Exchange Visitor Information System (SEVIS) does not seem to correspond with the number of students attending class or the size of the school’s physical space, all enrolled foreign students listed in SEVIS are living at the same address, and students repeatedly transfer to several different schools. Field representatives at one location we visited noted that DSOs with multiple job responsibilities may not have time to keep up with SEVP rules and policy updates. Similarly, DSOs at 7 of the 17 schools we spoke with mentioned that they have multiple job responsibilities beyond their DSO duties. In addition, SEVP officials indicated that DSOs have a high rate of turnover, especially at small schools, and may lack the expertise to effectively follow program requirements. SEVP officials acknowledged that mandatory training could help reduce the number of unintentional violations by DSOs who may not adequately understand the program’s regulations, thus allowing SEVP staff to focus their monitoring efforts on schools and individuals who may be engaged in intentional noncompliance and fraud. In June 2018, ICE officials told us that they recently received internal agreement to require all new DSOs to complete training prior to gaining full access to SEVIS once the officials release a new version of their DSO training program. However, SEVP officials could not provide documentation on their plans, including time frames for completing the revised DSO-training program, whether to require DSO training, or how they will track DSO compliance. Federal Internal Control Standards calls for agencies to demonstrate a commitment to competence, including recruiting, developing, and retaining competent individuals. Further, it recommends that agencies establish expectations of competence for key roles, including possessing the necessary knowledge, skills, and abilities, and training individuals appropriately. Without mandatory training and a process to verify that training is completed, SEVP does not have reasonable assurance that DSOs are familiar with, and understand, their roles and responsibilities as outlined in program regulation. SEVP’s voluntary DSO training emphasizes student and school compliance with program rules and the DSOs’ responsibilities to enter and maintain complete and accurate information in SEVIS in a timely manner but does not address fraud risks to the program, including previously identified fraud schemes or trends. According to ICE officials, some DSOs may receive fraud-specific training from ICE agents through the Project Campus Sentinel initiative; however, these visits are limited to a small portion of certified schools each year. During a Project Campus Sentinel visit, ICE guidance states that an ICE agent will meet with DSOs and provide information on how to detect potential fraud, including student visa exploitation and national security vulnerabilities. In addition, ICE guidance encourages ICE agents to remind DSOs to contact them when they encounter these instances. In fiscal year 2017, ICE officials reported that ICE agents visited 400 of the more than 18,000 SEVP- certified school campuses in existence at that time. According to ICE officials, the agency can only conduct a limited number of Project Campus Sentinel visits to schools each year due to competing investigative priorities. The DSOs we spoke with varied in their understanding of the role they should play in identifying and reporting fraud to SEVP. Specifically, DSOs at 8 of 17 schools told us they did not receive training on SEVP-related fraud risks or could not identify SEVP-provided, fraud-specific training. For example, DSOs at one school told us that there is confusion among DSOs about their role to prevent and report fraud and that this issue has been discussed at past training events and conferences. Specifically, they stated that there is some confusion over the difference between fraud and noncompliance. According to these DSOs, they are responsible for addressing issues of noncompliance, but they do not actively look for SEVP-related fraud. A DSO from another school told us she interprets the DSO role as providing program oversight, including oversight related to fraud, and that she previously reported an instance of potential student fraud to ICE when she encountered suspicious immigration paperwork. In addition, DSOs at another school told us that they were not aware of any training related to fraud risks within SEVP but noted that guidance about fraud trends or potential red-flag indicators could be useful. The Fraud Risk Framework identifies training as one way of demonstrating an agency’s commitment to combating fraud. Training and education intended to increase fraud awareness among stakeholders, managers, and employees serves as a preventive measure to help create a culture of integrity and compliance within the agency. Specifically, the Fraud Risk Framework discusses leading practices for training and education, including communicating responsibilities for implementing fraud controls and details on how and where to report fraud. In addition, increasing awareness of fraud schemes, including red flags and risk indicators, through training and education can serve a preventive purpose by helping create a culture of integrity and compliance within the program and can enable managers, employees, and stakeholders with responsibility for implementing aspects of the program to better detect potential fraud. According to ICE officials, DSOs can serve as the front line against SEVP-related fraud, and they provide a significant portion, if not the majority, of fraud-related tips. In June 2018, ICE officials told us that, in response to discussions that we had during our review, they plan to incorporate fraud training into the revised DSO training. However, because ICE officials just recently made that decision, they had not yet developed documented plans for this training or timelines for when it would be completed. While agreeing to incorporate fraud training into the revised DSO training is a good first step, the development and execution of those plans will be needed to strengthen fraud controls. Until ICE develops and implements a plan for fraud-specific DSO training, ICE will not have reasonable assurance that this training will be delivered and DSOs will have the information they need to address fraud within the program. Through SEVP, ICE oversees over 1.2 million foreign students at nearly 9,000 SEVP-certified schools across more than 18,000 campuses. Past instances of fraud and noncompliance in the program have resulted in ICE taking some steps to address fraud risks in the program, such as developing a Risk Assessment Model and Framework. However, ICE does not have a fraud risk profile that identifies all of SEVP’s fraud risks, discusses the likelihood of those risks, assesses related controls, and identifies the agency’s tolerance for risk. Such a fraud risk profile would help ICE more effectively assess whether additional internal controls or changes to policies or regulations are needed. Moreover, ICE has not yet fully employed the use of data analytics, such as network analysis, to help it identify potentially fraudulent schools before they become certified to enroll foreign students and help it better use its administrative and investigative resources. ICE has also made improvements to its processes for certifying and recertifying SEVP schools and monitoring DSOs—all of which can help reduce the risk of fraud in the program. However, ICE continues to delay the recertification process by initiating the school recertification reviews after the 2-year certification expiration date, which is not consistent with ICE regulations. Further, ICE has not included an assessment of residual risks posed by the current recertification queue—as a part of the fraud risk profile previously noted—and as a result does not have a full understanding of the risks associated with schools awaiting recertification. Although DSOs play an important role in helping ICE oversee students in the program, ICE has recognized they can pose fraud risks to the program. However, ICE does not routinely verify DSO-submitted eligibility information and DSO suitability for participation in SEVP, and therefore does not have reasonable assurance that only eligible and suitable DSOs are participating in the program. Finally, ICE has not developed or implemented mandatory and fraud-specific training to improve DSOs’ compliance with program requirements and aid its efforts to detect fraud in the program. We are making the following seven recommendations to ICE: The Director of ICE should develop a fraud risk profile that aligns with identifies inherent fraud risks affecting the program, assesses the likelihood and impact of inherent fraud risks, determines fraud risk tolerance, and examines the suitability of existing fraud controls and prioritizes residual fraud risks, including residual risks posed by the recertification queue. (Recommendation 1) The Director of ICE should build on existing efforts to use data analytics by employing techniques, such as network analysis, to identify potential fraud indicators in schools petitioning for certification. (Recommendation 2) As ICE works to complete its efforts to hire additional SEVP adjudicators, the Director of ICE should begin notifying certified schools 180 days prior to, and requiring submission of complete recertification petitions by, the 2-year certification expiration date, consistent with regulation, and evaluate whether additional resources are needed. (Recommendation 3) The Director of ICE should, as practicable, verify the eligibility information provided to establish the immigration or citizenship status of lawful permanent residents and naturalized U.S. citizens, as well as U.S.-born citizens, who have been nominated or renominated to serve as DSOs. (Recommendation 4) The Director of ICE should develop an implementation plan for the project aimed at strengthening background checks for DSOs; that plan should outline how the project will be executed, monitored, and controlled. (Recommendation 5) The Director of ICE should implement mandatory DSO training and verify that the training is completed. (Recommendation 6) The Director of ICE should complete the development and implementation of its plans for mandatory fraud-specific training for DSOs. (Recommendation 7) We provided a draft of this report to DHS for its review and comment. In its written comments, reproduced in appendix V, DHS concurred with our recommendations and described specific steps it plans to take in response to all seven of our recommendations. DHS also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretary of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Rebecca Shea at (202) 512-6722 or shear@gao.gov or Rebecca Gambler at (202) 512-8777 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Table 1 contains information on the eight recommendations that we made to U.S. Immigration and Customs Enforcement (ICE) in our 2012 report, and ICE’s actions to address them. We closed each of these recommendations as implemented. This report is a public version of a sensitive report that we issued on November 20, 2018, which examined the efforts that U.S. Immigration and Customs Enforcement (ICE) has taken since our 2012 report to address fraud risks, including the extent to which ICE has (1) taken steps to strengthen its management of fraud risks in the Student and Exchange Visitor Program (SEVP), (2) implemented controls to address fraud risks in the school certification and recertification processes, and (3) implemented fraud risk controls related to the eligibility, suitability, and training of Designated School Officials (DSO).The sensitive report included information related to SEVP internal controls used to help prevent and identify noncompliance or fraud in the program. The sensitive report also discussed some planned actions to improve these internal controls, some of which the Department of Homeland Security (DHS) deemed to be sensitive and must be protected from public disclosure. This public report omits the information that DHS deemed to be sensitive including details associated with (1) the oversight of schools during the certification and recertification process, (2) our covert testing of SEVP certification internal controls, and (3) current and planned actions to oversee DSOs. Although the information provided in this report is more limited, it addresses the same objectives and uses the same methodology as the sensitive report. For our first objective, to evaluate the extent to which ICE has taken steps to strengthen its management of fraud risks in SEVP, we assessed actions ICE, particularly SEVP and the Counterterrorism and Criminal Exploitation Unit (CTCEU), have taken since 2012 to design and implement controls to address fraud in the postsecondary, vocational, and English language school certification and recertification process. We reviewed documents including regulations, processes and procedures, and guidance related to fraud risk management, school certification, and recertification processes, and the role of DSOs. We evaluated the extent to which ICE’s practices were consistent with Standards for Internal Control in the Federal Government and GAO’s A Framework for Managing Fraud Risks in Federal Programs. In particular, we analyzed ICE documentation, such as standard operating procedures, policy statements, and guidance for adjudicators to determine how ICE’s processes and systems identify and assess risk in SEVP, including the SEVP Risk Assessment Model and Framework, Risk Assessment Tool, Risk Register, and other internal guidance. In addition, we reviewed information from ICE’s current SEVP administrative, watch, and criminal investigative cases and analyzed information on past cases of SEVP fraud, including indictments. Also, we interviewed ICE officials within SEVP to evaluate the extent to which the program has taken steps to strengthen its management of fraud risks since 2012. We met with senior officials from SEVP, including SEVP’s Director and management of the Risk Management Support Team, School Certification Unit (Certification Unit), Analysis and Operations Center (Compliance Unit), Policy Team, and Field Representative Unit. We interviewed officials from ICE’s Office of the Principal Legal Advisor to discuss regulatory priorities and legal authorities related to fraud prevention and detection. We also interviewed officials from ICE’s Identity Benefit Fraud Unit and Domestic Operations to discuss their roles in SEVP-related fraud prevention. In addition, we met with officials from CTCEU headquarters, including the Student and Exchange Visitor Information System Exploitation Section and criminal investigators from 5 of the 26 ICE field offices to discuss past cases of SEVP-related fraud and steps taken to identify and prioritize fraud risk. We visited ICE field offices in Washington, D.C.; Los Angeles and San Francisco, California; Newark, New Jersey; and New York, New York. We selected these locations based on a mix of criteria, including the following characteristics: (1) number of ongoing investigations of certified schools; (2) reported previous and current experience investigating SEVP-related fraud; (3) number of field representatives assigned to or located near the field office; and (4) number of schools that were located proximate to the field office and that were either pending recertification, as of July 2017, or have been recertified since August 2016. As we did not select a probability sample of ICE field offices to interview, the results of these interviews cannot be generalized to all of ICE’s 26 field offices. However, the interviews provided us with perspectives of ICE officials responsible for conducting school fraud investigations, including their views on the process SEVP has established for certifying and monitoring schools, fraud, and national security vulnerabilities related to foreign students, and any challenges field offices have faced in their investigations. We conducted a network analysis utilizing both public and proprietary information associated with currently certified schools to determine the potential to utilize additional data analytics to aid fraud risk-management efforts in SEVP. To develop this analysis, we identified a list of schools that, as of July 2017, had been identified by ICE as either being under active criminal investigation or subject to additional oversight or administrative action due to compliance concerns. We also selected a list of SEVP-certified postsecondary schools without such identified concerns as of September 2017. We restricted our set of schools to those with at least 20 foreign students as of September 2017. In total, 2,439 schools comprising 170 with concerns and 2,269 without such concerns were analyzed. We then used an outside vendor to provide public and proprietary information such as descriptive information associated with these schools including addresses, businesses, and past executives. Using these data, we used network-analysis techniques to identify connections between both those schools with criminal or compliance concerns and schools without such identified concerns. We determined whether each of the postsecondary schools without compliance concerns were linked to any of those with compliance concerns via executive employment. Specifically, we identified instances in which an official associated with a school with criminal or compliance concerns was associated with another school not identified as having those concerns. The underlying logic behind this focus was that schools associated with an official linked to a school of concern may potentially indicate the need for further review of possible criminal or compliance concerns. To further validate this information, we conducted additional research using investigative databases and the Internet to try to verify the instances identified in our analysis such as by ensuring the time frames of the connection appeared relevant or to verify the identity of individuals and schools involved. While such connections are not proof of criminal or compliance problems, they may potentially be indicative of them. This is a diagnostic that has been used in other fraud-related network research. For our second objective, to evaluate the extent to which ICE has implemented controls to address fraud risks in the school certification and recertification processes, we assessed documentation describing SEVP’s school certification and recertification controls, interviewed headquarters and selected field-office ICE officials, and analyzed agency-provided recertification data. Specifically, we assessed SEVP’s standard operating procedures, including its Adjudicator’s Manual, training materials, and other guidance to determine whether the certification and recertification controls described in these documents addressed the high-risk indicators ICE identified in its Risk Assessment Tool. We used this analysis to determine any potential noncompliance and fraud vulnerabilities in these controls. We also assessed SEVP’s controls in these areas against Standards for Internal Control in the Federal Government related to risk management, as well as principles of the Framework for Managing Fraud Risks in the Federal Government. Additionally, we interviewed ICE officials in SEVP’s Certification Unit, which is responsible for adjudicating certification and recertification petitions, and the Compliance Unit, which is charged with monitoring schools for ongoing compliance with regulatory record-keeping and reporting requirements. To understand how ICE Homeland Security Investigations agents in the field offices work with officials in SEVP and the CTCEU to investigate school fraud, we conducted semistructured interviews with ICE agents in five field offices. We also interviewed ICE officials from SEVP’s Field Representative Unit as well as eight field representatives assigned to or located near the selected field offices to gather information on the representatives’ roles and activities in identifying and reporting potential school fraud. Further, we conducted covert testing of SEVP’s internal control activities related to the school certification process. Specifically, we submitted certification petitions and conducted other covert investigative work for three fictitious schools, each of which are subject to particular petition requirements. For one of the fictitious schools, we tested SEVP certification controls that require schools to submit complete documentation by submitting a petition for the school that was missing several of the required documents. For our second school, we tested SEVP controls requiring schools to schedule and complete a site visit conducted by an SEVP field representative, by submitting a completed petition for the accredited business school, but avoiding the site visit and requesting that our paperwork move forward without it. For our third fictitious school, we submitted a petition and participated in a site visit with SEVP officials, using a rented space as a fictitious school location. We tested SEVP controls related to verifying petition documentation, and whether SEVP site-visit officials followed established procedures for the site visit. For all three petitions, we used publicly available information to construct our scenarios. We then documented any actions taken by SEVP on the submitted petitions, such as completeness checks, investigative steps, adjudication decisions or requests to provide additional supporting documentation, among other things. Results for all three covert tests, while illustrative, cannot be generalized to the full population of petitions. For our third objective, to determine the extent to which ICE implemented fraud risk controls related to the eligibility and suitability of DSOs, we assessed guidance, training, and policies related to DSOs. Specifically, we reviewed regulations for DSO eligibility and SEVP guidance and standard operating procedures to determine whether supporting evidence provided to meet these requirements is being verified, including the Field Representative Unit’s Site Visit Standard Operation Procedure and the Certification Unit’s Adjudicator’s Manual. We evaluated the extent to which ICE’s practices for verifying eligibility were consistent with the Framework for Managing Fraud Risks in the Federal Government. In addition, we reviewed the current and planned documentation and procedures on ICE’s existing and planned background checks, including the existing documentation for DSO vetting against relevant databases, initial requirements for planned biometric screening, and a draft policy document for the planned checks. To gather additional perspectives, we interviewed ICE officials in headquarters and selected field offices. We also interviewed selected DSOs in the field. We identified leading practices for project planning in the Project Management Institute’s A Guide to the Project Management Body of Knowledge. In addition, we reviewed the best practices associated with developing and maintaining a reliable, high-quality schedule in the GAO Schedule Assessment Guide. In assessing current training and oversight for DSOs, we examined guidance, policies, and procedures for the SEVP Field Representative Unit and CTCEU’s Project Campus Sentinel. We assessed the implementation of these controls against criteria in Standards for Internal Control in the Federal Government and A Framework for Managing Fraud Risks in Federal Programs. We reviewed DSO training materials, including the Online Training for DSOs and the Study in the States website. To determine how ICE identifies fraud risk associated with DSOs, the controls in place for addressing and mitigating these risks, and its efforts to identify potential vulnerabilities in its controls, we met with ICE officials at headquarters and five selected field offices, as discussed above. To identify the extent to which they have DSO training and antifraud responsibilities and requirements, we interviewed selected field representatives. Furthermore, we interviewed DSOs at 17 selected certified postsecondary schools on their roles and responsibilities and training resources. We selected these officials because, as of September 2017, they constituted a group of representatives from certified schools of various types and sizes and were located in proximity to our previously selected ICE field-office locations. As we did not select a probability sample of DSOs to interview, the information we obtained from these school officials cannot be generalized. These interviews provided us with the perspectives of DSOs on their roles and responsibilities, training, and fraud risks within the program. Further, we interviewed officials from NAFSA, an association of international educators, to discuss the organization’s views on fraud risks within SEVP, and we reviewed an extract from the NAFSA Advisor’s Manual of federal regulations affecting foreign students and scholars. GAO’s A Framework for Managing Fraud Risks in Federal Programs states that, in planning the fraud risk assessment, effective managers tailor the assessment to the program by, among other things, identifying appropriate tools, methods, and sources for gathering information about fraud risks and involving relevant stakeholders in the assessment process (see fig. 11). On the basis of our analysis of U.S. Immigration and Customs Enforcement (ICE) data, the Student and Exchange Visitor Program (SEVP) withdrew certification for approximately 2,600 schools during the period of fiscal years 2013 through 2017 (see fig. 12). The Enhanced Border Security and Visa Entry Reform Act of 2002 states that a material failure of an SEVP-certified school to comply with the record-keeping and reporting requirements to receive nonimmigrant students shall result in the suspension for at least 1 year, or termination, of the school’s approval to receive such students. Under federal regulation, SEVP can deny an SEVP-certified school’s recertification petition or, as a result of a subsequent out-of-cycle review, can withdraw certification, if the school or its programs are no longer eligible for certification. Denial of recertification or withdrawal on notice as a result of out-of-cycle review may be for any valid and substantive reason, including failure to comply with record-keeping and reporting requirements, willful issuance by a DSO of a false statement, or not operating as a legitimate institution, among other bases. According to SEVP officials, denials resulting from recertification reviews are often based on historical discrepancies in the DSO’s data entry, record-maintenance and Form I-20 issuance issues, or a negative change in the school’s operating status, such as a loss of state licensure. By regulation, an appeal of a notice of denial or withdrawal must be made within 15 days after service of the decision. Schools denied recertification must, according to regulations, wait at least 1 calendar year from the date of denial of recertification or withdrawal notice before being eligible to petition again for certification. If, upon the completion of an out-of-cycle review, SEVP determines that a school has failed to sustain eligibility or has failed to comply with the record-keeping, retention, reporting, or other requirements, SEVP will institute withdrawal proceedings by serving the school a notice of intent to withdraw SEVP certification. Failure of a school to respond to a notice of intent to withdraw within 30 days will result in an unappealable withdrawal of the school’s certification. At the conclusion of withdrawal proceedings, a school found to be ineligible for continued SEVP certification as a result of an out-of-cycle review will receive a notice of withdrawal. SEVP withdrew on notice approximately 211 certifications from fiscal years 2013 through 2017 (see fig. 12). If SEVP staff identify an issue during an out- of-cycle review that seems to be an error not warranting withdrawal, SEVP could issue a Remedial Action Plan to the school describing the issues it needs to address to retain its program eligibility. According to SEVP officials, once they have gathered enough evidence and made the decision to withdraw the school’s certification, SEVP can temporarily terminate the school’s ability to issue Forms I-20 to students. For example, SEVP officials explained that if a school that is otherwise in compliance lets its accreditation lapse, SEVP may revoke its authority to issue Forms I-20 until it renews its accreditation. Regarding automatic withdrawals, SEVP will serve a notice of intent to withdraw SEVP certification to the school 30 days prior to its certification expiration date if, up until that point the school has failed to file a complete petition for recertification. From fiscal year 2013 through fiscal year 2017, SEVP automatically withdrew 1,763 certifications (see fig. 12). SEVP will not accept a petition for recertification and the school will be automatically withdrawn immediately if such school has effectively relinquished its SEVP certification by not petitioning for recertification, abandoning its petition, or not submitting a complete recertification package by the certification expiration date. Certified schools can also voluntarily withdraw their certification at any time. In addition to the contacts named above, Latesha Love (Assistant Director), Kathryn Bernet (Assistant Director), Nick Weeks (Analyst-in- Charge), David Aja, David Dornisch, Gabrielle Fagan, April Gamble, Gina Hoover, Lauren Kirkpatrick, Kirsten Lauber, Barbara Lewis, Sasan J. “Jon” Najmi, Robin Nye, George Ogilvie, Ramon Rodriguez, Constance Satchell, Sabrina Streagle, Shana Wallace, and Helina Wong made key contributions to this report.", "summary": "As of March 2018, more than 1.2 million foreign students in the United States were enrolled in 8,774 schools certified by SEVP. ICE is responsible for managing SEVP, which certifies schools to enroll foreign students. Various ICE offices have a role in preventing, detecting, and responding to potential fraud in the program. GAO was asked to review potential vulnerabilities to fraud in SEVP. GAO examined, among other things, the extent to which ICE (1) implemented controls to address fraud risks in the school certification and recertification processes and (2) implemented fraud risk controls related to DSO training. GAO analyzed ICE policies and documentation, including fraud risk guidance and procedures for school certification and recertification; analyzed 2013 through 2017 recertification data; and interviewed officials from five ICE field offices that GAO selected based on their experience investigating program fraud. GAO also interviewed officials from 17 selected schools located near these ICE field offices. This is a public version of a sensitive report that GAO issued in November 2018. Information that DHS deemed sensitive has been omitted. The Department of Homeland Security's (DHS) U.S. Immigration and Customs Enforcement (ICE) has identified several fraud risks to the Student and Exchange Visitor Program (SEVP). As shown in the figure below, these include risks associated with school owners and designated school officials (DSO) who help ICE oversee students in the program. These fraud risks may occur as schools apply to become SEVP-certified, accept foreign students, and apply for recertification every 2 years. ICE has implemented controls to address fraud risks related to school certification, but long-standing delays in recertifying these schools exacerbate fraud risks. By statute and regulation, ICE must conduct recertification reviews every 2 years to ensure that schools continue to meet program requirements—an important fraud risk control. Between 2013 and 2017, ICE recertified about 12,900 schools. However, according to ICE officials, they have been unable to meet the 2-year time frame and, as of June 2018, had 3,281 recertification petitions waiting for review. To help manage its queue, ICE has lengthened the period between recertification reviews by extending schools' certification expiration dates by 180 days, which is inconsistent with its regulation and may allow fraudulent schools to operate longer without detection. Although ICE is taking steps to increase resources for recertification, it is unclear whether these steps will ensure recertification is conducted consistently with ICE regulations. ICE relies on DSOs to, among other things, update and maintain foreign-student data in ICE's foreign-student information system and report suspected fraud to ICE. However, ICE does not provide DSOs with training that addresses fraud risks to the program. In June 2018, ICE officials stated that they plan to develop this fraud training for DSOs, but do not have documented plans or timelines for when it would be completed. By developing these plans, the agency would be better positioned to ensure that DSOs receive the training needed to address potential fraud in the program. GAO is making seven recommendations, including that ICE (1) notify schools 180 days prior to the 2-year certification expiration date, as required, and evaluate whether additional resources for recertification are needed, and (2) develop a plan to implement fraud-specific training for DSOs. ICE concurred with all of GAO's recommendations.", "document_type": "gao"}
{"report": "Zika is spread to people primarily through the bite of an infected mosquito but can also be transmitted from mother to child during pregnancy or from person to person through sexual contact or blood transfusion. The disease can cause symptoms that include fever, rash, conjunctivitis (“pink eye” where the eyes appear red or pink), and joint and muscle pain. Although most people with Zika have only mild symptoms or none at all, Zika in pregnant women has been linked to adverse pregnancy outcomes, such as miscarriage and stillbirth, and severe birth defects. Zika can be passed to the fetus and cause a birth defect of the brain called microcephaly and other severe brain defects, according to CDC. Zika is also linked to other problems such as Guillain-Barré syndrome, an uncommon condition of the nervous system. Although at present no vaccine has been approved by the U.S. Food and Drug Administration to prevent Zika, several vaccines are in different phases of development. Zika was first identified in the Zika Forest in Uganda in 1947 and caused only sporadic human disease until 2007. In 2007, Zika was detected in Yap State, Federated States of Micronesia, and subsequent outbreaks occurred in Southeast Asia and the Western Pacific. In 2014, Zika spread east across the Pacific Ocean to French Polynesia, then to Easter Island. In May 2015, Brazil documented the first case of locally acquired Zika transmission in the Americas. See figure 1 for a timeline of the Zika outbreak and the U.S. Zika response overseas. According to WHO, in November 2015, Suriname, El Salvador, Guatemala, Mexico, Paraguay, and Venezuela reported cases of locally acquired Zika, followed by Panama, Honduras, French Guiana, Martinique, and Puerto Rico in December 2015. Zika continued to spread throughout the region, and on February 1, 2016, WHO declared that the recent association of Zika with clusters of microcephaly and other neurological disorders constituted a public health emergency of international concern. In November 2016, WHO declared an end of the public health emergency of international concern regarding microcephaly, other neurological disorders, and Zika. However, WHO announced that Zika and the associated health outcomes remained a significant public health challenge requiring intense action. Zika spread to multiple countries throughout the globe but primarily affected countries in Latin America and the Caribbean region. According to WHO, as of March 2017, transmission of the Zika virus was occurring in 79 countries or territories, most of which are located in the Western Hemisphere. According to WHO, from 2015 to 2017, there were approximately 583,000 suspected and 223,000 confirmed cases of Zika virus transmission in the Western Hemisphere. See figure 2 for the cumulative Zika incidence rates in each country in Latin America and the Caribbean from 2015 to 2017. In February 2016, the President submitted a request to Congress for emergency funding to enhance ongoing U.S. efforts to prepare for and respond to Zika, including a request for funding for USAID and State to respond to the outbreak overseas. In addition, in April 2016, USAID and State notified Congress of their intent to repurpose $215 million of fiscal year 2015 supplemental Economic Support Fund Ebola funding for the U.S. Zika response overseas, which included $78 million for CDC international Zika activities. In September 2016, Congress appropriated about $175.1 million in supplemental funding to USAID and State in the Zika Response and Preparedness Appropriations Act, 2016, for the U.S. Zika response overseas. USAID activities initially began in five countries—Haiti, Honduras, Guatemala, El Salvador, and Dominican Republic—based on an assessment of their Zika risk and limited host government capacity to prevent the spread and respond to the impact of the virus. USAID ultimately supported activities in 26 countries in the Latin America and Caribbean region. As of September 30, 2018, USAID and State had obligated about $385 million (99 percent) of the total $390 million available for the U.S. Zika response overseas and had disbursed approximately $264 million (68 percent). Specifically, USAID had obligated all of its funds available for the Zika response and disbursed about two-thirds, and State had obligated and disbursed more than three-quarters of its funding for Zika. USAID and State had disbursed a higher proportion of the repurposed Ebola funds than the funds appropriated in the Zika Response and Preparedness Appropriations Act, 2016. See figure 3 for USAID and State Zika response funding appropriations, obligations, and disbursements as of September 30, 2018. Of the $215 million in repurposed Ebola funds, USAID and State had obligated $215 million (100 percent) and had disbursed almost $201 million (93 percent) as of September 30, 2018. Of the approximately $175 million appropriated in the Zika Response and Preparedness Appropriations Act, 2016, USAID and State had obligated about $170 million (about 97 percent) and had disbursed about $63 million (36 percent) as of September 30, 2018. As of September 30, 2018, USAID had obligated all funds available for the Zika response and had disbursed about two-thirds, from three accounts. USAID has two sources of funding for Zika response activities: $211 million of fiscal year 2015 supplemental Economic Support Fund Ebola funding repurposed for the Zika response and about $155.5 million provided in the Zika Response and Preparedness Appropriations Act, 2016—including $145.5 million and $10.0 million through the Global Health Programs and Operating Expenses accounts, respectively—for a total of $366.5 million. As of September 30, 2018, USAID had obligated approximately $366.5 million (100 percent) and had disbursed approximately $245 million (67 percent), from the Economic Support Fund, Global Health Programs, and Operating Expenses appropriations accounts. See figure 4 for USAID Zika response funding obligations and disbursements by account. USAID obligated all funding for Zika response activities within a year after it was repurposed or appropriated. As of September 30, 2018, USAID had disbursed a higher proportion of repurposed fiscal year 2015 supplemental Economic Support Fund Ebola funding (93 percent) compared with Global Health Programs and Operating Expenses funding (28 percent and 72 percent, respectively), which was appropriated in the Zika Response and Preparedness Appropriations Act, 2016, in September 2016. The $211 million in Economic Support Fund obligations supported 56 USAID activities, as well as a $78 million interagency transfer to CDC. The $145.5 million in Global Health Programs obligations supported 25 activities and program support. Obligations for USAID-supported activities ranged from $12,000 to $37 million and included support for activities such as the procurement of insect repellent to assist pregnant women in avoiding Zika infection and strengthening the ability of civil society and community networks to disseminate information related to Zika. CDC supported 25 activities that ranged from $276,000 to $13.6 million, including activities such as collecting and analyzing public health data, conducting epidemiological studies to better understand the prevalence of Zika and related risk factors, building laboratory capacity, and providing training to conduct Zika virus testing. As of September 30, 2018, State had obligated and disbursed more than three-quarters of funding available for the Zika response, from two accounts. State has two sources of funding for Zika response activities: $4 million from a fiscal year 2015 supplemental Economic Support Fund appropriation for the Ebola response that was repurposed for the Zika response and about $19.6 million provided in the Zika Response and Preparedness Appropriations Act, 2016, the majority of which was provided through the Diplomatic and Consular Programs account, for a total of about $23.6 million. As of September 30, 2018, State had obligated and disbursed about $18.3 million (almost 78 percent) from the Economic Support Fund and Diplomatic and Consular Programs accounts. See figure 5 for State Zika response funding obligations and disbursements by account. Under the Zika Response and Preparedness Appropriations Act, 2016, State was provided almost $14.6 million through the Diplomatic and Consular Programs account, $4 million through the Emergencies in Diplomatic and Consular Services account, and $1 million through the Repatriation Loans Program account, for a total of almost $19.6 million. In September 2017, State notified Congress of its intent to transfer the $4 million from the Emergencies in Diplomatic and Consular Services account and $870,000 from the Repatriation Loans Program account to the Diplomatic and Consular Programs account. These transfers resulted in a total of $19.5 million available under the Diplomatic and Consular Programs account and $130,000 under the Repatriation Loans Program account. The $4 million in Economic Support Fund obligations supported research and development activities by the International Atomic Energy Agency to control disease-carrying mosquito populations. The $14.3 million in Diplomatic and Consular Programs obligations supported activities including medical evacuations to protect the health of pregnant U.S. government personnel and eligible family members, mosquito abatement training and other measures to reduce Zika risk to overseas staff, as well as public diplomacy efforts to further inform journalists and the public about the U.S. response to Zika. In their reporting to Congress on the uses of Zika funds, USAID and State included some country information but did not track or provide information on funding uses broken down on a country basis. In October 2016, USAID and State submitted a consolidated report to the appropriations committees on the anticipated uses of funds made available to USAID and State by the Zika Response and Preparedness Appropriations Act, 2016, in response to a reporting requirement in Section 203 of the act. After the initial submission, the act required the agencies to update and submit the report to the committees on appropriations every 60 days until September 30, 2017. The initial report described ongoing Zika response activities in five countries as well as planned activities in additional countries. Subsequent reports listed specific countries where USAID and State supported Zika response activities. However, USAID and State did not provide information to Congress on the uses of funding appropriated by the Zika Response and Preparedness Appropriations Act, 2016, broken down by country. The reports also included obligation and disbursement information for the fiscal year 2015 supplemental Economic Support Fund Ebola funding that was repurposed for the international Zika response; however, similar to the information provided regarding the funds appropriated by the Zika Response and Preparedness Appropriations Act, 2016, the reports’ information on the use of the repurposed Ebola funds was also not broken down by country. USAID officials told us that Zika activities were designed to be implemented on a regional and multicountry basis. While over 95 percent of all U.S. government funds available for the Zika response overseas were obligated by USAID, and the agency had a number of financial tracking systems in place, the agency did not take steps to record its funding by country at the outset of Zika response programming. Specifically, USAID officials noted that the contracts and grants the agency had signed with its implementing partners did not include provisions requiring partners to provide information to USAID that broke down their use of funds by country. Consequently, USAID was unable to track the uses of Zika funds on a country basis. Federal internal control standards state that management should use and communicate the necessary quality information both internally and externally to achieve the entity’s objectives and address related risks. According to USAID officials, tracking information on the uses of Zika response funding broken down by country would be helpful in the future for mission directors, chiefs of missions, and partner-country ministries of health, some of whom have requested this information. Moreover, data on USAID funding to address future infectious disease outbreaks if broken down by uses in each country could provide additional useful information to decision makers in assessing risks and planning responses. The ability to compile funding by country when responding to future infectious disease outbreaks would enable USAID to provide key decision makers, including Congress and agency officials, with additional information to better support spending oversight and inform budgetary and planning decisions. As part of the U.S. Zika response overseas, USAID provided assistance to several countries in the Caribbean, Central America, and South America and conducted a variety of activities related to mosquito control, public awareness, capacity building, and research. In support of mosquito control, USAID’s Zika AIRS Project (ZAP) conducted activities that included Entomological monitoring: collecting and reporting information on the location and population of mosquitoes; Larviciding: placing agents that kill mosquito eggs in likely breeding sites, such as water receptacles; Source reduction interventions: facilitating the removal or mitigation of likely breeding sites, such as tires, pots, barrels, or anything that may allow for standing water; and Indoor residual spraying: spraying insecticide that has a lasting effect in houses. We observed mosquito control activities during our fieldwork. For example, in Honduras we followed a team as they went house to house to implement and facilitate mosquito control activities. They collected information from mosquito egg traps, which serve as indicator of breeding activity, and recorded it for monitoring purposes. They also examined the premises for potential mosquito breeding sites, treated susceptible areas such as wash basins with larvicide, and spoke with residents about picking up trash and covering outdoor plant pots to reduce potential breeding sites. To support raising public awareness of the risk of Zika virus and to promote behavior change to reduce the spread of the disease, USAID implementing partners such as the Red Cross and CARE told us that they collaborated with communities, local government, and schools to communicate information about Zika. For example, in Trinidad, the Red Cross conducted educational campaigns at schools to improve students’ awareness. During our fieldwork, we observed a session led by adult volunteers during which children played games and engaged in discussions designed to teach Zika prevention and response methods. Implementing partners told us that the impact of such efforts extends beyond those reached directly; for example, they said the children who learned about Zika risks and prevention also conveyed the knowledge to their families, who in turn may pass it on to friends or others in the community. In Peru, CARE worked with schools to develop written education guides for application in the classroom and conducted communication campaigns. During our fieldwork, we went to schools and observed students delivering oral presentations on Zika risks and prevention. In addition, we witnessed other student activities, such as classroom discussions and art projects focused on Zika, designed to demonstrate understanding, raise awareness, and promote behavior change. To support capacity building, the Applying Science to Strengthen and Improve Systems (ASSIST) activity, which USAID funding supported, focused on improving Zika-related health services. Specific efforts included conducting a baseline assessment of the quality of care, improving clinical guidelines, training health care providers, and implementing a quality improvement program. During our fieldwork in Honduras, we visited a hospital and met with ASSIST-supported health workers who told us that they applied new guidance in their practice, and as a result, improved care in areas including counseling, screening, diagnosis, and follow-up of those affected by Zika. We also visited a hospital in Dominican Republic, where health care workers stated that they collaborated with ASSIST in responding to Zika by training staff and producing guidance materials. These activities raised awareness, increased prevention efforts, and improved care, according to health care workers. USAID supported research, training, and innovation activities through its “Grand Challenge” program as well as its interagency agreement with CDC. USAID launched a series of Grand Challenge efforts, providing $30 million in grants to foster innovation on new methods and technologies to respond to Zika. One grant, for example, supported the World Mosquito Program’s research into the feasibility and effectiveness of infecting mosquitoes with bacteria to hinder transmission of the Zika virus. We visited the program’s operations in Colombia, met with scientists, and observed the breeding lab. Program scientists told us that initial efforts have been promising and that if more tests prove successful, the potential for reducing Zika transmission could be significant. Another USAID Grand Challenge grant supports research into the possible use of genetically modified yeast to prevent mosquito eggs from hatching. We spoke with scientists, lab technicians, and viewed facilities supported by this grant in Trinidad during our field work. Scientists stated that yeast attracts mosquitoes and is inexpensive, commonly available, and environmentally friendly. Testing is ongoing, but if successful, the approach could help reduce populations of mosquitoes in critical areas, according to the scientists. The USAID–CDC interagency agreement identifies a range of activities that involve technical assistance to help strengthen surveillance, emergency operations and management, and epidemiological investigations and research. One CDC activity, for example, focuses on supporting public health surveillance and epidemiological studies to better understand the prevalence and risk factors for severe health outcomes related to Zika. Another activity aims to build laboratory capacity in areas such as Zika diagnostic test production and distribution. In addition, the objectives of CDC’s Field Epidemiology Training Program are to train qualified professionals, build sustainable capacity for detecting and responding to health threats, and develop in-country expertise so that disease outbreaks can be detected locally and prevented from spreading. In Dominican Republic, CDC officials told us that this program delivers 3 months of classroom and field project training, and that as of August 2018, four cohorts of approximately 80 students each had completed the training. CDC officials told us that in addition to implementing various activities, CDC’s Central America Regional Office in Guatemala played an important role in facilitating U.S. government cooperation with Colombia, which had the second largest outbreak of Zika after Brazil. We reviewed status reports for six USAID activities that received among the highest amounts of funding, and each identified various results. Below, we describe the activities and examples of reported results. For more information, see appendix II. ASSIST: This activity sought to strengthen Zika-related health services and systems in Latin America and the Caribbean with a focus on pregnant women, newborns, and women of reproductive age. ASSIST reported that it conducted virtual and in-person training, courses, and workshops on Zika prevention, diagnosis, and care. ASSIST also reported that 8,133 health care workers had been trained as of March 2017, and that its efforts had supported the development of Zika care protocols and guidelines with a new emphasis on clinical care and support for affected infants and families. ASSIST further reported that through March 2018, 75 percent of children affected by Zika in Dominican Republic received specialized care at Hospital Infantil Robert Reid Cabral, an ASSIST- supported hospital in the capital, Santo Domingo. Red Cross: This activity aimed to reduce risks associated with Zika infection through community involvement, sharing lessons learned, and improving practices. The Red Cross reported that its communication efforts reached approximately 3,000 students, 29 communities, and almost 140,000 people via TV, radio, and social media engagement, providing them with information on risk and protection methods. Zika AIRS Project (ZAP): This is a mosquito control activity focused on reducing Zika transmission in Latin America and the Caribbean. Specific activities supported by USAID funding included entomological monitoring, larviciding, source reduction interventions, and indoor residual spraying. ZAP reported that five countries (El Salvador, Guatemala, Haiti, Honduras, and Jamaica) implemented comprehensive mosquito control activities. Population Services International: The purpose of this activity was to improve the capacity and raise awareness of people in countries affected by and at risk of Zika and other vector-borne diseases. Population Services International reported that through March 2018, 35 health providers in Dominican Republic, El Salvador, and Guatemala had been trained in raising awareness about Zika prevention and the use of printed educational materials. In addition, 1,006 pregnant women received counseling on Zika prevention, and 967 received prevention kits containing condoms, mosquito repellent, and printed educational materials. Additionally, 227 pharmacy attendants from 195 pharmacies received information on Zika prevention. Save the Children’s Community Action on Zika (CAZ): The goal of this project was to reduce Zika transmission and minimize the risk of Zika-related microcephaly and other neurological disorders. The project focused on helping the most vulnerable through community- based prevention strategies in Colombia, Dominican Republic, and three Central American countries. CAZ reported that it had reached approximately 65,000 students and trained 3,838 community agents and volunteers who supported efforts to strengthen the capacity to prevent Zika in 921 communities. United Nations Children’s Fund (UNICEF): This activity focused primarily on four countries: Guatemala, El Salvador, Honduras, and Dominican Republic. UNICEF worked to promote the adoption of prevention behaviors among at-risk populations through actions to raise awareness at multiple levels: individual, interpersonal, community, institutional, and national policy levels. UNICEF reported that these efforts reached more than 5.5 million people with key risk- communication messages and more than 150,000 people through coordinated social mobilization and person-to-person communication. For example, in Guatemala, UNICEF worked with a local partner to train young people and adolescents in schools and social groups to lead prevention activities in their communities. Moreover, around 25,000 pregnant women benefited from counseling sessions on Zika- prevention behaviors. In response to Zika, State conducted public awareness and communication initiatives, medical evacuations for overseas staff, and other activities. According to a State official, State conducted Zika-related public outreach to U.S. citizens abroad through social media and the Smart Traveler Enrollment Program, a service that provides information from U.S. embassies about local safety conditions. According to a State official, State also implemented public diplomacy activities related to Zika awareness and communication. For example, one activity aimed to raise awareness of vector-borne diseases such as Zika and collect information on insect breeding grounds. Another supported the addition of a science envoy who focused specifically on Zika and mosquito-borne diseases. In addition, according to a State official, State conducted Zika-related medical evacuations as part of those normally offered to female staff who became pregnant while serving abroad. State’s medical services division also supported overseas posts by purchasing and distributing mosquito repellent. State officials also told us that they coordinated Zika response efforts internally and externally. For example, State participated in a U.S. government interagency group led by CDC to exchange information on Zika and coordinated with other agencies on the response effort. Over the course of our fieldwork, USAID and implementing partner officials identified two key challenges to the implementation of Zika response activities. The first was the long-term sustainability of Zika response activities. The second was the timely implementation of Zika response activities in countries without bilateral USAID health programs. While USAID took steps to address the challenge related to sustainability, it only partially mitigated the challenge to timely implementation of Zika response activities in countries without bilateral USAID health programs. Agency and implementing partner officials identified the sustainability of Zika response efforts as a key challenge. While USAID did not intend to continue U.S. Zika response activities after the one-time emergency funding, sustainability was a consideration and posed a challenge due to the short implementation time frame, according to agency and implementing partner officials. One official further elaborated that Zika funding efforts occurred during the acute phase of the outbreak, which made it difficult to focus on long-term needs. For example, an implementing partner said that Zika-affected children require long-term care that host country governments may not be able to support after U.S. assistance ends. In addition, host country government officials, U.S. government officials, and implementing partners said that some Zika activities may not be sustainable after U.S. assistance is finished due to a lack of funds and limited capacity to continue the work. To address this challenge and support the long-term continuation of Zika response activities, implementing partners aligned their activities with those of host country governments and other organizations. Implementing partners reported working with governments and other organizations to incorporate Zika activities into their plans and practices so they could continue over the long term. One implementing partner and the Dominican Republic’s Ministry of Health, for example, planned mosquito control efforts together, and a Ministry of Health official said they intend to continue those control efforts after the end of Zika funding. Implementing partners in various countries also stated that Zika activities brought broader benefits to mosquito control, disability services, maternal health care, surveillance efforts, and emergency preparedness, which facilitated partners’ efforts to align their Zika response activities. For example, an implementing partner reported using Zika funding to develop organizational guidelines for treating Zika-affected children, which will be used by the health care system in Dominican Republic to treat children with related disabilities in the long term. According to some implementing partners in countries we visited, they developed Zika protocols and guidelines in response to new scientific information, trained government and other personnel on the protocols, and worked with officials of host country governments and other organizations to encourage adoption of Zika activities. For example, according to an agency official, an implementing partner in Peru developed a curriculum for epidemiologists and trained them on how to detect and contain mosquito-borne diseases, such as Zika. The agency official said that the implementing partner shared the training curriculum and materials with Peru’s Ministry of Health so it could continue the trainings after the end of Zika funding. According to implementing partners, they also involved local communities in activities to increase community ownership and address sustainability. For example, an implementing partner official said they trained a cadre of community volunteers in Guatemala and El Salvador on behavior change practices so that they can continue activities after the end of Zika funding. In addition, implementing partner officials said that engaging with communities to learn about needs and resources is important to continued community interest in activities. For example, an implementing partner that works with communities on health priorities developed an approach that includes a toolkit for identifying a community’s specific risks for Zika and the efforts best suited to helping the community eradicate mosquito breeding sites. In places affected by violence, some implementing partners engaged with communities to better understand how to prioritize community worker and volunteer safety to enable the continuation of activities. For example, an implementing partner in Guatemala engaged with local communities to understand areas they recommended health workers avoid due to safety concerns. Agency and implementing partner officials described timely implementation of activities in some countries without bilateral USAID health programs as a second key challenge. Twenty-two out of the 26 countries where USAID implemented its Zika response activities were countries without bilateral USAID health programs. USAID officials stated that, as a result, there were no USAID health program officials present in these countries to build on relationships with host country health officials and help facilitate the start of implementing partners’ activities during the Zika response. USAID officials noted two reasons that working with host country governments took time. First, some U.S. Zika response activities started after a decline in Zika cases, when some host country governments were no longer as focused on countering the disease. Implementing partners responded to this situation by identifying related health service improvements that could stem from implementing a Zika response and were of interest to the host country governments. Second, agency and implementing partner officials said that in some countries without bilateral USAID health programs it also took time to identify the appropriate points of contact and establish relationships—preliminary steps needed to obtain approval from the host country government before activities could get underway. According to USAID officials, these relationships are critical to navigating bureaucratic systems and assist in designing activities that meet the needs of host country governments and communities, which are needed for timely implementation. USAID took some steps to address the timely implementation challenge in countries without bilateral health programs. For example, according to USAID officials, USAID worked with multilateral partners that had a health presence in those countries and relied on regional field-based Zika coordinators to build relationships with in-country points of contact. As noted above, however, agency officials indicated that Zika response activities took additional time to deploy in some of the countries without bilateral USAID health programs. Further, implementing partners reported it took additional time to start up activities in those countries because of the time it took to obtain approval for them from the ministries of health. For example, one implementing partner reported that activity startup was postponed for nearly 3 months until it received approval from the host country government. Another implementing partner said it was a challenge to get information on Zika from the host country government or establish dialogue until USAID officials became involved. USAID officials also said that efforts to start and integrate Zika response activities in countries with ongoing USAID health programs did not face a number of the obstacles to timely implementation experienced in countries without bilateral USAID health programs. According to federal internal control standards, agencies should design control activities, such as a plan, to achieve their objectives and address related risks, such as the challenge related to timely implementation. In an effort to enhance its planning for outbreaks, USAID developed an infectious disease response plan in July 2018 during the time frame of our review. However, the plan does not provide specific guidance on how to address the challenge of initiating emergency response activities in countries without bilateral USAID health programs, such as by noting particular practices that implementing partners and other officials can use to address that challenge. For example, our fieldwork and interviews with USAID officials indicate that the following may be helpful practices for infectious disease response: Immediately establish an in-country working group that includes implementing partners, host country government officials, and U.S. government officials to help initiate and coordinate outbreak response. Communicate a current list of health ministry and other relevant government officials to implementing partners and other officials so they can quickly identify the appropriate points of contact. According to USAID officials, USAID missions maintain regular contact with host country governments, maintain contact lists, and participate in coordination meetings. However, in the case of overseas Zika response, some implementing partner officials in the field told us that they did not initially know who to contact in the host country government. Likewise, a host country government official told us that a working group on Zika outbreak response was not established until after officials recognized that implementing partner and host country government officials did not have regular channels of communication. By taking steps to improve planning for countries without bilateral USAID health programs—such as by adding specific guidance for initiating emergency response activities in such countries to its July 2018 plan—USAID would be better positioned to quickly build relationships with health ministry and other key government officials in host countries and thus be better able to provide a timely infectious disease response to future outbreaks. The Zika virus quickly spread to dozens of countries in 2015 and 2016, prompting WHO to declare the virus and associated health risks an international public health emergency. As future infectious disease outbreaks arise, Congress will be called on to fund overseas response efforts, as it did with the Zika outbreak, and USAID is likely once again to play a vital role in those efforts. Because USAID did not provide key decision makers with information on how Zika funding was distributed across the various countries where it conducted response activities, decision makers lack visibility into a key aspect of the overall U.S. Zika response overseas. The ability to compile this information by country when responding to future infectious disease outbreaks would enable USAID to provide key decision makers, including Congress and agency officials, with additional information to better support spending oversight and inform budgetary and planning decisions. Further, while USAID took steps to address the challenge of sustaining Zika response activities over the long term, it did not fully mitigate the challenge of timely implementation of activities in countries without bilateral USAID health programs. As a result, the agency’s response to Zika took additional time in some countries without bilateral USAID health programs. Infectious disease response planning that addresses countries without bilateral USAID health programs would better position USAID to quickly respond to infectious disease outbreaks, such as Zika, whenever the need arises. We are making the following two recommendations to USAID: The Administrator of USAID should take steps to ensure that, in responding to future public health emergencies of international concern, the agency is able to compile funding information broken down by country. (Recommendation 1) The Administrator of USAID should take steps to improve its infectious disease response planning to address the challenge of initiating response activities in countries without bilateral USAID health programs in a timely manner. (Recommendation 2) We provided a draft of this report to USAID, State, and CDC for review and comment. USAID provided written comments, which we have reproduced in appendix III. In its comments, USAID agreed with our findings and recommendations and identified a number of actions it plans to take in response. Specifically, USAID stated that in responding to future public health emergencies of international concern, it plans to compile and report on funding by country. USAID also outlined the steps it plans to take to develop additional guidance for USAID officials in countries without bilateral health programs. State and CDC did not provide formal responses. CDC provided technical comments, which we incorporated throughout the report, as appropriate. We are sending copies of this report to the appropriate congressional committees, the Administrator of USAID, the Secretaries of State and of Health and Human Services, and to other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. The Zika Response and Preparedness Appropriations Act, 2016, included a provision for us to review the status of U.S. Agency for International Development (USAID) and Department of State (State) actions to respond to Zika. In this report, we examine (1) the status of USAID and State funding for the U.S. Zika response overseas, (2) activities supported by these funds, and (3) challenges, if any, to implementing Zika response activities and actions taken to address any challenges. To examine the status of funding for U.S. Zika response overseas, we reviewed USAID and State’s reports to the Senate and House Committees on Appropriations mandated by Section 203 of the Zika Response and Preparedness Appropriations Act, 2016. We reviewed agency reporting submitted to Congress and discussed the reports with agency officials. We also reviewed USAID and State’s reports to the Senate and House Committees on Appropriations mandated by the Department of State, Foreign Operations, and Related Programs Appropriations Act, 2015. We obtained additional funding and activity information from USAID covering a period beyond that included in the reports to Congress. We reviewed the interagency agreement between USAID and the Centers for Disease Control and Prevention (CDC), outlining the CDC’s Zika response activities supported by $78 million in funds USAID obligated to CDC. We also obtained additional funding data from CDC and interviewed CDC officials to discuss the status of the agencies’ obligations and disbursements for Zika response activities. We analyzed USAID’s and State’s obligations and disbursements that the agencies reported as supporting the U.S. Zika response overseas, as of September 30, 2018. We analyzed agency obligations and disbursements across agency bureaus, funding accounts, and activities for the Zika response. Additionally, we interviewed officials from USAID and State to discuss the agencies’ obligations and disbursements for Zika response activities. We then reviewed the funding data and related documentation and consulted with USAID and State officials on the accuracy and completeness of the data. In the small number of instances where we identified potential issues or inconsistencies in the data, we contacted relevant agency officials and obtained information from them necessary to resolve the discrepancies. We assessed USAID’s tracking of funding data against federal internal control standards related to using quality information. We also utilized information from data reliability assessments for two recent GAO reports that utilized funding data from the same USAID and State systems. We determined that the data we used were sufficiently reliable for our purposes of examining USAID’s and State’s obligations and disbursements of the funds. To examine activities that USAID and State implemented in response to Zika overseas, we conducted fieldwork, analyzed agency documents, and interviewed officials. We examined the status and progress related to Zika response activities. We conducted a teleconference with officials in Haiti and El Salvador and conducted fieldwork in Barbados, Colombia, Dominican Republic, Guatemala, Honduras, Peru, and Trinidad and Tobago. We selected these countries based on the following criteria: (1) geographic diversity to include the Caribbean, Central America, and South America; (2) coverage of the main lines of effort (mosquito control, public awareness, capacity building, and research); and (3) the presence of activities under way that accounted for a significant portion of funding. During our fieldwork, we interviewed agency officials who played a role in Zika response activities, which included officials from State, USAID, and CDC. We also interviewed host government officials, implementing partners, health care workers, community volunteers, and researchers. In addition, we visited offices, toured facilities, and observed operations. We also attended a conference in Guatemala that addressed topics including status, successes, challenges, and lessons learned related to USAID’s Zika response. We reviewed agency documents describing the plans and goals of activities. We also analyzed progress reports of six activities to provide illustrative examples of results. We selected activities from those with among with the highest amounts of funding and that together represented approximately 33 percent of all USAID funding for Zika response and a range of countries, lines of effort, and types of implementing partners (such as nongovernmental organizations and international organizations). The sample is not generalizable to all of USAID’s Zika response activities. To examine challenges, if any, to implementing Zika response activities and actions taken to address any challenges, we interviewed U.S. government officials, USAID implementing partners, and host government officials, and we analyzed progress reports from selected USAID-funded Zika response activities. We identified key challenges based on the nature of the description and the degree to which a diversity of interviewees and documents made mention of them. We reviewed USAID policy, USAID’s infectious disease response plan, federal internal controls, implementing partner progress reports, and interviews with officials to determine what agencies did to address these challenges. We assessed USAID’s infectious disease response plan against relevant federal internal control standards. We conducted this performance audit from December 2017 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate, evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. To provide illustrative examples of the results of Zika response activities funded by the U.S. Agency for International Development (USAID), we analyzed implementing partners’ progress reports for a sample of six activities. We selected activities from those with among the highest amounts of funding and that together represented approximately 33 percent of all USAID funding for Zika response and a range of countries, lines of effort, and types of implementing partners (such as nongovernmental organizations and international organizations). Quantitative figures related to individual indicators listed below reflect the targeted population of the activity. Start dates and funding information provided below reflect the date of the report to Congress in which the activity first appeared and the associated funds obligated. The sample is not generalizable to all USAID’s Zika response activities. Table 1 presents the progress on key indicators as of March 2018 reported to USAID by the Applying Science to Strengthen and Improve Systems activity. The aim of the activity was to strengthen Zika-related health services and systems in Latin America and the Caribbean with a focus on pregnant women, newborns, and women of reproductive age. Table 2 presents the progress on key indicators as of May 2018 reported to USAID by the International Federation of Red Cross and Red Crescent Societies Global Health activity. The activity aimed to reduce risks associated with Zika infection through promoting community involvement, sharing lessons learned, and improving practices. Table 3 presents the progress on key indicators as of March 2018 reported to USAID by the United Nations International Children’s Emergency Fund activity. The activity aimed to promote the adoption of prevention behaviors among at-risk populations through actions targeting multiple levels of their environment: individual, interpersonal, community, institutional, and national policy levels. Table 4 presents the progress on key indicators as of September 2017 reported to USAID by the Save the Children Community Action on Zika project. The goal of the project was to reduce Zika transmission and minimize the risk of Zika-related microcephaly and other neurological disorders among the most vulnerable through community-based prevention strategies. Table 5 presents the progress on an illustrative selection of key indicators, by objective, as of March 2018 reported to USAID by the Population Services International activity. The purpose of the activity was to improve the capacity and raise awareness of people in countries affected by and at risk of Zika and other vector-borne diseases. Table 6 presents illustrative examples of accomplishments as of March 2018 reported to USAID by the Zika AIRS Project (ZAP). This was a mosquito control project focused on reducing Zika transmission in Latin America and the Caribbean. Specific activities included entomological monitoring, larviciding, source reduction interventions, and indoor residual spraying. In addition to the contact named above, Joyee Dasgupta (Assistant Director), Marc Castellano (Analyst-in-Charge), Diana Blumenfeld, Alana Miller, Fatima Sharif, David Dayton, Francisco Enriquez, Christopher Keblitis, Amber Sinclair, and K. Nicole Willems made key contributions to this report.", "summary": "The World Health Organization (WHO) declared the Zika virus a public health emergency of international concern in February 2016. According to WHO, as of March 2017, 79 countries and territories—including 48 in the Western Hemisphere—reported evidence of ongoing Zika transmission. In April 2016, USAID and State repurposed $215 million for Zika from funds appropriated for Ebola. Subsequently, the Zika Response and Preparedness Appropriations Act, 2016, provided over $175 million in supplemental funding to USAID and State to support Zika response efforts overseas. The act also included a provision for GAO to review the status of USAID and State actions to respond to Zika. In March 2019, the Centers for Disease Control and Prevention downgraded its international travel warning for Zika. This report examines (1) the status of USAID and State funding for U.S. Zika response overseas, (2) activities supported by these funds, and (3) implementation challenges, if any, and responses to any challenges. GAO reviewed information from U.S. agencies and met with U.S. and host country officials in Washington, D.C. GAO also conducted fieldwork in a nongeneralizable sample of countries in Latin America and the Caribbean where agencies implemented key response activities. The U.S. Agency for International Development (USAID) and the Department of State (State) obligated $385 million of the total $390 million available for international Zika response and disbursed $264 million as of September 2018. USAID obligated 95 percent of the total funding. USAID and State provided some country information to Congress but did not provide, or take steps to track, funding on a country basis. According to USAID officials, tracking funding information by country would be helpful in the future. The ability to compile funding by country when responding to future infectious disease outbreaks would enable USAID to provide additional information to key decision makers to better support spending oversight and inform budgetary and planning decisions. In response to the Zika outbreak, USAID and State supported a broad range of activities overseas, including mosquito control, research efforts, and medical evacuations. In one activity, USAID implementing partners monitored mosquito populations; in another, they researched methods to reduce Zika virus transmission rates. USAID implementing partners reported various outputs from selected activities. For example, an implementing partner reported that its awareness campaign on Zika prevention reached more than 5 million people. USAID faced sustainability and timeliness challenges in implementing its Zika response. According to agency and other officials, one-time funding and a short time frame posed a challenge related to sustainability of Zika response activities. In response, USAID worked to align activities with those of host governments and other organizations so they could continue in the long term. However, USAID's emergency response planning did not fully address the challenge of timely implementation of response activities in countries without bilateral USAID health programs. Twenty-two of 26 countries with Zika response activities did not have bilateral USAID health programs when the Zika outbreak began. As a result, response activities took additional time to deploy in some countries where USAID first had to establish relationships with key host country officials. Although USAID developed an infectious disease response plan in 2018, the plan does not provide guidance on how to address the timely implementation challenge in countries without bilateral health programs. By improving its planning, such as by adding such guidance in its 2018 plan, USAID would be better positioned to respond quickly to future disease outbreaks. USAID should (1) take steps to ensure it is able to compile funding information by country for future infectious disease emergency responses and (2) take steps to improve its infectious disease response planning. USAID concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Prevalence of Forced Labor in Thailand A 2020 report by the United Nation’s International Labor Organization included research on hundreds of Thai workers employed on seafood vessels or in processing facilities. The report found that nearly 10 percent of interviewees shared circumstances of involuntary work and coercion that, taken together, constituted forced labor. The report concluded that if its selection of interviewees is representative of workers employed across the Thai seafood industry, it would indicate that tens of thousands of workers in Thai fishing and seafood processing are working in forced labor conditions. Fishing activities are vulnerable to a number of illicit practices, including illegal, unreported, and unregulated (IUU) fishing and forced labor. IUU fishing is a broad term that includes a variety of fishing activities with social, economic, and environmental impacts and concerns all aspects and stages of the capture and marketing of fish, according to the United Nations’ Food and Agriculture Organization. Examples of IUU activities include fishing without a license or in excess of quota for certain species; failing to report catches or making false reports on catches; and conducting unauthorized transshipments, such as transfers of fish to cargo vessels at sea or port, according to NOAA. IUU fishing poses a threat to food security and socioeconomic stability in many parts of the world, according to NOAA documents, and many crew members on fishing vessels that engage in IUU fishing are from poor or underdeveloped areas. In addition, the Food and Agriculture Organization has noted that activities of those engaging in IUU fishing can constitute, lead to, or be associated with organized crime or other crimes, such as human trafficking, that may include forced labor. The International Labor Organization—an agency within the United Nations that, among other things, sets labor standards—has identified common indicators of forced labor in any work sector, as shown in figure 1. The presence of a single indicator in a given situation may, in some cases, indicate the existence of forced labor; in other cases, several indicators taken together may point to forced labor, according to an International Labor Organization document. Forced labor can occur at various points along the seafood supply chain—which can be long and complex—with limited visibility at various points, making it difficult to detect (see fig. 2). For example, forced labor may occur if workers are held on fishing vessels for long durations without adequate breaks or the ability to return to land. It may also occur in later stages of seafood processing, such as during filleting and canning the fish for export and sale to consumer, according to an NGO report. Additionally, various NGO reports indicate that during the harvesting and processing stages, seafood caught with forced labor may be combined with legally caught seafood, making illegal shipments more difficult to identify. For example, companies may combine catches from several smaller boats onto a bigger vessel before transporting it to shore for processing. Moreover, some seafood supply chains have an additional layer of complexity because low-value fish may not be directly exported but, rather, used as feed for farm-raised seafood that could eventually be imported into the United States. Forced labor related to this type of situation can be difficult to detect because the source of feed for farm- raised fish is an early step in a supply chain that occurs well before the seafood is imported into the United States, according to CBP officials. Federal law prohibits the import of goods made with forced labor. In particular, section 307 of the Tariff Act of 1930 prohibits the importation of goods, wares, articles, and merchandise mined, produced, or manufactured, wholly or in part, in any foreign country by convict labor, forced labor, or indentured labor under penal sanctions. TFTEA, enacted in February 2016, among other things, amended section 307 of the Tariff Act of 1930 by repealing an exception referred to as the consumptive demand clause. The consumptive demand clause permitted the importation of certain forced labor-produced goods if they were not produced “in such quantities in the United States as to meet the consumptive demands of the United States.” determination that merchandise was produced with forced labor in violation of section 307, the Commissioner will publish a formal finding. Various other laws and regulations are also relevant to IUU fishing and the importation of illegally harvested seafood into the United States. For example, under the High Seas Driftnet Fishing Moratorium Act, among other things, the Secretary of Commerce is charged with identifying and certifying countries that have fishing vessels engaged in IUU. According to NOAA officials, a negative certification may result in restrictions on the importation of some fish and fish products at a port of entry. NOAA also administers and enforces a number of statutes that include prohibitions on the importation of illegally harvested seafood, most notably the Magnuson-Stevens Fishery Conservation and Management Act and the Lacey Act. Under these authorities and others, NOAA enforces a number of trade monitoring programs. For example, NOAA administers the Seafood Import Monitoring Program (SIMP), which establishes permitting, data reporting, and recordkeeping procedures for the importation of 13 species of fish and fish products identified as being at particular risk of IUU fishing or seafood fraud. NOAA also implements the Tuna Tracking and Verification Program, which monitors domestic cannery production and importation of all frozen and processed tuna products to ensure compliance with federal requirements regarding dolphin-safe certification. A number of federal agencies and stakeholders are involved in or have an interest in forced labor and seafood-related efforts. In particular, CBP is responsible for enforcing violations of section 307 as part of its overall efforts to deter and detect violations of U.S. customs and trade laws at the more than 300 ports of entry into the United States. CBP’s enforcement efforts include, but are not limited to, actions to identify, detain, seize or exclude illegitimate imports, including imports produced by forced labor, counterfeits, and goods evading customs duties. After passage of TFTEA in 2016, CBP initiated new efforts to emphasize and focus on enforcement of section 307. Specifically, CBP formally established its Forced Labor Division in March 2018, within its Office of Trade. Since its inception, the Forced Labor Division has grown in size, according to CBP officials, with about 12 staff onboard as of the end of 2019, mainly comprised of analysts and international trade specialists. The Forced Labor Division does not have staff in other countries, but CBP can leverage foreign attachés from other CBP offices, to the extent they are available, to assist with enforcement of section 307, according to CBP officials. Staff in the Forced Labor Division also collaborate with others throughout CBP, including the Office of Field Operations, which, among other things, oversees operations at U.S. ports. Other federal agencies, such as the Department of State and the Department of Labor, conduct activities and collect information related to forced labor. According to CBP officials, CBP may use information from these federal agencies to help support its enforcement of section 307 for particular cases, including those involving seafood. Also, since the enactment of TFTEA in 2016, a number of working groups or task forces have been established, primarily involving U.S. federal agencies, to share information collected related to forced labor and imports, in general, as well as illegal activities involving fishing more specifically, in some cases (see app. II). In addition, numerous stakeholders, such as NGOs, also have an interest in combating forced labor, including forced labor related to the seafood industry. Often stakeholders’ interests in forced labor include other human rights issues or are broader than specific commodities such as seafood. Stakeholders may provide a variety of services to advocate for workers and identify potential forced labor. For example, some NGOs investigate potential human rights abuses of workers in the seafood industry while others focus on collecting data to help other interested stakeholders identify cases of forced labor. Other NGOs may work with importers who have interests in corporate social responsibility by helping them identify potential issues in their supply chain and comply with U.S. laws, including section 307. CBP enforces section 307 involving seafood imports generally following the same process it uses for any other goods suspected of being produced with forced labor imported into the United States, such as apparel, electronics, or consumer products. CBP carries out its process through its Forced Labor Division, in collaboration with other offices across CBP. According to CBP documents and officials, the process CBP uses to enforce section 307 generally includes four phases: (1) assessing leads to determine whether to initiate a case; (2) investigating cases; (3) reviewing information for legal sufficiency to propose a WRO; and (4) implementing the WRO and detaining shipments (see fig. 3). An importer has several options if CBP detains its shipment, including contesting the WRO or deciding not to enter the good into U.S. commerce. Phase 1: Initiation. CBP analysts within the Forced Labor Division assess leads for credibility when deciding whether to initiate cases involving potential forced labor at any point in the supply chain for a particular good. According to CBP officials, an analyst would examine, for example, whether an allegation made by an external party is credible, the goods in question are being imported into the United States, and sufficient information is available on potential forced labor to initiate and build a case. CBP officials told us that they do not have the resources to gather firsthand information on labor practices such as on fishing vessels or processing operations overseas, but that they can initiate cases based on information obtained from external sources. For example, CBP may receive information through its e- allegations system, which is CBP’s online mechanism for the public to report any suspected violations of trade laws or regulations related to the importation of goods into the United States. In addition, CBP may receive an allegation directly from external entities, such as NGOs; letters from industry or other concerned parties; and information from other U.S. government agencies. Publicly available information, such as media reports or NGO publications, can also serve as leads for CBP to self-initiate a case. If CBP’s initial evaluation shows further evaluation is warranted, CBP initiates a case and moves to the next phase. Phase 2: Investigation. CBP analysts investigate cases by collecting information from various sources to help determine whether the evidence “reasonably but not conclusively” indicates that goods being imported into the United States were produced with forced labor, according to CBP officials. For example, analysts may ask other federal agencies for information, such as import data, or speak with NGOs that may have information about a particular good or supply chain overseas. CBP officials also said they may investigate the strength of the information collected as part of their case development. In doing so, they said the Forced Labor Division uses the International Labor Organization’s forced labor indicators, among other standards, to help evaluate the sufficiency of evidence for forced labor conditions. If there is insufficient evidence to continue investigating a case, the Forced Labor Division may either close or suspend it pending further information, according to CBP officials. If there is sufficient evidence to propose a WRO, the case moves to the next phase. Phase 3: Legal review. CBP’s Forced Labor Division prepares a package, which includes an assessment of evidence and a justification for a proposed WRO for the goods suspected to be produced with forced labor, and submits it to CBP’s Office of Chief Counsel for legal review. To propose a WRO, CBP officials said that the package must provide sufficient evidence to reasonably but not conclusively indicate a violation of section 307. This entails having sufficient supply chain information showing importation of a good harvested, produced, or otherwise manufactured with forced labor, according to CBP officials. During its legal review, the Office of Chief Counsel may request additional information or have discussions with the Forced Labor Division. If the Office of Chief Counsel determines there is insufficient evidence to proceed with a WRO, then the Forced Labor Division may choose to close the case or suspend it and consider whether to seek additional information for the case. If the Office of Chief Counsel determines there is sufficient evidence to proceed with a WRO, then the Forced Labor Division prepares a WRO package to be presented to the CBP Commissioner for review and approval. Phase 4: Implementation. Once the CBP Commissioner issues a WRO, CBP is responsible for implementing the parameters of the WRO. According to CBP officials, numerous officials within CBP, including those at U.S. ports and the Centers of Excellence and Expertise, are responsible for implementation. CBP officials located at U.S. ports screen import data to identify, hold, and detain shipments associated with a WRO. When CBP detains a shipment subject to a WRO at a port of entry, the importer has the option to reexport the shipment to a different country. Alternatively, officials said the importer can contest the detention and provide additional information to show that the shipment did not contain forced labor elements. If CBP determines the importer has provided sufficient evidence, it allows the shipment to enter into U.S. commerce. Should the importer not provide additional information, the shipment can be excluded (not admitted into U.S. commerce) and/or seized and destroyed in certain circumstances, according to CBP officials. A WRO remains in place until the circumstances surrounding the original WRO change to indicate that forced labor is no longer part of the production or manufacturing process, and the CBP Commissioner revokes the order, according to CBP documents. CBP officials said the agency can also issue civil penalties to importers for forced labor violations for importing goods in violation of section 307, where appropriate. CBP’s Forced Labor Division tracks its cases of suspected forced labor violations, including seafood cases, in a case-tracking spreadsheet throughout the various phases of the enforcement process. The spreadsheet notes the status of each case as (1) open and active, (2) suspended, or (3) closed/inactive. At any given time, the Forced Labor Division may be working on a number of seafood cases that are in various phases of the enforcement process, according to CBP officials. Further, officials said the status of these cases changes as new information becomes available. Data CBP provided to us showed a small number of open and active cases as well as suspended cases that were related to seafood. CBP officials stated that they suspended these seafood cases partly because they lacked personnel to obtain additional information to further investigate the cases. In other instances, they said they may suspend cases while waiting for additional information, which may take significant time to obtain. From February 2016—when TFTEA was enacted—through March 2020, CBP issued 13 WROs for goods suspected of violating section 307, of which one involved seafood, according to CBP data (see table 1). The seafood-related WRO was for all seafood imports caught by the fishing vessel Tunago No. 61, registered in Vanuatu, an island nation in Oceania. After issuing the WRO in February 2019, CBP detained multiple shipments of seafood, according to CBP data, but revoked the order at the end of March 2020. CBP’s other WROs cover a variety of goods such as cotton, toys, food, and agricultural products. Six of the 13 WROs included imports from China, while several WROs included goods from African countries. As of March 2020, CBP officials said they had not issued any civil penalties for forced labor violations involving seafood imports. CBP officials told us they obtain and use information from a variety of external sources, including media reports, other federal agencies, and stakeholders, that can help them initiate new forced labor-related cases or advance existing ones. CBP officials said that media reports can be a catalyst for its Forced Labor Division to initiate or investigate a case. For example, CBP officials noted that forced labor in Thailand’s shrimp industry had been in the news since 2015, and in response, CBP collected additional information from companies importing shrimp from Thailand. Additionally, the Forced Labor Division initiated the case that resulted in the seafood-related WRO based partially on news reporting, according to CBP officials. CBP officials also told us that they have formed working relationships with journalists that can be helpful in obtaining information to initiate or investigate cases. CBP also uses information on an as-needed basis from a variety of federal agencies to initiate and investigate cases, according to CBP officials. NOAA. CBP can use certain data collected through NOAA’s trade monitoring programs to help the agency support specific forced labor cases for seafood, according to CBP officials. For example, in the case of the seafood-related WRO, CBP officials told us they used vessel names collected through NOAA’s Tuna Tracking and Verification Program to link specific shipments of tuna to the vessel in question, which CBP officials said was essential information to confirm imports were being made to the United States. This information was available because the seafood in question was one of the species of fish subject to NOAA trade monitoring programs that generate data CBP can access; however, not all species of fish are included in these programs. Through its trade monitoring programs, NOAA collects harvest-related data, such as the name of the fishing vessel and the species of fish caught, but NOAA officials told us the agency does not collect data specific to labor conditions. NOAA officials and other stakeholders said that there have been discussions regarding potentially expanding the scope of data collected through NOAA’s trade-monitoring programs, such as SIMP, to collect labor-related data. However, the officials noted some potential difficulties in doing so. For example, NOAA officials said that they would need to determine what specific information would be feasible to collect from importers and how it would collect, review, and validate such information. Some NOAA officials raised concern that collecting data on labor conditions may be outside NOAA’s mission; as such, the agency may not have a clear use for the data once collected. The Department of Labor. CBP officials told us they may use reports published by the Department of Labor for context to inform section 307 investigations, including those involving seafood. CBP officials also said they may reach out to the department on an as-needed basis to seek additional information. According to Department of Labor officials, the department also contacts other U.S. government agencies, including CBP, on an ad-hoc basis to share information. The Department of State. CBP officials said they may use reports published by the Department of State for contextual information in their enforcement of section 307, including investigation of cases involving seafood. CBP officials also said that the Department of State may include CBP in official communications from embassies discussing potential instances of forced labor. Department of State officials also said that they may reach out to CBP on a case-by-case basis regarding issues of potential forced labor detected in the course of their work overseas. CBP officials said they may use information from stakeholders to initiate or investigate cases. For example, CBP officials stated that they have reached out to NGOs to obtain clarification on sources used in NGOs’ reports for specific cases. Stakeholders can also submit information or allegations proactively to the agency. CBP officials said that firsthand information collected in-country, including victim accounts, can be beneficial for initiating or investigating forced labor cases. We found many stakeholders collect such information. For example, a representative from one NGO told us that its organization conducts interviews with laborers from fishing vessels once the vessels dock to gather information on labor payment practices, which can serve as an indicator of potential forced labor. However, CBP officials said they also face challenges using information provided by stakeholders because information is often insufficient to initiate or investigate a forced labor case. For example, CBP officials said that information they receive from NGOs might not provide sufficient detail on the supply chain that includes the alleged forced labor, including the manufacturer or vessel committing forced labor, or the connection to a U.S. importer. Additionally, these officials told us that information from stakeholders may conflate poor working conditions with forced labor. Firsthand Account of Abuse and Potential Forced Labor Involving Workers on Fishing Vessels In its 2019 Seabound report, Greenpeace included testimonials of migrant fishers that detailed abuse and violent conditions on fishing vessels: “I witnessed horrible torture. We were working even on midnights. When the Fishing Master was angry, he hit my friend’s head near his left ear. After that he was forced to continue working until the work was finished and only then was he allowed to rest. In the morning when we woke up for breakfast, we found him dead in his room. The Captain wrapped up my dead friend’s body with a blanket and then stored him in the freezer.” According to stakeholders we interviewed and our review of information on CBP’s website, CBP has not clearly communicated its information needs externally. Representatives from 14 of the 18 NGOs we interviewed indicated that they had some uncertainty about the types and level of information CBP needs to investigate forced labor cases in the seafood industry. For example, representatives from one NGO said it was not clear what constituted a credible allegation for CBP, or what information CBP needs to make a section 307 determination. Additionally, representatives from two NGOs managing a grant program designed to support nonprofit organizations collecting firsthand evidence of forced labor said that they were unable to obtain specific guidance from CBP on the types of information the agency needs. As a result, these representatives said they could not communicate to potential grantees the specific kinds of information that would be most useful to submit to CBP. In asking CBP officials about this, CBP confirmed that the NGOs had reached out but they misunderstood the goals of the grant program at the time. CBP could improve the quality of information it receives from stakeholders, including NGOs, by better communicating what information is most useful to initiate and investigate forced labor cases, including those involving seafood, according to stakeholders. Of the 14 stakeholders that told us there was uncertainty, 11 indicated that additional or clearer information about the agency’s information needs could result in more reporting of information to CBP. For example, representatives from one NGO said there was reluctance among stakeholders that may have limited resources to develop an allegation without knowing whether it is helpful, and that they would be more likely to do so with a better understanding of CBP’s needs. Similarly, representatives from another NGO said it is not worth dedicating the time and resources to develop an allegation without a clear sense of the types of information CBP is looking for to investigate its forced labor cases. Many of these stakeholders indicated that they are collecting firsthand information about potential forced labor in seafood supply chains in countries where labor violations are prevalent, which is information CBP officials told us could benefit forced labor investigations. CBP officials said they have communicated in general about their information needs for forced labor cases, and that requisite information varies by case, including for seafood cases. The agency’s website contains some information, including a reference to a regulation identifying information individuals are to submit to CBP when making a forced labor allegation. However, CBP does not indicate what specific information to submit such as the timing or location of alleged forced labor activities. Similarly, CBP does not provide examples of the type of information—such as photos or testimonials from victims—that could be useful information for initiating or investigating cases. In addition, CBP officials said that their e-allegations system provides a means for stakeholders to submit allegations of potential forced labor, among other things, to the agency. However, as of April 2020, the instructions for submitting an allegation do not include specifics on the types of information CBP needs to initiate or investigate cases, such as whether photographs or firsthand accounts of forced labor could be helpful. CBP officials agreed with the need to better communicate to stakeholders the types of information that are helpful for initiating or investigating forced labor cases. They said that the Forced Labor Division had begun considering how it might do so but, to date, had yet to identify further details such as the approach it might take. Federal standards for internal control establish that management should externally communicate the necessary quality information to achieve an agency’s objectives. For example, an agency should use appropriate methods to communicate quality information so that external parties can help the agency achieve its objectives. With better communication to stakeholders about the types of information it needs to initiate and investigate forced labor cases, CBP may be able to improve its enforcement efforts through enhanced information from stakeholders. The exploitation of labor in the seafood supply chain is a global issue that, according to a recent United Nations report, affects millions of people working in the fishing sector. With the United States importing billions of dollars’ worth of seafood in 2018 and reliant on those imports for much of the seafood it consumes, it is important that CBP take action to detect and prevent imports produced with forced labor from entering the country. Following the enactment of TFTEA in February 2016, CBP created the Forced Labor Division and placed an increased emphasis on detecting forced labor in imports, including seafood. CBP officials told us they do not have the resources to gather firsthand information on labor practices. To this end, CBP uses information from a variety of sources, including external stakeholders such as NGOs, to initiate and investigate cases. However, stakeholders are unclear about the types of information CBP needs to initiate and investigate cases because CBP has not clearly communicated this information. As a result, CBP may be missing opportunities to obtain key information that stakeholders collect specific to forced labor in the seafood industry—information that could enhance CBP’s enforcement efforts. The Acting Commissioner of CBP should better communicate to stakeholders the types of information stakeholders could collect and submit to CBP to help the agency initiate and investigate forced labor cases related to seafood and, as appropriate, other goods. (Recommendation 1) We provided a draft of this report to the Departments of Commerce, Homeland Security, Justice, Labor, State, and the U.S. Agency for International Development for review and comment. We received written comments from the Department of Homeland Security and the U.S. Agency for International Development, which are reproduced in appendixes III and IV, respectively. The Department of Homeland Security concurred with our recommendation and noted that CBP is committed to continued collaboration and communication with stakeholders about the types of information needed to develop forced labor cases and improve enforcement efforts of section 307 of the Tariff Act of 1930, as amended. CBP described the actions it plans to take to address the recommendation, including steps to improve collaboration and information sharing during meetings with working groups. In addition, the Departments of Commerce, Homeland Security, Justice, Labor, and State provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees; the Secretaries of Commerce, Homeland Security, Justice, Labor, and State; and the Administrator of the U.S. Agency for International Development. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Anne-Marie Fennell at (202) 512-3841 or fennella@gao.gov or Kimberly Gianopoulos at (202) 512-8612 or gianopoulosk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. The objectives of this report are to examine (1) the process U.S. Customs and Border Protection (CBP) uses to enforce section 307 for seafood imports and the results of its civil enforcement actions and (2) the external sources of information CBP uses to help carry out enforcement of section 307 specific to seafood imports and stakeholder perspectives on CBP’s communication of information needs. To examine the process the Department of Homeland Security’s CBP uses to enforce section 307 for seafood imports, we reviewed laws, regulations, and CBP documents and data pertaining to section 307 enforcement. These laws included section 307 of the Tariff Act of 1930 and the Trade Facilitation and Trade Enforcement Act of 2015 (TFTEA). We interviewed CBP officials from the Office of Trade and Office of Field Operations, in both Washington, D.C., and the field, who are involved in forced labor detection and enforcement about the steps in CBP’s process to enforce section 307. In addition, we interviewed officials from U.S. Immigration and Customs Enforcement to learn about their involvement in addressing section 307. We examined CBP’s efforts to enforce section 307 since TFTEA was enacted, in February 2016, through March 2020, the most currently available information at the time of our review. We examined CBP’s enforcement of section 307 but did not include other forced labor laws in the scope of our review. To describe the results of CBP’s civil enforcement actions, we looked at enforcement actions CBP took from February 2016—when TFTEA was enacted—through March 2020. We reviewed CBP’s list of civil enforcement actions pertaining to all commodities it published on its website. In addition, we collected and analyzed information specific to seafood from a spreadsheet that CBP uses to track cases, which contains information on all of its active, suspended, and inactive forced labor investigations pertaining to section 307. CBP provided us with updated versions of this case-tracking spreadsheet in July 2019, November 2019, and March 2020. To assess the reliability of the data, we interviewed CBP officials about the accuracy and completeness of the data and discussed each seafood case to understand the data represented in the various fields, such as how the seafood case originated and what the outcome of the case was. We also discussed in detail the information that led to the one seafood withhold release order (WRO) CBP issued in February 2019. Based on our interviews with CBP officials, we determined that the seafood case data were sufficiently reliable for the purposes of describing CBP’s enforcement actions. To describe the external sources of information CBP uses to help carry out enforcement of section 307 for seafood, we conducted interviews with CBP officials, including officials in the Office of Trade, which includes the Forced Labor Division, and the Office of Field Operations, to learn about the types of information they gather from external sources and how they might use that information. In addition, we interviewed various other federal agencies and stakeholders that collect information that could be relevant to CBP’s enforcement of section 307. We identified these agencies by interviewing CBP officials about the external sources of information they use to help enforce section 307 cases. Federal agencies. We interviewed the following agencies: Department of Commerce’s National Oceanic and Atmospheric Administration (NOAA), Department of Justice, Department of Labor, Department of State, U.S. Immigration and Customs Enforcement, and U.S. Agency for International Development. We also interviewed officials from NOAA about the Seafood Import Monitoring Program and other trade-related programs that collect data aimed at preventing illegal, unreported, and unregulated seafood from entering the United States. In addition, we reviewed documents provided by NOAA that focused on trade programs that CBP could use as a source of information to help carry out enforcement of section 307. We interviewed officials from the Departments of State and Labor about each agency’s reports on human trafficking and forced labor and obtained copies of and reviewed their reports. Stakeholders. We interviewed current and former representatives from 18 nongovernmental organizations (NGOs) that have interests in forced labor in the seafood industry. Our original scope included 19 NGOs, but we eliminated one NGO from our scope since an official from this organization told us its responses would not vary from those we received from a larger parent NGO, and officials we interviewed did not have separate viewpoints on the extent to which they understood CBP’s information needs. We identified NGOs using internet searches for groups focused on seafood and forced labor and the recommendations of officials from federal agencies and NGOs we interviewed. We also selected NGOs that represented a variety of goals and missions, including those focused on helping U.S. importers remain compliant with section 307 and those focused on assistance to survivors of forced labor overseas. We asked representatives from each NGO a standard set of questions that addressed, among other things, information they may share with CBP. Statements these stakeholders made are not generalizable to all stakeholders but provide perspectives on information for enforcing section 307. To describe stakeholder perspectives on CBP’s communication of information needs, we interviewed stakeholders about their perspectives of CBP’s information needs. Specifically, we asked about the extent to which they understood CBP’s information needs for enforcement of section 307, the extent to which they have shared information with CBP about potential forced labor they have identified, and factors that may affect their sharing information with CBP. In some cases, NGO representatives we interviewed told us their organizations were unable to share information with CBP because of external factors, such as nondisclosure agreements or differing information collection objectives, but they noted that CBP actions could affect the likelihood of other stakeholders sharing information. Statements these stakeholders made are not generalizable to all stakeholders but provide perspectives on information sources. We also interviewed CBP officials about the extent to which information stakeholders provided was sufficient to use in initiating and investigating section 307 cases. To evaluate CBP’s communication of its information needs to initiate or investigate forced labor cases as part of its section 307 enforcement process, we interviewed CBP officials about how the agency communicates its information needs to initiate or investigate forced labor cases as part of its section 307 enforcement process. We also reviewed CBP documents and the agency’s website to identify what information CBP provided to the public about its information needs. We compared the agency’s existing communication efforts to federal standards for internal control, as appropriate. We assessed the agency’s procedures to determine whether CBP communicated information to external parties through appropriate methods. We also assessed the quality of available information to ensure it was appropriate, current, complete, and accessible, among other things. We conducted this performance audit from February 2019 to June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Since the enactment of the Trade Facilitation and Trade Enforcement Act of 2015 (TFTEA), a number of working groups or task forces have been established, primarily involving U.S. federal agencies, to focus on forced labor and imports in general as well as combating illegal activities in the seafood supply chain more specifically in some cases. Interagency Working Group on Forced Labor. The Department of Homeland Security established this working group in 2017. According to U.S. Customs and Border Protection (CBP) officials, the group’s purpose is information sharing and collaboration on forced labor topics with interagency partners, which include officials from the Departments of Labor, State, and Justice and the National Oceanic and Atmospheric Administration (NOAA), among other federal agencies. CBP officials stated that the group generally meets monthly. Task Force on Human Trafficking in Fishing in International Waters. In 2017, the Senate Appropriations Committee directed the Department of Justice to lead a multi-agency task force to examine the issue of human trafficking in seafood supply chains and report to Congress on the status of such issues, along with any related funding, policy recommendations, and legal decisions. Department of Justice officials said they launched the task force in October 2018, and it includes officials from NOAA; the Departments of Homeland Security, Justice, State, Labor, and Treasury; and the Office of the United States Trade Representative. Department of Justice officials said they drafted a report that identified relevant legal and jurisdictional issues, with recommendations to help further efforts to limit human trafficking in fishing in international waters. As of March 2020, the draft was undergoing interagency review and no publication date had been specified, according to the officials. Commercial Customs Operations Advisory Committee Forced Labor Working Group. The Commercial Customs Operations Advisory Committee is a longstanding public-private partnership between the federal government and the private sector. It advises the Department of Homeland Security on matters involving commercial operations, including significant changes that are proposed to CBP regulations, policies, or practices. After the enactment of TFTEA, a working group within the committee’s Intelligent Enforcement Subcommittee—the Forced Labor Working Group—began discussing a variety of issues related to the implementation of section 307, according to CBP officials. The officials said that in 2017 the Forced Labor Working Group sought information from several nongovernmental organizations knowledgeable about labor and human rights in sectors involving seafood and other goods to obtain their insights that could then be shared with CBP. In addition to the contacts named above, Alyssa M. Hundrup (Assistant Director), Christine Broderick (Assistant Director), Christina Werth (Assistant Director), Andrea Riba Miller (Analyst in Charge), and Emily Norman made key contributions to this report. Martin De Alteriis, Patricia Moye, Sheryl Stein, Sara Sullivan, and Nicole Willems also contributed to the report.", "summary": "The United States, which relies on imports for most of the seafood it consumes, imported about $40 billion in fishery products in 2018. Seafood imports often involve complex supply chains, which may include forced labor. A 2017 United Nations report estimated that there are 24.9 million people in forced labor around the world, 12 percent of whom work in the agriculture and fishing sectors. Section 307 of the Tariff Act of 1930, as amended in 2016, prohibits the importation of goods, including seafood, produced or manufactured, wholly or in part, in any foreign country by forced labor, among other things. GAO was asked to review CBP's enforcement of section 307. This report examines (1) the process CBP uses to enforce section 307 for seafood imports and the results of its civil enforcement actions; and (2) the external sources of information CBP uses to help carry out enforcement of section 307 for seafood imports and stakeholder perspectives on CBP's communication of its information needs. GAO reviewed laws and CBP documents pertaining to section 307 enforcement and interviewed officials from CBP, other federal agencies, and 18 NGO stakeholders. GAO selected NGOs with various goals and missions related to seafood and forced labor. The Department of Homeland Security's U.S. Customs and Border Protection (CBP) uses a four-phase process to enforce section 307 of the Tariff Act of 1930, which prohibits imports produced with forced labor, including seafood. CBP's Forced Labor Division, established in 2018, largely carries out this process. In phase 1, CBP assesses leads when deciding to initiate a case involving potential forced labor. In phase 2, CBP investigates cases using a variety of information to determine whether evidentiary standards have been met. In phase 3, CBP reviews information for legal sufficiency and, in phase 4, may take action at a port of entry to detain imports in violation by issuing a withhold release order. Between 2016 and March 2020, CBP issued one order for seafood, prohibiting tuna shipments from a specific fishing vessel from entering U.S. commerce. CBP uses information from external sources to help enforce section 307 for seafood imports but may miss opportunities to obtain key information from stakeholders. CBP officials said they use media reports and information from federal agencies and stakeholders to develop forced labor cases. For example, CBP initiated the case that resulted in the seafood order based partly on media reports and investigated it using vessel data from the Department of Commerce. CBP officials said that stakeholders such as nongovernmental organizations (NGOs) often have firsthand accounts of forced labor—valuable information for investigations. However, most stakeholders told GAO that they do not have a clear understanding of the information CBP needs to investigate seafood cases because CBP has not communicated such information. For example, CBP's website provides general information about what individuals can submit if forced labor is suspected but does not provide specific types of information that could be useful. With better communication to stakeholders about the types of information it needs to develop forced labor cases, CBP may be able to improve its enforcement efforts. GAO recommends that CBP better communicate to stakeholders the types of information stakeholders could collect and submit to CBP to help the agency initiate and investigate forced labor cases related to seafood and, as appropriate, other goods. CBP agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "SOCOM has a unique structure and responsibilities in that it has both combatant command responsibilities and military service-like functions for organizing, training, and equipping SOF. Under sections 164 and 167 of Title 10, United States Code, the SOCOM commander is responsible for training and ensuring the combat readiness of assigned forces and monitoring the preparedness to carry out assigned missions of SOF assigned to unified combatant commands. In addition, SOCOM is responsible for developing special operations strategy, doctrine, and tactics; the employment of forces of the command to carry out assigned missions; requirements validation; acquisition of special operations- peculiar equipment; and formulating and submitting requirements for intelligence support, among other things. In its combatant command function, the commander of SOCOM is responsible for and has the authority to conduct the following special operations activities: (1) direct action, (2) strategic reconnaissance, (3) unconventional warfare, (4) foreign internal defense, (5) civil affairs, (6) military information support operations, (7) counterterrorism, (8) humanitarian assistance, (9) theater search and rescue, and (10) other activities such as may be specified by the President or the Secretary of Defense. Congress initially established the position of the ASD-SO/LIC in the NDAA for Fiscal Year 1987. As previously discussed, in 2016 Congress enhanced the role of the ASD-SO/LIC in section 922, which is codified in section 138(b) of Title 10, United States Code. The ASD-SO/LIC’s current statutory responsibilities include overall supervision, including policy and resources, of special operations activities listed above; exercising authority, direction, and control of all special operations-peculiar administrative matters relating to the organization, training, and equipping of SOF; and assisting the Secretary of Defense and USD (P) in the development and supervision of policy, program planning and execution, and allocation and use of resources for irregular warfare, combating terrorism, and special operations activities. DOD Directive 5111.10, Assistant Secretary of Defense for Special Operations and Low-Intensity Conflict (SO/LIC), first issued in 1995 and most recently updated in 2011, also prescribes the roles and responsibilities for the ASD-SO/LIC. Among other things, the ASD- SO/LIC serves as the principal staff assistant to the USD (P) and the Secretary of Defense on special operations and low-intensity conflict matters and counterdrug policy, among others. DOD Directive 5111.10 also establishes responsibilities, functions, relationships, and authorities for the ASD-SO/LIC on issues such as the coordination and oversight of policy for humanitarian assistance, refugee affairs, and foreign disaster relief activities (e.g., emergency relief for Ebola). Prior to the enactment of section 922, OASD-SO/LIC coordinated regularly with SOCOM on administrative matters, such as reviewing SOCOM’s budget materials. Specifically, the administrative chain of command for SOF-related matters was formally changed by section 922 to give the ASD-SO/LIC more oversight over SOCOM through direct interaction with the Secretary of Defense. Section 922 provided the ASD-SO/LIC with the statutory authority to exercise authority, direction, and control of all special operations-peculiar administrative matters relating to organizing, training, and equipping SOF. Section 922 did not alter SOCOM’s operational chain of command as a combatant command. Section 1074 of the NDAA for Fiscal Year 2018 directed DOD to submit a report on the progress the department had made in implementing the requirements identified in section 922. Section 1074 specified seven reporting elements, such as the accounting of personnel currently assigned, that DOD’s report should address. DOD submitted its report on March 12, 2018, wherein it identified a high-level summary of actions taken, as shown in table 1 below. In 2018, DOD identified 166 recommendations to address the reforms required by section 922 that are aimed at increasing the ASD-SO/LIC’s role in the management of SOF and special operations. To identify these recommendations and support the implementation of service secretary- like responsibilities under section 922, OASD-SO/LIC and SOCOM created a “tiger team” to review broad functional areas typically performed by the military service secretariats and determine the need for potential changes to the roles and responsibilities of OASD-SO/LIC and SOCOM related to addressing requirements in section 922. The tiger team included five working groups to review potential roles and responsibilities for budget, special access programs, personnel and readiness, program and requirements, and acquisition functions. Two officials, respectively representing OASD-SO/LIC and SOCOM, co-led each of these working groups. OASD-SO/LIC established design principles to help the working groups identify new roles and responsibilities for OASD-SO/LIC and SOCOM under section 922. These principles included the following three broad categories of authorities that OASD-SO/LIC could be expected to take on: Monitor: This role requires that OASD-SO/LIC be informed, observe, and check the progress or quality of an activity throughout the lifetime of the activity. This includes, for example, monitoring SOCOM’s submission of its presidential budget justification material to Congress. Review and coordinate: This role requires that OASD-SO/LIC review, analyze, and coordinate throughout the lifetime of an activity to ensure compliance with authoritative policy and with statutory and other regulatory issuances, and to ensure achievement of broad program goals. Coordination does not imply authority to compel agreement, however. An example of the review and coordinate role is that OASD-SO/LIC liaises with the military departments on military personnel issues. Approve: This role requires OASD-SO/LIC’s concurrence to give explicit or official sanction, permission, or ratification of an activity. An example of approval authority is that ASD-SO/LIC approves SOCOM’s Program Objective Memorandum (POM). We found the largest share of the 166 recommendations made by the working groups strengthened OASD-SO/LIC’s roles related to monitor and to review and coordinate, as shown in figure 1. Specifically, 80 out of 166 recommendations (48 percent) would strengthen OASD-SO/LIC’s role regarding monitor or review and coordinate. Twenty-two out of 166 recommendations (13 percent) would give OASD-SO/LIC approval authority—requiring OASD-SO/LIC’s concurrence to give explicit or official sanction, permission, or ratification of an activity. Of these 22 recommendations, 16 involved either joint approval—requiring both OASD-SO/LIC and SOCOM to jointly approve the action—or partial approval—that is, OASD-SO/LIC would have approval authority on certain aspects of an action item. Sixty-four out of 166 recommendations (39 percent) did not recommend any change to OASD-SO/LIC’s role. In addition, the majority of the recommendations, about 156 out of 166 (about 94 percent) would not change SOCOM’s roles. OASD-SO/LIC used the 166 recommendations to inform the development of 87 actions in OASD-SO/LIC’s monthly reports to Congress. We found that with regard to the 87 actions identified in OASD-SO/LIC’s February 2019 monthly report, 49 percent of the action items (43 out of 87) focused on OASD-SO/LIC’s participation in meetings. For example, prior to the implementation of section 922, OASD-SO/LIC attended Joint Resources Management Board meetings. After implementing section 922, OASD- SO/LIC exercised its review and coordinate responsibility by attending Joint Resources Management Board meetings, thereby formalizing OASD-SO/LIC’s prior role. According to DOD officials, there is a value in adding OASD-SO/LIC as a participant in key meetings and formalizing OASD-SO/LIC’s review and coordinate role. For example, officials explained that, by participating in meetings, OASD-SO/LIC can have more situational awareness about key topics and can better advocate for the SOF enterprise. DOD, through OASD-SO/LIC, has taken various actions, including changes in roles and responsibilities, related to addressing requirements in section 922. According to OASD-SO/LIC officials, its actions reflect an incremental approach to strengthening OASD-SO/LIC’s roles and responsibilities. In February 2019 OASD-SO/LIC reported to Congress that it had completed 56 of its 87 actions. For example, one of the actions identified in the February 2019 monthly report was the need to enhance OASD-SO/LIC’s role in the development and approval of SOF-related program and budget matters. The report further identified a number of actions, including having OASD-SO/LIC approve SOCOM’s POM. According to the report, OASD-SO/LIC was briefed on and approved SOCOM’s POM for fiscal years 2020-2024. As another example, the report identified the need to enhance OASD-SO/LIC’s oversight of SOF- related military construction activities and contingency basing. This included a requirement that OASD-SO/LIC co-chair SOCOM’s Military Construction Summit, which according to officials deals with acquisition- related issues regarding military construction and is used to inform the POM. According to the February 2019 report, OASD-SO/LIC co-chaired the summit for fiscal year 2019, and its formal role as co-chair will be reflected in future updates to SOCOM guidance. The February report also explained that the Deputy Secretary of Defense approved a new Special Operations Policy and Oversight Council directive that identified the ASD- SO/LIC as the lead for that council. The Deputy Secretary of Defense also delegated the ASD-SO/LIC with authority to approve waivers to hire civilian personnel during a civilian hiring freeze. Many of the actions taken thus far formalize pre-existing, informal relationships between OASD-SO/LIC and SOCOM. According to OASD- SO/LIC officials, a formalization of a pre-existing role occurs when OASD- SO/LIC identifies a role that OASD-SO/LIC performed informally before addressing requirements under section 922 and continues to maintain the role officially under its section 922 responsibilities. Based on the February 2019 report to Congress, we found that 26 out of 56 implemented action items (about 50 percent) formalize ongoing OASD- SO/LIC roles and responsibilities that were previously conducted informally. Officials stated that all of the actions relating to budget execution are formalizations of previously existing informal roles and responsibilities. For example, according to OASD-SO/LIC and SOCOM officials, OASD-SO/LIC had an informal role in reviewing SOCOM’s POM prior to section 922, such as participating in the review of the POM without formal approval authority. According to DOD officials familiar with the POM process, giving OASD-SO/LIC approval authority for SOCOM’s POM essentially formalized what had been done in the past, while allowing OASD-SO/LIC to perform a more thorough review. Similarly, officials stated that OASD-SO/LIC had an informal role in developing SOCOM’s budget justification books prior to the passage of section 922. Another action identified in DOD’s February 2019 monthly report is OASD-SO/LIC’s role in budget submission. Officials explained that, in an effort to enhance OASD-SO/LIC’s role in budget submission, OASD- SO/LIC has formalized this role. According to the officials, the benefit of this formalization is that OASD-SO/LIC has greater access to the process of producing justification books. There have been similar examples of formalization of pre-existing roles in other areas as well. For example, prior to section 922, SOCOM’s public affairs requirements were coordinated with USD (P)’s public affairs office. Rather than duplicate SOCOM’s existing public affairs role with an additional public affairs office for the ASD-SO/LIC, OASD-SO/LIC coordinates with the USD (P)’s public affairs office. Most of the actions remaining to be implemented do not have clear time frames for implementation. Based on our analysis of the February 2019 monthly report, we found that 31 out of 87 identified actions remain unimplemented. Of these 31 actions, three have clear time frames for implementation. For example, one of the remaining actions involves enhancing the ASD-SO/LIC’s role in SOF military personnel-related issues. Among other things, this includes liaising with the military departments on relevant military personnel issues and coordinating on related policy issues. The February 2019 monthly report includes an action related to OASD-SO/LIC’s plans to coordinate a process to monitor promotions of SOF personnel and communicate issues with military departments. The report specifies that the ASD-SO/LIC expected to implement this process in 2019. As another example, documenting and funding for the Secretariat for Special Operations was expected to be resolved by the first quarter of fiscal year 2019. However, the remaining 28 actions do not have time frames for implementation. For example, some of the actions associated with implementing the ASD-SO/LIC’s key functions, such as acquisitions and legislative affairs, do not have clear time frames for implementation. Regarding acquisitions, OASD-SO/LIC is developing standard operating procedures, such as regular coordination and meetings, but it has not established time frames for the creation or implementation of these procedures. Similarly, OASD-SO/LIC and SOCOM are prescribing roles with regard to legislative affairs pending further departmental guidance, but they have not established time frames within which these roles will be defined. DOD officials identified some reasons for not having identified time frames for the remaining actions. First, according to OASD-SO/LIC officials, their initial efforts were focused on identifying and prioritizing the list of actions needed to implement section 922, as reflected in the March 2018 report required by law. Since then, according to OASD-SO/LIC and SOCOM officials, OASD-SO/LIC has taken an incremental approach to implementing these actions, addressing items on a case-by-case basis as they occur. For example, OASD-SO/LIC initially placed a higher priority on implementing its fiscal roles and responsibilities, partly because the POM cycle included deadlines associated with the President’s Budget for Fiscal Year 2020. Throughout the cycle, OASD-SO/LIC determined its specific role in each step of the POM process as the step arose. Second, OASD-SO/LIC officials stated that they had not established clear time frames linked to action items because the ASD-SO/LIC was new in that role and they were waiting for him to determine OASD-SO/LIC’s broader strategy and goals, which they could use to inform implementation time frames. However, we note that the ASD-SO/LIC has been in that position since December 2017, and OASD-SO/LIC has hired new personnel who could help develop and track time frames. Standards for Internal Control in the Federal Government emphasizes the need to establish time frames to implement actions effectively, and as we reported in June 2018, establishing time frames with key milestones and deliverables to track implementation progress are important for agency reform efforts. Failure to do so can have significant consequences. For example, by not establishing clear time frames for updating guidance that defines the ASD-SO/LIC’s acquisition roles, the ASD-SO/LIC is at risk for having unclear roles and responsibilities that may overlap between SOCOM and the Office of the Secretary of Defense on functions related to acquisitions. According SOCOM officials, having clearer time frames to update DOD guidance could enable OASD-SO/LIC and SOCOM to operate more efficiently and effectively. Without establishing clear time frames for the implementation of key oversight functions and other actions, the ASD-SO/LIC may not be able to fully execute OASD- SO/LIC’s service secretary-like authority, and DOD decision-makers may not be well positioned to track progress and evaluate whether or how the ASD-SO/LIC’s completed and pending actions support the full implementation of section 922. While the ASD- SO/LIC’s responsibilities, functions, relationships, and authorities are established in DOD Directive 5111.10, Assistant Secretary of Defense for Special Operations and Low-Intensity Conflict (ASD SO/LIC) (Mar. 22, 1995) (incorporating Change 2, Oct. 21, 2011), this directive is outdated and does not reflect the ASD- SO/LIC’s statutory roles under section 922 and codified at 10 U.S.C. § 138. For example, DOD Directive 5111.10 states that the ASD- SO/LIC shall serve under the authority, direction, and control of the USD (P). However, section 922 states that the ASD- SO/LIC’s exercise of authority of all special operations-peculiar administrative matters related to the organization, training, and equipping of SOF shall be subject to the authority, direction, and control of the Secretary of Defense. According to DOD officials, while there is other guidance that broadly lays out DOD roles and responsibilities, this guidance lacks details concerning operationalizing ASD- SO/LIC’s roles and responsibilities under the new administrative chain of command, creating potential confusion regarding the ASD- SO/LIC’s roles and responsibilities on some key SOF-related issues. For example: SOF personnel issues: SOF personnel activities include readiness reporting, training, education, warrior care, awards, decorations, and death notification. Support for SOF personnel issues is generally dispersed among different components, including the military services, SOCOM, the office of the Under Secretary of Defense for Personnel and Readiness (USD (P&R)), and OASD- SO/LIC. Although DOD Directive 5111.10 states that the ASD- SO/LIC “shall advise and coordinate with the Under Secretary of Defense for Personnel and Readiness on manpower” issues, it does not define whether manpower issues include SOF career management, such as special pay and promotion. According to DOD officials, DOD lacks overarching guidance that would clarify ASD-SO/LIC’s role on manpower issues. DOD Directive 5111.10 also does not provide specific information about the extent of the ASD- SO/LIC’s coordination role as it relates, for example, to issues such as career management, retirement, pay, or promotion with regard to USD (P&R) responsibilities on SOF personnel management. As a result, according to DOD officials, the lack of clear and updated guidance has caused some confusion among DOD components. According to OASD-SO/LIC officials, after section 922 was implemented, OASD- SO/LIC’s initial attempts to provide strategic outreach for SOF personnel faced some challenges because officials were not included in key personnel meetings. For example, OASD-SO/LIC officials told us they were not included in some meetings that discussed delegating civilian hiring waivers. By not participating in some key SOF personnel-related meetings, OASD-SO/LIC could have missed the opportunity to advocate for similar waiver authority. According to DOD officials, USD (P&R) officials did not fully understand the ASD- SO/LIC’s authorities under section 922 when OASD-SO/LIC officials attended some meetings. Despite this confusion, the ASD-SO/LIC has taken some steps to strengthen its role on SOF personnel issues. For example, according to DOD officials, during the federal government civilian employee hiring freeze, DOD delegated civilian employee hiring waivers to the secretaries of the military departments but did not include waivers for the ASD-SO/LIC or SOCOM. Without the waiver authority to re-instate SOF personnel, SOCOM would have to request a waiver separately through the military services. OASD-SO/LIC officials told us that by ensuring the ASD-SO/LIC was granted a similar waiver authority, OASD-SO/LIC officials streamlined the process and supported SOCOM’s efforts to hire additional SOF civilian personnel. However, the ASD-SO/LIC’s authority on SOF personnel matters remains unclear and SOF personnel issues are generally dispersed among the authorities of USD (P&R), military services, and SOCOM. Overall, it remains unclear what, if any, authorities the ASD-SO/LIC has with respect to leading and coordinating the department’s SOF personnel issues. Budgetary authority: SOF-related budgetary issues include the SOCOM special operations–specific–funding budget materials, the POM, acquisition, and congressional requests for information, among other things. DOD officials told us that before section 922 was enacted, the ASD-SO/LIC reviewed SOF-peculiar budget materials (generally linked to major force program funding) prior to submission of the POM, and the ASD-SO/LIC was notified of SOF-related congressional unfunded priority list submissions. The ASD-SO/LIC did not have principal staff assistant authority to approve the POM. DOD Directive 5111.10 states that the ASD-SO/LIC will provide overall supervision of the preparation and justification of the SOF budget and programs and will review the SOCOM POM. However, the DOD directive has not been updated to provide the ASD-SO/LIC with clear oversight and approval authority over special operations– specific funding, which traditionally has been controlled by SOCOM. DOD Directive 5111.10 also states that the ASD-SO/LIC will advise and coordinate with the Under Secretary of Defense for Acquisition and Technology on acquisition priorities, but this does not provide the ASD-SO/LIC with oversight of the SOF acquisition process. In addition, DOD does not have any guidance that gives ASD-SO/LIC clear oversight roles regarding the SOF acquisition process. By comparison, SOCOM is responsible for the development and acquisition of special operations-peculiar equipment, materiel, supplies, and services in accordance with section 167(e) of Title 10, U.S. Code, and it executes funding in operation and maintenance, procurement, and military construction accounts, among other things. According to OASD-SO/LIC senior officials, the ASD-SO/LIC has some authority over special operations–specific funding through the POM process. According to OASD-SO/LIC officials, after implementing section 922, the ASD-SO/LIC established a new principal staff assistant authority to approve the POM in 2018. However, DOD officials familiar with SOF-related budgetary issues stated that it is unclear how much authority the ASD-SO/LIC has over funding issues to adjudicate potential disagreements between the services and SOCOM on either SOF-specific or common funding issues. Special Access Programs (SAP): SAPs are programs established for a specific class of classified information that impose safeguarding and access requirements that exceed those normally required for information at the same classification level. Given the sensitive nature of these programs, DOD has established different levels of authorities to create and manage SAPs. According to DOD Directive 5205.07, Special Access Program (SAP) Policy, the Deputy Secretary of Defense designates certain DOD component heads, or DOD agency heads—for example, the secretary of a military department or the Commander, SOCOM—as cognizant authorities to manage and execute their respective SAPs. While the ASD-SO/LIC has always played a role in SOF-related SAPs, DOD officials stated that the role is expected to evolve as part of the implementation of section 922. OASD-SO/LIC’s February 2019 monthly report includes several actions intended to enhance the ASD-SO/LIC’s role in the management of SAPs, and OASD-SO/LIC has already begun participating in various SAP-related conferences and meetings. However, according to DOD officials, the ASD-SO/LIC’s future role related to SAPs remains unclear in existing guidance. For example, DOD Directive 5111.10 states that the ASD-SO/LIC will provide oversight over all special operations and low-intensity conflict related sensitive SAPs. Although the ASD-SO/LIC and SOCOM officials told us that they are currently further defining these roles, the DOD directive has not been updated to clarify whether the ASD-SO/LIC should be included in the SAP governance process, which includes designating the ASD-SO/LIC as a cognizant authority with service secretary-like SAP responsibilities. DOD officials expressed some concerns that until these matters are clarified in guidance, it will remain unclear whether the ASD-SO/LIC and SOCOM should work together on SAP issues, and how their relationships with the various Under Secretaries of Defense with oversight authority will be managed. Standards for Internal Control in the Federal Government states that management should define objectives clearly and assign responsibility for key roles throughout the organization. Specifically, the standards call for management to define objectives in specific terms so that they are understood at all levels of the entity. This involves clearly defining what is to be achieved, who is to achieve it, how it will be achieved, and time frames for its achievement. We have also previously reported that management practices key to program success include clearly identifying organizational roles and responsibilities and clarifying program objectives. OASD-SO/LIC and SOCOM officials stated that updated guidance is needed to help clarify the ASD-SO/LIC’s roles and responsibilities under section 922. In December 2018 OASD-SO/LIC officials told us that they were starting to update guidance on the ASD- SO/LIC’s roles and responsibilities under section 922 in DOD directive 5111.10. However, OASD-SO/LIC officials did not provide details about the information that would be updated, and did not provide a copy of that draft guidance. In addition, OASD-SO/LIC officials did not have clear time frames regarding when the guidance will be updated. As DOD updates the ASD-SO/LIC’s roles and responsibilities either in DOD Directive 5111.10 or through new guidance, it has an opportunity to clarify changes in its relationship with DOD components involved in overseeing SOF administrative matters related to personnel, budgetary authority, and SAPs. The SOF enterprise is a complex system, and without clearly identified roles and responsibilities for a service secretary- like role for the ASD-SO/LIC, other DOD components—such as the military departments, USD (P), and USD (P&R) —may not know the extent of the ASD-SO/LIC’s and SOCOM’s authorities in key issues where they have vested interests. For example, it will remain unclear what authorities the ASD-SO/LIC has with regard to SOF-related administrative matters, and which entities will have visibility over any problems or resourcing decisions related to the SOF enterprise. By clarifying the ASD-SO/LIC’s roles and responsibilities with regard to its relationship with SOCOM and other DOD components, DOD can more effectively implement the intent of section 922. OASD-SO/LIC has taken steps to develop a hiring plan to identify personnel requirements and an approach to hiring additional personnel. DOD’s efforts began in 2017, when OASD-SO/LIC commissioned the Army Office of Manpower and Reserve Affairs to conduct a manpower study to provide an analysis of manpower requirements based on unconstrained resources that are necessary to satisfy the service secretary-like responsibilities under section 922. The Army’s manpower study was based on nine functions, including budget, acquisitions, and legislative activities. For each function, the study identified corresponding tasks and the average man hours, or time needed, to complete each task. The study, which was included in DOD’s March 2018 report to Congress, ultimately estimated that up to 64 full-time equivalent (FTE) positions might be needed to implement the ASD-SO/LIC’s section 922 responsibilities. According to OASD-SO/LIC officials, the study provided an initial framework for OASD-SO/LIC to determine its staffing needs, but the study was not comprehensive and OASD-SO/LIC’s hiring needs will likely continue to change in the future. Over the past 2 years, according to OASD-SO/LIC officials, OASD- SO/LIC has begun to hire personnel to fulfill various roles and responsibilities. Specifically, the number of FTEs hired to support OASD- SO/LIC’s implementation of section 922 increased from 14 in March 2018 to 24 as of December 2018. In addition, section 361 of the John S. McCain NDAA for Fiscal Year 2019 gave the ASD-SO/LIC additional flexibility to hire staff in fiscal year 2019. For example, section 361 directed that not less than $4 million in fiscal year 2019 shall be used to fund additional civilian personnel to help implement section 922. Section 361 also provided the OASD-SO/LIC an exemption from the statutory civilian personnel limitation in the Office of the Secretary of Defense imposed by 10 U.S.C. § 143. Figure 2 shows OASD-SO/LIC’s hiring actions to date, along with key events related to the implementation of section 922. In December 2018 OASD-SO/LIC officials completed a basic hiring plan to guide future personnel growth as OASD-SO/LIC continues to implement actions related to section 922. The plan—documented in a 10 slide presentation—includes OASD-SO/LIC’s short-term hiring goals through the start of fiscal year 2020, a hiring approach involving a mix of permanent and temporary staff, and the identification of targeted skillsets for personnel hired. For example, the plan includes targets related to achieving key skills, such as force planning and shaping the President’s Budget for Fiscal Year 2021. The plan also calls for OASD-SO/LIC to grow from 27 current FTEs to a total of 55 FTEs in fiscal year 2020. While OASD-SO/LIC’s current hiring plan represents a first step toward developing a broad overview of its hiring goals and some key hiring considerations, it does not fully incorporate some leading practices for strategic workforce-planning. As we have previously reported, strategic workforce planning addresses two critical needs: (1) aligning an organization’s human capital program with its current and emerging mission and programmatic goals; and (2) developing long-term strategies for acquiring, developing, and retaining staff to achieve programmatic goals. While agencies’ approaches to workforce planning will vary, we have previously identified several key principles that strategic workforce planning should address, irrespective of the context in which the planning is done. GAO’s prior work on workforce planning identified the following five key principles: involve top management, employees, and other stakeholders in developing the strategic workforce plan; determine the critical skills and competencies needed to achieve long-term goals; develop strategies that are tailored to address critical competency gaps; build the capacity needed to address requirements important to supporting workforce strategies; and monitor and evaluate the agency’s progress toward its human capital goals. However, we found that as of December 2018, the OASD-SO/LIC’s hiring plan had not fully incorporated several of these key strategic workforce-planning principles, as described below: The hiring plan was not fully aligned with long-term goals. A key principle in strategic workforce planning is strategic alignment, which occurs when an agency’s human capital program is linked with its mission and goals. However, we found that OASD-SO/LIC has not clearly linked its hiring plan with its overall mission and goals. For example, the hiring plan mentions short-term goals, such as analyzing the budget for fiscal year 2021 and long-term goals, such as strategic assessment and aligning the organization with National Defense Strategy requirements. However, the plan does not define strategic assessment, and it lacks detail about how newly hired personnel in fiscal year 2019 will help OASD-SO/LIC meet long-term goals related to strategic assessment. For example, OASD-SO/LIC recently hired seven personnel, but it is not clear whether the newly hired personnel have skills that match competencies, such as the ability to work with Special Access Programs, identified in OASD-SO/LIC’s hiring plan. We have previously reported that unless hiring needs are clearly linked with long-term goals, the hiring plan may be incomplete or premature. OASD-SO/LIC’s approach did not fully involve stakeholders. While stakeholder involvement is not statutorily required, another key principle of effective strategic workforce planning is to involve top management, employees, and other stakeholders in developing, communicating, and implementing strategic workforce plans. We found several cases in which OASD-SO/LIC did not involve stakeholders in its key efforts. For example, although OASD-SO/LIC senior officials shared information about the hiring plan with one senior official at SOCOM, several OASD-SO/LIC and SOCOM officials stated that OASD-SO/LIC did not communicate the hiring plan’s expectations or strategies more broadly, to involve a full range of OASD-SO/LIC and SOCOM officials and other stakeholders, such as USD (P). In another example, when OASD-SO/LIC hired personnel from September 2018 through December 2018, several OASD- SO/LIC and SOCOM officials were unclear about the specific roles and responsibilities of new personnel hired. The hiring plan did not include strategies to address critical competency gaps and identify related personnel requirements. Leading principles of effective strategic workforce planning hold that agencies should develop strategies to address critical skill gaps and systematic personnel requirements processes, which are considered a good human capital practice across government. However, we found that OASD-SO/LIC’s hiring plan did not include completed competency-gap assessments or have procedures in place to periodically reassess personnel requirements. Without a systematic process to periodically assess personnel requirements, OASD-SO/LIC could not determine whether the Army study’s initial estimates were the most efficient choice for the workforce. For example with regard to the legislative affairs positions, OASD-SO/LIC and SOCOM officials told us that the Army manpower study’s initial estimate of eight FTEs was too high. OASD-SO/LIC officials eventually hired two FTEs for the legislative affairs office, but the hiring plan did not include a methodology to analyze the workforce and explain why two FTEs would fit within the Army study’s framework. According to OASD- SO/LIC officials, OASD-SO/LIC also did not use a standardized process to assess whether two FTEs would meet its requirements. According to OASD-SO/LIC officials, the hiring plan is the first step in developing an initial framework, and they stated that it lacked implementation details. OASD-SO/LIC officials stated that they anticipate building upon the hiring plan as the current workforce plan evolves over time. In addition, OASD-SO/LIC officials stated that key priorities include strengthening OASD-SO/LIC’s participation and oversight of SOF resources through the POM and fiscal guidance processes. As a result, the hiring plan includes information about new personnel focused on fiscal oversight, such as analyzing the budget in fiscal years 2020 through 2021, but it does not clarify long-term goals, competency gaps, and program results tied to other priorities, such as legislative and acquisition- related functions. Officials from OASD-SO/LIC and SOCOM agreed that incorporating key principles in the strategic workforce plan would help them determine the most appropriate size and composition of OASD- SO/LIC’s workforce. Until OASD-SO/LIC completes a comprehensive strategic workforce plan that includes key principles as outlined above, OASD-SO/LIC may not know what gaps exist in skills and competencies, and what their workforce strategies to fill those gaps should be. These issues could put OASD-SO/LIC at risk of hiring personnel who may not adequately meet its needs as defined by section 922. As DOD increasingly relies on SOF, the department has taken steps to implement section 922. Given the expanded statutory authority under section 922, the ASD-SO/LIC has greater authority to oversee and advocate for the SOF enterprise. The ASD-SO/LIC has implemented several actions to clarify and strengthen its oversight roles and responsibilities, and it has many additional planned actions underway. However, without time frames to implement action items and revised or new guidance that clearly articulates the ASD-SO/LIC’s roles and responsibilities with regard to SOCOM and the wider SOF enterprise, these changes may not be fully effective. In addition, without a strategic workforce plan that fully incorporates leading practices to ensure that the department has the right people, in the right place, at the right time, OASD-SO/LIC may not be well prepared to respond to future workload changes and manage its human capital strategically. As OASD-SO/LIC makes progress in its hiring plan, it is important for OASD-SO/LIC to develop a strategic workforce plan to ensure that it appropriately addresses the human-capital challenges of the future and better contributes to the agency’s efforts to meet its missions and goals. We are making three recommendations to the Secretary of Defense: The Secretary of Defense should ensure that the Assistant Secretary of Defense for Special Operations and Low-Intensity Conflict defines time frames for completing action items necessary to implement the Assistant Secretary of Defense for SO/LIC’s expanded section 922 responsibilities. (Recommendation 1) The Secretary of Defense should ensure that the Assistant Secretary of Defense for the Special Operations and Low-Intensity Conflict updates existing guidance or develops new guidance to clarify the roles and responsibilities of the Assistant Secretary of Defense for SO/LIC and relationships with DOD components that have vested interests in the SOF enterprise—such as the military services, SOCOM, the Under Secretary of Defense for Personnel and Readiness, and the Under Secretary of Defense for Policy. (Recommendation 2) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Special Operations and Low-Intensity Conflict builds upon its hiring plan by developing a strategic workforce plan that incorporates key principles, such as aligning the plan with long-term mission goals; fully involving stakeholders in developing the plan; and including strategies to address critical competency gaps and identify related personnel requirements. (Recommendation 3) In written comments on the draft of this report, DOD partially concurred with our recommendations. Comments from DOD are summarized below and reprinted in appendix I. DOD also provided technical comments, which we incorporated as appropriate. DOD partially concurred with the first recommendation that the ASD- SO/LIC define time frames for completing action items necessary to implement the ASD-SO/LIC‘s expanded section 922 responsibilities. In its response, DOD stated that most time frames have been established or the action completed. Additionally, DOD noted that some actions may not be completed because they depend on events, actions or leadership decisions that are outside of OASD-SO/LIC’s control. We agree that some DOD leadership decisions have yet to be made. However, 28 out of 31 already identified actions do not have clear time frames for implementation. Further, time frames can be modified as events change or better information becomes available. As we discuss in the report, establishing time frames with key milestones to track implementation progress are important for agency reform efforts. Without clear time frames, ASD-SO/LIC may not be able to fully execute its service secretary-like authority. DOD partially concurred with the second recommendation that the ASD- SO/LIC update DOD Directive 5111.10 to clarify the roles and responsibilities of the ASD-SO/LIC and relationships with DOD components that have vested interests in the SOF enterprise. DOD is in the process of revising this directive, but DOD noted that the purpose of DOD Directive 5111.10 is to define only specific Department-wide roles and missions for ASD-SO/LIC and is not the appropriate issuance to define ASD-SO/LIC’s relationship with other DOD components in the SOF enterprise. Given that DOD does not believe DOD Directive 5111.10 is the appropriate issuance to clarify ASD-SO/LIC’s relationships with DOD components, we modified our recommendation from focusing solely on updating DOD Directive 5111.10 to updating existing guidance and/or developing new guidance. Updating or developing guidance that clarifies ASD SO/LIC’s relationship with DOD components, such as the military departments, USD (P), and USD (P&R) would likely allow for improved oversight of and collaboration on SOF matters related to personnel, budgetary authority and SAPs. DOD partially concurred with the third recommendation that the ASD- SO/LIC build upon its hiring plan by developing a strategic workforce plan that incorporates key principles, such as aligning the plan with long- term mission goals; fully involving stakeholders in developing the plan; and including strategies to address critical competency gaps and identify related personnel requirements. In its response, DOD agreed that there is room to improve the involvement of stakeholders. In addition, DOD stated that it developed a strategic workforce plan that aligns with long-term mission goals and has identified strategies to address critical competency gaps, including target skillsets. However, as noted in our report, the 10 slide presentation that constitutes the hiring plan lacks details that would be included in a comprehensive workforce plan. For example, the hiring plan did not explain how the hiring needs would be specifically tied to long-term goals, such as National Defense Strategy requirements. Although the hiring plan mentions some skillsets, it does not include a competency gap assessment or assess personnel requirements. As noted in our report, OASD-SO/LIC and SOCOM officials stated that the initial personnel requirements developed by the Army study were inaccurate for several reasons, including the lack of a standardized process to assess personnel requirements. Accordingly, we continue to believe that until OASD-SO/LIC develops a comprehensive strategic workforce plan that includes key principles outlined in our report, OASD- SO/LIC could be at risk of hiring personnel who may not adequately meet its needs to perform the roles and responsibilities of section 922. We are sending copies of this report to other interested congressional committees and the Acting Secretary of Defense. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov. If you have any questions regarding this report, please contact me at (202) 512-5431 or at russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors are listed in appendix II. In addition to the contact named above, Jim Reynolds (Assistant Director), Tracy Barnes, Mikey Erb, Amie Lesser, Mike Silver, Cheryl Weissman, and Yee Wong (Analyst-in-Charge) made key contributions to this report.", "summary": "As DOD increased its reliance on special operations forces, SOCOM's budget has increased from $5.2 billion in 2005 to $12.3 billion in 2018. Section 922 of the NDAA for Fiscal Year 2017 included provisions to enhance the Assistant Secretary of Defense for Special Operations and Low-Intensity Conflict's responsibilities to be similar to those of a military department secretary regarding the organization, training, and equipping of special operations forces. The Joint Explanatory Statement accompanying the fiscal year 2018 NDAA included a provision for GAO to assess DOD's actions in response to section 922. This report assesses (1) the extent to which DOD has identified and taken actions to implement section 922; (2) what, if any, challenges it faces in completing implementation; and (3) the extent to which its hiring approach for the office of the ASD-SO/LIC has incorporated strategic workforce planning principles. GAO reviewed relevant documents and interviewed DOD officials. Since 2017 the Department of Defense (DOD) has made recommendations, developed actions, and taken steps to address requirements in section 922 of the National Defense Authorization Act (NDAA) for Fiscal Year 2017 to expand the Assistant Secretary of Defense for Special Operations and Low-Intensity Conflict's (ASD-SO/LIC) roles and responsibilities. DOD officials noted that they have taken an incremental implementation approach to addressing section 922. In 2018, DOD identified 166 recommendations to change the ASD-SO/LIC's oversight of special operations forces (SOF). These recommendations were used to develop 87 actions that were necessary to implement section 922. Since February 2019, DOD has implemented 56 of these actions. For example, the Deputy Secretary of Defense approved a new Special Operations Policy and Oversight Council directive that identified the ASD-SO/LIC as the lead for that council. The Deputy Secretary of Defense also delegated the ASD-SO/LIC with authority to approve waivers to hire civilian personnel during a civilian hiring freeze. Although the office of the ASD-SO/LIC has taken many actions to implement section 922, DOD faces two key challenges in completing its implementation of the ASD-SO/LIC's new roles and responsibilities: Lack of time frames . As of February 2019, 28 out of 31 unimplemented actions associated with section 922 did not have clear time frames for implementation. According to ASD-SO/LIC and U.S. Special Operations Command (SOCOM) officials, they did not prioritize establishing time frames because they took an incremental approach to implementing actions and addressed them on a case-by-case basis. Without clear time frames for implementation, ASD-SO/LIC and SOCOM may be less effective in implementing section 922. Unclear guidance . Current guidance about ASD-SO/LIC responsibilities is outdated: for example, it states that the ASD-SO/LIC shall report directly to the Under Secretary of Defense for Policy. However, section 922 states that special operation forces-related administrative matters are managed directly by the Secretary of Defense to the ASD-SO/LIC. The special operations force enterprise is a complex system, and unless roles and responsibilities are clarified in guidance, other DOD stakeholders, such as the military services, may not know the extent of the ASD-SO/LIC's and SOCOM's authorities and responsibilities. DOD officials expressed some concerns that until these matters are clarified in guidance, it will remain unclear whether the ASD-SO/LIC and SOCOM should work together—for example, on personnel issues—and how their relationships with stakeholders with oversight authority will be managed. DOD partially concurred, and based on its comments, GAO modified one recommendation. The office of the ASD-SO/LIC has made efforts to develop a workforce plan, including commissioning a manpower study and taking steps to develop a hiring plan; however, these efforts do not fully incorporate some leading principles for a strategic workforce plan. For example, ASD-SO/LIC did not share the hiring plan with its staff, including key officials from the office of the ASD-SO/LIC and SOCOM. Without completing a comprehensive strategic workforce plan that includes key principles, the office of the ASD-SO/LIC may not know what gaps exist in skills and competencies in order to develop effective workforce strategies to fill those gaps. These issues could put the office of the ASD-SO/LIC at risk of hiring personnel who may not adequately meet its needs as defined by section 922. GAO is making three recommendations to DOD to establish time frames for section 922 actions; update applicable guidance to clarify roles and responsibilities for the ASD-SO/LIC and SOCOM; and develop a strategic workforce plan that incorporates key principles. DOD partially concurred with the recommendations and GAO continues to believe the recommendations are valid, as discussed in the report. GAO also modified one recommendation to address DOD concerns regarding its applicability.", "document_type": "gao"}
{"report": "Two State entities play key roles in education assistance in the West Bank and Gaza—State’s Bureau of Population, Refugees, and Migration (State/PRM) and State’s U.S. Consulate General in Jerusalem (State/ConGen). State/PRM has an important role in funding and overseeing education assistance provided by UNRWA in the West Bank and Gaza. State contributes funds to and manages the institutional relationship with UNRWA on behalf of the U.S. government, while recognizing UNRWA’s independence and commitment to upholding humanitarian principles, including neutrality. This relationship is guided by the U.S.-UNRWA Framework for Cooperation, annually negotiated between State/PRM and UNRWA. The framework includes UNRWA’s commitment to meet the condition on U.S. contributions to UNRWA that U.S. funds do not support terrorism, pursuant to section 301(c) of the Foreign Assistance Act of 1961, as amended. The framework also sets forth the activities used to evaluate UNRWA’s conformance with this condition. According to State/PRM officials, some educational materials fit into the framework’s section involving broader U.S. priorities for UNRWA’s education sector. For example, continuing support for mutually identified special projects such as UNRWA’s Human Rights, Conflict Resolution, and Tolerance education program in all of UNRWA’s five fields of operation fit into the latter category. UNRWA’s five fields of operations are the West Bank (including East Jerusalem), Gaza, Jordan, Lebanon, and Syria. The framework also defines U.S. priorities for UNRWA’s education sector. The frameworks for fiscal years 2016 and 2017 state, “The United States is particularly interested in ongoing curriculum review process, which enables UNRWA’s educators to use consistent criteria in analyzing and enriching local textbooks, in order to promote UN values and principles in UNRWA classrooms.” The Secretary of State is required under Section 7048(d) of the Department of State, Foreign Operations, and Related Programs Appropriations Acts for fiscal years 2015 and 2016 to submit a report in writing to the Committees on Appropriations not less than, and for fiscal year 2016 no later than, 45 days after enactment. Section 7048(d) of the Department of State, Foreign Operations, and Related Programs Appropriations Act, 2017 states that this report must be submitted prior to initial obligation of funds. This report is to cover seven topics. One of the required topics in the report is whether UNRWA is taking steps to ensure that the content of all educational materials currently taught in UNRWA-administered schools and summer camps is consistent with the values of human rights, dignity, and tolerance, and does not induce incitement. State/ConGen also has a key role in funding and overseeing U.S. educational assistance. State/ConGen is responsible for the U.S. bilateral relationship with the Palestinian Authority, including efforts to combat incitement to violence and address problematic content in textbooks. In addition, according to the Consulate General’s Education Statement of Purpose, State/ConGen funds and implements education projects to improve the quality of education to equip Palestinians with the skills to grow their economy and build a democratic, secular, politically moderate, and outward-focused Palestinian civil society as a driver for peace. USAID funds education projects that support Palestinian Authority- administered schools, teacher and administrator training in the West Bank, and scholarships. USAID did not identify or address potentially problematic content in Palestinian Authority textbooks between fiscal years 2015 and 2017 because, according to USAID and State officials, reviewing textbooks is outside the scope of the work of USAID’s partners, including nongovernmental organizations, that implement projects in the West Bank and Gaza. USAID officials told us that they defer discussion of any potentially problematic content in textbooks to State as a bilateral policy issue. UNRWA is to provide humanitarian assistance to Palestine refugees in accordance with its mandate provided by the UN General Assembly. UNRWA provides education, health care, social services, microfinance, and emergency assistance to Palestine refugees; infrastructure and camp improvement within Palestine refugee camps; and protection. When UNRWA began operations in 1950, it was responding to the needs of about 860,000 Palestine refugees. UNRWA reports that over 5 million Palestine refugees are registered with UNRWA in the West Bank, Gaza, Jordan, Lebanon, and Syria and are currently eligible for its services. UNRWA administers its education system of more than 700 schools across its five fields of operation, educating approximately 526,000 children, according to UNRWA officials. This includes 370 schools in the West Bank and Gaza for grades 1 through 9 (and grade 10 in two East Jerusalem schools) serving over 300,000 children. UNRWA uses the curricula and textbooks of host governments. In keeping with this practice, UNRWA schools in the West Bank and Gaza use the Palestinian Authority curriculum and textbooks. This practice helps to ensure that UNRWA students can continue their education at government secondary schools and universities and can take national exams. According to UNRWA officials, using the host country curricula is also in line with good practice—affirmed by other UN agencies, such as United Nations High Commissioner for Refugees. The Palestinian Authority provides all textbooks used in UNRWA and Palestinian Authority schools in the West Bank and Gaza except for English language textbooks. Figure 1 shows an UNRWA girls’ school in Shufat refugee camp, located in East Jerusalem. Prior to the release of the first set of Palestinian Authority textbooks developed by the Palestinian Authority in 2000, schools in Gaza used Egyptian textbooks, and schools in the West Bank used Jordanian textbooks. The Palestinian Authority developed its first curriculum in the mid-1990s in cooperation with the United Nations Educational, Scientific and Cultural Organization. Since then, the Palestinian Authority has developed multi-year strategies to improve its educational system, including by modernizing its curriculum and improving its textbooks. The Palestinian Authority worked to implement its early strategies but could not fully do so because responding to other events took priority, according to Palestinian Authority documents. These events included the second Palestinian Intifada (uprising) that began in 2000, the government of Israel’s subsequent tightening of security, the rise of Hamas to power in the Palestinian government in 2006, and the resulting delays in donor funding. After donors resumed their support, the Palestinian Authority developed an education strategy for 2008 through 2012. This strategy’s stated goals include improving the quality of education by reviewing the curriculum and revising textbooks, among other things. Beginning in 2013 the Palestinian Authority undertook a multi-year effort to revise its curriculum and issue new textbooks to provide students with skills such as problem solving and analysis. As a result, the Palestinian Authority Ministry of Education and Higher Education issued new pilot textbooks for grades 1 through 4 in 2016 and 2017. The Palestinian Authority issued textbooks for the first semester of these grades in summer 2016 and textbooks for the second semester later in the year with the start of that semester. The Palestinian Authority issued the final textbooks for grades 1 through 4 and new pilot textbooks for grades 5 through 10 in 2017. As of August 2017, Palestinian Authority public schools and UNRWA schools in the West Bank and Gaza use these textbooks, according to State and UNRWA officials. Figure 2 shows examples of the pilot textbooks for grades 1 through 3. The U.S. government provided an estimated $243 million for education assistance in the West Bank and Gaza—State provided an estimated $193 million, and USAID provided about $50 million—for fiscal years 2015 through 2017, according to State and USAID data and UNRWA- provided information. Of State’s estimated $193 million contributions to education assistance in the West Bank and Gaza, UNRWA estimated that about $187 million went to its education assistance. State provided the remaining approximately $6 million to non-UNRWA education programs. UNRWA reported expending about $877 million for education in the West Bank and Gaza for fiscal years 2015 through 2017, including contributions from the United States and other donors. According to UNRWA officials, UNRWA used some of these funds to purchase English language textbooks that were used in UNRWA schools in the West Bank and Gaza. State, UNRWA, and USAID funds were not used to purchase or produce other textbooks used in the West Bank or Gaza, according to officials from these agencies. Of the estimated $243 million that the United States provided for education assistance in the West Bank and Gaza for fiscal years 2015 through 2017, State funded an estimated $193 million for UNRWA and non-UNRWA projects, according to State and UNRWA information. For UNRWA, State contributed an estimated $187 million for education in the West Bank and Gaza for fiscal years 2015 through 2017, out of a total contribution to UNRWA of about $1 billion for that timeframe. U.S. contributions support UNRWA’s core programs of education, health, relief and social services, microfinance, and infrastructure and camp improvement across its five fields of operation. State does not earmark the majority of its contributions to UNRWA’s program budget by either program area or field of operation. Rather, State contributes funds to UNRWA’s program budget, which UNRWA pools with contributions from other donors to provide general support to UNRWA’s core programs, according to State and UNRWA officials. State earmarks a small portion of its contributions to the program budget to support special projects of mutual priority to State and UNRWA, according to State officials. For each fiscal year from 2015 through 2017, State earmarked funds for the Human Rights, Conflict Resolution, and Tolerance project, an agency-wide, education-related project implemented in all five of UNRWA’s fields of operations, including in the West Bank and Gaza. UNRWA officials stated that UNRWA aims to support teachers in integrating human rights, conflict resolution, and tolerance into the regular curriculum. As part of its education reform, UNRWA developed a Human Rights, Conflict Resolution, and Tolerance Policy and Teacher Toolkit to further strengthen human rights education in UNRWA. According to UNRWA officials, UNRWA has built on international best practices to better integrate human rights education in all UNRWA schools. The United States exclusively funds the Human Rights, Conflict Resolution, and Tolerance project activities, according to State officials. UNRWA estimated expending about $0.3 million on the Human Rights, Conflict Resolution, and Tolerance project in the West Bank and Gaza for fiscal years 2015 through 2017. In addition to State’s funding for UNRWA, State’s U.S. Consulate General in Jerusalem (ConGen) officials said that State/ConGen provided about $6 million in funding for three non-UNRWA education programs focused on youth in grades 1 through 10 in the West Bank and Gaza for fiscal years 2015 through 2017. These three education programs include (1) a program that provides secondary school students in the West Bank and Gaza an opportunity to study at American high schools and live with American host families; (2) an afterschool English language program that targets academically gifted and economically disadvantaged high school students; and (3) a 2-week summer camp program for at-risk Palestinian youth ages 8 through 14 residing in refugee camps and other marginalized areas throughout the West Bank, Gaza, and Jerusalem. Of the estimated $243 million that the United States provided for education assistance in the West Bank and Gaza for fiscal years 2015 through 2017, USAID obligated about $50 million for active non- construction education projects for this timeframe, and it did not fund textbooks, according to USAID officials. USAID funds supported six education projects, of which four were scholarship projects. Two projects—the School Support Program and the Leadership and Teacher Development program—provided support directly to Palestinian Authority public schools in the West Bank. The School Support Program offers assistance to 50 schools, including infrastructure rehabilitation of schools, in-kind assistance (e.g., science lab equipment and school supplies), extracurricular activities (sports, arts and music, career counseling, and psychosocial support), and leadership and teacher development for the school administration. The Leadership and Teacher Development program supports teacher, principal, and supervisor training to make teaching and learning practices more learner-centered, in addition to the introduction of information technology in education (e.g., internet connectivity, equipment, teaching of coding), classroom assessment and testing methods, and administrative reform at the school, district, and central levels. According to UNRWA-provided information, UNRWA expended about $877 million on education for fiscal years 2015 through 2017 in the West Bank and Gaza with funds from the United States and other donors. These funds were expended for UNRWA’s education program, including the purchase of English language textbooks and other educational materials. Of the approximately $877 million UNRWA reported expending on education, it expended about $671 million for education in Gaza and $206 million for education in the West Bank. UNRWA’s expenditures for Gaza are significantly higher because, as of June 30, 2017, UNRWA operated 275 schools in Gaza serving approximately 270,000 students compared to 95 schools in the West Bank serving approximately 48,000 students. UNRWA’s largest reported expenditure within the education sector in fiscal years 2015 and 2016 was personnel-related expenditures, which represented about 85 percent of all education expenditures, according to UNRWA. Between fiscal years 2015 and 2017, including estimated expenditures in 2017, UNRWA reported that it expended about $2 million on educational materials—including about $1 million on English language textbooks for fiscal years 2015 through 2017 for UNRWA schools in the West Bank and Gaza. Of the approximately $1 million expended on English language textbooks, UNRWA estimates that the U.S. contributions totaled about $587,369, with about $28,763 for the West Bank and about $558,606 for Gaza. Educational materials made up less than one percent of UNRWA’s reported education expenditures in the West Bank and Gaza in part because UNRWA does not purchase or fund textbooks for use in its schools in the West Bank and Gaza, with the exception of English language textbooks. The Palestinian Authority provides UNRWA with textbooks for all but one academic subject (English) as an in-kind contribution, according to UNRWA officials. As such, U.S. funds do not contribute to the textbooks that are published by the Palestinian Authority, according to UNRWA information. However, to purchase English language textbooks used in Gaza, UNRWA sent payment from its program budget, which includes commingled donor funds, directly to the Palestinian Authority Ministry of Education and Higher Education, for which they subsequently paid a private publisher. According to information provided by UNRWA, doing so lowered the per unit cost through bulk ordering. According to UNRWA, UNRWA staff work on complementary teaching materials—educational materials that UNRWA develops to use alongside host government textbooks, as part of their regular course of work. They also work on student summer learning materials based on the textbooks. Therefore, the expenditures for these materials cannot be disaggregated from staff wages and salaries and are not included in UNRWA’s expenditures for educational materials. UNRWA has reviewed Palestinian Authority textbooks for the first semester of grades 1 through 10 to identify content it deemed not aligned with UN values and has developed complementary teaching materials to address this content when considered necessary. However, UNRWA did not train teachers on the materials or distribute materials to classrooms; as a result, these materials were not used in UNRWA classrooms. Since at least 2015, State has used several means to identify and address Palestinian Authority textbook content it deemed problematic, including examining nongovernmental organizations’ allegations about problematic Palestinian Authority textbook content, engaging with Palestinian Authority officials, and monitoring UNRWA’s efforts. UNRWA reported that it had reviewed 111 textbooks used in its West Bank and Gaza schools during three sessions since 2016 to identify content it deemed not aligned with UN values. UNRWA reported that it had developed specific complementary teaching materials for any page identified to address this content following each of the reviews. In addition, UNRWA reported that it had trained some field-level education staff but had not trained teachers on the materials or distributed materials to classrooms for several reasons including staff refusal to attend training and workshops. UNRWA reported that it reviewed the Palestinian Authority and English language textbooks in part based on the values contained in its Framework for the Analysis and Quality Implementation of the Curriculum (Curriculum Framework), through which UNRWA aims to ensure that the curricula taught in its schools reflect UN values, such as neutrality, tolerance, equality, and nondiscrimination, and human rights with regard to race, gender, language, and religion. However, UNRWA explained that, given the urgency of reviewing any newly issued textbooks, it developed a “rapid review” process. Appendix II provides an overview of the Curriculum Framework and rapid review processes. UNRWA reported conducting three rapid reviews of all newly released Palestinian Authority textbooks since 2016, in each case using the rapid review criteria as a guide: beginning in October 2016, for textbooks for the first semester of grades 1 through 4; beginning in January 2017, for textbooks for the second semester of grades 1 through 4; and beginning in August 2017, for all textbooks used in UNRWA schools for the first semester of grades 1 through 10. UNRWA slightly revised the criteria used over the course of its three rapid reviews. UNRWA officials noted that for the first rapid review, they reviewed textbooks to determine if the textbooks were aligned with UN values and the UN commitment to neutrality. For the second rapid review, UNRWA developed three criteria: (1) neutrality/bias, (2) gender, and (3) aggressiveness. For the third rapid review, UNRWA renamed the criterion of aggressiveness to age-appropriateness to better reflect the types of issues it was intended to capture. The criteria for the third rapid review are 1. neutrality/bias: taking sides or engaging in controversies of a political, racial, religious, or ideological nature 2. gender: gender stereotypes 3. age-appropriateness (formerly aggressiveness): content that is violent, frightening, or inappropriate for the student’s age. Appendixes II and III provide more detail on UNRWA’s textbook reviews. In fall 2017, UNRWA reported to donors that, based on its rapid review criteria, its August 2017 review identified issues on 3.1 percent of the pages in the 75 textbooks for the first semester of grades 1 through 10 used during the school year 2017-2018. In particular, UNRWA identified 203 issues covering a total of 229 pages (out of a total of 7,498 pages reviewed), the majority of which they identified as related to neutrality/bias. According to UNRWA-provided information, UNRWA found no cases of incitement to violence in the Palestinian Authority grades 1 through 10 textbooks during the August 2017 rapid review. More than half of the neutrality/bias issues it found were related to one of the following three categories—maps, Jerusalem, and cities—for example, regional maps that exclude Israel and refer to Israeli cities as Palestinian. Additional details about the issues UNRWA identified and the complementary teaching materials it developed have been omitted from this report because the information is classified. In addition to issues UNRWA identified using the three rapid review criteria, it identified positive attributes in the textbooks newly issued by the Palestinian Authority, such as promoting active learning, life skills, gender equality, higher-order thinking, and problem-solving skills, according to UNRWA officials. For the content that UNRWA identified as not aligned with UN values during all three rapid reviews, UNRWA officials reported that they developed specific complementary teaching materials for any page with issues identified, such as alternate photos, examples, and guidance for teachers, as needed, to use with the textbooks in UNRWA schools. UNRWA also developed training guides and presentations to support training on the complementary teaching materials for each of the reviews. According to UNRWA, it developed these materials to ensure that the lessons taught in UNRWA schools adhere to UN core values, such as neutrality. In addition, UNRWA officials reported that they trained some field-level education officials but were not able to train teachers or distribute materials to classrooms. UNRWA officials told us that UNRWA did not change the content of Palestinian Authority textbooks and that they do not have the authority or mandate to do so. UNRWA developed complementary teaching materials to address the following issues it identified, among others, during its rapid review process of pilot textbooks for the second semester of grades 1 through 4 textbooks. Details about these complementary teaching materials were omitted because the information is classified. For details about the issues UNRWA addressed, see appendix IV. UNRWA officials told us that as of April 2018 they have reviewed all textbooks for the second semester of grades 1 through 10. UNRWA did not train teachers or complete distributing complementary teaching materials after its first rapid review for several reasons. In a January 2017 briefing note to the United States and other donors, UNRWA reported that it had completed training for professional support staff on the complementary teaching materials for the pilot textbooks for the first semester of grades 1 through 4. However, UNRWA officials told us that UNRWA was not able to deliver the training for school staff, including principals or teachers, or disseminate these materials to classrooms before the end of the first semester of the 2016-2017 school year. They noted that this was due to collective employment actions between August 2016 and January 2017, including staff walkouts and a refusal to attend training and workshops, that were unrelated to the curriculum reform and having to complete the school exam period immediately following the resolution of these collective employment actions. For similar reasons, UNRWA was unable to distribute materials or train teachers after the second rapid review of pilot textbooks for the second semester of the 2016-2017 school year. UNRWA reported to the United States and other donors in March 2017 that it anticipated completing training on the complementary teaching materials for all professional support staff and teachers by the end of that month in the West Bank and by the end of the following month in Gaza, according to State/PRM officials. However, UNRWA officials told us that UNRWA halted the training following a Palestinian Authority announcement of suspension of ties with UNRWA in response to UNRWA’s use of complementary teaching materials, and the UNRWA staff union reactions. UNRWA then determined that these materials would be outdated because the Palestinian Authority planned to issue revised textbooks in August 2017, before the start of the new school year. UNRWA’s efforts to train teachers and issue complementary teaching materials as a result of the third rapid review were ongoing as of December 2017. As of that date, UNRWA officials told us that UNRWA had finalized the complementary teaching materials for the final textbooks for the first semester of grades 1 through 4 and pilot textbooks for grades 5 through 10, as well as the English Language textbooks for the first semester of grades 1 through 10, all of which are being used during the 2017-2018 school year. UNRWA officials told us that UNRWA has developed training materials for the final textbooks for first semester grades 1 through 4 and pilot textbooks for grades 5 through 10 and planned to begin training of all relevant professional support staff, who will, in turn, train teachers using a cascaded training model. In addition, UNRWA officials reported sharing the complementary teaching materials in PDF format with field education staff in the West Bank and Gaza for distribution to all teachers. However, in commenting on a draft report, UNRWA officials told us in April 2018 that they did not disseminate the training or the complementary teaching materials for the third rapid review for various reasons. For example, some UNRWA staff opposed the use of these materials in classrooms while other staff boycotted the training. In addition, UNRWA faced deteriorating operational and political environments during that time period, such as financial shortfalls, as well as an increased number of violent confrontations between Palestinians and Israeli Security forces in the West Bank and Gaza. According to UNRWA, these factors heightened sensitivities and risks associated with the training and curriculum enrichment materials. As a result, these materials were not used in UNRWA classrooms. To promote appropriate content in Palestinian Authority textbooks, State/ConGen officials have examined nongovernmental organizations’ studies and allegations about potentially problematic Palestinian Authority textbook content and confirmed instances of problematic material since fiscal year 2015. State/ConGen officials told us that the studies they reviewed raised concerns with a range of content, and they will continue their reviews of these studies in the future. In examining Palestinian Authority textbooks, State/ConGen has found material that ignores Israeli narratives, includes militaristic and adversarial imagery, and preaches the values of resistance, according to State officials. Although according to State officials there has been a general agreement in these studies on the absence of anti-Semitic content or explicit incitement to violence in Palestinian Authority textbooks, State/ConGen nonetheless has confirmed instances of inappropriate language, content, and imagery based on the grade level of certain textbooks. State/ConGen also noted that the textbooks do not mention Israel or Judaism, and they continue to include regional maps that exclude Israel. In response to allegations that two textbooks in particular—the National and Social Education (civics) textbooks for grades 3 and 4—contained problematic content, State/ConGen officials reported that they translated them into English and then analyzed two new pilot civics textbooks for grade 4 for the first and second semesters as well as previous versions of the same books and contracted for an external review of the textbooks. State/ConGen officials selected these textbooks for translation and analysis to examine a smaller subset of material reviewed in one independent study. State/ConGen officials told us in September 2017 that they had received the results of the external review and that these results informed their advocacy efforts and provided external perspective on additional material. To address incitement to violence, such as the inclusion of problematic content in textbooks, State/ConGen officials have engaged the highest levels of the Palestinian Authority officials, according to State officials. State/ConGen officials reported that, since 2015, they have encouraged Palestinian officials during these meetings to address incitement to violence in textbooks, and Palestinian officials have done so. Officials also noted that the Palestinian Authority President has publicly condemned incitement to violence and vowed to combat it. A case study of a particularly problematic lesson illustrates State/ConGen’s role and approach. State/ConGen officials reported that a specific math problem using the number of Palestinian casualties in the First and Second Intifadas (uprisings) was clearly objectionable even if it did not demonstrate a call for violence against Israel. The Consulate and Consul General subsequently raised this concern with Palestinian officials, including the Minister of Education. To discuss the Palestinian Authority’s ongoing textbook reform and address potential concerns, State/ConGen officials reported that they also convened a meeting in April 2017 of international donor groups and members of the international community that participate in the Palestinian-led Education Sector Working Group. A State official said that the group conducted a wide-ranging discussion about incitement to violence and agreed to discuss incitement bilaterally with the Palestinian Authority as appropriate. State/ConGen continued to raise the issue with the Palestinian Authority following the meeting. In accordance with State/PRM’s role in monitoring UNRWA’s efforts to identify and address potentially problematic content in Palestinian Authority textbooks, State/PRM reports that it engages regularly with UNRWA. It does so through reviews of UNRWA reports, site visits to UNRWA schools and classrooms when and where security permits, regular communication with UNRWA staff at UNRWA headquarters and in the field, and by attending UNRWA’s briefings on the status of its textbook reviews. In addition, State/PRM officials aim to ensure that UNRWA takes adequate steps to ensure neutrality in UNRWA’s operations. To do so, State/PRM meets regularly with UNRWA officials to ensure that UNRWA operates in a fully neutral way in line with UN standards across all sectors of operation, including education and content of textbooks. State/PRM submitted annual reports to Congress in response to provisions in the annual appropriations acts for fiscal years 2015, 2016, and 2017; however, these reports have several limitations regarding educational assistance. First, we found that State/PRM’s 2017 report inaccurately described certain UNRWA actions to address textbook content not aligned with UN values. Inaccurate information about UNRWA’s actions could limit the transparency of State’s and UNRWA’s activities and the usefulness of State’s reports as tools for congressional decision making and oversight. Second, while State’s reports explain generally how UNRWA is taking steps to ensure that educational materials in UNRWA schools are consistent with certain values, we found that the reports did not include some information about UNRWA’s textbook review that could be useful for congressional oversight. Specifically, State’s reports did not specify whether the educational materials are consistent with the value of dignity or not inducing incitement. In addition, we found that in its 2017 report, State did not include information provided by UNRWA about the nature and extent of content that UNRWA identified in Palestinian Authority textbooks as not aligned with UN values. This information, while not required by law to be included in State’s reports, could be useful to congressional decision- makers. State submitted reports to Congress each year in a timely manner in accordance with the requirements of the appropriations acts. In the annual appropriations acts for fiscal years 2015 through 2017, Congress required State to report on seven different topics, including whether UNRWA is taking steps to ensure that the content of all educational materials taught in UNRWA schools and summer camps is consistent with the values of human rights, dignity, and tolerance, and does not induce incitement. State’s reports explain that UNRWA applied its Curriculum Framework in reviewing textbook content and that the Curriculum Framework will help ensure all materials used in UNRWA classrooms reflect UN values and principles. These UN values address issues related to neutrality, human rights, tolerance, and non-discrimination. These values are aligned with the ones that are included in the laws, according to State officials. However, we found that State’s 2017 report to Congress inaccurately described some of UNRWA’s actions to address content that is not aligned with UN values. State correctly reported that UNRWA completed several actions related to its second rapid review, including that UNRWA reviewed 18 new Palestinian Authority pilot textbooks, with a particular focus on the issues of neutrality and bias, gender, and aggressiveness. However, State reported that UNRWA trained teachers on the application of the complementary teaching materials they developed and disseminated the materials to classrooms, actions that UNRWA officials told us they did not complete. State/PRM officials stated they became aware that UNRWA’s classroom training and dissemination of complementary teaching materials had been delayed in June 2017, after the school year ended and after submitting the report to Congress in May 2017. State/PRM officials stated that, based on conversations they had with UNRWA during tense discussions between UNRWA and the Palestinian Authority in March and April 2017, they believed UNRWA would train teachers and disseminate complementary teaching materials after the tensions dissipated. These officials said they did not provide the congressional report to UNRWA for it to review because it is considered an internal U.S. government document. While State/PRM officials stated they verified facts related to other aspects of the reporting requirement, they did not verify the implementation of training and dissemination of complementary teaching materials because they believed this information to be current given ongoing dialogue with UNRWA. In addition, State/PRM officials told us that they were not aware of the inaccuracy in their report to Congress until we brought it to their attention, although they were aware that the trainings had not been implemented in June 2017. In November 2017—about 6 months after the 2016-2017 school year ended—State/PRM officials told us that their understanding remained that UNRWA had trained some education staff on the application of the complementary teaching materials, though not all teachers, and that UNRWA had disseminated the materials to some education staff and schools, though not to all classrooms. From State’s perspective, the statement in its report to Congress about UNRWA training teachers and disseminating complementary teaching material was partly accurate. However, UNRWA officials confirmed that they did not disseminate the training or the complementary teaching materials related to the second rapid review to any school staff, including principals and teachers. In October 2017, State noted that it has taken, or plans to take, action to address the accuracy of reporting in the future. First, subsequent to learning that the training had been halted in June 2017, State/PRM officials reiterated to UNRWA the need to keep them informed in a timely manner when the situation in the field shifts with regard to textbooks and other issues. State/PRM officials also said that they would likely avoid misreporting facts in the future by taking additional actions, such as including specific dates of the actions taken in their reports and verifying key facts with UNRWA. Further, they said they plan to address the issue of inaccuracy in the fiscal year 2018 report, if needed. Standards for Internal Control in the Federal Government states that management should use quality information to achieve the entity’s objectives. Incomplete and inaccurate information about UNRWA’s actions could limit the transparency of UNRWA’s activities and usefulness of State’s reports as tools for congressional decisionmaking and oversight. Our analysis also showed that State’s required reports did not include some information that could be useful for congressional oversight of whether UNRWA is taking steps to ensure that all the content of all educational materials currently taught in UNRWA schools and summer camps is consistent with the values of human rights, dignity, and tolerance, and does not induce incitement. In particular, our analysis showed that while State’s reports partly explain how certain educational materials are consistent with two elements included in the law (human rights and tolerance), they do not address the other two elements (dignity and not inducing incitement). In addition, State’s reports do not include details about the nature and extent of content UNRWA identified in Palestinian Authority textbooks as not aligned with UN values. State’s reports for all 3 years partly explain how certain educational materials are consistent with the values of human rights and tolerance but do not specifically say whether the Palestinian Authority textbooks are consistent with these values. In particular, the reports discuss the U.S.- funded Human Rights, Conflict Resolution, and Tolerance project and accompanying teacher toolkit. The toolkit aims to ensure that teachers have the skills and resources to implement human rights education across UNRWA classrooms. The reports note that in Gaza, UNRWA students use a dedicated human rights curriculum anchored in the Universal Declaration of Human Rights. While the Human Rights, Conflict Resolution, and Tolerance project is relevant to the congressional reporting requirement, it is supplemental to the Palestinian Authority textbooks—the core educational materials used in UNRWA’s schools. State’s reports do not discuss whether these Palestinian Authority textbooks are consistent with the values of human rights and tolerance. Moreover, none of State’s reports for these 3 years explicitly state that the UN values UNRWA applied in reviewing textbooks encompass the value of dignity or not inducing incitement. State/PRM officials said that these topics are addressed implicitly, in that the value of “dignity” is encompassed by the concepts of human rights and non-discrimination, which are among the elements encapsulated by the “UN values” applied as part of the Curriculum Framework. State/PRM officials further assert that reporting to Congress on UNRWA’s application of “UN values” via the Curriculum Framework necessarily encompasses the concept of non- inducement of incitement. In State’s view, materials reviewed through the lens of UN values and principles as defined by the UN imply that such review is taking into consideration whether the materials include incitement to violence. However, State did not include language about dignity or not inducing incitement explicitly in its reports to Congress. Regarding the nature and extent of content UNRWA identified in Palestinian Authority textbooks as not aligned with UN values, State did not include details provided by UNRWA about UNRWA’s reviews of Palestinian Authority textbooks in its May 2017 report to Congress that, while not required by law to be included in State’s reports, could be helpful for congressional oversight. The May 2017 report states that UNRWA reviewed pilot textbooks for the first and second semesters of grades 1 through 4 and identified a “limited amount of problematic content in the Palestinian Authority materials.” However, State’s report did not cite the percentage of all pages UNRWA deemed as including content not aligned with UN values, the percentage of issues UNRWA identified in relation to each of the three rapid review criteria, or examples of such content (e.g., frightening pictures that they considered inappropriate for children), which UNRWA had reported to State/PRM and other donors at least 2 months earlier. We have previously reported that agencies should consider the differing information needs of various users to ensure that performance information will be useful in decision making. Standards for Internal Control in the Federal Government states that information should be communicated in a way that is useful to internal and external users. Less thorough information in State’s annual reporting could limit its usefulness as a tool for congressional oversight. In addition, the lack of certain relevant information may limit Congress’ ability to fully assess the nature and extent of material that may not be aligned with UN values in Palestinian Authority textbooks. The United States has funded education for Palestinian children for decades, including an estimated $243 million for fiscal years 2015 through 2017. State funds education projects to improve the quality of education to equip Palestinians with the skills to grow their economy and build a democratic, secular, politically moderate, and outward-focused Palestinian civil society as a driver for peace, according to the Consulate General’s Education Statement of Purpose. Congress remains interested in the role UNRWA plays in educating children under its purview, requiring State to report on steps UNRWA is taking to ensure that the content of all educational materials currently taught in UNRWA- administered schools is consistent with the values of human rights, dignity, and tolerance, and that those materials do not induce incitement. State’s 2017 report inaccurately describes certain UNRWA actions to address content not aligned with UN values. In addition, State’s reports to Congress did not specify whether the educational materials used in UNRWA schools are consistent with the value of dignity or not inducing incitement. Although State’s reports generally discuss whether UNRWA is taking certain steps, the lack of certain relevant information in State’s reports could limit their usefulness as a tool for congressional decision making and oversight. Accurate and complete information would help Congress more fully understand and assess the nature and extent of content in textbooks that is not aligned with UN values, as well as UNRWA’s actions to address this content. We are making the following four recommendations that could further enhance State’s annual reports to Congress: The Secretary of State should direct the Assistant Secretary for Population, Refugees, and Migration to establish a process to ensure that State’s reporting to Congress on the actions UNRWA has taken is accurate. (Recommendation 1) The Secretary of State should direct the Assistant Secretary for Population, Refugees, and Migration to provide information in its reports to Congress that could be useful for congressional oversight, including information that: discusses whether Palestinian Authority textbooks used in UNRWA schools are found to be consistent by UNRWA with the values of human rights and tolerance. (Recommendation 2) explicitly states whether the UN values UNRWA applied as part of the Curriculum Framework encompass dignity and do not induce incitement. (Recommendation 3) describes the nature and extent of textbook content that UNRWA identified as not aligned with UN values, including in the English language textbooks purchased by UNRWA. (Recommendation 4) We provided a draft of our April 2018 classified report to State and USAID for comment. We also provided UNRWA with relevant information for comment. In response, State and UNRWA provided written comments on the classified report. We have reprinted State’s updated written comments in appendix V and UNRWA’s original written comments in appendix VI. All three also provided technical comments, which we incorporated as appropriate throughout our report. In its written comments on this report, State noted that it has implemented all four of our recommendations contained in the classified report we issued in April 2018. To ensure the accuracy of information in its reports, State has developed standard operating procedures for drafting and verifying the information contained in its annual report to Congress on UNRWA, including clearly sourcing all information contained in the report and seeking written verification from UNRWA on any information previously obtained via oral communication. State implemented our recommendation that it discuss whether Palestinian Authority textbooks used in UNRWA schools are found to be consistent by UNRWA with the values of human rights and tolerance. State included additional qualitative details from UNRWA on its evaluation of the Palestinian Authority materials, and the degree to which UNRWA assesses that these materials are consistent with human rights and tolerance. State implemented the recommendation to explicitly state in its reports to Congress whether the UN values UNRWA applied as part of the Curriculum Framework encompass dignity and do not induce incitement. In addition, State implemented the recommendation to describe the nature and extent of textbook content that UNRWA identified as not aligned with UN values, including in the English language textbooks purchased by UNRWA. State provided additional qualitative and quantitative details from UNRWA’s evaluation of Palestinian Authority textbooks in its fiscal year 2018 report based on information provided by UNRWA. In its written comments, UNRWA said, among other things, that while using the curricula and textbooks of host nations, UNRWA’s education program strives to realize the potential of all its Palestine refugee students, to help them develop into confident, innovative, questioning, thoughtful, tolerant and open-minded critical thinkers, who uphold human values and tolerance, and contribute positively to the development of their society and the global community. In addition, UNRWA noted that it appreciates our understanding of the role of the Curriculum Framework and how UNRWA takes specific measures to rapidly review newly issued textbooks, including the large number of new textbooks released by the Palestinian Authority Ministry of Education and Higher Education throughout 2016 and 2017. UNRWA also commented that while it does not have authority to determine or alter national curricula, UNRWA is committed to taking all measures within its control to ensure that the delivery of its educational services is fully aligned with the values of the United Nations. UNRWA did not comment on our recommendations, since they were not directed to UNRWA. We are sending copies of this product to the appropriate congressional committees, as well as the Secretary of State, the Administrator of USAID, the Commissioner-General of UNRWA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9601 or melitot@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. This report examines (1) the amount of funding Department of State (State) and U.S. Agency for International Development (USAID) provided for education assistance to the West Bank and Gaza for fiscal years 2015 through 2017 and how it was used; (2) how the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA) and State have identified and addressed potentially problematic content in educational materials used by schools in the West Bank and Gaza; and (3) whether State has submitted required annual reports to Congress including information on whether UNRWA is taking steps to ensure that the content of all educational materials currently taught in UNRWA- administered schools is consistent with the values of human rights, dignity, and tolerance, and do not induce incitement. To determine which U.S. government agencies provide education assistance for the West Bank and Gaza, we reviewed documents and conducted interviews with State, USAID, and the Overseas Private Investment Corporation (OPIC). We initially conducted an interview with OPIC because it was included in a previous report we issued on a similar topic. We excluded OPIC from the scope of this engagement because they did not provide relevant education assistance to the West Bank or Gaza between fiscal years 2015 and 2017. We focused our review on State and USAID because these agencies provided education assistance to the West Bank and Gaza during this timeframe. For this review, we refer to State and USAID when we refer to the U.S. government. To examine the amounts of funding State and USAID provided for education assistance to the West Bank and Gaza and how it was used for fiscal years 2015 through 2017, we took the following steps. We examined actual funding where it was available and estimated funding where it was not. We obtained and analyzed financial data from State and USAID and expenditure data from UNRWA for education assistance to the West Bank and Gaza in fiscal years 2015 through 2017. We used these data to describe how much and for what types of activities State contributed funds to UNRWA. We also obtained and analyzed expenditure and contributions data from State and obligations data from USAID to describe non-UNRWA education programs that they administered in the West Bank and Gaza. We reported the amount of funds UNRWA expended in general on education in the West Bank and Gaza, including the amounts that UNRWA expended on educational materials and specifically on textbooks. We define educational materials to primarily include curriculum, textbooks, select videos and web-based tools, and any complementary teaching materials, including those developed by UNRWA that aim to supplement, replace, or mitigate materials that UNRWA deems not aligned with UN values. We exclude posters, library books, educational technology, education administration materials, extracurricular materials, handouts and worksheets, and teacher training materials, with limited exceptions, from materials produced by UNRWA and used to mitigate material that UNRWA deemed not aligned with UN values or to supplement the curriculum. According to UNRWA officials, the financial information they provided pertains to educational materials, including textbooks, complementary teaching materials, and costs related to an interactive learning portal in Gaza and UNRWA TV. Finally, we reported the amount of funds USAID obligated for education programs in the West Bank and Gaza during this timeframe. To analyze these data, we reviewed State-UNRWA contribution agreements, State reports on UNRWA emergency appeals expenditures, and USAID award documents. We examined the two types of funding that State contributed to UNRWA—program budget funding and emergency appeals. We also examined the three ways in which UNRWA expends that funding—through program budget expenditures, emergency appeals expenditures, and special project expenditures. We supplemented these data by interviewing State, UNRWA, and USAID officials about funding. While the majority of UNRWA data are actual expenditures, some UNRWA data are estimates. According to UNRWA officials, they estimated all UNRWA expenditure data for fiscal year 2017 because, as of December 2017, when we finished collecting data, UNRWA’s 2017 fiscal year was ongoing. In addition, UNRWA estimated its education expenditures provided by the United States because U.S. contributions to UNRWA are generally not earmarked. Rather, UNRWA’s core budget, its program budget, pools funding from all UNRWA donors. For this reason, we reported all UNRWA expenditure data on education assistance based on information UNRWA officials provided us. To make these estimates, UNRWA officials informed us that they calculated U.S. funding as a proportion of all UNRWA funding, and applied that proportion to their educational expenditures. Data on State’s contributions to UNRWA and USAID’s funding to education programs in the West Bank and Gaza active between fiscal years 2015 and 2017 are obligations; according to State, all funds disbursed to UNRWA were through contributions. Data on State’s funding for non-UNRWA education programs are expenditures. For the purposes of this report, we use the U.S. fiscal year (October 1 through September 30) for all State and USAID contributions data, while we use UNRWA’s fiscal year (January 1 through December 31) for all UNRWA expenditure data. In addition, State and USAID awarded several grants for additional years not included in our scope. For example, the USAID’s first obligation to the Leadership and Teacher Development program occurred in fiscal year 2011 and the latest obligation to that program occurred in fiscal year 2017. As a result, the data presented in this report may include additional contributions of funds beyond what State and USAID obligated for fiscal years 2015 through 2017. To determine the reliability of the obligations and expenditure data, we requested information from State, UNRWA, and USAID officials regarding the processes they used to collect and verify data, and we checked the data for reasonableness and completeness. When we found discrepancies or missing data fields, we worked with relevant agency officials to correct the discrepancies and missing fields. We compared State’s contribution data with UNRWA’s expenditure data to ensure consistency. We discussed UNRWA’s financial data for educational expenditures with knowledgeable officials, reviewed audited financial statements for confirmation, and reviewed vouchers they provided. However, we did not independently audit their financial data. To ensure completeness of the data, we reviewed initial grant documents or contribution agreements and all associated amendments for the (1) six education projects USAID funded in the West Bank and Gaza, and (2) annual UNRWA contributions State made between fiscal years 2015 and 2017. We discussed UNRWA’s procedures for estimating the proportion of U.S. funds that went to educational expenditures with knowledgeable officials. Based on our initial assessments of the data, we determined that the State and USAID funding data we collected were sufficiently reliable for the purposes of this report. In addition, we determined that the actual expenditure data we collected from UNRWA were sufficiently reliable for our purposes, and that the estimated expenditures it provided were reasonable for the purposes of this review. (To examine how UNRWA and State have identified and addressed potentially problematic content in educational materials used by schools in the West Bank and Gaza, we reviewed the policies and procedures that UNRWA and State have established and implemented. We focused on actions agencies took in response to the (1) pilot textbooks for grades 1 through 4 that the Palestinian Authority issued in 2016 and that UNRWA used during the 2016-2017 school year; (2) final textbooks for grades 1 through 4, and pilot textbooks for grades 5 through 10 the Palestinian Authority issued in in 2017 and used during the first semester of the 2017-2018 school year; and (3) English language textbooks that UNRWA and the Palestinian Authority purchased for grades 1 through 10 published in 2011 through 2014 and used during the 2017-2018 school year. According to UNRWA officials, these textbooks do not include the second semester Palestinian Authority textbooks for the 2017-2018 school year (released in late 2017) and the second semester English language textbooks, and therefore do not cover all the textbooks used in UNRWA and Palestinian Authority schools for grades 1 through 10. We examined how UNRWA and State have implemented their policies and procedures. We reviewed State’s cables and agencies’ policy documents and reports and met with officials from State, UNRWA, and USAID in Washington, D.C., and overseas. In addition, we interviewed international donors overseas and officials from the government of Israel, the Palestinian Authority, and Jerusalem municipality. We only interviewed official government entities and public international organizations and did not meet with non-governmental interest groups. We followed up with relevant officials on multiple occasions to assess the progress of textbook review and the status of implementation of other policies and procedures. We interviewed UNRWA officials about the methods they used to conduct the rapid reviews of textbook content and reviewed documents they provided that outline their procedures. While the methods and procedures described seemed generally reasonable, we did not independently review UNWRA’s underlying documents to fully assess the reliability of the rapid review results it reported because UNRWA is an international organization. Moreover, it was beyond the scope of our review to examine the underlying documents and textbooks themselves, most of which are written in Arabic. There can be a number of challenges to analyzing and coding content as UNRWA did in its rapid reviews, such as the need for those performing the review to exercise judgment, and while the overall process officials outlined generally appeared reasonable, we cannot comment on the extent to which it successfully overcame all of the potential challenges. We are presenting the results of the textbook reviews, attributed to UNRWA, to help support our finding that the agency has developed procedures to review textbooks, and that it found some concerns in its recent reviews. In addition, we are providing details about these reviews for context because the State Department summarized the results of the first two reviews in its May 2017 report to Congress, which we discuss in the third section of this report. This report is a public version of a classified report that we issued in April 2018. The Department of State deemed some of the information in our April 2018 report to be classified, which must be protected from loss, compromise, or inadvertent disclosure. Therefore, this report omits classified information about neutrality/bias, gender issues, and other textbook content identified in English language textbooks by UNRWA as not aligned with UN values. Although the information provided in this report is more limited, the report addresses the same objectives as the classified report and uses the same methodology. To examine whether State has submitted required annual reports to congressional committees, including information on whether UNRWA is taking steps to ensure that the content of all educational materials currently taught is consistent with the UN values of human rights, dignity, and tolerance, and do not induce incitement, we took the following steps. We reviewed the legal requirements for State to report on the steps UNRWA is taking to ensure that the content of all educational materials currently taught is consistent with the UN values. These requirements are found in the annual appropriations acts; for fiscal year 2017, the requirement is located in Section 7048(d)(5) of the Consolidated Appropriations Act, 2017. We reviewed State’s reports to Congress in 2015, 2016, and 2017, and compared data State reported regarding education assistance with data we gathered through meetings with State and UNRWA officials in in Washington, D.C., and overseas. We also reviewed UNRWA documents. The performance audit upon which this report is based was conducted from January 2017 to April 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with State from February 2019 to June 2019 to prepare this unclassified version of the original classified report for public release. This public version was also prepared in accordance with these standards. UNRWA’s Framework for Analysis and Quality Implementation of the Curriculum (Curriculum Framework) provides the overarching structure for the review and enrichment of educational materials used in UNRWA schools in all of its fields of operation, including the West Bank and Gaza. The Curriculum Framework, developed as part of UNRWA’s education reform process, aims to ensure that the curricula taught in its schools support the development of skills and competencies that are considered important for individual development in the 21st century. In addition, the Curriculum Framework aims to ensure that the delivery of the host country’s curriculum reflects UN values, such as neutrality, tolerance, equality, and nondiscrimination, and human rights with regard to race, gender, language, and religion as well as the development of respect for a child’s own cultural identity, language, and values in line with UN values. According to UNRWA officials, neutrality is one of the four “humanitarian principles” formally adopted by the UN General Assembly and endorsed by UNRWA and is a core obligation and value of UN staff as spelled out in the UN’s regulatory framework. According to UN humanitarian principles, the concept of neutrality means that, irrespective of their personal beliefs and opinions, “umanitarian actors must not take sides in hostilities or engage in controversies of a political, racial, religious or ideological nature.” The Curriculum Framework includes 10 Curriculum Framework principles and five student competencies against which UNRWA reviews educational materials used in its schools: Principle 1—Focuses on understanding and application and not just memorization Principle 2—Is active, practical, and encourages independent thinking Principle 3—Is relevant to students’ lives and situation, particularly as Principle 4—Provides a variety of teaching and learning approaches Principle 5—Integrates learning and emphasizes connections to other Principle 6—Is inclusive and provides learning opportunities for Principle 7—Provides for students’ personal development and well- Principle 8—Is free of biases (such as gender, disabilities, and ethnicity) Principle 9—Enables students to value their Palestinian culture, Principle 10—Reflects UN values Curriculum Framework Student Competencies: 1. Critical and creative thinking 3. Communication and literacy UNRWA’s Curriculum Framework includes tools to guide the analysis and review of host country textbooks and other learning material at the school and field levels, and remains the overarching framework for the review and enrichment of educational materials used in UNRWA schools agency-wide. However, given the urgency of reviewing any newly issued textbooks for use during the 2016-2017 school year, UNRWA developed a rapid review process. The rapid review process does not replace the Curriculum Framework process, as the Palestinian Authority textbooks reviewed through the rapid review process are also subject to the regular Curriculum Framework review process at the field and school levels, as follows: At the field level, field education staff are to use the Field-Level Analysis Tool of the Curriculum Framework to review textbooks against all five student competencies and 10 principles of the Curriculum Framework. At the school level, all UNRWA teachers and school principals in the West Bank and Gaza and UNRWA’s other fields of operations are to use the School-Level Analysis Tool of the Curriculum Framework to review their own teaching programs and lessons, including curriculum materials they use, while considering their context and diversity of needs. The School-Level Analysis Tool focuses on the five Student Competencies and select Curriculum Framework Principles: (1) Principle 4—provides a variety of teaching and learning approaches; (2) Principle 6—is inclusive and provides learning opportunities for students of all abilities; (3) Principle 8—is free of biases (such as gender, disabilities, and ethnicity); (4) Principle 9—enables students to value their Palestinian culture, heritage, and identity; and (5) Principle 10—reflects UN values. The Curriculum Framework is a more comprehensive pedagogical review—one that relates more directly to the theory and practice of education—than the rapid review process, which focuses specifically on three rapid review criteria linked to the UN values in the Curriculum Framework. According to UNRWA documents, UNRWA employed a multi-stage rapid review process to identify textbook content not aligned with UN values, and its efforts to address this content were ongoing as of November 2017. Figure 3 summarizes UNRWA’s process. Complementary teaching materials are educational materials that UNRWA developed to use alongside host government textbooks to ensure that the lessons taught in UNRWA schools adhere to UN core values, such as neutrality, according to UNRWA officials. UNRWA’s Agency Task Force is composed of the Chief of Staff and headquarters officials from the departments of Education and Legal Affairs, according to UNRWA officials. The cascade training model involves training groups of individuals who in turn train other individuals. UNRWA has established strategic support units in the fields that train educational specialists who then train school principals and teachers using a cascade model, according to UNRWA officials. Professional support staff include field-level strategic support unit staff, education specialists, and Chiefs of the Field Education Programs, according to UNRWA officials. Appendix III: 2016-2017 Rapid Review, as Reported by the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA) UNRWA reported that it has reviewed newly issued Palestinian Authority textbooks during three rapid review sessions since 2016 to identify content it deems not aligned with UN values and that it has developed complementary teaching materials to specifically address this content for any page with issues identified. Throughout the 2016-2017 school year, UNRWA reported reviewing pilot textbooks newly issued by the Palestinian Authority for grades 1 through 4 in two separate reviews. In August 2017, UNRWA reported reviewing the final textbooks for grades 1 through 4 for the first semester, pilot textbooks for grades 5 through 10 for the first semester, and English language textbooks funded with contributions from donor countries, including the United States, for grades 1 through 10 for the first semester. For the August 2017 review, UNRWA reported reviewing 75 textbooks (7,498 pages) in aggregate. Table 1 provides details on the number of textbooks and number of pages UNRWA reported reviewing between 2016 and 2017 for the textbooks used in its schools in the West Bank and Gaza. Table 2 provides detail on the academic subjects for which UNRWA reported reviewing Palestinian Authority textbooks in 2016 and 2017. Table 2. Select Academic Subjects for Which the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA) Reported Reviewing Content of Palestinian Authority and English Language Textbooks, 2016-2017 recitation) Legend: =UNRWA reviewed textbook for this subject. N/A= Not applicable because Palestinian Authority and UNRWA schools do not use these textbooks for the grades listed. Appendix IV: Textbook Content Issues Identified by the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA) Appendix IV: Textbook Content Issues Identified by the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA) During its August 2017 review of textbooks for grades 1 through 10 for the first semester, UNRWA identified 203 issues covering a total of 229 pages (out of a total of 7,498 pages reviewed), the majority of which it identified as related to neutrality/bias. Specific details about the percentage of pages with issues UNRWA identified in relation to each of the three rapid review criteria subjects, as well as the types and percentages of neutrality/bias issues UNRWA reported finding were omitted because the information is classified. Of the 203 issues UNRWA identified in the textbooks for the first semester of grades 1 through 10 for the 2017-2018 school year, UNRWA officials reported that they identified the largest number of issues in social studies textbooks (105 issues), followed by Arabic grammar (30 issues), Islamic education (20 issues), mathematics (18 issues), science and life (15 issues), English language (14 issues), and vocational education (1 issue). The 14 issues that UNRWA identified in the English language textbooks purchased by UNRWA for the first semester of grades 1 through 10 cover a total of 22 pages out of 664 textbook pages (3.3 percent), according to UNRWA officials. Of the 14 issues, UNRWA officials identified 10 of the 14 as neutrality/bias issues and 4 as gender issues. The neutrality/bias issues that UNRWA identified include issues related to maps, Jerusalem, and the Islamic religion. Details about the neutrality/bias and gender issues that UNRWA identified and the complementary teaching materials it developed were omitted because the information is classified. UNRWA officials identified four examples in the English language textbooks for the first semester of grades 1 through 10 that show a lack of gender balance in sports, hobbies, and professions. In response, they developed complementary classroom discussion questions to discuss gender bias with UNRWA students. Details about the gender issues that UNRWA identified and the complementary teaching materials UNRWA developed were omitted from this report because they included classified information. Appendix VI: Comments from the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA) Thomas Melito at (202) 512-9601 or melitot@gao.gov. In addition to the contact named above, Cheryl Goodman (Assistant Director), Jaime Allentuck (Analyst in Charge), Ashley Alley, Martin de Alteriis, and Lynn Cothern made key contributions to this report. Other contributors to this report include Neil Doherty, Mark Dowling, Aldo Salerno, and Mona Sehgal.", "summary": "The U.S. government has funded education assistance to Palestinians. The State Department oversees U.S. contributions to UNRWA, and USAID provides assistance to Palestinian Authority schools. UNRWA generally administers schools for Palestine refugees. The Palestinian Authority generally administers schools for non-refugee Palestinians who live in the WBG. During the 2016-2017 school year, it issued new pilot textbooks for grades 1 through 4 for use in both its and UNRWA's schools. GAO was asked to review issues related to U.S. education assistance to the WBG. This report examines (1) the funding the U.S. government provided for education assistance to the WBG for fiscal years 2015 through 2017, (2) how UNRWA and State have identified and addressed potentially problematic content in textbooks, and (3) whether State has submitted required annual reports to Congress including information on educational materials used in UNRWA schools. To address these objectives, GAO reviewed documents and interviewed U.S. government, UNRWA, and Palestinian Authority officials. For this report, GAO refers to potentially problematic content as that which State defined as inappropriate and that UNRWA defined as not aligned with UN values. The U.S. government funded an estimated $243 million for education assistance in the West Bank and Gaza (WBG) for fiscal years 2015 through 2017, including an estimated $193 million from the Department of State (State) and about $50 million from the U.S. Agency for International Development (USAID). Of State's contribution of approximately $193 million, the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA) estimated that about $187 million was provided for its education assistance. State provided the remaining approximately $6 million for non-UNRWA education projects. UNRWA purchased English language textbooks used in UNRWA schools with funds that consist of contributions from donor countries, including the United States. The U.S. government and UNRWA did not fund textbooks published by the Palestinian Authority because the Palestinian Authority provided these textbooks free of charge, according to agency officials. UNRWA and State have taken steps to identify and address potentially problematic content of textbooks used in UNRWA schools, such as maps that exclude Israel. UNRWA reviewed textbooks, including English language textbooks, and took actions to address content it deemed as not aligned with UN values. For example, UNRWA created complementary teaching materials, such as alternate photos, examples, and guidance for teachers to use with the textbooks in UNRWA schools. However, due to financial shortfalls and other constraints, UNRWA officials told GAO that UNRWA did not train teachers or distribute the complementary teaching materials to classrooms. As a result, these materials were not used in UNRWA classrooms. To address textbook content deemed problematic, State examined nongovernmental organizations' studies, encouraged Palestinian Authority officials to address the issue, and monitored UNRWA's efforts. The annual appropriations acts for fiscal years 2015 through 2017 require State to report to Congress on several topics, including steps UNRWA has taken to ensure that the content of all educational materials taught in UNRWA schools is consistent with the values of human rights, dignity, and tolerance, and do not induce incitement. Although State submitted its required reports to Congress on time, State included inaccurate information in the 2017 report and omitted potentially useful information in all three reports. In its 2017 report, State noted incorrectly that UNRWA had completed training teachers and distributed complementary teaching materials to address textbook content that UNRWA deemed as not complying with UN values. In all three of the reports, State omitted information concerning whether UNRWA found that any educational materials used in its schools do not comply with two of four elements, dignity and not inducing incitement. Standards for Internal Control in the Federal Government states that management should use quality information to achieve the entity's objectives and communicate it in a way that is useful to users. Without a fuller explanation, Congress may not have the information it needs to oversee efforts to identify and address potentially problematic textbook content. GAO made four recommendations in our April 2018 report that State improve its reports to Congress, including to ensure the information presented is accurate and to provide additional information on the textbook content UNRWA identified as not aligned with UN values. State implemented all of GAO's recommendations.", "document_type": "gao"}
{"report": "Table 1 describes the activities that USDA’s mission areas and major staff offices perform as part of five types of administrative services that USDA business centers are to provide under the Secretary of Agriculture’s November 2017 memorandum. At USDA, eight mission areas and three of the 13 major department-level staff offices, including five sub-offices, are responsible for delivering or overseeing these five types of administrative services (see fig. 1). USDA’s eight mission areas carry out the department’s program responsibilities through 18 agencies. Five mission areas consist of multiple agencies, while three consist of a single agency, as shown below. In general, USDA’s eight mission areas deliver the administrative services, and the staff offices develop regulations, guidance, and policies describing how mission areas should deliver those services and oversee the mission areas’ performance. In addition, the staff offices deliver some administrative services on a department-wide or shared-services basis. According to USDA officials, the mission areas are to follow the regulations, guidance, and policies developed by the staff offices but are allowed considerable discretion in how they deliver administrative services based on their missions and program needs. According to USDA officials and documentation, service delivery is typically handled by a mission area’s field offices at the regional, state, or local level; however, with the establishment of the business centers, more service is being delivered at the mission area’s headquarters level. USDA has consolidated administrative services and established business centers in all of its eight mission areas in accordance with the Secretary’s November 2017 memorandum. The eight existing business centers vary in when they were established. As shown in figure 2, three mission areas had business centers before the Secretary’s memorandum. However, even the mission areas that had business centers before the Secretary’s November 2017 memorandum subsequently changed the way they provide administrative services, specifically with regard to information technology services. Two mission areas—Marketing and Regulatory Programs and Research, Education, and Economics—added information technology to their business centers during fiscal year 2019. In 2019, the Natural Resources and Environment mission area, which already included information technology in its business center, changed the position descriptions of certain employees to more accurately reflect that their major duties are considered to be information technology work. Of the five new business centers established since the Secretary’s memorandum, establishment of the FPAC Business Center entailed the most significant transformation. Typically, each business center is located within one of the mission area’s component agencies and the center’s leader reports directly to that agency’s leadership (see table 2). The FPAC Business Center is the only business center established as a separate agency within a mission area. Changes that occurred at other mission areas in transitioning to new business centers included modifying reporting structures for services that had already been consolidated. For example, according to Rural Development officials, the mission area had a business services entity prior to the Secretary’s memorandum. To establish a business center as envisioned by the Secretary’s memorandum, the mission area changed the reporting structure for administrative operations in the field. Previously, field employees associated with an administrative service reported directly to leadership in Rural Development’s state offices. These employees now report directly to headquarters leadership specific to their administrative service. However, according to Rural Development officials, no employees were physically moved. As of November 2019, most of the business centers were providing all five of the main administrative services that the Secretary’s November 2017 memorandum envisioned—specifically, financial management, human resources, information technology, procurement, and property management. Two business centers have chosen to provide financial management services differently from the other administrative services. Specifically: Food Safety. According to officials in the Food Safety mission area, as part of its reorganization, that mission area grouped all of the administrative services except financial management under the Chief Operating Officer. However, it grouped the budget office, which performs financial management services, under the agency’s Chief Financial Officer because it preferred to keep this office with mission- related program offices, which report directly to the Deputy Administrator. Natural Resources and Environment. Officials in the Natural Resources and Environment mission area said that unlike other administrative services, which are grouped under the business center, financial management responsibilities are divided between the business center’s Office of Strategic Planning, Budget, and Accountability and the Forest Service’s Office of the Chief Financial Officer. According to these officials, this arrangement strengthens internal controls by separating responsibility for allocating and spending financial resources from responsibility for accounting for how the resources are spent. One business center—in the Trade and Foreign Agricultural Affairs mission area—provides information technology and financial management services for Foreign Agricultural Service employees and has agreements in place with other USDA components to provide human resources, procurement, and property management services for the mission area. According to the Deputy Assistant Secretary for Administration, USDA accepted these mission areas’ decisions about financial management because they ensured accountability of field-level staff to the administrative service’s headquarters leadership. According to USDA’s Deputy Assistant Secretary for Administration, the department regularly reviews data on administrative services, including services provided by the business centers. However, the department does not use these or other data to assess the effectiveness and impact of its business centers and as of November 2019 did not plan to do so. Beginning in 2018, USDA created an online monitoring system to compile data from mission areas on the status of their administrative services. The system has “dashboards” displaying data specific to financial management, human resources, information technology, procurement, and property management, among other things. Each of the dashboards presents metrics gathered from various databases across mission areas. For example, the dashboards for human resources include the number of employees by organization, along with their geographic location, retirement eligibility, occupation, and any skills gaps. According to USDA officials, the dashboards allow department-level review of a large number of metrics on a range of administrative activities performed by the business centers—data that previously were available only to each mission area. USDA’s Deputy Secretary discusses performance on various dashboards with mission area and staff office leadership at quarterly review meetings. However, the department has not used dashboards or associated metrics to assess the effectiveness and impact of the business centers. Specifically, the department has not assessed the impact that the business centers have had on USDA’s customer service; human resources, including hiring; and overall functionality. According to the Deputy Assistant Secretary for Administration, creating new business centers and changing existing ones has contributed to positive results, such as savings from reducing the size of USDA’s vehicle fleet, but USDA’s Departmental Administration has not systematically compared USDA’s ability to deliver its administrative services before and after these reforms. For example, the department has not examined whether the reforms have enabled mission areas to reduce costs, reduce processing times, or identify previously unknown issues that need to be addressed. According to USDA officials, these business center reforms broadly addressed the first policy goal in USDA’s May 2018 strategic plan for fiscal years 2018 through 2022—namely that USDA programs be delivered efficiently, effectively, and with integrity and a focus on customer service. However, USDA officials told us that they have not yet attempted to measure how the business center reforms have met the three overarching policy goals identified in the Secretary of Agriculture’s November 2017 memorandum, which called for the business center reforms to (1) improve customer engagement, (2) maximize efficiency, and (3) improve agency collaboration. In addition, some stakeholders we interviewed expressed concern about progress toward these goals as USDA works to implement the business center reforms. For example: Staffing vacancies. Some stakeholders raised concerns about the impact of vacancy rates at business centers on customer engagement. The two largest business centers created since November 2017—in FPAC and Rural Development—had position vacancy rates above 27 percent as of September 30, 2019. Officials with one group representing farmers who are customers of the FPAC and Rural Development mission areas told us they were concerned that (1) vacancies in the business center may be leading to vacancies among program staff in the field, (2) complaints related to staffing have increased over the past few years, and (3) staffing vacancies in the field are negatively affecting customer service. An official from another group representing farmers told us that the group is hearing from its members that there have been a lot of changes within USDA lately and field offices seem to be understaffed and overwhelmed even after the creation of the business centers, which could be negatively affecting the quality of customer service. Vacancies at the FPAC and Rural Development business centers, particularly among staff responsible for hiring USDA program staff in the field, could therefore affect both access to and the quality of technical assistance. Employee concerns. In the FPAC Business Center, officials from one union representing employees told us that confusion among employees about their roles and responsibilities could affect both internal employee satisfaction and the overall ability of the business center to serve the FPAC mission and its customers. Specifically, these union officials noted employees’ confusion about how to reconcile differences among the work procedures that each of the three FPAC agencies used before the reorganization. Officials from this and one other union also stated that employees have reported that business center leadership has not taken action to address such employee concerns. As a result, according to officials from both unions, FPAC business center employees are experiencing low morale, confusion, frustration, and anxiety about the changes, affecting their ability to deliver services. In response, FPAC officials told us in November 2019 that the FPAC Business Center is working on empowering employees, hiring, establishing a culture of accountability, building trust and engagement, and addressing other issues that have arisen in the business center’s first year of operation. For example, these officials said they were reviewing the business center’s organizational structure to determine whether there is a need for adjustments to further streamline operations and improve service. USDA officials cited several reasons the department has not assessed the effect of the business center reform effort undertaken in response to the Secretary’s November 2017 memorandum. According to the Deputy Assistant Secretary for Administration, the absence of evaluation is partly attributable to the department’s strategy of delegating responsibility to the mission areas to implement business centers; this strategy aims to give the mission area leadership ownership of the reform effort and help ensure their buy-in. The Deputy Assistant Secretary for Administration also said that the department has focused on implementing the reforms called for in the memorandum rather than on evaluating the results. USDA officials also pointed out that the reform effort is relatively recent, with five of the business centers having been created since June 2018. However, the Deputy Assistant Secretary for Administration acknowledged the importance of evaluating and communicating any benefits derived from the business center reform effort as it moves forward. Our prior work has shown that a key practice to consider during agency reform efforts is the establishment of clear outcome-oriented goals and performance measures to assess the reform’s effectiveness and impact. As we have previously reported, a performance goal is a target level of performance expressed as a measurable objective; a performance measure includes an assessment of results compared with intended purpose that can be expressed quantitatively or in another way that indicates a level or degree of performance. Monitoring performance against goals allows agencies to assess progress and address problems as necessary. While USDA has not developed goals and measures to assess the effectiveness and impact of the business center reforms, the department has set goals for a limited number of administrative services, including hiring, the number of fleet vehicles, and travel and conference spending. In addition, parts of the department have developed goals and measures for the administrative services their business centers provide. For example, officials in the Research, Education, and Economics mission area reported nine key performance indicators for their administrative services, such as specific goals and measures for the timeliness of posting job opportunity announcements. Developing appropriate performance goals and measures and systematically assessing the effectiveness and impact of the business center reforms could help the department determine whether the reforms are meeting the Secretary’s overarching policy goals and improving the delivery of administrative services to support the department’s mission and program goals. USDA has established business centers in all of its eight mission areas, and, according to USDA’s Deputy Assistant Secretary for Administration, the department regularly reviews data on administrative services, including services provided by the business centers. However, the department has not systematically assessed whether USDA’s ability to deliver its administrative services has improved since the establishment of its business center reforms or whether the reforms are meeting the policy goals that the Secretary intended them to achieve. Importantly, the department has not assessed the impact that the business centers have had on USDA’s customer service; human resources, including hiring; and overall functionality. Our prior work has shown that a key practice to consider during agency reform efforts is the establishment of clear outcome-oriented goals and performance measures to assess the reform’s effectiveness and impact. The department has set goals for a limited number of administrative services, including hiring, the number of fleet vehicles, and travel and conference spending, but it has not developed goals and measures to more broadly assess the effectiveness and impact of the business center reforms. Developing such goals and measures and using them to assess the effectiveness and impact of the business center reforms could help the department (1) determine whether the reforms are meeting the Secretary’s overarching policy goals and (2) identify whether the reforms have enabled mission areas to improve the delivery of their administrative services by, for example, reducing costs, reducing processing times, or identifying previously unknown issues that need to be addressed. The Secretary of Agriculture should direct Departmental Administration to work with the mission areas to develop department-level outcome- oriented performance goals and related measures for the business centers, and use them to assess the effectiveness and impact of the business center reforms. (Recommendation 1) We provided a draft of this report to USDA for comment. In an email, a Senior Advisor in USDA’s Office of Operations stated that USDA agreed with our recommendation about assessing the effectiveness and impact of the business centers. In addition, in comments, reproduced in appendix II, USDA generally agreed with the findings in our draft report. USDA stated that to address our recommendation, the department is evaluating options for the development of performance metrics and inclusion of these metrics and related information as part of the regular and recurring reviews by the department’s Deputy Secretary who is identified as the Chief Operating Officer. We are sending copies of this report to the appropriate congressional committees, the Secretary of Agriculture, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions regarding this report, please contact me at (202) 512-3841 or morriss@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Since the U.S. Department of Agriculture (USDA) established the Farm Production and Conservation (FPAC) Business Center in October 2018, Congress has appropriated a total of about $294 million to USDA for necessary expenses of the FPAC Business Center. USDA has also approved $1.1 million for efforts to modernize information technology at the center through fiscal year 2020. USDA has supported the FPAC Business Center with discretionary and mandatory appropriations. USDA budget documents and congressional report language indicate that these appropriations have been accompanied by corresponding reductions in funding to the other three agencies within the FPAC mission area—the Farm Services Agency (FSA), Natural Resources Conservation Service (NRCS), and Risk Management Agency (RMA). For fiscal year 2018, the Consolidated Appropriations Act, 2018, provided discretionary appropriations of about $1.0 million to the FPAC Business Center and further provided for the transfer into the FPAC Business Center account of another $145,000 in mandatory appropriations. Subsequent USDA budget justification documents state that the $145,000 included funds directed towards three NRCS programs—the Environmental Quality Incentives Program (EQIP), Conservation Stewardship Program (CSP), and Agricultural Conservation Easement Program (ACEP). As shown in table 3, for fiscal year 2019, the Consolidated Appropriations Act, 2019, provided for the FPAC Business Center to receive discretionary appropriations of about $216.4 million, an amount that an accompanying conference report states was offset by reductions to the appropriations for administrative functions in FSA, NRCS, and RMA; a transfer of about $16.1 million in discretionary appropriations from FSA’s Agricultural Credit Insurance Fund Program Account; and a transfer of about $60.2 million in mandatory appropriations that, according to USDA officials, came from the same three NRCS programs as in 2018 (EQIP, CSP, and ACEP). According to USDA officials, prior to the establishment of the FPAC Business Center, these funds were used to support the salaries of FSA, NRCS, and RMA personnel performing functions and tasks similar to those provided by the business center and for general operating costs such as rents, information technology, travel, and training expenses. The FPAC Business Center plans its spending and tracks its obligations using standard categories, including personnel compensation, benefits, travel, transportation, postage, contracts, supplies, and equipment. As shown in table 4, the FPAC Business Center planned to spend funds only for personnel compensation and benefits in fiscal year 2018. According to data provided by USDA, the business center obligated about $995,000 of the nearly $1.2 million in available funds, and those obligations were entirely for personnel compensation and benefits. In fiscal year 2019, the business center planned to obligate nearly 74 percent of the $292.7 million in available funds on personnel compensation and benefits, about 18 percent on contracts, about 8 percent on travel, and the rest on other activities. According to USDA officials, through the end of the fiscal year, the business center had obligated approximately $272 million, or about 93 percent, of its available funds. For fiscal years 2018 through 2020, USDA approved an investment of $10 million for information technology modernization across all FPAC agencies, including the following two efforts to modernize information technology in the FPAC Business Center at an estimated cost of $1.1 million: The Modernized Directives System, approved at a cost of $600,000. According to USDA officials, the FPAC Business Center is funding this project from its salaries and expenses budget. According to USDA documents, the business center’s Management Services Division wants to provide all FPAC employees an online tool to create, authorize, disseminate, and manage all of the agency’s policy directives in an FPAC Consolidated Directives Repository while minimizing the costs of operations. According to the agency, the tool would streamline the tasks performed by the division’s administrative staff. FPAC plans to gauge the success of the effort by measuring adoption of the new tool by employees, stakeholders, and the public. The National Office Information System, approved at a cost of $500,000. According to USDA officials, $41,000 of that amount is from the FPAC Business Center’s budget for salaries and expenses, while the remaining $459,000 is funded by the other three FPAC agencies. According to USDA documents, this operations support system would improve the agency’s ability to respond in a timely manner to congressional and departmental inquiries and meet reporting requirements from the Office of Management and Budget and other oversight organizations. According to FPAC Business Center officials, the business center obligated $600,000 and $41,000, respectively, toward these two projects in fiscal year 2019. In addition to the contact named above, Nico Sloss (Assistant Director), Stephen Cleary (Analyst in Charge), Ross Campbell, Caitlin Dardenne, Juan Garay, Scott Heacock, Serena Lo, Cynthia Norris, Lauren Ostrander, and Sara Sullivan made key contributions to this report.", "summary": "With budget authority of $146 billion in fiscal year 2018, USDA employs nearly 100,000 people organized into 13 major staff offices and eight mission areas comprising 18 agencies. In a November 2017 memorandum, the Secretary of Agriculture called for establishment of a business center in each mission area to provide consolidated administrative services. The memorandum identified three policy goals for these reforms: (1) improve customer engagement, (2) maximize efficiency, and (3) improve agency collaboration. The Agriculture Improvement Act of 2018 includes a provision for GAO to report on USDA's business centers. Among other things, this report examines the extent to which USDA has (1) established business centers and (2) assessed the effectiveness and impact of these business centers. GAO reviewed USDA documents and interviewed officials from USDA's Office of the Assistant Secretary for Administration, Office of Budget and Program Analysis, and eight mission areas about their efforts. GAO also interviewed representatives of USDA employee unions and USDA's external customers, such as farmers, for their perspectives on the establishment of the business centers. The U.S. Department of Agriculture (USDA) has established business centers to provide consolidated administrative services such as human resources and information technology in each of its eight mission areas, in keeping with reforms called for in a November 2017 memorandum from the Secretary of Agriculture. The business centers vary in when they were established; three preceded the Secretary's memorandum (see figure). Typically, each business center is located within one of the mission area's component agencies, and the center's leader reports directly to agency leadership. According to a USDA official, the department regularly reviews data on administrative services, including services provided by the business centers. However, the department has not assessed the effectiveness and impact of its business centers and as of November 2019, did not plan to do so. Beginning in 2018, USDA created an online monitoring system to compile data on the status of administrative services, with “dashboards” displaying data specific to different administrative services, among other things. However, the department has not used dashboards or associated metrics to assess the effectiveness and impact of the business centers, including their impact on USDA's customer service; human resources, including hiring; and overall functionality. GAO's prior work has shown that a key practice to consider during an agency's reform efforts is establishing clear outcome-oriented goals and performance measures to assess the reform's effectiveness and impact. Developing appropriate performance goals and systematically assessing the effectiveness and impact of the business center reforms could help the department determine whether the reforms are meeting the Secretary's overarching policy goals and improving the delivery of administrative services to support the department's mission and program goals. GAO recommends that USDA establish department-level outcome-oriented performance goals and related measures for the business centers, and use them to assess the effectiveness and impact of the business center reforms. USDA agreed with the recommendation.", "document_type": "gao"}
{"report": "TSA is the primary federal agency responsible for implementing and overseeing the security of the nation’s civil aviation system and is responsible for ensuring that all passengers and property transported by commercial passenger aircraft to, from, within, or overflying the United States are adequately screened. Specifically, TSA performs, or oversees the performance of, screening operations at about 440 TSA- regulated (i.e., commercial) airports nationwide. These airports range in size from smaller airports (category III and IV airports) to larger airports (categories X, I, and II airports). According to TSA policies and procedures in effect at these airports, all passengers, their accessible property, and their checked baggage are to be screened prior to entering the airport sterile area—the portion of an airport beyond the security screening checkpoint that provides passengers access to boarding aircraft. Among other things, these policies and procedures generally provide that passengers must pass through security checkpoints where their person, identification documents, and accessible property are to be screened by TSOs, and that all checked baggage must be screened by TSOs. Checkpoint Screening. The checkpoint screening process, as set forth in TSA’s procedures, is intended to deter and prevent passengers from carrying any unauthorized or prohibited items into the airport’s sterile area and onboard an aircraft. Upon entering the airport terminal security checkpoint, passengers provide travel document checkers their boarding passes for review. Based on the printed boarding pass result, travel document checkers are to direct passengers to designated areas for standard, enhanced, or expedited screening. Standard screening is generally applied to all passengers with boarding passes that are not marked for enhanced or expedited screening. This screening typically includes passing through either a walk-through metal detector or advanced imaging technology (the latter of which identifies objects or anomalies concealed on the person) and using X-ray equipment to screen the passenger’s accessible property. In the event that any of these screening devices identify a potential item of concern, additional security measures are to result as part of the alarm resolution process. These measures may include pat downs, explosives trace detection searches (which involve a device to detect explosive particles), and colorimetric testing to identify the concentration of certain chemical elements. Enhanced screening is generally required for passengers TSA identifies as high risk, such as passengers that have been matched to federal government lists of known or suspected terrorists. Enhanced screening involves the same procedures applied during a typical standard screening experience, as well as a pat down and an explosives trace detection search or physical search of the interior of the passenger’s accessible property, electronics, and footwear. Expedited screening is allowed for passengers TSA believes to be low risk. One group of passengers who routinely receive expedited screening are those enrolled in TSA’s Pre✓®—a program through which individuals vetted and approved by TSA are eligible for this level of screening. At airports with dedicated TSA Pre✓® lanes, expedited screening includes walk-through metal detector screening and X-ray screening of the passenger’s accessible property, and travelers do not have to remove their belts, shoes, or light outerwear, or remove items such as laptops from carry-on baggage. Checked Baggage Screening. TSA procedures for checked baggage screening establish a process intended to deter, detect, and prevent the transport of any unauthorized explosive, incendiary, or weapon aboard an aircraft. Checked baggage screening generally entails the use of explosives detection systems—which use X-rays and other technology to automatically measure the physical characteristics of objects in baggage and trigger an alarm when objects that exhibit the physical characteristics of explosives are detected. Inspection’s tests are intended to identify vulnerabilities related to any aspect of TSA’s checkpoint and checked baggage screening systems, to include the procedures for screening, the TSOs who implement these procedures, and the technology for screening (e.g., X-ray machines and advanced imaging technology). Security Operations’ testing focuses entirely on TSO performance of existing standard operating procedures for checkpoint and checked baggage screening, and unlike Inspection’s testing, does not test other aspects of screening, such as the performance of screening equipment. To carry out covert testing, both Inspection and Security Operations create test scenarios that describe the overall intent of the test, the threat item, the method of execution (e.g., an explosive device concealed in a shoe carried through the checkpoint), and other pertinent details. Generally, Security Operations’ scenarios have tested TSOs’ performance of procedures pertaining to one of three different paths travelers must follow to have either their persons or property screened (i.e., screening paths): checkpoint on-person—the tester travels through the checkpoint with the threat item concealed on his or her person; checkpoint in-property—the tester travels through the checkpoint with the threat item concealed in a carry-on bag; and checked baggage—the threat item is concealed in checked baggage. For both offices, covert tests begin when program managers notify an airport’s FSD and local law enforcement agency that testing is scheduled to begin. Testers typically pose as passengers and attempt to smuggle a threat object, concealed either on their person or in their property, through one or more layers of the checkpoint or checked baggage screening process (see fig. 1). These layers of screening include the travel document checker and the walk-through metal detector or the advanced imaging technology machine, among others. In general, TSA’s covert tests conclude with a meeting between either Inspection or Security Operations staff and the TSOs and their supervisors who were tested to discuss the results. These meetings, known as post-test reviews, allow officials to reinforce actions resulting in test successes, review the correct procedures for any failures, and collect additional data relating to factors contributing to success and failure. In addition, documented test results are reported to local TSA airport officials, so that they may schedule and track TSO participation in the remedial training that is required by law when screeners fail a test. More broadly, Inspection and Security Operations report test results to certain internal and external stakeholders. Historically, Inspection has reported its test results directly to TSA management to inform executive leadership about the aviation screening system’s potential vulnerabilities to new and evolving threats. In addition, Security Operations has reported test results for its prior testing program to the Office of Management and Budget quarterly and has also briefed TSA senior leadership on results periodically. DHS policy requires that its components, including TSA, use risk information and analysis to inform decision making. A risk-informed approach helps decision makers identify and evaluate potential risks so that actions can be taken to mitigate those risks. DHS defines risk as a calculation of threat, vulnerability, and consequence. These elements are defined as follows: Threat likelihood is estimated based on intent and capability of an adversary. Vulnerability is a physical feature or operational attribute that renders an entity open to exploitation or susceptible to a given hazard. In calculating risk, vulnerability is based on the likelihood that an attack is successful, given that it is attempted. Consequence refers to the negative effect of an event, incident, or occurrence. According to the 2010 DHS Risk Lexicon, which defines key risk- management terms for DHS agencies and components, risk-based decision making uses the assessment of risk as the primary decision driver, while risk-informed decision making may consider other relevant factors in addition to risk assessment information, for decision making. To guide agency efforts to make risk-based decisions, TSA issues annually its Transportation Sector Security Risk Assessment—a report on transportation security that assesses risk by establishing risk scores for various attack scenarios within different transportation sectors, including domestic aviation. These scenarios are continuously refined to reflect evolving threats to the various transportation modes and feedback from subject matter experts. In scoring risk scenarios for the Transportation Sector Security Risk Assessment, TSA considers the three elements of risk (threat likelihood, vulnerability, and consequence). In 2016, Inspection redesigned its process to conduct covert tests more consistently across airports, and began using quantitative methods to design tests and analyze results so that its findings might be applied more broadly across airports nationwide. Inspection officials explained that, prior to redesigning their process, Inspection’s findings could not be applied more broadly because of how tests were designed and executed. In addition, officials noted that some prior test practices risked diminishing the quality of testing. For example, some testers consistently ran tests at the same airports, increasing the likelihood that they might be recognized by TSOs and compromise the covertness of tests. As part of its new testing effort, Inspection recruited a technical team of employees with expertise in statistics and engineering to enhance the design, execution, analysis, and reporting of its covert tests. Inspection also documented its new covert test process and rationales for key program decisions, including its approach to performing quantitative analysis of test results, in overarching guidance issued in October 2016. These documents set forth a framework for conducting tests that includes the creation of detailed scenarios that specify Inspection’s covert test objectives and scope of testing. For example, for one Inspection test scenario conducted in fiscal year 2016, Inspection conducted 280 tests at larger airports to assess whether certain types of assembled explosive devices contained in carry-on luggage could evade detection at the checkpoint. Under new guidance, Inspection’s testers may not conduct tests at the same airport within a predetermined period, to limit the potential of being recognized by airport staff. In addition, under its new process, Inspection selects airports for testing so that it may apply its findings more broadly across airports nationwide. Once Inspection testers complete all tests for a given scenario, Inspection develops classified reports containing results of its quantitative analysis (including detection rates for specific threat items) and suggested actions aimed at addressing any identified vulnerabilities. Inspection uses a risk-informed approach to select locations and scenarios for covert tests, but has not fully documented this approach. According to Inspection officials, to select airport locations for tests, they use a tool to randomly select airports from various regions and of various sizes to ensure appropriate representation. According to our review of the locations Inspection tested in fiscal years 2016 and 2017, Inspection predominantly conducted testing at the larger airports. As previously discussed, this is consistent with a risk-informed approach, as TSA’s analysis has shown that larger airports face an increased threat of a terrorist attack. In addition, Inspection officials said that they use a risk-informed approach to select scenarios for their covert tests that takes into consideration all three aspects of a comprehensive risk assessment— threat, vulnerability, and consequence. According to officials, Inspection’s approach to each of the three components of risk is described below. Efforts to Consider Threats. According to Inspection leadership officials, Inspection has developed close working relationships with key intelligence community agencies to obtain current and specific intelligence information about threats to commercial aviation. Inspection uses this information to create test scenarios involving threat items and attack methods that correspond with the most current threat intelligence. Inspection officials explained that they also consult risk assessments such as the Transportation Sector Security Risk Assessment to help determine which scenarios to test, but do not rely solely on this information. Officials said this is because such assessments can lack specificity about the type and placement of threat items along different screening paths. For example, the Transportation Sector Security Risk Assessment may not convey the specific type of device or the mechanism by which an explosive device will be presented at the checkpoint (e.g., in a laptop). Inspection’s approach, which uses both current intelligence and risk assessments, is consistent with a risk-informed approach, which allows agencies to utilize resources beyond risk assessments to inform decision making. Efforts to Consider Vulnerability. Inspection officials told us they have considered vulnerability as a factor for making risk-informed decisions, and have found that it is not useful when deciding which scenarios to test for two reasons. First, their covert testing is intended to identify the existence of vulnerabilities in the aviation security system. Second, officials explained that vulnerabilities at some airports are well-documented and understood; therefore, they would generally not use their limited resources to test a vulnerability that is well-known. Efforts to Consider Consequence. Inspection officials explained that when selecting among possible scenarios to test, considering the consequences that might result from a scenario is less important than the likelihood of a given threat. However, Inspection officials explained that they require that any scenario tested is one that would result in the loss of life if the attack were actually to occur. Although Inspection program officials could articulate the risk-informed approach used to select scenarios for testing, they had not sufficiently documented this approach. Specifically, we found that Inspection documents its process for making risk-informed selections of scenarios in formal work plans. This documentation includes general criteria that Inspection leadership is to consider when developing threat scenarios, one of which is threat likelihood. However, the work plans we reviewed did not identify selection criteria that address the vulnerability or consequence components of risk. DHS’s Risk Management Fundamentals (2011) requires that agency documentation include transparent assumptions about the rationale behind risk management decisions. In addition, according to Standards for Internal Control in the Federal Government, agencies should document key decisions in a way that is complete and accurate. According to Inspection officials, they have not fully documented their risk-based process for selecting scenarios because their decision making is often informed by unforeseen events associated with the most exigent threats. Nevertheless, without documenting in its work plans how consequence and vulnerability are considered when determining which scenarios to test, current Inspection program managers may not be able to ensure that their scenario selection decisions are appropriately accounting for risk as called for by DHS and TSA guidance. Furthermore, although vulnerability and consequence are less important criteria for Inspection’s current risk-informed selections, documentation of its approach toward each would serve as a baseline for how Inspection makes risk-informed decisions for selecting scenarios to test. This baseline could inform future program managers and agency leadership seeking to make changes. In 2016, Security Operations replaced its Aviation Screening Assessment Program with a new covert test program. Security Operations issued guidance for this new program that, among other things, established a parallel test process carried out by headquarters staff to validate (i.e., determine the quality of) local covert test results from airports. In conjunction with this process, Security Operations also developed and launched a new web-based tool to collect more detailed information on covert tests. According to Security Operations officials, the new program is intended to address problems with its covert testing process identified by an independent contractor in 2015. Specifically, the contractor performed the same covert tests that TSA personnel at local airports conducted, and the contractor’s test results showed that screeners performed more poorly on its tests. In September 2016, we reported that, based on the results of the contractor’s study, TSA had determined that prior-year tests conducted by TSA officials at airports likely showed a higher level of performance than was actually the case. Further, TSA attributed these higher detection rates, in part, to local airport difficulties in successfully maintaining the covert nature of their tests. To address deficiencies identified by the TSA-contracted study, Security Operations issued test guidance in December 2016 and January 2017 that provides more structure to the planning and execution of tests and is intended to help ensure the quality of test results, among other things. For example, the guidance directs local test coordinators to schedule covert tests at varying times of day and varying days of the month, to prevent TSOs from becoming accustomed to testing at particular times. Also, to help ensure that testers are not recognizable by TSOs, the guidance states that airports must not recruit testers from the airport in which the test is to be conducted. Additionally, Security Operations’ guidance expands opportunities for recruiting testers at airports. Security Operations’ new covert test program also features a headquarters-based covert test effort, known as Headquarters Evaluation Team (HET) testing, to help validate the results of covert tests conducted by TSA officials at airports, known as Field Evaluation Team (FET) testing. Under the new process, FET teams, which are composed of TSA staff at airports and locally recruited testers, oversee testing at airports where FSDs are located and at any smaller airports under the FSD’s authority. FET teams perform tests of three different screening paths—checkpoint in-property, checkpoint on-person, and checked baggage—using a variety of scenarios assigned by Security Operations program managers every 6 months. FET teams test scenarios for a designated number of times over the 6-month period, after which, program managers are to select and assign a new set of scenarios for testing for the next 6-month period. For its HET tests, Security Operations is to select, on a quarterly basis, three scenarios to test from among the current set of scenarios assigned for FET testing. HET teams are to travel to airports quarterly to conduct these tests and help validate the FET testing results. Security Operations’ validation process involves comparing detection rates—the percentage of tests in which TSA screening recognized and prohibited a threat item from entering the sterile area of an airport—for similar scenarios from both groups of testers. To assist HET and FET teams in collecting more detailed information from its new test program, in April 2016, Security Operations developed a web-based data collection instrument called the Task Process Factor (TPF) tool that TSA officials use to record more detailed information on covert tests. According to program officials, collecting more detailed information about test failures was part of the agency’s effort to improve screener performance following the DHS Inspector General’s 2015 covert test findings that identified vulnerabilities in TSA’s checkpoint screening. The tool defines the key TSO activities for conducting checkpoint and checked baggage screening as tasks (e.g., interpret the X-ray image). The tool also identifies the various processes associated with a given task (e.g., move property into the X-ray scanner and stop when a full image appears). For any task in which a TSO fails, testers are to use the TPF tool to record the task and process associated with the failure—so that Security Operations may identify points of failure for tests with greater specificity. Furthermore, for all test failures, the tool requires HET and FET testers to identify the factor, or root cause, for failure. Although Security Operations considers some TSA risk information when selecting airport locations to test, we found that Security Operations does not fully consider this information when determining which scenarios to use for its covert tests, and also does not document its rationale for choosing the scenarios it selects. According to its planning documents for conducting HET and FET tests, Security Operations conducts more tests at larger airports than smaller airports. According to TSA officials, this is because larger airports generally have more TSOs who are subject to covert testing. TSA’s decision to allocate more testing resources to larger airports is based on its own risk analysis and, therefore, is consistent with a risk-informed approach. However, Security Operations has not taken steps to incorporate known risks—such as those documented in TSA’s annual Transportation Sector Security Risk Assessment, TSA’s primary risk assessment of threats for all transportation modes—into its process for selecting covert test scenarios. As our prior work has shown, implementing a risk-informed approach involves using risk assessments or other risk information to determine the most pressing security needs and developing strategies to address them. In reviewing TSA’s 2016 Transportation Sector Security Risk Assessment—the version that would have informed Security Operations’ selection of tests for fiscal year 2017—we identified numerous attack scenarios that could have been incorporated into Security Operations’ selection of scenarios to test. Specifically, the 2016 risk assessment included 20 scenarios that involved attacks that could be carried out through expedited screening conducted in dedicated TSA Pre✓® screening lanes. We reviewed all scenarios Security Operations selected to test in fiscal year 2017, but found that only one involved a test of the TSA Pre✓® lane. More generally, we also found that TSA’s selection of threat items to test at the checkpoint in fiscal year 2017 did not reflect threats identified in TSA’s 2016 Transportation Sector Security Risk Assessment. Security Operations officials acknowledged that they do not use formal TSA risk assessments to determine what threat scenarios or items to test. They also do not work with intelligence agencies or review classified information when developing covert test scenarios. Instead, Security Operations officials said they rely mainly on professional judgment regarding which areas of checkpoint and checked baggage procedures TSOs frequently overlook or may not perform correctly (e.g., pat downs). Officials explained that their judgment is informed by monitoring covert test results; unclassified media reports on threats; and requests from agency leadership, such as from TSA’s Administrator. Security Operations’ program managers further explained that because their tests are intended to assess TSO performance of screening procedures and identify any gaps, their selection of scenarios for testing is intended to cover the breadth of checkpoint and checked baggage screening procedures. However, as previously discussed, using a risk-informed approach would allow program managers to balance other goals of testing, such as the need to test a variety of screening procedures, with risk information, when making decisions on what to test. DHS’s Policy for Integrated Risk Management (2010) states that DHS components should use risk information and analysis to inform decision making. Additionally, the TSA Strategy 2018–2026 prioritizes structuring programs to manage risk and optimize resource allocation. Formal risk assessments such as the Transportation Sector Security Risk Assessment identify the most significant risks to checkpoint and checked baggage screening, and accordingly identify some of the most critical skills TSOs need to detect or prevent possible attack scenarios. Using a risk-informed approach to select scenarios that more fully account for known risks—such as those identified in the Transportation Sector Security Risk Assessment or a similar risk assessment—could better ensure that TSA is using its finite testing resources to target screening activities that will counter the most likely threats. Additionally, DHS’s Risk Management Fundamentals (2011) requires that agency documentation include transparent assumptions about the rationale behind risk management decisions. However, Security Operations has not documented its rationales for selecting covert test scenarios in any of its overarching guidance or planning documentation. Such rationales would delineate Security Operations’ framework for determining what screening activities to test, and specify how Security Operations officials balance a risk-informed selection of scenarios with their need to test scenarios that cover the breadth of requirements within existing screening procedures. Security Operations officials said they do not document their scenario selection process because they review covert test data on a frequent enough basis to identify which processes have low detection rates and, thus, are in need of testing. However, documenting a risk-informed rationale for its selection of scenarios would better enable Security Operations or an external party to assess TSA’s covert test programs and ensure that decisions are appropriately accounting for risk as called for by DHS and TSA guidance. It would also allow Security Operations to demonstrate how it balances its goal of promoting a risk-informed culture, as required by DHS, with program goals to ensure that TSOs are following all required screening procedures correctly. Inspection has established a new process and principles for conducting covert tests, as well as collecting and analyzing test data, intended to result in quality information on screening vulnerabilities. We reviewed two reports on results of Inspection’s covert testing that were completed using its new processes, and found they resulted in quality information on screening vulnerabilities. With respect to its new processes Inspection has implemented guidance to ensure a standardized process for developing and executing tests. Specifically, Inspection guidance requires that headquarters staff with expertise in relevant fields (including physical security, explosives, and intelligence analysis) develop all threat items used for testing and conceal these items within test bags or on testers in the same manner across tests. In addition, Inspection program managers require that testers have detailed background stories to explain the purpose(s) of their travel. Inspection now employs multiple standard practices to ensure test covertness. We observed several of these practices during four Inspection tests conducted at one airport. These four tests consisted of two scenarios that were each tested at two different checkpoints within the airport. First, we observed that Inspection teams notified the FSD of their presence only immediately prior to beginning tests, to limit the potential for local airport staff to be forewarned. We also observed that Inspection conducted tests simultaneously across checkpoints, and concluded testing at the airport after an initial round of testing. According to Inspection program managers, conducting tests simultaneously and leaving after the initial round of testing are necessary because once TSOs at a tested checkpoint become aware of testing, there is no reliable way to prevent this knowledge from spreading to other checkpoints. Inspection now integrates its technical operations team (technical team) into all aspects of test design and data collection and analysis. Inspection officials recruited staff with expertise in research and test design, statistics, and systems engineering, among other relevant fields, to analyze this information. Inspection has integrated these staff into all aspects of its test process to ensure the quality of test information collected and analyses performed. For example, according to TSA documentation, Inspection technical team members are to oversee the selection of airports for testing by first conducting an analysis to determine the number of airports to be tested, and then ensuring the selection of airports for testing is made using a random process—a requirement, given that Inspection intends to use test results to understand and describe screening activities at airports nationwide. Inspection now identifies data to be collected for each scenario and monitors this data as it is being collected for quality assurance. According to TSA documentation, Inspection’s technical team develops the data collection forms used to record test information for every scenario. Such data elements are specific to each scenario and can include, for example, the time when the tester entered the checkpoint, whether the TSO running the X-ray machine stopped the belt to review the tester’s bag, and the brand of X-ray machine. According to TSA documentation, the technical team is also to monitor incoming data from scenarios on a regular basis to address any problems as they arise. Inspection now uses guidance to ensure consistency in analysis and reporting. This includes requirements for reviewing all test data and applying rules about which data should be excluded. Inspection also developed guidance to specify the types of statistical analyses that may be used to draw conclusions about test results and how to report on the results to ensure that its analysis of test results is appropriate and transparent. For example, Inspection guidance identifies what technical information should be included in the report to help readers interpret Inspection’s conclusions that are based on statistical analysis of results. We reviewed the two full reports that Inspection issued using this new guidance and found that Inspection generally followed the guidance for using statistical analysis and reporting final results in these reports. As previously discussed, the primary method by which Security Operations tries to ensure that quality covert test results are generated at airports is by having HET and FET testers conduct the same test scenarios at airports, and then comparing detection rates identified by the two teams. Security Operations program managers explained that this method presupposes that test results collected by HET and FET (following Security Operations’ overarching guidance for conducting tests and using the same test scenarios) should produce similar detection rates at the national level. Security Operations program managers further explained that, because HET testers are unaffiliated with the airports they test, they can more easily maintain test covertness. According to program managers, this aspect of HET testing, along with additional training HET testers receive in conducting covert tests, gives them greater assurance that HET tests accurately reflect screener performance at airports. Therefore, program managers generally consider large disparities between HET and FET detection rates to indicate problems with the quality of local airport covert test results. According to our analysis of Security Operations national covert test data for fiscal years 2017 and 2018, checked baggage tests consistently met the Security Operations criterion for quality test results, but checkpoint tests did not. In fiscal year 2018, TSA included a new criterion for quality test results for Regional Director and FSD annual performance evaluations. The criterion requires that HET and FET covert test detection rates at airports under their supervision be within a designated percentage point difference for the three types of tests (checkpoint in- property, checkpoint on-person, and checked baggage). According to our analysis of Security Operations national covert test data for fiscal year 2017 and the first half of fiscal year 2018, checked baggage tests consistently met the criterion for quality test results, however, checkpoint on-person and in-property tests did not. Specifically, we calculated HET and FET detection rates for the three kinds of Security Operations tests (checkpoint on-person, checkpoint in-property, and checked baggage tests) for three 6-month periods from fiscal year 2017 through the first half of fiscal year 2018. We found that, for each 6-month period, HET detection rates for checkpoint tests were lower than FET detection rates, and the differences exceeded TSA’s established criterion for quality test information. Security Operations officials acknowledged the differences between HET and FET rates, but noted that the differences generally decreased from the last 6-month cycle of testing for fiscal year 2017 through the first 6-month cycle of 2018, and program managers are working to address them further. Nevertheless, our analysis showed that for the first half of fiscal year 2018 (the most recent cycle’s data available for our analysis) differences between HET and FET test detection rates for checkpoint on-person and checkpoint in-property remained greater than Security Operations’ criterion for quality test information. In our observations of FET tests, we identified practices in local airport testing that impact the covertness of tests, and thus may contribute to differences between HET and FET detection rates. First, in our observations of local airport FET tests in which TSOs correctly identified the threat items, at one airport the TSA airport official in charge of FET testing was present at the checkpoint, and his presence may have provided advance notice to the TSOs that testing was in progress. Further, we learned from airport testing officials that having the FET test coordinator present at the checkpoint was a routine practice when testing was in progress. At another airport visit, one TSO told us that TSOs often know a FET test is in progress because TSA airport officials use the same test bag to conceal threat items across all tests performed at the airport. According to TSA documentation, potential lapses in the covertness of covert tests, similar to those we observed and were told about, can make TSOs aware that they are being tested and lead to results on tests that overstate actual TSO performance. In addition, we found that the level of potential variability in how TSA airport officials build threat items and test bags for FET tests may affect the quality of the test results used for comparison purposes. Security Operations requires that FET personnel build the threat items, such as explosive devices, that are used for scenarios according to specifications included within TSA headquarters-disseminated scenarios. These scenarios provide a description of the test scenario, a list of materials needed for the threat item, assembly instructions, and directions on how to conceal the threat item within checked or carry-on baggage. TSA provides standard kits to local airports that contain some of the materials FET teams need to build threat items (e.g., an explosive simulant), but TSA staff at the airport must independently procure a number of items needed for each scenario. Given that approximately 80 different teams of FET testers use non-standardized items to build and conceal threat items for tests, the test bags used by teams of FET testers vary to a certain extent across test programs nationwide. According to TSA officials, variations in the construction of test bags (including the simulated explosive devices and test bag assembly) can affect how easy or difficult it is to detect a threat item. The program manager for the HET-FET testing program agreed there is a need for greater assurance of the quality of covert test results, but stated that Security Operations has not taken action on this issue due to resource constraints. However, quality assurance is critical to ensure that the resources TSA has invested in covert testing will yield valid and usable information. Moreover, given its resource constraints, Security Operations’ actions to improve local airport test results could encompass less resource-intensive undertakings, such as providing more standardized items for FET tests or improving guidance to address issues that impact the covertness and consistency of tests. Standards for Internal Control in the Federal Government states that management should use quality information to achieve an entity’s objectives, and that reliable internal sources should provide data that are reasonably free from error and bias and faithfully represent what they purport to represent. By assessing its current FET testing processes— including factors that may compromise the covertness and consistency of tests—Security Operations could identify opportunities to improve the quality of its testing. Further, making changes to its testing process based on its assessment of the current FET testing process could help improve the quality of test results. This, in turn, would better position those who use these results (including agency leadership and TSA airport officials) to reliably identify and address vulnerabilities based on TSO performance. In addition, we found that issues we identified with the quality of FET test results also affect Security Operations’ reporting to external stakeholders. As previously discussed, officials internal and external to TSA use Security Operations test results to assess the effectiveness of TSO performance. Currently, Security Operations reports quarterly FET detection rates as a performance measure to the Office of Management and Budget. The measure identifies the percent of time that TSOs correctly detect threat items at the checkpoint (concealed in carry-on baggage and on the passenger’s body) and within checked baggage. However, as previously discussed, we found that airport testers were not generating quality covert test information on checkpoint screening because their FET detection rates were higher than the HET rates used for comparison, and the difference between the rates exceeded the criterion TSA established for quality covert test information. TSA management officials acknowledged that the agency needs to use more reliable covert test results for measures reported to the Office of Management and Budget. In October 2018, TSA notified the Office of Management Budget that it is in the process of assessing the quality of covert test results it uses to report on TSO performance, and expects to develop new measures by fiscal year 2020. In addition to issues with the overall quality of airport test results, we found that Security Operations faced challenges with the quality of information it collected on the root cause of tests failures. For each test failure, HET and FET testers are to use the TPF tool to identify and record the factor, or root cause, leading to a covert test failure. The TPF tool groups test failure factors into three main categories—(1) failures characterized by the screener’s lack of knowing what is required to effectively accomplish a task or job (a knowledge deficiency); (2) failures caused by incorrectly performing a procedure (a skill deficiency); or (3) failures due to the TSO not assigning the correct level of importance to performing a specific screening procedure (a value deficiency). Although Security Operations has provided some guidance on when to apply a particular factor as a root cause for a covert test failure, this guidance may not be adequate and some testers may not be selecting factors appropriately as a root cause. In our analysis of the factors assigned by both Security Operations HET and FET testers for all covert test failures in fiscal year 2017, we found that testers assigned one factor more than the other two. To assist HET and FET testers in conducting root cause analyses for test failures, Security Operations provides definitions of the three root causes (knowledge, skills, and value). It also requires that all testers (HET or FET) complete three online exercises for using the TPF tool to record results, but the exercises do not provide additional guidance on how to appropriately select root causes. In addition, Security Operations provides in-person training to all HET testers that includes a practice case on selecting from among the factors, and the training course material indicates that the process can be subjective. In our observation of HET tests, we observed numerous failures in which HET testers had to assign a root cause. In a majority of these failures, the tester attributed the same factor as the root cause. HET testers who completed the root cause analyses for these failures all told us they assigned this particular factor by default, once they ruled out the other two causes. Our observations were consistent with a 2017 independent evaluation of the TPF tool performed by the DHS Science and Technology Directorate. Among other things, subject matter experts conducting the 2017 evaluation found that testers they spoke with were not clear on the meaning of the three root causes, and the evaluation recommended that Security Operations provide better guidance to testers on how to select the root cause of a test failure. Security Operations’ program managers concurred with the DHS Science and Technology Directorate’s recommendation that testers need better guidance on how to select among the factors as the root cause for test failures. They also stated they are working on guidance to assist testers in selecting the appropriate root cause for failures. However, in September 2018, program managers told us they had suspended these efforts to address the recommendation as a result of TSA efforts to transfer program operations to Inspection and in anticipation of broader changes to the Security Operations testing program. Inspection officials, who will assume responsibility for HET and FET testing once the transfer of the program to Inspection is complete, stated that they were unsure what changes they would make to Security Operations’ legacy testing process with respect to HET and FET tests at local airports, but stated both types of testing will continue to use their respective legacy testing processes in fiscal year 2019 until final decisions are made. Standards for Internal Control in the Federal Government states that management should use quality information to achieve an entity’s objectives, and that reliable internal sources should provide data that are reasonably free from error and bias and faithfully represent what they purport to represent. As long as Security Operations’ legacy testing process is in use, testers will continue to inconsistently and potentially incorrectly identify the root cause for test failures, and in doing so, will diminish the usefulness of root cause information for addressing TSO performance problems. Reviewing existing guidance and training and providing, where appropriate, additional clarification on applying the factors as a root cause would allow TSA to collect more reliable information on the factors leading to test failures. This, in turn, would better position those who use this information (including agency leadership and TSA airport officials) to address root causes of screener failures at individual airports and across the entire system. Security Operations has not fully documented its methodology for using HET testing as a quality assurance process for FET test results. While Security Operations has documented some aspects of the HET test process, such as training for HET testers on how to conduct tests and post-test reviews with TSOs, we found that Security Operations has not documented its methodology for using HET tests to ensure the quality of FET test results in either its program guidance or other internal documentation. For example, Security Operations has no documentation on how program managers should select airports (e.g., by airport category) and scenarios for HET testing, as well as how they should analyze, compare, and report on HET test results against FET test results. Security Operations officials described some aspects of how they calculate HET and FET test detection rates for comparison purposes, but they did not have a documented methodology for this quality assurance process. For example, Security Operations officials said that they only use data from the largest airports that receive both HET and FET tests (approximately 120 of the about 440 commercial airports) for comparison purposes. Security Operations officials also explained they exclude all HET and FET tests involving enhanced screening from the rates used for comparison purposes because enhanced screening involves a more detailed inspection of the subject that tends to result in the screeners identifying threat items at a higher rate. In addition to these explanations, program managers provided a document explaining Security Operations’ rationale for selecting each of the HET test scenarios used for the last half of fiscal year 2017. While these explanations and the accompanying documentation helped clarify aspects of Security Operations’ process, Security Operations has not developed a policy that provides a comprehensive description (and therefore understanding) of the quality assurance process that its program managers are to use for program planning purposes. Such a policy would describe Security Operations’ approach to selecting HET test scenarios used for ongoing covert testing, how it calculates and compares test results, and how it reports and uses the results. Security Operations program managers agreed that more transparent information regarding the use of HET test results to assess FET test results would be beneficial, but, given that the program was established in late 2016, they acknowledged that they have not had time to document this process. Standards for Internal Control in the Federal Government states that all transactions and other significant events need to be clearly documented, and this documentation should be readily available for examination. The documentation should appear in management directives, administrative policies, or operating manuals. By fully describing its methodology for comparing the results of HET testing with FET test results as a quality assurance process within its program guidance, Security Operations can better ensure that all aspects of this process are clear and available for assessment and validation by third party users of HET and FET test information, such as TSA senior leadership officials. Doing so can also ensure that future program managers for the HET-FET test program can continue to use this quality assurance method appropriately by following the guidance. Inspection submits its covert test findings that it determines to be security vulnerabilities to TSA’s Security Vulnerability Management Process. TSA established this agency-wide process in 2015 to review and address any systemic vulnerability facing TSA (including those related to checkpoint and checked baggage screening). However, it is unclear if vulnerabilities reviewed through this process are being addressed in a timely manner because the process lacks clear timeframes and milestones for mitigation steps, as well as an established method for monitoring the achievement of such timeframes and milestones. In 2015, before establishing the Security Vulnerability Management Process, TSA conducted a review of then-existing processes for evaluating and managing identified vulnerabilities, and found that they were not centralized and did not ensure the level of visibility and accountability needed to adequately mitigate and resolve (or close) the vulnerabilities. Consequently, TSA determined that its processes for tracking and managing the closure of identified security vulnerabilities represented an organizational deficiency that should be addressed. In addition, Inspection officials stated that, under the prior processes, they lacked complete knowledge of all agency resources that could be leveraged to develop mitigation strategies, as well as the necessary authority to compel offices to share these resources, which made it difficult to ensure identified vulnerabilities were addressed. As a result, TSA created the Security Vulnerability Management Process to better ensure the cooperation of various program offices within TSA that had the expertise needed to address vulnerabilities identified by Inspection or other offices within TSA. This process is intended to centralize agency efforts to mitigate vulnerabilities by ensuring that they receive agency- wide visibility and are evaluated, resourced, and managed by appropriate TSA program offices until fully addressed. TSA’s Strategy, Policy Coordination, and Innovation office is responsible for managing and overseeing the Security Vulnerability Management Process, as well as enforcing deadlines for vulnerability mitigation. The Strategy, Policy Coordination, and Innovation office submits vulnerabilities for review by one of two groups of TSA stakeholders—the Executive Risk Steering Committee or the Risk Assessment Integrated Project Team. These two groups are responsible for identifying all TSA program offices affected by the vulnerability in question and working with those program offices to determine whether and how vulnerabilities can be mitigated and formally closed (see fig. 2). According to TSA Strategy, Policy Coordination, and Innovation office officials, to close a given vulnerability, one of the two groups will assess whether the risk posed by the vulnerability aligns to the identified amount of risk that TSA is willing to accept. TSA officials told us that the agency is risk averse to any vulnerability that could cause catastrophic consequences, such as the loss of an airplane. The Strategy, Policy Coordination, and Innovation office has responsibility for enforcing deadlines for mitigating identified vulnerabilities, but our review of TSA documentation found that the office does not establish timeframes and milestones to ensure measured progress toward mitigation of those vulnerabilities. Moreover, we found that although the Security Vulnerability Management Process charter establishes a broad framework for developing and implementing mitigation strategies, it does not establish a method for how the Strategy, Policy Coordination, and Innovation office is to monitor mitigation activities to ensure that TSA program offices are meeting identified timeframes and milestones, such as by identifying a person or entity responsible for escalating cases when these requirements are not being met. Specifically, we found that Inspection has submitted nine vulnerabilities for consideration. With one exception, as of September 2018, none of the vulnerabilities have been formally closed as a result of mitigation steps taken via the vulnerability management process. Under the process, a vulnerability owner has responsibility for developing and leading mitigation efforts for a specific vulnerability. TSA closed one of the nine vulnerabilities 2 years after submission to this process because the relevant program office made policy changes that addressed Inspection’s interim findings. The remaining vulnerabilities have been in progress from 4 months to 2.5 years. Of these eight vulnerabilities, five have had TSA offices assigned as vulnerability owners, and three of these five have mitigation efforts in progress. The three remaining open vulnerabilities that did not yet have vulnerability owners assigned at the time of our review had been waiting for vulnerability owners for a period of 4, 5, and 7 months, respectively; however, TSA officials told us that these three open vulnerabilities had owners assigned in September 2018. TSA officials told us that timeframes for vulnerability mitigation can vary due to the number of stakeholders required to address the situation. They also explained that the complexity of certain threats affect the timeliness of final mitigation solutions (e.g., those requiring technology solutions can involve multiple TSA offices); and before such solutions are developed, Inspection works with program offices to help them develop interim mitigation procedures. Additionally, they cited factors beyond TSA’s control that can delay mitigation efforts, such as changes to agency leadership or in staff within a particular office. For example, mitigation has been delayed for one of the vulnerabilities under review for over 2 years, due to changes in agency leadership in 2016, among other things. In another example, TSA officials told us that mitigation for a vulnerability under review had been delayed for over two years due to personnel changes within the office tasked with developing and leading mitigation efforts. Inspection officials told us that while officials are working on mitigation solutions for identified vulnerabilities, Inspection will assist TSA program offices with implementing interim mitigation procedures before formal mitigation plans are developed. For example, Inspection officials stated that they worked with Security Operations to provide interim guidance to TSA airport officials to address an identified vulnerability that involved Transportation Security Specialists for Explosives using screening equipment incorrectly to clear passengers through the checkpoint. Although TSA has implemented interim mitigation steps for some vulnerabilities while its program offices develop long-term solutions, in some cases Inspection’s findings represent system-wide vulnerabilities to commercial aviation that could result in potentially serious consequences for TSA and the traveling public. For this reason, it is important that TSA make timely progress on formal mitigation solutions. Moreover, tracking progress for a given vulnerability against timeframes and milestones would not necessarily preclude TSA program managers from accounting for complex mitigation efforts. Program managers could, for example, establish longer timeframes at a mitigation effort’s onset and adjust these as needed, should challenges arise. The Standard for Program Management states that the governance of programs includes establishing minimum acceptable criteria for success and the standards by which they are measured and communicated to achieve desired outcomes. Additionally, programs should include the concept of time and incorporate schedules through which specific milestone achievements are measured to ensure that appropriate progress is made toward achieving a defined set of outcomes. In TSA’s case, this would mean the mitigation of identified vulnerabilities. The Standard for Program Management further states that program governance plans are to describe the systems and methods to be used to monitor a given program, and the responsibilities of specific roles for ensuring the timely and effective use of those systems and methods. TSA officials agreed that their vulnerability management process lacks a clear set of deadlines for the timely completion of mitigation steps, as well as a method for monitoring completion of these steps to ensure vulnerabilities are closed. By establishing timeframes and milestones for vulnerability mitigation, TSA would better ensure that progress toward addressing vulnerabilities continues, despite internal challenges, such as personnel changes, or external factors. In addition, by establishing the methods by which TSA’s Strategy, Policy Coordination, and Innovation office will monitor milestones for completion, and the steps it will take when mitigation is not progressing as planned, TSA will be better positioned to ensure that the agency is making measured progress toward addressing the vulnerabilities managed through this process. Security Operations program managers said that they continuously monitor covert test results to identify potential vulnerabilities and to assess progress at airports in addressing vulnerabilities identified through covert tests. Security Operations primarily monitors TSO performance by reviewing information within its TPF tool. Specifically, program officials said that they monitor the database each month to identify gaps between HET and FET detection rates at an individual airport and regional level. Security Operations officials said that they will alert TSA officials at airports if they detect anomalies or large disparities between their HET and FET test rates, and suggest strategies for conducting tests. While reviewing the data, Security Operations officials told us they may also identify specific test scenarios that TSOs are experiencing difficulties with, and sometimes develop strategies to improve performance. For example, officials said that when TSOs demonstrated difficulty with a scenario involving colorimetric testing, Security Operations developed a pamphlet for TSOs to clarify those procedures. Security Operations’ monitoring has also resulted in changes to processes and procedures. For example, according to TSA documentation, in early 2016 Security Operations officials conducted an ad hoc analysis of relevant covert test data. This analysis led to the implementation of Enhanced Accessible Property Screening procedures for personal property screened at airport checkpoints. According to TSA documentation, these new procedures are intended to help TSA officers obtain a clearer X-ray image to enhance screening effectiveness. Among other things, they involve advising passengers to remove organic materials from carry-on bags for X-ray screening, requiring that electronics larger than a cell phone be removed from carry-on bags and placed in bins for X-ray screening, and more targeted property search protocols. In addition to periodic monitoring of test data within the TPF tool’s database, Security Operations officials also told us they monitor Threat Detection Improvement Plans, which are based on recommended actions stemming from each airport’s covert testing results. TSA officials told us that these plans can include test-specific action plans and high-level improvement strategies. Security Operations now monitors airport progress against these plans in order to ensure that airports are taking the necessary actions to improve TSO performance deficiencies identified in covert testing. Security Operations officials told us they use covert test results as the basis for feedback and periodic reporting on TSO performance and the quality of covert test programs or results to headquarters, regional, and local TSA officials and other stakeholders. According to Security Operations officials, this feedback and reporting includes the following. HET reports and feedback: Security Operations directly communicates with TSA officials at airports on HET test performance. For example, in our observations of HET tests at airports, testers conducted an equal number of post-test reviews, during which they reviewed with TSOs and their supervisors the intent and results of the HET tests, reinforced actions resulting in test successes, and reviewed the correct procedures for any failures. In addition to post- test reviews, at the conclusion of each HET test at an airport, Security Operations program managers provide TSA management at the airport a report compiling the results of the recent HET test and statistics on the quality of the covert test program at the airport. According to TSA documentation, these reports include a comparison of local FET test results against the results of HET tests that were conducted during that visit. TPF Report: On a monthly basis, according to TSA documentation, Security Operations also provides a classified spreadsheet report to FSDs that contains a high-level analysis of HET and FET covert test data collected for the fiscal year to date, as well as a copy of the most current test results in the TPF tool’s database. Security Operations program managers stated that allowing airports access to the entire database allows FSDs to compare their airport’s performance against counterparts in other regions and address any areas in which they are lagging. In our interviews with FSDs, we found that officials from all of the airports we spoke with used the TPF data to help manage TSOs. For example five FSDs told us they download the raw test data into local systems for use in their local processes for monitoring TSO performance. Classified monthly conference calls: According to TSA officials, Security Operations hosts monthly classified conference calls with local and regional TSA officials to discuss issues related to covert testing. Security Operations officials told us these discussions typically include the results of specific covert test rounds, methods for using covert tests results, and FSDs’ beneficial practices for carrying out covert testing at their airports. Reporting to senior leadership and other stakeholders: Security Operations officials said they continue to use covert test results for monthly briefings to FSDs and TSA senior leadership. According to TSA documentation, these briefings include high-level analysis of regional covert test performance, as well as overall comparisons of detection rates for on-person, in-property, and checked baggage tests against the national averages. As previously discussed, TSA also uses FET test results as the basis of a performance measure reported quarterly to the Office of Management and Budget. FSDs we spoke with told us they find the feedback and reporting they receive from Security Operations program managers to be helpful. In particular, all 10 FSDs we spoke with told us they find both the HET test reports and accessibility to TPF data in the monthly spreadsheet report to be beneficial and useful. FSDs also noted that the HET reports help inform their assessments on individual and airport workforce performance and efforts to improve their airport’s screening operations overall. While Security Operations program officials perform some high-level analysis of TPF data for periodic reporting, they do not analyze all Security Operations-collected covert test data to identify potential national trends in screener performance that could constitute system-wide vulnerabilities. For example, according to officials and TSA documentation, Security Operations officials use FET and HET covert test data to describe broad trends in screening performance in monthly briefings to TSA management. However, the briefings do not include a breakdown of the different screening tasks and processes that may be most often associated with TSO failures nationally. In addition, although the TPF tool’s database contains information on the task, process, and factors associated with each TSO test failure, Security Operations does not typically include a comprehensive analysis of this information within the monthly covert test reports it provides to TSA leadership at airports. For example, based on our review of Security Operations’ monthly TPF reports, they identify which processes have resulted in the most failures, but do not identify which factors—knowledge, skill, or value—were the root cause of these failures. Moreover, none of this reporting reflects a broader analysis to identify whether failures or causes were associated with a certain size of airport or reflected across one or more regions. Standards for Internal Control in the Federal Government states that an agency should design its information systems to respond to the entity’s objectives and risks. Furthermore, agencies may use information from these systems to evaluate the agency’s performance in achieving key objectives. As discussed previously, Security Operations officials have performed similar types of analysis in the past with positive results. For example, when TSA developed the Enhanced Accessible Property Screening procedures in 2017, these actions were based (in part) on ad hoc analysis Security Operations conducted with national covert test data. At the time, Security Operations’ analysis showed that X-ray operators at checkpoints had problems determining the threat nature of certain categories of objects. This led to repeated failures in detection given the time and cognitive load requirements for interpreting those types of X-ray images. In response, TSA created or adjusted specific procedures based on the analysis of root causes of testing failures and the results of piloting new screening procedures at multiple sites to ensure effectiveness and efficiency could be sustained. Security Operations officials agreed that conducting a more comprehensive, national-level analysis, and utilizing more of the covert test data currently within the TPF tool’s database, would be useful in identifying system-wide vulnerabilities that could inform efforts to improve TSO performance. Security Operations officials told us that at present, they do not have a standard process to comprehensively analyze and report trends in TPF data across all airports. This is because the intent of the current program has been to make test data available to TSA airport and regional officials so they can identify factors affecting screener performance and take actions to remediate and improve any deficiencies. In addition, Security Operations officials cited a lack of resources available to dedicate to this activity, given that headquarters officials have been more focused on revising and improving their current covert test program. However, Security Operations’ TPF tool and database has enabled it to document and communicate detailed information on TSO performance, such as the different screening tasks (e.g., advanced imaging technology operation) and processes (e.g., resolving advanced imaging technology anomalies) where screeners encounter difficulties. Given the breadth of testing conducted and information collected, more comprehensive analysis of TPF data could help TSA identify and communicate important potential trends in the vulnerabilities that TSOs face across all airports. A comprehensive analysis of TSO performance at the national level beyond calculation of overall detection rates would provide Security Operations greater knowledge about the reasons for, and factors associated with, system-wide vulnerabilities due to TSO performance of checkpoint and checked baggage screening, which would better position TSA to address these security gaps. For example, having this information could allow Security Operations to provide more focused training and testing for these functions at the airport level. The information could also position TSA to allocate resources for high-priority issues across all airports. TSA officials at individual airports reported using different tools, techniques, and processes for conducting covert tests and using test data, but Security Operations does not document and disseminate this information. In our discussions with 10 FSDs and their management teams, officials identified a variety of tools, processes, and methods that were developed based on their experiences with covert tests and the resulting actions they took to utilize test data to improve TSO performance. Specifically, 5 of the 10 FSDs we spoke with said their teams developed some type of customized internal databases to aggregate all of their airports’ covert test results, other performance- related data, and any additional Inspection information. FSDs and their staff said such a tool helped present a holistic picture of TSO performance for training and development purposes. Likewise, 5 of the 10 FSDs we spoke with said that they use test results to develop TSO performance baselines and training plans with requirements that exceed TSA’s minimum standards for remediation. Additionally, 5 of 10 FSDs stated that they now include supervisory TSOs and/or TSA leadership officials at airports in remediation discussions with individual TSOs after covert tests take place to provide leadership officials with experience on how best to coach and develop staff. TSA officials we spoke with at airports and at the regional level said that individual airports are often a source for innovation with respect to executing covert tests and using test results, which has at times led to pilot efforts that were adopted at other airports either regionally or nationally. For example, officials from one TSA region told us that they were the first to develop and use performance scorecards (which incorporate covert test results) as an additional tool for improving screener performance. These scorecards were eventually adopted nationwide. Most of the FSDs we spoke with said they communicate with their counterparts at other airports to discuss covert test practices and beneficial methods for using test results at their respective airports. For example, officials from one airport we spoke with reported traveling to an airport in a different region to learn more about the team’s TSO remediation process, which involved using the results of covert testing, Threat Image Projections, and other assessments to create tailored corrective action plans for TSOs. The officials said that this process was an improvement from the one they used previously because it incorporated a greater variety of remediation actions, such as training courses or shadowing opportunities. As discussed previously, Security Operations officials communicate with TSA officials at airports on their covert test programs during a monthly classified call with all FSDs and their teams. This allows Security Operations program managers to provide FSDs with an update on results from recent HET and FET tests, among other things. Security Operations program managers stated that during these calls, they encourage TSA officials not only to discuss particular issues or challenges they have faced with respect to covert testing at their airports, but also to highlight beneficial practices for conducting tests and using test results to improve TSO performance that they and their teams have self-identified and implemented. Therefore, these calls also serve as a forum for FSDs to discuss successful techniques for running covert tests and using test results. In our discussions with 10 FSDs, 8 out of 10 told us they have independently adopted beneficial practices used by other airports. Security Operations program managers are privy to beneficial practices discussed during their teleconferences with local and regional TSA officials, but they told us that they do not regularly document or disseminate this information to TSA officials at airports. Security Operations program managers explained that the call itself is adequate for TSA airport officials to share information, and that local or regional officials can follow up with one another if they want to discuss them further. However, while a monthly conference call may be helpful for informal sharing of practices, it does not capture the breadth of methods or practices used by some TSA airport officials. Moreover, according to headquarters officials, while conference calls provide an opportunity for FSDs to discuss beneficial practices, sharing is ad hoc and the level of detail provided about methods and practices can vary. Systematically documenting and disseminating these practices would provide TSA officials at airports more accurate and complete information about beneficial practices in use at airports nationwide, so that they could be more readily implemented at other airports. The National Infrastructure Protection Plan states that in order to ensure that situational awareness capabilities keep pace with a dynamic and evolving risk environment, officials should improve practices for sharing information and applying the knowledge gained through changes in policy, process, and culture based on shared understanding of efforts to improve security and resilience. This plan also states that documenting and building upon beneficial practices is a key part of information sharing within a critical infrastructure risk management framework. Our interviews with FSDs revealed an array of tools, techniques, and processes for covert testing that TSA officials at airports developed to address local and regional needs. A process to systematically document and disseminate more accurate and complete information on these tools, techniques, and processes that captures the breadth of methods or practices used by some TSA airport officials could help TSA conduct better covert tests and more successfully use test results to improve TSO performance, as well as inform revisions to TSA’s national covert test program. Given the persistent threats to the aviation system, TSA must ensure that its covert testing program operates as effectively as possible to identify and address potential vulnerabilities in the checkpoint and checked baggage screening systems across the nation’s airports. TSA has strengthened the quality and rigor of its covert test programs since 2016, but additional steps are needed to better ensure that TSA targets the areas of highest risk in selecting attack scenarios for testing. Without using a risk-informed approach to selecting screening activities to test, TSA cannot ensure that it is targeting those aspects of TSA screening that pose the greatest known risks. In addition, without documenting its rationales behind how and why certain scenarios are selected for covert testing, TSA cannot demonstrate how its selections reflect identified risks in the aviation environment. New processes for covert testing implemented by Security Operations and Inspection have identified important vulnerabilities in checkpoint and checked baggage screening for fiscal years 2016 and 2017. However, these results can only be useful if they meet internal standards for quality test results. While Inspection’s new process generally produced quality test results on screening vulnerabilities, Security Operations continues to face challenges with the quality of test results collected by TSA staff at local airports. Without taking steps to ensure that Security Operations collects more valid and usable information on vulnerabilities, including the root cause of test failures, TSA will not be positioned to reliably identify and address important security vulnerabilities. In addition, without documenting its methodology for comparing the results of covert tests, TSA cannot ensure that its quality assurance process is consistently applied and transparent. Once vulnerabilities have been identified through covert testing, it is paramount that they are effectively and efficiently mitigated or addressed. Establishing the Security Vulnerability Management Process was a good step toward better tracking the vulnerabilities identified through covert tests and deploying resources to mitigate them, but key identified vulnerabilities have been stalled in the process and none have been closed using this process. This has largely been caused by the absence of timeframes and milestones for achieving mitigation and monitoring key activities in the process. Unless TSA incorporates these aspects into its vulnerability management guidance, it cannot ensure that it is effectively addressing security vulnerabilities that could result in potentially serious consequences for the traveling public. Additionally, while TSA shares some covert test information with TSA officials at airports, more comprehensive analysis of covert test information is needed to enhance TSA’s knowledge about the reasons for, and the factors associated with, TSO performance vulnerabilities that exist system-wide. Furthermore, although TSA officials at individual airports informally share information about beneficial practices they use to conduct covert tests and how they use test information, without systematically documenting and disseminating these practices, TSA cannot ensure that airport officials are fully informed about the different tools, techniques, and processes used by their colleagues. We are making the following nine recommendations to TSA: The Administrator of TSA should document its rationale for key decisions related to its risk-informed approach for selecting covert test scenarios, for both the Security Operations’ and the Inspection’s testing process. (Recommendation 1) The Administrator of TSA should incorporate a more risk-informed approach into Security Operations’ process for selecting the covert test scenarios that are used for tests conducted by TSA officials at airports. (Recommendation 2) The Administrator of TSA should assess the current covert testing process used by TSA officials at airports—including factors that may affect the covertness and consistency of the tests—to identify opportunities to improve the quality of test data, and make changes as appropriate. (Recommendation 3) The Administrator of TSA should assess Security Operations guidance for applying root causes for test failures, and identify opportunities to clarify how they should be applied. (Recommendation 4) The Administrator of TSA should document the methodology for using the results of covert testing conducted by headquarters staff as a quality assurance process for covert testing conducted by TSA officials at airports. (Recommendation 5) The Administrator of TSA should establish timeframes and milestones for key steps in its Security Vulnerability Management Process that are appropriate for the level of effort required to mitigate identified vulnerabilities. (Recommendation 6) The Administrator of TSA should revise existing guidance for the Security Vulnerability Management Process to establish procedures for monitoring vulnerability owners’ progress against timeframes and milestones for vulnerability mitigation, including a defined process for escalating cases when milestones are not met. (Recommendation 7) The Administrator of TSA should develop processes for conducting and reporting to relevant stakeholders a comprehensive analysis of covert test results collected by TSA headquarters officials and TSA officials at airports to identify vulnerabilities in screener performance and common root causes contributing to screener test passes and failures. (Recommendation 8) The Administrator of TSA should develop a standard process for systematically documenting and disseminating to airport Federal Security Directors beneficial practices for conducting covert tests and using test results. (Recommendation 9) We provided a draft of this report to DHS and TSA for review and comment. DHS provided written comments which are reprinted in appendix II. In its comments, DHS concurred with all 9 recommendations and described actions planned to address them. TSA also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. This report addresses the Transportation Security Administration’s (TSA) covert testing for checkpoint and checked baggage screening. More specifically, the report (1) describes how TSA has changed its covert test processes since 2016 and analyzes the extent to which these processes are risk-informed; (2) analyzes the extent to which TSA covert tests for fiscal years 2016 through March 2018 produced quality information; and (3) analyzes the extent to which TSA has used the results of covert tests to address any identified security vulnerabilities. To understand how both the Security Operations and Inspection offices changed their respective covert test processes since 2016, we reviewed agency documentation, interviewed agency officials, and observed 22 Security Operations and 4 Inspection covert tests at 5 different airports. In addition to Inspection testing, our observations included two types of testing overseen by Security Operations—Headquarters Evaluation Team (HET) testing and Field Evaluation Team (FET) testing. To gather information on how covert tests are carried out in different airport environments, we observed tests at four category X and one category I airports. We selected airports for observations on the basis of airport category and screener workforce (private vs. TSA-employed screeners). For all observations, we were able to observe TSOs performing checkpoint or checked baggage screening activities during tests. Following all observations, we observed post-test reviews and, when appropriate, interviewed TSA airport officials, including the Transportation Security Officers (TSO) and private sector screeners (collectively referred to as TSOs in this report) who were tested, about their experience with these tests. To determine the extent to which Security Operations and Inspection testing is risk-informed, we reviewed program documentation and spoke with agency officials. Specifically, we reviewed operational guidance and test scenarios, which describe the overall intent of the test, the threat item, and method of execution (e.g., an explosive device concealed in a shoe carried through the checkpoint) to identify how program officials incorporated the components of risk—threat, vulnerability, and consequence—in their selection of threats and airports to test. We also reviewed the TSA risk assessments that would have been available to Inspection and Security Operations when planning which threats and airports to test for fiscal year 2017, namely TSA’s 2016 Transportation Sector Security Risk Assessment and TSA’s 2012 Current Airports Threat Assessment. The 2016 Transportation Security Sector Risk Assessment contained attack scenarios for the five transportation modes for which TSA is responsible, including domestic and international commercial aviation, as well as other mass transit systems, such highway and mass transit. For our analysis, we used those scenarios relevant to our scope— domestic commercial checkpoint and checked baggage screening. We compared the results of these assessments to the threat items and locations that Security Operations selected for tests in fiscal year 2017 and Inspection selected for tests in fiscal years 2016 and 2017. We evaluated each office’s process for making risk-informed decisions with Department of Homeland Security (DHS) risk management policies, which require that agencies use risk information and analysis to inform decision making, and that risk management methodologies should be transparent and properly documented. To assess the quality of Security Operations data, we reviewed program guidance and interviewed program officials to understand how Security Operations uses HET test results to validate the quality of FET testing at local airports. We also reviewed a 2016 validation study of Security Operations’ test process conducted by the DHS Office of Science and Technology, and spoke with subject matter experts who conducted the study about their findings and recommendations related to improving the quality of test information. We concluded the study’s findings were reasonably sufficient to use as additional support for patterns we also observed during site visits. We were also informed by our HET and FET test observations, which included observations of 19 HET tests at 3 different airports, and 3 FET tests at 1 airport. We supplemented our understanding of how airports conduct FET tests through semi-structured telephone interviews with 10 different Federal Security Directors (FSD) and their staff. To select FSDs for interviews, we identified the airports at which TSA conducted more than the average number of HET covert tests in fiscal year 2017. We focused on the number of HET (as opposed to FET) tests because they are Security Operations’ quality assurance method for airport covert test programs, and we wanted to ensure FSDs had sufficient experience with these tests to provide us perspectives. From this group, we identified the airports with the highest and lowest pass rates for HET tests, and selected among these to reflect variation in several factors, including airport category, difference between HET and FET detection rates, and whether the airport had been tested by Inspection in fiscal years 2016 and 2017. Finally, to assess the quality of Security Operations’ testing, we calculated detection rates for its two types of testing—Headquarters Evaluations Team (HET) tests, in which Security Operations headquarters staff travel to airports to conduct tests, and Field Evaluations Team (FET) tests, which are conducted by staff at local airports. We assessed FET test results against Security Operations’ criterion stating that differences in HET and FET detection rates must be within a designated number of percentage points. We made these comparisons analyzing complete test results for fiscal year 2017 and the first 6 months of fiscal year 2018, over three 6-month periods in order to identify trends. We used for our analysis the12,000 fiscal year 2017 Security Operations TPF records documenting the results of individual covert tests, and an additional 3,600 records from fiscal year 2018. For our analysis, we calculated HET and FET detection rates (i.e., number of items successfully detected) for three screening paths: a checkpoint test with the item concealed on the tester, a checkpoint test with the item concealed in a carry-on bag, and a checked baggage test with the item concealed in the checked bag. In calculating these detection rates, we included only results for scenarios tested within the 18-month period that had both HET and FET tests, and we excluded any test results for scenarios involving enhanced screening. Also, in our calculation of the FET detection rate, we included FET test results for all airports, including those from smaller (category III and IV) airports, which HET teams generally do not visit. We chose to include FET results from all airports in our analysis because it better reflected the overall performance of airports on covert tests. In addition to comparing Security Operations’ quality assurance process against the program’s criteria, we assessed it against federal internal control criteria for documenting processes. To assess the quality of Inspection testing, we reviewed program guidance to identify testing requirements, methods, and limitations. We also observed four different tests conducted at a Category X airport. In addition, we reviewed Inspection guidance to identify and assess requirements for analyzing and reporting covert test results, and reviewed completed reports to identify the extent to which Inspection followed these requirements. We met with Inspection technical experts to discuss Inspection processes for selecting a sample of airports for tests and for analyzing and compiling covert test findings. To assess the extent to which Inspection and Security Operations address security vulnerabilities, we reviewed their efforts separately because each office utilized a different approach. To assess Inspection’s efforts, we focused on its use of the Security Vulnerability Management Process, an agency-wide process that Inspection designated in 2016 as the principal means by which it addresses its identified vulnerabilities. To obtain a more complete understanding of the extent to which this process has addressed Inspection vulnerabilities, we reviewed documentation related to the process (such as its charter) and other information pertaining to all vulnerabilities Inspection has submitted to the process, including those that were unrelated to checkpoint and checked baggage screening (e.g., cargo screening). We analyzed timeframes associated with the vulnerabilities reviewed under the process and the progress made toward closing nine Inspection-identified vulnerabilities. We assessed the vulnerability management process against standards for program management issued by the Project Management Institute, a not-for-profit association that provides global standards for, among other things, project and program management. Given the focus of Security Operations’ testing on screener performance, the vulnerabilities it identified involved TSO failures on tests of specific procedures. To determine how Security Operations headquarters officials address vulnerabilities involving screener performance, we reviewed program documentation, including program guidance and periodic reporting of results, and interviewed program managers. To understand how the results of covert testing are used at the airport level to improve TSO performance and address other identified vulnerabilities, we conducted semi-structured interviews with 10 TSA FSDs stationed at airports across the United States, and with three TSA Regional Directors. We selected the latter based on whether the Regional Director had under his or her direction at least 1 of 10 FSDs we selected for interviews, and to reflect variety in geographic location. We assessed Security Operations’ and TSA officials at airports’ efforts to use covert test results to address vulnerabilities against federal internal control standards and criteria within the National Infrastructure Protection Plan. This is the public version of a classified report that we issued on January 10, 2019. The classified report included an objective related to identifying the results of covert testing for fiscal years 2016 and 2017 and assessing the quality of this test information. DHS deemed covert testing results (including detection rates and identified vulnerabilities) to be classified information, which must be protected from loss, compromise, or inadvertent disclosure. Consequently, this report omits part of an objective identifying the results of covert testing. DHS also deemed some of information in our January report to be sensitive security information. Therefore, this report omits information describing TSA screening procedures, the results of agency risk assessments, and airport-level covert test results. The performance audit upon which this report is based was conducted from September 2017 to January 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient and appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained from this work provides a reasonable basis for our findings and conclusions based on our audit objectives. We worked with DHS from February 2019 through April 2019 to prepare this unclassified, non- sensitive version of the original classified report for public release. This public version was also prepared in accordance with these standards. William Russell (202) 512-8777 or RussellW@gao.gov. In addition to the contact named above, Ellen Wolfe (Assistant Director), Mona Nichols Blake (Analyst in Charge), James Ashley, Chuck Bausell, Jason Blake, Michele Fejfar, Eric Hauswirth, Susan Hsu, Tom Lombardi, Minette Richardson, and Nina Thomas-Diggs made significant contributions to this report.", "summary": "TSA uses covert testing to identify potential vulnerabilities in checkpoint and checked baggage screening systems at U.S. airports. In 2015, TSA identified deficiencies in its covert testing process, and in 2017, the Department of Homeland Security Office of Inspector General's covert testing identified deficiencies in screener performance. Since these findings, TSA has taken steps intended to improve its covert test processes and to use test results to better address vulnerabilities. GAO was asked to review TSA's covert test programs, including how the results are used to address vulnerabilities. This report analyzes the extent to which (1) TSA covert tests are risk-informed, (2) TSA covert tests for fiscal years 2016 through March 2018 produced quality information, and (3) TSA uses covert test results to address any identified security vulnerabilities. GAO observed 26 TSA covert tests, reviewed TSA guidance, analyzed test data for fiscal years 2016, 2017, and through March 2018, and interviewed TSA officials. Two offices within the Transportation Security Administration (TSA) conduct covert tests at U.S. airports—Inspection and Security Operations. The Department of Homeland Security requires that agencies use risk information to make decisions, and TSA issues annual risk assessments of threats that its program offices should consult when making risk-based decisions, such as what covert tests to conduct. Of the two TSA offices that conduct covert tests, Inspection officials used TSA's risk assessment to guide their efforts. However, Security Operations officials relied largely on their professional judgment in making decisions about what scenarios to consider for covert testing. By not using a risk-informed approach, TSA has limited assurance that Security Operations is targeting the most likely threats. Both Inspection and Security Operations have implemented processes to ensure that their covert tests produce quality results. However, GAO found that only Inspection has established a new process that has resulted in quality test results. Specifically, for the two reports Inspection completed for testing conducted in fiscal years 2016 and 2017 using its new process, GAO found that the results were generally consistent with quality analysis and reporting practices. On the other hand, Security Operations has not been able to ensure the quality of its covert test results, and GAO identified a number of factors that could be compromising the quality of these results. Unless TSA assesses the current practices used at airports to conduct tests, and identifies the factors that may be impacting the quality of covert testing conducted by TSA officials at airports, it will have limited assurance about the reliability of the test results it is using to address vulnerabilities. In 2015, TSA established the Security Vulnerability Management Process to leverage agency-wide resources to address systemic vulnerabilities; however, this process has not yet resolved any identified security vulnerabilities. Since 2015, Inspection officials submitted nine security vulnerabilities identified through covert tests for mitigation, and as of September 2018, none had been formally resolved through this process. GAO found that in some cases, it took TSA officials overseeing the process up to 7 months to assign an office responsible to begin mitigation efforts. In part, this is because TSA has not established time frames and milestones for this process or established procedures to ensure milestones are met, in accordance with best practices for program management. Without doing so, TSA cannot ensure efficient and effective progress in addressing security vulnerabilities. This is a public version of a classified report that GAO issued in January 2019. Information that TSA deemed classified or sensitive security information, such as the results of TSA's covert testing and details about TSA's screening procedures, have been omitted. GAO is making nine recommendations, including that TSA use a risk-informed approach for selecting covert test scenarios, take steps to improve the quality of airport covert test results, and establish time frames and milestones for the key steps in its vulnerability management process. TSA concurred with all nine GAO recommendations.", "document_type": "gao"}
{"report": "Of the eight coal mine operator bankruptcies we identified, three resulted in a transfer of estimated benefit liability from the coal operator to the Trust Fund and five did not, according to DOL. Figure 1 shows how many operators were self-insured or commercially insured at the time of bankruptcy, and if responsibility for benefits was shifted from the bankrupt operator to the Trust Fund. Federal law generally requires coal mine operators to secure their black lung benefit liability. A self-insured coal mine operator assumes the financial responsibility for providing black lung benefits to its eligible employees by paying claims as they are incurred. Operators are allowed to self-insure if they meet certain DOL conditions. For instance, operators applying to self-insure must obtain collateral in the form of an indemnity bond, deposit or trust, or letter of credit in an amount deemed necessary and sufficient by DOL to secure their liability. Operators that do not self-insure are generally required to obtain coverage from commercial insurance companies, state workers’ compensation insurance funds, or other entities authorized under state law to insure workers’ compensation. From 2014 through 2016, three self-insured coal mine operator bankruptcies resulted in a transfer of $865 million of benefit liabilities from the coal operators to the Trust Fund, according to DOL estimates (see table 1). DOL estimates for how these bankruptcies will affect the Trust Fund have considerably increased from what DOL had previously reported. In June 2019, we reported that DOL estimated that between $313 million to $325 million in benefit liabilities would transfer to the Trust Fund as a result of these bankruptcies. In January 2020, however, DOL provided updated estimates stating that $865 million in benefit liabilities would transfer to the Trust Fund as a result of these bankruptcies. According to DOL, their estimates increased, among other reasons, to account for higher black lung benefit award rates that occurred from fiscal years 2016 through 2019, and higher medical treatment cost inflation in recent years. Additionally, DOL’s prior estimate for the Patriot Coal (Patriot) bankruptcy did not account for future claims and their effect on the Trust Fund. The amount of collateral DOL required from these operators to self-insure did not fully cover their estimated benefit liabilities. When this occurs, benefit liabilities in excess of the collateral can be transferred to the Trust Fund. For example, the collateral DOL required from Alpha Natural Resources (Alpha) was about $12 million and approximately $494 million of estimated benefit liability transferred to the Trust Fund, according to DOL. The three other self-insured coal mine operator bankruptcies we identified did not affect the Trust Fund. Specifically, Arch Coal, Peabody Energy, and Walter Energy were also self-insured operators, but DOL officials said that their federal black lung benefit liabilities were assumed by a reorganized company or by a purchaser, and therefore did not transfer to the Trust Fund. Insurance contracts or policies to secure operators’ benefit liabilities are required by law to include a provision that insolvency or bankruptcy of an operator does not release the insurer from the obligation to make benefit payments. Additionally, state insurance regulation, insurer underwriting, risk management practices, and state guaranty funds help to protect the Trust Fund from having to assume responsibility for paying black lung benefits on behalf of bankrupt coal operators. Thus, by being commercially insured, the two operators we identified that filed for bankruptcy between 2014 and 2016—Energy Future Holdings and Xinergy Ltd—did not affect the Trust Fund, according to DOL (see fig. 1). Since 2016, several other self-insured operators have also filed for bankruptcy, according to DOL officials, including Cambrian Coal, Cloud Peak Energy, Murray Energy, and Westmoreland Coal. DOL officials said that about $17.4 million in estimated black lung benefit liability will transfer to the Trust Fund as a result of Westmoreland Coal’s bankruptcy. Given the uncertainty of the bankruptcy process in terms of whether liabilities will or will not transfer to the Trust Fund, however, DOL officials said that they could not speculate on how these other bankruptcies may affect the Trust Fund. In overseeing coal mine operator self-insurance in the past, DOL did not estimate future benefit liability when setting collateral or regularly review operators to monitor their changing financial conditions. DOL regulations require that collateral be obtained from operators in an amount deemed necessary and sufficient to secure the payment of the operators’ liability. To determine collateral amounts under the former process, agency procedures stated that an operator’s net worth be assessed by reviewing, among other factors, the operator’s audited financial statements and black lung claims information. The amount of collateral was to be equal to 3, 5, or 10 years of the operator’s annual black lung benefit payments made at the time of a coal operator’s self-insurance application depending on its net worth. Specifically, if net worth was $1 billion or greater, agency procedures set collateral equal to 3 years of benefit payments. If net worth ranged from $500 million to $1 billion, collateral was equal to 5 years of benefit payments. If net worth ranged from $10 million to $500 million, DOL set collateral equal to 10 years of benefit payments. Agency procedures did not permit operators with net worth less than $10 million to self-insure. DOL’s former process for determining collateral did not routinely consider potential future claims for which an operator could be responsible. The agency periodically reauthorized coal operators to self-insure by reviewing an operator’s most recent audited financial statement and claims information, among other things. DOL prepared memos documenting these reviews, and communicated with coal operators about whether their financial circumstances warranted increasing or decreasing their collateral. Regulations state that DOL may adjust the amount of collateral required from self-insured operators when experience or changed conditions so warrant, but regular monitoring of self-insured operators was not conducted. In reviewing the most recent reauthorization memos for each of the self-insured operators, we found that while some of these operators had been reauthorized more recently, others had not been in decades. One operator in particular had not been reauthorized since 1988. There were no written procedures that specified how often reauthorizations should occur after an operator’s initial 18-month reauthorization. DOL has other tools available to mitigate financial losses to the Trust Fund. These include revoking an operator’s ability to self-insure; fining mine operators for operating without insurance; and placing liens on operator assets. Based on our review of agency documentation, however, we found instances when officials did not use these tools to protect the Trust Fund, or were hindered from doing so because of an operator’s ongoing appeal or bankruptcy. James River. In September 2001, DOL required $5 million in additional collateral from James River Coal (James River), which would have increased its collateral from $0.4 million to $5.4 million. Although DOL did not receive the additional collateral, it did not revoke the operator’s authority to self-insure, which is a potential option under agency regulations. Further, DOL had not reauthorized James River at any point from August 2001 until it filed for bankruptcy in April 2014. If James River’s ability to self-insure had been revoked, DOL could have potentially prevented the Trust Fund from being responsible for claims based on a miner’s employment from 2001 through 2016, when James River liquidated. Additionally, if the operator had been unable to obtain commercial insurance, the agency could have potentially fined the operator for each day it operated without insurance. Instead, no action was taken during these years and estimated benefit liability of $141 million was shifted to the Trust Fund, according to DOL. DOL officials stated that they do not have records explaining why James River did not provide the additional collateral or why they did not revoke its authority to self-insure. Patriot. In August 2014, DOL required an additional $65 million in collateral from Patriot, increasing its collateral from $15 million to $80 million. Patriot appealed this decision and, in the 8 months that followed before Patriot filed for bankruptcy in May 2015, DOL did not obtain additional collateral, or revoke Patriot’s ability to self-insure because the appeal was still pending. DOL officials said they would not typically revoke an operator’s authority to self-insure during an ongoing appeal. As a result, DOL was hindered from using this enforcement tool. Liens on operator assets can be an effective tool to protect the Trust Fund if an operator defaults on its benefit liability, but DOL officials said they are hindered from using this tool if an operator files for bankruptcy. DOL can place a lien on a coal operator’s assets under federal law if they refuse the demand to pay the black lung benefit payments for which they are liable. In the event of bankruptcy or insolvency, federal law states that the lien imposed shall be treated in the same manner as a lien for taxes due and owing to the United States under certain laws. However, DOL officials said that operators rarely stop paying benefits until after they file for bankruptcy. Once a bankruptcy occurs, DOL officials said that they are generally prevented by the court from placing a lien and taking an operator’s assets in lieu of payment of current and future benefit liability. Under bankruptcy law, DOL officials said that they have no special status over other creditors with outstanding financial claims. DOL officials said that obtaining sufficient collateral is a better way to protect the Trust Fund. In July 2019, DOL began implementing a new process for coal mine operator self-insurance that may help to address some past deficiencies, if implemented effectively. Among other things, DOL will require operators to periodically submit financial and claims information, including an actuarial estimate of the operator’s current and future benefit liability. DOL plans to use this information to assess the insolvency risk of each operator. Depending on the results of their analysis, DOL plans to categorize the risk-level of each applicant as low, medium, or high. DOL will then set the amount of collateral required to self-insure by linking the operator’s risk category to a corresponding percentage of the operator’s actuarial estimated benefit liability. DOL policies state that they would require a high-risk operator to secure with collateral 90 percent of estimated benefit liability, a medium-risk operator to secure 45 percent, and a low-risk operator to secure 15 percent. However, in February 2020, DOL officials said they plan to revise these percentages to 100 percent, 85 percent, and 70 percent for high-risk, medium-risk, and low-risk operators, respectively. Coal mine operators that are already authorized to self-insure will be required to submit, among other things, an annual renewal application. DOL plans to use this information to update their insolvency risk analysis. If an operator’s risk category changes (e.g., from low-to medium-risk), DOL plans to send a form to the operator requiring an additional amount or type of collateral. Upon receiving the completed form, and proof that the collateral has been obtained, DOL stated that they will notify the operator that its authority to self-insure has been reauthorized. DOL’s new self-insurance process made important changes, but overlooked other key internal control improvements that are needed to protect the financial interests of the Trust Fund. DOL’s new requirements for setting collateral and for the more frequent review of self-insured operators are key components of internal controls, which call for agency management to implement control activities through policy. However, DOL’s new self-insurance procedures do not specify (1) the duration of an operator’s self-insurance authority, (2) the time frames for submitting renewal applications and supporting documentation, and (3) the conditions under which an operator’s self-insurance authority would not be renewed. Our report recommends that DOL implement procedures for coal mine operator self-insurance renewal that clarifies how long an operator is authorized to self-insure; when an operator must submit its renewal application and supporting documentation; and the conditions under which an operator’s self-insurance authority would not be renewed. DOL agreed with this recommendation and stated that it will ensure letters granting or renewing self-insurance authority will inform operators that their authorization expires in one year and that they must submit renewal information three months in advance of the expiration date. DOL staff are hindered from taking enforcement action during an operator’s ongoing appeal, as previously mentioned. DOL policies state that an operator may request reconsideration if its self-insurance application has been denied or if it believes the collateral required by DOL is too high to secure its benefit liability. However, DOL lacks procedures that specify, among other things, the length of time that operators have to submit supporting information. Further, DOL does not specify a goal for how much time DOL appeals decisions should take. For example, in October 2015, DOL recommended revoking Murray Energy’s (Murray) authority to self-insure due to deteriorating financial conditions. Murray appealed this decision, and DOL officials said they postponed responding to the appeal until their new self-insurance process was implemented. However, Murray filed for bankruptcy in October 2019 and DOL had not revoked its authority to self-insure or requested additional collateral because Murray’s appeal was still pending and DOL was still evaluating how much collateral it would require from the operator under its new self- insurance process. Our report recommends that DOL develop and implement procedures for self-insured coal mine operator appeals that identify time lines for self- insured operators to submit documentation supporting their appeals and that identify a goal for how much time DOL should take to make appeals decisions. DOL agreed with this recommendation and stated that they will ensure letters denying self-insurance will inform operators that they have a 30-day appeal period (limited to one extension), and that DOL has set a goal of resolving all appeals within 90 days of the denial letter. We found that DOL does not monitor coal mine operators that do not self- insure and, thus, must commercially insure their federal black lung liabilities to make certain they maintain adequate and continuous coverage as required by law. In the absence of effective DOL monitoring, we evaluated the potential risk that uninsured operators could pose to the Trust Fund. Specifically, in examining the 13 largest coal mine operators that do not self-insure, we found that some insurers erred in reporting black lung policies and in one instance an operator did not have adequate coverage. We found six operators (parent or subsidiary) that were not insured for the entire 3-year period from 2016 through 2018, according to our review of DOL data. When we discussed our findings with DOL, agency officials had to research each operator individually and in some cases contact the operator or their insurer to find out whether or not they had been covered. DOL concluded that these entities were insured. However, the insurers had not properly reported the federal black lung endorsement on new policies or subsequent renewals, in addition to other reporting issues. One of these six operators also had, inadvertently, not maintained adequate commercial coverage for its mining operations in Texas, and had not self-insured those operations. In this instance, the operator obtained an excess loss policy that only pays claims once they exceed a high threshold and, therefore, is not sufficient by itself to secure the payment of the operator’s benefit liability. Designing processes to achieve agency objectives and respond to risks is a principle of effective internal controls. Without a process to monitor operator compliance with program insurance requirements, DOL risks not identifying a lapse or cancellation of operator coverage. This could result in the Trust Fund having to assume responsibility for paying benefits that would otherwise have been paid by an insurer. Our report recommends that DOL should develop and implement a process to monitor operator compliance with commercial insurance requirements and periodically evaluate the effectiveness of this process. DOL agreed with this recommendation and stated that it will modify existing computer systems to identify lapses or cancellations of commercial insurance coverage, and require operators identified as having lapsed or cancelled coverage to obtain or provide proof of coverage within 30 days. Chairwoman Adams, Ranking Member Byrne, and Members of the Subcommittee, this concludes my prepared statement. I would be happy to respond to any questions you may have at this time. If you or your staffs have any questions concerning this testimony, please contact Cindy Brown Barnes at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Alicia Puente Cackley, (Director), Blake Ainsworth (Assistant Director), Patrick Ward (Assistant Director), Justin Dunleavy (Analyst-in-Charge), Alex Galuten, Rosemary Torres Lerma, Olivia Lopez, Scott McNulty, and Almeta Spencer. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In May 2018, GAO reported that the Trust Fund, which pays disability benefits to certain coal miners, faced financial challenges. The Trust Fund has borrowed from the U.S. Treasury's general fund almost every year since 1979 to make needed expenditures. GAO's June 2019 testimony included preliminary observations that coal operator bankruptcies were further straining Trust Fund finances because, in some cases, benefit responsibility was transferred to the Trust Fund. This testimony is based on GAO's report being released today, and describes (1) how coal mine operator bankruptcies have affected the Trust Fund, and (2) how DOL managed coal mine operator insurance to limit financial risk to the Trust Fund. In producing this report, GAO identified coal operators that filed for bankruptcy from 2014 through 2016. GAO analyzed information on commercially-insured and self-insured coal operators, and examined workers' compensation insurance practices in four of the nation's top five coal producing states. GAO also interviewed DOL officials, coal mine operators, and insurance company representatives, among others. Coal mine operator bankruptcies have led to the transfer of about $865 million in estimated benefit responsibility to the federal government's Black Lung Disability Trust Fund (Trust Fund), according to DOL estimates. The Trust Fund pays benefits when no responsible operator is identified, or when the liable operator does not pay. GAO previously testified in June 2019 that it had identified three bankrupt, self-insured operators for which benefit responsibility was transferred to the Trust Fund. Since that time, DOL's estimate of the transferred benefit responsibility has grown—from a prior range of $313 million to $325 million to the more recent $865 million estimate provided to GAO in January 2020. According to DOL, this escalation was due, in part, to recent increases in black lung benefit award rates and higher medical treatment costs, and to an underestimate of one company's (Patriot Coal) future benefit claims. Trust Fund, Filed from 2014 through 2016 DOL's limited oversight of coal mine operator insurance has exposed the Trust Fund to financial risk, though recent changes, if implemented effectively, can help address these risks. In overseeing self-insurance in the past, DOL did not: estimate future benefit liability when setting the amount of collateral required to self-insure; regularly review operators to assess whether the required amount of collateral should change; or always take action to protect the Trust Fund by revoking an operators' ability to self-insure as appropriate. In July 2019, DOL began implementing a new self-insurance process that could help address past deficiencies in estimating collateral and regularly reviewing self-insured operators. However, DOL's new process still lacks procedures for its planned annual renewal of self-insured operators and for resolving coal operator appeals should operators dispute DOL collateral requirements. This could hinder DOL from revoking operators' ability to self-insure should they not comply with DOL requirements. Further, for those operators that do not self-insure, DOL does not monitor them to ensure they maintain adequate and continuous commercial coverage as appropriate. As a result, the Trust Fund may in some instances assume responsibility for paying benefits that otherwise would have been paid by insurers. GAO made three recommendations to DOL to establish procedures for self-insurance renewals and coal operator appeals, and to develop a process to monitor whether commercially-insured operators maintain adequate and continuous coverage. DOL agreed with these recommendations.", "document_type": "gao"}
{"report": "State and local governments rely on a range of revenue sources to support their activities, including federal grants, user charges, and taxes. The share of revenue generated from different types of state and local taxes and user charges—also referred to as own-source revenue—varies by state or local government. State and local governments face fiscal pressures when, taken as a whole, spending exceeds revenues. Fiscal pressures may reflect growth in selected expenditure categories without corresponding revenue growth or other spending reductions. To alleviate fiscal pressures and comply with balanced budget requirements, state and local governments may seek to reduce spending, increase revenues, or both. For example, state and local governments may offset increased costs in one program by making cuts to other programs where they have more flexibility to adjust certain types of spending. Alternatively, if their ability to adjust spending is limited, they may seek additional revenue by increasing existing taxes or user charges or imposing new ones. For example, some programs may have spending that is defined or required in state law and must be funded annually, regardless of broader economic circumstances. Other spending may not be subject to legal or other requirements and is thus subject to decisions influenced by current fiscal pressures. Changes in the makeup of state and local government services, spending, and revenues may reflect economic or demographic changes, a change in spending priorities, or changes in federal policy. Fiscal pressures can result from spending growth or revenue declines that are not the direct result of current state and local policy choices. These choices may instead reflect automatic spending growth (for example, in response to population shifts or an increase in the number of people eligible for government programs) or declines in revenue due to changes in the economy (for example, a shift from goods to services without a corresponding shift in the tax base). Individual expenditure categories can also face fiscal pressures. For example, employee pension funds can experience investment returns below the rates of return assumed in budget forecasts, which can then become underfunded liabilities. From 1998 to 2018, state and local government expenditures increased from about $1.7 trillion in 1998 to about $2.8 trillion in 2018. Figure 1 shows that most state and local government expenditure categories experienced slight shifts during this period. While some categories declined as a share of total spending, inflation-adjusted spending increased in all expenditure categories. Health expenditures reflected the largest increase in inflation-adjusted spending, increasing from $288 billion in 1998 to $670 billion in 2018. As a share of total expenditures, health spending increased by 7 percentage points, from 17 percent in 1998 to 24 percent in 2018. Inflation-adjusted spending on education—the largest share of state and local expenditures—increased by more than $300 billion from 1998 to 2018. However, as a share of total spending, education expenditures decreased by 2 percentage points during the period, in large part, because of the sizable growth in health expenditures during this time period. From 1998 to 2018, state and local government revenues increased from about $1.6 trillion in 1998 to about $2.6 trillion in 2018 (see figure 2). In every year between 1998 and 2018, state and local government taxes (i.e., personal income, sales, excise, property, corporate, and other taxes) comprised the largest category of receipts for the sector, providing about $1.8 trillion or 69 percent of total revenues in 2018. With the exception of interest receipts, all revenue categories increased in inflation-adjusted dollars from 1998 to 2018. Interest receipts decreased from $108 billion or 7 percent of total revenues in 1998 to $72 billion or 3 percent of total revenues in 2018. Federal grants comprised the second largest category of state and local government revenues in both 1998 and 2018 (see figure 2). As a share of total revenues, federal grants increased from $288 billion or 17 percent of total revenues in 1998 to $569 billion or 22 percent in 2018, an increase of $281 billion or 5 percentage points. Figure 3 provides a more detailed breakdown of federal grants to state and local governments from 1998 to 2018. Compared to other grant categories, health grants reflected the only increase in state and local government federal grants, increasing from 53 percent in 1998 to 70 percent in 2018. Most of this growth occurred after 2010, following the enactment of the Patient Protection and Affordable Care Act (PPACA), which offered federal Medicaid funding for states choosing to expand their programs to low-income adults. As a share of total federal grants, income security grants reflected the largest decrease—from 26 percent in 1998 to 17 percent in 2018. However, income security grants increased in inflation-adjusted dollars, from $75 billion in 1998 to $96 billion in 2018. The decline in income security grants, as a share of total federal grants, reflects shifts in federal grants to state and local governments resulting from faster growth in health grants during the 20-year time period. In most states, growth in both state and local government expenditures and revenues exceeded growth in state gross domestic product (GDP) from 1997 to 2017. As shown in table 1, growth in expenditures equaled or exceeded growth in state GDP in each of the 5-year periods from 1997 to 2017. Revenues grew faster than state GDP, on average, during the 20-year period, though they grew somewhat slower than state GDP from 2008 to 2012. Table 1 also shows that state and local government expenditures, revenues, and state GDP all experienced more robust growth during the first half of the 20-year period (1997 to 2007) than in the second half of the period (2008 to 2017). On average, growth in state and local government expenditures outpaced growth in state and local government revenues by about 0.3 percentage points per year. As shown in figure 4, expenditures grew faster than revenues in 43 states from 1997 to 2017. We have previously reported on state and local government expenditure growth trending in excess of revenue growth and its implications for increasing state and local government fiscal pressures. For example, our most recent simulations suggest that the state and local government sector could continue to face a gap between expenditures and revenues during the next 50 years. Because many state and local governments are required to balance their operating budgets, they will most likely need to make policy changes involving some combination of reduced spending and increased revenue. Spending in most expenditure categories grew faster than or at the same rate as state GDP in a majority of states from 1997 to 2017 (see table 2). State and local government expenditures, as a whole, grew at an average annual rate of 2.8 percent from 1997 to 2017 and faster than state GDP in 43 states. Public welfare spending showed the fastest growth among all state and local government expenditure categories, growing at an average annual rate of 4.9 percent per year during the period. Public welfare. Public welfare—which includes Medicaid and welfare programs, such as Temporary Assistance to Needy Families—grew faster than all other spending categories from 1997 to 2017. Public welfare grew faster than state GDP in all but two states at an average annual rate of 4.9 percent during the period. The Centers for Medicare & Medicaid Services (CMS) Office of the Actuary projected that Medicaid spending would grow at an average rate of 5.7 percent per year, from fiscal years 2017 to 2026, with projected Medicaid expenditures reaching more than $1 trillion by fiscal year 2026. Since Medicaid is a matching formula grant program, the projected growth rate reflects expected increased Medicaid expenditures that will be shared by state governments. Furthermore, our long-term simulations of the state and local government sector’s fiscal outlook have shown that health expenditures are expected to continue to increase faster than the economy during the next 50 years. Hospitals and health. Expenditures on hospitals and health—which include state and local government spending on public health and hospitals, but not Medicaid—grew at an average rate of 2.6 percent per year from 1997 to 2017. Across all states, average annual growth in spending on hospitals and health ranged from -2.8 percent per year to 7.8 percent per year, reflecting the largest spread of any spending category. Further, growth in spending on hospitals and health was not distributed evenly across this range. In eight states, hospital and health expenditures grew at an average annual rate of less than 1 percent, while the average annual growth rate exceeded 3 percent in 20 states. Education services. Spending on education services (i.e., schools, colleges, other educational institutions, educational programs for adults, veterans, and other special classes) grew at an average rate of 2.6 percent per year and faster than state GDP in 36 states from 1997 to 2017. This average annual growth rate reflects faster growth of 4.1 percent per year, on average, from 1997 to 2007 and slower growth of 0.7 percent per year, on average, from 2008 to 2017. During the second half of the 20-year period, from 2008 to 2017, spending on education services grew more slowly than state GDP in 39 states. Public safety. Spending on public safety, which includes state and local government services, such as police, fire protection, and corrections, grew in all states at an average rate of 2.5 percent per year from 1997 to 2017. In 34 states, public safety spending grew faster than state GDP during the period. Further, public safety expenditures grew faster than 3 percent in 13 states and slower than 1 percent in three states during the same period. Transportation. Spending on transportation grew at an average annual rate between -1.4 percent and 7.2 percent from 1997 to 2017. In 35 states, transportation spending grew between 1 percent and 3 percent per year, on average, during this period. Transportation spending grew slower than 1 percent per year on average in seven states, while in nine states, transportation spending grew faster than 3 percent, on average, per year. Environment and housing. Expenditures on environment and housing, which include functions related to natural resources and housing and community development programs, grew, on average, at a rate equal to state GDP from 1997 to 2017 and ranged from a low of 0.3 percent to a high of 6.4 percent. Environment and housing spending exceeded state GDP growth in 24 states, while these expenditures grew more slowly than state GDP in 27 states. From 1997 to 2007, environment and housing spending grew at an average rate of 4.3 percent per year. From 2008 to 2017, this spending category grew at an average annual rate of .03 percent. Government administration. Government administration includes functions related to managing the government’s day-to-day work, such as financial administration, judicial and legal costs, and central staff services and personnel agencies. Spending in this category grew slightly slower than state GDP at an average rate of 2.1 percent per year from 1997 to 2017. Government administration spending grew faster from 1997 to 2007 (at an average rate of 3.6 percent per year) than from 2008 to 2017 (at an average rate of 0.4 percent per year). Other selected expenditures. Interest on debt spending (i.e., all spending on borrowed money except utility debt) grew slower than state GDP in 48 states, while annual growth ranged from -5.1 to 2.5 percent across states. From 2008 to 2017, spending on debt interest decreased by an average annual rate of 2.1 percent from 2008 to 2017. Insurance benefits and repayment expenditures, which include retirement benefits, was the fastest growing category of selected expenditures. Average annual growth in interest paid to finance debt equaled -0.1 percent. Salaries and wages for state and local government employees grew slower than state GDP in 46 states and slower than 1 percent per year in seven states. General revenues, as a whole, grew faster than state GDP in 35 states from 1997 to 2017 with the fastest growth in federal grants (3.5 percent per year) and user charges (3.1 percent per year). Table 3 shows state and local government revenue broken down into two larger categories: (1) federal grants, which include all federal fiscal aid to state and local governments; and (2) own-source revenue, which includes all general revenue state and local governments generate from their own sources, such as taxes and user charges. In the following section, we discuss trends in selected revenue categories identified in table 3. These selected revenue categories—federal grants, user charges, and property taxes—represent the three largest categories of revenue for the state and local government sector. Federal grants. Federal grants were the fastest growing source of revenue for the sector from 1997 to 2017, growing in every state and faster than state GDP in 45 states at an average annual rate of 3.5 percent. During the same period, state and local governments’ own- source revenue (i.e., taxes and user charges) grew at an average rate of 2.2 percent per year and ranged from -1.7 percent to 3.9 percent per year. However, state and local governments’ own-source revenue grew faster than state GDP in about half of the states. At the same time, this revenue growth varied among grant categories and across states. User charges. State and local government user charges comprised the second fastest growing revenue category for the sector from 1997 to 2017. User charges grew faster than state GDP in 40 states, at an average rate of 3.1 percent per year. In addition, user charges grew in every state, at an average rate between 0.9 percent and 6.2 percent per year. Total taxes. State and local government taxes, the largest category of own-source revenue, grew slower than state GDP from 1997 to 2017. Specifically, state and local government total tax revenues grew at a rate of about 2.1 percent per year, on average. As shown below, for the three major tax categories—property, sales, and individual income—growth varied overall and across states. Property taxes. Property taxes were the fastest growing category for the sector—growing in nearly all states at an average rate of 2.6 percent per year from 1997 to 2017. Property taxes grew faster than state GDP in 36 states and faster than 3 percent per year in 17 states. Property taxes drove own-source revenue growth during this time period. Compared to other tax revenue categories, property taxes have been a relatively stable revenue source for local governments. In addition, property taxes grew at an average rate of 1.4 percent per year from 2008 to 2012, while both sales and income taxes showed negative growth during the period. Sales taxes. Sales taxes grew at an average rate of 2 percent per year from 1997 to 2017, ranging from a low of -0.6 percent to a high of 4.1 percent. Revenue from sales taxes grew slower than state GDP in 28 states and slower than 1 percent per year in six states. Slower sales tax growth could reflect a shrinking sales tax base for state and local governments. Many states do not levy a tax on services—which represents more than two-thirds of all consumption. These states must therefore raise sales tax revenue from a smaller base. Individual income taxes. From 1997 to 2017, growth in individual income taxes showed greater variation across states and over time than either property or sales taxes. Similar to the growth in sales taxes, individual income taxes grew at an average rate of 2 percent per year, but reflected a wider range of growth from 1997 to 2017. Individual income taxes grew slower than state GDP in 26 states and slower than 1 percent per year in six states. From 2008 to 2017, growth in individual income taxes slowed to an average rate of 0.3 percent per year—representing a more than 3-percentage- point slower growth rate compared to the period from 1997 to 2007. Table 4 shows that public welfare grants to state and local governments—which include Medicaid—grew faster than state GDP in 47 states. Public welfare grants grew faster than 3 percent per year in 45 states from 1997 to 2017. During this period, public welfare grants grew in all states at an average rate of 4.6 percent per year, ranging from 1.8 percent to 9.5 percent per year. Grant funding for education and highways grew faster than state GDP at an average annual rate of 2.6 percent and 2.4 percent, respectively. Although a relatively small share of federal grants, natural resources grants had the largest average annual growth rate—4.9 percent—and grew faster than state GDP in 37 states from 1997 to 2017. Federal grants grew faster than own-source revenue overall and in a majority of states from 1997 to 2017. Figure 5 compares the rate of growth in own-source revenue to the rate of growth in federal grant revenue during the period. Figure 5 shows that, for the majority of states, revenue from federal grants grew faster than own-source revenue. State rainy day fund balances fluctuated as a median percentage of general fund expenditures from 1998 to 2018 and experienced consistent increases since 2010. Rainy day funds include state budget stabilization or reserve funds that state governments may use to supplement general fund spending during a revenue downturn or other unanticipated shortfall. Every state has some type of rainy day fund, though deposit and withdrawal rules vary considerably. Robust rainy day fund balances alone do not necessarily indicate strong fiscal positions, but they are one of the primary mechanisms available to states to offset a budget gap, along with spending reductions or tax increases. However, these funds will not necessarily relieve longer-term structural fiscal pressures. Median state rainy day fund balances as a percentage of total general fund expenditures increased to their highest level in the last 20 years in 2018. Figure 6 shows that states’ median rainy day fund balances increased from 1.6 percent of general fund expenditures in 2010 to 6.4 percent in 2018. Further, the median balance of state rainy day funds declined significantly after each of the last two recessions, while states gradually restored their balances each time. From 2016 to 2018, the majority of states maintained rainy day fund balances in excess of 5 percent of their general fund expenditures. The number of states with rainy day fund balances that exceeded 5 percent of their general fund expenditures doubled from 1998 to 2018, from 16 states in 1998 to 32 states in 2018 (see figure 7). Specifically, nearly half of the states maintained rainy day fund balances greater than 5 percent and less than 10 percent of their general fund expenditures in 2018. Six states had rainy day fund balances equal to 1 percent or less of their general fund expenditures, down from 11 states in 1998. Experts we interviewed identified a range of federal policies and other considerations that could affect the fiscal condition of state and local governments. While there are other issues that affect the state and local sector’s fiscal condition, this section focuses on the issues that emerged most frequently during the interviews related to the effects of federal policies on the sector’s fiscal condition, and the fiscal pressures facing states and localities that could require a federal policy response to ensure effective delivery of federal programs implemented by these governments. Those issues include: health care, federal budget uncertainty, physical infrastructure, tax policy, and natural disasters. Health care. Most experts agreed that health care costs and, in particular, Medicaid, have placed fiscal stress on state and local governments. A number of experts expressed concerns about the long- term sustainability of Medicaid and the states’ ability to meet future demand, given current demographic trends and expectations for escalating enrollment. As we discussed earlier, Medicaid has been the fastest growing category of state spending and, based on our simulations, is expected to rise faster than GDP during the next 50 years. Some experts noted that growth in Medicaid affects states’ fiscal conditions as it has become a larger portion of states’ budgets. They pointed out that even though states have experienced a recent leveling off in Medicaid enrollment, states have also experienced a faster rate of growth in spending. Two experts attributed this growth largely to the aged and disabled enrollment groups that account for a larger share of program spending. A number of experts said that states that expanded their Medicaid programs have seen the largest increases in enrollment—driven by adults who are newly eligible for the program. CMS’s Office of the Actuary projected that Medicaid enrollment is expected to grow by as many as 13.3 million newly eligible adults by 2026—as additional states may expand their Medicaid programs to cover certain low-income adults under the Patient Protection and Affordable Care Act (PPACA). The Congressional Budget Office also reported that Medicaid spending increased 36 percent from fiscal years 2015 to 2019, largely because of state Medicaid expansions. As of January 2020, 36 states and the District of Columbia expanded eligibility for their Medicaid programs under PPACA. Some experts noted that, while enrollment has grown for the expansion states, the federal government bears responsibility for a large portion of the costs. Specifically, the federal government reimbursed 100 percent of the costs of the expanded population beginning in 2014. The federal reimbursement then decreased to 94 percent in 2018, and to 90 percent in 2020. One expert told us that states had the benefit of anticipating the decrease in funding and the corresponding increase in the state share of the costs. At the same time, a number of experts generally agreed that states are not financially positioned to meet the future demands of Medicaid during a recession or economic downturn, given projected increases in enrollment. In particular, experts pointed to the costs of recession-related Medicaid enrollment increases and the resulting fiscal pressures this would place on federal and state governments to fund Medicaid obligations. One expert shared concerns related to the uncertainty of federal funding should a recession occur. Two experts also pointed to the pressures local governments, and more specifically, county governments, face from implementation of certain federal health care policies. Specifically, these experts pointed to the health care costs that county governments must incur as a result of local jails housing pretrial inmates who have medical needs and require treatment. Federal law prohibits the use of federal health benefits by inmates who are pending trial. Thus, to the extent that an inmate cannot afford to pay the costs of health care services, counties must assume the related health care expenses for providing the necessary treatment for the inmate without reimbursement for those expenses. Federal budget uncertainty. A number of experts told us that states continue to grapple with uncertainty stemming from unpredictability in the amount of federal assistance and timing of federal appropriations— including continuing resolutions and federal government shutdowns—and effects on states’ ability to plan and implement programs. Some experts raised concerns related to the federal government’s current fiscal condition and the potential effects on state and local governments. Specifically, experts noted that states are aware of the federal government’s current fiscal condition—including federal debt and deficit levels—and the level of support the federal government may or may not choose to provide in the event of an economic downturn or recession, as it has during past recessions. In light of the uncertainty, some states have engaged in “stress tests” of their own budgets using various revenue and expenditure scenarios to determine if they are in sufficient fiscal health to weather a mild-to-severe recession. Moody’s Analytics reported in 2019 that, based on the results of stress tests it performed on all fifty states, 28 states have the level of cash reserves necessary to manage a moderate recession without having to raise taxes or cut spending. Some experts further noted that state and local governments that have not been able to strengthen their cash reserves could undergo more severe fiscal stress in an economic downturn and require a greater level of assistance. Some experts also raised concerns related to the effects of federal government shutdowns and continuing resolutions on state and local governments and their ability to plan for and implement federally-funded programs. In all but 4 of the last 42 years, Congress has passed continuing resolutions to keep government services in operation until an agreement is reached on final appropriations bills. In some years, when new appropriations or a continuing resolution have not been enacted on time, this lapse in appropriations—or funding gap—caused the government to partially shut down, which halted some activities at federal agencies until appropriations were passed. A number of experts told us that interruptions in appropriations and subsequent delays in federal grants caused by shutdowns, for example, may require states to spend additional unbudgeted funds to ensure continuity of services in certain federally-funded programs, such as food and nutrition and transportation. According to one expert, not all state or local governments are in a position to access those funds in a timely manner. Furthermore, one expert noted the impacts of continuing resolutions on local governments by compressing the time available for federal grant applications. As a result, some applicants (e.g., cities or other localities) may not apply or miss deadlines for certain federal grant programs. We and others have reported on the effects of government shutdowns and its impact on some states. For example, we reported on the partial shutdown of the federal government in October 2013, which lasted for a period of 16 days due to a lapse in appropriations. Our report showed that even if a state wanted to use its funds to continue services for a federally-funded program, it might not have had sufficient liquid assets to do so quickly. At that time, at least 12 states publicly reported that funding for certain grant programs was only confirmed through October, meaning the funding may not have been available if the shutdown had continued into November. Some of these states expected to discontinue certain federally-funded programs or services if the shutdown had extended into November, while others expressed uncertainty regarding how they would have proceeded if the shutdown had been longer. Physical infrastructure. Physical infrastructure at the state and local government level includes a broad range of systems—including highways, mass transit, rail, water, and sewer systems. A number of experts pointed to concerns related to an aging infrastructure and the fiscal pressures that infrastructure demands place on state and local governments. The cost of repairing and upgrading the nation’s surface transportation infrastructure to meet current and future demands is estimated in the hundreds of billions of dollars. Further, our 2017 report noted that estimates from the Environmental Protection Agency put drinking water and wastewater infrastructure needs at approximately $655 billion nationwide during the next 20 years. State and local governments own a large portion of the nation’s physical infrastructure, while the federal government provides support to the sector in the form of grants, bonds, and loans. Funds made available from the Highway Trust Fund are distributed to states in the form of grants for eligible projects. The federal government also supports additional infrastructure spending through tax-exempt or tax-credit bonds, which provide a tax exclusion or tax credit to owners of municipal bonds issued by state and local governments. Further, through various loan programs, such as the Transportation Infrastructure Finance and Innovation Act program, the government supports project financing. State and local governments also generate revenues for transportation projects through their own sources including user fees and taxes. A number of experts shared concerns about the future of federal funding for state and local surface transportation needs. One expert acknowledged the benefits of highway grant programs provided through the Fixing America’s Surface Transportation Act. However, this expert also recognized that the act is set to expire in 2020 and its future, along with other sources of federal funding through the Highway Trust Fund, remains uncertain. We have also reported that traditional federal funding sources for surface transportation, such as the Highway Trust Fund, are eroding and the federal government lacks a long-term sustainable strategy for funding surface transportation. Moreover, experts noted that physical infrastructure needs represent only one among many competing priorities for state and local government spending. One expert expressed concern that the availability of state and local discretionary spending on infrastructure needs and maintenance will increasingly be affected by growing pressures from other mandatory spending categories, such as Medicaid. Many states have looked to modify or enhance other sources of revenue, such as the gas tax, to help meet highway transportation costs. According to the National Conference of State Legislatures (NCSL), since 2013, 31 states and the District of Columbia have enacted legislation that will or may increase their motor fuel tax to support surface transportation costs. Even so, two experts raised concerns about the viability of the gas tax as a reliable revenue source since gasoline consumption has declined. Further, NCSL reported that many states have received federal funding to study and pilot user-based alternative mechanisms through the Surface Transportation System Funding Alternative Program. We and others have also reported that some states have recognized the need for an alternative funding mechanism to meet future revenue demands. Some options that states have considered or implemented include tying gas tax rates to inflation or population, taxes based on the price of fuel, and taxing miles traveled instead of gas purchased—also referred to as mileage-based user fees. Further, experts pointed to the lack of a clearly articulated federal highway infrastructure policy and the implications for state and local governments. For example, one expert noted that states need the ability to plan multiyear programs for large-scale transportation projects and that an open dialogue about federal program implementation or renewal with all three levels of government could help state and local governments better plan for the future. This expert added that the uncertainty that state and local governments experience could be reduced if the federal government could better inform and communicate with state and local governments regarding legislative policy developments and was willing to engage in conversations with state and local governments. Tax policy. Experts discussed selected provisions of the law commonly known as the Tax Cuts and Jobs Act (TCJA) and other tax-related issues that could exacerbate or help ease fiscal pressures for state and local governments. Enacted in December 2017, TCJA included significant changes to corporate and individual tax law, with implications for state and local government tax collections. In particular, for individual taxpayers, for tax years 2018 through 2025, tax rates were lowered for nearly all income levels, some deductions from taxable income were changed (personal exemptions were eliminated, while the standard deduction was increased), and certain credits, such as the child tax credit, were expanded. A number of experts agreed that with just over 2 years since its passage, it is still too early to fully assess the effect of TCJA’s provisions on state and local government revenues. States are continuing to incorporate some of the provisions of TCJA into their own tax codes. Some states have adopted the federal definition of taxable income as a starting point for state tax calculations, while other states use the federal definition of adjusted gross income as a starting point. The choices states make regarding their linkage to these definitions have implications for their state tax revenues. Further, because TCJA placed a $10,000 annual cap on the federal deduction for taxpayers’ state and local taxes (SALT) from taxable income beginning on January 1, 2018, some high-income taxpayers prepaid their personal income and property taxes to take advantage of the uncapped SALT deduction in 2017. As a result, some states experienced an increase in revenues in late 2017. According to S&P Global Ratings, the imposition of SALT caps incentivized many taxpayers to accelerate their income tax payments into December 2017, but consequently made December 2018 tax payments look smaller by comparison. It also further reduced December 2018 payments by lessening the incentive for many taxpayers to make early income tax payments. Most experts raised the TCJA’s elimination of advance refunding for tax- exempt municipal bonds as a potential source of fiscal stress for the state and local government sector. State and local governments use these tax- exempt bonds to finance a broad range of projects and activities, including public infrastructure. Prior to its elimination, the provision allowed state and local governments to take advantage of favorable interest rates to reduce borrowing costs, restructure debt, and free up resources for other projects or investments. A number of experts explained that the elimination of the provision could result in increasing project costs—ultimately increasing infrastructure and debt costs over time. Some experts highlighted overall concerns about states’ eroding sales tax base. For example, the country has transitioned to a more service-based economy, due to changes in consumption. As services have begun to represent a larger and growing share of GDP, there has been an erosion of states’ sales tax bases. In contrast, a number of experts pointed to the outcome of the U.S. Supreme Court’s ruling in South Dakota v. Wayfair, Inc. and its potential for stimulating growth in sales tax revenue. The Court in South Dakota v. Wayfair, Inc. held that states could require out-of-state sellers to collect and remit sales taxes on purchases made from those out-of-state sellers, even if the seller does not have a substantial physical presence in the taxing state. A number of experts noted that remote sales taxes will likely increase state and local sales tax revenues, but that states are still realizing the effects of the ruling on their revenues. Following the U.S. Supreme Court’s decision, numerous states that levy a sales tax and the District of Columbia have taken some kind of action to enforce remote sales tax collections. According to NCSL, as of January 2020, 43 states and the District of Columbia currently require remote sales tax collection. Some states have taken legislative action to change their state laws in response to the outcome of the Wayfair case, while some collection efforts have been led by departments of revenue if statutory authority was already provided. However, it is too soon to determine the full effects of the Wayfair case on states’ sales tax revenue. Natural disasters. A number of experts pointed to the increasing fiscal pressure that state and local governments are under and will continue to face, given the increasing frequency, severity, and cost of natural disasters. We and others have reported on the increasing trend in the number of natural disasters and related costs. For example, in 2018 alone, there were 14 weather and climate disaster events with losses exceeding $1 billion each across the United States with total costs of at least $91 billion, according to the National Oceanic and Atmospheric Administration. Further, disaster costs are projected to increase as extreme weather events become more frequent and intense because of climate change as observed and projected by the U.S. Global Change Research Program and the National Academies. A number of experts acknowledged that the federal government plays a critical role in providing disaster assistance to state and local governments and stressed the need for continued financial support. Some experts discussed the importance of federal assistance since states may need to pay for immediate disaster costs, such as debris removal, out of current expenditures and may not have the funds available to cover those costs. Local governments in particular are generally the first responders in the event of a disaster, often times using their own personnel and funding in these circumstances. Some experts noted that these localities and communities may lack the available cash reserves needed for disaster response-related resources, such as public safety overtime and other types of public assistance. One expert underscored the federal government’s role as an economic stabilizer in providing assistance to local governments during disasters. Given the increase in federal disaster spending, we and others have underscored the importance of finding ways to address the growing costs of natural disasters, citing investment in mitigation as one approach. Some experts we interviewed also pointed to the importance of states’ adoption of mitigation strategies as a way to help states and localities reduce the environmental and fiscal effects of natural disasters. For example, the Pew Charitable Trusts reported in 2020 that a number of states and localities are looking to invest in infrastructure projects that will help mitigate the potential effects of disasters before they occur. For example, according to Pew Charitable Trusts, one state plans to limit development and move residents out of areas most prone to flooding, while improving infrastructure in communities on higher ground that are likely to receive displaced populations from neighboring towns. Another state plans to invest its federal funds in flood control, removing homes from high-risk areas and helping local governments pay for projects. Further, one locality plans to spend $500 million on infrastructure upgrades over the next few years, after its residents voted to authorize a bond to address flooding and other concerns. Further, one expert stressed the importance of the Disaster Recovery Reform Act of 2018 (DRRA) in developing state and local mitigation programs, in addition to strengthening federal, state, and local relationships in disaster response and recovery efforts. Among other things, the act increases the federal investment in predisaster mitigation, increases reimbursement caps for state and local governments on a range of disaster costs, and allows state and local governments to administer housing assistance grants. We reported in 2019 that it is too early to tell what effect implementation of DRRA will have on state and local resilience. In addition, economic literature we reviewed highlighted the potential long-term implications of natural disasters and climate change on state and local governments’ municipal bond ratings. For example, credit rating firms—Fitch Ratings, Moody’s Investors Service, and S&P Global Ratings—indicated that they are considering the effects of climate change in their credit analyses of state and local governments. Specifically, S&P Global Ratings has identified risk factors related to the environment, among other credit risk factors, such as extreme weather events and flooding that can affect an issuer’s ability to meet full and timely debt service. We are sending copies of this report to the appropriate congressional committees and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-6806 or sagerm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. This report examines fiscal pressures for state and local governments. Specifically, the objectives of our review were to (1) examine recent trends in state and local government expenditures and revenues; and (2) synthesize expert views regarding the effect of federal policy on state and local government fiscal pressures. To describe recent trends in state and local government expenditures and revenues, we analyzed categories of aggregate data on state and local expenditures and revenues using inflation-adjusted data from the Bureau of Economic Analysis’s (BEA) National Income and Product Accounts (NIPA) from 1998 to 2018. We analyzed changes in the shares of state and local expenditures and revenues as a percent of total expenditures and revenues respectively from 1998 to 2018. We determined that the NIPA data were the most recent available data for the purpose of examining aggregate state and local government revenue and expenditure trends. The NIPA data do not always match state and local government budget data due to methodological differences between how BEA calculates NIPA data and how state and local governments compute their budget data. We also reviewed our prior reports and those of others to identify what is known about these trends and the factors that affect them. To analyze trends in state and local government revenues and expenditures among states, we used the U.S. Census Bureau (Bureau) government finance data and gross domestic product (GDP) price index data from BEA to calculate inflation-adjusted values of selected expenditure and revenue categories for each state (including the District of Columbia) and for the United States for 1997 to 2017. Data for 1997, 2002, 2007, 2012, and 2017 are based on the Bureau’s Census of Governments, which surveys all state and local governments in the United States. Data for the other years are based on the Bureau’s Annual Survey of Government Finances. In these years, local government finance statistics are based in part on a sample of local governments in the United States. We determined that the Bureau’s data were the most comprehensive for the purpose of examining trends in state and local government expenditures and revenues. However, due in part to definitional differences among the states, such as those of coverage (what constitutes a government entity) or measurement (cash or accrual accounting), the data cannot be used as financial statements to measure a government’s fiscal condition or to calculate a surplus or deficit. We assessed the reliability of the data we used for this analysis and determined that BEA NIPA and the Bureau’s data were sufficiently reliable for our purposes. Our data reliability assessment included reviewing relevant documentation, interviewing knowledgeable BEA and Bureau officials, and reviewing the data to identify obvious errors or outliers. We examined patterns between state and local government revenue growth and growth in overall state and local government spending using data from the Bureau. For each state and the District of Columbia, we assessed how fast each expenditure and revenue category grew between 1997 and 2017 and calculated the average annualized growth rate based on year-to-year changes for each selected expenditure and revenue category. For each expenditure and revenue growth rate calculation, we identified the U.S. average annualized growth rate and the minimum and maximum growth rates across states. Because changes in the levels of expenditure and revenue categories can be affected by changes in state fiscal capacity—such as increased tax revenues due to population growth—we compared the average annual compound growth rate for each category of spending and revenues to the average annual compound growth rate in state gross domestic product (GDP). We chose state GDP as a proxy for each state’s resources or fiscal capacity. We determined state GDP to be the most appropriate representation of a state’s total resources or fiscal capacity. To compare the growth in these categories relative to growth in each state’s resources, we compared the growth rate for each selected expenditure and revenue category to the growth rate in each state’s GDP resources from 1997 to 2017. When expenditures in a state are growing faster than GDP, the share of the state’s resources that are dedicated to state and local government services is growing. Over the long run, such growth could create a fiscal pressure. This analysis also identified the number of states where growth in a category was (1) greater than GDP for that state or (2) less than GDP for that state. We also examined patterns between state and local revenue growth and growth in state and local spending and federal grants using data from the Bureau. For each state and the District of Columbia, we plotted the average annual growth rate in general revenues against the average annual growth rate in general expenditures from 1997 to 2017. We then counted the number of states in which spending grew faster, slower, and at the same rate as general revenues. We also analyzed growth in own- source revenues against growth in federal grant revenues using the same approach. We then counted the number of states in which own-source revenue grew faster, slower, and at the same rate as federal grants. To identify expenditure categories in the Bureau’s data, we selected all of the Bureau’s general expenditure categories. We included other expenditure categories, such as interest on debt and salaries and wages to document their low growth rates. We included insurance benefits and repayments because of its high growth rate and its inclusion of pension benefits, which experts identified as a growing expense in some states. As part of our analysis of trends in state and local government expenditures, we analyzed data from the National Association of State Budget Officers (NASBO) on state rainy day fund balances and general fund expenditures. NASBO’s Fiscal Survey of States surveys state budget officers in 50 states on general fund receipts, expenditures, annual tax and revenue changes, and balance data, which includes rainy day fund balances. We calculated state rainy day fund balances as a percentage of state general fund expenditures among states from 1998 to 2018. We then plotted the median state rainy day fund balances for each year from 1998 to 2018. We assessed the reliability of the data we used for this analysis and determined that NASBO’s data were sufficiently reliable for our purposes. Our data reliability assessment included reviewing relevant documentation and consulting knowledgeable officials about the data. To obtain expert views regarding the effect of federal policy on state and local government fiscal pressures, we conducted a series of structured interviews by telephone or in person with a nongeneralizable sample of individuals representing organizations with recognized expertise in state and local budgeting and finance economics, public policy, and intergovernmental issues. To select these experts, we reviewed their published or other publicly available work, professional affiliations, or recommendations by other experts. These considerations informed whether the experts we selected would be knowledgeable or have expertise related to state and local government fiscal and intergovernmental issues. We identified three categories of experts and selected individuals within each category. These three categories included: (1) officials representing state and local government organizations; (2) providers of financial and credit risk information, such as credit rating agencies; and (3) researchers representing think tanks with expertise in state and local government finance, including taxes, budgeting, and intergovernmental relations. We spoke with representatives from the following 17 organizations as part of our structured interviews: 1. The Council of State Governments 2. Federal Funds Information for States 4. International City/County Management Association 5. Moody’s Analytics 6. National Association of Counties 7. National Association of State Auditors, Comptrollers, and Treasurers 8. National Association of State Budget Officers 9. National Conference of State Legislatures 10. National Governors Association 11. National League of Cities 12. Pew Charitable Trusts 13. S&P Global Ratings 15. Urban-Brookings Tax Policy Center 16. The United States Conference of Mayors The results from the structured interviews are not generalizable and represent the opinions of the individuals from the 17 organizations we interviewed. However, we took steps to obtain opinions from experts with different types of expertise and perspectives. For each question in the structured interview, we coded, organized, and analyzed the responses to develop common themes among the responses, based on the issues that emerged most frequently. We use the terms “a number of,” “some,” and “most” to describe the number of experts who responded on a particular issue. We defined “a number of” or “some” as three or more experts and “most” as nine or more experts. To provide context on these themes and supplement our understanding of this information, we reviewed related research, literature from those interviewed and other organizations, including ourselves, and included relevant examples as appropriate. We conducted this performance audit from January 2019 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Brenda Rabinowitz (Assistant Director), Keith O’Brien (Analyst-in-Charge), Colin Ashwood, and Dylan Stagner made key contributions to this report. David Dornisch, J. Andrew Howard, Courtney LaFountain, Silda Nikaj, Robert Robinson, Ardith Spence, and Frank Todisco also provided support.", "summary": "State and local governments work together with the federal government to deliver a broad range of public services. GAO's prior work has shown that the state and local government sector will likely face fiscal pressures during the next 50 years due to a gap between spending and revenues. The fiscal sustainability of the state and local government sector is essential to effectively implement intergovernmental programs. GAO was asked to review recent trends in state and local government expenditures and revenues, fiscal pressures for state and local governments with intergovernmental implications, and the implications of federal policy for these pressures. This report (1) examines trends in state and local government expenditures and revenues during the past two decades; and (2) synthesizes expert views regarding the effects of federal policy on state and local government fiscal conditions. GAO analyzed data from the Bureau of Economic Analysis National Income and Product Accounts, the U.S. Census Bureau and the National Association of State Budget Officers. GAO also interviewed a nongeneralizable sample of experts from organizations that represent state and local governments, professionals who provide financial and credit risk information (credit rating agencies), and researchers from think tanks to better understand how federal policies affect state and local government fiscal conditions. During the past two decades, the state and local government sector experienced overall growth in spending and revenue. Specifically, inflation-adjusted spending increased from about $1.7 trillion in 1998 to about $2.8 trillion in 2018. Health spending accounted for the largest increase. Inflation-adjusted revenues increased from about $1.6 trillion in 1998 to about $2.6 trillion in 2018. Taxes comprised the largest revenue category. From 1997 to 2017, state and local government expenditures and revenues grew faster than state gross domestic product in most states. On average, growth in expenditures outpaced growth in revenues by 0.3 percentage points per year during the period. Increases in public welfare spending drove spending growth (spending largely for states' share of Medicaid), while federal grants and user charges drove revenue growth. Domestic Product (GDP) in Most States from 1997 to 2017 Source: GAO analysis of U.S. Census Bureau and Bureau of Economic Analysis data. | GAO 20-437 Experts identified a range of issues facing state and local governments that could affect the sector's fiscal condition. Those most frequently mentioned included: Health care. Experts expressed concerns regarding their ability to meet future Medicaid enrollment demands in an economic downturn. Federal budget uncertainty. Uncertainty in the future of federal assistance as well as the timing of federal appropriations, including federal government shutdowns, affected state and local governments' program planning. Physical infrastructure. Aging infrastructure costs and uncertainty in federal funding sources placed pressure on the sector to identify alternative revenue sources for transportation projects. Tax policy . Provisions of the law known as the Tax Cuts and Jobs Act had varied effects on the sector, but most experts agreed it is still too early to assess the act's full effects on state and local government revenues. Natural disasters . Experts acknowledged the important contribution of federal financial support for disaster response and recovery and noted some states' mitigation efforts to address the increasing frequency and cost of disasters. Credit rating firms are considering the effects of climate change in their credit analyses of state and local governments.", "document_type": "gao"}
{"report": "The number of AI/AN veterans eligible for both VA and IHS services is unknown. The U.S. Census Bureau estimates that in 2017 approximately 141,000 AI/AN individuals identified themselves as veterans. This estimate includes only individuals who identified as AI/AN alone and not in combination with another racial group. IHS and VA do not have an administrative mechanism for determining the number of AI/AN veterans who are users of both systems. Instead, each agency separately relies on individuals to identify either as veterans, or as AI/AN, resulting in different counts. Specifically, according to IHS, in fiscal year 2017, 48,169 active IHS users self-identified as veterans. According to VA, in fiscal year 2017, 80,507 VA-enrolled veterans self-identified as AI/AN. VA is charged with providing health care services to the nation’s eligible veterans, and served 6.8 million veterans in fiscal year 2017 with a total health care budget of about $69 billion. VA’s health care system includes 18 regional networks—Veterans Integrated Service Networks—to which each of VA’s facilities is assigned. VA has 170 medical centers, which offer a variety of inpatient and outpatient services, ranging from routine examinations to complex surgical procedures. VA’s health care system also includes community-based outpatient clinics and other facilities that generally limit services to primary care and some specialty care. When needed services are not available at VA facilities or within required driving distances or time frames, VA may purchase care from non-VA providers through its community care programs, such as the Veterans Choice Program. Eligibility for VA health care is based on several factors, including the veteran’s period of active service, discharge status, the presence of service connected disabilities or exposures, income, and other factors. VA uses factors such as these to categorize eligible veterans into eight enrollment priority groups—established to manage the provision of care. Some veterans qualify for free health care services based on service connected disabilities, income, or other special eligibilities, while others may be responsible for co-payments. IHS was established to provide health services to members of AI/AN tribes, and its facilities are primarily in rural areas on or near reservations. IHS’s fiscal year 2017 budget was approximately $5 billion, and the agency served about 1.6 million individuals. The agency is organized into 12 federally designated geographic areas. IHS provides services directly through a federally operated network of 25 hospitals, 53 health centers, and 30 health stations in 37 U.S. states. In addition, about 54 percent of IHS’s funds are provided to THPs to operate about 580 of their own facilities such as hospitals, health centers, clinics and health stations. IHS also provides funding to 41 nonprofit organizations through the Urban Indian Health program to provide health care services to AI/AN individuals living in urban areas. IHS and THP facilities are often limited to providing primary and emergency care services. When needed health care services are not available at IHS or THP facilities, in certain circumstances the facilities may pay external providers to provide these services through IHS’s Purchased/Referred Care (PRC) program. Before the PRC program can provide payment, patients must exhaust all health care resources available to them from private insurance, state health programs, and other federal programs, including VA. Furthermore, eligibility for PRC payment is not automatic, and IHS has reported that PRC funds are not sufficient to pay for all necessary care and, therefore, generally pay for only the highest priority costs, such as emergency care and transportation to that care. To be eligible for IHS health care services, an individual must generally be a member or descendant of one of the current 573 federally recognized Indian tribes, as evidenced by such factors as tribal membership, enrollment, residence on tax-exempt land, ownership of restricted property, active participation in tribal affairs, or other relevant factors. In instances where an AI/AN veteran is eligible for a particular health care service from both VA and IHS, VA is the primary payer. The 2010 MOU between VA and IHS set mutual goals and objectives to facilitate coordinating and resource-sharing between the two agencies. Specifically, the five MOU goals are as follows: 1. Increase access to and improve quality of health care and services to the mutual benefit of both agencies. Effectively leverage the strengths of the VA and IHS at the national and local levels to afford the delivery of optimal clinical care. 2. Promote patient-centered collaboration and facilitate communication among VA, IHS, AI/AN veterans, tribal facilities, and Urban Indian clinics. 3. In consultation with tribes at the regional and local levels, establish effective partnerships and sharing agreements among VA headquarters and facilities, IHS headquarters and facilities, tribal facilities, and Urban Indian Health Programs in support of AI/AN veterans. 4. Ensure that appropriate resources are identified and available to support programs for AI/AN veterans. 5. Improve health promotion and disease prevention services to AI/AN veterans to address community-based wellness. In accordance with these five goals, the MOU contains specific areas in which VA and IHS agreed to collaborate and coordinate, including: Reimbursement: development of payment and reimbursement policies and mechanisms to support care delivered to dually eligible AI/AN veterans. Sharing staff: sharing of specialty services, joint credentialing and privileging of health care staff, and arranging for temporary assignment of IHS Public Health Service commissioned officers to VA. Staff training: providing systematic training for VA, IHS, THP, and Urban Indian Health Program staff on VA and IHS eligibility requirements to assist them with appropriate referrals for services. Information Technology Interoperability: interoperability of systems to facilitate sharing of information on common patients, and establishment of standard mechanisms for VA, IHS, and THP providers to access records for patients receiving care in multiple systems. VA and IHS each designated certain staff to oversee and implement the MOU, but VA is generally responsible for administering the MOU. For example, VA’s Office of Community Care provides oversight of the reimbursement agreements—which are a key part of the MOU. Within that office, VA established the IHS/THP Reimbursement Agreements Program to carry out portions of the MOU related to the development of payment and reimbursement policies. Under these policies, in instances where an AI/AN veteran is eligible for a particular health care service from a VA facility, that veteran can instead receive the eligible service at an IHS or THP facility without prior VA approval and, under a reimbursement agreement, VA will reimburse the facility for the service. Some key aspects of the reimbursement agreement program are as follows: All IHS facilities are covered under one national reimbursement agreement between VA and IHS. THPs each negotiate their own separate reimbursement agreements with VA. While VA uses a reimbursement agreement template based on the agreement with IHS, the terms of each THP agreement may deviate from those in IHS’s national agreement. Urban Indian Health Programs are generally not eligible for reimbursement agreements. VA provides reimbursement for outpatient and inpatient direct care services provided at IHS and THP facilities. VA also reimburses IHS and THP facilities for costs of outpatient prescriptions for AI/AN veterans, as well as filling prescriptions for AI/AN veterans served at IHS and THP facilities through VA’s Consolidated Mail Outpatient Pharmacy program. VA does not provide reimbursement for those services from external providers paid for by IHS or THP PRC programs. VA reports that the process of establishing reimbursement agreements with THPs has multiple phases. The process begins with initial communication between the THP and VA, followed by an orientation briefing. The THP then begins to draft the agreement (based on VA’s template) and prepare required VA paperwork (e.g., an implementation plan and proof of certification or accreditation). Once drafted, the THP submits the draft agreement and paperwork for review by VA’s IHS/THP Reimbursement Agreements Program, followed by review by a VA contracting officer and legal team. The agreement is complete once it is signed by VA and the THP. A joint leadership team of VA and IHS officials continues to oversee the implementation of the 2010 MOU through meetings, regular reporting, and the establishment of goals and measures to assess performance— but these measures lack targets for assessing progress toward the goals. VA and IHS officials also told us they are drafting a revised MOU to be broader and more flexible than the existing MOU and are updating the performance measures. However, officials have not indicated that any revised measures will include targets. Since our last report in 2014, a joint national leadership team comprised of VA and IHS officials has continued to use quarterly meetings, routine reporting, and MOU goals and measures to oversee MOU implementation and help facilitate collaboration. VA and IHS officials told us that the leadership team consists of officials in VA’s Office of Rural Health and Office of Tribal Government Relations, and the IHS Deputy Director for Intergovernmental Affairs. Specifically, the leadership team has met to discuss the progress and status of the MOU, develop implementation policy and procedures, create performance measures and timelines, and evaluate progress on those measures. The leadership team also compiles annual reports on progress in MOU implementation that includes information about activities and challenges on meeting MOU goals using established measures, and information on the reimbursement agreements and outpatient pharmacy program. In addition, VA and IHS issue monthly data reports on the reimbursement agreements, including the total amount disbursed, the number of veterans receiving services reimbursed by VA, and the number of claims processed for IHS and THP facilities. The leadership team receives input from workgroups tasked with the responsibility for implementing and developing strategies to address the goals of the MOU. The workgroups primarily consist of VA and IHS staff who meet periodically to discuss goals and report quarterly to the leadership team. Tribal officials have participated in some MOU workgroups, though they are not a part of the MOU leadership team. Since our last report in 2014, the number of workgroups decreased from 12 to three groups. (See table 1.) VA and IHS officials said that there were a number of reasons why the number of workgroups had decreased over time, such as consolidation into broader groups because the missions of some groups were similar. VA officials noted that the 12 original workgroups reflected the structure of the MOU, but over time they realized that there was not a need for workgroups in some of these areas. With the establishment of the MOU, VA and IHS have been able to share resources and collaborate on activities to improve access of care for AI/AN veterans. VA and IHS reported that the MOU has helped both agencies develop an outpatient pharmacy program for AI/AN veterans, hold joint training and recruitment events, and establish the reimbursement agreement program, among other accomplishments. The VA, IHS, and THP facility officials we spoke with noted activities related to the reimbursement agreements and a few noted improvements in areas such as training and telehealth as a result of the MOU. However, most of the facility officials generally reported they had not observed improvements in national-level VA and IHS collaboration and coordination in other areas identified by the MOU. Additionally, these facility officials told us that their facilities have not implemented any new policies, procedures, or any specific facility performance goals or targets that were linked to the MOU. VA and IHS headquarters officials acknowledged that all areas of the MOU have not been implemented at all facilities, and noted that while improvements have been made in many areas, organizational challenges remain, such as in the area of information technology. One IHS headquarters official added that even though VA and IHS have not fully implemented all parts of the MOU, they have addressed each area of the MOU in some manner. For example, one of the goals of the MOU is to improve coordination of care by developing and testing innovative approaches and disseminating best practices. IHS headquarters officials indicated that the agency has addressed this goal in part by creating an Improving Patient Care program that was informed by using VA curriculum and utilizing lessons learned from VA’s Patient Aligned Care Teams. VA and IHS leadership said they are currently in the process of revising the MOU to be broader and more flexible to better meet the care needs of AI/AN veterans. Regularly monitoring and updating written agreements on collaboration, such as the MOU, is consistent with our key collaboration practices. IHS officials said that in contrast to the current MOU, in the new MOU, they are not looking to delineate every area of coordination and instead are grouping topics into broader areas of coordination. In the fiscal year 2017 MOU annual report, VA and IHS noted they were removing outdated language from the MOU and planned to create a more comprehensive, flexible MOU that would serve both agencies well into the future. VA and IHS officials indicated that these revisions will address some areas in the current MOU that they have not yet been able to implement. In June 2018, VA officials said that the leadership team had decided upon a revised set of MOU goals and associated objectives. In February 2019, VA and IHS reported that the target completion date for the new MOU was spring 2020. VA and IHS have improved their efforts to measure progress towards meeting the five MOU goals since 2014. In response to a recommendation made in our April 2013 report, VA and IHS revised their MOU performance measures in 2015—better aligning the measures with the MOU goals. In addition, as a result of our work in 2013, the agencies revised an existing data collection reporting template used to gather information for each measure—such as the measurable objective, rationale and intent of the measures, action plan, milestones, and barriers—to help determine whether MOU goals were being met. While we found that the three existing MOU workgroups had since stopped using this template, a VA official confirmed that they believe relevant information is still captured through its monthly and quarterly reports. Nonetheless, while VA and IHS improved their performance measurement efforts since our 2013 report, we found that the revised MOU performance measures still do not have quantitative and measurable targets to assess agency progress toward the goals. We have previously reported that performance measures should have numerical targets or other measurable values, which help assess whether overall goals and objectives were achieved by easily comparing projected performance and actual results. Besides having measureable targets, other key attributes of successful performance measures include linkage to an agency’s goals and mission, clarity, objectivity, and balance. None of the 15 revised measures have targets against which performance can be measured to assess progress and evaluate effectiveness. (The results of our assessment are shown in table 2.) For example, while the number of shared VA-IHS trainings and webinars is a performance measure, there is no target for the number of shared trainings VA and IHS hope to complete each year. VA officials we spoke with stated VA has not considered adding targets to these measures, noting that the nature of the measures and MOU work against establishing targets. For example, officials said that the measures related to the reimbursement agreements are dictated by the needs of the population seeking health care and the providers at the IHS and THP facilities. VA officials we spoke with said instead of targets, they assess success or failure by whether they see incremental growth in the measures. Officials added that they examine these measures quarterly to determine if they have increased, decreased, or remained stable. If the measures are stable or decrease, officials said they consider if these trends can be reversed. However, the absence of targets limits the ability of VA and IHS to use these measures to assess performance. Without defined measurable targets or goals, VA and IHS lack a clear basis for objectively and strategically evaluating how and where improvements should be made. For example, while it is helpful to count the number of tribal outreach activities conducted, setting an annual target for such activities would allow the agencies to better assess whether they are meeting their goals in this area. In addition, some of these measures also lacked other attributes important for assessing performance. Specifically, five of the measures listed the completion of an annual metric review, which is a task to execute rather than a desired performance outcome to be measured. VA and IHS also are not using two measures. Specifically, they have not collected any data to track results on the number of VA and IHS employees who attend training and on the quality of health care provided. Relatedly, for the measure on health care quality, VA and IHS have not developed a clear definition against which to measure performance, as specific quality measures have not been determined and data are not being collected. VA and IHS have documented challenges related to confusion and difficulty in tracking some measures; for example, at a meeting in March 2017, the MOU leadership team discussed that measures were not well tailored to the workgroup structure at that time. IHS officials also acknowledged that the measures currently in place are counting activities, but not necessarily always measuring performance—such as whether trainings held were effective. VA officials said that revising the MOU will give them an opportunity to revisit the performance measures used, and that they are looking to apply lessons learned to do a better job in the future at defining the measures. Similarly, IHS officials noted that the agencies are engaged in conversation about the performance measures to make them more useful. However, as previously noted, VA officials said that they have not considered establishing targets for the measures. THP facilities’ use of reimbursement agreements with VA increased from 2014 through 2018. The selected IHS and THP facilities we spoke with viewed the reimbursement agreements as beneficial, but also identified some concerns. The use of VA’s reimbursement agreements with THPs increased from 2014 through 2018, as measured by the number of agreements, claims reimbursed, and veterans served. In addition, there was also an increase in payments made for prescriptions filled through the VA’s Consolidated Mail Outpatient Pharmacy program for AI/AN veterans receiving services at IHS and THP facilities. As all IHS facilities are covered under a single national agreement that was instituted prior to 2014, there was less change in the use of reimbursement agreements by these facilities. Reimbursement agreements entered. The number of reimbursement agreements with THPs more than doubled from 2014 to 2018, increasing about 113 percent. We previously reported, as of May 16, 2014, that VA had 53 reimbursement agreements with THPs. VA data showed that as of December 2018 it had 113 reimbursement agreements with THPs, representing about 34 percent of the 337 total IHS-funded THPs. (See fig. 1.) VA also reported that there were 42 additional pending reimbursement agreements with THPs that were in varying phases of submission, processing, and review. In addition, as in 2014, IHS facilities are covered under a single national agreement, and the number of IHS facilities covered by it has remained similar. In 2014, we reported that VA officials had conducted outreach through tribal letters and events to educate THPs about the option of establishing reimbursement agreements, and officials told us this outreach has continued. As we reported previously, there are several reasons a THP might decide not to have an agreement with VA, such as deciding it was not worth the time and resources needed to establish an agreement. Officials from a national tribal organization we spoke with said that smaller tribes without many veterans or resources may not be interested. IHS officials also noted that if a THP’s veteran population has alternate payment resources (e.g., Medicaid or private insurance), it may not be worth the steps to implement a reimbursement agreement if the THP will not be billing VA for veterans’ services. Amount of claims reimbursed. In fiscal year 2014, VA paid IHS and THP facilities $11.5 million for services provided to AI/AN veterans, which grew to $20.1 million in fiscal year 2018. This increase mainly represents the growth in reimbursement to THP facilities—which grew 181 percent, from $4.3 million in fiscal year 2014 to $12.1 in fiscal year 2018. During this same time period, reimbursements to IHS facilities remained relatively stable, reflecting the stable number of IHS facilities receiving reimbursements. (See fig. 2.) Veterans served. Between fiscal year 2014 and fiscal year 2018, according to VA data, the number of unique AI/AN veterans receiving services reimbursed by VA each year has increased from about 3,800 in 2014 to a high of nearly 5,300. (See fig. 3.) While IHS facilities accounted for a larger percentage of veterans with reimbursed services compared to THPs, the number of veterans receiving services reimbursed by VA at THPs increased significantly. For fiscal year 2014, 2,965 AI/AN veterans received services reimbursed by VA at IHS facilities, which decreased slightly to 2,829 in fiscal year 2018. In comparison, 885 veterans received services reimbursed by VA at THP facilities in fiscal year 2014, which nearly tripled to 2,531 veterans in fiscal year 2018. Prescriptions filled. Similar to increases in the numbers of AI/AN veterans served under the reimbursement agreements, AI/AN veterans’ utilization of VA’s Consolidated Mail Outpatient Pharmacy program has also increased. Prescriptions filled through this program more than doubled—from more than 440,000 prescriptions in fiscal year 2014 to nearly 886,000 prescriptions in fiscal year 2018. (See fig. 4.) VA and IHS annual reports indicate that the pharmacy program has been one of the most successful collaborations between VA and IHS for AI/AN veterans, providing more than 2 million prescriptions for VA-IHS patients since the pharmacy program collaboration began in 2010. While this program was originally limited to AI/AN veterans served at IHS facilities, in December 2016, VA and IHS entered into an Interagency Agreement that extended the program to THPs. Officials from the majority of IHS and THP facilities we contacted said they were generally pleased with the reimbursement agreements. Among those, officials from one THP noted that the revenue received from their reimbursement agreement freed up other resources that allowed them to hire an additional part-time worker to conduct VA outreach activities. Additionally, a representative of a national tribal organization noted that IHS and THP facilities’ funding is limited and this revenue helps them extend services to eligible AI/AN veterans. However, officials from a number of IHS and THP facilities also had concerns about the agreements, including the lack of reimbursement for PRC program services provided by IHS and THP facilities, the length of time it took to enter into the agreements, and the time frames of the agreements: Lack of reimbursement for PRC program services. Officials at most IHS and THP facilities we contacted said they believed VA should reimburse facilities for services from external providers paid through the PRC program. Officials at some facilities said they have had to deny PRC services due to a lack of program funds. According to some facility and IHS area office officials, this issue is particularly relevant in states where Medicaid was not expanded under the Patient Protection and Affordable Care Act (PPACA). In states where Medicaid eligibility was expanded, more AI/AN individuals may therefore be eligible for Medicaid—potentially freeing up PRC funds. For example, an IHS official noted that prior to Medicaid expansion in his state they would have to limit PRC funds to be used only in life or death scenarios after May or June of each year, but that currently his facility was not limiting any PRC services. Given the limitations in PRC program funds, officials from a national tribal organization and some THPs noted they have raised the possibility of including the PRC program in the reimbursement agreements with VA, although the program was ultimately not included. VA officials noted that there is no statutory requirement for them to include the PRC program in the reimbursement agreements and also identified several other reasons for not including it. For example, they said that VA does not want to pay for services externally that it already offers internally and that it would prefer to coordinate the patient’s care within VA’s existing programs, such as VA’s own programs for purchasing care from external providers—like the Veterans Choice Program. The length of time to enter into an agreement. Officials from a few THP facilities and one national tribal organization we spoke with noted concerns about the amount of time it took to enter into reimbursement agreements. Our analysis of VA reimbursement agreement data shows that the median amount of time that it took to enter an agreement with THPs was over 1 year (about 403 days). We found that the number of days from the first contact by a THP to the actual signing of the agreement ranged from 96 days (over 3 months) to 1,878 days (more than 5 years). According to VA records and interviews, there were reasons for delays in completing reimbursement agreements, including lengthy negotiations, incomplete submission of information from the THPs, lapses in communication between VA and the THP, and a THP’s lack of medical certification or accreditation. VA officials explained that the amount of time increases if the THP does not want to use the VA-approved reimbursement agreement template or wants to change the terms of the agreement. For example, an official from one THP facility said that it took 2.5 years to finalize its reimbursement agreement due, in part, to internal challenges with their legal counsel and external challenges with negotiating the terms of the agreement during a time when the VA was developing a national reimbursement agreement template. VA officials also explained that entering the agreement with IHS was simpler than entering agreements with THPs because it was a national agreement between two federal agencies and, for example, did not require having a contracting officer review the agreement—an extra step needed for agreements with non-federal agencies. The length of time reimbursement agreements are in effect. Officials from a few THP facilities expressed a desire for longer reimbursement agreements that would permit greater planning ability. The agreement between VA and IHS was initially set for 3 years. It was then extended twice, once for 2 years and once for 1.5 years. The time frames for THP agreements have generally been extended consistent with extensions to the national agreement. Officials from one THP we spoke with said that having short-term reimbursement agreements causes problems with internal organizational planning and it would be beneficial to have a longer term non-expiring agreement that can be cancelled so that THPs do not continue to expend resources to complete new agreements or amendments every 2 years. In June 2018, VA and IHS signed an amendment to extend the terms of the national reimbursement agreement through June 30, 2022. VA officials said they are currently in the process of working with THPs to similarly extend their agreements. In speaking to officials at selected VA, IHS, and THP facilities about key issues related to coordinating care for AI/AN veterans, we found that the extent of coordination they reported varied widely. For example, three IHS and THP facilities said they had little to no care coordination with their local VA partners; noting, for example, that they rarely refer veterans to VA since they offer more services than the closest VA facilities. Other facilities described more extensive and formalized care coordination, including shared funding of certain VA and THP employees, or VA employees on site at THP facilities to manage veterans’ care and referrals to and from VA. In Alaska, for example, where services offered by VA are very limited, VA instead has formal sharing and reimbursement agreements established with 26 THPs, which provide the majority of services to AI/AN veterans, as well as some non-Native veterans. Two of the THP facilities we spoke with in Alaska have VA employees working on site to help coordinate veterans’ care. VA and IHS headquarters officials indicated that the MOU was intended to allow for variation in the level of coordination at the local, facility level not to create demands or obligations on facilities. One VA official noted that as the new MOU is developed, both VA and IHS want to continue to allow VA, IHS, and THP facilities to engage in whatever level of coordination makes sense. Despite variation in the extent of coordination, officials identified several common challenges regarding coordination between local VA, IHS, and THP facilities: Referring patients to VA facilities. Officials from 9 of the 15 VA, IHS, and THP facilities we contacted reported conflicting information about the process for referring AI/AN veterans from IHS and THP facilities to VA facilities for specialty care. For example, 4 of the IHS and THP facilities we spoke with said that AI/AN veterans generally could not be referred directly to VA specialty care by IHS or THP providers without first being seen and referred by a provider at VA. These facility officials indicated that this practice was a barrier to care. These officials also noted that this could result in the patient receiving, and the federal government paying for, duplicative tests. However, officials at another IHS facility indicated that IHS and THP facilities should be able to refer patients directly to VA specialty care. Additionally, during an interview at a VA facility, local and regional officials had differing understandings of whether IHS and THP facilities could refer patients directly to VA specialty care. VA and IHS headquarters officials both reported that in general, IHS or THP facilities cannot refer a patient to VA specialty care without that patient first being seen in VA primary care. However, VA officials reported that there is no national policy or written guidance on how to refer patients from an IHS or THP facility to a VA facility. VA officials said that the coordination process is left to the local VA facility and the respective IHS or THP facilities and the process can vary from one facility to another— explaining why differing information was reported by facility officials. Our past work on interagency collaborative mechanisms identifies that it is a leading collaboration practice to have written guidance and agreements to document how agencies will collaborate. Without a written policy or guidance about how referrals of AI/AN veterans from IHS and THP facilities to VA facilities may be managed, VA and IHS cannot ensure that VA, IHS, and THP facilities have a consistent understanding of the options available for these referrals. Information technology interoperability and access. Officials at 10 of the 15 VA, IHS, and THP facilities we contacted cited challenges related to accessing each other’s health information technology systems. Most stated that a lack of interoperability of their electronic health records caused challenges, while a few IHS and THP facilities also mentioned that the lack of access to VA systems makes it difficult to verify a veteran’s eligibility or determine the services for which VA will reimburse. For example, one THP noted that if an AI/AN veteran was sent to VA for a service, the THP provider would not receive the veteran’s follow-up records as quickly as if they had access to each other’s systems. Improving systems’ interoperability was a focus area identified in the MOU, and an IHS official indicated that while the agencies had some initial work on the topic, no systematic solutions were identified. We have previously identified VA’s lack of systems interoperability—particularly with the Department of Defense—as a contributor to the agency’s challenges related to health care. VA and IHS officials identified some potential workarounds to this lack of interoperability, although they noted that some of the described workarounds could be time consuming and may not be feasible for all facilities: An IHS headquarters official said that IHS and VA each have the ability to request the sharing of information from an individual electronic health record held by the other agency through secure emails—although the official noted that this is not as fast or efficient as being able to log in to each other’s systems. VA officials also reported that VA belongs to the eHealth Exchange— a national health information exchange—and said that IHS or THPs could join that, through which they would be able to access information about common veteran patients. However, IHS reported that although the agency explored connecting to the eHealth Exchange several years ago, testing and onboarding costs to participate were prohibitive. IHS noted that several individual facilities across the IHS system have elected to invest in connections with regional health information exchanges. Similarly, two THPs we spoke with reported being a part of other, more locally-based health information exchanges, but noted that VA was not part of these exchanges. A VA official noted that there is an enrollment guide that details how enrollment and eligibility verification will be managed between IHS, THP, and VA facilities. This guide describes how IHS or THP facilities can request veterans’ enrollment and eligibility information from the VA Health Eligibility Center using a templated spreadsheet that sends requests via email through a secure data transfer service. VA’s Health Eligibility Center verifies the list and returns the completed enrollment/eligibility excel spreadsheet to the IHS or THP facility securely. IHS and THP facilities can also contact the VA Health Eligibility Center directly by telephone for fewer than five veterans per call, or their local VA medical center by telephone to verify one AI/AN veteran’s enrollment and eligibility per call. IHS or THP facilities could also enter an arrangement with a local VA facility to have VA employees or co-funded employees on site at IHS or THP facilities, or to have VA-credentialed employees that can access VA systems to share information. However, these options may not be systemic solutions that work at all facilities. An IHS headquarters official noted, for example, that not all IHS or THP facilities have the type of relationship with their local VA facility that would lead to the establishment of such arrangements. In terms of the potential for improving interoperability in the future, VA is in the process of implementing a new electronic health record system, and we have previously reported that VA has identified increased interoperability as a key expected outcome of its decision to switch systems. Officials from two VA and THP facilities were hopeful that this new system will help improve interoperability since some THPs use an electronic health record system from the same company that VA has a contract with. Additionally, an IHS headquarters official said that IHS is also reevaluating its information technology platform and one requirement of any new IHS system will be to enhance interoperability with VA, pending the funding to do so. IHS also reported that the agency will consider health information exchange participation as part of the agency’s information technology modernization efforts. Staff turnover. Officials from 9 of 15 facilities identified staff turnover at VA, IHS, and THP facilities as an impediment to having better or consistent coordination. VA, IHS, and THP facility officials described situations in which the coordination between facilities was dependent on specific staff or facility leadership. According to officials, when there was turnover among these staff or positions went unfilled, or were eliminated, the coordination decreased or came to a halt. For example, officials at one VA facility said that they have found that if a sitting tribal government expresses interest in VA collaboration, they have to act quickly and work with the tribe before there is turnover and new tribal leadership comes in with different priorities. Additionally, officials from one IHS facility described a situation in which they had previously coordinated with their local VA facility through that facility’s AI/AN liaison. However, the coordination lapsed when the liaison left VA and the position went unfilled. Similarly, a THP official stated that coordination with VA was previously led by a nurse case manager on site who was a joint VA and THP employee. The official said that since that person’s retirement, she did not know who to contact at VA to coordinate veterans’ care. Officials at one IHS facility noted that due to turnover and attrition they would like to see more education for front line staff at both IHS and VA, so they can more efficiently obtain care for patients at the VA. VA headquarters officials acknowledged that staff turnover and retraining is a challenge that they will need to continually address as the MOU is carried out. In our prior work related to IHS and VA, we have found that both agencies face challenges related to staff turnover and training. VA Co-Payments. Officials at 3 of the 11 IHS and THP facilities we contacted, as well as IHS headquarters officials and representatives of two national tribal organizations said that the copayments that VA charges veterans represented a barrier to AI/AN veterans receiving care. While AI/AN veterans do not have any cost-sharing for care provided at IHS or THP facilities, they are subject to the same copayments as other veterans when they receive care from VA facilities. VA data shows, for example, that of the 80,507 VA-enrolled self-identified AI/AN veterans in in fiscal year 2017, about 30 percent were charged copayments, averaging about $281.56 billed per veteran. Officials from one THP noted that this kind of financial liability may discourage AI/AN veterans from getting care at VA, or lead them to return to the THP after they realize they will have to pay for care at VA. While some of our interviewees suggested that VA should waive copayments for AI/AN veterans, a VA official said they do not have the legal authority to do this. The official said that their statute specifies the categories of veterans for which they must charge copayments and VA is not authorized to waive the copayments for AI/AN veterans on the basis of their AI/AN status without statutory exemptions. While certain AI/AN veterans may qualify for waived copayments based on their inclusion in other statutory categories, AI/AN veterans are not specifically listed as a category for which copayments can otherwise be waived. VA officials also cautioned that because AI/AN veterans may qualify for waived copayments through these other categories, the possibility of copays should not discourage IHS or THP facilities from referring AI/AN veterans to VA. Since 2014, VA and IHS have continued to work together to oversee and implement their MOU aimed at improving the health care provided to dually eligible AI/AN veterans. While the agencies have made progress in certain areas of the MOU, especially those related to reimbursement, other parts have seen less attention. VA and IHS are now updating the MOU, and plan to revisit the related performance measures. This gives the agencies an opportunity to evaluate how well their existing oversight mechanisms have been working, and to improve these mechanisms accordingly in the future. Regardless of these updates, the agencies need to have effective performance measures. While the agencies took steps to improve MOU performance measures in response to one of our prior reports, these steps were not sufficient and the measures they set lack important attributes, including measurable targets. VA and IHS have indicated that they plan to reevaluate performance measures as they update the MOU, but have not indicated that these new measures will identify targets. Absent targets, VA and IHS are limited in their ability to measure progress towards MOU goals and ultimately make strategic decisions about how and where improvements should be made. At the local level, care for AI/AN veterans relies on coordination among individual VA, IHS, and THP facilities. However, variations in relationships among these many facilities and staff turnover creates challenges, which heightens the importance of clear and consistent guidance from the national level. Yet no written guidance exists related to referring AI/AN veterans to VA facilities for specialty care. Without such guidance, VA and IHS cannot ensure that facilities have a consistent understanding of the available referral options for AI/AN veterans. Enhancing their guidance in this area will help VA and IHS ensure that AI/AN veterans have access to needed care. We are making a total of three recommendations, including two to VA and one to IHS. Specifically: As VA and IHS revise the MOU and related performance measures, the Secretary of Veterans Affairs should ensure these measures are consistent with the key attributes of successful performance measures, including having measurable targets. (Recommendation 1) The Secretary of Veterans Affairs should, in consultation with IHS and tribes, establish and distribute a written policy or guidance on how referrals from IHS and THP facilities to VA facilities for specialty care can be managed. (Recommendation 2) As VA and IHS revise the MOU and related performance measures, the Director of IHS should ensure these measures are consistent with the key attributes of successful performance measures, including having measurable targets. (Recommendation 3) We provided a draft of this report to VA and the Department of Health and Human Services for review and comment. We have reprinted the comments from VA in appendix I and the comments from the Department of Health and Human Services in appendix II. Both departments concurred with our recommendations. The Department of Health and Human Services also provided technical comments, which we incorporated as appropriate. In response to our recommendations to ensure revised performance measures include key attributes of successful performance measures, VA and the Department of Health and Human Services provided information about the process for finalizing the new MOU, including conducting tribal consultation. They noted that VA and IHS will work together to ensure that performance measures under the new MOU include appropriate measurable targets. Regarding our recommendation to VA about establishing and distributing a written policy or guidance on how referrals from IHS and THP facilities to VA facilities for specialty care can be managed, VA noted the Office of Community Care is working on a process to enhance care coordination among all VA and non-VA providers—including IHS and THP providers. VA noted that for IHS and THPs, this will include establishing forms and procedures to refer patients to VA for specialty care, and that VA will provide training to applicable staff once the process and procedures are finalized. VA also noted that it is in the process of establishing an advisory group that will include tribal, IHS, and VA representation, and will make recommendations related to care coordination guidance and policies. The target completion date for establishing this group is spring 2020. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretaries of VA and the Department of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov/. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or farbj@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix III. In addition to the contact named above, Kathleen M. King (Director), William Hadley (Assistant Director), Christina Ritchie (Analyst-in-Charge), Jennie Apter, Shaunessye D. Curry, Jacquelyn Hamilton, and Vikki Porter made key contributions to this report.", "summary": "A 2010 MOU set mutual goals for VA and IHS collaboration and coordination related to serving AI/AN veterans. Under this MOU, VA has established reimbursement agreements with IHS and tribal health programs to pay for care provided to AI/AN veterans. In 2013 and 2014, GAO issued two reports on VA and IHS implementation and oversight of the MOU. GAO was asked to provide updated information related to the agencies' MOU oversight. This report examines (1) VA and IHS oversight of MOU implementation since 2014, (2) the use of reimbursement agreements to pay for AI/AN veterans' care since 2014, and (3) key issues identified by selected VA, IHS, and tribal health program facilities related to coordinating AI/AN veterans' care. To conduct this work, GAO reviewed VA and IHS documents, reports, and reimbursement data from 2014 through 2018. GAO interviewed VA and IHS officials at the headquarters level, and officials at 15 VA, IHS, and tribal facilities in four states—Alaska, New Mexico, North Carolina, and Oklahoma—selected based on factors including the number of reported AI/AN veterans served, and geographic diversity. GAO also interviewed organizations representing tribes and tribal health programs. The Department of Veterans Affairs (VA) and the Department of Health and Human Services' (HHS) Indian Health Service (IHS) established a memorandum of understanding (MOU) to improve the health status of American Indian and Alaska Native (AI/AN) veterans through coordination and resource sharing among VA, IHS, and tribes. Since GAO's last report on the topic in 2014, VA and IHS have continued to jointly oversee the implementation of their MOU—for example, through joint workgroups and quarterly meetings and reports—but they lack sufficient measures for assessing progress towards MOU goals. Specifically, while the agencies established 15 performance measures, they did not establish targets against which performance could be measured. For example, while the number of shared VA-IHS trainings and webinars is a performance measure, there is no target for the number of shared trainings VA and IHS plan to complete each year. GAO's work on best practices for measuring program performance has found that measures should have quantifiable targets to help assess whether goals and objectives were achieved by comparing projected performance and actual results. VA and IHS officials said they are currently in the process of revising the MOU and updating the performance measures used. However, officials have not indicated that any revised measures will include targets. Total reimbursements by VA for care provided to AI/AN veterans increased by about 75 percent from fiscal year 2014 to fiscal year 2018. This increase mainly reflects the growth in reimbursement from VA to tribal health program facilities—facilities that receive funding from IHS, but are operated by tribes or tribal organizations. Similarly, the number of VA's reimbursement agreements with tribal health programs and the number of AI/AN veterans served under the reimbursement agreements also increased during this period. The VA, IHS, and tribal facility officials GAO spoke with described several key challenges related to coordinating care for AI/AN veterans. For example, facilities reported conflicting information about the process for referring AI/AN veterans from IHS or tribal facilities to VA, and VA headquarters officials confirmed that there is no national policy or guide on this topic. One of the leading collaboration practices identified by GAO is to have written guidance and agreements to document how agencies will collaborate. Without a written policy or guidance about how referrals from IHS and tribal facilities to VA facilities should be managed, the agencies cannot ensure that VA, IHS, and tribal facilities have a consistent understanding of the options available for referrals of AI/AN veterans to VA specialty care. This could result in an AI/AN veteran receiving, and the federal government paying for, duplicative tests if the veteran is reassessed by VA primary care before being referred to specialty care. GAO is making three recommendations—one each to VA and IHS to establish measurable targets for performance measures and one to VA to establish written guidance for referring AI/AN veterans to VA facilities for specialty care. VA and HHS concurred with these recommendations.", "document_type": "gao"}
{"report": "The Kingpin Act authorizes Treasury to identify and apply sanctions to significant foreign narcotics traffickers and their organizations worldwide to protect the national security and economy of the United States. According to officials from OFAC and other partner agencies, key goals of the Kingpin Act include disrupting and dismantling drug trafficking organizations and blocking designees’ access to the U.S. financial system. The Kingpin Act mandates the participation of certain agencies in the Kingpin designation process. The Secretary of the Treasury, after consulting with partner agencies, is authorized to designate a foreign national or entity as a Specially Designated Narcotics Trafficker. The partner agencies participating in Kingpin Act designations are the Department of Justice (DOJ), State, DHS, DOD, CIA, FBI, and DEA. For Treasury to designate a foreign individual or entity under the Kingpin Act, it must identify that individual or entity as either a significant foreign narcotics trafficker or part of a designee’s network. The following offices in Treasury are involved in identifying designation targets, and managing and assessing the impact of sanctions: The Office of Terrorism and Financial Intelligence (TFI) has the twin aims of safeguarding the U.S. financial system against illicit use and combatting national security threats, including drug kingpins. TFI includes OFAC, the Office of Intelligence and Analysis (OIA) and the Office of Terrorist Financing and Financial Crimes (TFFC). OFAC is Treasury’s primary office for sanctions implementation and enforcement. OIA is responsible for TFI’s intelligence functions and performs some assessment of the impact of Treasury’s sanctions programs. TFFC works across the national security community and with the private sector and foreign governments to identify and address the threats presented by illicit finance to the international financial system. Treasury can designate a foreign individual or entity under the Kingpin Act if it identifies an individual or entity as either a significant foreign narcotics trafficker or part of a designee’s network. OFAC and its partner agencies have grouped these Kingpin Act designation categories into two tiers, Tier 1 and Tier 2, based on the procedures required for identification and designation under the act. (See tab. 1.) All identifications and designations under the Kingpin Act are subject to the same asset blockings and penalties. The names of persons and entities designated are published in the Federal Register and incorporated into Treasury’s Specially Designated Nationals and Blocked Persons List (SDN List). The majority of Tier 1 Kingpin Act designations are individuals and entities from countries in the Western Hemisphere, as shown in figure 1. Treasury is authorized to block assets of and prohibit transactions with designated individuals and entities and to impose penalties on individuals and entities that engage with designees. Blocking assets. Treasury blocks (i.e., denies access to) a designated individual or entity’s property and interests in property within the United States, or within the possession or control of any United States individuals or entities that are owned or controlled by the blocked individual or entity. Prohibiting transactions. Treasury generally prohibits United States individuals and entities from engaging in transactions in property or interests in property of designees. Denying visas. Treasury provides information to State so it can decide whether to cancel existing visas and deny visa applications of Kingpin Act designees. Penalties for nondesignees. Treasury may enforce criminal and civil penalties for any U.S. person who willfully violates the prohibitions in the Kingpin Act, associated regulations, or license rules. Penalties for violations of the Kingpin Act range from civil penalties of up to $1.5 million per violation to more severe criminal penalties. Criminal penalties for corporate officers in violation may include up to 30 years in prison and fines up to $5 million for individuals and $10 million for corporations. Treasury is required to report to Congress on the status of sanctions imposed under the Kingpin Act, including the personnel and resources directed toward imposing such sanctions during the preceding fiscal year. On July 1st of each year, the OFAC Director, as delegated by the Secretary of the Treasury, is required to submit a report to the Permanent Select Committee on Intelligence of the House of Representatives and the Select Committee on Intelligence of the Senate on the status of sanctions imposed under the Kingpin Act, the personnel and resources directed toward imposing sanctions under the Kingpin Act during the preceding fiscal year, and background information with respect to the newly identified significant foreign narcotics traffickers and their activities. Treasury is also required to report on foreign persons who are sanctioned under the Kingpin Act to the Director of the Office of National Drug Control Policy (ONDCP); ONDCP is the Executive Branch office responsible for issuing an annual National Drug Control Strategy and coordinating the efforts of the National Drug Control Program agencies implementing any aspects of the strategy. Treasury’s OFAC leads a process generally involving six steps to designate Kingpin Act targets. This process allows OFAC to coordinate its investigations and designations with U.S. partner agencies and foreign governments. (See fig. 2.) We determined the designation process through interviews with OFAC and partner agency officials, and selected nine cases to review the implementation of the designation process. OFAC’s Kingpin Act designation process includes the following six steps: 1. Identify potential targets. OFAC first identifies potential targets for investigation and Kingpin Act designation. OFAC’s partner agencies can submit recommendations for potential targets. According to OFAC officials, they consider information provided about potential Tier 1 targets from the recommending agency, such as whether the targets are on the U.S. multiagency list of priority drug trafficking targets, what unique identifiers the recommending agency can provide to minimize the chance of investigating the wrong target, and which drug(s) and quantities the targets traffic and to which markets. Additionally, OFAC considers (1) the likelihood that the target would meet the legal criteria for designation and have an impact, (2) the expectation that designation would complement rather than hinder law enforcement and foreign counterpart investigations and operations, (3) any unintended negative consequences on third parties, and (4) the current availability of OFAC resources. According to OFAC officials, Tier 2 targets are generally identified as part of the investigation of a Tier 1 target or designee. Officials said the decision to pursue designation depends on whether there is sufficient evidence to demonstrate that the target satisfies the designation criteria in the act. As early as at this step, but at some point before designation, OFAC coordinates with partner agencies to ensure that they do not have an ongoing investigation or other diplomatic interactions that will be adversely affected by a Kingpin Act designation. 2. Gather evidence. OFAC gathers evidence on the identified target to determine whether it meets the criteria for identification or designation and whether there is a network associated with the target. OFAC requests information on the target from other partner agencies. According to OFAC officials, they do not request information from all of OFAC’s partner agencies during the investigation of each target if they deem the information provided by a subset of the partner agencies to be sufficient evidence. OFAC also conducts its own research and uses all sources—including public and classified—to develop an evidentiary package. OFAC works with partner agency headquarters, and domestic and international field offices (as needed for each case) to collect information on either a person’s drug trafficking activities or activities that support drug trafficking organizations. OFAC and partner agency officials said they also collect information about targets from their foreign government partners and counterparts, as appropriate. OFAC also ensures that the derogatory information collected is linked to the target and not, for example, another person with the same name. 3. Assemble evidentiary package. OFAC compiles the collected information into an evidentiary package maintained in its electronic case management system. According to OFAC officials, the case management system documents the date when each step is completed and contains sign off by an approving official. In addition, OFAC officials said the case management system contains a summary of the evidence OFAC gathered to justify designating an individual or entity, and links to the source documents provided by partner agencies. Because the information in the evidentiary package may be sensitive, classified, and compiled from multiple sources, OFAC typically does not share the evidentiary packages with its partner agencies, with the exception of DOJ for legal review purposes. However, under certain circumstances, OFAC may allow partner agencies to review portions of an evidentiary package after ensuring that there is a specific need to know and that there is adherence to rules for disclosure to another agency. 4. Legal review. OFAC provides the evidentiary package first to Treasury’s Office of General Counsel and then to DOJ’s Civil Division for legal review. Treasury’s Office of General Counsel reviews the package for legal sufficiency, while DOJ assesses the risks associated with potential future litigation resulting from the identification or designation. According to OFAC and DOJ officials, attorneys often seek clarification or additional evidence from OFAC at this stage. In those cases where Treasury’s Office of General Counsel deems the basis for designation or identification to be legally sufficient and DOJ determines that the identification or designation presents an acceptable level of litigation risk, they give OFAC clearance to finalize the evidentiary package and proceed with the action. 5. Consult with partner agencies. Once the evidentiary package passes legal sufficiency, OFAC consults with all of its partner agencies to obtain concurrence. OFAC presents the names of individuals or entities it has decided to designate and a high-level summary of the reasons for designation to its partner agencies for final consultation and concurrence. According to officials from each of the partner agencies, this allows them the opportunity to identify if OFAC’s plan to designate a target will damage any of their operations or ongoing investigations or cause unacceptable damage to diplomatic relations with the host government in the country where the target resides or maintains citizenship. This consultation phase also allows for OFAC and other Treasury offices, such as the Office of Terrorist Financing and Financial Crimes (TFFC), as well as partner agencies to develop an engagement plan for outreach with relevant parties, including foreign governments and the press, as appropriate. While partner agencies at the U.S. embassy in the country of the proposed designation are given the opportunity to concur with OFAC’s decision to designate, agency representatives in headquarters give final agency concurrence. OFAC does not designate anyone unless all partner agencies concur. If an agency tells OFAC at any point during the process that designating a target would damage the agency’s investigation or operations, OFAC officials said they coordinate with the partner agency to determine how to proceed. For example, OFAC may delay the Kingpin Act designation until the partner agency has completed its investigation and can take simultaneous action against the target. 6. Designate the target(s). If all partner agencies concur with OFAC’s designation proposal, OFAC takes action to identify the Kingpins and designate any affiliated targets. The evidentiary package is provided to the OFAC Director who, if concurring with the designation, signs a memorandum that identifies or designates the targets. At this time, OFAC also adds the individuals and entities to the SDN List. OFAC announces the actions publicly and records them in the Federal Register. Figure 3 provides an example of an OFAC announcement of a Kingpin Act designation. According to Treasury officials, OFAC also coordinates with other Treasury offices and partner agencies at headquarters and U.S. embassies to execute an outreach and engagement plan. Once OFAC has taken these steps, it begins to monitor and enforce compliance with the sanctions it imposes against Kingpin Act designees. OFAC and U.S. partner agency officials said flexibility built into the process can affect the length of time it takes to investigate a target and the sequence of steps taken. For example, OFAC’s coordination with multiple U.S. partner agencies and foreign governments throughout the process may influence the sequence of steps taken. In addition, drug traffickers often change their organizations and operations in an attempt to evade investigators, which can contribute to the length of time to complete an investigation. According to OFAC officials, the process is intended to ensure that: designations do not jeopardize other agencies’ ongoing investigations, OFAC’s actions are coordinated with other planned civil or criminal actions against each target to maximize the disruption to the drug trafficking organization, and investigators can collect sufficient evidence to designate targets despite targets’ constantly changing efforts to evade detection. Coordination with partner agencies. Multiple U.S. agencies may have concurrent investigations of a Kingpin Act target, requiring coordination between OFAC and U.S. partner agencies to include decisions about how sharing information could affect their own investigations. When agencies withhold information about a target to ensure that their own investigation of the target is not compromised, it may take longer for OFAC to develop an evidentiary package that satisfies the Kingpin Act’s designation criteria. In addition, the length of the designation process and the sequence of steps also depend on how far along other agencies’ investigations of a target are. For example, if a law enforcement agency is able to provide enough evidence when the potential target is first identified and OFAC officials think little additional investigation is needed to further develop an evidentiary package, they may complete more than one of the designation steps concurrently in order to designate the target as quickly as possible. According to OFAC and partner agency officials, the coordination allows them to agree to and plan the civil and criminal actions to be taken to maximize the U.S. government’s efforts to disrupt the drug trafficking organization. Coordination with foreign government officials. According to OFAC and U.S. partner agency officials at headquarters, in Mexico, and in Colombia, foreign government officials determine whether to share derogatory information about Kingpin Act targets on a case-by-case basis. OFAC and partner agency officials in Colombia credited host government information sharing as a primary factor in OFAC’s ability to complete evidentiary packages for Colombian targets and one reason why OFAC has been able to investigate and designate more individuals and entities in Colombia than in other countries. Coordination with foreign partners also allows OFAC to time designations strategically to coincide with civil and criminal actions against the target by foreign governments. For example, on May 17, 2019, the Under Secretary for TFI and a Mexican government official announced coordinated, sanctions-related actions. The Under Secretary announced the Kingpin Act designation of seven individuals and six entities affiliated with the Cartel de Jalisco Nueva Generacion (CJNG) and its close ally, the Los Cuinis drug trafficking organizations. Treasury coordinated closely for months with the Mexican Financial Intelligence Unit, the Mexican Attorney General’s Office, and the Mexican Federal Police on this action. The Mexican Financial Intelligence Unit froze the Mexican bank accounts held by all of the designees, according to Treasury officials. Although actions like this sometimes require them to delay a designation, OFAC officials noted that the results of coordination can increase the impact of Kingpin Act designations. Changes to drug trafficking organizations. According to OFAC and partner agency officials, drug traffickers attempt to evade investigators by being unpredictable and making changes to their organizational structure and operations. Changes to the organization may result in the need for longer investigations if information gathered about an individual trafficker or a trafficking organization becomes outdated or irrelevant. Operational changes include such things as using shell companies or virtual assets, which several OFAC and partner agency officials said complicate their attempts to gather evidence of proceeds from drug trafficking, and can also lengthen the designation process. Based on our analysis of nine Kingpin Act designations, we found that the duration and sequence of steps leading to designations varied. According to OFAC officials, each investigation includes a unique set of circumstances that affect the length and sequence of steps. From initiation to designation, the nine cases we reviewed ranged from 6 months to 38 months (See fig. 4.) Time spent preparing the evidentiary packages for the cases ranged from 3 months to 31 months. Although OFAC got partner agency concurrence for seven cases after attorneys had begun the legal review of evidentiary packages, OFAC documented completion of this step before legal review had begun for two cases. The timing for submitting the case to partner agencies for initial designation consideration varied, including one case that OFAC did not submit until the month that attorneys completed legal reviews of the evidentiary package. For one case, OFAC followed most of the steps twice before designating the target. OFAC officials told us that if the decision is made to delay a designation after they have completed all of the steps leading up to designation, it may be necessary to go through the steps again to determine whether there is new derogatory information about the target and whether the information in the evidentiary package is still current and legally sufficient before designating the target. OFAC officials reported that they disseminate information about the designation process and agencies’ roles and responsibilities for the process in several ways. Treasury’s website includes Frequently Asked Questions that explain how agencies should interact with OFAC and each other and a hotline number that agencies can use if they need additional information. OFAC has provided presentations and memos to its partner agencies that further explain the Kingpin Act designation process. Treasury has also issued Kingpin Act regulations, which, among other things, define key terms related to the act and clarify prohibited activities. Because DEA is involved in the majority of OFAC’s Kingpin Act investigations, OFAC and DEA have signed a memorandum of understanding that further clarifies how they work together and share information related to Kingpin Act cases. Among other things, it establishes the terms for OFAC to have a staff person co-located at DEA and to have access to DEA files that support Kingpin Act investigations. According to OFAC officials, it does not have a similar formal collaboration mechanism with its other partner agencies. OFAC’s partner agencies reported that they generally understand their responsibilities under the Kingpin Act and how to find answers to their questions about the Kingpin Act designation process. Several officials stated that their responsibilities include recommending potential targets, participating in interagency group meetings, deciding whether to concur with OFAC’s decisions to investigate or designate persons, and responding to specific requests for information from OFAC. Officials from the headquarters of each of the Kingpin Act partner agencies said they found the information available from OFAC about the designation process sufficient to help them understand their roles. Most partner agencies in Colombia and Mexico, where the majority of Kingpin Act designations have taken place, reported that the presence of an OFAC Attaché in those countries made it easy for them to ask for clarification on the process as needed. OFAC monitors and enforces financial sanctions against Kingpin Act designees implemented by U.S. financial institutions. OFAC regulations and a memorandum of understanding with Federal Banking Agencies (FBA)—such as the Federal Reserve and the Office of the Comptroller of the Currency (OCC)—establish sanctions compliance and information sharing responsibilities. For example, OFAC regulations require banks to report all blockings of designee property to OFAC within 10 days of the occurrence and recommend that banks designate a Compliance Officer responsible for monitoring compliance with its programs, and an officer responsible for overseeing blocked funds. According to the memorandum, each FBA will provide OFAC the following types of information to help OFAC monitor bank compliance with sanctions programs, including Kingpin Act sanctions: Notification of any apparent unreported sanctions violations discovered during their examinations of financial institutions. Information on their examinations into a bank’s OFAC compliance policies, procedures, and processes. Notification of any deficiencies in a bank’s compliance programs, such as cases when a bank failed to respond to supervisory warnings concerning OFAC compliance violations. FBAs have established a schedule for regular examinations of U.S. banks, which generally include their OFAC compliance programs. Federal Reserve and OCC officials stated that their legal and bank examiner staff address sanctions compliance regimes as part of their general examination duties. They are not responsible for determining sanctions violations, but assess the bank’s compliance program as a whole for soundness. Both FBAs said they perform bank examinations every 12 to 18 months, and determine the extent to which they should review the bank’s OFAC compliance program during the examination. For example, OCC officials said that, in accordance with the guidelines, they review banks’ internal testing of their OFAC compliance programs. According to OCC and Federal Reserve officials, banks have compliance programs to identify and block OFAC designees, including Kingpin Act designees, from accessing the U.S. financial system. According to OFAC and FBA officials, the U.S. banks with the most international branches and non-U.S. clients are most likely to hold assets or facilitate financial transactions of foreign nationals. FBA examinations confirm that bank programs include procedures for ensuring compliance with OFAC sanctions, including Kingpin Act sanctions. We met with officials from the five U.S. banks that FBA officials said have the largest presence in Latin American countries, and bank officials reported that their compliance programs check daily for evidence they are maintaining any customer relationship or allowing any transactions involving designated individuals. OFAC has imposed a range of penalties on banks that have violated the terms of Kingpin Act sanctions. OFAC and bank officials said that a bank is noncompliant when it either fails to freeze a Kingpin Act designee’s assets at that bank or processes a transaction involving a Kingpin Act designee. According to Treasury officials, Treasury makes public any civil monetary penalties it imposes and OFAC has imposed monetary penalties on banks for Kingpin Act compliance violations in 12 cases for a total of $17 million since 2000. Officials from the five banks we spoke with said they self-report cases of noncompliance with OFAC sanctions against Kingpin Act designees as required. For example, one of the banks stated that they identify and report between six and 12 cases of noncompliance to OFAC each year. OFAC officials said when a bank self-reports a violation OFAC often issues them a cautionary letter. According to OFAC officials, they issue this as a warning when they have no reason to believe that the bank committed the violation intentionally or that it is evidence of a systematic problem that the bank has not taken steps to address. The OFAC officials said the letter may or may not include a required response from the bank. State has denied visa applications and revoked visas of Kingpin designees, prohibiting them from traveling to the United States after they were designated. According to both OFAC and State’s Consular Affairs officials, State contacts OFAC whenever a visa adjudicator finds information in State’s Consular Affairs database regarding a possible OFAC concern about a visa applicant. State’s officials use their Consular Lookout and Support System database to identify any information entered by U.S. government agencies, including OFAC, to indicate that an individual does not qualify for a U.S. visa. Although the consular database does not specify that OFAC’s concern is specifically related to a Kingpin Act designation, they do not issue a visa without discussing with OFAC whether the applicant’s designation disqualifies them from a visa. The consular database does not specify which OFAC flags are related to the Kingpin Act, and State was unable to provide us the number of visas that have been revoked or denied under the program. OFAC has also blocked (or denied access to) designees’ U.S. property as part of Kingpin Act sanctions. According to OFAC officials, they seek to identify U.S. property that designees own or control as part of their investigation of designees both before and after they are designated. As a result of those investigations, OFAC officials said they have blocked 15 U.S.-incorporated companies, nine real estate properties, and 21 other “tangible” properties (such as automobiles, aircraft, and boats), which remained blocked as of August 2019. An individual’s property is no longer blocked if that individual is removed from the SDN list. U.S. citizens, corporations, and financial institutions are not permitted to do business with blocked companies. OFAC has met the mandated requirement to report to Congress on agencies’ personnel and resources expended on the imposition of Kingpin Act sanctions, but provided limited guidance to partner agencies that has resulted in inconsistent data on Kingpin Act–related expenditures. Furthermore, OFAC has not disclosed limitations to the consistency or reliability of the expenditure data in its reports. The Kingpin Act requires Treasury, no later than July 1 each year, to provide the Permanent Select Committee on Intelligence of the House of Representatives and the Select Committee on Intelligence of the Senate a report describing the status of sanctions imposed, including the personnel and resources directed towards the imposition of such sanctions during the preceding fiscal year, and providing background information with respect to newly-identified significant foreign narcotics traffickers and their activities. OFAC has submitted annual reports to Congress since 2003. Each report includes information from OFAC on both agencies’ expenditures and on designations announced during the year. To prepare the report, OFAC requests partner agency expenditures. OFAC sends annual emails requesting the amount agencies spent on personnel and resources for their Kingpin Act activities. For at least the last 3 years, OFAC has sent the partner agencies a memo stating that for personnel expenses, agencies could estimate the percentage of time spent by staff members on activities directly attributable to implementing the Kingpin Act during the year covered in the report and multiply by the staff members’ salaries during the year. However, the guidance does not clarify or provide examples of types of personnel expenditures that agencies should consider as attributable to implementing the Kingpin Act. As a result, agencies must interpret for themselves what to include in their estimated personnel expenditure submissions to OFAC. The memo listed some examples of what agencies could include as resource expenditures related to the implementation of the Kingpin Act, such as research materials and information access, travel, equipment, supplies, outside services, and security. OFAC officials said they are not more prescriptive with their guidance because the Kingpin Act is not specific about which expenses to report. Agencies reported different methods for determining expenditure amounts and the information on agency personnel expenditures varied substantially from year to year. Officials from some of the partner agencies reported calculating estimates of personnel expenditures based on the paygrades of personnel engaged in Kingpin Act investigations or interagency meetings, while others stated that they did not report expenditures because they determined that their level of engagement was minor and did not warrant reporting. According to DEA officials, they do not report on personnel expenditures for the time they spend investigating Kingpin Act targets because the investigations simultaneously support their own cases against the targets. According to officials from some agencies, such as DHS, they only reported personnel expenditures for cases on which they were the lead investigative agency. As a result, DHS reported $2.4 million in personnel expenditures in fiscal year 2015, $0 between fiscal years 2016 and 2018, and then about $2 million in fiscal year 2019. Other partner agencies, such as DOD and State, report personnel expenditures even though they do not lead specific Kingpin Act investigations. As a result, the reported expenditures of agencies may not be consistent and may not represent a reliable total for Kingpin Act activities across the U.S. government. See figure 5 for the Kingpin Act–related personnel expenditures reported by Treasury and its partner agencies. Agencies’ determinations of what they include as resource expenditures vary as well. For example, several agencies have reported no resource expenditures for the last 3 years, but State has reported a small resource expenditure that, according to State officials, accounts for transportation expenses for Kingpin Act interagency meetings. OFAC officials said they do not know what agencies are including in their annual expenditure reports because OFAC does not seek information from agencies explaining their annual expenditure submissions and OFAC reports them as submitted. Moreover, OFAC officials said they did not verify the amounts reported to confirm, for example, why DHS and DOD reported personnel expenditures in some years many times greater than DOJ personnel expenditures, even though DOJ is the lead investigative agency for the majority of Kingpin Act cases. The Kingpin Act requires OFAC to report on the personnel and resources expended on the imposition of Kingpin Act sanctions each year. Additionally, federal internal control standards require entities to ensure that they are using quality information to achieve their objectives. Although Treasury reported in the most recent annual report from July 2019 that OFAC’s significant increase in resource expenditures was due to the addition of overseas costs and database contracts, the annual reports do not account for significant changes in agencies’ expenditures from year to year. Because OFAC does not provide guidance that clarifies what agencies are required to include in their annual expenditure submissions or disclose the limitations in the consistency and reliability of expenditure data from partner agencies, OFAC cannot provide assurance that its annual reports to Congress on Kingpin Act interagency expenditures contain quality information that is transparent and consistent across all reporting agencies. As a result, Congress may not be able to provide informed oversight of personnel and resources expended on implementing the Kingpin Act. OFAC and partner agency officials identified challenges that make it difficult—or impossible—to assess the overall effectiveness of the Kingpin Act sanctions in achieving U.S. policy goals to reduce illicit narcotics within the United States. These officials noted that the primary challenge in assessing the effectiveness of Kingpin Act sanctions is that they cannot isolate the impact of Kingpin Act sanctions from those of multiple other efforts and factors. For example, whether the estimates of the amount of drugs entering the United States is increasing or decreasing depends upon the sum total of activities of counternarcotics programs managed by organizations in the United States, other countries, and the international community. In addition, we have previously reported other challenges that agency officials have stated can make it difficult to assess the effectiveness of economic sanctions, including frequent shifts in policy goals and objectives, and a lack of reliable data. Treasury officials noted that sanctions are often used in conjunction with other policy tools, such as diplomatic engagement and export controls. According to Treasury officials, distinguishing the impact of each tool leveraged is exceedingly difficult due to the limited information available via intelligence and law enforcement channels. Moreover, while Treasury’s partner agencies said Kingpin Act designations contribute to their counternarcotics goals, these agencies’ are unable to quantify contributions specifically related to the Kingpin Act in measuring progress toward their own agencies’ goals. Partner agency officials said they do not consider the Kingpin Act to be a government program for which effectiveness can be assessed; rather, they stated that the Kingpin Act is one tool among many that U.S. government agencies can use where appropriate in their efforts to combat drug trafficking. According to partner agency officials, effectiveness of sanctions in achieving policy goals is often discussed at an interagency level, which allows the U.S. government to consider these issues in the larger policy context, because sanctions are often only one element of broader government-wide strategies to achieve U.S. policy goals. Treasury conducts some assessment of both the potential and observed impacts of specific Kingpin Act designations. The Office of Intelligence and Analysis (OIA), Treasury’s intelligence component, conducts both predesignation and postdesignation assessments. OIA officials noted that they consider it part of their mission to inform Treasury policymakers of potential impact before a designation occurs. According to OIA officials, OIA’s predesignation assessments are narrowly focused and can be delivered in any number of formats, including emails, spreadsheets, and briefings. OFAC officials said they provide OIA with name and summary evidentiary information on a potential target. According to OIA officials, they use the information to assess the potential level of impact (e.g., negligible or significant) a Kingpin Act designation may have on the target, its network, or other third parties, based on a variety of factors. For example, OIA may determine that a Kingpin Act designation can result in significant impact if evidence indicates that a designation will impose high costs and obstacles for a target to continue drug trafficking activity. According to OIA officials, such assessments have been required by the Under Secretary since 2018 and OIA has completed predesignation assessments on all Kingpin Act designations during that time period for senior Treasury officials’ consideration. Additionally, since 2018, OIA has completed two postdesignation assessments. OIA officials said they share these assessments with OFAC so it can incorporate the lessons learned into future investigations or to develop new designations. OIA officials said that the decision to conduct postdesignation assessments of Kingpin Act designations is based on resources and the availability of information to assess impact. OFAC officials said they have not undertaken formal, systematic assessments on the impact of Kingpin Act designations because OFAC’s staffing resources are primarily assigned to designation investigations and reviewing of petitions for Kingpin Act designation reconsideration. OFAC and its U.S. partner agencies reported on various results related to Kingpin Act sanctions. OFAC reported that it had designated more than 2,000 individuals and entities under the Kingpin Act as of June 2019. (See fig. 6 for the number of individuals and entities designated by year.) These designations are about evenly split between designations of individuals and designations of entities across the four designation classifications. OFAC reported 195 Tier 1 designations (B1 and B4 classification), and 2,033 Tier 2 designations (B2 and B3 classification). OFAC also reported that it has frozen more than half a billion dollars of sanctioned individuals’ or entities’ assets under the Kingpin Act between 2000 and 2019. According to OFAC data, almost 80 percent of the total assets frozen were from one individual in 2017. For the remaining years, the amount frozen fluctuated between $1.7 million and $36.4 million without a clear upward or downward trend. Further, law enforcement partner agencies cited the Kingpin Act as an important tool in aiding their investigations that may result in actions such as indictment or arrest of designees. For example, in one of our nine cases, a federal grand jury indicted Raul Flores Hernandez—the suspected leader of a Guadalajara-based drug trafficking organization—in August 2017 for moving large quantities of cocaine from South America to Mexico for distribution and further transportation into the United States. OFAC designated him (as well as 21 of his alleged criminal associates and 42 businesses and other entities affiliated with his drug trafficking organization) under the Kingpin Act concurrent with the indictment. According to OFAC and DEA officials, sharing information about Flores Hernandez was essential to both the designations and the indictment. According to these officials, disrupting the access of significant narcotics traffickers and their networks to the U.S. financial system and barring them from travel to the United States has been helpful in motivating several designees to cooperate with law enforcement investigations. Moreover, U.S. agencies report that the ability to sanction entire drug trafficking networks increases pressure on traffickers to cease involvement with illicit narcotics. OFAC officials stated that removing designees from the OFAC list is, in some cases, evidence of disruption of drug trafficking organization or other positive behavior change. To be removed, designees must petition OFAC and demonstrate that they no longer meet the criteria to be designated under the Kingpin Act. As of June 2019, OFAC had removed 399 individuals and entities previously designated under the Kingpin Act, of which five were Tier 1 designations (B1 and B4 classification), and 394 were Tier 2 designations (B2 or B3 classification). Foreign government officials also reported that Kingpin Act sanctions have assisted them in imposing penalties on drug traffickers. Foreign government officials we met with in Colombia reported that their Supreme Court issued a ruling that permits their countries’ banks to terminate accounts of, and deny service to, Kingpin Act designees because of the risk the banks would face if they continued those business relationships. According to Mexican government officials, a bankers’ association, the Financial Intelligence Unit, and the bank regulator in Mexico issued guidance supporting Mexican banks’ rights to deny service to Kingpin Act designees. Mexican government officials also stated that once the United States publicly identifies a Mexican national as a drug trafficker by designating him or her under the Kingpin Act, Mexican law enforcement entities face less public opposition when they arrest, imprison, or extradite the individual. According to government officials we met with in Colombia, information that OFAC and other U.S. agencies share as part of their Kingpin Act investigations help them justify seizing designees’ assets. For example, according to OFAC officials, OFAC, DEA, and Colombian authorities led a joint investigation that led to the October 2018 Colombian asset seizure of 202 assets of two individuals in Colombia valued at over USD $500 million. The Colombian seizure included farms, land, houses, hotels, apartments, businesses, commercial properties, emerald mines, horses and vehicles. Some Kingpin Act designations have had unintended consequences for foreign persons other than those targeted by the sanctions. The Congressional Research Service has reported that some designations have been associated with significant economic losses and unemployment by individuals not involved in illicit narcotics when large companies are liquidated in the process. Treasury officials stated that foreign drug trafficking organizations often attempt to integrate their illicit proceeds into the legitimate economy by owning or controlling businesses that may employ individuals who are not associated with drug trafficking activities. According to Treasury officials, it is imperative that Treasury designate businesses that are owned or controlled by drug trafficking organizations, despite the employment of individuals who may not have knowledge of the illicit activities. They said that prior to designating such foreign businesses, Treasury coordinates closely with other U.S. government agencies, the relevant U.S. embassy, and with the relevant foreign counterparts to minimize the impact on employees who lack knowledge of the illicit activities. According to the Congressional Research Service, some designations have also been associated with upticks in drug trafficking–related violence when, in combination with law enforcement action, drug trafficking organizations are dismantled and competing groups vie for abandoned territory. Furthermore, some designations have negatively affected public perceptions of the United States within the designee’s country of residence, according to OFAC and partner agency officials. For example, OFAC and State officials stated that there was significant public criticism of U.S. intervention when OFAC designated a Mexican celebrity in conjunction with a significant narcotics trafficker. OFAC officials said it can be difficult to address public opposition to a Kingpin Act designation because the information in the evidentiary package is sensitive and cannot be revealed publicly. The Kingpin Act enables the U.S. government to sanction significant international narcotics traffickers and their networks worldwide by designating foreign individuals and entities, resulting in the freezing of their U.S. assets and an inability to conduct transactions, including financial transactions, with U.S. businesses. OFAC and its partner agencies consider the Kingpin Act a valuable tool as part of U.S. counternarcotics strategy, but have noted that the plethora of counternarcotics efforts make it difficult to isolate the effects of the Kingpin Act. OFAC has reported on personnel and resources directed toward imposing Kingpin Act sanctions annually to Congress. However, OFAC provided limited guidance to agencies about what expenditure data to report. As such, we observed considerable inconsistencies in resource expenditures reported by various partner agencies, and also determined that methods for determining expenditures varied by agency. Moreover, OFAC does not disclose agency data limitations, such as explaining why the data may vary from year to year, before reporting the information to Congress. Without consistent agency data and disclosure of data limitations regarding information on agency resources devoted to Kingpin Act activities, Congress may be limited in its ability to conduct oversight of implementation of the Kingpin Act. We are making the following 2 recommendations to the Department of the Treasury: The Secretary of the Treasury should ensure that the Office of Foreign Assets Control provides its partner agencies more specific guidance regarding Kingpin Act–related expenditure data to improve the consistency of data submitted by these agencies. This could include, for example, how agencies account for expenditures that support Kingpin Act investigations when they are not the lead and for what types of activities resource expenditure data are required. (Recommendation 1) The Secretary of the Treasury should ensure that the Office of Foreign Assets Control discloses information about limitations in the consistency and reliability of the agency expenditure data in its annual reports to Congress. (Recommendation 2) We provided a draft of this report to Treasury, DHS, State, DOD, DOJ, CIA, the Federal Reserve, and ONDCP for comment. We received technical comments from Treasury, DHS, and the Federal Reserve, which we incorporated as appropriate. The remaining agencies informed us that they had no comments. Treasury did not agree or disagree with our recommendations. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of the Treasury, the Acting Secretary of Homeland Security, the Secretary of State, the Secretary of Defense, the Assistant Attorney General for Administration, the Director of the Central Intelligence Agency, the Chair of the Board of Governors of the Federal Reserve System, and the Deputy Director of the Office of National Drug Control Policy. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2964 or GurkinC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Our objectives were to examine (1) how U.S. agencies designate individuals and entities under the Foreign Narcotics Kingpin Designation Act (Kingpin Act); (2) the extent to which U.S. agencies monitor, enforce, and report on sanctions under the Kingpin Act; and (3) what agencies have done to assess the effectiveness of the Kingpin Act. To examine the process for designating individuals and entities under the Kingpin Act, we interviewed officials from the mandated partner agencies—the Departments of the Treasury (Treasury), State, Homeland Security (DHS), and Defense; the Federal Bureau of Investigation (FBI); and the Drug Enforcement Administration (DEA)—and reviewed documents, including the statutes that comprise the Kingpin Act. We also reviewed documentation on collaboration and information-sharing agreements between Treasury’s Office of Foreign Assets Control (OFAC) and its partner agencies to determine the ways in which agency participation has been formalized in the designation process. In addition, we received responses from the Central Intelligence Agency to questions we sent. We selected and reviewed a nongeneralizable sample of Kingpin Act designations made since 2015 to understand OFAC’s designation process and the extent of the variation in the timing and sequence of the steps leading to the designations. From the countries with the most designations, we considered only B1 and B4 designations. Additionally, we only considered designations that occurred after May 2015, when the authority to designate was delegated from the President to the Secretary of the Treasury, so that the process followed for designation would most closely resemble the current process. We considered cases with a range in number of B2 and B3 designations affiliated with the B1 or B4 designee. Furthermore, we ensured that we selected cases from both Western Hemisphere countries where most of the designations have occurred (including cases in Colombia and Mexico where we performed fieldwork), and non–Western Hemisphere countries to learn whether the process differs geographically. To account for those criteria, we selected nine cases to review. OFAC provided data on the milestone dates and results associated with the cases from its electronic case management system. We were unable to independently assess the data provided against the system, but we were able to corroborate some dates, such as designation dates, with public documents such as press releases. In addition, OFAC officials answered our questions about the variance in case data they provided by explaining factors that contributed to the length or sequence of investigative steps of each case. As a result, we deem the case study data provided by OFAC to be sufficiently reliable for the purposes of this report. To examine the extent to which U.S. agencies monitor, enforce, and report on Kingpin Act sanctions, we interviewed officials from OFAC and its partner agencies regarding their roles in sanctions implementation. We also interviewed officials from some of the Federal Banking Agencies (FBA) that OFAC officials said had responsibilities to help monitor bank programs for compliance with OFAC sanctions, including the Kingpin Act financial sanctions—the Office of the Comptroller of the Currency and the Board of Governors of the Federal Reserve System—and five U.S. banks recommended by the FBAs as having the largest presence in Latin American countries to assess implementation of economic sanctions, including any penalties incurred for sanctions violations. In addition, we met with officials from financial regulator agencies and the national banking associations in Colombia and Mexico to understand how U.S. enforcement of Kingpin Act sanctions affected their operations. To assess the extent to which OFAC included information required by the Kingpin Act for Treasury’s reports to Congress, we also reviewed the annual reports OFAC submitted to Congress from 2003 to 2019 and OFAC guidance sent to partner agencies from 2017 through 2019 seeking their input into the reports. We interviewed officials from each partner agency about the methodology they used to calculate their annual resource and personnel expenditures. Because we found that the agencies calculate their personnel expenditures differently and OFAC does not verify the amounts reported, we did not find the data reliable and are presenting the data to illustrate the problems with their reliability. To examine what agencies have done to assess the effectiveness of the Kingpin Act, we interviewed OFAC and U.S. partner agency officials in Washington, D.C., Colombia, and Mexico, regarding their efforts to assess effectiveness and results of Kingpin Act designations and any challenges in measuring effectiveness. We also held telephone interviews with U.S. partner agency officials in Panama. We interviewed OFAC and Office of Intelligence and Analysis (OIA) officials regarding the type of assessments being done on the Kingpin Act. We reviewed strategic planning documents from the partner agencies to identify their counternarcotics objectives and, if available, related performance measures they track. We also used information from the nine designation cases we selected and interviewed U.S. partner agency officials as well as host government, financial industry, international organization, and nonprofit officials in Colombia and Mexico, to get perspectives on the results of Kingpin Act designations. To report on designations and removals of the Kingpin Act, we used OFAC’s official brochure detailing the complete listing of Kingpin designations as of June 11, 2019. The data in the brochure are taken from OFAC’s Specially Designated Nationals and Blocked Persons (SDN) List. The data include designations from years 2000-2019 categorized by type of designation. We compared the data in the brochure against the SDN List for accuracy and asked OFAC officials about their efforts to ensure the reliability of the SDN List data. We determined the data were sufficiently reliable for the purposes of our report. To report the amount of foreign designees’ assets frozen, we collected available data from OFAC for calendar years 2008-2018. To obtain the types and number of U.S. assets that have been blocked under the Kingpin Act, we interviewed OFAC officials and reviewed their published data. It was beyond the scope of this engagement to independently verify the number of U.S. assets blocked. We conducted this performance audit from May 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Jennifer Grover, Mona Sehgal (Assistant Director), Jeffrey Baldwin-Bott (Analyst in Charge), Travis Cady, Marisela Perez, and Barbara Shields made key contributions to this report. Ashley Alley, Martin De Alteriis, Neil J. Doherty, Toni Gillich, Jeff Harner, John Hussey, and Triana McNeil also contributed to this report.", "summary": "Drug deaths in the United States have been rising for years. According to the Centers for Disease Control and Prevention, in 2017 there were over 70,000 U.S. drug overdose deaths. This national emergency results in part from the activities of international narcotics traffickers and their organizations. The Kingpin Act, enacted in 1999, allows Treasury to designate and sanction individuals and entities that contribute to illicit narcotics trafficking. Sanctions and other consequences include blocking a designee's property and assets, denying U.S. travel visas to designees, and penalizing U.S. persons who violate the prohibitions in the Kingpin Act. Treasury is required to submit an annual report to Congress on agencies' Kingpin Act–related personnel and resource expenditures and sanctions activities. This report examines (1) how U.S. agencies designate individuals and entities under the Kingpin Act; (2) the extent to which U.S. agencies monitor, enforce, and report on sanctions under the Kingpin Act; and (3) what agencies have done to assess the effectiveness of the Kingpin Act. GAO reviewed documents from and interviewed officials at Treasury, the Department of State, and other partner agencies. GAO also performed fieldwork in Colombia and Mexico. Under the Foreign Narcotics Kingpin Designation Act (Kingpin Act), the Department of the Treasury's (Treasury) Office of Foreign Assets Control (OFAC) leads a flexible interagency process to designate and sanction foreign individuals and entities that contribute to illicit narcotics trafficking. OFAC identifies potential Kingpin Act designees, compiles evidence, submits it for legal review, and seeks concurrence from partner agencies on designation decisions. OFAC and U.S. partner agencies monitor and enforce Kingpin Act sanctions, but OFAC has not ensured consistency and transparency of the expenditure data it has reported to Congress. Federal Banking Agencies monitor the OFAC compliance programs of U.S. banks through regular bank examinations. Additionally, OFAC handles enforcement through warnings, monetary penalties, and other methods. As required, OFAC reports annually to Congress on Kingpin Act designations and corresponding agency expenditures, but it has provided limited guidance to partner agencies on expenditure data they report. As a result, agencies use different methods to calculate the personnel and resource costs associated with their Kingpin activities. For example, the Department of Homeland Security said it only reports personnel expenditures when it is the lead investigative agency, but the Department of Defense reports personnel expenditures when it is not the lead. Furthermore, OFAC has not reported the limitations in agency data in its congressional reports. This lack of clear expenditure information could hinder oversight of the Kingpin Act. OFAC officials noted challenges to assessing the overall effectiveness of the Kingpin Act, but they and their U.S. and international partners track and report a range of results. The primary challenge cited is the difficulty of isolating the effect of the Kingpin Act from multiple other programs combating drug trafficking organizations. Results reported by OFAC and its partners include, for example, from 2000-2019, OFAC reported that it had designated more than 2,000 Kingpins and their supporters, and frozen more than half a billion dollars in assets under the act. In addition, host government officials reported that Kingpin Act sanctions assist them in imposing penalties on drug traffickers. GAO is recommending that Treasury ensure that OFAC (1) improve guidance to partner agencies on their Kingpin Act–related expenditures and (2) disclose expenditure data limitations in its annual Kingpin Act reports to Congress. Treasury did not agree or disagree with the recommendations.", "document_type": "gao"}
{"report": "USPS has the mission of providing prompt, reliable, and efficient universal postal service, and federal law requires USPS to “provide postal services to bind the Nation together through the personal, educational, literary, and business correspondence of the people.” USPS is required to serve, as nearly as practicable, the entire population of the United States. USPS has a number of key stakeholders, each with different interests in USPS and its operations (see fig. 1). USPS is a key part of the mailing industry, and over time, it has become both a competitor and partner to private companies that also operate in the broader mail and delivery industries. For example, although United Parcel Service (UPS) and FedEx both pay USPS to deliver packages that they enter into USPS’s system at local post offices where carriers pick up their mail, they also compete with USPS for end-to-end package delivery business, such as moving packages from the retailer to the purchaser. Similarly, FedEx is USPS’s largest contractor, providing air transportation for Priority Mail Express (formerly Express Mail), Priority Mail, and First-Class Mail. UPS is also one of USPS’s largest contractors, providing long-distance mail transportation. Over the years, legislation has changed key aspects of the business model used to provide the nation’s postal services. Until 1970, the federal government provided postal services via the U.S. Post Office Department, a government agency that received annual appropriations from Congress. At that time, Congress was involved in many aspects of the department’s operations, such as selecting postmasters and setting postal rates and wages. In addition, the President controlled the hiring and firing of Postmasters General, as it was a cabinet-level position. By the late 1960s, the department had several major problems including financial losses, management problems, service breakdowns, and low productivity. Because key postal business decisions were made by Congress through the legislative process, postal management had limited ability to plan and finance department operations and capital investments in accordance with postal needs. In order to improve and modernize postal services, the Postal Reorganization Act (PRA) was enacted in 1970 and replaced the U.S. Post Office Department with USPS, an independent establishment of the executive branch of the government of the United States. Congress designed USPS to be a self-sustaining, business-like entity headed by a Board of Governors that would cover its operating costs primarily with revenues generated through the sales of postage and postal-related products and services. However, by the early 2000s, USPS faced a bleak financial outlook that put its mission of providing universal postal service at risk, according to the 2003 Presidential Commission on the United States Postal Service. The Commission evaluated USPS’s business model and concluded that USPS must have greater flexibility to operate in a business-like fashion, but that this latitude required enhanced transparency to enable effective management and congressional oversight. The Postal Accountability and Enhancement Act (PAEA) was enacted in 2006. PAEA provided USPS additional pricing flexibility for mail products, but with provisions for increased transparency, oversight, and accountability, among other things. Specifically, PAEA gave USPS broader latitude to change postal rates in a more streamlined process that included review by the newly created Postal Regulatory Commission (PRC). The PRC, which replaced the former Postal Rate Commission, is an independent establishment of the executive branch responsible for regulating USPS. PRC is required to make annual determinations of USPS’s compliance with mail delivery standards and postal rate requirements. If PRC finds noncompliance, it is required to specify USPS actions to restore compliance. USPS’s current business model is not financially sustainable due to declining mail volumes, increased compensation and benefits costs, and increased unfunded liabilities and debt. USPS’s costs continue to rise faster than its revenues, and although USPS has made changes over the years to address these challenges, its efforts have been limited by stakeholder opposition and statutory requirements. As online communication and payments have expanded, USPS continues to face decreases in mail volume, its primary revenue source. First-Class Mail volume has declined 44 percent since fiscal year 2006, the year that total mail volume peaked. The long-term decline of First-Class Mail volume, which USPS has stated was exacerbated by the Great Recession and expects to continue for the foreseeable future, has fundamental implications for USPS’s business model because First-Class Mail is USPS’s most profitable class of mail. USPS Marketing Mail— which comprises most other mail volume—declined 27 percent from fiscal year 2007 to fiscal year 2019, in part due to electronic advertising alternatives. The volume of USPS competitive products more than tripled since fiscal year 2007. This volume, however, began to decline in the second half of fiscal year 2019 due to growing competition for package delivery. USPS has taken steps to right size its operations in response to declining mail volumes. For example, in both 2009 and 2011, USPS announced plans to close several thousand USPS retail facilities. However, due to stakeholder opposition—including from members of Congress, postal unions, and local communities, among others—USPS instead closed a few hundred retail facilities. USPS also expanded the alternative options for customers to access retail postal products and services outside of USPS-operated postal facilities—such as self-service kiosks and partnerships with other retailers such as contract postal units. According to USPS, as a compromise effort to right size the retail network and due in part to USPS’s efforts to expand retail alternatives, USPS began reducing retail hours at selected post offices in 2012, ultimately decreasing retail hours at approximately 13,000 post offices. Another major cost-cutting effort was its 2011 Network Rationalization Initiative, a multi-part plan to consolidate its mail processing network. USPS consolidated more than 160 mail processing facilities, but did not fully implement this initiative following opposition from various stakeholders. In addition to stakeholder opposition to changing postal services, federal laws also factor into USPS’s limited ability to respond to declining mail volumes. For example, federal laws define the level of postal services USPS is to provide, postal products, and pricing. Postal services to be provided: USPS has limited ability to make changes in the postal services it provides. Specifically, USPS is required to provide 6-days-a-week delivery and to operate postal facilities across the country. Federal law requires USPS to provide the maximum degree of effective and regular postal services to rural areas, communities, and small towns where post offices are not self- sustaining. Federal law also limits USPS’s ability to close retail facilities. For example, USPS cannot close a small post office solely because it is unprofitable. As of the end of fiscal year 2019, there were approximately 34,600 postal retail outlets nationwide, including approximately 31,300 USPS-managed post offices, branches, and stations, and, as we recently reported, USPS’s analysis showed that about 36 percent of its retail facilities were unprofitable in fiscal year 2018. Postal products and pricing: USPS’s pricing flexibility is limited by a price cap on market-dominant products that generally limits rate increases for these products to a common measure of inflation. Each competitive product is required to cover its attributable costs; competitive products collectively are required to recover their attributable costs; and competitive products collectively are required to cover a PRC-specified minimum of USPS’s institutional costs. In addition, USPS is prohibited from providing new nonpostal products and services. Such requirements affect USPS’s ability to increase revenues. While mail volumes have decreased, USPS’s compensation and benefits costs for current employees have been increasing since 2014, despite USPS’s efforts to control these costs. Although USPS reduced its total workforce (career and non-career employees) from 785,900 in fiscal year 2007, to 617,700 in fiscal year 2013, its workforce increased to about 630,000 in fiscal year 2019. Similarly, as we previously reported, recent trends show total work hours increased from a combination of new hiring and increased work hours for current employees. Specifically, we reported that from fiscal years 2014 through 2018, work hours increased by 5.4 percent. The number of work hours associated with higher costs— overtime and penalty overtime—have also been increasing. According to USPS, total compensation and benefits costs increased by almost $1 billion in fiscal year 2019 alone. USPS has implemented changes to help control employee compensation and benefits costs, including lowering pay for new career employees and increasing use of non-career employees. For example, as we previously reported, starting about 10 years ago, USPS’s collective bargaining agreements have included the ability to hire up to 20 percent of the workforce as non-career employees. Non-career employees are less costly because they generally have lower pay rates and are not entitled to the full federal benefits received by career employees. According to USPS officials, non-career employees are also “more flexible” because there are fewer restrictions on their tasks and schedules. We recently reported that our analysis estimated that USPS likely saved about $6.6 billion from fiscal years 2016 through 2018 from increased use of non- career employees. USPS has recognized trade-offs in increasing the use of non-career employees in entry-level positions, such as a high turnover rate, as would be expected for almost any entry-level position in the private sector. With respect to benefits costs for current postal employees, we have recently reported that USPS has also achieved savings by gradually decreasing its contribution percentage for employee health insurance premiums over the past decade, with corresponding increases in the contribution percentage paid by employees. These changes were negotiated with the four major postal labor unions and were included in successive collective bargaining agreements, each of which covered a multi-year period. We found that the reduction in USPS health insurance contributions generated estimated savings of about $1.4 billion for fiscal years 2016 through 2018. A number of restrictions limit USPS’s ability to control employee compensation and benefits costs. As we recently reported, USPS compensation and benefits costs—which represent about three-fourths of its total costs—are driven by a mix of USPS contracts and policies, including collective bargaining agreements negotiated with unions representing 92 percent of USPS employees, and statutory requirements governing USPS employee pay and benefits. When USPS and its unions are unable to agree, the parties are required to enter into binding arbitration by a third-party panel. USPS’s collective bargaining agreements with these labor unions, some of which were established through binding arbitration, have established salary increases and cost- of-living adjustments and, as mentioned above, have also capped the number of non-career employees at approximately 20 percent of the number of employees covered by the agreements. Federal law requires USPS to participate in the Federal Employees Health Benefits Program (FEHBP), which covers current employees and retirees, as well as federal pension and workers’ compensation programs. Further, USPS must provide fringe benefits that, as a whole, are no less favorable than those in effect when the Postal Reorganization Act of 1970 was enacted. USPS’s unfunded liabilities and debt, which consist mostly of unfunded liabilities for retiree health and pension benefits, have become a significant financial burden, increasing from 99 percent of USPS’s annual revenues at the end of fiscal year 2007 to 226 percent of its fiscal year 2019 revenues. At the end of fiscal year 2019, USPS’s unfunded liabilities and debt totaled approximately $161 billion. However, it has begun paying down this debt in recent years, leaving a balance of $11 billion at the end of fiscal year 2019 (see fig. 2). Total unfunded liabilities have risen in part due to USPS not making payments to fund its retiree health and pension benefits. USPS has stated that it prioritizes its “primary universal service mission” when it is unable to fulfill all of its financial obligations, and that it therefore did not make payments to fund its postal retiree health benefits and pensions to minimize the risk of running out of cash. In doing so, USPS cited its precarious financial condition and the need to cover current and anticipated costs and any contingencies. It has not paid $55.4 billion in required payments for funding these benefits through fiscal year 2019, including $47.2 billion in missed funding payments for retiree health benefits since fiscal year 2010, and $8.2 billion for funding pension benefits since fiscal year 2014. In addition, for many years, USPS had been at its statutory debt limit of $15 billion; however, it has begun paying down this debt in recent years, leaving a balance of $11 billion at the end of fiscal year 2019. A number of federal laws define the requirements for USPS’s retiree health and pension benefits that comprise most of its unfunded liabilities. Retiree health benefits: Federal law establishes certain requirements for postal retiree health benefits, including basic requirements for coverage eligibility and contributions. In administering the FEHBP, the Office of Personnel Management (OPM) negotiates with the insurance providers to establish the level of benefits provided to beneficiaries. USPS is required to prefund its share of health benefits for its retirees. Under PAEA, the first 10 years of prefunding payments were fixed—ranging from $5.4 billion to $5.8 billion annually from fiscal years 2007 to 2016. From fiscal years 2007 through 2016, USPS was also required to continue “pay-as-you-go” payments for its share of premiums for current retirees. The permanent schedule for USPS payments to prefund postal retiree health benefits under PAEA required USPS to make annual payments starting in fiscal year 2017. Currently, USPS no longer makes payments for retiree health benefits premiums. Starting in 2016, these premiums are paid out of the RHB Fund until it is depleted, whereupon USPS will resume paying premiums on a pay-as-you-go basis. As we previously reported, survey data we reviewed indicated that most companies do not offer retiree health benefits and that the number of companies providing such benefits is decreasing over time. Many companies that have retained their retiree health benefits have done so by making changes to control costs, including tightening eligibility and restructuring benefits. However, all approaches we identified have different potential effects and would require congressional action. Pension benefits: Federal law also requires USPS to finance its pension benefits under the Federal Employees Retirement System (FERS) and the Civil Service Retirement System (CSRS) and contains specific provisions defining USPS’s required contribution level to fund these benefits. USPS’s payments consist of a rolling 30-year amortization schedule to address unfunded FERS liabilities, an amortization schedule to address unfunded CSRS liabilities by 2043, and the normal costs of FERS benefits for current employees. The large domestic companies we selected in the airline, auto, and railroad industries took actions over a number of years to address major business challenges. Airlines such as Delta, American, and United faced competition from low-cost airlines, downward pressure on airfares, and rising compensation, benefits, and volatile fuel costs. These challenges were exacerbated by the economic downturn that began in 2000, the terrorist attacks of September 11, 2001, and the Great Recession that began in December 2007, all of which temporarily depressed demand for airline travel. Similarly, General Motors (GM) and Ford Motor Company (Ford) faced competition from lower cost competitors, the Great Recession, a workforce and networks too large to be supported by smaller sales volumes, and other changes in the market. Likewise, large U.S. railroads competed for freight and passengers from other transportation modes, such as the trucking and airline industries that operated over publicly provided infrastructure, while railroads had to invest in their own infrastructure. Based on our review of the selected companies’ annual reports and statements to the SEC as well as selected federal laws, and GAO, CRS, and other organizations’ reports, books, and academic articles, among other sources, we found that selected companies made changes to (1) products and services, (2) financial self-sustainment, and (3) use of the bankruptcy process. While some of the selected businesses restructured through a bankruptcy proceeding, other businesses took similar actions outside of the bankruptcy process. Mergers also played an important role for the airlines and railroads. The selected companies made multiple changes to their products and services. Specifically: Airlines: Selected airlines altered pricing by changing route structure to focus on more profitable routes and adding fees, such as for checked baggage. In addition, all three selected airlines merged with other major airlines, thereby broadening their routes and revenues. Automakers: Selected automakers focused on producing more profitable brands and models, discontinuing some models and introducing others. For example, during its financial difficulties about a decade ago, GM discontinued a number of unprofitable brands. In 2018, after years of declining car sales, Ford said it would eliminate some of its most well-known cars in North America, allowing it to devote more resources to sport utility vehicles and trucks. Railroads: Large railroads focused on more profitable routes and abandoned unprofitable routes or sold them to other railroads. For example, the federal government created a new freight railroad, Conrail, by merging several bankrupt railroads in the Northeast and Midwest. As we have reported previously, federal government deregulation of railroad pricing and contracts after 1980 also helped Conrail to reach profitability and increase capital investment. Cost reduction was a major theme for the selected businesses in the airline, automotive, and rail industries, particularly with respect to compensation, benefits, and infrastructure costs. Specifically: Airlines: The three selected airlines negotiated wage cuts and work rule changes with their unions; made workforce reductions, in part by outsourcing work; and cut pension and retiree health benefit programs. Wage cuts included all levels of employees, such as management, pilots, flight attendants, and mechanics. Benefit cuts involved reducing the level of pensions and retiree health benefits and transitioning pension programs from defined benefits plans to defined contribution plans that were structured to be less costly. Airlines also reduced infrastructure costs by eliminating some hubs, reducing the total number of aircraft, and changing the mix of aircraft in their fleet to save on maintenance and fuel costs. The airlines further cut costs by restructuring debt, reducing facility leasing costs, and renegotiating aircraft leases and vendor contracts. While in bankruptcy, the airlines took major actions to reduce their costs. For example, United implemented steep pay cuts, cut retiree health benefits, and terminated its defined benefit pension plans, resulting in the Pension Benefit Guaranty Corporation (PBGC) assuming responsibility for some of its pension payments, and a reduction in benefits for the plan’s participants. United also cut its workforce size by 31 percent, reduced the number of airplanes by 19 percent, and reduced the total number of flights by 13 percent. Delta and American also reduced pay and pension benefits while in bankruptcy, and the PBGC assumed responsibility for some of Delta’s pension liabilities. Automakers: The two selected automakers negotiated pay cuts, lower wages for entry-level employees, and changes to work rules designed to increase competitiveness; cut the workforce size in about half; made changes to employee benefits; closed many auto plants and dealerships; eliminated some vehicle brands and models; and changed the production process to increase efficiency. Specifically, The International Union, United Automobile, Aerospace and Agricultural Implement Workers of America (UAW) agreed to cuts in compensation for the automakers’ employees to levels paid by GM and Ford’s competitors. UAW also agreed to move retiree health care benefits into a private Voluntary Employee Beneficiary Association (VEBA) for current and former UAW-represented employees in 2007; the VEBA took over health benefits for retirees starting in 2010. The automakers also closed defined benefit pension plans to new participants and moved to defined contribution pension plans for eligible new employees. Railroads: The Staggers Rail Act of 1980 was enacted to improve the financial stability of the railroad system. Subsequently, railroads improved their financial health through, among other things, cost reduction measures such as reducing their workforce through layoffs and federal buyouts agreed to by unions, and abandoned or sold off unprofitable rail lines to reduce infrastructure and operating costs. Overall railroad employment fell greatly as railroads consolidated, reduced service, and changed work rules. For example, Conrail reduced its workforce from about 82,000 employees in 1977 to about 21,000 in 1996. Congress specifically facilitated Conrail’s downsizing by passing the Northeast Rail Service Act of 1981, which among other things, directed a $200 million a year reduction in labor costs and authorized Conrail to terminate employees. The U.S. government provided assistance to selected companies in various forms, including appropriated funds, loans, and other actions that helped enable companies to reduce their expenses. For example: Automakers: The federal government committed $49.5 billion in funding to help GM continue to operate while restructuring. After the government became the majority owner of the GM that emerged from bankruptcy, the Administration laid out core principles that included managing its ownership in a hands-off manner and voting as a shareholder only on core governance issues. Airlines: Under the 2001 Air Transportation Safety and System Stabilization Act, the federal government provided nearly $5 billion in compensation to airlines for losses due to the September 11, 2001 terrorist attacks, including $856 million for American, $668 million for Delta, and $782 million for United. Additionally, under the Emergency Wartime Supplemental Appropriations Act, $2.4 billion was appropriated to the Transportation Security Administration to compensate airlines for certain security expenses and fees, including $358 million to American, $411 million to Delta, and $300 million to United. Railroads: The federal government spent about $8 billion creating, subsidizing, and preparing Conrail for sale to the private sector. This funding included $7 billion through 1988 for purchasing properties of bankrupt railroads, operating subsidies, and capital improvements and employee buyouts. Amtrak, the national passenger railroad, took over money-losing intercity and commuter passenger rail services and funded federal payments of up to $25,000 for each laid-off employee as authorized by legislation. Some airlines, automakers, and railroads made changes through the bankruptcy process. Specifically: Airlines: All three selected airlines went through bankruptcy proceedings—United in December 2002, Delta in September 2005, and American in November 2011. The actions these airlines took to reduce costs while in bankruptcy are discussed above. Automakers: GM declared bankruptcy in June 2009 to implement its restructuring plan. The federal government became GM’s majority shareholder and continued to provide financial assistance while GM was in bankruptcy. The bankruptcy court approved the sale of substantially all of old GM’s assets to a newly formed company (“new GM”) in June 2009 as well the old GM’s amended bankruptcy plan in March 2011, and its assets and liabilities were transferred to liquidating trusts. These actions and the restructuring—which included major cost reductions described above—helped enable GM to report positive net income in every year from 2010 through 2019. Railroads: In 1970, the Penn Central Railroad, one of the largest in the country at the time, filed for bankruptcy. As Penn Central’s losses continued while in bankruptcy, the value of whatever assets that might have been available to satisfy its creditors’ claims was further eroded. Therefore, the bankruptcy proceeding was initiated to liquidate the railroad to meet the demands of its creditors. Faced with the potential cessation of railroad service for an entire section of the country, the federal government created Conrail to take over and operate specified portions of Penn Central as well as several other bankrupt railroads in the Northeast United States. Legislation was enacted that, combined with Conrail actions described above, enabled Conrail to become profitable. In the countries we selected, foreign governments also took actions to develop goals for their postal operators that enabled changes in the postal operators’ institutional structures and actions to address competitive pressures, economic downturns, and market changes. Based on our review of the selected countries’ government reports, including summaries of postal reform legislation, annual reports from foreign posts, and interviews with foreign government officials and representatives of foreign posts, we found that changes were made to foreign posts’: (1) products and services; (2) financial self-sustainment; and (3) institutional structure. Some of these actions were authorized by legislation that changed the status and duties of the postal operators; others were taken over a lengthy period that predated passage of key legislation. Some foreign posts also diversified into nonpostal products and services; however, postal-specific challenges and changes are the focus of this discussion. The selected foreign posts made multiple changes to their products and services. We found that, when transforming their postal operations, selected countries determined the level of postal services they deemed necessary to adequately serve the public. The main product changes involved expansion of their package delivery business by enhancing service and investing in facilities and sorting equipment dedicated to handling packages. Large increases in their package volumes and revenues helped offset declining letter mail volume and revenues. Two of the selected foreign posts reduced service levels to help control costs. For example, according to New Zealand Post officials, in response to reduced mail volume, New Zealand Post reduced its required frequency of mail delivery in urban areas from 5 to 3 days while maintaining 5-day delivery in rural areas with the stated goal of ensuring that postal service remained viable without government subsidization. Australia Post revised the service standards it provided for delivery of letter mail in 2016, resulting in slower delivery of some mail. In addition, the main pricing changes have involved price increases for all mail, as well as the introduction of discounted postal rates for letter mail entered at processing facilities that generally were closer to the final destination of the mail. For example, Royal Mail raised postal rates above the rate of inflation after relaxation of its price cap. In France, the postal regulator established a price cap in 2015 of 3.5 percent per year (in addition to inflation) and established a new cap in 2018 of 5 percent annual rate increases (including inflation) for 2019 through 2022 for letter mail and packages considered part of universal postal service. The 2018 price cap allows La Poste more price flexibility than the previous one. Likewise, Australia Post implemented an above-inflation rate increase in January 2020, citing the need to generate revenues to offset growing financial pressures from declining letter mail volume. Cost reduction was a major theme for the selected foreign posts, particularly with respect to compensation, benefits, and infrastructure costs. These actions were stimulated in part by reductions in mail volume and the associated revenues and workload, and in part by legislative or regulatory changes that allowed greater competition and created incentives, such as privatization, which resulted in shareholder pressure to enhance or encourage organizational profitability and efficiency. Because postal operations are labor-intensive, actions to address workforce costs were particularly important to improving financial results. These often went hand in hand with outsourcing, network restructuring, reductions in service levels to better align service with demand, and other changes to increase productivity and achieve cost savings. The governments and the postal operators of selected countries also considered the effects on stakeholders when making postal reform decisions. Specific changes varied from one foreign post to another. Workforce: In Germany, Deutsche Post officials told us that its employees hired after 1990 were designated private sector employees with lower pay and benefits than postal employees who were previously hired as civil servants. In France, La Poste officials told us that La Poste likewise transitioned its workforce in the 1990s from civil servants to private employees and ended recruitment of civil servants in 2000. They also said that this transition provided La Poste with a more flexible workforce and reduced its pension liability. Australia Post closed its defined benefit pension plan to new employees in 2012, while Royal Mail is transitioning to a defined contribution pension plan that it introduced in 2018. Royal Mail and New Zealand Post also have reduced the size of their workforce in recent years. Infrastructure: Some selected foreign posts consolidated their mail processing networks to reduce costs. For example, following the reunification of Germany, Deutsche Post replaced more than 320 mail processing facilities with 82 such facilities. Royal Mail and La Poste reduced the number of mail sorting centers by about 40 percent over the past two decades. In addition, all five selected postal operators have made changes to reduce retail network costs. As we reported in 2011, some foreign posts reduced the number of postal operator-owned and -operated facilities and in some cases closed facilities in an effort to reduce costs. At the same time, some minimized this disruption by expanding retail access through alternatives such as Internet sales and partnerships with retail businesses such as grocery stores or pharmacies. We reported that these changes either reduced operating and labor costs or improved customer service, in some cases because the partner retail facility stays open longer, or both. This trend continues. Deutsche Post, Post Office Limited in the United Kingdom, La Poste, Australia Post, and New Zealand Post have outsourced or franchised most of their postal retail functions to private nonpostal operators. For example, Deutsche Post franchised its postal retail outlets to local businesses to not only reduce expenses, but also increase the availability of postal retail services nationwide by putting retail counters in stores that were open longer than traditional post offices. Deutsche Post representatives stated that while there was some initial resistance to these changes, these concerns abated after a few months as customers realized they received better service and longer hours. The representatives also said the number of retail outlets has increased in recent years in response to increased demand for e-commerce package returns. La Poste has a substantial and growing proportion of retail facilities operated by private providers (in partnership with small shops, especially in rural areas) or co- located in local government-owned buildings (in partnership with local town halls in rural areas). In addition, Australia Post combined its letter and parcel delivery networks in 2018 to obtain efficiencies. Productivity: All of the postal operators in the selected countries took actions to enhance productivity, such as improving automation of mail processing, modernizing and streamlining operations, and changing work processes. For example, Deutsche Post officials stated they had streamlined their parcel sorting process and went from 140 parcel sorting centers in 1990 to 34 in 2019. Deutsche Post officials stated that this streamlining improved service performance for parcels; previously, most parcels were delivered within 3 or 4 days; by 2019, 93 percent of parcels were delivered within 1 day. Royal Mail officials also stated that they increased their level of automation and introduced new methods of parcel delivery, such as new high-capacity equipment for mail carriers, to increase efficiency as well. Government Assistance: The governments of some of the selected countries provided assistance to their postal operators in various forms, including assuming pension costs, granting tax exemptions, and providing subsidies to postal retail operations. For example, the governments of France and the United Kingdom assumed costs of defined benefit pensions for postal employees who are civil servants, while the government of Germany assumed these costs to the extent that they exceed the costs of private sector pensions. While all new employees are employed as private sector employees, German audit officials stated that the government’s pension obligation for postal employees who are or were civil servants and their dependents will last until 2079 and cost the government about €306 billion. The assumption of these pension plans was important in facilitating the privatization of these postal operators, according to Royal Mail and Deutsche Post officials, because without the reduced unfunded pension liabilities, the stock offerings for the newly created companies would have been much less attractive to private investors. France also provides subsidies for certain postal activities. In addition, the United Kingdom split off the postal retail network from Royal Mail into Post Office Limited, a separate entity owned and subsidized by the national government. Each of the five foreign countries we selected changed their institutional structure following the development of goals for postal transformation that were tailored to national needs and priorities. Each of these countries had definitions of universal postal services including provisions for nationwide delivery and access to postal retail services. As the following examples illustrate, the national context of each country has been of central importance to shaping these goals. In addition, postal transformation in the three selected European countries—Germany, the United Kingdom, and France—also had an international context in the broader effort to create and promote a single European internal market. Germany: In the 1990s, the German government changed its postal operator, Deutsche Post, from a government agency to a government- owned corporation. In 2000, the government changed Deutsche Post to a privately owned company so it could raise capital, modernize, and create a sustainable infrastructure. Goals for the newly created Deutsche Post were to maintain the high level of postal services, increase efficiency, and enhance profit. The legislation that created Deutsche Post also gave it more flexibility to respond to changes in the market. Currently, Deutsche Post remains a private company with the government holding a minority of its shares. United Kingdom: In 1969, the government of the United Kingdom changed the Post Office, its postal operator, from a government department to Royal Mail, a government-owned corporation prior to changing it to a privately owned company. It began privatizing Royal Mail in 2013 so Royal Mail could become more modern and competitive by raising private capital, operating with more flexibility, and be subject to shareholder scrutiny to drive efficiency. As mentioned above, the government of the United Kingdom also split off postal retail units into a new government-owned entity called “Post Office Limited” that is separate from Royal Mail. France: In 1991, the government of France changed La Poste, the postal service of France, from a government department to a public industrial and commercial establishment. In 2010, the government of France converted La Poste into a state-owned public limited company. This step allowed La Poste to raise additional public capital for investments to maintain and modernize its network, build a European parcel and express network, allow acquisitions outside Europe, and add nonpostal products and services, such as expanding its banking services. Australia: In 1989, the government of Australia changed its postal operator, Australia Post, from a government department to a government-owned corporation. It is required to earn a reasonable rate of return on its assets, maintain its equity, pay a reasonable dividend to the government, and be liable for the same taxes and charges as its competitors. New Zealand: According to a recent report, New Zealand Post began as a government department and became a state-owned enterprise in 1987, when legislation (State-Owned Enterprises Act 1986) created several such entities to address challenges in the national economy. Such corporations are required to be as profitable and efficient as a comparable business not owned by the state. Congress will face difficult choices in fundamentally reassessing the three critical foundational elements of USPS’s business model—level of universal postal service, financial sustainability, and institutional structure. These choices are likely to require changes in laws and will have differing effects on postal stakeholders. While the specific impacts will depend on the changes made, some or all of USPS’s stakeholders could be affected and these impacts should be considered as part of any reassessment. All three key areas are interrelated and significant changes in one area may affect another. For example, we have testified that Congress faces a tradeoff between the level of postal services the nation needs and the level of postal services the nation is willing to pay for. Based on our prior work, a starting point for a fundamental reassessment of USPS’s business model should be determining the level of postal services the nation needs. While mail volumes have declined since fiscal year 2006, businesses, governments, and households still pay USPS billions of dollars annually to deliver more than 140 billion of pieces of mail, demonstrating a continued nationwide demand for postal services. We and others—such as USPS, PRC, and USPS OIG—have called for a fundamental reexamination of what postal services the nation needs now and may need in the future. In particular, we have testified that USPS’s growing financial difficulties, combined with changing demand for postal services, have provided Congress with an opportunity to examine and potentially redefine what postal services should be provided on a universal basis and how they should be provided. As mentioned above, there are numerous federal laws and requirements related to the provision of universal postal service. For example, 6-day delivery has long been required by annual USPS appropriations acts. Over the past decade, legislation has been introduced, and USPS and others have proposed reducing the frequency of delivery. However, no legislation has been enacted that would allow USPS to reduce delivery frequency. There is also no consensus on the level of postal services the nation needs. Changes in service levels face opposition from some stakeholders, such as labor unions, affected communities, and the general public. Currently, legislation has been introduced that supports the preservation of both 6-day and door-to-door delivery for addresses that have it, and some mailer groups support one or both of these positions. Representatives from postal labor unions we spoke with stated that universal postal service is appropriate as currently defined and could be expanded to provide more products and services. Stakeholders have also expressed differing views on whether the frequency of delivery should be reduced to help USPS address its financial problems. USPS and PRC have estimated that eliminating Saturday delivery would reduce USPS’s costs but also would likely affect mail volume sent by business mailers, although USPS and PRC disagreed on the degree to which it would do so. USPS estimated that it could save $1.4 billion to $1.8 billion a year by reducing the frequency of mail delivery to 5 days while maintaining 7-day package delivery. To put these potential savings into context, delivery is USPS’s most costly operation. We reported, however, that USPS would face challenges in, among other things, how efficiently USPS would absorb the additional volume delivered in the remaining delivery days and its potential effect on mail volume. We also described potential trade-offs, such as possibly reducing the demand and value of USPS products if customers are not getting their delivery needs met. Further, key postal stakeholders hold opposing views on many other options that have been proposed. For example, to raise revenues, USPS and some postal labor unions favor eliminating or raising the price cap on market-dominant products, which would enable USPS to raise rates more than the rate of inflation but would require changing the current regulatory system. Mailers, however, have expressed opposition to increasing postage rates higher than the rate of inflation. Postal labor unions also favor increasing revenues by introducing new postal and nonpostal products and services. We have recently found, however, that USPS’s nonpostal revenues generated at postal retail facilities are small and that there are limited opportunities to generate revenues from nonpostal products and services from USPS’s delivery network. For example, we reported that nonpostal products and services offered through USPS’s postal retail facilities generated about $431 million in fiscal year 2018, accounting for less than 1 percent of USPS’s total revenue. In addition, we reported several potential limitations to USPS adding nonpostal services to USPS’s mail carrier activities, such as checking in on homebound and older residents and reporting signs of blighted properties. These limitations included, among other things, limited net revenue potential and a potential adverse effect on mail service delivery. A fundamental reassessment of USPS’s business model would include determining the degree to which USPS should be financially self- sustaining, i.e., the degree to which USPS’s operating costs and liabilities should be covered by ratepayers (such as businesses and individuals who pay USPS to send mail). If a reassessment concluded that USPS should be fully self-sustaining, past legislative proposals that would change elements of USPS’s costs and revenues may be worth congressional consideration. Stakeholders, however, have not reached a consensus on any of these proposals and none has been enacted. Another avenue is to focus on reducing costs. As we have also reported, compensation and benefits costs, which comprise about three-quarters of USPS’s operating costs, are driven by a mix of USPS contracts and policies, including collective bargaining agreements negotiated with unions representing 92 percent of USPS employees and statutory requirements governing USPS employee pay and benefits. USPS compensation and benefits costs for its active employees increased by almost $1 billion in fiscal year 2019 despite a slight decrease in the size of the workforce and declining workload from reduced mail volume. While USPS has been able to make some reductions in pay and benefits, its ability to control compensation costs is significantly inhibited by the collective bargaining process, which results in binding arbitration if an impasse is reached. According to USPS, all negotiations take place against the backdrop of binding arbitration (and the arbitrators have historically been reluctant to deviate from the status quo), resulting in only incremental changes. We have long supported changing the laws regarding collective bargaining to require that USPS’s financial condition be considered in binding arbitration. We have also reported that the collective bargaining structure, which was established many years ago, should be reexamined considering the dramatic changes in USPS’s competitive environment and rising personnel costs that have contributed to USPS’s losses. Multiple bills have been proposed changing the process and/or criteria for collective bargaining to a different standard. The 2018 report from the Task Force on the United States Postal System recommended that collective bargaining over compensation should be eliminated for postal employees. While eliminating or revising the collective bargaining process could potentially provide USPS greater flexibility in employee pay, there would be trade-offs. For example, we recently found that the potential annual cost savings associated with USPS implementing cuts for all current employee pay by 1 percent would be about $321 million; a 10 percent cut would potentially save $3.2 billion. However, we also reported that while USPS could reduce its compensation costs through efforts such as reducing mail delivery frequency, USPS would face challenges in realizing these savings, such as the extent to which workhours could be reduced. Furthermore, these savings could be offset by other factors including service or morale issues. With respect to benefits, we recently reported on a wide range of possible changes that would reduce or limit costs for postal retiree health benefits, nearly all of which would require a legislative change. Some approaches would shift costs to the federal government; some would reduce benefits or increase costs to postal retirees or employees; and some approaches would change how benefits are funded. Similar types of legislative changes could be considered with respect to postal pension benefits. In addition, if Congress decides that USPS should be financially self- sustaining but makes no changes to improve USPS’s financial condition, USPS will be unable to address unfunded liabilities for postal retiree health and pension benefits, an inability that could eventually translate into higher costs for future postal ratepayers. Ultimately, if USPS’s expenses continue to exceed its revenues, USPS is likely to continue to miss required payments, reduce operations, or seek federal appropriations through the annual appropriations process to cover its operating costs. If Congress determines that USPS should no longer be expected to be financially self-sustaining or if actions taken do not restore financial self- sustainability, Congress could provide financial assistance—not unlike what happened in other countries or for selected domestic business—to enable USPS to cover its costs, and to fulfill its obligation to provide federal health and pension benefits to postal employees and retirees. Federal financial assistance could be provided in various forms, such as: Appropriating funds to help cover USPS’s operating costs, essentially the same arrangement that was used to finance the former U.S. Post Office Department. Appropriating funds to supplement USPS’s payment of certain costs, such as to help fund its capital investments. For example, the federal government provides Amtrak, which is operated as a for-profit corporation with annual grants to operate and make capital investments in passenger rail service to supplement the revenues it generates. Assuming some or all of USPS’s unfunded liabilities for retiree health benefits. This could take different forms, such as direct assumption of responsibility for unfunded liabilities or, more indirectly, requiring postal retirees to participate in Medicare which would decrease USPS’s costs but increase Medicare’s costs. Assuming some or all of USPS’s unfunded liabilities for pension benefits. Writing off some or all of USPS’s debts to the U.S. Treasury. Options regarding the federal government providing ongoing financial assistance to USPS could have effects on both USPS and the federal government as a whole. Notably, this assistance would have to be funded in some way—either through offsetting reductions in federal expenditures in other areas, through tax increases, or through an increase in federal deficits. Moreover, reliance on federal funding could mean that USPS would be exposed to the uncertainty inherent in the annual appropriations process. In addition, access to annual appropriations to cover financial shortfalls could have an unintended consequence of reducing USPS’s incentives to become more cost-efficient. At present, there is no consensus on USPS’s level of financial self- sustainability should be. For example, representatives of labor unions we spoke with stated that Congress should address issues regarding postal retiree benefits before any reassessment of USPS’s financial self- sufficiency can occur. Increased federal financial support of USPS might also face political opposition, due to concerns about minimizing federal deficits and ensuring fair competition between USPS and the private sector. The final area of consideration in any reassessment of USPS’s business model is identifying what institutional structure could best deliver the level of postal services at the level of financial sustainability that Congress has determined. As an independent establishment of the executive branch, USPS must provide universal postal service while being expected to be financially self-sustaining. Thus, there may be a tension between attempting to fulfill public service missions while operating in an efficient, business-like and financially self-sustaining manner. USPS officials told us that as an entity of the federal government, its primary purpose is the achievement of its statutory universal service mission, and it has no incentive to seek to maximize profits at the expense of achieving its public service mission over the long term. Therefore, according to USPS, if it were maintained as an independent establishment of the executive branch or converted into a more typical government agency, it could continue to prioritize this public service mission. Additionally, there is widespread support for USPS’s institutional status as an independent establishment of the executive branch. Congressional resolutions have been introduced stating that “Congress should take all appropriate measures to ensure that the United States Postal Service remains an independent establishment of the Federal Government and is not subject to privatization.” Likewise, all four of the largest USPS unions, both of its management organizations, and a number of mailer groups and mailers support keeping USPS an independent establishment of the executive branch. Nonetheless, considering the depth of USPS’s financial problems and its poor financial outlook, now may be an appropriate time for Congress to reconsider what institutional structure will be most appropriate for USPS in the 21st century. However, any substantial change to USPS’s institutional status would require changing federal law. Based on our past work and options identified by USPS and others, Congress has a range of options it could consider in reassessing USPS’s structure (see table 1). The potential advantages and disadvantages of placing USPS into alternative institutional structures for USPS have long been debated. Several options have been discussed: USPS could revert to a traditional federal agency. USPS and its governance would be more consistent with other federal activities that are dependent on federal appropriations provided through the annual appropriations process. Many postal stakeholders, however, do not support such a change. For example, USPS told us that if it became a typical government agency reliant on federal appropriations to fill any operating gap, the political constraints that typically apply to government agencies could reduce USPS’s adaptability. Furthermore, changing USPS to a typical government agency could reduce its incentives to increase revenues or reduce costs in response to changing communication technologies and patterns. Consistent with this point, the 1968 presidential commission found that when it operated as a federal agency, the former U.S. Post Office Department had a lack of innovation, cost-control, and capital investment with major managerial decisions made through the legislative process. These and other issues led to persistent operational deficits, low productivity, and poor mail service. USPS could remain an independent establishment of the federal government with additional authority—relative to the status quo—over certain aspects of its business model. For example, USPS could be provided more flexibility to raise postal rates, introduce new nonpostal products, and make various changes to reduce its costs such as reducing the frequency of delivery or further consolidating its retail, transportation, and processing networks. USPS has long advocated for additional flexibility under its current institutional structure—such as to eliminate the price cap on market-dominant products and have greater flexibility to offer nonpostal products. Representatives from postal unions also stated that USPS should be provided additional flexibility, such as to expand into nonpostal products, which some representatives stated could help preserve its public service mission to provide universal postal services. Consensus does not, however, exist as to what flexibility should be given to USPS. For example, some mailer groups favor keeping the price cap unchanged, stating the cap is sufficient and provides incentives for increased efficiency. In addition, some stakeholders have supported further limiting USPS’s flexibility to reduce service standards, close retail outlets, or consolidate processing facilities, while other stakeholders noted that greater flexibilities in these areas would reduce USPS’s costs and enhance its efficiency. If USPS were to become a government-owned corporation or a government-sponsored enterprise, USPS could be incentivized to increase efficiency as a federally chartered entity providing a public service with a predominantly business nature. For example, three of the four third-party experts we spoke with stated that USPS should retain its current mission of universal postal service but become more like a private company with greater freedom to operate in a business- like manner. The new structure could promote greater incentives toward cost control and financial success. Government-owned corporations are federally chartered entities that provide a public service with a predominantly business nature. These corporations can have a board of directors that is appointed by the President. Government-sponsored enterprises are federally chartered entities that are privately owned and, typically, have a board of directors appointed by private sector owners. If USPS were to be a private company it would become accountable to the shareholders of that company. USPS told us that as a private company, its primary incentive could be to maximize profits and that, in USPS’s view, private shareholders would be most focused on short-term financial outcomes. Thus, any such design of a private USPS would need to balance its profit motive with the nation’s needs for universal postal service and the affordability of that service. For example, while Royal Mail is a private corporation owned by shareholders, the government of the United Kingdom still mandates 6- day delivery for letter mail (and 5-day delivery for packages) with specified delivery standards, and some mail types are subject to price controls. In addition, the United Kingdom monitors the provision of universal postal service and can take enforcement actions regarding regulatory conditions and competition law. Similarly, Germany has legal instruments to enforce the provision of universal postal services, although according to German government officials these instruments have not been used. Although some of the domestic businesses we examined reduced their costs through bankruptcy, this is likely not an option for USPS. As detailed in its report (see appendix I), National Bankruptcy Conference (NBC) found that USPS is not eligible to become a “debtor” under chapters 11 or 9 of the current Bankruptcy Code. According to NBC, a court likely would deem USPS to be a “governmental unit”—meaning it could not file for relief under chapter 11—and a court would deem USPS not to be a “municipality”—meaning it could not file for relief under chapter 9. Therefore, legislation amending the Code would be required to make USPS eligible for relief. According to NBC, however, even if the Bankruptcy Code were amended to allow USPS to file as a chapter 11 or 9 debtor, the Code would still not currently authorize a bankruptcy court to discharge the ongoing statutory obligations that have led to USPS’s current financial situation, and amending the Code to authorize such court action could raise constitutional (separation of powers) concerns. Moreover, NBC noted the bankruptcy process is designed to address obligations that have already accrued, not to override or amend statutes that apply to a debtor’s post- bankruptcy operations and obligations. In NBC’s opinion, because USPS’s pension and health care obligations are imposed by statute instead of by contract as in most bankruptcy reorganization proceedings, the bankruptcy process is not an effective or appropriate mechanism to address USPS’s obligations or potential transformation. NBC thus concluded that “although the bankruptcy process and bankruptcy tools raise interesting ideas for restructuring USPS’s existing and future obligations…all roads for doing so lead back to Congress.” Any changes that Congress makes to USPS’s business model will take time to implement and will need to be reevaluated as market conditions evolve. We have reported that fully implementing major transformations of government agencies can take years, and we also found that to be the case for the selected domestic businesses and foreign posts noted in this report, regardless of the changes needed. For example, railroads in the Northeast, airlines, and automakers took many years to implement a series of changes to their businesses. It took Germany more than a decade to fully liberalize and then privatize its postal operator, and the United Kingdom’s effort to privatize its postal operator took about 5 years. All of these organizations continue to adapt as they address ongoing challenges in a changing and highly competitive business environment. For example, GM recently stated that years after exiting bankruptcy and restoring profitability, it is closing some factories and focusing on developing electric and self-driving cars. Several freight railroads facing a downturn in freight traffic have also decided to run longer trains less frequently to reduce labor costs and increase efficiencies. Similarly, changes in the use of postal services will continue for the foreseeable future, necessitating continued adaptation. Some of the countries we selected are anticipating the need to be prepared for possible future changes. For example: In August 2019, German government officials said they would consider reducing postal delivery frequency from 6 to 5 days a week as part of an ongoing review to adapt Germany’s 20-year old postal law to changing market conditions and customer demands. In the United Kingdom, the postal regulator assessed postal users’ needs in 2020 in light of the changes in the postal market and to prepare for its regulatory review, which is to be concluded by 2022. A 2018 consultant’s report to the European Union (EU) recommended that the EU relax its universal service obligations to accommodate future changes in the postal market. The EU is currently studying how postal users’ needs are changing to determine if it needs to change its framework to allow member states to change their definitions of universal service obligations. In November 2019, the Australian government ordered a review of Australia Post’s long-term strategy to operate as a sustainable postal service provider, considering market conditions such as e-commerce, the regulatory environment and changes in business and consumer service needs. The government of New Zealand is scheduled to revise its memorandum of understanding with New Zealand Post defining universal service obligations by 2021. PAEA required GAO to evaluate strategies and options for the long-term structural and operational reform of USPS by December 2011. As USPS continued to face financial challenges, we accelerated this evaluation, which we issued in April 2010. However, we found that USPS’s business model, which was to provide universal postal service through self-supporting, business-like operations as an independent establishment of the executive branch, was not viable due to USPS’s inability to reduce costs sufficiently to respond to continuing declines in mail volume and revenue. In particular, we identified strategies to reduce compensation and benefit costs, reduce other operations and network costs, improve efficiency, and generate revenues through product and pricing flexibility. We also stated that while USPS may be able to improve its financial viability if it took more aggressive action to reduce costs, it was unlikely that those actions alone would fully resolve USPS’s problems unless Congress also took action. Therefore, we stated that Congress should consider, among other things, any and all options available to reduce USPS’s costs. While bills on these issues were introduced and in some cases passed congressional committees, postal reform legislation to address these considerations has not been enacted. In addition, in our most recent update to our High Risk List in 2019, we reiterated the basic elements of our 2010 matter for congressional consideration by stating that Congress should consider various options to better align USPS’s costs with its revenues. We stated that Congress should consider addressing constraints and legal restrictions that limit USPS’s ability to reduce costs and improve efficiency through considering a comprehensive package of legislative actions. To date, such a legislative package has not been enacted. Furthermore, we reported in 2018 that the financial outlook for the Postal Service’s Retiree Health Benefits Fund was poor, as USPS had not made any payments into it since 2010. OPM then forecasted the fund would be depleted by 2030 if USPS continued to not make payments. Therefore, we stated that Congress should consider passing legislation to put postal retiree health benefits on a more sustainable financial footing. However, legislation has not yet been enacted to address this issue. We have often reported over the past 10 years that USPS’s ability to take actions taken under its current authority is insufficient to fully address its financial situation. Absent congressional action on critical foundational elements of the USPS business model, USPS’s mission and financial solvency are increasingly in peril. USPS’s growing difficulties to provide universal postal service in a financially self-sustaining matter provide Congress with the need to consider fundamental reform of the entire framework of postal services in the United States. In so doing, we continue to believe that as we stated in 2010, Congress should consider any and all available options. Comprehensive postal reform has not taken place in part because of the difficulty in obtaining compromise among various stakeholders with divergent views. Comprehensive, effective, and successful reform cannot occur until there is leadership and clarity around: what services should be provided, whether USPS is to be fully financially self-sustaining or the extent of federal financial support, and what institutional structure best supports these changes. Congressional leadership is critical in transforming USPS because consensus on policy decisions involving value judgments, trade-offs, and effects on postal stakeholders will be difficult to achieve. In addressing these issues, while all stakeholders’ interests should be understood and taken into consideration, the fundamental needs of the nation must take precedence. Continued inaction will result in deepening financial problems—putting USPS’s mission to provide universal postal service at greater risk and minimizing the ability to make the most appropriate or sustainable policy decisions. We are making the following three matters for congressional consideration: Congress should consider reassessing and determining the level of universal postal service the nation requires. (Matter for Consideration 1) Congress should consider determining the extent to which USPS should be financially self-sustaining and what changes to law would be appropriate to enable USPS to meet this goal. (Matter for Consideration 2) Congress should consider determining the most appropriate institutional structure for USPS. (Matter for Consideration 3) We provided a draft of this report to USPS and PRC. USPS and PRC provided written responses which are reproduced in appendixes IV and V, respectively. In its response, USPS concurred with our first two matters to reassess and determine the level of universal postal services the nation requires and to determine the extent to which USPS should be financially self- sustaining. USPS noted that the recent COVID-19 pandemic has both highlighted USPS’s essential role in the nation’s infrastructure and has caused a significant and sudden decline in mail volume, leading to a short-term liquidity crisis. USPS stated that while action by Congress is critical to ensure its ability to operate in the short-term, its financial situation has long been unsustainable due to statutory and regulatory structures that limit their ability to increase revenues and decrease costs. USPS noted that these changes require Congress to adopt reforms to secure USPS’s long-term financial viability. In addition, USPS concurred with the National Bankruptcy Conference’s legal analysis that Federal bankruptcy laws do not apply to USPS and that all roads for USPS restructuring lead back to Congress. USPS generally agreed with our third matter, stating that determining the institutional structure could logically be a part of a comprehensive congressional examination of its business model. USPS stated that it does not believe that corporatization or privatization would unlock new efficiency potential in USPS and that sustainable postal service does not hinge on the provider’s institutional form. However, as we and USPS have stated, its current legal and regulatory structure does not provide flexibility in some key areas. While our report states that a corporate or privatized institutional structure could provide both the flexibility and a greater incentive to operate in a more business-like manner than USPS’s current structure, we also recognize there are advantages and disadvantages to any institutional structure. As a result, we are not recommending any particular institutional structure for USPS, but are urging that Congress identify what institutional structure could best deliver the level of postal services at the level of financial sustainability that Congress determines. In its response, PRC agreed with all of our matters for congressional consideration. Particularly, PRC noted that the matter to reassess and determine the level of universal postal service the nation requires must be addressed as soon as possible. The PRC noted that given USPS’s severe and worsening financial situation (even before the impacts of the current pandemic crisis), a clear and specific definition of universal postal service and how that obligation can be funded must be provided. The PRC stated that Congress may want to consider mandating that PRC define and update the universal service definition by regulation. Both USPS and PRC provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Postmaster General, the Chairman of PRC, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff making key contributions to this report are listed in appendix VI. Appendix II: Selected Legal Requirements Applicable to the U.S. Postal Service (USPS) Legal requirements USPS is required to provide prompt, reliable, and efficient services to patrons in all areas, render postal services to all communities, and serve as nearly as practicable the entire population of the United States. USPS is specifically required to receive, transmit, and deliver written and printed matter, parcels, and like matter throughout the United States, its territories and possessions, and pursuant to certain agreements, throughout the world. USPS is required to provide a maximum degree of effective and regular postal services to rural areas, communities, and small towns where post offices are not self-sustaining. No small post office can be closed solely for operating at a deficit, it being the specific intent of the Congress that effective postal services be insured to residents of both urban and rural communities. Statutory and regulatory requirements specify the process and criteria for post office closings, including appellate review by the Postal Regulatory Commission (PRC). In determining all policies for postal services, USPS is required to give the highest consideration to the requirement for the most expeditious collection, transportation, and delivery of important letter mail. In selecting modes of transportation, USPS is required to give the highest consideration to the prompt and economical delivery of all mail. Service standards 39 U.S.C. § 3691, 39 C.F.R. Pt. 121 Letter mail monopoly 18 U.S.C. §§ 1693-99; 39 U.S.C. §§ 601-06 Mailbox monopoly 18 U.S.C. § 1725 Collective bargaining 39 U.S.C. §§1004,1206-07 Benefit programs 39 U.S.C. § 1005; 5 U.S.C. §§ 8348(h), 8423, 8909a Level of benefits 39 U.S.C. § 1005(f) Comparability 39 U.S.C. §§ 101(c), 1003(a) Workers’ compensation 39 U.S.C. § 1005(c) For many years, provisions in annual appropriations acts have stated “hat 6-day delivery and rural delivery of mail shall continue at not less than the 1983 level.” USPS is required to establish modern service standards for each market-dominant product (e.g., delivery of First-Class Mail within the continental United States in 2-3 delivery days); these service standards are defined in the Code of Federal Regulations. USPS’s letter delivery monopoly is codified in criminal and civil laws known as the Private Express Statutes. These laws generally prohibit anyone from establishing, operating, or using a private company to carry letters for compensation on regular trips or at stated periods over postal routes or between places where mail regularly is carried. Restricts access to mailboxes by prohibiting anyone from knowingly and willingly placing mailable matter without postage in any mailbox, providing USPS exclusive access to mailboxes. USPS negotiates collective bargaining agreements with its labor unions. If the parties are unable to reach an agreement, binding arbitration by a third-party panel will ultimately be used to establish agreement. USPS is also required to consult with postal supervisory and managerial organizations concerning changes in pay, benefits, and other programs that affect their membership. USPS is required to participate in federal pension and health benefit programs, with specific provisions regarding the required level of USPS’s funding of these programs. For example, USPS is required to prefund both postal pension benefits and postal retiree health benefits, each with payments that fully cover USPS’s share of future benefit costs. The law requires USPS’s fringe benefits to be at least as favorable as those in effect when the Postal Reorganization Act of 1970 was enacted, unless variation of benefits is collectively bargained. Compensation for USPS officers and employees is required to be comparable to the rates and types of compensation paid in the private sector of the U.S. economy. USPS policy also is required to maintain compensation and benefits for all officers and employees on a standard of comparability to comparable levels of work in the private sector. USPS is required to participate in the federal workers’ compensation program, which covers postal and other federal employees and provides compensation to federal employees, as well as dependents, in the event of an employee’s death. Citation(s) Access to facilities 39 U.S.C. § 403(b) Legal requirements USPS is required to establish and maintain postal facilities of such character and in such locations, that postal patrons throughout the Nation will, consistent with reasonable economies of postal operations, have ready access to essential postal services. Generally, annual appropriations prohibit USPS from using funding to consolidate or close small rural or other small post offices. Appropriations restrictions See, e.g., Pub. L. 116-6, 133 Stat. 180 (2019) Processing/logistics facilities Pub. L. 109–435 § 302(c)(5), (2006), 120 Stat. 3219, codified at 39 U.S.C. § 3691 note Price cap 39 U.S.C. § 3622(d) The law requires USPS to provide public information and opportunities for public input and comment before closing or consolidating any mail processing or logistics facilities, and take comments into account when making a final decision. An inflation-based price cap generally limits rate increases for market-dominant products, including First-Class Mail, USPS Marketing Mail, Periodicals and Package Services such as Bound Printed Matter, Media Mail, and Library Mail. The PRC, an independent establishment of the executive branch, must review USPS proposals to change domestic postal rates and fees. Debt limits 39 U.S.C. § 2005 Restriction on nonpostal lines of business 39 U.S.C. §§ 404(e), 102(5) Investment of postal retiree funds 5 U.S.C. §§ 8348(c), 8909a(c) Whenever USPS proposes a change in the nature of postal services that will have an effect on a substantially nationwide basis, it must request an advisory opinion from the PRC on the proposal. USPS has the authority to borrow up to $15 billion from the U.S. Treasury. The annual net increase of obligations for capital improvements and defraying operating expenses is limited to $3 billion. USPS is limited to providing nonpostal services to those offered as of January 1, 2006 that PRC has authorized USPS to continue. Nonpostal service is defined to mean any service that is not a postal service. A postal service is defined as the delivery of letters, printed matter, or mailable packages, including acceptance, collection, sorting, transportation, or other function ancillary thereto. Funds set aside for postal pensions and retiree health benefits are required by law to be invested in U.S. Treasury securities. Lori Rectanus, (202) 512-2834 or rectanusl@gao.gov. In addition to the individual named above, Derrick Collins (Assistant Director); Greg Hanna (Analyst-in-Charge); Amy Abramowitz; Kenneth John; Hannah Laufe; Serena Lo; Michael Mgebroff; Joshua Ormond; Joshua Parr; Susan Sawtelle; Crystal Wesco; and Laurel Voloder made key contributions to this report.", "summary": "An independent establishment of the executive branch, USPS is required to provide prompt, reliable, and efficient services to the public. While USPS is to be self-sustaining, it lost about $78 billion from fiscal years 2007 through 2019 due primarily to declining mail volumes and increased costs. Given USPS’s poor financial condition, in 2009 GAO identified USPS’s financial viability as a high-risk area, a designation it retains today. GAO was asked to explore issues related the transformation of USPS and potential implications for stakeholders. This report (1) examines major challenges facing USPS, (2) identifies how selected domestic businesses and foreign posts reportedly have addressed serious challenges, (3) examines critical foundational elements of USPS’s current business model, and (4) identifies key previously issued GAO matters for congressional consideration regarding USPS and actions taken in response. GAO reviewed its prior reports and related matters for congressional consideration, analyzed laws and regulations, and assessed USPS documents on financial and operational performance. It also reviewed reports by the USPS Office of Inspector General, the Postal Regulatory Commission, and other selected groups such as the 2018 Task Force on the United States Postal Service. To identify how domestic businesses and foreign posts addressed similar serious challenges, GAO selected for review (1) six domestic organizations in the airline, automobile, and railroad industries and (2) five foreign posts in five countries—Australia, France, Germany, New Zealand, and the United Kingdom. The businesses and countries had characteristics similar to USPS, such as large unionized work forces, and had reportedly made significant changes to their business models. For each of these businesses and countries, GAO analyzed public reports on financial and operational performance, as well as institutional structure and requirements. GAO also interviewed government and postal officials from three selected countries and officials from the National Audit Offices of two of the selected countries. Because questions were raised regarding the application of the U.S. Bankruptcy Code to USPS, GAO also requested the National Bankruptcy Conference to assess whether USPS could use bankruptcy or other restructuring processes. To examine critical USPS business model elements, GAO reviewed its prior reports and reports from numerous other organizations, and obtained the views of stakeholders. Since GAO's 2009 high-risk designation, the U.S. Postal Service's (USPS) financial viability has progressively worsened due to declining mail volume, increased employee compensation and benefit costs, and increased unfunded liabilities and debt. First-Class Mail volume has declined 44 percent since fiscal year 2006. Additionally, employee compensation and benefits costs have been increasing. Although USPS's work force declined from about 786,000 in fiscal year 2007 to about 617,000 in fiscal year 2013, USPS's work force increased to about 630,000 in fiscal year 2019. Finally, total unfunded liabilities and debt continue their steady upward trend (see figure). To address these challenges, USPS has taken a variety of actions such as providing increased self-service options and reducing facility hours. Statutory requirements, however, limit USPS's ability to make changes in areas such as certain service offerings, pricing, and its employee compensation and benefits. In confronting similar types of challenges that are facing USPS, GAO selected large domestic businesses (companies) and foreign postal entities (widely known as “foreign posts”) that have seen significant change in foundational elements of their business models. Specifically, according to GAO's analysis of publicly available reports and interviews of cognizant officials, these organizations have had major changes in services and products, financial self-sustainment, and institutional structure: Companies and foreign posts have modified services and products to focus on profitable offerings, and two countries’ posts reduced postal service levels. For example, New Zealand Post reduced its mail delivery’s frequency from 5 to 3 days per week in urban areas. Companies have reduced their workforce, infrastructure, and operational costs, and some accepted government financial assistance to help remain financially viable. Cost reduction has also been a priority for all countries’ posts, especially in compensation and benefits, while three countries’ governments provided financial assistance to their posts. Four of the selected companies declared bankruptcy leading to restructured corporations; some merged with other companies to increase their revenues. Two countries privatized their posts, and three others restructured their posts from government departments into government-owned corporations. Regarding USPS, reassessing its business model should start with the level of required postal services. For example, delivery is USPS’s most costly operation; USPS officials estimate annual savings of $1.4 billion to $1.8 billion if delivery of mail were reduced to 5 days rather than 6 days per week. Second, USPS is to function as a financially self-sustaining entity; however, it does not. A reassessment could include determining whether some of USPS’s costs and liabilities should be borne by taxpayers. Third, alternative institutional structures for USPS range from a federal agency to a private company. A bankruptcy proceeding is not an effective or appropriate means to address the issues associated with a potential USPS restructuring, according to the National Bankruptcy Conference. Prior GAO reports have included suggestions for Congress to address USPS’s financial viability. For example, GAO’s 2010 report identified strategies to reduce compensation, benefits, and operational costs. GAO stated that Congress, among other things, consider all options available to reduce costs. While bills in this area were introduced and in some cases passed congressional committees, legislation was not enacted. In 2018, GAO reported that the financial outlook for the Postal Service Retiree Health Benefits Fund was poor—the Office of Personnel Management forecasted the fund would be depleted by 2030 if USPS continued not making payments into it. Legislation has not been enacted to place postal retiree health benefits on a more sustainable financial footing. Postal reform legislation has not taken place in part because of the difficulty in obtaining compromise among various stakeholders with divergent views (see figure below). However, since GAO’s 2010 report, USPS’s financial condition has significantly worsened raising fundamental questions about key elements of USPS’s business model. Such questions warrant congressional action. Congress should consider reassessing and determining the (1) level of postal services the nation requires, (2) extent to which USPS should be financially self-sustaining, and (3) appropriate institutional structure for USPS. Both USPS and the Postal Regulatory Commission (PRC) generally concurred with the matters.", "document_type": "gao"}
{"report": "The MESA Complex at Sandia comprises multiple production facilities and buildings, which total approximately 400,000 square feet (see fig. 1). In particular, the SiFab Facility, completed in 1988, is the primary production facility for microelectronics integrated into nuclear weapons. The SiFab Facility produces application-specific integrated circuits (ASIC) that are custom-designed to control certain nuclear weapon arming, fuzing, and firing functions. The MESA Complex also includes other buildings, such as the Micro Fabrication Facility, which was completed in 2006 and produces strategic radiation-hardened devices for manipulating electronic signals and electrical power. The physical layouts of these two production facilities center around a series of clean rooms that are designed to maintain an extremely low level of dust and other particulates, which can harm microelectronic functionality. The two facilities contain about 375 pieces of specialized production equipment, some of which cost millions of dollars, and have acid exhaust and liquid waste management systems for handling the byproducts of the production processes. The SiFab Facility produces all of the strategic radiation-hardened ASICs currently used in nuclear weapons. ASICs are produced on wafers—a thin slice of semiconductor material such as silicon—using what is referred to as a complementary metal-oxide semiconductor (CMOS) process technology. The production of ASICs requires hundreds of processing steps, which are completed over multiple weeks. For example, according to Sandia documentation, the production of a specific type of ASIC requires over 600 processing steps over an approximately 26-week period. Microelectronics are produced with characteristic dimensions (or “feature sizes”) measured in nanometers (nm), or one-billionth of one meter. The process technology together with an associated feature size is known as a technology “node.” In general, smaller nodes represent more advanced technologies. The SiFab Facility produces microelectronics at the 350 nm node, and NNSA and Sandia refer to the CMOS production process technology at the 350 nm node as “CMOS7.” Currently, state-of-the-art microelectronics are produced at the 32 nm or below node. For example, the Intel Corporation produces commercial microelectronics at the 14 nm node for use in personal computers and servers. However, such smaller nodes are more challenging to produce and have not been proven to perform at the strategic radiation-hardened level, according to Sandia contractor representatives. Figure 2 shows commercially produced microelectronics on a wafer (left photo) and diced into individual microelectronics parts next to a U.S. dime (right photo). As shown in table 1, NNSA is undertaking multiple LEPs and weapon modernization efforts, in which Sandia is participating. In addition, the 2018 Nuclear Posture Review calls for NNSA to consider additional weapon programs—specifically, a program to develop a modern nuclear- armed sea-launched cruise missile, and another to develop a new submarine-launched ballistic missile warhead (now referred to as the W93). To develop and produce microelectronics for these efforts, Sandia must (1) conduct research and development activities, (2) finalize the design of microelectronics to meet military requirements specific to the weapon program into which the microelectronics will be integrated, and (3) produce the microelectronics. Sandia must conduct all of these activities years before NNSA delivers a weapon program’s first production unit to DOD. According to Sandia documents and contractor representatives, microelectronics research and development efforts generally begin 10 to 15 years before a weapon program’s first production unit date, while microelectronics production generally begins 3 to 5 years before a first production unit date. DOD is also undertaking modernization efforts related to nuclear weapon delivery platforms, and Sandia is producing microelectronics to support those efforts. Specifically, DOD is responsible for designing and producing the arming and fuzing components on delivery platforms for certain types of nuclear weapons, and Sandia produces some of these components for DOD at the MESA Complex. For example, according to Air Force and Sandia documentation, the Air Force contracted with Sandia to design and produce microelectronics for its Intercontinental Ballistic Missile Fuze Modernization, which will provide a new fuze for use on both the current Minuteman III missile and its replacement, the Ground Based Strategic Deterrent missile. DOE and NNSA distinguish between projects and programs, and the agencies use different management approaches for each: Projects. DOE’s project management order governs NNSA’s management of capital asset acquisition projects with a total cost greater than $50 million. The order states that capital assets projects have a defined start and end point. Capital assets include land, structures, equipment and intellectual property that are used by the federal government and have an estimated useful life of 2 years or more. The order’s goal includes delivering projects within their original performance baselines (on time and within budget) and fully capable of meeting mission performance and other requirements, such as environmental, safety, and health standards. Programs. As we reported in 2018, DOE has not established a program management policy. However, NNSA issued its own program management policy in February 2019. The policy applies to all NNSA elements and requires them to establish additional program management requirements for respective NNSA programs based on needs, risk, complexity, and stakeholder involvement, among other things. The NNSA policy defines a program in part as an organized set of activities directed toward a common purpose or goal, undertaken or proposed in support of an assigned mission area. In addition, some NNSA offices have issued their own program management directives that are more specific than the NNSA policy. For example, NNSA’s Office of Defense Programs—which is responsible for, among other things, weapon modernization programs, including LEPs, and associated materials and components, such as microelectronics—issued a program management directive in June 2019 that establishes requirements and processes for managing the office’s programs. This directive establishes four program management categories and execution requirements for these categories. These management categories are risk-based and apply different execution requirements commensurate with program risk. The MESA Complex’s estimated fiscal year 2020 budget is $283 million, according to Sandia documentation. As shown in figure 3, this funding comes from a variety of sources, because Sandia uses the MESA Complex to meet both NNSA’s and DOD’s nuclear weapon production missions as well as for research and development for those and other federal entities through strategic partnership programs. Sandia documentation states that a portion of the MESA Complex’s budget is obtained from other, non-NNSA federal entities that pay Sandia directly to produce microelectronics for, among other thing, research and development purposes, and this amount of funding fluctuates annually. According to Sandia contractor representatives, the laboratory presents MESA’s budget as an estimate for this reason. Specific funding sources are discussed in greater detail below: NNSA provides about 60 percent (or $168 million) of the MESA Complex’s total estimated budget for fiscal year 2020. Two NNSA offices account for most of the agency’s funding: The Office of Defense Programs accounts for 42 percent (or about $71 million) and is responsible for ensuring the United States maintains a safe, secure, and reliable nuclear stockpile through the application of science, technology, engineering, and manufacturing activities. This funding comes from multiple sub- offices. For example, the Office of Research, Development, Test, and Evaluation provides funding for microelectronics research and development; the Office of Production Modernization provides funding for, among other things, refurbishing microelectronics processing capabilities; and the Office of Stockpile Management provides funding for microelectronics production, according to an NNSA official and NNSA documentation. The Office of Safety, Infrastructure and Operations accounts for 46 percent (or about $78 million), and this office is responsible for ensuring existing facilities are safely operated, effectively managed, and maintained to meet mission needs. DOE’s Strategic Partnership Programs account for about 13 percent (or $36 million) of the MESA Complex’s fiscal year 2020 budget. These programs include research and development projects sponsored by the Air Force and the Defense Advanced Research Projects Agency. DOE’s Laboratory Directed Research and Development work accounts for about 10 percent (or $28 million) of the MESA Complex’s fiscal year 2020 budget. Each of DOE’s 16 contractor-operated laboratories—including Sandia—may direct a portion of the funding they receive from DOE to scientists who conduct independent research. The statutory limit on this laboratory-directed research and development work is between five to seven percent of funds provided by DOE to the laboratories for national security activities. DOD provides about 6 percent (or $17 million) of the MESA Complex’s fiscal year 2020 budget through Strategic Partnership Programs. According to Sandia documentation, this funding comes directly from the Air Force and Navy to support the production of microelectronics that are integrated into nuclear weapon delivery platforms. Other sources account for about 12 percent (or $34 million) of the MESA Complex’s fiscal year 2020 budget. Among other things, this funding comes from indirect rates applied to all Sandia programs to support the MESA Complex’s management and operations. Over the past decade, NNSA completed several actions to sustain its existing strategic radiation-hardened microelectronics facilities at Sandia through 2025 while simultaneously identifying future alternatives for its microelectronics capability beyond 2025. In particular, during fiscal years 2012 through 2019, NNSA engaged in a $150 million effort at Sandia to sustain operations at the SiFab Facility through 2025. NNSA pursued this effort in response to a 2010 study conducted by Sandia that identified the need for millions of dollars in funding to sustain the SiFab Facility through 2025. NNSA’s sustainment efforts focused on the following two areas: Infrastructure. NNSA spent about $27 million to complete approximately 25 infrastructure projects that support microelectronics production. For example, NNSA installed two new 20,000-gallon tanks for water storage to improve the facility’s deionized water system, which provides ultra-high purity water for use in certain processing steps. NNSA also replaced a portion of the facility’s acid exhaust system. Equipment. NNSA spent about $123 million on production equipment for two main purposes: (1) to replace aging equipment that Sandia classified as being at high risk of failure; and (2) to refurbish existing equipment and procure equipment that will be used to produce microelectronics once Sandia completes its ongoing effort to convert the production process from using 6-inch silicon wafers to 8-inch wafers. Prior to these equipment investments, the SiFab Facility relied on aging equipment to perform certain processing steps using a manual process. In fiscal year 2018, Sandia refurbished existing equipment and purchased new equipment that is more automated and is intended to increase process reliability. In addition, according to Sandia documentation, Sandia needed to convert its production process to use 8-inch silicon wafers because the commercial sector had increasingly limited maintenance support and service for equipment that processed 6-inch wafers. While NNSA was working with Sandia to sustain the SiFab Facility through 2025, the agency also began identifying and evaluating options for producing microelectronics after 2025, such as constructing a new multibillion-dollar production facility at Sandia. However, because of changes to key assumptions, NNSA decided in November 2018 not to pursue any of the identified alternatives and instead stated that the agency was going to assess options to sustain its current capability at Sandia beyond 2025. See figure 4 for a summary of NNSA’s actions to sustain the SiFab facility and consider alternatives. More specifically, NNSA took the following actions during the past decade to identify alternatives for producing microelectronics beyond 2025: In 2011, NNSA’s Deputy Administrator for Defense Programs requested proposals from the agency’s three nuclear weapons laboratories for flagship experimental science, technology, and engineering facilities to help ensure that NNSA will have the capabilities to address future national security needs. In response, Sandia submitted a proposal to NNSA in 2012 to construct a new, multibillion-dollar microelectronics production facility, called the Center for Heterogeneous Integration, Packaging, and Processes (CHIP2). The Sandia proposal estimated that CHIP2 would take 14 years to design and build at an estimated cost of $2.5 billion. The proposal indicated that the facility would increase microelectronics functionality and trustworthiness by creating a trusted supply chain into the future for design, fabrication, testing, and packaging activities. As a result of the time needed to design and construct CHIP2, investment would still be needed to sustain the MESA SiFab Facility through 2025. NNSA commissioned two studies by The Aerospace Corporation, a federally funded research and development center sponsored by the Air Force, to help the agency evaluate Sandia’s CHIP2 proposal against other potential alternatives, such as contracting with commercial entities to produce microelectronics. These studies, completed in August and September 2014, generally ranked the CHIP2 proposal at or near the top of the alternatives but also stated that CHIP2 did not stand out as a decidedly better option. Nonetheless, in early 2015, NNSA’s Deputy Administrator for Defense Programs issued a memorandum recommending that NNSA pursue the CHIP2 proposal as a formal capital asset project, subject to DOE’s project management order on acquisition of capital assets. In 2016, in accordance with DOE’s project management order, NNSA developed two key documents during the initiation phase of its capital asset project supporting the CHIP2 proposal, which NNSA referred to as the Trusted Microelectronics Capability (TMC) project. NNSA first developed a mission need statement, which is a formal document that identifies a credible performance gap between current capabilities and those needed to achieve the goals stated in the agency’s strategic plan. The mission need should be stated in a way that is solution-neutral. The project’s mission need statement stated that, among other things, after 2025 the SiFab Facility faced a severe risk of equipment and facility failures that could have detrimental impacts on future microelectronics production schedules. The statement noted that continued refurbishment of the SiFab Facility beyond 2025 could result in significant downtime during critical weapon development and production cycles, as the facility was constructed in the 1980s and was not sized for modern microelectronics production equipment and supporting infrastructure. NNSA next developed a requirements document, which describes the ultimate goals the project must satisfy while also identifying key assumptions and constraints. The requirements document identified several key requirements, including that the TMC project must be able to provide NNSA with trusted access to produce microelectronics in support of the agency’s nuclear weapons mission. Between 2016 and 2017, in accordance with DOE’s project management order, NNSA conducted an analysis of alternatives for the TMC project based on achieving NNSA’s mission need statement. Such an analysis identifies, analyzes, and selects a preferred alternative to best meet the mission need by comparing the operational effectiveness, costs, and risks of potential alternatives, according to DOE documentation. During this process, NNSA considered 21 alternatives for meeting the mission need statement, among them the CHIP2 proposal as well as several alternatives that included partnerships with commercial industry and other government production facilities. The final TMC analysis of alternatives report, dated January 2018, did not identify the CHIP2 proposal as a preferred alternative because of the proposal’s high life-cycle costs, high total project cost, and long project schedule. Instead, the report identified two preferred alternatives as best meeting NNSA’s needs: (1) partnering with an existing, government-owned, contractor- operated production facility other than Sandia; and (2) entering into an interagency agreement with DOD and at least one member of the intelligence community, as well as a commercial entity, to design, build, and operate a state-of-the-art production facility. Ultimately, NNSA decided not to pursue either preferred alternative because of changing assumptions. For example, one of NNSA’s key assumptions for the TMC analysis of alternatives was that the SiFab Facility could not remain operational beyond 2025. However, NNSA tasked The Aerospace Corporation to validate this assumption, and in January 2018, The Aerospace Corporation completed a study concluding that the SiFab Facility could remain viable until 2040 with prioritized and well-planned infrastructure repairs and equipment replacements. Another example of changing assumptions concerned the preferred alternative under which NNSA would enter into an interagency agreement with DOD and at least one member of the intelligence community to design, build, and operate a state-of-the-art production facility. This preferred alternative assumed that DOD, the intelligence community, or both, would pay to develop and build the production facility (estimated to cost from $350 million up to $1.2 billion), while NNSA would pay to equip its portion of the production process. The TMC analysis of alternatives report stated that commitment from DOD and the intelligence community would be vital, and that this alternative carried significant execution risks. In January 2018, NNSA documentation stated that this interagency alternative was no longer viable because other agencies stated they were no longer interested in a potential partnership. Partly as a result of these changes in key assumptions, in November 2018, NNSA wrote in a letter to Congress that it was no longer requesting funding for the TMC and was assessing what investments were needed to extend the operational life of the SiFab Facility to 2040. As part of NNSA’s ongoing approach to managing its strategic radiation- hardened microelectronics activities, the agency plans to upgrade and sustain its microelectronics capability at Sandia through 2040, which it estimates will cost about $1 billion over the next 20 years. NNSA is also in the preliminary stages of identifying and evaluating options for a microelectronics capability beyond 2040. In addition, NNSA is starting to implement a revised management approach, including appointing a coordinator to guide certain aspects of its microelectronics activities. However, NNSA’s approach does not fully incorporate key management controls, such as developing an overarching management plan, which the agency has applied to other important activities. In 2019, NNSA made three key decisions related to upgrading and sustaining its microelectronics capability at Sandia through 2040. First, NNSA approved plans to further upgrade its process for producing microelectronics. This upgraded process, called CMOS8, contains some features of the currently employed CMOS7 process, but is a more advanced technology node that also includes many new features, according to Sandia documentation. Second, NNSA approved plans to produce and integrate into future nuclear weapons a more advanced type of microelectronics component called a field programmable gate array (FPGA). According to Sandia documentation, strategic radiation- hardened FPGAs can be produced using the CMOS8 process but not the CMOS7 process. Third, Sandia developed and NNSA approved a plan to identify, prioritize, and provide budget estimates to sustain Sandia’s microelectronics infrastructure and equipment at the MESA Complex over the next 20 years. This plan incorporates NNSA’s decisions to develop the CMOS8 process and produce FPGAs. According to NNSA and Sandia documents, the rationale behind and expected benefits of these three key decisions are as follows: The CMOS8 process will allow Sandia to produce microelectronics at a smaller, more advanced technology node (180nm) compared with the current CMOS7 technology node (350nm). NNSA documentation states that, among other things, the CMOS8 process is expected to produce microelectronics that have twice the processing speed compared with those produced using the CMOS7 process. Such advances are needed to help ensure that future nuclear weapons remain safe, secure, and reliable while operating in increasingly hostile threat environments and that the weapons meet increased performance requirements, according to Sandia documentation. According to NNSA officials, the agency agreed with Sandia’s assessment on implementing the CMOS8 production process based, in part, on findings and recommendations contained in an independent study commissioned by NNSA and completed by multiple entities including The Aerospace Corporation. According to Sandia documentation, while FPGAs have never been used before in a nuclear weapon, they may significantly reduce the cycle time for microelectronics research, development, and production compared with cycle times for ASICs used in nuclear weapons. This reduction may be possible because the ASICs currently used in nuclear weapons are uniquely designed and produced to carry out specific functions, whereas FPGAs can be produced using a common design and then programmed after production (but before insertion into a nuclear weapon) to carry out different functions, according to NNSA officials. Reduced cycle time from FPGAs could alleviate schedule pressure on future weapon modernization programs because cycle times for designing and producing ASICs for LEPs have historically been about 10 years before production of the first weapon, according to Sandia documentation. Sandia’s plan will provide NNSA with the basis for the investment profile needed to sustain the MESA Complex’s infrastructure and equipment through 2040. Because the sustainment effort will last at least 20 years, NNSA officials said that having a long-term planning document that provides a current baseline for the condition of Sandia’s microelectronics infrastructure and equipment, identifies challenges, and recommends specific sustainment activities will be a useful management tool. The plan for extending the life of the MESA Complex at Sandia provides cost and schedule estimates related to sustainment of existing facilities and equipment, as well as installation of new equipment for CMOS8 and development and maturation of the FPGA technology. Overall, the plan calls for spending about $1 billion over the next 20 years. Specifically, the plan identifies spending for the following activities: Sustainment of existing facilities and equipment. The plan identifies about $900 million in spending from fiscal years 2020 through 2040—or about $45 million a year for the next 20 years—to complete identified infrastructure and equipment projects. The plan calls for spending roughly half of the $900 million on projects to upgrade existing infrastructure within the MESA Complex. In particular, Sandia plans to spend about $120 million from fiscal years 2020 through 2024 on projects to improve or upgrade infrastructure within the SiFab Facility that is considered to be in “poor condition” based on information contained in NNSA’s infrastructure condition database. The SiFab Facility is to be the physical location for the majority of production tools for CMOS8. Two of these projects would replace electrical power and distribution equipment at an estimated cost of about $50 million, while another project would replace the facility’s chemical distribution system at an estimated cost of about $5 million. Sandia plans to spend the other half of the $900 million on equipment-related projects. For example, Sandia plans to spend about $85 million from fiscal years 2021 through 2026 on projects to support existing, non-CMOS8 production processes—such as producing transistors in the Micro Fabrication Facility—as well as activities that support microelectronics production, such as laboratory analysis, testing, and packaging. For example, Sandia plans to spend $1.5 million on a computerized tomography machine to support microelectronics testing. Development of CMOS8 and production of FPGAs. The MESA Complex extended life plan identifies about $170 million in spending from fiscal years 2020 through 2027 related to developing, maturing, installing, and implementing the CMOS8 process and the FPGA technology. Sandia contractor representatives told us that the CMOS8 process relies on newer and more advanced equipment to complete critical individual processing steps compared with the current CMOS7 process. As a result, the plan identifies about $70 million (out of the $170 million total) to acquire approximately 30 pieces of equipment, which Sandia will need to install and then qualify their performance. In addition, the plan identifies almost $90 million (out of the $170 million total) for developing and maturing the CMOS8 production process and the FPGA technology. According to Sandia documentation, Sandia plans to begin using the CMOS8 process to produce FPGAs for integration into a future nuclear weapon program at the end of fiscal year 2027. In addition to upgrading and sustaining Sandia’s microelectronics capabilities through 2040, NNSA is in the preliminary stages of identifying and evaluating options to ensure a continued microelectronics capability beyond 2040, according to NNSA officials and documentation. In particular, NNSA has identified the following two key options: NNSA is in the initial stages of identifying and evaluating options to construct a new facility for producing microelectronics by 2040 and beyond. In December 2019, NNSA officials provided us with documentation stating that the agency plans to begin evaluating options for a new microelectronics facility in 2021 with the goal of completing construction in 2030, installing needed equipment in the completed facility by 2033, and qualifying the production process and begin producing microelectronics for integration into nuclear weapons no later than 2035. In NNSA’s fiscal year 2021 budget request, which was released in February 2020, the agency requested funds to begin evaluation and early planning activities for this new microelectronics facility. NNSA is also evaluating whether the agency might be able to leverage a recent investment by DOD in a U.S. commercial microelectronics production facility to help meet NNSA’s microelectronics production needs after 2040. Specifically, DOD announced in October 2019 that it had awarded a contract to a U.S.- owned-and-operated microelectronics commercial production facility to, among other things, enhance its radiation-hardened microelectronics production process to meet DOD’s microelectronic needs for systems (such as satellites) that operate in environments with increased radiation levels. Over the next two years, the U.S. commercial microelectronics production facility plans to adapt its current production process and develop a new process that will produce microelectronics at a smaller node, according to DOD documentation. According to NNSA officials we interviewed in February 2020, NNSA and DOD are in preliminary discussions to determine if NNSA could make additional investments in this same facility to potentially produce strategic radiation-hardened microelectronics for integration into nuclear weapons. NNSA officials said that there was no firm timeframe for making an investment decision because such a decision would need to be made after the microelectronics facility begins producing microelectronics at the smaller node. NNSA is starting to implement a revised approach to managing its microelectronics activities. During our initial interviews with NNSA officials in early 2019, they stated that NNSA had not established a formal management structure to oversee the agency’s microelectronics activities. Instead, they said that NNSA had delegated primary responsible for overseeing such activities to two officials within NNSA’s Office of Defense Programs, who both served in multiple positions and had other duties within the office. According to these officials, once NNSA formally canceled the TMC project in November 2018, management efforts were focused on making initial determinations on the actions and budget estimates needed to sustain NNSA’s existing microelectronics capability at Sandia until 2040. These efforts included coordinating with multiple NNSA offices—such as the Office of Safety, Infrastructure and Operations—to understand their future microelectronics needs and requirements and to review draft MESA Complex sustainment documentation prepared by Sandia. However, officials from NNSA’s Office of Defense Programs told us that in late 2019 they determined that a more coordinated management approach would better position NNSA to oversee microelectronics activities and make informed budgetary and programmatic decisions. Specifically, NNSA officials stated that in November 2019 the Office of Defense Programs created and filled a new full-time microelectronics coordinator position within a sub-office, the Office of Research, Development, Test, and Evaluation. The microelectronics coordinator told us that NNSA has not yet finalized an official position description for the coordinator role. However, the coordinator said that the position will primarily be responsible for developing the CMOS8 process and the FPGA technology and integrating the research and development activities of the Office of Research, Development, Test, and Evaluation with another sub-office, the Office of Production Modernization. In addition, officials from NNSA’s Office of Defense Programs and Office of Safety, Infrastructure and Operations told us that they continue to use other existing processes to manage microelectronics activities at Sandia. For example, these officials said that they use the annual planning, programming, budgeting, and evaluation process, along with the annual work authorization process, to coordinate across NNSA offices on budgetary matters and work activities associated with microelectronics activities at Sandia. As part of these processes, agency officials told us that they issue annual implementation plans to direct the work of Sandia contractors related to microelectronics activities. NNSA officials then monitor the contractors’ progress toward completing the identified scope of work and work activities. For example, NNSA officials said that they conduct monthly meetings with contractor representatives to review status and financial reports. They also said that they hold mid-year and end-of-year program reviews with contractor representatives. To help management achieve desired results—such as ensuring a continued microelectronics capability—federal agencies design, implement, and operate internal controls, which comprise the plans, methods, policies, and procedures used to fulfill an entity’s mission, goals, and objectives. Federal standards for internal control state that management should, among other things: design control activities, such as by developing policies, procedures, techniques, and mechanisms that enforce management’s directives, to achieve objectives and respond to risk; and establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. NNSA has implemented internal controls at the agency level, in part, by developing and implementing directives that provide an organizational structure for the agency to plan, execute, control, and assess its programs and projects while also assigning responsibility and delegating authority for key management roles. For example, one purpose of NNSA’s 2019 program management directives is to increase management efficiency and effectiveness by, among other things, clearly defining management responsibilities and authorities. In addition, DOE’s project management order for the acquisition of capital assets lists principles for successful project execution such as disciplined, up-front planning; line management accountability; and effective implementation of all management systems (such as risk and performance management) supporting the project. In particular and as applicable to front-end planning, NNSA’s and DOE’s directives related to program and project management both include the following controls: Appointment of a federal manager, who is vested with the authority to carry out assigned responsibilities to meet program or project milestones on schedule and on budget, who manages the coordination of deliverables between the multiple entities (such as different program offices) involved, and who is responsible and accountable for planning, implementing, and executing a program or project, which includes responsibility for developing an overarching management plan; An overarching management plan, which establishes the procedures to define, execute, and monitor a program or project, as well as establishing specific requirements in a variety of areas—such as cost estimating, an integrated schedule, performance management, and risk management—to use to develop a baseline and against which to measure and monitor; A mission need statement, which identifies a credible gap between current capabilities and those needed to achieve the goals stated in the strategic plan; and A requirements document that describes the ultimate goals the program or project must satisfy while also identifying key assumptions and constraints. However, while some in NNSA and at Sandia have recognized the need to coordinate microelectronics activities to effectively carry them out and meet specific goals by specific dates, as evidenced by the hiring of a coordinator, Office of Defense Programs leadership have not fully developed controls to better manage and coordinate its microelectronics activities. Specifically, NNSA does not have or has not fully developed the following: Federal manager with coordination or oversight authority. NNSA has not established a federal management position with the authority and accountability to better coordinate or oversee NNSA’s microelectronics activities. Instead, as described above, agency officials told us that NNSA’s Office of Defense Programs established a coordinator position—within a sub-office, the Office of Research, Testing, Development and Evaluation—in November 2019 to help guide the agency’s efforts to develop the CMOS8 process and the FPGA technology, among other things. Moreover, in May 2020, NNSA stated that senior leadership within the Office of Defense Programs have not endorsed the formal role of a microelectronics coordinator and that the coordinator’s role and responsibilities are currently under review. NNSA also stated that the coordinator has not been given authority to manage an annual budget for microelectronics activities and that it was unlikely that such authority would be granted. This statement stands in contrast to earlier statements made to us that the coordinator would have responsibility for an annual budget of about $50 million, subject to future appropriations. Management plan. NNSA has not developed an overarching management plan to guide and coordinate the agency’s microelectronics activities. Instead, NNSA officials from the Office of Defense Programs and the Office of Safety, Infrastructure and Operations told us that the agency is in the very early stages of developing a NNSA plan that will incorporate key decisions and approaches outlined in the Sandia’s 20-year MESA sustainment plan, among other things. While NNSA officials are still evaluating the specific contents of this plan, they said that the plan may outline specific roles and responsibilities for each NNSA office involved in microelectronics, describe how these offices will interact with the microelectronics coordinator, and provide options for future microelectronics technology development efforts. However, it is unclear whether the document will define the planning approach, procedures, and processes that NNSA will use to ensure coordinated management in multiple areas and across multiple offices, such as developing cost estimates, an integrated schedule, and performance metrics. Agency officials said that this plan, when finalized, will provide a useful tool for coordinating various aspects of NNSA’s microelectronics activities, but they did not provide an estimated date for when the plan will be completed. Mission need statement and requirements document. NNSA has not developed a current mission need statement or a current program requirement document. In 2016, as required by DOE’s project management order on the acquisition of capital assets, NNSA issued a formal mission need statement and a requirements document to guide its assessment of the cancelled TMC project (as described earlier in this report). However, agency officials told us that these 2016 documents are no longer applicable to NNSA’s current approach to sustaining its microelectronics capability and evaluating options to ensure a continued capability after 2040. NNSA officials said that they intend to establish an updated set of requirements to guide the agency’s future microelectronics capability, and that they will consider these requirements in establishing a future mission need statement. However, NNSA officials did not provide a timeframe for finalizing these documents. NNSA officials acknowledged the importance of using management controls and that the controls described above would be useful, but they could not identify any specific DOE or NNSA directives, government-wide guidance, or best practices that they follow to manage their microelectronics activities. Instead, they offered three reasons why the agency has not implemented a more coordinated and robust set of management controls to oversee the agency’s microelectronics activities: Microelectronics production has historically been managed as a component production effort by an LEP, which is led by an NNSA program manager within the Office of Defense Programs who coordinates directly with other NNSA offices and Sandia contractors. Because NNSA has not designed microelectronics as a formal program, the requirements contained in the agency’s program management directives are not binding on microelectronics activities. NNSA officials said that the multiple projects (identified in the MESA Complex extended life plan) to upgrade and sustain the microelectronics capabilities at Sandia through 2040—at an estimated cost of over $1 billion over 20 years—will not be subject to DOE’s project management order, as these projects are for sustainment and not for new facility construction. According to officials from NNSA’s Office of Safety, Infrastructure, and Operations, infrastructure investments are being planned and managed as maintenance and repair efforts. NNSA officials told us that the agency’s current efforts provide the necessary structure for NNSA to oversee and manage its microelectronics capability. However, NNSA has recognized the importance of implementing a more coordinated and robust set of management controls for other important activities within its nuclear security mission that similarly have not been treated in the past as specific programs. For example, as we reported in June 2019, while NNSA historically managed its high-explosive capability without a formal mechanism to coordinate activities across multiple programs, it recently implemented a more robust set of management controls to oversee its high-explosive activities. Specifically, in 2018 NNSA appointed an enterprise manager to help coordinate these activities. NNSA also encouraged the enterprise manager to adopt, where appropriate, the program management controls contained in an NNSA directive on managing nuclear weapon life extension and strategic materials programs. Subsequently, the enterprise manager issued a strategic plan that provided an organizational structure for the agency’s high explosives capability. By taking a similar approach to its management of microelectronics activities and incorporating a more coordinated and robust set of management controls, the agency would have increased assurance that its planned microelectronics activities are clearly defined, efficiently executed, and effectively monitored. NNSA’s ability to produce unique microelectronics for nuclear weapons is essential to ensuring a credible U.S. nuclear deterrent. Producing such microelectronics is a complex task, and NNSA is limited in its ability to partner with the commercial sector for such production. Over the next two decades, NNSA will undertake an expensive and ambitious approach to upgrade and sustain its existing microelectronics production facilities and capabilities. Specifically, NNSA plans to spend about $1 billion over the next 20 years to, among other things, upgrade its process to produce a new type of microelectronic component that has never been integrated into a nuclear weapon. In addition, NNSA officials said that the agency will need to identify and analyze options for a continued capability after 2040, and that effort could begin as early as 2021. To increase its management and oversight of the agency’s microelectronics activities, NNSA has taken some positive steps such as appointing a microelectronics coordinator within the Office of Defense Programs and approving certain long-term planning documents. However, in contrast to other NNSA activities, including programs and projects, NNSA has not fully developed a coordinated and robust set of management controls to oversee its microelectronics activities. For example, NNSA has not established an overarching management plan to manage and coordinate the cost, schedule, and risks associated with its microelectronics activities. By incorporating a more coordinated and robust set of management controls, NNSA would have increased assurance that its planned microelectronics activities are clearly defined, efficiently executed, and effectively monitored. The NNSA Administrator should incorporate additional management controls to better oversee and coordinate NNSA’s microelectronics activities. Such management controls could include investing the microelectronics coordinator with increased responsibility and authority, developing an overarching management plan, and developing a mission need statement and a microelectronics requirements document. (Recommendation 1) We provided a draft of this report to DOD and NNSA for review and comment. DOD did not provide any comments. In its written comments, reproduced in appendix I, NNSA neither agreed nor disagreed with our recommendation but provided three main comments. First, NNSA stated that by December 2020 the agency plans to complete a strategic management plan that will more clearly articulate the integration of management controls for the various components of its microelectronics activities. NNSA stated that it believes this action is consistent with our recommendation. We are encouraged by this planned action and will evaluate the completed strategic management plan to determine if it meets the intent of our recommendation. Second, NNSA stated that our report did not clearly convey the differences between the management of microelectronics and other weapons or materials programs and did not include all aspects of its microelectronics activities (such as the procurement of commercial off the shelf components) in our audit’s scope. In response, we added references to the various aspects of NNSA’s microelectronics activities and clarified that our report focuses on NNSA’s strategic radiation- hardened microelectronics activities at Sandia’s MESA Complex. As stated in the report, we focused on this specific aspect of NNSA’s microelectronics mission because of the language in the Senate committee report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2019, which included a provision for us to review NNSA’s efforts to recapitalize its strategic radiation-hardened microelectronics design and production capacity. We also focused on this specific aspect of NNSA’s mission because the fiscal year 2020 Stockpile Stewardship and Management Plan lists the continued production of strategic radiation-hardened microelectronics as one of four key challenges to the agency’s nuclear stockpile mission. Third, NNSA stated that our audit did not include an assessment of management controls for the range of activities that work together to ensure the effectiveness of microelectronics planning and execution. However, our report identifies and describes these management controls, and as part of our work we considered how these controls work together. In addition and as stated above, NNSA intends to complete a strategic management plan to more clearly articulate the integration of its various microelectronics management controls, which is especially important as the agency invests about $1 billion dollars over the next 20 years while simultaneously needing to meet microelectronics production deliverables for multiple nuclear weapon modernization programs. NNSA also provided technical comments, which we incorporated in our report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, the Secretary of Defense, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or at bawdena@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Allison B. Bawden at (202) 512-3841 or bawdena@gao.gov In addition to the contact named above, Jason Holliday (Assistant Director), Patrick Bernard (Analyst in Charge), and Alisa Carrigan made key contributions to this report. Also contributing to this report were Jonathan Felbinger, Juan Garay, Lisa Gardner, Cindy Gilbert, Cynthia Norris, and Dan C. Royer.", "summary": "Microelectronics (see figure) form the basis of nearly all electronic products, including nuclear weapons. U.S. nuclear weapons use a unique supply of “strategic radiation-hardened” microelectronics that must function properly when exposed to high levels of radiation. NNSA's facilities at Sandia are the only source for these unique microelectronics, and the age of the facilities may pose significant risk to NNSA's capability after 2025. A Senate committee report accompanying the National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to review NNSA's strategic radiation- hardened microelectronics activities. This report (1) describes NNSA's actions over the past decade to sustain existing facilities and identify future alternatives; and (2) examines NNSA's ongoing approach to managing its microelectronics activities and the extent to which this approach incorporates key management controls. GAO reviewed documents and interviewed officials and contractor representatives from NNSA and Sandia, toured Sandia's microelectronics facilities, and reviewed NNSA program and project management controls. Over the past decade, the Department of Energy's (DOE) National Nuclear Security Administration (NNSA) completed several actions to sustain the condition of its existing microelectronics facilities at Sandia National Laboratories (Sandia), which are NNSA's only source for producing strategic radiation- hardened microelectronics that can operate in environments with extreme exposure to radiation. In particular, during fiscal years 2012 through 2019, NNSA carried out a multiyear, $150-million effort at Sandia to replace or refurbish infrastructure and equipment in its primary microelectronics production facility to ensure continued operations through 2025. While NNSA was working with Sandia to sustain current facilities, the agency also began identifying and evaluating options for producing microelectronics after 2025, including constructing a new multi-billion dollar production facility at Sandia. However, because of changes to key assumptions, including longer-term viability of existing facilities, NNSA decided in November 2018 not to pursue any of the identified alternatives and instead stated that the agency was going to assess options to sustain its current capability at Sandia. NNSA's ongoing approach to managing its strategic radiation-hardened microelectronics activities includes two key efforts. First, the agency decided in October 2019 to invest about $1 billion over the next 20 years to upgrade and sustain its microelectronics capability at Sandia through 2040. Specifically, NNSA plans to upgrade its production process as well as complete identified infrastructure (such as electrical distribution) and equipment projects. Second, in November 2019 NNSA created and filled a new full-time microelectronics coordinator position that, among other things, will have responsibility for certain aspects of the agency's microelectronics activities, according to agency officials. However, NNSA's approach does not fully incorporate key management controls that NNSA applies to other important activities. For example, DOE and NNSA require their programs and projects to establish an overarching management plan that describes the procedures to define, execute, and monitor a program or project as well as establishing specific requirements in a variety of areas such as cost estimating and performance management. NNSA has not established a similar management plan to oversee and coordinate its microelectronics activities. By incorporating these key management controls, NNSA would have increased assurance that its planned microelectronics activities are clearly defined, efficiently executed, and effectively monitored. GAO recommends that NNSA incorporate additional management controls, such as developing an overarching management plan, to better oversee and coordinate its microelectronics activities. NNSA neither agreed nor disagreed with this recommendation.", "document_type": "gao"}
{"report": "Generally, Congress provides budget authority to agencies through the passage of appropriations acts each fiscal year. Appropriations allow agencies to incur obligations and make payments for specified purposes. When an appropriation expires, and a new one is not enacted, a lapse in appropriations, also called a funding gap, results and the affected agency or program may lack sufficient budget authority to continue operations. Funding gaps can occur at the beginning of a fiscal year when new appropriations, or a continuing resolution, have not yet been enacted. Funding gaps also can occur any time during the year when a continuing resolution expires and may affect a few agencies or all agencies across the federal government. We have previously reported that funding gaps, actual or threatened, are both disruptive and costly. The ADA prohibits agencies from obligating or expending funds in excess or in advance of an available appropriation unless otherwise authorized by law as well as from accepting voluntary services for the United States except in cases of emergency involving the safety of human life or the protection of property. During a lapse in appropriations, employees may continue working if they are exempt from the lapse in appropriations or if an exception to the ADA applies (see figure 1). Exempt and excepted employees are defined as follows. Exempt employees are those who perform activities funded with budget authority that remains available despite the lapse in appropriations, such as multiple-year or no-year carryover balances. Available balances can also come from other authorities such as fee income that Congress made available for obligation. For the purpose of this report, we call employees who perform such functions exempt employees. Excepted employees are those who perform activities pursuant to a statutory authority that expressly authorizes an agency to enter into an obligation in advance of an appropriation, or to address emergencies involving the safety of human life or the protection of property, as described under the ADA. We have also recognized, in our prior legal opinions, other limited exceptions that may, under some circumstances, allow functions to continue during a lapse in appropriations. For example, Congress and the Executive branch may incur obligations to carry out core constitutional powers. Agencies also may incur those limited obligations that are incidental to executing an orderly shutdown of agency activity. Over the past 29 years, there have been six lapses in appropriations that led to government shutdowns, ranging in duration from 2 days to 35 days (see figure 2). Three shutdowns occurred in the past 7 years and two of these shutdowns were prolonged in that they lasted longer than 5 days (in fiscal years 2014 and 2019). In the event of a government shutdown, OMB is responsible for ensuring that agencies have addressed the essential actions needed to effectively manage the government shutdown. OMB does so by providing policy guidance and shutdown-related instructions. Specifically, OMB Circular A- 11 directs federal agencies to develop contingency plans for use in the event of a government shutdown and to update these plans on a recurring basis. These plans are key documents that help ensure an orderly shutdown following a lapse in appropriations, as well as continuity of appropriate agency operations. These plans also communicate policies and procedures to employees and external stakeholders that could be affected by the shutdown of operations. OMB’s Circular A-11 directs agencies to prepare contingency plans in anticipation of a lapse in appropriations. According to the guidance, contingency plans are to include information such as: (1) summaries of activities that will continue and those that will cease; (2) the amount of time needed to complete the shutdown activities; (3) the number of employees on-board prior to the shutdown; and (4) the number of employees to be retained during the shutdown. Agencies are also to explain the legal basis for each of their determinations to retain employees, including a description of the nature of the agency activities in which these employees will be engaged. Additionally, agencies’ contingency plans are to explicitly describe any changes in operations that would be necessary should a lapse in appropriations extend past 5 days. According to OMB officials, OMB reviews agencies’ contingency plans, but it does not formally approve plans. Agencies are ultimately responsible for determining which activities will continue during a lapse in appropriations and which activities will cease. Using OMB Circular A-11, we identified 14 key information elements for agencies’ contingency plans and used these as criteria to assess the selected agencies’ plans. Of the selected agency components, ITA and CBP operated under the contingency plans of their respective agencies. Similarly, USTR officials said that the component operated under EOP’s contingency plan. IRS, in contrast, had its own contingency plan for the fiscal year 2019 shutdown because Treasury did not have a department- wide plan. Agency contingency plans governing the shutdown operations of CBP, IRS, and ITA included most of the key information elements described in OMB Circular A-11. EOP did not address a majority of the key information elements in its contingency plan, which governed USTR’s shutdown operations. Figure 3 shows how selected agencies’ contingency plans aligned with OMB’s guidance. Three of our four selected agencies—Commerce, IRS, and DHS— provided summary information at the beginning of their contingency plans about activities that would and would not continue during a lapse in appropriations. EOP’s contingency plan did not include any information on activities that would and would not continue. The following table shows examples of exempt and excepted work activities from our selected agencies’ contingency plans (see table 2). All four agencies provided the total number of employees on-board before the shutdown and how many would continue to work during the government shutdown. However, EOP’s contingency plan did not break down these employees by ADA exception categories that may include addressing emergencies involving the safety of human life or the protection of property or carrying out core constitutional powers, as specified in OMB guidance. While the break out of employees by ADA exception category was not in the DHS department-wide plan, CBP’s component-level portion of the plan, which is not publically available because of law enforcement sensitivities, contained these details. This information is important for an agency to ensure it has proper oversight of operations and the right personnel performing excepted work to be in compliance with the ADA. None of the agencies we reviewed provided a complete description of potential changes to their activities and operations in the case of a prolonged lapse in appropriations—one lasting longer than 5 days—within their contingency plans. Officials at some of the selected agencies told us that the purpose of their contingency plans was only to document operations for the first 5 days of a shutdown, contrary to what is required in OMB’s Circular A-11 for planning and documenting operations in the anticipation of a potential prolonged shutdown. While three of the four agency contingency plans that we reviewed— Commerce, IRS, and DHS—provided some minimal details on how operational changes would be made in the event of a prolonged shutdown, such as designating personnel responsible, none provided the level of detail called for in OMB guidance. As discussed later, three of four selected agency components—ITA, IRS, and USTR—did have internal discussions on changes to operations in the event of a prolonged shutdown, according to officials. However, these discussions were not documented in the agency contingency plans. Given that shutdowns longer than 5 days have occurred in the past, it is important for agencies to consider and document the effects that a potential prolonged shutdown would have on operations in their contingency plans. Planning for potential prolonged shutdowns may assist the agencies with effectively managing changes in operations, and documenting these plans in public contingency plans may provide transparency to agency actions as a shutdown continues. OMB’s guidance states that if an agency anticipates changes during a potential prolonged shutdown, contingency plans should include information such as points in time when the furlough status of an employee may change, how many employees would be affected, and the legal basis for the changes. This information element is mentioned in two separate sections of Circular A-11 rather than in one consolidated location. Contingency plans for all selected agencies did not include complete information about (1) flexibilities available to supervisors if furloughed employees were unable to return to work on the day specified by the agency, including use of annual leave, compensatory time off, or credit hours; and (2) procedures for resuming program activities, including steps to ensure appropriate oversight and disbursement of funds upon the end of a shutdown, as specified in OMB’s guidance. Officials at selected agencies said that this information was available in internal guidance and fact sheets for employees, but that they did not include it in contingency plans, which are accessible to all employees during a shutdown. Including this information in contingency plans is important because it helps clarify agencies’ expectations for returning employees, and its inclusion may help agencies experience a more timely resumption of activities following a shutdown. As previously mentioned, USTR, as a component of EOP, operated under EOP’s contingency plan and did not have a separate plan for the fiscal year 2019 shutdown. While EOP’s contingency plan contained some information on USTR such as total number of employees on-board before the shutdown and employees to be retained during the shutdown, the plan did not fully address 10 of the 14 information elements outlined in OMB’s Circular A-11. Information that was not provided includes: (1) a breakout of exempt and excepted positions by category (e.g., available budget authority, emergencies involving safety of human life or protection of property, etc.); (2) summaries of activities that would or would not continue during a lapse; (3) designation of personnel responsible for implementing and adjusting the contingency plan if conditions change; and (4) methods for notifying employees that the shutdown has ended and when to return to work. Formal contingency plans that address the information elements specified in OMB guidance help agencies prepare for and oversee shutdown operations, and provide transparency to agency actions during a lapse in appropriations. Without a plan that covers these elements, USTR risks miscommunication with employees and other stakeholders that could negatively impact an orderly shutdown and the effective resumption of activities at the end of a lapse. Officials at IRS, ITA, and USTR discussed anticipated operational changes in the event of a prolonged shutdown internally while planning for the fiscal year 2019 shutdown. In one instance, an agency component documented these discussions in planning documents separate from agency contingency plans under which the component operated. However, as mentioned previously, potential operational changes were not documented in any of the contingency plans of our selected agencies. Potential operational changes generally involved recalling additional employees who had been furloughed at the beginning of the shutdown to carry out activities that the agencies categorized as excepted from the ADA or exempt due to other funding sources. IRS: IRS officials said that their initial planning was for a shutdown lasting 5 days or less. Within their contingency plan IRS noted that it would amend the plan if the shutdown lasted longer. On December 27, 2018, 6 days into the shutdown, IRS issued an updated contingency plan. According to IRS officials, this updated plan was assembled by contacting each of IRS’s 23 organizational offices to find any new activity requirements that would lead to changes in the contingency plan. In its amended plan, IRS added approximately 60 positions as excepted or exempt. Examples of activities that employees in these positions would support included: (1) communications efforts through IRS websites, (2) end-of-month financial operations, and (3) managing on-boarding for employees hired under Public Law 115-97, commonly referred to as the Tax Cuts and Jobs Act. IRS officials said they knew about these operational changes at the time of initial shutdown planning, but did not document all these operational needs. IRS program officials told us that being informed of anticipated operational changes as early as possible would have helped them prepare for the shift in workload. For its fiscal year 2020 contingency plan, IRS asked plan contributors to identify “as needed” positions that could be activated during a potential prolonged shutdown. Plan contributors also identified positions that would be needed if a shutdown lasting more than 5 days were to occur as IRS approached the tax filing season. ITA: Prior to the shutdown, ITA officials prepared a list of upcoming activities for the next 30 to 60 days to determine the potential scope of activities affected by a government shutdown. Activities included trade shows, meetings, and other critical operational deadlines. According to officials, ITA worked with their General Counsel to determine if activities could continue as excepted from the ADA or exempted because funding was available from another source. For those activities that could not be deemed excepted or exempt, ITA officials said that they were prepared to notify affected parties of the cancellation or postponement of the activities. For upcoming activities, ITA established dates when preparation would need to begin. In addition, ITA officials prepared temporary exception requests for employees to be recalled from furlough status in time to conduct needed preparation and carry out scheduled activities. ITA officials told us that they used and updated a tracker daily during the shutdown to ensure that all information remained current. Prior to the shutdown, ITA officials also said that they collected information on official travel planned for around the anticipated time of the shutdown. They said that it was important to gather this information because once employees were furloughed it becomes more difficult to gather complete and timely information on these travel plans. USTR: Prior to the shutdown, USTR officials asked offices to provide lists of positions that would need to be excepted during the first 2 weeks of a potential shutdown. This allowed USTR to anticipate operational needs if a shutdown lasted longer than 5 days. USTR officials said that flexibility was important as the potential shutdown approached because it allowed offices to adjust excepted position lists based on additional excepted activities or postponement of activities. USTR officials said that, in their experience, it is difficult to anticipate all the operational changes needed in the event of a shutdown longer than 2 weeks, especially as the agency component relies on partners at other agencies that may or may not be affected by the shutdown. CBP: Although the non-public portion of the DHS plan for CBP included sections that describe functions that may resume in the event of a prolonged shutdown, CBP officials said that these sections were not used in anticipation of the fiscal year 2019 shutdown. Specifically, the sections provide the opportunity for officials to indicate how many employees would be recalled to perform functions, but in the fiscal year 2019 plan almost every section indicates zero employees. Despite OMB guidance on prolonged shutdowns, officials said they believe that OMB guidance was exclusively for the first 5 days of a shutdown. Use of these sections of the CBP plan would help provide clearer expectations to the agency component’s workforce about who may be recalled to perform work activities during a shutdown. For the fiscal year 2020 contingency plan, CBP officials said that they asked offices to analyze and communicate what, if any, additional employees would be needed to work if a shutdown were to extend past 5 days. However, our review of the 2020 CBP plan found that, similar to the plan for fiscal year 2019, it largely does not indicate how many employees would be recalled to perform functions in the event of a prolonged shutdown. During the fiscal year 2019 shutdown, each agency component that we reviewed determined that changes needed to occur that affected the number of excepted employees working during the shutdown. According to agency component officials, these changes were due to the length of the shutdown, external events, and changes to the determination of excepted work. The length of the fiscal year 2019 shutdown was the most common reason cited by officials for operational changes. CBP: During the fiscal year 2019 shutdown, CBP responded to an increase in foreign nationals arriving at the southern U.S. border. In response to this external event, CBP officials told us that they identified a need to train additional law enforcement officers and agents to perform excepted activities. According to the DHS contingency plan, new hire training for law enforcement officers may be an excepted activity if the requesting agency component establishes a reasonable likelihood that a delay in new hire training would compromise the safety of human life or protection of property. According to DHS documents, this was a change from previous shutdowns, when new hire training was not an activity excepted from the ADA. CBP officials told us that they discussed this issue internally before the shutdown, but processing the change through DHS’s Chief Financial Officer, DHS General Counsel, and OMB occurred after the shutdown began. CBP has incorporated this change into its updated, non-public portion of DHS’s contingency plan. IRS: As the length of the shutdown increased, IRS identified mission requirements that it determined necessitated the recall of additional employees. For example, as it transitioned to its filing season operations, IRS recalled mail center employees to oversee the collection of taxes and protection of statute expiration. IRS’s updated fiscal year 2019 filing season contingency plan, published on January 15, 2019, incorporated this activity along with the additional 560 employees recalled for one division to perform the work. IRS said in the updated plan that the ADA exception for this work was the protection of life and property. According to IRS documents, Treasury officials evaluated plan updates for compliance with the ADA, and then shared the plan with OMB prior to implementing changes. IRS also made operational changes during the fiscal year 2019 shutdown that were based on changes to the determination of which work activities were excepted from the ADA. During the shutdown IRS announced that it would process tax returns beginning January 28, 2019, and refund taxpayers as scheduled. In 2011, OMB directed IRS not to pay tax refunds in the event of a lapse in appropriations. However, at the request of Treasury and IRS, OMB revisited this position and, on January 7, 2019, OMB informed Treasury that tax refunds may be paid during a lapse in appropriations. As a result of this determination, IRS added approximately 16,000 additional excepted positions to its filing season contingency plan for the purpose of issuing refunds. This change was documented in its updated contingency plan, published on January 15, 2019. In October 2019, we determined that the agency violated the ADA by processing tax returns and issuing refunds to taxpayers because it lacked available budget authority to support these activities and no exception to the ADA permitted IRS to incur these obligations. ITA: According to ITA officials, they updated ITA’s activity list during the course of the fiscal year 2019 shutdown. They said the update was needed to help determine which preparation activities could continue for future events, such as trade shows that bring international delegations, and which activities or events would have to be cancelled if the shutdown continued. ITA officials said they had to evaluate cancellation clauses in its contracts with these trade shows to decide whether and when to cancel. ITA recalled employees on a temporary basis, as needed, to perform these tasks. ITA officials told us that they followed departmental guidance in requesting employee recalls during the shutdown. ITA submitted proposed changes to Commerce’s Office of the Deputy Assistant Secretary for Administration, which coordinated department-level review and approval. Commerce officials told us that senior leadership discussed changes to the contingency plan with OMB officials over the course of the shutdown. Despite changes to the number of excepted employees, Commerce did not publish an updated contingency plan during the fiscal year 2019 shutdown. Commerce officials told us that, through discussions with OMB, they determined that publishing an updated plan was not necessary due to the relatively small number of changes to the total number of excepted and exempt employees. USTR: Prior to the beginning of the shutdown, USTR estimated that it could continue full operations for 3 to 4 weeks with available funding. Because the shutdown lasted beyond 3 weeks, USTR furloughed a majority of its employees on January 14, 2019, once those funds were no longer available. In the absence of available funding, USTR officials decided that some functions were excepted from the ADA under the justification that the agency component works to discharge the president’s constitutional duty and power to conduct foreign relations. USTR officials stated that component leaders identified the highest priority mission activities to continue during the shutdown, such as trade negotiations with China and work related to the North American Free Trade Agreement. Officials decided not to continue other activities, such as preparations for the 2019 Group of 20 Summit. USTR officials told us that, in consultation with OMB, USTR excepted more than 74 employees, the number listed in the EOP contingency plan published December 21, 2018. According to USTR documents, between 88 and 101 excepted employees were working during the last 2 weeks of the shutdown. Officials told us that these changes were made to carry out critical, excepted activities and that changes were communicated daily to EOP. Agency preparation for a government shutdown can require extensive changes in day-to-day operations. Having established policies and procedures prior to a shutdown can help agencies implement these changes successfully. Establishing these policies and procedures requires timely and transparent planning and communication to ensure that agencies function as effectively as possible during a shutdown. Internal controls related to planning for a government shutdown include designating roles and responsibilities, establishing processes for planning activities that help meet objectives, and documenting said processes. Internal controls related to communication prior to and during a government shutdown include ensuring that information communicated is timely, sufficient, and delivered to all appropriate individuals. Figure 4 summarizes the extent to which selected agency components incorporated applicable internal controls into their planning and operations prior to and during the fiscal year 2019 shutdown, as discussed in detail in the following sections. The agency components we reviewed identified staff needed to plan for the fiscal year 2019 shutdown and tasked each with certain responsibilities. According to Standards for Internal Control in the Federal Government (Internal Control Standards), agency component management should implement its control activities—processes, procedures, techniques, and mechanisms—through policies. Documenting roles and responsibilities for implementing the policies can help agencies meet their objectives related to managing a government shutdown. The following examples illustrate the roles and responsibilities of staff who helped determine which activities would continue during the fiscal year 2019 shutdown. CBP: CBP’s non-public portion of the DHS contingency plan described the key responsibilities and accountable parties for shutdown preparation. For instance, heads of offices determined which of their employees would remain at work to perform exempt or excepted functions during the shutdown. The CBP Hiatus Coordinator communicated daily with Hiatus Points of Contact within each CBP office who managed the offices’ shutdown processes. For instance, the Hiatus Points of Contact determined what functions would continue during the shutdown to help ensure activities aligned with OMB guidance. Officials said that CBP’s Office of Chief Counsel reviewed each excepted function to help ensure they met the legal standard for each ADA exception category. IRS: IRS internal process documents outlined the steps needed to prepare for a shutdown and the accountable parties for implementation. For example, IRS had a Lapse Program Manager who coordinated shutdown activities and helped develop the contingency plan, including identifying and evaluating excepted roles and aligning them with people, positions, and exception categories. IRS Chief Counsel was then responsible for reviewing the contingency plan for compliance with the ADA, followed by a review from Treasury’s General Counsel. According to the process documents, Treasury ultimately approves IRS’s contingency plan. ITA: ITA employed a “bottom-up” shutdown planning process, according to ITA officials. As part of this process, ITA officials said they identified activities to continue during a shutdown, as well as the ADA exceptions to justify the activities, before submitting plans to General Counsel for review. However, ITA did not document its roles and responsibilities because the component relied on the planning processes documented in Commerce’s shutdown contingency plan, according to ITA officials. The agency’s plan provided instructions for submitting component shutdown plans to Commerce’s Office of the General Counsel and Office of Human Resources Management. Commerce’s contingency plan did not, however, contain information about component-specific roles and responsibilities related to planning for a potential government shutdown. Without documenting roles and responsibilities, ITA cannot ensure that the appropriate officials take the necessary steps to effectively prepare and execute plans for any future potential government shutdowns. USTR: USTR instructed Assistant U.S. Trade Representatives to indicate which employees would perform excepted work based on the highest priority initiatives and activities. Two weeks prior to the shutdown, the Office of Administration and General Counsel used this information to develop a plan for the shutdown, followed by senior leadership approval, according to USTR officials. While USTR described the roles and responsibilities of its officials in planning for the shutdown, USTR did not document these roles and responsibilities because it used this same process in previous shutdowns, and responsible parties were accustomed to the process and knew their roles well, according to USTR officials. Documenting roles and responsibilities would help USTR ensure that the appropriate officials take the necessary steps to effectively prepare and execute plans for future potential government shutdowns, especially when officials currently familiar with the process no longer work for USTR. Internal Control Standards states that agency component management should implement its control activities through policies. Agency components can effectively do so, in part, by documenting processes for implementing policies related to government shutdowns. Shutdown preparation process documents at selected agency components included descriptions of activities to complete prior to the shutdown, such as updating and reviewing contingency plans, and preparing guidance and communication for managers and employees, among other steps. CBP: CBP’s non-public portion of the DHS contingency plan contained actions necessary to prepare for an impending shutdown, in addition to the roles and responsibilities discussed above. For example, CBP officials would need to identify executive points of contact who would continue working during the shutdown, prepare employee communications such as furlough notices, and prepare and distribute guidance for employee training during the shutdown, according to CBP’s shutdown guidance. This guidance also included descriptions of services, such as facilities maintenance, mail operations, and use of information technology equipment that would remain available and how, if at all, that work would be accomplished during a shutdown. IRS: IRS developed detailed process maps for its shutdown processes to document its planning and implementation activities and help improve understanding of the roles and responsibilities of staff at each step, according to IRS officials. IRS’s planning process map showed the order in which staff should perform certain tasks, a description of each task, and the responsible party for each task. For instance, the document showed who should draft, review, revise, and approve the shutdown contingency plan, and when each step should occur by each party. Figure 5 shows a streamlined version of IRS’s process map for the shutdown planning phase. Similarly, IRS’s implementation process map detailed steps for communicating with employees prior to a shutdown and updating contingency plans during a shutdown. During the shutdown, IRS distributed tools and guidance with instructions for implementing each step, according to an IRS official. ITA: ITA prepared a list of activities scheduled for the first 80 days of the fiscal year 2019 shutdown and determined the activities that would continue during the shutdown. While ITA officials described the process of assembling this list to us, they did not provide evidence to show that they had documented the process. According to ITA officials, ITA performed a similar exercise during the fiscal year 2014 government shutdown. Additionally, ITA officials said that they followed Commerce’s contingency plan to plan for the fiscal year 2019 shutdown. However, that document provided general information at the agency level. It did not provide information on the shutdown planning processes used by ITA, such as ITA-specific actions to take in the planning process. ITA did not provide documents showing these processes. Documentation of shutdown planning procedures would help ITA ensure that officials take the necessary steps to effectively prepare for future potential government shutdowns. USTR: USTR officials described the agency component’s shutdown processes but did not have the processes fully documented. Instead, USTR relied on the institutional knowledge of its officials to prepare for the fiscal year 2019 government shutdown. USTR officials told us that staff implementing shutdown processes for the fiscal year 2019 government shutdown also did so during the fiscal year 2014 shutdown. These officials told us that they used the same processes for both shutdowns, and that the staff involved were familiar enough with the processes to implement them effectively in fiscal year 2019. USTR communicated through email the steps for employees to take prior to furloughs, such as providing personal contact information to supervisors. USTR also provided EOP’s shutdown guidance to employees, which included additional information for employees, such as limitations to work site access and seeking outside employment while furloughed. However, EOP’s guidance did not contain details about USTR’s shutdown preparation process. USTR provided guidance to Assistant U.S. Trade Representatives about identifying excepted employees, but this guidance did not include information about other planning processes. Without documentation of all shutdown planning procedures, USTR cannot ensure that officials take the necessary steps to effectively prepare for future potential government shutdowns. Internal Control Standards states that management should communicate sufficient information, such as policies and procedures for implementing shutdown processes, to all appropriate individuals in a timely manner. We found that selected agency components used a variety of methods to communicate shutdown-related plans with employees in a timely manner prior to or at the beginning of the fiscal year 2019 shutdown. Methods included distributing policies through managers, referring employees to internal websites, and component-wide emails. Additionally, all selected agency components communicated individual furlough decisions to employees once the shutdown began. Representatives from employee organizations whose members worked at CBP and IRS said that, despite minor communication challenges between components and employees, they generally found shutdown-related communication to employees to be adequate. CBP: CBP encouraged supervisors to communicate to employees what could be expected of them should a shutdown occur, according to CBP officials from the Office of Field Operations. DHS directed CBP to email furlough notices to affected employees once the shutdown began, according to CBP officials, and CBP received email read receipts to help ensure the notices reached all employees. Each office confirmed with CBP that notices were sent to all affected employees, according to CBP officials. CBP held daily meetings with management during the shutdown to answer questions and share information, including information about travel, pay, contract actions, review and approval of employee recalls, and updates to the CBP contingency plan, according to CBP officials. Organizational points of contact then shared this information with managers, who provided appropriate information to employees. Furloughed employees did not have permission to access internal online resources as CBP had instructed them not to use CBP systems during the shutdown except in limited circumstances. In response, CBP developed a mobile application so that furloughed employees could see such updates on their personal cell phones in the event of a future shutdown, according to CBP officials. Representatives of CBP bargaining unit employees told us that, aside from limited instances of inaccurate or delayed information, CBP effectively communicated shutdown information to employees using multiple communication channels. IRS: IRS hosted internal training sessions prior to the shutdown to clarify roles and responsibilities for managers and excepted, exempt, and furloughed employees. IRS also made resources available to employees on its website, according to IRS officials, including shutdown checklists and a Frequently Asked Questions document with information on preparing for an orderly shutdown, among other things. Two days prior to the shutdown, OMB authorized IRS to direct managers to verbally inform employees of their furlough or excepted status in the event of a shutdown, according to IRS officials. These officials told us that IRS directed managers to not distribute status letters until December 22, 2018, the first day of the partial government shutdown. IRS’s implementation process map also shows that officials were to send status letters at the start of a shutdown. A representative of IRS bargaining unit employees told us IRS was responsive to employee questions during the shutdown and tried to address all issues raised. The representative noted that IRS had some challenges communicating with recalled employees as the shutdown continued but also said that IRS did the best it could, given its limitations, and did not identify ways to improve employee communication. ITA: Commerce directed ITA to distribute notices to employees explaining individuals’ furlough or excepted status after the shutdown began, according to ITA officials. On December 26, 2018, the first working day of the shutdown, ITA officials said they issued these notices along with a fact sheet about tasks for employees to complete that day. The fact sheet also communicated policies regarding scheduled leave and workspace access during the shutdown, among other things. ITA asked employees to confirm receipt of the notices during the orderly shutdown period, after which ITA certified to Commerce that it had issued all notices, according to ITA officials. A representative for bargaining unit Foreign Service Officers at ITA suggested that employees might benefit from receiving some information prior to a shutdown, including standard processes that ITA has established in policy and that remain the same between government shutdowns. USTR: Prior to furloughing employees, USTR instructed employees to visit its public website daily to verify USTR’s operating status. The website provided information on transit benefits, unemployment compensation, and an employee assistance program, among other things. USTR also communicated changes in operating status through notifications to employees’ personal telephone numbers and email accounts during the shutdown, according to USTR officials. These officials told us that in-person communication worked well to convey information to staff due to the small size of the agency component, approximately 250 staff. USTR officials said they emailed all employees about furloughs that would begin on January 14, 2019, updated the operating status on its phone line and website, and directed employees to stay apprised of USTR’s shutdown status. Before furloughs began, USTR instructed employees to provide managers with personal contact information, which, according to officials, managers used to recall employees during the shutdown. USTR officials said that managers also communicated with individual employees regarding whether they would continue to work after January 14, 2019. USTR employees were not represented by an employee organization. Internal Control Standards states that agency component management should design and implement control activities, such as shutdown processes, through policy. Agencies can effectively do so, in part, by documenting processes and roles and responsibilities for staff implementing those processes. During the fiscal year 2019 government shutdown, agencies recalled employees who were previously furloughed to return to work as the shutdown continued and circumstances changed. While each agency component had processes to recall employees back to work during the shutdown, not all components documented these processes. CBP: CBP’s non-public portion of the DHS contingency plan for fiscal year 2019 documented the employee recall process for government shutdowns. According to the plan, offices were to send a written request for a recall to the Executive Assistant Commissioner, Enterprise Services, specifying the number of employees to recall and the justification for doing so. DHS’s Budget Division and Office of the General Counsel also reviewed these recall requests, according to a DHS official. As with its initial excepted and furloughed employee notices, CBP used email read receipts to determine whether employees received updates to their furlough or excepted statuses and CBP recall processes. Additionally, each office had to verify with the CBP Hiatus Coordinator that updated status notices were sent to employees. IRS: IRS documented its procedures for recalling newly excepted employees during the shutdown in its implementation process map. IRS communicated these procedures to employees during the shutdown via its emergency web page and hotline, an updated Frequently Asked Questions document, and engagement with the employee organization representing IRS employees in the bargaining unit, according to IRS officials. IRS delegated the process of recalling employees to its 23 organizational offices. During the recall process, IRS managers contacted excepted employees to discuss duties and the date to report to work, according to IRS officials. IRS had many instances where the component recalled furloughed employees for a period of time and furloughed the employees again when needed, according to an IRS official. This official told us that IRS issued new furlough letters to employees each time this occurred. Similarly, IRS offices used an intermittent furlough letter when excepted employees planned to be away from work. According to the IRS official, doing so provided documentation of whether those excepted employees worked or were furloughed on a given day. ITA: According to ITA officials, once Commerce approved a temporary exception during the shutdown, ITA’s shutdown coordinator issued a recall letter to employees. ITA issued recall notices for temporary exceptions during the shutdown to perform specific work activities. Once employees completed those activities, ITA issued another furlough notice to those employees, according to ITA officials. ITA had a daily employee tracking document that showed exception start and end dates, and whether recall letters and subsequent furlough letters were issued to each employee. However, ITA did not document its recall process. Similar to its shutdown planning processes, ITA officials said they relied on Commerce’s employee recall processes instead of documenting its own specific processes. However, Commerce’s guidance did not contain information about how ITA developed temporary exception requests or how ITA processed the recalls. Without documentation of employee recall processes, ITA cannot ensure that officials are effectively implementing their processes during a potential future shutdown. Furthermore, officials who previously implemented shutdown-related processes may not be available during future shutdowns. Documentation ensures that processes that have been deemed to be effective can be replicated by others in the future. USTR: USTR recalled additional employees to perform excepted work during the shutdown. On each day after furloughs began, USTR recalled up to 30 employees beyond the 74 excepted employees in the Executive Office of the President’s (EOP) shutdown contingency plan. The Chief of Staff and Deputy Chief of Staff reviewed a list of excepted employees each day to identify adjustments to the number of excepted employees needed, according to USTR officials. These officials told us that when USTR offices requested employee recalls, the Chief of Staff and Deputy Chief of Staff consulted with the responsible Deputy U.S. Trade Representatives to make necessary changes. USTR officials did not have a documented process for recalling these employees. As with its preshutdown contingency planning processes, USTR officials said that they rely on institutional knowledge to carry out its recall procedures. Documentation of employee recall processes would help USTR ensure that officials effectively implement these processes during future shutdowns, especially given that officials who previously implemented shutdown-related processes may not be available during future shutdowns. All agency components we reviewed said that they had reviewed or planned to review their shutdown processes and incorporate any identified solutions into their internal planning documents or into agency contingency plans. CBP: CBP incorporated changes to its policies on employee leave and absences into its non-public portion of the DHS fiscal year 2020 contingency plan for a potential shutdown. For example, CBP’s fiscal year 2020 plan now contains examples of when supervisors may approve absences for excepted employees, such as for previously approved and ongoing requests under the Family and Medical Leave Act of 1993. IRS: Following the fiscal year 2019 shutdown, IRS reviewed its processes, requesting input from offices about ways to improve those processes in the case of future government shutdowns. Some improvements identified by offices included modifying current lapse plans to incorporate a medium- and long-term view, hosting training that focuses on frequently asked questions and managerial and employee responsibilities, and creating user-friendly access to information. ITA: ITA planned to cooperate with partner agencies on planned excepted activities going forward, according to ITA officials. These officials said they were in contact with their interagency partners at the time of our review and would work with them prior to a potential future shutdown to determine whether to submit requests for excepted work for certain activities. USTR: USTR reviewed its processes for the fiscal year 2019 shutdown and determined that it operated effectively and would not require changes for future shutdowns, according to USTR officials. Internal Control Standards states that agency component management should implement control activities through policies. During a government shutdown, agencies must limit the work performed to only exempt or excepted activities. Establishing limits for the number of employees working during a shutdown can help achieve this goal, and agencies can document these limits in their shutdown contingency plans. Tracking the number of employees working during a shutdown can help agencies ensure that they operate in accordance with their established contingency plans and prevent violations of the ADA. CBP: According to officials, CBP did not direct program offices to perform daily head counts of employees working and did not track the number of employees who worked during the shutdown. CBP officials told us that it would have been difficult to track employees because it did not have the systems or data to match the number of planned excepted employees with the number of employees who actually worked during the shutdown. Instead, CBP relied on managers to ensure that individual offices did not exceed their approved number of excepted positions during the shutdown. While individual offices could have opted to track the number of employees working each day for this purpose, CBP officials said that they did not direct all offices to do so. Tracking the number of employees who worked during the shutdown would help CBP ensure that controls to limit who can perform work during a shutdown function as intended. It would also ensure that its operations are consistent with contingency plans. IRS: IRS tracked the number of employees who worked each day during the shutdown but faced challenges in doing so. IRS directed managers to ensure that the number of excepted employees in each office did not exceed the number of approved positions in the contingency plan, according to IRS officials. An IRS official told us that each office had discretion for how it complied with this requirement, such as by requiring a daily headcount of employees. For example, during the shutdown, IRS’s Wage and Investment Division documented the office or function under which employees worked and the number of employees who worked in each. However, as headcounts proved to be time consuming for offices—Wage and Investment tracked up to 11,000 employees on one day—IRS officials told us they plan to move to an automatic tracking system in the future. According to IRS officials, IRS hosted daily calls with senior executives and Lapse Program Managers for each office to discuss the daily implementation of the shutdown contingency plan. IRS officials told us that the Heads of Office and Lapse Program Managers oversaw daily operations in each of their offices to help ensure operations were consistent with contingency plans. For example, Wage and Investment officials told us that the Wage and Investment Commissioner met daily with teams to discuss activities performed to help ensure that employees performed only the work in the contingency plan approved prior to the shutdown. ITA: ITA maintained a daily tracker of excepted employees who were scheduled to work each day. ITA used this tracker to record excepted employee names, projects and tasks, exception start and end dates, exception categories, and travel information as appropriate. Officials used this information to determine whether employees received the appropriate furlough or excepted status notice during the shutdown. USTR: USTR tracked which employees worked during the shutdown after January 14, 2019, in accordance with EOP guidance. EOP guidance says that “all EOP components are required to compile and report daily the name of each excepted employee and certify the hours worked that week for the duration of a lapse in appropriations to the group responsible for payroll.” USTR provided to EOP daily lists of excepted staff during the shutdown. These lists helped account for those who were guaranteed pay for work performed during the shutdown, according to USTR officials. Internal Control Standards states that agency component management should design and implement control activities through policies to help meet objectives. Effective implementation includes determining the policies necessary to operate a process based on objectives, such as limiting physical and virtual employee access to agency component workspaces and networks. CBP: In its notices provided to furloughed employees at the start of the fiscal year 2019 government shutdown, CBP advised employees that they must remain away from their workplace unless and until recalled. These furlough notices and CBP’s non-public portion of the DHS contingency plan also stated that employees could not use their government-issued devices for any purpose other than receiving updates and emergency notification from their supervisors. However, CBP did not have additional controls to limit employee access to physical or virtual workspaces, according to CBP officials, such as removing furloughed employees’ ability to logon to CBP networks or devices. DHS officials indicated that it would be difficult to monitor access for all excepted employees during a shutdown, especially given that most CBP employees continued to work during the fiscal year 2019 shutdown. IRS: IRS did not have sufficient controls to limit building access or virtual workspace access during the shutdown. While IRS developed lists of excepted and exempt employees who could work during the shutdown, IRS did not use these lists to grant or deny access to facilities, according to IRS officials. They told us IRS primarily used these lists to ensure it could provide sufficient services to each building based on the number of employees expected to work during the shutdown. IRS’s guidance to furloughed employees stated that employees should not use government-issued mobile phones or login to their government accounts remotely, and managers discussed this requirement with employees, according to IRS officials. IRS also directed employees not to use other government-furnished equipment such as computers, according to IRS officials. However, we found no additional controls to limit virtual network access during the shutdown. During the shutdown, IRS frequently substituted which excepted employees performed excepted functions, resulting in a rotating workforce, according to IRS officials. IRS officials believed it would be difficult to control physical or virtual access for all excepted employees in future shutdowns since access needs changed as frequently as each hour depending on which employees worked. ITA: ITA followed Commerce procedures to develop building access security lists to help ensure building access for excepted and exempt employees during the shutdown, according to ITA officials. Prior to the shutdown, Commerce directed agency components to prepare and submit building access security lists to the department’s Office of Security each day. If an employee tried to enter the headquarters building during the shutdown, the Office of Security would contact an ITA official to verify whether that employee could enter, according to ITA officials. These officials told us that employees not on the building access security list were not granted access to the headquarters building during the shutdown. While ITA had controls to limit physical workspace access during the shutdown, it did not have sufficient controls in place to limit virtual access. According to ITA officials, all furloughed employees were instructed not to use their government devices or access the ITA network virtually, and furlough notices stated that furloughed employees could not work at an alternative worksite during the shutdown. However, ITA officials believed that implementing additional controls, such as turning off network access for furloughed employees, would complicate its process for granting temporary exceptions for employees during a shutdown. USTR: USTR provided employees with EOP guidance prior to implementing furloughs on January 14, 2019. This guidance instructs furloughed employees not to access their place of work or use government-issued cell phones or computers. USTR officials told us they did not have controls in place to monitor employee building access or prevent furloughed employees from entering physical USTR workspaces. These officials told us they provided adequate communications, instructions, and guidance to employees about who can access physical and virtual workspaces. USTR officials also told us that they did not have controls in place to monitor or limit employee access to virtual USTR workspaces. According to these officials, USTR does not maintain or monitor the EOP-provided mobile communications devices and information technology network. Instead, provision and control of telecommunications and information technology, such as the ones identified, is the responsibility of the Presidential Information Technology Community. While agency components may face challenges implementing workspace access controls, such as limiting network access for a large number of employees, these steps are nevertheless important to take. Having sufficient controls to limit who can perform work during a shutdown would help agency components ensure that they operate consistently with the ADA and with contingency plans that are designed to help them operate effectively and avoid misuse of government resources during a shutdown. Agency component management can tailor controls to meet the component’s unique needs. Specific controls used by an agency component may be different than those used by other components based on a number of factors, such as differences in mission, size, or operational environment of the component. Government shutdowns are disruptive events that have spanned multiple weeks in recent years. Given the length of some shutdowns, it is important for agencies to have robust plans and established internal controls to effectively communicate, plan for potential changes, and manage operations prior to and during a shutdown. In addition, documentation of these plans and controls helps ensure that agencies can replicate their actions in the event of future shutdowns. According to OMB, agencies should have detailed contingency plans in place prior to a potential lapse in funding to ensure an orderly shutdown of operations. While three of four agencies’ contingency plans that we reviewed addressed most elements laid out in OMB’s guidance, we identified three elements for which all selected agencies had missing or incomplete information in their contingency plans. When asked about these deficiencies, agency officials often cited internal documents or discussions as addressing these information elements. Internal documentation and guidance can be useful in planning for a potential shutdown, but they do not provide the level of transparency of contingency plans, which are generally available to the public and furloughed employees. Contingency plans that address all information elements specified in OMB guidance ensure that agencies are prepared for potential shutdown scenarios, and provide transparency to agency actions during a lapse in appropriations. In addition to contingency plans, the agency components we reviewed all had internal processes related to planning for and managing operations during a shutdown. However, not all agency components documented these processes. Without documentation of shutdown operations, agencies may not be able to cease operations in a timely manner, and agencies’ actions may not be transparent to OMB, Congress, and the public during future shutdowns. Additionally, proper documentation of processes can help preserve institutional knowledge that might otherwise be lost. During a lapse in funding, agencies must ensure that they do not violate the ADA, which prohibits agencies from obligating or expending funds in the absence of appropriations unless otherwise authorized by law, and from accepting voluntary services for the United States except in cases of emergency involving the safety of human life or the protection of property. Contingency plans are one control that agencies use in this effort. Agencies must have assurance that the contingency plan is being followed daily during a shutdown. This assurance can be verified through controls that (1) track and document the number of employees who actually worked daily during the shutdown, and (2) limit physical and virtual workspace access to appropriate employees. Three of four agency components we reviewed tracked employees who worked, one had sufficient controls on physical access to workspaces, and none had sufficient controls to limit virtual access. Without these controls, agencies are at an increased risk that contingency plans will not be followed, thus diminishing their value as a mechanism to ensure ADA compliance. We are making a total of 14 recommendations, including four to USTR, three each to CBP and IRS, two to ITA, and one each to the Departments of Commerce and Homeland Security. The Secretary of Commerce should align the agency’s contingency plan with OMB guidance by including (1) plans for a potential prolonged shutdown; (2) flexibilities available to supervisors if furloughed employees were unable to return to work after the end of the shutdown; and (3) procedures for resuming program activities, including steps to ensure appropriate oversight and disbursement of funds upon the end of a shutdown. (Recommendation 1) The Secretary of Homeland Security should align the agency’s contingency plan with OMB guidance by including (1) plans for a potential prolonged shutdown; (2) flexibilities available to supervisors if furloughed employees were unable to return to work after the end of the shutdown; and (3) procedures for resuming program activities, including steps to ensure appropriate oversight and disbursement of funds upon the end of a shutdown. (Recommendation 2) The Commissioner of Internal Revenue should align the agency’s contingency plan with OMB guidance by including (1) plans for a potential prolonged shutdown; (2) flexibilities available to supervisors if furloughed employees were unable to return to work after the end of the shutdown; and (3) procedures for resuming program activities, including steps to ensure appropriate oversight and disbursement of funds upon the end of a shutdown. (Recommendation 3) The U.S. Trade Representative, in consultation with EOP as appropriate, should align the component’s contingency plan with OMB guidance. This could be accomplished through (1) revisions to the EOP contingency plan; or (2) by creating a separate USTR plan. (Recommendation 4) The Under Secretary for International Trade should document the component’s shutdown processes, including roles and responsibilities, planning processes for potential shutdowns, and recall processes for furloughed employees during a shutdown. (Recommendation 5) The U.S. Trade Representative should document the component’s shutdown processes, including roles and responsibilities, planning processes for potential shutdowns, and recall processes for furloughed employees during a shutdown. (Recommendation 6) The Commissioner of CBP should develop internal controls to track and document which employees worked and what work was performed daily during a government shutdown. (Recommendation 7) The Commissioner of CBP should develop internal controls to limit access to physical workspaces to appropriate employees during a government shutdown. (Recommendation 8) The Commissioner of Internal Revenue should develop internal controls to limit access to physical workspaces to appropriate employees during a government shutdown. (Recommendation 9) The U.S. Trade Representative should develop internal controls to limit access to physical workspaces to appropriate employees during a government shutdown. (Recommendation 10) The Commissioner of CBP should develop internal controls to limit access to virtual workspaces to appropriate employees during a government shutdown. (Recommendation 11) The Commissioner of Internal Revenue should develop internal controls to limit access to virtual workspaces to appropriate employees during a government shutdown. (Recommendation 12) The Under Secretary for International Trade should develop internal controls to limit access to virtual workspaces to appropriate employees during a government shutdown. (Recommendation 13) The U.S. Trade Representative should, in consultation with EOP, develop internal controls to limit access to virtual workspaces to appropriate employees during a government shutdown. (Recommendation 14) We provided a draft of this report to Commerce, DHS, EOP, IRS, OMB, and USTR for review and comment. We received written comments from Commerce, DHS, and IRS, summarized below and reproduced in appendixes II, III, and IV. USTR provided comments via email, also summarized below. OMB did not provide comments, citing its focused efforts on addressing the national emergency response to the coronavirus pandemic. DHS, EOP, IRS, and USTR provided technical comments, which we incorporated as appropriate. Commerce agreed with all three recommendations directed to it and ITA, and stated that ITA has taken steps to address two of the recommendations. Commerce stated that ITA has documented its shutdown planning processes and recall processes for furloughed employees during a shutdown (recommendation 5). According to Commerce, ITA has also established and documented internal controls to limit virtual workspace access to excepted or exempt employees during a government shutdown (recommendation 13). In addition, Commerce stated that it will develop an action plan to address the recommendation to better align its contingency plan with OMB guidance (recommendation 1). DHS agreed with all four recommendations directed to it and CBP, and stated that it has begun to take steps to better address OMB guidance on contingency plans (recommendation 2). In addition, DHS stated that CBP plans to analyze existing systems to determine which is best suited to track and document employee work during a government shutdown and will ensure that the chosen system is available should a future shutdown occur (recommendation 7). For the recommendation on developing controls for physical workspaces (recommendation 8), DHS stated that because CBP does not have systems capable of efficiently restoring physical access for furloughed employees, it would have to reinstate employee access individually and the cost would be substantial. DHS stated that CBP plans to update procedures to ensure more comprehensive workspace access guidance for furloughed employees. With regard to the recommendation on developing controls for virtual workspace access (recommendation 11), DHS stated that CBP believes that furloughed employees must be able to passively monitor the status of the government shutdown and access important agency communications using DHS-issued electronic devices. Additionally, disabling and reactivating thousands of employee user accounts during a shutdown posed a significant burden. DHS said that CBP plans to update shutdown procedures to clarify allowed use of DHS-issued electronic devices by furloughed employees. We agree that CBP should update procedures on workspace access as suggested, and continue to believe that physical and virtual access controls are important during shutdowns in order to prevent misuse of government resources. We encourage CBP to improve their systems to be able to efficiently implement such controls. IRS partially agreed with one recommendation addressed to it and disagreed with two others. IRS agreed with one element of our recommendation to include additional detail in its agency contingency plan (recommendation 3) and stated that it is in the process of adding procedures for resuming program activities following a government shutdown into its contingency plan. IRS did not agree with the other elements of the recommendation because it believes it has already addressed plans for a potential prolonged shutdown and flexibilities for supervisors if employees are unable to return to work at the end of a shutdown in its contingency plans. We agree that while IRS has included some details on these elements in its plans, we continue to believe that it should provide more detail, such as points in time when the furlough status of an employee may change, how many employees would be affected, and the legal basis for the changes, within its publically available contingency plan to fully address these elements. IRS disagreed with our recommendations on developing controls for physical and virtual workspace access during a shutdown (recommendations 9 and 12). For both recommendations, IRS stated that it believes that it has effective controls in place to manage physical and virtual workspace access during a shutdown. In addition, IRS said that it believes that implementing additional access controls do not justify the corresponding resource investments. We continue to believe that IRS should improve its access controls, which currently rely on managers and furlough letters to communicate limits on workspace access. While we recognize the costs of increased access controls, government shutdowns are unique events that require additional access controls in order to prevent potential misuse of government resources. In USTR’s emailed comments, its Assistant U.S. Trade Representative for Administration neither agreed nor disagreed with the four recommendations addressed to it. The official, however, stated that USTR has already begun addressing our recommendations on aligning its contingency plan with OMB guidance (recommendation 4) and documenting its shutdown processes (recommendation 6), and has made EOP aware of the recommendations on developing controls for physical and virtual workspace access during a shutdown (recommendations 10 and 14). As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Commerce, the Acting Commissioner of U.S. Customs and Border Protection, the Acting Secretary of Homeland Security, the Commissioner of the Internal Revenue Service, the Acting Under Secretary for International Trade, the Director of the Office of Management and Budget, the U.S. Trade Representative, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9110 or mctiguej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. James R. McTigue, Jr. Director, Strategic Issues. This report assesses the extent to which (1) selected agencies’ contingency plans were consistent with applicable Office of Management and Budget (OMB) guidance, (2) selected agency components planned for a potential prolonged shutdown and changed operations during the shutdown, and (3) selected agency components’ shutdown policies and procedures were consistent with relevant internal control principles. We selected four agency components under the jurisdiction of the Senate Committee on Finance that were affected by the fiscal year 2019 shutdown. When more than one agency component at an agency met these criteria, we selected the component that had the largest budget and the greatest planned number of employees performing excepted work during the shutdown. While the four components we selected are not generalizable to other agency components, they do reflect variation in size, funding type, and justification for excepted work that serve as illustrative examples of a range of experiences. These selected agency components are U.S. Customs and Border Protection (CBP), Department of Homeland Security (DHS); Internal Revenue Service (IRS), Department of the Treasury (Treasury); International Trade Administration (ITA), Department of Commerce (Commerce); and Office of the United States Trade Representative (USTR), Executive Office of the President (EOP). To address our first objective, we compared information in selected agencies’ government shutdown contingency plans to key information elements described in OMB guidance. Specifically, we identified 14 key information elements in the 2018 OMB Circular No. A-11 Section 124— Agency Operations in the Absence of Appropriations (Circular A-11), the applicable guidance, at the beginning of the partial government shutdown that began on December 22, 2018. This document details the information agencies should include in their contingency plans, such as significant agency activities that will continue or cease during a shutdown, the number of employees who will continue to work during a shutdown, and necessary actions for resuming orderly operations after a shutdown. Three of our four selected agency components—CBP, ITA, and USTR— operated under an agency-wide plan. Therefore, we evaluated the fiscal year 2019 contingency plans for Commerce, DHS, and EOP. Each component of DHS has a non-public, for official use only, portion of the agency-wide plan, and we included CBP’s non-public portion in our evaluation. Because Treasury’s contingency plan did not cover IRS, we evaluated IRS’s contingency plans for this objective. We also reviewed written responses from OMB and interviewed officials at selected agencies to understand the reasons for any discrepancies between the contingency plans and OMB guidance. To address our second objective, we assessed the extent to which selected agency components planned for a potential prolonged shutdown—one longer than 5 days—as outlined by Circular A-11, and changed operations during the shutdown. We reviewed shutdown contingency plans and other planning documents at CBP, IRS, ITA, and USTR to determine agency component processes for proposing, reviewing, and approving operational changes during a government shutdown. We interviewed officials at these agency components to determine what operational changes components made during the fiscal year 2019 shutdown and the key factors that led to these changes. To address our third objective, we assessed selected agency components’ shutdown processes to determine the extent to which the components followed relevant internal control principles in planning for the fiscal year 2019 government shutdown. We reviewed our Standards for Internal Control in the Federal Government (Internal Control Standards) and identified key principles related to agency components’ shutdown processes. Relevant internal control standards include designing and implementing appropriate policies and procedures and effectively communicating this information to stakeholders. This would include policies and procedures to ensure an orderly shutdown process and compliance with applicable laws such as the Antideficiency Act (ADA). We developed a questionnaire for selected agency components based on these internal control principles that reflected practices we determined to be associated with effectively implementing the controls in the context of a government shutdown, such as documentation of shutdown processes or employee communication. We reviewed the results of this questionnaire, reviewed agency component contingency plans and other internal planning documents, and interviewed component officials to determine the extent to which components followed these internal control principles. We assessed the sufficiency of selected agency components’ internal controls based on whether the evidence gathered contained relevant details about a component’s shutdown processes that demonstrated the component would have reasonable assurance of achieving its shutdown objectives. While we assessed agency components’ shutdown processes, we did not assess the results of those processes, such as whether components correctly or appropriately categorized activities as excepted from the ADA. We interviewed officials at selected agency components to understand the reasons for any inconsistencies between component planning and decision-making processes and internal control principles. We also interviewed representatives of employee organizations at the agency components we reviewed to determine if communication of shutdown- related policies and procedures was timely, sufficient, and transparent. In addition, we selected one program office within each reviewed agency component to identify illustrative examples of how components operationalized their shutdown processes. For CBP, IRS, and ITA, we selected the program offices with the largest budget based on available budget data. Based on this criterion, we selected CBP’s Office of Field Operations, IRS’s Wage and Investment division, and ITA’s Global Markets office. Selection of these program offices provided for a variety of justifications for excepted work and number of planned excepted employees. Due to the size of USTR, the agency component does not manage based on program offices, according to USTR officials. Because of this, we did not select a program office within USTR. We reviewed documents and interviewed officials in these program offices to determine how they planned for the fiscal year 2019 government shutdown, communicated with employees, and recalled furloughed employees back to work, among other things. We conducted this performance audit from March 2019 to June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. James R. McTigue, Jr. at (202) 512-9110 or mctiguej@gao.gov. In addition to the individual named above, Danielle Novak, Assistant Director; Shelby Kain, Analyst-in-Charge; Alyssia Borsella; Kendall Chan; Jacqueline Chapin; Ann Czapiewski; Kristine Hassinger; J. Andrew Howard; Ulyana Panchishin; Steven Putansu; and Melissa Wolf made major contributions to this report. Ted Hu and Triana McNeil also contributed to the report.", "summary": "A lapse in appropriations resulted in the federal government partially shutting down from December 22, 2018, to January 25, 2019. GAO was asked to evaluate agency contingency plans and operations during the FY 2019 shutdown. This report assesses the extent to which selected agencies and selected components (1) had contingency plans that were consistent with applicable OMB guidance, (2) planned for a potential prolonged shutdown and changed operations during the shutdown, and (3) had shutdown policies and procedures consistent with relevant internal control principles. GAO selected CBP, IRS, ITA, and USTR as agency components for review because they are under the jurisdiction of the Senate Committee on Finance and were affected by the FY 2019 shutdown. GAO reviewed OMB's guidance, agencies' contingency plans, and other documentation. GAO interviewed agency and component officials. The Office of Management and Budget (OMB) issues shutdown guidance for agencies in Circular A-11. Of four selected agency components, three—U.S. Customs and Border Protection (CBP), the Internal Revenue Service (IRS), and the International Trade Administration (ITA)—operated in fiscal year (FY) 2019 under contingency plans that included most of the key information elements specified in Circular A-11 . The plan that the fourth one—Office of the U.S. Trade Representative (USTR)—operated under, authored by the Executive Office of the President, did not include a majority of the key information elements. OMB guidance instructs agencies to have plans in place for both short and prolonged—longer than 5 days—shutdowns. None of the four selected agencies' FY 2019 contingency plans fully addressed anticipated changes in the event of a prolonged shutdown. GAO found that IRS, ITA, and USTR internally discussed and planned for anticipated operational changes in the event of a prolonged FY 2019 shutdown. CBP officials said they only focused on short-term operational needs. Having a comprehensive plan for a potential prolonged shutdown would help provide clearer workforce expectations during any future shutdowns. Having sufficient internal controls, such as documented policies and procedures, in place prior to a shutdown can help agencies implement changes in day-to-day operations during a shutdown. Selected agency components all incorporated some internal controls in their shutdown-related activities, as shown in the table below. However, none of the agency components had controls for limiting both physical and virtual workspace access for employees during a shutdown, each citing the difficulty of implementing such controls. Having these controls in place would help components ensure that they operate consistently with their contingency plans and avoid misuse of government resources. GAO is making 14 recommendations, including that certain agency components improve contingency plans, document shutdown procedures, and improve controls for physical and virtual workspace access during a shutdown. CBP and ITA agreed with the recommendations directed to them; IRS partially agreed with one and disagreed with two; and USTR did not state whether it agreed or disagreed, but has begun taking steps to implement two recommendations.", "document_type": "gao"}
{"report": "Several entities are involved with, and pay different prices for, prescription drugs as they move from the manufacturer to the beneficiary (a system referred to as the prescription drug supply chain). In general, manufacturers develop and sell their drugs to wholesalers, and wholesalers then sell the drugs to pharmacies. In the Part D program, CMS pays Part D plan sponsors to provide drug coverage, and plan sponsors may charge beneficiaries monthly premiums in exchange for coverage. Plan sponsors and PBMs negotiate reimbursement rates for the drugs provided to beneficiaries. When the beneficiary purchases a drug, the pharmacy is paid by the Part D plan sponsor, or through the PBM on the sponsor’s behalf, and by the beneficiary through any applicable cost-sharing. (See fig. 1 for a flow chart showing the relationship between certain entities in the prescription drug supply chain when a Part D plan sponsor uses a PBM.) Services associated with developing and managing a prescription drug plan performed by PBMs, Part D plan sponsors, or both, include: Formulary development. Determining the list of drugs covered under the plan (the formulary), including assignment of covered drugs to tiers that correspond to different levels of beneficiary cost sharing and placing restrictions on drugs included in the formulary. Part D plan sponsors submit formularies for their plans to CMS for review and approval annually. Pharmacy network development. Creating a network of pharmacies where beneficiaries may fill their prescriptions and negotiating drug prices and reimbursement rates with those pharmacies. This can also include developing “preferred networks,” whereby beneficiaries pay lower cost-sharing and pharmacies agree to receive lower prices for drugs in exchange for increased volume of prescriptions purchased. Utilization management services. Utilization management services include processes such as: Prior authorization. A requirement that beneficiaries obtain approval for a drug by the PBM or plan sponsor before obtaining the drug if it is to be covered by the plan. Step therapy. A requirement where more expensive drugs are covered only if beneficiaries try less expensive alternatives first and find them not to be effective. Medication therapy management. A program required by CMS designed to improve medication adherence and reduce the risk of adverse drug events through discussion with targeted beneficiaries and prescriber intervention. Drug utilization review. A concurrent examination by the PBM or plan sponsor of prescriptions at the time of purchase by the beneficiary to assess safety considerations, such as potential adverse interactions, and compliance with clinical guidelines (including quantity and dose). These reviews can also occur retrospectively to analyze beneficiaries’ drug utilization and physicians’ prescribing patterns. Negotiation of rebates from manufacturers. Negotiating rebates for Part D plan sponsors with manufacturers in exchange for driving more utilization of a manufacturer’s drug. This can include more favorable placement on the sponsor’s formulary. The rebate terms do not have to be disclosed to the public, but plan sponsors must report rebate amounts to CMS. PBMs may earn revenue from providing drug benefit management services to Part D plan sponsors in a number of ways, including: (1) payments from plan sponsors for administering services, such as drug benefits claim processing; (2) retention of a portion of drug rebates that PBMs negotiate on behalf of the plan sponsor and fees for managing and distributing those rebates; (3) spread pricing; and (4) payments from manufacturers for various services. PBMs may provide drug benefit management services to Part D plan sponsors and commercial plans, such as employer-sponsored health plans. Commercial plans may pay PBMs in ways similar to Part D plans (e.g., rebate retention and claims processing fees). Part D plan sponsors are also required to provide access to all or substantially all drugs covered under certain therapeutic classes of drugs, known as Medicare protected classes: (1) anticonvulsants, (2) antidepressants, (3) antineoplastics, (4) antipsychotics, (5) antiretrovirals, and (6) immunosuppressants for the treatment of transplant. Plans are limited in the formulary restrictions they can apply to these drugs. Additionally, CMS generally requires Part D plan sponsors to provide coverage for at least two drugs in each class. CMS makes payments prospectively to Part D plan sponsors for beneficiary drug coverage. CMS pays plan sponsors monthly, and these payments are determined through annual bids submitted in June of the preceding program year, which runs from January 1 through December 31. Those bids reflect the plan sponsors’ estimates of program costs and rebates and other price concessions that the sponsor expects to receive during the ensuing program year. At the end of the program year, CMS reviews cost data submitted by plan sponsors through PDE records and their submission of rebate and other price concession data and compares estimated payments with actual costs incurred, with CMS either reclaiming some funds or making additional payments. Thus, the final plan payments by CMS are based on the costs actually incurred by Part D plan sponsors minus rebates and other price concessions that are either passed along to the plan sponsors or retained by the PBMs. Rebates and other price concessions reduce the cost of the Part D program to beneficiaries and the federal government. In developing their bids, Part D plan sponsors may subtract rebates and other price concessions that are passed along to them from their estimated drug costs. When they do, rebates and other price concessions reduce a plan sponsor’s estimate of liability that is reflected in bid amounts, which, in turn, reduce beneficiary premiums because they are based, in part, on the bid amount. This downward pressure on premiums is one reason that premiums remained relatively unchanged between 2010 and 2015, according to CMS, even though total gross Part D drug costs grew about 12 percent per year in that period. Rebates have additional implications for Part D beneficiaries and the Part D program more generally. Since beneficiary cost sharing is calculated based on the price of the drug at the time of purchase (i.e., before rebates are paid), beneficiaries pay higher cost sharing than they would if rebates were paid at the point of sale. In addition, higher pre-rebate drug prices may result in beneficiaries more quickly reaching the catastrophic coverage phase, where the federal government’s share of drug costs increases, and the plan sponsors’ share decreases. Seventy-four percent of the drug benefit management services provided under 624 Part D plan sponsor contracts were performed by a PBM alone or in conjunction with a Part D plan sponsor in 2016. We found that plan sponsors performed the remaining 26 percent of services themselves. In addition, a PBM was used to provide one or more of the 10 key drug benefit management services under nearly all of the 624 Part D plan sponsor contracts (99.7 percent), and the manner in which they used them varied, as summarized below: Number of drug benefit management services provided. Part D plan sponsor contracts varied by the number of services provided by PBMs. Eighty-nine percent of Part D plan sponsor contracts used a PBM alone or in conjunction with a plan sponsor for at least half of the 10 drug benefit management services; 15 percent of contracts used a PBM alone or with a plan sponsor for all 10 services. Number of PBMs used. Part D plan sponsor contracts varied in the number of PBMs used to provide one or more of the 10 drug benefit management services. Fifty-four percent of contracts used one PBM, 35 percent used two or three PBMs, and 11 percent used four or more PBMs. Types of drug benefit management services provided. Part D plan sponsor contracts varied by the drug benefit management services they used a PBM to provide. PBMs alone or with the plan sponsor more frequently provided claims adjudication (99 percent of Part D plan sponsor contracts), pharmacy network development (92 percent), and rebate and other price concession negotiations (83 percent). In contrast, PBMs alone or with the plan sponsor less frequently provided a pharmacy and therapeutics committee (45 percent), enrollee appeals and grievance process-management (30 percent), and enrollment processing (34 percent). Part D plan sponsors mainly used five PBMs in 2016. Of the 103 PBMs that provided at least one drug benefit management service to the 624 Part D plan sponsor contracts in 2016, the following five provided at least one service to 528 (85 percent) plan sponsor contracts in 2016: CVS Caremark, OptumRx, Express Scripts, Medimpact, and Argus. These five PBMs also provided the largest number of services to Part D plan contracts in 2016. For example, CVS Caremark, by itself or with another PBM or plan sponsor, provided 17 percent of services that PBMs provided to Part D plan sponsors’ contracts in 2016, the most of any PBM. See appendix II for more information on variation in Part D plan sponsor contracts’ use of PBMs, factors that influence sponsors’ decision to use a PBM, and additional information on the PBMs used by Part D plan sponsors. Our review of 20 service agreements between Part D plan sponsors and PBMs found that the primary revenue source for PBMs from services they provided to Part D plans was (1) a volume-based fee paid by plan sponsors based on the number of paid claims that the PBM processed; (2) a flat monthly per-member, per-month fee paid by plan sponsors; or (3) a combination of the two. Nineteen of the 20 service agreements that we reviewed stated that PBMs were to be paid in one of these ways. None of the service agreements tied these fees to the price of a drug paid to the pharmacy. Representatives we interviewed from all seven of the PBMs confirmed that a Part D plan sponsor-paid fee for the PBM’s services was the primary way they earned revenue from their Part D clients. We also examined PBM revenue reported to CMS by Part D plan sponsors in their rebates and other price concession data—also referred to as direct and indirect remuneration (DIR)—in 2016, the most recent data available at the time of our analysis. These data show that PBMs passed nearly all rebates received from manufacturers through to Part D plan sponsors in 2016. Part D plan sponsors reported to CMS that, of the approximately $18 billion in rebates that PBMs negotiated with pharmaceutical manufacturers that year, PBMs retained $74.3 million, or about 0.4 percent, and passed through the remaining 99.6 percent to plan sponsors. The small amount of PBM rebate retention in the Part D program was also reflected in the service agreements we examined and in our interviews with PBM representatives. Sixteen of the 20 service agreements that we reviewed included provisions that required the PBM to pass through all rebates to the Part D plan sponsor; one other agreement required at least 95 percent to be passed through to the plan sponsor. The other three service agreements that we reviewed either did not include provisions related to rebate retention or redacted such information. Officials we interviewed from four of the seven PBMs told us their PBMs passed through to Part D plan sponsors all rebates obtained from manufacturers. Representatives of one PBM noted that plan sponsors, in turn, may use rebates to help offset the growth in drug costs, helping lower premiums for beneficiaries. Representatives from the other three PBMs noted that the amount of retained rebates was relatively small, consistent with the data reported to CMS. PBMs and Part D plan sponsors may earn non-rebate revenue from manufacturers for providing certain services. The service agreements we examined included examples of this revenue, including fees for rebate program administration, prescriber education programs, and programs designed to ensure patients adhere to, and comply with, recommendations regarding a particular prescription. The full amount that PBMs and Part D plan sponsors earned from manufacturers for non- rebate services in 2016 was $516.5 million. Although CMS requires these fees to be reported to the agency by plan sponsors, CMS does not break out how much of the money was received by PBMs and how much was received by plan sponsors. PBMs earned little Part D revenue from spread pricing—keeping the difference between the amount the PBM paid the pharmacy for a drug and the amount the PBM charged the plan for the drug, from 2014 through 2016. PBMs earned about $300,000 from spread pricing in 2016, according to CMS rebate and other price concession data. CMS data also show that PBMs earned no revenue from spread pricing in either 2014 or 2015. PBMs generally earn more from spread pricing and rebate retention from commercial plans than they do from Part D, according to officials from three PBMs. Officials from two of these PBMs said CMS reporting requirements have removed much of the incentive in Part D for PBMs to earn revenue from spread pricing because of the complexity of the requirements and the criticism from health care providers when reports to CMS containing these amounts are publicized. See appendix III for more information on Part D plan sponsor reporting to CMS of the amounts of revenue—other than rebates and discounts—that manufacturers provide to their PBMs; and on PBM and Part D plan sponsor perspectives on PBM revenue earned from spread pricing, the effect of CMS requirements on spread pricing revenue, and differences between PBMs’ Part D and commercial business lines. Growth in the amount of rebates and other price concessions provided by manufacturers and others to Part D plan sponsors and PBMs outpaced growth in gross and net Part D expenditures for all brand-name and generic drugs from 2014 through 2016. Gross expenditures reflect what was paid to the pharmacy by the Part D plan sponsor—or the PBM on the sponsor’s behalf—and by the beneficiary for a given drug. Net expenditures reflect any rebates and discounts obtained by plan sponsors and PBMs after a beneficiary receives a drug. During this time, gross Part D expenditures increased 20 percent, from $120.7 billion in 2014 to $145.1 billion in 2016. The amount of rebates and other price concessions obtained for these drugs increased 66 percent during the same period, from $17.5 billion to $29 billion. As a result, rebates and other price concessions as a proportion of gross expenditures increased from 14 percent of gross expenditures in 2014 to 20 percent in 2016. This resulted in an increase in net Part D expenditures of 13 percent, from $103.2 billion in 2014 to $116.1 billion in 2016 (see fig. 2). Rebates accounted for most of the total of rebates and other price concessions obtained for Part D drugs from 2014 through 2016. Rebates are generally paid by manufacturers to Part D plan sponsors, or PBMs on sponsors’ behalf, after a drug is purchased from a pharmacy. In 2016, rebates accounted for 92 percent ($27 billion) of the $29.1 billion in rebates and other price concessions. The proportion was generally consistent in 2014 and 2015, with rebates accounting for 93 and 91 percent of total rebates and other price concessions, respectively. Pharmacy-related price concessions, which include any monies obtained by plan sponsors and PBMs from a pharmacy after a beneficiary purchases a drug, accounted for nearly all the rest of rebates and other price concessions—7 percent—in 2016. The amount of pharmacy- related price concessions increased 295 percent from 2014 through 2016 ($538 million to $2.1 billion). The 444 highest expenditure, highest utilization brand-name drugs accounted for the majority of expenditures and received the vast majority of rebates and other price concessions in 2016. These drugs accounted for 65 percent of the $145 billion in Part D expenditures and received 90 percent of the $29.1 billion in rebates and other price concessions obtained for Part D drugs. Of the 444 highest expenditure, highest utilization brand-name drugs in 2016, the 200 highest utilization and the 200 highest expenditure drugs received a greater amount of rebates and other price concessions than the 200 highest expenditure per utilization drugs. (See table 1.) Furthermore we found that brand-name drugs received greater amounts of rebates and other price concessions than generic drugs. Specifically, among the 444 highest expenditure, highest utilization brand-name drugs and the 476 highest expenditure, highest utilization generic drugs, brand-name drugs received 98 percent of rebates and other price concessions in 2016. Consistent with the results for all Part D drugs, from 2014 through 2016 rebates and other price concessions outpaced growth in gross and net expenditures for the three groups of highest expenditure, highest utilization brand-name drugs in our analysis (see table 2 for information on these brand-name drugs). The three groups of brand-name drugs generally had higher percent changes in rebates and other prices concessions and in gross and net expenditures than did all Part D drugs, which includes generics. For example, from 2014 through 2016, net expenditures for the 200 highest expenditure brand-name drugs increased 27 percent compared to a 13 percent increase for all Part D drugs. Of the three groups, the 200 drugs with the highest expenditure per utilization had the largest percentage increases in expenditures and rebates and other price concessions. However, these drugs had relatively low gross expenditures, rebates and other price concessions, and utilization compared with the other two groups. Increases in expenditures for the three groups of drugs in our analysis were primarily accounted for by increases in the price per drug rather than changes in utilization, as indicated by the growth in expenditures exceeded growth in their utilization. Net expenditures per beneficiary were similar if a Part D plan sponsor used a PBM for rebate negotiations or if it conducted its own negotiations. Specifically, in 2016, median net expenditures per enrollee were similar for plan sponsors using a PBM and those that did not at $2,557 and $2,570, respectively. Rebates and other price concessions accounted for a median of 12 percent of gross Part D expenditures for plan sponsors using a PBM for their negotiations and a median of 10 percent for plan sponsors that did not. The majority—82 percent—of plan sponsors used a PBM to obtain rebates and other price concessions on their behalf. The plan sponsors that performed their own negotiations generally had higher enrollment than those that used a PBM—a median of approximately 47,000 beneficiaries, compared to approximately 13,000 beneficiaries (see table 3). See appendix IV for additional information on expenditures and rebates and other price concessions obtained for the 444 highest expenditure, highest utilization brand-name Part D drugs in 2016. The appendix also contains information on expenditures and rebates and other price concessions obtained by the Part D plan sponsors whose representatives we interviewed. In 2016, the highest expenditure, highest utilization brand-name drugs sold in retail pharmacies received discounts off of manufacturer list prices that were significantly higher than those sold in specialty pharmacies. Of the 444 highest expenditure, highest utilization brand-drugs in our analysis, 244 were sold in retail pharmacies. For this group, gross Part D prices—those paid to the pharmacy by the Part D plan sponsor, PBMs on the sponsor’s behalf, and the beneficiary—were 17 percent lower than manufacturer list prices for these drugs. When rebates and other price concessions were accounted for, net Part D prices were 41 percent lower than manufacturer list prices. In contrast, the 200 drugs sold in specialty pharmacies received fewer discounts off of manufacturer list prices. For these drugs, median gross and net prices were 15 percent and 16 percent, respectively, lower than manufacturer list prices (see fig. 3). As a result, drugs sold in retail pharmacies received median discounts (41 percent) that were 2.5 times larger than those sold in specialty pharmacies (16 percent). See appendix V for more information on prices for the highest expenditure, highest utilization brand-name drugs and for information on prices for selected generic drugs. Our review of 52 peer-reviewed studies indicates that utilization management services were associated with financial savings or improved beneficiary health indicators. However, the effects on ensuring that beneficiaries take their medication as prescribed (adherence) and access to clinically appropriate prescriptions were less clear. The studies examined the effects of 10 different types of utilization management services in three areas: (1) financial savings; (2) beneficiary health indicators; and (3) beneficiary medication adherence and access: Financial savings. Twenty-seven of the 36 studies we reviewed that examined financial savings found that utilization management services were associated with savings for the Medicare program, Part D plans, or beneficiaries. For example, all eight studies that examined the relationship between generic substitution and financial savings found savings. Of the 10 studies that did not find financial savings, five found no statistically significant impact of the utilization management service on savings, three found the utilization management service was associated with a decrease in savings, and two found both an increase and decrease in savings for different types of utilization management services. Beneficiary health indicators. Twelve of the 20 studies that examined beneficiary health indicators found that utilization management services were associated with improvement, such as a reduction in adverse drug events. Ten of the 12 studies that found improvement examined either medication therapy management programs or comprehensive medication reviews. The other two studies that found improvement looked at drug utilization reviews, which examine a beneficiary’s prescriptions to identify safety considerations, such as potential adverse interactions with other drugs and compliance with clinical guidelines. Of the eight studies that found no improvement, one found that a health indicator worsened, and four found improvement in at least one health indicator and a decline in at least one other indicator. Beneficiary medication adherence and access. Of the 15 studies that examined the effect of utilization management services on beneficiaries’ medication adherence or access to clinically appropriate drugs, 10 examined medication therapy management programs or comprehensive medication reviews. Seven of these 10 found improvement in medication adherence. In contrast, the other five studies that examined adherence and access found negative, mixed, or no effects associated with prior authorization and step therapy. For example, two studies examined the effect of prior authorization and step therapy and found that these utilization management services resulted in increased access problems. Two other studies examined the relationship of prior authorization and step therapy adherence and found a mixed impact. The remaining study examined the relationship of only prior authorization with the time needed to access medications and found no clinically significant impact. Stakeholders we interviewed generally agreed that utilization management services resulted in financial savings but differed in their views regarding the effect of utilization management services on beneficiaries’ medication adherence and access to clinically appropriate drugs. In interviews with representatives from PBMs, Part D plan sponsors, and a manufacturer trade association, these stakeholders generally agreed that utilization management services resulted in financial savings. While representatives from most Part D plan sponsors and PBMs told us that utilization management services have resulted in no adverse impact on medication adherence and access to prescriptions, representatives of the three drug manufacturers we interviewed told us that utilization management services limit medication adherence and access to medications by, for example, delaying therapy to needed drugs. See appendix VI for more information about the effects of utilization management services from the peer-reviewed studies we examined and the stakeholders we interviewed. See appendix VII for the articles included in our literature review. The Department of Health and Human Services provided technical comments on a draft copy of this report, which GAO incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees and the Secretary of Health and Human Services. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or dickenj@dickenj@gao.gov.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VIII. This appendix provides details on our scope and methodology in addressing each of our five reporting objectives: (1) the extent to which Part D plan sponsors contract with pharmacy benefit managers (PBM) to deliver drug benefit management services to Medicare beneficiaries; (2) how PBMs earn revenue from the services they provide to Part D plan sponsors; (3) trends in rebates and other price concessions obtained by Part D plan sponsors and PBMs from manufacturers and others for Part D drugs; (4) the extent to which prices for Part D drugs are discounted off of manufacturer list prices; and (5) what is known about savings and other effects from utilization management services commonly used in Part D. In addition, the appendix describes the steps we took to assure the reliability of the data we analyzed. For all our objectives, we obtained the perspectives of stakeholders on Part D plan sponsors’ use of PBMs as well as information on sponsors’ efforts to control Part D expenditures and drug utilization. We spoke to representatives from 17 small, mid-sized, and large Part D plan sponsors: Aetna, Anthem, Banner Health, Cambia Health, Cigna, CVS, Express Scripts, Kaiser, Health Care Service Corp, Health Plan of San Mateo, Henry Ford Health System, Humana, Missouri Highways and Transportation Commission, Rite Aid, United Health Care, University of Pittsburgh Medical Center, and WellCare. We spoke with seven PBMs: Argus, CVS Caremark, EnvisionRx, Express Scripts, MedImpact, Prime Therapeutics, and OptumRx. To obtain other drug industry perspectives, we spoke with representatives from three drug manufacturers: Eli Lilly, Gilead, and Amgen. We also spoke with one entity that is both a wholesaler and pharmacy services administrative organization: AmerisourceBergen. Additionally, we spoke with other industry and advocacy organizations, including groups representing drug manufacturers, Part D plan sponsors, pharmacies, and PBMs: America’s Health Insurance Plans, Biotechnology Innovation Organization, Community Oncology Alliance, National Association of Chain Drug Stores, National Association of Specialty Pharmacies, National Community Pharmacists Association, Patients for Affordable Drugs, Pharmacy Benefit Management Institute, Pharmaceutical Care Management Association, and Pharmaceutical Research and Manufacturers of America. To determine the extent to which PBMs provided services to Part D plan sponsors, we analyzed the Centers for Medicare & Medicaid Services’ (CMS) Health Plan Management System (HPMS) data that identified the entity or entities responsible for performing each of 10 key drug benefit management services for plan sponsors’ Part D contracts in 2016, the most recent available expenditure and rebate and other price concession data at the time of our analysis. CMS provided HPMS data for the 624 Part D plan sponsor contracts that were effective in 2016. The data contained the entity or entities reported by each plan sponsor as performing each service. Using this information, we identified for each contract whether the plan sponsor performed a service itself; contracted with a PBM to perform the service; or performed the service in coordination with a PBM. For a given contract, we counted as being a PBM any entity that was not the plan sponsor that performed one or more drug benefit management services. We manually reviewed those PBMs against a list of PBM members from a PBM trade organization. We used internet searches to confirm the entity was not the plan sponsor in instances when it was not listed in the trade organization’s member directory. In doing so, we also identified whether the plan sponsor shared common ownership with the PBM responsible for providing the drug benefit management service. For example, there were instances where the plan sponsor and PBM were sister organizations owned by the same parent company. In this situation, we counted the PBM as a separate entity from the plan. In addition, we analyzed PBM use by plan sponsor contract enrollment size using CMS contract enrollment information from June 2016. Additionally, we used HPMS data to examine plan sponsor contracts’ variation in the number of PBMs used, the types of services that PBMs provided, and the use of PBMs by contract enrollment size. We also identified the PBMs that provided the most services and described the services they provided. Last, we interviewed Part D plan sponsor representatives to understand the considerations that influenced their decision about how and whether to use a PBM. To determine how PBMs earned revenue from services they provide to Part D plan sponsors, we relied on four information sources. First, we reviewed selected service agreements between PBMs and Part D plan sponsors. The service agreements generally contain detailed information on the services that the PBM will provide, how the plan sponsor will pay the PBM for those services, and the rates that pharmacies will be paid for Part D drugs. We asked CMS for a list of all service agreements it approved between January 2016 and May 2018 that were in effect as of June 2018. CMS provided us with a list of 119 service agreements. Using June 2018 Part D publicly available enrollment data from CMS, we obtained from CMS the 20 service agreements for Part D plans sponsors with the largest enrollment in June 2018. While most of the service agreements included sufficient information to determine how the PBMs were paid, some did not, and, where appropriate, we noted these instances in our findings. Second, we examined PBM revenue reported to CMS by Part D plan sponsors in their rebates and other price concession data—also referred to as direct and indirect remuneration (DIR)—for 2014, 2015, and 2016. These rebate and other price concession submissions contain information on the various sources of revenue and expenses incurred by PBMs and plan sponsors. Third, we reviewed applicable CMS regulations and guidance on the reporting of PBM and Part D plan sponsor revenue and expenses. Fourth, we interviewed PBM representatives about the extent to which PBMs retained rebates or passed them through to plan sponsors and, in some cases, the reasons for this decision. We also asked certain PBM representatives whether their revenue sources for Part D, specifically rebate retention and spread pricing, differed from PBMs’ and plan sponsors’ commercial business and, if so, the reasons for any differences. To examine rebates and other price concessions obtained by Part D plan sponsors and PBMs from manufacturers and others for Part D drugs, relative to overall Part D expenditures, we analyzed plan sponsors’ gross and net expenditures for Part D drugs for 2014 through 2016, the most recent data available at the time of our analysis. Gross expenditures reflect what was paid to the pharmacy by the plan sponsor, PBMs on the sponsor’s behalf, and the beneficiary for a given drug. Net expenditures reflect any rebates and other price concessions obtained by Part D plan sponsors and PBMs after a beneficiary receives a drug. To calculate gross expenditures, we used Medicare prescription drug event (PDE) data to calculate gross brand-name and generic drug expenditure and utilization information for all Part D plan sponsors’ contracts. We used Red Book, a compendium published by Truven Health Analytics, to determine whether drugs were brand-name or generic. We then identified individual brand-name and generic drugs by grouping expenditure claims with the same active ingredient, strength, dosage form, and route of administration (known as ISDR). We calculated brand- name and generic drug expenditures based on a drug’s ingredient cost, dispensing fees, sales tax, and applicable vaccine administration fees. We used PDE data to calculate gross expenditures for all Part D plan sponsors at both the contract and plan sponsor level. We used DIR data to determine the amount of rebates and other price concessions and subtracted this amount from this data to calculate net expenditures. We also obtained plan sponsor enrollment data using publicly available CMS data for June 2016, which allowed us to calculate gross per beneficiary expenditures. We also examined differences in the amount of rebate and other price concessions obtained relative to expenditures for Part D plan sponsors that used a PBM relative to those that did not. We determined PBM involvement in rebate and other price concession negotiations for individual plan sponsors using 2016 HPMS data. We specifically looked at each entity listed in HMPS as negotiating rebates and other price concessions with drug manufacturers and others. We were able to determine whether a PBM or plan sponsor performed this service for 197 plans sponsors. However, there were 20 Part D plan sponsors where a PBM or plan was not solely listed as performing the rebate and other price concession service. In these instances, we could not identify which entity negotiated rebates and other price concessions and therefore excluded them from this analysis. To obtain more information on drugs that have the greatest fiscal impact on the Part D program and beneficiaries, we calculated gross and net expenditures for the brand-name and generic drugs with the highest expenditures, highest utilization, and highest expenditure per utilization in 2016. For both brand-name and generic drugs, we identified the following: the 200 brand-name and 200 generic drugs with the highest expenditures in 2016; the 200 brand-name and generic drugs with the highest utilization in 2016 (based on number of 30-day prescriptions); and the 200 brand-name and generic drugs with the highest expenditures per utilization (i.e., highest expenditure per number of 30-day prescriptions). As a result of overlap in the groups of drugs, these criteria yielded two groups: the 444 highest expenditure, highest utilization brand-name drugs and the 476 unique highest expenditure, highest utilization generic drugs. These 920 drugs accounted for 81 percent of total Part D expenditures in 2016. We used drug-level rebate and other price concessions data to calculate net drug prices for these drugs by subtracting rebate and other price concessions for each drug from gross expenditures. To determine the extent to which Part D drug prices are discounted off of manufacturer list prices, we compared the median gross and net prices for the 444 brand-name and 476 generic highest expenditure, highest utilization drugs to (1) list prices established by manufacturers, and (2) the cost to pharmacies of acquiring these drugs. For list prices, we used 2016 average wholesale price (AWP) data from Truven Health Analytics’ Red Book. AWP is a common benchmark drug price used in the negotiation of payment rates between Part D plan sponsors and pharmacies. Because AWP is updated on an ongoing basis, we calculated a day-weighted per unit price that takes into account the number of days that the reported price was in effect in 2016. We then determined the median AWP price for each drug product based on the ISDR. We refer to the median price as the manufacturer list price. For pharmacy acquisition costs, which reflect the price pharmacies paid to obtain the drug, we used retail community pharmacy acquisition cost data from National Average Drug Acquisition Cost (NADAC) data. NADAC does not contain data from non-retail pharmacies, such as mail- order or specialty pharmacies. For our groups of 444 brand-name and 476 generic drugs, we separated drugs sold in retail community pharmacies from those sold in specialty pharmacies. If a drug did not have pharmacy acquisition cost data from NADAC, we considered that drug to be sold in specialty pharmacies and, thus, a specialty drug. We used 2016 PDE data to determine the gross per unit Part D price for a drug by dividing the gross expenditures for the drug by the total quantity dispensed of it. For example, a drug that had 1,000 units prescribed to Medicare beneficiaries and $5,000 in gross expenditures would have a gross per unit price of $5. We determined net per unit Part D prices for the drugs in our two study groups by dividing the amount of rebates and other price concessions for each drug by the quantity dispensed of it and then subtracting the amount of rebates and other price concessions per quantity from the gross Part D price for each drug. For each drug, we then determined the median pharmacy acquisition cost (if available), median gross Part D price, and median net Part D price as a proportion of median manufacturer list price by dividing each price by the median manufacturer list price. We then reported the median value for these pricing points for the highest expenditure, highest utilization drugs in our analysis. To determine what is known about the impact of utilization management services that PBMs commonly provide to Part D plan sponsors, or that plan sponsors may perform themselves, we conducted a literature search for studies that examined the effect of utilization management services in Part D (regardless of whether they were provided by a PBM or another entity) on the following outcomes: (1) financial costs or savings, (2) beneficiaries’ health indicators, and (3) beneficiaries’ access to clinically appropriate medications or taking their medications as prescribed (adherence). The literature search was performed from April 2018 to July 2018 using keyword searches in bibliographic databases, including ProQuest, EBSCO, and Scopus. We limited our search to studies published beginning in 2006—the year the Part D program began. For our searches, we developed a list of search terms for our literature review by reviewing relevant background documentation and several database searches. The search terms included: “utilization management,” “prior authorization,” “quantity limits,” “step therapy,” “generic substitution,” “drug utilization review,” “quantity edit,” “medication therapy management,” and “comprehensive medication review,” combined with “access,” “adherence,” “health benefit,” “clinical outcome,” “generic use,” “cost effectiveness,” “savings,” “costs,” and “Medicare.” The literature search generated 700 studies. We reviewed this list by examining the abstracts for those studies that addressed the effects of utilization management services in Part D and were published in peer- reviewed journals. We identified 48 studies that met our criteria then added four more that met the criteria from several literature reviews we examined, resulting in a final group of 52 peer-reviewed studies that we analyzed. We analyzed these studies to group them by type of utilization management service evaluated and type of outcome measured. We documented any methodological limitations of these studies but did not exclude any of them on this basis. See the bibliography in Appendix VII for a list of the 52 studies in our review. We also interviewed PBMs, plan sponsors, and drug manufacturers to obtain their views regarding the impact of utilization management services in Part D plans and asked them to recommend additional studies on utilization management services. We did not assess the methodology or data reliability of the studies provided to us by these drug supply chain stakeholders; none of them met our criterion of being published in peer- reviewed journals. We used these studies to better understand stakeholder perspectives. To ensure the data used to produce this report were sufficiently reliable, we took several steps. We performed data reliability checks on the HPMS data by reviewing the data for missing values and errors, checking the information against other publicly available sources, and interviewing knowledgeable agency officials. We performed data reliability checks on the PDE and DIR data by reviewing relevant documentation, checking the data for outliers and errors, and interviewing knowledgeable agency officials. We performed data reliability checks of the AWP and NADAC data sets by testing the data for missing data and outliers and reviewing relevant documentation. After taking these steps, we determined the data were sufficiently reliable for the purposes of our reporting objectives. We conducted this performance audit from May 2017 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings based on our audit objectives. Appendix II: Medicare Part D Plan Sponsors’ Use of Pharmacy Benefit Managers (PBM) This appendix provides additional detail on the use of PBMs by Part D plan sponsors to provide prescription drug benefit management services to Medicare beneficiaries. We examined Centers for Medicare & Medicaid Services’ (CMS) data to identify the 10 key drug benefit management services provided by PBMs under 624 Part D plan sponsor contracts in 2016, the most recent available expenditure and rebate and other price concession data at the time of our analysis, and found the following variation in plan sponsor use of PBMs: Services provided by PBMs. Part D plan sponsors’ contracts varied by the services provided by PBMs in 2016. Plan sponsors’ use of a PBM for drug benefit services—either alone or with the plan sponsor—for their 624 contracts varied from 30 percent for enrollee appeals and grievance process-management to 99 percent for claims adjudication. For seven of the 10 drug benefit management services, PBMs—either alone or in conjunction with the plan sponsor—provided services to more than half the sponsor contracts (see fig. 4). Number of PBMs used. Part D plan sponsor contracts varied in the number of PBMs used to provide one or more of the 10 drug benefit management services. For example, 54 percent of plan sponsors’ contracts used a single PBM, while 11 percent used four or more PBMs (see fig. 5). Use of PBMs by enrollment. Smaller Part D contracts—those with contract enrollment below the median enrollment of all Part D contracts—used a PBM more often than larger contracts—those with enrollment at or above the median. For instance, 87 percent of smaller Part D plan sponsor contracts used a PBM alone or with the plan sponsor for rebate and price concession negotiations, compared to 77 percent of larger contracts. Similarly, 54 percent of smaller Part D contracts used a PBM alone or with a plan for a pharmacy and therapeutics committee, compared to 35 percent of larger contracts. Use of financially related PBMs. Part D plan sponsors’ contracts varied by their use of PBMs with which they were related by common ownership—either as a subsidiary or a sister company. In 2016, plan sponsors used a PBM with which they were related by common ownership for 17 percent of the 624 Part D plan sponsors contracts. Larger contracts—those with enrollment at or above the median— were more likely to use a PBM related by common ownership than smaller contracts. Larger contracts used a financially related PBM for 24 percent of drug benefit management services, compared to 10 percent of drug benefit management services provided to smaller contracts. The Part D plan sponsor representatives with whom we spoke noted several considerations that influenced their decision about how and whether to use a PBM. One plan sponsor noted that small plans may lack the resources to conduct their own rebate negotiations and, therefore, may use a PBM instead. Three other plan sponsors noted they switched from conducting their own rebate negotiation with manufacturers to using a PBM. Two plan sponsors said this switch was due to PBMs’ ability to obtain larger rebates than the plan sponsor could, and the third determined a PBM would help it achieve the best value and quality, while meeting Part D’s regulatory requirements. In contrast, representatives of three other Part D plan sponsors noted advantages of performing drug benefit management services themselves. For example, one plan sponsor noted that it performs almost all drug benefit management services internally, as it believes doing so improves quality through better communication and care coordination with pharmacies. Another plan sponsor noted the decision not to contract out certain services to a PBM may be influenced by a desire for more customization over formulary management and greater control over prior authorization. Representatives of one plan sponsor noted that their plan does not use a PBM because they believe they are more effective in developing formularies with better utilization management and greater use of generic drugs than are PBMs. Our analysis of CMS data for the 624 Part D plan sponsor contracts found that the five PBMs that provided the largest number of services to Part D plan sponsors’ contracts in 2016 also generally provided a full range of PBM services to them. Four of the top five PBMs provided all 10 drug benefit management services to plan sponsors’ contracts while the fifth PBM conducted claims adjudication but used an intermediary to conduct rebate negotiations. (See table 4). Furthermore, the top five PBMs provided a high proportion of the services that Part D plan sponsors most commonly used a PBM to provide. For example, CVS Caremark provided claims adjudication to 144 (23 percent) of Part D plan sponsor contracts, and OptumRx provided this service to 138 (22 percent). In contrast, we found that Part D plan sponsors used a large number of PBMs to provide a limited range of drug benefit management services. For example, 48 percent of PBMs provided only one type of drug benefit management service to plan sponsors’ contracts, and 22 percent of PBMs provided only one service to only one plan sponsor contract. For instance, there were 10 unique entities counted as PBMs in our analysis that provided only customer service support to one plan sponsor contract. One PBM representative noted in an interview that it is relatively common for plan sponsors and PBMs to contract with other vendors to provide additional assistance with drug benefit management services. One plan sponsor told us, for example, that its PBM uses a vendor to manage customer service calls. This appendix provides additional detail on (1) non-rebate revenue that PBMs may earn for services provided to manufacturers and Medicare Part D plan sponsors, and (2) PBM perspectives on Centers for Medicare & Medicaid Services (CMS) policies relating to spread pricing in Part D. PBMs and Part D plan sponsors may earn non-rebate revenue from manufacturers for providing certain services. Even though this money is reported to CMS as part of the rebate and other price concession submission, not all of it is considered rebates or other price concessions, which will lower plan liability in determining bids and thereby lower premiums. Of the $516.5 million in non-rebate revenue paid by manufacturers in 2016, $440 million, or about 85 percent, represented the amount paid for the services that exceeded the fair market value of the service and is considered rebates and other price concessions. These may be used to reduce the drug costs incurred by the plan sponsor. Therefore, this revenue factors into bid determinations and may be used to reduce premiums. The remaining $78.6 million in payments from manufacturers were considered “bona fide service fees”—fees paid by manufacturers to Part D plan sponsors and PBMs for services that the manufacturer would otherwise perform, or contract for, and that represented the fair market value of those services. Such fees do not reduce the plan sponsor’s drug costs and, therefore, could not factor into reducing premiums. The determination of a bona fide service fee as reported to CMS is made by the drug manufacturer and the Part D plan sponsor and is not routinely evaluated by CMS, agency officials told us. However, CMS requires that the PBM and manufacturer have information documenting the fair market value of the service. CMS requires Part D plan sponsors to report revenue earned from rebates retained by the PBM. This revenue increases the plan’s liability, which increases the amount of plan bids and, therefore, result in higher premiums. In contrast, rebate revenue passed through by PBMs to Part D plan sponsors lowers the plan’s liability, reduces plans bids, and, therefore, lowers beneficiary premiums. Some PBMs earn more revenue from spread pricing in their commercial business than in Part D, officials from three PBMs told us. Officials from two of these PBMs noted that CMS requirements create a disincentive to engage in spread pricing that is not present in the commercial sector. Beginning in 2010, CMS required that plan sponsors base the amount of beneficiary cost-sharing on the amount received by the pharmacy for a drug—known as the “pass-through price.” CMS also required that an estimate of rebates or other price concessions be included in the administrative costs submitted by the plan sponsor for bid determinations. Part D plan sponsors can still agree to pay the PBM based on the higher price of the drug without accounting for rebates, known as the lock-in price. However, the difference between that amount and the pass-through price would increase the bid determination and ultimately increase the premiums that plans charge beneficiaries. Because there are no similar requirements pertaining to the commercial prescription drug benefit market, spread pricing is more common there, CMS officials told us. This appendix provides information on (1) pharmacy-related price concessions for all Medicare Part D drugs and (2) expenditure and rebate and other price concession information for the 444 highest expenditure, highest utilization brand-name Part D drugs in 2016. The appendix also contains additional information on expenditures and rebates and other price concessions obtained by the 16 Part D plan sponsors whose representatives we interviewed. The amount of pharmacy-related price concessions obtained by Part D plan sponsors, or pharmacy benefit managers (PBM) on plan sponsors’ behalf, increased 295 percent from 2014 through 2016, from $538 million to $2.1 billion (see fig. 6). These monies account for any adjustments to the price of the drug paid to the pharmacy after the point sale, such as a pharmacy returning money that was overpaid by the plan sponsor or vice versa. It can also include monies paid based on pharmacies’ performance in meeting agreed-upon performance metrics—for example, fees a pharmacy pay plan sponsors, or bonuses pharmacies receive from plan sponsors, based on their performance. In 2016, Part D plan sponsors received $2.3 billion from pharmacies and paid out $211 million, for a net of $2.1 billion in pharmacy-related price concessions. Five of the seven PBMs and seven of the 12 Part D plan sponsors whose representatives we interviewed said they have performance-based arrangements with pharmacies. One plan sponsor noted that its performance agreement involves paying bonuses to pharmacies that exceed performance measures, while charging fees to pharmacies that did not meet the measures. The sponsor said this is part of an attempt to move from paying for volume to paying for value. Another plan sponsor told us there has been an improvement in pharmacy performance as a result of the program. Representatives from pharmacy industry groups said these pharmacy- related fees have put increasing pressure on pharmacies. For example, one group noted there is no standardization across measures with each plan sponsor using its own measures, and it is difficult for pharmacies to tie a fee to a specific pharmacy location or claim. Another group noted that fees may be imposed on pharmacies for performance measures not directly applicable to the pharmacy. For example, the group said specialty pharmacies have been assessed fees for beneficiary lack of adherence to maintenance medications, such as blood pressure medications, that these pharmacies do not commonly provide. PBMs and Part D plan sponsors obtained rebates and other price concessions for 441 (99 percent) of the 444 highest-expenditure, highest- utilization brand-name drugs in 2016. The amount of rebates and other price concessions for each drug ranged from $1,300 to $1.8 billion in 2016, with a median of $3.3 million. Rebates accounted for $24.5 billion of the $26 billion in rebates and other price concessions (94 percent) obtained by plan sponsors and PBMs for these 444 drugs. As a proportion of gross Part D expenditures—the amount paid by plan sponsors, or the PBM on the sponsors’ behalf, and by beneficiaries—for the 444 drugs ranged from -0.5 percent to 70.5 percent. (See fig. 7.) Expenditures and rebates and other price concessions varied by therapeutic class for the 444 highest expenditure, highest utilization drugs in 2016. Among those with 10 or more drugs in their class, gross expenditures ranged from $2.9 billion to $21.2 billion, and rebates and other price concessions ranged from $170 million to $8.7 billion (see table 5). Four classes—endocrine metabolic agents, anti-infective agents, respiratory agents, and central nervous system agents—accounted for 54 percent of the gross Part D expenditures, and 62 percent of rebates and other price concessions for the 444 highest expenditure, highest utilization drugs. When accounting for rebates and other price concessions, these drugs accounted for 51 percent of net Part D expenditures. Rebates and other price concessions as a proportion of gross expenditures varied from 4 percent to 27 percent in 2016 for the 17 Part D plan sponsors whose representatives we interviewed. Gross Part D expenditures per beneficiary ranged from $1,772 to $5,583, and net Part D expenditures per beneficiary ranged from $1,687 to $4,837 (see table 6). This appendix contains additional information on the gross and net discounts for the highest expenditure, highest utilization brand-name and generic Medicare Part D drugs in 2016. The amount of discounts in 2016 for the 444 highest expenditure, highest utilization brand-name drugs varied by whether they were sold in retail or specialty pharmacies. Discounts also varied by whether the brand-name drugs were highest expenditure, highest utilization or highest expenditure per utilization drugs. Of the 444 highest expenditure, highest utilization brand-name drugs, 244 were sold in retail pharmacies and 200 were sold in specialty pharmacies. Brand-name retail drugs. The three groups of drugs all had pharmacy acquisition costs that were 81 percent of manufacturer list prices and gross Part D prices that were between 83 and 84 percent of manufacturer list prices in 2016. However, the net prices varied, ranging from 55 percent of manufacturer list price for the highest utilization drugs to 77 percent for the highest expenditure per utilization drugs (see table 7). Brand-name specialty drugs. The 38 highest expenditure drugs and 187 highest expenditure per utilization drugs sold in specialty pharmacies had median gross prices that were between 84 and 85 percent of manufacturer list price and net prices that were 84 percent of manufacturer list price in 2016. We also found variation in brand-name prices across therapeutic classes for the 244 highest expenditure, highest utilization Part D drugs sold in retail pharmacies. In 2016, median gross Part D prices for the brand- name drugs sold in retail pharmacies were similar across the nine therapeutic classes we analyzed, ranging from 81 percent to 84 percent of the manufacturer list price. However, there was a much wider range among median net prices, from 43 percent to 83 percent of manufacturer list price. Anti-infective agents had the lowest percentage point changes in their prices from gross to net (1 percentage point), while endocrine metabolic agents, cardiovascular agents, respiratory agents, ophthalmologic agents, and genitourinary agents had the largest changes, with declines from gross to net of greater than 30 or more percentage points (see table 8). In contrast, there was little variation in both median gross and net prices across all therapeutic classes for brand-name drugs sold in specialty pharmacies. The range in median gross prices as a proportion of manufacturer list prices across the six therapeutic classes was 83 percent to 86 percent, and the range in median net prices as a proportion of manufacturer list prices was 80 percent to 84 percent. In 2016, discounts off of the manufacturer list price varied by whether the generic drug was sold in retail pharmacies or in specialty pharmacies. Of the 476 highest expenditure, highest utilization generic drugs in our analysis, the 367 sold in retail pharmacies had a median gross and net Part D price that were 66 percentage points lower than the manufacturer list price, and 13 percentage points higher than the pharmacy’s cost of acquiring the drugs. The 109 generic drugs sold in specialty pharmacies received far fewer discounts off of manufacturer list price than drugs sold in retail pharmacies. Median gross and net prices for those drugs sold in specialty pharmacies were both 26 percentage points lower than manufacturer list prices (see fig. 8). Therefore, generic drugs sold in retail pharmacies received median discounts (66 percent below manufacturer list prices) that were 2.5 times larger than those generic drugs sold in specialty pharmacies (26 percent below manufacturer list prices). We also found pricing variation by whether the generic drugs were in the 200 highest expenditure, 200 highest utilization group, or the 200 highest expenditures per utilization group. Generic retail drugs. Of the 367 generic drugs sold in retail pharmacies, 200 were in the group of the 200 highest utilization generic drugs, 198 were in the group of the 200 highest expenditure generic drugs, and 91 were in the group of the 200 generic drugs with the highest expenditure per utilization. We found that the gross Part D price for the highest utilization drugs was 14 percent of the manufacturer list price, while the gross price for the highest expenditure drugs was 34 percent of the manufacturer list price. However, the Part D gross price for the highest expenditure per utilization drugs was 63 percent of the manufacturer list price. The difference in gross and net Part D price as a percentage of manufacturer list price was one percentage point or less for all three groups of drugs (see table 9). Generic specialty drugs. Of the 109 generic drugs sold in specialty pharmacies, none was in the group of the 200 highest utilization generic drugs, two were in the group of the 200 highest expenditure generic drugs, and all 109 were in the group of the 200 highest expenditure per utilization generic drugs. The gross Part D price for the highest expenditure per utilization drugs sold in specialty pharmacies was 74 percent of the manufacturer list price, and these drugs received no additional rebates and other price concessions. There was variation in generic drug pricing across the eight therapeutic classes for generic drugs sold in retail pharmacies. Median gross Part D prices for generic retail drugs ranged from 14 percent of manufacturer list prices for cardiovascular agents to 56 percent of manufacturer list prices for dermatological agents (see table 10). However, there was little difference between in median gross and net Part D prices as a percentage of manufacturer list price for generic retail drugs in any therapeutic class, with the percentage difference ranging from 0 percent to 2 percent. There was little variation in median gross and net prices across the therapeutic classes for generic drugs sold in specialty pharmacies. The range in median gross prices as a percentage of manufacturer list prices was 73 to 75 percent (see table 11). There was little difference between median gross and Part D net prices as a percentage of manufacturer list price, with the percentage difference between median gross and net prices 1 percent or less for all classes. This appendix contains additional details on our review of 52 peer- reviewed studies on the effects of utilization management services on (1) financial savings, (2) beneficiary health indicators, and (3) beneficiary medication adherence and access, as well as stakeholders’ views on these effects. Of the 36 studies that examined the effect of utilization management services on financial savings, 18 examined medication therapy management programs and eight examined generic substitution. The two groups of studies found the following: Medication therapy management programs or comprehensive medical reviews. Thirteen of the 18 studies that examined the relationship between a medication therapy management program or comprehensive medical review and financial savings found an increase in savings. For example, one study found that a medication therapy management program conducted by telephone decreased beneficiary drug costs by $682 per beneficiary for participants, compared to an increase of $119 for those not in the program. Generic and therapeutic substitution and generic dispensing rate. Of the 8 studies that examined the relationship between generic and therapeutic substitution and financial savings, all found an increase in savings. For example, a 2013 study examined the potential financial savings to beneficiaries and Part D plan sponsors of generic and therapeutic substitution of commonly prescribed drugs. The study estimated that in 2007, generic and therapeutic substitutions could have resulted in an average annual savings of $127 and $389 per person, respectively. Additionally, eight of these 36 studies examined the generic dispensing rate, and all eight found that utilization management led to an increase in the rate. The generic dispensing rate—the percent of prescriptions dispensed with a generic drug instead of a brand-name drug—represents a source of financial savings through a reduction in the use of brand- name drugs, which are generally more expensive than generics. For example, a 2017 study analyzed 2012 Part D data to examine the impact of prior authorization and step therapy on generic use among low-income subsidy beneficiaries. This study found that those randomly assigned to a plan using both prior authorization and step therapy had an increased generic dispensing rate of 3 to 15 percentage points for all three classes of drugs examined. Twelve of the 20 studies that examined beneficiary health indicators found that utilization management services were associated with improved indicators, while the other eight found a mixed impact, no impact, or a decline. Examples of studies that looked at the association of utilization management services with beneficiary health indicators include: A study analyzing data from three Part D plan sponsors, which found there was a nearly 50 percent reduction in the use of potentially harmful drugs by beneficiaries 6 months after the implementation of a retrospective drug utilization review program. A randomized trial of medication therapy management for Part D beneficiaries found a nearly 60 percent reduction in beneficiaries’ drug therapy problems over time among two groups after the medication therapy management intervention. Fifteen studies examined the effect of utilization management services on beneficiary medication adherence and access. Seven of the 10 studies that examined the effect of either medication therapy management programs or comprehensive medication reviews on beneficiaries’ medication adherence (taking medication as prescribed) found improvement. For example, a 2016 study used data from Part D and the U.S. Renal Data System to examine the relationship of medication therapy management eligibility with immunosuppressant drug adherence 12 months after beneficiaries received a kidney transplant. The study found that medication therapy management-eligible transplant recipients were 14 percent more likely to have improved adherence than transplant recipients who were not eligible. The other three studies that examined medication therapy management programs or comprehensive medication reviews found no statistically significant impact on adherence. The effect of two other utilization management services—prior authorization and step therapy—on beneficiary medication adherence and access (the ability to obtain clinically indicated prescriptions) is unclear, according to the studies we reviewed. The two studies that examined the relationship of prior authorization and step therapy with adherence both found a mixed impact. For example, one study examined the impact of a health plan requiring either prior authorization or step therapy on medication use among dual-eligible nursing home residents. The study found that some residents whose new plan required prior authorization or step therapy for their current medication were more likely to have gaps in medication use than those without for two of six classes of drugs in 2006, but no gaps for any of the classes for in 2007 and 2008. The two studies that examined the relationship of prior authorization and step therapy with access found an increase in medication access problems, but they did not focus exclusively on the Medicare population. For example, one study used 2006 data from a random sample of psychiatrists surveyed about their patients to examine the relationship of prior authorization and step therapy with medication access problems among dual-eligible psychiatric patients. The study found that patients in plans with prior authorization and step therapy requirements were 2.8 and 1.8 times more likely, respectively, to have experienced medication access problems than patients in plans without these requirements. This study examined the transition of dual-eligible beneficiaries from Medicaid drug coverage to Medicare Part D when the program began in 2006, so the results may not be generalizable to the entire Medicare population at present. Most representatives of pharmacy benefit managers (PBM), Part D plan sponsors, and a manufacturer trade association we interviewed generally agreed that utilization management services resulted in financial savings by requiring the use of generic drugs. Representatives of 10 of 14 plan sponsors and six of eight PBMs we interviewed stated that utilization management services generally resulted in financial savings. Representatives of one Part D plan sponsor stated that its utilization management services resulted in annual savings of approximately 3 percent. However, representatives of one Part D plan sponsor and one PBM noted that not all utilization management services result in savings. For example, they noted that improving care with medication therapy management programs may increase drug costs through increased utilization. Additionally, representatives of one Part D plan sponsor noted the savings from utilization management services in commercial plans may be greater than in Part D because the use of manufacturers’ copay coupons are prohibited in federal health care programs, including Part D. While the coupons reduce or eliminate beneficiaries’ out-of-pocket co-payments for certain brand-name drugs, thereby encouraging their use, the coupons do not affect the amount that the plans pay for drugs. Therefore, to the extent that beneficiaries in their commercial plans use coupons, Part D plan sponsors have a greater incentive to employ utilization management services in these plans to reduce the use of more expensive brand-name drugs. Representatives of Part D plan sponsors and PBMs we interviewed differed with manufacturers and, in some cases, with each other on the effects of utilization management services on various non-financial aspects of drug utilization: Beneficiary health. Representatives from all three manufacturers we interviewed stated that utilization management services negatively affected beneficiary health by reducing their access to necessary medications. In contrast, seven of the 11 Part D plan sponsors and four of the five PBMs that discussed the effect of utilization management services on beneficiary health stated that utilization management services generally resulted in improved beneficiary health. Representatives of certain PBMs and one Part D plan sponsor provided us examples of the ways utilization management services have improved their beneficiaries’ health, such as through opioid quantity limits. One Part D plan sponsor noted that point-of-sale utilization management services warn pharmacies of therapeutic duplications, toxicities across multiple prescriptions, or interactions of certain drugs with health conditions. Medication access. Representatives from all three drug manufacturers noted that utilization management services impose limits on beneficiaries’ access to drugs, while seven of nine Part D plan sponsors and three of the four PBMs who discussed this stated utilization management services had no significant restrictions on beneficiaries’ access to necessary medications. Representatives from one plan sponsor noted there are appeals processes to ensure beneficiaries’ access is not adversely impacted by utilization management services. Medication adherence. Representatives from all three manufacturers told us that utilization management services limit beneficiaries’ adherence to their medications, such as by causing delays in therapy, while seven of eight Part D plan sponsors and all four PBMs who discussed this stated utilization management services had no adverse impact on beneficiaries’ adherence to their medications. Representatives from one plan sponsor and two PBMs stated that utilization management services may have a positive impact on adherence, such as by lowering copays through generic substitution. Medicare protected classes and utilization management. Representatives from Part D plan sponsors, PBMs, and manufacturers differed in their views on the effect of Part D utilization management services restrictions on protected class drugs on beneficiary health. Representatives of two PBMs told us the effect was positive, as beneficiaries who use these drugs do not experience disruptions in therapy. Representatives of two other PBMs said there was no effect, and one said there was a negative effect—as plan sponsors were required to cover certain less effective drugs. Representatives of one PBM said that, for example, patients in commercial health plans do not have any problems accessing protected class drugs that are subject to utilization management. These representatives noted that the Centers for Medicare & Medicaid Services provides for adequate access. However, one manufacturer told us that utilization management services for HIV drugs are rightly restricted in Part D, as these services may cause disruptions in therapy, which can lead to drug resistance and poorer health outcomes. Representatives of five Part D plan sponsors said Medicare’s restrictions on the use of utilization management services for protected class drugs have had a negative impact on beneficiary health because, for example, they limit plans’ ability to ensure that a prescribed drug is appropriate, such as ensuring that a cancer drug is appropriate for a beneficiary’s weight. Another plan sponsor representative told us the restrictions may have a positive impact by reducing increases in medical costs, while another plan sponsor said the restrictions have had no impact. Abbass, I. M., E. O. Caplan, D. B. Ng, R. Kristy, C. R. Schermer, P. Bradt, J. M. Collins, W. Man, M. Chan, and B. T. Suehs. “Impact of Overactive Bladder Step Therapy Policies on Medication Utilization and Expenditures among Treated Medicare Members.” Journal of Managed Care & Specialty Pharmacy, vol. 23, no. 1 (2017): 27-37. Agarwal, A., R. A. Freedman, F. Goicuria, C. Rhinehart, K. Murphy, E. Kelly, E. Mullaney, M. St. Amand, P. Nguyen, and N. U. Lin. “Prior Authorization for Medications in a Breast Oncology Practice: Navigation of a Complex Process.” Journal of Oncology Practice, vol. 13, no. 4 (2017): e273-e282. Almodovar, A. S., D. R. Axon, A. M. Coleman, T. Warholak, and M. C. Nahata. “The Effect of Plan Type and Comprehensive Medication Reviews on High-Risk Medication Use.” Journal of Managed Care & Specialty Pharmacy, vol. 24, no. 5 (2018): 416-422. Alston, G., and C. Hanrahan. “Can a Pharmacist Reduce Annual Costs for Medicare Part D Enrollees?” The Consultant Pharmacist, vol. 26, no. 3 (2011): 182-189. Ben-Joseph, R., C.C. Chen, A. P. De, R.L. Wade, and D. Shah. “Consequences of Patient Access Restrictions to Branded Oxycodone Hydrochloride Extended-Release Tablets on Healthcare Utilization and Costs in US Health Plans.” Journal of Medical Economics, vol. 17, no. 10 (2014): 708-718. Bergeson, J. G., K. Worley, A. Louder, M. Ward, and J. Graham. “Retrospective Database Analysis of the Impact of Prior Authorization for Type 2 Diabetes Medications on Health Care Costs in a Medicare Advantage Prescription Drug Plan Population.” Journal of Managed Care Pharmacy, vol. 19, no. 5 (2013): 374-384. Bloudek, L. M., D. Makenbaeva, and M. Eaddy. “Anticipated Impact of Generic Imatinib Market Entry on the Costs of Tyrosine Kinase Inhibitors.” American Health & Drug Benefits, vol. 8, no. 9 (2015): 472-480. Branham, A., J. Moose, and S. Ferrari. “Retrospective Analysis of Medication Adherence and Cost Following Medication Therapy Management.” Innovation in Pharmacy, vol. 1, no. 1 (2010): 1-8. Branham, A. R., A. J. Katz, J. S. Moose, S. P. Ferreri, J. F. Farley, and M. W. Marciniak. “Retrospective Analysis of Estimated Cost Avoidance Following Pharmacist-Provided Medication Therapy Management Services.” Journal of Pharmacy Practice, vol. 26, no. 4 (2013): 420-427. Buhl, A., J. Augustine, A. M. Taylor, R. Martin, and T. L. Warholak. “Positive Medication Changes Resulting from Comprehensive and Noncomprehensive Medication Reviews in a Medicare Part D Population.” Journal of Managed Care & Specialty Pharmacy, vol. 23, no. 3 (2017): 388-394. Caffiero, N., T. Delate, M. D. Ehizuelen, and K. Vogel. “Effectiveness of a Clinical Pharmacist Medication Therapy Management Program in Discontinuation of Drugs to Avoid in the Elderly.” Journal of Managed Care & Specialty Pharmacy, vol. 23, no. 5 (2017): 525-531. Caplan, E. O., M. C. Guy, J. Chang, and K. Boesen. “Telephone-Based Cardiovascular Medication Therapy Management in Medicare Part D Enrollees with Diabetes.” The American Journal of Pharmacy Benefits, vol. 9, no. 2 (2017): 47-54. Chinthammit, C., E. P. Armstrong, K. Boesen, R. Martin, A. M. Taylor, and T. Warholak. “Cost-Effectiveness of Comprehensive Medication Reviews Versus Noncomprehensive Medication Review Interventions and Subsequent Successful Medication Changes in a Medicare Part D Population.” Journal of Managed Care & Specialty Pharmacy, vol. 21, no. 5 (2015): 381-389. Chisholm-Burns, M. A., C. A. Spivey, E. A. Tolley, and E. K. Kaplan. “Medication Therapy Management and Adherence among US Renal Transplant Recipients.” Patient Preference and Adherence, vol.10 (2016): 703-709. Corsi, K., V. Lemay, K. K. Orr, and L. Cohen. “Pharmacist Medication Therapy Management in Home Health Care: Investigation of a Sustainable Practice Model.” Journal of the American Pharmacists Association, vol. 58, no. 4S (2018): S64-S68. Dai, R. and J. Robst. “The Relationship between Plan Characteristics and Medicare Prescription Drug Plan Bids.” Applied Economics Letters, vol. 19, no. 1 (2012): 99-104. De Lott, L. B., J. F. Burke, K. A. Kerber, L. E. Skolarus, and B. C. Callaghan. “Medicare Part D Payments for Neurologist-Prescribed Drugs.” Neurology, vol. 86, no. 16 (2016): 1491-1498. Dodson, S. E., J. F. Ruisinger, P. A. Howard, S. E. Hare, and B. J. Barnes. “Community Pharmacy-Based Medication Therapy Management Services: Financial Impact for Patients.” Pharmacy Practice, vol. 10, no. 3 (2012): 119-124. Duru, O. K., S. L. Ettner, N. Turk, C. M. Mangione, A. F. Brown, J. Fu, L. Simien, and C.W. Tseng. “Potential Savings Associated with Drug Substitution in Medicare Part D: The Translating Research Into Action for Diabetes (TRIAD) Study.” Journal of General Internal Medicine, vol. 29, no. 1 (2014): 230-236. Egilman, A. C., J. D. Wallach, J. S. Ross, and S. S. Dhruva. “Medicare Spending and Potential Savings on Brand-Name Drugs with Available Generic Substitutes Excluded by 2 Large Pharmacy Benefit Managers, 2012 through 2015.” JAMA Internal Medicine, vol. 178, no. 4 (2018): 567- 569. Fox, D., L. D. Ried, G. E. Klein, W. Myers, and K. Foli. “A Medication Therapy Management Program’s Impact on Low-Density Lipoprotein Cholesterol Goal Attainment in Medicare Part D Patients with Diabetes.” Journal of the American Pharmacists Association, vol. 49, no. 2 (2009): 192-199. Gellad, W. F., J. M. Donohue, X. Zhao, M. K. Mor, C. T. Thorpe, J. Smith, C. B. Good, M. J. Fine, and N. E. Morden. “Brand-Name Prescription Drug Use among Veterans Affairs and Medicare Part D Patients with Diabetes: A National Cohort Comparison.” Annals of Internal Medicine, vol. 159, no. 2 (2013): 105-114. Gernant, S. A., M. E. Snyder, H. Jaynes, J. M. Sutherland, and A. J. Zillich. “The Effectiveness of Pharmacist-Provided Telephonic Medication Therapy Management on Emergency Department Utilization in Home Health Patients.” Journal of Pharmacy Technology, vol. 32, no. 5 (2016): 179-184. Gold, J. A., B. French, and L. C. Vermeulen. “Reduction of Use of Potentially Inappropriate Medications in the Elderly.” Wisconsin Medical Journal, vol. 107, no. 4 (2008): 213-214. Hoadley, J. F., K. Merrell, E. Hargrave, and L. Summer. “In Medicare Part D Plans, Low or Zero Copays and Other Features to Encourage the Use of Generic Statins Work, Could Save Billions.” Health Affairs, vol. 31, no. 10 (2012): 2266-75. Hui, R. L., B. D. Yamada, M. M. Spence, E. W. Jeong, and J. Chan. “Impact of a Medicare MTM Program: Evaluating Clinical and Economic Outcomes.” The American Journal of Managed Care, vol. 20, no. 2 (2014): e43-e51. Huskamp, H. A., D. G. Stevenson, A. J. O’Malley, S. B. Dusetzina, S. L. Mitchell, B. J. Zarowitz, M. E. Chernew, and J. P. Newhouse. “Medicare Part D Plan Generosity and Medication Use among Dual-Eligible Nursing Home Residents.” Medical Care, vol. 51, no. 10 (2013): 894-900. Moczygemba, L. R., J. C. Barner, J. C. Brannier, and E. R. Gabrillo. “Outcomes of a Medicare Part D Telephone Medication Therapy Management Program.” Journal of the American Pharmacists Association, vol. 52, no. 6 (2012): e144-e152. Moczygemba, L. R., J. C. Barner, K. A. Lawson, C. M. Brown, E. R. Gabrillo, P. Godley, and M. Johnsrud. “Impact of Telephone Medication Therapy Management on Medication and Health-Related Problems, Medication Adherence, and Medicare Part D Drug Costs: A 6-Month Follow Up.” The American Journal of Geriatric Pharmacotherapy, vol. 9, no. 5 (2011): 328-338. Moore, J. M., D. Shartle, L. Faudskar, O.S. Matlin, and T.A. Brennan. Impact of a Patient-Centered Pharmacy Program and Intervention in a High-Risk Group. Journal of Managed Care Pharmacy, vol. 19, no. 3 (2013): 228-236. Newman-Casey, P. A., M. A. Woodward, L. M. Niziol, P. P. Lee, and L. B. De Lott. “Brand Medications and Medicare Part D: How Eye Care Providers’ Prescribing Patterns Influence Costs.” Ophthalmology, vol. 125, no. 3 (2018): 332-339. Null, K. D., K. Moll, A. Sadosky, M. K. Pasquale, J. C. Cappelleri, and B. Parsons. “Trends Associated with Implementing and Lifting a Pregabalin Step Therapy Policy.” American Journal of Pharmacy Benefits, vol. 8, no. 2 (2016): e17-e24. Olvey, E. L., M. C. Guy, J. Chang, and G. H. Skrepnek. “Cost- Effectiveness of Medication Therapy Management in Part D Diabetic Enrollees.” American Journal of Pharmacy Benefits, vol. 6, no. 5 (2014): e147-e156. Pai, A.B., A. Boyd, J. Depczynski, I.M. Chavez, N. Khan, and H. Manley. Reduced Drug Use and Hospitalization Rates in Patients Undergoing Hemodialysis Who Received Pharmaceutical Care: A 2-Year, Randomized, Controlled Study. Pharmacotherapy, vol. 29, no. 12 (2009): 1433-1440. Patel, R. A., M. P. Walberg, E. Tong, F. Tan, A. E. Rummel, J. A. Woelfel, S. M. Carr-Lopez, and S. M. Galal. “Cost Variability of Suggested Generic Treatment Alternatives under the Medicare Part D Benefit.” Journal of Managed Care Pharmacy, vol. 20, no. 3 (2014): 283-290. Pindolia, V. K., L. Stebelsky, T. M. Romain, L. Luoma, S. N. Nowak, and F. Gillanders. “Mitigation of Medication Mishaps via Medication Therapy Management.” The Annals of Pharmacotherapy, vol. 43, no. 4 (2009): 611-620. Shen, X., B. C. Stuart, C. A. Powers, S. E. Tom, L. S. Magder, and E. M. Perfetto. “Impact of Formulary Restrictions on Medication Use and Costs.” The American Journal of Managed Care, vol. 23, no. 8 (2017): e265-e274. Starner, C. I., S. A. Norman, R. G. Reynolds, and P. P. Gleason. “Effect of a Retrospective Drug Utilization Review on Potentially Inappropriate Prescribing in the Elderly.” The American Journal of Geriatric Pharmacotherapy, vol. 7, no. 1 (2009): 11-19. Stebbins, M.R., D.J. Kaufman, and H.L. Lipton. “The PRICE Clinic for Low-Income Elderly: A Managed Care Model for Implementing Pharmacist-Directed Services.” Journal of Managed Care Pharmacy, vol. 11, no. 4 (2005): 333–341. Steele, K. M., J. F. Ruisinger, J. Bates, E. S. Prohaska, B. L. Melton, and S. Hipp. “Home-Based Comprehensive Medication Reviews: Pharmacist’s Impact on Drug Therapy Problems in Geriatric Patients.” The Consultant Pharmacist, vol. 31, no. 10 (2016): 598-605. Stockl, K. M., D. Tjioe, S. Gong, J. Stroup, A. S. M. Harada, and H. C. Lew. “Effect of an Intervention to Increase Statin Use in Medicare Members Who Qualified for a Medication Therapy Management Program.” Journal of Managed Care Pharmacy, vol. 14, no. 6 (2008): 532-540. Suehs, B. T., A. Louder, M. Udall, J. C. Cappelleri, A. V. Joshi, and N. C. Patel. 2014. “Impact of a Pregabalin Step Therapy Policy among Medicare Advantage Beneficiaries.” Pain Practice, vol. 14 (5): 419-426. Tang, Y., W. F. Gellad, A. Men, and J. M. Donohue. “Impact of Medicare Part D Plan Features on Use of Generic Drugs.” Medical Care, vol. 52, no. 6 (2014): 541-548. Thatcher, E. E., E. M. Vanwert, and S. R. Erickson. “Potential Impact of Pharmacist Interventions to Reduce Cost for Medicare Part D Beneficiaries.” Journal of Pharmacy Practice, vol. 26, no. 3 (2013): 248- 252. Touchette, D. R., A. L. Masica, R. J. Dolor, G. T. Schumock, Y. K. Choi, Y. Kim, and S. R. Smith. “Safety-Focused Medication Therapy Management: A Randomized Controlled Trial.” Journal of the American Pharmacists Association, vol. 52, no. 5 (2012): 603-612. Ward, M. A., and Y. Xu. “Pharmacist-Provided Telephonic Medication Therapy Management in an MAPD Plan.” The American Journal of Managed Care, vol. 17, no. 10 (2011): e399-e409. Welch, E. K., T. Delate, E. A. Chester, and T. Stubbings. “Assessment of the Impact of Medication Therapy Management Delivered to Home-Based Medicare Beneficiaries.” The Annals of Pharmacotherapy, vol. 43, no. 4 (2009): 603-610. West, J. C., J. E. Wilk, I. L. Muszynski, D. S. Rae, M. Rubio-Stipec, C. L. Alter, W. E. Narrow, and D. A. Regier. “Medication Access and Continuity: The Experiences of Dual-Eligible Psychiatric Patients during the First 4 Months of the Medicare Prescription Drug Benefit.” The American Journal of Psychiatry, vol. 164, no. 5 (2007): 789-796. West, J. C., J. E. Wilk, D. S. Rae, I. L. Muszynski, M. Rubio-Stipec, C. L. Alter, K. E. Sanders, S. Crystal, and D. A. Regier. “First-Year Medicare Part D Prescription Drug Benefits: Medication Access and Continuity among Dual Eligible Psychiatric Patients.” The Journal of Clinical Psychiatry, vol. 71, no. 4 (2010): 400-410. Winston, S., and Y.S. Lin. “Impact on Drug Cost and Use of Medicare Part D of Medication Therapy Management Services Delivered in 2007.” Journal of the American Pharmacists Association, vol. 49, no. 6 (2009): 813-820. Woelfel, J. A., S. M. Carr-Lopez, M. D. Santos, A. Bui, R. A. Patel, M. P. Walberg, and S. M. Galal. “Assessing Medicare Beneficiaries’ Willingness-to-Pay for Medication Therapy Management Services.” The Consultant Pharmacist, vol. 29, no. 2 (2014): 104-109. Zillich, A. J., M. E. Snyder, C. K. Frail, J. L. Lewis, D. Deshotels, P. Dunham, H. A. Jaynes, and J. M. Sutherland. “A Randomized, Controlled Pragmatic Trial of Telephonic Medication Therapy Management to Reduce Hospitalization in Home Health Patients.” Health Services Research, vol. 49, no. 5 (2014): 1537-1554. In addition to the contact named above, Robert Copeland, Assistant Director; William A. Crafton, Analyst-in-Charge; Britt Carlson, Kaitlin Dunn, Andrew Emmons, Michael Rose, and Dan Ries made key contributions to this report. Also contributing were George Bogart, Yesook Merrill, Laurie Pachter, and Vikki Porter.", "summary": "Total expenditures for the Medicare Part D drug program exceeded $100 billion in 2016. Part D plan sponsors may use a PBM to provide drug benefit management services for Part D coverage, such as negotiating drug rebates and other price concessions and paying pharmacy claims. Policymakers have sought a better understanding of PBMs' roles in the drug supply chain and plans' and PBMs' efforts to manage Part D drug spending and use. GAO was asked to examine the role of PBMs in the Part D program. This report examines, among other objectives, (1) the extent to which Part D plan sponsors use PBMs, (2) trends in rebates and other price concessions obtained by both PBMs and plan sponsors for Part D drugs, and (3) how PBMs earn revenue for services provided to Part D plans. GAO analyzed Centers for Medicare & Medicaid Services (CMS) data on Part D plan sponsors' use of PBMs in 2016 as well as CMS drug expenditure, pricing, and rebate and other price concession data for all Part D drugs from 2014 through 2016 (the most recent available data at the time of our analysis). GAO reviewed service agreements between Part D plan sponsors and PBMs that were approved by CMS from January 2016 through May 2018 and had the highest enrollment as of June 2018. GAO spoke with CMS officials and 38 stakeholder groups including PBMs, Part D plan sponsors, pharmacy representatives and drug manufacturers. Medicare Part D plan sponsors used pharmacy benefit managers (PBM) to provide 74 percent of drug benefit management services and performed the remaining 26 percent of services themselves in 2016—the most recent year of data at the time of our analysis. Plan sponsors are private entities that operate drug plans; PBMs are organizations that help manage drug benefits. Rebates and other price concessions—discounts generally paid by manufacturers to Part D plan sponsors and PBMs after the sale of a drug at the pharmacy—grew faster than Part D expenditures from 2014 through 2016. Specifically, gross expenditures (the amount paid to pharmacies by plan sponsors, or by the PBM on the sponsor's behalf, and by the beneficiary) increased 20 percent, to $145.1 billion. During this period, rebates and other price concessions increased 66 percent, to $29 billion—20 percent of 2016 gross expenditures. Consequently, net expenditures (gross expenditures less rebates and other price concessions) increased only 13 percent, to $116.1 billion. PBMs primarily earned Part D revenue through a volume-based fee paid by plan sponsors based on PBM-processed claims; a per-member, per-month fee paid by plan sponsors; or a combination of the two. PBMs also earned revenue from the rebates they negotiated with manufacturers for Part D drugs, which accounted for $18 billion of the $26.7 billion in rebates in 2016. PBMs retained less than 1 percent of these rebates, passing the rest to plan sponsors. Plan sponsors in turn may use rebates to help offset the growth in drug costs, helping control premiums for beneficiaries. The Department of Health and Human Services provided technical comments on a draft of this report, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "Cultivating a strong biological defense requires an understanding of a multitude of biological threats. The nature of these threats can be intentional, naturally occurring, or accidental and can be exacerbated by changes in behavior and environment. The vast and evolving biological threat landscape includes threats of biological warfare, bioterrorism, infectious disease threats to humans and animals, crop failure, and safety and security lapses at facilities that house biological threat agents. The use of biological weapons or their proliferation by state or non-state actors presents a significant challenge to our national security, our population, our agriculture, the economy, and the environment. Despite ratification of the Biological Weapons Convention in 1975 and the end of the Cold War decades later, the threat of biological warfare persists today. For example, the State Department reported in 2019 that China, Iran, North Korea, Russia, and Syria continue to engage in dual-use or biological weapons-specific activities. Additionally, the biotechnology revolution presents opportunities to advance the life sciences, yet that same technology in the wrong hands could be used to catastrophic effect. For example, synthetic biology may lead to advances in public health, such as the development of biosensors that can permanently reside in the body to detect and treat abnormalities such as cancer. However, if used to create and combine agents to create biological weapons, synthetic biology poses a significant threat. Finally, non-state actors such as terrorist organizations, domestic militia groups, and “lone wolves” have both the interest and, in some cases, the limited capacity to develop biological weapons. Biological threats can be unpredictable, as humans, animals, and plants are vulnerable to a variety of naturally occurring infectious disease and pest threats. Urbanization, habitat encroachment, and increased and faster travel, coupled with weak health systems, increase the risk of infectious diseases to spread rapidly across the globe. Pandemic influenza presents a constant threat to global public health and exemplifies the susceptibility of humans to diseases with animal origins. For example, in 2009 when an H1N1 influenza virus emerged with a new combination of genes from swine, avian, and human influenza viruses, it demonstrated how the genetic compositions of some viruses naturally change, meaning most people have little or no immunity to the new virus. In 2009, this led to a global pandemic with a novel H1N1 influenza virus (see fig.1). Other examples of zoonotic disease threats—infectious diseases that are transmissible from animals to humans—include Ebola, Zika, and Eastern Equine Encephalitis. Biological threats may also arise from changes in human behaviors. Habitat loss and human encroachment on rural and wildlife environments are bringing populations of humans and animals into closer and more frequent contact. These changing relationships with animals increase the risk of disease transmission among people, pets, livestock, and wildlife. Other changes in human behavior—such as vaccine hesitancy, mass migration, and conflict—put stress on health care systems around the world. In an ever increasing interconnected world, building biological defenses globally can help maintain health security domestically, because a disease threat anywhere is a disease threat everywhere. Biodefense capabilities are also needed to address changes in the environment which have the potential to negatively affect human health and the agriculture industry. As we reported in October 2015, climate change may contribute to the spread of vector-borne diseases that are transmitted to humans by animals, including invertebrate animals such as mosquitoes and ticks. Additionally, extreme climate conditions, such as sustained drought and heat waves can affect crops and livestock, and excess precipitation can also increase flooding events and erosion, and decrease soil quality. Losses of livestock and crops from the biological threats of disease, pests, or extreme climate conditions could have devastating effects on trade and the national economy. Finally, in many countries around the world, pathogens are stored in laboratories that lack appropriate biosecurity measures where they could be diverted by actors who wish to do harm. Advances in science and technology bring revolutionary cures and progress, but they also have the potential to facilitate intentional misuse. As we reported in 2016, some laboratories do not have appropriate biocontainment or biosafety protocols. These shortfalls could lead to outbreaks through laboratory acquired infections or pathogens accidently being released into the environment. We have previously reported on a wide range of biodefense-related efforts carried out by multiple federal departments and agencies. Since 2009, we have identified broad, cross-cutting issues in leadership, coordination, and collaboration that arise from fragmentation throughout the complex interagency, intergovernmental, and intersectoral biodefense enterprise. For example, our past work has identified a number of key challenges related to the nation’s ability to detect and respond to biological incidents that transcend what any one agency can address on its own. They include: (1) assessing enterprise-wide threats, (2) determining optimal biodetection technologies, (3) building and maintaining emerging infectious disease surveillance, (4) establishing situational awareness and data integration, and (5) enhancing biological laboratory safety and security. (Additional detail on these challenges and our related reports is presented in appendix II.) The complexity and fragmentation of roles and responsibilities across numerous federal and nonfederal entities presents challenges to ensuring efficiency and effectiveness across the entire biodefense enterprise. We called for a national biodefense strategy and focused leadership because addressing these issues is a difficult and complex challenge that crosses mission areas, federal departments, and sectors. Additionally, we have reported on enterprise risk management principles that can support enterprise-wide decision-making under complex and uncertain conditions. Enterprise risk management is a strategy for helping policymakers make decisions about assessing risks, allocating resources, and taking actions under conditions of uncertainty. While often applied at an agency level, we have also recognized that the size and complexity of certain issues, such as homeland security, involves multiple partners which can add another degree of difficulty to enterprise risk management. For certain areas, like biodefense, where activities cut across multiple federal and nonfederal entities, applying enterprise risk management principles becomes more challenging, but equally important to help ensure the responsible parties can make decisions that help to ensure effectiveness and maximize opportunities to better manage risk. Enterprise risk management in the larger interagency and intergovernmental context does not replace what each agency needs to do to pursue its own core missions. Rather, it allows agency decision makers to consider their missions and the alternatives they have to meet them from an enterprise-wide perspective. In this manner, decision makers can consider the risk-reduction contributions their actions make to the larger enterprise—for example by selecting alternatives that meet their immediate needs and provide collateral benefits to some other part of the enterprise—as one of many factors in individual agency decision- making. On September 18, 2018, the White House released the National Biodefense Strategy and characterized it as a new direction to protect the nation against biological threats and that its implementation would promote a more efficient, coordinated, and accountable biodefense enterprise. The Strategy’s five high-level goals are to help enable the efficient assessment, prevention, preparation, response, and recovery from natural, accidental, or deliberate biological threats. When the National Biodefense Strategy was released, the White House issued NSPM-14: Presidential Memorandum on the Support for National Biodefense. According to the Strategy, NSPM-14 “creates a dedicated mechanism, housed within the U.S. Department of Health and Human Services, to coordinate federal biodefense activities and assess the effectiveness with which the National Biodefense Strategy’s goals and objectives are being met.” NSPM-14 details a governance structure and implementation process to achieve the Strategy’s goals. The governance structure includes the creation of a Biodefense Steering Committee chaired by the Secretary of HHS, and includes seven other agency heads as members: the Attorney General, the Secretaries from the Departments of State and VA, DOD, USDA, and DHS, and the Administrator of the EPA. Additionally, NSPM- 14 required the formation of a Biodefense Coordination Team to assist the Biodefense Steering Committee in carrying out its responsibilities. Administratively located within HHS, the Biodefense Coordination Team consists of staff from multiple agencies with biodefense responsibilities and is designed to assist the Biodefense Steering Committee in monitoring and coordinating implementation of the Strategy (see fig.2). The Biodefense Coordination Team may convene working groups and maintain awareness of biodefense activities across the biodefense enterprise and has responsibility for establishing policies, processes, and procedures to govern its activities, subject to the approval from the Biodefense Steering Committee. NSPM-14 also establishes that the Assistant to the President for National Security Affairs will serve as the lead for policy coordination and review, providing strategic input and policy integration for federal biodefense efforts. NSPM-14 also outlines an implementation process, which sets requirements and deadlines for the interagency group to achieve the Strategy’s goals and also requires the heads of agencies identified by the Biodefense Steering Committee as having responsibilities pertaining to biodefense to review the Strategy every 2 years, and revise as appropriate. The National Biodefense Strategy and associated plans bring together all the key elements of federal biodefense capabilities, which presents an opportunity to identify gaps and consider enterprise-wide risk and resources for investment trade-off decisions. However, challenges with planning to manage change, limited guidance and methods for analyzing capabilities, and lack of clarity about decision-making processes, roles, and responsibilities while adapting to a new enterprise-wide approach could limit the success of the Strategy’s implementation. The National Biodefense Strategy and its associated plans bring together the efforts of federal agencies with significant biodefense roles, responsibilities, and resources to address intentional, accidental, and naturally-occurring threats. The Strategy and plans also provide processes for collecting and analyzing comprehensive information across the enterprise, an important step toward the kind of enterprise-wide strategic decision-making we have called for. For example, our prior work identified the need for a strategy to help ensure efficiency and effectiveness across the entire biodefense enterprise by connecting strategic approaches and investment decisions across disparate but interrelated functions within the biodefense enterprise. These functions are (1) understanding and defining threats, (2) taking action to prevent and protect against attacks and significant national and international infectious disease outbreaks, (3) employing new and existing techniques and technologies to more quickly detect biological events, and (4) preparing to respond and recover. Consistent with characteristics of national strategies and leading practices for interagency collaboration, the National Biodefense Strategy clearly articulates the purpose of the Strategy and the scope of the problem, as well as high-level goals and objectives to guide implementation. As shown in Figure 3, the Strategy’s five high-level goals comprise a new framework that incorporates the distinct biodefense functional areas and includes the different sources of biological threat—accidental, intentional, and naturally occurring. It is within this framework that national biodefense capabilities will be assessed across the enterprise. According to the Strategy, its aim is to bring together a single, coordinated effort to orchestrate activities across the United States Government to protect the American people from biological threats. The Strategy defines the term “biothreat” broadly to include all sources of major catastrophic risk, including naturally occurring biological threats, the accidental release of pathogens, and the deliberate use of biological weapons. Officials from three of the eight participating agencies that we interviewed noted that this is the first time that the federal government has identified activities across the whole biodefense enterprise and assessed resources and gaps to address multiple sources of threat regardless of source (naturally occurring, accidental, or intentional). The Strategy also established common terminology, giving the agencies a shared basis for identifying biodefense-related programs and activities, which is consistent with our national strategy criteria and our leading collaboration practices. Developing common terminology can help to bridge organizational cultures when multiple agencies with varying missions work together for a common purpose. The Strategy also contains goals, objectives, and over 240 separate activities that cover the range of actions that comprise national biodefense capabilities, which provides a high-level framework to begin to guide agencies toward a shared vision for outcomes. While the Strategy outlined high-level goals and objectives to help define priorities, NSPM-14 established a structure and process by which the federal agencies can assess enterprise-wide biodefense capabilities and needs, and subsequently develop guidance to help inform agency budget submissions. NSPM-14 lays out, in broad strokes, a process to identify biodefense efforts and assess how current resources support the Strategy, how existing programs and resources could better align with the Strategy, and how additional resources, if available, could be applied to support the goals of the Strategy. As shown in figure 4, this process begins through a data call with participating agencies documenting all biodefense programs, projects, and activities within their purview in a biodefense memorandum. As part of this process, NSPM-14 calls for the Biodefense Coordination Team, in coordination with NSC staff through the NSPM-4 process, to develop and collectively agree on metrics, milestones, and end-states and roles and responsibilities. For each of the objectives within the Strategy where agencies have roles and responsibilities, HHS directed participating agencies, as part of a data call, to identify any resource, authority, policy, science and technology, or coordination gaps against those end states and propose solutions where needed. As outlined in NSPM-14, the Biodefense Coordination Team is then to use the information submitted by the individual agencies to identify gaps, shortfalls, redundancies, and challenges across the enterprise. Finally, NSPM-14 directs officials with biodefense responsibilities to create joint policy guidance in coordination with the Assistant to the President for National Security Affairs through the NSPM-4 process—to be updated on an annual basis—that can help guide individual agency budget submissions. The process outlined in NSPM-14 is intended to lead to a cross- government assessment of federal biodefense capabilities and is consistent with our past calls for a strategy that can guide investment across the whole enterprise and with leading practices for interagency collaboration and enterprise risk management. We have previously reported that defining shared outcomes—and processes by which to achieve them—and developing mechanisms to monitor and evaluate results can reinforce accountability for collaborative efforts. Working together to develop a set of draft metrics, milestones, and end-states requires interagency participants to establish a shared vision for outcomes, and metrics and milestones serve as accountability mechanisms. NSPM-14 describes how agencies will consider the agreed upon joint policy guidance developed by agencies with biodefense responsibilities and the White House when developing their budgets. Specifically, according to NSPM-14, these agencies shall include in their respective annual budget requests to OMB information on the programs within the budget requests that support the implementation of the Strategy and conform to budget formulation requirements established by OMB, including specified funding levels. Establishing goals, objectives, and desired end states that cut across the federal government also create a foundation for effective enterprise risk management. As we have previously reported, a shared understanding of the scope of the risks enables leaders across the enterprise to align agency goals and objectives and consider their own missions and purposes within a more expansive and comprehensive understanding of threats and opportunities. In our interviews, officials from participating agencies stated that the NSPM-14 processes constitute a new approach to identifying gaps and setting budget priorities for biodefense, and that they viewed the approach as generally well designed. Specifically, officials from six of the eight participating agencies said that the process for identifying gaps was somewhat well-designed. Officials from the other two participating agencies said that this process was very well-designed. Agency officials provided several reasons for optimism about the Strategy and the processes outlined in NSPM-14, including that: They provide a holistic picture of current biodefense programs and activities, which creates government-wide visibility so that gaps can be identified. They create a forum to discuss potential gaps and biodefense responsibilities, which has not existed previously. They contain a strong overarching architecture to map existing efforts, identify gaps, and inform future revisions (as necessary). Additionally, agency officials said that the assessment and joint policy guidance development process outlined in NSPM-14 offered some promise for helping agencies identify the resources necessary to achieve the Strategy’s goals, which is consistent with our national strategy criteria. Specifically, officials from five of the eight agencies said the process is somewhat well-designed to accomplish these goals. Officials from the other three agencies said the process is very well-designed to ensure the appropriate identification of resources and investments necessary to achieve the goals outlined in the Strategy. For example, officials from three agencies said it would help the implementation of the Strategy succeed where previous efforts failed because it is designed to allow the Strategy’s priorities to drive budget decisions. However, officials from all of the agencies we interviewed, even those with the most optimistic views on the leadership and governance structure design, tempered their responses with the caveat that implementation is in such early stages that it remains to be seen how effective these structures will actually be once tested. Although the Strategy and associated plans establish the foundation for enterprise risk management, in particular by bringing together all of the functional biodefense areas across different sources of threat, we and biodefense agency officials identified multiple challenges that could affect the Strategy’s implementation. These include challenges individual agencies faced during the initial data collection process as well as a lack of planning and guidance to support an enterprise-wide approach. In our analyses and interviews, we found that parts of the process in the first year were underdeveloped, raising questions about (1) the plans to support change management practices and ensure that early- implementation limitations do not become institutionalized in future years’ efforts; (2) guidance and methods for meaningfully analyzing the data; and (3) the clarity of decision-making processes, roles, and responsibilities. During our interviews, agency officials reported challenges they faced in the first-year’s data collection effort with (1) staffing and organizational resources within individual agencies, (2) quantifying biodefense activities, and (3) technology glitches. These challenges may have led to incomplete data collection, but are not wholly unexpected given they occurred in the context of adapting to cultural change that this kind of enterprise-wide approach to managing risk represents, while implementing new processes and procedures. We have previously reported that leaders of successful transformations seek to learn from best practices and create a set of systems and processes that are tailored to the specific needs and circumstances of the new organization. However, the agencies involved in implementing the Strategy do not have a plan that includes change management practices that can help prevent these challenges from being carried forward into future efforts, and help reinforce enterprise-wide approaches, among other things. Staffing and organizational resources. During our interviews, one challenge that arose involved having the personnel and expertise needed to complete the initial effort to document biodefense programs, projects, and activities. For example, officials from one agency told us that this data collection effort was especially challenging because policy and program managers were responsible for determining both programmatic and budgetary information, which exceeded their expertise. This agency ultimately had to bring in non-biodefense personnel—including from the comptroller’s office—to identify programs and resources to complete the information request. Officials from three of the eight agencies stated that staffing and organizational resource limitations also posed a challenge to the data collection process. For example, officials from one agency said that the agency does not have full-time staff assigned to the effort. Instead, it was seen as a collateral duty competing with regular priorities, which reduced the time devoted to identifying the necessary information. Quantifying biodefense activities. Officials we interviewed also highlighted the challenge of quantifying biodefense-related activities. Specifically, officials from four agencies noted that agencies without specific biodefense line items in their budgets have had difficulty fully quantifying how much their agency invests in biodefense-related activities. To help agencies attempt to capture and quantify this information in a consistent way, the Biodefense Coordination Team developed guidance to assist agencies in estimating the percentage of their chemical-biological-radiological-nuclear (CBRN) defense, all- hazards preparedness, and agriculture programs and activities, among others, that are specifically related to biodefense. Nevertheless, officials from two agencies said that distinguishing the biodefense-specific activities within their CBRN defense or all-hazards activities and budgets was inherently challenging, which in turn required officials to invest additional staff and time into the effort. Technology glitches. Officials we interviewed also cited challenges with the technology used to collect data. For example, officials from two agencies said that they had experienced glitches with the OMB Max Information System, which the Biodefense Coordination Team guidance directed them to use for the data collection effort. They stated that the technology issues prevented them from entering biodefense budget numbers in a timely manner. Officials noted that an integrated platform dedicated to biodefense enterprise needs would enhance their collaboration, which is consistent with our work on interagency collaboration that states technology is one means of establishing compatible processes for working across interagency boundaries. HHS officials are aware of the technology challenges and said they are collecting feedback and identifying ways to improve the data collection and analytical tool for future data collection efforts. These challenges with resources, identification of budget activities, and technology occurred in the context of the individual agencies and officials adapting to new procedures and a broader cultural shift from how they have approached their biodefense missions in the past. Officials told us that because of the learning involved the first time through the process and the 2018 government shutdown, coupled with the tight time frames set forth in NSPM-14, agencies may not have submitted complete or detailed information about their biodefense programs. For example, officials at one large agency told us they treated the first year as a learning experience and that in the coming years, when agencies have sufficient time to respond to the data call, the quality of the data submitted should improve. Some officials we interviewed voiced concern that this first-year effort could set a poor precedent for these activities in future years if the challenges are not acknowledged and addressed. For example, an official noted that committing to the first-year’s results as the “baseline” for future years of the Strategy’s implementation could compound or institutionalize the issues encountered in the first year. Officials cautioned against a “garbage in, garbage out” situation, meaning the output of any analysis would only be as good as the quality of the data fed into that analysis. As agency officials described their data collection efforts, it was clear to them that the focus was on meeting the time frames established in NSPM-14 to identify existing biodefense efforts in this first year and that not all processes had been fully developed prior to the data collection effort. OMB staff acknowledged that there were challenges in the first year’s data collection effort, and said data quality would likely improve in future years as agencies adjust their internal structures to suit the demands of the NSPM-14 process. Officials from HHS and OMB staff stressed that this process will be iterative, with the first year being primarily about outlining the existing biodefense landscape. Our prior work on organizational transformations states that incorporating change management practices improves the likelihood of successful reforms and notes that it is important to recognize agency cultural factors that can either help or inhibit reform efforts. We have also reported that identifying cultural features of the originating components, prior to, or early on, in the transformation process, can help leadership gain a better understanding of their beliefs and values. Incorporating this type of change management practice can help educate agencies to better understand the varying missions and how those missions support the broader enterprise-wide effort. We have also noted the importance of communication and obtaining feedback from participants to help promote ownership for the transformation. This type of approach to managing risk across a multi-agency, multi-sectoral enterprise like biodefense is complex and novel. During our interviews, agency officials recognized a need for change management practices to support this effort in future years. Agency officials we interviewed noted that the process for the identification of biodefense resources and activities across the federal government outlined by NSPM-14 could be “transformational” for the biodefense enterprise and approached the data collection process in good faith, but said that it will take time to get right. The biodefense agencies are currently assessing the activities and challenges of the first year of implementation, and they plan to develop an after action report on lessons learned. HHS has conducted a survey and interviews to collect information and the material is being analyzed, but the lessons learned document is not yet final. HHS has not worked with the other biodefense agencies, however, to undertake an intentional effort to manage key cultural aspects of the enterprise-wide approach—such as communication and education mechanisms to help bridge organizational cultures, promote ownership of the transformation, and emphasize awareness of joint national security responsibilities. Further, HHS has not worked with the other biodefense agencies to establish feedback and monitoring mechanisms or processes, that can help identify implementation challenges and develop solutions to address those challenges, particularly early implementation issues that might threaten the efficacy of the effort if they are institutionalized going forward. A systematically developed plan for managing change could help ensure effective planning to sustain and advance transformation in the early years. Such a plan could address (1) institutionalizing learning and feedback mechanisms that allow for corrective action and ensure that issues that arise in early implementation—for example, incomplete or unreliable data—do not become entrenched in a way that plagues the future years’ efforts; and (2) establishing a communication and education strategy to reinforce collaborative behaviors, enterprise-wide approaches, and to emphasize accountability for shared national security missions, outcomes, and procedures. We found a lack of clear procedures and planning to help ensure that the Biodefense Coordination Team is prepared to analyze the data, once it has been collected, in a way that that leads to recognition of meaningful opportunities to leverage resources in efforts to maintain and advance national biodefence capabilities. In particular, HHS (1) has not documented guidance and methods for analyzing the data, including but not limited to methods and guidance for how to account for the contribution of nonfederal capabilities; and (2) does not have a resource plan for staffing and sustaining ongoing efforts. Methods and guidance for analyzing data. We found that the processes for the Biodefense Coordination Team to analyze the results of all the individual agency data submissions and identify priorities to guide resource allocation were not agreed upon or documented prior to the agency efforts and continue to lack specificity and transparency. At the time of our interviews, agency officials were in the midst of compiling and assigning budget numbers to their programs, projects, and activities. Officials we spoke with expressed uncertainty about how the information would be used. For example, officials from four agencies said they were uncertain about fundamental elements of the implementation process, including how information gathered will be used to identify gaps and set priorities. The overarching purpose of the analysis described in NSPM-14 is identification of gaps, shortfalls, and redundancies to support the goals and objectives of the Strategy. However, NSPM-14 does not specifically articulate what is meant by these terms. In response to our question about how the analysis was to be conducted, the Office of the Assistant Secretary for Preparedness and Response—the HHS office responsible for leading the Biodefense Coordination Team—described a general process that reflects the high-level description laid out in NSPM-14. HHS officials also stated that the Biodefense Coordination Team had consulted with experts in budget, planning, and evaluation while developing the methodology. However, HHS has not documented specific guidance and methodologies to help ensure transparency and accountability across the interagency and consistency in the Biodefense Coordination Team’s analysis. Additionally, the initial effort to collect information on all programs, projects, and activities focused on existing federal activities did not include a complete assessment of biodefense capabilities at the nonfederal level. Processes for soliciting nonfederal capabilities that contribute to the biodefense enterprise and are necessary to support the Strategy’s implementation are not articulated in NSPM-14. Moreover, the guidance document that agencies used for the data call stated that the Biodefense Coordination Team—in coordination with National Security Council and OMB staff—was to, among other things, use the information provided by the agencies to analyze the extent to which current U.S. Government resources support the goals and objectives of the Strategy. Officials from two agencies also said that not gathering information from the private sector and other existing biodefense working groups was a limitation in the information gathering process for this first year. Officials said these entities provide valuable subject matter expertise and including input from them in the future could help identify gaps across the biodefense enterprise. Some agencies included information about their work to support nonfederal stakeholders in their data collection effort, for example, by listing their grant programs or cooperative agreements. In addition, during our interviews, officials from all eight agencies described efforts to involve nonfederal partners when developing the Strategy and many described outreach efforts to obtain information since the Strategy’s release. For example, HHS issued a notice in the Federal Register and the Biodefense Coordination Team held a summit related to the implementation of the National Biodefense Strategy to engage nonfederal stakeholders. However, the Biodefense Coordination Team was not explicitly required to analyze nonfederal resources and there was no guidance that would help ensure agencies consistently and systematically included the contributions of nonfederal capabilities. In 2011, we reported that few of the resources required to support national biosurveillance capabilities are wholly owned by the federal government. Effective response to significant national biological incidents also relies heavily on nonfederal resources and capabilities. Because nonfederal entities own many of the resources and capabilities needed to achieve the goals and objectives outlined in the Strategy, assessing the baseline and identifying investment needs for a national biodefense capability necessarily involves assessing nonfederal entities’ ability to support a national capability. Officials from one of the agencies initially tasked with developing the biodefense strategy said the Biodefense Coordination Team needs to develop engagement structures with nonfederal partners, because currently, there is not a system in place to get everyone’s views or learn of what is going on outside the federal government. Our enterprise risk management work calls for agencies to identify and assess risks to be able to select among risk reduction alternatives. Enterprise risk management requires good information and analysis to enable officials to make informed trade off decisions across alternatives. Although the NSPM-14 process is designed to enable this kind of assessment and selection, it will not be as effective without complete information at the risk identification stage. Effective enterprise risk management implementation starts with agencies establishing a customized program that fits their specific organizational mission, culture, operating environment, and business processes. In our guide for designing evaluations, we called for plans to analyze data in ways that allow for valid conclusions to be drawn. Although the NSPM-14 guidance provides a high-level process that serves as a solid foundation for an effort as complex as managing risk across the entire biodefense enterprise, it does not provide the kind of specific guidance that can help all the involved agencies ensure they are operating off a common set of procedures that fits the particular needs of this effort. Furthermore, an analysis that cannot consistently account for the contribution of nonfederal capabilities does not reflect the true enterprise operating environment and limits the selection of alternatives available for managing risk. Clear and specific documentation of methodologies and procedures for analysis—including guidance on the methods to account for nonfederal capabilities—would provide better guidance for agencies that submit information for the assessment, assurance of more complete information to assess the state of national capabilities, and better overall transparency, accountability, and consistency. Staffing, supporting, and sustaining ongoing efforts. Officials we interviewed expressed concern about the resources that the Biodefense Coordination Team had available to it, both in the first year and on an ongoing basis. According to officials from five of the eight agencies, in order for the team to be most successful, it would need to be staffed by detailees from the participating agencies. However, officials we spoke with told us that not all agencies were able to provide a full-time detailee to help support the office. Without a dedicated liaison to the Biodefense Coordination Team, agencies may have less access to information and more limited influence over the iterative process. We have previously reported that agencies need to identify how interagency groups will be funded and staffed. HHS, which serves a leadership role on the Biodefense Steering Committee, identified in its fiscal year 2020 budget request $5 million for the resources necessary to help carry out its administrative functions for implementing the National Biodefense Strategy. However, HHS appropriations for fiscal year 2020 did not include the $5 million HHS requested. In addition, in our work on leading practices for agency reform efforts we stated that having a dedicated implementation team that has the capacity—including staffing and resources—can help ensure successful transformation. However, officials from multiple agencies reported that the initial planning for the staffing and responsibilities for the Biodefense Coordination Team had not been finalized. Without a plan to help ensure resources and mitigate resource challenges for ongoing efforts, the Biodefense Coordination Team risks not having the capacity it needs to conduct meaningful analysis, which would undermine the vision created by the Strategy and NSPM-14. The governing bodies overseeing the National Biodefense Strategy’s implementation—the Biodefense Steering Committee and Biodefense Coordination Team—did not clearly document key components of the assessment process and roles and responsibilities for joint decision- making in the first year of NSPM-14 implementation. This raises questions about how these bodies will move from an effort to catalog all existing activities to decision-making that accounts for enterprise-wide needs and opportunities. For example, officials from multiple agencies were not certain how the group would make joint decisions regarding priority setting and the allocation of resources, how the group would assign new biodefense responsibilities if gaps were identified, and to what extent the Biodefense Steering Committee could enforce budgetary priorities, if at all. Process for leveraging or directing resources. We found a lack of shared understanding and agreement about how the interagency process would work to align resources toward any identified gaps and reconfigure resources for any identified redundancies or inefficiencies. To address needs for new appropriations, NSPM-14 lays out a process to identify the need for additional resources to support the goals of the Strategy and how agencies will consider the joint policy guidance in their budget requests to Congress, but this coordination process also remains ambiguous and untested. OMB staff said the 2022 budget cycle would be the first year that agencies consider the joint policy guidance to inform their budget submissions, as envisioned by the Strategy and NSPM-14 process, as that guidance is still being developed. Officials from four agencies expressed reluctance to redirect resources away from their core missions to better support any enterprise-wide identified needs. When asked about the process outlined in NSPM-14, officials from only one of the eight agencies we interviewed said that the governing bodies were well-positioned to assign new responsibilities in response to identified gaps. Further, officials we interviewed noted that new responsibilities or activities may be difficult to implement without additional appropriations or authorities approved by Congress, or they would compete with an agency’s other priorities. When discussing their understanding of the process for prioritization and determining which agencies require what resources to help implement the Strategy, officials from four agencies referenced the NSPM-4 process (within the White House) to help guide this process. NSPM-14 also references NSPM-4, as noted above, and states the Biodefense Steering Committee seeks to reach consensus on decisions, and should any disagreements arise, the issue will be addressed through the NSPM-4 process. Through this process, the Assistant to the President for National Security Affairs serves as the lead for policy coordination and review to provide strategic input and facilitate policy integration for federal biodefense efforts. When we asked HHS officials for more specific decision-making guidance, they continued to cite the existing processes and directives for interagency decision-making. However, we found that neither of these Presidential memorandums detailed specific decision-making principles or steps for reaching consensus or even for raising decision points about how to best leverage or direct resources across the enterprise in response to gaps and inefficiencies. Similarly, agency officials we interviewed were not clear how this process would work, how decisions would be made, or how agencies would agree to take on new responsibilities to bridge gaps to achieve the Strategy’s goals. Roles and responsibilities. Similarly, the governing bodies have not fully defined the roles and responsibilities for making enterprise-wide decisions that affect individual agency budgets and for enforcing enterprise-wide budget priorities. NSPM-14 directs the heads of agencies to monitor, evaluate, and hold accountable their agencies for implementation of the Strategy, and describes how agencies will develop their budgets with consideration of the agreed upon joint policy guidance developed by the agencies and the White House. However, as with other parts of the NSPM-14 implementation process, the details regarding specific roles and responsibilities for directing and enforcing budget decisions lack detail and specificity. Additionally, officials from four agencies stated that the charter for the Biodefense Coordination Team has not been finalized, further delaying the articulation of roles and responsibilities and the ability to establish a shared agenda and common operating picture. As a result, some officials remain skeptical of the effectiveness of any decisions made. For example, officials from four agencies said the Biodefense Steering Committee does not have the authority to decide how individual agencies in the broader biodefense enterprise should allocate resources or prioritize programs. Officials we spoke with also provided examples of how this part of the implementation process requires attention and will from stakeholders outside the Biodefense Steering Committee, including the National Security Council staff, OMB, and the Congress. For example, officials from two agencies said turnover within the National Security Council staff had contributed to a lack of consistent leadership from the White House, which created a “lapse in momentum” and disrupted the implementation process. Additionally, officials said that key parts of the implementation process, such as the finalization of metrics, milestones and end states, as well as agreement on the federal agency roles and responsibilities for the biodefense activities articulated in the Strategy, had not been approved by the National Security Council staff. As of January 2020, these documents had not received National Security Council staff approval as the process for the development of metrics, milestones and end states is considered ongoing, which could lead to inefficiencies and delay effective implementation of the Strategy’s goals. Finally, officials we interviewed also discussed Congress’s key role as part of the regular federal budget process in determining agency appropriations. For example, officials from two agencies said it will be hard to predict whether the budget component expressed in NSPM-14 to assess and prioritize biodefense programs and activities will achieve its intended outcome. Some agency officials also believed the process to use joint policy guidance to inform annual budget submissions would not be entirely dissimilar to the annual budgetary process, as agencies will continue to submit their proposed budgets and wait for Congress to make appropriation decisions. However, we have previously reported that sustained congressional attention helps ensure that agencies continue to achieve progress resolving complex issues. We previously reported that determining the sources and types of resources needed and where those resources should be targeted are key decisions that effective national strategies should support. We also reported that effective national strategies should help clarify implementing organizations’ relationships in terms of leading, supporting, and partnering—in the context of the Strategy, that includes how enterprise- wide decisions about leveraging or directing resources to fill gaps and reduce inefficiency will be made and by whom. These could include gaps in policy, programming, or funding. Similarly, our previous work has found that articulating and agreeing to a process for making and enforcing decisions can improve the clarity surrounding a shared outcome, and that articulating these agreements in formal documents can strengthen agency commitment to working collaboratively and provide the overall framework for accountability and oversight. Moreover, a key aspect of enterprise risk management is creating a foundation that will enable participants to consider and prioritize alternatives. This prioritization can be based on a number of factors, such as the degree of risk reduction alternatives afford and the cost and difficulty to implement them. However, to do this at the enterprise level, the interagency participants need to agree on processes, roles, and responsibilities for enterprise-wide decision-making. This is particularly important in the context of enhancing efficiency and effectiveness in a broad mission space like biodefense where there is a wide array of threats and the threat landscape continually evolves. Uncertainty around the mechanisms to identify enterprise-wide priorities along with the lack of clearly documented and agreed upon processes, roles, and responsibilities for joint decision-making jeopardize the Strategy’s ability to enhance efficiency and effectiveness of the nation’s biodefense capabilities. In the absence of clearly articulated and agreed upon processes and procedures for joint decision-making to leverage or direct resources across agency boundaries in order to enhance efficiencies, agencies run the risk of continuing to work in stovepiped mission spaces and collecting information that does not serve its intended purpose. Full development and documentation of the processes, roles, and responsibilities for leveraging or directing resources across the enterprise in response to identified gaps and inefficiencies would enhance transparency and clarity for future year’s efforts and help establish a common operating picture that enables trade-offs across agency missions. The National Biodefense Strategy, released in September 2018, and the establishment of interagency governance and budgeting mechanisms to help implement the Strategy constitute a promising new approach to establishing a transformational enterprise-wide endeavor that meaningfully enhances the effectiveness and efficiency of government- wide biodefense efforts. These efforts include establishing a framework to collect and compare biodefense programs, projects, and activities across the federal government, which could facilitate enterprise-wide decision- making and budget tradeoff decisions to help ensure the most efficient use of the nation’s biodefense resources. However, these efforts represent a start to a process and a cultural shift that may take years to fully develop. During the first year of implementation, agencies have faced numerous challenges that must be overcome to ensure long-term implementation success. While agencies remain optimistic about the potential benefits of this new approach, it is imperative that additional steps be taken to ensure the challenges experienced early on are not institutionalized and that there is an intentional communication, education, and feedback effort to reinforce collaborative behaviors and enterprise-wide accountability for national security missions. A plan that includes change management practices to help bridge agency cultures and missions, such as efforts to reinforce collaborative behaviors and enterprise-wide approaches, can help ensure agencies continue to refine their interagency efforts and adapt to changes and respond effectively to challenges along the way. In addition, without clear methods and guidance that articulate how all relevant information should be analyzed, including ensuring nonfederal roles, responsibilities, and resources are accounted for in the assessment, the Biodefense Coordination Team’s ability to effectively use the information to support enterprise risk management will be limited. Moreover, without a plan to help ensure resources for sustaining ongoing institutional support, the Biodefense Coordination Team risks not having the capacity it needs to conduct meaningful analysis and decision making processes. Finally, without the development and documentation of the processes, roles, and responsibilities for joint decision making regarding the identification of priorities and for raising decisions about resource alignment across agencies, it will be difficult to sustain an enterprise-wide approach to managing risk across the biodefense enterprise. These actions could help guide agencies towards a common operating picture and shared understanding of the efforts needed beyond their individual missions. The intersection of human, animal, plant, and environmental health, as well as the nexus to the national security and economic sectors, represent challenges that no single agency can address alone. The National Biodefense Strategy was written to help link these efforts and additional planning and guidance would help enable the agencies to achieve the Strategy’s goals. We are making the following four recommendations to the Secretary of HHS: The Secretary of HHS should direct the Biodefense Coordination Team to establish a plan that includes change management practices—such as strategies for feedback, communication, and education—to reinforce collaborative behaviors and enterprise-wide approaches and to help prevent early implementation challenges from becoming institutionalized. (Recommendation 1) The Secretary of HHS should direct the Biodefense Coordination Team to clearly document guidance and methods for analyzing the data collected from the agencies, including ensuring that nonfederal resources and capabilities are accounted for in the analysis. (Recommendation 2) The Secretary of HHS should direct the Biodefense Coordination Team to establish a resource plan to staff, support, and sustain its ongoing efforts. (Recommendation 3) The Secretary of HHS should direct the Biodefense Coordination Team to clearly document agreed upon processes, roles, and responsibilities for making and enforcing enterprise-wide decisions. (Recommendation 4) We provided a draft of this report to HHS, USDA, DOD, DHS, State, VA, Justice, EPA, the National Security Council staff, and OMB for review and comment. In its written comments, which are reproduced in appendix III, HHS concurred with our four recommendations and provided additional information on the steps the agency has taken or plans to take to address our recommendations. To address recommendation 1 for the Biodefense Coordination Team to establish a plan that includes change management practices, HHS reported that it had implemented change management practices to include strategies for feedback, communication, and education. Specifically, the letter describes plans to institutionalize an after-action survey following the interagency data collection effort each year and a communications and outreach plan that was informed by multiple sources of stakeholder input. In technical comments, officials also described meetings across different components of the participating agencies that the Biodefense Coordination Team has held to help bridge organizational cultures and promote ownership. These actions, if implemented effectively, are important steps toward addressing the intent of our recommendation. At the same time, it is important to recognize the extent to which the enterprise-wide approach—making resource decisions in the context not only of each agency’s separate mission and authorities, but also to further a shared national security mission—represents a cultural shift. In technical comments, HHS officials acknowledged that opportunities exist to continue to enhance cultural aspects of the enterprise-wide approach and noted that the participation of all the agencies will be important. In addition VA, State, and EPA—in technical comments and written responses—commented on the ability of the Biodefense Steering Committee and Biodefense Coordination Team to drive enterprise-wide decision-making. They noted challenges like the limitations in these bodies’ authority to direct action and the difficulty of achieving consensus across so many actors. (See Department of Veterans Affairs’ letter reproduced in appendix IV.) HHS also concurred with recommendation 2 about clear documentation of guidance and methods for analyzing the data collected from the agencies, including ensuring that nonfederal resources and capabilities are accounted for in the analysis. However, in its written response, HHS reiterated the assessment steps it already described during our review, but it did not provide additional documentation containing more concrete and detailed methods for the analysis. HHS noted the Biodefense Coordination Team’s limited responsibilities to address nonfederal resources in the annual assessment, as described in NSPM-14. HHS also expressed in its technical comments that NSPM-14 does not charge the Biodefense Coordination Team with analyzing or accounting for nonfederal capabilities in any formal or specific way. We recognize the challenges involved with assessing nonfederal capabilities, but disagree with HHS’s characterization of the Biodefense Coordination Team’s responsibilities. According to NSPM-14, the foundation for the United States Government’s role in the biodefense enterprise is the National Biodefense Strategy and its implementation plan. The memorandum further states that agency biodefense activities shall be conducted consistent with the National Defense Authorization Act for Fiscal Year 2017 (NDAA), which provides that the strategy is to include an articulation of related whole-of-government activities required to support the strategy. We have previously reported that parts of the biodefense enterprise, such as the resources that support surveillance capabilities, are heavily reliant on nonfederal resources. Moreover, the National Biodefense Strategy states that it is broader than a federal government strategy, rather a call to action for various nonfederal entities. Therefore, to fully address our recommendation, we continue to believe that NSPM-14 notwithstanding, HHS should develop and document clear guidance for the data collection and analytical methods that will support the NDAA’s call for articulation of the capabilities that support national biodefense and recommendations for strengthening those capabilities. Regarding recommendation 3 for the Biodefense Coordination Team to establish a resource plan to staff, support, and sustain its ongoing efforts, HHS concurred, and said it requested $5 million in no-year funding in its fiscal year 2020 budget request to support the administrative management of the National Biodefense Strategy. However, as we reported, the HHS appropriations for fiscal year 2020 did not include the $5 million HHS requested and officials from multiple agencies reported that the initial planning for the staffing and responsibilities for the Biodefense Coordination Team had not been finalized. To fully address our recommendations, HHS will need to establish a resource plan that would describe how the Biodefense Coordination Team plans to staff, support, and sustain its efforts. Finally, HHS concurred with recommendation 4, for the Biodefense Coordination Team to clearly document agreed upon processes, roles, and responsibilities for making and enforcing enterprise-wide decisions. In its response, HHS points to the authority NSPM-14 gives the Biodefense Coordination Team to establish governance, policies, and procedures, subject to the approval of the Biodefense Steering Committee. HHS stated that the Biodefense Coordination Team had developed charters and guidance to govern its activities, but said that these documents were still pending the approval of the Biodefense Steering Committee. We will continue to evaluate these actions to determine the extent to which they fully address our recommendation. To fully address our recommendation, HHS in partnership with other participating federal agencies should agree upon and document clear guidance, roles, and responsibilities for addressing shared national security concerns with interagency resources and solutions that transcend the mission and capabilities of the individual agencies. Irrespective of NSPM-14, clarifying decision making processes should help the agencies identify the recommendations for improved capabilities, authorities, command structures, and interagency coordination called for by the NDAA and make incremental progress over time toward implementing those recommendations. We are sending copies of this report to the appropriate congressional committees; the Secretaries of the Departments of Health and Human Services, Agriculture, Defense, Homeland Security, State, and Veterans Affairs; the Attorney General; the Administrator of the Environmental Protection Agency; and the Director of the Office of Management and Budget. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact Chris Currie at (404) 679-1875 or CurrieC@gao.gov, and Mary Denigan- Macauley at (202) 512-7114 or DeniganMacauleyM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. The National Defense Authorization Act for Fiscal Year 2017 (NDAA) articulated eight elements to include in the required National Biodefense Strategy (Strategy). The NDAA also included a provision that we review the Strategy. As part of our analysis, we assessed the extent to which the Strategy and its associated plans incorporated the elements listed in the NDAA. On March 14, 2019, we briefed the committees of concern (as identified in the NDAA) on our findings, which we present here. To determine the extent to which the National Biodefense Strategy incorporated the elements established in the NDAA, three analysts and an attorney independently evaluated the Strategy and NSPM-14 against each NDAA element, recording scores on separate matrices. The reviewers used the following descriptors to assess the extent to which the Strategy included an element: Great Extent – explicitly cites all elements, even if specificity and detail is lacking and thus could be improved upon; Some Extent – explicitly cites some, but not all, elements; No Extent – does not explicitly cite or discuss any elements, or any implicit references are either too vague or general. The analysts and attorney then convened as a panel to reconcile any differences in scoring to reach consensus. We also interviewed officials from the agencies which comprise the Biodefense Steering Committee to gain contextual information regarding the Strategy’s development as well to help identify any challenges that agencies faced in addressing any of the statutory elements during the development process. As of the date of our briefing in March 2019, the National Biodefense Strategy and associated plans generally addressed most of the elements in the NDAA, and agencies continued to develop additional key components. Specifically, for five of the eight NDAA elements, the Strategy and associated plans addressed the major parts of the elements with few or no omissions. For the other three NDAA elements, some parts were still under development. We found in March 2019 that the National Biodefense Strategy and its associated plans generally addressed five out of eight elements listed in the NDAA, even if some of these elements lack specificity and detail. For example, where we determined the Strategy and associated plans included an element to a great extent, we recognize that these documents reflect the intent of the required element, even if improvement could be made in future revisions. Figure 5 identifies the eight elements required by the NDAA and our assessment on the extent to which those elements were included in the Strategy and associated plans. Specifically, the Strategy and related documents include a description of biological threats and the capabilities necessary to address threats, as well as recommendations for improving current biodefense capabilities, authorities, structures and interagency coordination. Description of biological threats. The NDAA provides that one element to be addressed in the strategy is a description of various biological threats. The Strategy includes a description of biological threats, as well as additional contextual information about those threats and their place within the overall threat environment. For example, the Strategy describes biological warfare, bioterrorism, naturally occurring infectious diseases, and accidental exposures as significant threats. Articulation of necessary capabilities. One element listed in the NDAA is an articulation of related or required interagency capabilities and whole- of-Government activities required to support the Strategy’s priorities. The Strategy provides a list of five goals, with associated objectives and activities that articulate the capabilities necessary to fulfill the aims of the Strategy, such as the need to improve interagency capabilities. For example, one such activity describes the need to improve state, local, tribal, territorial, private sector, federal, regional, and international surveillance systems and networks to contain, control and respond to biological incidents. Another activity involves strengthening the ability to detect zoonotic diseases and incorporating forecasting into intelligence collection by federal agencies. This articulation of necessary capabilities addresses the NDAA element to a great extent, even though we noted that additional steps to include nonfederal capabilities in the annual assessment of programs, projects, and activities would enhance implementation efforts. Recommendations for improving current biodefense capabilities. Another element listed in the NDAA is to identify recommendations for strengthening and improving current biodefense capabilities, authorities, and command structures. The Strategy contains descriptions of activities necessary to improve upon current biodefense efforts and to help agencies establish new means to fulfill the goals of the Strategy. NSPM- 14 establishes a new governance structure (command structure) to help implement the Strategy and also includes a mechanism for continual revision of the Strategy, including recommendations for strengthening biodefense activities, based on identified needs. Recommendations for interagency coordination. The NDAA also provided that the Strategy include recommendations for improving and formalizing interagency coordination and support mechanisms with respect to a strong national biodefense. The Strategy and associated plans address this element by establishing collaborative interagency structures—the Biodefense Steering Committee and the Biodefense Coordination Team—intended to work continually on improving biodefense. NSPM-14 also identifies a focal point for coordination among agencies—the Secretary of HHS. Other matters identified by agencies. The final element is to include any other matters deemed necessary by the secretaries of Defense, Health and Human Services, Homeland Security, and Agriculture. According to officials from all eight agencies, the agencies originally tasked with authoring the Strategy opened the process up to all agencies with a stake in the biodefense enterprise because they recognized those four agencies could not develop a comprehensive biodefense strategy if all partners were not included. Officials from all of the agencies on the Biodefense Steering Committee cited the inclusive nature of the drafting process as contributing to a conceptually robust Strategy. Additionally, NSPM-14 includes a requirement for the development of metrics, milestones, and end states for implementing the Strategy, and officials from all eight agencies we interviewed said the interagency group drafted them and officials from 6 of the 8 agencies said they are under review by the National Security Council staff. As of March 2019, three of 8 elements listed in the NDAA were only included to some extent because agencies implicitly addressed the element through their work, or have started addressing parts of the elements but not yet completed them. The main body of the report discusses some of the ongoing challenges related to the Strategy’s implementation. Inventory and assessment of doctrine. To some extent, the Strategy addresses the element related to an inventory and assessment of all existing strategies, plans, policies, laws, and interagency agreements related to biodefense. The agencies implicitly addressed this element by incorporating existing doctrine in the process of drafting the Strategy. For example, officials at a majority of the 8 agencies said that agencies deliberately wrote the Strategy in a way that reflects their ongoing priorities in the area of biodefense or takes into account existing agency policies or strategies. The Strategy and NSPM-14 explicitly reference some existing executive orders, presidential directives, and international treaties related to biodefense, though it excludes reference to many relevant agency-level strategies, plans, policies, laws, and interagency agreements. For example, the Strategy reinforces obligations under the Convention on the Prohibition of the Development, Production, and Stockpiling of Bacteriological and Toxin Weapons and on their Destruction (Biological Weapons Convention) (1975), but does not mention the HHS’s National Health Security Strategy, which informs a number of HHS programs that contribute to the biodefense enterprise. According to HHS officials, an inventory of doctrine was completed and submitted to Congress along with the transmittal of the Strategy, when it was released. However, not all officials we spoke to believe this work is fully completed, and officials from several agencies said they are currently evaluating their internal policies and strategies to determine how they align with the new Strategy. Catalogue of current activities. The NDAA also included an element related to a description of the current programs, projects, or activities of the United States Government with respect to biodefense. While the Strategy itself does not include a catalogue of such activities, the NSPM- 14 process requires agencies to create this catalogue, and efforts to do so are described in the body of this report. NSPM-14 requires the Chair of the Biodefense Steering Committee to send written requests for information to agencies with biodefense responsibilities, including 17 agencies mentioned in the NSPM. According to HHS officials agencies completed this collection of information in June 2019. NSPM-14 directs the Biodefense Coordination Team to use the information gathered to produce an overall assessment of federal biodefense programs and coordinate the assessment with National Security Council staff and OMB prior to its finalization and approval by the Biodefense Steering Committee. Under NSPM-14, this process will occur annually as part of the budget cycle. We characterized this element as included to some extent because efforts to complete it were underway at the time of our briefing in March 2019. Additionally, as we describe in the body of the report, we identified areas of this process to be clarified for future years’ efforts. Agency roles and responsibilities. The Strategy and associated plans did not include a description of the roles and responsibilities of the Executive Agencies, including internal and external coordination procedures, in identifying and sharing information, as described in the NDAA. The Strategy’s implementation plan includes over 240 activities, but it does not assign roles and responsibilities for performing those activities. However, NSPM-14 includes a requirement to establish these roles and responsibilities, and officials from all of the 8 agencies said agencies drafted a document assigning roles and responsibilities to each agency. This document was submitted for review to the National Security Council staff. Agency officials also discussed their engagement with nonfederal partners on the Strategy, as they play a vital role in the Strategy’s implementation. However, as we describe in the body of the report, more can be done to articulate the nonfederal role in implementing the Strategy. Additionally, NSPM-14 describes a governance structure and initial responsibilities for executive agencies, such as identification of a senior-level official as the focal point for all federal biodefense efforts. However, as described in the body of this report, additional clarity is needed on specific roles and responsibilities regarding decision-making and leadership. Therefore, we consider this element addressed to some extent. As of October 2019, the agencies took additional steps to address the elements listed in the NDAA. For example, the data collection of the programs, projects, and activities was complete, and the assessment of those data submissions was in draft form. Additionally, the agencies drafted metrics, milestones, and end states, as well as roles and responsibilities for the over 240 activities outlined in the Strategy’s Implementation Plan. However, both of these documents had not received final approval from the National Security Council staff, and the charter outlining roles and responsibilities for the Biodefense Coordination Team had not been finalized. Since 2009, we have identified broad, cross-cutting issues in leadership, coordination, and collaboration that arise from fragmentation throughout the complex interagency, intergovernmental, and intersectoral biodefense enterprise. The biodefense enterprise is the whole combination of systems at every level of government and the private sector that contribute to protecting the nation and its citizens. It is composed of a complex collection of federal, state, local, tribal, territorial, and private resources, programs, and initiatives designed for different purposes and dedicated to mitigating both natural and intentional risk. In June 2019, we testified before the Subcommittee on National Security, Committee on Oversight and Reform, House of Representatives on our past work, which has identified a number of key challenges related to the nation’s ability to detect and respond to biological incidents that transcend what any one agency can address on its own. They include: (1) enterprise-wide threat determination, (2) biodetection technologies, (3) emerging infectious disease surveillance, (4) situational awareness and data integration, and (5) biological laboratory safety and security. Agencies have taken steps to address many of the recommendations we and others have made in these areas, and we continue to monitor ongoing efforts. Enterprise-Wide Threat Determination Needed to Help Leverage Resources and Inform Resource Tradeoffs. We reported in October 2017 that opportunities remain to enhance threat awareness across the entire biodefense enterprise, leverage shared resources, and inform budgetary tradeoffs among various threats and agency programs. Key biodefense agencies, including DHS, DOD, HHS, USDA, and EPA carry out activities within their own mission spaces to better understand threats and help make decisions about biodefense investments. Additionally, federal agencies in our October 2017 review had mechanisms to support specific federal activities and individual programs, or in response to specific biological incidents after they begin to unfold. However, there was no existing mechanism that could leverage threat awareness information to direct resources and set budgetary priorities across all agencies for biodefense. Without a mechanism that is able to assess the relative risk from biological threats across all sources and domains, we found that the nation may be limited in its ability to prioritize resources, defenses, and countermeasures against the most pressing threats. In June 2019, we said implementation of the National Biodefense Strategy offers the potential for the nation to progress toward more integrated and enterprise-wide threat awareness and to use that information to identify opportunities to leverage resources, but this will take time and entails a change in the way participating agencies have traditionally operated. Challenges Determining Optimal Biodetection Technology Solutions. We have previously reported on the challenges of determining and then implementing technologies capable of identifying biological threats in the environment. Since 2012 we have reported that DHS has faced challenges in clearly justifying the need for the BioWatch program and its ability to reliably fulfill its primary task of detecting aerosolized biological attacks. According to DHS officials, DHS is in the early stages of Biodefense 21 (BD21), a multi-year acquisition effort. DHS plans to develop requirements based on collected environmental data and input from first responders, public health officials, and other partners determine what the replacement to BioWatch needs to be. As part of the early acquisition cycle for BD21, DHS is currently conducting a technology demonstration for trigger and sensor technology; therefore we cannot yet determine how it will be implemented in the future or what decisions DHS will ultimately make regarding the existing BioWatch system. Additionally, in August 2017 we reported that from a homeland security and public health perspective, threats of bioterrorism, such as anthrax attacks, and high-profile disease outbreaks, such as Ebola and emerging viruses like dengue, chikungunya, and Zika, highlight the continued need for diagnostic tests that provide early detection and warning about biological threats to humans. One option being explored is multiplex point-of-care technologies which can simultaneously test (in minutes to a few hours) for more than one type of human infectious disease pathogen from a single patient sample (such as blood, urine, or sputum) in one run at or near the site of a patient. These technologies may be used for diagnosing different diseases, including more common diseases such as influenza, emerging infectious diseases, or diseases caused by weaponized biological agents. Advances in biological detection technologies present opportunities to provide early detection and warning of catastrophic biological incidents, and in June 2019 we said the agencies responsible for implementing the National Biodefense Strategy will need to engage on this issue in a way that helps to drive informed investment tradeoff decisions about technology alternatives. We also recognized that the National Biodefense Strategy and its interagency governing leadership offer the potential for the nation to better define the role of detection technologies in a layered national biodefense capability to help those that pursue these technologies better articulate the mission needs and align requirements and concepts of operation accordingly. Challenges Building and Maintaining Emerging Infectious Disease Surveillance. We have reported that establishing and sustaining biosurveillance capabilities can be difficult for a myriad of reasons. For example, maintaining expertise in a rapidly changing field is difficult, as is the challenge of accurately recognizing the signs and symptoms of rare or emerging diseases. We reported in October 2011 that funding targeted for specific diseases does not allow for focus on a broad range of causes of morbidity and mortality, and federal officials have said that the disease- specific nature of funding is a challenge to states’ ability to invest in core biosurveillance capabilities. According to federal, state, and local officials, early detection of potentially serious disease indications nearly always occurs first at the local level, making the personnel, training, systems, and equipment that support detection at the state and local level a cornerstone of our nation’s biodefense posture. In May 2018, we reported that officials from HHS told us that their grant awards funded by annual appropriations are intended to establish and strengthen emergency preparedness and capacity building, but may not fully support the need for surge capacity that states and other jurisdictions require to respond to an infectious disease threat. Further, we reported in May 2018 that although the awards funded by supplemental appropriations have allowed state and local public health departments, laboratories, and hospitals to surge during a threat—for example, the H1N1influenza and Zika virus outbreaks—most of the 10 non-federal stakeholders we interviewed, as well as HHS officials said that the timing of these awards can result in challenges to carrying out preparedness and response activities during infectious disease threats. In June 2019, we reported that how and to what extent implementation of the National Biodefense Strategy is able to efficiently leverage and effectively sustain capacity across both nonfederal and federal stakeholders will affect how prepared the nation is to more quickly gear up for whatever challenges emerge when outbreaks of previously non- endemic diseases threaten the nation. We also noted that the Strategy and its interagency governance structure offer the opportunity to design new approaches to identifying and building a core set of surveillance and response capabilities for emerging infectious diseases. Ongoing Challenges to Fulfill Enhanced Situational Awareness and Data Integration Requirements. Our prior work has identified challenges at DHS and HHS related to the sharing, collecting, and integration of data from various federal and nonfederal agencies for their public health situational awareness and data integration efforts. We have reported that DHS’s National Biosurveillance Integration Center (NBIC), which was created to integrate data across the federal government with the aim of enhancing detection and situational awareness of biological incidents, has suffered from long-standing issues related to its clarity of purpose. Since 2009, we have reported that NBIC was not fully equipped to carry out its mission because it lacked key resources—data and personnel— from its partner agencies, which may have been at least partially the result of collaboration challenges it faced. In September 2015, we reported that despite implementing our prior recommendations and NBIC’s efforts to collaborate with interagency partners to create and issue a strategic plan that would clarify its mission and efforts, a variety of challenges remained. In October 2019, officials acknowledged that situational awareness and data integration are still very challenging problems to solve, but overall the relationships between NBIC and partner agencies are improving. Similarly, in 2017, we reported on long-standing challenges faced by HHS—such as planning and implementation shortfalls—to create a public health situational awareness network, not unlike that envisioned for DHS. In June 2019 we observed that because the National Biodefense Strategy identified biosurveillance data integration among several information sharing activities that need to be enhanced, its implementation offers the potential for the nation to better define what kind of integrated situational awareness is possible, what it will take to effectively and efficiently achieve it, and what value it has. Continued Oversight Needed to Enhance Biological Safety and Security. We—along with congressional committees—have, for many years, identified challenges and areas for improvement related to the safety, security, and oversight of high-containment laboratories. For example, in response to reported lapses in laboratory safety at HHS and DOD in 2014 and 2015, we examined how federal departments oversee their high-containment laboratories and found that most of the 8 departments and 15 agencies that we reviewed had policies that were not comprehensive or were not up to date. Additionally, we found that while the departments and agencies we reviewed primarily used inspections to oversee their high-containment laboratories, some of them were not routinely reporting inspection results, laboratory incidents, and other oversight activities to senior officials. In October 2017, we found that the Federal Select Agent Program—jointly managed by HHS and USDA—oversees laboratories’ handling of certain hazardous pathogens known as select agents and toxins, but the program does not fully meet all key elements of effective oversight. For example, the Federal Select Agent Program was not independent from all laboratories it oversees, and it had not assessed risks posed by its current structure or the effectiveness of its mechanisms to reduce organizational conflicts of interest. In June 2019, we said the National Biodefense Strategy highlights the need for continuous improvement of biosafety and biosecurity for laboratories and other facilities, creating an opportunity for interagency partners to develop additional oversight or other practices to mitigate the risk of bioincidents at high containment laboratories. Chris P. Currie at (404) 679-1875 or CurrieC@gao.gov Mary Denigan-Macauley at (202) 512-7114 or DeniganMacauleyM@gao.gov. In addition to the contacts named above, Kathryn Godfrey (Assistant Director); Nick Bartine and Susanna Kuebler (Analysts-in-Charge); Jeff Cirillo; Michele Fejfar; Eric Hauswirth; Tracey King; Jan Montgomery; Matt Ray; and Adam Vogt made key contributions to this report.", "summary": "GAO has reported on the inherent fragmented nature of the federal and nonfederal resources needed to protect the nation from potentially catastrophic biological threats. GAO called for a strategic approach to help the federal government better leverage resources and manage risk The White House issued the National Biodefense Strategy and the Presidential Memorandum on the Support for National Biodefense to promote a more efficient and coordinated biodefense enterprise. The National Defense Authorization Act for Fiscal Year 2017 included a provision that GAO review the strategy. This report addresses the extent to which the Strategy and implementation efforts are designed to enhance national biodefense capabilities and any implementation challenges that exist. GAO analyzed the Strategy, plans, and NSPM-14, and compared them to selected characteristics of GAO's work on effective national strategies, enterprise risk management, organizational transformation, and interagency coordination. GAO interviewed officials from the eight federal agencies that comprised the Biodefense Steering Committee to learn about early implementation. Issued in September 2018, the National Biodefense Strategy (Strategy) and implementation plan, along with National Security Presidential Memorandum-14 (NSPM-14), are designed to enhance national biodefense capabilities. NSPM-14 established a governance structure composed of relevant federal agencies and chaired by the Secretary of Health and Human Services (HHS) to guide implementation. It also required federal agencies with biodefense responsibilities to collect and assess data on their biodefense activities to, among other things, identify gaps. The Strategy defined the scope of the biodefense enterprise (which includes partners at all levels of government and the private sector) and brought all of the biological threats—intentional, accidental, and naturally-occurring—together, establishing an overarching vision, goals, and objectives. There are a number of challenges, however, that could limit long-term implementation success. Among other things, there was no documented methodology or guidance for how data are to be analyzed to help the enterprise identify gaps and opportunities to leverage resources, including no guidance on how nonfederal capabilities are to be accounted for in the analysis. Many of the resources that compose national capbilities are not federal, so enterprise-wide assessment efforts should account for nonfederal capabilities. Agency officials were also unsure how decisions would be made, especially if addressing gaps or opportunties to leverage resources involved redirecting resources across agency boundaries. Although HHS officials pointed to existing processes and directives for interagency decision making, GAO found there are no clear, detailed processes, roles, and responsibilities for joint decision-making, including how agencies will identify opportunities to leverage resources or who will make and enforce those decisions. As a result, questions remain about how this first-year effort to catalogue all existing activities will result in a decision-making approach that involves jointly defining and managing risk at the enterprise level. Without clearly documented methods, guidance, processes, and roles and responsibilities for enterprise-wide decision-making, the effort runs the risk of failing to move away from traditional mission stovepipes toward a strategic enterprise-wide approach that meaningfuly enhances national capabilities. GAO is making four recommendations to the Secretary of HHS, including working with other agencies to document methods for analysis and the processes, roles, and responsibilities for enterprise-wide decision making. HHS concurred with all the recommendations and described steps to implement them.", "document_type": "gao"}
{"report": "Maintenance for the nuclear elements of the fleet (i.e., aircraft carriers and submarines) is generally performed at the four public Naval shipyards, while maintenance for the conventional elements of the fleet (e.g., cruisers, destroyers, amphibious assault ships, and Military Sealift Command ships) is generally performed at private shipyards and ship repair companies throughout the United States, as shown in figure 1. A number of organizations and commands within the Navy share responsibilities for setting maintenance policies and planning, scheduling, and executing ship maintenance, from the offices of the Secretary of the Navy and Chief of Naval Operations, to fleet commanders and ships’ crews. Naval Sea Systems Command is the primary Navy ship maintenance organization. It is charged with, among other things, maintaining ships to meet fleet requirements within defined cost and schedule parameters; managing critical modernization, maintenance, and inactivation programs; life-cycle management of maintenance requirements; and management and oversight of the public naval shipyards. Its offices also perform contract administration, program management, and planning for future maintenance periods informed by the historical maintenance needs of Navy ships. Our work has found that the Navy has been generally unable to complete ship and submarine maintenance on time, resulting in reduced time for training and operations and additional costs in a resource-constrained environment. The Navy’s readiness recovery is premised on the adherence to set deployment, training, and maintenance schedules. However, we reported in May 2016 on the difficulty that both the public and private shipyards were having in completing maintenance on time. We reported that, from 2011 through 2014, about 72 percent of scheduled maintenance for surface combatants, and 89 percent of scheduled maintenance for aircraft carriers, was completed late. We updated these data as of November 2019 to include ongoing and completed maintenance periods through the end of fiscal year 2019, and found that the Navy continues to struggle to complete maintenance on time, as we discuss below. The Navy was unable to complete scheduled ship maintenance on time about 75 percent of the time during fiscal years 2014 through 2019, which equates to about 33,700 days of maintenance delays (see figure 2). Furthermore, these delays have been growing longer and more frequent. In fiscal year 2014, about 20 percent of the Navy’s maintenance periods were more than 90 days late. However, in fiscal year 2019, more than 57 percent of its maintenance periods were similarly late (see figure 3). When maintenance is not completed on time, there are two primary effects. First, fewer ships are available to conduct training or operations, which can hinder readiness. For example, in fiscal year 2019, maintenance delays resulted in the Navy losing the equivalent of 19 surface ships. Second, maintenance delays are costly. In November 2018, we examined attack submarine maintenance delays and reported that the Navy incurred significant operating and support costs to crew and maintain attack submarines that are delayed during maintenance periods. We estimated that from 2008 to 2018, the Navy spent $1.5 billion to support attack submarines that provided no operational capability—attack submarines sitting idle no longer certified to conduct normal operations—while waiting to enter the shipyards and those delayed in completing their maintenance at the shipyards. We recommended that the Navy analyze how it allocates its maintenance workload across public and private shipyards. DOD concurred with our recommendation, and in December 2018, the Navy analyzed its workload allocation and moved two additional attack submarine maintenance availabilities to the private shipyards, with the possibility of moving additional availabilities to the private sector over the next 5 years. The Navy’s ability to successfully maintain its ships—completing all required maintenance on-time and within estimated cost—is affected by numerous factors that occur throughout a ship’s lifecycle (see figure 4). Some of these factors involve decisions made during the acquisition phase, years before a ship arrives at a shipyard for maintenance, while others manifest during operational use of the ship or during the maintenance process, as illustrated in figure 4. These decisions can be interrelated; for example, decisions to increase deployment lengths to meet the Navy’s operational demands can result in declining ship conditions and material readiness. The declining condition of the ships can increase the time that ships spend undergoing maintenance at the shipyards. Increased maintenance time at shipyards can lead to decisions to make further operational schedule changes to extend deployment lengths for other ships to compensate for ships experiencing maintenance delays. While our statement today focuses on factors occurring during operations and the maintenance process, we have previously reported that long-term sustainment costs can be affected by decisions made early in the acquisition process. The decisions made during the acquisition phase of a weapon system can affect maintenance strategies used throughout the lifecycle, as 80 percent of a program’s operating and support costs are fixed at the time a program’s requirements are set and the ship is designed. For example, the littoral combat ship (LCS) program initially planned to operate the ship with 40 sailors using contractors to complete all of the onboard maintenance tasks. After challenges with the first LCS deployments, the Navy began revising the ships maintenance strategy, including adding more sailors onboard the ship. In addition, decisions to acquire or not acquire rights to technical data can have far-reaching implications for DOD’s ability to sustain the systems and competitively procure parts and services. Furthermore, the Navy has shown a willingness to provide ships to the fleet that still have a number of unresolved construction and quality deficiencies, which add to its maintenance burden. For example, the Navy delivered the USS Somerset amphibious transport dock to the fleet with 52 significant defects, including an electronic system crucial to the ship’s mission effectiveness that the fleet had to replace shortly after it received the ship. We have ongoing work on the effect that acquisition decisions can have on maintenance that we expect to issue in early 2020. Some causes of delays are created or exacerbated during an operational deployment. Our work has shown that to meet heavy operational demands over the past decade with a smaller fleet, the Navy has increased ship deployment lengths and has reduced or deferred ship maintenance. Decisions to reduce crew sizes between 2003 and 2012 also left crews overburdened and contributed to deferred maintenance. These decisions have resulted in declining ship conditions across the fleet and have increased the amount of time that ships require to complete maintenance in the shipyards. Increased maintenance periods, in turn, have compressed the time during which ships are available for training and operations. Specifically, the Navy: Decreased crew levels. We reported in 2017 that the Navy’s effort to reduce crew sizes between 2003 through 2012 corresponded with increases in maintenance costs that outweighed the savings achieved through reduced personnel costs. Navy officials told us that shifts in maintenance workload from the organizational- and intermediate- levels to depot-level maintenance increased overall maintenance costs. This change occurred in part because reduced crew sizes resulted in minor maintenance being deferred, which developed into more costly issues that had to be addressed later at the depot level. Extended deployments. We have previously reported that Navy decisions to extend deployments can lead to maintenance challenges, as these decisions have resulted in declining ship conditions across the fleet, and have increased the amount of time that ships require to complete maintenance in the shipyards. Deferred maintenance. We reported in 2015, 2016, and 2017 that maintenance deferred while a ship is deployed can develop into more costly issues that must be addressed later, often during depot-level maintenance. Deferred maintenance can lead to new work at the shipyards, as the degraded ship conditions result in the need for additional maintenance. For example, maintenance officials told us that the focus for ships homeported overseas is on mission readiness, so overseas-homeported ships place priority on the maintenance of combat systems. This means that systems with the potential to reduce ship service life—such as fuel and ballast tanks that require extended in-port periods to properly maintain—can be subject to maintenance deferrals in order to allow the ship to sustain a high operational tempo. In our prior work, we identified numerous challenges that occur during the Navy’s planning and execution of a ship’s maintenance period that contribute to delays. For example: Difficulties in adhering to the maintenance planning process. We reported in 2016 that the Navy must accurately define the work for each ship’s maintenance period. To do this, the Navy’s maintenance planning process specifies planning milestones intended to ascertain the ship’s condition, identify the work needed, and plan for its execution. Missing or meeting planning milestones late can contribute to maintenance delays. However, the Navy does not always adhere to its own maintenance planning process due to high operational tempo, scheduling difficulties, or personnel shortages, among other factors, resulting in shipyards discovering the need for additional repairs after maintenance has begun and adding time to the schedule for planning, contracting, or waiting for parts. Navy shipyards have shortages of skilled personnel. The Navy has reported a variety of workforce challenges at the four public shipyards such as hiring personnel in a timely manner and providing personnel with the training necessary to gain proficiency in critical skills. The Navy has noted that some occupations require years of training before workers become proficient. According to Navy officials, a large portion of its workforce is inexperienced. For example, we reported in December 2018 that 45 percent of the Puget Sound and 30 percent of the Portsmouth Naval Shipyards’ skilled workforce had fewer than 5 years of experience. According to DOD officials, workforce shortages and inexperience contribute to maintenance delays. For example, at Pearl Harbor Naval Shipyard in 2014 and 2015, two submarines were delayed approximately 20 months each, in part because of shortages in ship fitters and welders, among other skilled personnel. Most of DOD’s depots, which include the naval shipyards, have taken actions to maintain critical skills through retention incentives, bonuses, and awards. However, we found that neither the depots, their higher-level service component commands, nor the services have conducted an assessment to determine the effectiveness of these actions. The condition of facilities and equipment at Navy shipyards is generally poor. We reported in September 2017 that poor condition of facilities and equipment at the shipyards contributed to maintenance delays for aircraft carriers and submarines, hindering the shipyards’ ability to support the Navy. Specifically, we found that the average condition of shipyard facilities was poor and that shipyard equipment was generally past its expected service life. For example, four of the five dry docks at Norfolk Naval Shipyard face flooding threats from extreme high tides and storm swells and average one major flooding event per year. In 2009 a dry dock at Norfolk Naval Shipyard required emergency repairs to prevent flooding while the USS Tennessee (SSBN-734) was undergoing maintenance. According to the Navy’s report on the incident, several days of high tides and winds, coupled with multiple leaks in the dry dock’s granite block joints, resulted in the dry dock flooding at an estimated rate of 3,000 gallons per minute before workers could repair it. In addition, at Puget Sound Naval Shipyard—located in an area identified by the U. S. Geological Survey as a “High Seismic Hazard Zone”—a 7.0 magnitude or greater earthquake could damage or ruin the only dry dock on the west coast that is capable of performing maintenance on aircraft carriers. We have also previously reported that the Navy shipyards do not track when facility problems leads to maintenance delays. Furthermore, the average age of equipment at the shipyards is beyond its average expected service life (see table 1). Equipment that is past its expected service life can pose an increased risk for maintenance delays or higher maintenance costs, affecting the depots’ ability to conduct work. As we have previously reported, aging equipment can present a number of challenges, such as more frequent breakdowns, less effective or efficient operation, and safety hazards. The Navy shipyards lack the capacity to conduct required maintenance in the future. We also reported in 2019 that the naval shipyards cannot support 68 of the 218—almost a third—of the maintenance periods that aircraft carriers and submarines will require through 2040, due to a lack of dry dock capacity. Specifically, several of the Navy’s 17 dry docks will become obsolete after the Los Angeles-class submarines are retired because they will be too small or lack the appropriate shore-side support for newer classes of submarines. For example, only 14 dry docks can support the early- flight Virginia-class submarines and only 11 dry docks can support the Virginia-class submarines outfitted with the longer Virginia Payload Module. In addition, no dry docks can currently support repairs to the Ford class aircraft carrier, even though the Navy accepted delivery of the first ship of that class in 2017. Private shipyards have told the Navy that they could have some additional capacity to conduct maintenance, but are hesitant to invest in creating this capacity without more certainty from the Navy. The Navy has begun to implement a major effort—the Shipyard Infrastructure Optimization Plan—that is intended to significantly improve the condition of shipyard facilities and equipment, but it will require significant time and resources to implement. This plan is designed to address the bulk of the Navy’s dry-dock capacity issues as well as identify the optimal placement of facilities and major equipment at each public shipyard. The Navy estimates these changes can ultimately increase its maintenance efficiency by reducing the distance that workers and material will have to travel around the shipyards during the maintenance period. According to the Navy, this equates to recovering about 328,000 labor days per year—an amount roughly equal to that of an additional submarine maintenance period annually. In addition, the Navy has created a program office to oversee its shipyard improvement effort, which we believe demonstrates leadership attention and commitment to the effort. However, the Navy estimated that the replacement of the facilities will take 20 years (see figure 5). Further, the Navy estimates that it will take 30 years to bring the average age of its equipment to within industry standards. The Navy estimated in 2018 that this effort will require $21 billion over 20 years to implement. However, this $21 billion estimate does not include inflation and other significant costs, such as those for utilities, roads, or environmental remediation. Our analysis of the Navy’s preliminary estimate is that it is understated due to a lack of inflation adjustments, which could add billions to the final cost. Navy officials stated that the $21 billion estimate is an initial indicator of the scope of the effort and is not intended as a cost estimate in its budget. However, even that $21 billion estimate would require funding levels beyond what the Navy has requested for shipyard infrastructure in recent years. We recommended in November 2019 that the Navy should prepare more accurate cost estimates using best practices so that the Navy can request accurate funding from Congress and avoid common pitfalls associated with inaccurate estimates such as cost overruns, missed deadlines, and performance shortfalls. We recommended that the Navy take steps to improve its cost estimate prior to the start of its primary facility improvement effort; the Navy has concurred with this recommendation. The Navy has additional efforts underway that should help reduce maintenance delays, though the results of these efforts likely will not be seen for several years. For example: Revising the size of ship crews. The Navy has taken steps to address some of our recommendations regarding the size of ship crews. Specifically, the Navy has begun reviewing and revising its ship crew levels—most notably adding 32 crewmembers to its DDG- 51 destroyers and 23 crewmembers to its LPD-17 fleet. However, officials noted that the process to update crew levels throughout the fleet would take about 4 years to complete. The Navy will also need to demonstrate that it actually can assign crew members to these ships to meet the higher crew levels. We have ongoing work examining this issue and plan to report on our findings in winter of 2020. Hiring additional workers at shipyards. Shipyards have increased hiring, going from about 30,600 workers in fiscal year 2014 to about 37,400 workers in fiscal year 2019. However, Navy officials have stated that it takes several years for workers to reach full productivity. In the past, officials expected that new hires would take about 5 years to become fully productive, although the Navy has testified that they hope to reduce that time through new training techniques. Performance to Plan. The Navy has begun an analytical effort to better understand maintenance challenges and its capacity needs for the future, called “Performance to Plan.” According to Navy officials and plans, this effort is intended to help the Navy improve full and timely completion of maintenance, including for aviation, surface ships, and submarines. For example, the effort for surface ship maintenance currently involves a pilot program looking at how to better plan and execute maintenance periods for DDG 51-class destroyers, including examining how to improve the accuracy of forecasted maintenance requirements and duration and better adhere to planning milestones, among other outcomes. We are encouraged by this effort, but note that it remains in the early stages, and it is not clear whether or when the pilot effort will be extended to examine the entire surface fleet. In sum, the Navy faces significant challenges in maintaining its current fleet and reaping full benefit of the ships it has in its inventory today due to persistent and substantial maintenance delays. The Navy has made progress identifying the causes of their maintenance challenges and has begun efforts to address them. However, delays continue to persist and these challenges will require years of continued management attention and substantial investment to be resolved. As part of this sustained management attention, the Navy would benefit from a continued focus on implementing our prior recommendations. Since 2015, we have made 17 recommendations to the Navy to address various concerns we identified with its maintenance process. The Navy agreed with 14 of those recommendations, partially concurred with 1 recommendation, and disagreed with 2 recommendations. However, as of November 2019, the Navy had fully implemented 6 of these recommendations. While the Navy has taken some additional action on the 11 remaining unimplemented recommendations, taking additional steps to fully address these recommendations could help the Navy address its maintenance challenges and better position it to sustain the current and future fleet. Looking to the future, the Navy is seeking to grow the fleet over the next 15 years. However, if it increases the size of the fleet before addressing its maintenance challenges, it is likely that the Navy will be faced with a growing number of both maintenance delays and ships that are unavailable for use. Even assuming the Navy’s efforts to improve shipyard operations succeed, it will be years before the Navy can maintain a significantly larger fleet. Chairmen Perdue and Sullivan, Ranking Members Hirono and Kaine, and Members of the Subcommittees, this concludes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff have questions about this testimony, please contact Diana Maurer, Director, Defense Capabilities and Management at (202) 512-9627 or maurerd@gao.gov. Contacts points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Suzanne Wren (Assistant Director), James Lackey (Analyst-in-Charge), A. S. Bagley, Chris Cronin, Amie Lesser, Felicia Lopez, Tobin McMurdie, Carol Petersen, Clarice Ransom, Matt Thompson, and Sally Williamson. In recent years, we have issued a number of reports related to ship and submarine maintenance. Table 1 summarizes the recommendations in these reports. The Department of Defense (DOD) concurred with most of the 17 recommendations; however, to date DOD has fully implemented 6 of the recommendations. For each of the reports, the specific recommendations and any progress made in implementing them are summarized in tables 2 through 9. Report numbers with a C or RC suffix are classified. Report numbers with a SU suffix are sensitive but unclassified. Classified and sensitive but unclassified reports are available to personnel with the proper clearances and need to know, upon request. Report numbers with a T suffix are testimonies. Naval Shipyards: Key Actions Remain to Improve Infrastructure to Better Support Navy Operations. GAO-20-64. Washington, D.C.: November 25, 2019. Military Depots: Actions Needed to Improve Poor Conditions of Facilities and Equipment that Affect Maintenance Timeliness and Efficiency. GAO-19-242. Washington, D.C.: April 29, 2019. DOD Depot Workforce: Services Need to Assess the Effectiveness of Their Initiatives to Maintain Critical Skills. GAO-19-51. Washington, D.C.: December 14, 2018. Navy and Marine Corps: Rebuilding Ship, Submarine, and Aviation Readiness Wil Require Time and Sustained Management Attention. GAO-19-225T. Washington, D.C.: December 12, 2018. Navy Readiness: Actions Needed to Address Costly Maintenance Delays Facing the Attack Submarine Fleet. GAO-19-229. Washington, D.C.: November 19, 2018. Navy Readiness: Actions Needed to Address Costly Maintenance Delays Affecting the Attack Submarine Fleet. GAO-19-192C. Washington, D.C.: October 31, 2018 Military Readiness: Update on DOD’s Progress in Developing a Readiness Rebuilding Plan. GAO-18-441RC. Washington, D.C.: August 10, 2018. (SECRET) Navy Shipbuilding: Past Performance Provides Valuable Lessons for Future Investments. GAO-18-238SP. Washington, D.C.: June 6, 2018. Weapon Systems Annual Assessment: Knowledge Gaps Pose Risks to Sustaining Recent Positive Trends. GAO-18-360SP. Washington, D.C.: April 25, 2018. Columbia Class Submarine: Immature Technologies Present Risks to Achieving Cost Schedule and Performance Goals. GAO-18-158. Washington, D.C.: December 21, 2017. Navy Readiness: Actions Needed to Address Persistent Maintenance, Training, and Other Challenges Affecting the Fleet. GAO-17-809T. Washington, D.C.: September 19, 2017. Naval Shipyards: Actions Needed to Improve Poor Conditions that Affect Operations. GAO-17-548. Washington, D.C.: September 12, 2017. Navy Readiness: Actions Needed to Address Persistent Maintenance, Training, and Other Challenges Facing the Fleet. GAO-17-798T. Washington, D.C.: September 7, 2017. Navy Shipbuilding: Policy Changes Needed to Improve the Post-Delivery Process and Ship Quality. GAO-17-418. Washington, D.C.: July 13, 2017. Department of Defense: Actions Needed to Address Five Key Mission Challenges. GAO-17-369. Washington, D.C.: June 13, 2017. Navy Force Structure: Actions Needed to Ensure Proper Size and Composition of Ship Crews. GAO-17-413. Washington, D.C.: May 18, 2017. Military Readiness: DOD’s Readiness Rebuilding Efforts May Be at Risk without a Comprehensive Plan. GAO-16-841. Washington, D.C.: September 7, 2016. Navy and Marine Corps: Services Face Challenges to Rebuilding Readiness. GAO-16-481RC. Washington, D.C.: May 25, 2016. (SECRET//NOFORN) Military Readiness: Progress and Challenges in Implementing the Navy’s Optimized Fleet Response Plan. GAO-16-466R. Washington, D.C.: May 2, 2016. Navy Force Structure: Sustainable Plan and Comprehensive Assessment Needed to Mitigate Long-Term Risks to Ships Assigned to Overseas Homeports. GAO-15-329. Washington, D.C.: May 29, 2015. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The 2018 National Defense Strategy emphasizes that restoring and retaining readiness is critical to success in the emerging security environment. The Navy is working to rebuild its readiness while also growing and modernizing its aging fleet of ships. A critical component of rebuilding Navy readiness is implementing sustainable operational schedules, which hinge on completing maintenance on time. We have reported that the Navy faces persistent challenges with completing required maintenance on time. This statement provides information on (1) the magnitude of maintenance delays for Navy ships and submarines, (2) factors contributing to maintenance delays, and (3) the Navy's efforts to address these factors. GAO also discusses its prior recommendations on the factors contributing to Navy maintenance delays and the Navy's progress in addressing the recommendations. This statement is based on previously published work from 2015 through 2019 on Navy maintenance, ship acquisition, crew size, ship maintenance and deployment schedules, the condition of Naval shipyards, and recruiting skilled maintenance personnel. The Navy continues to face persistent and substantial maintenance delays that affect the majority of its maintenance efforts and hinder its attempts to restore readiness. From fiscal year 2014 to the end of fiscal year 2019, Navy ships have spent over 33,700 more days in maintenance than expected. The Navy was unable to complete scheduled ship maintenance on time for about 75 percent of the maintenance periods conducted during fiscal years 2014 through 2019, with more than half of the delays in fiscal year 2019 exceeding 90 days. When maintenance is not completed on time, fewer ships are available for training or operations, which can hinder readiness. GAO identified multiple factors that contribute to maintenance delays, including insufficient shipyard capacity, shortage of skilled personnel, and deferred maintenance during operational deployments, among others. Ships awaiting or delayed in maintenance incur operating and support costs. For example, GAO estimated that the Navy spent more than $1.5 billion in support costs from fiscal years 2008 through 2018 due to delayed maintenance for attack submarines. The Navy has several efforts underway to improve its maintenance operations, but they will take years to implement, and will require sustained management attention and funding above current levels. For example, the Navy estimates it will take 20 years to improve the infrastructure at its shipyards, 4 years to restore ship crew levels, and several years to improve maintenance planning. Until the Navy addresses these challenges, it will be hindered in its ability to rebuild readiness and prepare for the future, particularly as it grows the size of the fleet. GAO made 17 recommendations in prior work cited in this statement. The Department of Defense generally concurred with most of GAO's recommendations, and has fully implemented 6. Continued attention is needed to ensure that the remainder of these recommendations are addressed.", "document_type": "gao"}
{"report": "Drawback refunds are a remittance of up to 99 percent of duties, taxes, or fees previously paid by an importer. CBP makes these refunds on imported goods on which the importer previously paid duties, taxes, or fees, and subsequently exported from the United States or destroyed. (See fig. 1.) According to CBP, the rationale for the drawback program was to encourage American commerce and manufacturing. It permits American manufacturers to compete in foreign markets without the handicap of including in their costs, and consequently in their sales price, the duty they paid on imported merchandise. Claimants can apply for and obtain the privilege of accelerated payment of drawback refunds. Accelerated payment allows estimated drawback refunds to be paid prior to liquidation of the drawback entry, provided that, among other things, claimants have acquired and posted with CBP a bond in an amount sufficient to cover the estimated amount of drawback to be claimed. There are three main categories of drawback refunds: (1) manufacturing drawback (direct identification and substitution), (2) unused merchandise drawback (direct identification and substitution), and (3) rejected merchandise drawback. Within each category, there are variations in drawback eligibility, such as the ability to substitute imported merchandise. a. Direct identification manufacturing drawback may be claimed on exported or destroyed articles that have been manufactured or produced in the United States with imported duty-paid merchandise, if those articles have not been used in the United States prior to export or destruction under CBP supervision. For example, a claimant could claim a drawback refund on exported pants made in the United States using imported foreign fabric. (See fig. 2.) b. Substitution manufacturing drawback may be claimed on exported or destroyed articles that have been manufactured or produced in the United States using domestic merchandise substituted for imported duty-paid merchandise meeting the statutory criteria, where the articles have not been used in the United States. As a result, domestic producers can select the most advantageous sources for their raw materials and components without regard to duties, saving them production costs. For example, a claimant could claim a drawback refund on exported pants made in the United States using domestic fabric substituted for imported foreign fabric. (See fig. 3.) a. Direct identification unused merchandise drawback may be claimed on imported merchandise that was exported or destroyed under CBP supervision, without having been used within the United States. For example, a claimant could claim a drawback refund on unused imported designer dresses upon their destruction. (See fig. 4.) b. Substitution unused merchandise drawback may be claimed on goods that were exported or destroyed under CBP supervision, without being used, and were substituted for imported merchandise meeting the appropriate criteria. For example, a claimant could claim a drawback refund on exported cars substituted for imported foreign-made cars. (See fig. 5.) 3. Rejected merchandise drawback may be claimed upon the exportation or destruction under CBP supervision of imported duty- paid merchandise entered or withdrawn for consumption, provided it meets the statutory criteria (i.e., not conforming to sample or specifications, shipped without consent, determined to be defective at the time of import, or ultimately sold at retail and returned). For example, a claimant could claim a drawback refund on foreign fabric it imported but returned to the seller because the fabric did not conform to the specification of the claimant’s order. (See fig. 6.) During calendar years 2009 through August 21, 2019, the total amount of drawback refunds claimed ranged from $631.6 million to $1.4 billion. The amount of drawback refunds claimed varied from year to year, but generally rose between 2011 and 2016. Overall, in dollar terms, substitution unused merchandise drawback remained the largest category of drawback refund, as shown in table 1. As originally enacted in 1789, the drawback program was limited to duties paid on certain imported merchandise if the merchandise was exported within a year. In the 1930s, drawback claimants could use substituted merchandise for imported merchandise in specified circumstances. Congress has continued to allow substitution for drawback refunds in various forms. (See fig. 7.) The U.S. International Trade Commission publishes and maintains the HTS. The HTS is used to determine tariff classifications for goods imported into the United States. Each item imported into the United States is classified in a category with an assigned 8-digit HTS subheading number. The category may be subdivided into 10-digit HTS subheading numbers for statistical purposes. The 4-digit and 6-digit nomenclature is consistent internationally. CBP is responsible for fixing the final classification. For unused merchandise substitution drawback claims, TFTEA also allows drawback using the U.S. Department of Commerce Schedule B commodity number. Pub. L. No. 114-125, § 906(e)(4). We do not discuss the use of the Schedule B commodity number in this report because, according to CBP, it is very rare that the Schedule B commodity number is not identical to the HTS number. stating that importers are now jointly and severally liable with claimants for refunds associated with their imported goods. Treasury and CBP had 2 years from the date of enactment of TFTEA to promulgate regulations implementing the TFTEA drawback provisions. TFTEA also provided for an additional 1-year transition period (February 24, 2018–February 23, 2019) during which drawback claimants could file under either the amended provisions or the drawback law as it previously existed. When the government did not meet the 2-year deadline for issuing regulations, which lapsed on February 24, 2018, a number of companies filed suit. Subsequently, Treasury and CBP published the Modernized Drawback Notice of Proposed Rulemaking in the Federal Register on August 2, 2018 and separately published the Regulatory Impact Analysis of the Modernized Drawback Notice of Proposed Rulemaking. In an October 12, 2018 order, the Court of International Trade ordered the United States to file the final rule developed pursuant to the Modernized Drawback Notice of Proposed Rulemaking with the Office of the Federal Register by December 17, 2018. The government met that deadline, publishing the Modernized Drawback Final Rule in the Federal Register and the Regulatory Impact Analysis of the Modernized Drawback Final Rule (RIA). In the final rule, CBP summarized and responded to public comments received on the Modernized Drawback Notice of Proposed Rulemaking and established new policies and procedures for the drawback program pursuant to TFTEA. In the RIA, CBP provided its predictions of the impact—primarily in terms of costs, benefits, and revenue transfers—of key changes to the drawback program on industry and the U.S. government. CBP did not make accelerated payments on or liquidate any TFTEA drawback claims until the final rule was issued. CBP also did not make any drawback payments during the partial federal government shutdown (December 22, 2018– January 25, 2019). Under drawback modernization, CBP transitioned its filing process for making claims for payment under the drawback program from its Automated Commercial System (ACS) to its Automated Commercial Environment (ACE). Previously, CBP required claimants to file a paper claim, and electronic transmission of a claim summary through ACS was optional. TFTEA required claimants to file all claims electronically on and after February 24, 2018, but also allowed for a 1-year transition period where claims could be filed under the existing drawback statute or under the statute as amended by TFTEA. CBP designated ACE as the electronic system for filing drawback claims. CBP initially partially deployed ACE for the drawback program on February 24, 2018, to allow electronic filing of claims. During the transition period, claimants could file claims under the existing drawback process (detailed in 19 C.F.R. part 191) or under the new drawback process (detailed in 19 C.F.R. part 190). CBP fully deployed ACE for the drawback program on February 24, 2019, the first day after the transition period when all drawback claims had to be filed under the amended statute and implementing regulations. After CBP mandated electronic filing in ACE, drawback entry summary data had to be filed at the more detailed line item level. ACE has expanded capabilities, such as accounting for line item reporting for drawback claims and automatically validating drawback claims against underlying import entries. Changes in the broader trade policy context may also impact CBP’s drawback program. In particular, 2018 witnessed a series of presidential and agency actions that resulted in higher tariffs on a range of goods. For example, in January 2018, the President issued Presidential Proclamation 9693 and Presidential Proclamation 9694, imposing tariff rate quotas and increased duties on imports of solar cells and panels, and washing machines and parts, effective February 7, 2018. Further, at the direction of the President, the United States Trade Representative has imposed additional duties on products of China in four tranches, in June 2018, August 2018, September 2018, and August 2019. According to the United States Trade Representative request for comments on a modification to the fourth tranche, the four tranches cover an annual aggregate trade value of approximately $550 billion. CBP has determined that the aforementioned tariffs (commonly referred to as section 201 and 301 duties, respectively) are eligible for drawback refunds and issued guidance on how to make such claims. For fiscal year 2019, Treasury reported that it collected $70.8 billion in customs duties, as compared to $41.3 billion in fiscal year 2018. Within the Department of Homeland Security, CBP’s Office of Trade is primarily responsible for managing the drawback program. CBP officials described the roles and responsibilities of the several offices within CBP that are involved, as follows: Trade Policy and Programs. The Office of Trade Policy and Programs provides policy and program oversight for the drawback program. Field Operations. The Office of Field Operations is responsible for implementing the drawback program, including ensuring that the Drawback Centers have the resources—allocations, staffing, equipment—to perform their duties and meet CBP’s trade mission. Drawback Centers. Drawback specialists located in one of the four Drawback Centers in Chicago, Houston, Newark, or San Francisco are responsible for reviewing and processing drawback claims. They review claims, in whole or in part, to determine eligibility for drawback refunds. (Appendix II describes CBP’s steps for filing and processing drawback claims.) They also review and make determinations concerning claimants’ (1) requests for drawback privileges for accelerated payment and waiver of prior notice, (2) applications for certain manufacturing rulings, and (3) protests of denied claims. Regulations and Rulings. The Office of Regulations and Rulings is responsible for issuing various types of binding rulings and decisions on drawback refunds. These include decisions on protest applications flagged for further review by the Drawback Centers as well as prospective ruling requests filed by drawback applicants, such as rulings on specific manufacturing drawback rulings and on the proper classification of merchandise for substitution manufacturing drawback. In addition to issuing binding rulings, the Office of Regulations and Rulings is responsible for drafting any regulatory changes involving the drawback program and provides technical advice for drawback policy and litigation. TFTEA generally expanded eligibility for drawback refunds, with some caveats, but CBP is not adequately managing its growing workload of claims resulting from the changes. The substitution standard for drawback claims under TFTEA generally allowed more merchandise to potentially qualify for drawback refunds. However, it also limited the eligibility of certain broadly categorized merchandise. TFTEA also expanded the scope of the refund of taxes and fees for manufacturing claims and standardized time limits to file claims. On balance, these changes, along with certain limitations in CBP’s Automated Commercial Environment (ACE), have led to an increase in the workload of drawback specialists. However, CBP did not anticipate the increased workload and does not have a plan to manage the increased workload, which has caused delays resulting in uncertainty for industry—potentially impeding trade. Change of substitution standard: According to CBP officials, the most significant change resulting from TFTEA is that it is now easier to substitute merchandise and still qualify for drawback refunds. TFTEA changed the substitution standard for certain drawback types, with new rules reflecting a shift from a subjective to a more objective standard. Previously, CBP applied a subjective ‘‘same kind and quality’’ standard for manufacturing substitution drawback and ‘‘commercially interchangeable’’ standard for unused merchandise drawback. For example, CBP did not permit a U.S.-based clothing manufacturer, Jockey, to substitute light blue underwear for dark blue underwear for an unused merchandise drawback claim before modernization. In 1995, Jockey submitted a request to CBP for a “commercially interchangeable” ruling to permit it to substitute underwear that is the same size, style, and specification, but different in color and part number—for example, substitute light blue underwear for dark blue underwear. CBP ruled that Jockey underwear was not “commercially interchangeable” for the purpose of the unused merchandise substitution drawback. Under the new substitution standard for manufacturing drawback and unused merchandise drawback, both the imported merchandise and the substituted merchandise generally must match at the 8-digit or 10-digit HTS classification to be eligible for drawback refunds. The new substitution standard has made more merchandise eligible for drawback refunds, such as the Jockey underwear that would now be eligible for unused merchandise substitution drawback, as shown in the example for one type of product in figure 8 below. It has also enabled automatic acceptance and verification of drawback claims in ACE. CBP officials told us that they had seen an increase in new claimants as a result of the changes to the substitution standard, among other factors. According to industry representatives we interviewed, the changes to the substitution standard have enabled new companies to file for drawback refunds and have expanded eligibility for existing clients. For example, they stated that the changes to the substitution standard have allowed the automotive industry to substitute domestic car exports for imported foreign-made cars, as mentioned earlier. One industry representative noted that as a result of the new substitution standard, an automotive company that had been recovering about $2 million in drawback refunds per year before TFTEA can now recover about $20 million a year. Drawback trading: The new substitution standard may also broaden the scope for “drawback trading,” according to industry representatives we interviewed. They described “drawback trading” as matching excess import and export activity through the use of a third-party special purpose entity that exists for the sole purpose of maximizing drawback refund recovery between currently unrelated importers and exporters with no existing commercial relationship. CBP officials we spoke to did not think the new substitution standard should have any bearing on the potential for “drawback trading.” CBP officials explained that although the substitution standard for certain drawback claims had changed, TFTEA should not significantly affect “drawback trading” because, as before TFTEA, the claimant would still need to fulfill the possession and assignment standards. Finished petroleum derivative drawback claims do not have a possession requirement. CBP has permitted drawback where a company set up relationships with the importer and exporter expressly to maximize drawback for finished petroleum derivatives. Limitation of basket provisions from unused merchandise substitution drawback: While TFTEA’s change to the use of HTS classifications generally expanded eligibility for certain drawback substitution claims, it concurrently limited eligibility in certain situations. Specifically, TFTEA prohibited eligibility for unused merchandise substitution of merchandise that is classified as “other” at both the 8-digit and 10-digit HTS subheadings for drawback refunds. Such classifications are considered basket provisions. For example, shrimps and prawns that fall under the HTS 1605.21.10.30 basket provision, as shown in figure 9, are not eligible for substitution unused merchandise drawback, as follows. If the shrimps and prawns are not in airtight containers, and are not products containing fish meat or prepared meals, they fall under “other” at the 8-digit HTS subheading (1605.21.10). If these shrimps and prawns are also frozen but not breaded, they fall under “other” at the 10-digit HTS statistical suffix (1605.21.10.30), categorizing them in a basket provision. According to CBP, the products most affected by the limitation on basket provisions from unused merchandise substitution drawback based solely on 2016 HTS counts will be screws, nuts, and bolts; motor vehicle parts and accessories; and transmission shafts. One company we spoke with had been able to claim over $1 million in unused merchandise substitution drawback a year prior to modernization, for an imported ceramic substrate used for cleaning emissions in cars. The company also makes domestically sourced ceramic substrate, which it exports. CBP considered these two products commercially interchangeable. However, according to the company, the ceramic substrate is classified as a basket provision and the company is no longer eligible for drawback refunds. from the same inventory. If the imported item is substituted for an exported item that is not fungible with the imported item, it does not qualify for direct identification. able to administer the new statistical reporting number. For example, the article description must be clear, the HTS classification must be correct, and the new number must not require difficult or prohibitively expensive laboratory or other testing. If merchandise is not eligible for direct identification drawback but is classified as a basket provision, it may still qualify for a drawback refund if a company can successfully petition the Committee for Statistical Annotation of Tariff Schedules for new 10-digit HTS statistical breakouts. However, such a workaround is time-intensive and not guaranteed, according to an industry representative. In one example, the representative explained that a chemical company with a product classified as a basket provision successfully petitioned for a new statistical breakout. The company produces chemical methanol and was unable to file a TFTEA drawback claim in 2018 because of the basket provision restriction. Such requests are generally considered by the committee twice a year. Standardizing time limits: TFTEA also expanded eligibility for drawback refunds by standardizing the drawback filing deadline. Previously, drawback claim filing deadlines varied based on type of claim and time between import and export or destruction, ranging from 3 years to 5 years from importation to exportation or destruction, followed by a 3-year window to file a claim. TFTEA generally standardized the timelines for the acceptance of claims to be up to 5 years from import. CBP expects the new eligibility time frames will give some drawback claimants more time to file for drawback and potentially increase drawback eligibility for some claimants. Expanding taxes and fees: TFTEA expanded the scope of drawback refunds by explicitly including taxes and fees for manufacturing drawback claims. Prior to TFTEA, the drawback statute did not specify that taxes and fees were eligible for manufacturing drawback. TFTEA extended drawback refunds to taxes and fees for manufacturing claims. Some industry representatives we spoke to told us they were benefiting from this expansion. For example, a representative from the U.S. oil industry noted that the new law is “much more lucrative” for oil companies that refine crude oil because they can now get drawback refunds on the oil spill tax and harbor maintenance fee. CBP has not adequately managed the growing workload drawback specialists have been experiencing since TFTEA. Drawback specialists told us that they had been experiencing increasing workloads since CBP implemented the changes from TFTEA. The largest Drawback Centers expect their backlog of old claims will take about 5 years to work through. This workload is the cumulative result of various factors that have caused delays with processing claims, rulings, and privileges applications. The workload of the Drawback Centers is growing because of a learning curve related to the switch from a paper-based to an electronic process, delays in processing claims, and an increase in the number of claims, as discussed below. Further, the Drawback Centers continue to face staffing shortages. Learning curve: According to CBP officials, drawback specialists face a learning curve as they become familiar with ACE and the new rules for drawback refunds. They explained that drawback specialists are still working through pre-TFTEA claims that were migrated into ACE. From January 1, 2019 to September 13, 2019, CBP Drawback Centers liquidated about 18 percent of the value of the remaining claims filed in CBP’s Automated Commercial System (ACS) and about 27 percent of the number of remaining claims filed in ACS. For TFTEA claims, CBP provided in-person training to drawback specialists before the final regulations were issued, as well as in May 2019 and September 2019. CBP has also been updating its guidance for processing claims, and, according to officials, plans to continue to offer trainings for drawback specialists as it finalizes the guidance. Nevertheless, adjusting to the changes has hampered the efficiency of drawback specialists. For example, drawback specialists explained that they had to learn to toggle between different systems that require separate logins to review event history, file uploads, and tax information within ACE in order to fully process a claim. Delays in processing claims: CBP faced a delay in processing drawback claims because of a hold relating to the issuance of the drawback final rule. Claimants could begin filing TFTEA claims on February 24, 2018, but CBP did not process any of these claims pending the final rule— which CBP issued on December 17, 2018. As a result, all 18,319 claims filed during this 10-month period were put on hold. CBP lifted the hold when the final rule was issued. CBP’s workload continued to grow because certain TFTEA manufacturing claims were on hold. Following TFTEA, the proposed and final rule required claimants who wanted to operate under an existing manufacturing ruling to file a supplemental application for a limited modification to the existing ruling, as previously discussed. According to CBP interim guidance, to ensure compliance with TFTEA drawback requirements, a limited modification must include a bill of materials or formula, annotated with the applicable HTS subheading numbers. Claimants who did not apply for a limited modification by February 23, 2019, would need to apply for a new manufacturing ruling. CBP received about 800 applications for limited modifications, which it began approving on September 16, 2019. Between February 2019 and July 2019, CBP also received about 50 applications for new manufacturing rulings, which it has not yet begun to process. These processes remain paper-based (see fig. 10). CBP officials explained that CBP generally does not process manufacturing drawback claims until claimants are issued up-to-date ruling numbers. Until the new or modified manufacturing rulings are approved, CBP officials explained, they will not provide accelerated payment or process manufacturing claims. Moreover, some manufacturing rulings can take years to finalize. For example, one chemical company noted that CBP’s lab analyzes every piece of the manufacturing process, and as a result, it is awaiting final decisions on new manufacturing rulings from 2013. Increase in number of claims: CBP has also seen an increase in the number of drawback claims because of TFTEA’s changes to the drawback program and limitations in ACE. Prior to TFTEA, the number of drawback claims per calendar year ranged from 11,690 to 13,291. CBP saw a large increase in the number of drawback claims in 2018 and 2019. (See table 2.) CBP limited the number of lines in a drawback claim in ACE, which increased the number of drawback claims filed. Prior to TFTEA, claims were not limited by line. Because of system constraints, claims filed in ACE are restricted to 10,000 lines per claim. CBP had predicted that this ACE line limitation would increase the number of claims by a factor of four. Evidence to date indicates a significant increase in workload for certain Drawback Centers. For example, the Chicago Drawback Center noted that two claimants had filed over 4,000 claims between February 24, 2018 and February 23, 2019, whereas these same claimants had filed less than 50 claims in the prior year. According to the industry representatives we spoke with, the line limit in ACE added more work for industry and CBP because it made it necessary for claimants to break up the volume of their claims into different applications. For example, one broker used to file drawback claims four times a year on behalf of one refinery, but now has to file 300 times per year to account for the line limit. Drawback specialists pointed out that each claim stands on its own. As a result, they explained that they must liquidate each claim in ACE, which involves a number of quality control steps such as verifying that the claim is ready to be liquidated, relevant rulings are valid, and all validation activities are complete. As discussed earlier, the changes to the substitution standard have also led to an increase in new drawback claimants, according to CBP officials. CBP has received applications from over 500 new claimants since February 24, 2018. New claimants require additional work, including drawback specialists’ manual reviews of claims, privilege applications, and ruling requests, as follows. Claims. Drawback specialists explained that drawback claims from new claimants are subject to a full desk review. The specialists will request supporting documentation to ensure that the appropriate statutory and regulatory requirements are met. They also determine drawback due on the basis of the completed drawback claim, the applicable general manufacturing drawback ruling or specific manufacturing drawback ruling, and any other relevant evidence or information. According to CBP, the time it takes a drawback specialist to conduct a full desk review varies by claim, based on the nature of the claim and the experience of the drawback specialist. CBP reported that it could take more than 3 years for CBP to conduct a full desk review and determine the final disposition of a drawback claim. Privilege applications. Claimants can also apply for privileges including accelerated payment privileges, a waiver of prior notice of intent to export or destroy, or a one-time waiver of prior notice of intent to export or destroy. Claimants must continue to submit paper applications for such privileges and drawback specialists must manually review the privilege applications. According to CBP, most claims are eligible for accelerated payment of drawback refunds. Manufacturing rulings. Lastly, if a claimant is seeking either a direct identification manufacturing drawback or a substitution manufacturing drawback, it must manually apply for a manufacturing ruling using a paper form submitted through email, which may require significant documentation and review, as discussed earlier. CBP maintains the manufacturing rulings as paper files. For example, the Drawback Center in Newark stores manufacturing rulings in rows of filing cabinets. (See fig. 11.) Additionally, CBP has not been able to respond to all privilege applications within 90 days, as set forth in the regulations. Between February 2018 and July 2019, CBP received almost 600 new privilege applications. CBP missed the 90-day deadline about 60 percent of the time. According to drawback specialists, they missed this deadline because of their workload. According to an industry representative, delays in processing privilege applications mean companies cannot receive their drawback money in a timely manner. Such delays cause uncertainty for industry, potentially impeding trade. Drawback specialists face new obstacles to managing automatic liquidation of drawback claims in ACE. According to CBP officials, previously, drawback specialists had at least 10 days of lead time to address an automatic liquidation. Now, Drawback Centers must continually monitor the automatic liquidation reports. Because of the way ACE operates, drawback specialists may only have 1 day of lead time before a claim automatically liquidates. According to drawback supervisors, such monitoring is significantly increasing their workload. Further, drawback specialists told us that one way they were managing their increased workload was by extending automatic liquidation, which can be done up to three times, as discussed in appendix II. This practice goes directly against CBP’s guidance. Moreover, if they continue this practice, specialists may be forced to liquidate claims at zero if they run out of extensions. Further, as the workload continues to grow, Drawback Centers continue to face staffing shortages. As of October 26, 2019, CBP met the congressionally mandated staffing level for drawback specialists of 37 for the first time in over 5 years. In CBP’s 2017 Resource Optimization Model, it reported an optimal staffing level of 40 to meet its drawback staffing needs. CBP’s staffing level of 37, as of October 2019, did not meet this target. According to CBP officials, although Drawback Centers are utilizing overtime, the drawback specialists are not able to keep up with the influx of work. CBP has not adequately managed its drawback workload because it did not anticipate the increase in workload and did not plan for the increase accordingly. Federal standards for internal control note that management should evaluate performance and hold individuals accountable for their internal control responsibilities, which include evaluating pressure on personnel to help personnel fulfill their assigned responsibilities in accordance with the entity’s standards of conduct. Management can adjust excessive pressures using many different tools, such as rebalancing workloads or increasing resource levels. However, CBP has not brought staffing to its optimal level, and has not adjusted the workload in Drawback Centers through ACE to account for the increase in claims, rulings, and privilege applications. Prior to TFTEA, CBP officials explained that CBP could not control the workload of the Drawback Centers because claimants mailed their paper-based claims to the Drawback Center of their choice. Now, CBP has greater visibility and flexibility to potentially control the work flow to the Drawback Centers through ACE, but has not done so. CBP officials said they had anticipated that ACE automation would reduce drawback specialists’ workload, but experience, to date, indicates that workload increased. Until CBP develops a plan for managing its increased workload, it risks further delays in drawback claim processing that result in uncertainty for industry, potentially impeding trade—which runs counter to its strategic goal of enhancing U.S. competitiveness by enabling lawful trade and travel, such as by reducing barriers to the efficient flow of trade and streamlining and unifying processes and procedures. CBP has taken steps to mitigate improper payment risks in the drawback program. Specifically, CBP expects three key changes to the drawback process under modernization will strengthen its ability to validate claims and recover inaccurately claimed drawback refunds: (1) requiring electronic filing in ACE, (2) extending the record retention period, and (3) broadening liability. However, CBP has not addressed several other risks for improper payments in the drawback program. These risks relate to (1) limitations in CBP’s existing desk review process, (2) establishing electronic proof of export, and (3) targeting a selection of claims for review. TFTEA contained provisions amending the drawback statute that CBP expects will help it to remediate certain internal control deficiencies over drawback claim processing. Prior independent audits identified significant or material internal control weaknesses related to CBP’s processing of drawback claims, including that CBP’s drawback system lacked effective automated controls to prevent the overpayment of drawback claims and that the record retention period was not appropriate to ensure that support for drawback claims was maintained for the length of the drawback claim lifecycle. CBP expects that three key changes to the drawback process under modernization will strengthen its ability to validate claims and recover inaccurately claimed drawback refunds, as follows. Requiring the electronic filing of drawback claims. On February 24, 2019, the drawback program fully transitioned to ACE. Specifically, all drawback claims are now required to be filed electronically in ACE and include drawback entry summary data at the more detailed line item level. Line item reporting requires claimants to provide certain relevant information for the designated imported merchandise on a drawback claim associated with the line item on an import entry summary, including the tariff classification, quantity, and value, as well as the duties, taxes, and fees assessed thereon. With electronic filing and line item reporting, CBP can now automatically compare and verify the amounts of duties, taxes, and fees claimed on the drawback claim against the amounts paid on the import entry summary, which CBP expects will help ensure that it does not overpay funds. CBP’s prior system for filing drawback claims did not have the capability to electronically compare and verify claims against underlying import entries upon which the drawback claim was based to determine whether an excessive amount had been claimed at the individual line item level, according to prior independent audits. CBP’s transition to ACE is intended to mitigate risks of improper payments on drawback-related imports, by helping to ensure through automated validations that the amount paid for drawback claims against a given import entry does not exceed 99 percent of the duties, taxes, and fees collected at the individual line item level. Extending the record retention period for certain drawback claims. For all TFTEA drawback claims, supporting records must now be maintained for a period of 3 years from the date of liquidation of the claim, rather than 3 years from the date CBP pays a drawback claimant. This new time frame requires claimants with accelerated payment privileges to maintain supporting records for a longer period than before modernization. Prior to modernization, the drawback record retention period sometimes fell short of the time in which CBP liquidated a drawback claim, preventing CBP from substantiating a claim with complete documentation. The extension of the record retention period provides CBP with more time to request documents needed to verify claims during desk reviews, which in turn should strengthen its ability to recoup over claimed drawback refunds. According to CBP officials, if a claimant fails to provide documents as directed, or if the documents do not support the claim as presented, CBP can liquidate the claim at $0, or other diminishment as appropriate, and ACE will then issue a bill for outstanding funds owed. Broadening liability for drawback claims. Following TFTEA, liability for the full amount of a drawback claim shifted from the claimant to both the claimant and the importer of the designated imported merchandise upon which drawback refunds are claimed. CBP expects that establishing joint and several liability, consistent with TFTEA, will help it to recoup over claimed drawback refunds by holding the importer of record, in addition to the claimant, responsible for payment of erroneous or false drawback claims. According to the industry representatives we spoke to, the impact of the joint and several liability change remains to be seen, but it could limit the incentive of importers to engage in drawback filing with exporters or claimants to avoid liability. In addition to implementing these statutory changes, CBP has been working with a statistician to develop a more robust basis for sampling and selecting claims for review. For example, CBP has determined that it will target higher-value claims for more frequent review. CBP lacks effective automated controls to prevent overpayment of drawback refunds related to export information. CBP guidance notes that a statutory prohibition on multiple drawback claims is set forth in 19 U.S.C. § 1313(v), which restricts the use of merchandise that is exported or destroyed to a single claim for drawback. Unlike import information, which is included in ACE to allow CBP to electronically compare and verify claims against underlying import entries, similar export information is not included in ACE. Therefore, CBP cannot perform electronic comparisons of export data within ACE to help ensure that it does not make overpayments on drawback-related exports. For example, if a claimant exported 10 widgets and filed one drawback claim for six exported widgets and another claim for five exported widgets, CBP would not be able to systematically verify that the second drawback claim was excessive and thus invalid. To compensate for the lack of automated controls, CBP designed an internal control for the drawback program that targets a selection of claims for a manual full desk review by drawback specialists. (See appendix II for an explanation of what such desk reviews involve). However, CBP has not addressed several other risks for improper payments in the drawback program. These risks relate to (1) limitations in CBP’s existing desk review process, (2) establishing electronic proof of export, and (3) targeting a selection of claims for review. CBP’s existing manual desk review process does not have the ability to systematically confirm the validity of export documentation and confirm that export documentation is accurately being used across multiple claims. CBP officials noted that, while export documentation could be used across multiple claims, by law, claimants cannot file multiple drawback claims based on the same exported merchandise, as discussed above. Under TFTEA, a person claiming drawback refunds based on the exportation of an item must provide proof of export. Such proof must establish fully the date and fact of exportation and the identity of the exporter and may be established through the use of records kept in the normal course of business or through an electronic export system, as determined by CBP. To comply with this requirement, CBP requires claimants to (1) provide summary data as part of the drawback claim in ACE that includes the date of export, name of exporter, description of the goods, quantity and unit of measure, tariff classification number, and country of ultimate destination; and (2) maintain actual proof of export, which can be records kept in the normal course of business, and provide such proof upon demand by CBP. However, CBP officials told us that claimants only provide proof of export upon request by the drawback specialist, and that such requests typically are made after the claim is accepted in ACE and only in the context of desk reviews. Drawback specialists do not routinely request, store, or compare export documentation except for claims selected for desk reviews. CBP has no way of tracking whether claimants are using their export information excessively, and, according to officials, CBP has not yet assessed the feasibility of doing so. CBP officials explained that having the ability to flag excessive export submissions across multiple claims would enhance CBP’s protection against over claiming, but that further review is needed to determine whether flagging is feasible with current system capabilities. CBP officials said that they intend to look further into the matter in fiscal year 2020. As a result, the drawback program remains at risk of improper payments on drawback related to export information as claimants could over claim drawback refunds by using non-existent, insufficient, or falsified export documentation, or by reusing export documentation across multiple claims for merchandise that was never exported. CBP has not taken any steps to establish electronic proof of export, although it has a longstanding goal to designate the Automated Export System as an electronic means of establishing proof of export. Federal standards for internal control call for agency management to design the entity’s information system and related control activities to achieve objectives and respond to risks. However, CBP has not yet deemed the Automated Export System as a reliable system of record for proof of export. At the time the final rule was issued in December 2018, CBP commented that the Automated Export System, as it stands, could not provide sufficient proof of export, and CBP would therefore continue to require documentary proof of export until further notice. Specifically, CBP determined that the Automated Export System does not establish the date and fact of exportation, or the identity of the exporter—information that can be relied upon to demonstrate drawback eligibility. CBP officials in headquarters told us that while being able to develop a reliable system of record for proof of export remains a goal, CBP does not have a plan or time frames for doing so as it intends to revisit the matter in fiscal year 2020. CBP officials explained that their focus has been on transitioning the drawback program to ACE, including by training staff and addressing industry concerns. Until CBP implements effective control activities for the drawback program, the government may be subject to revenue loss through duplicate or excessive claims for drawback related to export information. We cannot precisely estimate the potential savings that might result from CBP taking steps to prevent over-claims because the current rate of improperly claiming against the same export documentation multiple times is unknown. Further, the current number and amount of drawback claims improperly using export information is unknown. However, if these steps reduced drawback-related costs by even 1 percent of the over $1 billion in annual drawback refunds, this could equate to millions of dollars in savings. CBP has not targeted a selection of claims for a manual full desk review since it disabled the selection feature in ACE, and the number of claims not targeted for review continues to increase because CBP has not turned the selection feature back on. The lack of review increases the risks of improper payments for claims filed, which stood at over 35,000 as of August 23, 2019, and represented an estimated $2 billion. To mitigate risks of improper payments in the drawback program, CBP designed an internal control for the drawback program in which a selection of claims is targeted for a manual full desk review by drawback specialists. Prior to modernization, CBP officials told us that they would target 1 percent of the claims per claimant and 1 percent of the entries on a drawback claim for a full desk review. Drawback specialists provided examples of having conducted full desk reviews in which they discovered that the claimants had failed to substantiate the claim by, for example, providing insufficient proof of export. They explained that the claimants had to repay their drawback refund and had CBP target subsequent claims for a limited desk review. However, CBP officials explained that when CBP transitioned the drawback program to ACE starting on February 24, 2018, a system error forced CBP to disable the selection feature in ACE. Certain claims that have been submitted since the system error was discovered have not been targeted for a full desk review. Federal standards for internal control call for agency management to identify, analyze, and respond to risks related to achieving the defined objectives. These standards note that agency managers should comprehensively identify risks and analyze them for their possible effects, as well as design responses to these risks as necessary to mitigate them. CBP officials told us that they are working toward turning the selection feature back on as soon as CBP can address the system error. However, CBP did not expect the issue to persist as long as it has to date (22 months, as of December 2019). As a result, even when the selection feature is reactivated, it will only be applied to new claims filed after that point. CBP does not have a plan to retroactively target claims for review that had already been accepted in ACE during the system error, or to identify and analyze risks from targeting to adjust targeting in the future. For example, CBP has not determined whether specific claimant characteristics or claim types are more frequently associated with compliance problems. CBP officials explained that analyzing risks from targeting to identify non-compliance patterns across claimants is not something CBP has done in the past because CBP is account based and does not compare claims across claimants. However, CBP officials acknowledged the feasibility—with ACE’s new capabilities—of systematically pulling and analyzing non-compliance data input into ACE by the drawback specialists during limited or full desk reviews, and told us that they intend to explore this matter further in fiscal year 2020. These officials stated that taking these steps would be valuable for improving risk management in the drawback program and that doing so is likely to be feasible with current staff resources. Without finalizing or implementing procedures to retroactively target claims for review and taking steps to analyze non-compliance patterns to improve future compliance processes, CBP may miss opportunities to protect U.S. trade revenue from improper payments of drawback claims. We cannot precisely estimate the potential savings that might result from CBP pursuing claims from the period when the selection feature was disabled, because the amount of drawback recovery resulting from the review of this universe is unknown, and the actual amount would depend on the number of reviews conducted, amount of improper payments discovered, and ability to recover these payments. However, if these reviews recovered even half of 1 percent of the $2 billion in un-reviewed claims, this could equate to millions of dollars in additional recoveries. CBP published a required Regulatory Impact Analysis of the Modernized Drawback Final Rule (RIA) of new drawback regulations in 2018 to outline, prospectively, the anticipated consequences of this economically significant regulatory action. The RIA was to include a quantification and monetization of anticipated benefits and costs, to the best extent possible with information available at the time. As of December 2019, CBP’s RIA was the only formal analysis that had been conducted on the impact of changes to drawback eligibility under modernization on industry and government. We assessed three key portions of the RIA relating to impact on industry and changes to drawback eligibility against GAO’s standards for review of economic analysis, and found that CBP had not produced reliable estimates. Various factors limited the analyses that CBP could conduct. For example, because the RIA was published prospectively, post- modernization program data were, necessarily, not yet available. According to CBP officials, CBP also developed the RIA before it had transitioned to ACE, a database with enhanced capabilities. However, in some cases, we found that CBP was not transparent about the level of uncertainty in its assumptions resulting from these limitations. We did not comprehensively assess the entire RIA (a 251-page document containing more than 90 tables) or assess any of it against the Office of Management and Budget’s guidelines for an RIA. Therefore, the following discussion of the RIA is not an assessment of whether the RIA met the criteria for required regulatory analyses outlined in the Office of Management and Budget Circular A-4. Our assessment of each of the relevant portions of the RIA is based on GAO’s standards for review of economic analysis, and outlined below. Affected industries: CBP determined that a wide range of industries would be affected by modernization but did not determine whether the dollar impact of eligibility changes from modernization would be more concentrated in some industries than in others because of data limitations. To reach the conclusion that a wide range of industries would be affected by modernization, CBP took a sample of companies that had submitted drawback claims and examined these companies to determine their primary industry. According to GAO standards, an economic analysis should state its objective and the scope of the analysis should be designed to address this objective. According to CBP officials, CBP designed this sample to support statements about the number of companies affected but not the dollar size of the impact, although CBP did not explicitly state the intent of this design in the RIA. At the time of the RIA, according to these officials, designing a dollar-weighted sample—which could support statements about which industries were most affected in terms of financial costs and benefits—would have required a prohibitive amount of work with paper records. However, the officials noted that a dollar-weighted sample should now be feasible because most of the necessary information is now stored electronically in ACE. Expansion of substitution eligibility: CBP estimated that the expansion of substitution eligibility would account for $1 billion (98 percent) of the $1.02 billion estimated total 10-year amount of increased drawback refunds under modernization; however, we found that this estimate was not reliable because of the amount of uncertainty in key assumptions. According to GAO standards, an economic analysis should consider all relevant alternatives and describe and justify the analytical choices, assumptions, and data used. CBP’s estimate was based on assumptions about changes to the dollar amount per drawback claim and number of drawback claims as a result of modernization and system limitations in the number of lines per claim. Specifically, CBP assumed that claim values would remain equal to their historical average (adjusted for line limitations in ACE) and that the number of claims under modernization would grow primarily in the first year after modernization. However, CBP did not justify some key methodological assumptions about the amount and number of claims and did not take sufficient steps to inform on the extent to which the conclusions of the analysis would remain similar, even if it changed some of these assumptions. CBP estimated the dollar amount per claim based on a historical average of drawback claim amounts but did not explain in the RIA why the historical average is an appropriate assumption for drawback claim amount. CBP officials told us that they considered a range of different drawback claim amount values and growth rates as a result of significant annual variation in drawback claim amounts prior to TFTEA. However, CBP did not include variation in claim dollar amounts in its published sensitivity analyses for this table or otherwise discuss, within the scope of these analyses, whether its conclusions would have been affected by this variation in the assumed amount per claim. Additionally, CBP’s estimate of expected increase in the number of claims that would be filed under modernization contains several key assumptions that it justifies based on emails and discussions with industry representatives and CBP subject matter experts, the details of which are not transparent in the RIA. We reviewed these emails and found that the two industry representatives whom CBP cited expressed uncertainty about the effects of modernization and provided estimates of growth in substitution drawback claims that varied by 20 percentage points from one another. CBP also sought public comments on these estimates and did not receive any, according to officials. As the estimated effect of this change constitutes nearly all of the estimated increase in drawback refunds in the RIA, the uncertainty around key assumptions for this analysis means that the overall actual effects of modernization could differ widely from CBP’s estimate. Limitations on basket provisions: CBP estimated that eliminating claims with basket provisions would cost industry about $11 million over 10 years; however, we found that this estimate was not reliable because of the amount of uncertainty in key assumptions. According to GAO standards, an economic analysis should consider all relevant alternatives and describe and justify the analytical choices, assumptions, and data used. These standards further note that, when feasible, an economic analysis should adequately quantify how the statistical variability of the key data elements underlying the estimates of the economic analysis impacts these estimates. While CBP’s general methodology was reasonable, its sample design was too small to ensure reliable results and some assumptions were not fully explained or transparent. CBP sampled 50 out of 2,346 substitution unused merchandise claims from 2016, of which 16 contained lines classified under basket provisions in the HTS code, and used this sample to estimate the number of affected claims and lines, as well as average affected line value. CBP officials told us that CBP selected this sample size because of the labor-intensive process required to examine paper records from the relevant claims. However, in the RIA, CBP did not discuss how this small sample size caused imprecision in its estimates. Further, CBP did not establish that this time-limited sample was generalizable beyond 2016, either for the proportion of affected claims and lines or for the average affected line value. CBP officials said that, to alleviate these issues, CBP sought public comments on these estimates and did not receive any. According to CBP officials, at the time of their analysis, there was no evidence about the average dollar amount of future claims. However, CBP did not conduct a sensitivity analysis on these assumptions, for example, to determine how much its estimates would change if the number or dollar amount of claims utilizing basket provisions was larger or smaller than CBP had assumed. Beyond its RIA, CBP has not conducted economic impact analysis of the changes to drawback eligibility under modernization, including on industry, and does not have plans to do so in the near future. Because the changes are new and CBP has devoted many of its resources to rolling out modernization, CBP stated that, while it intends to follow relevant requirements for regulatory review, it has not yet prioritized developing a plan for further assessments of the economic impact of the regulation. CBP officials stated that any future plans for retrospective review would follow Treasury guidance. This guidance states that priorities for retrospective review projects of existing significant regulations should be based upon an understanding of the economic impact of the regulatory action on industry and the government, among other factors. According to the RIA, the drawback modernization regulations are an “economically significant regulatory action.” The Treasury guidance states that such an understanding can be achieved through an ex post analysis of the effects of the regulation on the public, industry, or the government, including increased revenue or costs. An ex post analysis of impact on industry and the impact of major changes to drawback eligibility would have fewer limitations than the RIA, which analyzed the changes prospectively (using historical data to predict future outcomes). For example, because of system updates, more detailed data about lines within claims are now stored electronically, which may reduce the need to conduct sampling in order to estimate the impact of changes. Additionally, because the regulation is now in effect, information such as the number of claims filed can be determined with actual data rather than by projection. According to CBP officials, within 3 to 5 years the agency will have sufficient data to conduct a reliable ex post analysis of the impact of the changes. Useful analysis might be possible sooner, as well. CBP assumed in the RIA that some of the most important effects of modernization would occur in the first year. According to GAO standards, the reliability of an ex post analysis will depend not only on the sufficiency of data, but also on whether the analysis has considered and properly dealt with elements such as objective and scope, methodology, analysis of effects, transparency, and documentation. At present, however, CBP has not prioritized developing a plan with time frames to conduct such an analysis when the data are available—a plan that could include identifying key areas of analysis, data sources, and appropriate methodologies. Without an ex post analysis, CBP cannot reliably determine the financial effects of changes to drawback refund eligibility on industry and the government. CBP disburses about $1 billion in drawback refunds per year and expects the amount of drawback refunds dispersed to continue growing. According to CBP, TFTEA modernized CBP’s system for processing drawback claims, transitioning it from a paper-based to an electronic system, in an attempt to mitigate longstanding risks in the program. Despite the expected increase in drawback claims, CBP did not anticipate and then adequately manage the increase in drawback specialists’ workload. As a result, CBP has delayed timely processing of some drawback claims, rulings, and privilege applications, which has resulted in uncertainty for industry—potentially impeding trade. Since modernization, drawback claims continue to be at risk of improper payments with vulnerabilities in CBP’s export verification and quality control system. While drawback modernization addressed longstanding risks associated with the program by automatically verifying import information, export information still creates a risk. CBP cannot systematically verify the validity and accuracy of a company’s proof of export. As a result, companies could still over claim drawback refunds by using non-existent, insufficient, or falsified export documentation, or by reusing export documentation across multiple claims. Additionally, while CBP established internal controls to mitigate improper payment risks in the program, such as by targeting a selection of claims for review, it disabled this quality control measure for claims submitted since drawback modernization began in February 2018. Over 35,000 claims accepted since drawback modernization—amounting to over $2 billion—remain at risk for noncompliance. Without CBP finalizing and implementing procedures to target claims retroactively and in the future, CBP will continue to miss opportunities to protect U.S. trade revenue. Further, if CBP does not design its targeting system to mitigate identified risks, future claims also are at risk of noncompliance. Prior to drawback modernization, CBP was not able to produce a reliable assessment of the economic impact of the changes to the drawback program on industry and government because of data availability constraints, systems limitations, and other factors. However, modernization has eliminated some of these constraints, and CBP estimates that within several years it will have sufficient data to conduct an ex post analysis. However, CBP has not prioritized developing a plan to do so. Without such an analysis, CBP cannot be certain about the economic impact of drawback modernization. We are making a total of six recommendations to CBP. Specifically: The Commissioner of CBP should ensure that the Office of Field Operations, in consultation with the Office of Trade, develops a plan for managing its increased workload. (Recommendation 1) The Commissioner of CBP should ensure that the Office of Trade assesses the feasibility of flagging excessive export submissions across multiple claims and takes cost-effective steps, based on the assessment, to prevent over claiming. (Recommendation 2) The Commissioner of CBP should ensure that the Office of Trade develops a plan, with time frames, to establish a reliable system of record for proof of export. (Recommendation 3) The Commissioner of CBP should ensure that the Office of Trade turns the claim selection feature in ACE back on and finalizes and implements procedures to target claims for review that were accepted into ACE during the period in which the selection feature was disabled. (Recommendation 4) The Commissioner of CBP should ensure that the Office of Trade analyzes the results of its targeting of claims for review and designs responses to mitigate identified risks. (Recommendation 5) The Commissioner of CBP should ensure that the Office of Trade prioritizes developing a plan to conduct an ex post analysis of the impact on industry and government of key changes to the drawback program, including time frames and methodology. (Recommendation 6) We provided a draft of this report to CBP and Treasury for comment. In its comments, reproduced in appendix III, CBP concurred with all six of our recommendations. CBP also provided technical comments, which we incorporated as appropriate. We requested comments from Treasury, but none were provided. We are sending copies of this report to the appropriate congressional committees, the Commissioner of CBP, and the Secretary of Treasury. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8612 or gianopoulosk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines (1) the extent to which modernization affects drawback refund eligibility and U.S. Customs and Border Protection’s (CBP) management of its workload, as well as the extent to which CBP has (2) taken steps to address risks of improper payments in the program and (3) analyzed the impact of the changes to the program on industry and government. To examine the extent to which modernization affects drawback refund eligibility and CBP’s management of its workload, we reviewed statutory, regulatory, and agency drawback documents to identify and describe expansions and limitations to drawback refund eligibility. Specifically, we reviewed the Trade Facilitation and Trade Enforcement Act of 2015 (TFTEA), the Modernized Drawback Notice of Proposed Rulemaking, and the Modernized Drawback Final Rule to identify key changes resulting from amendments made to the drawback statute and implementing regulations. We also reviewed CBP’s internal guidance, which defines the standards that drawback specialists must meet when processing claims for drawback refunds in the Automated Commercial Environment (ACE) and under TFTEA. To understand the regulations and policies for drawback modernization, we interviewed CBP officials with the Offices of Regulations and Rulings and Trade Policy and Programs within the Office of Trade. To gain insight into how drawback modernization is working, in practice, we interviewed CBP officials from the Office of Field Operations and the four Drawback Centers. We visited the largest Drawback Centers, located in Newark, New Jersey, and San Francisco, California, to observe how they manage their workload and process claims. As context for CBP’s increasing workload following drawback modernization, we also collected data on the following: The number of drawback claims filed in calendar years 2018 and 2019, as of August 21, 2019. We also compared these data against the historical number of drawback claims filed from 2009 through 2017, as reported by CBP in its Regulatory Impact Analysis of the Modernized Drawback Final Rule (RIA). In addition, we reported on the amount of drawback claimed during this period as context for the size of the drawback program. TFTEA (1) provided for a transition period, from February 24, 2018 to February 23, 2019, during which drawback claimants could file under either the amended provisions or the drawback law as it existed previously; and (2) thereafter required all claims to be filed under TFTEA starting on February 24, 2019. As such, claims filed between 2009 and 2017 reflect pre-TFTEA drawback claims. Claims filed in 2018 and 2019 reflect drawback claims filed under both the amended provisions and the drawback law as it existed previously. The number and value of claims migrated to ACE from the Automated Commercial System—CBP’s prior system for filing drawback claims— as well as the number and value of these claims liquidated in the first 9 months of 2019. The number of limited modifications to existing manufacturing rulings submitted between February 24, 2018 and February 23, 2019. Claimants who wanted to operate under an existing manufacturing ruling were required to file a supplemental application for a limited modification to the existing ruling by February 23, 2019. The number of new manufacturing rulings submitted between February 24, 2019 and July 22, 2019. Claimants who want to operate under a manufacturing ruling but did not apply for a limited modification by February 23, 2019, need to apply for a new manufacturing ruling. The number of privilege applications submitted between February 24, 2018 and July 22, 2019. Claimants can apply for and obtain drawback privileges for accelerated payment and waiver of prior notice. We incorporated data reliability questions in our interviews with agency officials, such as how the data are derived, maintained, and updated, and how CBP ensures their completeness and accuracy. Based on our interviews with agency officials, we found these data to be sufficiently reliable for providing context for CBP’s growing workload since modernization. We then discussed steps that CBP had taken to manage its workload, such as how it had updated its staffing models, managed processing privilege applications, and managed automatic liquidation. We assessed CBP’s responses against federal standards for internal control, which call for agency management to evaluate pressure on personnel to help personnel fulfill their assigned responsibilities in accordance with the entity’s standards for conduct. We reviewed staffing data covering fiscal years 2014 through 2019 for drawback specialists. We previously reported on staffing data from fiscal years 2014 through 2016. We incorporated data reliability questions in our interviews with agency officials for the fiscal years 2017 through 2019 staffing data. To determine staffing shortfalls, we compared actual staffing data against the minimum staffing level mandated by the Homeland Security Act and the optimal staffing level identified in CBP’s Resource Optimization Model for 2017. We determined these data to be sufficiently reliable for the purposes of comparing actual to optimal and mandated staffing levels. In addition, to understand how CBP is implementing the changes to the drawback program under modernization and the impact of the changes to the program, we interviewed a non-generalizable sample of 15 industry representatives from a variety of sectors who (a) had submitted public comments on the proposed rule, (b) were part of CBP’s Trade Support Network Drawback Subcommittee, or (c) met our criteria for both (a) and (b). According to CBP officials, this subcommittee was CBP’s primary forum through which officials obtained input on the modernized drawback regulations from industry. We developed a standard set of questions to ask industry representatives, for example, regarding their company’s involvement in the drawback program, how drawback modernization has impacted their company, what industries have been most impacted by the changes, and any unexpected or unintended results of the modernization. To examine the extent to which CBP has taken steps to address risks of improper payments in the program, we reviewed prior independent audits of the program, as well as statutory, regulatory, and agency documents delineating changes to the program, to understand how the changes are expected to remediate prior audit findings. These documents included TFTEA, as well as CBP’s proposed and final rules for modernized drawback, the RIA, and internal and external guidance for filing and processing drawback claims. We also interviewed agency officials in headquarters and in the field to discuss prior audit findings and the successes and challenges, if any, to drawback modernization addressing identified issues. We then assessed steps that CBP had taken to mitigate improper payment risks in the drawback program against federal standards for internal control, which call for agency management to identify, analyze, and respond to risks related to achieving the defined objectives. We collected data on the number of claims filed between February 24, 2018 and August 23, 2019, and the total amount claimed, that were not targeted for a full desk review. Based on our interviews with agency officials, we found the data to be sufficiently reliable for the purposes of reporting on the total number and value of claims that were not targeted for a full desk review during this period. To examine the extent to which CBP has analyzed the impact of the changes to the program on industry and government, we evaluated CBP’s RIA against GAO’s standards for review of economic analysis. We assessed those portions of the RIA that relate directly to the financial impact of changes to drawback eligibility, corresponding to three tables describing (1) affected industries, (2) expansion of substitution eligibility, and (3) limitation of basket provisions. We then compared our assessments against applicable Department of the Treasury standards to determine if a future assessment could overcome the prior data limitations, warranting a limited review of certain aspects of an existing rule. However, we did not comprehensively assess the RIA (a 251-page document containing more than 90 tables) or assess it against the Office of Management and Budget’s standards for regulatory impact analysis. Therefore, our discussion of the RIA is not an assessment of whether the RIA met the criteria for required regulatory analyses outlined in the Office of Management and Budget Circular A-4. We conducted this performance audit from February 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Drawback claim filing and processing generally follows three steps. Claims are (1) submitted for initial acceptance or rejection, (2) reviewed for drawback eligibility as applicable, and (3) liquidated with full, partial, or no payment. U.S. Customs and Border Protection (CBP) officials indicated that while the transition from the Automated Commercial System (ACS) to the Automated Commercial Environment (ACE) under drawback modernization has automated the initial intake of drawback claims, the review of claims to determine drawback eligibility, as presented, remains largely a manual process. 1. Claim submission. Prior to modernization, claimants had to file paper claims, including supporting documentation. CBP was required to accept or reject claims and authorize accelerated payment within 90 days of receiving the paper claim. However, claimants could also submit an electronic summary of the claim, known as a shell record, through ACS. For accelerated payment claims with a shell record, CBP was required to certify the approved claim for payment within 21 days of receiving the electronic summary of the claim. Under modernization, claims must be filed electronically. The drawback claim is transmitted electronically via ACE and supporting documentation, when required, is uploaded via the Document Image System component of ACE. CBP officials explained that the transition to ACE had automated the initial intake process of drawback claims. Instead of a drawback specialist having to manually validate the claim for completeness and mail a response back to the claimant, ACE is able to make that determination within seconds and provide immediate feedback to the claimant on whether the claim is accepted or, if rejected, what errors need to be addressed. 2. Claim review. CBP policy before and after modernization has been to require a full or limited desk review of selected claims, according to CBP officials. Claims necessitating a drawback specialist’s full desk review will undergo a more comprehensive verification of the complete drawback claim that often requires additional information from the claimant. If additional information is required to process the drawback claim, CBP will send a formal request for information to the claimant. Additionally, CBP officials said that before and after modernization, if CBP identified compliance issues during its review of a drawback claim, the drawback specialist could target any subsequent claims filed by the claimant for a limited desk review. According to CBP officials, the time it takes a drawback specialist to conduct a desk review varies by claim, based on the nature of the claim and the experience of the drawback specialist. CBP reported that it could take more than 3 years for CBP to conduct a full desk review and determine the final disposition of a drawback claim. 3. Claim liquidation and payment. Prior to modernization, CBP would manually verify that drawback claimants had the accelerated payment privilege on file. CBP stated that claimants with the privilege of accelerated payment of drawback generally received their refunds 14 days after CBP accepted claims and authorized accelerated payment. Now, under drawback modernization, a claimant can receive accelerated payment without a drawback specialist’s involvement. ACE is programmed to automatically make accelerated payment on claims that have on file the accelerated payment privilege and a drawback bond that equals or exceeds the amount of the claim(s). CBP stated that claimants with the privilege of accelerated payment generally receive their refunds within 21 days of claim acceptance. Before and after modernization, drawback claims are set to automatically liquidate if all the designated import entries within a claim are liquidated and final within 1 year of the claim date, according to CBP officials. CBP officials said that drawback specialists must extend the claim to prevent it from automatically liquidating before the necessary reviews have been completed. Drawback claims can be extended for three 1-year periods. CBP officials explained that liquidation extensions are intended to provide additional time to obtain information or documentation necessary to complete the review of a drawback claim. If the claimant fails to provide documents as directed, or if the documents do not support the claim as presented, the claim will be liquidated based on the information on file, which may result in liquidation at $0, or other diminishment, as appropriate. CBP officials described the liquidation and payment of drawback claims with and without accelerated payment privileges, as follows. At the time of liquidation, for claims with accelerated payment privileges, ACE issues an additional refund if the final claimed amount is greater than the accelerated payment amount, or a bill, if the accelerated payment amount is greater than the final claimed amount. If the accelerated payment amount is the same as the amount determined at liquidation, no further action is necessary. For claims without accelerated payment privileges, ACE will issue a refund for the drawback amount approved at liquidation. Claimants have 30 days from the issuance of a bill to repay CBP any amount due. Claims may be reliquidated up to 90 days from the date of an original liquidation. In addition to the contact named above, Kim Frankena (Assistant Director), Alana Miller (Analyst-in-Charge), Andrew Kurtzman, and Esther Toledo made key contributions to this report. The team benefited from the expert advice and assistance of Debbie Chung, Martin De Alteriis, Jeff Isaacs, Christopher Keblitis, Grace Lui, and Oliver Richard.", "summary": "The United States enacted the drawback program in 1789 to create jobs and encourage manufacturing and exports, according to CBP. CBP has primary responsibility for overseeing the drawback program. It disburses about $1 billion in drawback refunds per year. According to CBP, TFTEA modernized the drawback program, generally broadening the scope of potential claims and allowing electronic filing starting February 24, 2018. As of February 24, 2019, claimants could only file claims under the drawback statute as amended by TFTEA. TFTEA also included a provision for GAO to assess drawback modernization. This report examines the extent to which (1) modernization affects drawback refund eligibility and CBP's management of its workload, (2) CBP has taken steps to address risks of improper payments in the program, and (3) CBP has analyzed the impact of the changes to the program on industry and government. GAO reviewed statutory, regulatory, and agency documents, and interviewed agency officials and industry representatives. The Trade Facilitation and Trade Enforcement Act of 2015 (TFTEA) generally expanded eligibility for the drawback program, which provides refunds to claimants of up to 99 percent of certain customs duties, taxes, and fees. For example, a claimant could claim a drawback refund on exported pants made in the United States using imported foreign fabric. The expansion from TFTEA has resulted in Customs and Border Protection (CBP) facing a growing workload. According to CBP officials, the most significant change from TFTEA is that it is now easier to qualify for certain drawback refunds. Industry representatives explained that new claimants are seeking drawback refunds and existing claimants are able to increase claim amounts. However, CBP has not adequately managed the increased workload and has not developed a plan for doing so. As a result, CBP faces delays in processing drawback claims that could result in uncertainty for industry, potentially impeding trade. GAO Example of One Potential Drawback Claim CBP has taken some steps to address risks of improper payments in the drawback program, but several risks remain. To help ensure it does not overpay funds, CBP now electronically verifies drawback claims against underlying import information. However, CBP cannot verify drawback claims against underlying export information because it does not maintain detailed information about exports in its new electronic system. To compensate for this lack of automated controls, CBP requires manual full desk reviews of a selection of claims to mitigate improper payment risks. However, CBP has not targeted certain claims for a full desk review since switching to the new system on February 24, 2018. The lack of review for claims, which numbered over 35,000 and represented an estimated $2 billion in claims filed as of August 23, 2019, increases the risk of improper payments. CBP has not produced a reliable assessment of the economic impact of the changes to drawback refund eligibility because of data availability constraints, systems limitations, and other factors. CBP has not prioritized developing a plan to revisit its economic analysis, although new data and systems capabilities are becoming available. Without such a plan, CBP will not have a reliable assessment of the impact of the changes on industry and government. GAO is making six recommendations, including that CBP develop a plan for handling its drawback workload, improve its validation activities, and prioritize developing a plan for an economic analysis of the regulation to understand its impact. CBP concurred with all six recommendations.", "document_type": "gao"}
{"report": "Trauma or adverse childhood experiences may include physical and sexual abuse, neglect, bullying, community-based violence, extreme poverty, the loss of a parent or primary caretaker, or natural disasters, among other things. These experiences may overwhelm a child’s natural ability to cope and can cause stress reactions in children, including feelings of intense fear, terror, and helplessness. When children are exposed to chronic stressful events, their neurodevelopment can be disrupted. As a result, a child’s cognitive functioning or ability to cope with negative or disruptive emotions may be impaired, causing long-term harm to their physical, social, and emotional well-being. These adverse effects may include changes in a child’s emotional responses; ability to think, learn, and concentrate; impulse control; self-image; attachments to caregivers; and relationships with others. Traumatic experiences have been linked to a wide range of health-related conditions, including addiction, depression and anxiety, and risk-taking behavior, and may also increase the likelihood of chronic ill health conditions, such as obesity, diabetes, heart disease, cancer, and even early death. Not all children will experience all of these effects. Children’s responses to traumatic events are unique and affected by many factors, including their age at the time of the event, the frequency and perceived severity of trauma, and the child’s innate sensitivity, as well as protective factors such as the presence of positive relationships with healthy caregivers, physical health, and natural coping skills. While all children can be affected by trauma, trauma is common among children who enter the child welfare system. Many of these children have been abused or neglected, and involvement in the child welfare system, primarily through placements into a foster care home, may cause additional trauma due to the separation from family; changes in school placement, neighborhood, and community; as well as fear and uncertainty about the future. Child welfare experts generally believe that child welfare systems that use trauma-informed approaches are better able to address children’s safety, permanency, and well-being needs. Although trauma-informed frameworks may vary, they generally include interventions as well as a change in culture; thus if an agency or organization is taking a trauma-informed approach, it is incorporating knowledge of trauma and its effects into its policies, procedures, and practices. A trauma-informed child welfare system may offer services to help identify and mitigate the effects of trauma, including screening and assessing children for trauma, and providing or referring children to services. These approaches may produce improved outcomes for children in the child welfare system, including fewer children requiring crisis services, such as residential treatment, and fewer foster home placements, placement disruptions, and reentries into foster care. Other trauma-informed approaches may result in reduced lengths of stay in foster care and improved child functioning and increased well-being. In addition to child welfare agencies, school staff and members of the school community can play a key part in recognizing and responding to children who have experienced trauma. In a 2017 report on child well- being, GAO reported that an expert noted that health and human service agencies are not the only entities needed to address child well-being and suggested that community stakeholders work together to determine what resources are needed for the children in their community. A trauma- informed school, characterized by an understanding and a commitment of teachers and staff to an awareness of how trauma affects students, is an example of a coordinated approach to trauma. Trauma-informed teachers and staff are aware of trauma’s impact on students’ behavior, their relationships, their ability or inability to self-regulate behavior, and how it contributes to their classroom behavior. Specific elements of a trauma- informed school may include addressing and treating traumatic stress, developing partnerships with students and families, evaluating and revising school discipline policies and practices, and creating a trauma- informed learning environment. Federal agencies, academic institutions, and community-based treatment centers have generated evidence-based trauma treatments that clinicians and therapists can use when working with children and their families. See table 1 for examples of treatments. Recent studies have also found that trauma-informed approaches that are infused into the practices and work of child welfare and school staff can help children, their families, and others. While these studies are limited in terms of the number of participants, they indicate the positive effects of including trauma-informed approaches into the work of child welfare staff and educators. For example, one study that used child welfare administrative data for about 1,500 children from Kansas found that implementing a trauma-informed approach was associated with improved child well-being and placement stability for children in foster care. Another study of two public child welfare agencies that involved 52 children, as well as child welfare staff, mental health providers, and foster parents and kinship caregivers, suggests, among other things, that fewer children exited foster homes for negative reasons, such as running away or moving to a group home, when families were trained in a trauma- informed approach. In addition, a study of 126 female youths residing in two treatment centers in Massachusetts suggests that the youth at the center receiving the trauma-informed approach experienced a reduction in post-traumatic stress disorder symptoms compared with the youth in the residential center that did not offer this approach. A study of five schools that adopted a trauma-sensitive approach also reported positive outcomes. For example, the study found a decrease in disciplinary actions, and staff at one school reported that the school felt safer and calmer. School staff also reported improved relations among colleagues and with students, as well as better relations between students and increased parent engagement. HHS’s ACF and SAMHSA have awarded discretionary grants to states specifically to address childhood trauma. From 2011 to 2013, ACF awarded 20 state and local agencies and other organizations discretionary grants to address childhood trauma, according to ACF officials, totaling about $58 million. Each grantee, including two state child welfare agencies and a county agency as well as two universities in five of the six states we selected to review, received up to 5 years of funding. The grants were used to screen and refer children to treatment, implement or expand trauma-focused, evidence-based treatments, and bridge the gap between child welfare and mental health. According to HHS officials, funding for the last of these grants will end in September 2019. SAMHSA also awards discretionary grants specifically to address childhood trauma to state and local agencies, universities, and other organizations through an initiative to transform mental health care for children and adolescents affected by trauma. The National Child Traumatic Stress Network (NCTSN), a collaborative network of experts created through the National Child Traumatic Stress Initiative (NCTSI), conducts research on trauma treatment approaches and provides services to children affected by trauma. In fiscal year 2017, SAMHSA received over $48 million for the NCTSN, and it awarded four new grants and supported 82 5-year grant continuations through NCTSI. Officials that we spoke with from one state child welfare agency, three universities, and two nonprofits in four of the selected states received grants through this initiative. Several of these entities used these funds to train clinicians and educate other child serving professionals about trauma and mental health conditions. In addition to grants that were specifically meant to address childhood trauma, the selected states used other HHS discretionary grants to support children affected by trauma. For example, officials from five state education agencies in the selected states told us that they received SAMHSA’s Project Advancing Wellness and Resilience Education (Project AWARE) grant. Wisconsin officials also said they received Education’s School Climate Transformation Grant, which was used to create the state’s trauma-sensitive schools initiative. Washington officials credited SAMHSA’s Mental Health Transformation Grant with driving the state’s initial trauma-informed work, including its guide about trauma in schools. State agency officials also reported using formula funds, meant for broad purposes like mental health, substance abuse, child welfare, and education, to support their work with children affected by trauma. Officials from five agencies in the selected states reported using formula funding from Title IV-E of the Social Security Act to help children affected by trauma. According to Colorado officials, the state’s Title IV-E waiver has allowed child welfare workers to screen, assess, and provide interventions that are trauma-informed. Also, North Carolina officials told us that Title IV-E, combined with other funding sources, has helped pay for trauma-informed learning communities to help counties build trauma- informed programming. Two states reported using the Substance Abuse and Mental Health Block Grants. (See table 2 for additional grants states reported using to support children affected by trauma.) In addition to federal funding, officials in the six selected states reported receiving state funding to support children affected by trauma. For example, officials in North Carolina told us that, in 2013, the North Carolina General Assembly appropriated $1.8 million in annually recurring funds to train clinicians in evidence-based trauma treatments. Also, in Massachusetts, state funding may be used to create and support trauma- sensitive initiatives in schools, among other things. In addition to state funding, officials in three of the selected states reported using nonprofit funding to support their efforts. HHS offers information and funds training and technical assistance to help state and local agencies support children affected by trauma. For example, state and local child welfare officials in each of the six selected states cited the National Child Traumatic Stress Network (NCTSN) as an important resource for information, training, or technical assistance. State and local officials in four of the selected states told us that they use the NCTSN’s Child Welfare Trauma Training Toolkit curriculum to train their staff. The curriculum, designed to be completed in about 13 hours, covers topics such as the essential elements of a trauma-informed child welfare system, the impact of trauma on the brain and body, and the identification of trauma-related needs of children and families. Also, two state child welfare agencies told us that they use the Resource Parent Curriculum to train foster parents and others about trauma, and another used the Think Trauma curriculum to prepare trainers of group home and residential center staff; both curricula are provided through the NCTSN. In addition, the NCTSN makes other resources available to state and local communities on its website. For example, NCTSN offers fact sheets about various assessments and treatments, including those mentioned in table 1, as well as two evidence-based treatments for use in school settings. In addition to information and training provided through the NCTSN, in 2012, HHS’s ACF issued guidance to encourage state child welfare directors to focus on improving behavioral and social-emotional outcomes for children who have experienced abuse or neglect. In 2013, SAMHSA, in collaboration with ACF and CMS, issued joint guidance to encourage the integrated use of trauma-focused screening, functional assessments, and evidence-based practices in child-serving settings. Also, in 2014, SAMHSA, in an effort to help service sectors, such as child welfare, education, and juvenile justice, become more trauma-informed, released Concept of Trauma and Guidance for a Trauma-Informed Approach. This document included a framework of key assumptions and principles of a trauma-informed approach. SAMHSA intended that the trauma framework be relevant to its federal partners and their state and local system counterparts and to practitioners, researchers, and trauma survivors, families, and communities. (See table 3.) In addition to the information and training and technical assistance referenced above, HHS and Education fund technical assistance centers and make other resources available to states, including: SAMHSA’s National Center for Trauma-Informed Care and Alternatives to Seclusion and Restraint offers technical assistance to various publicly-funded systems and organizations on issues relating to trauma education, among other things. Education’s Readiness and Emergency Management for Schools Technical Assistance Center helps local education agencies before, during, and after emergency situations. Among its various activities, this technical assistance center offers information and technical assistance to local education agencies and others on Psychological First Aid for Schools, which is an intervention model to assist students, staff, and families in the immediate aftermath of an emergency. Education’s National Center on Safe Supportive Learning Environments as well as its Positive Behavioral Interventions and Supports Technical Assistance Center offer an array of materials about trauma and approaches to supporting children affected by it. ACF, through its Child Welfare Information Gateway website, provides information on building trauma-informed systems, assessing and treating trauma, and addressing secondary trauma in caseworkers. It also offers trauma resources for caseworkers, caregivers, and families, as well as information about trauma training. In some instances, the website directs users to SAMHSA or the NCTSN’s website. Officials we spoke with in the six selected states told us they used a variety of approaches to help staff understand trauma and its effects on children, identify children affected by trauma, and provide support to them. These approaches range from training child welfare workers, educators, and clinicians to screening children for symptoms caused by traumatic experiences. They also include developing support systems, including providing services, to children and their families who need more help. While we did not evaluate the effectiveness of the selected state and county initiatives, many of them incorporate key trauma principles and activities cited in the SAMHSA framework above. For additional information on examples of approaches taken in each selected state and in selected counties, see appendixes I and II. State and local child welfare and education agency officials in the six selected states use various approaches to train staff and birth and foster parents about trauma and its effects on children and families. Child welfare officials in two states, Wisconsin and North Carolina, told us that they use learning communities to train staff, and in some instances, foster parents. For example, North Carolina’s child welfare agency used a learning community approach—which included face-to-face training, as well as coaching and practice, over an extended period—to work with child welfare staff in 32 of the state’s 100 counties, according to a state official. In a 2016 agency report, state officials reported that the 9- to 12- month learning community process was designed to allow staff the time required to become steeped in trauma knowledge, to learn how to spread that knowledge into skills and practices, and to develop a sustainable program. Conversely, state and local education and child welfare officials in three states told us that they use online learning or university coursework to train staff. For example, Wisconsin education agency officials told us that they developed a three-tiered training, including online modules for educators and school staff. The modules are designed for self-study and, among other things, include guidance on making policies and procedures more trauma-sensitive, as well as information about the characteristics of safe, supportive learning environments. Also, Massachusetts state child welfare officials told us that they partnered with three universities to provide trauma-focused courses to child welfare workers, and local school officials told us that a university offers a graduate certificate in trauma and learning to area educators. corroborated, when possible, the information we received during our state and county interviews with relevant state documents. We provided officials the opportunity to review the content for accuracy and provide revisions or corrections. a 10-hour, self-paced webinar. According to Wisconsin’s child welfare website, clinicians who complete the training are eligible for certification as TF-CBT therapists and can be listed on a national website of certified clinicians. Similarly, North Carolina’s state child welfare agency, in partnership with a nonprofit organization, trains clinicians in four trauma- focused, evidence-based therapies, including TF-CBT and Parent-Child Interaction Therapy. Similar to the Wisconsin effort, over the course of a year, clinicians learn about these therapies and practice them with children and families. While training staff and parents is important to broaden understanding of trauma and its impact on affected children, identifying these children is also key to helping them receive needed support, including trauma- focused treatment. State and local child welfare and education officials in five of the six selected states told us that they screen certain children to determine whether they have experienced trauma, are exhibiting symptoms of trauma, or need to be referred for a trauma-informed mental health assessment. For example, North Carolina and Washington child welfare officials told us they screen children for trauma when they enter the child welfare system. North Carolina counties that participated in the state’s training efforts, described above, use two screening tools: one for children under age 6 and the other for those ages 6 through 21. The social worker, with input from the caregiver, completes the screening tool for children under age 6. Older children are asked questions about their exposure to trauma, including physical abuse, domestic violence, sexual abuse, and other traumatic events. According to the North Carolina child welfare agency, the trauma screen has a number of benefits for child welfare practice, including informing placement decisions for the youth, prioritizing children who might need to receive treatment quickly, and providing the mental health professional with a better understanding of a child’s issues. Child welfare officials in Washington also reported integrating trauma screening into the state’s child screening program, using a 2012 ACF trauma grant. Children and youth are screened within 30 days of placement in foster care if officials expect them to remain in care 30 days or more. With these grant funds, officials reported that Washington’s child welfare agency added a tool to screen for children’s trauma symptoms and developed a protocol that rescreens these children every 6 months. In addition, education agency officials from three states told us that schools have developed processes to identify students who may have experienced trauma. For example, one Wisconsin school district official told us that any staff member, family member, or student can refer a student for screening. This official explained that the school district formed school-based teams to review information, such as data on suspensions and class disruptions, to identify at-risk students. In addition to the screening process, the school district developed school-based and community mental health service partnerships at 23 schools where therapists provide mental health services, according to this official. State and local child welfare and education agency officials in five of six selected states told us they have developed support systems, which can include providing services, to try to help children affected by trauma. For example, Colorado and Ohio child welfare agencies have spearheaded efforts to provide services and support to children who may have experienced trauma. The Colorado child welfare agency, as part of its system of care, uses an evidence- and team-based planning model, referred to as high-fidelity wraparound services, to manage care for children with or at risk of serious emotional disturbance and who are involved in multiple systems, such as the child welfare and juvenile justice systems. As part of these wraparound services, county child welfare staff and local service providers and professionals work with the family to create a plan for them and their children. A coordinator sets up meetings, oversees the plan, and makes sure all team members participate in achieving the plan’s goals. In addition to the coordinator, a family advocate provides peer support, via weekly visits, to parents and caregivers of youth receiving wraparound services. In addition, depending on the needs of the child, wraparound services may include participating in a support group or meeting with a therapist or grief counselor, among other things. In Ohio, child welfare officials in two counties told us about a partnership that provides services to children and their families who have experienced trauma because of parents’ substance use disorder. As part of the program, children and parents are screened for trauma and may get referred for treatment and services. Families receive wraparound services that are provided by a caseworker and family peer mentor; the family peer mentor has personal experiences with addiction and is in recovery. In addition, state education agency officials in four selected states told us that they had at least one statewide effort administered by the state education agency to help support all children, including those affected by trauma. Colorado, Washington, and Wisconsin encourage schools to implement tiered systems of behavioral support, according to state officials. Tiered systems of support generally consist of three tiers of support: (1) universal supports that apply to all children; (2) specialized supports for smaller groups of children; and (3) supports for individual children who need intensive interventions. To implement the first tier, school staff support students in various ways, such as interacting with students and setting up a dedicated space in a classroom for students to regulate their behavior. The second tier may include convening small groups to help children with similar behavioral issues learn how to regulate their emotions, and the last tier may include intensive support for students who need more help, such as developing and implementing wraparound services plans. School district officials that we spoke with in Massachusetts told us that although they do not use tiered systems of behavioral support, they help children affected by trauma by employing practices to create safe classroom environments for all students, such as developing and building upon relationships and engaging students in structured conversations. Officials in all six selected states spoke of the importance of having engaged leadership in establishing and sustaining support for children affected by trauma. They cited a wide range of leaders, including state government officials; managers and supervisors; and those in partner agencies, such as schools or nonprofits, who supported these states’ trauma efforts. In some cases, these leaders helped establish new trauma initiatives. For example, Wisconsin’s former First Lady launched the work of a statewide, interagency trauma initiative. Additionally, Ohio county child welfare officials spoke about the value of obtaining management support for their plan to become a trauma-informed organization. In other cases, leaders were seen as important to sustaining trauma initiatives and ensuring their impact. In Massachusetts, university officials said that, to ensure the continued availability of evidence-based therapies, they train not only clinicians, but also the individuals who supervise them. Also, a county public health official in Washington, whose agency is implementing trauma initiatives in schools, told us that their efforts tend to be unsuccessful unless they first engage school leadership and align their health initiatives with the schools’ existing efforts. Federal officials and reports have also cited leadership as an important factor in the implementation of trauma initiatives, with some maintaining that leadership is necessary to support children affected by trauma because of the need to change an organization’s culture. In 2013, NCTSN reported on takeaways from a learning collaborative in which nine teams led by child welfare agencies developed, implemented, and tested trauma-informed child welfare practices. Based on the experiences of the teams, the NCTSN report stated that strong and consistent leadership is necessary to implement trauma-informed practice because it requires a shift in organizational culture. SAMHSA’s 2014 guidance for a trauma-informed approach similarly suggests that organizations consider the importance of leadership to initiate a systems- wide change. In addition, HHS officials, who worked with states on a series of trauma-related grants awarded between 2011 and 2013, also told us that leadership commitment was important for their grantees in building organizational and worker resiliency, acting upon data and evaluation, and sustaining initiatives. These documents and statements echo previous GAO work on organizational transformation; for example, in 2003 we reported on key practices found at the center of successful transformation efforts, noting that leadership must set the direction, pace, and tone and provide a clear, consistent rationale that brings everyone together behind a single mission. In addition to discussing the important role that leadership plays in establishing and sustaining support for children affected by trauma, officials in three states highlighted instances in which a lack of leadership hindered their efforts to support these children. The cases they described included delayed, incomplete, or unsuccessful implementation of trauma initiatives. Delayed implementation. Officials in one school district said they had developed policies around multi-tiered system of supports in 2009 but did not receive support from political leaders or funding for the initiative until 2016. They told us that this hindered the initiative’s implementation. Incomplete implementation. State education officials in that same state said that a lack of leadership hindered their ability to track school districts’ implementation of the state’s trauma initiatives. These officials said that a lack of requirements for districts to scale up trauma work was a barrier to collecting data on local activities. In another state, there was a county child welfare initiative to implement universal trauma screening which was conducted in partnership with a local university. The university reported that less than half of children with open cases were screened during the project period, which university officials attributed to some supervisors not supporting the screening initiative. Unsuccessful implementation. According to officials in a third state, turnover among high-level leaders contributed to difficulties integrating trauma-informed practices at the state’s child welfare agency, and the agency was not successful at implementing a trauma screening process. Officials in all six selected states talked about limitations on their agency’s or organization’s capacity to support children affected by trauma. Limitations included high rates of staff turnover, limited staff time to focus on trauma, insufficient numbers of clinicians trained in trauma-focused, evidence-based therapies, and insufficient funding for trauma initiatives. Some agencies and organizations had taken actions to address these challenges. Secondary Traumatic Stress According to the National Child Traumatic Stress Network (NCTSN), Secondary Traumatic Stress (STS) is the emotional duress experienced when hearing about another person’s traumatic experiences. Professionals working with children affected by trauma, such as child welfare workers, are commonly at risk of developing STS. STS can compromise these professionals’ ability to do their jobs and may drive them to leave their job or their professional field. NCTSN notes that several factors can increase the risk for developing STS, including heavy caseloads of children affected by trauma, social or organizational isolation, and feeling unprepared for the job due to lack of training. NCTSN suggests taking a multi- dimensional approach to STS, which includes both prevention and intervention. This could include strategies such as establishing self- care groups, helping workers maintain work- life balance, and training organizational leaders on STS. education agencies. Child welfare officials in all six states talked about high rates of staff turnover, while education officials did so in two states (Colorado and Wisconsin). Staff turnover resulted in difficulties maintaining staff trained in trauma-informed approaches and sustaining institutional trauma knowledge and trauma-related activities, according to officials. Colorado university officials partnering with a county child welfare agency said that staff turnover forced them to invest additional time in training replacement staff and made it more difficult for child welfare officials to conduct regular follow-ups. Similarly, one education official in another part of Colorado said that high turnover at many agencies, including education and child welfare, hindered the county’s efforts to maintain institutional knowledge about trauma-informed practices and sustain the services these agencies were providing to children affected by trauma. Some state and local officials in three states attributed high rates of staff turnover to fatigue and secondary traumatic stress, which is the emotional duress that staff may experience when they hear about children’s traumatic experiences (see sidebar). Some agencies said that they sought to address staff turnover by supporting employees through training on secondary traumatic stress; at least one agency in each of the six states offered such training. Officials from Ohio and Wisconsin told us that another way they were addressing the issue was by participating in an HHS-funded project to improve child welfare workforce outcomes. Many agencies also said they faced limitations on the time that staff could dedicate to trauma initiatives. This issue was more commonly raised by education agencies than by child welfare agencies. Education agency officials reported this limitation in three of four states that had education initiatives, whereas child welfare officials reported it in two of the six selected states. Some of these officials explained that lack of staff time to focus on trauma may have limited the implementation of their trauma initiatives. State education officials in Washington and local education officials in Massachusetts told us that they have the expertise to provide trauma training to schools and community groups, but time limitations restrict their ability to do so. A Colorado county child welfare official told us that some caseworkers see trauma screening as an additional burden due to their already large workload, and a child welfare official in another Colorado county told us that many caseworkers forget to do trauma screening because they are busy. At least one agency we interviewed in each of the six states has or had a staff position dedicated to trauma work, which could help address this limitation. Officials in all six selected states said that there were not enough clinicians trained in trauma-focused, evidence-based therapies to serve children affected by trauma. GAO has previously reported on difficulties finding specialty care for children. For example, in 2017 we found that limited access to mental health services was a challenge for several selected states due to a variety of factors, including insufficient numbers of providers in certain specialties, such as child psychiatrists. Some officials indicated that a shortage of clinicians trained in trauma-focused, evidence-based therapies can limit the ability of child welfare agencies to address trauma. For example, state child welfare officials in Massachusetts specifically noted that identifying children affected by trauma is not helpful if there are not enough clinicians trained in these therapies to treat them. County child welfare officials in Massachusetts and local healthcare partners in Ohio said that providers sometimes rely on interns to address the shortage of clinicians, but Massachusetts officials viewed this as problematic because interns have short tenures that prevent them from establishing relationships with the children. Officials in five of the six selected states told us about initiatives to address the shortage by training clinicians in trauma-focused, evidence- based therapies, and university officials in Massachusetts described an initiative to make trained clinicians more accessible. (See text box.) LINK-KID: A centralized trauma treatment referral service The Child Trauma Training Center at the University of Massachusetts Medical School trains clinicians and operates a centralized referral service called LINK-KID. The goal of LINK-KID is to facilitate connections between children in need of trauma-focused, evidence- based therapies and clinicians who have been trained to provide such therapies. LINK-KID maintains an active database of trained clinicians throughout the state of Massachusetts. University officials told us that anyone in Massachusetts with concerns about a child, including family, teachers, clinicians, and child welfare workers, may call the service. LINK-KID collects information about the child and family, works with them to decide which treatment is most appropriate, and ensures the child is referred for that treatment. University officials said that using LINK-KID is easier for families and child welfare workers, who otherwise might have to call multiple service providers to determine who offers the needed treatment and accepts their insurance. These officials also said they have seen a reduction in the time children must wait for treatment when using LINK-KID. They said that prior to LINK-KID, they saw many children waiting 6 months to a year to receive treatment after having been identified as having experienced trauma, whereas wait times are generally between 25 and 40 business days with LINK-KID. Finally, some agencies said they had difficulties getting or maintaining sufficient funding to support trauma initiatives. Officials in Washington, including, among others, state and local education officials and a local public health partner, reported this issue. In addition, local officials in four other states noted limited funding to support trauma initiatives. School district officials in Washington indicated that a lack of funding limited their implementation support for one major trauma initiative to approximately one-quarter of their schools. These schools were chosen based on need, as demonstrated by measures such as discipline and absenteeism rates. County child welfare officials in Ohio said they had to stop one of their trauma initiatives 3 years ago because the state funding supporting the initiative ran out. Those Ohio officials said they have relied on relationships and collaboration to address the issue of scarce funding. For example, they said that county organizations, including local government agencies, private healthcare providers, and nonprofits, share data extensively and pool funding to support various initiatives. One initiative they pointed to is a local interagency council which provides services to children affected by trauma. Child welfare and other officials in the six selected states, including officials with nonprofit partners, a state department of health, and a state interagency collaborative, also raised at least one other challenge. Challenges included sharing data while remaining in compliance with state and federal privacy laws; sharing data across incompatible systems; limitations on services billable to Medicaid; and Medicaid reimbursement rates. Some agencies had taken actions to address or avoid data sharing challenges. In the states where child welfare officials identified Medicaid- related challenges, state Medicaid officials offered a different perspective on perceived Medicaid challenges and cited alternative ways to support children affected by trauma. Officials in all six states talked about sharing data with other agencies for various purposes; however, privacy laws and regulations were sometimes cited by these officials as a barrier to sharing data about children affected by trauma. For example, officials in two Massachusetts school districts told us they are notified by police or child welfare workers when a child has been involved in an incident with those agencies. One official described the goal of this effort as making staff aware of incidents and events that may affect children’s learning and behavior and ensuring that children feel supported. However, child welfare officials in four of the six selected states and other officials in two states said that it was difficult to share data while remaining in compliance with state and federal privacy and confidentiality laws and regulations, though the reasons they cited for these difficulties varied. State child welfare officials in Massachusetts told us that the state has strict privacy laws in addition to federal laws such as the Health Insurance Portability and Accountability Act of 1996. These officials said that data sharing is possible but generally requires a specific memorandum of understanding because of privacy laws. In contrast, a state child welfare official in North Carolina said they had difficulties with counties not understanding what data they are allowed to share. That official told us that the state tries to mitigate this challenge by helping counties understand what they can share and encouraging them to share screening information with mental health and medical providers. Additionally, a North Carolina university has published state-specific guidance on sharing education, mental health, and other records. Systems incompatibility and technology issues were also sometimes seen as barriers to sharing data about children affected by trauma. Child welfare officials in three of the six selected states, and state health officials in a fourth state, said that incompatibility among various systems made data sharing very difficult or impossible. For example, county officials in Wisconsin said that the state’s child welfare and juvenile justice offices use one data reporting system while the state’s mental and behavioral health offices use another, and these two statewide data systems are unable to communicate. While state child welfare officials in Colorado also reported systems incompatibility issues, county child welfare officials in that state talked about efforts to make data systems more accessible to relevant partners. Officials in one county said that they have a database which is accessible by all members of the county’s multi- agency partnership, including child welfare, school districts, public health, and others. Those officials also said they use a universal release-of- information which includes all partner agencies, enabling them to share data at multi-agency meetings. Additionally, child welfare officials in Colorado, Ohio, and Massachusetts said that certain services for children affected by trauma or certain service providers were not billable to Medicaid, although Medicaid officials in these states offered a different perspective and cited alternative ways to support these children. Depending on the state, child welfare officials said they could not bill wraparound services, trauma assessments, transportation, or non-traditional therapies, such as animal therapy or community and relationship building. County child welfare officials in Ohio also mentioned restrictions on providers; they said that potential peer support specialists with a criminal background and interns could not bill Medicaid. However, Medicaid officials in these states generally said that such services were billable to Medicaid, and Ohio Medicaid officials said that interns and those with a criminal background could bill Medicaid, under certain circumstances. For example, they said that while certain severe criminal offenses, such as homicide, could exclude someone from providing services, those with lesser offenses could become eligible after a waiting period. Colorado and Ohio Medicaid officials we spoke with offered some alternative ways to use Medicaid to support children affected by trauma in cases where services could not be billed to Medicaid. For example, a Colorado Medicaid official and a child welfare official both said that Medicaid does not pay providers for travel time or mileage and that this can be a problem in rural areas; however, the state Medicaid official said that telehealth is available to address this issue and that reimbursement rates for services in rural areas can be higher to reflect the additional cost of travel. Finally, child welfare and Medicaid officials in Colorado and North Carolina also had different perspectives regarding Medicaid reimbursement rates. Child welfare and other officials in these states said that certain services for children affected by trauma, such as trauma assessments and trauma-focused, evidence-based therapies, are expensive, and that Medicaid reimbursement rates are too low to incentivize providers to offer these services. However, Colorado and North Carolina Medicaid officials explained that most children in Medicaid in their states receive mental health care through managed care, where the state pays a set rate per child to managed care organizations (MCOs) to provide or arrange for any mental health services a child may need, including trauma-related care. MCOs, in turn, reimburse providers for the services they deliver, and MCOs set the rates they pay providers for those services rather than the state. Medicaid officials in Colorado and North Carolina noted that MCOs have flexibility to negotiate rates with providers and may choose to reimburse at a higher rate. North Carolina Medicaid officials said that some MCOs in their state were reimbursing providers at a higher rate for comprehensive, trauma-informed mental health assessments, and a Colorado Medicaid official also noted that MCOs in their state may vary reimbursement rates based on provider availability, offering higher rates in areas where there are shortages. We provided a draft of this report to HHS and Education for review and comment. HHS did not provide written comments. Education provided technical comments, which were incorporated into the report as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretaries of HHS and Education, congressional committees, and other interested parties. In addition, this report will be available at no charge on the GAO website at https://www.gao.gov If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. In addition to the contact named above, Elizabeth Morrison (Assistant Director), Ramona L. Burton (Analyst-In-Charge), Isabella Guyott, and Robin Marion made significant contributions to this report. Also contributing to this report were Luqman Abdullah, Susan Aschoff, Sarah Cornetto, Kelsey Kreider, Hannah S. Locke, Jean McSween, Mimi Nguyen, Stacy Ouellette, Michelle Rosenberg, Almeta Spencer, Daren K. Sweeney, Shelia L. Thorpe, and Carolyn Yocom.", "summary": "Trauma is a widespread, harmful, and costly public health problem, and its effects are especially detrimental to children. Any frightening, dangerous, or violent event that threatens a child or their loved ones can potentially be traumatic. While not every child who experiences trauma will suffer lasting effects, trauma significantly increases the risk of mental health problems, difficulties with social relationships and behavior, physical illness, and poor school performance. GAO was asked to review selected states' efforts to support children affected by trauma. This report describes (1) the assistance that HHS and Education provide to help state and local agencies support children affected by trauma; (2) how child welfare and education agencies in selected states support these children; and (3) the challenges these agencies have faced in selected states in supporting these children. GAO interviewed state and local officials in six states that were selected based on recommendations from subject-matter experts and federal officials, among other factors; administered a questionnaire to 16 state agencies in the selected states; interviewed federal officials from HHS and Education; and reviewed relevant federal, state, and local agency documents, such as reports and guidance. Although our findings cannot be generalized to all states, they provide insight into government support for children affected by trauma. GAO is not making recommendations in this report. The Department of Health and Human Services (HHS) and the Department of Education (Education) provide grants, disseminate information, and fund training and technical assistance to help state and local agencies support children affected by trauma. HHS's Administration for Children and Families and Substance Abuse and Mental Health Services Administration (SAMHSA) have awarded discretionary grants specifically to address childhood trauma. In addition, state and local officials reported making use of other discretionary grants from HHS and Education—as well as formula funds meant for broad purposes like mental health, substance abuse, child welfare, and education—to support their work with children affected by trauma. In terms of non-financial support, state and local officials in six selected states all referred to the National Child Traumatic Stress Network, which is funded by SAMHSA, as an important resource for information, training, and technical assistance. Both HHS and Education have also made other guidance and informational resources available to states. Officials in child welfare and education agencies in the six selected states reported using a range of approaches to help children affected by trauma, including training staff, screening children, and providing services and support systems. To train child welfare workers, educators, and birth and foster parents to understand trauma and its effects on children, agencies in the six selected states used various approaches, such as learning communities, which include in-person learning and coaching, and online courses. Several state child welfare agencies also used learning communities to train clinicians in trauma-focused therapies. In addition, child welfare and education agencies in five states used screening tools to identify children exposed to and exhibiting symptoms of trauma. Children identified as experiencing trauma are referred for a trauma-informed mental health assessment. Also, to help children affected by trauma, child welfare and education agencies in five of the six states provide support and services. For example, in one state, caseworkers provide specialized services, including weekly visits, to children and families. Officials in the six selected states reported facing various challenges in their efforts to support children affected by trauma, and they emphasized the importance of engaged leadership in establishing and sustaining support for these children. In three states, officials said that a lack of such leadership hindered their efforts, and they described cases that included delayed, incomplete, or unsuccessful implementation of initiatives. Officials in all six states also talked about limitations on their agency's capacity to support children affected by trauma, including: high rates of staff turnover, especially in child welfare; limited staff time to dedicate to trauma initiatives; lack of clinicians trained in trauma-focused therapies; and insufficient funding to support trauma initiatives. Officials in some states reported strategies they have used to help address these challenges, including providing additional support to employees and coordinating with partner agencies to jointly leverage resources, expertise, and data.", "document_type": "gao"}
{"report": "OSC is an independent federal investigative and prosecutorial agency. Its primary mission is to safeguard the merit system in federal employment by protecting employees and applicants for federal employment from prohibited personnel practices, especially reprisal for whistleblowing. OSC reviews disclosures of wrongdoing within the federal government from current federal employees, former employees, and applicants for federal employment. These individuals, known as whistleblowers, make disclosures of alleged wrongdoing to OSC that the employee reasonably believes evidences either (1) a violation of law, rule, or regulation; (2) gross mismanagement; (3) gross waste of funds; (4) abuse of authority; (5) a substantial and specific danger to public health or safety; or (6) censorship related to research, analysis, or technical information. If a whistleblower believes his or her agency took, threatened to take, or did not take a personnel action because of a protected disclosure, the whistleblower may file a retaliation complaint with OSC. An employee may file a retaliation complaint with OSC even if the protected disclosure was made to another body such as an Inspector General’s office rather than OSC. Various statutory provisions have established protections for federal employee whistleblowers over the years. The Civil Service Reform Act of 1978 provided the first statutory whistleblower protections for disclosures of violations of laws, mismanagement, or gross waste of funds for federal employees, former employees, and applicants for employment. The 1978 act established both the Merit Systems Protection Board (MSPB) and OSC and placed OSC within MSPB. Under the act, OSC was authorized to review allegations of wrongdoing within federal agencies, to investigate and obtain corrective action over allegations of prohibited personnel practices, including whistleblower retaliation, and to initiate disciplinary actions against employees who commit prohibited personnel practices, among other things. Later, to strengthen protections for those who claim whistleblower retaliation, Congress passed the Whistleblower Protection Act of 1989. The 1989 act separated OSC from MSPB, making OSC an independent agency. The act also created the individual right of action, allowing whistleblowers to bring their appeals to MSPB after exhausting remedies at OSC. In 2012, the Whistleblower Protection Enhancement Act clarified the scope of protected whistleblowing under the Whistleblower Protection Act and mandated broader outreach to inform federal employees of their whistleblower rights, among other things. Further, the Dr. Chris Kirkpatrick Whistleblower Protection Act of 2017, among other items, enhanced disciplinary penalties for supervisors who retaliate against whistleblowers. Federal employees in the civil service are required to serve a period of probation when they begin serving initial appointments. These periods are typically for 1 to 2 years, and they allow an agency to evaluate the employee before the appointment becomes final. Our prior work notes that the probationary period provides a way for agencies to dismiss poorly performing employees or those engaging in misconduct before the process to do so becomes more complex and lengthy. In particular, we concluded that the probationary period could be more effectively used by agencies, which in turn could help agencies deal with poor performers more effectively. According to MSPB, the probationary period, if used fully, is one of the most helpful assessment tools available for supervisors to determine an individual’s potential to fulfill the requirements of the specific position. During the probationary period, the employee is still technically considered an applicant for employment. As such, probationary employees do not have the same protections against adverse personnel actions as other employees. Prior to firing a probationary employee for poor job performance or misconduct, an agency does not need to afford the same procedural protections required before removing a non- probationary employee. Therefore, it is reasonable to expect that probationary employees will be terminated at higher rates than permanent employees. Probationary employees also lack the same rights to appeal adverse actions, such as demotions or removals, to the MSPB that other federal employees have. However, probationary employees do have some legal protections. For example, probationary employees may file a complaint with OSC if they believe a personnel action such as reassignment, demotion, or removal was retaliation for whistleblowing. If OSC determines there are reasonable grounds to believe that retaliation has occurred, it may seek corrective action, including filing a petition with the MSPB. Additionally, a probationary employee who has filed a complaint with OSC may subsequently file an individual right of action with MSPB. Probationary employees also may appeal to MSPB if they believe they have been fired for partisan political reasons or because of discrimination based on their marital status. Probationary employees also have the right to file a complaint of discrimination with their agencies and subsequently file an appeal of a final agency decision with the Equal Employment Opportunity Commission or a civil action in federal district court if they believe that they have been discriminated against based on their race, color, religion, sex, national origin, age, disability, or genetic information. The average annual total of probationary and permanent federal employees from fiscal years 2014 through 2018 was approximately 1.9 million. During the same time period, 14,043 federal employees filed whistleblower disclosures, whistleblower retaliation complaints, or both. That is, an average of roughly 2,800 employees—about 0.15 percent of the federal workforce—filed complaints each year. For whistleblower disclosure complaints, whistleblower retaliation complaints, or both over this 5-year period, we estimate that probationary employees filed between 6.6 percent and 18.2 percent of complaints, while permanent employees filed between 76.8 percent and 93.4 percent of complaints. Because existing data are insufficient to determine probationary status of employees for more than 18 percent of each year’s complaints, it is not possible to determine whether probationary employees file at lower, comparable, or higher rates than their prevalence (about 13.5 percent, on average, across this time period) in the overall employee population. Figure 1 shows how many employees we could determine through matching were in probationary and permanent status when they filed whistleblower disclosure or retaliation complaints, along with the numbers of unmatched complaints for fiscal year 2018. The pattern is similar for the other years we examined; estimates for each year are available in appendix II. Overall, probationary employees—whether or not they have filed a complaint with OSC—are terminated at a higher rate than permanent employees, which is consistent with expectations that determining the suitability of employees for the particular position is a major purpose of the probationary period. In fiscal year 2018, 1.1 percent of probationary employees were terminated, regardless of whether they filed a whistleblower disclosure or retaliation complaint. In the same year, 0.3 percent of permanent employees were terminated, regardless of filing status. These percentages were consistent across the years we studied. As discussed below, estimated termination rates for permanent and probationary employees who filed either or both types of complaints we examined consistently exceeded these government-wide rates. Specifically, among permanent employees who filed, estimated termination rates could be anywhere from 1.7 to 17.1 percentage points higher than the 0.4 percent average for all permanent employees over this period. Among probationary employees who filed, estimated termination rates could be from 5.3 to 72.6 percentage points higher than the 1.3% average for these employees government-wide. Whistleblower disclosures. Estimated termination rates among employees who filed whistleblower disclosures from fiscal years 2014 to 2018 were higher than termination rates among all federal employees. This applies to both probationary and permanent employees. Specifically, estimated termination rates for probationary employees who filed were higher than estimated termination rates for permanent employees who filed. For example, as shown in table 1, in fiscal year 2018: The lowest estimated rate (minimum) of termination among probationary employees who filed whistleblower disclosures was 10.1 percent, compared to the overall 1.1 percent termination rate for all probationary employees. The lowest estimated rate (minimum) of termination among permanent employees who filed whistleblower disclosures was 2.9 percent, compared to the overall 0.3 percent termination rate for all permanent employees. Taking unmatched complaints into account, we estimated that the termination rate for probationary employees who filed whistleblower disclosures could be any percentage from 10.1 to 46.9 percent. Taking unmatched complaints into account, we estimated that the termination rate for permanent employees who filed whistleblower disclosures could be any percentage from 2.9 to 5.2 percent. The minimum estimated termination rate for probationary employees (10.1 percent) who filed whistleblower disclosures exceeds the maximum estimated rate for permanent employees who filed whistleblower disclosures (5.2 percent). Whistleblower retaliation complaints. We found that the lowest possible rates (minimums) of termination for employees who filed whistleblower retaliation complaints were higher than termination rates among all federal employees, both for probationary and permanent employees. Specifically, estimated termination rates for probationary employees who filed were higher than estimated termination rates for permanent employees who filed. For example, as shown in table 2, in fiscal year 2018: The lowest estimated rate (minimum) of termination for probationary employees who filed retaliation complaints was 17.4 percent, compared to the overall 1.1 percent termination rate for all probationary employees. The lowest estimated rate (minimum) of termination for permanent employees who filed retaliation complaints was 5.5 percent, compared to the overall 0.3 percent termination rate for all permanent employees. Taking unmatched complaints into account, we estimated that the termination rate for probationary employees who filed whistleblower retaliation complaints could be any percentage from 17.4 to 69.4 percent. Taking unmatched complaints into account, we estimated that the termination rate for permanent employees who filed retaliation complaints could be any percentage from 5.5 to 9.9 percent. The minimum estimated termination rate for probationary employees who filed retaliation complaints (17.4 percent) exceeds the maximum estimated rate for permanent employees who filed retaliation complaints (9.9 percent). Both whistleblower disclosures and retaliation complaints. For the category of employees who filed both whistleblower disclosures and retaliation complaints, termination rates were higher than termination rates among all federal employees, both for probationary and permanent employees. Specifically, estimated termination rates for probationary employees who filed were higher than estimated termination rates for permanent employees who filed. For example, as shown in table 3, in fiscal year 2018: The lowest estimated rate (minimum) of terminations among probationary employees who filed both whistleblower disclosures and retaliation complaints was 14.1 percent, compared to the overall 1.1 percent termination rate for all probationary employees. The lowest estimated rate (minimum) of terminations among permanent employees who filed both types of complaints was 7.8 percent, compared to the overall 0.3 percent termination rate for all permanent employees. Taking unmatched complaints into account, we estimated that the termination rate for probationary employees who filed both types of complaints could be any percentage from 14.1 to 56.3 percent. Taking unmatched complaints into account, we estimated that the termination rate for permanent employees who filed both types of complaints could be any percentage from 7.8 to 13.2 percent. The minimum estimated termination rate for probationary employees who filed both a whistleblower disclosure and a retaliation complaint (14.1 percent) exceeds the maximum estimated rate for permanent employees who, filed both types of complaints (13.2 percent). As previously discussed, probationary employees being terminated at a higher rate than permanent employees is consistent with expectations, given that determining the suitability of employees for the particular position is a major purpose of the probationary period. However, the higher rate of termination for filers generally, and the higher estimated rates for probationary employees specifically, suggests a potential relationship between filing and terminations that may disproportionately impact probationary employees. As stated earlier, we did not determine whether the disclosures and complaints filed had merit, whether termination actions were justified, or whether the terminations occurred before or after the filing of the whistleblower disclosure or retaliation complaint. As such, further examination would be needed to fully understand these relationships. OSC requires federal employees to use OSC Form-14 to submit a complaint alleging a prohibited personnel practice or a disclosure. Complainants begin the process by selecting a checkbox based on their particular complaint or disclosure. Depending on their selections, complainants are asked to provide additional information. Data fields on the form that are marked with an asterisk are mandatory. OSC instructions state that the agency cannot process forms lacking necessary information. OSC Form-14 includes a non-mandatory data field that asks whether the complainant is currently a probationary employee. Because it is not a required field, complainants may choose not to provide that information. According to OSC, it has designated only a limited amount of requested information as mandatory. OSC officials said that to avoid creating impediments for employees to file complaints, mandatory fields are limited to the information that is necessary for processing a complaint. In August 2019, according to OSC officials, OSC transitioned to a new electronic Case Management System (eCMS). This new system’s electronic version of the complaint form includes a data field as part of the question about employee status. Here employees can check off probationary status for OSC to capture and input complainants’ probationary status. According to OSC, when complainants provide this information, the agency is able to track the information in eCMS. OSC officials estimated that a number of filers voluntarily provide information on probationary status; however, the officials could not specify to what extent filers provide that information in their initial filings, or the extent to which this data is collected during processing of the case. OSC’s mission is to “safeguard the merit system by protecting federal employees and applicants from prohibited personnel practices, especially reprisal for whistleblowing.” Additionally, OSC’s 2017-2022 strategic plan includes an objective to ensure agencies provide timely and appropriate outcomes for referred whistleblower disclosures. One of the agency’s strategies to help achieve that objective is to monitor all whistleblower disclosures and referrals to agencies to identify trends or systemic challenges. Further, Standards for Internal Control in the Federal Government states that management should use quality information to achieve the entity’s objectives. OSC officials stated that OSC’s routine administration of disclosures and complaints allow them to identify trends. However, this process does not consistently use standard, structured data to identify trends, but rather relies on the personal experience of investigators. Without consistent quality information, including information on probationary status, OSC cannot have reasonable assurance that it is adequately identifying trends and challenges. OSC told us that because of limited resources it currently has no plans to conduct data studies or analyses of employees in their probationary period who file whistleblower claims. As previously discussed, the higher rates of termination we found for complainants, and in particular for probationary employees, suggests a potential relationship that warrants further examination. However, without consistent identification of probationary employees who file whistleblower claims, OSC will continue to lack complete data that would enable this analysis and support OSC’s goal of identifying trends and systemic challenges. Collecting and maintaining such information on every claimant, which could now be more easily done under eCMS, would provide OSC or other entities the ability to analyze termination rates or other issues related to a whistleblower’s probationary status. Having more complete information on trends and challenges could help OSC to ensure that its current level of resources are being distributed to support its mission. Probationary employees, by definition, are relatively new to their positions and are thus uniquely vulnerable to retaliation from employers due to the limited protections afforded them. Our estimates demonstrate that employees who file whistleblower disclosures and complaints of retaliation are terminated at a higher rates than employees government- wide, and suggest that these differences may be more pronounced for probationary employees. OSC has roles and responsibilities related to understanding key trends and challenges for whistleblowers, and could potentially further investigate whether these differences indicate a particular risk for probationary employees. However, they are not collecting data on probationary status that would enable them to do so. Without consistent information on probationary status, OSC is unable to properly analyze the effect of that status on those who file whistleblower disclosures, retaliation complaints, or both; and thus, cannot have reasonable assurance there is equal treatment of probationary employees. The Office of Special Counsel should require federal employees who are filing whistleblower disclosures or retaliation complaints to identify on their complaint forms their status as a permanent or probationary employee. We provided a draft of this report to OSC for review and comment. In its written comments, reproduced in appendix III, OSC disagreed with our conclusions and recommendation. While we continue to believe that our conclusions and recommendation are fully supported by the evidence— as discussed below—we made minor clarifications to our report to more clearly state the nature of our findings in response to OSC’s comments. OSC also provided technical comments, which we incorporated as appropriate. In its written comments, OSC expressed a concern that our report overreaches. OSC stated that our report appears to draw its conclusions based on correlative instead of causative data. Specifically, OSC stated that our report appears to connect the expected greater rate of termination of probationary employees to whistleblower retaliation, based on correlative data and without taking into account key factors such as justification for the termination, timing in relation to the disclosure or the filing of a complaint, or the merit of the individual’s complaint. Absent this type of crucial, detailed analysis that could help determine causation, OSC stated that few, if any, conclusions can be drawn regarding alleged retaliation experienced by probationary employees. As stated in our draft report, and noted by OSC, our estimates demonstrate that employees who file whistleblower disclosures and complaints of retaliation are terminated at higher rates than employees government-wide, and the estimates suggest that these differences may be more pronounced for probationary employees. Our draft report acknowledged that we did not assess certain factors: (1) whether the disclosures and complaints filed had merit, (2) whether the termination actions were justified, or (3) whether the termination actions occurred before or after the filing of the whistleblower disclosure or retaliation complaint. Because we did not control for these factors, we did not speculate about what caused these differences to occur or make causal claims about the relationship between probationary status and whistleblower retaliation. Instead, we stated that further examination and analysis would be needed to fully understand this indicator of potential risk. As we noted in the report, such analysis would require complete and accurate data on probationary status—data which OSC does not currently collect. Therefore, we recommended that OSC collect more complete data so that OSC could, if it chose, do exactly the type of crucial, detailed analysis that it says could help determine causation. Accordingly, we continue to believe that our recommendation for OSC to collect complete and accurate data on probationary status is warranted as such analysis is not possible without it. OSC also expressed a concern that our report appears to suggest that it perhaps may not be doing enough to protect probationary employees. OSC asserted that it already has reasonable assurance that it is appropriately protecting probationary employees from unlawful retaliation. We did not assess OSC’s review of the filed disclosures and complaints, and we made no claims or implications about whether OSC’s protection of whistleblowers is adequate or appropriate. Our report uses one specific outcome (terminations) as an example of an adverse employment action that could potentially signal retaliation. We did not present any findings about whether terminations were warranted, whether employees were appropriately protected, or any other information related to OSC’s handling of cases. We continue to believe, however, that OSC’s ability to run relevant data reports is constrained when the necessary data are not collected for the total population of filers. Without consistent quality information, including information on probationary status of all filers, OSC cannot have reasonable assurance that it is adequately identifying trends and challenges. Lastly, OSC stated that making employment status fields mandatory is onerous and unnecessary and that singling out probationary status from the list seems arbitrary and incomplete. The agency stated that the form includes the option for the individual to self-identify as a probationary employee, which OSC believes is sufficient. We do not believe that changing a field from optional to mandatory would place an undue burden on filers or OSC. We are sending copies of this report to relevant congressional committees, the Special Counsel and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2717 or jonesy@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our objectives were to (1) analyze the extent to which employees who filed whistleblower disclosures and retaliation complaints were in a probationary status, (2) analyze the extent to which these filings were associated with differences in termination rates, and (3) examine Office of Special Counsel (OSC) procedures related to probationary employees. We reviewed the Office of Special Counsel’s OSC 2000 database design documentation and submitted questions to OSC officials to determine what data were available. OSC does not collect or maintain data that identify whistleblowers and retaliation complaints filed by employees in probationary status in OSC 2000. In late August of 2019 OSC officials state that in late August of 2019 OSC launched a new system called the electronic Case Management System (eCMS) to replace OSC 2000. We submitted a series of questions pertaining to how OSC will collect and maintain probationary status information of employees filing complaints in eCMS. These questions pertained to the functionality of and reporting capability of eCMS in addition to OSC’s ability to conduct analysis of complainants who are in probationary status using eCMS. We obtained all closed whistleblower disclosure case data and closed prohibited personnel practices complaint data with allegations related to whistleblower retaliation from 2014 to 2018 from OSC’s previous electronic case management system (OSC 2000). We also requested and obtained 2014 to 2018 OPM Enterprise Human Resources Integration (EHRI) data. OSC 2000 is a case management system, so it was necessary to use combinations of variables associated with complaints filed, such as first name, last name, agency, email address, and job series to identify individual employees. We analyzed employees from federal agencies that submit human resources information to OPM. Factors such as complaints filed anonymously, name changes, and spelling variations could affect the precision of these counts of employees. However, because we are presenting these data in broad ranges throughout the report, these limitations do not likely affect our overall findings and message. After identifying employees in the OSC 2000 data, we then matched OSC 2000 data to OPM’s EHRI data. This was necessary because the OSC 2000 database does not include the probationary status of people filing complaints with OSC. We started by matching unique name and agency combinations. If that was not sufficient, we attempted to match using variables such as state, job series, and employee work email address. We matched OSC 2000 data to EHRI data using case data from OSC 2000 and federal probationary status as of the end of the fiscal year date from EHRI. We acknowledge that matching using these dates may not be precise, but because we present our results in ranges, we do not believe a more precise matching of dates would have resulted in substantive differences in the results overall. We matched 82 percent of the complaints in OSC 2000 to employees in EHRI. . Because it is not possible to determine the probationary status for unmatched cases, the rates of filing among matched cases may not precisely reflect the overall rates for all probationary employees. To account for this uncertainty, we estimated minimum and maximum rates of filing for permanent and probationary employees, and present these ranges in addition to the specific matched rates. Further, we calculated the number of instances in which matched employees who filed either a whistleblower disclosure or a retaliation complaint were terminated from federal employment. As we did with filing rates, we also estimated minimum and maximum termination rates to account for the uncertainty introduced by unmatched cases. Terminations were used because they represent adverse consequences for employees which could indicate retaliation. While other indicators, such as transfers could represent a potential retaliatory action, we focus on terminations because this is the most serious adverse action for which probationary employees have the little protection, and because OSC officials indicated that complaints with termination are prioritized. We did not determine (1) whether the disclosures or complaints had merit, (2) whether the termination actions were justified, or (3) whether the termination actions were before or after the filing of the whistleblower disclosure or retaliation complaint. Because these estimates do not consider the timing or merit of terminations, or other factors potentially associated with terminations, they do not represent proof of a causal relationship between filing and terminations, but rather one indicator of potential risk. To produce reasonably conservative estimates, we made certain assumptions in estimating the minimum and maximum rates in our ranges. Specifically, for unmatched cases we assumed that unknown characteristics, including probationary status and termination rate could be as much as 3.5 times their observed rate in known data. We believe these assumptions are reasonably conservative. While it is not impossible for this small group of unmatched complaints to be even more skewed, there is no evidence to suggest such an extreme assumption would be warranted. We assessed the reliability of the OSC 2000 and EHRI databases for the purposes of using limited data from these databases for our own analysis. We reviewed agency documents, electronically tested data for missing data and outliers, and submitted questions to agency officials about these databases. These two databases are the only sources of data that can be compared to determine the probationary status of individuals filing complaints with OSC. We determined that OSC’s data were sufficiently reliable to present the number of complaints filed by type. With regard to probationary status, the data were not available in OSC 2000. As a result, probationary status and termination rates were drawn from EHRI, which we found to be sufficiently reliable for this purpose. We conducted this performance audit from January 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The figure shown below details the distribution of probationary matched, permanent matched,and unmatched complaints for fiscal years 2014- 2018. In addition to the contact named above, Clifton G. Douglas Jr. (Assistant Director), Katherine Wulff (Analyst-In-Charge), Michael Bechetti, Karin Fangman, Steven Flint, Robert Gebhart, Steven Putansu and Wesley Sholtes made key contributions to this report.", "summary": "Federal employee whistleblowers—individuals who report allegations of wrongdoing—potentially help to safeguard the government from fraud, waste, and abuse. OSC was created to help protect whistleblowers. Probationary employees—generally those with less than 1 or 2 years of federal service—can be especially vulnerable to reprisal because they have fewer protections from adverse personnel actions, including termination. A 2017 law included a provision for GAO to examine retaliation against whistleblowers in their probationary period. This report examines (1) the extent to which probationary employees filed whistleblower disclosures or reprisal complaints, (2) termination rates of complainants, and (3) OSC procedures related to probationary employees. GAO used complaint data and workforce data to identify the probationary status of employees who filed claims with OSC from fiscal year 2014 to 2018 (the most recent full years of available data); estimated the number of instances where claimants were terminated; and reviewed OSC procedures. GAO found that existing data are not sufficient to determine if the rates of filing whistleblower disclosures, retaliation complaints, or both vary by probationary status. The average annual number of probationary and permanent federal employees from fiscal years 2014 to 2018 was approximately 1.9 million employees. Over this time frame, an average of approximately 2,800 employees—about 0.15 percent—filed complaints each year. Existing data were not sufficient to determine probationary status of employees for over 18 percent of each year's complaints. Therefore, it is not possible to determine whether probationary employees file at lower, comparable, or higher rates than their prevalence in the overall employee population. Specifically, probationary employees represented about 13.5 percent, on average, of the federal workforce, and GAO estimates that they filed from 6.6 percent to 18.2 percent of complaints. GAO estimates suggest that both permanent and probationary employees who filed complaints were consistently terminated at higher rates than federal employees government-wide. For example, in fiscal year 2018, the termination rate for probationary employees government-wide was 1.1 percent, while the lowest estimated rate of termination among probationary employees who filed a complaint was 10.1 percent. For permanent employees, the overall termination rate was 0.3 percent, while the lowest estimated rate for filers was 2.9 percent. GAO estimates also suggest that probationary employees who filed complaints were terminated at higher rates than permanent employees who did the same. For example, in fiscal year 2018: The lowest estimated termination rate for probationary employees who filed whistleblower disclosures (10.1 percent) exceeded the maximum estimated rate for permanent employees who did the same (5.2 percent). The lowest estimated termination rate for probationary employees who filed retaliation complaints (17.4 percent) exceeded the maximum estimated rate for permanent employees who did the same (9.9 percent). The lowest estimated termination rate for probationary employees who filed both types (14.1 percent) exceeded the maximum estimated rate for permanent employees who did the same (13.2 percent). The Office of Special Counsel's (OSC) complaint form allows but does not require complainants to identify whether they are probationary or permanent employees when filing a whistleblower disclosure or retaliation complaint. OSC officials said they try to limit mandatory data fields to the information that is necessary for processing a case, and that they have no plans to do any analysis of employees in their probationary period who file claims. However, the higher rates of termination GAO found for filers generally, and probationary employees specifically, suggests that there could be a risk of unequal treatment. Without first identifying probationary employees who file whistleblower claims, OSC would lack complete data should it decide at some point to analyze the effect of probationary status on filers. Collecting and maintaining such data on every claimant would provide OSC or other entities the ability to analyze termination rates or other issues related to a whistleblower's probationary status. GAO recommends that OSC require claimants to identify their status as permanent or probationary employees. OSC disagreed with GAO's recommendation. GAO continues to believe the recommendation is valid, as discussed in the report.", "document_type": "gao"}
{"report": "VHA manages one of the largest health care delivery systems in the United States and is responsible for overseeing the provision of health care at VA medical facilities. VA relies on its EHR system—VistA—to document the delivery of health care services to veterans. To facilitate care, clinical providers access patient medical records and document the care they provide in EHR systems. Patient information needs to be accessible and consistent to prevent risks to patients’ safety, particularly when shared between providers. Information that is electronically exchanged from one provider to another must adhere to the same standards to be consistently interpreted and used in EHRs. In our prior work, we found that EHR technology has the potential to improve the quality of care that patients receive and to reduce health care costs. VistA has served as VA’s EHR system for more than 30 years. Over the last several decades, it has evolved into a technically complex system that comprises about 170 modules that support health care delivery at more than 1,500 medical facilities. In addition, customization of VistA, such as changes to the modules by the various medical facilities, has resulted in approximately 130 versions of the system VA-wide. Furthermore, as we have reported, VistA is costly to maintain and does not fully support VA’s need to electronically exchange health records with other organizations, such as DOD and community providers. VA and DOD have historically operated separate EHR systems. In addition to patient data from its own EHR system, VA relies on patient data from DOD to help ensure that it has access to the necessary health information that could assist clinicians in making informed decisions to provide care to service members transitioning from DOD to VA’s health care system. We have previously reported on VA’s challenges in managing health information technology and modernizing VistA. In 2015, we designated VA health care as a high-risk area for the federal government, in part due to its information technology challenges. Specifically, we identified limitations in the capacity of VA’s existing information technology systems, including the outdated, inefficient nature of key systems and a lack of system interoperability, as contributors to the department’s challenges related to health care. In our 2019 update to the high-risk series, we stressed that VA should demonstrate commitment to addressing its information technology challenges by stabilizing senior leadership, building capacity, and finalizing its action plan for addressing our recommendations, and by establishing metrics and mechanisms for assessing and reporting progress. We also have issued numerous reports over the last decade that highlighted the challenges facing VA in modernizing VistA and improving EHR interoperability with DOD. VA created the Office of Electronic Health Record Modernization in 2018 to lead its EHRM program effort, which was intended to result in a more modern EHR system that would improve providers’ ability to deliver care, and share health data, including between VA and DOD and between VA and community providers. For example, with improved interoperability, medical providers would have the ability to query data from other sources while managing chronically ill patients, regardless of geography, or the network on which the data reside. In June 2017, the VA Secretary at the time announced that the department planned to acquire and configure the same EHR system that DOD is currently implementing across the military health system. According to the VA Secretary, the department decided to acquire the same system as DOD because it would allow all of VA’s and DOD’s patient data to reside in one system, thus assisting the departments in their goals of enabling seamless care between VA and DOD without the exchange and reconciliation of data between two separate systems. As VA planned to implement the same system DOD is implementing, experts recommended that VA and DOD coordinate to ensure that the departments could leverage efficiencies and minimize variation between the departments’ EHR system configurations when practical. DOD’s initial implementation of the Cerner EHR system occurred between February and October 2017 at four military treatment facilities in the state of Washington. In September 2019, the system was implemented at four additional military treatment facilities in California and Idaho. DOD plans to continue to implement the EHR system in 23 phases through 2023 with the next implementation expected to take place at eight additional military treatment facilities in California and Nevada. VA’s EHRM program originally planned to implement the Cerner EHR system at two VA medical facilities in spring 2020 with a phased implementation of the remaining facilities over the next decade. The EHRM program chose the Mann-Grandstaff VA Medical Center in Spokane, Wash. and the VA Puget Sound Health Care System in Seattle, Wash. as its initial operating capability sites. Information gathered from these sites will be used to help VA make EHR system configuration decisions and standardize work processes for future locations where the commercial EHR system will be implemented. In August 2019, the EHRM program adjusted its schedule to implement the commercial EHR system at these two sites in two phases, known as capability sets 1 and 2: Capability set 1 includes key EHR functionalities necessary to implement the system at the Mann-Grandstaff VA Medical Center, a level 3—that is, less complex—facility. Capability set 1 was originally scheduled for implementation in March 2020. Capability set 2 includes remaining functionalities necessary to implement the system at the VA Puget Sound Health Care System, a level 1—that is, highly complex—facility, in the fall of 2020. In February 2020, VA postponed the implementation of the Cerner EHR system at the Mann-Grandstaff VA Medical Center until July 2020. According to VA officials, the additional time will allow Cerner to develop and establish a more complete and robust training environment, as requested by VHA clinicians and other facility staff. In addition, according to VA EHRM program officials, the implementation delay will allow VA and Cerner to have time to develop additional interfaces between the Cerner EHR system and other VA systems, such as VA’s mail-order pharmacy system. These officials told us that the delayed implementation of the Cerner EHR system at the Mann-Grandstaff VA Medical Center was not expected to impact VA’s timeline for implementing the EHR system at the VA Puget Sound Health Care System in the fall of 2020. In April 2020, the VA Secretary announced that the department had shifted priorities to focus on caring for veterans in response to the pandemic created by the Coronavirus Disease 2019 (COVID-19). Further, the Secretary directed the EHRM program to allow clinicians who had been participating in EHRM program activities to focus on caring for veterans. According to program officials, they paused the implementation of the EHR system and were assessing the impact of the COVID-19 pandemic on VA’s planned implementation schedule. VA’s EHRM program used a multi-step process to make EHR system configuration decisions for the Cerner EHR system being implemented at the VA Mann-Grandstaff Medical Center and Puget Sound Health Care System. This process included forming EHR councils and convening these councils at national and local workshops to make configuration decisions used by VA’s contractor, Cerner, to configure the new EHR system. The EHR councils also assessed the compatibility of the EHR system with the processes VA clinicians and staff follow in delivering care. EHR councils. In fall 2018, VA’s EHRM program established 18 EHR councils, based upon specific clinical and administrative areas, to make VA-specific EHR system configuration decisions for these areas. Each EHR council included subject-matter experts from VA, such as health care providers in various clinical areas and other staff, as well as non-VA participants from DOD and Cerner. According to VA EHRM program officials, Cerner’s typical process for configuring its EHR system was modified to accommodate VA’s needs, which VA officials stated were more complex than those of Cerner’s commercial clients. According to Cerner officials, Cerner does not typically establish councils as part of its EHR system configuration process. National workshops. VA’s EHRM program planned and held eight national workshops from November 2018 to October 2019, during which members of all 18 EHR councils met to make standardized EHR system configuration decisions for the VA health care system. VA’s EHRM program utilized DOD’s version of the Cerner EHR system—MHS Genesis—as its starting point for the EHR system configuration process. During the workshops, Cerner assigned consultants to facilitate these workshops, who highlighted Cerner’s commercial best practices and prepared workflow designs, according to VA EHRM program and Cerner officials; facilitated EHR system configuration decision discussions and noted input from EHR council members and other session participants such as DOD representatives; held sessions that involved members from different EHR councils for system configuration decisions that required coordination between councils. For example, the Business Operations Council and the Ambulatory Council held joint sessions to address scheduling appointments for oncology patients; was responsible for identifying and documenting recommendations for EHR system configuration decision differences between VA sites, and each medical facility specialty/department; and provided weekly progress updates to VA that reflected overall progress of expected decisions to be completed compared to the actual approved EHR system configuration decisions during national workshops. Over the course of the eight national workshops, EHR council members were responsible for making EHR system configuration decisions in given clinical and administrative areas and communicating them to Cerner; providing progress updates to VA’s EHRM program and VA leadership; and notifying appropriate governing bodies (e.g., VHA program offices— such as the Office of Primary Care) of any local, state, federal, VISN, and department policies that impact configuration decisions. More specifically, each council discussed VA’s work processes and documented relevant information that informed the configuration of the EHR system, including: (1) “workflows”—”process maps” that capture the start-to-finish sequence and interactions of related steps, activities, or tasks for each work process that VA medical facilities follow. For example, VA has a medication administration workflow for describing the sequence of tasks needed for scanning a patient’s wristband and administering medication. (See fig. 1.) (2) “design decision matrices,” which are compilations of decisions and discussion topics that identify and resolve workflow questions to inform configuration decisions and support implementation of the EHR system. For example, the medication administration design decision matrix documents that clinicians should not be prevented from proceeding with medication administration if a patient’s wristband cannot be scanned. (See fig. 2.) (3) “data collection workbooks,” which capture all of the data needed to inform how the EHR system should be configured to support each workflow, such as user privileges and preferences. For example, a data collection workbook for medication administration includes data on user preferences and prescribing privileges. (See fig. 3.) The EHR system configuration decisions each council needed to make varied significantly in quantity and topic. For example, the Ambulatory Council, charged with focusing on primary care decisions, had over 200 EHR system configuration decisions to make, while the Behavioral Health Council had about 100. Once configuration decisions were made, the EHR councils assessed the compatibility of the configuration of the Cerner EHR system with VA work processes. To do so, VA’s EHR councils reviewed the capabilities of the system and identified work processes that the Cerner EHR system did not support (or only partially supported). For example, according to VA Mann- Grandstaff Medical Center staff, the Cerner EHR system did not originally interface with VA’s Patient Centered Management Module, which supports VA’s work processes for establishing provider-patient relationships. However, in March 2020, VA EHRM officials told us that the interface between the two systems would be available when the Cerner EHR system is implemented at the Mann-Grandstaff VA Medical Center, which was planned for July 2020. In addition, according to VA EHRM officials, Cerner is in the process of developing EHR system capabilities for prosthetics to support VA work processes. Furthermore, according to VA EHRM officials, Cerner has been documenting and tracking needed capabilities for EHR implementation and updating VA’s EHRM program accordingly. According to EHRM program officials, Cerner plans to include functionalities not available in capability set 1 in either capability set 2 or future capability sets, although the development of these capabilities is an ongoing process. Although the eight national workshops have concluded, since October 2019, these EHR councils have continued to meet as necessary, virtually, and in person, to complete capability set 1 and 2 configuration decisions. According to Mann-Grandstaff VA Medical Center staff, as of February 2020, VA still needed to make EHR system configuration decisions to address online prescription refills and assigning patients to primary care panels. Local Workshops. After standardized EHR system configuration decisions were made at the national workshops, they were reviewed at local workshops for site-specific needs. To do this, from December 2018 to October 2019, VA’s EHRM program held eight local workshops at each of the initial operating capability sites—the Mann-Grandstaff VA Medical Center and the VA Puget Sound Health Care System. Local workshops allowed VA and Cerner to identify variances from standardized EHR system configuration decisions made at the national workshops as well as manual processes that needed to be accounted for at local medical facilities. If variances were identified, Cerner reported them to the appropriate EHR councils. While VA tried to minimize the variances in system configuration decisions, in certain cases, necessary alternatives to these configuration decisions were approved for local medical facilities if practicable. For example, according to a Cerner official, the national emergency room triage workflow originally called for an emergency department registrar to register a patient; in response to input from a local workshop, VA developed an alternative workflow, in which an emergency department registered nurse completes the step if a VA facility does not have an emergency department registrar. If there were no variances, EHR system configuration decisions were approved and reported to Cerner to configure the EHR system. According to EHRM program officials, VA plans to hold local workshops in advance of the Cerner EHR system implementation at future VA medical facilities to focus on site-specific configuration decisions. Cerner will continue to facilitate these future local workshop sessions and configure the EHR system based on decisions made at these sessions. Figure 4 provides an overview of the EHR councils’ process for making system configuration decisions. VA met its schedule for making EHR system configuration decisions for capability set 1, which was scheduled for initial implementation at the Mann-Grandstaff VA Medical Center in July 2020. In addition, VA has formulated a schedule for remaining EHR system configuration decisions for capability set 2, which it planned to implement at the VA Puget Sound Health Care System in the fall of 2020. Our review of VA progress data shows that VA met the schedule for making EHR system configuration decisions it had established, which required VA’s 18 EHR councils to make at least 70 percent of decisions needed for capability set 1 by October 18, 2019. An EHRM program official stated that this threshold was required to enable Cerner to configure the EHR system for the Mann-Grandstaff VA Medical Center in anticipation of the system’s initial implementation. According to VA’s progress data, collectively, the 18 EHR councils met the requirement to make at least 70 percent of their total expected EHR system configuration decisions for capability set 1. Specifically, as of early November 2019, VA data for EHR configuration decisions needed for capability set 1 indicated that the EHRM program had developed: 877 of 966 (or 91 percent) of workflows; 1,397 of 1,412 (or 99 percent) of design decision matrices; and 1,364 of 1,610 (or 90 percent) of data collection workbooks. After the EHR councils collectively met VA’s goal to make 70 percent of EHR system configuration decisions by October 18, 2019, efforts continued to make the remaining decisions for capability set 1. In March 2020, VA data indicated that, combined, the EHR councils had developed an additional: 9 percent of workflows—874 of 878 (or nearly 100 percent); 1 percent of design decision matrices—1,459 of 1,467 (or nearly 100 10 percent of data collection workbooks—1,746 of 1,751 (or nearly 100 percent). (See Appendix I for additional details on specific changes from November 2019 to March 2020 by EHR councils.) As noted earlier, though the workshop process has concluded, a VA EHRM program official stated that they had plans to hold virtual—over teleconference or videoconference—meetings to allow the EHR councils to make remaining EHR system configuration decisions for capability set 1 at the Mann-Grandstaff VA Medical Center, by March 2020. VA’s EHRM program has formulated a schedule for making EHR system configuration decisions for capability set 2, which are necessary to support the implementation of the Cerner EHR system at the VA Puget Sound Health Care System planned for the fall of 2020. Specifically, VA’s EHRM program is continuing to make EHR system configuration decisions outside of the workshop process, which concluded in October 2019. Currently, EHRM program officials have plans to hold smaller meetings, about a fourth of the size of the national workshops, to make EHR configuration decisions that require input from multiple councils for capability set 2. According to EHRM program officials, the program set a goal of developing capability set 2 workflows, design decision matrices, and data collection workbooks by May 2020 so that the EHR councils could start validating the system configuration decisions at that time. EHRM program officials anticipate that this schedule for capability set 2 gives Cerner enough time to configure the EHR system and establish a training environment to enable implementation of the EHR system at the VA Puget Sound Health Care System planned for the fall of 2020. According to program officials, capability set 2 is composed of about 90 percent of configuration decisions for capability set 1 and 10 percent of additional workflows and data collection workbooks. These officials also told us that, as part of the process of making capability set 2 configuration decisions, they would determine the effectiveness of these decisions based on the implementation of capability set 1 at the Mann-Grandstaff VA Medical Center and make any necessary changes. VA’s EHRM program established EHR council decision-making procedures that were generally effective. In addition, the councils included a wide range of stakeholders, in terms of geographic representation and representation from VA central office, VISNs, and medical facilities. However, according to EHR council participants, VA did not always ensure adequate representation at local workshops. VA’s EHRM program’s decision-making procedures for the EHR councils were generally effective as demonstrated by adherence to applicable federal standards for internal control. According to these standards, management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. In addition, according to our leading collaboration practices, clarity can come from agencies working together to define and agree on their respective roles and responsibilities and participating agencies should document their agreement. VA’s EHRM program established the organizational structure, assigned responsibility, and delegated authority for system configuration decisions to the EHR councils. Specifically, the EHRM program developed a charter for the councils that outlined each council chair’s responsibility for managing council membership and ensuring it is consistent with guidelines for broad representation; outlined council member roles and responsibilities, such as participating in face-to-face meetings and conferences, providing subject matter expertise, and guiding EHR system configuration decisions; and delegated authority for EHR system configuration decisions from the EHRM Chief Medical Officer to the council chair and members. According to EHRM program documentation, VA established decision- making authority at the lowest level possible, beginning with the EHR councils, to ensure timely and appropriate decision-making. Based on our observations of national council workshop meetings, if a council had questions that involved coordination with another council, the Cerner consultant present would take note of the issue and coordinate a meeting with the relevant councils to discuss the issue. For example, participants from the Ambulatory Council met with participants from the Rehabilitation and Acute Clinical Ancillaries Council to discuss the EHR system configuration decisions for ordering glasses and contacts. Based on our review of the Functional Governance Board charter and meeting minutes, when a decision required coordination and could not be made at the EHR council level, it was identified and escalated to the Functional Governance Board. The Functional Governance Board provided guidance on addressing issues or, in turn, escalated unresolved issues to the higher-level Governance Integration Board, or if appropriate, to a joint VA and DOD coordination process. According to EHRM program officials, as of February 2020, there were no issues escalated from the Functional Governance Board to the Governance Integration Board because the council governance structure strived to make decisions at the lowest level possible. Figure 5 provides an overview of the EHRM program’s decision-making procedures. With respect to collaboration, because VA is using the same system as DOD, VA has had to coordinate with DOD on some decisions. Although both departments have procedures for configuring the Cerner EHR system for their individual needs, VA EHRM program officials noted the importance of coordinating to design a system that would allow sharing of information and tasks between VA and DOD. According to VA EHRM program officials, for example, VA and DOD coordination is necessary for workflows pertaining to durable orders for life-sustaining treatments—medical treatments intended to prolong the life of a patient who would die soon without the treatment (e.g., artificial nutrition and hydration, and mechanical ventilation). VA and DOD’s practices differed on how to address such treatment, and Cerner’s process did not accommodate VA’s need to maintain durable orders across patient encounters, so they would not need to be re-written every time a patient changed care setting or location. VA requested changes to the Cerner EHR system to allow it to continue to follow its current process for documenting life-sustaining treatments, but according to DOD officials, the proposed changes did not align with DOD’s position on such treatments, specifically resuscitation statuses. After multiple discussions between the VA and DOD clinicians, the two departments plan to adopt an interim solution. According to VA and DOD officials, VA and DOD’s joint decision-making body, the Functional Decision Group, has met weekly to address coordination issues since early 2019. These officials said that the joint Functional Decision Group determined whether it could make a decision, or whether additional information was needed and a team should be established to work on dispute resolution between the departments. VA EHRM program officials said that the coordination procedures for the joint Functional Decision Group would be formalized and that the roles and responsibilities for coordination between VA and DOD would be clearly defined, in response to a recommendation we made in a previous report. Specifically, VA and DOD have developed a charter for the joint Functional Decision Group, which was signed in April 2020. According to EHR council participants, VA and DOD had been developing their coordination procedures as system configuration decisions were made, and decisions that required input from both departments may not have been as timely as they could have been. According to EHRM program officials, the departments ultimately were able to address most decisions and coordination on remaining decisions was ongoing as of March 2020. VA’s EHRM Program Largely Met EHR Council Charter Goals for Representation VA generally included a wide range of stakeholders in its 18 EHR councils. Specifically, VA was largely in line with its EHR councils’ charter goals to include about 60 percent of council members from the field, with the remainder from the central office, and to have representatives from a range of geographic locations and with sufficient experience and expertise: VA data show that EHR councils had about 58 percent (607 of 1,039) of its members representing the field and about 40 percent (415 of 1,039) representing VA’s central office, roughly in line with VA’s goals. The councils included participants from a variety of geographic regions, including each of its 18 VISNs, with the most participants representing VISN 20, which oversees the two medical facilities where the new EHR system is scheduled to be initially implemented. Participants primarily represented the most complex level of VA medical facilities. Specifically, VA data show that about 83 percent (861 of 1,039) of participants represented level 1 VA medical facilities, whereas about 3 percent (33 of 1,039) and 7 percent (75 of 1,039) represented medium (level 2) and low (level 3) complexity VA medical facilities, respectively. EHRM program officials said that the majority of participants represented higher-complexity facilities because participants were drawn from national experts and published authors, and often performed VA-specific processes. Furthermore, smaller medical centers had fewer resources so clinicians were more likely to be needed to continue providing patient care at those facilities and less likely to be available to serve on councils. According to a voluntary questionnaire VA asked council participants to complete, about 37 percent of the 304 participants who completed the survey had at least 6 years of experience at VA; 29 percent had at least 16 years of experience; and, 19 percent had more than 25 years of experience. In addition to participants from the VA, we observed that EHR council national workshop meetings included participants from outside of the department—such as clinicians from DOD sites and commercial health care systems that had already implemented Cerner’s EHR system. These participants provided support for discussions and insight into industry best practices. While the EHR councils included a wide range of participants, in September and October 2019, council participants from both of the initial operating capability sites raised concerns that the councils did not include adequate representation from specialty areas at national workshop meetings. Specifically, these officials said that an insufficient number of specialty physicians, including pulmonologists and gastroenterologists, were included. In addition, VA’s summary from the last workshop, national workshop 8, observed that additional subject matter experts representing medical specialties should be included in the EHR system configuration decision process to enhance collaboration and decision- making. EHRM program officials, including the Chief Medical Officer and Ambulatory Council chairs, said they had not included certain specialists and scheduled workshops on specialty areas, such as pulmonology and gastroenterology as they decided to focus first on more foundational decisions, such as those for primary care. Starting in November 2019, following the completion of the eight national workshops, VA EHR councils continued to meet, as necessary, to complete capability set 1 and 2 configuration decisions and had begun to include clinicians from specialty areas in these meetings. VA plans to continue these meetings through September 2020. VA’s approach of including clinicians from specialty areas in ongoing configuration decision meetings is generally consistent with our leading collaboration practice that agencies should ensure that all relevant participants be included in any collaborative effort they undertake. By including relevant participants, the program increases the likelihood that it has considered input from participants with unique knowledge, skills, and abilities. Further, including relevant participants increases the likelihood that when implemented, the EHR system will be properly configured to meet the needs of clinicians, and effectively support their efforts to deliver care. VA’s EHRM Program Did Not Always Include Key Stakeholders at Its Local Workshops Local workshops at the Mann-Grandstaff VA Medical Center and VA Puget Sound Health Care System did not always include representation from relevant stakeholders, including facility clinicians and staff. Specifically, multiple participants in the local workshop meetings, including clinicians and department leads, at these facilities said that VA’s EHRM program did not always effectively communicate information about local workshop meetings to facility clinicians and staff to facilitate the designation of staff to participate and ensure relevant representation at local workshops. Local workshop participants stated that they did not always know which local workshop meetings they needed to attend, because they did not receive adequate information about the session topics. This is inconsistent with key collaboration practices identified in our prior work to ensure that relevant participants be included in any collaborative effort and that participating entities have agreed on common terminology. Furthermore, standards for internal control in the federal government call for effective communication and information sharing. Local workshop participants, including clinicians and department leads from medical facilities said that differences in the use of terminology between VA and Cerner sometimes made it challenging to identify the clinicians and staff that should attend local workshop meetings. For example, some officials reported that they did not believe that a meeting on “charge services” would be relevant to their work given that VA does not typically bill veterans for services. However, they later learned that the meeting actually covered topics beyond billing, such as capturing workload data that was relevant to their work. Because Cerner and VA did not always effectively communicate regarding workshop content for local workshops, local workshops did not always include all relevant stakeholders. As previously stated, VA plans to hold local workshops in advance of the Cerner EHR system implementation at future VA medical facilities. However, VA has not indicated how it will improve the ways in which it describes the topics of these workshops, including providing sufficient detail and defining key terms. If VA improves communication on workshop meeting topics, the EHRM program can increase the likelihood that it will obtain appropriate input from facility clinicians and staff at local workshops to consider in design decisions for the implementation of the EHR system. VA met its schedule for making the needed system configuration decisions that would enable the department to implement its new EHR system at the first VA medical facility, which was planned for July 2020. In addition, VA has formulated a schedule for making the remaining EHR system configuration decisions before implementing the system at additional facilities planned for fall 2020. VA’s EHRM program was generally effective in establishing decision- making procedures that were consistent with applicable federal standards for internal control. However, VA did not always ensure the involvement of relevant stakeholders, including medical facility clinicians and staff, in the system configuration decisions. Specifically, VA did not always clarify terminology and include adequate detail in descriptions of local workshop sessions to medical facility clinicians and staff to ensure relevant representation at local workshop meetings. Participation of such stakeholders is critical to ensuring that the EHR system is configured to meet the needs of clinicians and support the delivery of clinical care. We are making the following recommendation to VA: For implementation of the EHR system at future VA medical facilities, we recommend that the Secretary of VA direct the EHRM Executive Director to clarify terminology and include adequate detail in descriptions of local workshop sessions to facilitate the participation of all relevant stakeholders including medical facility clinicians and staff. (Recommendation 1) We provided a draft of this report to VA and DOD for comment. In its comments, reproduced in appendix II, VA concurred with our recommendation and described steps that it planned to take to address it. Specifically, VA noted that it planned and designed its workshops to enable collaboration between clinical and administrative experts and end- users so that the EHR system is designed, validated, and configured to promote interoperability and quality care for veterans. VA stated that it is further refining local workshop agendas and descriptions to facilitate VA subject matter expert identification and participation. VA also provided technical comments on the report, which we incorporated as appropriate. DOD provided technical comments on the report, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretaries of VA and DOD, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact Debra A. Draper at (202) 512-7114 or DraperD@gao.gov or Carol C. Harris at (202) 512-4456 or HarrisCC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In addition to the individuals named above, Mark Bird (Assistant Director), Michael Zose (Assistant Director), Merry Woo (Analyst-in-Charge), Bianca Eugene, and Paris Hawkins made key contributions to this report. Also contributing were Jennie F. Apter, Giselle Hicks, Monica Perez- Nelson, and Ethiene Salgado-Rodriguez.", "summary": "VA's existing EHR system is antiquated, costly to maintain, and does not fully support VA's need to exchange health records with other organizations, such as the Department of Defense. As a result, VA has undertaken a modernization effort to replace it. As VA prepares to transition from its existing EHR system to a commercial system, it has the opportunity to design standardized work processes to support the delivery of care and ensure information on veterans' care is consistently captured, regardless of site of care. GAO was asked to review VA's EHR system configuration process. This report examines, among other objectives: (1) how VA made EHR system configuration decisions and assessed the compatibility of the commercial EHR system with its work processes; and (2) the effectiveness of VA's decision-making procedures, including ensuring key stakeholder involvement. GAO observed national and local workshop meetings; visited planned initial implementation sites; reviewed documentation on the processes and schedule; and interviewed VA, DOD, and contractor officials. The Department of Veterans Affairs (VA) used a multi-step process to help ensure that its future commercial electronic health record (EHR) system is configured appropriately for, and is compatible with, its clinical work processes. To configure the EHR system, which VA planned to implement initially at the Mann-Grandstaff VA Medical Center, in Spokane, Washington, in July 2020, and at the Puget Sound Health Care System in the fall of 2020, VA established 18 EHR councils comprising VA clinicians, staff, and other experts in various clinical areas and held eight national workshops between November 2018 and October 2019. At these workshops, the councils decided how to design the functionality of the EHR software to help clinicians and other staff deliver care and complete tasks such as administering medication. VA also held eight local workshops at both medical centers to help ensure that the EHR configuration supported local practices. As of March 2020, the EHR councils were continuing to meet to complete configuration decisions. Furthermore, VA plans to hold local workshops in advance of the EHR system implementation at future VA medical facilities. In April 2020, the VA Secretary announced that the department had shifted priorities to focus on caring for veterans in response to the pandemic created by COVID-19. According to program officials, at that time, they paused the implementation of the EHR system and were assessing the impact of the COVID-19 pandemic on VA's planned implementation schedule. GAO found that VA's decision-making procedures were generally effective as demonstrated by adherence to applicable federal internal control standards for establishing structure, responsibility, and authority, and communicating internally and externally, but that VA did not always ensure key stakeholder involvement. Specifically, the councils included a wide range of stakeholders from various geographic regions. However, according to clinicians from the two initial medical facilities for implementation, VA did not always effectively communicate information to stakeholders, including medical facility clinicians and staff to ensure relevant representation at local workshop meetings. As a result, local workshops did not always include all relevant stakeholders. VA has not indicated how it plans to describe these future sessions and define key terms to ensure key stakeholder participation in local workshops. By ensuring that all relevant stakeholders are included, VA will increase the likelihood that it is obtaining input from a wide range of clinicians and staff who will use the EHR system and will increase the likelihood that when it is implemented, the EHR system will effectively support the delivery of care at VA medical centers. GAO is recommending that VA ensure the involvement of all relevant medical facility stakeholders in the EHR system configuration decision process. VA concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "Dual-eligible beneficiaries qualify for both Medicare and Medicaid, and may enroll in and receive benefits covered by each program. Individuals ages 65 or older can qualify for Medicare based on age, and individuals ages 18 to 64 can qualify for Medicare based on disability. Medicaid eligibility varies by state, but beneficiaries may qualify based on having a low level of income, a need for nursing home care, high medical expenses, or other criteria. For dual-eligible beneficiaries, Medicare is the primary payer for any benefits covered by both programs. As a result, Medicare is the primary payer for acute and post-acute care, such as physician services, hospitalizations, prescription drugs, and skilled nursing facility care. For many dual-eligible beneficiaries, Medicaid covers benefits not covered by Medicare. This includes long-term services and supports, which may include nursing home care, personal care services, or adult day care. Whether Medicaid covers these benefits varies between the two main categories of dual-eligible beneficiaries. Those in the first category are known as full-benefit, dual-eligible beneficiaries, because they may receive all Medicaid benefits, in addition to Medicare benefits. Medicaid also pays for their Medicare premiums and, in some cases, the cost- sharing for their Medicare benefits. Those in the second category are known as partial-benefit, dual-eligible beneficiaries, because Medicaid assistance is limited to payment of their Medicare premiums and, in some cases, the cost-sharing for their Medicare benefits. Partial-benefit, dual- eligible beneficiaries have limited income and assets, but their income and assets are not low enough to qualify them for full Medicaid benefits in their state. For Medicare, dual-eligible beneficiaries can choose to receive their Medicare services from either traditional Medicare or from MA plans. These options differ in key ways. For example, traditional Medicare may have a more extensive provider network than MA plans. However, MA plans may cover additional benefits, such as vision or dental care, which are generally not covered under traditional Medicare. If dual-eligible beneficiaries choose to enroll in MA plans, they may also have the choice between regular MA plans and D-SNPs, which offer certain services targeted at the needs of dual-eligible beneficiaries. For example, D-SNPs are required to perform health risk assessments, create individualized care plans, and provide an interdisciplinary care team for each beneficiary enrolled. They may also cover transportation services, home modifications, or other specialized services that are more likely to be used by dual-eligible beneficiaries. For Medicaid, states may allow or require Medicaid beneficiaries, including dual-eligible beneficiaries, to receive their Medicaid benefits through an MCO. In this managed care model, Medicaid MCOs are responsible for arranging for and paying providers’ claims for a specific set of Medicaid benefits provided to beneficiaries. More recently, some states have created new Medicaid managed care programs or expanded the benefits covered by existing Medicaid managed care programs in order to include additional populations previously covered through Medicaid fee-for-service. The new populations include seniors, persons with disabilities, and those who need long-term services and supports— many of whom may be dually eligible. A dual-eligible beneficiary may be able to enroll in a D-SNP and Medicaid MCO that are offered by the same or related companies, an arrangement known as aligned enrollment. In states with MLTSS, aligned enrollment means the same or related companies provide a beneficiary’s Medicare benefits, such as primary and acute care, through a D-SNP and Medicaid benefits, such as long-term services and supports, through a Medicaid MCO. State Medicaid agencies enter into contracts with both D-SNPs and Medicaid MCOs, and these contracts may include provisions to facilitate and encourage aligned enrollment. Since January 2013, all D-SNPs have been required to have an executed contract with the Medicaid agency in each state in which it operates. A state can enter into contracts with all, some, or none of the D-SNPs seeking to operate in the state, and any D- SNPs that the state declines to contract with cannot operate in the state. Each year, CMS reviews D-SNPs’ contracts with states to ensure that they include eight required elements, including the D-SNP’s responsibility for providing or arranging the provision of Medicaid benefits, among other things. According to CMS officials, in these reviews, CMS does not collect information regarding whether states are imposing requirements pertaining to aligned enrollment. States also have contracts with Medicaid MCOs, which can include requirements that could facilitate or encourage aligned enrollment. As shown in table 1, CMS’s Integrated Care Resource Center has identified five types of approaches that states can use to encourage aligned enrollment. For example, states can manage which D-SNPs operate in the state, such as only allowing D-SNPs with an aligned Medicaid MCO (that is, a MCO offered by the same company or a related company). This gives dual-eligible beneficiaries greater options for choosing aligned enrollment. As another example, states can allow the automatic assignment of certain dual-eligible beneficiaries to a D-SNP aligned with a Medicaid MCO, a process known as default enrollment. Default enrollment, which requires CMS approval, can directly increase the number of dual-eligible beneficiaries with aligned enrollment. In addition to D-SNPs with aligned enrollment, two other types of Medicare plans—Medicare-Medicaid plans and Program of All-Inclusive Care for the Elderly plans—exclusively or primarily serve dual-eligible beneficiaries and are responsible for both Medicare and Medicaid benefits. These three types of Medicare plans jointly served approximately 818,000 dual-eligible beneficiaries as of January 2019. Aligned enrollment in D-SNPs: As of January 2019, approximately 386,000 dual-eligible beneficiaries enrolled in D-SNPs had aligned enrollment, according to a report by the Medicare Payment Advisory Commission. This includes beneficiaries in a subset of D-SNPs that have been designated as fully integrated D-SNPs, which must meet additional specific requirements. For example, they must provide both Medicare and Medicaid benefits through a single managed care plan. In addition, the Medicaid benefits provided by the fully integrated D- SNPs must include long-term services and supports. Medicare-Medicaid plans: As of January 2019, approximately 388,000 dual-eligible beneficiaries in nine states were enrolled in these types of plans. These plans, which were established through CMS’s Financial Alignment Initiative, provide all Medicare benefits and all or almost all Medicaid benefits, and have some administrative processes that have been combined. In April 2019, CMS sent a letter to state Medicaid directors inviting additional states to express interest in the use of Medicare-Medicaid plans. Program of All-Inclusive Care for the Elderly plans: As of January 2019, approximately 44,000 beneficiaries in 31 states were enrolled in these types of plans. Most, but not all, are full-benefit, dual-eligible beneficiaries, and they are ages 55 or older and need the level of care provided in a nursing home. The plans are provider-sponsored and provide all Medicare and Medicaid benefits. In addition, each plan is required to have a physical site to provide adult day services. As of July 2019, of the 19 states with MLTSS and where aligned enrollment of dual-eligible beneficiaries in D-SNPs is possible, 16 have implemented at least one of the five approaches to encourage aligned enrollment identified by CMS’s Integrated Care Resource Center. (See fig. 1.) Of those 16 states, 11 managed which D-SNPs operate in the state, which is the foundation for promoting aligned enrollment, according to officials from the Integrated Care Resource Center. Of our seven selected states, all of them had implemented at least one of the five approaches to encourage aligned enrollment in 2019. The three most common approaches among our selected states were (1) managing which D-SNPs operate in the state; (2) limiting D-SNP enrollment to full- benefit, dual-eligible beneficiaries; and (3) encouraging D-SNP marketing to better support informed beneficiary decision-making. The details of the approaches implemented in each state varied widely. Managing which D-SNPs operate in the state. Five of the seven selected states (Arizona, New Jersey, Pennsylvania, Tennessee, and Virginia) managed which D-SNPs operated in 2019, but they varied in how they implemented this approach. For example, when Virginia established its Medicaid MLTSS program in 2017, only one D-SNP operated in the state, and Virginia required the companies with Medicaid MLTSS contracts to also start offering D-SNPs within 3 years. In contrast, when Pennsylvania and Tennessee implemented this approach, multiple D-SNPs already operated in each state. Pennsylvania and Tennessee required new D-SNPs to have aligned Medicaid MCOs, but allowed existing D-SNPs to continue operating. As a result, beneficiaries had the choice between D-SNPs that had aligned Medicaid MCOs and D-SNPs that did not have aligned Medicaid MCOs. Medicaid officials in these two states told us they chose not to cancel existing D-SNPs that did not have aligned Medicaid MCOs, as doing so could have disrupted beneficiary- provider relationships. As a result of the selected states’ differing approaches to managing which D-SNPs operated, the proportion of aligned to unaligned D-SNPs in each state varied. (See fig. 2.) Limiting D-SNP enrollment to full-benefit, dual-eligible beneficiaries. Five of the selected states (Arizona, Kansas, New Jersey, Pennsylvania, and Virginia) limited D-SNP enrollment in some or all of their D-SNPs to full-benefit, dual-eligible beneficiaries in 2019. In particular, Arizona and New Jersey Medicaid officials said that limiting D-SNP enrollment to full- benefit, dual-eligible beneficiaries allowed D-SNPs to provide a more straightforward benefit package. In turn, this can be more easily described in D-SNP materials and communications, which may help beneficiaries to make more informed decisions around aligned enrollment. Encouraging D-SNP marketing to better support informed beneficiary decision-making. Five of the selected states (Arizona, New Jersey, Pennsylvania, Tennessee, and Virginia) took steps to encourage D-SNP marketing to support informed beneficiary decision-making in 2019. For example, Arizona and Pennsylvania encouraged D-SNPs to directly market themselves to beneficiaries in the D-SNP’s aligned Medicaid MCO, in order to promote aligned enrollment. In addition, New Jersey Medicaid officials told us they review D-SNP marketing and work directly with D-SNPs to develop standard marketing language. In particular, the officials said some D-SNPs had marketed themselves as offering certain extra benefits, but those benefits were already a standard part of the state’s Medicaid package. The officials said they worked with the D-SNPs to correct the marketing, and they also developed standard language for marketing in the state. This can help reduce beneficiary confusion when making enrollment decisions. Automatically assigning certain beneficiaries to plans with aligned enrollment. Four selected states (Arizona, Florida, Pennsylvania, and Tennessee) allowed automatic assignment of certain beneficiaries to plans with aligned enrollment in 2019. For example, Arizona, Pennsylvania, and Tennessee allowed default enrollment by which certain Medicaid beneficiaries were automatically assigned to aligned D- SNPs. Under federal rules, beneficiaries have the opportunity to opt out prior to being default enrolled and select a different source of Medicare coverage; they also have the opportunity to disenroll within the first 90 days after default enrollment and select a different source of Medicare coverage. In addition, Florida and Pennsylvania automatically assigned certain dual- eligible beneficiaries to aligned Medicaid MCOs. For example, Florida law requires the state Medicaid agency to automatically assign certain D-SNP enrollees to aligned MLTSS plans when beneficiaries become eligible for long-term services and supports and have not voluntarily chosen an MLTSS plan. Engaging counselors to assist beneficiaries with aligned enrollment decisions. Two of the seven selected states (Arizona and Pennsylvania) engaged enrollment counselors to encourage aligned enrollment in 2019. For example, Arizona’s state Medicaid office works with the state’s Aging and Disability Resource Center and State Health Insurance Assistance Program counselors to increase beneficiary understanding of aligned enrollment and options to enroll in aligned plans. In 2019, Pennsylvania’s contracts with D-SNPs required collaboration between the D-SNPs and the state’s independent enrollment broker that assists beneficiaries with Medicaid enrollment. In addition to there being variation in the selected states’ use of approaches to encourage aligned enrollment, the proportion of D-SNP enrollees with aligned enrollment varied from 20 percent in Pennsylvania to 100 percent in New Jersey among the selected states that were able to provide data for 2019. (See fig. 3.) There can be multiple reasons for the varied levels of aligned enrollment between D-SNPs and MLTSS. For example, Arizona recently entered into new Medicaid MCO contracts, and this resulted in changes to the parts of the state served by each Medicaid MCO. According to state Medicaid officials, these new contracts somewhat reduced the extent of aligned enrollment. Medicaid officials in the seven selected states described various challenges with aligned enrollment. The most common challenge mentioned was difficulty using D-SNP data to implement and evaluate aligned enrollment policies. Medicaid officials in the selected states told us many of these challenges require ongoing monitoring and collaboration with CMS and the companies offering D-SNPs. Difficulty using data to implement and evaluate aligned enrollment. Medicaid officials in six of the selected states (Florida, Kansas, New Jersey, Pennsylvania, Tennessee, and Virginia) told us that using D-SNP and Medicare data to implement and evaluate aligned enrollment policies can be difficult. For example, Tennessee Medicaid officials told us that getting the data from CMS needed for default enrollment was a challenge. In particular, they said that, when the state was first starting to implement default enrollment, they had challenges with getting data from CMS in a timely fashion to identify which Medicaid beneficiaries were about to become dually eligible for Medicare, particularly those with eligibility due to disability. This meant that the state could not provide D-SNPs with the information needed by the D-SNPs to send notices to those beneficiaries in the required time frame. CMS officials also acknowledged that its data do not always identify individuals becoming eligible for Medicare early enough for D-SNPs to send notices in the required time frame. Tennessee Medicaid officials told us that CMS has worked with the state on this issue and it has now become easier for the state to receive the needed data. Furthermore, CMS and its Integrated Care Resource Center have also developed materials and, according to CMS officials, provided ongoing technical assistance for states on accessing data for default enrollment and other aspects of implementation of aligned enrollment. Medicaid officials in Virginia and New Jersey described related challenges with using D-SNP data to determine whether their policies work. Virginia Medicaid officials told us that it can be difficult to evaluate the health benefits of aligned enrollment, because data on quality measures can span multiple states. Specifically, one of the state’s D- SNPs operates in multiple states and therefore reports health outcome data to CMS for its entire service area. Virginia Medicaid officials told us they are not able to separate data for Virginia residents from those of other states. As a result, they said they currently cannot determine the effect of their aligned enrollment policies, and they plan to require the D- SNP to report Virginia-specific quality data in the future. New Jersey Medicaid officials described a challenge with receiving the relevant data to evaluate health outcomes for dual-eligible beneficiaries with aligned enrollment. The state has CMS approval to receive Medicare data directly from CMS. However, as of November 2019, the state’s data vendor was not in compliance with federal Medicare data security requirements for storing certain data, which meant that the state could not accept the Medicare data. The Bipartisan Budget Act of 2018 encourages CMS to require reporting of MA quality measures, including D-SNP quality measures, at the plan level. However, CMS has identified several challenges to developing such a requirement. One challenge CMS has identified is that about two- thirds to three-quarters of D-SNPs would not have reliable ratings, for example, because those plans had too few participants in the survey. Another challenge CMS has identified is the additional complexity and administrative burden for plans completing this reporting. As of December 2019, CMS officials told us they are continuing to work to determine the best reporting level for each quality measure. They also plan to collect additional feedback from stakeholders and a technical expert panel. Difficulties with information dual-eligible beneficiaries receive about Medicare enrollment choices. Medicaid officials in five of the selected states (Kansas, New Jersey, Pennsylvania, Tennessee, and Virginia) told us they have experienced challenges in ensuring that beneficiaries receive quality information about their Medicare enrollment choices. For example, in 2019, Pennsylvania’s contracts with D-SNPs required collaboration between the D-SNPs and the state’s independent enrollment broker that assists beneficiaries with Medicaid enrollment. However, Pennsylvania Medicaid officials told us the state’s independent enrollment broker did not have the capacity to provide this type of assistance in addition to its primary responsibility of assisting beneficiaries with Medicaid enrollment. As another example, Virginia Medicaid officials told us they have faced challenges using state D-SNP contracts to regulate D-SNP marketing. They told us that certain provisions in the state’s contracts with D-SNPs were intended to regulate the extent of D-SNP marketing in 2019. In particular, each D-SNP was supposed to only market to beneficiaries enrolled in that D-SNP’s aligned Medicaid MCO, which was intended to increase the extent of aligned enrollment in the state. However, state Medicaid officials told us that D-SNPs had different interpretations of the contract provisions, and one D-SNP had billboards and television advertisements available to the general public. Due to the difficulty of enforcement, among other reasons, Virginia Medicaid officials told us they chose to not include these provisions in the D-SNP contracts for 2020. Through the Integrated Care Resource Center, CMS has developed materials describing how states can regulate D-SNP marketing in their contracts with D-SNPs, and the agency reviews and may disapprove D- SNP marketing materials that do not follow federal requirements. CMS officials also told us they make themselves available to states to explain how to include marketing restrictions in the contracts that states have with D-SNPs. Limits of staff knowledge. Medicaid officials in four of the selected states (Florida, Kansas, New Jersey, and Pennsylvania) told us that limited staff knowledge of Medicare presents a challenge. For example, Medicaid officials in Kansas told us only one or two staff in the state’s Medicaid agency are knowledgeable about Medicare and would have the knowledge to implement aligned enrollment approaches. Similarly, Medicaid officials in Florida said they only recently learned about one of the approaches for encouraging aligned enrollment, which is that the state can decline to contract with certain D-SNPs. In addition, New Jersey and Pennsylvania Medicaid officials told us staff knowledge of Medicare is limited and that they would like to increase their level of knowledge as they continue to foster aligned enrollment. Competition from look-alike MA plans targeted to dual-eligible beneficiaries. Medicaid officials in four of our selected states (Arizona, Pennsylvania, Tennessee, and Virginia) identified certain MA plans that are so-called “look-alike” plans to the D-SNPs, which create a potential challenge to fostering aligned enrollment. According to CMS, look-alike plans are MA plans that are designed for and marketed exclusively to dual-eligible beneficiaries, but that are not D-SNPs. Therefore, look-alike plans do not need a contract with the state to operate and do not have to comply with state approaches that foster aligned enrollment. Medicaid officials from our selected states and the Medicare Payment Advisory Commission gave examples of the impact of look-alike plans. For example, Tennessee Medicaid officials told us that dual-eligible beneficiaries in look-alike plans do not receive care coordination between Medicare and Medicaid, in contrast with dual-eligible beneficiaries in D- SNPs, which are required to provide such coordination. In addition, Arizona Medicaid officials told us that look-alike plans have affected levels of aligned enrollment in the state. Similarly, according to the Medicare Payment Advisory Commission, look-alike plans can undermine states’ efforts to develop D-SNPs that integrate Medicare and Medicaid by encouraging dual-eligible beneficiaries to instead enroll in look-alike plans. CMS has also identified look-alike plans as a challenge and is considering some steps in response. In 2018, CMS revised its marketing guidelines to prohibit look-alike plans from marketing themselves as designed for dual-eligible beneficiaries and as having a relationship with the state Medicaid agency. In its April 2019 policy update for MA plans, CMS said that look-alike plans enable companies to offer plans that circumvent state and federal requirements for D-SNPs, which undermines efforts to improve the quality of care. In February 2020, CMS published a proposed rule that, if finalized, would prohibit the offering of MA plans whose enrollment of dual-eligible beneficiaries exceeds specific projected or actual enrollment thresholds in states with a D-SNP. According to CMS, this would prevent look-alikes from undermining the statutory and regulatory framework for D-SNPs. Extent of overlapping provider networks. Medicaid officials in two of our selected states (Pennsylvania and Tennessee) reported challenges with aligned D-SNPs and Medicaid MCOs that do not have completely overlapping networks of relevant providers. That is, even though the D- SNP and Medicaid MCO are offered by the same or related companies, certain providers may be in only the D-SNP network or only the Medicaid MCO network—but not both. For example, representatives from a beneficiary group in Pennsylvania told us that a dual-eligible beneficiary’s provider may be in the Medicaid MCO network, but not the D-SNP network. This can disrupt that beneficiary’s continuity of care if he or she is default enrolled into the D-SNP. There are no requirements for the state or D-SNP to ensure that a beneficiary’s primary care provider is in the D-SNP into which he or she is default enrolled. CMS’s model for the notice sent to beneficiaries identified for default enrollment suggests (but does not require) that the D-SNP include information on whether or not the beneficiary’s primary care provider is in the D-SNP’s network. CMS officials said they did not know of any complaints the agency has received on the issue. They also said they have not analyzed how the provider network of a D-SNP compares to the provider network of its aligned Medicaid MCO. Furthermore, in the preamble to the default enrollment final rule issued in April 2018, CMS said that it did not include any criteria related to provider networks, but that network adequacy requirements would apply and states can use their contracts with D-SNPs to create requirements for continuity of care. One state that does this is Tennessee, which specifically requires D-SNPs to develop provider networks that have substantial overlap with the provider network of their aligned Medicaid MCOs. The state also requires D-SNPs to ensure continuity of care for beneficiaries who have been default enrolled. For example, Tennessee Medicaid officials said that if a beneficiary who has been default enrolled has a long-standing primary care provider with the D-SNP’s aligned Medicaid MCO, the state requires the D-SNP to continue covering services by that provider for at least 30 days and to attempt to contract with the provider. CMS has assisted states with aligned enrollment. In particular, CMS has provided technical assistance to states on implementing the various approaches that encourage aligned enrollment. One way that CMS has done this is through its Integrated Care Resource Center, which has developed materials on how states can use their contracts with D-SNPs to align enrollment and promote integration. The Integrated Care Resource Center has also facilitated peer-to-peer assistance between states. For example, Integrated Care Resource Center officials said they facilitated conversations and assistance between state Medicaid officials in New Jersey and Pennsylvania on D-SNP marketing. Medicaid officials in six of our selected states said they had utilized CMS’s technical assistance, and they had overall positive views of CMS’s assistance. CMS reviews some aspects of the contracts between states and D-SNPs, including checking that the contracts include the eight required elements. According to CMS officials, in these reviews, CMS does not collect information regarding whether states are imposing requirements pertaining to aligned enrollment. CMS’s program audits of MA plans similarly do not include reviews of such state requirements pertaining to aligned enrollment. CMS has a direct role with one aspect of aligned enrollment: default enrollment. In particular, CMS approves D-SNPs to receive beneficiaries through default enrollment, and it processes the enrollment transactions of beneficiaries being default enrolled. D-SNPs’ approval for default enrollment: Before a D-SNP can receive beneficiaries through default enrollment, it must submit a proposal to CMS for approval. CMS reviews the D-SNP’s proposal and checks that the D-SNP meets an established list of requirements outlined in regulation. Among other requirements, the D-SNP must demonstrate it has the state’s support for default enrollment and that the required elements have been included in its template for the notice that is sent to beneficiaries identified for default enrollment. CMS also checks that the D-SNP is not facing any CMS enrollment sanctions and that the D-SNP has a quality rating of three or more stars. CMS grants approval for up to 5 years if it determines the D- SNP meets these requirements. Default enrollment transactions: CMS processes the enrollment transactions of dual-eligible beneficiaries being default enrolled, and it tracks these transactions in a monthly report. The monthly report lists the total number of beneficiaries identified for default enrollment for each applicable D-SNP, and the report lists numbers for certain subsets of beneficiaries who were ultimately not default enrolled. These subsets include beneficiaries who opted out prior to being default enrolled and beneficiaries whose default enrollment was not allowed by CMS for various reasons. Despite its direct role in default enrollment, CMS lacks quality information on the experiences of dual-eligible beneficiaries after they are default enrolled. This is inconsistent with federal internal control standards on information and communication, which state that management should use quality information to achieve the agency’s objectives. In particular, the monthly reports on enrollment transactions do not include data on the extent to which dual-eligible beneficiaries choose to disenroll after being default enrolled. Although the reports include data on the number of beneficiaries who opt out prior to being default enrolled (which CMS officials said was low), they do not include data on beneficiaries who choose to disenroll in the first 90 days after being default enrolled. This 90-day time frame for disenrollment is specified by federal regulation, and beneficiaries may choose to disenroll for various reasons. For example, one reason for disenrollment given by one beneficiary group we interviewed is that some beneficiaries may not realize they have been default enrolled into a D-SNP until they next see their provider, and that provider may not be in the D-SNP’s provider network. They said that beneficiaries may not have seen the notice or other information about being default enrolled, or they may not have understood the information. In addition, CMS cannot systematically review beneficiary complaints for trends or concerns related to default enrollment. Dual-eligible beneficiaries, like other Medicare beneficiaries, can submit complaints to CMS. These complaints are entered in the agency’s complaint tracking module, and D-SNP account managers, like other MA plan account managers, are responsible for monitoring complaints. CMS officials said that the D-SNP account managers have not identified any trends or concerns about default enrollment. However, CMS officials said default enrollment is not tracked as a distinct category in the complaint tracking module, and the guidance on monitoring complaints that is provided to the D-SNP account managers does not direct them to look for issues explicitly related to default enrollment. Quality information on the experiences of dual-eligible beneficiaries after they are default enrolled would allow CMS to better identify the extent to which these beneficiaries face challenges as a result of default enrollment and to determine how, if at all, to address the challenges. Future studies may provide CMS with additional information on beneficiaries in D-SNPs with aligned enrollment, but that information will not be available until 2022 or later. In particular, federal law directs the Medicare Payment Advisory Commission, in consultation with the Medicaid and CHIP Payment and Access Commission, to compare the quality of the different types of D-SNPs, including those with aligned enrollment, as well as comparing them to other types of plans. The commission is to develop an initial report by 2022 with subsequent reports afterward. Better care for dual-eligible beneficiaries is one of CMS’s strategic initiatives, and the agency has supported states’ decisions to encourage aligned enrollment in order to encourage better coordination of care. However, CMS lacks quality information on the experiences of beneficiaries who have aligned enrollment as the result of the use of default enrollment. For example, CMS’s monthly reports on default enrollment do not include data on beneficiaries who choose to disenroll after being default enrolled. CMS lacks this information even though selected states and others have reported challenges that could affect the care received by those beneficiaries. Quality information on the experiences of these dual-eligible beneficiaries would allow CMS to better identify the extent to which beneficiaries are facing challenges as a result of default enrollment and to determine how, if at all, to address those challenges. We are making the following recommendation to CMS: The Administrator of CMS should take steps to obtain quality information on the experiences of dual-eligible beneficiaries who have been default enrolled into D-SNPs, such as by obtaining information about the extent to which and reasons that beneficiaries disenroll from a D-SNP after being default enrolled. (Recommendation 1) We provided a draft of this report to the Department of Health and Human Services (HHS) for comment. In its comments, reproduced in appendix I, HHS concurred with our recommendation. HHS stated that it is committed to increasing the number of dual-eligible beneficiaries in integrated care and that it supports states with these efforts, such as the use of aligned enrollment. HHS also said that it has not identified any trends or areas of concern in its monitoring of beneficiaries who opted out prior to being default enrolled. In response to our recommendation, HHS stated it will evaluate opportunities to obtain more information on dual-eligible beneficiaries who disenroll from a D-SNP after being default enrolled. HHS also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Administrator of the Centers for Medicare & Medicaid Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Martin T. Gahart (Assistant Director), Corissa Kiyan-Fukumoto (Analyst-in-Charge), Jason Coates, Kelly Krinn, Virginia Lefever, Drew Long, Jennifer Rudisill, and Ethiene Salgado-Rodriguez made key contributions to this report. Medicare and Medicaid: Additional Oversight Needed of CMS’s Demonstration to Coordinate the Care of Dual-Eligible Beneficiaries. GAO-16-31. Washington, D.C.: December 18, 2015. Disabled Dual-Eligible Beneficiaries: Integration of Medicare and Medicaid Benefits May Not Lead to Expected Medicare Savings. GAO-14-523. Washington, D.C.: August 29, 2014. Medicare and Medicaid: Consumer Protection Requirements Affecting Dual-Eligible Beneficiaries Vary across Programs, Payment Systems, and States. GAO-13-100. Washington, D.C.: December 5, 2012. Medicare Special Needs Plans: CMS Should Improve Information Available about Dual-Eligible Plans’ Performance. GAO-12-864. Washington, D.C.: September 13, 2012. Medicare and Medicaid: Implementing State Demonstrations for Dual Eligibles Has Proven Challenging. GAO/HEHS-00-94. Washington, D.C.: August 18, 2000.", "summary": "Congress authorized the establishment of D-SNPs in 2003 to address the unique needs of dual-eligible beneficiaries. For example, D-SNPs are required to provide certain specialized services targeted at the needs of dual-eligible beneficiaries, such as health risk assessments. D-SNPs must have approval of state Medicaid agencies to operate, and states can require D-SNPs to coordinate with Medicaid. Congress included a provision in statute for GAO to review D-SNPs’ integration with state Medicaid programs. This report, among other objectives, (1) describes what is known about selected states’ experiences with aligned enrollment in D-SNPs, and (2) examines CMS’s oversight of aligned enrollment. GAO reviewed relevant federal guidance and internal control standards. GAO also interviewed Medicaid officials in seven selected states and reviewed available documentation. The states (Arizona, Florida, Kansas, New Jersey, Pennsylvania, Tennessee, and Virginia) were selected, in part, for variation in experiences with aligned enrollment. GAO also interviewed officials from CMS, beneficiary groups, and companies that offered D-SNPs and Medicaid MCOs. Dual-eligible beneficiaries are Medicare beneficiaries who are also enrolled in the Medicaid program in their state. In certain states, they may receive both types of benefits through private managed care plans. As of January 2019, about 386,000 such individuals were enrolled in both a private Medicare plan known as a dual-eligible special needs plan (D-SNP) and a Medicaid managed care organization (MCO) that were offered by the same or related companies. This arrangement, known as aligned enrollment, may create opportunities for better coordination between Medicare's acute care services and Medicaid's long-term services and supports, such as nursing facility care or personal care services. Medicaid officials in seven selected states described challenges with aligned enrollment. One challenge cited by officials in six of the states was using D-SNP and Medicare data to implement and evaluate aligned enrollment. For example, officials in one state said they cannot separate D-SNP quality data for just their state, because some D-SNPs report data spanning multiple states to the Centers for Medicare & Medicaid Services (CMS). As of December 2019, CMS officials said they are determining the best way for D-SNPs to report these quality data. CMS has assisted states with aligned enrollment, but lacks quality information on the experiences of dual-eligible beneficiaries who have aligned enrollment through a process known as default enrollment. With default enrollment, states allow automatic assignment of beneficiaries who are enrolled in a Medicaid MCO and are about to become eligible for Medicare to the D-SNP aligned with that MCO. However, CMS's monthly reports on default enrollment do not include information on beneficiaries who choose to disenroll in the first 90 days after being default enrolled, a time frame specified in regulation. According to one beneficiary group, some beneficiaries may disenroll, because they did not realize they were default enrolled and their provider is not in the D-SNP's network. Quality information on the experiences of dual-eligible beneficiaries after default enrollment would allow CMS to better identify the extent to which beneficiaries face challenges and to determine how, if at all, to address the challenges. GAO recommends that CMS take steps to obtain quality information on the experiences of dual-eligible beneficiaries who have been default enrolled into D-SNPs. The Department of Health and Human Services concurred with the recommendation.", "document_type": "gao"}
{"report": "In addition to retirement benefits to older individuals and their families, SSA administers the nation’s largest disability benefit program, the Disability Insurance (DI) program. DI generally pays benefits to individuals if they are unable to work due to qualifying impairments that are expected to last at least 1 year or result in death. In fiscal year 2018, SSA paid DI benefits to more than 10 million beneficiaries each month for a total of about $144 billion that year. In addition to monthly financial benefits, which averaged about $1,234 per disabled worker in 2018, those eligible for DI also gain access to Medicare after a 2-year waiting period, which can help pay for their medical costs, including prescription opioids. Disabled workers claiming DI benefits must meet work and other requirements to be considered eligible for DI. First, they must have worked for a specified amount of time covered by Social Security as well as worked within a specified timeframe before becoming disabled, based on their age. If these work requirements are met, SSA will assess a number of medical and vocational requirements, including whether the claimant earned more than a set monthly amount, the severity of any impairments they have, and whether they are able to continue working in a similar or other capacity given their age, education, and prior work history (see fig. 1). DI claimants may also apply concurrently for SSA’s Supplemental Security Income (SSI) program, which provides income to individuals who are aged, blind, or disabled with limited income and resources. Such claimants may be deemed eligible for both programs if they meet certain income and resource requirements in addition to those for DI. Under SSI, they may receive additional financial benefits as well as access to Medicaid. Several different program staff are involved in processing DI claims. First, staff in SSA field offices receive applications and determine whether claimants meet nonmedical eligibility requirements, such as having a sufficient work history. Claims for those who meet these requirements are then forwarded to state government Disability Determination Services (DDS) offices, where DDS staff review the claimant’s eligibility based on the medical and vocational requirements outlined in figure 1 above. Specifically, DDS examiners assemble any medical and vocational information for the claim. This can involve contacting a claimant’s treatment providers, and third parties such as family members, friends, and employers, and referring the claimant for consultative exams, such as with physicians or psychologists if recent treatment records are unavailable. DDS examiners then confer with DDS medical consultants, such as in-house or contracted physicians and psychologists, to determine whether the claimant meets the law’s requirements for having a disability. DDS examiners use all of this information to decide whether claimants are eligible for DI. Claimants who are dissatisfied with the initial DDS decision have several opportunities to appeal. First, they generally may request a “reconsideration” of the claim, which is conducted by a DDS examiner who was not involved in the original decision. Next, they may request a hearing before an SSA administrative law judge, who may collect new evidence and ask other witnesses, such as medical and vocational experts, to testify at the hearing. If their claim is denied at this hearings level, claimants may request that it be reviewed by the Appeals Council, which is comprised of SSA administrative appeals judges and appeals officers. Beyond the Appeals Council, the claimant may appeal to a federal district court. Staff at each level of the process must document their decision in a claimant’s case file, in accordance with the agency’s policies. For example, staff are generally required to document the medical evidence they reviewed, any assessments regarding the claimant’s severity of impairments and vocational abilities, and the rationale for their decisions. For allowed DI claims, federal law requires beneficiaries’ cases to be periodically reviewed within specified timeframes to ensure the beneficiary continues to meet DI requirements. DDS examiners conduct such reviews, called continuing disability reviews, conferring with medical consultants and making a decision regarding a beneficiary’s disability in comparison to the evidence from when the claim was allowed to determine if medical improvement has occurred. According to SSA, benefits typically continue unless evidence exists that a beneficiary’s impairment has medically improved and that they are able to return to work. Musculoskeletal conditions, which are pain-related, make up the largest proportion of impairments allowed by SSA for DI benefits. Specifically, these conditions, such as back and joint impairments, made up nearly 33 percent of impairments for disabled workers in 2018. Treatments for pain-related symptoms can include prescription opioids. Nationwide data show that trends in the numbers of opioid prescriptions and DI claims have followed a similar pattern, with both peaking between 2010 and 2014 and then declining. From 2006 through 2017, total opioid prescriptions peaked at about 255 million prescriptions in 2012, and then decreased in each of the following years (see fig. 2). Similarly, DI claims peaked at a maximum of about 1.1 million claims in 2014 and have steadily declined since (see fig. 3). Claims in which individuals applied concurrently for DI and the SSI program (i.e., DI/SSI concurrent claims) peaked a little earlier—at about 1.3 million claims in 2010—before also steadily declining. While trends in opioid prescriptions and DI claims have moved in the same general direction over time, few studies and data sources provide information on the relationship between these trends. For example, we identified two studies, both funded by SSA, that examined the relationship between prescription opioids and disability. One preliminary study in 2017 found a positive correlation between prescription opioids and DI claims, but noted that this correlation was not statistically significant in every model. Researchers for this study noted that additional data and analysis are needed to refine the results. A second study in 2018 did not identify a direct relationship between opioid misuse and disability, but found that they may have an indirect relationship because of other factors such as having poor health, which may lead to unemployment due to disability. Other studies have examined the relationship between prescription opioids and employment, but not DI claims specifically. One such study noted that, based on available data, it is difficult to separate the effects of prescription opioid use and disability on employment outcomes. The study noted further that disentangling the relationship between prescription opioid use and disability is an area in need of additional work. In addition to funding research, SSA collects some administrative data on substance use among DI claimants, including use of prescription opioids. However, these data have limitations for analyzing prescription opioid use. Specifically, SSA collects administrative data on the medications claimants report using when filing their claim, which may include prescription opioids. However, these data may be incomplete because claimants may not report all substances they use. Further, researchers working on a study funded by SSA said analyzing these data is challenging because many claimants manually enter the names of their medications into an optional free-text field on their electronic applications rather than selecting from a dropdown menu, and that these entries often include misspellings or alternative names. SSA also collects administrative data on whether its staff evaluated a substance use disorder while processing a DI claim. However, SSA headquarters officials told us that staff are not required to record this information in the administrative data unless substance use disorders are the basis for a denial. Further, these data only indicate whether a substance use disorder involved alcohol or other drugs. They do not include additional details on the types of drugs involved (e.g., opioids versus methamphetamines). According to SSA headquarters officials, these details are not necessary for evaluating the claim or managing the process for DI eligibility decisions. Given the limitations with the claimant-level data described above, we analyzed county-level data for 2006 through 2017 and found that rates of opioid prescriptions and DI claims varied widely across counties. Specifically, the rate of opioid prescriptions ranged from nearly 0 to 396 opioid prescriptions per 100 people per year across all counties in 2017. Likewise, the rate of DI claims ranged from nearly 0 to 16.4 DI claims per 1,000 people. Most counties, however, were clustered around the median of 65 opioid prescriptions per 100 people and 3.7 DI claims per 1,000 people (see fig. 4). In examining counties with the highest rates of opioid prescriptions and DI claims (i.e., counties in the top third of the distributions for each rate), we found that those with the highest rates of both were generally concentrated in the Southeast (see fig. 5). Specifically, almost 30 percent of counties in the Southeast were among the highest for rates of both in 2017. In comparison, many counties in the West were among the highest for rates of opioid prescriptions, but not for DI claims. Conversely, many counties in the Northeast were among the highest for rates of DI claims, but not for opioid prescriptions. We also observed that these geographic differences were generally consistent over a 10-year period we analyzed. Our analysis shows a positive correlation between rates of opioid prescriptions and DI claims, as well as correlations between these rates and other factors (see fig. 6). Specifically, we conducted regression analyses to examine the relationship between rates of opioid prescriptions and DI claims at the county level from 2010 through 2017, taking into account economic, demographic, and other factors. However, we were unable to determine whether there is a causal relationship between rates of prescription opioids and DI claims or other factors, given readily available data. Further, given the small numbers of DI claims in most counties, we would not expect differences in the rate of DI claims to fully explain differences in the rate of opioid prescriptions. Correlation between opioid prescriptions and DI claims. We found that rates of opioid prescriptions and DI claims were positively correlated before and after accounting for other factors. Specifically, counties with higher rates of opioid prescriptions tended to have higher rates of DI claims and vice versa from 2010 through 2017. We would expect this correlation, given that many DI claimants experience pain, and prescription opioids are intended to help manage pain. Correlations between opioid prescriptions and other factors. Our analysis showed that rates of opioid prescriptions were correlated with poverty rates, population size, and access to health insurance. In particular, counties with higher rates of opioid prescriptions tended to have higher poverty, be less urban and with small- to mid-size populations, and have more people with health insurance from 2010 through 2017. Correlations between DI claims and other factors. Our analysis showed that rates of DI claims were also correlated with poverty rates, as well as unemployment, age, and race. In particular, counties with higher rates of DI claims tended to have higher unemployment and poverty from 2010 through 2017. Those with higher rates of DI claims also tended to have higher percentages of older adult and white populations. SSA’s policies require staff to deny DI benefits to claimants if substance use disorders (including opioids not taken as prescribed) are “material” to the impairments that preclude the claimant from work. For example, substance use disorders would be considered material to the claimant’s impairment if (1) they are the claimant’s only impairment, or (2) the claimant would not be considered disabled if they stopped using drugs or alcohol. To illustrate, program staff described an example, under SSA’s policies, in which they would deny a claimant with a mental health condition, such as depression, who also has a substance use disorder. In particular, if staff determined that substance use was affecting the claimant’s depression, and their mental health would improve to the point of non-disability in the absence of drugs or alcohol, SSA would deny the claim. In contrast, they may allow a claimant with permanent liver damage, even if caused by drug or alcohol use, because the damage is irreversible and would continue to be disabling even if the claimant were to stop using these substances. SSA uses a six-step process, referred to as the Drug Addiction and Alcoholism (DAA) evaluation, to determine whether substance use disorders are material to a claimant’s impairments. In the first two steps of this process, SSA determines whether a claimant is disabled and whether one of the claimant’s “medically determinable impairments” is a substance use disorder. Medically determinable impairments include physical or psychological abnormalities identified through medically acceptable diagnostic techniques and documented in objective evidence from an acceptable medical source, such as a physician or psychologist. If the answer is “yes” to both questions in the first two steps of the DAA evaluation, program staff use the remaining steps to help determine whether the substance use disorder is material to the claimant’s disability (see fig. 7). In conducting DAA evaluations, program staff can involve medical experts to assist them. At the initial level, DDS examiners confer with DDS medical consultants, such as in-house or contracted physicians and psychologists. At the hearings level, administrative law judges can also seek opinions from medical experts during the claimant’s hearing. Substance use disorders are seldom the key factor in DI eligibility decisions, according to SSA data and staff. Specifically, SSA data show that DAA evaluations of substance use disorders—aside from those that involved alcohol only—were the reason for a denial in about 0.1 percent of all decisions at the initial level and 0.3 percent of all decisions at the hearings level in 2017. Staff in our three selected states cited these potential reasons for why substance use disorders are seldom the key factor in DI eligibility decisions: Claimants with substance use disorders may not have qualifying impairments. Staff explained that those who do not have any impairment severe enough to meet SSA’s disability standards can be denied without a DAA evaluation. Medical records do not include enough evidence of a substance use disorder to warrant a DAA evaluation. Staff said some claimants may not have any evidence of a substance use disorder in their file because they may not report all substances they are taking or lack past medical treatment. In addition, staff said those with suspected substance use disorders may not have enough evidence of a disorder in their medical records to warrant a DAA evaluation. For example, they said pain clinics will often discharge a claimant from the clinic (i.e., stop providing services) due to drug-seeking behaviors. However, these pain clinics may not always document the reasons why the claimant was discharged. Further, staff said isolated instances of drug-seeking behaviors or discharges from pain clinics documented in medical records may not necessarily mean that a DAA evaluation is warranted. Some claimants have qualifying impairments, despite having substance use disorders. Staff said substance use disorders may not be the reason a claimant cannot work and may have little or no effect on a claimant’s impairments. For example, in one case file we reviewed, an administrative law judge conducted a DAA evaluation because of the claimant’s substance use disorders, likely involving alcohol and prescription medications, including opioids. The judge allowed the claim after determining that the claimant’s back issues were disabling, independent of the substance use disorders. Use of substances as prescribed by a treatment provider, including opioids, is not considered a substance use disorder. Program staff explained that, per SSA’s policies, they would not consider the use of opioids as prescribed to be a substance use disorder warranting a DAA evaluation, even if they thought the claimant was using unusually high amounts. SSA headquarters officials added that the use of prescription opioids could be considered a substance use disorder and result in a denial if medical records from an acceptable medical source included information about excessive or inappropriate use. Staff told us that making DI eligibility decisions for claims involving substance use disorders, including prescription opioids not taken as prescribed, can be complex. For example, staff in our three selected states noted challenges with subjectivity in conducting DAA evaluations, particularly when the claim involves mental health conditions. They said that certain conditions, such as depression or psychosis, can be exacerbated by substance use disorders. Thus, they said evaluating whether these conditions would continue to be disabling in the absence of drug or alcohol use can be difficult and subjective. We found that program staff faced challenges understanding or following SSA’s policies, based on our interviews with staff in three selected states and our review of 30 case files for DI beneficiaries, which included 15 in which a DAA evaluation had been conducted. Specifically, we found challenges with two aspects of the DAA evaluation process: Determining when to conduct a DAA evaluation. SSA headquarters officials told us that their policies do not require an official diagnosis of a substance use disorder from a treatment provider to conduct a DAA evaluation. Rather, they said a DAA evaluation is required if the potential disorder is considered a medically determinable impairment as defined by the current edition of the American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Disorders—which includes descriptions of many types of substance use disorders—and documented by an acceptable medical source. However, program staff in five of the six offices we visited in the three selected states, including DDS managers and examiners participating in group interviews and three administrative law judges, told us they believed they should not conduct a DAA evaluation unless they see an official diagnosis documented in the medical evidence. SSA headquarters officials discussed why staff may be confused about when to conduct a DAA evaluation, and acknowledged the potential effects. Specifically, they said staff may be confused about the policies for determining what is considered a medically determinable impairment for substance use disorders. Officials said there must be evidence of substance use that is consistent with the general definition of a substance use disorder as defined in the Diagnostic and Statistical Manual of Mental Disorders. They said staff may mistakenly interpret this requirement to mean that they need an official diagnosis to conduct a DAA evaluation. In fact, SSA’s operations manual for determining DI eligibility may also cause confusion. Though officials told us that SSA’s policies do not require an official diagnosis, the operations manual states that staff should only conduct a DAA evaluation when “an acceptable medical source establishes that a claimant is diagnosed with a substance use disorder.” SSA headquarters officials acknowledged that confusion about when to conduct a DAA evaluation could result in evaluations not being done when they should be, as well as claims being evaluated for substance use disorders unnecessarily when they do not meet the standards for being a medically determinable impairment. Documenting the rationale for why substance use disorders did not affect the claimant’s impairment. SSA’s policies for the DAA evaluation process generally require staff to document sufficient information about their evaluations so that a subsequent reviewer can understand the rationale for the decision, which is in keeping with federal standards for internal control. These policies also indicate that a single statement documenting that “DAA is not material” to the claimant’s impairments is not sufficient, and that documentation should be included in the determination and decision, or in other appropriate documents for DDS staff. In the 15 case files in which SSA had conducted a DAA evaluation, nine did not include a documented rationale. For example, in one case file we reviewed, a DDS examiner initially denied a claim for mental health issues after determining that these issues would not be disabling in the absence of the claimant’s substance use disorders, which involved benzodiazepines. An administrative law judge at the hearings level later allowed the claim, but did not document a rationale for why the claimant’s substance use disorders did not affect the claimant’s impairments. SSA headquarters officials agreed that a documented rationale was inappropriately missing in four of the nine case files mentioned above, although they did not indicate why the documentation was missing. For the remaining case files, while they agreed that there was no documented rationale, they asserted that neither a DAA evaluation nor a documented rationale was required. For example, for four of these five case files, officials stated that the substance use disorder was not established as a medically determinable impairment, that the claimant’s impairments were disabling by themselves regardless of whether there was any history of substance use disorders, and that the impairments were irreversible or could not improve to the point of non-disability. Nonetheless, a DAA evaluation was conducted in these case files, underscoring staff’s confusion about when an evaluation is necessary. Furthermore, regardless of whether a documented rationale was required in these case files, such documentation, if included, would ensure the rationale for the decision is clear to a subsequent reviewer, a recommended practice in federal internal control standards. SSA headquarters officials acknowledged that a poorly documented rationale could lead to reversals or remands if staff conducting appeals or quality reviews are unable to understand the decision. This could result in increased processing time for those conducting appeals and quality reviews, as well as for staff who may be required to revisit their decision. For example, in one case file we reviewed, an administrative law judge allowed a claim for mental health issues that had previously been denied at the initial level as a result of the claimant’s substance use disorders involving prescription opioids, alcohol, and marijuana. The case file was later randomly selected for quality review by the Appeals Council, which remanded the case back to the administrative law judge due, in part, to the lack of documented rationale regarding the claimant’s substance use disorders. As a result of the remand, the administrative law judge held a new hearing and issued a new decision that still allowed the claim, but provided a rationale for the DAA decision. SSA headquarters officials told us about efforts that could help ensure staff understand and follow policies for the DAA evaluation process. For example, they discussed training on DAA evaluation and documentation requirements. For DDS examiners, they said this training includes presentation slides and videos on these topics. Similarly, for new administrative law judges and other hearings-level staff, they said mandatory trainings include a module on the DAA policies. While SSA headquarters officials said they generally do not offer additional training beyond this, they noted that DDS examiners and administrative law judges are able to revisit the training materials and receive more local, ongoing training and resource materials as needed. We found examples of local, ongoing training and resource materials on the DAA evaluation process during our interviews in our three selected states. For example, one DDS office we visited had developed a DAA flowchart for its internal website, as well as a question and answer section derived from existing SSA information. Another DDS office had developed its own guidance specifically on documentation requirements for DAA evaluations. DDS managers and examiners in this office said they had sought clarification from the SSA office overseeing their region in developing the guidance, which was used during a local training for disability examiners in January 2019. In addition to training and guidance, SSA headquarters officials told us that compliance with policies for the DAA evaluation process is examined as part of the agency’s larger quality review processes. These processes are designed to ensure that cases are decided accurately. They include national and local reviews of randomly selected decisions at the initial level, as well as national reviews at the hearings level. Identified errors are reported back to the respective offices for correction. However, these reviews do not target claims involving substance use disorders. SSA headquarters officials said the agency does not track how often they review such claims at the initial level. DDS managers in the three selected states who are involved in local quality reviews also told us that such claims are not targeted for review. Despite SSA’s efforts to train staff on the DAA requirements, provide guidance, and conduct quality reviews that may cover DAA evaluations, we found that confusion about implementing the policies remains and staff are not always documenting the rationale for their evaluations as required. If SSA does not clarify its policies regarding when to conduct a DAA evaluation, as well as ensure that staff document the rationale for these evaluations, staff may not be in compliance with the policies. Further, if SSA does not take action, staff conducting subsequent appeals and quality reviews may not have the information needed to effectively examine prior evaluations of substance use disorders. Thus, the agency may expend resources re-working cases and, in turn, delay benefits to individuals eligible for assistance. The DI program helps people with eligible impairments even if they are also struggling with substance use disorders, including opioids not taken as prescribed, if the impairments would continue to be disabling in the absence of drugs or alcohol. Many people with disabilities have chronic pain for which prescription opioids are used as a legitimate treatment option. Thus, it is not surprising that many people who apply for DI benefits have opioid prescriptions, or that we would observe a positive correlation between these rates. Though SSA data show that substance use disorders are seldom the key factor in denying benefits, the agency nonetheless has a responsibility to show accountability for the decisions made by staff. Evaluating substance use disorders can be complex. However, without clarification to help staff better understand the policies for evaluating such disorders and ensuring staff document the rationale for their decisions, SSA likely cannot know whether claims are thoroughly assessed and efficiently examined as they move through subsequent reviews. Such inefficiencies can result in delayed benefits to those eligible for assistance. Further, while our review focused on prescription opioids, any improvements SSA makes to this process could help the agency stay ahead of shifting trends in the broader opioid epidemic. We are making the following two recommendations to SSA: The Commissioner of the Social Security Administration should clarify policies and procedures to remind staff that a diagnosis of a substance use disorder is not necessary to conduct a Drug Addiction and Alcoholism evaluation. (Recommendation 1) The Commissioner of the Social Security Administration should ensure that staff document their rationale for decisions involving the Drug Addiction and Alcoholism evaluation process. (Recommendation 2) We provided a draft of this report to SSA and HHS for review and comment. SSA provided technical comments, which we incorporated as appropriate, and formal comments. As part of its technical comments, SSA suggested that we revise the language of Recommendation 1 to focus more directly on the cause of staff’s confusion about when to conduct a DAA evaluation (i.e., staff’s misconception that a diagnosis of a substance use disorder is required). We agreed with this suggestion, and revised the recommendation accordingly. A letter conveying SSA’s formal comments is reproduced in appendix IV. SSA agreed with our recommendations. Regarding both recommendations, SSA stated that it will continue to train staff on the agency’s policies and procedures related to substance use disorders and the DAA evaluation process, as well as the importance of fully documenting these evaluations. HHS did not provide any comments. We are sending copies to the appropriate congressional committees, the Commissioner of the Social Security Administration, the Secretary of Health and Human Services, and other interested parties. The report will also be available at no charge on the GAO website at www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. We examined (1) what is known about the relationship between trends in prescription opioids and Disability Insurance (DI) claims, and (2) how the Social Security Administration (SSA) considers potential prescription opioid misuse in its DI eligibility decisions. This appendix provides a detailed account of the information and methods we used to answer these objectives. Section 1 provides an overview of our methods and key data sources. Sections 2 through 4 provide additional details on the three main methods we used to answer our objectives. To answer our first objective on the relationship between trends in prescription opioids and DI claims, we reviewed relevant literature and analyzed data from the Department of Health and Human Services’ (HHS) Centers for Disease Control and Prevention (CDC) and SSA. Specifically, we reviewed existing studies and interviewed several researchers currently examining the relationship between prescription opioids and DI claims. We also reviewed available data from SSA on prescription opioid use among DI claimants. In addition, we analyzed county-level data on the rates of opioid prescriptions from CDC and number of DI claims from SSA from 2006 through 2017, the most recent year of data available at the time of our review. We used aggregate data to illustrate nationwide trends over time. We also examined variation among counties, including differences among those with the highest rates of opioid prescriptions and DI claims. Lastly, we used these data to conduct multiple regression analyses to examine the relationship between opioid prescriptions and DI claims, taking into account economic, demographic, and other factors. We discuss these analyses in greater detail in Section 2. To answer our second objective on how SSA considers potential prescription opioid misuse in its DI eligibility decisions, we reviewed relevant information, interviewed program staff, and reviewed DI case files. We reviewed relevant federal laws, regulations, and SSA policies, as well as federal standards for internal control. We also interviewed SSA headquarters officials and staff involved in DI eligibility decisions in six offices in Alabama, Kentucky, and West Virginia. We discuss the criteria we used to select these states in Section 3. Lastly, we selected and reviewed 30 case files for DI beneficiaries who had been identified by the Centers for Medicare & Medicaid Services (CMS) as being at risk for prescription opioid misuse or abuse in 2017. We discuss the data and criteria we used to select these case files in Section 4. To answer our objectives, we used a variety of electronic data from data sources administered by CDC, SSA, and other federal agencies. Tables 1 and 2 summarize the key data sources and how they were used for each objective. For each data source, we conducted a reliability assessment by completing two or more of these steps: conduct electronic tests for completeness and accuracy, review relevant documentation, and interview knowledgeable officials about how the data are collected and maintained. We found that the data we used were sufficiently reliable for the purposes of our analyses. However, our analytical approach was limited by the availability of data, as discussed below and in appendix II. To answer our first objective on what is known about the relationship between trends in prescription opioids and DI claims, we conducted three sets of analyses using county-level data on the rates of opioid prescriptions and number of DI claims from 2006 through 2017. The data on opioid prescriptions are from CDC and represent the number of opioid prescriptions filled by retail (i.e., non-hospital) pharmacies per 100 people per year in each county. Though other datasets on prescription opioids exist, we chose to use CDC data because they show the actual number of prescriptions filled in each county, were publicly available at the time of our study, and included data through 2017. SSA provided data on the number of DI claims, which we used to calculate rates. We chose to include claims from individuals who are generally subject to a disability determination, such as disabled workers, widow(er)s, and adult children. We excluded individuals who are generally not subject to these determinations, such as dependent spouses and children under age 18. We examined DI only claims separately from DI/Supplemental Security Income (SSI) concurrent claims, and also examined similar data for DI allowances. We calculated rates of DI claims per 1,000 people per year in each county using population data from the U.S. Census Bureau. We used county-level data because claimant-level data, such as prescription opioid use by DI claimants, were not readily available. Our three sets of analyses examined: Nationwide trends. We used aggregate data from CDC on opioid prescriptions and data on DI claims from SSA to examine trends nationwide from 2006 through 2017. County variation. We used the data to examine variation among counties in their rates of opioid prescriptions and DI claims. Specifically, we examined the distribution of these rates among all counties. We had data available on both rates of opioid prescriptions and DI claims for 2,953 out of 3,142 counties nationwide. We then examined counties with the highest rates of opioid prescriptions and DI claims. We defined counties with the highest rates as those in the top third of the statistical distributions for each rate (i.e., at least 984 counties for each rate in 2017). Of these counties, 527 were in the top third of the statistical distribution for both rates. We plotted these counties with the highest rates on a U.S. map to observe any geographic differences across the Midwest, Southeast, Northeast, and West. In addition, we identified two counties to feature as illustrative examples. To select these counties, we first calculated the number of years from 2010 through 2017 a given county ranked in the top 10 for rates of opioid prescriptions and DI claims in each geographic region. We then selected two of these high-rate counties to serve as examples from different geographic regions and with different major industries. Regressions on the relationship between opioid prescriptions and DI claims. We used the county-level data to conduct regression analyses to examine the relationship between rates of opioid prescriptions and DI claims. In our regression models, we analyzed rates of opioid prescriptions and DI claims. In addition, we used data from a variety of sources to control for other county-level factors. Specifically, economic factors we accounted for included unemployment and poverty rates; demographic factors included sex, age, and race; and other factors included state, year, population size/degree of urbanization, and access to health insurance (i.e., uninsured rates). See table 1 above for additional information on the sources of these data, as well as appendix II for a detailed discussion of our regression analyses, including our models and limitations. To answer our second objective on how SSA considers potential prescription opioid misuse in its DI eligibility decisions, we conducted site visits to Alabama, Kentucky, and West Virginia. We selected these three states primarily because of their high rates of opioid prescriptions in 2016 and drug overdose deaths in 2017, and because a high percentage of their adult population received DI benefits in 2015. In each state, we visited one Disability Determination Services (DDS) office and one Hearing Office. These six offices included the Birmingham DDS and Birmingham Hearing Office in Alabama, the Frankfort DDS and Louisville Hearing Office in Kentucky, and the Charleston DDS and Charleston Hearing Office in West Virginia. We selected offices that were relatively larger, were nearest to or in counties with the highest rates of opioid prescriptions in the state in 2016, and where the DDS and Hearing Office were in close proximity, among other reasons. At each office, we interviewed a range of staff involved in making DI eligibility decisions. Specifically, for each DDS, we conducted group interviews with managers, disability examiners, and medical consultants. We initially conducted an exploratory site visit to the Frankfort DDS in Kentucky, where we met with all available managers, disability examiners, and medical consultants. In the remaining visits, we met with all available managers, but randomly selected five disability examiners and five medical consultants for the group interviews. Each group included between 5 and 15 participants. For each Hearing Office, we conducted individual interviews with three randomly selected administrative law judges, as well as the chief administrative law judge. For the purposes of our report, we include state government DDS staff in our general references to “program staff.” We used semi-structured interview protocols for all interviews that included open-ended questions about SSA’s processes for making decisions on claims involving potential prescription opioid misuse and any challenges doing so, among other topics. Because those we interviewed provided answers in response to open-ended questions, not all respondents commented on every process or challenge. In addition, because we visited a non-probability sample of DDS and Hearing Offices in three selected states, the results of our review cannot be generalized to all offices and states. To gain a deeper understanding of how SSA considers potential prescription opioid misuse in its DI eligibility decisions, we selected and reviewed 30 case files from SSA involving DI beneficiaries who had been identified by CMS as being at risk of opioid misuse or abuse. To select case files, we used a dataset from CMS on Medicare Part D beneficiaries that we matched with SSA data on DI beneficiaries. The CMS dataset contained information on Medicare Part D beneficiaries who CMS identified as being at risk of prescription opioid misuse or abuse in 2017. CMS identifies beneficiaries as being at risk of prescription opioid misuse or abuse if they received high amounts of opioids (had an average daily morphine dose equivalent of 90 mg or more) and appeared to have coordination of care issues (either had three or more opioid prescribers and three or more opioid dispensing pharmacies, or five or more prescribers regardless of the number of pharmacies) during a 6-month period. We identified DI beneficiaries within this larger dataset of Medicare Part D beneficiaries using an identifier in CMS’s data. This identifier signified that DI eligibility was a beneficiary’s reason for Medicare enrollment, since those eligible for DI may gain access to Medicare after a 2-year waiting period. We then worked with SSA to match these data on DI beneficiaries within CMS’s dataset with SSA data. Specifically, we obtained information for analysis from SSA’s database on various demographic characteristics of this population of DI beneficiaries, including their sex, age, race, and impairments. We also obtained administrative data on beneficiaries’ claims. Using the CMS dataset on Medicare Part D beneficiaries that we matched with SSA’s data on DI beneficiaries, we identified 30,273 DI beneficiaries who had been identified by CMS as being at risk of prescription opioid misuse or abuse in 2017. See appendix III for additional demographic and other information on this population. From the DI beneficiaries we identified, we selected 30 case files to review based on a number of claims characteristics related to potential prescription opioid misuse and SSA’s processing of the claim. First, we only selected case files for individuals who had been allowed benefits during or after 2013, when SSA formalized its policies for evaluating substance use disorders, including prescription opioids. In addition, we randomly selected 15 case files where the beneficiary had been evaluated by SSA for an identified substance use disorder and 15 where they had not. As part of the selection of 30 case files, we also randomly selected 16 case files where the beneficiary had self-reported the use of a prescription opioid and 14 where they had not, and 14 case files where the beneficiary had their case reviewed for potential medical improvement (called a continuing disability review) and 16 where they had not (these characteristics were not mutually exclusive). To systematically collect information on how or whether SSA considered potential prescription opioid misuse in each case file, we developed a data collection instrument to conduct our review of them. We designed the instrument to examine SSA’s implementation of its process for making DI eligibility decisions for claims involving substance use disorders, including opioids not taken as prescribed. For example, the instrument included questions about how SSA identifies and evaluates such disorders when making decisions, any documentation of this process, and how SSA reviews case files for potential medical improvement after allowing benefits. The instrument was not intended to examine the accuracy of decisions. In addition, we shared the instrument with SSA officials in advance, who provided notes on where the needed information could be found in the case files. Two GAO analysts independently reviewed each case file using the instrument, then met to review coding decisions and reconcile any differences between their reviews. We also discussed the results of our review with SSA headquarters officials. These officials provided comments on our observations for each case file, which we took into consideration. Though we examined information on all of the case file characteristics described above, we ultimately focused on SSA’s implementation of its process for evaluating the beneficiary for an identified substance use disorder. We did not focus on SSA’s implementation of its process for examining whether the beneficiary had self-reported the use of a prescription opioid. This is because the use of opioids as prescribed is not considered a substance use disorder under SSA’s policies. In addition, we learned during our case file review that beneficiaries had multiple opportunities to self-report such use that would not be captured in SSA’s administrative data, and that program staff also had multiple opportunities to examine such use when collecting and reviewing medical evidence. In addition, we did not focus on SSA’s implementation of its process for reviewing the beneficiary for potential medical improvement because we learned during our case file review that substance use disorders seldom factor into SSA decisions about whether to continue or cease DI benefits. According to SSA, benefits typically continue unless evidence exists that a beneficiaries’ impairment has medically improved and that they are able to return to work. SSA headquarters officials told us that staff would not evaluate a substance use disorder during the continuing disability review unless the beneficiary has medically improved and a new impairment that may be affected by a substance use disorder is to be assessed. Several limitations exist with our review of case files. Because we selected from a population of DI beneficiaries, the sample did not include case files for claimants who were ultimately denied. However, we did not see this as a significant limitation because SSA’s policies regarding the DAA evaluation are the same regardless of whether a claim is ultimately allowed or denied. In addition, 16 of the 30 case files we reviewed had been denied at the initial level before being allowed on appeal at later adjudicative levels. In addition, the case files may not have contained any evidence of prescription opioid misuse or abuse because of the timeframes we used to select them. Specifically, we selected case files for DI beneficiaries who had been allowed during or after 2013, but who were identified as being at risk of prescription opioid misuse or abuse in 2017. Because these beneficiaries may have been allowed benefits as early as 2013, they may not have had any issues with prescription opioids at the time SSA evaluated their claim (i.e., they may have developed potential issues after being allowed benefits). Lastly, because we reviewed a non-probability sample of 30 case files, the results of our review cannot be generalized to the larger population of DI beneficiaries. We conducted this performance audit from June 2018 to January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We used regression models and other data analyses to address our first objective on the relationship between opioid prescriptions and Disability Insurance (DI) claims. This technical appendix outlines the data, methodology, limitations, and results for the regression analyses in our report. We used county-level data from data sources administered by the Department of Health and Human Services’ Centers for Disease Control and Prevention (CDC) and other federal agencies from 2010 through 2017. These data included the rates of opioid prescriptions from CDC. We also used ZIP code-level data from the Social Security Administration (SSA) on the number of DI claims, which we transformed into county-level data using ZIP code-to-county crosswalk data from the Department of Housing and Urban Development. Similarly, we examined the numbers of DI allowances and DI/Supplemental Security Income (SSI) concurrent claims and allowances from SSA as well. In addition, we used data on a number of economic, demographic, and other factors. Economic factors included unemployment and poverty rates; demographic factors included sex, age, and race; and other factors included state, year, population size/degree of urbanization, and access to health insurance (i.e., uninsured rates). We used data from 2010 through 2017 because those were the years in which we had data for all of our factors, with the exception of degree of urbanization. We had data on degree of urbanization for 2013, and assumed that this factor was consistent from 2010 through 2017. For a list of the county-level data that we used in our analyses and their sources, see table 1 in appendix I. Table 3 provides summary statistics for factors included in our regression models. These include the mean, median, standard deviation, and range for the factors among counties from 2010 through 2017. We used linear regression models to analyze the relationship between rates of opioid prescriptions and DI claims, and controlled for the economic, demographic, and other factors described above at the county level. Our unit of analysis was the county-year, meaning that the observations are for each county each year. We had 22,789 observations, since there are over 2,977 counties and we used data from 2010 through 2017. Some of the factors we controlled for, such as unemployment rates, sex, age, race, and access to health insurance (i.e., uninsured rates), were similar to what other researchers used in examining the relationship between prescription opioids and employment variables. We included state and year fixed effects in our models to help account for additional factors that could vary across states or over time and national time trends. For example, differences in prescribing practices and increased law enforcement strategies across states could affect rates of opioid prescriptions. Further, factors that have previously been identified as possibly affecting the DI population include changes in the characteristics of the working-age population, federal policies (e.g., DI eligibility criteria), and employment opportunities. The results should be interpreted as changes in the dependent variable (i.e., rate of opioid prescriptions or rate of DI claims) associated with a change in the independent variables, within states. Compared to the previous model, the main dependent and independent variables are switched, but all other elements of the model are as described above. Though our primary focus was the relationship between rates of opioid prescriptions and DI claims, we also examined the relationship between rates of opioid prescriptions and DI allowances, as well as concurrent DI/SSI claims and allowances. We found that the data we reported on were sufficiently reliable for the purposes of our analyses. However, our analytical approach was limited by the availability of data. Consequently, our results should be interpreted with caution. Specifically, we were unable to establish whether there is a causal relationship between rates of opioid prescriptions and DI claims (e.g., whether higher rates of opioid prescriptions could have contributed to higher rates of DI claims or vice versa), in part because of potential reverse causality between these variables. While we could have potentially used an instrumental variable approach to establish a causal relationship, we did not identify an appropriate instrument to conduct that analysis. Moreover, individual-level data on opioid use among DI claimants were not readily available. Though we used county-level data, we were unable to account for variations within counties, also due to data not being readily available. Other researchers have noted similar limitations in their studies on prescription opioids. In addition, the opioid prescriptions data we analyzed only count the number of prescriptions filled, which could vary by number of pills, dosage, and potency (i.e., the morphine dose equivalent). The data also do not account for any potential diversion, or illicit transfer, of prescription opioids from one county to another. Further, we did not include county-fixed effects in our models. Though there may be constant or long-term characteristics of counties that are related to rates of opioid prescriptions and DI claims, we did not find enough variation in these rates within counties in the timeframe we analyzed to include county fixed effects in our models. In sensitivity analyses, we did include county fixed effects in our models and found that there was not a statistically significant relationship between rates of opioid prescriptions and DI claims with these effects included. However, this may be due to the large number of fixed effects introduced in the model (our analyses included about 3,000 counties) and the relatively short timeframe of 2010 through 2017. Lastly, we analyzed DI claims separately from DI/SSI concurrent claims in our models due to limitations with the units of analyses for these claims. Specifically, the number of DI claims represents the total number of claims an individual may have, rather than the number of individuals. For example, one individual may have five different DI claims and all five would be counted in the number of DI claims. On the other hand, the number of DI/SSI concurrent claims represents the number of individuals who had filed at least one DI and one SSI claim within a given year. The individual may have filed two DI claims and three SSI claims that year, but are counted as one DI/SSI concurrent claim. Though we were unable to determine whether there is a causal relationship between rates of opioid prescriptions and DI claims (e.g., whether higher rates of opioid prescriptions could have contributed to higher rates of DI claims or vice versa), as discussed above, we did find a significantly positive correlation between these rates across our models, on average, from 2010 through 2017. These results were consistent before and after accounting for the economic, demographic, and other factors described above. We also found correlations between rates of opioid prescriptions and some of the other factors. These correlations are detailed in figure 6 of our report. Table 4 also provides additional results from our regression analyses for rates of opioid prescriptions. In addition, we found correlations between rates of DI claims and other factors. Similarly, these correlations are detailed in figure 6 of our report. Table 5 provides additional results. We also examined the relationship between rates of opioid prescriptions and DI allowances, as well as DI/SSI concurrent claims and allowances, and found similar results. In various sensitivity analyses to check our results, we found that the positive correlation between rates of opioid prescriptions and DI claims remained consistent. For example, these results were consistent in models that: Included labor force participation rates instead of unemployment or poverty rates. Examined each year of data. Given that we did not find much variation in rates of opioid prescriptions and DI claims within counties from 2010 through 2017, we also ran our models for each year separately to explain variations across counties. Accounted for counties with small populations. There were eight counties that were omitted from our regression models because they had no DI claims. To ensure we accounted for all counties in our sensitivity analyses, we took an approach similar to other researchers and aggregated counties with less than 100,000 people in each state for each year. We ran our models when treating these counties with small populations as one county and found similar qualitative results. Using data from the Centers for Medicare & Medicaid Services (CMS) and the Social Security Administration (SSA), we identified 30,273 Disability Insurance (DI) beneficiaries who had been identified by CMS as being at risk of prescription opioid misuse or abuse in 2017. Figures 8 and 9 describe the demographics of this population, including beneficiaries’ sex, age, and race, as well as the primary impairments for which they were allowed DI benefits. In addition to the contact named above, the following staff members made key contributions to this report: Erin Godtland (Assistant Director), Nhi Nguyen (Analyst-in-Charge), Justin Gordinas, Kathleen McQueeney, and Paul Wright. Also contributing to this report were James Bennett, Joy Booth, Mari Calderón, Breanne Cave, Jessica Farb, Justin Fisher, Alex Galuten, Melissa Jaynes, Lorin Obler, Jessica Orr, Oliver Richard, William Simerl, Almeta Spencer, Shana Wallace, and Eric Wedum.", "summary": "The United States is in the midst of an unprecedented opioid epidemic. Opioids are prescribed to treat conditions such as chronic pain. However, opioid misuse can lead to addiction, disability, overdose, and death. Prior GAO and other reports have discussed the use of prescription opioids within federal programs, particularly Medicare. Less is known about the use of opioids in relation to SSA's DI program. GAO was asked to review any correlation between prescription opioids and rates of DI claims, and any related challenges for SSA. This report examines (1) what is known about the relationship between trends in prescription opioids and DI claims, and (2) how SSA considers potential prescription opioid misuse in its DI eligibility decisions. GAO analyzed county-level data on opioid prescriptions and DI claims from 2006 through 2017; interviewed program staff involved in DI eligibility decisions in Alabama, Kentucky, and West Virginia, selected because of their high rates of opioid prescriptions and percentage of the adult population on DI; and reviewed case files for DI beneficiaries identified by the Centers for Medicare & Medicaid Services as being at risk for prescription opioid misuse or abuse. The numbers of opioid prescriptions and claims for the Social Security Administration's (SSA) Disability Insurance (DI) program have each declined nationally in recent years, but rates vary widely across the country. National trends show both peaking between 2010 and 2014 and then declining. GAO's analysis shows counties with the highest rates of both were concentrated in the Southeast (see figure). After accounting for economic, demographic, and other factors, GAO found that counties with higher rates of opioid prescriptions tended to have higher rates of DI claims from 2010 through 2017. These rates were also correlated with other factors. For example, counties with higher rates of each tended to have higher poverty rates. However, GAO was unable to determine whether there is a causal relationship between rates of opioid prescriptions and DI claims or other factors, given readily available data. Program staff are required to evaluate and document substance use disorders (including opioids not taken as prescribed) when making certain DI eligibility decisions. Specifically, staff are required to evaluate potential substance use disorders for certain DI claims and deny benefits, for example, if the claimant would not be considered disabled if they stopped using drugs or alcohol. In addition, staff are generally required to document the rationale for their decision so that another reviewer can understand how they made the decision. However, staff in five of the six offices GAO visited in three states were confused about when to evaluate substance use disorders, and nine of 15 case files that GAO reviewed in which an evaluation was conducted did not have a documented rationale. SSA officials acknowledged the need to clarify policies on when to evaluate substance use disorders, and that a poorly documented rationale could lead to reversals or remands of decisions. Without ensuring that SSA's policies are understood and that staff document their rationale, the agency may expend resources re-working cases and, in turn, delay benefits to individuals eligible for assistance. GAO recommends that SSA 1) clarify policies and procedures to help staff better evaluate substance use disorders, and 2) ensure staff document their rationale. SSA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "As described in figure 1, to support U.S. exports, EXIM offers four major types of financing products: direct loans, loan guarantees, export-credit insurance, and working capital guarantees. Regardless of type, EXIM’s financing products generally have one of three maturity periods: Short- term transactions are for less than 1 year; medium-term transactions are from 1 to 7 years long; and long-term transactions are more than 7 years. As we reported in July 2018, for all financing types, EXIM currently conducts a number of preauthorization and postauthorization antifraud activities. See the examples shown in figure 2. Fraud and “fraud risk” are distinct concepts. Fraud—obtaining something of value through willful misrepresentation—can be challenging to detect and adjudicate because of its deceptive nature. Fraud risk exists when individuals have an opportunity to engage in fraudulent activity, have an incentive or are under pressure (e.g., financial pressures) to commit fraud, or are able to rationalize committing fraud. When fraud risks can be identified and mitigated, fraud may be less likely to occur. Although the occurrence of fraud indicates there is a fraud risk, a fraud risk can exist even if actual fraud has not yet been identified or adjudicated. According to the Standards for Internal Control in the Federal Government, executive-branch agency managers are responsible for managing fraud risks and implementing practices for combating those risks. Specifically, federal internal control standards call for agency management officials to assess the internal and external risks (including fraud risks) their entities face as they seek to achieve their objectives. The standards state that as part of this overall assessment, management should consider the potential for fraud when identifying, analyzing, and responding to risks. Risk management is a formal and disciplined practice for addressing risk and reducing it to an acceptable level. The leading practices in the Fraud Risk Framework call for agencies to identify inherent fraud risks affecting the program, examine the suitability of existing fraud controls, and then prioritize mitigating “residual” fraud risks—that is, risks remaining after antifraud controls are adopted. Specifically, according to the assess component of the Fraud Risk Framework, managers who effectively assess fraud risks attempt to fully consider the specific fraud risks the agency or program faces, analyze the potential likelihood and impact of fraud schemes, and then ultimately document prioritized fraud risks. Moreover, managers can use the fraud risk assessment process to determine the extent to which controls may no longer be relevant or cost-effective. Leading practices that are consistent with this component include conducting quantitative or qualitative fraud risk assessments at regular intervals, or both, of the likelihood and impact of inherent risks on the program’s objectives, and determining the agency’s risk tolerance for the inherent fraud risks; identifying specific sources for gathering information about fraud risks, including information on fraud schemes that are reflected in adjudicated cases of fraud; examining the suitability of existing fraud controls for preventing fraud and mitigating fraud risks identified; and documenting in the program’s fraud risk profile the analysis of the types of inherent fraud risks assessed, their perceived likelihood and impact, managers’ risk tolerance, and the prioritization of the inherent fraud risks and any residual fraud risks. As we reported in July 2018, the Fraud Reduction and Data Analytics Act of 2015 requires the Office of Management and Budget (OMB) to establish guidelines that incorporate the leading practices of GAO’s Fraud Risk Framework. The act also requires federal agencies to submit to Congress a progress report each year, for 3 consecutive years, on implementation of the risk management and internal controls established under the OMB guidelines. OMB published guidance under OMB Circular A-123 in 2016 affirming that federal managers should adhere to the leading practices identified in the Fraud Risk Framework. As we reported in December 2018, EXIM identifies itself as subject to the act, and, as such, follows it. The Fraud Risk Framework is also aligned with federal internal control standards, specifically Principle 8 (“Assess Fraud Risk”) of the Green Book. Federal internal control standards also state that excessive pressures, such as financial pressures (e.g., delinquent federal debt), can pose a fraud risk factor to agency programs as these pressures can provide an incentive or motive to commit fraud. Although the existence of financial pressure alone does not necessarily indicate that fraud exists or will occur, financial pressure is often present when fraud does occur. Applicants for EXIM programs who have delinquent federal debt may not be able to obtain certain types of financing until they resolve their debts. Specifically, under 31 U.S.C. § 3720B, applicants who are delinquent on federal nontax debts may not receive federal financial assistance, including such assistance provided by EXIM, until they satisfactorily resolve the delinquency (e.g., pay in full or negotiate a new repayment plan). However, 31 U.S.C. § 3720B also provides that an agency head may waive this restriction. Additionally, OMB’s Circular No. A-129, Policies for Federal Credit Programs and Non-Tax Receivables, prescribes to agencies the policies, procedures, and standards for screening program participants to determine whether they are delinquent on any federal debt when applying to federal credit programs. We identified fraud risks—generally involving four overall fraud risk factors—by examining EXIM-associated court cases of fraud adjudicated from calendar year 2012 through calendar year 2017. We then communicated these fraud risks to EXIM, and EXIM officials reported examples of existing controls it uses to help detect and mitigate these fraud risks. EXIM also provided documentation reflecting its efforts to conduct a fraud risk assessment that considered various fraud risks affecting its major financing product lines, including fraud risks we identified during this review. We identified fraud risks—generally involving four overall fraud risk factors—by examining 44 EXIM-associated closed court cases of fraud adjudicated from calendar year 2012 through calendar year 2017. Specifically, the various fraud risks we identified overall involved one or more of the fraud risk factors illustrated in figure 3 below: opportunities to falsify self-reported information on applications or financial pressures that potentially incentivized participants or employees to commit fraud; opportunities to circumvent or take advantage of EXIM or lender opportunities to circumvent the intent of EXIM’s programs by diverting loan proceeds and other EXIM financing for personal use or benefit instead of for the export of U.S. goods. See appendix I for a summary of these 44 cases we reviewed. These 44 cases illustrate the financial risks associated with fraud against EXIM. Federal and state courts combined have ordered restitution of $82.4 million in the 44 adjudicated cases, but much of that restitution has not yet been paid. For example, as of October 2018, the total remaining unpaid restitution amount is $71.6 million, or over 80 percent. In one fraud case we reviewed, which was adjudicated in 2013, a federal court ordered a convicted U.S. exporter to pay EXIM $8.6 million in restitution for the fraud that he committed in a loan guarantee program. Since 2013, the participant has paid back $25.00 of this amount. EXIM reported having existing antifraud controls to mitigate the fraud risks we identified. Specifically, we communicated to EXIM the fraud risks we identified from our review of the 44 adjudicated cases. In response, EXIM officials described general antifraud controls the agency currently uses to help detect and mitigate each of the fraud risks we identified. The officials stated that EXIM has experience with all the fraud risks we identified and stated that they were generally confident that EXIM’s antifraud controls were appropriate for mitigating the risks. EXIM officials consider many of the fraud risks that we identified as risks that could impact any of the agency’s financing programs (i.e., credit insurance, loan guarantees, direct loans, or working capital guarantee programs). EXIM officials provided examples of the general antifraud controls that they said EXIM uses to mitigate the fraud risks we identified across all agency financing products. According to EXIM officials and as illustrated in figure 4 below, these controls include: fraud prevention and detection procedures; due diligence standards; and a list of “red flags” that EXIM staff should be aware of and is used to identify indicators of potential fraud and corruption that may appear on EXIM transaction documents. Officials said that their confidence in the controls stems from seeing a reduction in fraud cases since the early 2000s after these antifraud controls were put in place. EXIM officials clarified that this confidence does not stem from completing a comprehensive fraud risk assessment of fraud risks impacting all of its financing products consistent with the leading practices in the Fraud Risk Framework. EXIM also provided documentation reflecting its efforts to conduct a fraud risk assessment that considered various fraud risks affecting its major financing product lines, including fraud risks we identified during this review. EXIM officials said that the fraud risks we identified were generally already known to EXIM as they relate to or are very similar to those fraud risk factors contained in EXIM’s list of red flags. EXIM officials acknowledged that assessing its fraud risks and evaluating the agency’s existing antifraud controls may indicate opportunities for EXIM to further adapt EXIM’s antifraud controls to mitigate any residual fraud risks within its tolerance level. Such assessments can further help EXIM mitigate fraud and the resulting effects across all product lines before they occur, which includes the length of time it can take for EXIM to fully recover from restitution losses after fraud has been perpetrated, as illustrated in the 44 cases presented in appendix I. EXIM has procedures for detecting delinquent federal debt owed by EXIM applicants and participants. However, EXIM is missing additional opportunities to use readily available SAM data to identify ineligible applicants or participants that may have delinquent federal debt, and to use such data to determine eligibility or assess repayment fraud risk. EXIM has procedures to detect delinquent federal debt owed by applicants and participants that include reviewing their credit reports and requiring applicants to certify that they and other participants do not have such delinquent debt. Under 31 U.S.C. § 3720B, applicants who are delinquent on federal nontax debts may not receive federal financial assistance, including direct loans, loan guarantees, or loan insurance until they satisfactorily resolve the delinquency (e.g., pay in full or negotiate a new repayment plan). 31 U.S.C. § 3720B does not address delinquent federal tax debt; however, such delinquent federal debt may also pose a fraud risk or repayment fraud risk to EXIM’s financing programs. Additionally, OMB Circular No. A-129 prescribes to agencies the policies, procedures, and standards for screening program participants to determine whether they are delinquent on any federal debt when applying to federal credit programs, including recommending that agencies ask applicants to self-certify on their applications that they have no delinquencies; requiring agencies to obtain and review applicants’ credit reports; and encouraging agencies to use appropriate databases, such as the Department of the Treasury’s Do Not Pay portal sources to identify delinquent federal debtors during the application screening process. According to EXIM officials, the agency employs procedures to ensure its policies and processes meet these requirements for applicable financing products. Specifically, and as illustrated in figure 5 below, these procedures include reviewing the following: Self-certifications: EXIM applications for relevant financing programs include a self-certification by the applicant that the applicant does not have delinquent federal debt. However, as we have reported in the past, relying on applicants to self-report adverse actions on their applications, instead of verifying such information, could cause an agency to miss opportunities to develop a more-complete picture of the applicants. Credit reports: EXIM obtains credit reports for applicants and participants in some financing products. In particular, EXIM’s internal Loan Guarantee and Credit Insurance Manual of 2015 communicates the 31 U.S.C. § 3720B restriction to loan officers and instructs them to review the borrower’s credit report to check whether the borrower is delinquent on any federal debt. If the loan officer finds that the credit report reflects such delinquent federal debt, the manual further instructs the loan officer to advise and request guidance from EXIM’s Trade Finance Director and the Office of General Counsel. However, as we have reported in the past, some delinquent federal tax debt may not appear on the credit reports unless the Internal Revenue Service has filed a lien on the delinquent federal tax debt. World Check: EXIM, through the assistance of a third-party vendor, also makes use of some data sources listed in the Do Not Pay sources as part of its prescreening application process and possibly during postauthorization risk-based reviews. Specifically, EXIM officials told us that EXIM uses Thomson Reuters’s World Check database to identify federal debts owed by applicants as part of its Character, Reputational, and Transaction Integrity (CRTI) review process that is managed by EXIM’s Credit Review and Compliance Division. The World Check database currently checks over 20 different watch lists and other databases, including lists of entities excluded from doing business with the federal government maintained in GSA’s SAM. According to EXIM, other sources in the World Check database that reveal such federal debts could also lead indirectly to the discovery of delinquent federal debt. However, as discussed below, this check of SAM does not involve a check of delinquent federal debt. This CRTI review process is conducted during the underwriting (i.e., the preauthorization review) phase and may occur throughout the life cycle of transactions, such as during EXIM’s postauthorization risk-based reviews. EXIM officials told us that, as part of this process, loan officers or other EXIM officials send the names of applicants to EXIM librarians, who perform a manual search of the World Check database, review results, and return relevant results to EXIM officials for their consideration. EXIM officials noted that this process can be challenging, particularly when librarians perform searches on applicants with common names, which produce many results that are not useful. EXIM officials told us that EXIM does not track information on instances in which an applicant’s delinquent federal debt prevents a transaction from moving forward or prevents a specific applicant’s participation in a transaction. Consequently, EXIM officials told us that EXIM has no records of this happening. However as described in greater detail below, EXIM does not make use of readily available SAM data to identify delinquent federal debts owed by applicants and participants, which could limit its ability to detect instances in which applicants and participants owe these debts. EXIM is missing additional opportunities to use readily available SAM registration data to identify potentially ineligible applicants and participants that may have delinquent federal debt or may otherwise pose a repayment fraud risk. Specifically, while EXIM employs procedures that may reveal applicants’ delinquent federal debts, as described above, EXIM’s procedures for identifying applicants and participants with delinquent federal debt do not include a search of a specific data element in the SAM database that can be used to detect delinquent federal debtors. The data element we refer to here is the Debt Subject to Offset flag, which may reflect both nontax and tax delinquent federal debts owed. As mentioned previously, SAM is a government-wide information system that federal agencies can use to obtain information on businesses that do business with the federal government, including an entity’s Debt Subject to Offset status. The Debt Subject to Offset data element in SAM indicates that the entity potentially has a delinquent federal debt subject to collection under the Treasury Offset Program. The GSA officials who maintain the SAM database told us that all federal agencies have the legal authority to use the SAM registration database free of charge. Specifically, all federal agencies can use this database to manually search by an entity’s name, Data Universal Numbering System number, or Tax Identification Number for the purpose of detecting whether the entity potentially has delinquent federal debt, such as by identifying whether an entity’s SAM record contains the Debt Subject to Offset flag. Further, GSA officials also told us that all federal agencies are able to request batches of SAM registration data free of charge, for the purpose of matching these data to agency data by entities’ names, Data Universal Numbering System numbers, or Tax Identification Numbers for the purpose of identifying entities that may have the Debt Subject to Offset flag in SAM, among other available data. Performing data analytics, such as batch matching, on available data is a leading practice cited in the Fraud Risk Framework that we have reported can help improve agency efforts to combat fraud. In particular, we have found in prior work that using available data to verify that EXIM’s transaction applicants are not delinquent on federal debt can help EXIM assure applicant eligibility is consistent with federal guidance, provide reasonable assurance of repayment, and help prevent fraud. We have also found that using available data to independently verify self-reported delinquent federal debt information, such as self-reported information on delinquent federal tax debt owed, is a key detection and monitoring component of fraud prevention. We identified additional opportunities for EXIM to manually use SAM’s online database or data-matching approaches to identify applicants or participants with potential delinquent federal debt. Specifically, we registered in SAM to conduct several manual searches (by entities’ Data Universal Numbering System numbers, Tax Identification Numbers, and names) and confirmed that it can be used to conduct such searches without incurring any external costs charged by GSA. For example, we conducted two Data Universal Numbering System number searches and found two active EXIM participants appearing in SAM’s registration database with a Debt Subject to Offset flag. We also obtained historical SAM data from GSA and EXIM transaction data and confirmed that these data sources could be used to identify EXIM applicants and participants with potentially delinquent federal debt in a batch match (rather than manual, case-by-case searches). As illustrated in our batch-matching results below, we found this data-matching process can provide an opportunity to match these data sets using the Tax Identification Numbers and Data Universal Numbering System numbers for the entities in both data sets. Our batch-matching analyses indicated that, from calendar year 2014 through calendar year 2016, EXIM authorized transactions that had an aggregate authorization value of approximately $34.3 billion. Of that amount, we found the following: An aggregate authorization value of about $1.7 billion was associated with 32 U.S.-based companies that had a delinquent federal debt indicator in SAM in the same month that these transactions were authorized. The transactions mostly involved U.S.-based applicants and exporters. As mentioned above, associated parties we reviewed included not only the applicant, but also participants involved, including the borrower, buyer, and exporter, which may or may not be the applicant. While the results of this analysis do not mean that EXIM should have suspended these transactions in accordance with 31 U.S.C. § 3720B, these results nonetheless indicate that the data in SAM that indicate delinquent federal debt could provide an opportunity for EXIM to identify important indicators of applicants or other transaction participants with potential delinquent federal debt when determining their program eligibility and assessing any related fraud risks or repayment risks they present during EXIM’s preauthorization CRTI reviews. Because the Debt Subject to Offset flag may indicate either nontax debts or tax debts, it is possible that some of these entities owed delinquent federal nontax debts that are applicable under 31 U.S.C. § 3720B, indicating EXIM should have considered suspending these transactions. However, it is also possible that some of these entities owed delinquent federal tax debts that are not applicable under 31 U.S.C. § 3720B, but that may pose a fraud risk or repayment risk nonetheless. By using the Debt Subject to Offset flag as an indicator of these delinquent federal debts and gathering additional information on the specific facts and circumstances of each case, EXIM would be better positioned to assess the relevant compliance, fraud, and repayment risks an applicant’s or participant’s delinquent federal debt may pose. An aggregate authorization value of about $4.1 billion was associated with 97 U.S.-based companies that had a delinquent federal debt indicator in SAM during the transaction maturity period (i.e., after the month they were approved, but before the transactions’ maturity date). These transactions mostly involved U.S.-based applicants and exporters. As mentioned above, associated parties we reviewed included not only the applicant, but also participants involved, including the borrower, buyer, and exporter, which may or may not be the applicant. 31 U.S.C. § 3720B may prevent applicants with federal financial debts from obtaining loans, guarantees, and insurance; thus, it does not apply to any delinquent federal debt accrued after loan approval. However, we looked at potential delinquent debt accrued after approval because delinquent debt accrued after approval and during the transaction maturity period might affect EXIM’s view of a financing product’s repayment risk. Further, EXIM already conducts similar postauthorization monitoring to identify such risks through its use of World Check as part of its CRTI process described above. Thus, these results nonetheless illustrate that EXIM can use SAM data during EXIM’s postauthorization CRTI reviews to identify transaction participants with potential delinquent federal debt and determine the extent to which they may pose a repayment fraud risk. Prior to sharing our results with EXIM, EXIM officials told us that they have access to SAM entity registration records, but they believe searching the SAM registration database is a time-consuming process that should be reserved for rare circumstances. Further, EXIM officials also told us that using the SAM registration database to identify applicants or participants that have the Debt Subject to Offset flag in SAM would yield few results because the vast majority of their financing program participants are foreign-based entities, and thus would not also be contractors for the U.S. government and registered in SAM. However, we identified many U.S.-based entities that had a delinquent federal debt indicator either in the month a transaction was approved, or during the transaction’s maturity period, by searching in the SAM database and analyzing SAM data for EXIM applicant and participants, as described above. Further, it is not clear whether performing manual searches or batch matches with SAM data to identify delinquent federal debtors would be any more time-consuming than EXIM’s current procedures for doing so, which include manual searches of World Check and obtaining and reviewing credit reports, as described above. When we met with EXIM officials to communicate our batch-matching results above, they expressed concern that these results could imply that EXIM is doing business with applicants or participants with delinquent federal debt. They then indicated that they were interested in obtaining SAM registration data so that they could determine whether it would be feasible for them to perform the same type of analysis that we performed. In a subsequent meeting, EXIM officials informed us that they were also able to obtain current SAM registration data, analyze the SAM data against active EXIM participant data, and find dozens of active EXIM participants with the Debt Subject to Offset flag in SAM. The results of our analyses, as well as EXIM’s own experience with the SAM data, suggest EXIM also has an additional and practical opportunity to incorporate searches of SAM entity registration data as part of its postapproval monitoring of transactions to enhance its monitoring of and response to risks in ongoing transactions. Standards for Internal Control in the Federal Government state that management should use quality data to achieve agency objectives. For example, this could include agencies obtaining relevant operational, financial, or compliance-related data from reliable internal and external sources in a timely manner based on identified information requirements, and then using such data to make informed decisions and evaluate performance in achieving program objectives and addressing risks. Without also pursuing available debt data in SAM’s registration database, as an additional layer of due diligence, to identify applicants with delinquent federal debt during underwriting and compliance reviews, EXIM is potentially forgoing practical opportunities to use such data when determining applicants’ program eligibility and to adopt leading practices for managing repayment fraud risks across EXIM’s financing programs. In particular, such available SAM data can provide opportunities to verify independently the applicants’ self-certification of delinquent federal debts they owe and assess whether the applicants may have misrepresented their delinquent federal debt status on their applications, which is a fraud risk in the application process; detect potential delinquent federal debts that are not apparent in credit make informed eligibility decisions during preauthorization CRTI reviews and assess repayment fraud risk during postauthorization CRTI reviews. EXIM assumes the credit and country risks that the private sector is unable or unwilling to accept, including the risk of losses due to fraud. EXIM’s financing products face various fraud risks, and EXIM has begun to take steps to consider these fraud risks as part of a full fraud risk assessment, as we recommended in July 2018. However, because it remains unclear whether EXIM’s actions fully respond to the recommendations of our July 2018 report, we will continue to monitor EXIM’s progress in fully assessing its fraud risks. EXIM also employs procedures to detect delinquent federal debt owed by EXIM applicants and participants. However, EXIM is missing opportunities to use readily available SAM data to identify applicants or participants that may misrepresent their delinquent federal debt status and pose a repayment fraud risk to EXIM financing programs. Applicants or participants with delinquent federal debt could be one of many repayment fraud risks that could indicate an increased risk of nonrepayment and incentives to commit fraud against EXIM. EXIM officials believe searching SAM is a time-consuming process that would yield few results. However, manually searching SAM’s online registration database for the purpose of determining whether an applicant or participant may have a Debt Subject to Offset flag may not be any more time-consuming than what EXIM currently performs through its preauthorization or postauthorization CRTI reviews. Nevertheless, we demonstrate in this report the practicality and illustrate results of using such data through multiple approaches, such as batch matching, without incurring any external costs charged by GSA. By assessing the practicality of searching SAM data, EXIM may determine that this source of data provides an additional tool for combating fraud. Implementing these antifraud activities could further help EXIM verify program eligibility, identify repayment fraud risk, and provide EXIM with reasonable assurance that it is effectively and efficiently carrying out its mission of supporting U.S. jobs and the export of U.S. goods. We are making the following two recommendations to EXIM: EXIM’s chief operating officer should direct EXIM’s Credit Review and Compliance Division to assess and document the practicality of incorporating into its preauthorization CRTI reviews searches of data elements in SAM that indicate delinquent federal debts owed by applicants, and, if practical, implement relevant approaches—such as manual searches or batch matching. (Recommendation 1) EXIM’s chief operating officer should direct EXIM’s Credit Review and Compliance Division to assess and document the practicality of incorporating into its postauthorization CRTI reviews searches of data elements in SAM that indicate delinquent federal debts owed by applicants and participants, and, if practical, implement relevant approaches—such as manual searches or batch matching. (Recommendation 2) We provided a draft of this report to EXIM for review and comment. In its written comments, reproduced in appendix II, EXIM concurred with our recommendations and stated that it will move forward to implement them. EXIM also provided technical comments, which we incorporated as appropriate. In its written comments, EXIM noted a number of points it referred to as “key concerns.” These points do not disagree with our findings, conclusions, or recommendations. Specifically, EXIM stated that the 44 cases we reviewed involved transactions that were approved between 2002 and 2012 and that it will continue to work with the Department of Justice to collect restitution payments. Additionally, EXIM stated that it is in full compliance with 31 U.S.C. § 3720B and the related provisions of OMB Circular A-129 guidance regarding restrictions on doing business with delinquent federal debtors. However, assessing EXIM’s compliance with 31 U.S.C. § 3720B or OMB Circular A-129 was outside the scope of this report. Finally, for the purpose of implementing our recommendations, EXIM requested the data pertaining to the U.S.-based companies that we found to have a delinquent federal debt indicator in SAM. To identify those companies, we used (1) an extract of data that EXIM provided to us, and (2) GSA SAM data, which EXIM told us it can and has already obtained directly from GSA. We will provide EXIM with a copy of the EXIM data it requested. However, we believe EXIM will be better positioned to assess the practicality of checking the SAM delinquent federal debt flag by continuing to obtain the SAM data directly from GSA. We are sending copies of this report to the appropriate congressional committees, the president and board chairman of EXIM, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-6722 or bagdoyans@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. The table below summarizes the information we reviewed during our review of the 44 Export-Import Bank of the United States (EXIM)- associated cases of alleged fraud that we were able to identify and determine were adjudicated from calendar years 2012 through 2017. Such information includes financing product types, dates adjudicated, fraud schemes, fraud risk factors involved, and the amount of EXIM restitution owed and paid to EXIM. As mentioned earlier, the fraud risks we identified in these 44 cases related to one or more of the following four fraud risk factors: (1) opportunities to falsify self-reported information on applications or transaction documents, (2) financial pressures that potentially incentivized participants or employees to commit fraud, (3) opportunities to circumvent or take advantage of EXIM or lender controls, or (4) opportunities to circumvent the intent of EXIM’s financing programs by diverting loan proceeds and other EXIM financing for personal use or benefit instead of for the export of U.S. goods. In addition to the contact named above, Jonathon Oldmixon (Assistant Director), Flavio Martinez (Analyst in Charge), Mason Calhoun, Marcus Corbin, Anthony Costulas, Adam Cowles, David Dornisch, Heather Dunahoo, Paulissa Earl, Colin Fallon, Dennis Fauber, Jennifer Felder, Dragan Matic, Maria McMullen, Christopher H. Schmitt, Albert Sim, Sabrina Streagle, and Steve Westley made key contributions to this report.", "summary": "As the export credit agency of the United States, EXIM's mission is to help support U.S. jobs by facilitating the export of U.S. goods and services through direct loans, loan guarantees, working capital guarantees, and credit insurance. In September 2018, the total outstanding and undisbursed amount of these products and unrecovered default claims was about $60.5 billion, according to EXIM. The Export-Import Bank Reform Reauthorization Act of 2015 included a provision for GAO to review EXIM's antifraud controls. This report (1) describes key antifraud controls EXIM says it has for mitigating fraud risks identified by GAO, and describes EXIM's efforts to perform a fraud risk assessment that considers these fraud risks; and (2) identifies EXIM's procedures to detect delinquent federal debt owed by applicants and participants, and assesses additional opportunities to use readily available data to do so. GAO analyzed 44 EXIM-associated court cases of fraud adjudicated from calendar years 2012 through 2017, examined EXIM transaction data, and interviewed EXIM and GSA officials. GAO also analyzed data identifying delinquent federal debt as well as EXIM's procedures for doing so. The Export-Import Bank of the United States (EXIM) reported having antifraud controls in place for mitigating the fraud risks that GAO identified and communicated to EXIM officials. GAO reviewed 44 EXIM-associated court cases involving fraud and identified fraud risks involving the four fraud risk factors illustrated in the figure below. GAO communicated these fraud risks to EXIM officials, and they provided examples of antifraud controls they use to help mitigate these fraud risks for their major financing products. In February 2019, EXIM also provided documentation reflecting its efforts to conduct a fraud risk assessment that considered various fraud risks affecting its major financing product lines, including fraud risks GAO identified during this review. EXIM has procedures to identify applicants and participants with delinquent federal debt, such as obtaining applicants' credit reports that may indicate these debts when they apply to EXIM's financing programs. However, EXIM is missing additional opportunities to use readily available data containing delinquent federal debt indicators from the General Services Administration's (GSA) System for Award Management (SAM) to detect applicants and participants that may have delinquent federal debt. Federal law states that applicants who are delinquent on federal nontax debts may not receive federal direct loans, loan guarantees, or loan insurance until the delinquent debt is satisfactorily resolved. Using data from SAM, GAO found that, from calendar years 2014 through 2016, EXIM authorized transactions that had an aggregate authorization value of about $1.7 billion and were associated with 32 U.S.-based companies that had a delinquent federal debt indicator in SAM in the same month EXIM authorized these transactions . While these results alone do not mean EXIM should have suspended these transactions, they do indicate that there is a practical opportunity to use SAM data to help determine applicants' eligibility. Without assessing the practicality of pursuing such readily available data, EXIM is potentially forgoing opportunities to perform additional due diligence that would help inform its decisions about applicants' and participants' program eligibility and fraud risks. GAO is recommending that EXIM assess the practicality of using available SAM data and data-analytical approaches to detect applicants and participants with potential delinquent federal debt. EXIM concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Our report found that SBA’s STEP grants management process does not provide reasonable assurance that STEP grant recipients meet two of the three TFTEA requirements we reviewed before the grant is closed out. TFTEA contains specific requirements for STEP, including: Proportional distribution requirement. SBA must distribute grant funds in a way that caps the amount of grant funds distributed to the 10 states with the largest numbers of eligible small businesses at 40 percent of the total amount awarded each year. This requirement ensures that states with fewer eligible small businesses receive funding, and is known as the “proportion of amounts” clause in the law. Total match requirement. States must provide either a 25 percent or 35 percent nonfederal total match to the federal grant amount. Cash match requirement. A state’s match cannot be less than 50 percent cash. First, we found that OIT has established a process for ensuring compliance with the TFTEA requirement outlined in the “proportion of amounts” clause of the statute. OIT officials told us they review data from the Department of Commerce’s Census Bureau that show the number of exporting small and medium-sized businesses in each state, and then use these data to determine the top 10 states. According to OIT officials, they use the most recent data available, with an approximately 2- to 3- year lag. OIT officials told us that they planned to use available 2016 Census data to determine the top 10 states for the fiscal year 2018 award cycle and then, after receiving applications, determine award amounts that would comply with this requirement. Second, we found SBA’s process did not document that states met TFTEA’s total match requirement before grant closeout. TFTEA requires that states provide matching funds, and the total match is typically 25 percent of the combined state-federal amount. At least half of the total match must be cash. Matching share requirements are often intended to ensure local financial participation, and may serve to hold down federal costs. If SBA determines that a state is not providing sufficient matching funds, it can withhold reimbursement for expenses incurred under the grant. Figure 1 illustrates the STEP funding proportions described above. In our report, we identified four instances where, according to OIT’s documentation, states reported insufficient total matches—one in fiscal year 2015 and three in fiscal year 2016. OIT’s documentation showed that these four states failed to meet the required total matching funds by about $76,000 combined over these 2 years of the program. SBA told us they nevertheless had closed these grants. OIT officials provided several explanations for their actions. First, OIT officials told us that of these four states, two submitted additional information after the grant had closed, indicating that the states had met the matching requirement. OIT officials stated that they did not verify the accuracy of the total match information before grant closure because of OIT staff error. With respect to the other two states, OIT initially stated that it was working with OGM to verify that the total match requirement had not been met, and how best to recover the funds. Subsequently, OIT reported OGM’s determination that one state had in fact met the match requirement, but that the other had not. In the case of the state that did not meet the requirement, OGM determined that SBA had overpaid federal funds to that state by about $19,600. However, after contacting the state and looking into the matter further, OGM conducted a review of quarterly reporting documentation for this state, and determined that the state had in fact exceeded its required match by about $3,800. Though all four of the states initially identified were eventually determined to have met the total match requirement, SBA did not have an adequate process in place to ensure documentation of a full match before grant closeout. Federal internal control standards state that management should design control activities. By designing and executing appropriate control activities, management helps fulfill its responsibilities and address identified risks in the internal control system. Without a process for effectively documenting that the total match requirement has been met and reviewing this documentation before grant closeout, SBA does not have reasonable assurance that states have complied with TFTEA’s total match requirement, and risks overpayment of federal funds. Third, we found that OIT’s process does not provide reasonable assurance that states have complied with the TFTEA cash match requirement. As previously noted, TFTEA requires that states provide at least half of their total match in the form of cash. TFTEA allows for the remaining half to be any mixture of cash, in-kind contributions, and indirect costs. OIT collects information about the types of expended matching funds, including the proportion provided in cash; however, OIT does not have a process in place to use this information to monitor states’ compliance with this requirement. OIT documents show that while proposed cash match amounts are recorded, OIT does not track or analyze states’ expended cash matching funds during or at the close of the grant cycle. OIT officials told us that this information is included in the states’ quarterly detailed expenditure worksheets, and therefore can be reviewed for compliance on a case-by- case basis. However, OIT program officials told us that they do not regularly analyze this information to determine what proportion of the total match the cash portion constitutes. The program’s authorizing legislation does not define “cash,” and neither does the Uniform Guidance. OIT considers the salaries of state trade office staff who work on administering the grant to be a form of cash and, according to OIT officials, most states use state staff salaries as their total match, including the required cash portion. In addition, we found that OIT does not have a process for ensuring that states reporting staff salaries as their required cash match are not also using grant funds from STEP to pay for portions of these same salaries. As such, SBA cannot consistently determine whether states are meeting the TFTEA cash match requirement, and risks closing out grants for which states have not met the cash match requirement. Using part of the grant to cover the cost of the state’s matching requirement in this way could have the effect of reducing the match below the thresholds mandated by TFTEA. In our discussions with officials from 12 low-use states that received STEP grants in fiscal year 2015, 2 states reported using the grant to offset state staff salaries. When we asked OIT officials what process they had in place to determine whether states were using staff salaries paid for with STEP funds as part of their match amount, OIT officials told us that they were not aware that STEP grantees had engaged in this practice, and therefore did not monitor for it. SBA’s grants management standard operating procedure states that the agency should monitor grantees for compliance with the terms and conditions of the awards, which includes compliance with applicable federal law. Further, according to federal standards for internal control, management should design and execute control activities, and use quality information to achieve the entity’s objectives. Management should process reliable data into quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks. Without processes to review whether states are meeting the cash match requirement, OIT is not implementing its responsibilities under SBA’s standard operating procedure because it cannot consistently determine whether states are meeting this requirement. Without making such a determination, SBA does not have reasonable assurance that states are contributing to the program as required by STEP’s authorizing statute. In our report, we recommended that the SBA Administrator should establish a process that ensures documentation of states’ compliance with the total match requirement before grant closeout, and develop a process to determine states’ compliance with the cash match requirement. SBA agreed with these recommendations. Next, we looked at STEP’s grant use rate. In our report, we found that nearly 20 percent of grant funds go unused each year, despite OIT officials stating that they seek 100 percent use of grant funds. Specifically: 2015. Across all 40 recipient states, combined grant use was 81 percent, leaving 19 percent, or nearly $3.4 million, unused. This included one state that left 77 percent, or over $432,000, of its funds unused that year. 2016. Across 41 of the 43 recipient states, combined grant use was 82 percent, leaving 18 percent, or nearly $3.2 million, unused. This included one state that left nearly 95 percent, or nearly $184,000, of its funds unused that year. We found that OIT made some changes to the program that could improve states’ ability to use all their grant funds. Changes included: (1) Extending funds usage period to 2 years. This change allows an additional 4 quarters to conduct program activities, which, in turn, may help enable states to use the full amount of their grant funding and achieve performance targets. (2) Eliminating travel preauthorization requirement. This change may reduce the administrative burden on state trade office staff and allow greater flexibility to use grant funds when opportunities that require travel arise with limited notice. (3) Reducing the length of the technical proposal. This change may help to streamline the program’s application paperwork. We interviewed officials from low-use states to identify the continuing challenges they faced. We grouped the most commonly reported challenges into the following categories: (1) Timing of the application and award processes. State officials discussed the variable and short application timeframes, and said that the award announcement happening close to the start of the grant period can make it difficult to use funds during the 1st quarter of the period. (2) Administrative burden. State officials described challenges due to inflexible application requirements, a difficult process for repurposing funds, and burdensome and changing reporting requirements. (3) Communication. State officials told us this was a challenge because of delays and inconsistent communication of requirements from OIT. In our report, we found that OIT had not assessed and fully addressed the risk posed by some states’ low use of funds. OIT officials told us that while they informally collect feedback from states, there is no systematic process to collect states’ perspectives on challenges with the program, including obstacles to their ability to use funds. Officials said that they seek 100 percent use for each state that receives an award, as well as for the program as a whole. Federal internal control standards specify that agency leadership should define program objectives clearly to enable the identification of risks and define risk tolerances in order to meet the goals of the program’s authorizing legislation. In addition, OIT has no systematic process to share best practices with sufficient detail that states struggling to use their STEP funds might apply those practices to improve their own programs. TFTEA requires SBA to publish an annual report regarding STEP, including the best practices of those states that achieve the highest returns on investment and significant progress in helping eligible small businesses. While 12 states used 75 percent or less of their grant funds in the fiscal year 2015 cycle, 19 states used all or almost all of their funds. SBA publishes high-level information on what it deems to be notable state activities in its annual report to Congress. OIT officials told us that, when possible, they share best practices with states that may have difficulty accessing external markets. However, OIT officials told us that they do not formally facilitate the sharing of best practices among the states, saying that best practices for promoting exports in one state might not be transferable to another state because each state is unique. According to the Uniform Guidance, grant recipients’ performance should be measured in a way that helps the federal awarding agency and other nonfederal entities improve program outcomes, share lessons learned, and spread the adoption of promising practices. We have also previously reported on the importance of collecting and sharing best practices, as well as the processes for doing so. By sharing detailed information with all participating states about the approaches that some grant recipients are using to successfully achieve STEP’s goals, SBA could encourage all grant recipients to improve the effectiveness of their state STEP programs, including increasing fund use rates in pursuit of OIT’s stated aim of 100 percent grant fund use. In our report, we recommended that the SBA Administrator assess the risk to achieving program goals posed by some states’ low grant fund use rates, and that assessing this risk could include examining the challenges that states reported related to the program’s application and award processes, administrative burden, and communication. We also recommended that SBA enhance collection and sharing of best practices among states that receive STEP grant funds. SBA agreed with these recommendations. Chairwoman Finkenauer, Ranking Member Joyce, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this statement, please contact me at (202) 512-8612 or gianopoulosk@gao.gov. Contacts for our Offices of Congressional Relations and Public Affairs are on the last page of this testimony. GAO staff who made key contributions to this statement are Adam Cowles (Assistant Director), Cristina Ruggiero (Analyst in Charge), Martin de Alteriis, Mark Dowling, Jesse Elrod, John Hussey, and Christopher Keblitis. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Small Business Administration's (SBA) management of the State Trade Expansion Program (STEP) does not provide reasonable assurance of compliance with some legal requirements. Specifically, the Trade Facilitation and Trade Enforcement Act of 2015 (TFTEA) requirements for STEP include: Proportional distribution requirement. SBA's Office of International Trade (OIT) must distribute grant funds so that the total amount awarded to the 10 states with the highest percentage of eligible small businesses does not exceed 40 percent of the program's appropriation that year. Total match requirement. States must provide a 25 or 35 percent non-federal match to the federal grant amount. Cash match requirement. A state's match cannot be less than 50 percent cash. GAO found that, while OIT has a process to meet the distribution requirement, it does not have a process for documenting that states have met the total match requirement before grant closeout, and does not have a process to determine whether states are meeting the cash match requirement. Without such processes, SBA cannot be reasonably assured that states are contributing per the law's requirements. GAO found that, while OIT has made changes to STEP in response to states' feedback, officials from states with low grant use described ongoing challenges with the program that affect their ability to fully use funds. These challenges include compressed application and award timelines, administrative burden, and poor communication. SBA has not adequately assessed risks to the program, including the risk to achieving program goals posed by some states' low grant fund use rates. Without such an assessment, OIT's ability to support U.S. exporters may be diminished. Further, SBA has not effectively facilitated sharing best practices among states. By doing this, SBA could help states make full use of funds to achieve the program's goals.", "document_type": "gao"}
{"report": "NASA awarded firm-fixed-price contracts in 2014 to Boeing and SpaceX, valued at up to $4.2 billion and $2.6 billion, respectively, for the development of crew transportation systems that meet NASA requirements and for the initial service missions to the ISS. Figure 1 shows the spacecraft and launch vehicles for Boeing and SpaceX’s crew transportation systems. These contracts encompass the firm-fixed-price design, development, test, and evaluation work needed to support NASA’s certification of the contractors’ spacecraft, launch vehicle, and ground support systems and begin operational missions to the ISS. The Commercial Crew Program manages two processes in order to support the contractors’ uncrewed test flight, crewed test flight, and certification milestone. The contractors must submit evidence, which the Commercial Crew Program must review and approve for both processes. A three-phased safety review process informs the program’s quality assurance activities and is intended to ensure that the contractors have identified all safety-critical hazards and implemented associated controls prior to the first crewed test flight. In phase one, the contractors identify risks in their designs and develop reports on potential hazards, the controls they put in place to mitigate them, and explanations for how the controls will mitigate the hazards. In phase two, the program reviews and approves the contractors’ hazard reports and develops strategies to verify and validate that the controls are effective. In phase three, the contractors will conduct the verification activities and submit the hazard reports to the program for approval. The verification closure notice process is used to verify that the ISS requirements, applicable to any spacecraft flying to the ISS, and Commercial Crew Program requirements. After the contractor has successfully completed its uncrewed and crewed test flights and the above processes, the program determines at the contractor’s certification milestone whether the crew transportation system meets NASA’s requirements for human spaceflight. Following this contract milestone is an agency certification review, which authorizes the use of a contractor’s system to transport NASA crew to and from the ISS. It is at this point that the contractors can begin operational missions. Figure 2 shows the path leading to operational missions. Both contractors have made progress building and testing hardware, including SpaceX’s uncrewed test flight. But continued schedule delays and remaining work for the contractors and the program create continued uncertainty about when either contractor will be certified to begin conducting operational missions to the ISS. The program has made progress reviewing the contractors’ certification paperwork, but contractor delays in submitting evidence for NASA approval may compound a ‘bow wave’ of work, which creates uncertainty about when either contractor will be certified. NASA acknowledged the schedule uncertainty in February 2019, when it announced plans to purchase two additional Soyuz seats from Russia, citing concerns about the difficulties associated with achieving first flights in the final year of development. Both contractors are building several spacecraft, some of which are near completion. Each contractor’s spacecraft includes two main modules: Boeing’s spacecraft—CST-100 Starliner—is composed of a crew module and a service module. The crew module will carry the crew and cargo. It also includes communication systems, docking mechanisms, and return systems for Earth landing. The service module provides propulsion on-orbit and, if needed, in abort scenarios—when a failure prevents continuation of the mission and a return is required for crew survival—as well as radiators for thermal control. SpaceX’s spacecraft—Dragon 2—is composed of a capsule, which we refer to as the crew module, and a trunk, which we refer to as the support module. The crew module will carry the crew and cargo. It also includes avionics, docking mechanisms, and return systems for a water landing. The support module includes solar arrays for on-orbit power and guidance fins for escape abort scenarios. Different spacecraft will be used for the uncrewed test flight and the crewed test flight, as well as to support other test events. See table 1 for a description of each contractor’s hardware builds, current status, and upcoming events. Additional details on select hardware testing follow. In June 2018, Boeing experienced an anomaly while testing its launch abort engines. During a test firing, four of the eight total valves in the four launch abort engines failed to close after a shutdown command was sent. In response to this event, Boeing initiated an investigation to identify the root cause. According to Boeing officials, Boeing plans to replace components on all of its service modules except for the uncrewed test flight service module. This is because the abort system will not be active for the uncrewed test flight. Boeing plans to resume testing its launch abort engines in May 2019. A NASA official told us that addressing this anomaly and identifying its root cause resulted in a 12-month schedule delay to launch abort propulsion system testing. In March 2019, SpaceX conducted its uncrewed test flight, which demonstrated that the capsule could dock with the ISS and return to Earth. NASA officials described SpaceX’s uncrewed test flight as a success with key systems such as the guidance, navigation, and control and the parachutes performing as expected. A SpaceX official told us that this was a very successful test and represented significant risk reduction from a schedule and technical perspective. Subsequently, the spacecraft used in the uncrewed test flight was destroyed in a testing anomaly. The anomaly occurred during a test that SpaceX was conducting in advance of an in-flight abort test scheduled for this summer. As of May 2019, SpaceX was investigating the anomaly. Continued schedule delays create uncertainty about when NASA will certify either contractor to begin conducting operational missions to the ISS. We have previously found that the contractors’ schedules regularly changed, and this pattern continues. As of May 2019, both contractors have delayed their certification milestone nine times since establishing dates in their original contracts. In the span of less than a year, since our July 2018 report, Boeing has again delayed its certification milestone four times and by 12 months, while SpaceX has again delayed its certification milestone three times and by 7 months. Both contractors are now planning for certification to occur more than 2 years beyond the original dates in their contracts–Boeing in January 2020 and SpaceX in September 2019, though this date is under review and could further slip (see figure 3). Over time, both program and contractor officials have told us that they struggle to establish stable schedules. In 2018, the Commercial Crew Program manager told us that she relied on her previous experience to estimate schedule time frames as opposed to relying on the contractors’ schedules, which were overly optimistic. In March 2019, a senior NASA official told us that the agency has struggled to establish schedules with both contractors, often needing to negotiate dates with senior company officials. Further, SpaceX officials explained that they would not know the schedule for the crewed test flight until they conducted the uncrewed test flight. However, even having conducted the uncrewed test flight in March 2019 and before the April 2019 anomaly, SpaceX and NASA were still re- evaluating the schedule for the crewed test flight. Both contractors are continuing to mitigate technical risks identified by program officials that need to be addressed in order to reach certification. The program will close a risk when the contractor is able to fully mitigate it. If all mitigation activities are exhausted, but a risk still remains, the program will determine if the risk is acceptable as part of the agency’s rationale for flight. As the contractors address these technical risks and proceed through integration and testing, any issues that arise during testing or the test flights could further delay certification. Program risks for Boeing include: Parachute System Certification. Boeing is conducting five parachute system qualification tests to demonstrate that its system meets the Commercial Crew Program’s requirements, which will be validated on two spacecraft flight tests. However, in August 2018, Boeing identified a faulty release mechanism for its drogue parachute—which initially slows down the capsule—during its third parachute qualification test that successfully deployed all parachutes. Identifying and fixing the faulty mechanism delayed its fourth parachute qualification test. According to a NASA official, Boeing is conducting testing to qualify an alternative design, and Boeing must qualify this alternative design before the crewed test flight. Launch Vehicle Engine Anomaly. Boeing is addressing a safety risk related to a launch vehicle component. Specifically, during a 2018 launch, the launch vehicle engine position during ascent deviated from commands but the launch vehicle provider stated that it achieved all mission objectives. Program officials told us that they have insight into the launch vehicle manufacturer’s ongoing investigation and have participated in a separate independent review team. Boeing will implement a set of corrective actions for the uncrewed test flight, and will continue testing the engines for the crewed test flight. Spacecraft-Generated Debris. Boeing is addressing a risk that under normal operating procedures the initiators that trigger separation events, such as the separation of the crew and service module prior to re-entry, may generate debris and damage the spacecraft. These components function as expected, but Boeing plans to install hardware to contain debris generated when the initiators fire. Program officials told us that they believe Boeing has identified a solution that will be sufficient for the uncrewed and crewed test flights, but the program is continuing to explore a possible redesign for future operational missions. Spacecraft Forward Heat Shield. We had previously found that Boeing was addressing a risk that during descent a portion of the spacecraft’s forward heat shield may re-contact the spacecraft after it is jettisoned and damage the parachute system. Since our last report, Boeing tested the performance of the forward heat shield in worst-case scenarios and found there was no damage to the parachute system or the spacecraft. After reviewing test data, the program determined that Boeing had completed the mitigation activities and, as of February 2019, no additional steps were needed. Program risks for SpaceX include: Parachute System Certification. Like Boeing, SpaceX is conducting several parachute tests to demonstrate that its system meets the Commercial Crew Program’s requirements. However, SpaceX experienced two anomalies with its parachute system in August 2018. As a result, a SpaceX official told us they enhanced the parachute design to improve robustness. NASA officials told us SpaceX’s enhanced parachutes performed well on its uncrewed test flight. Prior to the crewed test flight, SpaceX must demonstrate the performance of its parachute system. SpaceX plans to continue to test its parachutes, and according to a SpaceX official, will take all steps necessary to ensure that the flight design meets or exceeds minimum performance levels. Propellant Loading Procedures. SpaceX is continuing to address a safety risk related to its plans to conduct launch vehicle propellant loading procedures after the astronauts are on board the spacecraft. SpaceX officials told us that this loading process has been used in other configurations for multiple SpaceX flights. The Commercial Crew program has approved SpaceX’s proposed loading procedures, including the agreed upon demonstration of the loading procedure five times from the launch site in the final crew configuration before the crewed test flight. The five events include the uncrewed test flight and in-flight abort test. As of March 2019, SpaceX had completed the first two events. Redesigned Composite Overwrap Pressure Vessel. SpaceX is continuing to address a risk that its launch vehicle’s redesigned composite overwrap pressure vessel, which is intended to contain helium under high pressure, may serve as an ignition source. The program and SpaceX conducted tests on the redesigned vessel and the program determined that all possible ignition sources, with one exception, have a low likelihood of creating ignition. The program continues to assess this ignition source. According to a NASA official, there were no indications of any issues during SpaceX’s uncrewed test flight. SpaceX officials also told us that the redesigned vessel has successfully flown on multiple flights. The program will need to determine whether to accept the risk associated with this technical issue prior to SpaceX’s crewed test flight. Engine Turbine Cracking. NASA continues to assess a SpaceX risk related to the design of its launch vehicle engines, which has previously resulted in the turbine wheel cracking. To mitigate the turbine cracking risk, SpaceX conducted additional qualification testing and developed an operational strategy that resulted in no cracks. Consequently, the program accepted this risk for SpaceX’s uncrewed test flight but levied a constraint on the crewed test flight. Specifically, SpaceX has agreed to conduct a follow-on test campaign of the engines to demonstrate that it meets NASA’s standards in order to launch its crewed test flight. Program officials said SpaceX plans to build the launch vehicle engines for its crewed test flight concurrently with this follow-on testing series. The Commercial Crew Program’s ability to process certification data packages for its two contractors continues to create uncertainty about the timing of certification. Specifically, the program is concurrently reviewing and approving both contractors’ phased safety reviews and verification closure notices. We previously reported that program officials, the contractors, and independent review organizations had concerns about a “bow wave” of work for the program. For example, at that time, the program’s safety and mission assurance office identified the upcoming bow wave of work in a shrinking time period as a top risk to achieving certification. Three-Phased Safety Reviews. The program continues to make progress conducting its phased safety reviews, but it has not yet completed them. In February 2017, we found that the program was behind schedule completing its phased safety reviews and, as of April 2019, it had yet to complete this process. As shown in Table 2, the program is near completion of phase two reviews and phase three reviews are in progress. Program officials told us that they have started work on many of the phase three safety reviews, but the data only reflect their efforts once they complete a phased safety report in its entirety. Any additional delays to complete this process, however, would delay the crewed test flights and create uncertainty about when NASA will certify the contractors to begin operational flights. Verification Closure Notices. NASA has made progress verifying that the contractors have met ISS and Commercial Crew Program requirements, but much work remains. When a contractor is ready for NASA to verify that it has met a requirement, such as that the contractor’s system can detect and alert the crew to critical faults that could result in a catastrophic event, the contractor submits data for NASA to review through a verification closure notice. Table 3 shows the agency’s progress approving verification closure notices for each contractor. Program officials told us that, because the contract solicitation did not require an uncrewed test flight, they had not previously determined the minimum number of Commercial Crew Program requirements that the contractors should meet prior to an uncrewed test flight. Subsequently, both contractors included an uncrewed test flight as part of their schedules. As these test flights approached, NASA determined that it must verify that the contractors met approximately 20 percent of the program’s requirements before the contractors’ uncrewed test flight and the remaining 80 percent before the contractors’ crewed test flights. The program made this determination based on ensuring the contractors met requirements related to the spacecraft safely approaching and docking to the ISS; ensuring the safety of the ISS and its crew; and meeting any mission-specific requirements for cargo. Both contractors originally planned for the program to verify they had met more than 20 percent of the Commercial Crew Program requirements before the uncrewed test flight but have subsequently changed their plans. For both contractors, the program is allowing the contractors to submit more verification closure notices between the uncrewed and crewed test flight than initially envisioned. Program officials told us that contractors proposed deferring the submission of verification closure notices because they were having difficulties meeting the original targets. Figure 4 includes SpaceX and Boeing’s original and current plans for verification of requirements compared to the Commercial Crew Program’s minimum level of requirements it determined was necessary for the uncrewed test flight. As reflected in the figure, these new plans, which defer submission of work to the crewed test flight, may compound the program’s bow wave of work and create uncertainty about the timing of certification. Further, the Commercial Crew Program will need to reassess a subset of requirements closed for the uncrewed test flight prior to the crewed test flight. For example, of the 78 requirements Boeing plans to close prior to the uncrewed test flight, the program will re-assess 16; for SpaceX’s 49 requirements, the program will re-assess 32. Program officials told us that some of this work is expected based on known changes to the contractors’ systems between the uncrewed and crewed test flight. For example, officials told us that they approved a verification closure notice for SpaceX’s air conditioning system in order to support the uncrewed test flight, but they know that they will need to re-assess it because SpaceX is making changes before its crewed test flight. While these types of changes and those that are identified through testing are not uncommon, they further add to the program’s workload and create uncertainty about the timing of certification. Among the requirements that must be closed before the crewed test flight is loss of crew, which is a metric that captures the probability of death or permanent disability to one or more crew members. According to program risk charts, the program’s top safety risk continues to be that neither contractor will meet the contractual requirement of a 1 in 270 probability of incurring loss of crew. We previously found that NASA lacked a consistent approach for how to assess loss of crew and recommended that key parties, including the program manager, collectively determine and document how the agency will determine its risk tolerance level prior to certifying either contractor. NASA partially concurred with that recommendation, stating that, if neither contractor can meet the loss of crew requirement, the program will request a waiver through the human rating certification process to ensure transparency. As of March 2019, NASA officials told us they have not taken steps to address this recommendation. Officials told us that the Commercial Crew Program is currently reviewing Boeing’s loss of crew verification closure notice and SpaceX’s draft verification closure notice in order to verify if the contractors have met the loss of crew requirement. According to program officials, one of the biggest challenges for the program is balancing its workload to support the two contractors, but officials are making an effort to review each contractor’s data products as they are submitted. For example, program officials told us that they were able to review SpaceX submissions during the summer of 2018, while Boeing’s submissions slowed as it focused on addressing the test anomaly with its launch abort engines. However, based on current schedules, the program must complete its reviews of certification paperwork while supporting uncrewed, crewed, and abort system test flights for both contractors before the end of 2019. Both contractors said they have concerns about NASA’s ability to maintain its pace of processing paperwork in order to support the contractors’ planned test flights and certification dates. The potential bow wave of work continues to create uncertainty about the timing of certification for either contractor, which could result in delays to the first operational mission to the ISS. In February 2019, NASA announced plans to buy two more Soyuz seats from Russia, thereby acknowledging that delays to certification of the Commercial Crew Program contractors could continue. These seats would extend U.S. access to the ISS from November 2019 through September 2020. According to a senior NASA official, NASA is not purchasing a new Soyuz spacecraft, which we have previously found requires a 3-year lead time. Instead, two additional seats became available on existing vehicles after changes to the Soyuz manifest. In 2015, NASA paid approximately $82 million per seat through its contract with the Russian Federal Space Agency (Roscomos). Program officials stated they could not publicly disclose the price NASA paid for these two new additional seats, but noted that the cost was 5 percent higher per seat than the previous contract modification to purchase Soyuz seats and is consistent with inflation. In addition, NASA plans to extend the duration of Boeing’s crewed test flight. In March 2018, NASA modified its contract with Boeing to allow NASA to add a third crew member and extend the length of the crewed test flight. In July 2018, we reported that NASA was considering this option as one way to maintain a U.S. presence on the ISS, but noted it had limited usefulness if Boeing’s crewed test flight slipped past the return date of the last Soyuz flight. NASA’s actions—purchasing the two additional Soyuz seats and implementing an extended duration crewed test flight for Boeing—do not fully address our July 2018 recommendation to develop and maintain a contingency plan for ensuring a presence on the ISS until a Commercial Crew Program contractor is certified. NASA concurred with this recommendation but, to fully implement it, NASA needs to provide additional support regarding planning efforts to ensure uninterrupted access to the ISS if delays with the Commercial Crew Program contractors continue beyond September 2020. Continued NASA attention on this issue is needed given the uncertainty associated with the final certification dates. We provided a draft of this product to NASA for comment. In its response, reproduced in appendix I, NASA generally agreed with our findings and included an update on the progress made by Boeing and SpaceX. NASA also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to NASA Administrator and interested congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. Cristina T. Chaplain at (202) 512-4841 or chaplainc@gao.gov. In addition to the contact named above, Molly Traci, Assistant Director; Lorraine Ettaro; Laura Greifner; Kurt Gurka; Joy Kim; Christopher Lee; Katherine Pfeiffer; Roxanna T. Sun; Hai Tran; Kristin Van Wychen; and Alyssa Weir made significant contributions to this report.", "summary": "In 2014, NASA awarded two firm-fixed-price contracts to Boeing and SpaceX, worth a combined total of up to $6.8 billion, to develop crew transportation systems and conduct initial missions to the ISS. In July 2018, GAO found that both contractors continued to delay their certification dates and that further delays were likely. NASA must certify the contractors' crew transportation systems before the contractors can begin operational missions to the ISS. The contractors were originally required to provide NASA all the evidence it needed to certify that their systems met its requirements in 2017. The House Committee on Appropriations included a provision in its 2017 report for GAO to continue to review NASA's human space exploration programs. This is the latest in a series of reports addressing the mandate. This report examines the extent to which the Commercial Crew Program and its contractors have made progress towards certification. To do this work, GAO analyzed contracts, schedules, and other documentation and spoke with officials from the Commercial Crew Program, Boeing, and SpaceX. Both of the Commercial Crew Program's contractors, Boeing and SpaceX, have made progress on their crew transportation systems. However, neither is ready to begin carrying astronauts into space as both continue to experience delays to certification. Certification is a process that the National Aeronautics and Space Administration (NASA) will use to ensure that each contractor's spacecraft, launch vehicle, and ground support systems meet its requirements for human spaceflight before any operational missions to the International Space Station (ISS) can occur. Factors contributing to schedule uncertainty include: Fluctuating schedules. As the contractors continue to build and test hardware—including SpaceX's March 2019 uncrewed test flight— their schedules for certification change frequently. As of May 2019, both contractors had delayed certification nine times, equating to more than 2 years from their original contracts (see figure). This includes several delays since GAO last reported in July 2018. Program Workload. NASA's ability to process certification data packages for its two contractors continues to create uncertainty about the timing of certification. The program has made progress conducting these reviews but much work remains. In addition, the program allowed both contractors to delay submitting evidence that they have met some requirements. This deferral has increased the amount of work remaining for the program prior to certification. In February 2019, NASA acknowledged that delays to certification could continue, and announced plans to extend U.S. access to the ISS through September 2020 by purchasing seats on the Russian Soyuz vehicle. However, this arrangement does not fully address GAO's July 2018 recommendation to develop a contingency plan for ensuring access to the ISS until a Commercial Crew Program contractor is certified. NASA concurred with the recommendation but has not yet implemented it. Continued NASA attention on this issue is needed given the uncertainty associated with the final certification dates. GAO continues to believe that NASA should develop a contingency plan to ensure uninterrupted access to the ISS if delays persist beyond September 2020. NASA generally agreed with GAO's findings.", "document_type": "gao"}
{"report": "The Navy oversees the planning and execution of non-nuclear surface ship repair and maintenance through several organizations (see fig. 1), including the following: The Chief of Naval Operations is the senior military officer of the Department of the Navy and is responsible to the Secretary of the Navy for the command, utilization of resources, and operating efficiency of the operating forces of the Navy and of the Navy shore activities assigned by the Secretary. The Assistant Secretary of the Navy for Research, Development and Acquisition, as the Navy Acquisition Executive, has overall authority, responsibility, and accountability for all acquisition and sustainment functions and programs, including surface ship repair and maintenance. Naval Sea Systems Command (NAVSEA) and its subordinate organizations maintain ships to meet fleet requirements within cost and schedule parameters, among other duties for combat systems design and operation. The Navy contracts with private shipyards and other firms—collectively known as the ship repair industrial base—for the repair and maintenance of non-nuclear surface ships. This work may be performed in either government-owned or contractor-owned facilities, potentially including shipyards with piers, cranes or facilities for pipefitting and valve repair. Certain types of work, such as inspecting, repairing or otherwise maintaining a ship’s hull, might require placing a ship in the ship repair contractor’s dry dock. Ship repair availabilities can range from a few weeks to years depending on the extent of work required and degree of complexity. The types of availabilities include the following: Chief of Naval Operations (CNO) availabilities accomplish major repair work. This includes industrial maintenance requiring complex processes to complete restorative work, such as structural, mechanical, and electrical repairs. These may include modernization work to upgrade a ship’s capabilities along with repair work, and can last for over a year. Larger contractors typically execute these types of availabilities rather than small businesses. Continuous Maintenance availabilities (CMAV) accomplish non- major repair work, which includes routine maintenance work requiring relatively little time compared to CNO availabilities—typically only weeks to a few months in duration. Small business contractors commonly execute CMAVs, and, at some ports, larger companies that have contracts for CNO availabilities also take on this type of work. Within NAVSEA, several organizations oversee MAC-MO strategy implementation (see fig. 2), including key functions such as contract administration, program management, and planning for future availabilities. Prior to awarding a contract for ship repair work under MAC-MO, the Navy plans and defines requirements for upcoming availabilities as depicted in figure 3 below. Contracting for availability execution under the MAC-MO strategy differs from that under the Navy’s previous strategy, known as Multi-Ship, Multi- Option (MSMO), in several key ways, including by calling for: establishment of fixed contractual prices and completion time frames for an upcoming availability, rather than payment of contractors’ incurred costs; use of a third-party planning contractor under a cost-reimbursement contract to define contract specifications, rather than relying on planners employed by ship repair contractors; and award of indefinite delivery contracts to multiple contractors that can then compete for future availabilities, rather than all availabilities for a particular class of ships going to one contractor. Under the MAC-MO strategy, the Navy normally places fixed-price orders for availabilities with expected durations of 10 months or less using indefinite delivery/indefinite quantity (IDIQ) contracts. IDIQ contracts do not specify exact times for delivery of supplies or services at contract award; the Navy establishes those via orders placed during contract performance. With MAC-MO, the Navy generally solicits and awards contracts for five-year periods to a set of qualified contractors at specific home ports. These periods include an initial execution year and four additional option years. As a result, several qualified contractors are available to subsequently compete for availabilities in a specific home port under firm-fixed-price availability delivery orders until contract expiration. Availabilities that the Navy expects to last more than 10 months are not restricted to the ships’ home port. This allows for contractors outside the home port to compete for this work. The Navy then awards contracts for these coast-wide availabilities as stand-alone contracts to a single prime contractor, potentially at a port different from the home port of the ship. Shorter availabilities may be limited to the home port area provided there is adequate competition, which the Navy defines as the presence of two or more qualified bidders. If adequate competition is not available in the home port area, the geographic area for solicitation is expanded equally in all directions until adequate competition exists. Figure 4 below depicts contracting processes used for soliciting and awarding work under the MAC-MO strategy. In November 2016, we reviewed the Navy’s implementation of the MAC- MO strategy through pilot maintenance periods, including its potential benefits and effects on the industrial base. We found MAC-MO had some potential benefits compared to the previous MSMO contracting strategy, including increased opportunities for competition and control of costs through fixed-price contracts. We additionally found that some contractors saw uncertainty associated with the need to continually compete for work, which could result in decisions to reduce their workforce and facilities, and the stability of ship repair workloads in their ports, irrespective of contract type. The Navy has achieved some, but not all, of the goals it set under the MAC-MO strategy. Among the achievements, the Navy provides more opportunities for competition—and received more offers—under MAC-MO than under the prior strategy. Further, MAC-MO’s fixed-price contracts help enable the Navy to ensure quality of work, and we found no evidence of deficient work at availability completion in our review of four completed case studies. At the same time, the Navy also desired improved availability cost and schedule outcomes under the MAC-MO strategy. The Navy’s results in these two areas have been mixed. Through April 2019, the Navy had completed 41 CNO availabilities under its MAC-MO strategy with, on average, 5 percent cost growth and 30 percent schedule growth. Unplanned work, which can often be unavoidable in ship repair, has detracted from both cost and schedule performance. The MAC-MO strategy has provided more opportunities than MSMO for competition by awarding a delivery order for each ship repair availability. The Navy has competed over 500 delivery orders under the MAC-MO strategy from April 2015 to March 2019. This represents a departure from the MSMO strategy under which a single contract was awarded to one contractor to execute multiple availabilities for a class of ship. The MAC- MO strategy also allows small businesses in Norfolk and San Diego to compete for noncomplex maintenance. Previously, under the MSMO strategy, small businesses said that they were more likely to work as subcontractors for the businesses that held one of the MSMO contracts. Navy officials have since stated that small businesses are now acting as prime contractors. The Navy has also achieved competition for soliciting its delivery orders under the MAC-MO strategy. According to our analysis of data from the Federal Procurement Data System-Next Generation (FPDS-NG), from the start of the MAC-MO strategy (April 2015) through March 2019, at least 78 percent (435 of 554) of MAC-MO awards solicited within home ports received two or more offers. Further, in the 18 percent of instances when the Navy awarded a delivery order after receiving only a single offer, it may have attained the benefits of having solicited that delivery order in a competitive environment. Table 1 shows the number of offers for both complex and noncomplex MAC-MO awards through March 2019. The MAC-MO strategy also gives the Navy flexibility to ensure that a contract’s quality requirements are met under a fixed price by the time of availability completion. The Navy identified improving the quality of workmanship as a goal when it switched from MSMO to MAC-MO. The previous MSMO contracting strategy relied on use of cost-reimbursement contracts, which only require the government to reimburse the contractor its allowable incurred costs, regardless whether the contractor completed the work. The MAC-MO strategy uses firm-fixed-price contracts, which provide for a price that is not subject to any adjustment on the basis of the contractor’s cost experience in performing the contract. This contract type places upon the contractor maximum risk and full responsibility for all costs and performance, including meeting the quality requirements of the contract. NAVSEA officials stated that in the event that the contractor doesn’t meet the quality terms of the contract, the Navy has two options: (1) require the contractor to complete the deficient work, at the contractor’s cost, to meet the specifications or (2) reduce the contract price to reflect the reduced value of the services performed and descope the related work requirements from the existing contract for performance on a future availability. Our review of availability completion reports from the four case study availabilities that were complete at the time of our assessment, out of six total, showed no instances where the Navy accepted quality deficiencies at availability completion. Navy contracting officials stated that in a firm- fixed-price contracting environment, they would not agree to accept deficient work without first obtaining concessions from the performing contractor, which would require modifying the delivery order. In one of these availabilities, we found evidence that the Navy elected to descope a non-option work item and defer it to a future availability. The maintenance team stated that this decision followed poor planning of the work item, which would have caused delays in completing the availability if not deferred. We also discussed these four availabilities with the responsible Navy maintenance teams, and none of those teams reported to us any deficient work at the time each availability completed. Between April 2015, when the Navy implemented the MAC-MO strategy, and April 2019, the Navy completed 41 CNO availabilities with an average cost growth per availability of 5 percent, or $1.7 million in fiscal year 2020 dollars. However, more than half of these availabilities (21 of 41) were completed at a lower cost than the Navy initially estimated. The cost growth of the remaining CNO availabilities (20 of 41) ranged between 1 percent and 78 percent and drove the aggregate average increase. Figure 5 shows the variation in cost performance, or the actual cost compared to the Navy’s estimate, for the 41 CNO availabilities. Figure 6 shows the cost performance, or actual cost compared to the Navy’s estimate, for the 41 CNO availabilities grouped by their location. Figure 7 shows cost performance, or actual cost compared to the Navy’s estimate, for the 41 CNO availabilities grouped by ship class. Between the start of the MAC-MO strategy in April 2015 and April 2019, the Navy completed 41 CNO MAC-MO ship repair availabilities with an average schedule growth, or actual number of days from availability start to completion, compared to the Navy’s estimate, of 30 percent, or 64 days. Twelve of 41 availabilities finished on time, and none finished ahead of schedule. In addition, two availabilities more than doubled in length, with one finishing with 123 percent schedule growth. We discuss some factors that can contribute to schedule growth below. Figure 8 shows the schedule growth for individual CNO availabilities. Figure 9 shows the schedule growth for the 41 CNO availabilities grouped by location. Figure 10 shows the schedule growth for the 41 CNO availabilities grouped by ship class. Navy officials stated that one potential source of delays is unplanned work, which consists of both growth work and new work. The Navy defines growth work as additional work that is identified or authorized after contract award that is related to a work item included in the original contract. We previously found that growth work contributed to cost and schedule increases, and it remains a contributing factor. Navy officials stated they expect some growth work in availabilities, as officials stated that certain tasks are difficult to fully scope within the original contract. As an example, one official stated that they cannot fully inspect ballast tanks and accurately write work specifications for their repair until the ship is at the repair yard and the availability has begun. Alternatively, the Navy defines new work as any additional work that is identified or authorized after contract award that is not related to a work item included in the original contract. Maintenance team officials stated that new work can originate when an item that needs repair breaks or the maintenance team first discovers it after the Navy awards the contract. The Navy can also add new work to an availability whenever it sees fit. In our six case study availabilities, we found that five added growth work, including examples of growth items that the Navy considered unavoidable. Our analysis of RMC data showed that the USS Stout (DDG 55) CNO Availability had 60 instances of growth work that the Navy considered unidentifiable prior to the start of the availability, including welding for the fuel tanks and repair to the bulkheads. The maintenance team did not consider these growth items to be unusual. Some non-CNO availabilities, like Continuous Maintenance availabilities, are smaller in scope and less susceptible to growth work. Maintenance team officials at SERMC consequently stated that they can often complete CMAVs on schedule. We found that the Navy completed one of our case study availabilities, the USS Iwo Jima CMAV, on schedule, and maintenance team officials stated they had time to add three new work items to the availability. Figure 11 describes the USS Stout (DDG 55) case study. Figure 12 describes the USS Iwo Jima (LHD 7) case study. According to Navy officials, managing growth work under firm-fixed-price contracts has contributed to schedule delays. In our November 2016 report on the Navy’s transition to the MAC-MO strategy, we described the importance of contractors and RMC staff negotiating contract changes and agreeing on costs in a timely manner in order to minimize schedule impact. In our current review, Navy officials stated that negotiating change orders for unplanned work under MAC-MO is more difficult and time consuming than under the prior MSMO strategy because the Navy can no longer direct the contractor to continue to work without agreeing on the cost. In one of our case study availabilities, the USS Whidbey Island (LSD 41) CNO Availability, the maintenance team officials stated that they had difficulties negotiating contract changes. As a result, the officials stated that the Navy used unilateral modifications to direct the contractor to execute growth work items and avoid further schedule disruptions. See Figure 13 below for more detail on the USS Whidbey Island (LSD 41) case study. The Navy recognizes the negative schedule outcomes it currently faces with MAC-MO strategy implementation and has worked to mitigate them. It has implemented new contracting provisions and is moving key availability milestones to earlier in the process in an effort to better plan availabilities and facilitate their on-time completion. The Navy has also tried to better coordinate with the third-party planner to plan for availabilities and improve schedule performance. Statutory requirements and their implementation, however, have hindered the Navy’s ability to further mitigate schedule delays. Specifically, the Navy must obtain approval from the Under Secretary of Defense (Comptroller) before funding growth work that occurs in subsequent fiscal years and exceeds $4 million—an amount established under a 1990 law. Late last year, Congress established a pilot program in fiscal year 2020 that affords the Navy the ability to use procurement funds for availabilities, and these funds remain available for obligation for three years. A congressional statement accompanying the appropriations law that established the pilot program states that the Navy is to submit quarterly reports on the execution of ship availabilities funded through the pilot program. In our November 2016 report, we identified several key lessons learned stemming from MAC-MO pilot maintenance availabilities. When we revisited these lessons learned during interviews with Navy officials, they discussed two persistent MAC-MO strategy attributes that remain points of emphasis for lessons learned from 2016. These strategy attributes, namely the use of firm-fixed-price contracts and the use of a third-party planner, led to two new key lessons learned and another ongoing lesson learned from our 2016 report. Most of these center on the importance of the Navy accurately planning for and anticipating needs during availabilities in order to avoid schedule delays—a theme that was also evident in our November 2016 report. According to NAVSEA leadership officials, the Navy primarily relies on two activities to determine lessons learned and identify actions that NAVSEA needs to take to improve ship repair maintenance, including under the MAC-MO strategy. Surface Team One compiles lessons learned that the individual RMCs recommend and reviews the implementation and status of actions to address those lessons learned. Performance to Plan (P2P) is a data-centric, analytical approach the Navy uses for a variety of improvement initiatives, including ship maintenance, to clearly characterize availability performance goals and develop solutions to improve availability duration outcomes. As shown in Table 2 below, the Navy has developed new contracting provisions and milestones to respond to lessons learned the Navy has identified. Additional information on each action follows the table. In 2018, the Navy began implementing two new contract provisions originating from lessons learned regarding the MAC-MO strategy—Small Dollar Value Growth and Level of Effort to Completion—in an effort to mitigate schedule delays typically associated with growth work. The November 2018 SDVG provision specifically addressed schedule delays due to growth items that cost $25,000 or less. Under SDVG, during availability planning the Navy and contractor agree on a set price to be used anytime a growth work item equal to or under the $25,000 threshold is added to the work specification. This provision eliminates the need for the Navy and the contractor to engage in time-consuming negotiations on small dollar items during the availability. According to the Navy’s 2018 biennial assessment, small dollar growth work negotiations accounted for around 70 percent of all contract changes. According to Navy documentation, contract negotiations for small dollar growth work caused delays of up to a week. In our discussion with officials from the USS Whidbey Island maintenance team, they reported that the availability required 972 contract changes, which they suggested SDVG would have helped expedite. The Navy’s SDVG policy memo states that in using SDVG, the contractor can now typically begin work on the growth item 24 hours after discovery. Figure 14 describes the SDVG process. While it can expedite work on smaller dollar value items, the use of SDVG carries cost risk for the Navy and the executing contractors, which RMC leadership officials and contractor representatives acknowledged. According to these officials, under SDVG the Navy, at times, will likely pay more for growth items than it would if it devoted increased time to negotiate prices, with the same being true for the contractors. For example, the Navy awarded a contract delivery order for the USS Bulkeley (DDG 84) availability in February 2019 that included SDVG. The SDVG line item provided for up to 291 changes for growth during that availability at a firm-fixed-price of $7,144 per change based on historical needs of similar availabilities. This meant that the Navy could use SDVG up to 291 times during the availability, and each of those growth items would cost the Navy $7,144 regardless of whether the actual cost to the contractor underran or exceeded that amount. After the contractor identifies the in-scope growth item, the Navy only must determine that the cost is equal to or less than the $25,000 threshold in the contract. Nonetheless, Navy officials expressed that the benefit of significantly decreased negotiation time outweighs the potential cost risk. As reflected in table 2, the Navy implemented a second new contract change process, known as LOE to Completion, in November 2018. This process is used for growth work items when the price exceeds the SDVG threshold of $25,000. LOE to Completion allows the Navy, within the already awarded contract for the availability, to fund growth work that contractors regularly discover during availability execution without having to separately negotiate each item. Through LOE to Completion, RMC leadership officials stated they have decreased negotiations and schedule delays during availability execution. LOE to Completion allows the Navy to obligate funding for labor-hours and material costs for estimated growth work at the time of award, rather than having to obtain appropriate funds after repair work begins. The Navy can then use those labor-hours and materials for individual growth work items over the course of the availability. According to RMC leadership officials, this provision allows them to avoid incurring additional delays. To establish the amounts of funding, the Navy reviews historical cost for growth work by class type and whether the availability is a docking or non-docking availability. For example, the Navy provided up to 134,002 work hours and $1.4 million for materials under the LOE to Completion contract process for the USS Bulkeley (DDG 84) availability. Figure 15 describes the LOE to Completion process. Because the Navy just recently implemented this process in November 2018, it has collected only limited data to date on its effectiveness. However, as described in figure 16, an availability involving complex ship repair work for the USS Princeton (CG 59) included contract terms that Navy officials described as a precursor to LOE to Completion. In August 2019, the Navy began targeting award of delivery orders for individual availabilities 120 days prior to the scheduled work start date. Previously, the Navy awarded these delivery orders 60 days prior to the scheduled work start date. According to Navy supply officials, awarding the delivery orders 120 days prior to the start of scheduled work allows the officials involved in the planning process to procure long lead-time materials early enough so that material delays do not impact schedule—a challenge they cited under the 60-day schedule. Figure 17 shows how the change awarding delivery order 120 days before work is scheduled to begin will affect availability milestones. As reflected in figure 17, another change is that long lead time materials are now ordered 365 days ahead of the start of work, as opposed to the prior schedule of 170 days ahead. Navy supply officials said that some materials require lead times from 1 year to 18 months. Consequently, ordering these materials 170 days before an availability begins increased the likelihood that they would arrive too late to fulfill the Navy’s stated goal of procuring all materials 30 days prior to the start of repair work. Unless repair work requiring these materials is nonessential and can be deferred to a future availability, these material delays can delay completion of availabilities by several months. Several ship repair contractor representatives we interviewed with pointed to long lead-time materials as drivers for schedule growth. While noting the potentially positive effects of shifting award date to 120 days before the availability begins, Navy officials also raised some challenges. They said that locking ship repair requirements almost a full year before an availability actually begins means that the Navy could finalize a ship’s upcoming availability work specifications before a ship even begins its next deployment. During this deployment, equipment breakages or other deficiencies not anticipated and subsequently not included in the work package could arise on the ship, all of which would likely become growth work during the availability. This new time frame for delivery orders has only recently been implemented. The first MAC-MO delivery order awarded 120 days prior to the start of work occurred in January 2020, with another awarded since then. The Navy was scheduled to award availabilities 120 days prior to the start of work in November 2019, but, according to Navy officials, lacked necessary funds to award several availabilities due to the continuing resolution in place at the time. The Navy is not yet certain whether awarding delivery orders earlier will improve the Navy’s ability to provide long lead-time materials on time. Both the Navy and the third-party planner recognize the need for the two parties to work closely together to produce the best specifications and work packages possible under MAC-MO. As within the Navy, third party planner staff also seek to identify lessons learned, in order to improve the quality of ship repair specifications they produce. According to third-party planning contractor representatives, they monitor contract changes involving growth work, assess whether that growth is due to planning deficiencies or other causes, and then identify lessons learned, which they use to improve their specification writing process. For example, contractor representatives stated that they used lessons learned during the USS Bainbridge (DDG 96) availability to create a template for a section of the forecastle deck plate. This template could be used on future availabilities for ships of the same destroyer class, providing potential cost savings to future availabilities. However, RMC officials across the three ports implementing the MAC-MO strategy expressed concerns over the quality of third-party planning contractor specifications used in ship repair availability solicitations and contracts. They stated that the specifications developed by the third-party planning contractors have frequently included errors and discrepancies. As a result, the maintenance teams have had to work with the third-party planning contractor to resolve the issues prior to award. According to RMC officials, maintenance teams within a given port have their own preferences with regard to how the third-party planning contractor writes specifications. Consequently, a specification written and approved in one RMC is sometimes deemed inadequate within another RMC. Figure 18 describes how specification deficiencies and other events affected a USS Roosevelt (DDG 80) availability. Even with the issues that Navy maintenance teams have encountered with third-party planner-developed specifications, RMC officials stated that they continue to find ways to enhance their coordination with the third-party planning contractor. For instance, according to MARMC officials, they found that when availability maintenance teams physically worked alongside third-party planning contractor staff, the planning process went much more smoothly. After SWRMC officials learned of this practice, SWRMC’s maintenance teams were co-located with the third- party planning contractor staff in an effort to improve its process as well. According to RMC staff, they found that having all parties coordinating closely in the planning process to be an effective way to mitigate some of the specification writing issues. In contrast to RMC officials, from NAVSEA leadership officials’ perspective, the third-party planning contractor is currently accomplishing the goals the Navy has set forth and has provided accurate enough specifications to earn the incentive fees outlined in its contract. The NAVSEA officials noted that the contractor has also received annual incentive fees for providing recommendations to the Master Specification Catalog utilized by the Navy to incorporate lessons learned and improve specifications written at all RMCs. Historically the Navy has used its operation and maintenance account to pay for ship repair. By law, those funds have generally only been available for new obligations for one fiscal year—which corresponds with the fiscal year in which the availability contract is awarded– after which the funds expire. In order for the Navy to use any remaining expired funds in the subsequent fiscal year for an in-scope contract change, the executing RMC must request what is called an upward obligation. The Navy can request an upward obligation at the fleet level as long as the request for a specific availability is less than $4 million. RMC officials stated this type of request involves a short process. However, if the upward obligations request exceeds $4 million for an availability, the executing RMC must receive approval from the Office of the Under Secretary of Defense (OUSD) Comptroller. According to RMC leadership officials, this process can take several months. We found that the Navy has requested upward obligations from the OUSD Comptroller 25 times across 14 ship repair availabilities since implementing the MAC-MO strategy in April 2015. In November 2016, we reported that the Navy identified the need for training for staff on how to obtain upward obligations funding. In our interviews with RMC leadership officials and Navy financial officials, some said they now had experience with upward obligations because of their regular need to obtain funding for ship availabilities that crossed fiscal years. Nonetheless, in our discussions with the RMC commanding officers, they described the upward obligations process to obtain OUSD Comptroller approval for upward obligations as cumbersome and unnecessarily complicated. Other Navy officials and contractors echoed these views and highlighted the upward obligations request process as a significant impediment to schedule performance. According to the RMC commanders, it requires several months to successfully execute and complete the upward obligations process for many availabilities because of reviews required within the Navy and the Office of the Secretary of Defense before approval is granted. Navy officials said the process also results in significant delays to the availabilities, as work cannot proceed without funding. For example, of six availabilities for which the Navy provided data, the shortest upward obligations request took 26 days, with the longest request spanning 189 days. Figure 19 describes how for one of our case studies, the USS Chosin (CG 65), the Navy experienced several months of schedule delay due in part to the upward obligations process. Navy officials stated they have attempted to identify legislative solutions to reduce the frequency under which they must obtain upward obligations, given the negative schedule effects this process precipitates. In 2018, the Office of the Assistant Secretary of the Navy (Financial Management and Comptroller), in conjunction with the Office of the Under Secretary of Defense (Comptroller), proposed two legislative initiatives to Congress intended to accomplish this goal. The first of these proposals seeks to raise the legal threshold for ship repair upward obligations requiring Navy and Defense Comptroller approval from $4 million to $10 million. The proposal also provides for a pilot ship availability with these new thresholds, which would allow the Navy to determine the proposal’s effectiveness before fully implementing the new threshold. According to Navy and DOD comptroller officials, this proposal holds merit on several levels. First, the upward obligations threshold has not changed since 1990, when the law implementing the process first passed. The proposed increase to the threshold would account for inflation and subsequent increases in the cost of ship repair over the last 30 years. For example, the average maintenance availability for a DDG 51 Arleigh Burke class destroyer cost $6 million in 1991, but costs $36 million when the Navy proposed the legislative change. Additionally, the scope of the Operations and Maintenance, Navy (O&M) budget has increased by a factor of 2.5 since the law’s 1990 passage. Navy officials believe that increasing the threshold to $10 million would potentially raise this amount to a level corresponding to increases in Navy ship repair budgets since that time. The second proposal would permit Navy O&M funds—which the Navy uses to fund ship repair, among other sustainment-related activities—to be available for the Navy to obligate for up to 2 fiscal years following their appropriation by Congress. Currently, these funds are available to be obligated by the Navy for only 1 year. According to Navy financial officials, since most ship repairs extend into a second year, this proposal would allow ship availabilities to avoid using upward obligations. A senior official with the Office of the Under Secretary of Defense (Comptroller) said that the threshold change was more logical, as the thresholds are no longer practical, and that the logistics of implementing 2-year funding were likely to be more complicated because of the various DOD software systems that would be affected. In December 2019, Congress and the President enacted legislation that— although differing from the Navy’s legislative proposals—is responsive to the Navy’s concerns relating to the process of approving upward obligations more than $4 million in its MAC-MO availabilities. In the Fiscal Year 2020 Consolidated Appropriations Act, Congress established a pilot program that allows the Navy to use the Other Procurement, Navy (OPN) account to fund Pacific fleet surface ship repair availabilities for 2020. Our review of Navy budget documentation shows that the Navy plans to execute 16 pilot availabilities using fiscal year 2020 OPN funds, and it has requested funding for another 26 pilot availabilities in fiscal year 2021. Unlike the Operations and Maintenance, Navy account, which the Navy typically uses to fund ship repair availabilities in 1-year increments, the OPN account provides the Navy with funding that will not expire for 3 years. Consequently, for availabilities the Navy funds through the pilot program, any growth work that necessitates an availability stretching into a second or even third year will avoid upward obligations and the related approval processes, provided sufficient funding remains in the OPN appropriation to cover the work. The joint explanatory statement accompanying the enacted legislation further stated that the Secretary of the Navy is to provide quarterly reports to Congress on the execution of ship availabilities funded through the pilot program in the OPN account. In these quarterly reports, the Navy is to report on the estimated or actual start or end dates of pilot availabilities, as well as the actual funded amount and estimate to complete. The Navy already completes systematic, biennial assessments of MAC- MO implementation, in response to our November 2016 report. While the Navy recognized upward obligations as an issue in its 2018 biennial assessment, the Navy did not examine potential solutions to the schedule delays that these obligations cause. Further, according to NAVSEA officials, the Navy has yet to determine whether it will address schedule outcomes and lessons learned from its pilot program availabilities within future biennial assessments. Our prior work identified leading practices for designing a well-developed and documented pilot program. These leading practices include the following: Establish well-defined, appropriate, clear, and measurable objectives Clearly articulate assessment methodology and data gathering strategy that addresses all components of the pilot program and includes key features of a sound plan Identify criteria or standards for identifying lessons about the pilot to inform decisions about scalability and whether, how, and when to integrate pilot activities into overall efforts Develop a detailed data-analysis plan to track the pilot program’s implementation and performance and evaluate the final results of the project and draw conclusions on whether, how, and when to integrate pilot activities into overall efforts Ensure appropriate two-way stakeholder communication and input at all stages of the pilot project, including design, implementation, data gathering, and assessment These practices enhance the quality, credibility, and usefulness of evaluations and help ensure that time and resources are used effectively. As the Navy moves into implementation of the OPN-funded pilot program, establishing a plan for analysis of the pilot program would provide a means to identify opportunities to take the data on availability schedules, which Congress directed, and compare it to the schedule performance the Navy has attained in its other non-pilot, MAC-MO availabilities. Such evaluations would provide information to the Navy and Congress to determine if the pilot approach should be expanded to help address persistent schedule challenges. In addition, similar to the lessons the Navy has learned in implementing the MAC-MO strategy, the Navy is likely to learn lessons from its OPN-funded pilot availabilities, including ones that relate to schedule drivers currently overshadowed by delays cast by the upward obligations process. Unless the Navy documents within an analysis plan a process for evaluating lessons learned, it runs the risk of missing opportunities to improve its overall performance outcomes across availabilities executed under the MAC-MO strategy. Representatives of private ship repair contractors that the Navy relies on to execute availabilities under the MAC-MO strategy told us that their workforce and facilities investment decisions are driven by two key considerations. First, the contractors seek visibility on planned workload within a given port, which, under current law, the Navy must publicly report on a quarterly basis. Second, the contractors assess that planned workload to determine what share of the work they are most likely to receive. This assessment affects whether a contractor hires more or fewer people, recapitalizes or expands facilities, and, ultimately, elects to remain part of the Navy’s industrial base for ship repair. In recognition of these considerations, the Navy has taken recent steps to increase predictability of workloads at each port, for example by bundling contracts for both sequential and concurrent availabilities. The Navy anticipates that these steps will help further increase contractors’ confidence in their ability to forecast their share of future workloads. As we found in our November 2016 report, various factors regarding the Navy’s level of demand for maintenance and repair work at each of the three home ports implementing MAC-MO, including the deployment of ships, can affect the demand for work in each of the home ports. Based on our analysis of Navy data, this workload remains cyclical in nature, and at times fluctuates above and below what port capacities ordinarily support, as it was under the prior contracting strategy. In May 2016, we found that wide swings in port workload can have a negative effect on the private-sector industrial base, and various factors can affect those workloads. Subsequent to that report, Congress required the Navy to publicly release on a quarterly basis workload projections covering the three ports implementing MAC-MO. Navy’s forecasts indicate that ports implementing MAC-MO will, at times during the next 3 years, be assigned workloads beyond their current capacity, particularly for the Southeast Regional Maintenance Center in Mayport, Florida. Figures 20, 21, and 22 identify the Navy’s port workload projections for each of the three ports as of December 2019. Although the Navy projects that overall workload at the ports implementing MAC-MO will fluctuate with periodic increases, lack of certainty about company-specific workload is driving mixed views among contractors on their willingness to make facility and workforce investments. Multiple contractor representatives we interviewed stated they have always worked within an environment of peaks and valleys of workload regardless of the Navy’s contracting strategy. Representatives of large ship repair contractors we interviewed commented on challenges and changes they have made to remain competitive in the MAC-MO strategy’s competitive, firm-fixed-price contracting environment. Under MAC-MO, which requires competition for every availability within a home port, large contractor representatives stated that they do not have a high level of confidence or visibility into future work that the Navy will award to their companies. They have noted that this uncertainty has affected their planning for hiring and facilities investments. Specifically, contractor representatives cited the following: Of the eight large MAC-MO contractors in our review, four reported that they have increased their full-time workforce and the other four have reported a decreased workforce since 2015. Representatives of three contractors selected in our review noted that they have had to rely more heavily on temporary labor to conduct work on Navy availabilities because of inability to predict workloads. For example, a representative of one large contractor noted that their company retains a permanent core workforce, which it then supplements with temporary labor, as needed, depending on the number of contracts it is awarded by the Navy. Representatives of another large contractor noted that the company recently reinstated a training program for new ship repair workers. A representative from the third contractor stated that the company is considering reinstating its equivalent training program based on workload forecasts and confidence in their amount of workload, which underpins investments in workforce training. Representatives of multiple large contractors in our review also stated that they increasingly rely on their subcontractors to execute ship repair work. For example, a representative from one noted that although the company reduced its full time workforce, it is still able to execute availabilities through their use of subcontractor labor. A representative of another large contractor noted that their company staffed a recent availability with about 70 percent subcontracted labor, in part to help the contractor work within the contract’s price as agreed to with the Navy and to help the company make a profit. Representatives of another large contractor stated their company’s preference is to use subcontractors rather than to surge its permanent staff, especially given the contractor’s uncertainty about its portion of future Navy ship repair and maintenance workloads. Representatives of three of the large contractors we interviewed also stated that unstable workloads have limited their plans for significant capital investments in new or expanded facilities. However, representatives of two large contractors reported making new investments in facilities due to high volume of work at their ports. For example, representatives of one large contractor noted that their parent company invested $100 million into building a new dry dock as part of the company’s commitment to win new availabilities and complete them on schedule. The company reported that it was willing to make this investment, in part, because Navy forecasts show an increase in ships being homeported at that location. These contractor representatives further stated their company is considering additional facilities investments. Apart from the considerations that affect their hiring and facilities investments, representatives from all of the large companies we interviewed told us that they plan to continue competing for Navy ship repair work under the MAC-MO strategy. For seven out of the eight of these contractors, the Navy is their primary customer. A representative of one large contractor noted their company’s preference for the MAC-MO strategy, as compared to earlier Navy contracting strategies, especially as a means to increase its ability to propose on and compete for availabilities. Representatives of two additional large contractors also echoed the positive effect of increased opportunities to propose on Navy ship repair and maintenance contracts as a means to potentially grow their workloads. Representatives of the three small business contractors we interviewed told us that they have each increased their workforces since 2015, when the Navy began implementing the MAC-MO strategy. Under this strategy, small businesses are able to compete for noncomplex ship repair work as prime contractors. Overall, these small business contractor representatives stated they intend to further grow their workforces and facilities, correspondent with the amounts of ship repair work they receive. Specifically, representatives of these small business contractors told us the following: Representatives of one small business prime contractor reported that their company grew its workforce from 625 to 982 between December 2015 and March 2019 as they stated that MAC-MO provided additional opportunities to propose on ship repair contracts. A representative of another small business prime contractor we interviewed estimated that their company hired an additional 100 personnel at two locations because of new, increased workloads related to MAC-MO’s implementation. One small business prime contractor included in our review completed a major facilities expansion, including the addition of a dry dock intended to serve all lines of business, including commercial business customers. A representative of one small business prime contractor stated that their company is considering significant infrastructure upgrades and plans to aggressively compete for noncomplex Navy ship repair and maintenance work. The Navy has recently begun implementing two new contractual approaches—horizontal and vertical contract bundling—within its MAC- MO strategy, but has not yet had sufficient time to collect or assess results. These approaches are intended to increase contractors’ visibility into and confidence regarding future ship repair workloads. Navy leadership officials stated that by awarding multiple availabilities, industry receives a body of work that creates confidence in hiring and retaining a skilled workforce and investment in infrastructure. These approaches provide for contractors to propose on multiple ship repair availabilities that the Navy has bundled within a single request for proposal. Figure 23 illustrates these new contractual approaches. Horizontal Contract Bundling: Navy leadership officials testified to Congress in October 2019 that horizontal bundling helps them decide where to direct ship repair and maintenance work, especially as a means to not surpass capacity at a given port. A representative of one large contractor told us the company anticipates positive effects from horizontal bundling to include being awarded two availabilities from one proposal process and guarantees of work for a longer period than one availability. Another large business contractor representative noted that horizontal bundling would help in stabilizing workloads over a longer period of time, which would also help with its hiring planning. The Navy awarded its first horizontally bundled availabilities in September 2019, and the contractor is expected to complete work on the two ships at its shipyard in Seattle, Washington in June 2021 and May 2022, respectively. NAVSEA leadership officials noted that Navy intends to implement horizontal contract bundling at all of its ports in the future. Vertical Contract Bundling: This contract bundling approach has the potential to allow contractors to increase their workload through only one proposal process, as they may then have the possibility to work on two availabilities at one time. The Navy awarded its first vertically bundled availabilities in February 2019 to three contractors. The second award, in September 2019, resulted in one contractor receiving two simultaneous availabilities. Additionally, NAVSEA leadership officials state they are undertaking other initiatives intended to avoid (1) large fluctuations in ship repair work at individual ports, and (2) the need for contractor workforce layoffs and surge hiring. These initiatives are outlined in further detail below: Attempting to Level Port Workloads: Through its P2P initiative, the Navy intends to use historical timelines from recent availabilities to more accurately plan and forecast future availability time frames. This effort is using computer modeling to avoid either underutilizing or exceeding the available port loading capacity of the industrial base in any given timeframe. On average, NAVSEA leadership stated that they intend to lengthen planned availability timeframes by 56 days to more accurately reflect completion times. The officials assessed that this strategy will help ship repair contractors better manage their workforce planning. They further stated that if contractors have increased visibility in port loading, they will be more likely to hire an increased number of permanent staff in key ship repair trades. According to NAVSEA leadership officials, this could then allow for increased workload capacity at a given port, as those permanent— rather than temporary—staff would become more skilled over time and therefore would require less on-the-job training. Contractor Workforce Capacity Reporting: NAVSEA leadership officials also noted that the Navy is considering options for including language in future ship repair contracts requiring contractors to identify their workforce capacity, including by trade and skill set. NAVSEA leadership officials noted that the intention of such an initiative would be to obtain better workforce capacity data to better plan future port workloads. Although the MAC-MO strategy appears to have stabilized the cost and quality components, completing maintenance availabilities within allotted schedules continues to elude the Navy. The Navy has taken steps to more readily accommodate growth work needs as they emerge, however these likely cannot completely eliminate the Navy’s need for upward obligations. The Navy has pointed to the low cost threshold and upward obligations approval process, as provided for in statute, as not providing it with the agility it needs to fund growth work on a schedule that minimizes disruption to an availability. Recently, Congress enacted legislation, signed into law by the President, which establishes an OPN-funded pilot program and provides the Navy a platform to potentially demonstrate that it can meet its MAC-MO schedule goals when freed from the time intensive process of upward obligations. Nonetheless, every pilot program should be thought out before it starts, including consideration of what data need to be collected and how the data will be analyzed. Otherwise, the pilot could be poorly run or could miss opportunities to gain information and lessons learned. Such planning for the OPN-funded pilot could enhance the quality, credibility, and usefulness of the pilot program. The Secretary of the Navy should establish an analysis plan for the evaluation of OPN-funded pilot program availabilities, based on the leading practices for pilot programs. This analysis plan should identify opportunities to evaluate schedule outcomes of pilot program availabilities as compared to non-pilot program availabilities and document a process for evaluating lessons learned from the pilot program (Recommendation 1). We provided a draft of this report to the Navy for review and comment. In written comments provided by the Navy (reproduced in appendix II), the Navy concurred with our recommendation. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; and the Secretary of the Navy. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or at oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In 2015, the Navy transitioned to the Multiple Award Contract, Multi Order (MAC-MO) contract strategy for the maintenance and modernization of surface ships. This report (1) examines competition, cost, schedule, and quality outcomes under the strategy; (2) evaluates actions the Navy has taken related to recent lessons learned; and (3) describes considerations informing contractors’ plans for future hiring and facilities investments. To examine the competition outcomes of the MAC-MO strategy, as well as number of offers received, we analyzed delivery orders for all of the MAC-MO availabilities in Norfolk, Mayport, and San Diego from the start of the strategy in April 2015 through March 2019. Navy provided a list of MAC-MO Indefinite Delivery, Indefinite Quantity contracts and identified which were complex and noncomplex. We used the Federal Procurement Data System – Next Generation (FPDS-NG) to identify the delivery orders associated with these contracts, and the number of offers received for each order. To assess the reliability of the FPDS-NG data, we reviewed documentation, interviewed Navy officials, performed logic checks, and compared the FPDS-NG data to contract documents. To confirm that we had correctly identified orders related to MAC-MO availabilities, we reviewed the order description in FPDS-NG to confirm that it was a valid ship repair availability and the type of availability. For cases in which the FPDS-NG description did not contain the availability type, we obtained the contract to confirm that it was a valid ship repair availability. To assess the reliability of the number of offers, we performed a logic check to confirm the number of offers received for the delivery order was generally different from the number of offers received for the base contract. Documents reviewed included the FPDS-NG data dictionary, FPDS-NG data validation rules, and Fiscal Year 2013-2018 Federal Procurement Data Quality Summary, which contains results of agency testing of selected fields in FPDS-NG. We determined the FPDS-NG data were reliable for the purpose of assessing the competition outcomes of the MAC-MO strategy. To assess the quality outcomes of MAC-MO availabilities, we reviewed Federal Acquisition Regulations to identify differences between fixed- price and cost reimbursement contract types, and interviewed Navy officials regarding the steps the Navy takes to manage quality in a fixed- price environment. To examine the cost and schedule outcomes of the MAC-MO strategy, we collected ship maintenance availability data from NAVSEA and the Commander, Navy Regional Maintenance Center (CNRMC). This data contained the planned cost and schedule of Chief of Naval Operations (CNO) availabilities, as well as the actual cost and schedule for the availabilities that the Navy closed out between February 2, 2011 and January 15, 2019. While we were directed to assess the MAC-MO outcomes against the Multi-Ship, Multi-Option outcomes, differences in how the availability cost and schedule are estimated between the two strategies prevented us from comparing their cost and schedule outcomes. To assess the reliability of the data, we (1) gathered information from the Navy’s users of the data related to its reliability, (2) compared different snapshots of the data over time to check the consistency of completed entries, including the version that the Navy used to publish its first assessment of the MAC-MO strategy, and (3) compared availability documentation from our completed case study CNO availabilities. We determined the data were reliable for the purpose of assessing cost and schedule outcomes. To narrow our sample, we filtered the data to the ship classes and locations covered under the MAC-MO strategy and eliminated availabilities that had yet to report final cost and schedule entries. This yielded 41 closed out CNO availabilities since the start of the MAC-MO strategy in April 2015. We then adjusted all dollar values for inflation to fiscal year 2020 dollars by using the deflators for Operations and Maintenance funding found in table 5-9 of the Department of Defense budget estimates for fiscal year 2020. To calculate cost and schedule change, we determined the difference between the final cost and completion date, and the planned cost and completion date. The planned cost and schedule represents the Navy’s estimate at the time the Navy awarded the contract. We then calculated the average cost and schedule change for all 41 availabilities, as well as the availabilities at each of the three maintenance centers and classes of ships. To help examine the cost, schedule, and quality outcomes of the MAC- MO strategy, as well as to identify lessons learned, we selected six availabilities as non-generalizable case studies, four of which were completed at the time of our review. To select the availabilities, we used a list of MAC-MO Indefinite Delivery, Indefinite Quantity (IDIQ) contract numbers provided by the Naval Sea Systems Command. We used the Federal Procurement Data System, Next Generation (FPDS-NG) to collect the descriptions of contract actions to determine the ship and availability type, estimated cost, estimated completion dates, contractor, and place of performance. We selected a combination of six availabilities that provided a variety of the following characteristics: We selected two availabilities of each class of ship under the MAC- MO strategy, including destroyers, cruisers, and amphibious ships. We selected two availabilities from each maintenance center executing the strategy: Mid-Atlantic Regional Maintenance Center, Southeast Regional Maintenance Center, and Southwest Regional Maintenance Center. We selected availabilities awarded to a variety of ship repair contractors, including two from BAE Systems, two from General Dynamics NASSCO, one from Marine Hydraulics International, and one from Huntington Ingalls Industries. We selected a variety of availability types to describe different types of ship repair work, including two Selected Restricted Availabilities, a Special Selected Restricted Availability, Depot Modernization Period, Phased Maintenance Availability, and a Continuous Maintenance availability. For each of the case study availabilities, we collected and reviewed Navy availability documentation including the delivery order, correspondence between the maintenance teams and contractors, availability completion reports, weighted progress reports at the time of completion, and briefings containing lessons learned following completion of the availability. We reviewed the documents to: 1) confirm our selection criteria, 2) identify any deficiencies in quality of work and contract changes as a result, 3) identify the presence of growth work items, new work items, or deferred work items, 4) corroborate interview statements, and 5) identify any other issues during the availability and solutions that could be lessons learned for future availabilities. To evaluate the actions the Navy has taken related to recent MAC-MO strategy lessons learned, we analyzed Navy documentation containing lessons learned that aim to improve the Navy’s implementation of MAC- MO. We identified a total of three lessons learned as key based on our assessment of the Navy’s documentation of the MAC-MO contracting strategy. These three lessons learned were also identified as such in one or more interviews with NAVSEA officials knowledgeable about the challenges associated with MAC-MO implementation and the steps the Navy has taken to fix those issues. To evaluate the Navy’s progress in taking actions to address potential challenges posed by the key lessons learned, we reviewed Navy documents, including Navy assessments of the contracting strategy’s effectiveness, documents implementing revised planning milestones and contracting processes, strategy and planning documents, documents from availability completion meetings, case study contract file documents and other documentation related to lessons learned. To assess the extent to which the Navy has taken actions, we developed the following three-point scale: Not Complete—The Navy has not taken any action to respond to identified lessons learned. Partially Complete—The Navy has taken some action to respond to the identified lessons learned, but has not completed the action needed to address the identified risk. Complete—The Navy has completed the action needed to address the identified lesson learned. To describe considerations informing ship repair contractors’ plans for future hiring and facilities investments under the MAC-MO strategy, we conducted semi-structured interviews with and reviewed questionnaire responses from 11 non-nuclear surface ship repair contractors. This included all eight contractors responsible for executing major ship repair work under this strategy at the three home ports implementing it, including Mayport, Florida, Norfolk, Virginia and San Diego, California. We randomly selected a non-generalizable sample of three small business contractors performing noncomplex ship repair work at the three home ports implementing MAC-MO, to obtain the views of small businesses executing MAC-MO contracts. We used FPDS -NG data to identify those small businesses that have been awarded MAC-MO delivery orders. Further, we used a data collection instrument to gather information from each of the selected 11 contractors on their facilities, workforce, and sources of revenue. For example, we collected contractor-reported information on what types of facilities the contractor owned, such as a dry dock or a pier, the number of the contractor’s full-time staff, and the percentage of revenue from entities other than from the Navy. To identify the Navy’s projected workload for non-nuclear surface ships where the MAC-MO strategy is implemented, we obtained data from the Navy from fiscal years 2019 through the end of 2023. Since the purpose of our analysis was to show the Navy’s projections in anticipated port workload, we did not conduct our own assessment of the accuracy of this data. We also interviewed key Navy officials and reviewed statements from testimonies to Senate subcommittees, including of the NAVSEA Commander and of the Assistant Secretary of the Navy for Research, Development, and Acquisition, on their approaches to provide increased visibility and avoid large fluctuations of workloads at Navy ports, including the three home ports implementing the MAC-MO strategy. We collected documentation on these approaches, such as for the Performance to Plan initiative on how the Navy intends to use computer modeling to more accurately plan and forecast future availability timeframes, leveraging Navy historical datasets to provide more accurate and realistic planning forecasts. In addition, for all three objectives, we interviewed officials responsible for overseeing, planning, administering, and funding the Navy’s ship repair contracts, including representatives of the Office of the Under Secretary of Defense (Comptroller); Office of the Assistant Secretary of the Navy (Financial Management and Comptroller); the Office of the Chief of Naval Operations; Commander, Navy Regional Maintenance Center (CNRMC) and Deputy Commander, Surface Ship Maintenance and Modernization (SEA 21); Surface Maintenance Engineering Planning Program (SURFMEPP); Commander, Naval Surface Force, Atlantic; Commander, Naval Surface Force, Pacific; Mid-Atlantic Regional Maintenance Center (MARMC) in Norfolk, Virginia; the Southwest Regional Maintenance Center (SWRMC) in San Diego, California; and the Southeast Regional Maintenance Center (SERMC) in Mayport, Florida. We additionally interviewed management representatives of 11 ship repair contractors included in our review and the third party planning contractor. We conducted this performance audit from November 2018 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, GAO staff who made key contributions to this report include Christopher R. Durbin (Assistant Director); Sean Seales (Analyst-in-Charge); Pete Anderson; Sonja Bensen, Lorraine Ettaro; Lori Fields; Suellen Foth, Kurt Gurka; Cale Jones; Ethan Kennedy; Sophia Payind; and Carol Petersen.", "summary": "The Navy relies on its fleet of over 150 surface ships to be ready to operate when needed for the defense of the United States. The Navy spends billions annually in maintaining this fleet. In 2015, the Navy changed how it contracts for such maintenance work, aiming to better control costs and improve quality. The new approach, called MAC-MO, generally uses firm-fixed-price contract delivery orders for individual ship availabilities competed among pre-qualified contractors at Navy regional maintenance centers. House Report 115-676 included a provision for GAO to review the Navy's implementation of the MAC-MO strategy. This report (1) examines outcomes under the strategy; (2) evaluates actions the Navy has taken related to recent lessons learned; and (3) describes contractors' considerations when planning for hiring and facilities. GAO analyzed data on ship repair under MAC-MO; reviewed six case studies involving different availability types, classes of ships, maintenance centers, and contractors; and interviewed Navy officials and contractors. Since shifting to the Multiple Award Contract-Multi Order (MAC-MO) contracting approach for ship maintenance work in 2015, the Navy has increased competition opportunities, gained flexibility to ensure quality of work, and limited cost growth, but schedule delays persist. During this period, 21 of 41 ship maintenance periods, called availabilities, for major repair work cost less than initially estimated, and average cost growth across the 41 availabilities was 5 percent. Schedule outcomes were less positive and Navy regional maintenance centers varied in their performance (see figure). To mitigate these delays, the Navy has identified and taken actions to implement lessons learned, including negotiating and funding undefined but expected increases in work at the time of contract award. However, these actions have not resolved the delays that result from the approval process the Navy often must use to obtain funds to complete this maintenance work. Namely, if an availability extends into a new fiscal year and needs more than $4 million in additional prior-year funding, both Navy and Defense Department approvals are required. GAO found this approval process took between 26 and 189 days based on Defense Department data. In December 2019, Congress established a pilot program that would potentially allow the Navy to avoid this process. Leading practices GAO identified for pilot programs call for development of an analysis plan to track implementation and performance and for evaluating final results. As the Navy moves into implementation of its pilot program, developing an analysis plan would provide it with a means to identify opportunities to evaluate schedule outcomes of pilot program availabilities, as compared to non-pilot program availabilities, and document a process for evaluating lessons learned from the pilot program. Such evaluations would provide information to determine if the pilot approach should expand to help address persistent schedule challenges. Ship repair contractors now operating in the MAC-MO environment told GAO that two key considerations drive their decisions on workforce and facilities investments: visibility regarding planned workloads within a given port and their assessment of the share of that work they are most likely to win. In recognition of these considerations, Navy officials have begun taking steps to increase predictability of workloads at each port. These officials anticipate that these steps, coupled with increasing workloads at the ports, will help increase contractors' confidence in their ability to forecast their share of future work. GAO recommends that the Navy establish an analysis plan for the evaluation of the pilot program. The Navy concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "We and others have identified challenges specific to VA’s management and oversight. These challenges have affected VA’s ability to accomplish its mission economically, efficiently, and effectively. For example, in April 2019, we summarized priority open recommendations from our previous reports to address these VA challenges. These recommendations cover areas affected by shortcomings in human capital management, such as veterans’ access to timely health care and reform of the appeals process for disability benefits. VA agreed or partially agreed with 28 of our 30 priority recommendations and is taking steps to implement them. We have also previously reported on human capital challenges across VA. For example, we reported in March 2017 that VA determined VBA staff resources have not sufficiently kept pace with increased pending appeals, and additional staff were needed to improve timeliness and reduce its appeals inventory. We found that VA’s written workforce plans—which cover recruiting, hiring, and training—were not consistent with sound workforce planning practices. We recommended that VBA ensure the development of a timely, detailed workforce plan for recruiting, hiring, and training new hires. As of October 2018, VA had taken steps to address this recommendation, but still needed to address risk mitigation strategies for ensuring it has appropriate capacity to manage appeals workloads and improve timeliness of appeals decisions. In addition, VA officials told us in August 2017 that VA had taken actions to hire more staff to update regulations on disability eligibility criteria. However, as of September 2018, the agency was still working to hire these staff. Furthermore, we reported in August 2018 that VHA’s Sterile Processing Services experienced workforce challenges such as lengthy hiring time frames and limited pay and professional growth potential. Officials told us that these challenges resulted in difficulty maintaining sufficient staffing. These challenges pose a potential risk to VA medical centers’ ability to ensure access to sterilized medical equipment. We recommended that VHA examine the services’ workforce needs and take actions based on the assessment. As of July 2019, this recommendation remained open. In 2018, VA’s Office of Inspector General identified leadership and workforce investment as a major management challenge. The Inspector General noted that the root cause for many of the issues it identified at VA was poor and unstable leadership and staffing shortages. Also, a 2015 Independent Assessment found that VHA’s leadership pipeline was not robust enough to meet its current and future needs. The report also concluded that VHA could not identify potential leaders and prepare them to assume their future roles. It stated that inadequate succession planning and unfocused leadership development efforts contributed to these problems. Finally, the report found that VHA may have difficulties meeting projected demand for services if it does not increase its total number of clinical employees, such as physicians, and their productivity. Effective succession planning can help agencies ensure they have a pipeline of talent to meet current and future mission requirements, according to OPM and our past work. Succession planning is a proactive and systematic process where organizations identify the positions they consider to be too critical to be left vacant or filled by any but the best qualified persons, according to OPM guidance. Organizations then develop a plan to fill those positions with qualified and capable employees. The guidance also states that organizations should take a planned, deliberate, and holistic approach to selecting, developing, and engaging their workforce. In our prior work, we noted that effective succession planning is more than filling existing vacancies with people with the same occupational skills and competencies. Rather, succession planning focuses on current and future needs, and develops pools of high-potential staff to meet the organization’s mission over the long term. Our assessment found that VA’s succession planning efforts partially met one leading practice and did not meet four. VHA met two and partially met three leading practices. VBA partially met three and did not meet two leading practices (see table 1). VA lacks a current, department-wide succession plan. It also has not met four of the key leading practices for succession planning, but has partially met one practice. VA Directive 5002 requires that VA use the administrations’ plans to develop a workforce and succession plan annually. However, VA has not produced a leadership-approved, department-wide succession plan since 2009. VA officials said the 2009 plan does not reflect their current succession planning efforts. Obtain active support and participation from leadership: Not met. According to VA officials, VA has tried to update its 2009 succession plan; however, leadership has not approved a revised plan because of leadership turnover. VA has developed a draft workforce plan, but as of July 2019, VA leadership had not approved the draft plan. Active leadership support for succession planning could help VA strengthen its current and future capacity to serve veterans. Develop succession plans aligned with strategic goals: Not met. VA officials did not provide evidence that VA’s succession planning process was aligned with strategic goals. OPM strategic human capital management regulations require an agency’s human capital policies and programs to align with its missions, goals, and strategic objectives. Developing an up-to-date succession plan aligned with the department’s strategic goals would help VA to establish a strategic process for meeting its current and future workforce needs. Analyze current and future workforce gaps: Not met. VA officials told us that they have conducted some analyses of workforce data for mission-critical occupations, but they did not provide evidence that VA analyzes or projects workforce gaps for leadership positions or for each mission-critical occupation. For example, in accordance with the VA MISSION Act of 2018 (MISSION Act), VA reported on the steps it is taking to achieve full staffing capacity. The report included data on VA’s onboard employees, turnover rates, and growth rates for the department’s total workforce, and growth and turnover rates for clinical positions and a limited number of other positions. VA also forecasted its overall hiring requirements for the current and upcoming fiscal year based on the budget and average turnover. However, the report, which VA produces to meet the specific requirements of the MISSION Act, does not include an analysis of workforce gaps for leadership positions or for specific mission-critical occupations. OPM’s strategic human capital management regulations require agency human capital policies and programs be based on comprehensive workforce planning and analysis, and use comprehensive data analytic methods and gap closure strategies to monitor and address skill gaps within mission-critical occupations. Further analyzing workforce gaps could help VA identify current and emerging workforce challenges and inform succession planning strategies. Identify strategies for closing workforce gaps: Partially met. VA has identified some strategies for addressing workforce gaps, though not within a succession planning process. VA’s Corporate Senior Executive Management Office (CSEMO) is responsible for managing the Senior Executive Service (SES) across the department and its administrations. CSEMO coordinates the hiring, placement, training, and development of VA’s SES employees. VA also has an SES Candidate Development Program, which identifies and develops talent to fill key SES positions. Further, in its MISSION Act report, VA identifies several strategies to achieve full staffing capacity. For example, the report discusses efforts to recruit and retain clinical staff through the VHA Education Debt Reduction Program and the VA Health Professional Scholarship Program. The report also discusses the Hire Right Hire Fast model initiated in 2017 that aims to fill open positions and reduce the time to hire for the medical support assistance occupation. VA officials have not provided evidence that they developed strategies for addressing future workforce gaps as part of the agency’s succession planning process. VA’s strategies are focused on closing current vacancies and achieving full staffing capacity. However, our prior work has found that leading organizations do more than just focus on replacing individuals; rather, they engage in broad, integrated succession planning and management efforts to strengthen both current and future organizational capacity. Additionally, OPM’s strategic human capital management regulations require agencies to plan for and manage current and future workforce needs, and to make progress towards closing any knowledge, skill, and competency gaps. Furthermore, because VA has not conducted a full analysis of its future workforce gaps, VA cannot identify strategies for closing those gaps. Monitor, evaluate, and update succession plans and strategies: Not met. VA provided limited evidence that it monitors and evaluates workforce planning strategies. For example, VHA and VBA produce action trackers to monitor the progress of some human capital initiatives at the administration level. However, because VA officials did not provide a current succession plan with strategies for closing workforce gaps, VA’s limited monitoring and evaluation efforts are not clearly linked to a succession planning process. Monitoring and evaluating the outcomes of strategies, policies, programs, and activities is one of the key systems established in OPM’s strategic human capital management regulations, and requires agencies to identify, implement, and monitor process improvements. Monitoring and evaluating activities as part of its succession planning process could help VA ensure that it is implementing effective strategies. Furthermore, regularly updating its succession plans would help VA identify and address current and emerging workforce gaps. VHA developed a succession plan in 2016, and its efforts have met two of the five succession planning leading practices for both leadership and mission-critical occupations. However, leadership has not ensured that VHA has complete workforce data. In addition, VHA’s monitoring and evaluation of its succession plans is limited. Obtain active support and participation from leadership: Partially met. VHA leadership has dedicated resources to succession planning. For example, VHA leadership dedicated staff and financial resources to develop a succession plan for VHA in 2016. VHA leadership also established the Healthcare Leadership Talent Institute (HLTI) in 2015 to strategically manage and develop VHA’s leadership talent. In addition, the former Undersecretary for Health approved the 2016 plan and encouraged staff to use the plan to develop talented staff, improve workplace culture and employee engagement, and address workforce challenges to improve the veteran experience. However, VHA leadership has not ensured that VHA’s plan incorporates leading practices and departmental requirements for succession planning, primarily related to analyzing workforce gaps and monitoring and evaluating its plan. VHA officials also told us that VHA does not provide leadership succession planning guidance to Veterans Integrated Service Networks (VISN) or medical centers because VHA has started to centralize leadership succession planning at the national level. Additional support and involvement from top leadership, such as providing additional oversight and guidance, could help to ensure VHA is meeting department-level succession planning requirements, and ensure that succession planning efforts achieve workforce goals. Develop succession plans aligned with strategic goals: Met. VHA has developed a succession plan and strategies that align with the administration’s and department’s strategic goals. The 2016 succession plan discusses VHA’s strategic direction—which includes strategic goals, major initiatives, and legislation that affect VHA’s workforce—and succession planning priorities. For example, the plan describes VHA’s strategies to adapt to a changing veteran population and to ensure it can provide sufficient, patient-driven primary and mental health care to meet the needs of veterans. Analyze current and future workforce gaps: Partially met. VHA’s 2016 succession plan analyzes current and projected workforce trends for both leadership and mission-critical occupations. For example, the plan presents the total number of executive leadership positions and the number of vacancies in those positions. In addition, the plan includes analyses of recent historical and projected workforce trends for mission- critical occupations, including prior and anticipated onboard, retirement, quit, and total loss rates. VHA collects workforce data from facilities annually and displays these data on its internal website, which is accessible to VHA staff who make human capital and workforce planning decisions. Although VHA tracks workforce data, our prior work has identified weaknesses with these data. For example, in October 2017, we found that VHA was unable to accurately count the total number of physicians in VA medical centers. Medical centers annually report data through a workforce planning tool; however, this tool does not include information on contract physicians, fee-basis physicians, and physician trainees. All of these arrangements help medical centers meet their demand for physicians, which have regularly been identified as one of VHA’s top shortage occupations. We recommended VHA develop and implement a process to accurately count all physicians providing care at each medical center. VA disagreed with this recommendation and, as of March 2019, had not implemented this recommendation. Improving the completeness and accuracy of its data would help VHA better address workforce gaps. Identify strategies for closing workforce gaps: Met. VHA’s 2016 plan identified strategies for closing workforce gaps. For leadership positions, HLTI offers training programs focused on developing future healthcare leaders. In addition to managing development programs, HLTI has implemented several initiatives to address specific gaps in leadership positions and build a succession pipeline of talent. For example, HLTI facilitates an annual talent review process by identifying and developing clinical and administrative leaders at medical centers who are interested in moving up into medical center director positions, the highest position in a VA medical center. VHA has also identified strategies to close gaps for its mission-critical occupations. For example, VHA requires facilities to develop action plans as part of the annual workforce planning cycle to reduce the risk of having critical staffing shortages. For instance, one VA medical center identified increasing human resources training and awareness of recruitment, retention, and relocation funding as an action to address shortages in dentist positions—the clinical occupation with the most severe shortage of candidates at that medical center. In addition, VHA established an initiative for hiring mental health providers, which involved hosting a virtual career event, partnering with professional organizations, and implementing other marketing and recruitment strategies. Monitor, evaluate, and update succession plans and strategies: Partially met. VHA has taken some steps to monitor, evaluate, and update its succession planning. VHA updates its succession plan approximately every 4 years and issues limited updates to the plan annually. VHA’s Office of Workforce Management and Consulting tracks workforce data nationally and provides data and risk scores by occupation to VISNs and medical centers so they can monitor workforce trends. VHA also uses these data to assess to what extent facilities’ efforts are achieving workforce goals. For leadership positions, VHA officials told us that HLTI evaluates its leadership development programs and that these evaluations are used to modify the programs to better meet VHA’s succession needs. However, VHA’s 2016 plan only included limited evaluations of previously identified strategies because VHA has not established a process for evaluating its succession planning efforts. While VHA tracks facility-level metrics for various occupations, VHA’s plan did not discuss specific methods for monitoring and evaluating its succession planning strategies. For example, VHA tracks the vacancy rates for medical center director positions; however, VHA has not identified a process to monitor and evaluate the effectiveness of the talent review process it has implemented for identifying and developing medical center director candidates. The plan also mentioned that subject matter experts within VHA suggested expanding monitoring efforts of certain recruitment and retention programs. As noted above, agencies are required to identify, implement, and monitor process improvements under the evaluation system established in OPM’s strategic human capital management regulations. Additional monitoring and evaluation of VHA’s succession plans and strategies could help VHA to assess the effectiveness of its strategies, and to identify and address emerging workforce challenges. VBA has partially met three key leading practices for succession planning and has not met two practices. Its strategic workforce plan, which VBA officials said is their primary succession planning document, only incorporates some key leading practices for mission-critical occupations. The plan does not address succession planning for leadership positions. Obtain active support and participation from leadership: Partially met. VBA’s leadership has taken some steps to promote succession planning, but has not fully incorporated departmental requirements or key leading practices. VBA officials told us that VBA’s leadership prioritizes filling vacancies for mission-critical occupations. For example, human capital staff brief VBA leadership monthly on vacancies and hiring initiatives. However, our prior work has found that leading organizations do more than simply backfill specific positions; rather, they engage in broad, integrated succession planning and management efforts to strengthen both current and future organizational capacity. As noted above, OPM’s strategic human capital management regulations require agencies to plan for and manage current and future workforce needs. In addition, VBA officials told us that VBA’s leadership reviewed and approved its strategic workforce plan. However, unlike VHA’s plan, VBA’s plan does not indicate that it was reviewed and approved by leadership. VBA leadership also has not ensured that VBA’s plan incorporates departmental requirements or key leading practices. Some of the missing leading practices discussed below—such as aligning plans with strategic goals, identifying strategies to close workforce gaps, and monitoring and evaluating those strategies—are also required by VA’s succession planning directive. VBA leadership also has not ensured that VBA is performing succession planning for leadership positions, which is required by VA’s succession planning directive and recommended by leading practices. In addition, strategic human capital management regulations require agencies to ensure leadership continuity by, in part, implementing and evaluating succession plans for leadership positions. According to VBA officials, VBA’s strategic workforce plan does not include succession planning for leadership positions because VBA plans Senior Executive Service (SES) development in coordination with the Corporate Senior Executive Management Office (CSEMO). While CSEMO manages SES development, VBA officials told us that VBA provides input to CSEMO on VBA’s SES needs. VBA officials did not provide evidence that they are identifying current and future leadership needs. Furthermore, VBA’s leadership also includes General Schedule (GS)-13 to GS-15 managers, who are below the SES level. Planning for those managers would not involve coordination with CSEMO. Therefore, incorporating leadership succession planning into its existing workforce planning processes could help VBA strategically identify and better meet current and future leadership needs. Develop succession plans aligned with strategic goals: Not met. Officials stated that VBA’s strategic workforce plan is the primary document that would discuss succession planning, but this document does not discuss strategic goals and how VBA’s plans align with those goals. As noted earlier, OPM strategic human capital management regulations require agency policies and programs to align with the agency’s mission, goals, and strategic objectives. Aligning plans with strategic goals could help VBA better achieve current and future mission requirements. For example, VBA does not clearly describe how succession plans and strategies for its veterans claims examining occupations—a mission-critical occupation series—will address VBA’s goal to provide veterans benefits and services in a timely manner. It can also help VBA officials create a clear and convincing case for agency leaders to dedicate resources―both budget and personnel―to succession planning. Analyze current and future workforce gaps: Partially met. VBA’s strategic workforce plan includes some analysis of current and future workforce gaps for mission-critical occupations, but not for leadership positions. For example, according to the plan, VBA has increased the number of employees in its veterans claims examining occupations. VBA also anticipates that it will need additional claims processors to meet future demand. However, the plan does not contain similar information for leadership positions, either at the SES level or at lower levels. As noted above, agency human capital policies and programs are to be based on comprehensive workforce planning and analysis. Analyzing workforce gaps in leadership could help VBA better understand its current and future workforce requirements to meet its evolving mission requirements. Identify strategies for closing workforce gaps: Partially met. VBA’s strategic workforce plan does not identify strategies or actions to close anticipated workforce gaps. VBA’s plan states that a forthcoming action plan will develop specific goals and corresponding targets, but VBA officials told us they are still developing this plan. As noted earlier, agencies are required to plan for and manage current and future workforce needs, and make progress towards closing knowledge, skill, and competency gaps. However, VBA does have training and development programs designed to ensure a pool of capable employees is available to take over leadership positions. For example, the Assistant Director Development Program helps prepare GS-14 and GS-15 employees for leadership positions within VBA. Nonetheless, identifying a coordinated set of strategies in its plan for filling leadership positions and closing mission-critical workforce gaps could help VBA address challenges in these areas. Monitor, evaluate, and update succession plans and strategies: Not met. VBA updates its strategic workforce plan every 4 years and issues limited updates to the plan annually. However, VBA’s plan does not provide any information on monitoring or evaluating strategies to close workforce gaps for mission-critical occupations or leadership positions. It also does not include updates to actions or strategies based on past performance. Monitoring and evaluating the outcomes of strategies, policies, programs, and activities is one of the key systems established in OPM’s strategic human capital management regulations, and requires agencies to identify, implement, and monitor process improvements. VA has not updated its succession planning directive since 2003 and VA officials told us that the directive does not incorporate legal requirements put in place since then. VA’s succession planning directive establishes the requirements and assigns the roles and responsibilities for succession planning across the department. VA’s directive identifies succession planning as an ongoing activity intended to best meet the needs of the department over time. According to VA officials, VA has attempted to update the directive twice since 2003, but has not completed the update due to leadership turnover. In addition, officials stated that they had to delay updating the directive to revise it to incorporate new regulatory and legislative changes that occurred during those past efforts to update the directive. A key update to legal requirements since 2003 is OPM’s strategic human capital regulations. These regulations establish the framework agencies are to use to plan, implement, evaluate, and improve human capital policies and programs. OPM originally issued the regulations in 2008 and then revised them in December 2016. In addition, the VA Choice and Quality Employment Act of 2017 requires VA, among other things, to establish a single database that lists each vacant position in VA that the Secretary determines is critical to the mission of VA, difficult to fill, or both. Updating the directive could help to ensure it reflects legal requirements put in place since 2003, such as OPM’s strategic human capital regulations. Updating the directive is also consistent with GAO’s Standards for Internal Control in the Federal Government, which requires management to identify and respond to significant changes, such as new laws and regulations. In addition, we found that VA, VHA, and VBA do not follow all of the requirements outlined in the directive. For example, the directive assigns responsibility to VA, VHA, and VBA for monitoring and evaluating their succession planning strategies, which is consistent with leading practices. However, as stated above, we found that VA, VHA, and VBA do not conduct sufficient monitoring and evaluation. Updating the directive could help VA clarify and recommunicate succession planning roles and responsibilities across the department and its administrations. We and others have previously identified leadership turnover and mission-critical vacancies that have affected VA’s ability to provide services to veterans. Addressing these challenges will require a planned and holistic approach to succession planning that focuses on current and future mission requirements over the long term rather than on filling existing vacancies with people with the same occupational skills and competencies. VA, VHA, and VBA have taken important steps to develop a pipeline of talent to fill leadership positions and mission-critical occupations. For example, each has developed training and development programs for aspiring leaders. However, VA lacks a current department-wide succession plan for leadership positions and mission-critical occupations, as required by its own directive. Establishing a department-wide succession plan and improving existing workforce plans would help VA identify and develop pools of high-potential staff to meet VA’s mission over the long term. Meanwhile, VHA and VBA could each take additional steps to fully incorporate key leading practices into their succession planning. Addressing VA’s challenges will require active leadership support and clear departmental guidance outlining VA’s and its administrations’ responsibilities for succession planning. However, VA has not updated its succession planning directive since 2003 due to leadership turnover, among other factors. Updating the directive could help ensure VA and its administrations are complying with relevant legal requirements—including OPM’s strategic human capital management regulations—and ensure they understand their roles and responsibilities for succession planning. We are making a total of four recommendations, including two to VA, one to VHA, and one to VBA: The Secretary of Veterans Affairs should develop a department-wide succession plan for leadership and mission-critical occupations that incorporates key leading practices for succession planning. (Recommendation 1) The Under Secretary for Health should incorporate key leading practices into VHA’s succession planning processes, including monitoring and evaluating VHA’s succession planning. (Recommendation 2) The Under Secretary for Benefits should develop a succession planning process for all leadership positions and incorporate key leading practices into VBA’s succession planning for leadership positions and mission- critical occupations. These practices include aligning the plans with strategic goals, identifying strategies to close workforce gaps, and monitoring and evaluating VBA’s succession planning. (Recommendation 3) The Secretary of Veterans Affairs should update VA’s 2003 directive on workforce and succession planning to incorporate relevant legal requirements, including OPM strategic human capital management regulation requirements. (Recommendation 4) We provided a draft of this report to the Secretary of Veterans Affairs for review and comment. VA provided written comments, which are reproduced in appendix III. VA concurred with all four recommendations. VA also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretary of Veterans Affairs. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Table 2 summarizes leadership positions and mission-critical occupations at the Department of Veterans Affairs (VA), Veterans Health Administration (VHA), and Veterans Benefits Administration (VBA) as identified by the department and administrations. We reviewed our past reports and Office of Personnel Management guidance to identify the following key leading practices for succession planning. The list below explains the importance of the practices and provides examples of how agencies can demonstrate them. 1. Obtain active support and participation from leadership. Agencies’ top leadership actively participates in, regularly uses, and ensures the needed financial and staff resources for key succession planning and management initiatives. This leadership is important because it can provide (1) stability during plan development and implementation, (2) champions within the agency, and (3) integration with other key management planning efforts. This practice may be demonstrated by, for example, leadership participating in key succession planning meetings and ensuring succession planning policies are up-to-date. 2. Develop succession plans aligned with strategic goals. Agencies discuss how workforce knowledge, skills, and abilities for leadership and mission-critical occupations will contribute to the achievement of strategic and annual performance goals. This alignment helps ensure agencies’ plans provide the talent needed to meet their current and future mission requirements. This practice may be demonstrated by, for example, integrating succession planning into strategic planning and annual strategic objectives review assessments. 3. Analyze current and future workforce gaps. For leadership and mission-critical occupations, agencies identify the current talent state and critical skills in the workforce, future workforce needs, and current and future workforce gaps. This gap analysis is important for identifying the skills and competencies needed for achieving its missions and goals even as the agency’s operating environment changes. This practice may be demonstrated by, for example, conducting and documenting current and projected workforce analysis, including workforce gaps. 4. Identify strategies for closing workforce gaps. Agencies identify strategies for closing workforce gaps for leadership and mission- critical occupations, such as recruitment strategies, training, and developmental opportunities. This planning is important for aligning strategies to eliminate gaps, and tailoring workforce programs and processes to the agency’s needs. This practice may be demonstrated by, for example, developing and implementing action plans and training and development programs. 5. Monitor, evaluate, and update succession plans and strategies. Agencies identify and track performance measures and progress against goals to measure the effectiveness of succession management programs, and regularly update plans to reflect lessons learned. This performance monitoring is important for measuring both the outcomes of strategies and how the outcomes have helped accomplish the agencies’ missions and goals. This practice may be demonstrated by, for example, conducting progress assessments or revising programs based on past performance. Robert Goldenkoff, (202) 512-2757 or goldenkoffr@gao.gov. In addition to the contact named above, Shannon Finnegan (Assistant Director), Alexander Ray (Analyst-in-Charge), Colleen Corcoran, Karin Fangman, Robert Gebhart, and Sarah Green made key contributions to this report. Steven Flint, Shelby Kain, Christy Ley, Marcia Mann, Rachel Stoiko, and James Whitcomb also made contributions.", "summary": "VA operates one of the largest health care delivery systems in the nation and provides billions of dollars in benefits and services to veterans and their families. However, VA faces serious and long-standing problems with management challenges and veterans' access to care and disability benefits. For example, as of December 2018, VA reported an overall staff vacancy rate of 11 percent at VHA medical facilities, including vacancies of more than 24,000 medical and dental positions, and around 900 human resource positions. Ensuring VA, VHA, and VBA have a pipeline of talent to fill leadership positions and mission-critical occupations is key to addressing these challenges. The VA Choice and Quality Employment Act of 2017 includes a provision for GAO to review succession planning policies and guidance at VA and its administrations. This report addresses the extent to which succession planning policies and procedures at VA, VHA, and VBA are consistent with key leading practices. GAO reviewed agency documents related to succession planning for leadership positions and mission-critical occupations, and interviewed agency officials. To identify key leading practices, GAO reviewed GAO’s past work and Office of Personnel Management guidance. The Department of Veterans Affairs (VA), the Veterans Health Administration (VHA), and the Veterans Benefits Administration (VBA) have not fully incorporated key succession planning leading practices (see table). Legend: ● Met ◒ Partially Met ○ Not Met Source: GAO analysis of VA's, VHA's, and VBA's succession planning efforts. │ GA O-20-15 VA lacks a current, department-wide succession plan. According to VA officials, VA has not produced a department-wide succession plan since 2009 due to leadership turnover. VA officials said the 2009 plan does not reflect their current succession planning efforts. Establishing a succession plan would help VA identify and develop high-potential staff to meet VA's mission over the long term. VHA's succession plan is consistent with some leading practices, but our prior work found that VHA's physician staffing data are incomplete. Also, VHA performs limited monitoring and evaluation of its plans. Additional monitoring and evaluation could help VHA assess the effectiveness of its strategies in achieving its goals. VBA's plan includes some analysis of workforce gaps for mission-critical occupations. However, VBA's plan does not address leadership positions or fully incorporate key leading practices for mission-critical occupations, such as veterans claims examiners. Developing a succession planning process for leadership positions and fully incorporating key leading practices into its existing processes could help VBA better meet its current and future workforce needs. VA has not updated its succession planning directive since 2003 and VA officials told us that the directive does not incorporate legal requirements put in place since then. The directive establishes requirements and responsibilities for succession planning across VA. VA officials stated that they have not updated the directive because of leadership turnover and changes in legal requirements. Updating the directive could help to ensure it reflects relevant legal requirements. In addition, we found that VA, VHA, and VBA do not follow all of the requirements outlined in the directive. Updating the directive could help to clarify and recommunicate succession planning roles and responsibilities across the department. GAO is making four recommendations. VA should develop a department-wide succession plan and update its succession planning directive. VHA and VBA should fully incorporate key leading practices for succession planning. VA agreed with the recommendations.", "document_type": "gao"}
{"report": "The United States currently has two primary missions in Afghanistan: the U.S-led counterterrorism mission and the NATO-led Resolute Support mission to train, advise, and assist the ANDSF. For U.S. purposes, both of these missions are a part of Operation Freedom’s Sentinel, commanded by U.S. Forces-Afghanistan. Combined Security Transition Command-Afghanistan is the command under NATO’s Resolute Support mission that conducts the train, advise, and assist mission in Afghanistan. These efforts are carried out via the regional Train Advise Assist Commands (TAACs) that collectively cover all of Afghanistan. Specifically, Train Advise Assist Command–Air (TAAC-Air) focuses on developing and advising the Afghan Air Force. The ASFF is generally a 2-year appropriation that is used to provide assistance, with the concurrence of the Secretary of State, to the security forces of Afghanistan, including the provision of equipment, supplies, services, training, facility and infrastructure repair, renovation, construction, and funding. The ASFF presently comprises four budget activity groups: Afghan National Army, Afghan National Police, Afghan Air Force, and Afghan Special Security Forces. Each budget activity group includes four sub-activity groups: sustainment, infrastructure, equipment and transportation, and training and operations. According to officials, the training and operations sub-activity group encompasses most of CSTC- A’s efforts to train the ANDSF, including the Afghan National Army. CSTC-A has established processes to identify capability gaps within the ANDSF, develop and select training needed to address those gaps, and identify associated funding requirements. To do so, CSTC-A works with various requiring activities—partner organizations, such as the Train Advise Assist Commands—to identify ANDSF training needs. CSTC-A then incorporates these needs and associated funding requirements into the ASFF budget request, typically a year or more before the training is initiated. CSTC-A has established processes to identify capability gaps within the ANDSF, develop and select training needed to address those gaps, and identify associated funding requirements for inclusion in ASFF budget justification documentation. To help execute these processes, CSTC-A has developed standard operating procedures and other guidance for planning, resourcing, and executing the ASFF. These procedures and other guidance include information on processes to validate training requirements and associated resources. CSTC-A works with various partner organizations—referred to as “requiring activities”—to identify capability gaps and training needs for the ANDSF. Requiring activities are the organizations that request the resourcing of ANDSF capability needs through ASFF. They include CSTC-A, the TAACs, and other U.S. or NATO organizations partnered with the ANDSF. According to DOD officials, a partner organization can identify capability gaps in a number of ways. For example, Train Advise Assist Command– Air, which develops and advises the Afghan Air Force, works with subject matter experts from the relevant U.S. military services and other organizations to identify potential Afghan Air Force capability gaps. Additionally, according to DOD officials, in 2015 DOD tasked the MITRE Corporation to conduct a study of Afghan Air Force capabilities. According to DOD officials, MITRE’s November 2015 study highlighted capability gaps within the cadre of Afghan Air Force fixed- and rotary-wing pilots and maintenance personnel. Further, officials stated that the study concluded that the training of additional pilots constituted a critical need for the Afghan Air Force. Once a capability gap has been identified, the requiring activity develops potential courses of action to address it, such as proposals to train the ANDSF to develop needed capabilities. Through CSTC-A’s procedures these proposals are validated, along with associated resources. The validation process is intended to ensure that a transparent and accountable process is followed when allocating ASFF resources to emerging requirements. For example, as part of the fiscal year 2018 budget process, TAAC-Air identified a capability gap within the Afghan Air Force and then worked with various subject matter experts to develop courses of action to address the gap. Specifically, TAAC-Air worked with personnel from the Program Executive Office for Simulation, Training, and Instrumentation (PEO-STRI), which provides simulation, training, and testing solutions for the Army and joint community. Subject matter experts from PEO-STRI provided details regarding various options for addressing the capability gap. PEO-STRI officials noted that they also provided cost estimates for delivering the solution based on historical data. According to PEO-STRI officials, this was a highly interactive process entailing frequent formal and informal discussions among multiple organizations to develop the most effective solution for pilot training for the Afghan Air Force. Once details and cost estimates were solidified, the requirement owner presented them to a Council of Colonels, an officer group responsible for requirement validation for training needs, among other capability needs. The requirement was then taken to the General Officer Steering Committee, which votes to validate the requirement and approve the proposed solution. CSTC-A’s process incorporates validated training needs and their associated funding requirements as part of DOD’s annual budget process. DOD’s planning, programming, budgeting, and execution (PPBE) process, which is governed in part by DOD Directive 7045.14, along with other DOD guidance, is conducted under four phases (see figure 1). Specifically, DOD uses the PPBE process to determine and prioritize requirements and allocate resources to provide capabilities necessary to accomplish the department’s missions. According to officials, as part of this process, CSTC-A provides inputs, including training requirements and associated funding needs, and later works with various contracting commands to execute appropriated funds. In the case of ASFF, CSTC-A’s guidance indicates that a proposed activity (for example, fixed-wing pilot training classes) should generally be included in the ASFF budget justification book in order to later use ASFF funds for that activity. To do so, CSTC-A’s Program and Analysis Division develops and incorporates the requests from requirement owners for funding for the operations, sustainment, and development of the ANDSF into the ASFF budget request and associated budget justification materials. The Program and Analysis Division works with the requirement owners to write a narrative describing their proposed activity and associated cost estimate for delivering the activity. The division then works with the OUSD-Comptroller to consolidate requirements for all budget activities and sub-activity groups into a single draft budget justification book. One significant aspect of this process is that many of the key decisions, and associated cost assumptions, on how CSTC-A and TAAC-Air (in the case of Afghan pilot training) intend to carry out ASFF training efforts are proposed 18-24 months before the training will occur. For example, as shown in figure 2, preparation of the ASFF budget justification book for fiscal year 2019 began in the summer of 2017. In turn, the budget justification book was subsequently submitted to the OUSD-Comptroller in December 2017, and funds were not available for use until the start of the new fiscal year, in October 2018. These time frames can present a challenge in developing accurate cost estimates for CSTC-A, given that situations in Afghanistan can change significantly in the time between CSTC-A’s developing a proposed capability requirement and associated cost estimate for inclusion in the ASFF budget justification book and the execution of that requirement, according to officials. If conditions change, officials noted, the proposed actions and associated cost estimates for a given requirement may no longer be appropriate or accurate. For example, the Special Inspector General for Afghanistan Reconstruction reported in January 2019 that CTSC-A may have overestimated the cost for UH-60 Blackhawk rotary- wing pilot training by as much as $1 billion over a 7-year period— attributing the overestimation mainly to unrealistic assumptions regarding student or pilot attrition and the English language program. In the case of initial entry fixed-wing pilot training classes, CSTC-A’s original proposal, as reflected in its budget justification book was to have classes of 25 students. However, during the implementation of this training, the class size fell to 12 students because not all 25 students achieved the required English language proficiency, and one student had dropped out of the program. Consequently, the resulting class was half the projected size underlying the estimated funding requirement, which resulted in funds being excess to CSTC-A’s actual need. CSTC-A officials acknowledged the challenges they faced in filling classes with the expected number of students, adding that they had purposely built in significant flexibility in the training approach to be able to adjust to the realities of the ANDSF’s ability to generate qualified students. According to CSTC-A officials, the number of English-proficient Afghan student candidates varies from year to year. For cases like these, where CSTC-A requested more funding than it ultimately obligated, in some circumstances DOD may reprogram the unobligated amounts within the same appropriation account, or may transfer it to other appropriation accounts, if there is authority to do so. Otherwise, time-limited appropriations, such as the ASFF, expire after their period of availability and are unavailable for new obligations. According to CSTC-A officials, in cases where they have unobligated funding due to changing conditions such as smaller-than-expected class sizes, they try to reprogram that money for related needs within the same sub-activity group in the ASFF budget prior to expiration. For example, if certain Afghan Air Force training costs are lower than expected, the money could be reprogrammed for other efforts within the Afghan Air Force training and operations sub-activity group. ASFF-funded training contracts for the ANDSF are developed and executed through a process that is modeled on the U.S. government’s foreign military sales process. Until April 2019, ASFF-funded orders to train the Afghan National Army were generally filled under a contract with a single provider. At that point, DOD began to transition to an approach using several contracts, including one with multiple providers. ASFF-funded training contracts are developed and executed under a process modeled on the U.S. government’s foreign military sales (FMS) program, referred to as “pseudo-FMS.” As indicated by CSTC-A guidance, these pseudo-FMS procurements are FMS-like cases and use U.S. funds to purchase items, services, and training for ANDSF capability requirements. The process is outlined in the Security Assistance Management Manual, which provides DOD-wide guidance to DOD components engaged in the management or implementation of DOD security assistance and security cooperation programs over which the Defense Security Cooperation Agency has responsibility. We have previously reported that while the many steps of the process used for FMS and pseudo-FMS cases can be grouped in different ways, they fall into five general phases: assistance request, agreement development, acquisition, delivery, and case closure. First, CSTC-A works with the resource coordinator, requirement owner, and other elements to develop a Memorandum of Request, and it submits that memorandum to the implementing agency and the Defense Security Cooperation Agency, requesting assistance to contract for ANDSF needs using ASFF funds. For example, when developing the Memorandum of Request for initial entry fixed-wing pilot training, CSTC-A worked with TAAC-Air, the requirement owner, to identify details regarding the agreed- upon training solution. Officials noted that CSTC-A also worked with the subject matter experts from PEO-STRI to develop the independent government cost estimate. Second, as described by officials, the agreement development phase begins with the Defense Security Cooperation Agency’s receiving the Memorandum of Request. The Defense Security Cooperation Agency opens a case and assigns it to an implementing agency—that is, the military department or defense agency responsible for overall management of the actions that will result in the delivery of materials or services. According to contracting officials, the implementing agency for training foreign military ground and air forces outside of the United States—such as the Afghan National Army—is the U.S. Army Security Assistance Command. The implementing agency then works with the appropriate Program Executive Office to develop the Letter of Offer and Acceptance—which serves to document the transfer of articles and services to the U.S. government requesting authority. For example, for the out-of-country fixed-wing pilot training requirement, contractors delivered the training, and the appropriate implementing agency was PEO-STRI, according to officials. Once the Letter of Offer and Acceptance is completed and signed by the implementing and requesting agencies, it is reviewed and approved by the Defense Security Cooperation Agency and Department of State, as appropriate. Third, the Program Executive Office works with the appropriate contracting command to acquire the requested defense goods or services as part of the acquisition phase. According to contracting officials, the contracting command solicits and receives bids from contractors and selects the best value option (including price plus deliverables). Fourth, the contractor delivers the required good or service. According to officials, the relevant Program Executive Office is responsible for monitoring the contractor’s performance by ensuring compliance with applicable contract clauses. Fifth, following contract completion and payment of outstanding obligations, the implementing agency initiates case closure with the Defense Security Cooperation Agency. Prior to April 2019, ASFF-funded training requirements for the Afghan National Army, including out-of-country fixed- wing pilot training, were generally executed under a single award indefinite delivery, indefinite quantity contract known as the Warfighter Field Operations Customer Support (WFF) contract. The WFF contract provided integrated training system sustainment and training services world-wide for the U.S. Army, Marine Corps, Navy, Air Force, and Special Operations Command. According to Army contracting officials, WFF was the most expedient way to contract for various types of training for the Afghan National Army due to the contract’s broad scope and $11.2 billion ceiling. These officials said it provided the capacity and flexibility needed to fulfill the Afghan National Army’s requirements and time frames in a streamlined way because the competition and award process had already occurred, enabling officials to move directly to awarding task orders for support. However, while the single award indefinite delivery, indefinite quantity contract streamlined the process for contracting ANDSF training, it limited DOD’s ability to negotiate some costs. According to contracting officials, only certain types of costs could be negotiated, such as those associated with housing, travel, and the number of advisors supporting the training. The officials stated that other costs were established as a per-unit cost at the time of the contract award. In addition, various administrative fees were established when the WFF contract was awarded in 2007 and could not be renegotiated, according to contracting officials. As a result, any task orders under this contract, including those to train the Afghan National Army, had to include these administrative fees and established labor wages. To illustrate the various costs associated with the Afghan Air Force training program, we reviewed documentation associated with training provided under the WFF contract. One training program cost $12.1 million for the delivery of an 86-week fixed-wing pilot training course (from February 2018 through September 2019) for 13 Afghan Air Force students at the Fujairah Aviation Academy in the United Arab Emirates. The pilot training was conducted by contractors and comprised aviation English language training, theory of flight, basic and advanced instrument ground school, advanced flight instrumentation, and simulation training for the Afghan Air Force Cessna C-208 Caravan aircraft. The $12.1 million total included amounts paid to the contractor and administrative charges to cover the costs of entities within the U.S. government. The costs associated with the training are shown in figure 3 below. The largest cost factor in this task order was the cost of the flight school itself, which accounted for 68.4 percent (or $8.2 million) of the total cost, according to contracting officials. The flight school included ground school, simulation, advanced instruments, and flying hours training, and it represented a cost per each of the 13 students who actually attended the training. The flight school also included the cost of housing, electronic books / manuals, and campus security, some of which costs were negotiable, according to officials. Other costs, such as the Defense Security Cooperation Agency 3.5 percent surcharge and contract administration services 1.2 percent surcharge, were established based upon rates current at the time of the letter of offer and acceptance. According to officials, the contractor’s profit was established at the time of award of the contract in 2007. Officials stated that the costs that could be negotiated were limited and included costs associated with travel, lodging, and adding more advisors to augment the training. According to contracting officials, these limitations were not unique to this ASFF training but applied broadly to all ASFF training task orders they executed under WFF. In 2018 DOD decided to replace WFF, which was nearing expiration, with a series of new contracts. DOD has begun to transition work previously performed under WFF to these new contracts, the first of which was awarded in 2018. According to contracting officials, ASFF-funded training efforts are expected to be executed primarily under two of the new contracts – the Enterprise Training Services Contract and the Training, Instructor Operator Support Services Contract. The Enterprise Training Services Contract is a multiple award indefinite delivery, indefinite quantity contract with a total contract ceiling of $2.4 billion that was awarded to multiple contractors in June 2018. According to officials, the Training, Instructor Operator Support Services Contract is a single award indefinite delivery, indefinite quantity contract with a ceiling of $197.6 million that was awarded in July 2018. According to Army contracting officials, the contracting process for ASFF training services will include competition among multiple contractors for each task order under the Enterprise Training Services Contract. Army contracting officials stated that under a multiple-award contract, each contract holder is to be provided a fair opportunity to compete for each task order, in part to use competition to ensure that the proposed prices are fair and reasonable. According to Army contracting officials, the Enterprise Training Services Contract also affords the opportunity to negotiate more elements than previously under the WFF contract, such as labor rates or travel costs associated with training. The first training task order under the Enterprise Training Services Contract in support of Afghan forces was issued in April 2019. As this task order has only recently been issued, it is too early for us to comment on the efficacy of these contracts. DOD has varying degrees of visibility over ASFF-funded training contracts. At the broadest level, OUSD-Comptroller and contracting officials stated that they have visibility of the overall execution of the ASFF budget, including funding associated with Afghan National Army training. For example, OUSD-Comptroller tracks and reports ASFF obligations and disbursements in monthly status-of-funds reports, known as Defense Financial and Accounting Services 1002 Reports. In addition, the Special Inspector General for Afghanistan Reconstruction tracks and reports ASFF obligations and disbursements via its Overseas Contingency Operations quarterly reports to Congress. At the individual contract level, the military services’ contracting commands, such as PEO-STRI and Army Contracting Command, develop and maintain contract files for individual ASFF-funded contracts and task orders. However, according to officials, DOD does not have a centralized system or reporting mechanism for tracking all ASFF training contracts, because the systems used by the services for managing funding and those used for contract management do not interface with each other. According to OUSD-Comptroller officials, the systems used for financial management were not designed or intended to identify ASFF funds specifically obligated for training contracts because there is no requirement for them to do so. Officials said that consequently, in the single instance in which they have had to develop a comprehensive list of all ASFF-funded training contracts, they had to work with the contracting commands at the respective military services to gather this information. For example, to respond to congressional direction related to contracts funded with ASFF, OUSD-Policy contacted all of the military services to request a list of all training contracts funded through the ASFF under the respective services’ responsibilities, according to OUSD officials. In turn, Army contracting officials stated that they identified the requested information by using the lines of accounting fields in their contract management systems to identify those training contracts funded with ASFF. OUSD-Policy officials provided us with the resulting list of 40 contracts and task orders, totaling over $483 million in estimated contract value, but they acknowledged that the list was likely incomplete. OUSD-Policy officials who compiled the list of training contracts told us that the precision of the list was affected by inconsistent interpretations among the services of what constitutes a training contract. According to these officials, training for the Afghan National Army can also occur under procurement or maintenance contracts that have embedded training components. For example, according to officials, the Army’s National Maintenance Strategy contract provides logistic support to the Afghan National Army and includes a training component. Similarly, the Navy’s ASFF-funded ScanEagle unmanned aerial vehicle reconnaissance procurement contract includes a training component. Because these contracts are not primarily training-oriented, according to contracting officials, they were not identified under the training and operations subactivity group in the ASFF budget, and therefore would not be easily identifiable as ASFF training contracts. Despite these limitations, DOD officials stated that, given their existing systems and processes and their ability to reach out to contracting officials to obtain additional data when needed, they believe they have sufficient tools to identify most ASFF- funded training contracts. Additionally, DOD officials stated that the congressional direction associated with ASFF-funded training was a one- time request, not a recurring task. We provided a draft of this report to DOD, and DOD responded that it would not be providing formal comments. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and to the Secretary of Defense. In addition, the report is available at no charge on our website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Cary Russell at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in the appendix. In addition to the contact named above, James A. Reynolds, Assistant Director, and Jerome Brown, William Chatlos, Alfonso Garcia, Steve Pruitt, Michael Shaughnessy, McKenna Stahl, and Cheryl Weissman made key contributions to this report.", "summary": "The United States has made a commitment to building Afghanistan's security and governance structure in order to counter terrorist threats and create sustainable security and stability in Afghanistan. Since 2005 Congress has appropriated more than $78.8 billion for the ASFF to build, equip, train, and sustain the Afghan National Defense and Security Forces. Over that period, nearly $4.3 billion has been expended to support the training and operations of the Afghan National Army. Training requirements are primarily fulfilled through contracts. In recent years, concerns have been raised in Congress about the high costs of some of these training contracts. The Joint Explanatory Statement accompanying the Consolidated Appropriations Act, 2018, included a provision for GAO to examine the ASFF training contracts. This report describes DOD's processes to (1) identify Afghan National Army training needs and associated funding requirements; (2) develop and execute ASFF training contracts; and (3) provide visibility over ASFF training contracts. GAO reviewed DOD guidance for identifying and executing training needs, and interviewed DOD officials. GAO also reviewed documentation associated with task orders issued against an indefinite delivery, indefinite quantity contract for training completed in fiscal years 2017 through 2019 for the Afghan National Army. Combined Security Transition Command-Afghanistan (CSTC-A) has established processes to identify capability gaps within the Afghan National Defense and Security Forces (ANDSF), develop and select training needed to address those gaps, and identify associated funding requirements. CSTC-A generally includes these requirements in the Afghanistan Security Forces Fund (ASFF) budget justification book. Many of the key decisions and associated cost assumptions on how CSTC-A and Train Advise Assist Command–Air (in the case of Afghan pilot training) intend to carry out ASFF training efforts are proposed 18-24 months before the training will occur (see figure). ASFF-funded training contracts are developed and executed under a process modeled on the U.S. government's foreign military sales program. Prior to April 2019, most ASFF-funded training requirements were filled under a single-award indefinite delivery, indefinite quantity (IDIQ) contract that supported a wide range of DOD training needs. An IDIQ contract provides for an indefinite quantity, within stated limits, of supplies or services during a fixed period. The government places orders for individual requirements. According to an Army official, that contract's broad scope and high contract value ceiling made it a highly expedient way to contract for various types of training for the ANDSF. However, contracting officals stated that using a single-award contract limited DOD's ability to negotiate some costs. At that point, DOD began to transition to an approach using several contracts, including one with multiple providers. Given that DOD executed its first task order under these new contracts in April 2019, it is too early for GAO to comment on the efficacy of this new approach. DOD has varying degrees of visibility over ASFF-funded contracts. DOD officials stated that they have visibiliity at the broadest level of the overall execution of the ASFF budget, including funding associated with Afghan National Army training. At the individual contract level, the military services' contracting commands maintain contract files, but the services' systems do not interface with one another. According to DOD officials, although DOD can obtain visibility over ASFF training contracts in the aggregate, the department must work with the contracting commands at the respective military services to gather information specific to training contracts.", "document_type": "gao"}
{"report": "As SBA’s largest matching grant program, the SBDC Program provides funding to SBDCs to deliver business advising and technical assistance to prospective and existing small businesses. SBDC lead centers manage the program, including submitting annual funding applications. Approximately two-thirds of SBDCs are funded on a calendar-year basis and the rest on a fiscal-year basis. Universities, community colleges, or state governments host SBDC lead centers. SBA provides grants covering 50 percent or less of SBDC program costs. As a condition of the grant, SBDCs are required to provide 100 percent matching funds from nonfederal sources (one nonfederal dollar for each federal dollar provided by SBA), which are used to cover remaining program costs. At least 50 percent of the match must be in cash and the remaining amount can include combinations of additional cash, in-kind contributions, or waived indirect costs. Organizations that provide matching contributions include state legislatures, private-sector foundations, state and local chambers of commerce, economic development entities, and colleges or universities. SBA considers expenditures of nonfederal funds that an SBDC spends on the program in excess of the statutorily required match as “overmatch.” SBA generally reimburses SBDCs for allowable program costs, up to the amount of the federal award and provided such costs adhere to the budget approved by SBA. As with other federal programs, the President submits a budget request to Congress for the SBDC Program in or around February of each year, and Congress thereafter engages in its appropriations process. In or around July of each year, SBA publishes a funding opportunity announcement for the SBDC Program. This announcement includes a funding estimate (because the final appropriation is not known at this time) for awards to be made by SBA during the upcoming fiscal year, which begins on October 1. Each SBDC submits an initial application for funding based on its proportional share of the funding estimate. The application must include detailed budgets setting forth program costs, broken out separately for the SBDC and each of the service centers it oversees. If the appropriation were to match the initial funding estimate, SBDCs would be required to adhere to the budgets they initially submitted or request approval for a modification from SBA. If a continuing resolution is in place, SBA awards partial funding to the SBDCs based on amounts available under the continuing resolution and apportioned by OMB. According to SBA officials, SBDCs are required to submit revised budgets after a continuing resolution. After appropriations are enacted for the full year, the funding amounts for the SBDC Program are revised and SBDCs submit a final, revised budget. SBDCs are informed of funding decisions through a notice of award that includes the budget approved by SBA and the other terms and conditions under which the award is made. We previously reported on the effects of budget uncertainties and disruptions. In a February 2018 testimony, we noted that Congress annually faces difficult decisions on what to fund among competing priorities and interests, and often postpones final funding decisions to allow additional time for deliberations. Under a continuing resolution, agencies can continue to operate, but the funding expires on a certain date and therefore creates uncertainty about both the timing of final appropriations and the level of funding ultimately available. And when a lapse in appropriations—or funding gap—is possible, the affected agencies must prepare for an orderly shutdown of operations, even if a shutdown is ultimately averted. In the same testimony, we reported that continuing resolutions and lapses in appropriations leading to government shutdowns created inefficiencies and other management challenges for agencies, such as delayed hiring and additional work. The following are select statutes, guidance, and regulations that govern agency budgeting and obligation and expenditure of federal funds, including for the SBDC Program: Antideficiency Act. This act prohibits agencies from obligating or expending funds in excess or in advance of an available appropriation unless otherwise authorized by law and, with some exceptions, from accepting voluntary services for the United States. Impoundment Control Act. An impoundment is any action or inaction by an officer or employee of the federal government that precludes obligation or expenditure of budget authority. When Congress appropriates funds to the executive branch, the President, unless otherwise authorized to withhold such amounts, must prudently obligate them. The Impoundment Control Act is rooted in this principle, and grants the President strictly circumscribed authority to temporarily withhold funds from obligation by transmitting a special message pursuant to procedures established by the act. Transmission of a special message is the only mechanism through which an agency may withhold budget authority from obligation. OMB guidance. Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards, 2 CFR Part 200 (Uniform Guidance), is meant to provide a government-wide framework for grants management and to reduce administrative burden for nonfederal entities receiving federal awards while reducing the risk of waste, fraud, and abuse. To comply with the Uniform Guidance, federal agencies are responsible for developing requirements for grant applicants and recipients of their program awards, with consultation with OMB’s Office of Information and Regulatory Affairs. Under the Uniform Guidance, federal agencies must provide the projected total amount of funds available for programs providing federal financial assistance, and this amount is then made publically available. Estimates based on the previous year’s funding are acceptable if current appropriations are not yet available. In addition, federal agencies must publically announce specific funding opportunities. These announcements must include sufficient information to help an applicant make an informed decision about whether to apply, such as the total amount of funding the agency expects to award through the announcement. SBA has adopted the Uniform Guidance, thereby giving regulatory effect to the guidance with respect to SBA awards. Preparation, Submission, and Execution of the Budget, OMB Circular No. A-11, contains instructions and schedules for agency submission of budget requests and justification materials to OMB. It provides agencies with an overview of applicable budgetary laws, policies for the preparation and submission of agency budgets, and information on financial management and budget data systems. Statutory funding formula for SBDCs. The amount of an SBDC formula grant received by a state is determined by a statutory formula. The formula divides the annual amount made available for the entire SBDC Program—for the fiscal year the grant begins—on a pro rata basis, based on the population of each state and subject to minimum funding levels specified in statute. The maximum grant amount for each recipient (SBDC) is the greater of the minimum statutory amount, or their pro rata share of all SBDC grants as determined by the statutory formula. In fiscal year 2016, SBA changed the methodology it used for the estimate in the SBDC funding opportunity announcement. Before fiscal year 2016, SBA officials stated that the agency used the prior year’s appropriated amount for the program as the funding estimate in the funding opportunity announcement. However, we found that this was not always the case. The funding estimates in fiscal years 2012 through 2014 ranged from $2 million to $10 million lower than the prior year’s appropriation (see fig. 1). The officials were unable to provide information on the justification for the different pre-2016 practices, stating that no officials involved in such determinations during those years remained at the agency. Since fiscal year 2016, SBA has instructed SBDC grantees to submit their funding applications for the upcoming year based on the President’s budget request. In fiscal year 2016, the prior year’s appropriation and the President’s budget request were the same—$115 million—and the final appropriated amount was $117 million. In fiscal year 2017, the President’s budget request of $115 million was slightly lower than the prior year’s appropriation of $117 million, and the final appropriated amount increased to $125 million. Beginning in fiscal year 2018, the funding estimate and the prior year’s appropriation began to diverge significantly (funding estimates decreased, appropriations increased). In that year, SBDCs were required to submit funding applications based on a funding estimate of $110 million, which was 12 percent lower than the prior-year appropriation of $125 million. The final appropriated amount increased to $130 million. In fiscal year 2019, the funding estimate was $110 million, 15 percent lower than the prior-year’s appropriation of $130 million. The final appropriated amount increased to $131 million. By fiscal year 2020, SBDCs were required to submit funding applications based on the President’s budget request of $101 million, which was 23 percent lower than the prior year’s appropriation. The final appropriated amount increased to $135 million. If SBA continues this practice for fiscal year 2021, SBDCs will be required to submit funding applications at a level that is 35 percent lower than the 2020 appropriation, in order to match the President’s budget request of $87.9 million (13 percent lower than the President’s budget request for fiscal year 2020). SBA cited two reasons for changing the way it estimates funding in the annual funding opportunity announcement: to conform to federal financial management practices and to address concerns about violating the Antideficiency Act. SBA officials cited federal financial management practices as one reason for changing the funding estimate methodology. The officials stated that it was a management decision to use the only known amount, the number in the President’s budget, for the funding opportunity announcement. They noted that the difference between the Senate and House markups for fiscal year 2020 exceeded $20 million. SBA officials also told us that they rely on the lowest estimate available to be prudent with taxpayer dollars. According to a December 2019 letter from SBA to the Senate and House Small Business Committees, the change to the funding estimate methodology was adopted when the SBDC Program office identified areas requiring process improvements and stronger internal controls to help the program operate and plan more effectively and efficiently and in consistency with federal financial management standards. SBA’s letter also stated that it is standard federal financial management practice to plan to the lowest budget estimate in the absence of a full-year appropriation, and SBA therefore provided the President’s budget to SBDCs for planning purposes. Although SBA characterized its use of the President’s budget as a standard practice within the federal government, SBA officials did not point to a specific regulation or guidance to support this view, either in the letter or in our interviews. OMB’s Uniform Guidance does not advise federal agencies to use the lowest estimate (that is, the lowest among the House mark, Senate mark, President’s budget request, and the prior year’s appropriation) when reporting funding available under federal financial assistance programs. Rather, the guidance states that estimates based on the previous year’s funding are acceptable if the current appropriations are not yet available, as was the case when recent SBDC funding opportunity announcements were issued. Although OMB’s Uniform Guidance does not identify any other accepted practice in this regard, it does not expressly prohibit other practices. OMB staff told us that, in the absence of a full-year appropriation, it is permissible for agencies to use the lowest estimate as the funding estimate in a funding opportunity announcement. The staff said that from their perspective, SBA’s use of the figure in the President’s budget is consistent with OMB Circular A-11 and the Uniform Guidance. However, they also acknowledged that there is no requirement to use the lowest estimate in a funding opportunity announcement. They also said that practices vary across agencies. During our limited review of other SBA and federal award programs, we did not find other programs that had funding application practices similar to those for SBDCs. However, those programs may not be entirely comparable to the SBDC Program, which publishes its funding opportunity announcement before the annual appropriation is known, requires applicants to initially apply based on a funding estimate, and is subject to a statutory formula that calculates individual grant amounts based on the final appropriation. The following are examples of ways in which the programs we reviewed differed from the SBDC Program: According to SBA officials, the funding for its Women’s Business Centers is awarded at the end of the fiscal year, and thus the final appropriation is known during the application process. In a fiscal year 2018 funding opportunity announcement for SBA’s Veterans Business Outreach Center Program, the total amount of available funding was known at the time applicants submitted their proposals. Like the Veterans Business Outreach Center Program, a fiscal year 2018 funding opportunity announcement for SBA’s Federal and State Technology Partnership Program listed the amount of funding available. Applicants for the Defense Logistics Agency’s Procurement Technical Assistance Program (PTAP) can apply for specific amounts of funding based on their service areas. For example, in a fiscal year 2020 funding announcement for PTAP, applicants providing statewide coverage could apply for up to $1 million in funding and those providing less than statewide coverage could apply for up to $600,000. In addition, the funding announcement did not include a funding estimate for the program as a whole. SBDC Program officials also cited adherence to the Antideficiency Act as a reason for the change. SBA stated in its letter to the Senate and House Small Business Committees that its practice of providing SBDCs with the President’s annual budget request as the dollar amount for planning purposes ensures that SBA abides by the terms of the Antideficiency Act. In doing so, they noted the Antideficiency Act prohibits obligating or expending funds in excess of amounts available through appropriations. In interviews, SBA officials said that if they consistently took a conservative approach, they would be less likely to violate the Antideficiency Act when appropriations were enacted. However, OMB staff told us that SBA’s use of other estimates in a funding opportunity announcement would not violate the Antideficiency Act. SBA’s funding opportunity announcements make clear that funding awards will be based on the appropriated level of funding, not the estimate provided in the funding opportunity announcement. OMB staff told us that because SBA’s funding opportunity announcement does not obligate the federal government, it does not present the potential for a violation of the Antideficiency Act. They further stated that using a higher amount, such as the prior year’s appropriation, also would not violate the Antideficiency Act. Consistent with that position, SBA’s Office of General Counsel agreed that the Antideficiency Act does not apply to the SBDC funding opportunity announcement because it does not create an obligation on behalf of SBA. Counsel also stated that SBA’s decision to use the President’s budget as the funding estimate was a management decision and was not required by the Office of the General Counsel. Most SBDCs that responded to our survey told us that using the lowest budget estimate—the President’s budget request—as the funding estimate for fiscal year 2020 had created budgeting, operational, and performance burdens and challenges. The burdens and challenges largely stemmed from the large gap between the initial funding estimates and final appropriations. According to SBA officials, using the amount in the President’s budget as the funding estimate should not negatively affect SBDCs that are on a calendar-year budget cycle (approximately two-thirds of SBDCs) because they have approved funding through December 31 of each year. They stated that by January 1 of each year, there is either a budget or continuing resolution in place that eliminates uncertainty for these SBDCs. They also stated that SBDCs on a calendar-year cycle are not required to submit their initial applications based on the estimated funding amount in the funding opportunity announcement (that is, the amount in the President’s budget), and can instead use a higher amount under a continuing resolution or other approved budget. However, SBA did not provide any documentation authorizing SBDCs on a calendar-year budget cycle to submit initial applications using a higher estimated funding amount. Rather, the fiscal year 2020 Funding Opportunity Announcement states that the funding estimate will be based on the President’s budget, and the announcement applies to both fiscal-year and calendar-year SBDCs. Lastly, SBA’s view that calendar-year SBDCs should not be negatively affected by SBA’s use of the lowest budget estimate is not consistent with SBDC responses to our survey. Both fiscal-year and calendar-year SBDCs reported in their survey responses that using SBA’s funding estimate had hindered different aspects of their operations. In the following discussion of these and other survey results, we note the limited instances in which calendar-year SBDCs reported a different experience than fiscal-year SBDCs. The vast majority of SBDCs responding to our survey said the large gap between the initial funding estimates and final appropriations imposed an additional administrative burden for SBDCs as they developed their budgets. Fifty-one out of 58 SBDCs (88 percent) responding to our survey question estimated that staff time to prepare the initial and final 2020 funding applications was somewhat or greatly increased compared to previous years (when the estimated funding amount was approximately the same as the current year’s award). Many SBDCs noted that using SBA’s funding estimate for fiscal year 2020 had created an additional administrative burden in their responses to open-ended questions on our survey. For example, one SBDC said that requiring two budget justifications (an initial funding application and a revised application) added time, complexity, duplication of effort, and considerable paperwork to the budgeting process. Another SBDC said the process of submitting multiple budgets took extra time and increased the likelihood of human error. Survey comments help explain why the use of the lowest budget estimate made budgeting more burdensome. For example, one SBDC explained that the lower amount required more time for negotiations to assure their matching partners that the actual number would likely be higher. Another SBDC noted the lower funding estimate required more proposed budget cuts in the initial application. Most SBDCs said using the funding estimate in the 2020 Funding Opportunity Announcement (which was $30 million below the 2019 appropriation and $34 million below the 2020 appropriation) required them to propose cuts in their initial budgets. Forty of the 57 SBDCs (70 percent) that responded to the survey question said their initial budget proposals eliminated some salaries, fringe benefits, and travel; 34 of 57 (60 percent) reduced supplies; and 33 of 57 (58 percent) reduced their contractual obligations. See table 1 for a full breakdown. Our survey asked SBDCs to provide a brief description of how they made adjustments to account for the decrease in the estimated funding amount in 2020: Some SBDCs stated that they had zeroed out the lead center or one or more service centers. A few SBDCs said they eliminated all part-time staff. Some also told us they decreased travel and professional development. Some SBDCs moved contractual line items (such as rent and software licenses) off the budget or reduced use of service providers (such as business consultants and independent contractors). Other survey respondents provided insight on why they made certain proposed cuts. To limit the burden on its service centers, one SBDC budgeted zero federal funds for the lead center and budgeted its service centers at the prior year’s amount so that its service centers would only need to budget once. Another SBDC stated that it chose the largest single line item that could be quickly reduced to meet the funding estimate. It noted that this allowed it to quickly scale down the budget and would allow it to quickly scale the budget back up to “reality.” Some of the survey respondents and SBDCs we interviewed indicated that they did not expect to have to make the cuts they proposed in their initial budgets because they were confident that Congress would appropriate the same or more funding as the prior fiscal year. One SBDC surveyed said that it took all budget deductions from the lead office personnel, but that it would not do so in a final budget if the lower amount in the President’s budget was to be appropriated. Three SBDCs we interviewed said that to account for the decrease, they proposed cuts in their initial budget they did not think they would have to make, such as eliminating an entire service center or a core information technology system. One SBDC we interviewed described the process as putting together two budgets simultaneously: one budget using the President’s budget request and one budget that was based on a funding amount that it believed was more consistent with historical norms. Still other survey respondents reported that they actually reduced their expenditures after receiving the funding estimate, although the amounts ultimately appropriated for SBDCs in fiscal years 2019 and 2020 were higher than for prior years. In response to the same open-ended question asking for a brief description of how they made adjustments to account for the decrease in the estimated funding amount in 2020, SBDCs provided the following examples: One SBDC said that to maintain all of its full-time personnel within the constraints of the initial application, some of its service centers had to stop providing training classes or attending conferences. Another SBDC told us a key staff person was given a layoff notice. Another said that due to the budget uncertainty, the SBDC eliminated all travel, cancelled almost all of its business development software subscriptions, and delayed replacing old computer equipment. Thirty-eight of the 60 SBDCs that responded to our survey (63 percent) said that using the initial funding amount as the basis for their initial applications somewhat or greatly hindered their ability to obtain matching funds. As discussed previously, SBDCs are required to match SBA funding at a 1:1 ratio. Host institutions (often supplemented by local governments, higher education institutions, and private-sector groups) provide matching funding. In response to open-ended survey questions, SBDCs noted the following: Difficulty obtaining full matching funding. Some survey respondents said that submitting a funding application based on the initial estimate put the SBDC’s ability to secure 100 percent of matching funding at risk. For example, one SBDC stated that the initial funding amount was the basis for its host institution’s budget for the match and that once the host’s budget was approved, it was difficult to amend it to increase the match (to meet the 1:1 requirement). Another SBDC said that host institutions only contribute a match equal to the federal allocation and that if the final allocation amount is not known, the host may choose to invest in other initiatives. Negative effect on relationships. Most survey respondents expressed concern that the change to the funding estimate would create confusion or uncertainty with their hosts or service centers (partners). For example, one SBDC told us that its host institutions do not react well to ambiguity in planning. This SBDC described the change in amounts between the initial estimate and the final appropriation as an unwelcome surprise to its partners, which needed to match the difference. Another SBDC noted that the decrease in the initial funding amount in 2020 was so great that several funding partners indicated they would partner with other organizations that were better funded and supported. Many survey respondents stated they were able to mitigate the impacts of the lower funding estimate by using matching funds, either because their host provided the same amount it had provided the previous year or overmatched (provided funds in excess of the statutorily required match). For example, one SBDC told us it was fortunate to have strong support from its host to be able to temporarily fund program operations at full capacity, until the revised funding amount from SBA was released. Three survey respondents who reported that using the initial funding amount neither helped nor hindered their efforts to obtain matching funds either stated they did not share the estimated amount with their hosts or local partners or that the amount they shared was equal to their portion of the appropriated amount. Forty-two of 60 SBDCs that responded (70 percent) said using the initial funding amount as the basis for their initial applications somewhat or greatly hindered their ability to spend grant funds. In response to an open-ended survey question asking for examples of how using the initial funding amount had affected their ability to spend grant funds, SBDCs noted the following: Having to spend conservatively early in the year. Some SBDCs told us that the funding process requires them to spend conservatively at the beginning of the budget cycle, only to ultimately receive more than the prior year’s amount. As shown in figure 2, SBDCs did not receive the notice of award for their full fiscal year 2019 appropriation until April 2019. One SBDC noted the extreme pressure in the first half of the year to operate on a lean budget and then having to switch to increased activity in the second half of the year once the final, higher amount was awarded. Another SBDC required its service centers to propose special projects but did not fund them at the beginning of the year because it did not know what the funding level would be. Having to carry over funding to the next year. Some SBDCs said it has become common for them to have unexpended funding left at the end of the fiscal or calendar year. In these instances, they carry over their funding to the next year. For example, one SBDC said that the late-in-year increases in funding prevented certain activities from being completed within appropriate project dates and led them to carry over funds. While the ability to carry over funding for one additional fiscal year was considered helpful, some expressed concern that this was not the best use of federal dollars over the course of the year. For example, one SBDC said it was forced to carry over unspent funds partly as a result of difficulties in forecasting monthly spend rates and adjusting the rates midyear. A few survey respondents noted that operating under a continuing resolution at the beginning of the year made spending grant funds more difficult. For example, one SBDC stated that multiple continuing resolutions in one year meant that by the time its host rebudgeted, it had less than 3 months to spend the increase. Another noted that because the initial funding amount varied so greatly from the final amount, the shortened time frame for deploying funds can make it difficult to maximize use of grant funds. Two SBDCs that operate on a calendar-year budget cycle told us that they had not experienced the same challenges as SBDCs that operate on a fiscal-year budget cycle because there is either a continuing resolution or final award by January. A smaller proportion of calendar-year SBDCs responded that using the initial funding amount somewhat or greatly hindered their ability to spend grant funds compared to fiscal-year SBDCs. However, the majority of calendar-year SBDCs still responded that using the amount somewhat or greatly hindered their ability to spend grant funds. Forty-four of 60 SBDCs that responded to our survey (73 percent) said using the initial funding amount as the basis for their initial applications somewhat or greatly hindered their ability to hire personnel. As mentioned previously (see table 1), 40 of the 57 SBDCs (70 percent) that responded to the survey question about reducing budget items in their initial funding application reduced salaries in their initial funding application. In response to various open-ended questions in our survey, SBDCs noted the following: Hiring delays or freezes. Many SBDCs reported delays in hiring or hiring freezes. For example, one SBDC told us it had to leave vacant counseling positions unfilled, although it knew the funding for those positions almost certainly would materialize. Another SBDC noted that using SBA’s current funding estimate created a minimum of a 6-month delay in hiring a new adviser at a local center. Another SBDC said its host institution would not allow any staff to be hired in the period from SBA issuance of the funding estimate through the congressional appropriation. Thus, when consultants and staff retire or leave the program, there is a staffing gap, which results in less service to clients. Reliance on short-term or contractor positions. Some SBDCs told us that using the funding estimate already had forced them to rely on or might force them to rely on part-time staff, short-term contracts, or contractors to provide services. For example, one SBDC stated the initial funding estimate restricted its ability to hire full-time personnel and instead required it to hire part-time individuals. This SBDC also noted it had a difficult time finding qualified individuals, since it could not guarantee funding for the new position in 6 months. Another SBDC said it was forced to put all personnel on short-term contracts. Another SBDC opted to use contractors for specialized projects and subject matter experts, rather than making a long-term investment in a core business advisor, because those hires were more flexible in the face of budget uncertainty. In addition to difficulty in hiring personnel, 35 of 60 SBDCs that responded to our survey (58 percent) said that using the initial funding amount as the basis for their initial applications had somewhat or greatly hindered their ability to retain personnel. In response to various open- ended questions in our survey, SBDCs cited morale and retention issues: Staff morale. Some SBDCs said that using the lower funding estimate had affected staff morale. Some respondents attributed this decline in morale to the lack of job security and funding certainty. Two others told us they withheld personnel management information from their staff and other centers to minimize the impact on morale. Staff retention. Some SBDCs noted that staff members left the SBDC to seek employment elsewhere in response to the uncertainty created by the lower initial funding amount. For example, one SBDC said that for 6 months its staff heard about the uncertainty and lack of a stable budget, which led to staff members leaving for other jobs. In addition, a few SBDCs noted that they were unable to offer a competitive salary. For example, one SBDC said that the initial funding estimate dictates matching funding, which constricts its ability to raise salaries to a competitive level. Survey respondents provided examples of how using the lower estimate in the President’s budget as the basis for the SBDC funding estimate negatively affected their ability to plan for new operations and expand services. Some SBDCs said they were not able to plan and promote a program until far into the fiscal year when the final notice of award was released. For example, one SBDC said it was very difficult to plan for expansion because the initial funding amount was barely enough to sustain operations. Two SBDCs said they had delayed opening one or more centers. For example, one SBDC said it would not be able to expand to additional rural areas and instead only would be able to maintain existing operations. The SBDC noted that it would have been able to expand with the amount of funding it eventually received, if that funding had been received at the start of the budget cycle and made available in a predictable manner. Two SBDCs stated that the funding ambiguity affects their host institutions’ ability to plan their own budgets, which directly affects the amount of matching funding they are able to provide. Thirty-one of 59 SBDCs that responded to the survey question (53 percent) said that using the initial funding amount as the basis for their initial applications somewhat or greatly hindered their ability to meet performance goals. SBDCs have four performance goals against which they agree to be evaluated when applying for SBA funding—number of jobs supported, number of new business starts, number of clients served, and amount of capital infusion. In response to an open-ended question that asked for examples of how using the initial funding amount had affected their ability to meet performance goals, many SBDCs pointed to the staffing difficulties discussed earlier as a reason for the increased difficulty in meeting performance goals. Some SBDCs surveyed also mentioned that performance goals did not decrease when the initial funding estimate was lower than the prior year’s funding amount. One SBDC noted that its goals are based on the assumption that the full funding amount will be available over 12 months, rather than from 4 to 5 months after the start of the year. Thirty-five of 59 SBDCs that responded to the survey question (59 percent) said using the initial funding amount as the basis for their initial applications somewhat or greatly hindered their ability to provide services. As noted previously, the purpose of the SBDC Program is to deliver business advising and technical assistance to prospective and existing small businesses. In response to an open-ended question that asked for examples of how using the estimated funding amount affected their ability to provide services, a few SBDCs mentioned the budget process took time away from providing services. For example, one SBDC said that its time and energy was split between the core mission and addressing budget uncertainty. Similarly, another SBDC stated that the extensive funding application work took time away from providing direct services to clients. Two SBDCs also pointed to gaps in service created by the uncertain funding situation. For example, one SBDC noted gaps in service delivery in terms of geographic coverage and of expanding technology training for clients because of the inconsistent budget environment. In prior work, we described legislative authorities and agency actions that may mitigate challenges associated with budget uncertainties. For example, in a 2018 testimony, we noted that Congress may include specific provisions in continuing resolutions (called legislative anomalies) that provide some agencies or programs with funding or direction different from those specified in the standard provisions that require agencies to spend more conservatively. For example, programs that previously received a specific or additional amount of funding under a continuing resolution have included wildfire management, veterans healthcare and benefits, and disaster relief. In addition, agencies can take actions to mitigate challenges associated with continuing resolutions and shutdowns. For example, agencies may have the ability to shift grant cycles to later in the fiscal year when they are less likely to be under a continuing resolution. Shifting these cycles can help minimize disruption of services. In response to an open-ended survey question on how SBA could help mitigate any of the challenges posed by the funding application procedure, SBDCs offered a number of suggestions. Many recommended that SBA allow SBDCs to submit funding applications at the prior year’s appropriated amount, as generally had been the practice historically. As noted previously, OMB’s Uniform Guidance offers SBA the flexibility to use other estimates, such as the prior year’s appropriation. A few also suggested a more streamlined process that does not involve multiple funding applications (and budgets), but a process in which SBDCs submit budgets only once. A few recommended 5-year funding for the program. SBDCs face administrative burdens and operational challenges stemming from SBA’s current practice of using the lowest budget estimate—the amount in the President’s budget—as the funding estimate for SBDC funding applications. More specifically, because of the large gap in recent years between the President’s budget and appropriated amounts for the program, SBDCs told us they now spend more time on budgeting, have a harder time obtaining matching funds, and have difficulty hiring and retaining staff. This in turn affected their ability to meet their performance goals and program objectives of serving small businesses. The use of continuing resolutions in recent years also has put strains on SBDCs’ ability to effectively plan, budget, and operate throughout a year. And if the funding estimates continue to decrease (as is already the case for fiscal year 2021) and diverge from appropriations in such an environment, SBDCs will face increased challenges in the areas cited above. SBA could take actions to alleviate some of the burden on SBDCs, in particular by reevaluating its funding application requirements for the SBDC Program. For example, the agency could reconsider the amount it uses as the basis for the funding estimate. While SBA previously cited a desire to improve operational efficiencies as a reason for changing the funding estimate methodology, most SBDCs told us their operational efficiencies have decreased. SBA also cited a need to align with federal financial management standards, but OMB’s Uniform Guidance permits the use of other estimates, such as the prior year’s appropriation. Lastly, SBA cited concerns over violating the Antideficiency Act, but OMB staff and SBA’s Office of General Counsel confirmed that the Antideficiency Act does not present a barrier to using other estimates. SBA might also reconsider other aspects of the funding application process that cause burden, such as the timelines for submitting applications and the number of times SBDCs must submit detailed budgets. We are making the following recommendation to SBA: The Associate Administrator of the Office of Entrepreneurial Development should reevaluate the SBDC funding application requirements, including reexamining the funding estimate SBDCs are required to use and considering other changes that could reduce administrative burdens on SBDCs. (Recommendation 1) We provided a draft of this product to SBA and OMB for their review and comment. OMB did not provide comments. SBA provided written comments that are reprinted in appendix III. In its written comments, which are summarized below, SBA partially agreed with our recommendation and recognized room for improvement in how it sets the funding estimates for the SBDC Program. SBA did not explicitly state with which part of our recommendation it disagreed; rather, SBA reiterated its view that the agency’s practice of planning to the more conservative President’s annual budget request affects only fiscal-year SBDCs. In addition, SBA listed two changes it was considering to improve the funding application process. SBA first stated that it is exploring moving the program start date for fiscal-year SBDCs to January 1, which would make all SBDCs operate on a calendar-year basis. SBA also indicated that it is considering publishing the fiscal year 2021 Funding Opportunity Announcement later in the fall (for example, on the first day of October), by which time the agency would be operating on a continuing resolution or final appropriation and would no longer be working with funding estimates. These steps are promising, but we would need to evaluate their implementation to determine whether they fully address our recommendation. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Administrator of SBA, the Acting Director of OMB, and other interested parties. In addition, the report will be available at no charge on the GAO website at https://gao.gov. If you or your staff members have any questions about this report, please contact William B. Shear at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs maybe found on the last page of this report. Major contributors to this report are listed in appendix IV. This report discusses the (1) Small Business Administration’s (SBA) rationale for changing the way it estimates funding in the annual funding opportunity announcement for the Small Business Development Center (SBDC) Program and (2) views of SBDC grantees on the effect of that change on their budgeting and operations. To determine SBA’s rationale for changing the way it estimates funding in the annual SBDC funding opportunity announcement, we reviewed SBDC funding opportunity announcements for fiscal years 2012 through 2020 and program guidance that governs the SBDC funding application procedure. We compared the funding estimates in the funding opportunity announcements, appropriations, and Presidents’ budget requests from fiscal years 2012 through 2020. We also interviewed SBA officials in the SBDC program office and Office of General Counsel to obtain the agency’s rationale for changing the way it estimates the funding amount in its SBDC funding opportunity announcements. In addition, we examined select laws (such as the Antideficiency Act) and regulations (such as the Office of Management and Budget’s (OMB) Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards). We interviewed OMB staff and officials in SBA’s Office of General Counsel to obtain their views on the relevance of the Antideficiency Act and the extent to which SBA’s funding proposal procedure is consistent with that law. We also reviewed documentation on other selected programs to compare the funding application processes used: the Women’s Business Center, Veterans Business Outreach Center, Federal and State Technology Partnership, and Procurement Technical Assistance Programs. We selected these programs because they were other SBA grant programs or because SBDCs we interviewed mentioned them as federal grant programs that used different budget practices. (We discuss below how we selected the SBDCs to interview.) To gather the views of SBDC grantees on the effect of the funding estimate change on their planning and operations, we interviewed representatives of a nongeneralizable sample of eight SBDC lead centers, selected to achieve diversity in funding amount, budget cycle, and host institution. Their views are not generalizable to other SBDCs but offered important perspectives. We also reviewed funding application documentation from these eight lead centers to determine their funding timelines. We then focused on the fiscal year 2019 timeline for SBDCs that use a calendar-year budget cycle and SBDCs that use a fiscal-year budget cycle to identify any differences in their timelines. We selected fiscal year 2019 because it was the most current complete funding cycle at the time we conducted our work. In addition, we conducted a web-based survey of all 62 SBDC lead centers to obtain their perspectives on the effect on their operations of the change in how SBA estimates SBDC funding. In total, we obtained 60 responses (a 97 percent response rate). We conducted four pretests of our draft questionnaire by telephone with officials at four SBDC lead centers with varying characteristics, such as amount of funding, budget cycle, and host institution. We used these pretests to help refine our questions, develop new questions, clarify any ambiguous portions of the survey, and identify any potentially biased questions, and we made revisions as necessary. We launched our web-based survey on January 30, 2020, and emailed log-in information to the directors of the SBDC lead centers. The survey was available through February 7, 2020. From February 10, 2020, through February 14, 2020, we conducted follow-up with nonrespondents by telephone and email. See appendix II for the full set of survey results. The survey included both closed- and open-ended questions. To analyze open-ended comments provided by the SBDCs that responded to the survey, GAO analysts read the comments and developed categories for the responses. An initial coder assigned a category that best summarized the comments. A separate coder reviewed and verified the accuracy of the initial categorizations. The initial coder and reviewer discussed orally and in writing any disagreements about code assignments and documented consensus on the final analysis results. For purposes of this report, we used the following terms to describe the number of SBDCs (out of 60) that were assigned to categories: “few” to describe three to five SBDCs, “some” to describe six to 15, “many” to describe 16 to 30, “most” to describe 31 to 45, and “vast majority” to describe 46 and over. We conducted this performance audit from October 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. To obtain the views of Small Business Development Centers (SBDC) on the Small Business Administration’s (SBA) change in the way it sets funding estimates, we conducted a web-based survey of all 62 lead SBDCs. Our survey comprised closed- and open-ended questions. This appendix includes all survey questions and results of each closed-ended question; it includes only the number of responses for each open-ended question. We received surveys from 60 lead SBDCs (a 97 percent response rate). The total number of responses to individual questions may be fewer than 60, depending upon how many lead centers chose to respond to a particular question. For a more detailed discussion of our survey methodology, see appendix I. For questions 7 through 13, the “initial funding amount” refers to your SBDC’s portion of the estimated funding of $101 million total available in the 2020 SBDC Funding Opportunity, which was ultimately below the $135 million later authorized under the Consolidated Appropriations Act, 2020, Pub. L. No. 116-93 (Dec. 20, 2019). (Question 7a.) Please provide a specific example or examples of how using the initial funding amount has affected your SBDC’s ability to obtain matching funds. We received 49 responses to this question. (Question 8a.) Please provide a specific example or examples of how using the initial funding amount has affected your SBDC’s ability to spend grant funds. We received 51 responses to this question. (Question 9a.) Please provide a specific example or examples of how using the initial funding amount has affected your SBDC’s ability to retain personnel. We received 49 responses to this question. (Question 10a.) Please provide a specific example or examples of how using the initial funding amount has affected your SBDC’s ability to hire personnel. We received 52 responses to this question. (Question 11a.) Please provide a specific example or examples of how using the initial funding amount has affected your SBDC’s ability to ensure continuous operations. We received 54 responses to this question. (Question 12a.) Please provide a specific example or examples of how using the initial funding amount has affected your SBDC’s ability to meet performance goals. We received 44 responses to this question. (Question 13a.) Please provide a specific example or examples of how using the initial funding amount has affected the ability of all your centers to provide services. We received 49 responses to this question. (Question 14a.) Please provide a brief description of how your SBDC made adjustments to account for the decrease in the estimated funding amount in 2020. We received 60 responses to this question. (Question 19) How could SBA help mitigate any of the challenges posed by the funding application procedure, if at all? We received 56 responses to this question. (Question 20) Is there anything else related to the current SBDC funding application procedure on which you would like to elaborate? We received 42 responses to this question. In addition to the contact name above, Paige Smith (Assistant Director), Meredith P. Graves (Analyst in Charge), Kristine Hassinger, Jill Lacey, Jason Marshall, Marc Molino, Kirsten Noethen, and Barbara Roesmann made significant contributions to this report.", "summary": "SBA's SBDC Program provides training and counseling to small businesses through a nationwide network of 62 lead centers and more than 900 service centers. Each year, SBDC lead centers submit grant applications based on an estimated amount in SBA's funding opportunity announcement. GAO was asked to review SBA's procedure for the SBDC funding estimate. This report discusses SBA's change to the way it estimates funding in the funding opportunity announcement, its rationale for the change, and views of SBDC grantees on the effect of the change on their budgeting and operations. GAO reviewed SBDC funding opportunity announcements, Presidents' budget requests, and appropriations for fiscal years 2012–2020; examined relevant laws and guidance; and interviewed SBA officials and OMB staff. GAO also reviewed documentation and interviewed officials from a nongeneralizable sample of eight SBDCs (selected to achieve diversity in funding amount, budget cycle, and host institution) and surveyed all 62 lead SBDCs. The Small Business Administration (SBA) annually issues a funding opportunity announcement with an estimate of total funding for the Small Business Development Center (SBDC) Program. Individual SBDCs are required to use this estimate to apply for their portion of the funding. In fiscal year 2016, SBA began using the lowest funding estimate—the amount in the President's budget—rather than an estimate reflecting historical funding levels. In fiscal year 2019, the amount in the President's budget was 15 percent lower than the prior-year appropriation and in 2020, 23 percent. If SBA continues its practice for fiscal year 2021, the funding estimate will be 35 percent lower than the 2020 appropriation. When appropriations are enacted for the program, the funding amount is revised and SBDCs submit a final budget. SBA officials said they changed how they set the funding estimate to conform to federal standards and appropriations law. In a 2019 letter to the House and Senate Small Business Committees, SBA said it adopted the change to help the program operate more effectively and be consistent with federal financial management standards. SBA officials could not point to specific regulations or guidance to support this statement. Office of Management and Budget (OMB) guidance for grants states that estimates based on the previous year's funding are acceptable if current appropriations are not yet available, as was the case when recent SBDC funding opportunity announcements were issued. SBA officials also cited the Antideficiency Act, which prohibits federal agencies from obligating or expending federal funds in advance or in excess of an appropriation. But staff from OMB and SBA's Office of General Counsel told GAO that the Antideficiency Act does not apply to a funding opportunity announcement because the announcement does not obligate federal funds. A majority of SBDCs that GAO surveyed said using the President's budget request for the initial funding estimate created budgeting, operational, and performance burdens and challenges—mostly stemming from the large gap between the initial estimate and appropriated amounts. For example, SBDCs surveyed said that they now spend more time on budgeting (determining what to cut from initial budgets to meet the lower estimate and then recalculating for final budgets); have a harder time obtaining matching funds (from state, local, or private-sector sources) or increasing the amounts from initial to final funding levels; have difficulty hiring or retaining staff; face challenges providing services to small businesses (particularly if SBDCs have staffing gaps); and thus also face challenges meeting performance goals (which include number of clients served). Under SBA's current practice for funding estimates, SBDCs will continue to experience (or may experience increasing) challenges given the growing divergence between the initial estimate and appropriated amounts. GAO is making one recommendation that SBA reevaluate its funding application requirements, including the initial funding estimate SBDCs are to use. SBA partially agreed and outlined steps it plans to take that could address the intent of the recommendation.", "document_type": "gao"}
{"report": "The United States historically has maintained close ties to Central America and played a role in the region’s political and economic development because of geographic proximity and common interests. The United States has provided assistance to the governments of Central America, including those of the Northern Triangle, under multiple initiatives over many years. In 2008, the United States began a multiyear assistance package to Central America under the Mérida Initiative to help address violence and criminal activity, especially from drug trafficking and other criminal organizations. In 2010, U.S. assistance continued under CARSI. CARSI was a collaborative partnership between the United States and Central American partner countries, including El Salvador, Guatemala, and Honduras, designed to improve citizen security within the region by taking a broad approach to security beyond traditional counternarcotics activities. Multiple U.S. agencies implemented projects in Central America, particularly in the Northern Triangle, to support and complement these initiatives. These projects focused on, among other things, improving law enforcement and criminal justice, promoting the rule of law and human rights, preventing youth violence in violence-prone areas, enhancing customs and border control, and encouraging economic and social development. Introduced in 2014, and updated in 2017, the Strategy is the latest U.S. government initiative in the region. The Strategy notes that prior U.S. assistance did not yield sustained, broad improvements in social or economic conditions and thus the Strategy intends to take a comprehensive, an integrated, and a whole-of-government approach that aligns activities and resources required to achieve systemic and lasting improvements. Under this approach, the Strategy promotes three mutually reinforcing objectives—prosperity, governance, and security. These three objectives seek to address challenges facing Central American countries, including the three Northern Triangle countries. For example: Prosperity Challenges: Northern Triangle countries have had high rates of poverty, low per capita income, and a lack of employment opportunities. The World Bank reported that, in 2014, over half of the population of Guatemala lived below the poverty line and, in 2017, almost one-third of the population of El Salvador and more than half of the population of Honduras lived below the poverty line. The World Bank also reported that El Salvador, Guatemala, and Honduras had among the lowest per capita incomes in Latin America in 2017. In addition, more than 27 percent of the population aged 15 to 24 in each of the Northern Triangle countries were not employed or seeking education or training in 2016, according to the World Bank. Governance Challenges: Northern Triangle countries have experienced widespread corruption, weak government institutions, and poor adherence to the rule of law. According to the 2018 Transparency International Corruption Perception Index, which ranks 180 countries by their perceived levels of public sector corruption, the Northern Triangle countries ranked among the bottom half. In addition, in 2018, Guatemala and Honduras ranked in the lowest 15 percent of countries in the World Justice Project’s Rule of Law Index, which measures countries’ adherence to the rule of law. Security Challenges: Northern Triangle countries have had weak security structures, high rates of crime and gang activity, and a lack of legitimate employment opportunities for youth susceptible to being drawn into criminal activity. While Northern Triangle countries experienced a decline in homicide rates from 2014 to 2017, the average homicide rate for El Salvador, Guatemala, and Honduras remains much higher than the averages for Latin America and the Caribbean for recent years and five to 12 times higher than the 10- year average for the United States. In addition, the percentage of people in the Northern Triangle who reported feeling safe walking in their neighborhoods at night was about 50 percent in 2017. Agencies reported implementing various assistance projects in the Northern Triangle to support the prosperity, governance, and security objectives from fiscal year 2013 through fiscal year 2018. We found that these projects generally correspond to 18 sectors that align with the three objectives of the current Strategy. Figure 1 shows the alignment of the 18 sectors with the objectives of the Strategy, including the six sectors we selected for an in-depth review. Table 1 shows the definitions for each of the 18 sectors we identified and the three objectives of the Strategy. Multiple agencies implemented assistance projects in the Northern Triangle to support the prosperity, governance, and security objectives from fiscal years 2013 through 2018. State, USAID, DOD, and USDA were the primary agencies that implemented such projects in the Northern Triangle during this period. In particular, State and USAID manage foreign assistance to support the Strategy’s objectives, and play key roles in monitoring and evaluating this assistance. According to agency officials, State’s Bureau of Western Hemisphere Affairs (WHA) is responsible for managing the implementation of the Strategy’s objectives among agencies. For example, WHA manages regular coordination meetings with USAID and State’s Bureau of International Narcotics and Law Enforcement Affairs (INL) as well as larger coordination meetings with other relevant agencies, including DOD and USDA, according to officials. In addition, WHA gathers information across agencies on a quarterly basis to produce and disseminate cables that discuss progress and challenges related to the Strategy’s objectives. WHA also collaborated with USAID to develop a plan to monitor and evaluate U.S. assistance and report results. Based on our review of agency funding data, we found that State, USAID, DOD, and USDA allocated about $2.4 billion in assistance to the Northern Triangle to support projects related to prosperity, governance, and security objectives from fiscal years 2013 through 2018. USAID reported the largest amount of allocations with approximately $1.44 billion, while State reported $464 million, and USDA and DOD each reported less than $235 million. For fiscal years 2013 through 2018, the four agencies reported allocating the largest amount of funding for projects in Guatemala, followed by Honduras and El Salvador. Specifically, the agencies reported allocating approximately $1.07 billion or 45 percent of total allocations to fund projects in Guatemala, approximately $749 million or 32 percent of total allocations to fund projects in Honduras, and approximately $496 million or about 21 percent of total allocations to fund projects in El Salvador. Some agencies also reported allocations for multi-country projects implemented in two or more countries, including at least one Northern Triangle country. For example, USAID funded a regional initiative to improve clean energy investment and reduce overall energy consumption throughout many Central American countries. The agencies reported allocating approximately $53 million for multi-country assistance projects implemented exclusively in two or three Northern Triangle countries, or about 2 percent of the total. See Table 2 for reported amounts of allocated funding by country and agency from fiscal years 2013 through 2018. State, USAID, DOD, and USDA implemented at least 370 technical assistance projects in the Northern Triangle to support prosperity, governance, and security objectives from fiscal years 2013 through 2018. The total number of projects that we report is lower than the actual number of projects implemented because some agencies and bureaus could not report data at the project level. Specifically, DOD and INL reported some broader assistance data that encompassed two or more projects and officials told us they were unable to disaggregate this data at the project level. Among the four agencies, USAID implemented the largest number of projects in the Northern Triangle during our time frame. Specifically, USAID reported that it implemented 218 projects or 59 percent of the projects reported across the four agencies. State reported that it implemented 124 projects or about one-third of the projects. DOD and USDA each reported 14 projects to support prosperity, governance, and security or about 4 percent each of the total projects. Collectively, the agencies reported they implemented the largest number of projects in Guatemala (126), followed by Honduras (106), and El Salvador (86). Agencies reported they implemented 52 multi-country projects that included at least one Northern Triangle country. See table 3 for the number of projects reported by country and agency. Agency officials typically reported implementing similar types of projects in each of the Northern Triangle countries, although there were some differences in the number of projects implemented for each objective and sector based on each country’s needs (see fig. 2). For example, officials told us that agencies implemented fewer agricultural development projects in El Salvador because its agriculture industry is small relative to Guatemala and Honduras and the majority of its population lives in urban rather than rural, agricultural areas. Instead, agency officials in El Salvador said agencies focused their prosperity assistance on projects in the economic growth sector that targeted more prominent business areas such as technology or manufacturing. For example, USAID supported a youth training center in El Salvador where students develop computer skills to work in the information technology fields (See fig. 3). Some agencies funded projects that supported multiple sectors and objectives, while others focused on a specific sector supporting one of the three objectives (see fig. 4). For example, USAID and State supported all three objectives by implementing projects in a variety of sectors. However, USDA supported only the prosperity objective by implementing projects primarily in the agricultural development sector and DOD supported the security objective by implementing projects primarily in the professionalize the military and develop defense capabilities sector. We also identified some specific assistance projects that supported more than one of the three objectives. For example, some of USAID’s workforce development projects targeted at-risk youth, which supported both the prosperity and security objectives. Other USAID projects worked with government officials in the Northern Triangle to improve health, environment, or economic growth, which supported both the prosperity and governance objectives. In addition, State’s rule of law projects, which trained police and other personnel in the judicial sector, supported both the governance and the security objectives. Below is an overview of the agencies’ general roles and responsibilities for supporting the three objectives: Prosperity: USAID, State, and USDA implemented projects supporting the prosperity objective. USAID implemented projects to assist populations to meet basic needs, help businesses access markets for goods and services, build a skilled workforce, and enhance health systems and education institutions. For example, one USAID economic growth project in El Salvador provided assistance to small enterprises through university-affiliated training centers where representatives of firms received training and advice to improve their business practices. State also implemented projects to assist businesses and entrepreneurs develop their capabilities. For example, State implemented a multi-country project to provide training to small and medium businesses on e-commerce platforms to access new markets and increase sales. USDA and USAID both implemented projects intended to help farmers improve agricultural management practices and increase their access to markets and capital. For example, a USAID agricultural development project in Honduras provided training to local farmers to increase their household incomes, strengthen access to food markets, and diversify their crops (see fig. 5). A USDA project provided schools in Honduras with food assistance, infrastructure improvements, and trainings to support school feeding, sought to improve educational outcomes (see fig. 6). Governance: USAID and State were the primary agencies supporting the governance objective. USAID projects provided technical assistance to governments to increase accountability, transparency, revenue collection, and provision of basic services. For example, a USAID project in Guatemala provided technical assistance to municipal governments to improve their financial management and increase the quality of government-provided services such as water and sanitation systems. State and USAID also supported this objective by supporting projects to strengthen justice institutions, combat corruption, improve democratic processes, and advocate for the protection of human rights. For example, we visited a morgue in Honduras where USAID and INL collaborated to provide forensic training and equipment and improve evidence collection and analysis capabilities, to better prosecute crimes (see fig. 7). Security: State, USAID, and DOD implemented projects to support the security objective. USAID and INL projects supported community based activities to prevent violence by supporting community youth centers, strengthening community policing, and implementing workforce development projects for at-risk youth. For example, a USAID project in Honduras provided technical training, mentorship, and job placement support for at-risk youth. INL also provided training and equipment to law enforcement to improve its capabilities and reputation in communities and to better identify and prevent crime, violence, and gang activity. For example, we visited the International Law Enforcement Academy in El Salvador, where U.S. assistance provides a variety of training courses to Central American and South American police, judges, and prosecutors, to increase capacity and coordination among law enforcement officials (see fig. 8). In addition, State funded and DOD funded and implemented projects to train and equip Northern Triangle militaries. DOD officials in Honduras, for example, told us they provide a range of trainings to Honduran military leaders at U.S. military schools. See appendix II for a summary of U.S. assistance projects in the Northern Triangle for our six selected sectors. State, USAID, DOD, and USDA reported mixed results, primarily focused on outputs, for the 190 projects in the six sectors we reviewed. While some projects in these sectors achieved the targets that agency officials established, others did not. We reviewed a variety of performance-related documents for the 190 projects that aligned with our six selected sectors—economic growth, agricultural development, good government service, justice reform, community based violence prevention, and professionalize the military and develop defense capabilities. Specifically, we reviewed State and USAID’s PPRs for fiscal years 2013 through 2018 for each Northern Triangle country, and State’s INCSRs for fiscal years 2013 through 2018. We also reviewed State and USAID’s Progress Report for the Strategy for fiscal years 2018 and 2019, and State’s quarterly country cables reporting on agencies’ progress in implementing projects that support the Strategy’s objectives in each of the Northern Triangle countries for available quarters of fiscal years 2016 through 2018. In addition, we reviewed implementer progress reports for a sample of 19 projects to obtain more detailed information on project-specific outputs and outcomes, as well as all available evaluations related to the six sectors completed from fiscal years 2013 through 2018. Examples of results for projects related to each of the six sectors include the following. Economic Growth: USAID implemented projects to assist workers improve their access to employment, and help firms improve their business practices and access markets. According to the PPRs we reviewed, USAID achieved 81 of 123 (66 percent) of its targets for performance indicators related to the economic growth sector for fiscal years 2013 through 2018. In addition, all nine evaluations in the sector reported generally positive project results. For example, according to the PPR, USAID assisted 176 firms to invest in improved technologies and 329 firms to improve their management practices in Guatemala in fiscal year 2017, exceeding the targets of 141 and 310, respectively. In addition, 5,067 individuals completed workforce development programs with U.S. assistance in the Northern Triangle countries in fiscal year 2018, according to the PPRs. USAID reported that 1,376 individuals completed workforce development programs in Guatemala, which exceeded the target of 1,000. However, USAID reported that 3,040 individuals in El Salvador and 651 individuals in Honduras completed such programs, which did not meet the fiscal year targets of 7,300 and 5,000, respectively. According to an evaluation of a USAID project in El Salvador that focused on providing training to individuals to improve their job opportunities, 3,585 individuals completed the training, which was 175 fewer than expected due, in part, to the project’s focus on training individuals for existing jobs and the scarcity of job opportunities for some individuals who completed the training. Agricultural Development: USAID and USDA implemented projects that provided assistance to apply improved agricultural technologies or management practices, and increase agricultural productivity and food security. According to the PPRs we reviewed, USAID achieved 58 of 86 (67 percent) of its targets for performance indicators related to this sector for fiscal years 2013 through 2018, and six of eight evaluations of agricultural development projects generally reported positive project results. For example, USAID reported in the PPR that 35,245 individuals in Honduras received short-term training with U.S. government support on agricultural productivity or food security in fiscal year 2018, exceeding the fiscal year target of 32,500, but 40,492 individuals received such training in Guatemala, which did not meet the target of 52,417. According to an implementer progress report, as of March 2017, an ongoing USDA school feeding project in Honduras had helped to construct and rehabilitate kitchens and food storage facilities at five of the 30 schools targeted by the project in 2017. An evaluation of a USDA project in El Salvador reported that the project issued 307 agricultural loans to improve agricultural production, which did not meet the target of 345 loans due, in part, to a delay in implementing the project. Good Government Service: USAID implemented projects to help create accountable and effective government institutions through improved provision of government services, increased citizen oversight, and greater ethics and transparency. According to the PPRs we reviewed, USAID achieved 22 of 30 (73 percent) of its targets for performance indicators related to this sector for fiscal years 2013 through 2018. Some of the projects achieved mixed results, according to an evaluation of projects in this sector. For example, USAID in the PPRs reported that in Honduras it exceeded targets in fiscal year 2018 by providing assistance to 94 local governments to improve public service and by training over 2,600 individuals in Guatemala in fiscal management to strengthen local government and foster decentralization. USAID met the target for fiscal year 2018 by having 81 public policies introduced, adopted, repealed, changed, or implemented with citizen input in Honduras. A USAID project in Guatemala designed to better manage public resources and government services reported in its fiscal year 2017 annual report that it helped 76 percent of the municipalities involved in the project increase their average monthly revenues following the project’s financial management training. However, an evaluation of two USAID projects in Honduras found that one project did not meet 70 percent of targets and struggled to successfully promote decentralization laws or increase municipal fiscal autonomy. Justice Reform: USAID and State provided technical assistance and equipment to help improve the efficiency of the courts and forensic laboratories, and strengthen the capabilities of prosecutors and judges. According to the PPRs we reviewed, USAID achieved 27 of 41 (66 percent) of its targets for performance indicators related to this sector for fiscal years 2013 through 2018. For example, according to the PPRs, 2,298 government officials in El Salvador received anti- corruption training with U.S. assistance in fiscal year 2018, surpassing the fiscal year target of 1,845. However, according to the PPRs, 150 individuals affiliated with nongovernmental organizations received such anti-corruption training in Guatemala in fiscal year 2017, which was below the fiscal year target of 550. The Progress Report for the Strategy for fiscal year 2019 reported that USAID assisted 244 courts in Guatemala to improve their case management systems in fiscal year 2018, which surpassed the target of 220. The Progress Report for the Strategy also reported that State and USAID trained 12,557 justice system personnel, including prosecutors and criminal investigators, in the Northern Triangle in fiscal year 2018; which surpassed the target of 2,275. Although State did not report targets, it provided data in its annual INCSR on U.S.-supported trainings, including training more than 1,000 police and justice sector personnel in El Salvador in 2016 and 2017, and 262 students in criminal investigations in Honduras in 2013. An evaluation of a USAID project in Guatemala noted the project helped improve prosecution practices and court management, but the evaluation also noted that continuous support would be required to preserve and consolidate reforms. Community Based Violence Prevention: USAID and State supported a number of efforts under the security objective to prevent violence in communities. According to the PPRs we reviewed, USAID achieved 7 of 18 (39 percent) of its targets for performance indicators related to this sector for fiscal years 2013 through 2018. For example, in El Salvador, 13 U.S. government-supported schools or other learning spaces met the criteria for the safe schools program in fiscal year 2018, surpassing the target of 10 schools. However, according to the PPR, in Honduras approximately 161,300 individuals participated in U.S.-funded gang prevention and education in fiscal year 2018, which did not meet the fiscal year target of 219,600. The Progress Report for the Strategy for fiscal year 2018 reported that State’s Gang Resistance Education and Training Program (GREAT) reached tens of thousands of youth and hundreds of police officers received instructor certifications to deliver anti-gang and crime prevention training through the program in the Northern Triangle in fiscal year 2017. However, State did not report targets for the program for the fiscal year. According to an implementer progress report, as of June 2018, an ongoing USAID project in Honduras that provides workforce development services for at-risk youth, had enrolled 2,528 of the project’s target of 6,500 youths for fiscal years 2017 and 2018. In addition, 440 of the project’s target of 2,488 youths for those fiscal years had completed the workforce development services as of June 2018, according to the report. Professionalize the Military and Develop Defense Capabilities: DOD and State supported efforts to professionalize the militaries of the Northern Triangle countries and develop their defense capabilities. While DOD and State reported positive output results for this sector, they also reported some limitations. According to the PPRs we reviewed, State achieved 48 of 71 (68 percent) of its targets for performance indicators for this sector for fiscal years 2013 through 2018. For example, in fiscal year 2018, State reported that 100 military personnel in Guatemala received technical or tactical training, which met the fiscal year target. State also reported that Guatemalan military personnel completed 12 exercises with U.S. or coalition personnel as a result of U.S. government assistance, which also met the target for fiscal year 2018. However, State reported that it supported the training of 44 fulltime peacekeeping staff in El Salvador in fiscal year 2017, which did not meet the target of 155. In its monitoring progress reports from fiscal years 2013 to 2018, DOD reported that it provided international military education and training to over 2,000 military personnel in the Northern Triangle, although DOD did not report targets. DOD personnel also engaged directly with Central American military personnel to improve their professionalism. For example, in Guatemala, DOD helped to establish a defense budget system designed to increase transparency and accountability of funds within the Ministry of Defense. However, DOD has reported ongoing challenges regarding the professionalism of Northern Triangle militaries and noted that public trust in the militaries remains low. Based on our review of various performance-related documents, we found limited information on progress toward improving prosperity, governance, and security in the Northern Triangle. Specifically, agencies generally reported more information about progress toward prosperity than toward governance and security. Some of the evidence about governance and security may be limited because evaluations were conducted unevenly across agencies and sectors. In addition, project implementers did not consistently collect key information to assess progress toward the Strategy’s objectives. Nevertheless, agency officials cited examples of important results from U.S. assistance as well as challenges to achieving progress toward the objectives. In addition, the Strategy’s monitoring and evaluation plan is not comprehensive because, while the plan specifies that State and USAID should track evaluations of their projects, it does not include a plan for evaluations of projects conducted by agencies other than State or USAID. For the sectors we reviewed, agencies generally reported more information on progress toward prosperity for projects related to economic growth and agricultural development, than toward governance and security. In addition, agencies generally reported positive information on progress toward prosperity for projects related to these sectors. For example, an evaluation of a USAID economic growth project in Guatemala reported the project supported 64 public-private partnerships that managed $39.1 million in investment, primarily from the business sector, for health, nutrition, and education activities to improve economic growth and development. In addition, USAID reported in the PPR that small and medium-sized firms assisted by its projects in El Salvador increased annual sales by approximately 40 percent in fiscal year 2016, which exceeded the target of 29 percent. In Guatemala, USAID also helped to increase crop yields by about $62 million and reduced household poverty by about 12.6 percent through two projects that trained agricultural producers in farm management practices and helped them access markets, according to an evaluation. Finally, an evaluation of a USDA agricultural development project in El Salvador reported that it helped generate approximately 12,930 new jobs, significantly exceeding the project’s goal of 900 jobs, in part, through increased access to credit and credit competency training. In general, however, little information was available from agency reports about progress toward the governance and security objectives. For example, an evaluation of a USAID project in good government service in Honduras that provided technical assistance to local governments to improve citizen satisfaction with services reported improvements in the quality of water and health services in most of the targeted municipalities, although the evaluation noted that the project had not developed appropriate indicators to measure results that were directly attributable to the project’s activities. Despite these improvements, the evaluation reported that the services remained largely unable to satisfy citizen needs adequately, and there was little evidence that municipalities would have the capabilities or resources to continue to improve the services without donor assistance. The evaluation also noted that the project promoted citizen advocacy by providing training to citizen oversight committees and establishing well-attended town halls in rural municipalities. However, it found no evidence such efforts were effective because the organizations remained too weak to advocate effectively for improved accountability and service. Another evaluation of a USAID project to prevent community based violence in Honduras reported significant reductions in homicide rates, ranging from 42 percent to 68 percent, in four of the six targeted communities, but also noted that these outcomes might not be attributable to the project’s activities. Although there were no evaluations of projects in the sector for professionalize the military and develop defense capabilities, DOD reported in its after action reports that it trained dozens of personnel who subsequently held positions of prominence within Northern Triangle militaries. The differences in results information for the three objectives are likely due, in part, to variations in the number of evaluations agencies conducted for their Northern Triangle projects. For example, we found that evaluations had been conducted unevenly across the agencies and six sectors we reviewed. Figure 9 shows the number of projects and completed evaluations of projects in the Northern Triangle that support the Strategy by agency and selected sector from fiscal years 2013 through 2018. From fiscal years 2013 through 2018, agencies completed 23 evaluations across the six sectors, which related to the 190 projects that agencies implemented in these sectors during this period. USAID completed 16 of these evaluations, with more than half of them in economic growth, although only 19 of the 116 projects USAID implemented in the sectors we reviewed related to economic growth. USDA completed six of these evaluations in agricultural development. State completed one evaluation in justice reform. DOD did not conduct any evaluations of its efforts to professionalize the military and develop defense capabilities in the Northern Triangle. In January 2017, DOD established agency-wide guidance for conducting assessment, monitoring, and evaluation of security cooperation programs and activities. We found that project implementers for State and USAID did not consistently collect key information to evaluate progress towards outcomes. Specifically, 12 of the 23 evaluations we reviewed from fiscal years 2013 through 2018 cited instances in which projects had not established measures or collected data to measure outcomes. Six of the 17 evaluations we reviewed for the sectors for economic growth and agricultural development noted that implementers had not collected sufficient data to measure the projects’ outcomes. For example, an evaluation of a USAID project that supported municipalities to mobilize financial resources for economic development noted that evaluators were unable to measure whether the project’s activities improved the municipalities’ competitiveness in providing services to businesses and investors. The evaluators could not perform this assessment because the project implementers did not consistently collect data to measure improvements in the local business climate. An evaluation of USAID projects in agricultural development in Guatemala noted that evaluators were unable to assess the total welfare impacts of the projects, such as changes in household incomes, because the projects had not collected information on household or farmer incomes from all sources with which to compare results following project activities. All four evaluations we reviewed in the sectors for good government service and justice reform noted that the projects did not sufficiently establish or measure the projects’ outcomes. For example, an evaluation of two USAID projects in Honduras for good government service found that one project did not incorporate indicators to measure outcomes. While the other project incorporated outcome indicators, the evaluation found most of these indicators to be poorly defined and inadequate to measure the project’s results. An evaluation of a State project in justice reform in Honduras also found that project indicators were focused on outputs and not outcomes. The evaluation also noted that the indicators were established after the project started and thus did not establish a true baseline or capture results from the beginning. As a result, evaluators reported that they lacked the data to evaluate key results. The two evaluations of projects to prevent community based violence we reviewed discussed deficiencies with progress indicators. For example, an evaluation of a project in Honduras that focused on reducing homicide rates noted that the implementing partner relied on the Honduran government to obtain data on homicides, although the government had limited capability to document and report such data. USAID officials noted that USAID and project implementers have made improvements to projects’ monitoring and evaluation plans in response to evaluation findings. For example, project implementers have added outcome indicators and USAID officials have provided technical assistance to implementers to help them design new methods for collecting data in response to evaluation findings and recommendations, according to USAID officials. Although our review of various performance-related documents related to the six sectors show that limited information from evaluations is available on progress toward prosperity, governance, and security, agency officials described some important results from U.S. assistance in the Northern Triangle related to these sectors. For example, USDA officials noted that technical assistance and training helped to enhance crop research and water and soil conservation, which contributed to increased agricultural production. USAID officials noted that the technical assistance the agency has provided to small and medium sized firms has helped them access markets and increase sales. State and USAID officials also described improvements in the use of forensic evidence through technical assistance and training provided to judges and prosecutors and enhanced court management, which contributed to timely criminal investigations and prosecutions. In addition, State officials explained that U.S. assistance along with support from other donors and host governments has contributed to positive results, including the passage of laws that prevent organized crime from donating to political campaigns, multiple anti-corruption investigations, as well as reductions in homicide rates through community based violence prevention projects. Furthermore, DOD officials noted that assistance in defense planning and management helped support oversight and accountability in the use of military funds and enhanced the capacity of security forces to respond to disaster relief and drug interdiction efforts. Agency officials also noted that from fiscal years 2013 through 2018 they achieved results toward enhanced prosperity, governance, and security for the 180 projects that corresponded to the 12 sectors outside of the scope of our review. In particular, USAID officials noted that environment sector projects increased incomes for thousands of individuals through improved management and conservation of natural resources, such as watershed management. State officials also described important results from projects in the human rights sector, including strengthening the capacity of labor union networks to monitor and document hundreds of incidents of violence against union activists in Guatemala and Honduras and increasing the number of investigations into such incidents. In addition, State officials identified results in the police reform sector, including passage of police reform legislation, professionalization of police academies, and sharing of information among law enforcement. Agency officials we interviewed also cited examples of challenges to achieving progress toward prosperity, governance, and security. For example, USDA and USAID officials noted that drought and coffee rust— a fungal disease that harms coffee plants—reduced agricultural production in affected areas. USAID officials also pointed out that the health of the economy and labor markets affect the results of economic growth projects, particularly with regard to firms’ sales and the placement of individuals in jobs following their completion of workforce development programs. In addition, State and USAID officials cited the importance of government officials’ willingness to implement reforms as an important factor that affects the achievement of results across sectors. Furthermore, high turnover of civil service and military professionals affects the achievement and sustainability of results in various sectors, according to State, USAID, and DOD officials. Agency officials also explained that they have taken steps to modify projects to address such challenges. For example, USAID and USDA projects have provided technical assistance and training to farmers on how to prevent coffee rust and cultivate coffee varietals resistant to the disease. In its coordinating role for the implementation of the Strategy, State has not created a comprehensive monitoring and evaluation plan that specifies an approach to evaluating progress across all agencies. Our prior work regarding effective foreign assistance strategies found that development of a monitoring and evaluation plan is a key element in terms of assessing agencies’ common goals and objectives, and mutually reinforcing results. Additionally, we found that foreign assistance involves the collaborative efforts of multiple agencies, and strategies that consistently address agencies’ roles and responsibilities and include interagency coordination mechanisms can guide effective collaboration among agencies and prevent fragmentation. In addition, Standards for Internal Control in the Federal Government indicates that managers should identify the information needed to achieve objectives and use such information to evaluate performance in achieving objectives. State, in coordination with USAID, has developed and updated a monitoring and evaluation plan for funds appropriated to them to implement the Strategy in response to direction contained in committee reports accompanying several State, Foreign Operations, and Related Programs appropriations acts. However, the plan that State and USAID developed for the Strategy, while consistent with the committee reports’ direction, is not comprehensive. In particular, it does not incorporate all the relevant agencies, sectors, and activities that support the Strategy’s objectives. The plan notes that State and USAID will monitor and evaluate foreign assistance supporting the Strategy. While the plan specifies that State and USAID should track completed, ongoing, and planned evaluations of their projects supporting the Strategy’s objectives, it does not include a plan for evaluations of projects conducted by agencies other than State or USAID, such as DOD and USDA. Additionally, the plan notes that each agency requires project monitoring, including progress indicators, baselines, targets, and expected outcomes of projects. The plan specifies that State will compile and report performance data, which will provide an important source of information to assess progress toward Strategy objectives. However, the plan does not specify how State and USAID would include reporting on many activities conducted by other agencies that support the Strategy’s objectives. As a result, State officials noted the monitoring and evaluation plan does not include indicators for DOD and USDA activities that contribute to the objectives of the Strategy, with the exception of DOD activities funded through State. For example, State, in addition to determining the scope of security assistance and funding level for each recipient of International Military Education and Training (IMET) programs, also identifies annual IMET goals and objectives for each country. DOD administers IMET in coordination with State. State and USAID’s monitoring and evaluation plan includes indicators to measure progress of these programs. DOD, however, conducts a number of other programs to professionalize the military that State and USAID have not included in the monitoring and evaluation plan. For example, DOD provides training to Northern Triangle militaries and Ministries of Defense that is outside of the IMET program, such as Defense Government Management and Training engagements. The Progress Report for the Strategy for fiscal year 2018 indicated that under the IMET program there were 13 U.S.-trained personnel in positions of prominence, or positions of military or government leadership, in the Northern Triangle in fiscal year 2017. DOD, though, in a separate report on these military training and education programs, noted there were over 100 U.S.-trained personnel in positions of prominence in the Northern Triangle in fiscal year 2017. In addition, the monitoring and evaluation plan does not include any of USDA’s activities or activities related to the health sector that support the Strategy’s objectives, despite the fact USDA completed six evaluations of its agricultural development projects that could be used to inform an understanding of progress toward the Strategy’s objectives. By not capturing information on DOD and USDA activities, State and USAID have limited ability to assess the progress made by all U.S. government agencies in the Northern Triangle. State officials stated that the monitoring and evaluation plan is not inclusive of DOD and USDA activities because the legislative direction for the plan did not require it. The Strategy, however, intends to take a comprehensive, integrated, and whole of government approach to engagement in Central America. DOD and USDA officials in headquarters and at the Missions in El Salvador, Guatemala, and Honduras told us that their activities also support the Strategy’s objectives. Given its coordinating role in the Strategy’s implementation and in foreign policy objectives in general, State is well positioned to work collaboratively with officials from other agencies to develop a comprehensive approach to monitoring the impact of all activities across all sectors that directly support the Strategy’s objectives. A comprehensive monitoring and evaluation plan that specifies an approach to evaluating progress across all agencies would help State and USAID to determine to what extent U.S. government activities in the Northern Triangle are achieving the Strategy’s desired results. The Northern Triangle, an area of strategic interest to the United States, faces high levels of poverty, weak governance, and widespread violence and insecurity. To respond to these challenges, the U.S. government has for many years provided assistance to the region. Multiple agencies have allocated billions of dollars to implement hundreds of projects that have provided technical assistance, equipment, and training to thousands of individuals and organizations. Agencies have reported mixed results from these projects, relative to targets set, yet little is known about progress on meeting broader objectives to improve prosperity, governance, and security in the region. Under the U.S. Strategy for Engagement in Central America, State and USAID developed a monitoring and evaluation plan, for their own projects, that is an important tool for assessing impact in the region. A more comprehensive approach to monitoring and evaluation of projects that may address the Strategy’s objectives to include all relevant agencies, sectors, and activities would enable the U.S. government to have a better understanding of progress under the Strategy and how U.S. assistance is addressing the underlying challenges that confront El Salvador, Guatemala, and Honduras. Given State’s coordinating role in the implementation of the Strategy among U.S. government agencies, including DOD and USDA, it is uniquely positioned to ensure that agencies collaborate effectively and that monitoring and evaluation are well coordinated and documented in a comprehensive plan. The Secretary of State, working with the Administrator of the U.S. Agency for International Development, should collaborate with the Departments of Defense and Agriculture and other Departments as necessary, to develop a comprehensive approach to the monitoring and evaluation of projects that directly support the objectives of prosperity, governance, and security, and incorporate this approach into the Strategy monitoring and evaluation plan. We provided a draft of this report to State, USAID, DOD, USDA, DOJ, and DHS. We received written comments from State, USAID, and DOD, which we reprinted in appendixes V through VII. We received technical comments from State, USAID, DOD, and DHS, which we incorporated as appropriate. USDA and DOJ informed us in writing that they had no comments. State and USAID did not concur with our recommendations, indicating that neither agency has the authority to direct DOD or USDA to design and implement programs. USAID indicated that while greater interagency coordination would be appropriate, it does not have the authority to direct DOD or USDA to monitor and evaluate their projects against objectives developed for the Strategy. DOD noted that while some of its programs enable progress toward the Strategy’s objectives, it is not appropriate for State to specify how to monitor and evaluate DOD-funded programs. State also asserted that our recommendation is not consistent with the explanatory statements accompanying the Department of State, Foreign Operations, and Related Programs Appropriations Act, which directs State and USAID to develop a monitoring and evaluation plan for the Strategy for programs funded by appropriations to them, but does not direct that the plan include monitoring and evaluation of programs funded by appropriations to DOD and USDA. We are not recommending that State and USAID direct DOD and USDA to monitor and evaluate projects, but rather that State collaborate with DOD and USDA to develop a more comprehensive approach to monitoring and evaluating projects that support the Strategy’s objectives and that State document the results of this collaboration in the Strategy’s monitoring and evaluation plan. We do not prescribe the format or content for how the Strategy’s monitoring and evaluation plan might be updated. We have modified relevant sections of our report and our recommendation to make this clearer and eliminated the recommendation to USAID, since State coordinates implementation of the Strategy by the various agencies of the U.S. government. We found that DOD and USDA have designed and implemented programs that directly support the objectives of the Strategy. While we acknowledge that some coordination among agencies occurs in Washington and in the Northern Triangle, we found that such coordination does not formally extend to monitoring and evaluation. We agree with USAID’s comment that interagency coordination on a comprehensive monitoring and evaluation plan for the Strategy would be appropriate. Consistent with USAID’s comment, we believe that our recommendation encourages greater coordination among agencies, including DOD and USDA, by ensuring that comprehensive monitoring and evaluation efforts of the entire U.S. government are in sync with the monitoring and evaluation plan for the Strategy. Excluding DOD and USDA projects from the monitoring and evaluation plan for the Strategy could result in an incomplete or unclear understanding of the results of U.S. assistance in the Northern Triangle. Without a complete and clear understanding of the results across all agencies involved, agencies may miss important lessons about the types of assistance that are most effective in achieving U.S. objectives in this region, potentially limiting overall progress. Furthermore, while the explanatory statement accompanying Pub. L. No. 114-113 directs State, in coordination with USAID, to develop a monitoring and evaluation plan for funds appropriated to them, we are recommending that State, as coordinator for the implementation of the Strategy, work with the other agencies to develop a more comprehensive approach to monitoring and evaluating projects that support the Strategy’s objectives. State should update the monitoring and evaluation plan that was created in response to the congressional direction to document the comprehensive approach to monitoring and evaluation. State indicated that the credibility of our report was limited by the following five methodological issues: (1) our inclusion of projects implemented by DOD and USDA; (2) our inclusion of projects implemented with funds appropriated prior to fiscal year 2016; (3) our use of inconsistent reporting methods for funding allocations among the four State bureaus providing data and among State, USAID, DOD, and USDA; (4) our classification of program sectors, which was not consistent with the sub-objectives used by State and USAID as part of the Strategy; and (5) our exclusion of several “primary” sectors for our in-depth review, such as police professionalization, reducing violence at the local level, and reducing the influence of organized crime and gangs. We believe that our methodology enhanced the credibility and reliability of our report. Overall, we designed our objectives, scope, and methodology, as outlined in detail in appendix I, to provide a reasonably comprehensive review of the results of U.S. assistance to the Northern Triangle toward achieving key U.S. objectives. First, we chose to review all agencies that have allocated a significant amount of funding from their appropriations to implement projects in the Northern Triangle from fiscal year 2013 through fiscal year 2018 to support prosperity, governance, and security. DOD and USDA officials confirmed that DOD and USDA projects support these objectives and we believe that the inclusion of these agencies significantly enhanced the accuracy and completeness of our reporting on the results that have been achieved from U.S. assistance as well as the gaps in the current monitoring and evaluation approach and implications for State’s ability to assess results comprehensively. Second, we believe our inclusion of projects implemented from fiscal years 2013 through 2018 provided a reasonable time frame for our review because it included projects that supported the objectives of improving prosperity, governance, and security— long standing objectives that predated appropriations for the Strategy, and even the Strategy itself. Including projects implemented between fiscal years 2013 and 2018 increased our ability to report on the results of agencies’ projects and their overall progress toward the Strategy’s objectives because projects funded since fiscal year 2016 were in too early a stage of implementation to report meaningfully on such results. However, we considered, as appropriate, any results information we were able to obtain on such projects. Third, we acknowledge that the precision of our estimates for reporting on funding allocations was limited due to the inconsistent nature of reporting of financial data by different bureaus and agencies. However, taking into consideration qualifications noted throughout our report, we believe that our reporting of funding allocations provides a reliable description of how agencies used allocated funding to support prosperity, governance, and security objectives. Fourth, we believe that our classification of projects under different sectors we identified provides a detailed, comprehensive, and meaningful analysis of projects and related results. Because some of the sub-objectives developed by State and USAID, such as “reduce poverty,” were very broad and did not lend themselves to an analysis of specific project sectors that supported the Strategy’s objectives, we identified more specific sectors, including health, economic growth, and agricultural development. State and USAID officials validated the accuracy of our definitions, and we revised them as appropriate, given input from agency officials. Fifth, our selection of six sectors for in-depth review of projects and results limits the generalizability of our findings to all sectors, which we note. Due to the large number of projects, sectors, and sub-objectives associated with U.S. assistance to the Northern Triangle, we determined that a case study approach was the most effective methodology for our review. We devised selection criteria to reflect a meaningful selection of projects across sectors, agencies, and countries. Moreover, two of the sectors we selected for in-depth review—community based violence prevention, and justice reform—encompass several projects classified as relating to “reducing violence at the local level,” and “reducing the influence of organized crime and gangs.” Thus our report addresses results in these sectors. We omitted certain sectors, such as police professionalization, in part, because we had ongoing work related to this sector. We acknowledge limitations with this case study approach and do not attempt to generalize results beyond the sectors we reviewed. We believe that this methodological approach provides a reasonable basis for our overall conclusion that projects in the sectors we reviewed achieved mixed results. USAID also raised several methodological concerns, some of which were similar to those raised by State. In particular, USAID (1) questioned the validity of our analysis, since it was based on a case study of six of the 18 sectors we identified, and commented that we did not discuss the limitation of this approach; (2) questioned the validity of our use of monitoring information relating to the achievement of annual targets to analyze results; and (3) asserted that we focused on negative evaluation findings to assess results and did not mention or analyze planned and ongoing evaluations or programmatic changes made in response to monitoring and evaluation information. We believe our methodological approach provides a reliable basis for our findings and conclusions, and concerns USAID raised do not limit the credibility of our report. First, we acknowledge the limitations of our case study approach and included statements throughout our report to make these limitations clear. Second, we believe that the use of data on the achievement of annual targets is a valid approach to assessing results, although the agencies collecting the data may also intend to use it in making decisions about ongoing projects. Furthermore, these data provided only one element of our analysis. We also analyzed State and USAID implementer progress reports, mid-point and final evaluations, and other performance reports, which provide a longer-term perspective on results. Collectively, we believe that this information provides meaningful insight into the successes and shortcomings of the projects in the sectors we reviewed. Third, we sought to present a balanced picture of results within the sectors we reviewed, highlighting both positive and negative outcomes described in the reviewed documents. We reviewed completed evaluations to provide insight into project results, but excluded ongoing and planned evaluations because conclusions about project results are not available until such evaluations are completed. Similarly, our report acknowledges that agency officials described progress and challenges to achieving the prosperity, governance, and security objectives, as well as the steps taken to modify projects to address such challenges. However, such modifications fell outside the scope of our analysis of results, absent documentation of their specific impact on the achievement of objectives. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of the Department of State, the Administrator of the U.S. Agency for International Development, the Secretary of the Department of Defense, the Secretary of the Department of Agriculture, the Secretary of the Department of Homeland Security, the Attorney General, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7141 or groverj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. This report examines (1) the projects that the U.S. government has implemented in the Northern Triangle from fiscal year 2013 through fiscal year 2018 to support prosperity, governance, security, (2) what is known about the results of these projects, and (3) what is known about progress toward the U.S. Strategy for Central America’s (Strategy) objectives. To determine the projects that the U.S. government has implemented in the Northern Triangle, we collected and analyzed agency project and funding data concerning foreign assistance projects supporting prosperity, governance, and security objectives from the U .S. Agency for International Development (USAID), and the Departments of State (State), Justice (DOJ), Homeland Security (DHS), Defense (DOD), and Agriculture (USDA). We focused our analysis on State, USAID, DOD, and USDA because they allocated the largest amounts of funding for the largest number of projects in the Northern Triangle from fiscal years 2013 through 2018. We included projects from fiscal years 2013 to ensure we examined projects that had undergone sufficient implementation to assess results. We obtained the data and information from several bureaus at State that administer these projects and funds: International Narcotics and Law Enforcement Affairs; Western Hemisphere Affairs; Political-Military Affairs; and Democracy, Human Rights, and Labor. We also obtained data from DHS and DOJ concerning projects implemented through agreements with State, which we included under State’s project and funding counts. Although agencies use different terms to describe agencies’ assistance, including programs, projects, and activities, we use the term “projects” to refer to assistance funded by the key agencies that are implemented directly by the agencies or through awards made to the implementing partners. In general, the term project consists of a set of activities that are designed and executed over a time frame to achieve a specific aim. While agencies and bureaus typically provided us with project-level data, some agencies and bureaus were unable to report data at the project level, and instead provided us with data that combined multiple activities or awards to implementing partners to accomplish a broader aim. In addition, most agencies reported project and funding data by country, including separating funding data for multi-country projects that were implemented in two or more countries, including at least one Northern Triangle country. Some agencies were not able to report multi-country projects by country, which we included in the multi-country project category. Since most agencies and bureaus provided us with project-level data separated by country, we use the term “projects” to encompass all available data on agencies’ assistance in each of the three countries. We analyzed agencies’ data and information to identify the number of projects implemented by agency and country and the total funding agencies allocated for these projects from fiscal years 2013 through 2018. We excluded from our analysis those projects that encompassed solely administrative and monitoring and evaluation activities and costs that did not provide technical assistance, although we included the funds allocated for these projects in our analysis of funds allocated by each agency for projects that supported prosperity, governance, and security. We assessed the reliability of the data that agencies reported for these projects. We requested and reviewed information from agency officials regarding the underlying data systems and the checks and reviews used to generate the data and ensure its accuracy and reliability. We also conducted logical checks and analysis to confirm the accuracy of the data. When we found potential duplicate data and discrepancies, we contacted relevant agency officials in Washington, D.C. and obtained information from them necessary to resolve these data issues. As a result of these steps, we determined that the data were sufficiently reliable for the purposes of reporting the number of projects that supported prosperity, governance, and security in El Salvador, Guatemala, and Honduras and funding allocations for these projects from fiscal years 2013 through 2018. To select a subset of the projects to review, we reviewed agencies’ project information as well as Strategy documents to categorize all projects into 18 different sectors of assistance that generally aligned with the current objectives of the Strategy. Specifically, we grouped similar projects by sector such as economic growth, justice reform, and community based violence prevention, and aligned them according to the Strategy’s three objectives of prosperity, governance, and security. We requested that officials from State, USAID, DOD, and USDA review our analysis to confirm our alignment of projects to the sectors and the three objectives. We incorporated revisions from agency officials as appropriate. We then selected a judgmental, nongeneralizable sample of six of the 18 sectors for an in-depth review of performance-related documentation for projects supporting each of the objectives. The six sectors selected included agricultural development, economic growth, good government service, justice reform, community based violence prevention, and professionalize the military and develop defense capabilities. We selected these six sectors to achieve variation by agency, funding allocation amount, country, and to include projects supporting each of the three objectives. Specifically, we selected the six sectors to include two sectors supporting each objective, a distribution of projects across the three Northern Triangle countries, and the largest amounts of allocated funding and number of projects. We excluded from our sample selection the migration and police reform sectors because of our ongoing work in those sectors concerning the Northern Triangle. To determine what is known about project results, we reviewed agency performance-related documents corresponding to the 190 projects implemented from fiscal years 2013 through 2018 in the six sectors we reviewed. Specifically, we examined State and USAID’s Performance Plans and Reports for El Salvador, Guatemala, and Honduras for each of fiscal years 2013 through 2018; State’s International Narcotics Control Strategy Reports for fiscal years 2013 through 2018; State and USAID’s Progress Report for the U.S. Strategy for Central America’s Plan for Monitoring and Evaluation for fiscal years 2018 and 2019; and State’s quarterly country cables reporting on agencies’ progress in implementing projects in support of prosperity, governance, and security objectives in each of the Northern Triangle countries for the available quarters of fiscal years 2016 through 2018. We also requested and reviewed all 23 evaluations completed from fiscal years 2013 through 2018 by State, USAID, and USDA related to the six selected sectors in each Northern Triangle country. In addition, we selected a nongeneralizable sample of 19 projects within the six selected sectors to gain more in-depth information and context about project implementation and results. For the nongeneralizable sample of projects, we reviewed performance-related documentation, including, among other things, implementing partners’ quarterly, semi-annual, and annual progress reports, to examine project results. We selected the19 projects based on a variety of criteria, including the types of project activities and the objectives they supported, as well as to obtain a range of funding allocation amounts, countries, and agencies. We excluded from our sample selection those projects that encompassed solely administrative and monitoring and evaluation activities and costs, and those that agencies reported as pilot projects not yet implemented. To examine what is known about progress toward the Strategy’s objectives, we reviewed Strategy documents, including monitoring and evaluation plans, to assess if they included key elements of effective strategies that we have identified as related to assessment of progress toward strategic goals. We developed these elements on the basis of prior work related to U.S. government strategies and interagency collaboration as well as prior work on addressing fragmentation, overlap, and duplication in the federal government. Our prior work suggests that strategic documents offer an opportunity to consider the roles and responsibilities of various stakeholders involved in achieving those goals, and information on how progress toward those goals will be measured. The Strategy documents were reviewed and rated by two analysts to determine the extent the planning and reporting procedures aligned with the key elements for foreign assistance strategies in situations where multiple agencies work together to deliver foreign assistance. These elements related to (1) delineation of agencies’ roles and responsibilities and coordination mechanisms; and (2) assessment of progress toward strategic goals, including identifying activities to achieve results, performance indicators, and monitoring and evaluation plans. Additionally, in assessing the monitoring and evaluation plan, we considered the Standards for Internal Control in the Federal Government, which specify that managers should identify the information needed to achieve objectives and use such information to evaluate performance in achieving objectives. To determine State and USAID’s rationale for not including other agencies’ activities that support the objectives of the Strategy, we met with State and USAID officials in Washington, D.C. We also reviewed relevant Strategy documents and Congressional legislation, particularly Public Law 115-31, 131, the Consolidated Appropriations Act, 2017, which State and USAID cited as the basis for the creation of the Strategy’s results architecture and monitoring and evaluation plan. To support our work on all three objectives, we conducted fieldwork in El Salvador, Guatemala, and Honduras. During the fieldwork, we observed selected project activities, and interviewed agency officials, implementing partners, and project beneficiaries about the project activities and results, and factors that affected project results. We also interviewed agency officials in Washington, D.C. from relevant State bureaus, USAID, DOD, and USDA Foreign Agricultural Service as well as officials of the U.S. Southern Command in Doral, Florida about project activities, project results, factors affecting results and actions to address these factors, as well as efforts to monitor and evaluate project results. We conducted this performance audit from December 2017 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate, evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix provides a summary of information on U.S. Agency for International Development (USAID), and the Departments of State (State), Defense (DOD), and Agriculture (USDA) assistance projects in the three Northern Triangle countries—El Salvador, Guatemala, and Honduras—to support the prosperity, governance, and security objectives of the U.S. Strategy for Central America (Strategy) from fiscal years 2013 through 2018. We provide a summary of information for the following sectors we selected by country, agency, funding allocation amount, and objective of the Strategy. The sectors include economic growth, agricultural development, good government service, justice reform, community based violence prevention, and professionalize the military and develop defense capabilities. For each sector, we provide an overview and examples of projects, including project objectives, activities, and results that State, USAID, DOD, and USDA reported toward improving prosperity, governance, and security in the Northern Triangle. The information about each sector also includes the following data, selected to illustrate the scope of U.S. assistance in each sector and the underlying conditions that impact prosperity, governance, and security in the Northern Triangle: Total number of projects: The total number of projects we identified that supported each sector in each country from fiscal years 2013 through 2018. Approximate Reported Funding: An estimate of the total allocated funding reported for the projects in each sector. Context Indicators: Data reported from various organizations relevant to each of the sectors, including the World Bank, and reported in State and USAID’s Progress Report for the Strategy for fiscal years 2018 and 2019. We did not independently verify these reported data. Economic growth projects are intended to assist populations living below the poverty line meet basic needs, help businesses improve their business practices and access markets and investment, and promote workforce development. USAID and State implemented 26 economic growth projects in the Northern Triangle from fiscal years 2013 through 2018. industries and higher education institutions to develop educational programs and research. The project trained 100 researchers at universities on how to complete applied research studies on the economy. This training, along with 26 applied research studies funded by the project, allowed for collaborative research between academia and the private sector that had not previously existed in El Salvador. The project also upgraded or created 28 new degree programs to align with industry demands. The project awarded 900 scholarships to students enrolled in these degree programs. A USAID project in Honduras provided assistance to rural micro- enterprises to improve their access to markets and competitiveness. The project helped 2,270 of these enterprises adopt new inputs, technologies, and practices for a range of entrepreneurial activities, such as installing solar panels and cultivating organically grown coffee, according to an evaluation. It also helped micro-enterprises achieve certifications from trade and business associations to help them access new markets with higher quality standards to obtain better prices for products, such as high quality chocolate. A USAID project in El Salvador encouraged public-private partnerships and provided funds to help municipalities mobilize financial resources for improving economic development. It also intended to help municipalities streamline their administrative procedures to improve the local business climate. The streamlined procedures reduced the time required to complete business processes and diminished the chances for bribery and other illegal practices, according to an evaluation. A USAID project in El Salvador targeted over 10,000 micro- enterprises and 20 local governments to strengthen the capacity of providers of business development services to help these micro- enterprises improve innovation and technology, access financing, and increase exports. According to an implementer progress report, the project provided trade capacity building assistance to at least 369 micro-enterprises to help them export. It also trained at least 491 entrepreneurs and 14 business consultants to develop export opportunities. A USAID project in El Salvador offered assistance to help workers obtain employment. It provided training to more than 5,600 individuals, including at-risk youth and disabled persons, to improve their job placement opportunities, according to an evaluation. The project also placed 4,886 participants in new or improved jobs. The evaluation also noted that the firms participating in the project reported that the project’s methods reduced their recruiting and hiring costs and risks and contributed to a decrease in employee turnover. A USAID project in Honduras installed irrigation systems to grow lettuce and other crops. Agricultural development projects are intended to assist farmers to increase the quantity and quality of crops through training, research, and better access to capital. They also sought to assist farmers to gain access to markets and address food security. USAID and USDA implemented 40 agricultural development projects in the Northern Triangle from fiscal years 2013 through 2018. Rural Population (Approximate percent of total population, 2017) Rural Poverty (Approximate percent of rural population, 2014) productivity and expand trade. The project provided training to more than 500 individuals, approximately 99 percent of whom reported using the lessons they learned to improve their farm management practices, according to an evaluation. In addition, approximately 97 percent of them reported that they made business decisions based on economic considerations or analysis following the training. The project provided 35,215 microfinance loans, valued at approximately $37.5 million. Approximately 82 percent of the beneficiaries reported an increase in agricultural production and approximately 88 percent reported an increase in business sales because of the loans, according to an evaluation. Although the evaluation noted that the loans had the potential to expand agricultural trade, the effects were mixed. A USDA project in Guatemala that provided school meals doubled the number of schools that reported having access to food in six municipalities and provided more than 40,400 school-age children with daily meals, according to an evaluation. The evaluation also reported that the reduction in hunger from the project contributed to a decline in absentee rates for students at the participating schools, from 20 percent before its implementation to 5 percent. The project also constructed or rehabilitated kitchens at 106 schools and provided utensils and equipment for preparing food. A USDA school feeding project in Honduras provided meals to more than 50,000 children in 1,047 schools. The project also conducted education campaigns using local media to inform the population about the importance of education and the steps for enrolling children in school. Following the project’s implementation, school attendance for boys increased by approximately 6 percent and for girls by approximately 2 percent, according to an evaluation. USAID projects in Guatemala that aimed to help small farmers improve their farming practices and gain access to markets had mixed results. For example, the evaluation noted that per capita incomes or household incomes of municipalities included in the projects fared worse than municipalities that were not. However, municipalities included in the projects fared better in access to electricity and rates of home ownership. Municipal watershed reforestation project in Guatemala supported by USAID. Good government service projects are intended to increase the effectiveness, efficiency, accountability, and transparency of government services and institutions. They do so by providing training and technical assistance to improve revenue collection and management, promote transparency and citizen oversight, and enhance the quality of government services. USAID funded 29 good government service projects from fiscal years 2013 through 2018. Government effectiveness (est.): (Percentage points changed 2013 to 2017) government services to better respond to citizen needs. An evaluation noted that the project helped draft stronger decentralization laws, but these were not passed due to lack of political will. The evaluation also reported the project provided technical assistance and training to municipal governments on revenue collection, fiscal management, and financial software systems intended to help raise revenue. However, the evaluation also found that 39 percent of municipalities reported decreases in fiscal autonomy. The evaluation also cited resource constraints, data inconsistencies in income records, and concerns about the sustainability of the training. A USAID project in Guatemala sought to strengthen select municipalities to better manage public resources and deliver services in a efficient and transparent manner in order to foster development. According to the project’s 2017 annual report, 76 percent of the target municipalities increased their average monthly revenues by 19 percent following finance management trainings. A USAID project in El Salvador aimed to improve government transparency and accountability. It did so by supporting citizen oversight and government compliance with regulations and standards related to transparency, professionalism, and ethics. According to a 2018 implementor monitoring report, the project met a majority of its expected performance goals. In addition, 11 of the targeted municipalities noted in their self-assessments an increased capacity to provide access to information and promote ethics in their institutions. US-provided forensic equipment at a criminal forensic lab in Honduras. Justice reform projects are intended to provide training, equipment, and technical assistance to the justice system to decrease impunity, combat corruption, improve prosecution and forensic capacities, and increase the efficiency and management of courts. USAID and State implemented 42 projects in justice reform from fiscal years 2013 through 2018. Percentage of the Population with Trust in the Courts: (Percentage points change, 2014 and 2018) transparency, accountability, and ethics, and increase civil society participation in government through technical assistance and training. An evaluation found that the project increased awareness of these topics, and led to some improvements in laws and regulations, such as improving the legal framework for anti-corruption efforts. However, the project was unable to achieve any significant changes intended due to lack of political will. A State project in Honduras implemented activities that sought to reduce violence and homicide by increasing access to justice, strengthening institutions and local organizations’ capacity to deliver legal and support services for victims of violence and rehabilitation and reintegration services for prisoners. A mid-term evaluation found that the project successfully convened stakeholders to discuss women and children’s access to justice and carried out a campaign to disseminate information on human rights and access to justice. The evaluation also found the project helped maintain, but not increase rehabilitation and reintegration services for prisoners. El Salvadoran police meeting with youth in a police athletic league. Community based violence prevention (CBVP) projects are intended to reduce the levels of crime and violence, including addressing some of the root causes of insecurity. USAID and State implemented 31 CBVP projects from fiscal years 2013 through 2018. These projects sought to support anti-gang education, employment opportunities for at-risk youth, and efforts to increase institutional capacity and citizen responsibility for crime prevention in municipalities plagued by violence. aimed at increasing access to comprehensive, long-term social, education, and health services for high-risk populations. As part of these efforts, 242,029 individuals participated in U.S. government- funded gang prevention and education programs in Honduras in fiscal year 2017. USAID projects in Honduras worked with civil society organizations to provide violence prevention services with a focus on vulnerable populations. In fiscal year 2018, USAID reported that 202 people received U.S. government-funded gender-based violence services, including health, legal, and counseling services. Percentage of the Population Who Feel Safe Walking in their Neighborhood at Night: (Difference in percentage points, 2014 and 2017) A USAID project in Honduras sought to lower rates of homicide and other violent crime through alliances of communities and government institutions, especially the police. A mid-term evaluation of the project reported significant decreases in homicide rates, ranging from 42 percent to 68 percent, in three of the six communities where USAID targeted its assistance. A USAID project aimed to improve educational options for out-of- school youth by offering them alternatives to criminal and gang activity. An evaluation of the project reported that more than 90 percent of the more than 15,000 individuals who enrolled in school did not pass exams to demonstrate competency at the end of courses. The evaluation further noted that 30 percent of the youth did not remain in school, which likely resulted in a small fraction of them meeting the goal of increasing their income. Honduran Special Forces demonstrate U.S. training. Projects to professionalize the military are intended to increase the acountability, competency, and capabilities of militaries in the Northern Triangle. DOD and State implemented a number of these activities from fiscal years 2013 through 2018. The projects provided military equipment and training to military personnel and technical assistance to Ministry of Defense personnel. Policy and a budgeting system for its Ministry of Defense that supports transparency and accountability. Total Number of US Trained Personnel at National Leadership Levels: (Fiscal year 2018) An After Action Report of a DOD Defense Governance workshop in Guatemala noted that DOD continued to support the Guatemalan Ministry of Defense to identify national policy and strategy priorities, determine capabilities, and develop a data-driven approach to problem solving and making decisions on resources. A DOD report noted that DOD training in El Salvador that focused on fighting corruption had improved relations between military and civilian institutions. Andrade Costa, Melissa, and Irene García Palud, Evaluation Report: Mid- term Evaluation of the Program, “Reducing Violence and Homicide Through Access to Justice in Chamalecón, Satelite, and Rivera Hernández Neighborhoods of San Pedro Sula, Honduras”, August 2018. USAID/El Salvador Monitoring, Evaluation and Learning Initiative. Final Performance Evaluation of the Higher Education for Economic Growth Activity, May 17, 2018. DevTech Systems, Inc. Programa de Monitoreo y Evaluación: Evaluación final del Poyecto Cadenas de Valor Rurales (PCVR), August 2017. Mendéz England and Associates, Evaluación de Desempeño de Medio Término de la Actividad de Educación para la Niñez y Joventud 2011- 2017, August 2017. Management Systems International, A Tetra Tech Company. Performance Evaluation of the Partnership for Growth in El Salvador, March 20, 2017 (Revised July 24, 2017). Asociación de Desarrollo Organizacional Comunitaria (ADOC). Mid-term Evaluation of the Investment for Educational Development of the Highlands (IDEA) Project, Save the Children/USDA, 2016. Advisem Services, Inc. Final Evaluation Report: Final Evaluation of FINCA’s Food for Progress (FFPr) in El Salvador, November 30, 2016. The Cadmus Group, Inc. Performance Evaluation of USAID/Honduras Proparque Program, June 2016. Boston College School of Social Work, Final Evaluation Report: Food for Education (FFE) Project – USDA Catholic Relief Services (CRS) Honduras, April 2016. Khanti, S.A. Project Concern International, Food for Education II, Mid- term Evaluation Final Report, December 2015. Boston College School of Social Work. Mid-term Evaluation Report: Food for Education “Learning for Life” Guatemala, October 2015. Social Impact, Inc. Honduras Convive! Mid-term Evaluation Report, July 10, 2015. DevTech Systems, Inc. Final Evaluation of the USAID/Alianzas Project, December 12, 2014. Optimal Solutions Group, LLC. Partnership for Growth: El Salvador– United States (2011-2015), Mid-term Evaluation Final Report, September 30, 2014. DevTech Systems, Inc. Informe final Evaluación del Proyecto Apoyo en Políticas y Regulaciones para el Crecimiento Económico de Guatemala (PRS), September 20, 2014. Optimal Solutions Group, LLC. Final Report: Does Assistance to Farmers Translate into Community Welfare Improvements? Non-Experimental Program Evaluation of USAID Assistance to Smallholder Farmers in Guatemala, August 18, 2014. Notre Dame Initiative for Global Development. Food for Education Mid- term Evaluation, July 2014. Democracy International, Inc. Final Report: Mid-term Performance Evaluation of the Transparent Local Governance and Improved Service Delivery Project (USAID/NEXOS) and the Decentralized Enabling Environment Project (USAID/DEE), May 2014. International Business and Technical Consultants, Inc. Evaluation Report: Final Performance Evaluation of the USAID Municipal Competitiveness Project in El Salvador, January 29, 2014. Development Training Services, Inc. Report on the Mid-term Performance Evaluation of the USAID Transparency and Governance Project El Salvador, December 24, 2012. Rivera Cira Consulting, Inc. USAID/Guatemala Final Performance Evaluation for the Project Against Violence and Impunity (PAVI), December 20, 2012. Amex International and DevTech Systems, Inc. USAID/Guatemala Mid- term Performance Evaluations for Two Economic Growth Office Projects, October 25, 2012. International Business and Technical Consultants, Inc. Performance Evaluation of the “Improving Access to Employment Program in El Salvador”. October 17, 2012. The Department of State (State) and the U.S. Agency for International Development (USAID) produced the results architecture for the U.S. Strategy for Central America (Strategy). The results architecture presents the desired end-state of the Strategy; the three primary objectives of prosperity, governance, and security; and sub-objectives that support each of the primary objectives. State and USAID defined the Strategy’s mission as to secure U.S. borders and protect U.S. citizens by addressing the economic, governance, and security drivers of illegal immigration and illicit trafficking, and to promote private sector investment in Central America. The result architecture’s overall objective is an economically integrated Central America that is fully democratic; provides economic opportunities to its people; enjoys more accountable, transparent, and effective public institutions; and ensures a safe environment for its citizens. The Strategy’s prosperity objective is to work with Central American governments to improve the business environment, create jobs, enhance food security, expand energy security, and increase U.S. investment and trade. The Strategy’s governance objective focuses on reducing impunity and corruption through the creation of more transparent, efficient governments that deliver services, including justice, effectively. The Strategy’s security objective includes enhancing citizen security, re-establishing state presence and security in communities at risk, scaling up violence prevention and law enforcement activities in communities, and targeting individuals most susceptible to gang recruitment. Figure 10 depicts the overall summary of the Strategy’s results architecture, which focuses on the objectives of prosperity, governance, and security. 1. We are not recommending that State direct DOD and USDA to monitor and evaluate projects, but rather that State collaborate with DOD and USDA to develop a more comprehensive approach to monitoring and evaluating projects that support the Strategy’s objectives and that State document the results of this collaboration in the Strategy’s monitoring and evaluation plan. We do not prescribe the format or content for how the Strategy’s monitoring and evaluation plan might be updated. We have modified relevant sections of our report and our recommendation to make this clearer and directed the recommendation to the Secretary of State, since State coordinates implementation of the Strategy by the various agencies of the U.S. government. We found that DOD and USDA have designed and implemented programs that directly support the objectives of the Strategy. While we acknowledge that some coordination among agencies occurs in Washington and in the Northern Triangle, we found that such coordination does not formally extend to monitoring and evaluation. We believe that our recommendation encourages greater coordination among agencies, including DOD and USDA, by ensuring that monitoring and evaluation efforts by U.S. government agencies are in sync with the monitoring and evaluation plan for the Strategy. Excluding DOD and USDA projects from the monitoring and evaluation plan for the Strategy will continue to result in an incomplete or unclear understanding of the results of U.S. assistance in the Northern Triangle. Without a complete and clear understanding of the results across all agencies involved, agencies may miss important lessons about the types of assistance that are effective in achieving U.S. objectives in the region, potentially limiting overall progress. 2. While the explanatory statement accompanying Pub. L. No. 114-113 directs State, in coordination with USAID, to develop a monitoring and evaluation plan for funds appropriated to them, we are recommending that State, as coordinator for the implementation of the Strategy, work with the other agencies to develop a more comprehensive approach to monitoring and evaluating projects that support the Strategy’s objectives, and that they utilize the monitoring and evaluation plan that they have already created in response to the congressional direction as a place to document the comprehensive approach to monitoring and evaluation. 3. We chose to review all agencies that have allocated a significant amount of funding from their appropriations to implement projects in support of prosperity, governance, and security objectives in the Northern Triangle. State, USAID, DOD, and USDA officials confirmed that DOD and USDA projects support the objectives of the Strategy, and we believe that the inclusion of these agencies enhanced the accuracy and completeness of our reporting on the results that have been achieved from U.S. assistance as well as the gaps in the current monitoring and evaluation approach and implications for State’s ability to assess results comprehensively. 4. We believe our inclusion of projects implemented from fiscal years 2013 through 2018 provided a reasonable time frame for our review because it includes projects that supported the objectives of improving prosperity, governance, and security—long standing objectives of U.S. assistance to the Northern Triangle that predated appropriations for the Strategy, and even the Strategy itself. Including projects implemented between fiscal years 2013 and 2018 increased our ability to report on the results of agencies’ projects and their overall progress toward the Strategy’s objectives because projects funded since fiscal year 2016 were in too early a stage of implementation to report meaningfully on such results. However, we considered, as appropriate, any results information we were able to obtain on such projects. 5. We acknowledge that the precision of our estimates for reporting on funding allocations was limited due to the inconsistent nature of reporting of financial data by different bureaus and agencies. However, taking into consideration qualifications noted throughout our report, we believe that our reporting of funding allocations provides a reliable description of how agencies used allocated funding from fiscal years 2013 through 2018 to support prosperity, governance, and security objectives in the Northern Triangle. 6. We believe that our classification of projects under the different sectors we identified enabled us to provide a more detailed, comprehensive, and meaningful analysis of projects and related results. Because some of the sub-objectives that State and USAID developed, such as “reduce poverty,” were very broad and did not lend themselves to an analysis of specific project sectors that supported the Strategy’s objectives, we identified more specific sectors, including health, economic growth, and agricultural development. State and USAID officials validated the accuracy of our definitions, and we revised them as appropriate, given input from agency officials. 7. Our selection of six sectors for in-depth review of projects and results limits the generalizability of our findings to all sectors, which we note. Due to the large number of projects, sectors, and sub-objectives associated with U.S. assistance to the Northern Triangle, we determined that a case study approach was the most effective methodology for our review. We devised selection criteria for our case study to reflect a meaningful selection of projects supporting each of the three objectives across a range of sectors, agencies, and countries. Moreover, two of the sectors we selected for in-depth review—community based violence prevention and justice reform— encompass several projects classified as relating to “reducing violence at the local level,” and “reducing the influence of organized crime and gangs.” Thus our report addresses results in these sectors. We omitted projects relating to police professionalization, in part, because we had ongoing work related to this sector. We acknowledge limitations with this case study approach and do not attempt to generalize results beyond the sectors we reviewed, but we believe our methodological approach provided a reasonable basis for our overall conclusions. 1. We eliminated the recommendation to USAID because State plays a coordinating role in the Strategy’s implementation and is well positioned to work collaboratively with officials of other agencies, including DOD and USDA. We believe our recommendation to State, in which we recommend that they work with USAID, encourages greater coordination among agencies, including DOD and USDA, to ensure that their efforts are included in a comprehensive monitoring and evaluation plan for the Strategy. 2. We believe our inclusion of projects implemented from fiscal years 2013 through 2018 provided a reasonable time frame for our review because it included projects agencies implemented to support the long standing objectives of prosperity, governance, and security in the Northern Triangle—objectives that the U.S. government has supported under various initiatives that predated the Strategy and appropriations for the Strategy. Furthermore, including projects implemented between fiscal years 2013 and 2018 increased our ability to report on the results of agencies’ projects and their overall progress toward prosperity, governance, and security because projects funded since fiscal year 2016 were in too early a stage of implementation to report meaningfully on results. However, we considered, as appropriate, any results information we were able to obtain on such projects. 3. We requested and reviewed all USAID evaluations completed during the time frame for our review—from fiscal years 2013 through 2018 or October 2012 through September 2018—to gain insight into the results of projects supporting the long standing U.S. assistance objectives of prosperity, governance, and security in the Northern Triangle. While we reviewed four evaluations that USAID completed at the beginning of fiscal year 2013, as shown in appendix III, three of these were mid-point evaluations of ongoing projects that continued implementation in fiscal years 2013 and 2014, during the time frame for our review. Although we reviewed one final evaluation of a project that had ended prior to the beginning of fiscal year 2013, the evaluation was a key aspect of the project’s implementation and lessons learned, which provided information pertinent to future USAID programming in the areas of justice reform and security. Furthermore, while our report noted examples of actions that agencies took in response to challenges to achieving progress toward prosperity, governance, and security, analysis of actions taken in the design of specific projects based on the findings and recommendations of the evaluations we reviewed was outside the scope of our review. 4. We believe that our classification of projects under different sectors we identified provides a detailed, comprehensive, and meaningful analysis of projects and related results. Because some of the sub- objectives developed by State and USAID, such as “reduce poverty,” were very broad and did not lend themselves to an analysis of specific project sectors that supported the Strategy’s objectives, we identified more specific sectors, including health, economic growth, and agricultural development. State and USAID validated the accuracy of our definitions, and we revised them as appropriate, given input from agency officials. We acknowledge that our selection of a judgmental sample of six sectors for in-depth review of projects and results limits the generalizability of our findings to all sectors, which we noted throughout our draft report. However, due to the large number of projects, sectors, and sub-objectives associated with U.S. assistance to the Northern Triangle and the extensive amount of documentation to obtain and analyze for each project, we determined that this case study approach was the most effective methodology for our review. We devised our selection criteria for our case study to reflect a meaningful selection of a significant number of projects across objectives, sectors, agencies, and countries. We do not believe that omitting some sectors from our in-depth review limited the credibility of the findings of our report. 5. We believe that the use of data on the achievement of annual targets is a valid approach to assessing project results, although the agencies collecting the data may also intend to use it in making decisions about the progress of ongoing projects. These data were only one element of our analysis. We also analyzed data and information from USAID implementer progress reports, mid-point and final evaluations, and other performance reports, which provided a longer-term perspective on results. Collectively, we believe that this information provided meaningful insight into the successes and shortcomings of the projects in the sectors we reviewed. Our report acknowledges that agency officials described progress and challenges to achieving the prosperity, governance, and security objectives, as well as the steps taken to modify projects to address such challenges. However, such modifications fell outside the scope of our analysis of results, absent documentation of the specific impact of such modifications on the achievement of objectives. 6. We reviewed completed evaluations to provide insight into project results, but excluded ongoing and planned evaluations because conclusions about project results are not available until such evaluations are completed. Similarly, our draft report acknowledged that agency officials described progress and challenges to achieving the prosperity, governance, and security objectives, as well as the steps taken to modify projects to address such challenges. However, such modifications fell outside the scope of our analysis of results, absent documentation of their specific impact on the achievement of prosperity, governance, and security objectives. 1. We believe that the inclusion of DOD projects significantly enhanced the accuracy and completeness of our reporting on the projects that the U.S. government has implemented in the Northern Triangle from fiscal years 2013 through 2018, and the important lessons learned from these projects on progress toward the Strategy’s objectives. State and DOD officials confirmed that DOD has designed and implemented projects from its appropriation that support the security objective of the Strategy in the Northern Triangle. Furthermore, we are not recommending that State and USAID specify how DOD monitors and evaluates such projects, but rather that State and USAID collaborate with DOD to specify a comprehensive approach to the monitoring and evaluation of projects across all agencies that directly support the Strategy’s objectives. Excluding DOD projects from the monitoring and evaluation plan for the Strategy could result in an incomplete or unclear understanding of the results of U.S. assistance in the Northern Triangle. Without a complete and clear understanding of the results across all agencies involved, including DOD, agencies may miss important lessons learned about the types of assistance that are most effective in this region, potentially limiting overall progress. In addition to the contact named above, James Michels (Assistant Director), Bradley Hunt (Analyst-in-Charge), Sophie Broach, Jon Fremont, Kayli Westling, Pedro Almoguera, Neil Doherty, Mark Dowling, Justin Fisher, Christopher Mulkins, Zamir Ruli, Aldo Salerno, and John Villecco made key contributions to this report.", "summary": "The United States has provided assistance to the Northern Triangle of Central America for many years to address poverty, weak governance, and insecurity. Introduced in 2014, and updated in 2017, the U.S. Strategy for Engagement in Central America (Strategy) supports the objectives of improving prosperity, governance, and security. State coordinates implementation of the Strategy's objectives among agencies. This report examines: (1) the projects the U.S. government has implemented from fiscal years 2013 through 2018 to support the Strategy's objectives in the Northern Triangle, (2) what is known about project results, and (3) what is known about progress toward the objectives. GAO reviewed results for a subset of 190 projects in a nongeneralizable sample of six sectors selected based on funding, country, and objective; analyzed Strategy documents and key elements of effective strategies; interviewed officials; and conducted fieldwork in the Northern Triangle. To support their prosperity, governance, and security objectives, the Departments of State (State), Defense (DOD), Agriculture (USDA), and the U.S. Agency for International Development (USAID) allocated about $2.4 billion from fiscal years 2013 through 2018 for 370 projects in the Northern Triangle—El Salvador, Guatemala, and Honduras. USAID and State implemented most of these projects, with some supporting more than one sector and objective. For example, USAID implemented projects to address poverty, while State trained prosecutors and police to address governance and security needs. State, USAID, and other agencies reported mixed results for the 190 projects in the six sectors GAO reviewed. For example, in fiscal year 2018, USAID assisted 1,376 individuals in workforce development programs in Guatemala, exceeding the target of 1,000, while it assisted 651 individuals in Honduras, falling short of the target of 5,000. State and USAID trained 12,557 justice system personnel in the Northern Triangle, exceeding the target of 2,275. USDA rehabilitated school kitchens in Honduras as part of its school feeding program. DOD helped Guatemala establish a budget system to increase accountability for military funds, but DOD reported persistently low public trust in Northern Triangle militaries. Limited information is available about how U.S. assistance improved prosperity, governance, and security in the Northern Triangle. Agencies generally reported more information about progress toward prosperity than toward governance and security, in part because evaluations were conducted unevenly across agencies and sectors. In addition, project implementers did not consistently collect key information needed to evaluate progress, but officials noted improvements. Nevertheless, agency officials described examples of progress through technical assistance, and noted challenges, such as drought. GAO has reported that development of a monitoring and evaluation plan is key to assessing agencies' common goals and objectives, and mutually reinforcing results. While State has a monitoring and evaluation plan for the Strategy, the plan does not include activities by DOD and USDA that support the Strategy's objectives and thus does not establish a comprehensive approach to assessing progress. GAO recommends that State collaborate with DOD and USDA to develop a comprehensive approach to monitoring and evaluation of projects that support Strategy objectives. State did not concur, citing lack of authority to direct other agencies' actions. GAO modified the recommendation to clarify that a collaborative effort would allow State to include information about all relevant projects as it evaluates progress under the Strategy as discussed in this report.", "document_type": "gao"}
{"report": "School-age children can access the internet in a number of ways. Their households may subscribe to in-home fixed internet, which is generally provided by cable television or telephone companies. School-age children, and other users, can connect a variety of devices to in-home fixed service through a wired connection or a Wi-Fi connection. They may also access the internet through mobile wireless service, which is provided through cell towers, with data transmitted over radio frequency spectrum. Mobile service providers usually sell internet access as an option in mobile telephone-service plans. A number of devices may connect to mobile wireless, such as smart phones, tablets, and mobile devices that enable laptops to connect to mobile wireless service. Finally, school-age children and others may access the internet outside the home through other ways, including publicly available Wi-Fi access at places such as libraries and coffee shops. FCC has found that Americans in lower-income areas are less likely to have access to both in-home fixed and mobile wireless internet than those in higher-income areas. Similarly, according to our analysis of data from the November 2017 CPS: Computer and Internet Use Supplement, among all school-age children, those in lower-income households are less likely to use the internet at home than those in higher-income households (see fig. 1). A number of factors explain the digital divide, or the varying levels of access among different populations. For example, as we have reported in the past, rural areas tend to have conditions such as low population density or difficult terrain that can increase the costs for internet providers to deploy and maintain internet networks. Furthermore, lower-income households with access to the necessary infrastructure for internet service may not be able to afford it. (See fig. 2.) While some in-home fixed internet providers offer low-cost service for lower-income households with school-age children, according to a 2016 survey, an estimated 5 percent of households with school-age children ages 6 to 13 and incomes at or below the federal poverty guidelines had ever signed up for such programs. Lower rates of internet access by lower-income households may make it more difficult for school-age children in those households to do homework. According to a 2018 Pew Research Center survey, a higher percentage of surveyed teens in lower-income households said that the lack of a dependable computer or internet connection sometimes prevents them from finishing their homework compared to teens in higher- income households. In addition, according to the Consortium for School Networking, the lack of in-home access makes it more difficult for parents to support their children academically. Specifically, as much communication between schools and parents has moved online, the lack of access may make it difficult for parents to stay connected to teachers and be informed about school notices, homework assignments, and other important information. FCC, which regulates commercial and other nonfederal spectrum, conducts activities that affect the ability of schools to address the homework gap. Specifically, it plays a role in expanding internet access by assigning licenses for Educational Broadband Service (EBS) spectrum, which permits schools and other eligible entities to transmit educational materials electronically. Currently, EBS license holders are allowed to lease excess capacity to others, including commercial wireless providers, for up to 30 years as long as the license holder has 20 hours of educational use per week per licensed channel and reserve the right to access 5 percent of the capacity for educational use. Schools that have such leases may need to wait years to regain full use of their EBS license. Furthermore, the last opportunity for school districts to apply for new EBS licenses was in 1995, and according to FCC, EBS licenses cover about half the geographic area of the United States, with rural areas west of the Mississippi River generally lacking licenses. However, FCC recently adopted a Report and Order with rules that, once effective, will change the eligibility requirements for EBS licenses, among other things. In addition, FCC supports internet investments at schools through the E- rate program, which provides discounts on telecommunications and internet access services, internal connections, and basic maintenance of internal connections. This program provides schools with higher percentages of lower-income students greater discounts on these services; for example, the most disadvantaged schools, where at least 75 percent of students are eligible for free or reduced price school lunch, receive a 90 percent discount. All services supported by the E-rate program must be used primarily for “educational purposes,” which FCC has defined as meaning “activities that are integral, immediate, and proximate to the education of students.” Education’s Office of Educational Technology also plays a role related to internet access for students by developing national educational- technology policies and providing guidance to schools and school districts on technology use in schools. For example, in January 2017 the office issued a letter to schools and school districts about Education grant funds that could be used to support the use of technology to improve instruction and student outcomes. It also issued a report in 2017 on the use of technology in schools; the report provided guidance on how to modernize the technology needed for digital learning, such as schools’ internet networks and internet-enabled devices. Education also collects, analyzes, and reports on a range of data from schools and school districts. For example, every year from 1994 to 2005 (except 2004 due to a lack of funding according to Education officials), the department collected data on internet access in schools and classrooms. In 2008, Education conducted three similar surveys at the district, school, and teacher levels on the availability and use of a range of educational technology resources, such as networks, computers, devices that enhance the capabilities of computers for instruction, and computer software. Due to a lack of funding, Education did not conduct additional similar surveys. However, the department recently finished administering a different survey effort, funded from different sources, that we discuss later in this report. According to our analysis of November 2017 CPS: Computer and Internet Use Supplement data, lower-income households with school-age children may be more likely than those in higher-income households to be reliant on mobile wireless service, such as through smart phones, for internet access. As seen in figure 3, among all households with school-age children, an estimated 22 percent with incomes of less than $25,000 per year use mobile wireless to access the internet but not in-home fixed high-speed internet service, in contrast to 8 percent with incomes of $75,000 or more per year. School-age children whose households only have mobile wireless internet access may face challenges in using it for homework, including: Device limitations. Students in mobile wireless-only households may have to rely on devices like smartphones that may not be well suited for academic tasks. A recent Pew survey found that an estimated 45 percent of teenagers in lower-income households say they sometimes have to do homework on a smartphone. However, most of the stakeholders we interviewed told us that smartphones are not adequate for doing homework for various reasons, including that they are too small for typing papers and that not all educational websites are compatible with smartphones. According to these stakeholders, other devices such as desktops or laptops are better suited for homework; however, among all school-age children, those in lower- income households are less likely than those in higher-income households to use these devices (see fig. 4). Data limitations. A majority of the stakeholders we interviewed said that wireless plans’ data caps—a limitation on the amount of data the subscriber can download and upload per month—could make it difficult for school-age children to do homework, because, for example, once the data cap is reached, the provider may decrease connection speeds or impose additional costs for further data use, which could hinder completion of homework. A 2016 survey found that an estimated 39 percent of lower-income households with school- age children—in this case those with incomes less than the federal poverty guidelines—had reached a data cap, compared to 25 percent of higher-income households. Varying service quality. Mobile wireless may be less reliable and slower than in-home fixed service, which can make doing homework more challenging. In 2018, FCC concluded that mobile wireless services are not full substitutes for in-home fixed service, because mobile wireless quality can be affected by user location, indoor obstructions, outdoor foliage, and weather, among other factors. In addition, we reported in 2015 that the availability and quality of mobile wireless service connections vary based on location and terrain. For example, according to officials with Albemarle County Public Schools in Virginia, while most students who participated in a recent survey indicate that they have mobile wireless internet access at home, that access may only offer poor quality connections and slow speeds due to mountainous terrain. As a result, mobile wireless access may have limited usefulness for homework purposes. A 2018 survey by the Pew Research Center found that about 20 percent of teens from lower-income households say that they sometimes have to use public Wi-Fi for homework given a lack of access at home. As shown in figure 5, stakeholders we interviewed and literature we reviewed identified a number of potential challenges students may encounter in using methods to access the internet outside the home to do their homework. The six selected school district projects we reviewed have taken various approaches to address the homework gap by providing wireless internet service to students who may lack access at home. Most of these projects provide wireless internet access to students who lack in-home fixed internet and do not necessarily limit it to students in lower-income households. In addition, all but one of these projects provide filtered access, meaning that students using these services are subject to the same usage restrictions as if they were on-site in school. Approaches included: Provide wireless hot-spot devices. The Green Bay Area Public School District in Wisconsin loans out mobile wireless hot-spot devices to students throughout the district who do not have access at home, providing them filtered internet access in their homes or elsewhere in the community. The hot-spot devices are available on loan from school libraries to any student who claims a need for one regardless of household income. Students may use district-issued Chromebooks or other internet-enabled devices, which then connect to the district’s internet resources via the hot-spot device using service provided by a commercial mobile-wireless provider. Build or use a private network. Some districts have built new or expanded existing networks to provide internet access to students using a variety of approaches. Albemarle County Public Schools in Virginia uses EBS spectrum to provide access to students in community centers in mobile home parks in this mountainous district where, according to school district officials, many students lack service at home. The district also plans to install wireless receiver devices in selected students’ homes through which those students will be able to connect internet-enabled devices via Wi-Fi. Desert Sands Unified School District in California also built out an EBS network to provide internet access to students who lack service at home. According to officials with that district, the benefit of this approach is that it involved only a one-time cost to build the network, rather than recurring annual payments to a commercial mobile-wireless provider for service. Two rural, low-income school districts in Virginia—Charlotte County Public Schools and Halifax County Public Schools—partnered with Microsoft to provide service through unlicensed white space devices (which operate on frequencies not being used by television broadcasters or 600 MHz wireless providers) to students who lack access at home, regardless of income . According to Microsoft, the use of unlicensed white space devices is a good solution to providing wireless access in rural areas where other technologies may be uneconomical and such frequencies tend to be available. Students who use this service receive a device that is installed in their home that wirelessly connects to the district’s network and transmits to other devices in the home via Wi-Fi. The Boulder Valley School District in Colorado allowed a local wireless provider to build antennas on some school buildings in order to serve its customers in exchange for providing free service to lower- income students, determined based on student eligibility for free or reduced price lunch. According to a school district official, the provider has installed antennas at three schools, providing access to students living within a 3-mile radius, and plans to install antennas at most remaining schools in the district. That official told us that this model may not work in many other school districts, as there may not be sufficient population density to make it economically beneficial for a commercial provider to agree to provide such service. Equip school buses with Wi-Fi. The Coachella Valley Unified School District, which covers a large geographic area in California where many students lack in-home fixed access, equipped its fleet of about 100 school buses with Wi-Fi in 2014, enabling students to do homework during long bus rides. A commercial mobile-wireless provider connected the Wi-Fi router on the bus to the district’s network. In order to access Wi-Fi on the buses, students had to use district-issued devices that they were allowed to bring home after school. The district also parked Wi-Fi-equipped school buses and other district vehicles overnight in neighborhoods with a high proportion of students who brought district-issued tablets home in order to provide access to students who likely lacked internet at home. However, the district stopped this initiative in 2017 due to limited funding and is now seeking out alternative funding sources to reactivate the program. While none of the projects described above used any funding from Education, the department has identified six existing grants that schools and districts could use under certain conditions to support internet investments, although not necessarily wireless investments specifically. While the purpose of each of these grant programs isn’t specific to internet investments, Education identified specific types of internet investments that these grant funds can be used for. We did not make a determination as to whether any of the grant funds could have supported the efforts we reviewed. Representatives of two of the school districts we met with stated that they would like to see additional information on Education grants that could be used to support internet investments. Education officials said the department has taken the first step to developing a strategy to share information about these grants by developing a coordinated communications strategy through its Office of Rural Engagement. They added that the department will then continue to build a broader strategy. Education is also finalizing data collection on a survey that will collect some data regarding the homework gap. As mentioned earlier, until 2008 Education collected survey data over a number of years about information technology and internet access in schools and classrooms. According to Education officials, the department stopped collecting such data due to a lack of funding. However, the department is now finalizing a survey that is collecting nationally representative data about public school teachers’ use of computers and the internet, and their knowledge of students’ access to computers and the internet outside the classroom. The survey is collecting data that pertain to the homework gap, including the extent to which schools provide wireless hot-spot devices to students to take home; the extent to which teachers think students access the internet outside of school, such as at home, libraries, or businesses; and the extent to which teachers think smartphones are useful for doing homework. According to Education, the department finished administering the survey in June 2019 and plans to release the results in April 2020. The survey data may provide Education and others, including FCC and Congress, with useful information that can inform policy and other decisions related to the homework gap, such as how best to support schools’ efforts to expand wireless access for underconnected students. FCC had a minor role in some of the school district projects by having previously granted EBS licenses to some districts that use EBS spectrum to provide wireless access. However, according to FCC documentation, many schools and school districts do not have EBS licenses—such as those in rural areas in the western United States—and some that have obtained a license now lease their capacity out on a long-term basis to commercial providers. As a result, school districts may be limited in using EBS to provide wireless access to students or have to take additional steps to use EBS. Desert Sands Unified School District officials said that the district did not have an EBS license and that the local license holder had leased it out to a commercial provider, so the district worked with that provider to build out its EBS network. Albemarle County Public Schools had leased out its EBS license to a commercial provider years ago, but because that provider was not utilizing that spectrum, the school district was able to reclaim it. FCC has taken recent steps that may affect the extent to which school districts are able to use EBS to provide wireless access. In May 2018, FCC issued a Notice of Proposed Rulemaking seeking comment on proposed changes to how it manages EBS to encourage and facilitate its efficient use. In July 2019, FCC adopted a Report and Order that makes a number of changes to the EBS spectrum and its use. Specifically, once effective, these rules will eliminate eligibility restrictions for EBS licenses and eliminate the educational use requirement of the spectrum. While FCC’s E-rate program supports schools’ connectivity by providing discounts for eligible services, program rules may limit the ability of schools and school districts to address the homework gap. Specifically, program rules specify that off-premises use of such services is not eligible for E-rate support and require that any off-premises traffic must be cost allocated out of school districts’ E-rate discounts. For example, any off- premises traffic supported by existing E-rate-supported products or services requires a reduction in the E-rate discount for those existing E- rate supported products and services. This reduction may increase costs for school districts as they would no longer receive all their potential E- rate discounts. Officials representing all six of the school district projects we reviewed suggested that program rules limiting eligibility for off- premises use and requiring cost-allocation may inhibit the ability of school districts to expand off-premises wireless access, and thus address the homework gap. For districts that do provide wireless access off-premises, E-rate program restrictions may still pose challenges. For example, according to an official with Desert Sands Unified School District, the district had to buy a separate line of internet access to avoid having that off-premises traffic travel through the district’s existing E-rate-supported network, which would have required cost-allocation and a reduction of the E-rate discount for that existing E-rate supported network. According to officials with Microsoft, Charlotte County Public Schools and Halifax County Public Schools had to separate their off-premises unlicensed white space device traffic from internet traffic that passed through E-rate-discounted access in the schools. An official with Boulder Valley School District said that the district had to terminate an earlier effort to extend access to students in a housing development after being told that it could not provide off- premises access with program-discounted equipment without cost- allocation. In September 2016, FCC issued a Public Notice requesting public comment on two petitions filed with the agency seeking to allow the petitioning school districts to use existing E-rate-program-supported services and equipment for off-premises access without having to cost- allocate that traffic out of their existing E-rate discounts. Cost allocating out that traffic would result in reduced E-rate discounts for school districts, and therefore higher costs, for existing services and equipment supported by E-rate. FCC rules allow parties to petition for waivers of rules if they can demonstrate that special circumstances warrant deviation from the existing rules and doing so serves the public interest. According to FCC officials, the petitions are pending and the agency has not yet taken further formal action on this Public Notice. The petitions are described in more detail below. In May 2016, the Boulder Valley School District filed a petition requesting a waiver of the cost allocation rules in order to use its E- rate-program-supported network to provide internet access to students at public housing facilities after school hours. In the petition, the district argued that because traffic on its E-rate program- supported network dramatically decreased after school hours, using that network to provide access during that time would not impose any additional costs on the E-rate program. Microsoft and others—including the school districts in Charlotte and Halifax counties—filed a petition in 2016 to obtain clarification that those school districts could provide wireless access to students’ homes for educational purposes by extending the districts’ existing E- rate-supported services using the districts’ unlicensed white space device network without cost allocating that traffic from the existing E- rate discounts. The petition stated that the infrastructure to provide service to unlicensed white space devices would not be funded with E-rate program funds, and that these districts were not well served by commercial internet providers. In comments filed with FCC, Microsoft argued that projects covered by both petitions would provide in-home access for students without imposing any additional costs to the E- rate program and that the projects would increase the productivity of E-rate by using existing resources more efficiently. Previously, FCC explored the possibility of making wireless off-premises access an allowable E-rate program expense—which would eliminate the requirement to cost-allocate such traffic—in a 2011 to 2012 pilot program. When establishing this pilot program, FCC noted commenter concerns regarding the potential administrative, legal, technological, and procedural challenges of expanding E-rate funding to off-campus premises. The pilot program provided funding from July 2011 to June 2012 and sought to “investigate the merits and challenges of wireless off- premises connectivity services” and to “gain a better understanding of operation and administrative issues associated with off-premises use and connectivity, as well as the financial impact on the E-rate program overall.” Furthermore, the pilot program sought to help FCC determine whether off-premises connectivity services “should ultimately be eligible for E-rate support.” FCC provided a total of $9 million in grants to 20 pilot-program participants—19 schools or school districts and one community library system—to implement projects enabling innovation in learning outside the boundaries of school buildings and the traditional school day, including those that provided off-premises wireless access and wireless devices to students. Recipients were not required to cost allocate the off-premises traffic as part of the pilot. FCC required all pilot participants to file interim and final reports that included information about project benefits, such as the extent to which students provided with wireless devices used them and the effect of increased internet access on academic outcomes; project costs; the effectiveness of measures to prevent project waste, fraud, and abuse, to filter content, and to ensure that students only used the devices for educational purposes; and lessons learned. According to FCC, those reports would allow it to assess the impact of selected pilot projects on the schools and to gather lessons learned that would help others implement similar projects in the future. In addition, FCC said it would evaluate the effectiveness of the pilot program to determine whether off- premises wireless access should be eligible for E-rate program support. While FCC received interim and final reports from most pilot participants, it did not determine a methodology for evaluating the data provided in those reports. Furthermore, FCC did not publish a report evaluating the effectiveness of the pilot program, including the potential costs, benefits, and challenges of off-premises wireless access to make a determination regarding whether off-premises access should be eligible for E-rate program support. Although the order establishing the pilot did not require FCC to determine an evaluation methodology and publish a formal analysis, according to FCC officials, staff reviewed the interim and final reports prior to the Commission adopting a 2013 Notice of Proposed Rulemaking that sought input on ways to modernize the E-rate program, including input on using E-rate-supported wireless hot-spots for community use. In two subsequent E-rate program modernization orders in 2014, the Commission did not expand the E-rate program’s support for off-premises access. FCC officials explained that given the changes in technology, costs, and student learning in recent years, the data collected from the pilot may have some limitations. FCC has not announced any plans to conduct another pilot program, and aside from its consideration of the petitions previously mentioned, FCC has not announced an intention to revisit whether off-premises wireless access should be eligible for E-rate support. Federal internal control standards state that agencies should use quality information to make decisions and communicate information to external parties. Specifically, agencies should collect data from reliable sources in a timely manner, process these data into quality information, and use that information to make informed decisions. Agencies should also communicate such information to external parties that can help the agencies achieve their objectives. Furthermore, in previous work we identified as pilot-program design best practices: determining a methodology for gathering and evaluating data, evaluating pilot results to make conclusions on whether to integrate pilot activities into broader efforts, and communicating with stakeholders—such as by publishing results. As discussed earlier, school districts we met with said that existing E-rate program rules that require cost-allocation of off-premises access to E-rate discounts limit their ability to address the homework gap and providing off-premises access remains a challenge for schools and school districts. Determining and executing a methodology for collecting and analyzing data on the potential costs, benefits, and challenges of making schools’ efforts to expand off-premises wireless access eligible for program funding could help inform FCC decisions regarding the two pending petitions and any future petitions. As petitions may only cover petitioning entities, determining and executing such a methodology could also help inform more widespread changes to E-rate rules regarding off-premises access that would affect all E-rate program recipients. FCC could collect such data through another pilot program or from school districts now providing off-premises wireless access. Publishing the results of this analysis could help FCC ensure that such information will be accessible to inform future related efforts and provide transparency to external stakeholders, including school districts. The differences in internet access—and therefore in the ease of doing homework—between school-age children in lower-income households and those in higher-income households that are more likely to be well connected has resulted in a “homework gap” that could inhibit the academic success of underconnected students. While school districts have made efforts to address the homework gap, such efforts may be inhibited by existing restrictions in FCC’s E-rate program. Although FCC explored the possibility of making wireless off-premises access an allowable E-rate program expense in a 2011 to 2012 pilot program, FCC’s lack of an analysis of the data it collected at the time or since then means that it may not have sufficient and relevant information to make a decision on pending petitions from local school districts regarding off-premises access. Determining the best way to collect and analyze data on the potential benefits, costs, and challenges of making off-premises wireless access eligible for E-rate program support; conducting such analysis; and publishing the results could provide relevant information and transparency to external stakeholders. Doing so could also enable FCC to make a determination on whether it would be appropriate to ease restrictions on off-premises access, a step that may give school districts more flexibility in addressing the homework gap. We are making the following recommendation to FCC: The Chairman of the Federal Communications Commission should determine and execute a methodology for collecting and analyzing data— such as conducting a new pilot program regarding off-premises wireless access or analyzing other data—to assess the potential benefits, costs, and challenges of making off-premises wireless access eligible for E-rate program support, and publish the results of this analysis. (Recommendation 1) We provided a draft of this report to FCC, Education, and the Department of Commerce for review and comment. FCC provided written comments, which are reproduced in appendix II. In these written comments, FCC stated that it agreed with our recommendation and noted steps it plans to take to assess the potential benefits, costs, and challenges of making off- premises broadband access eligible for E-Rate program support. FCC also provided technical comments, which we incorporated as appropriate. Education provided written comments, which are reproduced in appendix III and also provided technical comments that we incorporated as appropriate. The Department of Commerce reviewed our report and told us it did not have any comments. We are sending copies of this report to interested congressional committees, the Chairman of the FCC, the Secretary of Commerce, and the Secretary of Education. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our objectives for this report were to examine: (1) challenges lower- income school-age children who lack in-home fixed internet face in doing homework that involves internet access and (2) what selected school districts are doing to expand wireless internet access for their students, and the federal role in such efforts. To examine challenges lower-income school-age children who lack in- home fixed internet face in doing homework that involves internet access, we analyzed data from the Census Bureau’s November 2017 Current Population Survey: Computer and Internet Use Supplement, which is sponsored by the National Telecommunications and Information Administration (NTIA). The Computer and Internet Use Supplement collected household information from all eligible Current Population Survey households, as well as personal information from household members age 3 and older. The supplement provided data about households’ computer and internet use, and about each household member’s use of the internet from any location during the previous six months. One member of a household was generally interviewed and answered questions on behalf of every other member. Interviews were conducted from November 12–18, 2017. The probability sample selected to represent the universe consisted of approximately 56,000 households. We included variables on ages of household members to determine if the household had one or more school-age children. We considered a household to have school-age children if it had any children between the ages of 6 and 17, an age range used in other analyses of internet use by school-age children, such as analyses by NTIA and Pew Research Center. We analyzed data on the use of in-home fixed and mobile- wireless internet, as well as of various computing devices. In our analysis we also included variables on household income, to allow us to report results based on different income ranges. When analyzing responses by household income, we grouped household income into similar ranges that NTIA publishes on its Data Explorer website, but we consolidated the top two ranges used by NTIA into one range. To determine the reliability of these data, we reviewed NTIA technical documentation on the survey, interviewed NTIA officials, and compared our estimates of selected variables with estimates presented by NTIA on its website. We found these data were sufficiently reliable for reporting on data on internet and computing device use by household income levels. In addition, we conducted a literature search to review challenges lower- income school-age children who lack in–home internet face in doing homework that involves internet access. We searched multidisciplinary databases using relevant terms such as “low-income,” “wireless,” “internet,” and “school-age children.” We searched for scholarly articles, including working and conference papers, government reports, think tank publications, and trade publications published between 2013 and 2018. We reviewed the abstracts of results from the search for publications most relevant to our work and fully reviewed publications that, based on their abstract, were most suited to this engagement. We used relevant publications to support findings we collected from other sources, including interviews. We also conducted semi-structured interviews with a range of stakeholders, including education industry associations, researchers, and advocacy organizations we selected based on literature, internet searches, and recommendations from those we interviewed. Specifically, we interviewed eight education or technology industry associations or advocacy organizations, one education researcher, one technology industry researcher, and representatives of one technology company that provides internet services and products to schools. In addition, we interviewed officials with the Federal Communications Commission (FCC) and Department of Education (Education). We also reviewed a non-generalizable sample of six projects involving seven local school districts taking steps to provide wireless internet access outside of school for students who may lack internet at home. We identified these projects based on keyword searches and recommendations from other interviewed associations and researchers, as well as officials with FCC, NTIA, and Education. From this list, we then selected those projects that were frequently cited in the press or by others we interviewed; that covered a variety of geographic locations, including those in both urban and rural areas; and that included a variety of approaches to addressing the homework gap. During these interviews, we asked interviewees about a range of topics, including the extent to which school-age children have access to in-home and wireless internet and challenges faced by students who may only have mobile wireless access. In total we interviewed 17 stakeholders, including the industry associations, researchers, and school districts detailed above. We analyzed the content of the interviews to identify key challenges identified by stakeholders. These interviews did not provide a complete list of all challenges, and the results of these interviews are not generalizable but do provide insight into a range of issues. To determine what selected school districts are doing to expand wireless internet access for their students and the federal role in such efforts, we conducted semi-structured interviews with officials at the school districts listed above and officials at Microsoft regarding its efforts to expand wireless access for students who may lack internet at home. During these interviews, we asked the districts about what steps they are taking to expand wireless access, the goals and challenges of the relevant project, and the federal role in the effort. We analyzed the content of the interviews to identify key themes. We also interviewed officials with FCC and Education to determine and review federal efforts related to school initiatives to expand wireless access for students. We reviewed documentation from FCC and Education regarding relevant federal efforts including rulemaking documents such as FCC’s 2018 Notice of Proposed Rulemaking and 2019 Report and Order regarding Educational Broadcast Service spectrum. We reviewed other relevant FCC documents related to the Schools and Libraries Universal Service Support Mechanism (also known as the E-rate program), which provides schools with discounts on telecommunications and internet services. E-rate documents we reviewed included reports related the 2011 E-rate pilot program exploring off- premises wireless access. We compared FCC efforts to federal internal control standards related to using quality information and communicating externally and pilot program design best practices. We reviewed information, provided to us by department officials, on existing Education grant programs that can be used by schools and school districts to support internet investments. We also reviewed information on Education’s relevant survey efforts. We conducted this performance audit from May 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Andrew Von Ah at (202) 512-2834 or vonaha@gao.gov. In addition to the contact above, Mark Goldstein (Director); Derrick Collins (Assistant Director); Matthew Rosenberg (Analyst in Charge); Dwayne Curry; Sherri Doughty; Rachel Frisk; Hayden Huang; Gina Hoover; Dan Luo; Josh Ormond; Cheryl Peterson; Matt Ray; Hai Tran; and Laurel Voloder made key contributions to this report.", "summary": "School-age children without internet access may have difficulty in completing homework. Those without in-home fixed access may go online wirelessly outside the home to do homework. A provision was included in statute for GAO to review wireless internet access for school-age children in lower-income households. This report examines (1) challenges lower-income school-age children who lack in-home fixed internet face in doing homework involving internet access, and (2) selected school district efforts to expand wireless access for students and the federal role in those efforts. GAO analyzed 2017 CPS data; reviewed six local projects that were selected based in part on education industry stakeholders' recommendations, that included a range of geographic locations, and that took steps to address the homework gap; compared FCC efforts to federal standards for internal controls and pilot-program design best practices; reviewed FCC and Department of Education documents; and interviewed 17 stakeholders, including school districts. According to GAO's analysis of 2017 Census Bureau Current Population Survey (CPS) data, children ages 6 to 17 in lower-income households are more likely than peers in higher-income households to lack high-speed in-home internet and rely on mobile wireless service. GAO found that students who use mobile wireless for homework may face challenges, including slower speeds and limitations smartphones present in completing tasks like typing papers. These “underconnected” students may seek out ways to access wireless internet outside of the home to do homework; however, these methods also pose challenges (see figure). The inequity in internet access—and therefore in the ease of doing homework involving access—between students of varying income levels is known as the “homework gap.” Efforts by six selected projects involving seven school districts expanding wireless access for students who may lack it at home varied. According to officials with most school district projects GAO reviewed, rules for the Federal Communications Commission's (FCC) E-rate program, which allows schools to purchase discounted internet equipment, may limit schools' ability to provide wireless access off-premises. Specifically, off-premises access is not eligible for E-rate support, and schools that provide such access using existing services supported by E-rate must reduce their E-rate discounts. FCC conducted a pilot project in 2011 and 2012 to help decide whether to make wireless off-premises access eligible for E-rate support, but FCC did not determine and execute a methodology to assess the potential costs, benefits, and challenges of doing so. In 2016, FCC received two requests from school districts seeking waivers of rules to allow them to use E-rate program support to provide off-premises access, but FCC has not made a decision on the waivers. Determining and executing a methodology to analyze data about the potential benefits, costs, and challenges of easing E-rate rules on off-premises use and publishing the results could provide transparency to stakeholders such as school districts. This step could also help FCC act on pending and future waiver-of-rule requests and broader changes to rules that may help schools address the homework gap. GAO recommends that FCC take steps to assess and publish the potential benefits, costs, and challenges of making off-premises wireless access eligible for E-rate support. FCC agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "When a disaster overwhelms the ability of state, local, or voluntary agencies to adequately provide essential services on their own, the federal government, when requested, supports disaster response and recovery, providing selected resources where they are needed. The federal government has provided significant funds for transit services following past catastrophic disasters. For example, Congress provided roughly $232 million in response to the 2005 Gulf Coast hurricanes and over $10 billion in response to Hurricane Sandy. FEMA is the federal government’s primary agency for disaster response. In addition to coordinating disaster response and recovery operations, FEMA’s Public Assistance Program provides funding to state and local governments and some nonprofit organizations for recovery efforts after a disaster, including removing debris, implementing emergency protective measures, and repairing or replacing damaged public equipment or facilities. Once the President has declared a disaster, FEMA; the state or territorial government (the recipient); and the local or territorial entities (the subrecipient) work together to develop damage assessments and formulate project worksheets for eligible projects. Project worksheets detail the scope of work and estimated cost for repairing or replacing disaster-damaged infrastructure. After a project has completed FEMA’s review process and is approved, funding is available to FEMA for obligation from the Disaster Relief Fund. The recipient draws down—or withdraws—funding to pay the subrecipient for eligible work upon completion. Because FTA’s Public Transportation Emergency Relief Program is focused on public transportation specifically—unlike FEMA’s more general program—FTA has primary responsibility for reimbursing emergency response and recovery costs after an emergency or major disaster affects a public transportation system if FTA receives funds for the program in an annual or supplemental appropriation or continuing resolution. The Public Transportation Emergency Relief program is a reimbursable grant program and allows FTA to make grants for capital projects to protect, repair, reconstruct, or replace equipment and facilities of a public transportation system as well as for eligible operating costs. Such costs include reestablishing, expanding, or relocating public- transportation route service in the event of a natural disaster that affects a wide area or a catastrophic failure from any external cause. Congress has not provided an annual appropriation for FTA’s Public Transportation Emergency Relief Program but has provided supplemental appropriations following a specific event. Eligible recipients (referred to in this report as “FTA grantees”) of FTA’s Public Transportation Emergency Relief funding are entities that receive funds directly from FTA. Following the appropriation for the 2017 hurricanes, FTA staff and contractors visited sites to develop damage assessments—these assessments provide information on, among other things, the specific location, type of facility or equipment, nature and extent of damage, and a preliminary cost estimate to restore, replace, or reconstruct the damaged system. FTA then uses the information in these damage assessments to determine how to allocate funding among the affected FTA grantees. After FTA announces the allocations, FTA grantees can submit an application for funding to FTA. After FTA has approved the application and obligated funds, recipients must execute the grant agreement to draw down funding for reimbursement of eligible expenses. As required by MAP-21, FTA and FEMA have entered into a memorandum of agreement (MOA) to delineate the roles and responsibilities of the two agencies and establish procedures to coordinate assistance for public transportation following a disaster. We reported in 2014 that because FTA’s Public Transportation Emergency Relief Program is inherently limited by its inability to fund any activities without specific congressional action (in contrast to the other emergency program we examined), FTA and FEMA face challenges clearly delineating the responsibilities and costs each agency will assume during future disasters. We recommended that FTA and FEMA establish specific guidelines to monitor, evaluate, and report the results of collaborative efforts for future disasters. FEMA concurred with this recommendation and FTA took no position. The agencies addressed the recommendation by: (1) implementing a communications protocol to coordinate the two agencies in providing funding to transit agencies and (2) committing to jointly monitoring, evaluating, and reporting on the effectiveness of agency collaboration following events in which both agencies provided funding. In August and September 2017, Hurricanes Harvey, Irma, and Maria made landfall in Texas, Florida, the U.S. Virgin Islands, and Puerto Rico, affecting over 28 million people and causing significant damage to public transit infrastructure (see fig. 1). FEMA funding was made available through presidential disaster declarations. In February 2018, 6 months after the first hurricane made landfall, Congress appropriated funds to FTA’s Public Transportation Emergency Relief Program for the 2017 hurricanes. FTA announced on May 31, 2018, that it would allocate about $233 million of appropriated emergency relief funds to 52 transit agencies for response, recovery, and rebuilding projects, with approximately 85 percent of the funds ($198 million) going to Puerto Rico. Most of Puerto Rico’s funds, and around half the funds FTA allocated for response, recovery, and rebuilding ($116 million), will be distributed to San Juan’s rail transit service provider, Tren Urbano (see fig. 2). FTA allocated emergency relief funding to transit agencies based on preliminary cost estimates that the agencies submitted to FTA in damage assessment reports. Transit agencies developed these preliminary cost estimates through field surveys, which are meant to determine the general type and extent of damages. As shown in table 1, FTA allocated funds for various purposes including repairs to rail stations and bus terminals, repair and replacement of vehicles, and repairs to transit buildings and facilities. As previously noted, after FTA allocates funds, transit agencies must submit grant applications with detailed information about each eligible project activity and expense. As of October 2019, 19 transit agencies had submitted grant applications to FTA, and FTA approved and obligated funding for each of the 19 applicants. FTA officials told us they are working with the remaining transit agencies on submitting and finalizing their grant applications. Uncertainty regarding whether FTA will receive an appropriation can lead to FTA grantees’ applying to FEMA for funding since FEMA is the federal government’s primary agency for disaster response and recovery and can fund transit. This situation increases the importance of FEMA and FTA coordination. FTA did not receive an appropriation until roughly 6 months after the first hurricane’s landfall. FTA grantees, unaware of when or whether FTA would receive an appropriation, could apply during this period to FEMA’s Public Assistance Program for funding. Indeed, more than half of FTA grantees that responded to our survey (25 of 44) reported some interaction with FEMA’s Public Assistance Program by the time of our survey (see fig. 3). Fourteen reported reaching the quality assurance step on a grant application—the final step before receiving funds from FEMA. Six transit agencies received FEMA funds. Once FTA received an appropriation, FTA and FEMA instructed transit agencies to work with FTA, rather than FEMA, on funding requests. As a result, some transit agencies that initially worked with FEMA had to begin a new application with FTA. Fourteen FTA grantees in our survey reported spending more than 3 months working on their FEMA application; however, 10 stated that they could use the work from the FEMA application toward their FTA emergency relief application. In addition, most of the transit agencies we interviewed anticipated this issue, noting that FTA or FEMA officials explained the situation to them before FTA received an allocation. After Congress appropriated funds to FTA for the 2017 hurricanes, FTA and FEMA initiated their communication and coordination agreements, including the MOA and the communications protocol, which define coordination activities between the two agencies. Federal agencies, such as FTA and FEMA, that administer programs as a result of a major disaster or emergency, cannot provide funding for losses that have been covered by insurance or other programs, but are not prohibited from awarding funds to any entity that could receive funding from another agency so long as that entity has not yet received these funds and promises to repay any duplicate assistance. FTA’s and FEMA’s communications protocol also states that it may be appropriate for an agency to receive funding from both FTA and FEMA in a situation where the grantee provides both public transportation services and services other than public transportation. Thus, FTA’s and FEMA’s MOA states that the agencies will coordinate to avoid duplicate funding and to ensure a streamlined reimbursement process. When implementing coordination activities such as FTA’s and FEMA’s MOA and communications protocol, federal internal control standards state that management should design control activities to achieve objectives and respond to risks, such as the risk of providing duplicate funding. FTA and FEMA officials informed us of, and provided documentation of, their coordination efforts, such as biweekly conference calls, and email correspondence among staff. For example, when Congress appropriated funds to FTA, FEMA provided FTA a list of agencies that had applied to FEMA for funding. In addition, when FTA reviewed grant applications, FTA staff emailed FEMA staff to inquire whether applicants had already requested funding from FEMA. To avoid delays in processing applications, FEMA and FTA established an agreement that if FEMA did not respond to such requests in 5 days, then FTA could proceed with processing the application. Based on our document reviews, we found that FTA staff also emailed FEMA staff a copy of the final award. Finally, transit agencies applying to FTA for funding were required to certify whether they had received any transit funding from FEMA and that they would reimburse FTA for any federal funds that duplicated funding provided by FEMA. While FTA and FEMA took steps to coordinate, both agencies approved about $35,000 in funding to one applicant for the same expenses. In June 2019, we found a case in which FEMA and FTA both approved roughly $6,000 to repair a light pole at a bus stop in Collier County, Florida. Specifically, although FEMA had obligated funds to Collier County for the light pole in January 2019, FTA awarded funds for the same light pole in April 2019. One month prior to FTA’s award to Collier County, we notified FTA that Collier County had indicated in our survey that it had been in contact with FEMA. Subsequently, FTA staff twice emailed FEMA staff to inquire as to whether Collier had requested funds from FEMA, but FEMA staff did not respond. Per their agreement, FTA moved the application forward after receiving no response from FEMA within the 5-day timeframe. FTA awarded the funding to Collier County in April 2019. After we notified FTA and FEMA that they both appeared to have awarded funds for the same expense, FEMA de-obligated the funds for Collier County. In addition, FEMA conducted an additional review and found that both agencies had also approved $29,000 in funding for repairs to a transit facility in Collier County. FEMA officials stated they were in the process of de-obligating those funds as well. Although both agencies awarded funds to Collier County, the County had not yet executed the FTA grant or drawn down any of the funds. FTA and FEMA officials noted that both agencies can face challenges in identifying transit expenses submitted to both agencies. For example, FTA may be unaware of transit agencies receiving FEMA funds if these agencies are not direct recipients of such funds, but rather receive funds through a larger entity such as a city, county, or state government. Thus, although FEMA provides FTA with a list of entities that applied for FEMA funds, the list may only show a county’s name, rather than the name of a transit agency. In addition, while FTA also asks applicants whether they have received FEMA funds, applicants may be unaware of the status of their FEMA reimbursement. For example, officials from Collier County’s public transit department told us they were unaware that FEMA had obligated funding for their transit expenses until May 2019 (one month after the FTA award), because it took several months for the funding from FEMA to be processed at the state and county level. While FTA officials shared proposed and final awards with FEMA, we identified 10 cases, including Collier County, in which FEMA officials did not respond within the established 5-day time frame. When we asked why FEMA did not respond within the 5-day time frame, FEMA regional staff stated that the responsible person had since left that office. However, officials noted challenges they face identifying transit expenses contained within applications sent to FEMA by larger entities that may contain hundreds of pages, while at the same time processing a large number of applications related to the hurricanes. Specifically, in order to identify transit expenses within an application, FEMA staff may need to search these hundreds of pages using various transit-related word searches. For example, according to FEMA officials, Collier County currently has a total of 126 active ongoing and obligated projects and 86 inactive projects that were either withdrawn or determined ineligible. After we notified FTA and FEMA that they had approved funding to Collier County for the same expense, both agencies took steps to limit the potential for duplicate funding in future awards. As noted above, FEMA conducted an additional review of applications for which FEMA had not responded to FTA’s inquiries within the 5-day time frame and identified the $29,000 for transit facility repairs that FTA and FEMA both approved for Collier County. In addition, FTA officials updated their internal grants guidance to indicate that FTA staff should not process an application if FEMA has not responded and FTA has reason to believe there may be a potential for duplicate funding (for example, the recipient notifies FTA that it had previously worked with FEMA to reimburse transit expenses). In such cases, FTA may only proceed after FEMA has replied in writing that they have not identified any expenses in the FTA grant that are also in a FEMA grant, or, if FEMA does identify duplicate funding, after one agency removes such expenses from their grant to the recipient. In 2014, we noted that evaluating and reporting the results of collaborative efforts can identify areas for improvement and recommended that FTA and FEMA establish specific guidelines to monitor, evaluate, and report the results of collaborative efforts. FTA and FEMA implemented this recommendation and committed to jointly monitoring, evaluating, and reporting on the effectiveness of the agencies’ collaboration following future events in which both agencies provide funding. In addition, FTA and FEMA took action to address the duplicate award of funding we identified in our review. Nonetheless, FEMA staff continue to face challenges identifying transit expenses within applications submitted by larger entities, and FTA may be unaware of whether transit entities are included in such applications. Without identifying and implementing systematic measures to detect duplicate expenses, FTA and FEMA are at risk of awarding funds for the same expenses. Given that FTA may not receive an appropriation until months after a disaster, transit agencies will continue to submit applications to FEMA when it is unclear whether Congress will provide funding to FTA. This underscores the importance of FTA’s and FEMA’s coordination to avoid providing duplicate funding. FTA and FEMA have taken important steps to coordinate, including establishing an MOA and communications protocol that outline how FTA and FEMA staff should share information. Although FEMA and FTA both approved a relatively small amount of funding for the same expenses in Collier County, the issues that contributed to this outcome involve a risk of providing duplicate funding in the future. FTA took steps to strengthen its processes after we identified this duplicate funding, and FEMA conducted additional retroactive reviews to identify any additional duplicate funding. However, FEMA will continue to face challenges in identifying transit expenses when they are included in the application of a larger entity such as a city, county, or state government. Moreover, FTA may continue to be unaware when transit entities are included in FEMA applications. FEMA and FTA have committed to monitor, evaluate, and report the results of collaborative efforts on an ongoing basis. Without identifying and addressing the factors that contributed to duplicate funding in the federal response to the 2017 hurricanes, FTA and FEMA will continue to face the risk that both agencies will approve funding for the same expense in the future. We are making two recommendations, including one to DOT and one to DHS. The Secretary of Transportation should direct the Administrator of FTA to identify and develop controls, such as methods to more easily identify transit expenses within applications submitted by larger entities, such as a city, county, or state government, to address the risk of duplicate funding. (Recommendation 1) The Secretary of Homeland Security should direct the Administrator of FEMA to identify and develop controls, such as methods to more easily identify transit expenses within applications submitted by larger entities such as a city, county, or state government, to address the risk of duplicate funding. (Recommendation 2) We provided a draft of this report to DOT and DHS for review and comment. We received written comments from DOT and DHS that are reproduced in appendixes I and II. In comments, reproduced in appendix I, DOT concurred with our recommendation. DOT described some of the steps that FTA has taken to coordinate with FEMA, which we note in our report, such as updating its procedures to ensure that an FTA grant does not contain any expenses for which the applicant may have previously requested reimbursement. We continue to believe FTA would benefit from identifying additional internal controls to address the risk of duplicate funding, particularly since FTA and FEMA may still face challenges identifying entities that have applied to both agencies for funding. In comments, reproduced in appendix II, DHS concurred with our recommendation. DHS stated that FEMA is enhancing its Public Assistance Grants Manager System to address the risk of duplicate funding we identified in our report. This includes implementing a new functionality for data exporting, sorting, and filtering to better identify transit-related damages and improved tracking to identify projects that have received FTA funding. DHS estimates these improvements will be completed September 30, 2020. DOT and DHS both provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, the Secretary of the Department of Homeland Security, the Administrator of FTA, the Administrator of FEMA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In addition to the contact named above, Steve Cohen (Assistant Director); Crystal Huggins (Analyst in Charge); Matt Cook; Christopher Currie; Danielle Ellingston; Susan Irving; Kathryn Godfrey; Janet McKelvey; Cheryl Peterson; Brenda Rabinowitz; Malika Rice; Amy Rosewarne; Rebecca Shea; Joe Thompson; Matthew Valenta; and Elizabeth Wood made key contributions to this report.", "summary": "In August and September 2017, Hurricanes Harvey, Irma, and Maria made landfall in Texas, Florida, the U.S. Virgin Islands, and Puerto Rico, causing hundreds of millions of dollars in damage to public transit facilities. Access to transit plays an important role in a community's post-disaster recovery. FTA has primary responsibility for providing disaster assistance funding to transit agencies if it receives an appropriation from Congress. If FTA does not receive an appropriation, transit agencies can apply to FEMA for funding. GAO was asked to evaluate the federal government's response and recovery efforts related to the 2017 hurricanes. This report provides information on FTA's emergency relief allocations and examines FTA's and FEMA's coordination. GAO reviewed FTA's allocation of emergency relief funds; conducted site visits to Texas, Florida, and Puerto Rico; obtained survey responses from 44 of 52 transit agencies; and interviewed and reviewed documentation from FTA and FEMA officials. In response to hurricanes in 2017, the Federal Transit Administration (FTA) announced in May 2018 that it would allocate about $233 million of appropriated emergency relief funds to 52 transit agencies for response, recovery, and rebuilding projects, with most of that funding going to Puerto Rico ($198 million). Most of Puerto Rico's funds, and around half the funds FTA allocated ($116 million), will be distributed to one transit system—Tren Urbano—San Juan's rail-transit service provider (see figure below). While FTA and the Federal Emergency Management Agency (FEMA) shared information and coordinated efforts, both agencies still approved about $35,000 to one applicant for the same expenses. GAO found that FTA awarded a grant in April 2019 that included expenses for which FEMA had already obligated funds in January 2019. Although FTA contacted FEMA prior to the award to inquire whether the applicant had received FEMA funding, FEMA did not respond within 5 days, and per an agreement between FTA and FEMA, FTA processed the application. After GAO identified the duplicate funding, FTA and FEMA took steps to limit the potential for duplicate funding; FTA, for example, changed its policy of moving applications forward after 5 days if FEMA does not respond. FTA and FEMA officials noted challenges they face in identifying transit expenses in the applications they receive. For example, they may be unaware that a transit agency received FEMA funds if it received those funds through a larger entity such as a city, county, or state government. Although the amount of funding FEMA and FTA approved for the same expenses was relatively small, without addressing these challenges, FTA and FEMA will continue to face the risk that both agencies will approve funding for the same expense in the future. GAO recommends that FTA and FEMA identify and develop controls, such as methods to more easily identify transit expenses within larger applications, to address the risk of duplicate funding. The Department of Transportation agreed with the recommendation and noted steps FTA has taken to address it. However, GAO believes FTA would benefit from identifying additional internal controls to address the risk of duplicate funding. The Department of Homeland Security agreed with the recommendation and outlined steps FEMA plans to complete in 2020.", "document_type": "gao"}
{"report": "The majority of Americans receive their health coverage through private health insurance, either by purchasing health coverage directly or receiving coverage through their employer. Many of those with private coverage are enrolled in plans purchased from state-licensed or state- regulated issuers. Others are covered by plans where their employer sets aside funds to pay for employee health care, known as self-funded plans. In general, those who obtain private health coverage do so in one of three market segments: individual, small group, or large group. Enrollees in the individual market purchase private health insurance plans directly from a state-regulated issuer—not in connection with a group health plan. In the small group and large group markets, enrollees generally obtain health insurance coverage through a group health plan offered through a plan sponsor (typically an employer). Health benefits commonly include plan design features that require enrollees to pay for a portion of their health care, limit the amount or number of treatments enrollees can receive, and limit the scope or duration of treatments that enrollees may receive. Prior to the implementation of the MHPAEA, health plans offered through employers covering MH/SU often used plan design features that were more restrictive or provided lower levels of coverage for MH/SU benefits than for medical/surgical benefits. For example, prior to MHPAEA, an employer’s plan could cover unlimited hospital days and outpatient office visits and require 20 percent coinsurance for outpatient office visits for medical/surgical treatment while, for MH/SU, that same plan could cover only 30 hospital days and 20 outpatient office visits per year and impose 50 percent coinsurance for outpatient office visits. Congress passed MHPAEA in 2008 to help address discrepancies in health care coverage between mental illnesses and physical illnesses. MHPAEA both strengthened and broadened federal parity requirements enacted in 1996, including extending parity to cover the treatment of substance use disorders. MHPAEA requires coverage for MH/SU services—when those services are offered by group health plans sponsored by large employers (generally employers with more than 50 employees)—be no more restrictive than coverage for medical/surgical services. PPACA extended MH/SU parity requirements to individual insurance plans and some small group health plans. See figure 1 for a timeline of the laws and regulations establishing federal parity requirements and the types of plans affected. In general, MHPAEA requires that the financial requirements and treatment limitations imposed on MH/SU benefits cannot be more restrictive than the predominant financial requirements and treatment limitations that apply to substantially all medical/surgical benefits. Financial requirements. The most common types of financial requirements include: (1) deductibles, which are required payments of a specified amount made by enrollees for services before the issuer begins to pay; (2) copayments, which are payments made by enrollees and are a specified flat dollar amount—usually on a per-unit-of-service basis—with the issuer reimbursing some portion of the remaining charges; (3) coinsurance, which is a percentage payment made by enrollees after the deductible is met and until an out-of-pocket maximum is reached; and (4) out-of-pocket maximums, which are the maximum amounts enrollees have to pay per year for all covered medical expenses. Quantitative treatment limitations (QTL). QTLs are treatment limitations that can be expressed numerically, such as annual, episode, and lifetime day and visit limits. For example, QTLs include annual limits on the number of office visits an enrollee can make for a certain condition and lifetime limits on the coverage of benefits for a certain type of treatment. Non-quantitative treatment limitations (NQTL). NQTLs are non- numerical limitations on the scope or duration of MH/SU services. Common NQTLs include (1) medical management standards that limit or exclude benefits based on medical necessity or medical appropriateness, or based on whether the treatment is experimental or investigative; (2) refusal to pay for higher-cost therapies until it can be shown that lower cost therapy is not effective—known as fail first or step therapy protocols; (3) exclusions based on failure to complete a course of treatment; (4) standards for providers to be admitted to participate in a network, including the factors used to set provider reimbursement rates; and (5) requiring pre-authorization of services—the requirement that an enrollee receives prior approval for care. The MH/SU parity regulations established a two-part analysis to determine if the financial requirements or QTLs in a plan are in compliance with MH/SU parity requirements. The first test determines if a particular type of financial requirement or QTL (such as a copay) applies to substantially all medical/surgical benefits in the relevant classification of benefits (e.g., inpatient in-network or outpatient out-of- network). Generally, a financial requirement or QTL is considered to apply to “substantially all” medical/surgical benefits if it applies to at least two- thirds of the medical/surgical benefits in the classification, according to the regulations. Once the first test is met, the second test checks for parity in the level or magnitude of the requirement (e.g., copay of $15 or $20 or treatment limit of 21 or 30 inpatient days per episode). Specifically, by regulation, the financial requirement or QTL cannot exceed the predominant level—that is, the level that applies to more than half of the medical/surgical benefits subject to the financial requirement or QTL in the classification. For example, if at least two-thirds of outpatient, in- network, medical/surgical benefits are subject to a copay, and 75 percent (i.e. more than half) of outpatient, in-network visits involving medical/surgical benefits are subject to a copay of $30, the copay for outpatient, in-network visits involving MH/SU benefits cannot exceed $30. The MH/SU parity regulations extended parity requirements to NQTLs and establish a different test for assessing parity of NQTLs between medical/surgical and MH/SU benefits. Under the regulations, a plan generally cannot apply an NQTL on an MH/SU benefit unless—both as written and in operation—it is comparable to and applied no more stringently than the NQTL applied to medical/surgical benefits. According to guidance issued by HHS, DOL, and Treasury, the NQTL analysis in the regulations focuses on the underlying factors (such as processes, strategies, and evidentiary standards) used to apply the NQTL and ensuring there are not arbitrary or discriminatory differences in how a plan or issuer applies those factors to MH/SU benefits as compared to medical/surgical benefits. HHS, DOL, and Treasury share joint oversight responsibilities for certain federal laws applicable to private health coverage, including MHPAEA. The oversight of plans and issuers for compliance with MHPAEA is split between the states, HHS, DOL, and Treasury, depending on the type of coverage and whether the plan is self-funded or fully insured. Individual and fully insured group plans sold by issuers. States have primary responsibility for regulating insurance, and health insurance products sold within a state must meet both federal and state requirements, including MH/SU parity requirements. States oversee health insurance sold by issuers (1) in the individual market, where individuals purchase private health insurance plans directly from an issuer or through an exchange; and (2) in the group market, where a plan sponsor (typically an employer) purchases coverage from an issuer. Of the estimated 216 million Americans who were enrolled in private health insurance in 2016, the estimated enrollment in these state-regulated markets was 17.3 million in the individual market, 14.2 million in the small group market, and 42.9 million in the large group market. State oversight of health insurance applies only to fully insured health plans offered by state-licensed issuers. Because self-funded plans are financed directly by the plan sponsor, these plans are generally not subject to state law or oversight. With respect to health insurance issuers selling products in the individual and fully insured group market, HHS has primary enforcement authority over MH/SU parity requirements in two instances: (1) when a state notifies HHS that it does not have the authority to enforce MH/SU parity requirements or the state notifies HHS that it is not otherwise enforcing the requirements, or (2) when HHS determines the state failed to substantially enforce MH/SU parity requirements. States falling into these categories are known as direct enforcement states, and, in these states, the Centers for Medicare & Medicaid Services (CMS) within HHS assumes the responsibility for directly enforcing federal MH/SU parity requirements and other federal health laws covered by PPACA with respect to issuers. CMS is currently responsible for enforcing MH/SU parity requirements against issuers in four states: Missouri, Oklahoma, Texas, and Wyoming. While CMS enforces MH/SU parity requirements and other PPACA requirements for these direct-enforcement states, these states maintain enforcement authority over issuers for state-level regulatory requirements. For the estimate of overall enrollment in private health plans in 2016, see U.S. Census Bureau, “Coverage Numbers and Rates by Type of Health Insurance: 2013 to 2016,” Current Population Survey, 2014 to 2017 Annual Social and Economic Supplements, table 1. products they offer, DOL oversees the plans themselves for compliance through its Employee Benefits Security Administration (see table 1). DOL does not have the authority to enforce MH/SU parity requirements directly against issuers to correct noncompliant health policies that are designed, marketed, and sold by the issuer to numerous employers for the purposes of offering health plans to their employees. DOL has primary authority for overseeing compliance with MH/SU parity requirements for self-funded, private employer-sponsored group plans, as states generally do not have authority over these plans. The Internal Revenue Service (IRS) within Treasury is authorized to impose an excise tax on employers that sponsor private group plans that are not in compliance with MH/SU parity requirements. Similarly, HHS has primary authority for MH/SU parity requirements over employer-sponsored plans for state and local governments—known as non-federal governmental plans. Within HHS, CMS oversees both fully insured and self-funded non-federal governmental plans. In 2017, an estimated 13 million state and local government employees enrolled in these plans. Sponsors of self-funded, non-federal governmental plans may elect an exemption from, or “opt-out” of, certain federal health care requirements, including MH/SU parity requirements. If a plan elects to opt-out of MH/SU parity requirements, CMS also reviews the plan’s election to ensure they meet requirements for doing so. Through our survey and interviews with officials from the three selected states, we found that nearly all states conduct some type of review for MH/SU parity compliance as part of their oversight of issuers selling fully insured large and small group plans and individual plans. The reported type and frequency of these reviews vary, particularly for the reviews conducted after consumers enroll in plans. A product is a discrete package of health insurance coverage benefits that are offered using a particular product network type (e.g., health maintenance organization or preferred provider organization) within a service area. Issuers then sell plans to consumers by pairing health insurance coverage benefits with a particular cost sharing structure, provider network, and service area. The only two states that did not report that they conduct reviews for MH/SU parity compliance before products are approved for sale in their states are Missouri and Wyoming, which are two of the four states where CMS is directly enforcing MH/SU parity requirements. In the four direct enforcement states, CMS conducts reviews of issuer policies and documentation for compliance with federal MH/SU parity requirements before products are approved for sale in the states. The two other states—Texas and Oklahoma—reported in our survey that they review products for state-level MH/SU parity compliance; however, CMS maintains primary authority for reviewing products for compliance with federal MH/SU parity requirements in those states. consumers enroll in a plan. Additionally, according to CMS officials, the HHS mental health parity tool is not designed to facilitate an evaluation of NQTLs due to the nature of reviewing NQTLs. State oversight after consumers enroll in plans. In addition to the review they conduct prior to consumers enrolling in plans, 27 states reported in our survey they have conducted some type of review related to MH/SU parity after consumers enroll. The types of reviews states conduct vary. These review types include: targeted reviews based on consumer complaints or other information, random audits, and conducting broad routine reviews of issuers’ compliance with state and federal health insurance laws—called market conduct examinations. Through our interviews of states and stakeholders we identified additional enforcement activities some states are using to assess the issuer compliance with MH/SU parity requirements after consumers enroll. These reviews and additional enforcement activities are described below: Conducting targeted reviews. Twenty states reported in our survey that they had conducted a targeted review that focused on specific issuers or particular MH/SU parity compliance concerns, while other states reported they had never performed such a review. Consumer complaints were most commonly identified as the reason—at least in part—that these 20 states conducted targeted reviews to assess compliance with MH/SU parity requirements. Thirty-eight states reported in our survey that they track MH/SU parity complaints, which can be submitted by consumers, providers, or advocates. For example, after receiving consumer complaints, Massachusetts examined the accuracy of the information on behavioral health services—services that address mental health or substance use issues—contained in issuers’ provider directories and compared this to the accuracy of medical/surgical provider information in a 2018 report. Officials from another state told us they frequently use targeted reviews in response to complaints because these focus on a specific issue, rely on more recent data, and are less time consuming than more comprehensive market conduct examinations that review an issuer’s compliance with all state health requirements. States reported additional reasons for starting targeted reviews related to MH/SU parity requirements, including reviews initiated after receiving referrals from other departments, reviews driven by predictive analytics or market analyses, and reviews in response to media attention. In 2017 and 2018, the frequency of receiving MH/SU parity-related complaints and conducting targeted reviews varied across states. (See table 2). Market conduct examinations. Nearly all states conduct market conduct examinations and states have not routinely included a review for MH/SU parity compliance as part of the examinations. Market conduct examinations are a review of an insurer’s marketplace practices. The examination is an opportunity for the state to verify data provided by the insurer and to confirm that companies’ internal controls and operational processes result in compliance with state laws and regulations. Eighteen states reported in our survey that they routinely conduct market conduct examinations (ranging from every 3 or 5 years), and, of those, nine states reported that they usually or always include a review of MH/SU parity compliance. Twenty-nine states reported that their market conduct examinations are not routine; they are conducted on an as-needed basis or in response to risk factors, such as market analysis or complaints. In order to assist states’ ongoing oversight of MH/SU parity compliance, NAIC developed guidance on MH/SU parity for its Market Regulation Handbook, which most states use to guide their market conduct examinations, an NAIC official told us. The guidance includes a data collection tool for mental health parity analysis. While the guidance was finalized in August 2019, an NAIC official told us most states were already using the guidance to conduct their market conduct examinations while it was in draft form. State-wide comprehensive reviews of issuers. Officials we interviewed from two of the three selected states told us they have conducted reviews of all issuers in their state as part of their oversight of MH/SU parity compliance after consumers enroll in plans. For example, as requested by its state legislature, Maryland conducted three annual MH/SU parity surveys with the state’s major issuers. Maryland officials told us the first two surveys focused on MH/SU parity compliance in the issuers’ plan documentation, and the last survey assessed compliance in plan practices and operations. Maryland officials told us the review of all issuers in the state will give them a baseline understanding of issuer compliance with MH/SU parity requirements reviewed. Officials from Washington told us they are using a CMS grant to evaluate issuer claims data and to understand issuers’ NQTLs in operation, which officials say will enable them to identify statewide MH/SU parity-related concerns. Annual compliance reporting. At least eight states have established annual requirements for issuers to demonstrate their MH/SU parity compliance through data reporting or self-certifications, according to officials from one of the three selected states in our review and a provider organization. To fulfill the states’ requirements, issuers submit information such as the percentage of claims paid for in- network and out-of-network MH/SU services compared to those paid for medical/surgical services and the number of consumers denied prior authorizations for MH/SU services. For example, in 2012, Massachusetts began requiring issuers to submit annual reports certifying that their plans comply with federal and state MH/SU parity requirements and instructing issuers to compare denials of care for MH/SU and medical/surgical services, among other things. These certifications must be signed by the issuer’s chief executive officer and chief medical officer, which Massachusetts officials told us ensures that issuer leadership is aware of the MH/SU parity requirements. Additionally, an official from NAIC told us that NAIC now includes data reporting requirements related to MH/SU parity, such as requiring information on prior authorizations and denials of care, in its annual nationwide collection of issuers’ post-enrollment information. An NAIC official told us states can use these data to compare information on MH/SU and medical/surgical services and examine issuers that operate in multiple states. In their survey responses, 47 states identified enforcement actions they can take if they find, through a review, that an issuer violated MH/SU parity requirements. States reported that these enforcement actions include: financial penalties, license termination, orders to pay claims or interest, and orders to pay restitution. However, an official from NAIC told us that in the majority of cases, issuers voluntarily come into compliance after state regulators identify an issue or parity violation. Both DOL and CMS oversee employer-sponsored group plans to ensure their compliance with MH/SU parity requirements. Specifically, the agencies conduct what are known as targeted reviews after consumers enroll in these plans. The agencies initiate these reviews after they receive complaints or other information regarding possible noncompliance with either MH/SU parity requirements or other, unrelated issues, such as a plan failing to provide a document explaining the health benefits covered. Unlike states, the agencies do not conduct any type of review of employer-sponsored plans before consumers enroll and do not have the authority to conduct such a review, according to DOL and CMS officials. DOL oversight. DOL’s targeted reviews are triggered by inquiries, including complaints, or other information that identifies possible noncompliance with MH/SU parity requirements or other applicable federal health care laws. These targeted reviews can also originate from additional techniques DOL uses to target plans for review, such as reviewing bankruptcy filings or financial and operational information filed annually by employers. According to DOL’s enforcement manual, DOL investigators generally identify the reasons for starting each review, obtain relevant information from the plan or issuer, and conduct a full review of compliance with applicable federal health care laws. These reviews which are performed by DOL’s 10 regional offices can focus on specific private, employer-sponsored group plans, service providers (such as third party administrators), or issuers; however, DOL does not have the authority to take direct enforcement actions against issuers for violations of MH/SU parity requirements. DOL reported that it completed 302 reviews of private, employer- sponsored group plans that included a review for compliance with MH/SU parity requirements in fiscal years 2017 and 2018. According to DOL officials, these reviews can take 2 to 3 years to complete and investigators follow an extensive compliance checklist to conduct these reviews. The checklist includes specific questions to help determine compliance with all applicable requirements, including a section with questions on MH/SU parity. Because investigators complete the compliance checklist for every plan level health investigation, reviews not triggered by a parity complaint may still uncover a parity violation. For example, DOL might review a private employer-sponsored group plan in response to a consumer complaint about how long the plan covered a hospital stay for a mother and her newborn. The review would include a review of compliance with the related law (the Newborns’ and Mothers’ Health Protection Act of 1996), MH/SU parity-related requirements, and all other applicable federal health care requirements. Nearly all DOL reviews that assess compliance with MH/SU parity requirements originate from sources unrelated to MH/SU parity, including complaints or other information about potential noncompliance with other federal health care laws and DOL reviews of the annual financial and operational information filed by employers, based on data provided by DOL on the reasons targeted reviews were opened. DOL received few MH/SU parity complaints and opened few reviews based on a potential MH/SU parity violation, compared to complaints related to other federal health requirements, in fiscal years 2017 and 2018 (see table 3). When DOL identifies a violation of MH/SU parity requirements through one of its reviews, investigators first seek to bring the private employer- sponsored group plan or issuer into compliance voluntarily, according to DOL officials. When that is not possible, DOL can sue the plan for equitable relief, which can result in the plan being required to reimburse members whose claims were improperly denied. DOL can also request that the Treasury levy an excise tax on the non-compliant private employer-sponsored group plan, but DOL officials noted that the excise tax goes to the Treasury rather than toward payment of claims for plan members, and DOL’s focus is on obtaining payment of claims. DOL officials told us they have never referred a plan to IRS to levy an excise tax based on an MH/SU parity violation. The 21st Century Cures Act requires DOL to conduct an audit of a private employer-sponsored group plan when DOL has identified five or more MH/SU parity violations; however, DOL officials told us the use of this authority has not been triggered, as of October 2019. DOL identified audit resources challenges faced by the agency given the universe of plans DOL oversees and reported that the agency is taking steps to better leverage its resources through targeted exams in its September 2019 enforcement report to Congress. Specifically, DOL reported that DOL has less than one investigator for every 12,500 employee benefit plans the agency oversees, including private health, pension, life, and disability insurance. In light of these challenges, DOL officials said they are focusing their targeted reviews on issuers and other service providers to obtain voluntary corrections whenever possible so they can address noncompliance across multiple private employer- sponsored group plans. To date, they have completed at least two investigations at the issuer level and brought an issuer into voluntary compliance after one investigation identified MH/SU parity noncompliance affecting over 4,000 private, employer-sponsored group plans and 7 million consumers. According to DOL officials, focusing on issuers will result in their opening fewer targeted reviews than in prior years, but will have more meaningful results. DOL officials also noted other efforts underway to assist in MH/SU parity oversight. For example, the DOL’s Kansas City Regional Office has convened a task force that focuses on parity in opioid use disorder treatment coverage. DOL officials told us they require senior advisors in each of the 10 regions to identify trends in the types of violations DOL identifies and to identify when a violation could be happening at the issuer level, rather than the individual employer-sponsored group plan level. CMS oversight. CMS oversight of employer-sponsored, non-federal governmental plans for compliance with MH/SU parity requirements consists of targeted reviews. Like DOL, these targeted reviews originate from complaints or information about noncompliance—about MH/SU parity or issues with other federal health care laws. The reviews are used to assess compliance with all applicable health requirements, and CMS officials told us CMS has broad authority to review or request information as a part of these reviews. However, according to CMS officials, CMS has limited authority to review or request information from these plans outside of these targeted reviews. Specifically, CMS officials said CMS does not have the authority to conduct random audits, reviews, or examinations of employer-sponsored, non-federal governmental plans, or to require the plans to provide documentation to demonstrate compliance with MH/SU parity requirements. CMS officials also said they do not have the authority to review employer-sponsored, non-federal governmental plans for compliance with MH/SU parity requirements prior to enrollment. While large, self-funded, employer-sponsored, non-federal governmental plans may opt-out of MH/SU parity requirements and certain other federal health requirements, CMS may identify MH/SU parity noncompliance if these plans did not properly opt-out. CMS officials told us that they review documentation for all plans that elect to opt out of MH/SU parity requirements to ensure it was properly submitted. If CMS finds a plan may have opted-out incorrectly, CMS officials said they can request additional information from the plan and can ultimately decide the opt-out was invalid. CMS reported that it closed five reviews related to MH/SU parity in fiscal years 2017 and 2018. Two targeted reviews originated from MH/SU parity complaints and three reviews were related to plans opting-out of MH/SU parity requirements. CMS officials told us that they received four complaints related to MH/SU parity in employer-sponsored, non-federal governmental plans in fiscal years 2017 and 2018. The officials told us all four complaints resulted in targeted reviews, two of which were ongoing as of September 2019. When CMS identifies MH/SU parity noncompliance through one of these targeted reviews, the agency takes one of several actions: working with the plan to implement a corrective action plan; initiating a full market conduct examination of the plan; or imposing civil money penalties. Like DOL, the 21st Century Cures Act requires CMS to audit an employer- sponsored, non-federal governmental plan or issuer when CMS has identified noncompliance five or more times. According to CMS officials, as of November 2019, the use of this audit authority has not been triggered. Under DOL’s and CMS’s oversight through targeted reviews, self-funded employer-sponsored group plans do not undergo review for compliance with MH/SU parity requirements unless the agencies receive complaints or other information about potential noncompliance with an applicable federal health care law, or the review is opened as a result of a targeting technique unrelated to MH/SU parity—such as bankruptcy filing review. Relying on the receipt of such information to trigger a targeted review of MH/SU parity is a concern given the low number of complaints DOL receives related to MH/SU parity when compared to other federal health requirements. For example, as we have noted, DOL received 129 complaints in fiscal year 2018, and most of the noncompliance with MH/SU parity requirements DOL identified was found through reviews triggered by complaints and information unrelated to MH/SU parity in fiscal years 2017 and 2018, based on our review of DOL data. Further, as discussed later in this report, consumer advocates have noted that there is a lack of consumer awareness about MH/SU parity requirements, which may result in fewer complaints than would otherwise be made if consumers understood the requirements. Federal internal control standards state that agencies should identify, analyze, and respond to risks related to achieving their defined objectives. DOL has stated that its defined objective is the full implementation of MH/SU parity requirements through vigorous compliance assistance and enforcement. HHS has stated that it is committed to enforcing MH/SU parity requirements through CMS and to providing the sponsors of employer-sponsored, non-federal governmental plans the information needed to ensure that the plans are fully compliant with MH/SU parity requirements. DOL and CMS officials told us they have not completed any statistical analysis or study regarding the effectiveness of their targeted review approach to MH/SU parity compliance, nor whether this approach increases the risk of noncompliance. Specifically, they have not analyzed whether relying on targeted reviews alone increases the risk of noncompliance with MH/SU parity requirements in employer-sponsored group plans. The risk of noncompliance may be increased because incentives for plans to comply are limited when investigations are initiated only after receiving complaints or information about noncompliance. DOL and CMS officials also said they have not analyzed whether additional strategies, such as the attestation or issuer documentation requirements used by some states, would reduce the risk of noncompliance. For example, such an evaluation could assess whether a sample of health plans reviewed for compliance identified similar types of noncompliance as those identified when plans were reviewed in response to MH/SU parity complaints. According to officials from a provider organization, one such strategy to improve compliance would be to require issuers or plans to affirm that (1) their plans comply with MH/SU parity requirements and (2) they have documentation showing that they analyzed their plans for compliance. According to these officials, requiring this documentation from plans and issuers can increase compliance, even if there is a low probability that a plan will be audited. DOL and CMS officials told us that they currently do not have the authority to conduct oversight activities of this type. Specifically, they told us that for self-funded private or non- federal governmental employer-sponsored group plans they do not currently have the authority to: (1) review plans for compliance with MH/SU parity requirements before coverage is offered to consumers, (2) require plans to develop documentation to demonstrate compliance with MH/SU parity requirements, and (3) monitor or examine plans for compliance with MH/SU parity requirements outside of an investigation. Without evaluating the effectiveness of their targeted review approach, DOL and CMS do not know whether their oversight is adequate for ensuring compliance with MH/SU parity requirements, or whether they need to adopt additional strategies and seek new authorities, if needed. States, DOL, and CMS identified some plan or issuer noncompliance with specific MH/SU parity requirements in 2017 and 2018 through their various oversight efforts. Specifically, after consumers enrolled in plans: Seventeen of the 51 states that responded to our survey reported identifying noncompliance a total of 254 times among issuers of individual plans and fully insured, employer-sponsored group plans. DOL reported identifying noncompliance 113 times among private, employer-sponsored group plans or the issuers of these plans. CMS reported identifying noncompliance two times among employer- sponsored, non-federal governmental plans. Both states and DOL most commonly identified noncompliance with MH/SU parity NQTL requirements. Eleven of the 14 states that provided information on the types of MH/SU parity noncompliance in our survey reported that the noncompliance they found was related to NQTLs half the time or more. Similarly, DOL reported that 55 percent of noncompliance the agency found in fiscal year 2018 was related to NQTLs, while 40 percent was related to financial requirements or QTLs. Through our review of DOL letters informing plans of noncompliance, we found that the most common types of noncompliance with MH/SU parity requirements were related to (1) copayments or coinsurance, such as a higher copayments for MH/SU treatment than those generally applied to equivalent medical/surgical treatment (a financial requirement); (2) prior authorizations, such as requiring approval in advance for MH/SU treatment but not requiring it for equivalent medical/surgical treatment (an NQTL); and (3) the total number of treatments allowed, such as a limit on inpatient hospital days for MH/SU treatment that is not applied to equivalent medical/surgical treatment (a QTL). The scope of noncompliance with MH/SU parity requirements identified by states, DOL, and CMS in 2017 and 2018 varied—both in terms of the number of consumers affected and the steps needed to come into compliance. While MH/SU parity requirements apply to plans, regulators may identify and seek to correct noncompliance in the underlying health policies that issuers use to design, market, and sell as health plans to numerous employers. For example, DOL letters show one particularly widespread violation affected more than 7 million enrollees. Most plans or issuers resolved the noncompliance identified by regulators voluntarily. For example, DOL officials told us that plans or issuers resolved all instances of noncompliance voluntarily. Nine states reported in our survey taking a total of 20 enforcement actions to bring plans or issuers into compliance in 2017 and 2018. See table 4 for examples of noncompliance and steps required to come into compliance. Additionally, while the literature we reviewed suggested that the individual, small group, and large group plans assessed by the studies were generally compliant with MH/SU parity requirements assessed by the studies, the studies identified some noncompliance or possible noncompliance. For example: One case study found that, in 25 percent of the total products offered on two state-based health insurance exchanges between October 2013 and March 2014—the first year of operation for the exchanges established by PPACA—the financial requirements and certain NQTLs reviewed appeared to be noncompliant with MH/SU parity requirements. The study also found variation in the types of noncompliance in each of the states. The case study concluded that on one exchange more than half the products appeared inconsistent with MH/SU parity requirements, particularly the NQTLs reviewed; on the other exchange, 11 percent of the products had a financial requirement that violated MH/SU parity requirements. One study found that 18 percent of benchmark plans were not compliant with MH/SU parity requirements for substance use disorder benefits specifically. For example, five plans had limits on the number of inpatient and/or outpatient visits for substance use disorder services only. (See app. II for additional information about the studies we reviewed.) Each of the studies we reviewed were limited because they evaluated only selected requirements, with the authors of four studies noting there was insufficient information in plan documents to evaluate additional MH/SU parity requirements. As such, none of the studies could determine the extent of issuer compliance with all MH/SU parity requirements. A 2018 survey of employer-sponsored group plans suggests that there could be employer-sponsored plans that have not come into compliance with MH/SU parity requirements. Specifically, this nationally representative survey of employers that offer employer-sponsored group plans found that 61 percent of large and midsized employers reported they had taken steps to address compliance with MH/SU parity requirements—such as reviewing plan documents. An additional 13 percent of large and midsized employers reported that they planned to take action to come into compliance and some plans may have already been in compliance. According to advocacy groups and state and federal officials we interviewed and some of the research we reviewed, the full extent of compliance with MH/SU parity requirements is not known. As NAIC and consumer advocacy stakeholders have reported, regulators often rely on both individual complaints and aggregate consumer complaint statistics to identify problem issuers and problem areas for additional oversight. However, stakeholders from eight consumer advocacy groups told us that complaints are not a good measure of whether MH/SU parity issues exist and do not accurately reflect the number of enrollees facing problems with parity. Further, CMS, DOL, and state officials, as well as stakeholders and researchers, also noted the complexity of assessing plans for MH/SU parity compliance for NQTLs in particular, which may result in inconsistent identification of MH/SU parity violations or the inability to fully assess compliance. Limitations of relying on complaints to trigger enforcement activities. Stakeholders and state officials reported on the limitations of relying on complaints to trigger enforcement activities—which contribute to the challenges in determining the full extent of compliance with parity requirements. Stakeholders from eight consumer advocacy groups told us that if regulators rely on complaints to identify possible noncompliance after consumers enroll in plans, they will not know the full extent of compliance with MH/SU parity requirements. These stakeholders identified several reasons complaints do not accurately reflect the number of consumers facing problems related to plan or issuer compliance with MH/SU parity requirements: Consumers may not be aware of MH/SU parity requirements, such as how to determine if the treatment challenge they are experiencing is a potential parity violation, how to file a parity-related complaint, or which entity they should contact to file a complaint, according to five consumer advocacy stakeholders we spoke to and one professional organization. For example, while a consumer would be aware of a denial for a particular treatment for a mental health condition because the issuer did not consider it to be medically necessary, the consumer could not easily determine if this standard was applied more stringently than to similar medical/surgical benefits and thus signaled a parity issue. Further, in our survey, officials from 21 states reported they do not provide any public information to consumers about MH/SU parity requirements, which may contribute to a general lack of consumer awareness in these states. Consumers may decide not to file a complaint due to the stigma associated with MH/SU treatment, three consumer advocacy stakeholders and state officials in one state told us. One stakeholder also noted that consumers expect substance use disorder services to be treated differently than medical services and are therefore less likely to file a complaint if they receive disparate treatment. Consumers may be hesitant to file a complaint that includes sensitive personal details, such as a mental illness diagnosis, two stakeholders told us. One of these stakeholders told us consumers in need of substance use disorder services in particular may not want to raise a complaint that documents their participation in illegal activities, such as drug misuse. In addition, two stakeholders and state officials in one state stated that individuals or families experiencing an immediate crisis associated with MH/SU conditions may not be well-equipped to navigate the complaint process or wait for a complaint resolution. Providers face barriers helping consumers file complaints or appeals related to MH/SU parity requirements, four consumer advocacy stakeholder groups and one professional organization told us. The barriers identified by these stakeholders include: providers being unable to file complaints on behalf of consumers in some states; the time consuming nature of the appeals or complaint processes; and provider fear that an issuer will drop them from their network if they file a complaint. Two consumer advocacy groups have identified that providers may be in a better position to understand a denial decision and justify a consumer’s need for treatment, but noted that barriers discourage providers from filing an appeal or complaint. One of these consumer advocacy groups reported that providers might be unaware of what issuer actions would violate MH/SU parity requirements. Officials from the three selected states provided examples of specific efforts taken that may address stakeholder identified challenges consumers face in understanding parity requirements and filing related complaints. For example, Maryland officials told us they developed a webinar to help consumers with filing complaints related to substance use disorder treatment. Officials from Massachusetts told us they review for parity violations any complaint related to coverage of mental health- related services, regardless of whether the consumer indicates that the complaint might be a parity violation. While this process is still dependent on a consumer to make a complaint, it does not rely on the consumer having an in-depth understanding of parity requirements for their complaint to be reviewed for potential noncompliance. In light of concerns about consumers not filing complaints, officials from Washington told us their statewide comprehensive review includes an assessment of how issuers implemented state and federal MH/SU parity requirements and aims to help them assist consumers who are not reaching out directly. Additionally, 30 states reported in our survey that they provide public information—such as frequently asked questions or brochures—for consumers about MH/SU parity requirements. Complexity of assessing NQTLs for MH/SU parity compliance. CMS, DOL, NAIC, and state officials, as well as some stakeholders and researchers, identified complexities in assessing NQTLs for compliance with MH/SU parity requirements. As a result, regulators may fail to identify noncompliance, or may not always identify noncompliance, making current numbers on noncompliance with MH/SU parity requirements an unreliable indicator of the extent of noncompliance. Difficult to assess plan implementation of NQTLs. Officials from three states reported in our survey or interviews that it is challenging to determine how an NQTL described in plan documents is actually being implemented and experienced by consumers in practice. This can make it difficult to determine both if noncompliance has occurred and the extent of any noncompliance. Further, some state regulators do not conduct the types of detailed analyses necessary to determine if an NQTL is in compliance with MH/SU parity requirements, according to one consumer advocacy group. Finally, four studies we reviewed identified that researchers were unable to observe the plans’ implementation of NQTLs. Thus, they were unable to draw conclusions about whether or not the way plans implemented the NQTLs complied with MH/SU parity requirements. To address the complexities of these analyses for their own reviews, DOL officials told us that for its targeted reviews of MH/SU parity compliance, DOL uses seasoned investigators, early litigation support, technical guidance from DOL’s regulations office, and outreach to other federal and state agencies. Lack of documentation on medical/surgical NQTLs. A lack of documentation on the factors used to apply NQTLs to medical/surgical benefits makes it difficult for issuers to demonstrate compliance with MH/SU parity requirements, according to two industry officials. They told us that information on NQTLs—such as when to require prior authorization—has to be created for medical/surgical benefits so that the information can then be compared to the application of NQTLs to MH/SU benefits to assess compliance with MH/SU parity requirements. One industry official noted that this poses an additional hurdle when MH/SU benefits are carved out or separately managed from the rest of a health plan. This lack of explicit information about medical/surgical benefits and difficulty drawing parallels between medical/surgical and MH/SU care also makes it difficult for regulators to determine parity compliance, officials from one of the three selected states told us. Lack of resources. Eight states reported in our survey that lack of staff resources, staff training, or clinical expertise are additional challenges to assessing compliance with MH/SU parity requirements. Further, states may hesitate to determine an issuer violated federal MH/SU parity requirements due to a lack of confidence or clarity in applying the federal laws and may cite state laws instead, according to officials from one of our three selected states and a provider organization. Officials from the provider organization told us this could result in an undercount of MH/SU parity violations if a state cites a potential violation of an MH/SU parity requirement as a violation of a state law unrelated to federal MH/SU parity requirements. One state official identified consumer protection laws as an alternative to pursuing possible MH/SU parity requirement violations. Officials from one state told us some state laws have more clear cut standards than federal MH/SU parity requirements, due to the lack of clarity regarding federal MH/SU parity requirements. However, different strategies were used in three states to obtain the needed clinical expertise to review NQTLs, including regular meetings with clinicians from the state mental health department and using grant money to contract with physicians with clinical expertise to help with compliance reviews. HHS, DOL, and Treasury have coordinated on oversight of MH/SU parity requirements by providing support and jointly developing guidance for state regulators, insurance industry officials, providers, and consumers. HHS described several recent and planned coordination activities in its public action plan to improve state and federal coordination of the oversight of MH/SU parity requirements. This plan was required by the 21st Century Cures Act. Recent and ongoing support and coordination activities include: Formal agreements with states. HHS and DOL officials told us they have established formal agreements—such as collaborative enforcement agreements—with states to help coordinate, share information about, or assist states with MH/SU parity enforcement activities. For example, DOL officials told us they have general enforcement and common interest agreements with nearly 40 states that allow them to share information related to MH/SU parity enforcement. HHS officials told us they have collaborative enforcement agreements with six states that allow HHS to intervene if a state’s efforts to bring an issuer into compliance with MH/SU parity requirements are unsuccessful. In response to our survey, state officials reported few formal referrals between the states and HHS or DOL. Informal communication with states. HHS and DOL officials told us that state regulators can contact regional coordinators and individuals in their respective headquarters for assistance with MH/SU parity enforcement outside of formal agreements. HHS and DOL officials told us that referrals of specific complaints are informal and infrequent, noting that if a complainant contacted their office by mistake they would provide the contact information for the appropriate state or federal agency. Technical assistance and outreach. HHS and DOL jointly conduct technical assistance for state regulators and have conducted outreach with stakeholders, including consumers, consumer advocates, providers, issuers, and employers, to improve compliance with MH/SU parity requirements. DOL officials told us that they meet regularly with state regulators and NAIC to provide technical assistance and foster implementation and enforcement coordination. For example, in 2017, HHS and DOL held a commercial market parity policy academy— technical assistance for teams of state officials on strategies to advance MH/SU parity compliance and lessons learned from other states’ implementation efforts. According to the HHS action plan, representatives from 20 states and territories attended. Additionally, DOL held a roundtable discussion with stakeholders to discuss NQTLs, disclosure, and federal-state coordination in January 2019. Grant funding. HHS has also awarded funding, provided by PPACA, to states to help improve oversight of MH/SU parity requirements. In 2016, CMS awarded $9.3 million to 20 states specifically for enforcement and oversight related to MH/SU parity. Maryland, for example, used these funds to create a position specific to MH/SU parity oversight, which the state made permanent after the funding period ended. In 2018, CMS awarded funding through the State Flexibility to Stabilize the Market Grant Program that focused on supporting state implementation and planning around several PPACA market reforms and consumer protections. Washington, for example, is using this grant to review issuer’s implementation of state and federal MH/SU parity requirements and to assess access to MH/SU treatment. HHS, DOL, and Treasury also coordinate with state regulators and NAIC to issue guidance for stakeholders in an effort to increase understanding of and compliance with MH/SU parity requirements. From December 2010 to September 2019, the three agencies issued 10 guidance documents that included 58 frequently asked questions and answers specific to MH/SU parity requirements. These guidance documents cover a range of topics, including describing the types of plans covered by MH/SU parity requirements, providing definitions of QTLs and NQTLs, and using specific scenarios to show if a practice—such as requiring prior authorization for certain medications to treat a substance use disorder—is permissible under the law. HHS, DOL, and Treasury have also developed guidance or support on MH/SU parity aimed specifically at consumers. For example, as part of HHS’s action plan, HHS developed a web-based portal to assist consumers in identifying, based on the consumer’s insurance type, the appropriate entity to contact for filing a parity-related complaint—HHS, DOL, or state insurance regulators. See appendix III for examples of guidance published by the agencies and the target audience. States have reported that existing guidance and support from the agencies helped states in their reviews of issuers for compliance with MH/SU parity requirements; however, some states and other stakeholders have identified a need for additional guidance. Specifically, officials from 43 states reported in our survey that guidance or other support from the agencies has helped inform state reviews of plans or issuers for compliance with MH/SU parity requirements, and officials from 24 states reported in our survey that additional guidance or support is needed. In written survey responses, state officials most commonly identified the need for additional guidance around reviewing NQTLs. In their comments for the 2017 HHS public listening session, some stakeholders identified the need for additional compliance information. Similarly, two industry stakeholders and one consumer advocacy organization also told us that additional guidance around NQTLs would be helpful to improve compliance with MH/SU parity requirements. HHS, DOL, and Treasury issued additional guidance after seeking public comment, as required by the 21st Century Cures Act. This guidance covers the types of information plans must release to consumers or providers related to MH/SU parity, known as disclosure requirements, and NQTL requirements. Specifically, the guidance document contains (1) answers to 11 additional frequently asked questions on NQTLs and disclosure requirements and (2) a disclosure template consumers can use to request MH/SU parity-related information from their employer- sponsored health plans and issuers of individual plans. Released in September 2019, the guidance may address the concerns identified by states and stakeholders. Employer-sponsored group plan and issuer compliance with federal MH/SU parity requirements is important to ensure that individuals seeking MH/SU treatment do not face discriminatory practices. DOL’s and CMS’s oversight of employer-sponsored group plan compliance with federal health care laws is driven by information and complaints they receive about potential noncompliance; however the agencies receive relatively few consumer complaints about MH/SU parity and DOL refers a small percentage of those complaints to its investigators. DOL’s and CMS’s reviews of compliance with relevant federal health care laws—including those related to MH/SU parity even when the origin of the investigation was unrelated to MH/SU parity concerns—has enabled the agencies to identify some plan and issuer violations of MH/SU parity requirements. However, the frequency with which compliance issues are identified in these reviews suggests that noncompliance with MH/SU parity requirements may be common. Given stakeholder-identified concerns with relying on complaints for MH/SU parity, the complexity of MH/SU parity requirements, and the limited complaints received in this area, DOL and CMS may not be identifying and responding to the risks posed by the agencies’ oversight approach. As a result, consumers may be enrolled in plans that fail to comply with MH/SU parity requirements. Until DOL and CMS evaluate whether the current approach of targeted oversight in response to information received is effective for identifying compliance issues with MH/SU parity, they will not know whether this approach is effective or whether additional strategies are needed to help ensure that their oversight meets their commitment to full implementation of MHPAEA. We are making a total of two recommendations, including one to DOL’s Employee Benefits Security Administration and one to HHS’s CMS. Specifically: The Assistant Secretary of Labor for the Employee Benefits Security Administration should evaluate whether targeted oversight in response to information received is effective for ensuring compliance with MH/SU parity requirements. If this evaluation determines the current targeted oversight approach results in significant program risks, the Employee Benefits Security Administration should develop a plan to more effectively enforce MH/SU parity requirements and if necessary seek additional oversight authority, as warranted. (Recommendation 1) The Administrator of CMS should evaluate whether targeted oversight in response to information received is effective for ensuring compliance with MH/SU parity requirements for non-federal governmental plans. If this evaluation determines the current targeted oversight approach results in significant program risks, CMS should develop a plan to more effectively enforce MH/SU parity requirements and if necessary seek additional oversight authority, as warranted. (Recommendation 2) We provided a draft of this report to DOL, HHS, and Treasury for review and comment. DOL and HHS both concurred with our recommendations. DOL’s comments are reproduced in appendix IV and discussed below. HHS’s comments are reproduced in appendix V and discussed below. DOL, HHS, and Treasury also provided technical comments, which we incorporated as appropriate. In its written comments, DOL elaborated on its current strategy to review its health enforcement program. Specifically, DOL noted that it reviews all MH/SU parity-related investigation findings and case closings, and all health plan investigations include a review of MH/SU parity requirement compliance, regardless of the source or reason for the investigation. DOL also stated that its current enforcement strategy to identify violations at the plan level and seek corrections of systemic violations at the service provider level has been successful. However, as explained in our report, DOL has not analyzed whether relying on targeted reviews alone increases the risk of noncompliance with MH/SU parity requirements in private, employer-sponsored group plans. Such an evaluation could help DOL identify and determine if additional enforcement strategies related to MH/SU parity requirements are needed. In its comments, DOL also noted its resource limitations. Specifically, DOL stated that despite the Employee Benefits Security Administration’s small size and limited resources, it is responsible for overseeing 2.4 million health plans, among other things. DOL noted that it will consider GAO’s recommendation in light of its resource constraints. Given these constraints, an evaluation could help ensure DOL’s resources are most efficiently targeted. In its comments, HHS stated that it is committed to enforcing MH/SU parity requirements. HHS described its responsibilities for enforcement and noted that it works with plans and issuers to help them understand and comply with MHPAEA. HHS also stated that it collaborates with state regulators, DOL, and Treasury in an effort to increase understanding and compliance. We are sending copies of this report to the appropriate congressional committees, the Secretaries of Health and Human Services, Labor, and Treasury, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or DickenJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Appendix I: Mental Health and Substance Use Disorder Parity Requirements in Medicaid and the State Children’s Health Insurance Program (CHIP) In 2016, the Centers for Medicare & Medicaid Services (CMS), an agency within the Department of Health and Human Services, issued a final rule addressing the application of the Paul Wellstone and Pete Domenici Mental Health Parity and Addiction Equity Act of 2008 (MHPAEA) to Medicaid managed care organizations (MCO), Medicaid Alternative Benefit Plans (ABP), and CHIP. Under this final rule, all beneficiaries enrolled in Medicaid MCOs, ABPs, and CHIP are entitled to mental health and substance use disorder (MH/SU) benefits that comply with certain MH/SU parity requirements of MHPAEA, which generally requires that MH/SU benefits be no more restrictive than medical or surgical benefits when MH/SU benefits are offered. The CMS final rule defines the role of the states in evaluating overall compliance of state Medicaid and CHIP programs with MH/SU parity requirements. The final rule establishes the processes by which states must assess and document that their Medicaid and CHIP programs comply with MH/SU parity requirements. CMS guidance provides detailed information to help states assess their compliance with MH/SU parity requirements. These processes vary by program type, as described below. Medicaid MCOs. The final rule requires either the state or the Medicaid MCO to complete a parity analysis, depending on how Medicaid benefits are provided. In general, CMS guidance requires states or MCOs to assess if a plan’s MH/SU benefits are no more restrictive than medical or surgical benefits for the following items: aggregate lifetime/annual dollar limits, financial requirements, quantitative treatment limitations (QTL), and non-quantitative treatment limitations (NQTL). The MCO must complete this analysis when it provides all Medicaid benefits—both medical and MH/SU benefits. The state must complete the parity analysis if the benefits are provided through multiple delivery systems, such as through multiple MCOs or the state’s fee-for-service Medicaid program, and provide the parity analysis to CMS for review. States are also required to make the documentation of compliance with the final rule available to the general public. The final rule also requires states to include contract provisions requiring compliance with MH/SU parity requirements in all MCO and other applicable contracts. CMS guidance encourages states to consider including provisions in their contracts with MCOs to ensure adequate oversight of the MCO’s parity-related monitoring and compliance activities, such as ensuring the state can see the MCO’s parity analysis. ABPs and CHIP. The final rule requires states to document that their ABP and CHIP plans comply with MH/SU parity requirements in the comprehensive state plans that describe the state’s Medicaid and CHIP programs. CMS guidance requires that states conduct a parity analysis demonstrating this compliance as part of the documentation the states submit to CMS to request a change to the state plan, known as a state plan amendment. In certain CHIP programs and ABPs, the state does not have to complete the full parity analysis, known as deemed compliance. A plan may be deemed to be in compliance with MH/SU parity requirements for plan members aged 20 and under if Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) benefits are provided to those individuals, because EPSDT benefits include MH/SU services. CMS guidance requires that states demonstrate that EPSDT benefits are covered by their CHIP plans through documents such as member handbooks. (The state or MCO would still be required to conduct a parity analysis to ensure that plan benefits for those not eligible for EPSDT benefits satisfy parity requirements.) To ensure state Medicaid and CHIP benefits comply with MH/SU parity requirements, CMS must review states’ documentation of compliance. For Medicaid managed care, CMS must review state contracts with managed care plans to ensure they are compliant with CMS requirements. CMS reviews the parity provisions in MCO contracts and the state’s parity analysis as part of the normal contract review process. Additionally, for states in which some but not all benefits are provided by an MCO, CMS reviews documentation of the state’s parity analysis to ensure the full scope of services being provided complies with MH/SU parity requirements. For ABP and CHIP, CMS staff are required to review the state plan amendments submitted by the states and supporting documentation for compliance with MH/SU parity requirements. See figure 2 for a map of the parity compliance review process by program type. We conducted a literature review to identify information about compliance with federal mental health/substance use disorder (MH/SU) parity requirements by individual and employer-sponsored small and large group health plans. We identified literature through keyword searches of several bibliographic databases, including ProQuest, MEDLINE, Scopus, and WorldCat. We focused our review on literature published between January 2011 and May 2019. Of the 828 study citations we identified, we reviewed 77 full studies; of those, we determined there were six relevant studies. We also identified four additional studies through web searches, interviews with stakeholders, and citations included in the literature we reviewed. Our review included studies that contained information collected about compliance by individual and employer-sponsored group health plans with federal MH/SU parity requirements by assessing compliance, comparing MH/SU plan benefits and requirements to medical/surgical benefits, or by assessing changes in MH/SU plan benefits over time. Our review excluded studies that focused on the effects of federal MH/SU parity requirements on consumer utilization of MH/SU services, consumer spending on MH/SU services, and plan spending on MH/SU services. The 10 studies are described in more detail below. Berry, Kelsey N., et al. “A Tale of Two States: Do Consumers See Mental Health Insurance Parity When Shopping on State Exchanges?” Psychiatric Services, vol. 66, no. 6 (2015): pp. 565–567. Methodology: The case study reviewed documents for all small group and individual health insurance products offered on two state health insurance exchanges between October 2013 and March 2014 and assessed compliance with observable quantitative treatment limitations (QTL) and non-quantitative treatment limitations (NQTL). Examples of key findings: The case study found that for 75 percent of products offered, the financial requirements and certain NQTLs reviewed appeared to be compliant with MH/SU parity requirements, but compliance varied by state. On one state health insurance exchange (with fewer products) more than half the products appeared inconsistent with the parity requirements reviewed, particularly the NQTLs. On the other state health insurance exchange, 11 percent of the products contained a financial requirement that violated MH/SU parity requirements. The case study was not able to assess all aspects of NQTL requirements because the available documents did not provide information about all NQTLs, such as whether or not a specific MH/SU treatment would be considered medical necessary. Cowell, Alexander J., et al. Changes in Individual and Small Group Behavioral Health Coverage Following the Enactment of Parity Requirements: Final Report. A report prepared for the United States Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation, Office of Disability, Aging and Long-Term Care Policy. January 2017. Methodology: The study reviewed plan documents for a sample of individual and small group plans and assessed changes in MH/SU and medical/surgical benefits before implementation of MH/SU parity requirements in 2013 and after implementation in 2014. Examples of key findings: The study found that in 2014 most plans’ financial requirements and QTLs were compliant with MH/SU parity requirements. However, the plans included different limits on the quantity of prescription drugs covered for medications used for MH/SU treatments and those used for other chronic health conditions. This difference indicated possible noncompliance with MH/SU parity requirements for NQTLs, and the study noted that differences in NQTLs between MH/SU and other health conditions is an issue in need of additional study. The study stated that plan documents did not contain all information necessary to fully assess NQTLs. Friedman, Sarah, et al. “The Mental Health Parity and Addiction Equity Act Evaluation Study: Impact on Mental Health Financial Requirements among Commercial ‘Carve-In’ Plans.” Health Services Research, vol.51, no. 1 (2018) pp.366-388. Methodology: The study analyzed a sample of health benefit design data from 2008 to 2013. This data on large group plans was obtained from a managed behavioral health organization and was analyzed for changes in cost-sharing requirements for plan members before and after parity requirements were implemented. Examples of key findings: The study found that there were both increases and decreases in cost-sharing after MH/SU parity requirements went into effect. For example, among plans that covered both in-network and out-of-network benefits and required coinsurance for inpatient stays, the likelihood of using coinsurance increased by 4 percentage points, and the coinsurance rate increased by .75 percentage points. However, outpatient copayments were reduced by $3.88 among plans that offered only in-network benefits. Goplerud, Eric. Consistency of Large Employer and Group Health Plan Benefits with Requirements of the Paul Wellstone and Pete Domenici Mental Health Parity and Addiction Equity Act of 2008. A report prepared for the United States Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation, Office of Disability, Aging and Long-Term Care Policy. November 2013. Methodology: The study summarized the results of multiple assessments of compliance with MH/SU parity between 2009 and 2011 based on both plan data available from private databases and the Department of Labor, and survey data and interviews with health plan representatives. Examples of key findings: The study found that between 2009 and 2011 large group health plans made substantial changes to their plan designs to meet the parity requirements. By 2011, most large group health plans had removed most financial requirements that did not meet MH/SU parity requirements, although 20 percent still had a non- compliant copayment for outpatient services. Nearly all had eliminated the use of separate deductibles for MH/SU treatment and medical/surgical treatment. The study also noted that assessing consistency with NQTLs was difficult based on document reviews. Hodgkin, Dominic, et al. “Federal Parity and Access to Behavioral Health Care in Private Health Plans.” Psychiatric Services, vol. 69, no. 4, (2018): pp. 396–402. Methodology: The study reported results of surveys of senior executives at commercial health plans regarding changes to MH/SU benefits over time. The surveys were conducted between September 2010 and June 2011, and again between August 2014 and April 2015. The study did not independently verify the self-reported data from senior executives. Examples of key findings: The study did not find significant noncompliance with MH/SU parity requirements. It found that fewer plans required prior authorization for outpatient MH/SU treatment than medical treatment. This suggests compliance with the requirement that NQTLs applied to MH/SU treatment be no more restrictive than those for medical/surgical treatment. The study also found that 6 percent of products used coinsurance for MH/SU treatment and copayments for other medical care. While this is not necessarily noncompliant, this could result in noncompliant higher cost-sharing for MH/SU treatment than other medical care in some cases, because coinsurance may result in higher cost-sharing than a copayment. Horgan, Constance M., et al. “Health Plans’ Early Response to Federal Parity Legislation for Mental Health and Addiction Services.” Psychiatric Services, vol. 67, no. 2 (2016): pp. 162–168. Methodology: The study reported results of surveys of senior executives at commercial health plans regarding changes to MH/SU benefits over time. The surveys were conducted between September 2010 and June 2011. The study did not independently verify the self- reported data from senior executives. Examples of key findings: The study found that plans complied with MH/SU parity requirements by lifting QTLs that only applied to MH/SU benefits, although 4 percent of plans had QTLs that applied to mental health treatment that did not apply to medical/surgical treatment. This study also found that fewer plans had prior authorization requirements for outpatient MH/SU treatment than outpatient medical treatment, which suggests compliance with the requirement that NQTLs applied to MH/SU treatment be no more restrictive than those for medical/surgical treatment. The study was not able to assess if prior authorization requirements were implemented differently between MH/SU and medical/surgical treatment. Huskamp, Haiden A., et al. “Coverage of Medications That Treat Opioid Use Disorder and Opioids for Pain Management in Marketplace Plans, 2017.” Medical Care, vol.56, no 6(2018) pp.505-509. Methodology: The study compared coverage for medications used to treat opioid use disorder (an MH/SU benefit) and opioids used to treatment pain management (a medical/surgical benefit) in 2017 health insurance marketplace exchange plans, using publicly available data for a sample of 100 plans. Examples of key findings: The study found that most plans covered at least one of the four primary medications intended for opioid use disorder treatment, while 100 percent of plans cover short-acting opioid pain medications. For example, 80 percent of plans cover a generic combination of buprenorphine and naloxone for treatment of opioid use disorder, while 100 percent of plans cover the generic version of Oxycodone and Fentanyl for treatment of pain disorder. The study states that additional monitoring is needed to ensure that plan coverage of MH/SU medications complies with MH/SU parity requirements. Thalmayer, Amber Gayle, et al. “The Mental Health Parity and Addiction Equity Act Evaluation Study: Impact on Nonquantitative Treatment Limits for Specialty Behavioral Health Care.” Health Services Research, vol. 53, no. 6 (2018): pp. 4584–4608. Methodology: The study analyzed a sample of health benefit design data from 2008 to 2013. This data on large group plans was obtained from a managed behavioral health organization and was analyzed for changes in NQTL requirements for plan members before and after parity requirements were implemented. Examples of key findings: The study found plans were less likely to require NQTLs, such as prior authorization and financial penalties for failure to obtain prior authorization for MH/SU treatments after MH/SU parity requirements were implemented, among plans that manage MH/SU benefits separately from other medical benefits. However, the study also found that plans were more likely to include a penalty for failing to obtain prior authorization for MH/SU treatments after MH/SU parity requirement implementation if the MH/SU benefits were managed by the same plan that managed other health benefits. The study was limited in that it did not assess how NQTLs were implemented by plans and so could not determine if there were differences in how MH/SU and medical requirements were applied. Thalmayer, Amber Gayle, et al. “The Mental Health Parity and Addiction Equity Act (MHPAEA) Evaluation Study: Impact on Quantitative Treatment Limits.” Psychiatric Services, vol. 68, no. 5 (2017): pp. 435–42. Methodology: The study analyzed a sample of health benefit design data from 2008 to 2013. This data on large group plans was obtained from behavioral health organizations and was analyzed for changes in QTL requirements for plan members before and after MH/SU parity requirements were implemented Examples of key findings: The study found that QTLs were nearly eliminated after MH/SU parity requirements were implemented. This suggests that plans became compliant with parity requirements because if a QTL does not exist it cannot be more stringent than a medical/surgical QTL. The study noted that plans that continued to have QTLs might be noncompliant with MH/SU parity requirements, but did not assess that. Center on Addiction, Uncovering Coverage Gaps II: A Review and Comparison of Addiction Benefits in ACA Plans, (New York: March 2019). Methodology: The study reviewed plan documents to assess compliance with MH/SU parity requirements from a sample of 2017 benchmark plans and plans sold on health insurance exchanges. Examples of key findings: The study identified nine benchmark plans and 10 states that sold plans that were not compliant with MH/SU parity requirements (where this could be identified through plan documents). The study was able to identify non-compliant financial requirements in three benchmark plans and non-compliant QTLs in six benchmark plans, and found one state that sold a plan to with a possible non-compliant QTL. The study also identified two benchmark plans that had possibly noncompliant NQTLs, and 21 states that had either NQTL violations or indications of possible NQTL violations that could not be fully assessed with the available information. The study noted that plan documentation did not contain sufficient information to fully assess compliance with MH/SU parity requirements related to NQTLs. Summary This October 2016 publication provides an overview of federal disclosure laws affecting private-sector, employer-sponsored group health plans and health insurers. Between December 2010 and September 2019, the three agencies issued 10 guidance documents with 58 frequently asked questions about MH/SU parity requirements. These frequently asked questions are designed to help people understand the law, and benefit from it as intended through examples that illustrate the requirements. Topics include the types of plans covered by MH/SU parity requirements and specific examples of how to determine if a practice or policy is permissible under the law. This June 2016 brochure gives a high-level overview of MH/SU parity requirements and lists common limits placed on MH/SU services that are subject to parity. This April 2018 action plan released by HHS covers recent and planned actions related to HHS, DOL, and Treasury’s implementation of MH/SU parity requirements. The plan, required by the 21st Century Cures Act, includes information about a public listening session the agencies held in July 2017. This February 2016 publication describes MH/SU parity requirements for people with employer-sponsored health plans who need MH/SU treatment. It describes why some MH/SU benefit claims are denied and how to file a claim, the denial of a claim, and the appeals process. Self-Compliance Tool for the Mental Health Parity and Addiction Equity Act (MHPAEA) DOL issued this self-compliance tool in April 2018 to help both issuers and regulators determine if a plan or issuer complies with MH/SU parity requirements and other related federal health care laws. In May 2016, DOL and HHS published this brief guide of examples of plan provisions that—absent similar restrictions on medical/surgical benefits—could be “red flags” that a plan or issuer may be imposing an NQTL that is out of compliance with MH/SU parity requirements and should be reviewed. In addition to the contact named above, Kristi Peterson (Assistant Director); Summar C. Corley (Analyst-in-Charge); Kerry Casey; and Eric J. Schwab made key contributions to this report. Also contributing were Leia Dickerson, Cynthia Khan, Laurie Pachter, Ethiene Salgado- Rodriguez, and Emily Wilson Schwark.", "summary": "MHPAEA requires large group health plans that offer MH/SU benefits to ensure parity between MH/SU and medical/surgical benefits. To meet the essential health benefits requirements of the Patient Protection and Affordable Care Act, certain issuers offering small group and individual plans must comply with MHPAEA's MH/SU parity requirements. The 21st Century Cures Act included a provision for GAO to review federal and state oversight of MH/SU parity requirements and the extent to which health plans comply with these requirements. This report, among other objectives, (1) examines how DOL, HHS, and states oversee health plan compliance with MH/SU parity requirements; and (2) describes what is known about the extent to which health plans are complying with MH/SU parity requirements. For this report, GAO reviewed DOL and HHS policies, guidance, and reports; conducted a survey and received responses from all 50 states and the District of Columbia about oversight practices; interviewed officials from DOL, HHS, and selected states; interviewed national and state stakeholders; and reviewed available research studies regarding health plan compliance with MH/SU parity. The Paul Wellstone and Pete Domenici Mental Health Parity and Addiction Equity Act of 2008 (MHPAEA) generally requires that coverage for mental health and substance use disorder (MH/SU) be no more restrictive than coverage for medical/surgical services. State agencies and the Departments of Labor (DOL) and Health and Human Services (HHS) share responsibility for overseeing compliance with these MH/SU parity requirements among group and individual health plans. These oversight practices vary. While nearly all of the state officials who responded to GAO's survey reported that they perform some review of group and individual insurance plans for compliance with MH/SU parity requirements before they are approved to be sold to consumers, states vary in the frequency and type of reviews they conduct after consumers enroll in plans. For example, officials from 12 states reported that they conducted a targeted review of specific MH/SU parity concerns in 2017 and 2018, with the number of reviews ranging from one to 22 reviews per state. DOL and HHS conduct targeted reviews of certain employer-sponsored group plans when they receive information—such as consumer complaints—about possible noncompliance with MH/SU parity requirements or other federal heatlh care requirements. Unlike states, these reviews only occur after consumers enroll in these plans. For example, in fiscal years 2017 and 2018, DOL completed 302 reviews that included a review of MH/SU parity compliance in its oversight of 2.2 million plans. Nearly all these reviews originated from complaints or other information about potential noncompliance with federal health care laws unrelated to MH/SU parity. According to DOL and HHS officials, the departments have not analyzed whether relying on targeted reviews alone increases the risk of noncompliance with MH/SU parity requirements in employer-sponsored group plans. Without such an evaluation, DOL and HHS do not know if their oversight is effective or whether they need to adopt additional strategies. While states, DOL, HHS, and the research GAO reviewed identified some instances of noncompliance with MH/SU parity requirements, the extent of compliance with these requirements is unknown. States, DOL, and HHS have identified some noncompliance with MH/SU parity requirements based on consumer complaints and other information about potential noncompliance. For example, DOL reported citing 113 violations of MH/SU parity requirements through its reviews in 2017 and 2018. The available research studies GAO reviewed also identified noncompliance with some of the requirements by reviewing plan documentation and benefit data, among other methods. However, according to stakeholders GAO interviewed, complaints are not a reliable indicator of the extent of noncompliance because consumers may not know about MH/SU parity requirements or may have privacy concerns related to submitting a complaint. GAO is recommending that DOL and HHS evaluate whether relying on targeted oversight is effective for ensuring compliance with MH/SU parity requirements or whether alternative approaches are needed. DOL and HHS concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "In 1903, the War Department under President Theodore Roosevelt established the National Board for the Promotion of Rifle Practice, today known as CMP, with the general purpose of promoting the development of marksmanship skills and preparing individuals in the event that they were called upon to serve in the military. For the next several decades, the Army managed and operated CMP. In 1990, we reported that CMP was of limited value to military preparedness because, among other things, CMP’s objectives and goals were not linked to Army mobilization and training plans and program-trained personnel were not tracked. The NDAA for Fiscal Year 1996 moved CMP out of the Army and established CMP as a federally chartered, nonprofit corporation. The act also required the Secretary of the Army to transfer all firearms, ammunition, and funds from sales previously under the control of the Army program to CMP. The governing statutes for CMP and Army support of CMP activities are generally found in chapter 407 of Title 36, U.S. Code. Among other things, these provisions provide for the organization, governance structure, and functions of CMP. These functions include instructing U.S. citizens in marksmanship, promoting practice and safety in the use of firearms, and conducting competitions. For purposes of training and competition, CMP may issue or loan certain rifles, ammunition, repair parts, and other supplies necessary for activities related to CMP to affiliated organizations that provide firearms training to youth, the Boy Scouts of America, 4-H Clubs, Future Farmers of America, and other youth-oriented organizations. CMP is required to give priority to activities that benefit firearms safety, training, and competition for youth and that reach as many youth participants as possible. As one of its functions, CMP conducts rifle and handgun marksmanship competitions such as the annual National Matches. The National Matches is open to members of the armed forces, the National Guard, the Reserve Officers’ Training Corps, and rifle clubs, among other entities, as well as to civilians. Additionally, CMP may sell certain surplus rifles and M1911 handguns to affiliated organizations that provide training in the use of firearms, such as gun clubs. Finally, CMP is authorized to sell to U.S. citizens who are members of affiliated gun clubs, at fair market value, surplus .22 caliber rimfire rifles, .30 caliber rifles, and .45 caliber M1911/M1911A1 handguns, as well as ammunition, repair parts, and other supplies necessary for target practice. The Army and CMP have entered into various agreements governing their relationship. An MOU from 2016 currently delineates Army and CMP responsibilities for, among other things, the transfer of surplus firearms and associated parts and ammunition. Appendixes to the MOU identify approximately 170 surplus rifles and handguns that may be transferred to CMP. These surplus firearms include the M1 Garand .30 caliber rifle and other rifles, such as the 1903 Springfield and 1917 Enfield. See figure 1 for a photograph of a surplus M1 .30 caliber rifle packed for shipment. The Army provides a variety of support to CMP, including identifying and reserving certain surplus firearms, ammunition, and parts. At CMP’s request, the Army can transfer these firearms, ammunition, and parts to CMP under procedures established in the MOU. TACOM is the executive agent for small arms. Per the MOU between the Army and CMP, TACOM provides various forms of support to CMP, including facilitating transfers of surplus firearms from U.S. sources and recovery of firearms from foreign countries before transfer to CMP. For surplus firearms transfers within the United States, CMP reimburses the Army for the cost of preparation and transportation, including for the Army’s standard depot operations costs. The costs of the recovery of firearms, ammunition, and parts from foreign countries are treated as incremental direct costs of Army logistical support, and are also to be reimbursed by CMP. The MOU contains further provisions related to Army and CMP responsibilities and procedures, such as provisions regarding Army support for competitions and the Small Arms Firing School, and CMP’s role in the Army’s Ceremonial Rifle Program. The MOU also contains procedures and responsibilities related to funding. Finally, the MOU specifies certain management internal controls to be undertaken by CMP, including those related to the sale of firearms and the accountability of transferred materiel. CMP’s implementation and management of these controls is to be assessed and documented in an audit report to the Army required by the agreement. To implement the transfer of surplus M1911 handguns, parts, and accessories, the Army and CMP entered into a Memorandum of Agreement in January 2018, and the Army began transferring the surplus M1911 handguns to CMP the same month. The Memorandum of Agreement establishes procedures and requirements for the Army and CMP additional to those in the 2016 MOU. Among other things, it requires CMP to provide the Army with transaction data for all surplus handguns received and sold on a quarterly basis, including the number transferred to CMP, the number sold, a listing of the serial numbers for handguns sold, and any information CMP has regarding crimes committed with a purchased M1911 handgun. The Memorandum of Agreement further required CMP to take certain actions with respect to security and accountability procedures for surplus M1911 handgun processing and storage. See figure 2 for a photograph of surplus M1911 handguns. Chapter 407 of Title 36, U.S. Code authorizes CMP to sell firearms to individuals who (1) are U.S. citizens, (2) are legally of age, and (3) are members of CMP-affiliated gun clubs. CMP’s sales of surplus firearms are generally subject to applicable federal, state, and local law. For example, the minimum age to purchase a rifle from a federal firearms licensee (FFL) is 18, while the minimum age to purchase a M1911 handgun from an FFL is 21. Additionally, CMP must establish procedures to obtain a criminal records check with federal and state law enforcement agencies. Certain federal requirements and restrictions related to the sale of firearms are contained in section 922 of Title 18, U.S. Code. Among other things, section 922 prohibits selling or otherwise disposing of firearms to certain prohibited persons. Generally, only FFLs may engage in the business of dealing in firearms. Additionally, FFLs generally may not sell firearms directly to out-of-state customers other than another FFL. However, these restrictions do not apply to CMP for the sale of surplus .22 caliber rimfire and .30 caliber rifles. Specifically, CMP may sell these rifles without operating as an FFL and ship these rifles directly to customers around the country, unless prohibited by that customer’s state or local law. With respect to the sale of the surplus M1911 handguns, CMP must obtain a license and operate as an FFL. The MOU between the Army and CMP delineates a number of responsibilities for both organizations regarding the transfer of surplus firearms. Furthermore, we found that both organizations have established procedures to carry out these responsibilities. Appendixes to the MOU list approximately 170 firearms that the Army has identified as surplus to its needs. If any of the surplus firearms described in the MOU are identified by the Army in a domestic location, the Army reserves those firearms for transfer to CMP pending a formal written request from CMP for the transfer of the surplus firearms in question. For example, in fiscal year 2017 the Army identified and reserved for transfer to CMP more than 1,000 surplus rifles that various Department of Defense museums found to be surplus to their needs. Under the MOU, once TACOM informs CMP it has reserved surplus firearms that may be transferred, CMP can submit a transfer request for the surplus firearms in writing to TACOM. CMP’s written request must acknowledge that the requested materiel is on the list of firearms approved for transfer; certify that CMP will provide all security, oversight, and accountability—as required by law—of the materiel; and describe how CMP will use the requested materiel. TACOM facilitates the transfer of surplus firearms to CMP as required by the MOU. In some instances, the Army directly ships surplus firearms within the United States to CMP. In other instances, TACOM relies on the Defense Logistics Agency (DLA) to ship the surplus firearms to CMP. If DLA transfers the surplus firearms, the firearms are either shipped directly to CMP or to the DLA facilities located in Anniston, Alabama, where they are released to CMP. Under the MOU, CMP reimburses the Army for certain costs associated with transportation, supply depot operations, and administrative support. Both CMP and DLA officials told us that surplus firearms located at DLA’s facilities in Anniston, Alabama, did not incur shipping cost to the Army because CMP arranges the transfer from the DLA facilities directly to CMP’s facilities also located in Anniston, Alabama. For example, according to TACOM and CMP officials the Army did not incur any transportation costs for transferring 8,000 surplus M1911 handguns to CMP in January 2018. According to an Army official, this was because CMP transported 6,736 M1911 handguns from the DLA facility, an additional 1,242 M1911 handguns from the Center of Military History Museum Support Center, and 22 M1911 handguns from TACOM facilities—all located in Anniston, Alabama—back to its own facility in Anniston for storage. In addition to firearms from domestic locations, the Secretary of the Army may also recover certain surplus firearms furnished to foreign countries on a grant basis under the Foreign Assistance Act and transfer them to CMP. If the Secretary of the Army decides to transfer surplus firearms from a foreign country, TACOM and the Office of the Administrative Assistant to the Secretary of the Army works with the State Department, Office of Defense Cooperation, representatives located in the respective foreign country to recover and facilitate the transfer and shipment of surplus firearms from these recipients to CMP. For example, according to Army officials, CMP received approximately 100,000 surplus M1 rifles in fiscal year 2018; more than 13,000 surplus M1 rifles were recovered and transferred from Turkey in addition to nearly 87,000 surplus M1 rifles from the Philippines. After CMP receives the surplus firearms, the MOU requires CMP to perform all accounting procedures required by the Army for inventory control, including compiling the surplus firearms’ serial numbers. According to CMP officials, CMP uses a commercial point of sale system to track inventory and performs an audit of firearms stored at its facilities annually. For example, according to CMP officials, to count and verify the inventory of incoming shipments of surplus firearms, CMP staff open and inspect each box of surplus firearms upon receipt of the firearms in their facilities in Anniston, Alabama. CMP staff then inventory each firearm by matching the unique serial number found on the receiver of each firearm to the manifest included with the shipment. CMP then enters these firearms into its inventory using each firearm’s unique serial number. After shipping firearms to CMP, the MOU requires TACOM to update the Army’s Unique Item Tracking database for tracking the firearms, which it does by serial number. By statute, title to a transferred firearm does not vest with CMP until immediately before CMP delivers the firearm to an eligible purchaser. CMP primarily uses sales procedures and the Federal Bureau of Investigation’s National Instant Criminal Background Check System (NICS) to address requirements related to the sale of surplus firearms. See figure 3 below for a description of several of the processes used by CMP to address the requirements related to the sale of surplus firearms. While various federal statutes and provisions from the agreements between the Army and CMP apply to selling both the surplus rifles and the surplus handguns, there are some differences. For example: Because section 40733 of Title 36, U.S. Code, exempts CMP from certain federal firearms requirements and restrictions, CMP can ship surplus rifles directly to a customer’s home, unless that would conflict with state or local laws applicable where the firearm is being shipped. In contrast, CMP must operate as an FFL for selling surplus M1911 handguns and will ship purchased handguns to an FFL, such as a certified gun shop, in the customer’s state. The local FFL repeats the background check before turning over the firearm to the customer. The MOU limits CMP’s sale of surplus rifles to eight per customer per calendar year, while the Memorandum of Agreement limits CMP’s sale of M1911 handguns to one per customer while it is in effect. The minimum age to purchase any rifle from CMP, including surplus rifles, is 18 while the minimum age for purchasing handguns, including surplus handguns, is 21. CMP is required to ship the surplus handguns with a security device such as a trigger lock, which it is not required to do for the sale of surplus rifles. CMP uses sales procedures to address federal requirements and agreements with the Army. According to CMP officials, the sales procedures, specifically the application to purchase the surplus firearms that customers are required to complete and have notarized, address some of the federal requirements and agreements between the Army and CMP for the sale of surplus firearms. According to CMP, customers are required to mail the original completed application package, including copies of substantiating documentation and notarization, to CMP in order to apply to purchase a surplus firearm. The application includes a form requiring potential customers to certify that they do not fall within any of the categories of individuals prohibited from being sold or receiving a firearm. The form must be signed and notarized, and specifically lists the prohibited categories, as well as certain CMP-unique categories. As part of the form, potential customers must also certify that by receipt or possession of the firearm they will not be in violation of any state law or published ordnance applicable where they reside. Additionally, applicants must provide proof of the following: Citizenship and age: Applicants must include a copy of a U.S. birth certificate; passport; proof of naturalization; a military identification card for certain ranks (active duty, reserve component, National Guard, or retired); or any official government document that shows that an individual was born in the United States or that otherwise identifies U.S. citizenship. According to the application procedures, a copy of a driver’s license is proof of age, but not of citizenship. Membership in CMP-affiliated organization: Applicants are required to provide a copy of their current membership card or another proof of membership in a CMP-affiliated organization. According to CMP, this requirement can also be satisfied by providing proof of membership in one of the federally chartered veterans’ organizations such as the Veterans of Foreign Wars or American Legion; proof of either current or retired military service; or proof of current or retired status in a law enforcement department, agency, or association. Marksmanship or other firearms-related activity: Applicants are required to show proof of participation in a marksmanship-related activity or otherwise show familiarity with the safe handling of firearms and range procedures. According to CMP, this can be accomplished by providing documentation of current or past military or law enforcement service, participation in a shooting competition, completion of a marksmanship clinic that included live-fire training, a concealed carry license, or a FFL license, among other things. Proof of license, permit, or firearms owner identification card: If the state or locality where the applicant resides requires a license, permit, firearms owner identification card, or other documentation, applicants are also required to include a photocopy of such a document with the application for purchase of the surplus firearm. Federal Firearms Licensee: Applicants purchasing surplus rifles who reside in states or localities where shipments must be made to an FFL and applicants purchasing surplus M1911 handguns must provide a copy of the license for the FFL that will be receiving the shipped firearm. According to CMP officials, CMP staff verifies the completeness of the application, including all required documentation while entering applicants’ information into CMP’s commercial point of sale and inventory system. According to CMP officials this involves staff entering the customer’s data into the point of sale and inventory system and verifying the customer’s name, address, proof of age, proof of citizenship, and membership in a CMP-affiliated organization, among other data. CMP uses a separate version of the same commercial point of sale and inventory system to enter and verify customers’ information for the purchase of surplus M1911 handguns. According to CMP officials this is due in part to the requirement to operate as an FFL in order to sell the surplus M1911 handguns. According to CMP, in addition to meeting additional record-keeping requirements required of FFLs, using two different systems helps CMP ensure it addresses certain sales requirements included in the agreements it has with the Army. For example, according to CMP officials, this helps them address the Memorandum of Understanding and Memorandum of Agreement provisions regarding the maximum number of sales of each type of firearm per customer. During our site visits to CMP’s southern headquarters in Anniston, Alabama in August 2018 and November 2018 we observed CMP officials processing applications from the public to purchase surplus firearms to better understand how CMP addressed certain federal requirements and agreements between the Army and CMP. Specifically, we observed 11 transactions from the receipt of an order through processing and packaging for shipment. Six of these transactions involved rifles and five involved handguns. The 11 transactions we observed were consistent with the sales procedures we identified above. For example, in all cases, we saw CMP staff verify and update customer information from the application packet for existing customers of surplus rifles and input customer information from the application packet for new customers of both the surplus rifles as well as the surplus handguns. We also observed that CMP staff could not move forward with the sale without entering and verifying the information supplied in the application. In one instance, we observed a CMP employee entering an application for the purchase of a surplus rifle that had not been notarized and signed. CMP employees stopped the process for this application and informed us the applicant would be contacted directly and requested to provide the required notarization in order for CMP to proceed with the sale. In another instance, CMP staff demonstrated what would occur if information pertaining to the documentation required to demonstrate membership in a CMP-affiliated gun club was not entered. We saw that CMP employees could not continue to the next screen without entering these data. CMP addresses various other federal requirements for the sale of firearms via background checks. Once the application procedures we described above are completed, CMP staff then enter the prospective customer’s information into the system used by the Federal Bureau of Investigation to perform background checks. CMP intends this National Instant Criminal Background Check System (NICS) to address certain federal requirements for the sale of firearms that we identified above. For example, the NICS background check analyzes various databases to determine whether a prospective customer falls into any of the categories of persons prohibited from being sold or receiving a firearm. The prohibitions involve sale to or receipt by a prospective customer who is under indictment for or has been convicted in any court of a crime punishable by imprisonment for a term exceeding 1 year; is a fugitive from justice; or has been convicted in any court of a misdemeanor crime of domestic violence, among other things. The NICS background check also searches databases to identify a prospective customer who has been discharged from the U.S. Armed Forces under dishonorable conditions, was a U.S. citizen but has since renounced his or her citizenship, or is an unlawful user of or addicted to any controlled substance. Additionally, CMP uses the NICS background check to confirm the applicant is of the minimum age necessary to purchase either a rifle or a handgun. Specifically, CMP uses the NICS background checks to ensure the applicant is at least 18 in order to purchase a surplus rifle or a minimum of 21 in order to purchase a surplus handgun. If an age is entered into NICS that is younger than the minimum age required for purchasing a firearm, the system will not continue performing the background check and will notify CMP staff that the buyer is not old enough to purchase the firearm(s). During our site visits to CMP’s southern headquarters in Anniston, Alabama in August 2018 and November 2018 we observed CMP employees performing the NICS background checks for 10 of 11 transactions, and we observed that the employees were unable to proceed with the background check without certain required information. Specifically, CMP employees demonstrated the result of a change to the birthdate while processing the sale of a surplus M1911 handgun so that the customer would be under 21 years of age. This resulted in the NICS system automatically not allowing the background check to proceed. As previously discussed, the MOU requires (1) CMP to reimburse the Army for certain costs associated with the transfer of firearms to CMP, and (2) TACOM to account for the funds reimbursed by CMP. Specifically, CMP is responsible for assuming or reimbursing TACOM for certain costs associated with transportation, standard depot operations, and administrative support. According to Army officials, these administrative support costs include TACOM’s annual cost of one full-time equivalent position to help administer the identification and shipment of surplus firearms to CMP. The MOU also requires TACOM to provide CMP with semi-annual reports identifying the reimbursable costs the Army incurred for any firearms transfers. According to TACOM and CMP officials, TACOM has met this requirement since at least fiscal year 2012 by providing briefings at CMP’s biannual Board of Director’s meetings. CMP reimburses TACOM by depositing funds into a TACOM-managed reimbursement account. TACOM uses the funds in this account to pay the costs associated with transferring firearms to CMP. According to the MOU, TACOM is also responsible for maintaining accountability of funds provided by CMP in support of certain transportation, supply depot operations, and administrative support. The MOU further provides that administrative funding will be evaluated at the end of each fiscal year. For fiscal years 2008 through 2015, TACOM did not have complete information on the reimbursable costs incurred by the Army and on the amounts CMP reimbursed the Army for those costs. This is because, according to TACOM officials, information on transactions involving the reimbursement account prior to fiscal year 2016 was maintained under a different accounting system and could no longer be accessed. According to TACOM officials, TACOM began using a new financial system in fiscal year 2016 to track the information used to maintain accountability including, among other things, the Army’s reimbursable costs and CMP’s payments for those costs. For fiscal years 2016 and 2017, TACOM officials provided us with examples of the documentation from the current system demonstrating that in addition to tracking CMP’s reimbursement payments TACOM tracks reimbursable costs for transportation, standard depot operations, and administrative support using five specific categories of information: travel, labor, commercial transportation, intra- Army purchases, and contract service. For example, these documents showed that CMP reimbursed the Army for a total of $5 million in fiscal year 2017. CMP’s primary source of revenue from fiscal years 2008 through 2017 was the sale of surplus firearms. During this time frame, according to CMP’s internal financial documents, CMP generated $196.8 million in revenue from the sales of surplus Army rifles. However, the profit that CMP realized from these sales could not be determined. CMP’s internal financial documents show that CMP incurred $84.7 million in costs for those sales, but CMP’s methodology for calculating costs associated with the transfer and sale of surplus rifles did not account for depreciation and administrative expenses. CMP officials anticipate generating additional revenue from the future sale of surplus M1911 handguns and surplus rifles that CMP currently has available to sell. We estimate these sales could generate as much as $104.9 million, or enough to fund CMP’s operations for several more years. Based on our analysis of CMP’s IRS filings and the corporation’s internal financial documents, we identified four primary sources of revenue, of which the sale of surplus Army rifles accounted for the largest share. Sale of surplus Army rifles. According to CMP’s internal financial documents, CMP generated $196.8 million in revenue from the sale of surplus Army rifles during fiscal years 2008 through 2017. The vast majority of these firearms were M1 rifles (see app. II for additional details on the specific types of rifles CMP sold during that time frame). Although the number of surplus rifles CMP sold varied from year to year, as shown in figure 4, except for fiscal years 2012 and 2013, the number of rifles sold has trended downward from fiscal years 2008 through 2017. Sale of ammunition and memorabilia. CMP purchases bulk quantities of commercially available ammunition at a discounted rate due to the size of the order, and then sells this ammunition to its affiliated groups. CMP also sells memorabilia such as T-shirts and hats. According to CMP’s fiscal years 2008 through 2017 internal financial documents, the sale of ammunition and memorabilia was CMP’s second largest source of revenue and generated $76.4 million. Figure 5 shows boxes of commercially purchased ammunition stored in CMP’s warehouse in Anniston, Alabama. Investment account income. CMP officials told us that CMP established an investment account to ensure it had the financial resources to continue to meet its mission should the transfer of surplus firearms from the Army cease. According to CMP’s fiscal years 2008 through 2017 IRS filings, CMP reported earning $49.8 million in interest and dividend income from the corporation’s investment account. As seen in figure 6, CMP’s investment account grew by approximately $88 million, from $100.3 million at the end of fiscal year 2008 to $188.6 million by the end of fiscal year 2017. Although CMP’s investment account grew over this 10-year period, in some years CMP made net deposits into the account, and in other years CMP had net withdrawals from the account. For example, CMP’s net deposits from fiscal years 2008 through 2013 were $73.5 million. However, CMP had net withdrawals of $21.5 million from fiscal years 2014 through 2017. CMP officials stated that they used withdrawals from the investment account in those years to expand marksmanship- related programs and to finance construction of the Talladega Marksmanship Park, completed in 2015, shown in figure 7. Marksmanship-related programs. CMP charges fees for individuals to participate in its marksmanship-related programs such as training programs, matches, youth camps, and competitions. According to CMP’s fiscal years 2008 through 2017 IRS filings, CMP generated approximately $9.2 million in revenue from these fees. CMP’s IRS filings for those years also indicate that CMP’s expenses associated with CMP’s marksmanship programs exceeded revenue by approximately $85.8 million. According to CMP officials, CMP heavily subsidizes participation fees for both matches and youth camps to help make the corporation’s programs as accessible as possible, and revenue generated from the sale of surplus firearms covered any program deficits. Figure 8 shows competitors during the National Matches event we observed in July 2018. Other revenue. According to CMP’s fiscal years 2008 through 2017 internal financial documents, CMP also generated some additional revenue from a variety of other sources. For example, CMP’s Talladega Marksmanship Park has generated over $1.5 million in revenue from range rental and match fees, among other things. CMP reported an overall profit of $125.9 million on its IRS filings for fiscal years 2008 through 2017, but this amount includes all categories of revenue and expense for business operations, not only those categories specific to surplus rifle sales. We were therefore unable to use CMP’s IRS filings to determine CMP’s profits from the sale of surplus rifles. The amount of profit specific to surplus rifle sales also could not be determined from CMP’s internal financial documents. CMP’s internal financial documents showed $84.7 million in expenses to sell surplus rifles in fiscal years 2008 through 2017, including costs associated with labor, shipping, and other expenses to prepare the surplus firearms for sale. This is less than the $196.8 million CMP’s internal financial documents show CMP generated in revenue from the sale of surplus Army rifles during fiscal years 2008 through 2017. However, in its internal financial documents, the methodology CMP used to calculate the expenses to sell surplus Army rifles did not include all of CMP’s expenses for these sales. Specifically, the methodology CMP used did not account for depreciation and administrative expenses. CMP did not begin selling surplus M1911 handguns until November 2018, and therefore had just begun generating revenue from these sales at the time of our report. CMP’s internal financial documents reported some costs associated with the surplus M1911 handguns. For example, in fiscal year 2018, in response to an Army requirement, CMP spent approximately $0.7 million upgrading a facility used to house CMP’s M1911 handgun operations. CMP also reported expenses specific to the M1911 handguns of just over $8,000 in fiscal year 2017. According to CMP officials, CMP anticipates selling most, if not all, of the M1911 handguns because there has been a higher demand for the surplus M1911 handguns than the quantity available to CMP for sale. For example, CMP officials reported that they received more than 19,000 orders for the 8,000 surplus M1911 handguns transferred from the Army in January 2018. How much CMP will sell each surplus handgun for depends on the quality, or grade, of the handguns as determined by CMP. Specifically, CMP officials told us CMP will sell service grade surplus M1911 handguns for $1,050, field grade handguns for $950, and rack grade handguns for $850 each. CMP officials reported that as of December 2018, CMP had sold 632 service grade surplus M1911 handguns for $1,050 each, which generated $663,600 in revenue. Further, CMP officials told us they had determined that 145 of the surplus M1911 handguns were in unsellable condition. As a result, as of December 2018, 7,223 surplus M1911 handguns remained from the original 8,000 CMP received from the Army. If CMP sold all of the remaining handguns, we estimate that CMP could generate from $6.14 million to $7.58 million in additional revenue, depending on the grade of each surplus M1911 handgun sold. As of December 2018, CMP officials told us they expected to complete the processing and sale of the surplus M1911 handguns in the spring of 2019. We estimate that by the time these sales are completed CMP could generate total revenue of from $6.8 million to $8.2 million from the sale of surplus M1911 handguns. CMP may also be able to continue to generate revenue from surplus rifles that are currently available for sale. Based on CMP’s reported sales of 304,233 surplus rifles from fiscal years 2008 through 2017 and revenue generated from these sales of $196.8 million, we determined the average sale of these surplus rifles to be approximately $650 per rifle. According to CMP, as of August 16, 2018 it had approximately 148,714 sellable surplus rifles. Based on our calculation of the average sales price of $650 per surplus rifle, we estimate CMP could generate approximately $96.7 million in revenue from selling surplus rifles currently available for sale. Combined with the potential revenue from the sale of M1911 handguns, we estimated CMP could generate from $103.5 million to $104.9 million from the future sale of surplus firearms. Given CMP’s fiscal year 2017 expenses of $15.8 million, and assuming a similar level of future annual expenses, we estimate CMP could fund a similar level of operations for several more years from the sale of all of the surplus firearms it currently has available for sale. Further, as discussed earlier, CMP has other sources of revenue. As of September 30, 2017, CMP reported having cash of $3.6 million and an investment account that was valued at $188.6 million, for a total of $192.2 million. This could also allow CMP to continue operations for several additional years if it did not receive any additional transfers of surplus firearms. In addition to CMP, we examined five other federally chartered corporations–the U.S. Naval Sea Cadet Corps, the Civil Air Patrol, Big Brothers Big Sisters of America, Future Farmers of America, and the Boy Scouts of America–that have a similar focus on the development, education, or training of youth. Four of the six corporations, including CMP, have received federal funding or resources, and each of the six corporations is governed by some form of a board of directors. However, CMP’s relationship with members, which CMP officials refer to as “affiliated groups” (e.g., gun clubs throughout the United States), differs from the other five corporations we selected for comparison. Organizational mission. All five of the other federally chartered corporations we examined have a focus on the development, education, or training of youth. The Naval Sea Cadet Corps identifies itself as a national youth leadership development organization that promotes interest and skill in naval disciplines while instilling strong moral character and life skills through leadership and technical programs modeled after the Navy’s professional development system. The Civil Air Patrol’s mission statement includes the development of youth and promotion of air, space, and cyber power. Further, the Civil Air Patrol identified that it promotes aviation and related fields through aerospace/science technology engineering and math education and by helping shape future leaders through its cadet program. Big Brothers Big Sisters of America’s overall mission includes providing children facing adversity with strong and enduring, professionally supported relationships that change their lives for the better, including helping children to achieve educational success. Future Farmers of America’s mission statement involves making a positive difference in the lives of students by developing their potential for premier leadership, personal growth and career success through agricultural education. The Boy Scouts of America identified that its goal is to train youth in responsible citizenship, character development, and self-reliance through participation in a wide range of outdoor activities, and educational programs, among other things. Federal funding or resources. CMP, the Naval Sea Cadet Corps, the Civil Air Patrol, and Big Brother Big Sisters of America received some form of federal funding or resources during fiscal years 2015 through 2017. CMP is the only one of these four corporations that relies on the transfer and sale of federally donated surplus firearms for the majority of its revenue. According to officials from the corporations, the Naval Sea Cadet Corps and the Civil Air Patrol rely on federal appropriations and federal grants from the Navy and the Air Force, respectively. For example, according to officials from the Naval Sea Cadet Corps, the corporation received approximately $5.1 million in federal grants from the Navy from fiscal years 2015 through 2017. Civil Air Patrol officials stated that federal funds were the largest source of revenue. According to officials from Big Brothers Big Sisters of America, the corporation received approximately $3.8 million in federal grants from the Department of Labor and $8.2 million in federal grants from the Department of Justice’s Office of Juvenile Justice and Delinquency Prevention from fiscal years 2015 through 2017. Officials from Big Brothers Big Sisters of America told us that these grants were the corporation’s largest source of funding. Officials at both Future Farmers of America and the Boy Scouts of America told us they raise funds through membership dues and merchandise sales, among other things, but do not receive any federal funding or resources. Organizational structure. The leadership structure of CMP and the five selected federally chartered corporations was similar. That is, officials from CMP and the five selected corporations told us that each corporation has a board of directors or board of governors that may or may not have term limits. For example, CMP’s Board of Directors includes 11 board members with repeatable 2-year term limits, for which the Chairman of the Board also serves as the Chief Executive Officer. According to Boy Scouts of America officials, the corporation’s National Council’s Board of Directors is elected through a nominating process and has no fixed term limits. The Board of Directors in turn elects representatives to the Executive Committee and there is also an Advisory Council. The Advisory Council, according to Boy Scouts of America officials, reports to the Board of Directors and comprises both former members of the board and members who may become future directors on the board. According to officials from Future Farmers of America, that corporation has a Board of Directors of which four members are designated by the Department of Education including a designated Chairperson, and these four members serve open-ended terms. According to these officials, the remaining members of the board not designated by the Secretary of Education serve 3-year terms. The Civil Air Patrol has an 11-member Board of Governors: four are appointed by the Secretary of the Air Force; four are from its volunteer force; and three are from outside the corporation. Organizational relationships. CMP’s relationship with what it refers to as affiliated groups (e.g., gun clubs) throughout the United States, differs when compared with the five federally chartered corporations we selected for review. CMP is located in two facilities: one in Anniston, Alabama, that, according to CMP officials, primarily handles sales and operations and one in Port Clinton, Ohio, that, according to CMP officials, manages mission-related programs, such as the National Matches. CMP also sells surplus firearms to members of groups affiliated with CMP from throughout the United States. But, while CMP officials identified 5,002 affiliated clubs throughout the United States and referred to them as being “affiliated” with CMP, none of these entities are actually part of CMP. According to CMP officials, the clubs pay a small annual fee to become affiliated with CMP, which allows them to participate in CMP- sanctioned marksmanship matches and so that their members are eligible to buy surplus firearms from CMP, among other things. In contrast, according to officials from the other five selected corporations, those corporations have members or affiliates throughout the United States— meaning that these members and affiliated groups are part of the organization as a whole. For example, officials from the Boy Scouts of America told us that they divide the country into regions, then local councils, local districts, counties or communities, and then to local sponsors of individual units or troops; all members are part of the Boy Scouts of America. Similarly, Naval Sea Cadet Corps officials told us the organization is comprised of regional and local units; there is open communication between headquarters and the local units, and a standardized training program is implemented at the local level. We provided copies of a draft of this report to the Secretary of the Army, the Civilian Marksmanship Program, and other interested parties for comment. The Secretary of the Army and Civilian Marksmanship Program provided technical comments, which we incorporated into this report as appropriate. We are sending copies of this report to appropriate congressional committees and the Secretary of the Army. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9627 or at maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. The National Defense Authorization Act for Fiscal Year 2018 required the Army to transfer surplus M1911 handguns to the Civilian Marksmanship Program (CMP) during fiscal years 2018 and 2019, including no fewer than 8,000 in fiscal year 2018 and no more than 10,000 in any fiscal year. The act also included a provision for us to review certain matters related to CMP. This report (1) examines the Army’s and CMP’s procedures to address requirements governing the transfer and sale of firearms; (2) examines CMP’s primary sources of revenue, costs and profits, and estimated future revenue associated with the sale of surplus firearms; and (3) compares certain aspects of CMP’s business operations with those of five selected youth-focused, federally chartered nonprofit corporations The scope of our review focused primarily on fiscal years 2008 through 2017. We compiled 10 years of the sale of surplus rifles from fiscal years 2008 through 2017 to understand the numbers of surplus rifles transferred as well as the revenue, costs, and profits associated with the sale of surplus rifles. To identify the requirements governing the transfer and sale of surplus firearms, we reviewed applicable federal statutes including relevant provisions from chapter 407 of Title 36, and section 922 of Title 18, U.S. Code, as well as agreements between the Army and CMP such as the 2016 Memorandum of Understanding and the 2018 Memorandum of Agreement. We also reviewed transfer and sales procedures from fiscal years 2018 and 2019 to provide a current status regarding CMP’s sale of surplus M1911 handguns, which CMP began selling in November 2018. To identify procedures put in place by the Army to address the requirements governing the transfer of firearms, we reviewed documentation of reimbursements CMP made to the Tank-automotive and Armaments Command (TACOM), the organization within the Army responsible for facilitating the transfer of surplus firearms to CMP as well as for managing the related reimbursement account. Our review of the procedures associated with the reimbursement account included obtaining cash collection vouchers submitted to the Army by CMP and TACOM briefings presented at CMP’s biannual Board of Director’s meetings. Further, we interviewed TACOM and Defense Logistics Agency (DLA) officials to gain an understanding of how reimbursable costs are identified and requested from CMP. We also compared multiple source documents related to transfers. To understand how TACOM identifies and reports costs associated with the transfer of surplus firearms, we reviewed documentation related to reimbursement for labor, transportation, and standard depot operation costs associated with the transfer of firearms from the Army to CMP. To identify procedures put in place by CMP to address the requirements governing the transfer and sale of firearms, we conducted site visits to CMP’s northern and southern headquarters in Port Clinton, Ohio and Anniston, Alabama, and observed the inventory and sales processes for rifles and handguns. During our site visits to CMP’s southern headquarters in Anniston, Alabama, in August 2018 and November 2018 we observed 11 examples of firearm transactions and compared the procedures with various federal requirements and the agreements between the Army and CMP. We also reviewed documentation of sale order forms, and of CMP’s sales operating system processing an order in order to identify how CMP enters and confirms certain information related to sales. In addition, we interviewed CMP officials to obtain further clarification on the organization’s sales processes. To determine CMP’s primary sources of revenue, as well as the costs and profits associated with the sale of surplus rifles, we reviewed financial information provided by CMP. Our review included an analysis of CMP’s IRS filings and internal financial documents for fiscal years 2008 through 2017. We used CMP’s IRS filings to provide information on revenue generated from overall sales, investments, and programs, as well as on the growth of CMP’s investment account. We relied on the internal financial documents for a more granular account of the revenue CMP generated specifically from the sale of surplus rifles as well as commercially purchased ammunition and memorabilia. CMP officials provided us with a methodology for determining which data within the organization’s internal financial documents are revenue and expenses specific to the sale of surplus rifles. We assessed the reliability of the data by interviewing CMP officials to gain an understanding of how CMP’s IRS filings and internal financial documents are produced and found it sufficiently reliable for our purposes. To assess the reliability of the surplus firearms transfer data provided by TACOM we spoke with TACOM officials for clarification and further explanation of the data provided, including firearm nomenclature and identification codes. The additional information TACOM provided allowed us to identify 17 different types of .22 or .30 caliber surplus rifles that could be grouped together based on make, model, and/or caliber. TACOM officials confirmed our groupings for the types of firearms transferred from fiscal years 2008 through 2017, and we used the results of our analysis to summarize the number and types of surplus rifles transferred to CMP during this time frame. We found the data to be sufficiently reliable for our purposes. To determine potential future revenue associated with the sale of surplus M1911 handguns we obtained current sales price information from CMP, and used this information to project a range of potential future revenue based on the number of surplus rifles and handguns CMP currently has on hand. Specifically, to determine the range of potential revenue for the sale of surplus handguns, we asked CMP to provide information on the sales prices for each of the three grades of preordered M1911 handguns. CMP reported that it had sold 632 surplus M1911 handguns as of December 13, 2018 and further that it had identified another 145 surplus handguns as unsellable. To determine the number of surplus handguns remaining to be sold, we subtracted both the 632 surplus handguns CMP reported as sold and the 145 surplus handguns CMP had determined to be unsellable from the total of 8,000 surplus M1911 handguns the Army originally transferred to CMP. To calculate the range of potential revenue from the remaining 7,223 surplus M1911 handguns, we then multiplied the 7,223 remaining surplus handguns by the lowest and the highest sales prices, $850 and $1,050 respectively. This gave us a range of revenue from the future sales of from $6.14 million to $7.58 million. We then added the known $663,600 in revenue from the sale of the 632 service grade handguns CMP identified to our low and high end calculations to determine the range of future revenue of from $6.8 million to $8.2 million from the sale of surplus M1911 handguns. To determine potential future revenue associated with the sale of surplus rifles, we reviewed inventory and sales data provided by CMP and used this information to estimate potential future revenue based on the average price of the surplus rifles CMP has sold from fiscal years 2008 through 2017. Based on CMPs reported sales of 304,233 surplus rifles from fiscal years 2008 through 2017 and revenue generated from these sales of $196.8 million, we determined the average sale of these surplus rifles to be approximately $650 per rifle. According to CMP, as of August 16, 2018, it had 228,791 rifles on hand, of which CMP identified 148,714 as being in sellable condition. We then multiplied the number of rifles available for sale as of August 2018 by $650, assuming the average sales price would remain the same going forward, to obtain the potential future revenue from the sale of surplus rifles. We then added the range of potential surplus M1911 handgun sales to determine a potential range of CMP’s future sales of surplus firearms. Given CMP’s fiscal year 2017 expenses of $15.8 million, and assuming those expenses remained the same, CMP could fund a similar level of operations for several years from the sale of all of the surplus firearms it currently has available to sell. In order to compare CMP’s business operations with those of other federally chartered nonprofit corporations, we focused on CMP’s youth- focused mission and identified eight other youth-focused, federally chartered nonprofit corporations. Specifically, we reviewed 93 federally chartered nonprofit corporations to identify corporations that focused on the education, training, or development of youth. We developed a set of relevant questions and interviewed officials from five of the eight federally chartered nonprofit corporations we identified—the Naval Sea Cadet Corps, the Civil Air Patrol, Big Brothers Big Sisters of America, Future Farmers of America, and the Boy Scouts of America. Of the remaining three corporations, two did not respond to our requests for meetings and the third declined to meet. We posed the same questions to all the corporations’ officials we met with and compared certain aspects of CMP’s business operations with the federally chartered nonprofit corporations regarding governance, organizational structure and relationships, and funding sources. We conducted this performance audit from May 2018 to February 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Army transferred 279,032 surplus rifles to the Civilian Marksmanship Program (CMP) from fiscal years 2008 through 2017. The surplus rifles transfer data characterized rifles with different descriptions for nomenclatures (e.g., M1903, Mossberg M144, and M1917 Enfield) and firearm identification codes. Our analysis determined that the different nomenclatures could be combined into 17 distinct groups because many of the rifles were variants of the same type of .30 caliber rifle or carbine, or .22 caliber rimfire rifle. The 17 types of rifles we identified were grouped together based on make, model, and/or caliber. Through our analysis, we determined that the majority of rifles transferred to CMP by the Army from fiscal years 2008 through 2017 have been surplus M1 rifles. Our analyses determined that 203,644 of the 279,032 surplus rifles transferred to CMP from fiscal years 2008 through 2017 were serviceable M1 rifles. The second largest type of surplus rifles transferred during this period were drill rifles—rifles not capable of firing live or blank rounds of ammunition—although CMP received nearly four times the number of M1s as it did drill rifles. See table 1 for a description of the surplus rifles transferred to CMP by the Army from fiscal years 2008 through 2017. In addition to the contact named above, Marilyn Wasleski, Assistant Director; Scott Behen, Analyst-in-Charge; Mae Jones; Richard Kusman; Amie Lesser; Rebecca Mendelsohn; Mike Shaughnessy; Mike Silver; Carter Stevens; and Roger Stoltz made key contributions to this report.", "summary": "Since 1996, the Army has transferred more than 700,000 surplus rifles and handguns to CMP. The National Defense Authorization Act (NDAA) for Fiscal Year 1996 authorized CMP to sell certain types of surplus Army firearms to U.S. citizens, including M1 .30 caliber rifles. CMP reimburses the Army for the costs to prepare and transport surplus firearms to CMP. The NDAA for Fiscal Year 2018 required the Army during fiscal years 2018 and 2019 to transfer to CMP surplus M1911 .45 caliber handguns, including not fewer than 8,000 in fiscal year 2018 and not more than 10,000 in any fiscal year, and included a provision for GAO to conduct a review of certain matters related to CMP. Among other things, GAO examined (1) the Army and CMP's procedures to address requirements governing the transfer and sale of firearms and (2) CMP's primary sources of revenue, costs and profits, and estimated future revenue associated with the sale of surplus firearms. GAO reviewed applicable federal statutes and agreements between the Army and CMP; analyzed firearms transfer data, and CMP's Internal Revenue Service filings and internal financial documents; and visited both CMP's northern headquarters in Port Clinton, Ohio and its southern headquarters in Anniston, Alabama. The Civilian Marksmanship Program (CMP) is a federally chartered, nonprofit corporation that, among other things, instructs U.S. citizens in marksmanship; promotes practice and safety in the use of firearms; and sells surplus Army firearms (see figure), ammunition, repair parts, and other supplies. CMP is required to give priority to activities that benefit firearms safety, training, and competition for youth and that reach as many youth participants as possible. CMP also charges fees for individuals to participate in some of its programs. The Army and CMP have established procedures to address federal requirements for the transfer and sale of surplus firearms. Both organizations established procedures to carry out the transfer of surplus Army firearms as identified in a 2016 Memorandum of Understanding (MOU) and a 2018 Memorandum of Agreement, both between the Army and CMP. To address requirements for selling surplus firearms, CMP uses a combination of procedures, including an application requiring prospective customers to provide proof of citizenship and age, among other things, and a check against the National Instant Criminal Background Check System. Per the MOU, the Army's Tank-automotive and Armaments Command oversees the Army's costs and reimbursements from CMP for certain costs associated with storing, transporting, and administering the transfer of surplus firearms. The primary source of CMP's revenues from fiscal years 2008 through 2017 was from the sale of surplus rifles, which, according to CMP's internal financial documents, generated $196.8 million in revenue. CMP also sold commercial ammunition and memorabilia, which, according to the same documents, generated $76.4 million in revenue. Further, according to its Internal Revenue Service filings for this time frame, CMP reported earning $49.8 million in interest and dividends from its investment account. CMP began selling surplus M1911 handguns in November 2018 and had just begun generating revenue from these sales at the time of GAO's review. The profit that CMP realized from the sales of surplus rifles could not be determined because CMP's methodology to calculate expenses did not account for all of CMP's costs associated with the sale of these rifles. GAO estimates future sales of CMP's surplus handgun and rifles currently available for sale could generate as much as $104.9 million, or enough to fund CMP's operations for several years. Further, as of September 30, 2017, CMP reported having cash of $3.6 million, and an investment account valued at $188.6 million. This could also allow CMP to continue operations for several years.", "document_type": "gao"}
{"report": "In 1999, DOE issued a report stating that, based on experience from a decade of planning and conducting cleanup work at the sites for which it is responsible, complete restoration to levels acceptable for unrestricted use could not be accomplished at many of its sites. According to the report, a variety of hazards would remain at many DOE sites after these sites had been cleaned up in accordance with applicable requirements. These hazards include long-lived radionuclides left in place in soils or contained in on-site disposal cells and residual contaminants in surface water and groundwater. The report cited technical challenges—such as lack of existing technology for completely removing some types of waste—and economic limitations—such as prohibitive costs to employ available technology—as reasons why these hazards would remain. As a result, DOE reported that long-term management would be needed at these sites to ensure that the cleanup remedies—i.e., the actions, systems, or other measures put in place to clean up a site—would protect human health and the environment from these hazards into the future. Several DOE organizations, including the Office of Environmental Management (EM), were responsible for long-term management of post- cleanup sites until the department established LM in 2003. As of the end of fiscal year 2019, LM had assumed responsibility for 100 sites across the United States, including sites in Alaska and Puerto Rico (see fig. 1). Several different entities conducted cleanup of sites before LM assumed responsibility for the sites. These different entities conducted cleanup under a variety of authorities: EM. Established in 1989, DOE’s EM is responsible for the cleanup of legacy waste that resulted from the development and production of nuclear weapons and government-sponsored nuclear energy research dating back to World War II and the Cold War. Such waste includes radioactive waste, spent nuclear fuel and nuclear material, and contaminated soil and water, among other things. EM cleaned up 83 of the 100 sites that are now within LM’s portfolio. Key laws that governed EM’s cleanup of these sites include the Comprehensive Environmental Response, Compensation, and Liability Act of 1980 as amended (CERCLA); the Resource Conservation and Recovery Act of 1976 as amended (RCRA); and Title I of the Uranium Mill Tailings Radiation Control Act of 1978 (UMTRCA). Title I of UMTRCA authorizes a cleanup program for uranium mill tailings sites—which produced uranium for nuclear weapons and other defense purposes— that were no longer operational as of 1978, the year of the law’s enactment. DOE is generally responsible for financing the cleanup of these sites. EM also cleaned up sites that are now within LM’s portfolio under the Formerly Utilized Sites Remedial Action Program (FUSRAP). This program was established in 1974 to identify, investigate, and clean up sites where radioactive contamination remained from Manhattan Project and early Atomic Energy Commission operations. EM was responsible for cleaning up FUSRAP sites until 1997, when Congress directed the U.S. Army Corps of Engineers (USACE) to assume responsibility for the cleanup work of the remaining designated FUSRAP sites. USACE. USACE cleaned up 10 FUSRAP sites that are now within LM’s portfolio. Under a memorandum of understanding signed by DOE and USACE in 1999, DOE is responsible for the long-term management of FUSRAP sites after USACE completes cleanup. Key requirements that govern USACE’s cleanup of FUSRAP sites include CERCLA and the National Oil and Hazardous Substances Pollution Contingency Plan. Private licensees. LM’s portfolio includes seven sites cleaned up by private licensees, i.e., commercial operators who were permitted to operate uranium mills or other facilities under a license from the Nuclear Regulatory Commission (NRC). In all except one case, private licensees cleaned up these sites under Title II of UMTRCA, which assigned responsibility to the licensee for reclamation of uranium mill sites operating on or after the law’s enactment in 1978. When a private licensee has completed all cleanup requirements, NRC approves transfer of a site to LM for long-term management. Cleanup activities conducted by these entities included decontaminating, decommissioning, and demolishing buildings; containing and disposing of a variety of hazardous and radioactive wastes; excavating and stabilizing contaminated soil; constructing engineered disposal cells for contaminated materials; containing and treating contaminated surface water and groundwater; and preparing the land for future public, industrial, or commercial use. Depending on the legal and regulatory framework governing cleanup, other agencies or groups may have played a role in setting cleanup standards and helping to select a site’s cleanup remedy. For example, sites cleaned up under Title I of UMTRCA must meet regulatory cleanup standards established by the Environmental Protection Agency (EPA). For certain sites cleaned up under CERCLA and RCRA, DOE has entered into agreements with EPA and the relevant state regulator regarding the necessary cleanup actions, and EPA and the state have provided input in selecting the cleanup remedy. As cleanup of a site nears completion, LM works with the entity responsible for cleanup to prepare the site for transition into LM’s portfolio. The transition process for a given site may take up to 5 years, during which time LM and the cleanup entity develop a long-term surveillance and maintenance plan. Depending on the authority under which a site has undergone cleanup, this plan may require approval by regulators such as EPA or NRC. Other transition responsibilities include identifying and preserving records and checking that administrative institutional controls and other real property instruments are in place. DOE considers site cleanup to be complete when, among other things, short-term cleanup activities have been completed and long-term cleanup measures, such as groundwater treatment, are in place. According to a DOE document, ongoing groundwater remediation continues at many sites after the official completion of cleanup because of the long timeframes required to capture and remediate contaminated groundwater. Once LM acquires a site, it places each site into one of three categories based on the actual or anticipated long-term surveillance and maintenance activities associated with the site. LM has nine “category 3” sites, which require the most intensive surveillance and maintenance due to the extent of residual contamination, according to LM officials. These sites typically have an ongoing remediation system—such as a groundwater treatment system, according to officials—that LM must monitor and maintain. LM has 49 “category 2” sites, which require routine inspection, monitoring, and maintenance. LM has 42 “category 1” sites, which require management of records or stakeholder requests for information. LM also maintains a list of 52 sites that, as of September 2019, are expected to transition into its portfolio over the next three decades. Figure 2 illustrates sites’ transition from cleanup entities and their categorization. Appendix II provides additional details about the current sites in LM’s portfolio as of fiscal year 2019, and appendix III provides details about sites that, as of September 2019, are scheduled to transition to LM by 2050. According to LM officials, LM does not have a schedule or process for retiring sites from its portfolio. Depending on the sites’ clean- up standards and intended reuse, LM will likely be managing some sites for centuries. LM’s budget includes funding for other activities that are not directly associated with its 100 sites. These activities include conducting an inventory of abandoned defense-related uranium mines, overseeing pensions and post-retirement benefits for former contractor workers at closed DOE sites, and leading and coordinating DOE’s environmental justice activities. As of fiscal year 2019, LM’s overall budget was about $159 million. Federal accounting standards require agencies that are responsible for cleaning up contamination to estimate future cleanup and waste disposal costs and to report such costs in their annual financial statements as environmental liabilities. According to these standards, environmental liability estimates are to include probable and reasonably estimable costs of cleanup work. Environmental liability estimates do not include cost estimates for work for which reasonable estimates cannot currently be generated, such as cleanup costs at sites where no feasible remedy exists, according to the standards. In fiscal year 2019, DOE reported $505 billion in environmental cleanup and disposal liabilities, of which about $64 billion are categorized by DOE as “other legacy environment” costs. LM’s environmental liability is part of this category, along with several other types of environmental liability costs. LM estimated its environmental liability in fiscal year 2019 at $7.35 billion, an amount that has been relatively stable over the last 5 years. However, LM expects its environmental liability to increase as it acquires additional sites, according to LM officials. According to LM financial data, LM’s environmental liability estimate in fiscal year 2019 was $7.35 billion. LM’s guidance defines its environmental liability as an estimate of life-cycle costs associated with five main activities—determined by DOE—occurring over 75 years (see fig. 3). LM develops guidance on how its site managers should estimate their sites’ environmental liability. In accordance with this guidance, site managers are to develop estimates of the direct costs over the upcoming 75-year period. They are also to determine a certain amount of contingency to account for potential changes in LM’s project scope because of unknown and unpredictable events over the upcoming 75- year period. As shown in figure 4, LM activities related to long-term surveillance and maintenance of its sites accounted for about $3 billion—or 40 percent—of its fiscal year 2019 environmental liability. LM activities related to program direction and to archives and information management each accounted for about 23 percent and 22 percent, respectively, of LM’s fiscal year 2019 environmental liability, and activities related to asset management and to communication, education, and outreach combined for about 15 percent. Of LM’s approximately $3 billion in costs for long-term surveillance and maintenance, LM’s category 3 sites—the nine sites that require the most intensive level of management—accounted for almost half of these estimated costs (see fig. 5). The Rocky Flats site in Colorado accounted for the largest share of this portion of the liability (about $452 million), and the Fernald Preserve site in Ohio accounted for the second-largest share (about $308 million). Long-term surveillance and maintenance responsibilities for category 1 and category 2 sites, transition costs associated with sites that LM will acquire in future years, and other program-wide activities—such as exploring new technologies and operating a laboratory—accounted for the remaining share (about $1.5 billion) of LM’s environmental liability related to long-term surveillance and maintenance. LM’s total environmental liability has generally remained stable in recent years, although there have been some notable fluctuations at individual sites. In fiscal years 2015 through 2018, LM’s total environmental liability remained between $6 billion and $7 billion per year, and increased to slightly over $7 billion in fiscal year 2019 (see fig. 6). Most notably, LM’s total environmental liability increased by about $2 billion (about 41 percent) between fiscal years 2014 and 2015. LM officials attributed this increase to adopting a more thorough approach for estimating future costs associated with sites scheduled to be transferred from USACE under FUSRAP. LM officials said that, before fiscal year 2015, LM had used a standard cost estimate for all of USACE’s sites, which resulted in an underestimate of the associated liability. According to LM officials, in fiscal year 2015 LM began estimating costs based on individual sites’ specific conditions, which allowed LM to capture more potential costs. Similar to LM’s overall environmental liability, the long-term surveillance and maintenance portion of LM’s environmental liability has generally remained stable in recent years, though individual sites have seen some notable changes. From fiscal year 2015 through 2018, LM’s environmental liability related to long-term surveillance and maintenance remained between about $3 billion and $3.5 billion. Similar to LM’s overall environmental liability, the long-term surveillance and maintenance portion of LM’s liability saw a more significant increase between fiscal years 2014 and 2015, from about $2.2 billion to about $3.4 billion. At the site level, of LM’s nine category 3 sites, the Fernald Preserve and Mound sites in Ohio are examples of sites that have had mostly steady decreases from fiscal year 2014 to 2019, which LM officials attributed in part to adjustments to groundwater treatment strategies at Fernald Preserve as well as transferring ownership of most of the Mound site to another party. In contrast, several other sites (including Rocky Flats and Grand Junction in Colorado and Weldon Spring in Missouri) saw overall decreases from fiscal year 2014 to 2016 followed by steady increases from fiscal year 2016 to 2019, which LM officials generally attributed to costs of site maintenance at Rocky Flats, construction at Weldon Spring, and planning activities for the potential closure of the disposal cell at Grand Junction. LM officials provided additional details on specific factors driving sites’ changes in environmental liability. For example: At the Fernald Preserve site, the long-term surveillance and maintenance liability has decreased overall from about $367 million in fiscal year 2014 to about $308 million in fiscal year 2019 (about a 16 percent decrease). The site manager for Fernald attributed this decrease to improvements in the site’s groundwater treatment strategy. In 2014, LM made changes to optimize the site’s “pump-and- treat” system (which brings contaminated water above ground so that it can be treated and contaminants removed) by increasing pumping from the wells in the portion of the site with the most contamination, according to the site manager. Further, the site manager said that this change increased the amount of water coming from the more contaminated areas, making the water treatment more efficient and cost-effective in the long-term. At the Mound site, the long-term surveillance and maintenance liability has decreased from about $124 million in fiscal year 2014 to about $68 million in fiscal year 2019 (about a 45 percent decrease). According to LM officials, this decrease is in part due to a transfer in ownership. Specifically, LM transferred ownership of the majority of the site to the Mound Development Corporation to sell or lease parcels of the land to third parties for commercial use. Transferring ownership meant that LM gave up some of its responsibilities and their associated costs (such as maintenance and repairs at buildings that are now privately owned), although it continues to fulfill ongoing groundwater treatment and records management responsibilities. At the Rocky Flats site, the long-term surveillance and maintenance liability has increased substantially since fiscal year 2016, from about $269 million to about $452 million in fiscal year 2019 (about a 68 percent increase). According to the site manager for Rocky Flats, this increase can be attributed to additional costs needed to repair aging infrastructure. Specifically, a landfill on the site, which was constructed in the 1950s, has been damaged by erosion in recent years, and LM is currently undertaking a large-scale project to repair and stabilize it after previous repairs failed to provide a long-term fix. This project, which is due to be completed in the summer of 2020, includes installing about 260 steel anchors of up to 95 feet in length into the soil around the landfill. These anchors are intended to keep the soil intact while drains route groundwater away from the areas of the landfill that are particularly vulnerable to erosion. LM’s environmental liability is likely to grow as it acquires more sites in future years, even as LM takes steps to reduce the environmental liability associated with its current sites, according to LM officials. According to an LM document, as of September 2019, LM is scheduled to acquire 52 additional sites by 2050, including six category 3 sites, 45 category 2 sites, and one category 1 site. Since LM does not account for the environmental liability related to long-term surveillance and maintenance for a portion of its sites until it acquires them, LM officials could not tell us by how much its total environmental liability will increase as a result of acquiring these sites. However, officials said that some sites transitioning to LM in the future will be increasingly complex, which will likely mean increased long-term surveillance and maintenance costs. In particular, one official told us that the FUSRAP sites LM is set to acquire from USACE will be larger and have more extensive residual contamination than FUSRAP sites that LM had previously acquired. As a result, these sites will likely require LM to undertake more extensive and costly long-term surveillance and maintenance activities, according to this official. At the same time, LM officials said they are taking steps to help reduce the environmental liability at LM’s current sites, such as exploring ways to improve the cost-effectiveness of managing residual groundwater contamination. For example: At the Shiprock site in New Mexico, LM has initiated an environmental assessment to evaluate the impacts of removing an evaporation pond into which contaminated groundwater is being pumped, according to the site manager. The site manager also told us that removing this pond could mean reducing the scope of the site’s water pumping activities and ultimately adopting a different groundwater treatment strategy that could prove to be more efficient. Further, the site manager said that this removal would result in reduced long-term surveillance and maintenance costs associated with ongoing repairs to the pond. At the Tuba City site in Arizona, LM is conducting an environmental assessment to weigh options for a new groundwater treatment strategy. According to the site manager, the current strategy, which involves injecting clean water into the site’s contaminated aquifer to flush out contamination, does not cost-effectively address the root cause of the groundwater contamination. Among other options, LM may use its assessment to seek alternate concentration limits accompanied by restrictions to grazing and water use, which LM officials said could be a cost-effective way to manage residual contamination. LM officials we interviewed identified a number of challenges that LM faces in providing long-term surveillance and maintenance of sites. In particular, officials identified challenges related to three main areas: (1) the performance of remedies on its sites, (2) environmental conditions, and (3) new requirements and regulations. LM is taking some actions to address the challenges that officials identified. However, it has not planned for how to address challenges with remedies at some sites that may require additional cleanup work outside the scope of its expertise and resources, and it has not developed plans to assess and mitigate challenging environmental conditions that may become more frequent or intense because of climate change. According to LM officials, LM faces challenges with cleanup remedies not performing as predicted or intended at some sites. For example: At the L-Bar site in New Mexico, officials told us that the disposal cell, which was constructed by a private licensee under UMTRCA Title II and holds about 2.1 million tons of radioactive mill tailings, began experiencing erosion problems shortly after NRC transferred the site to LM in 2004. This erosion is threatening to undermine the disposal cell, according to LM officials (see fig. 7). At the Monticello site in Utah, monitored natural attenuation—the groundwater treatment remedy originally agreed to by DOE, EPA, and the Utah state regulator—proved ineffective in meeting cleanup goals within a few years of being implemented and of the site being transferred to LM. As a result, in 2015, LM implemented a pump-and- treat approach that reduced contamination; however, officials told us that the efficacy of this approach has declined over time, and LM is again seeking to change the remedy. To address challenges related to the performance of remedies, LM is currently undertaking a risk analysis effort to rank sites according to several types of risks, including the risk that a site will not attain compliance with cleanup goals or that compliance will not be maintained into the future. According to LM officials, LM plans to use the results of the risk analysis to inform decisions about where to focus resources, to identify systemic technical challenges, and to identify possible opportunities for reducing LM’s environmental liability, such as through technology development. LM is also addressing challenges related to remedy performance by updating some sites’ remedies. For example, LM has implemented an erosion monitoring program for the L-Bar site and, at the Monticello site, is collecting data that could allow it to seek regulatory approval for a new groundwater compliance strategy, according to LM officials. LM officials said that, in general, they consider such updates to be routine and to fall within LM’s mission to provide long-term surveillance and maintenance of these sites. Nonetheless, LM officials told us that as LM acquires additional sites and as remedies age, future challenges related to remedy performance could result in the need for more extensive work, including active cleanup work that is outside the scope of LM’s mission, capabilities, and resources. We found that LM has developed agreements and procedures for addressing such challenges at sites cleaned up by USACE, but has not developed such agreements and procedures for sites cleaned up by EM or by private licensees under Title II of UMTRCA. Specifically, regarding sites cleaned up by USACE under FUSRAP, under the 1999 memorandum of understanding between DOE and USACE, USACE is responsible for carrying out additional cleanup actions when it determines such actions are necessary. In addition, LM guidance related to transition and transfer of FUSRAP sites includes examples of situations in which LM would return a site to USACE for additional cleanup, such as situations in which routine monitoring identifies new areas of contamination. Conversely, for sites where EM was responsible for active cleanup, a 2005 memorandum co-signed by the leadership of LM and EM includes a brief statement about the need for LM and EM to coordinate in instances of “significant remedy failures.” LM officials told us that structural or engineering damage could signify evidence of a “significant remedy failure,” but said that such criteria have not been documented. They also said that LM has not defined a process by which such failures would be addressed. Finally, LM officials said that there is no mechanism in place under UMTRCA for LM to return a site to NRC or to seek recovery of costs from a private licensee for any additional cleanup that needs to be done. According to agency officials, LM has not developed agreements or procedures for addressing challenges that require active cleanup work at sites cleaned up by EM because LM has not yet encountered such instances at any of its sites. They also noted that LM has been more focused on long-term surveillance and maintenance and the process of transitioning sites into its portfolio from EM and private licensees, rather than a process for moving sites back to these entities if a cleanup remedy fails. However, under federal internal control standards, management is to design control activities to achieve objectives and respond to risks, such as by clearly documenting internal control in management directives, administrative policies, or operating manuals. By working with EM and NRC to develop agreements and procedures for identifying and addressing circumstances at LM sites that require new cleanup work beyond the scope of LM’s mission, capabilities, and resources, LM can help ensure mitigation by the most appropriate entity of the risks to human health and the environment that such instances would present. LM faces challenges with environmental conditions at the sites—some of which may become more frequent or intense—and, according to its mission, LM must react to these challenges to ensure the sites remain protective of human health and the environment. For example: At the Rocky Flats site in Colorado, officials told us that extreme rainfall events over the past few years have caused soils covering an on-site landfill to “slump,” or slip downhill. In particular, rainfall during 2015—the site’s wettest year on record, according to LM officials— caused a 20-foot slump in the landfill. The Boiling Nuclear Superheater site in Puerto Rico and the Pinellas County site in Florida were both in the path of Hurricane Irma in 2017, though neither site sustained substantial damage. At the Weldon Spring site in Missouri, the site manager said that tornadoes pose a risk to the site’s infrastructure, and that a strong tornado in 2013 damaged the site’s interpretive center. To address challenges related to environmental conditions, LM has been repairing damages caused by extreme weather events. For example, at the Rocky Flats site, LM is undertaking a major project to repair and stabilize its aging landfill, as discussed earlier. At the Weldon Spring site, LM installed a tornado shelter in 2014 and is currently building a new interpretive center. In addition, according to the 2020 LM Site Sustainability Plan, LM has taken a number of steps to implement emergency and security measures, such as completing emergency drills and tabletop exercises. The U.S. Global Change Research Program—which coordinates and integrates the activities of 13 federal agencies that research changes in the global environment and their implications for society—reported in its November 2018 Fourth National Climate Assessment that climate change is playing a role in the increasing frequency of some types of extreme weather, such as extremely heavy rainfall and hurricanes; these are environmental conditions that have presented challenges at LM sites. The assessment reported that climate models are consistent with temperature and precipitation extremes becoming more frequent, more intense, or longer in duration, which may make certain natural disasters more frequent or more intense. As a result of the significant risks posed by climate change and the nation’s fiscal condition, in February 2013, we added Limiting the Federal Government’s Fiscal Exposure by Better Managing Climate Change Risks to our list of areas at high risk for fraud, waste, abuse, and mismanagement, or most in need of transformation. In our March 2019 update to this high-risk area, we reported that the federal government needs to improve the resilience of facilities it owns and operates, and land it manages, against the effects of climate change. In addition, in October 2019, we found that EPA needs to improve management of risks from climate change at Superfund sites where remedies may need to be operational indefinitely (see sidebar). We Found That EPA Should Take Additional Actions to Manage Risks from Climate Change Superfund is the federal government’s principal program to address sites with hazardous substances. It was established by the Comprehensive Environmental Response, Compensation, and Liability Act of 1980 and is administered by the Environmental Protection Agency (EPA). EPA lists some of the most seriously contaminated sites on the National Priorities List (NPL) and has recorded over 500 contaminants at those sites. Some NPL sites are located at federal facilities, where departments such as the Department of Energy are responsible for cleanup. However, most NPL sites are nonfederal, where EPA generally carries out or oversees the cleanup conducted by one or more potentially responsible parties. In October 2019, we reported that available federal data on flooding, storm surge, wildfires, and sea level rise suggest that about 60 percent of all nonfederal NPL sites are located in areas that may be impacted by these potential climate change effects. According to EPA officials, remedies at nonfederal NPL sites may have to be operational indefinitely, during which time the potential effects of climate change may become more extreme. We found that EPA has taken some actions to manage risks from the potential impacts of climate change effects at nonfederal NPL sites, but that its actions did not fully align with essential elements of enterprise risk management. For example, we found that EPA officials do not always have direction to ensure that they consistently integrate climate change information into site-level risk assessments and risk response decisions, according to EPA officials. Without providing such direction, EPA cannot ensure that remedies at nonfederal NPL sites will protect human health and the environment in the long-term. We made four recommendations to EPA, including that it provide direction on how to integrate information on the potential impacts of climate change effects into risk assessments and risk response decisions at nonfederal NPL sites. EPA agreed with one recommendation and disagreed with the other three. We continue to believe that all four are warranted. LM’s 2016-2025 Strategic Plan acknowledges the challenges posed by climate change. To support the objective of improving the long-term sustainability of environmental remedies, the plan includes a strategy to “assess the effect of climate change on environmental remedies and develop plans to mitigate significant impacts.” However, LM provided minimal information about ongoing or planned efforts to carry out this strategy. Specifically, the 2020 LM Site Sustainability Plan, which officials said provides information about LM’s future plans to adapt to changing climate conditions, includes the term “climate change” one time, in reference to sustainable buildings—not to remedies. The plan describes one pilot project conducted at the Monticello site to evaluate the site’s main climate stressors and capacity to adapt to those stressors, but it does not describe whether or how LM intends to use the results of the pilot project, such as any specific plans to roll out the project to other sites. Aside from the 2020 LM Site Sustainability Plan, LM officials said they have a goal to review sites’ conceptual models, which predict how remedies should perform under different conditions, with the aim of updating the assumptions in the models to better account for real-world conditions. However, LM did not provide details about how it intends to meet this goal, such as a schedule for implementing this review across its sites. According to LM officials, LM has not developed a plan or schedule for reviewing sites’ conceptual models because of competing priorities. In addition, LM officials told us they have not assessed the effects of climate change or developed plans to mitigate those effects because of a lack of concern about the risks posed by climate change. Specifically, site managers in charge of several of LM’s category 3 sites—including Rocky Flats, which has the highest environmental liability of LM’s 100 sites and is currently implementing the large-scale project described above to address erosion caused by extreme precipitation—told us that they have not assessed the potential effects of climate change on their sites because they do not believe climate change is a concern. Recognizing the federal government’s significant role in managing climate-related disaster impacts, GAO’s Disaster Resilience Framework provides three broad principles that those who oversee or manage federal efforts can consider when analyzing opportunities to enhance their contribution to national disaster resilience. For instance, under the information principle, the framework states that accessing authoritative, understandable information can help decision makers to identify current and future risk and the impact of risk-reduction strategies. In addition, the integration principle states that integrated analysis and planning can help decision makers take coherent and coordinated resilience actions. By developing plans to assess the effect of climate change on LM’s sites and to mitigate any significant impacts and, as part of these plans, incorporating principles from GAO’s Disaster Resilience Framework, as appropriate, LM could better ensure that its remedies will protect human health and the environment in the long term. According to LM officials, LM faces challenges when regulators update or adopt new requirements and regulations for contaminants, meaning that remedies in place when LM received a site may no longer meet standards. For example: At several sites, such as the Fernald Preserve and Mound sites in Ohio and the Rocky Flats site in Colorado, LM officials told us they are investigating for per- and polyfluoroalkyl substances (PFAS) or vapor-forming chemicals, which are emerging contaminants that EM was not required to address when cleaning up these sites. EPA has published information regarding potential impacts to human health and the environment from these and other emerging contaminants. Federal regulatory standards issued by EPA in the future could affect LM sites. At the Bluewater site in New Mexico, LM officials said that the state recently adopted an updated, more stringent uranium drinking water standard. Under the new standard, the area of groundwater that is considered contaminated is much larger than the area of groundwater considered contaminated under the standard in place when NRC approved transfer of the site to LM, according to officials. To address challenges related to new requirements and regulations, LM is monitoring changes to federal and state standards. For example, LM participates in interagency working groups, such as a PFAS working group led by DOE’s Office of Environment, Health, Safety, and Security. Participation in the working groups helps LM monitor the evolution of a federal PFAS regulatory standard, according to LM officials. In addition, LM officials told us that they routinely review state and federal regulatory changes, with the aim of providing sites time to prepare for any changes. LM also evaluates its surveillance and maintenance practices against current regulatory and best management requirements to identify any gaps. For instance, in 2018, the contractor that provides support services to LM reviewed site management practices listed in UMTRCA Title I and II sites’ site management plans against current regulatory requirements. The review identified a number of discrepancies between practices and requirements. For example, the review found that some site management plans were developed many years ago and had not been updated to reflect changes in remedy requirements. LM indicated it planned to take steps to address the discrepancies identified by this review. For example, LM is planning to update its site management plans to include the most current remedy requirements for each site. At many sites contaminated from nuclear weapons production and nuclear energy research dating back to World War II and the Cold War, completely eliminating risks to human health and the environment is unlikely. LM is responsible for protecting human health and the environment from the risks that remain after other entities have cleaned up these sites, and its mission is long-term—LM sites will require surveillance and maintenance for hundreds or even thousands of years. Over this period, the likelihood that cleanup remedies will experience performance challenges is high, and these challenges may exceed the scope of LM’s mission, capabilities, and resources. LM acquires sites from several cleanup entities, but has not developed agreements or procedures with EM or NRC for addressing challenges that require new, active cleanup work. By working with EM and NRC to develop agreements and procedures for identifying and addressing circumstances at LM sites that require new cleanup work beyond the scope of LM’s mission, capabilities, and resources, LM can help ensure mitigation by the most appropriate entity of the risks to human health and the environment that such instances would present. Environmental conditions also present challenges to LM’s sites, and some of these conditions may become more frequent or intense in the future, according to the 13-agency U.S. Global Change Research Program. To ensure the long-term protectiveness of remedies, it is important for LM to understand how climate change may affect its sites. LM’s strategic plan includes a strategy to assess the effects of climate change on its sites, but the agency provided minimal information about how it plans to carry out this strategy. GAO’s Disaster Resilience Framework outlines a set of principles that those who oversee or manage federal efforts can consider when analyzing opportunities to enhance their contribution to national disaster resilience. By developing plans to assess the effect of climate change on LM’s sites and to mitigate any significant impacts, and, as part of these plans, incorporating principles from GAO’s Disaster Resilience Framework, as appropriate, LM could better ensure that its remedies will protect human health and the environment in the long term. We are making three recommendations to DOE: The Secretary of Energy should direct the Director of LM and the Assistant Secretary of the Office of Environmental Management to develop agreements and procedures for identifying and addressing circumstances at LM sites that require new cleanup work beyond the scope of LM’s mission, capabilities, and resources. (Recommendation 1) The Secretary of Energy should direct the Director of LM to work with the Nuclear Regulatory Commission to develop agreements and procedures for identifying and addressing circumstances at LM sites that require new cleanup work beyond the scope of LM’s mission, capabilities, and resources. (Recommendation 2) The Secretary of Energy should direct the Director of LM to, as called for in LM’s strategic plan, develop plans to assess the effect of climate change on LM’s sites and to mitigate any significant impacts. These plans should incorporate principles from GAO’s Disaster Resilience Framework, as appropriate. (Recommendation 3) We provided a draft of this report to DOE for comment. In its comments, reproduced in appendix IV, DOE agreed with our three recommendations. In its letter, DOE officials stated that in response to our first two recommendations, it plans to work with DOE’s Office of Environmental Management and the Nuclear Regulatory Commission to develop agreements and procedures for identifying and addressing new cleanup work beyond LM’s mission scope of long-term stewardship. DOE officials also stated that in response to our third recommendation, LM will develop site assessment and mitigation plans, taking into account any significant effects of climate change and incorporating principles from GAO’s Disaster Resilience Framework, as appropriate. DOE also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committee, the Secretary of Energy, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Rocky Flats site (Colorado) Shiprock Disposal site (New Mexico) Tuba City Disposal site (Arizona) Weldon Spring site (Missouri) DOE considers site cleanup to be complete when, among other things, short-term cleanup activities have been completed and long-term cleanup measures, such and groundwater treatment, are in place. DOE Office of Environmental Management (EM) Planned transfer in FY 2022 Durita Disposal site East Tennessee Technology Park site Gas Hills East Disposal site Gas Hills North Disposal site Ray Point Disposal site Split Rock Disposal site Planned transfer in FY 2023 Bear Creek Disposal site Hazelwood site private licensee U.S. Army Corps of Engineers (USACE) David C. Trimble, (202) 512-3841 or trimbled@gao.gov. In addition to the contact named above, Amanda K. Kolling (Assistant Director), Katherine Killebrew (Analyst in Charge), and Rachel Pittenger made key contributions to this report. Also contributing to this report were Mark Braza, Ellen Fried, Susan J. Irving, Richard Johnson, Keegan Maguigan, Katrina Pekar-Carpenter, Dan Royer, and Doris Yanger.", "summary": "After over 70 years of nuclear weapons production and energy research at hundreds of sites across the country, DOE faces over $500 billion in environmental liabilities associated with cleanup of hazardous contamination and long-term management of these sites. LM is responsible for the portion of these liabilities associated with long-term management of sites after active cleanup has been completed. LM oversees 100 sites across the country. Depending on the sites' clean-up standards and intended reuse, LM will likely be managing some sites for centuries. Senate Report 116-48 accompanying the National Defense Authorization Act for fiscal year 2020 includes a provision for GAO to review LM's operations, including the nature of its environmental liability. This report examines (1) LM's environmental liability, and (2) any challenges LM faces in managing its sites and how it is addressing those challenges. GAO analyzed data on LM's environmental liability; interviewed officials at LM headquarters and those responsible for the nine sites requiring the most intensive level of management; and reviewed relevant policies, procedures, and guidance. The environmental liability of the Department of Energy's (DOE) Office of Legacy Management (LM) was estimated at $7.35 billion in fiscal year 2019 and, according to LM officials, is expected to grow as LM acquires more sites (see figure for LM's current sites). Long-term surveillance and maintenance activities associated with radioactive and hazardous waste, such as treating residual groundwater contamination, account for about 40 percent of the costs. LM's environmental liability has generally remained stable over the past 5 years. As of September 2019, LM is scheduled to receive 52 additional sites by 2050, and officials expect LM's environmental liability to grow as a result. Officials said LM is taking steps to reduce its environmental liability at its current sites, such as exploring alternative approaches for reducing residual contamination. LM officials identified challenges in providing long-term surveillance and maintenance of sites related to: (1) the performance of remedies that contain or reduce contamination, (2) environmental conditions, and (3) new regulatory requirements. LM is taking some actions to address these challenges. For example, at its Rocky Flats, Colorado, site, LM is repairing an aging landfill that was damaged by extreme rainfall events. However, LM has not yet planned for how to address challenges at some sites that may require new cleanup work that is not in the scope of LM's expertise and resources. By developing agreements and procedures with the entities that would be responsible for conducting this new cleanup work, LM can help mitigate risks to human health and the environment. In addition, LM has not made plans to assess the effects of climate change on its sites or to mitigate those effects, as called for in its strategic plan. By developing plans to assess the effect of climate change on its sites and to mitigate any significant impacts, LM could better ensure that its remedies will protect human health and the environment in the long term. GAO is making three recommendations, including that DOE develop agreements and procedures for circumstances that require new cleanup work and that it develop plans to assess and to mitigate the effects of climate change on its sites. DOE agreed with all three recommendations.", "document_type": "gao"}
{"report": "The Aviation and Transportation Security Act designated TSA as the primary federal agency responsible for security in all modes of transportation. Public and private transportation entities have the principal responsibility to carry out safety and security measures for their services. As such, TSA coordinates with these entities to identify vulnerabilities, share intelligence information, and work to mitigate security risks to the transportation modes. See table 1 for examples of the entities TSA works with to secure the various surface transportation modes. TSA’s Surface Programs’ Program, Project, or Activity (Surface Programs account) supports TSA programs that are to protect the surface transportation system. According to DHS’s Congressional Budget Justifications, this account received about $113 million on average annually from fiscal years 2009 through 2018, about 1.5 percent of TSA’s average annual appropriation of more than $7 billion. During that time, the appropriations directed to the Surface Programs account ranged from about $63 million to nearly $135 million annually. For example, in fiscal year 2018, TSA’s Surface Programs account received about $129 million, which was less than 2 percent of TSA’s appropriation (see figure 1). In addition, the Surface Programs account staff (full-time equivalents) ranged from 353 to 843 annually from fiscal years 2009 through 2018, consistently representing between 0.68 and 1.53 percent of TSA’s total staff. I-STEP was created in response to provisions in the Implementing Recommendations of the 9/11 Commission Act of 2007. According to PPE, the I-STEP program offers three main services: Exercise Management Services assist transportation operators, emergency responders, local law enforcement, and government officials in enhancing security preparedness and resilience; Training Support Services help partners improve security awareness, training gaps, security plans, emergency procedures, and incident management skills; and Security Planning Tools and Services help partners gain an understanding of transportation security lessons learned and best practices to inform risk-based decision-making. The program conducts multi-agency, multi-jurisdictional activities ranging from seminars to full-scale exercises. Seminars provide a starting point for industry stakeholders developing or making major changes to their plans and procedures. Full-scale exercises deploy personnel and resources for real-time scripted events that focus on implementing and analyzing plans, policies, and procedures. The voluntary exercises are conducted across surface transportation modes including mass transit, passenger and freight rail, highway, and pipeline. TSA’s Surface Programs account received $123 million in fiscal year 2017 and $129 million in fiscal year 2018, according to DHS. Surface activities are primarily carried out by three TSA offices—Security Operations; Law Enforcement/Federal Air Marshal Service; and Policy, Plans, and Engagement. TSA reported that these offices were collectively allocated about 99 percent of the funding in TSA’s Surface Programs account in fiscal year 2017 and 93 percent in fiscal year 2018. Security Operations (SO). This office is to provide risk-based security that includes regulatory compliance and other programs designed to secure transportation. Within SO, surface transportation security inspectors, known as surface inspectors, conduct a variety of activities to implement TSA’s surface transportation security mission. These activities are to include (1) regulatory inspections for freight and passenger rail systems, (2) regulatory Transportation Worker Identification Credential inspections, and (3) non-regulatory security assessments and training which surface transportation entities participate in on a voluntary basis. Law Enforcement/Federal Air Marshal Service (LE/FAMS). This office is to conduct protection, response, detection, and assessment activities in transportation systems. For example, LE/FAMS administers the Visible Intermodal Prevention and Response (VIPR) program. Since late 2005, TSA has deployed teams to conduct VIPR operations as a way to augment security of and promote confidence in surface transportation systems. These capabilities can include random bag searches and law enforcement patrols at mass transit and passenger rail systems to deter potential terrorist threats. Policy, Plans, and Engagement (PPE). This office is to develop and coordinate both domestic and international multimodal transportation security policies, programs, directives, strategies and initiatives, while overseeing engagement with industry stakeholders and associations. For example, each modal section within PPE—mass transit, passenger and freight rail, highway, pipeline, and maritime—is to be responsible for outreach to their respective industry and with federal security partners. Their primary role is to align industry interests and actions with the TSA mission. The modes are to share intelligence and information with the industry to develop a shared understanding of risks, conduct vulnerability gap analysis, develop security policy, share best practices, provide risk mitigation and training tools, and conduct drills and exercises. These TSA offices further allocate surface program resources within their respective offices to carry out surface transportation activities (see table 2). Within PPE’s Surface Division, PPE reported allocating six Surface Program account staff to each surface transportation mode office— mass transit and passenger rail, freight rail, highway and motor carrier, and pipeline—in fiscal years 2017 and 2018. TSA may realign funds within an appropriation account through reprogramming and also has limited authority to realign funds between appropriation accounts through transfers, pursuant to its appropriations acts and subject to notification provisions. According to TSA officials, TSA reprogrammed or transferred the following surface transportation resources enacted from fiscal years 2017 through 2019: In fiscal year 2018, TSA reprogramed $5 million from Surface Programs to Mission Support activities to address security requirements and increase hiring of transportation security officers. Transportation security officers conduct security screening of passengers, baggage, and cargo at airports to prevent any deadly or dangerous objects from being transported onto an aircraft. In fiscal year 2018, DHS transferred $100,000 from the Surface Program account to (1) the Immigration and Customs Enforcement’s Custody Operations account to provide adequate funding for detention beds, (2) Immigration and Customs Enforcement’s Transportation Removal Program account to support transportation and removal activities for migrants, and (3) the U.S. Secret Service’s Protection of Persons and Facilities account to support upgrading protections for the White House. In fiscal year 2019, DHS transferred over $6 million to the Immigration and Customs Enforcement’s Custody Operations and Transportation Removal Program accounts for the same purposes. In fiscal year 2019, TSA reprogrammed $200,000 from Mission Support and Secure Flight to Surface Programs to ensure sufficient funds were available to make payroll payments to employees during the fiscal year 2019 government shutdown. Staff funded from the Surface Programs account may be used for aviation-related activities. For example: TSA funds VIPR teams from the Surface Program account; however, VIPR teams are often used for aviation security activities. TSA’s program guidance stated they use a risk-based approach to prioritize and schedule VIPR program operations. According to TSA, in fiscal year 2017, 41 percent of VIPR program operations were conducted in surface modes and 59 percent were conducted in aviation security. In fiscal year 2018, TSA reported that 61 percent of VIPR program operations were conducted in surface modes and 39 percent were conducted in aviation security. TSA also funds surface inspectors and their supervisors from the Surface Program account; however, surface inspectors can assist with aviation-related activities, as we reported in 2017. At that time, we found that TSA had incomplete information on the total time surface inspectors spent on those activities because of limitations in TSA’s data system. Since then, TSA updated its system to include a field indicating whether the activity was conducted in the surface or aviation mode, demonstrating that TSA has visibility over all activities surface inspectors conduct. In fiscal year 2018, TSA reported that surface inspectors spent about 16 percent of hours on aviation-related activities. TSA’s 2016 Surface Division Internal Operating Procedure details the planning and implementation process of I-STEP, but does not fully identify the roles and responsibilities for key TSA offices or time frames for when those offices should coordinate to support training and exercise planning. PPE has primary responsibility for planning and implementing I-STEP under the procedure and coordinates with other TSA offices to facilitate exercises and accomplish the program’s goals. Specifically, PPE officials stated that SO and the Intelligence and Analysis (I&A) offices, have important roles in helping PPE to plan and conduct tabletop exercises using I-STEP’s online exercise tool to facilitate planning in the field. For example, PPE officials stated that SO conducts external outreach to surface transportation stakeholders to identify participants and exercise locations, and I&A provides intelligence briefings that give background context to participants. The roles and responsibilities of SO and I&A are not captured in the operating procedure in part because program responsibilities have changed since the procedure was issued in 2016. For example, the operating procedure describes PPE’s primary responsibility for industry engagement, but does not discuss SO’s surface inspectors’ role in stakeholder and industry outreach for I-STEP. Specifically, surface inspectors reach out to industry stakeholders to identify participants interested in conducting an exercise. Surface inspectors also help handle logistics, such as coordinating with local responders and stakeholders. However, the operating procedure has not been updated since 2016 to capture this transition of SO responsibilities. In the absence of a policy that clearly defines all current offices that should coordinate and when, PPE may also be missing consistent input and important information from relevant offices across TSA. For example, PPE officials indicated that I&A officials can support I-STEP exercises by providing intelligence briefings, when requested, and can assist at or before initial PPE planning meetings. However, I&A officials stated that they do not typically participate in the PPE planning meetings that help identify and prioritize exercises based on risk-based intelligence documents, because they are not consistently invited to attend. Further, according to I&A officials, they sometimes receive a few weeks’ notice, or no notice at all to prepare intelligence briefings for upcoming exercises. I&A officials explained that while they have supported exercise planning, there is no formal role for the office in the procedure or expected time frames for providing information. Our Standards for Internal Control in the Federal Government states management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. Management then develops the overall responsibilities from the entity’s objectives that enable the entity to achieve its objectives. TSA officials stated that they plan to revise the 2016 Surface Division’s Internal Operating Procedure. This planned revision presents an opportunity to identify and clarify roles and responsibilities for all offices involved in the coordination of the exercise, including when they should to coordinate. TSA allocates resources for surface transportation activities, including I- STEP voluntary training and exercises with system operators and governmental security partners. While PPE coordinates with several offices across TSA to accomplish the program’s goals, coordination guidance could be improved. Although PPE has discussed the roles and responsibilities for offices outside of PPE, how and when these offices should coordinate has not been clearly defined in its sole guidance document. As a result, TSA may be missing input and information from relevant offices. Formalizing planning responsibilities, specifically with I&A, would allow for consistent involvement in the planning process and give analysts more time to prepare intelligence briefings for exercises. Also, with surface inspectors performing stakeholder outreach in addition to PPE’s primary role for industry engagement, formalizing planning and external outreach roles and responsibilities for SO would ensure consistent outreach in the field. We are making the following recommendation to TSA: The TSA Administrator should clarify roles and responsibilities for all offices involved in the coordination of surface transportation exercises, including when these offices are to coordinate, as part of the planned revision of the Surface Division’s Internal Operating Procedure for I- STEP. (Recommendation 1) We provided a draft of this report for review and comment to DHS. DHS provided written comments, which are reproduced in Appendix I. In their comments, DHS concurred with the recommendation and described actions planned to address it, including an estimated timeframe for completion. If fully implemented, these actions should address the intent of the recommendation and better position TSA’s offices to execute roles and responsibilities for planning and implementing I-STEP. TSA also provided technical comments, which we incorporated as appropriate We are sending copies of this report to the appropriate congressional committees and the Acting Secretary of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-8777 or RussellW@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in Appendix II. In addition to the contact named above, Ellen Wolfe (Assistant Director), Amber Edwards (Analyst-in-Charge), Lilia Chaidez, Dominick Dale, Tracey King, Leah Nash, Natasha Oliver, and Michael Silver made key contributions to this report.", "summary": "The global terrorist threat to surface transportation–freight and passenger rail, mass transit, highway, maritime and pipeline systems–has increased in recent years, as demonstrated by a 2016 thwarted attack on mass transit in New Jersey and the 2017 London vehicle attacks. TSA is the primary federal agency responsible for securing surface transportation in the United States. The FAA Reauthorization Act of 2018 includes a provision that GAO review resources provided to TSA surface transportation programs and the coordination between relevant entities related to surface transportation security. This report addresses TSA's: (1) allocation of resources to surface transportation programs for fiscal years 2017 and 2018; and (2) coordination within TSA to implement the Intermodal Security Training and Exercise Program. GAO analyzed TSA data on surface program resources for fiscal years 2017 and 2018, reviewed TSA program guidance, and interviewed TSA officials responsible for implementing the Intermodal Security Training and Exercise Program. This program is intended to assist transportation operators and others in enhancing security through exercises and training. Transportation Security Administration (TSA) reported allocating most of its surface transportation program account, which was $123 million in fiscal year 2017 and $129 million in fiscal year 2018--to three offices (see figure). The surface program account represented about 1.6 percent of the agency's appropriation in both fiscal years, according to Department of Homeland Security data. Security Operations is to conduct regulatory inspections for freight and passenger rail systems, non-regulatory security assessments, and voluntary training. Law Enforcement/Federal Air Marshal Service is to administer the Visible Intermodal Prevention and Response (VIPR) Program to augment the security of and promote confidence in surface transportation systems. Policy, Plans, and Engagement (PPE) is to develop and coordinate security policies, programs, directives, strategies, and initiatives, while overseeing industry engagement. In fiscal years 2017 through 2019, TSA reported using surface program resources for non-surface activities. For example, in fiscal year 2018, TSA reprogrammed $5 million from the Surface Programs account to the Mission Support account to address security requirements and increase hiring of transportation security officers. In that same year, about 39 percent of VIPR operations were conducted in aviation security. TSA has not fully identified coordination roles and responsibilities for its training and exercise program for offices outside of PPE—the office with primary responsibility for the program. PPE coordinates with several other offices to accomplish the program's goals, including the Intelligence and Analysis (I&A) office that provides intelligence briefings that give background context during program exercises. I&A officials explained that while they have supported exercise planning, there is no formal role for the office in the procedure or expected time frames for providing information. As a result, I&A officials stated that they do not typically participate in the PPE planning meetings because they are not consistently invited to attend. In the absence of a policy that clearly defines all current offices that should coordinate and when, PPE may be missing consistent input and important information from relevant offices across TSA. GAO recommends that TSA clarify roles and responsibilities for all offices involved in the coordination of surface transportation exercises, including when these offices are to coordinate. DHS concurred with the recommendation.", "document_type": "gao"}
{"report": "The U.S. electric grid comprises three distinct functions: generation and storage, transmission, and distribution (see fig. 1). Generation and Storage. Power plants generate electric power by converting energy from other forms—chemical, mechanical (hydroelectric or wind), thermal, radiant energy (solar), or nuclear— into electric power. Energy storage, such as batteries or pumped hydroelectric, can improve the operating capabilities of the grid while also regulating the quality and reliability of power. Transmission. The power transmission system connects geographically distant power plants with areas where electric power is consumed. Substations are used to transmit electricity at varied voltages and generally contain a variety of equipment, including transformers, switches, relays, circuit breakers, and system operations instruments and controls. Distribution. The distribution system carries electric power out of the transmission system to industrial, commercial, residential, and other consumers. Three large electric grids, or interconnections, exist in the contiguous United States that collectively constitute the U.S. electric grid: the Eastern Interconnection, Western Interconnection, and Electric Reliability Council of Texas Interconnection (see fig. 2). These interconnections, which extend into parts of Canada and Mexico, operate independently with limited ability to move electric power between them; electric power is produced within an interconnection to meet demand in the same interconnection. The grid is generally considered to be resilient. Historically, grid operators have been able to respond quickly to the adverse consequences of an incident—whether it is damage from a major hurricane or a falling tree—and quickly restore service. In some cases, electricity may be restored long before utilities fully recover from an incident. For example, in instances with physical damage to grid components, such as an event that damages many substations, it could take months or years to fully restore the equipment. The electricity industry has refined its power restoration processes after decades of experience in responding to disaster-related events, but restoration from a cyber-related event may be more challenging. For example, disaster-related events—such as hurricanes—may involve significant lead time before the incident. This allows owners and operators to take preemptive measures to protect their systems, develop restoration plans, and activate personnel. In contrast, cyberattacks may occur without warning, leaving owners and operators no time to prepare for a response. In addition, cyberattacks could target and damage specific types of components or facilities across a dispersed geographic area. Responding to such an attack could be more difficult than to a localized disaster-related event since resources may be geographically distributed rather than concentrated in the same area. Industrial control systems are typically network-based systems that monitor and control sensitive processes and physical functions, such as the opening and closing of circuit breakers on the grid. These systems support the control of electric power generation, transmission, and distribution. System operators—which are sometimes affiliated with a particular utility or sometimes independent and responsible for multiple utility areas—manage electricity flows through these systems. Early industrial control systems operated in isolation, running proprietary control protocols using specialized hardware and software. In addition, many industrial control system components were in physically secured areas, and the components were not connected to IT systems or the internet. However, industrial control systems are changing in ways that offer advantages to system operators but that also make them more vulnerable to cyberattacks. In particular, proprietary devices in these systems are being replaced by cheaper and more widely available devices that use traditional IT networking protocols—including those that support remote access. These newer devices can provide the system operator with more detailed data on the conditions of the transmission and distribution systems and with better tools to observe and manage the grid. Remote access capabilities in the devices can also make them easier to maintain. Further, industrial control systems are being designed and implemented using traditional IT computers and operating systems, which allow corporate business and industrial control system networks to be connected more easily. Nonetheless, cyberattacks on industrial control systems supporting grid operations may require a degree of sophistication and knowledge beyond what is needed to conduct cyberattacks on IT systems. For example, industrial control systems often use operating systems and applications that may be considered unconventional to typical IT personnel. Federal policy and public-private plans establish roles and responsibilities for the protection of critical infrastructure, including the electric grid. Presidential Policy Directive 21, issued in February 2013, shifted the nation’s focus from protecting critical infrastructure against terrorism to protecting and securing critical infrastructure and increasing its resilience against all hazards, including natural disasters, terrorism, and cyber incidents. The directive identified 16 critical infrastructure sectors, such as the energy sector, which includes the grid. In addition, the directive identified energy and communications systems as uniquely critical because of the enabling functions they provide across all sectors. The directive also outlined roles and responsibilities for protecting these sectors. For example: The directive designated DOE as the sector-specific agency for the energy sector. According to the directive, DOE and other sector-specific agencies are responsible for, among other things, collaborating with critical infrastructure owners and operators, identifying vulnerabilities, and helping to mitigate incidents. In addition, the Fixing America’s Surface Transportation Act of 2015 codified DOE’s role as the sector-specific agency for the energy sector and gave DOE the authority to order emergency measures, following a Presidential declaration of a grid security emergency, to protect or restore the reliability of critical electric infrastructure. The Office of Cybersecurity, Energy Security, and Emergency Response is the lead for DOE’s energy sector cybersecurity efforts. The directive called for DHS to coordinate the overall federal effort to promote the security and resilience of the nation’s critical infrastructure. Within DHS, the Cybersecurity and Infrastructure Security Agency’s National Cybersecurity and Communications Integration Center is the lead for cyber and physical infrastructure security. Private-sector critical infrastructure owners and operators are encouraged, but not required, to report cybersecurity incidents to the center. The directive emphasized that critical infrastructure owners and operators are uniquely positioned to manage risks to their individual operations and assets and to determine effective strategies to make them more secure and resilient. The National Infrastructure Protection Plan, updated by DHS in December 2013, among other things, further integrates critical infrastructure protection efforts between government and private sectors. It describes a voluntary partnership model as the primary means of coordinating government and private-sector efforts to protect critical infrastructure. As part of the partnership structure, the designated sector-specific agencies serve as the lead coordinators for the security programs of their respective sectors. The plan also called for each sector to have a government coordinating council, consisting of representatives from various levels of government, and many sectors have a coordinating council consisting of owner-operators of these critical assets or members of their respective trade associations. For example, the Energy Sector Government Coordinating Council has been established (comprising the electricity subsector, as well as the oil and natural gas subsectors), and an Electricity Subsector Coordinating Council has been established to represent electricity asset owners and operators. Cybersecurity, issued in 2013, among other things, addresses the need to improve cybersecurity through information sharing and collaboratively developing and implementing risk-based standards. It called for NIST to lead the development of a framework to reduce cybersecurity risks to critical infrastructure. It also called for sector- specific agencies to develop mechanisms to encourage adoption of the framework. NIST issued its Cybersecurity Framework in 2014 and updated it in April 2018. The framework provides a set of cybersecurity activities, desired outcomes, and applicable references that are common across all critical infrastructure sectors, including the energy sector. The executive branch has taken steps toward outlining a federal strategy for confronting cyber threats—including those facing critical infrastructure such as the grid. For example: Executive Order 13800: Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure, issued in May 2017, required federal agencies to take a variety of actions aimed at improving the cybersecurity of federal networks and critical infrastructure. Among other things, the order required DOE and DHS to assess the potential scope and duration of a prolonged power outage associated with a significant cyber incident, the readiness of the United States to manage the consequences of such an incident, and any gaps or shortcomings in assets or capabilities required to mitigate the consequences of such an incident. The National Cyber Strategy, issued in September 2018, builds upon Executive Order 13800 and describes actions that federal agencies and the administration are to take to, among other things, secure critical infrastructure. For example, one of the strategy’s seven goals is protecting critical infrastructure. To achieve this goal, the strategy outlines a number of priority actions, such as prioritizing risk- reduction across seven key areas, including energy and power. The DHS Cybersecurity Strategy was released in May 2018 with the intent of providing the department with a framework to execute cybersecurity responsibilities during the next 5 years. The plan outlines seven goals the department plans to accomplish in support of its mission related to managing national cybersecurity risks. For example, for the goal of protecting critical infrastructure, the plan outlines a number of objectives and sub-objectives, such as expanding and improving the sharing of cyber threat indicators, defensive measures, and other cybersecurity information. In our 2018 and 2019 updates on government high-risk areas, we reported that these executive branch strategy documents did not include key elements of desirable characteristics that can enhance the usefulness of a national strategy as guidance for decision makers in allocating resources, defining policies, and helping to ensure accountability. Federal and state authorities play key roles in regulating the reliability of the grid, which can be impaired by cybersecurity attacks. FERC is the federal regulator of interstate transmission of electricity with responsibility to review and approve standards to provide for the reliable operation of the bulk power system. In addition, FERC oversees NERC, which is the federally designated U.S. electric reliability organization. NERC is responsible for conducting reliability assessments and enforcing mandatory standards to ensure the reliability of the bulk power system—a term that refers to (1) facilities and control systems necessary for operating the electric transmission network and (2) the output from certain generation facilities needed for reliability. NERC develops reliability standards collaboratively through a deliberative process involving utilities and others in the electricity industry. NERC then sends the standards to FERC, which can either approve them or remand them to NERC for revision. These reliability standards include critical infrastructure protection standards for protecting electric utility-critical and cyber-critical assets from cyberattacks. FERC has approved 11 such cybersecurity standards, 10 of which are currently enforced. The standards call for organizations to classify their cyber systems as low-, medium-, or high-impact based on the adverse impact that loss, compromise, or misuse of those systems could have on the reliable operation of the bulk electric system. The classifications are made based on criteria and associated thresholds for, among others, generation resources and transmission substation operations. In turn, the standards apply differently to cyber systems based on whether they are classified as low-, medium-, or high-impact systems. For example: Low-impact systems. Systems that affect net aggregate generation capacity of less than 1,500 megawatts at one power plant location within a single interconnection are classified as low-impact systems and are subject to the requirements in two of the 11 cybersecurity standards. Medium-impact systems. Systems that similarly affect net aggregate generation capacity of at least 1,500 megawatts are classified as medium-impact systems and are subject to requirements in the full set of cybersecurity standards. High-impact systems. Systems that are used by and located at certain control centers are classified as high-impact systems and are subject to the full set of cybersecurity standards. The standards generally require organizations to implement similar controls for medium- and high-impact systems, with more stringent variations of certain controls for high-impact systems. As of December 2017, at most about 20 percent of the nation’s generation capacity comes from power plants with medium-impact systems and therefore is subject to requirements in the full set of cybersecurity standards. Both NERC and FERC have authority to enforce reliability standards. In addition, FERC has the authority to oversee NERC’s enforcement of the FERC-approved reliability standards. Cyber incident reporting is also an important part of federal and nonfederal regulatory efforts. Federal law requires grid owners and operators to report bulk power system incidents to DOE when certain criteria are met, such as a cyber event that causes interruptions of electrical system operations or that could potentially affect power system reliability. In addition, FERC-approved reliability standards require certain registered grid owners and operators to report cybersecurity incidents—that is, cybersecurity events that have compromised or disrupted one or more reliability tasks—to NERC. State regulators generally oversee the reliability of distribution systems, and cybersecurity regulations related to the distribution grid may vary across states. In 2017, the National Association of Regulatory Utility Commissioners released an updated version of its cybersecurity primer for state utility regulators that aims to provide guidance to state regulators. The primer highlights the NIST Cybersecurity Framework as well as the FERC-approved cybersecurity standards as helpful tools for utilities and state regulators. The U.S. electric grid faces significant cybersecurity risks—that is, threats, vulnerabilities, and impacts—and grid owners and operators face significant challenges in addressing these risks. Threat actors are becoming increasingly capable of carrying out attacks on the grid. At the same time, the grid is becoming more vulnerable to attacks. With respect to the potential impacts of the threats and vulnerabilities, U.S. cybersecurity incidents reportedly have not caused a domestic power outage. In addition, federal agencies have performed three assessments of the potential impacts that cyberattacks could have on the grid, but the potential scale of any associated outages is uncertain due to limitations in the assessments. As grid owners and operators attempt to address cybersecurity risks, they face a number of challenges, such as difficulties in hiring a sufficient cybersecurity workforce and limited public-private information sharing. A variety of threat actors pose significant cybersecurity threats to the electric grid, and many of these threat actors are becoming increasingly adept at carrying out attacks on industrial control systems, such as those supporting grid operations. Relatedly, the skill needed to attack industrial control systems is decreasing, as tools for exploiting industrial control system vulnerabilities become more available. According to the 2019 Worldwide Threat Assessment of the U.S. Intelligence Community, nations, criminal groups, and terrorists pose the most significant cyber threats to U.S. critical infrastructure. In addition, hackers and hacktivists, as well as insiders, pose significant cyber threats to the grid, according to officials and representatives of key federal and nonfederal entities whom we interviewed. Nations, including nation-state, state-sponsored, and state-sanctioned groups or programs, use cyber tools as part of their information-gathering and espionage activities. According to the 2019 Worldwide Threat Assessment, China and Russia pose the greatest cyberattack threats; of particular concern, they possess the ability to launch cyberattacks that could cause localized, temporary disruptive effects on critical infrastructure. For example, the assessment states that China has the ability to disrupt a natural gas pipeline for days to weeks (which could in turn disrupt grid operations), and Russia has the ability to disrupt an electrical distribution network for at least a few hours. The assessment also states that Russia is mapping U.S. critical infrastructure with the long-term goal of being able to cause substantial damage. Separately, DHS and the Federal Bureau of Investigation have described Russian activities as an intrusion campaign by actors on U.S. government entities and critical infrastructure organizations. In addition, a nation-state has successfully demonstrated its capability to disrupt the grid of another country. Specifically, according to the Office of the Director of National Intelligence, in December 2015 a state-sponsored actor conducted a cyberattack on the Ukrainian power grid that systematically disconnected substations, resulting in a power outage that lasted 3 hours. Officials and representatives of key federal and nonfederal entities we interviewed identified nations as the most capable threat actor but also noted that nations may not take action to disrupt the U.S. grid. For example, representatives from two utilities stated that nation-state actors are of the most concern because they have the resources to persist in their operations. However, officials from Los Alamos National Laboratory explained that nation-states may choose not to sponsor an attack because they could be easily identified. In addition, a representative from one of the utilities that we met with stated that nation-states may not pursue a cyberattack on the U.S. grid because they may be concerned about the potential response by the United States. Federal officials we interviewed noted that nation-states may be interested in gathering information about U.S. critical infrastructure with the intent of conducting a cyberattack at a later date. Criminal groups, including organized crime organizations, seek to use cyberattacks for monetary gain. According to the 2019 Worldwide Threat Assessment, financially motivated cyber criminals will likely expand their targets in the United States in the next few years, and their actions could disrupt critical infrastructure in non-energy sectors. The intelligence community does not identify criminal groups as a threat specifically to the energy sector, but these groups could still have a large impact on the grid. For example, criminal organizations often use ransomware—malicious software used to deny access to IT systems or data—to hold systems or data hostage until a ransom is paid. Criminal groups have not used ransomware to target industrial control systems, but ransomware has been used to infect IT systems tied to industrial control systems. For example, the Center for Internet Security reported in March 2019 that the LockerGoga ransomware disrupted industrial and manufacturing firms’ networks, including a Norwegian aluminum company, which had to temporarily move to manual production. According to DHS’s Industrial Control Systems Computer Emergency Response Team, ransomware continues to be a major threat to both IT and industrial control systems that support the grid. In addition, officials and representatives of key federal and nonfederal entities we interviewed suggested that nations could hire criminal groups to achieve their objectives. For example, an official from the National Renewable Energy Laboratory stated that criminal groups could be leveraged by other threat actors that have different incentives, such as nations focused on intelligence-gathering operations. Terrorists seek to destroy, incapacitate, or exploit critical infrastructures in order to threaten national security, inflict mass casualties, weaken the economy, and damage public morale and confidence. Terrorist groups may be highly motivated to disrupt or damage the grid, but they do not currently have the sophisticated tools or skill necessary to execute a cyberattack that could cause a widespread outage or significantly damage the power system, according to the 2019 Worldwide Threat Assessment. However, terrorist groups could cause disruptive effects, such as defacing websites or executing denial-of-service attacks against poorly protected networks. Hackers break into networks for a challenge, revenge, stalking, or monetary gain, among other reasons. By contrast, hacktivists are ideologically motivated and use cyber exploits to further political goals, such as free speech or to make a point. Hackers and hacktivists no longer need a great amount of skill to compromise IT systems because they can download commonly available attack tools. Officials and representatives of key federal and nonfederal entities we interviewed told us that hackers and hacktivists may have less capability to do harm than the most significant threat actors identified by the intelligence community, but they still pose a threat to the grid. For example, officials from the National Energy Technology Laboratory explained that while hacktivists generally are less capable than nations, their intent to inflict harm or to damage operations is typically more immediate than nations’ longer-term goals. In addition, representatives from nonfederal entities stated that hacktivists may be capable of causing problems for electric utilities and systems supporting the delivery of power. Insiders are entities (e.g., employees, contractors, vendors) with authorized access to an information system or enterprise who have the potential to cause harm through destruction, disclosure, modification of data, or denial of service. Such destruction can occur wittingly or unwittingly. For example, in 2009, a disgruntled former IT employee of a Texas power plant allegedly disrupted the company’s energy forecast system when the company failed to deactivate the employee’s account access and confiscate his company-issued laptop after firing him two days earlier. By contrast, in another case in 2009, contractors were reported to have unwittingly introduced malware on a uranium enrichment facility’s workstations in Iran. Specifically, the attackers introduced malware on the contractor’s business network. The malware then reportedly spread to universal serial bus (USB) devices that were used to transfer information between the contractors’ business IT network and the uranium enrichment facility’s workstations. Officials and representatives of key federal and nonfederal entities that we interviewed stated that while the threat posed by insiders varies, they could cause damaging effects. For example, Sandia National Laboratories officials explained that insiders could include knowledgeable employees with privileged access to critical systems or contractors with limited system knowledge. Further, representatives from another nonfederal entity explained that insider threats are a concern because of the economically valuable information they could steal. The electric grid is becoming more vulnerable to cyberattacks via (1) industrial control systems, (2) consumer Internet of Things (IoT) devices connected to the grid’s distribution network, and (3) the global positioning system (GPS). As previously noted, cheaper and more widely available devices that use traditional IT networking protocols are being integrated into industrial control systems. The use of these protocols, as well as traditional IT computers and operating systems, has led to a larger cyberattack surface—the different points in a network where attackers can try to enter or extract information—for the grid’s systems. In particular, many industrial control system devices include remote access capabilities, and industrial control systems are increasingly connected to corporate business networks. Remote access capabilities. Vendors are increasingly including remote access capabilities, including modems and wireless networking, as part of industrial control system devices. These capabilities are susceptible to exploitation by malicious actors. For example, malicious actors could scan a range of potential telephone numbers common to an area or published on a company website to find open modem connections to these devices (referred to as “war dialing”). In addition, malicious actors could scan for unsecured wireless networks connected to industrial control system devices while in close proximity to the devices (referred to as “war driving”). If implemented effectively, modern cybersecurity practices often protect against techniques used to remotely access industrial control system devices, and only allow trusted connections. However, to circumvent these practices, a malicious actor could, for example, compromise a vendor’s network—which is often trusted by owners and operators—and use the trusted connection to remotely connect to industrial control system devices. Connections to corporate business networks. Industrial control systems, which were once largely isolated from the internet and business IT systems, are increasingly connected in modern energy systems, allowing cyberattacks to originate in business IT systems and migrate to industrial control systems. For example, malicious nation-state actors used spear phishing emails to deploy malware on business IT networks in the 2015 attack on Ukrainian electricity utilities. After gaining initial access to the business IT networks, the attackers reportedly used a variety of techniques to migrate to the industrial control system networks of the utilities. Moreover, even if industrial control systems are not physically connected to business IT systems, malicious actors can exploit the use of removable media between the two networks. For example, as previously mentioned, contractors were reported to have unwittingly introduced malware on uranium enrichment facility workstations in Iran by using USB devices that were infected with the malware on the contractors’ business IT network to transfer information to the uranium enrichment facility’s workstations. Figure 3 illustrates how malicious actors could leverage this increasing attack surface to compromise industrial control systems. Compounding the risk associated with the increased attack surface, many legacy industrial control systems were not designed with cybersecurity protections because they were not intended to be connected to networks, such as the internet. For example, many legacy devices are not able to authenticate commands to ensure that they have been sent from a valid user and may not be capable of running modern encryption protocols. In addition, some legacy devices do not have the capability to log commands sent to the devices, making it more difficult to detect malicious activity. Additionally, even in the case of more modern devices, the safety and efficiency goals of the grid and the supporting industrial control systems can conflict with the goal of security in the design and operation of industrial control systems. According to an Idaho National Laboratory analysis, grid owners and operators may not always be able to identify industrial control system vulnerabilities in a timely manner. Vulnerability scanning is often used in IT systems to validate proper system configuration and to identify any vulnerabilities that may be present. However, conventional IT vulnerability scanning can disable or shut down energy delivery systems, and testing may not always detect vulnerabilities deep within industrial control system software. Further, even if owners and operators are able to identify industrial control system cybersecurity vulnerabilities, they may not be able to address those vulnerabilities in a timely manner because certain industrial control system devices may have high availability requirements to support grid operations. These devices typically need to be taken offline to apply patches to fix cybersecurity vulnerabilities. In addition, grid owners and operators need to rigorously test the patches before applying them. Security patches are typically tested by vendors, but they can degrade or alter the functionality of industrial control systems, which can have serious consequences for grid operations. Consequently, there is increased risk that malicious actors may be able to exploit vulnerabilities in industrial control system devices before patches can be applied. According to DHS, the number of vulnerability advisories for industrial control systems devices has steadily increased, from 17 advisories in 2010 to 223 advisories in 2018 (see fig. 4). Moreover, supply chains for industrial control systems can introduce vulnerabilities that could be exploited for a cyberattack. For example, there is a potential for manufacturers and developers to—wittingly or unwittingly—include unauthorized code or malware in industrial control system devices and systems that provides a back door into the equipment or that allows the program to “call home” once installed. Further, manufacturers and software developers create their products in many different locations around the world, thus making them potentially susceptible to foreign-based threats. For example, a capable nation-state could gather useful information on the types of equipment used at a particular utility with the intent to undermine security controls at a later time. In addition, manufacturers and developers have made sensitive information publicly available regarding the operation of their hardware and software. For example, manufacturers and developers have published vendor manuals, which include information such as default passwords and operating instructions. These manuals often appear on the internet and can aid malicious actors in conducting cyberattacks on industrial control systems. Researchers and federal agencies have recently identified concerns about the potential introduction of cyber vulnerabilities to the grid through the connection of consumer IoT devices to the grid’s distribution network. For example, university researchers in 2018 used large, real-world grid models to simulate the feasibility and impact on the grid of a coordinated cyberattack on smart home appliances. Specifically, the researchers found that malicious threat actors could compromise a large number of high-wattage IoT devices (e.g., air conditioners and heaters) and turn them into a botnet—a network of devices infected with malicious software and controlled as a group without the owners’ knowledge. The malicious actors could then use the botnet to launch a coordinated attack aimed at manipulating the demand across distribution grids. For example, according to the researchers, one such attack could involve synchronously switching on all of the compromised devices. Such an attack could disrupt the balance of power generation and consumption and ultimately cause an outage. An official from the National Renewable Energy Laboratory explained that the likelihood of attacks on the distribution network using IoT devices is low but could increase in the future. In particular, the official explained that the wattage needed to create a significant disruption in the balance of supply and demand would require a botnet of tens of thousands of smart appliances. Botnets of this size have been created, but the laboratory official explained that it would be very difficult to manipulate all of those devices to turn on at precisely the same time. However, the official cautioned that such an attack could become more plausible in the future as additional high-wattage systems and devices, such as building energy management systems and electric vehicles, are connected to the internet. The grid is dependent on GPS timing to monitor and control generation, transmission, and distribution functions. According to DOE, the GPS signal is susceptible to exploitation by malicious actors. For example, a malicious actor could inject a counterfeit GPS signal (known as GPS spoofing) that could result in disruptions to grid operations. According to the three entities responsible for collecting information on cybersecurity incidents that affect the electric grid—DHS, DOE, and NERC—none of the cybersecurity incidents reported in the United States have disrupted the reliability or availability of the grid, and none have resulted in a power outage. Even though cyber incidents involving the grid reportedly have not caused power outages in the United States, cyberattacks on foreign industrial control systems have resulted in power outages. For example, in December 2015, malicious actors linked by Ukrainian officials to the Russian government conducted cyberattacks on three Ukrainian power distribution operators, resulting in a loss of power for about 225,000 customers. GAO did not find evidence that these attacks physically damaged grid components, but cyberattacks on industrial control systems in other sectors demonstrates that this is possible. For example, in 2014, malicious cyber actors compromised industrial control systems and caused failures that led to massive damage to a blast furnace at a German steel mill. Further, federal agencies have performed three assessments of the potential impacts of cyberattacks on the industrial control systems supporting the grid. Specifically, DOE and FERC have conducted three assessments of the potential impact of cyberattacks on the grid at the scale of multiple system operators through the scale of an interconnection. The two DOE assessments—which according to DOE officials are early drafts and have not gone through intra-agency review— focused on the impact of a cyberattack within a single interconnection and produced varying reports of the potential scale of power outages that could result from a cyberattack. The remaining assessment—which FERC conducted in 2013—reviewed the impact of a cyber or physical attack on all three interconnections and concluded that an attack could result in a widespread blackout spanning the contiguous United States. Table 1 below describes the three assessments. However, because of limitations in the three federal assessments, the scale of any power outages that may result from a cyberattack is uncertain. In particular: Federal agencies have conducted one study—FERC’s 2013 study— that assesses the potential impact of a coordinated attack in each of the three interconnections. However, in 2015, DOE officials raised concerns about the scenario and related assumptions used in that study that called into question the findings. Specifically, at that time, DOE officials reported that they found several of the scenario’s assumptions highly unlikely, including peak capabilities at all targeted generation stations at the time of an attack and the loss of all safety systems designed to prevent the consequences described in the analysis. Further, DOE officials reported that they found the study’s scenarios even more unlikely to result in a total loss of power or any other consequence that could be reasonably expected to result in damage to national security. The 2017 assessment conducted by DOE’s Argonne National Laboratory was limited in scope to a six-state region. In addition, the assessment focused on a single cyberattack scenario and noted that many other grid cyberattack methods and outcomes were possible. The 2017, 2018, and 2019 editions of DOE’s draft Electricity Subsector Risk Characterization Study have significant methodological limitations. Specifically, officials from Lawrence Livermore National Laboratory who were contracted to perform the analyses cautioned that they used a reduced model of the Western Interconnection as it existed around 1980 and emphasized that their methodology should not be used to predict the behavior of the actual bulk power system. For example, those researchers told us that their selected model of the Western Interconnection had less than a quarter of its actual capacity in 2018. The DOE official responsible for the studies said that the assumption for the worst-case scenario was from that official’s professional judgement, not a documented analysis. Later, officials at Sandia National Laboratories told us that the worst-case scenario in the DOE draft study was a point solution used as a proof of concept, that the study was not of a high level of rigor, and that the assumptions may not represent a vulnerability in the actual bulk power system. Further, the DOE methodology assumed that all assets removed from service were treated equally; accordingly, the researchers did not distinguish the loss of specific assets (such as a substation or transmission line) in the calculation of attack difficulty and likelihood. Because of these limitations, some of the draft studies’ conclusions may not be realistic. For example, one of DOE’s major conclusions in the 2017 Risk Characterization Study—that a cyberattack may result in a relatively small loss of load in the United States about 8 times per year—may not be plausible because there have not been any reported cyberattacks that have caused an outage in the United States. In addition, the three draft DOE studies have widely varying conclusions on the likelihood of cyberattacks across the selected range of loss of load. For example, the 2018 draft study concluded that a cyberattack resulting in a more substantial loss of load had an average likelihood of occurring nearly once every 10 years, while the 2019 draft study concluded that such an attack would occur about once every 100 years. According to a DOE official, there is no documentation of the technical basis for the significant changes in the assessment outcomes between the 2017 and 2018 draft studies and between the 2018 and 2019 draft studies. In addition, DOE officials told us that all three studies are early drafts and have not gone through intra-agency review. Moreover, none of the federal assessments reviewed the risk associated with a cyberattack involving a botnet of high-wattage consumer IoT devices. As previously mentioned, university researchers demonstrated that malicious actors could use a botnet of IoT devices to launch a coordinated attack aimed at manipulating the demand on distribution systems across the grid. A federal official we interviewed agreed that such an attack could occur and could disrupt grid distribution systems— especially as additional high-wattage systems become connected to the internet—but they said it is unclear what impact, if any, such attacks could have on the reliability of the bulk power system. Officials and representatives of key federal and nonfederal entities we interviewed generally identified five significant challenges grid owners and operators face in addressing cybersecurity risks: (1) difficulties in hiring a sufficient cybersecurity workforce, (2) limited public-private information sharing of classified information, (3) limited resources to invest in cybersecurity protections, (4) reliance on other critical infrastructure that may be vulnerable to cyberattacks, and (5) uncertainties about how to implement cybersecurity standards and guidance. Officials and representatives of key federal and nonfederal entities we interviewed identified difficulties in hiring a sufficient cybersecurity workforce as a significant challenge to addressing cybersecurity risks to the grid. For example, a representative of a nonfederal entity told us that there are a limited number of trained cybersecurity personnel interested in working in the energy sector. The representative added that there are a large number of vacancies for cybersecurity positions and that they are difficult to fill due to the limited amount of available talent and organizational resource constraints, such as providing salaries that are competitive with other sectors. A laboratory official commented that larger grid entities are able to attract the majority of skilled cybersecurity professionals, leaving smaller entities with less skilled personnel. Further, an asset owner explained that training personnel so that they have sufficient cybersecurity knowledge and skills is difficult, and the requisite knowledge of industrial control systems further complicates training these personnel. DOE has also identified difficulties in hiring a sufficient cybersecurity workforce as a challenge. Specifically, according to DOE’s Assessment of Electricity Disruption Incident Response Capabilities, the electricity subsector continues to face challenges in recruiting and maintaining experts with strong knowledge of cybersecurity practices as well as knowledge of industrial control systems supporting the grid. Officials and representatives of key federal and nonfederal entities we interviewed identified limited public-private sharing of classified information, including the sharing of threat intelligence, as a significant challenge to addressing cybersecurity risks to the grid. For example, a laboratory official told us that many grid owners and operators do not have security clearances. Consequently, the official explained, deeming information on certain cybersecurity threats to the grid to be “classified” leaves many utilities without the awareness to address those threats to the grid. The official added that when details are removed from classified threat intelligence in order to develop an unclassified alert, that alert often lacks the specific information utilities need to address the threat. Asset owners told us that, even for those grid owners and operators who are permitted to initiate the clearance process, it can take an extended period of time to complete the associated adjudication to obtain that clearance. In addition, two asset owners noted that, even after clearances have been received and fully adjudicated, it is often difficult to obtain access to secure locations to review classified information. DOE has also identified limited public-private information sharing as a challenge. Specifically, according to DOE’s Assessment of Electricity Disruption Incident Response Capabilities, the bidirectional flow of information and intelligence between industry and government has been highlighted by stakeholders as a continued challenge for the electricity subsector. The assessment explains that the sharing of information is impeded by the slow adoption of automated capabilities and the difficultly of sharing classified information between government and industry— particularly in real time during an incident. Officials and representatives of key federal and nonfederal entities identified limited resources for cybersecurity protections as a challenge to addressing cybersecurity risks to the grid. In particular, most of the asset owners that we met with stated that it can be costly to implement required cybersecurity protections. In addition, officials and representatives of key federal and nonfederal entities that we spoke with explained that costs— including those for cybersecurity protections—must be recovered through electric rates to customers. As a result, a laboratory official explained that many utilities prioritize cybersecurity protections that are the most cost- effective over protections that may be needed to address risks. Officials and representatives of key federal nonfederal entities we interviewed identified the grid’s reliance on other critical infrastructure (e.g., natural gas pipelines) that may be vulnerable to cyberattacks as a challenge to addressing cybersecurity risks to the grid. For example, a representative of a nonfederal entity stated that the electricity subsector inherits cybersecurity risks from other critical infrastructures, since the electricity subsector relies on those critical infrastructures for its own operations. As such, that representative added that it is difficult to holistically determine how vulnerable the grid may be to a cyberattack. In addition, as previously mentioned, according to the 2019 Worldwide Threat Assessment, China has the ability to disrupt a natural gas pipeline for days to weeks. Officials and representatives of key federal and nonfederal entities we interviewed identified uncertainties about how to implement cybersecurity standards and guidance as a challenge to addressing cybersecurity risks to the grid. In particular, several representatives noted that these uncertainties have led their organizations to devote additional resources to implementing the standards and guidance. For example, one asset owner explained that FERC-approved cybersecurity standards do not always include details that are needed to understand how they apply to that owner’s environment. In addition, another asset owner stated that significant time and effort is required to understand the standards and how they might be implemented. DOE, DHS, and other federal agencies have performed a variety of critical infrastructure protection activities aimed at addressing grid cybersecurity risks, including implementing programs that help protect grid systems from cybersecurity threats and vulnerabilities. In addition, FERC has performed a variety of regulatory activities aimed at addressing grid cybersecurity risks, such as approving mandatory cybersecurity standards for the bulk power system. DOE, DHS, and other federal agencies have performed a variety of critical infrastructure protection activities aimed at addressing grid cybersecurity risks. These activities generally align with the functions in the NIST Cybersecurity Framework, which include (1) protecting systems to mitigate cybersecurity threats and vulnerabilities; (2) identifying cybersecurity threats and vulnerabilities and detecting potential cybersecurity incidents; and (3) responding to and recovering from such incidents. Protecting systems to mitigate cybersecurity threats and vulnerabilities Federal agencies assist grid asset owners and operators in implementing protections that mitigate cybersecurity risks by providing capabilities aimed at preventing cybersecurity intrusions and offering training and guidance on cybersecurity practices. For example, DHS’s Enhanced Cybersecurity Services program provides intrusion-prevention capabilities to U.S.-based entities and to state, local, tribal, and territorial organizations. To carry out this voluntary program, DHS provides classified and unclassified threat information to designated commercial service providers. These providers use the information to block access to (1) specific malicious internet addresses and (2) email with specific malicious criteria. NIST, DHS, and DOE also provide cybersecurity training and guidance. For example, NIST has developed numerous special publications on cybersecurity protections for IT and industrial control systems, such as the previously mentioned Cybersecurity Framework and its Guide to Industrial Control Systems. In addition, DHS provides in-person and online training on leading cybersecurity practices for industrial control systems through its National Cybersecurity and Communications Integration Center. Lastly, DHS has taken initial steps to help grid entities manage supply chain cybersecurity risks. For example, in July 2018 DHS created a public-private partnership, known as the Supply Chain Risk Management Task Force. The task force aims to examine risks to the global information and communications technology supply chain and develop consensus recommendations to manage such risks. Identifying cybersecurity threats and vulnerabilities and detecting potential cybersecurity incidents Federal agencies help grid entities identify cybersecurity risks and detect incidents by providing threat and vulnerability information, performing risk assessments, performing forensic analysis, and conducting research. For example, DOE piloted and launched the Cybersecurity Risk Information Sharing Program, which is now managed by the Electricity Information Sharing and Analysis Center. It provides a voluntary, bi-directional public- private IT data sharing and analysis platform. Using both classified and unclassified sources, DOE’s Pacific Northwest National Laboratory analyzes the information to (1) identify threat patterns and attack indicators, and (2) deliver alerts to owners and operators. In addition, DHS’s Automated Indicator Sharing program provides a server housed at each participant’s location that can be used to exchange threat indicators with the department’s National Cybersecurity and Communications Integration Center. Further, the center provides asset owners with alerts, advisories, and situational reports, including information on threats, vulnerabilities, or activity that could affect IT or industrial control system networks. DOE and DHS also offer services aimed at helping grid owners and operators assess cybersecurity risks and perform forensic analysis. For example, DOE has an evaluation tool known as the Electricity Cybersecurity Capability Maturity Model that aims to help the electricity industry evaluate, prioritize, and improve its cybersecurity capabilities. In addition, DHS offers technical assessments through its National Cybersecurity and Assessment and Technical Services Team that can help identify vulnerabilities and simulate a malicious adversary. Further, DHS can review potential cybersecurity incident artifacts, such as malware, phishing emails, and network logs, at its National Cybersecurity and Communications Integration Center to determine the existence or extent of a cybersecurity threat or incident. Moreover, DOE’s Cybersecurity for Energy Delivery Systems program sponsors grid cybersecurity research through DOE’s national laboratories. For example: Oak Ridge National Laboratory has conducted research on mechanisms that could help critical infrastructure entities better detect vulnerabilities in software used in industrial control systems. Four national laboratories have engaged in a project that aims to improve the capability of grid entities to collect and analyze data from their industrial control system networks and detect cybersecurity incidents. Oak Ridge National Laboratory and Pacific Northwest National Laboratory have a joint project to develop mechanisms for more quickly detecting and eradicating malware on industrial control systems. Responding to and recovering from cybersecurity incidents Federal agencies have developed policies, strategies, and plans to define their roles and responsibilities for responding to and recovering from grid cybersecurity incidents. In particular, DHS has responsibility for leading the federal effort to mitigate or lessen the impact of such incidents, the Department of Justice has responsibility for the federal law enforcement response to the threats, and DOE has authority, in designated emergencies, to impose measures to restore the reliability of critical electric infrastructure. DOE is also responsible for coordinating the energy sector-specific response with DHS and the Department of Justice. Federal agencies have also taken steps to help prepare asset owners for cyber response and recovery efforts. For instance, DHS has worked with nonfederal entities to simulate response and recovery efforts to a cyberattack through exercises such as Cyber Storm. In addition, DOE, in conjunction with the National Association of State Energy Officials, has conducted regional energy assurance exercises. These exercises aim to promote state and local preparedness and resilience for future energy emergencies stemming from a cyber incident. FERC has performed a variety of regulatory activities aimed at addressing grid cybersecurity risks. These activities include (1) approving mandatory cybersecurity standards for the bulk power system, (2) enforcing regulatory requirements through imposition of civil penalties, (3) auditing the performance of the electric reliability organization—NERC— and its regional entities, and (4) auditing bulk power entities for compliance with the mandatory cybersecurity standards. Approve mandatory cybersecurity standards. FERC has approved mandatory reliability standards relating to cybersecurity protections. For example, in October 2018, FERC approved a new standard to bolster supply chain risk management protections for the nation’s bulk electric system. This new standard, which will become enforceable in July 2020, is intended to augment existing standards that aim to mitigate cybersecurity risks associated with the supply chain for grid- related cyber systems. Enforce regulatory requirements through imposition of civil penalties. FERC has referred violations of its approved cybersecurity standards to NERC to impose penalties on the bulk power entities that committed the violations. For example, such a notification occurred in January 2019 when NERC assessed a $10 million penalty based on 127 violations of the cybersecurity standards made by an undisclosed entity. Audit the performance of the electric reliability organization. FERC has audited NERC’s performance as the electric reliability organization. In this audit, which it completed in 2012, FERC evaluated NERC’s budget formulation, administration, and execution. With respect to cybersecurity, FERC recommended that NERC (1) assess its existing staffing levels to ensure adequate resources to accomplish critical infrastructure protection work related to cybersecurity and (2) devote greater resources to carrying out its oversight duties. In 2013, FERC closed these recommendations after reviewing NERC’s plans for evaluating its staffing levels and its commitment to add resources in its business plan. According to FERC officials, FERC continues to monitor the level of resources NERC devotes to cybersecurity oversight through its annual review of NERC’s budget Audit bulk power entities for compliance with standards. FERC has audited bulk power entities’ compliance with its approved cybersecurity standards. From 2016 through 2018, FERC conducted its own independent audits of eight bulk power entities for compliance with those standards and produced public lessons learned reports based on the results. According to FERC officials, the agency plans to conduct four such audits every fiscal year starting in fiscal year 2019 and to continue producing annual lessons learned reports based on the results. In addition, since the first of the cybersecurity standards became enforceable in 2009, FERC has observed eight NERC regional entity-led audits a year—one in each NERC region—focused on bulk power entity compliance with those standards. National strategies are critical tools used to help address longstanding and emerging issues that affect national security and economic stability. In 2004, we identified a set of desirable characteristics for effective national strategies. These characteristics include: Purpose, scope, and methodology. Addresses why the strategy was produced, the scope of its coverage, and the process by which it was developed. Problem definition and risk assessment. Addresses the particular national problems, assesses the risks to critical assets and operations—including the threats to, and vulnerabilities of, critical operations—and discusses the quality of data available regarding the risk assessment. Goals, subordinate objectives, activities, and performance measures. Addresses what the strategy is trying to achieve; steps to achieve those results; and the priorities, milestones, and performance measures that include measurable targets to gauge results and help ensure accountability. Discussion of needed resources and investments. Addresses what the strategy will cost and the types of resources and investments needed. Organizational roles, responsibilities, and coordination. Addresses who will implement the strategy, what their roles will be, and mechanisms to coordinate their efforts. As previously noted, the executive branch has taken steps toward outlining a federal strategy for confronting cyber threats—including threats to critical infrastructure such as the grid. In addition, as the sector- specific agency, DOE has led the development of approaches to implement the federal cybersecurity strategy for the energy sector, including the grid. Table 2 identifies and describes these approaches— specifically, two agency plans and an assessment—for addressing grid cybersecurity risks and challenges. The two plans and the assessment do not fully address all of the key characteristics needed for a national strategy. Collectively, the plans and assessment fully address one characteristic—purpose, scope, and methodology—and partially address the other four characteristics of a national strategy (see table 3). Purpose, scope, and methodology The plans and assessment fully address the characteristic of outlining their purpose, scope, and methodology. For example, the Energy Sector- Specific Plan explains that it was produced to help integrate and guide the sector’s continuing effort to improve the security and resilience of critical infrastructure. In addition, the plan explains that DOE worked closely with the Energy Sector Coordinating Council and the Energy Sector Government Coordinating Council, among others, to develop the plan. Problem definition and risk assessment The plans and the assessment partially address the characteristic of defining the problem and performing a risk assessment. Each defines the problems that it was intended to address and assesses cybersecurity risks to the grid. For example, DOE’s Assessment of Electricity Disruption Incident Response Capabilities states that it was developed in response to Executive Order 13800’s requirement that DOE examine the potential scope and duration of a prolonged power outage associated with a significant cyber incident. In addition, as previously mentioned, the assessment describes the potential range of load loss resulting from four cyberattack scenarios. However, the discussion of the quality of data available regarding DOE’s assessment is inaccurate. According to the assessment, the potential range of load loss resulting from four cyberattack scenarios was based on rigorous modeling and analysis from multiple DOE national laboratory experts. However, these results were based on the 2017 Electricity Subsector Risk Characterization Study, which as previously described, has significant limitations affecting the quality of data. In addition, neither the plans nor the assessment fully analyzed the cybersecurity risks and challenges to the grid. In particular, none of them analyzed the threat of, and vulnerabilities to, a cyberattack spanning all three interconnections. In addition, the initiatives did not assess the vulnerability of the grid to a cyberattack involving high-wattage consumer IoT devices connected to the grid’s distribution system. Goals, subordinate objectives, activities and performance measures The two plans partially address the characteristic of outlining goals, subordinate objectives, activities, priorities, milestones, and performance measures. Both plans outline the goals, objectives, and activities for addressing cybersecurity risks facing the electric grid. For example, the Energy Sector-Specific Plan describes five goals for the energy sector and three related priorities for the electricity subsector. However, the plans’ goals, objectives, and activities do not fully address the cybersecurity risks to the grid. For example, neither plan includes goals and activities that address the vulnerability of the grid to a cyberattack involving high-wattage consumer IoT devices connected to the grid’s distribution system. Further, in light of the previously identified gaps in the analysis of cybersecurity risks and challenges, the plans’ goals, objectives, and activities are likely not commensurate with grid cybersecurity risks and challenges. Moreover, only one of the plans—DOE’s Multiyear Plan for Energy Sector Cybersecurity—includes milestones and performance measures for achieving the goals, objectives, and activities. Additionally, this plan does not include performance measures with measurable targets for all objectives, including those aimed at providing timely cyber threat briefings to energy sector partners and developing cyber incident response processes and procedures. The two plans partially address the characteristic of describing resource and investment needs. Specifically, although the plans identify many resources and investments needed to achieve their goals and objectives, they do not fully identify resource and investment needs. For example, one of the objectives of DOE’s Multiyear Plan for Energy Sector Cybersecurity is to establish a coordinated national cyber incident response capability for the energy sector. However, the plan does not describe the resources or investments needed to meet this objective. This is of particular concern because, as previously mentioned, the Fixing America’s Surface Transportation Act of 2015 authorized DOE to order emergency measures, following a Presidential declaration of a grid security emergency, to protect or restore the reliability of critical electric infrastructure. In addition, the plans do not describe specific investment costs associated with carrying them out. For example, DOE’s Multiyear Plan for Energy Sector Cybersecurity describes the need to develop a laboratory for identifying and analyzing cybersecurity vulnerabilities to energy delivery systems. However, the plan does not identify the specific costs associated with this investment. Further, given the previously discussed gaps in risk analysis, goals, and objectives, it is unclear to what extent the identified resources and investment needs are sufficient to address electric grid cybersecurity risks and challenges. Roles, responsibilities, and coordination The two plans partially address the characteristic of describing roles, responsibilities, and coordination mechanisms for carrying out the goals, objectives, and activities. Specifically, the plans describe mechanisms for coordinating but do not always identify organizations responsible for achieving the goals, objectives, and activities. For example, DOE’s Multiyear Plan for Energy Sector Cybersecurity states that the department will partner with DOE’s national laboratories to carry out several activities in the plan. However, the plan does not indicate which of the 10 national laboratories DOE will partner with for each activity. In a written response, DOE explained that executive branch documents that outline the broader federal strategy for confronting cyber threats— such as the National Cyber Strategy and the DHS Cybersecurity Strategy—address the key characteristics of a national strategy not addressed in DOE’s plans and assessment. In addition, DOE stated that the department’s plans and assessment for addressing risks and challenges facing the grid support and fit within the context of that broader cybersecurity framework while allowing the agency flexibility to accomplish its goals. Although the broader executive branch strategy documents on confronting cyber threats provide a framework for addressing critical infrastructure cybersecurity risks and challenges, they do not address the specific risks and challenges facing the electric grid. In addition, as previously mentioned, we have reported that these broader executive branch strategy documents also do not include key characteristics of a national strategy. Until DOE ensures it has a plan aimed at implementing the federal cybersecurity strategy relating to the grid that addresses all of the key characteristics of a national strategy—including a full assessment of cybersecurity risks—the guidance the plan provides decision makers in allocating resources to address risks and challenges will likely be limited. FERC has not ensured that its approved grid cybersecurity standards fully address leading federal guidance for improving critical infrastructure cybersecurity—specifically, the NIST Cybersecurity Framework. In addition, FERC has not evaluated the risk of a coordinated cyberattack on geographically distributed targets in approving the threshold for which grid cyber systems must comply with requirements in the full set of grid cybersecurity standards. The NIST Cybersecurity Framework provides a set of cybersecurity activities, desired outcomes, and applicable references that are common across all critical infrastructure sectors. The framework also states that while it is not exhaustive, it is capable of being extended, allowing organizations, sectors, and other entities to use references that are most appropriate to enable them to manage their cybersecurity risk. NIST recommends that organizations use the Cybersecurity Framework functions, categories, and subcategories to identify the key controls needed to meet their security objectives (see Table 4 for the functions and categories). To promote widespread adoption of the framework, Executive Order 13636 called for sector-specific agencies to develop mechanisms to encourage the framework’s adoption. In addition, the order called for regulatory agencies to review the framework and determine if current cybersecurity regulatory requirements are sufficient given current and projected risks. However, the FERC-approved cybersecurity standards do not fully address the NIST Cybersecurity Framework’s five functions and associated categories and subcategories. More specifically, the cybersecurity standards substantially address two of the five functions and partially address the remaining three functions. Table 5 depicts the extent to which these standards address the framework’s five functions and 23 categories. (Appendix II contains more detailed information regarding the extent to which the standards address the framework’s 108 subcategories.) Legend: ●—Fully address: the standards address all of the related subcategories. ◕—Substantially address: the standards address at least two-thirds, but not all, of the related subcategories. ◑—Partially address: the standards address at least one-third, but less than two-thirds, of the related subcategories. ◔—Minimally address: the standards address less than one-third of the related subcategories.○—Do not address: the standards do not address any of the related subcategories. As shown in table 5, the FERC-approved cybersecurity standards either fully address or substantially address eight of the 23 categories. For example: The standards fully address the identity management, authentication, and access control category by fully addressing seven associated subcategories. For instance, the standards fully address the subcategories for credentials to be issued, managed, verified, revoked, and audited for authorized devices, users, and processes; network integrity to be protected; and physical access to assets to be managed and protected. The standards fully address the response planning category by fully addressing the associated subcategory—a response plan is to be executed during or after an incident. Conversely, the FERC-approved cybersecurity standards partially address or do not address the remaining 15 of 23 categories. For example: The standards partially address the category for supply chain risk management. In particular, the standards fully address associated subcategories for establishing supply chain risk management processes, security measures in contracts with suppliers and third- party partners, and evaluations of suppliers and third-party partners to ensure they meet their contractual obligations. However, the standards do not address subcategories for response and recovery planning and testing with suppliers and third-party providers, and for using the supply chain risk management process to identify, prioritize, and assess suppliers and third-party partners. The standards do not address the three subcategories associated with the risk management strategy category. Specifically, the standards do not call for risk management processes to be established, organizational risk tolerance to be determined, or for the risk tolerance to be informed by the organization’s role in critical infrastructure and sector-specific risk analysis. In a written response, FERC officials said that the agency did not conduct an assessment to determine how the leading practices identified in the NIST Cybersecurity Framework could be applied to the cybersecurity standards. In addition, FERC officials stated that, while the Commission uses the NIST Cybersecurity Framework as a resource and its approved standards incorporate certain facets of the framework, there is not a one- on-one alignment because the NIST Cybersecurity Framework is not industry specific. According to FERC officials, the framework addresses certain issues outside FERC’s jurisdiction. For example, FERC officials stated that the Commission does not have authority to directly impose obligations on suppliers, vendors, or entities outside its jurisdiction that provide products or services to electric industry stakeholders. However, full implementation of the NIST Cybersecurity Framework does not require regulatory agencies to impose obligations on entities over which the regulatory agencies do not have authority. Framework categories and subcategories that reference suppliers and vendors call for the organization responsible for implementing the framework to establish and implement processes for managing cybersecurity risks relating to those suppliers and vendors. In addition, in a written response, NERC officials disagreed with our assessment and stated that a separate comparison by NERC subject matter experts found substantially more overlap between the FERC- approved cybersecurity standards and the NIST Cybersecurity Framework. Moreover, NERC officials said that the intended purpose of the standards differs from the framework’s voluntary nature, and that NERC must ensure all mandatory standards are auditable and implemented by electric utilities nationwide. The officials noted the importance of the NIST Cybersecurity Framework and emphasized that NERC has considered the framework in developing and updating grid cybersecurity standards. However, we believe our analysis accurately reflects the extent that the FERC-approved standards address the NIST Cybersecurity Framework. Without a full consideration of how the FERC-approved cybersecurity standards address NIST’s Cybersecurity Framework, there is increased risk that bulk power entities will not fully implement leading cybersecurity practices intended to help critical infrastructure entities address cybersecurity risks. As previously mentioned, FERC requires cyber systems affecting a generation capacity of 1,500 megawatts or more to comply with requirements in the full set of approved cybersecurity standards since the loss, compromise, or misuse of those systems could have a medium to high impact on the reliable operation of the bulk electric system. FERC approved the 1,500-megawatt threshold based on the results of a NERC analysis. Specifically, NERC staff selected a threshold value based on the loss of one large electric grid asset from a single disruptive event and assumed a loss of power could be compensated, in part, by power from a neighboring region. However, the analysis did not evaluate the potential risk of a coordinated cyberattack on geographically distributed targets. A coordinated cyberattack could cause multiple power plants, transmission lines, or related grid components in different regions to disconnect from the grid. Such a cyberattack could target, for example, a combination of low- impact systems, each affecting a generation capacity below 1,500 megawatts that, in aggregate, might present a significant risk to the grid. FERC officials told us that the agency considered but did not evaluate the potential impact of a coordinated cyberattack on geographically distributed targets at the time it approved the threshold because the agency did not have the information it needed to develop a credible threat scenario. FERC officials said they anticipate that a future update to the approved cybersecurity standards may require the collection of relevant data on suspicious cyber activity that could inform a threat scenario for evaluating the potential impact of a coordinated cyberattack on geographically distributed targets. Further, NERC officials told us that, while NERC has not determined that a modification of the 1,500 megawatt threshold is warranted at this time, they continue to monitor the risk of a coordinated cyberattack against multiple low-impact systems and acknowledged that the FERC-approved standards must adapt with the evolving understanding of cyber threats. In addition, NERC officials explained in a written response that the intent of the 1,500-megawatt threshold is to ensure that industrial control systems with vulnerabilities that are attributable to a common cause (e.g., cybersecurity vulnerabilities in common hardware or software) that could result in the loss of 1,500 megawatts or more of generation capacity are adequately protected. Those officials added that NERC encourages entities to disaggregate their industrial control systems so that individual systems operate and maintain less than 1,500 megawatts of generation capacity. NERC officials noted that the systems associated with the disaggregated generation capacity are very diverse and are therefore less likely to provide any large single point of failure. NERC officials further explained that this disaggregation minimizes the risk to the grid by requiring a malicious actor to conduct a cyberattack on more facilities to achieve a similar loss of power. However, encouraging grid entities to design industrial control systems so that individual systems operate and maintain less than 1,500 megawatts of generation capacity could still leave the grid vulnerable to a cyberattack on those systems. For example, although a malicious actor may need to attack more systems that fall under the threshold at multiple locations to achieve the attacker’s objective for loss of power (when compared with systems that meet or exceed the threshold), the difficulty of carrying out an attack on additional systems could be less significant if the attacker identifies and exploits vulnerabilities common across the systems. In addition, as previously mentioned, systems that fall under the 1,500- megawatt threshold are not required to follow all of the requirements of the FERC-approved cybersecurity standards; as such, there is increased risk that important security controls have not been implemented for these systems. According to federal standards for internal control, management should identify, analyze, and respond to risks related to achieving organizational objectives. For example, management comprehensively identifies risks that affect its objectives and analyzes the identified risks to estimate their significance, which provides a basis for responding to the risks. Without information on the risk of a coordinated cyberattack on geographically distributed targets, FERC does not have assurance that its approved threshold for mandatory compliance with all cybersecurity standards adequately responds to that risk and sufficiently provides for the reliable operation of the grid. The U.S. electric grid faces an increasing array of cybersecurity risks, as well as significant challenges to addressing those risks. To their credit, federal agencies have performed a variety of critical infrastructure protection and regulatory activities aimed at addressing those risks. In particular, DOE has developed plans and an assessment aimed at implementing the federal strategy for confronting the cyber threats facing the grid. However, those documents do not fully address all of the key characteristics needed to implement a national strategy, including a full assessment of cybersecurity risks to the grid. Until DOE ensures it has a plan that does, the guidance the plan provides decision makers in allocating resources to address grid cybersecurity risks and challenges will likely be limited. Additionally, FERC has approved mandatory cybersecurity standards for bulk power entities, but those standards address some but not all of the leading cybersecurity practices identified in NIST’s Cybersecurity Framework. Without a full consideration of how the FERC-approved cybersecurity standards address NIST’s Cybersecurity Framework, there is increased risk that bulk power entities will not fully implement leading cybersecurity practices needed to address current and projected risks. Finally, the threshold for which entities must comply with requirements in the full set of FERC-approved standards is based on the results of an analysis that did not evaluate the potential risk of a coordinated cyberattack on geographically distributed targets. Without information on the risk of such an attack—particularly one that might target low-impact systems that are subject to fewer requirements but in aggregate could affect the grid—FERC does not have assurance that its approved threshold for mandatory compliance adequately responds to that risk and sufficiently provides for the reliable operation of the electric grid. We are making a total of three recommendations—one to DOE and two to FERC. Specifically: The Secretary of Energy, in coordination with DHS and other relevant stakeholders, should develop a plan aimed at implementing the federal cybersecurity strategy for the electric grid and ensure that the plan addresses the key characteristics of a national strategy, including a full assessment of cybersecurity risks to the grid. (Recommendation 1) FERC should consider our assessment and determine whether to direct NERC to adopt any changes to its cybersecurity standards to ensure those standards more fully address the NIST Cybersecurity framework and address current and projected risks. (Recommendation 2) FERC should (1) evaluate the potential risk of a coordinated cyberattack on geographically distributed targets and, (2) based on the results of that evaluation, determine whether to direct NERC to make any changes to the threshold for mandatory compliance with requirements in the full set of cybersecurity standards. (Recommendation 3) We provided a draft of this report for review and comment to DOE and FERC—the two agencies to which we made recommendations—as well as DHS, the Department of Commerce (on behalf of NIST), and NERC. DOE and FERC agreed with our recommendations, DHS and the Department of Commerce stated that they had no comments, and NERC disagreed with one of our findings. DOE and FERC agreed with our recommendations. In its written comments, reproduced in appendix III, DOE concurred with our recommendation and stated that it is working through an interagency process to develop a National Cyber Strategy Implementation Plan that will consider DOE’s Multiyear Plan for Energy Sector Cybersecurity. In its written comments, reproduced in appendix IV, FERC stated that our recommendations were constructive and that it would take steps to implement them. DOE and FERC also provided technical comments, which we incorporated as appropriate. In its written comments, reproduced in appendix V, NERC stated that it disagreed with our conclusion that the FERC-approved cybersecurity standards do not fully address the NIST Cybersecurity Framework. NERC recognized the importance of the NIST Cybersecurity Framework and emphasized that NERC has considered the framework in developing and updating its grid cybersecurity standards. However, NERC stated that a separate analysis by NERC subject matter experts found substantially more overlap between the standards and the framework than our analysis. In addition, NERC cited a 2011 GAO report that found that the FERC-approved standards, in combination with NERC supplementary guidance, mostly addressed the information security controls in certain NIST guidance at that time. We reviewed NERC’s analysis comparing the FERC-approved cybersecurity standards to the NIST Cybersecurity Framework and continue to believe our analysis accurately reflects the extent to which the standards address the framework. Further, in this report we assessed the extent to which the FERC-approved standards addressed the NIST Cybersecurity Framework, which is more recent and broader guidance than the NIST guidance that we examined in our 2011 report. In its comments, NERC also stated it has not determined that any changes are needed to the threshold for mandatory compliance with the full set of cybersecurity standards at this time, but it agrees with the concern that low-impact systems may be more vulnerable to a cyberattack and will continue to evaluate whether the current threshold is appropriate given evolving cybersecurity risks. For example, NERC explained that it is studying cybersecurity supply chain risks, including those associated with low-impact assets not currently subject to its supply chain standards. We believe that this effort could help to better position electric grid entities to address supply chain cybersecurity risks. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretaries of Commerce, Energy, and Homeland Security, the Chairman of FERC, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact Frank Rusco at (202) 512-3841 or ruscof@gao.gov, and Nick Marinos at (202) 512-9342 or marinosn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. Our objectives were to (1) describe the cybersecurity risks and challenges facing the electric grid, (2) describe federal efforts to address grid cybersecurity risks, (3) assess the extent to which the Department of Energy (DOE) has a defined strategy for addressing grid cybersecurity risks and challenges, and (4) assess the extent to which Federal Energy Regulatory Commission (FERC)-approved cybersecurity standards address grid cybersecurity risks. To address our first objective, we developed a list of cyber actors that could pose a threat to the grid, identified vulnerable components and processes that could be exploited, reviewed the potential impact of cyberattacks on the grid, and identified key cybersecurity challenges facing the grid. To develop the list of cyber threat actors, we reviewed our prior work on cyber-based threats facing the grid as well as the threats identified by the 2019 Worldwide Threat Assessment of the U.S. Intelligence Community. In addition, we interviewed officials or representatives from the following key federal and nonfederal entities to confirm, add, or remove cyber threat actors identified in our prior work based on their potential impact on grid operations: Federal agencies. We interviewed officials from DOE, the Department of Homeland Security (DHS), FERC, and the National Institute of Standards and Technology (NIST). Nonfederal regulatory organizations. We interviewed representatives of the North American Electric Reliability Corporation (NERC). Grid owners and operators. We interviewed five grid owners and operators. To select these grid owners and operators, we reviewed a membership list of the Electricity Subsector Coordinating Council as of May 2018, divided that list into three categories—investor-owned, municipal, and cooperative utilities—and then randomly selected entities from each of those three categories to interview. The views of the grid owners and operators we selected are not generalizable to the population of utilities in the United States but provide valuable insight into the cybersecurity risks and challenges grid owners and operators face. National associations. We interviewed representatives of national associations that represent various types of asset owners, entities with regulatory or state interests, and those with grid cybersecurity interests generally. Specifically, we interviewed representatives from the American Public Power Association, Edison Electric Institute, Electric Power Research Institute, Independent System Operator/Regional Transmission Operator Coordinating Council, National Rural Electric Cooperative Association, National Association of Regulatory Utility Commissioners, National Association of State Energy Officials, and North American Transmission Forum Association. The views of the association representatives are not generalizable to the industry but provide valuable insight into the cybersecurity risks and challenges facing the grid. To identify grid cybersecurity vulnerabilities, we reviewed reports developed by key federal and nonfederal entities and others related to grid vulnerabilities and met with the key federal and nonfederal entities to understand the scale and complexity of these vulnerabilities. We also compiled DHS-provided advisories from 2010 through 2018 related to industrial control system devices. We then summarized information from the DHS website to determine how many DHS issued per year. With respect to the potential impact of cyberattacks, we reviewed cybersecurity incidents reported to DOE, DHS, and NERC from 2014 through 2018. We also asked these agencies for information on any cybersecurity incidents that occurred prior to 2014 or after 2018 that affected the reliability or availability of the grid. In addition, we reviewed federal reports on cyberattacks that caused power outages in foreign countries and a report developed by the German government regarding a cyberattack on industrial control systems that damaged a German steel mill. Further, we reviewed federal studies assessing the potential for widespread power outages resulting from cyberattacks, and we met with federal officials to discuss the methodologies used to perform these studies. Finally, to identify key cybersecurity challenges facing the grid, we reviewed our prior reports on such challenges as well as federal and industry reports recommended by entities we met with. We also asked the key federal and nonfederal entities to identify challenges facing grid entities in addressing cybersecurity risks, and we compiled the challenges they most often cited. To address the second objective, we identified critical infrastructure protection and regulatory actions that federal agencies are taking to address grid cybersecurity risks by reviewing federal strategies, plans, and reports describing activities that have been conducted or that are under way and by interviewing the key federal and nonfederal entities to obtain additional details on these activities. We also reviewed FERC- approved cybersecurity standards for the bulk power system. We then categorized critical infrastructure protection activities using the functions in NIST’s Framework for Improving Critical Infrastructure Cybersecurity (commonly referred to as NIST’s Cybersecurity Framework). For our third objective, we reviewed two DOE-led plans and one assessment aimed at implementing the federal cybersecurity strategy for the energy sector, including the grid. We then compared those plans and assessment with leading practices identified by GAO on key characteristics for a national strategy. In doing so, we assessed each characteristic as follows: fully addresses—the plan or assessment addresses all aspects of the characteristic, partially addresses—the plan or assessment addresses some but not all of the characteristic, or does not address—the plan or assessment does not address any aspects of the characteristic. We also provided our analysis to DOE officials to review, comment, and provide additional information. For our fourth objective, we compared the FERC-approved cybersecurity standards with leading federal practices for addressing critical infrastructure cybersecurity risks identified in NIST’s Cybersecurity Framework. Specifically, a GAO analyst compared the FERC-approved cybersecurity standards with the subcategories in the Cybersecurity Framework, and another GAO analyst reviewed and confirmed the results of that analysis. We then summarized the results of these assessments for each of the framework’s five functions, 23 categories, and 108 subcategories as follows: fully address—the standards address all of the related subcategories; substantially address—the standards address at least two-thirds, but not all, of the related subcategories; partially address—the standards address at least one-third, but less than two-thirds, of the related subcategories; minimally address—the standards address less than one-third of the do not address—the standards do not address any of the related subcategories. We also provided our analysis to FERC and NERC officials to review, comment, and provide additional information. We also examined the applicability of the FERC-approved cybersecurity standards to non-nuclear power plants and reviewed FERC and NERC information on the analytical basis for that threshold. To calculate the number and aggregate capacity of plants that met the 1,500-megawatt threshold for complying with all FERC-approved cybersecurity standards, we used data from Form EIA-860, “Annual Electric Generator Report,” which includes U.S. plants with generators having nameplate capacity of 1 megawatt or greater. As a proxy for the net real power capability specified in the standards, we selected the generator’s net summer generating capacity. To calculate a total capacity for each individual power plant, we combined the data on the capacity of each plant’s individual operating electric power generators. We then filtered these data to identify plants whose primary purpose is generating electricity for sale as reported on the Form EIA-860. Ultimately, we compared the number and capacity of non-nuclear plants exceeding the 1,500-megawatt threshold to the total number and total U.S. capacity for plants. We used U.S. Energy Information Administration (EIA) data to estimate the number and capacity of non-nuclear plants exceeding the 1,500- megawatt threshold. To assess the reliability of these data, we reviewed EIA documentation, discussed the quality of the data with EIA officials, and electronically tested the data set for missing data, outliers, or obvious errors. Based on this assessment, we determined that the EIA data were sufficiently reliable for our purposes. We conducted this performance audit from January 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The table below provides additional detail on our assessment of the extent to which Federal Energy Regulatory Commission (FERC)- approved cybersecurity standards address the National Institute of Standards and Technology’s (NIST) Framework for Improving Critical Infrastructure Cybersecurity’s (commonly known as the NIST Cybersecurity Framework) 23 categories and 108 subcategories. ID.AM-1: Physical devices and systems within the organization are inventoried. ID.AM-2: Software platforms and applications within the organization are inventoried. ID.AM-3: Organizational communication and data flows are mapped. ID.AM-4: External information systems are catalogued. ID.AM-5: Resources (e.g., hardware, devices, data, time, personnel, and software) are prioritized based on their classification, criticality, and business value. ID.AM-6: Cybersecurity roles and responsibilities for the entire workforce and third-party stakeholders (e.g., suppliers, customers, and partners) are established. ID.BE-1: The organization’s role in the supply chain is identified and communicated. ID.BE-2: The organization’s place in critical infrastructure and its industry sector is identified and communicated. ID.BE-3: Priorities for organizational mission, objectives, and activities are established and communicated. ID.BE-4: Dependencies and critical functions for delivery of critical services are established. ID.BE-5: Resilience requirements to support delivery of critical services are established for all operating states (e.g. under duress/attack, during recovery, normal operations). ID.GV-1: Organizational information security cybersecurity policy is established and communicated. ID.GV-2: Cybersecurity roles and responsibilities are coordinated and aligned with internal roles and external partners. ID.GV-3: Legal and regulatory requirements regarding cybersecurity, including privacy and civil liberties obligations, are understood and managed. ID.GV-4: Governance and risk management processes address cybersecurity risks. ID.RA-1: Asset vulnerabilities are identified and documented. ID.RA-2: Cyber threat intelligence is received from information-sharing forums and sources. ID.RA-3: Threats, both internal and external, are identified and documented. ID.RA-4: Potential business impacts and likelihoods are identified. ID.RA-5: Threats, vulnerabilities, likelihoods, and impacts are used to determine risk. ID.RA-6: Risk responses are identified and prioritized. Risk Management Strategy (ID.RM): The organization’s priorities, constraints, risk tolerances, and assumptions are established and used to support operational risk decisions. ID.RM-1: Risk management processes are established, managed, and agreed to by organizational stakeholders. ID.RM-2: Organizational risk tolerance is determined and clearly expressed. ID.RM-3: The organization’s determination of risk tolerance is informed by its role in critical infrastructure and sector-specific risk analysis. Supply Chain Risk Management (ID.SC): The organization’s priorities, constraints, risk tolerances, and assumptions are established and used to support risk decisions associated with managing supply chain risk. The organization has established and implemented the processes to identify, assess, and manage supply chain risks. ID.SC-1: Cyber supply chain risk management processes are identified, established, assessed, managed, and agreed to by organizational stakeholders. ID.SC-2: Suppliers and third-party partners of information systems, components, and services are identified, prioritized, and assessed using a cyber supply chain risk assessment process. ID.SC-3: Contracts with suppliers and third-party partners are used to implement appropriate measures designed to meet the objectives of an organization’s cybersecurity program and Cyber Supply Chain Risk Management Plan. ID.SC-4: Suppliers and third-party partners are routinely assessed using audits, test results, or other forms of evaluations to confirm they are meeting their contractual obligations. ID.SC-5: Response and recovery planning and testing are conducted with suppliers and third-party providers. Identity Management Authentication and Access Control (PR.AC): Access to physical and logical assets and associated facilities is limited to authorized users, processes, and devices and is managed consistent with the assessed risk of unauthorized access to authorized activities and transactions. PR.AC-1: Identities and credentials are issued, managed, verified, revoked, and audited for authorized devices, users, and processes. PR.AC-2: Physical access to assets is managed and protected. PR.AC-3: Remote access is managed. PR.AC-4: Access permissions and authorizations are managed, incorporating the principles of least privilege and separation of duties. PR.AC-5: Network integrity is protected (e.g. network segregation and network segmentation). PR.AC-6: Identities are proofed and bound to credentials and asserted in interactions. PR.AC-7: Users, devices, and other assets are authenticated (e.g., single-factor, multi-factor) commensurate with the risk of the transaction (e.g., individuals’ security and privacy risks and other organizational risks). Awareness and Training (PR.AT): The organization’s personnel and partners are provided cybersecurity awareness education and are adequately trained to perform their information security-related cybersecurity duties and responsibilities consistent with related policies, procedures, and agreements. PR.AT-1: All users are informed and trained. PR.AT-2: Privileged users understand their roles and responsibilities. PR.AT-3: Third-party stakeholders (e.g., suppliers, customers, and partners) understand their roles and responsibilities. PR.AT-4: Senior executives understand their roles and responsibilities. PR.AT-5: Physical and information security cybersecurity personnel understand their roles and responsibilities. PR.DS-1: Data-at-rest is protected. PR.DS-2: Data-in-transit is protected. PR.DS-3: Assets are formally managed throughout removal, transfers, and disposition. PR.DS-4: Adequate capacity to ensure availability is maintained. PR.DS-5: Protections against data leaks are implemented. PR.DS-6: Integrity checking mechanisms are used to verify software, firmware, and information integrity. PR.DS-7: The development and testing environment(s) are separate from the production environment. PR.DS-8: Integrity checking mechanisms are used to verify hardware integrity. PR.IP-1: A baseline configuration of information technology/industrial control systems is created and maintained incorporating security principles (e.g. concept of least functionality). PR.IP-2: A System Development Life Cycle to manage systems is implemented. PR.IP-3: Configuration change control processes are in place. PR.IP-4: Backups of information are conducted, maintained, and tested periodically. PR.IP-5: Policy and regulations regarding the physical operating environment for organizational assets are met. PR.IP-6: Data are destroyed according to policy. PR.IP-7: Protection processes are continuously improved. PR.IP-8: Effectiveness of protection technologies is shared with appropriate parties. PR.IP-9: Response plans (Incident Response and Business Continuity) and recovery plans (Incident Recovery and Disaster Recovery) are in place and managed. PR.IP-10: Response and recovery plans are tested. PR.IP-11: Cybersecurity is included in human resources practices (e.g., deprovisioning and personnel screening). PR.IP-12: A vulnerability management plan is developed and implemented. Maintenance (PR.MA): Maintenance and repairs of industrial control and information system components are performed consistent with policies and procedures. PR.MA-1: Maintenance and repair of organizational assets are performed and logged, with approved and controlled tools. PR.MA-2: Remote maintenance of organizational assets is approved, logged, and performed in a manner that prevents unauthorized access. PR.PT-1: Audit/log records are determined, documented, implemented, and reviewed in accordance with policy. PR.PT-2: Removable media is protected and its use restricted according to policy. PR.PT-3: The principle of least functionality is incorporated by configuring systems to provide only essential capabilities. PR.PT-4: Communications and control networks are protected. PR.PT-5: Mechanisms (e.g., failsafe, load balancing, hot swap) are implemented to achieve resilience requirements in normal and adverse situations. Anomalies and Events (DE.AE): Anomalous activity is detected and the potential impact of events is understood. DE.AE-1: A baseline of network operations and expected data flows for users and systems is established and managed. DE.AE-2: Detected events are analyzed to understand attack targets and methods. DE.AE-3: Event data are aggregated, collected, and correlated from multiple sources and sensors. DE.AE-4: Impact of events is determined. DE.AE-5: Incident alert thresholds are established. Security Continuous Monitoring (DE.CM): The information system and assets are monitored at discrete intervals to identify cybersecurity events and verify the effectiveness of protective measures. DE.CM-1: The network is monitored to detect potential cybersecurity events. DE.CM-2: The physical environment is monitored to detect potential cybersecurity events. DE.CM-3: Personnel activity is monitored to detect potential cybersecurity events. DE.CM-4: Malicious code is detected. DE.CM-5: Unauthorized mobile code is detected. DE.CM-6: External service provider activity is monitored to detect potential cybersecurity events. DE.CM-7: Monitoring for unauthorized personnel, connections, devices, and software is performed. DE.CM-8: Vulnerability scans are performed. Detection Processes (DE.DP): Detection processes and procedures are maintained and tested to ensure awareness of anomalous events. DE.DP-1: Roles and responsibilities for detection are well defined to ensure accountability. DE.DP-2: Detection activities comply with all applicable requirements. DE.DP-3: Detection processes are tested. DE.DP-4: Event detection information is communicated to appropriate parties. DE.DP-5: Detection processes are continuously improved. Response Planning (RS.RP): Response processes and procedures are executed and maintained, to ensure response to detected cybersecurity events. RS.RP-1: Response plan is executed during or after an event. RS.CO-1: Personnel know their roles and order of operations when a response is needed. RS.CO-2: Incidents are reported consistent with established criteria. RS.CO-3: Information is shared consistent with response plans. RS.CO-4: Coordination with stakeholders occurs consistent with response plans. RS.CO-5: Voluntary information sharing occurs with external stakeholders to achieve broader cybersecurity situational awareness. Analysis (RS.AN): Analysis is conducted to ensure effective response and support recovery activities. RS.AN-1: Notifications from detection systems are investigated. RS.AN-2: The impact of the incident is understood. RS.AN-3: Forensics are performed. RS.AN-4: Incidents are categorized consistent with response plans. RS-AN-5: Processes are established to receive, analyze, and respond to vulnerabilities disclosed to the organization from internal and external sources (e.g. internal testing, security bulletins, or security researchers). RS.MI-1: Incidents are contained. RS.MI-2: Incidents are mitigated. RS.MI-3: Newly identified vulnerabilities are mitigated or documented as accepted risks. Improvements (RS.IM): Organizational response activities are improved by incorporating lessons learned from current and previous detection/response activities. RS.IM-1: Response plans incorporate lessons learned. RS.IM-2: Response strategies are updated. Recovery Planning (RC.RP): Recovery processes and procedures are executed and maintained to ensure timely restoration of systems or assets affected by cybersecurity events. RC.RP-1: Recovery plan is executed during or after a cybersecurity event. Improvements (RC.IM): Recovery planning and processes are improved by incorporating lessons learned into future activities. RC.IM-1: Recovery plans incorporate lessons learned. RC.IM-2: Recovery strategies are updated. RC.CO-1: Public relations are managed. RC.CO-2: Reputation after an event is repaired. RC.CO-3: Recovery activities are communicated to internal and external stakeholders as well as to executive and management teams. Legend: ●—Fully address: the standards address all of the related subcategories. ◕—Substantially address: the standards address at least two-thirds, but not all, of the related subcategories. ◑—Partially address: the standards address at least one-third, but less than two-thirds, of the related subcategories. ◔—Minimally address: the standards address less than one-third of the related subcategories.○—Do not address: the standards do not address any of the related subcategories. In addition to the contact named above, Kaelin Kuhn (Assistant Director), David Marroni (Assistant Director), Andrew Moore (Analyst in Charge), Dino Papanastasiou (Analyst in Charge), David Aja, Christopher Businsky, Kendall Childers, Travis Conley, Rebecca Eyler, Philip Farah, Jonathan Felbinger, Quindi Franco, Wil Gerard, Cindy Gilbert, Mike Gilmore, Andrew Howard, Paul Kazemersky, Lisa Maine, Carlo Mozo, Cynthia Norris, Sukhjoot Singh, Adam Vodraska, and Jarrod West made key contributions to this report.", "summary": "The nation's electric grid—the commercial electric power generation, transmission, and distribution system comprising power lines and other infrastructure—delivers the electricity that is essential for modern life. As a result, the reliability of the grid—its ability to meet consumers' electricity demand at all times—has been of long-standing national interest. GAO was asked to review the cybersecurity of the grid. Among other things, this report (1) describes the cybersecurity risks facing the grid, (2) assesses the extent to which DOE has defined a strategy for addressing grid cybersecurity risks, and (3) assesses the extent to which FERC-approved standards address grid cybersecurity risks. To do so, GAO developed a list of cyber actors that could pose a threat to the grid; identified key vulnerable components and processes that could be exploited; and reviewed studies on the potential impact of cyberattacks on the grid by reviewing prior GAO and industry reports, as well as interviewing representatives from federal and nonfederal entities. GAO also analyzed DOE's approaches to implementing a federal cybersecurity strategy for the energy sector as it relates to the grid and assessed FERC oversight of cybersecurity standards for the grid. The electric grid faces significant cybersecurity risks: Threat actors . Nations, criminal groups, terrorists, and others are increasingly capable of attacking the grid. Vulnerabilities . The grid is becoming more vulnerable to cyberattacks—particularly those involving industrial control systems that support grid operations. (The figure below is a high-level depiction of ways in which an attacker could compromise industrial control systems.) The increasing adoption of high-wattage consumer Internet of Things devices—“smart” devices connected to the internet—and the use of the global positioning system to synchronize grid operations are also vulnerabilities. Impacts . Although cybersecurity incidents reportedly have not resulted in power outages domestically, cyberattacks on industrial control systems have disrupted foreign electric grid operations. In addition, while recent federal assessments indicate that cyberattacks could cause widespread power outages in the United States, the scale of power outages that may result from a cyberattack is uncertain due to limitations in those assessments. Although the Department of Energy (DOE) has developed plans and an assessment to implement a federal strategy for addressing grid cybersecurity risks, these documents do not fully address all of the key characteristics needed for a national strategy. For example, while DOE conducted a risk assessment, that assessment had significant methodological limitations and did not fully analyze grid cybersecurity risks. One such key limitation was that the assessment used a model that covered only a portion of the grid and reflected how that portion existed around 1980. Until DOE has a complete grid cybersecurity plan, the guidance the plan provides decision makers in allocating resources to address those risks will likely be limited. The Federal Energy Regulatory Commission (FERC)—the regulator for the interstate transmission of electricity—has approved mandatory grid cybersecurity standards. However, it has not ensured that those standards fully address leading federal guidance for critical infrastructure cybersecurity—specifically, the National Institute of Standards and Technology (NIST) Cybersecurity Framework. (See table below for an excerpt of GAO's analysis of two of the five framework functions.) Without a full consideration of the framework, there is increased risk that grid entities will not fully implement leading cybersecurity practices. In addition, FERC's approved threshold for which entities must comply with the requirements in the full set of grid cybersecurity standards is based on an analysis that did not evaluate the potential risk of a coordinated cyberattack on geographically distributed targets. Such an attack could target, for example, a combination of geographically dispersed systems that each fall below the threshold for complying with the full set of standards. Responding to such an attack could be more difficult than to a localized event since resources may be geographically distributed rather than concentrated in the same area. Without information on the risk of such an attack, FERC does not have assurance that its approved threshold for mandatory compliance adequately responds to that risk. GAO is making three recommendations—one to DOE and two to FERC. GAO is making a recommendation to DOE to develop a plan aimed at implementing the federal cybersecurity strategy for the grid and ensure that the plan addresses the key characteristics of a national strategy, including a full assessment of cybersecurity risks to the grid. GAO is also making the following two recommendations to FERC: 1. Consider adopting changes to its approved cybersecurity standards to more fully address the NIST Cybersecurity Framework. 2. Evaluate the potential risk of a coordinated cyberattack on geographically distributed targets and, based on the results of that evaluation, determine if changes are needed in the threshold for mandatory compliance with requirements in the full set of cybersecurity standards. DOE and FERC agreed with GAO’s recommendations.", "document_type": "gao"}
{"report": "A series of laws and policy directives dating back to 1904 require DOD to rely in large part on U.S.-flag commercial ships over government-owned or foreign-flag ships for its sealift needs. More recently, a 1989 National Security Directive reaffirmed the policy of relying on U.S.-flag commercial ships to provide sealift in times of peace, crisis, and war. These requirements and policies align with the following principles from the Merchant Marine Act of 1936, as amended: A fleet of commercial ships with military utility that are owned and operated by U.S. citizens and are able to provide reliable support during difficult wartime missions is necessary for national defense. According to testimony by the Commander of U.S. Transportation Command (USTRANSCOM), during Operation Desert Shield, 7 percent of foreign-flag ships refused to go into war zones, whereas U.S.-flag ships continued to deliver cargo as promised. DOD officials we interviewed also noted that U.S. mariners have a history of providing outstanding support to the nation, but cited several situations in which civilian mariners refused to complete a government mission due to security concerns. A pool of trained U.S. mariners is needed to crew the U.S.-flag fleet. According to USTRANSCOM and MARAD, U.S. mariners are necessary to crew not only the U.S.-flag commercial ships but also the U.S. government-owned reserve cargo ships. When put into full operating status—such as for a surge related to a wartime effort—the government needs additional trained and qualified mariners to operate these U.S. reserve cargo ships. U.S.-flag commercial ships, which are required to be staffed by U.S.-citizen mariners, provide a pool of mariners who can be used for this task. Because mariners work on ships for months at a time, commercial ships typically have at least two full sets of mariners to crew a single ship—one set of which is on the ship while the other is on leave. In times of crisis, one set of mariners could continue to work on the commercial ship, while some of those on leave could be called upon to voluntarily crew ships in the government-owned reserve fleet. A U.S. presence in international trade is needed to carry goods overseas. According to MARAD, a U.S. presence in international trade helps ensure that both commercial shippers and the military can access ships, and associated transportation networks, to carry their goods overseas at all times, both in times of peace and in times of war. The U.S. government financially supports oceangoing U.S.-flag ships in two key ways: (1) Maritime Security Program (MSP) stipends and (2) cargo preference requirements. Maritime Security Program: Since fiscal year 1996, the MSP has provided an annual stipend set by statute, subject to annual appropriations, to support a specific number of internationally trading U.S.-flag ships. In return for receiving the stipend, the MSP ship operator agrees to keep the ship or an equivalent ship under the U.S. flag for the life of the MARAD-issued operating agreement, and enrolled in a Voluntary Intermodal Sealift Agreement. By statute, the MSP is to enroll no more than 60 ships and provide each with a stipend of $5 million annually in fiscal years 2018-2020, subject to the availability of appropriations. The MSP was designed as a less costly replacement for the Operating Differential Subsidy that, since 1936, had subsidized the higher operating costs of the U.S.-flag fleet compared to foreign-flag ships operating on similar routes and trades. The MSP currently covers approximately 71 percent of the average annual operating cost differential between U.S. and foreign-flag ships, although this share varies across ships in the MSP, according to DOT’s estimates. The other key way that these ships can make up the operating cost differential is by carrying government cargo under cargo preference requirements. Cargo Preference: A series of laws requires federal agencies to transport some portion of their cargo on U.S.-flag ships, to the extent such ships are available at fair and reasonable rates. For example, current law requires that 100 percent of military cargo be transported on U.S.-flag ships, unless the rates are found by the President to be excessive or otherwise unreasonable. According to a 2015 MARAD report, DOD accounts for 59 percent of total government cargoes. For non-military cargo, including food aid, current law requires federal agencies to transport a minimum of 50 percent of their cargo on privately owned U.S.-flag commercial ships. Federal agencies can meet cargo preference requirements by transporting cargo on any privately owned U.S.-flag commercial ships, including those in the MSP. As we reported in 2018, federal stakeholders have differing views on cargo preference requirements. On the one hand, these requirements result in higher shipping costs for food aid agencies, costs that agency officials said negatively affect their missions. On the other hand, these requirements help support the financial viability of U.S.-flag ships by helping to offset the cost differential between U.S.-flag and foreign-flag ships. According to the 2015 MARAD report, the higher freight rates that DOD and other federal agencies pay to transport government cargo on U.S.-flag ships are critical to the financial viability of U.S.-flag ships in international trade, including MSP ships. In addition, the law commonly referred to as the Jones Act generally requires that maritime transport of cargo between points in the United States be carried by ships that are owned by U.S. citizens, registered under the U.S.-flag, and built in the United States. One of the purposes of the Jones Act is to provide the nation with a strong domestic maritime industry that can serve as a naval or military auxiliary in time of war or national emergency. As of August 2019, there were 99 oceangoing ships operating domestically (i.e., in the Jones Act fleet), according to MARAD data. We reported in 2013 that the effect of any potential modifications to the Jones Act on the U.S.-flag maritime industry would be uncertain. While repealing the Jones Act could increase competition with foreign-flag ships and reduce costs for shippers, it could also affect the reliability of the industry and have a negative effect on the U.S.-flag maritime industry and national security. DOT, DOD, and the Department of Homeland Security, among others, play a key role in federal policy related to the U.S.-flag maritime industry. Specifically: DOT, through MARAD, is the primary federal agency responsible for federal policy in support of the industry. DOT administers the MSP in consultation with DOD, provides funding to federal and state maritime academies, provides financial assistance to shipyards, and maintains a fleet of 56 government-owned cargo ships in reserve to provide sealift during war and national emergencies. DOD, through USTRANSCOM, jointly administers the MSP with DOT, and uses the U.S.-flag maritime industry to meet its sealift needs. DOD also maintains a fleet of 15 government-owned ships in reserve to provide sealift during war and national emergencies. We refer to DOT’s and DOD’s fleets together as the government-owned reserve fleet. The Department of Homeland Security, through the U.S. Coast Guard, oversees and regulates the U.S. maritime industry and marine transportation system. This includes overseeing and approving merchant mariner training programs, credentialing U.S. merchant mariners, documenting U.S.-flag ships, and maintaining the U.S, registry, among other functions. The Department of Agriculture and United States Agency for International Development administer multiple international food-aid programs. Under cargo preference requirements, they must use the U.S.-flag maritime industry to transport at least 50 percent of their government cargo when U.S.-flag ships are available at fair and reasonable rates. Since 2014, DOT has been required by law to develop two strategies: one to address industry challenges and the other to ensure the viability of U.S. sealift capability. First, the Howard Coble Coast Guard and Maritime Transportation Act of 2014 mandated that DOT, in consultation with U.S. Coast Guard, submit a national maritime strategy to Congress by February 2015. The law mandated that this strategy: Identify federal regulations and policies that reduce the competitiveness of the U.S.-flag maritime industry. Provide recommendations to make the fleet more competitive in international trade. Enhance U.S. shipbuilding capacity. In January 2018, the John S. McCain National Defense Authorization Act for Fiscal Year 2019 provided a new deadline of February 2020 for this strategy to be submitted. The second strategy, due in April 2014 and mandated by the Consolidated Appropriations Act of 2014, was to develop a national sealift strategy in collaboration with DOD to ensure the long-term viability of the U.S. Merchant Marine. This act additionally required DOT to identify the impact of reduced cargo preference requirements. DOT plans to submit a single maritime strategy to meet both these 2014 mandates. DOT completed a draft national maritime strategy that went through OMB interagency review in 2016. However, DOT did not finalize this strategy and submit it to Congress prior to the change in presidential administration. In August 2018, we recommended that DOT complete the strategy and publish a timeline for finalizing the strategy. DOT agreed to implement our recommendation. The U.S.-flag maritime industry faces an array of challenges that could negatively affect national defense. Federal assessments, as well as the federal officials we interviewed, underscored that the industry is critical to national defense, and that some potential sealift needs could be difficult for the U.S. industry to meet. All 10 of the industry stakeholders we interviewed identified at least one challenge related to each of the three broad sectors of the industry: (1) ships, (2) shipyards, and (3) mariners. Ships. Seven of the 10 stakeholders we interviewed expressed concern that declines in the size of the U.S.-flag fleet could lead to shortfalls in overall capacity or number of certain types of ships needed to carry defense cargo. Defense officials we interviewed and recent DOD needs assessments indicated that the current internationally trading U.S. fleet was generally sufficient to meet current needs but also raised some concerns about potential future gaps in certain situations. For example, the current U.S.-flag internationally trading fleet has 6 petroleum tankers—down from 36 in 1990—and USTRANSCOM has estimated potential needs for 86 tankers to fulfill DOD sealift requirements under the National Defense Strategy. Currently, according to USTRANSCOM officials we interviewed, U.S.-flag tankers and tankers flagged in other countries currently meet DOD needs, but these officials stated that access to allied foreign-flag petroleum tankers is increasingly uncertain in the current geo-political environment. Likewise, roll on/roll off ships (commonly referred to as Ro-Ros because it is possible to drive vehicles on and off the ships) are essential to move military vehicles, and DOD officials we interviewed stated they currently have assured access to roughly 3.5-million square feet of commercial capacity, which just meets current needs. A recent DOD analysis estimated 3.9 million square feet of Ro-Ro capacity will be needed in 2023. Seven of the stakeholders we interviewed raised concerns about limits in the overall capacity or a mismatch between the types of ships most needed for defense and those needed for commerce. Additionally, three stakeholders added that the Jones Act fleet—which is larger than the U.S.-flag fleet of internationally trading ships and also includes ships with sealift capabilities—would likely not be available in a time of crisis without significant disruption to U.S. domestic trade. Shipyards. Seven of the 10 stakeholders we interviewed expressed concerns about declines in U.S. shipyard capacity. According to MARAD and DOD officials, U.S. shipyards are an important part of ensuring government-owned cargo ships can be fully activated. According to a USTRANSCOM official, a shortage of shipyard capacity has contributed to increasing repair time for the government reserve fleet. In August 2017, we reported that incidents of degraded or out of service equipment in the government reserve fleet had increased over the previous 5 years. According to two stakeholders that operate U.S.-flag ships, U.S.-flag carriers are also experiencing maintenance delays at U.S. shipyards. For example, a representative of one U.S.-flag international carrier stated that it has difficulties scheduling needed work in a timely manner in the United States. In the face of these difficulties, as well as other business considerations, international ocean carriers may turn to foreign shipyards for repair services. Currently, according to MARAD, in April 2019, there were nine active shipyards in the United States with facilities capable of building large commercial ships. MARAD officials noted that while the domestic tug and barge industry is doing well, the side of the industry building large, self-propelled oceangoing ships is struggling due to declines in new orders. One stakeholder observed that U.S. shipyards are building very few new ships and noted that the industry could lose additional capacity in the coming years without a stream of new orders. Three stakeholders also expressed concern that shipyard workers have lost some of the necessary skills to support oceangoing commercial ships. Mariners. Nine of the 10 stakeholders we interviewed identified potential gaps in the skills or availability of U.S. citizen mariners. Likewise, federal officials we interviewed, as well as a recent government study, indicated there could be too few mariners to support sustained military sealift operations. When put into full operating status—such as for a surge related to a wartime effort—the government’s reserve fleet needs additional crew, and DOD counts on mariners working on oceangoing U.S.-flag ships to meet this need. MARAD and DOD have raised concerns about the sufficiency of U.S.- citizen mariners to meet this need. For example, in September 2017, in a statutorily mandated report, MARAD’s Maritime Workforce Working Group estimated a shortage of over 1,800 mariners in the event of a drawn-out military effort, although it also recommended data improvements to increase the accuracy of the count of available mariners. USTRANSCOM officials we interviewed added that they are concerned with not only the total number of mariners but also their specific mix of skills. Similarly, five stakeholders we interviewed identified potential mariner skills gaps because the U.S.-flag commercial fleet has modernized more quickly than the government- owned reserve fleet, so U.S.-citizen mariners in the commercial sector may lack experience with the technologies used on aging government-owned ships. Four stakeholders specifically noted potential shortages in mariners qualified to operate the 26 steam- powered ships that are in the government-owned reserve fleet, noting this older technology is no longer common on commercial ships. Seven stakeholders we spoke with stated that fleet and cargo reductions have led to fewer opportunities to crew ships, limiting career development paths for mariners. Current federal actions to address industry challenges and meet defense needs include administering long-standing policies and programs as well as studying underlying issues, rather than new efforts to confront these challenges. Established federal policies and programs—including the MSP, cargo preference requirements, and the Jones Act, among others— have not markedly changed in recent years. Officials explained that within the existing statutory framework, they have tried to better align the MSP fleet with defense needs. For example, within the last 2 years, they enrolled three roll on/roll off ships that provide a net increase in square- footage compared to the ships they replaced, among other improvements. Cargo preference requirements, also, have largely remained the same since 2012. MARAD has made efforts to better ensure these requirements are understood and followed by federal contracting officers, contractors, and sub-contractors who make shipping decisions and are supposed to abide by these requirements. Specifically, MARAD has developed training on cargo preference and conducted outreach to various agencies and industries. Further, agencies have taken other actions to improve existing programs in ways that could aid defense. These actions include initiatives to make it easier for veterans to earn merchant marine credentials and bureaucratic improvements to speed the process to flag a ship in the United States. Five of the 10 stakeholders we interviewed noted that these established policies and programs, collectively, are vital to the U.S.-flag maritime industry. Four stakeholders emphasized that the internationally trading U.S.-flag industry is supported by three sources of revenue—MSP stipends, government cargo, and commercial cargo—and stated that reductions in any of these three sources would likely cause further declines in the international-trading fleet. Similarly, an industry organization representing Jones Act carriers and a representative of a shipyard that builds ships for the fleet emphasized that the legal requirements for domestic shipping were essential to the viability of the fleet. In recent years, MARAD and other key federal agencies with maritime roles have focused on studying the industry and recent trends. DOT and DOD officials we interviewed identified several recent and ongoing efforts (see table 1). Currently, in response to a recommendation in the previously mentioned Maritime Workforce Working Group report, MARAD has begun a new effort to survey mariners to determine the number who are qualified, available, and willing to serve on short notice on U.S. government-owned sealift ships or commercial ships in times of national emergencies or to meet defense sealift needs. At this early date, DOT does not have specific plans for how to use the information gathered to change programs or practices. Likewise, USTRANSCOM regularly studies DOD’s sealift needs, through formal studies, ongoing cargo forecasts, and drills. For example, the Mobility Capabilities and Requirements Study of 2018 assessed the ability of mobility forces— including sealift capacity—to accomplish wartime missions as delineated in the 2018 National Defense Strategy based on anticipated fiscal year 2023 fleet capabilities and capacities. This study updated a similar study completed in 2010. Federal agencies have taken limited actions to address challenges industry stakeholders have identified, and the effects of those actions are unclear. For example, in addition to procuring or repairing ships as a customer, MARAD administers the small-shipyard grant program to provide cash support to sustain some shipyard capacity. MARAD officials we interviewed explained that while this grant program focuses on shipyards that tend to be too small to serve larger commercial ships needed to support defense sealift, it does help maintain the shipyard workforce. Similarly, the effect on carriers of U.S. food aid shipments is ambiguous in light of recent budget uncertainties. For example, officials we interviewed at the Department of Agriculture did not have estimates of food-aid cargo volumes beyond current appropriations because recent budget proposals from the administration have proposed eliminating much of the funding for these programs. According to 7 of the 10 stakeholders we interviewed, federal actions have not adequately addressed industry challenges, and many expressed concern that defense needs are at risk because of certain weaknesses in the federal approach. For example, four stakeholders noted that MARAD’s strategy to operate its fleet of government-owned reserve cargo ships in reduced operating status limits the opportunities for U.S.- citizen mariners to get experience on these ships, which may require distinct skills to operate (e.g., steam engines). Five stakeholders worried that federal actions were not working toward a common purpose or spurring industry innovation. Seven stakeholders stated that a comprehensive national strategy is needed to ensure, for example, that federal actions are working toward common goals to support the industry and are concerned that DOT had not yet submitted such a strategy, despite working on one since 2014. After a stalled strategy development process that did not include key stakeholders, in September 2019 DOT established a new interagency working group to finalize the strategy prior to the February 2020 deadline. Since 2017, the draft national maritime strategy, initially completed in 2016, has gone through three subsequent phases of development—DOT revision, OMB’s interagency review, and a renewed interagency working group. However, key federal agencies were omitted from DOT’s revisions and OMB’s interagency review. In September 2019, DOT formed a new interagency working group through the Committee on the Marine Transportation System (CMTS), an established interagency group for improving federal coordination and policies that affect the marine transportation system, as a way to bring key federal stakeholders together to finalize the strategy. DOT’s strategy revision. In 2017, the new administration instructed DOT to revise the existing draft strategy—which had been completed but not submitted to Congress under the prior administration—to align with its priorities. These priorities included DOD’s revised National Defense Strategy. Whereas DOT had held symposiums of maritime industry stakeholders and a broad array of federal agencies in 2014 when developing the initial draft strategy, DOT’s efforts to revise the strategy in 2017 and 2018 did not include substantive coordination activities with industry or other federal agencies. Subject matter experts within DOD reported to us in June 2019 that they had not seen a draft of the strategy since they provided comments during the OMB interagency review process that occurred in 2016. In addition, these DOD officials were unaware that the strategy was under revision. Similarly, in June and July 2019, officials at the Department of Homeland Security and subject matter experts within the U.S. Coast Guard told us they had not been consulted during the revision of the strategy since 2017. Accordingly, the largest government user of the U.S. flag fleet—DOD—and the agency overseeing credentialing of the U.S. Merchant Marine—the U.S. Coast Guard—were not able to provide input to DOT on revisions to the strategy mandated to ensure the long term viability of the U.S.-flag maritime industry. DOT officials we interviewed cited two main reasons for not engaging in new outreach and coordination specific to the revision of the strategy. First, they stated that the input they received in 2014 remained relevant as the challenges facing the industry have remained consistent. Moreover, DOT officials stated they are in regular contact with other federal agencies about maritime topics in general and, therefore, had a good understanding of these agencies’ positions. As a result, DOT officials told us they did not expect that the input they would receive from renewed outreach would be different from what they received in 2014. While DOT did not engage in substantive coordination during the strategy’s revision, it did provide status updates to some stakeholders on the progress of the strategy. For example, during a June 2018 meeting of the Marine Transportation System National Advisory Committee, DOT officials briefed industry representatives and participating federal agencies on the status of the strategy. During this briefing, DOT officials stated that the strategy had “undergone extensive revisions since 2015…but the vision, mission, and guiding principles are largely the same,” with the strategy refocused on areas where DOT plays a lead or major role. DOT officials stated this meeting afforded participants an opportunity to comment on topics germane to the strategy. DOT, however, did not circulate a draft of the strategy at this meeting, and so substantive reviews of the draft’s content were not possible. In August 2018, DOT completed its revisions and submitted the draft strategy to OMB for interagency review. OMB’s interagency review process. After receiving the revised strategy from DOT, OMB staff initiated the interagency review process. In August 2018, OMB staff sent the strategy to 12 federal agencies and 2 policy councils in the Executive Office of the President, according to OMB staff. DOT officials did not provide input to OMB on which agencies should review the strategy. According to DOT officials, they do not typically provide this type of input. We inquired with all 12 agencies and both councils whether they received the strategy from OMB and provided comments. As of September 2019, officials at six agencies or councils confirmed they had received the strategy in August 2018, and relevant officials at five agencies stated they did not have records of receiving the strategy. OMB staff we interviewed emphasized that it is the responsibility of each agency to make sure the strategy is provided to the right people within the agency. OMB did not include the Department of Agriculture, a major shipper of food aid, in the interagency review process, and Department of Agriculture subject matter experts told us they were not consulted by DOT during the revision of the strategy. Shortly after circulating the draft strategy, OMB suspended the interagency review process following a request from the EOP. According to OMB staff, in August 2018, an EOP policy council planned to convene a Policy Coordination Committee to address policy questions related to the strategy. As a result, OMB did not pass on the interagency comments it had already received to DOT, but instead provided those comments to the EOP policy council. According to DOT officials, OMB did not inform them that OMB had halted the interagency review process at the request of the EOP policy council. According to DOT officials, the process remained suspended until September 2019, when DOT officials learned from OMB staff that this committee had not and would not convene on the draft national maritime strategy. Furthermore, these DOT officials told us that until September 2019, when we informed them that OMB had suspended the process, they had been unaware that any such committee had been under consideration. Moreover, they indicated DOT had not worked with any EOP policy councils to resolve policy questions or concerns during that time. Likewise, DOD officials we interviewed also were unaware of any Policy Coordination Committee related to the strategy and had not worked with any EOP policy councils to resolve policy questions or concerns. As a result, from approximately September 2018 to September 2019, DOT was not working to advance the strategy, according to DOT officials we interviewed, nor did the OMB interagency process provide DOT with input from other agencies. Renewed interagency working group. Following our inquiries about both DOT’s and OMB’s interagency collaboration, in September 2019, DOT formed an interagency working group to finalize the strategy. According to DOT officials, following discussions with OMB, they understood that DOT could renew its efforts to finalize the strategy. According to DOT officials, DOT determined that the CMTS was the best forum for this finalization to occur. DOT officials explained the purpose of the working group is to receive substantive input from other agencies, fine tune the content of the strategy, and coordinate final edits. DOT officials told us the working group is open to any member of CMTS that elects to participate. Officials with CMTS we interviewed told us that it functions as an interagency forum for policy discussion and coordination, at the discretion of member agencies, and can help address issues that cut across multiple agencies. As of October 2019, participating agencies in the CMTS working group included DOT, OMB, DOD, and the Department of Homeland Security, among other agencies with maritime roles and responsibilities relevant to the strategy. DOT officials expected the working group to complete its work by the end of November 2019. After that time, DOT plans to send the strategy to OMB for an additional round of interagency review and clearance and to submit the finalized strategy to Congress. DOT officials stated the department remains committed to meeting the deadline to submit a finalized strategy by the February 2020 deadline. See figure 1 for the timeline of these three phases of development. Our previous work has found that national strategies are a mechanism for interagency collaboration, and that accordingly they can be used to address a range of purposes, including policy development and program implementation. We also found that collaborative mechanisms benefit from certain leading practices, including ensuring that all relevant participants have been included. These participants should have full knowledge of the relevant resources in their agency, the ability to commit those resources, and the knowledge, skills, and abilities to contribute to the collaborative effort. In addition, OMB guidance states that prior to submitting a document to OMB for interagency review, the submitting agency should make intensive efforts to reach agreement on policy issues in areas where there is overlapping interest between agencies. Since 2017, and throughout DOT’s revision of the strategy and the OMB interagency review initiated in 2018, key federal agencies and personnel were not included in the strategy’s development and lacked opportunities to provide their input on the strategy at that time. Without these agencies’ input, DOT did not have assurance that the strategy incorporated the agencies’ expertise or the most up-to-date information relevant to the strategy, including on DOD’s most recent sealift needs and priorities. Given the interconnected nature of maritime issues and the breadth of the statutory requirements for DOT to address in the strategy, including provisions that call for collaboration with DOD, interagency collaboration is an important step toward developing an effective national strategy. DOT’s work with the CMTS interagency working group should help ensure such collaboration and the input of key stakeholders that had previously not contributed to the revision of the strategy. In light of this new effort and our prior recommendation in 2018 that DOT complete and finalize the strategy, we are not making a new recommendation in this report. We provided a draft of this report to DOD, OMB, DOT, and the Department of Homeland Security for review and comment. DOD, DOT, and the Department of Homeland Security provided technical comments, which we incorporated as appropriate. OMB told us that they had no comments on the draft report. We are sending copies of this report to the appropriate congressional committees, Secretaries of Transportation, Defense, Homeland Security, and Director of OMB and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the individual named above, Alwynne Wilbur (Assistant Director); John Stambaugh (Analyst in Charge); David Blanding; Lilia Chaidez; Emil Friberg; Geoffrey Hamilton; Dawn Hoff; Diana Maurer; Jan Montgomery; Valerie Nowak; Josh Ormond; Molly Ryan; Travis Schwartz; Sarah Veale; Michelle Weathers; and Suzanne Wren made key contributions to this report.", "summary": "DOT's efforts related to a national maritime strategy aimed at helping to ensure the sustainability and competitiveness of the U.S.-flag fleet were first mandated in statute in 2014. In 2018, the due date for the national maritime strategy was extended to February 2020. A provision in statute directed GAO to identify the challenges facing the U.S. maritime industry and the status of the national maritime strategy. This report (1) identifies selected stakeholders' views on the key national defense implications of the challenges facing the U.S. maritime industry, among other things, and (2) examines the status of the national maritime strategy and the extent to which DOT coordinated the strategy's development with relevant federal agencies. GAO reviewed relevant laws and analyzed DOT and DOD documents related to the U.S. flag fleet. GAO also interviewed: (1) staff in the Executive Office of the President, including OMB, and (2) officials in DOT, DOD, and other federal agencies as well as selected industry stakeholders. Interview selections were based on a range of factors to gather different perspectives across the industry and results are not generalizable to all industry stakeholders. Selected stakeholders identified some key national defense implications of the challenges facing the U.S. maritime industry. This industry—which includes oceangoing U.S.-registered (i.e., U.S.-flag) ships and U.S. citizen mariners—provides global transportation capabilities to the Department of Defense (DOD) in times of peace, crisis, and war. The Department of Transportation (DOT), in cooperation with DOD and other federal agencies, is responsible for federal programs to ensure that this industry meets defense needs. Stakeholders, as well as DOD officials, cautioned that continued declines in the size and capabilities of the oceangoing U.S.-flag fleet could lead to inadequate capacity for DOD to transport military cargo during a national defense crisis. Likewise, a potential shortage of mariners could lead to DOD not having adequate crews to operate government-owned reserve ships that may be activated during a wartime surge. Seven of the 10 industry stakeholders GAO interviewed stated that a comprehensive national strategy could help address industry challenges. After a stalled strategy development process that did not include key stakeholders, DOT established a new interagency working group, in September 2019, to finalize the national maritime strategy. DOT has been working on a draft strategy since 2014 to address statutory mandates. In 2017, DOT began revising the draft strategy to align with the new administration's priorities. Interagency coordination, however, was limited as DOT did not include DOD or other key federal stakeholders. In August 2018, DOT submitted the revised draft to the Office of Management and Budget (OMB) for interagency review. OMB staff told GAO they circulated this draft to 12 agencies and two policy councils in the Executive Office of the President. However, according to OMB staff, OMB suspended this process shortly after it began at the request of the Executive Office of the President because of its plans to convene a committee to consider policy matters related to the strategy. According to DOT officials, the process remained suspended until DOT learned in September 2019 that the Executive Office of the President had not convened and no longer planned to convene such a committee. DOT then established a new interagency working group to revise and finalize the strategy, ending a year-long delay in the strategy's development (see figure). This working group includes DOD and other key agencies that were not previously consulted and should address gaps in interagency coordination. DOT officials told GAO that they intend to submit the strategy to Congress by February 2020, as required.", "document_type": "gao"}
{"report": "The NDAA requires DHS to develop and implement 43 border security metrics in four domains—between POEs, at POEs, the maritime border, and air and marine security in the land environment. Within DHS, CBP and the Coast Guard have primary responsibility for border security within these four domains. CBP and its subcomponents are to secure U.S. borders at and between POEs by preventing inadmissible people and illicit goods from entering the United States, among other responsibilities. Within CBP, the primary offices and components involved in border security are the Office of Field Operations at POEs, Border Patrol between POEs, and Air and Marine Operations for air and marine security in the land and maritime domains. The Coast Guard and CBP’s Air and Marine Operations share responsibility for security of the nation’s maritime borders. Table 1 shows examples of border security metrics by domain and responsible DHS component. According to DHS officials, within DHS, two subcomponents within the Office of Strategy, Policy, and Plans were responsible for coordinating the department’s effort to develop the fiscal year 2017 Border Security Metrics Report. A senior DHS official explained that the report was initially tasked to the Unity of Effort Integration Office, which was part of the Unity of Effort initiative started in 2014 to better understand border security efforts along the southwest border including exploring the development of border security metrics. OIS assumed responsibility for the report in 2017. According to OIS officials, to prepare the report, they obtained data and information related to each NDAA metric from the administrative records of the DHS components with primary responsibilities for border security in the four domains. For example, OIS requested data and information on “turn backs” and “got aways” from Border Patrol—the lead component for the between POE domain—which records sector estimates of turn backs and got aways based on direct and indirect observations. Of the 43 metrics the NDAA listed for inclusion in the Border Security Metrics Report, the majority were counts and rates of border security activities. The remaining metrics were estimates or were not specifically described. For example, the number of apprehensions in each Border Patrol sector is a count metric. In contrast, a rate metric compares one value or number against another. For example, the wait time ratio compares the average wait times to total commercial and private vehicular traffic volumes being processed at a land POE. An estimate is used for metrics of flows or activities that are largely undetected and therefore cannot be measured directly and must be estimated, such as the number of undetected unlawful entries. A few metrics are a combination of counts or rates with an estimate. For example, the metric for total inadmissible travelers at POEs counts known inadmissible travelers that are intercepted at POEs, and also requires an estimate of how many inadmissible travelers may have successfully entered at a POE without being detected, which cannot be directly measured. The NDAA did not specifically describe some metrics. For example, while the NDAA asked for an examination of each of the eight consequences under the Consequence Delivery System, it did not specify how this examination was to be carried out or what it was to include. While many of the metrics required by the NDAA can be addressed with data from DHS’s administrative records, certain metrics that rely on estimates necessitate the use of alternative methodologies and in some cases, specialized technical expertise. For example, DHS contracted with the Institute for Defense Analyses to assist with the development of a statistical model for estimating undetected unlawful entries. In its fiscal year 2017 Border Security Metrics Report, DHS provided information on its methodological approaches, such as how it estimated undetected unlawful entries. In its first Border Security Metrics Report, DHS reported information on 35 of the 43 metrics called for by the NDAA. The metrics DHS provided spanned the four domains outlined in the NDAA and included a mix of counts, rates, estimates, or a combination thereof as shown in figure 1. For 18 of the 35 border security metrics DHS included in its report, we found DHS generally included elements listed in the NDAA. For example, the NDAA asked for the number of detected unlawful entries between POEs, and in its report DHS provided information on the number of detected unlawful entries over a 10-year period. As another example, the NDAA asked for the number of cargo containers at sea ports that were identified to be potentially high-risk. In response, DHS provided information on the number of potentially high-risk containers from fiscal years 2013 through 2016 and also provided contextual information about trends in the volume of such containers over time. See table 2 for more information on these examples as well as other examples of the types of information included in DHS’s fiscal year 2017 Border Security Metrics Report. For some metrics, DHS also provided information in addition to the elements listed in the NDAA. For example, the NDAA described the “AMO apprehensions assisted” metric as a count of the number of apprehensions that were assisted by CBP’s AMO through the use of unmanned aerial systems and manned aircraft. In addition to the counts for such assists, DHS also provided data on the flight hours expended to assist with these apprehensions. For 17 of the 35 reported metrics, we identified differences between the metric as described by the NDAA and as reported by DHS. The differences we identified generally fell into two categories: Metric differed in scope or calculation. Some of the metrics DHS reported on differed in scope or in their calculation from what the NDAA described for reasons such as data availability, among other factors. For example, DHS’s fiscal year 2017 Border Security Metrics Report scoped three metrics on unlawful border crossings between POEs (the “attempted unlawful border crosser apprehension rate,” the “estimated undetected unlawful entries,” and the “probability of detection rate”) to only include data for the southwest border. In these instances, the report noted that a methodology for estimating data on unlawful crossings for the northern border had not yet been completed but that research was underway to do so. As an example of a difference in calculation, DHS presented the interdiction effectiveness rate for each southwest border sector as an alternative to the metric “unlawful border crossing effectiveness rate in each Border Patrol sector.” According to DHS’s report, the department used the interdiction effectiveness rate because it had not yet produced and validated sector-level estimates of unlawful entries required to calculate the unlawful border crossing effectiveness rate. In its report, DHS stated it expects these estimates to be available for the 2019 report. Alternative metric provided. For the situational awareness in the maritime environment metric, DHS stated that it is in a multi-year process to develop a metric that meets the intent of the NDAA. As an alternative, DHS instead provided data on the number of aircraft and vessel operational hours that contributed to maritime domain situational awareness. See appendix I for additional information about any differences we identified for each metric. The eight metrics on which DHS did not provide information spanned all four domains. In its report, DHS explained that the eight omitted metrics were either still in development, under review within the department, or officials were in the process of collecting data for them. Table 3 lists the eight metrics on which DHS did not provide information and the date DHS estimated it will report on each metric. In general, DHS components responsible for collecting the data used in the metrics DHS reported have processes to help ensure the reliability of the data and the quality of the information provided. DHS also identified and disclosed limitations with some of the data elements or methodologies used for the metrics in its report. However, DHS does not have a systematic process for reviewing the reliability of data to identify limitations related to the metrics, and we identified at least one additional limitation for 21 of the 35 metrics on which DHS reported where DHS did not disclose such limitations or could have been more transparent about the limitations or assumptions in its report. Data are considered reliable when they are reasonably free from error and bias. Quality information is derived from relevant and reliable data and is considered to be, among other things, complete, accurate, and timely. The specific processes DHS components use to ensure data reliability vary from metric to metric. Examples of processes DHS or its components have implemented to help ensure the reliability of the data and the quality of information provided include: Issuing guidance and monitoring implementation. In September 2012, Border Patrol headquarters officials issued guidance to help provide a more consistent, standardized approach for the collection and reporting of turn back and got away data by Border Patrol sectors. Each sector is individually responsible for monitoring adherence to the guidance. According to DHS’s report, command staff at Border Patrol stations ensure agents are aware of and utilize proper definitions for apprehensions, got aways, and turn backs at their respective stations and also ensure that the necessary communication takes place between and among sectors and stations to minimize double-counting when subjects cross over multiple areas of responsibility. Supervisory reviews of data entries. With regard to data on AMO vessel and aircraft missions, AMO guidance mandates that supervisors perform a review of all pre- and post-mission data entries to help ensure accurate entry of mission information. AMO officials confirmed that supervisors review the data being entered into the database. Additionally, officials said AMO data teams run monthly validation checks of data entered to check for completeness and accuracy, such as out-of-range values. Using built-in electronic safeguards. CBP’s databases for entering and maintaining data elements—including travelers or passengers seeking admission, known inadmissible aliens at POEs, referrals for secondary examinations, major infractions, and private vehicles processed at a POE—have built-in processes to detect and prevent potential data entry errors. More specifically, as an officer enters a record, the systems check for valid entries into relevant fields and provide an error message to the officer for entries that appear to be invalid (e.g., if an officer leaves a mandatory field blank or enters contradictory information such as charging an individual with a crime while also entering a request for expedited removal). In some cases, the systems will prevent a record from being saved if any required fields are blank. Comparing data against other sources. As part of the Coast Guard’s data reliability processes for data on maritime migrant interdictions used in the “known maritime migrant flow rate” metric, Coast Guard officials said that analysts cross-check the data entered into their database with other Coast Guard reporting documents, such as internal spreadsheets, to ensure accuracy. Independent assessment of performance measure data. Some border security metrics are similar to, or use the same data elements as, performance measures DHS reports annually in response to the Government Performance and Results Act Modernization Act (GPRAMA) of 2010. For those performance measures, DHS annually assesses a subset of measures and their data for completeness and reliability using independent review teams. For example, in May 2017 an independent review team assessed the “migrant interdiction effectiveness in the maritime environment” performance measure, which uses the same data as the border security metric, “known maritime migrant flow rate.” The review team found the measure to be complete and reliable and the data to be of good quality overall, but also recommended that the Coast Guard and DHS continue work on an improved database to enhance the consistency of data collection, among other things. In addition to the components having processes to help ensure the reliability of the data and the quality of the information used in the report, DHS took steps to be transparent in its presentation of the metrics by identifying and disclosing known limitations with some of the data elements or methodologies used for the metrics in its report. Communicating the extent to which such limitations exist and their potential impact is important to help facilitate the appropriate use and understanding of the data and the metrics. DHS identified and disclosed limitations related to the potential for misclassification of observations, the potential for cases not being entered or recorded correctly, and methodological limitations, among other things. For example, one of the key limitations DHS’s report identified for the data on turn backs and got aways is that they are based on potentially subjective observations of agents who have to make a determination on how to classify them based on what they observed or the available evidence (e.g., tracks, sensor activations, interviews with apprehended subjects, camera views, etc.). Further, DHS’s report explained that agents may face challenges in making that determination because some unlawful border crossers may enter the United States to drop off drug loads or to act as decoys to lure agents away from a certain area and then return to Mexico, and therefore may be misidentified as turn backs, for example. As another example, DHS identified limitations due to cases not being entered or recorded correctly. For the “known maritime migrant flow rate” metric, DHS used data on the total number of maritime migrants interdicted. In its report, DHS explained that a potential limitation of this data element is that the Coast Guard relies on international and domestic partners to report their interdictions for compilation in its database. Consequently, the accuracy and completeness of the data depend on whether those reports are made by those partners and the accuracy of their reports. See appendix I for additional information about the limitations identified for each metric. Even as DHS identified and disclosed limitations related to some of its metrics, we identified at least one additional limitation for 21 of the 35 metrics on which DHS reported where DHS did not disclose such limitations or could have been more transparent about the limitations in its report. Examples of such instances include: Potential for cases not being entered or recorded correctly. In our previous work we found that mission data for unmanned aerial systems were inconsistently collected across operation locations. Specifically, in February 2017 we reported that there were instances where no assist information was recorded in AMO’s data system even though such assets participated in investigations and operations. Because AMO’s data may not reflect all asset assists, we recommended that AMO update and maintain guidance for recording mission information in its data collection system and provide training to users of the system. For its fiscal year 2017 Border Security Metrics Report, DHS used asset assists data in metrics such as the “AMO individuals detected,” “AMO apprehensions assisted,” and “illicit drug seizures assisted by AMO,” but did not disclose this limitation in its report. Potential for data to be changed over time. Border Patrol officials told us that data on the apprehension of unaccompanied alien children may change over time because original apprehension records from a shared database have, in some instances, been updated by staff from U.S. Immigration and Customs Enforcement (ICE) Enforcement and Removal Operations (ERO). Officials said that in January 2015 they noticed that ERO staff were inadvertently overwriting Border Patrol’s original data entries about the status of apprehended children when they made updates to those children’s records. For example, if a child was unaccompanied at the time of his or her apprehension and was recorded as such by Border Patrol in the initial record entry, ERO may have changed the “unaccompanied” status in the system after they matched the child with a family member or sponsor. As a result, data may not be reconcilable with initial apprehension counts over time. DHS did not fully disclose limitations for some metrics. We identified instances where DHS could improve transparency about the assumptions or limitations of the data presented in its report. For example, in 2014 Border Patrol implemented a standard, southwest border-wide methodology to improve reporting of turn backs and got aways. While DHS made mention of this change in the text of the report, the data for these metrics are presented in tables without any table notes or disclosures within the table about this change. Further, DHS’s report does not discuss how the change may affect comparability of the data. Consequently, a reader may not be aware that data for before 2014 in a table are not necessarily comparable to the data for 2014 and after in the same table. Without a comprehensive identification of the limitations of the metrics and their associated data, and without an adequate disclosure of those limitations, the value of DHS’s report as a source of information to Congress, policymakers, and the public may be diminished. The metrics in the report were specifically identified and requested by Congress in the NDAA, and provide Congress with important information about the outputs and outcomes of DHS’s border security policies and investments that could be used to inform decision-making. However, those reading the report may not be aware of important contextual information because DHS did not identify and disclose some limitations, thereby creating the potential for the data to be misinterpreted. According to DHS officials who prepared the report, while they took steps to identify methodological limitations of the metrics, no process currently exists to systematically review the reliability of operational data used for public reporting purposes, such as in the metrics report. Specifically, DHS officials within OIS told us that while they were responsible for leading and managing the preparation of the report, they largely relied on the DHS components from which they collected the data to assess the data’s reliability and communicate identified limitations. OIS officials explained that many of the data elements used, such as those from AMO or the Coast Guard, were ones with which they were not familiar or had not worked with previously in their area of immigration statistics. OIS officials also noted that in some cases, the data had previously been used in performance measures or had been collected and tracked for several years, so they trusted the components’ processes for ensuring their reliability and identifying limitations, but reviewed the data provided where possible and consulted with the components as needed. However, OIS officials said that while they included as much information in the report as was known about identified limitations with the existing operational data, no additional effort was made to systematically review the underlying reliability of the data to comprehensively identify limitations that should be acknowledged when publicly reported because no department-wide process exists to do so. Standards for Internal Control in the Federal Government state that management officials should evaluate data sources for reliability and communicate quality information, including relevant data from reliable sources, to achieve an agency’s objectives. The quality information can then be used by agency management and external stakeholders such as policymakers, to make informed decisions and evaluate performance, among other things. Further, DHS’s Management Directive on Information Quality states that data and information disseminated by the department should, among other things, have full, accurate, transparent documentation, and error sources affecting data quality should be identified and disclosed to users. Additionally, our previous work on approaches for verifying and validating performance information found that communicating significant data limitations and their implications allows stakeholders to judge the data’s credibility for their intended use and to use the data in appropriate ways. By developing and implementing a process to systematically review the reliability of the data or consider the results of assessments components have completed, comprehensively identify any limitations, and communicate the data or methodological limitations with the metrics, DHS would improve the quality of the information available to Congress, DHS leadership, and the public. Doing so would also facilitate a better understanding and appropriate interpretation and use of the data in the context of the Border Security Metrics Report, thereby enhancing the report’s value as a source of information for future decision-making. Based upon statistical modelling, DHS developed a Model-based Apprehension Rate to calculate the total number of unlawful border entries between land POEs, including entries both detected by Border Patrol and “estimated undetected unlawful entries.” DHS reported that in fiscal year 2016 there were about 624,000 detected entries (which include apprehensions, turn-backs, and got aways) and estimated that there were about 62,000 undetected unlawful entries. DHS also used the Model- based Apprehension Rate to develop two other metrics in the fiscal year 2017 Border Security Metrics Report: (1) A “probability of detection rate,” which is the estimated proportion of the number of detected unlawful border entries to the total number of unlawful entries between land POEs. DHS estimated that in fiscal year 2016, 91 percent of unlawful border crossers were detected and 9 percent were not detected. (2) The “attempted unlawful border crosser apprehension rate,” which is the estimated proportion of unlawful border entrants apprehended by Border Patrol to the total number of unlawful entrants between land POEs. DHS estimated that in fiscal year 2016, 65 percent of individuals were apprehended by Border Patrol and 35 percent of individuals attempting an unlawful border entry either got away or entered the United States undetected. DHS based its statistical model upon research conducted by the Institute for Defense Analyses that leveraged long-standing research using capture-recapture models. Originally developed and utilized in biological and ecological sciences, capture-recapture models have been applied to other disciplines, including social science. According to the Institute for Defense Analyses, capture-recapture models have been the core approach for academic efforts to model the process of unlawful entry into the United States across land borders for several decades. To develop its statistical model, DHS used a capture-recapture methodology to calculate a probability of apprehension by counting the number of unlawful border crossers that were apprehended multiple times. At a high-level, capture-recapture involves taking an initial sample of the population of interest, in this case individuals attempting to cross the border unlawfully. Then, separately, a second, independent sample of the same population is taken. The samples are then compared to determine the number of individuals who appear in both samples. When the number of individuals who appear in both samples (e.g., individuals who have been apprehended twice) is low, it can be inferred that the overall population of interest (e.g., total unlawful border crossers) is much larger than the total number of individuals in the two samples. On the other hand, if the recapture rate is high, then it can be inferred that the overall population of interest is not much larger than the total number of individuals in the two samples. In the context of unlawful border crossing, when an individual’s first attempt at unlawfully crossing the border is successful, the individual enters the United States and no apprehension is made. However, if an individual is apprehended, Border Patrol records an apprehension of this individual in a DHS data system and the individual is potentially subject to consequences for entering unlawfully, such as administrative enforcement and removal, criminal prosecution, or being barred from legally entering the United States in the future. The individual is then returned to his or her home country, where the individual can then choose whether or not to make another attempt to unlawfully cross the border. During a second attempt to unlawfully cross the border, the individual faces the same possible outcomes (enter the United States unlawfully or apprehension by Border Patrol). Figure 2 provides the framework for DHS’s Model-based Apprehension Rate. DHS modified the traditional capture-recapture methodology by calculating a deterrence rate of 60 percent in fiscal year 2016 to account for individuals who choose not to make another unlawful border crossing attempt. The deterrence rate accounts for an individual being deterred from attempting to unlawfully cross the border again; that is, DHS assumed that some percentage of apprehended individuals, once returned to their country, will remain in their home country. DHS calculated the deterrence rate based upon a survey of Mexican individuals who were apprehended and returned to the border region of Mexico by U.S. immigration authorities. DHS assumed the remaining 40 percent of individuals who were apprehended and removed to their home country in fiscal year 2016 remain undeterred and will attempt to unlawfully cross the border again. Historically, DHS (and its predecessor the Immigration and Naturalization Service) did not use statistical models to calculate an apprehension rate but relied on apprehensions as a proxy measure for all unlawful entries (both observed and unobserved) between POEs. DHS also included in its report information on the apprehension rate using this method. Specifically, DHS also calculated an Observational Apprehension Rate based on direct observations (unlawful border crossers observed by Border Patrol) and indirect observations (residual evidence of a border crosser, i.e., footprints) of attempted unlawful border crossers. Using the observational apprehension rate, DHS calculated that in fiscal year 2016, it apprehended 79 percent of unlawful border crossers. DHS made assumptions about border crossers to develop its statistical model and described these assumptions in its report; however, DHS did not validate some of these assumptions or determine how they potentially could affect the accuracy of the Model-based Apprehension Rate through the use of sensitivity analyses. More specifically, DHS’s model incorporates several assumptions related to border crossers. Among others, these assumptions include: the rate at which individuals will be deterred from crossing again remains the same, regardless of the number of attempts an individual has made; individuals who indicate an intent to stay near the U.S.-Mexico border will attempt re-entry; a single apprehension rate applies to diverse groups of border crossers, regardless of their nationality or the number of attempts an individual has made; and certain individuals will not evade Border Patrol. However, the validity of some of these assumptions—which affect the Model-based Apprehension Rate—is uncertain. For example, DHS’s model estimates the rate at which a diverse group of border crossers attempting to evade detection will be apprehended by Border Patrol. This group includes both Mexicans and non-Mexicans and individuals who attempt to cross again after varying amounts of time. Despite this diversity, the model assumes that all crossers have the same chance of apprehension on each attempt to cross the border. This assumption allows DHS to apply the estimated apprehension rate developed based on a sample of Mexicans re-apprehended within 90 days—the group for whom relevant data exist—to a broader population of individuals regardless of the number of attempts the border crossers have made or their nationality. However, DHS did not make efforts to determine the extent to which an apprehension rate based on Mexican citizens re- attempting entry within 90 days would reflect apprehension rates for non- Mexicans and individuals crossing again after longer periods. Additionally, DHS assumes that the apprehension rate never varies between an individual’s attempts at crossing the border. For example, DHS assumes that an individual making a first attempt at crossing the border faces the same odds of apprehension as an individual making a fourth or fifth attempt at crossing the border. However, DHS has not explored the possibility that, for example, individuals may gain experience and knowledge from border-crossing attempts that could help them better evade Border Patrol on subsequent attempts. Further, DHS’s model assumes that certain individuals unlawfully crossing the border, such as those seeking asylum, will not evade apprehension and will turn themselves in to Border Patrol. Specifically, in addition to individuals who ultimately do seek asylum, DHS also includes within this group and applies this assumption to individuals apprehended as a family unit and unaccompanied minors. Under this assumption, 100 percent of such individuals are apprehended. According to DHS’s fiscal year 2017 Border Security Metrics Report, these individuals have historically been released into the United States with a Notice to Appear in immigration court for legal proceedings on a future date, rather than being subject to immediate DHS enforcement consequences such as voluntary return. Therefore, DHS assumes that 100 percent of these individuals will self-present to Border Patrol because, in doing so, they are able to claim asylum or other protection and potentially remain in the United States. However, representatives from the Institute for Defense Analysis stated that while anecdotally self-presenting rates of these individuals are high, more rigorous analysis is needed to accurately estimate a self- presentation rate. For example, it is possible that not all families crossing the border unlawfully may seek to self-present to Border Patrol; some may attempt to evade capture and enter the United States undetected. In this case, DHS may be underestimating the number of individuals who unlawfully cross the border and enter the United States by assuming 100 percent of these individuals will self-present to Border Patrol agents. Additionally, DHS noted in its fiscal year 2017 Border Security Metrics Report that this assumption does not reflect the actual behavior of all border crossers in this group. OIS officials stated that they based this assumption on interviews with Border Patrol agents but had not done formal or quantitative analysis to support this assumption. Further, OIS officials stated that they did not have a strong alternative assumption to use instead and therefore assumed that 100 percent of individuals within this group are apprehended. DHS described these assumptions in its report but did not provide quantitative information on the extent to which these assumptions affected the Model-based Apprehension Rate through the use of sensitivity analyses. Sensitivity analyses help to convey the extent to which changing the values of variables, assumptions, data, or other input affects statistical estimates. For example, sensitivity analyses could provide information on how different assumptions about unlawful border crossers’ behavior and other inputs to the statistical model could have affected the Model-based Apprehension Rate. OIS officials stated that while they had started to run sensitivity analyses by modifying certain assumptions, they had not completed the analysis and did not include results of the sensitivity analyses in the report. The Office of Management and Budget’s (OMB) statistical standards for federal agencies include providing the results of sensitivity analyses for key methodological assumptions to ensure that these assumptions do not unduly affect the results of the model. By including the results of sensitivity analyses in its Border Security Metrics Report, DHS would allow Congress and the public to better understand the potential limitations associated with its model and make independent assessments on its accuracy. DHS used a statistical model to develop the Model-based Apprehension Rate but did not provide information on the level of uncertainty related to this estimate. Rather, the fiscal year 2017 DHS Border Security Metrics Report provided a single rate that does not fully convey the difficulty and uncertainty of estimating partially unobserved metrics, such as unlawful entries and the probability of detection. Specifically, using the Model- based Apprehension rate, DHS estimated that 65 percent of unlawful border crossers were apprehended in fiscal year 2016, and the remaining 35 percent entered the United States. However, like all statistical models, DHS’s estimate is based upon a limited sample of data and may be affected by random variation, meaning that DHS does not have complete certainty that its rate is accurate. DHS included a discussion of limitations in the report but did not quantify its degree of uncertainty. According to the OMB statistical standards for federal agencies, possible variation in estimates should be noted, such as by reporting the range of each estimate. Measures of statistical uncertainty, such as margins of error or confidence intervals, help to convey the amount by which estimates might vary due to randomness in the data and allows consumers of the estimates to evaluate their accuracy. OIS officials stated that they agree that providing measures of statistical uncertainty would help Congress and the public better understand the Model-based Apprehension Rate to evaluate border security. Officials told us that the office had begun to develop measures of statistical uncertainty but did not complete this effort because the staff member who was working on the analyses recently left the office. Further, OIS officials stated that they were unsure when they would be able to provide measures of statistical uncertainty in future reports. Including measures of statistical uncertainty in future reports would allow Congress, policy makers, and the public to more fully evaluate the extent to which the metrics that use the Model-based Apprehension Rate are valid. Further, while DHS may ultimately adopt a new, simulation-based model in the future, described later in this report, it plans to use the current Model- based Apprehension Rate for estimates in its Border Security Metrics Report for the foreseeable future. Therefore, providing this additional information about the estimates would allow DHS to more accurately convey how limitations in available data and methods could affect the results and provide more useful information about migration and border enforcement. Additionally, to the extent DHS adopts a new estimating metric, that estimate may have some level of uncertainty associated with it. DHS is developing another model because its current statistical model may not sufficiently reflect conditions at the southwest border. Specifically, DHS’s current statistical model does not fully account for the changing population of unlawful border crossers. The capture-recapture methodology, which underlies the Model-based Apprehension Rate, was developed to sample homogenous populations that behave in set, uniform ways. However, those crossing the border have become increasingly diverse in recent years. Our analysis of DHS data used to develop the Model-based Apprehension Rate shows that the number of unlawful border crossers whose characteristics and behavior are best reflected in the statistical model has declined. For example, our analysis illustrated that the population that conforms best to the model’s assumptions—adult Mexicans travelling without dependents who do not plan to claim asylum and who are returned to Mexico in a short amount of time—has fallen from over 60 percent of apprehensions in fiscal year 2000 to less than 25 percent of apprehensions in 2016, as shown in figure 3. Conversely, the number of individuals who are excluded from the statistical model such as non-Mexicans, and individuals whose behavior may not reflect the model’s assumptions, such as asylum-seekers or those who have not departed the United States (e.g., because they are awaiting immigration court proceedings) have increased over time, as shown in figure 4. For example, the percentage of individuals apprehended at the border who are excluded from the model because they await immigration court proceedings increased from 26 percent in fiscal year 2000 to almost 70 percent in fiscal year 2016. DHS acknowledged these trends in its fiscal year 2017 Border Security Metrics Report and noted them as a limitation to the effectiveness of its model. OIS officials further noted that some of these limitations are difficult to address within the bounds of the statistical model. For example, to properly account for non-Mexicans, OIS officials stated that they would need information on the rate at which non-Mexicans are deterred from crossing the border. However, it would be difficult and costly to obtain this information through the use of a survey and real-world data does not already exist, according to OIS officials. To help address limitations of its current statistical model, DHS has invested in another research project to estimate the number of unlawful border crossers between land POEs, including unknown border entries. Border Patrol contracted with Johns Hopkins Applied Physics Laboratory to undertake a project that aims to use a combination of statistical modeling and data from sensors along the border to estimate the total number of unlawful border entries between land POEs, including entries both detected by Border Patrol and those not detected by Border Patrol. According to project documentation we reviewed, the project plans to leverage the CBP Tactical Simulation, an agent-based simulation of tactical border operations. CBP Tactical Simulation incorporates information on terrain at the border based on geographic information systems and sensors along with probability models that reflect how Border Patrol agents and unlawful border crossers behave in given circumstances. Border Patrol and OIS officials told us that this project would be more adaptable to changing border conditions and could help the agency address limitations associated with the Model-based Apprehension rate. Specifically, according to OIS officials, a simulation-based estimate would rely upon fewer assumptions about the types of individuals who unlawfully cross the border as compared to the current Model-based Apprehension rate. However, Border Patrol officials noted that estimates of unobservable phenomena, such as unobserved border entries, always face some limitations in their accuracy and that the new model may still rely upon samples of data that would have associated uncertainty as well as assumptions that would need to be validated. Ultimately, though, Border Patrol officials stated that the simulation-based model may be an improvement upon the current Model-based Apprehension rate. Border Patrol officials stated that the first iteration of the model would be presented to Border Patrol leadership for their review at the end of fiscal year 2019 and if at that time Border Patrol leadership approves the model, the earliest the simulation-based estimate could potentially be incorporated into the DHS Border Security Metrics Report would be for fiscal year 2020. Exploring alternative models is a positive step for DHS, however given that the project is in the early stages, it is too early to tell if it will be able to address the limitations we identified associated with the current model. In addition to the NDAA metrics, we have identified other metrics that DHS could use to help measure the effectiveness of border security. In particular, based on the findings from our previous reviews of border security programs and efforts, we have recommended that DHS use metrics that are relevant to each of the four domains listed in the NDAA— between POEs, at POEs, the maritime border, and for air and marine security in the land domain. For example, Between POEs domain. In February 2017, we reported on the use of border fencing along the southwest border and found that CBP collects data that could help provide insight into how border fencing contributes to border security operations, including the location of illegal entries. For example, we found that CBP collects data it could potentially use to determine the extent to which border fencing diverts illegal entrants into more rural and remote environments, and border fencing’s impact, if any, on apprehension rates over time. However, CBP had not developed metrics that systematically use these data to assess the contributions of border fencing to its mission. To better position CBP to make resource allocation decisions with the best information available to inform competing mission priorities and investments, we recommended that the Chief of the Border Patrol develop metrics to assess the contributions of pedestrian and vehicle fencing to border security along the southwest border using CBP data. DHS agreed with the recommendation and stated that it planned to develop metrics for use in its operational control framework for southwest border security operations. As of October 2018, DHS stated that the department planned to test the metrics and implement them in the framework by September 2019. At POEs domain. In July 2017, we reported on the Importer Security Filing (ISF) program and found that while ISF rule data have improved the program’s ability to identify high-risk cargo shipments, CBP could collect additional performance information to better evaluate program effectiveness. While evaluating the direct impact of using ISF rule data to assess shipment risk is difficult, we identified examples of how CBP could better assess the ISF program’s effectiveness. For example, CBP could track the number of containers not listed on a manifest—which could pose a security risk—it identifies through reviewing vessel stow plans. Collecting this type of additional information would help CBP better assess whether the ISF program is improving its ability to identify high-risk shipments. Therefore, we recommended that CBP identify and collect additional performance information on the impact of the ISF rule data, such as the identification of shipments containing contraband, to better evaluate the effectiveness of the ISF program. DHS agreed with the recommendation and reported that it is working to assess additional performance metrics to evaluate the effectiveness of the ISF program and anticipates completing the assessment by end of December 2019. Maritime border domain. In October 2017, we reported on the Coast Guard’s performance goals and found that although the Coast Guard’s performance goals are generally aligned with its statutory missions, the Coast Guard does not explain why certain aspects of mission performance are measured while others are not. For example, we found that while the Coast Guard’s mission is to interdict all illegal drugs, the agency’s two performance goals related to that mission were for cocaine interdiction only, excluding many other substances. We recommended that the Coast Guard either develop new performance goals to address mission activity gaps, or explain in the Coast Guard’s Annual Performance Report why certain aspects of mission performance are measured while others are not. Developing new goals to address missions, or describing how existing goals sufficiently assess mission performance, could better convey the Coast Guard’s progress in achieving its missions. DHS agreed with the recommendation and in February 2018, the Coast Guard provided us with its updated fiscal year 2017 Annual Performance Report. We found that while the updated report explained why performance goals related to its drug interdiction mission focus solely on cocaine interdiction, for the four other performance goals we previously identified as not fully addressing all related mission activities, the updated report did not include additional goals or explain why certain aspects of mission performance are not measured. We continue to believe that in instances in which performance goals do not fully address all of the respective mission activities, the Coast Guard’s Annual Performance Report should include an explanation. Air and marine security in the land domain. In May 2017, we reviewed DHS’s efforts to address subterranean, aerial, and maritime smuggling of drugs and humans. We found that while DHS established high-level performance measures and collected data on smuggling by ultralight aircraft, it had not assessed its efforts specific to addressing this smuggling method. Additionally, we found that DHS had similarly not assessed smuggling methods such as tunnels, panga boats (a fishing vessel), and recreational vessels. We recommended that DHS direct CBP, ICE, and Coast Guard to establish and monitor performance measures and targets related to ultralight aircraft, cross-border tunnels, panga boats, and recreational vessel smuggling to help provide reasonable assurance that efforts to address these smuggling methods are effective. By establishing measures and monitoring performance against targets, managers could obtain valuable information on successful approaches and areas that could be improved to help ensure that technology investments and operational responses to address these smuggling methods are effective. DHS agreed with the recommendations for measures related to ultralight aircraft and cross-border tunnels. DHS reported that AMO and Border Patrol have drafted a performance measure for ultralight aircraft, however, reviews and approval of the measure will not be completed until November 2019. As of June 2018, DHS reported that ICE was leading the development of measures related to cross-border tunnels. DHS did not agree with the recommendation to establish measures and monitor performance against targets for smuggling by panga boats and recreational vessels because the department believed measures and targets would not provide the most useful strategic assessment of operations to prevent all illicit trafficking, regardless of area of operations or mode of transportation. We continue to believe that the recommendation is valid and recognize the value of high-level strategic performance measures. However, such high-level measures may not provide sufficiently detailed performance information to allow DHS to identify successful approaches to addressing smuggling by panga boats and recreational vessels and areas for improvement. Further, establishing performance measures and targets related to smuggling by panga boats and recreational vessels could, in turn, better position DHS to understand the overall smuggling threat. Appendix II provides additional information on these and other metrics we have previously recommended that DHS could use to help measure the effectiveness of border security in the four domains. Securing U.S. borders is a complex undertaking that spans multiple domains and locations. It is also a key part of DHS’s mission for which DHS has made significant investments over the years. Given the complexity and breadth of border security efforts, having data and information available on the state of border security is important for DHS as well as policymakers and the public to understand the effectiveness of those investments. DHS’s fiscal year 2017 Border Security Metrics Report makes an important contribution in providing such data and information. DHS components generally have processes to help ensure the reliability of the data used in the metrics report and DHS identified and disclosed some data and methodological limitations with the metrics. However, DHS did not systematically review the reliability of data used in all metrics to identify and disclose limitations and their potential implications for the metric. Without complete information about the limitations of the data or the metric methodologies used in the report, Congress, policymakers, and the public may not be aware of important context or information needed to fully and appropriately understand the data being presented. By developing and implementing a process to systematically review the reliability of the data, as well as comprehensively identify limitations and communicate limitations of the metrics, DHS would improve the quality of the data and information provided in the report which would facilitate a better understanding and appropriate interpretation of the data and information provided. To develop three metrics in the report, DHS used a statistical model that incorporated untested assumptions about the behavior of unlawful border crossers that may not reflect real-world conditions. DHS was transparent about the limitations of its model, but providing the results of sensitivity analyses and measures of statistical uncertainty related to the model would allow Congress, policymakers, and the public to better understand its potential limitations and more fully evaluate the validity of DHS’s metrics that use estimates. We are making the following four recommendations to DHS: The Secretary of Homeland Security should develop and implement a process to systematically review the reliability of the data used in its Border Security Metrics Report and comprehensively identify any limitations with the data and methodologies that underlie its metrics. (Recommendation 1) The Secretary of Homeland Security should ensure the communication of the limitations of the metrics identified through the systematic review in the department’s annual Border Security Metrics Report. (Recommendation 2) The Under Secretary for the Office of Strategy, Policy, and Plans should include the results of sensitivity analyses to key assumptions in its statistical models of unlawful entry estimates in its annual Border Security Metrics Report. (Recommendation 3) The Under Secretary for the Office of Strategy, Policy, and Plans should include measures of statistical uncertainty for all metrics based on estimates derived from statistical models in its annual Border Security Metrics Report. (Recommendation 4) We provided a draft of this report to DHS and the Office of National Drug Control Policy for review and comment. DHS provided written comments, which are reproduced in appendix III and discussed below. DHS also provided technical comments, which we incorporated as appropriate. The Office of National Drug Control Policy indicated via e-mail that it did not have any comments on the draft report. In its comments, DHS concurred with our recommendations and stated that it planned to implement 3 of the 4 by October 2020. With respect to our second recommendation, DHS requested that we consider it closed as implemented because the department already detailed some of the limitations in its fiscal year 2017 report, and plans to continue to identify known limitations and the progress made to mitigate previously identified limitations in future reports. As discussed in this report, we agree that DHS identified and disclosed limitations for some metrics in its fiscal year 2017 Border Security Metrics Report; however, we identified at least one additional limitation for 21 of the 35 metrics on which DHS reported that DHS did not disclose or about which it could have been more transparent. To address the intent of this recommendation, once DHS has implemented a process to systematically review the reliability of the data used in its report and comprehensively identified related limitations, it should disclose those limitations in its annual Border Security Metrics Report. We are sending copies of this report to the appropriate congressional committees and the Secretary of the Department of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. Contacts points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or gamblerr@gao.gov. GAO staff who made key contributions to this report are listed in appendix IV. This appendix provides additional information on our analysis of the suitability and validity of the metrics the Department of Homeland Security (DHS) reported in its fiscal year 2017 Border Security Metrics Report for each of the four domains listed in the National Defense Authorization Act for Fiscal Year 2017 (NDAA)—between ports of entry, at ports of entry, the maritime border, and air and marine metrics in the land domain. Specifically, this appendix provides information on the metrics including their status, descriptions, differences between what DHS reported for the metrics and how they were described or defined by the NDAA, limitations, and any additional information or planned actions by DHS, where applicable. Description This metric is a rate comparing apprehensions to the total number of attempted unlawful border crossers. As such, this metric requires an estimate of the number of unlawful entry attempts that are not detected, which is added to the number of detected unlawful border crossers to create the denominator. The Department of Homeland Security (DHS) provided two methods for calculating this rate in its report. The first method, called the Model-based Apprehension Rate, uses a statistical model based on the capture-recapture methodology to estimate the rate.The second method, called the Observational Apprehension Rate, calculates the ratio of apprehensions to the sum of apprehensions and got aways. unlawful entries is limited to the southwest border. According to the report, research is underway on methods to produce estimates for the northern border. Limitations The Observational Apprehension Rate incorporates data on apprehensions, and got aways, while the Model- based Apprehension Rate is based on an estimate for undetected unlawful entries. Consequently, the limitations for those metrics also apply here. For more information on the limitations for those metrics, see the respective sections below. DHS identified: The observational apprehension rate excludes unobserved got aways. Additional information and planned actions by the Department of Homeland Security In its report, DHS noted that it has taken steps to improve situational awareness along the border and mitigate limitations. These steps include investing in technology, refining observational estimates, and developing a methodology to estimate statistical reliability. According to U.S. Border Patrol officials, investments in new technology have enabled U.S. Border Patrol to better detect cross-border activities. For additional information on the data elements used for this metric and DHS’s planned actions, see the respective sections below on apprehensions, got aways, and the estimate for undetected unlawful entries. Description This metric is a count of the total number of attempted unlawful border crossers between land ports of entry who were directly or indirectly observed or detected by U.S. Border Patrol. The Department of Homeland Security (DHS) calculated this metric by adding turn backs, got aways, and apprehensions of unlawful border crossers. Patrol officials, the northern border has different immigration dynamics than the southern border and accounts for a significantly smaller number of turn backs and got aways overall, so northern border data were not included. Limitations Because this metric incorporates data on apprehensions, got aways, and turn backs, the limitations for those metrics also apply here. For more information on the limitations for those metrics, see the respective sections below. Additional information and planned actions by the Department of Homeland Security For additional information on the data elements used for this metric and DHS’s planned actions, see the respective sections below. Description This metric is an estimate of the number of attempted unlawful border crossers that are not directly or indirectly observed or detected by U.S. Border Patrol (Border Patrol). The Department of Homeland Security (DHS) used a statistical model, based on capture-recapture methodology, to estimate total successful unlawful entries, and subtracted detected got aways to calculate the total number of undetected unlawful entries. unlawful entries is limited to the southwest border. According to DHS’s report, research is under way to produce this estimate for the northern border. DHS does not currently have reliable data on the estimated share of migrants who, following an unsuccessful unlawful entry attempt, are deterred from making a subsequent reentry attempt. For its model, DHS used data from a survey of recently removed Mexicans, which asked them about their intentions to re-enter the United States. According to DHS’s report, a shortcoming of the survey is that it does not take account of shifting border enforcement efforts, potential changes in behavior by individuals who have been exposed to consequence programs, or other deterrent factors along the border. Consequently, any resulting undercount in the estimate of the deterred population results in a downward bias. The population that conforms best to the model’s assumptions represents a diminishing share of southwest border apprehensions. Specifically, in its report DHS said that Mexican adults removed to the nearest border accounted for about 95 percent of apprehensions in the 1990s. However, because of recent changes at the border, including changes in the composition of border flows (i.e., rising numbers of Central Americans and asylum seekers) and in Border Patrol’s enforcement strategy, the population best reflected in the model has declined to as few as 20 percent of apprehensions in recent years. Further, DHS noted that some alien populations, such as those seeking asylum and who do not evade detection by Border Patrol agents, are also excluded from the model. However, these populations make up an increasing share of apprehensions in recent years. The model uses restrictive assumptions about which re-apprehensions to include. For example, the model excludes apprehensions occurring at check points and other remote locations and those occurring more than 4 days after an illegal entry. According to DHS, these assumptions result in a downward bias. We identified: DHS described assumptions it made in its report but did not provide quantitative information on the extent to which they affected its estimated undetected unlawful entries through the use of sensitivity analyses. Sensitivity analyses help to convey the extent to which changing the values of variables, assumptions, data, or other input affects statistical estimates. By including the results of sensitivity analyses in its Border Security Metrics Report, DHS would allow Congress and the public to better understand the potential limitations associated with its model and make independent assessments on its accuracy. DHS did not provide information on the statistical level of uncertainty related to this rate, such as margins of error or confidence intervals. This information would help convey how the estimates might vary due to randomness in the data. Instead, DHS provided a single rate that does not fully convey the difficulty and uncertainty of the estimate. This metric incorporated data on apprehensions and got aways. For more information on the limitations associated with those metrics, see the respective sections below. Additional information and planned actions by the Department of Homeland Security According to DHS, officials are continuing to improve the accuracy of the existing statistical model for estimating unlawful border crossers but are also considering alternative methodologies. U.S. Customs and Border Protection has contracted with Johns Hopkins Applied Physics Laboratory to develop a new model for estimating the flow of unlawful border crossers. This model uses a combination of statistical modelling, data from sensors along the border, and probability models that reflect how Border Patrol agents and unlawful border crossers behave in given circumstances. Border Patrol officials estimated that the earliest the simulation-based estimate could potentially be incorporated into the DHS Border Security Metrics Report would be for fiscal year 2020. Description This metric is a count of the number of unlawful border crossers who, after making an unlawful entry into the United States, responded to law enforcement efforts by returning promptly to the country from which they entered. These data came from U.S. Border Patrol (Border Patrol) records. Border Patrol officials, the northern border has different immigration dynamics than the southern border and accounts for a significantly smaller number of turn backs overall, so northern border data were not included. Officials stated that while the current emphasis of reporting is on the southwest border, efforts are underway to identify and find ways to capture data that are important and reflective of the effectiveness in addressing threats specific to the northern border. The estimate aggregates potentially subjective observations from thousands of individual agents. Some unlawful border crossers may enter the United States to drop off drug loads or to act as decoys to lure agents away from a certain area and then return to Mexico, and therefore may be misidentified as turn backs. In our previous work we identified differences in the procedures for reporting and classifying turn backs across sectors, and noted how factors such as terrain and weather may impact agents’ abilities to accurately detect turn backs. According to DHS, since 2014, Border Patrol has implemented a standard, southwest border-wide methodology to improve reporting and mitigate the potential subjectivity of observations by agents. Therefore, data before 2014 are not necessarily comparable to data from 2014 and later. DHS presented the data in a table without explaining that the methodology used to categorize and count turns backs changed in 2014. Additional information and planned actions by the Department of Homeland Security According to DHS’s report, Border Patrol has taken steps to implement a standard, southwest border-wide methodology to improve reporting of potential turn backs. In addition, DHS’s report said that command staff ensure all agents are aware of and utilize proper definitions for apprehensions, got aways, and turn backs at their respective stations. They also ensure necessary communication takes place between and among sectors and stations to minimize double-counting when subjects cross through more than one station. DHS’s report noted that Border Patrol headquarters components validate data integrity. Description This metric is a count of the number of unlawful border crossers who are directly or indirectly observed entering unlawfully, are not apprehended, and are not turn backs. These data came from U.S. Border Patrol (Border Patrol) records. Border Patrol officials, the northern border has different immigration dynamics than the southern border, so northern border data were not included. Officials stated that while the current emphasis of reporting is on the southwest border, efforts are under way to identify and find ways to capture data that are important and reflective of the effectiveness in addressing threats specific to the northern border. Limitations DHS identified: The count aggregates potentially subjective observations from thousands of individual agents. In previous work we identified differences in procedures for reporting and classifying got aways across sectors, and noted how factors such as terrain and weather may impact agents’ abilities to accurately detect got aways. According to DHS, since 2014, Border Patrol has implemented a standard, southwest border-wide methodology to improve reporting and mitigate the potential subjectivity of observations by agents. Therefore, data before 2014 are not necessarily comparable to data from 2014 and later. DHS presented the data in a table without explaining that the methodology used to categorize and count turns backs changed in 2014. For information on limitations with the model-based estimate for undetected unlawful entries, see the section for estimated undetected unlawful entries above. Additional information and planned actions by the Department of Homeland Security According to DHS’s report, Border Patrol has taken steps to implement a standard, southwest border-wide methodology to improve reporting of potential got aways. In addition, DHS’s report said that command staff ensure all agents are aware of and utilize proper definitions for apprehensions, got aways, and turn backs at their respective stations. They also ensure necessary communication takes place between and among sectors and stations to minimize double-counting when subjects cross through more than one station. DHS’s report noted that Border Patrol headquarters components validate data integrity. As a comparison against the counts of documented got aways, DHS also provided an estimate of total successful unlawful entries along the southwest border using a statistical model based on capture-recapture methodology. For more information on the methodology for this estimate, see the section titled “Estimated Undetected Unlawful Entries” in this appendix. Description This metric is a rate comparing the number of apprehensions and turn backs to the number of apprehensions, estimated undetected unlawful entries, turn backs, and got aways in each U.S. Border Patrol sector. rate is not available because sector-level estimates of unlawful entries and attempts have not yet been produced and validated. As an alternative, DHS presented data using the interdiction effectiveness rate. With this rate, the estimated undetected unlawful entries measure is replaced with known got aways. However, DHS does not have an interdiction effectiveness rate for the northern border so it solely provided data for the southwest border. According to DHS’s report, the department has not yet developed a northern border interdiction effectiveness rate because there are only a small number of attempted and successful entries along the northern border. Limitations None identified. Additional information and planned actions by the Department of Homeland Security DHS reported that sector-level estimates of unlawful entries and attempts are projected to be available in its 2019 annual Border Security Metrics Report to Congress. Authorization Act for Fiscal Year 2017 (NDAA) defined this metric as a rate comparing the estimated total undetected unlawful border crossing attempts to the unlawful border crossing effectiveness rate. The Department of Homeland Security (DHS) calculated this metric by dividing the detected unlawful entries by the estimated total unlawful entries. The number of detected unlawful entries is calculated by adding turn backs, got aways, and apprehensions. Estimated total unlawful entries is calculated by adding turn backs, apprehensions and estimated total successful unlawful entries derived from DHS’s statistical model. unlawful entries is limited to the southwest border. Additionally, DHS used detected unlawful entries as the numerator, instead of the estimated total unlawful border crossing attempts not detected as called for in the NDAA. For the denominator DHS used the estimated total unlawful entries instead of the unlawful border crossing effectiveness rate, as called for in the NDAA. Limitations Because this metric incorporates data on apprehensions, got aways, and turn backs, as well as the estimate for undetected unlawful entries, the limitations for those metrics also apply to this metric. For more information on the limitations for those metrics, see the respective sections for those metrics. Additional information and planned actions by the Department of Homeland Security For additional information on apprehensions, got aways, turn backs, and the estimate for undetected unlawful entries, and any planned actions by DHS for those metrics, see the respective sections for those metrics. Description This metric is a count of the number of apprehensions in each U.S. Border Patrol (Border Patrol) sector. Data come from Border Patrol records, and each apprehension of the same unlawful crosser in a fiscal year is counted separately, meaning these data do not represent a count of unique crossers apprehended. Border Patrol officials, the northern border has different immigration dynamics than the southern border, so northern border data were not included. Officials stated that while the current emphasis of reporting is on the southwest border, efforts are under way to identify and find ways to capture data that are important and reflective of the effectiveness in addressing threats specific to the northern border. Limitations DHS identified: In its report, DHS said that apprehensions are not a useful indicator of successful unlawful border crossings over the long-term and across multiple locations because the relationship between apprehensions and successful unlawful entries depends on the apprehension rate, which changes over time and may differ by location. Additional information None. Description This metric is a count of the number of apprehensions of unaccompanied alien children (UAC), and the nationality of such children, in each U.S. Border Patrol (Border Patrol) sector. A UAC is a child under 18 years old with no lawful immigration status, and no parent present and available in the United States to provide care and physical custody. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report The Department of Homeland Security (DHS) only included data for the southwest border. Limitations DHS identified: Agents may not be able to reliably distinguish among older children and young adults or confirm whether children are traveling alone or in family groups. We identified: We previously reported that it can be challenging to obtain accurate information about a child’s country of origin because of absence of documentation, language barriers, and coached responses by smugglers, among other reasons. Border Patrol officials said that the data on UAC may have reliability issues because original data from a shared database had been changed. Specifically, officials said that in January 2015 they noticed that Enforcement and Removal Operations staff were inadvertently overwriting Border Patrol’s original data entries about the status of migrant children apprehended once those children were placed with relatives or a foster family. Additional information and planned actions by the Department of Homeland Security According to Border Patrol officials, agents rely on statements provided by the child to determine the nationality of UACs when verifiable documentation is not available. Verifiable documentation could include biometric checks, birth certificates, state-issued identification cards, and passports. However, officials noted that this list is not all-inclusive and the processing agent determines the validity of any presented documents. Border Patrol officials said that a data integrity team regularly examines data on apprehensions and they conduct biweekly data reliability checks. Additionally, they are working with Enforcement and Removal Operations to modify the data entry process so that updates can be made without overwriting the original apprehension data entered by Border Patrol. Description This metric is a count of the number of apprehensions of family units, and the nationality of such family units, in each U.S. Border Patrol (Border Patrol) sector. A family unit is the number of individuals apprehended with a family member. For example, a mother and child apprehended together are counted as two family units. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report The Department of Homeland Security (DHS) only included data for the southwest border. Limitations DHS identified: DHS noted that the count of apprehensions for family units is considered reliable, but that agents may not be able to reliably identify family units. We identified: According to Border Patrol officials, their data entry system did not have a dedicated field for agents to record apprehensions of persons within a family unit for all of the years presented in the report. In December 2014, Border Patrol added specific data entry fields to its data entry processes for agents to input information about family units. These fields incorporated built in safeguards and edit checks to help ensure that agents make an appropriate family unit classification. Previously, Border Patrol officials said they used proxy data to identify family units. Given the additional safeguards and checks included with the new family unit data entry fields, Border Patrol officials stated that the data after December 2014 may be more reliable overall compared to previous years. Border Patrol officials stated that they have high confidence in the proxy count for data pre-2014, but acknowledged that those data may contain misclassifications of family units. Additional information According to Border Patrol officials, agents are trained in interviewing techniques and the processing agent will consider all available evidence to determine the validity of claims to familial relationships. Border Patrol officials also noted that in order to be categorized as a family unit, at least one member of the family unit must be at least 18 years of age. Consequently, related individuals younger than 18 years of age that are apprehended together would not be categorized as a family unit. Description This metric is a rate comparing the amount and type of illicit drugs seized between ports of entry in any fiscal year to the average of the amount and type of illicit drugs seized between ports of entry in the immediately preceding 5 fiscal years. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations None identified. Additional information None. Description This metric was not specifically defined in the National Defense Authorization Act for Fiscal Year 2017 (NDAA); the NDAA called for an estimate of the impact of the Consequence Delivery System (CDS) on the recidivism rate of unlawful border crossers over multiple fiscal years. The Office of Immigration Statistics (OIS) calculated this metric by providing the average annual recidivism rate for the 3 years prior to fiscal year 2012— when the CDS was implemented—and the average annual recidivism rate for the subsequent 3 years. The annual recidivist rate is calculated by dividing the number of unique crossers apprehended multiple times in a fiscal year by the total number of unique crossers in the fiscal year. to DHS’s report, recidivism data for the northern border were not available due to the small number of attempted illegal entries along the northern border. Noting the findings from our January 2017 review, DHS stated that its current recidivism measure could be strengthened by using the date an unlawful border crosser is removed or returned instead of the date they are apprehended, as well as by counting re-apprehensions within a fixed period of time defined by the crosser’s repatriation date instead of by the fiscal year. In January 2017, we reported that using a crosser’s apprehension history beyond 1 fiscal year, and excluding crossers that have not been previously removed, among other things, produces a significantly different rate compared to how DHS currently calculates it. Consequently, we recommended that DHS calculate recidivism for a period of time longer than 1 fiscal year and that DHS exclude from the recidivism calculation aliens for whom there is no record of removal and who may remain in the United States. As of December 2018, this recommendation remained open. DHS stated that changes in the recidivism rate after 2012 cannot be attributed solely to CDS because enforcement is a complex, dynamic system. We identified: Given that DHS’s methodology is to provide the 3-year average of the recidivism rate before and after CDS was implemented in fiscal year 2012, the data presented will remain static for subsequent annual reports because the periods of comparison for analyzing recidivism are fixed around a specific point in time. According to OIS officials, to help address this issue, in the next report they plan to provide individual rates for each year instead of the 3-year average. Additional information and planned actions by the Department of Homeland Security In its report, DHS noted that future reports will include estimates of the impact of CDS on both the annual recidivism rate and a longer-term recidivism rate. For example, OIS officials said they plan to update the way they calculate recidivism for future issues of the report and are developing a multivariate impact analysis that would take into consideration factors such as crossers’ demographics and immigration history. Description This metric was not specifically defined in the National Defense Authorization Act for Fiscal Year 2017 (NDAA); the NDAA called for an examination of each consequence under the Consequence Delivery System (CDS), including (1) voluntary return, (2) warrant of arrest or notice to appear, (3) expedited removal, (4) reinstatement of removal, (5) alien transfer exit program, (6) criminal consequence program, (7) standard prosecution, and (8) Operation Against Smugglers Initiative on Safety and Security. The Department of Homeland Security (DHS) presented data on the recidivism rates for each consequence between fiscal years 2012 through 2016. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report As noted above for the metric “estimates of the impact of the Consequences Delivery System on recidivism,” DHS only included data for the southwest border because recidivism data for the northern border were not available due to the small number of attempted illegal entries along the northern border. Differences in recidivism rates among the consequences may also reflect differences in the propensity of the targeted populations to attempt to re-enter. As with the metric for estimating the impact of the CDS on recidivism discussed above, DHS noted the limitation that current recidivism data are based on apprehensions within a given fiscal year, and not the date when an individual was repatriated to their country of origin. In January 2017, we reported that some unlawful border crossers were incorrectly classified based on CDS guidance. U.S. Border Patrol (Border Patrol) agents implement CDS by classifying apprehended aliens into one of seven noncriminal or criminal categories and then applying one or more of eight different consequences; therefore, determining the correct classification of the unlawful border crosser is important for identifying and applying the appropriate consequence. Our analysis of Border Patrol apprehension data from fiscal year 2013 through 2015 showed that Border Patrol did not classify 11 percent of apprehensions in accordance with the agency’s guidance. We recommended that Border Patrol provide consistent guidance for classification and take steps to ensure the integrity of classification data. Border Patrol implemented this recommendation as of December 2017, but the issue could potentially have implications for the data DHS used in this metric, which was for fiscal years 2012 through 2016. Additional information and planned actions by the Department of Homeland Security According to its report, DHS is refining its analysis and will seek to specifically address the limitations discussed above in the fiscal year 2018 version of the Border Security Metrics Report. Ports of entry are U.S. government facilities that provide for the controlled entry into or departure from the United States. There are 328 ports of entry in the United States. Specifically, a port of entry is any officially designated location (seaport, airport, or land border location) where U.S. Customs and Border Protection (CBP) officers or employees are assigned to clear passengers, merchandise and other items, collect duties, and enforce customs laws; and where CBP officers inspect persons seeking to enter or depart, or apply for admission into, the United States pursuant to U.S. immigration law and travel controls. CBP’s Office of Field Operations (OFO) is the lead DHS component responsible for carrying out activities at POEs. The 15 metrics in this domain measure the number of travelers attempting to enter the United States at ports of entry, illicit drugs seized at ports of entry, and cargo entering the United States, among other things. DHS included 11 of the 15 metrics called for in the NDAA for this domain in its fiscal year 2017 Border Security Metrics Report, as shown in table 5. DHS reported that the four metrics for which it did not provide information did not yet have a reliable methodology or were under review, and that DHS was in the process of developing methodologies to capture the data needed for the requested metrics. DHS officials said these four metrics would not be ready for inclusion in the next annual report. Description This metric is a count of total inadmissible travelers, and requires an estimate of the number of inadmissible travelers who successfully enter at a port of entry without being detected. The metric is the sum of the number of inadmissible travelers interdicted and the estimated number of inadmissible travelers who successfully enter at a port of entry without being detected. inadmissible travelers who successfully enter at a port of entry without being detected. Therefore, DHS only presented data on known inadmissible travelers. Limitations None identified. Additional information and planned actions by the Department of Homeland Security DHS projected that the department may be able to include estimates on the number of inadmissible travelers who successfully enter at a port of entry in its fiscal year 2019 Border Security Metrics Report to Congress. According to U.S. Customs and Border Protection (CBP) officials, they are in the process of determining whether CBP’s Compliance Measurement Examination (COMPEX) program could be used as a means to reliably measure undetected inadmissible travelers. Description These metrics are rates that require data on travelers seeking admission at a port of entry, interdictions of inadmissible travelers, and an estimate of the number of inadmissible travelers who successfully enter at a port of entry without being detected. The refusal rate is calculated by dividing the number of inadmissible travelers interdicted by all people seeking admission at a port of entry. The interdiction rate is calculated by dividing the number of inadmissible travelers interdicted by the total number of inadmissible travelers who attempt to enter at a port of entry. inadmissible travelers who successfully enter at a port of entry without being detected. Therefore, DHS only presented data on the refusal rate. Limitations None identified. Additional information and planned actions by the Department of Homeland Security DHS projected that the department may be able to include estimates on the number of inadmissible travelers who successfully enter at a port of entry in its next Border Security Metrics Report to Congress. According to U.S. Customs and Border Protection (CBP) officials, they are in the process of reviewing data and program policies for CBP’s Compliance Measurement Examination program to determine if the program could be used as a means to reliably measure undetected inadmissible travelers, which would then be used in calculating the interdiction rate. Description This metric is a count of the amount in kilograms of illicit drugs seized by U.S. Customs and Border Protection officers at ports of entry. In an appendix to the report, the Department of Homeland Security listed out 34 different types of illicit drugs and the amounts seized for each for fiscal years 2007 through 2016. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations None identified. Additional information None. Description This metric is a rate that compares the amount of illicit drugs seized (in kilograms) by Office of Field Operations officials at ports of entry in 1 fiscal year to the average amount seized in the immediately preceding 5 fiscal years. the metric. The Department of Homeland Security provided rates for cocaine, methamphetamine, marijuana, and heroin for fiscal years 2012 through 2016. Limitations None identified. Additional information None. Description This metric is a count of the number of infractions related to travelers and cargo committed by major violators, and an estimate of the number of major infractions not interdicted. The Department of Homeland Security (DHS) calculated an infraction rate by dividing the number of major infractions by the total number of passengers at ports of entry for fiscal years 2007 through 2016. National Defense Authorization Act for Fiscal Year 2017 (NDAA). As an alternative, for the purpose of its report, DHS defined a major infraction as an arrest, including arrests related to terrorism, drugs, criminal aliens, and currency, among other things. DHS reported that it does not have a methodology in place to estimate the number of undetected major infractions. Therefore, only data on known infractions are included. DHS only included data for passenger infractions and not cargo-related infractions. Although not requested by the NDAA, DHS provided an infraction rate by dividing the number of known infractions by the total number of travelers at ports of entry. Limitations We identified: Given that DHS’s alternative approach to this metric involves using arrests as a proxy for major infractions, it is unclear whether there is a one-to-one correspondence between the arrest of a major violator and the number of infractions committed. Additional information According to U.S. Customs and Border Protection (CBP) officials, they plan to use data from CBP’s Compliance Measurement Examination program as a means to report estimated undetected major infractions starting with DHS’s fiscal year 2019 report. Description This metric is a rate that compares the amount of cocaine seized at land ports of entry to the total estimated flow of cocaine. the total flow of cocaine through land ports of entry. The Office of National Drug Control Policy produces annual estimates for total cocaine flow into the United States, but does not have a methodology to estimate the flow of cocaine through land ports of entry alone. Therefore, the estimates the Department of Homeland Security used included cocaine flow through all domains. According to the U.S. Drug Enforcement Administration’s National Drug Threat Assessment, the southwest border remains the key entry point for the majority of the cocaine entering the United States. Limitations None identified. Additional information None. Description This metric is a rate that compares the average wait time for vehicles to pass through a land port of entry to the total number of commercial and private vehicles at each land port of entry. data were not available for every port of entry, such as small ones with negligible wait times. Limitations We identified: We reported in July 2013 that commercial vehicle wait time data were unreliable due to inconsistent data collection processes at ports, and made two recommendations to DHS to improve the reliability of the data. While DHS implemented these recommendations in 2018, older data, including the data for the years presented in the report (fiscal years 2012 through 2016), remain unreliable. Additional information and planned actions by the Department of Homeland Security U.S. Customs and Border Protection (CBP) officials clarified that the wait times shown in the report reflect the average of all hourly recordings for wait times at ports of entry rather than the average passenger or vehicle experience because CBP did not report a volume-weighted measure of wait times. According to the report, CBP’s wait time policy is currently under review and new guidance will be issued in the future to account for improvements in automation and recording. Description This metric is a rate that measures traffic volume at land ports of entry against the physical and staffing capacity at each land port of entry. The Department of Homeland Security (DHS) calculated the average number of vehicles processed per booth, per hour at each land port of entry. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None. Limitations None identified. Additional information In addition to reporting utilization at each port of entry, DHS provided the average utilization rate for all northern border land ports of entry and all southern border land ports of entry. This metric is a rate that measures the frequency of secondary examinations at each land port of entry. The Department of Homeland Security (DHS) calculated the rate by dividing the recorded number of passengers sent for secondary inspection by the total number of recorded passengers at each land port of entry. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report DHS did not include data on secondary examinations of cargo or shipments. Limitations None identified. Additional information None. Description This metric is a count of the number of cargo containers at sea ports that DHS identified as potentially high-risk using National Targeting Center (NTC) security criteria. According to the Department of Homeland Security (DHS), all international cargo shipments coming to the United States are screened to identify potentially high-risk containers, which may then be reviewed, scanned, or physically inspected prior to lading at a port of entry. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations We identified: U.S. Customs and Border Protection (CBP) officials said that the process of defining and identifying “high-risk” shipments can exclude some shipments, such as those in free trade zones. Additional information DHS’s report said that the NTC periodically refines, improves, and revises the security criteria for high-risk shipments, which can affect the number of cargo shipments identified as high-risk. Description This metric is a rate comparing the number of potentially high-risk cargo containers scanned at each sea port of entry during a fiscal year to the total number of high-risk cargo containers that entered the United States at each sea port of entry during the previous fiscal year. separate from cargo containers that were reviewed or assessed; instead, DHS tracks these inspection methods collectively. Therefore, DHS also included data on potentially high-risk cargo containers that were reviewed or assessed as well as those that were scanned in its report. Limitations DHS identified: In its report, DHS noted that ratio data are not available for fiscal year 2014 because U.S. Customs and Border Protection did not collect comparable container-level data (as opposed to shipment-level data) in fiscal year 2013. DHS also noted that the totals across the ports or field offices may include duplicate container counts. We identified: NTC officials said that the definition of “high-risk” shipments excludes some shipments, such as those in free trade zones. NTC officials noted that assessing, reviewing, and scanning containers are different activities and reflect different levels of inspection or review. For example, NTC officials said that while all containers are “assessed” in order to determine their risk level, only higher risk containers may be scanned using radiation detection and nonintrusive inspection equipment. Consequently, when DHS included data on containers that were assessed or reviewed but not scanned, the resulting count was higher. In an appendix to its report, DHS presented a column of data called the “percentage of potentially high-risk containers scanned (same fiscal year)” for each fiscal year. Given DHS’s inability to separate data on the different inspection methods, the data in this column included containers that were reviewed by all inspection methods, not just scanning. In its appendix, DHS did not present data on the number of containers that “entered the United States,” even though it used those data to calculate the ratio and they are specified in the National Defense Authorization Act for Fiscal Year 2017. As a result, it is not possible to verify the accuracy of DHS’s ratio calculations. Additional information None. The U.S. maritime border domain encompasses ports, internal or inland waters, and coastal waters, as well as the territorial sea (waters 12 nautical miles seaward of the U.S. coast), contiguous zone (waters adjacent to and seaward of territorial sea and extending 24 nautical miles from shore), and exclusive economic zone (waters seaward of and adjacent to territorial sea and extending out to 200 nautical miles from shore). U.S. Coast Guard (Coast Guard), Air and Marine Operations, and U.S. Border Patrol share responsibility for patrolling the U.S. maritime borders, and territorial sea. The Coast Guard is a component of DHS and the lead federal maritime law enforcement agency on the high seas (waters beyond 12 nautical miles seaward of the U.S. coast) and all other waters under U.S. jurisdiction. The Coast Guard responds to a variety of maritime border security issues, including trafficking of narcotics, people, illicit goods, unlawful migration, illegal exploitation of natural resources, potential terrorist activities, and the disruption of maritime commerce. The metrics in this domain measure the number of migrants and illicit drugs removed, among other things. DHS included 4 of 6 metrics called for in the NDAA for this domain in its fiscal year 2017 Border Security Metrics Report, as shown in table 6. Description This metric was not specifically defined in the National Defense Authorization Act for Fiscal Year 2017 (NDAA). The NDAA described situational awareness as the knowledge and understanding of current unlawful cross- border activity, including (1) threats and trends concerning illicit trafficking and unlawful crossings, (2) the ability to forecast future shifts in such threats and trends, (3) the ability to evaluate such threats and trends at a level sufficient to create actionable plans, and (4) the operational capability to conduct persistent and integrated surveillance of the international borders of the United States. developing a measure for situational awareness in the maritime domain that meets the intent of the NDAA. While this effort is in process, DHS presented data on U.S. Coast Guard and U.S. Customs and Border Protection (CBP) asset (aircraft and cutter or boat) hours contributing to situational awareness or interdiction support and the number of vessel manifests screened. Limitations None identified. Additional information According to CBP Air and Marine Operations officials, they did not have confidence that the data for years prior to fiscal year 2016 were consistent enough for making comparisons across years. Consequently, only data for fiscal year 2016 were included in DHS’s report for the metrics related to CBP. Description This metric is a count of the total number of undocumented migrants interdicted, identified directly or indirectly but not interdicted, or otherwise believed to have unlawfully entered the United States through the maritime border. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: The accuracy of migrant flow counts depends on partners to report interdictions and the ability to detect migrants. According to the DHS report, the U.S. Coast Guard relies on partners to report interdictions for compilation in the U.S. Coast Guard’s database. Interdictions may be double-counted by the U.S. Coast Guard and its partners because they cooperate during operations and some interdictions by partners may not get reported. Further, some migrants may not be apprehended and leave no evidence, and are therefore excluded from the known flow figures. We identified: According to U.S. Coast Guard officials, there is no centralized database for tracking migrant interdictions, and the decentralized nature of the data collection could lead to errors. Additional information According to the U.S. Coast Guard, about 90 percent of the data on migrant interdictions and flow originate from U.S. Coast Guard records. U.S. Coast Guard officials said that as part of a department-wide initiative to standardize illegal immigration statistics, they are in the preliminary stages of building a centralized database to enter and maintain information on migrant interdictions. Additionally, officials said they take steps to ensure the reliability of externally reported data such as communicating with partners and working together to reconcile any errors. Within the U.S. Coast Guard, meetings are held regularly to discuss and vet the accuracy of migrant flow data. Description This metric is a rate comparing the amount and type of illicit drugs removed by the Department of Homeland Security (DHS) maritime security components in any fiscal year, including drugs abandoned at sea, to the average amount removed or abandoned in the immediately preceding 5 fiscal years. by all DHS maritime security components, but DHS only provided data on removals by the U.S. Coast Guard. DHS did not explain in its report why it only included data from the U.S. Coast Guard. DHS officials said that the U.S. Coast Guard is the primary DHS component involved in this activity and was the only component that provided data for this metric, but this was not noted in the report. According to U.S. Coast Guard officials, some of the data for fiscal 2013 was misreported. Specifically, the quantity removed for methamphetamine should be 0 (report shows 17.4) while the value should be 7.9 kilograms for heroin (report shows 0). Additional information None. Description This metric is a rate comparing the amount of cocaine removed by the Department of Homeland Security (DHS) maritime security components inside and outside the maritime transit zone to the total documented cocaine flow rate. DHS used estimates of noncommercial maritime cocaine flow from the Consolidated Counter Drug Database, which are derived from intelligence reporting and case data. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations DHS identified: There is less robust intelligence on noncommercial maritime cocaine flow outside the transit zone than inside the transit zone, so data for outside the transit zone are not considered reliable. Precise cocaine flow estimates through a particular mode or domain can be difficult to obtain. In our prior work, officials with the Office of National Drug Control Policy and other departments and agencies involved in U.S. counternarcotics efforts told us that it is difficult to obtain precise estimates of cocaine flow because of the difficulty in obtaining specific information about the production of cocaine and how it gets to the United States. We have also previously reported that when confronted with threats to their activities, drug-trafficking organizations use a variety of techniques to quickly change their modes of operation, thus avoiding capture of their personnel and seizure of their illegal drugs. For example, when air interdiction efforts have proven successful, traffickers have increased their use of maritime and overland transportation routes. Additional information According to U.S. Coast Guard officials, DHS officials hold quarterly inter-agency meetings to review the reliability of performance data related to cocaine interdiction performance. Air and Marine Operations (AMO) is a federal law enforcement agency within CBP that interdicts unlawful people and cargo approaching U.S. borders, investigates criminal networks, and provides domain awareness in the air and maritime environments, among other things. The metrics in this domain measure AMO’s flight hours, individuals detected, and apprehensions, among other things. DHS included 7 of 8 metrics within this domain called for in the NDAA in its fiscal year 2017 Border Security Metrics Report, as shown in table 7. DHS reported that the “AMO actionable intelligence” metric was under review and estimated that the department would provide information on this metric in its 2019 annual report to Congress. Description This metric is a rate comparing the number of flight hour requirements to the number of flight hours flown by Air and Marine Operations (AMO) in the land domain. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. We identified: DHS used the terms “funded flight hours,” “unfunded flight hours,” and “unconstrained flight hours” in the report without clearly defining them. AMO officials stated that a definition of these terms will be included in the next report. Additional information AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a rate comparing the number of funded flight hours appropriated to Air and Marine Operations (AMO) to the number of actual flight hours flown. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. Additional information AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a rate comparing the number of aviation missions flown by Air and Marine Operations (AMO) to the number of aviation missions cancelled by AMO due to maintenance, operations, or other causes. the number of missions cancelled due to causes within AMO control, such as maintenance, personnel, and asset availability. However, the Department of Homeland Security (DHS) used the total number of mission requests, which also includes the number of missions flown in addition to the number of missions cancelled for reasons within AMO control. Limitations DHS identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. Additional information AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a rate comparing the number of missions cancelled by Air and Marine Operations (AMO) due to weather compared to the total planned missions. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. Additional information AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a count of the number of individuals detected by Air and Marine Operations (AMO) through the use of unmanned aerial systems and manned aircraft. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None identified. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. DHS data on detections from manned aircraft were limited to those that led to apprehensions and arrests, and data from unmanned aircraft were limited to the number of Vehicle and Dismount Exploitation Radar (VADER) detections. AMO did not track data from all sensors on unmanned and manned aircraft, and considers this metric to be a work in progress. We identified: In February 2017 we reported that some mission data (such as asset assists) for unmanned aerial systems were collected inconsistently across operation locations, which could affect the accuracy of the counts provided. We recommended that U.S. Customs and Border Protection—of which AMO is a component—update and maintain guidance for recording mission information in its data collection system, and provide training to users of the system. DHS completed implementation of these recommendations in July 2018. Although the recommendations have been implemented, this limitation is relevant because the data presented (for fiscal year 2016) were collected prior to their implementation. Additional information and planned actions by the Department of Homeland Security DHS expects to provide more comprehensive data for this metric in the next annual report. AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a count of the number of apprehensions assisted by Air and Marine Operations (AMO) through the use of unmanned aerial systems and manned aircraft. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. We identified: In February 2017 we reported that some mission data (such as asset assists) for unmanned aerial systems were collected inconsistently across operation locations, which could affect the accuracy of the counts provided. We recommended that U.S. Customs and Border Protection—of which AMO is a component—update and maintain guidance for recording mission information in its data collection system, and provide training to users of the system. DHS completed implementation of these recommendations in July 2018. Although the recommendations have been implemented, this limitation is relevant because the data presented (for fiscal year 2016) were collected prior to their implementation. Additional information In addition to the number of apprehensions assisted, DHS also provided the number of enforcement flight hours used for the assists. AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Description This metric is a count of the number and quantity of illicit drug seizures assisted by Air and Marine Operations (AMO) through the use of unmanned aerial systems and manned aircraft. Differences between the National Defense Authorization Act for Fiscal Year 2017 and the Department of Homeland Security’s report None. Limitations Department of Homeland Security (DHS) identified: Data prior to fiscal year 2016 were unavailable. According to AMO officials, this is because AMO did not collect these data prior to fiscal year 2016, or because older data were not comparable. We identified: In February 2017 we reported that some mission data (such as asset assists) for unmanned aerial systems were collected inconsistently across operation locations, which could affect the accuracy of the counts provided. We recommended that U.S. Customs and Border Protection—of which AMO is a component—update and maintain guidance for recording mission information in its data collection system, and providing training to users of the system. DHS completed implementation of these recommendations in July 2018. Although the recommendations have been implemented, this limitation is relevant because the data presented (for fiscal year 2016) were collected prior to their implementation. Additional information In addition to the drug seizures assisted (in pounds), DHS also provided the number of enforcement flight hours used for the assists. AMO officials said they have taken steps to improve how they track flight hour data, such as by adding new data fields to AMO’s system and providing training to staff. Based on findings from previous reviews of border security programs and efforts, we have recommended other metrics that the Department of Homeland Security (DHS) could use to help measure the effectiveness of border security. The tables that follow provide information about these recommended metrics in each of the four domains listed in the National Defense Authorization Act for Fiscal Year 2017—between ports of entry, at ports of entry, in the maritime border domain, and the air and marine security in the land domain. The recommendations listed in the tables below remain open; however, implementing them would provide DHS with additional indicators and metrics that could provide important insights into the state of border security. In addition to the contact named above, Taylor Matheson (Assistant Director), David Alexander, Kelsey Burdick, Lilia Chaidez, Kathleen Donovan, Michele Fejfar, Sally Gilley, Christopher Hatscher, Eric Hauswirth, Mikaela Meyer, Sasan J. “Jon” Najmi, Kevin Reeves, and Jeff Tessin made key contributions to this report.", "summary": "According to DHS, the United States has approximately 6,000 miles of land borders, 95,000 miles of coastline, and more than 300 ports of entry where travelers and cargo are inspected and processed for entry. Securing U.S. border areas is a key part of DHS's mission, and the department's ability to measure its border security efforts is essential for it to manage its responsibilities effectively and efficiently. The NDAA for Fiscal Year 2017 requires DHS to report annually on 43 border security metrics. DHS issued its first report in May 2018. The Act also includes a provision for GAO, within 270 days of receipt and biennially for the following 10 years, to review and report on the data and methodology contained in DHS's report. This report assesses the extent to which DHS: (1) reported metrics as outlined in the NDAA using quality information; and (2) validated assumptions and conveyed statistical uncertainty for unlawful entry metrics, among other objectives. GAO assessed the methodology and data in DHS's report, analyzed DHS's use of statistical models, and interviewed officials from DHS offices and components involved in developing the metrics. The Department of Homeland Security (DHS) reported on 35 of 43 metrics called for by the National Defense Authorization Act (NDAA) for Fiscal Year 2017 (see figure); it generally used quality information, but did not identify some data limitations. GAO found that about half of the 35 metrics generally included elements as called for by the NDAA, while 17 metrics differed, such as in scope or calculation. For example, DHS only provided information on the southwest border for some metrics, such as the estimate of undetected unlawful border crossers for which a methodology for estimating unlawful crossings for the northern border had not yet been completed. DHS components responsible for collecting the metric data generally have processes in place to ensure the reliability of the data and the quality of the information provided. DHS also identified and disclosed limitations for some, but not all, of the data elements and metrics used. For example, GAO found that DHS did not disclose limitations on data related to apprehensions of individuals that were assisted by unmanned aerial systems. By developing and implementing a process to systematically review the reliability of the data and comprehensively identify and communicate limitations, DHS would improve the quality of the information provided. DHS used a statistical model to estimate three metrics on unlawful border entries but did not validate some assumptions the model employs through sensitivity analyses and provide measures of statistical uncertainty in accordance with standards for federal agencies. For example, DHS's model assumes that 100 percent of families unlawfully crossing the border will be apprehended, but DHS did not provide information on the extent to which the assumption affected its metrics. DHS also did not provide information on the level of statistical uncertainty for the metrics, such as margins of error. Providing such information would allow Congress and the public to better understand the potential limitations and accuracy of these metrics of unlawful entry. Additionally, DHS's statistical model, which is based on Mexican adults not seeking asylum, represents a small and declining share of those apprehended at the border and DHS is developing a new model to account for current border conditions. GAO is making four recommendations, including that DHS develop and implement a process to systematically review the reliability of metric data, identify and communicate limitations of the metrics, and include the results of sensitivity analyses and measures of statistical uncertainty for metrics derived from statistical models. DHS concurred with the recommendations.", "document_type": "gao"}
{"report": "Beneficiaries have several Medicare options from which to select, which can have important consequences for their out-of-pocket expenses and access to care. These decisions include the following: What type of coverage? The first coverage decision faced by Medicare beneficiaries is choosing between original Medicare or MA. Original Medicare includes coverage for Medicare Part A services, such as inpatient hospital stays, and for Medicare Part B services, such as outpatient hospital care and physician office visits. Under MA—the private plan alternative to original Medicare—beneficiaries enroll in MA plans that generally must provide coverage for all the services included under original Medicare, and may also offer extra benefits. MA plans generally establish a network of health care providers to provide services to enrolled beneficiaries. Add prescription drug coverage? Beneficiaries in original Medicare and those in certain MA plans may also choose whether to add prescription drug coverage (Medicare Part D). Prescription drug plans are administered by private insurance companies that contract with CMS. Beneficiaries in original Medicare obtain drug coverage by purchasing a separate prescription drug plan (PDP), while those in MA generally obtain coverage by selecting a MA plan that offers prescription drug benefits. MA prescription drug plans and separate PDPs vary in the amount beneficiaries need to pay and in the drugs that are covered. Add supplemental coverage? Beneficiaries in original Medicare can also purchase Medicare supplemental insurance—known as Medigap plans—offered by private insurance companies. These plans help pay for Medicare’s required cost sharing and some out-of-pocket costs not covered under original Medicare, such as emergency health care during international travel. Figure 1 illustrates the decisions beneficiaries have to make when selecting their Medicare coverage options. Two research studies we reviewed indicate that cost is a key consideration for Medicare beneficiaries when selecting Medicare coverage. Beneficiaries may want to know what their likely out-of-pocket costs will be monthly, annually, or both. Beneficiaries may also want to know what their costs may be if they have a change in health status, such as by experiencing an illness. Beneficiaries may be responsible for several specific types of health care costs, including the following: Premiums—Beneficiaries generally make monthly payments to purchase coverage. Medicare Part A generally does not require beneficiaries to pay a premium. Part B premiums are established by statutory formula and are means-tested so that beneficiaries with higher incomes pay higher premiums. The premiums charged by MA plans and Part D plans are established by each plan and can vary widely. Beneficiaries in original Medicare who opt to purchase Medigap will also pay a monthly premium for coverage, with the amount of the premium varying across the 10 standardized plans and by the different companies offering these plans. Cost sharing—Beneficiaries are typically responsible for paying a portion of the costs for the services they receive as either a copayment or coinsurance. A copayment is a fixed dollar amount for each doctor visit, medical service, or medication. With coinsurance, a beneficiary pays a percentage of the allowed charge for each health care service or medication. Deductibles—Beneficiaries must pay out-of-pocket a specified annual amount of expenses before Medicare will begin paying for approved services or medications. MA plans establish out-of-pocket maximums or set limits on the amount a beneficiary will have to spend a year. In contrast, original Medicare has no limit on beneficiary out-of-pocket costs. In 2019, two Medigap plans provide maximum out-of-pocket limits, and beneficiaries with these plans do not have to pay costs above the limits. The same two research studies identified access to particular health care providers as another key consideration for beneficiaries when selecting Medicare coverage. Beneficiaries in original Medicare may see any doctor or use any facility that accepts Medicare payment, and referrals are not needed to see specialists. In contrast, MA beneficiaries must typically use the MA plan’s network of health care providers, including doctors, hospitals, and outpatient facilities, and referrals are generally needed to see specialists. Further, beneficiaries in MA plans that allow access to out-of-network providers may be required to pay more when receiving services from such providers. MA provider networks can change during the year and from year to year. According to CMS officials, MPF was launched in 1998 in response to the Balanced Budget Act of 1997, which required the Department of Health and Human Services—the agency responsible for overseeing CMS—to maintain MA plan information on the internet, among other things. According to CMS, MPF is a primary CMS resource for beneficiaries to compare costs and coverage of different Medicare health and prescription drug coverage options in their area, including comparing original Medicare to MA plans, and Part D plans. As illustrated in figure 2, the MPF landing page—the first web page users see when accessing MPF— includes a section where beneficiaries start the process of searching for and comparing coverage options (see A in fig. 2), and a section providing links to additional decision support tools for beneficiaries (see B in fig. 2). Beneficiaries begin searching in MPF by entering their zip code and following a 4-step process that moves them through different MPF website pages. Step 1—Basic search: Beneficiaries provide responses to requested information, including identifying whether they have Medicare coverage and whether they would like to add prescription drug coverage to their search. Step 2—Enter drugs: Beneficiaries may add a list of prescription drugs, along with the dosage and dosing frequency, to identify which plans cover these drugs and the cost sharing amount under each plan. Step 3—Select pharmacies: Beneficiaries select up to two pharmacies that they prefer for obtaining their medications. Step 4—Refine plan results: Beneficiaries see a list of available coverage options—original Medicare, MA plans, and separate PDPs—based on the zip code they entered. Beneficiaries can filter these search results by variables such as monthly premium or deductible amounts, and then they can sort those results by variables such as lowest estimated annual costs or lowest plan deductible. Beneficiaries can then select up to three choices, view specific coverage and cost details for each, and do a detailed side-by-side comparison of each. The plan results page shows this comparison and includes beneficiaries’ estimated annual out-of- pocket costs for each coverage option they choose to review. The additional decision support tools available on the MPF landing page that beneficiaries may use to help select their Medicare health and drug coverage include the following: 1. Help with Your Medicare Choices, which uses filtering questions to help new beneficiaries understand their Medicare coverage choices; 2. Estimate Medicare Costs, which helps beneficiaries compare the average estimated costs of original Medicare options, such as original Medicare with a prescription drug plan and a Medigap plan, to the costs of MA with prescription drug coverage; and 3. Find and Compare Medigap Policies, which helps beneficiaries find information on the different standardized Medigap plans offered by zip code. Since its inception, MPF has undergone many modifications as new parts were added to the Medicare program, such as the addition of Medicare Part D. According to CMS officials, the agency has also taken steps to make additional changes to improve the website, including technology updates to improve system stability and performance, such as page load times and error rates. In addition, CMS seeks feedback from stakeholders, such as the customer service representatives at the 1-800- MEDICARE help line, SHIP personnel, and others, which according to agency officials, has resulted in additional changes. Changes have included allowing beneficiaries to log into their Medicare account to access some of their existing data, such as their prescription drugs, and the addition of a help feature that can connect beneficiaries to 1-800- MEDICARE customer service representatives for live help. Stakeholders, research studies, and SHIP directors responding to our survey generally indicated that MPF is difficult for beneficiaries to navigate and understand. All 13 stakeholder groups we interviewed reported that MPF is challenging for Medicare beneficiaries to use. Specifically, most stakeholders cited difficulty navigating as beneficiaries click through multiple complex pages in order to find and compare coverage options. For example, two stakeholders noted that beneficiaries must answer questions about their current Medicare health and drug coverage and then go through a series of pages and steps before they can view detailed information on their coverage options. One of these stakeholders also told us that MPF navigation is cumbersome because users cannot jump directly to certain pages or sections that address their needs, such as viewing the availability of preferred pharmacies. One of the stakeholders we interviewed also noted the lack of prominent instructions on how to use MPF contributed to difficulties navigating the four steps. Finally, in our interviews two stakeholders also noted that navigation is difficult because beneficiaries are uncertain of the information needed to make different comparisons or identify specific plans. For example, the ability to filter and sort plan information does not appear until later in the plan search process, where users are refining plan results. This makes it hard for users to narrow options specific to their needs because they first must go through all the options presented. Specifically, beneficiaries will first see a list of plans available in their zip code—on average 24 plans—and then must narrow down that list before they can compare up to three selected plans. A 2018 report conducted jointly by two advocacy groups cited difficulties locating the filter and sort functions in MPF, which contributed to navigation problems. CMS user testing conducted on MPF found that overall beneficiaries are confused about how to find a MA plan on MPF. For example, this testing showed that some users had difficulties with the steps for refining plan results because they overlooked or ignored the filters. A 2017 CMS study noted that MPF navigation is difficult and is better suited for specialist users who assist beneficiaries in determining their coverage options, such as 1-800 MEDICARE customer service representatives and SHIP counselors. Further, CMS officials said the study found that beneficiaries would benefit if navigation through the site were more tailored to the tasks they were undertaking. Our survey of SHIP directors, who provide assistance to Medicare beneficiaries and therefore are familiar with MPF usability, also found that it is difficult for beneficiaries to navigate and find information. Specifically, 73 percent (29 of 40) of the SHIP directors who responded to our survey reported that it is difficult or very difficult for beneficiaries to find information in MPF. While SHIP directors reported that it is easier for SHIP counselors to find information, they noted that some also experience difficulty. Eighteen percent (7 of 40) SHIP directors reported that it is difficult for SHIP counselors to find information in MPF. (See fig. 3.) In addition to website navigation, it is also difficult for beneficiaries to understand the information in MPF, according to stakeholders, research studies, and SHIP directors responding to our survey. All seven beneficiary advocacy groups interviewed reported that beneficiaries find it challenging to understand information in MPF. For example, some stakeholders noted that beneficiaries do not always understand terminology, such as the differences between cost sharing, copayment, and out-of-pocket costs. Most stakeholders also noted that beneficiaries struggle to understand cost estimates and interpret how much they will have to pay. CMS user testing of MPF in 2018 found that beneficiaries were overwhelmed by the number and complexity of options from which they had to choose. According to a 2018 research study conducted by two advocacy groups, the website explains health coverage terminology poorly and does not use plain language. As a result, users with low health insurance literacy may not understand, for example, the cost differences between generic versus brand-name drugs. Sixty-five percent (26 of 40) of the SHIP directors we surveyed reported that the information in MPF is difficult or very difficult for beneficiaries to understand, while 23 percent (9 of 40) reported that it is difficult for SHIP counselors to understand information (see fig. 4). SHIP directors identified health coverage terminology as a challenge, with 38 percent (15 of 40) reporting that MPF does a poor or very poor job explaining health coverage terminology, such as non-network providers, drug formularies, and drug tiers to beneficiaries. MPF provides incomplete estimates of beneficiaries’ costs under original Medicare, making it difficult to compare coverage options, according to stakeholders and SHIP directors responding to our survey. The cost estimates on the plan results pages are incomplete because they do not include the effect of Medigap—which helps cover beneficiaries’ cost sharing responsibilities under original Medicare. As a result, beneficiaries who want to use MPF to compare original Medicare with a Medigap plan to specific MA plans are unable to do so. Most—4 of 7—beneficiary advocacy group stakeholders that we interviewed noted that beneficiaries must leave MPF to obtain information about Medigap plans, such as the specific benefits covered under those plans and their estimated costs. Six of seven beneficiary advocacy groups that we interviewed noted that MPF’s incomplete information on estimated beneficiary costs is a concern because beneficiaries need this information for understanding and comparing their Medicare options. CMS’s other coverage decision support tools—Help with Your Medicare Choices and Estimate Medicare Costs—provide general information intended to help beneficiaries understand and compare their Medicare options. However, these tools are separate links; their information is not included on the plan results pages in MPF. The SHIP directors we surveyed also noted lack of information as a concern, with 75 percent (30 of 40) reporting that the lack of Medigap information in MPF limits the ability of beneficiaries to compare original Medicare and MA plans. Further, SHIP directors surveyed reported more general concerns with MPF’s cost estimates, with 80 percent (32 of 40) reporting that improvements are needed to better estimate total annual beneficiary costs, and 63 percent (25 of 40) of the SHIP directors reporting that MPF does a poor or very poor job comparing the costs of original Medicare to MA. Stakeholders and SHIP directors responding to our survey reported that MPF also provides incomplete information on MA plan provider networks. According to a CMS-sponsored study, determining if specific providers are in an MA plan provider network is a key factor for beneficiaries when making coverage decisions, and beneficiaries stated in user testing that they must have this information. However, to obtain information on the providers in specific MA plans, MPF users must exit the website and go to the individual plan websites. Most stakeholders—10 of 13—cited the lack of information on provider networks as a shortcoming for beneficiaries in using MPF to select a plan, with one group stating that MPF users may need to call individual plans to determine if providers are in a plan’s network. SHIP directors also cited this issue as a problem, as 85 percent (34 of 40) who responded to our survey reported that the lack of a provider directory limits MPF as a resource for beneficiaries to compare MA plans. Without provider information, beneficiaries are not able to use MPF to narrow their options to MA plans that include desired providers or make comparisons among these plans. According to CMS officials, the agency is redesigning MPF to make it more usable for beneficiaries and is planning to release the redesigned MPF in early August 2019. With the redesign, CMS plans to improve the navigation of MPF by providing more prominent explanations on how to use MPF; reducing the steps users must take to get to more detailed coverage information; configuring MPF so users can more easily switch between different topics inside MPF, such as switching between MA plan information and Part D plan information; and improving the filter and sort functions so users can narrow down their coverage options more quickly. CMS also plans to make information easier to understand by simplifying and reducing the volume of information on the pages and revising frequently misunderstood terms with more user-friendly language. As part of the redesign, CMS is also taking steps to provide more complete cost information in MPF to help compare coverage options, according to agency officials. CMS plans to provide more information to the redesigned MPF to help beneficiaries understand their coverage options and decide whether original Medicare or MA is right for them. CMS officials also told us in June 2019 that the redesigned MPF will allow beneficiaries to do estimated cost comparisons of MA to all their original Medicare options, such as original Medicare with a Medigap plan and a prescription drug plan. Officials also told us that CMS is incorporating the functionality of the additional decision support tools currently available on the MPF landing page—Help with Your Medicare Choices and Estimate Medicare Costs—into the redesigned MPF to help beneficiaries understand their coverage options and compare their estimated costs across these options. In June 2019, CMS officials stated these additional tools will also continue to appear as separate links on the MPF landing page. CMS officials also told us that they are currently examining how to integrate MA plan provider information, but this is not part of the redesigned MPF being released in August 2019. The officials said they are working with the plans to develop requirements to help support the integration of provider directories into future versions of MPF. According to CMS, the redesign of MPF is not finalized and CMS will continue to evaluate the extent to which the changes will make MPF easier for beneficiaries to use and whether it provides complete information for making coverage decisions. As of June 2019, CMS officials told us they are continuing to gather feedback from stakeholders, such as 1-800-MEDICARE customer service representatives and SHIP personnel, and conduct user testing on a redesigned MPF model. CMS then plans to publicly launch the redesigned MPF to a subset of users in early August 2019. Once launched, CMS plans to incorporate feedback from this subset of users to confirm the core features that will be released in the redesigned MPF prior to the Medicare open enrollment period starting October 15, 2019. According to CMS officials, the development of the redesigned MPF is an incremental process that will involve continuous changes based on feedback and user testing. According to the agency, CMS will know more about how well the redesigned MPF addresses user needs after it is used by beneficiaries. We provided a draft of this report to the Department of Health and Human Services for review and comment. The Department of Health and Human Services provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Health and Human Services and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the contact named above, Tim Bushfield, Assistant Director; Maggie G. Holihan, Analyst-in-Charge; Sylvia Diaz Jones; Anne Hopewell; Dennis A. Antonio; and Dan Ries made key contributions to this report. Also contributing were Cathy Hamann, Krister Friday, Ethiene Salgado-Rodriguez, Julie Flowers, and Jennifer Rudisill.", "summary": "Medicare beneficiaries—more than 60 million as of 2019—have a series of decisions to make when selecting their Medicare health and prescription drug coverage. Beneficiaries must first choose between two main options for their Medicare coverage: either original fee-for-service Medicare or MA. Within these two options, beneficiaries have many additional choices, and they are permitted to change their coverage at least annually. These selections can be difficult due to the Medicare program's complexity and can have important implications for beneficiaries' out-of-pocket costs and access to providers. According to CMS, the MPF website is intended to help beneficiaries make informed decisions regarding their health care and prescription drug coverage. However, some stakeholders have raised concerns that beneficiaries experience challenges using MPF to compare their Medicare coverage options. GAO was asked to review MPF. This report examines what is known about the usability of MPF and the completeness of its information. GAO reviewed research and CMS documentation on MPF, and surveyed 51 directors of SHIP offices that have counselors who assist beneficiaries with Medicare decisions. Forty SHIP directors completed the survey, resulting in a 78 percent response rate. GAO also interviewed CMS officials and officials with 13 stakeholder groups, including seven beneficiary advocacy groups. GAO provided a draft of this report to the Department of Health and Human Services. The department provided technical comments, which GAO incorporated as appropriate. The Medicare Plan Finder (MPF) website—a primary resource for comparing Medicare coverage options—is difficult for beneficiaries to use and provides incomplete information, according to stakeholders and research studies. These sources and directors of State Health Insurance Assistance Programs (SHIP) GAO surveyed—who assist beneficiaries with their Medicare coverage choices—reported that beneficiaries struggle with using MPF because it can be difficult to find information on the website and the information can be hard to understand. For example, MPF requires navigation through multiple pages before displaying plan details, lacks prominent instructions to help beneficiaries find information, and contains complex terms that make it difficult for beneficiaries to understand information. In response to GAO's survey, 73 percent of SHIP directors reported that beneficiaries experience difficulty finding information in MPF, while 18 percent reported that SHIP counselors experience difficulty. Stakeholders and SHIP directors reported that MPF provides incomplete estimates of costs under original Medicare, making it difficult to compare original Medicare and Medicare Advantage (MA), the program's private heath plan alternative. Specifically, MPF's plan results pages do not integrate information on Medigap plans. (These plans help cover some of beneficiaries' out-of-pocket costs.) Seventy-five percent of the SHIP directors surveyed reported that the lack of Medigap information in MPF limits the ability of beneficiaries to compare original Medicare to MA. The Centers for Medicare & Medicaid Services (CMS)—the agency that administers MPF—is aware of the difficulities beneficiaries face using MPF and is planning to launch a redesigned website in August 2019. According to CMS, redesigning MPF involves multiple iterations of changes and ongoing user testing, and CMS will know more about how well the redesigned MPF addresses user needs after it is used by beneficiaries.", "document_type": "gao"}
{"report": "The government can choose from a wide selection of contract types to acquire the variety and volume of supplies and services agencies require to meet their needs. Contract types vary according to the degree and timing of the responsibility assumed by the contractor for the costs of performance, and the amount and nature of the profit incentive offered to the contractor for achieving or exceeding specified standards or goals. The primary contract types described by the Federal Acquisition Regulation (FAR) fall into two broad categories—cost-type and fixed- price-type—and table 1 summarizes key features of each. As illustrated in figure 1, within these categories the specific contract types range from cost-plus-fixed-fee, in which the contractor has minimal responsibility for the performance costs and the negotiated fee (profit) is fixed, to firm-fixed-price, in which the contractor has full responsibility for the performance costs and resulting profit (or loss). In between are the various incentive contracts, under which the contractor’s responsibility for the performance costs and the profit or fee incentives offered are tailored to the uncertainties involved in contract performance. For contracts with incentive fees or profits, the amount of fee or profit payable is related to the contractor’s performance, and generally involves an objective evaluation by the government of the contractor’s performance toward cost, schedule, or technical goals. Award fees, on the other hand, typically emphasize multiple aspects of contractor performance that are more subjectively assessed, such as the contractor’s responsiveness, technical ingenuity, or cost management. Furthermore, the basic types of contracts may be used in combination, with both fixed-price-type and cost-type contract line item numbers, unless otherwise prohibited. For example, a firm-fixed-price contract may have a cost-type line item for travel. The FAR states that selecting the contract type is generally a matter for negotiation and requires the exercise of sound judgment by the contracting officer. Negotiating the contract type and negotiating prices are closely related and should be considered together. The objective is for the government to negotiate a contract type and price (or estimated cost and fee) that will result in reasonable contractor risk and provide the contractor with the greatest incentive for efficient and economical performance. As also noted in the FAR, the government usually assumes greater risk in its contracts for more complex requirements, particularly those unique to the government. This is especially true for complex research and development contracts, where performance uncertainties or the likelihood of changes make it difficult to estimate performance costs in advance. Cost-type contracts are suitable for instances when uncertainties about contract performance do not allow accurate enough cost estimates to use a fixed-price-type contract—in other words, when programs choose to accept more risk. The level of risk drives the contract type chosen, with the contract then reflecting the risk of the work. DOD programs may use different contract types across the life of the MDAP. For example, DOD guidance notes that the preferred contract type for development efforts is cost-type, and requires particular consideration of fixed-price-incentive contracts for acquisitions moving from development to production. Consistent with the FAR, DOD guidance also notes that firm-fixed-price production contracts may be in the government’s best interest once costs have become stable. DOD and Congress have encouraged use of fixed-price-type contracts where appropriate. For example, DOD’s Better Buying Power initiative, which started in 2010, called for increased use of fixed-price-incentive contracts for programs transitioning from development to production. In addition, the National Defense Authorization Act (NDAA) for Fiscal Year 2017 required DOD to establish a preference for fixed-price-type contracts in the determination of contract type and specified approval requirements for use of cost-type contracts above certain dollar thresholds. Congress has also limited DOD’s ability to use cost-type contracts to acquire production units absent congressional notification. Our prior work contains many recommendations related to incentive-type contracts. For example, in March 2017 we recommended that the Navy remind contracting officials to follow guidance on documenting the rationale for using fixed-price-incentive contracts, and in April 2017, the Navy issued a memorandum addressing this issue. In July 2017 we recommended that DOD collect and analyze data to determine the extent to which incentive contracts achieved desired outcomes. While DOD agreed with the recommendation and developed a template for the military departments to use to collect relevant information, it is still gathering updates from the military departments about the status of this effort. DOD acquires MDAPs through the Defense Acquisition System, which implements an adaptive acquisition framework that allows DOD officials to develop acquisition strategies and employ acquisition processes that match the characteristics of the capability being acquired. The pathway for acquiring major capabilities generally includes four phases, three of which we focus on in this report: (1) technology maturation and risk reduction; (2) engineering and manufacturing development; and (3) production and deployment. Programs typically complete a series of milestone reviews and other key decision points that authorize entry into a new acquisition phase, as illustrated in figure 2. These milestones also typically mark critical contract award decisions. For example, the Milestone B decision commits the resources, including authorizing award of the program’s development contract, needed to conduct development leading to production. Milestone C represents the decision to move forward with initial production, including award of the initial production contract. A number of officials and agencies are involved in DOD’s choice and monitoring of MDAP contracts. Milestone decision authority: The designated individual with overall responsibility for the program who, at the time of key milestone reviews, approves the acquisition strategy with specified contract types. In approving the acquisition strategy, this individual must ensure that the strategy considers how to manage risk and how the contract type selected relates to the level of program risk in each acquisition phase. This individual is to use the acquisition strategy to assess the viability of the proposed approach, ensuring that it clearly explains how it is to be implemented with available resources, and is tailored to address program requirements and constraints. Milestone decision authority for most MDAPs now resides with the military departments following a reform enacted in the NDAA for Fiscal Year 2016. Prior to this reform going into effect, a position within the Office of the Secretary of Defense typically served as the milestone decision authority for MDAPs until they entered the production and deployment phase. Following a reorganization of the Office of the Secretary of Defense enacted in the NDAA for Fiscal Year 2017, the USD(A&S) now serves as milestone decision authority for a small number of MDAPs, such as the F-35 program. For other MDAPs, the following officials serve as milestone decision authority within the military departments: the Assistant Secretary of the Air Force (Acquisition, Technology, and Logistics); the Assistant Secretary of the Army (Acquisition, Logistics, and Technology); and the Assistant Secretary of the Navy (Research, Development, and Acquisition). Program manager: The designated individual with responsibility for and authority to accomplish program objectives for development, production, and sustainment to meet user operational needs. The program manager plans acquisition programs, prepares programs for key decisions, and executes approved acquisition and product support strategies. Contracting officer: The individual with the authority to enter into, administer, or terminate contracts and make related determinations and findings. Contracting officers are responsible for ensuring performance of all necessary actions for effective contracting, ensuring compliance with the terms of the contract, and safeguarding the interests of the United States in its contractual relationships. In order to perform these responsibilities, contracting officers are allowed wide latitude to exercise business judgement. Defense Contract Management Agency (DCMA): The entity that provides contract administration services for most DOD buying activities. Its contract management offices work with defense contractors to help ensure they deliver goods and services that meet performance requirements on time and at projected cost. Supervisor of Shipbuilding, Conversion and Repair (SUPSHIP): The entity that is the Navy’s on-site technical, contractual, and business authority for the construction of Navy ships. SUPSHIPs are co-located with the nation’s major shipbuilders and oversee the construction of every Navy ship, from patrol craft to the Navy’s most complex surface combatants and nuclear submarines and aircraft carriers. In addition to serving as milestone decision authority for certain MDAPs, USD(A&S) is responsible for improving outcomes by gathering and distributing best practices and lessons learned across the military departments. One such mechanism related to contract type choice, established in 2008, was mandatory preaward peer review—conducted by DPC, an office within USD(A&S)—for solicitations and contracts valued at over $1 billion and noncompetitive procurements over $500 million. For these competitive procurements, DPC conducted phased peer reviews prior to three events—issuance of the solicitation, issuance of the request for final proposal revisions, and contract award. The peer review teams—composed of senior DOD contracting leaders and officials from other military departments, and whenever possible comprising the same personnel across the three phases—discussed contract type and structure, and reviewed key program documentation such as acquisition strategies. Upon completion of a review, the team provided its findings and recommendations to the contracting officer, among other officials. However, in August 2019, DPC announced that it would no longer conduct peer reviews for most competitive procurements above $1 billion. Further details of this change are discussed later in this report. While the individual military departments have distinct requirements for the weapon systems they acquire, they also on occasion procure similar types of platforms, and use the same relatively small pool of contractors. For example, the Air Force and Navy both purchase fighter aircraft, and all three military departments buy missile systems. In 2019, we analyzed the 183 major development and procurement contract awards for MDAPs reported by DOD at that time, and found that almost half went to five corporations and entities connected with them, constituting 72 percent of the dollars associated with those contracts. From fiscal year 2011 through fiscal year 2019, a small proportion—an average of less than one-fifth—of obligations for programs in DOD’s portfolio of MDAPs was on cost-type contracts, although this proportion varied across the military departments. The remainder were on fixed- price-type contracts, split between firm-fixed-price and fixed-price- incentive, as illustrated in figure 3. Figure 4 illustrates the proportion of obligations by contract type for each of the military departments across the 9-year period. The Air Force made the most use of cost-type contracts, at an average of around one-quarter of obligations. While the Army made the least use of cost-type contracts, it made the most use of firm-fixed-price contracts. The Navy made the most use of fixed-price-incentive contracts. We have previously reported that the Navy has generally used cost-type contracts for lead ships and fixed-price-incentive contracts for follow-on ships. We found that the choice of cost-type contracts for MDAPs by contracting officers is based on assessments of program risk and uncertainty, underpinned by a number of statutory, regulatory, and policy provisions. Risk assessment also drives the application of additional reporting and surveillance requirements—designed to help the program office monitor cost and schedule performance—once DOD has awarded a cost-type contract for an MDAP. A range of statutory, regulatory, and policy provisions emphasize the importance of considering program risk and uncertainty when planning acquisitions and determining contract types for MDAPs. These provisions guide the decisions of contracting officers when choosing contract type and establish documentation requirements such as acquisition strategies. Table 2 describes key provisions related to program risk and uncertainty. Contracting and program officials, among others, collaborate and determine the appropriate contract type based on assessments of risk, considering factors such as availability of historical contract information, use of new technologies, cost stability, and the level of definition of requirements, such as software. In arriving at these determinations, officials we met with noted the importance of contracting officers having experience using a range of contract types. The seven MDAP cost-type contracts included in our review had documented rationales for their choice that all indicated areas of risk and uncertainty, addressing provisions noted in table 2. For example, four were development contracts, and FAR Part 35 states that the use of cost- type contracts for research and development is usually appropriate given the absence of precise specifications and difficulties in accurately estimating costs. The other three cost-type contract rationales noted that, consistent with the FAR, uncertainties in contract performance did not allow for costs to be estimated with sufficient accuracy to use a fixed- price-type contract. Table 3 summarizes these rationales. Contract types that shift more risk onto the government—including cost- type contracts—and exceed certain dollar thresholds have additional contractual reporting requirements. These requirements are designed to help the program office to monitor cost and schedule performance. In order to receive a cost-type or incentive contract valued at $20 million or more, a contractor must have an earned value management (EVM) system that complies with certain guidelines. These systems integrate the scope of work with cost, schedule, and performance elements to support project planning. They also provide program offices with monthly contract performance reports that include cost and schedule status and risks. Our prior work contains recommendations related to DOD’s use of EVM. For example, in 2009 we recommended that DOD modify policies governing EVM to ensure they addressed a number of weaknesses we had identified. In response, DOD developed and incorporated into its program management curricula a new EVM training course. Among the duties of two specialized government contract administration agencies—DCMA and SUPSHIP—are the review and approval of contractor EVM systems, and ongoing surveillance of data generated by the systems. The regular reports provided to program offices by these agencies include EVM data and analysis and highlight areas of concern and contract performance risk. In addition to use of EVM data, contracting officials from the seven cost- type MDAP contracts included in our review noted the importance of regular interactions between DOD—whether the program office, DCMA, or SUPSHIP—and the contractor in order to proactively identify drivers of cost or schedule overruns. These interactions can range from day-to-day tracking to comprehensive quarterly reviews. Several officials also noted the importance of having DCMA and SUPSHIP representatives on-site at contractor facilities, overseeing the contract and communicating with the contractor. Our analysis of program cost and schedule outcomes for 21 MDAPs did not find a clear relationship between these outcomes and the contract type used. DOD’s current portfolio of MDAPs contains a total of 85 programs. The 21 MDAPs in our review are the non-shipbuilding subset of the 85 that, as of January 2019, had completed system development, held a critical design review, and started production. Thus, these 21 programs are sufficiently far along the acquisition process that we can analyze their cost and schedule outcomes. We found that they demonstrated a range of cost and schedule performance, regardless of contract type chosen. Table 4 notes the contract types used for these MDAPs as well as unit cost and schedule change between each program’s first full estimate and our most recent in-depth assessment of the program as of May 2019. As reflected in the table, all but four of the MDAPs used some mix of cost-type and fixed-price-type contracts. Performance varied widely for programs using cost-type contracts at some stage, with unit cost change varying from 44 percent reduction to 183 percent growth, and schedule change varying from zero to 146 percent growth. In addition, while two of the three programs that used only fixed-price-type contracts had unit cost reductions, they also experienced schedule growth of over 40 percent. Programs generally made greater use of cost-type contracts than fixed-price-type contracts during development, and greater use of fixed-price-type contracts during procurement, as knowledge built over time. While we did not find a clear relationship between contract type and cost and schedule performance, we have found a relationship between improved outcomes and implementation of certain knowledge-based acquisition practices on these 21 programs. These are practices identified in our body of prior work that ensure a high level of knowledge is achieved at key junctures in development. We apply these practices as criteria in weapon system reviews, including our annual assessment of weapon systems. As shown in table 5 and based on analysis of the 21 programs, in general MDAPs that implemented certain knowledge practices—thus reducing risk—before the start of system development and critical design review had better unit cost and schedule outcomes than those that did not. The first such practice—completing preliminary design review before system development start—means that a program has held a review that assesses the maturity of the preliminary design, supported by the results of activities including prototyping and critical technology demonstrations. The second practice—release of at least 90 percent of drawings by critical design review—refers to the design drawings released or deemed releasable to manufacturing by that point. Our prior work has shown that establishing a sound business case is essential to achieving better program outcomes. A solid, executable business case provides credible evidence that the warfighter’s needs are valid and can best be met with the chosen concept. The business case should also demonstrate that the chosen concept can be developed and produced within existing resources such as technologies, design knowledge, funding, and time. At the heart of a business case is a knowledge-based approach, in which knowledge supplants risk over time. Establishing a business case calls for a realistic assessment of risks and costs; doing otherwise undermines the intent of the business case and invites failure. Over the years, we have identified a number of factors that undermine business cases and drive cost and schedule overruns, several of which are illustrated in figure 5. Undesirable outcomes such as cost and schedule growth reflect decisions made to move forward with programs before the knowledge needed to reduce risk and make those decisions is sufficient. For example, we have previously found that the majority of cost growth occurs after production start, which may be a sign that programs are entering production without attaining key knowledge about technology maturity, design stability, and production readiness in preceding phases of development. The primary consequences of risk are often more time and money, and these consequences flow through the acquisition phases, with unplanned overlap—known as concurrency—in development, testing, and production. Our annual assessment of weapon systems has identified numerous examples of programs proceeding without sufficient knowledge to reduce risk, and their subsequent cost and schedule growth. These examples have included the following from among the 21 MDAPs reviewed in this report: The F-35 program started development without a match between resources and requirements and without a stable design. Critical technologies were immature, development and production occurred concurrently, and critical deficiencies were still not resolved well into production. As of May 2019, the program had experienced unit cost growth of 75 percent and schedule growth of 35 percent since its first full estimate in October 2001. The MQ-4C program did not achieve technology maturity or design stability prior to development start and critical design review, respectively, and developmental challenges delayed production start. As of May 2019, the program had experienced unit cost growth of 10 percent and schedule growth of 70 percent since its first full estimate in February 2009. The CH-53K program failed to demonstrate technology and design maturity at appropriate points earlier in system development. As of May 2019, the program had experienced unit cost growth of 21 percent and schedule growth of 60 percent since its first full estimate in December 2005. A year after the production decision for the Ground/Air Task Oriented Radar program, the Marine Corps revised the program’s reliability requirements in response to an expert panel finding that the existing requirements did not reflect operational needs, contributing to delayed full-rate production. As of May 2019, the program had experienced unit cost growth of 168 percent and schedule growth of 146 percent since its first full estimate in August 2005. We have identified and recommended solutions to these issues, including that MDAPs establish firm and feasible requirements, mature technologies, incremental acquisition approaches, and realistic cost estimates. While DOD has agreed with most of our recommendations in these areas, it has not always implemented them. As we noted in our most recent High Risk List report, as of November 2018, 88 recommendations related to DOD weapon systems acquisition remained open. Furthermore, while we had previously reported better cost performance on newer programs initiated after implementation of major acquisition reforms in 2010, more recently we found cost growth on those programs. We attributed the deteriorating performance of newer programs to the inconsistent implementation of knowledge-based acquisition practices, as the negative effects of entering development with insufficient knowledge cascade throughout the acquisition cycle. In August 2019, DPC announced that it would no longer conduct mandatory peer reviews for competitive procurements above $1 billion, except for the small number of MDAPs for which USD(A&S) remains milestone decision authority, and other programs of special interest to USD(A&S). As part of the same announcement, DPC stated that it planned to continue to perform peer reviews for noncompetitive procurements of $500 million or more. DPC officials expect that the procurements no longer covered by DPC’s peer review will instead be covered by the military departments’ own review processes, which already address competitive procurements up to $1 billion. While these review processes exist within the military departments, there is not an active mechanism for sharing across the departments any best practices and lessons learned—including about contract choice—found in the course of the reviews. DPC does not currently have plans to address the reduced potential for information sharing resulting from this change. Figure 6 depicts key developments related to the DPC peer reviews since their establishment in 2008, including the last update to an online compendium—a tool designed to share best practices, lessons learned, and recommendations from peer reviews across DOD—in 2013. According to DPC officials, updates to the compendium stopped as personnel became more familiar with the peer review process. They also noted that the change to peer reviews in 2019 resulted from resource constraints and staff reductions associated with recent acquisition reforms. The officials expect this change to reduce the number of DPC peer reviews by half to approximately 50 per year, consisting primarily of the reviews for noncompetitive procurements of $500 million or more. The peer review process was established with the following objectives: 1. to ensure that contracting officers across DOD consistently and appropriately implement policies and regulations; 2. to continue to improve the quality of contracting processes across 3. to facilitate cross-sharing of best practices and lessons learned across DOD. In support of this third objective, procedures for conducting peer reviews stated that the predecessor office to DPC would look for common trends and issues to be shared with the broader DOD contracting community, and maintain information about best practices and lessons learned on its website. This public website currently houses the online compendium, although, as noted above, the last update was in 2013. Contracting officials we met with noted the value of being able to learn from the experiences of officials in other military departments through peer reviews. For example, contracting officials on an Air Force program that had a peer review involving Navy officials stated that lessons shared by those officials reduced the time it took to subsequently execute a contract. Officials from across the military departments cited benefits that resulted from these opportunities to learn from the real-world experience of peers across DOD, including the ability to share contracting information and expertise, review cost-sharing arrangements, and recalibrate contracting decisions. The online compendium is a spreadsheet with a row for each example of feedback, with the program and officials concerned kept anonymous. Columns include the category of feedback (e.g., source selection, terms and conditions), the type of feedback (e.g., recommendation, lesson learned, best practice), and the phase of review (e.g., issuance of the solicitation). Our analysis of the compendium found that it captures practices and recommendations related to contract type, as illustrated by the following examples: Use of incentives: Consider development of cost and performance incentives, rather than use of an award fee. Different contract type: Reconsider plan to award a fixed-price- incentive contract, given historical use of a cost-plus-incentive-fee arrangement under which contractor delivered at or around target cost. Source selection: Throughout solicitation for an award combining firm-fixed-price and cost-type line items, tell offerors what they are expected to provide and how they will be evaluated, and document that evaluation occurred in this exact way. Officials from the military departments confirmed that they are aware that they will now be expected to perform the reviews that DPC previously conducted. They have taken steps to adjust procedures accordingly, including updating their acquisition regulations as necessary. However, DPC does not currently have plans to encourage sharing of findings from military department-level reviews across the departments. For example, there are no plans to solicit updates to the online compendium or a similar centralized resource. USD(A&S) is responsible for improving acquisition results—including cost, schedule, and performance—by gathering and distributing data, best practices, and lessons learned across the military departments. Without a centralized resource for sharing findings, and as most reviews transition to the military departments, it will become more difficult for USD(A&S) to identify contracting trends across DOD and perform this assigned role. An updated compendium or other centralized resource could help contracting officials continue to learn from the experiences of peers across DOD—including when acquiring similar platforms and from similar contractors—by exposing them to good practices for structuring contracts and prompting consideration of alternative contract types. With DPC conducting fewer peer reviews and no updates to the compendium since 2013, contracting officials might not have insight into how other programs across DOD structure contracts. As the reviews will now primarily occur within the military departments, these officials could lose exposure to alternative contracting approaches suitable for their programs. A centralized resource such as the compendium takes on a new significance as a means for sharing information between the military departments as they proceed with their own peer reviews. USD(A&S) remains well-positioned to facilitate information exchange and contribute to positive program outcomes by requiring the military departments to share the findings of their peer reviews. The Under Secretary of Defense for Acquisition and Sustainment should establish procedures requiring the military departments to collect and share findings from their peer reviews of MDAP contracting approaches— including choice of contract type—such as by updating the existing online compendium of best practices and lessons learned as they complete their reviews. We provided a draft of this report to DOD for review and comment. DOD concurred with our recommendation and provided written comments, which are reprinted in appendix II. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report addresses: (1) the extent to which the Department of Defense (DOD) uses cost-type contracts for major defense acquisition programs (MDAP); (2) how DOD chooses among cost-type and other contract types for MDAPs and monitors their cost and schedule performance; (3) the range of cost and schedule outcomes across MDAPs that used cost-type contracts; and (4) the extent to which DOD shares information about choosing MDAP contract types across the military departments. To assess the extent to which DOD uses cost-type contracts for MDAPs, we analyzed Federal Procurement Data System-Next Generation (FPDS- NG) data regarding obligations by contract type from fiscal year 2011 through fiscal year 2019 on contracts for programs in DOD’s MDAP portfolio awarded from fiscal year 2010 through fiscal year 2018. These data reflect programs that were part of DOD’s MDAP portfolio and contracts that were reported in Selected Acquisition Reports at any point during this period. The basic types of contracts may be used in combination, with both fixed-price-type and cost-type contract line item numbers, unless otherwise prohibited. Per the Defense Federal Acquisition Regulation Supplement (DFARS) Procedures, Guidance, and Information, when entering contract type information info FPDS-NG, the data entrant is to choose the contract type that is applicable to the predominant amount of the contract action, based on the value of the line items; the selected contract type automatically populates any subsequent contract action reports for modifications. We aggregated obligations on orders under indefinite-delivery contracts and basic ordering agreements by contract type for each fiscal year. We used the Defense Acquisition Management Information Retrieval (DAMIR) system to identify those contracts reported in Selected Acquisition Reports for programs in the MDAP portfolio awarded from fiscal year 2010 through fiscal year 2018. Our dataset includes only obligations on MDAP contracts awarded since fiscal year 2010 due to problems identified in a prior GAO report regarding how data on contract types were reported in FPDS-NG for contracts awarded prior to that date. Specifically, prior to fiscal year 2010, data entrants could select the contract types “combination” and “other”, or not enter a contract type at all. The Office of Federal Procurement Policy subsequently removed those contract types as options in FPDS-NG, and made completion of the field mandatory. Contracts retain their original designation in FPDS-NG when modifications to those contracts are subsequently made. Therefore, in order to avoid including contracts coded as “combination” or “other”, we limited our analysis to contracts awarded since fiscal year 2010. We assessed data reliability by comparing the contract types identified in FPDS-NG for each contract with information on contract types contained in DAMIR and in another DOD database—Earned Value Management- Central Repository—and determined the data were sufficiently reliable for the purposes of analyzing the extent of DOD’s use of cost-type contracts for MDAPs. Contractors for programs with earned value management (EVM) reporting requirements submit EVM data to Earned Value Management-Central Repository. EVM reporting is generally required for cost-type or incentive contracts valued at $20 million or more. We included obligations associated with contract types contained in FPDS- NG if they matched contract types contained in either DAMIR or Earned Value Management-Central Repository. When there was no match with either source, we reviewed the narrative discussion of contract types contained in Selected Acquisition Reports in order to determine the most appropriate contract type with which to label those obligations. To assess how DOD chooses among cost-type and other contract types for MDAPs and monitors their cost and schedule performance, we reviewed relevant statutes, regulations, and policies. We analyzed documentation and interviewed officials regarding contract choice and monitoring from the following DOD and military department offices and selected contracting commands: Under Secretary of Defense for Acquisition and Sustainment Acquisition, Analytics and Policy Defense Pricing and Contracting Cost Assessment and Program Evaluation Defense Contract Management Agency Deputy Assistant Secretary of the Air Force for Contracting Deputy Assistant Secretary of the Army for Procurement Deputy Assistant Secretary of the Navy for Procurement Air Force Materiel Command Space and Missile Systems Center Marine Corps Systems Command Naval Air Systems Command Naval Information Warfare Systems Command Naval Sea Systems Command As illustrative examples of contract choice and monitoring under a variety of conditions, including different military departments and appropriation types, we also selected a nongeneralizable sample of seven MDAP contracts. Specifically, we selected for each of the three military departments the most recently awarded cost-type MDAP Research Development, Test, and Evaluation contract and the most recently awarded cost-type MDAP Procurement contract as reported in the December 2017 Selected Acquisition Reports. We also selected the most recently awarded cost-type MDAP contract for the Marine Corps. Table 6 notes the selected MDAPs and contracts, as well as the milestone decision authority responsible for approving the acquisition strategy associated with that contract. We interviewed contracting officials for these programs and reviewed key documentation such as acquisition strategies relating to each one of these contracts. We also reviewed our past work related to contract types used for MDAPs, including DOD’s use of incentive contracts and the Navy’s use of fixed-price-incentive contracts for shipbuilding. To assess the range of cost and schedule outcomes across MDAPs that used cost-type contracts, we identified the contract types as reported in DAMIR or GAO’s April 2018 and May 2019 annual assessments of weapon systems for 21 non-shipbuilding MDAPs that as of January 2019 had completed system development, held a critical design review, and started production. Table 7 notes the 21 MDAPs, as well as the dates of their first full estimate, and their most recent individual assessment by GAO as of May 2019. We compared the contract types reported in DAMIR or GAO’s annual assessments of weapon systems with the percentage unit cost and schedule change between the first full estimate and our most recent in- depth assessment of each program as of May 2019. Since 2018, as part of our annual assessment of weapon systems, we have conducted a statistical analysis evaluating programs’ completion of knowledge-based acquisition practices and corresponding performance outcomes. Our report cites results of this analysis as it pertains to these 21 MDAPs. We reviewed prior GAO work on the drivers of cost and schedule overruns for MDAPs. To assess the extent to which DOD shares information about choosing MDAP contract types across the military departments, we reviewed DOD and military department documentation related to contracting review processes. We compared this information to DOD memorandums establishing practices and policies for sharing of acquisition information across DOD. We also interviewed officials from offices including Defense Pricing and Contracting within the Office of the Under Secretary of Defense for Acquisition and Sustainment, and the cognizant Deputy Assistant Secretaries of the military departments. We conducted this performance audit from February 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Shelby S. Oakley, (202) 512-4841 or oakleys@gao.gov In addition to the contact named above, Raj Chitikila (Assistant Director), Robert Bullock, Jenny Chanley, Jasmina Clyburn, Andrea Evans, Lori Fields, Suellen Foth, Kurt Gurka, Stephanie Gustafson, and Grace Haskin made key contributions to this report.", "summary": "When acquiring major weapon systems, DOD can choose between several different contract types. One of these is cost-type, under which DOD pays allowable costs incurred by the contractor. Historically, DOD has struggled to manage its major acquisition programs. The result has been billions in cost growth and schedule delays in providing systems to the warfighter. GAO was asked to review DOD's use of cost-type contracts for its major acquisition programs. This report addresses the use of and range of cost and schedule outcomes for cost-type contracts for major weapon system acquisitions, and how military departments share information about contract choice. GAO analyzed government contracting data on obligations by contract type for fiscal years 2011 through 2019 on contracts in DOD's portfolio of major acquisition programs. GAO compared contract types for 21 major acquisition programs with their cost and schedule outcomes; reviewed seven recently awarded cost-type contracts for major acquisition programs, selected to reflect the different military departments and appropriation types; and interviewed contracting officials. To acquire new major weapon systems, such as aircraft, ships, and satellites, the Department of Defense (DOD) uses a variety of contract types including cost-type contracts, under which the government assumes more risk. DOD is required to document its risk assessment in choosing contract types for major programs. Risks assessed can include use of new technologies and stability of system costs and requirements. Once awarded, cost-type contracts have additional reporting requirements to help monitoring of cost and schedule performance. GAO analyzed program cost and schedule outcomes for 21 major acquisition programs, and did not find a clear relationship between these outcomes and contract types used. However, programs that completed certain knowledge-based acquisition practices generally had better cost and schedule outcomes than programs that did not implement those practices. These practices include completing preliminary design review before the start of system development and releasing at least 90 percent of design drawings by critical design review. From fiscal years 2011 through 2019, DOD used cost-type contracts for a small proportion—under one-fifth on average—of obligations for its major acquisition programs. This proportion varied across the military departments (see figure). A change to DOD's peer review process for its largest contract awards reduced a means for sharing best practices and lessons learned about contract choice across the military departments. In 2019, the Office of the Secretary of Defense announced the end of its peer reviews for most competitive procurements above $1 billion. While these contracts will instead be reviewed through the military departments' own processes, DOD currently does not require the departments to collect and share their findings. DOD has an online compendium of peer review findings; however, this was last updated in 2013. Using an existing centralized resource such as the compendium could help contracting officials learn from the experiences of peers across DOD by exposing them to good practices for structuring contracts. GAO recommends that DOD establish procedures requiring the military departments to collect and share findings from their reviews of contracting approaches, such as by updating the existing online compendium. DOD agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "DOD’s acquisition of weapon system programs has been on our High Risk List since 1990 because DOD programs consistently fall short of cost, schedule, and performance expectations. Congress and DOD have long explored ways to curtail these cost, schedule, and performance problems, and both took related actions about a decade ago, with Congress passing the Weapon Systems Acquisition Reform Act of 2009 and DOD implementing its “Better Buying Power” initiatives. The Weapon Systems Acquisition Reform Act of 2009 aimed to improve the organization and procedures of DOD for the acquisition of major weapon systems, for example by revising the certifications that programs were expected to complete before approval for system development start. The new certifications included the need to conduct trade-offs among cost, schedule, and performance objectives and for independent verification of technology maturity. In 2010, DOD started its own acquisition reform initiatives through “Better Buying Power.” These reforms required DOD programs to conduct analyses of program affordability and set cost targets, among other things, which placed cost constraints on programs and encouraged programs to find cost improvements during program execution. These and other reforms championed sound management practices, such as realistic cost estimating, increased use of prototyping, and systems engineering. In 2016, we found that DOD was beginning to decrease the amount of cost growth in its major defense acquisition program portfolio. Despite DOD’s improvements in cost control, however, members of Congress remained concerned that the DOD acquisition process was overly bureaucratic and too slow to deliver capability to the warfighter. Congress enacted numerous additional acquisition-related provisions in the National Defense Authorization Acts for Fiscal Year 2016 and subsequent years that addressed the processes with which DOD and the military departments acquire goods and services and encourage innovation. These provisions addressed a wide range of acquisition issues, such as the: creation of new processes for oversight of major defense acquisition programs; development of streamlined alternative acquisition paths; and changes to DOD’s other transaction authority, which allows DOD to enter into agreements that generally do not follow a standard format or include terms and conditions required in traditional mechanisms, such as contracts or grants. Congress also required that DOD establish a panel in the National Defense Authorization Act for Fiscal Year 2016, referred to as the “Section 809 Panel,” to identify ways to streamline and improve the defense acquisition system. The panel issued its final report in January 2019, which, together with its earlier reports, included a wide range of recommendations aimed at changing the overall structure and operations of defense acquisition. DOD acquisition policy defines an acquisition program as a directed, funded effort that provides a new, improved, or continuing materiel, weapon, or information system, or a service capability in response to an approved need. DOD Directive 5000.01, The Defense Acquisition System, provides management principles and mandatory policies and procedures for managing all acquisition programs. Oversight levels and procedures for DOD’s acquisition programs are outlined in DOD Instruction 5000.02, Operation of the Defense Acquisition System. Traditionally, defense acquisition programs are classified into acquisition categories based on the value and type of acquisition. DOD’s most costly programs have historically been referred to as major defense acquisition or Acquisition Category I programs. Programs with lower costs are categorized as Acquisition Category II or III programs. The acquisition category of a program can affect oversight levels and procedures, such as what program information and documents are required and who is designated as the milestone decision authority. Among other responsibilities, the milestone decision authority approves entry of an acquisition program into the next phase of the acquisition process and is accountable for cost, schedule, and performance reporting. DOD’s acquisition process includes three major milestones at which program offices provide information to or receive a waiver from the milestone decision authority. The milestone decision authority then makes a decision on whether the program is ready to transition to the next acquisition phase. The milestones normally represent transition points in programs at which there is a marked increase in the funding required for the program. Milestone A is the decision for an acquisition program to enter into the technology maturation and risk reduction phase. Milestone B is the decision to enter the engineering and manufacturing development phase. Milestone C is the decision to enter the production and deployment phase. Programs may start at different milestones depending on the circumstances of the particular program, such as whether the technologies the program plans to use are mature. Some major defense acquisition programs, such as the Marine Corps’ Amphibious Combat Vehicle program and the Navy’s Next Generation Jammer-Mid Band program, entered the acquisition system at milestone A. Other programs, such as the Air Force’s Combat Rescue Helicopter program and the Army’s Armored Multi-Purpose Vehicle program, entered directly at milestone B without having a milestone A because technologies were considered mature by the Office of the Secretary of Defense and an independent review team, respectively. Figure 1 illustrates the key milestones associated with the defense acquisition system. DOD’s acquisition policy encourages tailoring the acquisition process, including tailoring of documentation or information requirements. In previous work, we identified opportunities for DOD to tailor the documentation and oversight needed for major defense acquisition programs. In 2015, we found that 24 acquisition programs we surveyed spent, on average, over 2 years completing up to 49 information requirements for their most recent milestone decision. We found that DOD’s review process was a key factor that influenced the time needed to complete the information requirements. In total, the requirements averaged 5,600 staff days to document, yet acquisition officials considered only about half of the requirements as high value. We recommended that DOD eliminate reviews and information requirements that do not add value or are no longer needed. DOD agreed with both recommendations and took some actions through its Better Buying Power initiatives to streamline documentation and staff reviews. Among the information requirements that acquisition officials considered most valuable were those that support a sound business case. A solid, executable business case provides credible evidence that (1) the warfighter’s needs are valid and that they can best be met with the chosen concept, and (2) the chosen concept can be developed and produced within existing resources—such as technologies, design knowledge, funding, and time. Establishing a sound business case for individual programs depends on disciplined requirements and funding processes, and calls for a realistic assessment of risks and costs; doing otherwise undermines the intent of the business case and increases the risk of cost and schedule overruns and performance shortfalls. The program’s business case typically includes documentation of the capabilities required of the weapon system, the strategy for acquiring the weapon system, sound cost estimates based on independent assessments, and a realistic assessment of technical and schedule risks. Several entities at the enterprise level (meaning the Office of the Secretary of Defense, Joint Chiefs of Staff, and Joint Staff) and the military department level play a role in the oversight and budgeting for DOD weapon system acquisition programs. In general, at the enterprise level, the acquisition and budgeting processes are managed by subordinate offices within the Office of the Secretary of Defense. More specifically: The Under Secretary of Defense for Research and Engineering is responsible for establishing policies on and supervising all aspects of defense research and engineering, technology development, technology transition, prototyping, experimentation, and developmental testing activities and programs, including the allocation of resources for defense research and engineering. This organization has a significant role in activities prior to milestone B, but also interacts with major defense acquisition programs throughout their life cycles with regard to technical risks. For major defense acquisition programs, the Under Secretary conducts assessments in areas such as technology maturity, interoperability, and cyber security. The Under Secretary of Defense for Acquisition and Sustainment is responsible for establishing policies on and supervising all matters relating to acquisition (including (1) system design, development, and production; and (2) procurement of goods and services) and sustainment (including logistics, maintenance, and materiel readiness). This organization has certain oversight responsibilities for major defense acquisition programs throughout the acquisition process, such as collecting and distributing performance data. The Under Secretary is the Defense Acquisition Executive and serves as the milestone decision authority for certain major defense acquisition programs. The Director, Cost Assessment and Program Evaluation and the Under Secretary of Defense (Comptroller) manage the annual budget preparation process for acquisition programs. These organizations have cost assessment and budgetary responsibilities, respectively, for major defense acquisition programs leading up to each milestone and once these programs have been fielded. At the military department level, the service acquisition executive, also known as the component acquisition executive, is a civilian official within a military department who is responsible for all acquisition functions within the department and can serve as the milestone decision authority. The following officials serve as the service acquisition executive for the military departments: the Assistant Secretary of the Air Force (Acquisition, Technology, and Logistics) for the Air Force; the Assistant Secretary of the Army (Acquisition, Logistics and Technology) for the Army; and the Assistant Secretary of the Navy (Research, Development and Acquisition) for the Navy and the Marine Corps. We focused our review on five selected reforms from the National Defense Authorization Acts for Fiscal Years 2016 and 2017. Three of the reforms affect the processes related to DOD’s oversight of major defense acquisition programs, the fourth restructured acquisition oversight functions in the Office of the Secretary of Defense, and the fifth provides alternative acquisition pathways for programs that are not considered major defense acquisition programs and have an objective of being completed within 5 years. Table 1 identifies the source of the five reforms that we reviewed and provides a brief summary of each reform. For additional detail on the statute, amendments, and related DOD guidance we reviewed, see appendix II. We found that DOD has made progress implementing reforms that have affected the oversight of major defense acquisition programs. Decision- making authority for these programs has been realigned between the Office of the Secretary of Defense and the military departments. In addition, new processes are in place to improve DOD’s consideration of program cost, fielding, and performance goals and assessment of technical risk although questions remain about how they will be implemented. The Office of the Secretary of Defense has also restructured in an effort to increase innovation in the earlier stages of the acquisition process and reduce cost, schedule, and performance risks in later stages. While the restructure has begun to take shape, additional steps remain to be completed, including developing charters and fully staffing new offices. These steps are important to determining how acquisition oversight roles within the Office of the Secretary of Defense— which had been executed by a single office for decades—will be divided and how new offices will be structured to effectively carry out their work. Milestone decision authority for most major defense acquisition programs now resides with the military departments, a reform generally required for programs starting after October 1, 2016 by section 825 of the National Defense Authorization Act for Fiscal Year 2016. According to data from DOD’s Defense Acquisition Visibility Environment system, as of March 2019, milestone decision authority was at the military department level for 80 of 89 major defense acquisition programs. The 80 programs include all six programs that started at milestone B or an equivalent milestone since this reform became effective on October 1, 2016, and 74 other programs that started before the reform became effective. The nine programs retained by the Office of the Secretary of Defense all began prior to the reform becoming effective and include programs that are high risk, joint, or have had significant cost or schedule growth, such as the F-35 Joint Strike Fighter program and the Army’s Integrated Air and Missile Defense program. See appendix III for more information about milestone decision authority, including a list of the major defense acquisition programs as of March 2019 and the milestone decision authority for each. Prior to this reform going into effect, the Under Secretary of Defense for Acquisition, Technology and Logistics within the Office of the Secretary of Defense typically was the milestone decision authority for major defense acquisition programs until they entered the production and deployment phase—that is, for the milestone A, B, and C decisions. The Under Secretary then typically delegated milestone decision authority to the military departments after the milestone C decision. Under the new reform, the Secretary of Defense may designate an alternate milestone decision authority under certain circumstances. For example, the Secretary may determine that the program meets one of several characteristics outlined in statute, such as addressing a joint requirement or the program being critical to a major interagency requirement. There are now substantially more major defense acquisition programs with decision authority at the military department level. This change resulted from both the statutory reform for newly started programs and changes to milestone decision authority for existing programs resulting from a separate review conducted by the Office of the Secretary of Defense after the reform became effective, wherein the military department was designated the milestone decision authority for approximately 20 programs. See figure 2 for trends in the level of milestone decision authority from 2012 to 2019. major defense acquisition program will provide an options matrix to stakeholders including the Under Secretaries of Defense for Research and Engineering and Acquisition and Sustainment, Cost Assessment and Program Evaluation, and the Joint Staff, which must include at least three options that represent differing assumptions about possible solutions, technical risks, cost, schedule, and affordability. gathering and distributing data and lessons learned, and conducting or approving independent cost estimates. These stakeholders must be granted the access necessary to complete independent analysis in their area of responsibility. This analysis will consider aggregated risk regarding technical feasibility, cost, schedule, and affordability, and will be submitted to the milestone decision authority. A goal establishment meeting will be held within 30 days of the program’s analysis of alternatives outbrief and will be co-chaired by the milestone decision authority and Vice Chief of the pertinent military service(s) and supported by the stakeholders identified above. As of March 2019, no programs have held a milestone A since the reform became effective, and no programs have had goals established under the new process. establish a process to develop program cost, fielding, and performance goals for major defense acquisition programs that reach milestone A after October 1, 2017. The statute described the goals as follows: (1) the cost goal is for both procurement unit cost and sustainment cost, (2) the fielding goal is the date for initial operational capability, and (3) the performance goal is for technology maturation, prototyping, and a modular open system approach. DOD issued a policy for the process in November 2018, stating that stakeholders will complete independent analyses in their areas of responsibility to consider the aggregated risk regarding technical feasibility, cost, schedule, and affordability, which will be submitted to the milestone decision authority (typically at the military department level). The policy stated that it applies to all major defense acquisition programs that enter the acquisition process after October 1, 2017, without regard to what milestone initiates the program. The policy also stated that the Office of the Secretary of Defense will have the opportunity to consult with the milestone decision authority on revised goals if the program exceeds its initial cost or fielding goals prior to the next milestone or production decision. DOD acquisition policy already required programs to document objectives for system cost, schedule, and performance in an acquisition program baseline at milestone B and affordability cost goals were to be set at milestone A. Under the new process, fielding and performance goals are established earlier and all three goals (cost, fielding, and performance) are required to be established before funds are obligated for technology development, systems development, or production, rather than being set at specific program milestones. The new process also adds a meeting to review and discuss the goals before they are approved by the milestone decision authority. Officials from the Office of the Under Secretary of Defense for Acquisition and Sustainment told us that this new process is intended to consolidate existing information to inform earlier decisions on which investments the department wants to make. As of March 2019, no major defense acquisition programs have held a milestone A since the statutory requirement became effective, and no major defense acquisition programs have had goals established under the new process. According to officials from the Office of the Under Secretary of Defense for Acquisition and Sustainment, no new programs have been required to have goals established since DOD’s policy for the process was issued in November 2018. These officials told us they rely on the milestone decision authority to notify them when goals need to be established and that the first programs expected to have goals established under the new policy are the Army’s Gator Landmine Replacement Program and the Air Force’s Mk21A Reentry Vehicle. Both programs are slated to go through the process in mid-2019. Independent technical risk assessments. The Under Secretary of Defense for Research and Engineering is now responsible for conducting or approving independent technical risk assessments for major defense acquisition programs prior to milestones A and B and before production decisions. According to DOD’s December 2018 independent technical risk assessment policy, the assessments will consider the full spectrum of technology, engineering, and integration risk, including critical technologies and manufacturing processes, and the potential impacts to cost, schedule, and performance. The reform required the assessments for major defense acquisition programs reaching milestone A after October 1, 2017; no major defense acquisition programs have held a milestone A since that date. DOD policy issued in December 2018 implementing the statute states that the assessments will be conducted for all major defense acquisition programs at each upcoming milestone throughout the acquisition process, effective December 3, 2018. As a result, the assessments will be conducted regardless of whether the program reached milestone A after October 1, 2017. As of March 2019, the Office of the Under Secretary of Defense for Research and Engineering had conducted eight independent technical risk assessments on major defense acquisition programs. One additional assessment on the Infrared Search and Track Block II program was delegated to the Navy to conduct, although the Office of the Under Secretary of Defense for Research and Engineering still approved the assessment. While DOD acquisition guidance previously provided for similar types of assessments, they were not always required to be conducted or approved at the Office of the Secretary of Defense level for all major defense acquisition programs. DOD acquisition guidance previously provided for the Office of the Secretary of Defense to request broad program assessments related to systems engineering, including risk areas, at all milestones for major defense acquisition programs with milestone decision authority at the Office of the Secretary of Defense level. Additionally, all major defense acquisition programs were required to have a separate assessment of critical technology elements prior to entering the system development phase or the production and deployment phase if the system enters the acquisition life cycle after system development. DOD’s December 2018 policy requires that independent technical risk assessments be conducted or approved at the Office of the Secretary of Defense level by the Office of the Under Secretary of Defense for Research and Engineering unless this responsibility is delegated, regardless of the level of the milestone decision authority. The Office of the Secretary of Defense officially reorganized its acquisition organization on January 31, 2018, in response to Section 901 of the Fiscal Year 2017 National Defense Authorization Act. Under the reorganization, responsibilities of the former Under Secretary of Defense for Acquisition, Technology and Logistics were divided between two new offices—the Under Secretary of Defense for Research and Engineering and the Under Secretary of Defense for Acquisition and Sustainment (see fig. 3 and app. IV for organizational charts). According to the conference report accompanying the legislation, the priorities framing the conference discussions on reorganization included elevating the mission of advancing technology and innovation within DOD, and fostering distinct technology and acquisition cultures. The report further states that the conferees expect that the Under Secretary of Defense for Research and Engineering would take risks, test, and experiment, and have the latitude to fail, as appropriate. Additionally, the report states that the conferees expect the Under Secretary of Defense for Acquisition and Sustainment to focus on timely, cost-effective delivery and sustainment of products and services, and seek to minimize any risks to that objective. It is too early to say whether the goals of the reorganization have been realized. In July 2018, the Deputy Secretary of Defense issued a memorandum outlining the overall organizational structures, roles, and responsibilities of the two new Under Secretary offices. Responsibilities of many prior subordinate offices were realigned to one of the two new Under Secretary offices as part of the reorganization. For example, systems engineering falls under the Under Secretary of Defense for Research and Engineering and contracting policy and oversight falls under the Under Secretary of Defense for Acquisition and Sustainment. New offices or positions were also created during the reorganization. For example, the Office of the Under Secretary of Defense for Research and Engineering created eight assistant director positions to serve as resident experts in strategic technology areas, such as cyber, quantum science, and hypersonics. Similarly, the Office of the Under Secretary of Defense for Acquisition and Sustainment created an Assistant Secretary of Defense for Sustainment. Previously, sustainment activities were spread across several organizations headed by two Assistant Secretaries of Defense. While foundational steps to stand up the two new Under Secretary offices have been taken, as of March 2019, reorganization actions were ongoing in two major areas: completing chartering directives that define the scope of responsibilities for the two new offices and hiring additional people for the new offices, including for several senior leadership positions. Chartering directives: Officials from the Office of the Chief Management Officer originally expected charters for the two offices to be completed by January 2019, but progress has been delayed. According to DOD policy, chartering directives are required to define the scope of functional responsibilities and identify all delegated authorities for the chartered organizations. According to a July 2018 memorandum issued by the Deputy Secretary of Defense, the Chief Management Officer is to oversee the development of the charters. Officials from the Office of the Chief Management Officer stated that they are doing so with significant input from the Under Secretaries of Defense for Research and Engineering and Acquisition and Sustainment. These officials told us that the development of the charters has taken longer than expected because redistributing the responsibilities of a single office into two new offices was complicated due to the number of shared or partially overlapping interests. Officials from the Offices of the Under Secretaries of Defense for Research and Engineering and Acquisition and Sustainment now estimate the charters will be completed in July 2019 after department- wide coordination, though they said that time frame may be optimistic given the challenges to date. These officials also told us they expect that they will need to make additional changes to other existing acquisition policies and guidance to incorporate the new content of the chartering directives once complete. Hiring additional employees: In order to stand up the two newly- created organizations, on February 1, 2018, 516 civilian and military positions from the former Office of the Under Secretary of Defense for Acquisition, Technology and Logistics were divided between the two new Under Secretary offices. Finalizing staffing for both offices has been a gradual process that will not be completed until at least fiscal year 2020 because of the need to: (1) reduce positions to meet statutorily-directed cost-savings objectives; (2) realign positions between the two offices; and (3) hire additional staff. Table 2 provides additional detail on past and expected changes to authorized positions. Both Under Secretaries are still working to staff their offices, with approximately 30 percent of current positions vacant in the Office of the Under Secretary of Defense for Research and Engineering, and 8 percent of current positions vacant in the Office of the Under Secretary of Defense for Acquisition and Sustainment. See figure 4 for the current status of staffing within both offices. Both Under Secretaries have experienced challenges while staffing their offices. For example: The Office of the Under Secretary of Defense for Acquisition and Sustainment has experienced challenges stemming from needing to meet required personnel reductions while also hiring staff to align with the revised priorities from the reorganization. As part of the restructuring, the office will absorb all of the 57 remaining civilian and military position reductions that were originally assigned to the former Office of the Under Secretary of Defense for Acquisition, Technology and Logistics. These reductions will occur during both fiscal years 2019 and 2020. At the same time, officials said they are still working to hire staff with skills in needed areas such as data analytics. Officials said they are leveraging existing authorities such as voluntary early retirement authority and voluntary separation incentive payments to meet their targeted number of authorized positions by the end of fiscal year 2020. Officials from the Office of the Under Secretary of Defense for Research and Engineering said their challenges have primarily been negotiating the appropriate number of positions for the organization and staffing the organization in a timely manner. For example, 13 positions are not currently available to be filled because they will not be transferred from the Office of the Under Secretary of Defense for Acquisition and Sustainment until fiscal year 2020. The officials also stated that there have been delays related to developing new position descriptions, revalidating existing position descriptions, and finding individuals with the right skill sets for positions. Both offices have been delayed in filling key leadership positions. According to officials from these offices, vacant positions include the Deputy Director of Mission Engineering and Integration, the Director of Systems Engineering, and the Principal Director of Defense Pricing and Contracting. Senior officials from both offices told us that they have been unable to fill some vacant senior executive positions since the most recent Secretary of Defense resigned on December 31, 2018. The inability to fill these positions is due to the Office of Personnel Management’s general policy to suspend processing for senior executive service career appointments when an agency head leaves, until a successor is appointed at the agency. As of March 2019, a new Secretary of Defense had yet to be confirmed. Senior level officials also told us that some decisions about structure and staffing may be held up until after these executive positions are filled, but that in the interim, they are moving forward with daily operations and in some instances have other employees acting in those roles. As of March 2019, the military departments had begun using middle-tier acquisition pathways for over 35 rapid prototyping and rapid fielding programs under interim guidance issued by the Under Secretary of Defense for Acquisition and Sustainment and the military departments. However, DOD has yet to determine certain aspects of program oversight, including what information military departments should consider in selecting programs and what metrics and data the Office of the Secretary of Defense and military department leaders should use to assess performance. The Departments of the Air Force, Army, and Navy have begun to execute over 35 unclassified and classified acquisition programs using new acquisition pathways distinct from the traditional DOD acquisition process. Section 804 of the National Defense Authorization Act for Fiscal Year 2016 required DOD to issue guidance establishing two new streamlined acquisition pathways for DOD—rapid prototyping and rapid fielding—under the broader term “middle tier of acquisitions.” According to the Joint Explanatory Statement accompanying the National Defense Authorization Act, the guidance was to create an expedited and streamlined “middle tier” of acquisition programs intended to be completed within 5 years. The Joint Explanatory Statement noted that middle-tier programs would be distinctive from rapid acquisitions that are generally completed within 6 months to 2 years and traditional acquisitions that last much longer than 5 years. Statute lays out more specific intended time frames and expectations for programs using these two pathways: The rapid prototyping pathway is to provide for the use of innovative technologies to rapidly develop fieldable prototypes to demonstrate new capabilities and meet emerging needs. The objective of a rapid prototyping program is to field a prototype that can be demonstrated in an operational environment and provide for a residual operational capability within 5 years of the development of an approved requirement. The rapid fielding pathway is to provide for the use of proven technologies to field production quantities of new or upgraded systems with minimal development required. The objective of a rapid fielding program is to begin production within 6 months and complete fielding within 5 years of the development of an approved requirement. Middle-tier acquisition pathways are distinct from the traditional acquisition system for major defense acquisition programs. These pathways allow for programs to be exempted from the acquisition and requirements processes defined by DOD Directive 5000.01 and the Manual for the Operation of the Joint Capabilities Integration and Development System. The statute does not identify a dollar limit for programs using middle-tier acquisition pathways. Middle-tier programs are typically approved for initiation by the service acquisition executive, although Air Force policy also allows for smaller programs to be initiated by the program executive officer. Table 3 shows the number of unclassified programs initiated by the military departments as of March 2019. The middle-tier programs initiated to date represent a range of products, dollar amounts, and complexity. For example, one of the smaller dollar value programs is an approximately $30 million Navy effort to develop a prototype rocket motor that would support extended ranges for an existing missile. One of the larger dollar value programs is a multibillion dollar Army effort to develop the next generation combat vehicle. The military departments generally require funding these programs through the traditional budget process, using DOD’s existing planning, programming, budgeting, and execution process. Based on estimated program costs reported by the military departments, we found that approximately half of the programs initiated to date would be categorized as major defense acquisition programs if they were not being pursued under a middle-tier pathway. In some cases, such as the Army’s Lower Tier Air and Missile Defense Sensor program, an existing program planned as a major defense acquisition program shifted to a middle-tier acquisition pathway. Appendix V includes a list of middle-tier acquisition programs started by the military departments as of March 2019. Although DOD and the military departments have issued interim guidance for using middle-tier acquisition pathways, we found that DOD has not provided department-wide guidance on how certain aspects of program oversight will be conducted. DOD has yet to determine what types of business case information should be submitted to decision makers to help ensure well-informed decisions about program initiation and how program performance will be measured consistently. Section 804 of the National Defense Authorization Act for Fiscal Year 2016 required the Under Secretary of Defense for Acquisition, Technology and Logistics to establish guidance for middle-tier acquisitions. In response, the Under Secretary of Defense for Acquisition and Sustainment issued interim guidance in April 2018 that provided the military departments and other DOD components with the authority to implement middle-tier acquisition programs on an interim basis through September 30, 2019. The guidance laid out the broad purposes and requirements of middle-tier acquisition authorities, and encouraged the military departments and other DOD components using middle-tier acquisition pathways to develop specific implementation processes and procedures to implement the interim authority. Between April 2018 and September 2018, the military departments each issued their own implementing guidance, which provided additional details on how middle- tier programs would be selected and overseen within their department during the period of the interim authority. Subsequently, the Under Secretary of Defense for Acquisition and Sustainment issued two additional interim guidance memorandums: the first in October 2018, which described how the Office of the Secretary of Defense and the Joint Staff would conduct oversight of the military departments’ use of middle-tier acquisition pathways, and the second in March 2019, which addressed sustainment planning considerations for programs using the rapid fielding pathway. The Director, Cost Assessment and Program Evaluation, also issued guidance in April 2019 that included a life-cycle cost estimating policy for programs using the rapid fielding pathway. Statute requires that the guidance from the Office of the Secretary of Defense include a “merit-based process” for considering potential middle- tier programs, although the interim guidance does not describe what the process should include or what information should be considered by decision makers to assess merit other than meeting the needs communicated by the Joint Chiefs of Staff and the combatant commanders. Guidance from each of the military departments provides additional detail on the program selection process, to include describing generally the type of information decision makers should consider when selecting programs. Neither the Office of the Secretary of Defense’s guidance nor the military departments’ guidance fully identifies key elements of a business case to be provided as part of the program initiation process. Our past work has shown that in order to make sound decisions about initiating acquisition programs, it is important that decision makers have the information they need to assess the business case, including that (1) the warfighter need exists and that it can best be met with the chosen concept and (2) the concept can be developed and produced within existing resources. Information needed to establish a business case for a traditional acquisition program typically includes a requirements document (which provides information on the capabilities required of the weapon system); the strategy for acquiring the weapon system; sound cost estimates based on independent assessments; and a realistic assessment of risks, including those risks related to technology and schedule. For a middle-tier acquisition program, business case information would help decision makers make well-informed decisions, to include assessing whether the program is likely to meet objectives established in statute to complete a prototype with a residual operational capability (in the case of a rapid prototyping program) or complete fielding (in the case of a rapid fielding program) within 5 years of an approved requirement. Programs using a middle-tier pathway are intended to be completed within 5 years, and guidance may provide for expedited and streamlined procedures. As a result, the appropriate documents to provide business case information for a middle-tier acquisition program may not need to be as detailed as those for a major defense acquisition program. These documents may also vary to some extent depending on whether a program is a rapid prototyping or a rapid fielding program. However, having this type of information available in some form at program initiation can help decision makers to assess the soundness of a program’s business case at the time a decision is made to start a new program. Oversight at this time is critical because, as we have previously reported, program initiation presents the greatest point of leverage in the program life cycle for decision makers. Table 4 provides additional detail about certain types of business case documentation that are to be considered at program initiation for middle-tier acquisition programs according to the Office of the Secretary of Defense and the military departments’ guidance. Section 804 of the National Defense Authorization Act for Fiscal Year 2016 directed the Under Secretary of Defense for Acquisition, Technology and Logistics to establish guidance for middle-tier acquisitions within 180 days of enactment of the statute (which would have been May 2016), but guidance was not issued until April 2018. According to officials who were involved in efforts to develop the guidance, the Office of the Secretary of Defense circulated multiple iterations of draft guidance, but was unable to reach agreement with the military departments because of concerns that the guidance was too burdensome. These officials told us that as a result, the Under Secretary of Defense for Acquisition and Sustainment decided instead to issue broad interim guidance and allow each of the military departments and other DOD components to develop processes and procedures to implement the interim authority. As stated in the Under Secretary of Defense for Acquisition and Sustainment’s April 2018 interim guidance, the Under Secretary of Defense for Acquisition and Sustainment would develop final guidance for the department in 2019 based on lessons learned from the military departments and other DOD components. According to officials from the Office of the Under Secretary of Defense for Acquisition and Sustainment, the Under Secretary began the process in February 2019 to develop this final guidance. The process was in its initial stages as of March 2019 and officials involved told us they hope to complete the final guidance by September 2019. The business case information programs provided to decision makers at initiation varied widely for nine middle-tier acquisition programs we reviewed. We found that certain types of business case information, such as an assessment of schedule risk that would indicate whether a program could realistically be expected to be completed within time frame objectives in statute, were often not completed at the time of program initiation. For example: Six programs had approved requirements at program initiation. Three of these programs had requirements validated through DOD’s traditional requirements process prior to the decision to start under a middle-tier pathway. Two of these programs, both of which were Air Force programs, had previously planned to start as major defense acquisition programs. The third program, an Army program, had requirements based on those approved for an existing major defense acquisition program. The other three of these programs, all of which were Navy programs, had high-level requirements that described, for example, what environments the system should be tested in or what quantity should be fielded. These requirements were approved by the Navy’s Accelerated Acquisition Board of Directors, which includes the Chief of Naval Operations and the Assistant Secretary of the Navy for Research, Development and Acquisition, among other officials. Three programs were still in the process of developing requirements at the time of program initiation. Only one of the nine programs had an approved acquisition strategy at the time of program initiation. Officials from the other programs told us they planned to develop an acquisition strategy or were in the process of developing or updating one. While all nine of the programs had developed at least a draft cost estimate at program initiation, only one of the nine programs had an assessment of its program cost estimate completed by the military department cost agency at the time of program initiation. Officials from three other programs said that an assessment by the military department cost agency was in progress or planned. Officials from the other five programs told us they had developed a draft cost estimate at program initiation that in some cases was still expected to change and that they did not plan for an assessment by the military department cost agency. The programs varied in the extent to which they assessed risk at program initiation. Four programs had risk assessments that addressed schedule and technology risks, which are types of risks we have identified in our previous work as important to understanding a program’s business case. Two other programs had risk assessments that included either schedule or technology risks but not both. Officials from the other three programs stated that they were still in the process of assessing risks and had yet to assess risks related to meeting statutory schedule objectives at the time of program initiation. Without the Office of the Under Secretary of Defense for Acquisition and Sustainment identifying in its final guidance the minimum program information needed to help decision makers evaluate the program’s business case, DOD cannot ensure that the military departments are consistently considering these types of information. As a result, DOD is not well positioned to ensure that approved middle-tier acquisition programs represent sound investments and are likely to meet the objective of delivering prototypes or capability to the warfighter within 5 years. The Office of the Secretary of Defense and the military departments generally collect program data but we found that neither the Office of the Secretary of Defense nor the military departments has identified metrics that would allow them to use that data to measure and report on program performance in a consistent manner. Developing such metrics would allow senior leaders in the Office of the Secretary of Defense and the military departments to monitor and assess performance across the portfolio of middle-tier programs during program execution, including whether programs are on track to meet statutory objectives for rapid prototyping and rapid fielding. Table 5 provides additional detail on the extent to which guidance addresses the collection of program data and identification of metrics to measure program performance. The Office of the Secretary of Defense began collecting middle-tier program data from the military departments in November 2018 as part of an effort to ensure that middle-tier authority was being used appropriately within the department. However, the office has yet to determine what metrics it will use to measure program performance consistently across the portfolio. Officials within the Office of the Secretary of Defense who are involved with collecting the data told us that they are still refining what data should be collected, determining how to standardize definitions to improve the consistency of data, and considering how to use the data collected to monitor program execution. For example, they are still trying to determine the appropriate triggers that would allow them to know that a middle-tier program may be experiencing cost or schedule challenges. Similarly, guidance from two of the three military departments requires the collection of program data, but the military departments also have not identified metrics to consistently measure performance across programs. The Navy’s guidance does not require the collection of program data or identify metrics to measure program performance. Interim guidance from the Air Force and the Army requires the collection of program data and also requires programs to develop metrics to measure performance, but these metrics are not required to be consistent across programs. Decisions about specific metrics to be reported are left to the discretion of the decision authority for each program, who is typically the service acquisition executive or a program executive officer. As a result, these metrics may not allow consistent measurement of performance across programs because, for example, programs may have a different starting point for reporting data, or may change the metrics that are being assessed at different points within the life of a program. According to federal internal control standards, the ability of agency management to compare actual performance to planned or expected results throughout the organization and analyze significant differences is important to help ensure that the agency is meeting objectives and addressing risks appropriately. These standards also state that agency management should define objectives in quantitative or qualitative terms to permit reasonably consistent measurement of performance toward those objectives. For middle-tier acquisition programs, statute includes objectives related to fielding time frames for both rapid prototyping and rapid fielding programs. Additionally, for rapid prototyping, part of the objective is that the prototype fielded can be demonstrated in an operational environment and provide for residual operational capability. Middle-tier acquisition programs are to be provided streamlined processes, including for program oversight. Decisions about how to measure program performance therefore should be considered in light of how to facilitate oversight without losing the benefits of the flexibilities offered by middle-tier pathways. However, without the Office of the Under Secretary of Defense for Acquisition and Sustainment identifying in its final guidance a minimum set of metrics that can be used to measure performance of programs across the military departments, DOD risks not knowing how the department’s portfolio of middle-tier programs is progressing, including whether programs are on track to meet statutory objectives for rapid prototyping and rapid fielding. As a result, senior leaders in the Office of the Secretary of Defense and the military departments may lack insight needed to identify and address emerging challenges in a timely manner. This is particularly important given that the portfolio includes complex, costly programs that address important capability gaps for the department. While DOD has made progress implementing individual reforms, it continues to face challenges that affect the implementation of the reforms we reviewed. First, we found that senior DOD leadership has not fully addressed disagreements about the division of acquisition oversight roles and responsibilities between the Office of the Secretary of Defense and the military departments. As a result, there have been continuing differences of opinion about how to implement specific reforms. Second, DOD has yet to address persistent portfolio management challenges that affect its ability to effectively manage its portfolio of weapon system investments. Lastly, DOD has yet to develop processes to assess the effectiveness of recent reforms. Without developing such processes, DOD officials will not be well positioned to assess whether reforms are having the intended effects, such as improving innovation and delivering capability to the warfighter more quickly, or if additional changes are necessary to achieve such outcomes. Top DOD leadership has not fully addressed disagreements that remain about the division of acquisition oversight responsibilities between the Office of the Secretary of Defense and the military departments. Our past work has shown that in times of significant organizational transformation, top leadership must set the direction, pace, and tone for the transformation. Personal involvement of these leaders in driving change, including the Secretary and Deputy Secretary, helps provide stability. Internal control standards for federal agencies also emphasize the importance of management communicating information down and across organizational levels in order to enable personnel to perform key roles in achieving objectives and addressing risks. The Deputy Secretary of Defense has weighed in on the division of acquisition oversight responsibilities within the Office of the Secretary of Defense and has addressed specific roles and responsibilities for certain reforms. However, despite continuing disagreements about the division of oversight roles and responsibilities between the Office of the Secretary of Defense and the military departments, DOD’s top leadership has not provided a detailed framework addressing the appropriate roles of each party for acquisition oversight. Officials from the Office of the Secretary of Defense and the military departments we met with expressed different opinions on the appropriate oversight role of the Office of the Secretary of Defense. For example, the Under Secretaries of Defense for Research and Engineering and Acquisition and Sustainment both stated that in cases where the milestone decision authority is at the military department level, the military departments do not see the value in having the Office of the Secretary of Defense involved. This is consistent with concerns that officials from all three military departments have raised in speaking with us. Specifically, officials from all three departments raised concerns that the Office of the Secretary of Defense is overreaching on its oversight responsibilities in some cases, and creating new oversight processes that contradict the intent of recent reforms to speed up the acquisition process. Implementation of several of the reforms we reviewed has resulted in disagreements between the Office of the Secretary of Defense and the military departments that have yet to be resolved. For example: Cost, fielding, and performance goals. Despite the issuance of policy by the Deputy Secretary of Defense in November 2018 on the establishment of cost, fielding, and performance goals, military department officials have continued to express concerns that the process is too burdensome and involves too many stakeholders from the Office of the Secretary of Defense. These officials stated that Office of the Secretary of Defense involvement in programs with decision authority at the military departments, such as participation in meetings with the milestone decision authority to provide advice on cost, fielding, and performance goals, would slow down programs that other reforms were intended to accelerate. They added that they had expressed these concerns to the Office of the Secretary of Defense during the drafting of the policy, but they did not feel that their input was appropriately considered in the final policy. Officials from the Office of the Under Secretary of Defense for Acquisition and Sustainment stated that the analysis and meetings that involve the Office of the Secretary of Defense are ways for stakeholders to advise the milestone decision authority on program decisions based on information from existing oversight mechanisms, such as independent cost estimates and analyses of alternatives. Previously this type of oversight was conducted via multiple meetings leading up to program milestones. The policy states that the policy procedures will be revisited in 6 months and lessons learned incorporated where needed. Independent technical risk assessments. Debates about who should conduct independent technical risk assessments were elevated to the Deputy Secretary of Defense. Subsequently, the Deputy Secretary issued guidance in December 2018 to reiterate that the Under Secretary of Defense for Research and Engineering would conduct or approve these assessments for all major defense acquisition programs, although that responsibility may be delegated. However, despite the issuance of new guidance, there continue to be ongoing debates about when assessments will be delegated to the military departments. The December 2018 guidance does not include criteria for when responsibility for the assessments may be delegated. Officials from the Office of the Under Secretary of Defense for Research and Engineering said that decisions about whether to delegate assessments should be based primarily on the risk level of the program, but officials from military departments stated that these assessments should be conducted within the military department. Officials from the Office of the Under Secretary of Defense for Research and Engineering told us that they had convened a joint working group with the military departments in February 2019 to address this and other implementation issues related to independent technical risk assessments. In the meantime, nearly all assessments continue to be conducted by the Office of the Under Secretary of Defense for Research and Engineering. Middle-tier acquisition. Office of the Secretary of Defense and military department officials also disagree on the extent to which the Office of the Secretary of Defense should weigh in on the appropriateness of a program using a middle-tier pathway. DOD’s October 2018 interim governance guidance provided that the Office of the Secretary of Defense may determine that specific programs were not appropriate for a middle-tier pathway. However, officials from the Air Force and the Army expressed concerns to us about whether that determination was appropriate to be made by the Office of the Secretary of Defense since, from their perspective, programs should be selected at the military department level. Office of the Secretary of Defense officials also told us that there are differences of opinion between them and the military departments on the appropriate amount of information that programs should report to the Office of the Secretary of Defense, including whether the same information should be provided by all middle-tier programs, regardless of expected program cost. As stated earlier, DOD is in the process of finalizing guidance for middle-tier acquisition programs, which could address these issues. Documents that could outline roles and responsibilities of the various parties for acquisition oversight are still being developed. For example, as discussed earlier, officials from the Offices of the Under Secretaries of Defense for Research and Engineering and Acquisition and Sustainment told us that chartering directives for these offices, which are expected to be completed in July 2019, may address to some extent how the offices should work together and with the military departments and other external organizations. In addition, officials from the Office of the Under Secretary of Defense for Acquisition and Sustainment told us that while reforms are currently being implemented under multiple different polices and guidance documents, DOD Instruction 5000.02 will be substantially revised, including to reflect the latest reforms. When completed, the instruction is expected to provide further detail on how oversight activities will be carried out by various acquisition entities. Officials stated that they hoped to complete a version of the revision of DOD Instruction 5000.02 by the end of 2019, but they acknowledged that this estimate was optimistic and that it might take longer than expected to come to agreement on this policy. Without a comprehensive framework from top leadership in the near term that addresses acquisition oversight roles and responsibilities in detail, DOD’s ability to continue with reform implementation, including its ability to finalize policies that could clarify roles and responsibilities, may be slowed by ongoing disagreements. In the longer term, without resolving these issues, DOD cannot ensure that it is achieving the balance between oversight and accountability and efficient program management that senior leadership expects as an outcome of acquisition reform. With too little oversight, acquisition programs may not be properly scrutinized before they are started, which could lead to poor program cost and schedule outcomes. Alternatively, if new oversight processes are too burdensome, DOD may not achieve the expected benefits of streamlining its acquisition processes. As part of this review, we also assessed DOD’s efforts to implement our previous portfolio management recommendations and identified opportunities and challenges related to portfolio management that DOD may face as it continues to implement acquisition reforms. Our past work has shown that when investments are not managed as a portfolio at the enterprise level (meaning at the level of the Office of the Secretary of Defense, Joint Chiefs of Staff, and Joint Staff), the military departments plan to acquire more weapons than DOD can afford, sometimes develop potentially duplicative solutions to common needs, and do not always choose an optimal mix of investments to ensure the department can maintain its technological edge in the future. Realigning roles and responsibilities for decisions related to weapon system programs between the Office of the Secretary of Defense and the military departments could lead to further questions about who is ultimately responsible and accountable for portfolio management decisions if leadership roles are not clearly defined. Officials we met with from the Office of the Secretary of Defense told us that questions remain about the division of responsibilities between the Office of the Secretary of Defense and the military departments for making these types of portfolio management decisions. They told us that concerns we had previously identified about the division of decision-making authority for portfolio management had yet to be addressed during the implementation of recent acquisition reforms, and that in some cases, the reforms had led to additional questions. For example, the Under Secretary of Defense for Research and Engineering told us that while the statute that created his position as part of the restructuring of the former Office of the Under Secretary of Defense for Acquisition, Technology and Logistics assigns him responsibility for allocating resources for defense research and engineering, because he does not have control over the research and engineering budget, in actuality the military departments decide how to prioritize their investments. We found in August 2015 that DOD has had difficulty implementing portfolio management at the enterprise level in part due to diffuse decision-making responsibilities that make it difficult to determine who is empowered to make enterprise-level weapon system investment decisions. At that time, we recommended that DOD revise its portfolio management directive in accordance with portfolio management best practices. We also recommended that the Secretary of Defense designate the Deputy Secretary of Defense or some appropriate delegate responsible for the directive’s implementation, among other recommendations. DOD partially concurred with the recommendations, but the planned actions DOD identified at the time of our report did not fully address the issues we identified. For example, DOD stated that it did not plan to revise its portfolio management directive as we recommended, but instead planned to rescind it and direct stakeholders to participate in portfolio management through the requirements, acquisition, and budget processes. In response, we expressed concern that this approach could reinforce the stove-piped governance structure that we found to be an impediment to integrated portfolio management. As of March 2019, DOD had yet to implement our recommendations. An official from the Office of the Under Secretary of Defense for Acquisition and Sustainment told us that DOD is revising its portfolio management directive, but that there was not yet an estimated completion date. We are not making new recommendations on portfolio management in this report, but we continue to believe that DOD should implement our prior recommendations in order to improve its portfolio management capabilities. See appendix VI for additional details on our assessment of DOD’s progress in this area. DOD is beginning to monitor the implementation of certain reforms, but has yet to establish processes to assess the overall effectiveness of its reform efforts. Collectively, the reforms offer the potential for DOD to significantly reduce the time needed to approve and field acquisition programs by allowing the military departments additional opportunities to tailor documents needed for approval and limiting oversight by the Office of the Secretary of Defense. Ultimately, DOD anticipates that this opportunity will improve the speed at which new capabilities are delivered to the warfighter. Our prior work has identified steps that agencies, such as DOD, can take to help ensure successful implementation of reform efforts, including establishing clear outcome-oriented goals and performance measures putting in place processes to collect the needed data and evidence to effectively measure the reforms’ outcome-oriented goals. The Office of the Secretary of Defense has taken some initial steps to collect data that may help to measure the outcomes of a few reforms, but has yet to determine goals or processes for assessing the overall effectiveness of the reforms. For example, as previously discussed, the Office of the Under Secretary of Defense for Acquisition and Sustainment began initial efforts to collect middle-tier acquisition program data, such as cost and schedule data, from the military departments in November 2018. Officials from that office told us that once they address reliability concerns with the data they are receiving, such as ensuring that programs report schedule data in a consistent fashion, they anticipate that they will be able to use the data to better understand the military departments’ use of middle-tier acquisition pathways. However, according to officials we spoke with from the offices of the Under Secretaries of Defense for Research and Engineering and Acquisition and Sustainment, DOD has not determined how it will assess whether the reforms are collectively resulting in an acquisition process that is more efficient or how it will measure their effect on cost and schedule outcomes. These officials told us that it is important to have data to assess the effect of recent acquisition policy and organizational changes, but they have not determined specifically who will do the assessment, how it will be done, and what data will be needed. They told us that as a part of the reorganization of the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, they are still in the process of assessing data gaps and needs within the newly-formed organizations and that this type of analysis needs to be completed before they determine how they will assess recent reforms. We recognize that assessing the cumulative effect of recent acquisition reforms on the acquisition process and on the cost and schedule performance of the major defense acquisition program portfolio could take several years because a critical mass of programs will need to go through the new acquisition processes. In the interim, however, determining how an assessment of reforms will be conducted is an important first step in determining whether the reforms are having their intended effect. If DOD officials wait too long to plan for how the department will assess the effect of recent acquisition reforms, including identifying who will be responsible for the assessment and what data will be needed, they may miss the opportunity to collect data from the beginning of implementation needed to measure progress. As a result, they may not be informed about early indications of improvements or problems in the cost, schedule, and performance of programs. Recent acquisition reforms have given DOD significant opportunity to focus on delivering innovative capability to the warfighter more quickly and reduce bureaucratic processes that had built up over time. While DOD has made progress in implementing these reforms, continued attention from top leadership would help ensure that the progress the department has made is not unnecessarily slowed or halted. Middle-tier acquisition will require careful consideration as the department proceeds with the development of final guidance. Middle-tier programs are generally exempt from traditional acquisition and requirements processes, but they may still be large, expensive programs critical to the department’s ability to meet its mission. Identifying the types of business case elements decision makers should consider when initiating programs would improve the department’s ability to ensure that the programs the military departments select are sound investments and likely to succeed using a middle-tier acquisition pathway. Identifying metrics to track performance consistently across the portfolio of middle-tier programs will provide necessary information to senior leaders once programs have been started to assess the performance of middle-tier acquisition programs, including whether they are well positioned to meet statutory objectives. The department also faces challenges that affect the implementation of the reforms we reviewed. These sweeping changes have resulted in some disagreements about oversight roles and responsibilities between the Office of the Secretary of Defense and the military departments that have not been fully resolved. Clear communication from top leadership of a framework for oversight roles and responsibilities that is detailed enough to address areas of continued disagreement would help the department to move forward with effective implementation of the reforms. Developing an approach to assess the effects of recent acquisition reforms is also critical so that DOD can monitor whether reforms are collectively having the effect of speeding up the acquisition process without unintended negative consequences on cost and performance of acquisition programs. We also continue to believe that DOD should address our past recommendations to clarify and strengthen roles and responsibilities at the enterprise level for making portfolio management decisions to make sure that its investments are strategy-driven, affordable, and balance near- and long-term needs. In fact, these recommendations may take on more importance for DOD in light of the implementation of acquisition reforms that will further diffuse responsibility for initiating and overseeing acquisition programs. We are making the following four recommendations to DOD: The Secretary of Defense should direct the Under Secretary of Defense for Acquisition and Sustainment to identify in final guidance the types of business case elements potential middle-tier acquisition programs should develop and decision makers should consider at program initiation to assess the soundness of programs’ business cases, including whether programs are well positioned to meet statutory objectives. (Recommendation 1) The Secretary of Defense should direct the Under Secretary of Defense for Acquisition and Sustainment to determine and identify in final guidance for middle-tier acquisition programs the metrics that will be used to assess the performance of middle-tier acquisition programs across the military departments, including whether programs are meeting statutory objectives. (Recommendation 2) The Secretary of Defense should ensure that a comprehensive framework that clarifies the roles and responsibilities of the Office of the Secretary of Defense and the military departments for acquisition oversight is communicated by senior leadership. This framework should be detailed enough to address areas of continued disagreement among key stakeholders and serve to inform the department’s revisions of other acquisition policies such as DOD Instruction 5000.02. (Recommendation 3) The Secretary of Defense should develop a plan for how the department will assess the effect of recent acquisition reforms, including identifying who will be responsible for the assessment and what data will be needed. (Recommendation 4) We provided a draft of this product to DOD for comment. In its comments, reproduced in appendix VII, DOD concurred with our four recommendations. DOD also provided technical comments with regard to improving the clarity of the discussion of certain reforms and providing additional context about military departments’ oversight practices for middle-tier acquisition programs, among other issues. We incorporated DOD’s technical comments as appropriate. In its written comments, DOD described planned actions to address our recommendations. Specifically, in response to our first recommendation, to identify the types of business case elements that should be considered by decision-makers for middle-tier programs at program initiation, DOD stated that it expects to identify these business case elements in its final guidance on middle-tier programs, which it expects to complete in September 2019. In response to our second recommendation, to identify metrics that will be used to assess the performance of middle-tier programs, DOD stated that it plans to determine performance metrics in coordination with its release of its final guidance on middle-tier programs. DOD expects to release this guidance in late 2019. In response to our third recommendation, for senior leadership to clarify acquisition oversight roles and responsibilities, DOD stated that these roles and responsibilities will be finalized through the issuance of chartering directives and updated acquisition policy; issuance is expected by the end of 2019. Finally, in response to our fourth recommendation, to plan for assessing the effects of acquisition reforms, DOD stated that it has included a division in the Office of the Assistant Secretary of Defense for Acquisition to analyze and assess this and other high-level oversight and policy issues. DOD’s planned actions to address our first, second, and fourth recommendation, if implemented effectively, should address the intent of our recommendations. With regard to our third recommendation, however, we do not believe that the steps outlined in DOD’s written comments are likely to fully address the disagreements about acquisition oversight roles and responsibilities that we identified in the report. We acknowledge in the report that DOD plans to issue chartering directives and re-issue DOD Instruction 5000.02 as part of its efforts to outline the roles and responsibilities of various parties for acquisition oversight, as DOD reiterated in its written comments. However, without a comprehensive framework to inform the revisions of acquisition policies, such as DOD Instruction 5000.02, DOD’s ability to finalize these policies may be hindered by the disagreements between the Office of the Secretary of Defense and the military departments that we identified in our report. These disagreements are persistent and focused on fundamental acquisition oversight issues. Simply issuing chartering directives and finalizing policy as planned may not be enough to ensure that areas of disagreement are resolved and that officials within the Office of the Secretary of Defense and the military departments have a shared understanding of an acquisition oversight framework for the entire Department that will serve as the basis for any policy. Furthermore, without senior leadership within DOD communicating this framework to the Office of the Secretary of Defense and the military departments in sufficient detail to address areas of disagreement among key stakeholders, disagreement will likely persist and the intended impacts of reforms could be stymied. We are sending copies of this report to the appropriate congressional committees and the Acting Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. This report addresses (1) the progress the Department of Defense (DOD) has made to implement selected oversight reforms for major defense acquisition programs; (2) how DOD has used middle-tier acquisition pathways and the extent to which DOD has developed guidance on middle-tier program oversight; and (3) challenges DOD faces related to reform implementation. The conference report for the National Defense Authorization Act for Fiscal Year 2018 and the Senate Armed Services Committee report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2018 also contained provisions for GAO to review project, program, and portfolio management standards within DOD. Appendix VI of this report includes our assessment of DOD’s efforts to implement our previous portfolio management recommendations and identifies opportunities and challenges related to portfolio management that DOD may face as it continues to implement acquisition reforms. We focused our review on five selected reforms from the National Defense Authorization Acts for Fiscal Years 2016 and 2017 that we determined substantially affected DOD’s oversight of acquisition programs. Our selections were informed by our analysis of the National Defense Authorization Acts for Fiscal Years 2016 and 2017 and our past work on factors affecting the oversight of major defense acquisition programs. We also interviewed officials from the Office of the Secretary of Defense and the military departments to obtain their perspectives on the most significant reforms to acquisition oversight and considered those perspectives when we made our selections. For the purposes of our report, when we refer to a reform, we are referring to a specific change to DOD’s acquisition oversight processes or roles and responsibilities. Two of the reforms we reviewed align with sections of the National Defense Authorization Act for Fiscal Year 2016, and the other three align with one or more sections from the National Defense Authorization Act for Fiscal Year 2017. Table 6 identifies the specific sections or subsections that we reviewed for each reform. We also reviewed related amendments to these sections from National Defense Authorization Acts for subsequent years to determine whether the National Defense Authorization Act sections we reviewed, or sections of the U.S. Code that were added by sections we reviewed, had been modified since being signed into law. When we identified amendments, we assessed DOD’s progress in implementing the statute as amended. Appendix II provides additional details about the original legislative requirements and amendments, if any, to each of the reforms we selected. To identify the progress DOD has made to implement selected oversight reforms for major defense acquisition programs, we analyzed three selected reforms that affect processes related to DOD’s oversight of major defense acquisition programs: designating military departments to be the milestone decision performing independent technical risk assessments; and establishing cost, fielding, and performance goals. We also analyzed one reform that restructured acquisition oversight functions in the Office of the Secretary of Defense. We analyzed the associated National Defense Authorization Act sections and reviewed related acquisition policies and guidance from the Office of the Secretary of Defense and the military departments (see app. II for a list of key guidance we reviewed for each reform). For each reform, we analyzed DOD and military department policies and guidance to determine steps DOD and the military departments had taken to implement the reforms. We also compared new or updated policies and guidance, when available, with prior policies and guidance to determine how oversight roles, responsibilities, and processes had changed for DOD’s major defense acquisition programs. To obtain additional insight into how designation of milestone decision authority had changed as a result of recent reforms, we requested and analyzed data provided by DOD about the milestone decision authority levels for the major defense acquisition program portfolio. To assess the reliability of these data, we discussed the data and sources used to compile them with DOD officials, reviewed the data for errors, reviewed related documentation on programs with milestone decision authority at the military department level, and compared the data when possible to other sources, such as publicly available lists of major defense acquisition programs. On the basis of these steps, we determined that the data we used were sufficiently reliable to identify changes in the level of milestone decision authority over time for major defense acquisition programs. To assess changes resulting from the reorganization of Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, we also reviewed updated organizational charts and staffing and vacancy data for the successor offices (the Office of the Under Secretary of Defense for Research and Engineering and the Office of the Under Secretary of Defense for Acquisition and Sustainment) and compared these to past organizational charts and staffing data for the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics. To determine the current percentage of positions vacant in each office, we compared actual data for filled positions as of March 2019 to the total number of vacant positions as of the same point in time. The vacancy numbers do not include vacant positions that are slotted for future reduction or transfer. To assess the reliability of these data, we requested and reviewed written responses from DOD officials on the reliability of the data and sources used to compile it, reviewed the data for logical inconsistencies, and compared the data when possible to other sources, such as related data provided for other time frames. On the basis of these steps, we determined that the data we used were sufficiently reliable to identify the current staffing status for the two new Under Secretary offices. To determine how DOD has used middle-tier acquisition pathways, we reviewed the relevant statute and guidance, and obtained information from the military departments about the number and types of programs using middle-tier acquisition pathways as of March 2019. We analyzed the guidance from the Office of the Under Secretary of Defense for Acquisition and Sustainment and the military departments to determine how they were implementing the statute with regard to selection of programs and program oversight. We also compared the guidance with our past work on elements of business cases that should be completed at program initiation to determine what elements were addressed by DOD guidance. At each of the military departments, we judgmentally selected three middle-tier programs to review in additional detail. We selected programs to obtain a range of program costs (including programs that were above the equivalent threshold cost for designation as a major defense acquisition program if the program were not using a middle-tier acquisition pathway, as well as those below that threshold) and types of programs being executed under middle-tier acquisition pathways (such as space, artillery, software, missile, and ground vehicle programs). Programs we selected include: Air Force: Hypersonic Conventional Strike Weapon; Next Generation Overhead Persistent Infrared Space; Protected Tactical Enterprise Service; Army: Extended Range Cannon Artillery; Optionally Manned Fighting Vehicle; Rapid Opioid Countermeasures System; and Navy: STANDARD Missile-2 Block IIIC; STANDARD Missile-6 Block IB Phase IA Rocket Motor; STANDARD Missile-6 Block IB Phase IB All Up Round. For these programs, we collected and analyzed additional information such as acquisition decision memorandums, acquisition strategies, program cost and schedule estimates, and risk assessments. We also interviewed or received detailed written responses from program officials that addressed issues such as how decisions were made to execute programs under middle-tier acquisition pathways and how oversight for programs was being conducted. Further, we reviewed interim guidance from the Office of the Under Secretary of Defense for Acquisition and Sustainment and the military departments to determine how DOD planned to measure middle-tier program performance. We compared DOD and the military departments’ guidance on developing metrics and collecting data to assess middle-tier program performance to relevant internal controls related to consistent measurement of program performance. To assess the challenges DOD faces with regard to reform implementation, we reviewed policy and guidance issued by top DOD leadership that outlined roles and responsibilities for the Office of the Secretary of Defense and the military departments with regard to acquisition oversight and compared them to leading practices for leadership involvement in agency transformations that we had identified in prior work. We also collected and analyzed information about DOD’s actions taken to implement prior recommendations we have made to improve portfolio management at DOD and analyzed the acquisition oversight reforms we included in this review to identify opportunities and challenges related to portfolio management that DOD may face as it continues to implement acquisition reforms. Lastly, we reviewed DOD’s plans and ongoing efforts to develop performance measures and collect data to assess the effects of acquisition reforms and compared these efforts with success factors for reform implementation identified in our past work. For all objectives, we also conducted interviews with officials from the Office of the Secretary of Defense, the Joint Staff, and the military departments to obtain additional insight into implementation status, implementation challenges, and future plans, including: Office of the Secretary of Defense: The Office of the Under Secretary of Defense for Research and Engineering, the Office of the Under Secretary of Defense for Acquisition and Sustainment, the Office of the Under Secretary of Defense (Comptroller), the Office of the Chief Management Officer, the Office of the Director of Operational Test and Evaluation, the Office of the Director of Cost Assessment and Program Evaluation, and the Office of the General Counsel. Joint Staff: Force Structure, Resource and Assessment Directorate, J-8. Military departments: For each of the three military departments (Air Force, Army, and Navy) we interviewed acquisition officials from the Service Acquisition Executive’s office, requirements officials supporting the Chief of Staff of the respective armed force, and officials from the military department cost agencies. At the Air Force we also interviewed officials from the Office of the General Counsel. We conducted this performance audit from March 2018 to June 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix III: Milestone Decision Authority for Major Defense Acquisition Programs as of March 2019 Programs with milestone decision authority at the Air Force level (26) Advanced Extremely High Frequency Satellite AIM-120 Advanced Medium Range Air-to-Air Missile Air Force Intercontinental Ballistic Missile Fuze Modernization Airborne Warning and Control System Block 40/45 Upgrade B61 Mod 12 Life Extension Program Tailkit Assembly F-15 Eagle Passive Active Warning Survivability System Family of Advanced Beyond Line-of-Sight Terminals Global Positioning System III Follow-On Production Military Global Positioning System User Equipment Increment 1 MQ- 9 Reaper Unmanned Aircraft System Small Diameter Bomb Increment II Space Based Infrared System High Space Fence Ground-Based Radar System Increment 1 Wideband Global SATCOM Programs with milestone decision authority at the Army level (18) AH-64E Apache New Build Airborne & Maritime/Fixed Station Joint Tactical Radio System AN/TPQ-53 Counterfire Target Acquisition Radar Common Infrared Countermeasure Programs with milestone decision authority at the Army level (18) Guided Multiple Launch Rocket System/Guided Multiple Launch Rocket System Alternative Warhead Handheld, Manpack, and Small Form Fit Radios M88A2 Heavy Equipment Recovery Combat Utility Lift Evacuation System MQ-1C Gray Eagle Unmanned Aircraft System Patriot Advanced Capability-3 Missile Segment Enhancement RQ-7B Shadow Tactical Unmanned Aircraft System Warfighter Information Network-Tactical Increment 2 Programs with milestone decision authority at the Navy level (36) Advanced Arresting Gear AGM-88E Advanced Anti-Radiation Guided Missile Air and Missile Defense Radar Amphibious Combat Vehicle Phase 1 Increment 1 CVN 78 Gerald R. Ford Class Nuclear Aircraft Carrier DDG 1000 Zumwalt Class Destroyer DDG 51 Arleigh Burke Class Guided Missile Destroyer H-1 Upgrades (4BW/4BN) Joint Precision Approach and Landing System LHA 6 America Class Amphibious Assault Ship Littoral Combat Ship Mission Modules LPD 17 San Antonio Class Amphibious Transport Dock MQ-4C Triton Unmanned Aircraft System Programs with milestone decision authority at the Navy level (36) MQ-8 Fire Scout Unmanned Aircraft System Offensive Anti-Surface Warfare Increment 1 (Long Range Anti-Ship Missile) The conference report for the National Defense Authorization Act for Fiscal Year 2018 and the Senate Armed Services Committee report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2018 included provisions for GAO to review project, program, and portfolio management standards within DOD. This appendix includes our assessment of DOD’s efforts to implement our previous portfolio management recommendations and identifies opportunities and challenges related to portfolio management that DOD may face as it continues to implement acquisition reforms. Portfolio management is used by leading commercial companies to help ensure their investments are optimized to meet customer needs within available resources. Portfolio management focuses on products collectively at an enterprise level and involves evaluating, selecting, prioritizing, and allocating limited resources to projects that best accomplish strategic or organizational goals. It is also a vehicle to make a wide variety of decisions, including capability and funding trade-offs, to achieve the optimal capability mix for a given level of investment. For DOD, effective portfolio management can help to ensure that weapon system investments are strategy-driven and affordable and balance near- and long-term needs. Take a hypothetical example in which DOD starts with 10 programs and $50 billion to invest. Without portfolio management, program managers may seek to get the most that they can out of each of the 10 programs, without assessing their aggregate contributions to defense. Using portfolio management, DOD executives would look at different combinations of and approaches to the 10 programs to determine what, collectively, would provide the best capabilities for $50 billion. This would enable executives to decide, for example, whether it is better to concentrate more investment in seven programs rather than fund all 10 as best as possible. In another example, if a program began to have cost or performance problems, portfolio management would consider whether the other programs in the portfolio could address the requirements of the problematic program rather than just putting more money into it. Portfolio management activities at DOD are carried out at both the enterprise level and the military department level and responsibilities are divided among the requirements community, the acquisition community, and the budget community. At the enterprise level, the primary offices responsible for portfolio management are the Joint Staff (representing the requirements community), the Under Secretary of Defense for Research and Engineering and the Under Secretary of Defense for Acquisition and Sustainment (representing the acquisition community), and the Director of Cost Assessment and Program Evaluation (representing the budget community). In 2007, we identified several best practices for portfolio management, which we revalidated in 2015. These leading practices encourage organizations to assess product investments collectively from an enterprise level, rather than as independent and unrelated initiatives; continually make go/no-go decisions through a gated review process to rebalance portfolios based on investments that add the most value; use an integrated approach to prioritize needs and allocate resources in accordance with strategic goals; rank and select investments using a disciplined process to assess the costs, benefits, and risks of alternative products; empower leadership to make investment decisions and hold leadership accountable for investment outcomes; and provide sustained leadership for portfolio management. Portfolio management best practices and the Project Management Institute’s portfolio management standards also state that organizations should conduct regular reviews to adjust to strategic changes or changes in the mix of products within a portfolio, among other reasons. From a DOD perspective, portfolio reviews can help increase return on taxpayers’ investments in weapon systems in a number of ways, such as: helping to ensure investments align with national security and military strategies; prioritizing the most important investments; selecting the optimum mix of investments; identifying and eliminating unwarranted duplication; monitoring programs’ health to determine whether changes to the portfolio are warranted; and determining whether investments are affordable. We have previously reported that DOD was not effectively using portfolio management to optimize its weapon system investments. In 2015, we identified several factors that inhibited DOD’s ability to do so, including fragmented governance, a lack of sustained leadership, and a perceived lack of decision-making authority at the enterprise level. We also found that DOD’s portfolio management policy was dated, not fully consistent with best practices, and was not being implemented by the Department, in part due to changes in leadership priorities. Further, DOD’s enterprise- level requirements, acquisition, and budgeting communities—meaning those at the Office of the Secretary of Defense, Joint Chiefs of Staff, and Joint Staff level—were not consistently conducting portfolio reviews or collaborating to integrate key information. As a result, we reported that DOD may have been missing opportunities to better leverage its resources and identify investment priorities that best reflect DOD-wide needs. We recommended that DOD update its portfolio management policy; designate a senior official responsible for its implementation; conduct annual portfolio reviews that integrate key information from the requirements, acquisition, and budget processes; and invest in analytical tools to support its portfolio management efforts. DOD partially concurred with the recommendations, but the planned actions DOD identified at the time of our report did not fully address the issues we identified. As of March 2019, DOD had yet to implement our recommendations from 2015 (see table 16 for details of implementation status). It is too soon to assess the effect of the acquisition reforms we reviewed on DOD’s portfolio management efforts because a critical mass of programs has not yet gone through the new acquisition processes. Depending on how the department implements these reforms, some aspects of these reforms could help to address previously-identified deficiencies in portfolio management in the department. For example: Officials in the Office of the Under Secretary of Defense for Acquisition and Sustainment told us that now that milestone decision authority for major defense acquisition programs has largely shifted from the Office of the Secretary of Defense to the military departments, they expect that they will have more time to focus on portfolio-level issues such as identifying how systems need to work together to fill capability gaps since they are less involved in the details of individual programs. We previously reported that DOD’s processes were too focused on optimizing individual investments rather than considering investments across the department. The process developed by DOD to establish cost, fielding, and performance goals brings together officials from DOD’s acquisition, requirements, and budget communities, the three key entities with responsibility for portfolio management, to provide advice on the establishment of program goals. We previously reported that DOD’s enterprise-level processes, organizations, and decision makers oversee weapon system investments generally as stove-pipes and not as an integrated whole. While the process assesses programs on an individual basis rather than collectively from an enterprise level as called for by portfolio management best practices, it may still provide additional shared insight across the acquisition, requirements, and budget communities to help assess portfolios in a more integrated fashion at an enterprise level. However, other aspects of certain reforms have the potential to exacerbate challenges we have previously identified with DOD’s portfolio management approach if not actively managed. For example: Realigning roles and responsibilities for decisions related to weapon systems programs could lead to further questions about who is ultimately responsible and accountable for portfolio management decisions if leadership roles are not clearly defined. We previously reported that DOD’s governance structure for portfolio management was fragmented in part as a result of widely-dispersed decision making responsibilities for weapon system investments. We found that this dispersion of responsibility made it difficult to determine who was empowered to make enterprise-level weapon system investment decisions and who can be considered portfolio managers. According to portfolio management best practices, leadership should be clearly defined and held accountable for outcomes. Programs under middle-tier acquisition pathways have fewer requirements to report program information to offices within the Office of the Secretary of Defense and the Joint Staff than major defense acquisition programs. For example, middle-tier acquisition programs are generally exempted from the Joint Capabilities Integration and Development System for requirements development. Therefore Joint Staff officials may have less information about program requirements than for a major defense acquisition program. Office of the Secretary of Defense officials told us they are working with the military departments and other stakeholders to determine what information is needed for oversight and portfolio management for middle-tier acquisition programs. Office of the Secretary of Defense and Joint Staff officials told us that guidance issued by the Under Secretary of Defense for Acquisition and Sustainment in October 2018 that gives the Office of the Secretary of Defense and Joint Staff formal roles in a governance process may help to ensure sufficient insight. DOD’s ability to develop a common set of portfolios to facilitate integrated portfolio analysis may be more difficult. We previously reported that the requirements, acquisition, and budget communities at DOD were using different portfolio constructs, meaning that they defined their portfolios differently and did not use a standard approach to group investments into portfolios. We identified the use of different approaches as a barrier to taking an integrated approach to prioritize needs and allocate resources in accordance with strategic goals, as called for by portfolio management best practices. For example, the requirements community uses eight joint capability areas for examining warfighter needs, acquisition portfolios vary by military department, and budget data are organized into 11 major force programs. In our prior work, many officials at DOD said that using a wide variety of portfolio constructs is necessary and sometimes beneficial given the different roles and perspectives of the organizations involved. However, as notionally illustrated in figure 8, the different communities need to go through an extensive mapping exercise when they want to analyze their portfolios from another perspective—for example, examining funding associated with joint capability areas. With the reorganization of the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, officials from the Offices of the Under Secretaries of Defense for Research and Engineering and Acquisition and Sustainment told us that portfolio management activities that used to be conducted by the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics are now split between their offices. Officials from the Office of the Under Secretary of Defense for Research and Engineering told us that they were still in the process of determining what portfolio construct they would use to group investments for portfolio management purposes. If that office decides to use a different portfolio construct than other entities, that decision will increase the already complex process of mapping together portfolios in order to perform an integrated portfolio analysis. Officials from both offices told us that they were working on a pilot effort to conduct portfolio management by focusing on DOD’s missions rather than programs, which could help to standardize the portfolio constructs if the approach is accepted on a wider scale. In addition to the contact named above, Cheryl Andrew (Assistant Director), Marie Ahearn, Peter W. Anderson, David Dornisch, Anne McDonough, Melissa Pope, Scott Purdy, Juli Steinhouse, Sara Sullivan, Anne Louise Taylor, Alyssa Weir, and David Wishard made key contributions to this report.", "summary": "Amid concerns about the ability of DOD's acquisition process to keep pace with evolving threats, Congress included numerous reforms in recent National Defense Authorization Acts that could help to streamline acquisition oversight and field capabilities faster. GAO was asked to examine DOD's efforts to implement these reforms. This report addresses (1) the progress DOD has made implementing selected oversight reforms related to major defense acquisition programs; (2) how DOD has used middle-tier acquisition pathways; and (3) challenges DOD faces related to reform implementation. GAO reviewed five reforms: milestone decision authority designation; cost, fielding, and performance goals; independent technical risk assessments; restructuring of acquisition oversight offices; and middle-tier acquisition. GAO analyzed applicable statutes and implementing guidance, collected information from DOD about the number and types of middle-tier acquisition programs, reviewed relevant documentation, and interviewed DOD officials. The Department of Defense (DOD) has made progress in implementing reforms to restructure the oversight of major defense acquisition programs. As a result of one of these reforms, decision-making authority for many programs shifted from the Office of the Secretary of Defense to the military departments (see figure). Questions remain about how some reforms GAO reviewed will be carried out. For example, no programs have been required to have cost and fielding goals set under DOD's new process yet, and DOD has formed a working group to determine when to delegate risk assessments to the military departments. DOD also began using new pathways referred to as middle-tier acquisition to rapidly prototype and field new weapon systems. Middle-tier programs are expected to field capabilities within 2 to 5 years. As of March 2019, military departments were using this authority for 35 unclassified programs (see table). Source: GAO analysis of Department of Defense data. | GAO-19-439 DOD has yet to fully determine how it will oversee middle-tier acquisition programs, including what information should be required to ensure informed decisions about program selection and how to measure program performance. Without consistent oversight, DOD is not well positioned to ensure that these programs—some of which are multibillion dollar acquisitions—are likely to meet expectations for delivering prototypes or capability to the warfighter quickly. DOD also continues to face implementation challenges, including one related to disagreements about oversight roles and responsibilities between the Office of the Secretary of Defense and the military departments. Senior DOD leadership has not fully addressed these disagreements. As a result, DOD is at risk of not achieving an effective balance between oversight and accountability and efficient program management. GAO is making four recommendations, including that DOD should identify the types of information needed to select and oversee middle-tier acquisition programs consistently, and clarify the roles and responsibilities of the Office of the Secretary of Defense and the military departments for acquisition oversight. DOD concurred with GAO's recommendations and described actions planned to address them.", "document_type": "gao"}
{"report": "Signed into law on May 9, 2014, the DATA Act expands on previous federal transparency legislation. It requires a greater variety of data related to federal spending by agencies, such as budget and financial information, to be disclosed and agency spending information to be linked to federal program activities so that policymakers and the public can more effectively track federal spending through its life cycle. The act gives OMB and Treasury responsibility for establishing government-wide financial data standards for any federal funds made available to, or expended by, federal agencies. As Treasury and OMB implemented the DATA Act’s requirement to create and apply data standards, the overall data standardization effort has been divided into two distinct, but related, components: (1) establishing definitions which describe what is included in each data element with the aim of ensuring that information will be consistent and comparable and (2) creating a data exchange standard with technical specifications that describe the format, structure, tagging, and transmission of each data element. Accordingly, OMB took principal responsibility for developing policies and defining data standards. Treasury took principal responsibility for the technical standards that express these definitions, which federal agencies use to report spending data for publication on USAspending.gov. Under the act, agencies are required to submit complete and accurate data to USAspending.gov, and agency-reported award and financial information is required to comply with the data standards established by OMB and Treasury. See app. V for more information on the sources of data and process for submitting data under the DATA Act. Since the DATA Act’s enactment in 2014, we have issued a series of reports and made recommendations based on our ongoing monitoring of DATA Act implementation. In November 2017, we issued our first report on data quality, which identified issues with, and made related recommendations about, the completeness and accuracy of the Q2 FY2017 data that agencies submitted, agencies’ use of data elements, and Treasury’s presentation of the data on Beta.USAspending.gov. In addition, as part of our ongoing monitoring of DATA Act implementation, and in response to provisions in the DATA Act that call for us to review IG reports and issue reports assessing and comparing the quality of agency data submitted under the act and agencies’ implementation and use of data standards, we issued a report in July 2018, based on our review of the IG reports of the quality of agencies’ data for Q2 FY2017. Our prior reports identified significant data quality issues and challenges that may limit the usefulness of the data for Congress and the public. These data quality challenges underscore the need for OMB and Treasury to make further progress on addressing our 2015 recommendation that they establish clear policies and processes for developing and maintaining data standards that are consistent with key practices for data governance. Such policies and processes are needed to promote data quality and ensure that the integrity of data standards is maintained over time. In March 2019, we reported on the status of OMB’s and Treasury’s efforts to establish policies and procedures for governing data standards. We found that OMB and Treasury have established some procedures for governing the data standards established under the DATA Act, but a formal governance structure has yet to be fully developed. Therefore, we made recommendations to OMB to clarify and document its procedure for changing data definition standards, and to ensure that related policy changes are clearly identified and explained. For Q4 FY2018, 107 agencies, including all 24 CFO Act agencies and 83 non-CFO Act agencies, determined they were required to submit data, or they would voluntarily submit data, under the DATA Act. Of these 107 agencies, 96 submitted data for Q4 FY2018. This is an increase over the initial submissions for Q2 FY2017 when 78 agencies submitted data that covered 91 federal entities. This represents an improvement in the number of agencies reporting. However, not all the required files submitted by agencies were complete, and the data submitted were not always accurate (i.e., consistent with agency source records and other authoritative sources and applicable laws and reporting standards). In addition, we found that some CFO Act agencies did not include certain financial assistance programs that made awards during fiscal year 2018 in their submissions. Finally, some agencies continued to have challenges in reporting some data elements in accordance with standards. While the total number of agencies that submitted data for Q4 FY2018 increased compared to Q2 FY2017, more agencies submitted their data for Q4 FY2018 after the due date compared to Q2 FY2017. In addition, the data for Q4 FY2018 available on USAspending.gov are not complete because some agencies failed to submit data or submitted partial data. Fourteen agencies submitted late. Agencies were required to submit their DATA Act files for Q4 FY2018 by November 14, 2018—45 days after the end of the quarter. Eighty-two agencies submitted their data on time. These 82 agencies represented about 84 percent of the total obligations government-wide reported to Treasury on the SF 133 for Q4 FY2018. Fourteen agencies submitted their data after the November 14, 2018 due date. Our prior review of data submitted for Q2 FY2017 found that one agency submitted data after the due date. Eleven agencies did not submit data. Eleven non-CFO Act agencies did not submit any DATA Act files for Q4 FY2018. By contrast, in reviewing Q2 FY2017 data, we identified 28 agencies that determined they should have reported data under the DATA Act, but did not. Agencies told us that they did not submit data for Q4 FY2018 because (1) there was confusion or miscommunication between the agency and its shared service provider about who was responsible for reporting the data; (2) their officials had determined the agency was not required to report; (3) new staff were unfamiliar with DATA Act requirements; and (4) technical or systems issues, such as a financial system upgrade in process, prevented them from reporting their data. Multiple agencies submitted blank files. Of the 96 agencies that submitted DATA Act files for Q4 FY2018, 35 non-CFO Act agencies submitted the file that links budget and award information (i.e., File C) or the file containing procurement data (i.e., File D1) that did not contain any data (i.e., files were blank). Specifically, 34 non-CFO Act agencies submitted a blank File D1, which contains procurement data, and 16 of those 34 also submitted a blank File C. Another non-CFO Act agency submitted a blank File C only. File C data are particularly important to oversight and transparency because they link budget and award information, as required by the DATA Act. Without this linkage, policymakers and the public may be unable to effectively track federal spending because they would be unable to see obligations at the award and object class level. Agencies told us they submitted files without data for reasons including: (1) their data was submitted by and comingled with their shared service provider’s DATA Act submissions; (2) they did not have award activity to report or award activity was below the micro-purchase threshold for reporting; and (3) they do not use the Federal Procurement Data System- Next Generation or their systems were unable to produce the data necessary to create the files. We did not assess the completeness of File D1 in 2017, but we found that 13 agencies submitted a blank File C in Q2 FY2017. Of these 13 agencies, two were CFO Act agencies with large amounts of award activity —the Departments of Defense (DOD) and Agriculture (USDA)— both of which did submit a File C with data for Q4 FY2018. Two agencies submitted incomplete files. DOD and Treasury submitted all seven required DATA Act files for Q4 FY2018, but the data in some of those files were not complete. According to DOD officials, its File C submission for Q4 FY2018 included data from six of its 18 accounting systems. DOD officials said they are working to report data from all 18 systems in File C by the fourth quarter of fiscal year 2019. They said prior to Q4 FY2018, OMB granted DOD extensions for reporting financial and payment information in File C, as permitted by the act. DOD officials said the extensions allowed DOD to focus on financial statement audit readiness, build a single source tool from which File C obligation data could be aligned with procurement and grant data, and coordinate with the intelligence community on concerns over increased transparency. According to Treasury officials, the agency’s data submission did not include the spending of one of its component organizations—the Treasury Executive Office for Asset Forfeiture, Equitable Sharing Program— because OMB guidance does not allow for reporting aggregate transactions when Primary Place of Performance, a required data element, is at a multistate or nationwide level. According to Treasury officials, Treasury is working with OMB and the Treasury DATA Act Program Management Office to allow for these types of transactions to be reported. In our 2017 review, we identified similar challenges with the completeness of agencies’ DATA Act submissions for Q2 FY2017 and made recommendations to Treasury and OMB to improve the completeness of data on USAspending.gov. We recommended that Treasury reasonably assure that ongoing monitoring controls to help ensure the completeness and accuracy of agency submissions are designed, implemented, and operating as intended. Treasury agreed with this recommendation. In September 2019, Treasury officials told us that they are working to formalize a process for monitoring agency submissions that will include emailing reminders to agencies prior to submission deadlines, following up with agencies that do not submit required data on time, and forwarding a list of non-compliant agencies to OMB. We also recommended that OMB continue to provide ongoing technical assistance that significantly contributes to agencies making their own determinations about their DATA Act reporting requirements and that it monitor agency submissions. While OMB generally agreed with our recommendation, it has not yet taken steps to monitor agency submissions to help ensure their completeness. In October 2019, OMB staff told us that they believe monitoring agency submissions is not their responsibility. During this review we asked agencies why they did not submit data for Q4 FY2018. Subsequently, five of them submitted their data late (out of the initial 18 agencies that had not submitted data), demonstrating that simple monitoring tasks such as a follow up call or email can result in actions taken by the agencies. To address ongoing challenges with the completeness of agencies’ DATA Act submissions, we continue to maintain that Treasury and OMB should monitor agencies’ submissions to help ensure the completeness and accuracy of those data submissions. See app. IV for more information on the status of these recommendations. Agencies did not report awards made to 39 financial assistance programs. Seven of the 24 CFO Act agencies did not report spending for at least one financial assistance program that made awards during fiscal year 2018. File D2 contains detailed information about individual financial assistance awards. We compared the spending data reported by the 24 CFO Act agencies in File D2 against the Assistance Listings, formerly known as the Catalog of Federal Domestic Assistance (CFDA), a government-wide compendium of federal programs, projects, services, and activities that provide assistance or benefits to the American public. As of March 2019, the Assistance Listings website contained 2,926 programs for the CFO Act agencies. Of these, 39 programs (approximately 1 percent) were not included in the Q4 FY2018 DATA Act submissions, even though these agencies stated that they made reportable awards during fiscal year 2018. In comparison, in July 2017, the CFDA listed 2,219 programs for the CFO Act agencies. Of these 2,219 programs, 160 programs (approximately 7 percent) were not included in the Q2 FY2017 DATA Act submissions even though they made reportable awards. The remaining programs either reported at least one award or did not make awards that were subject to reporting. To provide a sense of magnitude of the underreporting, we obtained estimates of the total projected annual spending for these programs for fiscal year 2018 from the Assistance Listings website and applicable agencies. Based on the estimated obligations, the 39 programs account for approximately $11.5 billion in estimated annual obligations in fiscal year 2018. The omitted amounts largely resulted from USDA’s failure to report 27 programs representing more than 99 percent of the estimated annual obligations. According to USDA officials, USDA did not submit awards for some of these programs because it maintains that the information in legacy reporting systems is incompatible with the Treasury broker. USDA is working on solutions to resolve identified reporting challenges with its financial and awards systems. Treasury took steps to address findings on completeness issues for financial assistance programs we reported in 2017. At Treasury’s request, we provided details regarding the programs that were omitted from the USAspending.gov database for fiscal year 2017, which Treasury shared with the appropriate agencies. In our review of fiscal year 2018 data, we found that only nine of these programs did not report. Based on the results of testing performed on a sample of budgetary and award transactions, we found that the overall completeness within individual transactions and accuracy of the reported data was high. We selected a projectable government-wide sample of 405 transactions and tested 41 data elements and subelements associated with them for completeness and accuracy. We determined data completeness within the transaction based on whether the element included a value and whether the value was appropriate. We determined accuracy of data elements by determining consistency with agency source records as well as applicable laws and reporting standards. Specifically, based upon our sample we estimate with a 95 percent confidence level that all the data in the population were between 99 and 100 percent complete and between 90 and 93 percent accurate. We further analyzed accuracy at the transaction and individual data element levels as follows: 1. Transaction level, which describes the extent to which all applicable data elements within an individual transaction are complete and consistent with agency source records, and applicable laws and reporting standards. 2. Data element level, which describes the extent to which the data elements and subelements used for reporting budgetary and award information were consistent with agency source records and applicable laws and reporting standards. Consistency of transactions. For data submitted in Q4 FY2018, we found that the level of consistency differed between budgetary and award transactions, but both improved compared to the data we sampled for our review of Q2 FY2017 data. Based on our projectable government-wide sample of Q4 FY2018 data, we estimate with 95 percent confidence that between 84 and 96 percent of the budgetary transactions and between 24 and 34 percent of the award transactions in the USAspending.gov database were fully consistent with agency sources. We considered a transaction to be “fully consistent” if the information contained in the transaction was consistent with agency records for every applicable data element. This result represents an increase in consistency from what we reported in 2017, when we estimated that between 56 and 75 percent of budgetary transactions were fully consistent, and between 0 and 1 percent of award transactions were fully consistent. In addition to the transactions that were fully consistent, we estimate that 94 to 100 percent of budgetary transactions and 62 to 72 percent of award transactions in the population were significantly consistent. We considered a transaction significantly consistent if 90 percent or more of the data elements and subelements in the transaction were consistent with agency source records and applicable laws and reporting standards. Consistency of data elements. We also found improvements in the consistency of budgetary and award data elements with agency records, and applicable laws and reporting standards. As shown in figure 1, more data elements were significantly consistent and fewer were significantly inconsistent in Q4 FY2018 than Q2 FY2017. We considered a data element to be “significantly consistent” if the estimated consistency rate was at least 90 percent. Five of six of the budgetary data elements were significantly consistent in Q4 FY2018, compared to four of seven data elements in our 2017 review. We also found improvements in the consistency of award data elements and subelements compared to our 2017 review. Eighteen of the 35 award data elements and subelements in our sample were significantly consistent in Q4 FY2018, compared to only one of 26 data elements and subelements we tested in our 2017 review. See figure 2 for the data elements and subelements in our sample that were significantly consistent. We considered a data element “significantly inconsistent” if it was either not consistent with agency records or incomplete at least 10 percent of the time. We found that no budgetary data elements were significantly inconsistent, which is an improvement from our 2017 review where we found one budgetary data element—Obligation—significantly inconsistent. Similarly, we found fewer significantly inconsistent award data elements compared to our 2017 review. Specifically, we found five of 35 award data elements and subelements significantly inconsistent in Q4 FY2018, compared to 11 of 26 in our 2017 review. See figure 3 for the data elements and subelements in our sample that were significantly inconsistent. Unverifiable data elements. We found no data elements that exhibited a significant amount of unverifiable information—incomplete or inadequate agency source records that prevented us from determining whether the data element was significantly consistent or inconsistent. See app. III for details. While we tested the consistency of agency records and applicable laws and reporting standards for the 41 data elements and subelements previously discussed, we performed a different test for three other data elements that contained a value derived by FPDS-NG and FABS. These data elements and subelements—Legal Entity County Name, Primary Place of Performance County Name, and Primary Place of Performance Congressional District—were assessed against the other sources from which they were derived, such as data from the U.S. Census Bureau and house.gov, rather than agency records. We found that each were neither significantly consistent nor significantly inconsistent with their sources. See appendix III, table 5 for details. The DATA Act requires OMB and Treasury to establish data standards to produce consistent and comparable reporting of federal spending data. While we found improvements in the overall completeness and accuracy of the data when compared with the results of our 2017 review, we identified persistent challenges with the implementation and use of two award data elements—Award Description and Primary Place of Performance Address that limit the usefulness of these data. We previously reported that these data elements are particularly important to achieving the transparency goals envisioned by the DATA Act because they inform the public what the federal government spends money on and where it is spent. In our sample results, we found agencies reported values for Award Description that were significantly inconsistent with agency sources and with the established standard for reporting this data element which is defined by the DATA Act data standard as a “brief description of the purpose of the award.” Based on our testing of a representative sample of Q4 FY2018 transactions, we estimate that the Award Description data element was inconsistent with agency source records or contained information that was inconsistent with the established standard in 24 to 35 percent of awards. While this represents an improvement over the results we reported for this data element in 2017, we found in our testing that agencies continue to face challenges in reporting Award Description consistent with the established standard. See figure 4 for several examples of the Award Description data submitted by agencies in our sample, which illustrates the range of agency interpretations of this data element from understandable to incomprehensible. Lengthy, technical description. For example, the National Aeronautics and Space Administration (NASA) included several paragraphs for the description of procurement and financial assistance award transactions in our sample that were long and highly technical. These descriptions did not meet the data standard because they contained acronyms, jargon, and other technical terminology that might be challenging for others outside the agency to understand. NASA officials said they use the Award Description field internally to search for vendors when making awards for similar services. Thus, they instructed contract officers to include as much information as possible to maximize the Award Description field for later use. As of June 2019, the General Services Administration decreased the character limit for reporting Award Description in FPDS-NG for procurement awards from 4,000 characters to 250 characters to discourage agencies from copying and pasting sizeable portions of a contract’s contents rather than thoughtfully including a brief description of what is being procured. NASA officials said that the new maximum will limit the flexibility to search for contractors. They are seeking alternatives for these searches. No description provided. The Department of Education reported “unknown title” for the Award Description for the majority of the financial assistance award transactions in our sample. This does not meet the data standard because it does not provide any information about the award. Agency officials said the Award Description is provided by the applicant and if one is not provided, their system automatically will populate it as “unknown title.” Geographic information. DOD reported location information for the Award Description in several transactions in our sample. The locations reported in the description field were not understandable except to agency officials. For example, one field contained the text “4542874050!TRBO REGION 1.” DOD officials explained that this description includes the part number for a medical supply item and the region of the country and is auto populated by an agency system. While the description is consistent with agency sources, it is not easily understood by the public. The Defense Federal Acquisition Regulation Supplement Procedures, Guidance, and Information provides instructions to use plain English as much as possible, and to explain numbers and acronyms. DOD officials said the agency is investigating methods to improve how similar transactions are auto-populated. Description of modification. The Department of Homeland Security (DHS) used the Award Description field to describe modifications to contracts instead of the good or service being procured. Specifically, DHS reported “de-obligate excess funds and closeout” for a modification to a contract that procured information technology products and services. DHS officials said reporting the nature of the modification, rather than the original purpose of the award, is consistent with practices used in contract writing systems across the federal government and is intended to inform the public of changes made to the contract by the modification. DHS is working with Treasury to clarify how this information is displayed on USAspending.gov and suggested that additional information on how award descriptions for modifications are to be reported would be beneficial and should be provided in the DAIMS. We found that some individual agencies have taken steps to provide additional guidance on Award Description to ensure agency personnel are providing information that is consistent with the standard. Four agencies in our sample had additional guidance for their contracting officers. For example, officials from the Department of Veterans Affairs (VA) said that in June 2019, VA trained hundreds of members of its contracting workforce with curriculum that included an interactive game to illustrate how to provide a brief description of an award that meets the standard for reporting this information. Officials from 11 agencies said additional guidance on Award Description could help ensure those entering the data understand the standard definition and report appropriate information, for example, by providing examples of award definitions that meet the standard. In the absence of government-wide guidance, agencies have reported values that are inconsistent with the data standard and not comparable between agencies. Agencies also reported several challenges with reporting Primary Place of Performance Address for nonroutine locations, which OMB and Treasury defined as “where the predominant performance of the award will be accomplished.” Taking into account each of its subelements, we found the information regarding Primary Place of Performance Address had higher rates of inconsistency than the majority of the data elements in our review. Multiple subrecipients. Agency officials reported challenges with identifying Primary Place of Performance Address in cases where an award is made to a recipient that further distributes the funding to subrecipients. For example, the U.S. Agency for Global Media (USAGM) awards Radio Free Europe/Radio Liberty a grant that funds work globally. Officials from USAGM said that as a U.S. not-for-profit organization, Radio Free Europe/Radio Liberty, maintains corporate headquarters in Washington, D.C., but, as an international media organization, maintains many offices abroad. USAGM reports the Primary Place of Performance Address as Washington, D.C. because it is where the organization maintains its corporate office, but much of the performance takes place in other locations. In another example, the Department of Health and Human Services’ (HHS) Centers for Medicare and Medicaid Services (CMS) reports the Primary Place of Performance Address for Medicare payment data as the county of its payment processing centers, even though each processing center makes payments to recipients in multiple states and counties. CMS contracts with Medicare Administrative Contractors (MAC) to process and pay Medicare fee-for-service claims. For each type of Medicare claim, the number of jurisdictions and the number of MACs that handle that type of claim vary. At the time of our review, there were 12 jurisdictions for Medicare Part A and B claims handled by MACs. As shown in figure 5, the jurisdictions are made up of multiple states. In addition to the MAC jurisdictions for Medicare Part A and Part B claims, there were four home health and hospice jurisdictions and four durable medical equipment jurisdictions. Thus, there are 20 MAC jurisdictions, almost all of which covered multiple states. As a result, the spending for Medicare payments is reported in a small number of counties instead of where the beneficiaries of Medicare services are located. Software. Officials from three agencies in our review said that it is challenging to determine Primary Place of Performance Address for software licenses when purchased as a service. For example, there could be multiple performance locations, but none of these locations are predominant. Large or undefined locations. Officials from the agencies in our review reported challenges in meeting the standard for reporting large or undefined performance locations. For example, officials from the Delta Regional Authority said that it was difficult, at times, to determine the Primary Place of Performance Address for watersheds because they can cover a large area and cross multiple jurisdictions. Officials from the National Science Foundation (NSF) said that for projects that may not have a single location, they report the location that corresponds to the research asset’s physical location or the primary site. For example, for a research vessel, NSF officials report the awardee’s address, which is generally the vessel’s homeport as the Primary Place of Performance Address. In another example, NASA officials said that when they let contracts for services performed on the International Space Center, they report the command center in Houston as the Primary Place of Performance Address. For some of these non-routine locations, the FPDS-NG data dictionary provides guidance for procurement transactions. For example, for services being performed in oceans and seas, it directs agencies to report the closest U.S. city. For services being performed in the atmosphere or space, the FPDS-NG Data dictionary directs agencies to report the location from which the equipment conducting the services was launched. However, the DATA Act Information Model Schema (DAIMS) Data Dictionary does not include the same level of detailed guidance for reporting financial assistance awards and directs agency officials to report the location where the predominant performance of the award will be accomplished. Officials from several agencies said it would be helpful for OMB and Treasury to issue guidance on Primary Place of Performance Address for financial assistance awards to help agencies report this information consistent with the established standard. In the absence of more specific guidance, agencies are using different decision rules to identify the Primary Place of Performance Address for financial assistance awards which could limit the usefulness of this information to the public. We previously identified similar issues with Award Description and Primary Place of Performance Address on USAspending.gov. We recommended that OMB and Treasury provide agencies with additional guidance to address potential clarity, consistency, or quality issues with the definitions for specific data elements including Award Description and Primary Place of Performance Address and that they clearly document and communicate these actions to agencies providing these data as well as to end-users. OMB issued guidance in June 2018 which provides clarification on reporting requirements for some data element definitions. However, additional guidance is needed to clarify how agencies are to report spending data using standardized data element definitions that may be open to more than one interpretation, and then broadly communicate this information to agencies and the public. We continue to believe additional guidance is needed to facilitate agency implementation of certain data definitions to produce consistent and comparable information. Given the challenges we identified in this report and in previous reports with Award Description and Primary Place of Performance Address, we have concerns about whether the guidance OMB issued provides sufficient detail for agencies to consistently interpret and implement the definitions. See app. IV for more information on the status of this recommendation. Treasury does not fully disclose all known data limitations on USAspending.gov. According to OMB guidance, federal agencies should be transparent about the quality of information and identify the limitations of the data they disseminate to the public. Further, Treasury’s Information Quality Guidelines state that, when disseminating information to the public, information should be presented within the proper context to disseminate information in an accurate, clear, complete, and unbiased manner. In November 2017, we identified data quality limitations that were not disclosed on USAspending.gov. We recommended that Treasury disclose known data quality issues and limitations on USAspending.gov. Treasury agreed with this recommendation and has taken steps to better disclose some of these limitations, but many of the issues we identified in 2017 continue to present challenges. Some of these challenges apply widely, while others were specific to particular agencies. They include the following: Data not submitted or incomplete. One step taken by Treasury to improve disclosure was to create a webpage in USAspending.gov that provides information on unreported data. However, it is unclear exactly what this information covers. For example, it is unclear whether the information on unreported data includes financing accounts, agencies that should have reported but did not submit data, missing data for agencies that did submit, or spending that was not reported because obligation amounts fell below $25,000 and was therefore not required to be reported. As a result, users do not clearly know what data are unreported or the amount that was required to be reported. Optional data elements and subelements. Another issue we identified in 2017 and found again in our current review was that key information about the reporting requirements for some data elements and subelements was not adequately disclosed to the public. Specifically, for Q4 FY2018 certain data elements were listed in guidance as optional for agencies to report. According to Treasury officials, agencies were not required to report these data elements because the data standard was not fully implemented. For example, prior to fiscal year 2019, the data element Funding Office Name was optional for financial assistance awards. Additionally, as of September 2019, Period of Performance Start Date and Period of Performance Current End Date remained optional for reporting pending government-wide agreement on the standard. USAspending.gov does offer some information regarding optional data elements by providing a link to the DAIMS Reporting Submission Specifications document. However, this document is not labeled in a way that would make it clear to the user what information can be found there. Moreover, some agencies may voluntarily submit data for optional fields so only partial information for optional data elements may be displayed on USAspending.gov. Because data limitations related to optional data elements are not prominently displayed on USAspending.gov, users may not know which data elements or subelements are potentially incomplete. A more systematic approach for identifying and disclosing known data limitations on USAspending.gov—including procedures for addressing wide ranging issues such as communicating changes in the reporting requirements for certain data elements and information about data that may be unreported or incomplete—could help users of the data better understand potential quality issues with particular data elements and sources, and how to appropriately interpret the data. While Treasury has taken steps to better disclose data limitations, it needs to take further action to implement a more systematic approach, in line with our 2017 recommendation. In addition to such broader challenges, we identified two specific data limitations involving DOD and HHS: Delay in availability of DOD procurement data. A third issue we identified in our 2017 review, and again in our current review, concerns how information on DOD procurement data is presented on USAspending.gov. Specifically, information related to a 90-day delay in data availability for DOD procurement awards is not posted on USAspending.gov. FPDS-NG—which collects information on contract actions for display on USAspending.gov—releases DOD-reported procurement data to the public after a 90-day waiting period to help ensure the security of these data before they are released to the public. This also results in a 90-day delay in reporting these data to USAspending.gov. FPDS-NG clearly states that DOD data are subject to a 90-day delay as seen in figure 6. While DOD reports this data limitation in its senior accountable official certification statement, it is not presented prominently to users who are viewing DOD’s spending data. For example, DOD’s delay in data availability is not presented on DOD’s agency profile page or with queries on specific transactions associated with DOD. Until such information is transparently communicated, users of USAspending.gov who access DOD procurement data directly or as a result of broader government-wide searches are likely unaware that the information may be incomplete or not comparable. Medicare payment data. Additionally, in this review we found limitations in how Medicare payment data are made available to the public. According to HHS officials, CMS reports the Primary Place of Performance Address for Medicare payment data as the county for the applicable Medicare Administrative Contractor (MAC) because the MAC is the direct recipient of the agency’s contract award. As a result, Medicare spending data on USAspending.gov are not reported in the county where the Medicare beneficiaries are located. There are more than 3,200 counties and county equivalents in the United States and Puerto Rico, but only 20 Medicare MAC jurisdictions. Although Medicare payments may reach every county in the country, the users of USAspending.gov will only see this spending in the counties in which a MAC is located. We found that this information is not described on USAspending.gov. HHS officials said that they identified this limitation to the transparency of Medicare payment data to Treasury in 2016. They suggested that Treasury add information about how Medicare payments are reported on USAspending.gov to avoid confusion for users of the data. However, at that time, Treasury determined that it was unnecessary to provide this additional information on USAspending.gov. Until such information is transparently communicated, it will be unclear to the user that Medicare payments are consolidated in the counties where MACs are located. One of the purposes of the DATA Act is to establish government-wide data standards to provide consistent and comparable data that are displayed accurately for taxpayers and policymakers on USAspending.gov. As we have reported previously, establishing a data governance structure—an institutionalized set of policies and procedures for providing data governance throughout the life cycle of developing and implementing data standards—is critical for ensuring that the integrity of data standards is maintained over time. Such a structure, if properly implemented, would greatly increase the likelihood that the data made available to the public will be accurate. Accordingly, in 2015, we recommended that OMB, in collaboration with Treasury, establish a set of clear policies and procedures for developing and maintaining data standards that are consistent with leading practices for data governance. This recommendation has not been implemented. Having formalized policies and procedures in place for one of these key practices—managing, controlling, monitoring, and enforcing the consistent application of data standards once they are established—could help address some of the data quality challenges we identified in this and previous reviews. As described earlier, agencies experience challenges reporting Award Description and Primary Place of Performance Address. We continue to believe that having a robust data governance structure that includes policies and procedures for enforcing the consistent application of the established standards would lead to greater consistency and comparability of reporting for data elements, such as Award Description and Primary Place of Performance Address. OMB and Treasury have established some procedures for governing the data standards established under the DATA Act, but a robust governance structure has yet to be fully developed and operational. Since the enactment of the DATA Act in 2014, OMB has relied on a shifting array of advisory bodies to obtain input on data standards. In March 2019, we reported that the governing bodies involved in initial implementation efforts had been disbanded, and that their data governance functions were to be accomplished within the broader context of the cross-agency priority (CAP) goals established under the 2018 President’s Management Agenda (PMA). Since we issued our report, OMB has taken additional steps to develop a government-wide data structure and to establish data governance programs at each agency. OMB staff told us that they envision agencies as incubators of data governance where they can learn lessons on data governance. Toward that end, OMB, in collaboration with other interagency groups, has taken a number of steps to further develop data governance at both the agency and government-wide levels: In October 2019, OMB issued a set of grants management data standards under the Results Oriented Accountability for Grants CAP Goal. According to OMB staff, they received more than 1,100 public comments on draft standard data elements which were released for public comment in November 2018. OMB issued a memorandum in April 2019 that outlines approaches to shared services and the governance structure established to support shared services used for data reporting. In June 2019, as part of the CAP Goal Leveraging Data as a Strategic Asset, OMB issued the draft 2019-2020 Federal Data Strategy Action Plan (Action Plan). This document identifies both government-wide and agency-level action steps for improving data governance. To address government-wide data governance, the Action Plan calls for improvement in the standards for financial management data and geospatial data. The Action Plan directs agencies to establish a body of internal stakeholders responsible for data governance. These bodies will be made up of senior level staff and be responsible for assessing agency capability and ensuring monitoring and compliance with policies and standards related to data. Agencies are also instructed to assess data and related infrastructure maturity, identify opportunities to increase staff data skills, and identify data needs to answer key agency questions. OMB also issued initial guidance in July 2019 to support agency efforts to implement the first phase of the Evidence Act. For example, the Evidence Act requires, among other things, agencies to designate a Chief Data Officer by July 13, 2019. OMB also guidance directs agencies to establish a data governance body, chaired by the Chief Data Officer, with participation from relevant senior-level staff from agency business units, data functions, and financial management by September 30, 2019. In July 2019, the Federal Data Strategy Team issued a data governance playbook. According to OMB officials, this playbook is not guidance, but is meant to be a framework for agency-level data governance accompanied by forthcoming resources. OMB staff told us that updates to the playbook would come relatively quickly, but also said they had no planned time frames for doing so. Agencies have taken initial steps to establish data governance programs and develop data quality plans. As of September 2019, seven of the 30 agencies included in our review reported that they have taken steps to designate a Chief Data Officer as required by the Evidence Act. Twenty reported establishing internal bodies similar to the data governance bodies as directed by OMB guidance. The make-up and function of data governance bodies varies across agencies. The Department of Labor reported its Data Board was formalized and that the acting Chief Data Officer had become the official Chief Data Officer. The U.S. Agency for International Development reported establishing a DATA Act Governance Council to facilitate the effective implementation of the DATA Act. Other agencies reported similarly structured bodies referred to as working groups, steering committees, and consortiums. As of September 2019, 19 agencies reported that they have completed a data quality plan as required by OMB Memorandum, M-18-16. Nine agencies that do not have a data quality plan will have one completed by September 30, 2019. The data quality plans from the agencies in our sample varied in scope and content. Features of data quality plans we reviewed included a description of a data governance board, an assessment of existing and planned internal controls for data quality, and determination of priority data elements based on assessments of risk of data quality issues. For example, the Departments of Commerce and the Interior each conducted a risk assessment on the likelihood and consequence of improper reporting for assistance and procurement data. They will employ strategies or controls to mitigate risks related to the highest risk elements. Similarly, Treasury named targeted data elements based on their relevancy and further assessed the risk of improper reporting of each element based on existing internal controls. Agencies in our review reported using a variety of sources of guidance in developing their data quality plans, including the Data Quality Playbook issued by the Leveraging Data as a Strategic Asset Working Group in November 2018, OMB Circular M-18-16, and guidance on conducting required reviews under the DATA Act from the Council of inspector general for Integrity and Efficiency. While some agencies in our review reported that the information from these sources was helpful, they also noted the need for additional guidance, including help understanding the reporting requirements for certain data elements. In the 5 years since enactment, OMB, Treasury, and federal agencies have made significant strides to address many of the policy, technical, and reporting challenges presented by the DATA Act’s requirements. We found improvements in the overall quality of the data on USAspending.gov compared to our 2017 review of data quality. To continue moving forward with this progress and to fully realize the DATA Act’s promise of helping to improve data accuracy and transparency, more needs to be done to address continued challenges with the completeness and accuracy of key data elements. For example, OMB and Treasury have not fully addressed our recommendations to monitor agency submissions and ensure agencies are accountable for the completeness and accuracy of their data submissions. In addition, without the transparent disclosure of known data limitations, users may view, download, or analyze data made available on the website without full knowledge of the extent to which the data are timely, complete, accurate, or comparable over time. This could lead users to inadvertently draw inaccurate information or conclusions from the data. We have previously recommended that Treasury disclose known data limitations on USAspending.gov. The agency has taken some steps toward this goal. However, as we have shown, work remains for Treasury to develop a more systematic approach for disclosing known data limitations on its website. In the meantime, we believe it is important to address the specific data limitations we identify in this report. These include the need to provide users with information about the delay in the availability of DOD procurement data, and how Medicare payment data are reported. Finally, the challenges we have found with data completeness and accuracy, and the transparency around data limitations also demonstrate the importance of continued progress by OMB and Treasury in addressing our previous open recommendations to develop a robust and transparent data governance structure, and implement controls for monitoring agency compliance with DATA Act requirements. We maintain that OMB and Treasury should address our prior recommendations on DATA Act implementation, including recommendations on monitoring agency submissions, providing additional guidance on reporting established data standards, implementing a systematic approach to facilitate the disclosure of known data limitations on USAspending.gov, and developing a robust and transparent governance structure. We are making a total of two new recommendations to Treasury regarding the disclosure on USAspending.gov of specific known data limitations: The Secretary of the Treasury should ensure that information about the 90-day delay for displaying DOD procurement data on USAspending.gov is transparently communicated to users of the site. Approaches for doing this could include prominently displaying this information on the DOD agency profile page, in the unreported data section, and in search results that include DOD data. (Recommendation 1) The Secretary of the Treasury should ensure that information regarding how the Primary Place of Performance Address for Medicare payment data are reported is transparently communicated to the users of USAspending.gov. (Recommendation 2) We provided a draft of this report to the Departments of Agriculture (USDA), Defense (DOD), Commerce, Education, Health and Human Services (HHS), Homeland Security, the Interior (DOI), Labor (DOL), the Treasury, and Veterans Affairs (VA); the Office of Management and Budget (OMB); the National Science Foundation (NSF); the National Aeronautics and Space Administration (NASA); the Small Business Administration (SBA); the U.S. Agency for International Development (USAID); the U.S. Agency for Global Media (USAGM); and the Delta Regional Authority (DRA) for review and comment. USAID and Treasury provided written responses, which are summarized below and reproduced in appendixes VII and VIII, respectively. DHS and OMB provided technical comments, which we incorporated as appropriate. USDA, DOD, Commerce, Education, HHS, DOI, DOL, VA, NSF, NASA, SBA, USAGM, and DRA had no comments on the draft report. In its written comments, USAID stated that it is committed to DATA Act reporting and the accessibility and transparency of its spending data. In its written comments, Treasury stated its commitment to fully realizing the DATA Act’s promise of helping to improve data accuracy and transparency. Treasury agreed with our two recommendations on the disclosure of specific known data limitations and stated that it will work with HHS and DOD to implement them in the coming months. Treasury also stated that it remains committed to fully implementing our prior recommendations on DATA Act implementation. We are sending copies of this report to the relevant congressional committees; the Secretaries of Agriculture, Defense, Commerce, Education, Homeland Security, the Interior, Labor, the Treasury, and Veterans Affairs; the Directors of the Office of Management and Budget and the National Science Foundation; the Administrators of National Aeronautics and Space Administration, the Small Business Administration, and U.S. Agency for International Development; the Chief Executive Officer of the U.S. Agency for Global Media; the Chairman of the Delta Regional Authority; and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff has any questions about this report, please contact Michelle Sager at (202) 512-6806 or sagerm@gao.gov or Paula M. Rascona at (202) 512-9816 or rasconap@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. Key contributors to this report are listed in app. IX. National Science Foundation Nuclear Regulatory Commission (NRC) File B (Budgetary) File D1 (Procurement) The Broadcasting Board of Governors changed its name to the U.S. Agency for Global Media in August 2018. The Digital Accountability and Transparency Act of 2014 (DATA Act) requires that we report on the timeliness, completeness, accuracy, and quality of the data submitted under the act and the implementation and use of data standards. This review responds to the act’s requirement by addressing the following: (1) the timeliness, completeness, accuracy, and quality of the data and the implementation and use of data standards; and (2) the extent to which progress has been made to develop a data governance structure consistent with key practices, and how it affects data quality. We also update the status of select implementation issues and our previous recommendations related to implementing the DATA Act and data transparency. To assess the timeliness, completeness, accuracy, and quality of the data submitted and the implementation and use of data standards, we analyzed agency submission files for the fourth quarter of fiscal year 2018 (Q4 FY2018) on USAspending.gov and reviewed a representative stratified random sample from the Department of the Treasury’s (Treasury) USAspending.gov database download for Q4 FY2018. Specifically, to assess timeliness, we accessed agency submission files on USAspending.gov for Q4 FY2018 and determined whether agencies submitted their data by the established deadline—45 days after the end of the quarter or November 14, 2018—based on the date agencies certified their submissions. To help understand the proportion of spending that agencies reported by the due date, we obtained and analyzed a file from Treasury containing SF 133 Report on Budget Execution and Budgetary Resources (SF 133) data—which includes unaudited balances reported by agencies—for Q4 FY2018. These obligation balances are only used for illustrative purposes in our report. They include financing accounts, among other things, which are not required to be reported under the DATA Act. To assess completeness, we determined whether (1) all agencies that determined they are required to or would voluntarily submit DATA Act files did so, (2) the transactions reported in the files submitted by agencies contained all required data for that transaction, and (3) the database contained required assistance award data from the 24 Chief Financial Officers Act of 1990 (CFO Act) agencies. To determine whether all agencies that should have reported Q4 FY2018 data did so, we compared Treasury’s list of agencies that determined they were required to or would voluntarily report data to the agency file submissions on USAspending.gov for Q4 FY2018. We followed up with agencies that had not reported to find out the reasons for not reporting, but we did not verify the accuracy of their responses. To assess the completeness of files submitted by agencies, we accessed the agency submission files for Q4 FY2018 available on USAspending.gov and determined whether all files for each agency contained data (i.e., were not blank). We followed up with agencies that submitted a blank File C and/or File D1 that did not contain any data to find out why the files were blank, but we did not verify the accuracy of their responses. We also made inquiries of agencies to determine whether any agency components or systems did not submit data. Finally, we tested completeness of agency submissions through our sample testing, described in detail below. To assess the completeness of assistance data in the USAspending.gov database, we determined the extent to which federal agencies were reporting required award data based on a list of potential award-making agencies/programs from Assistance Listings on beta.SAM.gov, formerly the Catalog of Federal Domestic Assistance. We identified all programs listed in the Assistance Listings, as of September 2018. For the 24 CFO Act agencies only, we compared programs listed in the Assistance Listings to data in the USAspending.gov database to determine which programs reported information on at least one assistance award for fiscal year 2018. For any program reporting no assistance award information for the year, we asked agency officials why information was not reported. For all programs that agency officials determined either made an award but did not report it, or reported awards late to USAspending.gov, we extracted the agencies’ obligation estimates for fiscal year 2018 as reported in the Assistance Listings. To further assess completeness of the data and to assess accuracy of the data and the implementation and use of data standards, we extracted all records included in the scope of our review from a database used to display data on USAspending.gov. The records covered activity during Q4 FY2018 (July through September 2018). To extract all records from the database, we mapped the database fields to the data elements within the scope of our audit. Once we had the data within the scope of our audit for Q4 FY2018, we performed the following steps: Sampling data to determine completeness and accuracy: From the database we extracted, we selected a stratified random probability sample of 405 records for Q4 FY2018. Data records were stratified into procurement award transactions, assistance award transactions, and budgetary records. We randomly selected 158 procurement awards, 150 financial awards, and 97 budgetary records. Estimates for the results of the procurement, assistance, and budgetary samples have sampling errors of +/- 7.8, 8, and 10 percentage points or less, respectively, at the 95 percent level of confidence. The probability sample was designed to estimate the overall rate of reporting errors for a data element with a sampling error of no greater than plus or minus 5.3 percentage points at the 95 percent level of confidence. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval (e.g., +/- 7 percentage points). This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. For 41 data elements and subelements required by FFATA or the DATA Act, we first assessed the extent to which a data element was complete—whether there was a value and if that value was appropriate. If the data element was not complete, then we also considered that data element to not be accurate. For those elements that were complete, we then assessed the extent to which the data were accurate by comparing the information in our sample to the information contained in the originating agency’s underlying source documents, where available, and determining whether the data were consistent with applicable laws and reporting standards, as applicable. Therefore we determined an element was inconsistent if it was either inconsistent with the agency documents, applicable laws or reporting standards, or incomplete. For three data elements that contained values derived by Federal Procurement Data System-Next Generation (FPDS-NG) and Financial Assistance Broker Submission (FABS) based on other values provided by agencies, we compared the information in the sample to other sources, such as data from the U.S. Census Bureau and house.gov. This allowed us to verify whether the values in our sample were consistent with the systems from which they were derived. We then interviewed agency officials to discuss differences between the information in our sample and information in agency or other sources. Data element and subelement testing: Table 3 shows the 44 data elements and subelements tested in the statistical sample—including six budgetary data elements and 38 award data elements and subelements. Individual data elements may vary with their representation in the sample (e.g. Legal Entity Address Lines 1 and 2) because the data element was not required for all of the sampled data records. Specific error rates by category can be found in app. III. The government-wide results are a weighted total of the three strata of our sample: (1) procurement award transactions, (2) assistance award transactions, and (3) budgetary records. For reporting purposes, we combined some of the results for the award strata because some data elements appear in both Files D1 (procurement) and D2 (financial assistance). See app. I for the list of agencies and number of records randomly selected and tested in each strata. If we determined, after reviewing agency source documents, that a data element was not applicable to the sampled record, we did not factor the data element into our evaluation of completeness and accuracy. We determined an element to be unverifiable if no agency source records were provided or the records provided did not meet our audit standards. To test the controls over the reliability of agency data, we obtained supporting documentation to confirm that the agency provided only official agency source documents, such as a system of records notice. When such a supporting document was unavailable, we reviewed agency transparency policy documentation, data verification and validation plans or procedures, or system source code information to ensure the reliability of the data. We did not assess the accuracy of the data contained in sources provided by agencies. For the purposes of our review, we defined data quality as encompassing the concepts of timeliness, completeness, and accuracy. Therefore, our assessment of overall data quality is reflected in our specific assessments of these components. We also reviewed OMB, Treasury, and agency documents related to DATA Act implementation. We interviewed OMB and Treasury officials on their role in DATA Act implementation and interviewed officials from the agencies in our sample to discuss their test results and efforts to submit data under the DATA Act. To describe changes in data quality since our prior work, we compared the results of our review of Q4 FY2018 data to the results of our review of quarter two fiscal year 2017 (Q2 FY2017) data performed in our first assessment of data quality. For both reviews, we examined a projectable sample of budgetary and award transactions from a database that, according to Treasury, is partly used to display data on USAspending.gov. However, there were the following differences: (1) our 2017 sampling frame was confined to the 24 CFO Act agencies (which represented 99 percent of obligations in our data set at that time), while our sampling frame for this review included all agencies that submitted Q4 FY2018 data files as of February 11, 2019; (2) more agencies and their components reported data in Q4 FY2018 than in Q2 FY2017; (3) in 2017 our estimated error rate calculations included elements of certain sampled transactions that were determined to be not applicable to the transaction and were classified as consistent with agency sources in both the numerator and denominator while in this review, we excluded not applicable elements from both the numerator and denominator of the estimated rate calculations; (4) our sampling frame for this review included more data elements and subelements than were in our Q2 FY2017 sampling frame; (5) in this review, since three data elements we reviewed were derived by FPDS-NG and FABS rather than provided by agencies, we compared the information in the sample to other sources rather than agency documents and therefore did not include those results in our comparisons to Q2 FY2017; (6) agencies’ Q4 FY2018 data were submitted under policies and procedures outlined in DAIMS v1.3 which reflects changes in validation rules and reporting requirements from the DAIMS v1.0 that was in effect in 2017; (7) OMB issued additional guidance on DATA Act reporting since we reported in 2017; and (8) changes were made to the Treasury broker since our last report. To evaluate how the current data governance structure affects data quality, we compared data quality challenges we identified during our review to key practices for data governance identified in our prior work to underscore the need for a more robust structure consistent with key practices. To assess progress made to develop a data governance structure consistent with key practices, we reviewed policy and other documentation related to ongoing efforts to develop a government-wide structure for governing the standards established under the act and interviewed OMB staff about these efforts. We also reviewed agency data quality plans—guidance intended to facilitate agency efforts to establish data governance programs—and interviewed agency officials on their data governance efforts. To update the status of our recommendations related to the implementation of the DATA Act, we reviewed new guidance and other related documentation, and interviewed OMB staff and Treasury officials. See app. IV for an update on our recommendations related to DATA Act implementation. We conducted this performance audit from November 2018 to November 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix III: Estimates of Consistency Rates for Award Transactions and Budgetary Accounts/Balances Accurate/consistent (%) Q4 FY2018 Q2 FY2017 Q4 FY2018 Q2 FY2017 Q4 FY2018 Q2 FY2017 3-8 97-100 83-91 0-1 5-11 0-3 Inconsistent (%) Catalog of Federal Domestic Assistance Number (CFDA) Inconsistent (%) Data element Primary Place of Performance Address (all subelements) Accurate/consistent (%) Unverifiable (%) Legal Entity Address City Name refers to two subelements under DAIMS v.1.3 (Legal Entity Address City Name and Foreign City Name), which we combined for reporting purposes. Legal Entity Address State Name refers to three subelements under DAIMS v.1.3 (Legal Entity Address State Description for procurement awards and Legal Entity Address State Name and Foreign Province Name for financial assistance awards), which we combined for reporting purposes. Legal Entity Address Zip Code refers to four subelements under DAIMS v.1.3 (Legal Entity Address Zip+4 for procurement awards, Legal Entity Address Zip 5 and Last 4 for financial assistance awards, and Legal Entity Address Foreign Postal Code for foreign financial assistance awards), which we combined for reporting purposes. Primary Place of Performance Address Zip Code is one subelement under DAIMS v.1.3 (Primary Place of Performance Address Zip+4), which contains both the first five digits from the zip code and the last 4. However, the USAspending.gov database we obtained our sample from contained the zip code information for this element in two parts: 5 digit zip code and +4. Therefore, we present these subelements separately for reporting purposes. Element was optional for fourth quarter of fiscal year 2018. Unverifiable includes data elements rates as inaccurate because agency records were insufficient to complete the test or because the agency did not provide supporting documentation. Accurate/consistent (%) Estimated ranges Inconsistent (%) Unverifiable (%) In our prior Digital Accountability and Transparency Act of 2014 (DATA Act) reports, we have made recommendations to both the Department of the Treasury (Treasury) and the Office of Management and Budget (OMB) on a range of topics. Treasury and OMB have collectively taken action that resulted in closure of nine prior recommendations on the data transparency and implementation of the DATA Act. Table 7 provides a listing of open DATA Act recommendations at the time this report was issued as well as a short discussion of their status. Full and effective implementation of the open recommendations listed below will contribute to more reliable and consistent federal data to measure the cost and magnitude of federal investments as well as facilitate efforts to share data across agencies to improve transparency, accountability, decision- making, and oversight. The Digital Accountability and Transparency Act of 2014 (DATA Act) requires the Office of Management and Budget (OMB) and the Department of the Treasury (Treasury) to establish government-wide data standards that to the extent reasonable and practicable produce consistent, comparable, and searchable spending data for any federal funds made available to or expended by federal agencies. These standards specify the data elements to be reported under the DATA Act and define and describe what is to be included in each data element, with the aim of ensuring that data will be consistent and comparable. The DATA Act requires OMB and Treasury to ensure that the standards are applied to the data made available on USAspending.gov which has many sources of data. Some data are from agency systems, while other data are pulled or derived from government-wide reporting systems. Key award systems that generate data files that are linked to agency submitted files include the Federal Procurement Data System-Next Generation (FPDS-NG), which collects information on contract actions; the Financial Assistance Broker Submission (FABS) which collects information on financial assistance awards; the System for Award Management which is the primary database for information on entities that do business with the federal government (i.e., contractors and grantees), and in which such entities must register; and the Federal Funding Accountability and Transparency Act of 2006 (FFATA) Subaward Reporting System (FSRS), which provides data on first-tier subawards reported by prime award recipients. Agencies submit procurement award information to FPDS-NG daily and financial assistance award information (grants, loans, insurance and other financial assistance) to FABS at least twice monthly. These award data are reflected in USAspending.gov daily. As depicted in figure 7, agencies are expected to submit financial data linked to award data and certified on a quarterly basis, 45 days after the close of the quarter. They submit three data files with specific details and data elements to Treasury’s DATA Act Broker (broker) from their financial management systems quarterly (Files A, B, C). In February 2019, to reduce agency burden, Treasury made updates including an optional new broker feature that agencies can use to generate a provisional File A which agencies can choose to upload and submit as their File A in the regular submission process. The new feature produces an agency’s provisional File A based on budget and financial information reported by the agency to the Government-wide Treasury Account Symbol Adjusted Trial Balance System for the creation of the SF 133 Report on Budget Execution and Budgetary Resources. The broker then extracts award and subaward information from existing government-wide reporting systems to build four files that include procurement information, information on federal assistance awards such as grants and loans, and recipient information (Files D1, D2, E, and F). Each agency’s data must pass a series of validations in the broker and then be certified by the agency’s senior accountable official (SAO) before they are submitted for display on USAspending.gov. According to OMB guidance, the purpose of the SAO certification is to provide reasonable assurance that the agency’s internal controls support the reliability and validity of the data submitted to Treasury for publication on the website. The SAO assurance means that, at a minimum, the data reported are based on appropriate controls and risk management strategies as described in OMB Circular A-123, Management’s Responsibility for Enterprise Risk Management and Internal Control. In addition, agencies should include information about any data limitations in their SAO certification statements. Committee for Purchase from People Who Are Blind or Severely Disabled (AbilityOne Commission) District of Columbia Courts (DC Courts) In addition to the above contacts, Peter Del Toro (Assistant Director), Kathleen Drennan (Assistant Director), Michael LaForge (Assistant Director), Maria C. Belaval (Auditor-in-Charge), Barbara Lancaster (Analyst-in-Charge), Diane Morris (Auditor-in-Charge), Carl Barden, Daniel Berg, Mark Canter, Jenny Chanley, Shelby Clark, Tracy Davis Ross, Tabitha Fitzgibbon, Valerie Freeman, Jamaika Hawthorne, Michael Kany, Roy Kilgore, Peter Kramer, Seraé LaFache-Brazier, Krista Loose, Tonyita Muschette, Quang Nguyen, Kristine Papa, Joseph Raymond, Lisa Rowland, Susan Sato, John A. Schaefer, Sara Shore, James Skornicki, Andrew J. Stephens, James Sweetman, Jr., Silvia Symber, and Lisa Zhao made key contributions to this report. Additional members of GAO’s DATA Act Internal Working Group also contributed to the development of this report. DATA Act: Customer Agencies’ Experiences Working with Shared Service Providers for Data Submissions. GAO-19-537. Washington, D.C.: July 18, 2019. DATA Act: Pilot Effectively Tested Approaches for Reducing Reporting Burden for Grants but Not for Contracts. GAO-19-299. Washington, D.C.: April 30, 2019. DATA Act: OMB Needs to Formalize Data Governance for Reporting Federal Spending. GAO-19-284. Washington, D.C.: March 22, 2019. Open Data: Treasury Could Better Align USAspending.gov with Key Practices and Search Requirements. GAO-19-72. Washington, D.C.: December 13, 2018. DATA Act: Reported Quality of Agencies’ Spending Data Reviewed by OIGs Varied Because of Government-wide and Agency Issues. GAO-18-546. Washington, D.C.: July 23, 2018. DATA Act: OMB, Treasury, and Agencies Need to Improve Completeness and Accuracy of Spending Data and Disclose Limitations. GAO-18-138. Washington, D.C.: November 8, 2017. DATA Act: As Reporting Deadline Nears, Challenges Remain That Will Affect Data Quality. GAO-17-496. Washington, D.C.: April 28, 2017. DATA Act: Office of Inspector General Reports Help Identify Agencies’ Implementation Challenges. GAO-17-460. Washington, D.C.: April 26, 2017. DATA Act: Implementation Progresses but Challenges Remain. GAO-17-282T. Washington, D.C.: December 8, 2016. DATA Act: OMB and Treasury Have Issued Additional Guidance and Have Improved Pilot Design but Implementation Challenges Remain. GAO-17-156. Washington, D.C.: December 8, 2016. DATA Act: Initial Observations on Technical Implementation. GAO-16-824R. Washington, D.C.: August 3, 2016. DATA Act: Improvements Needed in Reviewing Agency Implementation Plans and Monitoring Progress. GAO-16-698. Washington, D.C.: July 29, 2016. DATA Act: Section 5 Pilot Design Issues Need to Be Addressed to Meet Goal of Reducing Recipient Reporting Burden. GAO-16-438. Washington, D.C.: April 19, 2016. DATA Act: Progress Made but Significant Challenges Must Be Addressed to Ensure Full and Effective Implementation. GAO-16-556T. Washington, D.C.: April 19, 2016. DATA Act: Data Standards Established, but More Complete and Timely Guidance Is Needed to Ensure Effective Implementation. GAO-16-261. Washington, D.C.: January 29, 2016. Federal Spending Accountability: Preserving Capabilities of Recovery Operations Center Could Help Sustain Oversight of Federal Expenditures. GAO-15-814. Washington, D.C.: September 14, 2015. DATA Act: Progress Made in Initial Implementation but Challenges Must be Addressed as Efforts Proceed. GAO-15-752T. Washington, D.C.: July 29, 2015. Federal Data Transparency: Effective Implementation of the DATA Act Would Help Address Government-wide Management Challenges and Improve Oversight. GAO-15-241T. Washington, D.C.: December 3, 2014. Government Efficiency and Effectiveness: Inconsistent Definitions and Information Limit the Usefulness of Federal Program Inventories. GAO-15-83. Washington, D.C.: October 31, 2014. Data Transparency: Oversight Needed to Address Underreporting and Inconsistencies on Federal Award Website. GAO-14-476. Washington, D.C.: June 30, 2014. Federal Data Transparency: Opportunities Remain to Incorporate Lessons Learned as Availability of Spending Data Increases. GAO-13-758. Washington, D.C.: September 12, 2013. Government Transparency: Efforts to Improve Information on Federal Spending. GAO-12-913T. Washington, D.C.: July 18, 2012. Electronic Government: Implementation of the Federal Funding Accountability and Transparency Act of 2006. GAO-10-365. Washington, D.C.: March 12, 2010.", "summary": "The DATA Act requires federal agencies to disclose roughly $4 trillion in annual federal spending and link this spending information to federal program activities so that policymakers and the public can more effectively track federal spending through its life cycle. The act also requires OMB and Treasury to establish data standards to enable consistent reporting of agency spending. The DATA Act includes a provision for GAO to report on the quality of the data collected and made available through USAspending.gov. Specifically, this report addresses: (1) the timeliness, completeness, and accuracy of the data, and the implementation and use of data standards; and (2) progress made in developing a data governance structure consistent with key practices, and how it affects data quality. GAO examined a projectable government-wide sample of Q4 FY2018 spending data from a Treasury database that populates data on USAspending.gov by comparing them to agency source records and other sources. GAO also compared the results of Q4 2018 with results from its previous review of Q2 FY2017 data. The Digital Accountability and Transparency Act of 2014 (DATA Act) requires federal agencies to report spending data to USAspending.gov, a public-facing website. A total of 96 federal agencies submitted required spending data for quarter four of fiscal year 2018 (Q4 FY2018). GAO examined the quality of these data and compared the results with the results of its prior review of quarter two of fiscal year 2017 (Q2 FY2017) data, as appropriate. GAO identified improvements in overall data quality, but challenges remain for completeness, accuracy, use of data standards, disclosure of data limitations, and overall data governance. Completeness. The number of agencies, agency components, and programs that submitted data increased compared to Q2 FY2017. For example, 11 agencies did not submit data in Q4 FY2018, compared to 28 in Q2 FY2017. Awards for 39 financial assistance programs were omitted from the data in Q4 FY2018, compared to 160 financial assistance programs in Q2 FY2017. Accuracy. Based on a projectable governmentwide sample, GAO found that data accuracy for Q4 FY2018—measured as consistency between reported data and agency source records or other authoritative sources and applicable laws and reporting standards—improved for both budgetary and award transactions. GAO estimates with 95 percent confidence that between 84 a 96 percent of the budgetary transactions and between 24 and 34 percent of the award transactions were fully consistent for all applicable data elements. In Q2 FY2017, GAO estimated that 56 to 75 percent of budget transactions and 0 to 1 percent of award transactions were fully consistent. Use of data standards. GAO continued to identify challenges related to the implementation and use of two data elements— Award Description and Primary Place of Performance Address— that are particularly important to achieving the DATA Act's transparency goals. GAO found that agencies continue to differ in how they interpret and apply The Office of Management and Budget's (OMB) standard definitions for these data elements. As a result, data on USAspending.gov are not always comparable, and in some cases it is difficult for users to understand the purpose of an award or to identify the location where the performance of the award occurred. USAspending.gov presentation. GAO identified known data limitations that were not fully disclosed on USAspending.gov. For example, the 90-day delay for inclusion of Department of Defense procurement data is not clearly communicated. In addition, although the website provides a total figure for unreported spending it is unclear whether it includes the 11 agencies that did not submit data. Not knowing this information could lead users of USAspending.gov to inadvertently draw inaccurate conclusions from the data. Data governance. OMB and the Department of the Treasury (Treasury) have established some procedures for governing the data standards established under the DATA Act, but procedures for enforcing the consistent use of established data standards have yet to be developed. Persistent challenges related to how agencies interpret and apply data standards underscore GAO's prior recommendations on establishing a governance structure that ensures the integrity of these standards. GAO maintains that OMB and Treasury should address prior recommendations on monitoring agency submissions, implementing data standards, disclosing data limitations, and developing a robust data governance structure. In addition, GAO makes two new recommendations to Treasury regarding disclosing on USAspending.gov specific known data limitations. Treasury agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "The military services – Army, Navy, Air Force, and Marine Corps – have highly trained EOD personnel to eliminate explosive hazards in support of a variety of events and activities, ranging from major combat operations and contingency operations overseas to assisting the Secret Service in its protection of the President of the United States (see fig.1). EOD forces are dispersed worldwide to meet combatant commanders’ operational requirements related to these missions. Although the services’ EOD forces support combatant commanders, NORTHCOM’s Joint Force Headquarters-National Capital Region coordinates EOD force support of land-based homeland defense and DSCA missions. EOD forces conduct combat-related and DSCA missions that support national military objectives. EOD combat-related missions include preparations for combat such as training and exercises, and the wartime execution of EOD missions. EOD forces play a major role in all phases of combat operations. For example, these forces contribute to information gathering during operations and serve to enable the safe conduct of operations within an operational area. Additionally, EOD forces support freedom of maneuver and force protection. Further, they may directly support missions such as counterterrorism, deterring and defeating aggression, and countering weapons of mass destruction, among others across the spectrum of operations. Officials from each service stated that EOD forces prepare for these combat-related missions during predeployment in-garrison periods. EOD forces also conduct DSCA missions when they are not engaged in combat-related missions. DOD provides EOD forces when requested in advance by specific federal agencies and approved by the appropriate DOD official. Officials stated that generally, EOD forces undertake VIP support missions during in-garrison periods, just after returning from combat-related deployments or while preparing for the next deployment. The military services collectively have more than 6,300 EOD positions to fulfill combatant command missions, and demand for EOD manpower and expertise is high. Each service determines the number of EOD technicians it needs based on its respective requirements, which consider combatant commanders’ wartime missions and plans. According to a DOD official, the services take into account the long lead times—up to 3 years in one service—that can be necessary to produce qualified and experienced EOD specialists. In accordance with DOD policy, when considering EOD wartime requirements, service officials should make certain that national military objectives can be accomplished using a minimum of manpower that produces maximum combat power. DOD policy also states that a formal validated process is to be used to determine wartime manpower requirements. Generally, manpower requirements are the amount of personnel needed to accomplish a job, mission, or program. Joint doctrine outlines mission tasks associated with EOD units. Once a service determines the tasks required of a particular community (such as EOD), the service then sizes its forces (i.e., determines the manpower requirement) according to the demand for those tasks among the combatant commands. Risk is the effect of uncertainty on objectives with the potential for either a negative outcome or a positive outcome or opportunity. In the military, accurately appraising risk allows leaders and staffs to manage and communicate risk effectively to inform decisions across disparate processes. Joint doctrine describes a planning process that aligns resources and military activities, and enables leaders to examine risks, among other factors, to determine a preferred course of action to achieve an objective. Planning for EOD involves military manpower systems that accurately determine the required EOD forces and decision makers who decide how much risk is acceptable if or when there is a shortfall of EOD forces. According to DOD doctrine on joint planning, regardless of the efforts to mitigate it, some level of risk will remain and should be identified to senior leaders so there is a common understanding of the decisions required and the potential effects of those decisions. Commanders must include a discussion of risk in their interaction with DOD senior leaders and that discussion must be in concrete terms that enable and support decision- making. In the context of strategic and military risk evaluation during joint planning, combatant commanders and DOD’s senior leaders work together to reach a common understanding of risk, decide what risk is acceptable, and minimize the effects of accepted risk by establishing appropriate risk controls. The military services’ processes for determining EOD manpower levels are based on combat-related missions and, accordingly, do not fully consider DSCA missions. However, DOD provides EOD resources for various DSCA missions such as: aiding in the protection of the President of the United States and dignitaries through VIP support missions; providing assistance to law enforcement agencies and other civil authorities in the United States and its territories when necessary to save lives under DOD’s immediate response authority; and rendering safe military munitions when requested by civil authorities (see fig. 2). EOD and manpower officials from each of the military services explained that, in practice, their respective services focus on combat-related missions and do not consider DSCA missions in determining the number of EOD personnel needed. Specifically: According to Army officials, the Total Army Analysis process that is used to size Army forces considers core functions for combat operations and warfighting requirements. They explained that this process does not consider DSCA requirements in determining the number of EOD forces needed. In Army guidance, manpower is based on wartime missions and wartime requirements for sustained combat operations, among other types of information. Due in part to force structure adjustments and the drawdown of EOD forces, since 2014, according to information provided by the Army, the Army has reduced more than 800 EOD positions, the equivalent of two EOD battalions and 13 EOD companies. According to Navy officials, the Navy makes manpower decisions with a focus on wartime requirements by analyzing required operational capabilities against the projected operational environment. In Navy manpower guidance, this analysis is critical to developing fleet manpower requirements for units such as EOD forces. Navy officials explained that the process does not consider the DSCA mission in determining EOD manpower. The Air Force’s EOD manpower standard, which has been updated through 2013, is based on in-garrison needs and wartime requirements. In Air Force manpower guidance, manpower is described as a critical resource that enables combat capability; the guidance further notes that manpower requirements are identified and resources are subsequently allocated for peacetime and wartime missions. However, Air Force officials stated that the process focuses on results for combat-related missions and does not specifically include DSCA requirements. According to Marine Corps officials, the Marine Corps’ EOD forces are sized to support Marine Expeditionary Forces for deployment for overseas combat operations. The Marine Corps’ manpower guidance describes a force structure process designed to identify and provide the capabilities, including personnel and equipment, necessary to accomplish mission essential tasks. Marine Corps officials stated that the service does not receive additional EOD manpower specifically for DSCA missions. Although service manpower calculations do not reflect DSCA missions, one of the DSCA missions—VIP support—is manpower intensive and occurs frequently. Specifically, the workload for VIP support can be substantial and has increased from about 248,000 man-hours in fiscal year 2007 to over 690,000 man-hours in fiscal year 2017. According to a NORTHCOM official, this rise is due to an increase in the different types and complexity of threats requiring more EOD personnel to sufficiently support civil authorities. Figure 3 below illustrates the increase in the amount of time the EOD forces have spent on VIP support missions. The military services have a long-standing practice of providing support to civil authorities, including EOD support. DOD support to civil authorities is grounded or reflected in statute and DOD guidance. For example, the Presidential Protection Assistance Act of 1976 requires executive agencies, including DOD, to assist the Secret Service on a temporary basis in protecting the President, the Vice President and other persons— such as visiting foreign dignitaries (see fig.4). In addition, the National Military Strategy and current homeland defense strategy prioritize defending the homeland and providing support to civil authorities. Moreover, DOD guidance addresses DSCA generally as well as specific support to the Secret Service, Department of Homeland Security, and law enforcement. Further, the Secretary of Defense approved a Joint Staff standing execute order (EXORD) which is used to execute routine VIP support missions related to the protection of dignitaries on short notice. This order authorizes NORTHCOM to provide EOD support to the Secret Service and U.S. Department of State within the NORTHCOM area of operations, and to coordinate that support at locations worldwide. Joint doctrine for EOD also lists DSCA as one of nine military missions that EOD forces may directly support, and states that the majority of EOD DSCA missions will be in support of law enforcement or emergency support agencies. Finally, the military services’ Inter-Service Responsibilities for Explosive Ordnance Disposal lists several common responsibilities of the military services’ EOD assets that include providing support to civilian agencies such as the Secret Service. While the DSCA mission is emphasized in departmental guidance and support of civil authorities has placed increasing and significant demands on EOD forces, the military services do not fully consider these factors in determining the appropriate number of EOD forces. According to EOD officials, this is because the primary mission of EOD forces is to conduct combat missions in support of combatant commanders and meet operational plans. Service officials stated that DSCA missions are not priority missions when it comes to sizing their respective forces, and that they do not routinely increase EOD manpower in order to provide support to other federal agencies. Standards for Internal Control in the Federal Government state that management should design control activities to achieve objectives and respond to risks. Specifically, management should ensure policies and procedures are relevant and effective in achieving an entity’s objectives. In addition, the standards state that management should use quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives. Quality information is information that is appropriate, current, complete, accurate, accessible, and provided on a timely basis. DOD manpower policy states that manpower management shall be flexible, adaptive to program changes, and responsive to new management strategies, and that existing policies, procedures, and structures shall be periodically evaluated to ensure efficient and effective use of manpower resources. However, the military services’ current processes for determining the appropriate amount of EOD manpower do not fully account for the increase in DSCA missions requiring EOD support. While it is understandable that the services prioritize combat missions when determining EOD requirements, they are not considering all available information in their decision-making process. This lack of consideration limits their ability to efficiently and effectively achieve their objectives and manage risks. Accounting for the increase in EOD manpower demand may not necessarily result in an increase in manpower; however, the services will be better prepared to understand the demand on existing EOD forces and evaluate any resulting risks. Ultimately, unless the military services update appropriate guidance to ensure that they consider the total EOD force required to support combat-related and DSCA missions, decision makers cannot accurately assess the sufficiency of EOD forces to meet both missions and the associated risks. DOD cannot evaluate the effects of VIP support missions on military preparedness because current VIP support mission guidance does not require the military services to notify the Joint Staff and appropriate combatant commands when military preparedness is negatively affected by these missions. According to officials from the military services, the execution of VIP support missions introduces risk that threatens the services’ abilities to execute combat-related missions. Specifically, military preparedness is degraded when the services’ EOD forces are unable to concurrently complete predeployment tasks, such as training for combat, because the forces are called upon to execute routine VIP support missions. Officials told us that EOD forces can only conduct these VIP support missions during the time period when EOD forces are scheduled to conduct predeployment tasks and accomplish training. As a result, according to officials, VIP support missions can deleteriously affect military preparedness for EOD forces. In multiple instances, missions supporting civil authorities have stressed the Army’s EOD capabilities, resulting in missed training and the inability to participate in exercises and activities supporting combat-related missions, according to statements and data provided by the Army. Furthermore, fulfilling VIP support missions can be particularly difficult because short-notice demand for EOD teams often exceeds the planned VIP support demand that can be supported. As a result, Army EOD teams are sometimes dispatched at the expense of military preparedness for combat-related missions in support of combatant commands, according to Army officials. Specific details of the effect recent VIP support missions have had on the Army’s EOD capabilities are included in our July 2019 restricted report. Officials from other services also acknowledged that undertaking routine VIP support missions comes at the expense of training for combat-related missions because of the high demand for and limited number of EOD forces. According to a senior Navy official, that service has sometimes refused mission requests to protect dignitaries because of its inability to meet operational demands, such as deployments and training for its EOD forces and support missions to protect dignitaries simultaneously. When this occurs, however, NORTHCOM will ask another service to accept the mission, thereby putting increased demand on that other service’s EOD forces that, in turn, may conflict with their scheduled training and preparations for combat missions, according to military service officials. Because NORTHCOM has few permanently assigned forces to conduct VIP support missions, it must instead rely on EOD forces from each of the military services that are in-garrison and preparing for but not currently deployed to a combat-related mission. According to DOD guidance, DOD’s ability to grant Secret Service requests for support is to be evaluated based on a number of factors, one of which is the effect on military preparedness. For example, DOD Directive 3025.18, Defense Support of Civil Authorities (DSCA), specifies that requests from civil authorities for assistance shall be evaluated for several factors, including the impact on DOD’s ability to perform its other primary missions. The guidance also provides that the Chairman of the Joint Chiefs of Staff is responsible for advising the Secretary of Defense on the effects of requests for civil support on national security and military readiness. According to joint doctrine, a commander’s tasks associated with the function of command and control include managing risk—such as that arising from EOD support for other agencies protecting dignitaries—as well as communicating and ensuring the flow of information across the staff and joint force, and to higher authorities. Additionally, in the context of evaluating strategic and military risk during joint planning, combatant commanders and senior DOD leaders work together to reach a common understanding of risk, decide what risk is acceptable, and minimize the effects of accepted risk. The Standards for Internal Control in the Federal Government also addresses the importance of an entity using quality information to achieve its objectives. Specifically, management should use quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks. As previously mentioned, the Joint Staff has issued a Secretary of Defense-approved EXORD that provides guidance for the military to provide EOD support to the Secret Service and Department of State for routine VIP support missions. However, this EXORD does not specify a requirement for the services to notify DOD stakeholders regarding the effect on military preparedness for combat missions. As a result, the military services are not advising the Joint Staff or NORTHCOM when these VIP support missions are adversely affecting EOD military preparedness for combat-related missions. Regarding military preparedness, the absence of a notification requirement precludes decision makers from understanding the risk to EOD forces’ ability to perform their primary mission. Decision makers need this information to carry out their responsibilities and assess risk to ensure efficient and effective accomplishment of both VIP support missions and preparation for combat-related missions for combatant commands. The military services’ EOD forces provide the combatant commanders necessary capabilities for combat and combat-related missions. They also provide capabilities through their DSCA missions that are important to supporting U.S. law enforcement agencies and other federal, state, and local civil authorities. DOD has manpower processes that result in careful consideration of the requirements of the combatant commander for combat-related missions. However, those manpower processes do not fully consider DSCA missions, such as the VIP support mission and its accompanying substantial workload. Until DOD processes begin to consider the demand for EOD support for both types of missions, decision makers cannot know the complete manpower requirement for EOD. Consequently, the extent to which the services’ EOD forces are sufficient or insufficient to meet national military objectives cannot be fully known. Furthermore, DOD lacks a requirement in guidance specific to the VIP support mission to notify stakeholders regarding the effects of such missions on military preparedness for combat-related missions. As a result, DOD may not be fully considering risks associated with the use of EOD forces for VIP support on the preparation and training of those forces for combat-related missions. We are making the following four recommendations to DOD. The Secretary of the Army should update Army manpower guidance, or other guidance as appropriate, to ensure that all missions conducted by EOD forces, including DSCA missions, are considered in determining the required number of EOD forces. (Recommendation 1) The Secretary of the Air Force should update Air Force manpower guidance, or other guidance as appropriate, to ensure that all missions conducted by EOD forces, including DSCA missions, are considered in determining the required number of EOD forces. (Recommendation 2) The Secretary of the Navy should update Navy and Marine Corps manpower guidance, or other guidance as appropriate, to ensure that all missions conducted by EOD forces, including DSCA missions, are considered in determining the required number of EOD forces. (Recommendation 3) The Secretary of Defense should ensure that the Chairman of the Joint Chiefs of Staff, in collaboration with the combatant commands, incorporate into the appropriate guidance a requirement that the military services notify the Joint Staff and the affected combatant commands when the execution of VIP support missions negatively affects the preparedness of EOD units for combat-related missions. (Recommendation 4) We provided a draft of this report to DOD for review and comment. DOD did not provide comments. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Secretaries of the Army, Navy, and Air Force. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have questions about this report, please contact me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the contact named above, Guy LoFaro, (Assistant Director), Ben Atwood, Naba Barkakati, Christopher Gezon, Amie Lesser, Dennis Mayo, Paulina Reaves, Michael Silver, Michael Shaughnessy, Stephen Woods, and Lillian Yob made contributions to this report.", "summary": "EOD forces are a high demand, critical asset that support DOD's ability to execute military operations. DOD increased the number of EOD forces by more than 70 percent from 2002 to 2012 because of increased demand. When not deployed, EOD forces provide support to civil authorities. One of these missions is protecting U.S. and foreign dignitaries—also referred to as VIP support missions. House Report 115-200 included a provision for GAO to review matters related to EOD capabilities and requirements. This report assesses the extent to which (1) the military services consider all combatant command EOD requirements, including DSCA, in determining the number of EOD personnel needed, and (2) DOD evaluates the effect of VIP support missions on the military preparedness of EOD forces. GAO reviewed relevant guidance, analyzed EOD data, and interviewed EOD and manpower officials. This is a public version of a sensitive report that GAO issued in July 2019. Information that DOD deemed sensitive has been omitted. The military services' processes for determining the necessary number of explosive ordnance disposal (EOD) personnel are based on combat-related missions. However, these processes do not fully consider some defense support of civil authority (DSCA) missions that EOD forces conduct. Demand for EOD forces for DSCA missions can be manpower-intensive and frequent. For example, EOD forces' workload for protecting U.S. and foreign dignitaries—also referred to as Very Important Person (VIP) support missions—increased from about 248,000 to over 690,000 man-hours in fiscal years 2007 to 2017 (figure). However, according to officials, the services do not consider DSCA missions in determining the number of EOD personnel needed, instead focusing on combat-related missions. Unless the Department of Defense (DOD) ensures that the services update guidance to consider the total EOD force required to support both missions, decision makers cannot accurately assess the EOD forces' sufficiency. DOD guidance specific to VIP support missions does not include a requirement for the services to report on the effect of VIP support missions on military preparedness. According to officials, military preparedness is degraded when the services' EOD forces are unable to concurrently complete predeployment tasks such as training for combat. Per DOD guidance, Secret Service support requests are to be evaluated based on their effects on military preparedness. Without this information, decision makers are precluded from understanding the risk to EOD forces' military preparedness resulting from the routine VIP support missions. Decision makers need this information to ensure efficient and effective accomplishment of both VIP support missions and preparation for combat-related missions for affected combatant commands. GAO is making four recommendations including that DOD (1) update the appropriate service guidance to ensure that all EOD missions, including DSCA missions, are considered in determining the required number of EOD forces, and (2) incorporate into appropriate guidance a requirement for the military services to notify the Joint Staff and combatant commands when VIP support missions negatively affect the military preparedness of EOD units. DOD did not provide comments on the draft of this report.", "document_type": "gao"}
{"report": "In response to the 2014 access crisis, VA launched the MyVA initiative, which was designed to transform the health care experience of veterans. In concert with the MyVA initiative, VA introduced the MyVA Access Declarations in April 2016 with the goal of improving access by providing veterans more control as to how they receive their health care. The MyVA Access Declarations was a list of nine “access declarations” that were intended to serve as the foundational principles for improving and ensuring access to care. Two of these “access declarations” required providing timely primary and mental health care and included same-day services. VHA had policies in place for same-day services in primary and mental health care clinics for several years prior to the same-day-services initiative. In primary care, the 2014 Patient-Aligned Care Team (PACT) handbook required all primary care providers and registered nurses to ensure they provide same-day access (unless it is too late in the day as determined by the individual facility) for face-to-face encounters, telephone encounters and, when required by VHA guidance or policy, other types of encounters. The PACT handbook was supplemented by a 2015 VHA memo on unscheduled patient walk-ins. The memo states that if an unscheduled patient presents at a PACT clinic with a clinical concern, the patient cannot be turned away without evaluation by a clinical member of the team, regardless of clinic hours, resource availability, or eligibility/enrollment status. VHA also had previously developed policies stating that veterans are entitled to timely access to mental health care. Specifically, a 2007 VHA memo required that all veterans requesting or referred for mental health care or substance abuse treatment receive an initial evaluation within 24 hours. VHA’s 2015 Uniform Mental Health Services handbook also noted that all new patients requesting or referred for mental health care services must receive an initial evaluation within 24 hours and a more comprehensive diagnostic and treatment planning evaluation within 30 days. Additionally, since 2008, VHA has required the integration of primary care and certain mental health care services at VA medical centers serving a veteran population greater than 5,000. This care model, known as Primary Care–Mental Health Integration (PC-MHI), integrates mental health staff into each primary care PACT clinic, allowing veterans to receive services for depression, anxiety, post-traumatic stress disorder, and substance use without needing to obtain a separate referral to providers in the mental health care clinic. According to VHA guidance, PC-MHI has been shown to improve access to same-day mental health care and reduce no-show rates to appointments. VHA’s veterans access to care office was created in 2016 as the national oversight office for VHA access-to-care issues. Additionally, each VISN is responsible for overseeing the VA medical centers within their designated regions. This oversight includes oversight of access issues and the implementation of initiatives such as the same-day service initiative. VA medical center directors are responsible for ensuring local policies are in place for the effective operation of their primary and mental health care clinics, including affiliated CBOCs. VHA used a five-pronged approach to design its same-day services initiative: VHA (1) defined same-day services, (2) developed guidance, (3) updated its mental health policies, (4) offered training, and (5) assessed VA medical center readiness to implement the initiative. VHA defined same-day services. As an initial step, VHA leadership developed the following definitions of same-day services in primary and mental health care: Same-day services in primary care: “When a veteran requires primary care services right away, during regular business hours, he or she will receive services the same day at a VA medical center. If a veteran calls after normal business hours, he or she will receive care the next business day.” Same-day services in mental health: “If a veteran is in crisis or has another need for mental health care right away, he or she will receive immediate attention from a health care professional at a VA medical center.” VHA also identified a variety of ways in which veterans can receive same- day services, including: (1) providing a face-to-face visit; (2) returning a phone call; (3) arranging a telehealth or video care visit; (4) responding to a secure email; or (5) scheduling a future appointment. VHA developed guidance for the same-day service initiative. To help VA medical centers implement its definition of same-day services, in April 2016, VHA developed written guidance—the MyVA Access Implementation Guidebook. The guidebook provides a variety of solutions to help VA medical centers meet the intent of the same-day service initiative. The guidebook includes specific solutions for VA medical centers struggling to provide same-day services in primary or mental health care for veterans with urgent care needs: Implementing open access in primary and mental health care: Open access aims to balance the supply of (for example, available appointments) and demand for (for example, the number of patients assigned to a provider and annual visits per patient) services to increase patient access. Achieving open access requires implementing specific strategies including achieving full staffing, planning for contingencies such as clinical staff absences or vacancies and managing the number of times patients see a provider each year, among other strategies. Implementing primary care-mental health integration: In order to complete the implementation of PC-MHI across the VA system, the guidebook suggests facilities address staffing vacancies, develop a PC-MHI implementation plan, and choose an open access scheduling model (for example, full open access where there are no scheduled appointments and patients are seen on a first come, first served basis), among other things. Utilizing same-day referrals to mental health for suicide prevention: This solution reiterates many of the mental health policy changes that VHA introduced in conjunction with the same-day service initiative such as implementing an initial screening evaluation, developing a process for same-day care for established patients with an urgent need, and deploying open access scheduling, among other things. The guidebook states that all of the solutions were chosen because they were used successfully at other VA medical centers; can be quickly implemented; and have a high impact on veterans’ access to care. The guidebook also notes that flexibility is a key element when choosing solutions and explains that VA medical centers should select and modify solutions as needed. The guidebook does not make any of the solutions mandatory; however, several of the mental health solutions were introduced to facilities through separate VHA memos and are required. VHA updated mental health policies. VHA updated certain mental health policies to facilitate the implementation of the same-day services initiative. Specifically, in April 2016 VA issued a memo updating its mental health policy to require that any veteran new to mental health services requesting or referred for care in person be seen the same day by a licensed independent provider to screen for and address immediate care needs. This was a change from the previous timeframe of 24 hours for an initial evaluation. The memo also created new processes for VA medical centers to assess same-day services in mental health care clinics, including a medical chart review and a one-time review of standard operating procedures to ensure that the new guidelines are being followed. VHA also distributed other memos that either sought to clarify existing guidance or expand same-day services into other areas of mental health care, such as substance abuse. Additionally, VHA provided a memo to VA network directors and mental health leads about scheduling models for mental health care that all VA medical centers needed to implement for the same-day service initiative. VHA provided training on the same-day-services initiative. VHA provided voluntary training for same-day services some of which discussed the solutions from the guidebook and the updated mental health policy. The trainings began in February 2016 for primary care and in May 2016 for mental health. The trainings consisted of national telephone calls (often with slide presentations) that any VA medical center staff member could join, and the presentation materials were posted to VHA’s internal website. The telephone trainings generally occurred twice a month in primary care and every week in mental health care. VHA assessed VA medical center same-day service readiness. Beginning in January 2017, VHA provided technical assistance around same-day services to VA medical centers. VHA reviewed several aspects of same-day services, including how VA medical centers were able to provide same-day services and identified any approaches that may have needed improvement. Generally, low-performing VA medical centers received continuous on-site support; moderate performing VA medical centers received a combination of virtual and on-site support; and, high performing VA medical centers primarily received virtual support. To determine the progress that VA medical centers were making in providing same-day services, VHA conducted surveys that required medical center directors to self-certify—and, in some cases, VISN directors to validate—that their VA medical centers (including affiliated CBOCs) were able to provide same-day services. In the event that a VISN director could not validate medical center survey information, VHA followed up with the medical center and VISN director to create an action plan to mitigate any issues that were delaying validation. These surveys were conducted in 2016 and 2017; focused on either primary care, mental health care or both; and varied in the information collected to determine how VA medical centers were providing same-day services (See Table 2 for information on the same-day-services readiness assessment surveys used by VHA). According to VHA, all VA medical centers were offering same-day services in primary and mental health care by December 2016. In January 2018, VHA announced that same-day services in primary and mental health care had been achieved in all VA medical centers and CBOCs (more than 1,000 facilities). Officials we spoke with from all six VA medical centers in our review told us they were providing same-day services in primary and mental health care prior to the same-day service initiative, an assertion supported by VHA survey data. For example, in a VHA survey conducted in May 2016, around the same time as the launch of the same-day service initiative, 142 out of 165 officials (86 percent) that responded to the survey said that their medical centers offered same-day appointments “always” or “very frequently” in primary care for urgent concerns. We found that the VA medical centers in our review used a variety of approaches in providing same-day services in primary and mental health care, most of which were in existence before the initiative. As noted earlier, VHA did not require the implementation of any specific solutions in the guidebook and afforded VA medical centers the flexibility to choose appropriate local solutions for the same-day service initiative. Many VA medical centers used this flexibility to continue providing same-day services as they had prior to the initiative often because that is what their resources allowed them to do or, in the case of mental health, because it was built into the foundation of their service line. VHA officials noted that mental health services—particularly PC-MHI—were built around same- day services so VHA’s guidance was familiar to them. The approaches used by the selected VA medical centers included using “float providers” who had not already been assigned specific patients to assist those who requested same-day services; carving out specific appointment times in the schedule for walk-ins; overbooking appointments in providers’ schedules, and offering walk-in clinics. VHA suggested that certain solutions should be prioritized if VA medical centers were struggling to provide same-day services and, in particular for mental health, created new requirements around same-day services. However, officials at selected VA medical centers noted that some of the suggested solutions in the guidebook—particularly open access—and requirements in updated mental health policies were difficult to implement because of longstanding challenges with staffing, space, or competing VHA policies. For example, VHA’s guidebook suggests the implementation of open access in primary and mental health care in such situations. However, officials at four of the six VA medical centers we visited noted that open access was difficult to implement because of the long-standing challenges mentioned above. In addition, VHA updated its mental health policy to include that any veteran new to mental health services requesting or referred for care in person be seen the same day by a licensed independent provider to screen for and address immediate care needs. However, one medical center official noted that they had designed their mental health clinic processes around registered nurses, who are responsible for completing the initial assessments of new patients. The official added that the medical center did not have licensed independent providers readily available at certain facilities to help complete the assessments in a timely manner. Officials at all six medical centers we visited noted that implementation was also sometimes challenging as veterans’ expectations shifted with the same-day-services initiative, with veterans’ expecting more immediate access to care from physicians for a variety of conditions. For example, one medical center official noted that veterans are presenting for care and wanting to see a provider because it is these veterans’ understanding that they could get care immediately for any condition including chronic, less urgent issues. Additional officials at the same facility echoed this concern and noted that they are not certain that this was the policy’s intent. Another medical center official noted that several medical center officials asked VHA to change the name “same-day service” because it gives the impression that veterans would always be able to see their provider immediately. This official added that there is some confusion for both staff and veterans about what are same-day services. Additionally, according to one veterans service organization official that we spoke with, a small number of veterans reported that the availability of same-day services varied by facility (VA medical center versus CBOC) and location (urban versus rural). Another medical center official noted that same-day services are not sustainable if the definition is immediate care by a provider for any condition, especially non-urgent issues. VHA officials told us that the same-day service initiative was a response to the 2014 access crisis and they wanted facilities to use the resources available to them rather than waiting on new policies and strategies. They stated that their main concern was that veterans’ needs were met, not necessarily how they were met. As such, VHA officials told us that they found VA medical centers’ implementation of same-day services acceptable. The VHA officials added that the guidebook is still the foundational document for same-day services. VHA officials told us that it is important for VA medical centers to educate patients on the appropriate use of same-day services. They added that in fiscal year 2019 they are (1) developing a more precise definition of same-day services; (2) developing a website to better explain the purpose of the initiative; and (3) requiring on-demand trainings to provide a clearer explanation about what same day services are available and what staff roles and responsibilities are, among other things. The training is expected to be completed no later than the first quarter of fiscal year 2020. VHA is limited in its efforts to assess the impact of same-day services due to its lack of documented objectives, developed performance goals and related performance measures. Our previous work has shown the benefit of fully connected objectives and performance goals with measurable targets. Objectives state the longer term desired impact or outcome to be achieved, while performance goals communicate the target the agency seeks to achieve within a certain timeframe. Performance measures are indicators of the progress the agency is making towards a goal or target within a particular time frame. VHA officials told us that the overall objectives of same-day services are to improve veterans’ access to care and customer service while having minimal impact on medical centers’ existing workflows. However, VHA has not documented these objectives—for example, in a directive. In addition, VHA has not developed and documented performance goals that, with associated performance measures, would facilitate monitoring of progress towards the desired outcome of the same-day services initiative. VHA officials stated that the same-day-services initiative was developed quickly in response to the 2014 access crisis, and noted that at the time, their focus was “to get something out quickly” instead of taking time to standardize the initiative around specific policies and procedures, which could include documenting objectives and developing performance goals. VHA officials acknowledged that their decision to focus on quickly implementing the initiative without documenting objectives and developing performance goals and associated performance measures makes assessing the impact of the same-day services initiative more challenging. VHA has taken some steps to collect data on same-day services. For example, VHA officials stated that they primarily rely on two measures to assess the impact of the same-day services initiative: patient experience scores and the number of same-day appointments. However, without performance goals these measures do not provide VHA with a means to monitor progress and provide limited information on same-day services’ impact. Patient experience score: VHA uses the Survey of Healthcare Experiences for Patients (SHEP) to measure veterans’ perceptions of their experience at VA medical centers. For same-day services, VHA monitors responses to two questions. According to VHA officials, the key measure is based on the survey question that asks “in the last 6 months, when you contacted this provider’s office to get an appointment for care you needed right away, how often did you get an appointment as soon as you needed?” While SHEP scores provide some data related to customer service and access to care, VHA has not developed performance goals that sets targets for these or other aspects of the same-day services initiative that would benefit from monitoring. Such goals would better enable VHA to identify gaps in performance and plan any needed improvements; ensure balance between agency priorities, such as customer service and access; and identify unintended effects, such as disruption to clinic workflows. For example, officials at one medical center told us that focusing on customer service creates issues with respect to routine care in that veterans’ definition of customer service is based on what makes them happy, while providers are focused on providing the best treatment. Officials added that these two definitions do not always align. In addition, officials at another medical center stated that implementing same-day services impacted their providers’ schedules and the resulting changes to their processes created chaos. Number of same-day appointments: VHA measures the number of same-day appointments, which, according to a VHA official, are identified in VHA data as appointments completed on the same day they are created in VHA’s scheduling system. According to a VHA training document, VA completed 12 million same-day appointments in fiscal year 2018. However, without performance goals with clear targets for same-day appointments, an official from one VISN said she was unclear how many same-day appointments medical centers should be scheduling. Additionally, same-day services performance goals may afford VHA the opportunity to monitor other key measures—such as those that capture services that do not require an appointment—which could provide VHA with important information on the impact of same-day services on access to care. Moreover, performance goals and additional performance measures may help prevent unintended consequences, such as an over-emphasis on same-day appointments as the way to provide same-day services, which VHA officials stated they are working to curb. For example, officials at two selected medical centers also noted that measuring the number or proportion of same-day appointments does not capture all the ways medical centers provide same-day services. Officials at two other selected medical centers noted they can meet veterans’ same-day needs through multiple avenues, such as a registered nurse providing patient education or by renewing a prescription, that do not require an appointment and therefore, would not be counted in the number of same-day appointments. VHA officials stated that the impact of the same-day services on access to care is difficult to measure and additional measures would help properly measure the impact. VHA’s lack of documented objectives and developed performance goals and related measures is inconsistent with our prior work on effective management practices and federal internal control standards. Specifically, we have previously reported that performance measures benefit from certain key practices, such as breaking down of broad long- term objectives into specific near-term performance goals with measurable targets and time frames, and key attributes, such as balance to prevent skewed incentives over-emphasizing certain goals. Additionally, Standards for Internal Control in the Federal Government states that documentation provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel. Without clearly documented objectives, performance goals, and related performance measures, VHA is hindered in its efforts to define success for its same-day service initiative and measure progress achieving it. VHA officials stated they rely on VISN and VA medical center officials to oversee same-day services; however, we found that without performance goals and related performance measures, VISNs and VA medical centers found it challenging to oversee the same-day services initiative. Specifically, officials at five of the six medical centers and two of the four VISNs we visited stated that it is difficult to measure same-day services; which in turn makes assessing the initiative’s impact on veterans’ access to care difficult. Officials at one medical center explained that the challenge stems from the fact that that VHA has not defined what outcome it wants to achieve. In addition, officials at another VA medical center stated that they have a number of access measures available to them, but it was unclear to them which measures they should be prioritizing as part of their oversight of the same-day services initiative. Further, absent performance goals, we found that VISNs and medical centers, which operate in a decentralized environment, varied in their oversight strategies. For example, one VISN required all medical centers to complete a self-assessment of their access capacity and sustainability, and collected information on a number of key open access elements, including Patient-Aligned Care Team staffing levels and provider panel sizes, among others. However, oversight by other VISNs was reportedly less robust. For example, at one VISN, officials stated it is difficult to audit access broadly and described their oversight of same-day services as “fairly minimal.” At the medical center level, oversight also varied as officials tried to develop their own oversight solutions. Officials at one medical center we visited used a feature within the outpatient appointment scheduling system that allowed them to count the specific services, such as pharmacy refills, that veterans seeking same-day mental health care had requested. According to these officials, the tool provided additional data not found in existing VHA access-related reports and allowed them to better understand veterans’ demand for specific same-day services and utilize resources more efficiently. These officials added that they developed this solution because they had not received guidance from VHA on how they should measure demand, and they had skilled staff with the ability to develop their own measures. However, not all VA medical centers we visited had the skilled staff to develop similar solutions. Developing performance goals and related performance measures would better position VHA to obtain useful, comparable information on the impact of same-day services on access to care across VISNs and medical centers. Moving forward, VHA is planning to conduct a “mystery shopper” evaluation of same-day services to assess the impact of same-day services. The mystery shopper evaluation will consist of various scenarios in which veterans, engaged through a contractor, will attempt to access same-day care at a variety of clinics in VA medical centers. As described in a VHA planning document, the evaluation is intended to provide VHA with information on veterans’ experience in obtaining same-day services and will attempt to understand variations in how same-day services are provided. However, VHA officials have not determined if the evaluation will be ongoing. VHA officials stated that in addition to the mystery shopper evaluation, they are considering additional measures to better assess the impact of same-day services beyond their current measures, such as the number of pharmacy refills completed the same day they were requested. However, as of May 2019, VHA had not developed specific performance goals to align these measures to, or set timeframes for their creation. Without overall performance measures that are tied to documented performance goals, VHA will continue to be limited in its ability to assess the impact of same-day services on veterans’ access to care. VHA’s same-day services initiative for primary and mental health care is one of several efforts by VHA to help improve veterans’ access to care in the 5 years since access issues garnered national attention. VHA’s stated objectives for the same-day-services initiative are to improve veterans’ access to care and customer service while having minimal impact on medical centers’ existing workflows. However, VHA has not documented these objectives or developed performance goals and related measures that provide for monitoring towards the desired outcomes. VHA primarily relies on veteran satisfaction scores and the number of same-day appointments to monitor the same-day-services initiative, but these measures alone do not enable an assessment of the impact of same-day services on access to care. Without documented objectives, and performance goals and related measures tied to these goals, VHA will continue to be limited in its ability to determine, how, if at all, the same- day-services initiative has improved veterans’ access to care. The Under Secretary for Health should document same-day services objectives and develop performance goals and related performance measures to facilitate the periodic assessment of the impact of same-day services on veterans’ access to care. (Recommendation 1) We provided a draft of this report to VA for review and comment. In its written comments, which are reproduced in appendix I, VA concurred in principle with our recommendation. VA stated that its Office of Veterans Access to Care will clarify objectives, develop performance goals, and explore the options for reliable performance measures. VA noted that identifying options for performance measures will take approximately 9 months and that additional time may be needed for development, testing and refinement. VA provided a target completion date of April 2020. We are sending copies of this report to the appropriate congressional committee and the Secretary of Veterans Affairs. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at DraperD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Ann Tynan (Assistant Director), Dan Klabunde (Analyst-in-Charge), Jennie F. Apter, and Q. Akbar Husain made key contributions to this report. Also contributing were Muriel Brown, Jacquelyn Hamilton, Ethiene Salgado-Rodriguez, and Merrile Sing.", "summary": "In 2014, a series of congressional testimonies highlighted problems with veterans' access to care after significant appointment wait times at VA medical centers reportedly resulted in harm to veterans. In response, VHA implemented several initiatives, including same-day services at its medical centers and outpatient clinics. GAO was asked to review the same-day services initiative and VHA's related oversight activities. This report (1) describes how VHA designed and how selected medical centers implemented the same-day services initiative; and (2) examines VHA's efforts to assess the impact of the same-day services initiative on veterans' access to care. GAO reviewed VHA documents, including policies, guidance, and requirements related to same-day services and interviewed VHA officials regarding implementation and oversight. GAO visited six VA medical centers selected for the complexity of services offered, range of wait times, and geographic variation, among other factors. GAO interviewed officials from (1) the six VA medical centers and affiliated outpatient clinics, (2) VHA's networks with oversight responsibility, and (3) two veterans service organizations. The Department of Veterans Affairs' (VA) Veterans Health Administration (VHA) introduced its same-day services initiative in primary and mental health care in April 2016, and used a five-pronged approach for its design: it defined same-day services, developed guidance, updated its mental health policies, offered training, and assessed VA medical center readiness to implement the initiative. Officials from all six VA medical centers GAO visited said they already were providing same-day services prior to the initiative and generally relied on previous approaches to implement VHA's same-day-services initiative. However, these officials told GAO that some of VHA's guidance and updated policies were difficult to implement due to long-standing challenges of staffing and space constraints, among others. For example, one medical center official stated that the medical center did not have the appropriate providers readily available to complete the initial mental health assessments of new patients in a timely manner—a new requirement under VHA's updated policies. VHA officials stated that the objectives of the same-day services initiative are to improve veterans' access to care and customer service. However, VHA has not documented these objectives in a directive or developed and documented performance goals that, with associated performance measures, would monitor progress. Although VHA does monitor patient experience scores and the number of same-day appointments, these measures are not tied to specific performance goals. For example, VHA has not specified targets for the number of same-day appointments medical centers should provide. Furthermore, monitoring the number of same-day appointments does not capture all of the ways VA medical centers provide same-day services, such as renewing prescriptions. VHA officials acknowledged the intitiative was quickly developed in response to the 2014 access crisis, and developing new policies or processes, which could include documenting objectives and developing performance goals, was not the priority. Without performance goals and related measures, VHA will continue to be limited in its ability to determine, how, if at all, the same-day services initiative has improved veterans' access to care. GAO recommends that VA document objectives and develop performance goals and related performance measures to facilitate the periodic assessment of the impact of same-day services on veterans' access to care. VA agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "The U.S. government implements an export control system to manage risks associated with exporting sensitive items and ensure that legitimate trade can still occur. The export control system is governed by a complex set of laws, regulations, and processes that multiple federal agencies administer to ensure compliance. State and Commerce each play a role in the U.S. export control system. Historically, State has controlled the export of military items, known as defense articles and services, while Commerce has controlled the export of less sensitive items with both military and commercial applications, known as dual-use items. In addition to firearms, artillery, and ammunition, State controls the export of items such as tanks, fighter aircraft, missiles, and military training, which it lists on the U.S. Munitions List (USML). Commerce controls the export of dual-use items such as computers, radars, and telecommunications equipment, which it lists on the Commerce Control List (CCL). State and Commerce both control the export of items within their jurisdictions by requiring a license or other authorization to export a controlled item; vetting the parties associated with export transactions; monitoring the end-use of exports and other compliance activities; and supporting law enforcement agencies’ investigations of possible violations of export control laws and regulations. Generally, unless a license exemption applies, exporters submit a license application to State if their items are controlled on the USML or to Commerce if they are controlled on the CCL to receive export approval. As part of the application review process, State and Commerce consult with other agencies, including DOD. Additionally, offices within Commerce, DHS, and the Department of Justice (DOJ) investigate potential violations of export control laws and regulations, and conduct enforcement activities. Items identified on the State and Commerce export control lists are subject to different laws and regulations. The Arms Export Control Act of 1976, as amended, (AECA) provides the statutory authority to control the export of defense articles and services, which the President delegated to the Secretary of State. State’s International Traffic in Arms Regulations (ITAR) implement this authority and identify the specific types of items subject to control in the USML. The USML is comprised of 21 categories of items, each with multiple sub-categories, encompassing defense items such as firearms, missiles, and aircraft. Firearms, artillery, and ammunition represent the first three categories of the USML (see table 1). Additional information on the 21 categories of the USML is presented in appendix II. Within State, the Directorate of Defense Trade Controls (DDTC) is responsible for implementing controls on the commercial export of these items. The Export Control Reform Act of 2018 (ECRA) provides the statutory authority for Commerce to control the export of less sensitive military items, dual-use items, and basic commercial items. Commerce’s Export Administration Regulations (EAR), which contain the CCL, implement this authority. The CCL classifies less sensitive military items, dual-use items, and basic commercial items in 10 categories, such as Nuclear & Miscellaneous, Electronics, and Telecommunications, and in 5 product groups. Appendix II shows the 10 categories and five groups of the CCL. Commerce’s Bureau of Industry and Security (BIS) is responsible for implementing these export controls (see table 2 for a summary of the legal and regulatory frameworks for State’s and Commerce’s export controls). In May 2018, State and Commerce published proposed rules in the Federal Register to request public comments on the proposed transfer of certain items in USML Categories I, II, and III (firearms, artillery, and ammunition) to the CCL. According to State and Commerce’s proposed rules, the purpose of the transfer is to limit the items that State controls to those that provide the United States with a critical military or intelligence advantage or, in the case of weapons, are inherently for military end use. According to the proposed rules, items that do not meet these criteria would be removed from State’s export control jurisdiction and moved to Commerce’s jurisdiction. The proposed rules state that some, but not all, of the firearms, artillery, and ammunition currently controlled for export by State would transfer to Commerce control. The items proposed for transfer to the CCL include non-automatic and semi-automatic firearms up to .50 caliber, and non-automatic shotguns with a barrel length less than 18 inches; as well as parts, components, accessories, attachments, and ammunition for these firearms and shotguns, among other items. According to the proposed rules, if finalized, State would continue to control fully-automatic firearms, shotguns, and modern artillery; silencers, components, parts, and accessories specially designed for automatic firearms and shotguns; and specific types of ammunition, including ammunition for automatic firearms. The proposed rules would also make a variety of conforming changes to the USML and CCL to accommodate the transferred items. The proposed transfer of firearms, artillery, and ammunition is part of an ongoing effort to reform the export control lists by reviewing the USML categories and transferring certain items considered less sensitive to the CCL. Since the export control reform initiative was first announced in 2010 with the objective of modernizing the export control system, State and Commerce have finalized various rulemakings that transferred certain items from USML Categories IV through XXI to Commerce’s control. Firearms, artillery, and ammunition are the last three USML categories proposed to undergo regulatory changes under export control reform. In accordance with the AECA, the President must notify Congress of items proposed for removal from the USML and describe the nature of any controls to be imposed on the items, and may not remove the items until 30 days after providing such notice. State and Commerce published the proposed rules in the Federal Register on May 24, 2018, opening a 45-day public comment period that ended on July 9, 2018. After reviewing public comments, State and Commerce submitted final rules to the Office of Management and Budget for regulatory review on November 7, 2018. The required 30-day congressional notification period pursuant to the AECA began on February 4, 2019, according to a State official. State reviewed 68,690 export license applications for firearms, artillery, and ammunition with a potential value of up to $45.4 billion during fiscal years 2013 to 2017. The number of export license applications for firearms, artillery, and ammunition remained relatively constant from fiscal years 2013 to 2017, averaging 13,738 annually, even as the total number of licenses reviewed by State declined as the export control reform process transferred items from State to Commerce control (see fig. 1). Firearms, artillery, and ammunition increased from about 16 percent of all license applications reviewed by State in fiscal year 2013 to about 36 percent in 2017. State processes export license applications for permanent exports, temporary exports and imports, and certain types of agreements. During fiscal years 2013 to 2017, about 91 percent of export license applications for firearms, artillery, and ammunition were for permanent exports, about 8 percent for temporary exports and imports, and about 2 percent for agreements. State can take various actions on the export license applications it receives, including approving the license, approving with conditions, returning without action, and denying the license. For fiscal years 2013- 2017, State approved 87 percent of the number of export license applications for firearms, artillery, and ammunition, returned without action 12 percent, and denied 1 percent. State can approve an application but place conditions on the export license, such as limiting the validity period or prohibiting certain types of intermediaries in the export transaction. State can also return without action export license applications that are missing information or that it is otherwise unable to review, and can deny, revoke, suspend, or amend a license for foreign policy or national security reasons. About two-thirds of the export license applications for firearms, artillery, and ammunition that State reviewed during fiscal years 2013-2017 were for firearms and related items controlled under Category I of the USML (see fig. 2). Of the applications for these items, about 57 percent involved non-automatic or semi-automatic firearms—most of which are proposed to transfer to the CCL under Commerce control—and about 4 percent involved fully-automatic firearms—which would remain on the USML under State control. The remainder of export license applications for Category I items included other types of firearms such as combat shotguns, firearm attachments such as silencers and riflescopes, firearm parts and components, and technical data and defense services related to these items. The proposed rules state that some of these items would transfer to Commerce control while others would remain under State control. As shown in figure 2, export license applications for Category II artillery were about 5 percent of all Category I-III license applications from fiscal years 2013 through 2017. According to State, under the proposed rules, modern artillery, their ammunition, and certain related parts and components would remain under State’s control. Category III ammunition represented about 21 percent of the Category I-III export license applications. As stated in the State and Commerce proposed rules, USML Category III would be revised to specifically list the ammunition that it controls, which would include ammunition that has only or primarily military applications. Generally, ammunition used in the non- automatic and semi-automatic firearms that are proposed to transfer to Commerce control would also transfer. About 8 percent of the export license applications involved items controlled in more than one category of USML Categories I, II, and III, which are shown as “Multiple” in figure 2. In fiscal years 2013 to 2017, 32 percent of license applications for the export of firearms, artillery, and ammunition were intended for end-users in countries in Europe and Eurasia, 29 percent to the Western Hemisphere, 24 percent to East Asia and the Pacific, 7 percent to the Near East, 3 percent to Africa, 3 percent to South and Central Asia, and 2 percent to multiple countries (see fig. 3). Export license applications for firearms, artillery, and ammunition during fiscal years 2013 to 2017 included applications for end-users spanning 189 countries and territories, yet the top 20 countries represented about 70 percent of the total number of applications (see fig. 4). State’s and Commerce’s export controls are guided by different laws, regulations, or policies that have several different requirements for registration, licensing, end-use monitoring, congressional notification, public reporting, and enforcement. The AECA requires manufacturers, exporters, and brokers of items on the USML to register with State whereas there is no registration requirement in the law for manufacturers, exporters, and brokers of items on the CCL under Commerce’s jurisdiction. Differences also exist in how State and Commerce screen export license applications and in their license requirements. For example, State and Commerce rely on different internal watch lists to screen applicants. In addition, according to Commerce, certain exports that currently require a State license would not require a Commerce license once transferred to Commerce’s jurisdiction. State and Commerce also conduct end-use monitoring of selected controlled exports differently. For example, State relies primarily on embassy staff to conduct end-use checks and Commerce relies primarily on several export control officers based overseas for this responsibility. In addition, congressional notification and public reporting requirements that under current law apply to firearms on the USML would not be applicable if they are transferred to the CCL. Finally, there are some differences in enforcement of export control laws, such as different maximum fines for civil violations, depending on whether the item is controlled by the ITAR under State’s jurisdiction or controlled by the EAR under Commerce’s jurisdiction. The AECA requires manufacturers, exporters, and brokers of defense articles or services listed on the USML to register annually with State’s Directorate of Defense Trade Controls (DDTC) whereas there is no requirement in the law for registration for manufacturers, exporters, and brokers of items on the CCL. State reported having 13,083 registrants across all 21 USML categories in fiscal year 2017. Registration, which requires a fee payment of at least $2,250 per year, is generally a precondition for obtaining a State export license, unless State grants an exception to a manufacturer or exporter, or a broker is eligible for an exemption. According to a State document, registration provides important information on the identity and location of defense companies and conveys management responsibility for compliance with export control laws. Those registering must disclose any foreign ownership or affiliations and certify that they have not been indicted, otherwise charged with, or convicted of export control violations and other crimes. Manufacturers and exporters whose entire product line transfers to the CCL would no longer have to register, according to Commerce’s proposed rule, while those that manufacture or export any items that remain on the USML, would continue to register with DDTC. State’s and Commerce’s processes for reviewing export license applications involve opportunities for other Departments to review applications. While DDTC has primary responsibility for reviewing State’s commercial export license applications, other bureaus within State, as well as DOD, also review certain applications, depending on the defense article, defense service, or the destination country. Commerce export license applications also involve an interagency review that includes State, DOD, and the Department of Energy, depending on the item to be exported. Both departments have a process for resolving disagreements among the reviewing bureaus or agencies on the disposition of the application. According to State officials, as part of the interagency review process for Commerce licenses, State has generally reviewed applications for items that have previously moved from the USML to the CCL and would continue to do so for items that would transfer to the CCL under the proposed rules. Moreover, DOD officials told us that DOD intends to review Commerce export license applications for these items during the interagency review process, if the proposed transfer is implemented. This would represent a change from DOD’s current practice to generally not review State’s firearms license applications. DOD officials told us that if the proposed rules are finalized, they believed it is prudent to begin reviewing Commerce license applications for items that would transfer under the proposed rules, at least initially. State and Commerce each maintain their own internal watch lists to screen all parties identified on license applications. A watch list match would trigger further review of the license and ultimately can result in a denial of the license in some cases. State and Commerce also use watch lists as a means of targeting transactions for possible end-use checks to verify legitimacy of end-users of controlled exports. Both departments’ watch lists include any derogatory information they collect internally from their past screening and end-use monitoring of licenses. For example, if information is identified raising questions about the legitimacy of a party to a license during the application review, that information would be used to update the watch list to inform future license application reviews. State’s and Commerce’s watch lists also include information from automated databases maintained by other U.S. agencies as well as information from law enforcement agencies and the intelligence community. State’s watch list contains over 200,000 entries, including sensitive details related to ongoing and previous law enforcement activities, according to State officials. According to Commerce officials, because State has been responsible for export controls of firearms, artillery, and ammunition, its internal watch list is also more likely than Commerce’s to include derogatory information collected from past screening and end-use monitoring related to exports of these items. However, Commerce does not have access to State’s watch list, according to State and Commerce officials. These officials noted that a Commerce licensing officer can ask State to screen an applicant with State’s watch list on a case-by-case basis, although such checks are not done routinely. State and Commerce officials told us that, in anticipation of the transfer of firearms, artillery, and ammunition to Commerce’s responsibility, the two departments are engaged in ongoing discussions to potentially share State’s watch list with Commerce. According to State officials, these discussions involve determining which specific watch list information Commerce would need and State is able to share, depending on the source of the information. State and Commerce also have to resolve the sharing and updating of information using different information technology infrastructures, according to department officials. As of February 2019, the departments had not reached agreement or established a documented process to achieve the goal of sharing watch list information before implementation of the proposed transfer would occur, according to State and Commerce officials. Information sharing is supported by a policy statement included in the ECRA. The statement says that among other factors, the “export control system must ensure that it is transparent, predictable, and timely, has the flexibility to be adapted to address new threats in the future, and allows seamless access to and sharing of export control information among all relevant United States national security and foreign policy agencies.” Without access to State’s watch list, if the proposed rules are finalized, Commerce may lack critical information needed to effectively screen license applicants for firearms and related exports and target possible cases for end-use monitoring to ensure that these exports are used as intended and by legitimate end-users. Both State and Commerce screen license applications for human rights concerns, but the federal law that prohibits exports to the governments of certain foreign countries on human rights grounds applies differently to items under State’s jurisdiction than under Commerce’s. Under Section 502B of the Foreign Assistance Act of 1961, as amended, in general, “no security assistance may be provided to any country the government of which engages in a consistent pattern of gross violations of internationally recognized human rights.” For this provision, “security assistance” is defined in part as any license in effect with respect to the export to or for the armed forces, police, intelligence, or other internal security forces of a foreign country of (1) any defense articles or defense services licensed for export under section 38 of the AECA, or (2) items listed under the 600 series of the CCL. Licenses under Commerce’s jurisdiction generally may not be issued for items defined as “crime control and detection instruments and equipment” to a country, the government of which engages in a consistent pattern of gross violations of internationally recognized human rights. For items under Commerce’s jurisdiction, the Commerce proposed rule specifies that concern for human rights is a regulatory reason for denying a license for firearms and ammunition under Commerce’s Export Administration Regulations (EAR). Within State, the Bureau of Democracy, Human Rights and Labor (DRL) is primarily responsible for screening export license applications to ensure that exports do not involve parties with human rights concerns. According to DRL officials, the bureau reviews applications for exports to specific countries where human rights concerns exist and prioritizes applications for firearms exports because they are often associated with human rights abuses committed by government police and military units. The officials noted, however, that State rarely denies an export license based solely on human rights concerns. If firearms are transferred to Commerce’s responsibility, DRL will continue to have the primary role in screening license applications for human rights as part of the Commerce-led interagency review process, according to DRL officials. For Commerce license applications, however, State’s position would be weighed together with the positions of Commerce, DOD, and Energy, according to Commerce officials. By contrast, for State export license applications, State alone makes the final determination, according to State officials. State and Commerce have different end-user certification requirements. State’s export control regulations require that for certain items, applicants provide a written certification from end-users that they will not re-export, resell, or otherwise dispose of the commodity outside of the country listed on the license. This requirement generally applies to all items on the USML that are designated as Significant Military Equipment, including firearms, and ammunition. In contrast, Commerce generally does not require end-user certification for items on the CCL but does require it when it has not verified the legitimacy of end-users and may also impose this requirement on a case-by-case basis. Written end-user certification provides additional assurance and accountability that end-users will comply with the terms and conditions of the license, according to State officials. It also is a deterrent and provides documentary evidence that can be later used in court, if necessary, according to an official from Immigration and Customs Enforcement (ICE). The AECA states that the Secretary of State shall require reporting on political contributions, gifts, commissions, and fees paid or offered, or agreed to be paid by any person in connection with a commercial sale of an item listed on the USML to or for the armed forces of a foreign country or an international organization. State’s export control regulations also require license applicants to disclose certain payment of political contributions, fees, and commissions for certain sales of defense articles and defense services. This requirement applies to exports of $500,000 or more. Applicants must report political contributions in an aggregate amount of $5,000 or more and paid fees or commissions in an aggregate amount of $100,000 or more. Applicants must provide a letter to DDTC containing specific information about the sale, including the amounts of political contributions, fees, or commissions paid, and the name and nationality of each recipient. The disclosures are intended to ensure that purchases made by foreign governments of U.S. defense articles are based on merit without improper influence. Failure of applicants to comply with these disclosure requirements can result in additional oversight measures and civil penalties. According to an ICE official, this disclosure information may provide valuable information in criminal or civil matters. There is no requirement in the law for these disclosures for items listed on the CCL and Commerce licenses do not require these disclosures. Therefore, this information would no longer be collected as part of the licensing process for firearms, artillery, and ammunition that are proposed for transfer to the CCL, according to Commerce officials. Consistent with export control regulations, there are several circumstances in which some exports proposed for transfer that currently require State licenses would either require fewer or no Commerce licenses if the proposed rules are finalized, according to Commerce. Multiple end-users on one license. State requires licenses to be limited to only one end-user, while Commerce allows multiple end-users on a single license. The applicant for a State export license must provide a purchase order documenting the proposed export to a single end-user and an additional license would be required for each additional end-user. According to Commerce officials, a Commerce license can have multiple end-users associated with a particular consignee, reducing the total number of licenses for which the applicant must apply. Technical data and defense services. State requires licenses for defense services and technical data whereas Commerce’s export controls do not generally apply to defense services and apply to technical data more narrowly than State. State’s regulations define defense services as “the furnishing of assistance (including training) to foreign persons … in the design, development, engineering, manufacture, production, assembly, testing, repair, maintenance, modification, operation, demilitarization, destruction, processing or use of defense articles.” State’s definition of defense services also includes military training of foreign units and forces including publications, training exercises, and military advice. State’s definition of technical data includes information, such as blueprints, drawings, or instructions. Commerce’s export control regulations generally do not apply to services. For example, training in the basic operation of a firearm controlled by Commerce would not be subject to export controls, according to State officials. In addition, Commerce’s regulations do not control technology or software, if it is “available to the public without restrictions.” For example, Commerce officials told us that Commerce would not require an export license for the posting of instructions for 3D printing of firearms on the internet, if they were publicly available without restrictions. Minimum level of U.S.-origin content. Items subject to State’s controls require a license when they are incorporated into a foreign-made product regardless of the percentage of controlled U.S. content in that product. Commerce does not require a license for items when they are incorporated into foreign-made items unless the controlled U.S.-origin content of a foreign-made product exceeds the applicable minimum percentage which, according to Commerce officials, may be 10 or 25 percent, depending on the destination. This minimum level of U.S.-origin content is referred to as “de minimis treatment.” Commerce’s proposed rule states that de minimis treatment in Commerce’s regulations would apply for all foreign-made items proposed for transfer to the CCL, unless they are being exported to a country that is subject to a United States arms embargo, in which case there would be no minimum threshold for U.S.-origin content. License exceptions. State regulations contain some country-based license exceptions, including for exports to Canada and, more narrowly, to Australia and the United Kingdom whereas Commerce has several different license exceptions under its regulations. For example, Commerce regulations have the “Strategic Trade Authorization” (STA) exception that permits exports of certain items to countries determined to be low risk, which includes NATO partners and other close allies, of which 37 are eligible for a broader STA authorization and seven are eligible for a much narrower STA authorization. Commerce’s proposed rule specifies that it would revise Commerce’s regulations to make firearms and most parts, components, accessories, and attachments ineligible for the STA license exception. However, Commerce estimates that 450 to 650 license applications per year involving certain eligible items would still be authorized under STA exceptions if the proposed rules are finalized. Commerce also has a “Limited Value Shipment” exception, which is available for proposed exports of certain less sensitive firearms parts and components with a value of $500 or less per shipment based on the actual selling price or fair market value. Commerce’s proposed rule specifies that this exception would only be available for certain parts, components, and accessories and attachments for firearms; complete firearms would be ineligible for this exception. State offers a similar exemption but only for licenses with a value of $100 or less, based on the wholesale price. State and Commerce both conduct end-use monitoring to verify the reliability of foreign end-users and legitimacy of proposed transactions and to provide reasonable assurance of compliance with the terms of the license and proper use of the licensed items. State recommends that end-use checks involve a site visit whenever possible, while Commerce policy requires that the end-use check include a physical verification on-site with a party to the transaction, according to Commerce officials. State and Commerce also apply their own means of risk-based targeting to select the licenses or exports that will undergo end-use monitoring, however, similarities exist involving selection criteria. For example, State and Commerce may target transactions that involve unfamiliar foreign parties, unusual shipping routes, or derogatory information from watch lists, according to the departments. The number of end-use checks conducted by State averaged about 1.3 percent of its license applications, and those conducted by Commerce averaged about 3.3 percent of its applications from fiscal years 2013-2017. State and Commerce end-use checks may result in either “favorable” or “unfavorable” findings. Commerce may also categorize an end-use check as “unverified.” An “unfavorable” or “unverified” result occurs if the end- use check cannot verify information in the license or reveals facts that are inconsistent with the license. For either State or Commerce, an unfavorable end-use check can lead to denying applications, revoking licenses, removing parties from licenses, updating the watch list, or making referrals to U.S. law enforcement agencies for investigation, according to a State report and Commerce officials. State closed 166 of 766, or 22 percent, of end-use monitoring cases as “unfavorable” in fiscal years 2013-2017 for firearms, artillery, and ammunition licenses. State’s three most common reasons for an unfavorable finding for end-use checks for firearms, artillery, and ammunition were derogatory information on a foreign party, inability to confirm order or receipt of goods, and involvement of an unlicensed party. State relies on U.S. embassy or consulate staff in the country or countries involved in the transaction to conduct its end-use checks. Commerce relies primarily on Export Control Officers (ECOs) positioned overseas to conduct end-use checks. ECOs conducted an average of about 60 percent of Commerce’s end-use checks per year from fiscal years 2013 to 2017. According to Commerce officials, Commerce had a total of nine ECO positions in Beijing, Dubai, Frankfurt, Hong Kong, Istanbul, New Delhi, and Singapore, as of October 2018 (see fig. 5). Six of these nine positions were filled as of this date. The ECOs have areas of responsibility covering multiple countries within their geographic region. For the remaining 40 percent of end-use checks, Commerce relied primarily on its “Sentinel Program” in which BIS special agents based in domestic field offices, along with other responsibilities, travel to destination countries not covered by ECOs to conduct end-use checks. In addition, a small percentage of Commerce’s end-use checks are conducted by Foreign Commercial Service officers or other personnel stationed at U.S. embassies, according to Commerce officials. State conducted 766 end-use checks for firearms, artillery, and ammunition in fiscal years 2013-2017 with the largest share, over 40 percent, in the Western Hemisphere (see fig. 6). None of Commerce’s overseas ECO positions are located in this region nor do any cover it within their areas of responsibility. According to Commerce officials, the number and locations of end-use checks for firearms, artillery, and ammunition, if these items are transferred to the CCL, will depend on how exports of these items factor into the department’s existing targeting criteria. To the extent that Commerce needs to conduct end-use checks for these items in the Western Hemisphere, Commerce officials told us that they plan to cover these checks via the Sentinel Program and, where necessary, through checks by Foreign Commercial Service Officers. The officials noted that they plan to reassess their end-use monitoring efforts after items are transferred to the CCL if the proposed rules are finalized. End-use checks include pre-license checks in support of the license application review or post-shipment verifications after the license has been approved and items have shipped. As shown in figure 7, more than 50 percent of State’s end-use checks specifically for firearms, artillery, and ammunition licenses from fiscal years 2013 to 2017 were pre-license checks. Conversely, about 90 percent of Commerce’s end- use checks for all items subject to the EAR for this period were post- shipment verifications. Commerce noted that it conducts mostly post- shipment verifications because it controls a higher share than State of items that are exported without a license. The AECA requires State to notify Congress before State can approve certain export licenses for firearms, artillery, and ammunition. These notification requirements depend on the proposed export value and type of export, among other factors. For example, the AECA requires State to notify Congress of proposed licenses for the export of USML Category I firearms in the amount of $1 million or more. Additionally, State must notify Congress of proposed licenses for commercial agreements that involve the overseas manufacture of certain USML items, including many firearms, artillery, and ammunition items, regardless of the proposed value. During fiscal years 2013 to 2017, State identified 240 export license applications involving firearms, artillery, and ammunition that required congressional notification, totaling approximately $2.5 billion. Additionally, State identified 41 license applications for commercial technical assistance or manufacturing license agreements involving the overseas manufacture of firearms, artillery, and ammunition that required congressional notification, totaling approximately $5.7 billion. According to State and Commerce officials, these congressional notification requirements would no longer apply to firearms, artillery, and ammunition that move from State’s to Commerce’s export control responsibility because the requirements apply specifically to USML controlled items. The proposed rule transferring firearms to Commerce’s responsibility does not revise Commerce’s export control regulations to add a congressional notification requirement for firearms, according to Commerce officials. The Foreign Assistance Act, as amended, requires State to report to Congress annually on military assistance and military exports to the governments of each foreign country and international organization and specifies that the report include “a statement of the aggregate dollar value and quantity of semiautomatic assault weapons, or spare parts for such weapons.” The Act also requires that State post all unclassified information from this report on the internet. To comply with this requirement, State posts an annual report that includes the aggregate dollar value and quantity of defense articles and services, by USML category, licensed to each foreign country and international organization, as well as data on the actual shipments occurring during the fiscal year. The report also includes an appendix that breaks out exports specifically for the USML sub-category I(a), which includes non-automatic and semi- automatic firearms, and sub category I(h), which includes firearms components, parts, accessories, and attachments. This reporting requirement only applies to exports of items on the USML, which are licensed by State under the AECA, but does not apply to exports controlled by Commerce. This information on exports, by country, would no longer be available for firearms and other items from Categories I-III of the USML after they are transferred to the CCL if the proposed rules are finalized, according to Commerce officials. The statutory penalties available for criminal violations of export control laws are the same regardless of whether the items are on the USML and controlled by State or on the CCL and controlled by Commerce. Criminal violations may result in fines up to $1 million and prison terms up to 20 years, or both. Under the AECA, civil violations of State’s export controls may result in a fine of up to $500,000 but, according to State officials, can be much higher based on inflation under the Federal Civil Penalties Inflation Adjustment Act of 1990, as amended. State told us that actual civil penalties for civil violations in 2018 ranged from $824,959 to $1,134,602. By contrast, the ECRA set the penalty for civil violations of Commerce’s export controls at up to $300,000 or twice the value of the transaction that is the basis of the violation, whichever is of greater value. According to Commerce officials, this can substantially increase the monetary penalty for civil violations. Criminal violations of either State’s or Commerce’s export control laws may result in prohibiting the violator from involvement in future exports of controlled items. The AECA also precludes the issuance of State licenses to persons convicted of violating certain federal laws, such as the Foreign Corrupt Practices Act. Similarly, Commerce can deny the export privileges, including the ability to obtain a license, of companies and individuals for a period of 10 years from the date of conviction for violating certain federal laws. This prohibition can be expanded to include other related parties, such as those connected with the denied person by virtue of affiliation, ownership, or control. Agencies with responsibility for export control enforcement can vary depending on whether items are controlled by State or Commerce. According to DHS officials, ICE has jurisdiction to investigate potential export control violations and U.S. Customs and Border Protection has primary enforcement responsibility for export control violations at the border, seaports, and airports. The Federal Bureau of Investigation (FBI) can also investigate these cases involving items controlled by either State or Commerce. According to Commerce, the Office of Export Enforcement in BIS has over 100 special agents in U.S.-based field offices authorized to investigate potential violations of Commerce’s export control laws. These investigative resources would be available, in addition to DHS and FBI, to address illegal firearms trafficking if the proposed transfer is implemented, according to Commerce officials. State expects to lose revenue from registration fees if the proposed transfer of firearms, artillery, and ammunition to Commerce is implemented. State estimates in its proposed rule that the transfer would result in about 10,000 fewer license applications per year for Category I- III items—a reduction of about 26 percent from the 38,862 applications that State processed in fiscal year 2017. State estimates a recurring annual registration fee revenue loss of about $2.5 million, according to its proposed rule. State officials told us, if the proposed rules become final, there would be additional revenue declines from an uncertain drop in the number of registrants that State cannot estimate. They explained that because many manufacturers and exporters would likely be involved in items controlled by State as well as Commerce, they would still need to register with State. Others involved only in items moving to Commerce would no longer have to register with State. For example, according to State officials, a manufacturer of both semi-automatic weapons that the proposed rules identify for transfer to the CCL and fully automatic weapons that would stay on the USML would still be required to register with State, if the proposed rules are finalized. State officials noted that the decline in the number of license applications resulting from previous transfers of items from the USML to the CCL has not produced a proportional decline in registration revenue. According to data provided by State, registration revenue has dropped less than 25 percent from about $47 million in fiscal year 2013 to about $36 million in fiscal year 2017, while the number of export license applications has dropped more than 50 percent from about 83,000 to almost 39,000. With the decline in license workload that State expects would result if the proposed rules are finalized, State officials told us that four contractors currently responsible for reviewing licenses for firearms and ammunition in DDTC could be moved to other teams with vacancies in order to review licenses for other controlled items. On the other hand, State’s Bureau of International Security and Nonproliferation (ISN), which has lead responsibility at State for reviewing Commerce licenses for items transferring from the USML to the CCL, expects to see an increase in its workload. An ISN official told us his bureau could potentially need an additional 2.5 full-time equivalent staff to review items transferred to the CCL as part of Commerce’s interagency review process. Commerce estimates in its proposed rule that it would gain 6,000 additional license applications from the proposed transfer—an increase of about 18 percent above the 34,142 license applications it reviewed in fiscal year 2017. Commerce officials told us that the increased workload to review license applications will also create more work for some related activities. For example, Commerce expects the number of investigative leads and export enforcement investigations to include more firearms- related actions. However, Commerce officials told us they have not estimated the magnitude of these changes. Commerce officials told us they believe they have enough resources to absorb the increase in workload. They noted that they have flexibility to shift license review staff to meet demand created by the additional licenses, if necessary. In addition, BIS received an 18 percent increase in full-time equivalent staff positions, from 367 to 432, in fiscal year 2018. This increase was in response to workload demands created by previous transfers of items from the USML to the CCL, according to Commerce officials. Commerce officials told us that they will continue to assess workload data after the proposed transfer is implemented to determine whether they have adequate staff levels to meet increased workload demands. If finalized, the proposed rules to transfer certain firearms, artillery, and ammunition from Categories I-III of the USML to the CCL would apply Commerce’s export control system to these items instead of State’s. However, critical information needed to effectively screen applicants and target licenses for end-use monitoring may be unavailable to Commerce unless State shares its watch list data. Further, because State has been responsible for export controls of firearms, artillery, and ammunition, its watch list is more likely than Commerce’s to include derogatory information collected from past screening and end-use monitoring related to exports of these items, according to Commerce officials. While State and Commerce officials said that they have held discussions regarding how to share relevant information from their internal watch lists, as of February 2019, they had not reached any agreement on how to share watch lists if the proposed rules are finalized. Without such an agreement or process to share State’s watch list, Commerce may lack critical information needed to ensure that items proposed for transfer are used as intended and by legitimate end-users. We are making a total of two recommendations, including one to State and one to Commerce. If responsibility for controlling the exports of certain firearms, artillery, and ammunition is transferred from State to Commerce, the Secretary of State should ensure that the Under Secretary of State for Arms Control and International Security Affairs develops a process for sharing State’s internal watch list with Commerce to enhance oversight of these items. (Recommendation 1) If responsibility for controlling the exports of certain firearms, artillery, and ammunition is transferred from State to Commerce, the Secretary of Commerce should ensure that the Under Secretary of Commerce for Industry and Security develops a process for receiving State’s internal watch list and integrating it into Commerce’s licensing review process to enhance oversight of these items. (Recommendation 2) We provided a draft of this report to State, Commerce, DOD, DHS, and DOJ for review and comment. In their written comments, reproduced in appendixes III and IV, State and Commerce agreed with our recommendations. Commerce provided some minor revisions to the recommendation, which we incorporated. DOD, DHS, and DOJ did not provide written comments. In addition, State, Commerce, and DOJ provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretaries of State, Commerce, Defense, and Homeland Security; and the Attorney General of the United States. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8612 or gianopoulosk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Our objectives were to assess (1) the volume and value of commercial export license applications State Department (State) reviewed for firearms, artillery, and ammunition—Categories I-III of the U.S. Munitions List (USML)—in fiscal years 2013-2017, (2) how certain export controls differ between State and Commerce, and (3) what is known about the resource implications for State and Commerce due to the proposed transfer. To assess the volume and value of export license applications for USML Category I-III firearms, artillery, and ammunition that State reviewed during fiscal years 2013 to 2017, we obtained data from the interagency export licensing database, USXPORTS. USXPORTS is the system of record for all munitions and dual-use export license applications and adjudications, and is maintained by the Defense Technology Security Administration, within the Department of Defense (DOD). The data on USXPORTS originates from private companies applying for export licenses which, in the case of munitions, State is responsible for adjudicating. The agencies use this database to review and adjudicate applications, and also to report back to the applicants. We interviewed officials from State’s Directorate of Defense Trade Controls (DDTC) in State’s Bureau of Political and Military Affairs to understand the data and identify any limitations on how we use them. We analyzed the data to describe the number and reported value of export license applications, the USML items in the applications, and the reported destination country, among other characteristics. We assessed these data and found them to be sufficiently reliable for the purpose of conducting these analyses, but recognized that approved applications may not necessarily result in actual exports. We also noted some minor data limitations in our report, such as the fact that amendments to export license applications are not associated with destination countries. We did not independently audit the underlying data submitted to DDTC by private companies. To analyze how certain export controls differ between State and Commerce, we reviewed the departments’ proposed rules, relevant laws and regulations, agency guidance, and annual reports related to State’s and Commerce’s export controls. We also interviewed officials from Commerce’s Bureau of Industry and Security; DDTC; State’s Bureau of Democracy, Human Rights and Labor; State’s Bureau of International Security and Nonproliferation; Immigration and Customs Enforcement and U.S. Customs and Border Protection in the Department of Homeland Security; and the Defense Technology Security Administration. We sought to present differences between State’s and Commerce’s export controls that are potentially relevant for items proposed for transfer from the USML to the CCL, rather than every possible distinction between the two departments’ export control systems. To describe the number of export license applications for firearms, artillery, and ammunition that required congressional notification, we reviewed the licensing data from the USXPORTS database. To describe the end-use monitoring conducted on exports of firearms, artillery, and ammunition, we extracted data from State’s Defense Trade Application database and interviewed agency officials to understand the data. We analyzed the data by the number of checks per year, the proportion of pre-license checks to post- shipment checks, the countries where the checks were conducted, and the outcome of the checks. We assessed these data and found them to be sufficiently reliable for these purposes. To assess what is known about the resource implications for State and Commerce due to the proposed transfer, we held discussions with State and Commerce officials, and reviewed annual budget documents and other agency reports. To better understand State’s estimated reduction of 10,000 license applications per year and Commerce’s estimated gain of 6,000 licenses that would result from the proposed transfer of items from the USML to the CCL, we reviewed State’s fiscal year 2013-2017 export license data and the proposed rules. We also discussed the estimates with agency officials. Commerce officials told us that their estimate was fairly broad, based on State’s estimate and their knowledge and experience of differences between the two agencies’ license requirements that account for the difference between the two estimates. We were not able to independently assess the accuracy of either estimate because the license data we collected from State were not disaggregated to identify which items on license applications would be transferring to the CCL under the proposed rules and which would be staying on the USML. Each State license application can involve multiple items across multiple USML Sub-Categories. We also reviewed the number of full-time equivalent staff responsible for export control activities and State’s annual revenue from registration fees paid by manufacturers, exporters, and brokers involved in items on the USML. We discussed State’s registration data with agency officials and while we assessed these data as sufficiently reliable for descriptive purposes, we also determined that these data could not be used to generate reliable estimates about the resource implications for the Department of State because there was no clear pattern in the relationship between applications, registrants, and revenue in the data provided. We conducted this performance audit from February 2018 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Defense articles and services subject to export controls under the Department of State’s jurisdiction are listed in the 21 categories of the United States Munitions List (USML). Table 3 shows the 21 USML categories and the dates of rule changes under export control reform that transferred certain items within these categories to the Commerce Control List (CCL). The CCL is divided into ten broad categories and each category is further subdivided into five product groups (see table 4). In addition to the individual named above, Drew Lindsey (Assistant Director), Howard Cott (Analyst in Charge), Ashley Alley, Martin de Alteriis, Neil Doherty, Adam Peterson, and Aldo Salerno made significant contributions to this report.", "summary": "The U.S. government implements an export control system to manage risks associated with exporting sensitive items while facilitating legitimate trade. State currently controls the export of most firearms, artillery, and ammunition. Regulatory changes proposed by State and Commerce would transfer this responsibility for many of these items to Commerce, which implements export controls under different legal and regulatory authorities. The proposed changes are part of a larger export control reform effort since 2010 to transfer control of less sensitive items from State to Commerce. GAO was asked to review the proposed changes to export controls of firearms, artillery, and ammunition. This report assesses (1) the volume and value of commercial export license applications State reviewed for these items in fiscal years 2013-2017, (2) how certain export controls differ between State and Commerce, and (3) what is known about the resource implications for State and Commerce due to the proposed transfer. GAO reviewed the proposed rules and related laws and regulations; analyzed data and documents related to licensing, end-use monitoring, and staff resources; and interviewed agency officials. The Department of State (State) reviewed approximately 69,000 commercial export license applications for firearms, artillery, and ammunition valued at up to $45.4 billion during fiscal years 2013 to 2017. About two-thirds of these applications were for firearms, and the majority involved the export of non-automatic and semi-automatic firearms, which are among the items proposed for transfer from State to Department of Commerce (Commerce) control. GAO identified several differences in Commerce's and State's export controls including those related to registration, licensing, end-use monitoring, and congressional notification that, according to the agencies, would apply to firearms, artillery, and ammunition proposed for transfer. Some of these differences are due to varying requirements in applicable laws and regulations. For example, the law requires manufacturers, exporters, and brokers to register with State for items controlled by State but not for items controlled by Commerce. Additionally, while Commerce and State both screen parties to licenses against relevant watch lists, Commerce officials said they do not have direct access to State's internal watch list, which contains derogatory information from past screening of licenses for firearms, artillery, and ammunition exports. State and Commerce officials stated that, while they have held some discussions, they have not established a process for sharing watch list information. Without access to State's watch list, Commerce may lack critical information to effectively screen parties to exports of firearms and related items. State and Commerce also both have end-use monitoring programs to confirm the legitimacy of end-users but some differences exist. For example, State relies on embassy staff to conduct end-use monitoring whereas Commerce relies primarily on several officers positioned overseas specifically for this purpose. In addition, a statutory requirement to notify Congress of proposed firearms exports over $1 million would no longer apply to firearms that transfer from State to Commerce, according to Commerce officials. According to the proposed rules and agency officials, the proposed transfer, if finalized, would result in a decline in licenses and revenues for State and an increase in licenses for Commerce, but the precise extent of these changes is unknown. State estimates that the transfer would result in a decline in revenue from registration fees but officials stated it is difficult to predict the extent of this decline. Commerce officials stated that they expected their licensing and enforcement workload to increase as a result of the transfer, if finalized, but they believe they have sufficient staff resources available to absorb the increase. GAO recommends that if the proposed regulatory changes become final, State and Commerce develop a process for sharing State's internal watch list with Commerce to enhance oversight of firearms, artillery, and ammunition exports. State and Commerce agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Federal agencies conduct a variety of procurements that are reserved for small business participation through small business set-asides. These set-asides can be for small businesses in general, or they can be specific to small businesses that meet additional eligibility requirements in the Service-Disabled Veteran-Owned Small Business, Historically Underutilized Business Zone (HUBZone), 8(a) Business Development (8(a)), and WOSB programs. The WOSB program enables federal contracting officers to identify and establish a sheltered market, or set- aside, for competition among women-owned small businesses (WOSB) and economically disadvantaged women-owned small businesses (EDWOSB) in certain industries. WOSBs can receive set-asides in industries in which SBA has determined that women-owned small businesses are substantially underrepresented. To determine these industries, SBA is required to conduct a study to determine which North American Industry Classification System (NAICS) codes are eligible under the program and to report on such studies every 5 years. Additionally, businesses must be at least 51 percent owned and controlled by one or more women who are U.S. citizens to participate in the WOSB program. The owner must provide documents demonstrating that the business meets program requirements, including a document in which the owner attests to the business’s status as a WOSB or EDWOSB. According to SBA, as of early October 2018, there were 13,224 WOSBs and 4,488 EDWOSBs registered in SBA’s online certification database. SBA’s Office of Government Contracting administers the WOSB program by, among other things, promulgating regulations and conducting eligibility examinations of businesses that receive contracts under a WOSB or EDWOSB set-aside. According to SBA, as of October 2018, there were two full-time staff within the Office of Government Contracting whose primary responsibility was the WOSB program. Initially, the program’s statutory authority allowed WOSBs to be self- certified by the business owner or certified by an approved third-party national certifying entity as eligible for the program. Self-certification is free, but some third-party certification options require businesses to pay a fee. Each certification process requires businesses to provide signed representations attesting to their WOSB or EDWOSB eligibility. Businesses must provide documents supporting their status before submitting an offer to perform the requirements of a WOSB set-aside contract. In August 2016, SBA launched certify.sba.gov, which is an online portal that allows firms participating in the program to upload required documents and track their submission and also enables contracting officers to review firms’ eligibility documentation. According to the Federal Acquisition Regulation, contracting officers are required to verify that all required documentation is present in the online portal when selecting a business for an award. In addition, businesses must register and attest to being a WOSB in the System for Award Management, the primary database of vendors doing business with the federal government. In 2011, SBA approved four organizations to act as third-party certifiers. According to SBA data, these four third-party certifiers completed a total of about 3,400 certifications in fiscal year 2017. In 2014 we reviewed the WOSB program and found a number of deficiencies in SBA’s oversight of the four SBA-approved third-party certifiers and in SBA’s eligibility examination processes, and we made related recommendations for SBA. In addition, in 2015 and 2018 the SBA Office of Inspector General (OIG) reviewed the WOSB program and also found oversight deficiencies, including evidence of WOSB contracts set aside for ineligible firms. In both reports, the SBA OIG also made recommendations for SBA. Further, in July 2015, we issued GAO’s fraud risk framework, which provides a comprehensive set of key components and leading practices that serve as a guide for agency managers to use when developing efforts to combat fraud in a strategic, risk-based way. As of early May 2019, SBA had implemented one of the three changes that the 2015 NDAA made to the WOSB program—sole-source authority. The two other changes—authorizing SBA to implement its own certification process for WOSBs and requiring SBA to eliminate the WOSB self-certification option—had not been implemented. The 2015 NDAA did not require a specific time frame for SBA to update its regulations. SBA officials have stated that the agency will not eliminate self-certification until the new certification process for the WOSB program is in place, which they expect to be completed by January 1, 2020. In September 2015, SBA published a final rule to implement sole-source authority for the WOSB program (effective October 2015). Among other things, the rule authorized contracting officers to award a contract to a WOSB or EDWOSB without competition, provided that the contracting officer’s market research cannot identify two or more WOSBs or EDWOSBs in eligible industries that can perform the requirements of the contract at a fair and reasonable price. In the final rule, SBA explained that it promulgated the sole-source rule before the WOSB certification requirements for two reasons. First, the sole-source rule could be accomplished by simply incorporating the statutory language into the regulations, whereas the WOSB certification requirements would instead require a prolonged rulemaking process. Second, SBA said that addressing all three regulatory changes at the same time would delay the implementation of sole-source authority. As of early May 2019, SBA had not published a proposed rule for public comment to establish a new certification process for the WOSB program. Previously, in October 2017, an SBA official stated that SBA was about 1–2 months away from publishing a proposed rule. However, in June 2018, SBA officials stated that a cost analysis would be necessary before the draft rule could be sent to the Office of Management and Budget for review. In response to the SBA OIG recommendation that SBA implement the new certification process, SBA stated that it would implement a new certification process by January 1, 2020. Further, in June 2018, SBA officials told us that they were evaluating the potential costs of a new certification program as part of their development of the new certification rule. On May 3, 2019, SBA officials explained that they expected to publish the proposed rule within a few days. In December 2015, SBA published an advance notice of proposed rulemaking to solicit public comments to assist the agency with drafting a proposed rule to implement a new WOSB certification program. In the notice, SBA stated that it intends to address the 2015 NDAA changes, including eliminating the self-certification option, through drafting regulations to implement a new certification process. The advance notice requested comments on various topics, such as how well the current certification processes were working, which of the certification options were feasible and should be pursued, whether there should be a grace period for self-certified WOSB firms to complete the new certification process, and what documentation should be required. Three third-party certifiers submitted comments in response to the advance notice of proposed rulemaking, and none supported the option of SBA acting as a WOSB certifier. One third-party certifier commented that such an arrangement is a conflict of interest given that SBA is also responsible for oversight of the WOSB program, and two certifiers commented that SBA lacked the required resources. The three third-party certifiers also asserted in their comments that no other federal agency should be allowed to become an authorized WOSB certifier, with one commenting that federal agencies should instead focus on providing contracting opportunities for women-owned businesses. All three certifiers also proposed ways to improve the current system of third-party certification—for example, by strengthening oversight of certifiers or expanding their number. The three certifiers also suggested that SBA move to a process that better leverages existing programs with certification requirements similar to those of the WOSB program, such as the 8(a) program. In the advance notice, SBA asked for comments on alternative certification options, such as SBA acting as a certifier or limiting WOSB program certifications to the 8(a) program and otherwise relying on state or third-party certifiers. SBA has not fully addressed deficiencies we identified in our October 2014 report, and these recommendations remain open. First, we reported that SBA did not have formal policies for reviewing the performance of its four approved third-party certifiers, including their compliance with their agreements with SBA. Further, we found that SBA had not developed formal policies and procedures for, among other things, reviewing the monthly reports that certifiers submit to SBA. As a result, we recommended that SBA establish comprehensive procedures to monitor and assess the performance of the third-party certifiers in accordance with their agreements with SBA and program regulations. In response to our October 2014 recommendation, in 2016 SBA conducted compliance reviews of the four SBA-approved third-party certifiers. The compliance reviews included an assessment of the third- party certifiers’ internal certification procedures and processes, an examination of a sample of applications from businesses that the certifiers deemed eligible and ineligible for certification, and an interview with management staff. SBA officials said that SBA’s review team did not identify significant deficiencies in any of the four certifiers’ processes and found that all were generally complying with their agreements. However, one compliance review report described “grave concerns” that a third- party certifier had arbitrarily established eligibility requirements that did not align with WOSB program regulations and used them to decline firms’ applications. SBA noted in the report that if the third-party certifier failed to correct this practice, SBA could terminate the agreement. As directed by SBA, the third-party certifier submitted a letter to SBA outlining actions it had taken to address this issue, among others. In January 2017, SBA’s Office of Government Contracting updated its written Standard Operating Procedures (SOP) to include policies and procedures for the WOSB program, in part to address our October 2014 recommendation. The 2017 SOP discusses what a third-party-certifier compliance review entails, how often the reviews are to be conducted, and how findings are to be reported. The 2017 SOP notes that SBA may initiate a compliance review “at any time and as frequently as the agency determines is necessary.” In March 2019, SBA provided an updated SOP, which includes more detailed information on third-party compliance reviews, such as how SBA program analysts should prepare for the review. However, the updated SOP does not provide specific time frames for how frequently the compliance reviews are to be conducted. In addition, in April 2018, SBA finalized a WOSB Program Desk Guide that discusses how staff should prepare for a compliance review of a third-party certifier, review certification documents, and prepare a final report. In March 2019, SBA provided GAO with an updated WOSB Program Desk Guide that contains information comparable to that in the 2018 version. Both Desk Guides do not describe specific activities designed to oversee third-party certifiers on an ongoing basis. Per written agreements with SBA, third-party certifiers are required to submit monthly reports that include the number of WOSB and EDWOSB applications received, approved, and denied; identifying information for each certified business, such as the business name; concerns about fraud, waste, and abuse; and a description of any changes to the procedures the organizations used to certify businesses as WOSBs or EDWOSBs. In our October 2014 report, we noted that SBA had not followed up on issues raised in the monthly reports and had not developed written procedures for reviewing them. At that time, SBA officials said that they were unaware of the issues identified in the certifiers’ reports and that the agency was developing procedures for reviewing the monthly reports but could not estimate a completion date. In interviews for our March 2019 report, SBA officials stated that SBA still does not use the third-party certifiers’ monthly reports to regularly monitor the program. Specifically, SBA does not review the reports to identify any trends in certification deficiencies that could inform program oversight. Officials said the reports generally do not contain information that SBA considers helpful for overseeing the WOSB program, but staff sometimes use the reports to obtain firms’ contact information. SBA’s updated 2019 SOP includes information on reviews of third-party certifier monthly reports, but it does not contain information on how staff would analyze the reports or how these reports would inform SBA’s oversight of third-party certifiers and related compliance activities, such as eligibility examinations. On May 3, 2019, SBA officials stated that, earlier in the week, they had initiated monthly meetings with the third-party certifiers. SBA officials explained that they intended to continue holding these monthly meetings to discuss best practices and potential issues related to the approval and disapproval of firms and to improve collaboration. Although SBA has taken steps to enhance its written policies and procedures for oversight of third-party certifiers, it does not have plans to conduct further compliance reviews of the certifiers and does not intend to review certifiers’ monthly reports on a regular basis in a way that would inform its oversight activities. SBA officials said that third-party certifier oversight procedures would be updated, if necessary, after certification options have been clarified in the final WOSB certification rule. However, ongoing oversight activities, such as regular compliance reviews, could help SBA better understand the steps certifiers have taken in response to previous compliance review findings and whether those steps have been effective. In addition, leading fraud risk management practices include identifying specific tools, methods, and sources for gathering information about fraud risks, including data on trends from monitoring and detection activities, as well as involving relevant stakeholders in the risk assessment process. Without procedures to regularly monitor and oversee third-party certifiers, SBA cannot provide reasonable assurance that certifiers are complying with program requirements and cannot improve its efforts to identify ineligible firms or potential fraud. Further, it is unclear when SBA’s final rule will be implemented. As a result, we maintain that our previous recommendation should be addressed—that is, that the Administrator of SBA should establish and implement comprehensive procedures to monitor and assess the performance of certifiers in accordance with the requirements of the third-party certifier agreement and program regulations. SBA also has not fully addressed deficiencies we identified in our October 2014 report related to eligibility examinations. We found that SBA lacked formalized guidance for its eligibility examination processes and that the examinations identified high rates of potentially ineligible businesses. As a result, we recommended that SBA enhance its examination of businesses that register for the WOSB program to ensure that only eligible businesses obtain WOSB set-asides. Specifically, we suggested that SBA should take actions such as (1) completing the development of procedures to conduct annual eligibility examinations and implementing such procedures; (2) analyzing examination results and individual businesses found to be ineligible to better understand the cause of the high rate of ineligibility in annual reviews and determine what actions are needed to address the causes, and (3) implementing ongoing reviews of a sample of all businesses that have represented their eligibility to participate in the program. SBA has taken some steps to implement our recommendation, such as including written policies and procedures for WOSB program eligibility examinations in an SOP and a Desk Guide. However, SBA does not collect reliable information on the results of its annual eligibility examinations. According to SBA officials, SBA has conducted eligibility examinations of a sample of businesses that received WOSB program set-aside contracts each year since fiscal year 2012. However, SBA officials told us that the results of annual eligibility examinations—such as the number of businesses found eligible or ineligible—are generally not documented. As a result, we obtained conflicting data from SBA on the number of examinations completed and the percentage of businesses found to be ineligible in fiscal years 2012 through 2018. For example, based on previous information provided by SBA, we reported in October 2014 that in fiscal year 2012, 113 eligibility examinations were conducted and 42 percent of businesses were found to be ineligible for the WOSB program. However, during our more recent review, we received information from SBA indicating that 78 eligibility examinations were conducted and 37 percent of businesses were found ineligible in fiscal year 2012. In addition, SBA continues to have no mechanism to look across examinations for common eligibility issues to inform the WOSB program. As we noted in 2014, by not analyzing examination results broadly, the agency is missing opportunities to obtain meaningful insights into the program, such as the reasons many businesses are deemed ineligible. Further, SBA still conducts eligibility examinations only of firms that have already received a WOSB award. In our October 2014 report, we concluded that this sampling practice restricts SBA’s ability to identify potentially ineligible businesses prior to a contract award. SBA officials said that while some aspects of the sample characteristics have changed since 2012, the samples still generally consist only of firms that have been awarded a WOSB set-aside. Restricting the samples in this way limits SBA’s ability to better understand the eligibility of businesses before they apply for and are awarded contracts, as well as its ability to detect and prevent potential fraud. We recognize that SBA has made some effort to address our previous recommendation by documenting procedures for conducting annual eligibility examinations of WOSB firms. However, leading fraud risk management practices state that federal program managers should design control activities that focus on fraud prevention over detection and response, to the extent possible. Without maintaining reliable information on the results of eligibility examinations, developing procedures for analyzing results, and expanding the sample of businesses to be examined to include those that did not receive contracts, SBA limits the value of its eligibility examinations and its ability to reduce ineligibility among businesses registered to participate in the WOSB program. These deficiencies also limit SBA’s ability to identify potential fraud risks and develop any additional control activities needed to address these risks. As a result, the program may continue to be exposed to the risk of ineligible businesses receiving set-aside contracts. In addition, in light of these continued deficiencies, the implementation of sole-source authority without addressing the other changes made by the 2015 NDAA could increase program risk. For these reasons, we maintain that our previous recommendation that SBA enhance its WOSB eligibility examination procedures should be addressed. SBA has also not addressed previously identified issues with WOSB set- asides awarded under ineligible industry codes. In 2015 and 2018, the SBA OIG reported instances in which WOSB set-asides were awarded using NAICS codes that were not eligible under the WOSB program, and our analysis indicates that this problem persists. Specifically, our analysis of data from the Federal Procurement Data System–Next Generation (FPDS–NG) on all obligations to WOSB program set-asides from the third quarter of fiscal year 2011 through the third quarter of fiscal year 2018 found the following: 3.5 percent (or about $76 million) of WOSB program obligations were awarded under NAICS codes that were never eligible for the WOSB program; 10.5 percent (or about $232 million) of WOSB program obligations made under an EDWOSB NAICS code went to women-owned businesses that were not eligible to receive awards in EDWOSB- eligible industries; and 17 of the 47 federal agencies that obligated dollars to WOSB program set-asides during the period used inaccurate NAICS codes in at least 5 percent of their WOSB set-asides (representing about $25 million). According to SBA officials we spoke with, WOSB program set-asides may be awarded under ineligible NAICS codes because of human error when contracting officers are inputting data in FPDS–NG or because a small business contract was misclassified as a WOSB program set-aside. Rather than review FPDS–NG data that are inputted after the contract is awarded, SBA officials said that they have discussed options for working with the General Services Administration to add controls defining eligible NAICS codes for WOSB program set-aside opportunities on FedBizOpps.gov—the website that contracting officers use to post announcements about available federal contracting opportunities. However, SBA officials said that the feasibility of this option was still being discussed and that the issue was not a high priority. Additionally, as of November 2018, the WOSB program did not have targeted outreach or training that focused on specific agencies’ use of NAICS codes, and SBA officials did not identify any targeted outreach or training provided to specific agencies to improve understanding of WOSB NAICS code requirements (or other issues related to the WOSB program). On May 6, 2019, an SBA official provided information that SBA has initiated a review to determine federal agencies’ use of ineligible NAICS codes and that SBA plans to share the findings with agencies and also provide training to procurement center representatives. Congress authorized SBA to develop a contract set-aside program specifically for WOSBs and EDWOSBs to address the underrepresentation of such businesses in specific industries. In addition, federal standards for internal control state that management should design control activities to achieve objectives and respond to risks, and that management should establish and operate monitoring activities to monitor and evaluate the results. Because SBA does not review whether contracts are being awarded under the appropriate NAICS codes, it cannot provide reasonable assurance that WOSB program requirements are being met or identify agencies that may require targeted outreach or additional training on eligible NAICS codes. As a result, WOSB contracts may continue to be awarded to groups other than those intended, which can undermine the goals of and confidence in the program. While federal contract obligations to all women-owned small businesses and WOSB program set-asides have increased since fiscal year 2012, WOSB program set-asides remain a small percentage. Specifically, federal dollars obligated for contracts to all women-owned small businesses increased from $18.2 billion in fiscal year 2012 to $21.4 billion in fiscal year 2017. Contracts awarded to all women-owned small businesses within WOSB-program-eligible industries also increased during this period—from about $15 billion to $18.8 billion, as shown in figure 1. However, obligations under the WOSB program represented only a small share of this increase. In fiscal year 2012, WOSB program contract obligations were 0.5 percent of contract obligations to all women- owned small businesses for WOSB-program-eligible goods or services (about $73.5 million), and in fiscal year 2017 this percentage had grown to 3.8 percent (about $713.3 million) (see fig. 1). In summary, the WOSB program aims to enhance federal contracting opportunities for women-owned small businesses. However, as of early May 2019, SBA had not fully implemented comprehensive procedures to monitor the performance of the WOSB program’s third-party certifiers and had not taken steps to provide reasonable assurance that only eligible businesses obtain WOSB set-aside contracts, as recommended in our 2014 report. Without ongoing monitoring and reviews of third-party certifier reports, SBA cannot ensure that certifiers are fulfilling their requirements, and it is missing opportunities to gain information that could help improve the program’s processes. Further, limitations in SBA’s procedures for conducting and analyzing eligibility examinations inhibit its ability to better understand the eligibility of businesses before they apply for and potentially receive contracts, which exposes the program to unnecessary risk of fraud. Also, since SBA does not expect to finish implementing the changes in the 2015 NDAA until January 1, 2020, these continued oversight deficiencies increase program risk. As a result, we maintain that our previous recommendations should be addressed. In addition, SBA has not addressed deficiencies related to WOSB program set-asides being awarded under ineligible industry codes. Although SBA has updated its training and outreach materials for the WOSB program to address NAICS code requirements, it has not developed a process for periodically reviewing FPDS–NG data, and has yet to provide targeted outreach or training to agencies that may be using ineligible codes. As a result, SBA is not aware of the extent to which individual agencies are following program requirements and which agencies may require targeted outreach or additional training. Reviewing FPDS–NG data would allow SBA to identify those agencies (and contracting offices within them) that could benefit from such training. Without taking these additional steps, SBA cannot provide reasonable assurance that WOSB program requirements are being met. As such, we made one recommendation in our March 2019 report to SBA. We recommended that SBA develop a process for periodically reviewing FPDS–NG data to determine the extent to which agencies are awarding WOSB program set-asides under ineligible NAICS codes, and take steps to address any issues identified, such as providing targeted outreach or training to agencies making awards under ineligible codes. As of May 2019, this recommendation remains open. Chairman Golden, Ranking Member Stauber, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact William Shear, Director, Financial Markets and Community Investment at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Andrew Pauline (Assistant Director), Tarek Mahmassani (Analyst in Charge), and Jennifer Schwartz. Other staff who made key contributions to the report cited in the testimony were Allison Abrams, Pamela Davidson, Jonathan Harmatz, Tiffani Humble, Julia Kennon, Rebecca Shea, Jena Sinkfield, Tyler Spunaugle, and Tatiana Winger. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In 2000, Congress authorized the WOSB program, allowing contracting officers to set aside procurements to women-owned small businesses in industries in which they are substantially underrepresented. To be eligible to participate in the WOSB program, firms have the option to self-certify or be certified by a third-party certifier. However, the 2015 NDAA changed the WOSB program by (1) authorizing SBA to implement sole-source authority, (2) eliminating the option for firms to self-certify as being eligible for the program, and (3) allowing SBA to implement a new certification process. This testimony is based on a report GAO issued in March 2019 ( GAO-19-168 ). For that report, GAO examined (1) the extent to which SBA has addressed the 2015 NDAA changes, (2) SBA's efforts to address previously identified deficiencies, and (3) use of the WOSB program. GAO reviewed relevant laws, regulations, and program documents; analyzed federal contracting data from April 2011 through June 2018; and interviewed SBA officials, officials from contracting agencies selected to obtain a range of experience with the WOSB program, and the three (out of four) private third-party certifiers that agreed to meet with GAO. The Small Business Administration (SBA) has implemented one of the three changes to the Women-Owned Small Business (WOSB) program authorized in the National Defense Authorization Act of 2015 (2015 NDAA). In September 2015 SBA published a final rule to implement sole-source authority (to award contracts without competition), effective October 2015. As of early May 2019, SBA had not eliminated the option for program participants to self-certify that they are eligible to participate, as required by the 2015 NDAA. SBA officials stated that the agency intended to address the third change made by the 2015 NDAA (meaning implement a new certification process for the WOSB program). SBA has not addressed WOSB program oversight deficiencies and recommendations in GAO's 2014 report ( GAO-15-54 ). For example, GAO recommended that SBA establish procedures to assess the performance of four third-party certifiers—private entities approved by SBA to certify the eligibility of WOSB firms. While SBA generally agreed with GAO's recommendations and conducted a compliance review of the certifiers in 2016, it has no plans to regularly monitor their performance. By not improving its oversight, SBA is limiting its ability to ensure third-party certifiers are following program requirements. Further, the implementation of sole-source authority in light of these continued oversight deficiencies can increase program risk. GAO maintains that its recommendations aimed at improving oversight should be addressed. In addition, GAO's March 2019 ( GAO-19-168 ) report found that about 3.5 percent of contracts using a WOSB set-aside were awarded for ineligible goods or services from April 2011 through June 2018. At that time, SBA was not reviewing contracting data that could identify which agencies may need targeted training. GAO recommended that SBA review such data to help address identified issues. In early May 2019, SBA said it had initiated such efforts. While federal contract obligations to all women-owned small businesses and WOSB program set-asides have increased since fiscal year 2012, WOSB program set-asides remain a small percentage (see figure). Note: Obligations to women-owned small businesses represent contract obligations to women-owned small businesses under WOSB-program-eligible North American Industry Classification System codes. FPDS-NG obligation amounts have been adjusted for inflation. GAO recommended in March 2019 that SBA develop a process for periodically reviewing the extent to which WOSB program set-asides are awarded for ineligible goods or services and use the results to address identified issues, such as through targeted outreach or training on the WOSB program. SBA agreed with the recommendation.", "document_type": "gao"}
{"report": "The LDA defines a lobbyist as an individual who is employed or retained by a client for compensation for services that include more than one lobbying contact (written or oral communication to covered officials, such as a high ranking agency official or a Member of Congress made on behalf of a client), and whose lobbying activities represent at least 20 percent of the time that he or she spends on behalf of the client during the quarter. Lobbying firms are persons or entities that have one or more employees who lobby on behalf of a client other than that person or entity. The LDA requires lobbyists to register with the Secretary of the Senate and the Clerk of the House, and to file quarterly reports disclosing their respective lobbying activities. Lobbyists are required to file their registrations and reports electronically with the Secretary of the Senate and the Clerk of the House through a single entry point. Registrations and reports must be publicly available in downloadable, searchable databases from the Secretary of the Senate and the Clerk of the House. No specific statutory requirements exist for lobbyists to generate or maintain documentation in support of the information disclosed in the reports they file. However, guidance issued by the Secretary of the Senate and the Clerk of the House recommends that lobbyists retain copies of their filings and documentation supporting reported income and expenses for at least 6 years after they file their reports. Figure 1 provides an overview of the registration and filing process. Lobbying firms are required to register with the Secretary of the Senate and the Clerk of the House for each client if the firms receive or expect to receive more than $3,000 in income from that client for lobbying activities. Lobbyists are also required to submit an LD-2 quarterly report for each registration filed. The LD-2s contain information that includes: the name of the lobbyist reporting on quarterly lobbying activities; the name of the client for whom the lobbyist lobbied; a list of individuals who acted as lobbyists on behalf of the client during the reporting period; whether any lobbyists served in covered positions in the executive or legislative branch, such as high-ranking agency officials or congressional staff positions, in the previous 20 years; codes describing general lobbying issue areas, such as agriculture and education; a description of the specific lobbying issues; houses of Congress and federal agencies lobbied during the reporting reported income (or expenses for organizations with in-house lobbyists) related to lobbying activities during the quarter (rounded to the nearest $10,000). The LDA requires lobbyists to report certain political contributions semiannually in the LD-203 report. These reports must be filed 30 days after the end of a semiannual period by each lobbying firm registered to lobby and by each individual listed as a lobbyist on a firm’s lobbying report. The lobbyists or lobbying firms must: list the name of each federal candidate or officeholder, leadership political action committee, or political party committee to which he or she contributed at least $200 in the aggregate during the semiannual period; report contributions made to presidential library foundations and presidential inaugural committees; report funds contributed to pay the cost of an event to honor or recognize an official who was previously in a covered position, funds paid to an entity named for or controlled by a covered official, and contributions to a person or entity in recognition of an official, or to pay the costs of a meeting or other event held by or in the name of a covered official; and certify that they have read and are familiar with the gift and travel rules of the Senate and House, and that they have not provided, requested, or directed a gift or travel to a Member, officer, or employee of Congress that would violate those rules. The LDA also requires that the Secretary of the Senate and the Clerk of the House guide and assist lobbyists with the registration and reporting requirements and develop common standards, rules, and procedures for LDA compliance. The Secretary of the Senate and the Clerk of the House review the guidance annually. It was last revised January 31, 2017, to (among other issues), revise the registration threshold to reflect changes in the Consumer Price Index, and clarify the identification of clients and covered officials and issues related to rounding income and expenses. The guidance provides definitions of LDA terms, elaborates on registration and reporting requirements, includes specific examples of different disclosure scenarios, and provides explanations of why certain scenarios prompt or do not prompt disclosure under the LDA. The offices of the Secretary of the Senate and the Clerk of the House told us they continue to consider information we report on lobbying disclosure compliance when they periodically update the guidance. In addition, they told us they email registered lobbyists quarterly on common compliance issues and reminders to file reports by the due dates. The Secretary of the Senate and the Clerk of the House, along with USAO, are responsible for ensuring LDA compliance. The Secretary of the Senate and the Clerk of the House notify lobbyists or lobbying firms in writing that they are not complying with the LDA reporting. Subsequently, they refer those lobbyists who fail to provide an appropriate response to USAO. USAO researches these referrals and sends additional noncompliance notices to the lobbyists or lobbying firms, requesting that they file reports or terminate their registration. If USAO does not receive a response after 60 days, it decides whether to pursue a civil or criminal case against each noncompliant lobbyist. A civil case could lead to penalties up to $200,000 for each violation, while a criminal case—usually pursued if a lobbyist’s noncompliance is found to be knowing and corrupt—could lead to a maximum of 5 years in prison. Generally, under the LDA, within 45 days of being employed or retained to make a lobbying contact on behalf of a client, the lobbyist must register by first filing an LD-1 form with the Secretary of the Senate and the Clerk of the House. Thereafter, the lobbyist must file quarterly disclosure (LD-2) reports that detail the lobbying activities, including filing a first report for the quarter in which the lobbyist registered. Of the 3,618 new registrations we identified for the third and fourth quarters of 2017 and the first and second quarters of 2018, we matched 3,329 of them (92.01 percent) to corresponding LD-2 reports filed within the same quarter as the registration. These results are consistent with the findings we have reported in prior reviews. We used the House lobbyists’ disclosure database as the source of the reports. We also used an electronic matching algorithm that allows for misspellings and other minor inconsistencies between the registrations and reports. Figure 2 shows lobbyists filed disclosure reports as required for most new lobbying registrations from 2010 through 2018. As part of their regular enforcement procedures, the Clerk of the House and the Secretary of the Senate are to follow up with newly filed registrations where quarterly reports were not filed. If the Clerk of the House and the Secretary of the Senate are unsuccessful in bringing the lobbyist into compliance, they may refer those cases to USAO as described earlier in figure 1. For selected elements of lobbyists’ LD-2 reports that can be generalized to the population of lobbying reports, our findings have generally been consistent from year to year. Most lobbyists reporting $5,000 or more in income or expenses provided written documentation to varying degrees for the reporting elements in their disclosure reports. Figure 3 shows that for most LD-2 reports, lobbyists provided documentation for income and expenses for sampled reports from 2010 through 2018, and our 2018 estimate does not represent a statistically significant change from 2017. Figure 4 shows that in 2018, 10 percent of lobbyists’ reported income or expenses differed by $10,000 or more. Additionally, for some LD-2 reports, lobbyists did not round their income or expenses as the guidance requires. In 2018, we estimate 20 percent of reports did not round reported income or expenses according to the guidance. We have found that rounding difficulties have been a recurring issue on LD-2 reports from 2010 through 2018. As we previously reported, several lobbyists who listed expenses told us that based on their reading of the LD-2 form, they believed they were required to report the exact amount. While this is not consistent with the LDA and the guidance, this may be a source of some of the confusion regarding rounding errors. In 2016, the guidance was updated to include an additional example about rounding expenses to the nearest $10,000. The LDA requires lobbyists to disclose lobbying contacts made with federal agencies on behalf of the client for the reporting period. This year, of the 99 LD-2 reports in our sample, 46 reports disclosed lobbying activities at federal agencies. Of those, lobbyists provided documentation for all disclosed lobbying activities at the federal agencies for 29 LD-2 reports. Figure 5 shows that lobbyists for most LD-2 reports provided documentation for selected elements of their LD-2 reports that include general issue area codes for lobbying activities, lobbying the House and the Senate, and individual lobbyists listed from 2010 through 2018. In 2017 and 2018, there was an improvement of compliance with documentation for lobbying the House and the Senate over the previous 7 years. Figure 6 shows that lobbyists for most lobbying firms filed contribution reports as required in our sample from 2010 through 2018. All individual lobbyists and lobbying firms reporting lobbying activity are required to file political contribution (LD-203) reports semiannually, even if they have no contributions to report, because they must certify compliance with the gift and travel rules. The LDA requires a lobbyist to disclose previously held covered positions in the executive or legislative branch, such as high-ranking agency officials and congressional staff, when first registering as a lobbyist for a new client. This can be done either on a new LD-1 or on the quarterly LD- 2 filing when added as a new lobbyist. This year, we estimate that 19 percent of all LD-2 reports may not have properly disclosed previously held covered positions as required. As in our other reports, some lobbyists were still unclear about the need to disclose certain covered positions, such as paid congressional internships or certain executive agency positions. Figure 7 shows the extent to which lobbyists may not have properly disclosed one or more covered positions as required from 2010 through 2018. Lobbyists amended 23 of the 99 LD-2 disclosure reports in our original sample to change previously reported information after we contacted them. Of the 23 reports, 10 were amended after we notified the lobbyists of our review, but before we met with them. An additional 13 of the 23 reports were amended after we met with the lobbyists to review their documentation. We consistently find a notable number of amended LD-2 reports in our sample each year following notification of our review. This suggests that sometimes our contact spurs lobbyists to more closely scrutinize their reports than they would have without our review. Table 1 lists reasons lobbying firms in our sample amended their LD-2 reports. As part of our review, we compared contributions listed on lobbyists’ and lobbying firms’ LD-203 reports against those political contributions reported in the Federal Election Commission (FEC) database to identify whether political contributions were omitted on LD-203 reports in our sample. The samples of LD-203 reports we reviewed contained 80 reports with contributions and 80 reports without contributions. We estimate that overall in 2018, lobbyists failed to disclose one or more reportable contributions on 33 percent of reports. Additionally, eight LD- 203 reports were amended in response to our review. Table 2 shows our results from 2010 to 2018; estimates in the table have a maximum margin of error of 11 percentage points. For this year’s review, the estimated change in the percent of LD-203 reports missing one or more FEC- reportable contributions was a statistically significant increase compared to each of the prior 9 years. As part of our review, we conducted interviews with 97 different lobbying firms in the 2018 sample of LD-2 disclosure reports. Consistent with prior reviews, most lobbying firms reported that they found it “very easy” or “somewhat easy” to comply with reporting requirements. Of the 97 different lobbying firms interviewed, 24 reported that the disclosure requirements were “very easy,” 61 reported them “somewhat easy,” and 11 reported them “somewhat difficult” or “very difficult.” One lobbying firm did not respond to this question (see figure 8). Most lobbying firms we surveyed rated the definitions of terms used in LD-2 reporting as “very easy” or “somewhat easy” to understand with regard to meeting their reporting requirements. This is consistent with prior reviews. Figure 9 shows what lobbyists reported as their ease of understanding the terms associated with LD-2 reporting requirements from 2012 through 2018. The U.S. Attorney’s Office for the District of Columbia (USAO) officials stated that they continue to have sufficient personnel resources and authority under the LDA to enforce reporting requirements. This includes imposing civil or criminal penalties for noncompliance. Noncompliance refers to a lobbyist’s or lobbying firm’s failure to comply with the LDA. However, USAO noted that due to attrition the number of the assigned personnel has changed from 2017 as indicated in table 3. USAO officials stated that lobbyists resolve their noncompliance issues by filing LD-2, LD-203, or LD-2 amendments, or by terminating their registration, depending on the issue. Resolving referrals can take anywhere from a few days to years, depending on the circumstances. During this time, USAO creates summary reports from its database to track the overall number of referrals that are pending or become compliant as a result of the lobbyist receiving an email, phone call, or noncompliance letter. Referrals remain in the pending category until they are resolved. The pending category is divided into the following areas: “initial research for referral,” “responded but not compliant,” “no response /waiting for a response,” “bad address,” and “unable to locate.” USAO officials noted that they attempt to review and update all pending cases every six months. USAO focuses its enforcement efforts primarily on the “responded but not compliant” and the “no response/waiting for a response” groups. Officials told us that, if after several attempts, it cannot contact the noncompliant firm or its lobbyist, it confers with both the Secretary of the Senate and the Clerk of the House to determine whether further action is needed. In the cases where the lobbying firm is repeatedly referred for not filing disclosure reports but does not appear to be actively lobbying, USAO suspends enforcement actions. USAO officials reported they will continue to monitor these firms and will resume enforcement actions if required. USAO received 3,798 referrals from both the Secretary of the Senate and the Clerk of the House for failure to comply with LD-2 reporting requirements cumulatively for filing years 2009 through 2018. Figure 10 shows the number and status of the referrals received, and the number of enforcement actions taken by USAO to bring lobbying firms into compliance. Enforcement actions include USAO attempts to bring lobbyists into compliance through letters, emails, and calls. About 40 percent (1,533 of 3,798) of the total referrals received are now compliant because lobbying firms either filed their reports or terminated their registrations. In addition, some of the referrals were found to be compliant when USAO received the referral, so no action was taken. This may occur when lobbying firms respond to the contact letters from the Secretary of the Senate and the Clerk of the House after USAO received the referrals. About 59 percent (2,250 of 3,798) of referrals are pending further action because USAO could not locate the lobbying firm, did not receive a response from the firm after an enforcement action, or plans to conduct additional research to determine if it can locate the lobbying firm. The remaining 15 referrals did not require action or were suspended because the lobbyist or client was no longer in business or the lobbyist was deceased. LD-203 referrals consist of two types: (1) LD-203(R) referrals represent lobbying firms that have failed to file LD-203 reports for their lobbying firm; and (2) LD-203 referrals represent the lobbyists at the lobbying firm who have failed to file their individual LD-203 reports as required. USAO received 2,629 LD-203(R) referrals from lobbying firms (cumulatively from 2009 through 2018) and 5,897 LD-203 referrals for individual lobbyists (cumulatively from 2009 through 2017) from the Secretary of the Senate and the Clerk of the House for noncompliance with reporting requirements). LD-203 referrals are more complicated than LD-2 referrals because both the lobbying firm and the lobbyists within the firm are each required to file an LD-203. Lobbyists employed by a lobbying firm typically use the firm’s contact information and not the lobbyists’ personal contact information. This makes it difficult to locate a lobbyist who is not in compliance and may have left the firm. In 2018, USAO officials confirmed that, while many firms have assisted USAO by providing contact information for lobbyists, they are not required to do so. According to officials, USAO has difficulty pursuing LD-203 referrals for lobbyists who have departed a firm without leaving forwarding contact information with the firm. While USAO utilizes web searches and online databases, including social media, to find these missing lobbyists, it is not always successful. Figure 11 shows the status of LD-203(R) lobbying firm referrals received and the number of enforcement actions taken by USAO to bring lobbying firms into compliance. About 42 percent (1,093 of 2,629) of the lobbying firms referred by the Secretary of the Senate and the Clerk of the House for noncompliance from calendar years 2009 through 2018 are now considered compliant because firms either filed their reports or terminated their registrations. About 58 percent (1,523 of 2,629) of the referrals are pending further action. The remaining 13 referrals did not require action or were suspended because the lobbyist or client was no longer in business or the lobbyist was deceased. USAO received 5,897 LD-203 individual lobbyists referrals from the Secretary of the Senate and the Clerk of the House for lobbyists who failed to comply with LD-203 reporting requirements for calendar years 2009 through 2017. Figure 12 shows the status of the referrals received and the number of enforcement actions taken by USAO to bring lobbyists into compliance. In addition, figure 12 shows that about 32 percent (1,880 of 5,897) of the lobbyists had come into compliance by filing their reports or no longer being registered as a lobbyist. About 68 percent (4,003 of 5,897) of the referrals are pending further action because USAO could not locate the lobbyist, did not receive a response from the lobbyist, or plans to conduct additional research to determine if it can locate the lobbyist. The remaining 14 referrals did not require action or were suspended because the lobbyist or client was no longer in business or the lobbyist was deceased. USAO received LD-203 referrals from the Secretary of the Senate and the Clerk of the House for 7,617 individual lobbyists who failed to comply with LD-203 reporting requirements for any filing year from 2009 through 2017. Figure 13 shows the status of compliance for individual lobbyists listed on referrals to USAO. About 36 percent (2,706 of 7,617) of the lobbyists had come into compliance by filing their reports or by not being registered as a lobbyist. About 65 percent (4,911 of 7,617) of the referrals are pending action because USAO could not locate the lobbyists, did not receive a response from the lobbyists, or plans to conduct additional research to determine if it can locate the lobbyists. USAO officials said that many of the pending LD-203 referrals represent lobbyists who no longer lobby for the lobbying firms affiliated with the referrals, even though these lobbying firms may be listed on the lobbyist’s LD-203 report. According to USAO officials, lobbyists and lobbying firms who repeatedly fail to file reports are labeled chronic offenders and referred to one of the assigned attorneys for follow-up. USAO also receives complaints regarding lobbyists who are allegedly lobbying but never filed an LD-203. USAO officials added that USAO monitors and investigates chronic offenders to ultimately determine the appropriate enforcement actions, which may include settlement or other civil actions. Additionally, USAO officials reported that they are working to resolve an active case involving a chronic offender firm and lobbyist that was pending as of 2018. USAO officials noted that the agency is continuing settlement discussions with the company that failed to respond to required LDA violation notices and its lobbyist did not respond to individual violations for semiannual reporting. The company is now current on filing its reports and USAO is working with the Secretary of the Senate and the Clerk of the House on settling past violations. USAO continues to review its records to identify additional chronic offenders for further action due to noncompliance. We provided a draft of this report to the Department of Justice for review and comment. The Department of Justice did not have comments. We are sending copies of this report to the Attorney General, the Secretary of the Senate, the Clerk of the House of Representatives, and interested congressional committees and members. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2717 or jonesy@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. The random sample of lobbying disclosure reports we selected was based on unique combination of House ID, lobbyist, and client names (see table 4). Lobbyist or lobbying firm Jessica Woolley National Multifamily Housing Council, Inc. Our objectives were to determine the extent to which lobbyists are able to demonstrate compliance with the Lobbying Disclosure Act of 1995, as amended (LDA) by providing documentation (1) to support information contained on registrations and reports filed under the LDA; (2) to identify challenges or potential improvements to compliance, if any; and (3) to describe the resources and authorities available to the U.S. Attorney’s Office for the District of Columbia (USAO), its role in enforcing LDA compliance, and any efforts it has made to improve LDA enforcement. We used information in the lobbying disclosure database maintained by the Clerk of the House of Representatives (Clerk of the House). To assess whether these disclosure data were sufficiently reliable for the purposes of this report, we reviewed relevant documentation and consulted with knowledgeable officials. Although registrations and reports are filed through a single web portal, each chamber subsequently receives copies of the data and follows different data-cleaning, processing, and editing procedures before storing the data in either individual files (in the House) or databases (in the Senate). Currently, there is no means of reconciling discrepancies between the two databases caused by the differences in data processing. For example, Senate staff told us during previous reviews they set aside a greater proportion of registration and report submissions than the House for manual review before entering the information into the database. As a result, the Senate database would be slightly less current than the House database on any given day pending review and clearance. House staff told us during previous reviews that they rely heavily on automated processing. In addition, while they manually review reports that do not perfectly match information on file for a given lobbyist or client, staff members approve and upload such reports as originally filed by each lobbyist, even if the reports contain errors or discrepancies (such as a variant on how a name is spelled). Nevertheless, we do not have reason to believe that the content of the Senate and House systems would vary substantially. Based on interviews with knowledgeable officials and a review of documentation, we determined that House disclosure data were sufficiently reliable for identifying a sample of quarterly disclosure reports (LD-2) and for assessing whether newly filed lobbyists also filed required reports. We used the House database for sampling LD-2 reports from the third and fourth quarters of 2017 and the first and second quarters of 2018, as well as for sampling year-end 2017 and midyear 2018 political contributions reports (LD-203). We also used the database for matching quarterly registrations with filed reports. We did not evaluate the Offices of the Secretary of the Senate or the Clerk of the House, both of which have key roles in the lobbying disclosure process. However, we did consult with officials from each office. They provided us with general background information at our request. To assess the extent to which lobbyists could provide evidence of their compliance with reporting requirements, we examined a stratified random sample of 99 LD-2 reports from the third and fourth quarters of 2017 and the first and second quarters of 2018. We excluded reports with no lobbying activity or with income or expenses of less than $5,000 from our sampling frame. We drew our sample from 49,918 activity reports filed for the third and fourth quarters of 2017 and the first and second quarters of 2018 available in the public House database, as of our final download date for each quarter. Our sample of LD-2 reports was not designed to detect differences over time. However, we conducted tests of significance for changes from 2010 to 2018 for the generalizable elements of our review. We found that results were generally consistent from year to year and there were few statistically significant changes (as noted in our report) after using a Bonferroni adjustment to account for multiple comparisons. For this year’s review, we estimated that 97 percent of LD-2 reports provided written documentation for the lobbying income and expenses. Our sample is based on a stratified random selection and is only one of a large number of samples that we may have drawn. Because each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95-percent confidence interval. This interval would contain the actual population value for 95 percent of the samples that we could have drawn. The percentage estimates for LD-2 reports have 95-percent confidence intervals of within plus or minus 12 percentage points or fewer of the estimate itself. We contacted all the lobbyists and lobbying firms in our sample and, using a structured web-based survey, asked them to confirm key elements of the LD-2 and whether they could provide written documentation for key elements in their reports, including the amount of income reported for lobbying activities; the amount of expenses reported on lobbying activities; the names of those lobbyists listed in the report; the houses of Congress and the federal agencies that they lobbied, and the issue codes listed to describe their lobbying activity. After reviewing the survey results for completeness, we interviewed lobbyists and lobbying firms to review the documentation they reported as having on their online survey for selected elements of their respective LD- 2 report. Prior to each interview, we conducted a search to determine whether lobbyists properly disclosed their covered position as required by the LDA. We reviewed the lobbyists’ previous work histories by searching lobbying firms’ websites, LinkedIn, Leadership Directories, Legistorm, and Google. Prior to 2008, lobbyists were only required to disclose covered official positions held within 2 years of registering as a lobbyist for the client. The Honest Leadership and Open Government Act of 2007 amended that time frame to require disclosure of positions held 20 years before the date the lobbyists first lobbied on behalf of the client. Lobbyists are required to disclose previously held covered official positions either on the client registration (LD-1) or on an LD-2 report. Consequently, those who held covered official positions may have disclosed the information on the LD-1 or a LD-2 report filed prior to the report we examined as part of our random sample. Therefore, where we found evidence that a lobbyist previously held a covered official position, and that information was not disclosed on the LD-2 report under review, we conducted an additional review of the publicly available Secretary of the Senate or Clerk of the House database to determine whether the lobbyist properly disclosed the covered official position on a prior report or LD-1. Finally, if a lobbyist appeared to hold a covered position that was not disclosed, we asked for an explanation at the interview with the lobbying firm to ensure that our research was accurate. In previous reports, we reported the lower bound of a 90-percent confidence interval to provide a minimum estimate of omitted covered positions and omitted contributions with a 95-percent confidence level. We did so to account for the possibility that our searches may have failed to identify all possible omitted covered positions and contributions. As we have developed our methodology over time, we are more confident in the comprehensiveness of our searches for these items. Accordingly, this report presents the estimated percentages for omitted contributions and omitted covered positions rather than the minimum estimates. As a result, percentage estimates for these items will differ slightly from the minimum percentage estimates presented in prior reports. In addition to examining the content of the LD-2 reports, we confirmed whether the most recent LD-203 reports had been filed for each firm and lobbyist listed on the LD-2 reports in our random sample. Although this review represents a random selection of lobbyists and firms, it is not a direct probability sample of firms filing LD-2 reports or lobbyists listed on LD-2 reports. As such, we did not estimate the likelihood that LD-203 reports were appropriately filed for the population of firms or lobbyists listed on LD-2 reports. To determine if the LDA’s requirement for lobbyists to file a report in the quarter of registration was met for the third and fourth quarters of 2017 and the first and second quarters of 2018, we used data filed with the Clerk of the House to match newly filed registrations with corresponding disclosure reports. Using an electronic matching algorithm that includes strict and loose text matching procedures, we identified matching disclosure reports for 3,329, or 92.01 percent, of the 3,618 newly filed registrations. We began by standardizing client and lobbyist names in both the report and registration files (including removing punctuation and standardizing words and abbreviations, such as “company” and “CO”). We then matched reports and registrations using the House identification number (which is linked to a unique lobbyist-client pair), as well as the names of the lobbyist and client. For reports we could not match by identification number and standardized name, we also attempted to match reports and registrations by client and lobbyist name, allowing for variations in the names to accommodate minor misspellings or typos. For these cases, we used professional judgment to determine whether cases with typos were sufficiently similar to consider as matches. We could not readily identify matches in the report database for the remaining registrations using electronic means. To assess the accuracy of the LD-203 reports, we analyzed stratified random samples of LD-203 reports from the 29,798 total LD-203 reports. The first sample contains 80 reports of the 9,502 reports with political contributions and the second contains 80 reports of the 20,296 reports listing no contributions. Each sample contains 40 reports from the year- end 2017 filing period and 40 reports from the midyear 2018 filing period. The samples from 2018 allow us to generalize estimates in this report to either the population of LD-203 reports with contributions or the reports without contributions to within a 95-percent confidence interval of within plus or minus 11 percentage points or fewer. Although our sample of LD- 203 reports was not designed to detect differences over time, for this year’s review, the estimated change in percentage of LD-203 reports missing one or more reportable contributions was a statistically significant increase compared to each of the prior 9 years. While the results provide some confidence that apparent fluctuations in our results across years are likely attributable to sampling error, the inability to detect significant differences may also be related to the nature of our sample, which was relatively small and designed only for cross- sectional analysis. We analyzed the contents of the LD-203 reports and compared them to contribution data found in the publicly available Federal Elections Commission’s (FEC) political contribution database. We consulted with staff at FEC responsible for administering the database. We determined that the data are sufficiently reliable for the purposes of our reporting objectives. We compared the FEC-reportable contributions on the LD-203 reports with information in the FEC database. The verification process required text and pattern matching procedures so we used professional judgment when assessing whether an individual listed is the same individual filing an LD-203. For contributions reported in the FEC database and not on the LD-203 report, we asked the lobbyists or organizations to explain why the contribution was not listed on the LD-203 report or to provide documentation of those contributions. As with covered positions on LD-2 disclosure reports, we cannot be certain that our review identified all cases of FEC-reportable contributions that were inappropriately omitted from a lobbyist’s LD-203 report. We did not estimate the percentage of other non-FEC political contributions that were omitted because they tend to constitute a small minority of all listed contributions and cannot be verified against an external source. To identify challenges to compliance, we used a structured web-based survey and obtained the views from 97 different lobbying firms included in our sample on any challenges to compliance. The number of different lobbying firms is 97, which is less than our original sample of 99 reports because some lobbying firms had more than one LD-2 report included in our sample. We calculated responses based on the number of different lobbying firms that we contacted rather than the number of interviews. Prior to our calculations, we removed the duplicate lobbying firms based on the most recent date of their responses. For those cases with the same response date, the decision rule was to keep the cases with the smallest assigned case identification number. To obtain their views, we asked them to rate their ease with complying with the LD-2 disclosure requirements using a scale of “very easy,” “somewhat easy,” “somewhat difficult,” or “very difficult.” In addition, using the same scale, we asked them to rate the ease of understanding the terms associated with LD-2 reporting requirements. To describe the resources and authorities available to the U.S. Attorney’s Office for the District of Columbia (USAO) and its efforts to improve its LDA enforcement, we interviewed USAO officials. We obtained information on the capabilities of the system officials established to track and report compliance trends and referrals and on other practices established to focus resources on LDA enforcement. USAO provided us with reports from the tracking system on the number and status of referrals and chronically noncompliant lobbyists and lobbying firms. The mandate does not require us to identify lobbyists who failed to register and report in accordance with the LDA requirements, or determine for those lobbyists who did register and report whether all lobbying activity or contributions were disclosed. Therefore, this was outside the scope of our audit. We conducted this performance audit from May 2018 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Clifton G. Douglas Jr. (Assistant Director), Shirley Jones (Managing Associate General Counsel), Ulyana Panchishin (Analyst-in-Charge), James Ashley, Krista Loose, Kathleen Jones, Amanda Miller, Sharon Miller, Robert Robinson, Stewart W. Small, Peter Verchinski, and Khristi Wilkins made key contributions to this report. Assisting with lobbyist file reviews were Adam Brooks, Jazzmin R. Cooper, Colleen Corcoran, Rianna B. Jansen, Benjamin Legow, Regina Morrison, Andrew Olson, Amanda R. Prichard, Alan Rozzi, Bryan Sakakeeny, Kate Wulff, and Edith P. Yuh. Lobbying Disclosure: Observations on Lobbyists’ Compliance with New Disclosure Requirements. GAO-08-1099. Washington, D.C: September 30, 2008. 2008 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-09-487. Washington, D.C: April 1, 2009. 2009 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-10-499. Washington, D.C: April 1, 2010. 2010 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-11-452. Washington, D.C: April 1, 2011. 2011 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-12-492. Washington, D.C: March 30, 2012. 2012 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-13-437. Washington, D.C: April 1, 2013. 2013 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-14-485. Washington, D.C: May 28, 2014. 2014 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-15-310. Washington, D.C.: March 26, 2015. 2015 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-16-320. Washington, D.C.: March 24, 2016. 2016 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-17-385. Washington, D.C.: March 31, 2017. 2017 Lobbying Disclosure: Observations on Lobbyists’ Compliance with Disclosure Requirements. GAO-18-388, Washington, D.C.: March 30, 2018.", "summary": "The LDA, as amended, requires lobbyists to file quarterly disclosure reports and semiannual reports on certain political contributions. The law also includes a provision for GAO to annually audit lobbyists' compliance with the LDA. GAO's objectives were to (1) determine the extent to which lobbyists can demonstrate compliance with disclosure requirements; (2) identify any challenges or potential improvements to compliance that lobbyists report; and (3) describe the resources and authorities available to USAO in its role in enforcing LDA compliance. This is GAO's 12th annual report under the provision. GAO reviewed a stratified random sample of 99 quarterly disclosure LD-2 reports filed for the third and fourth quarters of calendar year 2017, and the first and second quarters of calendar year 2018. GAO also reviewed two random samples totaling 160 LD-203 reports from year-end 2017 and midyear 2018. This methodology allowed GAO to generalize to the population of 49,918 disclosure reports with $5,000 or more in lobbying activity, and 29,798 reports of federal political campaign contributions. GAO also interviewed USAO officials. GAO is not making any recommendations in this report. GAO provided a draft of this report to the Department of Justice for review and comment. The agency stated that it did not have comments. For the 2018 reporting period, most lobbyists provided documentation for key elements of their disclosure reports to demonstrate compliance with the Lobbying Disclosure Act of 1995, as amended (LDA). For lobbying disclosure (LD-2) reports and political contributions (LD-203) reports filed during the third and fourth quarter of 2017 and the first and second quarter of 2018, GAO estimates that 92 percent of lobbyists who filed new registrations also filed LD-2 reports as required for the quarter in which they first registered (the figure below describes the filing process and enforcement); 97 percent of all lobbyists who filed could provide documentation for lobbying income and expenses. However, an estimated 20 percent of these LD-2 reports were not properly rounded to the nearest $10,000; 19 percent of all LD-2 reports did not properly disclose one or more previously held covered positions as required; and 33 percent of LD-203 reports were missing reportable contributions, which was a statistically significant increase compared to prior years. Except as noted above, these findings are generally consistent with prior reports GAO issued from 2010 through 2017. GAO continues to find that most lobbyists in the sample reported some level of ease in complying with disclosure requirements and in understanding the definitions of terms used in the reporting. However, some disclosure reports demonstrate compliance difficulties, such as failure to disclose covered positions or misreporting of income or expenses. The U.S. Attorney's Office for the District of Columbia (USAO) stated it has sufficient resources to enforce compliance. USAO continued its efforts to resolve noncompliance through filing reports or terminating registrations, as well as imposing civil and criminal penalties.", "document_type": "gao"}
{"report": "VA administers one of the largest health care systems in the United States and is charged, through the VHA, with providing health care services to the nation’s eligible veterans. VHA expects to provide care to more than 7 million veterans in fiscal year 2019 at health care facilities across the country through a system of 18 regional networks known as Veterans Integrated Service Networks. VHA has 172 medical centers that offer a variety of inpatient and outpatient services, ranging from routine examinations to complex surgical procedures. VHA’s health care system also includes community-based outpatient clinics and other facilities that generally limit services to primary care and some specialty care. When veterans need services that are not available at VHA medical facilities or within required driving distances or time frames, VHA may purchase care from non-VHA providers through one of its community care programs. VHA’s National Surgery Office is charged with overseeing the VA Organ Transplant Program, including the 12 VATCs that have established specialty services to provide solid organ transplant surgery and post- operative care, in some cases in conjunction with an academic affiliate. VATCs offer transplants for one or more organ types including heart, kidney, liver, and lung. (See fig. 1.) VHA considers transplant services provided through a VATC’s academic affiliate as care provided within the VA Organ Transplant Program. VHA’s National Surgery Office is responsible for clinical and operational oversight, as well as policies related to the VA Organ Transplant Program, including facilitating and monitoring the transplant referral process; overseeing quality of care; and monitoring outcomes of veterans receiving transplants. In 2013, VHA’s National Surgery Office established TRACER to track and monitor the referrals, evaluations, and outcomes for organ transplants performed at the VATCs. Referring VHA medical centers use the database to enter a referral for a veteran to be evaluated at a VATC; and VATCs use it to record referral reviews, patient evaluations, transplant outcomes, and follow-up care. In addition, the database provides the National Surgery Office with information used to monitor transplant volumes, the referral and evaluation process, and clinical outcomes across all VATCs. The VA Organ Transplant Program’s services include pre-transplant evaluation and testing, transplant surgery, post-transplant follow-up care, as well as transplant-related round-trip travel and lodging for both the veteran and a caregiver. VHA covers the cost of lodging for the veteran and caregiver through a variety of arrangements including contracts with local hotels and on-site VHA medical center housing, such as through the Fisher House Program. In addition, VHA may cover the cost of transplant services provided by non-VA providers; for example, when a veteran in urgent need of a heart transplant cannot travel to a VATC that provides that service. The VA MISSION Act includes provisions regarding VA’s authority to cover organ transplant services by non-VA providers—referred to as community care. Prior to the VA MISSION Act, VHA used its authority, as needed, to contract for transplant services with providers in the community when VHA care and services were not accessible in a timely fashion; however, the act provides additional authority to improve veterans’ access to transplant care and services through community providers, and authorizes transplant procedures with living organ donors who are not eligible for VHA care. On June 5, 2019, VA issued final regulations for the act. The Health Resources and Services Administration contracts with the United Network for Organ Sharing—a private, nonprofit organization—to manage the OPTN, which creates and maintains transplant policies and bylaws that are applicable to all transplant centers in the United States, including the VATCs and the academic affiliates performing transplants under contract with them. OPTN documents organ allocation policies, and collects and reports data on transplant recipients, donors, and outcomes. OPTN also conducts periodic audits of transplant program performance, including ensuring that transplant programs meet functional activity requirements (i.e., performing a minimum number of transplants in a proscribed period of time), and reviewing post-transplant patient survival rates. In addition, OPTN assesses whether transplant centers have established required quality assurance and performance improvement programs to help ensure the quality and safety of the transplant services provided. When transplant centers, including the VATCs, identify a candidate for organ transplantation, they register the patient in the OPTN’s centralized, national computer network that matches organ donors with transplant candidates, referred to in this report as the “national organ donation waitlist.” Veterans do not receive preference for organ allocation. When an organ becomes available, the computer network generates a list of transplant candidates ranked by a standard set of criteria that generally include factors such as blood and tissue type, size of the organ, medical urgency of the candidate, time on the waitlist, and geographic distance between the organ donor and transplant candidate. An organ procurement specialist then contacts the transplant program of the top-ranked transplant candidate to determine if the available organ is suitable for the candidate. If the organ is suitable, arrangements are made to procure, transport, and store the donated organ, and for the transplant candidate to travel to the transplant center for surgery. If the organ is not suitable for a given candidate, the procurement specialist contacts the transplant program of the next transplant candidate on the list until the organ is found to be suitable for a transplant candidate. Each year, VA allocates most of its appropriations for health care services to VHA’s 18 Veterans Integrated Service Networks through the Veterans Equitable Resource Allocation (VERA) system. VERA funds are allocated for general purposes, such as treatment for basic and complex patients, research and educational support, and equipment and maintenance costs; as well as for specific purposes, such as preventative and primary care initiatives and transplant care. The VERA model uses price groups— categories of veterans with similar resource needs based on the complexity of their medical conditions—to determine the funding level for each network. In addition, VHA’s National Surgery Office historically allocated transplant specific purpose funds to the VATCs for solid organ transplants, because the costs of transplant services were not fully covered by general purpose funds. Beginning in fiscal year 2019, the VERA model was modified to establish a new price group specifically for transplant patients, allowing full funding with general purpose funds for these services. VHA officials explained that this change is expected to reduce the need for specific purpose funds to supplement transplant care. VHA officials told us academic affiliate contracts are funded through the medical services appropriations allocated to the VHA medical center where the VATC is located. To receive an organ transplant in the VA Organ Transplant Program, a veteran must go through a five-step process: (1) initial referral, (2) pre- operative evaluation, (3) listing on the OPTN national organ donation waitlist, (4) transplant surgery, and (5) follow-up care that continues for the remainder of the veteran’s life. See figure 2 for an overview of the five steps. Step 1: Initial screening and referral. A veteran seeking an organ transplant begins the process by having an initial screening at a referring VHA medical center. If VHA medical center providers determine that the veteran is a potential candidate for an organ transplant, they may prepare a formal referral to a VATC. To prepare a referral, the providers use an organ-specific checklist and other tools developed by VHA’s National Surgery Office with input from other experts in the field to perform a standard set of assessments of the veteran’s clinical, social, and mental health status. In addition, VHA officials told us that the initial screening includes an assessment of the veteran’s social and family support; for example, identifying a caregiver who can accompany and stay with the veteran throughout the transplant process. In addition, there are organ- specific criteria, such as negative tobacco smoking screens for veterans seeking a heart transplant, and up-to-date dialysis information for liver and kidney transplant candidates. VHA officials noted that the providers may consult with staff at a VATC as needed during the initial screening phase. Following the initial screening, if VHA medical center providers determine that the veteran is a potential candidate for a transplant, they enter the checklist information into the TRACER database, include the results of the required assessments outlined in the checklist, and attach any additional medical information, such as testing performed through care in the community. VHA officials told us that the VATC to which the veteran is referred is chosen based on factors including distance from the veteran’s home and the types of organ transplants offered at the VATC. Once the VHA medical center completes a referral in TRACER, the information becomes available to the selected VATC. Step 2: VATC referral review and veteran evaluation. When the VATC receives a veteran’s referral, VATC staff review it to determine whether the referral information is complete and the veteran meets the criteria to continue the process. If so, VATC staff evaluate the veteran and perform additional testing and clinical preparation needed to determine whether the veteran is a transplant candidate. To reduce the travel burden on veterans and their caregivers, providers at the veteran’s referring VHA medical center may arrange for telehealth visits with the VATC for pre- transplant education and consultation. However, travel for in-person appointments at the VATC is required for most veterans referred for evaluation. According to VATC officials we spoke with, the VATC considers the severity of the veteran’s illness and overall need for a transplant. In some cases, this assessment is conducted by a panel of experts composed of VATC providers and providers from the academic affiliate, where applicable. For example, some VATCs hold regular review meetings to discuss cases up for consideration jointly with providers from the VHA medical center and the academic affiliate, because individual cases may be co-managed depending on the type of organ being transplanted and the services provided at an individual VATC. Providers at some VATCs provide care at both the VATC and its academic affiliate, allowing for integrated clinical management of patients. If the VATC determines the veteran is not a candidate for transplantation, the referring VHA medical center can request a second opinion by another VATC. If the veteran is once again determined not to be a candidate, the referring VHA medical center can make a final appeal. Appeals are forwarded to the VA’s Transplant Surgical Advisory Board, comprised of subject matter experts, for consideration. According to VHA policy, the board considers the appeal and makes a recommendation to the National Director of Surgery (the head of VHA’s National Surgery Office) who is responsible for facilitating second opinion requests, making the final determination, and notifying the referring VHA medical center regarding the final appeal determination. VHA reported that between fiscal years 2014 and 2018, 39 decisions were appealed to the Transplant Surgical Advisory Board, one of which was approved for resubmission to another VATC for consideration. Step 3: Listing on the national organ donation waitlist. If the VATC determines that the veteran is a candidate for an organ transplant, VATC staff add the veteran to the national organ donation waitlist. At this point in the process, veterans follow the same procedure as the general population seeking an organ transplant. To maximize the chances that the veteran will receive an organ, the VATC staff may also discuss options the veteran can pursue for personally identifying a potential living donor (if applicable for the organ needed). VATC officials noted that they may provide other clinical interventions to help prolong a veteran’s life and preserve his or her health while awaiting an organ; for example, implanting a ventricular assist device into a veteran awaiting a heart transplant. Step 4: Transplant surgery. According to VATC officials, once a veteran is placed on the OPTN national organ donation waitlist, depending on the type of organ needed, the veteran and their caregiver may be required to travel to the VATC and remain in close proximity while awaiting organ availability. In some cases, such as for a liver transplant, a harvested organ can be kept viable for longer periods, allowing time for a veteran to travel from their home to the VATC once the organ becomes available. Depending on the arrangement between a particular VATC and its academic affiliate, the veteran could receive the transplant surgery and post-operative care at either the VATC or the affiliate. For example, the VATC in Richmond performs heart transplants and contracts with its affiliate for liver transplants. The VATC in Nashville performs kidney transplants and contracts with its affiliate for heart and liver transplants. From fiscal years 2014 through 2018, 61 percent of the transplant surgeries provided within the VA Organ Transplant Program were performed by a VATC and 39 percent were performed by an academic affiliate. See table 1 for a list of VATCs, organ types transplanted, and contracts with academic affiliates. Step 5: Follow-up care. Following the transplant surgery and the immediate post-operative care provided by the VATC and its academic affiliate, the veteran receives on-going follow-up care from both the VATC and the referring VHA medical center. VHA providers monitor veterans post-transplant for the remainder of their lives; for example, to oversee post-transplant immunosuppression, and track survival rates and outcomes for organ recipients. VA policy states that the VATC has primary responsibility for providing care while the veteran is at the VATC for the transplant and for providing specialized follow-up care after the veteran is discharged. In general, however, following discharge from the VATC, the veteran’s referring VHA medical center maintains responsibility for the veteran’s care coordination. VHA has policies and processes to allow for some aspects of transplant- related care, including follow-up care, to be done via telehealth—that is, visits with a VATC provider remotely from the veteran’s referring VHA medical center. VHA medical centers may establish telehealth agreements with VATCs to ease the burden of travel for veterans and their caregivers, and to allow for ongoing monitoring of the veteran’s health post-transplant. Because VHA monitors transplant recipients for the rest of their lives, using telehealth can decrease the need for the veteran to travel back to the VATC unless a specific clinical need arises, such as biopsies for heart transplant recipients. VHA officials noted that follow-up care is facilitated by VA’s shared electronic health record, which allows VHA providers to share medical records and other patient information over time and across locations. Further, VHA providers noted that follow-up care and communication between VATCs and primary care teams can be more complicated in the private sector when transplant services are not generally part of a patient’s whole system of care. VATCs provided about 1,700 organ transplants between fiscal year 2014 and fiscal year 2018. The number of organ transplants provided each year generally increased, ranging from 300 transplants in fiscal year 2014 to a peak of 400 transplants in fiscal year 2017. During this 5-year period, kidneys and livers were the most frequently transplanted organs, representing 85 percent of all organs transplanted at VATCs. Heart and lung transplants were much less common and represented the remaining transplants. (See fig. 3.) For the programs that were active during all 5 years from fiscal year 2014 through fiscal year 2018, the number of solid organ transplants performed varied by VATC, ranging from 12 at the Birmingham VATC to 399 at the Pittsburgh VATC. (See table 2.) The nearly 1,700 transplants performed through the VA Organ Transplant Program represent a relatively small portion—less than 20 percent—of the VHA referrals for organ transplant between fiscal years 2014 and 2018. While thousands of veterans are referred for solid organ transplants, far fewer veterans ultimately receive transplants. According to VA officials, VHA considers all submitted transplant referrals; however, many patients do not meet initial screening criteria to proceed with a formal evaluation. For example, a veteran’s state of illness may not be severe enough to warrant a full transplant evaluation. Further, some veterans who are offered transplant evaluations decide not to proceed following education about the process. Officials noted that in many cases, the transplant evaluation reveals that a veteran does not meet the criteria for a transplant, such as not having a committed caregiver who can support the veteran through the evaluation and transplant procedure. Of the veterans who are listed on the national organ donation waitlist, VHA officials report that the number of transplants is limited by the supply of organs, which does not meet the demand in the U.S. general population, including veterans. For veterans who received an organ transplant from a VATC between fiscal year 2014 and fiscal year 2018, survival rates varied by organ type, with the 3-year survival rate ranging from about 95 percent for kidney transplants to 85 percent for lung transplants, according to National Surgery Office data. (See table 3.) For national-level general population survival rates, see appendix II. Consistent with the increase in the number of organ transplants provided between fiscal years 2014 and 2018, VHA’s allocations and spending for transplant services also increased, similarly peaking in fiscal year 2017. VHA funds these services using a combination of general purpose and specific purpose funds. VHA’s National Surgery Office allocates transplant specific purpose funding to the VATCs based upon the transplant-related workload the VATCs report through TRACER. See appendix III for additional information on transplant-related allocations and expenditures. VHA Allocation of Transplant Specific Purpose Funds. VHA allocated $292 million in transplant specific purpose funds during this 5-year period, ranging from $50.3 million in fiscal year 2014 to approximately $64.6 million in fiscal year 2018. (See table 5 in app. III.) Transplant specific purpose funds are used to support program overhead costs (infrastructure and maintenance) associated with organ transplants performed at VATCs. In addition, they are used for pre-transplant evaluations, lodging, and some miscellaneous costs associated with transplants, such as living donor evaluations. Further, transplant specific purpose funds are used to fund other VHA medical centers without a VATC that perform certain transplant follow-up care. VHA Expenditures for Transplant-Related Services VHA Expenditures of General Purpose and Specific Purpose Funds for Veterans Receiving a Solid Organ Transplant. VHA spent approximately $259 million for services provided to veterans who received a solid organ transplant at a VATC during this 5- year period, ranging from $44.6 million in fiscal year 2014 to a high of $57.7 million in fiscal year 2017. Similarly, VHA’s spending for pre- and post-transplant care provided at VHA medical centers totaled approximately $68.6 million during this time, ranging from $10.8 million in fiscal year 2014 to $15.6 million in fiscal year 2017. (See tables 6 and 7 in app. III.) VHA Contracts with VATC Academic Affiliates. VHA spent over $216 million on contract payments to academic affiliates for transplant services during this period, ranging from $34.9 million in fiscal year 2014 to a high of $49.9 million in fiscal year 2017. This increased spending corresponded to an increase in the number of transplants performed by academic affiliates, which totaled 669 transplants—nearly 40 percent of all VATC transplants from fiscal year 2014 through fiscal year 2018. The highest volume—146 transplants—and the highest cost—$49.9 million—occurred in fiscal year 2017. (See table 8 in app. III.) VHA Contracts for Community Care. From fiscal year 2014 through fiscal year 2018, VHA spent $7.9 million for solid organ transplant services provided to 53 veterans through community care programs. (See table 9 in app. III.) According to VHA data, over this 5-year period, 50 of these transplants were authorized using title 38 U.S.C. § 1703 (“Non-VA Medical Care Program”). The remaining three were funded using the Veterans Access, Choice, and Accountability Act of 2014 (Choice Act)—totaling approximately $411,000 of the $7.9 million. VA has reported that while the Choice Act allows VHA to provide an eligible veteran transplant care at a transplant center in the community, generally at Medicare rates, organ procurement is only partially covered at Medicare rates. This has resulted in community providers being less willing to provide transplant services for VHA patients through community care programs. From fiscal years 2014 through 2018, VATCs received 10,494 referrals from VHA medical centers. In that time, the percentage of VATC referrals that met timeliness standards outlined by VHA’s National Surgery Office improved. Stable condition referrals: For veterans in stable condition, VHA requires that VATCs review referrals and decide within 5 business days whether veterans are potential candidates for an organ transplant and should receive a full evaluation. The percentage of referrals for which VATCs met the timeliness standard increased from 95 percent in fiscal year 2014 to 99 percent in fiscal year 2018. Emergency condition referrals: For veterans in emergency circumstances, VHA requires that VATCs review referrals and document within 48 hours whether veterans are potential transplant candidates and should receive a full evaluation. The percentage of referrals for which VATCs met the timeliness standard increased from 94 percent in fiscal year 2014 to 98 percent in fiscal year 2018. (For more information, see app. IV.) National Surgery Office officials identified two possible drivers of the observed improvements: (1) increased monitoring, and (2) providing real- time feedback to VATCs through TRACER. Providers at one VATC noted that they use information from the National Surgery Office’s Transplant Quarterly Reports to identify areas to improve and they assigned a transplant team nurse the responsibility to monitor program quality, including that timeliness requirements are being met. A provider at a VATC where timeliness has improved since fiscal year 2014 and is now at 100 percent explained that his facility has provided training to staff at referring VHA medical centers they work with frequently. For example, the official said he has hosted a workshop for transplant coordinators to provide training on submitting transplant referral packets through TRACER. While VATCs almost always met timeliness standards in fiscal year 2018, VATC officials in our review noted that transplant coordinators at referring VHA medical centers sometimes submit referral packets that are incomplete, requiring additional time and effort by the provider to search for information not readily available and potentially adding delays to the VATC review times. VHA requires that a complete referral packet be submitted through TRACER using a referral progress note that contains the required assessments outlined in the organ-specific checklist. The referral packet can also include attachments to transmit some required information, such as results for tests performed by community providers. Providers at three VATCs told us that reviewing a complete referral packet generally takes 30 minutes to an hour. However, in cases where the packet is incomplete (for example, it does not include the results from all the required assessments) the process is much less efficient and, according to two providers we interviewed, can take up to 5 hours. VATC providers explained that when not all test results are available in the referral packet, they have to access another VHA medical center’s electronic medical record system and search for the required information. Accessing another medical center’s system adds time to the referral review process and can take time away from the provider’s other duties, such as providing follow-up care to transplant patients or monitoring transplant outcomes. Internal controls state that management should assign responsibility to discrete units and demonstrate a commitment to develop competent individuals in those units, such as through training, to enable the organization to operate in an efficient manner and help achieve the organization’s objectives. However, a lack of understanding or implementation of the required information needed in the referral packets can make the process for reviewing the referral packets inefficient in some cases. Specifically, one VATC provider told us that incomplete referral packets are often due to a lack of training for staff at the referring VHA medical centers on the process for submitting referrals through TRACER. In fact, four of the five transplant coordinators at referring VA medical centers we interviewed reported a lack of training on submitting transplant referrals through TRACER. Instead, for example, a transplant coordinator at one referring VHA medical center said she received assistance from a medical clerk at her facility on how to submit referrals through TRACER. Officials at VHA’s National Surgery Office told us that although there is no centralized, in-person training available for referring VHA medical centers, the office published a training guide, which is available on VA’s intranet and provides guidance on how to access TRACER and refer patients for transplant evaluation. Despite this resource, transplant coordinators from some referring VHA medical centers still cited a need for additional training or other guidance. For example, one official said training for new transplant coordinators would be helpful as would regular updates on transplant criteria or policy changes, such as through regular calls or a newsletter targeted at transplant coordinators. In addition to timeliness requirements for referral review, VHA requires that VATCs complete an evaluation of veterans within 30 calendar days of receiving a referral to determine whether they are a candidate for transplant and should be placed on the national organ donation waitlist. From fiscal years 2014 through 2018, VATCs increased the percentage of evaluations completed within this time frame, from 55 percent (576 of 1,045 appointments) in fiscal year 2014 to 89 percent (1,193 of 1,346 appointments) in fiscal year 2017, before dropping to 87 percent (1,325 of 1,521 appointments) in fiscal year 2018. National Surgery Office officials attributed the overall improvement to increased monitoring and the increased availability of telehealth for conducting transplant evaluations. See appendix IV for more information on the timeliness of transplant evaluations by VATCs from fiscal years 2014 through 2018. The extent to which delayed evaluations occurred varied by VATC location and by organ type each fiscal year. For example, in fiscal year 2018, we found that the average time from referral to completed evaluation was less than 30 days for 19 of the 20 organ transplant programs, and overall, 13 percent of evaluations were not completed within 30 days. Of note, 51 of 128, or 40 percent, of evaluations for kidney transplant at the Bronx VATC were completed more than 30 days after the referral was submitted, with evaluations ranging from 5 to 84 days after submission. In contrast, all 69, or 100 percent, of liver evaluations at the Nashville VATC were completed within 30 days, with evaluations ranging from 0 to 28 days after the referral was submitted. (See fig. 4.) According to VHA data and three VATC providers we interviewed, evaluation appointment availability is not a cause for delays in most cases; rather, delays are primarily due to veteran preference. According to VHA data for 1,617 evaluation appointments completed in fiscal year 2018, 1,412 appointments were scheduled within the 30-day requirement. For the remaining 205 appointments, 13 were delayed due to lack of appointment availability, and 192 appointments were delayed due to veteran preference. According to providers at the three VATCs we interviewed, while veterans may prefer to be seen at a later date for a number of reasons, including that their caregiver is not available to travel, veterans are not always aware that they should be evaluated within 30 days of being referred for a transplant. VHA requires the referring VHA medical center to discuss the 30 day evaluation requirement with the veteran prior to submitting the referral. According to VHA, in some cases evaluation timeliness is a critical factor affecting patient outcomes. Although a veteran may choose to be seen at a time beyond the 30-day standard, postponing an evaluation may delay their placement on the national organ donation waitlist, potentially having a negative impact on their health and well-being. Officials at five VATCs and two referring VHA medical centers reported that additional training for transplant coordinators would be helpful for improving evaluation timeliness. Additional training enables employees to develop competencies and reinforce requirements, which is consistent with internal control standards that state that management should develop competent individuals to achieve the entity’s objectives. According to one VATC provider, transplant coordinators at referring VHA medical centers should be trained to discuss travel with the veteran before submitting the referral, so the transplant coordinator and the veteran understand that the evaluation should be completed within 30 days of referral, increasing the likelihood that veterans will be able to schedule timely evaluations. A referring VHA medical center transplant coordinator also said that additional training about the VATC’s processes would be helpful in order to be better able to inform veterans and their caregivers about what to expect from the transplant process. Placing veterans on the national organ donation waitlist as soon as possible is critical for potential transplant candidates to be matched with a donor organ. Since fiscal year 2014, VHA has improved timeliness for reviewing transplant referrals to determine if a veteran is a candidate and for completing transplant evaluations. However, VHA medical center staff do not always submit complete transplant referral packets through TRACER, which can create inefficiencies and delay the referral review process. Similarly, inefficiencies in the transplant evaluation process occur when VATC and VHA medical center staff do not fully inform veterans of their role in the transplant evaluation process, specifically, that their evaluation be completed within 30 days of referral. Without additional training to address these inefficiencies a veteran’s placement on the national organ donation waitlist could be delayed. The Under Secretary for Health should establish a requirement that VHA’s National Surgery Office provide additional training to staff at referring VHA medical centers on (a) submitting referral packets through TRACER that are complete, and (b) understanding and communicating the veteran’s role in the evaluation process related to the timely completion of transplant evaluations. (Recommendation 1) We provided a draft of this product to VA for comment. In its comments, reproduced in appendix V, the department concurred in principle with our recommendation, reiterated the resources it currently makes available to staff at referring VHA medical centers, and described actions it plans to take to address the recommendation. Specifically, VHA’s National Surgery Office plans to distribute a memorandum to all VHA facilities to reinforce the available training and resources to support the staff at referring VHA medical centers with submitting complete referrals, and to ensure adequate communication of the veteran’s role in timely completion of transplant evaluations. VA also provided technical comments, which we incorporated as appropriate. In addition, we provided a draft of this report to the Department of Health and Human Services for review and they did not have any comments. We are sending copies of this report to the appropriate congressional committees, the Secretary of VA, the Secretary of the Department of Health and Human Services, and other interested parties. In addition, the report is available at no charge on GAO’s website at http://www.gao.gov/. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or silass@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix VI. Appendix I: Department of Veterans Affairs Transplant Centers Providing Solid Organ Transplants (printable) We analyzed national transplant survival rates by organ type using data from the Scientific Registry of Transplant Recipients. All transplant facilities across the United States, including Department of Veterans Affairs transplant centers, provide data to this database. Table 4 shows the national transplant survival rates by organ for fiscal years 2014 through 2018. Annually, the Department of Veterans Affairs (VA) allocates most of its appropriations for health care services to the Veterans Integrated Service Networks within the Veterans Health Administration (VHA) through a model called the Veterans Equitable Resource Allocation (VERA). The VERA model is designed to fund patient care based on a methodology that develops set, or “capitated,” rates for different groups or categories of veterans with similar resource needs based on the complexity of their medical conditions. Categories include oncology, visual impairment, chronic mental illness, and critical illness. VERA uses a national formula that considers the number of veterans and the complexity of care provided; and certain geographic factors, such as local labor costs, in determining how much each Veterans Integrated Service Network should receive. VERA determines this amount based on each network’s activities and needs in the following areas: patient care, equipment, nonrecurring maintenance, education support, and research support. The networks, in turn, allocate resources to their respective VHA medical centers, including those with VA transplant centers (VATC). The networks distribute VERA funds to VHA medical centers based on the complexity of patients treated at the medical center in previous fiscal years. This appendix provides VA’s reported allocations and expenditures for solid organ transplant services through its VATCs and contracts with academic affiliates and community providers from fiscal year 2014 through fiscal year 2018. Table 5 shows VHA allocation of transplant specific purpose funds by VATC for transplant-related services. Table 6 shows VHA expenditures at each VATC for veterans who received solid organ transplants. Table 7 shows VHA expenditures for pre- and post-transplant services provided by VHA medical centers without a VATC for veterans who received transplants. Table 8 shows the total number of transplants and contract payments to academic affiliates for solid organ transplant services. Table 9 shows total number and spending for solid organ transplants provided by community care providers. The Veterans Health Administration (VHA) has timeliness requirements for reviewing transplant referrals to determine whether veterans are potential candidates for organ transplant and should receive a full evaluation, and for completing timely evaluations for potential candidates. Timeliness of referral reviews improved from fiscal year 2014 through fiscal year 2018. VHA requires that for veterans in stable condition, Department of Veterans Affairs transplant centers (VATC) review referrals and decide within 5 business days whether veterans are potential candidates for an organ transplant and should receive a full evaluation. For emergency cases, VATCs should perform this review and document the results within 48 hours. Table 10 shows the number of referrals reviewed and the percentage of timely referrals for each VATC and organ transplant program from fiscal year 2014 through fiscal year 2018. VHA requires that VATCs complete an evaluation of stable veterans within 30 calendar days of receiving a referral to determine whether they are a candidate for transplant and should be placed on the national organ donation waitlist. The percentage of evaluations completed within the required time frame increased from 55 percent in fiscal year 2014 to 87 percent in fiscal year 2018, although some variation can be seen by organ type and location within each fiscal year. Table 11 shows the number of completed evaluations and the percentage of timely evaluations for each VATC and organ transplant program from fiscal year 2014 through fiscal year 2018. In addition to the contact named above, Marcia A. Mann (Assistant Director), Jill K. Center (Analyst-in-Charge), Colin Ashwood, Emily Binek, Emily Bippus, Shana Deitch, Keith Haddock, and Ebony Russ made key contributions to this report. Also contributing were Krister Friday, Jacquelyn Hamilton, Giselle Hicks, Drew Long, Vikki Porter, and Ethiene Salgado-Rodriguez.", "summary": "As of June 2019, over 113,000 people in the United States—including veterans—were waiting for an organ transplant. In 2018, more than 36,000 organ transplants were performed at transplant centers across the country, including those within the Department of Veterans Affairs' (VA) Organ Transplant Program. GAO was asked to review how VA administers and oversees its organ transplant program. This report, among other things, examines the process and timeliness with which VA reviews referrals and completes evaluations for organ transplants. GAO reviewed data from VHA's Transplant Referral and Cost Evaluation Reimbursement database, documents related to VA's transplant program, and federal internal control standards. GAO conducted site visits to three of the 12 VATCs, selected to obtain diversity in geography and types of organs transplanted. At the selected VATCs, GAO reviewed facility data and documents related to organ transplants and interviewed officials. GAO also collected and reviewed information from the remaining nine VATCs. The 12 Veterans Affairs' transplant centers (VATC), which are overseen by the Veterans Health Administration (VHA), almost always met the referral timeliness standard from fiscal years 2014 through 2018. When a veteran is determined to be a potential candidate for an organ transplant, he or she can receive a formal referral to a VATC. Depending on the type of referral, the VATC must meet specific timeliness standards for reviewing the referral and deciding if the veteran should receive a full evaluation. Likewise, VATCs have timeliness standards for conducting the full evaluation, and generally showed improvement in meeting that standard from fiscal years 2014 through 2018. For those delays in conducting full evaluations that did occur, GAO found they varied by organ type and VATC. Specifically, in fiscal year 2018, transplant evaluation timeliness ranged from 60 percent at two VATC kidney programs to 100 percent at kidney, liver, heart or lung programs across seven different VATCs. According to VATC officials, transplant evaluation delays are caused when patients or caregivers are not available or not aware that they are required to be evaluated within 30 days of being referred. Although veterans may prefer to be seen at a later date, untimely evaluations can delay veterans' placement on the national organ donation waitlist. According to VHA data, 192 of the 1,617 transplant evaluation appointments completed in fiscal year 2018 did not meet the 30-day requirement. VATC officials said this was because veterans were not available or not aware of the requirement. GAO found that staff at referring VHA medical centers lacked a full understanding of the transplant referral and evaluation process. For example, VATC providers told GAO that transplant referrals are sometimes incomplete, requiring providers to spend extra time searching for information that should have been readily available. GAO found that additional training for medical center staff would help to improve the efficiency of the transplant referral process and the timeliness of transplant evaluations provided to veterans, a critical factor affecting veteran outcomes. VHA should provide additional training for staff at VHA medical centers that refer patients for organ transplants on (1) submitting complete referrals and (2) understanding and communicating the veteran's role related to timely completion of transplant evaluations. VA concurred in principle with the recommendation and described actions the department will take to address the recommendation.", "document_type": "gao"}
{"report": "Our past work has identified progress and challenges in a number of areas related to DHS’s management of the CFATS program, including (1) the process for identifying high-risk chemical facilities; (2) how it assesses risk and prioritizes facilities; (3) reviewing and approving facility site security plans; (4) inspecting facilities and ensuring compliance; and (5) efforts to conduct outreach with stakeholders and first responders. In May 2014, we found that more than 1,300 facilities had reported having ammonium nitrate to DHS. However, based on our review of state data and records, there were more facilities with ammonium nitrate holdings than those that had reported to DHS under the CFATS program. Thus, we concluded that some facilities weren’t required to report to DHS and some that were required may have failed to do so. We recommended that DHS work with other agencies, including the Environmental Protection Agency (EPA), to develop and implement methods of improving data sharing among agencies and with states as members of a Chemical Facility Safety and Security Working Group. DHS agreed with our recommendation and has since addressed it. Specifically, DHS compared DHS data with data from other federal agencies, such as EPA, as well as member states from the Chemical Facility Safety and Security Working Group to identify potentially noncompliant facilities. As a result of this effort, in July 2015, DHS officials reported that they had identified about 1,000 additional facilities that should have reported information to comply with CFATS and subsequently contacted these facilities to ensure compliance. DHS officials told us that they continue to engage with states to identify potentially non-compliant facilities. For example, as of June 2018, DHS officials stated that they have received 43 lists of potentially noncompliant facilities from 34 state governments, which are in various stages of review by DHS. DHS officials also told us that they hired an individual to serve as the lead staff member responsible for overseeing this effort. DHS has also taken action to strengthen the accuracy of data it uses to identify high-risk facilities. In July 2015, we found that DHS used self- reported and unverified data to determine the risk categorization for facilities that held toxic chemicals that could threaten surrounding communities if released. At the time, DHS required that facilities self- report the Distance of Concern—an area in which exposure to a toxic chemical cloud could cause serious injury or fatalities from short-term exposure—as part of its Top-Screen. We estimated that more than 2,700 facilities with a toxic release threat had misreported the Distance of Concern and therefore recommended that DHS (1) develop a plan to implement a new Top-Screen to address errors in the Distance of Concern submitted by facilities, and (2) identify potentially miscategorized facilities that could cause the greatest harm and verify that the Distance of Concern of these facilities report is accurate. DHS has fully addressed both of these recommendations. Specifically, in response to the first recommendation, DHS implemented an updated Top-Screen survey in October 2016 and now collects data from facilities and conducts more accurate modeling to determine the actual area of impact (formerly called the Distance of Concern), rather than relying on the facilities’ calculation. In response to the second recommendation, DHS officials reported in November 2016 that they reassessed all facility Top-Screens that reported threshold quantities of chemicals posing a toxic release threat, and identified 158 facilities with the potential to cause the greatest harm. In April 2018, DHS officials reported that all of these facilities have since been reassessed using updated Top-Screen information and, where appropriate, assigned a risk tier. DHS has also taken actions to better assess regulated facilities’ risks in order to place the facilities into the appropriate risk tier. In April 2013, we reported that DHS’s risk assessment approach did not consider all of the elements of threat, vulnerability, and consequence associated with a terrorist attack involving certain chemicals. Our work showed that DHS’s CFATS risk assessment methodology was based primarily on consequences from human casualties, but did not consider economic consequences, as called for by the National Infrastructure Protection Plan (NIPP) and the CFATS regulation. We also found that (1) DHS’s approach was not consistent with the NIPP because it treated every facility as equally vulnerable to a terrorist attack regardless of location or on-site security and (2) DHS was not using threat data for 90 percent of the tiered facilities—those tiered for the risk of theft or diversion—and using 5-year-old threat data for the remaining 10 percent of those facilities that were tiered for the risks of toxic chemical release or sabotage. We recommended that DHS enhance its risk assessment approach to incorporate all elements of risk and conduct an independent peer review after doing so. DHS agreed with our recommendations and has implemented actions to address both of them. Specifically, with regard to our recommendation that DHS enhance its risk assessment approach to incorporate all elements of risk, DHS worked with Sandia National Laboratories to develop a model to estimate the economic consequences of a chemical attack. In addition, DHS worked with Oak Ridge National Laboratory to devise a new tiering methodology, called the Second Generation Risk Engine. In so doing, DHS revised the CFATS threat, vulnerability, and consequence scoring methods to better cover the range of CFATS security issues. Additionally, with regard to our recommendation that DHS conduct a peer review after enhancing its risk assessment approach, DHS conducted peer reviews and technical reviews with government organizations and facility owners and operators, and worked with Sandia National Laboratories to verify and validate the CFATS program’s revised risk assessment methodology. To further enhance its risk assessment approach, in the fall of 2016, DHS also revised its Chemical Security Assessment Tool (CSAT), which supports DHS efforts to gather information from facilities to assess their risk. According to DHS officials, the new tool—called CSAT 2.0—is intended to eliminate duplication and confusion associated with DHS’s original CSAT. DHS officials told us that they have improved the tool by revising some questions in the original CSAT to make them easier to understand; eliminating some questions; and pre-populating data from one part of the tool to another so that users do not have to retype the same information multiple times. DHS officials also told us that the facilities that have used the CSAT 2.0 have provided favorable feedback that the new tool is more efficient and less burdensome than the original CSAT. Finally, DHS officials told us that, as of June 2018, DHS completed all notifications and processed tiering results for all but 226 facilities. DHS officials did not provide an estimated target completion date for these pending risk assessments, noting that completing the assessments is highly dependent on the facilities providing the necessary Top-Screen information. DHS has also made progress reviewing and approving facility site security plans by reducing the time it takes to review these plans and eliminating the backlog of plans awaiting review. In April 2013, we reported that DHS revised its procedures for reviewing facilities’ security plans to address DHS managers’ concerns that the original process was slow, overly complicated, and caused bottlenecks in approving plans. We estimated that it could take DHS another 7 to 9 years to review the approximately 3,120 plans in its queue at that time. We also estimated that, given the additional time needed to do compliance inspections, the CFATS program would likely be implemented in 8 to 10 years. We did not make any recommendations for DHS to improve its procedures for reviewing facilities’ security plans because DHS officials reported that they were exploring ways to expedite the process, such as reprioritizing resources and streamlining inspection requirements. In July 2015, we reported that DHS had made substantial progress in addressing the backlog—estimating that it could take between 9 and 12 months for DHS to review and approve security plans for the approximately 900 remaining facilities. DHS officials attributed the increased approval rate to efficiencies in DHS’s review process, updated guidance, and a new case management system. Subsequently, DHS reported in its December 2016 semi-annual report to Congress that it had eliminated its approval backlog. Finally, we found in our 2017 review that DHS took action to implement an Expedited Approval Program (EAP). The CFATS Act of 2014 required that DHS create the EAP as another option that tier 3 and tier 4 chemical facilities may use to develop and submit security plans to DHS. Under the program, these tier 3 and 4 facilities may develop a security plan based on specific standards published by DHS (as opposed to the more flexible performance standards using the standard, non- expedited process). DHS issued guidance intended to help facilities prepare and submit their EAP security plans to DHS, which includes an example that identifies prescriptive security measures that facilities are to have in place. According to committee report language, the EAP was expected to reduce the regulatory burden on smaller chemical companies, which may lack the compliance infrastructure and the resources of large chemical facilities, and help DHS to process security plans more quickly. If a tier 3 or 4 facility chooses to use the expedited option, DHS is to review the plan to determine if it is facially deficient, pursuant to the reporting requirements of the CFATS Act of 2014. If DHS approves the EAP site security plan, it is to subsequently conduct a compliance inspection. In 2017, we found that DHS had implemented the EAP and had reported to Congress on the program, as required by the CFATS Act of 2014. In addition, as of June 2018, according to DHS officials, only 18 of the 3,152 facilities eligible to use the EAP had opted to use it. DHS officials attributed the low participation to several possible factors including: DHS had implemented the expedited program after most eligible facilities already submitted standard (non-expedited) security plans to DHS; facilities may consider the expedited program’s security measures to be too strict and prescriptive, not providing facilities the flexibility of the standard process; and the lack of an authorization inspection may discourage some facilities from using the expedited program because this inspection provides useful information about a facility’s security. We also found in 2017 that recent changes made to the CFATS program could affect the future use of the expedited program. As discussed previously, DHS has revised its methodology for determining the level of each facility’s security risk, which could affect a facility’s eligibility to participate in the EAP. In our July 2015 report, we found that DHS began conducting compliance inspections in September 2013, and by April 2015, had conducted inspections of 83 of the inspected 1,727 facilities that had approved security plans. Our analysis showed that nearly half of the facilities were not fully compliant with their approved site security plans and that DHS had not used its authority to issue penalties because DHS officials found it more productive to work with facilities to bring them into compliance. We also found that DHS did not have documented processes and procedures for managing the compliance of facilities that had not implemented planned measures by the deadlines outlined in their plans. We recommended that DHS document processes and procedures for managing compliance to provide more reasonable assurance that facilities implement planned measures and address security gaps. DHS agreed and has since taken steps toward implementing this recommendation. Specifically, DHS revised CFATS Standard Operating Procedures that, as of February 2019, we are reviewing to determine if they sufficiently document the processes and procedures currently being used to track noncompliant facilities and ensure facilities implement planned measures as outlined in their approved site security plans. In August 2018, we reported that our analysis of DHS data since our 2015 report showed that DHS has made substantial progress in conducting and completing compliance inspections. Specifically, our analysis showed that DHS increased the number of compliance inspections completed per year since DHS began conducting compliance inspections in 2013 and that, for the 2,466 high-risk facilities with an approved site security plan as of May 2018, DHS had conducted 3,553 compliance inspections. Of these, DHS issued corrective actions to two facilities that were not in compliance with their approved site security plan. In our August 2018 report, we also found that DHS developed a new methodology and performance measure for the CFATS program in order to evaluate security changes made by high-risk chemical facilities, but that the methodology does not measure the program’s impact on reducing a facility’s vulnerability to an attack. We found that DHS could take steps to evaluate vulnerability reduction resulting from the CFATS compliance inspection process. We recommended that DHS incorporate vulnerability into the new methodology to help measure the reduction in the vulnerability of high-risk facilities to a terrorist attack, and use that data in assessing the CFATS program’s performance in lowering risk and enhancing national security. DHS agreed and is taking steps to implement this recommendation. Specifically, in September 2018, DHS reported making progress towards the implementation of two new performance metrics by the end of the first quarter of fiscal year 2019. DHS officials stated that these metrics should, among other things, evaluate the progress of individual facilities in enhancing their security while part of the CFATS program and be used to demonstrate an increase in the security posture across the population of CFATS facilities. In April 2013, we reported that DHS took various actions to work with facility owners and operators, including increasing the number of visits to facilities to discuss enhancing security plans, but that some trade associations had mixed views on the effectiveness of DHS’s outreach. We found that DHS solicited informal feedback from facility owners and operators in its efforts to communicate and work with them, but did not have an approach for obtaining systematic feedback on its outreach activities. We recommended that DHS take action to solicit and document feedback on facility outreach consistent with DHS efforts to develop a strategic communication plan. DHS agreed and has implemented this recommendation by developing a questionnaire to solicit feedback on outreach with industry stakeholders and began using the questionnaire in October 2016. In August 2018, we reported that DHS shares some CFATS information with first responders and emergency planners, but these stakeholders may not have all of the information they need to minimize the risk of injury or death when responding to incidents at high-risk facilities. While certain facilities are required under the Emergency Planning and Community Right-to-Know Act of 1986 to report some chemical inventory information, which local officials told us they rely on to prepare for and respond to incidents at chemical facilities, we found over 200 chemicals covered by CFATS that may not be covered by these reporting requirements. We also reported that DHS developed a secure interface called the Infrastructure Protection (IP) Gateway that provides access to CFATS facility-specific information that may be missing from required reporting. However, we found that the IP Gateway is not widely used at the local level and officials from 13 of 15 selected Local Emergency Planning Committees we contacted—consisting of first responders and covering 373 CFATS high-risk facilities—said they did not have access to CFATS data in the IP Gateway. We recommended that DHS should take actions to encourage access to and wider use of the IP Gateway and explore other opportunities to improve information-sharing with first responders and emergency planners. DHS concurred with this recommendation and reported in September 2018 that they are taking actions to implement it. Specifically, DHS has revised three fact sheets and an outreach presentation to include information on the IP Gateway and how to request access to it. In addition, DHS plans to ensure contact is made with first responders representing the top 25 percent of CFATS high-risk chemical facilities by no later than March 2019 so that they are properly prepared to respond to incidents at these facilities. Chairman Thompson, Ranking Member Rogers, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff members have any questions about this testimony, please contact me at (202) 512-3841 or andersonn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals making key contributions to this work include Ben Atwater, Assistant Director; Hugh Paquette, Analyst-in-Charge; Chuck Bausell, Michele Fejfar, Tracey King, and Tom Lombardi. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Thousands of facilities across the United States contain hazardous chemicals that could be used by terrorists to inflict mass casualties or harm surrounding populations. In accordance with the DHS Appropriations Act, 2007, DHS established the CFATS program to, among other things, identify and assess the security risk posed by chemical facilities. DHS inspects high-risk facilities after it approves facility security plans to ensure that the facilities are implementing required security measures and procedures. This statement summarizes progress and challenges related to DHS's CFATS program management. This statement is based on prior products GAO issued from July 2012 through August 2018, along with updates as of September 2018 on actions DHS has taken to address GAO's prior recommendations. To conduct the prior work, GAO reviewed relevant laws, regulations, and DHS policies for administering the CFATS program; how DHS assesses risk; and data on high-risk chemical facilities. GAO also interviewed DHS officials and relevant stakeholders. The Department of Homeland Security (DHS) has made progress addressing challenges that GAO's past work identified to managing the Chemical Facility Anti-Terrorism Standards (CFATS) program. The following summarizes progress made and challenges remaining in key aspects of the program. Identifying high-risk chemical facilities. In July 2015, GAO reported that DHS used self-reported and unverified data to determine the risk of facilities holding toxic chemicals that could threaten surrounding communities if released. GAO recommended that DHS should better verify the accuracy of facility-reported data. DHS implemented this recommendation by revising its methodology so it now calculates the risk of toxic release, rather than relying on facilities to do so. Assessing risk and prioritizing facilities. In April 2013, GAO reported weaknesses in multiple aspects of DHS's risk assessment and prioritization approach. To improve this process, GAO recommended that DHS enhance its risk assessment approach to incorporate all elements of risk and conduct a peer review after doing so. DHS implemented both recommendations by revising the CFATS risk assessment methodology to include threat, vulnerability, and consequence to better cover the range of security issues, and conducting peer reviews and technical reviews to verify and validate the CFATS program's new risk assessment approach. Reviewing and approving facility site security plans . DHS is to review facility security plans to ensure their security measures meet DHS standards. In April 2013, GAO reported a 7- to 9-year backlog for these reviews. In July 2015, GAO reported that DHS had made substantial progress in addressing the backlog—estimating that it could take between 9 and 12 months for DHS to review and approve security plans for the approximately 900 remaining facilities. DHS has since taken additional action to expedite these activities and has eliminated this backlog. Inspecting facilities and ensuring compliance. In July 2015, GAO found that nearly half of the facilities DHS had inspected were not fully compliant with their approved security plans and that DHS did not have documented procedures for managing facilities' compliance. GAO recommended that DHS document procedures for managing compliance. DHS revised CFATS procedures that, as of February 2019, GAO is reviewing to determine if they sufficiently address the recommendation. Conducting stakeholder and first responder outreach. In August 2018, GAO reported that DHS shares some CFATS information with first responders and emergency planners but these stakeholders may not have all of the information they need to minimize the risk of injury or death when responding to incidents at high-risk facilities. GAO recommended that DHS should, among other things, take actions to explore opportunities to improve information-sharing with first responders and emergency planners. DHS concurred with this recommendation and reported in September 2018 that it is conducting additional outreach and taking other actions to implement it. GAO has made various recommendations to strengthen DHS's management of the CFATS program, with which DHS has generally agreed. DHS has implemented or described planned actions to address these recommendations.", "document_type": "gao"}
{"report": "DOD has long recognized that the contracts for items like weapon systems are capital intensive in nature and take a long time to produce. Contract financing assists the defense contractor in managing expenses, such as material, labor, and overhead. In such cases, DOD can agree to help finance these expenses as the work progresses through various types of contract financing payments, including progress payments and performance-based payments. Progress payments based on cost are determined as a percentage of the costs incurred by the contractor as work progresses. Currently, DOD pays 80 percent of incurred costs of large business concerns and 90 percent of incurred costs of small business concerns. To receive progress payments on the basis of cost, contractors are required to have an accounting system and controls that are adequate for proper administration of progress payments. The cognizant contract administration office is to maintain surveillance of the contractor’s accounting system as appropriate, and the Defense Contract Audit Agency is to audit the accounting system. DOD provides contract financing on fixed-price type contracts for non- commercial items. Performance-based payments enable the contractor to be paid for achieving certain contractual milestones, such as delivery of a major subcontracted component. DOD can pay up to 90 percent of either the contract’s price or the price of a deliverable item using performance-based payments. DOD’s performance-based payments guide states that these payments should not be structured such that they amount to advance payments, which in general terms are payments made before work is complete on a contract regardless of what performance milestones are met. Unlike progress payments, however, performance-based payments do not require that the contractor have an adequate accounting system. Lastly, contract financing can also be used when the terms and conditions of a contract are not yet “definitized,” a term that generally means finalized. These actions, which are termed undefinitized contract actions (UCAs) at DOD, are to be used only when the negotiation of a definitive contract action is not possible in sufficient time to meet DOD’s requirements and the department’s interest demands that the contractor be given a binding commitment so that contract performance can begin immediately. The government may incur unnecessary costs if requirements change before the contract is definitized. Defense acquisition regulations generally require UCAs to be definitized within 180 days of the UCA date or before more than 50 percent of the estimated contract price is obligated, whichever occurs first. During this period, progress payments are limited to 80 percent of work accomplished. DOD’s 2014 performance-based payments guide recommends that a UCA be awarded using progress payments first; performance-based payments should then be considered during the definitization process. Table 1 summarizes the conditions and rates applicable to progress payments based on costs and performance-based payments. Several offices and agencies within DOD have a role in managing contract financing. The office of DPC, within DOD’s Office of the Under Secretary of Defense for Acquisition and Sustainment, is responsible for all pricing, contracting, and procurement policy matters. This office formulates and oversees DOD-wide pricing policies and strategies supporting the procurement of major defense programs, including programs that use progress and performance-based payments. DCMA and other contract administration offices monitor contractors’ performance and management systems to ensure that cost, product, and performance are in compliance with the contract terms. DCMA generally maintains contract financing payment data for DOD progress and performance-based payments for contracts DCMA administers. Within DCMA, the Cost and Pricing Center supports DOD-wide analysis of contract data to support DOD-wide decision making, among other things. Contract financing has an impact on the price of negotiated contracts and, more generally, on the health and profitability of the defense contractor. On negotiated contracts, DOD requires contracting officers to use weighted guidelines, a structured approach used to develop profit objectives for individual defense contracts. DOD implements its profit policy through the weighted guidelines. As part of their efforts to determine the government’s negotiating position, including how much profit the contractor should receive under the contract, contracting officers are to consider various factors, including the degree to which the government is providing contract financing. Assuming other factors are held constant, the weighted guidelines suggest that the negotiated profit rate of a fixed-price defense contract might be 1 to 2 percentage points lower when the government provides contract financing. The contracting officer may vary the amount to consider other risk elements when establishing the government’s negotiating position. DOD-provided contract financing can also provide contractors higher rates of return on the amount of corporate funds contractors invest on that same contract. One measure of this benefit is “internal rate of return” (IRR), a tool that can be used to assess the impact of contract financing on overall contractor profitability. DOD’s 2001 Incentives Guidebook notes that IRR is one of the basic tools used by industry to determine where to invest its funds and assess the risks and potential rewards involved in contracting with the government or commercial entities. IRR is a measure that integrates both the contractor’s investment to produce the product and the profit earned on that product. In contrast to contracts in which the contractors must either self-finance or borrow from commercial lenders, when contractors receive financing on a contract from the government the contractor’s IRR can be significantly higher. Figure 1 provides a hypothetical example of how changes in the progress payment rate on a 40-month, fixed-priced contract affects the expected contract profit rate and the contractor’s IRR. As illustrated above, providing contract financing (in this case, progress payments) has a significant impact on the contractor’s IRR and a lesser impact on the actual profit that DOD expects the contractor to make. For example, if DOD provided no contract financing, the weighted guidelines would suggest a profit rate on this hypothetical contract of 13.8 percent, which would provide an internal rate of return to the contractor of 7.5 percent. If DOD provided progress payments at the customary rate of 80 percent, the weighted guidelines would suggest a profit rate on this hypothetical contract of 10.4 percent, or 3.4 percentage points lower than if no financing was provided. However, even though the contractor’s expected profit is lower, the IRR for the contract would increase to 30.9 percent, or a little more than four times what would be realized if the contractor had to finance the effort on its own. Several studies conducted by DOD, nonprofit organizations and GAO have assessed the impact of contract financing on contract profit or contractor profitability. These studies have generally found that, depending on the measure used, the defense industry generates high returns on investment. For example, In 1976, DOD’s Profit ‘76 study examined earnings’ relationship to capital investment and increased productivity. The Profit ‘76 study group concluded that government contractors were able to maintain higher profits by keeping investment low partly because DOD did not have profit policies in place to encourage investment in items such as facilities. As a result of the Profit ‘76 study, DOD made a number of changes to its profit policy to encourage corporate investment in facilities, among other things. In 1991, GAO suggested that using return on assets to measure profitability of defense contractors is beneficial because it recognizes how government financing can affect contractors’ levels of profitability. In 2008, the Institute for Defense Analyses reported that defense contractors generated high returns with low operating margins, in part because government-provided contract financing helped fund the contractors’ long, asset-intensive product cycles. According to DPC officials, however, the most comprehensive study of contract financing and profit policies was conducted by the DFAIR commission in 1985. We discuss this study in more detail below. Since the DFAIR commission issued its report in 1985, Congress and DOD have made a number of changes to the statutory and regulatory framework intended to (1) reduce the administrative burden associated with contract financing and (2) encourage the use of performance-based payments (see figure 2). Our review found that DOD paid less in performance-based payments after making some changes to contract financing policies, but started increasing these payments again in 2016. According to DPC officials, the most comprehensive study of contract financing and profit policies was conducted by the DFAIR commission in 1985. The DFAIR commission assessed, among other issues, whether DOD contract financing policies were equitable in maintaining the defense industrial base and cost-effective for DOD, the effectiveness of DOD contract financing policies as a means of encouraging contractor cost efficiencies, the profitability of defense work and its reasonableness in comparison with the profitability of the non-defense sector, and the interrelationship of DOD’s contract finance and profit policies. In evaluating contractor financing costs, DFAIR developed a model of a typical contract to use in calculating contractors’ contract financing costs, the amount of interest a contractor would have to pay if it were required to bear all those costs, and the effect of payment delays on contractor financing costs. The DFAIR commission reached a number of conclusions about DOD’s contract financing and profit policy in effect at that time. The study concluded that: The progress payment rate was appropriate for the time period studied but should be revised based on changes in short-term interest rates. DOD’s profit policy as reflected in the weighted guidelines at the time of the study did not explicitly take into account the cost of working capital (the difference between a contractor’s assets and liabilities). The profitability of individual defense contracts the commission reviewed had been consistently lower than the profit levels reported to have been negotiated by government contracting officers. DOD’s profit policy needed to be simplified and better integrated with contract financing policy. The study also concluded that there was a need to make DOD contract financing more responsive to economic conditions and that profit policy, contract financing, and contractor investment are related. We agreed with the conclusion that profit policy, contract financing, and contractor investment are related. We also highlighted the need for recurring DOD contract profitability studies using a generally accepted methodology in our 1986 report. Our work found that since the DFAIR study was issued, DOD made several changes to reduce the administrative burden associated with contract financing requirements. These changes included Elimination of flexible progress payments (1999) – DOD introduced flexible progress payments in 1981 as a new approach to contract financing. Under flexible progress payments, DOD contracting officers were to use the DOD Cash Flow Computer Model to develop an applicable progress payment rate for that contract. Under this approach, DOD specified the minimum percentage the contractor was required to invest and DOD would provide the remainder. The amount of contractor investment required by DOD varied from 5 to 25 percent, depending upon the year. Flexible progress payments were not allowed on contracts issued after November 11, 1993; the references were eliminated completely from the DFARS in 1999. Elimination of “paid cost rule” (2000) – The paid cost rule required large businesses to pay subcontractors before billing the government for payment. After DOD eliminated this rule in March 2000, large businesses were generally able to include subcontract costs incurred but not yet actually paid on progress payment requests to the government. Elimination of “financial need requirement” (2016) – Since 2000, one of the ways contractors could receive progress or performance- based payments under the FAR was on the basis of financial need or the unavailability of private financing. In that regard, an April 2013 DOD Inspector General report found that contracting personnel did not properly negotiate and verify contractors’ need for contract financing before authorizing performance-based payments. The Inspector General recommended that contracting personnel determine whether private financing is available to a contractor before authorizing performance-based payments. While DOD concurred with the recommendation, it subsequently amended the DFARS in 2016 to eliminate the requirement for DOD personnel to justify the use of contract financing for certain fixed-price contracts. In doing so, DOD stated it was in DOD’s best interests. Congress enacted the Federal Acquisition Streamlining Act (FASA) in 1994 to provide the executive branch with requirements to improve the process for acquiring goods and services. FASA, among other things, established performance-based payments “wherever practicable” as a form of contract financing. In 1995, the FAR Council amended the FAR to enable the use of performance-based payments up to a maximum amount of 90 percent of the contract’s price. In 2000, DOD issued a rule amending the DFARS to emphasize that performance-based payments were the preferred method of financing. The rule required contracting officers to consider and deem performance-based payments impracticable before deciding to provide progress payments. This rule was part of a larger effort by DOD to make contract financing procedures easier to understand and to simplify related provisions. DOD subsequently issued a user’s guide in 2001 to help its contracting personnel and contractors in using performance-based payments. Despite the provisions to encourage the use of performance-based payments when appropriate, DOD subsequently initiated department- specific actions that, according to industry officials, decreased the frequency with which they received performance-based payments on defense contracts. For example, the Under Secretary of Defense for Acquisition, Technology and Logistics’ September 2010 Better Buying Power memorandum instructed contracting officers to use progress payments as the basis for price negotiations. After the contractor and DOD contracting officer agreed on price using progress payments, contractors could propose using an alternate financing arrangement, including performance-based payments. The memorandum indicated that the rationale for this change was to provide increased incentives for contractor performance. In April 2011, the Director of Defense Procurement and Acquisition Policy (now known as DPC) issued a memorandum that focused on the “practicality” of performance-based payments, stating they “are not practical for use on all fixed-price contracts and require considerable effort between the contractor and Government.” The memorandum noted that if contractors wanted to use performance-based payments, then the contractor should submit a proposed schedule to include all performance-based payment events, completion criteria, and event values, along with the contractor’s expected expenditure profile. To implement its April 2011 performance-based payment policy, DOD issued a proposed rule to amend the DFARS in January 2012. This rule was finalized in March 2014. The 2014 version of the DOD performance-based payments user’s guide noted that performance-based payments are the preferred method only when they are deemed practical by the contracting officer. However, industry officials told us that they frequently cannot reach agreement with DOD regarding performance milestones, and therefore agree to the use of progress payments instead. The impact of DOD’s changes on the relative use of progress versus performance-based payments is uncertain. Between fiscal years 2010 and 2018, DCMA data indicates that DOD provided between $36 billion and $49 billion a year in contract financing on contracts DCMA administered. We found that nearly 98 percent of those contract financing payments were paid to medium and large defense contractors. We also found that the amount DOD paid out in performance-based payments on those contracts fell between 2010 and 2016 before increasing in 2017. In December 2016, Congress enacted Section 831 of the Fiscal Year 2017 NDAA to establish performance-based payments as the preferred type of contract financing for DOD in statute. Section 831 also directed the Secretary of Defense to ensure that nontraditional defense contractors and other private sector companies are eligible for performance-based payments, in line with best commercial practices. Figure 3 shows the differences in DOD’s progress and performance-based payments between fiscal years 2010 and 2018 for contracts administered by DCMA. In August 2018, DOD introduced a proposed rule that was intended to use contract financing rates to help incentivize contractor performance and to implement Section 831. The proposed rule would have set a base progress payment rate for large businesses (specifically, for other than small businesses) at 50 percent and small businesses at 90 percent. At the same time, however, the proposed rule provided opportunities to increase the rate if the contractor achieved certain enterprise-wide priorities such as meeting contract delivery dates. The proposed rule also eliminated some of the administrative requirements associated with performance-based payments to encourage their use. According to DPC officials, the rates would be subject to an annual adjustment based on the performance criteria provided in the rule. Table 2 summarizes key aspects of the proposed rule. DOD officials acknowledged that if implemented, contractors would initially receive a lower level of contract financing, but believe with improvements in their overall performance contractors would eventually receive much higher levels of financing than currently provided. Industry officials voiced a number of concerns about the proposed rule at the January and February 2019 public meetings held after the rule was proposed, as well as in our interviews with them. For example, these officials noted that the proposed rule would change the intent of contract financing from a means of assisting contractors to help meet short-term expenses to a mechanism for ensuring compliance with contract terms and conditions on an enterprise-wide basis. Industry officials said they believe compliance with contract terms and conditions should be addressed on a contract-by-contract basis. Further, industry officials stated that the changes suggested in the proposed rule could negatively impact the health, competitiveness, and resiliency of the defense industrial base and introduce significant uncertainty as to how much contract financing DOD would provide. Additionally, industry officials noted that the rule did not contain specific implementation details in such areas as whether the incentives would be applied on an enterprise-wide basis and how to ensure the data were reliable. DOD withdrew this rule in October 2018, citing the need to conduct additional outreach with industry regarding contract financing methods. Subsequently, DPC held three public meetings in January and February 2019 to obtain public comments on revising policies and procedures for contract financing, performance incentives, and associated regulations prior to proposing a new rule. DPC provided no timeframes for doing so. DOD officials issued the proposed rule in April 2019 to implement Section 831’s statutory preference for performance-based payments for public comment. The proposed rule notes that performance-based payments are the preferred method of contract financing at DOD whenever practicable. The period for public comments ends on July 1, 2019. DOD officials indicated that they hope to issue a final rule in early 2020. DOD has not conducted a comprehensive assessment of the impact of its contract financing and profit policies on the defense industry since the DFAIR study was completed in 1985. In the intervening time, there have been significant changes in the composition of the defense industry, business practices, and economic conditions. In December 2018, DPC officials acknowledged the need to assess contract financing policies against market and economic conditions on an ongoing basis and determine the effect these policies have on the defense industry, but did not provide a timeframe for doing so. DOD officials acknowledged that the department has not done a comprehensive assessment of how its contract financing policies affect the defense industry since the DFAIR study was issued in 1985. DOD had previously stated its intent to do such an assessment on a regular basis. Specifically, in 1991 DOD noted that it would issue progress payment rates each February. DOD also noted that it would use the methodology from the DFAIR study to determine the progress payment rate based on short-term commercial interest rates. However, DOD removed the DFARS provision related to flexible progress payments in 1999. Overall, we found that DOD has adjusted the progress payment rate five times since the DFAIR study was completed, but only adjusted the progress payment rate twice since 1991 when DOD indicated its intent to assess the rate annually. DOD last changed the progress payment rate in 2001 (see table 3). Since the DFAIR study was conducted and DOD last assessed progress payment rates, DOD and industry officials noted that the composition of the defense industry has changed, as we have noted in our prior work. For example, in 1997, we reported that the end of the Cold War and the subsequent declines in DOD budgets resulted in, among other changes, a reduction in the number of defense contractors through various mergers and acquisitions. In our current work, DPC officials pointed to a changing proportion of subcontractors relative to prime contractors. Industry officials also identified the emergence of contractors who do not typically work with DOD and technology companies into the defense sector as an issue that should be considered when looking at contract financing and profit policies. According to industry officials, the industrial base has moved away from heavy industrial manufacturing toward technology and more sophisticated industry partners, including contractors who do not typically work with DOD. These officials noted that these contractors may not be eligible for contract financing because they may not have an approved cost accounting system needed to receive progress payments. In that regard, in July 2017, we reported that one company conducted a study that determined it would take at least 15 to18 months and millions to establish a government-unique cost accounting system. Industry officials also noted that the emergence of high-technology companies may pose a challenge to traditional defense contractors in terms of attracting financing and investment from commercial and private investors at competitive rates. Industry officials also identified changing business practices, including the increased use of UCAs, which affect their ability to use performance- based payments. Industry officials stated that it is more difficult to negotiate performance-based payments on UCAs, noting that DOD’s guidance suggests that performance-based payments should not be provided for UCAs until definitization occurs. Our review of DOD’s semi- annual reports to Congress on the use of UCAs found that the number of UCAs and unpriced change orders reported by DOD has varied between March 2014 and September 2018 (see figure 4). DOD reported that the total not-to-exceed dollar value of all UCAs and unpriced change orders was approximately $76 billion as of September 2018. Finally, market and economic conditions have changed since the DFAIR study. For example, at the time of the DFAIR study, short-term interest rates were around 8 percent, whereas the short-term interest rate in 2018 was 2 percent. Figure 5 shows the changes in short-term interest rates and inflation since 1980. Industry officials noted, however, that a comprehensive economic assessment of defense industry returns and the cost of contract financing policies should be conducted. For example, they noted that a reduction to progress payment rates in times of higher interest rates would increase their cost of working on complex contracts. Industry officials acknowledged that while interest rates have been low, they anticipate rates increasing in the near future. DPC officials acknowledged that DOD’s August 2018 proposed rule did not assess the proposed rule’s impact on the health and profitability of the defense industry. DPC officials noted that since the proposed rule was focused on incentivizing contractor performance, DOD’s supporting analysis did not include an assessment of how the proposed rule would impact the overall profitability of defense contractors (such as assessing the impacts to a contractor’s internal rate of return) or of the profitability of defense work relative to non-defense industry opportunities. Rather, DOD’s analysis estimated the total financial impact the rule would have on large and small contractors primarily based on interest costs. Further, DOD stated in its supplementary material that it did not consider the extent to which the contract profit policy (in the form of weighted guidelines) would need to be adjusted given the proposed rule changes. DPC officials explained that changes to the weighted guidelines would need to consider how such changes would support the intent of providing higher rates of contract financing for higher levels of contractor performance. If DOD were to only propose a change to the progress payment rate, DPC officials acknowledged that such an assessment should consider what changes, if any, would need to be made to the weighted guidelines. DPC officials said they conducted an informal analysis that assessed contractor profitability, but this analysis was not made publicly available. In December 2018, DPC officials acknowledged the need to assess contract financing policies against market conditions on an ongoing basis and determine the effect these policies have on the defense industry. GAO’s Standards for Internal Control in the Federal Government call for monitoring the effectiveness of systems and policies throughout an organization on a recurring basis. Until DOD conducts a comprehensive assessment and updates that assessment on a recurring basis, it will not be in a position to understand whether current or future contract financing policies are achieving their intended objectives. DOD and industry officials have acknowledged that the defense industry, economic and market conditions, legislative and regulatory requirements, and business practices have all changed since the issuance of the DFAIR study in 1985. Despite this recognition, DOD has not conducted a comprehensive assessment of how its contract financing policies affect the defense industry in more than 30 years. Without assessing the collective impact of these changes, DOD may be assuming too much financial risk or providing contractors with levels of working capital that are not commensurate with what is needed to help finance long-term projects, and affecting its ability to attract new entrants into the defense market. That assessment, however, should not be a one-time effort. A prior DOD study, our work, and the department have acknowledged the need to do so on a regular and recurring basis. Without a comprehensive and systemic assessment, conducted on a recurring basis, of DOD’s contract financing policy’s effect on the defense industry, DOD will not be in a position to understand whether current or future policies are achieving their intended objectives. We recommend that the Acting Secretary of Defense direct the Under Secretary for Acquisition and Sustainment to ensure it conducts a comprehensive assessment of the effect that its contract financing and profit policies have on the defense industry and update that assessment on a recurring basis. (Recommendation 1) We provided a draft of this report to DOD for review and comment. DOD provided written comments, which are reprinted in appendix I, and concurred with our recommendation. In concurring with our recommendation, DOD stated it would seek fiscal year 2020 funds to contract a study on DOD contract financing policies and their effect on the defense industry. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of the report to the Acting Secretary of Defense; the Principal Acting Director, Defense Pricing and Contracting; the Director, Defense Contract Management and Agency; the Director, Office of Management and Budget; the Administrator for Federal Procurement Policy, and appropriate congressional committees. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3665 or dinapolit@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. In addition to the contact named above, Bruce H. Thomas, Assistant Director; Lorraine Ettaro, Elizabeth Field, Gina Flacco, Stephanie Gustafson, Kristen Kociolek, John Lopez, Beth Reed Fritts, Miranda Riemer, Anne Stevens, Megan Stewart, Anne Louise Taylor, Alyssa Weir, Robin Wilson, and Alex Winograd made key contributions to this report.", "summary": "Each year, DOD provides contractors with billions of dollars in contract financing on fixed-price contracts for major weapons systems and other long-term efforts. Contract financing helps contractors manage expenses until they begin delivering the contracted items to DOD. Contract financing can take several forms, including progress payments based on the cost incurred by the contractor, and performance-based payments, in which the government pays the contractor an agreed-to amount for achieving certain milestones. DOD last performed a comprehensive assessment of its contract financing polices in 1985. The Conference Report accompanying the Fiscal Year 2019 National Defense and Auhorization Act included a provision for GAO to analyze the level of financing currently provided to contractors, among other things. This report (1) describes changes in DOD contract financing policy since 1985 and (2) assesses the extent to which DOD has analyzed the effect of its contract financing policies on the defense industry. GAO assessed relevant legislation and DOD regulations; obtained data on DOD's use of progress and performance-based payments from fiscal years 2010 through 2018; and interviewed cognizant DOD and industry officials. Congress and the Department of Defense (DOD) have changed the contract financing legislative and regulatory framework since DOD last performed a comprehensive assessment, including eliminating a requirement that contracting officers justify a need for contract financing and establishing a preference for performance-based payments. However, Defense Contract Management Agency data indicates that the amount of performance-based payments it administered fell from 2010 to 2016 (see figure). DOD officials acknowledged that DOD has not comprehensively analyzed how its policies affect the defense industry since 1985. Industry and economic conditions, however, have since changed, including lower interest rates and the emergence of contractors who do not typically work with DOD. In August 2018, DOD proposed introducing performance-based elements into its process for setting progress payment rates. DOD officials stated that since the proposed rule focused on incentivizing contractors' performance, they did not assess how it would affect defense contractor profitability or whether other financing or profit policies changes would be needed. DOD withdrew the proposed rule in October 2018. GAO's Standards for Internal Control in the Federal Government call for organizations to monitor the effectiveness of their policies on a recurring basis. In December 2018, DOD officials acknowledged the need to do so. Until DOD conducts a comprehensive assessment and ensures they are done on a recurring basis, it will not be in a position to understand whether current or future contract financing policies are achieving their intended objectives. GAO recommends that DOD ensure it conducts a comprehensive assessment of the effect that its contract financing and profit policies have on the defense industry and update that assessment on a recurring basis. DOD concurred with the recommendation.", "document_type": "gao"}
{"report": "Schools generally report incidents of restraint and seclusion to their district, and districts are responsible for reporting incidents of restraint and seclusion to Education’s CRDC. Districts are expected to report the number of incidents and the number of students affected for all schools in their district and to use Education’s definitions of restraint and seclusion to determine whether an incident occurred. Education defines two types of restraint: physical and mechanical (see table 1). Education’s definition of a physical restraint makes a distinction between a restraint and a physical escort. Similarly, Education’s definition of seclusion makes a distinction between seclusion and a timeout (see figures 1 and 2). Every two years, OCR administers the CRDC to nearly every public school district in the country. In turn, districts self-report information on a wide variety of topics, including course offerings, discipline, and restraint and seclusion. Education collects these data through an online submission tool. CRDC activities, such as data collection and quality, are managed by a company under contract with Education. The data submission period for the 2017-18 school year ended June 21, 2019. School districts have one year from the end of the submission period to make a request to amend submitted data. As of March 2020, Education had not announced when it will publicly release these data. Education’s primary data quality checks for the CRDC data, including the restraint and seclusion data, are built into the CRDC submission tool. The online CRDC submission tool automatically performs checks that flag data errors or potential errors. These “business rules” occur in real time as districts enter data or after they upload files. The 2015-16 submission tool used three business rules related to restraint and seclusion; for the purpose of this report, we focused on the two rules most relevant to our work. The first business rule pertains to the reporting of zeros for very large districts only—that is, those districts with 100,000 or more enrolled students (see fig. 3). If a very large school district enters zero incidents of restraint and seclusion, it receives a message prompting it to review its enrollment counts and reported incidents, or provide an explanation using a reason code and comment. Importantly, if districts have not collected the data required for the CRDC—or if the data are unavailable for some other reason—districts are to leave relevant data cells blank. A zero in a data cell should represent an actual count— that is, the district restrained or secluded zero students. The second rule applies to schools that both report more than 100 incidents of restraint or seclusion and enter a greater number of students affected by restraint or seclusion than the number of incidents reported (see fig.4). Districts are asked to resolve this error by adjusting their counts so the number of students subjected to restraint or seclusion is less than the number of incidents of restraint or seclusion, or to provide an explanation using a reason code and comment. The last step in the data submission process is the district certification. To complete this step, the district superintendent or an authorized designee must indicate agreement with a statement that acknowledges that they are responsible for verifying the data, the information provided is “true and correct,” and a willfully false statement is punishable by law. The CRDC submission tool is designed so that it should not allow a district to certify its submission unless all required data pass the system validation checks, or all errors are explained. Education also reviews CRDC data quality during other phases of data collection to identify potential data quality issues to improve future collections. For school year 2015-16, Education’s contractor reviewed data quality during and after the collection phase. Education’s contractor contacted school districts about potential errors that Education determined were easily adjustable and asked them to review and correct data or provide an explanation if no corrections were determined to be necessary. Further, some states and school districts have laws, regulations, and/or policies regarding restraint and seclusion. These laws vary from state to state, and sometimes require schools or districts to annually report incidents of restraint and seclusion to either the state or local education agency. In January 2019, Education announced an initiative to address inappropriate use of restraint and seclusion on children with disabilities. As part of this initiative, OCR announced plans to conduct 50 data quality reviews of the 2015-16 restraint and seclusion CRDC data submitted across each of OCR’s 12 enforcement regions. This review had not been completed at the time of our review. In August 2019, Education announced plans for OCR to work collaboratively with Education’s National Center for Education Statistics (NCES). Education stated that the collaboration would help ensure that CRDC data are reliable and authenticated in a manner that provides a more accurate picture of key civil rights issues in education. According to the announcement, under a new agreement, NCES and OCR will work together to improve the quality of the CRDC data by providing school districts with technical assistance, and by reviewing and revising data quality procedures. NCES is the primary federal entity responsible for collecting and analyzing statistical data related to education. We identified four key issues for which Education’s quality control processes for its CRDC restraint and seclusion data are largely ineffective or do not exist (see table 2). Education’s business rule to detect potentially erroneous reporting of zero incidents applied to only 30 of the more than 17,000 school districts nationwide, rendering it largely ineffective for checking the 70 percent of districts that reported zero incidents of restraint or seclusion. This is because the rule only applied to districts with over 100,000 enrolled students. However, in its January 2019 data quality review of 50 districts’ restraint and seclusion data, OCR found erroneous reporting of zeros in districts of all sizes. Of the 50 districts OCR contacted, OCR determined that 40 districts should amend their original 2015-16 submissions. When we analyzed the 2015-16 CRDC restraint and seclusion data, we found that almost three-quarters of small districts reported zeros, while about one-third of large and one-fourth of very large districts reported zero incidents (see table 3). The findings from Education’s data quality review, along with those from our analysis, suggest that misreporting is a problem among districts of all sizes. For the 2017-18 CRDC data collection, Education lowered the threshold for the rule to detect potentially erroneous reporting of zeros to include districts with 50,000 enrolled students, rather than only districts with more than 100,000. However, the new rule counts students with disabilities and students without disabilities separately. To trigger the rule, a district would have to have at least 50,000 students with disabilities and report zero incidents for them, or have at least 50,000 students without disabilities and report zero incidents for them. Of the nation’s more than 17,000 school districts, only 3 (Chicago, Los Angeles, and New York) have at least 50,000 students with disabilities; only 95 have at least 50,000 students without disabilities. Education’s CRDC data show that restraint and seclusion disproportionately affect students with disabilities and its data quality review showed that substantial portions of districts of all sizes inaccurately reported zeros. However, Education could not provide a data-driven basis for the 100,000 or 50,000 student enrollment thresholds or for creating separate thresholds for students with and without disabilities. Rather, Education stated that the thresholds were a management policy decision inherited from previous administrations. Because Education’s business rule targeting districts that report zeros was inadequate, in June 2019, GAO recommended, among other things, that for the 2017-18 data collection Education contact districts that reported all zeros for restraint and seclusion to ask them to ensure that the zeros actually represented zero incidents, and Education did so after the data collection closed. Absent a business rule targeting all districts reporting zeros during data submission, inaccuracies in future data collections will likely be missed, and Education risks expending more time and resources with repeated manual follow up after the fact. Federal standards for internal control state that management should evaluate both internal and external sources of data for reliability. Absent reliable and accurate data, the public’s confidence in the CRDC restraint and seclusion data may be further undermined, and the utility of a dataset intended to assist with federal civil rights monitoring, enforcement, and oversight will remain limited. While it is clear that some school districts have reported inaccurate restraint and seclusion data, Education officials do not fully understand why this is occurring. In technical comments on a draft of this report, Education stated that it is committed to learning more about why this is occurring. While we do not know all of the reasons districts fail to report accurate data, our interviews with over 50 school and district officials provide some insight. School officials in the nine districts we visited cited a variety of reasons districts might not report, including that they were not collecting the data because their state did not require reporting, and that their school district only required them to collect data for students with Individualized Education Programs (IEPs). More fully understanding why districts report inaccurate data is key to correcting the issue. Federal standards for internal control also state that managers should use quality information to achieve the entity’s objectives, assess the risks facing the entity as it seeks to achieve its objectives, and use this assessment to develop appropriate risk responses. By not identifying school districts’ reasons for reporting zero incidents of restraint and seclusion, Education will not know how to best support districts in improving the accuracy of their reported data. Future CRDC data will remain inaccurate, significantly limiting the utility of a key tool on which OCR relies to help it enforce federal civil rights laws. Education has no business rules that flag school districts reporting very low or very high rates of restraint or seclusion, nor has it completed initial efforts to determine a range of rates that might warrant further exploration. Given widespread concerns about misreporting, we devised two possible ways to test for these types of outliers. First, we looked beneath the district level to examine school-level reporting patterns within districts. When we tested the nation’s 30 largest school districts (those with more than 100,000 students), we found patterns that may suggest underreporting in at least 13 of them, in addition to the 10 that reported zeros for the 2015-16 school year. In these 13 districts, we found that all of the incidents of restraint reported occurred in no more than 15 percent of a district’s schools; the rest of the schools in those districts reported zero incidents. (See fig. 5.) For example, the Chicago Public School District—the third largest school district in the country, with nearly 400,000 students enrolled—reported a total of 47 incidents of restraint for school year 2015-16. All of these incidents were reported by just two of its 579 schools. The district’s six incidents of seclusion were clustered in one school. In the Los Angeles Unified school district, the second largest school district in the country, 82 of its 785 schools reported a total of 108 incidents of restraint, with 65 schools reporting exactly one incident each. The district reported no incidents of seclusion. Education has a business rule that targets large districts (for 2017-18, those with over 50,000 students), but only when all schools in a district report zeros. Thus, as long as a large district reports at least one incident of either restraint or seclusion, the business rule would not be triggered. Education’s post-collection data quality reviews for school year 2015-16 did not test below the district level to look for potential underreporting within a district. Second, we tested for outliers by comparing per capita rates of restraint or seclusion in the 30 very largest districts (over 100,000 students enrolled) to average rates in all school districts. In the 30 districts, we found that in addition to the 10 districts that reported zeros, nine districts reported fewer than three incidents of physical restraint per 10,000 students, which is lower than 95 percent of all districts reporting incidents. (See fig. 6.) For example, DeKalb County school district in Georgia reported 0.3 incidents per 10,000 students, and Charlotte Mecklenburg school district in North Carolina reported 0.5 incidents per 10,000 students. We also tested for districts with very high rates of physical restraint. For the 2015-16 school year, we found 52 districts that were outliers, most of which served comparatively large populations of students with disabilities. Forty-nine of these 52 districts had rates of physical restraint per enrolled student higher than 99 percent of all districts that reported incidents of physical restraint. Almost half reported an average of 10 or more incidents per student affected, and almost two-thirds of the districts reported restraining from 25 to100 percent of their students. The Learning Tree preschools in Alabama, which enrolled a combined 135 preschool students ages 3 to 5, reported that it restrained nearly two-thirds of its students in school year 2015-16. Further, Learning Tree reported 5,963 incidents of physical restraint affecting 84 students, or an average of 71 incidents of physical restraint per preschooler. The Morris-Union Jointure Commission School District in New Jersey, where almost all of its 281 students were identified as having a disability, reported restraining over one-third of its students. These students were restrained an average of 20 times in school year 2015-16. (See table 4.) We found a similar pattern in the 2015-16 seclusion data, identifying 36 outlier school districts. For 22 of the 36 districts, the rates of seclusion were higher than 99 percent of districts reporting. (See appendix IV for more information.) For example, CRDC data for the Sangamon Area Special Education District in Illinois, which enrolled 74 students in grades 1 through 12, showed the district secluded one-third of its students an average of 27 times each in school year 2015-16. Similarly, data for the Bi-County Special Education Cooperative, also in Illinois, showed the district secluded over two-thirds of its 48 enrolled students an average of 13 times each in school year 2015-16. Federal standards for internal control state that management is to determine if controls individually and in combination with other controls are capable of achieving an objective and addressing related risks. An internal control design may be deficient when a control necessary to meet an objective is missing. Without business rules or similar analytical processes to flag these outliers, they may continue to go undetected by Education and other stakeholders. Education may be missing an opportunity to identify districts with disproportionately low or high incidents of restraint and seclusion to determine where technical assistance or other intervention may be warranted. Such information is particularly critical given widespread concerns about underreporting and misreporting, and its stated interest in protecting students’ civil rights. Education has a business rule that identifies illogical data; that is, when schools report more students affected than incidents of either restraint or seclusion. However, the rule is largely ineffective because it was not designed in a way that would detect logical inconsistencies in the majority of cases, as the rule would have only applied to schools with at least 100 incidents. When we tested Education’s rule on the 462 schools that reported at least 100 incidents in 2015-16, we found no logical inconsistencies in the data. However, when we tested the rule on all schools, we found logical inconsistencies in the data reported by 592 schools with fewer than 100 incidents. For example, a school in Indiana reported that it restrained 156 students, but only reported 80 incidents. (See table 5.) Education could not provide any data-driven basis for its threshold of 100 incidents for this business rule. Education officials said that the threshold was inherited from previous administrations’ business rules. Collecting accurate data is key to the Office for Civil Rights’ (OCR) mission to ensure equal access to education and to promote educational excellence throughout the nation. In addition, federal standards for internal control state that when evaluating the design of internal controls, such as business rules, management should determine if controls are capable of achieving an objective and addressing related risks. An internal control design is deficient if, even though it operates as designed, it does not meet the control objectives. Our analysis shows that the business rule is not effective in its current form, because 592 schools were able to report illogical, and therefore incorrect, data. We talked to more than 50 officials in nine school districts in Kentucky, Washington, and Wisconsin about their interpretations of the CRDC’s definitions of restraint and seclusion. These school districts all reported incidents of restraint and seclusion in 2015-16, but officials we interviewed differed in their interpretations of terms used in the CRDC definitions, such as alone and escort. As a result, districts varied in how they counted incidents of restraint and seclusion. Further, officials we spoke with in the three state educational agencies and all seven stakeholder groups with expertise on the use of restraint and seclusion in public schools also said there was ambiguity regarding terms used in the definitions. For example, an official from one stakeholder group that represents some of the nation’s largest school districts said that its constituents provided feedback that restraint and seclusion terms were ambiguous, open to interpretation, and do not provide enough clarity. Civil Rights Data Collection Definition of Seclusion: Seclusion is the involuntary confinement of a student alone in a room or area from which the student is physically prevented from leaving. It does not include a timeout, which is a behavior management technique that is part of an approved program, involves the monitored separation of the student in a non- locked setting, and is implemented for the purpose of calming. With respect to the definition of seclusion, district and school officials varied in their interpretations of the word alone, and consequently, whether the incident should be counted as seclusion. Officials in three districts said that an incident was not seclusion as long as a teacher was in the room with the student, while officials in several other districts said that even if a teacher was present, it could still be seclusion if the student was prevented from leaving. (See sidebar.) Officials in the nine districts we visited also varied in their interpretation of the word area. Because the CRDC’s definition of seclusion states that seclusion can occur in an area, officials from one stakeholder group representing thousands of school administrators wondered whether it should be considered seclusion if a child is in a classroom with 20 other students and is required to stay alone in the corner of the room. Officials from a district in Wisconsin said that if a student is taken away from peers and placed in one area of the same room, but cannot leave that area, it still might be seclusion, even if the student and peers are in the same room. Officials in another district in Wisconsin said that sending a student to a corner does not count as seclusion. However, they said that the use of mobile partitions to close off an area of a room could constitute seclusion. The phrase physically prevented from leaving also elicited differing interpretations. Officials from the stakeholder group representing administrators said the definition is not clear about what counts as “prevented from leaving”: a barrier, such as a door; the presence of another adult watching the child; or both. School officials we spoke with had differing interpretations of this phrase, which affected how they counted and reported incidents of seclusion. School officials in a district in Kentucky said that the phrase means closing the door and keeping it closed. However, officials in another school in the same district did not specify the use of a door, stating instead that “physically prevented from leaving” means the student cannot walk out of the room. A school official in Washington said that it would count as seclusion if staff put a student in a motorized wheelchair in a room and deactivated the wheelchair’s power. Civil Rights Data Collection Definition of Physical Restraint: Physical restraint is a personal restriction that immobilizes or reduces the ability of a student to freely move their torso, arms, legs, or head. The term does not include a physical escort. Physical escort means a temporary touching or holding of the hand, wrist, arm, shoulder or back for the purpose of inducing a student who is acting out to walk to a safe location. With respect to the definition of physical restraint, school district officials we interviewed varied in their interpretations of the term escort, which the CRDC definition specifies is not a physical restraint. (See sidebar.) While officials in three districts said that an escort meant providing a physical prompt to a student who was not resisting relocation, officials in four districts said that moving a student who was resisting staff still counted as an escort. For example, officials in a school in Wisconsin said that if the student who is resisting is “carried away” from a location, that action would not meet the definition of escort and would count as restraint. Yet a district in Kentucky counted moving students against their will—including by carrying them—as escorts and did not report them as restraints. Further demonstrating differing interpretations of these terms, officials in four districts said they reported all escorts as restraints in the CRDC. Education does not provide schools or school districts with any information that could help clarify its definitions or provide examples on how schools and school districts should apply the definitions of restraint and seclusion to common classroom situations. For example, while Education’s “Restraint and Seclusion: Resource Document” includes the CRDC definitions of restraint and seclusion, it does not include clarifying information or examples about how to apply the definitions. Officials in the schools and districts we visited inconsistently interpreted the definitions for restraint and seclusion; moreover, officials from the seven stakeholder groups we interviewed said the definitions were unclear. These findings raise concern about whether restraint and seclusion data reported by school districts to the CRDC are being reported in a way that is consistent with the CRDC definitions. Federal guidance on data reliability states that data should be well defined enough to yield similar results in similar analyses. In addition, federal standards for internal control state that agency management should use quality information to achieve the entity’s objectives, noting that such data should be reasonably free from error and bias and faithfully represent what they purport to represent. Absent data on restraint and seclusion that is what it purports to be, Education will continue to lack quality information key to fulfilling its mission of ensuring equal access to education nationwide. Officials in all nine school districts we visited said they used their data on restraint and seclusion to help reduce its use. In addition to collecting data for CRDC reporting purposes, these districts also collected and used more current and more detailed data to help reduce the use of restraint and seclusion. Officials in seven of the nine districts said they began collecting the data when their state passed a law requiring reporting. District officials identified several benefits to collecting data and using it to develop strategies to reduce use of restraint and seclusion. Specifically, officials said that the data helped them identify the following: Behavior patterns. Officials in several districts told us that collecting and reviewing data on restraint and seclusion helps them identify patterns in staff and student behavior that may contribute to use of these practices. Specifically, by identifying the circumstances under which a student’s behavior tends to escalate, staff can strategize how to more effectively respond so as to prevent the need to use restraint or seclusion. For example, one official in an elementary school in Wisconsin said that if staff notice more incidents occur on particular weekdays, they can examine those days to understand what may be affecting students’ behavior. Similarly, a teacher of students with autism in a middle school in Washington said that reviewing data helps staff, such as teachers, paraprofessionals, and administrators, determine what triggered a student’s behavior and then determine what to do differently to avoid triggering the student. In all three states, we visited districts that required staff to participate in a debriefing after each incident in an effort to understand what might have triggered the event and to discuss strategies to deescalate future incidents. For example, officials in Washington said that the building administrator and all staff involved discuss every incident. District officials consider this an important step for reducing use of restraint and seclusion, and said holding the discussions was a “game- changer.” Need for training. Officials in several districts said they examine data on restraint and seclusion at the classroom and school level to determine if staff need additional training, including on how to manage student behavior, or appropriately use restraint or seclusion. For example, a behavior coach for a Kentucky school district said that the data on restraint and seclusion helps her determine if certain teachers could benefit from more training on de-escalation techniques. A director of student services in Washington said that he was concerned about the rates of restraint and seclusion in the district, and after implementing more training for teachers, the rates declined. Officials in another Washington district said that after the district began collecting data in response to state law, they discovered that staff were using restraint and seclusion as punishment. As a result, district officials said that they coached teachers on how to manage behavior differently and emphasized that restraint and seclusion should not be an everyday occurrence. Officials at an elementary school in Wisconsin said that de- escalation training helps staff understand that students are trying to communicate with their behaviors. They said that when staff adopt the perspective that students are trying to communicate, staff also see the value of collecting data to improve how they respond to the students’ behaviors. Need for student supports. Officials in five of the nine districts we visited spoke about using restraint and seclusion data to assess when a student required additional support services to be successful in the classroom. For example, officials in one district in Kentucky said their data provides evidence for obtaining additional staff or social emotional learning resources for students. Similarly, officials in one Wisconsin district said the data can be used to allocate funding for school-based services to help address underlying causes of behavior. Officials in another Wisconsin district said that a jump in restraints or seclusions of a particular student could indicate that the student’s individualized education program needs to be adjusted. Officials in all the districts we visited also shared strategies on how they improved their CRDC data reporting, including communicating with staff about how data are used, training on how to report, and developing processes that encourage reporting. Specifically: Communication and culture. Officials in the majority of school districts said they routinely reviewed their data with school staff and emphasized the value of collecting data on restraint and seclusion. Officials in a district in Wisconsin said that they monitor data on restraint and seclusion on a monthly basis for students with and without disabilities, which increases interest among school staff about what causes the incidents. Officials in three districts we visited said that they explain to staff that documenting incidents of restraint or seclusion ensures that students obtain the support services they need. In a Wisconsin district, officials said they emphasize that reporting helps the students and keeps the school safe by making the district aware that more supports are needed. Officials at a school in Wisconsin said that some staff might worry that the data reflect poorly on them or might fear repercussions, but district officials have worked to shift the culture of reporting to focus on continuous improvement and problem solving. Similarly, officials in another Wisconsin district said that schools might be concerned about the data being used against them; therefore, district officials try to create a culture of curiosity around the data, rather than a culture of punishment. Accountability. To encourage staff to report incidents, officials in some districts developed processes that increased accountability for reporting. For example, school officials at an elementary school in Wisconsin said an administrative assistant in the main office immediately logs calls from classroom teachers requesting help managing a student’s behavior. Officials said this process provides accountability. Two districts said that they used a team approach for restraint or seclusion, which included someone to observe and someone to record details of the intervention, such as the time it began or the events that preceded it. Having multiple people involved increased the likelihood that relevant facts were recorded. Officials in a district in Washington said that keeping teachers and staff honest about reporting requires reiterating the process and procedures, reviewing the forms with staff, and following up with schools that fail to submit reports. Officials in a district in Wisconsin said they have advised staff to write the incident down on paper until staff are able to enter it in the district’s electronic reporting system. Training. Officials from all nine districts said they encourage reporting by provided training on how to report incidents. Generally, this information was incorporated into trainings on when to use restraint and seclusion and how to deescalate a student’s behavior. Officials from five stakeholder groups we interviewed, all of whom have expertise related to the use of restraint and seclusion in public schools, stated that training was necessary to both raise awareness of the requirement to report incidents and to ensure that incidents were reported accurately. For example, in de-escalation training for teachers in a Washington district, the trainer provides examples of restraint and seclusion; presents a variety of scenarios, including ambiguous ones, for discussion; and reviews the appropriate staff response. The Civil Rights Data Collection (CRDC) is a longstanding and critical aspect of Education’s Office for Civil Rights’ overall enforcement and monitoring strategy. Collecting accurate data through the CRDC can help Education in its mission to ensure equal access to education, promote educational excellence for all, and enforce various federal civil rights laws prohibiting discrimination on the basis of race, color, national origin, sex, and disability. However, the significant data quality problems that both Education and we identified with the CRDC data on restraint and seclusion, combined with the significant weaknesses we found in Education’s data quality control processes, cast serious doubt on the accuracy of these data. As a result, it is impossible to accurately determine the frequency and prevalence of restraint and seclusion among K-12 public school students. The four recommendations in our June 2019 report urged Education to take immediate steps to address the widespread potential misreporting of zeros for its 2017-18 CRDC. Education took some steps to address the issues we raised, but has not yet fully addressed them. Moreover, those recommendations were intended as stop-gap measures to improve the quality of the 2017-18 data being collected in real time precisely because the CRDC’s business rules related to restraint and seclusion are inadequate. Therefore, addressing our recommendations would not solve the issues that are the subject of this report. Our work makes it clear that an overhaul of the quality control processes is needed to correct fundamental problems with federal restraint and seclusion data collected through the CRDC. Two of the CRDC’s key business rules meant to check data quality and flag potential errors in restraint and seclusion data are poorly designed and the thresholds that trigger these rules have no data-driven basis. Further, Education does not have business rules designed to flag outlier schools and school districts that report relatively low or high rates of restraint and seclusion, nor has it determined a range of rates that might warrant further exploration. Until Education more fully understands why so many school districts are underreporting and misreporting federal restraint and seclusion data, it will likely not be able to help districts improve their reporting, thereby improving the accuracy and utility of the data. There were widely varied interpretations of federal restraint and seclusion definitions among the 50 school and district officials with whom we spoke and officials from the seven key stakeholder groups we interviewed echoed these concerns. As a result, we have concerns that school districts may be inconsistently counting and reporting instances of restraint and seclusion for federal reporting purposes. Clarifying the definitions, for example by explaining to districts how they can be applied to common classroom scenarios, could help produce more consistency in reporting. Ultimately, the issues we found with Education’s restraint and seclusion data have consequences for the students who are restrained or secluded in school and whose restraint or seclusion goes un-reported. When federal data are misreported to the public, it undermines confidence in that data and fails to provide decision makers with reliable information on which to make informed policy decisions to protect students. In addition, Education lacks information that could help it determine whether schools’ use of these practices may be excessive, discriminatory, or both. GAO is making six recommendations on restraint and seclusion to the Department of Education’s Office for Civil Rights. The Assistant Secretary for the Office for Civil Rights should revise its CRDC business rule to require that every school district reporting zeros, regardless of district size or numbers of students with disabilities, affirm the zeros are correct during the CRDC data submission process. (Recommendation 1) The Assistant Secretary for the Office for Civil Rights should develop and implement a CRDC business rule that targets schools and school districts that report very low numbers of incidents and set data-driven thresholds to detect such incidents. (Recommendation 2) The Assistant Secretary for the Office for Civil Rights should develop and implement a CRDC business rule that targets schools and schools districts that report very high number of incidents and set data-driven thresholds to detect such incidents. (Recommendation 3) The Assistant Secretary for the Office for Civil Rights should apply the CRDC business rule targeting illogical data at the school level to all schools, regardless of the number of incidents reported. (Recommendation 4) The Assistant Secretary for the Office for Civil Rights should identify the factors that cause underreporting and misreporting of restraint and seclusion and take steps to help school districts overcome these issues. (Recommendation 5) The Assistant Secretary for the Office for Civil Rights should further refine and clarify federal restraint and seclusion definitions and take steps to ensure that this information is conveyed to school districts. This could include providing common classroom scenarios that highlight the differences between a restraint and an escort, and a time out and a seclusion. (Recommendation 6) We provided a draft of this report to the Department of Education for review and comment. In its formal comments, which are reproduced in appendix II, Education agreed with all six recommendations. Education also provided technical comments, which we incorporated, as appropriate. In agreeing with GAO’s six recommendations, Education stated that it would determine the best means to implement them. Education also stated that it is fully committed to working with public schools, state educational agencies, and school districts to help ensure accurate reporting of federal restraint and seclusion data, and to improve the quality of the information for all users of CRDC data. We appreciate Education’s willingness to address the serious data quality issues affecting the CRDC restraint and seclusion data. In its response, Education stated that the agency has already made significant improvements to the CRDC in general and has made specific improvements with respect to restraint and seclusion data, especially in response to the four recommendations we made in our June 2019 correspondence. Education asked that we acknowledge the progress it feels it has made in this regard, and we have done so. Importantly, however, our June 2019 recommendations were intended as stop-gap measures to improve the quality of the 2017-18 data that was already being collected in real time precisely because the CRDC’s business rules related to restraint and seclusion were inadequate. Therefore, steps Education has taken toward addressing them do not address the underlying data quality issues that are the subject of this report. In other words, the recommendations in this report urge Education to address data quality problems at the front-end by applying adequate business rules at the time districts submit their data. This could reduce the need for follow-up with districts to correct potentially inaccurate data. More information about our assessment of the steps Education has taken to address the four recommendations from the June 2019 report are available on our website. Education also stated that because our draft report did not mention the methodological improvements OCR made to address the quality of restraint and seclusion data for the 2017-18 CRDC data collection, our draft report overstates the relevance of the data issues from the 2015-16 collection. Education also stated that it provided us with information about the methodological improvements in December 2019, and, in its formal response, requested that we reflect the information in this report. We disagree with this perspective. After we completed our audit work for this engagement, Education provided us an excerpt from its post-collection data quality report for school year 2017-18. At that time and again in its technical comments on this report, Education stated that “information shared with GAO about the results of the 2017-18 data quality review process and what might be addressed is still confidential.” As of March 23, 2020 Education described the 2017-18 data quality control process as “incomplete.” Lastly, the 2017-18 CRDC data, which are the topic of the excerpt Education provided to us, are not yet available. Under our auditing standards, we cannot opine on the quality of data we could not independently assess or on the efficacy of process improvements associated with those data. Education also raised concerns about how we weighted our interviews with school officials, and it questioned the relevance of our discussions about selected school districts’ use of restraint and seclusion data not reported for CRDC purposes. We disagree. Education was concerned about the weight GAO placed on information obtained from 50 officials in 11 school districts across 3 states whom we interviewed during the course of our audit work. As stated in the report, this information cannot be generalized to all districts. However, we believe it provides useful insights into how some districts use their restraint and seclusion data to reduce the incidence of these practices and improve the accuracy of their data. The widespread disagreement among the 50 school officials with whom we spoke also highlights confusion about how to accurately and consistently apply CRDC definitions of restraint and seclusion. This finding is supported by the views of seven nonfederal advocacy organizations that represent parents and families; individuals with disabilities; and other stakeholders, such as representatives of relevant school and special education professional associations. Education questioned the relevance of discussing the benefits that selected school districts said they derive from using restraint and seclusion data not reported for CRDC purposes. Education stated that “attempting to generalize comments about how these nine school districts use restraint and seclusion data” seems inconsistent with Governmental Accounting Standards Board (GASB) statistical principles. We believe that describing selected school districts’ use of their restraint and seclusion data is within the scope of our stated audit objectives. In addition, the explanatory statement from the House Committee on Appropriations accompanying the Consolidated Appropriations Act of 2018 includes a provision for us to provide examples of how schools are adopting effective alternatives to these practices and reducing the incidence of seclusion and restraint, among other things. Further, Education mistakenly asserts that none of the data and analyses that the school districts collected, performed, or used are part of the CRDC and none could be feasibly collected by the CRDC. We have further clarified in the final report that portions of the data these school districts collect are used for CRDC reporting purposes. For example, some of the data elements are the same ones that districts use to calculate aggregate incident counts, which are required by the CRDC. We do not recommend that Education collect such detailed data or perform such analyses. Regarding Education’s concern about “GASB statistical principles” and case selection, all GAO performance audits are subject to Generally Accepted Government Auditing Standards (GAGAS); in contrast, GASB’s Generally Accepted Accounting Principles apply to financial audits of public entities. The applicable methodological guidance we followed -- Selecting a Sample of Nongeneralizable Cases for Review in GAO Engagements -- is designed to ensure that GAO policies on evidence and GAGAS are met, and conforms to the generally accepted principles and practices of the appropriate disciplines. When providing illustrative examples, it is neither necessary nor appropriate to use statistical methods to analyze and interpret evidence. Finally, in its comments, Education stated that it is critical that we emphasize that the CRDC is an aggregate of self-collected and self- reported data from school districts and that the district superintendent or an authorized designee certifies that the data they submit are “true and correct.” We agree, and acknowledged this in several places in both the draft and final reports. At the same time, we believe that self-certified data does not absolve Education of its responsibility to ensure the quality of the data it collects and publicly reports – especially given the CRDC’s longstanding role in Education’s overall enforcement of various federal civil rights laws prohibiting discrimination on the basis of race, color, national origin, sex, and disability. Self-reported data by nature are subject to error, making the need for effective quality control measures before, during, and after collection a necessity. We are sending copies to the appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report examines (1) the effectiveness of CRDC data quality control procedures for its restraint and seclusion data, (2) how selected districts interpret the CRDC definitions of restraint and seclusion and (3) how selected districts use data on restraint and seclusion and encourage staff to report incidents. Below are the details of our analysis to determine the extent to which Education ensures the quality of restraint and seclusion data reported by school districts, and of our interviews with officials in selected districts about how they apply Civil Rights Data Collection (CRDC) definitions of restraint and seclusion and use restraint and seclusion data. To inform all of our objectives, we interviewed federal agency officials, representatives from several nonfederal advocacy organizations that represent parents and families, individuals with disabilities, and other stakeholders, such as representatives of professional associations. We also reviewed agency documentation, relevant federal laws, regulations and policies, and selected state laws. Analysis of National Restraint and Seclusion Data To determine the extent to which Education ensures the quality of restraint and seclusion data reported by school districts, we analyzed Education’s Civil Rights Data Collection (CRDC) for school year 2015-16. Specifically, we analyzed the CRDC to determine the extent to which districts reported zero incidents of restraint and seclusion, to identify outliers (districts that reported a high or low incidence of restraint and seclusion), and to identify illogical data. CRDC is a biennial survey that is mandatory for nearly every public school and school district in the United States and is conducted by Education’s Office for Civil Rights. The CRDC collects data on the nation’s public schools (pre-K through 12th grade) that includes the use of restraint and seclusion, student demographics and enrollment numbers, educational and course offerings, and disciplinary actions. In school years 2013-14 and 2015-16, the CRDC collected data from nearly every public school in the nation (approximately 96,000 schools in 17,000 school districts in school year 2015-16). CRDC data are self-reported by districts and schools, and consequently there is potential for misreporting of information. After reviewing their CRDC data, school districts can submit revised data to Education. The public-use data file of the CRDC for school year 2015-16 was the primary source of data for our analyses and the most recent data available at the time. We also used restraint and seclusion data from school year 2013- 14 primarily to analyze how use of restraint and seclusion may have changed between the two time periods. The CRDC collected data on (1) mechanical restraint, (2) physical restraint, and (3) seclusion. Using these data, we performed the following analyses to determine potential inaccuracies or underreporting in the CRDC. Analysis of Extent of Districts Reporting Zeros To examine the extent to which school districts reported zeros, we calculated the percentage of districts and schools reporting zeros for restraint (both mechanical and physical) and for seclusion. We performed this calculation for both districts and schools nationally and by state, district size, and school type (e.g., charter, traditional, and special education schools). Although Education has a business rule that targets very large districts that report zero incidents of restraint or seclusion, we calculated the number of all districts and schools that reported zeros to understand the prevalence of zeros in the reported data. Analysis of Relatively Low Rates of Restraint and Seclusion Incidents To test for potential underreporting, we first limited our analysis to the restraint and seclusion data reported by the 30 largest school districts in the nation (districts with over 100,000 students enrolled). Because of these districts’ size, we reasoned that they would be more likely to have incidents of restraint and seclusion to report. Our analysis found that 20 of the 30 largest school districts reported incidents, and thus we focused our analysis of underreporting on the 20 largest districts that reported incidents. For each of the 20 districts, we calculated the percentage of schools that reported incidents. To compare the 20 largest districts that reported incidents with all 5,252 districts that reported incidents, we calculated the rates of restraint and seclusion per enrolled student and calculated percentile ranges. (See table 6.) We determined that nine of the 20 districts had incidents of physical restraint per enrolled student that were below the 5th percentile of all districts reporting incidents of physical restraint. Analysis of Relatively High Rates of Restraint and Seclusion Incidents To identify school districts with relatively high rates of restraint and seclusion, we examined districts that reported having more incidents than students enrolled. This analysis potentially indicates that some students may have been restrained or secluded multiple times. To illustrate, if a school district reported that it had 24 students enrolled, and also reported that it had 100 incidents of restraint, these reported data would indicate that the reporting was erroneous or that some students were restrained multiple times. Based on this analysis, we then calculated the average number of incidents (of restraint and seclusion) per student affected. Analysis of Extent of Illogical Data To test for illogical data, we analyzed the restricted-use restraint and seclusion data file for schools that reported more students affected than incidents. To illustrate, if a district reported that a school had restrained 80 students, and also reported that the school had 40 incidents of restraint, these reported data are illogical. Education has a business rule to detect illogical data at the school level, but the rule applies only to schools with more than 100 incidents. For our analysis, we looked for all schools with illogical data to determine the prevalence. School District Interviews on Interpreting CRDC Definitions of Restraint and Seclusion To determine how selected school districts interpret the CRDC definitions of restraint and seclusion, we selected 11 schools and nine school districts in three states to serve as illustrative examples. In total, we interviewed about 50 school officials. Information we collected from our 11 selected schools and nine districts cannot be generalized to all districts and schools nationwide. We selected states, districts, and schools to obtain a range of perspectives on federal reporting of restraint and seclusion data. Our selection also accounted for other criteria, such as selecting states that had laws requiring reporting; high or low rates of reporting zeros among districts; relatively high or low rates of restraint or seclusion per capita; grade levels served (e.g., K-6 or 9-12); school type (e.g., traditional or charter); and significant changes—increase or decrease—in incidents across reporting periods. We also selected districts that had reported incidents. As a result, we selected nine school districts to visit: two in Kentucky, three in Washington, three in Wisconsin, and a charter district in Wisconsin (see table 7). To determine how district and school officials, such as assistant superintendents, program managers, department directors, principals, and teachers, were interpreting the CRDC definitions of restraint and seclusion, we made the following statements and asked the following questions in our interviews. 1. We are going to talk to you about the definitions of restraint and seclusion that appear in the CRDC. We have heard that these definitions are not always clear to educators, so we want to get your feedback. Mechanical Restraint: the use of any device or equipment to restrict a student’s freedom of movement. Do you think this definition is clear or does it leave room for ambiguity? Physical Restraint: a personal restriction that immobilizes or reduces the ability of a student to move his or her torso, arms, legs, or head freely. The term does not include a physical escort. Physical escort means a temporary touching or holding of the hand, wrist, arm, shoulder or back for the purpose of inducing a student who is acting out to walk to a safe location Do you think this definition is clear or does it leave room for ambiguity? How do you differentiate between physical escort and physical restraint? Does breaking up a fight constitute a restraint? Seclusion: the involuntary confinement of a student alone in a room or area from which the student is physically prevented from leaving. It does not include a timeout, which is a behavior management technique that is part of an approved program, involves the monitored separation of the student in a non-locked setting, and is implemented for the purpose of calming. Do you think this definition is clear or does it leave room for ambiguity? How do you differentiate between timeout and seclusion? What does physically prevented (from leaving) mean? In what types of physical spaces can seclusion occur? Does your district have dedicated spaces for seclusion rooms? Can you describe where they are generally located, e.g., which types of schools or classrooms? 2. How do staff determine when an incident needs to be recorded as a restraint? 3. How do staff determine when an incident needs to be recorded as a seclusion? We conducted this performance audit from November 2018 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Sherri Doughty (Assistant Director), Lara Laufer (Analyst-in-Charge), Morgan Jones, and Kristin Petroff, made key contributions to this report. Also contributing were James Bennett, Deborah Bland, Tonnye Conner-White, Holly Dye, Gretta Goodwin, Sheila R. McCoy, Jean McSween, John Mingus, James Rebbe, and Manuel Valverde.", "summary": "Every 2 years, Education requires nearly all school districts to report incidents of restraint and seclusion. Generally, restraint is restricting a student's ability to move, and seclusion is confining them alone in a space they cannot leave. The House Committee on Appropriations' explanatory statement accompanying the Consolidated Appropriations Act of 2018 included a provision for GAO to evaluate the CRDC's restraint and seclusion data. This report examines (1) the effectiveness of CRDC data quality control procedures, (2) selected districts' interpretation of CRDC's restraint and seclusion definitions, and (3) selected districts' use of data. GAO analyzed CRDC's quality control processes for school year 2015-16, and interviewed officials from seven stakeholder groups and over 50 school and district officials in three states. GAO selected states, districts, and schools to obtain a range of perspectives on using restraint and seclusion data and interpreting CRDC definitions of restraint and seclusion. Selection criteria included changes in reported incidents year to year and laws requiring districts to report incidents to states. The Department of Education's (Education) quality control processes for data it collects from public school districts on incidents of restraint and seclusion are largely ineffective or do not exist, according to GAO's analysis of school year 2015-16 federal restraint and seclusion data—the most recent available. Specifically, Education's data quality control processes were insufficient to detect problematic data in its Civil Rights Data Collection (CRDC)—data Education uses in its efforts to enforce federal civil rights laws (see figure). For example, one rule Education used to check the quality of data submitted only applied to very large school districts, although GAO and Education's own analyses found erroneous reporting in districts of all sizes. Education also had no rules that flagged outliers that might warrant further exploration, such as districts reporting relatively low or high rates of restraint or seclusion. GAO tested for these outliers and found patterns in some school districts of relatively low and high rates of restraint or seclusion. Absent more effective rules to improve data quality, determining the frequency and prevalence of restraint and seclusion will remain difficult. Further, Education will continue to lack information that could help it enforce various federal civil rights laws prohibiting discrimination. Officials in the nine school districts GAO visited lacked a common understanding of the CRDC's restraint and seclusion definitions. Similarly, officials GAO interviewed in all three state educational agencies (Kentucky, Washington, and Wisconsin) and all seven stakeholder groups expressed similar concerns about the clarity of these definitions. For example, officials inconsistently interpreted the word alone in the definition of seclusion and, therefore, on whether to count an incident if a teacher was in the room. Absent clearer definitions, Education will continue to lack quality information on restraint and seclusion in public schools. Officials in school districts GAO visited identified several benefits to collecting these data, including identifying patterns in student behavior and developing interventions that can reduce the need for restraint and seclusion. Officials also said that analyzing their data helped them identify needs for additional staff training and student support services. GAO made six recommendations, including that Education expand its CRDC business rules to cover all districts, develop additional quality controls to address misreporting, address factors underlying misreporting, and refine and clarify its definitions. Education agreed with these recommendations.", "document_type": "gao"}
{"report": "In January 2018, OMB released (M-18-04) Monitoring and Evaluation Guidelines for Federal Departments and Agencies that Administer United States Foreign Assistance (the “Guidelines”) in response to the 2016 FATAA legislation. (See appendix III for additional information on the requirements in the legislation). The Guidelines provide direction to federal departments and agencies that administer foreign assistance on monitoring the use of resources, evaluating the outcomes and impacts of the foreign assistance projects and programs, and applying the findings and conclusions of such evaluations to proposed project and program design. The goals of the Guidelines are to set forth key principles to guide each agency and to specify requirements, where appropriate, that agencies must cover in their own policies on M&E of foreign assistance. The Guidelines define monitoring and evaluation as follows: Monitoring is the ongoing and systematic tracking of data and information relevant to policies, strategies, programs, projects, and/or activities and is used to determine whether desired results are occurring as expected during program, project, or activity implementation. Monitoring often relies on indicators, quantifiable measures of a characteristic or condition of people, institutions, systems, or processes that may change over time. Evaluation is the systematic collection and analysis of information about the characteristics and outcomes of the program, including projects conducted under such program, as a basis for making judgments and evaluations regarding the program; improving program effectiveness; and informing decisions about current and future programming. Table 1 lists OMB’s M&E requirements and key excerpts of the descriptions as noted in the OMB M-18-04. In 2016, we reported on leading practices for foreign assistance program M&E. We identified 28 leading practices—14 for monitoring and 14 for evaluation. Table 2 lists and defines these monitoring practices. Table 3 lists the evaluation practice and corresponding definition. Based on our review, the Guidelines incorporate most of GAO’s leading practices for monitoring and evaluation. However, they do not incorporate practices on developing monitoring plans that are based on risks, ensuring that staff are appropriately qualified to conduct monitoring, establish procedures to close out programs, developing staff skills for evaluation, and following up on evaluation recommendations. OMB indicated that it intended the Guidelines to focus on elements required by the FATAA legislation. Nevertheless, incorporating these leading practices in the Guidelines can help ensure that all agencies address impediments, effectively manage foreign assistance, and meet their assistance goals. Based on our review, OMB incorporates 11 of 14 GAO’s leading practices. Figure 1 shows our assessment of the Guidelines with regard to monitoring foreign assistance. The OMB Guidelines do not incorporate practices on developing monitoring plans that are based on risks, ensuring that staff are appropriately qualified to conduct monitoring, and establishing close-out procedures for projects and programs. Developing monitoring plans based on an assessment of risk. The Guidelines do not incorporate GAO’s leading practice of developing monitoring plans based on as assessment of risks related to achieving the defined objectives. Identifying and assessing risks can help agencies determine if impediments exist that they might need to mitigate in order to manage their foreign assistance more effectively. Additionally, determining which activities warrant greater oversight and which require less can also help agencies ensure the appropriate allocation of foreign assistance. Ensuring Staff qualifications for monitoring. The Guidelines do not incorporate GAO’s leading practice for agencies to ensure that staff members responsible for monitoring programs or projects have the relevant knowledge, skills, and training. By having qualified staff for monitoring programs or projects, agencies can help ensure they meet their foreign assistance goals. By hiring qualified staff and providing them the right training, tools, structure, incentives, and responsibilities, agencies can make operational success possible. Establishing close out procedures for projects and programs. The Guidelines do not incorporate GAO’s leading practice for agencies to establish program closeout procedures for all required work and administrative actions completed by the implementing partner. By establishing such procedures, agencies can help ensure their foreign assistance is less susceptible to fraud, waste, and mismanagement; addresses increases to potential costs in fees for maintaining foreign assistance; and increases their ability to redirect foreign assistance to other projects. Based on our review, OMB incorporates 12 of 14 GAO’s leading practices. Figure 2 shows our assessment of the Guidelines with regard to evaluating foreign assistance. The OMB Guidelines do not incorporate some practices such as developing staff skills for evaluation and following up on evaluation recommendations. Developing staff skills regarding evaluating. The Guidelines do not incorporate GAO’s leading practice for agencies to establish requirements that the staff responsible for overseeing and using evaluations should continually undertake the relevant education, training, or supervised practice needed to learn new concepts, techniques, and skills. By having their staff continually undertake such education, training, or supervised practice, agencies can benefit more fully from program evaluations. Following up on recommendations. The Guidelines do not incorporate GAO’s leading practice for agencies to determine whether management or programs have accepted the recommendations made in evaluation reports and taken the actions needed to address them. By developing mechanisms to track recommendations, agencies can better address inefficient, mismanaged, or costly programs or projects. The FATAA requires the President to set forth guidelines “according to best practices of monitoring and evaluation” but does not define these best practices. Specifically, FATAA states, “the President shall set forth guidelines, according to best practices of monitoring and evaluation studies and analyses, for the establishment of measurable goals, performance metrics, and monitoring and evaluation plans that agencies can apply with reasonable consistency to covered United States foreign assistance.” OMB staff told us that the Guidelines were intended to focus on elements required by the FATAA legislation but noted that agencies are free to add additional requirements to their own M&E policies. However, we have previously reported that while some of these agencies have incorporated these leading practices, others have not. Furthermore, agencies that have incorporated these practices would not necessarily continue to include them if they are not required in the Guidelines. Regarding leading practices, officials noted that while these practices are important, there is no singular established standard for best monitoring practices. Nevertheless, both OMB’s circulars and recent legislation note the importance of leading practices for M&E. For example, Circular A-123 notes that management should identify internal and external risks that may prevent the organization from meeting its objectives. Additionally, the Foundations for Evidence-Based Policymaking Act of 2018 requires OPM, in consultation with the OMB, to identify skills and competencies needed for program evaluation, establish a new occupational series or update an existing one for program evaluation, and establish a new career path for program evaluation. Based on our review, most agencies incorporated all of OMB’s Guidelines for monitoring in their policies. However, DOD did not include the requirement to establish roles and responsibilities among agencies that participate in funding transfers or ensure that verifiable, reliable, and timely information is collected and available to monitoring personnel. We also found that agencies incorporate most of OMB’s Guideline requirements for evaluation in their policies, but some did not include the requirement to conduct impact evaluation on all pilot programs. Without incorporating these Guideline requirements, agencies risk losing accountability over their funding and monitoring and evaluating activities. They also risk replicating programs without fully understanding their effectiveness. We also found that all of the agencies we reviewed have taken initial steps to implement their M&E policies. Based on our review of agency monitoring policies, all the agencies except DOD incorporated relevant Guideline requirements. All six agencies we reviewed incorporated the requirement to establish monitoring policies that apply to their major foreign assistance programs. For example, State, USAID, and MCC have agency-wide policies for foreign assistance M&E. USDA and HHS have policies relevant to their major foreign assistance programs—for USDA, the Foreign Agriculture Service’s food aid programs, and for HHS, the President’s Emergency Plan for AIDS Relief (PEPFAR). All of the agencies with relevant monitoring policies—DOD, HHS, MCC, State, USAID, and USDA— incorporate the requirement to develop, collect, analyze, and report data on performance indicators. These policies help ensure the measurement of project implementation and progress, and promote the timely analysis and reporting of results that could identify any needed corrections. DOD did not incorporate Guideline requirements to establish agencies’ roles and responsibilities and ensure verifiable data for monitoring activities. Establishing agencies’ roles and responsibilities when funds are transferred. DOD did not include the Guideline requirement for agencies to establish roles and responsibilities in funding transfers. Without defined roles and responsibilities, agencies risk losing accountability over funding and monitoring activities. In addition, agencies could miss opportunities to collaborate and leverage interagency efforts to facilitate decision-making and address barriers across agency boundaries. Ensuring verifiable, reliable, and timely data. DOD did not include the Guideline requirement for agencies to ensure they collect and provide verifiable, reliable, and timely data to monitoring personnel. Without ensuring that such data are available to monitoring personnel, agencies risk employing inappropriate methods, continuing ineffective programs or projects, and making uninformed decisions. DOD officials told us these practices are currently not required because they are still in the process of fully aligning their policy with the Guidelines. Officials explained that working on prioritizing and directing resources towards M&E efforts has been a challenge. Officials noted they expect to update the policy to include these requirements in the future, but they have no specific timelines in place. The agencies we reviewed incorporated nearly all relevant Guideline requirements on evaluation. Three of the six agencies—DOD, HHS and USDA—did not include a requirement to conduct impact evaluations on all pilot programs or projects. Figure 4 shows our assessment of agencies’ evaluation policies against the Guidelines. All the agencies we reviewed have established project-specific evaluation plans. For example, HHS implements PEPFAR’s evaluation plan which indicates specific requirements for describing the evaluation component, strategy, or intervention, the reason for the evaluation, the type of evaluation, the key evaluation questions, the data sources, the methods by question, and the dissemination and utilization plan. All the agencies we reviewed also had policies on distributing their evaluation reports internally and publicly reporting them. For example, State and USAID have a web-based, customized Evaluation Registry system that they jointly maintain for bureaus and independent offices to record and track planned, ongoing, and completed evaluations. Conduct impact evaluations for pilot programs or projects. DOD, HHS, and USDA did not include the Guideline requirement for agencies to conduct impact evaluations for pilot programs or to conduct only a performance evaluation and to provide a justification for not conducting an impact evaluation. Without a requirement to conduct impact evaluations of pilot programs, agencies risk duplicating or scaling up programs without fully understanding the factors that could lead to their success or failure. DOD. DOD officials told us they do not require this practice because they are still in the process of fully aligning their policy with the Guidelines. According to DOD, it has determined that impact evaluations are impractical and inappropriate for the planned evaluations; instead, it plans to conduct only performance evaluations and provide justifications for not conducting impact evaluations, as required by the OMB Guidelines. DOD plans to address the evaluation methodology of pilot programs in future updates, according to officials. However, DOD has no specific timelines in place for these updates. HHS. PEPFAR’s M&E documents indicate that PEPFAR teams are encouraged but not required to evaluate all current pilot programs to see which should be taken to scale for specific populations. Officials from HHS and the Office of the U.S. Global Aids Coordinator noted that they conduct their own evaluation of pilot programs and use routine program data to inform scaling of programs. However, PEPFAR policies do not specifically require that such evaluations be like the impact evaluations described in the Guidelines. USDA. FAS’s M&E documents indicate that when selecting projects to undergo impact evaluation the agency will consider pilot projects. USDA officials told us they have no requirement to conduct impact evaluations on all pilot projects because impact evaluations may be cost prohibitive and project lifecycles are short (i.e., 3 to 5 years). Officials further noted that implementing partners can conduct an impact evaluation on pilot programs, but are not required to do so. Although the Guideline requirement indicates that agencies can forgo impact evaluations, they must provide a justification in their M&E policy. USDA officials have not provided such a justification provided in their M&E policy. Establish agencies’ roles and responsibilities for evaluation activities when funds are transferred. DOD did not include the Guideline requirement for agencies to define roles and responsibilities when there are funding transfers between or among U.S. government agencies to ensure accountability for evaluation activities. Without defined roles and responsibilities, agencies risk losing accountability over funding and evaluation activities. In addition, they could miss opportunities to collaborate and leverage interagency efforts to facilitate decision-making and address barriers across agency boundaries. Evaluate all programs at least once whose dollar value equals or exceeds that of a median sized program within the agency. DOD did not include the Guideline requirement for agencies to evaluate all programs, at least once during their existence, whose dollar value equals or exceeds that of a median sized program in the agency. Without a mechanism to evaluate all these types of programs, agencies risk continuing inefficient, mismanaged, or costly projects. DOD officials told us they do not currently require these practices because they are still in the process of fully aligning their policy with the Guidelines. They noted that they expect to update the policy to include these requirements, but they have no specific timelines in place. Since the six agencies we reviewed recently updated their M&E policies to align with the OMB Guidelines, many existing assistance projects and programs may not be governed by these requirements. Nonetheless, the agencies we reviewed have taken initial steps to help ensure implementation of agency M&E policies. In interviews, agencies provided us with the following examples of such steps. State. State developed a guidance document and tool-kit to operationalize and oversee its M&E policy to ensure it implements the Guidelines. According to State officials, they provide classroom training on the M&E policy and are piloting a revised online and classroom evaluation courses for staff. Officials also noted that they have dedicated staff to assist bureaus in implementing the Guidelines, among other agency policies. USAID. USAID has an approval process to ensure key deliverables include Activity plans that meet Guideline requirements. Additionally, USAID’s policy requirements indicate that each mission program office must identify a point of contact for monitoring and evaluation to ensure that USAID and its partners are complying with the agencies policies and foreign assistance M&E guidelines. MCC. MCC also has an approval process through their Department of Policy and Evaluation to ensure implementation of the Guidelines. As part of the process, the MCC Board of Directors or the appropriate partner country must approve initial M&E plans. HHS. Within HHS, the Centers for Disease Control and Prevention (CDC) are responsible for implementing the monitoring and evaluation guidance for their PEPFAR programs. CDC officials told us that they have existing mechanisms and supervisory structures in place to ensure that the Guidelines’ requirements are met in PEPFAR programs. USDA. USDA officials told us that the current M&E policy applies only to food assistance programs within FAS and not for other USDA programs. Officials explained they are trying to develop a structure that allows FAS to ensure all USDA components are implementing the OMB Guidelines. DOD. DOD developed guidance for fiscal year 2020 on implementing its M&E policy. DOD officials we spoke to noted they are working on identifying resources, skills, and capabilities to fully implement DOD’s M&E policy. OMB’s Guidelines set forth key principles to guide agencies and to specify requirements, where appropriate, which they must cover in their own policies on M&E of foreign assistance. However, they do not include key leading practices for M&E that GAO identified for ensuring agencies meet their foreign assistance goals and objectives. While OMB allows agencies discretion to include these or other best practices, it is unknown if the agencies will do so. By ensuring that OMB’s government-wide Guidelines include these best practices, agencies can help address impediments, effectively manage foreign assistance, and meet their goals. Although all agencies we reviewed developed or updated their M&E policies to align with the Guidelines, not all of them include important requirements. DOD, HHS, and USDA did not include the requirement for agencies to conduct impact evaluations for pilot programs or to conduct performance evaluations and provide a justification for not doing an impact evaluation. Without a requirement to conduct impact evaluations of pilot programs, agencies risk duplicating or scaling up programs without fully understanding the causes that could lead to their success or failure. We are making the following seven recommendations, including one to OMB, four to DOD, one to State, and one to USDA. The Director of the Office of Management of Budget should update the Guidelines to include GAO’s leading practices of developing monitoring plans that are based on risks, ensuring that monitoring staff have appropriate qualifications, establishing procedures to close-out programs, developing staff skills regarding evaluations, and establishing mechanisms for following up on evaluation recommendations. (Recommendation 1) The Secretary of Defense should update the Department’s monitoring and evaluation policies to define roles and responsibilities among agencies that participate in interagency funding transfers. (Recommendation 2) The Secretary of Defense should update the Department’s monitoring and evaluation policies to ensure verifiable, reliable, and timely data are available to monitoring personnel. (Recommendation 3) The Secretary of Defense should update the Department’s monitoring and evaluation policies to ensure that it evaluates all programs, at least once in their lifetimes, whose dollar value equals or exceeds that of the median program in the agency. (Recommendation 4) The Secretary of Defense should update the Department’s monitoring and evaluation policies to require the agency to conduct impact evaluations on all pilot programs before replicating or expanding, or conduct performance evaluations for those programs and provide a justification for not conducting an impact evaluation. (Recommendation 5) The Department of State’s U.S. Global AIDS Coordinator, in collaboration with HHS and other implementing agencies, should update the PEPFAR monitoring and evaluation policies to require these agencies to conduct impact evaluations on all pilot programs before replicating or expanding, or conduct performance evaluations for those programs and provide a justification for not conducting an impact evaluation. (Recommendation 6) The Secretary of Agriculture, in collaboration with the Foreign Agriculture Service, should update their monitoring and evaluation policies to require USDA to conduct impact evaluations on all pilot programs before replicating or expanding, or conduct performance evaluations for those programs and provide a justification for not conducting an impact evaluation. (Recommendation 7) We provided a draft of this product to the DOD, HHS, MCC, OMB, State, USDA, and USAID for comment. OMB commented on the draft report in an email from the staff responsible for economic policy, federal financial management, and international affairs. In the email, OMB disagreed with the recommendation to revise the Guidelines. It emphasized that an interagency group had developed the Guidelines and had consulted a number of expert sources on monitoring and evaluation policies and practices, including GAO’s leading practices. OMB also developed the guidelines to achieve the objectives contained in the Foreign Aid Transparency and Accountability Act of 2016 within the context of other existing OMB guidance. OMB suggested that it would be more effective to remind agencies that, in addition to the Guidelines specified in M-18-04, they should follow all guidance OMB had issued affecting monitoring and evaluation activities. This guidance includes policies for closeout procedures in the Uniform Guidance, for the Enterprise Risk Management and Internal Control in A-123, and for the Foundations for Evidence- Based Policymaking Act on using evaluation information and monitoring and evaluation staff skills and qualifications. We acknowledge that relevant monitoring and evaluation guidance is available to agencies in other forms beyond the Guidelines. However, we believe it is important for OMB to incorporate this guidance into its Guidelines, if only by reference, in order to emphasize the importance of these practices in the context of monitoring and evaluation of foreign assistance. This step would help ensure that OMB had integrated this guidance into the management of foreign assistance programs as appropriate. DOD concurred with our recommendations and indicated that it would address many of them in the next iteration of its M&E policy for security assistance (see appendix IV for written comments). DOD noted that two of our recommendations had limited applicability to DOD for security assistance, but described how it would implement them. First, DOD stated that it has not used its authority to transfer funds for security cooperation assistance to other departments and agencies. However, DOD indicated it would implement our recommendation to define roles and responsibilities among agencies that participate in interagency funding transfers, should such transfers become necessary. Second, DOD stated that conducting impact evaluations was not a feasible in the context of security assistance. Instead, DOD plans to conduct only performance evaluations, but it would provide justifications for not conducting impact evaluations, as required by the Guidelines. By documenting these approaches in its M&E policies, DOD would help ensure that those departments conducting M&E for DOD security assistance initiatives implement them as required. State agreed with the intent of the recommendation (see appendix V for written comments). State explained that impact evaluations are often not feasible in the context of assistance provided under PEPFAR and described its alternative approach to evaluating new initiatives. State indicated it would update appropriate PEPFAR policies to clarify when agencies should conduct impact and/or performance evaluations. These clarifications will reflect how State evaluates PEPFAR programs in practice in accordance with OMB guidance and legislation, according to State. USAID provided written comments (see appendix VI). HHS and USDA provided technical comments, which we incorporated as appropriate. MCC did not provide comments. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget; Secretaries of Agriculture, Defense, Health and Human Services, and State; Administrator of the U.S. Agency for International Development; and the Executive Officer of the Millennium Challenge Corporation and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3149 or GootnickD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. This report examines the extent to which (1) the Office of Management and Budget’s (OMB) monitoring and evaluation (M&E) Guidelines incorporate GAO leading practices and (2) agencies incorporate the OMB Guidelines in their M&E policies and plans. To address objective one, we examined the OMB Guidelines against GAO’s 28 leading practices—14 for monitoring and 14 for evaluation— identified in GAO-16-861R. In 2016, GAO developed the 28 leading practices. In 2019, we provide specific definitions for each of the practices noted. We made slight modifications to the language to align with the definitions provided. For monitoring, we developed this list of leading practices based on our review of the GPRA Modernization Act of 2010; OMB’s Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards; GAO’s Standards for Internal Control in the Federal Government (Greenbook); and others. The list of leading practices for monitoring includes developing monitoring plans; collecting, reviewing, and analyzing monitoring data; and establishing roles and responsibilities of personnel responsible for monitoring. For evaluation, we developed a list of leading practices based on the American Evaluation Association’s (AEA) 2016 An Evaluation Roadmap for a More Effective Government (AEA Roadmap) and Preface to Evaluators’ Ethical Guiding Principles. The list of leading practices for evaluation include developing evaluation plans; ensuring evaluator independence; developing staff skills regarding evaluation and use of evidence; and establishing roles and responsibilities of personnel responsible for evaluation. To perform these analyses, two analysts assessed if the Guidelines incorporated specific GAO leading practices. The analysts worked iteratively, comparing notes and reconciling differences at each stage of the analysis. In addition, GAO staff independent of the two analysts reviewed the final analysis, and made modifications as appropriate. We also interviewed relevant OMB officials in Washington D.C. involved in developing the memorandum and inquired about specific requirements and plans to ensure the implementation of these Guidelines. To address our second objective, we examined U.S. agency M&E policies against the requirements noted in the OMB Guidelines. We identified the six major agencies administering the most foreign assistance funds. The six agencies are the U.S. Agency for International Development (USAID), the Department of State (State), the Millennium Challenge Corporation (MCC), the Department of Health and Human Services (HHS), the U.S. Department of Agriculture (USDA) and the Department of Defense (DOD). We asked these agencies to identify or provide all relevant policies and guidance relating to foreign assistance M&E, including, where appropriate, standard operating procedures or other guidance. For USDA, we reviewed the Foreign Agricultural Service’s food assistance; for HHS, the President’s Emergency Program for AIDS Relief; and for DOD, security cooperation programs. To perform these analyses, two analysts assessed agency M&E policy documents against the requirements in the OMB Guidelines. We identified requirements as phrases that included the following language “required,” “must,” “mandatory,” or “should.” The analysts worked iteratively, comparing notes and reconciling differences at each stage of the analysis. In addition, other GAO staff independent of the two analysts reviewed the final analysis, and made modifications as appropriate. We also interviewed relevant OMB staff and agency officials in Washington D.C. involved in developing and implementing the M&E policies and inquired about specific requirements, and plans to ensure their M&E policies are implemented. We conducted this performance audit from July 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: The Office of Management and Budget Monitoring and Evaluation Guidelines (OMB Memorandum M-18-04) In January 2018, the Office of Management and Budget (OMB) released (M-18-04) Monitoring and Evaluation Guidelines for Federal Departments and Agencies that Administer United States Foreign Assistance (the “Guidelines”) in response to the Foreign Aid Transparency and Accountability Act of 2016 (FATAA). Table 4 shows the complete description of the requirements noted in the Guidelines. The Foreign Aid Transparency and Accountability Act of 2016 (FATAA) has required objectives on monitoring and evaluation for the Office of Management and Budget (OMB) to include in the Guidelines. We compared the 13 required objectives for the Guidelines set forth in the FATAA legislation with those in the OMB Guidelines. We found that all of the monitoring and evaluation requirements set forth in the legislation are included in the OMB Guidelines. Table 5 shows the FATAA legislation requirements, OMB Guidelines, and our assessment of the alignment between the legislation and OMB’s Guidelines. In addition to the individual named above, James B. Michels (Assistant Director), Farahnaaz Khakoo-Mausel (Analyst-in-Charge), Paulina Maqueda-Escamilla, Mark Dowling, Martin De Alteriis, Benjamin Licht, John Hussey, Neil Doherty, Aldo Salerno, Carolina Morgan and Michael Simon made key contributions to this report. Government Auditing Standards 2018 Revision (Supersedes GAO-12-331G. GAO-18-568G. Washington, D.C.: July 17, 2018. Foreign Assistance: Agencies Can Improve the Quality and Dissemination of Program Evaluations. GAO-17-316. Washington, D.C.: March 3, 2017. Foreign Assistance: Selected Agencies’ Monitoring and Evaluation Policies Generally Address Leading Practices. GAO-16-861R. Washington, D.C.: September 27, 2016. Program Evaluation: Some Agencies Reported that Networking, Hiring, and Involving Program Staff Help Build Capacity. GAO-15-25. Washington, D.C.: November 13, 2014. Government Efficiency and Effectiveness: Inconsistent Definitions and Information Limit the Usefulness of Federal Program Inventories. GAO-15-83. Washington, D.C.: October 31, 2014. Standards for Internal Control in the Federal Government. GAO-14-704G. Washington, D.C.: September 10, 2014. State Department: Implementation of Grants Policies Needs Better Oversight. GAO-14-635. Washington, D.C.: July 21, 2014. Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making. GAO-13-570. Washington, D.C.: June 26, 2013. President’s Emergency Plan for AIDS Relief: Agencies Can Enhance Evaluation Quality, Planning, and Dissemination. GAO-12-673. Washington, D.C.: May 31, 2012. Grants Management: Action Needed to Improve the Timeliness of Grant Closeouts by Federal Agencies. GAO-12-360. Washington, D.C.: April 16, 2012. Designing Evaluations: 2012 Revision. GAO-12-208G. Washington, D.C.: January 31, 2012. International School Feeding: USDA’s Oversight of the McGovern-Dole Food for Education Program Needs Improvement. GAO-11-544. Washington, D.C.: May 19, 2011. Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research. GAO-11-176. Washington, D.C.: January 14, 2011. Managing for Results: Enhancing Agency Use of Performance Information for Management Decision Making. GAO-05-927. Washington, D.C.: September 9, 2005. Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity. GAO-03-454. Washington, D.C.: May 2, 2003. Managing For Results: Federal Managers’ Views Show Need for Ensuring Top Leadership Skills. GAO-01-127. Washington, D.C.: October 20, 2000. Performance Plans: Selected Approaches for Verification and Validation of Agency Performance Information. GAO/GGD-99-139. Washington, D.C.: July 30, 1999. Agency Performance Plans: Examples of Practices That Can Improve Usefulness to Decisionmakers. GAO/GGD/AIMD-99-69. Washington, D.C.: February 26, 1999. Executive Guide: Effectively Implementing the Government Performance and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1, 1996.", "summary": "The Trump Administration requested $28.5 billion in foreign assistance in fiscal year 2019, to be administered by at least 22 federal agencies. Almost 95 percent of this assistance is administered by six agencies—the Departments of Agriculture (USDA), Defense (DOD), State (State), Health and Human Services (HHS), the Millennium Challenge Corporation (MCC), and the U.S. Agency for International Development (USAID). FATAA required the President to set forth guidelines for M&E of U.S. foreign assistance. In January 2018, OMB issued the required guidelines for federal agencies. FATAA also contained a provision for GAO to analyze the guidelines established by OMB; and assess the implementation of the guidelines by the agencies. In this report, GAO examined the extent to which (1) OMB's M&E Guidelines incorporate GAO leading practices, and (2) agencies incorporate the OMB Guidelines in their M&E policies and plans. GAO assessed the OMB Guidelines against GAO's 28 leading practices identified in GAO-16-861R . GAO also assessed the six agencies' foreign assistance M&E policies against the Guidelines and interviewed OMB and relevant agency officials in Washington, DC. The Office of Management and Budget's (OMB) foreign assistance Guidelines incorporate most of GAO's leading practices for monitoring and evaluation (M&E), but gaps exist (see figure). Monitoring : The Guidelines define monitoring as the continuous tracking of program or project data to determine whether desired results are as expected during implementation. The Guidelines do not require GAO's leading practices on risk assessments, staff qualifications, and program close-out procedures. Evaluation : The Guidelines define evaluation as the systematic collection and analysis of program or project outcomes for making judgments and informing decisions. They do not require GAO's leading practices on developing staff skills and following up on recommendations. OMB officials indicated the Guidelines are focused on elements required in the Foreign Aid Transparency and Accountability Act of 2016 (FATAA), but noted that agencies can add additional requirements to their own M&E policies. FATAA requires the President to set forth guidelines “according to best practices of monitoring and evaluation.” OMB staff acknowledged that GAO's leading practices are important, but stated that there is no singular established standard for best monitoring practices. Nevertheless, all of GAO's leading practices can help agencies address impediments, effectively manage foreign assistance, and meet their goals. When assessing agencies' M&E policies against OMB Guidelines, GAO found that agencies incorporated most of the requirements. However, for monitoring, one of the six agencies GAO reviewed—DOD—did not include the requirements to establish agencies' roles and responsibilities and ensure verifiable data for monitoring activities. For evaluation, agencies required most Guideline requirements, but not all. For example, DOD, HHS, and USDA did not require conducting impact evaluations for pilot programs or projects. Without a clear requirement to do such evaluations, agencies risk duplicating or scaling up programs without fully understanding the factors that could lead to their success or failure. Agencies GAO reviewed have plans or mechanisms in place to oversee the implementation of their M&E policies. For example, State developed a guidance document to operationalize and oversee its M&E policy to ensure the implementation of the Guidelines. GAO is making recommendations to OMB, DOD, State, and USDA. OMB did not agree with the recommendation to update the Guidelines, but GAO maintains that doing so can help to emphasize the importance of the M&E practices we identified. DOD, State, and USDA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "The 22 TCS programs covered by our review varied widely in total donations and number of scholarships awarded. For example, total donations during 2017 ranged from $854,326 in New Hampshire’s program to approximately $623 million in Florida’s largest program (see fig. 1 and app. I). Of the 17 states operating these programs, Arizona, Florida, and Pennsylvania operate the largest programs in terms of both dollars donated and number of scholarships awarded. These are also the only three states that operate multiple programs per state. Decisions about whether to develop and operate a TCS program—as well as how to structure requirements—are at the discretion of each state; there is no federal role in establishing these programs. For example, states sometimes choose to establish requirements that SGOs and schools must follow as a minimum condition for participation. Since scholarships are funded through donations rather than state appropriations, the financial impact to states from TCS programs primarily occurs through forgone revenue resulting from the associated tax credits. Various state agencies, SGOs, and participating private schools generally all play a role in administering state TCS programs, with the specific division of responsibilities varying by program. State agencies that are responsible for tax administration, education, or both, generally administer these programs. For example, in some programs, state agencies disseminate information to donors, scholarship students, or the public, and approve SGOs or private schools to participate in the program. SGOs are tax-exempt organizations that are generally responsible for managing some aspects of the donation process—such as collecting donations—as well as awarding scholarships to students. Participating private schools educate students receiving tax credit scholarships. They sometimes also facilitate donations or inform parents about TCS scholarship awards. Participating schools can vary in terms of characteristics such as their size, religious affiliation, and whether they focus on specific student populations, such as students with disabilities. In September 2018, we issued a report that described state TCS program requirements related to donations and student eligibility for programs operating at the beginning of 2018. In that report, we found that these programs offered tax credits to individuals, businesses, or both that ranged from 50 percent to 100 percent of the donation amount. We also found that most programs set limits on the amount of TCS program donations that could be made per year. In terms of eligibility, we found that TCS programs commonly determined student eligibility based on their household income, with income limits varying widely across programs. States have established requirements that SGOs must follow as a condition of participation in their TCS programs. See figure 2 for key requirements that some states have chosen to put in place for SGOs. All TCS programs have financial requirements that limit the percentage of donations that SGOs are permitted to use for non-scholarship expenses—such as staff salaries—and most also require SGOs to undergo annual financial audits or reviews. (See fig. 3.) The limit on the percentage of donated funds that SGOs are permitted to use on non-scholarship expenses ranges from 2 percent to 10 percent for 20 of the 22 programs. The other 2 programs—both in Pennsylvania— have a limit of 20 percent. About half (12) of programs also require SGOs to follow rules about how donated funds or related interest are managed and spent. For instance, South Dakota requires SGOs to spend all revenue from interest or investments on TCS scholarships. Most (19) programs require SGOs to undergo either an annual financial audit or review. Some of these programs require more extensive audits for SGOs that receive donations over a certain dollar threshold (e.g., over $1 million). Three programs require at least some SGOs to submit proof of fiscal soundness, such as a surety bond or letter of credit. For example, to insure against potential financial loss, Florida requires SGOs to submit a surety bond equal to the SGO’s unspent donations. Some programs require SGOs to take steps to guard against conflicts of interest. For example, four programs require SGOs to have a conflict of interest policy or a policy designed to prevent individual financial gain among SGO personnel. For instance, New Hampshire prohibits SGOs from awarding scholarships to children of any SGO employee or to children of any business owner whose business donates to the SGO. About a third (8) of programs prohibit SGOs and participating schools from sharing resources or personnel, while other programs do not include such prohibitions. For example, in Pennsylvania, schools are permitted to operate as SGOs and award scholarships directly to their students. In addition, TCS programs require SGOs to be recognized as tax-exempt organizations by the Internal Revenue Service (IRS), so they generally are also subject to applicable federal requirements for tax-exempt organizations, such as filing an annual information return or notice with the IRS. In addition, SGOs may be subject to applicable state laws for tax-exempt or non-profit organizations. Figure 4 shows the number of state TCS programs that had selected administrative requirements for SGOs. All programs require SGOs to register or receive state approval by, for example, submitting an application to the state or providing documentation showing the SGO is a tax-exempt organization. Slightly fewer than half (10) of programs require SGOs to follow rules about the qualifications of SGO leadership personnel. For instance, Nevada requires the SGO’s top board member to sign an affidavit stating that no member of the board of directors or SGO employee has ever been convicted of a felony, among other requirements. Nine of the programs have requirements related to marketing and fundraising practices. For instance, Louisiana requires SGOs to send the state any advertisements so state officials can review the materials. Almost all (21) programs require SGOs to report to the state the number or total dollar amount of scholarships they awarded. Also, 19 programs require SGOs to report information about the characteristics of scholarship students, such as household income or geographic location. These programs sometimes choose to require SGOs to report individual student-level information or aggregated information for all of their scholarship students combined. For example, Alabama requires SGOs to report identifying information and scholarship amounts for each student to the state. Alternatively, Georgia requires SGOs to report the number of families of scholarship recipients by income group. Figure 5 shows how many programs had key requirements for SGOs related to scholarship awards. Almost all (20) programs prohibit SGOs from awarding scholarships to only one school. Some of these 20 programs require SGOs to allow students to use their scholarships at any qualified school whereas others allow SGOs to work with a subset of two or more schools. The majority of programs prohibit donors from recommending that scholarships go to specific students while fewer programs prohibit donors from directing funds to specific schools. More than half (16) of programs prohibit SGOs from allowing donors to recommend that scholarships go to specific individuals, such as students they know personally. About one-third (8) of programs prohibit SGOs from allowing donors to recommend that scholarship funds be used at a specific school. In addition to requirements for SGOs, states with TCS programs also have requirements for private schools. See figure 6 for key requirements that some states with TCS programs have chosen to put in place for schools. Some requirements were specific to schools participating in the TCS program and some requirements were for all private schools, regardless of TCS program participation. We counted programs as having a particular requirement as long as the requirement applied to at least some of the participating private schools. Programs generally have one or more academic requirements for participating private schools. (See fig. 7.) About half (9) of the programs require private schools to register or be approved by the state TCS program before their students can receive TCS scholarships. For example, in Nevada private schools must register with the state by completing a form acknowledging that they will follow program requirements. Other programs, such as Georgia’s, require SGOs to determine if private schools meet program participation requirements. Separate from any requirements to register with the TCS program itself, over half (14) of the programs require participating private schools to be accredited by the state or another organization, such as a regional accreditation organization. For example, Pennsylvania requires participating private schools to be 1) licensed by the state, 2) accredited by an association approved by the state, or 3) operated by a religious institution. Private schools participating in most TCS programs are subject to requirements regarding minimum instructional time or student attendance and requirements related to curriculum content or core subjects, such as reading, mathematics, social studies, and science. These requirements may or may not be the same as requirements for all private schools in a state. For example, Nevada requires all private schools, including those participating in its TCS programs, to provide 180 days of academic instruction per year. South Carolina’s TCS program generally requires participating private schools to offer the courses required to receive a high school diploma in the state. Eleven programs require schools to give academic tests to TCS students. Of these programs, three require participating schools to administer the same test required of public school students and eight allow schools to select among multiple tests. For example, Louisiana requires participating private schools to give TCS students the same state tests in English and math that are used in public schools, whereas Florida allows participating private schools to choose from a state approved list of norm-referenced tests. These 11 programs also require schools to report TCS students’ test results to the state, parents, or other entities. For instance, one of Florida’s programs requires schools to report test results to a university selected by the state to analyze TCS students’ test scores. Participating schools are often required to ensure their teachers undergo background checks or meet minimum qualifications, and less frequently required to undergo site visits. (See fig. 8.) Most (18) programs require participating private schools to conduct background checks or fingerprinting of employees with direct, unsupervised contact with children. Additionally, 12 programs require schools to ensure teachers meet certain qualifications, such as holding a state-issued certificate or a college degree. For example, Alabama requires all private school teachers to hold teaching certificates issued by the state. Nine of the programs require private schools to permit state or other officials to conduct site visits. In the majority of these programs, site visits are a general requirement for private schools in the state or a component of state accreditation that some or all schools are required to obtain before participating in the TCS program. Across these programs, the frequency of the site visit varies. For instance, Indiana state officials are required to make random site visits to at least five percent of participating private schools each year, while Iowa requires a site visit for all private schools operating in the state at least once every five years. Among the 22 programs, financial requirements for participating private schools are generally less common than academic and administrative requirements. (See fig. 9.) Most (19) programs have requirements related to student withdrawals, such as requiring schools to report or repay the scholarship when students withdraw from the school for which the scholarship was originally awarded. For example, Illinois requires schools and SGOs to prorate scholarships for students who transfer to another private school during the school year, while Louisiana requires schools to immediately notify the SGO and state if a scholarship student is no longer enrolled. Few (4) programs require schools to complete an annual financial audit or review. Among programs with such requirements, Florida requires schools that receive more than $250,000 in scholarship funds to submit the results of a financial audit to the SGO that awarded the majority of those funds. South Carolina requires all schools to include a copy of an audit or other financial review when they initially apply to participate in the program and annually afterwards. About one-fourth (5) of programs require schools to provide surety bonds or other evidence to demonstrate financial viability. For example, Louisiana requires schools that have operated for fewer than five years and will receive more than $50,000 in TCS funds to either provide a SGO with a surety bond equal to the amount of TCS funds they expect to receive during the school year or other information showing financial viability. Florida requires schools operating for fewer than three years to provide the state with a surety bond equal to the amount of scholarship funds for any quarter. In the three states with the largest TCS programs—Arizona, Florida, and Pennsylvania—SGOs are generally responsible for recruiting donors while state agencies administer tax credits. (See fig. 10). SGOs generally recruit potential donors and sometimes help them apply for tax credits. SGO officials in all three states described ways they solicit donations, such as meeting with representatives from corporations to promote TCS programs or providing banners and pamphlets for private schools to display. In Arizona—where SGOs are permitted to allow donor recommendations for specific schools or students in certain programs— SGO and school officials described roles for schools or students and their families in soliciting donations. For example, family members of prospective scholarship students may encourage members of their community to donate and recommend their child or child’s school, according to officials from a SGO and a school we visited in Arizona. In addition, SGO officials we spoke with in all three states noted that they help donors navigate the process of obtaining tax credits. For example, officials from a SGO in Pennsylvania described how they can fill out the application for tax credit pre-approval as the donor’s delegate or review completed applications for errors before donors submit them to the state. State agencies administer state tax credits based on the rules of each TCS program. Specifically, five of the eight programs in the three states have a maximum total dollar limit on the amount of all scholarship tax credits that can be awarded in a year. In those five programs, the state requires donors to apply for pre-approval of the tax credits to ensure the limit has not been reached and tax credits are still available. In the Florida and Arizona programs that have such limits, state agencies consider all donor applications for tax credit pre-approval on a first come, first served basis. Pennsylvania considers returning donors for pre-approval before considering new donors (while tax credits remain). State officials in Pennsylvania and Florida described different methods for reviewing donors’ tax compliance before and after they file their taxes and claim the TCS tax credit. Specifically, the Pennsylvania Department of Revenue checks for any outstanding tax liability before approving tax credits for the TCS programs and reviews all tax returns that claim TCS credits to ensure the amount of tax credits claimed does not exceed the amount that was pre-approved. The Florida Department of Revenue reviews tax returns that claim a TCS credit to ensure the amount of tax credits used matches the approved amount that was allocated for the donor. Officials from the Arizona Department of Revenue said it does not have a separate tax compliance review process for its TCS credits. All three states established scholarship requirements while SGOs managed the scholarship awards process by determining which students are eligible for TCS scholarships, which eligible applicants will receive scholarships, and how much to allocate to scholarship recipients within program limits. In contrast, state agencies have a limited role—or no role at all—in determining the allocation of scholarships among eligible students, according to state and SGO officials in all three states. However, states may provide guidance documents to help SGOs navigate the state’s general policies for the awards process. The number of SGOs awarding scholarships varied across the three states, as did program policies for how those SGOs determine which eligible students receive scholarships. In fall 2017, Florida had two SGOs—one of which awarded 99 percent of scholarships. Meanwhile, Arizona’s four TCS programs had between 14 and 60 SGOs each and Pennsylvania’s two programs had approximately 190 and 260 SGOs each. In addition, as described in table 1, states varied in their requirements for how SGOs prioritize eligible students, the degree to which SGOs may work exclusively with a subset of schools, and whether SGOs may consider donors’ recommendations when deciding which students receive scholarships. Further, because each SGO is responsible for establishing its own procedures for awarding scholarships within program requirements, a TCS program with many SGOs could have substantial variation in how scholarships are awarded. Prioritizing eligible students: The three states provide varying levels of discretion to SGOs in how they prioritize eligible students when awarding scholarships. For example, Florida’s largest program requires SGOs to award scholarships on a first-come, first-served basis with first priority to students who previously received a scholarship, and then to students from lower-income households or who are in foster care; its other program requires SGOs to award scholarships to students on a first- come, first-served basis. Two programs in Arizona and one in Pennsylvania outline requirements for how SGOs must prioritize among eligible students, while the remaining programs do not. Further, in all four Arizona programs and both Pennsylvania programs, SGOs are permitted to set additional criteria for selecting scholarship recipients beyond requirements set by the state, as long as those criteria align with program rules and existing laws. For example, officials we spoke with at one SGO in Arizona noted that their selection committee considers written narratives from students and their parents about the student’s character and academic achievement when prioritizing eligible students, among other factors. Working with a subset of schools: State policies about partnerships between SGOs and schools can affect which students receive scholarship awards and where students can use those awards. SGOs in Florida award scholarships to students who can then use their scholarship award at any private school that qualifies to participate in the TCS program. In contrast, Arizona and Pennsylvania allow SGOs to partner with subsets of participating schools and award scholarships exclusively to students at those schools. For example, in Arizona and Pennsylvania, some SGO officials noted that the scholarships awarded through their SGO may only be used at partner schools that shared a religious affiliation with the SGO. Allowing donor recommendations: The TCS programs in the three states also had different rules on whether donors may recommend that scholarships be awarded to particular schools or students. Florida prohibits donors from making scholarship recommendations or designations for specific schools or students. In all Pennsylvania and Arizona programs, donors can recommend or designate donations for specific schools, but SGOs can take different approaches to distributing any such recommended funds. For example, one SGO in Pennsylvania sends recommended donation funds directly to the designated schools and the school decides how to distribute the scholarship funds among eligible students. In Arizona, one SGO tracks the amount of donations donors recommend for each of its partner schools and awards those funds to eligible students enrolled at those schools. Further, Arizona’s two TCS programs designed for individual donors (rather than businesses), allow donors to recommend that the funds they donate go to specific students. An official at one SGO we visited in Arizona said they provided its external scholarship award committee with information about students’ applications, including any student-specific recommendations, to inform the selection process. Pennsylvania programs do not expressly prohibit donor recommendations for specific students. In Pennsylvania, officials from one school that was also a SGO told us that they did not accept student-specific recommendations, while officials in a different SGO described how donors may not make student- specific recommendations, but may designate certain groups of students, such as children of first responders. In addition to establishing program policies regarding how eligible students are selected, the three states also have requirements regarding the amount of scholarship money that can be awarded per student, and SGO officials described different methods for determining the amount and frequency of scholarship awards for each student. Among the schools we visited the proportion of students who received TCS scholarships compared to students who did not receive TCS scholarships varied. For example, in one school in Florida, less than one percent of students received TCS scholarships and in a school in Arizona, school officials told us that about 80 percent of students receive TCS program scholarships. When awarding scholarships, officials at some SGOs we visited chose to issue a limited number of awards at the maximum allowable scholarship amount while others chose to issue scholarships to more students in smaller amounts—sometimes for shorter periods. In addition, students in Arizona and Pennsylvania may receive multiple concurrent scholarships from different SGOs, different TCS programs in the state, or both. This approach potentially increases the amount of funding students receive; however, it can also present logistical difficulties for the schools and families of scholarship recipients, according to SGO and school officials. For instance, officials at one school described wanting students to receive as much TCS funding as possible, but said it was also challenging to track the different SGO award cycles, incoming funds, and the projected impact on tuition balances for each student. As part of the scholarship award process, some SGOs we visited told us they collected information about tuition and fees at schools to ensure scholarship award amounts do not exceed school tuition, per program requirements. Costs for tuition ranged from approximately $6,000 per year to approximately $37,000 per year among the schools we visited. To participate in TCS programs, all three states require SGOs to provide a description of some of their operating procedures and regularly report donation and scholarship information. The type of information states collect and how they determine whether SGOs are following applicable TCS program requirements varies. (See table 2.) All three states require SGOs to complete an application to participate in TCS programs, which involves signing a form attesting that the SGO will follow program requirements and providing other types of documentation. This documentation includes evidence that the SGO is recognized as tax- exempt by the IRS, descriptions of the SGO’s procedures for awarding scholarships, and other information, depending on the program. When reviewing SGO applications, state officials in all three states described how they check that all required information is included in accordance with program rules, but generally do not evaluate the content. For example, in Pennsylvania, a program official noted that the state agency checks that SGOs submit all required documents and relies on attestation statements signed by SGO officials as an essential step for certifying that SGOs will follow program requirements. In addition, state agencies also generally provide SGOs with some guidance on how to interpret program requirements in all three states. State agencies in both Arizona and Pennsylvania have developed guidance manuals for SGOs. Florida collaborates closely with SGOs to interpret program rules and develop guidance, according to SGO and state officials. In addition to application materials, all three states require SGOs to regularly submit information about donations received, scholarships awarded, and the results of financial audits or reviews. The extent to which these audits or reviews include an assessment of SGO compliance with program requirements varies. For example, in addition to financial audits, Florida also requires the state Auditor General to review SGO operations for compliance annually. According to SGO officials, the Florida Auditor General conducts file reviews and on-site visits during these compliance reviews. In Arizona, the SGO manual includes optional procedures financial auditors may use to determine if SGOs are following certain program requirements as part of their review. Pennsylvania does not require an assessment of SGOs’ compliance with program requirements as part of its annual financial audit. According to officials, states typically work with SGOs to resolve any compliance issues and state agencies rarely permanently revoke SGOs’ approval to participate in TCS programs. According to officials in Florida, no SGOs have been removed from the programs due to noncompliance. The officials said that once a SGO has been approved through the states’ initial application process it is very likely that they will be renewed each year unless a large compliance issue arises. Officials in Pennsylvania noted that they contact SGOs to clarify discrepancies in documentation and have temporarily revoked approval from a small proportion of SGOs that failed to submit required information. Officials in Arizona described one instance where a SGO was decertified due to noncompliance; officials stated that the SGO would be recertified if it resolved the compliance issues and reapplied to be a SGO. State officials in the three states described different approaches to overseeing participating private schools’ compliance with program requirements. Florida state officials described using a variety of monitoring activities to oversee participating private schools, while Arizona and Pennsylvania state agency officials said they do not conduct ongoing monitoring activities due to the parameters of their statutory authority. (See table 3.) Florida state officials described conducting site visits to schools during the initial application process and when they determine schools are at risk of noncompliance. They also noted their monitoring practices have led them to identify multiple issues of noncompliance at certain schools and, as a result, they have temporarily or permanently suspended those schools from receiving TCS program funds. They said the state also delegates certain monitoring activities to SGOs, such as reviewing financial audit results, following up with schools to resolve any issues resulting from those audits, and reporting those issues to the state. State and SGO officials said that SGOs in Florida may also be responsible for implementing any penalties to schools, such as adjusting scholarship payments to schools that do not meet certain reporting requirements. For all Arizona programs and one of two Pennsylvania programs, officials told us that they generally rely on SGOs to verify that schools receiving scholarship funds meet program requirements. For example, in Arizona, SGOs are responsible for determining if a school qualifies to participate in the state’s TCS programs. SGO officials we spoke to in Arizona noted they require schools to sign documents attesting that the school meets the requirements to be a qualified school. In Pennsylvania’s Educational Improvement Tax Credit Program, SGOs determine whether schools are qualified to participate and each SGO may approach this differently, according to state officials. For example, in addition to meeting the state’s criteria for participation in the program, one SGO we spoke with in Pennsylvania also required schools to be tax-exempt, have a board and budget, and share its religious affiliation. Another SGO we spoke with in Pennsylvania required that a school attest that it meets program requirements. In Pennsylvania’s other TCS program (the Opportunity Scholarship Tax Credit Program) state officials told us that they determine whether schools are qualified to participate and do not conduct subsequent monitoring activities. We provided a draft of this report to the Department of Education for review and comment. Education’s comments are reproduced in appendix III. Education also provided technical comments, which we incorporated as appropriate. We also provided relevant excerpts from the report to the appropriate state agencies in each state we reviewed and incorporated their technical changes as appropriate. Education did not comment on the report’s findings. Instead, it provided information about the administration’s tax credit proposal and made observations about how the scope of TCS programs covered in our report was both similar and different from educational programs that are addressed in the administration’s proposal. In its comments, Education further stated that given these scope differences, the GAO report may not fully inform the debate around the administration’s proposal. As stated in the draft report, promoting school choice options—both private and public—through a variety of spending and tax expenditure programs continues to be a topic of national debate. The purpose of this report was to examine state TCS programs that are used to fund scholarships that students can use to attend private elementary and secondary schools by describing: (1) key requirements state TCS programs have chosen to establish for SGOs, (2) key requirements for private schools participating in state TCS programs, and (3) how selected states implement TCS programs and how they assess whether SGOs and participating private schools are following key state requirements. Education stated that the draft report does not note that several TCS programs allow scholarships to be used for educational expenses beyond tuition. As stated in the draft report, some states use tax credit scholarship programs to fund preschool, career and technical education, or public school initiatives; the draft report further noted that these programs are outside the scope of this review. This report is the second of two GAO reports examining K-12 TCS programs. The prior report (GAO-18-679) discussed various ways students can use state K-12 TCS scholarships. Specifically, we reported that, as of SY 2017-2018, more than half of the programs (13 of 22) allowed students to use their scholarship money for costs like transportation and books in addition to tuition, whereas the remaining programs (9 of 22) required scholarship funds to be used for tuition only. Education asserted positive fiscal effects associated with state TCS programs and cited several studies to this end. As stated in the draft report, tax credits are a form of forgone revenue. Assessing the fiscal impact of these programs was not among the purposes of this report. Thus, we did not assess the reliability of these studies or the validity of their results. Education also commented on our decision to exclude Montana’s TCS program from the scope of this report. As noted in our report, we did not include Montana’s program because it was the subject of pending litigation at the time of our review. Finally, Education noted, as also stated in the draft report, that states and school districts have obligations under the Elementary and Secondary Education Act and the Individuals with Disabilities Education Act to make equitable services available to eligible private school students, including those who participate in TCS programs. We agree and have reported extensively on equitable services in the context of private schools in GAO-16-712 and GAO-18-94. In GAO-18-94, we recommended that Education review information provided by states related to changes in federal special education rights when a parent places a student with a disability in private school and work with the states to correct inaccurate or incorrect information. In that report, we identified some private school choice programs that were providing information that Education confirmed inaccurately described rights under the Individuals with Disabilities Education Act when a student with a disability is moved from a public to a private school. Education agreed with this recommendation – a recommendation GAO considers to be among the highest priority of recommendations we have made to the Department. However, Education has not yet fully implemented this recommendation and, as of September 2019, we found that some information Education confirmed to be inaccurate remains in several states’ private school choice program documents. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in Appendix IV. of donors: businesses, individuals, or both) 1,957b (both) 15 (business) 795 (business) $1,856 88,109b (individual) $1,450 47,895b (individual) ——- (both) 286d (business) 23,411b (both) ——- (both) 6,768b,d (both) 3,054 (both) 153 (both) 79 (both) 69d (business) 71 (business) of donors: businesses, individuals, or both) Average donation amount per donor ——- (both) 1,671 (business) 662 (business) 40 (business) 788d (both) ——- (business) 1,428 (both) The profiles in this appendix describe key features of the largest tax credit scholarship programs in Arizona, Florida, and Pennsylvania, as measured in terms of the scholarships awarded in 2017. Information in these profiles was obtained from interviews with state agency and selected scholarship granting organization (SGO) officials and state program documents, and was verified by state officials. GAO did not independently review state laws or regulations. Unless otherwise specified, program requirements are as of school year 2018-2019. 10 percent of donations. Undergo annual audit or financial review depending on amount of donations received. donations. Administrative Receive state approval to operate. Ensure all promotional materials include required language. scholarship awards. Academic Follow guidelines for school curriculum or core subjects. Follow guidelines for minimum instructional time or attendance. Donations: Donors contributed over $68 million during calendar year 2017. The program is open to individual donors only (business donors may fingerprinting for staff. donate through other Arizona programs.) Financial Follow requirements if students withdraw (e.g., report or repay). Donations are eligible for 100 percent tax credits. The program does not limit annual program-wide donations. Individual donors (filing as single) were limited to $569 in tax credits during tax year 2019. Scholarship awards: 34,632 scholarships were awarded between July 1, 2017 and June 30, 2018. All K-12 students who live in Arizona and attend a private school are eligible for the program. SGOs are expected to consider financial need when prioritizing among eligible applicants. Students may receive more than one scholarship from this program and concurrently receive scholarships from other tax credit scholarship programs in the state, up to the full cost of tuition. SGOs may partner with a subset of qualified schools, such as schools with shared religious views or teaching methods. Donors may recommend that their donation fund scholarships at specific schools or be awarded to specific students (within certain parameters). 0 percent of donations in first 3 years and 3 percent afterwards. Provide a surety bond or line of credit. Maintain separate accounts for scholarship funds. Not share resources or personnel with private schools. Have a conflict of interest policy. Undergo annual financial audit. Report information about donations. Administrative Receive state approval to operate. Not have an owner or operator who recently filed for bankruptcy. Limit funds used for marketing. Report on scholarship awards. Academic Register with state. Follow guidelines for minimum instructional time or attendance. Donations: Donors contributed about $623 million during tax year 2017. The program is open to businesses (not individuals). Donations are eligible for 100 percent tax credits. Maximum allowable credit amounts per donor vary from 50 percent to Administer an approved academic test (not necessarily the test used in public schools). 100 percent of tax liability, depending on the type of business tax to which the tax credit is applied. Program-wide limit on tax credits was $698 million in state fiscal year 2017-2018. Administrative Hire teachers who meet minimum qualifications. Ensure background checks or fingerprinting for staff. Undergo state site visits. Scholarship awards: 108,098 scholarships were awarded during school year 2017-2018. Students are eligible for full scholarships if their household income level does not exceed 200 percent of the federal poverty level or if they are in foster care. Financial Undergo annual audit if receive a Students may only receive one tax credit scholarship at a time, up to $7,208 for tuition and fees, as of school year 2017-2018. given amount of scholarships. SGOs must allow students to use scholarships at any private school Provide a surety bond or line of participating in the program. credit if in operation for less than 3 years. Donors are prohibited from recommending that their donation fund scholarships at specific schools or for specific students. Follow requirements if students withdraw (e.g., report or repay). 20 percent of donations. Undergo annual audit or financial review. donations. Administrative Receive state approval to operate. Report information about scholarship awards. Academic Be accredited, licensed by the state, or a religious institution. Follow guidelines for minimum instructional time or attendance. Follow guidelines for core curriculum or academic subjects. Administrative Hire teachers who meet minimum qualifications. Ensure background checks or fingerprinting for staff. Donations: Donors contributed about $87 million during tax year 2017. The program is open to businesses, including “special purpose entities” Financial Follow requirements if students withdraw (e.g., report or repay). which allow individuals to donate to the program through the entity. Donors are eligible to receive tax credits for 75 percent of their donations if they donate for one year and tax credits for 90 percent of their donations if they donate for two consecutive years. The maximum allowable tax credit amount per donor is $750,000. The program-wide limit on tax credits was $135 million in fiscal year 2017. Donors must apply for pre-approval before claiming tax credits. Past donors may apply for credits before potential new donors apply for any remaining tax credits. Scholarship awards: 37,725 students received scholarships during school year 2017-2018. Students are eligible for scholarships if their household income level does not exceed $85,000, plus $15,608 per dependent in the household (higher for students with disabilities). Students may receive multiple tax credit scholarship at a time. Total scholarship award amount may not exceed the cost of tuition and fees. SGOs may partner with a subset of qualified schools, such as schools with shared religious views or teaching methods. Donors may recommend that their donation fund scholarships at specific schools. Donors are not prohibited from recommending scholarships to specific students. In addition to the individual named above, Nagla’a El-Hodiri (Assistant Director), Barbara Steel (Analyst-in-Charge), Andrew Emmons, and Jessica L. Yutzy made key contributions to this report. Also contributing to this report were Isabella Anderson, Jeff Arkin, Deborah Bland, Lilia Chaidez, Sarah Cornetto, Caitlin Cusati, Charles Ford, Monika Gomez, Alison Grantham, Kirsten Lauber, Sheila R. McCoy, Mimi Nguyen, Corinna Nicolaou, and Michelle Philpott.", "summary": "All TCS programs are state programs. States develop program policies and requirements, including establishing the roles and responsibilities of SGOs and participating private schools. The President's fiscal year 2020 budget request included a proposal for federal tax credits for donations to state-authorized SGOs. GAO was asked to review key characteristics related to accountability in state TCS programs that can fund K-12 educational expenses. This report examines (1) key requirements state TCS programs have chosen to establish for SGOs, (2) key requirements for private schools participating in state TCS programs, and (3) how selected states implement TCS programs and assess whether SGOs and participating private schools are following key state requirements. GAO identified key requirements states may choose to establish related to accountability for SGOs and schools based on relevant research and prior work. GAO also reviewed program documents from all 22 TCS programs to identify whether they had these key requirements as of school year 2018-2019 and then verified this information with state program officials. GAO did not conduct an independent review of state laws and regulations. GAO visited Arizona, Florida, and Pennsylvania, which have the largest TCS programs. In each of these states, GAO reviewed program documents and interviewed officials at state agencies and staff at selected SGOs and private schools (selected to provide variation in size and other characteristics). State tax credit scholarship (TCS) programs—programs that offer state tax credits for donations that can fund scholarships for students to attend private elementary and secondary schools—have established various key requirements for the scholarship granting organizations (SGO) that collect donations and distribute awards. For example, all 22 TCS programs in operation as of January 2019 require SGOs to register with or be approved by the state and limit the percentage of donations they can use for non-scholarship expenses. In addition, almost all of these programs—which received over $1.1 billion in donations and awarded approximately 300,000 scholarships in 2017—also require SGOs to undergo annual financial audits or reviews (19 programs). Fewer programs have requirements about SGO fundraising practices (9 programs) or the qualifications of SGO leadership personnel (10 programs), such as restrictions on officials having previous bankruptcies. States also have various key requirements that apply to private schools that enroll students with TCS scholarships. For example, private schools in most of the 22 programs must follow certain academic guidelines related to curriculum content (18 programs) and instructional time (19 programs), and have staff undergo background checks (18 programs). Schools in fewer programs are required to conduct academic testing (11 programs), ensure their teachers have specified qualifications (12 programs), or undergo an annual audit or financial review (4 programs). The three states with the largest TCS programs—Arizona, Florida, and Pennsylvania—implement and oversee their programs in different ways. In all three states, state agencies administer the tax credits while SGOs are generally responsible for managing donations and awarding scholarships; the details of these processes varied based on the requirements of each program. For example, Arizona and Pennsylvania's programs allow donors to recommend that funds go to specific schools, which can affect how SGOs solicit donations and award scholarships. Florida does not permit recommendations. All three states require SGOs to report on operations and undergo annual financial audits or reviews, while the states differ in how participating private schools are overseen. Florida's TCS programs use multiple monitoring methods, while all Arizona programs and one of two Pennsylvania programs generally rely on SGOs to confirm that schools comply with program requirements.", "document_type": "gao"}
{"report": "DHS has 15 components involved in achieving its broad strategic goals of countering terrorism and homeland security threats, securing U.S. borders and sovereignty, securing cyberspace and critical infrastructure, preserving U.S. prosperity and economic security, and strengthening preparedness and resilience. DHS relies on contracts to support these missions and has 10 contracting activities with authority to procure products and services within and across DHS’s components. For example, OPO within DHS’s Management Directorate is responsible for contracting for a number of DHS’s components and offices, including the Science and Technology Directorate, the Cybersecurity and Infrastructure Security Agency, and the Countering Weapons of Mass Destruction Office. See appendix II for DHS’s organizational chart, identifying operational and support components and contracting activities. The FAR requires that agencies take certain steps when identifying and developing requirements that need to be addressed through the execution of a contract. For example, the FAR requires that agencies conduct market research, as appropriate, and defines market research as the process used to collect and analyze information about capabilities in the market that could satisfy an agency’s needs. While the extent of market research will vary depending on characteristics of the requirement, the FAR provides general policies and procedures for conducting market research with the goal of arriving at the most suitable approach to acquiring, distributing, and supporting supplies and services. The FAR also requires agencies to perform acquisition planning activities for all acquisitions to ensure that the government meets its needs in the most effective, economical, and timely manner possible. In addition to the FAR, DHS relies on the Homeland Security Acquisition Regulation and Homeland Security Acquisition Manual—issued by DHS’s Chief Procurement Officer to implement and supplement the FAR—to establish policies and procedures for all acquisition activities within the department. For example, together the Homeland Security Acquisition Regulation and Homeland Security Acquisition Manual provide more specific department-wide policies and procedures for implementing acquisition requirements laid out by the FAR, such as competition, acquisition planning, and market research. Contracting activities may also implement their own procedures that support and implement the FAR, Homeland Security Acquisition Regulation, and Homeland Security Acquisition Manual. DHS also has its own policies and guidance for managing its service acquisition programs. For example, DHS generally defines major acquisition programs as those with life-cycle cost estimates of $300 million or more. However, DHS’s Acquisition Management Instruction 102-01-001 identifies additional thresholds for approval of stand-alone service acquisition programs—service contracts that are not part of a larger acquisition program. Specifically, service acquisition programs with annual cost estimates of $1 billion or more, or between $100 million and $1 billion are identified as Major Level 1 or 2 acquisition programs, respectively, and generally require approval from DHS’s Chief Acquisition Officer. Service acquisition programs with annual cost estimates under $100 million can be approved at the component level in accordance with component policies and processes. As of November 2019, DHS did not have any service programs identified as Major Level 1 or 2. In response to the 2009 Presidential Memorandum on Government Contracting, OFPP, within OMB, issued a policy letter in September 2011 to all executive agencies—including DHS—to clarify, in part, when governmental outsourcing of services is and is not appropriate. Specifically, the letter defines inherently governmental functions, according to the definition in the Federal Activities Inventory Reform Act, as those that are so intimately related to the public interest as to require performance by federal employees, such as determining agency policy or budget requests. Additionally, it identifies categories of service functions that agencies are allowed to contract for, but that require heightened management attention, as they pose a risk to the government losing control of either its responsibility to perform inherently governmental functions or its mission and operations. Figure 1 illustrates the increasing risk related to contracting for these types of functions. The letter also provides guidance on managing the performance of closely associated with inherently governmental and critical service functions, among others. In 2010, OFPP had also identified categories of services requiring heightened management attention. The three categories of service functions requiring special or heightened management attention follow: 1. Closely associated with inherently governmental functions. The 2011 OFPP policy letter adopts a single definition of an inherently governmental function, clarifies the types of services that constitute those closely associated with inherently governmental functions, and highlights the steps that agencies must take to ensure that the contractor does not ultimately perform functions that are reserved exclusively for federal employees. The response to public comments in the 2011 OFPP policy letter—in accordance with the FAR— provides the example that aspects of acquisition planning, such as determining requirements and approving a contract strategy, are inherently governmental functions. However, contractors may be used to support acquisition planning efforts through functions such as performing market research or drafting statements of work. These supporting functions are deemed closely associated with inherently governmental functions and can be contracted for. However, the OFPP policy letter states that agencies are required to take certain steps—such as assigning a sufficient number of qualified government employees to perform contract management—to ensure, among other things, that the contractor does not perform, interfere with, or undermine the integrity of the agency’s decision-making responsibilities. 2. Critical functions. The 2011 OFPP policy letter describes critical functions that, when contracted for, pose a risk that the agency could lose control of its mission and operations. Among other things, the policy established the criteria for identifying critical functions that are internally unique to each agency based upon their mission and operations. As an example, the 2011 OFPP policy letter notes that analyzing areas of tax law that impose significant compliance burdens on taxpayers may constitute a critical function for the Internal Revenue Service’s Office of the Taxpayer Advocate. OFPP notes that when contracting for a critical function, agencies must retain sufficient internal capability either through: dedicating an adequate number of federal personnel to perform the function in-house or alongside the contractors in the event the contractor fails to perform; or ensuring federal personnel are available to oversee and manage the contractor workforce. 3. Special interest functions. Special interest functions, according to OFPP, are functions that required increased management attention due to heightened risk of workforce imbalance. Some special interest functions may also be either closely associated with inherently governmental or critical functions. According to OFPP, contracting for these functions also poses a risk that the agency can lose control of its mission and operations. In a November 2010 memo, OMB instructed agencies to identify and analyze a list of product and service codes to be deemed special interest functions. DHS, with OMB approval, has chosen 17 product and service codes to categorize as special interest functions, including policy review and development and acquisition support services. To mitigate the risk associated with contracting for special interest functions, agencies are required to analyze their contracts for special interest functions annually to ensure the mix of federal employees to contractors is appropriately balanced. For examples of functions deemed closely associated with inherently governmental, critical, and special interest, see appendix III. Since March 2019, DHS has required program officials to complete its Inherently Governmental and Critical Functions Analysis job aid for all proposed service contract requirements above the simplified acquisition threshold—currently $250,000—with a product and service code that is not included on DHS’s exemption list. The department established the Inherently Governmental and Critical Functions Analysis job aid to enable it to systematically ensure that proposed service requirements do not include inherently governmental functions and to identify those that contain functions considered closely associated with inherently governmental or critical. The job aid collects general information about the proposed service contract, such as a brief description, followed by three discrete sections to check for these three functions. Section 1. This section includes a checklist for functions that the FAR has identified as being inherently governmental, such as developing federal agency policy and determining price reasonableness of vendor bids. In order to proceed with contracting for the service, the program official has to certify that none of these functions exist within the proposed requirement. Section 2. This section includes a checklist for functions that the FAR and OMB have identified as being closely associated with inherently governmental functions, such as conducting market research or drafting statements of work. If program officials identify any functions that are closely associated with inherently governmental in the proposed requirement, the job aid includes a narrative section where the program official is expected to input information on the nature of the work to be performed by the contractor and how heightened management attention will be given. Section 3. This section requires program officials to consider whether the proposed requirement is necessary for the agency to effectively perform and maintain control of its mission, which would designate the requirement as critical. Agencies are allowed to contract for critical functions so long as the program official certifies that the agency has sufficient internal capacity to undertake the work if, for any reason, the contractor is unable to provide the service. Special interest functions are not required to be identified in the job aid. The job aid concludes with the program official’s signature and is eventually forwarded to the contracting officer as part of the overall procurement package prior to soliciting for the proposed requirement. The job aid was put in place following the March 2019 decommissioning of DHS’s prior tool—the BWAT. The BWAT was used to implement DHS’s Balanced Workforce Strategy, which focuses on achieving the appropriate mix of federal and contractor personnel. This strategy was established in October 2009 to meet the statutory requirements in the 2009 Omnibus Appropriations Act. The 2009 Omnibus Appropriations Act directed most federal agencies—including DHS—to devise and implement guidelines and procedures to ensure that, on a regular basis, consideration is given to using federal employees to perform new functions, and functions that are performed by contractors but can be performed by federal employees. The Balanced Workforce Strategy established processes to enable DHS to achieve the appropriate mix of federal employees and contractors to accomplish the department’s mission, while minimizing risk to DHS’s missions from an overreliance on contractors. DHS implemented this strategy through the BWAT—an online questionnaire completed by individual program offices for certain service contracts. The function of the BWAT was to ensure the proposed service functions are not inherently governmental, and to identify whether the functions are closely associated with inherently governmental, critical, or special interest, among others. In addition, the BWAT recommended the ratio of federal employees to contractors needed to oversee those services. This analysis was then approved by the program and reviewed by the contracting officer as part of the procurement package. According to officials from an internal DHS working group, the Balanced Workforce Strategy—and BWAT by extension—were deemed no longer necessary based on the maturation of the department’s program and contracting officials’ ability to identify inherently governmental, closely associated with inherently governmental, critical, and special interest functions without a detailed questionnaire. In addition, the software used to conduct the BWAT was not supportable and faced obsolescence issues. For additional information on the differences between the BWAT and the job aid, see appendix IV. DHS relies on its planning, programming, budgeting, and execution process to plan for and allocate resources—including those for service contracts—across the department. DHS uses this process to develop its Future Years Homeland Security Program—a database that contains 5- year program funding plans and is used to prepare a report to Congress—and the department’s annual budget request. According to DHS guidance, at the outset of the annual planning, programming, budgeting, and execution process, the Office of Policy and Office of Program Analysis and Evaluation under the Chief Financial Officer provides resource planning guidance to the components outlining departmental priorities. Following the identification of departmental priorities, DHS guidance states that components should consider their objectives and commitments within fiscal guidance constraints, to estimate needs in their resource plans. The components then prepare their annual resource plans, based on their needs and in line with DHS priorities, which are reviewed by DHS leadership and culminate in a document reflecting the department’s resource decisions. See figure 2 for a depiction of the planning, programming, budgeting, and execution process. Beginning with the fiscal year 2017 budget request, DHS has used the common appropriation structure to organize the information in its budget requests. This common appropriation structure is comprised of four appropriation accounts: procurements, construction, and improvements; operations and support; and federal assistance. Each of these accounts has mission oriented program/project activities that correspond to the components’ different operations. For example, ICE’s fiscal year 2020 budget request includes program/project activities for the three operational directorates that accomplish its mission— Homeland Security Investigations, Enforcement and Removal Operations, and the Office of the Principal Legal Advisor. Within the component’s program/project activity accounts, service contract requirements are reflected in budget documents through object classes prescribed by OMB. OMB guidance establishes object classes as a measure for communicating resource needs in budget justifications and identifies eight object class codes for other contracted services, as shown in table 1. We have conducted prior work on the use of service contracts across the federal government, including how agencies have mitigated challenges overseeing and managing risks associated with service contracts that require heightened management attention, and how agencies have identified estimated service contract needs as part of agency budget requests. Specifically: In December 2011, we reported on how the Departments of Homeland Security, Transportation, and Housing and Urban Development, the United States Agency for International Development, and the National Science Foundation considered and mitigated risks associated with professional and management support service contracts—including contracts that are considered to be a special interest function and can increase the risk that contractors inappropriately influence the government’s authority, control, and accountability for decisions. We found that these agencies generally did not consider and mitigate the risks associated with selected professional and management support service contracts prior to their award. We recommended that OMB establish a deadline for agencies to develop procedures to improve their management of risks related to professional and management support service contracts. OMB agreed with our recommendation but did not establish such a deadline. In February 2016, we reported on what insights the Department of Defense had into the military department’s use of service contracts to fulfill current and future requirements, and how the department reported on service contract requirements in its annual budget requests to Congress. We found that while program offices within the military departments generally had information on current and future service contract requirements beyond the budget year, that future service requirements through the Future Years Defense Program were not identified to Department of Defense leadership in annual budget requests because there was no requirement to do so. We also found that the Department of Defense’s budget requests to Congress did not include all planned service contract needs and that its contracted services budget exhibit intended to meet certain statutory reporting requirements significantly underreported its estimated budget request for contracted services. We suggested that Congress should consider revising statutory reporting requirements to include estimated requirements beyond the budget year. In August 2018, Congress included a provision in the National Defense Authorization Act for Fiscal Year 2019 requiring the Department of Defense to include information on planned service contract requirements in the Future Years Defense Program. We also recommended that the military departments revise budgeting guidance to collect service contract information beyond the budget year, and that the Department of Defense modify its approach for reporting on service contracts in budget exhibits to ensure that certain service contract requirements are included. The department generally agreed with these recommendations, and has taken some steps to update military department budget guidance and modified its approach for reporting service contract requirements in its budget requests. In September 2019, we reported on the extent to which the National Nuclear Security Administration reports information on service contract requirements in its congressional budget justification documents and manages potential risks of service contracts that are at risk of performing inherently governmental functions. We found that the National Nuclear Security Administration did not consistently include information on all of its service contracts in budget justification materials. We also found that the agency may not be effectively managing the risks of contractors performing inherently governmental activities because contracting officers are not required to document how they will oversee contracts for services closely associated with inherently governmental functions, and the agency does not verify that planned oversight is performed. We recommended that the National Nuclear Security Administration report on all professional support services contracts with obligations as part of its budget justification materials, ensure contracting officers document plans to oversee service contracts at risk of performing inherently governmental functions, and develop a process to ensure that contracting officers are carrying out planned oversight. The National Nuclear Security Administration generally agreed with these recommendations. DHS obligated about $70.7 billion, or 76 percent, of its $93.7 billion in total contract obligations on services from fiscal years 2013 through fiscal year 2018. See figure 3 for details on DHS’s obligations on services and products from fiscal years 2013 through 2018. DHS annual service contract obligations increased by 40 percent from fiscal years 2013 to 2018, from about $10.5 billion to $14.7 billion. This increase in service contract obligations was largely driven by increases in Federal Emergency Management Agency and CBP service contract obligations, which grew by $2.2 billion and $927 million respectively. In fiscal year 2018, the Federal Emergency Management Agency had the highest service contract obligations, at $3.3 billion, followed by DHS headquarters organizations, and CBP. Of the Federal Emergency Management Agency’s fiscal year 2018 service contract obligations, $2.5 billion, nearly 75 percent, were identified as disaster-related. See figure 4 for additional detail on fiscal year 2018 service contract obligations by DHS component. DHS relies on a variety of services to accomplish its missions. For example, about $2.1 billion, or 14 percent of DHS’s total fiscal year 2018 service contract obligations, were for guard services to protect federal buildings or other security needs. DHS obligated about $2 billion, or 13 percent of its total fiscal year 2018 service contract obligations, towards various information technology and telecommunications services—such as satellite services and hardware and software maintenance. DHS’s five service categories with the highest amount of contract obligations in fiscal year 2018 accounted for about 40 percent of its total service contract obligations that year. See figure 5 for additional details on DHS’s top service obligations. In fiscal year 2018, 65 percent of DHS’s total service contract obligations were for services in need of heightened management attention or oversight due to being a closely associated with inherently governmental, critical, or special interest function. DHS’s obligations on contracts for these types of services increased by about 58 percent, from about $6 billion in fiscal year 2013 to $9.5 billion in fiscal year 2018. See figure 6 for additional details on the proportion of contract obligations for services in need of heightened management attention over time. Within our selected components, obligations for service contracts in need of heightened management attention increased the most from fiscal years 2013 to 2018 for contracts awarded by ICE—increasing by $732.1 million. CBP’s obligations for service contracts in need of heightened management attention increased over this time frame by $598 million. Service contracts in need of heightened management attention accounted for more than three quarters of all service contract obligations in fiscal year 2018 for DHS headquarters organizations and ICE. See figure 7 for additional detail on fiscal year 2018 contract obligations for services in need of heightened management attention by DHS component. DHS has policies and guidance to identify its service and product needs and develop contract requirements. In addition to the FAR, Homeland Security Acquisition Regulation, and Homeland Security Acquisition Manual, which combined establish DHS’s acquisition regulations and contracting policies, DHS has developed additional guidance specific to identifying needs and developing contract requirements. For example, DHS’s Developing and Managing Contract Requirements Desk Guide for the Acquisition Workforce is available to program personnel as a resource for how to define requirements, including processes and required documents and templates. DHS has also developed guidance for program and contracting officials for specific activities related to the requirements development process—such as market research, acquisition planning, and source selection guides—as well as guidebooks for specific participants involved in identifying needs and developing contract requirements, such as the contracting officer’s representative. Based on DHS policies and guidance, we identified key processes DHS undertakes to identify needs and develop contract requirements for services and products. Of these key processes, assessing for inherently governmental functions is specific to DHS’s development of service requirements. In response to the 2011 OFPP policy letter’s requirements to screen service contracts for the performance of inherently governmental functions and consider how contractor employees are used to perform agency functions, DHS implemented the BWAT in 2013. As previously noted, this tool has now been replaced by the Inherently Governmental and Critical Functions Analysis job aid. These tools have been required for service contracts specifically to ensure that contractors are not performing tasks that should be reserved for federal employees. Once completed, the output from these tools are reviewed by the contracting officer and included in the procurement package. Figure 8 summarizes key processes we identified that DHS uses to identify and develop service requirements. In addition to the policies and guidance DHS has for identifying and developing service requirements, DHS components in our review have implemented additional guidance and tools. For example, USCIS has developed specific guidance to support the program office’s development of requirements, including information on how to define requirements, conduct market research, and develop a cost estimate and acquisition strategy. Further, all of the components in our review reported using tools, such as templates and checklists, to help guide program and contracting officials through the requirements development process. For example, all of the components in our review use templates for market research, acquisition plans, and requirements documents that identify what information officials should include in these documents. The components in our review also provided program and contracting officials with checklists for what documents are required in the procurement package, depending on the type of contract being solicited. Some of the components in our review maintain this information on acquisition websites that serve as repositories for DHS and component guidance, templates, and other requirements. For example, ICE’s Office of Acquisition Management’s portal provides guidance, documents, and templates by phase of the acquisition process, from acquisition planning and solicitation preparation through contract administration and close-out. DHS components in our review also relied on subject matter experts to assist in their requirements development efforts, with the level of involvement varying depending on the requirement. Specifically, officials associated with two of the eight contracts in our review stated they used integrated product teams to assist with developing their service requirements. For example, officials involved in requirements development for services at USCIS’s 135 Application Support Center locations told us they established an integrated product team with program officials, the contracting officer, cost estimators, Field Office Directorate personnel, and Office of Security and Integrity personnel. Officials from the other six contracts relied on more informal subject matter expert involvement. Component officials from three of our contracts that relied on more informal coordination methods said that when the requirement is recurring and has previously been contracted for, formal coordination through an integrated product team may not be necessary. DHS has established a process for reviewing the procurement strategy for certain service and product procurement actions prior to award, but has not developed an approach to ensure proposed service contract requirements are clearly defined or that it is consistently reviewing what DHS considers to be high-risk service procurement actions. In 2018, OCPO and the Office of Program Accountability and Risk Management began piloting a DHS-wide Service Requirements Review to validate, optimize, prioritize, and approve service requirements early in the development process. However, DHS discontinued these efforts before the pilot was finalized. According to DHS officials, they initiated this pilot because there had been no consistency or rigor for reviewing service contract requirements even though these contracts account for over 70 percent of DHS’s contract obligations. According to DHS documents and officials, the main objectives of the pilot were to: ensure service requirements are clearly defined and reviewed before planning how the services are obtained; assess whether the services should be provided in whole or in part by foster collaboration and opportunities to leverage efficiencies for similar service requirements to avoid duplication in services across the department; and assess whether the requirement should be managed as a service acquisition program. To accomplish these objectives, DHS identified stakeholders from within DHS’s Management Directorate to be headquarters-level reviewers for service requirements based on the type of service being contracted for. However, according to OCPO and Office of Program Accountability and Risk Management officials, the pilot was discontinued in April 2019 before any service requirements were reviewed because it was determined to be too resource intensive. According to DHS officials, the discontinuation of the Service Requirements Review pilot coincided with the implementation of the Procurement Strategy Roadmap, a separate OCPO-led initiative to review and approve the procurement strategy for all service and product acquisitions with a total estimated value over $50 million. The Procurement Strategy Roadmap requires contracting activities, along with their procurement teams, to present and discuss the procurement strategy with the DHS Chief Procurement Officer, members of OCPO, and other stakeholders as needed, prior to drafting an acquisition plan or other decision documents. According to OCPO officials, it was intended to require procurement staff to meet with OCPO officials early in the acquisition planning process, prior to the service contract requirement being finalized, to discuss how services and products would be purchased. Specifically, the Procurement Strategy Roadmap is intended to address what OCPO considered as key elements of the procurement process, such as the requirement, competition, the availability of strategic sourcing or small business options, and contract type. Following the discontinuation of the Service Requirements Review pilot in April 2019, OCPO and Office of Program Accountability and Risk Management officials discussed expanding the Procurement Strategy Roadmap to incorporate some elements of the Service Requirements Review pilot, including reviewing proposed requirements to determine if they are clearly defined and valid, when appropriate. For example, OCPO officials said they have included the Office of Program Accountability and Risk Management and the Office of the Chief Information Officer to facilitate additional DHS stakeholder involvement in some reviews, and to broaden the discussion beyond how services and products will be purchased and include what the requirement is and whether it needs to be purchased at all. However, as of February 2020, OCPO officials told us that reviewing requirements to ensure they are clearly defined and collaborating with additional DHS stakeholders to identify opportunities to leverage existing service requirements was not the intent of the Procurement Strategy Roadmap. For example, OCPO officials stated that proposed requirements may only be reviewed by additional DHS stakeholders during the Procurement Strategy Roadmap if the requirement is new, “unique,” or “high risk,” and that this decision is based on their review of the information in the Procurement Strategy Roadmap and professional judgment. When asked what constitutes a unique or high risk requirement, officials told us a proposed requirement could be high risk if it had historical procurement issues, but noted that ultimately the decision to review the requirement and whether to involve additional DHS stakeholders in that review is subjective and based on whether OCPO leadership believes other stakeholders may add value in developing and reviewing the proposed requirement. According to OCPO officials, some Procurement Strategy Roadmap requirements supporting major acquisition programs undergo separate review by DHS headquarters stakeholders in the Acquisition Review Board. However, high-dollar service acquisitions that are not associated with a major acquisition program or not above $100 million do not currently receive headquarters- level scrutiny to determine whether requirements are clearly defined or to leverage efficiencies and buying power for similar service requirements across the department. In addition, OCPO has not established a process to ensure it is consistently reviewing proposed procurement actions through the Procurement Strategy Roadmap. Our review of the fiscal year 2019 Procurement Strategy Roadmap eligible procurement actions found that OCPO subjectively waived the review for 18 of the 49 eligible actions— over 36 percent of the actions that should have been subject to a Procurement Strategy Roadmap. The waived procurement actions included three out of six Federal Emergency Management Agency actions, eight out of 16 OPO actions, four out of 12 CBP actions, and two out of six Transportation Security Administration actions. According to OCPO officials, the decision to waive a procurement action is a subjective one, made by OCPO leadership based on the initial information provided. For example, officials said the review may be waived if the procurement action is recurring or will be fulfilled using an already established DHS contract vehicle. We found, however, that the subjective decision to waive the reviews does not take into account other acquisition risks. For example, our review of the description of waived procurement actions found that 11 of the 18 actions were for services, including some for administrative and professional support and information technology services that DHS considers to be in need of heightened management attention. Two of the waived actions were for requirements that resulted in orders placed off General Services Administration Schedule contracts, despite an OCPO official telling us that these orders are expected to receive increased scrutiny to ensure that any existing DHS contract vehicles have been fully considered. Our review of waived procurement actions also found that OCPO waived several Federal Emergency Management Agency actions for disaster response activities and CBP actions for services at temporary soft-sided facilities used for holding detainees on the U.S.-Mexico border. Our prior work has noted challenges in requirements development and acquisition planning for these types of contracts. For example, in April 2019, we reported that contracting officers at FEMA were receiving requirements packages for disaster contracts that lacked technical specificity or had inaccurate estimates of the products and services needed. In March 2020, we also reported on acquisition planning, requirements development, and information sharing challenges with one of the waived procurement actions—a CBP delivery order for a soft-sided facility and services to hold and care for detainees—finding that these challenges led to CBP spending millions of dollars on services that were not ultimately needed. Federal internal control standards state that management should identify and respond to risk to achieve its objectives. OCPO officials acknowledged that the intent of the Procurement Strategy Roadmap was not to replace the Service Requirements Review that preceded it, and that expanding the scope of the Procurement Strategy Roadmap to review requirements would require additional resources. However, the department’s previous efforts to devote management attention to its growing proportion of service procurements are indicative of its concerns about its use of service contracts. While the Procurement Strategy Roadmap is not specific to services, it can provide a mechanism to address these concerns. Moreover, without documenting factors OCPO considers when waiving certain Procurement Strategy Roadmap eligible procurement actions, DHS is at risk of not consistently reviewing service procurement actions that could benefit from headquarters-level review. Given DHS’s reliance on service contracts, which accounted for 78 percent of DHS’s contract obligations in fiscal year 2018, developing a risk-based approach for reviewing proposed service requirements through the Procurement Strategy Roadmap or other means could help to improve DHS’s use of service contracts by identifying opportunities to leverage efficiencies and ensuring service requirements are clearly defined across the department. DHS does not have a formal process for identifying all service requirements in need of heightened management attention or for planning, documenting, and updating the amount of federal personnel necessary to perform or oversee these requirements. In order to maintain control of their mission and performance of inherently governmental functions, part of contracting for services in need of heightened management attention—including functions that are closely associated with inherently governmental, critical, and special interest—is ensuring that agencies dedicate an adequate number of federal employees to oversee these functions. Specifically, OFPP notes that prior to contract award, for services that require heightened management attention agencies should complete an analysis that among other things and depending on the service, establishes that they can: retain sufficient capacity and capability to give heightened management attention to contractor performance or retain control of its operations; limit or guide the contractor’s exercise of discretion; ensure reasonable identification of contractors and contractor work avoid or mitigate conflicts of interest. Functions Requiring Heightened Management Attention The Office of Management and Budget’s (OMB) Office of Federal Procurement Policy (OFPP) guidance identifies three categories of service contracts requiring heightened management attention—those closely associated with inherently governmental functions, critical functions, and special interest functions. Depending on the function, these categories of service contracts may involve contractor work products that support policy development and program evaluation, and other tasks that are essential to the agencies’ ability to perform its mission. According to OMB, these contracts require management attention to ensure that they do not result in the performance of inherently governmental functions by the contractor and that agencies retain control of their mission and operations. DHS officials stated that, as of March 2019, they use the Inherently Governmental and Critical Functions Analysis, or job aid, to screen proposed service requirements to ensure that there are no inherently governmental functions and to identify functions that may be contracted for that are closely associated with inherently governmental or critical. If a function is identified as closely associated with inherently governmental or critical, program officials must certify that there is sufficient internal capacity to oversee contractor activities and maintain control of its missions and operations. Further, if a function is closely associated with inherently governmental, the job aid includes a narrative section where the program office should document mitigation strategies to ensure heightened management attention and enhanced oversight occur throughout the life of the contract. We found that the job aid does not provide a place to identify special interest functions that require heightened management attention. In addition, the job aid does not require program officials to analyze or document the expected federal personnel necessary to perform or oversee service requirements in need of heightened management attention following contract award; therefore information available for planning purposes is limited. We analyzed all nine of the 27 completed job aids that included closely associated with inherently governmental functions, and found that none included any calculation of federal oversight personnel necessary or mentioned the federal personnel who will be expected to perform oversight activities. The narrative section of the job aid instructs components to document mitigation strategies for functions identified as closely associated with inherently governmental functions. We found that two of the nine job aids identified mitigation strategies, such as noting that federal employees will ensure the contractor’s presence is announced at all meetings. However, neither provided any detail about who—such as the program manager or contracting officer representative—would be responsible for performing and overseeing the contractor employees performing the contracted functions or tasks. Program and contracting officials from ICE, CBP, and USCIS stated that analyzing and documenting the expected federal oversight personnel necessary prior to contract award with the BWAT provided visibility—both within the program and across the component—into resource needs. However, only ICE continues to analyze and document federal oversight necessary outside of the job aid. Following the BWAT’s decommissioning, ICE established a Service Contract Review Template for all service contracts above $1 million. ICE program officials are expected to complete this template with information such as a description of the requirement, the anticipated product and service code, identification of special interest functions, expected number of contractors needed, the number of federal employees available to oversee the work, and a justification for outsourcing the requirement. From this information, the template produces a recommended percentage of federal personnel necessary to perform management oversight. ICE created this new process because it wanted to ensure that it has a repeatable, documented decision-making process that helps plan oversight, such as the proper balance of federal and contractor employees and determining the reasonableness of the contract. DHS’s job aid also does not provide a process to update oversight needs if the contracted tasks or functions change throughout the life of the service contract. Officials from three of the four components in our review reported not having a formal process for updating federal oversight needs when elements of the contract change—such as an increase in the number of contractor personnel performing tasks or a change in scope. For example, component program and contracting officials told us that, although one of the service contracts in our review experienced an increase in the number of contractor personnel, they did not update planning for federal oversight personnel needs. In contrast, officials from ICE reported having a process to reevaluate federal oversight needed that is triggered by specific contract funding actions. Funding actions that trigger the process include: establishing a new contract, exercising an option on an existing contract, or adding funding to a service contract. ICE officials explained that through this process they review the service contract for changes to the number of contractors and whether the current oversight levels are sufficient. If they find that current oversight levels are no longer sufficient, ICE officials stated that they would require the program office to develop a risk mitigation strategy, such as assigning additional oversight personnel or increasing the contractor’s reporting requirements. DHS headquarters officials told us they no longer have a formal process for analyzing and documenting federal oversight requirements because the department has matured since implementing the BWAT, and program and contracting staff are aware of how to plan for federal oversight requirements for service contracts in need of heightened management attention. Specifically, during our review, DHS and component officials from OPO and USCIS stated that they rely on their program and contracting officials’ historical knowledge and professional judgment to determine and communicate oversight needs informally at the component level. However, we found a lack of understanding and inconsistencies in how oversight was analyzed and documented prior to the BWAT’s decommissioning. Specifically, 25 of the 75 required BWATs for special interest functions we reviewed either could not be provided or did not contain the information used to calculate and, therefore, plan for sufficient federal employees to conduct oversight. In addition, according to DHS documents and officials, the department plans for federal oversight personnel needs more broadly through its annual workforce planning efforts; and therefore, it is not necessary to analyze federal oversight personnel needs at the contract level. Yet we found that DHS’s fiscal year 2018 annual workforce plan focused on DHS and government-wide mission critical occupations, like Border Patrol Agents and Transportation Security Administration Officers. The plan does not address oversight needs based on services in need of heightened management attention (i.e., contracted functions that are closely associated with inherently governmental, critical, or special interest). While DHS’s workforce plan accounts for government-wide mission critical occupations, such as contracting officers and specialists, there is not the same level of consideration given to program managers, employees who serve as contracting officer’s representatives, or other program staff that are responsible for performing oversight at the contract level. According to OCPO officials, program officials completing the job aid should document in the narrative section the federal personnel responsible for ensuring the task does not become inherently governmental. However, we found that the job aid instructions do not address how program officials should analyze or document the federal personnel who will be tasked with conducting oversight. The job aid also does not include similar instructions, or provide space, to depict this information for functions identified as critical. Moreover, although there is guidance on when an initial job aid needs to be completed, there is no guidance indicating when, and under what circumstances, program and contracting officials may need to update federal oversight needs based on changes to the functions or task being performed by the contractor. Officials associated with only three of the eight contracts in our review reported receiving some training on the new job aid, but OCPO officials explained that they have not provided additional training beyond the instructions in the acquisition alert that implemented the job aid. Federal internal control standards state that agency’s management should use and internally communicate quality information to achieve the entity’s objectives. OCPO officials told us that components—such as ICE—have the discretion to establish additional processes for identifying and calculating federal oversight beyond what is required by the job aid. However, without consistently identifying all service requirements in need of heightened management attention and establishing a repeatable process across the department for analyzing, documenting, and updating the federal personnel needed to perform or oversee the requirement when changes occur, program and contracting officials lack reasonable assurance that they are dedicating an adequate number of federal employees to oversee these functions. This places DHS components at risk of inconsistently planning federal oversight necessary to ensure the department retains control of its missions and the performance of inherently governmental functions. DHS components included in our review are at risk of not conducting the oversight tasks and safeguards necessary to ensure that, once the contract has been awarded, the contractor’s functions are performed in a way so as to not become inherently governmental, and that DHS retains sufficient internal capability to retain control of its mission for functions that are closely associated with inherently governmental, critical, or special interest. The 2010 Consolidated Appropriations Act states that agencies should have specific safeguards and monitoring systems in place to ensure the work that contractors are performing has not changed or expanded during performance to become an inherently governmental function. Additionally in 2010, OMB issued a memo that states agencies shall conduct meaningful analysis—through the annual service inventory—focused on special interest functions that require heightened management attention to ensure proper workforce balance. Based on our review of contract documentation and interviews with program and contracting officials associated with the eight contracts in our review, oversight of these service contracts in need of heightened management attention focused largely on assessing the quality of specific contractor tasks. Oversight of these contracts did not include a focus on ensuring the work of the contractors is not performed in a way so as to become inherently governmental, or that DHS retains sufficient internal capability to perform its missions. While assessing quality is important in monitoring contractor performance, it does not allow DHS to identify when tasks beyond what is detailed in the contract—including tasks that are potentially inherently governmental and require that final agency action reflects the independent conclusions of agency officials—are being performed. According to DHS’s most recent service contract inventory analysis and OCPO officials, DHS relies on well-trained contracting officer’s representatives to monitor contractor performance for inherently governmental functions. Yet one of the eight contracts in our review has not had a certified contracting officer’s representative assigned to the contract since its award in September 2018. For the remaining seven contracts, we found that their contracting officer’s representative appointment letters—which document oversight responsibilities— mentioned performing surveillance and inspections against the contract’s performance requirements. But only two of the seven letters—both from CBP—reference performing oversight tasks focused on how the contractor is completing the work. Specifically, both appointment letters stated that ongoing reviews should be completed focusing on the way work is performed and how the government is managing service acquisitions for closely associated with inherently governmental and critical functions. However, none of the letters we reviewed identified specific safeguards—such as vetting all contractor recommendations through a panel of federal employees—that federal personnel should perform to mitigate identified concerns with contractors performing closely associated with inherently governmental or losing sufficient internal capability for performing critical functions. Similarly, program and contracting officials associated with six of the eight service contracts in our review did not identify additional oversight tasks undertaken as a result of the contract requiring heightened management attention. Rather, these officials said they assess the contractor’s performance in terms of the quality of deliverables when asked about the types of oversight tasks performed. For example, DHS headquarters officials responsible for overseeing a service contract for technical support related to the development of nuclear detection technologies stated that their oversight largely focuses on tracking the completion of tasks included in the statement of work as well as available funding. These officials did not identify any additional actions taken to address the risk of contractors working in situations that permit or might permit access to confidential business or other sensitive information—a function closely associated with inherently governmental functions in need of heightened management attention. Additionally, acquisition officials from one of the DHS components in our review stated that they have previously relied on the contractor to report if they were performing work that was not specified in the contract. While performance monitoring is crucial to ensure that the contractor is meeting the terms of the contract, it alone does not provide DHS visibility into whether work is being performed that is outside the scope of the contract or inappropriate for contractors. Program and contracting officials associated with two of the eight service contracts in our review identified safeguards they have established to prevent contractors from performing inherently governmental work. For example, program and contracting officials associated with a USCIS contract awarded to assist in the preparation of Freedom of Information Act requests stated that they have safeguards in place to ensure the contractor does not approve agency responses to Freedom of Information Act requests—an inherently governmental function, according to the 2011 OFPP policy letter. Specifically, officials associated with this contract explained that they use a software program that does not allow a user without federal employee credentials to approve a request within the system. This is an example of a safeguard that can be instituted for similar service contracts when the risk of the contractor performing the inherently governmental function of approving requests is present. Additionally, officials associated with CBP’s service contract for maintaining its unmanned aircraft systems stated that they ensure that their onsite personnel do not direct contractors to perform unauthorized tasks by requiring these personnel to report directly to the program office. Contracting and program officials’ lack of focus on safeguards to mitigate risks associated with contract functions in need of heightened management attention is due, in part, to DHS not identifying—either in guidance or training that we reviewed—a list of oversight tasks that program and contracting officials can perform. DHS’s OCPO officials explained that there are unique aspects of each contract that should drive oversight needs so they have not established any required safeguards component program and contracting officials must employ. Despite the uniqueness of each contract, officials from OPO stated that it would be helpful to have a list of identified potential oversight tasks or safeguards for service contracts in need of heightened management attention to ensure they are managing the risk of the contractor performing work outside of scope. We found that at least one federal agency has such a list available. Specifically, the Department of State’s Contracting Officer’s Representative Handbook provides a list of mitigation strategies contracting officer’s representative can employ for contracts requiring risk mitigation—such as reserving final approval authority of any contractor proposed action for federal employees only. Additional strategies listed include requiring contractor affiliation be clearly displayed on all presentation material, and conducting conflict of interest reviews when contractors are performing services that involve or relate to evaluating another contractor’s performance. Without identifying what oversight tasks or safeguards component personnel can institute to prevent contractors from performing inherently governmental functions or from affecting the ability of the agency to maintain control of its mission and operations, DHS is at risk of its personnel not knowing which steps they should take to prevent that from occurring. DHS components in our review consider service contract requirements when identifying their resource needs and formulating their budget justifications, but DHS headquarters and Congress have limited visibility into requested and actual service contract requirement costs. DHS uses the planning, programming, budgeting, and execution process to allocate resources—including those for service contracts—across the department. DHS’s guidance for this process that we reviewed does not provide specific instructions for how the components should consider service contract requirements when budgeting, but program officials we spoke with said that they generally provide information on specific service contract costs, among other resource needs, to their budget offices during the programming phase. Components then include these resource needs in their budget justifications, which are submitted to DHS headquarters for review before being submitted to OMB and then Congress. Based on our review of component budget justifications, components communicate service contract requirements in three primary ways, but none provide complete visibility into service contract requirements. Object Classes: Object classes are broad spending categories identified in OMB guidance. As shown earlier in table 1, there are eight object classes for other contracted services. According to component officials, once the resource needs for service contracts are identified, they are grouped into the object classes that best represent the requirement by either program or budget officials before submitting budget justifications to DHS headquarters. However, object class codes do not provide visibility into just service contract requirements. For example, budget officials at ICE and USCIS told us that aligning service requirements across object classes is not always perfect or precise. According to ICE officials, object class codes may include other expenses, such as interagency agreements. Further, USCIS officials noted that some contract requirements can apply to multiple object classes, so how requirements are communicated by object class is subjective based on program officials’ judgment. Cost Drivers: According to DHS budget officials, cost drivers identified in budget documentation represent the requirements that make up the largest costs at the program/project activity level. Service requirements may be included as a cost driver, but only if the estimated value of the contract represents a large portion of the program/project activity’s costs. For example, ICE’s budget guidance instructs the program offices to identify major requirements that add up to at least 50 percent of the program/project activity resource needs as non-pay cost drivers. Based on that guidance, in fiscal year 2020 budget documentation, one of ICE’s service contracts included in our review—for Office of the Principal Legal Advisor document management services—is identified as a cost driver. Only one other contract included in our review—from USCIS—was identified as a cost driver. Capital Investment Exhibit: According to DHS budget officials, the five contracts with the highest dollar value supporting each component’s capital investment are identified in the component’s budget documentation. Service contracts may be included in the capital investment exhibit if they meet this criteria, but the details included are vague. For example, for each contract listed, the exhibit typically includes information such as the contract number and total value, but does not categorize whether the contract is for a product or service nor consistently provide a description of the contract itself. For the contracts in our review, one of the eight—a contract for nuclear detection technology technical support—was included in the capital investment exhibit in fiscal year 2020 budget documentation. Since component budget offices submit their proposed budget requests with service contract requirements aligned into object class codes, program/project activities, and capital investment exhibits, DHS lacks visibility into the components’ requested service contract requirement needs. For example, officials from the Office of the Chief Financial Officer stated that they do not have visibility into DHS and the components’ specific service contract requirements. Rather, officials said their visibility is limited to changes in service contract requirements that are justified as part of requested increases or decreases in components’ funds. While officials from the Office of the Chief Financial Officer stated that they can request additional information from the components on service requirements if needed, officials could not identify any specific circumstances that have led to them requesting this information for their own purposes or in response to congressional interest. See figure 9 for details on how service contract requirements are communicated to DHS headquarters in budget documentation. Moreover, although DHS obligates over three-quarters of its contract spending to services, neither the Office of the Chief Financial Officer or OCPO have full visibility into or track service contract requirement costs. For example, similar to how information is portrayed in budget documents, officials from the Office of the Chief Financial Officer stated they report obligations to Congress by object class level on a quarterly basis; therefore, visibility into service contract requirement costs is limited. Further, OCPO officials stated that they also do not have a system for tracking service contract obligations reported through FPDS- NG or otherwise. In a discussion held during the course of our review, congressional requesters expressed interest in receiving additional information and visibility into DHS’s estimated service contract requirements. Members of Congress have also previously expressed interest in having increased oversight and visibility into other aspects of DHS’s proposed spending as well as into the Department of Defense’s estimated service contract requirements. For example, DHS budget officials told us that the decision to include the top five highest dollar value contracts in its capital investment exhibits was driven by congressional interest in this type of information on service contracts. In addition, in 2009, Congress began requiring the Department of Defense to identify in its budget submission the amounts requested for its service contracts for each component, installation, or activity, excluding services related to research and development and military construction. For example, the Department of Defense has two budget exhibits that provide details on estimated service contract requirements—one that details its advisory and assistance services, and another that tracks contracted services across prior fiscal years. In February 2016, we found shortfalls in the Department of Defense’s reporting of service contract requirements in its budget documents. We recommended that it modify its approach for reporting service contracts in its budget justifications to include additional service requirements. The Department of Defense agreed with this recommendation and, in February 2016, took steps to fully report on these service categories in its service contract spending exhibit accompanying the fiscal year 2017 budget request. We have also reported on challenges with congressional visibility into DHS’s major acquisition programs in budget documents. In April 2018, we found that DHS budget practices limit Congress’s visibility into costs and recommended that DHS work with Congress to include information on operations and support funding requests for major acquisitions in its annual budget justifications. DHS agreed with this recommendation and addressed it by adding an operations and support funding information display for major acquisition programs to its congressional budget justification for fiscal year 2021. Federal internal control standards state that agency’s management should communicate quality information internally and externally to inform decisions. Although detailed information on service requirements is available at the component level, DHS’s budget justifications do not provide that level of visibility. Visibility into service requirements is especially critical given that increases in DHS’s service contract obligations—particularly those in need of heightened management attention—may pose risks to DHS maintaining control over its mission. Given these increases, additional visibility into how much of DHS’s mission is being accomplished through the use of services requiring heightened management attention could inform DHS’s decision-making on the tasks it chooses to contract for, and the balance of its federal and contractor workforce. Without working with Congress to determine the format and level of detail needed to communicate service contract requirements in budget information, DHS headquarters and Congress are at risk of not having the information for sound resource planning and decision-making related to DHS’s use of service contracts. Service contracts play a critical role in supporting DHS’s wide range of missions, but increases in service contract obligations—including significant increases in obligations for services in need of heightened management attention—necessitate DHS’s attention as it develops, reviews, oversees, and budgets for service contract needs. DHS’s recent effort to perform a headquarters-level review of certain service and product procurement actions is a positive step in improving the department’s visibility into how it is acquiring certain services and products. However, without developing a risk-based approach for reviewing certain proposed service contract requirements to ensure they are clearly defined and valid before they are procured and consistently reviewing eligible procurement actions, DHS cannot ensure it has established the rigor needed to review its service procurements. Further, changes in DHS’s processes and a lack of agency-wide guidance for planning, documenting, and updating federal oversight personnel and activities for services in need of heightened management attention have put the department at risk of not effectively addressing whether contractors are performing inherently governmental functions. These risks could pose challenges to DHS’s ability to maintain control over its mission and operations. Ensuring DHS has guidance for planning and updating the resources needed to oversee these contracts, and identifying the types of activities that federal personnel should be performing to mitigate the risks associated with these contracts are critical to DHS’s ability to address these concerns. Finally, despite the availability of information on specific service contract requirements within component program offices, DHS does not communicate most of this information in budget documentation provided to DHS headquarters or Congress, nor is DHS currently required to do so. Given that DHS’s service contract obligations—including those in need of heightened management attention—account for more than three quarters of DHS’s total annual contract obligations, DHS is missing opportunities to make more informed strategic decisions because it does not have visibility into its current or future service requirement spending for these services. We are making six recommendations to the Secretary of Homeland Security: The Secretary of Homeland Security should direct the DHS Chief Procurement Officer to, in coordination with the Office of Program Accountability and Risk Management, develop a risk-based approach for reviewing service requirements—through the Procurement Strategy Roadmap or other means—to ensure proposed service requirements are clearly defined and reviewed before planning how they are to be procured. (Recommendation 1) The Secretary of Homeland Security should direct the DHS Chief Procurement Officer to document the factors the Office of the Chief Procurement Officer considers when waiving procurement actions from its Procurement Strategy Roadmap to ensure it is consistently considering potential acquisition risks in its planning—including those specific to services. (Recommendation 2) The Secretary of Homeland Security should direct the DHS Chief Procurement Officer to update the Inherently Governmental and Critical Functions Analysis to require the identification of special interest functions. (Recommendation 3) The Secretary of Homeland Security should direct the DHS Chief Procurement Officer to update the Inherently Governmental and Critical Functions Analysis to provide guidance for analyzing, documenting, and updating the federal workforce needed to perform or oversee service contracts requiring heightened management attention. (Recommendation 4) The Secretary of Homeland Security should direct the DHS Chief Procurement Officer to develop guidance identifying oversight tasks or safeguards personnel can perform, when needed, to mitigate the risk associated with contracts containing closely associated with inherently governmental functions, special interest functions, or critical functions. (Recommendation 5) The Secretary of Homeland Security should direct the DHS Chief Financial Officer to work with Congress to identify information to include in its annual congressional budget justifications to provide greater transparency into requested and actual service requirement costs, particularly for those services requiring heightened management attention. (Recommendation 6) We provided a draft of this report to DHS for review and comment. In its comments, summarized below and reproduced in appendix V, DHS agreed with the third and fifth recommendations and identified steps it plans to take to address them. DHS disagreed with the first, second, fourth, and sixth recommendations. DHS also provided technical comments, which we incorporated as appropriate. DHS did not agree with the first recommendation, that OCPO, in coordination with the Office of Program Accountability and Risk Management, develop a risk-based approach for reviewing service requirements through the Procurement Strategy Roadmap, or other means, to ensure that proposed service requirements are clearly defined and reviewed before planning how they are to be procured. In its response, DHS cited Instruction 102-01-001 as codifying how DHS and its components acquire and sustain services for major acquisitions. However, as noted in our report, as of November 2019 none of DHS’s services programs rose to the level of being classified as a major service acquisition. Therefore, DHS is at risk of overlooking those service contracts that are not a service acquisition program or not associated with its major acquisitions. DHS also noted the use of existing key processes that enable it to identify needs and develop contract requirements for services. While we acknowledge in our report that DHS and selected components have these processes in place, we found they were not consistently used throughout the contracts in our review, and none can serve as a replacement for the kind of risk-based headquarters-level oversight that we believe is necessary. For example, among its processes, DHS cited the use of integrated product teams as a way to facilitate comprehensive reviews of service requirements. We noted in our report, however, that according to officials only two of the eight contracts in our review used such an approach. Further, DHS stated that other existing efforts meet the primary objectives of the Service Requirements Review pilot, thus making an additional headquarters-level review of service requirements unnecessary. However, all of these efforts were also already in place when the then Under Secretary of Management directed OCPO and the Office of Program Accountability and Risk Management to undertake its December 2018 pilot program to provide consistency and rigor to reviewing service contract requirements. Therefore, we continue to believe that given the amount DHS obligates in service contracts to support its mission, establishing a risk-based approach to review service requirements prior to and in coordination with its consideration of how those requirements are to be procured will help prevent negative acquisition outcomes and the potential for wasted resources. DHS also did not agree with the second recommendation. In its response, DHS stated that OCPO’s decision to waive a Procurement Strategy Roadmap review does not mean that the Chief Procurement Officer did not consider acquisition risks, and that it is unclear what other acquisition risks we believe are not being considered. The recommendation to document the factors considered when waiving the Procurement Strategy Roadmap is intended to ensure that the department is able to consistently apply a framework and maintain institutional knowledge—particularly given the risks and challenges that vacancies in top leadership positions throughout the department could pose to addressing management issues. Waiving procurements without documentation of what acquisition risks are being considered puts the department at risk of inconsistently making those decisions and not being able to leverage Procurement Strategy Roadmap lessons learned. DHS noted in its response that the decision to waive a procurement review is based on several considerations, such as the type of service, information provided to the Chief Procurement Officer by the Head of Contracting Activity, and historical and current knowledge of the procurement, among others. However, the department offered no further insights as to: what types of services may not warrant a Procurement Strategy Roadmap; what type of information provided by the Head of Contracting Activity may indicate a review is unnecessary; or how the Chief Procurement Officer maintains the historical knowledge of procurements that may have previously experienced challenges and thus warrant a Procurement Strategy Roadmap. We continue to believe that taking the step of documenting the factors considered—such as types of services that may require additional review, or challenges with prior procurements, some of which may have been awarded years prior—will help ensure that decisions to waive Procurement Strategy Roadmaps are made consistently and transparently. DHS did not agree with the fourth recommendation, that OCPO should update the Inherently Governmental and Critical Functions Analysis job aid to provide guidance for analyzing, documenting, and updating the federal workforce needed to perform or oversee service contracts requiring heightened management attention. In its response, DHS stated that the job aid requires components to certify that they have sufficient internal capacity to oversee and manage contractor activities and maintain control of its missions and operations when the requirement is a closely associated with inherently governmental or critical function. Further, DHS stated that the job aid requires components to certify that there are an adequate number of positions filled by federal employees to manage and monitor contractors if the requirement is a critical function. As noted in our report, each component is making its own determination, in the absence of guidance, as to what factors to consider. In its response, DHS stated that OCPO will assist components with examples of analysis by reviewing what some components are doing, and sharing those examples with others. However, in the absence of guidance about what DHS expects the components to analyze and document based on those examples, DHS does not know how or whether the components are considering the federal workforce available to oversee service contracts in need of heightened management attention, or what steps, if any, the components are taking to mitigate risks if there are not enough federal personnel available to oversee the contracts after award. In its response, DHS recognized the need to provide guidance for updating the job aid, if there is a change in the contract requirement, to help ensure it has sufficient internal capacity to oversee and manage contractor activities, maintains control of its missions and operations, and has the appropriate workforce in place. We consider this to be a positive step to address part of the recommendation; however, it is unclear what considerations the components will use to update their analysis without the presence of guidance for how to analyze the federal workforce needed prior to the contract being awarded. We maintain that without guidance, DHS is at risk of inconsistent consideration of federal oversight for service contracts across its components—an action at odds with its goals of improving integration, and centralizing and coordinating its many functions to ensure that its whole is greater than its parts. Finally, DHS did not agree with the sixth recommendation, to work with Congress to identify information to include in congressional budget justifications to provide greater transparency into requested and actual service requirement costs, particularly for services requiring heightened management attention. In its response, DHS stated that it does not believe including additional information on estimated or actual service contract requirement costs is appropriate, and stated that contract information can be found in congressional budget justifications in budget object class breakouts, cost drivers, and in the Procurement, Construction, and Improvement Appropriation Capital Investment exhibit. We acknowledge these same three sources of information in our report, and note the limitations with each (either over-estimating or under- estimating service contracts) to providing visibility into DHS’s estimated or actual service contract requirements—both internally to DHS and externally to Congress. For example, as we note in the report, contracts identified in the Capital Investment Exhibit are not categorized as being for a product or service nor does the exhibit consistently provide a description of what the contract is for. In its response DHS noted limitations with our analysis comparing contract obligation data from FPDS-NG with what is reported in DHS’s budget justifications, however, after discussion with DHS officials during our review, we did not include that comparison in our report. DHS also noted in its response that the congressional budget justifications are intended to focus on the request, not on the previous or current year’s contracts. However, the recommendation that DHS work with Congress is impartial as to what type of service contract information would be useful for providing greater transparency into DHS’s service contract requirements. Rather, the recommendation is intended to address the limited visibility both DHS and Congress have into DHS service requirements—in particular the significant increases in services requiring heightened management attention—and provide a means to report on that information to improve internal and external oversight over these requirements and to allow for more informed decision-making. The need for this visibility into service contract requirements is aligned with prior recommendations GAO has made related to the need to increase visibility in DHS’s congressional budget justifications for major acquisition programs’ funding requests; recommendations that DHS has agreed with and implemented. Given that service contracts accounted for over three quarters of DHS’s contract obligations from fiscal years 2013 through 2018, we continue to believe that our recommendation to work with Congress on how to convey that information in congressional budget requests is valid. We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the DHS Chief Procurement Officer, the Commissioner of U.S. Customs and Border Protection, the Director of Immigration and Customs Enforcement, and the Director of U.S. Citizenship and Immigration Services. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. You asked us to review the Department of Homeland Security’s (DHS) use of and planning for service contracts. This report addresses the extent to which DHS and selected components and offices (1) used service contracts from fiscal years 2013 through 2018; (2) identified, developed, and reviewed service contract requirements; (3) ensured oversight of service contracts requiring heightened management attention; and (4) considered service requirements in budgeting processes. To identify the extent to which DHS used service contracts, we reviewed the Federal Procurement Data System-Next Generation (FPDS-NG) data on DHS-funded contract obligations from fiscal years 2013 through 2018 adjusted for inflation using the Gross Domestic Product Price Index. We identified obligations for services using the codes associated with services in the General Services Administration’s Federal Procurement Data System Product and Service Codes Manual. We analyzed the FPDS-NG data to identify DHS service obligations compared to obligations for products, service obligations by DHS component, the types of services procured, and the proportion of service contracts for functions in need of heightened management attention—those deemed closely associated with inherently governmental, critical, and special interest functions. We assessed the reliability of FPDS-NG data by reviewing existing information about the FPDS-NG system and the data it collects— specifically the data dictionary and data validation rules—and performing electronic testing. We determined the FPDS-NG data were sufficiently reliable for the purposes of identifying DHS’s use of service contracts. We selected a non-generalizable sample of four DHS contracting activities that had high obligations for service contracts and special interest functions compared to other DHS contracting activities—U.S. Customs and Border Protection (CBP), U.S. Immigration and Customs Enforcement (ICE), U.S Citizenship and Immigration Services (USCIS), and the Office of Procurement Operations (OPO). For the purposes of this report, we will refer to these contracting activities, which include three components and one office, as components. From these components, we selected a non-generalizable sample of 100 contracts awarded in fiscal year 2018 that were above the simplified acquisition threshold and were not exempt from performing a Balanced Workforce Assessment Tool (BWAT)—a risk analysis tool used by DHS components at that time to identify the appropriate mix of federal and contractor employees. Seventy five of the 100 contracts were for special interest functions, with the remaining 25 randomly selected. From that sample, we selected eight contracts—two from each component—that were identified as requiring heightened management attention. We selected a range of contracts based on whether the contract contained functions requiring heightened management attention, the percent of recommended federal oversight, and whether the requirement was new, among other selection criteria. We conducted semi-structured interviews with program, contracting, and budgeting officials from the eight selected component contracts to identify how the selected service contract requirements were developed, overseen, and considered when budgeting. Information collected from the four components and eight contracts cannot be generalized to all components and contracts. For additional details on the contracts we selected, see table 2. To determine how DHS and selected components identified, developed, and reviewed service contract requirements prior to soliciting for a contract, we reviewed relevant documentation, including the Federal Acquisition Regulation (FAR), and DHS, CBP, ICE, USCIS, and OPO contracting policies. To determine what processes selected components have for identifying and developing service requirements, we reviewed documentation, and interviewed program and contracting officials associated with our four selected components and eight selected contracts. To determine how DHS is reviewing service contract requirements, we reviewed DHS Office of the Chief Procurement Officer (OCPO) and Office of Program Accountability and Risk Management guidance and documentation on recent DHS headquarters initiatives—the Procurement Strategy Roadmap and Service Requirements Review pilot—and federal internal control standards on risk assessment. We also interviewed officials on these efforts to identify similarities and differences, and the processes established to review certain service contracts. To determine the extent to which DHS and the selected components in our review ensured federal oversight of service contracts requiring heightened management attention, we reviewed relevant documentation and regulations including Office of Federal Procurement Policy (OFPP) memorandums, the FAR, DHS contracting policies and guidance, and federal internal control standards on information and communication and risk assessment. To understand how DHS and selected components planned and documented oversight needs we reviewed available BWATs from our non-generalizable sample of contracts that we identified as special interest functions. Using FPDS-NG data, we identified an additional 27 contracts with completed job aids that were awarded in fiscal year 2019 for special interest functions across our selected components following implementation of the Inherently Governmental and Critical Functions Analysis job aid in March 2019. We reviewed the completed job aid associated with each of these contracts to understand how the oversight planning process has changed. We interviewed OCPO and component program and contracting officials about their use of both the BWAT and the Inherently Governmental and Critical Functions Analysis job aid. To determine the extent to which DHS and selected components conducted federal oversight of service contracts requiring heightened management attention throughout the life of a service contract, we analyzed documentation—such as contracting officer’s representative appointment letters depicting oversight responsibilities and training for contracting and program officials—and interviewed officials responsible for performing oversight functions. Additionally, to understand the types of tasks oversight officials can perform to mitigate the risk of contractors performing inherently governmental functions or losing control of the department’s mission, we reviewed OCPO provided guidance and trainings and interviewed relevant officials. To determine the extent to which DHS and selected components consider service contracts when budgeting, we reviewed Office of Management and Budget (OMB), DHS headquarters, and component budgeting guidance, federal internal control standards on information and communication, and interviewed headquarters and component budget officials. To determine how service contract requirements are communicated during resource planning and budget formulation, we reviewed DHS and component budget justification documents to identify what ways service requirement information is reflected, including whether specific information on our selected contracts was visible. We compared the resources DHS reported needing and receiving in its fiscal year 2018 budget documentation with DHS’s use of service contracts as reported in FPDS-NG in the same fiscal year as a proxy for visibility of service contract requirements in DHS budgeting. We conducted this performance audit from February 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Department of Homeland Security Organizational Chart The Office of Procurement Operations is the contracting activity for the following DHS components and offices shown above: Countering Weapons of Mass Destruction; Cybersecurity and Infrastructure Security Agency; Civil Rights and Civil Liberties; General Counsel; Office of Intelligence and Analysis; Legislative Affairs; Office of Operations Coordination; Partnership and Engagement; Office of Strategy, Policy, and Plans; Public Affairs; Chief Information Officer; Chief Financial Officer; Secretary/ Deputy Secretary; Management Directorate; Privacy; and the Science and Technology Directorate. In September 2011, the Office of Federal Procurement Policy (OFPP) in the Office of Management and Budget (OMB) issued a policy letter to help agencies manage the performance of inherently governmental and critical functions. The guidance states contracts whose performance may involve closely associated with inherently governmental, critical, or special interest functions require heightened management attention. Specifically, guidance states that closely associated with inherently governmental functions are functions that require heightened management attention to ensure that contractor’s activities do not expand into inherently governmental functions. OMB’s response to public comments on the proposed policy letter provides examples of inherently governmental and closely associated with inherently governmental functions, as shown in table 3. In response to public comments on the proposed policy letter, OMB called critical functions core to the agency’s mission or operations. In addition, the policy letter states that critical functions, when contracted for, pose the risk that the agency can lose control of its mission and operations. Examples of work previously identified by DHS as critical functions for the department include: Intelligence services—proprietary software used to conduct deep and dark web searches on possible threats against senior officials. Risk mitigation services—supporting undercover agents’ identities Program support—immigration data integration Administrative services—working closely with agency senior leadership to conduct research, schedule and attend meetings, as well as develop policies. The policy letter states that agencies must retain sufficient internal capability to give critical functions heightened management attention by: dedicating an adequate number of qualified federal personnel to understand the agency’s requirements and perform functions alongside contractors, if necessary, in the event the contractor fails to perform; or ensure qualified federal personnel are available to oversee and manage the contractor workforce. OMB guidance also describes special interest functions as requiring heightened management attention. In a 2010 memo, OFPP issued guidance to help agencies conduct a required service contract inventory for fiscal year 2010. The guidance describes the service contract inventory as a tool to better understand how contracted services are used and whether contractors’ skills are utilized in an appropriate manner. According to the guidance, agencies should give priority consideration to special interest functions, which for fiscal year 2010 OFPP identified as the categories of professional management services and information technology support services. Special interest functions require increased management attention due to increased risk of workforce imbalance. DHS, in line with OMB guidance, has identified 17 product and service codes to categorize as special interest functions, as shown in table 4. The Department of Homeland Security (DHS) implemented its Balanced Workforce Strategy (BWS) in October 2009 to establish a set of processes that, when repeated on a regular basis, enables the department to achieve the appropriate mix of federal employees and contractors to accomplish the department’s mission while minimizing mission risk that may result from an overreliance on contractors. To accomplish the intended goals of the BWS, DHS instituted an online questionnaire called the Balanced Workforce Assessment Tool (BWAT). The BWAT was in place until March 2019 when DHS determined it— along with the strategy—were no longer necessary given the maturation of their acquisition workforce and their inability to support the software underlying the BWAT. In its place, DHS commissioned its Inherently Governmental and Critical Functions Analysis—known as the job aid. Marie A. Mak 202-512-4841 or MakM@gao.gov In addition to the contact named above, Penny Berrier (Assistant Director), Meghan Perez (Analyst in Charge), Erin Butkowski, Signe Janoska-Bedi, and Jacqueline Wade were principal contributors. In addition, the following people made contributions to this report: Pete Anderson, Lorraine Ettaro, Suellen Foth, Julia Kennon, Roxanna Sun, and Anne Louise Taylor.", "summary": "DHS's spending on services—such as guard services and technology support—represents over 75 percent of its annual contract obligations. The Office of Management and Budget has recognized that some service contracts require extra management attention because they pose a risk that the government could lose control of its decisions or operations. GAO was asked to review DHS's use of and planning for service contracts. This report addresses, among other objectives, the extent to which DHS and selected components and offices use, oversee, and budget for service contracts. GAO analyzed Federal Procurement Data System-Next Generation data from fiscal years 2013 through 2018; selected non-generalizable samples of four components with high service contract obligations and eight service contracts requiring heightened management attention; and interviewed DHS officials. From fiscal years 2013 through 2018, the Department of Homeland Security (DHS) increased its reliance on contracts for services, particularly those in categories that may need heightened management attention, such as drafting policy documents (see figure). These services include functions that are closely associated with inherently governmental, critical, or special interest, which could put the government at risk of losing control of its mission if performed by contractors without proper oversight by government officials. GAO found that DHS and selected components do not consistently plan for the level of federal oversight needed for these contracts because there is no guidance on how to document and update the number of federal personnel needed to conduct oversight. GAO also found that program and contracting officials from six of the eight contracts GAO reviewed did not identify specific oversight activities they conducted to mitigate the risk of contractors performing functions in a way that could become inherently governmental. DHS lacks guidance on what these oversight tasks could entail. Without guidance for documenting and updating the planned federal oversight personnel needed, and identifying oversight tasks, DHS cannot mitigate the risks associated with service contracts in need of heightened management attention. Selected DHS components have information on service requirements, but budget documentation—submitted to DHS headquarters as well as to Congress—does not communicate details about most estimated or actual service contract requirements costs. Given that services account for over three-quarters of DHS's annual funding for contracts, additional insights would shed light into how much of DHS's mission is being accomplished through services, including those requiring heightened management attention. Without more visibility into this information, DHS headquarters and Congress are at risk of not having complete information for sound resource planning and decision-making, particularly as it relates to determining what proposed service contract requirements DHS should prioritize when budgeting. GAO is making six recommendations, including that DHS provide guidance for documenting and updating the federal workforce needed to oversee certain service contracts and identifying oversight tasks, and report service requirement information in budget documents to Congress. DHS agreed with two of the recommendations and did not agree with four of them. GAO continues to believe the recommendations are valid, as discussed in the report.", "document_type": "gao"}
{"report": "FDA approves reference listed drugs and generic drugs that meet safety and efficacy standards for marketing in the United States. Generic drug companies must show that their drug is (1) the same as the reference listed drug with respect to the active ingredient(s), conditions of use, route of administration, dosage form, strength, and labeling (with certain permissible differences, as approved by FDA); and (2) bioequivalent to the reference listed drug, meaning it generally delivers the same amount of active ingredient(s) in the same amount of time as the reference listed drug. When the reference listed drug is available, it is also designated as the reference standard drug, which is the product generic drug companies must use to conduct bioequivalence testing. When the reference listed drug is not available, FDA will select an approved generic of the reference listed drug to serve as the reference standard drug. All drugs pose some level of safety risk to patients. According to FDA, for most drugs, routine, risk-minimization measures, such as FDA-approved professional labeling, are sufficient to protect the public from the drug’s risks. However, in some cases, FDA may require a drug company to take additional actions to ensure that the benefits of the drug outweigh its risks and to help mitigate or prevent serious risks of adverse side effects. Specifically, FDA may require the drug company to establish a REMS that includes one or more risk-mitigation strategies beyond the drug’s professional labeling. According to FDA, most REMS are designed to reinforce patients’ and health care providers’ behaviors and actions that support the safe use of the particular drug they cover. For example, FDA may require drug companies to give patients and health care providers additional information to reinforce certain safe use conditions or specific risks described in the approved labeling of a certain drug. FDA may require a REMS either before a drug is approved or after approval if FDA becomes aware of new safety information. When determining whether a REMS is necessary, FDA considers several factors, including, for example, the estimated size of the population likely to use the drug and the seriousness of the disease or condition being treated. FDA may require a REMS to include one or more components. For example, FDA may require drug companies to provide patients with certain information in the form of medication guides. Generally, medication guides include information on serious side effects, including those that might require emergency medical care or involve life- threatening conditions. Similarly, FDA may require drug companies to develop communication plans for how the drug company will disseminate information to health care providers. Communication plans can include, for example, information on any serious risks of the drug and any safety protocols to ensure its safe use. Thus, for one REMS, FDA could require a drug company to provide a medication guide. For a second REMS, FDA could require a drug company to provide both a medication guide and a communication plan. Table 1 below includes a list of selected REMS components. Additionally, if a reference listed drug is subject to a REMS, an approved generic drug is also subject to some of the same REMS requirements. FDA can also require drug companies to implement another REMS component, called “elements to assure safe use” (ETASU), if a drug has been shown to be effective, but is associated with a specific serious risk. Depending on the risk, FDA may require any or all of the following ETASU measures: Prescribers have specific training or special certifications; Pharmacies or health care settings where the drug is dispensed have Drugs are dispensed only in certain health care settings, such as hospitals; Drugs are dispensed with evidence of safe-use by the patient, such as requiring a patient’s acknowledgement that she has been counseled on a drug’s risks and understands and accepts these risks; Patients are monitored, for example, while taking the drug for specific adverse events or outcomes; or Patients are enrolled in a registry for collection of certain information, such as patient outcomes and adverse reactions associated with the drug. According to FDA, these measures are for drugs that can be marketed only if there are requirements in place to mitigate a specific serious risk listed in the drug’s labeling. If FDA requires certain ETASU measures, it may also require a drug company to develop an implementation system to enable the drug company to monitor and evaluate implementation of the ETASU measures by health care providers, pharmacists, and other responsible parties. Also, if a reference listed drug is subject to a REMS with ETASU and a generic version is being developed, the reference drug company and the generic drug company are required to develop a shared system—a system that is used by participating companies to coordinate their REMS activities and information about a drug’s risks. Under a required shared system, the generic drug company and the reference drug company use the same REMS documentation and other materials on the drug’s risks and generally share in the implementation and maintenance of any database and infrastructure (e.g., call center). According to FDA, shared systems can be beneficial in reducing the burden for patients and health care providers, such as prescribers and pharmacies, when accessing REMS informational materials or completing administrative requirements, including any required training or certifications for providers. Generic drug companies must submit REMS documentation and materials as part of their generic drug application. Generally, before FDA can approve generic drugs that are subject to REMS with ETASU, reference drug companies and generic companies must reach agreement on a required shared system. According to FDA, generic drug companies that are developing a required shared system should submit their proposed REMS materials to FDA by the midpoint of the application review process or another time as specified by the agency. Any delays in the development of a required shared system can affect FDA’s ability to approve a generic drug application. A generic company may request a waiver from FDA, which if granted, would allow the generic company to develop a separate system that includes the same ETASU measures required for the reference listed drug. For example, if the reference listed drug’s ETASU measures require prescriber certification and the dispensing of the drug in certain health care settings, then the generic drug company’s separate system must also include the same ETASU measures. In recent years, FDA and FTC have identified two practices that can hinder competition by preventing or delaying the development and marketing of generic drugs. The first practice the agencies identified involves limiting access to samples of reference standard drugs, which generic companies generally need to conduct bioequivalence testing. This practice can apply both to reference standard drugs subject to REMS, specifically those subject to certain ETASU measures, and those not subject to REMS. For example, some drug companies might limit access to samples of reference standard drugs subject to REMS, citing ETASU measures that limit distribution, such as the measure that limits distribution of drugs subject to REMS to only certain health care settings. Additionally, drug companies may limit access to samples of reference standard drugs that are not subject to REMS. Typically, generic companies obtain samples through normal distribution channels such as wholesale distributors. However, drug companies could, for example, limit the sale of their reference standard drugs to certain pharmacies, such as specialty pharmacies. FDA and FTC have testified before Congress that these distribution limits—for reference standard drugs with and without ETASU-related distribution measures—can hinder generic companies’ ability to develop generic drugs and to submit a generic drug application to FDA for review. The second practice involves circumstances when a reference drug company delays its negotiations with generic drug companies on a required shared system. The negotiations to develop a required shared system can be complex because all parties must agree on the implementation of the REMS as well as issues related to cost-sharing, confidentiality, and product liability concerns. As part of their generic drug application, generic companies must include an adequate REMS program in order to be approved. Therefore, delays in the development of a required shared system can affect FDA’s ability to approve a generic drug application. Our analysis of FDA data shows that as of March 18, 2019, there were 74 approved active REMS that apply to 523 drugs. These drugs pose a variety of risks to users, treat a variety of conditions, and some are generics. A REMS can apply to one drug, more than one drug, or to a large number of drugs. Specifically, the approved REMS apply to: 136 drugs because they pose a high risk of serious medical side effects, 384 drugs because they pose a high risk of serious medical side effects from misuse and abuse, and Three drugs because they have the risk of medical side effects from both the use of the drug and from misuse and abuse. These drugs also treat at least 15 different types of medical conditions such as cancer, cardiovascular, and respiratory conditions. Twenty-two are orphan drugs, which are drugs intended to treat rare diseases. One hundred forty-three of these drugs are reference standard drugs, and 64 of these reference standard drugs have one or more approved generics that are also subject to REMS. (See Table 2) For example, FDA approved a generic of the drug Clozaril, which is used to treat mental and mood disorders. Both Clozaril and its generic, Clozapine, are subject to a REMS to prevent adverse medical side effects. Medicare and Medicaid paid at least $11.8 billion in 2017 for reference standard drugs subject to REMS. Specifically, in 2017 Medicare paid at least $8.5 billion for 83 of 139 reference standard drugs subject to REMS. This amount accounted for at least 8 percent of all Medicare drug spending in 2017. In the same year, Medicaid paid at least $3.3 billion—or at least 15 percent of all Medicaid drug spending—for 83 of the 139 reference standard drugs subject to REMS. Appendix I provides information from available data on Medicare and Medicaid spending on reference standard drugs subject to REMS. Of the 74 active REMS in our analysis, 51 have at least one required ETASU measure, and 35 specifically limit how drugs are distributed. Thirty-one of the 74 active REMS also require medication guides explaining the risks of the drug to be given to patients, and 12 require a communication plan for how the company will disseminate information to health care providers. Similar to how the 74 active REMS can have more than one REMS component, the 51 active REMS with ETASU measures can have more than one required measure. For example, 19 active REMS have an ETASU measure requiring patients to be enrolled in a registry and an ETASU measure requiring drug companies to provide training to prescribers of the drugs. Over half of the 51 active REMS with ETASU include measures that may limit how drugs are distributed. Specifically, 35 active REMS with ETASU measures include a requirement for drug companies to ensure drug dispensing settings are specially certified before they distribute the drugs. The certification process may require dispensing pharmacies to enroll in education programs provided by the drug companies. For example, to mitigate the risk of accidental overdoses from the misuse and abuse of fentanyl products, dispensing pharmacies are required to complete an education program that addresses—among other things— the risks of fentanyl products, patient selection, drug dosage, and patient counseling. In addition, for 10 of the 74 active REMS, companies have entered into a shared system. In three of the 10 shared systems, generic companies received a waiver from the shared system requirement after they were not successful in negotiating a shared system with the reference drug companies. In these three cases, the generic drug companies entered into shared systems that are separate from the reference drug company systems. For example, after developing generics of Lotronex, which is subject to REMS with ETASU and intended to treat gastrointestinal conditions, the generic companies were required to enter into a shared system with the reference drug company. When these companies were not successful in negotiating a required shared system, FDA determined the burden of developing a required shared system with the reference drug company outweighed the benefits of having one and waived the requirement. Once FDA granted the waiver, multiple generic companies were allowed to share REMS materials and administrative requirements with heath care providers via one shared system that is separate from Lotronex’s REMS system. FDA and FTC have taken four actions to address circumstances when generic drug companies cannot access samples of reference standard drugs or experience delays in negotiating required shared systems. According to FDA and FTC, both circumstances can hinder generic drug companies’ ability to develop and market generic drugs. Three of the actions focus on making samples of reference standard drugs accessible and the fourth focuses on facilitating the development of a required shared system. While all four of the actions pertain to drugs subject to REMS, only two of the actions pertain to drugs both subject to REMS and not subject to REMS. According to FDA officials, the agency is even more limited in what actions it can take when drugs not subject to REMS are involved. Drug Companies’ Perspectives on Limited Access to Samples of Reference Standard Drugs with Elements to Assure Safe Use (ETASU) that Limit Distribution Officials from all four of the generic drug companies we interviewed told us that their inability to access samples of reference standard drugs with ETASU measures that limit distribution either delayed or discouraged them from developing generic drugs. Officials from two of the five reference drug companies we interviewed told us they were unaware of specific instances when generic companies had difficulty obtaining samples or that generic companies had requested samples of reference standard drugs with ETASU measures that limit distribution. Also, officials from two reference drug companies cited safety concerns as the reason for limiting the distribution of their drug. FDA issued draft guidance on how to obtain a safety determination letter. One of FDA’s actions focused on facilitating generic drug companies’ access to reference standard drugs with ETASU-related limited distribution measures. (See sidebar for drug companies’ perspectives on this practice.) In 2014, FDA issued draft guidance describing how a generic drug company could ask the agency to send what is known as a safety determination letter to the reference drug company on the generic drug company’s behalf. The draft guidance explains how FDA could send a letter stating that the agency had reviewed the generic company’s plans for its bioequivalence testing and determined that these plans included safety measures that were comparable to those in the ETASU measures for the reference standard drug. For example, if the reference standard drug’s ETASU required protections to prevent fetal exposure to the drug, the generic company’s plans should include the same protections. The safety letter would also note that FDA would not consider it a REMS violation to provide reference standard drug samples to the generic company requesting the safety determination letter. According to FDA, some reference drug companies were concerned that providing samples to the generic drug company would violate REMS requirements. From 2016 to 2018, FDA issued 12 safety determination letters to reference drug companies on behalf of generic companies, according to agency data. However, FDA did not issue a safety determination letter for all of the requests it received. According to FDA officials, there are various reasons why they might not issue a safety determination letter to the reference drug company. For example, a generic company must sign a disclosure form in order for FDA to send the letter to the reference drug company, but the generic company does not always choose to do this. Additionally, the generic company might have withdrawn its request for a safety determination letter, or FDA might be waiting for additional information from the generic company in order to complete its review. According to FDA officials, there is no need for a safety determination letter (which assures the reference drug company that providing samples to the generic drug company will not be considered a violation of their REMS) when there is no REMS for the product in the first place. Drug Companies’ Perspectives on Limited Access to Samples of Reference Standard Drugs Not Subject to Risk Evaluation and Mitigation Strategies (REMS) Officials from three of the four generic companies in our review told us they had experience with drug companies’ imposed distribution limits on reference standard drugs not subject to REMS. Of the four generic companies in our review, officials from one company said they were not able to obtain the samples they needed and chose not to pursue developing a particular drug. Officials from one of the five reference drug companies said they have limited distribution for reference standard drugs not subject to REMS. They said their companies do so to ensure that their products are efficiently distributed, in part by using certain pharmacies. FDA published a web page with information about inquiries that included drugs both subject to and not subject REMS. In February 2019, FDA published a web page with information on inquiries made to FDA by generic companies seeking to obtain samples of reference standard drugs in order to develop generic drugs. (See sidebar for drug companies’ perspectives on this practice.) FDA officials said they published this list to increase transparency about continuing issues related to accessing samples and to raise awareness about the potential effect these issues might have on reducing competition in the drug market. This list included drugs subject to and not subject to REMS, the names of reference drug companies, and the number of inquiries made. According to the web page as of February 2019, inquiries were made for 54 reference standard drugs, including 25 drugs with ETASU-related limited distribution measures and 29 drugs without such measures. According to FDA data, the number of inquiries had been generally decreasing in the years prior to when the list was published. FTC officials reviewed inquiries the agency received from FDA and generic companies and filed two briefs. FTC told us it reviewed inquiries the agency had received from generic companies and FDA, including those related to information on FDA’s published web page. However, FTC officials said, to date, they have not brought a case charging a reference drug company with violating federal antitrust law for refusing to provide samples to a generic drug company. In order to take enforcement action, FTC needs to find sufficient evidence of activity that violates the Federal Trade Commission Act or the Sherman Act. For example, FTC would need to find that a reference drug company’s practice constituted monopolization in violation of the Sherman Act. According to FTC officials, they have not brought any antitrust cases to the courts, but have filed two amicus briefs related to cases involving drugs subject to REMS. In both of these briefs, FTC noted that the generic companies’ respective allegations, if true, established an antitrust violation and that the generic companies’ lawsuits should be allowed to continue. Drug Companies’ Perspectives on Negotiating Required Shared Systems Officials from one generic company said that the respective reference drug companies would not meet with them to negotiate the development of a required shared system REMS. Officials from another generic company said the negotiation process with the reference drug company lasted almost 2 years. Officials from four of the five reference drug companies we interviewed had experience negotiating required shared systems. Officials from three of these four companies told us that developing a shared system is a difficult, challenging, and complex process. Officials from one reference drug company said that the level of complexity can increase based on the number of companies and the different people involved. FDA issued waivers that allowed generic drug companies to develop a separate system from the REMS of the reference listed drug and issued draft guidance on how to obtain such a waiver. According to FDA, since 2007, the agency has received 13 requests for a waiver from the shared system requirement, and at the time of our data collection and analysis, FDA had approved three, the first in 2013. (See sidebar for drug companies’ perspective on required shared systems.) These waivers allowed the generic drug company to develop a separate system that includes the same ETASU measures required for the reference listed drug. According to officials, FDA was unable to grant the remaining waivers for different reasons. For example, the agency may still be reviewing the generic drug application submitted by the company that requested a waiver. Officials explained that the waiver request is part of the overall generic drug application and the agency cannot approve a waiver without approving the application as well. To further facilitate the process, in 2018, FDA issued draft guidance describing what factors the agency considers when granting waivers. The statute authorizes FDA to grant a waiver (1) if the burden of creating a required shared system outweighs the benefit of having it, taking into account the impact on the health care providers, patients, and drug companies involved or (2) if an aspect of the ETASU is covered by an unexpired patent or entitled to trade secret protection, and the generic company was unsuccessful in obtaining a license for use. FDA’s guidance describes examples of the potential benefits of having a shared system and the burdens of forming a shared system on health care providers, patients and drug companies that FDA will consider. For example, having a shared system could benefit drug companies by making a REMS for multiple products more efficient. In contrast, the drug companies negotiating a required shared system could be market competitors and involved in patent litigation related to the drug product. In general, the four generic drug companies and five reference drug companies we interviewed disagreed on the usefulness of FDA’s and FTC’s efforts to address the practices that may affect the development of generic drugs. FDA’s safety determination letters. Officials from three of the generic companies in our review said that the safety determination letters were not useful because they were not enforceable and did not require a reference drug company to provide a generic company with samples of a reference standard drug. In its comments on FDA’s draft guidance on obtaining a safety determination letter, one stakeholder representing generic companies expressed concern that reference drug companies now use safety determination letters as another requirement to obtain samples. In contrast, officials from three reference drug companies we interviewed told us that FDA’s safety determination letters addressed their safety concerns regarding sharing samples of reference standard drugs with generic companies. Further, officials from two of these three reference drug companies said they request these letters from generic companies that request samples of reference standard drugs. Officials from the remaining two reference drug companies we interviewed said they were not aware of FDA’s safety determination letters or did not have concerns or a position on the issue. FDA’s publication of its web page. Officials from one of the four generic companies we interviewed told us they thought the inquiries web page published by FDA was helpful. However, this same company said it had not noticed a significant effect in being able to access samples of reference standard drugs because of the web page. Officials from another generic company said it was too early to tell about the usefulness of FDA’s web page. Of the remaining two generic companies, officials from one company were unaware of the web page and officials from the second company noted that they were uncertain why a generic company would be included in the list of companies on FDA’s web page. Officials from two of the five reference drug companies we interviewed, and whose companies appeared on the web page, said they were unaware of any inquiries made to their companies requesting samples of reference standard drugs. Additionally, officials from one company told us they did not know why they were on FDA’s published web page because the company had sold the reference standard drug to another company and had informed FDA that this had occurred. According to FDA, the web page reflects the owner of the reference standard drug at the time the agency received an inquiry, regardless of whether the drug was later sold. Additionally, some generic companies might contact FDA directly without contacting the reference drug company because they anticipate having difficulties accessing samples of the reference standard drug. FDA notes on its web page that the agency did not independently investigate or confirm the access limitations described in the inquiries it received. FTC’s filing of amicus briefs. Officials from two of the generic companies in our review said FTC’s filing of amicus briefs was generally a positive step. Officials from two companies said the amicus briefs helped negotiations with reference drug companies. A third generic company said the amicus briefs helped raise awareness about issues generic companies are having. Officials from a fourth generic company said FTC’s actions could impact the company’s efforts to develop generic versions of reference listed drugs in the future. Officials from the five reference drug companies we interviewed did not have any comments on FTC’s specific amicus briefs. Waivers for a required shared system. Officials from three of the four generic companies we spoke with had experience with waivers. Officials from one of these three companies said the waiver guidance was helpful. However, officials from this generic company and a second company said it took FDA almost a year to grant their waivers. According to officials from a third company, they obtained their waiver within a month, in part, because negotiations had been ongoing for more than a year. According to FDA officials, the review of a waiver request is part of the generic drug company’s drug application. FDA will not grant a waiver unless the generic drug company meets the waiver requirements and its generic drug application is approved. Reference drug companies and other stakeholders expressed concerns about these waivers. Officials from three of the five reference drug companies we spoke with said the burden on health care providers or patients should be considered when granting waivers. Officials from one company specifically expressed concerns that as FDA grants additional waivers, it could place an additional burden on the health care system. For example, health care providers could be required to use multiple systems to access REMS information on the drug’s risks or to complete administrative requirements, such as required certification. The remaining reference drug companies did not have comments on the topic. In comments we reviewed on FDA’s draft guidance on these waivers, stakeholders noted concerns similar to those raised by the reference drug companies. For example, two groups representing pharmacists and pharmacies said that if FDA grants additional waivers, it could place a burden on the health care system. Historically, FDA has attempted to limit the number of required shared systems created under waivers. If a generic drug company is granted a waiver, it is allowed to create a separate system that includes the same ETASU measures required of the reference listed drug. However, to date, FDA has only granted waivers to generic drug companies that agree to share their systems with other drug companies that concurrently or subsequently develop generic or brand versions of the same reference listed drugs. We provided a draft of this product to FDA and FTC for their review and comment. Both agencies provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of HHS, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In order to estimate Medicare and Medicaid spending on reference standard drugs subject to risk evaluation and mitigation strategies (REMS), we compared data from the Food and Drug Administration (FDA) on drugs subject to REMS as of March 18, 2019, to publicly available 2017 data on the Medicare Part D Drug Spending Dashboard and the Medicaid Drug Spending Dashboard, maintained by the Centers for Medicare & Medicaid Services (CMS). These data covered drug spending and utilization for both of these programs for calendar year 2017, the most current data available. However, not all spending and utilization data for reference standard drugs subject to REMS were available. Since we analyzed data as of March 18, 2019, we were able to identify 139 reference standard drugs subject to REMS with corresponding cost data. To assess the reliability of these data, we interviewed knowledgeable agency officials. We determined that the data were sufficiently reliable for the purposes of our report. Medicare and Medicaid paid at least $11.8 billion in 2017 for reference standard drugs subject to REMS, according to cost data available from the CMS’s drug pricing dashboard. Specifically, Medicare Part D paid at least $8.5 billion for reference standard drugs subject to REMS. This amount—which includes Medicare Part D plan sponsors and beneficiaries Part D payments such as copays, but not price concessions, such as manufacturers’ rebates—accounted for at least 8 percent of all Medicare drug spending in 2017. Similarly, Medicaid paid at least $3.3 billion for reference standard drugs subject to REMS, or at least 15 percent of all Medicaid drug spending in 2017. Of the 139 reference standard drugs in our analysis, the greatest share of these programs’ spending, across medical conditions, was on reference standard drugs subject to REMS for cancer, based on our analysis of available data. Specifically, Medicare and Medicaid spent at least $4.6 billion on 8 reference standard drugs that treat cancer. See table 3 below for Medicare and Medicaid spending for reference standard drugs subject to REMS by medical condition treated. Further, our analysis of available data showed that Medicare and Medicaid spent the most on Revlimid, a drug used to treat cancer, totaling $3.6 billion with Medicare accounting for $3.3 billion of this total. More than 37,000 Medicare beneficiaries used this drug, at an average cost per dosage unit of $626.94. (See table 4.) In contrast, Medicaid spent the most on Suboxone, a drug used to treat opioid dependence, totaling $0.7 billion, based on available data. More than 3 million Medicaid claims were filed for this drug in 2017, at an average cost per dosage unit of $7.89. Vivitrol was the third most utilized drug under Medicaid. (See table 5.) Based on our analysis of available data, the selected examples of reference standard drugs subject to REMS had higher average cost per dosage unit compared to the generic. For example, Medicare spent an average cost per unit of $12.20 for Clozaril, a drug used to treat mental health conditions, compared to $0.99 for clozapine, a generic version of Clozaril. Table 6 below shows selected examples comparing Medicare spending for reference standard drugs to Medicare spending for a generic version, based on our analysis of available data. Our analysis of available Medicaid data showed similar results to our analysis of Medicare data. For example, Medicaid spent an average cost of $11.71 for Clozaril, a drug used to treat mental health conditions, compared to $0.97 for clozapine, a generic version Clozaril. Table 7 below shows selected examples comparing Medicaid spending for reference standard drugs to Medicaid spending for a generic version, based on our analysis of available data. In addition to the contact named above, Geri Redican-Bigott and Tom Conahan, Assistant Directors; Carolyn Garvey, Analyst-in-Charge; Zhi Boon, Gay Hee Lee, and McKenna Storey made key contributions to this report. Also contributing were Sam Amrhein, Kaitlin Farquharson, Cathy Hamann, and Diona Martyn.", "summary": "To manage the risks posed by some drugs, FDA requires drug companies to establish risk evaluation and mitigation strategies. Companies developing generic drugs generally need samples of the reference standard drug to conduct bioequivalence testing. Generic companies may also have to negotiate a shared system with the reference drug company, when that company's drug is subject to certain REMS requirements. FDA and FTC officials acknowledge that some drug companies have used certain practices that prevent or delay the development of generic drugs. The practices include limiting access to samples of reference standard drugs with and without REMS and delaying negotiations for creating required shared systems. GAO was asked to review drugs subject to REMS and drug companies' experience with these practices. This report describes (1) the drugs subject to REMS, and (2) FDA and FTC's efforts to address these practices, and stakeholders' views on agencies' efforts. GAO analyzed FDA data on the conditions these drugs treat and the REMS requirements that apply to the drugs. GAO also interviewed FDA and FTC officials and representatives from five reference drug companies and four generic drug companies, which GAO selected based on a variety of factors, including the companies' experiences with drugs subject to REMS . GAO also reviewed public comments and related documents from FDA and FTC. HHS and FTC provided technical comments on a draft of this report, which GAO incorporated as appropriate. The Food and Drug Administration (FDA) can require drug companies to establish risk evaluation and mitgation strategies (REMS) for drugs with serious safety concerns to ensure that a drug's benefits outweigh its risks. As of March 18, 2019, FDA approved 74 active REMS that cover 523 drugs that treat various conditions. One hundred forty-three of the drugs are reference standard drugs, which are drugs generic drug companies must use to conduct bioequivalence testing. Of these 143, 64 have at least one approved generic that is also subject to REMS. Ten of the REMS are shared systems that allow health care providers to obtain information from multiple companies on a drug's risks and satisfy other administrative requirements through one REMS system. According to FDA and the Federal Trade Commission (FTC), drugs with and without REMS have been the subject of practices that can delay or prevent generic drug development and marketing. FDA and FTC have taken actions designed to address some of these practices. According to FDA officials, they are more limited in what actions they can take when drugs without REMS are involved. Drug company officials that GAO interviewed had different views on these actions. To address practices that may limit access to samples of reference standard drugs and keep generic drugs from the market: FDA issued draft guidance in 2014 on how generic companies could obtain a letter stating that the agency would not consider it a REMS violation to provide reference standard drug samples to the generic company requesting the letter. Three of the four generic companies GAO interviewed said these letters were not useful because they do not require drug companies to share samples. In contrast, officials from three of five reference drug companies said the letters addressed their safety concerns about providing samples to generic companies. FDA does not issue such letters for drugs without REMS. In February 2019, FDA published a list of drug companies whose reference standard drugs were the subject of access inquires made to FDA by generic drug companies. One of the four generic companies GAO spoke with said FDA's list was helpful, and one reference drug company said it was uncertain why it was included on the list. FTC has reviewed inquiries it received from FDA and generic companies, and has filed amicus briefs in two cases involving drugs with REMS. According to FTC, to date, the agency has not brought a case charging a drug company with violating federal antitrust law for refusing to provide samples to a generic drug company. To address practices that may delay negotiations between reference drug and generic drug companies for creating required shared systems, FDA issued waivers and related guidance that allowed generic companies to develop a separate, but comparable, REMS shared system. One generic drug company said the guidance on waivers was helpful; however, one drug company said the waivers put added burden on health care providers who have to use multiple REMS systems.", "document_type": "gao"}
{"report": "The F-35 plays a key role in DOD’s modernization efforts. However, it faces concerns in several areas that will inform the program’s cost and performance in the future. These include the risk in its modernization efforts, its aircraft not meeting all reliability targets, and sustainment and supply chain challenges. Specifically, the F-35 program plans to award Block 4 development contracts before it has key business case documents that would normally inform this decision. Also, the program is not meeting all of its Reliability and Maintainability (R&M) targets. Finally, the F-35 program’s sustainment costs are rising as it also faces significant supply chain challenges. The F-35 baseline aircraft program completed development in April 2018. It started formal operational testing of the baseline aircraft in December 2018 after a 3-month delay. This testing was delayed for two main reasons: (1) to resolve critical deficiencies identified in developmental testing, and (2) to accommodate an unexpected grounding following the crash of an F-35B in September 2018. According to a test official, the program expects to complete testing in December 2019, about 3 months later than planned due to delays with the simulator that is used for more complex testing. Until that testing is complete, there is still a risk that additional deficiencies may be identified. With the program wrapping up development of the baseline program, it is transitioning to early development and testing activities for the Block 4 modernization efforts, which the F-35 Joint Program Office estimates will cost about $10.5 billion. With Block 4, DOD plans to add new capabilities and modernize the F-35 aircraft to address evolving threats. In April 2019, we found that DOD will not have a complete business case for Block 4 before it plans to award development contracts in 2019. Section 224 of the National Defense Authorization Act for Fiscal Year 2017 required DOD to submit a report containing certain elements of an acquisition program baseline—in essence, a business case—to include cost, schedule, and performance information and independent estimates—for Block 4. In 2018, we found that DOD’s report to Congress was incomplete. In its report, DOD stated that the acquisition program baseline would continue to be refined over the next year. DOD officials stated that the updated F-35 program baseline, with the Block 4 efforts included, will be released in April 2019. Over the past year, the program has already invested over $1.4 billion, in part to gain the knowledge it needs to develop that business case, such as a preliminary design review, as well as to establish Block 4 testing facilities and support early capabilities’ development. The program incorporated some Block 4 activities into its acquisition strategy, which was approved in October 2018. However, we found that three key Block 4 business case documents will not be ready before the program’s planned development contract awards in May 2019: Independent Technology Readiness Assessment: A Technology Readiness Assessment is a systematic, evidence-based process that evaluates the maturity of hardware and software technologies critical to the performance of a larger system or the fulfillment of the key objectives of an acquisition program. The program office plans to conduct a partial independent assessment of initial capabilities sometime between October and December 2019 with additional assessments to follow. A program official stated that technologies will not be integrated into the aircraft until they are adequately mature. However, without a complete independent Technology Readiness Assessment, the program will not have identified potential critical technology elements and, as a result, may be at risk of delaying the delivery of new capabilities. Test and Evaluation Master Plan: Although the F-35 program has begun testing Block 4 capabilities, it does not have an approved Test and Evaluation Master Plan. The Test and Evaluation Master Plan documents the overall structure, strategy, and objectives, as well as the associated resources needed for execution. Developmental and operational test officials have expressed concerns about the lack of an approved test plan, uncertain funding, the number of test aircraft available, and the draft test schedule, among other things. An approved, properly resourced test plan is essential for planning and preparing for adequate testing of the Block 4 capabilities. According to these officials, without an approved plan, the F-35 program is providing the test authorities with capabilities to be tested without giving them the necessary direction on how to adequately prepare to conduct the tests, making it difficult to execute testing. While this is still a concern, F-35 program officials explained that over the past 3 months they have been providing the test authorities with the direction needed to conduct testing. Independent Cost Estimate: The Block 4 Independent Cost Estimate, which details the program’s total estimated life cycle cost, is not complete. In August 2017, we reported that DOD estimated the development funding needed for the first phase of Block 4 was projected to be over $3.9 billion through 2022. Since then, the program incorporated more fidelity and specific Block 4 efforts that were not in the original estimate into its Block 4 cost estimate. Based on the program office’s latest estimate, the cost of Block 4 capabilities is expected to be $10.5 billion through 2024. According to OSD’s Cost Assessment and Program Evaluation office, it will provide the Independent Cost Estimate between October and December 2019 to support the F-35 program’s pending full-rate production decision, but this would occur several months after the program plans to award the Block 4 development contracts. According to the GAO Cost Guide, an Independent Cost Estimate is considered one of the best and most reliable estimate validation methods as it provides an independent view of expected program costs that tests the program office’s estimate for reasonableness. Without an Independent Cost Estimate, Congress does not have insight into the full potential cost of Block 4. The expected completion dates for these documents are between October and December 2019, at the earliest. Figure 1 shows key Block 4 dates such as the Block 4 re-plan, which included revising the cost estimate for Block 4 that DOD established in 2017, the planned development contract awards, and planned completion dates for the three remaining critical business case documents. As seen in figure 1, the program office plans to award Block 4 development contracts in May 2019, at least five months before any of the critical business case documents will be available. Based on best practices identified by GAO, without an independent Technology Readiness Assessment, Test and Evaluation Master Plan, or an Independent Cost Estimate, program officials cannot have a high level of confidence that the requirements are firm and that risk has been adequately reduced before beginning efforts estimated to cost $10.5 billion in funding to develop Block 4. If program officials move ahead with Block 4 contracts without gaining the knowledge that a full business case would provide, Block 4 modernization efforts will be at risk of experiencing the same kind of cost and schedule growth the baseline development program experienced. To address this risk, in April 2019, we recommended to the DOD that it should ensure the F-35 program office complete its business case, to include the three documents discussed above, at least for the initial Block 4 capabilities under development before initiating additional development work. DOD did not concur with this recommendation. In its comments, DOD stated that the F-35 program office has adequate knowledge to begin Block 4 development. We maintain, however, that completing its business case before awarding its Block 4 development contracts would put DOD and the program in a better position to effectively and successfully develop Block 4 capabilities. As we reported in April 2019, the program has made slow, consistent progress in improving the F-35’s R&M metrics’ performance but half of the metrics are not achieving targets. All F-35 variants are generally performing near or above targets for four of the eight R&M metrics, while still falling short for the other four. Each F-35 aircraft variant is measured against eight R&M metrics, four of which are in part of the contract. All eight R&M metrics are described in the program’s Operational Requirements Document (ORD)—the document that outlines the targeted performance levels for these metrics that DOD and the military services agreed the F-35 should meet in 2000. Based on our analysis, while the program is on track to meet half of the targets, the program office has not taken adequate steps to ensure the others will be met. Additionally, in December 2018, the Director, Operational Test & Evaluation reported that, although performance for the four under-performing metrics has shown slow growth over the years, none of these metrics are meeting interim goals needed to reach requirements at each variant’s maturity. Each F-35 variants’ R&M performance against these metrics is shown in table 1. Since the program began tracking R&M performance in 2009, it has seen small, annual improvements. Over the past year, all variants showed a slight improvement in targeted performance levels for one metric, the mean flight hours between failure—design controlled, but saw little or no discernable improvement for the four metrics not meeting targets. However, based on current performance, the program does not expect to meet those targets by full aircraft maturity. According to F-35 program officials, the ORD R&M metrics should be re-evaluated to determine more realistic R&M performance metrics, but the program has not yet taken actions to do so. Until the program office does so, it remains accountable for ensuring those ORD R&M metrics are achieved. In June 2018, we recommended that the F-35 program identify steps it needs to take to ensure the F-35 aircraft meet R&M requirements before each variant reaches maturity and update its R&M Improvement Program (RMIP)—DOD’s action plan for improving R&M—with these steps. DOD concurred with our recommendation but has yet to take substantive actions to address it. DOD did, however, complete 16 improvement projects since we last reported on this. Despite completing these projects, there were not significant gains in the R&M metrics not meeting targets. Program officials advised, however, that measurable improvements in R&M can take time to manifest. To speed this process, the program is accelerating planned upgrades to older aircraft where appropriate, which officials stated should translate to an overall improvement in the program’s R&M performance. The F-35 program office has estimated that implementing all of the identified improvement projects currently contained in its RMIP could result in potential life cycle cost savings of over $9.2 billion by improving the F-35’s R&M. However, we found that, as of December 2018, the guidance the F-35 program office has used to implement the RMIP does not define specific, measurable objectives for what the desired goals for the F-35’s R&M performance should be or align improvement projects with R&M goals. Furthermore, the RMIP has not been a funding priority. Federal internal control standards state that programs should define objectives when implementing programs such as the RMIP. Although the F-35 program RMIP’s guidance has a general goal of improving R&M, it does not identify achieving the eight R&M targets listed in the ORD as an objective. Program officials acknowledged that the RMIP’s guidance does not include such an objective. Instead, officials stated they are using the RMIP to prioritize and fund projects that will improve aircraft availability and mission capability—neither of which are included in the eight R&M metrics, but are necessary and important initiatives. The program is focusing on these two areas in part because a September 2018 memorandum from the Secretary of Defense to the Secretaries of the military departments included a goal for the F-35 fleet to attain a mission capable rate of 80 percent by the end of fiscal year 2019. According to program officials, improving these two areas will translate into improvements in the F-35 overall R&M. However, we found that the RMIP’s guidance does not discuss these priorities or align how any improvement projects would ensure targets under all eight R&M targets will be met. In our prior work on weapon system acquisitions, we have identified a number of best practices for improving program outcomes if implemented, such as clearly establishing well-defined requirements and securing stable funding that matches resources to requirements. We found that the program office has not prioritized or dedicated funding in its budget to improve R&M, in part because program officials explained that they were focused on initiatives intended to lower the cost of the aircraft. In addition, any current funding for R&M improvement projects comes from the program’s operation and maintenance funds, which are only available for one fiscal year. Officials explained that, if the funding runs out or is used by the program for other efforts, then R&M projects will go unfunded or be suspended until new funding is available. In fiscal year 2018, for example, while some projects were completed, several other projects were suspended when that year’s funding ran out. As of December 2018, according to a contractor representative, all of the identified improvement projects currently unfunded in the program’s RMIP would cost about $30 million to implement, but were not funded. Program officials also stated that they are in the process of revising the RMIP and have considered including more specific objectives in addition to improving aircraft availability and mission capability, such as more focus on improving R&M performance where ORD R&M targets are not currently being met. According to the program, any revisions to the RMIP and changes to how it will be funded, however, will not be complete until April 2019. By not defining objectives in its RMIP guidance for meeting all eight R&M metrics, aligning which improvement projects will ensure those metrics are met, and prioritizing funding for those projects, the program is at risk of not fully meeting its R&M targets. As a result, the warfighter may accept aircraft that are less reliable than originally planned, and whose operation and sustainment costs may raise affordability questions. In addition, the military services recently identified the need to cut sustainment costs—by 43 percent in the case of the Air Force—to improve the F-35’s affordability in sustainment. Increasing costs from less reliable aircraft will add strain to an already unaffordable program. To address these issues, in April 2019, we recommended to DOD that it should ensure that the F-35 program office 1. assess whether the ORD R&M targets are still feasible and revise the 2. as it revises its RMIP, identify specific and measurable R&M objectives in its RMIP guidance; 3. as it revises its RMIP, identify and document which RMIP projects will achieve the identified objectives of the RMIP guidance; and 4. prioritize funding for the RMIP. DOD concurred with these recommendations and stated that it will take actions to address them. We have previously reported on the F-35 program’s rising estimated sustainment costs and challenges maintaining an expanding fleet. In October 2017, we reported that estimated F-35 life-cycle sustainment costs increased by 24 percent from fiscal years 2012 through 2016 due to an increase in projected flying hours and other factors. We also reported that sustainment costs were not fully transparent to the military services. For example, the Marine Corps received an initial funding requirement for fiscal year 2017 sustainment of $293 million, which then increased to $364 million in the execution without a full explanation from the program office. We recommended that DOD take steps to improve communication with the services and provide more information about how F-35 sustainment costs they are being charged relate to the capabilities received. DOD concurred with the recommendation and has begun taking actions to address it. In addition, DOD faces substantial supply chain challenges that are lowering F-35 aircraft performance. In April 2019, we reported that F-35 aircraft performance was falling short of warfighter requirements—that is, aircraft could not perform as many missions or fly as often as required. Specifically, F-35A aircraft were mission capable only 52 percent of the time from May through November 2018—far short of the 80 percent target set by the former Secretary of Defense. This lower-than-desired aircraft performance is due largely to F-35 spare parts shortages and limited part repair capabilities. For example, during this time period, F-35 aircraft were unable to fly about 30 percent of the time due to spare parts shortages. Additionally, DOD’s capabilities to repair F-35 spare parts at its depots are years behind schedule, which has resulted in an average of 188 days to repair an F-35 part and a backlog of about 4,300 spare parts awaiting repair at military depots or manufacturers. We also reported that DOD faces challenges managing, moving and maintaining accountability of F- 35 parts within the supply chain. We made eight recommendations to DOD, including that DOD determine what actions are needed to close the gap between warfighter requirements for aircraft performance and F-35 supply chain capabilities. DOD concurred with the recommendations and identified actions that it was taking or planned in response. Finally, the F-35’s Autonomic Logistics Information System (ALIS) has the potential to lead to increased costs for the program if key issues are not addressed. ALIS is the F-35’s central logistics system intended to support operations, mission planning, supply-chain management, maintenance, and other processes. In April 2016, we identified several risks, including that ALIS (1) was not initially designed to be deployable, (2) lacked redundant infrastructure, (3) did not communicate well with legacy aircraft systems, (4) had data accuracy and accessibility issues, and (5) had security risks. In addition, DOD had not included certain analyses and information, such as historical cost data, to increase the credibility and accuracy of ALIS’s estimated costs. Further, a 2013 DOD-commissioned study found that schedule slippage and functionality problems with ALIS could lead to between $20 billion and $100 billion in additional costs. We have made several recommendations to DOD to improve ALIS planning and cost estimates, and to develop a performance measurement process for ALIS to better address problems based on actual system performance and user requirements. DOD generally concurred with our recommendations and has taken some actions, including developing a plan that identifies and prioritizes key ALIS risks. However, more work remains. We are currently conducting a review examining DOD’s progress in implementing our ALIS-related recommendations, addressing concerns from ALIS users, identifying emergent financial and operational risks associated with ALIS, taking near-term actions to improve ALIS functionality, and assessing DOD’s actions regarding the long-term viability of ALIS to ensure capable sustainment of the F-35 fleet. We plan to issue a report based on our current work later in 2019. Based on our ongoing work, ABMS is early in the acquisition process, as the specific capabilities and overarching acquisition strategy are still to be determined by the Air Force. As a result, the Air Force has not yet established a cost and technical baseline for ABMS. When ABMS planning began in 2017, program officials stated that the intent of the program was to replace and modernize the capabilities of the AWACS system—which provides the warfighter with the capability to detect, identify, and track airborne and maritime threats. But changes in Air Force expectations for how it would fight during future conflicts led the department to assess options for developing a more robust and survivable air, land, and sea battle management system that can operate in contested environments. In July 2018, the ABMS Initial Capabilities Document—which describes capability needs and associated gaps—was approved by the DOD Joint Requirements Oversight Council. Our ongoing work also found that, in December 2018, the Air Force determined it would not continue its planned JSTARS Recapitalization program—which was intended to provide surveillance and information on moving ground targets—well into the future, as initially expected. As a result of a recent study, the Air Force has extended the estimated service life of the JSTARS fleet, and will incorporate its capabilities into the ABMS in the short term, and retire JSTARS in the 2030s. Our preliminary observations indicate that the details about ABMS are still to be determined. The Air Force expects to fully define ABMS through an Analysis of Alternatives (AOA) that it plans to complete by the summer of 2019, as shown in figure 2. The ABMS AOA, led by the Air Force’s Air Combat Command, will assess how ABMS will deliver air-centric capabilities, such as those currently provided by AWACS. Air Force officials explained that they plan to utilize an existing AOA completed for the JSTARS Recapitalization program, approved in May 2012, to identify and assess ABMS’s potential ground target tracking capabilities. Originally planned as a 9-month study, Air Force officials stated that the ABMS AOA was shortened to a 6-month effort. As a result, the Air Force received conditional approval to reduce the number of alternatives studied from five to three. Our ongoing work indicates that the Air Force plans to develop ABMS over three phases. The first phase began in fiscal year 2018 and goes through 2023. In this phase, the Air Force plans to integrate existing sensors, improve battle management systems, and upgrade communication networks across 10 existing acquisition programs. Table 2 includes information on three existing programs the Air Force plans to enhance during the first phase of ABMS. According to an Air Force acquisition official, the technologies associated with the first phase are considered to be mature but there may be risks as the Air Force integrates technologies. Air Force officials explained that their approaches to the second and third phases of ABMS are not fully developed, but noted that the phases would be informed by the AOA results. That said, the Air Force expects to start phase 2 in 2024 by integrating advanced sensors and software into its existing battle management command and control platforms while at the same time retiring JSTARS. Air Force officials have reported that the third phase, planned for the mid-2030s, is expected to provide multi-sensor, resilient battle management command and control capability using multiple types of communications methods, with an initial operational capability planned for 2035. The Air Force estimates that ABMS’s acquisition cost through fiscal year 2024 will be $3.8 billion. Because ABMS is composed of many different defense acquisition programs, the Air Force intends to manage it as a family of systems directed by a Chief Architect and not a traditional acquisition program manager. According to the Air Force, the ABMS Chief Architect is the first of its kind, and the Air Force believes the position will be instrumental in integrating the various programs and technologies into an overall system. Based on our preliminary analysis, the roles and responsibilities of the Chief Architect have not been fully defined. However, according to the Air Force, the Chief Architect is expected to be responsible for (1) leading a high-level analysis and determining the overall design of ABMS, (2) coordinating with the service-level commands and the acquisition programs involved to make sure they are aligned with the ABMS development, and (3) identifying the enabling technologies for integration into ABMS. Chairman Norcross, Ranking Member Hartzler, and members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions you may have. We look forward to continuing to work with the Congress as we to continue to monitor and report on the progress of the F-35 program and the ABMS. If you or your staff have any questions about this testimony, please contact Michael J. Sullivan at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement are Justin Jaynes (Assistant Director), Diana Maurer, Jennifer Baker, Desirée E. Cunningham, Alissa Czyz, Stephanie Gustafson, Kasea Hamar, Jeff Hubbard, Jessica Karnis, Matt Metz, Robin Wilson, and Lauren Wright. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In 2018, the F-35 program began operational testing. Also in 2018, the Air Force continued planning for the acquisition of ABMS, intended to modernize how DOD maintains command and control over and manages the future battlefield. Both the F-35 and ABMS are expected to play key roles in DOD's modernization efforts. This testimony statement discusses (1) the F-35 program's development and modernization efforts, and progress in improving the aircraft's R&M and (2) DOD's current planning efforts for ABMS. This statement is based on two GAO reports on the F-35 published in April 2019 and on GAO's ongoing work examining ABMS. To conduct this work, GAO analyzed DOD management reports; discussed the efforts with program and contractor officials; and compared both efforts to DOD policy and GAO acquisition best practices. The Department of Defense (DOD) wrapped up the F-35 development program in April 2018 and expects to complete operational testing in December 2019. DOD has turned its attention to modernization efforts—referred to as Block 4—to add new capabilities to address evolving threats. The program office estimates Block 4 to cost at least $10.5 billion through 2024. DOD plans to start Block 4 development without a complete business case identifying baseline cost and schedule estimates. Key documents for establishing a business case, such as an independent cost estimate, will not be ready before the program plans to award Block 4 development contracts in May 2019 (see figure). Without a business case—consistent with acquisition best practices—program officials cannot be confident that the risk of committing to development has been reduced adequately prior to planned contract awards. The program made slow, sustained progress in improving the F-35's reliability and maintainability (R&M). F-35 aircraft are assessed against eight R&M metrics, which inform how much time the aircraft will be in maintenance rather than operations. Half of these metrics are not meeting targets. While the program office has a plan for improving R&M, its guidance is not in line with GAO's acquisition best practices or internal control standards as it does not include specific, measurable objectives, align improvement projects to meet those objectives, and prioritize funding to match resources to R&M requirements. If the R&M requirements are not met, the warfighter will have to settle for a less reliable and more costly aircraft than originally planned. This contributes to the F-35's $1.12 trillion estimated sustainment costs and challenges with maintaining an expanding fleet that also has supply chain and logistics system problems. GAO's ongoing work indicates that the Air Force's Advanced Battle Management System (ABMS)—intended to provide battle management command and control and surveillance across air, land, and sea—is in the early stages of planning. The capabilities and the strategy to deliver those capabilities are still to be determined. The Air Force plans to manage ABMS as a family of systems, integrating sensors from existing and future weapons programs, and overseen by a Chief Architect—whose role is still to be determined. The Air Force expects to further define ABMS after analyzing different options for delivering the capability. That analysis is expected to be complete in summer 2019. In April 2019, GAO recommended that the F-35 program office complete its Block 4 business case before making more contract awards. DOD did not concur, citing that it has adequate knowledge to begin Block 4 development. GAO maintains that completing its business case before awarding its Block 4 development contracts would put DOD and the program in a better position to successfully develop Block 4 capabilities. GAO also recommended that DOD take action to improve its R&M performance. DOD concurred and noted the actions it would take.", "document_type": "gao"}
{"report": "A skilled acquisition workforce is vital to maintaining military readiness, increasing the department’s buying power, and achieving substantial long-term savings through activities such as systems engineering and contract administration. As of September 2018, DOD’s civilian acquisition workforce was comprised of about 157,000 civilian personnel (see figure 1). About 60 percent of DOD’s civilian acquisition workforce personnel held positions in 3 of 14 acquisition career fields: engineering, contracting, and life cycle logistics (see table 1). We have previously found that DOD has faced various challenges in growing and sustaining its acquisition workforce, including challenges with hiring, recruiting, and retaining personnel. In December 2015, we found that over the past 20 years, DOD has significantly reduced and then subsequently increased the size of its acquisition workforce. During the 1990s, as defense budgets decreased, DOD reduced the size of its military and civilian acquisition workforce, and by the early 2000s it began relying more heavily on contractors to perform many acquisition support functions. According to DOD, between 1998 and 2008, the number of military and civilian personnel performing acquisition activities decreased 14 percent, from about 146,000 to about 126,000 personnel. Due to concerns about skill gaps within the workforce and growing reliance on contractors, the Secretary of Defense announced his intention in 2009 to rebalance the workforce mix. In 2010, DOD issued an acquisition workforce plan that specified DOD would add 20,000 military and civilian personnel to its acquisition workforce by fiscal year 2015. DOD subsequently increased the size of its military and civilian acquisition workforce by 21 percent between 2008 and 2015 to about 153,000 personnel, but did not accomplish growth goals set for certain priority career fields, such as engineering and contracting. DOD officials stated that the shortfalls were largely the result of high attrition rates, difficulty hiring qualified personnel, and budget constraints. In May 2018, we found that DOD’s Science and Technology Reinvention Laboratories (defense laboratories), which include acquisition workforce personnel, experienced challenges with delays with security clearances and human resource processing of personnel actions, which contributed to a lengthy hiring process. We also found that the delays made it difficult for defense laboratories to hire highly qualified candidates. Similarly, in June 2018, the Section 809 Panel identified DOD’s cumbersome hiring process as a challenge for shaping its acquisition workforce. The Section 809 Panel emphasized that these challenges undermine DOD’s ability to successfully recruit top candidates into the acquisition workforce. Most recently, in March 2019, we reported that DOD had not developed metrics to track progress associated with shaping the future acquisition workforce, such as workforce targets as a whole or by specific career fields. For example, we reported that DOD issued an updated acquisition workforce strategic plan in October 2016 which, among other things, assessed its current capacity and capability, and identified risks that DOD needed to manage to meet future needs. In addition, in September 2017, DOD issued a workforce rationalization plan. However, neither the October 2016 strategic plan nor the September 2017 workforce rationalization plan established specific size targets. We noted that without such metrics, DOD would not be able to demonstrate that its strategic workforce planning efforts and associated initiatives were successful, despite increasing the size of its acquisition workforce beyond its earlier target. DOD’s challenges with hiring civilian acquisition workforce personnel are not unique within the federal government. The traditional method of hiring for the federal government, also known as the competitive examining process, has been characterized by federal agencies as rigid and lacking in flexibility. The traditional hiring method generally requires agencies to, among other things, notify the public that the government will accept job applications for a position, screen applications against minimum qualification standards, apply selection priorities such as veterans’ preference, and assess applicants’ relative competencies or knowledge, skills, and abilities against job-related criteria to identify the most qualified applicants. In 2008, OPM established a roadmap for the traditional hiring method, including an 80-day time-to-hire goal (see figure 2). To address some of the human capital challenges federal agencies face, statutes have provided hiring, recruitment, and retention flexibilities that provide agencies with tools to help manage their workforces. Legislation has also provided hiring flexibilities exclusively to DOD for specified purposes, including hiring acquisition workforce personnel. Hiring flexibilities can help the government fill critical skills gaps or achieve certain public policy goals, such as employing veterans. As of September 2018, we identified 46 hiring flexibilities that DOD could use to hire civilian acquisition workforce personnel, including the following. DOD Direct Hire Authorities. These authorities help expedite the hiring process by allowing DOD to hire candidates without regard to certain provisions in Title 5, such as veterans’ preference and applicant rating and ranking. According to DOD officials, using direct hire authorities can reduce the time to hire personnel by nearly half as compared to the traditional hiring method. We identified 14 DOD direct hire authorities in effect as of fiscal year 2018 that DOD could use to hire civilian acquisition workforce personnel. For example, the “expedited hiring authority for certain defense acquisition workforce positions” (expedited hiring authority for acquisition positions) permits the Secretary of Defense to determine that a shortage of candidates or a critical hiring need exists for certain acquisition workforce positions, and to recruit and appoint qualified persons directly to those positions. For the purposes of the expedited hiring authority, in December 2015, the Secretary of Defense had designated 12 of the 14 acquisition career fields as critical or understaffed, including the engineering, contracting, and life cycle logistics career fields. DOD Civilian Acquisition Workforce Personnel Demonstration Project (AcqDemo) Hiring Authorities. According to HCI officials, about 19 percent of DOD’s civilian acquisition workforce personnel participate in the AcqDemo performance management system, an alternative to the General Schedule pay system. Hiring managers under AcqDemo may use AcqDemo-specific hiring flexibilities, such as direct hire appointments for the business and technical management professional career path, in addition to hiring flexibilities available DOD-wide. Veterans-Related Hiring Authorities. These authorities allow agencies to hire certain veterans without regard to certain provisions in Title 5. For example, agencies may appoint eligible veterans under the Veterans’ Recruitment Appointment authority without competition under limited circumstances or otherwise through excepted service hiring procedures. Pathways Programs. These programs promote employment opportunities in the federal government for students and recent graduates through an exception to the competitive hiring rules for certain positions in the federal workforce. Appendix I provides additional information on the hiring flexibilities that were available to DOD’s civilian acquisition workforce as of September 2018. Sections of Title 5 outline recruitment and retention flexibilities that agencies can offer to prospective and current employees to help recruit and retain highly qualified personnel. Like other federal agencies, DOD can use these incentives to recruit and retain civilian personnel, including those in the acquisition workforce. Recruitment bonuses may be paid to a newly hired federal employee if the agency determines that the position would be difficult to fill in the absence of a bonus. Relocation bonuses may be paid to a current employee who must relocate for a position in a different geographic area if the agency determines that the position would be difficult to fill in the absence of a bonus. Retention bonuses may be paid to a current employee if the agency determines that the unusually high or unique qualifications of the employee or a special need of the agency for the employee’s services makes it essential to retain the employee who would likely leave federal service in the absence of such a bonus. Student loan repayments may be paid on behalf of a job candidate or a current employee to recruit or retain highly qualified personnel. The employees must sign a service agreement of at least 3 years with the agency that pays the loans. Federal agencies may pay up to $10,000 per employee per calendar year, totaling no more than $60,000 for any one employee. DOD can fund the four monetary incentives with the Defense Acquisition Workforce Development Fund (DAWDF)—a dedicated funding source for the recruitment, training, and retention of DOD’s acquisition personnel— as well as other sources, such as operations and maintenance appropriations. Appendix II provides additional information on the recruitment and retention flexibilities available to DOD’s civilian acquisition workforce as of September 2018. Several offices within DOD play key roles in managing how the department uses hiring, recruitment, and retention flexibilities for the civilian acquisition workforce. For example, HCI oversees department- wide acquisition workforce strategic planning; DCPAS develops implementation guidance on how DOD flexibilities should be used; and civilian personnel centers track the extent to which flexibilities are used (see table 2). From fiscal year 2014 to 2018, DOD increased its use of hiring, recruitment, and retention flexibilities for its civilian acquisition workforce. During this period, DOD used hiring flexibilities for 90 percent of its approximately 44,000 civilian acquisition workforce hiring actions. This high usage rate came as USD (A&S), USD (P&R), and the military departments’ leadership encouraged their hiring managers and human resource specialists to use the hiring flexibilities to help reduce the length of the hiring process. Additionally, during this period, DOD’s human resource specialists issued guidance that helped address confusion about the requirements governing the hiring authorities. Currently, USD (P&R) is leading a DOD-wide effort to consolidate direct hire authorities in an attempt to simplify their application. During this 5-year period, DOD also increased its use of recruitment and retention flexibilities for the civilian acquisition workforce, increasing the dollar amount authorized for these flexibilities from $13.9 million in fiscal year 2014 to $33.7 million in fiscal year 2018. This increase came as DOD leadership emphasized the benefits of recruitment and retention flexibilities and oversaw concerted efforts to increase their usage through the dissemination of information to human resource specialists. We found that DOD used hiring flexibilities for about 90 percent of DOD’s approximately 44,000 civilian acquisition workforce hiring actions between fiscal years 2014 and 2018. Further, DOD increased its use of these flexibilities, which include direct hire authorities, from 80 percent in fiscal year 2014 to 95 percent in fiscal year 2018 (see figure 3). From fiscal year 2014 to 2018, DOD used the expedited hiring authority for acquisition positions more than any other direct hire authority for its civilian acquisition workforce. Congress enacted this authority in fiscal year 2009 and in fiscal year 2010, amended the authority to, among other things, allow hiring of all qualified applicants instead of only highly qualified applicants. Additionally, in November 2015, legislation eliminated the expedited hiring authority’s expiration date and made the authority permanent. Command officials told us that they used this authority often because it does not have as many requirements as other direct hire authorities and because of their familiarity with it. Nine of the 14 DOD direct hire authorities identified were not available for use until fiscal year 2017 because they were enacted after that year or DOD had not yet implemented the authority, either through memorandums or federal register notices (see table 3). Since 2015, USD (A&S), USD (P&R), and leadership of the military departments have encouraged the use of hiring flexibilities—particularly direct hire authorities—over the traditional method. From July 2015 to November 2017, USD (A&S) and USD (P&R) convened five joint acquisition and human resource summits to provide a recurring forum for discussing leading practices in sustaining the acquisition workforce, including the improved use of hiring flexibilities. In October 2016, USD (A&S) issued its current acquisition workforce strategic plan for DOD and used this document to encourage implementation of direct hire authorities as appropriate. In November 2017, senior leadership in the USD (P&R) office issued a Federal Register Notice that updated and consolidated AcqDemo’s rules and guidance, including introducing additional AcqDemo hiring flexibilities (see appendix I, table 8 for additional information on these flexibilities). In 2018, the Secretary of the Navy, the Assistant Secretary of the Army for Manpower and Reserve Affairs, and the Assistant Secretary of the Air Force for Manpower and Reserve Affairs each issued memorandums to their respective departments stating that the use of direct hire authorities be considered first in the hiring process, as appropriate. These memorandums note that direct hire authorities provide significant advantages in timeliness compared to the traditional hiring process, and encourage maximum use of direct hire authorities to the extent appropriate. In addition to DOD leadership emphasis, command officials credited DCPAS and the military departments’ civilian personnel centers for taking actions to help DOD increase its use of direct hire authorities. These officials explained that confusion among hiring managers and human resource specialists over the numerous requirements that apply to each direct hire authority constituted one of the main challenges that had previously limited DOD’s use of direct hire authorities. To illustrate the potential for confusion, table 4 presents some of the direct hire authority requirements a hiring manager would have to consider under two different hiring authorities. To help address the confusion stemming from the direct hire authorities’ numerous requirements, in 2017 and 2018, DCPAS and the personnel centers consolidated information on the available direct hire authorities and the requirements that govern each of them into concise and comprehensive guidance documents. As a result, command-level and personnel center officials told us that human resource specialists can now quickly find and compare available direct hire authorities to determine what may work best for their hiring needs. We found that the military departments’ use of certain direct hire authorities was limited by the amount of time it took DOD leadership to implement some of the authorities. DCPAS officials told us that although Congress enacts direct hire authorities in legislation, DOD human resource personnel and hiring managers do not use the authorities until DOD and the components issue implementing guidance. We found that DOD implemented the 14 DOD direct hire authorities anywhere from 2 to 42 months after an authority’s enactment (see figure 4). In May 2018, we reported on the 30-month lapse between the enactment of the science, technology, engineering, and mathematics direct hire authority for students at the defense laboratories and DOD’s issuance of corresponding implementation guidance. Defense laboratory officials told us it took longer than anticipated to publish the federal register notice that allowed the laboratories to use the hiring authority, and they attributed the delays to coordination issues among relevant offices during the approval process. In December 2018, we found that the defense laboratories hired significantly fewer students than authorized because of the delays. To address the delays, in May 2018, we recommended that DOD establish and document time frames for its coordination process to help ensure the timely implementation of defense laboratory hiring authorities in the future. DOD concurred with our recommendation and identified actions the department plans to take to improve oversight and coordination of the defense laboratories’ hiring efforts. DOD acquisition workforce and human resource officials told us that they also did not use certain direct hire authorities as much from fiscal years 2014 to 2018 because the requirements associated with them made them harder to use. For example, according to DOD guidance documents we reviewed, most of the DOD direct hire authorities applicable to the civilian acquisition workforce have expiration dates or limits on the number of hires. Table 5 provides examples of requirements governing direct hire authorities that officials identified as making the authorities more difficult to use. Going forward, HCI and DCPAS officials told us that USD (P&R) is leading a DOD-wide effort to advise Congress on which direct hire authorities could be consolidated and which requirements could be eliminated. For example, HCI officials said that USD (P&R) recently provided Congress input on consolidating four cybersecurity-related authorities into one authority. DCPAS officials also told us they previously provided input to Congress on certain challenges hiring managers experienced in using some of the direct hire authorities. According to command officials, DCPAS recommended that Congress raise the limits on the number of personnel that could be hired under the defense laboratory direct hire authorities. We found that DOD increased its use of recruitment and retention flexibilities from fiscal year 2014 to fiscal year 2018. We also examined two other issues related to recruitment and retention—post-employment restrictions on military personnel and authorities to remove civilian acquisition workforce employees for unacceptable performance. DOD officials did not identify either issue as a major challenge for managing the civilian acquisition workforce. We found that DOD increased its use of recruitment bonuses, relocation bonuses, retention bonuses, and student loan repayments from $13.9 million in fiscal year 2014 to $33.7 million in fiscal year 2018 (see figure 5). As part of the increased total amount of funds authorized for recruitment and retention flexibilities, DOD had increased the number of awarded recruitment and retention flexibilities by approximately 140 percent between fiscal years 2014 to 2018 (see table 6). DOD leadership has emphasized the benefits of recruitment and retention flexibilities, which helped increase their use. For example, in DOD’s October 2016 acquisition workforce strategic plan, USD (A&S) stated that the acquisition workforce would increase the use of these flexibilities by leveraging DAWDF. Additionally, in November 2016, USD (A&S) and USD (P&R) held a joint acquisition and human resource summit, which highlighted efforts of an integrated product team established to expand the use recruitment and retention flexibilities. For example, the integrated product team developed a fact sheet to answer frequently asked questions about incentives from human resource specialists and hiring managers within the military departments. Officials from the commands and DACMs generally agreed that recruitment and retention flexibilities were useful tools in helping them recruit and retain acquisition workforce talent. To receive these monetary incentives, employees must enter into written service agreements to remain with the department for a specific period. DACM and defense agency officials, however, noted that retention bonuses were the least effective of the monetary recruitment and retention flexibilities. For example, Defense Contract Management Agency and Air Force officials told us they do not use retention bonuses as widely because they do not view them as effective tools in retaining talent. Defense Contract Management Agency officials explained that most of the personnel who leave their agency for other jobs go to other DOD organizations or federal agencies, and retention bonuses are generally used only for employees who are likely to leave federal service. The Air Force DACM’s representatives told us the Air Force decreased its use of DAWDF for retention bonuses as a result of a 2016 RAND Corporation study that found that private sector companies made minimal use of retention bonuses. According to this study, none of the 21 companies in RAND’s sample—among Fortune’s “100 best companies to work for”—identified retention bonuses as a primary tool to retain talent. Lastly, command and personnel center officials we interviewed also noted that a number of factors outside of monetary recruitment and retention flexibilities influence an employee’s decision to join or remain with DOD. These factors include the organization’s mission and work-life programs and policies. DOD military personnel are subject to certain post-employment restrictions that could potentially dissuade them from seeking further employment with the department as civilian personnel, but DACM and command officials told us these restrictions have not significantly affected their ability to recruit new hires. For example, the “180-day rule” prevents DOD from appointing retired military personnel to civil service positions within 180 days of the military person’s retirement unless the appointment, which must be in the competitive service, is authorized by the Secretary of Defense or a designee, OPM approves the appointment, and the minimum rate of basic pay has been increased. DACM and command officials noted that retired military personnel could elect to work for private sector companies during the 180-day period. However, these officials did not cite post-employment restrictions as a major recruitment challenge for the civilian acquisition workforce and instead cited other challenges, such as limited resources dedicated to recruiting civilian personnel and hiring delays due to the security clearance process. DOD does not have specific statutory authority in Title 10, U.S. Code for removing civilian acquisition workforce personnel for unacceptable performance. However, DOD’s civilian employees are subject to a longer probationary period than other civilian federal employees, and DACM and command officials told us that removing underperforming staff is easier during a probationary period than when staff are permanently employed. Officials also noted that staff tend to leave on their own accord if they are not performing well within the department. DOD does not regularly monitor its use of hiring, recruitment, and retention flexibilities for its civilian acquisition workforce, and despite ongoing efforts, is not yet able to systematically assess the effectiveness of these flexibilities. HCI, the office responsible for DOD-wide acquisition workforce strategic planning, regularly monitors the overall health of the acquisition workforce, in part by reviewing and reporting statistics on workforce size, workforce gains and losses, and other workforce-related metrics on a quarterly basis. Further, as previously noted, DOD has increased its overall use of human capital flexibilities. However, HCI does not regularly monitor the military departments and defense components’ specific use of hiring, recruitment, and retention flexibilities. As a result, HCI is missing opportunities to identify variations in usage rates, and use that information to determine whether there are specific issues or challenges being encountered. For example, we found that the Air Force and Navy used direct hire authorities twice as often as the Army in fiscal year 2018. Further, while DOD leadership has emphasized that using hiring flexibilities improves DOD’s ability to recruit and hire high-quality talent in a timely manner, HCI is not yet able to assess how effective the hiring flexibilities have been in achieving these goals. This is because DCPAS has not yet developed a plan to consistently measure how long it takes to hire new personnel across the department. Similarly, DCPAS has not yet established metrics to assess the quality of the new personnel DOD hires. DCPAS has efforts underway to address these issues and plans to start using these metrics in 2019. DOD policy states HCI should implement strategies and policies to help attract and retain acquisition workforce personnel. To this end, HCI officials told us they monitor the overall health of the acquisition workforce in various ways, including outreach to the DACMs on workforce challenges, as well as holding knowledge-sharing events, such as a May 2018 acquisition workforce human capital symposium. Additionally, HCI reviews and reports statistics on workforce size, workforce gains and losses, and other workforce-related metrics on a quarterly basis. For example, in its fiscal year 2019 first quarter assessment, HCI reported data on the current size of the acquisition workforce; acquisition workforce education and certification levels; and workforce gains, losses, retirement eligibility and attrition rates, among other things, both on a DOD-wide basis as well as by acquisition career field. HCI officials told us they use these data to identify potential or emerging workforce challenges. HCI officials noted that if they identify any issues, they further analyze data to identify the root cause of the issues. HCI officials acknowledged, however, that HCI does not regularly collect or review data on the defense components’ specific use of hiring, recruitment, and retention flexibilities as part of its quarterly assessments. HCI officials stated that they collect and review hiring flexibilities data on an as-needed basis, such as in preparation for DOD acquisition workforce governance forums, including senior steering board and workforce management group meetings, in which the use of the flexibilities will be on the agenda, or in response to Congressional requests. HCI officials also noted that because the use of hiring, recruitment, and retention flexibilities are made at the command level within the military departments, the military departments are better positioned to regularly monitor usage. However, the military departments are not in a position to identify variations in usage rates across DOD’s civilian acquisition workforce, which are significant. For example, we found that the Air Force and the Navy used direct hire authorities for 85 percent and 84 percent of their respective hiring actions in fiscal year 2018, while the Army used direct hire authorities for 42 percent of its hiring actions that year. Similarly, some career fields use the hiring flexibilities at higher rates than others. While hiring flexibilities comprised 95 percent of total civilian acquisition workforce hiring actions in fiscal year 2018, the auditing and purchasing career fields used hiring flexibilities for only 68 percent and 62 percent of their respective hiring actions that year. Without regularly monitoring usage rates for hiring flexibilities across the civilian acquisition workforce, HCI lacks visibility into these types of variations and opportunities to investigate and address them, as appropriate. Lastly, HCI focuses its efforts on those recruitment and retention flexibilities funded by DAWDF because HCI is responsible for DAWDF’s management. Based on DAWDF reports and DCPDS data, we found that the amount of dollars obligated for DAWDF-funded recruitment, retention, and recognition initiatives in 2017 was $15 million or about two-thirds of the total dollars authorized for the recruitment and retention flexibilities for DOD’s civilian acquisition workforce in fiscal year 2017. The remaining amount (about one-third) was funded by other sources, such as the military departments’ operations and maintenance appropriations, but is not included as part of HCI’s annual review. Since 2002, we have repeatedly found that agencies should strategically manage their use of human capital flexibilities—including hiring, recruitment, and retention flexibilities—to address human capital challenges. Additionally, federal internal control standards state that an agency’s management should obtain relevant data on a timely basis to effectively monitor activities to achieve objectives. Based on these standards, in May 2018, we recommended that DOD’s defense laboratories routinely monitor data on its use of hiring authorities. DOD concurred with our recommendation and planned to determine the appropriate data to be collected and establish routine reporting requirements. Because HCI is not regularly reviewing hiring flexibility usage, it may be similarly missing opportunities to help identify challenges, inconsistencies, or needed improvements in using the flexibilities. DOD leaders have repeatedly emphasized that hiring flexibilities— particularly direct hire authorities—can help the department hire high- quality talent in a more timely manner. We have previously found that time-to-hire and quality-of-hire are useful metrics that help agencies evaluate their hiring efforts, which can include the use of hiring flexibilities. To this end, DCPAS collects and reports time-to-hire data to measure DOD’s progress in improving hiring practices. For example, according to DCPAS, from fiscal year 2016 through 2018, DOD took an average of 127 days to hire civilian personnel under the traditional hiring method compared to an average of 110 days when using DOD direct hire authorities. DCPAS also noted variations in time-to-hire across the direct hire authorities, reporting that DOD took anywhere from 77 to 111 days to hire civilian personnel using the 14 DOD direct hire authorities applicable to the civilian acquisition workforce. For the expedited hiring authority for acquisition positions—the direct hire authority used most frequently to hire civilian acquisition workforce personnel—DCPAS reported an average time-to-hire of 106 days from fiscal year 2016 through 2018. While these time-to-hire metrics could be helpful in determining which direct hire authorities most effectively expedite the hiring process, HCI officials told us they do not use these metrics to inform management decisions for the civilian acquisition workforce because they are not yet consistently measured. DCPAS officials explained that the military departments and their major commands developed their own approaches for inputting and reporting time-to-hire data based on their individual needs and data systems. HCI and DCPAS officials acknowledged that this resulted in different ways to record and track the data, which in turn prevented meaningful comparisons between the time-to-hire metrics produced by each of the components. According to HCI and DCPAS officials, this difference, in part, is attributable to the variation in how DOD personnel input certain data. For example, one human resource specialist may initiate a request for a personnel action—which is generally the starting date for time-to-hire metrics—on the day the hiring manager submits a job description, while another human resource specialist may initiate a request for personnel action after the job announcement has been posted publicly. Moreover, our analysis of DCPAS data for all DOD civilian personnel hires from fiscal year 2014 to 2018 shows that about 36,000 of 420,000 personnel actions, or about 9 percent, were initiated on or after the individuals’ start dates, producing a zero or negative time-to-hire figure. DCPAS officials told us they omit these figures when they report time-to-hire metrics. Until time-to-hire metrics are consistently measured DOD-wide, HCI will not be able to use this data to assess which direct hire authorities have most effectively expedited the hiring process, which DOD components have been the most successful in using these authorities, or identify potential issues in using these authorities. In September 2018, to address inconsistent time-to-hire methodologies across DOD, we recommended that the DOD Chief Management Officer require that all DOD human resource providers adopt consistent time-to- hire measures. DOD concurred with our recommendation, and in June 2019, DCPAS officials told us they were in the process of developing a plan to implement consistent time-to-hire metrics across the department. DCPAS officials anticipate completing the plan by July 2019 and will start implementation after DOD leadership approves this plan. HCI officials told us that they plan to use the time-to-hire metrics to assess the civilian acquisition workforce’s hiring efforts, including the use of flexibilities, when the metrics are comparable. Similarly, HCI officials told us that they cannot systematically assess quality-of-hire across the civilian acquisition workforce because DCPAS has not developed guidance that outlines how quality-of-hire should be measured. DOD’s June 2018 civilian human capital operating plan outlines an initiative to improve the quality of civilian hires, among other things. As part of this initiative, DCPAS is to establish quality-of-hire metrics using data collected from an OPM hiring satisfaction survey tool. Using the OPM survey, DOD’s hiring managers are to rate the performance of new employees 6 months after they are hired. DCPAS officials stated that various DOD components have used the survey since 2011, but acknowledged hiring managers completed the survey for only 1 percent of all DOD hires in fiscal year 2018. In March 2019, USD (P&R) leadership issued a memorandum to DOD human capital offices encouraging wider implementation of the survey, including outlining roles and responsibilities and milestones for implementation. Starting in fiscal year 2020, USD (P&R) plans to set quality-of-hire goals using the fiscal year 2019 survey results and incorporating these into future civilian human capital operating plans. HCI officials told us that they plan to use the quality-of-hire metrics to evaluate the civilian acquisition workforce’s hiring efforts, including the use of flexibilities, once DCPAS completes its efforts. Congress has provided DOD with a number of hiring, recruitment, and retention flexibilities to help the department manage its acquisition workforce. DOD leadership has encouraged the use of these flexibilities across the department in recent years, and usage has increased significantly since 2014. However, HCI does not regularly monitor defense components’ use of hiring, recruitment, and retention flexibilities for their civilian acquisition workforce to identify challenges, inconsistencies, or needed improvements in using these tools. As a result, HCI may be missing opportunities to develop strategies or inform efforts aimed at improving the usage of these flexibilities. The Secretary of Defense should ensure that the Director of Human Capital Initiatives regularly monitors usage of hiring, recruitment, and retention flexibilities for the civilian acquisition workforce—across the military departments and acquisition career fields—to help develop strategies or inform efforts aimed at improving the usage of these flexibilities. (Recommendation 1) We provided a draft of this report to DOD for review and comment. DOD provided written comments, which are reprinted in appendix VII, and concurred with our recommendation. In concurring with our recommendation, DOD stated it would provide guidance to DOD components to monitor usage of flexibilities and provide the results to HCI at least annually. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; the Secretaries of the Army, the Air Force, and the Navy; the Under Secretary of Defense for Acquisition and Sustainment; the Under Secretary of Defense for Personnel and Readiness; the Director of the Defense Civilian Personnel Advisory Service, and the Director of Human Capital Initiatives. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or dinapolit@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. Sections of Title 5 of the U.S. Code include, among other things, requirements that agencies must follow to hire personnel, such as those associated with the competitive examining hiring authority. Competitive examining has been the traditional method of hiring for the federal government since 1978. The traditional hiring method requires agencies to notify the public that the government will accept applications for a job; screen applications against minimum qualification standards; apply selection priorities such as veterans’ preference, where applicable; and assess applicants’ relative competencies or knowledge, skills, and abilities against job-related criteria to identify the most qualified applicants. Hiring flexibilities were established beyond the traditional hiring method to expedite the hiring process and achieve certain public policy goals. We identified 46 hiring flexibilities available to the Department of Defense’s (DOD) civilian acquisition workforce as of September 2018—3 specific to the DOD acquisition workforce; 6 specific to DOD’s Civilian Acquisition Workforce Personnel Demonstration Project; 20 available DOD-wide, including its acquisition workforce; and 17 available government-wide. Further, of the 46 hiring flexibilities, 14 are DOD direct hire authorities provided in statute that we have identified as being directly applicable to the DOD civilian acquisition workforce—3 of which are specific to the DOD acquisition workforce and 11 of which are available DOD-wide. Tables 7 through 10 provide additional information on each of the 46 hiring flexibilities and denote the 14 DOD direct hire authorities and their legal authorities. Recruitment and retention flexibilities assist federal agencies in attracting and retaining employees who possess unusually high or unique qualifications, or who fill essential needs for the agencies. Additionally, they allow agencies more control over compensation and are intended to help government compete with the private sector. We identified nine recruitment and retention flexibilities available to the Department of Defense’s (DOD) civilian acquisition workforce as of fiscal year 2018— four monetary incentives and five work-life balance policies and programs (see tables 11 and 12). Section 843 of the National Defense Authorization Act for Fiscal Year 2018 included a provision for us to review and report on the effectiveness of hiring and retention flexibilities for the Department of Defense’s (DOD) acquisition workforce, with a focus on its civilian acquisition workforce, including (a) the extent to which DOD experiences challenges with recruitment and retention of the acquisition workforce, such as post-employment restrictions; (b) a description of the hiring and retention flexibilities available to DOD to fill civilian acquisition positions and the extent to which DOD has used the flexibilities available to it to target critical or understaffed career fields; (c) the extent to which DOD has the necessary data and metrics on its use of hiring and retention flexibilities for the civilian acquisition workforce to strategically manage the use of such flexibilities; (d) an identification of the factors that affect the use of hiring and retention flexibilities for the civilian acquisition workforce; (e) recommendations for any necessary changes to the hiring and retention flexibilities available to DOD to fill civilian acquisition positions; and (f) a description of the flexibilities available to DOD to remove underperforming members of the acquisition workforce and the extent to which any such flexibilities are used. This report: (1) provides information on DOD’s use of available hiring, recruitment, and retention flexibilities for its civilian acquisition workforce personnel from fiscal years 2014 to 2018; and (2) determines the extent to which DOD has monitored and assessed its use of hiring, recruitment, and retention flexibilities for its civilian acquisition workforce. In doing so, the report addresses elements (a) through (f) identified above. To examine DOD’s use of available hiring, recruitment, and retention flexibilities for its civilian acquisition workforce personnel from fiscal years 2014 to 2018, we reviewed relevant statutes, reports, and DOD policies and guidance to identify hiring, recruitment, and retention flexibilities available to DOD’s civilian acquisition workforce. Based on our review, we identified the following hiring authorities: competitive examination, which we refer to as “the traditional hiring method,” and 46 alternatives to the traditional hiring method, which we refer to as “hiring flexibilities” for the purposes of our review. Appendix I provides additional information on these 46 hiring flexibilities. We also identified four monetary incentives and five work-life balance programs that DOD can use to recruit and retain civilian acquisition workforce personnel. We scoped our analysis to the four monetary incentives DOD can use to recruit and retain civilian acquisition workforce personnel—(1) recruitment bonuses, (2) retention bonuses, (3) relocation bonuses, and (4) student loan repayments—and collectively refer to these four incentives as “recruitment and retention flexibilities” for the purposes of our review. We focused our review on the four government-wide monetary flexibilities with personnel data in the Defense Civilian Personnel Data System (DCPDS), DOD’s central repository for civilian personnel transactions data, and required for submission to the Defense Manpower Data Center. Appendix II provides additional information on these four recruitment and retention flexibilities. We also analyzed personnel data from DCPDS. We obtained DCPDS data on hiring actions from the Office of the Under Secretary of Defense (USD) for Personnel and Readiness (P&R) – Defense Manpower Data Center. We obtained DCPDS data on dollars authorized for recruitment and retention flexibilities from USD (P&R) – Defense Civilian Personnel Advisory Service (DCPAS). We also obtained acquisition workforce data for fiscal years 2014 through 2018 from DOD’s DataMart, a central repository of acquisition workforce data, from the Defense Manpower Data Center. We analyzed the DataMart data to determine which DOD civilian personnel were in DOD’s acquisition workforce at the end of each fiscal year, the military department or organization in which these personnel worked, and the career fields in which these personnel held positions. For our analysis of hiring flexibilities, we included all hiring actions for the DOD civilian acquisition workforce with effective dates from fiscal year 2014 through 2018, except for actions with legal authority codes designated as transfers. We did not include hiring actions designated as transfers because they include hiring actions between military departments as well as transfers from outside of DOD. We excluded all of these transfer hiring actions because the data did not include enough information for us to distinguish between internal and external transfers. We identified 44,291 hiring actions for this 5-year period, and used the legal authority code data fields for each hiring action to determine the type of hiring authority or flexibility that DOD used. We analyzed DOD’s usage of hiring flexibilities from fiscal years 2014 through 2018 across each of DOD’s 14 acquisition career fields and the military departments. Of the hiring flexibilities, we focused our analysis on DOD direct hire authorities because they comprised the single largest category of hiring authorities used by the DOD civilian acquisition workforce for hiring actions from fiscal year 2014 through 2018—26,385 of 44,291 DOD hiring actions or 60 percent. DCPDS, however, did not include enough information for us to determine which specific direct hire authority DOD used for each hiring action. For these actions, the human resource specialists manually entered the details of the specific type of DOD direct hire authority they used in DCPDS. To determine the type of DOD direct hire authority used, two analysts independently reviewed each description and identified the appropriate DOD direct hire authority. For 360 of the 26,385 the hiring actions (or 1.4 percent), the descriptions did not contain enough information for us to determine the specific DOD direct hire authority used. For the purposes of our analysis, we established three categories of hiring actions based on the DOD’s designations in DCPDS (see table 13). For our analysis of recruitment and retention flexibilities, we included all actions authorizing recruitment bonuses, retention bonuses, relocation bonuses, and student loan repayments for the DOD civilian acquisition workforce from fiscal year 2014 through 2018. We identified 13,643 authorization actions. We used the award amount data field for each authorization action to determine the amount of dollars authorized for these four types of incentives. We analyzed DOD’s usage of the recruitment and retention flexibilities from fiscal years 2014 through 2018 across each of DOD’s 14 acquisition career fields. We assessed data reliability by electronically testing these data, reviewing relevant data standards and guidance, and interviewing DCPAS and Defense Manpower Data Center officials. We determined that the data were sufficiently reliable for the purposes of reporting the frequency with which DOD’s civilian acquisition workforce used hiring, recruitment, and retention flexibilities for fiscal years 2014 through 2018. We also identified factors that affected DOD’s use of hiring, recruitment, and retention flexibilities for its civilian acquisition workforce by reviewing DCPAS and military departments’ policies and guidance for using human capital flexibilities, including implementation of 14 DOD direct hire authorities provided in statute, and efforts by DCPAS to improve DOD’s use of the flexibilities. To determine the extent to which DOD has monitored and assessed its use of hiring, recruitment, and retention flexibilities for its civilian acquisition workforce, we reviewed acquisition workforce human capital plans from the Office of Human Capital Initiatives (HCI) within USD for Acquisition and Sustainment (A&S); acquisition workforce plans from the Air Force, the Army, and the Navy; and data and metrics collected by HCI and DOD’s four Directors for Acquisition Career Management (DACM)— one for each of the military departments and a fourth for the defense agencies and field activities outside the military departments. We assessed DOD’s efforts against our key practices for effectively managing human resource flexibilities and federal internal control standards, including that management should use quality information to achieve the entity’s objectives. We also reviewed reports by the Advisory Panel on Streamlining and Codifying Acquisition Regulations—commonly referred to as the Section 809 Panel after the legislative provision that required the Secretary of Defense to establish an advisory panel on streamlining acquisition regulations—and interviewed Section 809 Panel commissioners to supplement our analysis. For both objectives, we interviewed officials from HCI, the office responsible for DOD-wide acquisition workforce DCPAS, the office responsible for developing DOD’s civilian human resources policies and programs; the Defense Manpower Data Center, the office responsible for collecting and maintaining DOD’s civilian personnel data; the Directors for Acquisition Career Management (DACM) for each military department and the Fourth Estate, which is responsible for the 30 defense agencies and field activities outside the military departments; the Air Force Personnel Center; Army’s Civilian Human Resources Agency; Navy’s Office of Civilian Human Resources; and the command within each military department that had the largest number of civilian acquisition workforce personnel in fiscal year 2018: Air Force Materiel Command, Army Combat Capabilities Development Command, and Naval Sea Systems Command. We also interviewed officials from the Defense Contract Management Agency, which had the largest number of civilian acquisition personnel of the other defense agencies with acquisition workforce personnel. Collectively, these four organizations comprised about 38 percent of DOD’s civilian acquisition workforce in fiscal year 2018. We also interviewed personnel from the Office of Personnel Management (OPM), which is responsible for developing and promulgating government-wide human capital policies; and personnel from the Society for Human Resource Management, the world’s largest human resources membership group, who were familiar with metrics used by the private sector to monitor hiring and retention efforts. We conducted this performance audit from August 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Section 843 of the National Defense Authorization Act for Fiscal Year 2018 included, among other things, a provision for us to review the extent to which the Department of Defense (DOD) has used hiring flexibilities available to it to target critical or understaffed career fields. In its December 2015 memo on using the expedited hiring authority for certain defense acquisition workforce positions, DOD designated 12 of the 14 acquisition workforce career fields as shortage or critical needs categories. We identified 44,291 DOD hiring actions from fiscal year 2014 to 2018 for the civilian acquisition workforce, and categorized them as (1) traditional hiring method actions, (2) actions using DOD direct hire authorities, or (3) other hiring flexibilities (see appendix I for additional information on these hiring flexibilities, including DOD direct hire authorities). We further categorized the hiring actions by DOD’s 14 acquisition workforce career fields and ordered the career fields by total number of hiring actions in fiscal year 2018. Figures 6, 7, 8, and 9 provide data on the use of hiring flexibilities for each of the 14 acquisition workforce career fields. We identified 44,291 Department of Defense (DOD) hiring actions from fiscal year 2014 to 2018 for the civilian acquisition workforce, and categorized them as (1) traditional hiring method actions, (2) actions using DOD direct hire authorities, or (3) other hiring flexibilities (see appendix I for additional information on these hiring flexibilities, including DOD direct hire authorities). We further categorized the hiring actions by military departments and defense agencies. Figure 10 provides data on the use of hiring flexibilities for each of the military departments and the Fourth Estate, which is responsible for the 30 defense agencies and field activities outside the military departments. Section 843 of the National Defense Authorization Act for Fiscal Year 2018 included, among other things, a provision for us to review the extent to which the Department of Defense (DOD) has used retention flexibilities available to it to target critical or understaffed career fields. In its December 2015 memo on using the expedited hiring authority for certain defense acquisition workforce positions, DOD designated 12 of the 14 acquisition workforce career fields as shortage or critical needs categories. We identified $123.9 million authorized in recruitment and retention flexibilities for DOD’s civilian acquisition workforce from fiscal year 2014 to 2018, and categorized them as (1) recruitment bonuses, (2) relocation bonuses, (3) retention bonuses, and (4) student loan repayments (see appendix II for additional information on these recruitment and retention flexibilities). We further categorized the recruitment and retention flexibilities by DOD’s 14 acquisition workforce career fields and ordered the career fields by total dollars authorized by DOD. See figures 11 through 14 below. In addition to the contact named above, Nathan Tranquilli (Assistant Director), Claire Li (Analyst-in-Charge), TyAnn Lee, and Ashley Rawson made key contributions to this report. Lorraine Ettaro, Christopher Falcone, Lori Fields, Cynthia Grant, Laura Greifner, and Sylvia Schatz also contributed to this report.", "summary": "DOD spends over $300 billion annually on contracts for products and services such as major weapons systems and military support services. By awarding and overseeing these contracts, DOD's acquisition workforce plays a critical role in maximizing DOD's buying power. DOD has increased the size of its acquisition workforce in recent years, but has also faced a number of challenges hiring and retaining personnel. DOD has a number of human capital flexibilities that help DOD in hiring, recruiting, and retaining its civilian acquisition workforce. The National Defense Authorization Act for Fiscal Year 2018 included a provision for GAO to review DOD's implementation of human capital flexibilities for its acquisition workforce. This report: (1) provides information on DOD's use of human capital flexibilities and (2) determines the extent to which DOD has monitored and assessed its use of these flexibilities. GAO reviewed relevant statutes, and DOD policies, guidance, and acquisition workforce plans; analyzed DOD's fiscal year 2014-2018 civilian acquisition workforce personnel data; and interviewed DOD officials. The Department of Defense (DOD) has used human capital flexibilities extensively to hire, recruit, and retain its civilian acquisition workforce. Since 2014, usage rates for hiring flexibilities—alternatives to the traditional, competitive hiring process—have generally increased. DOD leadership has encouraged its hiring personnel to use these flexibilities, such as direct hire authorities, to reduce the length of the hiring process. From fiscal year 2014 to 2018, DOD used hiring flexibilities for 90 percent of its approximately 44,000 civilian acquisition workforce hiring actions (see figure). DOD also increased its use of recruitment and retention flexibilities for its civilian acquisition workforce, increasing the dollar amount authorized from $13.9 million in fiscal year 2014 to $33.7 million in fiscal year 2018. This increase came as DOD leadership emphasized the benefits of these flexibilities, and oversaw concerted efforts to increase their usage through the dissemination of information to human resource specialists. While usage of human capital flexibilities has increased, DOD's Office of Human Capital Initiatives (HCI), which is responsible for DOD-wide acquisition workforce strategic planning, does not regularly monitor or assess how the department uses these flexibilities. HCI regularly monitors the overall health of the acquisition workforce, including by reviewing workforce metrics on a quarterly basis, but does not regularly monitor the military departments' use of human capital flexibilities. For example, GAO found the Air Force and Navy used direct hire authorities twice as often as the Army in fiscal year 2018. Without efforts to gain such insights through monitoring, HCI may be missing opportunities to identify challenges, inconsistencies, or needed improvements in using these tools. With regard to assessing the use of human capital flexibilities, HCI intends to study how long it takes to hire personnel when using the flexibilities. According to DOD officials, this analysis can begin following development of a plan to ensure that defense components consistently collect data on hiring timeframes. DOD officials said they expect to issue this plan in 2019. GAO recommends that HCI regularly monitors DOD's use of human capital flexibilities for its civilian acquisition workforce to help identify challenges, inconsistencies, or needed improvements in using these tools. DOD concurred with the recommendation.", "document_type": "gao"}
{"report": "Depots are government-owned, government-operated industrial installations that maintain, overhaul, and repair a multitude of complex military weapons systems and equipment for DOD. The military services operate 17 depots that perform depot-level maintenance on a wide range of vehicles and other military equipment, including aircraft, engines, combat vehicles, ships, and software. Of those, the Air Force operates three Air Logistics Complexes and the Navy operates three Fleet Readiness Centers for aviation depot maintenance (see figure 1). For the purposes of this report, we will be referring to them as Air Force and Navy aviation depots. The Air Force’s and Navy’s aviation depots operate through the Air Force and Navy working capital funds. Depot customers are charged for the anticipated full cost of goods and services. Over the past decade, we have audited the services’ working capital funds activities related to carryover, new orders, and revenue that can affect depot maintenance timeliness. For more information on the services’ working capital funds depot maintenance activities, see appendix II. The depots are part of a larger, DOD-wide logistics enterprise that involves a number of different organizations, which are identified in figure 2 and described below. Assistant Secretary of Defense for Sustainment. This official serves as the principal assistant and advisor to the Under Secretary of Defense for Acquisition and Sustainment on materiel readiness. Among other responsibilities, the Assistant Secretary of Defense for Sustainment prescribes policies and procedures on maintenance, materiel readiness and sustainment support. DOD officials report that the Office of the Deputy Assistant Secretary of Defense for Materiel Readiness is responsible for maintenance policy along with the development of a strategic vision for DOD’s organic depot base. Also, the Air Force and Navy each has its own logistics or materiel command component, which provides day-to-day management and oversight of the services’ depots. Air Force Materiel Command. This command develops, acquires, and sustains weapon systems and their components, providing acquisition and life-cycle management services and logistics support, among other things. The Air Force Life Cycle Management Center within Air Force Materiel Command is responsible for the life-cycle management of weapon systems from inception to retirement, with a specific program office managing each type of aircraft. Air Force Materiel Command works with the program offices to develop, review, validate and prioritize aircraft depot maintenance workload requirements and associated funding. Naval Air Systems Command. This command is responsible for providing life-cycle support of aircraft, weapons, and systems for the Navy and Marine Corps including, acquisition, repair and modification, and in- service engineering and logistics support. As with the Air Force, a specific program office manages each type of aircraft. According to Navy officials, Naval Air Systems Command (NAVAIR), and Commander, Fleet Readiness Centers (COMFRC) work with the program offices to plan and approve the maintenance depot work executed at the Navy aviation depots, including obtaining fixed-wing aircraft workload requirements and associated funding. The 36 Air Force and Navy fixed-wing aircraft types we selected for our review ranged from fighters to bombers. These aircraft completed a total of 4,513 depot maintenance events in fiscal years 2014 through 2019. See figure 3 for more information. For the selected aircraft in our review, the Air Force completed depot maintenance on time or earlier an average of 82 percent of the time during fiscal years 2014 through 2019. However, the Navy completed depot maintenance on time or early in the same period an average of 52 percent of the time. We found that a range of factors, such as unexpected repairs and aircraft operating beyond their designed service life, have affected Air Force and Navy depot maintenance timeliness for fixed-wing aircraft. Our analysis of aggregate depot maintenance data regarding fiscal years 2014 through 2019 shows that: Air Force aviation depots completed depot maintenance of the selected fixed-wing aircraft on time or early in 5 of 6 fiscal years. The annual average percentages for on-time or early-completion maintenance ranged from 78 to 90 percent. In total, selected Air Force fixed-wing aircraft have spent 22,572 fewer days in maintenance than expected since fiscal year 2014. Navy aviation depots were late in completing depot maintenance of the selected fixed-wing aircraft for each of the 6 fiscal years. The annual average percentages for on-time or early-completion maintenance ranged from 45 to 63 percent. In total, the maintenance for selected Navy fixed-wing aircraft has taken over 62,000 more days than expected since fiscal year 2014. Figure 4 shows the percentage of depot maintenance completed on time or early, as well as total days of maintenance delays, if applicable, for the Air Force and Navy. Analyzing the maintenance timeliness data on a per aircraft basis shows similar trends. The Air Force completed depot maintenance on average about 7 days early per aircraft during fiscal years 2014 through 2019, while the Navy completed depot maintenance on average nearly 55 days late per aircraft (see figure 5). In comparing depot maintenance timeliness for specific aircraft types, we found that timeliness varied for both Air Force and Navy aircraft. For example, Air Force aviation depots completed individual KC-135 aircraft maintenance an average of about 28 days earlier than projected and completed F-15E aircraft maintenance an average of almost 35 days later than projected. Navy aviation depots completed individual EA-6B aircraft maintenance an average of about 1 day earlier than projected and completed F/A-18A-D aircraft maintenance on average about 137 days later than projected. Figure 6 shows the average number of days—by aircraft type—that the Air Force and Navy aviation depots completed maintenance earlier or later than projected in fiscal years 2014 through 2019. Our prior work has identified multiple factors that contribute to depot maintenance delays, including the size and skill of the depot workforce, the condition of weapon systems upon arrival at the depot, the availability of spare parts, and the condition of the depot’s facilities and equipment, among others. In addition, all of these factors can be affected by funding and operational considerations (such as unexpected accidents). DOD officials have stated that disruptions to funding, to include continuing resolutions, also affect the ability to conduct depot maintenance. Over the course of this review, Air Force and Navy officials cited many of these factors as continuing to affect depot maintenance timeliness while offering specific details on issues contributing to the trends we identified above. Air Force’s perspectives on early and late completions: Air Force officials stated a variety of reasons for completing aircraft maintenance earlier than projected, including frequent communication between program offices and depot stakeholders. For example, Air Force Sustainment Center officials told us that they conduct weekly aircraft performance review meetings with commanders and senior staff to provide a comprehensive status update on aircraft maintenance performance that has occurred since the previous meeting. In addition, on the KC-135 depot production line, Air Force officials told us they document tasks that can be done concurrently during a specific phase in the maintenance process, which has helped them meet their timeliness targets. Air Force officials from across the sustainment enterprise agreed that proactive planning for depot maintenance requirements helps the depots provide the appropriate resources to perform aircraft maintenance. Officials cited unexpected repairs or shortage of skilled depot maintainers as reasons for later-than-projected completion of maintenance on an aircraft. Navy’s perspectives on late completions: Navy officials stated various reasons for completing aircraft maintenance later than projected, including growth in the scope of needed work after the aircraft was evaluated (e.g., finding damage that required tailored engineering instructions), a diminishing supply of manufactured parts for aircraft, and aircraft operating well beyond their designed service life—such as the F/A-18A-D fighters and C-2A cargo aircraft. In addition to operating F/A- 18A-Ds longer than originally planned, Navy officials stated that they also had to manage aircraft production delays related to the F-35, which was scheduled to replace the F/A-18A-Ds. Navy officials explained that they have implemented a variety of strategies to improve on-time maintenance. These initiatives primarily focus on mitigating or reducing maintenance delays in the year of execution. For example: Naval Sustainment System initiative: The Navy implemented this initiative at the beginning of fiscal year 2019, which led to the service implementing private industry best practices and employing new strategies such as “swarming,” which refers to many artisans being put to work on a particular aircraft to expedite completion. The initial focus of these strategies was on the F/A-18E-F. During a site visit to Fleet Readiness Center Southwest, officials showed us how the initiative prompted reconfiguration of workstations—clearing storage and material areas in the hangar and creating direct line of vision for maintainers—to maximize an artisan’s time spent on an aircraft. Tracking depots efficiency: Officials are using a new software program that enables real-time tracking of the progress of aircraft maintenance, which they told us has led to improved efficiency because it provides increased visibility into aircraft with delays. For example, officials stated over the past 2 years, they have decreased the number of aircraft undergoing maintenance from 390 in 2017 to about 270 across the aviation depots. The Air Force has largely accurately planned for aviation depot maintenance requirements for selected fixed-wing aircraft during fiscal years 2014 through 2019, but the Navy has not. Both services have initiatives underway to improve planning for aviation depot maintenance; however, we identified several planning challenges that the Navy has not fully addressed, such as not effectively using historical data to establish accurate planning targets for aircraft depot maintenance packages. Our analysis of Air Force planned maintenance workload data—estimates of the number of days planned for depot maintenance made 3 years in advance—found that the Air Force largely accurately planned for aviation depot maintenance requirements for selected aircraft for fiscal years 2014 through 2019. The difference between the number of days the Air Force planned in advance that it would need for maintenance and actually needed has been small and trending downward, from 12 percent in fiscal year 2014 to 3 percent in fiscal years 2018 and 2019 (see figure 7). Our analysis shows that, for the 6 fiscal years we reviewed, the Air Force slightly underestimated the amount of time it needed to complete fixed- wing aircraft maintenance by an average of about 6 days per aircraft. To accurately plan for aviation depot maintenance, Air Force Materiel Command officials told us they had implemented three key initiatives including: Conducting early inspections: Air Force officials stated that they have been conducting pre-inspections of selected aircraft a year before scheduled maintenance to check for unplanned maintenance needs, and to ensure the availability of parts. Officials stated that the early inspections can clarify the scope of work and avoid extended delays in completing maintenance. For example, the Air Force has been conducting pre-inspections of its KC-135—an aerial refueling aircraft—by sending Boeing engineers to pre-inspect a sample of KC- 135s that are scheduled for depot maintenance in the following year. The inspections can inform parts orders with long-lead times and initiate developing procedures to resolve any new repairs identified during inspections, Air Force officials stated. Developing and implementing a new metric: To help measure the effectiveness of planning, Air Force officials stated that in 2017 Air Force Materiel Command created a new metric—the Planned Obligations Weighted for Execution Review—comparing which customer orders were planned for funding versus which ones actually received funding. According to Air Force officials, the metric provides visual information to leaders of the degree of variance between the planned and actual aircraft that come into the depots for maintenance. Air Force Materiel Command officials stated that the metric has helped them identify factors affecting the differences between planned and actual aircraft entering the depots for maintenance and to adjust resources when needed to address the workload. Reviewing planning performance and making adjustments: Air Force Materiel Command annually conducts a two-phased planning process to establish the organic Air Force depot-level resources necessary to support the projected funded maintenance requirements for the next 2 fiscal years. Later, the Air Force conducts an after-action review of the performance of the planning process. In addition, command leadership, senior staff, and other members of the aviation depot community hold weekly performance review meetings and make necessary adjustments. This includes reviewing visual information such as standardized charts and graphs that provide the current status of aircraft undergoing depot maintenance, as well as any issues they are monitoring. Our analysis of Navy-planned maintenance workload data—estimates of the number of days planned for depot maintenance made 3 years in advance—found that the Navy generally has not accurately planned for aviation depot maintenance requirements for selected fixed-wing aircraft for fiscal years 2014 through 2019. We found a trend of underestimating actual days needed for aircraft maintenance. The difference between the number of days the Navy planned in advance it would need for maintenance and the number actually needed ranged from a low of 3 percent in fiscal year 2014 to a high of 69 percent in fiscal year 2018. However, we found the difference declined to 42 percent in fiscal year 2019. Figure 8 shows the difference between planned and actual work days for selected Navy fixed-wing aircraft in fiscal years 2014 through 2019. Our analysis, for the 6 fiscal years that we reviewed, showed that the Navy underestimated the amount of time it needed to complete fixed-wing aircraft maintenance by an average of about 55 days per aircraft. The Navy has acknowledged that it has not accurately planned for depot maintenance requirements. The Navy conducted risk assessments and internal control assessments in 2018 and 2019 and found material weaknesses, such as a trend of underestimating time needed to address aviation depot maintenance requirements. Specifically, the two risk and internal control assessments stated that Navy policies for defining, costing, and executing maintenance did not allow them to correctly predict cost estimates and duration of depot maintenance. In addition, the Navy’s 2019 risk and internal control assessment highlighted the need to improve planning accuracy; the report stated that internal reviews found workload standards did not accurately capture the required maintenance and that planned maintenance requirements exceeded depot capacity. Navy officials stated that they implemented an initiative in fiscal year 2020 to improve maintenance requirements called Performance to Plan. This initiative is focused on incorporating data collection and analysis to, among other things, improve forecasts of maintenance period durations according to Navy documentation. For example, the approach of Performance to Plan is to incorporate predictive data into planning to improve forecasts of maintenance period durations, according to the same documentation. While this initiative is a positive step, it is still in the early stages of implementation and we identified three reasons that have in part led to inaccurate planning that the Navy has not fully addressed. The Navy measures depot performance using turnaround time as one of the key timeliness metrics. Turnaround time is the overall duration of the maintenance cycle, from when the aircraft is inducted into the depot to when it is provided back to the squadron. According to Navy officials, the Navy reviews historical data to support the maintenance requirements planning process in various ways, including adjusting turnaround time based on historical depot performance. However, we found that the Navy has not effectively used historical data to analyze turnaround time—total days planned for depot maintenance periods—and to update maintenance requirements planning for selected fixed-wing aircraft. Specifically, our analysis of average turnaround time for selected aircraft depot maintenance packages shows that the Navy has not adjusted maintenance planning effectively to account for the actual days needed to perform maintenance. Figure 9 shows that the Navy kept planned turnaround time the same or with minimal changes for maintenance packages for the C-2A, the F/A-18A-D, and the F/A-18E-F, despite worsening trends in maintenance execution during fiscal years 2014 through 2019. C-2A: In fiscal years 2016 through 2019, the Navy did not adjust its planned turnaround time—270 days—while the average number of actual work days to complete maintenance increased from 451 days in fiscal year 2016 to a high of 722 days in fiscal year 2018. Further, the difference between the average planned turnaround time and the average actual number of days needed to complete maintenance increased from 181 days in fiscal year 2016 to 352 days in fiscal year 2019, and peaked at 452 days in fiscal year 2018. F/A-18A-D: In fiscal years 2014 through 2019, the Navy adjusted its planned turnaround time by a total of 82 days, while the average number of actual work days to complete maintenance increased from 148 days in fiscal year 2014 to 694 days in fiscal year 2019, and peaked to 857 days in fiscal year 2018. In addition, the difference between the average planned turnaround time and the average actual number of days needed to complete maintenance increased from 52 days in fiscal year 2014 to 412 days in fiscal year 2019, and peaked at 629 days in fiscal year 2018. F/A-18E-F: In fiscal years 2014 through 2019, the Navy adjusted its planned turnaround time by a total of 28 days, while the average number of actual work days to complete maintenance increased from 51 days in fiscal year 2014 to 92 days in fiscal year 2019. In addition, the difference between the average planned turnaround time and the average actual number of days needed to complete maintenance increased from 10 days in fiscal year 2014 to 23 days in fiscal year 2019, and peaked at 59 days in fiscal year 2016. Naval Air Systems Command Workload Standards Required for the Aircraft and Engine Programs at the Fleet Readiness Centers states that COMFRC should analyze and review naval aviation proposed workload standard packages and compare to actual production and historical data. It also states that the workload standard development process, which includes the estimation and development of turnaround time—is to provide a basis for the identification of resource requirements at the naval aviation depot, such as personnel skills mix and materials, and as a budgetary justification of workload for the repair of aircraft. In addition, the Navy 2014-2019 Depot Maintenance Strategic Plan states that NAVAIR and COMFRC will identify and sustain requisite core maintenance capabilities through a planning process that effectively estimates and monitors near and long-term workload. Navy officials explained that they have worked to incorporate historical data into their maintenance requirements planning process; however, they acknowledged that planning needs to improve and they are in the process of revising how COMFRC and NAVAIR determines planned turnaround time. For example, COMFRC and F/A-18 program office officials said as part of the Naval Sustainment System initiative, COMFRC is moving toward a 60-day fixed turnaround time on some F/A-18E-F depot maintenance packages in an effort to drive depot maintenance efficiency and, ultimately, improve aircraft mission capability rates. According to officials, they plan to apply a fixed turnaround time across all aircraft. As the Navy moves forward, it must ensure that it effectively uses historical data to analyze turnaround time and establish accurate turnaround time targets for fixed-wing aircraft depot maintenance packages. If it does not do so, the Navy will likely continue to underestimate the number of days required to perform depot maintenance and misalign the resources and funding needed at the depots to perform aircraft maintenance, which in part contributes to persistent maintenance delays that reduce the time aircraft are available for operations and training. We found that Navy depot maintenance planners do not have direct visibility into fixed-wing aircraft maintenance that is performed outside the Navy aviation depots by an operational unit or at an intermediate maintenance facility—information critical to planning for the condition and depot maintenance needs of individual aircraft. Navy officials said that data exists on maintenance conducted on an aircraft outside the Navy aviation depots—by an operational unit or at an intermediate maintenance facility—and on the condition of an aircraft while deployed with squadrons. However, depot planners do not have direct visibility over squadron-level information because the Navy has not provided depot planners regular reporting on fixed-wing aircraft maintenance performed outside the aviation depots. Instead, depot planners can access that data only through a request to the squadron and typically rely on general planning factors rather than aircraft-specific data when estimating maintenance needs. According to Navy officials, the lack of direct visibility into the condition and maintenance history of an aircraft has driven maintenance delays in the past. For example, COMFRC and F/A-18 program office officials said that high turnaround time on certain F/A-18A-Ds undergoing maintenance was due to extended squadron usage on the aircraft combined with a lack of logistics support to address these issues. Furthermore, for aircraft damage that was outside of the depot’s repair capabilities, long lead times on parts procurement and extensive engineering analysis resulted in aircraft being placed in delay for extended periods of time, sometimes years, according to the officials. Figure 10 shows an F/A-18 undergoing depot maintenance at a Navy aviation depot. In other cases, high usage by squadrons can cause unexpected corrosion on many types of fixed-wing aircraft that depots have not prepared to address. For example, Office of the Chief of Naval Operations officials said that some aircraft have panels that are taken off during depot maintenance events. If depot maintainers find unexpected corrosion behind a panel, it may require additional time to repair that aircraft, resulting in an increase in the turnaround time. In addition, a COMFRC official stated an AV-8B arrived at an aviation depot without the engine installed, which prevents full operational checks being performed during disassembly. Once maintenance was completed, the aircraft went through operational checks and officials found canopy seal issues, which could have been identified if the depot had received data from the intermediate-level maintenance facility. The Navy 2014-2019 Depot Maintenance Strategic Plan states that NAVAIR and COMFRC will identify and sustain the necessary capabilities to perform maintenance through a planning process that effectively estimates and monitors near and long-term workload. Navy officials said direct visibility into data on the current condition and maintenance history of an aircraft at the squadron level better prepares COMFRC and NAVAIR to more accurately plan aircraft depot maintenance. This has been corroborated by Naval Sustainment System initiative findings that revealed that the Navy should be better informed about the condition of aircraft in order to improve their maintenance requirements planning. However, depot planners do not have direct visibility over squadron-level information because the Navy has not provided depot planners regular reporting on fixed-wing aircraft maintenance performed outside the aviation depots. Without regular reporting on fixed-wing aircraft maintenance performed outside the Navy aviation depots by an operational unit or at an intermediate maintenance facility, depot planners cannot plan for the condition and depot maintenance needs of individual aircraft, which in part contributes to persistent maintenance delays that reduce the time aircraft are available for operations and training. We found that the Navy does not have formal processes and related guidance for communication and coordination between depot stakeholders to inform maintenance requirements planning. Navy officials explained that depot maintenance stakeholders communicate in a variety of ways to inform maintenance requirements planning. For example, the Navy conducts annual and mid-year workload planning meetings. At the annual workload planning meeting, COMFRC and aircraft program leads provide plans to meet aircraft workload requirements for the current year and the next 2 fiscal years. The mid-year review provides an update on the current year’s performance and the final workload plan for the next 2 fiscal years, according to Navy documentation. Various aircraft program office leads attend both the annual and mid-year planning meetings to provide an update on workload plans—among others—to COMFRC. Navy officials stated that they also informally communicate in a variety of ways to inform maintenance requirements planning. For example, depot maintenance engineers may find extra corrosion on an aircraft, and use those findings to update maintenance plans for other individual aircraft. While these meetings provide opportunities for collaboration and officials utilize other means to informally communicate, NAVAIR and COMFRC do not have formal processes and related guidance for communication and coordination between depot stakeholders to ensure they receive input from all key subject-matter experts regarding workload planning. Navy officials noted that not having formal processes and related guidance presents several challenges including: Navy officials said that there is no formal process or guidance for communication and coordination, and that the process instead involves a series of documents that COMFRC receives that are assembled to create a representation of future workload from the Commander, Naval Air Forces and from each of the aircraft program offices, among others. A COMFRC official said that different stakeholders manage various parts of workload planning and without guidance on specific documentation needs and process owners, it is challenging for the Navy to identify accountable stakeholders and discuss specific planning needs. NAVAIR officials said workload planners hold periodic meetings, but attendance by subject-matter experts is not mandatory. For example, subject-matter experts from the Fleet Support Teams—officials who provide engineering and logistics technical support to fleet and aviation depot maintenance organizations—are not required to attend workload planning meetings. Experts may potentially attend via video teleconference, but others, due to time zone differences, may not participate. As a result, workload planning meetings may not consistently include workload input from all relevant subject-matter experts. Navy officials said that once the Naval Sustainment System initiative began focusing on improving depot maintenance on the F/A-18E-F, deficiencies in the workload planning process became more apparent. They noted the challenges of coordinating key stakeholders along the maintenance planning timeline and its impact on planning and budgeting. In particular, Navy officials stated the current depot maintenance planning-time horizon was disconnected from long- range planning, such as the Program Objective Memorandum process. For example, due to a misalignment in the planning and budgeting processes, COMFRC reacts to the outcome of the Program Memorandum Objective process rather than influencing it, which results in many adjustments to their productions plans, such as improper staffing, material management, and facility-usage plans. NAVAIR Instruction 5221.1B, Workload Acceptance states that commanders will establish internal competency guidelines for communication and coordination of workload-related issues. In addition, the Navy 2014-2019 Depot Maintenance Strategic Plan, states the Navy will forge a strong liaison between maintenance activities and the acquisition community to ensure that maintenance requirements and planning are in sync. As a result of the Naval Sustainment System initiative, Navy officials said that COMFRC is developing a new workload planning process to become more proactive in depot maintenance planning and increase information exchanges. This includes ensuring that the new process involves all key depot maintenance stakeholders, such as COMFRC officials, program managers, and fleet officials. For example, NAVAIR officials said that most of the Fleet Support Team scheduled maintenance leads will be the primary point of contact to assist COMFRC with developing the future maintenance requirements planning and will be invited and asked to attend workload planning meetings. If they are unable to attend, they will then ask to have a program office representative attend in their place. However, Navy officials acknowledged that their efforts are still in the developmental stages and that the Navy needs formal processes and related guidance for communication and coordination between depot stakeholders to inform maintenance requirements planning. Without these in place, the Navy cannot be assured that all subject-matter expert input is proactively solicited and incorporated into depot workload planning, which in part can contribute to persistent maintenance delays that reduce the time aircraft are available for operations and training. The ability of the Air Force and Navy aviation depots to complete maintenance on time directly affects military readiness. Poor planning for depot maintenance contributes to longer delays and reduced unit readiness. The Air Force has generally accurately planned for aviation depot maintenance over the last 6 years and in turn has completed the vast majority of its depot maintenance on time or early over this timeframe. In contrast, the Navy has not accurately planned for aviation depot maintenance over the last 6 years and in turn has completed only half of its depot maintenance on time over this timeframe, which has adversely affected aircraft availability. While the Navy has implemented an initiative to improve maintenance planning, the Navy has not effectively used historical data to analyze turnaround time and establish accurate planning targets for aircraft maintenance packages. In addition, Navy depot planners do not have visibility into aircraft maintenance that is performed outside the depots by an operational unit or other maintenance facility—information critical to planning for the condition and depot maintenance needs of individual aircraft. The Navy also has not established formal processes and related guidance for communication and coordination between depot stakeholders to ensure they receive input from all key subject-matter experts to inform maintenance planning. Without addressing these challenges, the Navy cannot appropriately plan for depot maintenance workload and may continue to experience maintenance delays that reduce the availability of aircraft for operations and training. We are making three recommendations to the Department of Navy. The Secretary of the Navy should ensure that Naval Air Systems Command and Commander, Fleet Readiness Centers effectively use historical data to analyze turnaround time and establish accurate turnaround time targets for fixed-wing aircraft depot maintenance packages. (Recommendation 1) The Secretary of the Navy should ensure that Commander, Naval Air Forces and Commander, Naval Air Force, Pacific provide depot planners regular reporting on fixed-wing aircraft maintenance performed outside the Navy aviation depots by an operational unit or at an intermediate maintenance facility to ensure they have information on the current condition and depot maintenance needs of individual aircraft. (Recommendation 2) The Secretary of the Navy should ensure that Naval Air Systems Command and Commander, Fleet Readiness Centers establish formal processes and related guidance for communication and coordination between depot stakeholders to inform maintenance requirements planning. (Recommendation 3) We provided a draft of this report to DOD for review and comment. In written comments on a draft of this report, DOD concurred with all three of our recommendations. DOD’s comments are reprinted in their entirety in appendix III. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Secretaries of the Navy and Air Force. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff has any questions about this report, please contact Diana Maurer at (202) 512-9627 or maurerd@gao.gov or Asif A. Khan, at (202) 512-9869, or khana@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to this report are listed in appendix IV. Senate Report 115-262 accompanying a bill for the National Defense Authorization Act for Fiscal Year 2019 included provisions for us to examine the Department of Defense’s (DOD) aviation depots and maintenance operations. Our report examines the extent to which 1) the Air Force and Navy aviation depots completed selected fixed-wing aircraft maintenance on time from fiscal years 2014 through 2019, and 2) the Air Force and Navy accurately planned for depot maintenance requirements for selected fixed-wing aircraft from fiscal years 2014 through 2019 and addressed any associated challenges. We have separate ongoing reviews to examine maintenance timeliness and related issues at the Army and Marine Corps key weapons systems depots and Navy shipyards. For objective one, we selected a non-generalizable sample of 18 Air Force and 18 Navy fixed-wing aircraft types, including fighters, bombers, and aerial refuelers, based on information from Navy and Air Force maintenance data and our prior work. These aircraft had maintenance completed in fiscal years 2014 through 2019, at the Navy’s three Fleet Readiness Centers and the Air Force’s three Air Logistics Complexes. We selected this time period so we could identify and obtain insight on historical data trends regarding maintenance timeliness for the selected aircraft. For each aircraft, we collected data on the date maintenance began and was completed for individual aircraft, as well as the original estimate of time (in days) needed to complete maintenance. We also collected updated estimates if available. We used this information to calculate the difference between the number of days planned for maintenance (using the updated estimate if available) and the number of days used for maintenance in order to determine whether the services completed aircraft maintenance on time, early, or late. Additionally, we used the total number of aircraft completed in each fiscal year to calculate a measure of average maintenance timeliness by aircraft type. We presented the data based on aircraft that had maintenance completed in a given fiscal year; however, not all of the maintenance was necessarily completed in that given fiscal year. For example, an aircraft may have had maintenance begun on it in one fiscal year and its maintenance completed in the next fiscal year. In such case, we would count that aircraft in the second fiscal year. The aircraft types we selected were: F/A-18A-D Hornet T-6B Texan II Turboprop In addition, we interviewed DOD and service officials to gain a better understanding of factors influencing fixed-wing aircraft maintenance timeliness and reviewed our prior work on depot maintenance. For objective two, we collected information on the depot maintenance planning processes for Air Force and Navy fixed-wing aircraft. Using the non-generalizable sample of 18 Air Force and 18 Navy fixed-wing aircraft identified in objective one, we analyzed data on maintenance duration for maintenance completed in fiscal years 2014 through 2019, and compared the number of days planned for maintenance to the number of days used for maintenance to determine the extent to which planned and actual numbers aligned. We interviewed DOD, Navy, and Air Force officials to obtain their views on the challenges related to planning, incorporating historical data, and coordinating with stakeholders related to the maintenance requirements planning process for aircraft depot maintenance. For specific challenges identified in the Navy, we reviewed documents including Naval Air Systems Command (NAVAIR) workload standards, applicable Navy guidance, and the Navy depot maintenance strategic plan and interviewed Commander, Fleet Readiness Centers (COMFRC) and NAVAIR officials to determine the extent to which the Navy incorporates historical data into the maintenance requirements planning process and the extent to which Navy depot stakeholders communicate and coordinate to inform this planning process. In addition, we visited one Air Force aviation depot and one Navy aviation depot to interview officials from specific aircraft programs, depot production, and depot business offices to understand challenges associated with planning for depot maintenance. To assess the reliability of the maintenance timeliness and planning data collected for both objectives, we reviewed and evaluated two systems— one for the Air Force and one for the Navy—that are used to collect and track data on depot maintenance. We conducted these assessments by interviewing officials regarding their data-collection processes, reviewing related policies and procedures associated with the collection of the data, examining the data for missing values and other anomalies, and interviewing knowledgeable agency officials regarding their accuracy and completeness. Based on our assessments, we determined that the data used from these systems were sufficiently reliable for the purposes of summarizing trends in selected aircraft maintenance timeliness and planning accuracy for fiscal years 2014 through 2019. We also assessed the reliability of the working capital fund data related to aviation depot maintenance activities included in Appendix II, by (1) reviewing our prior work to determine if there were reported concerns with Air Force and Navy budgetary data, and (2) reconciling the working capital fund data that was previously published in our reports for consistency. Based on our assessment, we determined that these data were sufficiently reliable for the purposes of presenting information on the services’ working capital funds activities and budget estimates for fiscal years 2014 through 2019. The U.S. military use working capital funds to procure and provide certain materiel and commercial products and services to its forces. A Working Capital Fund (WCF) is a type of revolving fund that operates as a self- supporting entity conducting a regular cycle of businesslike activities, such as acquiring parts and supplies, equipment maintenance, transporting personnel, research and development. Department of Defense (DOD) WCFs are authorized under 10 U.S.C. § 2208 and their amounts are generally available until expended. Ongoing WCF operations and maintenance of a minimum cash balance are funded through reimbursements to the WCF comprised of customer payments for goods or services received from WCF-supported activities, such as Navy and Air Force aviation depots. DOD WCFs operate on a break-even basis, although they may realize gains or losses within each fiscal year. As part of the annual budget submission for each upcoming fiscal year, however, prior year gains and losses are taken into account when new rates are established at levels estimated to recover the budgeted costs of goods and services, including all general and administrative overhead costs. Regardless, WCFs must maintain a net-positive cash balance at all times. Section 2464 of title 10 of the United States Code requires DOD to maintain a core depot-level maintenance and repair capability that is government-owned and operated. Maintaining this capability provides a ready and controlled source of technical competence and resources to enable effective and timely response to mobilizations, contingencies, or other emergencies. Additionally, DOD must assign these government- owned and operated facilities (the depots) sufficient workload to ensure cost efficiency and technical competence during peacetime, while preserving the surge capacity and reconstitution capabilities necessary to fully support the strategic and contingency plans prepared by the Chairman of the Joint Chiefs of Staff. The three Air Force and three Navy aviation depots operate through the Air Force and Navy working capital funds. Depot customers are charged for the anticipated full cost of requested goods and services. We reviewed the Air Force’s and the Navy’s budget estimates for fiscal years 2014 through 2019 and describe the information at a summary level below: Carryover (funded maintenance work leftover at the end of the fiscal year): Both services’ aviation depots underestimated carryover for most years during fiscal years 2014 through 2019. New orders (funded workload customers place at the aviation depots for maintenance work to be performed on their aircraft): Both services generally underestimated the amount of funds that their aviation depots received from new orders placed by customers and the work performed did not keep pace with those orders from year to year, during fiscal years 2014 through 2019. Revenue (dollar amount of work performed by depots in a single fiscal year): The services have varying trends for revenue for fiscal years 2014 through 2019. In fiscal years 2014 and 2017 through 2019 the Air Force, and for fiscal years 2014 through 2017 the Navy, overestimated the amount of revenue that was actually earned. Conversely, the Air Force underestimated for fiscal years 2015 through 2016, and the Navy underestimated during fiscal years 2018 through 2019. Workload (workload projections are expressed in Direct Production Earned Hours (DPEHs) for the Air Force and Direct Labor Hours (DLHs) for the Navy): A DPEH or DLH is an hour earned by a direct employee against an established work order, and includes civilians, contractors and military personnel. In fiscal years 2014 through 2019, Air Force depots’ workload increased from 21,337,000 DPEHs to 24,511,000 DPEHs—an increase of about 3.2 million (14.9 percent) DPEHs. During those same years, Navy depots’ workload generally increased from 10,161,000 DLHs to 11,668,000 DLHs—an increase of about 1.5 million (14.8 percent) DLHs. Personnel (civilian and military personnel performing depot maintenance at aviation depots and civilian staff performing support functions, such as finance and budgeting and supply and acquisitions): In fiscal years 2014 through 2019 the number of civilian personnel working at the aviation depots has generally increased by around 3,000 for both the Air Force and the Navy. Each year, customers order billions of dollars of maintenance work that the depots cannot complete by the end of the fiscal year. To the extent that the depots do not complete work at year-end, the work and related funding will be carried into the next fiscal year. DOD refers to this reported dollar value of work that has been ordered and funded (obligated) by customers, but not completed by working capital fund activities at the end of the fiscal year as “Carryover”. DOD allows and the congressional defense committees recognize that some carryover from one fiscal year to the next is needed to ensure a smooth flow of maintenance work during the transition from one fiscal year to the next. However, past congressional defense committee reports have raised concerns that the level of carryover may be more than is needed. DOD has reported that approximately 6 months of carryover is optimal. Excess carryover (i.e., more unfinished work than allowed) may reflect an inefficient use of resources and tie up funds that could be used for other priorities. Excessive amounts of carryover may result in future appropriations or budget requests of depot customers being subject to reductions by DOD and the congressional defense committees during the annual budget-review process. Tables 1 and 2 show Air Force and Navy carryover for fiscal years 2014 through 2019, respectively. Accurate budgets for the amount of new orders to be received by the depots are essential for them to plan their work, such as determining the right number of personnel, parts, and material needed. For example, if the services include workload in their new order estimates that do not materialize, a depot is at risk of incurring unplanned financial loss because the depot is allocating its overhead costs over less work than planned. These losses may lead the depots to increase their rates for repairing assets. If the customer receives more funding (e.g., Operations & Maintenance or Procurement) than they originally anticipated and they in turn increase their orders with the depots (new orders or just an increase to an existing order), or if operational decisions lead to changes in requirements or priorities, unplanned workload may materialize at the depots resulting in additional carryover. Tables 3 and 4 show Air Force and Navy new orders for fiscal years 2014 to 2019, respectively. Revenue represents the dollar amount of work performed by depots in a single fiscal year. DOD WCFs conduct businesslike activities to generate revenue from the sale of goods or services to customers, such as the military services or combatant commands, to cover costs expended throughout the year in support of those services. The DOD FMR 7000.14- R directs DOD WCFs to operate on a “break-even” basis (revenue generated equals the cost associated with receiving the revenue). See tables 5 and 6 for Air Force and Navy Depots’ Revenue (Budgeted vs Actual) for fiscal years 2014 through 2019. The Air Force and Navy express depot workload projections in Direct Production Earned Hours (DPEHs) for the Air Force, and Direct Labor Hours (DLHs) for the Navy. A DPEH or DLH is an hour earned by a direct employee against an established work order in the performance of depot work on an end item. The Air Force and Navy include direct labor hours worked by civilians, contractors and military personnel in their DPEH and DLH projections. Tables 7 and 8 show Air Force DPEHs and Navy DLHs for fiscal years 2014 through 2019, respectively. The number of civilian personnel at the Air Force and Navy aviation depots—referred to as end strength—perform the majority of depot-level maintenance activities and are made up of personnel such as artisans and maintainers—welders, machinist, sheet metal mechanics, aircraft mechanics, aircraft electricians, engineers and scientists—performing aviation depot maintenance, but also includes personnel performing support functions such as finance and budgeting. Tables 9 and 10 show total civilian and military personnel employed at the Air Force and Navy aviation depots for fiscal years 2014 through 2019, respectively. As seen in table 9, in fiscal years 2014 through 2019, the number of civilian personnel working at the Air Force aviation depots has grown by over 3,000 civilians (25,540 to 28,576). As seen in table 10, in fiscal years 2014 through 2019, the number of civilian personnel working at the Navy aviation depots has grown by over 3,100 civilians (8,515 to 11,643). Diana Maurer at (202) 512-9627 or maurerd@gao.gov, or Asif A. Khan at (202) 512-9869 or khana@gao.gov. In addition to the contacts listed above, Chris Watson (Assistant Director), Delia Zee (Analyst-in-Charge), John Craig, Sergio Enriquez, Amie Lesser, Felicia Lopez, Amanda Manning, Keith McDaniel, Richard Powelson, Benjamin Sclafani, Michael Silver, and Roger Stoltz (Assistant Director) made key contributions to this report. Military Depots: DOD Can Benefit from Further Sharing of Best Practices and Lessons Learned. GAO-20-116. Washington, D.C.: January 30, 2020. Navy Maintenance: Persistent and Substantial Ship and Submarine Maintenance Delays Hinder Efforts to Rebuild Readiness. GAO-20-257T. Washington, D.C.: December 4, 2019. Naval Shipyards: Key Actions Remain to Improve Infrastructure to Better Support Navy Operations. GAO-20-64. Washington, D.C.: November 25, 2019. F-35 Aircraft Sustainment: DOD Faces Challenges in Sustaining a Growing Fleet. GAO-20-234T. Washington, D.C.: November 13, 2019. Depot Maintenance: DOD Should Adopt a Metric That Provides Quality Information on Funded Unfinished Work. GAO-19-452. Washington, D.C.: July 26, 2019. Military Depots: Actions Needed to Improve Poor Conditions of Facilities and Equipment That Affect Maintenance Timeliness and Efficiency. GAO-19-242. Washington, D.C.: April 29, 2019. DOD Depot Workforce: Services Need to Assess the Effectiveness of Their Initiatives to Maintain Critical Skills. GAO-19-51. Washington, D.C.: December 14, 2018. Navy and Marine Corps: Rebuilding Ship, Submarine, and Aviation Readiness Will Require Time and Sustained Management Attention. GAO-19-225T. Washington, D.C.: December 12, 2018. Navy Readiness: Actions Needed to Address Costly Maintenance Delays Facing the Attack Submarine Fleet. GAO-19-229. Washington, D.C.: November 19, 2018. Depot Maintenance: DOD Has Improved the Completeness of Its Biennial Core Report. GAO-19-89. Washington, D.C.: November 14, 2018. Air Force Readiness: Actions Needed to Rebuild Readiness and Prepare for the Future. GAO-19-120T. Washington, D.C.: October 10, 2018. Weapon System Sustainment: Selected Air Force and Navy Aircraft Generally Have Not Met Availability Goals, and DOD and Navy Guidance Need to Be Clarified. GAO-18-678. Washington, D.C.: September 10, 2018. F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency. GAO-18-75. Washington, D.C.: October 26, 2017. Naval Shipyards: Actions Needed to Improve Poor Conditions That Affect Operations. GAO-17-548. Washington, D.C.: September 12, 2017. Depot Maintenance: Executed Workload and Maintenance Operations at DOD Depots. GAO-17-82R. Washington, D.C.: February 3, 2017. Depot Maintenance: Improvements to DOD’s Biennial Core Report Could Better Inform Oversight and Funding Decisions. GAO-17-81. Washington, D.C.: November 28, 2016. Army Working Capital Fund: Army Industrial Operations Could Improve Budgeting and Management of Carryover. GAO-16-543. Washington, D.C.: June 23, 2016. Defense Inventory: Further Analysis and Enhanced Metrics Could Improve Service Supply and Depot Operations. GAO-16-450. Washington, D.C.: June 9, 2016. Navy Working Capital Fund: Budgeting for Carryover at Fleet Readiness Centers Could Be Improved. GAO-15-462. Washington, D.C.: June 30, 2015. Army Industrial Operations: Budgeting and Management of Carryover Could Be Improved. GAO-13-499. Washington D.C.: June 27, 2013. Marine Corps Depot Maintenance: Budgeting and Management of Carryover Could Be Improved. GAO-12-539. Washington, D.C.: June 19, 2012. Air Force Working Capital Fund: Budgeting and Management of Carryover Work and Funding Could Be Improved. GAO-11-539. Washington, D.C.: July 7, 2011.", "summary": "Three Air Force and three Navy aviation depots maintain critical fixed-wing aviation platforms, such as the KC-135 aerial refuelers and F/A-18 fighters. The ability of these depots to complete maintenance on time directly affects military readiness because delays reduce the time aircraft are available for operations and training. Senate Report 115-262, accompanying a bill for the Fiscal Year 2019 National Defense Authorization Act, contained provisions that GAO examine the Department of Defense's (DOD) aviation depots. GAO's report evaluates the extent to which 1) the Air Force and Navy aviation depots completed selected fixed-wing aircraft maintenance on time from fiscal year 2014 through 2019, and 2) the Air Force and Navy accurately planned for depot maintenance requirements from fiscal year 2014 through 2019 and addressed any associated challenges. GAO selected a non-generalizable sample of 18 Air Force and 18 Navy fixed-wing aircraft types; analyzed maintenance and planning data for fiscal year 2014 through 2019; and interviewed service officials. The Air Force and Navy varied in the extent that they completed depot maintenance on time for selected fixed-wing aircraft in fiscal years 2014 through 2019. Specifically, GAO's analysis of aggregate maintenance data found that: Air Force depots completed aircraft maintenance on time or early in 5 of 6 years, with percentages for on-time or early-completion maintenance ranging from 78 to 90 percent. Navy depots completed aircraft maintenance late for each of the 6 years, with percentages for on-time or early-completion maintenance ranging from 45 to 63 percent. Navy fixed-wing aircraft have spent over 62,000 more days in maintenance than expected since fiscal year 2014. The Air Force generally has accurately planned for depot maintenance requirements for selected fixed-wing aircraft during fiscal year 2014 through 2019, but the Navy has not. Both services have initiatives underway to improve planning for aviation depot maintenance; however, GAO identified planning challenges that the Navy has not fully addressed: The Navy has not effectively used historical data to analyze turnaround time—total days planned for depot maintenance periods—and established accurate planning targets for aircraft maintenance packages. Navy depot planners do not have visibility into aircraft maintenance that is performed outside the depots by an operational unit or other maintenance facility—information critical to planning for the condition and depot maintenance needs of individual aircraft. The Navy does not yet have formal processes and related guidance for communication and coordination between depot stakeholders to inform maintenance requirements planning. Without addressing these challenges, the Navy cannot appropriately plan for depot maintenance workload and will likely continue to experience maintenance delays that reduce the time aircraft are available for operations and training. GAO is making three recommendations to the Navy: to use historical data to set turnaround time targets for depot maintenance; provide planners information on maintenance performed outside the depots; and establish processes for communication between depot stakeholders. DOD concurred with all three recommendations.", "document_type": "gao"}
{"report": "This section describes tribal energy resources, EPACT05, and federal agencies’ authority and processes to purchase energy. Tribal lands have untapped energy resources that, if developed, could help to alleviate economic hardships among tribal populations. According to DOE, while tribal lands account for only 2 percent of all U.S. land, tribal land contains an estimated 50 percent of potential uranium reserves, 30 percent of coal reserves west of the Mississippi, and 20 percent of known oil and gas reserves. Furthermore, DOE’s National Renewable Energy Laboratory also reports that these tribal lands contain about 6.5 percent of all utility-scale potential U.S. renewable energy resources. Ninety percent of this potential renewable energy capacity is for solar energy. According to the laboratory, tribal lands have the potential for over 6,000 gigawatts of utility-scale solar photovoltaic capacity. To put this in perspective, a single gigawatt of power running at full capacity has approximately enough energy potential to power over 800,000 homes. However, 86 percent of tribal lands with energy potential are undeveloped, according to DOE. In July 2018, DOE announced the release of its Tribal Energy Atlas and an accompanying report on the renewable energy potential on tribal lands. The atlas, developed by the National Renewable Energy Laboratory, is an interactive, web-based geospatial application that provides information about energy resource potential on tribal lands (see fig. 1). According to DOE, the atlas is the first of its kind; further, it is available to tribal energy project planners, technicians, and investors to assist with analyzing energy options on tribal lands. Previously, DOE had identified multiple tribal lands with undeveloped energy resources that could potentially meet DOD energy needs. Specifically, in a May 2013 report, DOE identified 15 reservations that had, among other things, the potential to meet DOD energy needs and were near existing transmission lines that could be used to transport the energy from the reservation to the installation. The report was based on a DOE survey of DOD installations that could have an interest in purchasing energy from tribal sources based on the tribes’ proximity to the installations and the tribal energy sources’ potential to meet installation energy needs, among other factors. EPACT05 has several provisions related to tribal energy resource development. As previously noted, one of these provisions authorizes federal agencies to give preference to majority tribally owned energy suppliers over other potential energy suppliers when purchasing energy. More specifically, EPACT05 specifies that federal agencies “may” give such a preference as long as the agencies do not pay more than prevailing market prices or obtain less-than-prevailing-market terms and conditions. In addition, EPACT05 doubles the credit that agencies receive toward their mandated renewable energy goals if the renewable energy that agencies contract for is produced on tribal lands. As noted earlier, GSA has primary authority to enter into energy contracts for federal agencies, and it has delegated this authority to DOD and DOE as well, by regulation. In addition to these statutory and regulatory authorities, the acquisition and supply of energy for federal agencies is governed by the Federal Acquisition Regulation (FAR), which is issued and maintained by the Federal Acquisition Regulatory Council (FAR Council). The process GSA has prescribed for entering into federal energy contracts varies by location, depending on market conditions and state law. In traditional energy markets, retail customers such as GSA, DOD, and DOE are typically required to contract with the local utility operating in the area for energy. In deregulated markets, these agencies publicly issue requests for proposals for energy, and energy providers engage in a competitive bidding process for federal energy contracts. Federal officials seeking to enter into energy contracts may specify energy of certain types (for example, renewable sources may be given priority) in the requests for proposals, and the energy contracts are typically awarded to the best-value provider who meets the requirements of the request for proposal. Federal officials, tribal representatives, and stakeholders we interviewed identified a number of factors that have the potential to limit federal government energy purchases from tribal sources, and they offered suggestions to address some of these factors. The factors, which sometimes overlapped, included requirements to purchase from monopoly utilities, difficulty entering the market at the prevailing rate, access to transmission infrastructure, access to capital, and technical capacity. Requirements to purchase from monopoly utilities. In traditional regulated energy markets, retail customers, including federal agencies, generally can only purchase energy from the local monopoly utility in that region. According to officials from GSA, DOD, and DOE, this requirement prevents agencies from purchasing from tribes. A representative we interviewed from one tribal energy corporation concurred with the agency officials’ assessment. That tribal energy corporation currently sells energy in wholesale markets and is interested in selling energy to federal agencies, but it has not succeeded in doing so because agencies typically make purchases as retail, not wholesale, customers, according to the tribal representative. Nonetheless, according to the tribal representative, retail customers, including federal agencies, may have the option to purchase electricity as wholesale customers in traditional markets if the entity is large enough, which would allow them to purchase from sources such as the tribal corporation. GSA officials told us that purchasing energy as a wholesale customer may not be in the best interest of the federal government, given the associated technical requirements, including connecting to the grid in ways GSA is not currently equipped for, and regulatory risk, such as managing power in a way not required of retail customers. In particular, GSA officials expressed concern about the regulatory requirements associated with reselling any potential excess energy that may come with a wholesale purchase. Additionally, DOE officials said there might be cost considerations related to achieving and maintaining status as a wholesale customer, as well as risks in giving up retail customer status, including the loss of the utility’s obligation to service the agency’s facilities because it is no longer a retail customer. In addition, according to DOE officials, switching from retail to wholesale purchasing has historically presented significant litigation risk, such as the utility challenging the legal and technical basis for the government’s change from retail to wholesale customer. Difficulty entering the market at the prevailing rate. According to GSA procurement guidance, the contracting process for public utility services should obtain the best-value product for the government, which GSA officials said typically awards the contract to the lowest-cost provider that also meets technical requirements, potentially limiting federal agency opportunities to purchase energy from tribes. In particular, tribes may find it difficult to enter the energy market at competitive rates, according to four federal officials and one stakeholder. For example, two DOE officials provided examples of DOE receiving bids from tribes for federal energy contracts but stated that both bids were unsuitable because their price was higher than the market rate. One DOE official said that tribes developing renewable energy projects would have to compete with lower- cost natural gas and hydroelectric energy, which could prevent tribes from meeting the prevailing market rate. To help foster the success of such tribal projects, one DOE official and one stakeholder suggested allowing federal agencies to purchase energy from tribes at rates that exceed the prevailing market rates. However, some tribes have successfully entered the energy market and have sold energy at competitive rates. For example, representatives we interviewed from one tribe and a renewable energy development corporation owned by several tribes said they anticipate that their current or future projects will allow them to sell energy at competitive rates, and at least two tribes have entered into contracts with the federal government. Moreover, since the beginning of 2017, DOE has seen an increased interest from tribes in renewable energy projects because the price of renewable energy has become more competitive with other, lower-cost forms of energy, according to DOE officials. As tribes develop more renewable energy projects, there may be additional opportunities for federal agencies to purchase from tribes, which will also help these agencies meet federal renewable energy goals. Access to transmission infrastructure. Lack of access to energy transmission infrastructure may prevent tribes from transmitting their energy off tribal lands, according to 10 federal officials, tribal representatives, and stakeholders whom we interviewed. One DOE official said the biggest challenge in contracting for energy with tribes can be getting a physical connection to transmit power between the tribal energy providers and a federal building. Federal officials from GSA and DOE noted that there are few federal buildings close to tribal lands, making transmission from those lands to federal buildings more complex and expensive. A 2013 Edison Electric Institute report said that the cost of new construction of overhead transmission lines can range from $174,000 to $11 million per mile. DOE’s Tribal Energy Atlas may assist tribes in overcoming this factor because it provides information on existing infrastructure, including transmission lines, giving tribes access to data they need to make informed decisions about their energy development options. Further, tribes are not limited to developing energy projects on their own lands, which can eliminate issues with proximity to federal purchasers. For example, one tribe near San Diego partnered with a private developer to build a wind farm in Illinois to sell energy to GSA. The purchase was the largest wind energy purchase from a single source in federal contracting history, according to GSA officials. Access to capital. Tribal energy development may be hindered because of difficulty obtaining access to capital, potentially limiting federal energy purchases from tribes. For example, one industry official we interviewed who worked with a tribe in the process of developing a wind farm on its reservation said the tribe does not have the necessary capital to connect to the local transmission infrastructure. As a result, it cannot provide power beyond the reservation. Likewise, nine federal officials and stakeholders whom we interviewed said securing financing for energy development could be difficult for some tribes. To overcome this potential limitation, one group of tribes combined their resources and formed a multi-tribal power authority, which allowed them to raise the necessary capital to take on a larger-scale project while maintaining tribal ownership and creating jobs in the tribal communities. The multi-tribal authority plans to develop one of the largest wind farms in the country and sell the energy at a competitive price, according to representatives from a renewable energy development corporation owned by several tribes. Another option for tribes to address this limitation is tribes leasing their land to private developers to operate and maintain energy projects, thereby benefiting from their energy resources without having to raise the capital needed to develop them but still receiving additional benefits for the tribal community. For example, one tribal representative and one stakeholder mentioned that training and educational programs for tribal members could be part of these agreements between private developers and tribes. Technical capacity. Some tribes may not have the technical capacity to develop their energy resources, which can also limit federal energy purchases from tribal sources, according to tribal representatives and stakeholders we interviewed. For example, one tribal representative and four stakeholders we interviewed said that some tribes lack experience with energy development, which potentially limits their ability to take on large-scale projects that could meet federal energy needs. Two stakeholders noted the importance of tribes having access to professionals with experience in running energy development projects to help overcome this potential limitation. Federal agencies offer programs that could assist tribes with building technical capacity. For example, the Department of the Interior provides technical and financial assistance to tribes for the exploration, development, and management of tribal energy resources. In addition, DOE offers grants and education through webinars, forums, and workshops. For example, DOE in August 2018 selected 15 tribal projects to receive funding for developing their energy resources to reduce or stabilize energy costs, as well as to increase energy security and resilience on tribal lands. DOE has also provided technical assistance, technology and market analysis, and capacity building for tribes, as well as webinars on utility-scale energy development, fundamentals of energy markets for tribes, and effective tribal project partnerships. However, DOE’s efforts have focused primarily on reducing tribal energy costs and assisting tribes in developing energy for use on reservations, rather than on selling energy to outside sources, according to DOE officials. Since the establishment of the tribal energy preference, GSA, DOD, and DOE have not entered into an energy contract with a tribe using the preference. The preference and other tribal energy resource development provisions added in the tribal energy section of EPACT05 provide federal agencies with mechanisms to support tribal energy development and use. As noted previously, the section provided for grants to assist tribes in developing their energy resources, authorization for federal agencies to give preference to tribal energy sources when contracting for energy, and double credit towards mandated renewable energy goals when federal agencies contract for energy produced on tribal lands. GSA, DOD, and DOE officials we interviewed identified five instances in the past when a tribe bid on a federal energy contract, and agencies did not use the tribal energy preference in any of these instances. Two of the instances led to contracts with GSA because, in one of those instances, officials said that the tribe submitted the best bid, and in the other, GSA used the small business preference authority instead, as discussed further below. The other three instances were bids to DOD and DOE; these instances did not lead to contracts because either the cost was too high or the proposal was unsolicited and not needed by the agency, according to agency officials. Officials from GSA and DOD noted that EPACT05 makes use of the preference discretionary because it says that federal agencies “may give preference” to a majority tribally owned energy source. Officials from DOD said they cannot authorize agency officials to use the preference without a policy or FAR requirement to use the preference. DOD officials said they follow FAR regulations and guidance when implementing policy and guidance for the agency, but the FAR has no provisions specifically addressing the preference. Similarly, GSA officials told us they would be hesitant to use the preference because they believe it limits competition solely to tribal sources, which may not be in the best interest of the federal government. GSA officials attempted to use the preference to limit an energy contract solicitation solely to tribal sources in 2014, according to a stakeholder that worked on the project, but the GSA Administrator expressed concern about limiting competition in that manner. The stakeholder noted that GSA instead decided to open the solicitation to small businesses, and the tribe ultimately won the contract through the small business preference authority. When we reported on implementation of the tribal energy preference in November 2016, we found that federal agencies had not used the preference because of uncertainty about how to do so and lack of guidance. Because GSA has primary energy purchasing authority for the federal government, we recommended that GSA develop implementing guidance to clarify how contracting officials across the federal government should use the preference. GSA partially agreed with the recommendation, stating that guidance would be beneficial, but GSA officials stated that government-wide rulemaking from the FAR Council, of which GSA is a member, is necessary to clarify how agencies should use the preference. Subsequently, GSA officials told us that in April 2017, GSA presented the FAR Council with a business case that included an analysis of the problem we identified. After reviewing the business case, the FAR Council determined the preference has limited application government-wide. GSA officials told us that the FAR Council declined to pursue regulatory changes to the FAR because, according to the council, the preference only impacts agencies responsible for entering into federal energy contracts, mainly GSA, DOD, and DOE. Further, the FAR Council recommended that GSA consider nonregulatory paths, in keeping with Executive Order 13771, which aims to reduce costs associated with regulatory compliance. In response to the FAR Council’s recommendation, GSA added the preference language from EPACT05 to the form it uses to delegate purchasing authority to other federal agencies that may seek this authority in the future. As we reported in November 2016, DOE in 2013 issued agency-specific guidance on use of the preference, such as for limiting competition to qualified majority tribally owned suppliers for the purchase of renewable energy and energy by-products. DOE distributed the tribal energy preference guidance through a February 2013 acquisition letter. However, in our interviews with officials responsible for purchasing energy in nine DOE offices, we found that officials in five of these nine offices were unaware of the DOE guidance or unaware of the preference. DOE headquarters officials stated that the agency did not take further action to communicate the guidance or ensure relevant officials were aware of it after its initial distribution. Under federal standards for internal control, management should internally communicate the necessary quality information to achieve the entity’s objectives. According to DOE documentation, the objectives of the guidance are to promote tribal renewable energy development, reaffirm the federal government’s trust responsibility to tribes, and reinforce key national policy objectives such as the acquisition and use of clean energy products. DOE officials agreed that officials responsible for purchasing energy should be aware of the agency’s guidance and the preference. For example, officials from one DOE office stated that its contracting officials are aware of the preference because it has included the preference language in its requests for proposals. By taking steps to communicate the guidance to all DOE officials responsible for purchasing energy, DOE will be better positioned to ensure that these officials are aware of the preference, which may increase its use. In addition, officials from GSA, DOD, and DOE who are responsible for purchasing energy told us they are still uncertain about how they would use the preference. For example, officials from GSA stated that they would use the preference as a tiebreaker at a minimum, but they also noted that ties are unlikely and they had not seen any ties in bids to provide energy in the last 4 years. They also noted that they would rely on GSA’s legal and acquisition policy offices for any instruction regarding using the preference. Similarly, DOD officials responsible for energy purchases stated they would have to consult with DOD’s acquisition policy office, which stated that DOD does not have guidance and could not authorize use of the preference without a requirement in the FAR, as discussed previously. Likewise, DOE officials were unclear about how they would use the preference, stating that they would use the preference by awarding the energy contract to a tribe if the tribe had the lowest bid. However, the agency would not need to use the preference in such situations because the agency generally awards contracts to the lowest bidder, according to DOE officials. According to officials from GSA and DOD, other statutes that authorize agencies to apply preferences for acquisition of goods and services from specific sources include more specific requirements in their statutory language, making the requirements easier to apply. For example, GSA officials explained that the specific requirements and measurable goals set under the Small Business Act, as amended, increases contracts awarded to small businesses. In contrast, EPACT05’s tribal energy provision does not contain analogous specific requirements for how agencies should use the preference. DOD officials stated that the agency would potentially pursue using the tribal energy preference if EPACT05 required a certain amount of energy contracts to go to tribes, similar to the Small Business Act’s requirements for small businesses. Energy resources on tribal lands present an opportunity for individual tribes that pursue development of these resources to improve their socioeconomic status by generating income, jobs, and associated economic development. The federal government, as a significant energy consumer, is in a position to support energy development on tribal lands. Through EPACT05’s tribal energy resource development provisions, including the tribal energy preference, Congress has provided federal agencies with mechanisms for such support. GSA and DOE have taken steps intended to promote use of the preference—GSA by adding the preference language when delegating energy contracting authority in the future, and DOE by issuing guidance. However, no federal agency has used the preference since its establishment in 2005, in part because EPACT05 does not require its use or include goals specifying how agencies should use it. Further, officials we interviewed at GSA, DOD, and DOE told us they were uncertain about how to use the preference. Specific incentives or requirements for the use of the tribal energy preference could help create additional opportunities for federal energy purchases from tribes as they develop more renewable energy projects. DOE’s issuance of agency-specific guidance for implementing the preference is an important positive step. However, some DOE officials responsible for purchasing energy were unaware of the DOE guidance. By taking steps to communicate the guidance to all DOE officials responsible for purchasing energy, DOE will be better positioned to ensure that these officials are aware of the preference, which may increase its use. To the extent that Congress wants to further encourage federal agencies to use tribal energy sources, it should consider amending the Energy Policy Act of 2005 to provide more specific direction to federal agencies for implementing the tribal energy preference, to include consideration of additional incentives or requirements to use these energy sources. (Matter for Consideration 1) The Secretary of Energy should communicate DOE’s tribal energy preference guidance to all DOE officials responsible for purchasing energy. (Recommendation 1) We provided a draft of this report for review and comment to DOE, DOD, and GSA. In its written comments, reproduced in appendix I, DOE concurred with our recommendation and outlined planned action to implement it. Specifically, DOE plans to issue and disseminate a new policy flash to its acquisition personnel to draw renewed attention to its tribal energy preference guidance. DOD and GSA stated that they did not have any comments. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Administrator of General Services, the Secretary of Defense, the Secretary of Energy, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the individual named above, Karla Springer (Assistant Director), Andrew Moore (Analyst in Charge), Justin Bolivar, William Gerard, Cindy Gilbert, Cynthia Norris, and Caroline Prado made key contributions to this report.", "summary": "Tribal lands hold considerable energy resources—oil, gas, coal, wind, solar, geothermal, and biomass. Tribal energy projects can help tribes fund programs and services that improve tribal members' quality of life. Federal agencies are large consumers of energy in the United States, spending about $6 billion in 2017 on energy for their facilities. Congress has provided a mechanism for agencies to support development and use of tribal energy by authorizing agencies to give preference to majority tribally owned suppliers when purchasing energy. GAO was asked to review federal efforts to use the preference. This report examines, among other objectives, the extent to which GSA, DOD, and DOE have used the tribal energy preference. GAO reviewed available agency information on use of the preference and interviewed federal agency officials to understand how agencies would use the preference when entering into contracts with tribal suppliers. None of the three primary federal agencies with authority to enter into energy contracts—the General Services Administration (GSA) and the Departments of Defense (DOD) and Energy (DOE)—have used the tribal energy preference since it was established in the Energy Policy Act of 2005 (EPACT05). The section of the act that includes the preference provides federal agencies with mechanisms that can support development and use of tribal energy resources. The mechanisms include grants to assist tribes in developing their energy resources and authorization for agencies to give preference to majority tribally owned sources in federal energy purchases, so long as they pay no more than prevailing market prices and obtain no less than prevailing market rate terms and conditions. According to DOE, tribal lands account for 2 percent of U.S. land but contain about 6.5 percent of all utility-scale U.S. renewable energy potential. GSA, DOD, and DOE officials identified five instances in the past when a tribe bid for a federal energy contract, and the agencies did not use the preference in any of those instances. GSA awarded a contract to tribes in two of the instances. In the first instance, the tribe submitted the best bid. In the second, GSA officials attempted to use the preference by limiting the energy contract solicitation solely to tribal sources, according to a stakeholder that worked on the project, but the GSA Administrator expressed concern about limiting competition in that manner. GSA instead used the small business preference authority, through which the tribe ultimately won the contract. DOD and DOE received the other three bids, which did not lead to contracts because either the cost was too high or the bid was not needed by the agency, according to agency officials. Federal officials noted that use of the preference is discretionary. EPACT05, which says agencies “may give preference,” does not require use of the preference, and the Federal Acquisition Regulation does not specifically address the preference. In November 2016, GAO reported that one reason federal agency officials cited for not using the preference was uncertainty about how to do so. GAO recommended that GSA develop guidance to clarify use of the preference across the federal government. GSA agreed that such guidance would be beneficial but stated that the Federal Acquisition Regulatory Council is the regulatory body empowered to address this issue. In April 2017, GSA presented the council with a business case on the issue. However, GSA officials told GAO that the council determined that the preference has limited application government-wide because it mainly affects GSA, DOD, and DOE, and that, accordingly, the council declined to issue regulations and recommended GSA consider nonregulatory paths. GSA then added the preference language to the form it will use if it delegates purchasing authority in the future. In 2018, federal agency officials told GAO they were uncertain how to use the preference. According to GSA and DOD officials, other statutes that authorize agencies to apply preferences for acquisition of goods and services from specific sources include more specific requirements in their statutory language, making them easier to apply. GSA officials noted that the Small Business Act, as amended, contains specific requirements and measurable goals that increase contracts awarded to small businesses. DOD officials stated that the agency might use the tribal energy preference if EPACT05 had similar requirements. To the extent that Congress wants to further encourage use of tribally owned energy sources, it should consider amending EPACT05 to provide more specific direction to federal agencies for implementing the tribal energy preference, to include consideration of additional incentives or requirements.", "document_type": "gao"}
{"report": "VA pays monthly disability compensation to veterans with service- connected disabilities according to the severity of the disability. VA’s disability compensation claims process starts when a veteran submits a claim to VA. A claims processor then reviews the claim and helps the veteran gather the relevant evidence needed to evaluate the claim. Such evidence includes the veteran’s military service records, medical exams, and treatment records from VHA medical facilities and private medical service providers. If necessary to provide support to substantiate a claim, VA will also provide a medical exam for the veteran, either through a provider at a VHA medical facility or through a VBA contractor. According to VBA officials, VBA monitors a VHA facility’s capacity to conduct exams, and in instances when the facility may not have capacity to conduct a timely exam, VBA will send an exam request to one of its contractors instead. Once the contractor accepts the exam request from VBA, it assigns a contracted examiner to conduct the exam and complete an exam report designed to capture essential medical information for purposes of determining entitlement to disability benefits. The contractors send the completed report to VBA, which uses the information as part of the evidence to evaluate the claim and determine whether the veteran is eligible for benefits. In 2016, VBA established an exam program office to manage and oversee contractors, monitor their performance, and ensure that they meet contract requirements. For example, in 2018 we reported that the contracts require that contractors develop plans outlining how they will ensure examiners are adequately trained. We also reported that contractors are required to provide VBA with monthly exam status reports, which include the number of canceled, rescheduled, and completed exams, among other things. VBA also has an office dedicated to completing quality reviews of contractors’ exam reports, which are used to assess contractor performance. VBA awarded new contracts in 2018, in part, because it wanted to update performance measures for its contractors and to change how contractors were assigned to each region throughout the country, according to agency officials. For example, officials said that the agency restructured the service areas in its contracts from five U.S. geographic districts to four to balance the number of rural and urban areas contained in each region. In doing so, they said that VBA’s goal was to distribute exams in rural areas, where it can be more challenging to find examiners, more evenly across all contractors. VBA has not fully resolved issues in collecting information on contractors’ quality and timeliness, which continues to hinder its ability to oversee contractor performance. We previously reported that VBA’s lack of complete and accurate information on the quality and timeliness of exams limited its oversight of contracted examiners and contributed to other challenges in managing the contracts. For example, VBA officials had told us that as of late June 2018, VBA was behind in completing quality reviews for contracted exams that were completed in 2017, in part, due to lack of staff to complete the quality reviews. Further, VBA officials had acknowledged that they did not have accurate information on whether contractors were completing veterans’ exams in a timely manner as outlined in the contracts. We reported in 2018 that VBA measured timeliness as the number of days between the date the contractor accepts an exam request and the date the contractor initially sends the completed exam report to VBA. However, we previously found that the exam management system VBA used until spring 2018 did not retain the initial exam completion date when VBA sent an initial exam report back to a contractor for clarification or correction. In such cases, VBA’s system maintained only the most recent date an exam report was sent back to VBA. In such a situation, according to agency officials, VBA would not always be able to accurately assess a contractor’s timeliness as outlined in the contracts. Similar to our findings, the VA Office of Inspector General’s June 2019 report on VBA’s oversight of contracted exam cancellations also identified deficiencies due to staffing shortages and exam management system limitations, among other reasons. According to VBA officials in 2018, because VBA did not have complete and accurate information on contractor performance, it could not carry out key oversight activities. For example, VBA officials acknowledged that they were unable to track exams that needed corrections or clarifications, which we reported is needed to determine if VBA should reduce payment to a contractor. In 2018, we reported that the contracts required that contractors correct these exams within a certain number of days and bill VBA for these exams at half price. However, we found that VBA’s lack of complete and reliable information on insufficient exams hindered its ability to ensure such requirements were met. Further, in the absence of current and accurate quality and timeliness information, we reported in 2018 that VBA officials told us that they had not completed the quarterly reports that summarize how each contractor performed. VBA’s delay in completing these reports meant that it had not administered other provisions of the contracts. For example, we reported in 2018 that the contracts stated that VBA could use performance data to help determine how to allocate exams within specified areas in the United States that have two contractors; in particular, VBA could decide to allocate more exams to the contractor with higher performance results. However, VBA did not have performance data on which it could base its allocation of exams. Rather, the agency based allocation on contractor workload. Further, we reported that the contracts outlined how VBA could use performance data to administer financial incentives linked to performance targets. However, due to the lack of performance information, VA had not yet administered these incentives at the time of our review in October 2018. In our 2018 report, we recommended that VBA take steps to address the oversight issues we identified by developing and implementing a plan for using data from the new exam management system to accurately assess contractor timeliness, monitor time spent correcting exams, and verify proper exam invoicing. VBA has taken steps to address issues with both the incomplete quality information and inaccurate timeliness data. For example, to help resolve the delays in completing quality reviews, VBA officials said in November 2018 that the agency had hired additional staff to assess quality of contract exam reports. As of September 2019, officials said they have 16 out of 17 full-time positions filled in the quality review office because one employee left and that they are in the hiring phase for the final position. With the addition of quality review staff, officials stated that VBA is up-to- date on completing initial quality reviews. However, they said the agency has not yet finalized any quality scores, or completed the quarterly performance reports, under the new contracts. As such, according to VBA, it has not yet administered financial incentives linked to performance. To address the inaccurate timeliness data, VBA officials stated that the agency’s new exam management system, implemented in spring 2018, was designed to capture information that would allow VBA to accurately calculate contractor timeliness. Officials also said that VBA revised its performance measures to help it more fully assess contractors’ performance. In its agency comment response to our draft report in September 2018, VBA had a target completion date of December 2018 for implementing our recommendation. However, as of September 2019, VBA reported that it has not been able to fully implement its plan for using the new system to improve oversight of contractors and did not provide a target completion date for fully implementing our recommendation. In particular, VBA has not been able to implement an automated invoicing system that it plans to use to validate the accuracy of contractors’ invoices nor can it reconcile historical data in the exam management system. As a result, according to VBA, it still cannot ensure that it is paying contractors the correct amounts based on the terms of the contracts. According to VBA, the delay in implementation is, in part, a result of having to fix technical issues with exam scheduling requests and an ongoing effort involving multiple VA offices to align VBA’s systems with those of multiple contractors. To address these issues, VBA stated that it has completed testing of its invoice system with all of the contractors and anticipates completing analysis of the results of those tests by October 2019 and will provide an updated target completion date at that time. We also recommended that VBA regularly monitor and assess aggregate performance data and trends over time to identify higher-level trends and program-wide challenges. Without plans to conduct comprehensive performance analyses, we stated that VBA is limited in its ability to determine if the contract exam program is achieving its quality and timeliness goals in a cost effective manner. VBA stated that as it makes improvements to its exam management system data it will be able to implement this recommendation, but did not provide a specific date. VBA also noted that information collected in the new exam management system has helped them to identify potential issues with the metrics that they use to assess contractor performance and that the agency is in the process of identifying the best way to analyze the data to make improvements to the program. We previously reported that VBA relies on contractors to verify that their examiners complete required VA training and that VBA did not have information on whether the training effectively prepares examiners to conduct high quality exams. Specifically, we noted that the contractors, rather than VBA, access the contractor training systems to verify that examiners have completed the required training before they are approved to conduct exams. Further, VBA did not review contractors’ self-reported training reports for accuracy or request supporting documentation, such as training certificates, or solicit feedback from contracted examiners on the effectiveness of training or suggestions for improvement. Since VBA was without plans to verify completion of training, we noted that VBA risked using contracted examiners who are unaware of the agency’s process for conducting exams. This could lead to poor-quality exams that need to be redone and, thus, delays for veterans. Similarly, without information on the effectiveness of training, VBA may not know whether additional training courses are needed. To address these concerns, we recommended that VBA document and implement a plan and processes to verify that contracted examiners have completed required training and that it collect feedback on training for the purpose of assessing its effectiveness and making improvements as needed. As of July 2019, after VBA determined that none of its contractors were comprehensive in reporting all examiners’ training, VBA reported that the agency started conducting random audits of contractor training records. Additionally, VBA said that contractors can submit feedback following the completion of each VBA-developed training course and that it will use this information to make improvements. However, VBA is still in the process of developing a centralized training system to collect information on all training completed by contracted examiners and to obtain participant feedback on each course. VBA stated that it expects the system updates that would allow it to verify that all examiners have completed required training will be fully implement by the end of fiscal year 2020 and that it will continue random audits until full implementation. In conclusion, as VBA increasingly relies on contractors to perform veterans’ disability compensation exams, it is important that the agency ensures proper oversight of these contractors. Specifically, VBA needs to ensure that (1) it has accurate and up-to-date information on individual contractor performance to ensure veterans receive quality and timely exams and that contractors are properly paid, as well as a mechanism to asses overall performance of the contract; and (2) examiners are trained to conduct these exams in a manner that results in accurate exam reports that claims processors can use to make a disability ratings decision. Without sustained oversight, VBA also runs the risk of causing undue harm to veterans through delayed or inadequate exams. Chair Luria, Ranking Member Bost, and Members of the Subcommittee, this concludes my prepared statement. I would be happy to answer any questions you or other members of the subcommittee may have at this time. For questions about this statement, please contact Elizabeth Curda, Director, Education Workforce, and Income Security Issues at (202) 512- 7215 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contact above, Nyree Ryder Tee (Assistant Director); Justin Gordinas (Analyst-in-Charge); Alex Galuten; and Jessica Orr made key contributions to this testimony. Other staff who made key contributions to the report cited in the testimony are identified in the source product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "VBA has increased the use of contractors in conducting veterans' disability medical exams. From fiscal year 2012 through mid-September fiscal year 2019, VBA reported that the number of exams completed by contractors rose from about 178,000 to nearly 958,000, which is more than half of all exams completed to date in fiscal year 2019. The remaining exams were completed by medical providers from the Veterans Health Administration. According to VBA, its contracts are worth up to $6.8 billion over 10 years. In light of issues GAO identified with VBA's oversight of contracted examiners in its October 2018 report (GAO-19-13), this testimony provides updates on VA's efforts to 1) improve its oversight of contracted examiners to ensure quality and timely exams and proper invoicing, and 2) ensure that examiners are properly trained. The Veterans Benefits Administration (VBA) has not fully resolved issues regarding how it oversees the quality and timeliness of and invoicing for disability compensation medical exams that are completed by contracted examiners. VBA uses medical exam reports from both VHA and contract examiners to help determine if a veteran should receive disability benefits. GAO reported in October 2018 that VBA was behind in completing quality reviews of contracted exams and did not have accurate information on contractor timeliness. VBA's lack of quality and timeliness data hindered its oversight of contractors' performance. In 2018, GAO made recommendations for VBA to address these issues. VBA has begun to implement GAO's recommendations, but continued action is needed to: Develop and implement a plan for using data from its new medical exam management system to (1) assess contractor timeliness, (2) monitor time spent correcting exams, and (3) verify proper exam invoicing. According to VBA, the agency has not fully implemented its plan for using this new system to resolve challenges with oversight of contractors' performance. For example, due to system issues, VBA has not been able to implement an automated invoicing system it planned to use to validate the accuracy of contractors' invoices. Further, VBA has not yet completed quarterly performance reviews of contracted exams under its new contracts, including any reports for fiscal year 2019. As a result, VBA still is unable to ensure that it is paying contractors the correct amounts based on its contract terms. Monitor and assess aggregate performance data and trends over time to identify higher-level trends and program-wide challenges. VBA officials stated that as the agency makes improvements to the exam management system data it will be able to implement this recommendation, but officials could not provide a target completion date. VBA has taken steps to address issues GAO identified with its oversight of contracted examiner training requirements but has not yet fully addressed them. Having properly trained examiners who can provide high quality exam reports is critical to ensuring that claims processors can make timely and accurate disability determinations for veterans. In 2018, GAO recommended that VBA improve its training oversight by: Implementing a plan to verify that all contracted examiners have completed required training. In response, VBA began conducting random audits of training completed by contracted examiners, but it is still in the process of developing a centralized training system that will collect this information. Such a system could help ensure that contracted examiners complete training and, ultimately, conduct high-quality exams. Collecting information from contractors or examiners on training and use this information to assess training and make improvements. VBA has since developed a feedback tool for examiners to complete following training and plans to use it to improve the training, where needed. GAO made four recommendations in 2018, including that VBA (1) develop a plan for using its new data system to monitor contractors' quality and timeliness performance, (2) analyze overall program performance, (3) verify that contracted examiners complete required training, and (4) collect information to assess the effectiveness of that training. VA agreed with and initiated actions on all of these recommendations but has not yet fully implemented them.", "document_type": "gao"}
{"report": "Different aviation industry employers have distinct workforce needs and may require workers with specific skillsets depending on the type of work performed. The workforce includes FAA-certificated mechanics and repairmen, as well as non-certificated workers. FAA-certificated mechanics inspect, service, and repair aircraft bodies (airframe) and engines (powerplant), and only they can approve an aircraft for return to service. It can take between 1 and 3 years to obtain the required education or training to become certificated. FAA-certificated repairmen service aircraft components and must be recommended for certification by their employer to perform specific tasks such as welding or painting. It can take more than a year to obtain the required experience or training to become certificated. A repairman certificate is only valid at the employer for which it was issued. Non-certificated aviation maintenance workers include individuals who are supervised by certificated mechanics or repairmen in performing repair work. Existing federal data shed light on key workforce characteristics such as the number of FAA-certificated mechanics and repairmen, their age, sex, and education. Specifically: As of December 2018, about 295,000 individuals held a mechanic certificate and about 35,000 held a repairmen certificate. The median age of FAA-certificated mechanics and repairmen was 54 years old, according to our analysis of FAA data. Three percent of all aviation maintenance certificate holders were women as of December 2018. Attending AMT school was the most common pathway certificated individuals used to qualify for the FAA tests to become mechanics. Existing federal data also provide some information on employment characteristics such as the supply of certificated workers. Specifically, FAA certificated about 8,600 mechanics and repairmen on average each year for 2014 through 2018 (see fig. 1). BLS data project an annual average of 11,800 job openings in the United States from 2018-2028 for aircraft mechanics and service technicians due to growth and replacement, which include job openings for certificated and non- certificated workers. There are, however, certain limitations to existing federal data. For example, neither FAA nor BLS collects data on the race or ethnicity of certificated individuals. In addition, FAA officials said the number of certificated individuals likely overestimates the number of them working in the aviation industry. It is unknown how many of the approximately 330,000 certificate holders are retired, deceased, or working in other industries. Furthermore, BLS data indicate 136,900 individuals were employed in the aircraft mechanics and service technicians occupation in 2018, but it is not clear how many of those jobs were filled by FAA- certificated workers. There are also limitations to determining employment characteristics such as pay for certificated workers, specifically. BLS publishes some data on pay for aircraft mechanics and service technicians, such as average hourly and annual wages. However, the occupational classification system BLS and other federal statistical agencies use for aircraft mechanics and service technicians does not distinguish between FAA- certificated and non-certificated workers, making it difficult to determine employment characteristics such as pay for certificated workers, specifically. This is in part because workers are classified by the work they perform and not necessarily by certification or education, according to occupational classification system principles. BLS officials said they collected wage and employment data for certificated workers separate from non-certificated workers in employer surveys conducted between 2000 and 2012, but stopped collecting these data in part because employers inconsistently reported them. Employers we interviewed, including air carriers and repair stations, had differing perspectives on potential growth in demand for aviation maintenance workers; some said they were experiencing difficulty finding enough workers to meet their needs, while others said they were not experiencing difficulty. Employers we interviewed for our 2014 report also expressed varying levels of difficulty filling vacancies and recruiting individuals for certain aviation professions, including aviation maintenance workers. Small and medium-sized employers in particular cited some challenges to hiring due to the wage they offered. Some stakeholders we interviewed for our recent report voiced concerns about the potential for a labor shortage. In addition to these views, two of the three selected labor market indicators (unemployment rate and wage earnings) we reviewed from 2013 through 2018 were consistent with difficulties in hiring aircraft mechanics and service technicians, while the other indicator (employment) was not. Several federal agencies such as DOD, DOL, VA, Education, and the Department of Transportation administer grants or programs that support individuals pursuing aviation maintenance careers or facilitate coordination among different stakeholders to support them. For example: DOD’s Military Services’ Credentialing Opportunities On-Line (COOL) program. This program provides funding for service members to obtain professional credentials related to their military training and helps them translate their military experience into civilian occupations. DOL’s Registered Apprenticeship Program. DOL awards grants to support Registered Apprenticeship Programs— employer-driven training opportunities that combine on-the-job learning with related classroom instruction. The program facilitates coordination among different stakeholders such as industry, states, and educational institutions to support apprenticeships and employment opportunities. In addition, FAA established an Aviation Workforce Steering Committee in February 2019, in part to coordinate efforts across FAA to address various workforce related provisions included in the FAA Reauthorization Act of 2018. Additional examples of federal grants or programs that support this workforce can be found in our report. The report also includes examples of states, industry employers, and AMT schools coordinating or partnering to support the workforce including developing career grants and military pathway programs. Despite some of FAA’s recent efforts in support of this workforce, we found that FAA does not routinely analyze, collect, or coordinate with other stakeholders on certain data related to workforce development. FAA’s strategic plan includes an objective on promoting the development of a robust aviation workforce, and its Aviation Workforce Steering Committee charter emphasizes providing diverse populations, including youth, women, and minorities, with clear pathways into aviation careers to expand the talent pool from which both government and industry may recruit. However, neither the strategic plan nor the steering committee charter provides specific information on how FAA plans to select and measure any efforts it undertakes related to these objectives. Without routinely analyzing its own data or leveraging others’ data, FAA may not have certain information it needs to track or ensure progress toward its workforce development goals. We identified several areas in which improved data analysis, collection, or coordination could assist FAA in measuring progress and understanding how to target its resources in support of its workforce related objectives. For example, FAA could use the demographic or pathway data it already collects to identify patterns or relationships (such as the trend in female certificate holders by pathway), which could be useful information as FAA aims to increase opportunities for women to pursue aviation maintenance careers. FAA could also use existing AMT school data (such as enrollment or mechanic test pass-rate data) to analyze nationwide trends or aggregate information across AMT schools to better understand the AMT school pathway as a whole. In our 2020 report that issued last week, we recommended that the Aviation Workforce Steering Committee, as part of its ongoing efforts, take steps to use existing FAA data and coordinate with other federal agencies to identify and gather the information it needs to measure progress and target resources toward its goal of promoting a robust, qualified, and diverse aviation maintenance workforce. FAA agreed with our recommendation. Even as FAA’s strategic plan states the agency’s focus on promoting the development of a skilled aviation maintenance workforce to integrate new technologies, the agency has acknowledged that the current curriculum requirements for AMT schools and mechanic testing standards are outdated. FAA officials, employers, and AMT School officials we interviewed said the current curriculum requirements do not emphasize commonly used modern aircraft technologies, such as avionics and composite materials. Over the years, FAA has attempted several times to revise curriculum requirements for AMT schools through the rulemaking process, and efforts to revise these requirements are ongoing through this process. FAA is also currently updating the testing standards for mechanics. FAA officials have noted several challenges to updating the curriculum requirements including competing demands at the department level and the extent of comments FAA has received from stakeholders in response to proposed changes. In October 2015, FAA published a notice of proposed rulemaking (NPRM) with the stated goal of updating the existing AMT school curriculum. FAA issued a supplemental NPRM in April 2019 that expanded the scope of the NPRM it issued in October 2015. Comments on the supplemental NPRM were due in June 2019. As of October 2019, FAA officials said they were in the process of reviewing the comments. FAA officials told us that a final rule will be published some time toward the end of 2020. In a separate effort outside of the rulemaking process, FAA is currently updating the testing standards for mechanics. FAA has acknowledged that current mechanic testing standards are also outdated. As a result, aviation stakeholders have stated the mechanic tests include outdated or irrelevant questions. For example, the practical test may include projects on wood airframes and fabric coverings, which are not common to modern commercial aircraft. An FAA official noted that any delay in finalizing the rule would likely result in a corresponding delay to finalizing the testing standards. Delaying the release of the updated mechanic testing standards could result in the prolonged use of outdated or irrelevant questions on the mechanic tests. FAA officials said that once finalized and implemented, the updated curriculum requirements for AMT schools and the mechanic testing standards for individuals should be mostly aligned. Chairman Larsen, Ranking Member Graves, and members of the Subcommittee, this completes my prepared remarks. I look forward to answering any questions you may have. If you or your staff have any questions about this statement, please contact me at (202) 512-2834 or krauseh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this testimony were Betty Ward-Zukerman, Assistant Director, Vashun Cole, Chelsa Gurkin, Ellie Klein, Meredith Moore, Justin Reed, Andrew Von Ah, and Chris Woika. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "FAA requires that only mechanics who are \"certificated\" by the FAA approve aircraft for return to service. Some stakeholders have expressed concern that retirements and attrition could adversely affect the capacity of this workforce to meet the growing demand for air travel, and that the mechanic curriculum is outdated. The FAA Reauthorization Act of 2018 included provisions for GAO to examine the aviation workforce. This testimony examines (1) what federal data reveal about the characteristics of the aviation maintenance workforce, (2) how selected federal agencies and other key stakeholders provide support and coordinate to develop the skills of this workforce, and (3) FAA's progress in updating the curriculum and testing standards for mechanics. GAO analyzed FAA and BLS data; reviewed relevant federal laws and regulations; and interviewed selected federal agency, industry, and AMT school officials. Federal data provide some information on the Federal Aviation Administration (FAA)-certificated aviation maintenance workforce, though certain data limitations exist. FAA maintains data on the number of individuals newly certificated each year, but less is known about how many certificated individuals exit the aviation industry each year and the extent of growing demand. A sufficient supply of certificated workers is critical for safety and to meet the growing demand for air travel. Bureau of Labor Statistics (BLS) data provide some information on pay and demand for aviation maintenance workers more broadly, but do not differentiate between FAA-certificated and non-certificated workers due to data collection challenges. Demographic data may also be useful for workforce analysis and planning. FAA data provide some demographic information on certificated mechanics and repairmen, such as age and sex, but the agency lacks data on race and ethnicity. According to GAO analysis of FAA data, the median age of the roughly 330,000 mechanics and repairmen FAA had certificated as of December 2018 was 54 years old and three percent were women. Government agencies, educational institutions, and businesses coordinate to some extent in support of this workforce, but FAA does not routinely analyze, collect, or coordinate with other stakeholders on certain data related to workforce development. One of FAA's strategic objectives includes promoting the development of a robust, skilled aviation workforce, and the agency established a committee, in part, to explore ways to diversify this workforce; however, FAA is not currently positioned to understand whether its efforts are optimally targeted or effective. Without routinely analyzing its own data or leveraging others' data, FAA may not have certain information it needs to track or ensure progress toward its workforce development goals. FAA has acknowledged that curriculum requirements for Aviation Maintenance Technician (AMT) schools and mechanic testing standards are outdated. Efforts to revise the decades-old curriculum requirements for AMT schools are ongoing and FAA officials told GAO that a final rule will be published some time toward the end of 2020. FAA officials indicated that the revised mechanic testing standards would likely be finalized after. In its February 2020 report, GAO recommended that FAA use its existing data and coordinate with other federal agencies to identify and gather information to measure progress and target resources toward its goal of promoting a robust, qualified, and diverse aviation maintenance workforce. FAA agreed with the recommendation.", "document_type": "gao"}
{"report": "According to DHS and HHS officials, DHS has historically separated a number of children from accompanying adults at the border and transferred them to HHS custody, but these separations occurred only in certain circumstances. For example, DHS might separate families if the parental relationship could not be confirmed, if there was reason to believe the adult was participating in human trafficking or otherwise a threat to the safety of the child, or if the child crossed the border with other family members such as grandparents without proof of legal guardianship. HHS has traditionally treated these children as unaccompanied alien children (UAC)—children who (1) have no lawful immigration status in the United States, (2) have not attained 18 years of age, and (3) have no parent or legal guardian in the United States or no parent or legal guardian in the United States available to provide care and physical custody. The Attorney General’s April 2018 memorandum, also referred to as the “zero tolerance” policy, directed Department of Justice (DOJ) prosecutors to accept all referrals of all improper entry offenses from DHS for criminal prosecution, to the extent practicable. According to DHS officials, in implementing the April 2018 memo, DHS’s U.S. Customs and Border Protection (CBP) began referring a greater number of individuals apprehended at the border to DOJ for criminal prosecution, including parents who were apprehended with children. In these cases, referred parents were placed into U.S. Marshals Service custody and separated from their children because minors cannot remain with a parent who is arrested on criminal charges and detained by U.S. Marshals Service. In cases where parents were referred to DOJ for criminal proceedings and separated from their children, DHS and HHS officials stated they treated those children as UAC. In such cases, DHS transferred these children to the custody of HHS’s Office of Refugee Resettlement (ORR) and ORR placed them in one of their shelter facilities, as is the standard procedure for UAC. The President’s executive order issued on June 20, 2018, directed, among other things, that the Secretary of Homeland Security maintain custody of alien families during any criminal improper entry or immigration proceedings involving their family members, to the extent possible. This order stated that the policy of the administration is to maintain family unity, including by detaining alien families together where appropriate. In addition, on June 26, 2018, a federal judge ruled in the Ms. L. v. ICE case that certain separated parents must be reunited with their minor children (referred to in this testimony statement as the “June 2018 court order”). In this case, the American Civil Liberties Union filed a federal lawsuit on behalf of certain parents (referred to as class members) who had been separated from their children. As of September, 10, 2018, the government had identified 2,654 children of potential class members in the Ms. L. v. ICE case, which we discuss in greater detail later in this statement. As of January 31, 2019, this litigation was ongoing. Under the Homeland Security Act of 2002, responsibility for the apprehension, temporary detention, transfer, and repatriation of UAC is delegated to DHS, and responsibility for coordinating and implementing the placement and care of UAC is delegated to HHS’s ORR. CBP’s U.S. Border Patrol (Border Patrol) and Office of Field Operations (OFO), as well as DHS’s ICE, apprehend, process, temporarily detain, and care for UAC who enter the United States with no lawful immigration status. ICE’s Office of Enforcement and Removal Operations (ERO) is generally responsible for transferring UAC, as appropriate, to ORR, or repatriating them to their countries of nationality or last habitual residence. Under the William Wilberforce Trafficking Victims Protection Reauthorization Act of 2008 (TVPRA), UAC in the custody of any federal department or agency, including DHS, must be transferred to ORR within 72 hours after determining that they are UAC, except in exceptional circumstances. In addition, the 1997 Flores v. Reno Settlement Agreement (Flores Agreement) sets standards of care for UAC while in DHS or ORR custody, including, among other things, providing drinking water, food, and proper physical care and shelter for children. In 2015 and 2016, we reported on DHS’s and HHS’s care and custody of UAC, including the standard procedures that DHS follows to transfer UAC to ORR. ORR’s UAC policy guide states that the agency requests certain information from DHS when DHS refers children to ORR, including, for example, how DHS determined the child was unaccompanied. Depending on which DHS component or office is referring the child to ORR, DHS may provide information on the child in an automated manner directly into ORR’s UAC Portal—the official system of record for children in ORR’s care—or via email. ORR has cooperative agreements with residential care providers to house and care for UAC while they are in ORR custody. The aim is to provide housing and care in the least restrictive environment commensurate with the children’s safety and emotional and physical needs. In addition, these care providers are responsible for identifying and assessing the suitability of potential sponsors—generally a parent or other relative in the country—who can care for the child after the child leaves ORR custody. Release to a sponsor does not grant UAC legal immigration status. Children are scheduled for removal proceedings in immigration courts to determine whether they will be ordered removed from the United States or granted immigration relief. Once at the shelter, shelter staff typically conduct an intake assessment of the child within 24 hours, and then are to provide services such as health care and education. According to ORR’s UAC policy guide, shelter staff are responsible for meeting with the child to begin identifying potential sponsors, which can include parents. To assess the suitability of potential sponsors, including parents, ORR care providers collect information from potential sponsors to establish and identify their relationship to the child. For example, the screening conducted of potential sponsors includes various background checks and in June 2018, ORR implemented increased background check requirements that were outlined in an April 2018 memorandum of agreement with DHS. These changes required ORR staff to collect fingerprints from all potential sponsors, including parents, and all adults in the potential sponsor’s household and transmit the fingerprints to ICE to perform criminal and immigration status checks on ORR’s behalf. ICE was to submit the results to ORR, and ORR used this information, along with information provided by, and interviews with, the potential sponsors, to assess their suitability. However, in December 2018, ORR revised its background check policy to limit criminal and immigration status checks conducted by ICE to the potential sponsor, unless concerns about other adult household members are raised via a public records check, there is a documented risk to the safety of the child, the child is particularly vulnerable, or the case is referred for a home study. According to HHS and DHS officials we interviewed, the departments did not take specific steps in advance of the April 2018 memo to plan for family separations or a potential increase in the number of children who would be referred to ORR because they did not have advance notice of the memo. Specifically, ORR, CBP, and ICE officials we interviewed stated that they became aware of the April 2018 memo when it was announced publicly. Though they did not receive advance notice of the April 2018 memo, ORR officials stated that they were aware that increased separations of parents and children were occurring prior to the April memo. According to ORR officials, the percentage of children referred to ORR who were known to have been separated from their parents rose by more than tenfold from November 2016 to August 2017 (0.3 to 3.6 percent). In addition, the ORR shelter and field staff we interviewed at four ORR facilities in Arizona and Texas told us they started noticing an increase in the number of children separated from their parents in late 2017 and early 2018, prior to the April 2018 memo. The DHS officials we interviewed stated that, in some locations across the southwest border, there was an increase in the number of aliens CBP referred to DOJ for prosecution of immigration-related offenses after an Attorney General memo issued in April 2017. This memo prioritized enforcement of a number of criminal immigration-related offenses, including misdemeanor improper entry. In addition, CBP officials stated that there may have been an increase in children separated from non-parent relatives or other adults fraudulently posing as the child’s parents. According to ORR officials, in November 2017, ORR officials asked DHS officials to provide information about the increase in separated children. In response, DHS officials stated that DHS did not have an official policy to separate families, according to ORR officials. A few months prior to the April 2018 memo, ORR officials said they saw a continued increase in separated children in their care. ORR officials noted that they considered planning for continued increases in separated children, but HHS leadership advised ORR not to engage in such planning since DHS officials told them that DHS did not have an official policy of separating families. From July to November 2017, the Border Patrol sector in El Paso, Texas conducted an initiative to address an increase in apprehensions of families that sector officials had noted in early fiscal year 2017. Specifically, Border Patrol officials in the sector reached an agreement with the District of New Mexico U.S. Attorney’s Office to refer more individuals who had been apprehended, including parents who arrived with minor children, for criminal prosecution. Prior to this initiative, the U.S. Attorney’s Office in this district had placed limits on the number of referrals it would accept from Border Patrol for prosecution of immigration offenses. According to Border Patrol officials, under this initiative, the U.S. Attorney’s Office agreed to accept all referrals from Border Patrol in the El Paso sector for individuals with violations of 8 U.S.C. § 1325 (improper entry by alien) and § 1326 (reentry of removed aliens), consistent with the Attorney General’s 2017 memo directing federal prosecutors to prioritize such prosecutions. For those parents placed into criminal custody, Border Patrol referred their children to ORR’s care as UAC. According to a Border Patrol report on the initiative, the El Paso sector processed approximately 1,800 individuals in families and 281 individuals in families were separated under this initiative. Border Patrol headquarters directed the sector to end this initiative in November 2017, and Border Patrol officials stated that there were no other similar local initiatives that occurred prior to the Attorney General’s 2018 memo. When the April 2018 memo was released, there was no single database with easily extractable, reliable information on family separations. DHS and HHS subsequently updated their data systems in the spring and summer of 2018, but it is too soon to know the extent to which these changes, if fully implemented, will consistently indicate when children have been separated from the parents or will help reunify families, if appropriate. Specifically, prior to April 2018, CBP’s and ORR’s data systems did not include a designated field to indicate that a child was unaccompanied as a result of being separated from his or her parent, and ORR officials stated that such information was not always provided when children were transferred from DHS to HHS custody. According to agency officials, between April and August 2018, the agencies made changes to their data systems to help notate in their records when children are separated from parents. Regarding DHS, CBP’s Border Patrol and OFO made changes to their data systems to allow them to better indicate cases in which children were separated from their parents; however, ORR officials told us in September 2018, that they had been unaware that DHS had made these systems changes. According to Border Patrol officials, Border Patrol modified its system on April 19, 2018, to include yes/no check boxes to allow agents to indicate that a child was separated from their parent(s). However, Border Patrol officials told us that information on whether a child had been separated is not automatically included in the referral form sent to ORR. Rather, agents may indicate a separation in the referral notes sent electronically to ORR, but they are not required to do so, according to Border Patrol officials. While the changes to the system may make it easier for Border Patrol to identify children separated from their parents, ORR officials stated ORR may not receive information through this mechanism to help it identify or track separated children. Prior to this system modification, Border Patrol agents typically categorized a separated child as an unaccompanied child in its system and did not include information to indicate the child had been separated from a parent. CBP’s OFO, which encounters families presenting themselves at ports of entry, also modified its data system and issued guidance to its officers on June 29, 2018, to track children separated from their parents. OFO officials have access to the UAC Portal but typically email this information to ORR as part of the referral request. According to OFO officials, prior to that time, OFO designated children separated from their parents as unaccompanied. ORR updated the UAC Portal to include a check box for indicating that a child was separated from his or her parents. According to ORR officials, ORR made these changes on July 6, 2018, after the June 20 executive order and June 2018 court order to reunify families. According to ORR officials, prior to July 6, 2018, the UAC Portal did not have a systematic way to indicate whether a child was designated as unaccompanied as a result of being separated from a parent at the border. The updates allow those Border Patrol agents with direct access to the UAC Portal to check this box, and Border Patrol issued guidance on July 5, 2018, directing its agents to use the new indicator for separated children in the UAC Portal and provide the parent’s alien number in the UAC Portal when making referrals to ORR as of July 6, 2018. However, ORR officials also said that DHS components with access to the UAC Portal are not yet utilizing the new check box consistently. Staff at three of the four shelters we visited in Arizona and Texas in July and August of 2018 said that in most, but not all cases during the spring of 2018, DHS indicated in the custody transfer information that a child had been separated. Staff at one shelter estimated that for approximately 5 percent of the separated children in its care there was no information from DHS indicating parental separation. In these cases, shelter staff said they learned about the separation from the child during the shelter’s intake assessment. Staff at the same shelter, which cares for children ages 0 to 4, noted that intake assessments for younger children are different from intake for older children, as younger children are unable to provide detailed information on such issues as parental separation. While the updates that OFO and ORR have made to their data systems are a positive step, they do not fully address the broader coordination issues we identified in our previous work. Specifically, we identified weaknesses in DHS and HHS’s process for the referral of UAC. In 2015, we reported that the interagency process to refer and transfer UAC from DHS to HHS was inefficient and vulnerable to errors because it relied on emails and manual data entry, and documented standard procedures, including defined roles and responsibilities, did not exist. To increase the efficiency and improve the accuracy of the interagency UAC referral and placement process, we recommended that the Secretaries of DHS and HHS jointly develop and implement a documented interagency process with clearly defined roles and responsibilities, as well as procedures to disseminate placement decisions, for all agencies involved in the referral and placement of UAC in HHS shelters. In response, DHS officials told us DHS delivered a Joint Concept of Operations between DHS and HHS to Congress on July 31, 2018, which provides field guidance on interagency policies, procedures, and guidelines related to the processing of UAC transferred from DHS to HHS. DHS submitted the Joint Concept of Operations to us on September 26, 2018, in response to our recommendation. We are reviewing the extent to which the Joint Concept of Operations includes a documented interagency process with clearly defined roles and responsibilities, as well as procedures to disseminate placement decisions, for all agencies involved in the referral and placement of unaccompanied children, including those separated from parents at the border, in HHS shelters. Moreover, to fully address our recommendation, DHS and HHS should implement such interagency processes. DHS and HHS took various actions in response to the June 26, 2018, court order to identify and reunify children separated from their parents. The June 2018 court order required the government to reunite class member parents with their children under 5 years of age within 14 days of the order, and for children age 5 and over, within 30 days of the order. HHS officials told us that there were no specific procedures to reunite children with parents from whom they were separated at the border prior to the June 2018 court order. Rather, the agency used its standard procedures, developed to comply with the William Wilberforce Trafficking Victims Protection Reauthorization Act of 2008 (TVPRA), to consider potential sponsors for unaccompanied children in their custody; if a parent was available to become a sponsor, reunification with that parent was a possible outcome. DHS and HHS Efforts to Identify Potential Class Members. To create the list of potential class members (that is, those parents of a separated child covered under the lawsuit) eligible for reunification per the June 2018 court order, DHS and HHS officials told us that they generated the list based on children who were in DHS or HHS custody on that date. As a result, DHS and HHS officials told us that a parent of a separated child would only be a class member if his or her child was detained in DHS or HHS custody on June 26, 2018. After developing the class list, DHS and HHS officials told us that they next determined whether class members were eligible for reunification, as a class member could be determined ineligible for reunification if it was determined that the parent was unfit or presented a danger to the child. Parents of children who were separated at the border but whose children were released by ORR to sponsors prior to the June 2018 court order were not considered class members, and according to HHS officials, the department was not obligated to reunite them with the parent or parents from whom they were separated. Further, HHS officials told us that they do not know how many such children separated from parents at the border were released to sponsors prior to the order and that the court order does not require the department to know this information. Because there was no single database with information on family separations, HHS officials reported using three methods to determine which children in ORR’s custody as of June 26, 2018, had been separated from parents at the border: 1. Data Reviewed by an Interagency Data Team. An interagency team of data scientists and analysts—led by HHS’s Office of the Assistant Secretary for Preparedness and Response with participation from CBP, ICE, and ORR—used data and information provided by DHS and HHS to identify the locations of separated children and parents, according to HHS officials. 2. Case File Review. HHS reported that more than 100 HHS staff reviewed about 12,000 electronic case files of all children in its care as of June 26, 2018 for indications of separation in specific sections of each child’s case file, such as the phrases “zero tolerance,” “separated from ,” and “family separation.” 3. Review of Information Provided by Shelters. According to HHS officials, shelter staff were asked to provide lists of children in their care who were known to be separated from parents based on the shelter’s records. On the basis of its reviews, as of September 10, 2018, the government had identified 2,654 children of potential class members in the Ms. L. v. ICE case. Of the 2,654 children, 103 were age 0 to 4 and 2,551 were age 5 to 17. As previously discussed, the number of children of potential class members does not include those who were separated from parents but released to sponsors prior to the June 2018 court order or the more than 500 children who were reunified with parents by CBP in late June 2018, because these children were never transferred to ORR custody. As of September 10, 2018, 2,217 of the 2,654 identified children had been released from ORR custody, according to a joint status report filed in the Ms. L. v. ICE case. About 90 percent of the released children were reunited with the parent from whom they were separated and the remaining children were released under other circumstances. Children released under other circumstances could include those released to another sponsor such as a parent already in the United States, another relative, or an unrelated adult, or children who turned 18. Staff at one ORR facility we visited told us they planned to release some children under these circumstances. As of December 11, 2018, the government had identified additional possible separated children of potential class members for a total of 2,816. It had released 2,657 and 159 remained in ORR custody. However, the government has also reported that 79 of the children it initially identified as separated had not been separated from a parent. Excluding those 79 children from the 2,816 total would bring the total number of children separated to 2,737. Plan for Reunifying Children with Class Member Parents Within and Outside ICE’s Custody. The process used to reunify separated children with their class member parents in the Ms. L. v. ICE case evolved over time based on multiple court hearings and orders, according to HHS officials. After the June 2018 court order, HHS officials said the agency planned to reunify children using a process similar to their standard procedures for placing unaccompanied children with sponsors. However, according to agency officials, the agency realized that it would be difficult to meet the court’s reunification deadlines using its standard procedures and began developing a process for court approval that would expedite reunification for class members. As a result, from June 26, 2018 to July 10, 2018, the reunification process was refined and evolved iteratively based on court status conferences, according to HHS officials. ORR field and shelter staff we interviewed noted the impact of the continually changing reunification process; for example, staff at one shelter told us there were times when they would be following one process in the morning but a different one in the afternoon. On July 10, 2018, the court approved reunification procedures for the class members covered by the June 2018 court order. In the July 10, 2018 order that outlined these procedures, the court noted that the standard procedures developed by ORR pursuant to the TVPRA were meant to address “a different situation, namely, what to do with alien children who were apprehended without their parents at the border or otherwise” and that the agency’s standard procedures were not meant to apply to the situation presented in the Ms. L. v. ICE case, which involves parents and children who were apprehended together and then separated by government officials. The reunification procedures approved in the Ms. L. v. ICE case apply only to reunification of class members with their children and included determining (1) parentage and (2) whether the parent is fit to take care of the child or presents any danger to the child. Specifically: 1. Determining Parentage. Before July 10, 2018, to determine parentage for children ages 0 to 4, HHS officials said they initially used DNA swab testing instead of requiring documentation, such as birth certificates, stating that DNA swab testing was a prompt and efficient method for determining biological parentage in a significant number of cases. On July 10, 2018, the court approved the use of DNA testing “only when necessary to verify a legitimate, good-faith concern about parentage or to meet a reunification deadline.” HHS officials told us that at that point, to determine parentage, ORR relied on the determinations made by DHS when the family was separated and information ORR shelter staff had already collected through assessments of the children in their care. Unless there were specific doubts about the relationship, ORR did not collect additional information to confirm parentage, according to HHS officials. 2. Determining Fitness and Danger. To reunify class members, HHS also followed the procedures approved by the court on July 10, 2018 for determining whether a parent is fit and whether a parent presents a danger to the child. HHS used the fingerprints and criminal background check of the parent conducted by DHS when the individual was first taken into DHS custody rather than requiring the parent and other adults living in the household to submit fingerprints to ORR, as potential sponsors were typically required to do for unaccompanied children. According to HHS officials, ORR personnel also reviewed each child’s case file for any indication of a safety concern, such as allegations of abuse by the child. HHS did not require fingerprints of other adults living in the household where the parent and child will live. HHS also did not require parents to complete an ORR family reunification application as potential sponsors are typically required to do for unaccompanied children. The specific procedures for physical reunification varied depending on whether the parents were inside or outside of ICE custody. DHS and HHS took steps to coordinate their efforts to reunify children with parents who were in ICE custody, but experienced challenges. Generally, for parents in ICE custody, DHS transported parents to a detention facility close to their child and HHS transported the child to the same facility. At the facility HHS transferred custody of the child to ICE for final reunification. HHS officials said that in some instances children had to wait for parents for unreasonably long amounts of time and parents were transported to the wrong facilities. In one case, staff at one shelter told us that they had to stay two nights in a hotel with the child before reunification could occur. According to HHS officials, for families in which the parent was released into the interior of the United States, the reunification process involves ORR officials and shelter staff attempting to establish contact with the parent and determining whether the parent has “red flags” for parentage or child safety. These determinations are based on DHS-provided criminal background check summary information and case review of the child’s UAC Portal records. In cases where no red flags are found, HHS transports the child to the parent or the parent picks the child up at the ORR shelter. For more information on DHS and HHS reunification procedures for class member parents inside and outside ICE custody, see GAO-19-163. Chair DeGette, Ranking Member Guthrie, and Members of the Subcommittee, this concludes our prepared remarks. We would be happy to answer any questions that you may have. For further information regarding this testimony, please contact Kathryn A. Larin at (202) 512-7215 or larink@gao.gov or Rebecca Gambler at (202) 512-8777 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony include Kathryn Bernet (Assistant Director), Elizabeth Morrison (Assistant Director), David Barish (Analyst-in-Charge), Andrea Dawson, Jason Palmer, and Leslie Sarapu. In addition, key support was provided by Susan Aschoff, James Bennett, Sarah Cornetto, Michael Kniss, Sheila R. McCoy, Jean McSween, Jan Montgomery, Heidi Nielson, and Almeta Spencer. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "On April 6, 2018, the Attorney General issued a memorandum on criminal prosecutions of immigration offenses. According to HHS officials, this resulted in a considerable increase in the number of minor children whom DHS separated from their parents after attempting to cross the U.S. border illegally. On June 20, 2018, the President issued an executive order directing that alien families generally be detained together, and on June 26, 2018, a federal judge ordered the government to reunify separated families. DHS is responsible for the apprehension and transfer of UAC to HHS. HHS is responsible for coordinating UAC placement and care. This testimony discusses DHS and HHS (1) planning efforts related to the Attorney General's April 2018 memo, (2) systems for indicating children were separated from parents, and (3) actions to reunify families in response to the June 2018 court order. It is based on a report GAO issued in October 2018. This testimony also includes updated data reported by the government on the number children separated from their parents subject to the court's reunification order and the number of those children in ORR custody as of December 11, 2018. Department of Homeland Security (DHS) and Department of Health and Human Services (HHS) officials GAO interviewed said the agencies did not plan for the potential increase in the number of children separated from their parent or legal guardian as a result of the Attorney General's April 2018 “zero tolerance” memo because they were unaware of the memo in advance of its public release. The memo directed Department of Justice prosecutors to accept for criminal prosecution all referrals from DHS of offenses related to improper entry into the United States, to the extent practicable. As a result, parents were placed in criminal detention, and their children were placed in the custody of HHS's Office of Refugee Resettlement (ORR). DHS and ORR treated separated children as unaccompanied alien children (UAC)—those under 18 years old with no lawful immigration status and no parent or legal guardian in the United States available to provide care and physical custody. Prior to April 2018, DHS and HHS did not have a consistent way to indicate in their data systems children and parents separated at the border. In April and July 2018, U.S. Customs and Border Protection's Border Patrol and ORR updated their data systems to allow them to indicate whether a child was separated. However, it is too soon to know the extent to which these changes, if fully implemented, will consistently indicate when children have been separated from their parents, or will help reunify families, if appropriate. In response to a June 26, 2018 court order to quickly reunify children separated from their parents, HHS determined how many children in its care were subject to the order and developed procedures for reunifying these families. As of September 2018, the government identified 2,654 children in ORR custody who potentially met reunification criteria, which does not include separated children released to sponsors prior to the June 2018 court order. On July 10, 2018, the court approved reunification procedures for the parents covered by the June 2018 court order. This July 10, 2018 order noted that ORR's standard procedures used to release UAC from its care to sponsors were not meant to apply in this circumstance, in which parents and children who were apprehended together were separated by government officials. Since GAO's October 2018 report, the government identified additional children separated from parents subject to the court's reunification order and released additional children from its custody (see figure). GAO recommended in 2015 that DHS and HHS improve their process for transferring UAC from DHS to HHS custody. DHS and HHS concurred and have taken action, but have not fully implemented the recommendation.", "document_type": "gao"}
{"report": "Different aviation industry employers have distinct workforce needs and may require workers with specific skillsets depending on the type of work performed. The aviation maintenance workforce includes FAA-certificated mechanics and repairmen, as well as non-certificated workers. FAA-certificated mechanics inspect, service, and repair aircraft bodies (airframe) and engines (powerplant), and only they can approve an aircraft for return to service. FAA-certificated mechanics can earn an airframe rating, a powerplant rating, or an airframe and powerplant (A&P) rating. It can take between 1 and 3 years to obtain the required education or training to become certificated. If an FAA- certificated mechanic changes employers, the certificate remains valid. FAA-certificated repairmen service aircraft components and must be recommended for certification by their employer to perform specific tasks such as welding or painting. It can take more than a year to obtain the required experience or training to become certificated. FAA-certificated repairmen are employed by entities such as repair stations that are authorized by FAA to perform specific tasks. A repairman certificate is only valid at the employer for which it was issued. Non-certificated aviation maintenance workers include individuals who are supervised by certificated mechanics or repairmen in performing repair work. FAA maintains data on certificated mechanics and repairmen, including data on characteristics such as age and sex. FAA also maintains some data on non-certificated workers, such as the number employed by FAA- certificated repair stations, but neither the federal government nor the aviation industry maintains data on the total number of non-certificated aviation maintenance workers. Career pathways consist of education, training, and support services that enable individuals to obtain industry-relevant certification and employment. There are three distinct pathways to become eligible to take the FAA mechanic tests—military training and experience, AMT School, and practical, or civil work experience (see fig. 1). FAA collects data on the use of the different pathways to becoming a certificated mechanic. Individuals must pass the FAA mechanic tests to become certificated, regardless of the pathway they take to become eligible to take the tests. There are three tests—written, oral, and practical. FAA publishes testing standards for the oral and practical skills tests. Military training and experience. The Community College of the Air Force administers an FAA-approved A&P training program which consists of on-the-job training and various courses for military service members with certain experience. When service members successfully complete the program, the Joint Services Aviation Maintenance Technician Certification Council issues them a certificate of eligibility to take the FAA mechanic tests (see side bar). Aviation Maintenance Technician School. Individuals may also attend an FAA-approved AMT School to become eligible to take the FAA mechanic tests. FAA approves and oversees AMT Schools and it maintains enrollment and mechanic test pass-rate data for each school. The minimum curriculum requirements for these schools are currently prescribed in regulation. The regulation includes the subjects the curriculum must cover and the number of training hours students must complete to become eligible to take the FAA mechanic tests. Given that AMT School curriculum requirements are in regulation, FAA has in the past attempted to amend the requirements through the federal rulemaking process. Practical work experience. People can also become eligible to take the FAA mechanic tests by demonstrating practical, or civil work experience. Individuals may work under the supervision of a certificated mechanic for 18 months for either an airframe or powerplant certificate, or 30 months for an A&P certificate. Practical work experience includes apprenticeships, which combine on-the-job training with classroom instruction. For certificated repairmen, there is no prescribed test, though repairmen must demonstrate their practical experience or have completed formal training to be certificated. Avionics technicians also have no prescribed test, but may seek certain related certifications. The Standard Occupational Classification (SOC) system is a federal statistical standard used to classify workers into occupational categories for purposes of collecting, calculating, or disseminating data such as employment levels and pay. The SOC structure forms the basis for the occupational coding system used by BLS’ Occupational Employment Statistics survey and Current Population Survey. Aviation maintenance workers generally fall into the avionics technicians or the aircraft mechanics and service technicians occupational group (see fig. 2). Both occupational groups include certificated and non-certificated individuals. The most recent revision to the SOC was for 2018. In addition to the Department of Transportation (DOT) and FAA, several other federal agencies play a role in developing and maintaining a qualified aviation professional workforce. For example, we previously reported on related efforts administered by DOD, DOL, VA, and Education. These agencies provide either financial assistance for education or training in aviation maintenance related fields or administer programs that support career pathways to becoming an FAA-certificated mechanic or repairman. As of December 2018, about 295,000 individuals held a mechanic certificate and about 35,000 held a repairman certificate. The median age of FAA-certificated mechanics and repairmen was 54 years old, according to our analysis of the FAA data. Specifically, 52 percent were between the ages of 50 and 70 years old; 19 percent were between 39 and 49; and 19 percent were between 18 and 38. The remaining 10 percent were between the ages of 71 and 89 years old (see fig. 3). In comparison, about 23 percent of the overall workforce was age 55 or over according to BLS data as of 2018. Our analysis of FAA data also found that 3 percent of all aviation maintenance certificate holders were women as of December 2018. This percentage has not changed since we last reported on this workforce in 2014. In comparison, BLS data as of 2018 show that women made-up 47 percent of the total workforce. We were not able to analyze other demographic characteristics for these certificate holders, such as race or ethnicity, because neither FAA nor BLS collects these data. FAA data for 2015 through 2018 also provide some information on the education and work experience of certificated mechanics. These data show that attending AMT School was the most common pathway certificated individuals used to qualify for the FAA tests to become mechanics (see fig. 4). In addition, FAA data provide information on the number of newly certificated individuals and indicate that FAA certificated about 8,600 mechanics and repairmen on average each year for 2014 through 2018 (see fig. 5). BLS data project an annual average of 11,800 job openings in the United States from 2018-2028 for the aircraft mechanics and service technicians occupation due to growth and replacement, which include job openings for both certificated and non-certificated workers. The supply of workers to fill any open or projected job openings in the aviation industry, however, not only depends on the number of people qualified to do the work, but also their availability and willingness to work at a certain wage and under particular working conditions. While FAA data provide information on the number of mechanic and repairman certificate holders who are qualified to perform certain work, less is known about the number of them who are available and willing to work in the aviation industry. FAA data show there were approximately 330,000 certificated mechanics and repairmen as of December 2018, but FAA officials said this number likely overestimates the number of individuals working in the aviation industry. BLS data indicate 136,900 were employed in the aircraft mechanics and service technicians occupation in 2018, but it is not clear how many of those jobs were filled by FAA- certificated workers. In addition, it is unknown how many of the approximately 330,000 certificate holders are retired, deceased, or working in other industries. Individuals who obtain a mechanic certificate from FAA may never work in the aviation industry, or may begin their career in the aviation industry and leave for a job in another industry. Several stakeholders we interviewed said FAA-certificated mechanics possess certain skills that are transferrable to other industries and leave the aviation industry to work for other employers, such as amusement parks. Furthermore, officials explained that once certificated, there is no certification renewal requirement for mechanics. BLS publishes some data on pay for aircraft mechanics and service technicians, such as average hourly and annual wages. However, the occupational classification system BLS and other federal statistical agencies use for aircraft mechanics and service technicians does not distinguish between FAA-certificated and non-certificated workers. This is in part because workers are classified by the work they perform and not necessarily by certification or education, according to SOC classification system principles. As a result, it is difficult to determine employment characteristics such as pay for certificated workers, specifically. BLS data as of May 2018 show that annual wages for aircraft mechanics and service technicians ranged from about $37,000 to about $98,000. According to BLS officials, it is not uncommon for there to be a wide salary range across an occupation, as wages may vary depending on factors such as experience, education, and skills. A DOD official we interviewed also said that employers have told him that they pay certificated aviation maintenance workers more than non-certificated workers. BLS officials said they collected wage and employment data for certificated workers separate from non-certificated workers in employer surveys conducted between 2000 and 2012. However, officials said they stopped collecting these data in part because employers inconsistently reported them. Data limitations at the federal and state levels also make it difficult to determine the demand for certificated aviation maintenance workers. BLS occupational data. On the federal level, BLS occupational outlook data provide some information on potential future demand nationwide for aviation maintenance workers, but the data do not distinguish between certificated and non-certificated workers. As a result, the data provide limited detail about the demand for certificated workers, specifically. According to BLS data, total employment for the aircraft mechanics and service technicians occupation is projected to grow about 3 percent over the 2018 to 2028 time frame, which is slower than the average for all occupations. As previously mentioned, these data project an annual average of 11,800 job openings for this occupation from 2018 to 2028 due to job growth and replacement. DOL certification data. On its public website for career planning and job search, CareerOneStop, DOL provides information on certifications that are frequently mentioned in online job postings and considered to be in-demand. DOL also indicates in its online resources which certifications may draw on training and experience gained in the military. However, DOL does not track or publish data on the demand for occupational licenses, including federal licenses such as FAA’s A&P certification. DOL officials said currently there are no plans to expand the agency’s data collection to include information on the demand for occupational licenses. DOL officials added that for certain jobs that require licenses, the demand for the required licenses mirrors occupational demand for those jobs so collecting those data may not be as meaningful. Workforce Innovation and Opportunity Act plans. On the state level, Workforce Innovation and Opportunity Act plans, intended in part to outline states’ use of federal funds to help workers meet employers’ needs, provide some geographically-specific information on the demand for workers in the aviation industry. Our review of states’ most recent Workforce Innovation and Opportunity Act plans found that 19 states identified the aerospace and aviation industry as a targeted sector for development. However, only certain plans specifically mention the need for certificated mechanics; others refer to the aviation industry more broadly. Employers we interviewed had differing perspectives on potential growth in demand for aviation maintenance workers; some said they were experiencing difficulty finding enough workers to meet their needs, while others said they were not experiencing difficulty. While some stakeholders voiced concerns about the potential for a labor shortage, the selected labor market indicators we reviewed for aircraft mechanics and service technicians (unemployment, wages, and employment) from 2013 through 2018 were not all consistent with the existence of hiring difficulties. See appendix I for our analysis. Officials we interviewed from a regional airline said the majority of the airline’s certificated mechanics come to them directly after completing AMT School and that the airline was having a difficult time finding enough mechanics to fill 60 open full-time positions. On the other hand, officials we interviewed from a major airline said the airline rarely hires certificated mechanics right out of AMT School and that their employees typically come to them with a number of years of experience. The officials from the major airline said that they were not experiencing difficulty recruiting and retaining aviation maintenance workers, but noted that regional airlines may experience hiring difficulties first if there is a shortage of these workers because certificated mechanics often start their careers at regional airlines to gain practical experience before moving on to work at a major airline. Registered Apprenticeship Program Serving Underrepresented Populations The Department of Labor awarded the Connecticut Department of Labor Office of Apprenticeship Training a $1,550,000 grant to fund the Connecticut Apprenticeship Expansion Rx project, which targets the aviation industry, among others. The project aims to serve over 1,600 apprentices and provide underrepresented populations, including women, dislocated, and under- employed individuals an opportunity to acquire industry required credentials. Key partners include industry, educational institutions, and labor unions. Registered Apprenticeship Program for Airframe and Powerplant Mechanics As part of the State Apprenticeship Expansion grant, the Alaska Department of Labor and Workforce Development (DOLWD) is implementing registered apprenticeships in aviation, a relatively new industry in using the apprenticeship model. With the help of the U.S. Department of Labor Office of Apprenticeship in Alaska, two aviation occupations have been approved: Airframe & Powerplant Mechanic and Air Transport Pilot. The Alaska state Apprenticeship Coordinator is working closely with the U.S. Department of Labor Office of Apprenticeship and individual air carriers across Alaska to develop and implement registered apprenticeships for these occupations. Alaska DOLWD has approximately 12 mechanic apprentices and 2 air transport pilot apprentices with various air carrier employers. support Registered Apprenticeship Programs— employer-driven training opportunities that combine on-the-job learning with related classroom instruction. The program facilitates coordination among different stakeholders such as industry, states, and educational institutions to support apprenticeships and employment opportunities. DOL awarded almost $3.8 million in grants and contracts from 2014 through 2018 to promote these apprenticeships for aviation maintenance workers. For example, one grantee is aiming to serve underrepresented populations in the aviation industry, including women, and another is coordinating with industry to develop a registered apprenticeship program for certificated mechanics (see side bars). In addition, the United Services Military Apprenticeship Program, a DOL registered program, provides service members an opportunity to improve skills and qualify for employment in a recognized civilian trade, including as an A&P mechanic, upon completion of military service. VA’s Post-9/11 GI Bill Program. The Post-9/11 Veterans Educational Assistance Act of 2008 (Post-9/11 GI Bill) provides funding for veterans to pursue an approved program of education, including undergraduate and graduate degrees, non-college degree programs, apprenticeships, and on-the-job training. VA data show approximately $42 million in Post-9/11 GI Bill funds were awarded in fiscal year 2018 to 4,200 veterans enrolled in aviation maintenance post-secondary programs, which include programs at FAA-approved AMT Schools. Education’s financial assistance. Education provides billions of dollars in federal assistance to support students pursuing higher education, which may include training in aviation-related fields. We previously reported that in academic year 2011-2012, Education disbursed $918 million in federal grants to 142,708 recipients and $1.3 billion in federal loans to 114,564 recipients pursuing aircraft mechanic and avionics programs. DOT’s workforce development grant program. DOT is also developing a process for administering a workforce development grant program for aviation maintenance workers. Specifically, the FAA Reauthorization Act of 2018 included a provision for DOT to establish an aviation maintenance workforce development grant program. Once established, eligible entities such as aircraft repair stations, unions, educational institutions, and state or local governments may apply for grants. The program may provide grants for projects such as establishing new educational programs or scholarships for individuals seeking employment in the aviation maintenance industry and supporting service members transitioning into aviation maintenance related careers. In addition, FAA has taken steps to engage other key stakeholders on aviation workforce development initiatives. In September 2018, FAA sponsored an Aviation Workforce Symposium that included participants from industry, AMT Schools, and federal agencies such as Education and DOL. Discussion topics included building the pipeline of workers, maximizing efficiency in training, and promoting productive partnerships. Subsequent to the 2018 symposium, FAA established an Aviation Workforce Steering Committee in February 2019, in part to coordinate efforts across FAA to address various workforce related provisions included in the FAA Reauthorization Act of 2018. The steering committee finalized its charter in April 2019, and the charter states FAA’s intentions of developing productive partnerships with industry, government, and educational institutions to expand the pipeline of aviation safety professionals. As of October 2019, FAA was finalizing a working group structure to carry out the steering committee’s work that will focus on: (1) marketing/communications, (2) educational outreach, (3) training, and (4) partnerships. FAA officials also told us they plan to collaborate with other federal agencies moving forward, including Education, DOL, and DOD. For example, FAA and DOD officials said they are currently discussing options for streamlining pathways for service members with aviation maintenance backgrounds to move into civilian careers in aviation maintenance. According to a DOD official, streamlining pathways could increase the number of service members who become FAA-certificated mechanics and leverage the skills of the over 250,000 current service members with aviation maintenance backgrounds. Additional examples of states, industry employers, and AMT Schools coordinating or partnering to support the aviation maintenance workforce include: Career grants. One state we visited developed a career grant to align students’ programs of study with in-demand occupations in the state. The grant provides tuition assistance to in-state residents working toward selected certificates or degrees at eligible in-state colleges or universities, including aviation maintenance programs. Military pathway program. Officials from a regional airline we interviewed said the airline developed a military transition program to assist service members in preparing for the FAA mechanic tests. The airline funds 100 percent of the program cost, which according to officials is about $11,000 per person. As part of the program, airline officials told us they provide about $5,000 worth of tools to each participant. Training equipment and funding. Officials we interviewed from one school said they strategically opened their AMT program next to a major cargo airline so that students could benefit from employment opportunities there. The officials said the airline also benefits from the close proximity of the school in that it is able to leverage local talent, and the airline provides AMT School students with scholarships, technical support, and surplus equipment to use for training. In another example, officials from a major commercial airline told us the airline partners with over 40 AMT Schools and provides them with funding to improve operations and recruitment. Officials said the goal of the program is to ensure the airline has a pipeline of workers to fill any future job openings. Despite FAA’s recent efforts to coordinate with other federal agencies on expanding and streamlining pathways for aviation maintenance careers, we found that FAA does not routinely analyze, collect, or coordinate with other stakeholders on certain related data. Such activities could assist FAA in measuring progress toward meeting its workforce related objectives and inform strategic decisions for promoting the development of this workforce. For example, FAA’s strategic plan includes an objective on promoting the development of a robust aviation workforce. In addition, FAA’s Aviation Workforce Steering Committee charter emphasizes providing diverse populations, including youth, women, and minorities, with clear pathways into aviation careers to expand the talent pool from which both government and industry may recruit. However, neither the strategic plan nor the steering committee charter provides specific information on how FAA plans to select and measure any efforts it undertakes related to these objectives. Prior GAO work has emphasized that strategic workforce planning requires monitoring and evaluating progress toward goals, and federal internal control standards state that management should use quality information to achieve its objectives. We identified several areas in which improved data analysis, collection, or coordination could assist FAA in measuring progress and understanding how to target its resources in support of its workforce related objectives. Demographic data. FAA collects certain demographic data on its A&P certification application, such as the age and sex of individuals; however, FAA currently uses these data only to determine eligibility and issue certificates, according to FAA officials. These data could also be used to identify patterns or relationships, such as the trend in female certificate holders by pathway, which could be useful information as FAA aims to increase opportunities for women to pursue aviation maintenance careers. In addition, FAA does not currently collect data on the race and ethnicity of certificated individuals. Such data could provide additional information on the demographics of certificated individuals and help FAA or other stakeholders monitor the progress of any efforts to diversify this workforce. FAA could also leverage BLS data on the race and ethnicity of certificated and non-certificated aircraft mechanics and service technicians more broadly as it begins to develop and implement any activities related to expanding and diversifying the talent pool for recruiting workers into aviation maintenance careers. Pathway data. FAA also maintains mechanic pathway data, but these data do not provide a complete picture of certificated individuals’ education, training, and work experience due to certain data limitations. For example, FAA does not require AMT Schools to report program completion data. As a result, it does not have information such as how many students who enter FAA-approved AMT Schools complete the program. Moreover, FAA does not analyze nationwide trends for AMT Schools using existing data on these schools (such as enrollment or mechanic test pass-rate data) or aggregate information across AMT Schools to better understand the AMT School pathway as a whole. In addition, pathway data collected by FAA do not clearly differentiate between civil and military work experience. Specifically, FAA officials said practical, or civil, work experience pathway data may include information on individuals with both prior military and civil experience. Moreover, according to an FAA official, FAA’s military experience pathway data may include individuals who completed DOD’s FAA- approved A&P training program as well as individuals who met FAA’s on-the-job training requirements through relevant military experience. Combined pathway data may limit FAA’s and DOD’s understanding of DOD’s contributions to this workforce, including the number of individuals who completed DOD’s FAA-approved A&P training program and subsequently obtained mechanic certification from FAA. Supply and demand data. Other federal agencies, such as BLS and DOD, maintain data that relate to this workforce more broadly that could be useful to FAA. For example, FAA could leverage BLS data on the projected employment of certificated and non-certificated aircraft mechanics and service technicians in conjunction with its data on newly certificated workers each year to better understand worker supply and demand. DOD also maintains information on separating service members with aviation maintenance backgrounds, who may be attractive to the commercial aviation industry. For example, according to a DOD official, in fiscal year 2018 over 22,000 service members with aviation maintenance backgrounds separated from the Air Force and Navy. Additional data analysis and coordination could potentially yield useful information on worker supply and demand and areas for promoting the development of this workforce. Without routinely analyzing its existing data on certificated workers, collecting additional data, or leveraging existing workforce data maintained by other federal agencies, FAA will not have certain information it needs to measure progress and strategically target its resources toward its objective of promoting the development of a robust aviation workforce. A robust aviation workforce, including certificated mechanics and repairmen, is necessary for maintaining a safe aviation system. FAA’s recently developed Aviation Workforce Steering Committee presents the agency with an opportunity to engage other federal agencies in discussions on how to leverage data to expand and diversify this workforce. Even as FAA’s strategic plan states the agency’s focus on promoting the development of a skilled aviation maintenance workforce to integrate new technologies, the agency has acknowledged that the current curriculum requirements for AMT Schools and mechanic testing standards are outdated. Efforts to revise the curriculum requirements for AMT Schools are ongoing through the rulemaking process, and FAA is also currently updating the testing standards for mechanics. The curriculum requirements for AMT Schools have remained largely unchanged for several decades despite numerous attempts to update them as aviation technology has evolved. The minimum requirements are established in regulation and list the subjects that AMT Schools must include in their training curriculum for individuals to be eligible to take FAA’s mechanic tests. FAA officials, employers, and AMT School officials we interviewed said the current curriculum requirements do not emphasize commonly used modern aircraft technologies, such as avionics and composite materials. Because the curriculum requirements are established in federal regulation, FAA has attempted several times to revise them through the rulemaking process. Table 1 provides selected changes or actions relating to these requirements. FAA officials noted several challenges to updating the AMT School curriculum requirements, including competing demands at the department level, the extent of comments FAA has received from stakeholders in response to proposed changes, and the amount of time required to coordinate with internal stakeholders during the review process. We previously reported on factors that affect the amount of time needed to issue a rule for selected agencies, which included similar challenges such as the complexity of an issue, agency management priorities, and the amount of review required at different phases of the rulemaking process. In October 2015, FAA published a notice of proposed rulemaking (NPRM) with the stated goal of updating the existing AMT School curriculum and providing an efficient means of changing specific course items by including them in each school’s operations specifications (see fig. 6). This would eliminate the need to go through the federal rulemaking process to update the curriculum. As part of its ongoing efforts to revise the curriculum requirements for AMT Schools through the rulemaking process, FAA issued a supplemental NPRM in the April 2019 Federal Register that expanded the scope of the NPRM it issued in October 2015 (see fig. 6). Comments on the supplemental NPRM were due in June 2019. As of October 2019, FAA officials said they were in the process of reviewing the comments. In a separate effort outside of the rulemaking process, FAA is currently updating the testing standards for mechanics. The standards were last revised in 2015. FAA has acknowledged that current mechanic testing standards are also outdated. As a result, aviation stakeholders have stated that the mechanic tests include outdated or irrelevant questions. For example, the practical test may include projects on wood airframes and fabric coverings, which are not common to modern commercial aircraft. FAA has stated that the revised testing standards will provide a comprehensive framework for the mechanic tests and serve as a guide for reviewing and revising the oral and written test questions and the practical projects. FAA officials said two offices within the agency are responsible for updating AMT School training curriculum requirements and mechanic testing standards and that these offices have been coordinating efforts to align the two. FAA’s efforts to modernize the curriculum requirements for AMT Schools and its efforts to update the mechanic testing standards started on slightly different paths, in part due to differences in when the working groups were formed and recommendations to address these issues were made (see fig. 7). As of October 2019, FAA had not issued a final rule for modernizing AMT School curriculum requirements as required by the FAA Reauthorization Act of 2018, and it was still in the process of updating testing standards. FAA officials have indicated that they have informed the appropriate committees in Congress that the proposed schedule for issuance of a final rule is in October 2020 and said that the revised mechanic testing standards would likely be finalized after the publication of the final rule amending the curriculum requirements for AMT Schools. An FAA official noted that any delay in finalizing the rule would likely result in a corresponding delay to finalizing the testing standards. Delaying the release of the updated mechanic testing standards could result in the prolonged use of outdated or irrelevant questions on the mechanic tests. FAA officials said that once finalized and implemented, the updated curriculum requirements for AMT Schools and the mechanic testing standards for individuals should be mostly aligned. A sufficient supply of aviation maintenance workers is critical to maintaining a safe and robust aviation system and meeting the growing demand for air travel. Current training and skills requirements for these workers are also important because of changing flight technology. Both the federal government and other industries benefit from having a professional, trained, and qualified workforce, and addressing aviation workforce needs is a shared responsibility among these different stakeholders. As the federal agency responsible for certificating aircraft mechanics and repairmen, FAA maintains certain demographic information on these individuals that could shed light on the characteristics and employment of these personnel. However, without strategically using or analyzing the data it has along with data other stakeholders collect, FAA will not have certain information it needs to target its resources or measure and improve progress toward its aviation workforce goals. It may also miss the opportunity to provide other stakeholders with valuable information for supporting these workers. Other agencies and stakeholders may also assist FAA in understanding and promoting the development of the aviation maintenance workforce. FAA’s recently developed Aviation Workforce Steering Committee presents the agency with an opportunity to engage other federal agencies to explore potential data sources and their usefulness and discuss ways to expand, diversify, and strengthen career pathways for the aviation maintenance workforce. The Administrator of FAA should direct the Aviation Workforce Steering Committee, as part of its ongoing efforts, to take steps to use existing FAA data and coordinate with other federal agencies to identify and gather the information it needs to measure progress and target resources toward its goal of promoting a robust, qualified, and diverse aviation maintenance workforce. For example, FAA could task a committee working group with developing and implementing ways to improve data sharing among federal agencies to inform decision-making on how to strengthen career pathways and better understand the supply and demand of certificated workers. (Recommendation 1) We provided a draft of this report to DOT, DOL, Education, DOD, and VA for review and comment. DOT provided written comments, which are reprinted in appendix II. DOT concurred with our recommendation. Specifically, DOT agreed that using existing data could potentially contribute to its efforts to develop the aviation maintenance workforce. DOT said it will ask the Aviation Workforce Steering Committee to consider using existing FAA data and to coordinate with other federal agencies regarding other potential data sources to support the FAA’s aviation maintenance workforce goals. DOL provided technical comments, which we incorporated in the report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, the Secretary of the Department of Labor, the Secretary of the Department of Education, the Secretary of the Department of Defense, the Secretary of the Department of Veterans Affairs, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or gurkinc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. While no single metric can be used to determine whether a labor shortage exists, certain indicators in conjunction with views of stakeholders can provide insight on this issue. We previously found, based on our review of economic research, that an occupation experiencing a labor shortage would exhibit the following: (1) a low unemployment rate signaling limited availability of workers in that profession, (2) increases in wages offered to draw people into that profession, and (3) increases in employment due to increases in demand for that occupation. Table 2 shows these specific indicators from 2013 through 2018 for the aircraft mechanics and service technicians occupation, measured using Bureau of Labor Statistics (BLS) Current Population Survey data. According to our analysis of BLS data from 2013 through 2018, unemployment rate and change in median wage earnings for the aircraft mechanics and service technicians occupation, which includes both Federal Aviation Administration-certificated and non-certificated workers, were consistent with the existence of hiring difficulties, while the percent change in employment was not. Data on two of the three indicators (unemployment rate and wage earnings) were consistent with difficulties in hiring aircraft mechanics and service technicians. However, because these data combine information for certificated and non-certificated workers, it is difficult to know the extent to which any hiring difficulties represent demand for certificated workers, specifically. In addition, the indicators should be viewed with appropriate caveats. For example, while median wages increased for aircraft mechanics and service technicians in 2018 compared to 2013, they did not increase in every year—and they exhibited decreases of as much as 6.7 percent. The direction of change of the employment indicator was not consistent with hiring difficulties for this occupation. As shown in table 2, from 2013 through 2018, employment for aircraft mechanics and service technicians does not appear to have changed, while employment for all other occupations increased. However, employment for this occupation did not remain constant in every year over that time period and exhibited increases of as much as 12.5 percent and decreases of as much as 21.9 percent. Chelsa Kenney Gurkin at (202) 512-7215 or Gurkinc@gao.gov. In addition to the contact named above, Betty Ward-Zukerman (Assistant Director), Meredith Moore (Analyst-in-Charge), Ellie Klein, and Chris Woika made key contributions to this report. Additional assistance was provided by James Bennett, Lilia Chaidez, Holly Dye, Serena Lo, Sheila R. McCoy, John Mingus, Michael Naretta, James Rebbe, Almeta Spencer, and Andrew Von Ah.", "summary": "FAA requires that only mechanics who are “certificated” by the FAA approve aircraft for return to service. The required training to become a certificated mechanic can take between 1 and 3 years. FAA also oversees the certification of repairmen who work on aircraft parts. Some stakeholders have expressed concern that retirements and attrition could adversely affect the capacity of this workforce to meet the growing demand for air travel, and that mechanic curriculum is outdated. The FAA Reauthorization Act of 2018 included provisions for GAO to examine the aviation workforce. This report examines (1) what available federal data reveal about the FAA-certificated aviation maintenance workforce; (2) how selected government agencies, educational institutions, and businesses provide support and coordinate to develop the aviation maintenance workforce; and (3) the progress FAA has made in updating its curriculum and testing standards for mechanics. GAO analyzed FAA data on certificate holders as of December 2018; reviewed BLS employment data for 2013 through 2018; reviewed relevant federal laws and regulations; and interviewed selected federal agency, industry, and AMT School officials. Federal data provide an incomplete picture of the Federal Aviation Administration (FAA)-certificated aviation maintenance workforce. A sufficient supply of certificated workers is critical for safety and to meet the growing demand for air travel. However, supply and demand data for certificated workers are limited. FAA maintains data on the number of individuals newly certificated each year (see figure), but less is known about how many certificated individuals exit the aviation industry each year and the extent of growing demand. Bureau of Labor Statistics (BLS) data provide some information on pay and demand for aviation maintenance workers more broadly, but do not differentiate between FAA-certificated and non-certificated workers due to data collection challenges. Demographic data may also be useful for workforce analysis and planning. FAA data provide some demographic information on certificated mechanics and repairmen, such as age and sex, but the agency lacks data on race and ethnicity. According to GAO analysis of FAA data, over half of the roughly 330,000 mechanics and repairmen FAA had certificated as of December 2018 were between 50 and 70 years old and 97 percent were men. Government agencies, educational institutions, and businesses coordinate to some extent in support of this workforce, but FAA lacks certain information—including information maintained by other agencies that administer related programs—that could advance its workforce development efforts. One of FAA's strategic objectives includes promoting the development of a robust, skilled aviation workforce, and the agency established a committee, in part, to explore ways to diversify this workforce; however, FAA is not currently positioned to understand whether its efforts are optimally targeted or effective. Without routinely analyzing its own data or leveraging others' data, FAA may not have certain information it needs to track or ensure progress toward its workforce development goals. FAA has acknowledged that curriculum requirements for Aviation Maintenance Technician (AMT) Schools and mechanic testing standards are outdated. Efforts to revise the curriculum requirements for AMT Schools are ongoing and FAA officials told GAO that a final rule will be published some time toward the end of 2020. FAA officials indicated that the revised mechanic testing standards would likely be finalized after. GAO recommends that FAA use its existing data and coordinate with other federal agencies to identify and gather information to measure progress and target resources toward its goal of promoting a robust, qualified, and diverse aviation maintenance workforce. FAA agreed with the recommendation.", "document_type": "gao"}
{"report": "CMS and states jointly administer the Medicaid program and generally share in the financing of Medicaid payments according to a formula established in law. States may deliver health care services to Medicaid beneficiaries through fee-for-service payments to participating providers or through Medicaid managed care plans, through which states pay plans a fixed amount per beneficiary—typically per member per month—to provide a specific set of Medicaid-covered services. States finance their share (nonfederal share) of Medicaid program spending in a variety of ways, including state funds, such as state general funds appropriated to the state Medicaid program and funds collected through taxes levied on health care providers. Within limits, however, states may also use other sources of funds—including funding from local government providers, such as county-owned or county-operated hospitals, or from local governments on behalf of government providers. Federal law allows states to finance up to 60 percent of the nonfederal share of Medicaid payments from local government funds. State Medicaid agencies have two primary mechanisms for making payments to hospitals—base payments and supplemental payments— and both can qualify for federal matching funds. Base payments are payments to hospitals for specific services provided to Medicaid beneficiaries through both fee-for-service and managed care. These payments are set by state Medicaid programs or managed care plans, and can vary considerably across states for the same services. Payment amounts for the same service may also vary within a state. States’ Medicaid base payments are typically lower than other payers’, and often are below the costs of providing services. Supplemental payments are typically lump sum payments made to hospitals that are not specifically tied to an individual’s care. Like all Medicaid payments, supplemental payments are required to be economical and efficient. Supplemental payments can be grouped into two broad categories: (1) DSH payments, which states are required to make to certain hospitals; and (2) non-DSH payments, which states are allowed to make, but are not required by law. DSH payments are designed to help offset uncompensated care costs for hospitals serving a high proportion of Medicaid beneficiaries and uninsured low-income patients. In fiscal year 2017, total DSH payments to hospitals nationally were about $18.1 billion. States may distribute DSH payments to any eligible hospital in the state; however, under federal law, the total amount of DSH payments to a hospital must not be more than the total amount of uncompensated care provided by the hospital (both the Medicaid shortfall and uncompensated costs for care for the uninsured). To be eligible for a DSH payment, hospitals must meet minimum requirements such as having a Medicaid inpatient utilization rate of at least 1 percent. States are required to make DSH payments to certain hospitals—termed deemed-DSH hospitals—with a Medicaid inpatient utilization rate of at least one standard deviation above the mean for hospitals in the state that receive Medicaid payments, or a low-income utilization rate that exceeds 25 percent. The amount of federal funding each state may claim for DSH payments is limited by federal law. Since fiscal year 1993, each state is subject to a federal DSH allotment that establishes the maximum federal funding available for the payments. A state’s DSH allotment is largely based on its fiscal year 1992 DSH spending, although Congress has since made several incremental adjustments to these allotments. Ultimately, however, the states that spent the most in fiscal year 1992 continue to have the largest allotments; conversely, the states that spent the least in fiscal year 1992 have the smallest allotments. States may choose to make DSH payments to institutions for mental disease (IMD), which can include state-operated psychiatric hospitals. Prior to 1997, a large share of DSH payments went to state-operated IMDs, where they were used to pay for services not covered by Medicaid and any remaining funds were returned to the state treasuries. In general, Medicaid excludes fee-for-service base payments for beneficiaries aged 21-64 who are residents of IMDs—called the IMD exclusion—and using DSH payments allowed states to support the costs of IMDs. In 1997, Congress restricted the total amount of DSH payments a state could make to IMDs as a group by establishing an annual limit on payments to IMDs for each state. Any unspent funds within the IMD-designated limit can be used for other hospital types. Non-DSH payments include four types of supplemental payments that states may make, but are not required to do so, to hospitals and other providers. Medicaid upper payment limit (UPL) payments are lump-sum payments that are made in addition to fee-for-service base payments. The UPL is a limit or ceiling on the amount of a state’s Medicaid payments for which the federal government will match spending. The UPL is based on the difference between Medicaid fee-for-service base payments and an estimate of what Medicare would pay for comparable services. The UPL is not a hospital-specific limit, but is applied in the aggregate across certain categories of providers. States have some flexibility in deciding which hospitals will receive a UPL payment, and how to allocate UPL payments among hospitals. In fiscal year 2017, UPL payments totaled nearly $13 billion. Uncompensated care pool payments are payments that some states make to hospitals specifically for uncompensated care costs in conjunction with section 1115 demonstration waivers and pilot projects for which they have received approval from the Secretary of HHS. Specifically, section 1115 of the Social Security Act authorizes the Secretary of HHS to waive certain federal Medicaid requirements and allow costs that would not otherwise be eligible for federal matching funds for experimental, pilot, or demonstration programs that, in the Secretary’s judgment, are likely to assist in promoting Medicaid objectives. States have received approval to make supplemental payments for hospital uncompensated care in their Medicaid programs. In fiscal year 2017, states reported total spending of about $8 billion through uncompensated care pools. Delivery system reform incentive payment (DSRIP) programs, which have also been authorized under section 1115 demonstrations, allow states to make supplemental payments to providers engaging in various improvement projects that align with state delivery system reform objectives. Examples of reform objectives include improving care for patients with specific conditions or increasing capacity. In fiscal year 2017, DSRIP program payments totaled about $7.3 billion. Graduate medical education payments help support teaching hospitals, and can include teaching costs, such as physician resident salaries, though states are not required to make such payments to teaching hospitals. States have significant flexibility in designing and administering these payments; however, the payments are subject to the UPL. In fiscal year 2017, Medicaid graduate medical education payments totaled about $2 billion. Effective January 1, 2014, PPACA allowed states to expand Medicaid eligibility to certain non-pregnant, non-elderly individuals. PPACA also required a phased reduction in DSH allotments to states, reflecting the expectation that the number of uninsured individuals would decline—and so would hospital spending on uncompensated care. As of May 2019, there were 37 “expansion states”—those states that chose to expand Medicaid eligibility—and 14 “non-expansion states”—those that did not choose to expand Medicaid. Congress has delayed the reduction in DSH allotments several times. The reductions are scheduled to begin in fiscal year 2020. Between 2013 and 2014, both expansion and non-expansion states reported different degrees of change in care for the uninsured and Medicaid shortfall. In particular, MACPAC reported that between 2013 and 2014, the year in which most state Medicaid expansions took effect, expansion states’ uncompensated care costs for the uninsured declined by $2.2 billion (19 percent), while non-expansion states’ uncompensated care costs for the uninsured increased by $0.6 billion (5 percent). During the same period, expansion states’ Medicaid shortfall increased by $2.2 billion (36 percent), and non-expansion states’ Medicaid shortfall increased by $1.8 billion (546 percent). States’ use of supplemental payments has grown in recent decades, partly due to the flexibility supplemental payments provide. This flexibility is twofold: supplemental payments provide states with flexibility in financing the nonfederal share of supplemental payments, and flexibility to target the payments to specific hospitals or types of hospitals. Total supplemental payments to hospitals have grown over time, while states’ base payments have often been frozen or reduced. Congress imposed limits on DSH spending in the 1990s, and since then states’ use of non-DSH payments has grown. Between fiscal year 2000 and fiscal year 2017, DSH payments increased about 16 percent, from $15.6 billion to $18.1 billion. In prior work, we reported that in fiscal year 2006 state Medicaid agencies made at least $6.3 billion in non-DSH payments, though the exact amounts are unknown, because states did not report all their payments to CMS. By fiscal year 2017, the amount of non-DSH payments had increased to $30.4 billion. Both uncompensated care pool payments and DSRIP programs are relatively new types of non-DSH payments, and thus contributed to the overall increase in supplemental payments. In prior work, we reported that, as of February 2017, CMS authorized nearly $38.7 billion in DSRIP spending nonconsecutively over 2011 to 2022 in four states with the largest DSRIP programs. Our prior work found that new or increased supplemental payments helped mitigate the increasing gap between Medicaid base payments and hospital costs. While supplemental payments increased, the number of states reducing or freezing base payments to hospitals has increased, in part, because states reported challenges paying the nonfederal share with state general funds. Our work found that from 2008 to 2011, across all providers, the number of states making at least one base payment reduction grew from 13 to 34, while the number of states increasing at least one base payment fell over the same period. Across all 4 years, states most frequently reported reducing base payments for hospitals. The Kaiser Family Foundation’s annual survey data shows the trend continued in more recent years. Specifically, over half of states froze or reduced inpatient hospital base payments each fiscal year from 2011 to 2018, ranging from a low of 28 states in 2011 and 2018, to a high of 39 states in 2012. (See table 1.) In a September 2018 study of five states, MACPAC found that hospitals and state Medicaid officials often prefer increases to supplemental payments rather than increases to base payments, because supplemental payments come with more predictability. MACPAC found that all five states reported reducing hospital base payments from 2007 to 2011. After 2011, all five states kept base payments frozen with no adjustment for inflation. As a result, base payments to hospitals in these states were lower in 2018 relative to other payers and hospital costs. To address the growing gap between base payments and hospital costs, states collaborated with hospitals to establish or increase supplemental payments. In the five states, supplemental payments ranged from 18 percent to 61 percent of total hospital payments. More often than with base payments, states have relied on sources other than state general funds to finance the nonfederal share of supplemental payments. For example, states may receive funds for the nonfederal share of supplemental payments through taxes levied on health care providers. (See fig. 1.) In previous work, we found that funds from local governments and health care providers constituted about 50 percent of the nonfederal share for DSH and non-DSH payments in fiscal years 2008 through 2012. In contrast, funds from local governments and health care providers constituted approximately 30 percent of base payments during the same time period. The MACPAC study of five states also found that states and hospitals preferred supplemental payments, because hospitals can track the extent to which their tax assessments are recouped through supplemental payments. In a July 2014 report, we found that the number of states relying on provider taxes increased, and that provider tax revenues were then used for the nonfederal share of supplemental payments. In particular, the total number of provider taxes increased from 119 taxes in 42 states in 2008 to 159 taxes in 47 states in 2012—a 34 percent increase. Kaiser Family Foundation data show this trend has continued. According to state survey data, the number of states using inpatient hospital provider taxes has steadily increased from fiscal year 2011 to 2018, ranging from a low of 34 states in 2011, to a high of 42 states in 2017 and 2018. (See table 2.) Supplemental payments provide states with flexibility that allows them to address states’ goals by targeting payments to particular hospitals or hospital types, such as public hospitals or teaching hospitals. States may choose to target supplemental payments to hospitals that may not have the highest uncompensated care costs. Our prior work found some states’ DSH payments were not proportionally targeted to hospitals with the highest uncompensated care costs, which DSH payments are designed to address. Based on our prior analysis of annual hospital-specific 2010 DSH data, we reported that in 30 of 42 states, hospitals receiving the largest share of state DSH payments did not provide the largest share of total uncompensated care. Moreover, our prior review of the independent DSH audits found that 41 states made DSH payments to 717 hospitals that exceeded the individual hospitals’ uncompensated care costs as calculated by the auditors, 9 states did not accurately calculate the uncompensated care costs of 206 hospitals in those states for purposes of making DSH payments, and 15 states made DSH payments to a total of 58 hospitals that either did not retain their DSH payments or were not qualified to receive them. States’ criteria for identifying eligible DSH hospitals and how much funding they receive vary, but were often related to hospital ownership, hospital type, and geographic factors. Our prior work found that 2006 DSH payments to individual hospitals varied widely, ranging from 1 cent to about $395 million. For example, California reported both the lowest and highest 2006 DSH payment amounts; the state made a total of only $160 in DSH payments to 96 private hospitals and paid $2 billion in DSH payments to 51 government hospitals. Based on our analysis of 2014 DSH audits, several states targeted DSH payments to certain hospitals and hospital types, including the following: Public hospitals: States targeting nearly all (93 percent or higher) of their DSH funding to public hospitals included Arkansas (99 percent), California (100 percent), Illinois (99 percent), Iowa (93 percent), Maine (100 percent), and Washington (97 percent). Nonprofit hospitals: Nebraska targeted 98 percent of its DSH funding to nonprofit hospitals. High-teaching hospitals: Arkansas targeted 98 percent of DSH funding to high-teaching hospitals, defined as teaching hospitals with an intern-and-resident-to-bed ratio of 0.25 or greater. IMDs: Maine makes DSH payments to the two state-run IMDs. In 2014, 18 states directed their entire IMD-designated DSH limit to IMDs. (For additional information on DSH payments to IMDs, see table 9 in app. II). Similarly, states can target UPL payments to certain hospitals. We and the HHS Office of the Inspector General have reported that some states concentrated these payments to a small number of providers. Our work has highlighted a number of concerns about the use of non- DSH payments from various perspectives, highlighting the need for transparent reporting, ensuring expenditures meet Medicaid purposes, and concerns regarding arrangements that shift costs from the states to the federal government. For example, in November 2012, we recommended that Congress consider requiring CMS to improve the transparency of and accountability for non-DSH payments by requiring facility-specific payment reporting and annual audits. The report noted that the annual DSH reports and audits that states began submitting in 2010 were important steps toward improving transparency and accountability for Medicaid DSH payments; however, similar information is lacking for non-DSH payments. Moreover, the report stated that the limited information available on non-DSH payments shows that a large share of these payments are paid to a small number of hospitals; when these payments are combined with Medicaid base payments, hundreds of hospitals may be receiving Medicaid payments well in excess of their actual costs of providing Medicaid services. As of March 2019, Congress has not taken any action, but CMS announced in fall 2018 that it was planning a proposed rule on supplemental payments that, if finalized, would improve transparency by requiring states to provide CMS with certain information on Medicaid supplemental payments. The agency plans to release the proposed rule for comment by fall 2019. In 2014, we recommended that CMS develop a data collection strategy ensuring states report accurate and complete data on all sources of funds used to finance the nonfederal share of Medicaid payments. Such data are needed to (1) track trends in financing the nonfederal share, and (2) oversee compliance with current limits on sources of financing the nonfederal share. CMS did not concur with our recommendation, but did acknowledge the agency does not have sufficient data to oversee compliance with the 60 percent limit on local government contributions to a state’s nonfederal share. Among hospitals receiving DSH payments in 2014, total uncompensated care costs varied by state, ranging from $5.9 million in North Dakota to $6.2 billion in New York. In the hospitals, most uncompensated care costs were related to costs to care for uninsured patients, rather than the Medicaid shortfall. For example, among hospitals receiving DSH payments in the 48 states studied: Costs related to care for the uninsured comprised about two-thirds (67.9 percent) of total uncompensated care costs for DSH hospitals. The remaining share of DSH hospital uncompensated care costs consisted of the Medicaid shortfall. In 34 states, costs for care for the uninsured exceeded the Medicaid shortfall. In the remaining 14 states, the Medicaid shortfall exceeded costs related to care for the uninsured. In 15 states, Medicaid paid hospitals more than the total cost of services provided to Medicaid beneficiaries, resulting in a surplus of Medicaid payments—even prior to receiving DSH payments. Termed a negative Medicaid shortfall, these surplus funds can be the result of non-DSH Medicaid supplemental payments. The remaining 33 states had some Medicaid shortfall. (See table 3.) No states had a surplus of total uncompensated care costs. (For additional information on state uncompensated care costs and DSH payments in 2014, see table 10 in app. II.) DSH payments—both the federal and nonfederal share—varied significantly in the amount that each state paid to hospitals in 2014. (See table 4 and fig. 2.) Wyoming made the smallest amount of DSH payments at about $500,000, while New York made the largest amount in DSH payments at $3.5 billion. Differences in DSH payments are largely the result of differences in the state allocations established in law. The proportion of total DSH hospital uncompensated care costs covered by total DSH payments in 2014 also varied considerably by state. Nationally, DSH payments ($18.3 billion) covered about half of DSH hospital uncompensated care costs ($36.2 billion). Nineteen states made DSH payments totaling at least 50 percent of uncompensated care costs for the states’ DSH hospitals, while 29 states made DSH payments of less than 50 percent of uncompensated care costs for the states’ DSH hospitals. (See table 5.) Four states (California, Illinois, Maryland, and Missouri) made DSH payments that exceeded aggregate hospital uncompensated care costs. (For additional information on state uncompensated care costs and DSH payments in 2014, see table 10 in app. II.) Among hospitals receiving them, DSH payments accounted for 13.6 percent of total Medicaid payments, nationally, but there was considerable variation across states. For example, DSH payments comprised 96.6 percent of Medicaid payments to DSH hospitals in Maine and 0.7 percent of Medicaid payments to DSH hospitals in Tennessee. In 40 states, DSH payments accounted for less than 20 percent of total Medicaid payments to hospitals, but in 8 states, it exceeded 20 percent. (See table 6.) (For additional information on state Medicaid payments to hospitals, see table 11 in app. II.) Among deemed and non-deemed DSH hospitals, overall deemed-DSH hospitals received larger relative DSH payments compared to non- deemed DSH hospitals. Deemed-DSH hospitals received 69.9 percent of DSH payments in 2014, but carried 51.2 percent of uncompensated care costs, relative to all hospitals receiving DSH payments that year. Each of the 48 states that distributed DSH payments in 2014 had at least one deemed-DSH hospital. (See table 7 for hospital type definitions.) Most of these states (36) provided deemed-DSH hospitals with a greater share of DSH payments relative to their share of total uncompensated care costs. (See table 8 for a summary of how states’ DSH payments to deemed-DSH hospitals compared to the hospitals’ share of uncompensated care costs, and table 12 in app. II for additional information by state.) In terms of ownership and teaching hospital status, hospitals that were publicly owned or teaching hospitals also generally received a greater proportion of DSH payments relative to their share of total uncompensated care costs. Among the three different ownership groups (public, non-profit, and private), public hospitals generally received a larger share of DSH payments relative to their share of uncompensated care. Among hospitals receiving DSH payments in 2014, public (36.7 percent) and nonprofit (53.7 percent) hospitals accounted for more uncompensated care costs than that of privately owned hospitals (9.6 percent). States generally provided more DSH payments to public hospitals (62.8 percent) relative to their share of total uncompensated care costs (36.7 percent). (For additional information on DSH payments and hospitals’ uncompensated care costs by ownership, see table 13 in app. II.) States distribute DSH payments to teaching hospitals at different rates, but generally provided a greater proportion of DSH payments to high-teaching hospitals (56.5 percent) relative to their share of total DSH hospital uncompensated care costs (44.0 percent). (For additional information on DSH payments and hospitals’ uncompensated care costs by hospital teaching status, see table 14 in app. II.) Nationally, urban hospitals received a greater share of DSH payments relative to rural hospitals, with 89.6 percent of DSH funds distributed to urban hospitals and the remaining 10.4 percent distributed to rural hospitals. This proportion corresponds to a similar distribution of uncompensated care costs, with 88.2 percent of uncompensated care costs among DSH hospitals carried by urban hospitals and the remaining 11.8 percent carried by rural hospitals. (For additional information on DSH payments and hospitals’ uncompensated care costs by urban/rural status, see table 15 in app. II.) For additional information on variation in uncompensated care and DSH payments by hospital category and sole community provider status, and state characteristics, see tables 16 through 19 in appendix II. We provided a draft of this report to HHS for review. HHS provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to appropriate congressional committees, to the Secretary of Health and Human Services, the Administrator of CMS, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114, or yocomc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This bibliography contains citations for the eight Kaiser Family Foundation reports referenced in the report. Kaiser Family Foundation and Health Management Associates. States Focus on Quality and Outcomes Amid Waiver Changes: Results from a 50-State Medicaid Budget Survey for State Fiscal Years 2018 and 2019. Washington, D.C.: Kaiser Family Foundation, and National Association of Medicaid Directors, October 2018. Kaiser Family Foundation and Health Management Associates. Medicaid Moving Ahead in Uncertain Times: Results from a 50-State Medicaid Budget Survey for State Fiscal Years 2017 and 2018. Washington, D.C.: Kaiser Family Foundation, October 2017. Kaiser Family Foundation and Health Management Associates. Implementing Coverage and Payment Initiatives: Results from a 50-State Medicaid Budget Survey for State Fiscal Years 2016 and 2017. Washington, D.C.: Kaiser Family Foundation and National Association of Medicaid Directors, October 2016. Kaiser Family Foundation and Health Management Associates. Medicaid Reforms to Expand Coverage, Control Costs and Improve Care: Results from a 50-State Medicaid Budget Survey for State Fiscal Years 2015 and 2016. Washington, D.C.: Kaiser Family Foundation and National Association of Medicaid Directors, October 2015. Kaiser Family Foundation and Health Management Associates. Medicaid in an Era of Health & Delivery System Reform: Results from a 50-State Medicaid Budget Survey for State Fiscal Years 2014 and 2015. Washington, D.C.: Kaiser Family Foundation, and National Association of Medicaid Directors, October 2014. Kaiser Family Foundation and Health Management Associates. Medicaid in a Historic Time of Transformation: Results from a 50-State Medicaid Budget Survey for State Fiscal Years 2013 and 2014. Washington, D.C.: Kaiser Commission on Medicaid and the Uninsured, Kaiser Family Foundation, October 2013. Kaiser Family Foundation and Health Management Associates. Medicaid Today; Preparing for Tomorrow: A Look at State Medicaid Program Spending, Enrollment and Policy Trends Results from a 50-State Medicaid Budget Survey for State Fiscal Years 2012 and 2013. Washington, D.C.: Kaiser Commission on Medicaid and the Uninsured, Kaiser Family Foundation, October 2012. Kaiser Family Foundation and Health Management Associates. Moving Ahead Amid Fiscal Challenges: A Look at Medicaid Spending, Coverage and Policy Trends Results from a 50-State Medicaid Budget Survey for State Fiscal Years 2011 and 2012. Washington, D.C.: Kaiser Commission on Medicaid and the Uninsured, Kaiser Family Foundation, October 2011. To conduct this analysis, we used data compiled by Acumen for the Medicaid and CHIP Payment and Access Commission. These data consist of measures from several sources. The measures used within this report were collected from state disproportionate share hospital (DSH) audits and Medicare cost reports. The 2014 DSH audits, which report data on hospital uncompensated care costs and DSH payments to hospitals, were submitted by 48 states and the District of Columbia. These data do not include a census of all hospitals, but only those hospitals that were reported in the 2014 DSH audits. As a result, these data do not capture all uncompensated care costs in each state, only uncompensated care costs for those hospitals reported in the 2014 DSH audits. Two states, Massachusetts and Hawaii, did not submit a 2014 DSH audit, because they did not make DSH payments. Additionally, while South Dakota submitted a 2014 DSH audit, we excluded the state from our analysis because of concerns about the reliability of the reported cost measures. In addition, not all hospitals reported every data element we analyzed. As a result, total uncompensated care costs and total DSH payments vary between tables, as hospitals were excluded from a given table if they did not report the characteristic described by the table. The numbers of hospitals excluded because they did not report a given data element are noted in each table for which this is the case. Likewise, as uncompensated care costs are an important focus of the report, we also excluded from all analyses 13 hospitals that did not report a value for uncompensated care costs. In addition to the contact named above, Lori Achman (Assistant Director), Dawn Nelson (Analyst-in-Charge), Sean Miskell, and Jeffrey Tamburello made key contributions to this report. Also contributing were Tim Bushfield, Drew Long, Vikki Porter, and Emily Wilson.", "summary": "Medicaid, the joint federal-state program that finances health care coverage for low-income and medically needy individuals, spent an estimated $177.5 billion on hospital care in fiscal year 2017. About a quarter ($46.3 billion) of those hospital payments were supplemental payments—typically lump sum payments made to providers that are not tied to a specific individual's care. States determine hospital payment amounts within federal limits. In fiscal year 2017, DSH payments totaled about $18.1 billion. Beginning in fiscal year 2020, the amount of DSH payments each state can make is scheduled to be reduced. GAO was asked to study Medicaid DSH payments to hospitals. Among other things, GAO examined hospital uncompensated care costs and DSH payments by state Medicaid program and hospital characteristics. GAO analyzed data from the 2014 DSH audits—states' independently audited and certified reports of hospital-level uncompensated care costs and DSH payments—from 47 states and the District of Columbia (48 states). Three states were excluded from the analysis because they either did not make DSH payments or the submitted data were unreliable. The 2014 data were the most recently available audited, hospital-specific, data at the time of GAO's analysis. We provided a draft of this report to HHS for review. HHS provided technical comments, which we incorporated as appropriate. Medicaid disproportionate share hospital (DSH) payments are one type of supplemental payment and are designed to help offset hospitals' uncompensated care costs for serving Medicaid beneficiaries and uninsured patients. Under the Medicaid DSH program, uncompensated care costs include two components: (1) costs related to care for the uninsured; and (2) the Medicaid shortfall—the gap between a state's Medicaid payment rates and hospitals' costs for serving Medicaid beneficiaries. GAO's analysis of hospitals receiving DSH payments showed that in 2014, costs related to care for the uninsured comprised 68 percent of total uncompensated care costs, and the remaining 32 percent was the Medicaid shortfall. Across states, GAO found that total DSH payments varied significantly in 2014. DSH payment levels are generally tied to state DSH spending in 1992 and since 1993 states have been subject to a limit on the amount of federal funding that may be used for DSH payments. Medicaid DSH payments covered 51 percent of the uncompensated care costs. In 19 states, DSH payments covered at least 50 percent of uncompensated care costs. DSH payments comprised about 14 percent of total Medicaid payments, yet wide variation existed. For example, DSH payments comprised about 97 percent of Medicaid payments to DSH hospitals in Maine and 0.7 percent of Medicaid payments to DSH hospitals in Tennessee. Some types of hospitals received a greater proportion of DSH payments relative to their share of total uncompensated care costs. For example, states generally provided more DSH payments to public hospitals (in comparison to private and non-profit hospitals) and teaching hospitals (as compared to non-teaching hospitals) relative to their share of total uncompensated care costs.", "document_type": "gao"}
{"report": "FAA air traffic controllers are responsible for guiding aircraft that are departing, landing, and moving around the terminal area at 518 U.S. airports. Airport terminal areas include “movement areas,” such as runways and taxiways, and “non-movement areas” such as ramp areas (see fig. 1). Incidents can occur in either the movement or non-movement area and include: Runway incursions: These incidents involve the incorrect presence of an aircraft, vehicle, or person on a runway. Incursions fall into three categories—pilot deviations, operational incidents, and vehicle or pedestrian deviations—depending on their cause (see fig. 2). Runway excursions: These incidents occur when an aircraft veers off the side, or overruns the end, of a runway. Wrong-surface: These incidents occur when an aircraft lands or departs, or tries to land or depart, on the wrong runway or on a taxiway (see fig. 3). Wrong surface incidents also include when an aircraft lands or tries to land at the wrong airport. Ramp area: These incidents occur when aircraft, vehicles, or people cause damage or injuries in the ramp area. FAA oversees the safety of runways and taxiways and works with partners such as airlines, airports, pilots, and others to improve safety in these areas. FAA’s oversight of ramp areas is generally exercised indirectly through its certification of airports and airlines, which have been more directly responsible for safety in these areas. Several FAA offices—with staff in D.C. headquarters, FAA regional offices, and local district offices—oversee terminal area safety, including: The Air Traffic Organization (ATO) manages air traffic control, validates reports of terminal area incidents, develops and maintains runway safety technology, and leads investigations of operational incidents. ATO also administers the mandatory reporting system, which requires air traffic controllers to report certain incidents, including runway incursions, excursions, and wrong surface landings. ATO’s Runway Safety Group leads and coordinates all FAA terminal area safety efforts. The goal of the Runway Safety Group is to improve runway and taxiway safety by reducing the risk of runway incursions, excursions, and other incidents. The Office of Airports oversees airport-related safety, including inspecting and certifying operations at commercial airports and establishing airport design and safety standards. The Office of Airports also provides grants to airports to help support safety improvements, and leads investigations of incursions caused by vehicle/pedestrian deviation. Office of Aviation Safety investigates aircraft incidents and accidents, sets aviation safety standards, and certifies aircraft and pilots. Office of Aviation Safety, Flight Standards Service (Flight Standards) inspects and certifies airlines, promotes runway safety initiatives, and provides policies and guidance for pilots. Flight Standards also administers a reporting program to obtain information on incidents involving pilots and leads investigations of incursions caused by pilot deviation. Office of Aviation Safety, Accident Investigation and Prevention oversees investigations of terminal area-safety accidents and incidents, a role which includes coordinating with the NTSB, OSHA, and other FAA offices. Runway and taxiway safety has long been a focus of FAA efforts. FAA’s fiscal year 2019-2022 strategic plan establishes four safety initiatives related to its data-driven, risk-based safety oversight approach, known as a Safety Management System (SMS), including two fiscal year 2019 safety initiatives: proactively addressing emerging safety risk by using data-informed approaches to make risk-based decisions, and reducing the risk of runway incursions and wrong surface incidents. Further, FAA’s SMS guides its terminal area oversight. For example, FAA’s order establishing the Runway Safety Program states that FAA use SMS to ensure the safety of the national airspace through evaluations, data tracking, and analysis of incidents to identify new hazards and risks, and to assess existing safety controls. In our 2011 report on FAA’s oversight of terminal area safety, we made three recommendations related to excursions, ramp areas, and information sharing, all three of which FAA has since implemented. FAA uses data from reports and investigations to analyze runway incursions. For example, a team of representatives from the Air Traffic Organization, the Office of Airports, and the Office of Flight Standards, uses information on each incursion to classify its severity into one of four categories—A through D. An example of a category A incursion occurred in June 2018 in Springfield, Missouri, when an aircraft with 53 people on board accelerated for takeoff before noticing an airport operations vehicle crossing the runway. No injuries or damage were reported, but a collision was narrowly avoided. An example of a Category C or D incursion is a pilot entering a runway without authorization, but without significant potential for a collision. FAA reports the rate of severe category A and B incursions to Congress and the public in its annual performance plan. FAA also uses data to analyze runway incursions over time. For example, FAA data show that the number and rate of reported runway incursions nearly doubled from 954 in fiscal year 2011 to 1804 in fiscal year 2018 (see fig. 4). The majority of reported runway incursions (62 percent) were pilot deviations followed by operational incidents (20 percent) and vehicle/pedestrian deviations (18 percent). According to our analysis of FAA data, the increase in reported incursions was largely due to an increase in less severe incursions. Our analysis showed that severe incursions (category A and B) in which there is a significant potential for a collision, are relatively infrequent. Category C and D incursions, in which there is less potential for a collision, are more frequent. According to FAA officials, the increase in less severe incursions may be due to increased reporting of these incidents, which we also noted in our 2011 report on terminal area safety. However, the number and rate of reported runway incursions has continued to steadily increase since then, and may also indicate an increase in the actual occurrence of incidents. In 2017, FAA developed a new metric to analyze excursions and other incidents, as well as incursions. According to FAA officials, the new metric (“Surface Safety Metric”) measures the relative riskiness of terminal area incidents by assigning a different severity weight to each incursion, excursion, or other incident depending on its proximity to a fatal accident. For example, FAA documentation states that the new metric assigns a severity weight of 1 to incidents that result in a fatal injury, 0.6 to incidents with serious injuries, and 0.3 to incidents with minor injuries. Incidents in which there are no injuries are assigned even lower severity weights—for example 0.003 for a category A incursion and 0.002 for a category B incursion. FAA officials said they will analyze these severity weights year-to-year, so they can identify trends in each type of incident and across all incidents. For example, FAA officials noted that despite an increase in the number of runway incursions from fiscal years 2011 through 2018, the estimated risk of these incidents, as measured by their severity weights, declined. FAA has developed new performance goals tied to this metric, which it plans to report to Congress and the public by the end of fiscal year 2019. FAA has analyzed excursion data through special FAA task teams and other joint industry efforts with airlines, associations, and other government agencies. Excursions occur when an aircraft veers off the side or end of a runway, and can result in serious injury, death, or property damage. For example, on September 27, 2018, a small aircraft slid off the side of the runway at Greenville Downtown Airport in South Carolina shortly after landing. The aircraft continued down a 50-foot cliff, resulting in the deaths of two people. According to data FAA provided to us, nearly 700 excursions were reported in fiscal year 2018. Additionally, several joint industry efforts and special task teams have recently analyzed excursions. For example, the Commercial Aviation Safety Team (CAST), which FAA co-leads, found that about a third of the commercial accidents in the U.S. that resulted in fatalities or irreparable damage to the aircraft from 2006 through 2015 were attributed to runway excursions. In 2013, FAA began collecting additional data on excursions, but our review of FAA’s data found the excursion data FAA has collected since then contain duplicates. In 2011, we found that FAA was not formally tracking runway excursions and recommended that FAA develop a plan to track and assess them, which FAA began doing in 2013. Prior to 2013, FAA collected excursion data from two sources—the NTSB Aviation Accident Database, which contains information gathered during NTSB investigations, and FAA’s own Aviation Safety Information Analysis and Sharing (ASIAS) database, which includes information on incidents that may not reach the level of an NTSB investigation, such as an incident without serious injuries or fatalities. In 2013, FAA began identifying excursions in a third source—mandatory occurrence reports that FAA requires air traffic controllers to file when they observe an incident. FAA officials said that the additional excursions they identified through these mandatory occurrence reports added 15 percent more annual reports to those that they had identified through only the other two sources. However, FAA officials said there are likely duplicate records in their excursion data as a single excursion could be reported in more than one of these three sources. Although we did not have enough identifying information in the excursion data FAA provided to confirm the number of duplicate reports, our analysis of excursion data did identify possible duplicates. Further, despite containing possible duplicates, FAA recently began using these excursion data in its new surface safety metric. Federal standards for internal control state that data should be appropriate, current, complete, and accurate. A 2017 FAA internal analysis also noted the importance of identifying duplicates in order to ensure accurate runway excursion data. FAA officials said that they do not know how many duplicate records there are, and that they do not have an automated way to identify (and remove) all duplicates. FAA officials said that they could manually identify and remove duplicates, but that they do not currently do this nor plan to do so because duplicate excursion records would not affect their assessment of excursion risk. FAA officials said that excursions captured solely by the mandatory occurrence reports tend to be minor, lower-risk events. However, without a process to identify duplicates, FAA is not able to verify that this statement is true, and therefore cannot accurately assess and mitigate the risk excursions pose to terminal area safety. FAA does not use data to analyze most ramp area incidents, and does not plan to do so in its new surface safety metric. While the manager of the Runway Safety Group said FAA analyzes fatal ramp accidents through its participation in CAST, it does not analyze non-fatal ramp incidents, which are estimated to occur more frequently. In addition to some airport and airline officials telling us that they likely collect ramp data, FAA’s Runway Safety Group manager said that FAA likely has data on some non-fatal ramp incidents. For example, some air traffic controllers we interviewed said that they would report any ramp area incidents they observed through FAA’s mandatory reporting process, and officials from a pilot association told us they would also report such incidents. However, FAA officials said that FAA does not plan to analyze ramp incidents in the agency’s new surface safety metric. FAA’s Runway Safety Program Manager said that FAA has not analyzed most ramp area incidents because the risk of these incidents is lower than that in other areas, such as runways, and therefore does not merit analysis. For example, the manager said that aircraft speed in the ramp area is generally slower than take-off or landing speed, and fatalities are infrequent. However, we have previously reported that ramp areas are typically small, congested areas in which departing and arriving aircraft are serviced by ramp workers, who include baggage, catering, and fueling personnel. These areas can be dangerous for ground workers and passengers. The Flight Safety Foundation, which has collected its own data on ramp safety, estimated that each year 27,000 ramp accidents and incidents occur worldwide and can be costly due to effects such as damage to aircraft and schedule disruptions. In addition, ramp areas are complex because safety responsibilities in these areas vary by airport and even by terminal. For example, officials at Boston Logan International Airport told us that the airport operator shares some responsibilities with airlines but maintains control over all ramp areas. By contrast, officials at Los Angeles International Airport told us that in terminals leased by individual airlines, the airline controls the ramp area, while the airport operator controls the ramp areas in terminals where multiple airlines operate. Officials from the Air Line Pilots Association told us that ramp areas are the “scariest part of airports.” One official gave an example of inconsistencies between airports that can cause confusion and risk, such as some airport ramp areas being marked with painted lines while others are not. Federal internal control standards state that data should be appropriate, current, complete, and accurate. In addition, FAA’s own SMS calls for FAA to use a data-driven approach to analyze safety risks so that it can control that risk. As part of those efforts, FAA began the rulemaking process in 2010 to require airports to implement SMS, through which airports would analyze risks in runways, taxiways, and ramp areas, but as of August 2019 this rule had not been finalized. Although some airport officials we interviewed said they are voluntarily implementing SMS and could be collecting data on ramp area incidents, FAA—with its role in overseeing safety at all commercial airports—is better positioned to take steps to analyze ramp incidents across all U.S. airports. For example, an individual airport implementing SMS would analyze ramp area incidents at that airport, but FAA could analyze ramp area incidents and identify trends across hundreds of airports as it does for other terminal area incidents described above. Beginning to analyze ramp area incidents, for example in its new metric, would provide FAA with information necessary to mitigate ramp area incidents and ensure that it is directing its efforts to the riskiest parts of the terminal area. FAA, airports, and airlines have implemented multiple efforts, including technologies, to improve runway, taxiway, and ramp safety; FAA’s efforts, which are coordinated by the Runway Safety Group, focus primarily on runway and taxiway safety. FAA’s primary runway and taxiway safety effort is the Runway Safety Program, whereby staff develop national and regional runway safety plans, analyze data on runway and taxiway incidents, and help local air traffic control managers organize annual Runway Safety Action Team (RSAT) meetings at which FAA, airport operator, and other stakeholders at each airport discuss recent runway and taxiway incidents. Prior to each RSAT, FAA’s Regional Runway Safety Program Managers we met with told us they compile and share available information on each incident that occurred in the last year at the airport with the local air traffic manager. This information may include trends in incursions, the location of each incident on an airport map, and results from vehicle/pedestrian deviation investigations conducted by the FAA Office of Airports. Each air traffic manager then presents this information to attendees, who may include staff from FAA’s Office of Airports or Flights Standards, the airport operator, and local pilots. Participants discuss the prior year’s incidents, identify risks, and develop a plan to mitigate these risks. For example, attendees at an RSAT in Phoenix, Arizona, discussed risk factors that could be contributing to pilot deviations, and identified that pilots could be missing taxiway markings that instruct pilots to stop before proceeding onto a runway. Consequently, these RSAT attendees developed a plan to add lights to the surrounding area to improve visibility. The attendees also tasked air traffic managers with developing a program to provide annual tours of the tower and airfield to local pilots and personnel working on the airfield to show both parties what the other sees during flight operations. Another important FAA effort is the Runway Incursion Mitigation (RIM) Program established by the Office of Airports in 2015 to identify strategies to mitigate areas of airport runways or taxiways that do not meet current FAA airport design standards and have high incursions rates (“RIM locations”). There can be multiple RIM locations at a single airport. FAA considers locations for inclusion in the RIM inventory based on whether the location has a non-standard design and has experienced three or more incursions in a given calendar year, or averaged at least one incursion per year over the course of the RIM program. At RIM locations, FAA provides funding and technical assistance to airports to mitigate the risk of incursions, such as by changing airport design and by improving runway and taxiway signage. For example, the airport may reconfigure a taxiway to intersect a runway at a 90-degree angle (the FAA standard), or install “hold position” signs at intersections between two runways. According to FAA, at the end of fiscal year 2018, FAA had helped airports mitigate 33 RIM locations through the program, leaving 135 locations across 79 airports that still needed to be mitigated. FAA also collaborates with industry stakeholders to identify and address runway and taxiway safety issues. For example, FAA serves as Co-Chair of CAST, which analyzes data across airports to identify root causes of incidents and develop and track mitigations to address those causes. For instance, through CAST, FAA and industry stakeholders developed training for air traffic controllers to mitigate the risk of runway excursions. The training described factors that can contribute to runway excursions such as adverse winds, wet or contaminated runways, or unstable aircraft approaches. In addition, in 2015, FAA convened a forum of aviation stakeholders representing government, industry, and labor called the Runway Safety Call to Action which developed 22 short-, medium-, and long-range mitigations to address the rising number of reported runway incursions. In 2018, the DOT Office of Inspector General reviewed FAA’s progress in implementing these 22 mitigations and made three recommendations to address implementation challenges it identified, including consolidating duplicate mitigations and, as mentioned below, developing a plan to measure their effectiveness. As of August 2019, FAA had not implemented these recommendations. Individual airport operators and airlines have implemented their own efforts to improve runway, taxiway, and ramp safety. For example, officials who manage Daniel K. Inouye International Airport in Honolulu, Hawaii, told us that they changed the location of markings in an airport area known to be confusing to some pilots, which reduced incursions at this location. In addition, officials from Airlines for America and the Regional Airlines Association told us airlines host safety meetings where they leverage their collective data to identify and address industry-wide safety trends. Officials told us that one of the working groups at these airline safety meetings specifically discusses issues and solutions pertaining to the ramp area. FAA, airports, and airlines fund multiple technologies to improve runway and taxiway safety, primarily through increasing air traffic controller, pilot, and vehicle operator awareness of their surroundings. See Table 1 for technologies in place or in development. FAA surveillance technologies are multi-million dollar programs designed to help air traffic controllers identify aircraft and vehicles in the terminal area. For example, at the 35 airports where ASDE-X has been installed since 2011, FAA estimated the total program cost to FAA to be more than $800 million. In-aircraft technologies like those mentioned above help pilots identify their location on runways and taxiways, and could mitigate risks of injuries and damage caused by excursions. FAA has taken steps to improve terminal area safety, but has not assessed the effectiveness of many of its runway and taxiway safety efforts. For example, FAA has not evaluated how its primary efforts such as ASDE-X, ASSC, or the Runway Safety Program contribute to runway and taxiway safety, despite having implemented these efforts years ago. In some instances, FAA has taken steps to evaluate its terminal-area safety efforts. For example, FAA tracks the Runway Incursion Mitigation Program’s outcomes and the number of runway excursions safely stopped by an Engineered Material Arresting System (EMAS). FAA also contracted with a research organization in 2017 to evaluate the effectiveness of Runway Status Lights on the runway incursion rate at 15 airports. Further, the Runway Safety Program manager described other instances in which local airport officials have taken steps to evaluate the effect of mitigations at those airports. For example, one of FAA’s runway safety offices assessed the effect of five informational videos it produced, to highlight issues identified at specific airports, on runway incursions at those locations after the videos were released. However, FAA has not assessed the effectiveness of many of its numerous other runway and taxiway efforts described above and FAA officials told us that FAA does not have a plan to do so. Officials told us that they believe that the assessments described above are sufficient, based on the availability of agency resources. In June 2018, the DOT IG reported a similar finding related to its assessment of FAA’s 2015 Runway Safety Call to Action, described above. The DOT IG reported that FAA had a plan to track the completion of mitigations aimed at improving runway and taxiway safety, but not to link the mitigations to quantifiable goals or metrics that would measure their effectiveness in reducing runway incursions. FAA’s guidance on the Runway Safety Program states that FAA may evaluate the effectiveness of its runway safety programs, and the extent to which they are helping FAA meet its safety goals. In addition, in the 2016 Evaluation Roadmap for a More Effective Government, the American Evaluation Association stated that agencies should consistently use program evaluation and systematic analysis to improve program design, implementation, and effectiveness and to assess what works, what does not work, and why. Evaluating a program’s effectiveness can include methods such as surveying a program’s managers (e.g., regional runway safety program managers), or comparing a program’s performance to an evaluative criterion (e.g., a measure of terminal area safety). Without assessing the effectiveness of its range of efforts, FAA cannot determine the extent to which each of its efforts contribute to its goal of improving runway and taxiway safety, or whether other actions are needed. As discussed previously, FAA has efforts designed to increase runway and taxiway safety that range from periodic stakeholder meetings to multi-million dollar ground surveillance systems. By assessing the effectiveness of its primary efforts, FAA may be better positioned to make decisions about how to target its limited resources within and among these efforts. We also found that FAA may be missing opportunities to improve its terminal-area safety efforts, including improving communication within FAA. Specifically, FAA Regional Runway Safety Program staff told us that they do not receive the results of most runway incursion investigations— information that could aid RSAT discussions about preventing these incidents in the future. Four of FAA’s five Regional Runway Safety Program Managers we interviewed reported that, they did not receive the results of investigations of pilot deviations—which constitute the majority of runway incursions—from the Office of Flight Standards. As part of its investigations of these incursions, Flight Standards identifies possible causes and implements mitigations, such as additional pilot training. However, FAA does not require Flight Standards to automatically provide their investigations of runway and taxiway incidents to the Runway Safety Group, which could enhance runway and taxiway safety. FAA officials said that FAA requires Flight Standards to make its investigations available to Runway Safety Group staff, if requested, but acknowledged that this does not always result in Runway Safety Group staff receiving these investigations in a timely manner. FAA officials said they are in the process of implementing additional processes to improve communication between Flight Standards and the Runway Safety Group, but documentation on these processes FAA provided to us did not address getting investigations to Runway Safety program staff in a timely manner. Without this information, the Regional Runway Safety Program Managers may be unable to provide air traffic managers with relevant information on most incursion investigations as they prepare to host their annual RSAT meetings. The manager of the Runway Safety Group told us that Regional Runway Safety Program Managers may request individual investigations from regional Flight Standards officials, but that it would be time consuming for these regional managers to make such requests for every pilot deviation. One of FAA’s objectives is to improve runway and taxiway safety, and federal internal control standards state that management should internally communicate the information necessary to help meet its objectives. Without timely access to the results of Flight Standards’ incident investigations, Regional Runway Safety Program Managers—and therefore, local air traffic control managers—may not have all of the relevant information they need to develop appropriate runway and taxiway safety mitigation strategies and plans. Selected airport operators we interviewed also reported that they may not have all information they need to develop appropriate terminal area safety mitigation strategies. Specifically, most of those we interviewed reported that air traffic control managers did not provide them with complete and timely information on all runway and taxiway incidents. Six of 10 airport operators we interviewed told us that air traffic control managers did not notify them of all runway and taxiway incidents as they happened. Further, some airport operators told us that they were not aware of all incidents until the annual RSAT meeting. For example, the operator of one airport told us that the air traffic manager notifies the airport of vehicle/pedestrian deviations immediately, but not of operational incidents or pilot deviations. The Manager of the Runway Safety Program also confirmed that communication varies by airport operator and air traffic manager. According to federal internal control standards, management should communicate quality information externally so that external parties can help the entity achieve its objectives and address related risks. Further, according to air traffic control procedures, controllers are required to report as soon as possible to airport managers and others “any information which may have an adverse effect on air safety.” However, this requirement does not specify the types of terminal area safety incidents to which this applies. Also, through a 2018 internal risk management process, FAA identified the need for enhanced communication among airport management, the FAA Air Traffic Organization, and pilots at towered airport facilities, in order to mitigate the safety risks associated with runway incursions. Lacking complete information on runway and taxiway incidents at their airports could hamper airport operators’ ability to develop appropriate safety strategies or make investment decisions related to safety in a timely manner. For example, the operator of one airport told us that not being notified of operational incidents means the airport does not have a complete picture of the safety incidents there, which limits their ability to identify trends or training needs. FAA’s safety oversight approach is designed to use data to identify hazards, manage risks, and mitigate them before an accident occurs. FAA uses data to analyze runway incursions, and recently developed a new metric to track the risk of terminal-area incidents. However, without leveraging data to analyze all terminal-area incidents, FAA may be missing opportunities to better target the agency’s resources, and ultimately to further improve safety. For example, because FAA does not have a process to eliminate all duplicates from its excursion data, it does not have assurance that its excursion data are accurate, and it may be missing opportunities to mitigate the risks excursions pose. Similarly, taking steps to analyze ramp area incidents by identifying such incidents in its new metric would help FAA determine whether it needs to focus more on improving safety in ramp areas. In addition, establishing a plan to evaluate all of its runway and taxiway safety efforts would help FAA direct its resources toward activities and technologies proven to enhance safety and identify ways to strengthen those efforts. Moreover, improving internal communication among FAA offices could make the annual Runway Safety Action Team meetings—a key component of FAA’s terminal area safety efforts—more effective. And last, improving external communication between air traffic managers and airport operators would help airports identify and implement needed mitigations more quickly. We are making the following five recommendations to FAA: 1. The Runway Safety Manager should develop a process to identify and remove duplicate excursion records. (Recommendation 1) 2. The Runway Safety Manager should take steps to analyze data on ramp area incidents in FAA’s new surface safety metric. (Recommendation 2) 3. The Runway Safety Manager should establish a plan to assess the effectiveness of all of FAA’s terminal area-safety efforts, including Airport Surface Detection Equipment, Model X (ASDE-X) and the Runway Safety Program. (Recommendation 3) 4. The Administrator of FAA should require Flight Standards to share the results of its investigations with the Runway Safety Group, in a timely manner. (Recommendation 4) 5. The Administrator of FAA should require air traffic control managers to share information on terminal area incidents, such as operational incidents and pilot deviations, with airport operators, in a timely manner. (Recommendation 5) We provided the Department of Transportation (DOT), the Department of Labor (DOL), the National Aeronautics and Space Administration (NASA), and the National Transportation Safety Board (NTSB), with a draft of this report for review and comment. In its written comments reproduced in appendix I, DOT concurred with our recommendations. DOL, NASA, and NTSB did not provide technical comments. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 11 days from the report date. At that time, we will send copies to the appropriate congressional committees, DOT, DOL, NASA, NTSB, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or krauseh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. In addition to the individual named above, other key contributors to this report were Heather MacLeod (Assistant Director); Sarah Farkas (Analyst-in-Charge); Dave Hooper; Josh Ormond; Madhav Panwar; Steven Rabinowitz; Laurel Voloder; Madeline Welter; and Elizabeth Wood.", "summary": "U.S airspace system is one of the safest in the world, but incidents and near misses at and around U.S. terminal areas still occur. FAA oversees the safety of runways and taxiways and works with industry partners—including airlines, airports, pilots, and others—to improve safety in these areas. Despite FAA's continued efforts, the number of reported terminal area incidents has increased over time. GAO was asked to review various issues related to runway safety and to update its prior work on airport terminal areas. This report examines: (1) the extent to which FAA uses data to analyze terminal area incidents and (2) efforts FAA and others have implemented to improve terminal area safety, and how FAA assesses their effectiveness. GAO analyzed FAA data; interviewed officials from 10 airports selected based on high runway incident rates in the past 3 years, among other factors; and interviewed federal and industry officials. The Federal Aviation Administration (FAA) uses data to analyze some types of incidents in airport “terminal areas”—runways, taxiways, and ramps. For example, FAA uses data to analyze runway “incursions”—the incorrect presence of an aircraft, vehicle, or person on the runway. According to FAA data, the rate of reported runway incursions nearly doubled from fiscal years 2011 through 2018, with most of this increase due to a rise in reports of less severe incursions, or those without immediate safety consequences. However, GAO found that FAA has not identified or removed all duplicates from its data on runway “excursions”—when an aircraft veers off or overruns a runway—which limits FAA's ability to accurately analyze these incidents. Additionally, FAA does not use data to analyze incidents that occur in ramp areas—the parts of terminal areas where aircraft are prepared for departure and arrival—where injuries to workers and damage to aircraft can occur. Without a process to leverage accurate excursion and ramp incident data, FAA may not be able to assess the risk these incidents pose to passengers, airport staff, and others. FAA, airports, and airlines have implemented multiple efforts to improve terminal area safety, but FAA has not assessed the effectiveness of many of its efforts. For example, FAA has funded multiple technologies to improve runway safety, such as Airport Surface Detection Equipment, Model X (ASDE-X)—a ground surveillance system that enables air traffic controllers to track landing and departing aircraft and alerts controllers of potential collisions. However, FAA has not assessed the effectiveness of ASDE-X. Similarly, FAA has not assessed the effectiveness of its Runway Safety Program, whereby FAA staff, along with local airport stakeholders, provide data and support to local air traffic managers to help identify and manage terminal area safety incidents. FAA has taken steps to evaluate some of its terminal-area safety efforts, such as tracking the number of runway excursions safely stopped by a lightweight, crushable concrete designed to stop or greatly slow an aircraft that overruns the runway. However, without assessing how all of FAA's efforts contribute to its goal of improving runway and taxiway safety, FAA cannot determine the extent to which it is targeting its limited resources to the most effective strategies. GAO is making five recommendations including that FAA identify and remove duplicate excursion data, develop processes to analyze ramp area incidents, and establish a plan to assess the effectiveness of its terminal area safety efforts. FAA concurred with the recommendations.", "document_type": "gao"}
{"report": "OCWR allocates functions among its Board of Directors, Executive Director, and General Counsel (see fig. 1). This organizational structure is largely due to statutory requirements in the CAA. As of February 2019, OCWR had 28 full-time equivalent positions, which includes five part-time board members (counted as one full-time equivalent) appointed by congressional leadership. This represents an increase of five full-time equivalents since April 2018. OCWR manages an Administrative Dispute Resolution (ADR) process to resolve alleged violations of workplace rights and protections, such as discrimination. The Reform Act overhauled the ADR process, including removing mandatory counseling and mediation periods and a waiting period prior to filing a claim (see fig. 2). To advance worker protections, the Reform Act mandated that OCWR implement various new requirements. OCWR has implemented three of the four requirements that generally became effective on June 19, 2019 (see table 1). As of October 2019, OCWR had completed three requirements. Managing changes to the ADR process. OCWR officials stated that because they had initiated a multi-year process to revise procedural rules in 2016, they were more familiar with the steps and timeline needed to implement this requirement in 2019. Appointing a confidential advisor. Similarly, an OCWR official stated that the confidential advisor role was similar to OCWR’s counselor role prior to the Reform Act, which made implementing this requirement more manageable. Creating a secure electronic system to file claims. The online system, SOCRATES, was operational starting June 26, 2019, 7 days after the requirement’s effective date of June 19, 2019. Between June 19 and June 26, 2019, OCWR implemented a fillable PDF form so that claims could be submitted electronically (email or fax). OCWR officials reported that no claims were filed during the 7-day delay, and therefore, they believe that the delay did not negatively affect employees’ ability to file claims. According to OCWR, testing the system the week prior to June 19, 2019, revealed numerous problems with SOCRATES. For example, if a user did not submit his or her claim within a certain amount of time, the system refreshed the page without saving the user’s data, forcing the user to restart the claim. Also, during a June 17, 2019, meeting between OCWR and congressional staff, OCWR received requests to further revise forms associated with SOCRATES. OCWR was unable to implement these changes before the June 19, 2019, deadline. As a result, OCWR delayed the launch of SOCRATES until June 26, 2019, to allow time to resolve these issues and fully test the system. However, OCWR did not communicate the decision to delay the full launch of SOCRATES to congressional stakeholders who had expected that the system would be delivered on time. As of October 2019, OCWR had not completed one requirement that was due by June 19, 2019. Establishing a program to permanently retain records. The Reform Act required OCWR to establish and maintain a permanent records retention program, which includes records of preliminary reviews, mediations, hearings, and other proceedings. Since November 2017, OCWR has operated under an interim records retention policy that requires it to permanently keep all records. According to OCWR, it is not destroying or deleting any records. OCWR’s interim permanent records retention policy states that OCWR will establish standards and procedures for records integrity, privacy, and confidentiality. However, as of October 2019, about 4 months after this requirement became effective, OCWR had not developed these standards or established other policies or procedures for maintaining a permanent records retention program other than the interim policy. According to OCWR, it scanned paper records to create electronic files and hired a separate contractor in September 2019 to help further develop its records retention program. As of October 2019, OCWR was implementing the other three requirements which have varying deadlines, time frames, and effective dates extending beyond June 19, 2019 (see table 2). Tracking and reporting data and assessments. The Reform Act created new reporting requirements for OCWR. For example, it required OCWR to issue annual, itemized reports on awards and settlements. The Reform Act also required OCWR to issue a one- time report on awards and settlements previously paid, which OCWR published on January 20, 2019. OCWR plans to issue the report on 2019 awards and settlements by January 31, 2020, and subsequent reports annually. The Reform Act also required OCWR to use SOCRATES data to assess the effectiveness of ADR procedures in resolving claims in a timely manner and to publish these assessments in semi-annual reports to Congress. OCWR plans to issue the first semi-annual report by January 31, 2020. Conducting a workplace climate survey. The Reform Act required OCWR to conduct a secure survey of legislative branch offices covered by the act by December 20, 2019 (within one year of enactment), and every 2 years thereafter. The survey would assess the workplace environment, including attitudes toward sexual harassment. As of October 2019, OCWR officials reported that they were waiting for additional input from congressional staff before proceeding with certain aspects of the survey. According to OCWR officials, OCWR’s House and Senate oversight committees had different views of what the survey should include. Therefore, OCWR plans to conduct separate surveys for House offices, Senate offices, and other legislative branch offices. According to OCWR officials, they may be able to launch the House survey by the December 20, 2019, deadline, with the other surveys following. However, the timeline for conducting these surveys is uncertain until OCWR can confirm the surveys’ content with congressional staff and conduct various tests, such as separately pilot testing each of the three surveys. Additional work remaining includes: reviewing changes to the survey questions, obtaining input from legislative branch stakeholders, conducting internal testing of the survey, pilot testing the survey with legislative offices, and finalizing the survey and communications to survey recipients. Educating and assisting legislative branch agencies. OCWR updated various education and training materials, such as: creating a new workplace rights brochure; redesigning a poster notifying employees of rights, protections, and procedures under the CAA; and establishing audio and visual teleconferencing access for out-of- area employees (i.e., legislative branch employees in elected officials’ district and state offices). An OCWR official reported that, in October 2019, OCWR developed a training video on new procedures under the Reform Act. A link to the training video was included in the September 2019 quarterly e- newsletter sent to all legislative branch employees covered under the CAA. According to the official, OCWR also plans to launch another training video in November 2019 and develop three new training classes. We found that OCWR incorporated some key management practices when implementing Reform Act requirements (see appendix II for a list of management practices we used to assess OCWR). However, we also found that OCWR did not consistently incorporate key management practices for some requirements and that opportunities exist to improve the remaining implementation and administration of Reform Act requirements. We found that OCWR incorporated some key change management or project management practices applicable to implementing Reform Act requirements. For example: OCWR defined the Reform Act requirements and created 21 task teams for implementing them. OCWR dedicated an implementation team to manage the transformation process. OCWR designated a manager to track the implementation status for all task teams. The task team leaders also met weekly. OCWR established an overall project schedule with interim milestones and time frames for revising procedural rules, part of the requirement to manage changes to the ADR process. OCWR also established an overall project schedule for conducting the workplace climate survey. OCWR officials reported that having this schedule has enabled them to track progress, determine that the survey is behind schedule (as of October 2019), and communicate revised expectations to stakeholders. In addition, OCWR officials stated they identified and addressed risks associated with the requirement to appoint a confidential advisor. These risks included the perception of a potential conflict of interest that could arise if an attorney contracted from a private law firm served as the confidential advisor. To mitigate this perception, OCWR hired the confidential advisor as an employee to ensure that the confidential advisor cannot refer claimants to his or her own law firm for legal representation. Project schedules provide a detailed plan for delivering products, services, and results in a timely manner, as well as serve as a communication tool for managing stakeholder expectations. OCWR used project schedules to revise the procedural rules and develop the workplace climate survey but did not use schedules to manage the implementation of other requirements. In particular, for SOCRATES, OCWR officials reported that they proposed a draft schedule but did not finalize this draft or otherwise document changes to the schedule. According to these officials, they did not update the schedule because their implementation plans had changed significantly, and the compressed timeframe resulted in making changes “on the fly.” For example, they revised the system architecture as late as 3 weeks before the mandated deadline, which required signing an interagency agreement for hosting the system with the Library of Congress the day before the mandated deadline. In addition, OCWR encountered last-minute issues when testing the system, as we previously discussed. As a result, OCWR delayed the full launch of SOCRATES but did not communicate this decision to congressional stakeholders who had expected that the system would be delivered on time. Although not a long delay, a project schedule could have helped manage stakeholder expectations. Without a schedule for SOCRATES, OCWR missed opportunities to take corrective actions earlier, communicate with congressional stakeholders, and better manage expectations. OCWR has ongoing cybersecurity activities and planned upgrades to other information technology (IT) systems, but has not yet established project schedules for them. Moving forward, it will be important for OCWR to establish project schedules to manage IT projects and allow key stakeholders to monitor OCWR’s progress. OCWR has taken interim steps to establish a permanent records retention program. These steps include changing its records retention policy in November 2017 to make all records permanent, hiring a contractor in May 2019 to scan paper records and store them electronically, and hiring another contractor in September 2019 to help develop its records retention program. Key management practices call for organizations to identify and assess risks that could affect their ability to achieve their goals and objectives and to monitor and manage these risks as the projects progress. OCWR identified the largest potential risk to establishing and maintaining a permanent records retention program as the loss of control over confidential files. For example, an OCWR official confirmed that OCWR maintains a physical file for every electronic file, which results in multiple storage locations and duplicate records. Although this ensures the availability of records, multiple storage locations can make ensuring the confidentiality and security of these records more difficult. However, as of October 2019, OCWR has not yet fully addressed this risk by developing a policy to manage it or identified other risks. OCWR officials stated that the contractor will help with these tasks. They also reported that they plan to develop policies for the records retention program, particularly for maintaining the privacy and security of records, based on records management requirements for executive branch agencies. According to OCWR officials, addressing risks for its records retention program is not a high priority, and demand for records is low. Nevertheless, if OCWR does not address the potential risks, and any emerging risks, associated with permanently retaining sensitive records, OCWR may be less able to manage its records and ensure their confidentiality, integrity, and availability. We have previously reported that a critical element in an organization’s efforts to manage for results is its ability to set meaningful goals for performance and to measure progress toward these goals. Strategic goals are intended to be the starting point for an organization’s performance measurement efforts. To provide a connection between the long-term strategic goals and the day-to-day activities, organizations should also establish near-term performance goals and measures. Finally, an organization needs to report on its performance to provide information to its stakeholders on the extent to which it has met its performance goals and what it accomplished with its resources. Leading organizations then apply this performance information to continuously improve organizational processes, identify performance gaps, and set improvement goals. OCWR’s 2019-2023 strategic plan includes several broad, long-term, outcome-related goals that address Reform Act requirements. These goals are supported by objectives, called initiatives. For example, OCWR has a long-term strategic goal to “provide an efficient and effective ADR program.” A supporting initiative is to “ensure that ADR processes meet statutory and regulatory mandates, including mandates for maintaining confidentiality.” However, this initiative does not state what is to be achieved and by when, and none of the supporting initiatives reflect near- term performance goals that allow for an objective assessment of progress. Performance goals, which are used to assess progress toward long-term goals, should be stated in objective, measureable, and quantifiable terms. OCWR identifies performance measures in its strategic plan, but the measures lack target levels of performance which would help assess progress toward goals. For example, one of OCWR’s initiatives is to “empower stakeholders to effectively resolve their workplace disputes without having to engage in protracted dispute resolution proceedings.” A supporting performance measure is to “track the rate of cases resolved by negotiated settlements.” This measure provides a starting point for collecting performance information but does not specify how it can be used to assess progress on the initiative. We have previously reported that successful performance measures commonly demonstrate results, are limited to the vital few, respond to multiple priorities, and link to responsible programs. OCWR does not report on progress toward goals in its annual report, partly because of the lack of performance goals and measures assessing progress. OCWR’s annual reports summarize statistical data about the number of employees using OCWR’s services and reasons for ADR claims from each fiscal year, which is information required to be published under the CAA. However, these statistics do not compare actual performance against measurable performance goals. Further, OCWR does not report how it used the data to improve its programs. For example, in its fiscal year 2018 annual report, OCWR reported the number and types of workplace issues that employees inquired about in general information requests and raised during formal counseling requests. However, OCWR did not report how it used this information to identify trends and develop training programs to target the indicated issues. According to OCWR officials, OCWR does not set more specific or measurable goals and measures beyond what is included in its strategic plan. In addition, they stated that OCWR’s current performance goals and measures reflected their concern that changes from the Reform Act could affect their workload, such as the number of cases filed and how they would be settled. They plan to reassess their performance starting in June 2020, about 1 year after many Reform Act requirements became effective, and establish new performance measures and targets based on updated baseline performance levels. Clearly defining performance goals, such as establishing measureable performance targets and milestones, and related performance measures would provide OCWR information to determine if it is making progress toward its long-term goals and better communicate with congressional and other key stakeholders about its progress. Moreover, such performance data would allow OCWR to make more informed decisions to improve performance, such as determining what activities are working as intended and achieving results, and which are not and could be improved. Finally, sharing this information in publically available annual reports could make it more useful and transparent for stakeholders, as well as strengthen OCWR’s accountability for making progress toward its goals. OCWR has a broad mandate to provide education and information to Members of Congress and covered legislative branch offices and employees about their rights, protections, and responsibilities under the CAA. OCWR routinely conducts educational activities, such as holding brown bag events, creating online training, and posting resources on its website and social media channels. OCWR also performs outreach by meeting with congressional committees regularly, communicating with stakeholders (e.g., House and Senate Employment Counsel), meeting with heads of legislative branch employing offices at least sending an annual notice of rights to all legislative employees. However, we found that OCWR’s assessment of these activities is limited, such as collecting feedback forms from attendees of its brown bag events. While important, these efforts do not enable OCWR to assess the effectiveness of its education and outreach activities and the extent to which they are reaching all covered legislative branch populations. Key management practices call for continuous monitoring to identify areas that require additional attention. In 2004, we recommended that OCWR use various approaches, such as feedback surveys, to increase its understanding of the actual level of awareness of its activities among target populations. In response to the recommendation, from 2008 to 2009, OCWR surveyed legislative branch employees but had a low response rate, which rendered the survey data of limited value. OCWR officials attributed the low response rate to not having all respondent email addresses, as well as the lack of statutory authority to conduct surveys. Through the Reform Act’s requirement to conduct a workplace climate survey every 2 years, OCWR has new opportunities to collect data on the extent to which legislative branch employees are aware of OCWR’s services and their rights under the CAA. Because the Reform Act states that OCWR must consult with congressional committees on how to carry out the survey, OCWR has sought guidance from its oversight committees on what information to collect for the survey and the use of the results. In addition to developing the climate survey, an OCWR official stated that, in March 2019, OCWR also hired a training and education project manager who is responsible for developing an education and outreach strategy. This effort is intended to include assessing awareness levels of OCWR’s activities among legislative branch populations and improving awareness where needed. However, as of October 2019, OCWR did not provide documentation of the strategy and a timeline for its completion. A mechanism for collecting feedback more widely from all covered legislative branch employees could improve OCWR’s understanding of the reach and effectiveness of its education and outreach efforts. For example, it could help OCWR determine if it may be missing certain subsets of legislative branch populations, such as maintenance workers who may have limited computer access. Further, such information could help inform management and resource allocation decisions, such as where to focus education and outreach efforts and how to increase their effectiveness. In 2004, we reviewed OCWR’s management practices and made 20 recommendations to help OCWR: strengthen strategic planning and develop results-oriented performance measures; ensure an effective, results-oriented program structure; build effective communication emphasizing outreach and coordination with congressional and legislative branch stakeholders; and create and sustain an enhanced management control environment, particularly for managing human capital and performance. Between 2004 and 2013, we determined that OCWR had implemented 18 of the 20 recommendations. In this review, we found that, of these 18 recommendations, OCWR subsequently stopped implementing an information technology (IT) planning recommendation that could have strengthened its ability to better manage and implement the requirements in the Reform Act. We had recommended that OCWR ensure that IT planning and implementation be an integral component of the strategic planning process. This recommendation focused on positioning OCWR to effectively leverage technology in achieving strategic mission goals and outcomes. To do this, OCWR needed to establish certain basic IT management capabilities, such as: developing a picture or description, based on OCWR’s strategic plan, of what it wanted its future IT environment to look like; establishing and following a defined and disciplined process for allocating limited resources across competing IT needs; using a rigorous IT system acquisition management process; and ensuring that needed IT skills have been identified and obtained. OCWR took steps in 2003 and 2005 to create an IT task force and issue a multi-year IT plan, respectively. However, these efforts were not sustained. An OCWR official explained that OCWR had not conducted IT planning, including developing an IT strategic plan, in recent years because of limited resources and difficulties attracting a candidate for the IT manager position. These challenges resulted in the position remaining vacant for approximately 18 months from 2016 to 2018. In past work, we have reported that IT strategic planning can help an organization align its IT goals and investments with its strategic goals. A key element of IT strategic planning is developing an IT strategic plan that can serve as an organization’s vision or road map to guide its efforts and investments. OCWR officials reported that they will be developing an IT strategic plan. However, as of October 2019, they were unable to provide additional documentation or a timeline for completion. Without IT strategic planning, OCWR may be less able to set forth a long- term vision of OCWR’s IT environment and measure progress in carrying out its strategic initiatives. For example, OCWR envisioned developing an electronic claims filing system as one of its strategic initiatives as early as fiscal years 2013 to 2015. However, that system was not implemented until 2019, in part because OCWR did not have an IT strategic plan and related IT expertise to support the initiative. With increased funding for implementing Reform Act requirements, OCWR has recently re-focused on human capital management. In September 2018, it hired an IT manager whose responsibilities include IT planning and cybersecurity. In September 2019, OCWR hired a contractor to help update its human capital plan, which had not been updated since 2009. We have previously reported that effective human capital management is critical to sustaining an IT workforce with the necessary skills to execute a range of management functions that support the agency’s mission and goals. Given OCWR’s strategic and ongoing IT initiatives, it will be important for OCWR to consider leading practices in human capital management to ensure that it has the appropriate skills and capacity to meet its current and future responsibilities. These leading practices include consulting with key stakeholders when developing human capital strategies, having a system in place to continually assess and improve human capital planning and investment, determining critical skills and competencies its workforce needs to achieve current and future agency goals, linking the strategic workforce plan with the organization’s strategic plan, developing customized strategies to recruit highly specialized and having an ongoing succession planning process for identifying and developing a diverse talent pool. If OCWR does not continue to strategically assess and manage its human capital needs, it could again find itself with IT or other skills gaps that could negatively affect its ability to meet its mission. Incorporating these leading practices in human capital management could help OCWR develop a workforce plan that better aligns with its mission and goals, as well as develop long-term strategies for recruiting and retaining staff to achieve those goals. Although small in size, OCWR has important responsibilities—to advance the safety, health, and workplace rights of employees and employers in the legislative branch. The Reform Act updated how OCWR carries out these responsibilities, such as requiring OCWR to offer an electronic option for filing Administrative Dispute Resolution (ADR) claims and to conduct a workplace climate survey of legislative branch employees. To meet these new requirements, OCWR developed new procedures, trained and hired staff, and balanced new responsibilities with existing ones. As a result, OCWR completed three requirements—managing changes to the ADR process, appointing a confidential advisor, and creating a secure electronic claims reporting system. However, as of October 2019, OCWR had not fully completed the requirement to establish and maintain a program for permanent records retention. To meet this requirement, OCWR needs to develop and implement policies and procedures to administer and manage the program, as well as identify and address potential risks related to the privacy and security of records, among other actions. To help OCWR meet requirements with ongoing work and build upon completed work, it will be important for OCWR to incorporate key practices for project management, such as consistently developing and using project schedules and assessing risk. These practices could help OCWR better manage expectations and prioritize high-risk work. Further, establishing results-oriented performance goals and measures and collecting and using performance information could help OCWR evaluate and better focus its education and outreach efforts, as well as assess progress toward its strategic goals. Finally, OCWR should use its strategic planning process to clearly articulate how its IT initiatives support its organizational goals, such as how the electronic claims reporting system supports a more efficient and effective ADR program. Establishing performance measures and monitoring actual-versus-expected performance of those measures can help determine whether IT is making a difference in improving performance, and in turn whether OCWR is better serving the legislative community. Additionally, OCWR needs to evaluate its human capital management strategies to ensure that it can recruit and retain staff with the appropriate skills to plan and manage IT projects, as part of a larger effort to conduct IT planning. We are making the following six recommendations to OCWR: The Executive Director of OCWR, in collaboration with relevant managers, should establish a policy that requires a schedule of tasks to be developed, documented, and updated throughout the lifetime of IT system projects. (Recommendation 1) The Executive Director of OCWR should identify and assess risks in establishing and maintaining a permanent records retention program, and develop policies and procedures to ensure that risks are properly addressed. (Recommendation 2) The Executive Director of OCWR should identify desired performance results, develop performance measures that demonstrate the degree to which the desired results were achieved, and report progress toward those results in OCWR’s annual reports. (Recommendation 3) The Executive Director of OCWR should collect relevant data through a survey or other mechanisms, and use the information to evaluate the effectiveness of education and outreach efforts and the extent to which they are reaching all covered legislative branch populations. (Recommendation 4) The Executive Director of OCWR should integrate IT planning and implementation into the agency’s strategic planning process. (Recommendation 5) The Executive Director of OCWR should incorporate key strategic human capital management practices, such as developing strategies to recruit and retain staff with mission-critical skills, into the strategic planning process. (Recommendation 6) We provided a draft of the report to OCWR for review and comment. In its comments—reproduced in appendix III—OCWR agreed with our findings and concurred with our six recommendations. To address the recommendations, OCWR noted that it has already taken some actions, such as hiring a contractor to assess risks related to permanent records retention. It intends to implement additional changes, such as developing a policy for IT project planning. Going forward, OCWR agreed to update us on its progress implementing the recommendations. We are sending copies of this report to the appropriate congressional committees, the Executive Director of OCWR, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or jonesy@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our first objective was to determine the status of Office of Congressional Workplace Rights’ (OCWR) efforts to address new requirements from the Congressional Accountability Act of 1995 Reform Act (Reform Act). To meet this objective, we reviewed applicable laws and identified the new requirements. We reviewed the Reform Act and grouped Reform Act requirements into seven categories of activities based on similar characteristics, such as requirements related to amending the claims process, and how these requirements aligned with OCWR’s task teams working on these requirements. We also collected and reviewed documentation on OCWR’s implementation process and management practices, such as OCWR’s list of tasks and task teams, task team meeting notes, progress reports, agreements with outside vendors, and email communications. Our second objective was to assess how OCWR is incorporating key management practices to implement the Reform Act’s new requirements. To meet this objective, we analyzed OCWR’s implementation of new requirements against key practices for organizational change management we identified in our 2003 report, Results-Oriented Cultures: Implementation Steps to Assist Mergers and Organizational Transformations (GAO-03-669) and key practices for project management from the Project Management Institute Inc.’s A Guide to the Project Management Body of Knowledge, PMBOK Guide®. We determined which key practices and related implementation steps were applicable to OCWR based on the following factors: (1) if the practices aligned with the scope and nature of OCWR’s work, and (2) if the practices applied to OCWR’s implementation timeline given Reform Act deadlines. We shared these key management practices with OCWR. Our third objective was to determine the extent to which OCWR implemented recommendations from our 2004 report, Office of Compliance: Status of Management Control Efforts to Improve Effectiveness (GAO-04-400). To meet this objective, we reviewed OCWR’s plans and documentation of its activities, such as strategic plans and annual reports, to address the recommendations. We then assessed OCWR’s plans and activities against our original recommendations and the recommendations’ implementation status to determine the extent to which OCWR implemented the recommendations in the past and has continued to take similar actions. For all three objectives, we interviewed OCWR officials and conducted semi-structured interviews with a nonprobability sample of key stakeholders and officials from offices covered by the Reform Act. Although results from these interviews are not generalizable to all stakeholders or offices covered by the act, they provided views and illustrative examples about OCWR’s efforts to address new Reform Act requirements, OCWR’s efforts to incorporate key management practices to implement those new requirements, and the extent to which OCWR implemented some of our previous recommendations. These stakeholders and offices include the Architect of the Capitol, Senate Chief Counsel for Employment, and Office of House Employment Counsel. To obtain perspectives from a range of stakeholders and offices, we considered the following factors in our selection: size of the office or agency by number of employees; extent to which offices/agencies are involved in outreach by number of Administrative Dispute Resolution cases and Occupational Health and Safety Inspections/Americans with Disabilities Act inspections; past interviews with offices/agencies discussing OCWR outreach for balance of perspective (e.g., both chambers of Congress). We also interviewed the House Office of Employee Advocacy and the House Office of the Chief Administrative Officer safety team to obtain additional views on their interactions with OCWR. We conducted this performance audit from January 2019 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We determined that the following key organizational change management practices and key project management practices, as well as related implementation steps, were relevant for assessing the Office of Congressional Workplace Rights’ (OCWR) management practices. In addition to the contact name above, Steven Lozano (Assistant Director), Elizabeth Fan (Analyst in Charge), David Blanding, Giny Cheong, Patrick Dibattista, Karin Fangman, Ben Licht, Patricia Powell, and Edith Yuh made key contributions to this report. Karen Brindle, Hannah Brookhart, Kisa Bushyeager, Terrell Dorn, Robert Gebhart, Lisa Hardman, Ted Hu, Susan Irving, Sonya Johnson, Amalia Konstas, Kaelin Kuhn, Patricia McClure, Zina Merritt, Edda Emmanuelli Perez, Robert Robinson, Sukhjoot Singh, Jon Ticehurst, Alicia White, and Rebecca Woiwode also provided valuable assistance.", "summary": "OCWR is an independent, non-partisan office that administers and enforces various provisions related to fair employment and occupational safety and health within the legislative branch. Responding to concerns about sexual harassment in the workplace, Congress passed the Reform Act in 2018, which expanded worker protections and overhauled the process for resolving workplace claims, including claims relating to discrimination and harassment. The act also required OCWR to create a secure, electronic claims system and appoint a confidential advisor to assist claimants, among other requirements. The Reform Act includes a provision for GAO to review OCWR's management practices. This report examines (1) the status of OCWR's efforts to address new requirements in the Reform Act; (2) how OCWR is incorporating key management practices to implement the new requirements; and (3) the extent to which OCWR implemented recommendations from a related 2004 GAO report. GAO reviewed documentation on OCWR's processes, interviewed officials from OCWR and selected legislative branch offices, and assessed how OCWR's actions aligned with key organizational change management practices that GAO identified and key project management practices from the Project Management Institute. The Office of Congressional Workplace Rights' (OCWR) mission is to effectively implement and enforce the Congressional Accountability Act of 1995 (CAA), as amended in 2018 by the Congressional Accountability Act of 1995 Reform Act (Reform Act). OCWR has implemented three of the four Reform Act requirements that generally became effective June 19, 2019, as shown below. Three other Reform Act requirements—track and report data and assessments, conduct a workplace climate survey, and educate and assist legislative branch offices—are in progress. OCWR has incorporated some key management practices when implementing requirements, such as managing risks associated with appointing a confidential advisor. However, opportunities exist to further incorporate key management practices in OCWR's work. For example: Addressing risks . OCWR has not yet developed policies and procedures to address the risks associated with permanently retaining sensitive records, such as ensuring they remain confidential when stored in multiple locations. Measuring performance . OCWR has not established measurable performance targets and milestones or related performance measures. Doing so would allow OCWR to determine if it is making progress toward its long-term goals and better communicate with congressional and other stakeholders about its progress. Monitoring effectiveness . OCWR routinely conducts educational activities, such as holding brown bag events and online training, and performs a variety of outreach activities. OCWR has new opportunities every 2 years to collect data through the workplace climate survey on the extent to which legislative branch employees are aware of OCWR's services and their rights under the CAA. GAO found that OCWR implemented most recommendations from a 2004 GAO report examining OCWR's management controls. GAO also found that OCWR later stopped implementing a recommendation related to information technology (IT) planning, including ensuring that it obtained necessary IT skills. Without IT strategic planning, including recruiting and retaining staff with mission-critical IT skills, OCWR may be less able to carry out its strategic initiatives. GAO is making six recommendations to OCWR to better incorporate key management practices as it implements requirements, and to improve its strategic planning. OCWR agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Several bureaus within Interior are responsible for the leasing, permitting, and inspecting of mineral extraction activities on federal lands and waters. Interior’s Bureau of Land Management (BLM) is responsible for onshore activities and manages approximately 700 million acres of subsurface mineral rights throughout the country, including the acreage it leases to companies for oil and gas development. At the end of fiscal year 2016, about 41,000 oil and gas leases accounted for approximately 28.2 million acres in 32 states, according to BLM data. For offshore oil and gas activities, the Bureau of Ocean Energy Management is generally responsible for leasing and resource planning and evaluation, among other functions, and the Bureau of Safety and Environmental Enforcement is generally responsible for permitting and inspecting as well as verifying production volumes on offshore leases, among other functions. Under the Outer Continental Shelf Lands Act, as amended, Interior is responsible for leasing and managing approximately 1.71 billion offshore acres. To begin the leasing process, Interior holds auctions through which companies may secure the rights to federal leases that allow them to drill for oil and gas upon meeting certain conditions. Once a company obtains a lease, it may conduct further exploration and subsequently determine whether it would like to drill a well. If a company plans to drill, it must first secure a permit from Interior. To secure a permit to drill under an onshore lease, a company must submit an application for a drilling permit to the appropriate BLM field office. BLM officials then evaluate the company’s proposal to ensure that it conforms to the relevant BLM land use plan for the area as well as applicable laws and regulations, including those focused on protecting the environment. To secure a permit to drill on offshore leases, a company must submit an application for a drilling permit to the Bureau of Safety and Environmental Enforcement, where it is reviewed for completeness and whether all technical elements conform to applicable regulations. Once a company secures a permit and begins producing, oil and gas is transported to market and sold. As part of this process, companies may elect to process the natural gas into various products before its sale. Under ONRR regulations, companies may deduct certain costs associated with transportation and natural gas processing from the royalties due. Companies can continue to produce oil and gas until the lease is no longer capable of producing in paying quantities, regardless of the length of the lease. To ensure compliance with applicable laws, regulations, and other requirements, both BLM and the Bureau of Safety and Environmental Enforcement have inspection and enforcement programs that are designed to verify that companies comply with all requirements at the lease site, including those related to measuring oil and gas volumes. The authority for inspecting wells and leases for this purpose is derived from FOGRMA. The act requires the Secretary of the Interior to develop guidelines that specify the coverage and frequency of inspections. Interior has delegated responsibilities for implementing the act to BLM for onshore leases and to the Bureau of Safety and Environmental Enforcement for offshore leases. ONRR’s oversight of federal royalties includes collecting company-paid royalties, disbursing these royalties to appropriate accounts, and verifying the company-paid royalties through its compliance activities. Collecting: Companies that obtain federal onshore or offshore oil and gas leases are typically obligated to pay royalties on any oil or gas they produce from the leases and then sell. As a condition of producing oil and gas under federal and Indian leases, companies are required to submit two key monthly reports to ONRR—one specifying the total production and disposition of oil and gas and the other stating the royalties due based on production. However, because of various leasing and development arrangements made by companies, these two reports are often submitted by different companies. The companies physically developing the lease, referred to as the operators, are responsible for reporting the production volumes to ONRR in monthly production reports. The companies with a financial interest in the lease, referred to as the payors, are responsible for reporting the cash royalty owed on the federal and Indian oil and gas production in their monthly royalty reports. Each month, payors are to calculate the royalty payment owed to the federal government using the four key variables illustrated in the following equation: Royalty payment = ((volume sold x sales price) less deductions) x royalty rate Companies are to submit monthly production and royalty reports via a web-based portal to ONRR’s royalty information technology (IT) system. In addition to filing the royalty report with ONRR, companies typically make the actual cash royalty payment via an electronic fund transfer to an account at the Department of the Treasury (Treasury). Disbursing: Once ONRR reconciles the self-reported royalty payment data from the monthly royalty reports with the payments to Treasury, ONRR is to disburse the royalties from the Treasury account to the appropriate federal, state, tribal, or Individual Indian Money (IIM) accounts. All these transactions are to be recorded and stored in ONRR’s IT system. Verifying: ONRR is responsible for verifying royalties through its compliance program, which includes ensuring that the royalty revenues generated from the sale of oil and gas extracted from leased federal lands are accurately reported and paid. In conducting its compliance activities, ONRR is to assess the elements of the royalty equation: commodity price, volume of oil and gas, transportation and processing allowances, and royalty rate. ONRR also is to ensure that all relevant laws, regulations, and lease terms have been followed. ONRR has two key statutory requirements for its compliance program: FOGRMA and the Federal Oil and Gas Royalty Simplification and Fairness Act of 1996 (RSFA). FOGRMA requires that ONRR establish a comprehensive auditing system to provide the capability to accurately determine oil and gas royalties. RSFA directs ONRR not to conduct audit activities if it and the relevant state determine that the cost of conducting or requiring the audit exceeds the expected amount to be collected by the activity, based on the most current 12 months of activity. ONRR’s Work Planning Group identifies which companies or leases will be subject to compliance activities. The three primary levels of compliance activities ONRR conducts are audits, compliance reviews, and data mining—each of which provides a varying degree of assurance that royalties are accurately paid. Audits: According to ONRR documents, an audit involves detailed examinations of companies’ royalty payments and corresponding reporting to ONRR. As part of an audit, ONRR staff are to assess the accuracy and completeness of the companies’ self-reported production and royalty data compared to third-party documents, such as sales contracts and oil and gas sales receipts from pipeline companies. According to ONRR documents, it is to design its audits to ensure that royalty payments and other obligations to ONRR are in substantial compliance with applicable lease terms, federal laws and regulations, and other policies. Compliance reviews: ONRR describes compliance reviews as an analysis designed to determine the reasonableness of company- reported production and royalty data. In contrast to audits, compliance reviews are quicker, more limited checks on the accuracy and completeness of companies’ self-reported data and do not include systematically examining the underlying source documentation used to generate the self-reported data. Data mining: ONRR began its data mining program in 2011 and officially organized it within the compliance program beginning in fiscal year 2018. Data mining is a partially automated activity to identify and resolve data errors prior to audits and compliance reviews. According to ONRR officials, data mining examines large sets of company- reported data for certain common errors, such as irregularities in the volume of oil or gas extracted. Officials stated that data mining generally identifies obvious data errors that ONRR staff work with companies to correct. The process companies are to follow to produce oil and gas from federal leases, bring it to market, transmit required data to Interior, and pay royalties is outlined in figure 1. FOGRMA authorizes the Secretary of the Interior to enter into cooperative agreements with states to share oil and gas royalty management information and carry out inspection, audit, investigation, and enforcement activities on federal and Indian lands. Currently, the nine states that are members of STRAC have delegated authority to conduct compliance activities for federal lands in their respective state. These agreements form the framework of ONRR’s relationship with states for mineral revenue compliance activities. A governor or other appropriate official with delegation authority may request that Interior enter into a cooperative agreement with a state by sending a letter to the Director of ONRR. States may also elect to end these agreements at their discretion with a 120-day notice. States have a vested interest in ensuring that all royalties are paid accurately because states receive a portion of the royalties that the federal government collects, including additional collections resulting from compliance activities identifying underpayment. ONRR also reimburses states for the costs of performing approved and eligible compliance activities, including compliance activities under the cooperative agreement. State audit offices that have entered into agreements with ONRR are to submit yearly work plans identifying the compliance activities they propose to conduct in the next fiscal year, which ONRR is to review and approve. Member states can conduct both audits and compliance reviews, and ONRR requires that the state auditors follow the procedures established in generally accepted government auditing standards and ONRR’s audit and compliance review manuals. To ensure that compliance activities are conducted in accordance with generally accepted government auditing standards and relevant ONRR manuals, states are to undergo an external peer review every 3 years, during which they are assessed on their adherence to the standards and manuals and whether they provided corrective actions to any identified problems. A key practice in results-oriented management for federal agencies is establishing agency-wide, long-term strategic goals. The Government Performance and Results Act of 1993 (GPRA), which was significantly enhanced by the GPRA Modernization Act of 2010 (GPRAMA), requires federal agencies, among other things, to develop strategic plans with long-term, outcome-oriented goals; annual goals linked to achieving the long-term goals; and annual reports on the results achieved, as assessed through the use of performance measures and targets. Federal departments and agencies must comply with these requirements and are to follow associated Office of Management and Budget guidance when developing their agency-wide strategic plans. We have reported that these requirements also can serve as leading practices for strategic planning at lower levels within federal agencies, such as planning for individual divisions, programs, or initiatives. These leading practices include defining the mission and goals of an agency or a specific program and developing and using performance measures that allow an agency to track its progress toward its mission and goals. ONRR issued a fiscal year 2017 strategic priorities document that contains the agency’s mission statement: “to collect, account for, and verify natural resource and energy revenues due to states, American Indians, and the U.S. Treasury.” ONRR stated in the document that it planned to achieve Interior’s strategic goals to (1) timely disburse 98 percent of federal and Indian revenues, (2) close 85 percent of Interior’s OIG and GAO recommendations targeted for implementation in fiscal year 2017, and (3) report results of ONRR’s supporting performance measures for Interior’s strategic goals on total ONRR compliance collections and a 3-year average compliance return on investment. ONRR also stated that it planned to create an ONRR strategic plan. In the 1970s and early 1980s, we and Interior’s OIG reported on Interior’s management of the oil and gas revenue collection system. Interior’s OIG issued five reports critical of the program from 1969 through 1977 that raised concerns about royalty collections. In 1981, we reported that Interior was not collecting potentially hundreds of millions in royalties due from federal oil and gas leases. In response, in 1981, the Secretary of the Interior established the Commission on Fiscal Accountability of the Nation’s Energy Resources, also known as the Linowes Commission, to investigate allegations of irregularities in royalty payments, among other issues. The Linowes Commission raised a number of concerns, and its 1982 report stated that management of royalties for the nation’s energy resources had been a failure for more than 20 years. The report found that because the federal government had not adequately managed this multibillion-dollar enterprise, the oil and gas industry was not paying all the royalties it rightly owed. The report cited a range of problems, including the failure to verify data that companies reported as well as late payments and underpayments. Following this report, Interior and Congress took actions aimed at improving revenue collection, including reorganizing oil and gas revenue collections under a new bureau within Interior, passing FOGRMA in 1982, and passing RSFA in 1996. In December 2006, Interior’s OIG analyzed ONRR’s compliance processes and issued a report that made several recommendations to improve these processes and the agency’s systems for tracking them. The report identified deficiencies with how ONRR maintained compliance-related information and recommended changes for how ONRR measures its compliance activities’ performance. In 2007, Interior’s Subcommittee on Royalty Management—a subcommittee of the Royalty Policy Committee—issued a report that reiterated several of the findings from Interior’s OIG report on ONRR and further stated that several aspects of royalty management activities required prompt and, in some cases, significant management attention. In particular, the report included over 100 recommendations for improving Interior’s management of oil and gas resources, including recommendations related to audit, compliance, and enforcement. Appendix I provides a list of the subcommittee’s recommendations and the status of their implementation, according to Interior documents and interviews with Interior officials. We identified several challenges with Interior’s management of federal oil and gas in the 2000s. In February 2011, in part because of the challenges identified in our past work, we added Interior’s management of federal oil and gas resources to our list of program areas at high risk for fraud, waste, abuse, and mismanagement. In the March 2019 update of our High-Risk List, we found that Interior had made progress improving its management of federal oil and gas resources. However, additional steps are needed to improve Interior’s royalty determination and collection. ONRR, according to officials, has begun implementing several initiatives that seek to make the agency operate more effectively. In March 2017, ONRR initiated Boldly Go, an effort to assess its organizational structure and identify and implement potential improvements. According to ONRR officials, this initiative was in response to March 2017 comments from the Secretary of the Interior, in which he said the department, in general, should undergo a “bold restructuring.” ONRR officials said that the Boldly Go organizational restructuring was implemented in October 2017 and included several changes to how ONRR conducts its compliance work. Before the reorganization, audits and compliance reviews were part of the same management group—referred to as the Audit and Compliance Management group. After the reorganization, audits and compliance reviews are managed separately. Audits now have their own management group, referred to as Audit Management. According to ONRR officials, the new Audit Management group conducts audits of multiple companies and properties and will attempt to identify more systemic misreporting issues common to those companies and properties. ONRR staff who conduct compliance reviews were moved into the same management group as the data mining staff in the new Compliance Management group. According to ONRR officials, the merger occurred because both groups use similar data sources to conduct less in-depth checks of the royalty data than audits. Additionally, officials stated that putting these activities under the same management could assist in better targeting companies for similar compliance issues. Prior to the reorganization, identifying and selecting cases for audits and compliance reviews was a function of the Audit and Compliance Management group. After the reorganization, this function was moved to a new Analytics and Risk Management group that is also tasked with using data analytics methods, such as computerized analysis of spatial and geographic data, to better identify noncompliant royalty payments. ONRR is also in the process of implementing a new electronic compliance case management and work paper tool referred to as the Operations and Management Tool (OMT). According to ONRR documents, OMT is to combine multiple systems into one and is intended to serve a variety of functions. ONRR documents state that OMT is designed to be a single standardized system that reduces manual data entry, creates a single system of record for ONRR case data, offers error checks to eliminate data entry errors, and provides greater transparency for outside auditors. One ONRR official stated that the agency plans to have ONRR’s data mining, compliance review, and audit teams all using OMT to manage their compliance work in 2019. According to some ONRR and state audit officials, ONRR piloted OMT’s electronic compliance case management system in North Dakota in 2018, and ONRR expects to offer OMT as an option to other STRAC partners for their audit and compliance review case management needs. Finally, the agency introduced a new auditor training curriculum in April 2018. Shortly after new auditors are hired, they are expected to begin ONRR’s training program, and according to ONRR’s training manual, they are expected to complete the training within 2 years of their hire dates. According to ONRR officials, courses will also be available to existing audit staff upon request. ONRR reported generally meeting its annual royalty compliance goals for fiscal years 2010 through 2017. To meet its compliance goals, ONNR used all three levels of compliance activities—audits, compliance reviews, and data mining—each of which provides a different level of assurance. However, ONRR’s compliance goals may not align with the agency’s mission to ensure the accuracy of royalty payments and other statutory requirements. ONRR reported generally meeting its annual compliance goals—those from Interior’s strategic plan and bureau-specific goals—for its royalty compliance program for fiscal years 2010 through 2017, and the agency made multiple revisions to its goals during this period. Our analysis of Interior’s annual budget justifications for fiscal years 2010 through 2017 found that ONRR reported meeting its compliance goals for 6 of the 8 fiscal years we reviewed (see table 1). According to ONRR officials we interviewed, the 2 years when the agency did not report meeting its compliance goals largely resulted from a shift in the focus of its goals that created a short-term misalignment of planned work and available resources. During fiscal years 2010 through 2017, ONRR revised its annual compliance goals multiple times. These included both compliance goals supporting Interior’s strategic plans covering fiscal years 2007 through 2018 and bureau-specific goals. In the revisions to its compliance goals, ONRR generally shifted from goals focused on the extent to which its compliance program was ensuring the accuracy of royalty payments to those focused on the efficiency of the program. ONRR’s accuracy goals, which included conducting compliance activities to cover a specific percentage of royalties, companies, or properties, helped it assess the extent to which it was ensuring the accuracy of royalty payments. That is, by measuring the portion of, for example, royalties subject to compliance activities, it was able to quantify the percentage of royalties that were reasonably correct or accurate. ONRR’s efficiency goals, which included conducting compliance activities to obtain a certain return on investment and additional amount of royalty collections, helped it assess whether resources spent on compliance activities were used cost effectively. According to ONRR officials, these revisions were made in an effort to continually improve its compliance performance. Table 2 identifies ONRR’s annual compliance goals for fiscal years 2010 through 2017 and establishes two categories for these goals corresponding to ONRR’s requirements under FOGRMA and the RSFA. Appendix II provides more detailed information on ONRR’s annual compliance goals and the agency’s reported compliance program performance. Conduct compliance activities on specified percentage of companies (coverage) Conduct compliance activities on specified percentage of properties (coverage) Conduct compliance activities on specified percentage of companies (coverage) Conduct compliance activities on specified percentage of companies (coverage) Conduct compliance activities on specified percentage of companies (coverage) Conduct compliance activities on specified percentage of payors and operators (coverage) Conduct compliance activities on specified percentage of payors and operators (coverage) Generate specified return on investment from compliance activities Generate specified return on investment from compliance activities Generate specified amount in total additional royalties Generate specified return on investment from compliance activities Generate specified amount in total additional royalties Legend: FOGRMA = Federal Oil and Gas Royalty Management Act of 1982, as amended; N/A = not applicable; RSFA = Federal Oil and Gas Royalty Simplification and Fairness Act of 1996; — = does not apply. While it is within ONRR’s purview to revise its compliance goals or targets, frequent changes may complicate management’s ability to assess performance over time because consistent goals are needed as a baseline from which to assess performance. For example, for the time period we reviewed, ONRR revised its compliance goals or goal targets nearly every year. This makes it difficult to assess, for example, how variations in resource allocations to and among its compliance activities may have affected the compliance program’s performance. The following are the types of compliance goals that ONRR used and revised for the period: Royalty coverage goal. Prior to fiscal year 2010, one of ONRR’s compliance goals was to conduct compliance activities on a specified percentage of royalties within 3 years of the date it received payment. In December 2006, Interior’s OIG issued an audit report that found, among other things, that the royalty compliance coverage goal had reduced the number of companies and properties subject to compliance work. The report stated that ONRR should consider modifying its compliance program strategy to ensure appropriate coverage of properties and companies within a reasonable time frame even if this resulted in a reduction in the overall percentage of dollars covered and recommended that ONRR develop separate performance measures for companies and properties subjected to compliance coverage. ONRR concurred with the recommendation and developed an implementation action plan. For fiscal year 2010, ONRR eliminated its royalty coverage goal in response to the OIG recommendation. Company/operator/payor and property coverage goals. For fiscal year 2010, ONRR revised its compliance goals to address company and property coverage, or conducting compliance activities—including audits and compliance reviews—on a certain percentage of companies and properties. ONRR’s fiscal year 2010 budget justification stated that the new compliance goals would reflect the cumulative percentage of unique companies and properties covered by audits, compliance reviews, or the royalty-in-kind compliance strategy. For fiscal year 2010, ONRR’s company coverage goal was to cumulatively conduct compliance activities on 57.6 percent of companies that paid royalties from fiscal years 2008 through 2012. ONRR’s property coverage goal was to cumulatively conduct compliance activities for 35 percent of properties where oil and gas had been extracted and sold from fiscal years 2008 through 2012. For fiscal year 2011, ONRR revised its compliance goals, eliminating the property coverage goal. ONRR also revised its company coverage goal to cumulatively conduct compliance activities on 66 percent of companies that paid royalties for fiscal years 2011 through 2016. ONRR further revised this goal for fiscal year 2014 to consider operators and payors instead of companies. In fiscal year 2014 ONRR established a compliance goal of conducting compliance activities on 90 percent of operators and payors but dropped the goal to 52 percent for fiscal year 2015. The goal for covering a percentage of operators and payors was eliminated beginning in fiscal year 2016, which left ONRR without a compliance goal addressing its coverage of royalty payments. Agency officials we interviewed told us that they eliminated ONRR’s company coverage goal because they concluded that the compliance program was reviewing too many companies and properties with smaller royalty payments, which officials deemed an inefficient use of limited compliance resources. ONRR officials added that budgetary constraints and the complexity of company bankruptcies and consolidation in the oil and gas industry also contributed to the goal’s elimination. Additionally, in 2008, ONRR established a data mining program to examine large sets of operator-reported data to identify royalty and reporting errors, such as when the production volumes that payors and operators reported for the same lease did not match. This work led to additional royalty collections, but ONRR did not consider these results when calculating its annual performance measure for company and property coverage. According to officials, data mining was the responsibility of ONRR’s Royalty Reporting group and was not considered compliance work. Return on investment goal. ONRR has had a goal for return on investment for fiscal years 2010 through 2017 that measured the efficiency of the compliance work that all of its program areas performed. However, Interior elevated this goal from a bureau-specific goal to a strategic plan goal for fiscal year 2017. This goal is a ratio of costs to collections for compliance activities—and is to assess whether ONRR collected additional royalties for every additional dollar the agency spends on compliance reviews, audits, and data mining. To account for variations in collections and oil and gas prices, ONRR is to calculate its performance on return on investment based on the royalties from the previous 3 years. For example, the return on investment the agency reported for fiscal year 2017 was based on revenues collected from fiscal years 2014, 2015, and 2016. According to ONRR officials, the goal for fiscal year 2017—to collect an additional $2 in royalties for every $1 spent on compliance activities—was developed based on trends from prior years. Achieving this return on investment would indicate that ONRR met its goal. Total additional royalty collections goal. In fiscal year 2016, ONRR developed a bureau-specific goal for total additional royalties collected from compliance reviews, audits, and data mining. The goal for fiscal year 2016 was to collect an additional $110 million from compliance activities. In the following fiscal year, 2017, ONRR elevated this goal to a strategic plan goal and kept the amount the same, at $110 million in additional royalties. To generally meet its compliance goals during fiscal years 2010 through 2017, ONRR used all levels of its compliance activities: audits, compliance reviews, and data mining. The number of audits completed annually generally remained the same for fiscal years 2010 through 2017, declining slightly from 162 in 2010 to 153 in 2017. During the same time period, the number of completed compliance reviews decreased, declining from 1,233 in 2010 to 683 in 2017 (see fig. 2). During this time frame, ONRR’s Data Mining group increased the number of exceptions resolved to address instances of incorrectly reported data from 4,323 in 2010 to over 26,000 in 2017. Our analysis of ONRR’s data on compliance activities showed that adding data mining financial results in 2011 was associated with a decrease in the return on investment for ONRR’s other compliance activities. Prior to including data mining, compliance reviews earned a 6 to1 return on investment, STRAC compliance work earned about a 4 to1 return on investment, and audits earned a 2 to1 return on investment. By the end of fiscal year 2017, data mining proved to be far more cost-effective for royalty compliance, with a return on investment of 9 to1. During the same time, return on investment declined for all compliance reviews (including STRAC compliance work) and all audits (see fig. 3). According to ONRR officials, the reason for this decline was that data mining was identifying royalties that might otherwise have been identified through audits or compliance reviews. Additionally, ONRR officials we interviewed stated that data mining has been more cost-effective than audits or compliance reviews in identifying additional royalties. Officials we interviewed stated that data mining often identifies more simple reporting errors. Return on investment is an indicator of the efficiency of ONRR’s compliance program. As long as ONRR is collecting more royalties through its compliance activities than it is spending on identifying those royalties, the federal government will obtain additional revenues. According to ONRR officials we interviewed, the agency does not calculate the potential additional royalty revenues that would be generated if it conducted additional compliance activities. However, ONRR officials said they do calculate the effect of reduced funding on compliance activities. For example, ONRR stated in its fiscal year 2018 budget justification document that reductions in its budget for compliance work would directly result in reductions to additional royalty collections. ONRR’s fiscal year 2017 compliance goals, the most recent compliance goals we reviewed, may be useful for assessing certain aspects of ONRR’s performance but may not be effectively aligned with the agency’s stated mission or fulfill other statutory requirements. ONRR’s 2017 strategic priorities document states that the agency’s mission is to collect, account for, and verify energy revenues. Additionally, statutory requirements under RSFA direct ONRR not to conduct audit activities if it and the relevant state determine that the cost of conducting or requiring the audit exceeds the expected amount to be collected by the activity, based on the most current 12 months of activity. ONRR’s fiscal year 2017 return on investment compliance goal helps the agency comply with RSFA by assessing whether the agency’s compliance program is cost- effective. Moreover, ONRR’s statutory requirements under FOGRMA require that it establish a comprehensive auditing system to provide the capability to accurately determine oil and gas royalties, among other requirements. However, ONRR’s fiscal year 2017 compliance goals do not sufficiently address its mission or FOGRMA requirements, in part, because its goals do not address accuracy—or consider the extent to which its compliance work is covering, for example, royalty payments. While ONRR previously had coverage goals, agency officials told us that they eliminated their company and property coverage goals because they concluded the compliance program was reviewing too many companies and properties with smaller royalty payments. ONRR officials told us that this was deemed an inefficient use of limited compliance resources. However, it is difficult for ONRR to provide reasonable assurance that it is accurately collecting royalties when it does not have data on the extent to which, for example, royalties or companies were subject to compliance activities. According to agency officials we interviewed, ONNR stopped tracking these data when ONRR eliminated its coverage goals for fiscal year 2016. As a result, ONRR could be determining that it is meeting its current annual compliance goals but potentially doing so by examining a small percentage of royalties or companies. For example, ONRR may be able to achieve a 2 to 1 return on investment, but only conduct compliance activities on 10 percent of the approximately $5 billion in royalties paid in calendar year 2017. This raises questions about the extent to which ONRR can provide reasonable assurance that its compliance program is assessing the accuracy of oil and gas royalty payments because it does not have a goal for, or data on, the amount of royalties subject to compliance activities. Finally, because ONRR no longer has a coverage goal—which helps it assess the extent to which it has ensured the accuracy of royalty payments—it does not track the amount of royalties subject to its differing level of compliance activities. ONRR has established a compliance program with three activities—audits, compliance reviews, and data mining—each of which offers varying levels of assurance for determining the accuracy of royalty payments. However, the extent to which its compliance program allows ONRR to accurately determine and collect royalty payments is unclear because the agency does not track each compliance activity’s contribution toward a coverage goal. Interior’s OIG reported a similar finding in December 2006. In its report, the OIG found that ONRR’s compliance goal for coverage of royalties was misleading because it weighed audits and compliance reviews equally, although the two compliance activities provided differing levels of assurance about whether royalties were accurately paid. The OIG recommended that ONRR should revise the compliance goal to account for each compliance activity separately. While ONRR did not concur with establishing a goal for each of the compliance activities, it agreed to internally track separate measures for them. According to ONRR documentation, the agency took steps to identify what amount of royalties was covered by audits or compliance reviews but did not report this information. When ONRR eliminated its coverage goals for fiscal year 2016, it no longer tracked information on the extent to which royalty payments were subject to its different levels of compliance activities. By establishing a coverage goal (e.g., identifying the number of companies or percentage of royalties subject to compliance activities over a set period of time) that aligns with the agency’s mission and tracking the extent to which each of its compliance activities contributes toward this goal, ONRR would have greater assurance that its compliance program has the capability to accurately determine oil and gas royalties. ONRR’s process to select compliance cases for audits and compliance reviews is not documented. Additionally, the agency does not have performance measures for determining whether its case selection process aligns with the agency’s compliance goals. Finally, while ONRR has a risk model to assist in selecting compliance cases, it has not analyzed the effect the risk model has had on its selection process. ONRR does not have a documented case selection process with procedures for how to select cases. According to ONRR officials, ONRR’s Work Planning Group reviews royalty information on federal oil and gas leases and selects leases from specific companies or properties to undergo either an audit or a compliance review. These officials also stated that while the process for selecting cases for audits and compliance reviews differs, the agency has no written procedures for either compliance activity on how cases should be selected. For audits, ONRR officials we interviewed told us that cases are generally selected based on research from ONRR’s recently established Analytics and Risk Management group, which includes the relocated Work Planning Group and other offices that analyze particular aspects of the oil and gas industry, such as pricing. According to these officials, the work planners or analytics staff review a variety of royalty payment and oil and gas production information to identify trends and outliers that may indicate potential royalty noncompliance. The officials told us that they also consider other factors in their selection decisions, such as whether a company was new—and therefore may be unfamiliar with how to correctly report royalties—or had undergone a change in ownership—which can lead to reporting errors. Additional factors that ONRR officials told us they considered were referrals by ONRR staff based on recently completed compliance activity on a specific company or property and the risk scores for the relevant companies and properties generated from the agency’s compliance risk model. For compliance reviews, ONRR officials we interviewed told us that the Work Planning Group includes information from a Go/No-go analysis, which they said allows ONRR to make a decision early in the process to cost effectively decide whether to initiate a compliance review. According to ONRR officials, the use of the Go/No-go analysis began in fiscal year 2015 as means to better ensure that the compliance activities they select will identify a finding of royalty noncompliance. According to the officials, the Work Planning Group then compares the list of companies and properties to other sources of information, such as the findings of recently completed work and ONRR’s royalty compliance risk model, to select cases based on the group’s professional judgment. Overall, ONRR officials we interviewed said that the Work Planning Group maintains a small pool of cases for either an audit or compliance review for when staff become available after completing other work. ONRR officials said that as there was no requirement that they develop documented procedures for case selection, they rely on the experience and training of members of the Work Planning Group to review the available information and select cases based on requests from the Audit Management and Compliance Management groups. Under federal standards for internal control, management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives, including the documentation of the internal control system. Documentation provides a means to retain organizational knowledge and mitigate the risk of consolidating that knowledge to a few personnel, as well as a means to communicate that knowledge as needed to external parties, such as external auditors. By developing a documented case selection process that includes procedures for how to select compliance cases, ONRR could better ensure that it retains the organizational knowledge needed to carry out the process effectively and can defend it to external parties. ONRR does not have performance measures for determining the extent to which cases selected align with the agency’s compliance goals. ONNR’s Work Planning Group is responsible for selecting cases—that is, companies or properties—to undergo a compliance review or audit. As mentioned previously, ONRR’s fiscal year 2017 goals for its compliance program are to achieve a specified return on investment and total amount of additional royalties collected from the cases it selects to undergo compliance activities. However, according to ONRR officials we interviewed, these goals are not considered when selecting cases. Rather, these officials told us that the Work Planning Group attempts to select cases for compliance activities that are the most likely to result in a finding of royalty noncompliance. A finding of noncompliance for a company can result from a variety of circumstances, such as reporting an incorrect volume of oil or gas sold or claiming allowances for transportation and processing costs above established limits. ONRR officials stated that the agency’s IT system tracks whether a completed compliance case resulted in a finding, but the agency does not regularly assess the percentage of completed cases that produce findings. Prior to 2015, ONRR had performance measures for determining the extent to which cases selected aligned with its compliance goals but stopped using these measures after it made changes to the goals. Prior to fiscal year 2010, for example, ONRR had a goal for conducting compliance activities on a certain percentage of royalties within 3 years from the date it received payment. To support this goal, ONRR sought to select companies with relatively high royalty dollar amounts. ONRR then assessed its performance toward achieving this goal by reviewing all completed audits and compliance reviews over a 3-year period and calculating the percentage of total royalties paid over this period from completed compliance cases. For example, in fiscal year 2008, ONRR reported that compliance cases covered 69 percent of royalties received in calendar year 2004. However, as we noted previously, the 2006 OIG report found that the focus on coverage of royalties resulted in ONRR providing limited coverage of its universe of companies and properties. Additionally, ONRR officials we interviewed confirmed that selecting cases with higher royalty amounts to achieve the royalty coverage goal resulted in more limited coverage of companies because the goal directed ONRR toward repeatedly selecting many of the same large companies each year for compliance activities. In response to the recommendations in the OIG’s report as well as ONRR’s own recognition of the reduced company coverage resulting from its selection of companies that pay high royalties, ONRR transitioned to a new performance measure for case selection along with a new compliance goal in fiscal year 2010. ONRR’s new performance measure assessed the number of unique companies and properties for selected compliance activities. This new performance measure, according to ONRR officials, was driven by ONRR’s new compliance goal for cumulatively covering a certain percentage of unique companies and properties over a 3-year period. According to ONRR officials, after the company and property coverage goal was in place for approximately 5 years, officials determined that this goal and corresponding performance measure was driving the compliance case selection process to select too many companies and properties with smaller royalty payments, which they deemed an inefficient use of limited compliance resources. As a result, ONRR officials told us that the agency decided to change its compliance goal in fiscal year 2015 to focus on return on investment and total additional dollars collected. However, ONRR did not establish a corresponding performance measure for its compliance case selection process that would determine the extent to which cases selected contributed to ONRR’s compliance goals. As stated previously, we have reported that the requirements in GPRA and GRPAMA for establishing performance metrics serve as leading practices for divisions, programs, and initiatives. Performance measures help agencies make resource decisions, provide managers information on which to base their organizational and management decisions, and create powerful incentives to influence organizational and individual behavior. Furthermore, successful performance measures are aligned with division and agency-wide goals and missions. According to ONRR officials we interviewed, they have not established performance measures for determining whether the way such cases are selected aligns with the agency’s compliance goals because there is no specific requirement to do so. By developing performance measures (e.g., establishing a specified percentage of compliance cases that identify findings of royalty noncompliance or total additional royalties) that assess whether the agency is selecting cases that are helping it achieve its compliance goals, ONRR would be able to better monitor its performance in achieving its goals and whether changes to its selection process affect its performance. ONRR has developed a model that assesses the risk of noncompliance for companies and properties. Officials from the Work Planning Group use this model to inform their compliance case selections. However, it is unclear whether use of the model has improved case selection because ONRR has not analyzed the model’s effect on such selections. ONRR began a pilot program in 2006 to analyze the risk factors for royalty noncompliance, which included developing a quantitative risk model. In December 2006, Interior’s OIG recommended that ONRR consider additional factors that may indicate a risk of noncompliant royalty payments when making case selection decisions. In addition to the factors that ONRR was using to select cases to help achieve its compliance goals, such as cases with high royalty dollars, the OIG recommended that ONRR incorporate other risk factors, including companies or properties having a history of underreported royalties and falsely reported information to other federal agencies, such as the Environmental Protection Agency. In December 2007, Interior’s Subcommittee on Royalty Management reiterated the importance of using a risk-based process for compliance and made a number of related recommendations to ONRR. Among these were that ONRR should fully implement the quantitative model it was developing as part of its pilot program. Additionally, the subcommittee recommended that ONRR evaluate its risk model’s performance and then establish a process to continually validate and update the model to ensure that it remains effective. In response to these recommendations, ONRR worked with a contractor from 2006 through 2012 to develop an initial risk model. This model evaluated the risk of royalty noncompliance for each lease based on four characteristics: the type of lease, the specific location of the lease, the region of the country the lease was in, and the type of commodity extracted. The model looked at a number of indicators of risk, which were grouped into four overall risk drivers: complexity of the oil and gas market, complexity of regulations, commodity-specific practices, and transparency of the market. According to officials from the Work Planning Group we interviewed, they used the risk scores generated from this model to help inform the list of compliance cases to be reviewed the following year. These officials told us that they stopped using the scores from this model around 2012 for two reasons. First, the model allowed for the scores to be manually weighted based on the judgment of those selecting the compliance cases, and this weighting process was believed to have eventually hurt the accuracy of the risk scores. Second, agency officials determined that the risk scores the model was producing did not correlate closely with cases resulting in significant findings. In 2013, ONRR tasked a different contractor with developing a new set of risk models. According to an initial development document, ONRR requested separate risk models for companies and properties that would determine the propensity for a company to submit an incorrect royalty payment using historical royalty compliance data that ONRR and third- party sources provided. The contractor produced two risk models, one that assigned a risk score to companies and one to properties. The risk scores—which ranged from 0 to 100—attempted to quantify the risk of royalty noncompliance. The initial models were completed in 2014, and ONRR began including the risk scores from these models in the data that the Work Planning Group reviewed during the case selection process. Documents from the contractor show that the models then went through an initial validation process using the results of cases that the contractor selected when the models were instituted in 2014 and completed cases from 2012 onward. According to documents summarizing the contractor’s efforts, the validation showed a correlation between higher risk scores on the company model and cases that resulted in findings and additional royalty revenues. However, the contractor reported that higher risk scores on the property model did not correlate with either findings or additional royalty revenues. The documents we reviewed also included a number of recommendations to ONRR to improve its risk modeling, including adding third-party and commercial data sources, adding data sources from within ONRR and other oil and gas bureaus within Interior, and attempting to redefine property risk and building a new property risk model. However, according to officials we interviewed, they have not yet acted on any of these recommendations. ONRR requested that the contractor update the models with data from recent royalty reporting and completed compliance cases, which it did in both 2015 and 2018 but does not do either regularly or periodically. According to ONRR officials, the Work Planning Group currently considers the risk scores based on the company model when selecting cases but does not consider the risk scores based on the property model, as the group considers those scores less reliable. ONRR officials told us that they do not believe that their current risk approach is entirely effective and are considering having staff from the Analytics and Risk Management group develop a risk model for the agency. To date, ONRR has not analyzed how the use of the risk scores has affected case selection or findings of royalty noncompliance and is therefore unable to identify whether its risk model is effective. As a result, the agency does not have sufficient information to make a decision on whether to continue using the model as it exists today, consider potential improvements, or discontinue the model in favor of another approach. Federal standards for internal control state that management should design control activities to achieve objectives and respond to risks, such as by comparing actual performance to planned or expected results and analyzing significant differences. By periodically analyzing whether the risk model is effectively identifying potential royalty noncompliance and whether the model’s results are being effectively used to assist in case selection, and making changes to the model (e.g., updating it) or developing a new model based on this analysis, ONRR would be better able to determine how to proceed with using risk analysis to inform its case selections. STRAC officials we interviewed from the nine member states that had agreements with ONRR for conducting royalty compliance generally expressed satisfaction with ONRR’s coordination of compliance activities, including both the frequency of interaction as well as support for budget and training. STRAC officials from all nine member states generally expressed satisfaction with the frequency of interaction between STRAC and ONRR. STRAC officials stated that this interaction occurred primarily through three mechanisms. First, ONRR and STRAC hold semiannual in-person meetings. At these meetings, STRAC officials said that attendees discuss a range of topics. For example, at the March 2018 STRAC meeting in Sacramento, California, which we attended, there were two training sessions as well as a session on updates to ONRR’s IT systems. Second, ONRR and STRAC hold quarterly teleconferences. These teleconferences, according to STRAC officials, are opportunities for both ONRR and STRAC to highlight any significant or systematic issues that they may be identifying in their compliance activities. Third, ONRR assigned agency points of contact to each STRAC member state for technical questions. The STRAC officials stated that ONRR has been responsive when they have reached out with questions or concerns. Overall, STRAC officials from seven of the nine member states said that coordination with ONRR had improved over the past approximately 10 years. STRAC officials from two of the member states attributed this improvement to ONRR leadership’s concerted effort to work more effectively with STRAC. Additionally, STRAC officials from the majority of member states generally expressed satisfaction with the support ONRR has provided STRAC member states with respect to resources and training. STRAC officials from seven of nine member states told us that the current contracted budget was sufficient to conduct oversight of their states’ federal oil and gas royalties. STRAC officials from two member states stated that the budget was insufficient. One official stated that the budget did not allow the state to review all of the federal properties for which it was responsible. The officials from the other state indicated that a larger budget would allow the state to hire additional auditors. According to these officials, additional auditors could help the state conduct compliance activities on more royalty payors and in particular small royalty payors that may not be as familiar with the requirements for federal royalty payments. STRAC officials from several member states said that the flat budget that ONRR provided over the past several years may lead to changes in their federal royalty compliance activities. For example, one official stated that without additional funding in the future, the state may have to move more experienced and higher paid auditors to state royalty compliance activities, thus leaving less experienced and lower paid auditors to conduct federal royalty compliance activities. Another official stated that the state had offered less training and reduced the amount of funds for travel to address potential budget shortfalls. Additionally, another official stated that flat budgets could make it difficult to offer staff merit pay increases. Finally, officials from seven of the nine STRAC member states said ONRR provided sufficient training on policies, procedures, and IT systems used to conduct compliance activities on federal oil and gas royalties. A STRAC official from one member state said that ONRR’s training had improved recently, while another official said that support had improved. However, STRAC officials from three member states expressed uncertainty about ONRR’s training for companies. These officials stated that they would like to understand the content of the training so they would better understand how ONRR is training companies to report royalties. None of the nine STRAC member states had documented case selection processes. Specifically, officials from all nine STRAC member states we interviewed said that either they did not have, or were unable to provide, documented procedures for the processes they used to select federal oil and gas compliance cases. Rather, STRAC officials stated that they relied on a variety of factors to select cases for compliance reviews. Staff expertise about companies and properties was the factor that all nine STRAC officials identified as key for case selection. For example, one official stated that she had over 10 years of experience and therefore knew what companies or properties to review. Another official stated that because staff also work on state tax audits, they can use knowledge from that work to help identify compliance cases. Another factor officials identified as assisting in the case selection process was ONRR’s company and property risk scores, though they were given varying degrees of consideration. Other factors officials identified included referrals from BLM or ONRR, and risk scores generated from their own models. Under federal standards for internal control, management should establish an organization structure, assign responsibility, and delegate authority to achieve the entity’s objectives, including the documentation of the internal control system. Documentation provides a means to retain organizational knowledge and mitigate the risk of consolidating that knowledge to a few personnel, as well as a means to communicate that knowledge as needed to external parties, such as external auditors. Because STRAC members do not have a documented process, ONRR cannot, for example, assess whether STRAC members are selecting compliance cases in a manner that aligns with ONRR’s compliance goals or that future STRAC members will know how to select compliance cases. In the agreements between the nine STRAC members and ONRR, the agency includes terms and conditions that the members agree to, but ONRR does not require STRAC members to have documented procedures for compliance case selection. By including in ONRR’s future agreements with STRAC members requirements to develop a documented case selection process, including procedures for how to select compliance cases and how to document which factors were considered in selection decisions, ONRR could better assess whether members select cases that align with the agency’s compliance goals. We reviewed STRAC members’ annual work plans to determine whether the compliance activities discussed aligned with ONRR compliance goals. STRAC member agreements from eight of the nine STRAC members included language that the “state will contribute to ONRR’s GPRA goals and thereby the performance goals of this Agreement by performing audits, compliance reviews and other investigations in coordination with ONRR. The yearly performance goals are listed on the state’s annual work plan.” However, when we reviewed the STRAC members’ corresponding annual work plans, we found no information on how the members’ compliance activities contributed to ONRR’s goals. For example, several of the STRAC members’ work plans included information on the leases and properties selected for compliance activities but did not include information on how those selections would contribute to ONRR’s compliance goals. In addition, the majority of STRAC officials from member states said they did not consider ONRR’s compliance goals for return on investment or total additional royalty collections when selecting compliance cases. When we asked STRAC members about their goals, three of nine STRAC member states noted that they had compliance program goals. For example, one STRAC member’s goal was to “maximize revenue to the state” and “implement on behalf of ONRR and the state, a constantly improving and efficient royalty audit program.” Another member’s goal was to “protect the US Citizens’ Federal Mineral Interest within the boundaries of the state by ensuring that a fair value, as established by the federal regulations, is received.” Officials from the other six STRAC members told us that their states do not have goals for federal oil and gas royalty compliance activities because ONRR does not require that they do so. For STRAC members that did not have compliance goals, officials provided examples of informal goals—or goals that were not documented. For example, one STRAC official reported that the state’s goal was to try to audit 50 percent of royalties paid to the state every 2 years. Another STRAC official stated the state tries to review major market areas in the state once every 7 years. When we compared STRAC officials’ responses on their goals to ONRR’s broader compliance goals for return on investment and total additional royalty collections, we found that the majority of states’ compliance goals did not align with ONRR’s goals. Federal standards for internal control state that management should define objectives clearly to enable the identification of risk and define risk tolerances, such as by defining objectives in alignment with the organization’s mission, strategic plan, and performance goals. In requiring eight of the nine STRAC members to conduct compliance activities consistent with the agency’s compliance goals, ONRR was following these standards. However, ONRR approved the STRAC members’ work plans, although those work plans did not specify how the described members’ compliance activities would contribute to ONRR’s goals as the agency stated they would in the agreements between the seven of the nine STRAC members and ONRR. By requiring STRAC members to describe in their annual work plans how their compliance activities would align with ONRR’s current compliance goals, ONRR would have better assurance that activities were aligned with its compliance goals. Finally, ONRR does not track STRAC member states’ contributions against its annual compliance goals. ONRR has the data available to track these contributions because the results of STRAC members’ compliance activities are retained in ONRR’s IT system. For example, we obtained reports on the aggregate overall return on investment of STRAC members and reviewed individual data entries from STRAC members’ work that included a data field for revenue collections. According to regulations, if a state accepts delegated authority, it is to assist ONRR in meeting the requirements of GPRA as well as in developing and endeavoring to comply with ONRR’s Strategic Plan and Performance Measurements. Because ONRR does not track STRAC member states’ contributions toward its annual compliance goals, the agency has limited information for assessing whether the funding they are providing to STRAC members is achieving its goals. ONRR officials we interviewed stated that they do not track states’ contributions to ONRR’s overall compliance goals as there is no requirement to do so. However, by tracking the performance of each state and its contribution toward ONRR’s compliance goals, ONRR could better assess the effectiveness of states’ performance in supporting the agency’s mission of ensuring accurate royalty payments. ONRR is taking steps intended to improve its royalty compliance program and better verify that all royalties paid on the sale of oil and gas extracted from leased federal lands are accurate. These steps include reorganizing the management structure of its compliance program, implementing new systems for managing compliance cases electronically, and instituting a training curriculum for newly hired auditors. However, although ONRR reported generally meeting its compliance goals for fiscal years 2010 through 2017, its current goals may not align with the agency’s mission or other statutory requirements. For example, ONRR’s fiscal year 2017 compliance goals do not sufficiently address its mission to collect, account for, and verify revenues, in part, because its goals do not address accuracy, such as through a coverage goal. Establishing a coverage goal (e.g., identifying the number of companies or percentage of royalties subject to compliance activities over a set period) that aligns with the agency’s mission, and tracking the extent to which each of its compliance activities contributes to this goal, would provide ONRR more reasonable assurance that its compliance program is assessing the extent to which oil and gas royalty payments are accurate. Furthermore, ONRR’s audits, compliance reviews, and data mining efforts each provide a different level of assurance that royalties are accurately paid, but the agency does not measure how each of the compliance activities contributes to the FOGRMA requirement to establish a system with the capability to accurately determine and collect royalties in a timely manner. By tracking the extent to which each of its compliance activities contributed to any future coverage goal, ONRR would have greater assurance that its compliance program has the capability to accurately determine oil and gas royalties. In addition, ONRR’s compliance program relies on its Work Planning Group, which is responsible for reviewing information on companies and properties to select cases for audits or compliance reviews. The Work Planning Group, however, does not have a documented case selection process. By developing a documented case selection process that includes procedures for how to select compliance cases, ONRR could better ensure that it retains the organizational knowledge needed to effectively select compliance cases and defend the process in external reviews. In addition, ONRR does not have performance measures to determine the extent to which cases selected align with ONRR’s compliance goals. By developing performance measures (e.g., establishing a specified percentage of compliance cases that identify findings of royalty noncompliance or total additional royalties) that assess whether the agency is selecting cases that are helping it achieve its compliance goals, ONRR would be able to better monitor its performance in achieving its goals and whether changes to its selection process affect performance. Moreover, since 2006, ONRR has worked to develop a model to assess the risk of royalty noncompliance for use in its compliance case selection process. After several iterations with two contractors, ONRR began using the risk scores from its model to assist with case selection in 2014. However, according to ONRR officials, the agency is considering discontinuing the use of its current model in favor of one that is internally developed. ONRR has not analyzed how the use of the risk scores has affected case selection or findings of royalty noncompliance and is therefore unable to identify whether its risk model is effective. By periodically analyzing whether the risk model is effectively identifying potential royalty noncompliance and whether the model’s results are being effectively used to assist in case selection and making changes to the model (e.g., updating it) or developing a new model based on this analysis, ONRR would be better able to determine how to proceed with using risk analysis to inform its case selections. Furthermore, none of the nine STRAC members had documented case selection processes. In the agreements between the nine STRAC members and ONRR, the agency includes terms and conditions that the members agree to, but ONRR does not require STRAC members to have documented procedures for compliance case selection. By including requirements in ONRR’s agreements with STRAC members to develop a documented case selection process, including procedures for how to select compliance cases and how to document which factors were considered in selection decisions, ONRR could better assess whether members select cases that align with the agency’s compliance goals. Additionally, ONRR does not require that STRAC members specify how their compliance activities included in annual work plans contribute to ONRR’s compliance goals, although those goals appear on the work plans. ONRR approved the work plans but did not specify how the members’ compliance activities would contribute to its goals as the agency stated they would in the agreements between eight of the nine STRAC members and ONRR. By requiring STRAC members to describe in their annual work plans how their compliance activities would align with ONRR’s current compliance goals, the agency would have better assurance that activities were aligned with its performance goals. Lastly, ONRR does not track STRAC members’ contributions toward its annual compliance goals though it has the data to do so. By tracking the performance of each state and its contribution toward ONRR’s compliance goals, ONRR could better assess the effectiveness of states’ performance in supporting its mission of ensuring accurately royalty payments. We are making a total of seven recommendations to ONRR. Specifically: The Director of ONRR should establish an accuracy goal (e.g., identifying the number of companies or percentage of royalties subject to compliance activities over a set period of time) that aligns with the agency’s mission of collecting, accounting for, and verifying royalty payments. In doing so, ONRR should track the extent to which each compliance activity (audits, compliance reviews, and data mining) contributes toward achieving this goal. (Recommendation 1) The Director of ONRR should develop a documented case selection process that includes procedures for how to select all compliance cases. (Recommendation 2) The Director of ONRR should develop performance measures (e.g., having a specified percentage of compliance cases identify findings of royalty noncompliance or total additional royalties) that assess whether the cases the agency is selecting are helping it achieve its compliance goals. (Recommendation 3) The Director of ONRR should periodically analyze whether the risk model is effectively identifying potential royalty noncompliance and whether the model’s results are being effectively used to assist in case selection, and should use this analysis to make changes to the model (e.g., updating it) or develop a new model. (Recommendation 4) The Director of ONRR should include requirements in ONRR’s agreements with STRAC members to develop a documented case selection process, including procedures for how to select compliance cases and how to document which factors were considered in selection decisions. (Recommendation 5) The Director of ONRR should require STRAC members to describe in their annual work plans how their compliance activities would align with ONRR’s current compliance goals. (Recommendation 6) The Director of ONRR should track the performance of the compliance work of each state STRAC member and the contribution that each state makes to ONRR’s compliance goals. (Recommendation 7) We provided a draft of this report to Interior for review and comment. Interior concurred with all seven recommendations. Agency comments are reproduced in appendix III. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of the Interior, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made major contributions to this report are listed in appendix IV. We reviewed and summarized recommendation closure documentation that the Department of the Interior (Interior) provided for royalty compliance recommendations made to the department by the Royalty Policy Committee’s Subcommittee on Royalty Management in 2007 and Interior’s Office of Inspector General (OIG) in 2006. For the subcommittee recommendations, Interior officials told us that the subcommittee did not assess the implementation of its recommendations. As a result, we present information that Interior provided on its decision about the status of the recommendations and a summary of actions taken. (See table 3.) For the OIG recommendations, we present the status of recommendations according to the OIG and a summary of the actions according to Interior. (See table 4.) We did not independently assess the implementation of the recommendations. See table 5 for detailed information on the Office of Natural Resources Revenue’s (ONRR) performance goals, including goal type, goal, fiscal year goal, fiscal year performance, and long-term target for performance goal. In addition to the contact named above, Christine Kehr (Assistant Director), Glenn C. Fischer (Analyst-in-Charge), Tim Bober, John Delicath, Sarah Detweiler, Wil Gerard, Cindy Gilbert, Michael Kendix, Eli Lewine, Ben Licht, Anne Stevens, and Sara Sullivan made important contributions to this report.", "summary": "Royalties paid on the sale of oil and gas extracted from leased federal lands and waters are a significant source of revenue for the federal government. However, Interior has faced challenges verifying the accuracy of royalty payments. In the 2000s, GAO issued reports highlighting weaknesses in Interior's royalty compliance program. In 2011, GAO added Interior's management of federal oil and gas resources to its High-Risk List, in part because its work showed Interior did not have assurance that it was collecting its share of revenue from oil and gas produced on federal leases. Interior has taken steps to operate more effectively. GAO was asked to examine ONRR's federal oil and gas royalty compliance efforts. This report examines, among other objectives, the extent to which ONRR reported meeting its compliance goals for fiscal years 2010 through 2017, the most recent data available. GAO reviewed relevant laws, regulations, agency guidance, and Interior's annual performance plan and report and annual budget justifications for the period; analyzed ONRR compliance data for the period; and interviewed ONRR officials and state auditors who conducted work in coordination with ONRR. The Department of the Interior's (Interior) Office of Natural Resources Revenue (ONRR) reported that it met its annual performance goals for its royalty compliance program in 6 of the 8 years from fiscal years 2010 through 2017. Under this program, ONRR conducts three levels of compliance activities—audits, compliance reviews, and data mining—to help ensure that oil and gas royalty payments submitted by companies that produce oil and gas from federal leases are accurate and comply with federal laws and regulations (see figure). Specifically, GAO's analysis of Interior's annual budget justifications for fiscal years 2010 through 2017 found that ONRR reported meeting its compliance goals for 6 of the 8 fiscal years. According to ONRR officials, ONRR did not report meeting its compliance goals for 2 years because of a shift in the agency's goals that created a short-term misalignment of planned work and available resources. ONRR's fiscal year 2017 goals for its compliance program were (1) to obtain a return of $2 of additional royalties for every dollar spent on compliance activities and (2) to collect a defined amount of additional royalties. ONRR's compliance goals generally aligned with the agency's requirement that resources should not be expended without an expected return. However, these goals may not align with the agency's mission to collect, account for, and verify royalty payments and other statutory requirements because the goals do not address accuracy—or the extent to which its compliance work is covering, for example, royalty payments. By establishing a goal that addresses accuracy, for example, by covering a portion of royalty payments with its compliance activities, ONRR could increase the extent to which it had reasonable assurance that its compliance program is fully accounting for federal oil and gas royalty payments. GAO is making seven recommendations, including that ONRR establish an accuracy goal that addresses coverage that aligns with its mission. Interior concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Micronesia, the Marshall Islands, and Palau are among the smallest countries in the world. In fiscal year 2017, the three FASs had a combined resident population of approximately 175,000 (102,622 in Micronesia; 54,354 in the Marshall Islands; and 17,901 in Palau). Interior’s Office of Insular Affairs (OIA) has primary responsibility for monitoring and coordinating U.S. assistance to the FASs, and State is responsible for government-to-government relations. The U.S. relationship with the FASs began when American forces liberated the islands from Japanese control near the end of World War II. In 1947, the United States entered into a trusteeship with the United Nations and became the administering authority over Micronesia, the Marshall Islands, and Palau. Voters approved the Constitution of the Federated States of Micronesia in 1978 and approved the Constitution of the Marshall Islands in 1979. Both Micronesia and the Marshall Islands remained subject to the authority of the United States until 1986, when a compact of free association went into effect between the United States and the two nations. The Palau constitution took effect in 1981, and Palau entered into a compact of free association with the United States in 1994. Micronesia and Marshall Islands became members of the United Nations in 1991, while Palau joined the organization in 1994. Under its compacts with Micronesia, the Marshall Islands, and Palau, the United States provided economic assistance that includes access to certain federal services and programs, among other things, for defined time periods. Economic assistance to Micronesia and the Marshall Islands. The 1986 compact of free association between the United States and Micronesia and the Marshall Islands, respectively, provided about $2.6 billion in funding for fiscal years 1987 through 2003. In 2003, the United States approved amended compacts of free association with the two countries. According to Interior, economic assistance under the amended compacts is projected to total $3.6 billion, including payments for compact sector grants and trust fund contributions for both countries in fiscal years 2004 through 2023. Funding under the original compact and amended compacts has been provided to Micronesia and the Marshall Islands through Interior. Economic assistance to Palau. The compact of free association between the United States and Palau entered into force in 1994 and provided $574 million in funding through Interior for fiscal years 1995 through 2009 for assistance to the government, contributions to a trust fund, construction of a road, and federal services. In September 2010, the United States and Palau signed an agreement that would, among other things, provide for additional assistance to Palau, including contributions to its trust fund. The 2010 agreement and subsequent amendments entered into force in September 2018. According to Interior, direct assistance to Palau under the compact will total $229 million for fiscal years 2010 through 2024, including $105 million that Congress provided in annual appropriations in fiscal years 2010 through 2017. Under the compacts, the United States has responsibility for defense and security matters in, and relating to, each of the FASs, and subsidiary agreements pursuant to the compacts provide for U.S. military use and operating rights in these countries. According to the Department of Defense, the compacts have enabled it to maintain a critical strategic position in the Indo–Pacific region. The compact with the Marshall Islands also provided for a separate agreement that constituted a full and final settlement of all claims resulting from U.S. nuclear tests conducted in the Marshall Islands during the period 1946 through 1958. In addition, a subsidiary agreement with the Marshall Islands secured the United States’ access to the U.S. military facilities on Kwajalein Atoll, which are used for missile testing and space tracking activities. Under the compacts, eligible FAS citizens are exempt from certain visa and labor certification requirements of the Immigration and Nationality Act as amended. The migration provisions of the compacts allow eligible FAS citizens to enter the United States (including all states, territories, and possessions) and to lawfully work and reside in the United States indefinitely. The implementing legislation for the 1986 compact with Micronesia and the Marshall Islands stated that it was not Congress’s intent to cause any adverse consequences for U.S. territories and commonwealths and the state of Hawaii. The legislation further declared that Congress would act sympathetically and expeditiously to redress any adverse consequences. In addition, the legislation authorized compensation to be appropriated for these areas that might experience increased demands on their educational and social services from compact migrants from Micronesia, the Marshall Islands, and Palau. The legislation required the President to report and make recommendations annually to Congress regarding adverse consequences resulting from the compact and provide statistics on compact migration. In November 2000, Congress made the submission of annual reports about the impact of compact migration in affected jurisdictions—that is, compact impact reports—optional and shifted the responsibility for preparing these reports from the President to the governors of Hawaii and the territories. In December 2003, Congress took steps in the amended compacts’ implementing legislation to address compact impact in designated U.S. areas. The legislation restated Congress’s intent not to cause any adverse consequences for the areas defined as affected jurisdictions— Hawaii, Guam, the CNMI, and American Samoa. In addition, the legislation authorized and appropriated funding for compact impact grants to the affected jurisdictions, to be allocated on the basis of the proportion of compact migrants living in each jurisdiction. Further, the legislation required an enumeration of compact migrants to be undertaken at least every 5 years. The legislation also permitted affected jurisdictions to submit compact impact reports to the Secretary of the Interior. The implementing legislation for the amended compacts authorized and appropriated $30 million for each fiscal year from 2004 through 2023 for grants to the affected jurisdictions. According to the legislation, the grants are provided to aid in defraying costs incurred by these jurisdictions as a result of increased demand for services due to the residence of compact migrants. OIA reviews the affected jurisdictions’ annual proposals for the use of the funds and provides the funds to the jurisdictions as compact impact grants. The grants are to be used only for health, educational, social, or public safety services or for infrastructure related to such services. Figure 1 shows the locations of the FASs and the affected jurisdictions. The implementing legislation for the amended compacts requires Interior to conduct an enumeration of compact migrants, which is to be supervised by the Census Bureau or another organization selected by Interior, at least every 5 years beginning in fiscal year 2003. On the basis of these enumerations, each affected jurisdiction is to receive a portion of the annual $30 million appropriation in proportion to the number of compact migrants living there. The legislation permits Interior to use up to $300,000, adjusted for inflation, of the annual appropriation for compact impact to conduct each enumeration. The amended compacts’ implementing legislation defines a compact migrant, for the purposes of the enumeration, as “a person, or their children under the age of 18, admitted or resident pursuant to [the compacts] who as of a date referenced in the most recently published enumeration is a resident of an affected jurisdiction.” Compact migrants have varying eligibility for certain U.S. federal government programs. Eligibility for some federal programs changed as a result of the 1996 Personal Responsibility and Work Opportunity Reconciliation Act. For example, when the compacts were signed, FAS citizens were eligible for Medicaid; however, the act removed this eligibility. Table 1 shows compact migrants’ eligibility status for selected federal benefit programs as of November 2019. From 2009 to 2018, the number of compact migrants living in U.S. states and territories rose by an estimated 68 percent, from about 56,000 to about 94,000. In 2011, we reported that combined data from the Census Bureau’s 2005-2009 American Community Survey and 2008 enumeration showed an estimated 56,345 compact migrants living in U.S. areas. During the period 2013 to 2018, an estimated 94,399 compact migrants lived in U.S. areas, according to combined data from the Census Bureau’s 2013-2017 American Community Survey and 2018 required enumeration in Guam, the CNMI, and American Samoa. This estimate includes Micronesian and Marshallese citizens who entered the United States after 1986, Palauan citizens who entered the United States after 1994, and certain U.S.-born children younger than 18 years. Data from the 2013-2017 American Community Survey and the 2018 enumeration indicate that an estimated 50 percent of compact migrants lived on the U.S. mainland and an estimated 49 percent lived in the affected jurisdictions during this period: 26 percent in Hawaii, 20 percent in Guam, and 3 percent in the CNMI. This estimate indicates growth in the number of compact migrants on the U.S. mainland since 2011, when we reported that the Census Bureau estimated 58 percent of compact migrants lived in the affected jurisdictions. The Census Bureau estimated that 11 states in the U.S. mainland, in addition to three of the four affected jurisdictions—Hawaii, Guam, and the CNMI—had compact migrant populations of more than 1,000, according to the 2013-2017 American Community Survey and the 2018 enumeration (see fig. 2). Stakeholders we interviewed—including FAS embassy and consular officials, FAS community members, state government officials, and representatives of private sector and nonprofit organizations—expressed concerns about the Census Bureau’s prior estimates of compact migrants. Some Arkansas stakeholders cited other, higher estimates of the FAS population in their state. Moreover, some stakeholders said that compact migrant populations are apprehensive or distrustful about being formally counted through surveys or the census. Stakeholders also noted that some compact migrant communities have felt frustrated at having been encouraged to respond to surveys and be counted but not experiencing any benefit from these efforts, according to a nonprofit official and FAS community members. Marshallese consular officials said that they believed the 2010 census undercounted their citizens, noting that the Census Bureau did not employ any Marshallese surveyors in the Arkansas counties with Marshallese populations. Stakeholders also expressed concern about the decennial census to be conducted in 2020, which, like the 2010 decennial census, will collect information on race. Nonprofit organization officials whom we interviewed expressed concern that the 2020 census could result in an undercounting of compact migrants because of language barriers and compact migrants’ difficulty accessing the census form online. Arkansas health care and private sector representatives and the Marshallese consulate described plans to address barriers to obtaining a more accurate count of the population in the 2020 census. Hawaii is making a statewide effort to ensure that compact migrants are counted in the 2020 census, according to Hawaii state officials. According to Guam officials, an outreach effort in Guam has leveraged “trusted voices,” or parties known to compact migrant communities there, to communicate the importance of responding to the 2020 census. Data from the American Community Survey showed an estimated 72,965 compact migrants living in the 50 states, the District of Columbia, and Puerto Rico in 2013 through 2017. An estimated 31,425 compact migrants living in these areas (43 percent) were U.S. citizens. The remaining estimated 41,540 (57 percent) were not U.S. citizens. The U.S. citizens who were counted included naturalized citizens and minor-age U.S. citizen children of compact migrants, who would no longer be counted as compact migrants after reaching 18 years of age. An estimated 25,555 compact migrants living in these areas were born in Micronesia; 20,545 were born in the Marshall Islands; and 3,435 were born in Palau. These totals do not include compact migrants born in the FAS and living in Guam, the CNMI, or American Samoa, because the American Community Survey does not cover these territories. An estimated 27,735 compact migrants living in these areas who were 18 years and older (69 percent) were in the civilian labor force. Of those, 24,540 (89 percent) were employed and 3,195 (12 percent) were unemployed. An estimated 1,660 compact migrants living in these areas—4 percent of compact migrants 17 years and older—were on active duty in the U.S. military or had served on active duty in the past. For additional American Community Survey data on compact migrant demographics, see appendix IV. Compact migrants move to U.S. areas for a range of reasons, including greater economic and educational opportunities, better access to health care, a desire to join family members in the United States, and a wish for greater personal freedom. In some communities we visited, stakeholders noted that FAS citizens had come to the United States for school or work before the compact with Micronesia and the Marshall Islands and the compact with Palau went into effect but that the compacts had opened the option of migration to a broader range of individuals. Economic opportunities. Compact migrants described moving to U.S. areas for better, more reliable jobs and higher wages. Having a better-paying job in the United States sometimes allows individuals to send remittances or consumer goods to family members living in an FAS. Other compact migrants move to U.S. areas to join the military. Educational opportunities. Compact migrant families often move to U.S. areas so that their children will have access to improved primary and secondary education, according to compact migrants. Some compact migrants travel to U.S. areas to attend college and choose to stay to work, including to pay off their student loans, according to consular officials and compact migrants. Health care access. Compact migrants sometimes migrate to U.S. areas to obtain medical treatment for themselves or family members, according to FAS community members and consular officials. Some medical procedures or treatments, such as dialysis or access to specialists, are not available in the FASs, according to federal and nonprofit officials. Family. Many compact migrants relocate to the United States to join family members and communities already living there, according to consular and nonprofit officials. Personal freedom. Some compact migrants said that they have more personal, social, and cultural freedom in the United States than in their more traditional home country. Changes in the natural environment in the FASs have also prompted migration from those areas, according to FAS representatives. Depleted food resources and effects of climate change—including more-frequent typhoons, coral reef bleaching, and depletion of fishing stocks—have contributed to migration, according to an FAS official. In addition, members of Marshallese communities cited rising sea levels and frequent tidal flooding as reasons for migrating from the Marshall Islands to U.S. areas. Some Marshallese community members also noted that the legacy of U.S. nuclear testing had contributed to their decision or need to move. Compact migrants cited varied reasons for choosing to migrate to specific locations. For example, representatives of FAS communities in Guam and the CNMI noted the FASs’ closer proximity to those territories than to the U.S. mainland as well as the similarity of Guam’s and the CNMI’s island cultures to those of their home countries. Also, some compact migrants in Arkansas and Oregon cited the lower cost of living and a perception of less discrimination or greater safety there than in Hawaii. Marshallese community members often migrate to Arkansas for jobs in the poultry industry. Consular officials noted that, because of comparatively lower wages and fewer housing options in the FASs, returning to their countries after living in U.S. areas can be difficult for some compact migrants. Some compact migrants said that it is also difficult to find a good job in their home countries without family or political connections. According to an FAS official, some compact migrants retire to their home countries. However, several compact migrants we spoke with said they planned to stay in U.S. areas to be close to medical care or to children and grandchildren born there. The affected jurisdictions of Hawaii, Guam, and the CNMI reported estimated compact impact costs (i.e., costs incurred as a result of increased demands on public services from compact migrants) that totaled $3.2 billion during the period fiscal years 2004 through 2018 and increased over time for Hawaii and Guam. Interior has provided compact impact grants totaling more than $30 million annually to the affected jurisdictions, each of which uses the funds differently. In October 2019, Census discovered an error in the 2013 and 2018 enumerations, which Interior had used to determine the distribution of compact impact grant funds and which resulted in misallocation of these funds for fiscal years 2015 through 2020. In February 2020, Interior officials told us that the department had developed a modified plan for compact impact grants in fiscal years 2021 through 2023 that, according to the officials, is intended to correct the misallocation. Hawaii, Guam, and the CNMI reported a total of $3.2 billion in estimated compact impact costs during the period fiscal years 2004 through 2018, with estimated annual costs increasing over time for Hawaii and Guam and fluctuating for the CNMI. Hawaii reported $1.8 billion in total estimated compact impact costs. Hawaii’s reported annual costs increased from $55 million in fiscal year 2004 to $198 million in fiscal year 2018. Guam reported $1.2 billion in total estimated compact impact costs. Guam’s reported annual costs increased from $33 million in fiscal year 2004 to $147 million in fiscal year 2017. The CNMI reported $116 million in total estimated compact impact costs. The CNMI’s reported annual costs amounted to $10 million in both fiscal year 2004 and fiscal year 2018 but fluctuated over time, ranging from a low of about $3 million in fiscal year 2011 to a high of $12 million in fiscal year 2014. For a summary of the estimated compact impact costs reported by the three affected jurisdictions, see figure 3. For more details of their compact impact reporting, see appendix V. The three affected jurisdictions reported compact impact costs for education, health, public safety, and social services (see table 2). As the table shows, the highest total costs in fiscal year 2017 were for education and health services. In November 2011, we found that Interior’s reporting to Congress on compact impact had been limited, and we identified weaknesses in existing compact impact reporting. We found that some jurisdictions did not accurately define compact migrants, account for federal funding that supplemented local expenditures, or include revenue received from compact migrants. Our November 2011 report recommended that the Secretary of the Interior disseminate guidelines to the affected jurisdictions that adequately addressed concepts essential to producing reliable impact estimates and that the Secretary call for the use of these guidelines in developing compact impact reports. Although Interior developed a draft of compact impact reporting guidelines in 2014, it had not disseminated such guidelines to the affected jurisdictions as of February 2020. In 2019, Interior awarded the Guam government a technical assistance grant for $280,000 to conduct a cost-benefit analysis to determine compact migrants’ economic contribution to the local economy. The effort will reportedly also seek to address weaknesses and methodological concerns related to compact impact costs calculated by Hawaii, Guam, and the CNMI. Guam officials said that the grant application was prepared in response to our prior critique of their compact impact estimation methodology. The grant was awarded to the Guam Bureau of Statistics and Plans, which contracted with University of Guam consultants to carry out the work beginning in October 2019. Guam officials expected this work to result in two reports—one identifying economic contributions by compact migrants (expected September 2021) and another proposing a methodology for determining compact impact costs (expected August 2022). During fiscal years 2004 through 2019, Hawaii, Guam, and the CNMI received a combined total of approximately $509 million in compact impact grant funding. This total includes (1) annual compact impact grant funding allocated from $30 million authorized and appropriated in the amended compacts’ implementing legislation and (2) additional compact impact grant funding allocated from annual appropriations. In fiscal years 2004 through 2019, Interior made annual allocations of the $30 million of compact impact grant funds authorized and appropriated in the amended compacts’ implementing legislation. Interior provided these allocations as compact impact grants to each affected jurisdiction to defray their costs due to the residence of compact migrants. Interior used the four most recent enumerations— conducted in 2003, 2008, 2013, and 2018—as the basis for these annual allocations. Since fiscal year 2012, Interior has provided additional compact impact grant funding to the affected jurisdictions from annual appropriations. This additional funding has ranged from approximately $3 million to $5 million per year since fiscal year 2012. Interior has allocated the additional funding on the basis of the 2013 and 2018 enumerations. Table 3 shows the total amounts that Hawaii, Guam, and the CNMI received as compact impact grant funding in fiscal years 2004 through 2019. Affected jurisdictions use their compact impact grant funding in varying ways and report on their use of the funds to Interior. Hawaii allocates the entirety of its compact impact grant—approximately $13 million annually since fiscal year 2015—to the state’s MedQuest division to defray costs of providing medical services to compact migrants. Guam has used some of its approximately $15 million of compact impact funding each year for new schools constructed through leasebacks (see fig. 4 for photos of several schools built by the Guam government with compact impact funds). The CNMI allocates its approximately $2 million of compact impact funding each year across the education, health care, public safety, and social service sectors. Hawaii, Guam, and CNMI officials have emphasized that compact impact funding does not fully compensate for the expenses associated with compact migration. For stakeholder suggestions related to compact impact funding and other issues, see appendix VII. In October 2019, Census Bureau officials discovered an error in the 2013 and 2018 Census Bureau enumerations that caused inaccurate counts of compact migrants in Hawaii and, according to Interior officials, resulted in misallocation of compact impact funding for Hawaii, Guam, the CNMI, and American Samoa in fiscal years 2015 through 2020. Relative to the proportion of compact migrants in each jurisdiction, allocations to Hawaii were a total of $16.9 million lower than they would have been without the enumeration error while allocations to Guam, the CNMI, and American Samoa were higher than they would have been without the error. Table 4 summarizes the under- and overpayments of compact impact funding to each affected jurisdiction that, according to Interior officials, resulted from the enumeration error. The enumeration error was discovered in late October 2019, near the beginning of fiscal year 2020. As of February 2020, OIA officials had developed a modified planned allocation of compact impact funds for fiscal years 2021 to 2023. Beginning in fiscal year 2021, OIA plans to divide the $30 million of annual compact impact grant funding in fiscal years 2021 through 2023 using corrected base allocations from the updated 2018 enumeration from Census Bureau, according to an Interior preliminary assessment. The base allocations will be adjusted upward for Hawaii and downward for Guam, the CNMI, and American Samoa to correct for the erroneous payments in fiscal years 2015 through 2020. See table 5 for a comparison of the originally planned fiscal year 2020 allocation (based on the erroneous enumeration) and the revised allocation (based on the corrected enumeration) as well as the grant amounts that OIA proposed for fiscal years 2021 through 2023 to correct for the erroneous payments. The governments of some of the U.S. areas we visited identified effects of providing public education and health care services to compact migrants. Compact migration’s effects in U.S. areas we visited also include budgetary contributions from compact migrants’ payment of taxes and fees as well as budgetary costs of other government programs and services to compact migrants. Stakeholders in the U.S. areas additionally discussed the participation of compact migrants in those areas’ workforces and communities in terms of contributions and impacts of compact migration. Children of compact migrants attending U.S. public primary and secondary schools sometimes receive additional or specialized services, such as support for English language learners, according to state and territorial officials. In the U.S. areas we visited, state and territorial departments and school districts have identified and counted compact migrant students by means of one or more criteria, including ethnicity, language, and place of birth. See table 6 for estimated numbers of compact migrant students in the states and territories we visited and the criteria that each state or territory used to count students as compact migrants. Compact migrants are eligible for in-state tuition at some U.S.-based colleges and universities, according to university, nonprofit, and state officials. For example, in Guam, compact migrants attending the University of Guam are eligible for in-state tuition. In Oregon, FAS citizens are eligible for in-state tuition after a 1-year residency period in the state, according to nonprofit officials. In Arkansas, Marshallese citizens are eligible for in-state tuition after a 3-year residency period in the state, according to state tuition policy and officials. States and territories have reported budget and program effects related to health care for compact migrants who are eligible for federal benefits as well as health care for individuals, including compact migrants, who are ineligible for federal benefits and lack private insurance or other means of payment. U.S. area governments sought to enable compact migrants’ access to health care in several ways, including extending access to the federal Children’s Health Insurance Program (CHIP) or Medicaid and leveraging federal health insurance tax credits and other federal funding. According to some U.S. area government officials, some of these programs are provided specifically because compact migrants are ineligible for certain programs at the federal level. Extended Access to Children’s Health Insurance Program or Medicaid The Children’s Health Insurance Program Reauthorization Act of 2009 included an option for states to cover children younger than 21 years and pregnant women in both CHIP and Medicaid who are lawfully residing in the United States—a definition that includes compact migrants—and who are otherwise eligible under the state plan. Therefore, in some U.S. areas, non-U.S. citizen compact migrants who are children or pregnant may access federal health insurance coverage through CHIP or Medicaid. As of February 2020, 38 states and territories and the District of Columbia had extended such coverage to lawfully residing non–U.S. citizen pregnant women or children, including compact migrants, who met all other eligibility requirements (see fig. 5). According to Arkansas officials, their state’s decision to extend this coverage was sought in part to address unmet needs of compact migrants living in Arkansas. Subsidized Coverage in Patient Protection and Affordable Care Act Exchanges Compact migrants are eligible to purchase individual market health insurance plans through health insurance exchanges established under the Patient Protection and Affordable Care Act (PPACA). Individuals purchasing coverage through the exchanges may be eligible, depending on their incomes, to receive financial assistance in the form of premium tax credits to offset the costs of their coverage. Premium tax credits, which are designed to reduce an eligible individual’s premium costs, may be either paid in advance on a monthly basis to an enrollee’s issuer (referred to as advance premium tax credits) or received after the individual files federal income taxes for the prior year. Some state governments have elected to cover the remaining balance of some individuals’ exchange plans, leveraging a combination of advance premium tax credits and state funds to fully cover health insurance premiums on certain exchange plans. For example, Hawaii created the Health Care Premium Assistance Program, a special state program that covers the cost of premiums on eligible plans for qualified residents who do not qualify for Medicaid. While Hawaii’s program was not created specifically in response to compact migration and is not limited to compact migrants, most of its enrollees are compact migrants, according to Hawaii government officials. Since its launch in 2015, the program pays the balance of health insurance premiums not covered by advance premium tax credits for those who would otherwise be qualified for federal Medicaid if not for their citizenship status, including compact migrants. According to state officials, the program covered 3,223 compact migrants residing in Hawaii as of June 2017. Oregon and Washington developed premium assistance programs specific to compact migrants that leverage advance premium tax credits to eliminate health care premium costs. In addition to covering premiums, these programs provide for out-of-pocket health care costs, according to the programs’ websites and state officials. Oregon COFA Premium Assistance Program. The Oregon COFA Premium Assistance Program was launched in 2017, expressly to help compact migrants gain access to health care. In Oregon, participants pay for out-of-pocket costs at the time of service and subsequently apply to the program for reimbursement. Oregon’s program covered 780 compact migrants as of October 2019, according to state officials. The officials estimated that this program leverages $9 of federal funds through advance premium tax credits for every $1 of Oregon state funds contributed. Washington COFA Islander Health Care. The Washington COFA Islander Health Care program was launched in 2019, expressly to help compact migrants gain access to health care, and was based in part on the Oregon program, according to state officials. Washington will also cover dental insurance costs for compact migrants beginning in 2021, according to the program’s website and state officials. Participants in Washington’s program receive a payment card with preloaded funds to use for out-of-pocket costs. The program covered approximately 1,100 compact migrants in 2019, according to state officials. (Fig. 6 shows an example of an advertisement for Washington’s program, presenting information in six languages spoken by compact migrants.) Additional Federal Health Care Funding in U.S. Territories All U.S. territories, including Guam and the CNMI, receive federal funding through Medicaid, which is subject to an annual cap. Section 2005 of the PPACA, as amended, increased the funding caps for the territories for the period beginning on July 1, 2011, and ending on September 30, 2019, and provided a total of $6.3 billion in additional federal funding for health care to the territories. Guam and the CNMI have used some of this funding, in addition to other federal funding for health care, to partially support compact migrants’ health care costs or to alleviate the burden on programs that cover compact migrants. Guam. PPACA Section 2005 funding partly alleviated the financial shortfall of Guam’s Medically Indigent Program, according to a territory official. The Medically Indigent Program pays for health care costs of primarily non-U.S. citizens living in Guam, including compact migrants, who do not have other health insurance. Most compact migrants in Guam qualify for this program after meeting the 6-month residency requirement, according to Guam officials. In fiscal year 2019, compact migrants participating in the program numbered 8,616, according to Guam officials, and made up 73 percent of the program’s total participation. The officials said that the program is also funded through Guam local appropriations and federal Medicaid Undocumented Emergency Services funding. CNMI. Territorial hospital officials said that PPACA Section 2005 funding available in fiscal years 2011 through 2019 partially covered patient care costs in excess of the territory’s annual Medicaid cap, including care for compact migrants. The CNMI Medicaid program uses federal Disaster Relief Assistance funding to reimburse the hospital for emergency services provided to compact migrants, according to CNMI officials. Other Health Care Services Available to Compact Migrants Non-U.S. citizens, including compact migrants, may access health care through the U.S. Department of Health and Human Services Health Resources and Services Administration’s Health Center Program and through state government–supported clinics. The Health Center Program was established in the mid-1960s to help low-income individuals gain access to health care services. Health centers are responsible for delivering affordable, accessible, high-quality, comprehensive primary health care regardless of recipients’ ability to pay, according to Department of Health and Human Services officials. Figure 7 shows the entrance to Kokua Kalihi Valley, a federally qualified health center in Honolulu that estimates one-third of its patient population to be compact migrants, mostly from Micronesia. State clinics provide health services such as screening and treatment of certain infectious diseases to compact migrants, among other state residents. For example, the Arkansas Department of Health established the Dr. Joseph Bates Outreach Clinic to provide public health services to Marshallese in the region. As of September 2019, approximately 95 percent of the clinic’s patients were Marshallese, according to clinic officials. In addition, the University of Arkansas for Medical Sciences Northwest Campus facilitates research and community health programs in the Marshallese community and has established a clinic focused on diabetes. The budgetary effects of compact migration in the U.S. areas we visited include contributions by compact migrants, such as payment of federal and state taxes and fees, and also include several types of government program costs related to compact migration. Budgetary contributions. Compact migrants pay payroll taxes, including income taxes, and contributions to Social Security and Medicare. They also pay fees associated with state or territorial documentation or licensing, including driver’s licenses. In general, reliable data on budgetary contributions of compact migrants are not available, because state and territorial tax filings and related databases do not provide data on citizenship or ethnicity, according to state and territorial officials. However, the Hawaiian government reported that in 2017, compact migrants generated an estimated $36.6 million in state revenue from fees and taxes, such as the individual income tax, general excise taxes, and taxes generated from state government spending. According to University of Guam officials and an FAS community member, the presence of FAS communities may have helped Guam institutions obtain funding, including funding for research. Budgetary costs. State and territorial officials identified budgetary costs related to compact migration. For example, officials cited costs of providing translators or interpreters for government programs and costs associated with compact migrant interactions with police and the justice system. Some states have elected to extend state-level programs for food or cash-based assistance to compact migrants who are ineligible for the federal equivalents. For example, Washington’s Cash Assistance and Food Assistance Programs provide financial support to FAS citizens who are ineligible for the federal Supplemental Nutrition Assistance Program and Temporary Assistance for Needy Families. In Guam, some compact migrants qualify for the federal earned income tax credit, according to officials of Guam’s Department of Revenue and Taxation. The officials noted that because Guam’s tax system mirrors the federal system, any earned income tax credit paid in Guam is an expense to the territorial government. Compact migrants are eligible to work in U.S. areas and have contributed to the workforces of receiving communities, holding jobs in a range of industries. According to stakeholders we interviewed, compact migrants have encountered challenges while participating in the workforce. In the U.S. areas where they reside, compact migrants participate in the local economies in part by serving in the workforce in a variety of fields, including manufacturing, service industries, and professional industries, according to stakeholders we interviewed. See table 7 for examples. The following describes compact migrants’ participation in the areas we visited. Arkansas. Arkansas private sector representatives described Marshallese workers as essential to poultry plant operations, comprising one-quarter to one-third of some plants’ workers. At one such plant, most Marshallese employed are line workers on the floor of the plant, while others work as trainers and translators. Other compact migrants in Arkansas work at an airport; in hotels; in retail; or as caregivers, including in adult day care, according to FAS consular officials and nonprofit representatives. CNMI. CNMI officials and a private sector representative described compact migrants as a valuable resource in supplementing the CNMI’s small labor pool. Officials also noted that without compact migrants, businesses would have to recruit more foreign labor and face more-severe hiring challenges than they do now. Officials and a private sector representative stated that several businesses and franchises were founded by, and employ, compact migrants. Guam. Guam Chamber of Commerce representatives indicated that compact migrant workers would not be easily replaced if they were no longer eligible to work in Guam and that hiring other foreign workers in Guam involves difficult visa processes. Compact migrants tend to hold entry-level and low-skill jobs in Guam and have high turnover rates, according to representatives from one company. Several businesses in Guam were founded by, or cater to, compact migrants, according to private sector representatives. Hawaii. Micronesian officials noted that established communities of compact migrants in Hawaii help other FAS citizens to migrate, network, and find job opportunities. FAS community members in Hawaii identified multiple local businesses that either are owned by compact migrants or employ a large number of compact migrants. Oregon. In Oregon, some compact migrants work as caregivers or in a plant manufacturing reusable plastic containers for food storage and transport, according to an FAS official and community members. Oregon state government officials noted that compact migrants play an important role in working with adults and children with intellectual and developmental disabilities and in other paid caregiver capacities. The Governor of Oregon noted that compact migrants bring a tremendous amount of value to Oregon communities as educators, social workers, caregivers, and as members of the U.S. military. Other jobs or industries in which compact migrants work include warehousing, fast-food restaurants, and airport jobs, according to FAS officials. Washington. Some compact migrants work in caregiving, including at senior care homes; in manufacturing, warehousing, fast-food restaurants, or nonprofits; as artisans; or at airports, according to state and FAS officials and FAS community members. Stakeholders reported that compact migrants have encountered various challenges related to participation in the U.S. workforce. See appendix VII for additional challenges experienced by compact migrant communities. Form I-94. Compact migrants from Micronesia and the Marshall Islands may present an unexpired FAS passport and Form I-94 Arrival/Departure Record (known as Form I-94) to employers to demonstrate their identity and employment authorizations. Before 2013, compact migrants entering the United States received a paper copy of the form to document their legal entry and their ability to legally reside indefinitely in the United States. The DHS transition in 2013 from issuing Forms I-94 on paper to issuing them electronically created challenges for compact migrants, according to FAS community members. According to consulate officials, communities were not adequately notified that DHS would maintain these records in publicly accessible databases for only 5 years. As a result, some compact migrants who entered the United States after mid-2013 did not download their Forms I-94 before they became unavailable and thus did not have a Form I-94 to show to employers, according to stakeholders we interviewed. REAL ID–compliant driver’s licenses. Some employers require employees to have REAL ID–compliant driver’s licenses, according to FAS officials and community members. Before September 2019, DHS required compact migrants and other nonimmigrants applying for a REAL ID–compliant driver’s license to present an unexpired passport with an unexpired visa and Form I-94 or to present an employment authorization document. However, because compact migrants do not receive a visa and are not otherwise required to obtain an employment authorization document, they were unable to obtain the licenses. In September 2019, DHS changed its requirements specifically to allow compact migrants to receive REAL ID–compliant driver’s licenses by presenting an unexpired passport and Form I- 94. Some compact migrants in Guam said that challenges related to REAL ID before the DHS regulation change had negatively affected their employment because some military base jobs required these documents for employment or for base access. In addition, some compact migrants have lost jobs at airports because of difficulty in obtaining REAL ID–compliant identification, according to Marshallese embassy officials. See appendix X for information about legislative and DHS policy changes that affected compact migrants’ ability to access full-term REAL ID–compliant driver’s licenses and identification cards. Commercial driver’s licenses. Various stakeholders discussed difficulties that compact migrants had encountered in obtaining commercial driver’s licenses required by certain jobs and obtaining standard driver’s licenses that are compliant with REAL ID requirements in some states. Marshallese officials said that compact migrants’ inability to obtain or renew commercial driver’s licenses had prevented them from being able to work in related jobs, such as truck driving. Labor abuse and discrimination. In September 2019, the government of Micronesia requested that the Department of State provide assistance to investigate abuse and mistreatment of Micronesian citizens who were recruited to move to the United States to work for a U.S. company in Iowa. In addition, compact migrants in Hawaii, Guam, and Oregon told us that they had faced workplace discrimination or were seen as harming the local economy. For example, compact migrants in Guam said that they had experienced discrimination in hiring and pay and sometimes were made to feel like a burden on the community. Additionally, a March 2019 report by the Hawaii Advisory Committee to the U.S. Commission on Civil Rights concluded that some compact migrants find it difficult to report workplace discrimination because they are concerned about retaliation from employers. The report found, among other things, that compact migrants face discrimination in access to employment and housing and also face widespread negative public perception in Hawaii. Stakeholders expressed some concerns about compact migration with respect to public health and law enforcement interactions. In addition to participating in the workforce, compact migrants participate in social institutions and create diversity and cultural exchange in their receiving communities. State and territorial health department officials and health care providers in the U.S. areas we visited noted concerns about the prevalence of communicable diseases such as tuberculosis and Hansen’s disease in compact migrant communities. Tuberculosis. State and territorial health departments have worked to identify and treat cases of active and latent tuberculosis in compact migrant communities. About 15 to 20 percent of active, communicable tuberculosis cases in Hawaii have occurred in the FAS community, including several cases of antimicrobial drug–resistant variants of tuberculosis, according to Hawaii government officials. In 2019, 23 communicable tuberculosis cases were diagnosed in compact migrants in Hawaii. In Arkansas, public health officials estimated that they had screened 30 percent of the Marshallese population since 2000 and reported 202 active cases and 500 cases of latent tuberculosis infection between 1997 and 2019. Arkansas officials also said that they screened 1,728 Marshallese and reported five cases of active disease and 95 cases of tuberculosis infection in fiscal years 2018 and 2019. In 2017 and 2018, Arkansas officials traveled to the Marshall Islands to conduct screening for active and latent tuberculosis in addition to diabetes and Hansen’s disease. Hansen’s disease. Hansen’s disease affects some members of compact migrant communities, according to health care providers and state government officials. For example, the Hawaii Department of Health has a registry of 281 patients who are on active treatment or monitoring for recurrence of Hansen’s disease or complications from the disease. The department manages 10 to 20 new cases of Hansen’s disease each year. According to Hawaii public health officials, 95 percent of the individuals diagnosed with Hansen’s disease in the state were from the Micronesian or Marshallese communities. From 2003 to 2019, the Arkansas Department of Health reported that 54 individuals, including 42 compact migrants, had been diagnosed with Hansen’s disease. Some stakeholders reported concerns regarding public order and law enforcement interactions with compact migrants in Guam, Hawaii, and Washington. Guam. Guam law enforcement agencies report on crimes committed by, or attributed to, FAS groups in each location. Guam private sector representatives we interviewed expressed a belief that social tension with the FAS communities was driven in part by some compact migrants’ public drunkenness or violence. In addition, language barriers can hinder compact migrants’ social integration into receiving communities, according to Guam law enforcement officials. Hawaii. Common offenses for which compact migrants are cited or arrested in Hawaii include quality-of-life or social-order offenses, such as trespassing, disorderly conduct, drinking in public or driving under the influence of alcohol, assault, or harassment, according to state officials. These interactions with the public or with law enforcement officials may contribute to a strained relationship between compact migrants and receiving communities. Hawaii officials estimated that 20 to 25 percent of the population using the state’s homeless services self-identify as part of the FAS community. Compact migrants may sleep in public parks, which can lead to legal charges. A lack of affordable housing may be a cause for homelessness among FAS communities. Washington. Marshallese embassy officials cited sporadic problems with gang activity and drug use among some younger Marshallese community members, particularly those living in Washington. These officials suggested that some migrant children who feel bullied or pressured may band together, resulting in a negative or gang-like situation. In some U.S. areas we visited, stakeholders we interviewed said that compact migrants seek to contribute to, or engage with, their surrounding U.S. communities through volunteer work, including church activities, environmental work, and other efforts. For example, FAS communities described participating in environmental cleanup efforts, including efforts to control invasive species and leveraging their agricultural knowledge to help Hawaiian farmers grow a more resilient variety of taro. Several community representatives in multiple states noted that some compact migrants spend a significant amount of time supporting their fellow community members as translators or interpreters or volunteering to help others navigate complex systems in U.S. areas. FAS citizens also serve in the U.S. military. The FAS countries have a high rate of military service, according to FAS officials and State documentation. Stakeholders in some U.S. areas we visited described compact migrant populations as contributing to the diversity of receiving communities and educational institutions. For example, University of Guam officials said that FAS student association groups sponsor cultural events and activities that help to define the character of the university. The officials also noted that FAS students contribute to research portfolios and bring FAS government and community perspectives to classroom discussions. The officials observed that the presence of compact migrants increases the university community’s diversity and its cultural awareness and competency. In Arkansas, Marshallese community members said that they had helped to teach local U.S. residents about Marshallese culture and history not otherwise taught in U.S. schools. Marshallese community members in Arkansas also expressed a belief that the community brought a greater emphasis on family and respect for elders to the region. We provided a draft of this report for review and comment to the Departments of Agriculture, Commerce, Health and Human Services, Homeland Security, Housing and Urban Development, the Interior, State, and Transportation; the Social Security Administration; the Governors of Hawaii, Guam, the CNMI, Arkansas, Oregon, and Washington; and the Ambassadors of Micronesia, the Marshall Islands, and Palau to the United States. The Departments of Agriculture, Health and Human Services, Homeland Security, and State and the Social Security Administration provided technical comments, which we incorporated as appropriate. The Departments of Commerce, Housing and Urban Development, the Interior, and Transportation did not provide comments. U.S. areas and the FAS Ambassadors provided written comments that we have summarized below and reproduced in appendixes XI through XIX, and responded to their comments, where appropriate, at the end of those appendixes. Hawaii. The government of Hawaii commented that the health and economic impacts of the coronavirus pandemic, in addition to Hawaii’s high cost of living and public charge concerns, affect the state’s compact migrant communities in particular. The government also observed that racial disparities and other determinants of health and well-being are exacerbated for compact migrants. Noting that compact migrants lack access to Medicaid and the Supplemental Nutrition Assistance Program, the government urged that compact migrants’ access to health care and food nutrition programs be treated as a federal priority. Guam. The government of Guam advocated, in the context of the coronavirus pandemic, for restoring debt relief provisions associated with compact migration to offset unreimbursed compact expenses. The government noted that from the time the compacts went into effect until 1996, FAS citizens maintained access to federal health coverage. The government also commented that in 2017, Interior’s Office of Insular Affairs reported to Congress that restoring this eligibility would be in line with Congress’ intent to never cause adverse consequences to the territories and Hawaii. In addition, the government observed that moving compact migrants from Guam’s locally funded Medically Indigent Program to Medicaid would help Guam provide government services to all residents who need them. The government of Guam noted that the ongoing absence of an agreed definition for compact migrant for the purposes of data collection creates confusion. CNMI. The government of the CNMI commented on the importance of compact migrants’ contributions to the territory’s workforce needs and noted that they enrich the cultural makeup of the CNMI. Separately, the government stated that the response to the enumeration error discovered by the U.S. Census Bureau that led to a misallocation of compact impact funds has penalized the territories. According to Interior’s modified plan, future allocations to the CNMI (in addition to Guam and American Samoa) would be adjusted downward to account for past overpayment. The CNMI commented that reducing the future amounts of compact impact funds because of an error of the federal government does not recognize the present needs of the CNMI. The CNMI government also noted that the territories receive less data collection support from the American Community Survey, the U.S. Bureau of Economic Analysis, and the U.S. Bureau of Labor Statistics than other U.S. areas receive. Arkansas. The government of Arkansas commented that it considered the Census Bureau data in our report to underestimate the compact migrant population in Arkansas, and it cited several higher estimates. The government noted that the state does not receive compact funding, despite its high population of Marshallese, because it is not an affected jurisdiction as defined in the Compacts of Free Association Amendments Act of 2003. The government projected that approximately 12,000 compact migrants reside in Arkansas and estimated its annual costs related to compact migration at about $72 million. We believe that the Census Bureau data are sufficiently reliable for our purposes of estimating the number of compact migrants in U.S. areas. However, our report includes a discussion of stakeholder concerns that the compact migrant population in Arkansas may have been undercounted. We reported that the Census Bureau had estimated the compact migrant population in Arkansas during the period 2013 to 2017 at 5,895 on the basis of the definition of “compact migrants” used for its enumerations—citizens of Micronesia, the Marshall Islands, and Palau who entered the United States after 1986 (from Micronesia and the Marshall Islands) or 1994 (from Palau) and their U.S.-born children (biological, adopted, and step-) and grandchildren younger than 18 years. Oregon. The government of Oregon advocated for more reporting on the effects that U.S. military access to, and U.S. testing of 67 nuclear weapons in, the Marshall Islands has had on compact migration, citing the devastating impact of nuclear fallout on inhabitants’ health and the environment. The government of Oregon also cited a need to report on compact migrants’ positive contributions to receiving areas. Our report provides qualitative descriptions of compact migrants’ contributions, including budgetary, workforce, and social contributions, and also provides high-level data on estimated mean and median incomes among compact migrants. We have incorporated additional statements by the government of Oregon about compact migrant contributions in our report. The government noted that it has taken steps at the state level to provide health care access to compact migrant populations while also urging Congress to restore this populations’ access to federal programs such as Medicaid and Temporary Assistance for Needy Families. Further, the government called on Interior to expand the definition of “affected jurisdiction” and appropriate grant funds equitably. As we note in our report, this definition and the associated grant funding were established by Congress in the amended compacts’ implementing legislation. Washington. The government of Washington commented that our report did not provide a detailed history of U.S. military nuclear testing in the FASs and subsequent impacts on them and their citizens. The government noted that such information is necessary to explain FAS citizens’ current challenges and why additional resources are required to meet their needs. Further, the government commented that our report omits the personal narratives that are critical to a holistic account of the FAS experience in the United States, including the struggles many compact migrants face. Our report incorporates information that we obtained through our interviews with members of compact migrant communities, including those in Washington, such as reasons for migration, workforce and other challenges they faced, and stakeholder suggestions for improving experiences or outcomes of compact migration (see app. VII). The government of Washington stated that it hoped our report would prompt the federal government to make additional resources available to U.S. areas with sizeable compact migrant populations, and it called for inclusion of Washington among affected jurisdictions receiving compact impact grant funding. Micronesia. The Embassy of the Federated States of Micronesia emphasized the importance of quantifying not only costs but also economic benefits of compact migration, including job creation, taxes paid, and community contributions. The embassy also called for guidelines and enumeration methods that better capture actual costs and revenue. The embassy noted the relationship between FAS citizens’ ineligibility for federal programs such as Medicaid and the costs borne by local governments and communities in the absence of these federal programs. According to the embassy, the continuing challenge of Micronesian citizens’ ineligibility for Medicaid since 1996, compounded by the effects of relevant social determinants of health, make their successful integration in U.S. areas more difficult. Noting that these circumstances have a direct effect on Micronesian migrants’ ability to contribute positively in receiving areas and become less reliant on public assistance programs, the embassy expressed support for the restoration of FAS citizens’ eligibility for Medicaid and for expanded veterans’ health care in Micronesia. The embassy commented that compact impact grant funding is a domestic issue and that discussions related to this issue should not diminish the priority of ongoing U.S. assistance to Micronesia under the compact. The embassy also raised concerns about challenges facing compact migrants, including the challenges described in our report. Marshall Islands. The Embassy of the Republic of the Marshall Islands described the migration rights provided in the compact as fundamental and essential to its country’s relationship with the United States. Additionally, the embassy observed that restoring Medicaid eligibility for its citizens living and working in the United States would greatly benefit its citizens and substantially reduce impact costs to certain areas. The embassy noted that, although Marshall Islands citizens living in the United States are eligible to purchase individual market health insurance plans through exchanges established under the Patient Protection and Affordable Care Act, many who are employed lack access to affordable health care because of the limited insurance benefits offered by most service industries or the high cost of covering family members. Furthermore, the embassy called for an objective accounting of revenue received from compact migrants and depiction of their contributions to, for example, the health and food security of the United States through employment in the food processing industry and other essential work. Last, the embassy commented that the addition of Marshallese workers to the 2020 census effort may remedy the potential undercounting of its citizens in the previous census. Palau. The Embassy of the Republic of Palau observed that it would be helpful to know the number of compact migrants from each FAS country who are able to access the federal programs for which they are eligible. This question was outside the scope of our review. Further, the embassy commented that it would like the U.S. federal government to inform and educate state departments of motor vehicles regarding the special status of FAS citizens in the United States, and it highlighted the difficulties that compact migrants historically have faced in obtaining REAL ID–compliant identification. We are sending copies of this report to the appropriate congressional committees and to the Departments of Agriculture, Commerce, Health and Human Services, Homeland Security, Housing and Urban Development, the Interior, State, and Transportation; the Social Security Administration; and the Governors of Arkansas, the CNMI, Guam, Hawaii, Oregon, and Washington; and the Ambassadors of Micronesia, the Marshall Islands, and Palau. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix XX. We were asked to review topics related to migration to U.S. areas from the freely associated states (FAS)—the Federated States of Micronesia (Micronesia), the Republic of the Marshall Islands (Marshall Islands), and the Republic of Palau (Palau)—under those countries’ compacts of free association with the United States. This report (1) presents estimates of compact migrant populations and describes recent trends in compact migration; (2) summarizes the reported costs related to compact migration (compact impact costs) in three affected jurisdictions—Hawaii, Guam, and the Commonwealth of the Northern Mariana Islands (CNMI); and (3) describes effects of compact migration on governments, workforces, and societies in these and other U.S. areas. To present estimates of compact migrant populations in U.S. areas and describe recent trends in compact migration, we obtained special tabulations of data from the U.S. Census Bureau’s 2013-2017 American Community Survey (ACS) for the 50 U.S. states, the District of Columbia, and Puerto Rico. For Guam and the CNMI—U.S. territories that are not covered by the ACS—we used the Census Bureau’s revised 2018 enumeration of compact migrants in these areas. The special tabulations of ACS data and the Census Bureau’s 2013 and 2018 enumeration reports defined compact migrants as individuals residing in U.S. areas who were born in the FASs and entered the United States after 1986 (for Micronesia and the Marshall Islands) or 1994 (for Palau) and their U.S.-born children (biological, adopted, step-) and grandchildren younger than 18 years. We calculated percentage changes in states and territories that had more than 1,000 estimated compact migrants (or were designated as affected jurisdictions by the 2003 compacts’ implementing legislation) by comparing 2005-2009 ACS data and the 2008 enumeration with 2013-2017 ACS data and the revised 2018 enumeration. To identify and describe changes in Census Bureau methods and definitions for enumerating compact migrants over time (see app. VI), we reviewed the definitions of “compact migrant” in the bureau’s enumeration reports for tabulations before and including 2018. We also interviewed Census Bureau and Department of the Interior (Interior) officials. For example, we asked when and where grandchildren were counted among compact migrants younger than 18 years. To estimate net arrivals to U.S. areas by travelers with FAS passports (see app. III), we analyzed data from the Department of Homeland Security’s (DHS) Customs and Border Protection’s (CBP) Arrival and Departure Information System (ADIS). According to CBP officials, ADIS consolidates data from several DHS systems to create unique, person- centric travel records for all travelers regardless of citizenship. We calculated monthly FAS net arrivals to U.S. areas from 2017 through 2019 by using ADIS data that DHS provided, showing numbers of individuals with FAS-issued passports entering and exiting U.S. ports of entry each month during the period. To assess the reliability of ADIS data, we spoke with DHS officials to identify potential data reliability concerns and other limitations of ADIS. Officials said that any compact migrant who enters on an FAS passport and holds U.S. citizenship will be masked or not appear in the ADIS system. Officials also said that compact migrants who become U.S. citizens after arrival or are later discovered to be U.S. citizens are removed from the data; CBP officials believed these numbers to be small. We also conducted statistical checks for consistency and completeness of the ADIS data, including validating the ADIS data against publicly available passenger data from the U.S. Department of Transportation Air Carrier Statistics (TranStats) T-100 database for 2015 to 2019 (data for 2019 were partial). We used flight segment data from the T-100 database containing total passenger counts reported by both U.S. and foreign air carriers for flights that compact migrants take to U.S. areas. We found that data from ADIS and the T-100 database were positively correlated for 2015, 2017, 2018, and 2019 but were not correlated for 2016. According to CBP officials, ADIS was significantly changed in 2016 and may contain duplicate entries for that year. As a result, we determined that ADIS data for 2017 and later were sufficiently reliable for our intended use. To quantify costs related to compact migration that were reported by the affected jurisdictions included in our review—Hawaii, Guam, and the CNMI—we reviewed documents that they had published or provided to Interior, such as compact impact reports submitted by Hawaii and Guam and grant documents submitted by the CNMI. We used the most recent data available for 2004 through 2018. To identify the amount of funding distributed by Interior as compact impact grants to the affected jurisdictions, we interviewed Interior officials and reviewed relevant documentation. To identify and describe effects of compact migration on governments, workforces, and societies of receiving U.S. areas, we reviewed relevant documentation and conducted interviews with stakeholders in six U.S. areas that we visited. Documentation that we reviewed included program information and counts of compact migrants using state-level benefits programs, treated by state or local health clinics, enrolled in public schools or higher-education institutions, or using interpreters. Because various sources may define compact migrants by ethnicity, place of birth, language of origin, or other metrics, we noted the definition used for each count in this report. To identify the eligibility of compact migrants for selected federal programs, we reviewed relevant statutes and regulations and held discussions with officials from the U.S. agencies that oversee the programs. We selected the programs included in table 1 on the basis of those we included in a prior report, and we added other selected programs that we learned about in the course of interviews for our current report. We traveled to, and interviewed stakeholders in, six U.S. states and territories where compact migrants live, including three of the U.S. areas designated in the 2003 amended compacts’ implementing legislation as affected jurisdictions—Hawaii, Guam, and the CNMI— and three mainland states—Arkansas, Oregon, and Washington. We selected these areas on the basis of previously reported compact migrant population distributions in U.S. areas and of the locations of consulates or honorary consuls established by Micronesia, the Marshall Islands, and Palau. Stakeholders we interviewed included officials from nine federal agencies; state and territorial government officials in areas we visited; private sector and nonprofit organization representatives such as chambers of commerce, employers of compact migrants, and nonprofit service providers; officials from the FAS embassies and consulates or honorary consuls in areas we visited; and compact migrants living in areas we visited (see table 8). FAS embassy officials in Washington, D.C., connected us with local community members who helped us promote and organize the local community meetings in areas we visited. Participants whom we interviewed in the meetings do not represent a generalizable sample of compact migrants, and the challenges they discussed are not comprehensive (see app. VII for a discussion of challenges faced by compact migrants, according to stakeholders we interviewed). To describe academic studies of workforce and fiscal impacts of new migrants (see app. IX), we conducted a search, using keywords relevant to the economic impact of migration, in American and European economics academic journals published during the period 2015 to 2019. We reviewed a subset of these articles that we deemed most relevant to the context of compact migration, including articles that related to migration of lower-skilled workers and that included empirical analysis of the impact of this migration on various economic aspects. We also reviewed survey articles reviewing the conclusions of prior relevant publications. We conducted this performance audit from March 2019 through June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 9 presents Census Bureau data for U.S. states and territories that had estimated compact migrant populations of more than 1,000 in 2013 through 2018 and shows percentage changes in these populations from 2005-2009 to 2013-2018. Data for U.S. areas not covered by the American Community Survey, including Guam, the Commonwealth of the Northern Mariana Islands (CNMI), and American Samoa, are from compact migration enumerations that the Census Bureau performed on behalf of the Department of the Interior. According to 5-year data from the Census Bureau’s 2013-2017 American Community Survey, 72,965 compact migrants resided in the 50 U.S. states, the District of Columbia, and Puerto Rico. (The American Community Survey does not cover American Samoa, the CNMI, Guam, or the U.S. Virgin Islands.) For estimates of the number of compact migrants in each of the 50 U.S. states, the District of Columbia, and Puerto Rico, see table 10. The American Community Survey captures, among other things, respondents’ place of birth (by country) and state of residence. Table 11 provides Census Bureau estimates, using 2013-2017 American Community Survey data, of the numbers of compact migrants born in the freely associated states—the Federated States of Micronesia, the Republic of the Marshall Islands, and the Republic of Palau—and residing in each of the 50 states, the District of Columbia, and Puerto Rico. From 2017 through 2019, an average of about 366 more migrants from the Federated States of Micronesia (Micronesia), Republic of the Marshall Islands (Marshall Islands), and Republic of Palau (Palau) arrived in U.S. areas per month (4,390 per year) than departed, according to the Department of Homeland Security’s Customs and Border Protection’s Arrival and Departure Information System (ADIS). As figure 8 shows, this trend was driven by migrants from Micronesia and the Marshall Islands (3,343 and 1,487 per year on average, respectively). Each year during this period, an average of about 440 more Palauan citizens departed from the United States than arrived. The Census Bureau’s American Community Survey is an ongoing survey that provides information on a yearly basis, including employment status, educational attainment, veteran status, and age of survey respondents, among other topics. The survey covers the 50 U.S. states, the District of Columbia, and Puerto Rico. (The survey does not cover American Samoa, the Commonwealth of the Northern Mariana Islands, Guam, or the U.S. Virgin Islands.) Some data for compact migrant populations are available through the American Community Survey. See table 12 for demographic information about compact migrant populations in the 50 U.S. states, the District of Columbia, and Puerto Rico in 2013 to 2017. See table 13 for demographic information about the compact migrant population in Hawaii only. Since 1986, Hawaii, Guam, and the Commonwealth of the Northern Mariana Islands (CNMI) have submitted to the Department of the Interior (Interior) intermittent compact impact reports and other documents that include descriptions of, and estimated costs for, education, health, public safety, and social services that local government agencies provided to compact migrants. Hawaii and Guam have submitted compact impact reports, which are available on Interior’s Office of Insular Affairs’ website. The CNMI has not submitted a compact impact report since fiscal year 2003 but reports compact impact costs to Interior in the CNMI’s annual plan for the use of compact impact grants. Table 14 shows the estimated costs that these affected jurisdictions reported to Interior or provided to us for 1986 through 2018. The Census Bureau, working under an interagency agreement with the Department of the Interior (Interior), has conducted six sets of enumerations of compact migrants in affected jurisdictions for the purpose of allocating compact impact grant funding and has performed the enumerations every 5 years. Enumeration methods and definitions have changed over time. During the course of our work, an error was discovered that affected the accuracy of the 2013 and 2018 enumerations and also affected Interior’s allocations of compact impact grants for several fiscal years. Census Bureau methods of gathering new data or analyzing existing data for compact migrant enumerations on behalf of Interior have changed over time. In 1993,1998, and 2003, the bureau used the “snowball” technique; in 2008, 2013, and 2018, the bureau employed a two-pronged approach. For enumerations in 1993, 1998, and 2003, the Census Bureau employed a survey method known as snowball sampling to count compact migrants in Hawaii, Guam, and the Commonwealth of the Northern Mariana Islands (CNMI). Because the surveys relied on referrals by respondents to identify new respondents, they were likely to undercount compact migrants who were not referred. For the enumerations in 2008, 2013, and 2018, the Census Bureau used a combination of existing and new survey data to count or estimate the numbers of compact migrants in Hawaii, Guam, and the CNMI. However, for the enumerations in Hawaii, the bureau used a different approach than it used for the enumerations in Guam and the CNMI. Also, for the enumerations in Guam and the CNMI, the bureau used a different approach in 2013 than it used in 2008 and 2018. To estimate the number of compact migrants in Hawaii, the Census Bureau used existing American Community Survey data. To estimate the numbers of compact migrants in Guam and the CNMI, the bureau used existing decennial census data in 2013 and gathered new survey data in 2008 and 2018. See table 15 for a summary of the enumeration methods that the Census Bureau has used over time. The definition of “compact migrant” that the Census Bureau used for the enumerations has changed over time. Each enumeration has counted as a compact migrant any individual, of any age, who was born in the Federated States of Micronesia (Micronesia), the Republic of the Marshall Islands (Marshall Islands), or the Republic of Palau (Palau) and who entered the United States after the effective date of their country’s compact. However, the enumerations in 2003, in 2008, and in 2013 and 2018 used various criteria for counting U.S.-born (U.S. citizen) individuals as children of compact migrants and therefore as compact migrants. 2003 enumeration. The definition of “compact migrant” in the 2003 amended compacts’ implementing legislation indicates that the children of compact migrants were to be considered compact migrants until 18 years of age. Interior interpreted the legislation’s definition of “compact migrant” as including all children younger than 18 years who were born to a compact migrant or migrants in the United States, thus including some U.S. citizens. 2008 enumeration. For the 2008 enumeration, the Census Bureau, on behalf of Interior, counted as compact migrants all children (biological, adopted, and step-) younger than 18 years who were born in the United States to a compact migrant head of household or to his or her spouse, were adopted by a compact migrant head of household or by his or her spouse, or were stepchildren of a compact migrant head of household or of his or her spouse. 2013 and 2018 enumerations. Starting with the 2013 enumeration, the Census Bureau also began counting as compact migrants all children (biological, adopted, and step-) younger than 18 years who were born in the United States to a compact migrant or to his or her spouse, regardless of whether they were the children of the head of household or of his or her spouse, and all grandchildren of a compact migrant who were younger than 18 years, regardless of whether they were the grandchildren of the head of household or of his or her spouse. Also starting with the 2013 enumeration, the Census Bureau introduced a requirement that to be counted as a compact migrant, a child or grandchild of a compact migrant must never have been married. For a summary of “compact migrant” definitions used for the enumerations over time, see table 16. The six sets of enumerations of compact migrants that the Census Bureau conducted on behalf of Interior in affected jurisdictions from 1993 through 2018 showed these populations growing in Hawaii and Guam and fluctuating in the CNMI. During our work with the Census Bureau to obtain American Community Survey data related to compact migrant populations, bureau officials discovered a programming error in the 2013 and 2018 enumerations of compact migrants that had resulted in an underestimate of certain compact migrants in Hawaii. The bureau revised these estimates in October 2019 to correct for the error in Hawaii. Figure 9 shows the Census Bureau’s revised estimates of compact migrants in the affected jurisdictions as of October 2019. In February 2020, Interior requested that the Census Bureau further revise its estimates for 2013 and 2018 to no longer count grandchildren. As of March 2020, the results of this revision were not yet available. In the U.S. areas we visited, stakeholders from state and territorial governments, private sector and nonprofit organizations, and freely associated state (FAS) consulates and communities made suggestions for improving experiences or outcomes of compact migration for both the receiving areas and the migrants themselves. Stakeholders recommended that some actions be taken in both the United States after compact migrants’ arrival and in the FASs before the migrants’ departure. Provide more information and education about the compacts. Several stakeholders said that U.S. agencies should better understand the compacts and coordinate their related work. These stakeholders, including members of compact migrant communities, noted that U.S. government officials in some cases have seemed uncertain or unaware that compact migrants are able to live and work in U.S. areas without a visa or other documentation and have asked them to present immigration documents they do not possess or are not required to obtain. An FAS official and community members noted a need for more education of employers and state government officials regarding the migration terms of the compacts and the migration status of FAS citizens in the United States. Restore Medicaid eligibility and expand benefits access. State government officials and health care providers advocated restoring Medicaid access to FAS populations. An FAS Consul General advocated restoring Medicaid eligibility to its pre-1996 status for compact migrants. FAS community members suggested extending Supplemental Nutrition Assistance Program benefits and expanding federal student loan access to compact migrants. Provide more information and guidelines about federal programs and policies. State government officials suggested that changes to federal government policies should include specific information about the applicability of the changes to FAS citizens. Health care providers suggested that the federal government should share more data about compact migration and noted a need for federally established guidelines to support accurate, rather than exaggerated, cost reporting. The providers noted that compact impact estimating was chaotic and had a negative effect on the community. FAS community members expressed interest in federally provided educational sessions and clear eligibility criteria for federal benefits. Simplify Form I-94 access for compact migrants. FAS consular officials and community members said that compact migrants entering the United States should receive information about the importance of their Form I-94 Arrival/Departure Record (Form I-94) and how to retrieve it online. Because compact migrants have had difficulty in accessing these forms, and given the cost of replacing them, FAS community members requested that federal agencies be enabled to retrieve migrants’ Forms I-94 for them. FAS consular officials recommended that compact migrants’ Forms I-94 be made accessible on the Customs and Border Protection website indefinitely, not only for the current 5-year period, since compact migrants’ forms do not expire. Provide more and broader funding to U.S. states and territories. State government officials, nonprofit representatives, and FAS community members said that more federal funding and resources were needed to accommodate the compact migrant population or to support the receiving states and territories. State government officials also said that the federal government should increase compact impact funds to a “reasonable amount,” even if the full costs cannot be covered. They noted that the compacts represent a federal obligation and expressed a belief that the federal government should take care of compact migrants. According to some health care providers, the United States’ treatment of the compact migrant population in U.S. areas could affect the FASs’ compact negotiations with the U.S. government. State government officials also suggested that allowing compact migrants access to more federal benefits would help alleviate compact impact on states and territories. Clarify immigration provisions under the compacts of free association. FAS community members in some locations we visited expressed a need for clarification about the status of migration provisions of the compacts. Specifically, they expressed concern that they might have to leave the United States in 2023. For example, in one FAS community we visited, community members registered confusion about whether provisions of the compacts (including migration provisions) are scheduled to end in 2023 and whether FAS citizens in U.S. areas can become U.S. citizens. One community member expressed concern that compact migrants would be “chased” out of U.S. areas after 2023 and that “all of their rights” under the compacts would be revoked. FAS community members also sought clarification about the implementation of the DHS rule for considering public charge while determining admissibility to U.S. areas. According to community members and other stakeholders, the rule has caused uncertainty in compact migrant communities, which may result in some compact migrants’ not enrolling in, or unenrolling from, public benefits programs. FAS community members said that they are uncertain whether and how the rule change will apply to them and whether enrolling in public benefits or enrolling eligible children will make them ineligible to reenter the United States. FAS consular officials and community members also suggested revising certain immigration provisions—for example, changing compact migrants’ nonimmigrant status to allow them access to a wider range of jobs, including law enforcement and military officer positions. Expand health care access and clinics in U.S. areas. State government officials said they believed that more health education and outreach to FAS communities were needed. A nonprofit representative noted that FAS communities lack vision care and that the extension of postpartum care to FAS communities would improve maternal and child health. FAS community members suggested the creation of a Pacific Islander–specific health clinic in the Pacific Northwest, with translators on staff and on-site enrollment for health insurance. Representatives of a nongovernmental organization in Hawaii that is led and staffed by compact migrants noted that a series of changes in compact migrants’ eligibility for the Hawaii state health care program, Med-QUEST, had caused confusion about compact migrants’ eligibility for public health care benefits. Address preventative care, dialysis needs, and communicable diseases in the FASs. State and territorial government officials and health care providers said that greater access to in-country care, including more resources for primary care, was needed in the FASs. They recommended making more preventative treatment available in the FASs, including diabetes prevention, and establishing clinics in the FASs to potentially reduce the number of individuals moving to the United States for health care. Health care providers suggested that the Department of the Interior (Interior) should produce or fund a study on dialysis in the FASs, including an analysis of whether high-quality dialysis services in the FASs would decrease migration solely for access to dialysis. Territorial government officials suggested that compact migrants should receive health screenings before departing for the United States to identify any serious conditions or communicable diseases. Some health care providers and state government officials proposed that the U.S. federal government focus on reducing or eliminating the transmission of tuberculosis in the FASs. Offer predeparture education to compact migrants in the FASs. State government officials and nonprofit representatives suggested that videos be aired on television in the FASs to support predeparture education, to explain differences they would find in the United States, and to reduce culture shock after arrival. Some state government officials and health care providers suggested that FAS citizens be encouraged to gather documentation, such as immunization and medical records, school records, and anything necessary to obtain a U.S. driver’s license, before departing for the United States. State government officials also suggested that lists of community-based organizations, by U.S. state or territory and city, be provided to FAS citizens before their departure. Offer orientation and information to compact migrants arriving in the United States. State government officials said that U.S. areas should offer and fund location-specific orientations for FAS citizens after arrival. The officials suggested that these orientations should cover how health care eligibility works, what resources are available to compact migrants, and how they can contact interpreters. State government officials also said that proactive education about U.S. laws could help compact migrants avoid behavior or circumstances that might cause them to run afoul of the law, given cultural differences and misunderstandings. Health care providers noted that compact migrants could be given more information to encourage better nutritional choices and more exercise. Expand and professionalize translation and interpretation resources. Compact migrants who are not fluent in English may experience challenges accessing or navigating health care, the judicial system, and educational institutions, according to state government officials, FAS consulate officials, private sector and nonprofit organization representatives, and compact migrant communities. State government officials reported frequent difficulty in finding interpreters and translators for the multiple languages spoken by compact migrants. State government officials recommended that grants be made available to help pay for interpreters until more FAS community members graduate from college and become qualified. The officials also said that interpreters should be encouraged to develop greater proficiency in fields such as law and medicine so that they can serve in multiple capacities. In addition, the officials identified a need for more in-person interpreters in hospitals and medical facilities. State government officials noted that FAS communities speak many different languages, and they acknowledged the need for a culturally-specific approach for each group. They said that, in addition to translating content, interpreters should fully explain the context of programs to ensure compact migrants’ understanding. FAS community members proposed the creation of a group of paid, full-time interpreters and a language certification requirement to guarantee the availability and quality of language services. Create “one-stop shops” with information and resources for compact migrants. State government officials and health care providers identified a need for one-stop shops—centers that serve compact migrant populations—in areas that do not currently have them. According to stakeholders in a U.S. state without such a center, a one-stop shop could reduce duplication and increase coordination among the many groups that serve the FAS community. Other stakeholders suggested that each state government establish a single point of contact for compact migrants. FAS community members and nonprofit representatives identified a need for a cultural center or other physical space that could be used to hold events and provide centralized communication and resources for the FAS community in the Pacific Northwest, in particular. Emphasize community-based approaches to supporting compact migrants. State government officials noted the importance of community- based approaches to supporting compact migrants. For example, stakeholders recommended hiring community health workers from the FAS population to engage with their communities in U.S. areas. According to the officials, community health workers, as known and trusted entities, are better sources of information for FAS communities than any government agency. The officials also acknowledged the importance of engaging with FAS community leaders (including embassy or consular officials and church leaders) in U.S. areas to successfully connect with FAS community members. Provide compact migrant–dedicated housing. State government officials, FAS consulate officials, and nonprofit organization representatives discussed discrimination that compact migrants experienced in housing. For example, stakeholders in some areas we visited described landlords who failed to maintain or repair housing leased to compact migrants, who targeted compact migrants for evictions, or who avoided renting units to compact migrants. Officials in one state suggested that FAS communities need access to dedicated housing options that align with their community traditions and cultural norms, such as units that can accommodate large or multiple families. Nonprofit organizations provide compact migrants with a range of assistance, such as assistance with housing or rent, food, documentation and legal matters, and enrollment in health insurance. Some organizations, such as “one-stop shops” (i.e., centers serving compact migrant populations), serve only compact migrants, while other organizations serve compact migrants among other members of the receiving community. Additionally, some companies that employ compact migrants offer programs intended to help them adjust to life in the United States. The information presented in this appendix is based mainly on documentation provided by the organizations and interviews with their representatives. Several nonprofit organizations in U.S. areas that we visited target their services to compact migrants. Two of these organizations—one-stop shops in Hawaii and Guam—aim to support the compact migrant communities by connecting the migrants to existing resources and, in some cases, creating new programs and services to support freely associated state (FAS) communities, according to nonprofit and government officials and documentation. These one-stop shops receive funding from the U.S. Department of the Interior (Interior) as well as other governmental and nongovernmental sources. In Guam, the Micronesian Resource Center One-Stop Shop was developed with input from various communities in Guam and government agencies and launched in October 2015, according to one-stop shop officials and Interior documentation. The one-stop shop has received an Interior grant each year starting in fiscal year 2016. The amount of the grant has steadily increased, rising from $210,000 in fiscal year 2016 to $217,000 in fiscal year 2017, $250,000 in fiscal year 2018, and $267,000 in fiscal year 2019, according to Interior documentation and officials. The one-stop shop employs both case workers and cultural mediators and uses a mobile van to bring services directly to FAS communities, according to one-stop shop officials. These services include outreach to communities, including youths; workshops for parenting and driving; and assistance with lost or replacement documentation. For example, when conflict escalated among compact migrants living in a Guam apartment complex, Interior and Guam officials noted that the one-stop shop worked with police to facilitate meetings and participation in neighborhood watch programs. The one-stop shop has hosted “Welcome to Guam” orientations to educate compact migrants about finding housing, setting up utilities, and opening a bank account in Guam; employees’ rights; medical insurance; deportable offenses; and the danger of human trafficking, according to one-stop shop officials. In Hawaii, the one-stop shop We Are Oceania was established with Interior funding in 2015. The organization provides case management, helping compact migrants to find jobs, address housing or legal issues, and enroll in health insurance through Hawaii’s Premium Assistance Program, according to one-stop shop officials and documentation. We Are Oceania has also provided cultural consultations and trainings to Hawaii public school teachers and service providers to educate them about cultural differences and potential challenges that compact migrants may face, according to nonprofit representatives and documentation. The officials also said that the organization hosts a youth summit and helped open a newcomer welcome center at a middle school. Figure 10 shows photos of the We Are Oceania facility, including desks where compact migrants can apply for health insurance and other services. Other nonprofit organizations were also founded specifically to assist the compact migrant community in navigating various U.S. systems, such as education and health care, and obtaining documentation such as driver’s licenses or Forms I-94 Arrival/Departure Records. The Arkansas Coalition of Marshallese in Springdale, Arkansas, according to representatives of the organization, helps local compact migrants with tasks such as retrieving new Forms I-94 from Customs and Border Protection; translating state driver’s license applications into Marshallese; providing education about diabetes prevention and management; and enrolling compact migrant children in ARKids, the state’s public health insurance program that extends federal health insurance coverage for children younger than 19 years. In 2018, the Micronesian Islander Community Organization in Oregon announced a study among local compact migrants to identify barriers that they faced in the region, such as a lack of certified health care interpreters. Additionally, the Oregon-based COFA Alliance National Network conducts policy and advocacy work aimed at supporting compact migrant communities, according to representatives of the organization. Charities, legal services, and other programs assist compact migrants and other eligible individuals in selected U.S. areas. For example: In Hawaii, the Salvation Army of the Hawaiian and Pacific Islands provides assistance with rent, utilities, and food; interpreters to assist non–English speakers with accessing health and legal services; and digital literacy training (e.g., how to use email), according to Salvation Army officials. In Guam, the Salvation Army Guam Corps provides assistance with rent, utilities, food, and clothing and also provides case management services, according to Guam officials. In the Commonwealth of the Northern Mariana Islands (CNMI), Karidat provides a food pantry, clothing assistance, rental assistance, and victim advocacy, among other services. (Fig. 11 shows a public bulletin board and donated clothing in Karidat’s offices.) In 2018, compact migrants made up 20.4 percent of individuals accessing Karidat’s food pantry and 39.5 percent of individuals receiving clothing assistance, according to Karidat estimates. The Hawaii and Arkansas chapters of the Legal Aid Society provide legal services to local residents, such as victims of crime, according to Hawaii and Arkansas officials. According to Hawaii chapter officials, they served 569 compact migrants (8.5 percent of their total clients) in fiscal year 2019. The Asian Family Center within Oregon’s Immigrant and Refugee Community Organization provides similar services, including defense for parties engaged in deportation removal proceedings, according to representatives of the organization. Some employers with compact migrant workers provide employee services, programs, or accommodations specific to these workers’ needs. In Arkansas, Tyson, Inc., provides written materials in Marshallese and operates a program that appoints chaplains to help the company’s Marshallese workers, as well as other non–U.S. citizen employees, navigate life in the United States generally and in Arkansas specifically, according to private sector representatives. Additionally, the representatives told us that the company provides free classes in financial literacy and English as a second language to its employees, including compact migrants. Another company in the region, Cargill Protein, has partnered with local nongovernmental organizations to educate its compact migrant employees about U.S. driving laws and help prepare them for driver’s license tests. We examined academic studies published from 2015 through 2019 to determine what is known about the likely effects of migration similar to compact migration on the workforces of receiving countries. Because we were unable to identify articles published during this period that focused specifically on compact migration, we focused our search on studies examining the effects of migration by other groups with relatively few skills. Studies that we reviewed sometimes reached differing conclusions about whether migration is associated with a negative, neutral, or positive effect on the employment and earnings of nonmigrant workers in the receiving countries. Some studies found that migration may result in worsened employment prospects or wages—particularly in the short term and if the influx of migrants is sudden—for nonmigrant workers who are most similar to the migrants in terms of demographics and skills. If the migrant workers are close substitutes for nonmigrants, they may intensify competition for jobs, increasing unemployment and lowering wages for such nonmigrant workers as well as for similar prior migrants. In the case of compact migration, this might include younger and less educated nonmigrants. However, according to other studies and survey papers that we reviewed, nonmigrants, both low and high skill, could benefit as a whole from migration. For example, one study of the effects of migration on 20 countries found that both low- and high-skill nonmigrants clearly benefited from an influx of migrant workers about two-thirds of the time. Nonmigrant workers may benefit from migration if the migrant workers specialize in different skills and vocations than the nonmigrant population, leading to complementary effects from scale and specialization. For example, larger numbers of construction workers may result in greater efficiency and quality in the building of more restaurants and bars, benefitting workers in nonconstruction trades as well as nonmigrant investors and business owners. Institutions may play an important role in determining the effects of migration on the receiving country’s workforce. For example, a study estimating the effect of migrant workers in European Union countries and controlling for institutional and noninstitutional factors showed that the effect of migrants varied between countries, driven in part by differences in their institutional environment, such as the extent of unemployment insurance, fiscal redistribution, and government spending on services and public goods. This study found that, while fiscal redistribution to migrant workers through taxation and unemployment benefits somewhat worsens outcomes for nonmigrants, this effect is often outweighed by the economic contribution of these migrants. Distinctions in statistical methodologies and assumptions may explain studies’ seemingly contradictory conclusions about the effects of migration on the workforce of receiving countries. According to a survey paper reviewing other previously published work, the statistical controls selected for studies of the impact of migration can result in subtle but economically important distinctions in what the studies attempt to measure, such as the total effects of migration on a given region or the effects of migration on a specific group (e.g., a particular education or skill group). According to this and a second survey paper we reviewed, contradictory conclusions may also result from differing assumptions about factors such as the extent to which migrants “downskill” (compete for jobs for which they may be overqualified) and, therefore, about the nonmigrants that should be used as a comparison group to examine the effect of migrants of a particular skill and education level. According to a third survey paper we reviewed, studies also vary in whether they measure the shorter- or longer-term effects of migration; the survey found that negative effects are more often reported when studies measure migration’s shorter-term effects. Academic journal articles that we examined also discuss the potential fiscal effect of migration. Several studies argue that evaluations of migration’s fiscal effect should consider the potential effects over multiple generations and should also consider the indirect fiscal effect of migrants’ influence on native workers. For example, a panel discussion report of the National Academies of Sciences, Engineering, and Medicine states that descendants of immigrants are often studied only as children, in cross- sectional data providing a point-in-time snapshot. As a result, according to the report, the average immigrant household is counted as a net fiscal burden in part because young children of immigrants, like the children of natives, receive public education. The report stated that studying the descendants of immigrants as they complete their education, become workers, and start paying taxes provides a more complete measure of migration’s fiscal effect, because such an analysis may include not only the cost of their education but also the delayed fiscal benefits of that education: larger tax payments made possible by the investment in human capital that education represents. Another paper we reviewed argues that because migrant workers can positively influence the upward mobility of native workers, the higher taxable income from these native workers should be considered, in addition to the low taxable income of the migrants, to avoid negatively biasing the estimated fiscal effect of migrants. The REAL ID Act, passed by Congress in 2005, set minimum document requirements and issuance standards for driver’s licenses and personal identification cards. The act also prohibits federal agencies from accepting for certain purposes driver’s licenses and identification cards from states that do not meet the act’s minimum standards. Citizens of the freely associated states (FAS)—the Federated States of Micronesia, the Republic of the Marshall Islands, and the Republic of Palau—have always been eligible for REAL ID–compliant driver’s licenses or identification. However, the term of the licenses’ or identification’s validity and the documents that the Department of Homeland Security (DHS) required to establish compact migrants’ identity have varied over time. Currently, compact migrants are eligible for full-term REAL ID–compliant identification. Since September 2019, they have been required to present an unexpired FAS passport and the most recent Form I-94 Arrival/Departure Record (Form I-94) as evidence of identity to obtain the identification. DHS regulations previously required compact migrants to provide documents they were not required to have. Before September 2019, DHS required compact migrants applying for REAL ID–compliant identification to present, in addition to their unexpired foreign passport and Form I-94, either an unexpired U.S. visa (affixed to the passport) or an employment authorization document (EAD). However, under the compacts of free association, compact migrants are not required to obtain a visa or an EAD. On September 4, 2019, DHS modified its policy, designating an unexpired passport from one of the FAS countries, in combination with an individual’s most recent Form I-94, as acceptable evidence of identity that compact migrants may present to obtain REAL ID–compliant identification. Federal law previously restricted the term of the REAL ID–compliant identification that compact migrants could receive. Before December 2018, compact migrants were eligible to receive temporary, limited-term REAL ID–compliant identification, valid until the expiration date on their EAD, which could be a maximum of 5 years, according to DHS officials. In December 2018, the REAL ID Act Modification for Freely Associated States Act made FAS citizens eligible for full-term REAL ID–compliant identification. Since then, compact migrants have been eligible for full- term REAL ID–compliant identification, valid for the maximum number of years for any license or identification as set by individual U.S. states and territories, according to DHS officials. In several areas that we visited, compact migrant communities described challenges they had experienced in obtaining or renewing their REAL ID– compliant identification. Some compact migrants spoke of difficulty due to the requirement to present a visa or an EAD as evidence of identity. In one location, FAS community members said that other members of the community had lost employment on a military base because they were unable to obtain REAL ID–compliant identification. (We heard many of these observations before September 2019, when DHS modified the policy that required applicants for REAL ID–compliant identification to present a visa or EAD.) Some compact migrants reported being unable to obtain REAL ID– compliant identification for other reasons. number on their current foreign passports. When an FAS citizen’s passport expires and he or she renews it while in the United States, the new passport has a different number than the former passport number displayed on the FAS citizen’s Form I-94. 1. Section 104(e)(9)(A) of the amended compacts’ enabling legislation authorized the President of the United States, at the request of the Governor of Guam or the Governor of the Commonwealth of the Northern Mariana Islands (CNMI), to reduce, release, or waive all or part of any amounts owed by the Guam or CNMI government (or either government’s autonomous agencies or instrumentalities), respectively, to any department, agency, independent agency, office, or instrumentality of the United States. According to section 104(e)(9)(B)(iv), that authority expired on February 28, 2005. 2. The Census Bureau data that we report reflect a definition of “compact migrants” that includes citizens of the Federated States of Micronesia (Micronesia), Republic of the Marshall Islands (Marshall Islands), and Republic of Palau (Palau) who entered the United States after 1986 (from Micronesia and the Marshall Islands) or 1994 (from Palau) and their U.S.-born children (biological, adopted, and step-) and grandchildren younger than 18 years. 1. The Arkansas Department of Education’s data estimating the number of compact migrant students at 4,175 is based on students’ ethnicity (Hawaiian and Pacific Islander) in the 2018-2019 school year. As a result, Arkansas’s estimate may include students who are not Marshallese. In addition, Arkansas’s estimate may include second- generation U.S. citizens, including Marshallese children born in the United States to Marshallese parents who were also born in the United States. The American Community Survey data that we report reflect a definition of “compact migrants” that includes only citizens of the Federated States of Micronesia (Micronesia), Republic of the Marshall Islands (Marshall Islands), and Republic of Palau (Palau) who entered the United States after 1986 (from Micronesia and the Marshall Islands) or 1994 (from Palau) and their U.S.-born children (biological, adopted, and step-) and grandchildren younger than 18 years. The 5,895 compact migrants that the Census Bureau estimated resided in Arkansas during the period 2013 to 2017 (a different time period from that of the data cited by the government of Arkansas) includes only adults and children who met those criteria. We believe that the Census Bureau data are sufficiently reliable for our purposes of estimating the number of compact migrants in U.S. areas. However, our report includes a discussion of stakeholder concerns that the compact migrant population in Arkansas may be undercounted. 2. The population estimate cited in the published study from Arkansas is based in part on a 2013 statement by a Marshallese consulate official. The Arkansas Department of Education estimated there were 4,175 Hawaiian and Pacific Islander students in Arkansas schools in the 2018-2019 school year. 3. Costs related to compact migration in U.S. areas not considered affected jurisdictions are outside the scope of our review. 4. We updated our report to reflect the data that the government of Arkansas cites for the period 1997 to 2019. 1. We have previously reported on defense issues in the Federated States of Micronesia and the Republic of the Marshall Islands (Marshall Islands). For more information about the United States’ right to use part of the Kwajalein Atoll in the Marshall Islands for missile tests and space tracking operations, see GAO, Foreign Relations: Kwajalein Atoll Is the Key U.S. Defense Interest in Two Micronesian Nations, GAO-02-119 (Washington, D.C.: Jan. 22, 2002). For more information about the Marshall Islands’ Nuclear Claims Trust Fund, see GAO, Marshall Islands: Status of the Nuclear Claims Trust Fund, GAO/NSIAD-92-229 (Washington, D.C.: Sept. 25, 1992). 2. Our report provides some information about contributions by compact migrants, including qualitative statements about their budgetary, workforce, and community contributions as well as high-level data on their average per-capita income (see app. IV). We have added the government of Oregon’s statements about the contributions of compact migrants to our report. 3. As our report notes, the affected jurisdictions are defined in the amended compacts’ implementing legislation, which also establishes funding for the associated compact impact grants for those jurisdictions. 4. We made revisions in our report to help direct readers to stakeholders’ suggestions for improving experiences or outcomes of compact migration, presented in appendix VII. 1. Our report incorporates the results of our interviews with members of compact migrant communities, including their reasons for migrating to U.S. areas, workforce challenges and other challenges they face, and their contributions to U.S. communities. Our report also includes these and other stakeholders’ suggestions for improving experiences or outcomes of compact migration (see app. VII). 2. We have previously reported on defense issues in the Federated States of Micronesia and the Republic of the Marshall Islands (Marshall Islands). For more information about the United States’ right to use part of the Kwajalein Atoll in the Marshall Islands for missile tests and space tracking operations, see GAO, Foreign Relations: Kwajalein Atoll Is the Key U.S. Defense Interest in Two Micronesian Nations, GAO-02-119 (Washington, D.C.: Jan. 22, 2002). For more information about the Marshall Islands’ Nuclear Claims Trust Fund, see GAO, Marshall Islands: Status of the Nuclear Claims Trust Fund, GAO/NSIAD-92-229 (Washington, D.C.: Sept. 25, 1992). (from Micronesia and the Marshall Islands) or 1994 (from Palau) and their U.S.-born children (biological, adopted, and step-) and grandchildren younger than 18 years. Given this definition, any individual older than 18 years who was not born in an FAS would not be counted as a compact migrant in the Census Bureau enumerations or the American Community Survey data in this report. 8. Table 10 in appendix II of our report includes estimates of the number of compact migrants in states with fewer than 1,000 estimated compact migrants, except when the data were suppressed by the Census Bureau or the number was unreportable because the margin of error exceeded the estimate. 9. Our report notes that some FAS citizens move to U.S. areas to join the military and that the FASs have a high rate of U.S. military service, according to FAS officials and Department of State documentation. 10. The amended compacts’ implementing legislation permitted the affected jurisdictions (Hawaii, Guam, the Commonwealth of the Northern Mariana Islands, and American Samoa) to submit compact impact reports to the Secretary of the Interior. The definition of “affected jurisdictions” in the legislation did not include any mainland states. 11. Our report notes that compact migrants work in professional industries, including jobs in government and education. 12. We updated our report to include information about the locations of COFA Alliance National Network chapters in states other than Oregon. 1. Our report describes policies allowing compact migrants to access in- state tuition at colleges and universities in some U.S. areas but does not include a comprehensive description of such policies in all U.S. areas. 2. Our report describes this and other challenges related to Form I-94 and includes freely associated state consular officials’ recommendations to their citizens experiencing this challenge (see app. VII). Pacific in the 2005 legislation was an error, it had no impact on FAS citizens’ eligibility for limited-term REAL ID–compliant identification. David Gootnick, (202) 512-3149 or gootnickd@gao.gov In addition to the contact named above, Emil Friberg (Assistant Director), Caitlin Mitchell (Analyst-in-Charge), Topher Hoffmann, Andrew Kurtzman, Reid Lowe, Moon Parks, and Nicole Willems made key contributions to this report. Kathryn Bernet, Justin Fisher, Rebecca Gambler, Christopher Keblitis, Ty Mitchell, Mary Moutsos, and Michael Simon provided technical assistance.", "summary": "The U.S. compacts of free association permit eligible citizens from the freely associated states (FAS), including Micronesia, the Marshall Islands, and Palau, to migrate to the United States and its territories without visa and labor certification requirements. In fiscal year 2004, Congress authorized and appropriated $30 million annually for 20 years to help defray costs associated with compact migration in affected jurisdictions, particularly Hawaii, Guam, and the CNMI. This funding ends in 2023, though migration to U.S. areas is permitted to continue and is expected to grow. GAO was asked to review topics related to compact migration. This report describes (1) estimated compact migrant populations and recent trends in compact migration; (2) reported costs related to compact migration in Hawaii, Guam, and the CNMI; and (3) effects of compact migration on governments, workforces, and societies in these and other U.S. areas. GAO reviewed Census Bureau data to determine the numbers of compact migrants in U.S. areas. In addition, GAO interviewed federal, state, and territory government officials; representatives of private sector and nonprofit groups employing or serving compact migrants; FAS embassy and consular officials; and members of compact migrant communities. In commenting on a draft of this report, U.S. area governments and FAS Ambassadors to the United States identified areas for additional study related to compact migration and impact. Some also discussed policy considerations, including restoration of Medicaid benefits to compact migrants. More than 94,000 compact migrants—that is, citizens of the Federated States of Micronesia (Micronesia), the Republic of the Marshall Islands (Marshall Islands), and the Republic of Palau (Palau) as well as their U.S.-born children and grandchildren younger than 18 years—live and work in the United States and its territories, according to Census Bureau data. Data from Census Bureau surveys covering the periods 2005-2009 and 2013-2017 and an enumeration in 2018 show that the combined compact migrant populations in U.S. areas grew by an estimated 68 percent, from about 56,000 to about 94,000. Historically, many compact migrants have lived in Hawaii, Guam, and the Commonwealth of the Northern Mariana Islands (CNMI). From 2013 to 2018, an estimated 50 percent of compact migrants lived on the U.S. mainland. Hawaii, Guam, and the CNMI track and report the financial costs related to compact migration, or compact impact, for their state or territory. These areas reported estimated costs totaling $3.2 billion during the period fiscal years 2004 through 2018. In fiscal years 2004 through 2019, Hawaii, Guam, and the CNMI received a combined total of approximately $509 million in federal grants to help defray the costs of providing services to compact migrants. In the U.S. areas GAO visited—Arkansas, the CNMI, Guam, Hawaii, Oregon, and Washington—state and territorial officials identified effects of providing public education and health care services to compact migrants. Some area governments use a combination of federal and state or territorial funds to extend health care coverage to compact migrants. For example, some states help compact migrants pay for coverage through health insurance exchanges, created under the 2010 Patient Protection and Affordable Care Act, by covering the cost of premiums not covered by advanced premium tax credits available to eligible compact migrants. Effects of compact migration in these U.S. areas also include compact migrants' budgetary contributions through payment of taxes and fees as well as their workforce contributions—for example, through jobs in hotels, manufacturing, the U.S. military, poultry processing, caregiving, and government.", "document_type": "gao"}
{"report": "The United States took control of the Northern Mariana Islands from Japan during the latter part of World War II. After the war, the U.S. Congress approved a trusteeship agreement making the United States responsible to the United Nations for the administration of the islands. In 1976, the District of the Mariana Islands entered into a covenant with the United States establishing the island territory’s status as a self-governing commonwealth in political union with the United States. The covenant granted the CNMI the right of self-governance over internal affairs and granted the United States complete responsibility and authority for matters relating to foreign affairs and defense affecting the CNMI. The covenant also preserved the CNMI’s exemption from certain federal laws that had previously been inapplicable to the Trust Territory of the Pacific Islands, including certain federal minimum wage provisions and immigration laws, with certain limited exceptions. In 2008, the CNRA amended the joint resolution approving the U.S.– CNMI covenant to generally apply federal immigration law, including the INA, to the CNMI, with a transition period for foreign workers and investors. In addition, the INA provides DHS with discretionary authority to grant parole to certain noncitizens, on a case-by-case basis, allowing them to be temporarily present in the United States, including the CNMI. To provide for an orderly transition from the CNMI immigration system to the U.S. federal immigration system under the immigration laws of the United States, DHS, through USCIS, established the CNMI-Only Transitional Worker program in 2011. Through the program, employers petition for nonimmigrant CW-1 permits that allow foreign workers who meet certain requirements to work temporarily in the CNMI. The CNRA limits the number of permits DHS may issue annually and reduces that number each year until the end of the transition period. Since 2008, Congress has amended the CNRA several times, with provisions that affected the length of the transition period, the number of CW-1 permits allocated, and the distribution of permits (see table 1). Figure 1 shows the past numerical limits on CW-1 permits established by DHS and the current and future numerical limits for permits specified in the Northern Mariana Islands U.S. Workforce Act of 2018, Pub. L. No. 115-218. The limits shown are the maximum number of permits available for each fiscal year through the end of the transition period and may not reflect the number of permits for which employers would petition and that DHS would approve. In addition, the INA provides authorization for several types of visas for nonimmigrant workers and their families—for example, H-2B visas for temporary nonagricultural workers—that became applicable to the CNMI with the passage of the CNRA. The CNRA allows CNMI employers to bring temporary workers to the CNMI under the H-2B program without counting against the numerical restriction for H-2B visas. The CNRA and its implementing regulations established E-2 CNMI Investor (E-2C) status, a classification for certain foreign investors who previously had been lawfully admitted to the CNMI under the territory’s immigration system and who met certain eligibility requirements. Such investors could petition for E-2C status prior to January 18, 2013, according to USCIS. Eligibility criteria include, among others, providing evidence of maintaining financial investments in the CNMI of at least $50,000. DHS may grant E-2C status for up to 2 years, and such status can be renewed. Under the INA, DHS has discretionary parole authority to allow certain noncitizens, on a case-by-case basis, to be temporarily present in the United States. DHS has used this authority to grant parole to individuals who may be inadmissible or otherwise ineligible for admission to allow them to remain in the CNMI, according to DHS. In 2017, the President issued Executive Order 13767, calling for, among other things, the Secretary of Homeland Security to take appropriate action to ensure that parole authority is exercised only on a case-by-case basis in accordance with the plain language of the statute and, in all circumstances, only when an individual demonstrates urgent humanitarian reasons or a significant public benefit derived from such parole. Proposed bill H.R. 560 includes several provisions, among others, that would provide CNMI resident status to eligible individuals. To be eligible for CNMI resident status under H.R. 560, an individual must have been lawfully present in the CNMI under U.S. immigration laws on the date of enactment or on December 31, 2018; be admissible as an immigrant to the United States under the INA, although no immigrant visa is required; have resided continuously and lawfully in the CNMI from November 28, 2009, through the date of enactment; and not be a citizen of the Federated States of Micronesia, Republic of the Marshall Islands, or Republic of Palau. Individuals who meet each of these four criteria would be eligible to apply for CNMI resident status if they fall into one of the categories shown in table 2. As figure 2 shows, the number of CW-1 permits approved by USCIS remained well under the annual numerical limits established by DHS for fiscal years 2012 through 2015 and exceeded or neared the annual limits for fiscal years 2016 and 2017. According to USCIS data, most individuals with approved CW-1 permits for fiscal years 2015 through 2018 were born in the Philippines or China. In addition, as table 3 shows, four times more CW-1 permits were issued to Chinese nationals for fiscal years 2016 and 2017 than for fiscal year 2015. As we reported in 2017, firms involved in building a new casino in Saipan have primarily employed Chinese workers. About one-third of fiscal year 2018 CW-1 permit holders had maintained continuous employment in the CNMI since 2015 and could be eligible for CNMI resident status under H.R. 560, if they had been admitted every year under CW-1 status and were otherwise eligible. USCIS CW-1 permit data for fiscal years 2015 through 2018 show that, of the 8,995 foreign workers with CW-1 permits approved by USCIS for fiscal year 2018, 2,875 workers (about 32 percent) had maintained continuous employment in the CNMI since fiscal year 2015. (Of this group, 2,287—80 percent—were born in the Philippines.) Under H.R. 560, a foreign national who meets additional eligibility requirements, including having resided continuously and lawfully in the CNMI from November 28, 2009, through the date of enactment, may be admitted to the CNMI under CNMI resident status if that individual was admitted to the CNMI as a CW-1 worker during fiscal year 2015 and during every subsequent fiscal year beginning before July 24, 2018. As a result, according to our analysis of USCIS data, 2,875 workers could be eligible under H.R. 560 to apply for CNMI resident status if they were admitted as CW-1 workers every fiscal year until 2018 and met all other eligibility conditions. Table 4 shows the numbers of foreign workers who received CW-1 permits for fiscal year 2018 and had maintained continuous employment in the CNMI since fiscal years 2012 through 2017. USCIS data show a reduction from fiscal year 2017 to fiscal year 2018 in the number of CW-1 permit holders and a significant increase in the number of H-2B beneficiaries. While the number of approved CW-1 permit holders declined from 12,889 in fiscal year 2017 to 8,995 in fiscal year 2018, the number of H-2B beneficiaries for those years increased from 0 to 3,058. In addition, our analysis of USCIS data found that the number of CW-1 permit holders for the construction trade declined from 2,981 to 545—by 82 percent—from fiscal year 2017 to fiscal year 2018. Meanwhile, the number of H-2B beneficiaries for the construction trade in the CNMI increased from 0 for fiscal year 2017 to 1,801 for fiscal year 2018. In August 2017, Congress amended the CNRA to, among other things, restrict CW-1 permits for workers in construction and extraction occupations (as defined in the U.S. Department of Labor’s Standard Occupational Classification system) by allowing only extensions of CW-1 permits first issued before October 1, 2015. The CNRA allows CNMI employers to petition for H-2 visas to bring temporary workers, such as construction workers, to the CNMI without counting against the numerical restriction for such visas. According to a senior USCIS official, the new casino employer in Saipan began petitioning in 2018 for foreign workers under the H-2B program instead of petitioning for CW-1 permits for its construction workers. The official noted that Pub. L. No. 115-53’s restriction on the use of CW-1 permits for construction trade workers may account for the decrease in petitions for CW-1 permit holders and increase in petitions for H-2B beneficiaries from fiscal year 2017 to fiscal year 2018. Table 5 shows the numbers of approved CW-1 permit holders and H-2B beneficiaries for the construction trade in fiscal years 2016 through 2018. In October 2016, DHS announced the list of countries whose citizens were eligible to participate in the H-2 program from January 18, 2017, to January 18, 2018. Asian countries on the list included the Philippines, South Korea, Taiwan, and Thailand, among others, but did not include China. In January 2019, because of concerns about overstays and human trafficking, DHS removed the Philippines from the list of countries eligible for the H-2B program. CNMI government and Chamber of Commerce officials have voiced concerns that the removal of the Philippines from the list will make it difficult to hire construction workers in the aftermath of two recent typhoons. USCIS began approving 2-year E-2C status for eligible foreign long-term investors and their dependents in the territory in fiscal year 2011. According to USCIS, as of February 5, 2019, 56 investors who had previously resided in the CNMI as investors under CNMI immigration law were residing in the CNMI with E-2C status. Under H.R. 560, foreign nationals who otherwise meet additional eligibility requirements may be granted CNMI resident status if they resided in the CNMI as investors under CNMI immigration law and are presently resident under E-2C status. As a result, under H.R. 560, these 56 investors could be eligible to apply for CNMI resident status if they met all other eligibility conditions. According to USCIS testimony, after the CNRA was passed in 2008, USCIS implemented DHS’s discretionary parole authority by making parole available to groups of individuals residing in the CNMI who would not be covered by INA classifications and for whom the classifications established in the CNRA did not appear to be appropriate. These individuals previously had immigration status under CNMI immigration law that allowed them to potentially remain in the CNMI indefinitely, according to USCIS. Without USCIS action, these individuals would have been deemed unlawfully present in the United States, according to USCIS documents. To provide such individuals with a means to remain temporarily in the CNMI during the transition period, USCIS announced several discretionary parole policies to cover the following groups, among others, which were potentially eligible for parole: CNMI permanent residents, immediate relatives of CNMI permanent residents, spouses and children of deceased CNMI permanent residents, and immediate relatives of citizens of the freely associated states (November 2009) Certain in-home foreign national caregivers of CNMI residents (October 2011) Immediate relatives of U.S. citizens, especially parents of U.S. citizen children, and stateless individuals in the CNMI (November 2011) In response to Executive Order 13767, on December 27, 2018, USCIS announced the termination of parole for immediate relatives of U.S. citizens and certain stateless individuals; CNMI permanent residents, immediate relatives of CNMI permanent residents, and immediate relatives of citizens of the freely associated states; and certain in-home foreign worker caregivers of CNMI residents. To provide an opportunity for individuals in these categories to prepare to depart or seek a different lawful status, USCIS announced that the affected individuals were allowed to remain in the CNMI with a transitional parole status for up to 180 days, not to extend beyond June 29, 2019. According to a senior USCIS official, from December 2, 2016, through December 14, 2018, USCIS had granted parole until December 31, 2018, to 1,039 individuals in the terminated parole categories. Under H.R. 560, some of these individuals could be eligible to apply for CNMI resident status if they met all other eligibility conditions. Vice Chairman Sablan, Republican Leader Gonzalez-Colon, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff have any questions about this testimony, please contact David Gootnick, Director, International Affairs and Trade, at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Emil Friberg (Assistant Director), Julia Ann Roberts (Analyst in Charge), Sada Aksartova, Andrew Kurtzman, Reid Lowe, and Alexander Welsh. Technical support was provided by Kathryn Bernet, Justin Fisher, Christopher Keblitis, Mary Moutsos, and Moon Parks. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The 1976 covenant defining the political relationship between the CNMI and the United States exempted the CNMI—a U.S. territory north of Guam—from certain federal immigration laws. However, the covenant preserved the right of the U.S. government to apply federal law in these exempted areas. The CNRA, which amended a joint resolution approving the covenant, generally established federal control of CNMI immigration beginning in 2009. In 2009, DHS began implementing, among other things, a foreign worker permit program to address CNRA provisions specific to the CNMI. DHS also began using its discretionary authority under the INA to parole certain groups of individuals into the CNMI (i.e., allow them to be temporarily present). Congress has amended the CNRA several times with provisions that affected the total number of permits allocated and the distribution of permits. Proposed bill H.R. 560 would further modify the CNRA by establishing a CNMI resident status for certain individuals. Among its other provisions, the CNRA allows CNMI employers to petition for H-2 visas for temporary workers without counting the visas against a numerical restriction. Drawing from ongoing work, this testimony discusses DHS's implementation of (1) selected CNRA provisions regarding foreign workers, among others, in the CNMI and (2) its discretionary parole authority under the INA as applied in the CNMI. GAO updated information from May 2017 ( GAO-17-437 ) and February 2018 ( GAO-18-373T ), reviewed relevant legal documents, and analyzed DHS data. Under the Consolidated Natural Resources Act of 2008 (CNRA), the Department of Homeland Security (DHS) established the nonimmigrant Commonwealth of the Northern Mariana Islands (CNMI)–Only Transitional Worker program in 2011. Through the program, eligible foreign nationals can obtain CNMI-Only Transitional Worker (CW-1) permits to work temporarily in the CNMI. Under H.R. 560, foreign nationals who meet additional eligibility requirements could be eligible to receive CNMI resident status if they were admitted annually to the CNMI as a CW-1 worker in fiscal years 2015 through 2018. GAO's preliminary analysis of DHS data found that 2,875 (about 32 percent) of 8,995 workers with CW-1 permits for fiscal year 2018 had maintained continuous employment each fiscal year since 2015 (i.e., received a CW-1 permit annually). While DHS data show the number of approved CW-1 permit holders declined from fiscal year 2017 to fiscal year 2018 (see figure), the number of H-2B beneficiaries—who often fill construction jobs—increased from 0 to 3,058. In January 2019, DHS removed the Philippines from the list of countries eligible for the H-2B program. In 2009, DHS began granting discretionary parole that authorized temporary stays for certain CNMI residents, such as spouses and children of U.S. citizens. These individuals may have been inadmissible or otherwise ineligible for admission to the United States, according to DHS. However, in December 2018, DHS announced that it was terminating parole for certain categories of residents in response to Executive Order 13767, issued in 2017. The order called on DHS to take appropriate action to ensure that parole authority is exercised only on a case-by-case basis, among other things. According to DHS, 1,039 individuals in the terminated categories had been granted parole until December 31, 2018. Under H.R. 560, some of these individuals could be eligible to apply for CNMI resident status.", "document_type": "gao"}
{"report": "NRC was established by the Energy Reorganization Act of 1974 and is headed by five commissioners, collectively referred to as the Commission, with members appointed by the President and confirmed by the Senate. One commissioner is designated by the President to serve as the Chair, who, among other things, serves as the official spokesperson of the Commission. The Commission is responsible for, among other things, revising budget estimates and determining the distribution of appropriated funds according to major programs and purposes. NRC staff from program offices in headquarters and four regional offices implement the agency’s programs for developing regulations, licensing, inspection, enforcement, and emergency response, among other responsibilities. In addition, NRC’s Office of the Chief Financial Officer establishes, maintains, and oversees the implementation and interpretation of the agency’s regulatory user fee policies and regulations and is responsible for assessing annual and service fees to licensees for each license they hold and sending invoices to licensees. The Office of the Chief Financial Officer also leads the agency’s budget formulation and execution processes. NRC’s authority to charge service and annual fees is derived from two laws: the Independent Offices Appropriations Act of 1952 and the Omnibus Budget Reconciliation Act of 1990, as amended. The Independent Offices Appropriations Act of 1952 provides broad authority to federal agencies, including NRC, to assess user fees or charges to identifiable beneficiaries through regulation. The Omnibus Budget Reconciliation Act of 1990 requires that NRC recover approximately 90 percent of its annual budget authority through fees assessed to licensees, excluding amounts appropriated for any one of a number of specified purposes. The law requires that NRC first use its authority from the Independent Offices Appropriation Act of 1952 to collect service fees for specific services provided. However, because those fees do not equal 90 percent of NRC’s budget authority, NRC also assesses annual fees. To the maximum extent practicable, the annual fees assessed must have a reasonable relationship to the cost of the regulatory services provided and may be based on how NRC allocates resources for regulating licensees or fee classes. The licensees regulated by NRC encompass a broad range of commercial uses of nuclear material such as use in commercial nuclear power reactors; the use of radioactive materials in medical, academic, and industrial settings; and the transport, storage, and disposal of radioactive materials and waste. For the purpose of setting service and annual fees, NRC established nine fee classes. Figure 1 shows NRC’s total budget authority and the amounts NRC collected through service and annual fees from licensees for fiscal years 2014 through 2019. NRC sets its service fees and annual fees through the federal rulemaking process every year. Under this process, NRC first drafts and then publishes a proposed fee rule in the Federal Register, after which interested parties have 30 days to comment. NRC develops the proposed fee rule by allocating its resources for regulating the fee classes and calculating its proposed hourly fee rate, which is the same rate for all fee classes, and annual fees, which vary by fee class. NRC bases its calculations for the proposed fee rule on its appropriation for the current fiscal year, if enacted. If the agency has not received its appropriation by the time it begins calculating fees, it bases these calculations on the President’s Budget. As part of this process, NRC generally posts fee work papers providing additional details to support the proposed fee rule. After the 30-day comment period, NRC adjusts its hourly fee rate for service fees and annual fees, as needed, and drafts a final fee rule. Finally, NRC publishes a final fee rule that includes its responses to comments received on the proposed fee rule. The final fee rule becomes effective 60 days after publication. In recent years, GAO and NRC internal initiatives identified several key findings and made recommendations to improve NRC’s fee-setting process, including the following: In 2016, NRC’s Fees Transformation Initiative recommended process improvements related to updating NRC’s fee rules and associated work papers. In 2017, we recommended that NRC clearly present information in NRC’s proposed fee rule, final fee rule, and fee work papers such as by defining and consistently using key terms so that stakeholders could understand fee calculations. In 2017, we also recommended that NRC develop objective, measurable, and quantifiable performance goals and metrics that would enable NRC to assess its efforts to improve the transparency and timeliness of its fee-setting process. NRC’s billing process for annual fees is based solely on the annual fee rate that is set through the rulemaking process. NRC’s billing process for service fees begins by identifying work that can be billed to a specific licensee and ends when the licensee pays its invoice for service fees. Once NRC determines that billable work needs to be done, NRC program office staff and contractors perform the work. NRC follows the steps in the billing process shown in figure 2. NRC assigns an activity code under which billable work is accomplished. Both NRC staff and contractors can perform work under the same billing codes. Once the hours charged have been reviewed, NRC aggregates the charges in its financial accounting system. Then, NRC staff and supervisors verify the hours billed to that activity code on a monthly basis through a formal, agency-wide billing validation process. On a quarterly basis, NRC will send invoices through the U.S. Postal Service or electronically to licensees, who have 30 days from issuance of the invoice to review and pay the invoice before being assessed late fees. Licensees may also dispute charges at this point. According to NRC staff, most disputes are handled informally and generally entail explanations of the agency’s billing or licensing policies. NRC provides licensees information on billing through other methods as well. For example, as part of the Fees Transformation Initiative, NRC posted public cost estimates for common regulatory actions to help licensees better plan for the cost of those actions. In addition, during the course of regulatory actions, NRC staff are to communicate with licensees about the status of the work being performed, and NRC can provide licensees biweekly estimates of charges from NRC staff and contractors to supplement its billing invoices. Over the years, GAO, NRC’s OIG, and NRC internal initiatives identified several key findings and made recommendations to improve NRC’s billing process, including the following: NRC began an initiative aimed at improving its billing codes in 2013, and in 2017, NRC’s Cost Accountability and Management Project plan stated that NRC’s methods for requesting and managing billing codes place the agency at risk of collecting inaccurate data. In 2018, we found that NRC was working to improve internal controls over the billing codes NRC staff use to record their work hours, which did not describe the work being performed and did not have a consistent naming convention. Relatedly, in 2015, NRC’s OIG recommended that NRC establish policies and procedures to centralize control of its billing code structure, link billing codes to specific tasks, and design and implement controls regarding the billing codes to which staff can charge time. In 2014, NRC’s Fee Billing Process Improvement Project report recommended that NRC standardize and document its fee-billing validation process, along with developing and issuing guidance for the process. In 2018, we found that NRC’s billing validation procedures for verifying the accuracy of time charged to licensees was not standardized across the regional and program offices, but NRC was planning to pilot a standardized process. Relatedly, in 2015, NRC’s OIG recommended that NRC design and implement a plan to improve its billing validation process. Similarly, in 2017, NRC’s OIG recommended that NRC implement a streamlined and consistent billing validation process and define the roles and responsibilities for billing validation staff. In 2016, NRC’s Fees Transformation Initiative recommended process improvements related to the information NRC provides to licensees through cost estimates and on invoices. In 2017, NRC’s OIG recommended that NRC create consistent, well-defined processes and reporting to calculate and explain its cost estimates. In 2018, we found that NRC had posted cost estimates for common regulatory activities to its public website. In 2018, we recommended that NRC formally communicate to licensees the availability of supplemental billing information, including biweekly reports and monthly status reports on contractor charges. Relatedly, in 2015, NRC’s OIG recommended that NRC design and implement a process to provide information regarding contractor charges on invoices that identifies the specific tasks performed and related reimbursable contractor costs. In 2018, we found that NRC intended to transition to electronic billing to address challenges some licensees were experiencing with the format and timeliness of invoices, but did not have planning documents for this transition. We recommended that NRC develop a project plan for the transition to electronic billing incorporating plans for schedule and cost, steps that involve soliciting and considering licensee feedback, and steps to assess the results of implementing electronic billing. NRC submits an annual budget justification to Congress with estimates and other information that support the policies and proposed spending decisions represented in the President’s Budget. This includes information on what NRC plans to achieve with the resources the agency requested. After Congress enacts appropriations providing NRC’s budget authority for the fiscal year, NRC allocates these appropriated funds to its offices, which obligate them to carry out the agency’s mission. Though NRC receives its funding from these congressional appropriations, the agency then collects approximately 90 percent of its budget from service and annual fees, and the fees collected are then deposited to the U.S. Treasury. NRC’s budget structure is currently grouped by programs and business lines, among other subsets. For fiscal year 2020, NRC’s two major programs are (1) Nuclear Reactor Safety and (2) Nuclear Materials and Waste Safety. Under these two programs, seven business lines relate to key regulatory groups of licensees as follows: The Nuclear Reactor Safety Program New Reactors (including Advanced Reactors) The Nuclear Materials and Waste Safety Program Spent Fuel Storage and Transportation Decommissioning and Low-Level Waste In addition to these seven programmatic business lines, there is a Corporate Support business line which encompasses agency-wide support activities, including acquisitions, administrative services, financial management, human resource management, information management, information technology, outreach, policy support, and associated training and travel. The Corporate Support business line supports all of the programmatic business lines, and Corporate Support costs are allocated across the other business lines in NRC’s budget. Over the years, GAO, NRC’s OIG, and NRC internal initiatives identified several key findings and made recommendations to improve NRC’s budgeting process, including the following: NRC’s OIG reported in 2013 and we similarly reported in 2017 that NRC’s budget justification and related systems did not align with its budget execution. NRC’s OIG recommended NRC enforce the consistent use of financial management system codes to help address this issue. Relatedly, in 2016, an NRC internal initiative identified the need to present actual obligation data in its budget justifications. In 2017, we found that NRC did not present actual obligation data in its budget justifications for fiscal years 2010 through 2017, which made it difficult for users of the budget justification—including Congress and licensees—to understand how NRC spent its appropriations. Since 2017, NRC has implemented changes to its fee-setting process in response to GAO and internal NRC findings and recommendations. Those changes have improved the fee-setting process in two main areas: (1) the clarity and consistency of terms used in fee rules, and (2) performance goals and measures for transparency and timeliness of NRC’s fee-setting process. NRC began using clear and consistent terms in its fee rules in response to GAO and internal NRC recommendations. Specifically, in 2017 we recommended that NRC clearly present information in NRC’s proposed fee rule, final fee rule, and fee work papers by defining and consistently using key terms, providing complete calculations for how fees are determined, and ensuring the accuracy of the fee rules and work papers. In addition, NRC’s Fees Transformation Initiative identified process improvements related to updating NRC’s fee rules and associated work papers. Beginning with NRC’s fiscal year 2017 proposed fee rule, NRC made the following changes: NRC provided definitions of key terms used in the calculation of its hourly-fee rate for service fees. After providing these definitions in its fee rules and workpapers for fiscal years 2017 and 2018, NRC codified these definitions in its regulations in June 2018. NRC posted fee-related spreadsheets in electronic format on its public website to supplement the proposed and final fee rules. Specifically, NRC included an additional supplemental spreadsheet with downloadable data comparing budgeted resources from the proposed fee rule to the prior year’s amounts to enhance transparency on changes from year to year. Seven of the 11 licensees we interviewed said that NRC uses clear and consistent information in its fee rule and associated work papers. One licensee said that that the language in the fee rule and work papers is difficult to follow given the finance terminology but noted that its organization has not done a thorough review of the fee rule. Another licensee said that the fee rule has a large amount of data that is easy to follow given NRC’s detailed work papers, but the licensee would like to see additional narrative information in the fee rule justifying increases or decreases to fee categories. The remaining two licensees had no comment. Additionally, one of the seven licensees who said NRC uses clear and consistent information in its fee rule also told us that determining what licensees pay for through NRC’s fees at a more detailed level is difficult because NRC does not stipulate which NRC actions are specifically recovered through service fees and which are recovered through annual fees. In 2017, we reported that NRC’s budgeting system is not designed to provide information on which budget items are recovered specifically through service fees and which are recovered through annual fees. At that time, NRC staff told us that the agency was trying to determine if its budget formulation system could be modified to address this concern. According to NRC officials, since we last reported, NRC has modified the system so that, beginning with its fiscal year 2021 budget justification, the agency can provide more detailed information for the operating reactor fee class in accordance with legislative requirements in the Nuclear Energy Innovation and Modernization Act. NRC officials said that this information will include which budgeted activities are proposed to be recovered through service fees versus through annual fees for the operating fee class. The agency has not made similar modifications to its budgeting system to provide more detailed information for the other fee classes, according to NRC officials, because NRC has prioritized making system upgrades to address legislative requirements and is only required to provide more detailed information for the operating reactor fee class, as well as because the formulation of NRC’s budget is done two years in advance of the fee rule and the information is subject to change. NRC developed performance goals and measures for the transparency and timeliness of its fee-setting process in response to a GAO recommendation and internal NRC findings and recommendations. Specifically, in 2017 we recommended that NRC develop objective, measureable, and quantifiable performance goals and measures to enhance the transparency and timeliness of NRC’s fee-setting process. NRC established three performance goals for its fee-setting process: (1) increased transparency, (2) increased equitability, and (3) increased timeliness. To meet the first two performance goals of increased transparency and equitability, NRC developed several performance measures, including implementing 80 percent of identified improvements in NRC’s Fees Transformation Initiative, holding two public outreach meetings with stakeholders on fee-setting or billing topics, and soliciting public comments on improvement activities. For NRC’s performance goal of increasing timeliness, its performance measure is to meet NRC’s planned date for issuance of the proposed and final fee rules. NRC’s goal is to issue its proposed fee rules in January and final fee rules in May of a given fiscal year. However, NRC finalizes its fee rule after it receives its annual appropriations, and according to NRC officials, NRC’s publication of the final fee rule may be delayed depending on when NRC receives its annual appropriations. Based on our review of NRC documents and interviews with agency officials, we have determined that NRC has met these performance measures. Specifically, NRC closed as implemented about 93 percent of improvements—37 of 40—NRC identified as a part of its Fees Transformation Initiative. Further, NRC has held numerous public outreach meetings on these topics since 2017 and solicited public comments, with the most recent public meeting occurring on February 13, 2019, to discuss key features of NRC’s fiscal year 2019 fee rule. For fiscal years 2017 through 2019, NRC issued its proposed fee rule in January. NRC published its final fee rules for fiscal years 2017 and 2018 in June, and published its fiscal year 2019 final fee rule in May. Since 2017, NRC has implemented changes to its billing process in response to GAO, NRC OIG, and internal NRC findings and recommendations to improve the transparency, accuracy, and timeliness of the process, but some billing information NRC provides licensees is still not transparent. NRC improved transparency by standardizing its billing codes, updating its invoices, formally communicating some supplemental billing information, and creating public cost estimates, but it has not ensured the estimates clearly define what costs are included or provided work progress information throughout the course of ongoing regulatory activities. NRC also implemented a standardized process to validate charges to licensees to improve accuracy. In addition, NRC enhanced the timeliness of its billing process by implementing an electronic billing system. NRC implemented changes to increase the transparency of its billing process in response to GAO, NRC OIG, and internal NRC findings and recommendations, in four main areas: (1) standardized billing codes, (2) updated invoices, (3) supplemental billing information, and (4) public cost estimates. NRC improved its standardized billing codes—codes that NRC staff use to record their work hours on time cards—in response to GAO findings and NRC OIG recommendations as well as NRC internal initiatives. Specifically, in 2018 we reported that NRC’s billing codes did not adequately describe work performed and did not have a consistent naming convention, which increased the risk of staff charging their time to the wrong billing codes. We reported that this, in turn, could lead to billing errors. In addition, in 2015 NRC’s OIG recommended, among other things, that NRC establish policies and procedures to centralize control of billing codes. Moreover, NRC began an initiative aimed at improving its billing process in 2013, and in 2017, NRC’s Cost Accountability and Management Project plan stated that NRC’s methods for requesting and managing billing codes place the agency at risk of collecting inaccurate data. Based on our review of NRC’s updated billing codes, the agency’s Enterprise Project Identifiers, (EPID)—umbrella codes for regulatory actions such as inspections, licensing actions, and licensing renewals— now have a consistent naming structure, and NRC has centralized control of billing codes. In particular, the EPID alpha-numerical naming structure denotes the type of regulatory work, the calendar year the work began, and includes a 4-digit number to make the code unique, among other elements. NRC also created and implemented Cost Activity Codes (CAC), which are numerical codes that capture the ways in which NRC staff spend billable time working on an EPID, including the time spent preparing and documenting an action as well as performing the direct work. In addition, NRC added controls to ensure staff charge the correct billing codes. For example, NRC management must now grant staff permission before they are able to charge these codes. Eight of the 11 licensees we interviewed said that NRC has consistently used both EPIDs and CACs after NRC revised the accounting structure. The remaining three licensees we interviewed had no comment on the revised EPIDs and CACs. NRC officials stated that while they have completed standardizing the billing codes, they are continuously working on refining them to respond to stakeholder feedback, and NRC started a working group in November 2019 to further review the codes. NRC updated the service fee invoices it sends to licensees in response to internal NRC initiatives, but it has not implemented an internal NRC recommendation to provide licensees with information on the progress of work performed on ongoing regulatory actions. As part of our prior review of NRC’s billing process, we found that NRC expected to issue updated invoices to licensees. We were unable to assess licensees’ satisfaction with the updated invoices because NRC issued them after we had completed our review. In January 2018, NRC updated the service fee invoices it sends to licensees, in response to NRC internal initiatives, to include the names of NRC staff and of contractors billing time, along with the updated EPIDs and CACs. According to our analysis of a sample of invoices from before and after January 2018, NRC has consistently made these changes to invoices. As shown in figure 3, NRC’s updated invoices provide the quarterly total of all charges for a given regulatory action as an EPID total. CACs are no longer specific to a project or site and can now be reused to represent the same type of work for different EPIDs. NRC staff and contractors can charge multiple CACs to the same EPID during a given quarter. All 11 licensees we interviewed stated that the changes NRC made to the invoices were positive. Seven licensees stated that the inclusion of staff names made it easier to understand what they were being billed for, and five licensees stated that the inclusion of CACs improved NRC’s billing process. In 2016, NRC’s Fees Transformation Initiative recommended a process improvement to include information on the progress of work performed on inspection reports, but NRC ended this initiative in 2018 without making updates to the inspection reports. NRC determined that the updated information on invoices, described previously, resulted in sufficient improvements to transparency. However, two licensees we interviewed told us that including information on the progress of work performed would assist licensees with their planning and budgeting. According to our analysis of NRC documents and a licensee we interviewed, some NRC regulatory oversight actions can take several years to complete, with charges to licensees from a single action spanning multiple quarterly invoices. One licensee explained that, as a result, not having information on the progress of work performed on ongoing regulatory actions can make it more difficult to budget. This is because the licensee does not know how far NRC is in completing an activity, and NRC may invoice for a large amount of additional costs that the licensee did not anticipate. One of NRC’s program offices has a policy regarding when to communicate information on the progress of work performed on ongoing regulatory actions, but the remaining NRC program offices we spoke with do not. Specifically, the Office of Nuclear Reactor Regulation has a policy to communicate with operating reactor licensees if it anticipates significant changes to the forecasted completion date or hours billed to complete the action. Furthermore, officials in this office said that it has a practice to notify the licensee when it estimates that NRC will expend over 125 percent of the initial estimate of hours for a given regulatory action. These officials said that the office created this policy and practice to improve its communication with licensees, in support of NRC’s Principles of Good Regulation, which includes guidance on transparency. They further stated that this policy and practice benefit licensees by allowing them to better budget and plan for NRC’s work. Additionally, they benefit NRC by helping the agency to better manage its resources and workload, according to these officials. In contrast, officials from the Office of New Reactors and the Office of Nuclear Material Safety and Safeguards stated that they do not have a policy regarding communicating with licensees about the progress of work performed on ongoing regulatory actions. Furthermore, there is no agencywide policy or guidance regarding this communication. Officials from the Office of New Reactors stated that the office tracks percent completion as an internal metric, but does not communicate this information to licensees. The Office of Nuclear Material Safety and Safeguards does not track regulatory actions by percent completion. NRC officials told us that it is difficult to provide accurate estimates of work progress to licensees because NRC’s ability to meet anticipated cost and schedule estimates depends on the complexity of the NRC action. NRC’s Principles of Good Regulation and NRC’s Organizational Values list openness as a key principle and value, respectively. According to those documents, being open—that is, transparent and forthright—should guide every action NRC takes, how it performs administrative tasks, and how it interacts with stakeholders, such as licensees. Additionally, Standards for Internal Control in the Federal Government state that management should externally communicate the necessary quality information to achieve the entity’s objectives. According to NRC officials, NRC generally provides licensees with an estimate of the number of hours and length of time NRC anticipates it will take to complete certain regulatory actions upon beginning the action. NRC officials said that NRC project managers are in regular contact with licensees about the status of ongoing NRC activities; however, three licensees we interviewed stated that NRC’s project managers do not always communicate about the status of the regulatory action, which can make planning and budgeting more difficult. This is in part because the program offices do not each have a policy regarding when NRC should provide updates on cost and schedule. Formalizing when NRC staff are to communicate information to licensees on the progress of work performed could enhance transparency and make planning and budgeting easier for licensees, as they would have more information about when an action is expected to be completed or when it will cost more than NRC’s initial estimate. NRC formally communicated to licensees that supplemental billing information about contractor charges is available and developed guidance on how that information should be provided in response to two GAO recommendations. Specifically, we reported in 2018 that, upon request, NRC can provide information on contractor charges to licensees through a summary of work performed or a biweekly summary of charges that lists all billable activities charged during a 2-week period. NRC officials stated that the purpose of the biweekly summaries is to provide licensees with information on costs that accrued in that particular period to help licensees estimate their quarterly bill amount. We recommended that NRC formally communicate to all licensees that these two supplemental billing reports were available and how to request them, as we found that not all licensees were aware this information was available. We also recommended that NRC develop policy and guidance on what billing information related to contractor charges NRC staff could provide to licensees and how it should be provided. In January 2019, NRC formally communicated to licensees that supplemental billing information about contractor charges is available, but it has yet to formally communicate to licensees that biweekly summaries of charges are also available. Specifically, NRC created a process for licensees to request narrative information on contractor charges through a standard form, and formally communicates that process to licensees through a reference to the form on agency invoices. NRC also developed guidance on what billing information related to contractor charges NRC staff can provide to licensees, along with a process map for how to respond to licensee requests for contractor information. In contrast, while NRC has continued to provide biweekly summaries to licensees upon licensee request, the agency has not formally communicated the existence of these reports to licensees. Seven of the 11 licensees we interviewed receive the biweekly reports, and five of these licensees said the reports allow them to better track billable activity through the quarter. The remaining four licensees we interviewed were unaware that NRC can provide these reports. Agency officials stated that they do not have the capacity to provide these biweekly reports to all licensees, as the current process is manual and labor-intensive. However, NRC officials stated that they plan to create an automated process to provide these biweekly summaries as an enhancement to the agency’s electronic billing initiative. NRC plans to implement this enhancement by March 2020, according to NRC officials. NRC created and posted public cost estimates for common oversight activities to its website in response to an internal NRC recommendation, but it has not consistently updated those estimates or ensured the estimates clearly defined what costs were included. Specifically, in 2017, NRC’s Fees Transformation Steering Committee, chaired by a representative in the Office of the Chief Financial Officer, tasked NRC program officials with creating public cost estimates for common regulatory actions to increase transparency and enhance stakeholder awareness of the costs associated with activities such as site permitting, design certifications, inspections, license amendments, and license renewals. We have previously reported that licensees had identified challenges with planning for future work and budgeting to pay future costs because NRC had not provided certain information about the agency’s billable work, such as cost estimates. Beginning in September 2017, staff from several NRC program offices posted public cost estimates relating to six types of regulated entities: operating reactors, new reactors, fuel facilities, spent fuel storage and transportation, decommissioning, and uranium recovery. The cost estimates, which are based on historical expenses and are calculated using a sample of licensing and oversight actions, include the low, high, and average number of NRC staff hours billed for each action, as well as some estimates for contractor charges for certain tasks. According to agency officials, NRC does not use these estimates as part of its budgeting and fee-setting processes since these public cost estimates are a resource for identifying possible costs, but are not tailored to a site- specific NRC action. Instead, the estimates assist stakeholders with planning for the costs of future NRC work. The Fees Transformation Steering Committee created guidance that the program offices should update the estimates periodically, and NRC also posted on its public website that these estimates would be updated biennially. However, we found that, as of December 2019, NRC’s program offices had updated only two of the six estimates. When we discussed this with NRC officials at that time, the Office of the Chief Financial Officer sent out a reminder to the program offices to update their estimates by January 31, 2020. Additionally, we found that NRC program offices did not clearly define what costs—such as project management—are included across the six public cost estimates, which may limit stakeholders’ ability to understand them. For example, the cost estimate for operating reactors included “inspection support” activities and defined what types of costs are included in this category. In contrast, the cost estimate for fuel facilities included a category for “project management activities,” but did not define what types of costs are included in this category. The remaining four cost estimates did not mention project management costs, so it is unclear whether the estimates include these types of costs. According to our analysis of NRC documents and licensees we spoke with, project management costs for some NRC actions can account for about two thirds of total hours billed. Thus, increased transparency of these costs could help stakeholders—such as NRC licensees or potential applicants—better understand the full cost of NRC’s regulatory actions. The Fees Transformation Steering Committee provided high-level guidance to the program offices for developing cost estimates, but the guidance did not specify what costs to include when creating these estimates. According to NRC officials, the Committee did not provide specific guidance regarding cost estimates because activities in the cost estimates may vary based on the specific activities conducted by the program offices and it wanted the program offices to have flexibility when creating the estimates. We recognize that some activities in the cost estimates will vary based on the different activities conducted by the program office. However, certain costs, such as project management, are relevant across all cost estimates, and it is not always clear whether these costs are included. As previously discussed, NRC’s Principles of Good Regulation and NRC’s Organizational Values list openness—that is, being transparent and forthright—as a key principle and value, and it is applicable to the agency’s cost estimates. Additionally, Standards for Internal Control in the Federal Government state that management is to externally communicate the necessary quality information to achieve the entity’s objectives. By clearly defining what costs are included in its public cost estimates, NRC could enhance transparency and increase the value of these estimates as a budgeting and planning tool for stakeholders. In response to internal NRC and NRC OIG recommendations, NRC implemented a standardized process to validate charges to licensees to improve accuracy. Specifically, in 2014 NRC’s License Fee Billing Business Process Improvement report recommended that NRC standardize and document its fee-billing validation process, along with developing and issuing guidance for the process. Furthermore, in 2017 NRC’s OIG recommended that NRC implement a streamlined and consistent billing validation process and define the roles and responsibilities for billing validation staff. Reinforcing these recommendations, we reported in 2018 that NRC did not have formal guidance on validating charges and that the process varied among NRC’s program offices. At the time of our review, NRC was planning to standardize the process and establish clear roles and responsibilities for staff participating in the process. In August 2019, NRC implemented a revised process for validating time charged to licensees in order to improve the accuracy of invoices, identify billing errors in a timelier manner, and standardize billing validation throughout the agency. This revised process came out of the work of NRC’s Fee-Billing Validation Working Group, which began work in December 2017. NRC implemented several changes to standardize the process agency-wide, including creating formalized roles throughout the process, a handbook outlining the steps of the process, and an internal controls checklist for management to complete in order to certify fee- billing validation. In addition, NRC changed the frequency of the billing validation process from a quarterly to a monthly basis. NRC officials we interviewed stated that the biggest changes in the new process are the increased role of management-level personnel throughout the process and the increased frequency of the reviews. NRC enhanced the timeliness of its billing process by implementing an electronic billing system in line with a project plan the agency developed in response to a GAO recommendation. Specifically, in 2018 we reported that NRC was undertaking an initiative to transition to an electronic billing system known as eBilling, but it had not developed planning documents for the initiative. We recommended that NRC develop a project plan for eBilling that would (1) establish plans for schedule and cost, 2) involve licensees in developing system capabilities, and (3) include steps to assess the results of implementing eBilling. Based on our review of NRC’s eBilling documents, NRC implemented these recommendations as part of its planning process. For example, NRC solicited feedback about eBilling usability, organization, content, and functionality from nine licensees it selected for an eBilling pilot. NRC also established plans for schedule and cost and included metrics assessing eBilling on the timeliness of invoices, licensee participation rates, and the accuracy of invoices in its eBilling project plan. As a result, in September 2019, NRC was able to begin distributing electronic invoices through eBilling and sent all licensees receiving service fee invoices an informational brochure giving instructions for how to enroll in the program in October 2019. Six of the 11 licensees we interviewed stated that they anticipated eBilling would improve the timeliness of NRC’s billing process. Figure 4 summarizes some of the key features now available to licensees through eBilling. Since 2017, NRC has implemented changes to its budgeting process that address some but not all of its internal initiatives, prior GAO, and NRC OIG findings and recommendations in two main areas: (1) NRC’s annual budget justification, and (2) NRC’s budget formulation and budget execution systems. Annual budget justification. In 2017, we reported that NRC did not present actual obligation data in its annual budget justifications for fiscal years 2010 through 2017, and without this information, it was difficult for users of the budget justification—including Congress and licensees—to understand how NRC used its appropriations. We also reported that, in spite of an agency initiative to decrease overhead costs, NRC’s obligations for overhead—currently named Corporate Support—increased each year from fiscal year 2011 to 2015 due to increases in rent, utilities, and information technology investments, among other things. As a part of its Fees Transformation Initiative, NRC planned to include additional information on actual obligation data to better enable stakeholders to determine how NRC spent its appropriation. Starting with fiscal year 2018, NRC began presenting actual obligation data in its annual budget justification. NRC data show that the agency had about a 4 percent decrease in actual obligations for Corporate Support from fiscal year 2016 to fiscal year 2019, from $302.9 million to $291.2 million, as shown in table 1. These reductions were a result of NRC’s corporate workload reductions to reflect efficiencies as well as current and projected declines in agency workload, among other things. However, actual obligations for Corporate Support as a percentage of NRC’s total agency-wide obligations increased by about 2.3 percent during this same time period. Specifically, in fiscal year 2016, Corporate Support was about 30.4 percent of total NRC obligations ($302.9 million of $996.6 million), whereas in fiscal year 2019, Corporate Support was about 32.7 percent of total NRC obligations ($291.2 million of $891.5 million). In some years, reductions in Corporate Support were offset by pay increases consistent with federal government-wide guidance and investments in information technology, among other items. In addition, NRC officials said that Corporate Support as a percentage of NRC’s total obligations increased because program resources decreased as NRC’s projected workload declined. In addition to presenting actual obligation data in its annual budget justification, NRC began presenting more detailed information on the status of funds it carried over from previous fiscal years starting in its fiscal year 2018 budget justification. Specifically, NRC began reporting the amounts of carryover funds that were allocated in a given fiscal year and the amounts of these funds available for obligation at the beginning of a fiscal year, as shown in table 2. According to NRC officials, the agency generally allocates carryover funds based on (1) congressional direction to use carryover funds to supplement annual appropriations, and (2) the agency’s discretion in order to address urgent mission needs. In its fiscal year 2018 and 2019 budget justifications, NRC presented this carryover data by appropriation funding category, while it presented the rest of the information in its budget justification by the agency’s business lines. According to NRC officials, the difference in presentation limited the ability of users of the budget justification to understand where these carryover funds were being allocated. In response, in its fiscal year 2020 budget justification, NRC began presenting data on its congressionally- directed carryover funds using the same business lines it used to present the rest of the information in its budget justification. However, NRC did not present its discretionary use of carryover using those business lines. NRC officials told us that they started an initiative to enhance NRC’s carryover tracking process, and that NRC will continue to refine how the agency presents carryover data in future budget justifications. In addition to presenting data in its annual budget justification, NRC included additional information in its budget justifications to increase transparency, in response to NRC’s Fees Transformation Initiative. For example, in its fiscal year 2018 budget justification, NRC included a crosswalk of business lines’ allocation to NRC’s nine fee classes with the goal of helping licensees understand how NRC’s planned workload in its budget formulation impacts licensees’ fees. Budget formulation and execution system. NRC’s OIG reported in 2013 that NRC’s budget formulation process did not align with its budget execution process, and we similarly reported in 2017 that these processes were not aligned from fiscal years 2010 through 2015. NRC used two different systems—one to formulate its budget and another to execute its budget through obligation of funds. The two systems differed in that they used different account structures for NRC’s personnel and other costs. Specifically in 2013, NRC’s OIG found that NRC’s budget formulation and execution processes were not aligned, recommending NRC enforce the use of financial management system codes. In 2017, we reported that there were no specific requirements for an agency’s budget formulation process to align with its execution process, but without this information, it was difficult to track how NRC used its funds in relation to its budget authority. According to NRC officials, the agency has prioritized making system upgrades to address new legislative requirements in the Nuclear Energy Innovation and Modernization Act before fully addressing other challenges with the systems. However, as of December 2019, officials told us that NRC recently began the planning phase of work to address these system challenges, and that NRC plans to implement system upgrades in fiscal year 2020, with a tentative completion date in fiscal year 2021. Furthermore, in a 2017 letter to NRC’s OIG, NRC noted that it had begun updating its systems to address NRC OIG’s 2013 recommendation on financial management system codes. However, the system modifications did not accomplish the entire task, and NRC has established a monthly process to manually reconcile the codes between the two systems while NRC further updates its systems to meet NRC OIG’s recommendation. Since 2017, NRC has made a number of changes to its fee-setting, billing, and budgeting processes in response to GAO, NRC OIG, and internal NRC findings and recommendations, and those changes have improved those processes and addressed some challenges previously raised by licensees. However, additional steps could further enhance NRC’s efforts to improve its billing process. First, NRC program offices do not consistently provide information on the progress of work performed on ongoing regulatory actions. By developing guidance about when NRC staff are to communicate information to licensees on the progress of work performed, NRC could enhance transparency and facilitate planning and budgeting, as licensees would have more information about when an action is expected to be completed or will cost more than NRC’s initial estimate. Second, NRC program offices do not clearly define what costs are included across their public cost estimates for common oversight activities. By doing so, NRC could enhance transparency and increase the value of these estimates as a budgeting and planning tool for stakeholders, consistent with NRC’s Principles of Good Regulation. We are making the following two recommendations to NRC: The Executive Director for Operations of NRC should ensure relevant NRC program offices develop policy and guidance for when to communicate information on work progress to licensees, such as through communications to licensees at specified timeframes or thresholds. (Recommendation 1) The Chief Financial Officer of NRC should, in consultation with NRC program offices, develop guidance to ensure NRC staff clearly define what costs—such as project management—are included in its public cost estimates. (Recommendation 2) We provided a draft of this report to NRC for review and comment. In its comments, reproduced in appendix I, NRC neither agreed nor disagreed with our recommendations but did describe actions that it intends to take in response to our recommendations. NRC stated that it will review its current practice of providing information on work progress to licensees and develop or revise any policy and guidance where necessary. NRC also stated that it will review its current web-based cost estimates to determine if changes are necessary and implement those changes as appropriate. Although further review of NRC’s practices on providing work progress information to licensees and cost estimates could be worthwhile, we believe our review sufficiently demonstrated that by taking additional steps, NRC could further enhance transparency and facilitate planning and budgeting for licensees. As a result, we continue to believe that implementing our recommendations on work progress and cost estimates could further improve NRC’s processes. NRC also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Chairman of NRC, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, David Marroni (Assistant Director), Margaret Childs (Analyst-in-Charge), and Jon Muchin made key contributions to this report. Kevin Bray, Cindy Gilbert, Jessica Lemke, Susan Murphy, Dan Royer, Sheryl Stein, and Doris Yanger made additional contributions.", "summary": "NRC regulates the commercial nuclear industry. In that role, the agency provides services for regulated entities that hold licenses—that is, licensees. NRC recovers the majority of costs for these services by setting fee rates and using those rates to bill licensees. In 2017 and 2018, GAO recommended actions to improve NRC's fee-setting, billing, and budgeting processes, and NRC OIG and internal agency initiatives recommended additional actions. However, industry stakeholders continue to identify challenges with these processes. GAO was asked to review NRC's (1) fee-setting, (2) billing, and (3) budgeting processes. This report examines NRC's progress since 2017 implementing changes to those processes in response to GAO, NRC OIG, and internal agency findings and recommendations. GAO identified relevant GAO, NRC OIG, and internal agency recommendations and evaluated NRC's progress implementing those using evidence such as NRC's fee rules and budget documentation. GAO also spoke with NRC officials and interviewed a non-generalizable sample of NRC licensees, who were selected based on the amount of fees NRC charged them from fiscal years 2014 through 2018. Since 2017, the Nuclear Regulatory Commission (NRC) has implemented changes to its fee-setting, billing, and budgeting processes in response to GAO, the NRC Office of Inspector General (OIG), and internal agency findings and recommendations: Fee-Setting . NRC has improved the clarity, consistency, and transparency of its fee-setting process by, among other things, defining key terms used in the calculation of its hourly-fee rate and by developing and meeting performance measures for the transparency and timeliness of the fee-setting process. Second, NRC did not clearly define what costs are included across all its public cost estimates for common regulatory actions. NRC created the estimates as a transparency measure to assist stakeholders—including licensees and potential applicants—with planning for the costs of future NRC oversight activities. However, NRC did not specify what costs are included across these cost estimates, such as those related to project management. According to GAO's analysis of NRC documents, such costs for some NRC actions can account for about two thirds of total hours billed. By clearly defining the costs in its public cost estimates, NRC could enhance transparency and increase the value of the estimates as a budgeting and planning tool for stakeholders, in accordance with NRC's Principles of Good Regulation . Budgeting . NRC has made some changes to its budgeting process to better enable stakeholders to determine how it spent its appropriation. For example, starting in fiscal year 2018, NRC began presenting actual obligation data and more detailed information on the status of funds it carried over from prior fiscal years in its annual budget justification. GAO recommends that NRC (1) develop guidance on when to communicate work progress information to licensees, and (2) ensure costs are clearly defined in its public cost estimates. NRC neither agreed nor disagreed but plans to review these processes. GAO believes its report supports implementation of these recommendations.", "document_type": "gao"}
{"report": "Within CBP, Border Patrol is responsible for securing U.S. borders and apprehending individuals arriving at the border between ports of entry. Also within CBP, OFO is responsible for inspecting travelers and cargo seeking to enter the United States through ports of entry and encounters or apprehends individuals determined to be inadmissible to the country. Upon apprehension of individuals at or between ports of entry, Border Patrol agents and OFO officers generally decide whether to (1) place apprehended adults and family units into expedited removal proceedings, or (2) initiate full immigration proceedings, according to CBP officials. If agents or officers place individuals into expedited removal proceedings, CBP will transfer them to DHS’s U.S. Immigration and Customs Enforcement (ICE) for longer-term detention (see appendix II for more information on eligibility, screening standards, and possible screening outcomes for credible and reasonable fear cases). Noncitizen adults and family units may make a fear claim in CBP custody at any point after apprehension, and during the pendency of their expedited removal proceedings in ICE custody (see appendix III for data on apprehensions of noncitizens placed into expedited removal who claimed fear of returning to their country, along with other characteristics of their cases). ICE is generally responsible for referring any fear claims to USCIS for a fear screening after individuals enter detention. If USCIS makes a negative determination and the determination is either not reviewed by an immigration judge, because the noncitizen has declined immigration judge review, or, if reviewed, is upheld by a reviewing immigration judge, ICE is then responsible for removing the person from the country. Within USCIS, the Refugee, Asylum, and International Operations Directorate (RAIO) is to provide, among other things, services for people who are fleeing oppression, persecution, or torture or facing urgent humanitarian situations. RAIO is made up of two divisions: the Asylum Division and the International and Refugee Affairs Division. USCIS’s Asylum Division is responsible for, among other responsibilities, adjudicating affirmative asylum applications—that is, claims made at the initiative of the individual who files an application for asylum with USCIS—and screening credible and reasonable fear cases. As of March 2019, USCIS had 546 asylum officers on board and eligible to screen credible and reasonable fear cases (out of 745 authorized positions). Asylum officers screen cases at the Asylum Prescreening Center in Arlington, Virginia, and eight asylum offices nationwide. USCIS established the Asylum Pre-Screening Center in fiscal year 2016 to provide additional support for the credible and reasonable fear caseload. As of April 2019, the Asylum Pre-Screening Center and the Arlington asylum office together had jurisdiction over 27 ICE detention centers across the United States. EOIR is responsible for conducting immigration court proceedings, appellate reviews, and administrative hearings to fairly, expeditiously, and uniformly administer and interpret U.S. immigration laws and regulations. As of September 30, 2019, 442 immigration judges presided over EOIR’s 63 immigration courts nationwide. In addition to removal proceedings, immigration judges also conduct certain other types of hearings, such as the review of negative credible fear determinations. Table 1 provides additional information about DHS’s and DOJ’s roles in the credible and reasonable fear processes. In July 2019, USCIS made several changes to its credible fear screening processes in response to an interim final rule implementing a new mandatory bar to asylum, known as the “third country transit bar.” Under the interim final rule, noncitizens who enter, attempt to enter, or arrive in the United States across the southern land border on or after July 16, 2019, after transiting through at least one country outside their country of citizenship, nationality, or last lawful habitual residence en route to the United States, must be found ineligible for asylum unless they demonstrate that they fall under an exception to the third country transit bar. As shown in table 2, USCIS’s credible and reasonable fear caseloads nearly doubled from fiscal year 2014 (over 56,000 referrals to USCIS) to fiscal year 2018 (almost 109,000 referrals)—the most recent full year of USCIS data available at the time of our analysis. From fiscal year 2014 through the first two quarters of fiscal year 2019, referrals to USCIS for credible fear screenings comprised about 89 percent of all credible and reasonable fear referrals. The number of referrals for credible fear screenings in the first two quarters of fiscal year 2019 alone was larger than the total number of referrals in each of fiscal years 2014 and 2015. Referrals for reasonable fear screenings also increased from fiscal years 2014 to 2018, and comprised between 9 and 15 percent of all referrals during that time period. Appendix III contains additional information on the characteristics of credible and reasonable fear applicants from fiscal year 2014 through March 2019. As shown in figure 1, USCIS asylum officers made positive determinations in about 71 percent of all credible and reasonable fear screenings between fiscal years 2014 and the first two quarters of fiscal year 2019. The remaining credible and reasonable fear screenings were almost evenly divided between negative determinations and administrative closures (approximately 14 percent each) with a small remainder of screenings pending resolution (0.1 percent). Individually, from fiscal year 2014 through the first 2 quarters of fiscal year 2019, USCIS asylum officers made positive determinations in nearly 77 percent of all credible fear screenings; officers made positive determinations in about 30 percent of reasonable fear screenings. Regarding credible fear screenings specifically, the percentage of positive determinations ranged from about 73 to 80 percent of total credible fear cases completed each year from fiscal year 2014 through the first two quarters of fiscal year 2019 (see fig. 2). Regarding reasonable fear screenings, as shown in figure 3, outcomes for reasonable fear cases from fiscal year 2014 through the first two quarters of fiscal year 2019 were generally split evenly each year among positive determinations (from 28 to 32 percent), negative determinations (from 29 to 35 percent), and administrative closures (from 35 to 42 percent). EOIR’s credible and reasonable fear workload increased by about 16 percent—from about 8,100 reviews to about 9,400 reviews each year— between fiscal year 2014 and fiscal year 2018. According to EOIR data, from fiscal year 2014 through the third quarter of 2019 (the most recent data available at the time of our analysis), EOIR’s immigration judges, at the noncitizens’ requests, reviewed about 55,000 cases in which USCIS asylum officers made a negative credible or reasonable fear determination (see figure 4). Approximately 10 percent of these reviews were for individuals detained at the Karnes, Dilley, or Berks family residential centers. As shown in figure 4, immigration judges upheld USCIS’s negative credible and reasonable fear determinations in 77 percent of all reviews judges conducted from fiscal year 2014 through the third quarter of fiscal year 2019. During this time period, immigration judges vacated (or overturned) 22 percent of USCIS’s negative determinations—meaning, judges found that those individuals had a credible or reasonable fear, as appropriate. As a result, individuals found to have a credible fear were to be placed in full removal proceedings and individuals found to have a reasonable fear were to be placed into more limited removal proceedings to consider the applicants’ eligibility for withholding of removal or deferral of removal. Immigration judges upheld 45 percent of USCIS’s negative determinations and vacated 54 percent of USCIS’s negative determinations for individuals in ICE’s Dilley, Karnes, or Berks family residential centers. In addition, EOIR publicly reports data on the outcomes of removal cases across immigration courts that originated with a positive credible fear determination. EOIR reported that, from fiscal years 2014 through March 2019, immigration judges completed about 135,000 cases that began with a positive credible fear determination. Individuals in about 75,800 of the completed removal cases filed applications for asylum (56 percent). In about 59,200 of the completed removal cases (44 percent), individuals did not file an asylum application. However, as previously described, individuals who have received positive credible fear determinations may apply for other forms of relief or protection besides asylum, such as withholding of removal, and those applications are not represented in the statistics on EOIR’s website. Further, EOIR officials told us that, for data reporting purposes, each member of a family who receives a Notice to Appear before an immigration judge is counted as one EOIR removal case and each removal case may or may not include an asylum application. However, for a number of immigration applications before the court, including asylum and the related screening for credible fear, a spouse or child (defined as an unmarried natural or legally adopted child under 21 years of age) may be included as a dependent on a principal’s application and derive lawful immigration status from the principal applicant if the application is granted. As previously discussed, individuals detained in family residential centers—including individuals who could be eligible dependents for credible fear screening and asylum application purposes—comprise a substantial proportion of those who receive positive credible fear determinations. As such, according to EOIR officials, each family member would not be expected to file a separate asylum application. For example, a mother and her two children whose removal cases originated with a positive credible fear screening would comprise three removal cases in EOIR’s publically reported data, but it is likely that only the mother’s case would include an application for asylum, with her children as dependents on that application. For those removal cases in which the noncitizen applied for asylum, immigration judges granted asylum in about 19,300 cases (25 percent of the 75,800 completed removal cases with an asylum application). USCIS has developed various policies and procedures related to managing and overseeing credible and reasonable fear cases in accordance with the regulations governing credible and reasonable fear screenings, including setting requirements for interview procedures, background and security checks, and supervisory review. In particular, USCIS has a Credible Fear Procedures Manual and a Reasonable Fear Procedures Manual that outline the procedures officers are to follow in screening these cases. Interview procedures. As of July 2019, an asylum office is to wait a minimum of one full calendar day from the applicant’s arrival at an ICE detention facility before conducting a credible fear interview; an asylum office is to wait 48 hours after an initial orientation on the reasonable fear process before a reasonable fear interview, according to USCIS policy. However, both credible and reasonable fear interviews generally occur at least 48 hours after the applicant’s arrival at a detention facility, according to USCIS officials. Asylum officers may conduct credible and reasonable fear interviews either in-person or on the phone. Asylum officers are to arrange the assistance of an interpreter, generally connected over the phone, if the applicant is unable to proceed effectively in English pursuant to regulation. Asylum officers are to verify and document that applicants have received and understood information regarding the credible or reasonable fear process before they begin asking substantive questions during the interview about the applicant’s claim. According to USCIS documents and officials, during the interview, asylum officers are to elicit all information relevant to a credible or reasonable fear claim, and regulation requires they conduct interviews in a non- adversarial manner. For example, asylum officers are to ask applicants questions to determine whether they can establish a credible or reasonable fear of persecution based on their race, religion, nationality, membership in a particular social group, or political opinion. In addition, asylum officers are to ask applicants questions to determine whether they can establish a credible or reasonable fear of torture if returned to their home country. During our observations of in-person and telephone interviews, we observed asylum officers asking questions to ensure they fully explored any aspect of the claim related to a protected ground that could result in a positive determination. For example, we observed asylum officers asking applicants separate questions about each protected ground, even if the applicant had not previously expressed they were harmed because of their political beliefs or race. USCIS policy notes the applicant’s credibility is dependent on various factors such as comparing information provided during the interview with that previously provided in the applicant’s sworn statement to Border Patrol or OFO when initially apprehended. If asylum officers identify an issue with the applicant’s credibility, they are to inform the applicant of the concerns and ask the applicant for his or her perspectives. During our site visits, we observed asylum officers questioning applicants on inconsistencies, in a non-adversarial manner, between information provided during the interview as compared to the applicant’s sworn statements to Border Patrol agents upon apprehension. At the end of the interview, asylum officers are to provide a verbal summary of the material facts of the applicant’s claim, and provide an opportunity for the applicant to make any corrections or additions. We observed asylum officers providing such summaries in all but one of the interviews that we observed in full. According to USCIS policy, asylum officers are to record key information about the applicant’s claim, as well as specific details of the determination, on required forms that serve as the official record of the credible or reasonable fear screening. In addition, asylum officers use a “checklist” to record more detailed legal analysis related to the applicant’s claim. Asylum officers also generally type notes during interviews in a question and answer format, capturing each question and follow-up question they ask, and each response the applicant provides. We observed asylum officers documenting interviews in this way during all of the interviews where we observed the asylum officer in person. Background and security checks. USCIS policy requires asylum officers to ensure certain background and security checks are conducted. If security checks or information discovered during the interview raises concerns related to fraud, public safety, or national security, asylum officers are to refer the case to USCIS’s Fraud Detection and National Security Directorate (FDNS) for assistance. FDNS officials told us the short time frames in the credible and reasonable fear process, among other factors, make direct involvement in individual cases less likely than in other caseloads at USCIS, such as affirmative asylum cases. As such, the scope and extent of FDNS investigations into credible and reasonable fear cases is limited relative to other USCIS caseloads. FDNS data indicate that asylum officers referred approximately 1,400 total credible and reasonable fear cases to FDNS between fiscal years 2017 and 2018. Of those, 13 cases resulted in a formal finding, called a Statement of Finding. FDNS officials told us referrals from asylum officers on credible and reasonable fear cases typically result in FDNS conducting research related to an applicant’s criminal history or travel patterns. FDNS may refer this information, in turn, to ICE to reference in the applicant’s removal proceedings, as appropriate. In contrast, according to FDNS officials, a fraud referral in the affirmative asylum context may result in a more formal finding of fraud in a Statement of Finding. Supervisory review. USCIS oversight of credible and reasonable fear cases includes a required supervisory review of each case after an asylum officer makes a positive or negative determination. USCIS officials said supervisors are to review cases for legal sufficiency and accuracy, including a review of the screening checklist and the asylum officer’s supporting interview notes. According to officials, supervisors are to communicate the results of their review to the asylum officer informally (e.g., via email or in-person discussion) for small issues, such as an administrative error, or through a formal write-up for larger issues, such as if the asylum officer’s legal analysis was insufficient and requires a second interview with the applicant. USCIS oversight of credible and reasonable fear cases includes basic training for new asylum officers and ongoing training for incumbent officers at asylum offices; these trainings include information specific to credible fear and reasonable fear screenings. As of the time of our review, the initial training program for asylum officers is comprised of two main components: Distance Training. New asylum officers participate in 3 weeks of self- paced RAIO Directorate and Asylum Division distance training in their respective asylum offices. During distance training, asylum officers are expected to participate in webinars, read the training materials and complete exercises and quizzes in preparation for residential training. The Asylum Division distance training includes course readings on credible and reasonable fear, and observations of credible and reasonable fear interviews. Residential Basic Training. Asylum officers participate in a 6-week residential basic training program, which includes 3 weeks of training in issue areas common across USCIS’s Refugees, Asylum, and International Operations Directorate, as well as three weeks of Asylum Division-specific training. In the first 3-week session, courses include classroom instruction, practical exercises, and interviewing exercises on a variety of topics and skills relevant to multiple areas of USCIS’s work, such as on affirmative asylum and refugee adjudications. The legal topics and skills covered in this initial training include eligibility for asylum, an applicant’s nexus to protected grounds, and eliciting testimony, among others. The second 3-week session focuses on division-specific policy, procedure, and law related to asylum adjudications and screenings. For example, the 3-week session includes training on the affirmative asylum process, and multiple mock affirmative asylum interviews, among others, as well as 2 days of training specific to credible and reasonable fear cases. These 2 days include practical exercises; one mock credible fear interview exercise; and formal presentations on interviewing skills and security checks in a credible fear context, forms required for credible and reasonable fear, and on the Convention against Torture. At the end of the 6-week residential training course, new asylum officers must pass final exams with a score of at least 70 percent. We reviewed a version of the exam and found that it included questions specific to credible and reasonable fear screenings. Asylum Division officials said the 9 combined weeks of distance and residential basic training constitute the minimum amount of formal training required for asylum officers to effectively screen credible and reasonable fear cases. However, Asylum Division officials said it is important for individual asylum offices to provide additional, on-the-job training to new officers assigned to screen credible and reasonable fear cases, specifically. Asylum officers screen credible and reasonable fear cases under shorter time frames and with less corroborating documentation compared to affirmative asylum cases. As such, Asylum Division officials told us that officers accustomed to adjudicating affirmative asylum cases may need to adjust to the shorter time frames required in credible and reasonable fear cases. For example, some asylum offices have developed formal presentations on local policies and procedures, or provide officers with an opportunity to observe other officers conducting credible or reasonable fear interviews and gradually increase the number of cases they screen per day. Given their caseloads, the Houston and Arlington asylum offices provide 3 and 4 weeks of additional credible and reasonable fear training for new asylum officers, respectively. By comparison, the San Francisco and Newark asylum offices provide 1 week of training on credible and reasonable fear procedures for new asylum officers and Los Angeles provides 2 days of such training, according to officials. For incumbent asylum officers, USCIS policy requires asylum offices to allocate four hours per week for formal or informal training. The training can range from classroom instruction by a training officer, to individual study time that asylum officers can use to review case law, research country conditions affecting asylum applicants, or read new USCIS procedures and guidance. Individual asylum offices design their weekly training programs based on the types of cases their office generally receives, according to Asylum Division officials. The Asylum Division requires training officers to track the date and topic of each weekly training session and report that information to Asylum Division headquarters on a quarterly basis. Our analysis of fiscal year 2018 quarterly training reports for all asylum offices and sub-offices indicates that offices with larger credible and reasonable fear caseloads generally provided more weekly trainings on these topics. For example, Houston and Arlington conducted seven or more weekly training sessions on credible and reasonable fear screenings in fiscal year 2018. By comparison, two offices with smaller credible and reasonable fear caseloads—Newark and New York—conducted one or fewer weekly sessions on credible and reasonable fear (see app. III for credible and reasonable fear workload data by asylum office). In addition to this training program for asylum officers, USCIS trains officers from outside the Asylum Division to screen credible and reasonable fear cases, including refugee officers and others. Refugee officers receive some of the same basic training as asylum officers, as they participate in the same RAIO distance training and RAIO Directorate residential training. Refugee officers do not participate in Asylum Division distance training or residential training. As a result, USCIS provides refugee officers with 3 days of training on screening credible fear cases before they can begin screening cases. We reviewed training materials for the refugee officer training, and found the sessions are similar to Asylum Division residential training sessions on credible fear screening. In addition, some materials provide information and guidance on the differences between adjudicating refugee cases and screening credible fear cases. Officials said refugee officers generally screen credible fear cases, including at the family residential centers, only if they are detailed to the Houston and Arlington asylum offices. Both Houston and Arlington provide refugee officers detailed to their offices with 1-2 weeks of additional training on credible fear screening, similar to the procedural training they provide to new asylum officers. At both offices, trainings include formal presentations or exercises on legal concepts and procedures specific to credible fear, credible fear interview observations, and a gradual increase in the number of cases refugee officers screen each day. Although all new asylum officers receive basic training on the credible and reasonable fear screening process and may also receive on-the-job training in their home offices, not all offices provide additional pre- departure training to asylum officers before they begin screening cases for family units at ICE family residential centers. Credible fear screenings at ICE’s family residential centers, in particular, represent a significant percentage—about 34 percent—of all credible fear cases asylum officers screened from fiscal year 2014 though the second quarter of fiscal year 2019. As discussed previously, asylum offices with relatively small credible and reasonable fear local caseloads generally provide less on- the-job training throughout the year on credible and reasonable fear. However, almost all asylum offices send officers to the family residential centers in Texas for in-person interviews, including those offices with small credible and reasonable fear caseloads at the local level. Asylum Division officials said they require asylum offices to send a specific number of asylum officers—a number in proportion to the size of the office—with the largest offices sending the most officers to the family residential centers each year. For example, in fiscal year 2018, Newark, Los Angeles, Houston, and Chicago sent the most officers to the family residential centers, as shown in figure 5 below. At least two asylum offices provide pre-departure training to asylum officers being sent to ICE’s family residential centers. To support officers who are more accustomed to adjudicating affirmative asylum cases, the Los Angeles asylum office provides pre-departure training for officers before they travel to the family residential centers. In Los Angeles, officers observe credible and reasonable fear interviews and gradually increase to a full caseload of credible or reasonable fear cases at their home office, according to officials. In San Francisco, officers receive pre- departure training highlighting procedures unique to the family residential centers or to processing family units in credible and reasonable fear. Specifically, San Francisco pre-departure training includes a formal presentation on family residential center procedures, including discussion of challenges officers may experience, according to officials. By comparison, officials from the Chicago and New York offices told us they do not provide formal pre-departure training but rather direct or recommend that officers review Asylum Division guidance and procedures on family processing independently before they travel. Officials from two other offices told us they rely on the training asylum officers may receive throughout the year related to credible and reasonable fear, which can vary, as previously discussed. Asylum officers also noted inconsistent pre-departure training prior to their temporary duty during our February 2019 visits to two Texas family residential centers in Dilley and Karnes. For example, some asylum officers we interviewed said they screened credible and reasonable fear cases at their home office in preparation for their assignment. Others said they reviewed procedures independently on family processing in credible and reasonable fear cases. Officers from one asylum office said they relied primarily on an email from USCIS support staff located at the family residential centers to learn about screening family cases. Asylum officers are to review the procedures on family processing in credible and reasonable fear before they arrive at the family residential centers, according to officials. However, there is no minimum amount of pre-departure training, or required content for such training, that all asylum offices are to provide before officers begin screening family units. Asylum Division officials acknowledged that training on screening of family units for credible and reasonable fear varies by asylum office and noted that offices have been given discretion to determine what, if any, pre-departure training to provide on screening family units. Arlington and Houston asylum office officials stated that inconsistent asylum officer training on credible and reasonable fear cases negatively impacts efficiency at the family residential centers. Specifically, these officials noted that asylum officers who typically adjudicate affirmative asylum applications benefit from training on key differences between credible fear, reasonable fear, and affirmative asylum. For example, officials said more training could reduce administrative errors in applicants’ paperwork, and changes needed during supervisory review, both of which occur more often for officers with less experience and training, according to officials. Further, officials said asylum officers less experienced in credible and reasonable fear may not be able to handle a full caseload at the family residential centers when they first arrive. As a result, Houston officials said they may spend the first week of a 2-week assignment providing additional support to inexperienced officers as they gradually increase to a full caseload. Standards for Internal Control in the Federal Government states management should demonstrate a commitment to recruit, develop, and retain competent individuals. The standards also note that competence is the qualification to carry out assigned responsibilities, and requires relevant knowledge, skills, and abilities, which are gained largely from professional experience, training, and certifications. As previously noted, Asylum Division officials told us additional training for asylum officers before they begin screening cases at the family residential centers is important. Officials also said their intention is to balance such training against the need for rapid deployment, in some cases. Although additional training may not be feasible before every deployment, providing asylum officers additional pre-departure training before they begin screening credible and reasonable fear cases for family units would better prepare officers and help ensure efficient and effective case processing at ICE’s family residential centers. USCIS relies primarily on two quality assurance reviews for assessing quality of credible and reasonable fear cases, but does not document the results of one of these reviews in a consistent manner. Annual, Asylum Division-wide reviews. The Asylum Division conducts division-wide quality assurance reviews on a random sampling of credible fear, reasonable fear, or affirmative asylum cases selected proportionally from asylum offices nationwide. To do so, the Asylum Division works in collaboration with the RAIO Directorate. The reviews occur each year, and rotate between a sampling of credible fear cases, reasonable fear cases, and affirmative asylum cases. For credible and reasonable fear reviews, USCIS randomly selects a specified number of cases after supervisory review, but before officers serve determinations to the applicant. Reviewers use a checklist to identify and track quality issues arising in each reviewed case, such as accurate data entry, appropriate legal analysis, asylum officer notes that reflect a skilled interview, and others. The review process for each case includes two lines of review. If the two reviewers come to different conclusions, they discuss any differences in their reviews and reach consensus about how to score the case. USCIS records the results of these reviews in a document that lays out the numbers and percentages of errors in areas covered in the review checklist. For example, for the 2018 review of credible fear cases, the document states asylum officer notes did not reflect a skilled interview in an estimated 58 percent of cases. For most of these cases, the reason for the error that reviewers noted was insufficient follow-up questions. USCIS officials said while the sample across asylum offices is generalizable with respect to the credible or reasonable fear caseloads nationwide, the samples taken from each asylum office are not large enough to draw conclusions about trends at individual asylum offices. Officials said they rely on periodic reviews to identify trends by asylum office. Periodic, asylum office reviews. In addition to the Asylum Division-wide quality assurance reviews, the Asylum Division began conducting periodic reviews at asylum offices in November 2017. As of November 2019, the Asylum Division had conducted periodic reviews of credible and reasonable fear cases or affirmative asylum adjudications at some asylum offices, as well as a review of credible and reasonable fear cases at the family residential centers. For periodic reviews, the Asylum Division selects cases over a period of several weeks. For example, based on the Asylum Division’s draft standard operating procedures for the periodic reviews, an asylum office may send two to four credible and reasonable fear cases every day for several weeks to reach the required total number of cases. According to the draft standard operating procedures, Asylum Division reviewers are to use a reviewer checklist, modeled off the checklist used for the Asylum Division-wide reviews, as a starting point for what factors to review. However, USCIS does not document the results of the periodic reviews in a consistent manner. We reviewed the reports resulting from six periodic reviews conducted at the Arlington, Chicago, Miami, and Houston asylum offices, the New Orleans sub-office, and the family residential centers. We found that all reports included information on strengths and weaknesses, and some reports further organized analysis into additional categories. For example, some reports had analysis on details related to procedures, eliciting testimony, and issues related to fraud detection and national security. Other reports included analysis of specific trends in persecution cases and in Convention against Torture cases. Some reports also included analysis on legal sufficiency, applicant country of origin, determination outcomes, and others. According to the draft standard operating procedure, reviewers are to note trends, common errors, and collect samples to create a deliverable, such as a short report or other deliverables, for the asylum office at the end of the review. However, the Asylum Division has not provided guidance on what specific information is important to include in reports resulting from periodic reviews in order to track trends within an asylum office over time, or across asylum offices. Standards for Internal Control in the Federal Government states management should establish and operate monitoring activities to oversee the internal control system and evaluate the results. Management should document the results of ongoing monitoring and separate evaluations to identify internal control issues, and should use this evaluation to determine the effectiveness of the internal control system. Asylum Division officials told us the primary purpose of the periodic reviews is to collect information about current, office-specific trends, and provide timely support in the form of training sessions and other guidance. Further, the Asylum Division historically has not used the periodic reviews to compare one office to another, though they have sometimes noted issues from these reviews requiring similar guidance across multiple offices. Documenting the results of periodic reviews in a consistent manner would help the Asylum Division identify trends and provide support across asylum offices. The draft standard operating procedures for the periodic review provides general directions for reviewers to share information on trends to asylum office personnel, such as strengths, weaknesses, and other developing trends. However, the draft standard operating procedures do not specify requirements for documenting the results of these reviews. Asylum Division officials told us the periodic review standard operating procedures are in draft form, and that they may provide more specific guidance on aspects of the reviews in the future. However, officials also said they are not planning any changes or additions to the standard operating procedure as of September 2019. More specific guidance on requirements for documenting results would better position USCIS to track trends in a consistent manner for credible and reasonable fear reviews within and across asylum offices. By regulation, dependents, specifically a spouse or child, of a noncitizen (referred to as the “principal applicant”) can be included in the applicant’s credible fear determination if the dependent (1) arrived in the United States concurrently with the principal applicant, and (2) desires to be included in the principal applicant’s determination. However, any noncitizen may have his or her credible fear determination made separately, if he or she expresses such a desire. USCIS policy is to include any dependents on a principal applicant’s credible fear determination if the principal applicant receives a positive determination, resulting in both the principal applicant and any dependents being issued a Notice to Appear for full removal proceedings. For example, USCIS may process credible fear cases together for family units detained at ICE’s family residential centers, including children as dependents on a parent’s case, or issuing a Notice To Appear for the parent and children in the interest of family unity (see figure 6). We observed asylum officers at the family residential centers asking principal applicants whether they were apprehended with any family members. If yes, asylum officers asked for the names and dates of birth of those family members, and recorded the information in their typed notes. For parents who received a positive determination, we observed asylum officers including the child on the parent’s case as a dependent. Further, if a parent receives a negative credible fear determination, and his or her child receives a positive credible fear determination, USCIS may issue a Notice to Appear to the child as a positive credible fear determination and to the parent in the interest of family unity. In that case, because a parent could not be a “dependent” of a child under the regulation, USCIS policy is to use its discretion to issue a Notice to Appear to both the child receiving a positive determination and the parent he or she arrived with, in the interest of family unity, even though the parent initially received a negative determination. Issuing the parent and child a Notice to Appear places them into full removal proceedings where they can apply for multiple forms of relief or protection before an immigration judge, including asylum, rather than being expeditiously ordered removed in accordance with the expedited removal process. The exercise of this discretion to issue a Notice to Appear to both the child receiving a positive determination and the parent he or she arrived with in the interest of family unity is limited to cases in which the children are under the age of 18 because, according to the policy, family unity interests are more compelling when the child is a minor. USCIS data indicate that asylum officers screened more than 141,000 credible fear cases at ICE’s four family residential centers between fiscal years 2014 and the first two quarters of 2019 (see table 3). In addition, USCIS data indicate that positive credible fear determination rates are higher at the family residential centers—87 percent compared with the nationwide rate of 77 percent from fiscal year 2014 through the second quarter of fiscal year 2019 (see app. III for data on reasonable fear cases screened at ICE’s family residential centers). Asylum officers are to record individual case outcomes for all family members in USCIS’s automated case management system. However, Asylum Division officials said their system does not allow asylum officers to record whether an individual receives a credible fear determination as a principal applicant, dependent, or in the interest of family unity. Instead, asylum officers are to record positive determinations in the USCIS case management system for both (1) dependents on the basis of the principal applicant’s positive case, and (2) parents with negative determinations, on the basis of their child’s positive case. USCIS does record more specific information related to outcomes for family units in the family members’ individual hardcopy alien files, but this information is not readily available in an automated manner. USCIS’s case management system allows asylum officers to record family relationships—that is, officials stated that asylum officers are to record who is a principal applicant, and who is a spouse, child, parent, or sibling of the principal applicant. According to USCIS officials, asylum officers are to record a parent who receives a positive credible fear determination as the principal applicant, and record any children as a child. Further, asylum officers are to link known family members’ cases in the system, but adding a description of the family relationship is up to asylum officers’ discretion. However, the system does not allow officers to record whether an applicant’s determination stems from his or her own case, or from a family member’s case. As a result, USCIS does not maintain automated data in a readily accessible manner on outcomes for family members in a manner that indicates whether (1) an eligible family member received a positive determination as a dependent on a principal applicant’s positive case or (2) whether a parent was issued a Notice to Appear based on his or her child’s positive determination, after the parent received a negative determination. Standards for Internal Control in the Federal Government states that management should process obtained data into quality information that supports the internal control system. Quality information is appropriate, current, complete, accurate, accessible, and provided on a timely basis. Although USCIS data indicate that positive credible fear determination rates are higher at the family residential centers compared to rates of positive credible fear determinations across all detention facilities, USCIS officials stated the higher rates result from the ability to (1) include children under 18 on a parent’s positive credible fear determination, and (2) record all family members as positive in the system when USCIS uses its discretion in the interest of family unity. USCIS officials told us that systematically recording all outcomes of credible fear screenings for family members in a more complete manner would require changes to their case management system; according to Asylum Division officials, they are continually exploring options to improve the system’s capabilities. Without complete data in its case management system on all outcomes of credible fear screenings at family residential centers, USCIS is not well-positioned to report on the scope of either the agency’s policy for family members who are treated as dependents, pursuant to regulation, or USCIS’s use of discretion in the interest of family unity. USCIS manages its credible and reasonable fear workloads using national- and local-level staffing models to inform staffing allocation decisions. Specifically, USCIS has an agency-wide staffing model to allocate staff to different workload categories, including credible and reasonable fear workloads, for each upcoming fiscal year. Asylum Division headquarters officials stated they collaborate with USCIS’s Office of Performance and Quality, USCIS administrative offices, and local asylum offices to develop the national staffing model. Headquarters officials said the national staffing model is intended to allocate staff for each workload category for the upcoming fiscal year. They begin working on the staffing model in June for any given year in anticipation of resource decisions the agency will make before a new fiscal year begins, usually in September or October. According to officials, the national staffing model for credible and reasonable fear is based on historical case receipt and workload data, historical staffing data and future staffing workload forecasting data, bi-weekly reports on staffing and workload data, and observations from asylum offices submitted to headquarters, among other things. The Asylum Division also maintains staffing models that guide local staffing deployment, according to headquarters officials. Asylum offices make local staffing allocation decisions in collaboration with the Asylum Division at headquarters. Headquarters officials stated they consider several factors in allocating staff specifically for the credible and reasonable fear workloads for local asylum offices. Such factors include workload projections, available facilities and planned facilities projects, and existing workforce and vacancy levels (table 4 shows the number of asylum officers authorized and on board for each asylum office in March 2019). In addition, headquarters officials said they work with local asylum offices to review the number of credible and reasonable fear case receipts and current staffing allocations by asylum office on a daily basis. Headquarters officials stated they change staffing allocation as necessary to address changes in credible or reasonable fear case receipts. In the Houston and Arlington asylum offices, in particular, officials stated they review headquarters data on workload projections to assign personnel to the credible and reasonable fear workloads for their offices. For example, officials in the Houston office said they look at the projected credible and reasonable fear workload to determine the number of officers they may need. Houston office officials said they assign officers based on officer availability, considering factors such as leave or training schedules. Similarly, a senior official responsible for staffing in the Arlington office said they look at the projected numbers of credible and reasonable fear cases, as well as the location of the cases, to determine the target number of officers assigned to a specific workload. Once they set targets for the number of officers needed, a senior official responsible for staffing in the Arlington office said they assign personnel based on a number of factors, including officer preferences, seniority, and locations with the greatest need. Although USCIS uses national and local staffing models for determining staffing needs and allocating staff at and across field offices, senior Asylum Division officials stated that predicting future workload for credible fear cases is challenging. Moreover, headquarters officials told us that USCIS’s credible fear workload projections have been off by as much as 50 percent when comparing projected and actual credible fear workload volume in recent years. For example, the USCIS projections for credible fear cases in fiscal year 2015 were 78,485, but actual case receipts totaled 48,052. More recently in fiscal year 2018, USCIS projected 70,000 credible fear case receipts, but actual case receipts totaled 99,035. Headquarters officials stated that a variety of external factors— unpredictable changes in country conditions, and CBP and ICE decisions to either place individuals in expedited removal or issue Notices to Appear before an immigration judge—make it difficult to project this workload. Furthermore, headquarters officials stated the volume of credible fear cases can fluctuate on a weekly basis, while reasonable fear projections have been fairly accurate, since the number of reasonable fear cases has remained relatively stable in recent years. To manage its workload of credible and reasonable fear cases, USCIS relies on a flexible workforce to respond to fluctuations in cases, in addition to asylum officers who generally screen credible and reasonable fear cases. For example, USCIS pulls asylum officers from affirmative asylum adjudications and uses overtime hours to handle surges in credible and reasonable fear case receipts, according to officials. Senior Asylum Division officials stated they do not receive staffing increases to account for lost or stopped work in other workload categories, such as affirmative asylum, that result from surges in credible fear case receipts. They stated surges in credible and reasonable fear case receipts may require immediate staff redeployment from the affirmative asylum workload. As a result, asylum offices have sometimes canceled planned affirmative asylum interviews and have prioritized credible and reasonable fear screenings over affirmative asylum cases, which have significantly contributed to the current backlog in pending affirmative asylum cases, according to headquarters officials. As previously discussed, asylum offices across the country also send officers on details to ICE’s family residential centers to conduct credible and reasonable fear screenings. In addition, the Asylum Division headquarters tracks the number of asylum officers assigned to the credible and reasonable fear workload, among other workload categories such as affirmative asylum, through biweekly reports received from local asylum offices. Headquarters officials told us they use the reports to respond to specific requests for information about Asylum Division staffing allocation. For example, Congress may request information on the number of USCIS personnel working on credible fear cases for a particular time period, according to headquarters officials, so they maintain these reports to fulfill such requests. Specifically, with regard to the biweekly reports, asylum offices record the number of asylum officers assigned to credible and reasonable fear cases for each day in the 2-week pay period. The resulting biweekly reports are spreadsheets with 15 tabs, one tab for each day in a pay period and one tab summarizing the pay period, with 26 separate spreadsheets for each year per asylum office. Headquarters officials stated the biweekly reports are manually compiled and may contain errors, but the biweekly reports have historically provided the overall number of personnel performing credible and reasonable fear work for any particular date or pay period. As of October 2019, headquarters officials said they are developing automated software that will track information similar to that collected in the biweekly reports, which will allow more systematic analysis of the staffing data that the current biweekly reports contain. USCIS sets and monitors timeliness goals for completing credible and reasonable fear cases. Monitoring timeliness goals for credible fear cases. USCIS monitors credible fear processing times by setting timeliness goals for completing credible fear cases and those goals have changed over time. USCIS regulation does not require that credible fear cases be completed in a specific time frame; however, Asylum Division headquarters officials said they have used timeliness goals to help monitor their credible fear workload. In addition, case delays may occur for credible fear cases (discussed further below). Specifically, from fiscal year 2009 through the first quarter of fiscal year 2018, USCIS used a 14-day goal to monitor credible fear case processing times. In other words, USCIS monitored the extent to which officers completed credible fear cases within 14 calendar days of USCIS receiving referral documents from ICE and created an electronic file for the case in their case management system. According to our analysis, USCIS completed at least 81 percent of credible fear cases in 14 or fewer days for each fiscal year from 2014 to 2017—the last full fiscal year under the 14-day goal (see table 5). In February 2018, USCIS lowered its credible fear processing time goal to 10 days. USCIS completed 68 percent of credible fear cases in 10 or fewer days between February and September 2018. Monitoring timeliness requirements for reasonable fear cases. USCIS monitors reasonable fear processing times by setting a 10-day goal. Pursuant to regulation, asylum officers are to conduct reasonable fear interviews and make a determination within 10 days of receiving a referral from CBP or ICE with an indication that the individual has made a fear claim, absent exceptional circumstances. Additionally, a 2015 settlement agreement in the Alfaro-Garcia v. Johnson case (“Alfaro- Garcia” Settlement Agreement) requires USCIS to achieve an average national reasonable fear determination period of no more than 10 court days (i.e. business days), calculated on a monthly basis, for cases in which individuals are detained by DHS. For reasonable fear cases subject to this settlement agreement that take longer than 20 court days to complete, asylum offices are to notify the Chief of the Asylum Division in writing and provide an explanation for the delay. Further, USCIS must provide class counsel in the Alfaro-Garcia case a notice and remedial plan of action for cases that exceed 20 days that are subject to that settlement agreement. Consistent with USCIS policy and the Alfaro-Garcia settlement agreement, officers may pause the clock for reasonable fear cases— and thus case processing times—in the following limited circumstances: the applicant or the applicant’s representative requests to defer the reasonable fear interview; the applicant refuses to participate in the reasonable fear interview or accept service of a reasonable fear determination; or exceptional circumstances. USCIS pauses processing times for detained reasonable fear cases by recording the dates when the case was paused and when processing resumed, once the basis for pausing the clock no longer exists. Asylum Division headquarters officials said pauses in reasonable fear case processing times are separate from case delay reasons, but case delays may occur for reasonable fear cases. In our case processing time analysis of USCIS data, we excluded approximately 13 percent of reasonable fear cases that had at least one pause in case processing time from our analysis because, in conducting our analysis, we could not systematically confirm the appropriate order of dates for those cases. As shown in table 6, our review of USCIS data for cases that did not include pauses found that USCIS completed at least 91 percent of reasonable fear cases within 10 or fewer court days from fiscal year 2016 to the second quarter of fiscal year 2019. Although the Asylum Division monitors overall processing times for credible and reasonable fear cases, it does not collect comprehensive data in its case management system on some types of case delays. For example, USCIS tracks whether cases are delayed for certain reasons related to the individual—such as if he or she has a medical condition that prevents the asylum officer from conducting the interview, if the individual requests that the interview be rescheduled, or if the individual is detained in a remote location. In addition, USCIS’s system can track if cases are delayed for logistical or resource constraints. Specifically, asylum officers may select “lack of resources” as one case delay reason in the system. However, this field in the system does not allow officers to distinguish more specific types of delays—such as a lack of space in detention facilities for officers to screen fear cases, telephones not working properly, and other types of delays—which officers told us occur on a regular basis. Asylum officers we interviewed in the Arlington and Houston offices stated that logistical delays could affect the number of credible or reasonable fear cases they can complete each day. Specifically, some asylum officers said they have experienced delays up to 30 minutes waiting for phone lines to work properly at detention facilities. Moreover, supervisors we interviewed in the Arlington office stated telephone and interpreter delays could add 20 or 30 minutes per case, resulting in a cumulative delay that could affect an officer’s productivity for any given day. Moreover, supervisors in the Arlington office said it is challenging to identify the appropriate number of cases to assign to officers because the number depends on whether or not disruptions occur. Asylum officers in Arlington said they are expected to conduct a certain number of credible or reasonable fear screenings per day, but expectations for completing their assigned cases may be tempered by circumstances such as interpreter availability or if there are issues at the detention facility, including physical space shortfalls or difficulty in locating the individual at the facility. Similarly, asylum officers in Houston said they are expected to complete a certain number of credible or reasonable fear cases per day, but supervisors understand that they may face logistical challenges such as interpreter or telephone issues. In addition to system limitations in tracking case delay reasons, Asylum Division headquarters officials said their case management system does not have the capability to track how long case delays may last. Our analysis of USCIS data from fiscal year 2014 through the second quarter of fiscal year 2019 indicates that 21,528 credible fear cases and 6,724 reasonable fear cases had delays. USCIS’s system can calculate the number of days for each credible and reasonable fear case—in other words, the total processing time for each case—and the system can produce daily reports noting these overall processing times. However, officials in the Houston office told us they must investigate individual cases on an ad hoc basis to understand how long cases have been delayed during processing. Specifically, officials in the Houston office said they maintain weekly “late reports” using information from USCIS’s case management system that show pending credible and reasonable fear cases with the longest processing times and that they must spend time researching cases on the report to determine the length of the delays. Standards for Internal Control in the Federal Government state that management should obtain data on a timely basis so that they can be used for effective monitoring. These standards also state that management should process the obtained data into quality information that supports the internal control system. As previously discussed, USCIS’s case management system does not track specific logistical reasons for any delays in credible and reasonable fear cases, which affect the number of cases an officer can complete in a day. Furthermore, USCIS’s system can calculate the number of processing days for each credible and reasonable fear case. However, the system cannot track how long a case delay lasts. Headquarters officials said they evaluate the usefulness of their system, and consider options for improvements or changes, on an ongoing basis. However, as of October 2019, they stated they did not have plans for significant changes to the system to track more specific case delay reasons. Collecting additional information in its automated case management system on case delays would provide USCIS with more readily available information and analyzing such data could help USCIS identify case delay reasons relevant in the current environment for officers conducting fear screenings and better position USCIS to mitigate the reasons for the delays and improve efficiency in case processing. EOIR has developed processes for immigration courts and judges to help manage its workload related to credible and reasonable fear reviews. As previously discussed, in the event of a negative outcome of their credible or reasonable fear screening, noncitizens can request a review of USCIS’s negative determination by an immigration judge. The Immigration and Nationality Act, as amended, and regulation require that such reviews occur within certain time frames. Specifically, immigration judge reviews of negative credible fear determinations are to be conducted no later than 7 days after referral from USCIS, to the maximum extent practicable, and immigration judge reviews of negative reasonable fear determinations are to be conducted within 10 days of referral, in the absence of exceptional circumstances. EOIR officials told us that increased resources, beginning in fiscal year 2015, and a faster process for hiring immigration judges have allowed EOIR to increase the number of immigration judges. As of September 30, 2019, EOIR reported that it had 442 immigration judges on board, including 173 judges hired in fiscal year 2018 and fiscal year 2019. EOIR reports that the number of immigration judges has increased each year from fiscal year 2015 through fiscal year 2019. EOIR officials told us that they plan to hire an additional 100 judges in fiscal year 2020. Additionally, EOIR officials told us that they prioritize credible and reasonable fear reviews and that these reviews can generally be accommodated within EOIR’s existing resources—specifically, by finding efficiencies within judges’ existing schedules to add credible or reasonable fear review hearings or by conducting hearings via video teleconferencing (VTC). EOIR officials also said that credible and reasonable fear reviews for individuals in ICE’s family residential centers comprise a small portion of EOIR’s overall workload. According to EOIR officials, each ICE detention facility is assigned to the jurisdiction of an immigration court, and the workload for credible and reasonable fear reviews is managed locally by the court to which each detention facility is assigned. ICE officers are to initiate the immigration judge’s review by filing a request with the appropriate immigration court. Some courts are co-located with ICE detention facilities in which the detainee requesting the credible or reasonable fear review is housed. EOIR officials said that reviews in those locations are typically heard in person by immigration judges assigned to that facility, and that the court finds room in the judge’s regular calendar to hear credible and reasonable fear reviews. For individuals in detention facilities without a co-located immigration court, including ICE family residential centers, immigration judges typically conduct credible and reasonable fear reviews via VTC. Judges conducting credible or reasonable fear reviews via VTC may be located in any immigration court in the United States. According to EOIR officials, the Assistant Chief Immigration Judge for each court is responsible for managing the court’s workload, including seeking support from judges outside the court in circumstances where there are too many cases for the court’s assigned judges. EOIR officials told us that the use of VTC technology—which is available in all courtrooms—provides flexibility to the courts in balancing workloads related to credible and reasonable fear reviews, among other workloads. In addition, EOIR officials stated that judges’ credible and reasonable fear workload is impacted, in particular, by immigration enforcement priorities and USCIS credible or reasonable fear determinations. For example, if DHS places more noncitizens into expedited removal proceedings who subsequently express fear or intent to apply for asylum, EOIR’s related workload might increase. In addition, because immigration judges do not review USCIS’s positive credible fear determinations, if USCIS’s screenings result in more negative determinations, EOIR’s caseload related to credible or reasonable fear reviews might increase. As of January 2018, EOIR has performance measures that include timeliness goals for credible and reasonable fear reviews, and these timeliness goals align with the required credible and reasonable fear review time frames. However, EOIR data we reviewed indicate that about 30 percent of credible and reasonable fear reviews are not completed within the required time frames. Specifically, EOIR’s memorandum on Case Priorities and Immigration Court Performance Measures states that 100 percent of credible fear reviews should be completed within seven days of an asylum officer’s negative determination and that 100 percent of reasonable fear reviews should be completed within 10 days of the filing of a negative reasonable fear determination. Further, according to EOIR officials, courts are to assign credible and reasonable fear reviews to a judge within 48 hours of receipt of the request from ICE, and immigration judges are to complete such reviews within 24 hours after they are assigned. EOIR officials said their automated case management system maintains data on the date when courts receive a request from ICE for an immigration judge review, the date the review is assigned to a judge, and the date the review takes place. EOIR headquarters officials told us that they monitor the extent to which judges are completing reviews within 24 hours after they are assigned using an automated immigration judge performance dashboard, which allows officials to review this performance measure for all judges combined, for individual courts, or for individual judges. Further, EOIR officials told us that if courts are scheduling credible and reasonable fear reviews within 48 hours after receipt and judges are completing reviews within 24 hours after they are assigned, they expect that EOIR should be meeting the required time frames (7 days after ICE’s referral for credible fear reviews and 10 days after ICE’s referral for reasonable fear review) for conducting credible and reasonable fear reviews. EOIR data we reviewed indicate that, from fiscal year 2014 through June 2019, approximately 28 percent of credible fear and 36 percent of reasonable fear reviews exceeded the required time frames, as shown in table 7 below. As previously discussed, the Immigration and Nationality Act and regulation allow for some flexibility with regard to the required credible and reasonable fear review time frames. Specifically, credible fear reviews are to be completed within 7 days, to the maximum extent practicable, and reasonable fear reviews are to be completed within 10 days, absent exceptional circumstances. EOIR officials we spoke with said there are a variety of court, judge, or applicant-related reasons that reviews could exceed the required time frames. For example, case file documentation sent from USCIS to the court may be incomplete. Further, a detention facility may have a medical quarantine that restricts court proceedings for a certain period of time. EOIR headquarters officials told us that, as of October 2019, they review weekly reports that include the median processing times for completed credible and reasonable fear reviews. For example, according to one weekly report from October 2019, the median completion time for credible fear reviews was 7 days. These reports also include information about the average and median number of days pending per case, for those credible and reasonable fear reviews that are not complete. For example, the weekly report we reviewed from October 2019 showed that EOIR had 553 pending credible fear reviews that week, with a median of 7 days pending and an average of 18 days pending. While these weekly reports allow EOIR headquarters officials to monitor some information about their credible and reasonable fear workload, they do not provide information to EOIR officials about the proportion of EOIR’s credible and reasonable fear reviews that are completed within the required time frames, or whether any reviews are delayed for reasons within the limits set out in the law or regulation. EOIR officials said they plan to implement an automated court operations dashboard in early 2020 which is to, among other things, monitor court performance against the performance goals EOIR established in January 2018, including the credible and reasonable fear performance goals. This automated dashboard is to be similar to the immigration judge performance dashboard, which EOIR implemented in early 2019. According to EOIR, the court operations dashboard is intended to operationalize EOIR’s performance measures—including completion of 100 percent of credible fear reviews with 7 days and 100 percent of reasonable fear reviews within 10 days—by providing court staff with daily alerts and warning notices to help court administrators prioritize the scheduling of cases based on the performance measures. This prioritization, combined with EOIR’s monitoring of judge performance to ensure that credible and reasonable fear reviews are completed within 24 hours after they are scheduled, should provide EOIR officials with sufficient information to monitor EOIR’s adherence to the required credible and reasonable fear review time frames. Because implementation of the court operations dashboard is planned for early 2020, it is too soon to know if EOIR will use the dashboard to monitor adherence to the required credible and reasonable fear review time frames or if it will help EOIR understand reasons for delays in those cases that take longer than 7 or 10 days. The number of credible and reasonable fear cases has increased since fiscal year 2014, and USCIS policies and procedures require completion of those cases within short time frames. The Asylum Division provides training for credible and reasonable fear cases to new asylum officers in basic training, given the differences between these screenings and affirmative asylum adjudications. However, not all offices provide additional training on screening such cases at the family residential centers. Ensuring that all asylum offices provide such training, in addition to basic training for new officers, would better prepare them to screen those cases efficiently and effectively. In addition, USCIS relies on its periodic quality assurance reviews to assess the quality of credible and reasonable fear cases across asylum offices. Developing and implementing more specific guidance on requirements for documenting the results of its periodic quality assurance reviews would better position the agency to track trends for credible and reasonable fear reviews across asylum offices. USCIS data show that positive credible fear determination rates are higher at the family residential centers than they are nationwide, in part because USCIS’s automated case management system does not track whether an individual receives a credible fear determination as a principal applicant, dependent, or in the interest of family unity. Without systematically recording credible fear determinations involving family members, USCIS may not have complete data on credible fear determination rates, and the agency may not be in a position to report on the scope of its policy for family members in the credible fear process. Asylum officers have experienced logistical delays that can affect the number of credible and reasonable fear cases they complete each day. Although USCIS tracks some of these delays in its case management system, the system does not distinguish between specific reasons for logistical case delays, such as telephones nor working properly or lack of space at detention facilities for officers to screen cases. Furthermore, USCIS’s system can calculate the number of processing days for each credible and reasonable fear case. However, the system cannot track how long case delays last. By collecting and analyzing additional information on case delays, including specific reasons for delays and how long they last, USCIS can identify relevant case delays for officers conducting fear screenings. Moreover, analyzing specific case delay information could help USCIS mitigate reasons for case delays and improve efficiency in case processing. We are making the following four recommendations to USCIS: The Director of USCIS should ensure that, in addition to USCIS’s basic asylum officer training, all asylum offices provide pre-departure training on the credible and reasonable fear processes before their officers begin screening cases at the family residential centers. (Recommendation 1) The Director of USCIS should develop and implement more specific guidance on requirements for documenting results of Asylum Division periodic quality assurance reviews. (Recommendation 2) The Director of USCIS should ensure asylum officers systematically record in USCIS’s automated case management system if individuals receive credible fear determinations as principal applicants, dependents, or in the interest of family unity, pursuant to regulation or USCIS policy. (Recommendation 3) The Director of USCIS should collect and analyze additional information on case delays, including specific reasons for delays and how long they last, that asylum officers may face when screening credible and reasonable fear cases. (Recommendation 4) We provided a draft of this report to DHS and DOJ for review and comment. DHS provided formal, written comments, which are reproduced in full in appendix IV. DHS also provided technical comments, which we incorporated as appropriate. DOJ told us they had no comments on the draft report. DHS concurred with our recommendations and described actions planned or underway to address them. For example, regarding our recommendation that all USCIS asylum offices provide officers with pre-departure training on credible and reasonable fear before they officers begin screening cases at family residential centers, DHS stated that USCIS plans to develop a standardized pre-departure training and provide this training to all asylum officers prior to their deployment to the family residential centers. In addition, regarding our recommendation that USCIS ensure that asylum officers record in their automated case management system if individuals receive credible fear determinations as principal applicants, dependents, or in the interest of family unity, DHS noted USCIS will explore ways to modify its case management system to ensure that asylum officers record such data and train officers on any subsequent system changes. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, the Attorney General, and other interested parties. In addition, the report is available at no charge on GAO’s website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V. We were asked to review processes for screening noncitizens who arrive at the southwest border expressing an intention to apply for asylum, a fear of persecution or torture, or a fear of return to their country, and the resources needed to carry out these screenings within applicable time frames by the Department of Homeland Security’s (DHS) U.S. Citizenship and Immigration Services (USCIS) and Department of Justice’s (DOJ) Executive Office for Immigration Review (EOIR). This report discusses (1) what USCIS and EOIR data show about the credible fear and reasonable fear processes, (2) the extent to which USCIS has policies and procedures for overseeing credible fear and reasonable fear screenings, and (3) the extent to which USCIS and EOIR have processes for managing their respective credible fear and reasonable fear-related workloads. To address all three objectives, we interviewed USCIS headquarters personnel from the Refugee, Asylum, and International Operations Directorate (RAIO) and RAIO’s Asylum Division who are responsible for managing USCIS’s credible and reasonable fear screening processes. We also interviewed officials from USCIS’s Fraud Detection and National Security Directorate (FDNS), which is responsible for leading USCIS’s efforts to detect and deter immigration benefit fraud and help detect national security and public safety threats. We conducted site visits at two of USCIS’s eight asylum offices—Houston, Texas and Arlington, Virginia—in April 2019. We selected these asylum offices based on the relatively large size of their credible and reasonable fear caseloads in fiscal year 2018—the most recent, complete data available at the time of our review. During these visits, we conducted in-person, semi-structured interviews with asylum officers, supervisory asylum officers, training officers, FDNS immigration officers, and asylum office management. During these interviews, we discussed topics related to data quality, supervisory review, training, quality assurance, family processing, and resource allocation. While the views expressed in these interviews do not represent those of all Houston and Arlington asylum office officials, they provide valuable insights from stakeholders who have experience with credible and reasonable fear policies and procedures. In addition, we collected written responses from the remaining six asylum offices on the same topics. Further, we conducted site visits to U.S. Immigration and Customs Enforcement (ICE) adult detention centers and family residential centers. Specifically, we visited single adult detention facilities in San Diego, California (September 2018), and Port Isabel and Pearsall, Texas (October 2018 and February 2019, respectively). We selected these ICE single adult facilities based on their geographic proximity to various CBP field locations we visited (discussed below). In addition, in February 2019, we visited ICE’s Enforcement and Removal Operations field office in San Antonio, Texas, as well as ICE’s South Texas Family Residential Center in Dilley, Texas, and Karnes County Residential Center in Karnes, Texas. We selected these two ICE family residential centers for field visits because they accounted for more credible and reasonable fear referrals to USCIS than any other single adult detention facility or family residential center. We also selected them to examine unique aspects of ICE and USCIS processing of credible and reasonable fear claims made by members of family units. During these visits to USCIS asylum offices and ICE detention facilities, we observed USCIS asylum officers conducting credible or reasonable fear screenings of single adults and family unit members either in person or via telephone. In total, we observed more than 20 credible and reasonable fear interviews across our site visits. Our observations are not generalizable to all USCIS asylum offices conducting credible and reasonable fear screenings, but provided us the opportunity to learn more about how USCIS personnel conduct interviews, make fear determinations, process these cases, and coordinate with ICE officials. For additional context about how noncitizens are apprehended at the border, processed into expedited or full immigration removal proceedings, transferred to ICE, and ultimately referred to USCIS for credible and reasonable fear screenings, as appropriate, we interviewed headquarters personnel from DHS’s U.S. Customs and Border Protection’s (CBP) Office of Field Operations (OFO) and U.S. Border Patrol (Border Patrol) who are responsible for apprehending noncitizens at or between U.S. ports of entry. In addition, we conducted site visits at CBP facilities in California and Texas from September 2018 to October 2018. In California, we visited Border Patrol’s San Diego sector headquarters and Imperial Beach station, and OFO’s San Ysidro port of entry. In Texas, we visited CBP’s Central Processing Center and McAllen Border Patrol station in McAllen, Texas; Border Patrol’s Fort Brown, Weslaco, and Harlingen stations; and OFO’s Hidalgo and Brownsville ports of entry. During these visits, we interviewed Border Patrol and OFO officials and observed how CBP personnel processed apprehended individuals and, as appropriate, documented whether those individuals expressed an intention to apply for asylum, a fear of persecution or torture, or a fear of return to their country. To select these locations, we assessed CBP data on Border Patrol and OFO apprehensions along the southwest border and targeted specific locations that saw the greatest increase in the number of apprehensions of individuals from fiscal year 2016 to 2017. As noted previously, we also considered the geographical proximity of multiple CBP and ICE facilities to maximize observations. Our observations during site visits are not generalizable to all Border Patrol and OFO operations along the southwest border, but provided us the opportunity to learn more about policies and procedures for processing noncitizens into removal proceedings and documenting any fear claims. To address the first objective, we obtained and analyzed data and documentation from USCIS and EOIR. Regarding USCIS, we analyzed record-level data from USCIS’s automated case management system from fiscal year 2014 through the second quarter of fiscal year 2019 (March 2019)—the most recent time period for which complete data were available at the time of our review. We analyzed these data to identify the number, characteristics, and outcomes of credible and reasonable fear cases. According to USCIS officials, USCIS’s system creates a unique number, or “case ID” for each case. USCIS officials told us that a previous system used a different identifier for each case—the individual’s Alien number (or “A-number”)—and did not use a “case ID” field. USCIS transitioned from its previous system to its current system in February 2018 and, according to USCIS officials, cases originally opened prior to the transition to the new system may have been re-opened under the same “case ID” number in the new system. As part of our data reliability testing, we checked for unique “case ID” numbers by searching for duplicate values and determined the data did not have duplicate values for “case ID” numbers. For our analysis of USCIS data specifically for ICE detention facilities and family residential centers, we only included credible and reasonable fear cases for detained individuals. To assess the reliability of USCIS data, we completed a number of steps, including (1) performing electronic testing for obvious errors in accuracy and completeness, such as running logic tests; (2) reviewing existing information about the data and the systems that produced them, such as relevant training materials for USCIS officers who use the system; and (3) discussing data entry issues and data limitations with USCIS officials. We determined the data were sufficiently reliable to describe the number, outcomes, and characteristics of credible and reasonable fear cases. Regarding EOIR, we reviewed data on immigration judge reviews of credible and reasonable fear cases posted on its public website. Specifically, we reviewed EOIR data on credible and reasonable fear reviews from fiscal year 2014 through June 2019—the most recent, complete data available at the time of our review. We also obtained and analyzed summary data from EOIR on credible and reasonable fear reviews for those individuals detained in ICE’s family residential centers. We analyzed the data to determine the outcomes of all credible and reasonable fear reviews and compared the outcomes of all reviews with the outcomes of reviews at ICE’s family residential centers. Finally, we reviewed EOIR data on the outcomes in immigration court for those completed removal cases that began with a positive credible or reasonable fear determination. In addition, we interviewed immigration judges and other court personnel serving both detained and nondetained dockets from EOIR’s Otay Mesa Immigration Court and San Diego Immigration Court in California, and from EOIR’s Harlingen Immigration Court in Texas. We also observed two immigration judge reviews of negative credible fear determinations. Our observations are not generalizable to all immigration judge reviews, but provided us the opportunity to learn more about EOIR’s processes. We interviewed EOIR officials about their data entry and management practices for credible and reasonable fear reviews. We determined that the data EOIR provided, much of which they report publicly on their website, are sufficiently reliable for analyzing the number and duration of credible and reasonable fear reviews that are received, completed, and pending. To provide additional context on the numbers, characteristics, and outcomes of CBP apprehensions, we obtained and analyzed record-level data on all apprehensions by Border Patrol and OFO from fiscal year 2014 through the second quarter of fiscal year 2019. We also obtained and analyzed record-level data on ICE detentions from fiscal year 2014 through fiscal year 2018 (see app. III for the results of our analyses). To assess the reliability of Border Patrol, OFO, and ICE data, we completed a number of data reliability steps, including (1) performing electronic testing for obvious errors in accuracy and completeness, such as running logic tests; (2) reviewing existing information about the data and systems that produced them, such as relevant training materials for Border Patrol agents and OFO, and ICE officers who use agency data systems; and (3) discussing data entry issues and data limitations with Border Patrol, OFO, and ICE officials. We also received demonstrations on the data systems from Border Patrol, OFO, and ICE officials at headquarters and in the field. As described below, we determined that the data are sufficiently reliable for providing information on the numbers, characteristics, and outcomes of CBP apprehensions and ICE detentions. Border Patrol data. For our analysis of Border Patrol data, we used “apprehensions” as our unit of analysis, instead of the number of individuals apprehended, because an individual may have been apprehended multiple times in the same year. We identified a small number of Border Patrol apprehension records that had the same date of apprehension and unique identifier (“A-number”). It is possible that these apprehension records represented one apprehended individual that Border Patrol agents processed as two apprehensions. These records comprised less than one percent of the more than 2.3 million apprehension records we analyzed. We included these apprehension records in our analysis because Border Patrol considers them unique apprehensions and because their small number did not materially affect our analysis. In addition, Border Patrol did not systematically track family relationships in its data systems until fiscal year 2016, as we have previously reported. Therefore, our analysis of Border Patrol apprehensions of family unit members processed under expedited removal proceedings is for fiscal years 2016 through the first two quarters of 2019. Further, according to Border Patrol officials, Border Patrol did not record reasonable fear cases in its automated data system before April 2016. Therefore, we are reporting the number of reasonable fear cases recorded by the Border Patrol in its automated system from fiscal year 2017 (the first full year for which Border Patrol recorded this information in its system) through the second quarter of fiscal year 2019. We did not include the 860 reasonable fear cases that Border Patrol recorded in its automated system for fiscal year 2016, since this number represents only partial-year data. According to Border Patrol officials, prior to April 2016, these reasonable fear cases would likely have been recorded under other case dispositions in their automated system, such as one indicating the reinstatement of a prior removal order. We determined that Border Patrol data are sufficiently reliable to describe the numbers and demographic characteristics of individuals and family unit members apprehended from fiscal year 2014 through the second quarter of fiscal year 2019. OFO data. For our analysis of OFO data, we used “apprehensions” as our unit of analysis, instead of the number of individuals apprehended. We excluded approximately 13 percent of all apprehension records (including single adults, unaccompanied alien children, and parents and children that arrived as part of a family unit) from our analyses because we could not confirm an A-number for those apprehensions. Among the apprehension records missing an A-number, 44 percent were cases in which OFO officers paroled the individuals and, according to OFO officials, officers are not required to assign an A-number to these individuals. In addition, 47 percent of the records with a missing A- number were cases that involved the individual withdrawing their application for admission into the United States, in which OFO officers have discretion whether to assign an A-number. According to OFO officials, additional records with missing A-numbers may be due to data entry errors or problems with the data system saving this information in the database that OFO used to pull the data. Finally, we collapsed 182,266 apprehension records into 86,597 apprehension records because we determined that they were duplicate records for the same individual and the same apprehension, based on factors such as alien number, birth date, and date and time of apprehension. As a result, we determined that we could not present precise figures for analyses that include OFO data and instead provided approximations throughout the report. We rounded all data and figures on OFO apprehensions down to the hundreds place and described relevant data using modifiers such as “at least” because of possible missing information. In addition, according to OFO officials, OFO does not capture information in its automated data system on individuals who were processed under expedited removal with a reasonable fear claim. OFO officials stated that, since OFO has historically processed a relatively small number of such apprehensions, it does not collect automated data on reasonable fear claims. With the previously-described modifications, we determined that OFO data are sufficiently reliable to generally describe the numbers and demographic characteristics of individuals and family unit members apprehended from fiscal year 2014 through the second quarter of fiscal year 2019. ICE data. To report on ICE detentions of adults and family unit members, we obtained and analyzed ICE detention data from fiscal years 2014 to 2018, the most current data available at the time of our review. The ICE data we obtained contained information on whether adults and family members booked-in to an ICE detention facility had a fear claim recorded in ICE’s data system as of the date our data were pulled. Specifically, we divided our analysis of ICE detention data into two parts. First, we obtained data on all individuals (all adults and children without consideration of any family relationships) detained from fiscal year 2014 through fiscal year 2018. Second, we obtained data specifically on family unit members apprehended by CBP and housed at the four ICE family residential centers from fiscal year 2014 through fiscal year 2018. Regarding our analysis of family unit members who made a fear claim in one of ICE’s family residential centers, we excluded less than one percent of all detention records from our analyses because we could not confirm a unique identifier for the individual. In addition, for individual family unit members who were detained more than once in a fiscal year, we included the most recent record for the individuals in our analyses to report on the most recent information available about each individual. This accounted for less than one percent of all detention records in our time period of analysis. We determined that the data were sufficiently reliable to describe the numbers of individuals (adults and family unit members) who were apprehended by CBP and recorded by ICE as having made a credible fear claim. To address our second objective, in addition to our aforementioned interviews and site visits, we reviewed relevant laws and regulations governing the credible and reasonable fear screening process. We collected and analyzed documentation on key USCIS oversight mechanisms related to credible and reasonable fear screenings— supervisory review, asylum officer training, and quality assurance reviews. In particular, we reviewed the Credible Fear Procedures Manual, and the Reasonable Fear Procedures Manual, standard operating procedures, training and quality assurance records and materials, and guidance on conducting credible and reasonable fear screenings for families in ICE detention. Specifically, we reviewed USCIS asylum officer basic training materials from RAIO and the Asylum Division, and training materials for officers from outside the Asylum Division who screen credible and reasonable fear cases. In particular, we reviewed USCIS Asylum Division quarterly training reports for fiscal year 2018 and used them to analyze the weekly training activities in each asylum office for each week of the reporting quarter. We compared RAIO and Asylum Division training materials with federal internal control standards related to developing competent individuals qualified to carry out assigned responsibilities. We also reviewed documents associated with the quality assurance reviews that the Asylum Division conducted, including those reviews conducted in collaboration with RAIO. Specifically, we reviewed standard operating procedures, reviewer checklists, and resulting reports and analysis for three RAIO nationwide reviews of credible and reasonable fear cases and for the six periodic reviews of credible and reasonable fear cases the Asylum Division conducted at asylum offices and at the family residential centers between November 2017 and May 2018. We compared these policy documents and their role in providing oversight of the credible and reasonable fear process against federal internal control standards related to ongoing monitoring activities and evaluation of results. We also reviewed USCIS standard operating procedures, requirements, and training materials for processing family members, and corresponding data on applicant family relationships. We then compared the procedures, requirements and data against federal internal control standards related to obtaining high quality data. To address our third objective, we reviewed USCIS and EOIR documents and data, and interviewed relevant officials to evaluate the extent to which USCIS and EOIR have process for managing their respective credible and reasonable fear-related workloads. USCIS. In particular, we reviewed USCIS documentation and spoke with officials from Asylum Division headquarters and local asylum offices regarding the Asylum Division’s staffing allocation model for the credible and reasonable fear workload. In addition, we obtained and analyzed record-level data from USCIS’s automated case management system to identify processing times and case delays for credible and reasonable fear cases between fiscal year 2014 through the second quarter of fiscal year 2019 (March 2019). We included cases that had a fear determination that was served or an administrative closure for both detained and nondetained individuals. In this report, we present information on both credible and reasonable fear case receipts and analysis of processing times for the cases using the “clock-in” date recorded in USCIS’s automated case management system. However, while USCIS relies on the “clock-in” date to track case processing times, according to an Asylum Division official USCIS tracks and reports the number of credible and reasonable fear case receipts based on the date cases are input into, or “created” in, its automated system. According to the official, these “created” and “clock-in” dates are often the same, but can differ slightly. Therefore, the number of case receipts tracked and reported by USCIS may differ slightly from those presented in this report. Regarding credible fear cases, we determined case processing times by calculating the difference between the beginning and end dates for credible fear cases. We considered credible fear case processing times for detained individuals to begin on the day when USCIS receives referral documents and records a “clock-in” date in the automated case management system, as noted previously. For nondetained individuals, the clock starts for credible fear cases when a USCIS asylum office conducts the interview for a credible fear screening. We used the starting clock date for detained and nondetained individuals provided by USCIS for our analysis. We considered credible fear case processing times to end on the day when cases either had a fear determination that was served or an administrative closure. We included credible fear cases that had a fear determination that was served or an administrative closure for detained and nondetained individuals. We also reviewed USCIS’s publicly-reported data on credible fear processing times during this time period. Regarding reasonable fear cases, we used USCIS data to count the number of processing days and percent of cases completed in certain time intervals. We determined reasonable fear processing times by calculating the difference between the beginning and end dates for reasonable fear cases. We considered reasonable fear case processing times for detained individuals to begin on the day when USCIS receives referral documents and records a “clock-in” date in the automated case management system, as noted previously. For nondetained individuals, the clock starts for reasonable fear cases when a USCIS asylum office conducts the interview for a reasonable fear screening. We used the starting clock date for detained and nondetained individuals provided by USCIS for our analysis. We calculated reasonable fear processing times in court days by excluding weekends and federal holidays. USCIS may also pause the clock when processing reasonable fear cases in certain circumstances. We excluded approximately 13 percent of reasonable fear cases that had at least one pause in case processing time from our analysis because in conducting our analysis we could not systematically confirm the appropriate order of dates for those cases. We considered reasonable fear case processing times to end on the day when cases either had a fear determination that was served or administrative closure. We also included reasonable fear cases that were served a fear determination or received an administrative closure for detained and nondetained individuals. To identify the reasons for delays in credible and reasonable fear cases during the time period of our analysis, we identified the fields that USCIS’s case management system tracks for case delays related to the credible and reasonable fear workload. In addition, we reviewed USCIS’s manuals and documentation on its case management system. We compared USCIS’s recording and tracking of data on case delays to federal internal control standards related to obtaining data on a timely basis for management to use for effective monitoring and that data should be processed into high quality information. We determined that the USCIS data we reviewed on credible and reasonable fear processing times and case delays were sufficiently reliable for our purposes. EOIR. To evaluate EOIR’s process for managing its credible and reasonable fear-related workload, we interviewed EOIR officials about their practices to manage the credible and reasonable fear workload, including immigration judge hiring, oversight of credible and reasonable fear review processing times, infrastructure requirements for credible and reasonable fear reviews, and the use of video teleconferencing by judges to conduct credible and reasonable fear reviews. We reviewed publicly available data about EOIR’s workload and case adjudications, including data about the number of credible and reasonable fear reviews EOIR judges completed and data about judges hired from fiscal year 2014 through fiscal year 2019. We also reviewed guidance documents, such as EOIR’s 2018 Case Priorities and Performance Measures memorandum, which established performance measures for credible and reasonable fear reviews. In addition, we used EOIR data to analyze the timeliness of EOIR’s completion of credible and reasonable fear reviews and compared EOIR’s processing times for fiscal year 2014 through the third quarter of fiscal year 2019 with required time frames. By reviewing documentation on EOIR’s case management system and interviewing officials with knowledge about EOIR’s case management system and the methodology used to calculate the publicly-reported data, we determined that the EOIR data we reviewed on credible and reasonable fear review processing times and outcomes was sufficiently reliable for analyzing the number of credible and reasonable fear reviews completed and pending, and the duration of the reviews. We conducted this performance audit from November 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient and appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix provides detailed information on eligibility, screening standards, and possible screening outcomes for both credible fear and reasonable fear cases. Noncitizens placed into expedited removal who make fear claims will be referred to U.S. Citizenship and Immigration Services (USCIS) for a credible fear screening by an asylum officer or, if the individual has been issued a final administrative removal order after conviction for an aggravated felony or has a prior order of removal that is reinstated, and expressed a fear of return, for a reasonable fear screening. Table 8 below describes the eligibility and screening standards, as well as the potential outcomes for USCIS’s credible fear screening cases. Similarly, table 9 details eligibility, screening standards, and potential outcomes for reasonable fear screening cases. If noncitizens are placed into expedited removal proceedings instead of full removal proceedings, they are to be ordered removed from the United States without further hearing before an immigration judge unless they indicate either an intention to apply for asylum or a fear of persecution or torture or a fear of return to their country (referred to throughout this appendix as making a “fear claim”). This appendix provides information on the number and dispositions (such as full removal proceedings or expedited removal proceedings, among others) of noncitizens who were apprehended by the Department of Homeland Security’s (DHS) U.S. Customs and Border Protection’s (CBP) U.S. Border Patrol (Border Patrol) and Office of Field Operations (OFO) at or between U.S. ports of entry from fiscal year 2014 through the second quarter of fiscal year 2019. It also includes U.S. Immigration and Customs Enforcement (ICE) data on detentions of noncitizens who made a credible fear claim. For cases in which noncitizens were referred to U.S. Citizenship and Immigration Services (USCIS) for a fear screening, this appendix also provides additional information on the characteristics of these cases, including their country of origin, age, gender, whether they had representation, and location of their screenings. As shown in table 10, Border Patrol apprehensions totaled more than 2.3 million from fiscal year 2014 through the second quarter of fiscal year 2019. Further, Border Patrol data indicate that agents processed about 687,000 (or 30 percent) for full immigration proceedings and nearly 931,000 (or 40 percent) under expedited removal proceedings. For those apprehensions that agents processed under expedited removal, more than 197,000 (approximately 9 percent of total apprehensions) included a credible fear claim made in Border Patrol custody. As also shown in table 10, during fiscal years 2017 through the first two quarters of 2019, Border Patrol apprehended more than 10,000 additional noncitizens who made reasonable fear claims. As shown in figure 7, the number of Border Patrol apprehensions of individuals who were placed into expedited removal proceedings with a credible fear claim increased from more than 16,000 apprehensions in fiscal year 2014 to more than 51,000 in fiscal year 2018. These apprehensions of individuals claiming fear ranged from 3 percent to 13 percent of total apprehensions during these fiscal years. Border Patrol data include various characteristics of each apprehension such as age, gender, and whether a noncitizen was a member of a family unit. For example, of the nearly 208,000 apprehensions processed under expedited removal with a credible or reasonable fear claim during fiscal years 2014 through the first half of fiscal year 2019, approximately 166,000 (or 80 percent) were adults age 18 and above with the remaining 42,000 (or 20 percent) encompassing children age 17 and under (see table 11). Of the nearly 208,000 apprehensions processed under expedited removal with a credible or reasonable fear claim during fiscal years 2014 through the first half of fiscal year 2019, approximately 117,000 (or 56 percent) were male and the remaining 90,000 (44 percent) were female (see table 12). As shown in table 13, for fiscal years 2016 through the first two quarters of 2019, Border Patrol apprehended nearly 456,000 noncitizens who were members of families. Of these, Border Patrol processed more than 120,000 (or 26 percent) under expedited removal proceedings. Nearly 71,000 apprehensions during this time period (15 percent of total family unit members apprehended and 59 percent of those placed in expedited removal) included a credible fear claim. From fiscal year 2014 through March 2019, OFO apprehensions at ports of entry totaled at least 546,900. Of these 546,900 apprehensions, OFO officers placed at least 193,500 (or 35 percent) into expedited removal proceedings. For those in expedited removal proceedings, OFO data indicate that at least 104,600 apprehensions included a credible fear claim in OFO custody (19 percent of total apprehensions). In addition, OFO issued Notices to Appear before an immigration judge for full immigration proceedings to at least 167,400 (or 31 percent) of the approximately 546,900 total apprehensions (see figure 8). As shown in figure 9, the number of OFO apprehensions in expedited removal proceedings with a credible fear claim generally increased over this time period from at least 11,600 apprehensions in fiscal year 2014 to at least 27,000 in fiscal year 2018 (the last full year of data available at the time of our analysis). In addition to this overall increase, the percentage of OFO’s total apprehensions placed into expedited removal proceedings with a credible fear claim also increased. Specifically, these apprehensions increased from about 17 percent of all apprehensions in fiscal year 2014 to about 26 percent in fiscal year 2018. OFO apprehension data include various characteristics such as age, gender, and whether an apprehension involved a member of a family unit. For example, as shown in table 14, of the approximately 104,300 OFO apprehensions with credible fear claims, at least 78,500 (or 75 percent) were adults age 18 and above with about 25,700 (or 25 percent) of the remaining credible fear claims encompassing children age 17 and under. Also, for each year during this period, the percentage of adults versus children was generally consistent with this overall percentage with the exception of fiscal year 2019, for which the partial year’s data show that about 98 percent of those apprehensions processed under expedited removal with a credible fear claim were adults. In addition, for fiscal years 2014 through the first two quarters of fiscal 2019, at least 56,500 (or 54 percent) of these apprehensions involving a fear claim were male and at least 47,400 (or 45 percent) were female (see table 15). Also, for each year during this period, the number of males and females were almost evenly split with the exception of fiscal year 2019, for which the partial year’s data show a larger proportion of males claiming fear. As shown in table 16, for fiscal years 2016 through the first two quarters of 2019, OFO had a total of at least 144,100 apprehensions involving members of family units. Of these approximately 144,100 apprehensions, OFO placed at least 39,100 (27 percent) into expedited removal proceedings of which at least 32,900 (about 23 percent of total family unit members apprehended and approximately 84 percent of those placed in expedited removal) claimed a credible fear of returning to their country. The number of individuals in expedited removal proceedings detained in ICE facilities with a credible fear claim increased from fiscal years 2014 to 2018. Specifically, as shown in table 17, ICE data indicate that the number of individuals in expedited removal proceedings with a recorded credible fear claim while in ICE detention increased from about 37,000 (or 9 percent) in fiscal year 2014 to about 99,000 (or 26 percent) in fiscal year 2018. The period of greatest percentage increase was from fiscal years 2015 to 2016 when the percentage of individuals in expedited removal proceedings with a credible fear claim while in ICE custody increased from approximately 15 percent to 25 percent. For fiscal years 2014 through 2018, the majority of family unit members in ICE’s four family residential centers had a credible fear claim (81 percent), as demonstrated in table 18. The number of family unit members with a fear claim ranged from approximately 69 percent in fiscal year 2015 to 88 percent in fiscal year 2018. For fiscal years 2014 through 2018, slightly more than half of all family unit members in ICE’s four family residential centers with a credible fear claim were children under the age of 18 (55 percent). As also shown in table 19, the division between adults and children with fear claims varied little each year. As shown in Figure 10, the majority of credible fear cases referred to USCIS for screening from fiscal year 2014 through the first two quarters of fiscal year 2019 had applicants who were nationals of El Salvador, Honduras, Guatemala, or Mexico. Citizens of these countries accounted for 74 percent of all credible fear cases during this time period (approximately 306,000 referrals). As shown in table 20, El Salvador had the most credible fear referrals to USCIS each year from fiscal year 2014 through fiscal year 2017. However, beginning in fiscal year 2018, Honduras accounted for the most credible fear referrals to USCIS among these four countries. As shown in Figure 11, applicants from the countries of Mexico, Honduras, El Salvador, and Guatemala accounted for all but approximately 7 percent of the reasonable fear cases screened by USCIS for fiscal years 2014 through the first two quarters of fiscal year 2019. Overall, Mexican nationals accounted for the largest number of reasonable fear cases among these four countries (33 percent of total reasonable fear cases). As shown in table 21, Mexico had the most reasonable fear referrals to USCIS each year from fiscal years 2014 through the first two quarters of fiscal year 2019. As table 22 shows, noncitizens making credible fear claims who had representation present at their interviews with asylum officers more often received positive determinations of fear by the asylum officer. Overall, during this time period, the number of positive determinations in cases with representation was nearly 10 percentage points greater than those without representation. As table 23 shows, similar to credible fear cases, noncitizens making reasonable fear claims who had representation present at their interviews with asylum officers more often received positive determinations of fear by the asylum officer. Overall, during this time period, the number of positive determinations in cases with representation was over 20 percentage points greater than those without representation. As table 24 shows, two of ICE’s family residential centers (Dilley and Karnes family residential centers) accounted for the highest number of credible and reasonable fear referrals, among the top five facilities making these referrals, from fiscal years 2014 through the first two quarters of fiscal year 2019. As shown in table 25, reasonable fear screenings for those in ICE family residential centers comprised 6 percent of all such cases referred to USCIS during this same period with the percentage of positive determinations (77 percent) higher than that for all reasonable fear cases nationwide (30 percent). As shown in table 26, the Houston asylum office screened two-thirds (67 percent) of credible fear cases from fiscal year 2014 through the first two quarters of fiscal year 2019. Also, over this same time period, USCIS’s Los Angeles asylum office screened the second most credible fear cases (11 percent). However, since fiscal year 2018, USCIS’s Asylum Pre- Screening Center has screened the second most credible fear cases after Houston. As shown in table 27, the Houston asylum office screened nearly half (approximately 45 percent) of reasonable fear cases from fiscal year 2014 through the first two quarters of fiscal year 2019. Also, over this same time period, USCIS’s Los Angeles asylum office screened the second most reasonable fear cases (12 percent). However, since fiscal year 2018, USCIS’s Asylum Pre-Screening Center has screened the second most reasonable fear cases after Houston. In addition to the contact named above, Kathryn Bernet (Assistant Director), Michael Harmond (Analyst-in-Charge), Hiwotte Amare, Miranda Cohen, Benjamin Crossley, Michele Fejfar, Cynthia Grant, Jan Montgomery, Heidi Nielson, Mary Pitts, Adam Vogt, and Jessica Walker made key contributions to this work.", "summary": "Individuals apprehended by DHS and placed into expedited immigration proceedings are to be removed from the country without a hearing in immigration court unless they express an intention to apply for asylum, or a fear of persecution, torture, or return to their country. Those with such “fear claims” are referred to USCIS for a credible fear screening. Individuals who have certain criminal convictions or who have a reinstated order of removal and claim fear are referred for a reasonable fear screening. Those with negative outcomes can request a review by EOIR's immigration judges. GAO was asked to review USCIS's and EOIR's processes for fear screenings. This report examines (1) USCIS and EOIR data on fear screenings, (2) USCIS policies and procedures for overseeing fear screenings, and (3) USCIS and EOIR processes for workload management. GAO analyzed USCIS and EOIR data from fiscal years 2014 through mid-2019; interviewed relevant headquarters and field officials; and observed fear screenings in California, Texas, and Virginia, where most screenings occur. Data from the Department of Homeland Security's (DHS) U.S. Citizenship and Immigration Services (USCIS) and Department of Justice's Executive Office for Immigration Review (EOIR) indicate that their credible and reasonable fear caseloads generally increased from fiscal year 2014 through fiscal year 2018. USCIS's caseloads nearly doubled during this timeframe—from about 56,000 to almost 109,000 referrals for credible and reasonable fear screenings. Further, the credible fear caseload was larger in the first two quarters of fiscal year 2019 alone than in each of fiscal years 2014 and 2015. Referrals to USCIS for reasonable fear screenings also increased from fiscal years 2014 through 2018. USCIS asylum officers made positive determinations in 71 percent of all credible and reasonable fear screenings between fiscal years 2014 and the first two quarters of fiscal year 2019. The outcomes of the remaining screenings were generally split evenly (14 percent each) between negative determinations or administrative closures (such as if the applicant was unable to communicate). EOIR's caseload for immigration judge reviews of USCIS's negative credible and reasonable fear determinations also increased between fiscal year 2014 and fiscal year 2018. EOIR's immigration judges reviewed about 55,000 cases from fiscal year 2014 through the third quarter of 2019 (the most recent data available), and judges upheld USCIS's negative determinations in about three-quarters of all reviews. USCIS has developed various policies and procedures for overseeing credible and reasonable fear screenings in accordance with the regulations governing those screenings, such as interview requirements and mandatory supervisory review. USCIS provides basic training for new asylum officers and other training at individual asylum offices that includes credible and reasonable fear. The training at asylum offices includes on-the-job training for officers newly-assigned to credible and reasonable fear cases and ongoing weekly training for incumbent officers—some of which includes credible and reasonable fear. However, USCIS asylum offices do not all provide additional pre-departure training before officers begin screening families in person at DHS's family residential centers. Asylum Division officials told GAO that additional training for asylum officers before they begin screening such cases is important—in particular, credible fear screenings at these facilities represent about one-third of USCIS's caseload. Almost all USCIS asylum offices send officers to the family residential centers, including those offices with small fear caseloads at the local level. Some asylum offices provide pre-departure training to officers being sent to screen families, but such training is inconsistent across offices. By comparison, officials from the Chicago and New York offices stated they do not provide formal pre-departure training, but rather direct or recommend that officers review Asylum Division guidance and procedures on family processing independently before they travel. Officials from two other offices stated they rely on the training asylum officers may receive throughout the year related to credible and reasonable fear, which can vary. Providing pre-departure training, in addition to USCIS's basic training for new asylum officers, would help USCIS ensure that officers from all asylum offices are conducting efficient and effective fear screenings of families. Further, consistent with regulation, USCIS policy is to include any dependents on a principal applicant's credible fear determination if the principal applicant receives a positive determination, resulting in the principal and any dependents being placed into full removal proceedings with an opportunity to apply for various forms of relief or protection, including asylum. For example, a parent as a principal applicant may receive a negative determination, but his or her child may receive a separate positive determination. In the interest of family unity, USCIS may use discretion to place both the parent and child into full removal proceedings rather than the parent being expeditiously ordered removed in accordance with the expedited removal process. However, USCIS's case management system does not allow officers to record whether an individual receives a determination on his or her case as a principal applicant, dependent, or in the interest of family unity. Without complete data on all such outcomes, USCIS is not well-positioned to report on the scope of either the agency's policy for family members who are treated as dependents, pursuant to regulation, or USCIS's use of discretion in the interest of family unity. USCIS and EOIR have processes for managing their respective credible and reasonable fear workloads. For example, USCIS uses national- and local-level staffing models to inform staffing allocation decisions. USCIS also sets and monitors timeliness goals for completing credible and reasonable fear cases. Although USCIS monitors overall processing times, it does not collect comprehensive data on some types of case delays, which officers told us can occur on a regular basis. Asylum officers whom GAO interviewed stated that certain delays could affect the number of credible or reasonable fear cases they can complete each day. Collecting and analyzing additional information on case delays would better position USCIS to mitigate the reasons for the delays and improve efficiency. EOIR has developed processes for immigration courts and judges to help manage its workload that include performance measures with timeliness goals for credible and reasonable fear reviews. EOIR data indicate that about 30 percent of credible and reasonable fear reviews are not completed within the required timeframes. EOIR officials said they plan to implement an automated tool in early 2020 to monitor court performance, including the credible and reasonable fear performance goals. Because implementation of the automated tool is planned for early 2020, it is too soon to know if EOIR will use the tool to monitor adherence to the required credible and reasonable fear review time frames or if it will help EOIR understand reasons for case delays. GAO is making four recommendations, including that USCIS provide additional pre-departure training to USCIS asylum officers before they begin screening families, systematically record case outcomes of family members, and collect and analyze information on case delays. DHS concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Throughout history, new technologies have transformed societies. Many technological advances, ranging from the steam engine to electricity and personal computers, have enhanced productivity and improved societal standards of living. At the same time, many technological advancements have led to increases in automation—modifying processes to become more automatic by reducing human involvement—and corresponding changes in the workforce. For example, researchers have noted that automation has replaced tasks performed by workers and also increased production, creating a greater demand for other types of workers. Although automation has historically been a labor disrupter in manufacturing and physical work, various researchers have observed that recent progress in fields such as artificial intelligence (AI) and robotics are enabling machines to perform cognitive tasks currently performed by humans. Artificial intelligence refers to machines and computers that attempt to mimic various aspects of human intelligence, as we have reported. The field of AI can be traced back to the 1950s. Early AI often consisted of expert systems programmed by humans to perform predefined tasks. This form of AI resulted in some degree of productivity gains and remains an active area of development. However, numerous factors, primarily the trends underlying big data (i.e., increased data availability, storage, and processing power), have contributed to rapid innovation and accomplishments in AI in recent years. Present-day AI innovation centers more on machine learning, including deep neural network architectures, in which systems are trained against observational or simulated outcomes—applications include language translation and machine vision (i.e., systems that use cameras, radar, or lasers to observe their surroundings or recognize content). Industrial robots and robotic machinery are often more comparable to expert systems that are programmed to perform predefined tasks, but they can also incorporate machine learning, such as having machine vision capabilities (e.g., object recognition). Below are some examples of expert system and machine learning applications of artificial intelligence. Examples of expert system applications of AI: software programs that prepare tax filings or schedule logistics; and industrial robots that perform predefined or routine tasks, such as lifting, placing, and welding pieces of metal together. Examples of machine learning applications of AI: software that uses a training dataset to “learn” how to read information from a form filled out by a person; collaborative robots that can sense when they touch a physical obstruction and shut down to safely work alongside humans; industrial robots with machine vision incorporated to identify and pick up specific parts from a collection of randomly strewn pieces; and automated guided vehicles that transport materials around a production plant and use cameras and radar to navigate independently and re-route around obstacles. Advanced technologies, including AI and other technological drivers of workforce changes, are continually progressing and new developments emerge regularly. For example, automated vehicles have varying levels of autonomy. Similarly, while robots have existed for decades, today’s generation of robots may be equipped with machine vision and learning capabilities that enable them to perform a more expansive array of tasks. How, when, or whether technologies progress from development to commercialization (i.e., readiness for adoption), and how, when, or whether firms adopt the technologies is generally dependent on context- specific considerations, which are difficult to predict. To better understand these developments and how they affect the economy, the National Academies report recommended developing three indexes (technology progress index; AI progress index; and organizational change and technology diffusion index) to measure technology progress and the extent of adoption. The study suggested that indexes could be valuable for identifying what fields are advancing rapidly and what benchmarks might indicate the imminence of significant economic impact, as well as tracking and predicting the types of human tasks that can be automated and the impacts of technology adoption by industry. Stanford University’s AI Index project is another initiative that aims to track, collate, and visualize data related to artificial intelligence. The data collected by the AI index measure, among other things, volume of AI activity (e.g., published papers, course enrollment, AI-related startups, job openings) and technical performance (e.g., object detection and speech recognition). However, the potential uses and limitations of the data being compiled are yet to be seen, as this initiative is still in its early stages. While national employment data measure jobs and workers by occupation and industry, the adoption of advanced technologies generally affects specific work tasks, and can materialize in a variety of ways. As shown in figure 1, industries are made up of various occupations, which in turn are formed by a group of jobs. Underlying all, jobs are comprised of a collection of varied work tasks. By analyzing tasks within jobs or occupations to determine their susceptibility to automation, a number of studies have developed models to estimate the future workforce effects of advanced technology adoption. The three example studies below each developed similar models, though differences in methods and data sources produced varying conclusions about the number of jobs that may be automated in the future. In a 2016 article, researchers Frey and Osborne estimate that 47 percent of total U.S. employment is in occupations that are at high risk of automation over the next decade or two (i.e., by 2030). For example, the authors observe both that industrial robots will be able to perform a wider scope of non-routine manual tasks and that a substantial share of employment in services, sales, and construction occupations exhibit high probabilities of automation. A 2017 report by the McKinsey Global Institute estimates that 23 percent of total U.S. work hours could be automated by 2030 or as high as 44 percent under other assumptions. The report predicts that while labor demand will enable some re-employment of displaced workers, up to one-third of the workforce may need to change occupational categories. In a 2016 paper, researchers Arntz, Gregory, and Zierahn estimate that 9 percent of all U.S. workers hold jobs that are at high risk of automation. The authors observe that susceptibility to automation is lower for jobs that require cooperating or influencing others. Studies by Autor and others also develop theoretical models exploring the effects of automation. For example, they noted that while automation can substitute for some tasks, it can also complement others. This can lead to increasing value for tasks that require other attributes like creativity and intuitive judgement. These models hypothesize that automation may have a net positive effect on employment, or at least on employment in certain sectors, which is consistent with historical employment trends. However, researchers have also noted that machine learning may affect different tasks than earlier forms of automation and may be less likely to automate low-wage jobs—though low-wage workers may be affected in other ways. Although the models discussed above represent ways of identifying jobs that may be affected by the adoption of advanced technologies, they do not provide a model for tracking the current or to-date workforce effects of technology adoption. As the recent National Academies report states, “making forecasts about social phenomena is perilous… doing so with respect to the fast-changing and dynamic area of technology is even more challenging.” According to a different project by some of these same experts, several factors unrelated to whether a task or job could be automated contribute to these challenges. For example, technologies may substitute for human labor in some tasks, but: may also complement human labor in other tasks—increasing the demand for, or value of, human labor (e.g., the automation of calculation tasks leading to increased demand for human programmers); prices and demand for products may counteract this human labor substitution (e.g., technology reducing the price of air travel, and thus leading to increased demand for flights, and thus increased employment in the aviation industry); and firms may redesign operations in response to the substitution in ways that lead to employment increases or decreases that are greater than the direct substitution. As discussed in the National Academies report and elsewhere, researchers have tried to disentangle workforce effects in various ways, such as analyzing productivity data to examine workforce trends in the context of other economic factors, such as globalization. As the National Academies report observes, “Predictions that new technologies will make workers largely or almost entirely redundant are as old as technological change itself…. However, predictions of widespread, technologically induced unemployment have not come to pass, at least so far.” Since recovering from the recession of 2007-2009, the economy has recently experienced low unemployment rates—4.0 percent in January 2019—despite continued strides in advanced technologies. However, other indicators have not recovered. For example, the labor force participation rate—the percentage of the population that is either employed or seeking work—declined significantly through the recession and has generally remained at this lower level. This may indicate that the post-recession decline in the unemployment rate may over-represent the health of the labor market, according to BLS. Advanced technologies and automation may also affect workers in other ways, beyond potential changes in the workplace, such as by reducing production costs and thus lowering the prices of consumer goods. There are currently no comprehensive data on firms’ adoption and use of advanced technologies. As a result, researchers have difficulty determining whether changes in the U.S. workforce observed in existing employment data are related to advanced technologies. The National Academies report states that federal household and employer surveys, such as the CPS, ACS, and OES, provide useful information about changes to the occupational mix of the U.S. workforce over time. However, these data cannot identify the causes of employment shifts. For example, these data do not identify whether an employment decline in one occupation is due to jobs being replaced as a result of automation, or to other factors unrelated to automation. Other federal data, such as the Job Openings and Labor Turnover Survey, provide useful information on employment turnover and opportunities. However, although these data are available by industry sector and firm size, the data do not capture reasons for layoffs and discharges, and thus cannot be linked to advanced technologies. In the absence of comprehensive data that definitively link employment trends to technology adoption, we analyzed occupations that researchers Frey and Osborne identified as being susceptible to automation (see sidebar) to determine whether changes due to advanced technologies are appearing in employment data. By exploring concentrations of these occupations in industries, job displacements in these occupations, and the characteristics of workers in these occupations, we found minor indications that advanced technologies are changing the workforce and could affect some worker populations. However, the conclusions that can be drawn from these analyses are limited by the unpredictability of when, if, or how automation materializes—e.g., whether worker positions are eliminated or shifted to other non-automated tasks. Industries with higher concentrations of jobs susceptible to automation were more likely than others to have experienced significant growth in their concentration of tech jobs from 2010 to 2016, according to our analysis of employment data from the American Community Survey. For example, as shown in figure 2, the plastics product manufacturing industry has a relatively high concentration of jobs susceptible to automation. Many of these jobs are in production occupations. From 2010 through 2016, this industry experienced about 11 percent annual growth in tech jobs (i.e., jobs in the fields of computing, engineering, and mathematics). More than half of this growth was the result of increases in industrial engineers, engineering technicians, and miscellaneous engineers. As we observed at some firms we visited, some of these engineers may have been hired to program or maintain newly installed robots. However, the data do not provide this level of information about job tasks. Similar dynamics could also be occurring in other industries. Across all 69 industries that had statistically significant changes in the concentration of tech jobs, we found a positive, though weak, correlation with the concentration of jobs susceptible to automation (see fig. 2). This suggests that growth in tech jobs may be an indicator of industries’ preparation for, or adoption of advanced technologies. However, given the complex causes of employment changes, there could be other reasons for tech job growth in these industries that are unrelated to firms’ adoption of advanced technologies. The growth in tech jobs in certain industries suggests firms in these industries may be using more advanced technologies, which could also signal that jobs susceptible to automation are being replaced. However, our analysis of ACS data showed no correlation between an industry having a higher concentration of jobs susceptible to automation and employment changes in that industry (i.e., total employment increases or decreases). We also found no meaningful differences in job losses, according to our analysis of employment data from the Current Population Survey’s Displaced Worker Supplement. Specifically, the relative rate at which workers in occupations susceptible to automation lost a job because their position or shift was abolished or there was insufficient work for them to do was not meaningfully different than workers in other occupations. There could be a number of reasons we did not find a relationship between susceptibility to automation and employment changes in both of these analyses, including: a relationship does not exist; such a relationship is too complex to measure in this way (e.g., automation may lead to decreases in employment in some industries, while also leading to increases in employment in other industries due to improved competitiveness, productivity, and profitability); it is too soon to observe the employment effects of automation (e.g., growth in tech jobs in an industry may be a leading indicator of employment disruption); or our analysis covered a period of overall economic growth, which could obscure or overwhelm other employment trends. Existing data cannot predict with certainty when or if automation will materialize in the workforce, as suggested by our analyses. However, the tendency of particular worker groups to hold jobs susceptible to automation suggests that some communities may be disproportionately affected by changes if they occur. For example, according to our analysis of 2016 ACS data, workers with lower levels of education are more likely than those with higher levels to hold jobs in occupations that the Frey and Osborne study identify as susceptible to automation. Specifically, 60.7 percent of workers with a high school degree or less hold these types of jobs, as compared to 46.7 percent of workers with some college, 26.9 percent of workers with a bachelor’s degree, and 11.3 percent of workers with a graduate degree. In addition, 54.1 percent of Hispanic workers hold jobs in occupations susceptible to automation, as compared to 46.4 percent of Black workers, 40.0 percent of White workers, and 35.9 percent of Asian workers. Certain geographic areas also rely more heavily than others on occupations identified as susceptible to automation, according to OES data. We identified areas where the proportion of jobs susceptible to automation is at least 5 percentage points greater than the national average (see fig. 3). These occupations are comprised of a diverse set of jobs that may experience automation in different ways and at different times, if at all. However, if employment disruptions are regionally concentrated, groups of workers with similar skills in the same labor market may need to adapt to changes simultaneously, which could strain the availability of local job opportunities and support resources. Workers in occupations that the Frey and Osborne study identify as susceptible to automation earn less on average than other workers. For example, the median hourly wage for workers in occupations susceptible to automation is $14.26, compared to $22.06 for other workers, according to our analysis of 2016 ACS data. After controlling for factors that may affect wages, such as age, education, and industry, we found that workers in jobs susceptible to automation earn about 17.2 percent less, on average, than similar workers in other occupations. These results show that, on average, workers in jobs susceptible to automation are already in more vulnerable economic circumstances than other workers. When or if changes brought on by automation materialize, these workers may face additional hardships in adapting to changing workforce demands. In the absence of comprehensive data, researchers have taken differing approaches to exploring the relationships between technology adoption and workforce trends. We identified some examples of recent and ongoing work that attempt to measure workforce effects directly attributable to technology adoption. These examples illustrate types of data that may be useful for better understanding and measuring the use of specific technologies (e.g., robot sales), the spread of technologies generally (e.g., automation patents), and how specific work tasks are changed by technology use (e.g., firm-level operations data). Some researchers have used data on industrial robot sales collected by the International Federation of Robotics (IFR) to approximate robotics adoption worldwide and in the United States and to model its direct effects on employment. Analysis by Furman and Seamans (2018) shows that annual sales of industrial robots in the United States increased substantially between 2010 and 2016. The analysis attributes this growth to a combination of factors, including lower robot prices, improved robot functionality, and greater awareness of the benefits of robots. They also observe that the automotive sector was the largest customer for industrial robot sales in the United States from 2004 through 2016, though robot sales to the consumer electronics sector grew the most over that period. Studies by Acemoglu and Restrepo (2017) and by Graetz and Michaels (2017) both use IFR data through 2007 to model the workforce effects of robot adoption in the United States, though their methods, results, and conclusions differ. Acemoglu and Restrepo estimate that each additional robot used in a geographic area reduces employment by about six workers in that area. They observe that their estimated employment effects are greatest in manufacturing and other industries most exposed to robots, in routine manual work-related occupations, and for workers with less than a college education. They do not find corresponding employment gains in any other occupation or education groups. They also estimate that one more robot used per thousand workers reduces wages by about 0.5 percent. They conclude by noting that, so far, relatively few robots have been used in the U.S. economy and thus the effect on jobs has been limited; however, they state that if robot usage continues to grow as researchers expect, these effects could be more substantial. Graetz and Michaels estimate that increased robot use did not significantly affect total hours worked across the 17 developed countries in their analysis, but that work shifted from low-skilled workers to middle-skilled and high-skilled workers. They also estimate that increased robot use increases productivity and average wages. While their analysis covers 17 developed countries, they note that robot use in the United States was marginally lower than the average across all countries. They also observe that while their results differ from Acemoglu and Restrepo, it is possible that the effects of robot usage are different in the United States than across the 17 countries they analyze. Other researchers have used U.S. patent data as an alternative way to approximate the spread of advanced technologies and to examine the resulting workforce effects. Mann and Püttman (2017) use machine learning algorithms to identify patents related to automation technology. They find that automation patents grew substantially from 1976 through 2014. After linking the patents to industries where they may be used, they estimate that automation causes manufacturing employment to fall, though it increases employment in the service sector, as well as overall employment. They observe that their results depict a more positive picture of the employment effects of new technology use than the studies that used industrial robot sales data (discussed above). Lee Branstetter, a researcher at Carnegie Mellon University, and his colleagues have a similar ongoing project that uses a machine learning algorithm to identify patents related to AI technologies. According to these researchers, their initial results suggest a rapid rise in AI patents over the past decade and also that AI patents are emerging in a variety of application areas. They are also in the early stages of work linking AI patents to industries to explore how new technology use affects the workforce. Researchers have also identified how important micro-level data could be for understanding the workforce effects of advanced technology adoption. For example, reports by the National Academies and others highlight the potential for firm-level information to augment traditional survey data to enable analyses of the conditions under which advanced technologies complement or substitute for workers, and what types of firms invest in advanced technologies. Other researchers have emphasized the importance of focusing on work tasks to analyze the effects of technological change at workplaces. Erica Fuchs, a researcher at Carnegie Mellon University, and her colleagues Christophe Combemale, Katie Whitefoot, and Laurence Ales use a combined firm-level, task- based approach by collecting and analyzing production floor data from four semiconductor firms with different levels of process automation and parts consolidation. They map out detailed versions of firms’ production processes and then use existing data and technical knowledge to simulate each step to analyze the effects of technology changes. Their preliminary results estimate that automation replaces some routine tasks, leading to estimated declines in the number of production floor jobs requiring medium skill levels. According to the authors, this firm-level, task-based approach may be applicable to other manufacturing industries and could provide insight on how the adoption of different technologies may produce different labor outcomes. However, they note that the approach requires detailed production process data, which may be difficult to collect for many firms or industries. Commerce’s Census Bureau has begun administering surveys with questions that focus specifically on firms’ adoption of advanced technologies and resulting workforce changes. According to Census, this data collection is part of a long-standing, coordinated effort to measure the impact of technology. In addition, consistent with Commerce’s strategic plan, these represent new efforts to provide a timely, in-depth, and accurate picture of the economy amidst the economic shifts and technological advances of the 21st century. However, none of the survey results will be available until late 2019 and later. The new Annual Business Survey (ABS) is a joint effort by Commerce and the National Science Foundation that has the potential to provide insight on the spread of advanced technologies in the economy and could be used to examine the workforce effects of technology adoption, but the first ABS results are not expected until late 2019. Census administered the 2017 ABS in June 2018 to collect information on firms’ use of advanced technologies, such as automated guided vehicles, machine learning, machine vision, and robotics, among other things (see example in sidebar). The survey asks whether firms are testing a given technology or using it for either less than 5 percent, 5 to 25 percent, or more than 25 percent of their production or service. Census officials said this question should provide information about the extent of technology adoption nationwide, including whether there are any industry concentrations of advanced technologies. Census plans to add questions on the workforce effects of advanced technologies when it administers the 2018 ABS during July through December 2019, pending final approval by the Office of Management and Budget. Census plans to release these survey results in December 2020. Specifically, Census plans to include new questions that ask firms about: (1) their use of advanced technologies such as AI, cloud computing, robotics, and specialized software and equipment; (2) their motivation for adopting and using artificial intelligence and advanced technologies; (3) the impact these technologies might have on the number and skill level of workers; and (4) the factors that could adversely affect the adoption or production of these technologies. The new questions also ask about changes in the number of production workers, non-production workers, supervisors, and non-supervisors. These new questions could be used to characterize the prevalence of workforce changes in the economy caused by advanced technology adoption (e.g., declines in production workers, or increases in supervisory workers) and whether this differs by industry sector. However, these planned questions are not intended to provide information to quantify the magnitude of workforce changes, in part to minimize respondent burden and potential survey error, according to Census. In addition, until the ABS data are available and evaluated, it remains unclear what limitations, if any, the data may have. Census also plans to expand other surveys to track the spread of advanced technologies in the economy, including its Annual Survey of Manufactures (ASM) and Annual Capital Expenditures Survey (ACES). Census plans to administer the 2018 ASM in May 2019, pending final approval by the Office of Management and Budget. The survey will collect capital expenditures data for industrial robotics at approximately 50,000 manufacturing plants, as well as the number of industrial robots purchased by and in use at these plants. Census officials stated these two measures might be useful in understanding the impact that industrial robots could have on productivity as well as the impact robots could have on the manufacturing labor force once the survey results are available in the spring of 2020. Census plans to administer the 2018 ACES during March through May 2019 and to have the survey results available in February 2020.The survey will include questions on robotics expenditures, similar to those in the 2018 ASM. However, the ACES collects expenditure data from 50,000 employer firms across all non-farm sectors of the economy—instead of just manufacturers—and will also ask about firms’ use of both industrial and service robots. Some Commerce offices also track issues related to the adoption and workforce effects of advanced technologies on a limited or intermittent basis. For example, National Institute of Standards and Technology officials stated that the Hollings Manufacturing Extension Partnership collects limited information about the number of jobs gained and retained by small and medium businesses adopting new technologies. National Telecommunications and Information Administration officials said they monitor developments in AI on an intermittent basis and also direct a project that examines new applications of small and large internet devices. DOL has a role in collecting data that track changes occurring in the U.S. economy and workforce, including developing new ways to track emerging economic trends, though as we previously discussed, currently available federal data do not link shifts in the workforce to technological changes. BLS is the principal federal statistical agency responsible for measuring labor market activity. According to DOL’s strategic plan, BLS is to support public and private decision-making and meet the needs of its many stakeholders, including the general public, educational institutions, and the public workforce system. This includes regularly identifying structural shifts in the economy and developing new data products that reflect economic changes. In addition, DOL’s Employment and Training Administration (ETA) is to assist workers’ entry and reentry into in- demand industries and occupations. This assistance includes providing job seekers with accurate labor market data and guidance about opportunities, aligning training services to industry needs, and helping connect businesses with properly skilled workers. Internal control standards state that agencies should use quality information to identify, analyze, and respond to significant changes, including external conditions such as economic and technological changes that may affect an agency’s ability to achieve its objectives. DOL collects workforce data through various surveys, including the Current Population Survey’s Displaced Worker Supplement, and produces other data products such as the occupational employment projections and Occupational Information Network database that include information related to advanced technologies. However, these data are limited, and according to BLS, provide some, but not all, of the information required to assess the impact of automation on the workforce. BLS’s Employment Projections program identifies and provides limited information about occupations expected to experience declines in their share of employment in an industry or group of industries as a result of the adoption of advanced technologies. On a biennial basis, this program analyzes changes in the economy to project how employment by occupation may change over 10 years, including which occupations may be affected by advanced technologies. Factors that can affect occupational employment include but are not limited to technological innovation; changes in business practices or production methods; organizational restructuring of work; changes to the size of business establishments; and offshore and domestic outsourcing, according to BLS. As part of this program, BLS develops a table of occupations that are projected to have direct employment changes due to some identified reason. This table identifies projected staffing pattern changes and BLS’s qualitative judgment of the most significant factor or factors projected to affect the occupation. The table also indicates whether an occupation’s share of employment is expected to change within a single industry or within multiple or all industries. For example, the table includes the following selected entries: Librarians: Employment share is projected to decline in the information services industry as internet-based research continues to displace library-based research. Stock clerks and order fillers: Employment share is projected to decline in two industries (the warehousing and storage industry and the grocery and merchant wholesalers industry) as firms increasingly adopt automated storage-and-retrieval systems. Aircraft structure and systems assemblers: Employment share is projected to decline in all industries as collaborative robotics increase efficiency, producing more output with the same amount of labor. We identified 100 occupations in BLS’s table that are projected to experience declines in their shares of employment in an industry or group of industries as a result of the adoption of advanced technologies. Similar to the examples above, reasons could be related to automation, the increased use of robots or artificial intelligence, advances in machine or software technologies, or other changes resulting from the adoption of advanced technologies. As shown in figure 4, most of these occupations are production occupations (40 of 100) or office and administrative support occupations (30 of 100). BLS officials told us they do not currently track groups of occupations projected to experience employment share declines due to specific reasons, such as advanced technology adoption. Officials also said they do not aggregate total projected employment effects stemming from similar causes because they are unable to identify ripple effects in all occupations—e.g., automation in one occupation affecting employment in a different occupation. Information contained in ETA’s Occupational Information Network (O*NET) database includes, among other things, information about work activities, tools and technologies used, and required skills associated with over 1,000 occupations. According to ETA officials, the primary purpose of O*NET is to assist job seekers in making employment decisions. However, the O*NET database can be used to identify occupations that use certain types of advanced technologies. For example, we identified 15 occupations in which workers monitor, install, develop, troubleshoot, debug, or perform other tasks with robots as part of their daily work activities and 63 occupations in which workers use robots as a tool or technology in their daily work activities (see table 1). In addition, states, federal officials including at BLS, and academic researchers use these data to inform, among other things, worker support programs. DOL officials told us they do not use O*NET data to analyze changes in occupations over time, such as robots being used in additional occupations, because the methodology is not currently structured to capture these kinds of changes systematically. For example, data are collected from a selection of occupations at varying frequencies, rather than at the same time, which could make it challenging to track changes in certain occupations over time. Without comprehensive data linking employment shifts and technological changes, policymakers and DOL may not be prepared to design and implement programs that both encourage economic growth and provide support for workers affected by changes. DOL-funded programs rely on accurate information to guide job seekers to employment opportunities and to help align training services with local industry needs. For example, the O*NET database identifies high-growth, high-demand occupations for job seekers based largely on BLS employment projections data. While these employment projections provide valuable information, they are not designed to identify the full extent of occupational shifts due to advanced technology adoption. Similarly, other workforce surveys, such as the Current Population Survey’s Displaced Worker Supplement and the Job Openings and Labor Turnover Survey, do not collect information about the causes of job losses and gains. This information could be a valuable tool for designing programmatic or policy supports for workers. For example, data on whether advanced technologies have resulted in worker displacements, work hour reductions, or substantial adjustments to work tasks could better position BLS to meet stakeholder needs. Congress has expressed concern that there continues to be insufficient data on the effects advanced technologies are having on the U.S. workforce. On January 2, 2019, BLS reported to Congress that it plans to work with a contractor during fiscal year 2019 to study the interaction between labor and capital in the workplace and how it is affected by new technologies; identify ways to supplement BLS data with additional information on automation; and produce a report that recommends data collection options to fill those gaps. In fiscal year 2020, BLS also plans to identify pilot projects to test the feasibility of new data collection based on the recommendations in its final report, resources permitting. However, these plans are still in their early stages, according to BLS officials. Officials at Commerce and DOL stated that collecting data on the adoption and workforce effects of advanced technologies is challenging because it is difficult to identify which new and emerging technologies to track; employment trends generally occur at the occupation and industry levels but the effects of advanced technologies typically occur at the task or job level; and employment trends have a complex and diverse set of causes. Specifically: Identifying which new and emerging technologies to track. Census officials said there is uncertainty about how an emerging technology might affect the economy and thus whether it should be tracked systematically. For example, self-service technology appeared at grocery stores in 1916, other self-service technology appeared at gas stations later, and more recently self-service technologies are being adopted by some restaurants, according to researchers. Periodically, Census has included questions in its firm surveys about the use of these technologies. Past surveys asked questions about the use of self-service at gas stations until the technology became ubiquitous and was dropped from the survey. As self-service technologies have expanded to other areas of the economy such as restaurants, Census has again added questions about self-service to recent surveys because information is lacking on the growth of this phenomenon. Trends and effects appear at different levels. BLS officials said employment changes due to technology typically occur at the individual task or job level and employment trend data are at the industry and occupation levels. Officials also said that identifying technology-related effects in occupations, such as changes related to uses of machine learning algorithms, is difficult because some workers within an occupation might be affected by the technology while others might not. For example, some computer scientists and engineers might be involved in the development or application of machine learning algorithms while others are not. Causes of trends are complex and diverse. BLS officials said that employment trends’ complex and diverse causes make it difficult to identify occupations that are changing because of advanced technologies. Changes in one occupation may have ripple effects in other occupations. Partly as a result of this complexity, BLS’s Employment Projections program identifies examples of technology- impacted occupations, but it does not attempt to identify all instances where technology impacts occupations nor does it attempt to quantify an overall projected employment effect of advanced technologies. The White House Office of Science and Technology Policy (OSTP) is responsible for coordinating AI related policy across government agencies and for overseeing the National Science and Technology Council’s subcommittees and their ongoing activities. For example, the Subcommittee on Machine Learning and Artificial Intelligence was originally chartered in 2016 to monitor machine learning and artificial intelligence and to watch for the arrival of important technology milestones in the development of AI, among other things. OSTP officials told us that the Subcommittee has been re-chartered, now receives direction from OSTP’s Select Committee on Artificial Intelligence, and is presently focused on federal resources related to AI research and development. Selected firms generally adopted advanced technologies through a phased process of innovation and technology adoption (see fig. 5). We met with officials representing 16 firms that are using advanced technologies and a systems integrator who spoke for a number of his customer firms. Many firm officials described the path to integrating technology into operations as lengthy, complex, and iterative. For example, some firms we visited have had to build and test different mechanical “grippers” attached to robot arms to pick up and handle particular objects; one firm had high school participants at a local training center develop a gripper solution for one of the firm’s robots. Some of the large firms we visited had their own internal teams that identified, tested, and integrated advanced technologies. Other firms we visited used third- party integrator companies to help with incorporating technologies into their operations. We spoke with firm officials about their motivations for adopting advanced technologies, as well as challenges they faced throughout the process, and they identified a number of similar issues. Most selected firms cited cost savings as a primary consideration for adopting advanced technologies. Firm officials discussed cost-related motivations in various forms, such as remaining competitive in a global economy, increasing productivity (i.e., lower cost per unit), decreasing labor costs, and saving on physical space. Firms said they adopted advanced technologies as a way of reducing operational costs—including labor costs—to increase competitiveness and profitability. Some officials also specifically identified the pressure of large low-cost competitors, both in the United States and globally, as a major motivation to reduce costs and product prices. Officials at a medium-sized door manufacturer told us that increased use of advanced technologies, such as robots, enabled the firm to increase efficiency, reduce labor costs, and re-focus its product line on custom doors to survive the entry of manufacturers in China that could sell mass-produced doors for lower prices. The original motivation for adopting robots at a medium-sized automotive parts manufacturer was a customer’s price demand that the firm could not meet and still remain profitable, according to officials. Integrating more robots enabled the firm to reduce production costs by using fewer workers. At a large manufacturing corporation of household and personal care goods, officials told us the company had a goal of reducing its workforce size by 1,500 full-time positions per year for 5 years (across its subsidiaries), and specifically using robotic automation to accomplish 40 percent of its reduction goal. The constant pressure to keep costs low in the health care sector motivated a university-affiliated medical center we visited to explore adopting more advanced technologies, such as autonomous mobile robots that could decrease expenses by reducing the number of positions in some departments. Firm officials also told us about other, non-labor-related cost savings considerations that led to the adoption of advanced technologies. Officials at a large automotive manufacturer told us they recently upgraded a laser welding system to use fewer, more advanced robots to save production line space—which is a valuable commodity in manufacturing. They also pursued this change to increase overall production capacity because the physical space they saved could be used to install more robots for other production steps. The integration of autonomous mobile robots to deliver prescription drugs to patient wards at a university-affiliated medical center was intended, in part, to save costs related to medicines that go missing when delivered and processed manually, according to officials. According to officials at selected firms, the desire to improve jobs led firms to adopt advanced technologies. The firms wanted to automate tasks that are dangerous, difficult, dull, or dirty in large part to improve worker safety, and to optimize the value added by workers. For example: Dangerous work: Two robots were installed to pick up doors weighing between 90 and 300 pounds, and place them on a paint line at a medium-sized door manufacturer we visited. Prior to the robots, workers who performed this dangerous task experienced work related injuries, and the firm paid large amounts of money in workers’ compensation claims, according to officials. Once the robots were installed, the firm experienced a decrease in the number of worker compensation claims. Dull work: A small automotive parts manufacturer we visited installed an industrial robot to perform a machine-to-machine transfer of a heavy part. Prior to the robot, the firm had three workers performing this task—even though the task only required two—because workers would eventually quit due to the tedium of the job and new workers would require time to be trained, according to officials. Value-added work: Some officials told us they adopted advanced technologies because they wanted to maximize human labor that provided value to the firm and reduce labor that did not. Officials at a warehouse for a regional grocery store chain and a university- affiliated medical center said they wanted to minimize time workers spent traveling between tasks (as opposed to performing tasks). Warehouse officials said their workers spend up to 60 percent of their time traveling back and forth between shelves and products, which is time that could be spent selecting and sorting items. Thus, at the time of our visit, the warehouse was in the early stages of adopting automated guided vehicles to eliminate the need for workers to travel between points. Similarly, officials at a university-affiliated medical center that adopted autonomous mobile robots to transport, among other things, prescription drugs, said nurses and pharmacy technicians used to walk back and forth between the patient ward and the pharmacy to pick up and deliver these drugs, which diverted them from performing other tasks. They said that the medical center wanted them to have more time to provide valuable work, especially for employees who are highly-paid. Officials at many firms said that adopting advanced technologies can help them deal with the challenges of recruiting and retaining skilled workers. They explained that worker shortages and high turnover can result from skill gaps in the local or national workforce, low unemployment, and certain work being viewed as unappealing, among other reasons. For example, officials at a warehouse for a regional grocery store chain we visited told us they struggle with high worker turnover and the constant need to hire new workers. In addition, low unemployment can make it difficult to retain workers with the right skills to operate machinery according to officials at a small automotive parts manufacturer. Similarly, at the university-affiliated medical center, an official said that positions for pharmacy and other types of medical technicians can be difficult to fill. By using autonomous mobile robots to automate some tasks, the medical center can streamline its operations to more efficiently use the technicians it already has. Recruitment in Manufacturing Officials at some manufacturing firms we visited said they have had trouble attracting new workers into the sector, and officials at two firms said that adopting advanced technologies is one way they have sought to make manufacturing more attractive and to appeal to more and younger workers. One younger worker at a small automotive parts manufacturer talked about how appealing his workplace was due to the firm’s use of advanced technologies, specifically robots. Officials at a large automotive manufacturer viewed their tech development facility, which includes spaces to tinker with virtual reality, augmented reality (i.e., technology that superimposes images on a user’s view of the real world; for example, by wearing augmented reality glasses), and other emerging technologies, as an asset to recruit young talent. Improving product quality, expanding product offerings, and supply chain reliability were primary motivations for adopting advanced technologies, according to officials at some firms. Product quality: Quality is paramount in the automotive industry, where mistakes are costly and can have implications for a firm’s reputation, according to officials at a medium-sized automotive parts manufacturer we visited. For this reason, they decided to use robots rather than workers for welding in order to standardize the processes, reduce errors, and improve product consistency and quality. Officials at a large automotive manufacturer similarly said that the firm has pursued machine learning technologies to ensure fewer defects and problems in vehicles. Engineers at the firm are developing a smart watch for workers who connect wires that will provide feedback to these workers if a proper connection is not made, based on the sound of the connection. The firm is already using machine vision technology that inspects vehicles as they pass through a section of the production line to ensure the correct pieces have been used for each vehicle model. Expanding product offerings: At a medium-sized fruit processing plant, an official said that integrating robots, an advanced conveyer system, and machine vision inspection technologies, among other advanced technologies, enabled the firm to begin producing applesauce in a highly automated and safe way. Had manual production been the only option, officials said they would not have considered producing applesauce due, in part, to safety issues. Supply chain reliability: One small manufacturer of rubber stamps and embossing seals (hereafter referred to as a small stamp manufacturer) used to rely on a single supplier for pre-cut materials, which was not always reliable. The firm adopted a collaborative robot, in part, so it could purchase raw materials directly and then have the robot cut the materials as part of the production process (see fig. 6). In addition to the capital cost of advanced technologies, which some firms told us can be substantial, firms face a number of risks that can affect their return on investment, such as the reliability of technology and working with new tech developers. While the firms we met with had already adopted advanced technologies, officials had to consider and overcome various risks during the adoption process. Some of these firms decided against adopting other advanced technologies upon evaluating these risks. Being an early adopter of a technology is risky because the new technology may not yet be sufficiently reliable for firms’ operations. Officials at a large appliance manufacturer we visited showed us technology that was supposed to use machine vision to autonomously inspect the wire connections for clothes dryers. They told us that the vision technology had been ineffective, so they took it off the production line for engineers to continue working with it in the lab; they planned to bring the technology back onto the line a few weeks after our visit. Officials at this firm said that the vision technology was still relatively immature, as it had a limited field of vision and yielded numerous false readings. Similarly, a warehouse we visited that invested in automated guided vehicles used them to move pallets for a short time, but then put them into storage because these vehicles did not have mature enough machine learning and vision capabilities for the firm’s purposes. Eventually, officials from this warehouse began working closely with the developer firm to improve the vehicle technology, which advanced enough that it could be used. For instance, officials from the warehouse suggested adding turn signals to the vehicles to alert nearby workers of intended movements and improving the vehicles’ ability to travel over spills without triggering the system’s sensors to shut down. Firm Size Might Affect Risk Tolerance An official at one small manufacturing firm stated that larger firms may be more willing to be early adopters of technology, as they may be able to absorb the high risks of experimenting with expensive technologies, while smaller firms tend to wait until a technology has been optimized before deciding to adopt it. Accordingly, his firm only purchases industrial robots from an established manufacturer, although it would like to experiment with newer technologies in the future, such as augmented reality. Officials at a large manufacturing firm told us they have purchased a number of advanced technologies to experiment with, even though they do not know yet how the technologies may ultimately be used in their production process. This firm also has teams of technicians and engineers who can adapt the technology for operations. During our visit, we met with engineers who demonstrated different potential applications of technologies that are still being tested, including using virtual reality to test new part design and augmented reality glasses to provide interactive training to workers. Officials at some firms explained that installing advanced technologies at times necessitated building manual redundancies into their operations due to reliability concerns. Officials at a construction consulting company and a municipal township that adopted a machine learning technology to inspect roads said the technology would miscategorize road quality at times, such as identifying tree branch shadows on the road as pavement cracks. While working with the developer to improve the technology, officials said they continued to conduct redundant manual inspections to ensure they were making road repair decisions based on accurate information. During our visit to a large appliance manufacturer, we saw multiple collaborative robots that were not working properly. As a result, workers were performing these tasks manually while the robots were down; officials told us that each of the firm’s automated processes has workers trained to perform the tasks in case a technology was not working properly. Technologies Viewed Differently by Firms Some firms find a technology to be useful while others find little practical application for that technology, as illustrated by the various opinions firm officials had about collaborative robots. Officials at one small manufacturer we visited said that a collaborative robot was well suited for the firm’s production process and environment because, among other reasons: (1) the firm produces small durable goods that require dexterity rather than speed, which the collaborative robot could provide; (2) the collaborative robot would be safe around workers and could be trained by non-technical staff, so the firm’s small workforce could adapt to its use; and (3) the collaborative robot could fit in the firm’s limited floor space, as it would not require a cage. On the other hand, officials at other manufacturing firms we visited told us that collaborative robots were less useful in their settings because they have significant weight and speed limitations in order to be safe enough to operate outside of a cage, limiting their usefulness for their firms. Some firm officials told us it could be risky to work with tech developers with limited experience. Officials at a large appliance manufacturer said that newer developers may go out of business or be bought out by a larger firm, which could render the technology acquired from them obsolete (especially in terms of future servicing of parts and software updates). The officials stated that emerging technologies, both hardware and software, tend to not be standardized, so investing in a developer likely means investing in a type of technology that may not be supported by other developers if issues arise. We heard from some firms that they purchased technology from developers who already had established reputations and longevity. For example, a small manufacturer of durable goods selected a robotics company because of the founder’s reputation and track record, among other reasons. Operational slowdowns: The time period between initial adoption and optimization of a technology varies widely and can sometimes be a lengthy and ongoing process, according to officials. One small stamp manufacturer experienced a lengthy and iterative implementation process for an off-the-shelf collaborative robot they purchased. For example, they had to construct a customized environment for the robot to function in, make parts by hand, purchase a 3-D printer to develop tools for the robot, and build additional parts to take care of increased byproducts like sawdust. Officials at a large automotive manufacturer told us that new technology, such as machine vision technology used for automated inspections, is often integrated on the weekends or during off-shifts. Then, on the first day of production after the new technology is integrated, the production line starts slowly and speeds up as worker comfort and experience increases. Outside of manufacturing, a consultant that helps facilitate the adoption of advanced technologies at firms said that firms’ existing, or legacy, computer infrastructure can be a barrier to integrating machine learning technology, increasing complexity and causing an extended implementation process as his firm integrates the new technology platform with the legacy infrastructure. Worker concerns: Officials at some manufacturing firms said they have encountered worker concerns with advanced technologies, and have employed various tactics to mitigate this, such as introducing workers to the technology in offsite demonstrations and involving them during the decision-making and planning before the technology was integrated. In one case, workers were able to ask questions about a collaborative robot as it was being installed and were provided with orientation training. The robot was then phased into operations—used initially for short periods of time so workers would become accustomed to its physical presence and proximity to their workstations. Deciding Not to Adopt Advanced Technologies Officials at the firms we visited identified instances in which they chose not to adopt certain advanced technologies, or not to use advanced technologies that were working well in other processes. Reasons we heard included: a product line had too much variation to benefit from advanced technologies (i.e., that some advanced technologies work better for standardized products and processes); a certain manufacturing process was too low-volume to invest time and resources into automation; and human dexterity is difficult to replicate. Officials from a large appliance manufacturer showed us an instance where using automation would not make sense. We observed a worker performing a simple, single task: grabbing a metal heat shield and plastic dishwasher spinner from separate bins and clipping one on to the other. Because of the shape of the pieces and because they were lying unorganized in boxes, the task requires human dexterity, making the process difficult to automate, according to officials. Officials at many of the firms we visited said they needed fewer workers in certain positions after adopting advanced technologies to perform tasks previously done by workers. Officials at these firms generally told us they adjusted by redeploying workers to other responsibilities and, in certain instances, reducing the firm’s workforce size through attrition. We also heard examples of direct layoffs due to the adoption of technologies. There may also be other types of adjustments firms can make that we did not observe or discuss with these officials. The complexity of these workforce adjustments makes it difficult to determine or measure the effects of technology adoption on workers. For example, although workers may not have lost their jobs due to an adopted technology taking over specified work tasks—either because of redeployments or attrition— fewer job opportunities might be available in the future for workers with similar skills. In addition, the iterative and sometimes lengthy process of incorporating advanced technologies can delay workforce effects. Thus, the absence of short-term effects of technology adoption does not necessarily preclude long-term implications, such as reductions or slower growth rates in workforce size over time (see text box below). As discussed in the prior section, one reason firm officials are motivated to use advanced technologies is to decrease labor costs. Slower Workforce Growth than Revenue Growth An official from a small automotive parts manufacturer told us that advanced technologies and automation resulted in revenue increasing by more than 400 percent over the last 12 years while the workforce increased about 15 percent. Production workers now make up a smaller percentage of the overall firm workforce than prior to automation, and sales and support staff now make up a greater percentage. The firm official described this change as an increase in higher-skilled jobs and a decrease in lower-skilled jobs. Similarly, according to firm officials at a different medium-sized automotive parts manufacturer, revenue has grown six times in the past 15 years while the workforce has grown four times, largely as a result of adopting robotics technology. Redeployments without job loss: When advanced technologies replaced positions, firms we visited often shifted, or redeployed, workers to different responsibilities. For example, officials at a medium-sized automotive parts manufacturer we visited told us they had nine workers who smoothed sharp edges and removed burrs on hydraulic cylinders prior to installing two robots to perform these tasks. Now, with the robots in these positions, three workers load the robots and then inspect and de- burr any parts of the cylinders the robots missed. The other six workers were redeployed to other tasks, according to a firm official. At a large appliance manufacturer we visited, officials told us that two workers used to move large parts from one line to another line to be painted. Now, as we observed, a collaborative robot performs this function alone; a worker monitors the operation to ensure it is running smoothly, and the original workers were moved to different tasks on the production line, according to officials. Although the size of these firms’ workforces did not decrease as a result of the technology adoption, the numbers of certain positions were adjusted—for example, production positions decreased while monitoring positions increased. Differences in skills required for these positions may also affect the ability of current workers to transition and could have implications for individual workers even though the number of jobs at the firm does not change. These sorts of changes may or may not appear in firms’ reported employment data, depending on whether redeployed workers change occupations or what other workforce changes may be occurring simultaneously (e.g., if other production workers are hired for reasons unrelated to the technology adoption). Redeployments with job loss through attrition: Officials at some of the selected firms that redeployed workers said they also reduced their overall workforce size through attrition, as a result of adopting advanced technologies. Autonomous mobile robots independently transported biohazardous waste, linens, meals, and prescription drugs throughout the university- affiliated medical center we visited. Officials told us they eliminated 17 positions after they deployed the robots. No workers were laid off; instead, they relied on high staff turnover rates and moved workers to vacant positions elsewhere. At a medium-sized fruit processing plant, firm officials told us they replaced 150 to 200 jobs with various advanced technologies over the past 3 to 4 years. However, they relied on attrition rather than layoffs. For example, the plant adopted a robot to pack food into boxes. Prior to using the robot, officials told us there were 26 workers per shift performing this job; as of our visit, there were 13 workers per shift. A medium-sized door manufacturer reduced its workforce from 650 employees to less than 500 over approximately the last 20 years due to, among other things, their adoption of robots, according to firm officials. For example, we observed industrial robots that load steel sheets into a cutting machine, reading a barcode on each sheet that tells them what size sheet is being lifted and how it should be placed in the cutting machine. This process only requires a single worker to monitor the robots during each of two shifts, where previously three workers per shift were on this production step (i.e., a change from six to two workers total). How quickly workforce reductions materialize for firms using attrition can vary greatly. We visited firms with low employee turnover rates and firms with high turnover rates. High worker turnover rates allowed some firms to more quickly adjust their workforces when deploying advanced technologies and may be a reason we were told about job loss through attrition rather than layoffs at these firms. Job loss through layoffs: An official from a systems integrator firm (“integrator”) provided examples of significant layoffs as a direct result of advanced technologies. This integrator provides machine learning technology and other similar products to automate office and administrative processes, among other things. One of the integrator’s customers—a U.S. automotive parts firm facing competition from online retailers—adopted machine learning technology to take over its accounts payable and distribution system. As a result, according to the integrator’s official, this firm reduced the number of employees in one of its U.S. offices from 500 to 200. Another of this same integrator’s customers—a firm that sells telecommunication circuits—adopted machine learning technology to automate product returns processing. As a result, the firm experienced a 30 percent reduction in customer care calls, and replaced about 150 jobs in a U.S. call center with 110 jobs at a call center in a different country (i.e., about 150 U.S. jobs lost; and an overall workforce reduction), according to the integrator’s official. According to officials at some selected firms, greater competitiveness and productivity due to the adoption of advanced technologies (see sidebar) has helped firms grow their workforces. For example, some hired additional production workers due to increased production (despite some production tasks being taken on by the adopted technologies), or new types of workers, such as technicians to maintain the technologies. Some officials also said that although they may not have grown their workforces, adopting advanced technologies helped them stay in business by allowing them to compete effectively, and thus to preserve jobs and retain workers. For example, officials at a medium-sized door manufacturer, where we observed numerous robots in the production facility, told us that their firm “could not survive” global competition without the use of advanced technologies. Productivity and Efficiency Gains Adopting advanced technologies has helped some firms improve their product quality and increase their production efficiency. For example, according to officials at a medium-sized fruit processing plant, after the firm began using an automated fruit grading technology, the process took significantly less time and resulted in far fewer complaints from farmers about the grading. Farmers thought the automated grading technology was fairer and more accurate than having workers manually and subjectively grade the fruit. A large appliance manufacturer that began using a collaborative robot to apply sealant to an appliance door observed improved consistency, which led to fewer service calls from retailers and customers about excessive, insufficient, or incorrect seals. One medium-sized door manufacturer said that automation technologies enabled them to produce and ship doors in 3 days, as opposed to 4 to 6 weeks. An official from a warehouse for a regional chain of grocery stores said that using automated guided vehicles allowed the firm to save time moving pallets from one end of the warehouse to the other, and also save worker hours. The warehouse saves just over $2 per pallet moved by an automated guided vehicle rather than a worker, and up to $3,500 a day based on volume, according to the official. Advanced technologies enabled some selected firms to increase production or produce a larger range of goods, and thus to hire additional production workers. This also led to workforce increases for suppliers and other firms, according to officials. One large appliance manufacturer increased its use of robots and other advanced technologies to produce more of its own component parts internally instead of relying on suppliers. As a result, the firm was also able to increase the number of production jobs, according to firm officials. Due to advanced technologies, a small automotive parts manufacturer was able to bid on a contract to produce a new and more intricate part for a major automotive manufacturer. An official described how the part was so intricate that it could not have been produced manually with the required level of consistency and speed. Although the firm adopted six robots to produce this part, winning the contract also created nine new jobs. While the robots are completing much of the production, the volume of parts demanded and the existence of some tasks that only workers can complete has led to this job growth. A developer of autonomous mobile robots said that, as a result of increased business, his firm has created jobs among its eight local suppliers where he buys parts, such as motherboards for the robots. Growth of Developer and Integrator Firms Selected developer firms we met with said they grew their technical and non-technical staff as a result of increasing demand for their technologies. A firm that develops and produces robots had tripled its workforce size, to about 130 employees, in the last year alone, according to officials. An official at another developer firm that makes inspection robots said they had grown from three workers to about 20 and envisions expanding to 100 in the near future. The official said that the firm’s first years were spent on technology development, but that once the technology was deployable to customers, the firm grew its workforce size. Integrator firms that help companies adopt advanced technologies have also grown in size, and new types have emerged, according to integrators we visited. For example, with the development of smarter robots, one integrator firm we visited entered the industry to recondition and sell old robots; the firm also adds newer technology to these robots if requested. This integrator has grown from 35 to 45 employees in the last 10 years, according to officials, with the new positions being primarily robot technician jobs. As a result of technology adoption, some firms hired more workers with technical skills, and in other instances lower-skilled workers, according to firm officials. An official from a warehouse for a regional chain of grocery stores said that adopting an advanced automation system created a need for three additional workers to provide preventive maintenance on the machines. These additional positions pay about 25 percent more than the standard warehouse positions, according to officials. At a large automotive manufacturer, officials told us the firm increased its number of lower-skilled cleaning jobs when robots began producing large amounts of byproduct. At the firms we visited, workers changed roles and tasks as a result of advanced technology adoption, such as focusing more on interactive, cognitive, higher-skilled, and monitoring tasks, and in other cases focusing more on lower-skilled tasks. Workers who can adapt and be flexible to task changes may experience positive effects, including work that is less physically taxing, safer, more ergonomic, less monotonous, or higher paying. On the other hand, workers who are unable to adjust to changing tasks may be negatively affected. Officials at some of the firms told us that their firms provided internal training or leveraged external resources to develop workers’ skills to help them move into new positions. During our visits to selected firms, we saw a variety of ways in which tasks for workers are changing. Interactive work: The use of autonomous mobile robots to deliver prescription drugs for patients enabled nurses at the university-affiliated medical center we visited to focus more of their time on patient interaction, according to officials. The small stamp manufacturer we visited would like to continue to automate its ordering process and focus more on providing customer service. Officials there said for future hires, they plan to recruit for data and people skills, rather than production skills. Cognitive work: A federal statistical agency adopted machine learning technology to automatically interpret text narratives on forms and assign codes to the data. As a result, staff who previously entered this information manually are able to spend more time on analytical tasks such as reviewing the accuracy of the auto-coding, correcting issues, obtaining clarifications about information submitted on the forms, and following up with non-respondents, according to officials. Higher-skilled work: At a large automotive manufacturer, due to increased use of advanced technologies, workers who are hired today need to have greater technical proficiencies than workers hired in the past. For example, to adapt to their changing roles working with robotic equipment, non-technical production staff need machine maintenance and technical skills, rather than only manual dexterity skills. Officials at a large appliance manufacturer that adopted an automated machine to stamp metal said that the resulting process required a single worker to monitor the machine and provide basic maintenance. This worker needed technical skills and at least 6 months of training to effectively perform these duties. In contrast, at another one of this firm’s global plants, four separate pressers are used and each requires workers to load and unload metal. Monitoring work: Officials at the large appliance manufacturer mentioned above showed us a step in their production process in which two small pieces of plastic and metal need to be attached. Three workers used to perform this task by hand, which caused ergonomic challenges, and inconsistencies in both quality and production cycle times. Now, the firm uses three robots to perform this work and a single worker loads the pieces for all three robots and monitors their performance. At a small automotive parts manufacturer, production operators who work in cells with robots monitor multiple machines and sometimes also monitor multiple work cells, so a greater aptitude level is needed. As a result, these operators earn $3 per hour more than operators in work cells without robots, according to a firm official. Less physically taxing work: Staff at some firms also told us how advanced technologies have made worker tasks less physically demanding. For example, we talked with one warehouse worker who used to lift heavy boxes, but who now operates a forklift after his old task was automated with a conveyer belt and sorting system. He described his new position as having ergonomic benefits, including experiencing less back pain. At a large automotive manufacturer, officials said the firm installed six robots to paint vehicle interiors. This production step was a major ergonomic hazard and workers who did this painting had a relatively high injury rate, according to officials. Officials told us that adopting the robots lowered the injury rate among these workers and resulted in faster vehicle painting. Simplified work: At a small stamp manufacturer that adopted a collaborative robot, officials told us that as the firm continues to redesign and optimize operations, the robot will take on more complex tasks. As a result, the remaining production work performed by the firm’s production worker will be simpler (see fig. 7). Officials said that in the future, after the firm’s current production worker retires, the firm may rely on contingent workers to perform any needed production work not completed by the robot because the tasks will be simpler and easier to train a new, temporary worker to complete. Officials said the firm may also hire a worker with a different and more varied skillset who can perform the few remaining production tasks along with other types of tasks. Lower-skilled work: Officials at a medium-sized door manufacturer installed a robot to facilitate the firm’s redesigned door sealant system and production process. The original process of manually applying door sealant was physically-intensive, ergonomically challenging, and required significant skill and experience to precisely apply the sealant. With the new design, a robot applies the sealant autonomously. As a result, workers perform lower-skilled tasks in this process, including placing a piece on a platform, visually inspecting the robot’s work, cleaning and setting up the robot’s work station, and confirming the correct program is entered in the computer. Adaptability to changing daily work demands: Officials from selected firms told us that due to advanced technology adoption, workers need to change tasks depending on the day and circumstances. For example, at a large appliance manufacturer some workers serve in different capacities depending if the robots are functioning properly and depending on the production needs of that day. On the day we visited the plant, several of the robots were malfunctioning and workers were performing the robots’ tasks. Firm officials said that some of their workers serve in swing roles and move around to different production processes and assist as needed. Training Centers for Advanced Tech Skills We met with officials at a training center that re-trains adults and teaches high school students to work with advanced technologies used in manufacturing. We visited two firms in the area that told us that this training center helps fill a local shortage in maintenance technician skills, and that they have hired workers who graduated from the center. Officials at the training center said that there is a high demand in the area for maintenance technicians. For example, they said that a large automotive manufacturer in the area is planning to hire 800 maintenance technicians over the next 3 years, and that the firm is worried about how it will fill these positions. Officials at the training center also said that some firms have such a high demand for maintenance technicians that they hire high school students who complete the training program before they graduate high school. The training center is piloting its adult training program. The program recruits adults who are underemployed and have some mechanical aptitude, then trains them in advanced technologies used in manufacturing. Most of the students who participated in an early pilot obtained higher paying jobs than those they held before the program, according to officials at the training center. Many firms we visited offered training for workers to adapt to their changing roles and tasks, particularly when the tasks or roles became more technical. Some firms used internal training resources and some leveraged local training centers (see sidebar). Some technology developers also offered training to firms that adopted their technologies. Officials at some firms told us that training current workers for more technical positions was easier than finding workers with the appropriate skills. For example, officials at one medium-sized door manufacturer said they needed highly specialized engineers, but could not find any in the region. As a result, this firm offered tuition reimbursement for workers who were willing to go back to school to become engineers. They also partnered with local community colleges to train students to become future maintenance technicians. Officials at a large automotive manufacturer said that due to increases in the firm’s use of advanced technologies, the plant has needed to hire more technicians. As a result, this firm added programs to its on-site training center to train workers for these roles. The complex job changes we observed at the selected firms we visited are not currently captured in federal data, though they may have significant implications for broader employment shifts. As the primary agencies responsible for monitoring the U.S. economy and workforce, the Departments of Commerce and Labor are aware of the importance of advanced technologies as major drivers of changes. For example, Census’ newly administered Annual Business Survey may provide valuable information in the future about the adoption and use of advanced technologies nationwide and the prevalence of resulting workforce effects. However, comprehensive data on firms’ adoption and use of advanced technologies do not currently exist, which prevents federal agencies and others from fully monitoring the spread of advanced technologies throughout the economy and linking their use to changes in employment levels or structural shifts in the tasks and skills associated with jobs. Observations from our visits to selected firms illustrate the complex and varied workforce effects that result from firms’ adoption of advanced technologies. In some circumstances, technology adoption will lead to increases in different types of jobs and in other cases technology adoption will lead to workforce reductions—either over time or immediately. Regardless of the firm-level workforce effects, worker roles and responsibilities are likely to change as advanced technologies take over tasks that workers previously performed. These changes could positively affect some workers, but could also have negative consequences for other workers, especially those who are unable to adapt to changes. For example, workers whose previous work tasks are automated and who are unable to perform new tasks required of them may need to seek new employment. If these changes occur occupation- wide, across many firms, workers may need to re-train or seek new employment in entirely different occupations or industries. To the extent that these changes are concentrated among occupations susceptible to automation, certain groups of workers (e.g., those with lower education levels) may be disproportionately affected and may lack the opportunity to develop skills needed to enter growing occupations. These workers will be in greater need of programmatic or policy supports, and federal workforce programs will need to be aligned with in-demand skills for the changing economy. Without comprehensive data that can measure the magnitude and variety of these firm-level changes, the workforce effects of the adoption of advanced technologies will remain unclear, job seekers may not be fully informed about their best future career prospects, and federally funded programs to support workers may be misaligned with labor market realities. DOL’s ability to collect information regularly on jobs and workers may enable the agency to fill these information gaps. Specifically, better data could be used by policymakers and DOL to proactively design and fund worker training programs that meet the job needs of the future. The Secretary of Labor should direct the Bureau of Labor Statistics (BLS) and the Employment and Training Administration (ETA) to develop ways to use existing or new data collection efforts to identify and systematically track the workforce effects of advanced technologies. For example, the Secretary could select any of the following possibilities, or could identify others. BLS could expand existing worker or firm surveys to ask respondents whether advanced technologies have resulted in worker displacements, work hour reductions, or substantial adjustments to work tasks. BLS could expand its employment projections work to regularly identify occupations projected to change over time due to advanced technologies. ETA could expand the O*NET data system to identify changes to skills, tasks, and tools associated with occupations, as the information is updated on its rotational basis, and consider how this could be used to track the spread of advanced technologies. (Recommendation 1) We provided a draft of this report to DOL, Commerce, NSF, and OSTP for review and comment. We received written comments from DOL that are reprinted in appendix II and summarized below. DOL and Commerce provided technical comments, which we incorporated as appropriate. NSF and OSTP told us that they had no comments on the draft report. DOL agreed with our recommendation to develop ways to identify and track the workforce effects of advanced technologies. DOL stated that it will continue coordinating with the Census Bureau on research activities in this area, and that it plans to identify and recommend data collection options to fill gaps in existing information about how the workplace is affected by new technologies, automation, and AI. DOL also stated that it plans to release employment projections annually instead of every 2 years, beginning in 2019. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Labor, the Secretary of Commerce, the Director of the National Science Foundation, the Director of the White House Office of Science and Technology Policy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. The objectives of this review were to examine (1) what is known about how the adoption of advanced technologies affects the U.S. workforce; (2) selected federal agency efforts to track and monitor the adoption and workforce effects of advanced technologies; (3) considerations that led selected firms to adopt advanced technologies and the risks they faced; and (4) ways technology adoption has affected the workforce at selected firms. Throughout the report, we use “advanced technologies” as a broad term to describe technological drivers of workforce changes, including but not limited to those identified in the National Academies study: artificial intelligence; machine learning; robotics; autonomous transport; advanced manufacturing; 3D printing; advanced materials; computing power; and internet and cloud technology. The technologies we observed at work sites could generally be categorized as applications of robotics, machine learning (e.g., machine vision or autonomous navigation), or both. However, not all technologies that may affect the U.S. workforce in the future—through automation or in other substantial ways—fall into these categories. Our use of the broad term “advanced technologies” leaves open the possibility that new technologies and other areas of focus are likely to emerge. To examine what is known about how the adoption of advanced technologies affects the U.S. workforce, we explored the extent to which available federal data could identify and measure these effects, and we identified limitations with available data. Because there was no comprehensive data that link employment trends to technology adoption, we used a study by Frey and Osborne to identify a group of occupations susceptible to automation. We then analyzed whether the concentration of these occupations in industries is correlated with growth in tech jobs or employment declines in those industries, whether job displacements are more common in these occupations than in others, the characteristics of workers who hold jobs in these occupations, and the geographic concentration of jobs in these occupations. We analyzed employment data from the Census Bureau (Census) and the Bureau of Labor Statistics (BLS); specifically, the American Community Survey (ACS), the Current Population Survey’s (CPS) Displaced Worker Supplement, and the Occupational Employment Statistics (OES) survey. For more information, see detailed discussions of our data analyses in sections 1-3 below. Identifying occupations susceptible to automation: Using a model that evaluates tasks within an occupation, Frey and Osborne estimate a probability of automation for 702 occupations. They identify occupations with a probability greater than 0.7 as being at high risk of automation. In our analyses, we thus consider this collection of occupations as those susceptible to automation. While there are different studies that attempt to predict what occupations or jobs may be automated in the future, we use the work by Frey and Osborne because it is widely cited and because its results are structured to allow us to identify a broadly inclusive collection of occupations susceptible to automation. The results of our analyses could be affected by using other studies to the extent that they identify different occupations as susceptible to automation. The accuracy of any collection of occupations is limited by the unpredictability of when or if jobs are automated, as well as the fact that occupations are comprised of a variety of jobs, which may experience automation to varying degrees or in different ways. We also reviewed examples of recent and ongoing studies that attempt to measure workforce effects directly attributable to technology adoption. We identified examples of research through interviews with knowledgeable individuals and from among those included in a recent review of the state of empirical work. Our review of studies was not meant to be comprehensive of the research in this area. To identify selected federal agencies’ current and planned efforts to collect data on, and monitor the prevalence and effects of advanced technologies in the economy, we met with the Departments of Labor (DOL) and Commerce (Commerce), as the principal federal agencies responsible for collecting data on the U.S. economy and workforce; the White House Office of Science and Technology Policy (OSTP), which leads interagency science and technology policy-coordination efforts across federal agencies; and the National Science Foundation (NSF), which was involved in the development of the Annual Business Survey. We interviewed officials and reviewed data and information collected by these agencies. We also reviewed the Annual Business Survey’s questionnaire to consider the potential uses of data being collected by the survey, and analyzed data from DOL’s Employment Projections program and Occupational Information Network (O*NET) database to identify information related to the adoption and workforce effects of advanced technologies. Annual Business Survey: The Annual Business Survey was administered for the first time in summer 2018, and collects information from firms about various topics, including innovation and technology use. The survey is a joint effort by the Census Bureau and the National Center for Science and Engineering Statistics within the National Science Foundation and Census plans to administer the survey annually for 5 years. The Annual Business Survey replaces the 5-year Survey of Business Owners, the Annual Survey of Entrepreneurs, the Business R&D and Innovation for Microbusinesses survey, and the innovation section of the Business R&D and Innovation Survey. Employment Projections program: BLS’s Employment Projections program analyzes changes in the economy, among other things, to project how employment by occupation may change over the next 10 years, including which occupations may be affected by advanced technologies. BLS’s projections are for the most part structured around the Occupational Employment Statistics, which produces employment and wage estimates for over 800 occupations. As part of this program, BLS develops a table of occupations that are projected to have direct employment changes due to some identified reason. According to BLS officials, the specific reason listed for each occupation is based on BLS’s judgment of the most significant factor or factors affecting the occupation (i.e., based on a qualitative assessment). We examined the reasons listed in this table and identified those related to the adoption of advanced technologies in an occupation, such as through automation, the increased use of robots or artificial intelligence, advances in machine or software technologies, or other similar changes. We then counted the number of unique occupations projected to experience declines in their shares of employment in an industry or group of industries due to one of these reasons. We also counted these occupations according to their major occupation group. BLS projected that some of these occupations would experience employment share declines in all industries and some would experience employment share declines in a single industry only. We counted unique occupations regardless of what industries or how many were noted (e.g., all industries or only one). We chose to do this to capture an inclusive list of occupations projected to be affected by advanced technologies, and because we are not using the list to quantify total projected employment changes. Of the 247 unique occupations BLS includes in its table as projected to have direct employment changes due to some identified reason, BLS projects that 163 will experience employment share declines— 100 of those occupations are projected to change broadly as a result of the adoption of advanced technologies. An employment share decline indicates that employment in an occupation will decline relative to others in a given industry or group of industries, not that the occupation will necessarily experience a decrease in employment in absolute terms. Occupational Information Network (O*NET) database: The O*NET database contains information about the skills, tasks, and tools (i.e., use of technology) associated with specific occupations. We downloaded two components of the database that (1) list the various work tasks associated with each occupation, and (2) list the various tools and technologies used by each occupation. In each database component, we searched for and identified tasks, tools, and technologies that involved robots in some way—e.g., tasks such as working with robots, robotic systems, or robotic applications, and tools such as welding robots, loading robots, or robot automation tools. We then counted the number of unique occupations that (1) had an associated work task related to robots, or (2) used a robot-related tool in the occupation. To understand firms’ adoption of advanced technologies and any resulting workforce effects, we met with officials representing 16 different firms that are using advanced technologies in their operations, as well as a systems integrator who provided detailed information about how several customer firms are using advanced technologies. Most of the meetings with firms were in-person site visits; three of the meetings with firms and the meeting with the systems integrator were by phone. Throughout this report, we use the term “firm” for simplicity, although the “firms” we met with included production plants of large manufacturers, single-location firms, public sector agencies, and other entities (see below). We also identify the manufacturing firms we visited as falling into one of three different size groups to describe their relative size differences from each other. The manufacturing firms we visited ranged from eight employees to thousands, according to firm officials. For the purposes of our study, we define small as fewer than 200 employees; medium as 200 employees to 1,000; and large as over 1,000 employees. Among the 16 firms we met with that are using advanced technologies, 10 are manufacturing firms: a small manufacturer of rubber stamps and embossing seals (also referred to as a small stamp manufacturer); two medium-sized door manufacturers; a small automotive parts manufacturer; a medium-sized automotive parts manufacturer; two large appliance manufacturers; a large automotive manufacturer; a large manufacturing corporation of household and personal care a medium-sized fruit processing plant. Six are non-manufacturing firms of various types: a construction consulting company; a federal statistical agency; a food retail corporation; a municipal township; a university-affiliated medical center; and a warehouse for a regional grocery store chain. The firms about which we received information from the systems integrator were business, administrative, and customer relations offices of various firm types. To identify firms to meet with, we consulted and sought referrals from a variety of knowledgeable sources, including academic researchers, technology developer firms, technology integrator firms, state economic development associations, and our own research. We selected firms that varied in size, industry sector, types of advanced technology used, and geography. We limited our focus to firms that had adopted advanced technologies and had experienced workforce effects. Our selection of firms is not a generalizable sample, but does provide illustrative examples of the adoption and workforce effects of advanced technologies. During our site visits at firms, we met with one or more management officials and, at times, with workers. We were also able to view the advanced technologies being used in operations. Our discussions with officials included topics such as motivations for adopting advanced technologies, the integration process, and any workforce effects that resulted from the technologies, including positions lost or gained and how workers’ tasks and skills may have changed. Our site visits and interviews with firm officials ranged from hour-long conversations to full-day visits, so some site visits yielded more detailed information than others. In addition to the firms that use advanced technologies, we interviewed seven technology developer firms and two robotics integrator firms (in addition to the systems integrator mentioned above). We met with these firms to learn more about some of the technologies being used and the adoption process, as well as about workforce effects at these firms. We identified these developer and integrator firms from various sources, including our conversations with academic researchers and our own research. We conducted additional interviews to obtain background and context for our work. We met with individuals knowledgeable about issues related to the adoption and workforce effects of advanced technologies, such as academic researchers and economists, officials from two unions representing manufacturing workers, officials at three industry-based organizations, officials from two state economic development associations, and officials at two worker training centers. For all objectives, we also reviewed relevant federal laws and regulations. The remainder of this appendix provides detailed information about the data and quantitative analysis methods we used to examine what is known about the workforce effects of automation and the adoption of advanced technologies (objective 1), as follows: Section 1: Analyses using data from the ACS Section 2: Analyses using data from the CPS’s Displaced Worker Section 3: Analyses using data from the OES survey For each of the datasets described below, we conducted a data reliability assessment of variables included in our analyses. We reviewed technical documentation and related publications and websites with information about the data. We spoke with BLS and Census officials who maintain the datasets to gain an understanding of and provide context for the various data that we analyzed, as well as to resolve any questions about the data and to identify any known limitations. We also tested the data, as applicable, to check for logical consistency, missing data, and consistency with data reported in technical documentation. We determined that the variables we used from the data we reviewed were sufficiently reliable for the purposes of this report. This section describes the quantitative analysis methods we used to examine employment trend correlations and the characteristics and earnings of workers in occupations susceptible to automation (as identified by Frey and Osborne; see above). We used ACS data for these analyses. The ACS is administered by the Census Bureau and is an ongoing national survey that uses a series of monthly samples to produce annually updated estimates for the same areas surveyed via the decennial census. The ACS collects a range of information about individuals from a large sample of households—over 2.2 million respondent households in 2016—including employment information such as occupation, industry, and earnings, and demographic information such as age, gender, race, ethnicity, and educational attainment. We limited our analysis to workers who were classified as current employees, and who had earned positive wage and salary income in the prior 12 months. In 2016, this resulted in observations representing 136 million workers, close to the number reported by BLS for that same period using a different survey. This report primarily used ACS data from 2010 through 2016—specifically, we relied on the Census Bureau’s Public Use Microdata Sample of the ACS for the single years 2010, 2011, 2012, 2013, 2014, 2015, and 2016. To test whether industries with higher concentrations of individuals in occupations susceptible to automation (as identified by Frey and Osborne) have experienced employment changes, we examined their correlation with changes in tech job concentration and changes in overall employment from 2010 through 2016. We limited the analysis to this period both because the ACS occupation codes changed in 2010 and because it allowed our results to post-date the economic recession of 2007-2009. We used industry definitions set by the ACS data, which groups some industries together—e.g., residential and nonresidential construction industries are combined in a single construction industry grouping. We defined tech jobs as those in computing, engineering, and mathematics occupations, consistent with previous GAO work on the tech field. We also examined an alternative definition of tech jobs in which we included those with “computer” in the occupation title. For both definitions, we estimated the number of tech jobs in each industry in each year, 2010-2016. We then calculated the growth rate in the number of tech jobs in each industry, and correlated that growth rate with the percentage of workers in that industry in occupations susceptible to automation (as identified by Frey and Osborne). We also estimated the number of workers overall in each industry in each year (2010-2016) and correlated the trend in total employment with the percentage of workers in that industry in occupations susceptible to automation (as identified by Frey and Osborne). We restricted our correlation analyses to those industries where the tech job growth rate or the overall employment trend was statistically significant. We performed two correlation tests. The Spearman test measures correlation between the rank of the two sets of values. The Pearson test measures correlation between the values themselves. As shown in table 2, we found a positive but weak correlation between industries with higher concentrations of jobs susceptible to automation and their concentration of tech jobs, based on both correlation tests and both definitions of tech jobs, and we found no meaningful correlation with change in overall employment in either test. To explore an example industry—the plastics product manufacturing industry—in further detail, we identified the number of jobs susceptible to automation within that industry, by occupation and groups of occupations. We also examined the growth in tech jobs within the industry, by tech occupation. We approximated each occupation’s contribution to the overall growth of tech jobs in the industry by multiplying their individual growth rates over the period 2010-2016 by their employment in 2010. The growth rates for the three engineering occupations, which when combined, account for more than half of the industry’s growth in tech jobs, were each significant at the 85 percent confidence level. To analyze the characteristics of workers in occupations susceptible to automation (as identified by Frey and Osborne), as well as the characteristics of workers with tech jobs, we used 2016 ACS data. We examined data on the workers’ gender, level of education, age, race and ethnicity, and hourly wage, and compared distributions of workers in occupations susceptible to automation and workers in all other occupations (see table 3). For race and ethnicity categories, we included only non-Hispanic members of White, Black, Asian, and Other categories, and the Hispanic category included Hispanics of all races. The “Other” category included American Indian or Alaskan Native, Native Hawaiian or Pacific Islander, two or more races, and other race. To analyze education level, we combined all attainment levels from a high school degree or less. To estimate the hourly wage of workers, we divided the wage and salary earnings of the worker by their usual hours worked and weeks worked. To test the reliability of this measure, we compared our results to average hourly wages reported by other BLS surveys; we found that the average values were sufficiently close to determine that this method was sufficiently reliable for our purposes. To investigate whether differences in hourly wage might be due to other factors, we estimated multiple regression models that enabled us to control for additional variables. Specifically, we estimated wage differences between workers in occupations susceptible to automation and workers in other occupations—i.e., whether a worker was in an occupation susceptible to automation (as identified by Frey and Osborne) was our primary independent variable (a binary, yes/no variable). Because we used the natural log of the hourly wage as the dependent variable, the standard interpretation of the regression coefficient of this variable is that it represents the average log point difference in hourly wages between occupations susceptible to automation and all other occupations. This coefficient can be made to more closely approximate a percentage difference in hourly wages or an earnings gap by taking the exponent and subtracting 1. As noted previously, we limited our analysis to workers who earned positive wage and salary income in the prior 12 months. We also removed observations with outlier values for wages (e.g., wage rates above $140 per hour); this represented about 1 percent of the sample in 2016. We ran five regression models with different sets of independent variable controls. Regression (1) estimates the earnings gap without any controls (the uncorrected earnings gap). Regression (2) estimates the earnings gap with a set of independent variables that control for characteristics of the individual; these variables included age, race and ethnicity, gender, marital status, state of residence, and education level. Regression (3) estimates the earnings gap with independent dummy variables for 2-digit industry codes added; this corrects for any differences between industries at the 2-digit level. Regression (4) estimates the earnings gap with independent dummy variables for 2-digit occupation codes added; this corrects for any differences between occupations at the 2-digit level. Regression (5) includes both 2-digit industry and 2-digit occupation code dummy variables. As table 4 shows, we found a significant difference in hourly wages between workers in occupations susceptible to automation compared to workers in other occupations, even after independent variables to control for worker characteristics, industry, and occupation codes were included. Including the additional independent variables caused the earnings gap to fall from just over -34 percent to just over -10 percent. Regression model 3, which estimated an earnings gap of about -17.2 percent, is our preferred model, as it controls for individual worker characteristics and for any differences between industries at the 2- digit level, but does not include occupation as an independent variable. Including occupation variables controls for any differences between occupations at the 2- digit level. However, because we identify workers in jobs susceptible to automation based on their occupations, these occupation control variables are likely highly predictive of Frey and Osborne’s estimated probability of automation, which is used to categorize workers in jobs susceptible to automation. We also ran these regression models for other years from 2010 to 2016 and we found substantively similar results. This section discusses the quantitative analysis methods we used to compare relative job displacement rates between workers in occupations susceptible to automation (as identified by Frey and Osborne; see above) and workers in other occupations. We used data from the CPS’s Displaced Worker Supplement for these analyses. The CPS is sponsored jointly by Census and BLS and is the source of official government statistics on employment and unemployment in the United States. The basic monthly survey is used to collect information on employment, such as employment status, occupation, and industry, as well as demographic information, among other things. The survey is based on a sample of the civilian, non-institutionalized population of the United States. Using a multistage stratified sample design, about 56,000 households are interviewed monthly based on area of residence to represent the country as a whole and individual states; the total sample also includes additional households, some of which are not interviewed in a given month for various reasons, such as not being reachable. The CPS Displaced Worker Supplement has been administered every other year since 1984, and provides supplemental data on persons age 20 years or older who lost a job involuntarily in the prior 3 years, including data on reasons for job displacement, as well as industry and occupation of the former job. This report used data from the January 2016 Displaced Worker Supplement. To analyze whether workers in occupations susceptible to automation (as identified by Frey and Osborne) experience job displacement at differing rates than workers in other occupations, we used data from the CPS’s January 2016 Displaced Worker Supplement. We identified workers who lost or left a job involuntarily during the 3 calendar years prior to the survey (i.e., January 2013 through December 2015) because their position or shift was abolished or because there was insufficient work for them to do. We focused on these reasons for displacement as those that most closely approximate how advanced technologies could replace workers at a given firm. We also limited our analysis to those workers who did not expect to be recalled to their jobs within the next 6 months. We categorized these displaced workers according to the occupations from which they were displaced (e.g., workers displaced from occupations susceptible to automation and workers displaced from all other occupations). We calculated relative job displacement rates as the number of displacements over the period 2013-2015 reported by a given population (e.g., workers in occupations susceptible to automation), over that population’s total current employment in January 2016. Although this measure does not represent the total number of jobs that existed annually that could have resulted in displacements, it allows us to control for population size and to approximate a relative displacement rate. We examined various populations, including occupations identified as susceptible to automation by Frey and Osborne, occupations BLS projects will experience declines in their share of employment due to advanced technologies (see above), and production occupations. To categorize occupations, Frey and Osborne and BLS use Standard Occupational Classification (SOC) codes, whereas the Displaced Worker Supplement uses Census occupation codes. We used a crosswalk provided by Census to match these occupation classifications. SOC codes have a hierarchical structure—e.g., a “broad” occupation group contains a subset of “detailed” occupations. For example, SOC code 13- 1031 is the detailed occupation “claims adjusters, examiners, and investigators” within the broad group SOC 13-1030 (“claims adjusters, appraisers, examiners, and investigators”). When a direct crosswalk between SOC and Census occupation codes was not available at the detailed level, we used the associated broad SOC group to identify a Census occupation code. There were some respondents in the Displaced Worker Supplement who did not report the occupation from which they were displaced, and these were dropped from our analysis. To estimate the sampling errors for each estimate, we used strata defined by state because the Displaced Worker Supplement data did not provide replicate weights or the sampling strata necessary to obtain standard errors. When estimating the number of job displacements over the period 2013-2015 reported by a given population (e.g., workers in occupations susceptible to automation), we used the supplement weight for respondents. When estimating the population’s total current employment in January 2016, we used the CPS 2016 weight for respondents. We used a Taylor series linearization to estimate the sampling error of the ratio of estimated number of job displacements over the period 2013- 2015 to the estimated number of current employment in 2016. While our primary analysis examined relative displacement rates for workers in occupations susceptible to automation, we also conducted sensitivity analyses by considering other groups of occupations. Specifically, we examined the relative displacement rates of the following groups: Jobs susceptible to automation had a relative displacement rate of 3.4 percent +/- 0.3, and all other jobs combined had a relative displacement rate of 2.9 percent, +/- 0.2. Jobs in occupations BLS projects will experience relative declines in employment due to advanced technologies (see above) had a relative displacement rate of 3.7 percent, +/- 0.5, and all other jobs combined had a relative displacement rate of 3.6 percent, +/- 0.2. Jobs in production occupations had a relative displacement rate of 3.7 percent +/- 0.8, and all other jobs combined had a relative displacement rate of 3.1 percent, +/- 0.2. This section discusses the quantitative analysis methods we used to analyze geographic reliance on occupations susceptible to automation (as identified by Frey and Osborne; see above). We used OES data for these analyses. The OES survey is a federal-state cooperative effort between BLS and state workforce agencies, which collects information on occupational employment and wage rates for wage and salary workers in nonfarm establishments. The survey is based on a sample drawn from about 7.6 million in-scope nonfarm establishments in the United States that file unemployment insurance reports to the state workforce agencies. Using a stratified sample design, about 200,000 establishments are surveyed semiannually and employment estimates are based on six panels of data collected over a 3-year cycle. The final in-scope sample size when six panels are combined is approximately 1.2 million establishments. The OES survey includes all full- and part-time wage and salary workers in nonfarm industries, but excludes self-employed workers, owners and partners in unincorporated firms, household workers, and unpaid family workers. OES data provide occupational employment estimates by industry for the country as a whole, for individual states, and for more local geographic areas (e.g., metropolitan and nonmetropolitan areas). This report used data from the May 2017 Occupational Employment Statistics. To analyze what U.S. geographic areas rely more heavily on employment in occupations susceptible to automation, we used data from the May 2017 OES. For each local geographic area, we estimated how many jobs were in occupations identified as susceptible to automation by Frey and Osborne (see above) and how many jobs were in all other occupations. We also estimated how many jobs were in each group of occupations nationwide (using national-level data). We then calculated a location quotient for each local geographic area, which measures the proportion of each area’s jobs that were in occupations susceptible to automation compared to the national proportion of employment in these occupations. This measure depicts the extent to which a local geographic area relies on certain jobs for the employment of its population, relative to other areas. Based on their location quotients, we categorized and mapped 589 local geographic areas in the following three groups: Relatively High Concentration: Areas where the proportion of jobs susceptible to automation is at least 5 percentage points greater than the national average, and the difference is statistically significant at the 95 percent confidence level. This translates to an estimated location quotient of at least 1.1. Average or Relatively Low Concentration: Areas where the proportion of jobs susceptible to automation is within 5 percentage points above the national average or lower. Undetermined Reliance: Areas where the proportion of jobs susceptible to automation is undetermined. We classify an area’s proportion as “undetermined” if the estimated margin of error at the 95 percent confidence level is larger than 5 percentage points. We conducted one sided z-tests at the 95 percent confidence level to analyze each area’s estimated location quotient. The null hypothesis is that the area location quotient is less than or equal to 1.1 (i.e., the proportion of employment in the group of occupations in an area is 1.1 times the national proportion). The alternative hypothesis is that the area location quotient is greater than 1.1. Because estimated area employment proportions are based on a sample, we also restricted our tests to those areas that were reliable for our purposes by requiring that areas had sampling errors of no greater than 5 percentage points for a 95 percent confidence interval. According to BLS, employment estimates for individual occupations in individual local geographic areas may not be available in the public data for a variety of reasons, including for example, failure to meet BLS quality standards or to ensure the confidentiality of survey respondents. Because we aggregate data across multiple occupations, our methodology treats these cases as if employment in the given occupation in the given area was zero, which is not the case and which introduces imprecision into our analysis and the resulting location quotients. However, because ensuring confidentiality is a primary concern, we assume that most of these cases where data are suppressed would have relatively small numbers of jobs, and thus have minimal effects on our results. To test this assumption and to ensure the appropriateness of our methods, we compared the total number of jobs we analyzed across all local geographic areas to the total number of jobs reported at the national level (which do not have data suppressed). The total number of jobs analyzed across our local geographic areas was 5.5 percent lower than the total number of jobs reported at the national level, which we concluded was within an acceptable threshold to determine that the data were sufficiently reliable for our purposes and our analysis. In addition, according to BLS, because occupational employment estimates are rounded to the nearest 10 before publication, estimates of location quotients calculated from the public data will be subject to some rounding error, compared with location quotients calculated from the unrounded pre-publication data. In addition to the contact named above, Blake Ainsworth (Assistant Director), Michael Kniss (Analyst-in-Charge), Shilpa Grover, and John Lack made key contributions to this report. Also contributing to this report were James Bennett, Benjamin Bolitzer, Melinda Cordero, Holly Dye, Jonathan Felbinger, Sheila R. McCoy, Jean McSween, James Rebbe, Krishana Routt-Jackson, Benjamin Sinoff, Almeta Spencer, and Sonya Vartivarian.", "summary": "Advanced technologies—including artificial intelligence and robotics—are continually changing and emerging. While robots have existed for decades, modern robots may be equipped with learning capabilities that enable them to perform an expansive array of tasks. Advanced technologies are likely to affect the U.S. workforce by enabling firms to automate certain work tasks. Questions exist about how prepared federal agencies are to monitor workforce changes, promote economic growth, and support workers who may be negatively affected by automation. GAO was asked to examine workforce issues related to the adoption of advanced technologies. This report examines (1) what is known about how the adoption of advanced technologies affects the U.S. workforce ; (2) federal efforts to track these effects; (3) considerations that led selected firms to adopt advanced technologies and the risks they faced; and (4) ways technology adoption has affected the workforce at selected firms. GAO identified available federal workforce data, analyzed the extent to which those data could identify and measure workforce effects due to advanced technologies, reviewed selected research, and analyzed federal data on occupations susceptible to automation. GAO used data from the American Community Survey (2010-2016), the Current Population Survey's Displaced Worker Supplement (2016), and the Occupational Employment Statistics (2017). GAO met with 16 firms that are using advanced technologies in their operations and seven firms that develop advanced technologies, and interviewed managers and workers, and observed firms' use of technologies. The selected firms varied in size, industry sector, types of technologies used, and geographic location. Findings from discussions with the fims are not generalizable, but provide illustrative examples about the adoption of advanced technologies. GAO interviewed officials from federal agencies, including Commerce and DOL, academic researchers, economists, labor union officials, industry association officials, officials from state economic development associations, and other knowledgeable individuals. GAO also reviewed relevant academic work. Although existing federal data provide useful information on the U.S. workforce, they do not identify the causes of shifts in employment. As a result, it is difficult to determine whether changes are due to firms adopting advanced technologies, such as artificial intelligence and robots (see photo), or other unrelated factors. In lieu of such data, GAO analyzed employment trends and characteristics of jobs that selected researchers identified as susceptible to automation, and found that: industries with a greater proportion of jobs susceptible to automation were more likely to have experienced growth in tech jobs (i.e., computing, engineering, and mathematics) from 2010 to 2016—possibly an indicator of industries preparing to adopt advanced technologies; occupations susceptible to automation and industries with a greater share of these jobs did not experience meaningfully higher job loss rates in this period, though it could be too soon to observe these effects; and certain groups, such as workers with no college education and Hispanic workers, tended to hold jobs susceptible to automation in 2016, and thus could be disproportionately affected by changes if they occur. The Department of Labor (DOL) has a role in tracking changes in the U.S. workforce, but the data it collects related to the workforce effects of advanced technologies are limited. DOL's Bureau of Labor Statistics (BLS) identifies occupations projected to experience staffing pattern changes and the most significant causes, such as use of robotics, but its efforts are not designed to capture all instances of changes due to advanced technologies. DOL's Occupational Information Network program also collects data on tasks and technologies in occupations, such as robotics, but it was not designed to track changes over time. According to BLS, these efforts and other data they collect provide some, but not all, of the information required to identify and systematically track the impact of automation on the workforce. Without comprehensive data that link technological changes to shifts in the workforce, DOL lacks a valuable tool for ensuring that programs it funds to support workers are aligned with local labor market realities, and employers and job seekers need to rely on other sources of information to decide what training to offer or seek. The Department of Commerce's Census Bureau (Census) has started tracking technology adoption and resulting workforce effects in the new Annual Business Survey, which was administered for the first time in June 2018 with significant support from the National Science Foundation. This first survey asked firms about their use of advanced technologies and initial results will be available in late 2019. When the survey is next administered in summer 2019, Census plans to ask additional questions about firms' motivations for adopting technologies and effects the technologies might have on workers. This survey could provide information about the prevalence of technology adoption and workforce changes (e.g., declines in production workers or increases in supervisory workers), but it is not intended to provide information on the magnitude of workforce changes. Also, it remains unclear what limitations, if any, the survey data may have. According to officials from the 16 firms GAO interviewed, cost savings and other considerations led them to adopt advanced technologies, despite facing certain risks with the new technologies. Officials from these firms typically identified cost savings and improving job or product quality as primary motivations for adopting advanced technologies. For example, an automotive parts manufacturer said the firm adopted robots to reduce costs by using fewer workers. A door manufacturer said the firm installed two robots to lift heavy doors onto a paint line to reduce the number of worker injuries. A rubber stamp manufacturer said acquiring a robot (pictured above) allowed it to purchase and process raw materials instead of buying precut materials. Firm officials also identified risks related to adopting advanced technologies that could affect their return on investment, such as risks related to the reliability of technology and working with new tech developers. Among the firms GAO met with, officials described various ways technology adoption has affected their workforces. On one hand, officials at many firms said they needed fewer workers in certain positions after adopting technologies. The firms generally redeployed workers to other tasks, and in some cases, reduced the size of their workforces, typically through attrition. For example, a medical center GAO visited adopted autonomous mobile robots to transport linens and waste, among other things, which officials said eliminated 17 positions and shifted workers to other positions. On the other hand, officials at some firms said advanced technologies helped them increase competitiveness and add positions. An appliance manufacturer used advanced technologies to produce more of its own parts instead of relying on suppliers and, as a result, increased the number of production jobs, according to officials. Firm officials also noted that workers' tasks and skills have been changing due to advanced technologies (see figure). Workers who can adapt to new roles may experience positive effects, such as work that is safer, while those who cannot adapt may be negatively affected. GAO recommends that DOL develop ways to use existing or new data collection efforts to identify and systematically track the workforce effects of advanced technologies. DOL agreed with GAO's recommendation, and plans to identify and recommend data collection options to fill gaps in existing information about how the workplace is affected by new technologies, automation, and artificial intelligence. DOL also stated that it will continue coordinating with the Census Bureau on research activities in this area.", "document_type": "gao"}
{"report": "While female participation in the military dates back to the American Revolution, women have formally served in United States military units since 1901 with the establishment of the Army Nurse Corps. The Act of May 14, 1942 authorized the president to establish and organize a Women’s Army Auxiliary Corps for the purpose of “making available to the national defense when needed the knowledge, skill, and special training of the women of this Nation.” In 1948, the Women’s Armed Services Integration Act of 1948 authorized the military services to, subject to the provisions of the act, enlist and appoint women to their active and reserve components. Certain provisions of the Women’s Armed Services Integration Act of 1948, including limits on the number of women in the Navy and Marine Corps, were repealed in 1967, and additional changes to DOD policies have been made since then. For example, the Department of Defense Appropriation Authorization Act, 1976 directed the secretaries of the military departments to, among other things, take such action as may be necessary and appropriate to insure that women were eligible for appointment and admission to the military service academies. Almost two decades later, the National Defense Authorization Act for Fiscal Year 1994, among other things, required the Secretary of Defense to ensure that qualification of members of the armed forces for military occupational career fields open to both male and female members is evaluated on the basis of common, relevant performance standards without differential standards or evaluation on the basis of gender. It also repealed the remaining statutory prohibitions on the Secretary of the Navy assigning female servicemembers to duty on vessels and aircraft engaged in combat missions or expected to be assigned combat missions. In January 1994, the Secretary of Defense issued a memorandum creating the Direct Ground Combat Definition and Assignment Rule, which made servicemembers eligible for assignment to all positions for which they were qualified, but it excluded female servicemembers from assignment to units below the brigade level whose primary mission was to engage in direct combat on the ground. The memorandum required the services to coordinate approved implementing policies and regulations—including certain service restrictions on the assignment of women—with the Assistant Secretary of Defense for Personnel and Readiness prior to their issuance. The memorandum also permitted the services to propose additional exceptions. In its 2011 final report, the Military Leadership Diversity Commission stated that the services’ have been leaders in providing opportunities for all servicemembers, regardless of racial/ethnic background, or gender, and stated that the DOD’s mission-effective force is a living testament to progress in the areas of military equal opportunity policies and related recruiting and management tactics. The report also stated that more needs to be done to address 21st century challenges and that the Armed Forces have not yet succeeded in developing a continuing stream of leaders who are as demographically diverse as the nation they serve. A 2013 Secretary of Defense and Chairman of the Joint Chiefs of Staff memorandum rescinded the 1994 Direct Ground Combat Definition and Assignment Rule. That memorandum also directed the military services to open currently closed units and positions to female servicemembers, consistent with certain principles and with the implementation of certain standards. The memorandum also directed that the integration of female servicemembers into these newly opened positions and units occur as expeditiously as possible, considering good order and judicious use of fiscal resources, and no later than January 1, 2016. The military services also took action through issuing guidance. For example, in 2013, the Commandant of the Marine Corps issued a letter to Marine Corps leadership stating that it is imperative for the Marine Corps to take a fresh approach to diversity and establishing four task force groups, including one titled “Women in the Corps: Attract, Develop, and Retain Women Officers.” Subsequently in June 2014, the Secretary of the Air Force and Air Force Chief of Staff released a memorandum establishing active-duty officer applicant pool goals, which are intended to reflect the nation’s highly talented, diverse, and eligible population. More recently, in 2015, the Secretary of Defense determined that no exceptions were warranted to the full implementation of the rescission of the Direct Ground Combat Definition and Assignment Rule and directed the secretaries of the military departments and chiefs of the military services to begin to execute the implementation of their approved plans to open all military occupational specialties, career fields, and branches for accession by female servicemembers as soon as practicable and not later than April 1, 2016. Figure 1 presents a timeline of selected events in female participation in the military, including changes to laws and policies. Overall, the percentage of female active-duty servicemembers slightly increased from fiscal year 2004 through 2018. However, our analyses also determined that for fiscal years 2004 through 2018, female enlisted servicemembers and commissioned officers had higher attrition rates than their male counterparts, and the percentage of female active-duty servicemembers began to decrease at the 10–to-less-than-20-years of service career point, meaning a smaller pool of female servicemembers being available for leadership opportunities. We also found that female servicemembers are generally more likely to separate from the military, and that the reasons active-duty servicemembers separate from the military vary by gender, pay grade category, and length of service. In addition, other factors—such as access to quality childcare or family planning—have been found to influence female active-duty servicemembers’ separation decisions based on our review of existing literature. The services have experienced slight increases in their populations of female active-duty servicemembers from fiscal year 2004 through 2018. More specifically, the overall percentage of female active-duty servicemembers increased slightly department-wide within that 15 year period, from 15.1 percent in fiscal year 2004 to 16.5 percent in fiscal year 2018, with slight decreases identified in some years—for example, fiscal years 2005 through 2009. Comparatively, the percentage of males serving on active duty decreased from 84.9 percent in 2004 to 83.5 percent in 2018. In fiscal year 2018, the Air Force had the highest percentage of female active-duty servicemembers (20.2 percent), followed by the Navy (19.6 percent), the Army (15.1 percent), and the Marine Corps (8.6 percent). The Air Force also had the highest percentages of female enlisted and officers in fiscal year 2018 (20.0 percent and 21.3 percent, respectively). The Marine Corps (8.7 percent female enlisted and 7.9 percent female officer), had the lowest percentages in fiscal year 2018. Figure 2 shows the representation of active-duty servicemembers, by gender, organization, and pay grade for fiscal year 2018. The Air Force and the Army had higher percentages of female servicemembers than the Navy and Marine Corps in fiscal year 2004— the first year of the data we analyzed–-and those percentages remained relatively stable over the full 15 fiscal years of data we analyzed. Additionally, the percentage of female servicemembers in the Air Force remained higher in each year than in the three other services over that 15 year period. The Navy and the Marine Corps experienced larger increases in their overall percentages of female active-duty servicemembers from fiscal year 2004 through fiscal year 2018. For example, the overall percentage of female active-duty servicemembers in the Navy increased by 4.9 percentage points, from 14.7 percent in fiscal year 2004 to 19.6 percent in fiscal year 2018. The Marine Corps experienced an increase of 2.5 percentage points in that same time period, from 6.1 percent in fiscal year 2004 to 8.6 percent in fiscal year 2018. Figure 3 shows the percentage of female active-duty servicemembers across all services in select years from fiscal years 2004 through 2018, by their organization. We also found that although the percentage of female active-duty servicemembers generally increased across the department from fiscal year 2004 through 2018, the percentage of female active-duty servicemembers was higher for those with fewer years of service and generally decreased as years of service increased. Specifically, as figure 4 shows, the percentages of female enlisted and commissioned officers in all four services with either 10 to 20 years of service or 20 or more years of service were generally lower than those with less than 10 years of service. We also found that the percentages of women with more years of service were higher in more recent years, specifically in fiscal years 2014 through 2018 as compared to fiscal years 2004 through 2009. For example, in fiscal years 2014 through 2018, the percentage of female enlisted with 20 or more years of service (12 percent) was 2.2 percent higher than the percentage of female enlisted in fiscal years 2004 through 2009 (9.8 percent). Similarly, the percentage of female commissioned officers with 20 or more years of service in fiscal years 2014 through 2018 (12.1 percent) was 1.4 percent higher than female commissioned officers with the same length of service in fiscal years 2004 through 2009 (10.7 percent). In addition, the percentage of female warrant officers with 20 or more years of service in fiscal years 2014 through 2018 (8.3 percent) was 2.3 percent higher than female commissioned officers with the same length of service in fiscal years 2004 through 2009 (6 percent). From fiscal year 2004 through 2018, female active-duty enlisted servicemembers and commissioned officers had higher annual attrition rates than corresponding males during that same time period. However, the gaps between male and female attrition rates for enlisted and commissioned officers have narrowed in more recent years. Specifically, for fiscal years 2004 and 2018, enlisted female active-duty servicemembers’ annual attrition rates were 33.1 and 8.6 percent, respectively. In fiscal years 2004 and 2018, enlisted male active-duty servicemembers’ annual attrition rates were 22.7 and 6.1 percent, respectively. For fiscal years 2004 and 2018, female commissioned officer annual attrition rates were 10 and 0.7 percent respectively, while male commissioned officer annual attrition rates were 6 and 0.4 percent in those same years, respectively. In fiscal years 2004 and 2018, female warrant officer annual attrition rates were 12.5 and 0 percent, and male warrant officer annual attrition rates were 3.2 and 0 percent in fiscal years 2004 and 2018, respectively. Figure 5 shows active-duty servicemember annual attrition rates over time from 2004 through 2018, by gender and pay grade. Additionally, we developed a set of statistical models—all discrete time duration analysis—using data from fiscal years 2004 through 2018 which accounted for active-duty servicemembers’ time in service (i.e., the period of time from when they joined the military until their separation). The models estimated the association of gender with separation. We accounted for specific servicemember characteristics, such as gender, branch of military service, pay grade, race or ethnicity, marital status, and the existence of dependents to estimate the associations that these characteristics have with active-duty servicemembers separating from the service. The results of our statistical models show that female active-duty servicemembers are more likely to separate from the military than males at any given period of time in service. The average estimated likelihood of female active-duty servicemembers’ separation for each quarter year of time in service is 2.3 percent, while the average estimate for male active- duty servicemembers is 1.8 percent. In relative terms, the likelihood of separation for female active-duty servicemembers is 28 percent higher than the likelihood of separation for male active-duty servicemembers. When controlling for various individual and occupational characteristics— including pay grade categories, marital status, race or ethnicity, education level, occupation, and whether the servicemember has dependents— among others—female active-duty servicemembers’ average estimated likelihood of separating from the military per quarter year of time in service ranges from 1.8 percent to 3.1 percent, depending on their branch of service, while that for their male counterparts ranges from 1.4 percent to 2.3 percent, if other personal characteristics remain the same. In relative terms, the likelihood of separation for female active-duty servicemembers is estimated to be 13 to 46 percent higher than that of their male counterparts. Based on our statistical models, we also found the following by particular characteristics: Married versus unmarried without dependents: In all of the services, both female and male married active-duty servicemembers without dependents are more likely to separate from the military than unmarried male and female active-duty servicemembers without dependents. For example, the likelihood of separation for both female and male married active-duty servicemembers without dependents in the Air Force and the Navy are twice as high as male and female unmarried active-duty servicemembers without dependents in the same services. Married with dependents versus unmarried without dependents: Married male active- duty servicemembers with dependents in all of the services except the Air Force are less likely to separate from the military than unmarried males without dependents. However, married female active-duty servicemembers who have dependents and are serving in the Army, the Navy, and the Air Force are more likely to separate compared to unmarried female active-duty servicemembers without dependents. For example, in the Navy, the likelihood of separation for married female active-duty servicemembers who have dependents is 17 percent higher relative to that for unmarried female active-duty servicemembers without dependents. Comparatively, we estimate that the likelihood of separation for married male active-duty servicemembers in the Navy who have dependents is 28 percent lower than the likelihood of separation for unmarried male active-duty servicemembers in the Navy who do not have dependents. Unmarried with dependents versus unmarried without dependents: In all four services, unmarried female active-duty servicemembers who have dependents are more likely to separate from the military than their unmarried counterparts who do not have dependents. Our analysis produced similar results for unmarried male active-duty servicemembers with dependents, except for those serving in the Navy, who we found are less likely to separate than unmarried male active-duty servicemembers without dependents. More specifically, we estimate that the likelihood of separation for unmarried male and female active-duty servicemembers who have dependents and serve in the Army, Marine Corps, or Air Force, is from 9 percent to 32 percent higher than that for their unmarried male and female counterparts who do not have dependents. Further, we estimate that the likelihood of separation for unmarried female active- duty servicemembers who are serving in the Navy and who have dependents is 35 percent higher relative to the likelihood of separation for those female servicemembers who serve in the Navy and are unmarried and do not have dependents. Pay grade categories: Our analysis found that enlisted male and female active-duty servicemembers in all of the services are more likely to separate from the military than male and female active-duty officers and warrant officers within the same service. For example, we estimate that the likelihood of separation for male and female officers serving in the Navy is 62 and 63 percent lower, respectively, relative to the likelihood of separation for enlisted male and female active-duty servicemembers serving in the Navy. Race or ethnicity minority groups versus whites: In all of the services, black and Hispanic female active-duty servicemembers are less likely to separate from the military than white female active-duty servicemembers. All other racial or ethnic minority female active-duty servicemembers are also less likely to separate from the military than white female active-duty servicemembers except in the Army. More specifically, we estimate that black, Hispanic, and all other racial or ethnic minority female active-duty servicemembers in all of the services (except in the Army) are at least 13 percent less likely to separate from the military relative to white female active-duty servicemembers. All other racial or ethnic minority female active-duty servicemembers (except black and Hispanic) serving in the Army are estimated to be 26 percent more likely to separate from the military relative to white female active-duty servicemembers. In 2011, the Military Leadership Diversity Commission’s final report discussed explanations for discrepancies in representation among senior military leaders, including lower retention of mid-level female enlisted and officer servicemembers. Additionally, OSD officials stated that, in 2017, ODEI conducted an assessment of diversity and inclusion among officers that analyzed fiscal year 2012 through 2016 data to determine whether there was a difference between male and female retention within each of the services. According to DOD, ODEI found various increases and decreases in female retention; however, the officials stated that the assessment did not include an analysis to identify the reasons for the differences in retention among female servicemembers within the services. In its 2017 and 2018 reports, DACOWITS identified dual-military couples as facing retention challenges and the 2017 report stated that, proportionally, more female servicemembers are married to a military spouse than are male servicemembers. Additionally, in the 2017 report, DACOWITS stated that servicemembers who are separated from the military because of issues related to parenthood, including family care plans, are disproportionately female. The DACOWITS report further stated that, according to data provided to DACOWITS by the services, between fiscal year 2007 and 2016, female servicemembers represented between 65 and 83 percent of parenthood-related discharges. We also analyzed 15 years of separation code data (fiscal years 2004 to 2018) to identify the documented reasons why active-duty servicemembers separated from the military during that time. Our analysis of these data found that the reasons active-duty servicemembers separate from the military vary slightly based on gender, pay grade category, and length of service, as well as by time period. For example, misconduct was a top reason for separation from 2004 through 2013 for enlisted male servicemembers with 5 or fewer years of service, whereas pregnancy was one of the top three reasons for separation for female enlisted with 5 or fewer years of service, during that same period. However, neither misconduct for male servicemembers nor pregnancy for female servicemembers were found to be in the top three reasons for separation in fiscal years 2014-2018. The results of this analysis are shown below in figures 6, 7, and 8. To better understand other factors that may underlie a servicemember’s decision to separate, we reviewed a variety of studies on female active- duty servicemember retention in the military. Through our review, we identified six factors that were reported to influence female active-duty servicemembers’ separation from the military: work schedules, deployments, organizational culture, family planning, sexual assault, and dependent care. Work schedules. Specifically, four of the six studies in our literature review cited work schedule as a reason for or factor influencing separation by female active-duty servicemembers. For example, in several studies female active-duty servicemembers cited the demands and uncertainty of their work schedules. In one study, which asked senior female enlisted Army personnel about the primary factors responsible for their decision to leave the military, a review of the participants’ responses indicated that the primary factor responsible for female servicemembers exiting the service sooner than their male counterparts was that the female members believed they constantly had to sacrifice family time for their careers. In another study, former female active-duty naval surface warfare officers cited the uncertainty of their work schedules as having a strong influence on their decision to separate from the military. Deployments. The occurrence of deployments and their effects on family life were also highlighted in four of the six studies as factors influencing female servicemembers’ decisions to separate from the military. For example, one study of female Air Force pilots identified deployments as a factor that caused them to consider leaving active duty. In another study, which included 295 active-duty and reserve female officers in grades O-1 to O-5, participants in 94 percent of the 54 focus group mentioned deployments as an important negative influence on retention, given their effect on spouses and children. Organizational culture. Organizational culture also had an effect on female servicemembers’ decisions to separate from the military in four of the six studies we reviewed. In one study, female active-duty, reserve, and Air National Guard officers in the Air Force mentioned the lack of female mentors and role models in leadership positions, and the experience of sexism as factors influencing the decision to separate. Female servicemembers also discussed how having leaders who are not supportive or understanding of family needs can contribute to a negative or toxic work environment. Study participants also noted that they often faced sexism and the existence of an “old boy’s network,” especially in career fields dominated by males. As such, these female servicemembers felt they had to work harder to prove themselves and also felt they were sometimes not treated equally because they were female. Family planning. Three of the six studies in our literature review cited family planning as being another characteristic that influences separation for female active-duty servicemembers. In one study, female officers in a majority of focus groups (85 percent of 54 focus groups) mentioned issues related to pregnancy that could affect their decisions to stay in or leave the Air Force. More specifically, Air Force female officers (active duty, reserve, and Air National Guard) cited the difficulty of timing pregnancies to fit within rigid career timelines. These female servicemembers stated that they felt they needed to ensure that pregnancy occurred at certain times in their careers to minimize negative career effects. Even with that effort, the female servicemembers stated that negative effects still persisted due to missed opportunities while pregnant, such as in-residence professional military education, or career- field specific problems, such as loss of flying time for pilots. Sexual assault. Two of the six studies in our literature review cited sexual assault as a reason for separation by female active-duty servicemembers. In one study, female military veterans mentioned both the occurrence of a sexual assault and how it was handled by the military as contributing to their separation. For example, two females stated that the perpetrator was not punished, and another woman cited the lack of support from other servicemembers as contributing to their decisions to separate from the military. In another study examining female officer retention (active duty, reserve, and Air National Guard) in the Air Force, a few participants cited cases in which either they or individuals they knew had decided to leave specifically because of a sexual assault. Participants commented that female officers often do not want to report the incident, deciding instead to separate. Dependent care. Two of the six studies in our literature review also mentioned challenges with dependent care as influencing female servicemembers’ decisions to separate from the military. For example, in one study, female military veterans cited difficulties being separated from their children for long time periods as a reason for ending military service. These difficulties were both emotional and practical, including limited stable and safe placement options for children while mothers were deployed. In another study, female Air Force officers in 59 percent of 54 focus groups stated that difficulties with childcare development centers on military bases—including service hours that were incompatible with their work schedules, inconsistent quality of care, and long waitlists—could influence their separation decisions. Participants in that study’s focus groups stated that childcare development centers often have limited hours that make it difficult to coordinate childcare with long work hours or shift work. For example, according to the study’s focus group participants, pilots are sometimes required to fly at night and regularly need overnight child care, outside of typical childcare development center hours. Further, participants stated that some female servicemembers also raised concerns about the quality of care at childcare development centers, noting that the quality of employees is not consistent across locations and that the childcare development centers generally do not provide day-care services that include educational activities to enhance children’s learning, unlike some off-base options. In addition, some female servicemembers in that same study’s focus groups cited problems setting up childcare with childcare development centers before the end of their maternity leave due to lengthy wait lists. Our analyses determined that for fiscal years 2004 through 2018, female active-duty servicemember promotion rates were slightly lower for enlisted in most years, but higher for officers as compared to their male counterparts. We also found that the percentage of promotions for eligible female and male active-duty servicemembers decreases at certain grade levels, and the likelihood of promotion varies across certain characteristics, including gender and pay grade. Overall, we estimated that in most years from fiscal years 2004 through 2018, promotion rates for female enlisted active-duty servicemembers were slightly lower than those for male enlisted active-duty servicemembers. Specifically, female enlisted promotion rates were lower than male enlisted promotion rates by a range of 0.1 percentage points to 2.5 percentage points during much of that time period. However, in fiscal years 2015 and 2018, female enlisted promotion rates were higher than their male counterparts by 0.1 percentage points and 0.4 percentage points, respectively. In contrast, female commissioned officers had higher promotion rates than male commissioned officers each year during that same period. Specifically, from fiscal years 2004 through 2018, female commissioned officer promotion rates ranged from 3.3 to 5.3 percentage points higher than male commissioned officer promotion rates. Similarly, from fiscal year 2004 through 2018, female warrant officer promotion rates were higher—a range of 1.5 to 19.3 percentage points—than male warrant officer promotion rates in most years. However, in fiscal years 2015 and 2016, promotion rates for male warrant officers were higher by 1.4 percentage points and 1.9 percentage points, respectively. Figure 9 shows active-duty servicemember annual promotion rates over time, by gender and pay grade category, for fiscal years 2004 through 2018. We also present additional data in appendix III on servicemember promotion rates in fiscal years 2004 through 2018. The 2017 DACOWITS report stated that female servicemembers are particularly underrepresented in military leadership and, as of July 2017, the percentages of female servicemembers in the highest ranks were much lower than in the lowest ranks, particularly among officers. Further, according to DACOWITS, the percentage of female servicemembers declined by nearly two-thirds from the lowest to highest- ranking commissioned officer position, and by nearly half from the lowest to highest-ranking enlisted position. Through our analysis of DMDC data, we found a similar trend in 2018 with the percentage of female servicemembers declining by nearly three quarters from the lowest to highest-ranking commissioned officer positions (21 percent to 5.4 percent). Additionally, the trend was also similar for enlisted personnel for which the percentage of female enlisted declined by nearly half from the lowest to highest-ranking positions (16.6 percent to 9.1 percent). Based on our discrete time duration analysis, we estimated that promotion rates may vary for female active-duty servicemembers relative to their male counterparts across the services, after adjusting for certain demographic and occupation-specific factors, including gender, time in service, branch of service, pay grade, marital status, and whether the active-duty servicemember has dependents. We estimated that in the Navy, enlisted female active-duty servicemembers may have a lower likelihood of promotion than their male counterparts, whereas the evidence is mixed for the Army, the Marine Corps, and the Air Force after controlling for certain individual- and occupation-level characteristics. Figure 10 presents the likelihood of female promotion as compared to males when controlling for time in service, while figure 11 presents the difference in likelihood of promotion when controlling for various demographic factors. Officials from the Service Women’s Action Network told us that, with regard to career progression, the rigidity and timing of some job requirements for certain military occupational specialties are not conducive to becoming pregnant or raising a young family. Specifically, these officials stated that such requirements—for example, Naval surface warfare tours—often occur at the time in a female active-duty servicemember’s life when she may try to become pregnant or have young children. However, according to these officials, such tours must occur at these specific points in one’s career in order to get promoted. Similarly, the Military Leadership Diversity Commission reported in its 2011 final report that, although the services do not have a checklist of assignments required for promotion, each service, community, and career field has a notional career path comprising key work and educational assignments, including leadership and staff assignments early on in one’s career, holding command assignments, meeting certain educational milestones, and holding executive officer or assistant positions to current flag or general officers. Further, the report stated that women and minorities face barriers to serving in such key assignments which can affect their ability to reach senior leadership ranks. The Military Leadership Diversity Commission also reported that one barrier may include lack of sufficient knowledge about these key assignment opportunities, perhaps because women and minorities may not receive the same career counselling or mentoring about key assignments as their white male counterparts. DOD officials stated that as part of the 2017 ODEI assessment, female promotion rates were also analyzed across the services. According to those officials, the assessment found variations in promotion rates from fiscal year 2012 through fiscal year 2016 among female servicemembers; however officials also stated that the assessment did not include an analysis to identify the reasons for the differences in promotion rates among female and male servicemembers. DOD has identified that female recruitment and retention is important to diversity in the military, but the services do not have plans that include goals, performance measures, or timeframes to guide and monitor current or future efforts to recruit and retain female active-duty servicemembers. While recruiting is an important first step in building a diverse force and increasing the representation of female servicemembers, retention plays a similarly important role in maintaining that diversity once it is achieved. DOD’s 2012-2017 Diversity and Inclusion Strategic Plan, quoting the 2011 National Military Strategy, stated that the all-volunteer force must represent the country it defends and benefits immensely from the different perspectives and linguistic and cultural skills of all Americans. According to ODEI officials, the department is currently updating its diversity and inclusion strategic plan to guide efforts through 2024. However, neither the 2012-2017 plan nor the draft updated plan, according to officials, has a focus on goals, such as recruitment or retention goals, for any one particular demographic group. Officials we interviewed stated that there is a general goal to recruit a force that reflects the makeup of the country it represents as a method for encouraging trust in the military among the population at large. However, according to OSD and service officials, the department emphasizes gender-neutral occupational standards and policies, with its focus on recruiting and retaining the best and brightest service members. Specifically, OSD officials stated that the department’s priorities and goals are aimed at improving the retention and promotion rates of all active-duty servicemembers, while ongoing OSD efforts to evaluate diversity within the department focus more broadly on the overall state of diversity of both the military and civilian workforces. OSD officials further stated that retention goals have, in the past, been misconstrued as quotas based on gender and, as such, the department does not set goals or targets for gender. While we recognize the department’s concern about goals being misconstrued as quotas, goals are not quotas and we have previously reported that quantitative and qualitative performance measures “help organizations translate their diversity aspirations into tangible practice.” For example, an organization can track data on its workforce to evaluate the effectiveness of the organization’s diversity management efforts and the progress it is making in those efforts. In addition to analyzing quantitative workforce data, we further reported that organizations can use qualitative data derived from interviews, focus groups, and surveys to identify employee perceptions—including available opportunities and work environment or culture—among various segments of their workforces. In its 2017 report, DACOWITS stated that each of the military services experiences challenges retaining women to a varying degree, with a particularly wide gender gap in operational specialties. DACOWITS’ report further stated that concerns persist that this attrition will result in a disproportionate impact to mission readiness if left unresolved. DACOWITS has also made a number of recommendations specific to the services’ efforts to address and increase female representation in the military through the use of goals and targets. For example, in 2014, DACOWITS recommended that the services should have targets to increase the representation of enlisted female servicemembers and that these targets should be benchmarked against the pool of eligible recruits. Subsequently in 2015, DACOWITS recommended, among other things, that the services should set goals to systematically increase the representation of women in the officer and enlisted ranks. However, according to officials from the four services, the services currently do not have plans that include goals, performance measures, and timeframes to guide and monitor efforts to recruit and retain female servicemembers. For example, Marine Corps officials stated that DOD has not tasked the Marine Corps to prioritize gender with regard to retention or promotion. Marine Corps officials also stated that the Marine Corps does not have any programs or initiatives that focus specifically on reducing attrition and increasing retention of female servicemembers and that its programs focus on increasing the retention of quality Marines—regardless of gender. As another example, Air Force officials stated that, while the Air Force has some specific initiatives that each have their own goals, performance measures, and timeframes included as part of those initiatives, these efforts have not been consolidated into a deliberate plan that targets female servicemembers. Navy and Army officials also stated that their respective services do not have plans specific to female retention efforts. We found that OSD has not provided guidance to the services to develop and implement plans to guide and monitor their efforts to recruit and retain female servicemembers. While DOD is in the process of updating its diversity and inclusion strategic plan to guide efforts through 2024, the updated plan will focus—like the 2012-2017 plan—on providing an overarching construct for the department’s diversity efforts. DOD’s 2012- 2017 Diversity and Inclusion Strategic Plan recognized that, due to the significant amount of time it takes to develop senior DOD leaders, it is essential that the department act to tap into the nation’s growing diverse talent pool. We have previously reported that pressures facing DOD— including increased competition for resources and involvement in more than a decade of conflict—underscore the importance of using a strategic approach to recruiting, developing, and retaining its workforce. In addition, although DOD has reported that the services generally met overall recruiting and retention goals—goals that do not consider gender—we have also reported in recent years on challenges associated with meeting its goals for certain critical skills and specialties—for example, the medical field and pilots—and rebuilding readiness across the force. Given appropriate planning and monitoring, the department could, as the former Secretary of Defense stated in 2015, benefit by drawing strength from the broadest possible pool of talent, which includes the female population that makes up over 50 percent of the population. Our prior work on effective strategic workforce planning states that agencies should periodically measure their progress toward meeting human capital goals and the extent to which human capital activities contribute to achieving programmatic goals and provide information for effective oversight by identifying performance shortfalls and appropriate corrective actions. In addition, internal control standards for the federal government state that management should define objectives clearly, including what is to be achieved , who is to achieve it, how it will be achieved—and in what timeframes—in addition to helping ensure that terms are understood at all levels. Finally, the standards also stipulate that management should develop information needed for corrective action, if necessary. Until DOD provides clear guidance and the services establish plans for monitoring and guiding their efforts to recruit and retain female active-duty servicemembers, including establishing goals, performance measures, and timeframes, the department may continue to experience slow growth of the female population and miss opportunities to retain a valuable segment of the population for its active-duty force. Women have been eligible for appointment and admission to the military service academies for over 40 years and, more recently, DOD has taken steps to open more positions to female servicemembers, including ground combat positions. However, while DOD has identified that it intends to increase diversity—including gender diversity—across the services, data show that the overall percentage of female servicemembers across the department has increased slightly from fiscal years 2004 through 2018. In addition to this slight overall growth, female enlisted and commissioned officer rates of attrition during that same period were slightly higher in comparison to their male counterparts. The percentage of female active- duty servicemembers tends to decrease at the 10-to-less-than-20 years of service category, and female active-duty servicemembers are more likely to separate from the military than their male counterparts. Moreover, from fiscal years 2004 through 2018, promotion rates for female active- duty servicemembers were slightly lower among the enlisted ranks in most years, but higher for officers as compared to their male counterparts. DOD has an ongoing effort to study the state of diversity in the department and is in the process of developing a new Diversity and Inclusion Strategic Plan for 2019-2024. However, these efforts address the department’s overall diversity and do not provide guidance to the services for developing plans to guide and monitor efforts to recruit and retain female active-duty servicemembers. Without such guidance and clear plans that include goals, performance measures, and timeframes to guide and monitor efforts to recruit and retain female servicemembers in the active-duty force, the services are not positioned to achieve the department’s goals of maintaining a ready force that includes the best and the brightest and is also representative of the population it serves. We are making a total of five recommendations—one to the Secretary of Defense and one to each of the military services. Specifically: The Secretary of Defense should ensure that the Under Secretary of Defense for Personnel and Readiness provides guidance to the services, for example, in its forthcoming diversity and inclusion strategic plan, to develop plans, with clearly defined goals, performance measures, and timeframes, to guide and monitor recruitment and retention efforts of female active-duty servicemembers in the military. (Recommendation 1) The Secretary of the Army should develop a plan, with clearly defined goals, performance measures, and timeframes, to guide and monitor the Army’s female active-duty servicemember recruitment and retention efforts. (Recommendation 2) The Secretary of the Navy should develop a plan, with clearly defined goals, performance measures, and timeframes, to guide and monitor the Navy’s female active-duty servicemember recruitment and retention efforts. (Recommendation 3) The Secretary of the Navy should ensure that the Commandant of the Marine Corps develops a plan, with clearly defined goals, performance measures, and timeframes, to guide and monitor the Marine Corps’ female active-duty servicemember recruitment and retention efforts. (Recommendation 4) The Secretary of the Air Force should develop a plan, with clearly defined goals, performance measures, and timeframes, to guide and monitor the Air Force’s female active-duty servicemember recruitment and retention efforts. (Recommendation 5) We provided a draft of this report to DOD for review and comment. In its written comments, reproduced in appendix V, DOD and the services concurred with our recommendations and noted steps the department has taken and would be taking. DOD also provided technical comments, which we incorporated, as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense; the Secretary of the Army; the Secretary of the Navy; the Commandant of the Marine Corps; the Secretary of the Air Force; the Office for Diversity, Equity, and Inclusion; and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. Caswell, David C.USAF Female Pilot Turnover Influence: A Delphi Study of Work-Home Conflict.Wright-Patterson Air Force Base, Ohio: Department of the Air Force, Air Force University, Air Force Institute of Technology (June 2016). https://ntrl.ntis.gov/NTRL/dashboard/searchResults/titleDetail/AD1054221 .xhtml. Dichter, Melissa E. and Gala True. ““This is the Story of Why My Military Career Ended Before It Should Have”: Premature Separation From Military Service Among U.S. Women Veterans.” Affilia: Journal of Women & Social Work, vol. 30, no. 2 (2015): 187-199. http://dx.doi.org/10.1177/0886109914555219. https://dialog.proquest.com/professional/docview/1681926518?accountid =12509. Keller, Kirsten M., Kimberly Curry Hall, Miriam Matthews, Leslie Adrienne Payne, Lisa Saum-Manning, Douglas Yeung, David Schulker, Stefan Zavislan, and Nelson Lim. Addressing Barriers to Female Officer Retention in the Air Force (Santa Monica, California: RAND Corporation, 2018). https://www.rand.org/pubs/research_reports/RR2073.html. Pierce, Penny F., TriService Nursing Research Program. Women Veterans Project: Operation Iraqi Freedom. Ann Arbor, Michigan: University of Michigan (2008). https://ntrl.ntis.gov/NTRL/dashboard/searchResults/titleDetail/PB2013101 316.xhtml. Stoker, Carol and Alice Crawford. Surface Warfare Officer Retention: Analysis of Individual Ready Reserve Survey Data. Monterey, California: Naval Postgraduate School, Graduate School of Business and Public Policy (January 22, 2008). https://ntrl.ntis.gov/NTRL/dashboard/searchResults/titleDetail/ADA476863 .xhtml. Williams, Nanette Marie, The Influence of Contemporary Army Culture on Senior Enlisted Women’s Decision to Commit to a Lifelong Career. Flint, Michigan: Baker College, 2013. https://search.proquest.com/docview/1427847908?accountid=12509. This report examines (1) trends in the percentage of female active-duty servicemembers in the military and their attrition rates from fiscal year 2004 through 2018, including the reported factors leading to that attrition; (2) how female active-duty servicemember promotion rates compare with those of their male counterparts and among female servicemembers with differing characteristics from fiscal years 2004 through 2018, and what factors influence these rates; and (3) the extent to which DOD and the military services have plans to guide and monitor female active-duty servicemember recruitment and retention. To address these objectives, we focused our review on active-duty enlisted, officers, and warrant officers in all ranks and pay grades, serving within the four military services (the Army, the Navy, the Marine Corps, and the Air Force). For our first and second objectives, we obtained and analyzed servicemember personnel data for fiscal year 2004 through 2018 from the Defense Manpower Data Center (DMDC), including, for example, service start date, branch of service, status, grade, gender, race, marital status, and whether the servicemember has dependents. We selected fiscal year 2004 through 2018 because, at the time we submitted our request for data, this was the most recent 15-year time period for which DMDC had complete data available. These data were obtained from three different files maintained by DMDC, including the (1) Active-Duty File Monthly Snapshots, (2) Transaction data for active-duty separations for October 1, 2003 through September 30, 2018, and (3) the Defense Enrollment Eligibility Reporting System. The data obtained from DMDC are granular down to the single month and single servicemember. We aggregated these data into a single file that allowed us to analyze them for (1) descriptive statistics to show trends and (2) modeling using duration analysis to show trends to examine the likelihood of occurrence for specific events for various demographic and DOD-specific administrative characteristics. We analyzed these data based on specific demographic characteristics, including gender, race, ethnicity, pay grade, and other variables. While the focus of this review was female active-duty servicemembers, we analyzed data on male active-duty servicemembers, using the same demographic and administrative characteristics, as the primary comparison group. We also analyzed the data to identify and compare the reasons for separation by these different groups and characteristics based on assigned separation designator codes. To assess the reliability of the data obtained from DMDC, we reviewed related documentation, for example, the data dictionary associated with the active-duty file; interviewed knowledgeable officials from DMDC; and conducted both electronic and manual data testing to look for missing or erroneous data. For example, within the data, some servicemembers changed their race, ethnicity, and/or gender over time. Through discussions with DMDC, we determined that these are often errors in the data, but in some instances can be the result of personal decisions by the servicemember. DMDC recommended using the last known instance for each of these attributes for each point on the servicemembers’ timeline. We implemented this recommendation, as it improved the results and findings and avoided servicemembers being counted across multiple, exclusive demographics—i.e., double counted. Based on these steps, we determined that these data were sufficiently reliable for the purposes of analyzing and reporting on the representation of servicemembers with specific demographic characteristics and the rates of attrition and promotion among those servicemembers for fiscal year 2004 through 2018. We also determined that fiscal year 2004 through 2018 DMDC data were sufficiently reliable for the purposes of constructing a duration analysis statistical model to estimate the likelihood of attrition by servicemembers with specific demographic factors. We used the fiscal year 2004 through 2018 DMDC data to construct descriptive statistics of the demographic composition of the services’ active-duty forces and drew comparisons between female and male servicemembers, and across demographic and administrative characteristics. According to service officials, the department does not have a universal definition for attrition. We, therefore, constructed attrition rates for active-duty servicemembers by capturing (1) any enlisted servicemember who separated more than 1 week from the end of his or her first service contract, and (2) any officer who separated within 3 years of his or her start date. Attrition rates were calculated by taking the total number of members who attritted, per the definitions above, in a given fiscal year and dividing that number by the total number of officers or enlisted servicemembers in that year, times 100 to express as a percent. To prevent double counting of non-attritted members across multiple fiscal years, attritted and non-attritted members were counted in the year that they entered service and not the year that they separated. In order to construct promotion rates for active-duty servicemembers, we used the servicemembers’ time-in-grade, time-in-service, and each service’s policy for time-in-service and time-in-grade minimums for each pay grade to determine eligibility for promotion. For every fiscal year, if a servicemember was eligible for promotion whether they promoted or not, the servicemember was counted as eligible. If the servicemember did promote, then the servicemember was counted as promoted. The promotion rate for each category was calculated as the total number of promoted servicemembers divided by the total number of promotion- eligible servicemembers, times 100 to express as a percent. We also conducted an analysis of associations between each of separation and promotion outcomes and certain demographic characteristics for servicemembers using the servicemember personnel data from DMDC for fiscal years 2004 through 2018, which included quarterly data on individual servicemembers. These data also contain information for each servicemember on the timing of his or her separation and promotions, if any. Specifically, we implemented a discrete time method for the analysis of event histories, using a logit specification. This is a type of duration analysis methodology that is suited to the analysis of event occurrences and their timing—which is the time elapsed until the event occurs (e.g. number of years until separation or promotion). We examined the extent to which each active-duty servicemember’s separation and promotions (or lack thereof) may be associated with certain factors related to that servicemember’s demographic and occupational characteristics. These factors were time-invariant (e.g. race, gender, etc.) or time-varying (e.g. occupation, marital status, etc.). For our duration models for separation, we generally included (1) gender, (2) marital status, (3) the existence of dependents, (4) race and ethnic groups, (5) pay grade categories, (6) having a bachelor’s degree or higher education degree versus not, (7) whether the individual has been assigned to an overseas duty location, (8) occupation, (9) fiscal year fixed effect, and (10) quarter-year time-in-service fixed effect. We tested multiple models and included various sets of factors. Since the number of female active-duty servicemembers decreases at higher pay grades, this was taken into account for our duration models for promotion. To ensure convergence of our promotion models, we made the following adjustments in control variables. We started with the Marine Corps’ promotion data because the service has the smallest proportion of female active-duty servicemembers among the four services. After testing with multiple sets of different control variables with the data, we decided to use the following control variables. (See table 1.) We could not control for all factors that may affect separation and promotion, such as a servicemember’s performance and labor market conditions. We also did not model the promotion process in the services. Our modeling should thus be viewed as evidence that may inform on possible associations in the data, and does not establish a causal relationship. Additional inquiry into each of the observed separation and promotion cases would be needed to truly ascertain the role of certain factors, such as gender, in each of these cases. Additionally, we conducted a literature review and content analysis of existing research on promotion and retention in the military, with a focus on female servicemembers. To identify studies, we conducted searches of various databases, including ProQuest, EBSCO, Westlaw Edge, Scopus, Dialog, and the National Technical Information Service, for English-language sources published in calendar year 2008 through 2018. We searched for peer-reviewed material, government and non- governmental reports, conference papers, books, and dissertations or theses. The database search was conducted from December 21, 2018 to January 10, 2019. This search and review process yielded 213 potentially relevant studies after initial scoping by a research librarian and, after additional screening of titles and abstracts for relevance, resulted in the selection of 87 studies for full text review. Specifically, two analysts sequentially reviewed the full texts for substantive content and reconciled any differences. Two methodologists sequentially reviewed the full texts for methodological considerations and reconciled any differences. Then the analysts and methodologists discussed and reconciled any remaining differences. To be included in our review, studies had to either (1) include factors servicemembers reported about intended or actual separations, including retention; or (2) report analyses designed to identify characteristics that statistically predict service separation or attrition differences among female servicemembers or between female and male servicemembers. The studies had to include primarily one or more of the four military services within DOD and could not focus exclusively on the Coast Guard. The studies also had to include primarily active-duty personnel and could not focus exclusively on reserve component personnel. Studies that focused only on recruitment or accessions, exit or lateral transfer from a career field but not separation from service, or data collected only from military spouses were also deemed out of scope. The studies we included in our literature review were published between 2008 and 2018 and included information relevant to our research objective on female servicemember retention, attrition, or promotion. From the group of 87 studies, we excluded 81 studies because they did not meet our inclusion criteria or the results were deemed not relevant to this review. The resulting six studies were further reviewed for content. We conducted a content analysis in order to be able to summarize the relevant results from the literature search by identifying recurring themes. To conduct this content analysis, the team developed a list of six overarching themes with three to seven sub-themes associated with each main theme. The resultant 54 sub-themes were documented in the team’s data collection instrument as a paired main theme and sub-theme. First, an analyst recorded an assessment of whether the study included the theme and sub-theme. A second analyst independently reviewed the same information and recorded an assessment. The two analysts reconciled their two independent assessments to produce the analysts’ consensus and recorded that consensus in the team’s final spreadsheet. All results reported from the studies reviewed were found to be sufficiently reliable for how they are used in this report and any limitations are mentioned in the text. For our third objective, we reviewed documentation on the Office of the Secretary of Defense’s (OSD) and services’ efforts to collect and analyze data on diversity in the department, as well as servicemember retention. We reviewed the department’s plans for developing and promoting diversity and inclusion in the force, including the department’s 2012-2017 Diversity and Inclusion Strategic Plan. We also reviewed a draft version of the department’s forthcoming plan for 2019-2024 Diversity and Inclusion Strategic Plan. We evaluated their efforts to determine whether they met federal internal control standards, including that management should design appropriate types of control activities such as defining objectives clearly and helping ensure that terms are understood at all levels. We reviewed other publications on female recruitment and retention in the military, including reports and briefings developed by the Defense Advisory Committee on Women in the Services (DACOWITS) and the 2011 final report of the Military Leadership Diversity Commission to determine what others had found and recommended with regard to female retention and participation in the military. We also analyzed our past reports and recommendations, for example, on military personnel management and DOD’s Career Intermission Pilot Program, among others. For all three objectives, we also interviewed officials from the Office of Military Personnel Policy Office and the Office for Diversity, Equity, and Inclusion (ODEI), both under the Office of the Under Secretary of Defense for Personnel and Readiness, as well as officials from the four military services. We also interviewed representatives from DACOWITS and the Service Women’s Action Network. Further, we reviewed previously made recommendations by DACOWITS and the Military Leadership Diversity Commission aimed at improving promotion and retention, specifically of female servicemembers, and interviewed OSD officials about any progress made by the department and the services to address these recommendations. We conducted this performance audit from September 2018 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Tables 2 through 14 present snapshots of active-duty data from the Defense Manpower Data Center, spanning the years of 2004 through 2018. We developed a set of statistical models—all discrete time duration analysis–using data from fiscal years 2004 through 2018, which accounted for active-duty servicemembers’ time in service—that is, the period of time from when they joined the military until their separation. We controlled for specific servicemember characteristics such as gender, branch of military service, pay grade, race or ethnicity, marital status, and the existence of dependents to estimate the association of these characteristics on the likelihood of active-duty servicemembers separating from the service. Table 15 depicts the results of our analysis. Positive numbers higher than 1.0 indicate the comparison group (e.g., married female servicemembers without dependents) is more likely to separate than the baseline group (e.g., unmarried female servicemembers without dependents). Positive numbers lower than 1.0 indicate the comparison group (e.g., female officers) is less likely to separate than the baseline group (e.g., female enlisted). Odds ratios from the duration analysis allow us to compare the relative relationships between various characteristics and separation from the military. For categorical variables, increase or decrease in the likelihood of separation is in comparison to an omitted category, or reference baseline group. Odds ratios that are statistically significant and greater than 1.00 indicate that servicemembers with those characteristics are more likely to separate than the baseline group. Odds ratio that are less than 1.00 indicate that servicemembers with those characteristics are less likely to separate. For example, the odds ratio for married female servicemembers with dependents in the Air Force are 1.203. This implies that the odds of separation for married female servicemembers with dependents in the Air Force are 1.203 times the odds of separation for unmarried female servicemembers without dependents in the Air Force, holding other factors constant, or that the odds of separation for married female servicemembers with dependents in the Air Force are about 20 percent higher than single female servicemembers without dependents in the Air Force, if other conditions remain constant. In addition to the contact named above, Kimberly Mayo (Assistant Director), Jennifer Weber (Analyst in Charge), Adriana Aldgate, Emily Biskup, Charles Culverwell, Edda Emmanuelli-Perez, Cynthia Grant, Chad Hinsch, Yvonne Jones, Zina Merritt, Amie Lesser, Samuel Moore, Moon Parks, Steven Putansu, Leigh Ann Sheffield, Michael Silver, Pamela Snedden, Carter Stevens, Elaine Vaurio, and Lillian M. Yob made key contributions to this report. Military Personnel: Observations on the Department of Defense’s Career Intermission Pilot Program. GAO-17-623R. Washington, D.C.: May 31, 2017. Military Personnel: Oversight Framework and Evaluations Needed for DOD and the Coast Guard to Help Increase the Number of Female Officer Applicants. GAO-16-55. Washington, D.C.: November 13, 2015. Military Personnel: DOD Should Develop a Plan to Evaluate the Effectiveness of Its Career Intermission Pilot Program. GAO-16-35. Washington, D.C.: October 27, 2015. Military Personnel: DOD Is Expanding Combat Service Opportunities for Women, but Should Monitor Long-Term Integration Progress. GAO-15-589. Washington, D.C.: July 20, 2015. Military Child Care: DOD Is Taking Actions to Address Awareness and Availability Barriers. GAO-12-21. Washington, D.C.: February 3, 2012. Women in the Military: Attrition and Retention. GAO/NSIAD-90-87BR. Washington, D.C.: July 26, 1990.", "summary": "The role of female servicemembers in the military has expanded in the last half century as restrictions on female servicemembers serving on active duty, including in combat, have been eliminated. DOD has also stated that recruiting and retaining women is important in order to reflect the nation's population and ensure strong military leadership. House Report 115-676 includes a provision that GAO review female retention and promotion in the military. This report examines (1) trends in the percentage of female active-duty servicemembers in the military and their attrition rates, including reported factors leading to attrition; (2) how female active-duty servicemember promotion rates compare with those of males and among females with differing characteristics, and what factors influence these rates; and (3) the extent to which DOD and the military services have plans to guide and monitor female active-duty servicemember recruitment and retention. GAO analyzed fiscal year 2004 through 2018 personnel data to identify attrition and promotion rates and conducted statistical modeling to determine the likelihood of separation and promotion, reviewed DOD reports and other literature on servicemember attrition, and interviewed officials from DOD and other military organizations. The Department of Defense (DOD) experienced slight increases in the overall percentage of female active-duty servicemembers from fiscal year 2004 through 2018 (15.1 percent in fiscal year 2004 to 16.5 percent in fiscal year 2018), with those percentages varying by pay grade category (see figure). During that period, female enlisted and commissioned officers had higher annual attrition rates than corresponding males. However, the gaps between male and female attrition rates have narrowed. For example, in fiscal years 2004 and 2018, female enlisted servicemembers' annual attrition rates were 33.1 and 8.6 percent, respectively, and enlisted males' annual attrition rates were 22.7 and 6.1 percent respectively. GAO's statistical model found that the likelihood of separation for female servicemembers is 28 percent higher than that of males. GAO's literature review of selected studies on reasons why females separate from the military identifed six themes, including family planning, sexual assualt, and dependent care, as influencing separations. GAO's analysis of fiscal year 2004 through 2018 data estimated that promotion rates were slightly lower for female enlisted in most years, but higher for officers as compared to their male counterparts. Specifically, female enlisted promotion rates ranged from 0.1 to 2.5 percentage points lower than male enlisted promotion rates during much of that period. However, from fiscal year 2004 through 2018, female commissioned officer promotion rates ranged from 3.3 to 5.3 percentage points higher than the rates of their male counterparts. GAO's statistical model also estimated that the likelihood of promotion outcomes varies by certain characteristics, such as gender and pay grade. For example, GAO estimated that the likelihood of promotion for female enlisted in the Navy may be lower than male enlisted, and the evidence is mixed for the other services. DOD has identified female recruitment and retention as important to diversity in the military, but the services do not have plans that include goals, performance measures, and timeframes to guide and monitor current or future efforts to recruit and retain females. According to officials, DOD is currently updating its diversity and inclusion strategic plan; however, neither its prior plan nor the updated plan include goals, such as recruitment or retention goals, performance measures, and timelines for any one particular demographic group. DOD officials stated that retention goals have, in the past, been misconstrued as quotas and, as such, the department does not set goals or targets for gender. However, goals are not quotas and can help guide continued improvement. Without DOD guidance and service plans with goals, performance measures, and timeframes to monitor female recruitment and retention efforts, DOD may continue to miss opportunities to recruit and retain a valuable segment for its active-duty force. GAO recommends that DOD provide the services with guidance to develop plans with goals, performance measures, and timelines to address female recruitment and retention efforts, and for the services to develop such plans. DOD concurred with the recommendations.", "document_type": "gao"}
{"report": "Education administers federal student aid programs, including the William D. Ford Federal Direct Loan (Direct Loan) program, through the Office of Federal Student Aid. Only Direct Loans are eligible for the TEPSLF and PSLF programs. Under the Direct Loan program, Education issues and oversees federal loans provided to students and contractors service these loans. Education currently contracts with nine loan servicers that each handle the billing and other services for a portion of the over $1 trillion in outstanding student loans provided through the Direct Loan program. These servicers track and manage day-to-day servicing activities. Education contracts with a single loan servicer to implement PSLF and TEPSLF, which includes responding to borrower inquiries, reviewing requests for loan forgiveness, and processing loan forgiveness for qualifying borrowers. Borrowers interested in pursuing loan forgiveness under either PSLF or TEPSLF must have their loans transferred to this loan servicer in order to proceed. TEPSLF is a temporary expansion of the PSLF program and the eligibility requirements for TEPSLF are largely the same as those of the PSLF program with a few key exceptions. Both provide eligible borrowers with forgiveness on the remaining balance of their Direct Loans after they have met all program requirements. To receive forgiveness for a loan under either PSLF or TEPSLF, borrowers are required to be employed in a public service job for 10 years when making 120 qualifying payments, at the time they apply for forgiveness, and at the time they receive forgiveness for their loans. Specifically, borrowers are generally required to: Work full-time for at least 10 years at a public service organization, a government organization, agency, or entity at any level (federal, state, local, or Tribal); a nonprofit, tax exempt organization (under section 501(c)(3) of the Internal Revenue Code); or another private nonprofit organization that provides certain public services. Not be in default on their loans. Make 120 on-time monthly loan payments for the full amount due on their bill. These monthly payments do not need to be consecutive. Key differences between PSLF and TEPSLF include: Qualifying repayment plans. PSLF generally requires borrowers to repay their loans through one of the eligible income-driven repayment plans or the Standard repayment plan. TEPSLF allows borrowers to qualify for loan forgiveness through several additional types of repayment plans that do not qualify for PSLF, including the Graduated repayment plan, Extended repayment plan, Consolidated Standard repayment plan, and Consolidated Graduated repayment plan. Funding. TEPSLF loan forgiveness is temporarily available to borrowers on a first-come, first-served basis until the $700 million is expended. The PSLF program will continue unless repealed by Congress. Specific payment requirements. For TEPSLF, the following two payments generally must be at least as much as the borrower would have paid under an income-driven repayment plan: (1) the payment made immediately prior to applying for TEPSLF, and (2) the payment made 12 months prior to applying for TEPSLF. In the context of high denial rates in the PSLF program and evidence that some borrowers were being misinformed by loan servicers about which repayment plans would qualify for PSLF, Congress appropriated $4.6 million for Education to conduct outreach on PSLF and TEPSLF. The legislation called for this outreach to be targeted to, among others, borrowers who would qualify for PSLF loan forgiveness except that they have made some or all of their payments through plans that do not qualify. Congress directed Education to implement a simple method for borrowers to apply for TEPSLF within 60 days after the legislation funding the program was enacted. In response, Education established a process in which borrowers send an email to the TEPSLF loan servicer with their name and date of birth to initiate their TEPSLF review and establish their place in line for TEPSLF funds. In addition to sending an email to initiate a TEPSLF request, Education requires that a borrower has submitted a PSLF application before they can be considered for TEPSLF (see fig. 1). While a PLSF application is not explicitly required by statute for a borrower to be considered for TEPSLF loan forgiveness, Education officials said that the department added this step to the process because the PSLF application form captures information the TEPSLF loan servicer needs to determine a borrower’s eligibility for TEPSLF. Education officials said that they added this step in order to roll out the TEPSLF program within the required 60 days. Education’s TEPSLF website states that borrowers interested in this temporary expanded loan forgiveness option must submit a PSLF application in order to be considered for TEPSLF. Even with this information, our review of TEPSLF loan servicer data found that 71 percent of denied TEPSLF requests were denied because the borrower had not submitted a PSLF application. Education officials said that they believed that many borrowers send a TEPSLF request without submitting a PSLF application because TEPSLF funding is temporary and borrowers know that sending the email request will hold their place in line for the limited funds. However, borrowers who have not submitted the PSLF application are sent a denial letter from the TEPSLF loan servicer. According to Education officials, these borrowers would lose their place in line and have to reapply if they want to be reconsidered for TEPSLF. Officials from Education, the TEPSLF loan servicer, and representatives from selected organizations representing student borrowers all said that the requirement to submit a PSLF application to be eligible for TEPSLF loan forgiveness can confuse borrowers. For example, Education officials acknowledged that the majority of TEPSLF requests come from borrowers who have not first submitted a PSLF application, and officials from the TEPSLF loan servicer said that borrowers who called were frequently confused when they received a TEPSLF denial based on the fact that they had not first submitted the PSLF application. In addition, representatives from the three student borrower groups we interviewed said that having to apply for PSLF before requesting TEPSLF often confuses borrowers and, in the opinion of officials from two of the three groups, leads directly to large numbers of TEPSLF denials. We also found some examples of borrower confusion about the PSLF application requirement in our review of borrower complaints. In three TEPSLF borrower complaints filed with Education that we reviewed, the borrowers expressed confusion and frustration about why they were being asked to submit an application for a program—PSLF—they knew they did not qualify for in order to receive TEPSLF loan forgiveness. Education’s policy of requiring all borrowers requesting TEPSLF to first submit a PSLF application has created a confusing process for borrowers. Education officials said that integrating the TEPSLF request into the PSLF application—for example, by including a checkbox that borrowers could use to request a TEPSLF review if the loan servicer finds they are ineligible for PSLF—would eliminate the need for borrowers to take multiple steps, reduce the number of borrowers who are denied, and improve service to borrowers. Education officials estimated that integrating the TEPSLF request into the existing PSLF process would require roughly a year in order to revise the PSLF application form, borrower communications, and program procedures, and to work with the loan servicer to implement new contractual requirements. Education officials told us that they will be implementing a new online portal in fall 2019 to provide better overall service to borrowers. They also stated that the new portal could incorporate an online integrated PSLF and TEPSLF application if they had sufficient resources and time, but that there were currently no specific plans to do so. While Education rolled out the process for requesting TEPSLF loan forgiveness in 2 months, it has not created a borrower-friendly TEPSLF process. This does not align with Education’s strategic plan objective to improve the quality of service to customers across the student aid life cycle. Further, Congress created the temporary expansion to the PSLF program to help certain borrowers who faced barriers obtaining PSLF loan forgiveness and required Education to develop a simple method for borrowers to apply for TEPSLF. While initiating a TEPSLF request through an email is straightforward, requiring borrowers to have submitted a PSLF application to successfully pursue TEPSLF loan forgiveness is confusing and inefficient for borrowers because borrowers must take multiple steps and complete an application for a program they do not qualify for. If Education were to allow borrowers to request TEPSLF loan forgiveness through an integrated application form, it would improve service to borrowers, reduce borrower confusion about how to obtain loan forgiveness, and better align with its strategic plan objective to improve service to borrowers. Further, although TEPSLF is a temporary opportunity, it could be years before the $700 million appropriation is exhausted, and it is therefore worthwhile for Education to invest resources in improving the process now. From May 2018 through May 2019, about 40,000 borrowers submitted TEPSLF requests for loan forgiveness and Education has approved or denied about 54,000 separate TEPSLF requests. Education has approved 1 percent (661) and denied 99 percent (53,523) of these requests, according to the most recent data from the TEPSLF loan servicer (see fig. 2). Of the 53,523 denied TEPSLF requests, about 38,000 were ineligible for consideration and were therefore denied because the borrower had not submitted a PSLF application, according to data from the TEPSLF loan servicer. Of the remaining denied requests, about 15,000 were denied because they did not meet other program requirements (see fig. 3). As we previously noted, under the current TEPSLF review process, the loan servicer first checks to see if the borrower requesting TEPSLF has submitted a PSLF application. If the borrower has not done so, the loan servicer does not conduct any further review of the borrower’s request and sends the borrower a denial letter informing them they have to submit the PSLF application before the borrower can be considered for TEPSLF. Without the loan servicer conducting any further review of a borrower’s request, it is impossible to know how many of the roughly 38,000 requests that were denied because the borrower had not yet submitted a PSLF application would have otherwise met all other program requirements and been approved for TEPSLF loan forgiveness. The large number of TEPSLF requests denied for not submitting a PSLF application provides further evidence that many borrowers may be confused about the process for obtaining TEPSLF loan forgiveness. Further, more than 5,000 (about 10 percent) of the TEPSLF requests were denied because the borrower had not been repaying their loans for at least 10 years, which indicates that they had not yet made 120 qualifying payments—a straightforward program requirement. Since TEPSLF became available in May 2018, Education has approved TEPSLF loan forgiveness totaling about 4 percent (approximately $26.9 million) of the $700 million appropriated for TEPSLF loan forgiveness, according to the most recent data available from the TEPSLF loan servicer at the time of our review (see fig. 4). Of the 656 borrowers approved for TEPSLF loan forgiveness, the borrowers received an average of about $41,000 in loan forgiveness, with loan forgiveness amounts ranging from about $190 to about $227,000. Education does not provide complete information to borrowers about options they have to contest a denied TEPSLF request. Specifically, the letter the TEPSLF loan servicer sends to the borrower communicating a decision to deny the TEPSLF request includes the reason for the denial and the TEPSLF loan servicer’s customer service number for borrowers to call with questions. An FSA official told us that while there is no formal process for borrowers who are dissatisfied with their TEPSLF or PSLF determinations to contest them, borrowers do have additional options for addressing concerns, such as an additional review by the TEPSLF servicer, or a complaint to the FSA Feedback System or Ombudsmen (see fig. 5). According to Education officials, when a borrower is denied loan forgiveness, they can call the TEPSLF loan servicer’s customer service number with questions about the denial. TEPSLF servicer officials said that if the customer service representative is unable to resolve the borrower’s questions, the representative can elect to elevate the borrower’s concern internally within the TEPSLF loan servicer, which may result in a second review by loan servicer management. Education and TEPSLF loan servicer officials said that borrowers who are not able to resolve their issues with the loan servicer can bring their issues directly to Education. Specifically, if a borrower is dissatisfied with their TEPSLF decision, they can submit their concern through the online FSA Feedback Tool. Borrowers can also contest the decision with the FSA Ombudsman Group. Education officials told us it does not provide information about these options in its denial letters or on its TEPSLF website, noting that borrowers could find this information at the bottom of FSA’s main website. Education officials also stressed the importance of borrowers resolving their concerns first with their loan servicer directly before pursuing other avenues, and said that this is part of the reason why they do not include this information in letters sent to borrowers. All TEPSLF denial letters include a statement at the bottom of the letter indicating that if borrowers had questions about the information in their denial letter, they should call the general customer service number at the TEPSLF loan servicer for assistance. The letters did not explain how the servicer could potentially do a second review or subsequently refer the matter to Education. Information about the potential for a second review at the loan servicer and the option to raise an issue with Education directly would help borrowers who are unable to resolve their concerns by calling the general customer service number at the loan servicer. Additional information on options for contesting decisions is not necessary for all TEPSLF borrowers who are denied. For example, it may not be appropriate to include this information in denial letters sent to borrowers who do not meet basic program requirements—for example, those who have no federal Direct Loans. However, borrowers who are denied for reasons that are more prone to error, such as having fewer than 120 qualifying payments, are not made aware of all the available options so they can make informed decisions about how to best resolve their concerns. We previously reported that Education does not ensure that the loan servicer responsible for PSLF and TEPSLF is receiving consistent loan payment history information from other loan servicers, increasing the risk of inaccurate qualifying payment counts. This also raises the risk of inappropriate denials for TEPSLF. Our review of TEPSLF complaints made to Education from borrowers found eight examples of borrowers contesting the loan servicer’s determination of the number of qualifying payments. In six of these instances, the TEPSLF servicer determined that the borrowers were correct and had met requirements for loan forgiveness. Given the risk of denial errors, additional information about options for borrowers who are dissatisfied with their TEPSLF denial determination is especially important. While there is information about options for contesting decisions at the bottom of FSA’s main website, it is not incorporated into the TEPSLF website and borrowers may not know where to find this information. Federal internal control standards for external communication with stakeholders call for communication of quality information with external parties to achieve program objectives. Providing this information in relevant denial letters and Education websites will increase the likelihood that borrowers with valid concerns will have their TEPSLF requests appropriately resolved. Education and the TEPSLF loan servicer conduct direct outreach to certain borrowers about TEPSLF. For example, when TEPSLF was first rolled out, Education sent a notice to over 1,000 borrowers who had been denied PSLF due to a lack of 120 qualifying payments, but who had been in repayment for at least 10 years. Education officials told us that they had identified this group of borrowers as the most likely to be eligible for TEPSLF. This notice informed borrowers of the new TEPSLF loan forgiveness opportunity and told them how to apply for it. Education officials told us that they continue to review the PSLF denial list on a weekly basis and send notices to those whom they have determined to be the most likely to qualify for TEPSLF loan forgiveness. In addition, borrowers who have previously expressed interest in TEPSLF by sending an email to request TEPSLF loan forgiveness will be sent a TEPSLF outreach letter by the TEPSLF loan servicer under certain conditions: after submitting a new employment certification form, or after submitting a new PSLF application that is subsequently denied. In these two circumstances, the TEPSLF loan servicer sends the borrower a letter suggesting that they may now be eligible for TEPSLF loan forgiveness and would need to re-request such loan forgiveness. Education officials told us that the agency has focused on a broad, general outreach strategy; however, we found that its online information is limited because TEPSLF information is not included in several key online sources. Education and TEPSLF loan servicer officials told us that they primarily direct borrowers to online sources to inform them about TEPSLF requirements. For example, Education has created a TEPSLF-specific website and the TEPSLF loan servicer’s website references the TEPSLF opportunity and links to Education’s TEPSLF website if borrowers would like more information. However, we found that while all nine of the loan servicers’ websites contain some information on the PSLF program, none of them (other than the TEPSLF loan servicer) included TEPSLF information on their websites or provided a link to Education’s TEPSLF website. While Education officials told us that only the TEPSLF servicer is required to have TEPSLF information on its website, other loan servicers may also serve borrowers who are potentially eligible but may be unaware of TEPSLF. In addition, borrowers with other loan servicers who are interested in TEPSLF may not be aware that they have to transfer to the TEPSLF loan servicer to obtain loan forgiveness. Further, according to agency officials, Education’s PSLF Online Help Tool also does not include any TEPSLF information, and Education has not created a similar tool for TEPSLF. Education officials told us that the PSLF Online Help Tool, which assists borrowers with determining whether their loans and employment qualify and which PSLF forms they need to submit, is one of the primary PSLF outreach mechanisms to inform borrowers about PSLF eligibility. According to Education data, since the rollout of the online tool in December 2018 through the beginning of March 2019, about 340,000 users have used the online tool, and about 100,000 have logged on and have collectively generated about 40,000 PSLF-related forms, such as PSLF application forms. However, according to Education officials, the online tool does not include any information on TEPSLF. Education officials stated that the first phase of the Online Help Tool was focused on informing borrowers about eligibility requirements for PSLF and that as the department makes enhancements to phase two of the Online Help Tool, it could consider adding TEPSLF information and functionality. Both FSA and TEPSLF loan servicer officials stated that having information on TEPSLF integrated into the PSLF Online Help Tool would be beneficial for borrowers and would reduce confusion about TEPSLF. Federal internal control standards state that management should externally communicate the necessary quality information to achieve the entity’s objectives. Including TEPSLF information in the PSLF Online Help Tool and noting it on all loan servicer websites could increase borrower awareness of TEPSLF and the likelihood that borrowers are able to take advantage of this opportunity. The loan forgiveness opportunity through TEPSLF is an expansion of the PSLF program and helps borrowers who hoped to qualify for PSLF but who did not realize they were in an ineligible loan repayment plan. Instead of integrating the expanded loan forgiveness opportunity into the existing PSLF process, Education required borrowers to have submitted a separate PSLF application before the loan servicer will consider a borrower’s TEPSLF request. The large number of requests denied because borrowers had not submitted a PSLF application suggests that borrowers are confused about this requirement. In some cases, these borrowers may have been working in public service jobs for years believing they were on track for loan forgiveness, only to find out later that they did not qualify. While the loan forgiveness opportunity through TEPSLF is only available until the $700 million in funding has been spent, a relatively small amount of total funding has been spent so far. It is possible that the program could continue for years, supporting the case for investing in improvements to the process now. Integrating the process for obtaining loan forgiveness through TEPSLF into the PSLF application would be easier for borrowers and help Education meet its goal to improve customer service. Information provided in TEPSLF denial letters and on the TEPSLF website does not explain what options are available to borrowers if they want to contest the loan servicer’s determination. While additional information on this topic is not necessary for borrowers who do not meet basic program requirements—for example, those who have no qualifying federal loans—this information would help certain borrowers whose TEPSLF requests may have been denied. By including this additional information on the TEPSLF website and in denial letters to these borrowers, the borrowers can then pursue additional options to contest the denial and help Education avoid denial errors. Finally, Congress provided funding and tasked Education with conducting outreach to borrowers to help increase overall borrower awareness of the public service loan forgiveness programs. While Education has engaged in some outreach activities, Education is missing opportunities to reach out to borrowers potentially eligible for TEPSLF—specifically, by not requiring all loan servicers’ websites to include information about TEPSLF and not including TEPSLF information in the PSLF Online Help Tool. TEPSLF was created to provide relief to a group of borrowers who were ineligible because they were repaying their loans on repayment plans that were not eligible for the original PSLF program. Without improved TEPSLF outreach in these two areas, however, many of these borrowers who were initially unable to qualify for the PSLF program may be unaware of the TEPSLF opportunity that was designed to help them. We are making the following four recommendations to Education’s Office of Federal Student Aid: The Chief Operating Officer of the Office of Federal Student Aid should integrate the TEPSLF request into the PSLF application, for example, by including a checkbox on the PSLF application, to provide borrowers a more seamless way to request TEPSLF consideration. (Recommendation 1) The Chief Operating Officer of the Office of Federal Student Aid should provide certain borrowers, for example, those who are denied TEPSLF for not having 120 qualifying payments, with more information about options available to contest TEPSLF decisions on the TEPSLF website and in their denial letters. (Recommendation 2) The Chief Operating Officer of the Office of Federal Student Aid should require all loan servicers to provide TEPSLF information on their websites. (Recommendation 3) The Chief Operating Officer of the Office of Federal Student Aid should include TEPSLF information in its PSLF Online Help Tool. (Recommendation 4) We provided a draft of this report to Education for its review and comment. In its comments, reproduced in appendix I, Education concurred with each of our recommendations and identified steps it plans to take to implement them. To make the TEPSLF loan forgiveness process easier for borrowers, Education stated that it will integrate the TEPSLF request into the PSLF application as part of the improvements planned for the PSLF application under its new online interface for student borrowers. Regarding our recommendation to provide certain borrowers with more information about options available to contest TEPSLF decisions, Education stated that it will add information for borrowers on the procedures for contesting TEPSLF decisions to FSA’s specific TEPSLF website and in relevant TEPSLF denial letters. To improve outreach and help increase overall borrower awareness of TEPSLF, Education stated it will require all loan servicers to provide TEPSLF information on their websites within 120 days. In addition, Education stated that it will also include TEPSLF information in the PSLF Help Tool. We also provided relevant report sections to the TEPSLF loan servicer for technical comments. The TEPSLF servicer provided technical comments, which we incorporated as appropriate. We are sending copies of this report to relevant congressional committees, the Secretary of Education, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0534 or emreyarrasm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Michelle L. St. Pierre (Assistant Director), Nora Boretti (Analyst-In-Charge), and Aaron Karty made significant contributions to this report. Also contributing to this report were James E. Bennett, Deborah Bland, Alicia P. Cackley, Marcia L. Carlsen, Linda A. Collins, William W. Colvin, Alex Galuten, Sheila R. McCoy, Jean L. McSween, Jessica S. Orr, Debra Prescott, and Ashanta Williams. Public Service Loan Forgiveness: Education Needs to Provide Better Information for the Loan Servicer and Borrowers. GAO-18-547. Washington, D.C.: Sept. 5, 2018. Federal Student Loans: Further Actions Needed to Implement Recommendations on Oversight of Loan Servicers. GAO-18-587R. Washington, D.C.: July 27, 2018. Federal Student Loans: Education Could Improve Direct Loan Program Customer Service and Oversight. GAO-16-523. Washington, D.C.: May 16, 2016. Federal Student Loans: Key Weaknesses Limit Education’s Management of Contractors. GAO-16-196T. Washington, D.C.: Nov. 18, 2015. Federal Student Loans: Education Could Do More to Help Ensure Borrowers Are Aware of Repayment and Forgiveness Options. GAO-15-663. Washington, D.C.: Aug. 25, 2015.", "summary": "In the context of high denial rates in the PSLF program, Congress appropriated $700 million in 2018 for a temporary expansion to the public service loan forgiveness program for certain borrowers who were not eligible for the original PSLF program. TEPSLF funds are available on a first-come, first-served basis. GAO was asked to review TEPSLF. This report examines (1) the extent to which the process for obtaining TEPSLF is clear to borrowers, (2) what is known about loan forgiveness approvals and denials, and (3) the extent to which Education has conducted TEPSLF outreach. GAO analyzed data from the TEPSLF servicer on loan forgiveness requests from May 2018 through May 2019 (the most recent available at the time of our review); reviewed Education's guidance and instructions for the TEPSLF servicer; assessed Education's outreach activities; interviewed officials from Education, the TEPSLF servicer, and selected groups representing borrowers; and reviewed borrower complaints about TEPSLF submitted to Education. The Department of Education's (Education) process for obtaining Temporary Expanded Public Service Loan Forgiveness (TEPSLF) is not clear to borrowers. Established in 2007, the Public Service Loan Forgiveness (PSLF) program forgives federal student loans for borrowers who work for certain public service employers for at least 10 years while making 120 payments via eligible repayment plans, among other requirements. In 2018, Congress funded TEPSLF to help borrowers who faced barriers obtaining PSLF loan forgiveness because they were on repayment plans that were ineligible for PSLF. Congress also required Education to develop a simple method for borrowers to apply for TEPSLF. Education established a process for borrowers to initiate their TEPSLF requests via e-mail. The agency also required TESPLF applicants to submit a separate PSLF application before it would consider their TEPSLF request. Agency officials said they established this process to quickly implement TEPSLF and obtain the information needed to determine borrower eligibility. However, the process can be confusing for borrowers who do not understand why they must apply separately for PSLF—a program they are ineligible for—to be eligible for TEPSLF. Requiring borrowers to submit a separate PSLF application to pursue TEPSLF, rather than having an integrated request such as by including a checkbox on the PSLF application for interested borrowers, is not aligned with Education's strategic goal to improve customer service to borrowers. As a result, some eligible borrowers may miss the opportunity to have their loans forgiven. As of May 2019, Education had processed about 54,000 requests for TEPSLF loan forgiveness since May 2018, and approved 1 percent of these requests, totaling about $26.9 million in loan forgiveness (see figure). Most denied requests (71 percent) were denied because the borrower had not submitted a PSLF application. Others were denied because the borrower had not yet made 120 qualifying payments (4 percent) or had no qualifying federal loans (3 percent). More than a year after Congress initially funded TEPSLF, some of Education's key online resources for borrowers do not include information on TEPSLF. Education reported that it has conducted a variety of PSLF and TEPSLF outreach activities such as emails to borrowers, social media posts, and new website content. However, Education does not require all federal loan servicers (who may serve borrowers interested in public service loan forgiveness) to include TEPSLF information on their websites. Further, Education's Online Help Tool for borrowers—which provides information on PSLF eligibility—does not include any information on TEPSLF. Requiring all loan servicers to include TEPSLF information on their websites and including TEPSLF information in its online tool for borrowers would increase the likelihood that borrowers are able to obtain the loan forgiveness for which they may qualify. GAO is making four recommendations, including that Education integrate the TEPSLF request into the PSLF application, require all loan servicers to include TEPSLF information on their websites, and include TEPSLF information in its PSLF Online Help Tool. Education agreed with GAO's recommendations. (617) 788-0534 or emreyarrasm@gao.gov .", "document_type": "gao"}
{"report": "U.S. and foreign air carriers have cooperated in a variety of ways to expand their international reach and service offerings. Legal requirements in the United States and other countries prevent mergers between U.S.- owned airlines and foreign owned airlines and also place restrictions on carriers providing end-to-end service between locations within other countries as well as between third countries. Air carriers may cooperate with each other to provide a wider range of services, more seamlessly, despite these restrictions. Simple forms of cooperation include, for example, “interlining,” which are voluntary commercial agreements to carry passengers across two or more carriers on the same itinerary, and “codesharing,” an agreement whereby carriers place their marketing code on a flight operated by another carrier. This practice allows consumers to book a single ticket for an itinerary involving two separate airlines, with one airline selling tickets under its own code for travel on the other carrier’s flight. These cooperative activities allow carriers to access each other’s network with varying degrees of cooperation. As part of their cooperative efforts, some carriers have formed global alliances. An alliance is an agreement between two or more airlines to link each of the airlines’ route networks and coordinate on specified activities, such as marketing and sales; coordination of airport operations (e.g., sharing gates or baggage facilities); and frequent flyer program accrual and redemptions. Alliances represent more involved coordination than interline or codeshare relationships. This expanded cooperation, according to DOT, allows participating carriers to further expand the geographic reach of their respective networks that the carriers would not be able to do on their own, because of the aforementioned legal restrictions and due to the economic and operational difficulties a single carrier would face implementing such an expansion in foreign markets. As of January 2019, there were three global airline alliances, each with a major U.S. member airline and multiple foreign partners: Oneworld (American Airlines); SkyTeam (Delta Air Lines); and the Star Alliance (United Airlines). These three airline alliances have 61 airline members: 13 for Oneworld, 20 for SkyTeam, and 28 for the Star Alliance. Many of the carriers within each of these alliances, as well as other carriers, have pursued antitrust immunity from DOT to cooperate more closely on key economic elements of their businesses that U.S. antitrust laws might prohibit. The specific activities are delineated in cooperative agreements and carriers have the option to implement such agreements without antitrust immunity from DOT. Carriers are more likely to pursue immunity when the proposed cooperation—and risk of antitrust violations—involves increasingly integrated business functions, according to DOT. However, once carriers that are party to such an agreement are immunized, carriers can cooperate more comprehensively than through interlining and codesharing arrangements (see fig. 1). For example, these agreements may stipulate that carriers share revenues across their flights, regardless of which carrier operates the flight, and jointly coordinate on schedules, prices, and sales. Since 1993, when DOT immunized the first cooperative agreement, between Northwest Airlines and KLM Royal Dutch Airlines, DOT has adjudicated 38 cases which involved one or more U.S. carriers and foreign carriers. Currently, United, Delta, and American—and their major foreign airline partners—are each members of multiple immunized cooperative agreements with their foreign airline partners. As a result, immunized carriers now provide air service across the globe. For example, in 2017, immunized carriers across these three alliances provided approximately 75 percent of the available seats on trans-Atlantic flights between the United States and Europe, and also provided on trans-Pacific service to Asia, and Australia, as well as service to South America. DOT’s process for reviewing each application for antitrust immunity includes two analytic steps. First, DOT must decide whether to approve a proposed cooperative agreement. In this step, by statute, DOT is directed to approve cooperative agreements deemed “not adverse” to the public interest. DOT conducts a competitive analysis to make this determination. Second, DOT decides whether to grant antitrust immunity to the agreement’s partners for activities undertaken pursuant to the approved agreement. DOT’s statutory authority provides for such a grant of immunity only to the extent necessary for the parties of the agreement to go forward with the transaction and only if the immunity is “required by the public interest,” vis-a-vis the creation of consumer, commercial, or other public benefits that would not otherwise occur. These steps will be discussed in detail in the following section. The statute does not detail specific competitive metrics or public benefits that DOT must consider in its evaluation but rather provides DOT leeway in making such determinations. The Department of Justice, which is responsible for reviewing and approving domestic mergers, may provide DOT with input during deliberations. DOT may also consult with relevant authorities in the foreign partner’s country. In granting antitrust immunity, DOT may require carriers to comply with specific conditions and, for grants of antitrust immunity approved since 2009, reporting requirements. DOT’s process to consider requests for immunity follows procedural steps delineated in the Administrative Procedure Act (APA). The APA provides for public notice and comment. At the beginning of the proceeding, carriers applying for immunity place information about the proposed cooperative agreement in a public docket. DOT staff then review this material, may request additional information to address any questions raised by their review, and will solicit comments from the public. The APA, in contrast to Department of Justice merger review procedures, specifies steps that afford public involvement and requires agencies to respond to the public comments. In DOT’s proceedings, the Department typically issues “show cause” orders that articulate the tentative approval or disapproval of the application. After publishing this show-cause order, DOT solicits additional public comments for review prior to issuing a final decision. See figure 2 for a summary of this process. DOT’s statutory authority indicates that DOT may conduct “periodic reviews,” but the statute does not include a definition of the nature or frequency of these reviews. All of DOT’s orders granting antitrust immunity state DOT may amend or revoke a grant of immunity at any time. Further, after DOT issues a final order that approves a request for antitrust immunity, the public docket remains open and provides a forum for ongoing public comments that DOT is obligated to respond to. DOT analyzes competitive and public benefit effects, taking into consideration the potential effects on consumers, when deciding whether to approve cooperation agreements and grant carriers antitrust immunity, based on our review of DOT’s processes. In competitive and public benefit analyses, DOT uses the professional experience and expertise of staff to identify and assess relevant market factors, the terms of proposed cooperative agreements, supporting documents, and other information in light of the facts and circumstances specific to each case. DOT’s competitive analysis focuses on the likely effect of the cooperative agreement on competition in key airline markets, while the public benefits analysis focuses on the likelihood of carrier integration yielding consumer benefits. As discussed earlier, DOT’s process includes opportunities for stakeholders’ participation. Stakeholders we interviewed considered the overall review process transparent, though some had criticisms of the underlying economic evidence DOT uses to predict if, and how, consumer benefits might arise. The potential effects of proposed cooperative agreements on competition, and thus consumers, are central to DOT’s analysis. Specifically, DOT looks to see how the agreement may affect competition across routes affected by the alliance agreement. To make this assessment, according to DOT documentation that we reviewed and officials whom we interviewed, DOT focuses on three key elements of the proposed agreement. Specifically, DOT identifies (1) the geographic scope of the proposed alliance and which markets that the agreement would affect; (2) the number of competitors in each market, their market shares, and the level of market concentration; and (3) the feasibility and likelihood of market entry by new competitors into markets that might be adversely affected by the agreement as well as the ability of existing carriers to compete in such markets (see table 1). DOT’s assessment is based on an array of information provided by applicants and third parties. This information may include competitive analyses or other studies conducted by consulting economists for the applicants, and business plans and data, among other things. DOT may also independently use departmental databases to conduct its own analysis, including those data DOT collects from foreign carriers pursuant to data-reporting requirements in existing grants of antitrust immunity. DOT looks at competitive issues at the region-to-region (e.g., United States to Europe), country-to-country (United States to France), and city- to-city levels (e.g., New York-to-Paris city pair market), or airport-to- airport pairs (Chicago O’Hare-to-London Heathrow). The analysis focuses largely on city- or airport-pairs because the sale of air transportation between cities/airports is the product being sold by airlines and purchased by the consumer, according to DOT officials. Consequently, DOT looks most closely at those city-pair markets where the number of competitors is expected to decline, such as from 3 to 2 or 2 to 1, when the applicants are counted as a single competitor. According to DOT officials, this approach to competitive analysis is consistent with legal and economic practice and in the application of antitrust laws and principles used by other competition authorities, such as the Department of Justice. Officials then recommend determinations as to whether such a reduction in competitors in these markets is likely to be harmful to competition and, in turn, to consumers. According to DOT officials, the department has no predetermined threshold for defining substantive competitive harm because it would not be appropriate to pre-define what constitutes a “substantial reduction in competition” that would necessitate disapproval of an application. Instead, the Department looks at the characteristics of discrete markets where there is a reduction. In addition to looking for potential competitive harms in the city-pair analysis, DOT’s competitive analysis also assesses if the agreement could enhance competition in some markets. In particular, DOT may find that certain markets will have an increase of an effective competitor due to the agreement. Specifically, based on applicants’ filings, DOT may expect the cooperating carriers to enter new routes that neither had previously served. For example, DOT approved a grant of immunity in 2010 based on expectations that the applicants would have increased opportunities for new or expanded transpacific routes and service and enhanced connecting options, among other benefits. Additionally, if two carriers each served a market with a market share under 5 percent—the threshold DOT uses for deeming a carrier as providing competitive service on the route—the agreement may push that market share above the 5-percent threshold and effectively result in a new competitor on the route. Also, according to DOT, the carriers’ agreement could result in connecting flights across two carriers to become effectively “online” (as opposed to “interline”) for some city-pair markets due to the agreement. This could potentially offer consumers competing options among airlines that provide direct flights on a given route. We reviewed DOT documentation in which its analyses had projected these improved competitive outcomes across thousands of city-pair markets based on an application for a cooperative agreement. Finally, according to DOT orders, carrier agreements can promote competition in various markets, if the agreements strengthen inter-alliance network competition. For example, DOT approved and immunized the cooperative agreement between the major partners of the Oneworld alliance, in part, based on the finding that a third immunized global network could better discipline the fares and services offered by the Star and SkyTeam alliances. Specifically, in approving the immunity application, DOT noted consumer benefits stating that “enhanced inter-alliance competition is beneficial for consumers across many markets, in particular the hundreds of transatlantic markets in which the applicants become more competitive as a direct result of the alliance. Travelers in those markets gain new competitive options.” Though DOT may find prospective competitive harm from the agreement, such as a reduction in the number of competitors in certain markets, DOT does not necessarily reject the application if a DOT-stipulated remedy can potentially mitigate those harms, according to department officials. DOT has used different potential remedies over the years, including carving out specified city-pairs from a grant of immunity and requiring carriers to divest from slots at specific airports (see table 2). DOT officials indicated carve-outs are less favored now than in the past because carve-outs on specific routes can, in DOT’s view, diminish broader public benefits of the alliance by limiting the degree carriers can merge their operations. DOT currently has 11 active carve-outs in three alliances, with the last carve- out issued in 2009. More recently, DOT officials indicated mitigations based on slot divestitures have the potential to better target competitive harms on specific routes by enabling new entrants to these cities with slot-constrained airports. DOT required slot-based remedies in two grants of immunity, one in 2010 and one in 2016. In the 2010 immunity grant, DOT required applicants to relinquish slots at London’s Heathrow airport and specified that two slots must be for Boston-Heathrow services and two for services between any U.S. location and Heathrow. DOT expected these remedies, once implemented, to enable other carriers to start new services to compete with the newly immunized alliance, thereby ensuring adequate competition remains in the affect market. Whether and what mitigation strategies are pursued can be a contested aspect of the proceeding, in which DOT, the applicants, and third-parties debate the competitive implications of the agreement and the mitigations based on the facts and circumstances of each situation. In a 2016 case involving Delta and Aeromexico, DOT included two new or rarely used conditions in the grant of antitrust immunity. Specifically, to address competitive concerns specific to this case, DOT made its approval conditional upon the removal of exclusivity clauses in the joint venture agreement that precluded specified types of cooperation with other carriers. Though the carriers argued that such clauses were necessary to encourage long-term investment in their cooperative products and services, DOT took into account the perspectives from stakeholders’ docketed comments, concluding that such clauses could give the carriers an undue ability or incentive to foreclose actual or potential competition. Additionally, DOT placed a 5-year sunset provision on its grant of antitrust immunity to Delta and Aeromexico to allow DOT a defined opportunity to revisit whether specific slot constraints identified at the Mexico City airport had been resolved. Prior grants of immunity regularly included requirements for carriers to resubmit their cooperative agreements to DOT after 5 years as part of DOT’s subsequent monitoring (discussed below), but the immunity was not time limited. DOT officials explained the inclusion of the sunset provision was to address concerns specific to this case, rather than a new departmental policy. Once the competitive analysis and any decisions on mitigations are complete, DOT determines whether, on balance, the proposed agreement would likely have an overall positive, neutral, or negative competitive effect and decides whether to approve the agreement. In all cases where DOT has granted antitrust immunity, DOT found the proposed cooperative agreements, on balance or with any specified remedy in place, to be either neutral or pro-competitive. However, DOT has denied approval of a proposed agreement, citing that the carriers’ combined market share on routes where they both operate service would be so dominant they could, for example, raise prices to the detriment of consumers. DOT conducts public benefits analyses to determine if there are benefits of proposed cooperative agreements for consumers. Based on our review of applications, carriers typically point to varied benefits such as the potential for lower fares on certain routes, improved connectivity, and reciprocal frequent flier benefits for consumers. In considering the public benefits claims made by applicants as well as any potential benefits of the proposed agreement identified by DOT, the department assesses whether (1) the public benefits identified are significant and likely to be realized in a timely fashion and (2) if a grant of immunity is necessary for the carriers to go forward with the agreement such that benefits will be achieved. DOT officials emphasized that this assessment focuses on the carriers’ anticipated level of integration. The officials said higher levels of cooperation in a proposed agreement, given the nature of the airline industry and depending on the economic incentives employed, can lead to lower fares, especially for connecting itineraries. Though DOT officials acknowledged that the flow of consumer benefits due to high levels of carrier cooperation is not absolute or certain, they said DOT’s analysis has consistently supported the notion that connecting passengers who traverse carriers on a given itinerary pay less as cooperation between alliance carriers increases. DOT has applied this policy in each of the proceedings involving grants of antitrust immunity to the three major air alliances—SkyTeam (2008), Star Alliance (2008), and Oneworld (2010)— as well as subsequent cases. For example, DOT approved immunity within the Star Alliance based on its expectation that fares for connecting itineraries for Star’s transatlantic routes would decrease, benefiting the majority of its transatlantic passengers. DOT further noted that this connecting service would “discipline fares on non-stop routes,” as well. The practical consequence of this policy, according to DOT officials, is that DOT expects applicants to present detailed cooperation agreements, which show integrative efficiencies and processes, at the time the requests for antitrust immunity are made. In other words, DOT expects antitrust immunity, when provided, will provide consumers with an array of benefits—lower connecting fares, new route offerings, among others— that follow from these business efficiencies. DOT’s public benefits analysis considers the specific provisions of each proposed agreement to assess how the applicants plan to coordinate a wide range of business functions. These can include network and capacity planning, scheduling, pricing, sales, revenue management, and customer service, among other considerations. DOT officials told us that they examine the carriers’ revenue-sharing plans, corporate strategic documents, and other relevant documentation. For example, DOT may look to see if carriers plan to: Share revenue in a manner to provide incentives to carriers to coordinate the management and selling of their combined networks to make more seats and more frequencies on routes linking their respective networks available, substantially increasing connectivity and time-of-day schedule options and improving customer service by treating their partner’s customers just as they would their own. Align their different ticket fare and availability classes and procedures such that their revenue management systems make seats available on domestic flights for passengers connecting from the foreign partner’s flights at the same levels and on the same terms as if customers were connecting online from their own international flights. Coordinate marketing and incentivize sales staff to promote the carriers’ combined, rather than individual networks, and thereby creating more options for consumers. Align products for a consistent, seamless passenger experience (e.g., baggage fees, upgrade policies, frequent flyer program rules). According to DOT, the agency further reviews governance and revenue- sharing provisions to ensure that sufficient economic incentives exist to substantially increase the number of passengers flowing through the combined networks and to significantly increase capacity (particularly on hub-to-hub routes and home country hub-to-beyond foreign hub routes). Further, DOT has sought detailed information from applicants on their plans to increase capacity beyond what they would do in the counterfactual scenario in which DOT did not grant immunity. These officials said that DOT places particular emphasis on the quantity, likelihood, and viability of additional capacity when determining whether the application will produce substantial benefits that might not occur if applicants choose not to go forward with the agreement in the absence of a grant of immunity. DOT also considers filings from other parties that support or cast doubt on the applicants’ claims. For example, in 2005, DOT denied an application from six carriers seeking immunity for the SkyTeam Alliance. According to DOT officials, based on the case record and competitive circumstances at the time, DOT found that immunizing the proposed agreement would not provide sufficient public benefits. This finding comported with arguments from objecting parties that immunity was not required to produce benefits because there was a high likelihood that SkyTeam members would continue integrating their management and operations, in order to maintain and maximize the profitability of their existing relationships. As with the competitive analysis, DOT officials use their professional experience and expertise, as well as the case record of each application, to determine the likelihood of benefits, and the necessity of antitrust immunity for carriers to implement their proposed plan quickly. As a general practice, DOT does not attempt to replicate the benefits analyses that carriers may provide as part of their application, according to DOT officials. DOT officials explained that they use their knowledge of the industry to verify and validate the applicants’ benefit claims by qualitatively assessing the reasonableness of the market and broad economic assumptions underlying these claims. Based on this assessment, DOT may condition a grant of immunity on carriers’ first demonstrating a readiness and ability to implement the agreement. For example, in one case, the department did not initially grant antitrust immunity to the partners of a cooperative agreement because DOT determined that incompatibilities in the carriers’ information technology systems would prevent the partnership from yielding consumer benefits. Consequently, DOT officials said they advised the applicants to reconcile these shortcomings, or risk DOT finding the benefits of the proposed cooperative agreement implausible and, in turn, antitrust immunity unwarranted. Similarly, DOT has also conditioned several grants of antitrust immunity on the carriers’ expeditious implementation of the proposed cooperative agreement. Based on our analysis of DOT’s antitrust immunity proceedings, DOT has ultimately approved most of the requests for antitrust immunity that it has received, with some stipulating competitive remedies. Specifically, DOT has adjudicated 38 applications involving a U.S. and foreign carrier(s) since 1993, granting antitrust immunity 31 times, according to our analysis. Twenty-three of these grants remain in effect across 13 different carrier agreements. See appendix I for information on adjudicated immunity proceedings involving U.S. and foreign carriers. In two proceedings, DOT denied antitrust immunity based on findings from its public benefits analysis. Specifically, in one proceeding, DOT found that the overall level of public benefit was small because the proposed alliance focused on a single route and was not likely to create new routes or a significant number of new travel options for consumers. In the other proceeding, DOT noted that code sharing or other less-involved forms of collaboration could produce similar benefits, namely new and expanded service additions, suggested by the carriers. Consequently, DOT denied these applicants’ requests for antitrust immunity. Most stakeholders, in particular representatives from major carriers, we interviewed considered DOT’s final decisions and application review process to be largely transparent, but lengthy. DOT officials and some stakeholders we interviewed underscored that there are opportunities for interested parties, including competing airlines, to examine all submitted application materials—including confidential and proprietary information— and to provide substantive comments. DOT officials emphasized the importance of a complete record of information on the official docket as the basis for their decisions. DOT is required to make a final decision within 6 months from the date of an application but may issue a notice to suspend the procedural schedule in order to establish a complete record. Some carriers we interviewed said that DOT’s review and efforts to establish a complete record can cause a proceeding to be lengthy. For example, the most recently completed proceeding to date was over 18 months from when the application was filed until DOT issued a decision. This proceeding involved a number of filings that pointed to the likely harm to present and future competition from independent carriers in specific markets due to the potential for exclusionary behavior by the applicant carriers. Our documentation review affirms DOT’s and stakeholders’ view that available proceedings’ records include DOT’s analyses and findings. With the exception of confidential or proprietary information, all applications, notices, DOT orders, and other documentation related to an application can typically be found on the public docket. Our review of all the proceedings found that each DOT order providing a grant of immunity included discussion of DOT’s findings from its competitive and public benefits analyses, as well as discussion of why and how DOT arrived at stipulated remedies, if any. For example, as previously discussed, in the 2010 Oneworld order, DOT described the potential competitive harm at specific airports that the department identified in its analysis and rationale for requiring a divestiture of slots at those airports as a remedy for those potential harms. Though we found consensus among stakeholders that DOT’s process is transparent, there is disagreement among the stakeholders we interviewed about the potential benefits of immunity for consumers. Specifically, two third-party stakeholders and representatives of all non- immunized carriers we interviewed suggested that carriers do not need antitrust immunity to cooperate in ways that benefit consumers, such as through codeshare and interlining agreements. Some of these stakeholders noted that immunized carriers, through their cooperative agreement, could have access to better market data than non-immunized carriers or leverage their increased network size to gain unfair competitive advantages. Representatives for all three U.S. carriers with approved immunized agreements indicated these immunities were, and continue to be, essential to their ability to provide high-quality service to their customers. Moreover, these carriers believed that changes to DOT’s process should be focused on expediting the process so that public benefits achievable only through grants of antitrust immunity could be realized more quickly. DOT officials indicated they are aware of the controversial nature of grants of antitrust immunity and noted that it takes time for DOT to gather and assess the evidence in each proceeding. These officials indicated that the department considers different views when considering applications, monitors academic and other literature on the topic, and applies these ideas as the officials deem appropriate in their decision-making. DOT conducts a number of activities to oversee and monitor individual immunized cooperative agreements and to understand how broad trends in international air competition affect immunized agreements. For example, DOT officials responsible for the program explained that they analyze a variety of international and domestic airline-competition issues including, but not limited to, airline alliances and, accordingly, keep track of market developments, such as new carriers entering markets and changes in market shares of established carriers. By monitoring these broad trends, DOT is able to better understand industry dynamics, according to officials we interviewed. For specific grants of immunity, DOT officials emphasized that they may tailor some monitoring activities to the nature of the agreement and the specific requirements set forth in DOT’s grant of immunity. For example, DOT officials explained they track compliance with the required slot divestitures in one grant of immunity through a designated trustee or, for immunities that require carriers to maintain capacity on certain routes, by DOT officials’ own review of existing flight schedule databases. DOT officials noted that the department’s specific monitoring activities are undertaken to track the implementation of cooperative agreements and to assure carriers comply with the terms of immunity grants (see table 3). In recent years, DOT’s monitoring activities have focused on the status of cooperation under immunized agreements and whether that cooperation is leading to merger-like efficiencies. To that end, according to DOT officials, all seven grants of immunity approved since 2009 require carriers to submit confidential annual reports to DOT. These reports cover topics including the public benefits of the agreement and commercial developments between the partners. Each year, DOT develops a template for these reports that delineates what information must be included on operational aspects of the implemented agreement (e.g., integration of routes and service planning) and the extent that partnered carriers have aligned their customer service policies to provide customers with a consistent experience across partners, among other topics. These reports, and DOT’s associated reviews, are the core of DOT’s current monitoring efforts, according to DOT officials and, according to representatives of the carriers submitting these reports, provide DOT with extensive information on the implementation status of the immunized agreement. Our examination of the most recent of these reports, for 2017, affirms they include considerable information on the implementation of the agreement and status of the alliance. DOT’s monitoring activities also include some review of empirical information on the effects of individual immunities. Specifically, as discussed above, carriers seeking immunity routinely identify anticipated consumer benefits, such as lower fares and greater frequency of service, and DOT has predicated grants of immunity on these expected benefits. According to DOT officials, they monitor available schedule, pricing, and other data to check whether observed outcomes are consistent with expectations, and if not, whether other factors, such as fuel prices or other market changes, provide a qualitative explanation of observed trends. The 2017 annual reports that carriers submitted to DOT also included information on these trends, based on our review of these documents. Likewise, according to DOT officials, DOT takes steps to track the status of remedies—such as whether airport slots were, in fact, divested and market entry occurred as was expected. DOT’s specific steps to do so vary depending on the nature of the remedy and the availability of relevant information. Furthermore, DOT officials commented that third parties, such as other air carriers, have incentives to alert DOT to concerns about violations of exclusivity prohibitions that help DOT verify and enforce this condition of some immunity grants. DOT’s monitoring activities do not typically include independent econometric analysis to examine the effects of the immunities it has granted, according to DOT officials, but the department tracks economic literature on these effects and has recently commissioned its own study. As we have noted, DOT looks for substantial integration among carriers requesting immunity as an indication that pricing efficiencies will be attained and benefit consumers. For a connecting airline route where one carrier serves one leg of the route and a different carrier serves the other leg, it is broadly recognized by economists that joint price-setting by the carriers will generally result in a lower airline fare. However, in cases where two airlines are competing on the same route—as could be the case on nonstop routes between the U.S. and another country—carrier coordination could reduce the extent of effective competition and lead to higher fares. Additionally, lesser forms of coordination that do not rely on a grant of immunity may also address the “double pricing” inefficiencies on connecting routes. Academic literature that uses statistical modeling to examine the effect of antitrust immunity has come to differing conclusions on the effect of immunity on fares for airline passengers. For example, one study found that connecting routes served by carriers with immunized cooperative agreements had lower prices compared to connecting routes served by carriers with other forms of cooperating agreements that were not immunized, and this study also found that immunities did not lead to higher fares on nonstop routes. However, another study found that antitrust immunity reduced competition and, thus, caused higher prices on nonstop routes; this study also found that pricing efficiencies on connecting routes did not require antitrust immunity. Recognizing the varying findings of the available literature, DOT commissioned a specialized study in 2016 to improve its understanding of the effect of immunities and airline joint ventures on consumer prices. According to DOT, the department provided guidance, data, and other input to support this work, but did not assist in the analysis or guide its conclusions. The report was provided to DOT in the summer of 2018, and according to DOT officials, as of December 2018, the department was reviewing the study’s findings, and considering how, if at all, it might apply the methodologies used in the study to DOT’s own monitoring activities in the future. DOT officials also indicated they have not made any final determinations about what, if any, adjustments may be appropriate to existing grants of immunity or to DOT’s process for considering future immunity applications based on the study’s findings. Based on our review of antitrust immunity proceedings, DOT has rarely amended or modified, and has seldom revoked immunity of an approved cooperative agreement. However, DOT has changed some terms of approval when carriers have sought immunity for updated agreements that, for example, added other carriers to an existing agreement. DOT officials explained that initiating a change in an existing immunity grant is a time-consuming and technically difficult process because it would involve the same administrative steps as in the initial approval process. Further, DOT officials indicated that carriers have been generally responsive to the requirements laid out in DOT’s grants of immunity, and as a result, DOT has not needed to pursue many corrective actions. Moreover, these officials explained that they are well aware of carriers’ plans to pursue new immunized agreements, and as a result, DOT officials are able to await those proceedings to make incremental changes to the terms of DOT’s original approval. For example, DOT’s early grants of immunity did not include annual-reporting requirements, but as carriers updated their agreements and sought new immunities, DOT used these new proceedings as an opportunity to add this requirement. There is generally little, if any, information from DOT available to external stakeholders and the public regarding DOT’s monitoring efforts and its findings on the effects of granted antitrust immunities. DOT publishes one summary document on its website that lists every active and inactive immunized cooperative agreement. This document, which according to officials, DOT updates periodically with each new grant of immunity, includes web links to the dockets of formal proceedings associated with each immunity application and grant. This document provides a single portal for anyone to access materials related to antitrust immunities that are spread across multiple dockets. Each docket remains open for public comment as long as DOT’s grant of immunity remains active. For example, in 2017, stakeholders submitted public docket comments critical of the market effects of a cooperative agreement awarded antitrust immunity 15 years earlier. In this case, DOT provided a formal, public response, as required, on the issues raised. DOT does not report information on its own voluntary monitoring activities in public dockets or elsewhere. For example, DOT does not post information on whether immunized carriers have submitted required annual reports or, as periodically required, resubmitted their cooperative agreements to DOT. Moreover, DOT does not release its assessments of these materials nor does DOT make any public statements on whether a grant of immunity yielded, in actuality, the types of carrier cooperation expected, whether DOT-imposed remedies were implemented and had the expected results, or whether the immunity generated the public benefits as expected when approved. As described previously, DOT has approved grants of immunity based on the expectation of various public benefits. These potential benefits include, for example, lower consumer prices for connecting flights, expanded route and schedule offerings, and increased market entry and competition. DOT provides no reports to the public or Congress related to whether these expectations were met. Internal controls help program managers achieve desired results and adapt to shifting environments, evolving demands, changing risks, and new priorities. As part of an internal control system, management should externally communicate quality information. Attributes of this principle call on federal program managers to communicate quality information externally so that external parties can help the government achieve its objectives and address related risks. Generally, according to this internal control standard, government reporting is intended for the executive branch’s decision makers and Congress as well as the general public. Management may select appropriate methods for external reporting. Accordingly, program managers should consider what methods are appropriate for such a broad audience, considering factors such as the nature of information and cost. In the context of grants of antitrust immunity, relevant parties include Congress, industry stakeholders, and the general public. Each of these groups may have distinct needs and abilities to access, understand, and act upon information about the effects of antitrust immunities in the marketplace. DOT officials cited several reasons for not reporting on their monitoring activities and related findings. DOT officials underscored that much of the information gathered in its voluntary monitoring efforts—annual reports, in particular—are proprietary and, therefore, not information DOT could publicly disclose. Representatives from immunized carriers we interviewed also stressed that public disclosure of the business plans and alliance status assessments provided to DOT would be damaging to their business if made public. DOT officials also expressed concern that commentary from the department about the effects of immunities could be construed as departmental promotion of a specific alliance, or “prejudgment” of an issue that could come before the department in a future proceeding. DOT officials also said competition authorities, such as the Department of Justice, do not typically address the results of a case (e.g., post-merger analyses) and are only involved with the process and guidelines associated with reviewing and adjudicating a case. While there are valid concerns about the publication of proprietary information and statutory prohibitions on doing so, there are available avenues for DOT to report on the findings of its monitoring activities and assessments of the consumer effects of antitrust immunities broadly. Further, many of the expected benefits of grants of immunity—such as changes in prices, schedules, and markets served—can be evaluated without relying on proprietary information. For example, the number of competitors serving city-pair markets and carriers’ market shares can be calculated—as DOT does during the approval process—using publicly available data. Prices changes under the immunity can also be evaluated using publicly available information. Likewise, an assessment of the market outcomes of competitive remedies—such as whether slots were divested and competitors provided new service as expected—does not require business-sensitive information about the internal workings of an immunized alliance, but rather data on the public actions of carriers in the marketplace. These data are publicly available through schedule data and information in DOT datasets. Government reporting can also protect proprietary information from improper disclosure, either by issuing restricted reports to Congress or through stating findings at a very general level. For example, the Federal Trade Commission has balanced the protection of proprietary information from public disclosure while also reporting on the commission’s findings of the effects of its commission- imposed competitive remedies. Specifically, the Federal Trade Commission published two merger remedies studies, eliminating the names of and financial information about the merging parties and the buyers of the divested assets in publicly available versions. The Federal Trade Commission made both of its studies public. The lack of information available on the observed effects of immunities in the marketplace, including the effects of DOT-stipulated remedies, can make it difficult for external stakeholders to assess what consumer benefits have, or have not, been realized. According to consumer and antitrust organizations we interviewed, the lack of available information left them speculating that DOT did not conduct any monitoring of granted immunities after approval. Likewise, representatives from two of the three non-immunized carriers we interviewed noted the contrast between the transparency of DOT’s approval process and the opacity of its monitoring process. Additionally, two stakeholders we interviewed opined that airline alliances have harmed consumers by, for example, creating restrictive rules that make certain types of travel more difficult than in the past, among other anti-consumer effects. Some stakeholders mentioned they had no basis to review or comment on whether DOT monitoring activities are sufficient. Another stakeholder mentioned that in the absence of any reports or other information from DOT, they did not know if alliances have delivered the consumer benefits initially expected. DOT officials stressed that because the process for consideration of immunity is public any outside party may petition the department for review of an existing immunized alliance and provide information on the docket—which remains open—if any party believes that an alliance is acting contrary to the public interest. However, two stakeholders we interviewed indicated that it was difficult to use the docket comments process to lodge observations or criticisms without, for example, disclosing their own competitively sensitive information and absent information on the implementation of immunized alliances. Further, the information available on dockets does not provide congressional policymakers with readily available information on the findings of DOT’s many ongoing monitoring activities. During the approval process, DOT publishes key aspects of its analytic findings in show-cause and final orders to the public docket. These documents provide insights into the basis for DOT’s decisions. DOT could periodically provide information on the effects of immunities, based on its monitoring activities, on the docket, or through other mechanisms, such as public reports or through confidential reports to Congress. This information could provide greater transparency and be useful in considering changes in DOT’s authority to grant antitrust immunity, an authority the Congress and others have considered at various points. With more information about DOT’s monitoring activities and findings, policymakers, stakeholders, and the public would have an improved understanding of the competitive effects of immunities. As U.S. and foreign air carriers have pursued more integrated forms of cooperation through international air alliances, DOT has extended American Airlines, Delta Air Lines and United Airlines antitrust immunities with their major foreign partners with the expectation that the immunities would yield public benefits. Cooperation between international air carriers can lead to certain benefits for consumers, and immunizing such cooperation from antitrust laws may yield additional benefits. DOT’s review of requests for immunity and oversight of immunized agreements are important to ensuring robust competition and, thus, consumer benefits in the marketplace. DOT’s ongoing monitoring pays significant attention to whether and how grants of immunity affect consumers. However, DOT generally has not reported on its monitoring activities and market outcomes of immunities. As the authority responsible for granting antitrust immunity, DOT holds a unique responsibility for reporting on these effects. Per internal control standards, the department’s responsibilities extend to communicating information to key stakeholders about the effect of immunities, based on DOT’s monitoring activities. DOT must balance providing information to policy makers and the public with statutory requirements that protect proprietary information from disclosure. DOT rightly keeps information on the status of cooperation under immunized agreements confidential. However, the market outcomes of immunities are not proprietary and DOT could publicly report on them. Such reports feasibly could include DOT’s views on whether the prospective benefits projected at the time of immunities’ approval have been realized and whether the department’s remedies have been implemented by immunized carriers and have had the effects expected by DOT. Like DOT’s current practice of periodically updating the summary document on immunities, DOT could issue such reports at a time interval it determines appropriate. Doing so would improve transparency and provide the public with improved information on the effects of antitrust immunities on consumers. The Director of DOT’s Office of Aviation Analysis should provide periodic external reporting, at a time interval DOT determines appropriate, to the public and policymakers, on the effects of antitrust immunity—based on the range of monitoring activities undertaken by DOT—including whether grants of immunity have achieved anticipated benefits and the status of remedies—such as airport slot divestitures—imposed as part of DOT’s approval. (Recommendation 1) We provided a draft of this report to DOT and the Department of Justice for review and comment. We received written comments from DOT, which are reproduced in appendix II and summarized below. In email, the Department of Justice told us they had no comments on the draft report. DOT and the Department of Justice also separately provided technical comments, which we incorporated as appropriate. In its written comments, DOT partially concurred with the recommendation. More specifically, DOT stated it will provide additional public information about the status of its monitoring activities and remedies, but it did not agree to report publicly on its findings about whether grants of immunity have achieved anticipated benefits. As discussed below, after evaluating the concerns that DOT raised, we continue to believe that periodically reporting on the effects of antitrust immunities would improve transparency and accountability. In its written response, DOT stated that if DOT were to release any additional materials than it already does, it could have a chilling effect, not just on competition by revealing proprietary information and insight on the real-time commercial strategies of a particular alliance, but also on the carriers’ willingness to share detailed and sensitive information with DOT that is necessary to conduct oversight. We disagree with DOT’s assertion that reporting on the effects of immunities would have a chilling effect on competition and the willingness of airlines to share information with DOT. Our report explains that DOT is prohibited from releasing proprietary information to the public and we expressly called on DOT to balance protecting this information while making appropriate information available to policy makers and the public. Moreover, contrary to DOT’s implication, we are not recommending DOT release the information DOT reviews during the annual reporting process, such as alliances’ revenue management and competitive strategies. Instead, the recommendation calls for DOT to report on the market effects of immunity relative to DOT’s anticipated benefits cited in DOT’s approvals of antitrust immunity and the status of remedies. As we noted in the draft report, these include trends in consumer fares, schedule offerings, and the like that DOT could report on without relying on proprietary information. DOT also stated that it must balance the importance of transparency with its statutory obligations to adjudicate each request for antitrust immunity fairly. Further, it stated that making such findings independently from the decision-making process in dockets with pending matters raises issues with prejudgment and ex parte communications, and is administratively unworkable. Doing so for cases that are not pending also raises issues of prejudice and prejudgment of “issues that are likely to be raised in future cases involving amendment of the alliance agreements (e.g., when membership changes).” We agree that DOT’s role as an impartial adjudicator is critical. We do not agree with DOT’s assertion that making public its assessment of the effects of immunities that have been granted would jeopardize its impartiality, because DOT could report this information and still consider each case based on its particular facts and circumstances. Further, the recommendation provides DOT with flexibility on how, when, and exactly what to report on that should allow DOT to avoid any prohibited ex parte communication. DOT described existing activities it believes maintain transparency for the public and ensure an ability for interested parties to seek review on the record of previously granted authorities. These activities include DOT’s public dissemination of passenger ticket and schedule data and the publication of DOT’s own orders that summarize departmental assessments of the state of competition as well as its immunity decisions. We note that our draft report described these activities in detail and recognized the overall transparency of DOT’s application review process. Nonetheless, we maintain that these activities do not provide regular or reliable information on the actual effects of antitrust immunities, based on DOT’s monitoring activities, and that DOT could do more to increase transparency through external reporting on these matters. For example, DOT’s provision of data to the public does not diminish the value of DOT providing its own independent reporting on whether expected consumer benefits, in fact, have materialized. Likewise, DOT’s published orders on specific immunities come at time intervals largely determined by the applicants and, naturally, when reviewing these applications, DOT’s competitive analysis focuses only on those markets relevant to the application at hand. More intentional reporting on the effects of immunity from DOT could address these shortcomings of existing activities. In other comments that were not included in DOT’s letter, DOT questioned the applicability of internal control standards to its role in monitoring grants of antitrust immunity. The principle of internal control we applied calls on management to externally communicate quality information that helps the agency achieve its objectives and manage risks. As we stated in the report, such communication can help program managers achieve desired results and adapt to shifting environments, which is relevant to DOT’s responsibility in this area. Ultimately, the recommendation, in full, aims to improve the transparency on the effects of antitrust immunity. Providing external stakeholders with additional information on DOT’s monitoring activities, as DOT agrees to do, should enhance confidence that DOT is undertaking oversight activities. Providing information on whether grants of immunity have achieved anticipated benefits, will further improve transparency and provide the public and Congress with useful information to inform policymaking in the future. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, the Attorney General, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In addition to the individual named above, Heather MacLeod (Assistant Director); John Stambaugh (Analyst in Charge); Friendly Vang-Johnson; Jim Geibel; Amy Suntoke; Delwen Jones; Amy Abramowitz; and David Hooper made key contributions to this report.", "summary": "Each year, millions of passengers travel internationally by plane. Many of these passengers are served by U.S. and foreign air carriers that have formed alliances to coordinate and integrate their networks. With antitrust immunity provided by DOT, airline alliances pursue a wide range of cooperative activities as outlined in joint venture agreements between the airlines. While this cooperation is meant to provide consumers with better services, it could also affect the extent of airline competition. GAO was asked to review consumer issues related to immunized international air alliances. This report (1) describes how DOT's review of antitrust immunity applications considers the potential effects on consumers and (2) evaluates how DOT monitors approved grants of antitrust immunity. GAO analyzed DOT's antitrust immunity proceedings, interviewed officials from DOT, the Department of Justice, as well as a nongeneralizable selection of 13 stakeholders, including consumer organizations and domestic air carriers with and without antitrust immunity. Potential effects on consumers are included in the analyses the Department of Transportation (DOT) conducts when reviewing international air carriers' requests for antitrust immunity. If granted, this immunity allows the airlines to engage in certain cooperative activities, such as coordinating prices and schedules, without risk of violating U.S. antitrust laws (see figure). DOT's analyses examine: The potential competitive effect of the proposed cooperative agreement in terms of relevant markets, on changes in the number of competitors and market shares, and on market entry. The potential for the close integration of carriers to create public benefits, such as lower consumer prices or expanded service offerings. Such analyses involve DOT staff's reviewing an array of data, documents, and reports filed in a public docket by carriers and interested parties and, ultimately, making a decision based on their assessment of the application. DOT has premised its decisions to grant immunity on the expectation that consumer benefits flow from high levels of integration of critical business functions between carriers. To date, DOT has granted antitrust immunity 31 times, with 23 grants currently in effect, which cover agreements made among carriers in each of the three major international air alliances. DOT has rejected three applications due to concerns about potential anticompetitive harm or insufficient public benefits for consumers. Stakeholders GAO interviewed generally agreed that DOT's decisions were transparent, but some disagreed on the potential benefits of immunity for consumers. DOT takes multiple steps to monitor alliances and understand the effects of immunity. Since 2009, DOT has required all transatlantic and transpacific partnerships to submit annual reports on the status of their immunized agreement. Additionally, DOT recently commissioned an empirical evaluation of immunities' effects and is currently reviewing the findings. However, DOT does not externally report information on the effects of granted immunities to Congress, industry stakeholders, and the public. As a result, these external entities are unable to determine what, if any, steps DOT is taking to ensure that grants of antitrust immunity remain in the public interest. Further, without additional transparency and information on DOT's findings on the effects of immunities, external entities do not know if immunized alliances have delivered the expected consumer benefits that DOT used as a basis to approve the carriers' request for antitrust immunity. GAO recommends that DOT externally report to policymakers and the public on the effects of antitrust immunity, based on DOT's monitoring activities. DOT agreed to provide public information on its monitoring, but not to report on the effects of antitrust immunity. GAO continues to believe its recommendation, in full, is valid as discussed further in the report.", "document_type": "gao"}
{"report": "After it is extracted, natural gas—a colorless, odorless fossil energy source—is stored in three types of underground geologic formations: salt caverns, depleted aquifers, and depleted oil and gas reservoirs. Two physical characteristics govern the suitability of each type of geologic formation for storage, including: (1) its capacity to hold natural gas for future use and (2) the rate at which natural gas can be withdrawn to meet demand. As of July 2019, about 80 percent of the approximately 400 natural gas storage sites in the United States are depleted natural gas or oil reservoirs because they are available in greater numbers than other types of formations, according to EIA. Underground salt caverns and depleted aquifers each account for about 10 percent of the sites. Natural gas storage sites are located in 31 states. California, Louisiana, Michigan, Pennsylvania, and Texas together contain natural gas storage sites that provide more than half of the natural gas storage capacity in the United States. Figure 1 illustrates the types of geologic formations used for natural gas storage and the locations of natural gas storage sites in the United States. The wells that inject natural gas into, or withdraw it from, the underground storage sites can extend thousands of feet underground. According to information from PHMSA, about 17,000 wells are used to inject and withdraw gas at approximately 400 natural gas storage sites, ranging from a few wells per site to more than a hundred wells at some larger sites. Wells are constructed with multiple layers of steel pipe, called casing, which are cemented in place. The layers of steel casing are intended to isolate the internal portion of the well from the outlying geological formations, which may include underground drinking water supplies. As a well is drilled deeper, progressively narrower casing is inserted further down the well and cemented in place. The wells at natural gas storage sites can be constructed to prevent leaks by installing multiple control points at each well, according to API. If a well is not constructed with such multiple points of control, it could be subject to a single point of failure, in which the failure of a single component, such as a casing or a safety valve, can lead to a large release of natural gas—a factor that contributed to the Aliso Canyon incident, according to PHMSA officials. From October 23, 2015, through February 11, 2016, the Aliso Canyon Underground Storage Facility in Los Angeles County, California, experienced a large and uncontrolled natural gas leak. The Aliso Canyon natural gas storage site is a depleted oil field that was converted into a natural gas storage reservoir in the 1970s and that is near the Porter Ranch community, a residential community of about 30,000 people. It provides natural gas to the Los Angeles region for residential heating and cooling, commercial and industrial uses, and as fuel for electric power plants. According to the Energy Information Administration, the Aliso Canyon site has the fourth largest capacity among the approximately 400 underground natural gas storage sites in the United States. The leak reportedly was caused by damage to a well casing approximately 500 feet underground. California state government officials identified the damage as being caused by the aging infrastructure of that well, which had been drilled in 1953, and a lack of redundant safety valves at the well that prevented the leak from being stopped. Across a 4-month period, the site operator made multiple attempts to stop the leak. About 8,000 families near the Aliso Canyon leak were temporarily relocated in November 2015 due to ongoing odors and symptoms including headaches or migraines; nausea, vomiting, stomach aches, or diarrhea; nosebleeds; respiratory or breathing problems; chest tightness, coughing, or palpitations; and light-headedness and dizziness. Various agencies, including public health and regulatory agencies from state and local governments such as the Los Angeles County Public Health Department and California’s Office of Environmental Health Hazard Assessment, responded to the leak. Additionally, several studies about the leak have been conducted or are planned. CCST, a nonpartisan, nonprofit organization established in response to a California state legislative resolution, published an independent review of the viability of underground natural gas in California, including an analysis of the health effects from stored natural gas releases. An interagency task force established pursuant to federal law, led by DOE and PHMSA, studied the Aliso Canyon incident and in 2016 provided a report to relevant congressional committees with recommendations to enhance safety. According to the 2016 interagency task force report, natural gas stored in geologic formations is under high pressure, which can force the gas through underground fissures or unplugged oil and gas wells and allow the gas to find its way to the surface. Leaks can also occur if the wells lose integrity because of cracking of the cement used to seal them, among other factors. Older wells used for natural gas storage were often drilled for other reasons, such as oil and gas production, and are more likely to have age-related degradation. As part of its work, the interagency task force chartered a Public Health and Environment Subgroup, led by EPA, to summarize the actions taken by local, state, and federal agencies to monitor and mitigate impacts to public health and the environment. The subgroup was to also recommend actions to prepare local, state, and federal agencies if a release from a natural gas storage facility should occur in the future. When the Aliso Canyon leak occurred in 2015, federal safety regulations applied to conventional surface pipelines and above-ground equipment at all natural gas storage sites. Only state safety regulations applied to underground natural gas storage sites at that time. The PIPES Act of 2016 significantly changed the regulation of natural gas storage. It requires, among other things, that DOT establish minimum safety standards for all natural gas storage sites. Within DOT, PHMSA's mission is to protect people and the environment by advancing the safe transportation of energy and other hazardous materials, and because natural gas storage is a part of this mission, PHMSA is responsible for natural gas storage safety. In response to the act’s requirement, PHMSA issued an interim final rule in December 2016 that took effect in January 2017. The rule included minimum safety standards based largely on recommended practices from API and generally required compliance by natural gas storage sites by January 2018. PHMSA provided for a public comment period, and after reviewing the public comments received on the interim final rule, PHMSA may modify aspects of the interim final rule by issuing a final rule. In August 2019, PHMSA officials told us they planned to issue a final rule in October 2019. PHMSA's interim final rule contains four different reporting requirements for operators of all natural gas storage sites, including an annual report with gas storage volumes, gas storage pressures, well depths, gas injection and withdrawal rates, and maintenance information that is conducted to ensure the safety of a facility. The interim final rule also requires operators to develop emergency response plans, but the required elements for such plans vary depending on the type of natural gas storage site. While PHMSA has authority for oversight of underground natural gas storage facilities, the PIPES Act also authorizes states to participate in such oversight by annually obtaining certification from or entering into an agreement with PHMSA (which we refer to as partnering with PHMSA). Authorized states are responsible for inspecting intrastate underground natural gas storage facilities on sites fully within their borders. According to PHMSA officials, 25 of the 31 states where underground natural gas storage sites are located have such intrastate sites, and PHMSA expected to partner with these 25 states by granting them oversight authorization, according to PHMSA officials. In addition, the PIPES Act requires PHMSA to set and charge user fees for operators of underground natural gas storage sites. The act restricts the use of these fees to activities related to natural gas storage site safety. The act also prohibits PHMSA from collecting fees unless the expenditure of these fees is provided in advance in an appropriations act; as a result, PHMSA can only collect fees up to the amount provided in advance in an appropriations act. Human health can be affected by breathing hazardous chemicals in the air; drinking water contaminated by such chemicals; or making skin contact with contaminated soil, dust, or water. Chemicals that can affect human health include several types of hazardous materials that pose a risk to human health and safety. Environmental effects of chemicals can include greenhouse gas emissions and groundwater contamination. Several federal agencies have a role in assessing the public health and environmental effects from exposure to hazardous chemicals, although these efforts may not be specifically related to underground natural gas storage as described in this report. For example, the Toxic Substances Control Act authorizes EPA to review the environmental and health effects of certain chemicals and regulate those that pose unreasonable risks to human health or the environment. According to EPA's July 2018 Report on the Environment, relationships between environmental exposures and health outcomes can only be established through well- designed epidemiological, toxicological, and clinical studies. Developing evidence that environmental contaminants cause or contribute to the incidence of adverse health effects can be challenging, particularly for effects that occur in a relatively small proportion of the population or effects with multiple causes. For example, there may be factors related to both the exposure and the health effect—confounding factors—that can make it difficult to detect a relationship between exposure to environmental contaminants and disease. In its 2018 report, EPA stated that it uses the results of scientific research to help identify linkages between exposure to environmental contaminants and diseases, conditions, or other health outcomes. These linkages, in turn, identify environmental contaminants and health outcomes of potential agency interest. Research has established a relationship between exposure and disease for some environmental contaminants, including radon and lung cancer, arsenic and cancer in several organs, and lead and nervous system disorders. OSHA established the Air Contaminants Standard to limit employees’ occupational exposure to more than 400 chemicals. It also established the Hazard Communication Standard, which requires employers to provide information to their employees about the hazardous chemicals to which they are exposed by means of Safety Data Sheets, among other things. Other federal agencies have responsibilities related to the human health effects of chemicals, including ATSDR and NIOSH. ATSDR has authority to, among other things, perform health assessments for releases or facilities where information was provided that individuals were exposed to a hazardous substance for which the probable source of such exposure is a release. NIOSH researches the safe use of chemicals in the workplace and provides information on how to measure chemicals in the work environment, among other things, for understanding and managing chemicals safely at work. In November 2017, we reported on PHMSA’s natural gas storage program. At the time of our 2017 review, PHMSA was still establishing its program, and we reviewed its planning efforts for developing the program. We found that although PHMSA had established a strategic goal for its natural gas storage program and set a performance goal for training inspectors, it had not yet followed other leading practices for strategic planning. PHMSA officials told us that the program would be guided by one of PHMSA’s existing strategic goals: to promote continuous improvement in safety performance. We found that PHMSA had not defined the level of performance to be achieved and did not have performance goals that addressed other core program activities, such as conducting inspections. We recommended that PHMSA define levels of performance, address core program activities, and use baseline data to develop performance goals for its natural gas storage program. At that time, we also found that PHMSA had not yet used initial baseline data it gathered early in the program to inform the development of its performance goal. We recommended that PHMSA use other data and information about budgetary resources as they become available to inform and refine its performance goals. PHMSA agreed with these recommendations and in May 2018 established a performance goal for inspections of natural gas storage sites. PHMSA officials told us in July 2019 that they were continuing to inform and refine agency performance goals based on budgetary information. After our report in November 2017, PHMSA began inspecting natural gas storage sites but has not fully assessed resource needs for its changing workload or established a performance goal that measures PHMSA’s progress toward its relevant strategic goal to improve safety. First, because PHMSA has not used an analysis of its workforce needs to inform its budget requests, the agency may not have assurance that it has enough resources to meet its performance goal of inspecting all of the approximately 400 natural gas storage sites within 5 years (from early 2018 through early 2023). Second, although PHMSA has established a performance goal that focused on the number of inspections completed, the goal does not reflect the agency’s contributions toward its strategic goal to promote continuous improvement in safety. In November 2017, we reported that PHMSA had established a strategic goal for its natural gas storage program but had not yet set performance goals that define the level of performance officials hope to achieve or that address all core program activities, such as conducting effective inspections. PHMSA's inspections of natural gas storage sites are designed to determine the extent to which these sites meet PHMSA’s 2016 minimum safety standards for natural gas storage sites, according to PHMSA officials and documents. In our November 2017 report, we stated that our prior work had identified several leading practices for strategic planning that PHMSA had not yet followed, such as setting goals that define a certain level of performance and address all core program activities. We recommended that PHMSA develop such goals, and the agency concurred. In 2018, PHMSA officials told us that the agency had established a performance goal to inspect all of the approximately 400 natural gas storage sites over 5 years (from early 2018 through early 2023), with the expectation that state partners would help PHMSA inspect the sites. The officials also told us the agency has started inspecting sites to meet that goal. Currently, 10 states have agreed to partner with PHMSA to help inspect natural gas storage sites, according to agency officials. Natural Gas Storage Site Inspections Conducted by the Pipeline and Hazardous Materials Safety Administration (PHMSA) At a PHMSA inspection of a natural gas storage site in rural Iowa, we observed PHMSA inspectors conducting visual inspections of natural gas storage wells in the field to ensure that the site operator's wells matched the operator's documentation and that the wells were operating within safe limits. During the inspection, PHMSA's inspectors also conducted a review of the storage site operator's safety procedures, such as the operator’s schedule for inspecting its wells for potential leaks or pressure changes, its emergency contact protocols, and its procedures for ensuring the integrity of wells. As part of the review, PHMSA inspectors reviewed the site operator’s documentation to evaluate the operator’s efforts to implement the agency’s 2016 minimum safety standards for natural gas storage sites. To meet its performance goal, PHMSA set targets for each of the 5 years (see app. II for details about PHMSA's annual targets for this performance goal). For example, PHMSA set a target that its inspectors and state partners would inspect a total of 41 sites in 2018. According to PHMSA officials, the agency completed 35 inspections, and its state partners inspected an additional 30 sites, for a total of 65 inspections in 2018. In future years, according to PHMSA planning documents, PHMSA’s annual site inspection targets will almost double from 41 total site inspections in 2018 to 80 total site inspections in 2019. However, PHMSA's inspection workload for its natural gas storage program has increased since November 2017, which may affect its ability to meet its inspection performance goal. We reported in November 2017 that PHMSA had developed a preliminary estimate of the workforce it would need to inspect half of the approximately 400 natural gas storage sites. That estimate was based on the agency’s experience from its pipeline safety program. Specifically, in 2017, agency officials said that they expected 25 state governments would partner with PHMSA to inspect about 200 of the sites and that six agency employees would inspect the remaining approximately 200 sites. Specifically, in 2017 PHMSA estimated the inspections would require about 203 work weeks of inspectors’ time. However, in October 2018, PHMSA officials told us that their inspectors would need more time than previously estimated to complete each natural gas storage site inspection, due to requirements for operators in the 2016 minimum safety standards. Furthermore, in its 2017 estimate, PHMSA assumed that all 25 state governments eligible to partner with PHMSA on inspections would agree to do so. However, as of June 2019, only 10 of the 25 eligible states had agreed to partner with PHMSA, according to agency officials. PHMSA officials told us that more states may decide to participate in the future. However, there are a variety of reasons why states may be reluctant to partner with PHMSA. For example, officials from two states told us that PHMSA had not offered enough funding to cover the cost of partnering with the agency. Officials from two states told us that partnering with PHMSA required some lead time to obtain funds through their states' legislative processes for such inspections. In addition, PHMSA officials told us that some states are waiting until the interim final rule is issued as a final rule before determining whether to partner. As a result, according to PHMSA data, unless additional states partner with the agency, PHMSA will need to increase the number of sites it inspects from about 200 to 322 in order for the agency to meet its performance goal of inspecting all of the approximately 400 sites by 2023. This would increase PHMSA’s inspection workload by about 60 percent, as shown in figure 2. Because of the increase in its inspection workload over its preliminary estimate, PHMSA does not have assurance that it has enough resources to meet its inspection goal. Specifically, PHMSA has requested and received the same budget authority for its natural gas storage safety activities—$8 million—for each fiscal year from 2017 through 2019. Of the $8 million, PHMSA requested $2 million for federal employees to inspect about 200 of about 400 natural gas storage sites. PHMSA requested the remaining $6 million for grants to authorized states to conduct inspections of the remaining sites. However, of the 25 states PHMSA expected to request such authority, only 10 did so and are partnering with PHMSA to conduct inspections, according to PHMSA officials. This means that the number of sites that states could inspect is about 90 rather than about 200, as PHMSA had initially estimated. In comparison, PHMSA's workload for its natural gas storage inspection program is more than three times higher than the workload for PHMSA’s pipeline inspection program. We also recommended in November 2017 that PHMSA use other data and information about budgetary resources to inform and revise its performance goals. PHMSA concurred with our recommendation. However, officials told us that as of July 2019, the agency had not yet fully addressed this recommendation to use workforce data to inform and revise its goals. In December 2018, PHMSA issued a strategic workforce plan that indicates it represents a thorough analysis of the agency’s current workforce composition as of 2018 and the collective viewpoints of employees and senior leadership regarding the future. PHMSA stated in this plan that workforce planning will allow the agency to respond to emerging challenges and responsibilities and improve overall mission effectiveness and efficiency. Specifically, the plan states that PHMSA leadership recognizes that while the agency has implemented some foundational elements of workforce management and the overall workforce is staffed with skilled professionals, the agency’s workforce planning has tended to be more reactive than proactive. The plan cites as evidence underdeveloped succession plans, inconsistent hiring results, increased turnover, and limited workforce analysis and forecasting. To address these gaps, the plan identifies the following three high-level strategies to supplement and expand agency capabilities: expand and enhance PHMSA’s recruitment and hiring plans, conduct operational workforce planning and workload analysis by program office, and implement succession planning and develop leadership and staff. PHMSA officials said that the agency has been assessing its workforce, but they told us this assessment will not guide the agency’s budget requests for its natural gas storage program. PHMSA officials told us they did not plan to change the workforce levels reflected in the agency’s budget requests until 2022 or 2023. This is because although PHMSA has been collecting and assessing workforce data since March 2018, the agency does not expect to have the workforce data it needs to further inform workforce analysis until 2022 or 2023, according to PHMSA officials. The officials indicated that the additional data they have begun gathering may include variables such as the number of additional states that may partner with PHMSA in the future; resources used, by region; and the capacity of inspection teams of different sizes. In technical comments PHMSA provided on a draft of this report, PHMSA officials stated that the agency recently concluded a workforce assessment of its pipeline inspection program—including its natural gas storage program— covering the 5 years from 2020 through 2024. PHMSA’s workforce assessment indicated that the state of Texas is likely to partner with PHMSA beginning in 2020, which would reduce the number of natural gas storage sites PHMSA would need to inspect. Based on our preliminary review of the information PHMSA officials provided, however, PHMSA’s assessment does not address the reasons its inspectors’ workload increased by about 60 percent, such as the factors affecting states’ participation in inspections. Moreover, PHMSA officials did not indicate whether PHMSA would use this workforce information to guide its workforce planning or budget requests. We have reported that strategic workforce planning is an essential tool to help agencies align their workforces with their current and emerging missions and develop long-term strategies for acquiring, developing, and retaining staff. Furthermore, we have reported that existing strategic workforce planning tools and models and our own work suggest that there are certain principles that such a process should address. These principles include developing strategies tailored to address gaps in number, deployment, and alignment of human capital to enable and sustain the contributions of all critical skills and competencies. We also have reported that workforce planning should include (1) identification of the knowledge, skills, and abilities and other characteristics (i.e., competencies) needed by the future workforce; the competencies of the current workforce; and gaps between the two; (2) development of a workforce action plan designed to address these gaps; and (3) monitoring and evaluation of the workforce planning actions taken. Furthermore, we have found in our prior work that completing and regularly updating staffing models in a timely manner can help support agencies’ activities and decision-making. By analyzing the factors affecting states’ participation in inspections and analyzing the agency’s workforce needs on an ongoing basis and using this information to guide its budget requests, PHMSA would have more reasonable assurance that it has the necessary staff to meet its inspection goal. PHMSA has established a strategic goal for its natural gas storage program to promote continuous safety performance but as of April 2019 had not established performance goals that reflect the agency’s contributions to protecting human health and the environment. According to PHMSA officials, PHMSA’s natural gas storage program is guided by the agency’s strategic goal to promote continuous improvement in safety performance. PHMSA officials acknowledged that the agency’s inspection performance goal provides information about activities or outputs— specifically, the number of inspections. However, this goal does not provide information on the outcomes or results of PHMSA's contributions toward its strategic goal of improving safety at natural gas storage sites, consistent with leading practices under GPRA. An example of an outcome-oriented performance goal could be to measure reductions in the volume of gas released from natural gas storage wells, which could indicate that operators of natural gas storage sites are reducing safety risks through improved maintenance. Based on our previous work, measuring performance outcomes is an important management tool for agencies, and leading practices indicate that results-oriented performance goals focus on expected results to show progress toward, or contributions to, intended results. By establishing performance goals that demonstrate improvements to safety outcomes, PHMSA would have better assurance that it can show its progress toward meeting the agency’s strategic goal of continuously improving safety performance. In addition to the performance goal PHMSA established, agency officials told us that DOT applied an outcome-oriented, department-wide performance goal to its natural gas storage program. Based on our review of DOT’s 2018-19 Annual Performance Plan—2017 Annual Performance Report, PHMSA is responsible for meeting the department-wide performance goal of reducing incidents involving death or major injury resulting from the transport of hazardous materials by all modes, including pipelines. While PHMSA officials told us this was an outcome-oriented goal, we believe it would not provide a meaningful measure of safety improvements at natural gas storage sites because, according to PHMSA data, there have been zero incidents involving death or major injuries at natural gas storage sites since 2017, when PHMSA started tracking incidents. While no deaths or major injuries have been reported at natural gas storage sites since 2017, PHMSA reported seven incidents—four in fiscal year 2017 and three in fiscal year 2018—that did not result in death or major injury. These seven incidents resulted in natural gas releases of 3 million cubic feet or more or caused estimated property damage of $50,000 or more. By tracking reductions to these incidents, PHMSA may have additional opportunities to measure outcomes in safety improvements. Several federal agencies—including EPA, ATSDR, OSHA and NIOSH— have documented potential health effects of chemicals that may be found in stored natural gas. These chemicals—some at trace amounts—are known to cause health effects at specific levels of exposure. Stored natural gas primarily consists of methane, and during large releases at natural gas storage sites, downwind methane concentrations can be higher than flammability or explosion limits, creating health and safety concerns, according to CCST. In addition, other chemicals occur naturally in natural gas or are residues from the storage site’s previous use. For example, hydrogen sulfide, a flammable, colorless gas that smells like rotten eggs, can occur in depleted oil and gas reservoirs. Figure 3 shows a building containing a well at a natural gas storage site with a notice that warns of hydrogen sulfide, which may collect in confined spaces in amounts that are acutely toxic. Hydrogen sulfide can cause a range of human health effects, from eye irritation to serious lung injury, according to ATSDR. In addition, some chemicals may be added to natural gas, such as sulfur odorants that are added to give natural gas a distinct smell in case of leaks. The combination of such chemicals varies from one storage site to another based on the attributes of that site, such as its geologic type and the extent to which sulfur odorants are added to the natural gas before storage. Many of these chemicals have been linked to adverse health effects. However, research is limited on the health effects of exposure to stored natural gas in general and on the effects in particular from exposure to chemicals that may occur in natural gas storage leaks or be present at the storage sites. Reports linking health effects are available on specific chemicals but not in the context of natural gas storage, based on our literature review. Scientific studies are important for establishing the association between chemicals in stored natural gas and symptoms community members may experience during leaks to determine health effects. EPA, through its Integrated Risk Information System (IRIS) Program, identifies and characterizes the health hazards of chemicals found in the environment and has produced assessments on several chemicals that may be present in natural gas. EPA established the IRIS Program in 1985 to help develop consensus opinions within the agency about the health effects from lifetime exposure to chemicals. The IRIS database of chemical assessments contains EPA’s scientific positions on the potential human health effects that may result from exposure to various chemicals in the environment. As of November 2018, the database included information on 510 chemicals. To conduct an assessment of a chemical, the agency follows a multi-step process that includes identifying credible health hazards associated with exposures to a chemical and characterizing the quantitative relationship between chemical exposure and each credible health hazard. The program derives toxicity values through this quantitative relationship. EPA has completed assessments on several chemicals that may be in stored natural gas, including hydrogen sulfide, benzene, toluene, ethylbenzene, and xylene. In its IRIS assessment on benzene, EPA found that, as is the case with many other organic solvents, benzene has been shown to produce neurotoxic effects in test animals and humans after short-term exposures to relatively high concentrations. ATSDR develops toxicological profiles—summaries of its evaluations concerning whether, and at what levels of exposure, adverse health effects occur and levels at which no adverse effects occur—for several chemicals that may be present in natural gas, including hydrogen sulfide, benzene, toluene, ethylbenzene, and xylene. For example, ATSDR has found that inhaling benzene can cause drowsiness, dizziness, and unconsciousness and that long-term benzene exposure affects the bone marrow and can cause anemia and leukemia. Also, ATSDR found that toluene may affect the nervous system and at low to moderate levels can cause tiredness, confusion, weakness, memory loss, nausea, and loss of appetite. However, these symptoms usually disappear when the exposure stops. NIOSH researches the safe use of chemicals in the workplace and provides information on how to measure chemicals in the work environment, engineering controls and personal protective equipment, risk assessments, and communication tools for understanding and safely managing chemicals at work. NIOSH publishes information on chemical hazards in the workplace to inform workers, employers, and occupational health professionals. For example, NIOSH reports on occupational exposure limits for ethylbenzene. NIOSH’s Pocket Guide to Chemical Hazards provides key facts on the health effects from exposures to chemicals and recommends occupational exposure limits to chemicals that can affect human health. In addition, NIOSH helped initiate the International Chemical Safety Cards, a joint international agency effort. The cards, which provide essential safety and health information in a clear and concise way, are drafted and peer-reviewed by an international group of scientists from institutions concerned with occupational safety and health. The cards provide information about some chemicals that can occur in natural gas storage sites, including hydrogen sulfide, benzene, toluene, ethylbenzene, and xylene. OSHA collects information on chemicals and occupational health effects for workers and compiles that information into a database. OSHA accumulates information from several government agencies, including EPA, ATSDR, and NIOSH. This information includes chemical identification and physical properties, occupational exposure limits, and sampling information. OSHA’s Occupational Chemical Database provides information on chemicals, including those that can be present in stored natural gas, such as hydrogen sulfide, benzene, toluene, ethylbenzene, and xylene. In addition, among other general information, OSHA regulations require employers to maintain and make available to employees Safety Data Sheets in the workplace for each hazardous chemical they use. Releases at natural gas storage sites are known to emit greenhouse gases—mainly carbon dioxide and methane—into the atmosphere, according to EPA and CCST reports. In addition, we identified two natural gas storage site releases from 2000 through 2018 that potentially impacted groundwater, but information about such releases is limited. Releases at natural gas storage sites emit greenhouse gases into the atmosphere, according to data from EPA’s program on greenhouse gas emissions. These can be major releases, such as the Aliso Canyon leak, or other emissions, such as leaking pipes and valves. According to the 2019 EPA annual report Inventory of U.S. Greenhouse Gas Emissions and Sinks, the main greenhouse gases released from natural gas storage sites are methane, the largest component of natural gas, and carbon dioxide, the main greenhouse gas produced by natural gas combustion. Of the two, methane makes a greater pound-for-pound contribution to climate change—the comparative impact of methane is more than 28 to 36 times greater than carbon dioxide over a 100-year period, according to EPA officials who cited the Intergovernmental Panel on Climate Change. As a result, leaks such as the Aliso Canyon incident contribute to climate change, according to EPA. For example, the Aliso Canyon leak resulted in the single largest release of methane in U.S. history, with a release of 78,000 metric tons of methane in 2015 and an additional 22,000 metric tons in the first 2 months of 2016. The Aliso Canyon leak equaled the greenhouse gas emissions from approximately 529,000 passenger vehicles driven for 1 year, according to EPA data. In most years since 1995, an annual average of 15,000 metric tons of methane were released from natural gas storage, according to EPA data on greenhouse gases. In 2015, however, due to the Aliso Canyon leak, greenhouse gas emissions from all natural gas storage wells increased to more than 92,000 metric tons of methane—about 6 times greater than the release for an average year—according to EPA estimates. Figure 4 shows EPA‘s estimates of annual methane emissions from natural gas storage sites from 1995 through 2016, including the estimated emissions from the Aliso Canyon leak in 2015 and 2016. Chronic releases during routine operations at natural gas storage sites, such as small leaks from valves or from equipment exhaust, also emit greenhouse gases into the atmosphere and may persist for long periods of time. These chronic releases tend to be slow leaks from natural gas wells, such as releases from seals and valves. Slow leaks can persist for long periods because, unlike major leaks, they are less likely to be detected, according to a CCST report. Moreover, slow leaks, if identified, may not be prioritized due to a perception that they present few implications for worker safety and public health, according to CCST’s report. However, the CCST report also stated that chronic releases may routinely occur, although the amount of the release is difficult to measure since it may not be known when the release started, and these chronic releases may lead to a significant release of greenhouse gas. In 2016, California conducted an assessment of all its natural gas storage wells across its 11 natural gas storage sites and found 229 chronic leaks. Methane releases from these slow, chronic leaks generally represent a small share of the statewide reported methane releases in California. However, over a 10-year period, the cumulative impact of these releases from routine operations in California can equal the amount of methane released in the Aliso Canyon leak, according to CCST, using estimates from the California Air Resources Board. In some instances, groundwater has been contaminated by the release of natural gas from storage sites, but the extent of the risk to groundwater is not known because data are limited. We identified two examples of releases from 2000 through 2018 that potentially affected groundwater: a 2003 release at the Playa Del Rey storage site in California and a 2006 release at a storage site near Fort Morgan in Colorado. Natural gas storage site releases can impact groundwater sources in different ways. For example, these releases can impact groundwater sources above the storage site when they involve the upward migration of gas and other fluids mixed with the gas. According to CCST, this occurred at the Playa Del Rey site, where stored natural gas has leaked into a freshwater aquifer for a number of years. In other cases, faulty natural gas well design and construction, such as inadequate cementing, can allow natural gas to migrate through fractures and infiltrate overlying groundwater sources or enter drinking water wells. For example, gas infiltrated an aquifer that served drinking water wells in Fort Morgan, Colorado, which led to an evacuation of about a dozen families until the release was stopped. Subsurface leaks can also result from abandoned wells in which the casings or cement have degraded over time or from improperly plugged wells. In January 2017, PHMSA started collecting data from operators on incidents, including releases of natural gas from underground storage sites that cause more than $50,000 of property damage; these incidents could include leaks that harm groundwater resources, according to PHMSA officials. Based on our review of PHMSA incident information, no reported incidents have included groundwater contamination. Moreover, PHMSA officials told us they are not aware of any incidents involving groundwater contamination that meet reporting thresholds. PHMSA does not require operators to submit information about groundwater contamination unless that contamination meets the regulatory definition of an incident. Natural gas storage is an integral part of the nation’s energy system, ensuring that energy is available to meet peak demands across the nation. PHMSA’s safety program for natural gas storage fills a gap that existed in the regulation of underground storage prior to 2017. PHMSA met its inspection targets in the first year of its program, but it faces challenges in meeting its performance goal to inspect 400 storage sites by 2023 because fewer states agreed to partner with the agency on inspections than PHMSA originally envisioned. Because of the increase in its inspection workload from its preliminary estimate, PHMSA does not have assurance that it has enough resources to meet its inspection goal. PHMSA officials told us that while the agency has conducted a workforce assessment, it will not have the data to complete a workforce analysis it can use to guide its workforce allocations and budget requests until 2022 or 2023. The officials also told us that more states may decide to participate in the future. By analyzing the factors affecting states’ willingness to participate in inspections and analyzing its workforce needs on an ongoing basis, PHMSA would have more reasonable assurance that it has the necessary staff to meet its inspection goal. In addition, while PHMSA addressed one of the two recommendations in our November 2017 report and has established a performance goal that provides information about the number of completed inspections, this performance goal does not provide information on the outcome of PHMSA's efforts to improve safety at natural gas storage sites, consistent with leading practices under GPRA. By establishing performance goals that demonstrate improvements to safety outcomes, such as tracking reductions in incidents ranging from releases of natural gas to death or major injury, PHMSA would have better assurance that it can measure its progress toward meeting its strategic goal to improve safety. We are making the following two recommendations to PHMSA: The PHMSA Administrator should analyze the factors affecting states’ participation in underground natural gas storage inspections and analyze its workforce needs on an ongoing basis to guide its budget requests. (Recommendation 1) The PHMSA Administrator should establish performance goals that demonstrate improvements to safety outcomes for the natural gas storage program, such as tracking reductions to incidents. (Recommendation 2) We provided a draft of this report to DOT for review and comment. In written comments, DOT concurred with the report’s recommendations and provided additional information on steps it is taking or plans to take as part of its implementation of the underground natural gas storage program. In addition, DOT stated that it would provide a detailed response to each recommendation within 180 days of our final report’s issuance. The complete comment letter is reproduced in appendix IV. If you or members of your staff have any questions about this report, please contact me at (202) 512-3841, or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report (1) assesses the extent to which the Department of Transportation’s Pipeline and Hazardous Materials Safety Administration (PHMSA) has further developed its natural gas storage program since our November 2017 report, (2) describes what is known about the potential health effects from chemicals in stored natural gas, and (3) describes what is known about the potential environmental effects of releases at natural gas storage sites. To address these objectives, we reviewed documents from PHMSA, the Department of Energy, and the Environmental Protection Agency (EPA) and met with officials from these agencies to understand their roles in natural gas storage safety. Since there was no comprehensive list of natural gas storage releases, we conducted a literature search for reports of incidents that occurred in the United States from 2000 through 2018. Later, we expanded our search to include reports of incidents related to mercaptan, an odorant added to natural gas, regardless of whether these incidents occurred at a natural gas storage site. We sought reports and studies from news reports and trade and peer-reviewed journals. We conducted searches in research databases such as Nexis’ All English Language News, Elsevier’s Scopus, Ei EnCompassLIT, and Chemical Safety Newsbase. We further expanded our search to include state or county reports that had conducted studies or released reports on these issues. We also reviewed three reports referred to us by agency officials we interviewed that compiled lists of natural gas storage releases to identify those releases that occurred from 2000 through 2018 at underground natural gas storage sites in the United States. The specific reports we reviewed were: An Appraisal of Underground Gas Storage Technologies and Incidents, for the Development of Risk Assessment Methodology; “Analysis of Occurrences at Underground Fuel Storage Facilities and Assessment of the Main Mechanisms Leading to Loss of Storage Integrity”; and U.S. Natural Gas Storage Risk-Based Ranking Methodology and Results. We also included a list of incidents at natural gas storage sites in 2017, the first year for which PHMSA collected and compiled these data for underground natural gas storage. From these sources, we identified 93 releases of natural gas from storage sites; these 93 releases include incidents as defined by PHMSA regulations as well as releases of natural gas that may not meet that definition. The releases we identified could include releases, leaks, explosions, or fires that occurred at natural gas storage sites, and we included these releases regardless of the severity of their impacts, such as injury, death, cost associated with release, or volume of gas released in the incident. We excluded releases at other types of storage, such as aboveground storage or oil storage. This list may not represent the complete universe of releases because not all releases may have been documented, and no federal agency or independent source cataloged all releases for this time period. We reviewed the list of releases to identify any documented examples of health or environmental effects associated with a release. We identified one example of reported health symptoms associated with a natural gas storage release at the Aliso Canyon Storage Site in 2015; the studies we identified did not empirically link the release of natural gas at Aliso Canyon to health effects. The studies also identified two examples of potential groundwater impacts from two other natural gas storage leaks. We visited natural gas storage facilities selected to represent each of the three types of underground storage—for depleted fields, Aliso Canyon in California; for salt caverns, Moss Bluff in Texas; and for aquifers, Redfield in Iowa. We reviewed documentation from each site and interviewed these sites’ operators. We selected these sites for specific reasons: Aliso Canyon because of the 2015 leak, Redfield because it was scheduled to undergo an inspection by PHMSA at the time of our visit, and Moss Bluff because it was readily accessible from a major urban area (Houston, Texas). Our findings from the sites we visited and officials we interviewed are not generalizable to sites and officials we did not include in our review but provide illustrative examples of such sites. We also met with officials from industry groups that represent companies that operate natural gas storage sites—the American Gas Association, American Petroleum Institute, and Interstate Oil and Gas Compact Commission—to better understand these groups’ perspectives on the natural gas storage safety program. We also met with the Environmental Defense Fund to understand its perspective on natural gas storage. To examine the extent to which PHMSA has taken action since our 2017 report to continue developing its program for natural gas storage, we reviewed documents related to the program, including strategic plans, business plans, guidance and plans related to inspections, data on the number of trained inspectors and completed inspection counts, and workforce planning. We also met with PHMSA officials to discuss the program. We selected a nongeneralizable sample of seven states: four of the five states with the largest amount of working natural gas storage (Michigan, Texas, Louisiana, and California), one state in which PHMSA was conducting an inspection (Iowa), and two additional states that had considered partnering with PHMSA (Alaska and Colorado). We met with officials representing these seven states to understand their perspectives on PHMSA’s natural gas storage safety program and their efforts to partner with PHMSA and conduct inspections. We compared PHMSA efforts on its natural gas storage program’s workforce planning with our prior work on best practices in workforce planning. We also compared PHMSA’s efforts on strategic planning with leading strategic planning practices that our past work has identified. For example, we have previously reported that requirements of the Government Performance and Results Act of 1993, as amended—such as performance goals—that apply at the departmental or agency level can serve as leading practices for planning at lower levels, such as component agencies, offices, programs, and projects, within federal agencies. To describe what is known about the potential health effects from chemicals in stored natural gas, we used our literature search results that identified releases from 2000 through 2018 to determine whether there were any studies that empirically linked the releases of natural gas in storage sites with health effects; we did not find any such studies. Since no list of natural gas storage site composition exists, we took steps to identify the components and chemicals that may be present in stored natural gas. First, we identified operators of natural gas storage sites that represented 49 percent of the total storage capacity of all natural gas storage sites within the United States. We identified these operators by reviewing Energy Information Administration data on natural gas storage working capacity from 2016. Next, we obtained and analyzed each operator’s Safety Data Sheet for natural gas and identified the components of natural gas. Also, we reviewed the interagency task force report to identify any additional chemicals that may be present in natural gas, and we reviewed reports to identify chemicals that had been identified as present in the Aliso Canyon storage site release in 2015. We then met with and obtained documents from federal agencies that focused on public health and occupational health to determine the extent to which chemicals within natural gas storage had documented potential health effects. We reviewed databases from EPA and the Agency for Toxic Substances and Disease Registry to identify the health effects that may be caused by exposure to chemicals. We also reviewed documents from and met with officials from the Occupational Safety and Health Administration (OSHA) and the National Institute for Occupational Safety and Health (NIOSH). To examine the health symptoms associated with the Aliso Canyon storage site leak, we (1) visited the storage facility; (2) met with officials from California state agencies, including the Los Angeles Department of Public Health, Division of Gas and Geothermal Resources, and South Coast Air Quality Management District to discuss the Aliso Canyon natural gas leak; and (3) reviewed reports related to potential health effects during and after the Aliso Canyon leak, including results on community health (2016); indoor dust samples (2016); and air monitoring for methane, benzene, volatile organic compounds, and sulfur odorants. Additionally, we reviewed reports from the Public Health and Environment Subgroup of an interagency task force that studied the Aliso Canyon incident and from the California Council on Science and Technology (CCST). To describe what is known about the potential environmental effects of releases at natural gas storage sites, we reviewed documentation and data from EPA on greenhouse gas emissions in general and specifically for the Aliso Canyon natural gas leak in 2015, and we spoke with officials from EPA knowledgeable about the agency’s greenhouse gas reporting program and inventory program. In addition, we obtained data from EPA estimating methane emissions from natural gas storage sites from 1995 through 2016. We assessed the reliability of these data by (1) corroborating these data with other published sources, (2) reviewing existing information about the data and the methods that produced them, and (3) interviewing agency officials knowledgeable about the data. We determined that these data were sufficiently reliable for the purposes of our reporting objectives, specifically to illustrate the relative size of the Aliso Canyon leak relative to estimated releases from natural gas sites. We identified an EPA report summarizing the amount of air emissions at the Aliso Canyon leak. For the Aliso Canyon incident in 2015, we reviewed reports that we identified through officials related to the release of methane, including results from air samples for methane taken by California agencies. We visited the Aliso Canyon storage facility and met with relevant California state agency officials. Also, through our literature search, we identified two examples of natural gas storage releases of chemicals into groundwater: the Playa Del Rey storage site in California and a storage site near Fort Morgan, Colorado. Additionally, we met with California Council on Science and Technology officials and reviewed the council’s report, Long-Term Viability of Natural Gas Storage in California, to better understand how a natural gas storage incident could impact groundwater. We also reviewed recommendations made in an October 2016 report by the Interagency Task Force on Natural Gas Storage Safety. We conducted this performance audit from December 2017 to October 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Pipeline and Hazardous Materials Safety Administration (PHMSA) has set a goal to inspect all of the approximately 400 storage sites over 5 years, from early 2018 to early 2023, according to PHMSA officials. To meet this five-year goal, PHMSA divided its workload of approximately 400 inspections over the 5 years it planned to meet its goal. PHMSA planned that its state partners would complete about one-quarter of the inspections while its federal inspectors would complete the remaining three-quarters of inspections. PHMSA’s targets for inspections, and its actual inspections according to PHMSA officials, are illustrated in table 1 below. The Pipeline and Hazardous Materials Safety Administration (PHMSA) funds its enforcement activities, such as inspections by PHMSA employees and grants to states, partially through user fees paid by operators of natural gas storage sites. However, PHMSA cannot collect user fees from operators unless expenditure of the fees is provided in advance in an appropriations act. Annually, prior to the start of the fiscal year, PHMSA submits a budget request to Congress that identifies the amount of budget authority it needs for the underground natural gas storage program. The annual appropriations act then provides for expenditure of a certain amount of fees and PHMSA is authorized to collect that amount in fees. PHMSA then obligates the fees it receives either (1) for federal activities, such as inspections by PHMSA employees, or (2) for grants to state governments, which carry out inspections at some natural gas storage sites. Table 2 provides details about the PHMSA’s budget request, budget authority, user fees, and obligations. In addition to the contact named above, Diane Raynes and Janet Frisch (Assistant Directors), Lee Carroll (Analyst in Charge), Ellen Fried, Cindy Gilbert, Jennifer Gould, Rich Johnson, Jessica Lemke, John Mingus, Katrina Pekar-Carpenter, Rebecca Parkhurst, Jeanette Soares, Sheryl Stein, Sara Vermillion, and Kiki Theodoropoulos made important contributions to this report.", "summary": "About 400 natural gas storage sites are important to the U.S. natural gas system, providing about 30 percent of the nation's energy. During a 2015 leak at a storage site near Los Angeles, about 8,000 families were temporarily relocated due to symptoms such as migraines, nausea, and respiratory problems. The leak raised concerns about health and safety risks from other storage sites. In 2017, GAO recommended that PHMSA take actions, including using baseline data to develop performance goals for its natural gas storage program. GAO was asked to review the health and environmental effects of activities at natural gas storage sites. This report, among other objectives, (1) assesses the extent to which PHMSA has developed its natural gas storage inspection program and (2) describes what is known about the potential health effects from chemicals in stored natural gas. GAO reviewed available documents about natural gas storage incidents from 2000 through 2018; compared PHMSA research, goals, and plans against leading planning practices; visited sites representing the three types of storage sites; and interviewed agency officials. In 2018, the U.S. Department of Transportation's Pipeline and Hazardous Materials Safety Administration (PHMSA) set a goal for its natural gas storage inspection program to inspect all approximately 400 natural gas storage sites within 5 years, according to agency officials. PHMSA expected that all 25 eligible states would help inspect sites, but only 10 states agreed to partner with the agency. As a result, the agency's inspection workload increased by almost 60 percent from when it set its goal, according to PHMSA data. Because of the increase in its inspection workload over its preliminary estimate, PHMSA does not have assurance that it has enough resources to meet its inspection goal. Furthermore, PHMSA has not used a workforce analysis to inform its budget requests. PHMSA officials said that the agency does not expect to have enough data until 2022 or 2023 to further inform analysis of its workforce. By analyzing factors affecting states' willingness to partner with PHMSA and its workforce needs on an ongoing basis, the agency would have better assurance that it has the staff it needs to meet its inspection goal. Health effects have been reported related to chemicals that may be found in stored natural gas. Several federal agencies—including the Environmental Protection Agency and the Agency for Toxic Substances and Disease Registry—have documented potential health effects of chemicals that may be found in stored natural gas. In addition, some chemicals may be added to natural gas, such as sulfur odorants that give natural gas a distinct smell in case of leaks. The combination of such chemicals varies from one natural gas storage site to another, based on the attributes of that site such as its geologic type and the extent to which sulfur odorants are added to the natural gas before storage. Many of these chemicals have been linked to adverse health effects. However, research is limited on the health effects of exposure to stored natural gas in general and on the effects in particular from exposure to chemicals that may occur in natural gas storage leaks or be present at the storage sites. Reports linking health effects are available on specific chemicals but not in the context of natural gas storage, based on GAO's literature review. GAO is making two recommendations, including that PHMSA should analyze factors affecting states' willingness to partner with PHMSA and analyze its workforce needs on an ongoing basis. The agency concurred with the recommendations.", "document_type": "gao"}
{"report": "When DOT issued regulations requiring accessible lavatories on twin- aisle aircraft in 1990, single-aisle aircraft were used primarily for shorter distances. However, technological advancements—such as the use of lighter, stronger composite materials—have enabled aircraft to fly longer distances with greater fuel efficiency. As a result, both Airbus and Boeing now offer single-aisle aircraft designs that can routinely fly 3,000 to almost 4,000 nautical miles—or easily from coast to coast in the continental U.S. as well as some overseas routes to and from the U.S. Of the eight U.S. carriers we interviewed for this review, five of them operate only single-aisle aircraft and the remaining three fly a mixture of single- and twin-aisle aircraft. As shown in figure 1, twin-aisle aircraft are rarely used for domestic flights by U.S. carriers. In 2018, 99 percent of U.S. aircraft departures for domestic flights occurred on single-aisle aircraft: 71.7 percent on the larger single-aisle aircraft; 26.61 percent on smaller, regional aircraft; and 0.68 percent on turbo jets. According to the 2010 U.S. Census, 57 million Americans (roughly 1 in 5) have a disability, of whom half have mobility issues that may require accommodations when flying. Furthermore, older Americans are representing a greater share of the U.S. population. By 2020, 16.5 percent of the nation’s population will be over age 65, and by 2030, 20 percent of the population will be over age 65, and the likelihood of this group’ needing assistance to access lavatories may increase as they age. As an indication of the number of people with reduced mobility flying, DOT’s monthly aviation travel data indicate that for the first 6 months of 2019, aviation passengers checked a total of 294,216 wheelchairs and scooters. Over the past 30 years, some efforts have been made to address the issue of ensuring non-discriminatory treatment of aircraft passengers, including access to aircraft lavatories. The enactment of the ACAA, which prohibits discrimination by airlines on the basis of disability, charged DOT with promulgating implementing regulations. DOT promulgated a final rule in 1990, in which it required aircraft with more than one aisle (twin-aisle aircraft) in which lavatories are provided to have at least one wheelchair accessible lavatory. DOT, however, deferred setting regulations for single-aisle aircraft, noting cost and feasibility concerns for carriers. Since that initial regulation, DOT has taken several steps to study the issue of accessible lavatories for single-aisle aircraft, but as of December 2019, none of these actions has resulted in a regulation for accessible lavatories in single-aisle aircraft. These steps include: DOT issued an advance notice of proposed rulemaking (ANPRM) to study, among other things, the issue of accessible lavatories on single-aisle aircraft that was issued in conjunction with its final rule mandating that twin-aisle aircraft must have a lavatory that is accessible to passengers who use wheelchairs. DOT created an advisory committee in 1992 to provide guidance to DOT concerning access to lavatories on single-aisle aircraft for persons with disabilities, including persons who rely on the aircraft’s onboard wheelchairs. In 1996, the committee reported to DOT that it would be feasible to provide accessible lavatories on single-aisle aircraft but acknowledged that there could be a cost to doing so. As part of a final rule that DOT issued in 2008 to amend the ACAA regulations to include foreign carriers that fly to the U.S., DOT acknowledged that requiring accessible lavatories on single-aisle aircraft was an ongoing issue. While the department noted that accessible lavatories on single-aisle aircraft would benefit passengers with disabilities, it also expressed concerns that revenue loss and other cost impacts could be too great for the carriers. The department said that it would continue to study the issue and review ongoing developments. DOT published a notice of intent in December 2015 to explore the feasibility of conducting a negotiated rulemaking concerning, among other things, accessible lavatories on single-aisle aircraft for travelers with disabilities. As a result of the 2015 notice of intent, DOT established the ACCESS Advisory Committee in 2016—composed of representatives from air carriers, aircraft manufacturers, disability groups, and other aviation stakeholders. Its charge was to negotiate and develop a proposed amendment to DOT regulations for DOT’s consideration concerning accommodations for air travelers with disabilities that would address whether to require accessible lavatories on new single-aisle aircraft, among other issues. The committee noted that the issue of requiring accessible lavatories on single-aisle aircraft merited exploration because of two developments: (1) the increased use of single-aisle aircraft on long flights, and (2) the availability of new accessible- lavatory designs for single-aisle aircraft. In late 2016, the ACCESS Advisory Committee agreed on proposed amendments that included short-term and long-term solutions to address the challenges persons with mobility impairments face when traveling on single-aisle aircraft. The committee, taking into account costs to industry, recommended accessible lavatories on new aircraft, did not recommend requiring the retrofit of existing aircraft, and proposed a multi-tiered approach to meet this goal. In 2019, DOT publicized its intent to issue notices of proposed rulemaking regarding accessible lavatories to address the ACCESS Advisory Committee’s final resolution, which we discuss in more detail below. Both Airbus and Boeing offer their customers a range of standard lavatory designs. For example, both Airbus and Boeing offer a lavatory with a contoured design (see fig. 2). This design offers a smaller sink and different dimensions than previous lavatories and has a contoured or angled wall on the exterior allowing seats in the last row to recline into the bottom portion of the contour. Air carriers can also choose to move the last row of seats back into the contour and then add an extra row of seats after making other changes to the configuration of seating rows. Airbus and Boeing also offer flat-wall lavatory designs that are similar to the standard flat-wall lavatories that had previously been available on single- aisle aircraft for years but have slightly different interior dimensions. Compared to the contoured lavatory design, this current flat-wall lavatory design could offer a larger sink or more countertop space. According to measurements and diagrams the aircraft and lavatory manufacturers provided for these lavatory styles, some interior lavatory dimensions have decreased while other dimensions have increased. For example, changes in these two lavatory styles have resulted in increased interior space in some areas, such as the sitting knee space and diagonal shoulder width, and decreased space in other areas, such as the entry width and door height. These changes were to provide carriers options to help them meet their business strategies. In addition to making changes to the standard lavatories, since 2015, both Airbus and Boeing offered lavatory configurations for their single-aisle aircraft designed to provide greater access for passengers who rely on the use of onboard wheelchairs. According to officials for Airbus and Boeing, both manufacturers use a design that connects two adjacent lavatories with a retractable wall or partition. As shown in figure 3, when the folding partition is open, this configuration is designed to enable the person who relies on the aircraft-onboard-wheelchair to enter in one of the lavatories and then transfer or be transferred to the toilet in the other lavatory. While there are differences between the Boeing and Airbus models, they operate similarly. Both the Airbus and Boeing designs are for the rear of the aircraft and take up space in the area normally used for the galley where food and drink carts are located for flight attendants’ access. According to officials we interviewed from two carriers that have purchased aircraft with this design, a reduced galley area is less of a concern because their flights provide limited food and beverage service and do not need a full galley. They said that the space where the traditional lavatories were located could be used for other purposes, such as more seats. According to the manufacturers, the lavatory models that are designed for greater accessibility accommodate the onboard wheelchair to varying degrees. Airbus offers two designs to accommodate a passenger with an onboard wheelchair. The Space Flex version 1 design consists of two adjacent lavatories with a connecting retractable partition. This retractable partition can open to allow for a passenger who relies on the aircraft onboard-wheelchair to enter the lavatory with or without the help of an assistant. A representative from a disability organization was generally positive about the Space Flex version 1 and said it was a good design for both carriers and travelers with disabilities. Airbus also offers another lavatory design specifically for its A220 single-aisle aircraft model. Airbus officials told us that it is a single lavatory that is designed to accommodate a wheelchair but cannot accommodate both a passenger in an onboard wheelchair and an assistant. Boeing offers one lavatory designed to accommodate a passenger using an onboard wheelchair for single-aisle aircraft for its 737 aircraft family. This design, known as the Pax Plus, consists of two adjacent lavatories with a removable partition designed to enable a wheelchair and assistant to enter. In addition, officials from the eight selected air carriers told us that their crews are trained to assist passengers with reduced mobility to use lavatories. These officials from the eight air carriers stated that they provide their cabin crew with initial and, in some cases, recurrent training about how to assist passengers with reduced mobility, pursuant to DOT regulations. DOT regulations further stipulate that if there is an on-board wheelchair, the carrier must provide assistance to enable the passenger to move to and from the lavatory if, in general, such assistance is requested by or on behalf of a passenger with a disability. While aircraft manufacturers offer lavatories designed to accommodate passengers with mobility impairments, carriers do not often choose to acquire them. Of the eight U.S. carriers we interviewed, we found that four have some aircraft—all of which are Airbus aircraft—with lavatories that are designed to accommodate passengers with mobility impairments to some extent. Only one of these carriers is among the four with the largest number of aircraft in their fleet. Specifically, these four carriers have either the Space Flex version 1 or the Airbus A220 lavatory. Despite Boeing’s offering of the Pax Plus lavatories since 2017, Boeing officials told us that as of November 2019 no U.S. carriers have ordered these lavatories for their current or future single-aisle Boeing aircraft. Overall, about 4.5 percent of the combined single-aisle fleet of the eight selected carriers have lavatories designed to provide some measure of greater access to passengers with reduced mobility, including those who require the use of the onboard wheelchair (see fig.4). According to the carriers we interviewed, they consider the configuration of the aircraft among other factors, including their business strategy, when ordering lavatories for new aircraft. Providing a lavatory designed to accommodate onboard wheelchairs on single aisle aircraft may require financial tradeoffs for carriers, such as reducing the number of revenue generating seats in the aircraft cabin. According to airline officials, this reduction can result in higher costs for carriers that subsequently might be passed onto consumers through higher fares. Officials from all eight selected carriers, however, stated that all of their aircraft lavatories have features designed to increase access to certain lavatory functions, such as assist handles or grab bars, accessible call buttons, door locks, and faucets that passengers with disabilities can use. Carrier officials also stated that they need to make trade-offs between competing priorities; for example, taking into account how onboard wheelchair-accommodating lavatory designs may affect food service. According to officials from two carriers, an onboard wheelchair- accommodating lavatory can result in less galley space, and a full galley at the back of the aircraft is needed for the type of services they wish to provide to their customers without compromising customer seating capacity. Conversely, officials from two other carriers told us that trading galley space for onboard wheelchair-accommodating lavatories did not affect their food service, as they do not provide full meal service. For example, they said that because they did not need the full galley space, the Space Flex lavatory enabled them to add not only a lavatory that accommodates onboard wheelchairs but also an additional row of passenger seats. Representatives from stakeholder groups we interviewed told us that the lack of accessible lavatories makes flying challenging for persons with reduced mobility. They described how some passengers with reduced mobility take precautionary measures to avoid the need to use an aircraft lavatory, such as severely limiting food and fluid intake in advance of the flight, risking dehydration; using a catheter; or wearing a protective undergarment. Some passengers with reduced mobility reportedly may avoid long flights altogether by purchasing flights with connections or layovers. However, according to one stakeholder group, these precautionary measures may not alleviate the fear and anxiety that passengers who rely on the onboard wheelchair to get to the lavatory may face during air travel as there is always the possibility of having to deal with circumstances beyond their control. For example, unforeseen events such as increased flight time or delays in getting to the gate can increase the time a passenger has to postpone attending to normal bodily functions. Finally, stakeholder groups report that passengers may choose not to travel at all, or to drive rather than fly, choices that may increase the cost and time of travel, particularly if it involves an overnight stay. Even when an aircraft has a lavatory that can accommodate an onboard wheelchair, which exists on about 4.5 percent of the combined fleet of single-aisle aircraft for the 8 airlines included in our review, passengers may have difficulty determining whether or not their flight has such a lavatory. According to officials of air carriers, passengers may call the carriers’ customer service department for this information, although not all phone representatives may have this information readily available. In addition, our review of selected carriers’ websites revealed that most do not have information about which flights or aircraft may have such a lavatory, although we found that two carriers include descriptions of aircraft amenities or diagrams denoting onboard wheelchair- accommodating lavatories. However, even if this information were made available, it may not guarantee that a passenger with a mobility impairment will be able to fly on an aircraft with this type of lavatory because air carriers sometimes switch aircraft at the last minute without notice, such as when, for example, an aircraft has a mechanical problem. While stakeholders described challenges, neither air carriers nor DOT receive a large number of complaints regarding the lack of lavatories designed to accommodate passengers who use onboard wheelchairs or lavatories in general. As we have previously reported, DOT receives and processes complaints from passengers and uses complaint data to help identify which carriers to inspect for consumer protection violations. From 2014 through 2018, DOT received 59,846 complaints about U.S. carriers. Of these, we reviewed 1,263 complaints related to accessibility, inadequate facilities, and flight delays and identified 69 complaints about lavatories in general and 5 about the accessibility of lavatories. Of the 69 lavatory complaints identified: 64 related to non-functioning lavatories (e.g., non-operational or unclean lavatories, sinks lacking running water, etc.); 5 related to lavatories being inaccessible by persons with disabilities (e.g., lavatory grab bars at an improper height, passenger using onboard wheelchair unable to enter lavatory); and 2 related to lavatory size (e.g., lavatory size has been reduced). We also discussed lavatory-related complaints with the eight selected air carriers, three of which reported that these complaints made up about 1 percent or less of the total passenger complaints they received in 2018. Four air carriers reported that lavatory complaints related to accessibility made up an even smaller portion—around 0.05 percent or less of their total passenger complaints in 2018. However, the small number of complaints related to lavatory accessibility does not necessarily indicate that individuals who use onboard wheelchairs are not affected by inaccessible aircraft lavatories, as some may choose not to fly, and others may take precautionary measures as described above to avoid having to use the aircraft lavatory. Furthermore, because accessible lavatories are not required on single-aisle aircraft and there is no expectation that the lavatory would be accessible, passengers may not see grounds to complain or may not take the time to submit a complaint. As we have previously reported, complaint data are inherently limited because, according to academic literature, a substantial portion of dissatisfied individuals never complain and are therefore not represented in the complaint data. Finally, when they do complain, their complaints may not be representative of other individuals. We also found that there were very few complaints about non-functioning lavatories. As noted above, DOT received 64 passenger complaints on non-functioning lavatories. Carrier officials also told us that they have received few complaints about non-functioning lavatories. According to the air carrier officials we interviewed, depending on the flight, some flights may operate with one or more lavatories not functioning. However, most carrier officials reported that according to data they collect, this occurred on less than 2 percent of flights. In such instances, some carrier officials stated they would notify passengers of nonoperational lavatories to give them the opportunity to use the airport lavatories prior to boarding. These officials also stated that if all lavatories are inoperable it is the responsibility of the pilot—in consultation with flight dispatchers—to decide if the aircraft will take off or, if lavatories become inoperable during a flight, to divert to an airport other than the destination. Carriers further noted that flights with no operational lavatories are extremely rare. As previously noted, in late 2016, the ACCESS Advisory Committee reached a consensus on proposed amendments that would require accessible lavatories on single-aisle aircraft. DOT announced in 2019 that it would address the issue in rulemaking. On December 16, 2019, DOT issued a notice of proposed rulemaking to solicit comments on short-term accessibility improvements on single-aisle aircraft through the installation of accessibility features within the lavatory, such as assist handles, call buttons, and lavatory controls, without changing the size of lavatories. In addition, DOT has announced its intention to issue an advanced notice of proposed rulemaking to address long-term accessibility improvements, also addressed by the Advisory Committee, and to solicit comments and gather information on the costs and benefits of requiring carriers to increase the size of the single-aisle lavatory on new aircraft models to enable passengers using an onboard wheelchair to enter and use the lavatory with an assistant, if necessary. In addition to the two rulemakings, DOT has recently established another advisory committee. The ACAA Advisory Committee was created in response to a requirement in the FAA Reauthorization Act of 2018, has a 2-year charter, and is required to report its findings to both DOT and Congress on current DOT regulations on barriers to persons with disabilities who want to travel by air. The ACAA Advisory Committee is also required to determine the extent to which DOT is addressing those barriers, recommend improvements to implement the ACAA, and improve the flying experience for travelers with disabilities. The committee— comprised of members representing aircraft manufacturers, national disability organizations, air carriers, and airports—plans to hold its first meeting in early 2020. According to DOT officials, although it is within the purview of this committee to consider issues regarding accessible lavatories, it does not plan to do so at the present time given that the two proposed rulemakings are proceeding and that there is a Congressional mandate for the committee to report on other issues within 6 months of the first meeting. We provided a draft of this report to DOT for review and comment. DOT provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, DOT, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have questions concerning this report, please contact me at (202) 512-2834, or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the individual named above, other key contributors to this report were Ed Laughlin, Assistant Director; Martha Chow, Analyst-in- Charge; James Geibel; Geoffrey Hamilton; Elke Kolodinski; Gail Marnik; Malika Rice; Amy Rosewarne; Travis Schwartz; Pamela Snedden; and Elizabeth Wood.", "summary": "Flying can pose significant challenges for persons who rely on wheelchairs, including the lack of wheelchair accessible lavatories on most flights. In 1990, DOT required wheelchair accessible lavatories on twin-aisle aircraft used mainly for long flights. It did not require them for single-aisle aircraft, although DOT continued to study the issue. Since 1990, technological advances have enabled single-aisle aircraft to fly longer distances, and these aircraft now make 99 percent of domestic flights. In 2016, a DOT advisory committee recommended that DOT require accessible lavatories in certain single-aisle aircraft in the future. The Federal Aviation Administration (FAA) Reauthorization Act of 2018 included a provision that GAO examine the availability and designs of lavatories on commercial aircraft and the ability of passengers with disabilities to access them. This report describes (1) what is known about lavatory designs and accessibility for persons with reduced mobility and (2) the challenges wheelchair-bound passengers and others face while traveling on single-aisle aircraft without accessible or functional lavatories. GAO reviewed DOT's guidance and rulemaking and analyzed DOT's aircraft complaint data and fleet data for the eight largest U.S. air carriers. GAO interviewed officials from the eight largest mainline carriers and reviewed their fleet and lavatory data. GAO also interviewed officials from Airbus and Boeing and subsidiary lavatory manufacturers, as well as representatives from cabin crew labor associations and consumer groups representing persons with disabilities. Aircraft manufacturers offer lavatories that carriers can provide and that are designed to accommodate users of onboard wheelchairs, but carriers do not choose to acquire this option for their single-aisle aircraft. We found designs for lavatories that enable a passenger in an onboard wheelchair to use them, to varying degrees. In recent years, both Airbus and Boeing—makers of single-aisle aircraft—began offering similarly designed lavatories to provide greater access for these passengers. For example, one design consists of two adjacent lavatories located in the rear galley area with a connecting retractable wall to allow for a wheelchair-bound passenger to enter one lavatory and transfer or be transferred to the toilet in the other lavatory. Another design is a single lavatory large enough to accommodate a passenger using an onboard wheelchair. Four of the eight U.S. carriers—and only one of the four with the largest fleets—GAO interviewed have Airbus aircraft with an adjacent lavatory design (Space Flex version 1) or the single lavatory design found on the A220 aircraft, constituting about 4.5 percent of the carriers' combined single-aisle fleet (see figure). None of the eight U.S. carriers have purchased a similar lavatory for their Boeing's single-aisle aircraft. Carrier officials told GAO that they consider many factors when ordering lavatories, including financial and service tradeoffs such as the potential to lose seating spaces, or reduced food and beverage service for passengers. While the Department of Transportation (DOT) receives few complaints on lavatory inaccessibility, consumer groups told GAO that the lack of an accessible lavatory on single-aisle aircraft presents challenges for persons with reduced mobility. For example, some passengers take precautionary measures to avoid the need to use the aircraft lavatory and others avoid flying altogether. Additionally, although some aircraft have wheelchair-accommodating lavatories, they are not well advertised to passengers, making it difficult for passengers to know whether their flight may have such a lavatory. To address such challenges and the findings of its 2016 advisory committee, DOT issued, on December 16, 2019, a notice of proposed rulemaking to require carriers to install accessibility features without changing the size of the lavatories. DOT also expressed intent to study the costs and benefits of enlarging single-aisle aircraft lavatories to enable use by passengers using the onboard wheelchair.", "document_type": "gao"}
{"report": "In the primary market, lenders originate mortgage loans to borrowers to purchase homes. To evaluate the creditworthiness of a potential borrower (called underwriting), the lender considers the borrower’s credit scores and history, monthly debts including mortgage payments relative to income (debt-to-income ratio), and the amount of the mortgage loan relative to the home’s value (loan-to-value ratio). Borrowers with strong credit histories typically receive prime mortgages with the most competitive interest rates and terms. Lenders generally require borrowers to purchase private mortgage insurance when the loan-to-value ratio is higher than 80 percent. Some borrowers also may qualify for federal mortgage insurance programs (discussed later in this section). Mortgage lending creates certain risks: Credit risk is the risk that the borrower will default on the mortgage by failing to make timely payments. Prepayment risk is the risk that borrowers will pay off the principal of the loan before the mortgage term ends. Prepayment reduces or eliminates future interest payments. The lender must relend or reinvest the prepaid amount and may have only lower-interest options available for lending or investing the funds if interest rates have decreased. Interest rate risk is the risk that an increase in interest rates will reduce the value of a loan for the lender. For example, a lender might fund mortgage lending through short-term deposits. If interest rates rise and the lender previously made a long-term fixed-rate mortgage at a lower rate, the difference between the interest payments the lender receives from the mortgage and the interest the lender has to pay to its depositors decreases. Liquidity risk is the risk that an institution will be unable to meet its financial obligations as they come due without incurring unacceptable losses. For example, firms can be exposed to liquidity risk by funding longer-term asset purchases with shorter-term debt obligations. After origination, mortgages are serviced until they are paid in full or closed due to nonpayment. Servicers can provide borrowers with account statements, respond to customer service questions, and collect monthly payments, among other duties. The servicer can be the same institution that originated the loan or the servicer can change as institutions sell servicing rights. Lenders hold mortgage loans in their portfolios or sell them to institutions in the secondary market (see fig. 1). Lenders sell their loans to transfer risk (such as interest rate risk in the case of fixed-rate mortgages) or to increase liquidity. Secondary market institutions can hold the mortgages in their portfolios or pool them into MBS that are sold to investors. Participants in the secondary market include federal entities, issuers of private-label MBS, and investors. Private institutions, primarily investment banks, may issue MBS (known as private-label securities) which are backed by mortgages that are not federally insured and do not conform to the enterprises’ requirements. The federal government participates in the primary and secondary mortgage markets as both an actor and a regulator. In the primary market, the federal government operates mortgage guarantee and insurance programs to promote homeownership for certain types of borrowers. For example, the Federal Housing Administration (FHA), Department of Veterans Affairs (VA), Department of Agriculture’s Rural Housing Service, and HUD’s Office of Public and Indian Housing offer programs that insure mortgages against default or guarantee lenders payment of principal and interest. In the secondary market, the federal government facilitates mortgage lending through the enterprises (discussed below) and the Government National Mortgage Association (Ginnie Mae). Ginnie Mae is a federally owned corporation within HUD that guarantees the timely payment of principal and interest to investors in securities issued through its MBS program. Ginnie Mae-guaranteed MBS consist entirely of mortgages insured or guaranteed by federal agencies (such as FHA) and are issued by financial institutions it approves. The federal government also regulates the housing finance system through FHFA, which oversees the enterprises; the Bureau of Consumer Financial Protection, also known as the Consumer Financial Protection Bureau (CFPB); and the federal banking regulators, which enforce regulatory standards for mortgage lending. Congress chartered Fannie Mae and Freddie Mac as for-profit, shareholder-owned corporations in 1968 and 1989, respectively. They share a primary mission to enhance the liquidity, stability, and affordability of mortgage credit. The enterprises generally purchase mortgages that meet certain criteria for size, features, and underwriting standards (known as conforming loans) and hold the loans in their own portfolios or pool them into MBS that are sold to investors. In exchange for a fee, the enterprises guarantee the timely payment of interest and principal on MBS that they issue. The enterprises also have obligations to support housing for certain groups. Following the enactment of the Federal Housing Enterprises Financial Safety and Soundness Act of 1992, the enterprises have been required to meet specific goals for the purchase of mortgages supporting underserved groups (such as low- and moderate-income families) or certain geographic areas. In 2008, the Housing and Economic Recovery Act (HERA) tasked the enterprises to fund new affordable housing programs, including the Housing Trust Fund and the Capital Magnet Fund. The enterprises fund these programs with a dollar amount based on their unpaid balance of new business, purchases, and the funds distribute the money to states and housing organizations to support affordable housing. HERA established authorities for providing capital support to the enterprises and established FHFA as an independent regulatory agency for the enterprises. HERA also authorized the Director of FHFA to appoint FHFA as a conservator or receiver for the enterprises. FHFA put the enterprises into conservatorship in September 2008. FHFA has a statutory responsibility to ensure that the enterprises operate in a safe and sound manner and that their operations and actions of each regulated entity foster a liquid, efficient, competitive, and resilient national housing finance market. FHFA sets strategic goals for its conservatorship of the enterprises. According to FHFA, the enterprises’ boards of directors oversee day-to-day operations, but certain matters are subject to FHFA review and approval. For example, FHFA officials told us that FHFA reviews and approves some pilot programs. Fannie Mae and Freddie Mac retain their government charters and continue to operate legally as business corporations. Using authority provided in HERA, Treasury has committed to providing up to $445.6 billion in capital support to Fannie Mae and Freddie Mac while they are in conservatorship through the senior preferred stock purchase agreements. If Fannie Mae or Freddie Mac has a net worth deficit at the end of a financial quarter, Treasury will provide funds to eliminate the deficit. Under the most recent agreement in December 2017, the enterprises must pay Treasury a dividend of all their quarterly net income above a $3 billion capital reserve that each enterprise is allowed to retain. Since the 2007–2009 financial crisis, Congress has taken steps to improve regulation and consumer protection related to the housing finance system. For example, to address challenges related to limitations on mortgage information, HERA requires FHFA to collect market data. The Dodd-Frank Wall Street Reform and Consumer Protection Act (Dodd- Frank Act) created CFPB, which has undertaken a number of consumer protection initiatives related to mortgage lending and servicing. The Dodd-Frank Act also updated the Truth in Lending Act to prohibit lenders from making certain mortgage loans without regard to a consumer’s ability to repay the loan (known as the ability-to-pay rule). A lender is presumed to have met the ability-to-repay requirement when it originates a qualified mortgage—a category of loans that have certain more stable features that make it more likely a borrower will repay the loan. Congress also has considered proposals to make significant changes to the housing finance system. During the 113th Congress (January 2013– January 2015), three proposals—the Housing Finance Reform and Taxpayer Protection Act of 2014, S. 1217; the FHA Solvency Act of 2013, S. 1376; and the Protecting American Taxpayers and Homeowners Act of 2013, H.R. 2767—were reported out of committee but no further action was taken. In September 2018, the Protecting American Taxpayers and Homeowners Act of 2018 (H.R. 6746) was reintroduced in the 115th Congress and referred to committee. As of the end of the 115th Congressional session, no further action had been taken. Industry groups and think tanks also have published reform proposals. We discuss reform proposals made since 2014 in more detail later in this report. Federal agencies also have commented on housing finance reform. In early 2018, the Director of FHFA sent a letter to the Chairman and Ranking Member of the Senate Committee on Banking, Housing, and Urban Affairs stating that conservatorship is not sustainable and needs to end, and provided suggestions on how the enterprises could be reformed. For example, the letter states that the housing finance system should preserve 30-year fixed-rate mortgages, end taxpayer bailouts for failing firms, maintain liquidity, and provide a level playing field for lenders of all sizes. It also states that secondary market activities should be managed by shareholder-owned firms chartered by a regulator and operating as utilities with an explicit paid-for federal guarantee on MBS issued by regulated firms. In June 2018, the Office of Management and Budget released recommendations to reform the federal government in a number of areas, including housing finance. The recommendations propose privatizing the enterprises, allowing new private entities to enter the market, and providing an explicit federal guarantee on MBS that could only be accessed in limited, exigent circumstances. In a 2014 report, we outlined a framework composed of nine elements we consider to be critically important to help policymakers assess or craft proposals to change the housing finance system (see table 1). The housing market has recovered since the financial crisis, with significant federal support. Indicators of recovery include rising house prices and declining mortgage delinquency rates. However, the federal government has continued to support the housing market with guarantees on more than two-thirds of new mortgages each year since 2008, either through government-insured originations or by guaranteeing timely payment to investors on mortgage loans purchased and securitized by the enterprises. The government also has continued to play a very substantial role in the secondary market, guaranteeing around 95 percent or more of all MBS issued annually since 2008. But recent trends—some loosening of underwriting standards, the rise of nonbank mortgage lenders and servicers, and less access to affordable housing and homeownership—may pose additional risks and challenges to the housing market and participants, including the enterprises. Several indicators demonstrate that the housing market has recovered since the financial crisis of 2007–2009. For example, real national average house prices have consistently risen each year since 2012 (see fig. 2). The rise in house prices also has been complemented by consistent economic growth, declining unemployment, and low mortgage rates since 2009. Higher house prices have some positive implications for the financial soundness of the enterprises: higher prices can reduce the enterprises’ potential losses due to defaulted loans because the enterprises can recover more value from properties securing the loans. Serious delinquency rates (90 or more days delinquent) for mortgages purchased by Fannie Mae and Freddie Mac have declined steadily and since 2014 have remained between 1 and 2 percent for both enterprises (see fig. 3). Examining delinquency rates for mortgages by origination year reveals significant differences for mortgages originated before and after the financial crisis. According to Fannie Mae and Freddie Mac reports, mortgages originated since 2009 have had lower delinquency rates than those originated before 2009. For example, in 2017, Fannie Mae’s serious delinquency rate was 6.6 percent for mortgages originated in 2005–2008, compared to 0.5 percent for mortgages originated since 2009. As of October 2018, mortgages originated since 2009 represented more than 90 percent of Fannie Mae’s and 80 percent of Freddie Mac’s outstanding held loans and guaranteed MBS. Serious delinquency rates for mortgages insured by FHA are higher on average than those purchased by the enterprises but generally have followed similar trends. Compared to pre-2007 levels, trends in mortgage originations indicate a smaller-volume market largely composed of prime conforming and government-insured mortgages, as shown in figure 4. During 2008–2017, total mortgage origination volume—the dollar value of mortgage loans— remained below pre-crisis levels. Much of the decrease in volume resulted from large declines in prime jumbo and nonprime originations since 2008. Prime jumbo and nonprime originations represented a significant share of originations (market share) before 2007 but declined sharply since 2008. Prime jumbo market share recovered somewhat, increasing from a low of 6 percent in 2009 to approximately 18 percent of originations each year since 2014. Riskier nonprime originations remain very low compared to their pre-crisis levels. Meanwhile, federally insured mortgages (such as those insured by FHA or guaranteed by VA) grew significantly in 2008 and retained a market share between 19 and 25 percent in 2008–2017. Finally, prime conforming origination volume varied year-to-year but these mortgages have represented the majority of originations since 2007. Federally insured and prime conforming mortgages represented 80 percent or more of originations every year since 2008. The federal government has continued to support a significant share of the mortgage markets since the financial crisis. For instance, while down from the peak in 2009, the federal government has guaranteed more than two-thirds of new mortgages since 2014, either by insuring mortgages or by guaranteeing timely payment to investors on loans purchased and securitized by the enterprises (see fig. 5). Government-insured mortgages declined leading up to the financial crisis, largely due to the availability of nonprime mortgages and securitization by fully private institutions. But when the availability of these products declined sharply, government agencies such as FHA and VA insured or guaranteed significantly higher volumes of mortgages. For instance, the share of mortgages insured or guaranteed by federal agencies grew from 6 percent ($134 billion) in 2007 to more than 20 percent ($328 billion) in 2008. As of 2017, federally insured mortgages were 25 percent ($444 billion) of total originations. Similarly, as the share of conventional mortgages held in banks’ portfolios declined during the financial crisis, the enterprises purchased and securitized large volumes of these mortgages. The share of mortgage originations purchased by the enterprises peaked at 65 percent in 2008 and still accounted for nearly half of new mortgages in 2017. The federal government also has maintained a very substantial role in the secondary mortgage market since the financial crisis. The enterprises and Ginnie Mae guaranteed around 95 percent or more of all MBS issued each year since 2008, despite a nearly decade-long economic expansion. In line with the rise in federally insured originations, Ginnie Mae’s market share increased substantially, from 5 percent ($110 billion) in 2007 to 22 percent ($301 billion) in 2008, and about 33 percent ($455 billion) in 2017 (see fig. 6). Conversely, private-label MBS issuance since 2008 has been minimal, as many private-label issuers left the market and nonprime originations declined. The growth in the market share of Ginnie Mae and the enterprises resulted in part from actions by Congress and the Board of Governors of the Federal Reserve System (Federal Reserve). Congress increased the loan limits for FHA-insured loans and loans eligible for securitization by the enterprises. The federal government also made its backing of securities issued by the enterprises explicit by committing to provide them financial assistance, and Ginnie Mae continued to provide guarantees for securities backed by federally insured mortgages. According to several mortgage originators, securitizers, investors, and researchers with whom we spoke, the enterprises will continue to dominate the MBS market because the federal guarantee through conservatorship offers a competitive advantage over other participants without such a guarantee. In response to the financial crisis, the Federal Reserve provided additional support for the mortgage market, becoming one of the largest purchasers of MBS issued by the enterprises and guaranteed by Ginnie Mae. Among other impacts, this action made these securities somewhat more attractive to secondary market participants. In June 2017, when the Federal Reserve’s MBS holdings had peaked at $1.78 trillion, it announced plans to gradually reduce its MBS holdings as part of its efforts to reduce the size of its balance sheet. As of November 2018, the Federal Reserve had $1.66 trillion in MBS holdings. Recent trends—particularly changes in underwriting standards and borrowers’ credit risk profiles, the rise of nonbank mortgage lenders and servicers, and limited access to affordable housing and homeownership— pose risks and challenges to the housing market and participants, including the enterprises. Indicators of borrower credit risk and surveys of loan officers indicate a loosening of underwriting standards in recent years. More specifically, indicators of borrower credit risk for mortgages the enterprises purchased suggest underwriting standards tightened in 2008 but loosened slightly since 2012, which could pose increased risk to the enterprises. Specifically, average combined loan-to-value ratios (for all loans on the property) and debt-to-income ratios have increased, while average borrower credit scores have declined. The enterprises and FHA include assessments of these measures in setting their underwriting standards. As discussed earlier in the report, mortgages originated since 2009 have performed much better than those originated before 2008, but remain untested by a large-scale stressful economic event. Furthermore, mortgages to refinance an existing mortgage (as opposed to mortgages for purchasing a home) declined since 2012. According to FHFA officials, credit scores, loan-to-value ratios, and debt-to-income ratios tend to be stronger for refinance mortgages than purchase mortgages. FHFA and HUD officials also told us that reduced refinancing volume due to rising interest rates may put additional pressure on lenders to maintain volume and profitability by offering more relaxed credit terms to borrowers. Average combined loan-to-value ratios for mortgages purchased by the enterprises peaked in 2014 and have remained roughly similar to pre- crisis levels (see fig.7). In December 2014, FHFA began allowing the enterprises to purchase mortgages with loan-to-value ratios up to 97 percent. In the first three quarters of 2018, 22 percent of mortgages Fannie Mae purchased included a loan-to-value ratio over 90 percent, which is higher than shares in 2005–2008. FHA’s loan-to-value ratio is limited to 96.5 percent, and the average among borrowers has remained relatively consistent around 93 percent since 2008. The higher the loan- to-value ratio when a loan is originated, the less equity borrowers will have in their homes and the more likely they are to default on mortgage obligations, especially during times of financial stress or falling home values. Additionally, house price valuation—measured by the price-to- rent ratio—has increased substantially since 2012 to levels last seen in 2004. Higher valuations could increase the risk of future price decreases—which would reduce collateral values that protect the enterprises against losses in the event of default—or more modest price increases. This could signal increased risk when associated with higher loan-to-value ratios. Average credit scores for enterprise-purchased loans rose significantly from their pre-crisis lows and remained historically high through 2012 but have since slightly declined (see fig. 8). The average credit score of FHA-insured borrowers, while lower than those for loans purchased by the enterprises, followed a trend similar to those of the enterprises. Generally, a higher score indicates a greater credit quality and potentially lower likelihood of default. Lenders continue to use credit scores as a primary means of assessing whether to originate a loan to a borrower. Average debt-to-income ratios for mortgages purchased by the enterprises remained below their pre-crisis levels but have deteriorated since 2012, and the share of high debt-to-income mortgages rose. Additionally, according to Fannie Mae financial reports, in the first three quarters of 2018, roughly 25 percent of mortgages it purchased included a borrower debt-to-income ratio over 45 percent, up from roughly 7 percent of mortgages in the first three quarters of 2017. The share of high debt-to-income ratios for FHA-insured borrowers also has risen significantly. For example, nearly half (49 percent) of FHA- insured borrowers in fiscal year 2017 had high debt-to-income ratios, surpassing the previous high of 45 percent of borrowers in 2009. According to FHA, as of March 2018, about 24 percent of mortgages included debt-to-income ratios above 50 percent, up from 20 percent of mortgages in March 2017. The Dodd-Frank Act requires mortgage lenders to make “a reasonable, good faith determination” of a borrower’s ability to repay the loan. A lender that originates a “qualified mortgage” is presumed to have met this requirement. All qualified mortgages must meet mandatory requirements including restrictions on points and fees, and loan structure. In addition, the borrower’s debt-to-income ratio must be 43 percent or less; however, loans eligible for purchase by the enterprises or to be insured by the FHA, VA or USDA are not subject to a specific debt-to-income ratio. Additionally, according to results from the October 2018 Senior Loan Officer Opinion Survey on Bank Lending Practices, more loan officers reported loosening than tightening their underwriting standards for enterprise-eligible mortgages every quarter from 2015 through the second quarter of 2018. More officers reported loosening their standards for government-insured mortgages during 12 of the last 16 quarters. Our review found that the increased role of nonbank mortgage lenders and servicers in recent years has helped provide liquidity and access to mortgage credit but also presented additional liquidity risks. FHFA and HUD officials reported that the share of nonbanks mortgage originators and servicers grew since the financial crisis. According to data from Inside Mortgage Finance, nonbanks originated roughly half of all mortgages sold to the enterprises in 2017 and the first three quarters of 2018. Of the top 10 mortgage sellers to the enterprises in the first three quarters of 2018, six were nonbanks that originated more than 20 percent of all enterprise purchases during that period. Nonbank servicers of loans backing enterprise MBS have grown from 25 percent in 2014 to 38 percent as of the third quarter of 2018. For FHA-insured mortgages, nonbank originations represented 74 percent in 2003, declined to 56 percent in 2010, and then increased to 86 percent in fiscal year 2017. While FHFA and HUD officials told us nonbanks have helped provide access to mortgage credit, several stakeholders and experts in all four of our panels identified the increased presence of nonbank lenders as a current risk in the housing finance system. A 2018 paper published by the Brookings Institution cited that nonbanks are exposed to significant liquidity risks in their funding of mortgage originations and servicing of mortgages, because nonbank lenders rely more on credit lines provided mostly by banks, securitizations involving multiple players, and more frequent trading of mortgage servicing rights than banks. For instance, during times of financial stress, lenders to nonbanks have the right to quickly pull their lines of credit and seize and sell the underlying collateral if nonbanks do not maintain certain levels of net worth. HUD officials identified similar risks and added that this may reduce borrower access to credit in the event of financial stress or a liquidity crisis. Additionally, while nonbanks are subject to some federal and state oversight, they are not federally regulated for safety and soundness. State regulators may require nonbanks to be licensed and may examine their financial soundness and compliance with relevant state laws, but there are no such federal regulations, unlike with banks. The Conference of State Bank Supervisors has a series of initiatives with the goal of all state regulators adopting a nationwide nonbank licensing and supervisory system by 2020. CFPB oversees nonbank issuers for compliance with consumer financial protection laws but not for financial safety and soundness. We reported in 2016 that incomplete information on the identity of nonbank servicers may hinder those responsible for their oversight. The lack of federal safety and soundness oversight of nonbank lenders and servicers may pose risks for the enterprises and federal housing finance entities. The enterprises conduct financial and operational reviews of their counterparties in accordance with FHFA guidance. But, as we reported in 2016, FHFA does not have the authority to independently evaluate the safety and soundness of entities that conduct business with the enterprises. In 2014, the FHFA Office of Inspector General found that nonbank lenders may have limited financial capacity and are not subject to federal safety and soundness oversight, creating an increased risk that these counterparties could default on their financial obligations. They also found that rapid business growth among specialty servicers could put stress on their operational capacity or overrun their quality control procedures, potentially increasing representation and warranty claims and credit losses on mortgages they sell to the enterprises. Representation and warranty claims allow the enterprises and other federal entities to recover some losses from lenders in the event of misrepresentation by the seller. From 2009 through 2013, the enterprises received $98.5 billion through repurchase requests to sellers (that is, they required sellers to repurchase the enterprises’ interests in the loans). According to the FHFA Office of Inspector General, due to lower capital levels, nonbanks may be less able to honor these representation and warranty commitments. FHFA and HUD officials also told us nonbanks have helped increase servicing capacity. We previously reported that nonbank servicers provide benefits to the housing market through increased capacity to service delinquent loans and contribute to liquidity by broadening participation in the market for mortgage servicing rights. In particular, larger numbers of individual servicers also can reduce market concentration, suggesting that servicers may be more likely to behave competitively and can, for instance, increase innovation. Furthermore, large nonbanks are generally not as interconnected with the financial system as large banks, potentially limiting broader market effects in the event of the failure of a single large nonbank servicer. But the enterprises and Ginnie Mae likely would incur costs in the event of a failure of a large nonbank servicer whose portfolio cannot be easily absorbed by others. Mortgage servicers must continue making payments to investors when borrowers do not make payments. For mortgages backed by the enterprises, servicers can be reimbursed for principal and interest and certain other expenses, but they must finance them in the interim. Servicers of mortgage pools guaranteed by the enterprises must advance payments until the borrower is 120 days delinquent on the loan. Servicers of Ginnie Mae-guaranteed pools are not limited in how long they must advance principal and interest on delinquent loans, and they additionally may be required to absorb losses not covered by FHA insurance or VA guarantees. In the event of a failure of a large nonbank servicer with a not readily absorbable portfolio, Ginnie Mae and the enterprises likely would bear most of the associated costs, and consumers also likely would see some effects, such as service interruptions. In 2015, FHFA and Ginnie Mae raised their minimum financial eligibility requirements for sellers and servicers (including for net worth, capital ratio, and liquidity criteria for counterparties), but these requirements may not fully account for the high interest rate and default risks that nonbanks face. Challenges related to affordable housing and access to homeownership also remain. Fannie Mae and Freddie Mac are subject to affordable housing goals for their purchases of single-family and multifamily mortgages that benefit families with lower incomes. However, a number of factors affect the development of affordable housing and access to homeownership. For example, according to a 2018 study on the state of the nation’s housing, competition for the historically low supply of existing homes on the market has pushed up home prices in most metropolitan areas, raising concerns about affordability. The study also noted that although better housing quality accounts for some of the increase in housing prices, sharply higher costs for building materials and labor, among other factors, have made housing construction considerably more expensive. Land prices also increased as population growth in metropolitan areas increased demand for well-located sites. Along with rising housing costs, the study also reported that weak income growth among low- and moderate-income households contributed to affordability pressures. As homeownership becomes less affordable with house price increases, the enterprises’ affordable housing goals become more difficult to achieve. For example, for calendar year 2016, Freddie Mac met all of its affordable housing goals, and Fannie Mae met most of its affordable housing goals, but failed to meet its goal for the single-family home purchase, very-low income category. For calendar year 2017, based on FHFA’s preliminary determinations, Fannie Mae met all of its affordable housing goals, but Freddie Mac missed its single-family home purchase goals for both the very low-income and low-income categories. Experts and stakeholders we interviewed identified other contributing challenges. For example, a few experts and stakeholders cited lower levels of lending in minority communities and to low- and moderate- income borrowers, which are typically most in need of affordable housing, as contributing challenges. A few other experts and stakeholders stated that borrowers increasingly have been holding other types of debt, such as student loan debt, which makes it more difficult for them to obtain an affordable mortgage. Lastly, the qualified mortgage rule exception, which may have helped some borrowers with a debt-to-income ratio above 43 percent to obtain a mortgage, expires in 2021 or earlier if conservatorship of the enterprises ends before then. When this happens, this could also hinder the ability of certain borrowers with a debt-to-income ratio higher than 43 percent to obtain mortgages. Fannie Mae and Freddie Mac have taken actions in recent years that could further increase the scope of their activities and present challenges or barriers to entry for other market participants. Both enterprises have recently introduced pilot programs that affect mortgage insurance decisions and terms typically made by lenders. In 2018, Fannie Mae introduced a pilot program to offer an enterprise-paid mortgage insurance option—an alternative to the borrower-paid and lender-paid options currently available. Under the program’s structure, Fannie Mae is the entity responsible for purchasing mortgage insurance on loans with high loan-to-value ratios. To do so, Fannie Mae secures an insurance arrangement from a qualified insurer, which in turn transfers the risk to a panel of approved reinsurers. Fannie Mae pays the mortgage insurance premiums, while the lender is responsible for paying an additional, loan-level price adjustment. Freddie Mac launched a similar pilot program earlier in 2018 known as the Integrated Mortgage Insurance program. Under this program, simultaneous with purchasing single-family mortgages, Freddie Mac purchases mortgage insurance from a panel of pre-approved reinsurance companies that it has allocated risk among. In addition, the reinsurers post collateral to provide further assurance that claims will be paid, and they cannot deny or rescind coverage. According to Fannie Mae and Freddie Mac documents, these pilot programs allow the enterprises to better manage their counterparty risk and streamline the operational requirements of participating lenders. For example, each participating reinsurer undergoes a thorough counterparty review in order to be approved for participation in the programs. Additionally, under the programs, lenders are not required to purchase mortgage insurance for loans with loan-to-value ratios above 80 percent, which would simplify the process of selling loans to the enterprises. However, according to several experts and stakeholders with whom we spoke, by allowing the enterprises to play a role in selecting the mortgage insurer, these pilot programs widen the scope of activities of the enterprises. They also allow them to become more dominant by potentially growing their role beyond the secondary market and into the primary market. They explained that these programs promote greater vertical integration of private-sector activities into the enterprises, and create challenges for market participants. For example, they stated that they promote an uneven playing field in the private market by allowing for different terms and standards for enterprise-paid mortgage insurance versus other sources of private capital. Experts and stakeholders also identified other enterprise pilot programs or activities, such as Freddie Mac’s financing of nonbank mortgage servicers and the enterprises’ standardization efforts, as potential challenges. Freddie Mac’s Mortgage Servicing Rights pilot program provides financing to nonbank servicers, with some limitations, secured by the servicers’ mortgage servicing rights. The program is intended to address impediments nonbank mortgage servicers face in obtaining financing and extends credit to nonbank mortgage servicers when they need access to cash. However several experts and stakeholders with whom we spoke stated that this could lead to certain servicers having a competitive advantage. For example, they stated that under this program, Freddie Mac may target its financing at the biggest servicers and charge comparatively low interest rates, putting small lenders and servicers at a disadvantage. The enterprises also have efforts to standardize appraisal data, loan applications, and closing disclosures. While these efforts are intended to streamline and standardize aspects of the mortgage process, several experts and stakeholders explained that the results of these activities can be costly to smaller lenders and servicers who have to bear the costs of adapting their systems to enterprise requirements. They also indicated that participants in the primary market have become reliant on the enterprises for standards and innovation. Several experts and stakeholders also stated that the cost for market participants to adopt new programs or standards set by the enterprises can be high and could inhibit other participants from entering the housing finance market. In addition, the enterprises are currently developing a common securitization platform to support the issuance of a common single mortgage-backed security by both enterprises. The platform will support the enterprises’ single-family mortgage securitization activities, including issuance by both enterprises of a common mortgage-backed security to be known as the uniform mortgage-backed security. FHFA expects the issuance of the uniform mortgage-backed security to improve the overall liquidity of the enterprises’ securities and promote liquidity of the nation’s housing finance markets. The common securitization platform also would integrate the various securitization infrastructure systems within each enterprise, which is expected to lower costs and increase efficiency. However, several stakeholders we interviewed explained that the platform presents concerns. For example, mortgage securitizers and investor stakeholders who participated on our panels expressed concern about the platform and its availability to other market participants. Specifically, they stated that the goal of the project has, at times, been unclear and that it has been difficult to tell to what extent or when the platform will be accessible to other secondary market participants. They also stated that if the platform would not be accessible to other secondary market participants, it would take away opportunities from participants willing and able to pool eligible securities. FHFA officials told us the platform currently is intended for use only by Fannie Mae and Freddie Mac, but that the agency is aware that potential reforms to the housing finance system may bring about the inclusion of other guarantors. As such, the platform is being designed to be adaptable for use by other participants in the secondary market in the future. (We discuss recent proposals to reform the housing finance system in detail later in this report.) FHFA-directed actions (including retained mortgage portfolio reductions, credit risk transfer, and foreclosure prevention) have improved the condition of the enterprises by mitigating some of the enterprises’ exposures to potential losses. Treasury’s remaining funding commitment through the senior preferred stock purchase agreements leaves taxpayers exposed to risk, especially in the event of adverse market or other external conditions and considering the recent growth in the enterprises’ guarantee business. Total MBS outstanding guaranteed by the enterprises and held by external investors has increased each year since 2012. As of the end of 2017, the enterprises’ combined MBS outstanding held by external investors peaked at $4.8 trillion (see fig. 12). Under the terms of the senior preferred stock purchase agreements with Treasury, Fannie Mae and Freddie Mac do not maintain a capital cushion—as a private financial institution would—to guard against the risk of unexpected losses such as those that might occur during a recession or downturn in the housing market. Instead, Treasury, through taxpayer funds, committed $445.6 billion of financial support to the enterprises. As of August 2018, Treasury had provided the enterprises with $191.4 billion of the total amount since they were placed under conservatorship in 2008, leaving $254.1 billion in potential taxpayer exposure should Treasury need to provide additional support. In return, the enterprises must pay to Treasury as dividends all of their quarterly positive net worth amount (if any) over $3 billion. Thus, any losses on this amount not recovered through loss-mitigation efforts or covered by private investors or insurers would be borne by taxpayers through additional financial support from Treasury. While private institutions could absorb a share of losses on mortgages covered by credit risk transfer and private mortgage insurance (discussed earlier in this section), any additional losses would come from Treasury’s remaining funding commitment through the senior preferred stock purchase agreements. Because of this arrangement, credit rating agencies have linked the enterprises’ strong long-term credit ratings directly to that of the U.S. government and their equity to Treasury’s remaining funding commitment. Since the second quarter of 2012, Fannie Mae and Freddie Mac have not required additional support from Treasury, with the exception of the first quarter of 2018, when both enterprises required Treasury support due to devaluation of their deferred tax assets as a result of changes to the tax code. As of the end of September 2018, the enterprises had cumulatively returned $285.8 billion to Treasury through senior preferred stock agreement dividend payments. However, in addition to economic circumstances, changes in market conditions or other external factors— such as changes in interest rates, house prices, accounting standards, or events such as natural disasters—could lead to volatility in the enterprises’ quarterly financial results, potentially requiring additional taxpayer support. The extended duration of the conservatorships continues to create uncertainty about the goals and future role of the enterprises. We previously reported that FHFA’s priorities can shift, sometimes due to changes in leadership. For example, FHFA initially outlined its understanding of its conservatorship obligations and how it planned to fulfill those obligations in a 2010 letter to Congress. In February 2012, FHFA sent Congress a strategic plan that set three strategic goals for conservatorship and elaborated on how FHFA planned to meet its conservatorship obligations. However, under a new Director in 2014, FHFA issued an updated strategic plan that reformulated its three strategic goals. This same Director’s term expired in early January 2019, and the process is underway for a new, permanent Director to be confirmed. The upcoming change in leadership could shift priorities for the conservatorships again and change enterprise goals. Continuing conservatorship also presents challenges to FHFA, as it has to balance its role as conservator with its role as regulator. FHFA must follow the mandates assigned to it by statute and the missions assigned to the enterprises by their charter. This entails consistently balancing governing of the enterprises, ensuring they employ sound risk-management practices, and ensuring they continue to serve as a reliable source of liquidity and funding for housing finance. In our interviews with experts and stakeholders, at least one expert or stakeholder from each of the groups (mortgage originators, mortgage securitizers and investors, academics and researchers, and consumer advocates) also identified the duration of the conservatorships as a challenge. For example, they said that the duration of the conservatorship has led to a more substantial role for the enterprises than envisioned when they were placed under conservatorship, which could make potential changes to their structure more difficult to implement. The duration of the conservatorships also has led to uncertainties in the housing finance market. As we previously reported, under conservatorship, the enterprises are subject to agency policy decisions and are insulated from competition and other market forces. As a result, according to several mortgage originators and securitizers, and consumer groups with which we spoke, uncertainty about the future of the enterprises also makes it challenging for them to develop their own strategic plans and goals. They explained that they hesitate to make longer-term strategic plans and goals due to potential housing finance reform changes, particularly to the enterprises, that could markedly affect their industries. Additionally, the dominant role of the federal government in guaranteeing MBS since the crisis has continued, and private capital generally has not been positioned to absorb losses in the secondary mortgage market during a potential economic downturn. The current structure of the secondary mortgage market will continue to leave taxpayers at risk to potential losses. The significant federal role in the housing market likely will continue if the enterprises remain under conservatorship and without a defined future role. We assessed 14 proposals for housing finance reform against our framework to assess potential changes to the housing finance system. The framework consists of nine elements we determined to be critically important, such as recognition and control of federal fiscal exposure, protections for investors and borrowers, and clear goals (see the Background for more information). We found that the proposals generally aim to manage fiscal exposure—the risk the housing finance system poses to the federal government and taxpayers—but only six have clear goals and only seven consider other federal housing finance entities, such as FHA or Ginnie Mae, in addition to the enterprises. Reform proposals we reviewed generally fit into four different models: (1) reconstituted enterprises, (2) multiple guarantor, (3) government corporation, or (4) privatization (termination of the enterprises). Based on our review of the proposals, relevant literature, and expert interviews, each model has potential strengths and limitations. Four proposals we reviewed call for the enterprises to be recapitalized and then released from conservatorship, retaining their federal charters. Under these proposals, the enterprises would be regulated by an independent regulator that would oversee their safety and soundness. These proposals also recommend a federal guarantee on MBS under the senior preferred stock purchase agreement or by legislation. To mitigate fiscal exposure from the enterprises, the proposals include the continuation of credit risk transfer programs, and also require the enterprises to have risk-based capital reserves. In its report analyzing alternative housing finance market structures, CBO reported that under this model, taxpayers would have a higher exposure to risk compared with the multiple-guarantor and privatization models. According to industry stakeholders, potential strengths of this model include feasibility, minimal market disruption, and the continuation of policies familiar to key stakeholders. For example, one proposal argues that its reforms could be completed under existing legal authority, with no new legislation required. Five primary market stakeholders in our panels also stated that they would prefer a system similar to the current model with minor reforms because larger changes might disrupt the market and have unforeseen consequences. Industry stakeholders that rely on specific policies of the enterprises also generally support a recapitalization and release model. For example, four associations of small lenders have released statements in support of reconstituting the enterprises to ensure the continuation of the cash window. In addition, groups that advocate for financial inclusion and civil rights also have expressed support for reconstituting the enterprises to ensure the continuation of the affordable housing goals and other policies to help low-income borrowers. However, this model may not include sufficient safeguards to mitigate the risk that the enterprises—even in a reconstituted form—could pose to the stability of the mortgage market. As previously discussed, as of 2017, the enterprises issue more than half of new MBS and, in our panels, two participants from industry groups criticized the enterprises for their expansion into other areas of the housing market. In 2018, a former FHFA director stated in Congressional testimony that the enterprises were more entrenched in the market than ever before, the market depended entirely on them, and any weaknesses in their risk management could disrupt the entire housing market. If the enterprises were recapitalized without sufficient safeguards, shareholders again might have incentive to take on excessive risk. To mitigate these concerns, two of the four proposals recommend that the reconstituted enterprises operate as utilities. Utilities have a regulated rate of return, which supporters say would limit profit-maximizing motivations and encourage more prudential behavior and underwriting standards. The utility model is traditionally used in industries that tend to operate as monopolies or near monopolies, such as the electric power industry. Some industry experts believe that the securitization market operates similarly to a monopoly. Three industry stakeholders, two researchers, and one participant from a consumer protection group we interviewed also supported restructuring the enterprises as utilities. Additionally, three industry groups representing small lenders endorsed turning the enterprises into utilities. Six of the proposals we reviewed recommend transitioning to a system with multiple guarantors operating in the secondary market. Under this model, multiple private-sector firms would purchase eligible mortgages and aggregate them into MBS. The MBS would be eligible for an explicit federal guarantee if the guarantor arranged for private credit enhancements to absorb a certain amount of loss and if it met certain regulatory criteria, such as securitizing mortgages that comply with all qualified mortgage standards. A federal agency—FHFA or a successor— would charge and collect guarantee fees from the guarantors and set capital requirements. The six proposals use the guarantee fees to fund a mortgage insurance fund that would provide the federal guarantee. According to CBO’s analysis, under this model, taxpayers would have less exposure to risk compared with models for reconstituted enterprises or a government corporation, but more than under a fully private market. Proposals within this model vary in a few key ways: Enterprises: Four of the six proposals call for the enterprises to become guarantors in the new system, while two call for them to be put into receivership and replaced with successor entities. One of the proposals that would keep the enterprises recommends that they and other guarantors operate as utilities and another suggests the enterprises remain in the new system but transition to be mutually owned by lenders instead of shareholders. Securitization: Three of the six proposals would retain the common securitization platform, while one proposal would rely on Ginnie Mae- approved issuers, allowing them to issue securities including mortgages that obtained credit enhancement from a private guarantor (instead of just federal programs). One proposal that retains the common securitization platform would convert the platform into a government corporation that issues securities from any regulator- approved entity. The fifth proposal would rely on both Ginnie Mae issuers and the common securitization platform to issue securities. The sixth proposal does not specify an entity to issue securities. Number of guarantors: Proposals vary in the number of guarantors needed in the new system. For example, the Mortgage Bankers Association’s proposal suggests having more than two guarantors, while Moody’s Chief Economist said in a congressional testimony that from five to seven would be feasible (using the private mortgage insurance industry as a guide). The potential strengths of this model include the benefits arising from competition and replacing reliance on two large firms with multiple smaller guarantors. The Mortgage Bankers Association’s proposal stated that, while subject to strong regulations, guarantors can compete on price, products, and service. Multiple guarantors could provide lenders with a variety of options to sell their loans, instead of just the enterprises. More competition also could encourage innovation in the secondary market. The secondary mortgage market also might reduce its reliance on two “too-big-to-fail” entities with multiple guarantors. Because credit risk would be more dispersed across a number of entities, the failure of one firm would be less likely to disrupt the broader system, thus reducing the likelihood the government would have to rescue a struggling firm. According to four representatives of investor groups and a former HUD official we interviewed, a potential limitation of the multiple-guarantor model is that it could be difficult for new firms to enter the market and compete with the enterprises. One proposal addresses this concern by terminating the enterprises. However, if there are only a few guarantors, the failure of any one firm could pose a systemic risk and might require federal assistance. In addition, two researchers we interviewed said that because the guarantors would operate in the same market and thus would face the same market trends, having multiple guarantors might not diversify risk. For example, in a financial crisis, it is possible that all the guarantors would struggle and in such a scenario, the government would have to assist many firms. Some industry experts expressed concern that competition could have negative consequences. We previously reported that leading up to the financial crisis, the enterprises faced new competition from private-label securitizers and, in the absence of strong federal oversight, they relaxed their underwriting standards to regain market share. Thus, two researchers, four primary market stakeholders, and a former HUD official we interviewed warned that a system dependent on competing entities could face similar risks, particularly if oversight and regulation were not strong. Five of the proposals we reviewed would require all guaranteed securitized mortgages to meet qualified mortgage standards, limiting potential reductions in underwriting standards, and one of the five also would address this concern by regulating the guarantors as utilities. Two proposals would replace Fannie Mae and Freddie Mac with a single government corporation that would issue MBS. For example, in one proposal we reviewed, lenders would sell loans meeting certain requirements (such as qualified mortgages) to the corporation, which would operate the common securitization platform to issue MBS with a federal guarantee. The government corporation would manage fiscal exposure by transferring credit risk to the private sector and through capital requirements set by an independent regulator. The two proposals also would use guarantee fees to fund a mortgage insurance fund that would add an additional level of taxpayer protection. Based on its analysis, CBO reported that under this model, taxpayers would have more exposure to financial risk than under the multiple guarantor or privatization models. The potential benefits of a government corporation include stable lending during financial crises, equitable lender access, and better targeting of underserved groups. According to CBO, a government agency is more likely than private actors to promote stable mortgage lending during financial crises due to federal support. Additionally, according to a proposal by a think tank, a government corporation could provide lenders of all sizes with equal access to securitization, potentially reducing barriers to entry for new firms in the primary market. We previously reported that compared with other models, a government corporation would be well-positioned to facilitate lending to targeted groups because it does not have potentially conflicting priorities, such as maximizing shareholder value. Finally, a key benefit of creating a government corporation would be to mitigate the potential challenges posed by relying on private-sector entities (reconstituted enterprises, multiple guarantors, or a fully private market). For example, we previously reported that as for-profit corporations with government sponsorship, the enterprises had an incentive to engage in potentially profitable but risky business practices, in part because of the perception of an implied federal guarantee. In contrast, a government corporation would not be motivated by profit and thus should have less incentive to engage in potentially risky actions. The government corporation also could end reliance on a few large private firms by transferring securitization to a single entity in the public sector. There are potential limitations to relying on a government agency to support the secondary market. According to CBO, under this model, the government would still retain most credit risk and thus originators might not have a strong incentive to thoroughly vet borrowers’ credit risk, which could lead to potential losses. We also reported in 2009 that because of the limitations on government entities relative to private firms, a government corporation might have more difficulty in attracting and retaining capable staff, responding to market developments, or promoting innovation. If unaddressed, these issues could pose safety and soundness concerns because the agency might not have the skills and capabilities to assess risks and manage a complex industry. Two proposals we reviewed would terminate the enterprises and completely privatize the housing finance industry, with no federal guarantee on MBS. Under these proposals, the enterprises’ charters would be revoked and the enterprises would be wound down over a multiyear transition period during which their guarantee fees would increase and their loan limits decrease until they no longer guaranteed new mortgages. One proposal would keep the common securitization platform and make it available to all market participants, but it would operate as a nongovernmental entity and would be prohibited from guaranteeing MBS. The main benefit of this model would be to minimize fiscal exposure by having private firms form the secondary market for mortgages that are not federally insured, similar to the private-label MBS sector before the crisis. CBO noted that private actors should have a stronger incentive to control lending risk without a government backstop. Additionally, if a number of firms replaced the enterprises, then a largely private market likely would reduce the systemic risk of relying on a few large firms. However, a fully privatized market has some potential limitations related to an implied federal guarantee, and credit availability. CBO reported that although taxpayers’ would have less explicit exposure to risk compared to the other models, risk exposure could be very high even without an explicit guarantee. That is, the government likely would assist or prevent the failure of private firms in an economic downturn to ensure financial stability (also known as an implicit federal guarantee). We previously reported that private-sector actors may benefit from an implicit guarantee and this may incentivize firms to engage in potentially risky actions and expose the government to potential losses. Additionally, privatizing the market could increase fiscal exposure through FHA. The CBO report noted that a privatized model could reduce the availability of credit to marginal borrowers, and predicted it would lead to a large increase in FHA-insured loans. A largely private market also might not sustain mortgage lending during periods of economic stress. For example, the private-label market largely disappeared after the 2007–2009 financial crisis and has yet to recover, as previously discussed. Finally, CBO reported that during a financial crisis, there could be large increases in mortgage interest rates, large declines in house prices, and limited availability of 30-year fixed-rate mortgages. The 14 proposals we reviewed generally meet the following elements of our housing finance reform framework: recognizing and controlling federal fiscal exposure, protecting mortgage investors, adhering to an appropriate regulatory framework with government entities that have the capacity to manage risks, emphasizing the implications of the transition to a new housing finance system, protecting mortgage borrowers and addressing market barriers, and considering market cyclicality and impacts on financial stability. Legislative proposals and those from other sources generally address these elements in similar ways. Every reform proposal we reviewed attempts to recognize and control federal fiscal exposure—the risk that the federal government and taxpayers will have to provide financial support to the housing finance system. Twelve of the 14 proposals we reviewed support an explicit government guarantee on MBS. Some supporters of a federal guarantee maintain that if the government were to support the mortgage industry in a crisis, then such support should be explicit, which will allow it to be priced and reflected in the federal budget. In addition, every expert with whom we spoke—including industry stakeholders, consumer advocates, researchers, and former agency officials—supported an explicit government guarantee on MBS. In 11 proposals, the federal guarantee would be administered through a mortgage insurance fund managed by a federal regulator and funded through guarantee fees. To manage and limit fiscal exposure, the 12 proposals structure the federal guarantee so that it would only be accessed after a certain amount of private-sector loss. Private capital would be introduced through increased, risk-based capital requirements for the enterprises, successor entities, or new market entrants (such as guarantors). The proposals also would continue to transfer credit risk to the private sector. These proposals vary in how much private capital would be required ahead of the government guarantee. For example, one proposal would require 10 percent but another proposal would require 5 percent. In its proposed rule for enterprise capital requirements, FHFA reported that capital reserves of about 5.5 percent would have covered the enterprises’ losses during the financial crisis. However, according to CBO, the initial increases in capital requirements could increase mortgage interest rates. The two proposals without an explicit federal guarantee aim to address fiscal exposure by eliminating the enterprises and relying entirely on the private sector. However, some industry experts have asserted that there likely will always be an implied federal guarantee for the housing finance market (even without the enterprises) as the federal government will not allow the market to fail. These experts stated that they believe that this guarantee should be explicitly recognized and accounted for in the federal budget. Thirteen of 14 proposals fully meet the element of providing protections for mortgage securities investors. We previously reported that investors need to receive consistent, useful information to assess risks. We also reported that prior to the crisis, MBS investors may have lacked reliable information to accurately assess the credit risk of their investments. Twelve reform proposals we reviewed attempt to remedy these weaknesses by first providing an explicit federal guarantee on MBS. In a 2017 testimony, a former FHFA Director said that a federal guarantee signaled to MBS investors that they were protected from credit risk and a meaningful segment of investors would not continue to invest in this market without the guarantee. In addition to the federal guarantee, proposals would aim to protect investors in the following ways: Increased transparency: Proposals recommend providing investors with more information on the mortgages underlying MBS. If investors had more information about asset quality, it would help them to more accurately price risk. For example, one proposal would require market participants to make available to investors all documents (including servicing reports) related to the mortgage loans collateralizing the security. Standard securitization platform: Currently, the enterprises each have their own platforms to issue MBS and different rules governing their MBS. To improve investor protections, FHFA and others recommend a standard platform for issuing securities. As previously discussed, FHFA has been developing such a platform, which will result in a both enterprises issuing a uniform security. Twelve of 14 proposals emphasize an appropriate regulatory framework with federal regulators that have the capacity to manage risk. Proposals generally recommend that an independent federal agency, such as FHFA or a successor, regulate housing finance market participants. The regulator also may oversee the securitization platform. Three proposals that would expand Ginnie Mae recommend that Ginnie Mae become an independent agency to strengthen its counterparty oversight capabilities. The proposals also generally recommend that the regulator have risk- management capabilities to determine market participants’ capital requirements. The regulator also would be able to adjust these and other requirements, such as credit risk transfer targets, based on market circumstances. In 11 proposals, the regulator would set and collect guarantee fees and use these fees to create a fund for mortgage insurance that would act as the federal guarantee on MBS. However, we previously noted that federal agencies sometimes have faced challenges in accurately pricing risk in other insurance programs, such as deposit or flood insurance. Eleven of the 14 proposals we reviewed fully consider the implications of transitioning to a new system and mitigating potential disruptions. Because transitioning to a new system could disrupt market operations and consumers’ access to mortgage credit, we previously noted the importance of a deliberate, well-defined transition. In our expert panels, participants from investor groups noted that unless there is a clear transition plan (particularly one that addresses any changes to the enterprises), it would be difficult for new market entrants and investors to plan accordingly. The 11 proposals that meet this element include multiyear transitions to help minimize disruption. For example, one proposal that would eliminate the enterprises would allow for a 10-year transition to a new fully privatized system and create a temporary federal entity to oversee the transition. Five primary market stakeholders and a representative from a consumer advocacy group we interviewed emphasized the importance of minimizing market disruption and maintaining market liquidity. These industry experts noted that some parts of the system currently work well, and these aspects should be maintained and transitioned in reform. The 11 proposals would transition the enterprises to the new market structure or transition their personnel and facilities to successor entities. One proposal that does not meet this element does not discuss transition plans. Two other proposals do not fully meet this element because they do not address what would happen to the enterprises’ current assets, human capital, and intellectual property. Nine of 14 reform proposals explicitly address protections for mortgage borrowers. The relevant policy mechanisms to protect mortgage borrowers include maintaining CFPB’s qualified mortgage and ability-to- repay rules, as well as additional services to support borrowers. For example, one proposal would increase support for programs that help prepare renters to become homeowners. Another proposal recommends modifying servicing guidelines for nonperforming loans to ensure consumers are treated fairly and would establish consistent procedures for servicers. The five proposals that do not fully meet this element do not address it at all or do not describe specific programs or policies. Eleven of 14 proposals explicitly address barriers to accessing the mortgage market. For example, eight proposals aim to maintain access to 30-year fixed-rate mortgages, a key instrument for promoting access to homeownership. Eleven proposals would support funds dedicated to affordable housing, such as the Housing Trust Fund and Capital Magnet Fund, through fees on securitized loans. Five proposals also would collect fees for a new Market Access Fund dedicated to increasing the number of families able to achieve homeownership and access credit. However, two proposals would eliminate the Housing Trust Fund. Proposals vary in their support of the enterprises’ affordable housing goals. Eight proposals call for the affordable housing goals to be eliminated and eight industry stakeholders we interviewed doubted the effectiveness of such goals, stating that homeownership should not be addressed through the secondary market. In 2009, we reported that there was limited evidence to support the effectiveness of the enterprises’ affordable housing goals in supporting homeownership for the targeted groups. However, affordable housing and consumer advocates we interviewed stated that they want to maintain the goals because they believe that the goals improved access to credit for minority and low- to moderate-income borrowers. Regardless of their position on the affordable housing goals, we found that proposals with a federal guarantee generally would require market participants to serve all eligible borrowers in all markets to receive the guarantee. Thirteen proposals also recommend policies that would promote small lender access to the market, such as maintaining the enterprises’ cash windows or creating a similar structure in their successors. Through the cash windows, lenders can sell individual loans directly to the enterprises and retain servicing rights. According to the Center for Responsible Lending, keeping loan servicing within community-based financial institutions often results in better loan performance and customer service outcomes. One former HUD official we interviewed stated that minority communities are often served by smaller lenders and these lenders need the cash window as a way to continue making affordable loans. Ten experts in our panels—including housing advocates, primary and secondary market participants, and researchers—said that reform plans should give fair treatment to all lenders, regardless of size. Nine of 14 reform proposals fully meet the element relating to consideration of the cyclical nature of the housing finance market and its impact on financial stability. We previously reported that the housing finance market is characterized by cyclical fluctuations and its market cycles may pose risks to overall financial and economic stability because housing is a significant part of the economy. The five proposals that do not fully meet this element do not address how the reformed system would attempt to mitigate market cycles. We previously reported that financial regulatory action or inaction can exacerbate housing finance cycles, and thus reform proposals should consider the potential impact of new regulations on market cyclicality. To mitigate market cycles, the nine proposals that meet this element generally include policies that would allow the regulator to adjust regulations based on market cycles. In one proposal, the regulator would establish risk-based capital requirements for the enterprises or successor entities and could adjust the requirements temporarily based on market cycles. One proposal that would create a government corporation also would allow the corporation to maintain a small portfolio to manage distressed loans. Eight of the proposals we reviewed do not have clearly defined goals and seven do not fully consider other entities in the housing finance system— two key elements in our housing finance reform framework. Eight of 14 proposals we reviewed do not have clearly defined goals for the housing finance system, including four legislative proposals. Additionally, none of the proposals prioritize their goals. Among the six proposals with clearly defined goals, we identified some common goals, such as minimizing the risk of taxpayer-funded bailouts, supporting market liquidity, and maintaining a level playing field for lenders of all sizes. We also identified different goals among the proposals, reflecting differences in reform models. For example, proposals similar to the multiple-guarantor model explicitly include market competition as a goal, while a proposal for reconstituting the enterprises includes stable transition as a goal. As we reported in 2015, clearly defined and prioritized goals are a key element to consider when assessing changes to the housing finance system. Clear goals help guide agencies’ activities and establish accountability. Experts with whom we spoke also emphasized the importance of clearly defined goals in housing finance reform proposals. For example, one researcher said it would be difficult to discuss any necessary policy changes until the government clearly articulated goals for its role in the housing finance system. Furthermore, prioritizing goals can help guide agencies’ actions and provide clarity to market participants, particularly if there are conflicting goals. For example, three proposals we reviewed include the goals of both minimizing risks to taxpayers and promoting affordable homeownership, but there is a trade-off between these goals—promoting homeownership may mean encouraging lending to riskier borrowers. As of early January 2019, Congress had not enacted legislation that establishes clear and prioritized objectives for the future federal role in housing finance. The lack of such goals in many of the proposals we reviewed raises questions as to whether the proposals that Congress may consider in the future will give adequate attention to these critical elements of housing finance reform. Without clearly defined and prioritized goals, agencies’ housing finance activities may lack focus and consistency. We previously reported that because Congress did not provide clearly defined and prioritized goals to FHFA for conservatorship, each FHFA director has been able to shift agency priorities within statutory requirements. For instance, the first FHFA director raised guarantee fees to encourage the return of private capital to the MBS market, while the next director stopped the increase out of concern for its effect on credit availability. Additionally, we reported that FHFA’s shifting priorities for conservatorship contributed to uncertainty among market participants. Therefore, by identifying a primary objective for housing finance reform, Congress would be better positioned to determine appropriate steps and policies and provide clarity to market participants. Seven of 14 proposals we reviewed—including proposed legislation—do not consider if and how they would affect other federal entities in the housing finance system, such as FHA and Ginnie Mae. The proposals that consider other federal entities include policies to help them manage the effects of reform and ensure agencies’ policies are consistent with overarching goals. For example, proposals that would expand Ginnie Mae’s guarantee to include the enterprises’ market also recommend that Ginnie Mae become an independent agency to better manage its expanded role. Another proposal with the broad goal of reducing the federal role in the mortgage market by terminating the enterprises also aims to manage fiscal exposure through FHA by increasing its capital reserve ratio from 2 to 4 percent. Finally, one proposal with a goal of promoting market liquidity recommends that FHA should become an independent agency to buttress its countercyclical role (that is, its ability to provide credit availability across market cycles). We previously reported that aligning policies and mechanisms with goals is a key element of housing finance reform, and that reform should have a comprehensive approach that considers all relevant entities. A comprehensive approach would help to promote consistency, transparency, and reduce unnecessary overlap and duplication between the enterprises and other federal entities. As of early January 2019, Congress had not enacted legislation with a system-wide approach to housing finance reform that considers the enterprises and other federal entities. The lack of a comprehensive approach in half of the proposals we reviewed highlights the need for policymakers to consider these key elements when reforming the housing finance system. Housing finance reform that does not consider all federal entities or participants may not account for how changes in the enterprises’ activities could affect risk exposure of other federal entities. For example, CBO reported that transitioning to a fully private market likely would lead to large increases in the volumes of loans insured by FHA. Industry experts with whom we spoke—including stakeholders from the primary and secondary markets, researchers, and former agency officials—also stated that any reforms to the enterprises must consider FHA too. Thus, considering the impacts of potential reforms on other federal entities would help ensure consistency and avoid unintended consequences. The enterprises have remained in conservatorship since 2008 (over 10 years), perpetuating uncertainty about their future and the federal role in the housing finance market. Determining those future roles and the enterprises’ structures has become both more urgent and more challenging as federal fiscal exposures have grown and new risks emerged in the housing finance markets (such as the growing role of nonbank lenders and servicers). Congress and industry stakeholders have introduced a number of proposals to reform the housing finance system, including addressing the prolonged conservatorship of the enterprises, but several proposals lack clearly defined and prioritized goals or do not consider all relevant federal entities in the housing finance system. By incorporating these key elements in future reform efforts, Congress could facilitate a more focused and comprehensive transition to a new housing finance system. Moreover, reform efforts that are both focused and comprehensive could allow market participants to confidently engage in long-term planning and help increase private-sector participation in the markets. Congress should consider legislation for the future federal role in housing finance that addresses the structure of the enterprises, establishes clear, specific, and prioritized goals and considers all relevant federal entities, such as FHA and Ginnie Mae. (Matter for Consideration 1) We provided a draft of this report to FHFA, Treasury, and HUD for review and comment. FHFA provided a technical comment that we incorporated. We also received technical comments from HUD and Treasury on sections of the draft report, which we incorporated as appropriate. Further comments on the full draft report from HUD and Treasury were not available due to the partial government shutdown. We are sending copies of this report to the appropriate congressional committees and FHFA, Treasury, and HUD. This report will also be available at no charge on our website at http://www.gao.gov. Should you or your staff have questions concerning this report, please contact me at (202) 512-8678 or garciadiazd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Our objectives in this report were to examine (1) recent developments in the housing and financial markets that could affect the safety and soundness of Fannie Mae and Freddie Mac, two government-sponsored enterprises (enterprises); (2) risks and challenges that the ongoing conservatorships pose to the status and operations of the enterprises and other aspects of the housing finance system, and (3) housing finance reform options that have been proposed and their relative strengths and limitations. To examine trends in the housing market and assess related risks, we reviewed and analyzed data that we considered relevant to various aspects of risk and developments in the housing market. Specifically, we reviewed and analyzed: house prices from the Federal Housing Finance Agency (FHFA) and Standard and Poor’s (a financial services company); mortgage delinquency rates from FHFA; the Board of Governors of the Federal Reserve System (Federal Reserve); the Bureau of Consumer Financial Protection, also known as the Consumer Financial Protection Bureau (CFPB); and Inside Mortgage Finance (a housing market data provider); mortgage origination and securitization data from Inside Mortgage Finance, FHFA, the Mortgage Bankers Association, and the Securities Industry and Financial Markets Association; and measures of underwriting standards from Fannie Mae, Freddie Mac, the Department of Housing and Urban Development (HUD), and the Senior Loan Officer Opinion Survey on Bank Lending Practices (conducted by the Federal Reserve). We adjusted house prices for inflation using the Bureau of Labor Statistics’ Consumer Price Index and mortgage origination and securitization volume using the Bureau of Economic Analysis’s Implicit Price Deflator for gross domestic product to make dollar amounts reflective of real 2017 dollars. To further inform our assessment of these developments and risks, we reviewed prior GAO work on these issues. Specifically, we reviewed prior GAO work that identified and analyzed key national housing market indicators, including house prices and loan performance, since the 2007– 2009 financial crisis. To examine risks and challenges that conservatorship poses to the status of the enterprises and other aspects of the housing finance system, we reviewed FHFA reports and Fannie Mae and Freddie Mac financial statements. Specifically, we reviewed progress reports and program updates from FHFA regarding its credit risk transfer and foreclosure prevention actions, and reviewed FHFA’s scorecard progress and other FHFA reports (such as the 2017 Report to Congress), strategic plans, and FHFA Office of Inspector General reports. For financial information on Fannie Mae and Freddie Mac, we reviewed filings with the Securities and Exchange Commission, quarterly financial supplements, and reports from credit rating agencies. We also reviewed selected academic literature that reported on risks and challenges identified in these sources and the potential effectiveness of risk-mitigation efforts. We also reviewed our prior work on the enterprises’ instability during the financial crisis. We took a number of steps to assess the reliability of the data, including interviewing agency officials; corroborating trends across data from multiple sources that we analyzed for these two objectives; reviewing related documentation; and reviewing relevant, prior GAO work. We used data that had been collected for prior GAO reports and reviewed the data reliability assessments that had been completed for those reports to determine if the data were reliable for our purposes. Based on these actions, we determined the data were sufficiently reliable to report on recent trends in the housing market and developments under the conservatorships of the enterprises. To address our third objective, we reviewed 14 proposals proposed by Congress, federal agencies, industry groups, or think tanks for reforming the single-family housing finance system. We selected proposals for review based on the following criteria: Time frame: We selected proposals that were released from 2014 through 2018. Source of proposal: We selected proposals from the following sources: (1) Congress (either proposed legislation or discussion drafts by members), (2) federal agencies, and (3) industry groups or think tanks (limited to those that were discussed in congressional hearings). We excluded some proposed legislation that only would modify certain aspects of the conservatorships of the enterprises and did not contain broader reforms. For example, three proposed legislative acts would have amended the terms of the senior preferred stock purchase agreements but did not address other aspects of housing finance and thus we excluded them from our review. We also excluded documents that outlined principles and objectives for reform but did not include specific policies, such as reform principles documents that some industry and advocacy groups released. We used elements of GAO’s framework for assessing potential changes to the housing finance system to analyze the content and assess the potential strengths and limitations of the reform proposals. For each element, we defined a series of responses to determine if the proposal fully, partially, or did not meet the element and provided examples of relevant policies for each element. Generally, a proposal fully met an element if it described specific policies and programs relevant to that element, partially met an element if it the element was addressed but the proposal did not describe specific policies or programs relevant to it, or did not meet an element if it did not address it at all. We also gathered descriptive information on the policies and programs on which the proposals relied. We used the information we collected from the proposals to determine the potential strengths and limitations of the proposals. We generally considered a proposal’s strengths to be the elements it fully met and its limitations to be elements that were partially met or not met. We did not make an individual, overall determination about each proposal, but instead examined whether each proposal fully considered key elements of housing finance reform. For example, a proposal could have useful ideas for reform but had yet to consider some key elements. Using this information, we used the number of proposals that fully met each element to determine which elements were most frequently met. We noted which elements were met least often to determine the gaps in the reform proposals as a whole. We also grouped the individual proposals into the different reform models. We determined the main reform models and their potential strengths and weaknesses based on our review of the proposals, prior GAO reports, Congressional Budget Office reports, industry stakeholder reports, and information we obtained during panels and interviews we conducted. To address all three objectives, we convened four, 2-hour panels of experts and stakeholders representing (1) mortgage originators and insurers, (2) securitizers and investors, (3) consumer and affordable housing advocates, and (4) researchers. We selected the experts and stakeholders based on the extent to which they developed reform proposals, testified before Congress on housing finance reform, or had participated in prior GAO studies of housing finance issues. Each panel had from three to five participants. In cases in which key experts or stakeholders could not attend our discussion panels, we interviewed them separately. We also interviewed officials at FHFA, HUD, and the Department of the Treasury. We conducted this performance audit from March 2018 to January 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. For this report, we reviewed the following housing finance reform proposals released between 2014 and September 2018 (see appendix I for more information about how we selected the proposals): Bipartisan Housing Finance Reform Act of 2018 (discussion draft). Released by House Financial Services Chairman Jeb Hensarling, Representative John Delaney, and Representative Jim Hines on September 6, 2018. Housing Finance Reform and Taxpayer Protection Act of 2014 (S. 1217). Released by Senate Banking Committee Chairman Tim Johnson and Ranking Member Michael Crapo on March 16, 2014. Housing Opportunities Move the Economy (HOME) Forward Act of 2014 (discussion draft). Released by House Financial Services Committee Ranking Member Maxine Waters on March 27, 2014. Mortgage Finance Act of 2015 (S. 495). Introduced by Sen. Johnny Isakson on February 12, 2015. Partnership to Strengthen Homeownership Act of 2014 (H.R. 5055). Introduced by Representative John Delaney on July 10, 2014. Protecting American Taxpayers and Homeowners Act of 2018 (H.R. 6746). Introduced by House Financial Services Chairman Jeb Hensarling on September 7, 2018 (originally introduced on July 22, 2013). Bright, Michael, and Ed DeMarco. Toward a New Secondary Mortgage Market. Washington, D.C.: Milken Institute, September 2016. Federal Housing Finance Agency. Perspectives on Housing Finance Reform. Washington, D.C.: January 2018. Independent Community Bankers of America. ICBA Principles for GSE Reform and a Way Forward. Washington, D.C.: 2017. Moelis & Company LLC. Blueprint for Restoring Safety and Soundness to the GSEs. June 2017. Mortgage Bankers Association. GSE Reform: Creating a Sustainable, More Vibrant Secondary Market. Washington, D.C.: April 2017. National Association of Home Builders. Why Housing Matters: A Comprehensive Framework for Reforming the Housing Finance System. Washington, D.C.: September 2015. Office of Management and Budget. Delivering Government Solutions in the 21st Century: Reform Plan and Reorganization Recommendations. Washington, D.C.: June 2018. Parrott, Jim, Lewis Ranieri, Gene Spalding, Mark Zandi, and Barry Zigas. A More Promising Road to GSE Reform. Washington, D.C.: Urban Institute, March 2016. In addition to the contact named above, Karen Tremba (Assistant Director), Tarek Mahmassani (Analyst in Charge), Miranda Berry, M’Baye Diagne, Michael Hoffman, Risto Laboski, Melanie Magnotto, Marc Molino, Matthew Rabe, Barbara Roesmann, Jessica Sandler, and Andrew Stavisky made significant contributions to this report.", "summary": "Since 2008, the federal government has greatly increased its role in financially supporting housing markets. In September 2008, FHFA placed Fannie Mae and Freddie Mac under conservatorship, which created an explicit fiscal exposure for the federal government. As of October 2018, the dollar amounts of their outstanding MBS have grown by more than $800 billion since the end of 2008. Since 2013, GAO has designated the federal role in housing finance as a high-risk area. GAO examines (1) recent housing market developments, (2) risks and challenges posed by the current federal role, including ongoing conservatorship, and (3) housing finance reform proposals and their strengths and limitations. To address these issues, GAO reviewed housing finance data; FHFA and enterprise reports; and 14 housing finance reform proposals introduced in Congress or proposed by industry stakeholders since 2014. GAO also convened panels with housing finance experts and stakeholders (including consumer advocates, mortgage originators, insurers, and investors), who developed reform proposals, testified before Congress, or participated in prior GAO studies. Federal support of the housing finance market remains significant even though the market has largely recovered since the 2007–2009 financial crisis. While down from the peak in 2009, in 2017, the federal government directly or indirectly guaranteed about 70 percent of single-family mortgage originations. The Federal National Mortgage Association (Fannie Mae) and the Federal Home Loan Mortgage Corporation (Freddie Mac)—two government-sponsored enterprises (enterprises) that purchase and securitize mortgages into mortgage-backed securities (MBS)—securitized and guaranteed about 46 percent of mortgage originations in 2017. In 2017, federal programs, such as those offered by the Federal Housing Administration (FHA), insured about 25 percent of mortgage originations. Together, the enterprises and the Government National Mortgage Association (Ginnie Mae)—a federally owned corporation that guarantees MBS backed by federally insured mortgages—have issued or guaranteed 95 percent or more of all MBS issued annually since 2008 (see figure). However, recent market trends pose risks to these entities and the housing finance system. For example, mortgage lending standards have loosened slightly in recent years, which could increase the risk of borrower default—especially in a recession or downturn in the housing market—and losses to federal entities. Nonbanks have increased their presence in mortgage lending and servicing, which involves collecting monthly mortgage payments, among other duties. For instance, the share of nonbank originations of FHA-insured mortgages increased from 56 percent in fiscal year 2010 to 86 percent in 2017. The share of nonbank servicers of mortgages in enterprise MBS also grew from 25 percent in 2014 to 38 percent as of the third quarter of 2018. While nonbank lenders and servicers have helped provide access to mortgage credit, they are not subject to federal safety and soundness regulations. The Federal Housing Finance Agency (FHFA) has taken actions to lessen some of Fannie Mae and Freddie Mac's risk exposure. For example, under FHFA's direction, the enterprises have reduced the size of their riskier retained mortgage portfolios which hold assets that expose them to considerable interest rate and other risks from a combined $1.6 trillion in 2008 to $484 billion in 2017. Since 2013, the enterprises also have transferred increasing amounts of risk on their guaranteed MBS to private investors and insurers through credit risk transfer programs. However, federal fiscal exposure remains significant. The Department of the Treasury's remaining funding commitment through the senior preferred stock purchase agreements—which provide financial support to the enterprises—leaves taxpayers exposed to risk, especially in the event of adverse market or other conditions and given the recent growth in the enterprises' guarantee business. The value of outstanding MBS on which the enterprises guarantee principal and interest payments to investors grew from about $2.1 trillion in 2003 to about $4.8 trillion in 2017. The long duration of the conservatorships also raises uncertainty among market participants. Several experts and stakeholders GAO interviewed said that they have hesitated to make longer-term strategic plans and goals due to potential housing finance reforms that could markedly affect their industries. The figure below shows 2003–2017 trends in the enterprises' guarantee business and retained mortgage portfolios. GAO reviewed 14 housing finance reform proposals from Congress, agencies, industry groups, and think tanks. The proposals generally fit into four different models: reconstituted enterprises, a multiple guarantor system with an explicit federal guarantee, a government corporation, and a completely privatized market without an explicit federal guarantee. The 14 proposals generally meet key elements of GAO's framework for assessing potential changes to the housing finance system, such as addressing fiscal exposure, protecting investors, and considering the implications of the transition to a new system. However, many proposals lack clearly defined and prioritized goals or do not address the role of other federal entities in the housing finance system, such as FHA and Ginnie Mae—two key elements in GAO's framework. By incorporating these elements, policymakers could facilitate a more focused and comprehensive transition to a new housing finance system and provide greater certainty to market participants. Congress should consider legislation for the future federal role in housing finance that addresses the structure of the enterprises, establishes clear and prioritized goals, and considers all relevant federal entities, such as FHA and Ginnie Mae.", "document_type": "gao"}
{"report": "Many older Americans are retired and rely on different parts of the U.S. retirement system for their financial security. The U.S. retirement system is often described as being composed of Social Security, employer- sponsored pensions and retirement savings plans, and individual savings. In addition, older Americans may work past traditional retirement ages or phase into retirement. Social Security’s Old-Age and Survivors Insurance program is the foundation of the U.S. retirement system and provides benefits to retired workers, their families, and survivors of deceased workers. In 2018, about 53 million retirees and their families received $844.9 billion in Social Security retirement benefits, according to the Social Security Administration. However, Social Security is facing financial difficulties that, if not addressed, will affect its long-term stability. If no changes are made, current projections indicate that by 2034, the retirement program Trust Fund will only be sufficient to pay 77 percent of scheduled benefits. Employer-sponsored pensions include DB plans, which generally promise to offer a monthly payment to retirees for life. Employers also sponsor defined contribution (DC) plans, such as 401(k)s, in which individuals accumulate tax-advantaged retirement savings in an individual account based on employee and/or employer contributions, and the investment returns (gains and losses) earned on the account. Participants in both DB and DC plans receive certain tax preferences provided the plans comply with requirements outlined in the Internal Revenue Code (IRC). For fiscal year 2018, estimated tax expenditures related to retirement plans and savings amounted to about $188 billion. The Employee Retirement Income Security Act of 1974 (ERISA) outlines minimum standards and requirements that must be met by most private sector employer- sponsored retirement plans; it does not, however, require any employer to establish, or continue to maintain, a retirement plan. Assets rolled over from employer-sponsored DC plans when individuals change jobs or retire are the primary source of funding for individual retirement accounts (IRAs). Over the past 40 years, private sector employers have increasingly moved from offering DB plans to offering DC plans. While DC plans offer more portability, some financial risks—such as poor investment returns, decreases in interest rates, and increases in longevity—have shifted from the employer to the employee, with important implications for individuals’ retirement planning and security. Individual savings are any other non-retirement plan savings and investments. Home equity is an important asset for many households. Other sources of savings or wealth may include amounts saved from income or wages, contributions to accounts outside of a retirement plan, non-retirement financial wealth that is inherited or accumulated over time, and equity from other tangible assets such as vehicles. Wealth: For analyses in this report, we defined wealth as net worth, i.e., assets minus debt. Assets could be financial (e.g., savings accounts, stocks, bonds, retirement accounts) or nonfinancial (e.g., the value of any houses or vehicles). Retirement accounts include defined contribution plans, such as a 401(k), or individual retirement account (IRA)s. Net worth is a measure often used by researchers studying retirement security. Present value of future income from Social Security and defined benefit pensions: Older Americans may also have other future retirement resources, not included in net worth, such as the present value of benefits expected from defined benefit (DB) pension plans and Social Security. These present value estimates could be included in a broader definition of economic resources or wealth, and we were able to produce estimates of these additional retirement resources to supplement our analysis of the distribution of income and wealth among older Americans over time. While all estimates produced using survey data are subject to some uncertainty, our present value estimates for these additional retirement resources are also subject to additional uncertainty that arises from using another data source—the Financial Accounts of the United States—to create a measure of aggregate defined benefit entitlements; having limited information about lifetime earnings in the Survey of Consumer Finances; and making assumptions about life expectancy, real discount rates, and retirement ages, which are unlikely to hold for all households. Data limitations prevented us from producing this broader measure of retirement resources for our analysis examining the distributions of income and wealth as a cohort of older Americans aged. Income: For analyses in this report, we defined household income as the sum of income across all sources, including wages and salaries, Social Security benefits, traditional pension benefits from defined benefit plans, withdrawals from retirement accounts, and income from any other sources, such as interest on financial assets or benefits from social safety net programs such as the Supplemental Nutrition Assistance Program (SNAP). See appendix I for more information on our definitions and the methods used to produce estimates of wealth, the present value of future income expected from Social Security and defined benefit plans, and income. Older Americans may also have wages or salaries from working longer as they transition to retirement. According to data from the Bureau of Labor Statistics, more older Americans are working. From 1989—the earliest starting year for our analyses—to 2018, the labor force participation rate for Americans aged 55 or older increased from 30 percent to 40 percent. In addition, some older Americans may receive income from financial assets, such as interest or dividends, and from other benefit programs, such as Social Security Disability Insurance. The number of older Americans is increasing faster than the population as a whole. In 1990, about 52 million, or around 1 in 5, people in the United States were aged 55 or older. By 2030, that number is expected to be about 112 million, or around 1 in 3. The aging of the baby boomers— that is, people born between 1946 and 1964—as well as increasing longevity and lower fertility have contributed to this trend. The oldest baby boomers turned 55 in 2001 and the youngest are turning 55 this year. In addition, average life expectancy for those ages 65 or older has increased significantly over the past century and is projected to continue to increase. For example, a man turning 65 in 2030 is expected to live, on average, to age 85.0, an additional 5.3 years compared to a man who turned 65 in 1980, who was expected to live, on average, to age 79.7. A woman turning 65 in 2030 is expected to live, on average, to age 87.3, an additional 3.5 years compared to a woman who turned 65 in 1980, who was expected to live, on average to age 83.8. Since life expectancies are averages—some individuals will live well beyond their life expectancy— longer life expectancies, combined with the possibility of living well beyond life expectancy, mean that people must now prepare for the potential for more years in retirement with greater risk of outliving their savings. Disparities in income and wealth among older households have become greater over the past 3 decades, according to our analysis of 1989 to 2016 data from the SCF. For our analysis, we divided older households in the data into five groups, or quintiles, based on income or wealth. Each year of data in our analysis used a different set of households. Therefore, each quintile includes different sets of households over time. In other words, the households in the top 20 percent in 1989 are not the same households as those in the top 20 percent in 2016. While the households included in the SCF are different for each year of data we used in our analysis, we were able to examine how the distribution of income and wealth across older households changed over time. We found mostly higher income and wealth across all quintiles over time, disproportionately so for the top quintile. For example, we estimated that average income of households in the top 20 percent in 1989 was about $242,000. In 2016, estimated average income of households in the top 20 percent was about $398,000, which is about 64 percent higher (see fig 1). In comparison, estimated average income of households in the bottom quintile—bottom 20 percent—was about $9,000 in 1989. In 2016, estimated average income of households in the bottom 20 percent was about $14,000, which is about 55 percent higher. We found similar results when we analyzed changes in median income. Our findings were similar when we analyzed changes in wealth (defined as net worth). Estimated average wealth of households in the top 20 percent was about $2.1 million in 1989. In 2016, estimated average wealth of households in the top 20 percent was about $4.6 million, which is more than twice as high. (See fig. 2.) In comparison, average wealth of households in the bottom 20 percent was similar over time from 1989 to 2013. In fact, in both 2010 and 2013, estimated average wealth of households that were in the bottom 20 percent in either of those years was negative, meaning that those households, on average, had more debt than assets. (See text box for discussion of how recessions during the time period of our analysis could affect retirement security.) Within the top quintile, a disproportionate share of income and wealth is held by the top 1 percent compared to the next 19 percent. (See figs. 3 and 4 for average income and wealth of households in the top 1 percent.) For example, we found households in the top 1 percent in 1989 had estimated average wealth that was about $13 million more than estimated average wealth for households in the next 19 percent (about 10 times as much estimated average wealth). By 2016, households in the top 1 percent had about $34 million more in estimated average wealth compared to households in the next 19 percent (about 13 times as much estimated average wealth). Social Security is the foundation of retirement security in the United States, and along with income from traditional DB pensions, can be particularly important for older households with lower wealth. As discussed in the text box above, some older Americans will expect future income from Social Security, DB pensions or both. We analyzed the present value of these sources for two subsets of older Americans: 1) those who expect future income from Social Security but not DB pensions, and 2) those who expect future income from both Social Security and DB pensions. On average, households with lower wealth, and that expect future income from Social Security but not DB pensions, may receive a significant income stream from future Social Security benefits, according to our analysis of SCF data (see fig. 5). The bottom 20 percent have little in wealth, on average, but the estimated present value of future Social Security benefits provides them relatively significant financial security in retirement. On the other hand, for the top two quintiles, wealth was the most important retirement resource, as households in the top quintile have wealth that, on average, far exceeds the estimated present value of benefits provided by any future Social Security or pension benefits. We found similar results for households with lower wealth and that expect future income from Social Security and DB pensions. While the lower quintiles may have little in wealth, on average, they may expect to receive a significant income stream from future Social Security and DB pension benefits (see fig. 6). Wealth was the most important financial retirement resource for the top two quintiles, on average. While disparities remain, the present value of future income expected from Social Security and DB pensions mitigate these disparities to some extent for those households that expected such income, as illustrated by the examples below. Estimates for all older households in 2016 that expect future income from Social Security but not DB pensions: Households in the top quintile had, on average, about $6.1 million in assets, about 272 times as much as the bottom quintile, which had estimated assets of, on average, about $22,000. When looking at a broader definition of retirement resources (assets plus the present value of future income from Social Security), we estimated that the top quintile had, on average, $6.6 million in these resources, about 27 times as much as the bottom quintile, which had, on average, about $241,000. Estimates for all older households in 2016 that expect future income from Social Security and DB pensions: Households in the top quintile had, on average, about $3.2 million in assets, about 61 times as much in assets as the bottom quintile, which had estimated assets of, on average, about $52,000. When looking at a broader definition of retirement resources (assets plus the present value of future income from Social Security and DB pensions), we estimated that the top quintile had, on average, about $4.3 million in these resources, about 8 times as much as the bottom quintile, which had, on average, about $535,000. Recent research has theorized that benefits expected from Social Security “ a long way” to explaining why having little in DC accounts and future income expected from pensions does not necessarily translate into dramatic changes to living standards as people retire. In particular, the progressivity of Social Security, meaning Social Security benefits replace a higher percentage of pre-retirement earnings for lower-earning households, could be helpful for these households, especially in the absence of other resources, such as retirement accounts. Income and wealth were consistently lower over time for older households headed by someone who was a racial minority, single, or hadn’t attended college, according to our analysis of 1989 through 2016 SCF data. (See fig. 7 for an example using the middle quintile.) We found these disparities existed across all quintiles and all years (see fig. 8 for another example, this time using the top quintile). Generally, the largest disparities from 1989 to 2016 were between 1) households in which the head had not attended college and households in which they had and 2) coupled households and single women. These results are consistent with our prior work, which found that women age 65 and older had less retirement income, on average, and live in higher rates of poverty than men in that age group. Disparities were also sizeable for households headed by someone who was white and non-Hispanic compared to those headed by a minority. There are multiple reasons why households headed by someone with at least some college education may have more wealth in retirement. Most notably, those with more education may have access to higher-paying jobs and be able to save more. Our review of the literature identified several other theories to explain this association. These include (1) education increases awareness about the need to save, (2) highly- educated individuals may have more financial education and achieve higher rates of return on savings, (3) those with more education may be willing to work longer, and (4) highly-educated individuals may have wealthier parents and thus may have received larger bequests. Our prior work has explored how recent trends in marital patterns and saving for retirement, among other factors, can negatively affect retirement security for minorities, women, or those who are single. The percentage of households with retirement accounts was higher across all wealth quintiles in 2016 compared to 1989, and it was disproportionately higher for the top quintile, according to our analysis of SCF data. In 1989, the percentage of households with retirement accounts—amounts in DC plans and IRAs—ranged from 4 percent of the bottom quintile to 65 percent of the top quintile (see fig. 9). By 2016, 11 percent of households in the bottom quintile had retirement accounts compared to 86 percent of households in the top quintile. These increases reflect the transition to more employers offering DC plans, among other factors. Further, the percentage of households in the bottom quintile with retirement accounts had not returned to its pre- recession rate. As discussed earlier, households with less wealth may be more reliant on income from Social Security and DB plans. Further, we found the amount in retirement accounts was often low, particularly for the lower quintiles. In 2016, 89 percent of the households in the bottom quintile had no retirement accounts, and another 10 percent had account balances of less than $50,000 (see fig. 10). In comparison, over half the households in the middle quintile had retirement accounts, and almost all of these households had less than $200,000 in their accounts. Older Americans may rely on resources other than those discussed above for financial security in retirement (see fig. 11), and these “non- retirement assets” remained important over the time span of our analysis, regardless of their value relative to retirement account balances or the present value of future income from Social Security or DB pensions. Home equity. We estimated that over 80 percent of households in each of the top four quintiles of the wealth distribution owned a home in each year of our analysis. However, the home ownership rate for households in the bottom quintile in each year of our analysis was consistently much lower than for the other quintiles–ranging between 18 and 32 percent. Further, the home ownership rate for households in the bottom 20 percent in 2016 (19 percent) was significantly lower than the home ownership rate for households in the bottom 20 percent in 2007 (28 percent), the starting year for the most recent recession. In 2016, the estimated average amount of home equity of households in the bottom quintile was about $2,000, and $50,000 for the second-from-the-bottom quintile, compared to about $118,000 for the middle quintile, about $208,000 for the fourth (or second-from-the- top) quintile, and about $559,000 for the top quintile. According to researchers, most households appear to treat a house as a source of reserve wealth that can be tapped in the event of a substantial expense, further pointing to the importance of home ownership for many older Americans. Vehicles. A majority of households in each quintile of the wealth distribution owned a vehicle across all years in our analysis, although the bottom quintile had ownership rates that were disproportionately lower. However, despite this, we estimated that vehicles provided higher value, on average, relative to other non-retirement assets for households in the bottom quintile from 2010 onward. For example, in 2016, the estimated average value of vehicles among households in the bottom quintile was about $7,000 in 2016, compared to estimated average values of less than $2,000 in home equity and about $3,000 in all other non-retirement assets. All-other non-retirement assets. For the top quintile of households, the average value of these “other assets”—which included stocks, bonds, and other savings outside of retirement accounts, among other things—was more than average home equity or the average value of vehicles over the period of our analysis. Estimated average wealth in this other assets category was about $3.3 million in 2016 for the top quintile. Individual income sources and debt were also important factors in older households’ financial security. Researchers have examined the importance of income sources for households and found Social Security is more important for households with lower incomes, while older households with the most income tend to have a diverse range of income sources, such as earnings from financial assets and income from DB plans. We found that debt could have a substantial effect on households’ financial security, particularly for the bottom 20 percent. For example, in 2010 and 2013, average net worth for this group was negative because debt was greater than assets. A substantial number of older Americans born from 1931 through 1941 lived into at least their 70s or early 80s, according to our analysis of data on a cohort of people born in these years. (See text box and app. I for more on how we analyzed Health and Retirement Study (HRS) data on this cohort.) However, this same cohort faced disparities in longevity. Further, our analysis, as well as that of other researchers, found income and wealth each have strong associations with longevity, as do certain demographic characteristics, such as gender and race. However, even among those with multiple factors associated with a shorter life, such as having lower mid-career earnings and not having attended college, a significant proportion from our cohort were alive in 2014, when they were in their 70s or early 80s. Taken all together, individuals may live a long time, even individuals with factors associated with lower longevity, such as low income or education. Those who live a long time and have little or nothing in DC account balances or pension benefits may have to rely primarily on Social Security or safety net programs. Analyzing Income, Wealth and Longevity We examined the association of income and wealth with longevity in a nationally representative sample of Americans born from 1931 through 1941. Throughout this analysis, our references to “older Americans” and “households” apply to that specific subset of older Americans born from 1931 through 1941 and their households. The Health and Retirement Study (HRS) began in 1992 and first surveyed these individuals when they were 51 to 61 years old. The same individuals have been re-interviewed every 2 years since, provided they continued to participate in the survey, and the most recent complete data is from 2014, when those who were still alive were 73 to 83 years old. We were able to measure deaths over a period of 22 years (1992 through 2014). Every 2 years, the HRS attempted to measure whether the original respondents were still alive, but these longevity data were incomplete because some of the original respondents declined to participate in later waves of the survey. Once these respondents left the survey, their actual longevity could not be followed. Therefore, we used survival analysis to estimate the proportion of individuals in the1992 sample alive in 2014. Survival analysis accounts for survey respondents with complete or incomplete longevity data and allowed us to estimate the chance of death by any given time in the observation period. Most importantly, our analysis assumed actual longevity from 1992 to 2014 of the individuals in our analysis did not have a systematic relationship with whether the original HRS respondents continued to participate in the study except that leaving the study implied a later death. We believe this assumption to be reasonable for the purpose of our analysis for two reasons. First, a small percentage (8 percent) of the original respondents dropped out of the survey, so that the impact of any longevity differences among the population who dropped out would likely have been small. Second, while some baseline characteristics of respondents do appear correlated with non-response over time, the population that dropped out of the study does not appear to vary significantly from those completing each wave, except for race and ethnicity. We conducted this analysis, at the individual level, for HRS respondents in 1992, and any spouses or partners also born in 1931 through 1941. Additional details and caveats to this analysis are available in appendix I. We broke the sample into quintiles based on their income or wealth. To determine an individual’s place in the income distribution, we measured mid-career household earnings using administrative records from the Social Security Administration that are linked to the HRS data. Specifically, we defined mid-career household earnings based on average annual earnings reported to the Social Security Administration for years when the survey respondent we identified as the household head was ages 41 to 50 as well as the earnings of their spouse or partner during those years if the respondent was part of a couple in 1992. This measure of earnings provides a relatively stable indicator of the household’s labor market experience, compared to using a single year of earnings, which could be unusually high or low. For wealth, we used the household’s initial net worth in 1992, including any balances in defined contribution accounts or individual retirement accounts, but excluding second homes, which HRS did not consistently capture in all years. In both instances, the sample was broken into quintiles. For additional details on our methodology, see appendix I. Overall, an estimated 63 percent of the individuals in our sample were alive in 2014 (ages 73 to 83), and greater levels of income and wealth were associated with greater longevity in our analysis of HRS data. For income, an estimated 52 percent of individuals from households in the bottom quintile of the mid-career earnings distribution were alive in 2014, compared to an estimated 74 percent of individuals from households in the top quintile. (See fig. 12.) The percentages by wealth quintile were similar. Other researchers have similarly found that greater levels of income and wealth are associated with greater longevity. For example, a researcher at the Social Security Administration has established that men with higher earnings had seen greater gains in longevity than those with lower earnings. Understanding the association among income, wealth, and longevity is complicated because of relationships among the characteristics, as well as their relationships with demographic characteristics (see text box). Besides income and wealth, several demographic characteristics were also associated with longevity in our analysis of HRS data, and these relationships have also been noted in other researchers’ studies. Women tended to live longer than men: Women had greater longevity through 2014, with an estimated 69 percent living to at least ages 73 to 83 compared to an estimated 58 percent of men. Non-Hispanic whites and Hispanics tended to live longer than blacks: For Hispanics, an estimated 68 percent lived to at least 2014, as did an estimated 65 percent of non-Hispanic whites, compared to an estimated 52 percent of non-Hispanic blacks. More educated individuals tended to live longer than those with less education: An estimated 75 percent of college graduates lived to at least 2014, compared to an estimated 65 percent of those who graduated from high school and an estimated 50 percent of those with less than a high school diploma or GED. Individuals who self-reported being in good health tended to live longer than those who reported being less healthy: Among those who self-reported being in excellent health in 1992, an estimated 78 percent lived to at least 2014, compared to an estimated 31 percent of those who reported being in poor health. Income, Wealth, and Demographics Are Interrelated The relationships of income, wealth, and demographics with longevity are complex because of interactions among these characteristics themselves, which make it difficult to determine the direction or extent of causality. For example, there are many potential interactions among educational status, income, and wealth. Higher levels of education could provide access to better job opportunities, increasing income. Education could contribute to greater financial literacy and better financial decision making, increasing wealth. Having access to wealth could make it easier to attain additional education. While income, wealth, and education all are associated with longevity, it is difficult to interpret their individual associations with longevity because of their possible interactions with each other. We estimated that individuals whose households were in the top two quintiles (top 40 percent) of the mid-career earnings distribution were more likely than their counterparts in the bottom 60 percent to be alive in 2014 (ages 73 to 83) in an analysis controlling for race and ethnicity, gender, age, education level, and initial self-reported health status on entry into HRS in 1992. In a similar analysis, we found that individuals from households in the top quintile (top 20 percent) of wealth in 1992 were more likely to be alive than their counterparts in the bottom four quintiles. Our findings are consistent with the work of other researchers who also controlled for such factors. However, such observational studies are only able to demonstrate that a statistical association exists between two characteristics. For example, one study that found a strong association between income and life expectancy specifically notes that unmeasured factors likely affect the association. Similarly, we cannot determine from our analysis the extent to which income or wealth causes differences in longevity. Even among individuals with characteristics associated with decreased longevity, a substantial proportion of older Americans lived at least into their 70s or early 80s, according to our analysis of 1992 to 2014 HRS data. For example, we constructed three scenarios to illustrate how longevity varies for those with different mid-career earnings and education. Among those in the “bottom” scenario–those individuals who had no college education and were from households in the bottom 20 percent of the earnings distribution–an estimated 50 percent were still alive in 2014 (see fig. 13). We estimated that the corresponding percentages for our “middle” scenario and “top” scenario were 65 percent and 80 percent, respectively, of individuals still alive in 2014. Thus, even among those with education and earnings associated with lower longevity, a significant proportion, 50 percent, were still alive in 2014, and these individuals will need to provide for themselves through their remaining years. We also analyzed a subset of our bottom scenario that included those who had no college education and were from households in the bottom 20 percent of the earnings distribution and whose self- reported health status was fair or poor. While the percentage of the individuals who survived was lower, an estimated 39 percent were alive in 2014, which is a substantial proportion. Most individuals have the potential for an unexpectedly long life, including individuals with demographic characteristics associated with lower longevity, income or wealth. In addition, individuals may face major expenses as they age. For example, several experts we spoke with noted that health care costs can pose a particular challenge at older ages. Taken all together, individuals may live a long time and face financial challenges in their later years, including those with less income and wealth. For example, of the individuals in the bottom group of our scenarios illustrating the effects of earnings and education on longevity, an estimated 50 percent were still alive in 2014. Should these individuals not have DC accounts or have little in them, or should they have little to no DB pension benefits, they may have to rely primarily on Social Security (which itself faces financing difficulties) or safety net programs. Using HRS data and following the same households over time, we examined how income and wealth distributions changed and found that, in general, disparities in income decreased while disparities in wealth persisted among a cohort of older Americans as they aged (see text box for more information on our analysis). Households with the top 20 percent of mid-career earnings saw larger drops in income than households in other mid-career earnings groups, decreasing income disparities overall. During the same time period, the amount of wealth held by most households remained steady and wealth disparities persisted. We also found important differences in the distribution of income and wealth among households by race and ethnicity and education level. Analyzing Income and Wealth for Households Over Time We analyzed Health and Retirement Study (HRS) data to estimate how income and wealth distributions changed as a particular cohort of older Americans aged over time. We analyzed income, wealth, and select financial resources for the same group of survey respondents (heads of households) or their spouses or partners who responded to the survey in 1992 and were still alive and responded in 2014, which is the most recent year for which the data are complete. We defined wealth as net worth. Data limitations prevented us from producing estimates of the present value of future income expected from Social Security or defined benefit pensions. The heads of households we analyzed were from the original HRS cohort and were born in 1931 to 1941. If neither the head of household or the spouse or partner interviewed in 1992 was still alive in 2014, their household was not included in our sample. As a nationally representative longitudinal survey, the HRS allows us to follow the same set of Americans from their 50s through the remainder of their lives; these household heads or their spouses or partners had reached their 70s or early 80s by 2014, allowing us to estimate how income and assets changed for the households as they progressed through retirement. We are reporting medians, as our analysis indicated that means were not consistently reliable. Appendix VI contains additional figures examining how assets and income changed for households headed by individuals in HRS’ “War Babies” cohort, who were born from 1942 through 1947. For our analysis, we divided older households in the data into five equally sized quintiles, or earnings groups, based on the number of households and their mid-career household earnings. We defined mid-career household earnings based on earnings reported to the Social Security Administration for years when the survey respondents were ages 41 through 50, as well the earnings of their spouses or partners during those years if the respondents were part of a couple in 1992. For more on our analysis, see appendix I. As described in the textbox above, our analysis included households in which either the head of the household or their spouse or partner were still alive in 2014, and table 1 shows the race and ethnicity and education level of the household head, as well as the composition of the household. As discussed in the previous section, certain demographic characteristics, such as being a minority or being less educated, are associated with a shorter life. However, not everyone with these demographic characteristics will have a shorter life. As the table below shows, there are households in which the head had at least one of these characteristics and lived into his or her 70s or early 80s. We analyzed HRS data and found that household income declined as heads of households born from 1931 through 1941 and their spouses or partners aged, with decreased earnings from work contributing to the decline as people retired. Those households that had the highest mid- career earnings—those in the top earnings group—experienced the largest declines in income from 1992 when the heads of household were ages 51 to 61 to 2014 when the surviving heads of household or their spouses or partners were ages 73 to 83 (see fig. 14). For example, estimated median income for the top earnings group decreased by 53 percent, from about $121,000 in 1992 to about $57,000 in 2014. In comparison, for those with the lowest mid-career earnings—those in the bottom earnings group—estimated median income declined by 36 percent, from about $28,000 to about $18,000 over this same period. The decrease in income disparities may reflect the shift from work-related earnings to Social Security as the largest source of income for households in the top 20 percent, indicating the possible transition from working to retirement. More specifically, in 1992, 94 percent of households in the top mid-career earnings group had work-related earnings, which contributed the largest amount to their income. By 2014, only 25 percent of the top earnings group still had work-related earnings, and Social Security provided the highest median value of all income sources. Among households in the bottom mid-career earnings group, 68 percent had work-related earnings in 1992, and 15 percent continued to have work-related earnings in 2014. Similarly, work-related earnings provided the greatest source of income for these households in 1992, and Social Security provided the highest median value of all income sources for these households in 2014. However, concerns about retirement insecurity for those with lower earnings may remain. Social Security is progressive, meaning it replaces a higher percentage of income for those with lower earnings, but the formula for calculating Social Security benefits provides a higher benefit amount to those with higher lifetime earnings. In addition, those households with higher mid-career earnings maintained relatively higher income in retirement, perhaps due to their having higher levels of other types of non-wage income after retiring. For example, in 2014, a significantly greater percentage of households in the top two earnings groups had income from employer-sponsored retirement accounts compared to those in the bottom earnings groups, although households may not be consistent in how they spend down these funds. We analyzed HRS data from 1992 to 2014—when heads of households were in roughly their 50s to when they were in their 70s or early 80s—and found that for most households, the level of wealth was relatively consistent as they aged, and disparities in wealth persisted over time. As shown in figure 15, wealth remained relatively steady for households in the bottom three mid-career earnings groups over the time period we examined while households in the top two mid-career earnings groups experienced larger fluctuations in wealth. More specifically, households in the top two earnings groups saw their wealth increase overall from 1992 to 2014. However, while wealth increased from 1992 to 2006, this was followed by declines in wealth from 2006 to 2014. Looking at the overall time period of our analysis, wealth disparities persisted between households in the top earnings groups and households in the bottom earnings groups. For example, in 1992, households in the bottom 20 percent had estimated median wealth of about $93,000 while households in the top 20 percent had estimated median wealth of about $432,000, a difference of about $339,000 (or the top had about 4.6 times the median wealth of the bottom). In 2014, households in the bottom 20 percent had estimated median wealth of about $66,000 while households in the top 20 percent had estimated median wealth of about $539,000, a difference of about $473,000 (or the top had about 8.2 times the median wealth of the bottom). Other researchers have found that that some households may not spend down their wealth as much during retirement due to factors including a generally higher propensity to save, a desire to leave bequests, and the desire to self-insure against medical costs. Households in the top 20 percent of mid-career earnings had greater participation in retirement accounts (see sidebar) and increased home equity relative to other households, which may have contributed to wealth disparities over the time period of our analysis. Retirement Accounts. Among households that had retirement accounts, the median value of retirement accounts increased for all of our income groups (see fig. 16); however, the continued wealth disparities between higher- and lower-earning households may be due to significant differences in the value of retirement accounts and in household participation. The value of retirement accounts for households in the top and bottom earnings groups increased substantially between 1992 and 2014 (a 93 percent and 138 percent increase, respectively). Some of the increase in retirement account balances over time may be due to contributions to DC plans and IRAs during years in which individuals worked, as well as waiting until age 70 ½, when many individuals are required to take minimum distributions from their IRAs. Despite this potential for gains in account balances across the distribution, disparities still exist. In 2014, among households that had retirement accounts, we estimated that households in the top 20 percent had about three times more in their retirement accounts compared to households in the bottom 20 percent (about $176,000 compared to about $54,000). Higher-earning households may not spend down their retirement account balances as much in retirement whereas lower-earning households may have spent down all or part of their account balances. In addition to having more in their retirement accounts, a greater percentage of households in the top earnings group had retirement accounts compared to households in the bottom earnings group. For example, in 2014, an estimated 69 percent of households in the top 20 percent had retirement accounts compared to an estimated 19 percent of households in the bottom 20 percent. Home equity. From 1992 to 2014, home equity increased across all mid-career earnings groups for households with home equity; however, households in the top two earnings groups saw greater increases in the value of their home equity compared to households in the bottom two earnings groups (see fig. 17). Over this time period, a greater percentage of households in the top 20 percent had home equity compared to households in the bottom 20 percent. More specifically, from 1992 to 2014, the percentage of households in the bottom 20 percent with home equity ranged from an estimated 61 percent to 70 percent. For the top 20 percent, the percentage of households with home equity ranged from 88 to 94 percent. Despite the recession from 2007 to 2009, which may have caused home values to depreciate, median home equity for households in the top 20 percent that had home equity increased by an estimated 30 percent from 1992 to 2014. At the same time, median home equity for the bottom 20 percent of households with home equity increased by an estimated 14 percent, though this change was not statistically significant. One expert we interviewed also noted recent real estate appreciation as benefiting wealthier retirees. Significant differences in income and wealth associated with race and ethnicity, as well as education levels, continued as households aged, according to our analysis of heads of households and their spouses or partners as they aged from roughly their 50s to their 70s or early 80s using 1992 through 2014 HRS data. Non-Hispanic, white households in the bottom 40 percent of mid-career earnings had higher estimated median incomes, and non-Hispanic, white households across the mid-career earnings distribution generally had greater wealth, than minority households. In terms of income, the gap between non-minority and minority households in the bottom 40 percent persisted even as median income decreased overall for households as they aged. For example, we estimated that, in 1992, non-Hispanic, white households in the bottom 20 percent had about $20,000 more in income than minority households. The income disparity was smaller (about $9,700) in 2014, but still remained. In terms of wealth, non-Hispanic, white households had persistently higher wealth compared to minority households across all levels of the mid-career earnings distribution. For example, among the bottom 20 percent of households, in 1992, non-Hispanic, white households had about $138,000 more in estimated median wealth than minority households. While this difference decreased to about $119,000 in 2014, the wealth difference remained. Similarly, for the top 20 percent of households, in 1992, non-Hispanic, white households had about $170,000 more in estimated median wealth than minority households, and, in 2014, the wealth disparity increased to about $294,000. Households headed by someone with at least some college education generally had higher median incomes and more wealth than households headed by someone who did not attend college. Income disparities existed across the mid-career earnings distribution from 1992 to 2014. For example, we estimated that, in 1992, households in the top 20 percent with heads who attended college had about $44,000 more in income compared to households in the top 20 percent with heads who did not attend college. We estimated that, in 2014, households with heads in the top 20 percent who had attended college still had greater income, though the difference was smaller (about $25,000). Similarly, heads of households in the bottom 20 percent who had attended some college had more income than heads of household who had not. For example, in 1992, households with heads who had attended some college had about $31,000 more in income than households with heads who had not, and that difference decreased to $9,700 in 2014. Wealth disparities generally existed across the mid-career earnings distribution over time. For example, in 1992, households in the top 20 percent with heads who had attended some college had about $166,000 more in estimated median wealth compared to households in the top 20 percent with heads who did not attend college. In 2014, the difference in estimated median wealth between these same groups was about $386,000. Similarly, households in the bottom 20 percent with heads who had attended some college had greater median wealth than households in the bottom 20 percent with heads who had not attended college. For example, we estimated that, in 1992, households in the bottom 20 percent with heads who attended college had about $176,000 more in wealth than heads who had not. In 2014, the difference in median wealth between these groups was about $120,000. Our findings are consistent with those of other researchers, who found that educational attainment was an important determinant of wealth at age 65, and that it was strongly correlated with wealth even after controlling for lifetime earnings. We provided a draft of this report to the Department of Labor, the Department of the Treasury, the Internal Revenue Service, and the Social Security Administration for review and comment. While none of the agencies provided official comments, the Department of Labor and Social Security Administration provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Labor, the Secretary of the Treasury, the Commissioner of the Internal Revenue Service, and the Commissioner of the Social Security Administration. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or jeszeckc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. To determine how growing disparities in the distributions of income and wealth affect older Americans, we examined (1) the distributions of income and wealth among all older Americans over time; (2) the association between income, wealth, and longevity among older Americans; and (3) how the distributions of income and wealth have changed over time for a cohort of individuals as they aged. This appendix provides a detailed account of the data sources used to answer these questions and the analyses we conducted. The appendix is organized into three sections. Section I describes how we reviewed literature relevant to this report’s objectives and provides information on the interviews we conducted. Section II describes the information sources and methods we used to analyze the distributions of income and wealth among all older Americans over time. Section III describes the information sources and methods we used to analyze how income and wealth among older Americans are associated with longevity, and how the distributions of income and wealth changed as a cohort of individuals aged. For the purposes of our analysis, we defined wealth to be a household’s net worth—that is, total assets minus total debt. Net worth is a measure often used by researchers studying retirement security. Older Americans may have other future retirement resources, such as the present value of future income expected from defined benefit (DB) pension plans and Social Security. We supplemented our data analysis with a literature review and interviewed researchers to identify appropriate background information and context. We had two primary methods for identifying literature to include in our literature review: a snowball technique and a database search. To apply the snowball technique, we first identified possible relevant literature by examining the studies cited in our 2016 report examining the relationship between Social Security benefits and longevity. Then we reviewed the citations included in those studies. Finally, we reviewed relevant literature included in a weekly report called “Current Awareness in Aging Report,” produced by the Center for Demography of Health and Aging at the University of Wisconsin-Madison, which includes a comprehensive list of recently issued materials relating to aging, including retirement security. We compiled relevant citations across these sources and analyzed abstracts to identify working papers, journal articles, and reports that required further review. We identified reports for inclusion based on whether they provided insight into the following relationships: As older Americans age, the relationship between wealth and expenses, and income and wealth. For older Americans, how income and/or wealth inequality are (1) related to the topics below and (2) how, if at all, these relationships have changed over time or generations: Rural vs urban locations Role of inequality (income, wealth, longevity) in reliance on federal income security programs among older Americans To complement the snowball technique search, we also conducted a database search. We searched the Proquest database EconLit for scholarly journals and working papers for a 5-year span, from 2013 through 2018, that matched keywords related to our criteria for relevance. We took additional steps to enhance the robustness of our results. We solicited recommendations for literature from GAO stakeholders, agency officials, and contacts at the Congressional Research Service and Congressional Budget Office and added these recommendations to our list for consideration. During interviews with experts, we discussed contrary opinions and findings in the research and requested full citations as needed. We also attended retirement security events and reviewed news clippings for references to contrary opinions or findings in breaking research. Finally, an economist reviewed the methods and reliability of all studies. We included 26 out of 34 articles from the snowball technique search and expert recommendations and an additional 3 out of 160 articles from the database search (the database search identified some of the same articles as the snowball technique search). These 29 articles that best matched our criteria for inclusion were the articles we reviewed. We also identified and interviewed nine researchers whose work was relevant to our objectives and interviewed them in order to identify researchers’ explanations and theories about the relationships between inequality and longevity, health status, gender, education, and race and ethnicity. To select these researchers, we considered their areas of expertise; whether they worked for a federal agency, university, or other type of organization; and their ideological perspective, if known. This section describes the two main data sources we used to analyze trends in the distribution of income and wealth among all older Americans: the Survey of Consumer Finances (SCF) and the Financial Accounts of the United States (FA). To examine the distributions of income and wealth among all older Americans over time, we used 1989 through 2016 data from the SCF. The SCF is a triennial survey of household assets and income from the Board of Governors of the Federal Reserve System (Federal Reserve) and asks households detailed questions about their income—including pension benefits—and assets—including amounts in retirement accounts. The survey also asks about debt and demographic information, among other topics. A different sample of households was used for each year in our analysis. These data allow for comparison of the experiences of same-age households at different points in time. The SCF is conducted using a dual-frame sample design. One part of the design is a standard, multistage area-probability design, while the second part is a special over-sample of relatively wealthy households. This is done in order to accurately capture financial information about the population at large as well as characteristics specific to the relatively wealthy. The two parts of the sample are adjusted for sample nonresponse and combined using weights to make estimates from the survey data nationally representative of households overall. In addition, the SCF excludes people included in the Forbes magazine list of the 400 wealthiest people in the United States. Furthermore, the SCF omits observations that have net worth at least equal to the minimum level needed to qualify for the Forbes list. For example, the 2016 SCF surveyed 6,254 U.S. households and removed six households that had net worth equal to at least the minimum level needed to qualify for the 2016 Forbes list. Over time, the number of households interviewed has expanded (see table 2). We found the SCF to be reliable for the purposes of our report. While the SCF is a widely used federal data source, we conducted an assessment to ensure its reliability. Specifically, we reviewed related documentation and internal controls, spoke with agency officials, and conducted electronic testing. When we learned that particular estimates were not reliable for our purposes, or had sample sizes too small to produce reliable estimates, we did not use them. Nonetheless, the SCF and other surveys that are based on self-reported data are subject to nonsampling error, including the ability to get information about all sample cases; difficulties of definition; differences in the interpretation of questions; and errors made in collecting, recording, coding, and processing data. These nonsampling errors can influence the accuracy of information presented in the report, although the magnitude of their effect is not known. Estimates from the SCF are also subject to some sampling error since, for any given year, the sample is one of a large number of random samples that might have been drawn. Since each possible sample could have provided different estimates, we express our confidence in the precision of the sample results as 95 percent confidence intervals. These intervals would contain the actual population values for 95 percent of the samples that could have been drawn. In this report, we present 95 percent confidence intervals alongside the numerical estimates that were produced using SCF data. All financial figures using the SCF data are in 2016 dollars. We supplemented the SCF data with data from the Financial Accounts of the United States (FA). The FA include data on transactions and levels of financial assets, and liabilities, by sector and financial instrument; balance sheets, including changes in net worth, for households and nonprofit organizations, nonfinancial corporate businesses, and nonfinancial noncorporate businesses; Integrated Macroeconomic Accounts; and additional supplemental detail. These data provide an aggregate estimate of DB pension entitlements (or liabilities, as the FA refer to them), which can be apportioned across SCF respondents (see detailed explanation below). This section describes the analysis that we conducted using the SCF and FA to analyze trends in income and wealth over time for all older Americans. We chose to look at household-level resources because couples may pool their economic resources and the SCF asks some of its questions about resources for households. The Federal Reserve provides the underlying programming code for creating the variables presented in its publications. Where possible, we relied on variable definitions used for Federal Reserve publications using the SCF. For example, we used the race or ethnicity of the household head, defined as either 1) white, non- Hispanic or 2) non-white or Hispanic (which we renamed “minority” for ease of reporting). We also relied on the Federal Reserve’s definitions for net worth, which we refer to as “wealth” in this report; retirement account balances (DC plans and IRAs); income from withdrawals from retirement accounts; and income from Social Security, pension, or disability benefits or annuities. In other cases, we developed our own variables, based on the raw variables described in the SCF codebooks. For example: Older households: households in which the survey respondent or any spouse or partner were aged 55 or older. Household income: estimated total income by adding up all of the individual income components created by the Federal Reserve. Other assets: any other assets that are not retirement accounts, the present value of future income from Social Security or DB pensions, or the value of the household’s primary residence (if one is owned) or vehicles. Other income: any other income coming from a source besides wages; withdrawals from retirement accounts; and Social Security, pension, or disability benefits or annuities. The SCF is a cross-sectional survey, meaning it presents a nationally representative “snapshot” for each survey wave rather than following the same households over time. To create an income distribution, we rank ordered older households by household income and then broke them into five even groups, or quintiles. The “top” refers to the top 20 percent of households in this ranking while the “bottom” refers to the bottom 20 percent of households. We repeated this exercise for each year of the data. While the households included in the SCF are different every survey year, we were able to examine how the distribution of income and wealth across older households changed over time. We used the same method to create wealth distributions, except we rank ordered households by net worth, one measure of wealth, instead of income. To better understand increases in the top quintile, we also estimated the amount of income and wealth held among the top 10 percent, 5 percent, and 1 percent of households, when possible, for each survey year. We also created distributions of income and wealth for other subcategories of older households. As with the analysis for all older households, we broke the subcategory population into quintiles. We estimated distributions of income and wealth for the following subcategories for each survey year: Households in which the head was white and non-Hispanic Households in which the head was a minority Households in which the head attended at least some college Households in which the head did not attend college For all older households, we also estimated the percentage of households in each survey year that had 1) wage income, 2) income from retirement account withdrawals or 3) income from Social Security, pension, or disability benefits or annuities, as well as the amount of income provided by each source. Similarly, we estimated the percentage of older households that had a retirement account (DC or IRA), owned their home, or owned a vehicle, as well as the value of each of these assets. To better understand the importance of these asset types across the wealth distribution, we also estimated the percentage of households that had a retirement account (DC or IRA) with a balance of at least a $100; owned a vehicle worth at least $100; or had home equity of at least $100. We also analyzed the percentage of households with retirement account balances by bands of $50,000. Additional sensitivity analysis included comparing a household’s location in the income distribution to its location in the wealth distribution for each survey year. We found that the vast majority of households were in the same quintile of the income and wealth distributions or were only one quintile apart. Very few households were in the bottom quintile for income and top quintile for wealth or vice-versa. From 1989 through 2016, the percentage of households who fit these two scenarios was always under 1 percent. The literature on retirement adequacy emphasizes the importance of including measures of the value of future DB and Social Security benefits in measures of the wealth distribution. However, the SCF does not provide estimates of the present value of expected future DB and Social Security benefits. As a result, we did a separate analysis to estimate the present value of future income from DB and Social Security benefits using the SCF and FA data from the Federal Reserve, as well as life expectancy data from the Social Security Administration (SSA). In general, our analysis was done for respondents and spouses/partners separately at the individual level, and estimates were combined to create household totals. We generally followed methods presented in an 2016 paper entitled “Is the U.S. Retirement System Contributing to Rising Wealth Inequality?” by Devlin-Foltz, Henriques, and Sabelhaus (see bibliography for the full citation), but made some changes in the assumptions given our specific focus on older Americans. In order to estimate the present value of income expected from DB plans at the household-level, we started with the aggregate value of accrued DB benefits by survey year from the FA. Following Devlin-Foltz et al. (2016), we calculated aggregate DB pension entitlements as the portion of total pension entitlements not found in DC assets and annuities held in IRAs at life insurance companies. Then, we allocated aggregate DB entitlements across households in a series of steps, ultimately splitting the aggregate DB entitlements between SCF respondents who were already receiving benefits and those who were covered by DB plans but were not yet receiving benefits. In the first step of the allocation, we estimated the present value of promised DB benefits for current DB beneficiaries. The present value of promised DB benefits for those already receiving benefits was based on the reported values for DB benefits in the SCF, life tables from SSA, and an assumed 3 percent real discount rate. After solving for the present value of promised DB benefits for those currently receiving benefits, we subtracted the total amount of DB benefits promised to current DB beneficiaries from the aggregate DB assets to solve for the share to be distributed to future DB beneficiaries. By doing this, we effectively assumed that current DB beneficiaries had first claim to DB pension assets. We allocated the remaining DB assets to future DB recipients by assigning each future DB beneficiary a share of the amount of the residual of aggregate DB entitlements (left over after current beneficiaries claimed their share) based on their earnings, the number of years they participated in a DB plan, their expected retirement age as stated in the SCF, and a 3 percent real discount rate. We also estimated the present value of expected future Social Security benefits for current and future Social Security beneficiaries, using information from the SCF on Social Security benefits for current Social Security beneficiaries and earnings information for future Social Security beneficiaries. With respect to current Social Security beneficiaries, we solved for the present value of Social Security benefits using annual Social Security benefits as reported in the SCF, life tables from SSA, and an assumed 3 percent real discount rate, consistent with our DB analysis. For future Social Security beneficiaries, we used current earnings or earnings from the longest job held as reported in the SCF as the basis for the Social Security benefit. Given that our analysis focused on older Americans, we assumed that future Social Security beneficiaries were close enough to retirement that the earnings information in the SCF provided a reasonable proxy for lifetime earnings. We created a monthly average of these earnings, which we used as a simplified version of the average indexed monthly earnings (AIME). We used these thresholds to compute something similar to the primary insurance amount (PIA) by assigning 90 percent of earnings up to the first bend point, 32 percent of earnings between the first and second bend points, and 15 percent of earnings between the second bend point and the monthly taxable maximum. We assumed everyone who was not yet receiving benefits but would in the future started collecting benefits at 62 or at their current age if older than 62. We applied benefit rules associated with each individual’s birth year to the PIA as set by the Social Security Administration and made adjustments for spousal benefits. We estimated the present value of Social Security benefits for future beneficiaries using the estimated PIA, a retirement age of 62 or their current age if older than 62 and not yet receiving benefits, life tables from SSA, and a 3 percent real discount rate. While adding these present value estimates to wealth better captures the totality of resources available to older Americans, our estimates of the present value of income from future DB and Social Security benefits are subject to uncertainty and should be interpreted with caution. For example, our estimates of the present value of DB benefits for future beneficiaries are not based on SCF respondent-reported expected DB benefits. Instead, we used the aggregate DB entitlements in the FA data and allocated that amount across households with DB plans. We followed this method, in part, because it appears that workers do not have a good understanding of their pension plan parameters and confuse DB benefits with other types of payouts in the SCF data, according to Devlin-Foltz et al. (2016). Moreover, our estimates of the present value of Social Security benefits for future beneficiaries are not based on lifetime earnings since the SCF does not collect all of the inputs needed to project Social Security benefits for respondent-families. However, it is possible to get a sense of the distributional impact of Social Security by focusing on those near retirement in certain points in time. A general limitation of our analysis of the present value of future income from DB pensions and Social Security is that our estimates rely on assumptions about life expectancy, real discount rates, and retirement ages, which are unlikely to hold for all households. As a result, we conducted some sensitivity analyses, particularly with respect to real discount rates and retirement ages. For both the DB and Social Security sensitivity analyses, we varied the real discount rate given the uncertainty about future interest rates. In general, higher discount rates result in lower estimated present values, so our estimates of the present value of future DB and Social Security benefits are sensitive to the assumptions about the discount rate. This is especially important in the DB analysis, as changing the assumed discount rate affects the allocation of aggregate DB assets between current and future DB beneficiaries. For example, using a 2 percent real discount rate, as opposed to a 3 percent real discount rate, yielded a higher allocation of aggregate DB assets for current beneficiaries compared to our baseline estimates. Using a 4 percent real discount rate, as opposed to 3 percent, generated a higher allocation of aggregate DB assets for future DB beneficiaries relative to our baseline estimates. For future beneficiaries, we had to make assumptions regarding the respondent and spouse/partner’s retirement age. For the DB analysis, we used the SCF-reported expected retirement age, given that our focus is older Americans, and older people not yet claiming benefits are relatively close to retirement. Given these assumptions, we also did the analysis assuming that all future DB beneficiaries retired at 62 and 65. Assuming different retirement ages can change the amount of the share of aggregate DB assets allocated to individual future DB beneficiaries in the SCF. For the Social Security analysis, we generally assumed that future Social Security beneficiaries retired at 62, in part because a sizeable proportion of people claim Social Security at 62, despite increases in the full retirement age. In addition, according to Devlin-Foltz et al. (2016), assuming a low retirement age decreases the present value of benefits directly if the reductions for early retirement are not actuarially fair, and indirectly if the individual were to keep working at a high enough income to increase their average indexed monthly earnings. Agency officials raised technical concerns about choosing age 62. It is possible that setting the retirement age at 62 may overstate the present value of future Social Security benefits, depending on various factors including interest rates and mortality. We considered using alternative retirement ages and do not believe that choosing a different retirement age for those not yet retired would substantively change our findings. Alternative methods to using present value estimates of future income expected from Social Security and DB pensions for analyzing distributional disparities in retirement security exist. For example, one option would be to evaluate how future monthly income from Social Security and DB pensions would be expected to affect retirement security, perhaps by assessing how the standard of living for workers would be expected to change. Additionally, disparities in health in adulthood could contribute to subsequent disparities in income and wealth at older ages. However, for our analysis, it was useful to estimate the present value of Social Security and DB pensions so we could compare the value of these sources to retirement account balances. In addition, the SCF does not include sufficient data on health to consider its role in income and wealth disparities for this part of our analysis. This section describes the analysis we conducted to determine how the income and wealth of a specific cohort of older Americans were associated with longevity, and how the distributions of income and wealth changed as this cohort aged. For these analyses, we used data from the Health and Retirement Study (HRS), described below. We analyzed data collected through the HRS, a nationally representative survey of older Americans. The HRS is a longitudinal survey, meaning that it follows the same individuals and households over the course of the study, allowing us determine how households’ income and wealth changed over time. HRS is a project of the University of Michigan’s Institute for Social Research that is funded through a cooperative agreement with the National Institute on Aging (U01AG009740). It collects information on individuals over age 50 and, among other things, contains detailed data on their education, marital status, work history, health, assets, and income. When the HRS began in 1992, it consisted of a representative sample of Americans then aged 51-61, which is called the original or core HRS cohort. Since then, several additional cohorts of individuals have been added to the data to maintain representation of the older population, beginning in 1993 with the Asset and Health Dynamics Among the Oldest Old (AHEAD) cohort. Currently, a new cohort of participants aged 51-56 is added to the study every 6 years (see table 3). Respondents are surveyed every 2 years. We analyzed the HRS original cohort for our examinations of the association between longevity, income, wealth, and other factors; and our analysis of how income and assets change as the original HRS cohort aged. We also analyzed how income and assets changed for the War Babies cohort, which includes individuals born from 1942 through 1947. Figures from this analysis are presented in Appendix VI. We used three forms of HRS data: Public-Use HRS data: Most HRS datasets are available for download from the HRS website. For each wave, HRS makes an early release version of the data available prior to the final version. As of June 2019, final release files are available for each wave of the survey from 1992 through 2014, and the 2016 early release file is available. RAND HRS data: Researchers at RAND have created a more user- friendly version of the public-use HRS data (see below for more details). As of June 2019, RAND files are available through the 2014 final release data. Restricted-use HRS data: Some data resources in the HRS are restricted, meaning they are available only under special agreement because they contain sensitive and/or confidential information. For this report, we used restricted data containing earnings records from SSA. We conducted our analysis of the restricted-use files via a virtual desktop environment data enclave made available by the University of Michigan’s Center on the Demography of Aging (MiCDA). RAND, a research organization, cleans and processes the HRS data to create a user-friendly longitudinal dataset that has consistent and intuitive naming conventions and model-based imputations for missing wealth and income data. In most cases, we used the RAND version of the HRS variables due to the greater ease of use and the additional data cleaning already performed. RAND income and wealth variables were given in nominal dollars. We adjusted these variables to real 2016 dollars using the Consumer Price Index for All Urban Consumers. To calculate mortality, we supplemented the RAND files with information from the early release 2016 public use file to the extent that it provided additional information on mortality through 2014. See the data reliability section below for further discussion of the mortality data. We found the HRS variables presented in this report to be sufficiently reliable. We conducted a data reliability assessment of selected variables by conducting electronic data tests, reviewing documentation on the dataset, and reviewing related internal controls. When we learned that particular variables were not sufficiently reliable, we did not use them in our analysis. We selected our analyses to ensure there was sufficient sample size to produce reliable estimates. We produced variance estimates using a statistical technique chosen to account for the sample design of the HRS and adjusted the sample weights to account for potential bias due to the linkage to SSA administrative data, as described below. We identified additional limitations due to the survey responses being self-reported. As such, they are subject to the respondent’s possible errors in reporting specific financial amounts. We measured mortality from 1992 through 2014. Mortality data in the HRS, including an indicator for a respondent’s death in a given survey year and month and year of death, come from matches with the National Death Index or follow-up interviews with surviving family members. There is complete date of death (specifically month and year of death) information for nearly everyone who died prior to 2012. However, for deaths since 2012, the HRS data linked to the National Death Index was not available, which likely lead to more deaths without information on month and year of death. Since the 2012 and 2014 survey years, there has been time to gather death date information from follow up interviews with families, and less than 10 percent of those who died between the 2012 and 2014 survey years had incomplete data on month and year of death. However, in the 2016 survey year early release public use file, we found that a higher proportion of those who died did not have death dates, likely due to the lack of linkage with the National Death Index and a lack of time to follow up with families since the 2016 survey year to find out when survey participants died. As a result, we determined that we had reliable data on mortality through 2014. HRS contains restricted data drawn from SSA administrative sources for participants who have provided explicit consent to link their responses to administrative data and subsequently were successfully linked with the administrative data. It is possible that respondents who were linked may differ in systematic ways from respondents who were not linked, which would affect the generalizability of estimates derived solely from the subset of participants who were linked. The survey weights provided with HRS data account for the complexity of the survey design (e.g., oversamples of minorities and Floridians), nonresponse, and post- stratification adjustments for demographic distributions, but do not adjust for the administrative linkage. There is evidence that in at least some waves of the survey, there are modest but statistically significant differences in linkage rates on characteristics including race, income, and wealth. One technique to address this potential source of bias is to adjust the sample weights used in variance estimation for observed differences between those with and without linked administrative data. Kapteyn et al. suggest a technique for computing inverse probability weights to account for these differences. Following this technique, HRS has computed a set of weights that account for consent to SSA administrative linkage, but only for the 1992, 1998, and 2004 survey waves. However, this report needed adjusted household weights for all 12 waves and adjusted respondent weights for wave 1. We opted to address the potential non- linkage bias using a logistic model-based propensity score adjustment, rather than a weighting class adjustment for several reasons. First, we had the benefit of many variables with which to model the propensity of non-linkage. Second, weighting class adjustments, which involve creating mutually exclusive classes based on the variables associated with non- linkage, were not feasible because of the large number of variables we included in the adjustment. The number of respondents per cell would be too small. Third, the propensity score adjustment allows us to consider many variables at the same time. Finally, the propensity score adjustment allows us to rank respondents, rather than assume that the characteristics used in a weighting class adjustment would perfectly predict non-linkage. We compared estimates and standard errors obtained using the original weights to the non-linkage adjusted weights. The adjusted weights changed estimates and their standard errors in generally small amounts, but did not affect observed trends in this report. For instance, the median absolute value of the change was less than 1 percent for estimates of median household income for individuals by mid-career earnings quintiles from 1992 to 2014. The median absolute value of the change was 5.7 percent for the standard errors of those estimates. We used the balanced repeated replication method to estimate standard errors for the income and wealth statistics we reported using HRS because the income and wealth statistics were quantiles (i.e., medians). The standard Taylor series (Woodruff) variance estimation method assumes that quantiles can be expressed as a smooth function in the sample and population. However, quantile functions are not considered smooth. After ruling out Taylor series method, we explored replication methods such as jackknife, bootstrap, and balanced repeated replication. Of those, the balanced repeated replication is most suited for the two primary sampling units per stratum design of the HRS. The Fay adjustment stabilizes the estimates across strata when using the normal balanced repeated replication method. This adjustment is particularly relevant for smaller samples. The literature we reviewed suggested that the jackknife produces a poor estimate of the variance of quantiles (Lohr 2009 and Judkins 1990) and that the bootstrap requires more computations than balanced repeated replication. For our analyses, we wanted to classify HRS respondents into income groupings based on a relatively stable measure of income that uses multiple years of administrative data, to reduce measurement error in self- reported survey data and to reduce the chance of basing the income grouping on a single year of unusually low or high income. Several limitations prevent us from classifying households based on their full lifetime income from all sources. HRS does not contain administrative data on income sources besides earnings and Social Security benefits. Moreover, for years before 1978, the administrative earnings records are only available for earnings covered by Social Security and below the taxable maximum. Finally, not all sources of earnings are covered by Social Security. While around 96 percent of employment is currently covered by Social Security, this has not always been the case. In particular, successive expansions of coverage in the 1950s and 1960s greatly increased the proportion of the workforce covered by Social Security, such that relying on SSA earnings records going back to 1951 would underestimate the earnings of large numbers of older HRS participants. Thus, for our analysis, we constructed earnings groupings based on a measure of “mid-career” earnings, based on a household’s average annual reported earnings when the household head was age 41 to age 50. Earnings tend to peak (and remain relatively stable) for workers in their mid-40s through their early 50s. We begin measuring earnings at age 41 to avoid using data prior to expansions of Social Security coverage and to minimize our reliance on imputed earnings above the taxable maximum. In the early years of the study, HRS sought retrospective consent for administrative data linkages. As a result, some participants who only provided consent for the administrative linkage during their initial interview and did not provide consent in subsequent interviews did not have earnings records after age 50. Therefore, we set age 50 as the upper bound for our measure of mid-career earnings. Our goal was to determine how income, wealth, and other demographic and health-related factors are associated with the longevity of older Americans over age 50 in the original HRS cohort. We measured the proportion of original HRS participants still alive at the end of the survey to examine how longevity varied across the income and wealth distributions, as well as across different demographic and health-related variables, including race, educational attainment, gender, and self- reported health status at the beginning of the survey. In order to examine these relationships, we used data from the original HRS cohort to measure deaths over a maximum of 22 years (1992 through 2014). Every 2 years, the HRS attempted to measure whether the original respondents were still alive, but these longevity data were incomplete because some of the original respondents declined to participate in later waves of the survey. Once these respondents left the survey, their actual longevity could not be followed. This incomplete measurement of longevity is generally known as “censored data” in statistics. Special methods of “survival analysis” are required to avoid making inaccurate conclusions about actual longevity from this type of data, when the analyst can only measure longevity up to a certain time before death. Survival analysis accounts for survey respondents with complete or incomplete longevity data. Without making this distinction, ordinary statistical methods, such as linear regression models of the observed longevities, would not include the correct sample of respondents when estimating the chance that a respondent would die at any time within the observation period. In addition, ordinary methods would incorrectly treat the longevities observed in the observation period as actual longevities, when some of them are the shorter, censored longevities observed before the respondents dropped out of the study. Survival analysis methods correct for this problem, in order to reliably estimate the chance of death by any given time in the observation period. Most importantly, our analysis assumed actual longevity during the observation period did not have a systematic relationship with whether the original HRS respondents continued to participate in the study except that leaving the study implied a later death (“noninformative censoring”). In other words, participants with censored and actual longevities did not systematically differ in ways that affected longevity or the variables associated with it. We believe this assumption to be reasonable for the purpose of our analysis for two reasons. First, a small percentage (8 percent) of the original respondents dropped out of the survey, so that the impact of any longevity differences among the population who dropped out would likely have been small. Second, while some baseline characteristics of respondents do appear correlated with non-response over time, the population that dropped out of the study does not appear to vary significantly from those completing each wave, except for race and ethnicity. In our survival analysis, the dependent variable was composed of two parts, including the time in months to death and whether death was observed during the survey period. In general, we used continuous time survival models, including Kaplan-Meier and Cox proportional hazards regression models to estimate survival functions, which estimate the probability of surviving (or dying) up to the end of the survey period, and hazard functions, which estimate the probability of death, per time unit, given that an individual has survived up to that point in time. We used the Kaplan-Meier method to estimate survival probabilities as a function of time and to obtain univariate statistics on survival for different groups. For example, we estimated the percentage of survivors during the survey period across income and wealth quintiles. We also estimated survivorship across the demographic and health-related variables. Moreover, using the Cox proportional hazards regression models, we analyzed the relationship between income and longevity and wealth and longevity, controlling for related demographic and health-related variables, as well as age at the beginning of the survey. These regressions allow the relationships between various characteristics and death to be described as hazard ratios. For example, hazard ratios that are statistically significant and greater than 1.00 indicate that individuals with those characteristics are more likely to die during the survey period compared to a reference group. Hazard ratios that are statistically significant and less than 1.00 indicate that individuals with those characteristics are less likely to die in the study period compared to a reference group. We estimated survivorship among individuals with the following characteristics in combination: bottom income (earnings) quintile and no college; middle of the income (earnings) distribution (third quintile) and high school diploma or some college (excluding GED); and top of the income (earnings) distribution and college diploma. We then ran a subset of these scenarios using different combinations of self-reported health status for each of the three main scenarios. For example, we estimated survivorship among individuals in the bottom income (earnings) quintile, who had not attended college, and reported being in fair or poor health in 1992. Our results have limitations and should be interpreted with caution. Results from the survival analysis present correlations, not causal estimates. Moreover, while our main analysis includes self-reported health status at the beginning of the study period, we also excluded this variable as a sensitivity check given the interconnectedness of income, wealth, and health and the conclusions were similar. Furthermore, due to limitations with respect to the mortality data in later years of the HRS, we did not have specific months and years of death for 60 respondents we know died during the observation period due to death indicators in the interview status variables from HRS. As a result, we imputed their death dates based on the survey year they were indicated to have died in from the HRS interview status questions. While death is continuous in the sense that it can happen to any person at any time, we only observe death within a given month for those with death dates in the data, and only within a year for those whose death information we gathered for the interview status variables. As a sensitivity check, we redid the analysis using survival information at the person-year level and discrete survival analysis techniques and found similar results. This section describes how we used the HRS to determine how the distributions of income and wealth change as older Americans in the original HRS cohort aged. We focused this analysis on the original HRS cohort (born 1931-1941). This cohort entered the study in 1992 at ages 51-61 and had reached their 70s or early 80s by 2014, allowing us to analyze how income and assets changed as these households progressed through retirement. We conducted our analysis and reported results at the household level because couples may pool financial resources or co-own assets. Also, RAND HRS variables on income and wealth are presented at the household level. When necessary, we combined respondent and spouse or partner level variables we used from the public-use file in order to obtain household-level variables. We restricted this analysis to survey respondents (“household heads”), or any spouses or partners, who were still alive in 2014 to ensure we followed the same group of people throughout our analysis. We grouped households into five earnings groups based on their mid-career earnings, as described above. Our primary goal was to examine how the distribution of income and wealth changed over time for households in the original HRS cohort, based on their mid-career earnings groups. We also examined how specific sources of income and wealth changed over time. We also wanted to determine how these trends varied based on household demographic characteristics, including race and ethnicity and education level, without attempting to ascribe causality. Our analysis included survey respondents (heads of households) or their spouses or partners who responded to the survey in 1992 and were still alive and responded in 2014, which is the most recent year for which the data are complete. The heads of households we analyzed were from the original HRS cohort and were born in 1931 to 1941. If neither the head of household or the spouse or partner interviewed in 1992 was still alive in 2014, their household was not included in our sample. In order to do so, we estimated median levels of household wealth and income every 2 years for each earnings group, as well as median levels for specific sources of income and wealth. We estimated the percentage changes and absolute changes in median wealth and income for each earnings group from 1992 through 2014 in order to determine whether income or wealth levels increased or decreased over time. For specific sources of income and wealth, we estimated medians for all households in each earnings group as well as for only those households which reported having the specific source of income or wealth. For example, we determined the median home equity for all households in each earnings group as well as the median home equity for only those households with home equity for each earnings group. Finally, we calculated the percent of our sample having each type of wealth and income (e.g. home equity, Social Security benefits) for each year in the data. As a sensitivity check, we also analyzed how total assets and income changed for the HRS’s “War Babies” cohort (born 1942-1947). For this analysis, we report 99 percent confidence intervals alongside the percentage or other numerical estimates. We chose to use this level of confidence to account for the use of imputation in the RAND HRS data in addition to the sampling error that using survey data introduces. All financial figures using the HRS data are in 2016 dollars. This appendix compares the top 1 percent of the wealth distribution of older households to several other groups in this distribution: (1) the next 19 percent, (2) the top 20 percent, (3) the bottom 80 percent, and (4) the bottom 20 percent. These comparisons provide context for the financial security of the top 1 percent relative to other households at the top of the wealth distribution, the remainder of the wealth distribution, and households at the bottom of the distribution, respectively. To draw these comparisons, we used 2016 data from the Survey of Consumer Finances, a triennial, cross-sectional survey produced by the Board of Governors of the Federal Reserve System. A different sample of households was used for each year in our analysis. These data allow for comparison of the experiences of same-age households at different points in time. We chose to look at household-level resources because couples may pool their economic resources, and the SCF asks some of its questions about resources for households. We conducted our analysis for older households, which were defined as those in which the household head or any spouse or partner were ages 55 or older. We defined wealth as net worth, or assets minus debt. Because the sample size for the top 1 percent is small, we presented dollar values rounded to thousands of 2016 dollars. dollars) Estimated median value (2016 dollars) Vehicle(s) All other assets Debt n/a Not available. There were insufficient data to produce a reliable estimate of median debt. Financial resource Retirement account(s) Home Vehicle(s) loans, lines of credit, and credit card balances after the last payment. Financial resource Retirement account(s) Estimated median value (2016 dollars) Home Vehicle(s) All other assets n/a Not available. There were insufficient data to produce a reliable estimate of median debt. loans, lines of credit, and credit card balances after the last payment. Financial resource Retirement account(s) Home Vehicle(s) loans, lines of credit, and credit card balances after the last payment. Financial resource Retirement account(s) Home Vehicle(s) Estimated median value (2016 dollars) Financial resource Retirement account(s) Home Vehicle(s) 95 percent confidence interval lower bound 434,000 95 percent confidence interval upper bound 556,000 n/a Not available. There were insufficient data to produce a reliable estimate of median debt. loans, lines of credit, and credit card balances after the last payment. Financial resource Retirement account(s) Home Vehicle(s) mortgages, loans, lines of credit, and credit card balances after the last payment. Financial resource Retirement account(s) Home Vehicle(s) mortgages, loans, lines of credit, and credit card balances after the last payment. Estimated median value (2016 dollars) Financial resource Retirement account(s) Home Vehicle(s) Financial resource Retirement account(s) Home Vehicle(s) mortgages, loans, lines of credit, and credit card balances after the last payment. Financial resource Retirement account(s) Home Vehicle(s) mortgages, loans, lines of credit, and credit card balances after the last payment. Estimated median value (2016 dollars) Financial resource Retirement account(s) Home Vehicle(s) 95 percent confidence interval lower bound 0 0 3,000 1,000 95 percent confidence interval upper bound 0 0 4,000 1,000 n/a Not available. There were insufficient data to produce a reliable estimate of median debt. mortgages, loans, lines of credit, and credit card balances after the last payment. This appendix contains additional results from our survival analysis, as shown in the tables below. Appendix V: 2014 Population in the Health and Retirement Study (HRS) This appendix compares the demographic characteristics, as of 2014, of the HRS sample we used in our analysis. This appendix contains estimates of income and wealth for households, where the heads of households were born from 1942 through 1947. The Health and Retirement Study (HRS) refers to this cohort as the “War Babies” cohort. In addition to the contact named above, Michael Collins (Assistant Director), Jennifer Gregory (Analyst-in-Charge), Garrick Donnelly, Kathleen McQueeney, Chris Wickham, and Christopher Zbrozek made key contributions to this report. Also contributing to this report were Susan Aschoff, James Bennett, Deborah Bland, Melinda Bowman, Nisha Hazra, Kirsten Lauber, Jeffrey Miller, Oliver Richard, Amrita Sen, Joseph Silvestri, Jeff Tessin, Frank Todisco, Adam Wendel, and Sirin Yaemsiri. Auerbach, Alan J., Laurence J. Kotlikoff, and Darryl R. Koehler. “U.S. Inequality, Fiscal Progressivity, and Work Disincentives: An Intragenerational Accounting.” NBER Working Paper, no. 22032 (2016). Banerjee, Sudipto. “Asset Decumulation or Asset Preservation? What Guides Retirement Spending?” EBRI Issue Brief, no. 447 (2018). Banerjee, Sudipto. “Income Composition, Income Trends and Income Shortfalls of Older Households.” EBRI Issue Brief, no. 383 (2013). Bee, Adam and Joshua Mitchell. “Do Older Americans Have More Income Than We Think?” SESHD Working Paper, vol. 2017, no. 39 (2017). Board of Governors of the Federal Reserve System. “Changes in U.S. Family Finances from 2013 to 2016: Evidence from the Survey of Consumer Finances,” Federal Reserve Bulletin, vol. 103, no. 3 (Washington, D.C.: September 2017). Bosworth, Barry P. and Kan Zhang. “Evidence of Increasing Differential Mortality: A Comparison of the HRS and SIPP.” Center for Retirement Research Working Paper, vol. 2015, no. 13 (2015). Bricker, Jesse, Alice M. Henriques, Jake A. Krimmel, and John E. Sabelhaus. “Measuring Income and Wealth at the Top Using Administrative and Survey Data.” Finance and Economics Discussion Series, vol. 2015, no. 030 (2015). Brookings Economic Studies Program. Later Retirement, Inequality in Old Age, and the Growing Gap in Longevity between Rich and Poor (Washington, D.C.: Brookings Institution, 2016). Burtless, Gary. “What Do Stock Market Fluctuations Mean for the Economy?” Brookings Opinions, February 23, 2018. Accessed March 14, 2019, https://www.brookings.edu/opinions/what-do-stock-market- fluctuations-mean-for-the-economy?/ Chen, Anqi, Alicia H. Munnell, and Geoffrey T. Sanzenbacher. “How Much Income Do Retirees Actually Have? Evaluating the Evidence from Five National Datasets.” Center for Retirement Research Working Paper, vol. 2018, no. 14 (2018). Chetty, Raj, Michael Stepner, Sarah Abraham, Shelby Lin, Benjamin Scuderi, Nicholas Turner, Augustin Bergeron, and David Cutler. “The Association Between Income and Life Expectancy in the United States, 2001-2014.” JAMA, vol. 315, no. 16 (2016). Choi, HwaJung and Robert F. Schoeni. “Health of Americans Who Must Work Longer to Reach Social Security Retirement Age.” Health Affairs, vol. 36, no. 10 (2017). Congressional Budget Office. Measuring the Adequacy of Retirement Income: A Primer. 53191. Washington, D.C.: October 2017. Congressional Budget Office. Trends in Family Wealth, 1989 to 2013. 51846. Washington, D.C.: August 2016. Congressional Research Service. The U.S. Income Distribution: Trends and Issues. R44705. Washington, D.C.: December 8, 2016. Devlin-Foltz, Sebastian, Alice Henriques, and John Sabelhous. “Is the U.S. Retirement System Contributing to Rising Wealth Inequality?” The Russell Sage Foundation Journal of the Social Sciences, vol. 2, no. 6 (2016). Federal Reserve Bank of St. Louis. “When the Stock Market Rises, Who Benefits?” On the Economy Blog, February 27, 2018. Accessed March 14, 2019, https://www.stlouisfed.org/on-the- economy/2018/february/when-stock-market-rises-who-benefits. Ghilarducci, Teresa, Siavash Radpour, and Anthony Webb. “Employer Retirement Wealth Inequality: 1992 and 2010.” Paper presented at the annual meeting of the American Economics Association Meetings, Philadelphia, P.A.: January 6, 2018. Goda, Gopi Shah, Shanthi Ramnath, John B. Shoven, and Sita Nataraj Slavov. “The Financial Feasibility of Delaying Social Security: Evidence from Administrative Tax Data.” Journal of Pension Economics and Finance, vol. 17, no. 4 (2018). Gustman, Alan, Thomas Steinmeier, and Nahid Tabatabai. “Distributional Effects of Means Testing Social Security: Income Versus Wealth.” NBER Working Paper Series, no. 22424 (2016). Johnson, Barry W. and Brian Raub. “How Much Longevity Can Money Buy? Estimating Mortality Rates for Wealthy Individuals.” Statistical Journal of the IAOS, vol. 34 (2018). Johnson, Richard W. Delayed Retirement and the Growth in Income Inequality at Older Ages (Washington, D.C.: Urban Institute, 2018). Kindig, David A. and Erika R. Cheng. “Even As Mortality Fell in Most US Counties, Female Mortality Nonetheless Rose in 42.8 Percent of Counties From 1992 to 2006.” Health Affairs, vol. 32, no. 3 (2013). Olshansky, S. Jay, Toni Antonucci, Lisa Berkman, Robert H. Binstock, Axel Boersch-Supan, John T. Cacioppo, Bruce A. Carnes, Laura L. Carstensen, Linda P. Fried, Dana P. Goldman, James Jackson, Martin Kohli, John Rother, Yuhui Zheng, and John Rowe. “Differences in Life Expectancy Due to Race and Educational Differences Are Widening, and Many May Not Catch Up.” Health Affairs, vol. 31, no. 8 (2012). Owyang, Michael T. and Hannah G. Shell. “Taking Stock: Income Inequality and the Stock Market.” Economic Synopses, No. 7 (2016). Pijoan-Mas, Josep and Jose-Victor Rios-Rull. “Heterogeneity in Expected Longevities.” Demography, vol. 51, no. 6 (2014). Poterba, James M. “Retirement Security in an Aging Population.” The American Economic Review, vol. 104, no. 5 (2014). Poterba, James, Steven Venti, and David A. Wise. “Longitudinal Determinants of End-of-Life Wealth Inequality.” Journal of Public Economics, vol. 162 (2018). Poterba, James M., Steven F. Venti, and David A. Wise. “Were They Prepared for Retirement? Financial Status at Advanced Ages in the HRS and AHEAD Cohorts.” NBER Working Papers, no. 17824 (2012). Poterba, James, Steven Venti, and David Wise. “The Composition and Drawdown of Wealth in Retirement.” Journal of Economic Perspectives, vol. 25, no. 4 (2011). Raskin, Sarah Bloom. “Downturns and Recoveries: What the Economies in Los Angeles and the United States Tell Us.” Remarks at the luncheon for Los Angeles business and community leaders, Los Angeles Branch of the Federal Reserve Bank of San Francisco. April 12, 2012. Ruiz, John M., Patrick Steffen, and Timothy B. Smith. “Hispanic Mortality Paradox: A Systematic Review and Meta-Analysis of the Longitudinal Literature.” American Journal of Public Health, vol. 103, no. 3 (February 2012). Saez, Emmanuel and Gabriel Zucman. “Wealth Inequality in the United States Since 1913: Evidence from Capitalized Income Tax Data.” The Quarterly Journal of Economics, vol. 131, no. 2 (2016). Waldron, Hilary. “Mortality Differentials by Lifetime Earnings Decile: Implications for Evaluations of Proposed Social Security Law Changes.” Social Security Bulletin, vol. 73, no. 1 (2013). Waldron, Hilary. “Trends in Mortality Differentials and Life Expectancy for Male Social Security-Covered Workers, by Socioeconomic Status.” Social Security Bulletin, vol. 67, no. 3 (2007). Wang, Haidong, Austin E Schumacher, Carly E Levitz, Ali H Mokdad, and Christopher JL Murray. “Left Behind: Widening Disparities for Males and Females in US County Life Expectancy, 1985-2010.” Population Health Metrics, vol. 11, no. 8 (2013). Retirement Security: Most Households Approaching Retirement Have Low Savings, an Update. GAO-19-442R. Washington, D.C.: Mar. 26, 2019. The Nation’s Retirement System: A Comprehensive Re-evaluation Needed to Better Promote Future Retirement Security. GAO-19-342T. Washington, D.C.: Feb. 6, 2019. The Nation’s Retirement System: A Comprehensive Re-evaluation Is Needed to Better Promote Future Retirement Security. GAO-18-111SP. Washington, D.C.: Oct.18, 2017. Older Workers: Phased Retirement Programs, Although Uncommon, Provide Flexibility for Workers and Employers. GAO-17-536. Washington, D.C.: June 20, 2017. Retirement Security: Low Defined Contribution Savings May Pose Challenges. GAO-16-408. Washington, D.C.: May 5, 2016. Retirement Security: Shorter Life Expectancy Reduces Projected Lifetime Benefits for Lower Earners. GAO-16-354. Washington, D.C.: Mar. 25, 2016. Retirement Security: Better Information on Income Replacement Rates Needed to Help Workers Plan for Retirement. GAO-16-242. Washington, D.C.: Mar. 1, 2016. Retirement Security: Most Households Approaching Retirement Have Low Savings. GAO-15-419. Washington, D.C.: May 12, 2015. Retirement Security: Trends in Marriage and Work Patterns May Increase Economic Vulnerability for Some Retirees. GAO-14-33. Washington, D.C.: Jan. 15, 2014. Retirement Security: Women Still Face Challenges. GAO-12-699. Washington, D.C.: July 19, 2012. Unemployed Older Workers: Many Experience Challenges Regaining Employment and Face Reduced Retirement Security. GAO-12-445. Washington, D.C.: Apr. 25, 2012.", "summary": "Income and wealth inequality in the United States have increased over the last several decades. At the same time, life expectancy has been rising, although not uniformly across the U.S. population. Taken together, these trends may have significant effects on Americans' financial security in retirement. GAO was asked to examine the distribution of income and wealth among older Americans, as well as its association with longevity, and identify the implication that these trends may have on retirement security. This report examines (1) the distributions of income and wealth among all older Americans over time; (2) the association between income, wealth, and longevity among older Americans; and (3) how the distributions of income and wealth changed over time for a cohort of individuals as they aged. To conduct this work, GAO analyzed data from two nationally representative surveys: the SCF, using data from 1989 through 2016, and the HRS. GAO used 1992 through 2014 HRS data linked to earnings records from the Social Security Administration. While preliminary 2016 HRS data are available, GAO used 2014 data, which contain more complete information for GAO's analysis. GAO also reviewed studies and interviewed researchers to further analyze the relationships between income, wealth, longevity, and retirement security. Disparities in income and wealth among older households have become greater over the past 3 decades, according to GAO's analysis of Survey of Consumer Finances (SCF) data. GAO divided older households into five groups (quintiles) based on their income and wealth. Each year of data in the analysis, and, thus, each quintile, included different sets of households over time. Average income and wealth was generally higher over time (see fig. 1 for average income), disproportionately so for the top quintile (top 20 percent). For example, in 2016, households in the top quintile had estimated average income of $398,000, compared to about $53,000 for the middle quintile and about $14,000 for the bottom quintile. GAO also found that for quintiles with lower wealth, future income from Social Security and defined benefit pensions provide a relatively significant portion of resources in retirement for those who expect such income. A substantial number of older Americans born from 1931 through 1941 lived at least into their 70s or early 80s, according to GAO's analysis of data from the Health and Retirement Study (HRS), a nationally representive survey which follows the same individuals over time. GAO divided individuals born from 1931 through 1941 into quintiles based on their mid-career household earnings using records from the Social Security Administration. GAO's analysis, as well as that of other researchers, shows that differences in income, wealth, and demographic characteristics were associated with disparities in longevity. However, even with these disparities, we found a substantial number of people in the sample were alive in 2014, including those with characteristics associated with reduced average longevity, such as low earnings (see fig. 2) and low educational attainment. Taken all together, individuals may live a long time, even individuals with factors associated with lower longevity, such as low income or education. Those with fewer resources in retirement who live a long time may have to rely primarily on Social Security or safety net programs. GAO's analysis of HRS data also found that disparities in household income decreased while disparities in wealth persisted as a cohort of older Americans aged from approximately their 50s into their 70s or early 80s. Income disparities decreased between higher- and lower-earning households because higher-earning households saw larger drops in income over time, indicating the possible transition from working to retirement. For example, we estimated median income for the top mid-career earnings group decreased by 53 percent while estimated median income for the bottom earnings group decreased by 36 percent over the same period. Wealth remained relatively steady for households in the bottom three earnings groups over the time period GAO examined, while households in the top two earnings groups experienced larger fluctuations in wealth. GAO estimated that median retirement account balances and median home equity increased across earnings groups for households that had these assets. However, the continued wealth disparities may be due to significant differences in the median value of retirement accounts and home equity between higher- and lower-earning households. GAO also found that white households in the bottom two earnings groups had higher estimated median incomes, and white households in all earnings groups generally had greater estimated median wealth, than racial minority households in those earnings groups. In addition, within each earnings group, households headed by someone with at least some college education generally had higher median incomes and wealth than households headed by someone who did not attend college.", "document_type": "gao"}
{"report": "FFRDCs arose from partnerships between the federal government and academic researchers and scientists during World War II. Those partnerships were later restructured into federal research centers to retain scientists, and they became known as FFRDCs by the mid-1960s. Since that time, FFRDCs have continued to perform tasks including technical studies and analyses, research and development, and systems engineering on behalf of federal agencies, such as DOD. In sponsoring an FFRDC, agencies draw on academic and private sector resources that can contribute to an agency’s ability to accomplish tasks that are integral to the mission and operation of the sponsoring agency. FFRDCs may be operated, managed, and/or administered by a university or consortium of universities, other nonprofit organizations, or a private industry firm as an autonomous organization or as a separate unit of a parent organization. As of May 2019, federal agencies sponsored a total of 42 FFRDCs, 10 of which are sponsored by DOD. These 10 DOD-sponsored FFRDCs can be divided into three categories: S&A Centers: These centers deliver independent and objective analyses and advise in core areas important to their sponsors in support of policy development and decision-making, among other things. Research and Development Laboratories: These laboratories conduct research and development, focusing on the development and prototyping of new technologies and capabilities to meet DOD needs. For example, these laboratories engage in research programs that emphasize the evolution and demonstration of advanced concepts and technology, and transfer new technology to the private sector. Systems Engineering and Integration Centers: These centers meet long-term technical and engineering needs to ensure complex systems meet operational requirements. Among other things, Systems Engineering and Integration Centers assist with testing system performance, development and acquisition of system hardware and software, integration of new capabilities, and the continuous improvement of system operations and logistics. Table 1 lists the 10 DOD-sponsored FFRDCs. As shown in table 1, each of the 10 DOD-sponsored FFRDCs is managed by a specific military department or organization within DOD—referred to as the FFRDC primary sponsor. More broadly, the Office of the Under Secretary of Defense for Research and Engineering oversees and manages DOD’s FFRDC program. DOD’s relationships with FFRDCs are defined through sponsoring agreements between the primary sponsor (i.e., the DOD organization responsible for the overall use of the FFRDC) and the FFRDC parent organization. According to the FAR and DOD instruction, sponsoring agreements define the FFRDC’s purpose and mission and may not exceed 5 years in duration. DOD’s instruction also states that sponsoring agreements are to establish conditions under which DOD may award an FFRDC contract and describe the overarching requirements for operation of the FFRDC. For example, the DOD instruction states that sponsoring agreements are to describe constraints on the FFRDC parent organization that are necessary to preserve the integrity of the FFRDC, such as provisions to prevent the occurrence or appearance of organizational or personal conflicts of interest that may undermine the independence, objectivity, or credibility of the FFRDCs. The DOD instruction also states that sponsoring agreements will preclude FFRDCs from performing commercial work. In this regard, the FAR provides that sponsoring agreements are required to address whether or not the FFRDC may accept work from other entities and if so, the procedures to be followed and the limitations as to the work that can be accepted. Further, the DOD instruction and the FAR provide that sponsoring agreements will generally preclude FFRDCs from competing with any organization in response to a formal request for proposals other than the operation of the FFRDC. After the primary sponsor identifies the need for FFRDC work, and has defined FFRDC core competencies, roles, and responsibilities in the sponsoring agreement, the primary sponsor awards a noncompetitive contract to the FFRDC to support the sponsor’s research requirements, such as addressing national security issues and systems development. Prior to extending a contract or sponsoring agreement for an FFRDC, the FAR requires that the primary sponsor conduct a comprehensive review of the use and need for the FFRDC at least every 5 years. The FAR describes elements of what the comprehensive review should include, such as examination of the sponsor’s special technical needs and mission requirements performed by the FFRDC and assessment of the efficiency and effectiveness of the FFRDC in meeting the sponsor’s needs. The FAR further requires that the head of the sponsoring agency approve continuing or terminating sponsorship based on the results of the comprehensive review. FFRDCs initiate work on specific projects at the request of “work sponsors,” or the entities that request the services of the FFRDC. Work sponsors can be the primary sponsor of the FFRDC or another entity. When initiating work at FFRDCs, the primary sponsor determines whether to approve research projects for the FFRDC before projects are placed on contract. Approval of research projects is based on the determination that work proposed is appropriate for the FFRDC and consistent with the FFRDC’s core competencies as documented in the sponsoring agreement. Additionally, the primary sponsor ensures FFRDC work efforts do not exceed available resources. Among other things, FFRDC work sponsors identify project requirements, propose an appropriate research design, confirm the work is appropriate and consistent with FFRDC core competencies, identify the source of project funding, and monitor the progress of the work to ensure FFRDC performance is satisfactory and meeting desired requirements. In some instances, S&A Centers serve only a specific military department or office, while in other cases an FFRDC may serve a range of DOD entities. For example, RAND Arroyo Center broadly supports the analytic requirements of the Army in order to provide timely advice to help senior Army leadership make informed policy choices. Accordingly, the RAND Arroyo Center sponsoring agreement with the Department of the Army provides that the scope of RAND Arroyo Center work is to support Army sponsors throughout the Army requiring comprehensive analytical support. In contrast, the Institute for Defense Analyses (IDA) and RAND National Defense Research Institute serve DOD more broadly on national security issues. For example, according to IDA’s sponsoring agreement with DOD’s Office of the Under Secretary of Defense for Acquisition and Sustainment, the primary mission of IDA is to assist the Office of the Secretary of Defense and other Defense organizations in addressing important national security issues, particularly those requiring scientific and technical expertise. DOD manages the overall level of FFRDC work using a metric known as staff years of technical effort (STE), which is roughly equal to the work of one employee working for 1 year. Congress typically sets an annual limitation on the STE that may be funded for DOD FFRDCs to support non-intelligence programs on behalf of the agency (hereafter, Defense STE). Between fiscal years 2013 to 2017, Congress established an annual ceiling of 5,750 Defense STE available to DOD, of which 1,125 could be allocated to S&A Centers. In fiscal year 2018, Congress raised the ceiling on Defense STE to 6,030; however, the limit on S&A Centers remained unchanged. In managing Defense STE, DOD: consolidates annual Defense STE requirements for each fiscal year based on projected primary sponsor requirements and submits STE requirements to Congress; establishes Defense STE allocations for each DOD-sponsored FFRDC and provides associated funding limitations to each primary sponsor; monitors Defense STE usage and associated obligations; and provides an annual report to Congress at the end of each fiscal year outlining the Defense STE funded and associated DOD funds obligated for each FFRDC. In addition to Defense STE, FFRDCs may support DOD intelligence activities under the Military Intelligence Program and the National Intelligence Program. Oversight for STE usage for these programs is provided by the Office of the Under Secretary of Defense for Intelligence and Office of the Director of National Intelligence, respectively. Military Intelligence Program and National Intelligence Program STE funding may not be used to support Defense STE requirements. In October 2008, we reported that Congress implemented the Defense STE ceiling during the 1990s in response to concerns that DOD was inefficiently using its FFRDCs. In addition, we found that STE ceilings aimed to ensure that FFRDC work was appropriate and that resources, which were limited, were being used on DOD’s highest priorities. In December 2018, we reported that officials in the Office of the Secretary of Defense’s Studies and FFRDC Management Office stated that the ceiling significantly constrains the use of DOD’s FFRDCs and that DOD customer demand for FFRDC services is significantly greater than the annual ceiling set by Congress. Further, officials indicated at that time that FFRDC-related work must be deferred to later years when the limits are reached, since there are no other legally compliant alternatives capable of fulfilling these requirements. We did not make any recommendations related to this issue. Following the completion of FFRDC work, the primary sponsor, with assistance from the work sponsor, reviews FFRDC performance in written assessments via questionnaires. In addition, the primary sponsor assesses FFRDC performance annually, addressing the technical quality, responsiveness, value, and timeliness of the work performed. Some of the information from the annual reviews may be used in support of the comprehensive review, such as to demonstrate the efficiency and effectiveness of the FFRDC in meeting the primary sponsor’s needs. From fiscal years 2013 through 2018, total DOD obligations to the 10 DOD-sponsored FFRDCs generally increased annually from about $2.7 billion in fiscal year 2013 to approximately $3.2 billion in fiscal year 2018. Approximately 70 percent of total annual DOD obligations to DOD- sponsored FFRDCs between these fiscal years went to support non- intelligence programs and were comprised of DOD obligations associated with utilized Defense STE, or Defense STE obligations. Specifically, DOD Defense STE obligations ranged from about $1.9 billion in fiscal year 2013 to $2.2 billion in fiscal year 2018, with S&A Centers representing approximately 18 percent of these obligations. In addition to DOD Defense STE obligations, about 30 percent of total DOD obligations to DOD-sponsored FFRDCs between fiscal years 2013 through 2018 went towards other FFRDC-related activities and costs, such as intelligence program activities through the Military Intelligence Program and National Intelligence Program and capital equipment costs. Figure 1 shows DOD obligations by fiscal year to DOD-sponsored FFRDCs. For fiscal years 2013 to 2018, the FFRDCs we reviewed in-depth—DOD’s S&A Centers—collectively accounted for about 18 percent of DOD Defense STE obligations annually, whereas Research and Development Laboratory FFRDCs and Systems Engineering and Integration Centers accounted for 27 and 55 percent, respectively (see figure 2). DOD Defense STE obligations to S&A Centers rose from about $320 million in fiscal year 2013 to approximately $380 million in fiscal year 2018 totaling about $2.3 billion during this period. Within each S&A Center, obligations remained relatively constant over the 6 years, with obligations for some FFRDCs higher than obligations for others. For example, on average DOD obligated about $134 million annually to IDA between fiscal years 2013 through 2018, whereas DOD obligated approximately $39 million annually to RAND Arroyo Center during this timeframe. DOD Defense STE obligations to S&A Centers were almost entirely awarded to support research projects requested by DOD. In some cases, work was done in response to congressional direction. For example, RAND Project Air Force (PAF) initiated a fiscal year 2017 independent review and assessment of the Ready Aircrew Program to respond to requirements outlined in the National Defense Authorization Act of Fiscal Year 2017. Overall, according to information provided by DOD sponsors and FFRDC representatives, between fiscal years 2013 through 2017, S&A Centers began work on about 600 research projects annually on behalf of DOD, with about 93 percent of these projects initiated at the request of DOD. The dollar value of these S&A Center projects ranged from about $2,000 to $11 million between fiscal years 2013 through 2017. Sponsoring agreements note and primary sponsors reported in comprehensive reviews that S&A Centers are utilized because of DOD’s strategic relationships with FFRDCs. As described in the FAR, FFRDCs meet special, long-term research or development needs of the sponsoring agencies. Sponsoring agreements with S&A Centers outline the importance of strategic relationships that have helped these FFRDCs to develop and maintain in-depth knowledge of their sponsors’ and users’ programs and operations. In our review of S&A Center sponsoring agreements and comprehensive reviews, we identified that strategic relationships between sponsors and S&A Centers are generally characterized by the stability of long-term capabilities in subject areas important to DOD, access to sensitive and proprietary data and information, and objectivity in the form of freedom from conflicts of interest. These documents also indicate that strategic relationships enable S&A Centers to maintain in-depth knowledge of work sponsor programs and operations. For example, in the 2015 sponsoring agreement between the Army and RAND Arroyo Center, the sponsoring agreement states that both the Army and RAND Arroyo Center share a strategic relationship, and that the RAND Arroyo Center is structured to maintain strong analytic expertise related to Army policy and operations. In addition, the sponsoring agreement outlines the importance of RAND Arroyo Center’s continuity of expertise to the Army, long-term research efforts, and high-quality staff. Office of the Under Secretary of Defense for Acquisition and Sustainment (OUSD(A&S)) officials told us that S&A Centers are oftentimes chosen to perform work for DOD due to unique long-term strategic relationships with sponsors for independent and knowledgeable expertise within core competencies to address sponsors’ specific analytic requirements. In some cases, these strategic relationships date back to World War II. Regarding these strategic relationships, OUSD(A&S) officials also told us the primary sponsor has a degree of control over an FFRDC’s business affairs that can limit the risks of organizational conflicts of interest at FFRDCs. DOD also cited strategic relationships between DOD and S&A Centers as a reason for using S&A Centers when initiating projects we reviewed. For example: Prior to initiating a 2016 assessment of the impact of long-term fiscal trends on Army capabilities, the Army determined RAND Arroyo Center was uniquely qualified to conduct the research because the project required knowledge of defense planning scenarios that would have given an industry contractor a competitive advantage, potentially leading to a conflict of interest. The Army also identified RAND Arroyo Center’s long-standing expertise on security cooperation when requesting a fiscal year 2013 study on assessing value in Army security cooperation as a reason RAND Arroyo Center was uniquely suited to complete the study. Navy primary sponsor officials identified the long-term relationship between CNA, the FFRDC, and the Navy, which has led to broad subject-matter expertise in naval matters, as a reason they used CNA for the fiscal year 2016 study on the assessment of the effects of possible policy changes to a career track program for military officers trained to work with other military services. CNA leadership chose two researchers to lead the effort, one of which had prior experience in this area. An OUSD(A&S) official cited RAND National Defense Research Institute’s (NDRI) longstanding portfolio on military workforce issues as a reason for using RAND NDRI for a fiscal year 2017 study on the military’s 40-year pay table. An official told us that RAND NDRI’s prior work in this area would allow for a quicker response and more in- depth analysis to respond to the work request. In addition to the strategic relationships, sponsoring agreements and comprehensive reviews cited FFRDC core competencies as key factors in establishing and continuing relationships with S&A Centers, which is consistent with provisions outlined in DOD Instruction 5000.77. The DOD instruction states that FFRDCs maintain long-term competencies and capabilities to meet DOD needs that cannot be met by government or other private sector resources as effectively, and these competencies derive from the sponsor’s analytical requirements. In general, core competencies include expertise in engineering, research and development, and analysis, and are further described in FFRDC sponsoring agreements and comprehensive reviews. For example: The Navy 2015 comprehensive review of CNA states that CNA satisfies the Navy’s need for highly specialized skills and competencies in Navy warfighting and warfighting support— particularly research staff from CNA’s studies and analyses division— to accomplish their operational missions. The 2019 sponsoring agreement between DOD’s OUSD(A&S) and IDA outlined the need for technical and analytical support, citing IDA’s four core competencies as the scope of work of the FFRDC: systems and capabilities evaluations, technology assessments, force and strategy assessments, and resource and support analyses. The Army 2010 and 2014 comprehensive reviews of RAND Arroyo Center stated that RAND Arroyo Center has currency in all requisite Army proficiencies, provides a multidisciplinary research process that integrates and applies competencies with an assurance of consistently high quality, and also has the ability to apply competencies with expedience when an Army request for analytic support requires a quick response. OUSD(A&S)’s sponsoring agreement with RAND NDRI defines RAND NDRI’s research capability and core competencies such as, but not limited to, global and national security, defense acquisition, intelligence, and system risk management as means to satisfy essential needs of the FFRDC’s work sponsors for policy research and analysis. Primary sponsor officials we spoke with also told us that FFRDC staff skills and knowledge related to FFRDC core competencies are important to DOD. For example: Navy officials said CNA is uniquely suited to perform work for the Navy due to CNA’s core competencies relating to maritime defense analysis and how those competencies align with Navy goals and requirements. Army officials told us that RAND Arroyo Center staff has extensive background knowledge and analytical skills relating to reserve affairs, manpower policy, and war game analysis, among other things, in providing work for the Army. Air Force officials told us that RAND PAF has robust knowledge of Air Force processes and maintains top staff and researchers in each core competency. OUSD(A&S) officials also told us that sponsors and FFRDCs have a relationship in which sponsors rely on FFRDCs for independent and knowledgeable expertise within their core competencies to address sponsors’ analytic requirements. As shown in figure 4, DOD’s S&A Center primary sponsors identified 3 to 15 core competencies in their sponsoring agreements with each S&A Center. DOD cited FFRDCs’ core competencies as factors that contributed to using S&A Centers when initiating projects we reviewed, as provided by DOD instruction. For example: When initiating a fiscal year 2016 CNA assessment on the effects of possible policy changes to a career track program for military officers who are trained to work with other military services, DOD’s Office of the Under Secretary of Defense for Personnel and Readiness cited CNA’s core competencies of analysis of maritime resources; maritime program planning; and maritime policies, strategies, and doctrines as justification for using CNA to perform the work, among other things. In initiating a fiscal year 2014 IDA analysis on satellite ground control, DOD’s Office of the Deputy Assistant Secretary of Defense for Space and Intelligence cited IDA’s core competencies related to technology, such as systems and capabilities evaluations, as justification for using IDA for the research. DOD reports that it uses studies and analyses to inform decision-making; shape guidance, policies, and training; and identify opportunities to save time and money. Inform decision-making. For example, a 2016 study conducted by the RAND Arroyo Center on linking Army cost and performance found that the Army needed an updated tool to inform more strategic allocation of its resources. Among other things, the study contributed to updated strategies to measure the Army’s performance regarding force structure and readiness as well as the cost implications for these activities. According to an Army official, the study contributed to the development of updated Army metrics for cost and other performance indicators. In another example, a 2013 research project conducted by RAND NDRI on effectiveness measures of a DOD program to reduce the threat from infectious diseases and biological weapons developed and recommended two sets of metrics to improve program evaluation efforts. According to OUSD(A&S) officials, DOD used the recommended metrics to develop program performance measures. Shape guidance, policies, and training. For example, a 2013 study conducted by RAND NDRI on the root causes related to DOD weapons programs cost overruns found, among other things, that DOD needed to re-examine its assumptions when estimating a program’s cost, schedule, and technical performance. OUSD(A&S) officials told us the study contributed to DOD’s decision to update its policy, processes, management practices, and training curriculum so as to improve estimates. In another example, a 2013 study conducted by RAND Arroyo Center on the value of security missions conducted by the Army’s geographically aligned forces found that the use of these forces improved the efficiency of security planning and preparation and recommended a range of process and planning improvements for the Army. According to an Army official, the Army used several of the recommendations to update guidance for preparation and planning for future missions involving regionally aligned forces. Identify opportunities to improve efficiency. For example, a 2013 study conducted by RAND Arroyo Center on marketing and resources needed for Army recruiting efforts identified strategies aimed at optimizing the Army’s annual spending, estimated at nearly $1 billion for recruiters, enlistment bonuses, and television advertising. An Army official said that the Army has used the recruiting tool developed by RAND Arroyo Center for this study to make decisions and the Army estimates the tool can reduce costs by potentially hundreds of millions of dollars annually. In another example, DOD reported in its 2015 comprehensive review of RAND PAF that a 2010 study conducted by RAND PAF on aircraft maintenance at centralized repair facilities found that these facilities should be consolidated. According to the Air Force primary sponsor, this study helped the Air Force make decisions that led to saving up to $300 million annually as well as saving time on aircraft inspections. In terms of assessing the outcomes of research, we found that DOD primary sponsors took steps to assess the value of S&A Center research and the centers. The DOD instruction requires that primary sponsors assess the efficiency and effectiveness of the FFRDC in meeting DOD needs in comprehensive reviews, including a review and summary of FFRDC accomplishments and their effectiveness utilizing factors such as quality and timeliness of the work produced and value of projects assessed. Additionally, the DOD instruction provides that the factors of technical quality, responsiveness, value, and timeliness be addressed in annual performance reviews. DOD’s FFRDC Management Plan—which preceded the DOD instruction and was in effect until the DOD instruction became effective in January 2018—also required primary sponsors to annually assess the value of FFRDC performance, among other factors, and include summaries of these annual assessments in comprehensive reviews. Primary sponsors generally assess the value of S&A Center research through annual performance reviews (through performance evaluation questionnaires to solicit feedback from work sponsors) and comprehensive reviews. To monitor the execution of research projects, primary sponsors regularly solicit work sponsor input regarding S&A Centers’ performance, including the value, technical quality, responsiveness, and timeliness of the work performed. Time frames for soliciting this input vary by primary sponsor but most do this annually. These questionnaires include one or more sections for work sponsors to add comments about S&A Center work and allow work sponsors to rate S&A Center performance. Some of these questionnaires use a numerical scale. For example, the Air Force questionnaire sent to RAND PAF work sponsors asks respondents to rate project value using a scale from 1 through 10, with 1 indicating “very poor” and 10 “very good.” The OUSD(A&S) questionnaire sent to IDA work sponsors asks respondents to rate the value of IDA’s work and results using a scale from 1 through 5, where 1 symbolizes either “strongly agree” or “outstanding performance” and 5 symbolizes “strongly disagree” or “poor performance.” FFRDC primary sponsors conduct comprehensive reviews at least every 5 years to, among other things, identify the accomplishments made by each FFRDC. In August 2014, we reported that DOD officials described the comprehensive review process as an opportunity to take a broad assessment of the FFRDC and its key competencies beyond the annual assessments of FFRDCs. Included in these comprehensive reviews is a summary of FFRDC accomplishments and effectiveness in meeting work sponsors’ needs since the last comprehensive review. In our examination of the most recent comprehensive reviews for each of the five S&A Centers, we found that the comprehensive reviews summarize the results from the performance evaluation questionnaires and assessed the value of the research in varying ways. For example, the Army questionnaire to RAND Arroyo Center work sponsors assessed value in terms of whether a project was worth the investment monetarily. OUSD(A&S) questionnaires sent to work sponsors assessed the value of IDA work in relation to whether the results were useful, consistent with the level of effort, and if IDA brought competence, expertise, and helpful perspectives to the issues. The Army reported in the 2014 comprehensive review of RAND Arroyo Center that between fiscal years 2010 through 2013, work sponsors provided overwhelmingly positive results that RAND Arroyo Center performance was “worth the level of effort.” DOD’s Office of the Under Secretary of Defense for Acquisitions, Technology, and Logistics—RAND NDRI’s primary sponsor prior to the DOD reorganization in 2018—reported in the 2014 comprehensive review of RAND NDRI that in fiscal year 2013, work sponsors provided overwhelmingly positive results that RAND NDRI performance provided long-term value. We also found that comprehensive reviews included anecdotal examples of how DOD used S&A Center research. For example, the Army 2014 comprehensive review of RAND Arroyo Center highlighted 53 of 114 research projects completed between fiscal years 2010 through 2013 to demonstrate how RAND Arroyo Center work met Army research requirements. Likewise, the Air Force primary sponsor’s 2015 comprehensive review of RAND PAF highlighted 28 of 207 research projects completed between fiscal years 2010 and 2014 to demonstrate how the Air Force leveraged RAND PAF work to improve efficiency in the department. An Air Force official told us that RAND PAF, and not the Air Force, collected 28 project examples for the purposes of the comprehensive review. Another potential way to assess the outcomes of research is to track to what extent a research project’s recommendations were implemented, and how. Neither DOD nor primary sponsors currently track the implementation of S&A Center research project recommendations. While primary sponsors are not tracking recommendations, in 2015 one of the S&A Centers—RAND PAF—began tracking recommendations made to the Air Force. According to a RAND PAF representative, the tracking system captures the issue, approach, conclusions, opportunities, and outcomes for each completed project. A RAND PAF representative told us that tracking recommendations is useful for demonstrating the value that RAND PAF provides the Air Force. In April 2019, a Navy official told us that the Navy is working on a database to track CNA reports, including recommendations, report topic, work sponsor, and project funding, among other things, to prevent duplication of requests. The Navy official said this effort is expected to be completed in 2019. Both OUSD(A&S) and Army officials told us that while they do not currently track recommendations, they are considering doing so as part of their oversight efforts. Further, Army officials told us that it is important for the sponsor that implements the recommendations to track how and whether that information was used. While tracking recommendations is useful according to some primary sponsors, some DOD officials cautioned that tracking recommendations would not provide insights into the overall value across all S&A Center research. DOD officials told us that recommendations are only one potential outcome of S&A Center research and that the value of a study may not be specifically linked to a recommendation. For example, Navy officials said that CNA’s projects may present the Navy with options and associated courses of action rather than formal recommendations, and DOD officials also told us that S&A Center work can provide value to DOD that is not always represented by recommendations, such as presentations or research aimed at contributing to the understanding of a particular issue, but without specific recommendations. In February 2019, DOD’s Office of the Under Secretary of Defense for Personnel and Readiness issued a memorandum related to the oversight of the Personnel and Readiness Studies and Analysis program. The memorandum tasked the program director with developing a studies and analysis program framework that improves accountability for project results and the implementation of study recommendations. Personnel and Readiness also issued a template “action memo” providing for an executive summary of completed projects as well as implementation plans delineating recommendations made, implementation approach, and plan of action for each recommendation. According to a senior Personnel and Readiness official, work sponsors with reports that were completed or published since September 2018 are subject to these actions. This official noted that the purpose is to increase accountability of the Personnel and Readiness staff regarding the use of FFRDCs and to develop an overall picture of the value proposition of FFRDC research. It is too soon to tell to what extent these memorandums will affect DOD’s insights on its implementation of S&A Center recommendations. What are Conflicts of Interest? A Personal Conflict of Interest exists when an individual employed by an organization is in a position that could materially influence research findings or recommendations and may lack objectivity due to their financial interests, personal activity, or relationships. An Organizational Conflict of Interest exists when, because of other interests or relationships, an entity is unable or potentially unable to render impartial assistance or advice to the government or the entity might have an unfair competitive advantage. The Federal Acquisition Regulation (FAR) requires an FFRDC to conduct its business in a manner befitting its special relationship with the government and to be free from conflicts of interest. To perform its responsibilities to the sponsoring agency, an FFRDC and its employees have access beyond that which is common in a normal contractual relationship, including access to sensitive and proprietary data and information, equipment, and property. To accomplish this, the FAR and DOD instruction state that an FFRDC must be free from conflicts of interest and fully disclose financial and outside interests to the sponsoring agency. Conflicts of interest can be personal or organizational. Personal conflicts of interest can be, but are not limited to, financial interests of the employee or close family members, other employment, gifts, consulting relationships, other forms of research funding or support, investment in the form of stock or bonds ownership, real estate, or business ownership. Additionally, the DOD instruction outlines steps FFRDC parent organizations should take to prevent and mitigate conflicts of interest. These steps include, but are not limited to, having procedures in place to screen employees for potential conflicts of interest; requiring disclosure of financial and other interests that might affect the employee’s objectivity; establishing policies and procedures to protect proprietary, privileged, and sensitive information from disclosure; and reporting any conflicts of interest to the applicable contracting officer or contracting officer’s representative and the primary sponsor as soon as it is identified. See figure 5 for DOD’s conflict of interest elements outlined in the DOD instruction that primary sponsors are to require from FFRDC parent organizations. Each of the five S&A Centers we reviewed has corporate-wide conflict of interest policies and practices which incorporate various key elements of the DOD instruction. For example, all S&A Center policies we reviewed have measures that require personnel to protect proprietary, privileged, and sensitive information. S&A Center representatives told us they undertake various approaches in practice that meet key elements in the DOD instruction in order to ensure they operate in the public interest with objectivity and independence. For example: Reviewing all personnel annually or on a task-by-task basis for conflicts of interest. Generally, representatives we spoke with from the five S&A Centers address conflicts of interest annually or task-by- task, which is an option outlined in the DOD instruction. For instance, RAND representatives said they perform task-by-task, instead of annual, conflict of interest reviews because staff do not know which projects they will be working on during the year. In addition, IDA and RAND representatives told us they screen employees upon hire as well as when an employee initiates a new project, and both IDA and RAND have automated their screening processes. IDA representatives explained that their automated tool screens personnel at the initiation of each project, by including, for example, a process to determine if staff assigned to a project have any affiliations with industry or companies and competitors in the particular field of study. If staff or members of their households do have affiliations, IDA may issue a waiver if the financial interest (such as but not limited to stocks, stock options, and bonds) in a single company is below $15,000, the threshold for disclosure outlined in the DOD instruction. IDA representatives also told us that IDA staff are required to self- report any changes to previous financial interest disclosures during the year. In another example, RAND representatives said their automated conflict of interest tool screens for conflicts of interest by comparing areas of work RAND performs to similar areas in the private sector. Additionally, the system will identify any staff that have not submitted a conflict of interest statement within a year. Providing initial and annual conflict of interest training for all personnel. S&A Center representatives told us that they perform training related to or specifically covering conflicts of interest in varying ways. IDA’s corporate-wide conflict of interest policy includes initial and annual conflict of interest training elements, as outlined in the DOD instruction. For example, IDA’s policy states that all employees are to participate in conflict of interest training upon initial hire, and in annual refresher training thereafter. The other four S&A Centers did not explicitly include annual conflict of interest training in corporate-wide policies, but representatives told us they provide annual ethics training, which includes training on conflicts of interest, to their employees. For example, CNA representatives told us they provide ethics and conflicts of interest training to staff, which is required by their contract with the Navy. CNA representatives told us that if CNA staff do not complete the required training, the staff will be blocked from accessing CNA’s time card system and will not receive pay until the training is complete. In another example, RAND representatives told us they have annual training that covers ethics, conflicts of interest, and culture and discrimination issues for newly hired staff. Representatives from each of the S&A Centers told us they attempt to mitigate potential conflicts of interest as soon as the potential conflicts become known and before they become a reportable conflict. For example, CNA representatives told us that in one instance, a CNA employee’s spouse worked for the Navy and CNA mitigated this potential conflict by transferring the employee to another project where the relationship did not pose a potential conflict. In another example, when a RAND employee inherited stock in the middle of a project, a potential conflict of interest was mitigated by the employee selling the inherited stock. In another instance, a RAND employee was initially staffed to a project related to an area of work a spouse worked on commercially, and RAND mitigated the potential conflict by recusing the employee from the project. We provided a draft of this report to DOD for review and comment. In its comments, DOD concurred with our findings. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or LudwigsonJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Table 2 provides detailed information on the 22 projects we selected for review. In addition to the contact named above, Janet McKelvey (Assistant Director), Andrew Burton (Analyst-in-Charge), Mallory Bryan, Lisa Fisher, and Jordan Kudrna made key contributions to this report. Additional assistance was provided by Marie Ahearn, Pete Anderson, Jenny Chanley, Joseph Cook, Julia Kennon, Tind Shepper Ryen, and Roxanna Sun.", "summary": "For decades, the government has contracted and entered into agreements to sponsor academic, nonprofit, or private organizations to operate FFRDCs. DOD military departments and other DOD components sponsor 10 FFRDCs to help develop innovative solutions to diverse national security threats. Five FFRDCs—referred to as S&A Centers—aim to provide independent analyses to support DOD policy development. Federal regulation and DOD guidance specify sponsors' oversight activities, including the establishment, use, and review of FFRDCs. A Senate Armed Services Committee report included a provision that GAO review DOD's use of FFRDCs. This report describes, among other objectives: (1) DOD obligations (in dollars) to DOD's FFRDCs from fiscal years 2013 through 2018; (2) factors that led DOD to use S&A Centers for research; and (3) how DOD used this research. GAO analyzed obligation data for DOD's 10 FFRDCs. GAO focused further review on DOD's five S&A Centers that primarily provide studies and analysis. GAO analyzed sponsoring agreements, comprehensive reviews, and 22 S&A Center research projects selected based on factors such as obtaining a mix of project costs, and interviewed DOD and FFRDC representatives. From fiscal years 2013 through 2018, the Department of Defense (DOD) obligated about $2 billion annually to 10 DOD-sponsored Federally Funded Research and Development Centers (FFRDC), excluding obligations related to two intelligence programs and capital equipment costs (such as antenna or radar systems). Of these obligations, roughly $400 million annually went to a subset of five FFRDCs called Study and Analysis (S&A) Centers. Note: Obligation amounts were not adjusted for inflation and totals may be affected by rounding. a Numbers in parentheses refer to the number of FFRDCs within each category. DOD primarily cited strategic relationships between the sponsor (the agency responsible for the overall use of the FFRDC) and the FFRDC and the core competencies of the FFRDC as factors when sponsoring S&A Centers and initiating projects. For example: Strategic relationships. The Army determined that an S&A Center was uniquely qualified to conduct a research project that required knowledge of defense planning scenarios, noting that awarding the project to an industry contractor would have given that contractor a competitive advantage. Core competencies. The Center for Naval Analyses has core competencies in Navy policy, strategy, and doctrine, among other things. S&A Centers perform hundreds of research projects annually on behalf of DOD, and DOD reported using them to inform decisions, shape guidance, and identify opportunities to improve efficiency. For example, one S&A Center's study on the causes of weapons system cost overruns found DOD needed to re-examine its assumptions when estimating program cost, schedule, and performance. DOD officials told us the study contributed to policy, process, and training updates.", "document_type": "gao"}
{"report": "VA’s mission is to promote the health, welfare, and dignity of all veterans in recognition of their service to the nation by ensuring that they receive medical care, benefits, social support, and lasting memorials. In carrying out this mission, the department manages one of the largest health care delivery systems in the United States that provides enrolled veterans with a full range of services. These services may include primary care; mental health care; and outpatient, inpatient, and residential treatment. The Veterans Health Administration (VHA), one of the department’s three major components, is responsible for overseeing the provision of health care at all VA medical facilities. Information technology (IT) is widely used and critically important to supporting the department in delivering health care to veterans. As such, VA operates and maintains an IT infrastructure that is intended to provide the backbone necessary to meet the day-to-day operational needs of its medical centers and other critical systems supporting the department’s mission. The infrastructure is to provide for data storage, transmission, and communications requirements necessary to ensure the delivery of reliable, available, and responsive support to all VA staff offices and administration customers, as well as veterans. The Office of Information and Technology (OIT) is responsible for managing the majority of VA’s IT- related functions. The office provides strategy and technical direction, guidance, and policy related to how IT resources are to be acquired and managed for the department. VA provides health care services to approximately 9 million veterans and their families and relies on its health information system—VistA—to do so. VistA has been essential to the department’s ability to deliver health care to veterans. It was developed based on the collaboration between staff in the VA medical facilities and VHA IT personnel. Specifically, clinicians and IT personnel at the various VA medical facilities collaborated to define the system’s requirements and, in certain cases, carried out its development and implementation. As a result of these efforts, the system has been in operation since the early 1980s. VistA supports a complex set of clinical and administrative capabilities. It is comprised of an architecture that ties together servers and personal computer workstations with various applications within VA facilities and the supporting infrastructure, such as data centers, storage, and messaging technologies. The core system and database code are programmed in the MUMPS programming language. Among other things, VistA contains an EHR for each patient and supports clinics and medical centers. In addition, the system provides functionality beyond the EHR and exchanges information with many other applications and interfaces. For example, the system also provides the functionality of a time and attendance program, asset management system, library, and billing system, among other things. Users interact with VistA through a number of interfaces that connect stored health data. These interfaces enable the system to communicate (send or exchange data) with other VA systems, as well as with other federal agencies (e.g., DOD), health information exchange networks, and COTS products. According to OIT officials, applications either interface with VistA directly through a messaging protocol or extract data from the system via a reporting mechanism. The Computerized Patient Record System is a graphical user interface to VistA that runs on workstations, laptops, and tablets and enables the department to support clinical workflows. Specifically, the Computerized Patient Record System enables the department to create and update an individual EHR for each VA patient. Among other things, clinicians can order lab tests, medications, diets, radiology tests, and procedures; record a patient’s allergies or adverse reactions to medications; request and track consults; enter progress notes, diagnoses, and treatments for each encounter; and enter discharge summaries. According to VHA officials, there are also more than 100 COTS products that interface with VistA. In addition to these commercial products, medical equipment or devices at local facilities may also require interfaces to the system, and these vary on a site-by-site basis. Over the last several decades, VistA has evolved into a technically complex system that supports health care delivery at more than 1,500 locations, including VA Medical Centers, outpatient clinics, community living centers, and VA vet centers. Customization of the system by local facilities has resulted in about 130 clinical versions of VistA—referred to as instances. According to the department, no two VistA instances are identical. Further, each instance is comprised of over 27,400 routines (executable modules of code), which are logically grouped into products or modules. VistA products or modules can also be comprised of one or more software applications that support health care functions, such as providing care coordination and mental health services. The department reported that there are approximately 140 to 200 products or modules that comprise the system. The 130 clinical instances of VistA are operated from four regional VA data centers. Users interact with the system through the Computerized Patient Record System. Aggregated clinical data from every instance of the system are located on servers hosted at VA’s National Data Center. Over time, VA has identified the need for enhancements and modifications to VistA in order to ensure that the system keeps up with current technology and health care delivery. However, according to the department, the system has become difficult and costly to maintain. This is a result of, for example, being programmed in MUMPS, a language for which there is a dwindling supply of qualified software developers. It is also due to years of decentralized customization of the system by staff members who were permitted to develop and implement applications at the local level. OIT and VHA serve as the technical and functional leaders, respectively, for the department’s health care delivery and, together, they have worked to develop and maintain VistA for decades. Specifically, OIT is responsible for managing the majority of VA’s IT-related functions. The office provides strategy and technical direction, guidance, and policy related to how IT resources are to be acquired and managed for the department. According to the department, OIT’s mission is to collaborate with its business partners (such as VHA) and provide a seamless, unified veteran experience through the delivery of state-of-the-art technology. The Assistant Secretary for Information and Technology/Chief Information Officer (CIO) serves as the head of OIT and is responsible for providing leadership for the department’s IT activities. The CIO also advises the Secretary regarding the execution of VA’s IT systems appropriation, consistent with the Federal Information Technology Acquisition Reform Act. For fiscal year 2019, the department has been appropriated $4.1 billion for IT. According to VA’s budget documentation, about $1.2 billion of this amount is intended to support IT staffing and associated costs for approximately 8,100 full-time employees. VHA provides information and expertise to OIT to support the department’s health-related information systems. For example, VHA officials help identify clinical and business needs used to inform IT requirements development. The Under Secretary for Health is the head of VHA and is supported by the Principal Deputy Under Secretary for Health, four Deputy Under Secretaries for Health, and nine Assistant Deputy Under Secretaries for Health. After nearly 2 decades of pursuing multiple efforts to modernize VistA, in June 2017, the former VA Secretary announced that the department planned to acquire the same EHR system that DOD is acquiring—Cerner Millennium. According to the department, it has chosen to acquire this product because Cerner Millennium should allow VA’s and DOD’s patient data to reside in one system, thus, potentially reducing or eliminating the need for manual and electronic exchange and reconciliation of data between two separate systems. Accordingly, the department awarded an indefinite delivery, indefinite quantity contract to Cerner Corporation in May 2018 for a maximum amount of $10 billion over 10 years. Cerner is to replace the 130 instances of VistA with a standard COTS system to be implemented across VA. This new system is to support a broad range of health care functions including acute care, clinical decision support, dental care, and emergency medicine. When implemented, the new system will be expected to become the authoritative source of clinical data to support improved health, patient safety, and quality of care provided by VA. The Electronic Health Record Modernization (EHRM) program is responsible for managing the Cerner contract implementation. For fiscal year 2019, the program was appropriated about $1.1 billion for planning and managing the transition from VistA to Cerner. Further, the department has estimated that an additional $6.1 billion in funding, above the Cerner contract amount, will be needed to fund additional project management support supplied by outside contractors, government labor costs, and infrastructure improvements over the 10- year contract period. VA plans to deploy the new EHR system at three initial operating capability sites within 18 months of October 1, 2018, with a phased implementation of the remaining sites over the next decade. Each VA medical facility is expected to continue using VistA until the new system has been deployed. The three initial deployment sites, located in the Pacific Northwest, are the Mann-Grandstaff, American Lake, and Seattle VA Medical Centers and related clinical facilities that operate the same instances of VistA. These are the first locations where the system is expected to “go live.” The task order to deploy the Cerner system at the three initial sites provides a detailed description of the steps Cerner needs to take in order to reach initial operating capability at the Mann-Grandstaff site in March 2020, and at the Seattle and American Lake sites in April 2020. According to the schedule, the initial operating capability sites are expected to be operational by July 2020. In order to maintain internal control activities over an IT system and its related infrastructure, organizations should be able to define physical and performance characteristics of the system, including descriptions of the components and the interfaces. Further, consistent with GAO’s Cost Estimating and Assessment Guide, a comprehensive system definition should identify customization and the environment in which the system operates. While defining a complex IT system can be challenging, having an adequate understanding of its characteristics will better position the organization to comprehensively project and account for costs over the life of a system or program as well as identify specific technical and program risks. Definition of VistA remains important because VA plans to continue using the system during the department’s decade-long transition to the Cerner system. VA maintains multiple documents and a database that describe parts of VistA, including various components and interfaces. However, despite these existing sources, OIT officials acknowledged that there is no comprehensive definition of the VistA system. Consequently, VA has completed a number of efforts to better define VistA and understand the environment in which it operates and additional work is planned in the future. Specifically, VA has documented descriptions of the system, including the components that comprise it. These descriptions are documented in multiple sources: the VA Monograph, VA Systems Inventory, and VA Document Library. The VA Monograph is a document maintained by OIT that provides an overview of VistA and non-VistA applications used by VHA. According to VHA officials, the VA Monograph is the primary document that describes the components of the system. The Monograph describes VistA in terms of modules. For modules identified, including VistA modules, information such as the associated business functions, VA Systems Inventory identification number, and a link to the VA Document Library for additional technical information are provided. The VA Systems Inventory is a database maintained by OIT that identifies current IT systems at the department, including systems and interfaces related to VistA. For systems identified, the database includes information such as the system name, the system status (i.e., active, in development, or inactive), and related system interfaces. The VA Document Library is an online resource for accessing documentation (i.e., user guides and installation manuals) on the department’s nationally released software applications, including VistA. VA has taken additional steps to further define the system. For example, EHRM program officials recognized the need to further understand the customization of VistA components at the various medical facilities and have conducted analyses to do so. These analyses include: Variance analysis: As part of its VistA Evolution program, which has focused on standardizing a core set of VistA functionality, the department implemented a process to compare the instances of VistA installed at sites to the Enterprise Standard version. The results of this analysis allowed the department to assess the criticality of each variance, which is expected to help with VA’s transition to the Cerner system. Module analysis: EHRM program subject matter experts undertook an analysis that involved reviewing and assessing capabilities provided by VistA modules. This analysis enabled department officials to determine whether the capability provided by a VistA module could be provided by the Cerner system, or whether another COTS solution would be required to support this function going forward. Visual mapping: EHRM program officials also directed an analysis that involved developing a notional visual mapping of VA’s health care applications, components, and supporting systems within the health delivery environment. The results of this analysis provided a description of the current state of one instance of VistA and the VA health environment, which is intended to inform the department of possible opportunities for business process and IT improvements as it proceeds with the Cerner acquisition. Nevertheless, even with these analyses, VA has not yet fully defined VistA, including, for example, identifying performance characteristics of the system and describing the environment in which it operates. The department’s three sources that describe VistA and the additional analyses undertaken do not provide insight into site specific customizations of the system. For example, the VA Monograph does not include information on module customization at local facilities. In addition, according to OIT officials, the systems inventory does not reflect differences among the 130 different instances of VistA and does not take into consideration regional and local customizations of related components. Further, the visual mapping analysis noted that there was not full insight of the intertwined structure of data and applications or the various local customizations of VistA. EHRM program officials stated that they have not been able to fully define VistA and understand all local customizations due to the decentralization of the development of the system and its evolution over more than 30 years. They explained that VistA’s complexity is partly due to the various instances of the system, compounded by local customizations, which have resulted in differences in VistA instances operating at various facilities. According to EHRM program documentation, Cerner’s contract calls for the company to conduct comprehensive assessments to capture the current state of technical and clinical operations at specific facilities, as well as identify site-specific requirements where the Cerner system is planned to be deployed. As of June 2019, Cerner had completed site assessments for the three initial operating capability sites in the Pacific Northwest and had planned additional assessments at future deployment sites. The initial site assessments included, among other things, an assessment of the unique VistA instances and the environment in which the system operates. The continuation of planned site assessments should provide a thorough understanding of the 130 VistA versions, help the department better define VistA, and position it for transitioning from VistA to Cerner’s COTS solution. When using public funds, an agency must employ effective management practices in order to let legislators, management, and the public know the costs of programs and whether they are achieving their goals. To make those evaluations for a program or for a system as large and complex as VistA, a complete understanding of the system and reliable cost information is required. By following a methodology and utilizing reliable data, an agency can ensure that all costs are fully accounted for, which in turn, better informs management decisions, establishes a cost baseline, and enhances understanding of a system’s performance and return on investment. Fundamental characteristics of reliable costs are that they should be accurate (unbiased, not overly conservative or optimistic), well- documented (supportable with source data, clearly detailed calculations, and explanations for choosing a particular calculation method), credible (identifying any uncertainty or biases surrounding data or related assumptions), and comprehensive (costs are neither omitted nor double counted). Identification of VistA’s costs remains important because VA plans to continue using the system during the department’s transition to the Cerner system over the next decade. VA identified costs for VistA and its related activities adding up to approximately $913.7 million, $664.3 million, and $711.1 million in fiscal years 2015, 2016, and 2017, respectively—for a total of about $2.3 billion over the 3 years. However, the department could not sufficiently demonstrate the reliability of certain costs that were identified. In addition, VA identified other categories of VistA-related costs, but omitted these costs from the total. Of the $2.3 billion total costs for VistA, VA demonstrated that only approximately $1 billion of these costs were reliable. Specifically, OIT officials identified VistA-related costs within seven categories. The officials were able to sufficiently explain why these categories were included in the development and sustainment costs for VistA and how they were documented by the department; the officials also presented detailed source data for our examination. As a result of our review, we determined that the cost data for these seven categories were accurate, well-documented, credible, and comprehensive and, thus, sufficiently reliable. Table 1 provides a summary of the program costs identified for VistA by OIT and VHA for fiscal years 2015 through 2017 that we determined to be reliable. As shown in the table, VA identified costs for the following seven categories for fiscal years 2015 through 2017: VistA Evolution – The VistA Evolution program costs were associated with VistA strategy, system design, product development, and program management. These costs totaled approximately $549.6 million. Interoperability – The Interoperability program focused on sharing electronic health data between VA and non-VA facilities, including private sector providers and DOD. For example, interoperability costs were associated with architecture, strategy, the Interagency Program Office, product development, and program management. These VistA-related costs totaled approximately $140.2 million. Virtual Lifetime Electronic Record (VLER) Health – This program focused on streamlining the transition of electronic medical information between VA and DOD. These VistA-related costs were associated with product development and program management and totaled approximately $81.2 million. Contracts – Contract costs for VistA Evolution included VHA’s obligations associated with workload management, change management, clinical requirements, and clinical interoperability. These VistA-related costs totaled approximately $202.8 million. Intergovernmental personnel acts – Intergovernmental personnel acts are agreements for the temporary assignment of personnel between the federal, state, and local governments; colleges and universities; Indian tribal governments; federally funded research and development centers; and other eligible organizations. These costs accounted for VHA’s need to use outside experts from approved entities for limited periods of time to work on VistA Evolution assignments. The total VistA-related costs were approximately $2.4 million. Memorandums of understanding – According to VHA, memorandums of understanding are agreements used by the administration to obtain the services of personnel between VA entities for VistA-related activities. These agreements accounted for approximately $2.3 million. Pay – Costs in this category included salaries for VHA staff who worked on VistA-related projects as well as travel, training, and supply costs associated with employment. These costs totaled approximately $34.1 million. However, VA was not able to sufficiently demonstrate the reliability of approximately $1.3 billion in costs related to VistA. Specifically, OIT officials identified the additional legacy VistA costs that generally fell into three categories: Legacy VistA: Infrastructure, hosting, and system sustainment – Legacy VistA costs are generally related to the maintenance of fully operational items, such as VistA Imaging and Fileman—two key components related to VistA’s operation. The costs also included obligations for costs related to hosting health data in both VA and non-VA facilities. The OIT officials and subject matter experts estimated these total costs to be approximately $343 million during fiscal years 2015 through 2017. However, we were not able to determine the reliability of these costs because, for example, source data were not well documented; changes in the cost information provided to us during our review indicated that the cost data may not be credible; and subject matter experts were unclear about how to separate VistA costs from non- VistA costs. Related software – Related software costs are associated with the software supporting, or closely integrated with, VistA that were identified by EHRM officials, yet not tracked directly for one of the VistA-related programs. Both OIT and VHA identified software licensing costs as VistA-related obligations. The EHRM program reported these costs to be approximately $389 million in total during fiscal years 2015 through 2017. However, we were not able to determine the reliability of the costs in this category for a variety of reasons, including that source data were not well documented. In addition, VA officials were not clear regarding how the total amounts in each category should be divided between OIT and VHA. Given this confusion, we were not able to determine if the costs were fully accurate or credible. OIT personnel (pay and administrative) – According to EHRM officials, OIT does not track labor costs by program. Instead, the department provided estimations of the amount of salaries paid to OIT government staff working on activities such as VistA Evolution, program management, and overall support of VistA and related applications. OIT personnel costs were estimated by the EHRM program office to be approximately $544 million total during fiscal years 2015 through 2017. However, we were not able to determine the reliability of costs in this category because assumptions made for estimating the personnel and salary costs were not well documented and could not be verified. In addition, VA omitted certain VistA costs from the total costs identified for fiscal years 2015, 2016, and 2017. Specifically, VA omitted the following costs: Additional hosting – OIT officials stated that additional costs related to hosting health data by an outside vendor, as well as hosting backup VistA instances at each of the medical center sites, should also be included in the total costs for VistA; however, VA omitted these costs from the total for fiscal years 2015 through 2017. Specifically, according to the officials, calculating costs for these hosting activities requires subject matter experts to identify equipment, space, utilities, and maintenance costs for resources allocated specifically for VistA. However, the department has not yet developed a methodology to calculate the costs. The officials said they were working on identifying a reliable approach for calculating these costs in the future. Data standardization and testing – OIT officials stated that additional costs related to work on clinical terminology mapping and functional testing were not included in the total costs for VistA for fiscal years 2015 through 2017. This work related to mapping existing clinical data to national standards and making updates to VistA or the Joint Legacy Viewer and included mapping data and building test scripts and reports. OIT officials noted that this work had been critical to the VistA Evolution program, but they did not provide actual cost data in this category. The lack of sufficiently reliable and comprehensive costs indicates that the department is not positioned to accurately report the annual costs to develop and sustain VistA. This is due in part to VA not following a well- documented methodology that describes how the department determined the total costs for the system. In lieu of a methodology, OIT officials said that leadership and staff from the program took efforts to identify and track the cost components and contracts associated with the system. However, they noted that costs associated with VistA were not all clearly labeled as VistA in an IT system and it was necessary to estimate other costs. The officials were also unable to verify how VistA-related costs were separated from other department costs in all areas and subject matter experts were not consistently familiar with the estimation methods employed and how VistA was defined for the purposes of calculating costs. Further, VA officials noted that they were still working on the best approach to identifying and calculating omitted costs. Without documenting the methodology for what costs are to be included and how they were identified and calculated, VA’s total does not accurately reflect the development and sustainment costs for VistA. As a result, the department, legislators, and the public do not have the comprehensive, reliable information needed to understand how much it actually cost to develop and maintain the system. Further, VA does not have the reliable information needed to make critical management decisions for sustaining the many versions of VistA over the next 10 years until the Cerner system is fully deployed. In our report, we are making a recommendation for VA to improve its reporting of VistA’s costs. Specifically, we are recommending that the department develop and implement a methodology for reliably identifying and reporting the total costs of VistA. The methodology should include steps to identify the definition of VistA and what is to be included in its sustainment activities, as well as ensure that comprehensive costs are corroborated by reliable data. In written comments on a draft of the report, the department agreed with the recommendation and stated that it will provide the actions it plans to take to address this recommendation within 180 days. In conclusion, although VA is not likely to be positioned to retire VistA for at least another 10 years, the department lacks the comprehensive and reliable cost information needed to make critical management decisions for sustaining the system. As the department continues to work toward acquiring a new electronic health record, it will be important for VA to take actions to address our recommendation for improving the reporting of VistA costs. Doing so is essential to helping ensure that decisions related to the current system are informed by reliable cost information and that there is an accurate basis for reporting on the return on its investment for replacing VistA. Chair Lee, Ranking Member Banks, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staffs have any questions about this testimony, please contact Carol C. Harris, Director, Information Technology Management Issues, at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony statement. GAO staff who made key contributions to this testimony are Mark Bird (Assistant Director), Rebecca Eyler, Jacqueline Mai, Monica Perez-Nelson, Scott Pettis, Jennifer Stavros-Turner (Analyst in Charge), and Charles Youman. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "VA provides health care services to approximately 9 million veterans and their families and relies on its health information system—VistA—to do so. However, the system is more than 30 years old, is costly to maintain, and does not fully support exchanging health data with DOD and private health care providers. Over nearly 2 decades, VA has pursued multiple efforts to modernize the system. In June 2017, the department announced plans to acquire the same system—the Cerner system—that the Department of Defense is implementing. VA plans to continue using VistA during the department's decade-long transition to the Cerner system. GAO was asked to summarize its report that is being released today which discusses, among other things, (1) the extent to which VA has defined VistA and (2) the department's annual costs to develop and sustain the system. In preparing the report on which this testimony is based, GAO analyzed documentation that defines aspects of VistA and identifies components to be replaced; and evaluated the reliability of cost data, including funding obligations associated with the development and sustainment of VistA for fiscal years 2015, 2016, and 2017. The Department of Veterans Affairs (VA) has various documents and a database that describe parts of the Veterans Health Information Systems and Technology Architecture (VistA); however, the department does not have a comprehensive definition for the system. For example, VA has identified components that comprise VistA, identified interfaces related to the system, and collected system user guides and installation manuals. VA has also conducted analyses to better understand customization of VistA components at various medical facilities. Nevertheless, the existing information and analyses do not provide a thorough understanding of the local customizations reflected in about 130 versions of VistA that support health care delivery at more than 1,500 sites. Program officials stated that they have not been able to fully define VistA due to the decentralization of the development of the system for more than 30 years. Cerner's contract to provide a new electronic health record system to VA calls for the company to conduct comprehensive assessments to identify site-specific requirements where its system is planned to be deployed. Three site assessments have been completed and additional assessments are planned. If these assessments provide a thorough understanding of the 130 VistA versions, the department should be able to define VistA and be better positioned to transition to the new system. VA identified costs for VistA and its related activities adding up to approximately $913.7 million, $664.3 million, and $711.1 million in fiscal years 2015, 2016, and 2017, respectively—for a total of about $2.3 billion over the 3 years. However, of the $2.3 billion, the department was only able to demonstrate that approximately $1 billion of these costs were sufficiently reliable. In addition, the department omitted VistA-related costs from the total. The lack of a sufficiently reliable and comprehensive total cost for VistA is due in part to not following a well-documented methodology that describes how the department determined the costs for the system. As a result of incomplete cost data and data that could not be determined to be sufficiently reliable, the department, legislators, and the public do not have a complete understanding of how much it has cost to develop and maintain VistA. Further, VA lacks the information needed to make decisions on sustaining the many versions of the system. In its report being issued today, GAO is recommending that VA develop and implement a methodology for reliably identifying and reporting the total costs of VistA. The department agreed with the recommendation.", "document_type": "gao"}
{"report": "The NHPA requires federal agencies to establish historic preservation programs to ensure the ongoing identification and protection of historic properties. A historic property is any building, structure, object, site, or district listed on or eligible for inclusion in the National Register of Historic Places (National Register). To be eligible for the National Register, a property must meet certain criteria, such as being associated with the lives of significant people from the past or yielding important information about prehistory or history, among others. Generally, properties that have achieved significance within the past 50 years are not considered eligible for the National Register unless they are of exceptional importance. The NHPA also established the ACHP, which advises the President and Congress on matters relating to historic preservation. The ACHP also recommends measures to coordinate activities of federal, state, and local agencies and private institutions and individuals relating to historic preservation. The ACHP can review the relevant policies and programs of federal agencies and make recommendations to improve their effectiveness, coordination, and consistency. Section 106 of NHPA requires federal agencies, including DOD, to take into account the effects of their undertakings (hereinafter referred to as projects) on historic properties, and to afford the ACHP a reasonable opportunity to comment on any such projects on historic properties by a federal agency. Part 800 of title 36, Code of Federal Regulations, establishes procedures to define how DOD and other federal agencies should meet these statutory responsibilities and how to accommodate historic preservation concerns with the mission of the agency, including DOD. Historic preservation concerns are reviewed in consultation with officials from the agency in question and other parties with an interest in the effects of the proposed project on historic properties. The goal of this consultation is to identify historic properties potentially affected by the project, assess its effects, and seek ways to avoid, minimize, or mitigate any adverse effects on historic properties. State Historic Preservation Offices—each led by a State Historic Preservation Officer (SHPO)—advise and assist federal agencies, including DOD, in carrying out their Section 106 responsibilities, and ensure that historic properties are taken into consideration during in project planning. A more detailed description of the relationship between DOD and SHPOs is presented in appendix II. A programmatic agreement is a document that federal agencies can, in consultation with the ACHP, SHPO, and/or other parties, negotiate and execute when a planned project will or may adversely affect historic properties and sets out the measures the federal agency will implement to resolve those adverse effects. Agencies can use programmatic agreements to satisfy their Section 106 responsibilities in the following circumstances: when effects on historic properties are similar and repetitive or are multi-state or regional in scope, when effects on historic properties cannot be fully determined prior to approval of a project, when nonfederal parties are delegated major decision-making where routine management activities are undertaken at federal installations, facilities, or other land-management units, or when other circumstances warrant a departure from the normal Section 106 process. Section 110 of the NHPA requires federal agencies to establish a preservation program to protect, identify, evaluate, and nominate historic properties to the National Register. Section 110 also states that agencies must designate qualified preservation officers to lead their respective agencies’ efforts to adhere to the NHPA, among other requirements. Further, Executive Order 13287, Preserve America, instructs all executive branch departments and agencies to maximize efforts to integrate the policies, procedures, and practices of the executive order and the NHPA into their program activities to advance historic preservation objectives. Preserve America also instructed executive branch departments and agencies to assess the current status of their historic property inventories (including general condition and management needs) and directs agencies with real property management responsibilities to report on efforts to identify, protect, and use historic properties every 3 years. DOD Instruction 4715.16 set forth the framework for a department-wide program that focuses on the management of cultural resources, which include historic properties. According to DOD officials, as part of DOD’s program to preserve historic properties, each military department designates federal preservation officers to coordinate its own separate historic property programs. Each department has an office or division that handles cultural resources and historic preservation and has staff who are generally knowledgeable about NHPA and its requirements. The military departments also issue their own guidance that establishes policies on historic preservation and delineates responsibilities for cultural resources personnel at the service and installation level. Each military department also is responsible for ensuring that military installations with cultural resources under their purview prepare Integrated Cultural Resource Management Plans (ICRMPs). These plans should include an inventory of all known historic properties, an inventory of properties that may be eligible for listing on the National Register, and standard operating procedures covering certain maintenance aspects of historic properties. According to officials from the military departments, installations are responsible for setting up a process where all maintenance/work order requests are reviewed for further action. For example, the review process can take the form of a maintenance/work order request review board and typically includes the installation’s cultural resources manager or members of the cultural resources manager’s staff. If the maintenance/work order request involves a historic property, then additional steps are taken at the installation level to consult with the appropriate stakeholders. Once officials at an installation complete their evaluation of the potential impact a maintenance request/work order would have on a historic property, they consult with the SHPO on how to move forward with the proposed maintenance/work order, according to installation officials. A more detailed description of the review of maintenance/work order requests is presented in appendix III. DOD generally uses its historic properties in one of two ways—to support mission needs or to house service members and their families. Generally, after consultations with the SHPO, historic properties can be repurposed or renovated to fulfill current mission and housing needs. For example, a historic aircraft hangar could be converted into additional administrative space or historic homes could be renovated by a private housing partner to house service members and their families. Figure 1 is an example of how a historic property could be reused. In October 2017, DOD reported that, of its approximately 375,000 properties on installations in the U.S. and its territories, it has identified and evaluated about 60,000 as historic and about 57,000 as not being historic. DOD has not yet evaluated the remaining roughly 258,000 properties for historic significance. Approximately 41,000 of these properties are greater than or equal to 50 years of age, according to DOD. DOD’s Cultural Resource Management Instruction requires DOD to conduct a survey of historic properties that includes the identification and evaluation of all cultural resources against the criteria of the National Register. According to ACHP officials, DOD does not routinely identify and evaluate every property under its purview for historic significance as those properties reach 50 years of age. Instead, DOD’s practice is to identify and evaluate property for historic significance as installations have an identified need for or a project planned for the property, according to both DOD and ACHP officials. Officials said that, generally, federal agencies do not have the funding to proactively identify and evaluate properties for historic significance. Rather, funding to identify and evaluate properties is included within a project’s funding; therefore, generally federal agencies cannot begin to identify and evaluate a property for historic significance until a project for that property is funded, according to officials from the ACHP. The initial process to identify, evaluate, and track real property, such as historic properties, occurs at the installation level. Installation officials are to record transactions; document new acquisitions, changes to existing facilities, and disposals; and collect information on the real property at each installation. Installation officials are then to enter this information into the corresponding military department or WHS real property data systems. The military departments and WHS use these databases to oversee and manage real property needs across DOD installations, such as how property is used to support the installations’ missions and how much to budget for required sustainment, restoration, or construction of real property. Figure 2 shows how data are intended to move from the installation level to the military department databases and then to the DOD-wide real property database, which DOD calls the “Real Property Assets Database (RPAD).” OSD requires that the military departments and WHS submit their real property inventories to be compiled into RPAD. DOD uses these data to provide information on its real property to Congress and other federal agencies, including the Office of Management and Budget and the General Services Administration, in order to assist in the oversight of federal real property. We identified some gaps in data, as well as data discrepancies between the data reported at the installation level and the department level regarding historic properties for fiscal year 2017. For example, one of the 10 installations we visited could not generate a list of historic properties on the installation with corresponding data fields such as the facility condition, plant replacement value, and facility utilization rate. Officials at this installation told us they are working on a long-term project to update their data on historic properties. Additionally, data we collected from three of the 10 installations we visited were inconsistent with data in the installations’ respective military department-level databases. For instance: One installation had 150 more historic properties listed in its installation real property data than were listed in the corresponding military department database. The installation’s data also showed 114 fewer properties coded as “Not Yet Evaluated” for historic significance than did the military department’s database. Similarly, the data in the military department database showed twice the number of privatized homes than did the installation database. A second installation had 119 properties coded as “Not Yet Evaluated” for historic significance, but none with this designation in the data provided by the installation. The data provided by the installation also included 164 privatized homes, none of which were included in the military department database. Further, this installation had nine historic properties that were not included in the military department database but that were included in the installation data, as well as 26 historic properties that were included in the military department database but that were not included in the installation data. A third installation had fewer discrepancies, with two historic properties that were included in the installation data that were not in the military department database. The data in the military department database contained six assets that the installation data did not contain. There were also four discrepancies regarding privatized housing between the installation data and the military department database, with each database containing two entries the other did not include. We asked five installation cultural resource managers about these discrepancies, and they stated that the military department databases most likely had not been updated to reflect the correct installation numbers. In November 2018, we reported that RPAD contained inaccurate and incomplete data due to weaknesses in DOD’s processes for recording and reporting real property, including historic property. The military services lacked complete data regarding real property transactions as well physical inventories of real property, to include historic properties. We also found that the military services have not consistently recorded real property transactions (i.e., the acquisition of, change to, and disposal of real property assets) and the results of physical inventories of assets. Finally, we found that the military services have not corrected previously- identified discrepancies in their data systems, such as missing entries for utilization and facility condition and overdue asset reviews. We recommended that each of the services develop monitoring processes for recording all real property (including historic properties) information. We also recommended that the Under Secretary of Defense for Acquisition and Sustainment work in collaboration with the services to develop corrective action plans to remediate inconsistencies in the data. DOD concurred with these recommendations and identified actions it plans to take to implement them. Implementing these recommendations would help DOD ensure more accurate and complete information on properties of historic significance and prevent further data discrepancies. Also, more accurate and complete information on the identification and evaluation of properties would help installations, military departments, and WHS oversee and manage their real property needs, including informing decisions regarding how much to budget for required sustainment, restoration, or construction of real property. We will continue to monitor DOD’s progress in addressing these recommendations. DOD may transfer the responsibility to identify and evaluate homes for historic significance to the private developers. However, the military department officials we interviewed could not confirm that private developers were meeting those responsibilities. The military departments have flexibility in how they structure their privatized housing projects, but project structures share certain similarities. For a typical privatization project, a military department leases land to a developer for a 50-year term and conveys existing homes located on the leased land to the developer for the duration of the lease. Given the length of these lease agreements, homes may move beyond 50 years of age while being maintained by the private developer. Military department officials told us that when a lease or programmatic agreement is signed with a private developer, the responsibility to identify and evaluate homes for historic significance is generally transferred to the private developer. Navy and Marine Corps officials stated that, when the leases for privatized military homes were signed, a list of historic properties was provided to each private developer. According to Navy officials, those private developers are now responsible for identifying and evaluating privatized homes for historic significance once the lease is signed and the homes are transferred to the private developer. Similarly, Air Force officials stated that, prior to conveying homes to a private developer all homes encompassed in the lease agreement should have been identified and evaluated for historic significance by the Air Force. According to these officials, after the transfer of properties under the lease, the private developer is responsible for identifying and evaluating homes for historic significance. Army officials also stated that the responsibility to manage privatized homes and assess their historic value falls to the private developer. However, private developers at seven of the nine of installations we visited that had privatized historic military housing told us that they do not identify or evaluate additional homes for historic significance. The private developers at the remaining two installations said they hire a third- party to identify and evaluate homes on the installations for historic significance as they age. DOD’s instruction on the management of cultural resources directs the establishment of a process to identify and evaluate cultural resources for historic significance. The need to identify and evaluate privatized military homes for historic significance would arise if a new project were planned for homes that could be of historic significance. Officials from all three military departments told us that they have addressed the identification and evaluation process by formally transferring those responsibilities to the private developers through documents such as land-lease agreements, installations’ programmatic agreements, and installations’ ICRMPs. However, DOD guidance also states that because privatization creates a long-term governmental interest in privatized housing, it is essential that the military departments attentively monitor these privatization projects. Taking steps to ensure that installation personnel verify that private developers are identifying and evaluating privatized properties for historic significance, as appropriate, could help to ensure that private developers do not make renovations or repairs to properties that could compromise their historic nature. Under DOD Instruction 4165.14, once a historic property has been identified, installations are required to complete a review of the real property asset record every 3 years, including a physical inventory that assesses the condition of the property. According to DOD, these inventories are important for planning, analysis, and decision making. However, we found that these required inventories are not routinely being conducted at six of the 10 installations we visited for a variety of reasons. Specifically, cultural resource management officials at six of the 10 installations told us that the inventory was not conducted because they were unaware of the requirement or thought that updating their ICRMPs was sufficient to satisfy the inventory requirements. As previously noted, ICRMPs should include an inventory of all known historic properties, an inventory of properties that may be eligible for listing on the National Register, and standard operating procedures covering certain maintenance aspects of historic properties. Officials at one of the six installations reported that they believe it is a best practice to inventory their historic properties every 5 years if they have sufficient staff to do so. Officials at two installations stated that they do complete the required inventory every 3 years. Officials at the remaining two installations either did not provide any comment or said they were unsure of when the last inventory was completed. However, officials from all of the services’ headquarters reiterated to us that the requirement under DOD Instruction 4165.14 is to inventory historic properties every 3 years. They explained that this inventory is separate and distinct from the annual inventory required under the ICRMP process. For example, Air Force headquarters officials stated that the 3 year inventory should consist of a physical check of the condition of the buildings, while the annual inventory required as part of the ICRMP update is a process to update data, such as status codes, for newly evaluated buildings. Until the military departments clarify the existing 3 year inventory requirement, current and accurate information on the condition of historic properties will not be available. Such information would better position officials who manage these properties to make informed management, maintenance, and planning decisions. We found that misunderstandings about how to maintain historic properties have led, in some instances, to problems with the preservation of these properties at installations. Each of the 10 installations that we visited has an established process and procedures for reviewing and approving maintenance/work orders on historic properties. These processes and procedures, articulated in installations’ ICRMPs, vary by installation and are generally intended to assist in preserving historic properties. However, cultural resource managers at five of the 10 installations said that past maintenance or renovation projects on some of their installations’ historic buildings may have compromised the historic significance of those buildings. In some cases, for instance, maintenance was performed improperly by tenants of historic properties or by contractors, according to installation officials. At one installation we visited, an official said a tenant made changes to a historic building without undergoing the formal approval process at the installation, which includes informing the cultural resource manager of the proposed change. The official said the tenant added additional office space and equipment, such as computers and other systems, in an unused attic without updating the capacity of the electrical panels. As a result, the official said a fire started in the attic, causing extensive damage to the building. An official at another installation we visited told us a contractor pressure washed a historic property that ended up damaging the building. The official said the damage was not intentional, as the contractor did not realize that pressure washing would harm the property. Unit members also noted some instances in which they were told by maintenance personnel that problems the members had reported could not be fixed because of the historic nature of the properties. For instance: At a Marine Corps installation, unit members said that maintenance and facilities management staff ignored or improperly handled issues they raised in their historic buildings. For example, unit members told us that maintenance personnel erroneously informed them they could not replace the air filters or clean out the mold in the ceiling because their building was historic. At an Air Force installation, unit members told us their requests for upgraded electrical outlets and roof fixes were denied because maintenance personnel told them those changes could not be completed because of the historic nature of the building. According to unit members, the existing outlets were not suitable for work on the aircraft being maintained in the building and thus presented a safety risk. Moreover, unit members told us that, to deal with the roof leaks, they ultimately resorted to using buckets to catch water. At an Army installation, unit members told us that maintenance personnel informed them they could not address certain problems, such as leaks, because of the historic nature of the building. For example, unit members at this installation resorted to boarding up their building with plywood during storms to keep rainwater from affecting the secure facility in the basement of the historic building because maintenance division staff told them addressing the leaks was not their responsibility, due, in part, to the historic nature of the building. One reason these problems may have occurred is that the individuals involved were not properly informed or trained about how to conduct maintenance on historic buildings. At nine of the 10 installations we visited, unit members who work in historic buildings told us that, based on their experiences requesting repairs to historic buildings, they believed maintenance personnel did not know what maintenance could or could not be done to the historic buildings. Officials from these installations expressed concerns about training, including a lack of training, related to historic preservation and maintenance of historic properties. For example, maintenance officials at three of the 10 installations we visited stated that they do not receive training on the special requirements associated with maintaining historic buildings; and cultural resource managers from four of the 10 installations told us that more training for installation staff, particularly maintenance staff, on historic preservation requirements would be helpful. Furthermore, officials from two of the four SHPOs representing the states where we visited military installations said they believe that tenants and maintenance personnel at installations do not have the proper training to adhere to historic preservation requirements. Officials from the Office of the Under Secretary of Defense for Acquisition & Sustainment (OUSD(A&S)) also said they were aware of misunderstandings within the military communities about aspects of historic preservation. For example, these officials said there were misunderstandings among installation personnel, including between personnel from department of public works’ offices, environmental offices, installation planners, and cultural resource managers about their roles and responsibilities concerning historic preservation. The OUSD(A&S) is responsible for establishing cultural resource guidance, designating responsibilities, and providing procedures to implement DOD’s cultural resources program. DOD Instruction 4715.16 states that ICRMPs act as the instrument DOD uses to comply with the statutory management requirements of the NHPA. It is also DOD policy that cultural resources under DOD control are to be managed and maintained in a sustainable manner through a comprehensive program that considers the preservation of historic, archaeological, architectural, and cultural values; is mission supporting; and results in sound and responsible stewardship. In addition, the Standards for Internal Control in the Federal Government state that management should communicate quality information down and across reporting lines to enable personnel to perform key roles in achieving objectives, addressing risks, and supporting the internal control system. However, officials from each of the military departments stated that they do not have department-wide or service-wide guidance related to historic preservation training. Instead, the content and frequency of training is determined by the installations, according to military department officials. When we analyzed the installations’ ICRMPs, we found that responsibilities for providing cultural resources training or technical guidance, feedback, and comments to installation personnel regarding historic preservation generally lie with the installation cultural resource manager. Installation personnel rely on individual cultural resource managers and the individual installations’ ICRMPs to ensure that all personnel at an installation have the training they need. Without providing installations with DOD or military department-wide guidance on training related to historic preservation, there could be more instances of improper or incomplete maintenance of historic properties on installations. According to the Advisory Council on Historic Preservation (ACHP), DOD is one of the most compliant federal agencies with regard to historic preservation requirements. DOD uses historic properties to support mission needs as well as to house military service members. Thus far, DOD has identified and evaluated 60,000 properties as historic. However, additional actions could enhance DOD’s efforts to identify, assess, and preserve historic properties. First, we recently made recommendations which DOD concurred with, to improve the quality of DOD’s real property data. Implementing the recommendations would help ensure that DOD has more accurate and complete information on properties of historic significance and prevent further data discrepancies. Second, taking steps to verify that private developers are identifying and evaluating privatized properties that could be historic would help mitigate the risk of developers making renovations to properties that could compromise their historic nature. Additionally, clarifying the requirement to inventory historic properties every 3 years to assess their condition would help ensure that DOD has the information it has identified as important for planning, analysis, and decision-making related to such properties. Further, establishing guidance on training for installation personnel would help ensure they possess the necessary knowledge to properly maintain historic properties on installations. We are making a total of seven recommendations to DOD. The Secretary of the Navy should take steps to ensure that Navy and Marine Corps’ installation personnel verify that private developers are identifying and evaluating privatized properties for historic significance, as appropriate. (Recommendation 1) The Secretary of the Army should take steps to ensure that Army installation personnel verify that private developers are identifying and evaluating privatized properties for historic significance, as appropriate. (Recommendation 2) The Secretary of the Air Force should take steps to ensure that Air Force installation personnel verify that private developers are identifying and evaluating privatized properties for historic significance, as appropriate. (Recommendation 3) The Secretary of the Navy should clarify the requirement for Navy and Marine Corps’ installation personnel to conduct a physical inventory of historic properties every 3 years, including an assessment of each property’s condition to ensure that facilities that have been identified and evaluated as historic are inventoried. (Recommendation 4) The Secretary of the Army should clarify the requirement for Army installation personnel to conduct a physical inventory of historic properties every 3 years, including an assessment of each property’s condition to ensure that facilities that have been identified and evaluated as historic are inventoried. (Recommendation 5) The Secretary of the Air Force should clarify the requirement for Air Force installation personnel to conduct a physical inventory of historic properties every 3 years, including an assessment of each property’s condition to ensure that facilities that have been identified and evaluated as historic are inventoried. (Recommendation 6) The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment, in collaboration with the military departments, develop and disseminate department-wide or service-wide guidance, on training related to historic preservation to installation personnel, including information on roles and responsibilities. (Recommendation 7) We provided a draft of this report to DOD for review and comment. In written comments, DOD concurred with each of our recommendations. DOD’s comments are reprinted in their entirety in appendix IV. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and to the Acting Secretary of Defense; the Under Secretary of Defense for Acquisition and Sustainment; and Secretaries of the Departments of Air Force, Army and Navy, and the Director of Washington Headquarters Services. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or our staff have any questions about this report, please contact me, Elizabeth Field, at (202) 512-2775 or FieldE1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are listed on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Senate Report 115-130, accompanying a bill for the Fiscal Year 2018 Military Construction, Veterans Affairs, and Related Agencies Appropriations Act, included a provision that GAO assess the historic properties in use on the Department of Defense’s (DOD) U.S. installations. This report assesses the extent to which (1) DOD identifies and evaluates properties for historic significance, including those that have been privatized, and (2) DOD assesses the condition of its historic properties and has guidance on the training of installation personnel maintaining and those working in historic properties. For both objectives, we reviewed relevant laws, regulations, executive orders, and DOD (including military service) guidance that govern efforts to identify, evaluate, manage, and maintain DOD’s historic properties. We interviewed officials from the Office of the Secretary of Defense (OSD) (the Office of the Under Secretary of Defense for Acquisition and Sustainment); Washington Headquarters Services (Facilities Services Directorate); the Army (Installation Management Command; Office of the Assistant Chief of Staff for Installation Management; Office of the Assistant Secretary of the Army for Installations, Energy and Environment; U.S. Army Corps of Engineers); the Navy (Office of the Assistant Secretary of the Navy for Energy, Installations, and Environment; Office of the Deputy Assistant Secretary of the Navy for Installations and Facilities; Office of the Chief of Naval Operations; Naval Facilities Engineering Command); the Marine Corps (Headquarters Marine Corps; Marine Corps Installations Command; Environmental Management Division); and the Air Force (Headquarters Air Force; Air Force Civil Engineer Center Installations Directorate). We reviewed DOD data, plans, and agreements, and compared DOD’s efforts to address criteria in the National Historic Preservation Act and DOD Instructions. Additionally, we met with officials from the Advisory Council on Historic Preservation and private developers, such as Balfour Beatty, Clark Realty Capital, Lendlease, Lincoln Military Housing, and Hunt Companies, to whom DOD has conveyed property under the Military Housing Privatization Initiative (MHPI). To gather detailed examples of DOD’s historic preservation efforts, we visited historic properties at a non- generalizable sample of 10 installations. We selected these installations by analyzing DOD’s fiscal year 2017 data on real property, limited our analysis to installations in the continental United States, and identified the number of buildings and structures (“properties”) in each state DOD reported as historic. We selected four states, California, Hawaii, Virginia, and Maryland, for reasons including the high concentration of historic properties in the state. To select installations in each state, we considered variation in military service representation, the number of historic properties at each installation, and geographic variation and proximity. During these visits, we interviewed officials representing environmental resource management, cultural resource management, and the department of public works, facilities management, along with privatized installation housing developers. Further, we met with relevant state stakeholders including State Historic Preservation officials in California, Hawaii, Maryland, and Virginia. We obtained documentary and testimonial evidence related to the identification, evaluation, management, and maintenance of historic properties. We also conducted semi-structured group discussions of those who work in historic properties. The results of our interviews and semi-structured group discussions are not generalizable to all DOD installations. To determine the extent to which DOD identifies and evaluates properties for historic significance, including homes that have been privatized, we reviewed prior GAO reports related to this issue, including a recent GAO report on DOD’s real property data, including historic properties. We also requested and reviewed data related to historic properties, for each installation that we visited, including data on: the facility condition, plant replacement value, and facility utilization rate, among other data fields. We reviewed and compared the data from the military departments and from these selected installations. As discussed in this report, we identified limitations of the reported data on historic properties that have been identified and evaluated by DOD. Further, we compared DOD’s efforts to ensure that privatized homes have been identified and evaluated for historic significance to guidelines in Department of Defense Instruction 4715.16, Cultural Resources Management, and Department of Defense Manual 4165.63, DOD Housing Management. We also obtained and assessed testimonial evidence about the process to identify and evaluate privatized homes for historic significance from officials from the military departments and private developers. To determine the extent to which DOD assesses the condition of its historic properties and has guidance on the training of installation personnel maintaining and working in historic properties, we conducted interviews with officials from within OSD, each military department and officials at the 10 installations we visited to identify efforts to manage and maintain historic properties. We also met with U.S. Army Corps of Engineers and DOD’s Washington Headquarters Services to further understand their roles in historic property maintenance. We interviewed major developers who have, under the Military Housing Privatization Initiative, leased military housing from DOD and analyzed the process that is used to manage and maintain historic properties. We compared DOD’s efforts to conduct inventories of historic properties to guidelines in Executive Order 13287, Preserve America, and DOD Instruction 4165.14, Real Property Inventory (RPI) and Forecasting. In addition, related to the maintenance of historic properties, we compared DOD’s efforts to guidelines in DOD Instruction 4715.16, Cultural Resources Management, and the Standards for Internal Control in the Federal Government. In addition, at the 10 installations we visited, we collected physical and documentary evidence of DOD’s management and maintenance practices at the installation level. We analyzed installation-level planning documents related to the management and maintenance of historic properties, specifically the installation Integrated Cultural Resource Management Plans (ICRMPs) of the installations we visited. The ICRMPs were from installations spread out across the country and represented all branches of the military. We analyzed the ICRMPs to determine if there were any common themes. We also reviewed a non-generalizable sample of 10 programmatic agreements—one provided by each installation we visited—to identify common themes. These themes cannot be generalized to all programmatic agreements. We conducted interviews with installation staff to understand their responsibilities for historic property management and maintenance. We interviewed state historic preservation officials to understand the relationship between installations and preservationists and efforts to preserve historic properties on installations. During our site visits to 10 installations, we conducted semi-structured group discussions with individuals who work in historic buildings to supplement our understanding of DOD’s compliance with required policy and guidance, as well as any impact working in historic properties has on DOD employees. We used the military department data that informed our site selection, and queried the data to generate a random list of properties DOD identifies as historic. We provided each installation we visited with a list of 20 randomly selected historic properties and requested the installation’s assistance in inviting unit members who work in these buildings to participate in a semi-structured group discussion. The participants of the semi-structured group discussions were asked to discuss their experiences working in historic buildings. The results of our semi-structured group discussions are not generalizable to all DOD installations. To conduct the analysis and summary of these discussion groups, we developed a record of analysis that listed the installations visited and overall topics posed to the unit discussion groups and assessed the extent to which unit members had similar or different experiences working in historic buildings. We identified themes that emerged for each discussion topic across these group discussions. We conducted this performance audit between March 2018 and June 2019, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Installation cultural resource managers we spoke to at all 10 installations we visited said that they cultivate and maintain active relationships with their state historic preservation office (SHPO) and regularly communicate with them on preservation issues affecting their installations. Five out of the 10 cultural resource managers noted that maintaining a good working relationship with their SHPO made the consultation process more efficient. Officials we interviewed for two of the four SHPOs stated that being involved early in the consultation process with installation officials is more efficient and makes historic preservation an easier process by enabling them to receive feedback on proposed projects on historic properties, approval of programmatic agreements, and concurrence on their Integrated Cultural Resource Management Plans (ICRMPs) in a timely manner. For example, DOD officials at one military installation said they were able to use non-historic materials during renovations on a historic property in place of more costly period-accurate materials because the agreement with that SHPO facilitated such a solution. According to officials at this installation, SHPOs generally prefer the use of period-accurate materials on historic properties when conducting repairs and renovations. The officials, however, stated that they began consultations with the SHPO early in the process and were able to reach agreement that the historic nature of the property would not be adversely affected if non-historic materials were used. See figure 3 below. Officials from two of the four SHPOs said that due to positive working relationships between the installation and the SHPO, a programmatic agreement has been put in place to help manage the installation’s historic properties. These programmatic agreements can be used to address routine maintenance activities for historic properties that can be carried out by the installation with no further consultation with the SHPO. In the four states that we visited, SHPO officials said they executed programmatic agreements with some installations that can save time and reduce the number of required consultation meetings. According to officials from two of the four SHPOs we interviewed, having programmatic agreements in place can increase the efficiency of the historic preservation process. Generally, these programmatic agreements can include the following: Standard operating procedures. Programmatic agreements can include a number of routine maintenance plans pre-approved by the SHPO (such as the replacement of historic windows, repairing leaking historic roofs, and painting historic buildings) that an installation cultural resource manager can then follow without having to go through the consultation process. Inventories of relevant properties. Programmatic agreements can include inventories of historic properties that are relevant to the agreement. Generally, the procedures outlined in the programmatic agreement would apply to all of the properties listed in the inventory. Dispute resolution and emergency plans. Programmatic agreements can also include dispute resolution mechanisms between parties to the agreement and contingency plans for the maintenance and repair of historic properties in the event on an emergency. DOD Instruction 4715.16 on cultural resource management states that installations should adapt and reuse existing structures at their installation before disposal, new construction, or leasing. Installations typically consult with the SHPO before renovation work can proceed on historic properties, but, according to officials at one installation, alternative solutions can be reached if there is a good working relationship. In the figure below, at one military installation we visited, a historic property formerly used by National Aeronautics and Space Administration (NASA) and now used by the installation is in the process of being renovated and converted into additional office space. The concrete dome was used to test the aerodynamics of some of NASA’s satellite and spaceship components and is being converted into a new conference room after the SHPO approved the installation’s plan. See figure 4 below. While all of the installation cultural resource managers we spoke to told us they regularly communicate with their SHPO and five of these cultural resource managers said that good working relationships with the SHPO made the consultation process more efficient, installation officials may still experience challenges when trying to address historic preservation concerns. For example, maintenance officials at four of the 10 installations expressed some concerns about a backlog of consultations due in part to the increased time that they felt it takes to conduct these consultations. According to these officials, consultation backlogs caused delays to maintenance projects on historic properties at their installations. DOD officials from every military service stated that each installation has a process for reviewing maintenance requests and work orders, including those involving historic properties. These procedures, articulated in installations’ Integrated Cultural Resource Management Plans (ICRMPs), vary by installation. For example, at seven of the 10 installations we visited, the ICRMPs state that all maintenance requests and work orders are reviewed by a board (or other body of internal stakeholders) composed of maintenance personnel, cultural resources staff (including the cultural resources manager), and other installation personnel. Officials from the military departments said that these boards are responsible for, among other duties, regularly identifying maintenance requests and work orders that affect historic properties and ensuring that the proper steps are carried out before addressing a maintenance request. Decisions by the board, results of SHPO consultations, and programmatic agreement requirements are then, according to officials from the military departments, passed down to maintenance personnel before they begin work on the historic property. At two of the other installations we visited, the installations’ department of public works reviews all maintenance requests and work orders, and at the remaining installation, the cultural resources manager reviews them, according to installation officials. During our visits to the military installations, cultural resource managers from eight of the 10 installations stated that they play a role in their installation’s maintenance request/work order review process and that maintenance personnel are typically included in the process. For example, one installation we visited set up a work induction board composed of staff from the installation’s Environmental Security Division (which handles cultural resources), maintenance staff, and other internal stakeholders. The senior official within the Environmental Security Division at this installation said the board meets on a weekly basis to determine whether proposed projects (such as maintenance requests and work orders) will affect historic properties at the installation. If the project involves a historic property, the installation’s cultural resources manager becomes involved and determines the extent of the affect to the property’s historic nature. This senior official also told us that the board also checks in regularly on ongoing projects and monitors work being done on historic properties. Officials at another installation we visited said they treat any building that is aged 50 or older in their database as historic and the maintenance division sends every new project to their installation’s historic preservation division to ensure a review of potential impacts of the maintenance requests or work orders. In addition to the contact named above, Brian Lepore, Director (retired); Maria Storts, Assistant Director; Whitney Allen; Ronnie Bergman; Aaron Chua; Christopher Gezon; Alexandra Gonzalez; Lori Kmetz; Amie Lesser; Emily Martin; Natalia Peña; Clarice Ransom; Jodie Sandel; Monica Savoy; and John Van Schaik made key contributions to this report. High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas. GAO-19-157SP. Washington, D.C.: March 6, 2019. Defense Real Property: DOD Needs to Take Additional Actions to Improve Management of Its Inventory Data. GAO-19-73. Washington, D.C.: November 13, 2018. Military Housing Privatization: DOD Should Take Steps to Improve Monitoring, Reporting, and Risk Assessment. GAO-18-218. Washington, D.C.: March 13, 2018. High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others. GAO-17-317. Washington D.C.: February 15, 2017. Defense Infrastructure: More Accurate Data Would Allow DOD to Improve the Tracking, Management, and Security of Its Leased Facilities. GAO-16-101. Washington, D.C.: March 15, 2016. High-Risk Series: An Update. GAO-15-290. Washington D.C.: February 11, 2015. Federal Real Property: Improved Data Needed to Strategically Manage Historic Buildings, Address Multiple Challenges. GAO-13-35. Washington, D.C.: December 11, 2012. Defense Infrastructure: Military Services Lack Reliable Data on Historic Properties. GAO-01-437. Washington, D.C.: April 6, 2001.", "summary": "The National Historic Preservation Act of 1966 requires each federal agency to establish a preservation program that ensures properties are identified and evaluated for historic significance, as well as managed and maintained in a way that considers their preservation. Senate Report 115-130 accompanying a bill for the Military Construction, Veterans Affairs, and Related Agencies Appropriations Act for fiscal year 2018, included a provision that GAO assess DOD's management of historic properties in use on U.S. installations. This report examines the extent to which DOD (1) identifies and evaluates properties for historic significance, including those that have been privatized, and (2) assesses the condition of its historic properties and has guidance on the training of installation personnel maintaining and those working in historic properties. GAO reviewed DOD fiscal year 2017 real property data and policies and procedures; visited a non-generalizable sample of 10 installations, selecting them based factors such as military service representation and concentration of historic properties; and interviewed DOD officials, privatized housing developers, and installation personnel. The Department of Defense (DOD) reported that it has identified and evaluated about 60,000 of its approximately 375,000 properties on installations as historic as of October 2017. DOD's practice is to identify and evaluate property for historic significance as installations have an identified need for or a project planned for the property, according to DOD officials. However, GAO identified opportunities for DOD to enhance its efforts in several areas. DOD lacks complete and consistent data on historic properties. Specifically, GAO identified data gaps and discrepancies between the data reported at the installation and department levels for fiscal year 2017. For example, for one installation, GAO found that 150 more historic properties were listed in its installation data than were listed in department-level data for that installation. In November 2018, GAO reported on issues concerning DOD's data and made recommendations to improve the data quality. DOD concurred and reported actions it plans to take to improve data quality. Doing so would help DOD to ensure it has complete information on properties of historic significance and prevent further data discrepancies. DOD has limited visibility of privatized homes that could be historic. When the military departments transferred military homes to private developers, DOD officials said they also transferred the responsibility to identify and evaluate homes for historic significance to the private developers. However, the military departments do not verify that private developers are doing so. Private developers at seven of the nine installations with privatized housing that GAO visited said they do not identify or evaluate homes for historic significance. Taking steps to verify that private developers carry out this responsibility could help DOD ensure that renovations or repairs are not made to privatized properties that could compromise their historic nature. Additionally, DOD does not routinely assess the condition of its historic properties and a lack of guidance on training could hamper maintenance and preservation efforts. First, inventories of historic properties, including physical inspections, required every 3 years, are not being conducted at six of the 10 installations GAO visited. Officials at these six installations said that the inventory was not conducted because they were unaware of or misunderstood the requirement. Second, while each installation GAO visited had an established process for approving maintenance work orders, DOD officials reported problems with the maintenance of historic properties at these installations, ranging from maintenance personnel not addressing issues, to maintenance being conducted improperly. At nine of the 10 installations GAO visited, individuals who work in historic buildings said that they believed maintenance personnel did not know what maintenance could or could not be done to the historic buildings, and installation officials expressed concerns about a lack of training related to historic preservation. By clarifying the requirement to conduct a physical inventory and developing guidance on training, DOD would be better positioned to preserve the historic properties under its purview. GAO is making seven recommendations, including that DOD take steps to verify that privatized military homes are identified and evaluated for historic significance; clarify the inventory requirement for historic properties; and develop guidance related to historic preservation training. DOD concurred with the recommendations.", "document_type": "gao"}
{"report": "Federal policy for rental housing has traditionally focused on assisting low-income households through rental assistance and incentives for the development of housing with below-market rents. In fiscal year 2020, Congress appropriated about $43.9 billion for HUD’s three largest federal rental assistance programs: public housing, Housing Choice Vouchers, and Project-Based Rental Assistance. These programs make rents affordable to eligible households, generally by paying the difference between the unit’s rent and 30 percent of a household’s adjusted income. Unlike certain other means-tested programs, federal rental assistance programs are not entitlements. The number of households that the programs can assist is limited by the amount of budget authority that HUD requests and Congress provides through the annual appropriations process. Historically, appropriations for rental assistance programs have not been sufficient to assist all households that HUD has identified as having worst case housing needs—that is, renter households that (1) have very low incomes; (2) do not receive housing assistance; and (3) use more than one-half of their income to pay for rent, live in severely inadequate conditions, or both. In 2017, HUD reported that 8.3 million households had worst case needs in 2015, an increase from 7.7 million in 2013. HUD reported that among very low-income renters in 2015, 25 percent of them received rental assistance, and an additional 43 percent had worst case needs. To determine program eligibility and identify populations in need of assistance, many federal rental assistance programs have specific income eligibility requirements. HUD sets income limits that determine eligibility for its assistance programs based on median family income and market rent estimates. These income limits can vary across different types of localities. The national rentership rate increased from 2001 through 2017 (see fig. 1). In 2004, the estimated rentership rate fell below 33 percent, the lowest in U.S. history, then climbed to 37 percent in 2013, a rate not seen since the 1960s. By 2017, almost 7 million more households rented their homes than in 2001, which brought the rentership rate to an estimated 36 percent. This increase of 7 million households reflects both overall growth in the population as well as the net shift from owning to renting. Many households experienced lasting financial effects of the financial crisis, such as impaired credit or loss of income, which hampered their ability to enter into or transition back into homeownership. Although the national foreclosure rate has slowed significantly in recent years, past research has shown that most households struggle to return to homeownership after foreclosure. Further, median home prices have risen faster than median incomes nationally, which makes achieving homeownership more challenging. Specifically, the gap between rising home prices and wage growth has likely contributed to increases in rentership in many metro areas. Nationally, the rentership rate increased from 2001 through 2017 across all age categories we analyzed, except for older households (65 years or older), as shown in figure 2. The greatest increase was among early middle-aged households (35–49 years old), an estimated increase of nearly 8 percentage points. In addition, rentership for late middle-aged (50–64 years old) and younger (20–34 years old) households increased by 5 percentage points. Renters are, on average, older than they previously were. The late middle-aged group (50–64 years) experienced the largest estimated increase in the number of renter households—an increase of 4 million households—and accounted for more than half of the total increase in renter households from 2001 through 2017 (see fig. 3). Many of these households have not recovered from the financial crisis, and this group has lower incomes and higher rentership rates than in previous generations, Harvard’s Joint Center for Housing Studies has reported. We previously reported that the homeownership rate for the poorest older households was significantly lower after the financial crisis than before it. Black households had higher estimated rentership rates than White, Hispanic, and Asian households, and rentership among Black households increased from 54 percent in 2001 to 58 percent in 2017 (fig. 4). In contrast, rentership among White households was lowest among the race/ethnicity groups and remained generally stable during our analysis period (ranging from 26 to 29 percent from 2001 through 2017). While rentership among Hispanic and Asian households increased slightly in the aftermath of the financial crisis, as of 2017, their rentership rates had returned to levels below those of 2001, although these rates were still higher than those of White households. As of 2017, high-growth and moderate-growth/high-density metro areas we analyzed tended to have more racially diverse renter populations than other areas, and renters in these metro areas were mostly from minority groups. For example, in Dallas, Texas, which is high-growth, an estimated 59 percent of renter households were minority households, and in Miami, Florida, which is moderate-growth/high-density, an estimated 75 percent of renter households were minority households. The most significant change in rentership from 2001 through 2017 by income group was for higher-income households (more than 120 percent of area median income), with the greatest change between 2010 and 2017. Nationally, higher-income households were the second smallest renter group in 2001, with an estimated 6.6 million households, or 17 percent of renter households. In 2017, higher-income households were the second largest renter group, with approximately 10.3 million households, approximately 20 percent of renter households (see fig. 5). Consistent with national trends, in all locality types—that is, those with higher and lower population density or rates of growth—the estimated number and proportion of higher-income renter households increased from 2001 through 2017 (see fig. 6). The greatest increase occurred in high- and moderate-growth metro areas. This trend could reflect (1) a change in income, (2) relocation from moderate-growth/high-density metro areas, and (3) consolidation of households—such as having multiple roommates, extended families occupying one housing unit, or households doubling up with relatives or others to make ends meet. There were modest changes in the number and proportion of low-income households during the same period. Rural areas and metro areas with shrinking populations had the highest proportion of renter households with low incomes as of 2017—for example, an estimated 63 percent of renters in negative-growth metro areas had low to extremely low incomes. Population growth and two other factors appear to have contributed to the growth in higher-income renter households. First, many homeowners who experienced foreclosure during the financial crisis became renters. Second, with rising housing costs, there has been a trend toward consolidated households. The share of households with three or more adults was higher in 2017 than in 2001. Some of these households may have chosen to combine as an alternative to eviction or homelessness, and they may have overcrowded or unstable living arrangements. Most renter households paid a larger share of their income in rent in 2017 than in 2001. Federal housing policy generally considers rents at or below 30 percent of household income to be affordable, and households that pay more than 30 percent of income in rent are considered to be rent burdened. We found that by 2017, an estimated 48 percent of renter households were rent burdened, 6 percentage points higher than in 2001. Severe rent burden, where more than 50 percent of household income is paid in rent, also became more common. Of the households that were rent burdened in 2017, about half were severely rent burdened. These households represented 24 percent of all renter households—an increase of 4 percentage points from 2001 (see fig. 7). The rising rent burden is part of a long-term trend in rental unaffordability, as supply has not kept pace with demand for rental units. With fewer affordable apartments available, rent burdens increased among lower- income households, who were forced to spend a greater proportion of income on rent. Government, academic, and industry research has identified several factors that contribute to this trend: Local regulation and geography have long constrained where and how much rental housing can be built. Cities have adopted zoning and land use regulations that can prohibit or increase costs for new rental units. Metro areas, particularly those in coastal or mountainous regions, have limited available land for new housing. Construction of new rental units has been limited since the 2007– 2009 financial crisis, in part because developers struggled to rebuild workforce capacity after layoffs of skilled construction workers. As a result, since 2009, the construction industry has focused on building luxury apartments, which have higher profit margins, and produce few units affordable to lower-income households. Conversion of lower-rent units to higher-rent units through renovation also reduced the number and share of rental units affordable to lower-income households. Demographic changes, particularly the aging of the millennial and baby boomer generations, have increased demand for rental units. As previously discussed, we found that renters were, on average, older in 2017 than in 2001. In addition, Harvard’s Joint Center for Housing Studies has reported that late middle-age renters (50–64 years) have lower incomes and higher rentership rates than previous generations. Populations with higher rentership rates—including minority households—are forecasted to continue growing through 2030. The spike in foreclosures during the financial crisis resulted in millions of households entering the rental market, increasing competition for available units. Tighter credit standards after the financial crisis have kept many of those who lost their home due to foreclosure from qualifying for a new mortgage. In the United States, rent burden has been most common among minorities and older adults and in dense metropolitan areas (see fig. 8): Rent burden was about 10 percentage points more common among Black and Hispanic households than White households in 2017. This disparity was due to sizable differences in median income. In 2017, estimated median income was $63,704 for White households, $49,793 for Hispanic households, and $40,232 for Black households. Rent burden was more than 10 percentage points more common among older adult (65 and over) households than working-age (20– 64) households in 2017. This disparity was also due to sizable differences in median income, as older adults were less likely to be in the workforce. In 2017, median income was $69,459 for households age 25–64 and $43,735 for households age 65 and over. Rent burden was nearly 10 percentage points more common among renters in high-density metro areas than in nonmetro areas in 2017. According to the Urban Institute, the shortage of affordable rental housing was more acute in urban areas than rural areas in 2014. See appendixes III and IV for more detailed information on rent burden by age, race/ethnicity, and locality type. In 2017, moderate and severe rent burdens were common among low- to extremely-low income households and relatively rare among moderate- to higher-income households (see fig. 9). From 2001 through 2017, the estimated number of renters with moderate or severe rent burdens increased across all income levels, but the increase was more pronounced among lower-income groups (see fig. 10). Specifically, we found the following: The estimated number of higher-income renters increased by more than 3.6 million households from 2001 through 2017, but relatively few of these households experienced rent burden. In contrast, the numbers of low-income, very low-income, and extremely-low income renters also increased over this period, and these groups saw significant increases in rent burden. In more recent years, the estimated number of extremely low-income renter households with severe burden actually decreased—from 7.4 million in 2011 to 6.6 million in 2017. This decrease, however, does not necessarily indicate improved conditions for these households because it was not accompanied by a corresponding increase in either (1) the number of extremely low-income households that were less burdened or (2) the number of very low-income households (the next highest income group). An increase in either of these groups could indicate that the poorest, most burdened households experienced either an increase in income or a decrease in rent burden. However, because these other groups did not increase, it is possible that some of these extremely low-income, severely burdened households moved in with other households or experienced some other form of homelessness. Rent burdens affect households differently depending on their income. Households with lower incomes may pay the same percentage of income in rent as moderate- or higher-income households but have less income left over for other necessities. Even relatively inexpensive units may not leave enough money for lower-income households to cover other necessities like food, clothing, transportation, or medical care. These households may also be sensitive to shocks, such as job loss and health emergencies, and may be at heightened risk of eviction and homelessness. Challenges that lower-income households face can vary across cities and regions due to differences in local market rents and incomes. For example, as figure 11 shows, in the San Francisco area in 2017 a very low-income family of four would experience a severe rent burden if it paid the fair market rent for a two-bedroom apartment ($3,018 per month). Such a family would struggle to pay the rent and afford other necessities even with two or three full-time minimum wage jobs. In contrast, a very low-income family of four in the St. Louis area in 2017 would experience a moderate or no rent burden if it paid the fair market rent for a two- bedroom apartment ($896). Such a family with at least two full-time minimum wage jobs would have relatively more money left over for other necessities. See appendixes III and IV for more detailed information on rent burden by household income. For moderate-income households, the consequences of rent burden are less dire than for lower-income households, but they are still significant. For example, a family of four earning the median income in San Francisco that paid fair market rent for a two-bedroom apartment would be rent burdened. A housing unit that would be considered affordable to them would cost at least $135 per month below fair market rent (or approximately $2,882 or less). Money that a family could save on a unit below fair market rate could help reduce household debt, add to retirement savings, or pay for necessities like child care. Rent burden among moderate-income households tends to be more common in large cities with strong economies and significant geographic and regulatory constraints on new housing, such as San Francisco and New York. The lowest-income households face challenges securing affordable rental units. There are not enough rental units that are affordable to the lowest- income households without rental assistance. Specifically, according to HUD, lower-income households face competition from moderate- or high- income households to rent affordable units. HUD’s analysis showed that although there were enough affordable units nationwide to house 66 percent of extremely-low income renters in 2015, 43 percent of those units were occupied by renters with higher incomes. We also found that for all income groups, rents rose faster than incomes and therefore became less affordable to varying degrees. Specifically, estimated median rent-to-income ratios, which indicate the median proportion of income devoted to rent, generally increased from 2001 through 2017, according to our analysis (see fig. 12). For the lowest- income households, even small declines in affordability have a big impact because these households face the highest rent burdens and have the fewest options in the housing market. See appendix III for more detailed information on rent-to-income ratios. Based on two indexes we created to analyze rental housing conditions using American Housing Survey data, we found that an estimated 15 percent of renter households—more than 5 million—lived in units with serious deficiencies in 2017. Specifically, an estimated 12 percent of renter households (more than 4 million households) lived in units with substantial quality issues. These units typically had a combination of issues, such as cracked walls and the presence of rodents, or multiple heating problems and the presence of rodents. An additional 3 percent of renter households (more than 1 million households) lived in incomplete units—that is, units lacking essential components of a dwelling (such as heating equipment or hot and cold running water). Further, an estimated 28 percent of households—nearly 10 million—rented units with less substantial quality issues. Table 2 presents these findings and how our indexes described different types of rental housing conditions. The proportion of rental units with the three types of deficiencies— substantial quality issues, less substantial quality issues, and absence of essential components of a dwelling—generally remained stable from 2001 through 2017 (see fig. 13). The proportion of rental units that had at least one of these deficiencies ranged from an estimated 39 to 47 percent from 2001 through 2017. We analyzed rental housing conditions by renter household and rental unit characteristics. Households with low incomes (those with low, very low, or extremely low incomes) or with rent burdens comprised half or more of renters living in units with substantial quality issues and incomplete housing units (those lacking essential components of a dwelling). Although incomplete housing units represented a small percentage of rental units overall (about 3 percent), there were more than an estimated 1 million such units in 2017. Low-income renters have fewer affordable options and, as a result, may end up in units with deficiencies out of necessity. Households with low, very low, or extremely low incomes represented an estimated 62 percent of renters overall in 2017. These households occupied an estimated 67 percent of units that had substantial quality issues and nearly 80 percent of incomplete units. Similarly, rent-burdened households represented an estimated 50 percent of renters overall in 2017 and occupied an estimated 53 percent of units with substantial quality issues and 60 percent of incomplete units. There were some notable differences in housing conditions by age and race/ethnicity. Older households (65 and older) were the most likely age group to live in rental units with no deficiencies in 2017. About half of renting households were White in 2017, and White households comprised the largest share of renters in each quality or completeness category we analyzed. The proportions of Hispanic and Asian households that rented incomplete units (estimated at 31 percent and 11 percent, respectively) were higher than the overall proportions of Hispanic and Asian renter households (estimated at 20 percent and 6 percent, respectively). In addition, the proportion of Black households that rented units with substantial quality issues (estimated at 24 percent) was slightly higher than the overall proportion of Black renter households (estimated at 21 percent). Rental housing conditions by unit age or type were generally consistent from 2001 through 2017. As expected, units built after 2000 had fewer deficiencies than those built before. Older rental housing—units built prior to 1980—were more likely to have substantial quality issues than those built after. An estimated 63 percent of units in high-growth metro areas had no quality issues as of 2017, compared to an estimated 55 to 57 percent of units in other types of localities. There was little other variation in housing conditions by locality type. We also found that detached single-family homes and mobile homes were somewhat more likely to have serious deficiencies than multifamily units. The proportion of units with these deficiencies remained relatively steady from 2001 through 2017. One reason for this is single-family units lack on-site building managers and other benefits of shared maintenance that multifamily units may provide. Some researchers and industry participants have noted possible maintenance challenges for a growing number of investor-owners of single-family rentals that manage thousands of properties of varying size, age, and condition. From 2001 through 2017, the proportion of single-family units with serious deficiencies (rental units lacking essential components of a dwelling or units with substantial quality issues) ranged from around 13 to 20 percent (see fig. 14). During the same period, the proportion of single-family units with less substantial quality issues ranged from an estimated 28 to 34 percent. We also analyzed household crowding trends based on American Community Survey data. We defined crowded households as those having more than two people per bedroom. From 2001 through 2017, the incidence of renter household crowding decreased, with the greatest percentage point declines for Hispanic households prior to the housing crisis. Generally, households that were younger, Hispanic or Asian, or had lower incomes were more likely to experience crowding. In addition, crowded households were more common in high-density and high-growth metro areas. Appendix VI includes information on household crowding by race/ethnicity, age, household income, and locality type. We provided a draft of this report to HUD for review and comment. HUD officials told us that they had no comments on the draft report. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Housing and Urban Development, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4529 or garciadiazd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. The objectives of this report were to analyze trends in (1) the share of households that rent and their characteristics, (2) the affordability of rental housing, and (3) rental housing conditions. We analyzed 2001–2017 data from the American Community Survey and American Housing Survey to describe renter household characteristics, rent affordability, rental housing conditions, and trends at the national level and across different types of localities. The American Community Survey is an ongoing survey administered by the Census Bureau of around 3.5 million households across the United States; the data we used in our analysis were current as of 2017, the most recently available data at the time of our review. The survey collects data on the economic, social, housing, and demographic characteristics of communities at various geographic levels, including metropolitan areas, states, and counties. The American Housing Survey is a biennial survey sponsored by the Department of Housing and Urban Development (HUD) and administered by the Census Bureau that collects a range of housing information, including the size and composition of the U.S. housing inventory, physical condition of housing units, characteristics of occupants, and other information. Findings from each survey are subject to sampling errors. To assess the reliability of the data, we reviewed technical information for each survey. In addition, we interviewed HUD and Census Bureau officials to identify differences across survey years and understand geographic limitations of publicly available data. We determined that the surveys were sufficiently reliable for purposes of reporting at the national level on renter household characteristics. However, we determined that additional, nonpublic data were needed from each survey to analyze renter household characteristics, rent burden, and rental housing conditions for smaller geographic units. To address this limitation, staff from HUD’s Office of Policy Development and Research provided us with aggregated Census Bureau data. To assess the reliability of these data, we analyzed the underlying programming code and related documentation from agency officials and reviewed for missing data, outliers, and errors. We determined that the data were sufficiently reliable for purposes of analyzing renter household characteristics, rent burden, and rental housing conditions from 2001 through 2017 at the national level and for different types of localities. For all objectives, to describe common trends and differences across localities—that is, localities with different population growth rates and densities—we developed metro area groupings. The groupings provide a general framework for describing metro areas that experienced varying degrees of population growth from 2000 through 2017 and how trends in renter household characteristics, rent affordability, and rental housing conditions compared to trends in other types of areas. To identify the locality types, we analyzed core-based statistical areas by population growth from 2000 through 2017 and population density as of 2017. We identified three growth categories (high, moderate, and negative) and further categorized the moderate growth group by density (high and moderate). We also identified a group of nonmetro areas consisting of all counties in each state that are outside the boundaries of any metro area. These areas included micropolitan areas, small towns, and low-density rural areas. The five locality types were high-growth metro areas, moderate-growth/high-density metro areas, moderate- growth/moderate-density metro areas, negative-growth metro areas, and nonmetro areas. To describe trends in the share of households that rent and their characteristics, we analyzed American Community Survey data from 2001 through 2017 at the national level and across different types of localities, with a focus on renter household age, race/ethnicity, and income. We defined four head-of-household age categories: younger (20–34 years old), early middle age (35–49 years old), late middle age (50–64 years old), and older (65 years and older). We reported on five race/ethnicity categories, combining some Census categories for our analysis: White, Black, Hispanic (an ethnicity that applies to individuals of any racial background), Asian (includes Asian, Native Hawaiian, and Other Pacific Islander), and Other (includes American Indian, Alaska Native, two or more races, and some other race). We defined five income categories based on income ranges that HUD uses for determining rental assistance eligibility or reporting to Congress on worst case needs: extremely low income (up to 30 percent of HUD area median family income (HAMFI)); very low income (more than 30, up to 50 percent of HAMFI); low income (more than 50, up to 80 percent of HAMFI); moderate income (more than 80, up to 120 percent of HAMFI) and higher income (greater than 120 percent of HAMFI). To describe trends in the affordability of rental housing, we analyzed American Community Survey data on gross rent as a percentage of household income from 2001 through 2017 at the national level and across different types of localities. Consistent with other housing research and HUD policies, we defined rent burden as spending more than 30 percent of household income on rent, moderate rent burden as spending more than 30 and up to 50 percent of household income on rent, and severe rent burden as spending more than 50 percent of household income on rent. Further, as described in appendix IV, we developed a supplementary analysis of rental housing affordability for rural areas by state. To describe trends in rental housing conditions, we analyzed data from the American Community Survey and American Housing Survey from 2001 through 2017 at the national level and across different types of localities. HUD designed the American Housing Survey to include indicators of housing quality. HUD analyzes and reports periodically on a housing adequacy measure as part of its worst case housing needs assessments for Congress. HUD’s adequacy measure and related research informed our methodology for developing two indexes to analyze rental housing conditions. We developed the indexes to more specifically define the range of housing conditions. The two indexes include 13 quality-related variables and nine variables we identified as essential components of a dwelling from the American Housing Survey, described in table 3. Appendix II includes more detailed information about our methodology. Appendix V includes information on the similarities and differences between HUD’s adequacy index and the indexes we developed for this report. With our indexes, we analyzed trends in rental housing conditions by renter household characteristics and rental unit characteristics. The renter household characteristics we analyzed included household income and affordability, race/ethnicity, and age. The rental unit characteristics we analyzed included location, age, and structure type. In addition, from American Community Survey data, we analyzed household crowding as another aspect of housing conditions. Further, we reviewed reports and studies on housing conditions and interviewed stakeholders including federal agency officials, academic experts, and research organizations. To further describe trends in renter household characteristics, rent affordability, and rental housing conditions during our review period, we reviewed reports and studies by federal agencies and research organizations and interviewed a variety of stakeholders selected for their knowledge of these issues, including federal agency officials from HUD, the Census Bureau, Congressional Research Service, the Department of Agriculture, the Federal Housing Finance Agency, and the Department of the Treasury; academic experts, including researchers from Harvard’s Joint Center for Housing Studies and others; research organizations that included the Bipartisan Policy Center, various researchers associated with the Board of Governors of the Federal Reserve System, Brookings Institute, Center on Budget and Policy Priorities, Housing Assistance Council, National League of Cities, National Rural Housing Coalition, Urban Land Institute, and Urban Institute; and industry groups that included the National Association of Home Builders, National Association of Realtors, and the National Housing Conference. We conducted this performance audit from February 2018 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. This appendix provides additional details on our analysis of the conditions of the national rental housing stock between 2001 and 2017. To assess rental housing conditions, we used data from the national American Housing Survey (AHS), which is administered by the Census Bureau and conducted every odd year. Specifically, we considered two concepts, unit completeness and unit quality, and relied on questions which were consistently asked over the 2001 to 2017 period to define nine completeness and 13 quality variables. These are described in table 4. The survey questions underlying uncomfortably cold periods and heating equipment breakdowns were only asked of respondents who occupied their unit in the winter prior to the survey year, so our main analysis of rental unit quality only considered cash-rent housing units occupied by households since the prior winter, while the analysis of unit completeness considered all cash-rent units. See table 5 for the distributions of the completeness and quality variables in 2017. We used each set of variables to construct two indexes, one of unit completeness (an indicator) and one of unit quality (a continuous measure). We collapsed the continuous quality index into three categories (no quality issues, less substantial quality issues, and substantial quality issues) to facilitate a summary of rental housing quality trends. To determine rental unit completeness, we first summed the number of missing components contained in the nine completeness components for each rental unit in the surveys. We obtained an estimate of Cronbach’s alpha associated with this sum of 0.40, which was low enough to suggest that a simple indicator would be a more appropriate summary measure. We therefore determined that each of the nine completeness components was essential for us to consider a unit livable, and assigned a completeness score to each rental unit based on the absence of any of them. The resulting index therefore measured incompleteness, where a score of 1 indicated that a unit was missing one or more of the essential components. See table 6 for the distribution of the completeness index in the survey years between 2001 and 2017. for inference that is robust to non-normal distributions of the latent continuous variables. Finally, we obtained quality score estimates by empirical Bayes, which selects the mode of the posterior distribution 𝑝𝑝(𝜂𝜂̂|𝑦𝑦𝑖𝑖) evaluated at the estimates of the model’s parameters. Note that because all quality variables increased in the presence or number of issues, the quality index correspondingly increased in poor quality. We first estimated two variants of the factor model, one accounting for the sample weights assigned to units in each survey (our preferred specification), and one that did not account for these weights. The robust root mean square error of approximation from the latter of 0.020 suggested that our single factor model provided an appropriate representation of the AHS data. Estimates of the polychoric correlation matrix and of the factor loadings are reported in tables 7 and 8 respectively. We assessed stability by estimating the model on each AHS year separately and broadly found that factor loadings estimates varied little over time. Given the estimates from our preferred specification, which accounts for the survey design, we then relied on empirical Bayes estimation to assign a quality score to each unit for which responses to all 13 quality variables were observed. The full distributions of the resulting quality index in the survey years between 2001 and 2017 are reported in figure 15. We then selected thresholds in the distribution of the continuous quality index to distinguish between units without any quality issues, units with a quality score indicating the presence of less substantial issues, and units with a score denoting more substantial issues (those with either a combination of some of the most severe issues as determined by the model, or a large number of issues of varying severities). The first threshold between units with no quality issues and units with at least one issue occurred at a score of -0.2280. Units with no issues represented between 54 and 62 percent of the rental units to which we were able to assign quality scores. To further separate units experiencing any issues into two groups, we inspected the quality score distribution for local minima in its density to find a score around which small perturbations in threshold choice would have little effect on the share of units falling into each of the two groups. We examined all quality issue profiles experienced in units with scores in the region around two candidates where the density nearly reached 0, and selected a score of 0.5240 as the second threshold, immediately above which were units with one or more holes in the floor large enough to catch someone’s foot. All units with a quality score of 0.5240 or higher were therefore considered to have substantial issues. Table 9 reports the share of cash-rent, previous-winter-occupied units for each quality level in the survey years between 2001 and 2017, and table 10 reports the most common quality issue profiles in 2017. Our analysis is subject to several limitations. In determining both unit completeness and quality, we were limited to the variables consistently available across all survey years. Therefore, we could not include features not observed in the AHS which could be deemed to be important components of either unit completeness (such as a unit’s access to an internet service provider) or quality (such as the presence of major defects in the structure of the unit’s building). In the quality factor model, we assumed that quality was uncorrelated with the error term from each measurement equation and that the error terms were uncorrelated with each other to obtain estimates of the model’s parameters, and ultimately the quality scores. A violation of these assumptions would bias the estimates. For example, if rental units located in regions with harsh weather were of systematically worse quality than units in fairer weather regions, the estimated effect of poor quality on a variable like the number of outdoor leaks could be overstated, which would in turn overweight the importance of outdoor leaks in the estimation of the quality scores, resulting in overly poor quality score estimates for units experiencing outdoor leaks. Conversely, if units in harsh weather regions were of systematically better quality than those in fairer weather regions (e.g. as a measure of resilience) the estimated effect of poor quality on outdoor leaks would be understated, biasing down the importance of outdoor leaks in the estimation of quality scores. In general, any systematic linear relationship between latent quality (𝜂𝜂) and the unobserved factors (𝜀𝜀𝑖𝑖) affecting one of the 13 unobserved latent continuous variables, or between the unobserved factors themselves, would be a violation of the model’s assumptions. Since the two quality variables recording uncomfortably cold periods and heating equipment breakdowns were only asked of respondents who occupied their unit in the winter prior to the survey year, we could not assign quality scores to the 10 to 25 percent of rental units across years which were occupied by recent movers. To assess potential biases on the quality distribution of the full cash-rent-occupied rental housing stock introduced by excluding this group, we therefore compared both groups along the remaining 11 dimensions Of the 11 observable quality variables, three exhibited an incidence of issues that differed meaningfully across the two groups. These differences were persistent throughout survey years and consistent in their direction: units whose respondents moved in later than the winter prior to the interview were between 5 to 10 percentage points less likely to experience any outside leaks, inside leaks, and to report evidence of rodents. The differences were meaningful in that they corresponded to over a halving of the incidence of the evidence of rodents, and up to a halving of the incidence of both types of leaks in the recent-mover units relative to the units for which all quality variables were available. To evaluate the effect of these differences on the quality distribution of the full universe of cash-rent units, we estimated a modified quality factor model in which we dropped the uncomfortably cold periods and heating equipment breakdowns variables. This allowed us to obtain quality scores for both the units with the original scores and the recent-mover units. The distributions of the modified quality indices in the two groups reached their largest difference at the share of units without any of the set of 11 quality issues, and we estimated that across all survey years, 1.3 to 2.4 percentage points more units would likely have no measured quality issues in the full cash-rent universe than we found in that which excludes the recent movers. Furthermore, the distributions of the modified indices truncated to exclude the respective units without any of the 11 quality issues were largely comparable. In the full universe of cash-rent units, we would therefore expect decreases in each of the shares of less substantial issues and substantial issues units proportional to their respective shares in the partial universe, and in sum corresponding to the magnitude of the increase in units with no issues each year. The alternative of including recent movers in our main model at the expense of the uncomfortably cold periods and heating equipment breakdowns variables would have yielded a share of units without any other quality issues that we estimated to be 3 to 4 percentage points higher than the share calculated using the original index in the partial universe. Because we believed that these variables should ultimately be included in the quality index, and because we considered the biases we estimated to be relatively small, we retained the original index. In this appendix we present our analysis of rentership and housing affordability by age, race/ethnicity, locality type, and income from 2001 through 2017. The data on renter households are from the American Community Survey’s 1-year estimates. In this appendix we present state-level analysis of housing affordability for rural renter households. While rental affordability is a challenge in both rural and urban areas, differences in demographics, economies, housing stock, and federal rental assistance programs make rural rental affordability issues unique We defined rural areas using the U.S. Department of Agriculture’s 2010 rural-urban commuting area (RUCA) codes. The data on renter households living in these areas are from the American Community Survey’s 5-year estimates for 2013 through 2017. While renter households lived in rural areas of all 50 states, generally the most populous states had the largest populations of renter households in rural areas (fig. 16). From 2013 through 2017, more than an estimated 2.2 million renter households lived in rural areas. The states with the largest estimated populations of renter households in rural areas were Texas (119,000), Missouri (96,000), Wisconsin (96,000), and Kentucky (93,000). The prevalence of rural renter households varied significantly by state. While only about 5 percent of renter households lived in rural areas from 2013 through 2017, some states had significantly larger proportions of renters in rural areas. States with higher estimated proportions of rural renter households generally had small populations and were in northern New England or along the Missouri, Mississippi, or Ohio Rivers (fig. 17). The states with the largest estimated proportions of renter households in rural areas were Vermont (39 percent) and Montana (32 percent). Renter households in rural areas generally had lower incomes than other renter households. From 2013 through 2017, while the median income for renter households overall was an estimated $36,653, nearly three in five rural renter households had incomes lower than $35,000. For context, a household with two full-time jobs earning the federal minimum wage in 2017 would earn approximately $30,160. In general, Southern states had the highest estimated proportion of rural renter households with incomes less than $35,000 (fig. 18). The states with the smallest proportion of rural renter households with incomes lower than $35,000 were New Jersey (25 percent), Rhode Island (32 percent), Alaska (35 percent), Hawaii (39 percent), and Connecticut (39 percent). Rent burden was common among renter households in rural areas, but prevalence varied by state. Rent burden was slightly less common among rural renter households from 2013 through 2017 (45 percent) than renter households in general in 2017 (48 percent). In eight of 48 states, at least 50 percent of rural renter households were rent burdened (fig. 19). In general, rural rent burden was most common in the Northeast, South, and West Coast, and least common in the U.S. interior. Louisiana had the highest estimated rate of rent burden among rural renter households (55 percent) and Wyoming had the lowest (33 percent). Rent burdens were more common among rural households with incomes below $35,000. From 2013 through 2017, an estimated 70 percent of these households were rent burdened, and in no individual state were less than 50 percent of these households rent burdened (fig. 20). The five states with the highest proportion of lower-income rural renter households that were rent burdened were Alaska (81 percent), Massachusetts (83 percent), Hawaii (83 percent), California (83 percent), and Delaware (85 percent). As discussed previously in this report, lower-income households with rent burdens may struggle to pay for essential needs like food, transportation, health care, and clothing. Rent burdens were uncommon among rural households with incomes of $35,000 or greater. From 2013 through 2017, only an estimated 9 percent of these households were rent burdened, and in no state were more than 30 percent of these households rent burdened (fig. 21). In 40 of 48 states, less than an estimated 15 percent of rural renter households with incomes of $35,000 or greater were rent burdened. The four states with the highest proportion of rural renter households with income $35,000 or greater that were rent burdened were Connecticut (28 percent), Hawaii (26 percent), California (24 percent), and Massachusetts (22 percent). This appendix describes how the indexes we developed to analyze rental housing conditions compare to an index the Department of Housing and Urban Development (HUD) uses to measure housing adequacy. Although our index uses many of the same American Housing Survey variables as HUD’s adequacy index, differences in our analytic methods allowed us to produce more detailed results on housing conditions. HUD measures housing adequacy as part of its ongoing efforts to analyze and report on worst case housing needs The adequacy index is a measure that is based on 19 variables in the American Housing Survey. It categorizes housing units as severely inadequate, moderately inadequate, or adequate based on whether a surveyed housing unit meets certain conditions or criteria. Severely inadequate housing units represented 2 to 3 percent of all rental units from 2001 through 2017. We developed two indexes based on a factor analysis of 13 quality- related variables and nine variables we identified as essential components of a dwelling. We determined that two indexes were needed to describe rental housing unit conditions based on American Housing Survey data, as relevant variables fell into two categories that required different statistical treatment and interpretation. Figure 22 provides a detailed comparison between the variables and scoring techniques of our indexes and HUD’s adequacy index. We compared HUD’s 2017 housing adequacy findings to the results of our indexes and identified some notable differences. Among rental units that HUD considered adequate in 2017, an estimated 8 percent had substantial quality issues as measured by our quality index—affecting 2.7 million households. In addition, another estimated 9.7 million units had less substantial quality issues. These units did not satisfy HUD’s scoring criteria for inadequate or moderately inadequate units, but they had a combination of issues that exceeded our statistical thresholds for substantial and less substantial quality issues. Figure 23 provides a detailed comparison of how our results compare to HUD’s. This appendix provides additional information on rental housing conditions by household income, affordability (rent burden), race/ethnicity, age, rental unit age, and structure type, based on two indexes we developed to analyze American Housing Survey data The appendix also includes information on household crowding based on our analysis of American Community Survey data by household income, rent burden, race/ethnicity, and age. In addition to the contact named above, Heather Chartier (Analyst in Charge), Jeremy Anthony, Daniel Benson, Abigail Brown, Stephen Brown, Nina Daoud, Davis Judson, John McGrail, Yann Panassie, Dae Park, Lena Richenberg, Paul Schmidt, Jennifer Schwartz, Jena Sinkfield, Farrah Stone, and Jeff Tessin made key contributions to this report.", "summary": "Since the 2007–2009 financial crisis, growth in the share of renter households has reversed a decades-long trend toward homeownership. This change has underscored concerns about the availability, affordability, and condition of rental housing, especially for low-income households. The federal government subsidizes rents for around 4.4 million households per year, but more households qualify for assistance than receive it. GAO was asked to provide a comprehensive assessment of the housing market. This report examines trends in the housing market prior to the COVID-19 pandemic and does not account for the profound impact it will likely have on renter households. This report, one of several GAO plans to issue, focuses on rental housing from 2001 through 2017 and analyzes (1) the share of households that rent, (2) the affordability of rental housing, and (3) rental housing conditions. GAO analyzed American Community Survey and American Housing Survey data from 2001 through 2017 (the most recent data available at the time of this review) at the national level and for different types of localities. GAO also reviewed recent reports by the Department of Housing and Urban Development (HUD), research organizations, and academic researchers on rental housing and obtained views from a variety of stakeholders selected for their knowledge of these issues, including federal agency officials, academic experts, research organizations, and industry groups. In 2017, almost 7 million more households rented their homes than in 2001, which brought the share of households that rent from an estimated 34 percent to 36 percent. Renting became more common after the 2007–2009 financial crisis as foreclosures and changes in household characteristics reduced the proportion of homeowners. Renting was more prevalent across most age and race/ethnicity groups in 2017 than in 2001, with notable increases among higher-income households. Rental affordability declined from 2001 to 2017. In 2017, 48 percent of renter households were rent burdened—that is, they paid over 30 percent of income for rent—which is 6 percentage points higher than in 2001. Rent burden was most common and most severe among lower-income households (80 percent or less than area median income), with almost three-quarters of extremely low-income households (30 percent or less than area median income) paying over half of their income in rent (see figure). Affordability declined because of a range of factors, including more households competing for rental units and the supply of low-cost rental units not keeping up with demand. Note: Estimates in this figure have a margin of error of ±2 percentage points or fewer, at the 95 percent confidence level. An estimated 15 percent of rental units in 2017—more than 5 million—had substantial quality issues (such as cracked walls and the presence of rodents) or lacked essential components of a dwelling (such as heating equipment or hot and cold running water), according to GAO's analysis of American Housing Survey data. The share of units with deficiencies was relatively stable from 2001 to 2017. Serious deficiencies more often affected households with extremely low incomes or rent burdens. In addition, lower-income households rented approximately two-thirds of the units with substantial quality issues and nearly 80 percent of units lacking essential components.", "document_type": "gao"}
{"report": "CERCLA established the Superfund program to clean up contaminated sites to protect human health and the environment from the effects of hazardous substances. CERCLA requires the President to establish procedures and standards for prioritizing and responding to releases of hazardous substances, pollutants, and contaminants into the environment and to incorporate these procedures and substances into the National Oil and Hazardous Substances Pollution Contingency Plan (National Contingency Plan). Under CERCLA, PRPs are liable for conducting or paying for the cleanup of hazardous substances at contaminated sites. EPA and PRPs can undertake two types of cleanup actions: removal actions and remedial actions. Removal actions are usually short-term cleanups for sites that pose immediate threats to human health or the environment. Remedial actions are generally long-term cleanups—consisting of one or more remedial action projects—that aim to permanently and significantly reduce contamination. Remedial actions can take a considerable amount of time and money, depending on the nature of the contamination and other site-specific factors. EPA’s Office of Superfund Remediation and Technology Innovation, which is part of the Office of Land and Emergency Management, oversees remedial actions at NPL sites, including nonfederal NPL sites. At each nonfederal NPL site, the lead official who is responsible for compliance with the National Contingency Plan is the remedial project manager. Management of nonfederal NPL sites is the responsibility of the EPA region in which a site is located. EPA has 10 regional offices, and each one is responsible for executing EPA programs within several states and, in some regions, territories. Figure 1 illustrates EPA’s 10 regions. The Superfund process begins with the discovery of a potentially hazardous site or notifications to EPA regarding the possible release of hazardous substances that may pose a threat to human health or the environment. EPA’s Superfund remedial cleanup process for nonfederal NPL sites includes the actions shown in figure 2. Site assessment. EPA, states, tribes, or other federal agencies evaluate site conditions to identify appropriate responses to releases of hazardous substances to the environment. During this process, EPA or other entities, such as state or tribal agencies, collect data to identify, evaluate, and rank sites using agency criteria. Site listing. EPA considers whether to list a site on the NPL based on a variety of factors, including the availability of alternative state or federal programs that may be used to clean up the site. Sites that EPA proposes to list on the NPL are published in the Federal Register. After a period of public comment, EPA reviews the comments and makes final decisions on whether to list the sites on the NPL. Remedial investigation and feasibility study. EPA or the PRP will generally begin the remedial cleanup process for an NPL site by conducting a two-part study of the site. First, EPA or the PRP conducts a remedial investigation to characterize site conditions and assess the risks to human health and the environment, among other actions. Second, EPA or the PRP conducts a feasibility study to assess various alternatives to address the problems identified through the remedial investigation. Under the National Contingency Plan, EPA considers nine criteria, including long-term effectiveness and permanence, in its assessment of alternative remedial actions. Record of decision. EPA issues a record of decision that identifies its selected remedy for addressing the contamination at a site. A record of decision typically lays out the planned cleanup activities for each operable unit of the site as well as an estimate of the cost of the cleanup. Remedial design and remedial action. EPA or the PRP plans to implement the selected remedy during the remedial design phase, and then, in the remedial action phase, EPA or the PRP carries out one or more remedial action projects. Construction completion. EPA generally considers construction of the remedial action to be complete for a site when all physical construction at a site is complete, including actions to address all immediate threats and to bring all long-term threats under control. Postconstruction completion. EPA, the state, or the PRP performs operation and maintenance for the remedy, if needed, such as by operating a groundwater extraction and treatment system. EPA generally performs reviews of the remedy at least every 5 years to evaluate whether it continues to protect human health and the environment. Deletion from the NPL. EPA may delete a site, or part of a site, from the NPL when the agency and the relevant state authority determine that no further site response is needed. Nonfederal NPL sites may include a variety of contaminants, and EPA may select different types of remedies to clean up the sites. EPA had recorded more than 500 contaminants at nonfederal NPL sites as of fiscal year 2014, the most recently available data. According to the Agency for Toxic Substances and Disease Registry, the highest-priority contaminants—based on a combination of their prevalence, toxicity, and potential for human exposure—are arsenic, lead, mercury, vinyl chloride, and polychlorinated biphenyls. For example, in 2016, the Agency for Toxic Substances and Disease Registry reported that exposure to arsenic in drinking water is associated with various health effects, such as pulmonary and cardiovascular disease, diabetes, and certain cancers. Contaminants may be found in different media at nonfederal NPL sites. In 2017, EPA reported that groundwater and soil were the most common contaminated media, including at the nonfederal NPL sites it analyzed. To clean up a nonfederal NPL site, EPA may select various on-site or off- site remedies. For example, EPA may select on-site remedies that include treatment as well as those that do not, such as on-site containment, monitored natural recovery, and institutional controls. In 2017, EPA reported that about a quarter of the decision documents for sites it analyzed included on-site treatment. EPA may also treat or dispose the contamination off-site. Examples of off-site treatment and disposal include incineration and recycling. EPA reported that sites it analyzed may have various combinations of remedies, including treatment, on-site containment, off-site disposal, and institutional controls. Various federal agencies provide nationwide data on flooding, storm surge from hurricanes, wildfires, and sea level rise. Data on flooding, storm surge, and wildfires are generally based on current or past conditions. NOAA models the extent of inundation for various heights of sea level rise compared to the most recently available data on average high tide. FEMA provides flood hazard and risk information to communities nationwide. Among other information, FEMA provides data on coastal and riverine flooding in the National Flood Hazard Layer, a database that contains the most current flood hazard data. Federal law requires FEMA to assess the need to revise and update the nation’s flood maps at least every 5 years. Among other uses, the flood hazard data are used for flood insurance ratings and floodplain management. The National Flood Hazard Layer identifies areas at the highest risk of flooding, which are those that have a 1 percent or higher annual chance of flooding. In some locations, the National Flood Hazard Layer also identifies areas with 0.2 percent or higher annual chance of flooding, which FEMA considers to be a moderate flood hazard, as well as other flood hazards. The National Flood Hazard Layer also identifies areas with minimal flood hazard, including those with less than 0.2 percent annual chance of flooding, and unknown flood hazard, including areas FEMA had not assessed for flood hazards. In 2018, the Technical Mapping Advisory Council noted that FEMA has produced modernized data (i.e., digital maps) for areas of the United States where 98 percent of the population resides, but has not determined the flood hazard for 40 percent of streams. In general, flood hazards are based on existing conditions in the watershed and floodplains. However, in certain cases, FEMA may include flood hazard information that is based on future conditions, according to FEMA regulations. NOAA provides estimates of hurricane storm surge using a model called Sea, Lake and Overland Surges from Hurricanes. Estimates are available for eastern U.S. coastal areas from Texas through Maine and other areas affected by storm surge, including Hawaii, Puerto Rico, and the U.S. Virgin Islands. As of June 2019, NOAA had not modeled storm surge for the West Coast of the United States or other Pacific islands. The model takes into account a specific locale’s shoreline, incorporating bay and river configurations, water depths, bridges, roads, levees, and other physical features. It estimates the maximum extent of storm surge at high tide by modeling hypothetical hurricanes under different storm conditions, such as landfall location, storm trajectory, and forward speed. NOAA models storm surge from Category 1 through Category 5 hurricanes for the Atlantic coast south of the North Carolina–Virginia border, the Gulf of Mexico, Puerto Rico, and the U.S. Virgin Islands and Category 1 through Category 4 hurricanes for the Atlantic coast north of the North Carolina–Virginia border and Hawaii. According to NOAA’s website, the model is to be used for educational purposes and awareness of the storm surge hazard at a city or community level. In accordance with federal law, the model is also used for other purposes, such as hurricane evacuation studies. According to NOAA’s website, the agency updates the model for portions of the shoreline each year to account for, among other changes, new data and the addition of flood protection devices, such as levees. The model does not account for future conditions such as erosion, subsidence (i.e., the sinking of an area of land), construction, or sea level rise. The U.S. Forest Service maps wildfire hazard potential based on landscape conditions and other observations. According to the U.S. Forest Service, the primary intended use of the wildfire hazard potential map is to identify priority areas for hazardous fuels treatments from a broad, national- to regional-scale perspective. The U.S. Forest Service maps an index of wildfire hazard potential for the contiguous United States, based on, among other factors, annual burn probabilities and potential intensity of large fires. The U.S. Forest Service categorizes the wildfire hazard potential index into five classes of very low, low, moderate, high, and very high. For example, the U.S. Forest Service designates as “high” those areas with wildfire hazard potential index from the 85th to the 95th percentile, and “very high” above the 95th percentile. The U.S. Forest Service also categorizes areas as nonburnable (including agricultural and developed lands) and water. According to the U.S. Forest Service, areas with higher values of wildfire hazard potential represent vegetation that is more likely to burn with high intensity under certain weather conditions. However, areas with moderate, low, and very low wildfire hazard potential may still experience wildfire, particularly if they are near areas that have higher wildfire hazard potential. Wildfire hazard potential is not a forecast or wildfire outlook for any particular season as it does not include any information on current or forecasted weather or fuel moisture conditions. NOAA models the extent of inundations from various heights of sea level rise (up to 10 feet above average high tides) for the contiguous United States, Hawaii, the Pacific islands, Puerto Rico, and the U.S. Virgin Islands and provides the results in a web mapping tool called the Sea Level Rise Viewer. NOAA’s guidance on the Sea Level Rise Viewer states that data are not available for Alaska. The uses of the sea level rise data include planning and education but not site-specific analysis, according to a NOAA document. NOAA labels areas as not mapped if elevation data of sufficient quality for the areas are not available. NOAA does not model natural processes, such as erosion, subsidence, or future construction, or forecast how much sea level is likely to rise in a given area. Rather, for various heights of local sea level rise, NOAA determines extent of inundation based on the elevation of an area and the potential for water to flow between areas. Enterprise risk management is a tool that allows agencies to assess threats and opportunities that could affect the achievement of their goals. In a December 2016 report, we updated our 2005 risk management framework to reflect changes to the Office of Management and Budget’s Circular A-123, which calls for agencies to implement enterprise risk management. We also incorporated recent federal experience and identified essential elements of federal enterprise risk management. Our December 2016 report states that beyond traditional internal controls, enterprise risk management promotes risk management by considering the effect of risk across the entire organization and how it may interact with other identified risks. Additionally, it addresses other topics, such as strategy determination, governance, communicating with stakeholders, and measuring performance. The principles of enterprise risk management apply at all levels of the organization and across all functions, such as those related to managing risk to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites. The six essential elements of enterprise risk management that we identified in our December 2016 report are as follows: 1. Align risk management process with goals and objectives. Ensure that the process maximizes the achievement of agency mission and results. Agency leaders examine strategic objectives by regularly considering how risks could affect the agency’s ability to achieve its mission. 2. Identify risks. Assemble a comprehensive list of risks—both threats and opportunities—that could affect the agency’s ability to achieve its goals and objectives. 3. Assess risks. Examine risks, considering both the likelihood of the risk and the impact of the risk to help prioritize risk response. 4. Respond to the risks. Select risk treatment response (based on risk appetite), including acceptance, avoidance, reduction, sharing, or transfer. 5. Monitor risks. Monitor how risks are changing and whether responses are successful. 6. Communicate and report on risks. Communicate risks with stakeholders and report on the status of addressing the risks. Available federal data on flooding, storm surge, wildfires, and sea level rise suggest that about 60 percent (945 of 1,571) of all nonfederal NPL sites are located in areas that may be impacted by one or more of these potential climate change effects. These data, however, may not fully account for the number of nonfederal NPL sites that may be in such areas because (1) federal data are generally based on current or past conditions; (2) data are not available for some areas; and (3) the NCA has reported that climate change may exacerbate flooding, storm surge, and wildfires in certain regions of the United States. In addition, EPA does not have quality information on the boundaries of nonfederal NPL sites, which could affect its ability to identify the number of sites that may be impacted by one or more of these potential climate change effects. Available federal data suggest that 945 of 1,571 nonfederal NPL sites, or about 60 percent, are located in areas that may be impacted by selected climate change effects—that is, 0.2 percent or higher annual chance of flooding or other flood hazards, storm surge from Category 4 or 5 hurricanes, high and very high wildfire hazard potential, and sea level rise of up to 3 feet. The locations of these sites are shown in figure 3; the full results of our analysis and additional information on these sites is available in the interactive map and downloadable data file, which can be viewed at https://www.gao.gov/products/GAO-20-73. Our analysis, however, may not fully account for the number of nonfederal NPL sites that may be impacted by the effects of climate change for various reasons. First, we represented the areas of nonfederal NPL sites based on a 0.2-mile radius around their primary geographic coordinates, which may not accurately reflect their area (i.e., they may be larger or smaller). We did not analyze site-specific information for these nonfederal NPL sites, including the extent of contamination and location of remedies. Such site-specific analyses would be needed to determine whether there is a risk to human health and the environment at nonfederal NPL sites as a result of these potential climate change effects. Further, according to the NCA, EPA documents, and interviews with EPA officials, there may be other climate change effects that could impact nonfederal NPL sites, such as potential increases in salt water intrusion (the movement of saline water into freshwater aquifers), drought, precipitation, hurricane winds, and average and extreme temperatures; we did not analyze these effects because we did not identify relevant national-level federal data sets. We identified 783 nonfederal NPL sites—approximately 50 percent—in areas that FEMA had identified as having 0.2 percent or higher annual chance of flooding, which FEMA considers moderate flood hazard, or other flood hazards, as of October 2018. Of these 783 sites, our analysis shows that 713—approximately 45 percent of all sites—are currently located in areas with 1 percent or higher annual chance of flooding, FEMA’s highest flood hazard category. We provide information on the number of sites in areas with moderate or other flood hazards because, according to the NCA, heavy rainfall is increasing in intensity and frequency across the United States and is expected to continue to increase, which may lead to an increase in flooding in the future. The full results of our analysis—which include information on the sites in areas that may have 1 percent or higher annual chance of flooding, 0.2 percent or higher annual chance of flooding or other identified flood hazards, unknown flood hazard or no data, and minimal flood hazard—are available in our interactive map, which can be viewed here. For example, there are a number of nonfederal NPL sites in EPA Region 7, where states experienced record flooding in early 2019. Specifically, as seen in figure 4, there are 51 sites that are located in areas with 0.2 percent or higher annual chance of flooding or other identified flood hazards, of which 42 are located in areas with 1 percent or higher annual chance of flooding. Nationwide, the number of nonfederal NPL sites in areas that may be impacted by flooding currently may be higher than 783. Specifically, 217 nonfederal NPL sites are located in areas that FEMA has not assessed for flood hazards or that we did not analyze because the data were not available in a form we could use with our mapping software. We identified 187 nonfederal NPL sites—12 percent—in areas that may be inundated by storm surge corresponding to Category 4 or 5 hurricanes, the highest possible category, based on NOAA’s storm surge model as of November 2018. Of these sites, 102 are located in areas that may be inundated by a storm surge corresponding to Category 1 hurricanes. We analyzed areas that may be inundated by a storm surge corresponding to the highest possible category because, according to the NCA, a projected increase in the intensity of hurricanes in the North Atlantic could increase the probability of extreme flooding because of storm surge along most of the Atlantic and Gulf Coast states, beyond what would be projected based solely on relative sea level rise. However, the NCA stated that there is uncertainty in the projected increase in frequency or intensity of Atlantic hurricanes, and other factors may affect the potential for flooding because of storm surge, such as changes in overall storm frequency or tracks. The full results of our analysis, which include information on the number of sites in areas that may be inundated by storm surge from Category 1 and from Category 4 or 5 hurricanes, are available in our interactive map, which can be viewed here. In EPA Regions 2 and 3, where states experienced damage from two major hurricanes in 2017, there are 87 nonfederal NPL sites located within areas that may be inundated by storm surge from Category 4 or 5 hurricanes. Figure 5 shows these 87 sites, of which 54 sites may be inundated by storm surge from Category 1 hurricanes. Nationwide, the number of nonfederal NPL sites in areas that may be impacted by storm surge may be higher than 187 because NOAA has not modeled areas along the West Coast and Pacific islands other than Hawaii. Further, our analysis did not include other potential impacts from hurricanes, such as rainfall. Figure 6 shows an example of the impact of rainfall caused by a hurricane at the American Cyanamid NPL site. We identified 234 nonfederal NPL sites—15 percent—located in areas that have high or very high wildfire hazard potential—those more likely to burn with a higher intensity, based on a U.S. Forest Service model as of July 2018. For this analysis, we combined the high and very high wildfire hazard potential categories; we did not identify the number of sites in each of these categories separately. We did not analyze areas that currently have moderate or lower wildfire hazard potential because those with moderate or lower wildfire hazard potential are less likely to experience high-intensity wildfire and the extent to which wildfire hazard potential may change in the future is unknown. The full results of our analysis on the number of sites in areas with high or very high wildfire hazard potential are available in our interactive map, which can be viewed here. As seen in figure 7, there are 22 nonfederal NPL sites in areas with high or very high wildfire hazard potential in EPA Region 9, a region that experienced wildfires in 2018, including the highly destructive Carr Fire. Nationwide, the number of nonfederal NPL sites in areas that currently have high wildfire hazard potential may be higher than 234 because wildfire hazard data are only available for the contiguous United States (i.e., there are no data for Alaska, Hawaii and other Pacific islands, Puerto Rico, and the U.S. Virgin Islands). According to the NCA, the incidence of large forest fires in the western United States and Alaska has increased since the early 1980s and is projected to further increase in those regions as the climate changes. However, the NCA noted that analyses regarding the effect of climate change on the incidence of wildfire in other parts of the United States are not readily available, so it is unknown how climate change will affect the number of nonfederal NPL sites in areas rated with high or very high wildfire hazard potential nationwide. As figure 8 shows, wildfires can pose risks at nonfederal NPL sites, such as the Iron Mountain Mine site near Redding, California. We identified 110 nonfederal NPL sites—7 percent—located in areas that would be inundated by a sea level rise of 3 feet, based on our analysis of EPA and NOAA data as of March 2019 and September 2018, respectively. Our analysis shows that if sea level in these areas rose by 1 foot, 97 sites would be inundated. If sea level in these areas rose by 8 feet, 158 sites would be inundated. We also identified 84 nonfederal NPL sites that are located in areas that may already be inundated at high tide. We provide the number of sites in areas that may be impacted by these sea level rise heights because, according to the NCA, global average sea levels are very likely to continue to rise by at least several inches in the next 15 years and by 1.0 to 4.3 feet by 2100. Further, the NCA states that a rise of as much as 8 feet by 2100 cannot be ruled out. The full results of our analysis, which include information on the number of sites in areas that may already be inundated at high tide and that would be inundated if sea level rose by 1 foot, 3 feet, and 8 feet, are available in our interactive map, which can be viewed here. There are 23 nonfederal NPL sites located within areas that may be impacted if sea level rose by up to 3 feet in EPA Region 6, a region that has experienced land loss because of sea level rise and coastal flooding, according to the NCA. In addition, as seen in figure 9, 16—or 70 percent—of these sites may already be inundated at high tide. Nationally, the number of nonfederal NPL sites that may be inundated by various heights of sea level rise will vary from the results of our analysis because different parts of the United States may experience higher or lower sea level rise than the global average. For example, the NCA states that sea level rise will be higher than the global average on the East and Gulf Coasts of the United States and lower than the global average in most of the Pacific Northwest and in Alaska. As can be seen in figure 10, sea level rise and other coastal hazards may impact nonfederal NPL sites, such as the one in the San Jacinto River Waste Pits site in Texas, parts of which are already under water. EPA does not have quality information on the boundaries of nonfederal NPL sites, which could affect its ability to identify the number of sites that may be impacted by one or more of these potential climate change effects. According to EPA officials, EPA has not validated data on site boundaries and EPA’s regional offices do not use a consistent geographic standard, which makes it difficult to produce a national data set. In general, EPA officials told us that information on the boundaries of NPL sites has not been a focus at a national level and is not yet subject to quality standards. For example, EPA officials told us that boundary information for each NPL site represents the remedial project manager’s professional judgment and remedial project managers may determine and record the boundaries of sites differently. EPA has taken some initial actions to improve the quality of information on the boundaries of nonfederal NPL sites. In November 2017, the Office of Superfund Remediation and Technology Innovation issued a directive to all regional Superfund division directors recommending national standards for collecting and maintaining geographic information, including site boundaries. EPA’s 2017 directive notes that using national standards to collect geographic information, including site boundaries, promotes EPA’s reporting and analytical efforts to support program implementation and evaluation. In addition, in May 2018, EPA’s Office of Land and Emergency Management developed technical guidance for all its regions and programs for collecting, documenting, and managing geographic information on Superfund sites, including their boundaries. EPA officials told us that in 2019 and 2020, the agency plans to move toward recording site boundaries in a consistent format across regions and instituting procedures to validate and update them at least annually. However, EPA officials told us that there is no schedule in place for completing this effort and they are uncertain when they will complete it because of competing priorities. By developing a schedule for completing the standardization and improvement of the quality of the information on the boundaries of nonfederal NPL sites, EPA could more reasonably ensure that it would have quality information with which to fully identify nonfederal NPL sites that are located in areas that may be impacted by climate change effects. EPA’s actions to manage risks from the potential impacts of climate change effects align with three of the six essential elements of enterprise risk management. Specifically, for the six essential elements, EPA’s actions do not align with one essential element, aligning its enterprise risk management process with goals and objectives; partially align with two essential elements, assessing risks and responding to risks; and align with three essential elements, identifying risks, monitoring risks, and communicating about and reporting on risks. Table 1 shows the alignment of EPA’s actions with the essential elements of enterprise risk management. This essential element calls for agencies to align their risk management processes with the goals and objectives of the agency, but EPA has not taken action to clearly align its process for managing risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites with agency-wide goals and objectives. For example, the 2018 to 2022 EPA strategic plan does not include goals and objectives related to climate change or discuss strategies for addressing the impacts of climate change effects. Moreover, neither the fiscal years 2018 to 2019 nor fiscal years 2020 to 2021 national program manager guidance for EPA’s Office of Land and Emergency Management mentions climate change among its goals and priorities. In contrast to the current strategic plan, the 2014 to 2018 EPA strategic plan included addressing climate change as one of four strategic goals and specifically discussed climate change as an external factor or emerging issue in the context of planned, current, and completed cleanups, including at nonfederal NPL sites. In addition, the fiscal years 2016 to 2017 national program manager guidance for the office that oversees the Superfund program listed climate change adaptation as one of four national areas of focus for the office. According to an EPA official, when the 2018 to 2022 strategic plan was drafted, senior agency officials were not aware of the potential risks to the Superfund program mission from the impacts of climate change effects. According to this official, senior EPA officials have expressed support for certain activities related to climate change, such as the work of the Cross- EPA Work Group on Climate Adaptation, but have not issued related documents or policy statements. Without clarifying how the agency’s ongoing actions to manage these risks at nonfederal NPL sites align with current agency goals and objectives, EPA will not have reasonable assurance that senior officials will take an active role in supporting these actions, which would help EPA achieve its mission of protecting human health and the environment. EPA’s actions to identify risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites align with this essential element of enterprise risk management. Specifically, EPA identified climate change effects that may impact nonfederal NPL sites—and pose risks to human health and the environment—in studies and climate change adaptation and implementation plans. For example, in a 2012 study of adaptation of Superfund remediation to climate change, EPA identified eight climate change effects that may impact certain NPL site remedies: flooding, sea level rise, extreme storms, large snowfall, wildfires, drought, extreme heat, and landslides. In 2014, EPA issued an agency-wide climate change adaptation plan, which identified climate change effects that may impact NPL sites. The same year, EPA issued a climate change adaptation implementation plan for the office that oversees the Superfund program that identified nine climate change effects that may impact NPL sites. Each of the 10 EPA regional offices identified relevant regional climate change effects in their 2014 climate change adaptation implementation plans. For example, the Region 3 plan states that increased flooding and sea level rise may increase risks of releases of contaminants, salt water intrusion may impact the performance of remedies, and increased temperatures may impact vegetation that prevents erosion. Additionally, five regional offices have conducted or are conducting additional screening-level studies to identify which climate change effects, if any, may impact each of the NPL sites in these regions. For example, Region 10 conducted a study in 2015 that identified, among other effects, sea level rise and wildfires as potential climate change effects that may impact NPL sites in the region. EPA’s actions to assess risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites partially align with this essential element. In a 2012 study of adaptation of Superfund remediation to climate change, EPA assessed the impacts of eight climate change effects on certain remedies to determine the risk they presented to the agency’s mission to protect human health and the environment. EPA issued climate change adaptation implementation plans for the office that oversees the Superfund program and all regions, as described above, which assessed potential impacts of climate change effects. In addition, five EPA regional offices assessed or are assessing potential impacts of climate change effects on NPL sites in their regions as a whole, and one of these regions assessed both the impacts and likelihood of climate change effects, consistent with this essential element. Specifically, Region 4 identified the sites most likely to face major climate change risks and then examined these sites in greater detail. Additionally, Region 3 has developed a mapping tool on climate change vulnerability that provides site-level assessments of sea level rise, among other potential impacts. EPA provides training and direction to remedial project managers—the lead EPA officials at nonfederal NPL sites—on conducting site-level risk assessments that incorporate information on potential impacts of climate change effects. Since 2014, EPA has offered optional training to remedial project managers and others on integrating climate change into the Superfund cleanup process. From 2013 through 2015, EPA issued fact sheets as guidance for assessing the potential impacts of climate change effects for three types of remedies. According to EPA officials, these fact sheets constitute the direction that EPA provides to remedial project managers on assessing risks from climate change effects. EPA plans to update these fact sheets in 2019 and is also in the process of developing a compilation of resources for assessing potential flood risks in coastal areas to inform cleanup and reuse decision-making, according to an EPA official. In addition, EPA provides resources on climate change on the Superfund program website, such as links to tools and data on drought and coastal flooding. EPA also offers technical assistance on incorporating climate change information into risk assessments to remedial project managers through groups such as the Contaminated Sediments Technical Advisory Group and the Cross-EPA Work Group on Climate Adaptation. EPA officials in four regions provided us with site-specific examples of how they used climate change information to assess risks from the potential impacts of climate change effects, but officials from other regions stated that they have not always integrated climate change information into their risk assessments. For example, according to a record of decision for the site, EPA Region 2 incorporated the potential for increased storm flow intensities into the model of the Passaic River used in the remedial investigation and feasibility study at the Diamond Alkali site in Newark, New Jersey. Conversely, officials in six regions told us that they have not used climate change projections for flooding or rainfall in site-level risk assessments. In addition, officials in Region 6 told us that they do not incorporate potential impacts of climate change effects or changes in the frequency of natural disasters into their assessments. EPA officials have not consistently incorporated climate change information into their assessment of site-level risks because they do not always have the climate data they need to do so, according to our review of EPA documents and interviews with EPA officials and stakeholders. For example, officials in three regions told us that they have not used rainfall or flood projections because the data are not available or they were unsure which data to use. In addition, in the record of decision for the Diamond Alkali site in New Jersey, Region 2 officials stated that they did not integrate sea level rise information into their storm flow modeling for the Passaic River at the site because of the uncertainty in expected future sea level rise values, especially at the regional and local levels. We reported on similar challenges with climate data in our 2015 report on climate information, which found that existing federal efforts do not fully meet the climate information needs of federal, state, local, and private sector decision makers, and we made a related recommendation in that report. Further, current EPA practice for assessing risks at NPL sites does not always include consideration of climate change, according to agency documents we reviewed and officials from three regions and a stakeholder we interviewed. EPA’s climate change adaptation plan noted that EPA and its partners will need to alter their standard practices—such as their standard methods for estimating the frequency of floods or runoff of pollutants into rivers—to account for a continuously changing climate. The Region 4 climate change adaptation implementation plan, for instance, noted that preliminary assessments and site investigations are typically based on historic information, not future projections and therefore may not fully address risks. Officials in two regions told us that they do not have direction on how to alter their practices to account for climate change. For example, officials in Region 2 said they do not have instructions that identify a particular set of expectations, data, or maps that they should use when considering future risks from flooding. Officials in Region 5 told us that they do not have any formal direction on how to address risks from climate change and are waiting for EPA headquarters to provide information on how to do so. According to EPA documents and a headquarters official, EPA believes that its existing direction, including general guidance on conducting risk assessments and the fact sheets for assessing potential impacts of climate change effects for three types of remedies, discussed above, provide a robust structure for considering such impacts. However, without providing direction to remedial project managers on how to integrate information on the potential impacts of climate change effects into site- level risk assessments at nonfederal NPL sites across all regions and types of remedies, EPA cannot ensure that remedies will protect human health and the environment in the long term. EPA’s actions to respond to risks that potential impacts of climate change effects may pose to human health and the environment at nonfederal NPL sites partially align with this essential element. In two national studies EPA conducted in 2012 and 2017, EPA examined potential impacts of some climate change effects on selected remedies at NPL sites, including nonfederal NPL sites, and generally found that it has taken actions to respond to risks through its existing cleanup processes. In 2012, as noted above, EPA studied the vulnerability of selected remedies to some climate change effects and found that existing processes—such as EPA’s Five-Year Review and operation and maintenance—could adequately address the potential impacts of climate change effects. In addition, EPA studied the impacts of three hurricanes in 2017 on sites with selected remedies in place, including nonfederal NPL sites, and found that the agency has generally taken resiliency measures to respond to risks at these sites. EPA also provided guidance and training to remedial project managers on responding to risks to human health and the environment from the potential impacts of climate change effects and recently added requirements for certain potential site contractors to describe their capacity to respond to such risks. EPA provided guidance in its fact sheets on integrating climate change information into risk response decisions at nonfederal NPL sites and optional training on integrating climate change into the Superfund cleanup process. In addition, EPA provided relevant information and resources for EPA officials on resiliency measures on the agency website. In 2016, EPA issued performance work statements to potential contractors for environmental services and operations and for remediation environmental services that required contractors to describe their ability to conduct climate change vulnerability analyses and adaptation, as needed, to ensure the resiliency of remedies to climate change impacts. According to an EPA headquarters official, EPA is currently working on developing technical guidance on how remedial project managers can integrate requests for climate change– related analysis into their task orders for contractors. With respect to site-level responses, EPA officials from three regions provided us with examples of site decision documents that described how climate change information will be incorporated into remedy selection and design. For example, the record of decision for the Portland Harbor site in Oregon states that a containment cap will be constructed to withstand more frequent floods with higher peak flows more common with climate change. Officials from Region 3 told us that they take into account a number of factors, including climate change impacts, if any, when they design and select site remedies. However, according to our interviews with regional officials, they have not consistently integrated climate change information into remedy selection and design. For example, officials from two regions stated that they are not aware of any remedial project managers in their regions who are taking action at nonfederal NPL sites to respond to climate change or consider future conditions. EPA officials have not consistently taken the potential impacts of climate change effects into account in site-level risk response decision making because they do not always have sufficient direction to do so, according to our interviews with EPA officials. EPA officials from three regions told us that they are unsure how to translate data on potential impacts of climate change effects into the design of remedies. For example, officials from Region 10 told us that EPA does not have direction for remedial project managers on how to integrate response to climate change impacts into remedial design. These officials noted that it is up to remedial project managers to be aware of this issue and it is done on an ad hoc basis. Further, EPA headquarters officials who review proposed remedies told us that additional guidance from EPA on managing the risks from potential impacts of climate change effects would be useful. According to EPA documents and another EPA headquarters official, EPA has determined that existing direction—guidance and processes— for risk response provide a robust structure to integrate climate change information into remedy selection and design. However, without providing direction for remedial project managers on how to integrate information on potential impacts of climate change effects into site-level risk response decision making at nonfederal NPL sites, EPA cannot ensure that remedies will protect human health and the environment in the long term. EPA’s actions to monitor risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites through its Five-Year Review process align with this essential element. In 2016, EPA introduced a new recommended template for the Five-Year Review that includes a section for officials to document their consideration of whether any newly available information related to climate change may call into question a remedy’s protectiveness. Officials in three regions told us they use the Five-Year Review process to identify and evaluate newly available information on climate change effects that may impact nonfederal NPL sites. For example, in the 2014 Five-Year Review report for the Publicker Industries site in Pennsylvania, Region 3 considered newly available information on projected sea level rise in the region to determine if those projections called into question the protectiveness of the existing remedies at the site. Officials in that region told us that they rely on their biological and technical assistance group to identify any new relevant climate change data to incorporate into their Five-Year Reviews. Region 7 officials also told us that they assess any potential changes in future conditions, especially flooding, during the Five-Year Review process. Officials from two other regions told us that they monitor changes in site conditions that may be related to climate change during the Five-Year Review process. For example, Region 2 officials developed additional guidance to help remedial project managers and site project teams consider changes in site conditions related to climate change in the Five-Year Review process. Region 6 officials told us that during the Five-Year Review process, they take into account any current flood hazard information from FEMA as well as current sea levels, but they do not monitor projections about sea level rise. EPA’s actions to communicate about and report on risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites align with this essential element. For example, as described above, EPA reported on the potential impacts of climate change effects—which may pose risks to human health and the environment—on NPL sites in its 2014 national climate change adaptation plan and the climate change adaptation implementation plans for the office that oversees the Superfund program and all regions. In addition, publicly available site-level documents, such as the records of decision described above, may include information on risks from climate change and EPA’s actions to manage these risks. EPA officials may also communicate this information in response to questions from the public. EPA officials from four regions told us that they have not received many direct questions on risks from climate change from the public. However, members of the public can comment on climate change risks through EPA’s existing public engagement mechanisms, and some have done so. For example, EPA officials in Region 7 received questions on the draft record of decision for the West Lake Landfill site in Missouri during the public comment period and responded to those questions in the final version of the document, describing how they addressed risks of increased flooding from climate change in the remedy selection processes. EPA has also communicated with stakeholders and the public on risks to human health and the environment from the potential impacts of climate change effects in other ways. For example, officials from Region 10 convened a workshop in 2017 to discuss climate change impacts on sediment cleanup and upland source control for the Lower Duwamish Waterway site in Washington with other federal agencies, state and local officials, universities, companies, and community groups. In addition, EPA provides an online mapping tool that can help members of the public identify sites located in areas that would be impacted by up to 6 feet of sea level rise or in flood hazard areas as determined by FEMA. EPA recognizes institutional, resource, and technical challenges in managing risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites, according to agency and other documents that we reviewed and EPA officials and stakeholders we interviewed. According to agency and other documents we reviewed and officials and stakeholders that we interviewed, EPA faces institutional challenges in managing risks to human health and the environment from the potential impacts of climate change effects. As discussed above, officials from three regions told us that they do not have the direction they need to manage these risks. For example, EPA officials in Region 2 told us that during Five-Year Reviews, engineers may analyze several different maps on flooding potential and must use their professional judgment to determine how resilient to design the remedy, because there is no standard guidance on how to do so. Further, EPA officials in two regions and stakeholders we interviewed stated that it may not be clear whether EPA could require PRPs to consider climate change impacts in the cleanup process. However, according to EPA headquarters officials, considering climate change is consistent with the National Contingency Plan and the CERCLA criterion that requires officials to consider the long- term effectiveness of remedies when evaluating cleanup alternatives. Another institutional challenge that EPA faces is that its ability to manage these risks may depend on actions of other entities that are outside of EPA’s control, according to EPA documents we reviewed and EPA officials we interviewed. For example, EPA officials from Region 1 told us that they are not certain whether a hurricane barrier built by the U.S. Army Corps of Engineers that protects the New Bedford Harbor site in Massachusetts is designed to withstand future storms. Managing risks may also require internal coordination within EPA, which presents another challenge. For example, an EPA headquarters official told us that it can be challenging for regional Superfund program staff to connect with EPA experts on climate change, who may be in different program offices. In April 2019, EPA restructured its regional offices, consolidating cross- cutting issue areas in the immediate office of each Regional Administrator and Deputy Regional Administrator. Although it is too early to evaluate the effect of this restructuring, EPA headquarters officials told us that the restructuring may help address this challenge. Furthermore, EPA officials from three regions told us that they face challenges related to the sensitive nature of climate change. For example, officials in Region 6 told us that when they engaged with the local community during the decision making process for the San Jacinto River Waste Pits site in Texas, they avoided using the term climate change because of concerns that the charged term would alienate some community members. Documents from four EPA regions and headquarters officials and officials from three regions we interviewed stated that insufficient or changing resources—specifically funding and staffing—makes managing risks to human health and the environment from the potential impacts of climate change effects challenging for EPA. For example, according to two regional climate change adaptation implementation plans and EPA officials, assessing these risks may require more resources than assessing risks based on current or past conditions. In addition, designing or modifying existing remedies to respond to these risks could increase costs, according to EPA documents we reviewed and EPA officials we interviewed. EPA officials from three regions told us that staffing constraints can make it difficult to manage risks. For example, EPA officials from Region 9 told us that the need for remedial project managers to respond to other emergencies, such as overseeing hazardous materials removal after fires, means that they have less time to oversee cleanup of nonfederal NPL sites. Officials from Region 10 told us that they had a climate change advisor who helped integrate climate change into all aspects of the region’s work, but that person retired, and the region was unable to fill the position because of resource constraints. As noted above, according to an EPA headquarters official, EPA’s recent restructuring of its regional offices may help address this challenge. EPA faces technical challenges in managing risks to human health and the environment from the potential impacts of climate change effects in terms of available expertise and data, according to documents we reviewed and EPA regional officials we interviewed. In its 2014 agency- wide climate change adaptation plan, EPA reported that site vulnerabilities may be difficult to assess because of limited scientific understanding. EPA officials told us that they need additional expertise and training to better manage risks. For example, an EPA official in Region 2 told us that it would be useful to have training on assessing risks for projects located in floodplains. As noted above, EPA has developed training for officials on managing risks from climate change, such as a course on building resilient Superfund remedies that EPA offered at the annual National Association of Remedial Project Managers meeting in August 2019. The course’s focus is to help remedial project managers incorporate consideration of adaptation and build resilience into Superfund remedies at extreme weather event–impacted sites, according to the course agenda. According to EPA documents and EPA officials from two regions, appropriate climate change data may not be available to inform assessments that help manage risk. For example, the Region 4 study of the vulnerability of NPL sites stated that climate model projections of temperature and precipitation patterns are not available at a spatial resolution that is useful for assessing vulnerabilities at the site level. In Region 6, officials told us that when the U.S. Army Corps of Engineers modeled flooding for the San Jacinto River Waste Pits site in Texas, it had to rely on past flooding data because the only information available was on historical storms. In addition, the level of uncertainty inherent in climate change data may make it challenging for EPA to incorporate that information into risk management processes, according to agency documents we reviewed and some agency officials we interviewed. As noted above, we made recommendations to address similar challenges with climate data in a prior report. Climate change may result in more frequent or intense extreme events, such as flooding, storm surge, and wildfires, among other effects, which could damage remedies at nonfederal NPL sites and lead to releases of contaminants that could pose risks to human health and the environment. Our analysis of EPA, FEMA, NOAA, and U.S. Forest Service data has shown that more than half of nonfederal NPL sites—based on a point coordinate with a 0.2-mile radius as a proxy for the site boundaries—are located in areas that may be impacted by selected climate change effects. To help ensure the long-term protectiveness of remedies, it is important for EPA to understand how climate change effects may impact nonfederal NPL sites, and the agency has taken steps to do this. However, EPA does not have quality information on the precise boundaries of nonfederal NPL sites, which could make it difficult to determine the nonfederal sites located in areas that may be impacted by climate change effects. The agency has taken initial steps to develop this information but does not have a schedule in place for completing this effort. EPA has taken actions to manage risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites. These actions align with three of the six essential elements of enterprise risk management. However, EPA has not clarified how its actions to manage risks from these effects at nonfederal NPL sites align with current agency goals and objectives, which could limit its senior officials’ ability to manage these risks. Further, EPA officials do not always have direction to ensure that they consistently integrate climate change information into site-level risk assessments and risk response decisions, according to EPA documents and officials. Without providing such direction for remedial project managers, EPA cannot ensure that remedies at nonfederal NPL sites will protect human health and the environment in the long term. We are making the following four recommendations to EPA: The Director of the Office of Superfund Remediation and Technology Innovation should establish a schedule for standardizing and improving information on the boundaries of nonfederal NPL sites. (Recommendation 1) The Administrator of EPA should clarify how EPA’s actions to manage risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites align with the agency’s current goals and objectives. (Recommendation 2) The Director of the Office of Superfund Remediation and Technology Innovation should provide direction on how to integrate information on the potential impacts of climate change effects into risk assessments at nonfederal NPL sites. (Recommendation 3) The Director of the Office of Superfund Remediation and Technology Innovation should provide direction on how to integrate information on the potential impacts of climate change effects into risk response decisions at nonfederal NPL sites. (Recommendation 4) We provided a draft of this report to EPA for its review and comments. In its comments, reproduced in appendix II, EPA stated that it recognizes the importance of ensuring Superfund sites cleanups are resilient in the face of existing risks and extreme weather events. EPA added that it has taken actions to include vulnerability analyses and adaptation planning in its Superfund activities. We acknowledge that EPA has taken some action to manage risks. However, EPA has not clarified how its risk-related actions align with agency goals and objectives. Further, it has not provided direction to ensure that officials consistently integrate climate change information into site-level risk assessments and risk response decisions. Regarding our recommendations, EPA agreed with one and disagreed with the other three. We continue to believe that all recommendations are warranted. In response to our recommendation that the Director of the Office of Superfund Remediation and Technology Innovation establish a schedule for standardizing and improving information on the boundaries of nonfederal NPL sites, EPA noted that it agrees with our finding and acknowledges a lack of consistent standards to identify site boundaries at the national level. According to EPA, it has taken initial steps to develop an approach to standardize and improve information on nonfederal NPL site boundaries. EPA stated that it expects to establish a schedule for this effort by the second quarter of fiscal year 2020, with the aim to have collected an initial set of site boundaries for all NPL sites by the fourth quarter of fiscal year 2021. In response to our recommendation that EPA clarify how its actions to manage risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites align with the agency’s current goals and objectives, EPA said that it believes managing risks from exposure to contaminants in the environment is integral to EPA’s current strategic goal 1.3, Revitalize Land and Prevent Contamination. We agree that protectiveness is a key part of strategic objective 1.3. However, this strategic objective does not include any measures related to climate change or discuss strategies for addressing the impacts of climate change effects. An essential element of enterprise risk management is to align risk management processes with goals and objectives. Consequently, we believe that our recommendation is still warranted. In response to our recommendations that the Director of the Office of Superfund Remediation and Technology Innovation provide direction on how to integrate information on the potential impacts of climate change effects into risk assessments and risk response decisions at nonfederal NPL sites, EPA said that it strongly believes the Superfund program’s existing processes and resources adequately ensure that risks and any effects of severe weather events are woven into risk assessments and risk response decisions at nonfederal NPL sites. However, as we noted in our report, EPA’s current direction does not address all types of cleanup actions or climate change effects. Further, EPA officials from some regions told us that current EPA practice for assessing risks at NPL sites does not always include consideration of climate change and that they have not consistently integrated climate change information into site- specific remedy selection and design. EPA noted that it may issue a memorandum to reinforce the tools and resources available to NPL site teams and would determine whether to issue this memorandum by the end of January 2020. EPA also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Administrator of the Environmental Protection Agency, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines (1) what available federal data suggest about the number of nonfederal National Priorities List (NPL) sites that are located in areas that may be impacted by selected climate change effects; (2) the extent to which the Environmental Protection Agency (EPA) has managed risks to human health and the environment from the potential impacts of climate change effects at nonfederal NPL sites; and (3) the challenges, if any, EPA faces in managing these risks. To determine what available federal data suggest about the number of nonfederal NPL sites that are located in areas that may be impacted by selected climate change effects, we reviewed the Fourth National Climate Assessment (NCA) to identify potential climate change effects. Based on our review of the NCA, we identified the following potential climate change effects: sea level rise, which may lead to increased frequency and extent of extreme flooding from coastal storms; greater frequency and magnitude of drought; increased intensity and frequency of heavy precipitation events, which may lead to increased local flooding; salt water intrusion; increased incidence of large wildfires; increased frequency and intensity of extreme high temperatures and sustained increases in average temperatures; decreased permafrost; and increased intensity—including higher wind speeds and precipitation rates—and frequency of very intense hurricanes and typhoons. We reviewed EPA documents (such as EPA’s climate change adaptation implementation plans) to identify potential climate change effects that may impact nonfederal NPL sites and interviewed EPA officials. Through a review of federal agencies’ documents and databases and interviews with officials about their data and research on these effects, we identified available national federal data sets on three current hazards: flooding, storm surge, and wildfires—which the NCA reports will be exacerbated by climate change—from the Federal Emergency Management Agency (FEMA), the National Oceanic and Atmospheric Administration (NOAA), and the U.S. Forest Service. We also identified data on sea level rise from NOAA. In this report, we refer to (1) flooding, (2) storm surge, (3) wildfires, and (4) sea level rise as potential climate change effects. We used the most recently available data for each of these climate change effects; these data do not provide estimates of the projected changes in the future. To the extent that data were available, we analyzed a range of these potential climate change effects. For example, we used the maximum extent of storm surge from Category 1 hurricanes as well as Category 4 or 5 hurricanes, the highest possible categories, as modeled by NOAA. We focused on a range because, for three of the four effects, we had data on current hazards, which may become more intense and frequent in the future, according to the NCA. Additionally, the Comprehensive Environmental Response, Compensation, and Liability Act of 1980 (CERCLA) directs EPA to give preference to remedies that would result in the permanent and significant decrease in toxicity, mobility, or volume of the contamination. According to EPA officials, remedies at nonfederal NPL sites may have to be operational indefinitely, during which time the potential effects of climate change may become more extreme. The range of estimates we provide in our report is as follows: For flooding, we used data from FEMA’s National Flood Hazard Layer as of October 2018. FEMA identifies a variety of flood hazards, and for reporting purposes, we grouped flood hazard zones into four categories: (1) 1 percent or higher annual chance of flooding, (2) 0.2 percent or higher annual chance of flooding or other flood hazards, (3) unknown flood hazards, and (4) minimal flood hazard. For storm surge, we used data from NOAA’s model on Sea, Lake and Overland Surges from Hurricanes as of November 2018 for Category 1 and Category 4 or 5 hurricanes. For wildfire, we used data from the U.S. Forest Service’s 2018 wildfire hazard potential map, which the U.S. Forest Service released in July 2018. We used areas with high or very high wildfire hazard potential in our analysis. The U.S. Forest Service based the 2018 map on wildfire likelihood and intensity data from 2016, spatial fuels and vegetation data from 2012, and point locations of past fire occurrence from 1992 to 2013. For sea level rise, we used NOAA data, last updated in September 2018. We downloaded inundation data on 0, 1, 3, and 8 feet of sea level rise and “not mapped” areas. We obtained data from EPA’s Superfund Enterprise Management System on the location and other characteristics of active and deleted nonfederal NPL sites. In our analysis, we used a 0.2-mile radius around the primary geographic coordinate point of each nonfederal NPL site, which may not accurately represent their actual areas because the sites vary in size and shape. The EPA data we used in our analysis on the location of nonfederal NPL sites are current as of March 2019. We also obtained EPA data on contaminants and types of remedies that are current as of fiscal year 2014 to determine the number of contaminants EPA has identified in nonfederal NPL sites. We did not conduct further site-specific analyses, such as those related to the extent of contamination and location of remedies. We reviewed documents from the Agency for Toxic Substances and Disease Registry on the health effects of hazardous substances in nonfederal NPL sites and interviewed officials from that agency. To analyze whether nonfederal NPL sites are located in areas that may be impacted by flooding, we used ArcGIS mapping software to intersect the area of a 0.2-mile radius around the primary coordinate of the sites with the categories we defined from the National Flood Hazard Layer. If a site overlapped with areas in more than one of the four reporting groups, we categorized the site in the group representing the highest flood hazard. For the purposes of our report, we considered the highest flood hazard to be, in descending order, 1 percent or higher annual chance of flooding, other flood hazards (including 0.2 percent or higher annual chance of flooding), unknown flood hazard or no data, and minimal flood hazard. To analyze whether nonfederal NPL sites are located in areas that may be impacted by storm surge, wildfires, and sea level rise, we used MapInfo mapping software to intersect the area of a 0.2-mile radius around the primary coordinates of sites with each of these layers. Overlap indicates that a site is potentially in an area that may be impacted. To assess the reliability of FEMA’s National Flood Hazard Layer, we reviewed FEMA’s methodology, guidelines, and standards; interviewed FEMA officials to assess the timeliness and accuracy of the data as well as any limitations of the data; conducted data testing to check for missing data and inconsistencies; and reviewed internal controls. We also reviewed a prior GAO report on the methodology FEMA uses to map flood hazards. To assess the reliability of NOAA’s data on Sea, Lake and Overland Surges from Hurricanes, we reviewed NOAA’s methodology for developing the model, interviewed NOAA officials to assess the timeliness and accuracy of the data as well as any limitations of the data, and reviewed internal controls. To assess the reliability of the U.S. Forest Service’s wildfire hazard potential data, we reviewed the agency’s documentation of the methodology, interviewed U.S. Forest Service officials to assess the timeliness and accuracy of the data as well as any limitations of the data, and reviewed internal controls. We also reviewed our past reports that cited the 2014 versions of these data. To assess the reliability of NOAA’s data on sea level rise, we reviewed the methodology NOAA used for developing the model, interviewed NOAA officials to assess the timeliness and accuracy of the data as well as any limitations of the data, and reviewed internal controls. To assess the reliability of EPA’s data, we reviewed agency manuals and data dictionaries to understand data elements, interviewed EPA officials to assess the timeliness and accuracy of the data and related internal controls, conducted data testing, discussed inaccuracies with EPA officials; and obtained corrected data. For example, we compared the zip code of each nonfederal NPL site to its coordinate to check the accuracy of site locations. We shared potential errors with EPA officials, who corrected the coordinates of six sites. As a result of the steps described above, we found data from EPA, FEMA, NOAA, and the U.S. Forest Service to be sufficiently reliable for our purposes. To determine the extent to which EPA has managed risks to human health and the environment from the potential impacts of climate change effects on nonfederal NPL sites, we examined relevant provisions in CERCLA, EPA’s implementing regulations, and executive orders. We also reviewed EPA documents, including climate change adaptation and implementation plans; vulnerability studies; training materials; and site- specific documents, our prior work, and relevant documents from other organizations, such as the National Research Council. We identified these documents by conducting a search of (1) websites of relevant agencies and organizations and (2) article databases. We also reviewed documents provided to us by agency officials and stakeholders that we identified as described below. We interviewed EPA officials at headquarters and all regional offices to identify information on agency actions for managing risks. In addition, to obtain their views of EPA’s actions, we interviewed former EPA officials, representatives of two associations representing state officials (the Environmental Council of States and the Association of State and Territorial Solid Waste Management Officials), a professor of environmental law, and a private consultant who has worked on Superfund issues, which we identified in the search described above and recommendations from other interviewees. We generally contacted all stakeholders that we identified who appeared to be currently working on issues related to Superfund and climate change and who agreed to speak with us. We also interviewed stakeholders at the three sites we selected as illustrative examples in order to obtain their views of EPA’s actions. We selected three nonfederal NPL sites as illustrative examples of how EPA has managed risks to human health and the environment from potential impacts of climate change effects and challenges EPA may face in managing these risks. The three sites we selected are the (1) American Cyanamid site in Bridgewater, New Jersey; (2) Iron Mountain Mine site near Redding, California; and (3) San Jacinto River Waste Pits site in Channelview, Texas. To select these sites, we initially identified 43 sites based on information in EPA documents, news articles, and interviews with EPA officials and other stakeholders as described above. We selected relevant sites in three different EPA regions that illustrate a variety of potential climate change effects and that had experienced an extreme weather event in the past 10 years. To gather more in-depth information about these sites, we reviewed EPA and other documents; toured the sites; and interviewed EPA officials and relevant stakeholders at these sites, including state and local officials, representatives of potentially responsible parties, and community organizations. The results from these illustrative examples are not generalizable to nonfederal NPL sites that we did not select. We compared EPA’s actions to manage risks to human health and the environment from the potential impacts of climate change effects with essential elements for managing risk as identified in our prior work on enterprise risk management. These essential elements are as follows: (1) align the risk management process with goals and objectives, (2) identify risks, (3) assess risks, (4) respond to the risks, (5) monitor the risks, and (6) communicate and report on the risks. We assessed information on EPA’s actions to determine the extent to which the agency’s actions aligned with these elements. In assessing EPA’s actions against these essential elements, we used “aligned,” “partially aligned,” or “not aligned” to reflect the extent to which EPA took actions aligned with each essential element. If EPA provided evidence that it had taken major actions in alignment with that essential element, we determined the actions were aligned. If EPA provided evidence that it had taken some actions in alignment with that essential element, we determined the actions were partially aligned. If EPA took only a few or no actions in alignment with that essential element, we determined the actions were not aligned. Two GAO analysts independently reviewed the information on EPA’s actions and then reached consensus on the extent to which EPA’s actions were aligned with each element. To identify the challenges EPA faces in managing these risks, we reviewed EPA documents; our prior work; and relevant documents from other organizations, including the National Research Council, that we obtained as described above. We interviewed EPA officials at headquarters and all regional offices and stakeholders in order to obtain their views on the challenges EPA faces. The views of stakeholders we interviewed are illustrative and not generalizable to all stakeholders. We reviewed the challenges that we identified in these documents and interviews and grouped all the challenges into three categories for reporting purposes: institutional, resource, and technical. Two GAO analysts independently reviewed the information and then reached consensus on the challenges and their grouping in the three categories. We conducted this performance audit from April 2018 to October 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Barbara Patterson (Assistant Director), Ruth Solomon (Analyst in Charge), Breanne Cave, Charles Culverwell, Cindy Gilbert, Richard Johnson, Gwen Kirby, Krista Mantsch, Patricia Moye, Eleni Orphanides, Ernest Powell Jr., Dan Royer, and Kiki Theodoropoulos made key contributions to this report.", "summary": "Administered by EPA, Superfund is the principal federal program for addressing sites containing hazardous substances. EPA lists some of the most seriously contaminated sites—most of which are nonfederal—on the NPL and has recorded over 500 contaminants, including arsenic and lead, at those sites. Climate change may make some natural disasters more frequent or more intense, which may damage NPL sites and potentially release contaminants, according to the Fourth National Climate Assessment. GAO was asked to review issues related to the impact of climate change on nonfederal NPL sites. This report examines, among other objectives, (1) what available federal data suggest about the number of nonfederal NPL sites that are located in areas that may be impacted by selected climate change effects and (2) the extent to which EPA has managed risks to human health and the environment from the potential impacts of climate change effects at such sites. GAO analyzed available federal data; reviewed laws, regulations, and documents; interviewed federal officials and stakeholders; visited three nonfederal NPL sites that experienced natural disasters; and compared EPA actions to manage risk to GAO’s six essential elements of enterprise risk management. Available federal data—from the Environmental Protection Agency (EPA), Federal Emergency Management Agency, National Oceanic and Atmospheric Administration, and U.S. Forest Service—on flooding, storm surge, wildfires, and sea level rise suggest that about 60 percent of all nonfederal National Priorities List (NPL) sites are located in areas that may be impacted by these potential climate change effects. Additional information on these sites can be viewed in an interactive map and downloadable data file, available here (see figure). EPA’s actions to manage risks to human health and the environment from potential impacts of climate change effects at nonfederal NPL sites align with three of the six essential elements of enterprise risk management GAO previously identified, partially align with two essential elements, and do not align with one essential element. For example, EPA has not taken actions consistent with one essential element because it has not aligned its process for managing risks with agency-wide goals and objectives, which do not mention climate change. Without clarifying this alignment, EPA cannot ensure that senior officials will take an active role in strategic planning and accountability for managing these risks. GAO is making four recommendations to EPA, including that it clarify how its actions to manage risks at nonfederal NPL sites from potential impacts of climate change align with current goals and objectives. EPA agreed with one recommendation and disagreed with the other three. GAO continues to believe that all four are warranted.", "document_type": "gao"}
{"report": "DOD acquires new weapon systems for its warfighters through a management process known as the Defense Acquisition System. This system is implemented by two key acquisition policies: DOD Directive 5000.01, which establishes the overarching framework for the Defense Acquisition System; and DOD Instruction 5000.02, which provides detailed procedures for the operation of the Defense Acquisition System and the management of acquisition programs. These policy documents establish the guiding principles for all aspects of the DOD acquisition process. Additionally, each of the military services has its own acquisition policies which incorporate and enhance the DOD acquisition guidance. Figure 2 depicts DOD’s acquisition process beginning with Milestone A in general terms. Several entities in the Office of the Secretary of Defense and the military departments play a role in the oversight of DOD weapon system acquisition programs, including the following: The Under Secretary of Defense for Research and Engineering is responsible for establishing policies on and supervising all aspects of defense research and engineering, technology development, technology transition, prototyping, experimentation, and developmental testing activities and programs, including the allocation of resources for defense research and engineering. DOD’s Reliability and Maintainability Engineering lead reports to this Under Secretary. The Under Secretary of Defense for Acquisition and Sustainment is responsible for establishing policies on and supervising all matters relating to acquisition (including (1) system design, development, and production; and (2) procurement of goods and services) and sustainment (including logistics, maintenance, and materiel readiness). This organization has certain oversight responsibilities for major defense acquisition programs throughout the acquisition process, such as collecting and distributing performance data. The Under Secretary is the Defense Acquisition Executive and serves as the milestone decision authority for certain major defense acquisition programs, meaning the Under Secretary authorizes these programs to proceed through the DOD acquisition process’s major milestones. At the military department level, the service acquisition executive, also known as the component acquisition executive, is a civilian official within a military department who is responsible for all acquisition functions within the department and can serve as the milestone decision authority. Congress has recently devolved much of the decision making authority for major defense acquisition programs from OSD to these service acquisition executives. According to a DOD official the service acquisition executive will normally assign a relevant program manager who will then assign a chief engineer or lead systems engineer and team members with responsibility for the engineering effort of a program, including the reliability engineering effort. The following officials serve as the service acquisition executive for the military departments: the Assistant Secretary of the Air Force (Acquisition, Technology, and Logistics); the Assistant Secretary of the Army (Acquisition, Logistics and Technology); and the Assistant Secretary of the Navy (Research, Development and Acquisition) for both the Navy and the Marine Corps. Major defense acquisition program managers, who can be either civilian or military, are tasked with developing and delivering new weapon systems while balancing factors that influence cost, schedule, and performance and ensuring that systems are high quality, supportable, reliable, and effective. According to DOD guidance, reliability is the probability of an item to perform a required function under stated conditions for a specified period of time. DOD’s acquisition environment has changed over time and this has affected the way the Department addresses reliability. Until the late 1990s, DOD’s goal was to achieve good reliability by focusing on specific reliability engineering tasks during design and manufacturing, and early testing to prevent, detect, and correct design deficiencies. In the late 1990s, in response to various NDAAs, DOD implemented certain acquisition reforms, eliminating and consolidating acquisition functions, and reducing the number of personnel assigned to the remaining functions. According to the Defense Science Board Task Force on Developmental Test & Evaluation, these reforms altered several aspects of the military acquisition process and DOD’s acquisition workforce. As a result, DOD lost experienced acquisition management and technical personnel. DOD officials stated this loss included reliability personnel who contributed to developmental testing and evaluation. DOD also canceled the Military Standard pertaining to reliability at this time. DOD officials explained that, after acquisition reform in the late 1990s, the department shifted much of the responsibility for reliability issues to contractors, and government personnel primarily focused on how systems performed during operational tests at the end of their development program. In the mid to late 2000s, Congress and DOD took actions to increase the focus on reliability engineering during weapon system design and development. Both Congress and DOD took steps to elevate the importance of reliability, which has continued through 2019. Figure 3 depicts selected laws related to reliability and DOD reliability efforts over time. Poor reliability can negatively affect the warfighters through low operational availability; that is, the amount of time a system is available to execute its mission. For example, the MV-22 aircraft was less reliable than intended, and required many more spare parts than expected. When the Marine Corps deployed to Iraq, MV-22 maintainers had to cannibalize parts from some MV-22s to keep other MV-22s flying, and as a result, the Marine Corps had fewer aircraft available to fly missions. Reliability can significantly influence a weapon system’s operating and support costs, which we have previously reported account for approximately 70 percent of a weapon system’s total life-cycle cost. Operating and support costs are a reflection of how programs achieve operational availability for weapon systems. Programs can achieve operational availability by building highly reliable weapon systems or, if the systems are not highly reliable, supporting them with an extensive logistics system that can ensure spare parts and other support items are available when needed. DOD has previously reported that deficiencies in DOD weapon systems—such as high failure rates and an inability to make significant improvements in reliability—have historically limited program performance and increased operating and support costs. In the commercial world, the manufacturer carries most of the risks that would result from developing a product with poor reliability. Such risks include increased warranty expenses that decrease profits. For example, reliability personnel from Ford, Cummins, and Thermo Fisher Scientific explained that more reliable products cost their companies less because they do not have to dedicate as many resources to fixing systems that fail, which would lead to warranty claims. In addition to increased costs, poor reliability can also negatively influence a company’s reputation. Ford representatives said that failures and product recalls are not just financial costs; recalls are highly publicized. A Thermo Fisher Scientific product manager explained that a customer’s bad experience can be shared in the media and negatively influence a company’s reputation. This may alter future buying behavior, especially in industries with relatively small customer bases in closely linked professional communities. This person shared a prior experience at a different company, where a design risk was identified during development. Instead of addressing the risk effectively, a standard cycle test was done to prove or disprove the risk. However, the test did not apply the stress necessary to cause the failure. The product was released to the market based on this successful but inadequate test. In the field, the components failed, and the company had to remove the product from the market. This damaged the company’s reputation and sales. We have previously reported that poor reliability is a concern for commercial companies because their customers demand products that work, or are reliable and do not experience failure, and the companies must develop and produce high-quality products to sustain their competitive position in the marketplace. In the commercial sector, reliability engineers told us their companies proactively address reliability from the beginning of the development process. We reviewed documentation from these companies and the 2019 Reliability and Maintainability Symposium and found engineers strive to identify reliability issues at the component and sub-system level early in the development process to avoid expensive rework after producing an entire system. We identified the following key practices in the commercial sector: leveraging reliability engineers early and often, establishing realistic reliability requirements—for example, not expecting a product to operate twice as long as its predecessor before failing, emphasizing reliability with their suppliers, and employing reliability engineering activities to improve a system’s design throughout development. Figure 4 shows some of the activities involved with these key practices. We found commercial companies in our review include reliability engineers as part of their development teams. In this role, reliability engineers implement reliability tools and methods that integrate statistics, physics, and engineering principles to help develop a reliable product. For example, HBM Prenscia identified that reliability engineers from several commercial companies said it was important to initiate their assessments early in the development life cycle when there is greatest opportunity to influence product design. According to leading reliability engineers, engineering activities can add value to decision-making by providing direction and feedback that helps development teams refine designs that lead to more reliable and cost effective systems. Researchers have reported reliability engineers should be empowered to influence decisions, such as delaying overall project schedule or negotiating for more resources when necessary. In addition, our analysis of reliability engineers’ documentation from the Reliability and Maintainability Symposium and commercial companies found it important that management provide sufficient resources and time dedicated specifically to improving reliability by discovering failures, implementing corrective actions, and verifying their effectiveness. Our analysis found that cost and schedule constraints can negatively influence reliability testing, which can limit development teams’ ability to discover failures and improve designs through corrective actions. Our analysis of documentation from the Symposium also highlighted the importance of having experienced reliability engineers. For example, Ford representatives told us they have a dedicated reliability engineering community that coaches the members of the company’s different product development teams. Ford’s reliability engineers said they focus on teaching development team members to ask the right questions at the right point in time with the right people in the room. We found companies in our review emphasize that reliability requirements should be realistic, be based on proven technologies, and reflect customer usage and the operating environment. To determine feasibility of meeting a requirement, reliability engineers we spoke with at Cummins and Thermo Fisher Scientific recommend conducting comparative analysis with historical data and assessing risk due to new, unique, or difficult technology. In addition, an independent reliability engineer with over 40 years of experience told us programs should provide justifications for how reliability requirements were established to demonstrate they are within the realm of technological possibility. If the reliability requirement turns out not to be technically feasible, it could have broad implications for the intended mission, life-cycle costs, and other aspects of the system. We have previously reported on the importance of making informed trade-offs when considering requirements to reduce program risk or total ownership costs. HBM Prenscia representatives told us the commercial companies they work with regularly make trade-offs involving capability, reliability, and cost requirements. Reliability representatives at Ford told us it is important to have the right people involved in these trade-off decisions, and that they work with user representatives and reliability engineers to define their systems’ reliability requirements. Systems produced by commercial companies in our review include parts or components produced by suppliers, and reliability engineers repeatedly told us the reliability of those parts or components directly impacts the reliability of the overall system. According to a leading reliability engineer, vendor quality can affect a part’s reliability, so it is critical that the reliability of vendors’ parts be evaluated before being approved for use. To emphasize reliability with suppliers, commercial companies in our review engage with suppliers early, clearly specify requirements with the supplier, and evaluate and monitor the supplier. Cummins representatives stated engaging the supplier early is critical. They explained that they engage the supplier early, during concept development, and ask the supplier to demonstrate it can meet requirements. According to Cummins representatives, this is to ensure the supplier is able to meet quality standards and to ensure there is enough lead time and testing of components. Reliability engineers at the Reliability and Maintainability Symposium also emphasized that reliability requirements must be clearly specified with suppliers, and product teams must actively monitor suppliers and assess their deliverables. Cummins representatives explained their engineers work directly with the supplier and hold it responsible for meeting reliability requirements. Ford representatives told us they evaluate and monitor the supplier to ensure the components it is providing are reliable. For example, they visit their suppliers’ testing facilities and evaluate their testing programs, focusing specifically on their failure analysis and reliability activities. We have previously reported that leading commercial companies use disciplined quality management practices to hold suppliers accountable for high quality parts through such activities as regular supplier audits and performance evaluations. A Thermo Fisher Scientific product manager provided a scenario where relying on an external supplier’s quality assurances would be insufficient. For example, a compressor is a critical – and commonly outsourced— component in complex industrial equipment. The product manager recommended in-house testing for critical components like a compressor rather than relying on a supplier’s testing that may not factor in real-world operating conditions. In house testing is recommended to avoid finding a failure after the product is brought to market. Post-sale failures result in dissatisfied customers, reputation damage, warranty claims and similar issues. The Thermo Fisher Scientific product manager said, in some cases, a company should establish a dedicated test facility for vital outsourced components provided by suppliers. Based on our review of commercial sector practices, we found companies use reliability engineering activities to identify potential product failures and their causes. They also use these activities to improve a system’s design early and often throughout development to avoid surprises that lead to expensive rework or excessive repairs after integrating components and subsystems. For example, HBM Prenscia representatives told us that failures should be identified early, and that identification should be viewed as an opportunity to improve the design and make the product better. According to leading reliability engineers, the earlier changes are made to designs, the less costly they are to the program. It is expensive, time consuming, and risky to make changes late in development, as late changes jeopardize product reliability. The commercial company representatives we spoke with also emphasized the need to conduct reliability engineering activities iteratively until the design is optimized. For example, HBM Prenscia has identified that a common mistake is establishing a reliability plan but not actively utilizing it throughout development. Reliability engineers use various reliability engineering activities to increase system reliability, and generally refer to these activities as design for reliability tools. These tools can be tailored to meet the specific needs of a particular development project, and can complement one another and increase reliability prior to any testing. These tools can help identify how long a part or component will work properly, how a part or component’s failure will affect a system, and what actions are needed to correct failures. See table 2 for some examples of design for reliability tools that can be used to help meet reliability goals. We have previously reported that leading commercial companies use a knowledge-based development process that enables decision makers to be reasonably certain that product quality, reliability, and timeliness are assured. Our analysis of documentation from reliability engineers found that reliability engineering activities should be integrated into the product development process, and their outputs should be reviewed at development milestones. These reviews can help ensure that reliability is a robust process rather than a paper exercise by providing an opportunity to assess data from reliability analysis or testing. For example, Cummins incorporates reliability reviews into its product development processes to ensure products meet reliability goals prior to moving to the next phase of development. This helps ensure the company is on track to fulfill its reliability commitments and will be able to deliver the promised product reliability to customers. The leading commercial practices we reviewed highlight the importance of consistently collecting, sharing, and analyzing data from reliability engineering activities to inform development efforts. Commercial companies we spoke with recognized the value of reliability data. For example, Cummins representatives stated they capture reliability data and share it across different product development teams to help inform estimates of reliability for new product development efforts. In addition, Cummins representatives noted that they are moving to an interactive database that personnel throughout the entire company can access. Similarly, HBM Prenscia representatives told us that failures and lessons learned from previous projects should be captured and shared within a company, and that doing so could help inform future product development efforts. We reviewed seven major defense acquisition programs and found they often reactively addressed reliability after identifying issues later in development. As shown below, these programs did not consistently reflect key practices we identified in the commercial sector, and instead prioritized other activities intended to have positive acquisition cost and schedule impacts. However, DOD officials noted that there has recently been a greater emphasis on reliability, and the three programs that started development in 2012 and 2014 reflected more of the key practices than the older programs. See figure 5, which notes a distinction between commercial companies’ suppliers and DOD contractors. For more detailed information on each program, see appendix I. The Expeditionary Fighting Vehicle (EFV) and F-22 programs did not involve reliability engineers early during system development. Instead, these programs leveraged engineers after reliability problems arose, including after they integrated components and subsystems and during system-level testing. At the end of system development, the programs brought in additional engineers and established more concerted reliability growth efforts. In one example, the EFV program did not have an overall systems engineer. Marine Corps acquisition officials stated that reliability was not a priority during the original system development process, and we have previously reported the program was instead focused on achieving other performance parameters, including water speed, survivability, and lethality. Prime contractor representatives identified some of their design engineers who lacked experience and did not comply with engineering standards as a root cause for problems discovered late in the development process. We also reported the lack of early systems engineering discipline and knowledge undermined the EFV program’s ability to develop informed and reasonable reliability requirements, delayed the identification of potential failures until integration, and contributed to poor vehicle reliability. In addition to frequent hydraulic system failures, leaks, and pressure problems, the EFV also suffered main computer failures that froze steering while operating in water. As we have previously reported, the EFV program was subsequently restructured. The program office hired additional engineers and consulted with Army reliability engineers to institute a reliability growth program. This program was intended to mitigate previously identified vehicle design issues related to reliability and other risks before proceeding into a second development and demonstration phase. However, the EFV program never got to fully realize the benefits of its new reliability approach, as less than 3 years after restarting development it was canceled due to continuing technology problems, development delays, and affordability concerns. For the F-22 program, officials stated that at points during development the program did not have a leadership position focused on reliability, and the official who oversaw reliability was also responsible for supply chain management. The officials noted that at the time these were not focus areas because the Air Force expected the contractor to conduct the needed reliability engineering. In 2004, we reported that, as early as low- rate initial production, however, the Air Force identified 68 parts that had a high rate of failure and needed to be removed or replaced, requiring additional contractor work. We also reported the F-22 canopy also experienced failures during testing, allowing it to achieve only about 15 percent of its expected lifetime. In 2014, we reported that later reliability maturation projects intended to address reliability deficiencies had a positive effect on availability over time, but as of 2018 the F-22 still had not met its availability target. As we have found in our prior reports as well as in this review, the EFV, F-22, F-35, and V-22 programs set unrealistic operational requirements for reliability. These requirements were, therefore, unachievable during development and before fielding the systems to warfighters. As we have previously reported, when programs overpromise a weapon’s prospective performance and deliver systems that cannot achieve their requirements, such as reliability goals, the warfighter receives less capability than originally promised. In one example, as we reported in 2019, more than 11 years after the start of F-35 production, none of the three aircraft variants (Air Force, Marine Corps, and Navy) had met the minimum targets for two of the program’s five reliability metrics. These include mean flight hours between part removals for replacement and mean flight hours between critical failures. We found that only the Navy variant had achieved the minimum target for a third goal, mean flight hours between maintenance events. As we reported, while the program has instituted an effort intended to improve reliability, the effort does not align improvement projects with the F-35’s reliability requirements. That is, the reliability improvement projects being funded may not improve the F-35’s performance against its reliability metrics. Ultimately, the program does not expect to achieve the unmet reliability metrics by full aircraft maturity, and program officials have acknowledged that the requirements should be reevaluated. As a result, the warfighter may not receive an aircraft that is as reliable as was expected. In a review of the V-22 program, DOD found that the program integrated complex technologies and unprecedented capabilities into its weapon system without accounting for unknown reliability risks. Specifically, these capabilities included a conceptually new design and multiple service and mission needs. However, officials stated that the program derived its reliability requirements from antecedent helicopters, systems that were not representative of the V-22 given its increased complexity. With a limited understanding of the V-22’s mission profile, program officials stated that they also underestimated the amount of time the system would be used in helicopter mode and its operating time on the ground. Subsequently, when the Marine Corps variant of the V-22—the MV-22—was deployed in Iraq from 2007 to 2009, a number of components experienced high rates of failure, affecting systems such as the engines and engine housing. This situation, combined with an immature parts supply chain, reduced the system’s availability significantly below minimum levels. At the time, as we reported in May 2009, the MV-22 had a stated minimum mission capability rate of 82 percent, but the three MV-22 squadrons in Iraq demonstrated an average of 62 percent. The development and integration of new technologies on the F-22— stealth, supersonics, and integrated avionics—were critical to achieving operational success, but also presented significant reliability risks. Officials told us that the F-22 was initially expected to cost less to acquire and operate than one of its predecessors, the F-15, and be more reliable as well. However, they also stated this was an unrealistic expectation. We have previously reported that the immaturity of technologies at the start of and throughout development weaken a system’s ability to achieve reliability requirements. Since 2005, when full rate production of the F- 22 began, the program has made substantial additional investments in increasing the system’s reliability through various improvement programs. But the program also changed its mean time between maintenance reliability requirement to an operational availability metric, a target that as of 2018 it had yet to meet and may need to reevaluate, according to program officials. If the F-22 cannot achieve its current reliability requirement, warfighters will have to execute their missions with a less capable aircraft than expected. The AMPV, EFV, F-35, and V-22 programs did not effectively emphasize reliability with DOD contractors. Specifically, according to DOD, the AMPV, EFV, and V-22 did not effectively incentivize reliability with the contractor and one program, the F-35, did not include all of the program’s reliability metrics in the contract. Each F-35 aircraft variant is measured against five reliability metrics, two of which are in part of the contract. Contractors are not responsible for achieving reliability requirements if programs do not include them in contracts. As of August 2018, two of the F-35’s three variants had not met minimum targets for any of the three metrics that are not in the contract. The last variant (Navy) has met the minimum target for only one of the three metrics. As we have previously reported, the warfighter may have to accept F-35 aircraft that are less reliable and more costly than originally expected. As we have reported, the F-35 program tried to encourage the aircraft’s manufacturer to improve reliability through an incentive fee in sustainment contracts. These contracts, for sustainment services, included incentives for meeting aircraft availability. Reliability of parts is one of the factors that influences aircraft availability, because broken parts prevent aircraft from flying. Program officials told us they hoped the incentive fee in the sustainment contract would incentivize the contractor to invest in and implement additional reliability activities, which would help improve aircraft availability, but according to the program office, the incentive has not been effective. Program officials told us the contractor has not pursued the incentive fee in the sustainment contract through efforts to improve aircraft reliability because it would have to invest significant resources to design and incorporate changes into production aircraft in order to do so. F-35 aircraft, especially early production aircraft, continue to face challenges related to parts that are failing more often than planned and are in short supply. For example, we have previously reported that DOD found the special coating on the F-35 canopy that helps maintain the aircraft’s stealth failed more frequently than expected and that the manufacturer could not produce enough canopies to meet demand, ultimately degrading system capability. According to program officials, to ensure that reliability growth was on track, the AMPV program offered an incentive fee of up to $16 million if the contractor could demonstrate at least 80 percent of the system’s reliability before low rate production. But officials stated that the AMPV contractor did not achieve the goal. The AMPV was a derivative system of the Army’s Bradley Fighting Vehicle with an accelerated development schedule, and officials stated that for this reason the contractor assumed the government would accept much of the Bradley’s initial design and changes to the AMPV’s performance resulting from legacy reliability issues. As a result of these expectations, officials stated that the contractor did not put enough resources, including a robust reliability team, toward the work that was eventually needed to improve reliability, and the contractor understaffed in this area. The AMPV, EFV, F-22, F-35, and V-22 programs deferred key reliability engineering activities, intended to improve system designs, until later in development. As a result, they missed opportunities to identify, understand, and mitigate reliability issues early in the development process. After realizing reliability shortfalls late in development, some programs initiated expensive redesign efforts that continued well into production and deployment, while others accepted degraded performance. Based on our prior reporting, we found the EFV program did not implement a proactive reliability approach, which would include identifying challenges early and designing reliability into the system in a cost- effective manner. Instead, the program used a test-fix-test approach that relied on identifying failure modes after the system-integration phase. Early in the acquisition process, officials noted in program documentation that the program had conducted little reliability growth planning before starting development, and officials stated that the EFV program did not plan for or conduct dedicated reliability testing. Then, the program prematurely conducted its critical design review, a key review during the development phase which confirms the system’s design is stable and is expected to meet system performance requirements, before the EFV prototype’s system-integration work was complete. The program did not have the time necessary to demonstrate design maturity as scheduled and officials stated that they did not schedule long enough corrective action periods to allow for proper failure mitigation. As a result, during a 2006 operational assessment, the EFV demonstrated very low reliability and failed to complete amphibious, gunnery, and land mobility tests. F-22 program officials stated that many of the aircraft’s components and subsystems had to be tested as part of an integrated system. This limited the discovery of reliability issues early in the development phase. DOD reliability experts told us programs should not use integrated system testing to demonstrate individual component reliability, and should instead use it to focus on how components work together and identify more complex system failure modes. F-22 officials also stated the program office frequently continued with development and other testing before implementing corrective actions for critical reliability issues. As we have previously reported, the F-22 program started a program to improve its reliability in 2005, near the start of full rate production, to mitigate hundreds of known reliability issues deferred from earlier in development. Nonetheless, we reported in 2012—nearly 3 years after DOD announced the end of F-22 production—that reliability deficiencies had increased support costs, and continued to prevent the aircraft from meeting its reliability requirement. According to program officials, the Army selected a derivative of the Bradley Fighting Vehicle to meet the AMPV requirements, even though that vehicle’s transmission had known reliability problems. According to AMPV program officials, the Army selected this vehicle because it had prioritized controlling costs and accelerating schedule. Program officials stated that the focus on cost and schedule caused the contractor to underestimate the necessary reliability work at the start of development and led to a backlog of test incident reports and deferred corrective actions. According to 2018 program documentation we reviewed, the AMPV’s reliability growth did not track to targets during development and the vehicle did not achieve its pre-production reliability goal. Moreover, some of the AMPV’s deferred work may need to be addressed during a future corrective action period that could continue through fiscal year 2021. Although there are differences between the DOD and commercial sector stemming from the statutory and regulatory structures that govern DOD’s acquisition processes, DOD has had long-established policy and guidance at both the department and service level that recognize the four key reliability practices we found in the commercial sector. For example, the Defense Acquisition Guidebook encourages acquisition programs to involve reliability engineers early and often, and DOD Instruction 5000.02 identifies the need for establishing realistic reliability requirements. Additionally, the 2005 DOD Guide for Achieving Reliability, Availability, and Maintainability addresses the importance of emphasizing reliability with contractors, and the service-level policies at all three military departments establish the importance of reliability engineering activities. However, most of these documents cover a wide range of acquisition issues or many aspects of reliability engineering, and they do not specifically emphasize the four key practices we identified in our review of the commercial sector. For example, the DOD Instruction 5000.02 is an overarching policy document covering the entire acquisition life cycle at a high level, from concept development to live fire test and evaluation, and only one section provides significant detail and direction on reliability. The service level instructions and Defense Acquisition Guidebook similarly cover the entire acquisition life cycle, and reliability is one of dozens of characteristics addressed in each document. The DOD Guide for Achieving Reliability, Availability, and Maintainability is largely focused on achieving reliability, but the reliability proponents at OSD, the Army, and the Navy said the guide is not consistently used throughout DOD, noting that it was issued in 2005 and has not been updated since. DOD policy provides decision makers flexibility to tailor regulatory activities that acquisition programs perform when developing weapon systems. The process is inherently complex, and these decision makers must balance many factors when overseeing and executing the programs. In the absence of an emphasis on the key reliability practices we identified, we found decision makers for the programs we reviewed prioritized other activities intended to have positive acquisition schedule and cost impacts. For example, AMPV program officials told us the program eliminated 7,500 miles of contractor reliability testing in order to proceed to the next development phase more quickly, believing that there would be sufficient time later to complete corrective actions. Recently, DOD has begun employing the Middle Tier Acquisition pathway—an alternative acquisition pathway with an objective of beginning production within 6 months and completing fielding within 5 years. This emphasis may encourage decision makers to prioritize activities that promise to reduce schedule. We found that for the programs we reviewed, however, such an approach can come at the expense of other activities, such as implementing effective reliability practices. DOD has recently taken steps that could introduce more balance when decision makers consider trade-offs between schedule and reliability. Specifically, DOD has highlighted the importance of one of the four key reliability practices we identified: emphasizing reliability with contractors, and Congress has passed legislation related to reliability. The NDAA for fiscal year 2018 included a provision mandating DOD program managers to include certain reliability requirements in weapon system engineering and manufacturing development and production contracts. In January 2019, the USD(A&S) implemented the NDAA by issuing a policy memorandum to Service Acquisition Executives and other DOD Directors echoing this key practice. However, USD(A&S) has not similarly emphasized the three other key reliability practices we identified in the commercial sector, nor have the Secretaries of the Air Force, Army, and Navy, who now have ultimate responsibility for most of DOD’s major acquisition programs. Specifically, these senior leaders have not emphasized the value of leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s design throughout development. As a result, it is less likely that acquisition programs will take the actions necessary to recognize and address potential reliability problems early in the development process. Without senior leadership emphasis on a broader range of key reliability practices, DOD runs the risk of delivering less reliable systems than promised to the warfighter and spending more than anticipated on rework and maintenance of major weapon systems. This risk is exacerbated in an environment where decision makers are striving to deliver systems in an accelerated manner. The best opportunity to influence the reliability of a weapon system is early on during the design of the system. Decisions and tradeoffs made at that time can increase the weapon system’s reliability, help warfighters execute their missions, and decrease operating costs for years to come. However, these decisions and tradeoffs are not easy, as acquisition decision makers are tasked with managing competing priorities such as cost, schedule, and performance. Many of the DOD acquisition program examples in this report illustrate what can happen when reliability is not prioritized. The programs often approached reliability in a reactive manner, discovered problems late in the development process, and then tried to fix them through costly and time-consuming rework. The programs did not consistently adhere to key practices we identified in the commercial sector: reliability engineers were not leveraged early in the development process, reliability requirements were not realistic, reliability was not emphasized with contractors, and reliability engineering activities were not utilized throughout design and development. Recent DOD actions have highlighted the importance of emphasizing reliability with contractors. DOD senior leaders can help improve reliability by highlighting the importance of the three other key reliability practices we identified in the commercial sector. In light of the current focus on accelerating the acquisition process, balancing the desire for speed with reliability considerations is critical. Given the delegation of acquisition decision authority to the military services, the Secretaries of the Air Force, Army, and Navy are in the best position to do so. We are making a total of three recommendations: one each to the Air Force, the Army, and the Navy. We recommend the Secretary of the Air Force issue policy emphasizing the following three key reliability practices when planning and executing acquisition programs: leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s design throughout development. (Recommendation 1) We recommend the Secretary of the Army issue policy emphasizing the following three key reliability practices when planning and executing acquisition programs: leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s design throughout development. (Recommendation 2) We recommend the Secretary of the Navy issue policy emphasizing the following three key reliability practices when planning and executing acquisition programs: leveraging reliability engineers early and often, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system’s design throughout development. (Recommendation 3) We provided a draft of this report to DOD for review and comment. DOD’s written comments are reprinted in appendix II. DOD stated that the Air Force, Army, and Navy concur with our recommendations to their respective Departments. The comments also state that the Air Force and Navy plan to update their policies in response to our recommendations. As for the Army, the comments state that the Army Acquisition Executive will issue direction emphasizing the three key reliability practices and highlight an existing Army regulation focused on reliability engineering. In addition to the responses to our recommendations, DOD’s written comments included technical comments that we addressed as appropriate. For example, we provided additional detail on an existing DOD policy, and clarified how a program engaged with a contractor. We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or mackinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This appendix summarizes key characteristics of seven selected major defense acquisition programs’ approach to reliability. The four key characteristics are categorized as: did not leverage government reliability engineers in decision making initially pursued unrealistic operational requirements for reliability; did not effectively emphasize reliability with contractors; and, deferred key reliability engineering activities until later in development. These summaries do not address all the reliability actions taken by each program; rather they focus on key characteristics we identified in our review of commercial companies and associated deficiencies. See figure 6, which notes a distinction between commercial companies’ suppliers and DOD contractors. In addition to the contact named above, Nathan Tranquilli (Assistant Director), Julie A. Clark (Analyst-in-Charge), Lori Fields, Laura Greifner, Brendan K. Orino, LeAnna Parkey, Christine Pecora, Andrew N. Powell, Timothy Saunders, and Michael J. Sullivan made key contributions to this report.", "summary": "DOD invests tens of billions of dollars each year in major defense acquisition programs, designing and developing technologically advanced weapon systems that warfighters expect will meet specific performance requirements, including reliability requirements. Systems that are not reliable make it more difficult for warfighters to perform their missions. GAO was asked to examine DOD weapon system reliability. This report addresses (1) how selected companies in the commercial sector address reliability, (2) how selected DOD acquisition programs addressed reliability, and (3) the extent to which DOD leadership has highlighted key reliability practices. GAO collected information on leading commercial practices at the 2019 Reliability and Maintainability Symposium and from four commercial companies known for delivering reliable products. GAO also assessed how seven DOD acquisition programs—both older and newer, and representing all the military services—addressed reliability; reviewed key documents and interviewed knowledgeable officials; and reviewed reliability-related guidance and policy from senior DOD leaders. The commercial companies GAO reviewed proactively address reliability. They strive to identify reliability issues at the component level early in the development process to avoid expensive rework after producing an entire system. GAO found these companies focus on the following key practices: 1. Leveraging reliability engineers early and often 2. Establishing realistic reliability requirements 3. Emphasizing reliability with their suppliers 4. Employing reliability engineering activities to improve a system's design throughout development GAO found that the seven Department of Defense (DOD) acquisition programs it reviewed did not consistently adhere to these key practices (see figure). These programs often prioritized schedule and cost over incorporating the key reliability practices, and these systems generally were not as reliable as promised. In 2019, DOD highlighted in a policy memorandum the importance of emphasizing reliability with contractors. However, the other three key practices have not been similarly highlighted. DOD has taken steps to accelerate weapon system development, and decision-making authority has been delegated to the military services. In an environment emphasizing speed, without senior leadership focus on a broader range of key reliability practices, DOD runs the risk of delivering less reliable systems than promised to the warfighter and spending more than anticipated on rework and maintenance of major weapon systems. GAO recommends the Secretaries of the Air Force, Army, and Navy highlight the importance of three key reliability practices: leveraging reliability engineers, establishing realistic reliability requirements, and employing reliability engineering activities to improve a system's design throughout development. DOD agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "The Secretary of the Army has the responsibility to recruit personnel, subject to the authority, direction, and control of the Secretary of Defense. The Assistant Secretary of the Army for Manpower and Reserve Affairs serves as the principal advisor to the Secretary for the Army’s management of its manpower and personnel and provides overall governance for marketing, advertising, and research. The Deputy Chief of Staff of the Army, G-1, is the principal military advisor to the Assistant Secretary of the Army for Manpower and Reserve Affairs and the Chief of Staff of the Army for all matters related to manpower across the Army. As of August 2019, the Deputy Chief of Staff of the Army, G-1, has responsibility for overseeing the new Office of the Chief Army Enterprise Marketing (AEMO) once it is fully established, as described later in this report. In addition, multiple other Army organizations from across the accessions enterprise—the collection of Army organizations involved in efforts to recruit and train soldiers for the Army—have roles and responsibilities in carrying out the Army’s marketing and advertising program, as summarized in table 1. The Army contracts with a primary advertising agency to develop and implement its marketing and advertising program. The advertising agency is responsible for providing a range of services from the development of the Army’s marketing and advertising strategy to the production of marketing and advertising activities, including television and print advertisements, event marketing, and social media. In November 2018, the Army awarded a contract to a new advertising agency for up to 10 years with a value not to exceed $4 billion. The contract with the previous advertising agency was awarded in March 2011, and from March 2011 through the end of fiscal year 2018, the Army issued 702 task orders and obligated about $1.6 billion on this contract, according to our analysis of data from the Federal Procurement Data System-Next Generation. The Army conducts a variety of marketing and advertising activities at the national and local levels in support of the Army’s recruiting goals. Figure 1 shows examples of the various types of Army marketing and advertising activities, such as mobile assets used at recruiting events and digital advertising on social media. Different marketing and advertising activities are useful for supporting the three phases—awareness, engagement, and activation—of an individual’s decision-making process, sometimes referred to as the consumer journey. The goal of marketing and advertising is to move a potential recruit through each of the three phases and, ultimately, to the decision to enlist: Awareness: Individuals learn about the opportunity to serve in the Army and the distinct characteristics of serving in the Army. The Army pursues awareness through marketing and advertising activities such as television commercials, print advertisements, banners at events, and billboards. Engagement: Individuals who are aware of the opportunities for service in the Army begin considering the possibility of joining the Army. During this phase, the Army seeks to provide recruits with additional information to aid in their decision-making process. Often this phase of advertising takes place in the digital environment, as the Army seeks to provide informative social media posts and use banner advertisements to attract individuals to visit its website for more information. Activation: Individuals have considered the Army and are ready to talk to a recruiter about enlistment. As such, activation activities seek to encourage these individuals to provide their contact information to schedule an opportunity to meet with a recruiter. Activation is often conducted in person, such as through recruiters’ presence at events like career fairs. Activation may also be conducted through other means—such as direct mail and online or print classified advertisements—as long as the advertisement prompts viewers to provide their contact information. Further, the Army employs the use of mobile assets, such as large trucks and trailers fitted with equipment and activities intended to draw crowds and encourage and facilitate public interaction with a recruiter at an event to generate leads. Figure 2 shows the awareness, engagement, and activation phases and examples of marketing and advertising activities that are used in support of each phase. To implement AAA’s recommendations, the Army has developed processes designed to improve its oversight of the primary contract for executing the Army’s marketing and advertising program. In April 2018, AAA reported that AMRG did not sufficiently evaluate the performance of its primary contractor, effectively oversee deliverables included in its three main marketing and advertising contracts, or effectively oversee the negotiation process of task orders for its primary marketing and advertising contract. AAA made seven recommendations to AMRG to improve its contract oversight, with which AMRG agreed. As shown in table 2, as of September 2019, AAA considered three of the recommendations implemented but not closed, with AMRG still taking steps to address the other four recommendations; as a result, it is too soon to assess the extent to which the Army’s steps have improved contract oversight within the marketing and advertising program. AMRG officials stated that the implementation of these recommendations has been slowed, in part, because of the recent award of its primary marketing and advertising contract to the new advertising agency. Among AMRG’s actions to improve its contract oversight are steps to develop processes for overseeing contractor performance, deliverables, and price negotiations. Specifically, AMRG has taken steps in the following areas: Created a new project management office. AMRG established a project management office to help address the challenges identified by AAA and to serve as a coordinating body that centralizes contract oversight. According to its charter, the office is responsible for maintaining cost, schedule, and performance for Army marketing and advertising programs to help ensure that they are completed on time and within reasonable costs to support the accessions mission. Documents outlining the contract management process indicate that the project management office reviews key contract documents, such as the statement of objectives and the quality assurance surveillance plan, before the documents are submitted to the contracting officer who is responsible for administering the contract. After the contractor submits its proposal for providing the requested product or service, the project management office reviews the proposal to ensure it meets the requirements of the statement of objectives. Also, the project management office coordinates and submits the technical evaluation form for review by the contracting officer’s representative and the contracting officer. Within the newly-formed AEMO, there will be a project management office with six authorized personnel, according to the organizational chart for AEMO. In July 2019, AMRG officials stated that they were developing standing operating procedures and continuity plans that AEMO could use as it establishes its project management office. Implementing training and tools to evaluate contractor performance. AAA found that AMRG did not sufficiently evaluate the performance of its primary contractor and recommended that AMRG require that all program managers receive contracting officer’s representative training and ensure that individual quality assurance surveillance plans are developed for each task order over $150,000. As of September 2019, AAA reported that this recommendation had not been fully implemented. In response to AAA’s recommendations, in April 2019, AMRG reported that all program managers had completed contracting officer’s representative training. Further, as of September 2019, AMRG reported that the Army had developed individual quality assurance surveillance plans for each task order issued in fiscal year 2019. According to AAA officials, AMRG provided them with examples of these quality assurance surveillance plans, and AAA provided AMRG with feedback on additional information that should be included, which AMRG officials stated they were taking actions to address. In September 2019, AMRG also issued standing operating procedures for program managers to provide internal policy and instruction for executing the Army’s marketing and advertising program as well as evaluating its performance, including overseeing contractor performance. We found that the standing operating procedures require contracting officer’s representative training for program managers and that the Director of Marketing, Director of Research, and AMRG contracts team are to monitor compliance quarterly. In addition, the procedures include steps outlining contract oversight mechanisms, such as information on the purpose and contents of quality assurance surveillance plans. Implementing a standardized technical evaluation form. AAA found that AMRG had not effectively negotiated prices for its primary marketing and advertising contract; AAA recommended that AMRG (1) define and implement a well-structured policy for conducting technical evaluations of contractor proposals and (2) establish a standardized form to ensure consistency during the evaluation process. As of September 2019, AAA reported that these two recommendations were implemented but not closed. The standing operating procedures that we reviewed outline the process for program managers to complete a standardized form for evaluating contractor proposals. The form is intended to ensure that program managers are consistently evaluating contractor proposals for performing work under the contract. For example, the form requires program managers to conduct a comparative price analysis by comparing the contractor’s proposed price to total task order cost in prior years. According to the instructions on the form, program managers are to submit the completed form to their supervisor or director for approval. The project management office then coordinates and submits the technical evaluation to the contracting officer’s representative and contracting officer for review. Overseeing contract deliverables. AAA found that AMRG did not effectively oversee deliverables in its marketing and advertising contracts and recommended that AMRG develop procedures to ensure that contracts or task orders do not contain deliverables already provided in other contracts. As of September 2019, AAA reported that this recommendation had not been fully implemented. AMRG and U.S. Army Mission Installation and Contracting Command officials have implemented processes to prevent duplicative deliverables (i.e., services or products to be provided through a contract) in the future, such as the standardized technical evaluation form noted above. We found that the standardized technical evaluation form requires program managers to certify that they have reviewed other tasks and contracts within their purview and to validate that the task order being requested does not duplicate existing or other requested work. In addition, AMRG and U.S. Army Mission Installation and Contracting Command officials stated that they consolidated all contract actions under one team at the U.S. Army Mission Installation and Contracting Command and that both AMRG’s contracting officer’s representative and budget office must verify that contract requests are not duplicative. AAA also found overlapping deliverables between AMRG’s primary marketing and advertising contract and a contract for creative technology support and recommended that AMRG use the creative technology support contract for all of the creative technical services within its scope. As of September 2019, AAA reported that this recommendation had not been fully implemented. AMRG officials told us that they plan to issue a modification to the contract for creative technology support to remove services duplicated in the primary marketing and advertising contract. As of September 2019, an AMRG official told us that the Army expected to issue the modification in November 2019. Revising process for contract award fees. AAA found that AMRG had minimal support to justify its higher award fee ratings for its primary contractor and recommended that AMRG update its award fee plan and award fee review process to include soliciting feedback from program managers, maintaining supporting documentation, and obtaining objective performance data, among other things. As of September 2019, AAA reported that this recommendation had not been fully implemented. We found that AMRG revised its process in fiscal year 2018 for determining award fee incentives for its advertising agency. For example, according to the documentation associated with the award fee decision for the agency’s performance from April 2017 through April 2018, AMRG reported that it, among other things, included feedback from program managers on the advertising agency’s performance and obtained objective performance data from an independent entity, DOD’s Joint Advertising, Market Research & Studies. In future work with the new advertising agency, AMRG officials stated that the Army plans to offer award fees for specific task orders rather than one fee for performance in a given year. According to AMRG officials, this change allows greater flexibility in deciding which programs should be incentivized with an award fee. As of September 2019, AMRG officials stated that they had not issued any task orders with an award fee under the new marketing and advertising contract. To implement AAA’s recommendations, the Army has taken steps to improve how it measures the effectiveness of its marketing and advertising program; these steps are consistent with commercial best practices for assessing the effectiveness of advertising identified in our prior work. In its 2018 report on return on investment, AAA found that AMRG had deficiencies in how it measured the effectiveness of its marketing and advertising efforts and made seven recommendations to AMRG, with which AMRG agreed. Of these recommendations, AAA considered four implemented but not closed as of September 2019, with AMRG still taking steps to address the other three recommendations, as shown in table 3. Since the Army’s steps were recently implemented or are ongoing, it is too early to determine if they will achieve their desired results. Based on our analysis of the Army’s actions, the Army’s steps to implement AAA’s recommendations fall into the following five areas: (1) revising strategic goals, (2) updating and documenting its assessment process, (3) improving the reliability and capabilities of data systems, (4) integrating national and local marketing and advertising efforts, and (5) obtaining new tools to determine required marketing and advertising resources. The steps the Army has taken in these areas thus far are consistent with commercial best practices for assessing the effectiveness of advertising we identified in our prior work. As the Army takes additional steps to establish the newly-formed AEMO, it will be important for the Army to continue to align its efforts with these commercial best practices for assessing the effectiveness of advertising to ensure advertising dollars are used efficiently to help meet stated recruiting goals. Revising strategic goals. AAA found that AMRG did not have specific goals to measure the long-term effects of investments in marketing and advertising efforts to support the Army’s accessions mission and recommended that AMRG develop such goals. As of September 2019, AAA reported that this recommendation was implemented but not closed. AMRG has revised its strategic marketing goals from tracking changes in individuals’ attitudes toward the Army, such as support for the Army among the general population, to tracking the behaviors of these individuals, such as the number of visits to GoArmy.com. For fiscal year 2018, AMRG had seven strategic marketing goals that tracked the attitudes of the general population and prospects toward the Army. For fiscal year 2019, AMRG revised the goals to four that track attitudes, two that track behaviors, one that tracks effectiveness, and one that tracks efficiency. Goals that track attitudes are aligned with the awareness phase of the consumer journey, whereas goals that track behaviors are more aligned with the engagement and activation phases. Figure 3 shows how the fiscal year 2018 and 2019 strategic marketing goals align with the three phases of the consumer journey. Looking ahead, AMRG officials told us that, consistent with feedback they received from marketing industry experts, the strategic goals in the fiscal year 2020 marketing plan will all be behavioral and will target the different stages of what AMRG refers to as a lead nurturing funnel. AMRG officials stated that their goal is to use information so they can quickly shift attention and funding to different stages of the funnel that are not meeting their goals, so as to ensure that those stages get the attention needed to reach mission success. AMRG’s recent and ongoing steps to revise its marketing goals are consistent with the commercial best practice to develop an evaluation framework that identifies the target audience and includes measurable goals. To institutionalize AMRG’s updated processes, AAA recommended that AMRG update Army Regulation 601-208 to reflect the new goals and processes it would implement to improve its program effectiveness. As of September 2019, AAA reported that this recommendation had not been fully implemented. The Army is in the process of revising its marketing and advertising regulation to reflect the updated strategic marketing goals and process. AMRG had drafted a revision to the regulation; however, AMRG officials had put this process on hold while senior Army leaders were making the decision about AMRG’s placement within the Army. Now that AMRG has been redesignated as AEMO and reassigned within the Army, as of August 2019 AMRG officials stated that the Army was revising the regulation to reflect the new organization’s relationship to other entities within the accessions enterprise. AMRG expects the Army to publish the updated regulation in 2020. Updating and documenting its assessment process. AAA found that only three of AMRG’s 23 national events during fiscal year 2016 provided the best value for their intended purpose and recommended that AMRG discontinue efforts that were not cost-effective in comparison to other options and assess the cost-effectiveness of current marketing and advertising efforts. As of September 2019, AAA reported that this recommendation was implemented but not closed. According to AMRG officials, AMRG discontinued all of the events that were deemed to not be cost-effective in AAA’s report. In its report, AAA also found that AMRG’s assessment process did not include USAREC’s and USACC’s marketing and advertising efforts and that AMRG did not formally document that process; AAA recommended that AMRG establish and formally document a process with roles and responsibilities to assess the effectiveness and efficiency of all Army marketing and advertising efforts. As of September 2019, AAA reported that this recommendation was implemented but not closed. We found that AMRG formally documented how it assesses the effectiveness of the marketing and advertising program, consistent with the commercial best practice of seeking to develop an understanding of how outcomes can be attributed to advertising. In January 2019, AMRG issued guidance that outlines its assessment process, including the types of information that are reviewed in each assessment. The guidance we reviewed outlines three levels of assessments: Level I is a review by program managers of their individual programs; level II, to be conducted on a quarterly basis, is a review at the operational level in which the directors of research and marketing review the results of marketing and advertising efforts across multiple marketing channels; and level III, also to be conducted on a quarterly basis, is a review at the strategic level in which the AMRG Director reviews the Army’s progress in meeting its strategic marketing goals. Further, in January 2019, AMRG reviewed a summary of the number, cost, and performance of USAREC’s and USACC’s local marketing and advertising activities as part of the level II assessment process. In addition, to facilitate comparison of options, AMRG developed a tool for comparing different programs within a given marketing channel. The tool, which AMRG calls a decision support matrix, allows officials to comparatively rank different programs based on weighting different factors. For example, based on our review of the decision support matrix, we found that AMRG assigned a higher weight to a program’s effectiveness than its timing. The tool also incorporates qualitative feedback based on how program managers, USAREC, and USACC rank the programs, and quantitative analysis on the cost per lead, impression, or engagement, depending on the type of program. Improving the reliability and capabilities of data systems. AAA identified discrepancies between information in the Enterprise Marketing Management (EMM) system and supporting documentation and recommended that AMRG establish and formally document a process to ensure that all Army marketing and advertising performance and cost data were regularly recorded in an official marketing system of record on a regular basis. As of September 2019, AAA reported that this recommendation had not been fully implemented. The Army has taken steps to improve the reliability of the data in EMM since AAA’s report. In August 2019, the Army issued a task order on its primary marketing and advertising contract covering EMM system support to include overseeing and improving the quality of data in EMM. According to the performance work statement, the Army’s new advertising agency is responsible for, among other things, accurately documenting current data, systems, and business processes, as well as analyzing EMM reports and documentation for completeness and accuracy. Further, the advertising agency is responsible for identifying and documenting business problems and recommending areas for improvement and technology solutions. During the focus groups we held with AMRG personnel, participants told us that AMRG leadership was focused on demonstrating the effectiveness of the Army’s marketing and advertising through reliable and readily-available data. For example, the Army implemented an electronic business reply card, which is a digital form to capture a potential recruit’s eligibility and contact information and a means of identifying the event where the recruit learned about the Army. USAREC and USACC officials told us that prior to the electronic business reply card, recruiters collected prospects’ information at events by using a paper card. Although that card reflected the event where the potential recruit heard about the Army, it would often take several days before the potential recruit’s information appeared in the recruiting system, according to these officials. As a result, recruiters would sometimes not send in the paper card and would instead enter the prospect’s information directly into the system. In these cases, the marketing and advertising event would not receive credit for generating the lead. USAREC and USACC officials stated that the electronic business reply card’s quicker turnaround time for leads showing up in the system should improve data reliability by ensuring that recruiters and recruiting operations officers consistently enter an individual’s demographic data into the system, along with the marketing and advertising activity they interacted with. These steps to better identify the number of leads generated by marketing and advertising activities are consistent with the commercial best practice of conducting ongoing analysis of performance using industry standard measures appropriate for the purpose of the advertising activity. In addition, in August 2019, the Army issued a task order on its new marketing and advertising contract for the maintenance and optimization of its system that tracks analytics on the Army’s marketing and advertising activities, which AMRG refers to as the Intelligence Hub. The advertising agency is responsible for monitoring this system and producing reports that track the effectiveness of marketing and advertising activities based on key performance indicators. The advertising agency is also responsible for upgrading the system to track the multiple marketing and advertising resources that a potential recruit interacts with. AMRG officials told us that the upgrade of this system is intended to equip the Army with more complete data to demonstrate the effectiveness of the Army’s marketing and advertising activities— consistent with the commercial best practice of seeking to develop an understanding of how outcomes can be attributed to advertising. Integrating national and local marketing and advertising efforts. AAA found that AMRG did not integrate and leverage both national and local marketing and advertising efforts to support the Army’s accessions mission and recommended that AMRG revise the Army’s marketing performance framework to include marketing and advertising efforts at both the national and local levels. As of September 2019, AAA reported that this recommendation was implemented but not closed. We found that the Army has created programs and instituted procedures designed to increase coordination of national and local marketing and advertising efforts. For example, AMRG reported that it included other Army components, including USAREC and USACC, in developing the fiscal year 2019 marketing goals and planned to include those organizations in its fiscal year 2020 process. In addition, TRADOC established the Army Accessions Resource Fusion Board, which brings together organizations from across the accessions enterprise for quarterly meetings at which they make operational resource sharing plans for marketing and recruiting assets. For example, according to a March 2019 briefing for an Army Accessions Resource Fusion Board meeting, representatives from USAREC brigades discussed their planned marketing and advertising activities for the first quarter of fiscal year 2020, including any requests they had for support from other Army stakeholders for those planned activities. Further, according to its charter, the Army Accessions Resource Fusion Board is responsible for assessing the effectiveness of local marketing and advertising efforts in the previous quarter. In fiscal year 2018, the Army also created a pilot program designed to improve how the Army’s marketing and advertising program coordinates with its recruiting components and to produce marketing and advertising messages that resonate more effectively with target populations. The Army began implementing the program in Chicago in fiscal year 2019 and as of April 2019 was planning to expand the program to Boston and four other cities. As of July 2019, AMRG had observed positive results from the program in Chicago. For example, AMRG reported an increase of 11 percent in the number of leads and an increase of about 7 percent in the number of recruits who signed contracts with the Army when compared to the prior year in that region. Officials from TRADOC, USAREC, and USACC told us that coordination with AMRG on marketing and advertising efforts has improved since the time AAA conducted its audits. For example, TRADOC officials stated that AMRG senior leaders have supported the accessions enterprise by providing analytic support to USAREC and USACC. Further, USAREC officials stated that in fiscal year 2018 AMRG started to provide funding for local marketing and advertising activities near USAREC’s requested levels, and that this change had been carried forward into fiscal year 2019. Obtaining new tools to determine required marketing and advertising resources. AAA found that AMRG did not use a resource requirements projection model that supported and linked to planned marketing efforts and recommended that AMRG develop such a model. As of September 2019, AAA reported that this recommendation had not been fully implemented. AMRG has contracted with the RAND Corporation and a consulting firm to develop tools to determine the resources AMRG needs to conduct its marketing and advertising activities. The RAND tools include three planned models, one of which is the recruiting resource model recommended by AAA. The recruiting resource model has been partially completed and is being updated with additional data with full completion scheduled for September 2020. According to the Army’s fiscal year 2020 budget request, the Army used the RAND report that developed this model as a justification for increasing its advertising budget for fiscal year 2020. Further, in consultation with a consulting firm, AMRG developed a channel allocation simulator that allows AMRG officials to test different funding levels for its marketing and advertising channels to see potential outcomes. For example, based on our review of the simulator, AMRG can enter a specific amount of funding for events to estimate how many leads it can expect to obtain from that level of funding. AMRG officials stated that they can use this tool to help them plan for their required level of resources for the upcoming fiscal year. The development of this simulator is consistent with the commercial best practice of using sophisticated marketing mix modeling to determine an appropriate spending strategy. ASA(ALT) and OPM conducted reviews of AMRG’s workforce and made recommendations to improve the workforce practices within the marketing and advertising program. From January to May 2018, ASA(ALT) conducted a review of AMRG’s business processes and found high-risk issues that contributed to organizational inefficiencies within five areas: (1) internal communications, (2) business performance, (3) training, (4) program performance and accountability, and (5) personnel. For example, ASA(ALT) found that AMRG personnel were unclear about AMRG’s core mission, objectives, and program priorities. Further, ASA(ALT) found that AMRG’s personnel, skills, training, and physical locations were not aligned to support AMRG’s mission. In addition, OPM conducted an assessment from March to September 2018 to identify organizational inefficiencies and propose solutions intended to transform AMRG into a high-performing organization and improve its workforce morale. Similar to ASA(ALT), OPM identified issues with a lack of mission clarity and insufficient communication and collaboration throughout AMRG’s workforce and with its stakeholders. In addition, OPM identified a number of organizational design issues within AMRG, including workforce acquisition, management, and optimization of its operational components and staff. ASA(ALT) and OPM made multiple recommendations to address these issues within AMRG. For example, ASA(ALT) recommended that AMRG establish and disseminate standard operating procedures and process charts; clarify roles and responsibilities of the various organizational components; and clearly communicate to staff the final annual marketing strategy. OPM recommended a multiphased approach to implementing its overall recommendations, identifying key actions to take in each phase. For example, within the first phase, OPM recommended that AMRG determine the new functional structure for AMRG because it would improve management and accountability, collaboration, and stakeholder satisfaction. In addition, within the second phase, OPM recommended that AMRG develop a human capital management plan and review and update its position descriptions regularly to ensure they align with changing goals, staffing needs, and the organizational structure of AMRG. AMRG took some steps to address ASA(ALT)’s and OPM’s recommendations to improve its workforce practices. Within its report, ASA(ALT) noted that AMRG had started to take actions to implement several recommendations, such as disseminating AMRG’s mission statement, priority objectives, strategic goals, and fiscal year 2019 annual marketing plan guidance to all AMRG personnel. Similarly, OPM noted that AMRG had established task groups to coordinate with stakeholders, participated in meetings with Congress and stakeholders, and was developing a new vision for AMRG. However, as of April 2019, senior AMRG officials stated that they had not taken steps to address all of the reports’ recommendations because the Army was considering broader organizational changes to the placement of AMRG within the Army. While the recommendations in ASA(ALT) and the OPM reports were generally specific to AMRG’s organization and workforce at that time, senior AMRG officials stated that the Army would take additional steps to incorporate ASA(ALT)’s and OPM’s recommendations, as appropriate, after senior Army leadership made decisions about those changes. In our review, AMRG personnel we met with continued to identify poor internal communications and morale as key challenges within AMRG, consistent with the findings from ASA(ALT) and OPM. During our focus groups with AMRG personnel, participants repeatedly stated that senior AMRG leadership did not communicate key information to staff. For example, participants told us that senior AMRG leadership did not communicate information about AMRG’s mission, strategic priorities, or pending organizational changes. As described below, subsequent to our focus groups, the Army began taking steps to fundamentally change the organizational structure, workforce, and leadership of its marketing and advertising program. In light of the timing of these substantial changes, we did not comprehensively assess the extent to which communication issues have been resolved in the reorganization of the marketing and advertising program. It will be important for the new leadership to focus on communication at the outset of this organizational change to establish positive morale within the workforce. The Army recently reorganized its marketing and advertising program to improve its organizational structure; the Army’s early steps to implement the reorganization are consistent with some key practices for agency reform efforts we identified in our prior work, as described below. In May 2019, the Secretary of the Army reassigned, redesignated, and stated that the Army planned to relocate AMRG. The Secretary of the Army redesignated AMRG as AEMO and reassigned the office to the Deputy Chief of Staff of the Army, G-1. The effective date of this reassignment was August 1, 2019. AEMO’s mission is to coordinate the Army’s national marketing and advertising strategy; develop and maintain relationships with the marketing and advertising industry; and develop marketing expertise and talent within the Army to support the Army, Army National Guard, and Army Reserve accessions. The offices will be moved from Arlington, Virginia, and Fort Knox, Kentucky, to Chicago, Illinois. Consistent with the key practice to designate a leader to be responsible for the implementation of the proposed reforms, the Secretary of the Army designated the Assistant Secretary of the Army for Manpower and Reserve Affairs as responsible for establishing AEMO and overseeing the transition. The Assistant Secretary stated that he expected AEMO to be fully operational by early 2020. The Army identified several reasons for transitioning from AMRG to AEMO and reassigning the office to the Deputy Chief of Staff of the Army, G-1, consistent with the key practice to define and articulate a succinct and compelling reason for the reforms. According to the execution order establishing AEMO, the Army needed an organization strategically positioned to: support Army senior leadership in advertising, marketing, and analysis; coordinate with the Army’s primary advertising agency; be talent diverse; provide effective marketing analysis; and be able to provide consistency of message and brand across the Army accessions enterprise. The Assistant Secretary of the Army for Manpower and Reserve Affairs also stated that AEMO is being assigned to the Deputy Chief of Staff of the Army, G-1, in part, because of the continuity in leadership that having a military officer lead the organization will provide. Previously, AMRG was assigned to the Office of the Assistant Secretary of the Army for Manpower and Reserve Affairs, whose leader is politically appointed and had been vacant for two years until January 2019. In addition, the Assistant Secretary stated that AEMO will be located in Chicago to increase its coordination with the new advertising agency, which is also headquartered in Chicago. The Assistant Secretary also told us that the Army hoped to recruit civilian staff and to leverage the marketing and advertising expertise at academic and other marketing and advertising institutions in the region. The Army has taken some initial steps to establish AEMO and its operations. The Army has outlined a three-phased plan with specific tasks and associated dates within each of these phases, which is consistent with the key practice to establish implementation goals and a timeline to build momentum and show progress for the reforms. Phase 1, which was to be completed by August 1, 2019, prioritized tasks to initially establish AEMO, such as publishing the Army directive establishing AEMO, issuing the execution order outlining roles and responsibilities for the transition from AMRG to AEMO, and identifying office space in Chicago. Phase 2, which is to be completed between August 1, 2019, and February 1, 2020, includes tasks to transition AEMO to being fully operational, such as establishing new position descriptions and equipping the permanent office space. Lastly, phase 3 identifies those tasks to be implemented after February 1, 2020, when AEMO is fully operational and conducting daily operations, such as updating roles and responsibilities in the Army’s regulation for its marketing and advertising program and developing policies to direct commission military personnel to the office. The plan also identifies offices and officials who are accountable for implementing specific tasks during the transition. The Assistant Secretary of the Army for Manpower and Reserve Affairs told us that the Army established an operational planning team to execute the transition from AMRG to AEMO. The execution order also identifies key stakeholders, including officials from TRADOC, Office of the Chief, Army Reserve, and National Guard Bureau, who are to participate in weekly working group meetings led by the Assistant Secretary of the Army for Manpower and Reserve Affairs. Looking forward, by January 2020 the Army plans to develop metrics to assess the effectiveness of the new AEMO organization, including the purpose, expectations, and desired outcomes, which is consistent with the key practice that calls for clear outcome- oriented goals and performance measures. In addition, the Army has taken some initial steps to establish AEMO’s workforce. As of September 2019, the Army had authorized 53 positions—31 military and 22 civilian—for AEMO and identified a Brigadier General from the Army Reserve with marketing and advertising experience as its leader. Senior Army leadership stated that they expected to fill almost all of the positions with new military and civilian personnel, in part, because the civilian position classifications in AMRG do not generally align with those in AEMO. As of June 2019, the Assistant Secretary of the Army for Manpower and Reserve Affairs told us that the Army was identifying Active Duty, Reserve, and National Guard officers with marketing and advertising education or experience to fill the military positions. In addition, the Assistant Secretary stated that they were working with OPM to develop position descriptions for the AEMO civilian personnel and to identify the skills and expertise needed within AEMO to fulfill its mission, consistent with the key practice to determine if the agency will have the needed resources and capacity, including the skills and competencies, in place for the reorganization. The Army has also established a working group led by the U.S. Army Office of Economic and Manpower Analysis at West Point to develop a new marketing career path that senior Army leadership stated is intended to create a pool of military personnel who could serve in AEMO and other Army accessions organizations in the future. As the Army carries out its steps to fully establish AEMO and reorganize the marketing and advertising program, it will continue to be important for the Army to consider and use the key practices for agency reform efforts to guide the transition. Doing so will help ensure the success of the new marketing and advertising organization. We provided a draft of this report to DOD for review and comment. The Army provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Secretary of the Army. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This appendix summarizes an Army report to Congress on actions taken to improve its marketing and advertising program. Section 599 of the John S. McCain National Defense Authorization Act for Fiscal Year 2019 required the Army to submit a report to Congress that addressed several elements, such as the mitigation and oversight measures implemented to assure improved program return and contract management, and the establishment of a review process to regularly evaluate the effectiveness and efficiency of marketing efforts. The Army submitted the report on February 5, 2019. Table 4 identifies the required elements of the report and the actions that the Army has reported taking to address these elements. This appendix provides a summary of actions that the Army has reported taking to address the recommendations in the April 2018 U.S. Army Audit Agency (AAA) reports about the Army’s marketing and advertising program. One report focused on contract oversight, and the other report focused on what AAA termed “return on investment.” Each report contained seven recommendations, with which the Army Marketing and Research Group (AMRG) concurred. AMRG and AAA officials stated that they have communicated about AMRG’s actions to implement the recommendations and that AAA has provided feedback, as appropriate, on actions taken. AAA officials stated that AAA may conduct a follow-up audit in fiscal year 2020 to determine if the actions have led to improvements in the marketing and advertising program. Table 5 summarizes the recommendations from the AAA report on contract oversight, the actions AMRG has taken to implement them, and the status—as of September 2019—of their implementation as reported by AAA. Table 6 identifies the recommendations from the AAA report on return on investment, the actions AMRG has taken to implement them, and the status—as of September 2019—of their implementation as reported by AAA. In addition to the contact named above, Margaret Best (Assistant Director), Kendall Banks, Timothy J. DiNapoli, Jacob Fender, Alexandra Gonzalez, Amie Lesser, Kristen Kociolek, Steven Lozano, Jonathan Meyer, Eve Nealon, Julia Kennon, Carol Petersen, Richard Powelson, Jerome Sandau, Jared Sippel, and Andrew Stavisky made key contributions to this report.", "summary": "The Army requested nearly $335 million for fiscal year 2020 to conduct marketing and advertising activities intended to increase awareness of Army service and ultimately generate leads for potential recruits. In April 2018, AAA made recommendations in two reports to improve the contract oversight and return on investment of the Army's marketing and advertising program. Further, in May 2018 and October 2018, respectively, the Army and OPM made recommendations to improve the workforce practices and organizational structure of the program. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to review the actions taken to implement AAA's recommendations and the effects of these actions on AMRG's leadership, workforce and business practices, and return on investment. This report assesses the extent to which the Army has taken steps to address recommendations (1) from AAA to improve the contract oversight and measurement of the effectiveness of the Army's marketing and advertising program and (2) from the Army and OPM to improve the workforce practices and organizational structure of the marketing and advertising program. GAO analyzed Army marketing and advertising data from fiscal year 2018; reviewed marketing and advertising plans and guidance; conducted focus groups with AMRG personnel; interviewed cognizant officials; and compared the Army's efforts to GAO-identified best practices. The Army provided technical comments, which GAO incorporated as appropriate. The Army has recently taken steps to improve the oversight of its primary marketing and advertising contract and measurement of the effectiveness of its marketing and advertising program in response to two U.S. Army Audit Agency (AAA) reports. In April 2018, AAA found that the Army Marketing and Research Group (AMRG)—the component responsible for conducting the Army's national-level marketing and advertising program—did not fully evaluate the performance of its contracted advertising agency or track the effectiveness of its marketing and advertising efforts. GAO found that AMRG has taken or is taking actions to address AAA's recommendations: Contract Oversight . AMRG has developed processes for overseeing the advertising agency's performance and services. For example, AMRG developed a form program managers use to validate that proposed advertising services are not already provided through other contracts. Program Effectiveness . AMRG has taken steps in several areas related to revising its strategic marketing goals to support Army recruiting, updating how it assesses marketing and advertising effectiveness, and improving the reliability of data systems. AMRG's steps are consistent with commercial best practices that GAO identified for assessing the effectiveness of advertising, such as identifying outcomes from advertising. The Army has also taken steps to improve the workforce practices and organizational structure of its marketing and advertising program in response to two workforce reviews. The two reviews—by an Army organization and the U.S. Office of Personnel Management (OPM)—found that AMRG, among other things, did not have regular communication throughout its workforce and with its stakeholders, and had a poor workforce climate. AMRG took initial steps to address the reviews' recommendations. The Army then established a new office effective August 2019—the Office of the Chief Army Enterprise Marketing—to replace AMRG and to assume all marketing and advertising activities. Some of the Army's early steps to establish the new office are consistent with key practices for agency reform efforts identified previously by GAO. For example, the Army outlined a three-phased plan with specific tasks and associated dates to fully establish the new office by early 2020 consistent with the key practice to establish implementation goals and a timeline.", "document_type": "gao"}
{"report": "According to OMB, federal agencies reported that they operated 432 data centers in 1998, 2,094 in July 2010, and 9,995 in August 2016. Operating such a large number of centers has been, and continues to be, a significant cost to the agencies. For example, in 2007, EPA estimated that the annual cost for electricity to operate federal servers and data centers across the government was about $450 million. Further, according to the Department of Energy (Energy), a typical government data center has 100 to 200 times the energy use intensity of a commercial building. However, in 2009, OMB reported server utilization rates as low as 5 percent across the federal government’s estimated 150,000 servers. All of these factors contributed to OMB recognizing the need to establish a coordinated, government-wide effort to improve the efficiency, performance, and environmental footprint of federal data center activities. Concerned about the size of the federal data center inventory and the potential to improve the efficiency, performance, and environmental footprint of federal data center activities, OMB’s Federal CIO established FDCCI in February 2010. This initiative’s four high-level goals were to reduce the overall energy and real estate footprint of government data centers; reduce the cost of data center hardware, software, and operations; increase the overall IT security posture of the government; and shift IT investments to more efficient computing platforms and technologies. In February 2010, OMB required all of the agencies participating in the FDCCI to submit a data center inventory and a consolidation plan. In October 2010, OMB also clarified the definition of a data center and noted that, for the purposes of FDCCI, a data center was to be defined as any room used for the purpose of processing or storing data that is larger than 500 square feet and meets stringent availability requirements. Under this definition, OMB reported that agencies had identified 2,094 data centers as of July 2010. However, in 2011, the Federal CIO expanded the definition to include a facility of any size and OMB published its revised definition in March 2012. Based on the revised definition, OMB estimated that there were a total of 3,133 federal data centers in December 2011. In addition, its goal was to consolidate approximately 40 percent, or 1,253 of these data centers, for a savings of approximately $3 billion by the end of 2015. Figure 1 shows data center server racks at SSA’s National Support Center in 2017. The number of federal data centers reported by agencies has continued to grow since 2011. In May 2018, we reported that agencies had collectively identified a total of 12,062 data centers in their inventories as of August 2017—an increase of about 9,000 data centers compared to OMB’s October 2011 estimate. According to the Federal CIO, the increase in the number of data centers was primarily due to the expanded definition of a data center (discussed later in this report) and improved inventory reporting by the agencies. See figure 2 for a depiction of the increase in the number of data centers from 1998 through August 2018. Further, OMB placed greater emphasis on data center optimization to improve the efficiency of federal data centers when it issued memorandum M-13-09 in March 2013. Specifically, OMB stated that, to more effectively measure the efficiency of an agency’s data center assets, agencies would also be measured by the extent to which their primary data centers are optimized for total cost of ownership by incorporating metrics for data center energy, facility, labor, and storage, among other things. Subsequently, in May 2014, OMB issued memorandum M-14-08, which established a set of data center optimization metrics to measure agency progress, along with target values for each metric. All agencies were expected to achieve the target values by the end of fiscal year 2015. Recognizing the importance of reforming the government-wide management of IT, Congress enacted FITARA in December 2014. Among other things, the law required agencies to: Submit to OMB a comprehensive inventory of the data centers owned, operated, or maintained by or on behalf of the agency. Submit, by the end of fiscal year 2016, a multi-year strategy to achieve the consolidation and optimization of the agency’s data centers. The strategy was to include performance metrics that were consistent with the government-wide data center consolidation and optimization metrics. Report progress toward meeting government-wide data center consolidation and optimization metrics on a quarterly basis to OMB’s Administrator of the Office of Electronic Government. In addition, according to FITARA, the Office of Electronic Government at OMB was to: Establish metrics applicable to the consolidation and optimization of data centers (including server efficiency), ensure that agencies’ progress toward meeting government-wide data center consolidation and optimization metrics is made publicly available, review agencies’ inventories and strategies to determine whether they are comprehensive and complete, and monitor the implementation of each agency’s strategy. Develop and make publicly available not later than December 19, 2015, a goal broken down by year for the amount of planned cost savings and optimization improvements to be achieved through FDCCI and, for each year thereafter until October 1, 2020, compare reported cost savings and optimization improvements against those goals. OMB issued memorandum M-16-19 in August 2016 to establish DCOI and included guidance on how to implement the data center consolidation and optimization provisions of FITARA. Among other things, the guidance required agencies to consolidate inefficient infrastructure, optimize existing facilities, improve their security posture, and achieve cost savings. For example, each agency was required to maintain a complete inventory of all data center facilities owned, operated, or maintained by or on its behalf, and measure progress toward defined optimization performance metrics on a quarterly basis as part of its data center inventory submission. OMB’s memorandum also directed each agency to develop a DCOI strategic plan that defined its data center strategy for fiscal years 2016 through 2018. Among other things, this strategy was to include a timeline for agency consolidation and optimization activities, with an emphasis on cost savings and optimization performance benchmarks that the agency could achieve between fiscal years 2016 and 2018. For example, each agency was required to develop cost savings targets due to consolidation and optimization actions and report any realized cost savings. OMB required each agency to publicly post its DCOI strategic plan to its agency-owned digital strategy website by September 30, 2016, and to post subsequent strategic plan updates by April 14, 2017 and April 13, 2018. Further, the memorandum stated that OMB was to maintain a public dashboard (referred to as the IT Dashboard) to display government-wide and agency-specific progress in areas such as planned and achieved data center closures, consolidation-related cost savings, and data center optimization performance information. In this regard, OMB began including data center consolidation and optimization progress information on the IT Dashboard in August 2016. OMB’s memorandum also provided new guidance for the classification of a physical data center and expanded the definition of a data center. According to the revised definition, a room with at least one server that provides services (whether in a production, test, staging, development, or any other environment) should be considered a data center, while a room containing only print servers, routing equipment, switches, security devices (such as firewalls), or other telecommunication components, was not to be considered a data center. In light of this new definition, OMB directed each agency to perform a comprehensive review of its data centers and maintain a complete and updated data center inventory. Further, OMB directed each agency to categorize each of its data centers as either a tiered data center or a non- tiered data center. OMB’s memorandum defined a tiered data center as one that uses each of the following: a separate physical space for IT infrastructure; an uninterruptible power supply; a dedicated cooling system or zone; and a backup power generator for a prolonged power outage. According to the memorandum, all other data centers were to be considered non-tiered. Moreover, OMB guidance included a series of performance metrics in the areas of data center closures, cost savings, and optimization progress. Data center closures: According to the guidance, agencies were to close at least 25 percent of tiered data centers government-wide, excluding those approved as inter-agency shared services providers, by the end of fiscal year 2018. Further, agencies were to close at least 60 percent of non-tiered data centers government-wide by the end of fiscal year 2018. OMB’s guidance further notes that, in the long term, all agencies should continually strive to close all non-tiered data centers, noting that server rooms and closets pose security risks and management challenges and are an inefficient use of resources. Cost savings: According to the guidance, agencies were to reduce government-wide annual costs attributable to physical data centers by at least 25 percent, resulting in savings of at least $2.7 billion for fiscal years 2016 through 2018. Data center optimization: According to the guidance, agencies were to measure progress against a series of new data center performance metrics in the areas of server utilization, energy metering, power usage, facility utilization, and virtualization. Further, OMB’s guidance established target values for each metric that agencies were to achieve by the end of fiscal year 2018. OMB’s guidance further noted that agency progress against these performance metrics was to be measured by OMB on a quarterly basis, using agencies’ data center inventory submissions and OMB-defined closures, cost savings, and optimization targets. In November 2018, OMB published proposed changes to DCOI for public comment. The changes focus federal consolidation and optimization efforts on agencies’ larger, tiered data centers and also de-emphasize the consolidation of non-tiered facilities and other smaller spaces. The draft guidance also revises the classification of data centers and data center optimization metrics. The draft guidance redefines a data center as a purpose-built, physically separate, dedicated space that meets certain criteria. Similarly, OMB does not plan to continue to report on spaces not designed to be data centers. According to the draft, OMB also plans to work with agencies to set agency-specific goals for data center closures and cost savings and to update these targets from those set in OMB’s August 2016 memorandum to match agencies’ current status and progress. Additionally, the proposed changes to DCOI make several changes to the metrics currently used by agencies to monitor the performance of their data centers. Specifically, of the five metrics currently in use (described in detail later in this report), OMB proposes updating three, removing two, and adding one new metric. The draft guidance states that public comments will be collected through the end of December 2018, but does not provide a date for when the proposed changes will be finalized and implemented. However, the draft does state that the new guidance will sunset on September 30, 2020, a date that coincides with the extension of FITARA’s data center provisions. Since the enactment of FITARA in December 2014, we have reviewed and verified annually for the quality and completeness of each agency’s (covered by the law) inventory and DCOI strategy. We have also published reports documenting the findings from each of our reviews. In addition, we have examined and reported on agencies’ efforts to optimize their data centers, as well as the challenges encountered and successes achieved. In a report that we issued in March 2016, we noted that agencies had reported significant data center closures—totaling more than 3,100 through fiscal year 2015—with the Departments of Agriculture, Defense (Defense), the Interior (Interior), and the Treasury (Treasury) accounting for 84 percent of the total. Although the agencies fell short of OMB’s fiscal year 2015 consolidation goal, their plans identified about 2,100 additional centers planned for closure through fiscal year 2019. Agencies also reported significant consolidation cost savings and avoidances—totaling about $2.8 billion through fiscal year 2015 and expected to increase to over $8.0 billion in future years. The Departments of Commerce, Defense, Homeland Security (DHS), Transportation (Transportation), and the Treasury accounted for 96 percent of the total planned savings. However, we pointed out that many agencies lacked complete cost savings goals for the next several years despite having closures planned. In addition, we reported that 22 agencies had made limited progress against OMB’s fiscal year 2015 data center optimization performance metrics, such as the utilization of data center facilities. Accordingly, we recommended that the agencies take actions to complete their cost savings targets and improve optimization progress. As of December 2018, 18 of the 32 recommendations from this report had yet to be fully addressed. In May 2017, we reported that the agencies were reporting significant data center closures—totaling more than 4,300 through August 2016— with Agriculture, Defense, Interior, and the Treasury accounting for 84 percent of the total. The agencies’ plans for 2016 had identified more than 1,200 additional centers for closure through fiscal year 2019. Agencies also reported significant consolidation and optimization cost savings and avoidances, which totaled about $2.3 billion through August 2016. However, reductions in the amount of achieved savings reported to OMB, particularly by the Treasury, resulted in a net decrease of more than $400 million in these savings, compared to amounts we previously reported as planned in 2015. Further, our report noted that, as of December 2016, agencies’ total planned cost savings of about $656 million were more than $3.3 billion less compared to the amounts that we reported in 2015, and more than $2 billion less than OMB’s fiscal year 2018 cost savings goal of $2.7 billion. This reduction in planned savings was the result of eight agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans compared to the savings amounts previously reported to us in November 2015. The reduction also reflected the absence of cost savings information for one agency (Defense) that did not submit its strategic plan in time for our review. In addition, our May 2017 report identified weaknesses in agencies’ DCOI strategic plans. Of the 23 agencies that had submitted their strategic plans at the time of our review, 7 agencies—Agriculture, the Department of Education (Education), DHS, and the Department of Housing and Urban Development (HUD); GSA; the National Science Foundation (NSF); and the Office of Personnel Management (OPM)—had addressed all five required elements of a strategic plan, as identified by OMB (such as providing information related to data center closures and cost savings metrics). The remaining 16 agencies that submitted their plans either partially met or did not meet the requirements. We also pointed out that there were inconsistencies in the reporting of cost savings in the strategic plans of 11 agencies. Given these findings, we recommended that OMB improve its oversight of agencies’ DCOI strategic plans and their reporting of cost savings and avoidances. We also recommended that 16 agencies and Defense (which did not submit a plan in time for our review) complete the missing elements in their strategic plans, and that 11 agencies ensure the reporting of consistent cost savings and avoidance information to OMB. As of December 2018, 10 of the 30 recommendations had not been fully addressed. In a subsequent report that we issued in August 2017, we noted that 22 of the 24 agencies required to participate in the OMB DCOI had reported (collectively) limited progress against OMB’s fiscal year 2018 performance targets for the five optimization metrics. The 2 remaining agencies, Education and HUD, did not own any data centers and, therefore, did not have a basis to report on progress. Specifically, for each of the five targets, no more than 5 agencies reported that they had met or exceeded that specific target. We reported that this limited progress against OMB’s optimization targets was due, in part, to agencies not fully addressing our prior recommendations in this area. In addition, we noted in the report that most agencies had not yet implemented automated monitoring tools to measure server utilization, as required by the end of fiscal year 2018. Specifically, 4 agencies reported that they had fully implemented such tools, 18 reported that they had not yet done so, and 2 did not have a basis to report on progress because they did not own any data centers. Accordingly, we recommended that OMB require that agencies include plans, as part of existing OMB reporting mechanisms, to implement automated monitoring tools at their agency-owned data centers. We also recommended that the 18 agencies that did not have fully documented plans take action, within existing OMB reporting mechanisms, to complete plans describing how they intended to achieve OMB’s requirement to implement automated monitoring tools at all agency-owned data centers by the end of fiscal year 2018. As of December 2018, none of our 19 recommendations had been fully addressed. Most recently, in May 2018, we noted that the 24 agencies participating in DCOI had reported mixed progress toward achieving OMB’s goals for closing their data centers by September 2018. Thirteen agencies reported that they had either already met, or planned to meet, all of their OMB- assigned goals by the deadline. However, 4 agencies reported that they did not have plans to meet all of their assigned goals and 2 agencies were working with OMB to establish revised targets. With regard to agencies’ progress in achieving cost savings, 20 agencies reported, as of August 2017, that they had achieved $1.04 billion in cost savings for fiscal years 2016 and 2017. In addition, the agencies’ DCOI strategic plans identified an additional $0.58 billion in planned savings— for a total of $1.62 billion for fiscal years 2016 through 2018. This total was approximately $1.12 billion less than OMB’s DCOI savings goal of $2.7 billion. This shortfall was the result of 12 agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans, as compared to the savings targets established for them by OMB. In addition, the 24 agencies reported limited progress against OMB’s five data center optimization targets for server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization. As of August 2017, 1 agency had met four targets, 1 agency had met three targets, 6 agencies had met either one or two targets, and 14 agencies reported meeting none of the targets. Further, as of August 2017, most agencies were not planning to meet OMB’s fiscal year 2018 optimization targets. Specifically, 4 agencies reported plans to meet all of their applicable targets by the end of fiscal year 2018; 14 reported plans to meet some of the targets; and 4 reported that they did not plan to meet any targets. Because GAO had made a number of recommendations to OMB and the 24 DCOI agencies to help improve the reporting of data center-related cost savings and to achieve optimization targets, we did not make new recommendations and noted that, as of March 2018, 74 of the 81 prior recommendations had not been fully addressed. While agencies have made considerable progress, as of December 2018, 47 of the 81 recommendations had not been fully addressed. According to OMB guidance, agencies were expected to close at least 25 percent of tiered data centers government-wide, by the end of fiscal year 2018. In addition, agencies were to close at least 60 percent of non-tiered data centers government-wide by this same deadline. Further, agencies were expected to reduce government-wide annual costs attributable to physical data centers by a least 25 percent by the end of fiscal year 2018, resulting in savings of at least $2.7 billion. The 24 agencies reported mixed results regarding their data center closure progress and plans, when compared with OMB’s goal for each agency to close at least 25 percent of their tiered data centers and at least 60 percent of their non-tiered centers. Specifically, as of August 2018, 13 agencies reported that they had already met the goal of closing 25 percent of their tiered data centers, another 3 agencies reported that they planned to meet the goal by the end of fiscal year 2018, and 6 agencies reported that they did not plan to meet the goal. Further, as of August 2018, 11 agencies reported that they had already met the goal for closing 60 percent of their non-tiered centers, 3 agencies reported that they planned to meet the goal by the end of fiscal year 2018, and 9 agencies reported that they did not plan to meet the goal by the end of fiscal year 2018. Table 1 displays a breakdown of the number of reported tiered and non-tiered data centers and completed and planned closures by agency, as of August 2018. As shown in the figure below, the 24 agencies reported a total of 6,250 data center closures as of August 2018, which represented about half of the total reported number of federal data centers. In addition, the agencies planned 1,009 closures by the end of fiscal year 2018, with an additional 191 closures planned through fiscal year 2023 for a total of 1,200 more closures. This would further reduce the number of open data centers to about 39 percent of the number reported in the agencies’ inventories. Figure 3 provides a summary breakdown of agencies’ data center inventories that were closed, planned for closure, or not planned for closure, as of August 2018. As noted, while about half of the agencies had met, or had planned to meet, their OMB targets as of August 2018, the other half planned to miss one or both of them. Officials from the 11 agencies that did not plan to meet one or both of their closure goals provided various reasons for why they had not planned to do so. For example, several agencies indicated that they were seeking revised closure goals because they viewed their goals as unattainable. Specifically, officials from Interior’s Office of the CIO stated that a number of the department’s non-tiered data centers were either mission-critical or not cost effective to close. Thus, the officials said Interior was working with OMB to establish a revised closure goal. Similarly, Transportation’s Director for IT Compliance stated that the department was working with OMB to establish a revised closure goal. The department reported having 186 tiered data centers in Federal Aviation Administration control towers that it believes should be excluded from its count of data centers when OMB sets the department’s goal for closures. In addition, officials in Defense’s Office of the CIO stated that the OMB closure targets for the department were based on including special purpose processing nodes that are mission critical and, therefore, are not subject to being closed. The officials noted that the department intends to continue operating its enterprise data centers, close its smaller data centers, and work with OMB to remove the special purpose processing nodes from DCOI consideration. When OMB launched DCOI in 2016, agencies originally had until the end of fiscal year 2018 to meet OMB’s stated time frame for closing their data centers. However, the extension of FITARA’s data center consolidation and optimization provisions through fiscal year 2020, and OMB’s planned revisions to DCOI goals, provide the 11 agencies that had not planned to meet one or both of OMB’s closure targets with additional time to meet their goals. Until these agencies take action to close enough data centers to meet OMB’s targets, they may not realize the efficiencies and cost savings that were expected from DCOI. Since 2013, federal agencies have been required to report on data center cost savings, with guidance from OMB regarding how agencies were to report cost savings and avoidances. Specifically, the guidance required agencies to report both data center consolidation cost savings and avoidances, among other areas, as part of a quarterly reporting process. FITARA also called for each agency to submit a multi-year strategy for achieving the consolidation and optimization of data centers that included year-by-year quarterly calculations of investment and cost savings through fiscal year 2018, which has now been extended to 2020. In addition, in August 2016, OMB M-16-19 provided guidance on how agencies should implement the requirements of FITARA. Specifically, agencies were to develop a strategic plan that included information on historical cost savings and avoidances due to data center consolidation and optimization through fiscal year 2015. This guidance stated that agency strategic plans were also to include year-by-year calculations of target and actual agency-wide spending and cost savings on data centers from fiscal years 2016 through 2018. Further, the guidance established a DCOI combined cost savings goal of $2.7 billion for all federal agencies to achieve from fiscal years 2016 through 2018. This overall goal was then broken down into agency-specific targets on the IT Dashboard. In August 2018, 22 agencies reported through the quarterly reporting process that they had achieved $1.94 billion in cost savings for fiscal years 2016 through 2018, while 2 agencies reported that they had not achieved any savings. Further, 21 agencies identified an additional $0.42 billion planned through fiscal year 2018, for a total of $2.36 billion in planned savings from fiscal years 2016 through 2018. Nevertheless, this total is about $0.37 billion less than OMB’s goal of $2.7 billion for overall DCOI savings. Figure 4 compares the total achieved savings as reported by the 24 agencies for fiscal years 2016 through 2018 and the agencies’ additional planned savings through 2018 to OMB’s DCOI savings goal for fiscal years 2016 through 2018. The 24 participating DCOI agencies had achieved $1.94 billion in savings as of August 2018. In addition, agencies identified an additional $0.43 billion, for a difference of $0.37 billion between planned and achieved savings from fiscal years 2016 through 2018. Table 2 provides specific data related to each agency’s total planned savings, total achieved savings, and additional planned savings through 2018. As shown in table 2, 13 agencies reported that they had met or planned to meet or exceed their OMB targets, and 3 agencies that did not have an OMB target also identified achieved savings. In contrast, 5 agencies reported that they did not plan to meet their targets. Three agencies did not have a savings target and did not report any achieved savings. Agencies provided various reasons for why they did not plan to meet their savings targets. For example, the Department of Veterans Affairs (VA) reported that the implementation of its DCOI-related projects in fiscal years 2016 through 2018 was dependent on funding approval and might not result in cost savings and avoidances until later in the projects’ life cycles (i.e., fiscal year 2019 or later). In another example, GSA stated that the OMB target may be difficult for the agency to reach due, in part, to the methods OMB used to set the target for fiscal years 2016 through 2018. According to GSA, OMB used the data that GSA had reported on the IT Dashboard regarding the agency’s expenditure for data center infrastructure and reduced that amount by 25 percent. Agencies have now been working toward OMB’s DCOI savings goals since fiscal year 2016; however, almost half of the agencies are still not planning to meet OMB’s targets. Until agencies plan to meet and achieve OMB’s data center-related savings targets, they will likely not realize the expected financial benefits from DCOI. FITARA required OMB to establish metrics to measure the optimization of data centers, including server efficiency, and to ensure that agencies’ progress toward meeting the metrics is made public. Pursuant to FITARA, OMB established a set of five data center optimization metrics intended to measure agencies’ progress in the areas of server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization. According to OMB, while the server utilization and automated monitoring metric applied to agency-owned tiered and non-tiered data centers, the four remaining metrics applied only to agency-owned tiered centers. OMB’s memorandum also established a target value for each of the five metrics, which agencies were expected to achieve by the end of fiscal year 2018. OMB measures agencies’ progress against the optimization targets using the agencies’ quarterly data center inventory submission and publicly reports this information on its IT Dashboard. Table 3 provides a description of the five data center optimization metrics and target values. As of August 2018, most (22 of the 24) DCOI agencies continued to report limited progress in meeting OMB’s fiscal year 2018 data center optimization targets identified on the IT Dashboard. The remaining 2 agencies—Education and HUD—reported that they did not have any agency-owned data centers in their inventory and, therefore, did not have a basis to measure and report optimization progress. With regard to the data center optimization targets, agencies reported the greatest progress against two metrics: power usage effectiveness and virtualization metrics. Specifically, 8 agencies reported that they had met OMB’s target for power usage effectiveness and 6 agencies reported that they had met the target for virtualization. However, for the energy metering, facility utilization, and server utilization and automated monitoring metrics, no more than 3 agencies reported meeting each. Figure 5 summarizes the 24 agencies’ progress in meeting each optimization target, as of August 2018. As of August 2018, NSF, SSA, and EPA reported the most progress against OMB’s metrics among the 22 agencies with a basis to report— each met 3 targets. Nine agencies reported that they had met only one target, and 10 agencies reported they had not met any of the targets. Further, OMB began requiring the implementation of automated monitoring tools in August 2016; however, as of August 2018, of the 22 agencies with a basis to report, 5 reported that they had either not implemented the tools at any data centers, or had experienced shortcomings in their implementation. For example, the Department of State (State) reported that it had limited centralized monitoring capability and is installing automated monitoring tools in several phases. Thus, these 5 agencies were not able to report any progress against either or both of the server utilization or power usage effectiveness metrics because their data centers lacked the required monitoring tools to measure progress in these areas. The remaining 17 agencies reported that they had implemented the tools in at least one data center. Table 4 depicts the performance of the agencies in meeting OMB targets for data center optimization, as of August 2018. As of August 2018, multiple agencies had made changes to their data center inventory and operational environment, such as closing all agency- owned tiered data centers or implementing automated monitoring tools. These changes impacted which metrics were applicable or an agency’s ability to report on the status of its optimization metrics. For example, GSA reported that it no longer had any agency-owned tiered data centers, and, therefore, did not have a basis to report on four of the five optimization metrics. Additionally, NSF, which previously had only owned one non-tiered data center, migrated from the non-tiered center to a tiered data center as part of its headquarters relocation. Accordingly, NSF began reporting on the metrics applicable to its tiered facility. Further, the Nuclear Regulatory Commission (NRC) did not report on power usage effectiveness due to delays in awarding a new contract that was to include monitoring tools that would impact the ability to report on this metric. Agency officials stated that NRC plans to have the monitoring tools in place during fiscal year 2019. Overall, these changes since last year’s report have resulted in no significant changes to the progression of these agencies on their optimization metrics. In addition, agencies’ limited progress against OMB’s optimization targets was due, in part, to not fully addressing our prior recommendations in this area. As previously mentioned, in March 2016, we reported on weaknesses in agencies’ data center optimization efforts, including that 22 agencies did not meet OMB’s fiscal year 2015 optimization targets. We noted that this was partially due to the agencies having challenges in optimizing their data centers, including in their decentralized organizational structures that made consolidation and optimization difficult, and in competing priorities for resources. In addition, consolidating certain data centers was problematic because the volume or type of information involved required the data center to be close in proximity to the users. Accordingly, we recommended that the agencies take action to improve optimization progress, to include addressing any of the identified challenges. Most agencies agreed with our recommendations or had no comments. However, as of December 2018, only 4 of the 22 agencies had fully addressed them. The continuing shortcomings in data center optimization can also be attributed, in part, to agencies viewing OMB’s optimization metric targets as unrealistic. For example, Transportation stated in its DCOI strategic plan that it could not meet multiple optimization metrics due to funds not being available and competing priorities. In addition, Treasury indicated in its DCOI strategic plan that it struggles to report on automated monitoring because many of its data centers do not have the ability to centrally aggregate and report on central processing unit data. Further, DHS officials noted that it has 7 smaller tiered data centers where it has determined that it is not cost effective to equip those centers with the tools needed to report on metrics such as power usage effectiveness. Given these types of challenges, the targets for each optimization metric may not be realistic for every agency. Unless agencies take action to meet the applicable OMB optimization metrics, their data centers may not operate efficiently enough to provide expected cost savings. In addition to reporting current optimization progress on the IT Dashboard, OMB required agencies to include in their DCOI strategic plans planned performance levels for fiscal year 2018 for each optimization metric. However, according to the 24 agencies’ DCOI strategic plan information as of August 2018, only 2—Commerce and the U. S. Agency for International Development (USAID)—reported plans to fully meet their applicable targets by the end of fiscal year 2018. Of the remaining agencies, 14 reported plans to meet some, but not all, of the targets; 6 reported that they did not plan to meet any targets, and—as already discussed—Education and HUD did not have a basis to report planned optimization milestones because they did not report having any agency- owned data centers. Figure 6 summarizes agencies’ progress, as of August 2018, in meeting OMB’s optimization targets and planned progress to be achieved by September 2018. At the time of our review, only two agencies planned to meet all of their applicable targets, and it was doubtful that the agencies would be able to achieve OMB’s collective optimization target of at least $2.7 billion in cost savings by the end of fiscal year 2018. Until the remaining agencies take the steps necessary to meet their optimization targets, it is unlikely that these agencies will achieve the expected benefits of optimization and the resulting cost savings. As we noted previously in this report, many agencies have reported challenges that have hindered their efforts to meet OMB’s DCOI targets. However, a number of agencies have also reported success in meeting OMB’s targets ahead of DCOI’s end of fiscal year 2018 deadline. As noted in our methodology section, six agencies that were among the best performers in achieving data center closures, cost savings, and optimization performance reported a number of key practices that had contributed to their success. These practices were: obtaining executive leadership support for consolidation and using experiences and lessons learned to refine consolidation increasing the use of cloud and shared services to consolidate or optimize data center operations; emphasizing closing data centers to meet OMB targets and achieve increasing the use of virtualization to optimize data centers; and employing an organization-wide communications plan to facilitate adoption of consolidation and optimization activities. Five of the six agencies (Agriculture, Commerce, Justice, EPA, and GSA) reported that their success in consolidation and optimization activities was due to obtaining support from executive leadership for the agency’s consolidation efforts. Each agency obtained sponsorship and support from its executive leadership (e.g., Deputy Secretary or agency CIO), such as through a memorandum or policy that directed all agency offices to participate in, or comply with, the consolidation effort. For example, The Deputy Secretary for Agriculture issued a memorandum in 2017 that, among other things, declared the department’s intent to consolidate from 39 data centers down to 2 by the end of 2019. According to officials in the Office of the CIO, this memorandum from the Secretary’s office focused all data center owners on the same project task of reducing the data center inventory. The Commerce CIO and the department’s CIO Council provided overall governance through organizational policies, processes, and procedures for the department’s data center consolidation effort. Leveraging this departmental guidance, each component of Commerce developed its own consolidation plan that identified specific approaches and activities. Using these plans, the department and its components focused on reducing spending on redundant software, infrastructure, and data center operations. The Deputy Attorney General issued a memorandum in 2014 to the heads and CIOs of all components. This memorandum formally established Justice’s Data Center Transformation Initiative, established the Department Program Review Board to provide oversight for the initiative, and also directed the consolidation of all data centers into 3 enterprise facilities. In addition, Justice’s CIO issued a memorandum to component CIOs that provided additional details on how to execute planned activities and established further governance associated with the initiative. These memoranda provided clear leadership buy-in and support for the department’s data center consolidation and optimization activities that could be used to resolve any challenges or issues at the departmental level. EPA attributed much of its DCOI success to a top-down approach from its CIO office, saying that such support was critical to achieve data center closures. For example, EPA leadership decided to adopt and enforce geographical consolidation of data centers within major areas to minimize costs of consolidation while still meeting closure objectives. In doing so, the agency leadership provided clear direction and support for the agency’s consolidation effort by adopting the strategy to consolidate data centers within specific geographic regions. GSA reported that it obtained leadership commitment that made its data center consolidation and optimization activities a priority. The agency noted that having strong CIO and executive leadership was important for sponsoring technology modernization. As a result of the buy-in, the agency reported that it had minimized resistance to change and improved acceptance of its consolidation and optimization activities. Four agencies (Agriculture, Commerce, Justice, and SSA) reported that their success with consolidation and optimization activities was due to the use of a refined consolidation plan or process. Each of these agencies developed an initial consolidation plan or process for closing data centers, and then refined their procedures based on their experiences and lessons learned as data centers were closed. For example, Agriculture developed a set of streamlined processes to facilitate DCOI closures that were based on the experiences gained from successful data center closures under FDCCI. The set of processes consisted of 5 steps: The planning step included the discovery and documentation of all data center assets, including applications and IT hardware, in a given data center. In addition, this step involved identifying the necessary resources to move the applications and associated data to a target data center. The preparation step included identification of the target data center and development of a project schedule. The data migration step included moving both applications and data to the target data center or cloud-services, as planned. The testing step included ensuring the applications and data that were moved were integrated into the target data center, and functional testing to ensure that the applications worked and data was accessible. The application cutover step included putting the migrated applications and data into operation and closing the original data center. Using and refining this set of processes allowed the department to become more efficient in closing its data centers. After closing 46 data centers in fiscal years 2011 through 2014, the department closed 2,185 data centers over the next 2 years. In total, Agriculture reported that it had closed 2,253 data centers as of August 2018. Commerce established departmental guidance and then each departmental component leveraged that guidance to develop its own consolidation plan. The plans identified specific approaches and activities intended to achieve the stated goals and milestones. According to Commerce, the department and its components leveraged their IT planning processes and established IT governance to, among other things, reduce spending on redundant commodity software, infrastructure, and operations. Justice’s Office of the CIO developed a master plan for the department’s data center consolidation effort in June 2015. The plan included a planning framework, transformation approach, and a master schedule for data center moves and closures. It also included process steps similar to those used by Agriculture. Further, Justice’s plan noted that the department would use its initial closure efforts to gain experience and to refine its plans. Justice reported that it used the plan’s schedule and semi-monthly progress reports to ensure that consolidation activities stayed on schedule, or the department could make adjustments as needed. As a result, the department closed 84 of its 110 data centers and achieved more than $128 million in cost savings and avoidances as of August 2018. SSA used a project management framework process and controls that it believed efficiently addressed requirements, critical path, and risk management. In addition, SSA reported that it used an incremental development approach to its data center optimization plans, with each project expected to accomplish specific tasks that would lead to another project. Accordingly, SSA noted that the agency used a multi-year plan with many initiatives focused on specific goals. Using this approach, the agency successfully moved SSA’s operations and infrastructure from an older facility to the newly-built National Support Center. The agency reports that this facility is state-of-the-art and provides similar capabilities and efficiencies to major cloud service providers. Three agencies (Commerce, GSA, and SSA) also attributed their success in consolidation and optimization activities to increasing their agency’s use of cloud and shared services. In doing so, each agency emphasized the move of data center assets and systems to cloud services to optimize their data centers and reduce costs. For example, Commerce identified moving to cloud services and utilizing shared services as being most effective in closing data centers. As an example, the department cited the National Oceanic and Atmospheric Administration’s (NOAA) “cloud-first” policy that emphasized using cloud services rather than an agency-owned physical data center whenever feasible. The agency attributed its ability to handle increased traffic as an operational benefit of its increased use of cloud services. For example, NOAA did not have the capacity in its agency- owned facilities to meet the computing demands and requirements of a sudden increase in web traffic on the websites for NOAA and the National Hurricane Center, such as during Hurricanes Irma and Harvey in 2017. Commerce stated that using cloud services allowed NOAA to handle 4.7 billion page hits during Hurricane Harvey over a 6-day span, ensuring the websites were not adversely impacted by the increase in traffic. GSA reported that it focused on moving services from agency-owned tiered and non-tiered data centers to cloud services or to shared centers. As a result, GSA had closed 118 data centers as of August 2018, including all of the agency’s tiered centers. SSA developed an agency cloud initiative that encourages the adoption of cloud technologies as part of the agency’s infrastructure modernization. The agency reported that it is employing a hybrid cloud strategy that is comprised of both private cloud and public cloud services for the agency’s back office applications. By doing so, the agency will consolidate and standardize SSA’s IT infrastructure systems and software to simplify management of those resources and reduce costs. Three agencies (Agriculture, Justice, and EPA) reported that their success in consolidation and optimization activities was due to focusing on the closure of data centers. In doing so, they emphasized the importance of closing data centers to reduce costs and achieve cost savings and avoidances. For example, Agriculture determined that the costs to improve DCOI performance metrics in its agency-owned data centers were prohibitive. Accordingly, the department decided that the only viable alternative was to close data centers to remove underperforming centers and improve optimization metrics performance and reduce costs. As a result, Agriculture reported that it had closed 2,253 data centers through August 2018. In addition, the department reported that it had improved its security posture, reduced its real estate footprint, and achieved realized cost savings and avoidance of $51.8 million from fiscal year 2012 through 2018. Justice reported that it took a practical approach to selecting the data centers that would remain as its enterprise facilities, considering factors such as the number of physical servers that could be eliminated, the efficiency of the remaining hardware, and potential labor savings. The department reported that it focused on retaining more efficient data centers (e.g., those with more efficient use of electricity or virtualization), rather than simply keeping its biggest existing data centers. As a result, Justice has closed 84 of its 110 data centers and achieved more than $128 million in cost savings and avoidances as of August 2018. EPA identified geographical consolidation as its best approach to meeting DCOI goals. Specifically, in its data center consolidation plan, the agency stated that, for geographic areas where it had multiple data centers, a single facility was identified into which data center IT assets would be consolidated. Using this approach, EPA had closed 43 of its 83 data centers as of August 2018. Three agencies (Commerce, EPA, and SSA) reported that their success in consolidation and optimization activities also was due to focusing on the increased use of virtualization to run more software on the same or a reduced amount of servers. In doing so, the agencies expected to reduce costs by avoiding the purchase of additional servers to meet computing demands or eliminating unnecessary hardware and floor space in their data centers. For example, Commerce focused on moving systems from physical hardware to virtual servers, as part of its component offices’ plans to update technology and in cases where the systems did not require a specific type of server. Using this approach, the department reported that it had reduced the number of physical servers in its data centers, and was working to improve server utilization. The department also cited the ability to automatically increase or decrease computing capability through virtualization, such as when NOAA handled the increased traffic to its hurricane-related web pages during Hurricanes Irma and Harvey in 2017. EPA used the agency’s data center consolidation plan to implement an agency-wide “physical-to-virtual” policy that required offices to convert existing physical servers to virtual servers wherever possible. The agency also defined server and software standards for virtualized platforms. SSA reported that the agency’s goal, using its “Virtual 1st” policy, was to have failover capability within the data center, disaster recovery capability for both data centers, and balanced load capacity between data centers. The agency reported that it has continued to virtualize not only servers but storage and network applications, as well. For example, SSA stated that it has taken steps to virtualize as much storage as possible and used similar techniques to reduce the physical hardware footprint on the data center floor, as well as power, cooling, and network bandwidth requirements. Two agencies (Justice and GSA) reported that their success in consolidation and optimization activities was due to employing an organization-wide communications plan. In doing so, the agencies adopted a structured method for communicating with agency offices to improve acceptance and adoption of consolidation and optimization activities. This also facilitated conflict resolution. For example, Justice reported that it prioritized communications related to its Data Center Transformation Initiative and established an all-encompassing approach to initiative-related communications. To help communicate all related directives, strategies, plans, statuses, and accomplishments, the department used a variety of methods that included: regular meetings to share information, a dedicated email box to provide easy communication for answers or information, without the need to know specific individuals, an intranet web page that provided general information, instructions, templates, decisions, status information, and accomplishments related to the initiative; and email broadcasts on an as-needed basis. GSA reported that it communicated and collaborated frequently with business stakeholders to identify the best time frames to move systems, stagger transfers to minimize impact, and determine which systems could be virtualized. The agency indicated that these important factors required continuous communication between system owners, system administrators, and business leadership. As a result, the agency experienced minimal staff resistance to change and a commitment to reach a consensus on moving forward with the agency’s consolidation efforts. The aforementioned practices included elements of sound management techniques, such as gathering leadership support for a project and developing a communications plan to foster adoption of organizational changes. The practices also included activities that aligned with the core tenets of DCOI to consolidate inefficient infrastructure, optimize existing facilities, and achieve cost savings. Further, these practices each proved effective for multiple agencies and, while they were not the only practices that could be effective, they represent concepts that could provide the foundation for an effective data center consolidation and optimization program. Federal data center consolidation efforts have been underway since 2010 and OMB’s fiscal year 2018 targets provided clear and transparent goals that helped define the tangible benefits that DCOI was expected to provide. However, most agencies continue to report mixed progress against those targets. Although agencies have taken action to close about half of the data centers in their combined inventories, 11 agencies did not plan to meet all of their closure targets. Further, the data center closures were expected to drive cost savings and avoidances and, to the agencies’ credit, the closures have led to more than $2.37 billion in planned and achieved cost savings and avoidances from fiscal years 2016 through 2018. However, five agencies did not plan to meet their cost savings targets. Until agencies consolidate the data centers required to meet their targets, as well as identify and report the associated cost savings, they will be challenged to realize expected efficiencies and the full benefits of DCOI will not be fully realized. Similarly, although OMB first established optimization metrics in May 2014, agencies continue to report only limited progress against the current performance targets. While two agencies do not have a basis to report any progress as they do not own any data centers, only two agencies reported that they planned to achieve all of DCOI’s fiscal year 2018 optimization targets. Ensuring the optimized performance of data centers is a key component to meeting OMB’s DCOI-wide savings goal and the 20 agencies that did not have plans to meet their targets call into question whether DCOI will realize its full potential savings. Although many agencies have struggled to meet their individual DCOI targets, other agencies have successfully met OMB’s goals for data center closures, savings, and optimization. Six such agencies that we identified reported on the importance of gathering leadership support, effective communication, and alignment with the core tenets of DCOI. Key practices such as these can play an important role in helping agencies better meet the overall goals and mission of DCOI. We are making a total of 36 recommendations to 22 of the 24 agencies in our review. Specifically: The Secretary of Agriculture should take action to meet the data center optimization metric targets established by OMB under DCOI. (Recommendation 1) The Secretary of Commerce should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 2) The Secretary of Defense should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 3) The Secretary of Defense should identify additional savings opportunities to achieve the targets for data center-related cost savings established under DCOI by OMB. (Recommendation 4) The Secretary of Defense should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 5) The Secretary of Energy should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 6) The Secretary of Energy should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 7) The Secretary of the Department of Health and Human Services (HHS) should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 8) The Secretary of HHS should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 9) The Secretary of DHS should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 10) The Secretary of DHS should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 11) The Secretary of Interior should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 12) The Secretary of Interior should take action to meet the data center- related cost savings established under DCOI by OMB. (Recommendation 13) The Secretary of Interior should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 14) The Attorney General should take action to meet the data center optimization metric targets established for Justice under DCOI by OMB. (Recommendation 15) The Secretary of the Department of Labor (Labor) should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 16) The Secretary of State should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 17) The Secretary of State should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 18) The Secretary of Transportation should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 19) The Secretary of Transportation should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 20) The Secretary of Treasury should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 21) The Secretary of VA should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 22) The Secretary of VA should take action to meet the data center-related cost savings established under DCOI by OMB. (Recommendation 23) The Secretary of VA should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 24) The Administrator of EPA should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 25) The Administrator of EPA should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 26) The Administrator of GSA should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 27) The Administrator of the National Aeronautics and Space Administration (NASA) should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 28) The Director of NSF should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 29) The Chairman of NRC should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 30) The Director of OPM should take action to meet the data center-related cost savings established under DCOI by OMB. (Recommendation 31) The Director of OPM should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 32) The Administrator of the Small Business Administration (SBA) should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 33) The Commissioner of SSA should take action to meet the data center- related cost savings established under DCOI by OMB. (Recommendation 34) The Commissioner of SSA should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 35) The Administrator of USAID should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 36) We requested comments on a draft of this report from OMB and the 24 agencies that we reviewed. Of the 22 agencies to which we made recommendations, 11 agencies agreed with our recommendations; three agencies agreed with some portion, but not all of the recommendations; one agency disagreed with our recommendations; and seven agencies did not state whether they agreed or disagreed with the recommendations. In addition, OMB and two agencies to which we did not make recommendations stated that they had no comments. Further, multiple agencies provided technical comments, which we have incorporated, as appropriate. The following 11 agencies agreed with our recommendations: In written comments from Commerce, State, NASA, SBA, and SSA, the agencies stated that they agreed with the recommendations and indicated their intent to address them. State also provided technical comments, which we have incorporated, as appropriate. The agencies’ comments are reprinted in appendices II through VI. In written comments, Energy agreed with our recommendations to meet its data center closure and optimization metric targets, and described actions that the department planned to take in order to address the recommendations. Energy initially estimated that it would complete these actions by March 1, 2019; however, the department subsequently revised its estimated completion date to April 15, 2019. Energy also provided technical comments, which we have incorporated, as appropriate. Energy’s comments are reprinted in appendix VII. In written comments, VA agreed with our recommendations to meet its data center closure, cost savings, and optimization metric targets. In addition, the department requested that we close our recommendation related to data center closures on the basis of its planned actions to implement a new inventory data collection tool and methodology to improve how the department collects data center inventory information, and a positive trend in its data center closures. The department estimated that its planned actions would be completed in March 2019 and reported that, as of November 2018, it had closed 78 data centers in fiscal year 2018, as compared with 24 in fiscal year 2017. However, as noted earlier in this report, we found that VA did not plan to meet the closure goal for either tiered or non-tiered data centers, which was the basis for our recommendation. While we acknowledge and encourage VA’s reported closure progress, the department still has not met its DCOI closure goals, as we recommended. Further, VA did not provide an update on the status of its planned actions in time for us to address them in this report. As such, we maintain that this recommendation is still appropriate. In addition, VA referred to OMB’s proposed changes to DCOI guidance when describing actions that it planned to take to meet the department’s cost savings and optimization metrics targets. However, OMB staff told us that the August 2016 DCOI guidance will remain in effect until the revised DCOI guidance is formally issued. Once OMB’s new DCOI guidance is finalized, we plan to assess agency progress against any revised targets, and we will continue to monitor the department’s efforts to address our recommendation. VA’s comments are reprinted in appendix VIII. We received emails from officials of Agriculture, Justice, Transportation, and OPM which stated that these agencies agreed with the recommendations we directed to them. In addition, three agencies agreed with some portion, but not all of our recommendations directed to them: In written comments, Defense stated that it agreed with our recommendation to meet its data center closure targets. However, the department partially agreed with our two other recommendations: to identify additional data center-related savings opportunities and to meet OMB’s data center optimization metric targets. In partially agreeing with our recommendation on data center savings, Defense asserted that it had already identified significant cost savings through activities such as the identification of system migration candidates and the use of cloud services, among others. The department further stated that, while it would continue to optimize its data centers, the need for IT would continue to grow, and this growth might ultimately lead to an increase in total data center costs, despite overall per unit cost reductions. However, the department’s planned savings of $205.46 million represented only 11 percent of its $1.8 billion savings goal by the end of fiscal year 2018 and, as such, this limited progress by the department formed the basis for our recommendation. As discussed in our report, OMB plans to revise DCOI guidance and work with agencies to set agency-specific targets. According to OMB staff, until the guidance is revised, the current guidance and its targets are still applicable. For these reasons, we maintain that our recommendation is still appropriate. Further, in partially agreeing with our recommendation to meet optimization metric targets, Defense stated that the department will continue to drive towards the achievement of data center optimization targets. It added, however, that it would not invest resources to improve the efficiency of data centers planned for closure and that, as a result, the composite view of Defense’s data center efficiency would fall short of meeting OMB’s targets. Our review found that Defense did not plan on meeting any of OMB’s five data center optimization metric targets by the end of fiscal year 2018. This finding was the basis for our recommendation. We acknowledge Defense’s position that investing resources into optimizing data centers that are already planned for closure would not be the best use of taxpayer dollars. We also noted in our report that OMB had proposed revising its optimization metrics, and that any such changes had not yet been finalized. Our recommendation is not intended to imply that an agency should meet a particular version of OMB targets but, rather, that the agency should meet any targets that are established by OMB. This would include any future changes to DCOI targets. Accordingly, we maintain that our recommendation is still appropriate and will continue to monitor the department’s efforts to address our recommendation. Defense’s comments are reprinted in appendix IX. In written comments, DHS stated that it agreed with our recommendation to meet its data center closure targets and disagreed with our recommendation to meet its data center optimization metric targets. Specifically, the department noted that it had met its tiered data center closure targets, and was reviewing the status of its remaining open non-tiered data centers. The department added that it expected to complete this activity by March 31, 2019. However, the department did not provide an update on its efforts in time to be included in this report. While we encourage DHS’s continued efforts to close its remaining non-tiered data centers, we note that the department’s letter cites an inventory of 18 open non-tiered facilities, which differs significantly from the 202 non-tiered centers counted in our draft report, and which DHS officials confirmed in November 2018. According to the department, this discrepancy is because OMB issued revised inventory reporting requirements in November 2018, and these revised requirements exempted certain types of facilities from DCOI reporting and resulted in the lower number. These changes in reporting requirements are similar to the proposed, but not yet finalized, revisions to the DCOI policy that are discussed earlier in this report. However, OMB staff told us that the August 2016 DCOI guidance will remain in effect until the revised DCOI guidance is formally issued. Once OMB’s new DCOI guidance is finalized, we plan to assess agency progress against any revised targets, and we will continue to monitor the department’s efforts to address our recommendation. Further, in disagreeing with our recommendation on meeting optimization metrics, the department stated that, while the recommendation was applicable under the original DCOI guidance that OMB issued in August 2016, OMB’s proposed changes to DCOI guidance would exempt most, if not all, DHS agency-owned data centers from the optimization metrics. Consequently, the department requested that our recommendation be closed. In our review, we found that the department did not plan on meeting any of OMB’s five data center optimization metric targets established under DCOI. This finding was the basis for our recommendation on meeting optimization metrics. Also, while OMB has proposed changes to its metrics, as we noted previously, it has not provided a date for when any such proposed changes will be finalized and implemented; and, according to OMB staff, until the changes to DCOI guidance are finalized, the current guidance is still applicable. Further, our recommendations do not specify that an agency should meet any particular version of OMB targets, but rather, that an agency should meet the targets established by OMB. This would include any future changes to DCOI targets. Accordingly, we maintain that our recommendation is still appropriate. DHS also provided technical comments, which we have incorporated, as appropriate. DHS’s comments are reprinted in appendix X. In written comments, Interior stated that it partially agreed with our recommendation to meet its data center closure targets and disagreed with our two recommendations to meet its data center-related cost savings target and its data center optimization metric targets. For all three recommendations, the department stated that OMB had proposed changes to DCOI guidance that would result in new targets for closures, cost savings, and optimization metrics and that Interior planned to adopt the new policy and work through OMB to establish its new targets. As noted in our report, Interior met its target for tiered data center closures, but did not plan to meet the closure goal for non-tiered data centers. Further, the department planned on achieving only $15.95 million of its $88.19 million savings target (18 percent) by the end of fiscal year 2018, and did not plan on meeting any of OMB’s five data center optimization metric targets. These three findings were the basis for our recommendations to the department. We also noted that, as part of OMB’s proposed changes to DCOI guidance, it planned to work with agencies to set agency-specific targets for data center closures and planned to modify the metrics currently used by agencies to monitor the performance of their data centers. However, as previously mentioned, OMB has not provided a date for when these proposed changes will be finalized and implemented and, according to OMB staff, until the changes to DCOI guidance are finalized, the 2016 guidance is still applicable. Furthermore, our recommendations do not specify that an agency should meet any particular version of OMB targets, but should meet any targets that are established by OMB. This would include any future changes to DCOI targets. As such, we maintain that our recommendations are appropriate. Interior’s comments are reprinted in appendix XI. One agency disagreed with all of our recommendations: In written comments, HHS disagreed with our two recommendations to meet its data center closure targets and data center optimization metric targets. In regard to both recommendations, the department disagreed with being held to what it termed “expired requirements” from DCOI guidance, pending the assignment of new targets being established by OMB. As noted in our report, HHS met its target for tiered data center closures, but did not plan to meet the closure target for non-tiered data centers. We also found that HHS did not meet any of OMB’s five optimization metric targets and had planned to meet only one of the five by end of fiscal year 2018. These findings were the basis for the two recommendations that we made to the department. We also noted that, as part of OMB’s proposed changes to DCOI guidance, OMB planned to work with agencies to set agency-specific targets for data center closures and planned to modify the metrics currently used by agencies to monitor the performance of their data centers. However, as previously mentioned, OMB did not provide a date for when these proposed changes will be finalized and implemented and, according to its staff, until the changes to DCOI guidance are finalized, the current guidance is still applicable. Further, our recommendations do not specify that an agency should meet any particular version of OMB targets, but rather, that the agency should meet the targets established by OMB. This would include any future changes to DCOI targets. Accordingly, we maintain that our recommendations are still appropriate. HHS’s comments are reprinted in appendix XII. Further, seven agencies did not agree or disagree with the recommendations: In written comments, EPA did not state whether it agreed or disagreed with our recommendations to meet its data center closure and data center optimization metrics targets. However, the agency requested that we close our recommendations, citing its reported progress in closing 21 of 34 targeted data centers and OMB’s proposed changes in its draft DCOI guidance that could result in revised closure targets and optimization metrics. As stated in our report, we found that EPA did not plan to meet its closure target for tiered or non-tiered data centers, nor did it plan to meet its data center optimization targets; these findings were the basis for our recommendations. We also noted that, as part of OMB’s proposed changes to DCOI, OMB planned to work with agencies to set agency-specific targets for data center closures and planned to modify the metrics currently used by agencies to monitor the performance of their data centers. However, OMB has not provided a date for when these proposed changes will be finalized and implemented and, according to OMB staff, until the changes to DCOI guidance are finalized, the current guidance is still applicable. Further, our recommendations do not specify that an agency should meet any particular version of OMB targets, but that it should meet the targets established by OMB. This would include any future changes to DCOI targets. Accordingly, we maintain that our recommendations are appropriate and should remain open. EPA also provided technical comments, which we have incorporated, as appropriate. The agency’s comments are reprinted in appendix XIII. In written comments, GSA did not state whether it agreed or disagreed with our recommendation to meet the agency’s data center optimization metrics targets. Specifically, the agency stated that it had complied with revised inventory reporting requirements, which OMB provided to agencies in November 2018 and which eliminated non- tiered data centers from the requirement to meet optimization targets. As a result, the agency noted that it no longer had a basis to measure and report on the one metric our report cited as applicable to GSA (i.e., server utilization and automated monitoring) and asked that we withdraw the recommendation. These changes in reporting requirements are similar to the proposed, but not yet finalized, revisions to the DCOI policy that are discussed earlier in the report. However, OMB staff told us that the August 2016 DCOI guidance is still in effect until the revised DCOI guidance is formally issued. Until OMB’s new DCOI guidance is finalized and agency progress against any revised targets can be evaluated, we maintain that our recommendation to meet the agency’s optimization metrics targets is appropriate, and we will continue to monitor the agency’s efforts to address it. GSA’s comments are reprinted in appendix XIV. In written comments, NSF did not state whether it agreed or disagreed with our recommendation. The agency’s comments are reprinted in appendix XV. In written comments, NRC agreed with the draft report, but did not state whether it agreed or disagreed with our recommendation. The agency’s comments are reprinted in appendix XVI. In written comments USAID did not state whether it agreed or disagreed with the draft report’s recommendation but agreed with our finding that the agency no longer had any tiered data centers. However, USAID stated that it had met DCOI’s closure targets for the agency by closing its 4 non-tiered data centers, and requested that we close our recommendation to meet those targets. While we encourage USAID’s continued efforts to close its remaining non-tiered data centers, we note that the agency’s letter cites an inventory of 4 non-tiered facilities, which differs significantly from the 83 non-tiered centers counted in our draft report, and which USAID officials confirmed in October 2018. As USAID communicated in subsequent emails, this discrepancy is because OMB issued revised inventory reporting requirements in November 2018 and these revised requirements exempted certain types of facilities from DCOI reporting, which resulted in the lower number. These changes in reporting requirements are similar to the proposed, but not yet finalized, revisions to the DCOI policy that are discussed earlier in the report. However, OMB staff told us that the August 2016 DCOI guidance is still in effect until the revised DCOI guidance is formally issued. Until OMB’s new DCOI guidance is finalized and agency progress against any revised targets can be evaluated, we maintain that our data center closure recommendation is appropriate, and we will continue to monitor the agency’s efforts to address it. The agency’s comments are reprinted in appendix XVII. In emails received from Labor’s GAO liaison in the department’s Office of the Assistant Secretary for Policy on January 8, 2019, and from an audit liaison in Treasury’s Office of the CIO on February 1, 2019, both departments did not state whether they agreed or disagreed with our respective recommendations. Finally, in emails received from a Management and Program Analyst in Education’s Office of the Secretary/Executive Secretariat on January 8, 2019; an audit liaison in HUD’s Office of the CIO, Audit Compliance Branch on February 15, 2019; and a GAO liaison in OMB’s Office of General Counsel on February 25, 2019, these agencies stated that they had no comments on the draft report. We are sending copies of this report to interested congressional committees, the Director of OMB, the secretaries and heads of the departments and agencies addressed in this report, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XVIII. Our objectives for this engagement were to (1) determine agencies’ progress in data center closures and achievement in related savings to date and describe plans for future savings, (2) evaluate the agencies’ progress against OMB’s data center optimization targets, and (3) identify effective agency practices for achieving data center closures, cost savings, and optimization. To address the first objective, for data center closures, we obtained and analyzed August 2018 data center inventory documentation from the 24 departments and agencies (agencies) that participate in OMB’s Data Center Optimization Initiative (DCOI). To determine data center closures to date, we totaled their reported closures from fiscal year 2010 through August 2018 and to identify future closures, we totaled their reported planned closures through fiscal year 2018. We also compared agencies’ completed and planned closures to OMB’s fiscal year 2018 consolidation goals, as documented in its August 2016 memorandum (M-16-19). To verify the quality, completeness, and reliability of each agency’s data center inventory, we compared information on completed and planned data center closures to similar information reported on OMB’s IT Dashboard—a public website that provides information on federal agencies’ major IT investments. We also checked for missing data and other errors, such as missing closure status information. In some cases identified, we followed-up with agency officials to obtain further information. We determined that the data were sufficiently complete and reliable to report on agencies’ consolidation progress and planned closures. For cost savings and avoidance we obtained and analyzed documentation from the 24 DCOI agencies. This documentation is required by OMB’s March 2013 and August 2016 memorandums and included the agencies’ quarterly reports of cost savings and avoidances posted to their digital services websites and their DCOI strategic plans. To determine cost savings achieved, we totaled agencies’ reported savings and avoidances from the start of fiscal years 2012 through August 2018, as found in the August 2018 quarterly reports posted to the agencies’ digital services websites. To identify future planned savings, we totaled the agencies’ projected savings and avoidances from fiscal years 2016 through 2018, as reported in their DCOI strategic plans. To assess the quality, completeness, and reliability of each agency’s data center consolidation cost savings information, we used the latest version of each agency’s quarterly cost savings report and DCOI strategic plan, as of August 2018. We also reviewed the quarterly reports and DCOI strategic plans for missing data and other errors, such as missing cost- savings information. In addition, we compared agencies cost savings and avoidances with data from our most recent data center consolidation report. As a result, we determined that the data were sufficiently complete and reliable to report on agencies data center consolidation cost-savings information. For our second objective, we analyzed the August 2018 data center optimization progress information of the 24 DCOI agencies. This progress information was obtained from the IT Dashboard—an OMB public website that provides information on federal agencies’ major IT investments. To assess agencies’ planned optimization progress, we obtained the planned optimization performance from the 22 agencies’ DCOI strategic plans. We then compared the agencies’ current and planned optimization progress information to OMB’s fiscal year 2018 optimization targets, as documented in its August 2016 memorandum. Although OMB’s memorandum establishes a single optimization target value for the server utilization and automated monitoring metric, the IT Dashboard displays agencies’ progress for tiered and non-tiered data centers separately. To report consistently with OMB’s implementation memorandum, we combined the progress information for tiered and non-tiered data centers into a single assessment in this report. In addition, to assess the reliability of the planned optimization milestones in the DCOI strategic plans, we reviewed agencies’ documentation to identify any missing or erroneous data. We also compared the planned data center optimization milestones contained in agencies’ documentation against current optimization progress information obtained from the IT Dashboard; we then discussed any discrepancies or potential errors that we identified with agency officials to determine the causes or request additional information. As a result of these efforts, we were able to determine whether each agency’s strategic plan information was sufficiently reliable for reporting on plans to meet or not meet OMB’s fiscal year 2018 optimization targets. To assess the reliability of agencies’ optimization progress information on OMB’s IT Dashboard, we reviewed the information for errors or missing data, such as progress information that was not available for certain metrics. We also compared agencies’ optimization progress information across multiple reporting quarters to identify any inconsistencies in agencies’ reported progress. We discussed with staff from OMB’s Office of the Federal Chief Information Officer any discrepancies or potential errors identified to determine the causes. To identify effective agency practices for achieving data center closures, cost savings, and optimization progress, we selected two of the highest performing departments or agencies for each of those three data center areas that we reported on in our May 2018 report. For the data center inventory closures area, we selected the Departments of Agriculture (Agriculture) and Justice (Justice) from among the five agencies that had, as of August 2017, reached or exceeded both their tiered and non-tiered data center closure targets for the end of fiscal year 2018. For the cost savings area, we identified two departments and two small agencies reporting the highest cost savings DCOI to date, as of August 2017. From those, we selected one department (Commerce) and one small agency (the General Services Administration) to provide balance relative to agency size. For effective practices related to optimization performance, we reviewed agencies’ reported optimization performance as of August 2017 and selected the two highest-performing agencies in this area (the Social Security Administration and the Environmental Protection Agency), since they were the only two agencies reporting that they met more than half of OMB’s optimization targets. Selecting these agencies was designed to provide anecdotal information that could assist agencies struggling with DCOI implementation. The examples they provided are not findings nor should they be taken to be representative of all the agencies participating in DCOI. We asked each selected agency to identify practices that they found effective in implementing DCOI at their agency and in meeting OMB’s established targets in each of the areas, not just the area for which they were selected. We also solicited examples that demonstrated how those practices helped agency implementation or the benefits from implementing DCOI. Additionally, we considered information and examples that these agencies provided as part of our work to identify FITARA best practices. We analyzed the responses to determine the practices and reported those that were identified by at least two agencies. We conducted this performance audit from April 2018 to April 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, individuals making contributions to this report included Dave Powner (director), Dave Hinchman (assistant director), Justin Booth (analyst-in-charge), Alexander Bennett, Chris Businsky, Nancy Glover, and Jonathan Wall.", "summary": "In December 2014, Congress enacted federal IT acquisition reform legislation that included provisions related to ongoing federal data center consolidation efforts. OMB's Federal Chief Information Officer launched DCOI to build on prior data center consolidation efforts; improve federal data centers' performance; and establish goals for inventory closures, cost savings and avoidances, and optimizing performance. The 2014 legislation included a provision for GAO to annually review agencies' data center inventories and strategies. Accordingly, GAO's objectives were to (1) evaluate agencies' progress and plans for data center closures and cost savings; (2) assess agencies' progress against OMB's data center optimization targets; (3) and identify effective agency practices for achieving data center closures, cost savings, and optimization progress. To do so, GAO assessed the 24 DCOI agencies' data center inventories as of August 2018; reviewed their reported cost savings documentation; evaluated their data center optimization strategic plans; and assessed their progress against OMB's established optimization targets. GAO also solicited practices that selected agencies reported to be effective in meeting DCOI goals. The 24 agencies participating in the Office of Management and Budget's (OMB) Data Center Optimization Initiative (DCOI) reported mixed progress toward achieving OMB's goals for closing data centers and realizing the associated savings by September 2018. As of August 2018, 13 agencies reported that they had met, or had plans to meet, all of their OMB-assigned closure goals by the deadline. However, 11 agencies reported that they did not have plans to meet their goals. Further, 16 agencies reported that, as of August 2018, they had met, or planned to meet, their cost savings targets, for a total of $2.36 billion in cost savings for fiscal years 2016 through 2018. This is about $0.38 billion less than OMB's DCOI savings goal of $2.7 billion. This shortfall is the result of 5 agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans, as compared to their savings targets established for them by OMB. Three agencies did not have a cost savings target and did not report any achieved savings. In addition, the 24 agencies reported limited progress against OMB's five data center optimization targets for server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization. As of August 2018, the agencies reported that 3 had met three targets, 9 had met one target, and 10 met none of the targets. Two agencies did not have a basis to report on progress as they do not own any data centers. Further, as of August 2018, 20 agencies did not plan to meet all of OMB's fiscal year 2018 optimization goals. Specifically, only 2 agencies reported plans to meet all applicable targets; 6 reported that they did not plan to meet any of the targets (see figure). We selected 6 agencies that had demonstrated success towards meeting their DCOI goals and those agencies reported a number of key practices that contributed to their efforts. The officials noted the importance of, among other things, obtaining executive leadership support for consolidation and optimization activities, employing an organization-wide communications plan, and focusing on data center closures. The officials also cited the use of past experience and lessons learned to inform improvements to future consolidation plans and processes. GAO is making 36 recommendations to 22 agencies to improve performance against established DCOI goals. Eleven agencies agreed with the recommendations, three did not fully agree, one disagreed, and seven neither agreed nor disagreed, as discussed in the report.", "document_type": "gao"}
{"report": "TAs are significant given their increasing importance to policymakers, and the growing effects of S&T on society, economy, and other areas. While technological changes can be positive, they can also be disruptive. Therefore, it is critical for Congress to be able to understand and evaluate these changes, to ensure, for example, national security and global competitiveness. Examples of potential uses of TAs related to enhancing knowledge and awareness to assist decision-making include: Highlight potential short, medium, and long-term impacts of a Elaborate on and communicate the risks and benefits associated with a technology, including early insights into the potential impacts of technology Highlight the status, viability, and relative maturity of a technology Plan and evaluate federal investments in S&T GAO TAs are most commonly requested by congressional committees, which may use them to, among other things, make decisions regarding allocating or reallocating resources to address research gaps, support updated rulemaking for a regulatory agency, or inform a legislative agenda or the development of a national strategy. Technologies present opportunities and challenges that may vary, depending in part on the policy context in which they are evaluated. Therefore, part of a TA is considering the policy context surrounding a given technology. GAO may, where appropriate, identify and analyze policy options as part of its TAs, which may also include: clarifying and summarizing policy-related issues and challenges, and providing information that can be used for decision-making. In this situation, policy options can be defined as a set of alternatives or menu of options (including the status quo) that policymakers, such as legislative bodies, government agencies, and other groups, could consider taking. Policy options can be used to articulate a range of possible actions a policymaker could consider in the context of a given technology and policy goal. Policy options do not state what policymakers should do in a given circumstance with a certain technology. Policy options do not endorse or recommend a particular course of action; they are not recommendations or matters for congressional consideration, which GAO makes in its audits. In addition, policy options are addressed to policymakers more broadly, and are not addressed to a specific federal agency or entity. Developing a written TA design helps TA teams agree on and communicate a clear plan of action to the project team and the team’s advisers, requesters, and other stakeholders. Written TA designs also help guide and coordinate the project team’s activities and facilitate documentation of decisions and procedures in the final report. In addition, focusing the TA on answering specific researchable questions can assist teams to define and select the appropriate scope, approach, and type of product, ensuring usefulness of the product to the intended users. More specific reasons for spending time on systematically designing a TA include: Enhance its quality, credibility, and usefulness Ensure independence of the analysis Ensure effective use of resources, including time Data collection and quality assurance of data can be costly and time- consuming. A thorough consideration of design options can ensure that collection and analysis of the data are relevant, sufficient, and appropriate to answer the researchable question(s), and helps to mitigate the risk of collecting unnecessary evidence and incurring additional costs. This chapter highlights design phases, cross-cutting considerations, and GAO TA design examples for sound technology assessment (TA) design. To ensure that the information and analyses in TAs meet policymakers’ needs, it is particularly useful to outline the phases and considerations involved in sound TA design, while remaining aware of the iterative and nonlinear process of designing a TA. The information presented in this chapter is based on review of results of a literature search, an expert forum, select GAO reports, and experiences of GAO teams and technical specialists. For more information, please refer to Appendix I: Objectives, Scope, and Methodology. Below are questions to consider for a sound TA design. Reflecting on these questions may help teams make important decisions (like selecting an appropriate design) and ensure quality TAs. Does the design address the needs of the congressional requester? Will the design yield a quality, independent, balanced, thorough, and objective product? Will the design likely yield information that will be useful to stakeholders? Will the design likely yield valid conclusions on the basis of sufficient and credible evidence? Will the design yield results in the desired time frame? Will the design likely yield results within the constraints of the resources available? How will policy options be identified and assessed, if applicable? Figure 1 outlines three phases and seven considerations for TA design. While Figure 1 presents TA design as a series of phases, actual execution is highly iterative and nonlinear. Teams may need to be prepared to re-visit design decisions as information is gathered or circumstances change. Below are some considerations for the team to think about while designing a TA and throughout the process of performing the TA. This list is not exhaustive, and some of the considerations may not be unique to TAs. of the technology) and context of the technology (such as social, political, legal, and economic factors) circumstances change and new information comes to light, it may be necessary to revisit scope and design. The initial situational analysis may also be used to: Inform the goal(s), purpose, and objectives (also known as researchable questions) challenges to design and implementation of the TA, such as: (1) possible changes in operating environment; (2) characterizing or quantifying anticipatory factors, uncertainty, and future condition(s); and (3) lack of or limitations with data. See Chapter 3 for more specific examples. Communication strategy: Consider potential users of the product(s) and how information regarding the TA will be communicated. How results are communicated can affect how they are used, so it is important for TA teams to discuss communication options. statement. TA teams will need to think about whether the initial policy options are appropriate to the size and scope of the TA, as well as whether they are in line with the policy goal and the overall TA purpose and objectives. In keeping with the iterative nature of TA design and execution, any initial policy option list will be revisited, modified, or refined, as needed, as the work progresses and more information is gained. TA teams may also need to plan to include policy analysis and exploration of the ramifications of each policy option during subsequent design and implementation phases. During this phase, TA teams continue to build on the situational analysis work and gather more background information. In addition, TA teams: Confirm and validate the scope from phase 1 Reach agreement with stakeholders on the initial design May perform an “environmental scan” to further highlight limitations, assumptions, divergent points of view, potential bias, and other factors that may help the team select a design Other specific activities that take place during this phase include: Identify and select appropriate design, methodologies, and analytical approaches (refer to the next section of this chapter for example TA design approaches and App. III for examples of TA methods) Examples of data collection and analytical techniques used in GAO TAs to date include: interviews, literature review, expert forums, site visits, technology readiness assessments, surveys, conceptual models, small group discussion, content analysis such as Delphi, among others. OTA reported using similar methodologies for its TAs (OTA, Policy Analysis at OTA: A Staff Assessment, 1983). Identify and select appropriate data sources, or the need to gather data Identify, select, and possibly develop appropriate dimensions of analysis, if applicable Develop possible policy goal(s) Clarify the possible initial policy options that will be considered and describe how they may be analyzed, if applicable Identify and consult with external experts to inform design and implementation, and assist with external review, as appropriate If policy options are being considered, it is important to determine the relevant dimensions along which to analyze the options. The dimensions will be highly context- specific, vary from TA to TA, and depend on the scope and policy goal statement of the TA. During this phase, the design and project plan are being implemented, potentially while aspects of phase 2 are still underway. It is important to consider changes in the operating context—such as changes in the operating environment, understanding of the issues, and access to information—and review and make changes to the design and project plan accordingly. We reviewed select GAO products that used policy analysis to present policy options. We found that these products used a variety of data collection and analytical approaches, such as: interviews, literature review, survey, expert forum, site visits, case studies, analysis of secondary data, content analysis, among others. If an initial policy options list was developed earlier in design, it may be necessary to revisit the list as work progresses. During this phase, TA teams may gather additional information regarding the policy options, further analyze policy options, and present the results of the analysis. Policy options are to be presented in a balanced way, including presentation of opportunities and considerations, and not resulting in a single overall ranking of policy options. We found that GAO TAs used a variety of design approaches and methodologies to answer various categories of design objectives (researchable questions). GAO TAs generally include one or more of the following categories of design objectives, which are not mutually exclusive: (1) describe status of and challenges to development of a technology; (2) assess opportunities and challenges arising from the use of a technology; and (3) identify and assess cost-effectiveness, other policy considerations, or options related to the use of a technology. Provided below are example questions, design approaches, and GAO TAs, for each of these categories of objectives. GAO TA examples were used given our familiarity with GAO products, though numerous non-GAO TA design examples exist. This is not intended to be a comprehensive list of design examples. For more examples of methodologies, please refer to App. III. Describing the status and challenges to the development of a technology. Table 2 provides example questions, design approaches, and GAO TAs, for design objectives related to describing the status and challenges to the development of a technology. Questions may address, for example, what the current state of the technology is, and may involve identifying and describing the status of the technology, which GAO TAs have done using a variety of methods. Assessing opportunities and challenges that may result from the use of a technology. Table 3 provides example questions, design approaches, and GAO TAs, for design objectives related to assessing opportunities and challenges that may result from the use of a technology. Questions may address, for example, what are the expected or realized benefits of the technology, and may involve gathering and assessing evidence on the results from using the technology, which GAO TAs have done using a variety of methods. Assessing cost-effectiveness, policy considerations, or policy options related to the use of a technology. Table 4 provides example questions, design approaches, and GAO TAs, for design objectives related to assessing cost-effectiveness, policy considerations, or policy options related to the use of a technology. Questions may address, for example, what are the economic trade-offs of a technology, and may involve gathering and analyzing evidence related to cost, which GAO TAs have done using a variety of methods. This chapter describes select challenges regarding technology assessment (TA) design and implementation, as well as possible strategies to mitigate those challenges. The information in this chapter is based on review of results of a literature search, an expert forum, select GAO reports, and experiences of GAO teams and technical specialists. The tables provided below are not intended to be a comprehensive list of challenges or strategies. For more information, please refer to Appendix I: Objectives, Scope, and Methodology. To be useful, TA assessment products must be readable and timely, among other things, which may present a challenge for numerous reasons. Table 5 provides examples of potential mitigation strategies to address these challenges. Another challenge in TA design arises from determining policy goals and policy options, and estimating their potential impacts. Many of the effects of policy decisions may be distant, and policy outcomes may be uncertain at the time of the TA. Table 6 provides examples of potential mitigation strategies to address these challenges. TAs are complex and interdisciplinary, and emerging technologies are inherently difficult to assess. Table 7 provides examples of potential mitigation strategies to address these challenges. An additional challenge in conducting TAs is engaging all relevant internal and external stakeholders, ensuring none are overlooked. Table 8 provides examples of potential mitigation strategies to address this challenge. This handbook identifies key steps and considerations in designing technology assessments (TAs). Below is a summary of methodologies used for all chapters of the handbook. We reviewed GAO documents, including: Designing Evaluations (GAO-12-208G) select GAO products utilizing policy analysis approaches to identify and assess policy options We reviewed and analyzed 14 GAO TAs, including their designs and considerations, using a data collection instrument that contained fields regarding each report’s purpose, methodologies, and key considerations for each methodology used (such as strengths and weaknesses). The data collection instrument also contained fields regarding whether policy considerations were presented or if specific policy options were identified and assessed in each TA report, what methodologies were used to identify and assess policy options, and key considerations associated with the methodologies used. We also reviewed GAO reports from non-TA product lines that utilized policy analysis approaches to assess policy options. An initial pool of 56 GAO reports was generated based on a keyword search of GAO’s reports database. Of the 56 GAO reports, 12 were selected for review based on the following criteria: (1) the reports were publicly released after January 1, 2013 and (2) the reports included identification and assessment of policy options (not solely a presentation of agency actions related to policy options or general policy considerations). Testimonies and correspondence were excluded. We analyzed each of these selected GAO reports according to a data collection instrument that contained the following fields regarding policy options in the report: purpose, methodologies, and key considerations for each methodology used (such as strengths and weaknesses). A list of GAO documents reviewed is provided below. Retirement Security: Some Parental and Spousal Caregivers Face Financial Risks. GAO-19-382. Washington, D.C.: May 1, 2019. GAO Science Technology Assessment, and Analytics Team: Initial Plan and Considerations Moving Forward. Washington, D.C.: April 10, 2019. Retirement Savings: Additional Data and Analysis Could Provide Insight into Early Withdrawals. GAO-19-179. Washington, D.C.: March 28, 2019. Critical Infrastructure Protection: Protecting the Electric Grid from Geomagnetic Disturbances. GAO-19-98. Washington, D.C.: December 19, 2018. Postal Retiree Health Benefits: Unsustainable Finances Need to Be Addressed. GAO-18-602. Washington, D.C.: August 31, 2018. Data Collection Seminar Participant Manual. Washington, D.C.: March 2018. Artificial Intelligence: Emerging Opportunities, Challenges and Implications. GAO-18-142SP. Washington, D.C.: March 28, 2018. Chemical Innovation: Technologies to Make Processes and Products More Sustainable. GAO-18-307. Washington, D.C.: February 8, 2018. Federal Regulations: Key Considerations for Agency Design and Enforcement Decisions. GAO-18-22. Washington, D.C.: October 19, 2017. Medical Devices: Capabilities and Challenges of Technologies to Enable Rapid Diagnoses of Infectious Diseases. GAO-17-347. Washington, D.C.: August 14, 2017. U.S. Postal Service: Key Considerations for Potential Changes to USPS’s Monopolies. GAO-17-543. Washington, D.C.: June 22, 2017. Internet of Things: Status and Implications of an Increasingly Connected World. GAO-17-75. Washington, D.C.: May 15, 2017. Flood Insurance: Comprehensive Reform Could Improve Solvency and Enhance Resilience. GAO-17-425. Washington, D.C.: April 27, 2017. Flood Insurance: Review of FEMA Study and Report on Community- Based Options. GAO-16-766. Washington, D.C.: August 24, 2016. Medicaid: Key Policy and Data Considerations for Designing a Per Capita Cap on Federal Funding. GAO-16-726. Washington, D.C.: August 10, 2016. Municipal Freshwater Scarcity: Using Technology to Improve Distribution System Efficiency and Tap Nontraditional Water Sources. GAO-16-474. Washington, D.C.: April 29, 2016. GAO Memorandum: Quality Assurance Framework Requirements for Technology Assessments. Washington, D.C.: April 6, 2016. Biosurveillance: Ongoing Challenges and Future Considerations for DHS Biosurveillance Efforts. GAO-16-413T. Washington, D.C.: February 11, 2016. Social Security’s Future: Answers to Key Questions. GAO-16-75SP. Washington, D.C.: October 2015. Water in the Energy Sector: Reducing Freshwater Use in Hydraulic Fracturing and Thermoelectric Power Plant Cooling. GAO-15-545. Washington, D.C.: August 7, 2015. Nuclear Reactors: Status and Challenges in Development and Deployment of New Commercial Concepts. GAO-15-652. Washington, D.C.: July 28, 2015. Veterans’ Disability Benefits: Improvements Needed to Better Ensure VA Unemployability Decisions Are Well Supported. GAO-15-735T. Washington, D.C.: July 15, 2015. Debt Limit: Market Response to Recent Impasses Underscores Need to Consider Alternative Approaches. GAO-15-476. Washington, D.C.: July 9, 2015. Temporary Assistance for Needy Families: Potential Options to Improve Performance and Oversight. GAO-13-431. Washington, D.C.: May 15, 2013. Private Pensions: Timely Action Needed to Address Impending Multiemployer Plan Insolvencies. GAO-13-240. Washington, D.C.: March 28, 2013. Designing Evaluations: 2012 Revision. GAO-12-208G. Washington, D.C.: January 2012. Neutron Detectors: Alternatives to Using Helium-3. GAO-11-753. Washington, D.C.: September 3, 2011. Climate Engineering: Technical Status, Future Directions, and Potential Responses. GAO-11-71. Washington, D.C.: July 28, 2011. Technology Assessment: Explosives Detection Technologies to Protect Passenger Rail. GAO-10-898. Washington, D.C.: July 28, 2010. Technology Assessment: Protecting Structures and Improving Communications during Wildland Fires. GAO-05-380. Washington, D.C.: April 26, 2005. Technology Assessment: Cybersecurity for Critical Infrastructure Protection. GAO-04-321. Washington, D.C.: May 28, 2004. Technology Assessment: Using Biometrics for Border Security. GAO-03-174. Washington, D.C: November 15, 2002. We spoke with and gathered input from GAO teams that are in the process of or have successfully assessed and incorporated policy options into GAO products. In addition, to augment our understanding of TA design and implementation challenges, we collected input from GAO staff who had provided key contributions to GAO TAs. Specifically, we asked for their thoughts regarding: (1) the strengths and limitations of TA methodologies and (2) challenges they faced, and strategies to address those challenges. A GAO librarian performed a search for relevant Office of Technology Assessment (OTA) reports, using keyword searches. From this initial list of OTA reports, we selected 17 reports to review that were frameworks, guides, models, or other compilations. We also reviewed the methodologies of the OTA reports selected for review. A list of OTA reports reviewed is included below. Office of Technology Assessment. Insider’s Guide to OTA. Washington, D.C.: January 1995. Office of Technology Assessment. Policy Analysis at OTA: A Staff Assessment. Washington, D.C.: May 1993. Office of Technology Assessment. Research Assistants Handbook. Washington, D.C.: June 1992. Office of Technology Assessment. Strengths and Weaknesses of OTA Policy Analysis. Washington, D.C.: 1992. Office of Technology Assessment. The OTA Orange Book: Policies and Procedures of the Office of Technology Assessment: Communication with Congress and the Public. Washington, D.C.: February 1986. Office of Technology Assessment. What OTA Is, What OTA Does, How OTA Works. Washington, D.C.: March 1983. Office of Technology Assessment. Draft: An OTA Handbook. Washington, D.C.: June 7, 1982. Office of Technology Assessment. Draft: A Management Overview Methodology for Technology Assessment. Washington, D.C.: February 2, 1981.* Office of Technology Assessment. Draft: Technology Assessment in Industry: A Counterproductive Myth. Washington, D.C.: January 30, 1981.* Office of Technology Assessment. Draft: Technology Assessment Methodology and Management Practices. Washington, D.C.: January 12, 1981.* Office of Technology Assessment. Draft: Technology Assessment in the Private Sector. Washington, D.C.: January 9, 1981.* Office of Technology Assessment. Draft: A Process for Technology Assessment Based on Decision Analysis. Washington, D.C.: January 1981.* Office of Technology Assessment. Draft: Technology as Social Organization. Washington, D.C.: January 1981.* Office of Technology Assessment. A Summary of the Doctoral Dissertation: A Decision Theoretic Model of Congressional Technology Assessment. Washington, D.C.: January 1981.* Office of Technology Assessment. Report on Task Force Findings and Recommendations: Prepared by the OTA Task Force on TA Methodology and Management. Washington, D.C.: August 13, 1980. Office of Technology Assessment. Phase I Survey Results: Draft Papers Prepared for the Task Force on TA Methodology and Management. Washington, D.C.: April 10, 1980. We identified a pool of 29 Congressional Research Service (CRS) reports to consider reviewing that were technology assessments or included an analysis of policy options, based on a keyword search of CRS’s website. We also interviewed CRS officials. Of the initial 29 CRS reports we identified, we selected six CRS reports to review, based on the following criteria: (1) published within the past 15 years (2004-2019) and (2) if a review of technology (technology assessment) and/or policy options was included. Reports were excluded based on the following criteria: (1) for technology assessment related reports—if they represented a summary of a technology assessment that was included in our review or (2) for policy options related reports—the report did not indicate how CRS arrived at the policy options (no methodology to review or analyze). A list of CRS reports reviewed is included below. Congressional Research Service. Advanced Nuclear Reactors: Technology Overview and Current Issues. Washington, D.C.: April 18, 2019. Congressional Research Service. Drug Shortages: Causes, FDA Authority, and Policy Options. Washington, D.C.: December 27, 2018. Congressional Research Service. Policy Options for Multiemployer Defined Benefit Pension Plans. Washington, D.C.: September 12, 2018. Congressional Research Service. Shale Energy Technology Assessment: Current and Emerging Water Practices. Washington, D.C.: July 14, 2014. Congressional Research Service. Carbon Capture: A Technology Assessment. Washington, D.C.: November 5, 2013. Congressional Research Service. Energy Storage for Power Grids and Electric Transportation: A Technology Assessment. Washington, D.C.: March 27, 2012. A GAO librarian performed a literature search based on keyword searches for two areas—TA and policy options. For TA literature, the team selected 29 documents to review that were frameworks, guides, models, or other compilations, based on a review of the literature titles and abstracts. In general, we excluded specialized types of TAs, such as health-related TAs, as we focused on TA design more broadly. For policy options literature, the team selected 14 documents to review that were frameworks, guides, models, or other compilations and focused on policy options related to science and technology. We also asked experts we consulted to suggest literature for our review; these suggestions confirmed the literature list noted below. A list of literature reviewed is included below. Grunwald, Armin. Technology Assessment in Practice and Theory. London and New York: Routledge, 2019. Armstrong, Joe E., and Willis W. Harman. Strategies For Conducting Technology Assessments. London and New York: Routledge, 2019. Noh, Heeyong, Ju-Hwan Seo, Hyoung Sun Yoo, and Sungjoo Lee. “How to Improve a Technology Evaluation Model: A Data-driven Approach.” Technovation, vol. 72/73 (2018): p. 1-12. Larsson, A., T. Fasth, M. Wärnhjelm, L. Ekenberg, and M. Danielson. “Policy Analysis on the Fly With an Online Multicriteria Cardinal Ranking Tool.” Journal of Multi-Criteria Decision Analysis, vol. 25 (2018): p. 55-66. Nooren, P., N. van Gorp, N. van Eijk, and R. O. Fathaigh. “Should We Regulate Digital Platforms? A New Framework for Evaluating Policy Options.” Policy and Internet, vol. 10, no. 3 (2018): p. 264-301. Smith, A., K. Collins, and D. Mavris. “Survey of Technology Forecasting Techniques for Complex Systems.” Paper presented at 58th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Grapevine, TX (2017). Ibrahim, O., and A. Larsson. “A Systems Tool for Structuring Public Policy Problems and Design of Policy Options.” Int. J. Electronic Governance, vol. 9 , nos. 1/2 (2017): p. 4-26. Christopher, A. Simon. Public Policy Preferences and Outcomes. 3rd ed. New York: Routledge, 2017. Weimer, David L., and R. Aidan Vining. Policy Analysis Concepts and Practice. 6th ed. London and New York: Routledge, 2017. Mulder, K. “Technology Assessment.” In Foresight in Organizations: Methods and Tools, edited by Van Der Duin, Patrick, 109-124, 2016. Coates, Joseph F. “A 21st Century Agenda for Technology Assessment.” Technological Forecasting and Social Change, vol. 113 part A (2016): p. 107-109. Coates, Joseph F. “Next Stages in Technology Assessment: Topics and Tools.” Technological Forecasting and Social Change, vol. 113 (2016): p. 112-114. Mazurkiewicz, A., B. Belina, B. Poteralska, T. Giesko, and W. Karsznia. “Universal Methodology for the Innovative Technologies Assessment.” Proceedings of the European Conference on Innovation and Entrepreneurship (2015): p. 458-467. Sadowski, J. “Office of Technology Assessment: History, Implementation, and Participatory Critique.” Technology in Society, vol. 42 (2015): p. 9-20. Larsson, A., O. Ibrahim. “Modeling for Policy Formulation: Causal Mapping, Scenario Generation, and Decision Evaluation.” In Electronic Participation: 7th IFIP 8.5 International Conference, 135-146, Springer, 2015. Moseley, C., H. Kleinert, K. Sheppard-Jones, and S. Hall. “Using Research Evidence to Inform Public Policy Decisions.” Intellectual and Developmental Disabilities, vol. 51 (2013): p. 412-422. Calof, J., R. Miller, and M. Jackson. “Towards Impactful Foresight: Viewpoints from Foresight Consultants and Academics.” Foresight, vol. 14 (2012): p. 82-97. Parliaments and Civil Society in Technology Assessment, Collaborative Project on Mobilization and Mutual Learning Actions in European Parliamentary Technology Assessment. The Netherlands: Rathenau Instituut, 2012. Blair, P. D. “Scientific Advice for Policy in the United States: Lessons from the National Academies and the Former Congressional Office of Technology Assessment.” In The Politics of Scientific Advice: Institutional Design for Quality Assurance, ed. Lentsch, Justus, 297-333, 2011. Paracchini, M.L., C. Pacini, M.L.M. Jones, and M. Pérez-Soba. “An Aggregation Framework to Link Indicators Associated With Multifunctional Land Use to the Stakeholder Evaluation of Policy Options.” Ecological Indicators, vol. 11 (2011): p 71-80. Roper, A. T., S. W. Cunningham, A. L. Porter, T. W. Mason, F. A. Rossini, and J. Banks. Forecasting and Management of Technology, 2nd ed. New Jersey: Wiley, 2011. Lepori, B., E. Reale, and R. Tijssen. “Designing Indicators for Policy Decisions: Challenges, Tensions and Good Practices: Introduction to a Special Issue.” Research Evaluation, vol. 20, no. 1 (2011): p. 3-5. Russel, A. W., F. M. Vanclay, and H. J. Aslin H.J. “Technology Assessment in Social Context: The Case for a New Framework for Assessing and Shaping Technological Developments.” Impact Assessment and Project Appraisal, vol. 28, no. 2 (2010): p. 109-116. Shiroyama, H., G. Yoshizawa, G., M. Matsuo, and T. Suzuki. “Institutional Options and Operational Issues in Technology Assessment: Lessons from Experiences in the United States and Europe.” Paper presented at Atlanta Conference on Science and Innovation Policy, Atlanta, 2009. Tran, T.A., and T. Daim T. “A Taxonomic Review of Methods and Tools Applied in Technology Assessment.” Technological Forecasting and Social Change, vol. 75 (2008): p. 1396-1405. Brun, G., and G. Hirsch Hadorn. “Ranking Policy Options for Sustainable Development.” Poiesis Prax, vol. 5 (2008): p. 15-31. Tran, T.A. “Review of Methods and Tools applied in Technology Assessment Literature.” Paper presented at Portland International Conference on Management of Engineering and Technology, Portland Oregon, 2007. Burgess, J., A. Stirling, J. Clark, G. Davies, M. Eames, K. Staley, and S. Williamson. “Deliberative Mapping: A Novel Analytic-Deliberative Methodology to Support Contested Science-Policy Decisions.” Public Understanding of Science, vol. 16 (2007): p. 299-322. Decker, M., and M. Ladikas. Bridges Between Science, Society and Policy: Technology Assessment — Methods and Impacts. Berlin: Springer-Verlag, 2004. Guston, D. H., and D. Sarewitz. “Real-time Technology Assessment.” Technology in Society, vol. 24 (2002): p. 93-109. Rip, A. “Technology Assessment.” In International Encyclopedia of the Social & Behavioral Science, vol. 23, edited by Smelster, N. J. and B. P. Baltes, 15512-15515. Amsterdam: Elsevier, 2001. Van Den Ende, J., K. Mulder, M. Knot, E. Moors, and P. Vergragt. “Traditional and Modern Technology Assessment: Toward a Toolkit.” Technological Forecasting and Social Change, vol. 58 (1998): p. 5-21. Wood, F. B. “Lessons in Technology Assessment: Methodology and Management at OTA.” Technological Forecasting and Social Change, vol. 54 (1997): p. 145-162. Janes, M. C. “A Review of the Development of Technology Assessment.” International Journal of Technology Management, vol. 11, no. 5-6 (1996): p. 507-522. Hastbacka, M. A., and C. G. Greenwald. “Technology Assessment - Are You Doing it Right?” Arthur D. Little – PRISM, no. 4 (1994). Rivera, W. M., D. J. Gustafson, and S. L. Corning. “Policy Options in Developing Agricultural Extension Systems: A Framework for Analysis.” International Journal of Lifelong Education, vol. 10, no. 1 (1991): p. 61-74. Lee, A. M., and P. L. Bereano. “Developing Technology Assessment Methodology: Some Insights and Experiences.” Technological Forecasting and Social Change, vol. 19 (1981): p. 15-31. Porter, A. L., F. A. Rossini, S. R. Carpenter, and A. T. Roper. A Guidebook for Technology Assessment and Impact Analysis, vol. 4. New York and Oxford: North Holland, 1980. Pulver, G.C. “A Theoretical Framework for the Analysis of Community Economic Development Policy Options.” In Nonmetropolitan Industrial Growth and Community Change, edited by Summers, G. and A. Selvik, 105-117. Massachusetts and Toronto: Lexington Books, 1979. Ascher, W. “Problems of Forecasting and Technology Assessment.” Technological Forecasting and Social Change, vol. 13, no. 2 (1979): p. 149-156. Majone, G. “Technology Assessment and Policy Analysis.” Policy Sciences, vol. 8, no. 2 (1977): p. 173-175. Berg, M., K. Chen, and G. Zissis. “A Value-Oriented Policy Generation Methodology for Technology Assessment.” Technological Forecasting and Social Change, vol. 4, no. 4 (1976): p. 401-420. Lasswell, Harold D. A Pre-View of Policy Sciences. Policy Sciences Book Series. New York: Elsevier, 1971. We held a forum to gather experts’ opinions regarding TA design. An initial list of experts was prepared based on a review of GAO TA reports, literature, and referral by other experts. Experts were selected based on their knowledge and expertise in the subject, including: (1) prior participation on a National Academy of Sciences panel or other similar meeting; (2) leadership position in one or more organizations or sectors relevant to technology research and development implementation or policy; and (3) relevant publications or sponsorship of reports. Care was also taken to ensure a balance of sectors, backgrounds, and specific areas of expertise (e.g., science, technology, policy, information technology, and law). We also asked the experts to suggest literature for our review; these suggestions confirmed the literature list noted above. A list of external experts consulted is included below. Dr. Jeffrey M. Alexander, Senior Manager, Innovation Policy, RTI International Dr. Robert D. Atkinson, President, Information Technology and Innovation Foundation Mr. David Bancroft, Executive Director, International Association for Impact Assessment Mr. Duane Blackburn, S&T Policy Analyst, Office of the CTO, MITRE Dr. Peter D. Blair, Executive Director, Division of Engineering and Physical Sciences, National Academies of Sciences, Engineering, and Medicine Ms. Marjory Blumenthal, Acting Associate Director, Acquisition and Technology Policy Center; Senior Policy Researcher, RAND Corporation Mr. Chris J. Brantley, Managing Director, Institute of Electrical and Electronics Engineers, Inc., USA Dr. Jonathan P. Caulkins, H. Guyford Stever University Professor of Operations Research and Public Policy, Carnegie Mellon University Mr. Dan Chenok, Executive Director, Center for The Business of Government, IBM Dr. Gerald Epstein, Distinguished Research Fellow, Center for the Study of Weapons of Mass Destruction, National Defense University Dr. Robert M. Friedman, Vice President for Policy and University Relations, J. Craig Venter Institute Mr. Zach Graves, Head of Policy, Lincoln Network Ms. Allison C. Lerner, Inspector General, National Science Foundation Mr. Mike Molnar, Director of Office of Advanced Manufacturing, National Institute of Standards and Technology Dr. Michael H. Moloney, CEO, American Institute of Physics Dr. Ali Nouri, President, Federation of American Scientists Dr. Jon M. Peha, Professor, Engineering and Public Policy; Courtesy Professor, Electrical and Computer Engineering, Carnegie Mellon University Dr. Stephanie S. Shipp, Deputy Director and Professor, University of Virginia, Biocomplexity Institute and Initiative, Social and Decision Analytics Division Dr. Daniel Sarewitz, Co-Director, Consortium for Science, Policy & Outcomes Professor of Science and Society, School for the Future of Innovation in Society, Arizona State University Ms. Rosemarie Truman, Founder and CEO, Center for Advancing Innovation Dr. Chris Tyler, Director of Research and Policy, Department of Science, Technology, Engineering and Public Policy (STEaPP), University College London (UCL) As part of GAO’s Quality Assurance Framework, GAO’s general design and project plan templates contain five phases that are followed in sequential order, with modifications or changes as needed. GAO technology assessments (TAs) use these templates, as applicable. Throughout the phases, the status of the work, including decisions, is communicated to stakeholders and congressional committees that requested the work. Provided below is a summary of the activities GAO staff undertake during each of the phases, and is based on a review of GAO documentation related to engagement phases. Phase I: Acceptance Engagement characteristics such as risk level or internal stakeholders are determined at a high-level Engagement Acceptance Meeting. Engagement teams obtain a copy of and review the congressional request letter(s), as applicable. Phase II: Planning and Proposed Design Staff are assigned to the engagement and set up the electronic engagement documentation set folders. Staff enter standard information regarding the engagement in GAO’s Engagement Management System (EMS), which is used to monitor the status of the engagement throughout the engagement process and regularly updated. Engagement teams hold an initiation meeting with engagement stakeholders to discuss potential research questions, design options, and stakeholder involvement. Engagement teams clarify engagement objectives and approach through discussions with the congressional requesters, as applicable. Engagement teams obtain background information. For example, to gather information about the topic and any work already performed, teams may conduct a literature review, search prior and ongoing GAO work related to the topic, or consult with external stakeholders, outside experts, and agency officials, including the Congressional Research Service, Congressional Budget Office, and Inspectors General of federal agencies. Engagement teams formally notify agencies of the engagement through a notification letter, and hold an entrance conference, as applicable. Engagement teams prepare a design matrix, project plan, risk assessment tool, data reliability assessment, and all participants on engagements, including stakeholders, affirm their independence. The design matrix is a tool that describes: researchable questions; criteria; information required and sources; scope and methodology; and limitations. The project plan identifies key activities and tasks, dates for completing them, and staff assigned. Engagement teams secure approval to move forward with engagement approach at a high-level Engagement Review Meeting. Phase III: Evidence Gathering, Finalizing Design, and Analysis Engagement teams finalize design: teams work with internal stakeholders to confirm soundness and reach agreement on proposed initial design. If engagement teams and stakeholders conclude that additional work is needed or the design faces significant implementation challenges, design is reviewed and modified, as needed. Engagement teams collect and analyze evidence: teams may collect and analyze evidence using a variety of methodologies including document review, interviews, surveys, focus groups, and various forms of data analysis. For example, engagement teams may meet with agency officials and outside experts, as applicable, to gather evidence. Engagement teams assess evidence and agree on conclusions: teams assess whether the evidence collected is sufficient and appropriate to support findings and conclusions reached for each objective. Once sufficient evidence is collected and analyzed, the team discusses how the evidence supports potential findings and shares these findings with stakeholders, generally in the form of a formal message agreement meeting. Engagement teams update congressional requesters, as applicable, on the engagement status and potential findings. Phase IV: Product Development Engagement teams draft product: after drafting the product, teams send draft to internal stakeholders for review. Teams also send draft to relevant external parties, including relevant agencies, to confirm facts and obtain their views. Teams identify sources of all information in the draft and an independent analyst (not on the team) verifies the sources through a process called indexing and referencing. Engagement teams perform exit conferences with agencies, as applicable, to discuss findings and potential recommendations. Agencies and external parties are given the opportunity to comment on the draft, as applicable. Engagement teams communicate findings and potential recommendations, as well as timeframes for issuing the product, to congressional requesters, as applicable. The draft product is copy-edited, prepared for issuance, and publicly released on GAO’s website, as applicable. Phase V: Results Engagement documentation is closed out. Engagement teams conduct follow-up, track the results, and prepare reports on the status of recommendations and financial and non-financial benefits, as applicable, using GAO’s results tracking system. This appendix provides examples of methods and analytical approaches that GAO technology assessment (TA) teams can use to examine different types of evidence. Also included in this appendix are considerations of the strengths, limitations, and synergies among evidence types and methods, which can be useful to consider throughout design to ensure that evidence is sufficient and appropriate to answer the researchable questions. Examples from GAO TAs were used given our familiarity with GAO products, though numerous other (non-GAO) examples of TA methods exist. This appendix included a review of GAO reports and select literature, and is not intended to be comprehensive. This is a simplified presentation of methods, and there is variation in the levels of structure of the example methods. This appendix is divided into several sections, including by evidentiary types: Testimonial, Documentary, and Physical. For each of these types of evidence, example methods are presented with low and high levels of structure, and include examples of considerations (such as general benefits and limitations) that analysts may consider. In general, more highly structured approaches generate increased consistency and comparability of results that allows for stronger quantification. Less structured approaches tend to provide more flexibility and context, and richer illustrative evidence. Testimonial evidence is elicited from respondents to understand their experience, opinions, knowledge, and behavior, and it can be obtained through a variety of methods, including inquiries, interviews, focus groups, expert forums, or questionnaires. Testimonial evidence can be gathered from individuals who may be responding personally based on their own experience in an official capacity to represent agencies or other entities, or groups, who may share individual level responses, or may present a single group response. Group testimony enables interactions that can be used to explore similarities and differences among participants, to identify tensions or consensus in a group, or to explore ideas for subsequent research and collaboration. It is important to evaluate the objectivity, credibility, and reliability of testimonial evidence. Analysts may use a combination of approaches to gather testimonial evidence, depending on the relevant population(s) of respondents, intended analytical approach(es), likely respondent burden, and resource considerations. Table 9 provides more examples. Documentary evidence is existing information, such as letters, contracts, accounting records, invoices, spreadsheets, database extracts, electronically stored information, and management information on performance. It is important to evaluate the objectivity, credibility, and reliability of documentary evidence. Analysts may use a combination of approaches to gather documentary evidence, depending on the relevant sources and types of documents, intended analytical approach(es), and resource considerations. Table 10 provides more examples. Physical evidence is obtained by direct inspection or observation of people, property, or events. The appropriateness of physical evidence depends on when, where, and how the inspection or observation was made and whether it was recorded in a manner that fairly represents the facts observed. Common considerations for physical evidence include the reliability of site selection, intended analytical approaches, and resource considerations. Table 11 provides more examples. GAO may also rely on agency and other secondary data. Considerations for those secondary data are dependent on the type, source, and collection method, and could include all of the considerations above. Use of secondary data is usually more efficient than collecting new data on a topic, and administrative records (a form of documentary evidence) are generally not as prone to self-reporting biases that may be present in testimonial evidence. However, when secondary data are used, more work may be required to assess whether data are reliable and appropriate for a given purpose. For example, analysts will gather all appropriate documentation, including record layout, data element dictionaries, user’s guides, and data maintenance procedures. Depending on the database, procedures and analysis can be very complex—and it would be important to note assumptions, limitations, and caveats pertaining to the data, which may affect the conclusions that can be drawn based on the analyses. Examples of analytical approaches found in the literature to analyze data include: Interpretive structural modeling: shows a graphical relationship among all elements to aid in structuring a complex issue area, and may be helpful in delineating scope. Trend extrapolation: is a family of techniques to project time-series data using specific rules, and may be helpful in forecasting technology. Scenarios: is a composite description of possible future states incorporating a number of characteristics, and may be helpful in policy analysis. Scanning methods, such as checklists: is listing factors to consider in a particular area of inquiry, and may be helpful in identifying potential impacts. Tracing methods, such as relevance trees: includes identifying sequential chains of cause and effect or other relationships, and may be helpful in identifying potential impacts. Cross-effect matrices: are two-dimensional matrix representations to show the interaction between two sets of elements, and may be helpful in analyzing consequences of policy options. Simulation models: are a simplified representation of a real system that is used to explain dynamic relationships of the system, and may be helpful in identifying impacts and forecasting technology. Benefit-cost analysis: is a systematic quantitative method of assessing the desirability of government projects or policies when it is important to take a long view of future effects and a broad view of possible side effects. Decision analysis: is an aid to compare alternatives by weighing the probabilities of occurrences and the magnitudes of their impacts, and may be helpful in determining impacts and assessing policy options. Scaling: is an aid that may include developing a matrix that identifies potential impact related to an activity and stakeholder group, and qualitatively or quantitatively assesses the potential impact, and may be helpful analyzing potential impacts, including of policy options. In addition to the contacts named above, key contributors to this report were R. Scott Fletcher (Assistant Director), Diantha Garms (Analyst-in- charge), Nora Adkins, Colleen Candrl, Virginia Chanley, Robert Cramer, David Dornisch, John De Ferrari, Dennis Mayo, Anika McMillon, SaraAnn Moessbauer, Amanda Postiglione, Steven Putansu, Oliver Richard, Meg Tulloch, Ronald Schwenn, Ben Shouse, Amber Sinclair, Ardith Spence, Andrew Stavisky, David C. Trimble, and Edith Yuh. The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is through our website. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. You can also subscribe to GAO’s email updates to receive notification of newly posted products. The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 Please Print on Recycled Paper.", "summary": "In January 2019, at the direction of Congress, GAO formed the Science, Technology Assessment, and Analytics team to expand its work on cutting-edge science and technology issues, and to provide oversight, insight, and foresight for science and technology. TAs can be used to strengthen decision-making, enhance knowledge and awareness, and provide early insights into the potential impacts of technology. TA design can enhance TA quality, credibility, and usefulness; ensure its independence; and ensure effective use of resources. Under the Comptroller General authority, we developed this handbook using the format of the 2012 GAO methodology transfer paper, Designing Evaluations . Below is a summary of the approach we used to affirm and document TA design steps and considerations for this handbook. Reviewed select GAO documents, including Designing Evaluations (GAO-12-208G), published GAO TAs, select GAO products that presented policy options, and other GAO reports Reviewed select Office of Technology Assessment reports Reviewed select Congressional Research Service reports Reviewed select literature on TAs and related to development and analysis of policy options Held an expert forum to gather experts’ input on TA design Considered experiences of GAO teams that have successfully assessed and incorporated policy options into GAO products, as well as GAO teams that are currently incorporating policy options into their TA design Collected input from GAO staff who provided key contributions to GAO TAs, regarding challenges to TA design and implementation and possible solutions The Technology Assessment Design Handbook identifies tools and approaches GAO staff and others can consider in the design of robust and rigorous technology assessments (TAs). The handbook underscores the importance of TA design (Chapter 1), outlines the process of designing TAs (Chapter 2), and describes approaches for mitigating selected TA design and implementation challenges (Chapter 3). While the primary audience of this handbook is GAO staff, we expect that other organizations engaged or interested in TAs will find portions of this handbook useful. We anticipate modifying and refining this handbook, as needed, based on experience and public comments received. We will accept comments on this handbook at TAHandbook@gao.gov for approximately 1 year after publication. The handbook identifies three general design phases, as appropriate, as shown in the figure below. T terative nature of TA design, the requester’s interests, resources, independence, stakeholder engagement, potential challenges, and communication. In addition, ormulating initial policy options to consider; gathering evidence, determining relevant dimensions to analyze, and analyzing the policy options; and presenting the results of the policy analysis. Summary of Key Phases of Technology Assessment Design We found that GAO TAs have and can use a variety of design approaches and methods. The handbook provides TA design and methodology examples, including related to objectives commonly found in GAO TAs, such as: describe a technology, assess opportunities and challenges of a technology, and assess policy considerations. One example provided is: some GAO TAs include an objective related to describing the status and feasibility of a technology, which GAO teams have done by using methodologies such as expert panels, interviews, literature and document reviews, site visits, and determining the Technology Readiness Level. Also included in the handbook are examples of TA design and implementation challenges we found, along with possible mitigation strategies. We identified four general categories of challenges, including: (1) ensuring TA products are useful for Congress and others; (2) determining policy goals and measuring impact; (3) researching and communicating complicated issues; and (4) engaging all relevant stakeholders. An example of a potential mitigation strategy to the specific challenge of writing simply and clearly about technical subjects includes: allowing sufficient amount of time for writing, including reviewing and revising writing.", "document_type": "gao"}
{"report": "IDEA was enacted to ensure that all children with disabilities have access to a free appropriate public education (FAPE); to protect the rights of those children and their parents; and to assist states, localities, educational service agencies, and federal agencies in educating those children. Part C of IDEA provides grants to states for Early Intervention services for infants and toddlers (birth through 2 years) with developmental delays or diagnosed conditions that have a high likelihood of developmental delay. Part B of IDEA provides grants to states to assist them in providing special education and related services to eligible children with disabilities beginning at age 3 and possibly lasting to the student’s 22nd birthday, depending on state law or practice. In fiscal year 2019, the total appropriation for IDEA Parts B and C was approximately $13.2 billion ($12.8 billion for Part B and $470 million for Part C). These funds are awarded through formula grants to state agencies which, in turn, provide these funds to eligible entities (school districts under Part B and early intervention service providers under Part C) to carry out applicable IDEA requirements. (See table 1.) Each state has a designated lead agency—called a Part C Lead Agency—that is responsible for administering, supervising, and monitoring Part C. Part C requires each state to have a continuous process of public awareness activities and evaluations designed to identify and refer as early as possible all young children with disabilities and their families who are in need of Early Intervention services. By law, public awareness efforts should include disseminating information to parents and those likely to make referrals, especially hospitals and physicians. States have disseminated this information in different ways, including through television ads, pamphlets, and posters describing Part C and how parents can obtain services for their child. Under Part C of IDEA, states must also provide services to any child under 3 years of age who is developmentally delayed. These delays must be measured by appropriate diagnostic instruments and procedures in one or more areas of cognitive development, physical development, communication development, social or emotional development, and adaptive development, or the child must have a diagnosed physical or mental condition that has a high probability of resulting in developmental delay. Once a child who is suspected of having a disability is referred, states must evaluate the child in accordance with applicable IDEA requirements. Figure 1 illustrates the typical process in Early Intervention programs. Infants and toddlers who are still receiving services by about age 2 and a half are evaluated again to determine if they are eligible for services under Part B. Under Part B, states and school districts must make FAPE available to all eligible children with disabilities in mandatory age ranges. FAPE includes special education (specially designed instruction) and related services (support services)—such as speech therapy, psychological services, and physical therapy—tailored to their needs based on an individualized education program (IEP). Figure 2 illustrates the typical process for identifying students for special education under Part B. Figure 3 shows the percentage of children served under IDEA by age and state as of fall 2016. Nationally, for each age group, the percentage of children receiving special education services remained relatively stable from 2012 through 2016, changing by less than 1 percentage point. IDEA requires states to have policies and procedures to ensure that school districts identify, locate, and evaluate all children suspected of having a disability who need special education and related services, regardless of the severity of their disability, but also gives states some latitude in establishing eligibility criteria and defining disability categories. In addition, states have some flexibility to determine their own processes for identifying and evaluating children, provided the state’s procedures are consistent with IDEA requirements. As a result, a child eligible for IDEA services in one state might be ineligible in another. Eligibility criteria. IDEA allows states some flexibility to establish their own definitions of developmental delay (when a child does not reach developmental milestones for certain skills, such as motor or language skills, at the expected times), including the level or severity of the delay. For example, in Maryland, a child must have at least a 25 percent delay in one or more developmental areas to be eligible for Early Intervention services, while in Arizona, a child must demonstrate a 50 percent delay in one or more developmental areas to be eligible. In Massachusetts, Part C lead agency officials we interviewed said that the state had, as IDEA allows, tightened eligibility criteria in 2009 to reduce the number of children eligible for Early Intervention services by narrowing the definition of developmental delay. Officials said that there were no current plans to change the eligibility criteria, but that they would consider tightening eligibility criteria again if the number of eligible children outpaces state fiscal resources for these services. Part C of IDEA also allows but does not require states to provide Part C services to at-risk infants and toddlers. States that choose to provide services to at-risk children may use IDEA risk factors to determine eligibility, such as low birth weight or history of abuse and neglect, or they may develop their own list of risk factors. For example, Massachusetts developed its own at-risk criterion for eligibility, which requires the presence of four or more defined child and family factors, including biological, medical, and trauma-related factors. As of 2018, seven states or territories were serving at-risk infants and toddlers, according to an Education official. Early Intervention process. The processes states use to deliver Part C Early Intervention programs can vary in a number of ways. First, the types of agencies designated as the Part C Lead Agency vary from state to state; these lead agencies are responsible for administering and monitoring Early Intervention programs in their states. For example, Iowa’s State Educational Agency (SEA) administers both its Parts C and B programs; Massachusetts and New York administer their Parts C and B programs through separate agencies; and, Colorado shares these responsibilities between two agencies. Second, the extent to which lead agencies directly provide Early Intervention programs, including locating and evaluating children, or do so through contractors varies. For example, both Colorado and Iowa administer their Early Intervention programs directly, while Massachusetts and New York contract with private entities to do so. In Massachusetts, early childhood officials said that they contract with 31 different vendors that operate 60 Early Intervention programs throughout the state. In addition to providing Early Intervention services, these programs are responsible for locating and evaluating children, according to the early intervention officials. Those officials also said that each of these individual programs have unique relationships with referral sources, which can affect the likelihood that the sources will make referrals to a given program. Regardless of the type of entity responsible for Early Intervention programs, having strong relationships with referral sources is important, according to early childhood officials in all four of the states we visited. Otherwise, according to these officials, some children who are likely to be eligible for Early Intervention services may not be identified or evaluated for needed services. In Colorado, where Early Intervention responsibilities are shared between the Part C lead agency and the SEA, state officials said that this arrangement can make it difficult to ensure a seamless process and can cause delays between evaluation and services. They said that this can result in incorrect identification or services because they do not have control over the evaluations—responsibility for evaluations is assigned to the Part B agency. Part C officials also said this can cause confusion for families as they are moved between agencies. Relatedly, some infants and toddlers may not be identified for Early Intervention services because of the challenges of sharing data between state agencies when more than one agency is responsible for providing special education to children. In three of the four selected states we visited, responsibility for special education services for children was shared by more than one agency and officials in all three states told us that difficulties in sharing Early Intervention program data could hamper efforts to identify potentially eligible children for special education services. Officials in one of the states said that sharing data could allow them to identify children being provided school-aged special education services that had not received Early Intervention services. The officials said that if commonalities were found among these children, it could help them find similar children and ensure they receive Early Intervention services in the future. Eligibility criteria. In practice, IDEA Part B’s disability definitions provide minimum standards that all states must meet. According to Education officials, IDEA allows states the flexibility to adopt more expansive definitions of disabilities than those provided in the IDEA statute and regulation, provided that the state definition would not exclude children who would be covered by the IDEA definition. For example, in New York an intellectual disability is defined as “significantly subaverage general intellectual functioning … that adversely affects a student’s educational performance,” while in Massachusetts an intellectual impairment is defined as occurring when “the permanent capacity for performing cognitive tasks, functions, or problem solving is significantly limited or impaired and is exhibited by…a slower rate of learning .” Also, states must establish their own eligibility criteria for determining the presence of a Specific Learning Disability (SLD)—a broad category of disorders related to understanding and using language. IDEA also requires that states allow the use of research-based procedures in establishing the presence of an SLD, but does not define the specific procedures to be used. Identification process. IDEA requires all states to have Child Find policies and procedures in place, and requires a practical method for determining which children with disabilities are currently receiving needed special education and related services, but does not specify the exact method to be used. In all four of the states we visited, school district officials we interviewed said that the schools in their respective districts were using the same type of approach as part of the Child Find identification process, but that some school districts were in different stages of implementation or that the approach was being used differently by schools within the same districts. Officials in one school district in New York said that, as part of their approach, there was a concerted effort to use student data to make decisions about intervention levels and special education evaluation decisions, while a school district official in Massachusetts said that the district had placed a greater emphasis on improving classroom instruction as a means to reduce the need for special education services rather than on intervention systems used for identifying and making decisions about potentially eligible children. Officials of school districts in two of the states we visited told us that they are in the midst of revising their identification processes to increase accuracy and consistency across the schools in their districts. Officials in one of those districts said that differences in the processes schools used resulted in variations in how the special education identification process worked in each of the schools. Appropriately identifying and evaluating children who may be eligible for special education services can be difficult, according to advocates, subject matter specialists, and state and local officials we interviewed. Representatives of two national special education advocacy organizations and special education subject matter specialists agreed that it may be difficult to identify disabilities and that differences in school district or in school special education processes can add to this challenge. Early Intervention services are intended to enhance the development of infants and toddlers with disabilities, minimize developmental delay, and reduce the need for special education later in life. However, officials we interviewed at state agencies in the four states we visited— Massachusetts, Colorado, New York, and Iowa—said that because of challenges in identifying and evaluating children, some infants and toddlers who are eligible and would benefit from Early Intervention services do not receive them. These challenges include navigating referral processes, obtaining parental consent, and dealing with staffing limitations. State early childhood officials and subject matter specialists we interviewed said it can be difficult to secure a parental or physician referral, which can cause delays in evaluating children and may lead to some infants and toddlers not being provided Early Intervention services. In all four states we visited, officials noted that some parents or physicians did not make referrals because they did not understand the referral process. State officials in Iowa expressed concern that some doctors may take a “wait-and-see” approach instead of referring an infant or toddler for evaluation when indications first arise. Early childhood officials in Colorado as well as Early Intervention subject matter specialists we spoke to said that physicians may also choose not to refer patients because they (1) cannot guarantee families that their children will ultimately receive services, (2) find the referral process difficult, or (3) receive little feedback about whether their referrals ultimately lead to children getting Early Intervention services. Before an infant or toddler can be evaluated for Early Intervention services, the parent(s) must give consent. In Massachusetts and Colorado, state early childhood officials said that parents sometimes do not provide consent for an evaluation, which can delay or even prevent the delivery of needed services. Officials from these states cited various reasons parents might withhold consent, such as opting to wait and see if the child’s problems are resolved over time. State early childhood officials in Massachusetts also said that parents will sometimes refuse to provide consent for evaluation due to a lack of awareness of Early Intervention services or the Early Intervention process. To better address this, officials said that they are working collaboratively with state early education and care providers to inform parents about these issues. Massachusetts officials stated that parents may mistrust government agencies or associate Early Intervention services or providers with child protective services agencies and mistakenly think they are being investigated. Insufficient personnel with the right qualifications to conduct evaluations is another reason infants and toddlers may not be consistently identified and evaluated, particularly in certain types of locations. Officials from lead agencies in Massachusetts, Colorado, New York, and regional education officials in Iowa, noted that it was difficult to find enough Early Intervention personnel with appropriate expertise in low population density areas which can complicate the process of identifying and evaluating children. Officials in Massachusetts noted challenges hiring staff that reflect the communities they serve and in hiring for specific disciplines, such as occupational and physical therapists. In addition, officials in New York said that they sometimes face staffing difficulties when children are located in areas with high crime rates. State and local officials as well as special education advocacy organizations said identifying and evaluating students for Part B special education services can be complicated by many factors, which may result in some students inappropriately being determined eligible or ineligible for services. These factors include confusion over IDEA requirements, challenges implementing Response to Intervention (RTI), a child’s lack of English proficiency, the difficulty of detecting certain types of disabilities, or the Part C to Part B transition. School district officials in Massachusetts said that confusion about IDEA requirements is common. For example, a school district official from that state told us that general education staff do not always understand when special education services are appropriate, versus when other options may meet students’ needs, such as Response to Intervention (RTI) or other supports. (See sidebar for more information about RTI.) Officials in another school district in the same state said there was confusion over and little consistency in the eligibility decisions made for special education and other supports. Additionally, officials in that district said that the expertise level among the decision makers varies and can affect eligibility decisions. Response to Intervention For those students who may need additional academic and behavioral supports to succeed in a general education environment, schools may choose to implement a multi-tiered system of supports, such as Response to Intervention (RTI). Regulations implementing the 2004 amendments to the IDEA include a provision mandating that states allow, as part of their criteria for determining whether a child has a Specific Learning Disability (SLD), the use of a process based on the child’s response to scientific, research-based intervention. See 34 C.F.R. § 300.307(a)(2). RTI is a school-wide approach that attempts to address the needs of all students, including struggling learners and students with disabilities, and integrates assessments and interventions to maximize student achievement. Key characteristics of RTI are: (1) students receive high-quality research- based instruction in the general education setting; (2) schools continually monitor and document student performance; (3) schools screen all students for academic and behavioral problems; and (4) schools provide multiple levels (tiers) of instruction that are progressively more intense, based on the student’s response to instruction. Children who do not respond to interventions are to be referred for evaluation to determine eligibility for special education and related services. School district officials in all of the states we visited and representatives from various advocacy organizations said that there were challenges related to implementing RTI. Representatives from advocacy organizations in all four states we visited cited concerns with school RTI practices that may delay student evaluations or contribute to incorrect eligibility determinations. Advocates in Massachusetts told us that some school districts are more likely than others to put students suspected of a disability through the RTI process for extended periods of time before evaluating them. Further, advocates said using RTI to delay or deny evaluations occurs more frequently at the elementary level and for students with specific types of disabilities, such as mental health and social or emotional disabilities. implementation, the type of disability a student has, the quality and quantity of data gathered on students, and the amount of support provided for the process. In all of the states we visited, school district officials cited efforts to address issues with RTI practices. For example, school district officials in all four states noted that training related to RTI was being provided to their schools. In Massachusetts, New York, and Iowa, school district officials cited recent initiatives specifically aimed at strengthening and implementing the RTI process in schools, such as by integrating social- emotional and behavioral components in RTI and better using student- level data to improve eligibility determinations. In one district, officials specifically noted that efforts to improve their schools’ RTI processes and core curriculum had reduced the number of special education students in their district. According to Education’s 2016-17 school year data, 73 percent of public school districts in the nation had English Learner students; nationwide, English Learner students comprise about 10 percent of public school students, an increase of almost 3 percent since 2010. School district officials we interviewed in all four states we visited described inherent challenges in properly identifying and evaluating English Learner students for special education disabilities. In Massachusetts and New York, school district officials we interviewed explained that they do not always have staff with the necessary expertise to perform evaluations in a child’s first language, which makes it more difficult to determine if a child’s learning difficulties are caused by a disability or by language proficiency issues. State education officials in New York told us that they are concerned about identification issues related to English Learner students, noting that over 200 languages are spoken by their students and about 12 percent of their students with disabilities were also English Learners in 2017-18. In the same state, officials in one school district said that over 100 different languages are spoken by their students and that it was a challenge to properly identify and evaluate them. Representatives of special education advocacy organizations in two states we visited—Massachusetts and New York—made similar observations, noting that English Learner students were at risk of being both over identified and under identified. For example, advocates we interviewed in Massachusetts said that under identification can occur when school districts do not communicate with parents in their home language and, as a result, the parents do not understand how to engage with the special education process. Advocates in both states told us that over and under identification may also occur if the lack of language proficiency is mistaken for a disability or if a disability is mistaken for language learning issues. Education and the Department of Justice have issued guidance to assist schools in meeting their obligations under federal law to ensure that English Learner students who may be eligible for services under IDEA are located, identified, and evaluated for special education services in a timely manner. This guidance instructs schools to consider the English language proficiency of the students appropriately so that they are not identified as students with disabilities because of their limited English language proficiency. Local officials we interviewed in four states said that some disabilities, such as those related to mental health or behavioral disorders, can be difficult to identify and may go undiagnosed. These officials noted that behavioral disabilities can be particularly difficult to correctly identify because they sometimes affect academic performance or behavior in more subtle ways. Some school district officials said they may not have the right tools or staff to identify these students. For example, officials in one school district in Colorado stated that a commonly used disability identification process on its own was not effective for students with mental health and behavioral disabilities. School district officials we spoke to in Massachusetts and Iowa noted that they often struggle to employ staff with the appropriate expertise to address mental health or behavioral issues and that there are fewer resources for schools to use in these areas. Part C to Part B transition Another area of confusion may arise when children transition from Part C services to Part B services, at about age 3. School district officials in the four states we visited said that they identify a significant number of their districts’ school-aged special education students through referrals from the state’s Early Intervention programs during the transition process. State education officials in Massachusetts indicated that the majority of children referred from the early childhood programs for Part B services are not found eligible for school-aged services, which may indicate a lack of a common understanding of the Part B eligibility criteria as the early childhood programs are required to refer the children they think could be eligible for those services. Education’s monitoring of state efforts to implement Child Find requirements is part of a broad framework—known as Results Driven Accountability (RDA)—the department uses to monitor certain aspects of IDEA implementation. Education’s monitoring activities specific to Child Find are based on data and information that states submit annually, as required by IDEA and as part of the RDA process. Because IDEA gives states some discretion in how to meet Child Find requirements, according to Education officials, it focuses on ensuring states have policies, procedures, and systems in place for monitoring local school districts’ special education programs, including their Child Find activities. To monitor state Child Find activities, Education relies, in part, on four indicators specific to the Child Find requirements and requires states to report data on them annually in the State Performance Plan/Annual Performance Report. Three of the indicators pertain to Part C Early Intervention programs and one pertains to Part B. Two Part C Child Find indicators compare the numbers of children served to two data points—the national Part C average (as a percentage) as well as the percentage Education would expect a state to serve based on the state’s population. Education requires states to report these Part C data for two subsets of children—birth to 1 year and birth through 3 years. Education has encouraged states whose Part C enrollment is significantly lower than the national average or below expected levels based on the state’s population, to examine compliance with related Part C requirements. The third Part C Child Find indicator measures state compliance with the 45-day timeline. For this indicator states must report on the number and percentage of children referred to Part C whose evaluations, assessments, and initial individualized family service plan meetings were held within 45 days of referral. The Part B indicator measures the percent of children who were evaluated within 60 days of receiving parental consent for initial evaluation. This indicator is a compliance indicator for which states must establish a target of 100 percent. According to Education officials, the department developed these Parts C and B indicators in response to requirements in the 2004 IDEA reauthorization, which directed the Secretary of Education to monitor the states, and require each state to monitor local educational agencies located in the state or as applicable, the early intervention providers located in the state, using quantifiable indicators in specific priority areas (including Child Find), and using such qualitative indicators as are needed to adequately measure performance in those areas. In developing the indicators, Education officials told us that the department sought to strike a balance between the statutory requirement that they be quantifiable and the inherent challenges in knowing how many children should be identified, evaluated, and found eligible—at the state level or in individual school districts. Education officials said that states and school districts are in a much better position to estimate how many children who have disabilities and who could potentially be found eligible for special education and related services because of their disability. Education officials told us they consulted internal stakeholders, states, school districts, and other special education experts to develop possible quantifiable measures given the inherent challenges in doing so. In addition to the Child Find indicator data submitted annually, under Part B, states provide other information related to Child Find as part of their annual data reporting to Education and the public. These data include the number and percentage of children with disabilities by race, ethnicity, English Learners, gender, and disability category that receive a free appropriate public education; participate in general education; are placed in separate classes, schools, or residential facilities; receive Early Intervention services; and are between birth to 2 years who are no longer receiving Early Intervention services. States are also required to report the number and percentage of infants and toddlers, by race and ethnicity, who are at risk of having substantial developmental delays and who are receiving Early Intervention services. Additionally, Education may receive information about states’ Child Find activities in states’ annual reports as part of the description of IDEA oversight policies and procedures; in explanations of any actions taken in response to Education’s finding of noncompliance with Child Find indicators in prior years; or in the comprehensive multi-year improvement plan Education requires as part of its RDA framework. Education supports states’ implementation of Child Find in a variety of ways, including a network of technical assistance centers, written guidance, and direct assistance from Education staff. The Technical Assistance and Dissemination (TA&D) program is the primary way Education provides educators, administrators, service providers, and parents with information regarding IDEA. This program assists state and local administrators on a range of topics including clarifying Child Find obligations, professional development for staff and administrators on various aspects of Child Find, and federal accountability requirements. Technical assistance offerings include training on data collection and Early Intervention issues for various audiences such as teachers, administrators, and special education service providers. Officials in each of the states we visited said they had used Education’s technical assistance. In addition to the TA&D program, Education has established six centers that specifically support states in the annual data collection process. Education provides written guidance to states through documents such as Dear Colleague Letters, Frequently Asked Questions, and Questions and Answers. These documents clarify provisions of Child Find and other IDEA requirements as well as respond to common inquiries from school administrators or the public. The written guidance may also address information gathered during oversight activities and changes in federal law or regulation. Topics Education has addressed in written guidance on Child Find include school districts’ uses of RTI and requirements for subgroups of children who may be difficult to find. For example, Education issued a memorandum in 2016 reminding states and districts that (1) RTI processes cannot be used to delay or deny a timely evaluation of a child suspected of having a disability and (2) implementation or completion of RTI is not required prior to evaluating a student for special education services. Officials in Colorado said they found this guidance helpful and issued guidance to their school districts based on Education’s memorandum. Additionally, in 2007 and 2008 Education addressed issues regarding Child Find requirements for certain groups of children, such as those who are homeless or those who are residing in Immigration and Customs Enforcement (ICE) residential facilities. Homeless children, for example, are inherently difficult to identify and evaluate for special education services because they and their families are highly mobile. Education’s guidance reminded states and school districts that their Child Find obligations include these hard to find subgroups and directed states to coordinate with emergency shelters and homeless advocacy programs, among others, to help find children suspected of having a disability. Education’s website notes that each state is assigned a customer service representative, a Part B contact, a Part C contact, and a team leader. Education officials we spoke to told us that staff hold monthly check-in meetings with state officials to provide information and discuss issues of concern. They also said that issues needing clarification sometimes arise during these check-in meetings. For example, they said that in a meeting with state directors they identified a lack of clarity around some English Learner issues. As a result, Education developed guidance to explain Child Find obligations regarding English Learner students as well as other obligations under IDEA. Education also has a customer service unit available to assist states with questions about IDEA, special education, and related services. State officials in all four states we visited told us they had good relationships with Education IDEA monitoring staff and rely on them to learn about available technical assistance and other resources. Officials we interviewed in one state said their Education contacts were instrumental in helping them improve their programs. facilities, although Education stated that an ICE facility and the state or local school district could enter into a voluntary agreement to provide Child Find or other educational services. U.S. Department of Education, Office of Special Education and Rehabilitative Services, Letter to David Anderson, General Counsel, Texas Education Agency (Dec. 21, 2007); U.S. Department of Education, Office of Special Education and Rehabilitative Services, Letter to David Anderson, General Counsel, Texas Education Agency, (Apr. 22, 2008). Education officials told us that if these children are released from ICE facilities into the care of a sponsor to await their immigration hearings, they do have a right under federal law to enroll in public elementary and secondary schools and to receive educational services, including special education services, if found eligible. States must monitor their local school districts’ implementation of IDEA requirements. As part of the State Performance Plan/Annual Performance Report, each state must establish measurable and rigorous targets for the indicators, including Child Find, and must analyze the performance of each local school district in the state in implementing the requirements of Part B or as applicable, each Early Intervention provider located in the state in implementing the requirements of Part C. Data analysis and regular audits are the primary means states use to monitor local school districts, according to officials we interviewed in each of the four states we visited. The Part C lead agencies in the four states we visited reported monitoring local implementation of Early Intervention programs through indicator data or on-site visits. In their State Performance Plan/Annual Performance Reports for federal fiscal year 2016, the states we visited reported various monitoring activities. For example: Colorado gathers data from an online system to monitor local programs and analyze performance. In addition to desk audits of local service providers, Colorado’s lead agency does on-site monitoring, selecting local agencies for monitoring visits based on its annual priority areas, or focusing on a cross-section of programs based on size, region, and program structure. Colorado’s annual priority areas have included topics such as increasing public awareness regarding Early Intervention services by providing developmental information to parents of newborns in the hospital and ensuring that the transdisciplinary team members who are responsible for evaluating infants and toddlers are effectively communicating. Massachusetts’ local Early Intervention programs complete and submit to the state lead agency annual reports and self- assessments based on federal indicators. Additionally, the Part C lead agency conducts on-site monitoring of selected sites on a cyclical basis, and focused monitoring to examine specific aspects of local Early Intervention programs. New York conducts comprehensive on-site monitoring of municipalities that administer local Early Intervention programs and approved providers who perform Early Intervention services including reviewing written policies and procedures regarding Early Intervention processes as well as examining a sample of client records at each service location. Iowa monitors all regional grantees on an annual basis. The process includes review of parent surveys and review of family outcome data, among other things. When performance or compliance issues are identified, the lead agency conducts desk audits and data verification checks. Although Part B monitoring activities in the four selected states are similar, they reflect the structure, policies and procedures of individual states. For example, Iowa officials said they monitor both Area Education Agencies and local school districts through desk audits and site visits. Officials told us that the SEA has developed (1) a process to evaluate the performance of the regional agencies regarding the provision of special education services and their oversight responsibilities for the local school districts, and (2) a separate process that examines the performance of school districts with regard to IDEA implementation. The State Performance Plan/Annual Performance Reports for federal fiscal year 2016 for the remaining three states we visited note the following monitoring activities: Colorado collects data and reviews the results of school district self- audits from each of its districts. Massachusetts reported reviewing indicator data and instituting a new monitoring process called Tiered Focus Monitoring. In the first year of the monitoring cycle, all local school districts are to conduct self- assessments on specific criteria related to the special education identification processes and other topics. The self-assessments inform the SEA’s on-site monitoring in the second year. In the third year, school districts are to continue internal monitoring; and in the fourth year, they complete a self-assessment regarding special education and legal requirements. New York reported reviewing data and using school district self- assessments, desk audits, and on-site monitoring. According to the annual report the selection of sites for on-site monitoring depends on a variety of information, including performance on indicator targets. IDEA requires states and lead agencies to provide professional development and technical assistance to local school districts. The State Performance Plan/Annual Performance Reports for federal fiscal year 2016 for each of the four states we visited described professional development activities provided on topics related to Part C Early Intervention and Part B programs. For Part C, states reported that they provided the following professional development activities among others: Colorado provided training on data management to ensure valid and reliable data for monitoring purposes. Iowa provided service coordination training which provides knowledge and skills to understand Early Intervention eligibility, the IDEA, and Early Intervention services. Massachusetts held training sessions for Early Intervention service providers regarding Early Intervention transitions to support children who are exiting Early Intervention services or are referred for Part B services. Early Intervention service providers were also able to receive training concerning functional assessments. New York employed contractors to provide training on best practices for delivering Early Intervention services and training about providing those services in a child’s natural environments. Additionally, they provided training to primary referral sources. For Part B, the states reported that they provided the following professional development activities among others: Colorado provided professional development on topics that were identified by teachers. The SEA surveys teachers, providers, and Special Education Directors annually to determine professional development topics. Officials we interviewed in selected school districts told us that they had received training on Child Find obligations and classroom interventions. Iowa requires each district to develop professional development plans that support the needs of district staff responsible for instruction. Districts officials said they have provided training concerning intervention strategies and Child Find responsibilities. Massachusetts has provided training in social emotional learning and behavioral interventions. New York provides ongoing statewide training regarding classroom and behavioral interventions, as well as a program for school principals regarding special education law and regulations as well as the principal’s responsibilities for implementing IDEA. Officials we interviewed in each of the four states we visited told us that they offer a range of technical assistance, including written guidance, webinars, meetings/conferences, telephone assistance, and one-on-one training to support local school districts and schools in implementing Child Find requirements. For example, New York instituted a Blueprint for Improved Results for Students with Disabilities. This Blueprint establishes expectations to improve instruction and results for students with disabilities, which in turn informs the state’s technical assistance networks. In each of the four states, officials reported (1) offering targeted assistance where there were concerns related to performance or results of Part B programs and (2) examining results and compliance data to identify areas of concern and potential recipients for targeted assistance. For example, Massachusetts reported in its annual report that it had provided one-on-one technical assistance to local school districts where there were performance concerns, while New York reported that its technical assistance improvement specialists review low-performing schools and help to develop tools for improvement. Similarly, the Part C lead agency officials in all of the states we visited told us they provided training and technical assistance to Early Intervention programs. These states offered assistance in a variety of ways including written guidance, information provided via phone or email, and formal training sessions. Officials from Colorado and Iowa reported holding monthly technical assistance calls, while officials from Massachusetts reported holding monthly webinars for local Early Intervention providers. In its annual report, Iowa reported providing training on using technology to provide Early Intervention services, while New York reported offering training on best practices in identifying and evaluating infants and toddlers. Each of the four states we visited reported offering targeted assistance to schools where monitoring efforts identified concerns or compliance issues. The targeted assistance is intended to improve performance in the areas identified. We provided a draft of this report to Education for review and comment. Education provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. The objectives of this study were to examine (1) factors that may account for differences in the percentage of children receiving special education services, and (2) how the U.S. Department of Education (Education) and selected states monitor and support Child Find requirements. To conduct this work we (1) reviewed federal special education data from school years 2011 through 2016 (the most recent available at the time of our review); (2) reviewed relevant Department information, such as Dear Colleague Letters, Frequently Asked Questions, and Questions and Answers; federal laws; regulations and policies; and selected state laws; (3) interviewed Education officials; (4) interviewed officials from state agencies responsible for administering Parts C and B of the Individuals with Disabilities Education Act (IDEA) special education programs in four states (Colorado, Iowa, Massachusetts, and New York) and fifteen school districts within those states; and (5) interviewed representatives from special education advocacy organizations that represent parents and families of individuals with disabilities and subject matter specialists to discuss issues related to Child Find. The following sections contain detailed information about the scope and methodology for this report. To determine the differences in the percentage of children receiving special education services across states we used Education’s Annual Reports to Congress on the Implementation of the Individuals with Disabilities Education Act (IDEA) to review national and state level special education data. We used the most recent five reports, 2014 through 2018, which reported on data for school years 2012 through 2016, to review the percentages of children that were receiving special education services under IDEA Part C and Part B during school years 2012 through 2016 nationally and by state. These data, known as Section 618 data, are self- reported by school districts. We focused our review primarily on data regarding the percentage of children served under IDEA Part C (ages 0- 2), Part B (ages 3-5), and Part B (ages 6-21), nationally and by state during school years 2012 through 2016. We determined that the data we used from the Annual Reports to Congress on the Implementation of IDEA were sufficiently reliable for the purposes of the report by reviewing technical documentation and interviewing Education officials to determine what mechanisms are in place to ensure data quality. To obtain information on the factors that may account for variation in the percentage of children receiving special education services and to examine how Education and selected states support and monitor Child Find requirements, we reviewed Education documents, such as Dear Colleague Letters, Frequently Asked Questions, and Questions and Answers. We also reviewed Education’s recent annual reports to Congress and documents containing guidance to states on required annual data submissions. Additionally, we reviewed relevant federal laws, regulations, and policies, and selected state laws and regulations. With both Education and state agencies responsible for supporting and monitoring Child Find requirements, we interviewed officials about the agencies’ responsibilities with respect to IDEA, as well as the processes the agencies put in place to monitor implementation of those requirements. We also discussed each agency’s guidance and support to school districts on these issues. In addition, we collected and reviewed relevant agency procedures and guidance documents. To obtain information on the factors that may account for differences among selected states and school districts in the percentage of children receiving special education services and how selected states support and monitor Child Find requirements, we conducted site visits in a non- generalizable sample of four states and 15 school districts. We selected states primarily for diversity in (1) the percentage of special education students; (2) changes in the percentage of special education students over a 5-year period; (3) geography; and (4) the agency responsible for state Early Intervention programs (i.e., the state educational agency or another state agency). We used data from the National Center for Education Statistics (NCES), Common Core of Data (CCD) for the 5-year period, 2011-2015 (the most recent available data at the time of our selection) to identify the percentage of special education students in each state as well as the change in the percentage of special education students in each state over the 5-year period. We determined that the data used were sufficiently reliable for the purposes of the report by reviewing technical documentation and interviewing Education officials to determine what mechanisms are in place to ensure data quality. In each state, we interviewed officials from the state educational agency, the agency responsible for Part B special education, as well as officials from the state agency responsible for Part C special education. In addition, we also interviewed officials from special education advocacy organizations that represent parents and families of individuals with disabilities. We selected school districts primarily for diversity of size. We used state department of education enrollment data for 2017-2018 to sort school districts based on the size of the student population. We selected three school districts in Colorado, five in Iowa, three in Massachusetts and four in New York. In each district, we interviewed district-level officials involved in special education and school Child Find processes. These officials included assistant superintendents, administrators, and directors of special education. While not generalizable, our interviews provided illustrative examples of a range of state and district Child Find processes, and the differences and challenges states and school districts face. To obtain information on the factors that may account for differences among states and school districts in the percentage of children receiving special education services and processes that states and school districts may use in implementing their Child Find requirements, we interviewed representatives from eight special education advocacy organizations that represent parents and families of individuals with disabilities and four special education subject matter specialists to discuss issues related to Child Find. Some of the issues we discussed included Early Intervention eligibility, assessment processes of students including Response to Intervention, and other topics to get a better sense of Child Find processes and issues. We conducted this performance audit from August 2017 to April 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Bill MacBlane (Assistant Director), Mindy Bowman (Analyst-in-Charge), Aaron Karty, Deborah Signer, Phillip Steinberg, and Shelia Thorpe made key contributions to this report. Also contributing to this report were James Bennett, Deborah Bland, Shilpa Grover, Serena Lo, Art Merriam, Sheila R. McCoy, Corinna Nicolaou, James Rebbe, Brian Schwartz, Daren Sweeney, and Kathleen van Gelder.", "summary": "About 13 percent of children aged 3 through 21 enrolled in public schools received special education services in school year 2015-16, and about 3 percent of children from birth through age 2 received special education services. The percentage of the population served under IDEA varies across states. For example, in fall 2016, the percentages of the population aged 6 through 21 served in individual states ranged from 6.4 percent to 15.1 percent. Concerns about the difficulties identifying and evaluating children for special education have been raised by the media, experts, and special education advocates. GAO was asked to examine how states implement Child Find and how Education monitors it. This report examines (1) factors that may account for differences in the percentage of children receiving special education services across states, and (2) how Education and selected states monitor and support Child Find efforts. GAO reviewed federal special education data, agency documentation, federal laws and regulations, and selected state laws; and interviewed Education officials, officials from four state agencies and 15 school districts in those states (Colorado, Iowa, Massachusetts, and New York), and representatives of organizations that advocate for families of individuals with disabilities as well as special education subject matter specialists. GAO selected the four states based on a variety of factors, including the percentage of special education students. Differences in states' eligibility criteria and the difficulty of identifying and evaluating some children suspected of having disabilities may contribute to differences in the percentages of children receiving special education services across states. The Individuals with Disabilities Education Act (IDEA), the primary federal special education law, requires states to have policies and procedures in place to ensure that all children with disabilities residing in the state who need special education services are identified, located, and evaluated. These policies and procedures—known as “Child Find”—are generally implemented by local school districts (see fig.). IDEA gives states some latitude in setting eligibility criteria and defining disability categories. In addition, states may determine their own processes for identifying and evaluating children. As a result, a child eligible for services in one state might be ineligible in another. According to advocates, special education subject matter specialists, and state and local officials GAO interviewed, a number of challenges related to correctly identifying and evaluating children suspected of having a disability can affect eligibility decisions. For example, school district officials in all four states GAO visited cited challenges in properly identifying and evaluating English Learner students, as districts do not always have staff who are conversant in a child's first language and skilled in distinguishing language proficiency from disabilities. The Department of Education (Education) monitors and supports Child Find efforts primarily by reviewing states' annual performance data and providing professional development and technical assistance. The four states GAO visited reported monitoring and supporting school districts' efforts in a similar manner to Education's.", "document_type": "gao"}
{"report": "The Coast Guard has 11 statutory missions, which are divided into homeland security and non-homeland security missions (see appendix II). The Coast Guard’s units that conduct operations to achieve its statutory missions are organized into shore-based forces such as boat stations, maritime patrol forces such as cutters and icebreakers, and Specialized Forces—the latter of which can serve as a force multiplier for the other units, such as by deploying for added capacity during homeland security missions, including port security, drug interdiction, and defense readiness. Table 1 details Specialized Forces teams, types of operations they conduct, and an example of an operation. Specialized Forces units deploy from their home locations, such as major U.S. port areas, to conduct operations in U.S. coastal waters and internationally. For example, some units such as MSRTs, MSSTs and PSUs deploy with specialized boats on trailers that can be towed or air lifted to the site of an antiterrorism patrol or defense readiness operation. Other Specialized Forces units do not maintain the vessels, such as cutters, or air assets, such as helicopters, from which they carry out operations. TACLETs, for example, do not maintain any boats and rely on and deploy via U.S. Navy or Allied vessels, as well as Coast Guard cutters, to conduct drug interdiction operations. Figure 1 shows Coast Guard personnel conducting a drug interdiction operation that included a TACLET member boarding a foreign, semi-submersible vessel, which resulted in seizing 17,000 pounds of cocaine. The Coast Guard is the lead federal maritime law enforcement agency on waters beyond 12 nautical miles offshore of the U.S. coast. The Coast Guard shares responsibility for patrolling the U.S. maritime borders and territorial sea (i.e., maritime approaches 12 nautical miles seaward of the U.S. coast) to interdict drugs and foreign nationals illegally entering the United States with U.S. Customs and Border Protection’s Air and Marine Operations and Border Patrol. Outside of DHS, the Department of Defense (DOD) is the lead federal agency for the detection and monitoring of the aerial and maritime transit of illegal drugs into the United States, and it operates systems, such as radar, that can be used in support of DHS and other federal, state, and local law enforcement activities. Figure 2 depicts the geographic areas in which Specialized Forces operate and resources, such as vessels or aircraft, used to support their operations. In July 2007, the Coast Guard reorganized the command structure of its Specialized Forces and aligned them as an independent Coast Guard command—the Deployable Operations Group. The Deployable Operations Group was intended to enhance operational effectiveness and interagency coordination in responding to a range of national emergencies and events, such as terrorist threats or natural disasters. Prior to the Deployable Operations Group, Specialized Forces aligned geographically under Atlantic Area and Pacific Area commands. In 2010, we found that the unified command structure achieved its intended benefits of standardized training and centrally managing assets. We also reported in 2010 that the Deployable Operations Group faced human resource challenges such as selecting qualified candidates and achieving and maintaining qualifications to perform certain high-skill techniques, such as vertical insertion from a helicopter onto the deck of a ship during maritime interdiction missions. Because of the ongoing program changes at that time, we did not make recommendations. In 2010, a DHS Inspector General report recommended a systematic review and analysis of the MSST program to determine, in part, the optimal staffing levels, training, and competency mix needed. The Coast Guard agreed and cited planned MSST program changes in its response to the Inspector General report. The Coast Guard’s Fiscal Year 2011 budget included a proposal to close five MSSTs and consolidate those forces to achieve cost savings, among other things. A 2011 Coast Guard report recommended that the Coast Guard integrate Specialized Forces units across the Coast Guard, balance the capacity of the Specialized Forces with proficiency and safety levels, and manage risk. In April 2013, the Coast Guard disbanded the Deployable Operations Group, and Specialized Forces units returned to regional commands. Figure 3 shows the evolution of Coast Guard Specialized Forces units since 1970. The April 2013 reorganization of the Coast Guard’s Specialized Forces units under regional commands more closely aligns with its original command structure that existed prior to the 2007 creation of the Deployable Operations Group. Figure 4 details the three command structures—pre-Deployable Operations Group, Deployable Operations Group, and Post-Deployable Operations Group. The Coast Guard generally applied three of five key practices for agency reform and partially applied two of five when developing its report that recommended the 2013 reorganization of its Specialized Forces units. Table 2 identifies the extent to which the Coast Guard’s reorganization applied key practices and considerations. Coast Guard generally applied three of five key practices for Specialized Forces reorganization, including establishing goals and outcomes, involving employees and key stakeholders, and addressing high risk areas and longstanding management challenges. Establishing goals and outcomes of reforms can help decision makers determine what problems genuinely need to be fixed, how to balance differing objectives, and what steps need to be taken to create long-term gains. The Coast Guard generally applied this key practice in its analysis of Specialized Forces units. For example, the Coast Guard’s 2011 report cites personnel safety as a main reason to reform Specialized Forces operations. According to Coast Guard officials, part of the rationale for this focus was because of a training mishap and a problem with equipment requirements. Specifically, officials stated that in 2010, a Coast Guard member drowned while training when he entered the water without self-inflating flotation equipment. Coast Guard officials told us that at the time of the incident, members of Specialized Forces units would carry in excess of 100 pounds of specialized gear and equipment. Officials also noted that at that time there were concerns that members’ self-inflating flotation devices could inflate onboard aircraft, which in the event of a crash in the water could result in personnel being unable to exit the aircraft. The Coast Guard subsequently established a goal for its reorganization to mitigate this safety risk by decreasing gear weight and personal flotation devices. Further, the report recommended reducing or eliminating inconsistencies between the Specialized Forces units and the rest of the Coast Guard. For example, a Coast Guard official told us that integration between the Deployable Operations Group and the rest of the Coast Guard was inconsistent, training programs were not standardized, and training took place at 15 different locations. This resulted in difficulties sharing assets, such as aircraft and boats, for use during training sessions. As a result of the report findings and its recommendation, the 2013 reorganization realigned Specialized Forces units under regional operational commands to integrate its logistics with the rest of the Coast Guard (fig 2.). Involving employees and key stakeholders in the process of developing reforms is part of an integrated approach that helps facilitate the development of reform goals and objectives, as well as incorporate insights from a frontline perspective and increase customer acceptance of any changes. The Coast Guard generally applied this key practice because it involved senior officials representing the agency to develop the goals of the reorganization, how to address them, and to make reform recommendations to improve the efficiency and effectiveness of Specialized Forces operations. The Coast Guard’s 2011 report included and incorporated input from a broad range of subject matter experts including high level officers representing a comprehensive mix of Coast Guard units, with a diverse mix of experience, and it reflected different programs throughout the Coast Guard to ensure a comprehensive review. During site visits, Coast Guard officials told us the reorganization from the Deployable Operations Group to Specialized Forces had a positive effect by helping to ensure tactics, training, and techniques became standardized and ensuring better cooperation within the Coast Guard as well as with other agencies. For example, Coast Guard officials told us that because their area of responsibility is large and busy, they use MSSTs to augment their local capabilities and to apply the MSST’s specialized capabilities that the local unit does not have. Coast Guard officials also emphasized an increase in safety, particularly with a decrease in a risk of drowning while in tactical gear. Addressing long standing weaknesses in how some federal programs and agencies operate is a key practice, which can improve the effectiveness and responsiveness of the federal government. The Coast Guard generally applied this key practice because it considered high risk areas when considering Specialized Forces reorganization. Specifically, the Coast Guard addressed retention and training, which it identified in its 2011 report to be high risk areas and longstanding management challenges. For example, the Coast Guard’s 2011 report identified the need for additional subject matter expertise and made recommendations to implement training standardization across the Specialized Forces. Our work has also identified retention and training as challenges. We found in 2010 that the Coast Guard was unable to retain qualified Specialized Forces personnel, in part because of additional training requirements. For example, while personnel working on a cutter may need a boat driver certification, an MSST or MSRT member would need an additional tactical boat driver course. The Coast Guard subsequently developed detailed guidance for Specialized Forces units that includes standardized training, requirements, and qualifications to be followed regardless of the unit location and to be applied consistently across organizational commands. During site visits to units in the Pacific and Atlantic Areas, we observed that equipment was standardized across Specialized Forces, and officials we spoke with described the benefits of the standardized training and equipment. Figure 5 shows Coast Guard MSST personnel conducting standardized training, which officials said has the added benefit of providing potential deterrence of illegal activities, such as drug smuggling, in the geographic area of the training. Coast Guard officials also told us that, prior to the 2007 reorganization to the Deployable Operations Group, Coast Guard personnel working in Specialized Forces units could not remain in those units and be competitive for promotions. Coast Guard officials told us that this was because the Coast Guard has certain requirements for career progression, including personnel working in various assignments within a given career path. In 2010, we reported that the Coast Guard had developed a career path for maritime law enforcement personnel—who are part of operations that generally address the Coast Guard’s homeland security missions. Coast Guard officials told us that this change was a response to challenges the agency faced retaining law enforcement personnel. Officials said this change created a maritime law enforcement career path within the Specialized Forces community. Coast Guard officials we spoke with also told that us the career path has helped them retain qualified Specialized Forces personnel. Coast Guard partially applied two of five key practices for agency reorganization, including using data and evidence, and considering to some extent the possibility of fragmentation, duplication, and overlap. However, it has not used data and evidence to fully assess Specialized Forces workforce needs and has not comprehensively evaluated the potential for overlaps or gaps in the capabilities among them. We have reported that agencies are better equipped to address management and performance challenges when managers use reliable data and evidence, such as evidence from program evaluations and performance data that provide information on how well a program or agency is achieving its goals. We have previously reported that when reforming a given program, the use of data and evidence is critical for setting program priorities and allocating resources. The Coast Guard used some data and evidence related to a specific management challenge—training mishaps—but did not use data and evidence to fully assess Specialized Forces workforce needs. As previously mentioned, the Coast Guard analyzed equipment weight data and scenarios and made recommendations based on the results of these analyses to reduce the risk of drowning. The 2011 report affirmed the locations of the Specialized Forces units to ensure that unit capabilities were geographically distributed, but it recommended additional analyses of some unit locations, such as TACLETs. The Coast Guard found that the geographic distribution of the Specialized Forces, at the time of the analysis, provided coverage for their tactical law enforcement and waterside operations and did not recommend changes to the geographic locations of these units. The Coast Guard partially applied this key practice because, when it reorganized its Specialized Forces command structure in 2013, it did not assess Specialized Forces workforce needs with regard to the number of personnel required to conduct its operations. The Coast Guard’s 2011 report identified some capability and capacity shortfalls, including inadequate capacity to conduct certain security operations, and recommended an analysis of staffing levels for all Specialized Forces units. Similarly, a 2012 Homeland Security Studies and Analysis Institute peer review of the Coast Guard’s 2011 report on its Specialized Forces noted the need for a more comprehensive analysis of all of the units to ensure the effective use of their specialized capabilities. In the eight years since the Coast Guard study recommended workforce needs analyses, the Coast Guard has not assessed the overall Specialized Forces workforce needs or established such an analysis as a priority. The Coast Guard conducted a unit level analysis of its PSUs in January 2014, but it did not use the results because the analysis focused on non-deployed personnel. Officials stated the analysis identified gaps in personnel and recommended that the Coast Guard expand the size of the units to be able to fulfill mission requirements. However, Coast Guard officials said they did not act on the recommendations of the study to request different resource levels. Officials told us that leadership changes among Specialized Forces can result in units, such as PSUs, getting study results based on scope decisions with which the new leader disagrees. We found that the Coast Guard might not have the right mix and number of personnel relative to the mix and number of operations Specialized Forces conduct to meet mission demands. Our analysis of Specialized Forces data for fiscal years 2016 through 2018 and planned for 2019 found variation in the number of operations requested of some units during this period, even though the number of personnel remained relatively constant. For example, our analysis of Coast Guard data found that PSU requests—and the number of operations carried out—changed from three operations in 2016 to six in 2018, with two operations planned in 2019, spread among a constant of approximately 1,000 personnel. In another example, our analysis of Coast Guard data found that the of number operations requested for MSSTs varied from 85 in 2016 to 67 in 2018, and 39 planned for 2019. Our analysis of Coast Guard data found that the number of MSST operations carried out was 152 in 2016, 141 in 2018, and 379 planned operations in 2019, while the number of personnel assigned to MSSTs decreased from 562 in 2016 to 547 planned for 2019. Such variations may affect the extent to which Specialized Forces units are used efficiently. Officials from some units we interviewed indicated that they experienced periods of underutilization, while other similar units turned down operations for lack of available personnel. For example, an official at one unit described efforts to increase the number of operations carried out by the unit, with officials describing outreach efforts to other Coast Guard units to encourage those units to call on them for specialized assistance. Officials at another unit conducted similar outreach, including passing out flyers describing Specialized Forces capabilities and contact information should the other Coast Guard units need assistance. In contrast, officials from a different Specialized Forces unit described instances where they had to decline operations because they did not have enough personnel to meet the demand. Further, an official from one Area Command responsible for assigning some of the Specialized Forces operations stated approximately 5 percent of requests for Specialized Forces assistance went unfulfilled. Without an analysis of the Specialized Forces units as a whole, the Coast Guard does not have the assurance that it has the requisite number of personnel in the right units to conduct the required missions. Such an analysis would better position the Coast Guard to identify capability gaps between mission requirements and mission performance caused by deficiencies in the numbers of personnel available, as required by the Coast Guard Authorization Act of 2015. Coast Guard officials from Specialized Forces units we interviewed in 2019 acknowledged that an analysis of each unit would be useful and in August 2019, officials from headquarters affirmed this and stated the Coast Guard aims to conduct analyses of the individual units. We found that these analyses consider each unit individually and do not comprehensively consider similar units, such as Specialized Forces. Therefore, without analyzing the Specialized Forces program as a whole, the Coast Guard may miss opportunities to optimize the allocation of personnel among Specialized Forces units, as well as the number of units. Using data and evidence to comprehensively assess workforce needs across Specialized Forces units would better position the Coast Guard to prioritize its Specialized Forces efforts to more effectively achieve desired outcomes. As we have reported since 2011, agencies may be able to achieve greater efficiency or effectiveness by reducing or better managing programmatic fragmentation, overlap, and duplication. We have also reported that these issues should be considered during agency reform efforts. We found that the Coast Guard partially considered how to reduce potential duplication and overlap when reorganizing the Specialized Forces units. The 2011 Coast Guard report identified some duplication of one specialized unit and challenges associated with uncoordinated training and fragmented guidance. The Coast Guard recommended the elimination of one Specialized Forces unit with that specialized capability, and to change training requirements to reduce the duplication of roles within one specific Specialized Forces unit. Further, the report recommended training standardization and associated guidance, which the Coast Guard subsequently addressed by updating its guidance and standardizing training requirements. In addition, the Coast Guard report recommended changes to the capabilities maintained by some units, such as MSSTs. Specifically, the report recommended that the Coast Guard focus MSSTs on waterside security capabilities and eliminate law enforcement roles, among others, to reduce duplicative training costs. Further, according to officials, in response to the Coast Guard Authorization Act of 2010, the Coast Guard eliminated the MSST in San Diego, California and replaced it with MSRT West, a second MSRT. The Coast Guard also placed all regional dive lockers under MSRT West. According to Coast Guard officials, structuring regional dive lockers under a single command in a single geographic location is safer and more efficient, because dive operations require a high level of subject matter expertise in the command as well as personnel actually participating in the dives. However, the Coast Guard partially applied this practice because it has not conducted the analyses necessary to fully identify potential overlap and the extent to which it could be unnecessarily duplicative. The Coast Guard categorizes Specialized Forces missions, such as drug interdiction or defense readiness, as primary, secondary or collateral, and assigns different levels of capabilities according to these categories. Specifically, multiple Specialized Forces are used to support the same Coast Guard missions, which often require similar capabilities from the units, such as the ability to perform enhanced law enforcement boardings. Figure 6 provides a visual representation of the Specialized Forces missions, the capabilities to carry out operations in support of those missions, and the units that address the mission areas. As shown in figure 6, MSSTs and PSUs primary and secondary missions overlap, as do the capabilities necessary to conduct three of the same missions—Ports, Waterways, and Coastal Security; Defense Readiness; and Search and Rescue. MSSTs and PSUs have operational differences, but there may be benefits to assessing when to use PSUs in place of MSSTs or vice versa, such as when one Specialized Force can be deployed more rapidly, or because Specialized Forces are located in close proximity. For example, MSSTs maintain the ability to deploy almost immediately to carry out an operation, while PSUs generally require around 24 months to deploy. PSUs generally have a deployment preparation cycle of at least 24 months and up to 48 months. Moreover, the variance in Specialized Forces utilization and the overlapping capabilities units maintain underscores a challenge and an opportunity, particularly given the close proximity of Specialized Forces units. For example, given that there are certain instances where Specialized Forces units appear to be substitutable, assessing the extent to which co-located units could be better leveraged could help the Coast Guard more efficiently manage its resources. Figure 7 shows the locations of Coast Guard Specialized Forces units and the close proximity of some units, such as co-located MSSTs and PSUs, which have overlapping primary and secondary defense readiness and ports waterways and coastal security missions (fig. 6) and related capabilities. In March 2019, as previously noted, Coast Guard leadership again called for a review of PSUs, citing overlap, personnel shortages, and excessive distance to training areas (such as waterways and weapon ranges). The challenge this new PSU study seeks to address underscores the importance of a contemporary and comprehensive assessment of these units’ workforce needs. It also presents the Coast Guard with an opportunity to consider whether it could more effectively use its co- located Specialized Forces. For example, instead of deploying a PSU within commuting distance of an operation occurring in San Francisco, CA that required surge capacity, the Coast Guard deployed an MSST from Seattle, WA for 7 days, even though both Specialized Forces are to maintain the same capabilities needed for the operation. Coast Guard officials stated that they decided to send the MSST to meet a surge capacity instead of the local PSU because Ports, Waterways and Coastal Security is a secondary mission for PSUs and PSUs do not bring law enforcement capability of boarding officers, among other things. According to Coast Guard officials, each PSU costs around $1 million a year to operate when not deployed to Guantanamo Bay, and two of eight PSUs are deployed annually. Assessing Specialized Forces workforce needs to determine the optimal mix of units and analyzing trade-offs, such as eliminating underutilized units, could identify opportunities for the Coast Guard to save millions of dollars over time. Elimination of even one PSU could save around a million dollars annually. Because the exact amount of savings would depend on the outcomes of those analyses and currently available cost data is not available for making estimates, we cannot precisely estimate the value of potential savings. However, given that the Coast Guard has begun an assessment of PSUs, it is reasonable to expect that a comprehensive analysis of Specialized Forces could find unnecessary duplication and could recommend PSU closures. Coast Guard officials did not state that they are considering this review as part of a comprehensive review of Specialized Forces that would include assessing the overlapping capabilities of other Specialized Forces units. In August 2019, Coast Guard officials told us that overlap or gaps in Specialized Forces’ capabilities could exist. Coast Guard officials also stated that some overlapping capability could be beneficial. While overlap may be beneficial, overlapping capabilities, if unnecessary, could indicate inefficiencies, such as excess capacity in some areas, including geographic areas, to the detriment of others where there may be capability gaps. The Coast Guard is not currently positioned to take action to reduce the risk of some potentially unnecessary overlap or duplication among the Specialized Forces units because it has yet to comprehensively assess the Specialized Forces program. Specifically, as reported above, Coast Guard officials stated that the Coast Guard has conducted some staffing analyses of standalone Specialized Forces units, but has not evaluated the Specialized Forces’ workforce or operations as a whole. Until the Coast Guard comprehensively assesses Specialized Forces’ needs, the Coast Guard will lack a complete picture of the extent to which overlapping capabilities are necessary or appropriate, or where there are capability gaps or areas where certain Specialized Forces units could be better leveraged to meet mission requirements. Assessing the extent to which unnecessary overlap or duplication exists among Specialized Forces’ capabilities, would better position the Coast Guard to identify capability gaps and reallocate resources, as needed, to use them more efficiently. The Coast Guard’s Specialized Forces units include a range of specialized capabilities that are vital to the agency’s ability to fulfill its mission, and they constitute a significant force multiplier to maintain readiness throughout major U.S. ports and cities. The Coast Guard faces the difficult decision of determining how best to invest its limited resources. Without having assessed its operational needs and mix of personnel for Specialized Forces units, the Coast Guard does not have the information it needs to ensure that it is investing its resources efficiently. GAO’s key practices and considerations provide a framework for agency reorganization and a decision-making approach that can help ensure that resources are allocated efficiently and do not result in unnecessary overlap or duplication. The Coast Guard did not fully apply these practices when reorganizing the Specialized Forces. By comprehensively assessing Specialized Forces’ workforce needs and determining the extent to which overlapping capabilities are necessary, or whether capability gaps may exist, the Coast Guard may be able to more efficiently allocate resources for its Specialized Forces. The Coast Guard should conduct a comprehensive analysis of its Deployable Specialized Forces’ workforce needs. (Recommendation 1) The Coast Guard should assess the extent to which unnecessary overlap or duplication exists among Deployable Specialized Forces’ capabilities. (Recommendation 2) We provided a draft of this report to DHS for comment. DHS provided technical comments, which we incorporated as appropriate. On November 5, 2019, DHS also provided comments, reproduced in full in appendix III. DHS concurred with one of our two recommendations, and described actions planned to address it, but did not concur with the other. DHS concurred with our first recommendation that the Coast Guard should conduct a comprehensive analysis of its Specialized Forces’ workforce needs. DHS stated in its comments that the Coast Guard will conduct individual unit analyses, prioritizing for units that were not previously examined. Initial requests, according to the comments, will be submitted to staff responsible for the analyses by January 31, 2020, and estimated completion dates for the analyses are expected to be determined after assessing the availability of funding to support the analyses. These actions, if fully implemented, should address the intent of the recommendation. DHS did not concur with our second recommendation that the Coast Guard assess the extent to which unnecessary overlap or duplication exists among Specialized Forces’ capabilities. In its comments, DHS stated that when the priority of the missions, capabilities, and subsequent geographic operating areas are appropriately considered for each DSF unit type, unnecessary overlap or duplication does not exist among DSF capabilities. DHS further stated that our conclusions illustrate a fundamental misunderstanding of the corresponding missions of DSF units. We note in our report that the way in which the Coast Guard deploys certain Specialized Forces units may not result in overlap, but overlapping capabilities amongst units could indicate inefficiencies in how they are used, such as excess capacity in some areas, including geographic areas, and missed opportunities for use in others. As noted in our report, the Coast Guard has not conducted the analyses necessary to fully identify potential overlap amongst units’ capabilities and the extent to which opportunities may exist to use the units more efficiently. The Coast Guard categorizes Specialized Forces missions, such as drug interdiction or defense readiness, as primary, secondary, or collateral, and assigns different levels of capabilities according to these categories. We found that multiple Specialized Forces are used to support the same Coast Guard missions, which often require similar capabilities from the units, such as the ability to perform enhanced law enforcement boardings. Further, as stated in our report, in August 2019, Coast Guard officials told us that overlap or gaps in Specialized Forces’ capabilities could exist and that some overlapping capability could be beneficial. While overlap may be beneficial, overlapping capabilities, if unnecessary, could indicate inefficiencies, such as excess capacity in some areas, including geographic areas. Also in its comments, DHS stated that we have not identified any substantive examples of unnecessary overlap or duplication nor provided any other compelling reasons for how implementing this recommendation could enhance Coast Guard mission effectiveness. DHS cited our use of MSST and PSU potential overlap as an example of misunderstanding DSF unit missions and active versus reserve personnel. However, MSST and PSU potential overlap is a prime example of why potential unnecessary overlap should be examined by the Coast Guard. Specifically, as noted in our report, MSST and PSU primary and secondary missions overlap, as do the capabilities necessary to conduct three of the same missions—Ports, Waterways, and Coastal Security; Defense Readiness; and Search and Rescue. MSSTs and PSUs have operational differences due to active versus reserve personnel status, but there may be benefits to assessing when to use PSUs in place of MSSTs or vice versa, such as when one Specialized Force can be deployed more rapidly, or when Specialized Forces are located in close proximity. Beyond the MSST and PSU potential overlap, active duty units such as MSSTs and MSRTs provide an additional example. As shown in Figure 6 of our report, MSRTs and MSSTs share the primary missions of Ports, Waterways, and Coastal Security, as well as common secondary missions, including Drug and Migrant Interdiction. Additionally, MSRT- San Diego and MSST-Long Beach are within close proximity to one another, offering an opportunity to examine potential overlap. Also, as noted in our report, officials from some units we interviewed indicated that they experienced periods of underutilization, while other similar units turned down operations for lack of available personnel. For example, an official at one unit described efforts to increase the number of operations carried out by the unit, with officials describing outreach efforts to other Coast Guard units to encourage those units to call on them for specialized assistance. Officials at another unit conducted similar outreach, including passing out flyers describing Specialized Forces capabilities and contact information should the other Coast Guard units need assistance. In contrast, officials from a different Specialized Forces unit described instances where they had to decline operations because they did not have enough personnel to meet the demand. Given that there are certain instances where Specialized Forces units appear to be substitutable, assessing the extent to which units could be better leveraged could help the Coast Guard more efficiently manage its resources. In addition, in March 2019, Coast Guard leadership called for a review of PSUs, citing overlap, personnel shortages, and excessive distance to training areas (such as waterways and weapon ranges). As noted in our report, the challenge this new PSU study seeks to address underscores the importance of a contemporary and comprehensive assessment of these units’ workforce needs and presents the Coast Guard with an opportunity to consider whether it could more effectively use its co- located Specialized Forces. According to Coast Guard officials, each PSU costs around $1 million a year to operate when not deployed to Guantanamo Bay, and two of eight PSUs deploy annually. Assessing Specialized Forces workforce needs to determine the optimal mix of units and analyzing trade-offs, such as eliminating or moving underutilized units, could identify opportunities for the Coast Guard to save millions of dollars over time. As noted in our report, because the exact amount of savings would depend on the outcomes of those analyses and cost data that is not currently available for making estimates, we cannot precisely estimate the value of potential savings. However, given that the Coast Guard has begun an assessment of PSUs, it is reasonable to expect that a comprehensive analysis of Specialized Forces could provide either a defensible basis for the existing structure or find unnecessary duplication and could recommend changes to the number and location of Specialized Forces. In summary, as we state in our report, a comprehensive assessment of Specialized Forces’ needs would, among other things, help the Coast Guard have a more complete picture of the extent to which certain Specialized Forces units could be better leveraged to meet mission requirements. Assessing the extent to which unnecessary overlap or duplication exists among Specialized Forces’ capabilities would better position the Coast Guard to identify capability gaps and reallocate resources, as needed, to use them more efficiently. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or AndersonN@gao.gov. GAO staff who made key contributions to this report are listed in appendix IV. This appendix provides Coast Guard data on Deployable Specialized Force (Specialized Forces) personnel, operations, costs, and resource hours showing a mix of operational tempos, including variation in the number of operations requested of some units such as Tactical Law Enforcement Teams (TACLETs) and Port Security Units (PSUs), but relatively constant numbers of personnel assigned to them. Table 3 provides operational and cost details for Coast Guard Specialized Forces units for fiscal years 2016 through 2018 and planned for 2019. This appendix details the Coast Guard’s 11 missions, which are characterized as non-homeland security and homeland-security missions (see Table 4). Nathan J. Anderson, (202) 512-3841 or AndersonN@gao.gov. In addition to the contact above, Ben Atwater (Assistant Director), Andrew Curry (Analyst-in-Charge), Chuck Bausell, David Dornisch, Michele Fejfar, Tracey King, and Calaera Powroznik made key contributions to this report. Also contributing were: Jason Berman, Dominick Dale, Eric Hauswirth, and Jan Montgomery.", "summary": "The U.S. Coast Guard, within the Department of Homeland Security (DHS), is the principal federal agency charged with ensuring the security and safety of the waters under U.S. jurisdiction. To help carry out its missions, the Coast Guard maintains Specialized Forces units with the capabilities needed to handle drug interdiction, terrorism, and other threats to the U.S. maritime environment. The Coast Guard reorganized the command structure of these units in 2007 and again in 2013. The Maritime Security Improvement Act of 2018 included a provision for GAO to evaluate Specialized Forces units and provide a report to Congress. This report examines the extent to which the Coast Guard addressed key practices and considerations for assessing reorganization of its Specialized Forces units. GAO assessed the Coast Guard report and associated workforce planning documentation and data used for its 2013 reorganization and analyzed the extent to which the agency applied key practices. GAO also analyzed guidance and data on Specialized Forces capabilities and operations to identify potential overlap or gaps and interviewed agency officials. In reorganizing its Deployable Specialized Forces (Specialized Forces) in 2013, the Coast Guard generally applied three of five key practices for agency reorganization, including establishing goals and outcomes, engaging stakeholders, and addressing longstanding management challenges, such as training shortfalls. However, the Coast Guard did not fully apply the other two key practices—using data and evidence and addressing potential overlap and duplication within the Specialized Forces workforce. For example: The Coast Guard has not assessed the overall Specialized Forces workforce needs, as this practice recommends. Officials from some units stated that they experienced periods of underutilization, while other units with the same or similar capabilities turned down operations for lack of available personnel. GAO identified some overlap among the capabilities of the different Specialized Forces units and the Coast Guard missions they support—in some cases Specialized Forces units were co-located with other Specialized Forces units with many of the same capabilities and similar missions. In August 2019, Coast Guard officials acknowledged that the 2013 reorganization did not conduct an analysis of potential overlap or duplication of capabilities and agreed that overlap or gaps in Specialized Forces capabilities could exist. Assessing workforce needs and the extent to which unnecessary overlap or duplication may exist among Specialized Forces would help ensure that the agency effectively allocates resources and uses them efficiently. GAO makes two recommendations to DHS. First, GAO recommends that the Coast Guard conduct an analysis of its Specialized Forces' workforce needs, with which DHS concurred. Second, GAO recommends that the Coast Guard assess the extent to which unnecessary overlap or duplication exists. Although DHS did not concur, GAO continues to believe the findings documented in the report support the recommendation.", "document_type": "gao"}
{"report": "Under the APA, agencies engage in three basic phases of the rulemaking process: they initiate rulemaking actions, develop proposed rulemaking actions, and develop final rulemaking actions. Built into agencies’ rulemaking processes are opportunities for internal and external deliberations, reviews, and public comments. Figure 1 provides an overview of the rulemaking process. The public comment portion of the rulemaking process generally comprises three phases: 1. Comment Intake: During this phase, agencies administratively process comments. This may include identifying duplicate comments (those with identical or near-identical comment text, but unique identity information), posting comments to the agency’s public website, and distributing comments to agency subject-matter experts within responsible program offices for analysis. 2. Comment Analysis: During this phase, subject-matter experts analyze and consider submitted comments. This may include the use of categorization tools within FDMS or outside software systems. 3. Comment Response: During this phase, agencies prepare publicly available responses to the comments in accordance with any applicable requirements. Agencies are required to provide some response to the comments in the final rule, but in some cases, an agency may also prepare a separate report to respond to the comments. As illustrated in figure 1 above, the public has the opportunity to provide input during the development of agencies’ rules. Among other things, the APA generally requires agencies to publish an NPRM in the Federal Register; allow any interested party an opportunity to comment on the rulemaking process by providing “written data, views, or arguments”; issue a final rule accompanied by a statement of its basis and purpose; and publish the final rule at least 30 days before it becomes effective. The APA requires agencies to allow any interested party to comment on NPRMs. The APA does not require the disclosure of identifying information from an interested party that submits a comment. Agencies therefore have no obligation under the APA to verify the identity of such parties during the rulemaking process. Instead, the APA and courts require agencies to consider relevant and substantive comments, and agencies must explain their general response to them in a concise overall statement of basis and purpose, which in practice forms part of the preamble of the final rule. Courts have explained that significant comments are comments that raise relevant points and, if true or if adopted, would require a change in the proposed rule. However, courts have held that agencies are not required to respond to every comment individually. Agencies routinely offer a single response to multiple identical or similar comments. As explained by Regulations.gov’s “Tips for Submitting Effective Comments,” “the comment process is not a vote,” and “agencies make determinations for a proposed action based on sound reasoning and scientific evidence rather than a majority of votes. A single, well-supported comment may carry more weight than a thousand form letters.” The APA includes provisions on the scope of judicial review that establishes the bases under which a court shall find an agency’s action unlawful. Among these APA bases are when the court finds that agency action, findings, and conclusions were “arbitrary, capricious, an abuse of discretion, or otherwise not in accordance with law” and “without observance of procedure required by law.” How an agency managed and considered public comments may be relevant during judicial review. For example, one basis for a court’s reversal of an agency action has been that, upon review of the statement of basis and purposes, the court concludes the agency failed to consider or respond to relevant and significant comments. Conversely, courts have upheld agency rules when the courts have found the statement of basis and purposes demonstrate the agency considered the commenter’s arguments. The E-Government Act of 2002 requires agencies, to the extent practical, to accept comments “by electronic means” and to make available online the public comments and other materials included in the official rulemaking docket. Executive Order 13563 further states that regulations should be based, to the extent feasible, on the open exchange of information and perspectives. To promote this open exchange, to the extent feasible and permitted by law, most agencies are required to provide the public with a meaningful opportunity to participate in the regulatory process through the internet, to include timely online access to the rulemaking docket in an open format that can be easily searched and downloaded. Most agencies meet these responsibilities through Regulations.gov, a rulemaking website where users can find rulemaking materials and submit their comments, but all agencies are not required to use that platform. In October 2002, the eRulemaking Program was established as a cross-agency E-Government initiative and is currently based within EPA. The eRulemaking PMO leads the eRulemaking Program and is responsible for developing and implementing Regulations.gov, the public- facing comment website, and FDMS, which is the agency-facing side of the comment system used by participating agencies. As of March 2018, Regulations.gov identified 180 participating and 128 nonparticipating agencies. These agencies may be components of larger departmental agencies. Some nonparticipating agencies, including FCC and SEC, have their own agency-specific websites for receiving public comments. The comment systems within the scope of this report are as follows: FDMS and Regulations.gov: FDMS is federal government-wide document management system structured by dockets (or file folders) that offer an adaptable solution to service a wide range of regulatory activities routinely performed by federal agencies. The public-facing website of FDMS is Regulations.gov, which is an interactive website that allows the public to make comments on regulatory documents, review comments submitted by others, and access federal regulatory information. Regulations.gov allows commenters to submit comments to rulemakings by entering information directly in an electronic form on the Regulations.gov website. This form also allows commenters to attach files as part of their comment submission, and can be customized by each participating agency. Appendix II provides an example of one comment form from Regulations.gov. Additionally, all participating agencies allow comments to be submitted by mail or hand delivery. At their discretion, some participating agencies also allow comments to be submitted via email. See table 1. FCC’s Electronic Comment Filing System (ECFS): ECFS is a web- based application that allows anyone with access to the internet to submit comments to FCC rulemaking proceedings. ECFS allows commenters to submit comments to rulemakings through two main avenues: brief text comments submitted as Express filings, and long- form comments submitted as Standard filings. Both types of filings can be submitted through an ECFS comment form, which requires commenters to enter information directly into an electronic form on the ECFS website. See appendix III for examples of the comment forms used by ECFS. Additionally, interested parties with the appropriate technical capabilities can submit either type of filing directly to ECFS via a direct application programming interface (API) or through a public API that is registered with the website Data.gov. Filing comments through an API allows interested parties the ability to file a large number of comments without having to submit multiple individual comment forms. Finally, to accommodate a large volume of comment submissions for the 2015 Open Internet rulemaking, FCC allowed interested parties to submit Express comment filings in bulk through formatted CSV files that were submitted via a dedicated email address and then uploaded into ECFS. Similarly, for the 2017 Restoring Internet Freedom rulemaking, FCC allowed commenters to submit Express comment filings in bulk through a dedicated file- sharing website, and the comments were then uploaded into ECFS. With the exception of these two rulemakings, FCC does not allow comments to be submitted electronically outside of ECFS. Figure 2 shows how ECFS facilitates public commenting by using the processes discussed above. SEC’s Comment Letter Log: When SEC requests public comments on SEC rule proposals, the public can submit comments to rulemakings through an online form, which requires commenters to enter information in an electronic form on SEC’s website. This form also allows commenters to attach files as part of their submission. When commenters submit a comment, it is sent to SEC staff as an email. SEC also allows comments to be submitted via email and mail. After review, staff upload the comment and any associated data into the Comment Letter Log, which is the internal database that SEC staff use to manage the public comment process, and post the comment to the public website. See appendix IV for an example of a comment form on SEC’s website. Consistent with the discretion afforded by the APA, Regulations.gov and agency-specific comment websites use required and optional fields on comment forms to collect some identity information from commenters. In addition to the text of the comment, each participating agency may choose to collect identity information from the Regulations.gov comment form by requiring commenters to fill in other fields, such as name, address, and email address before they are able to submit a comment. Participating agencies may also choose to collect additional identity information through optional fields. For example, while EPA does not make any fields associated with identity information available to commenters, CFPB makes all fields available and requires that commenters enter something into the first name, last name, and organization name fields before a comment can be submitted. Table 2 shows the fields on Regulations.gov in which each of the participating agencies we analyzed require commenters to enter information and the optional fields available for commenters to voluntarily enter information. FCC requires that all commenters complete the following fields on both the Standard and Express comment forms in ECFS: (1) name, (2) postal address, and (3) the docket proceeding number to which they are submitting a comment. The ECFS comment form also allows commenters to voluntarily provide additional information in optional fields, such as email address. Similarly, SEC’s comment forms require commenters to provide (1) first and last name, (2) email address, and (3) the comment content, before a comment can be successfully submitted. The comment form also allows commenters to voluntarily provide other information in optional fields, such as their city and state. Regardless of the fields required by the comment form, the selected agencies all accept anonymous comments in practice. Specifically, in the comment forms on Regulations.gov, ECFS, and SEC’s website, a commenter can submit a comment under the name “Anonymous Anonymous,” enter a single letter in each required field, or provide a fabricated address. In each of these scenarios, as long as a character or characters are entered into the required fields, the comment will be accepted. Further, because the APA does not require agencies to authenticate submitted identity information, neither Regulations.gov nor the agency-specific comment websites contain mechanisms to check the validity of identity information that commenters submit through comment forms. As part of the Regulations.gov modernization effort, the Office of Information and Regulatory Affairs (within the Office of Management and Budget) and the Department of Justice proposed language for a disclosure statement on every comment form that would require the commenter to acknowledge that they are not using, without lawful authority, a means of identification of another person with any comment they are submitting. Commenters would be required to acknowledge their agreement with the statement before their comment could be submitted. According to PMO officials, even with this disclosure statement, anonymous comments would still be permitted and accepted by Regulations.gov. This disclosure statement was proposed in response to allegations of comments being submitted to rulemakings on behalf of individuals without their permission. As of April 2019, this proposed language has not yet been approved by the Executive Steering Committee for Regulations.gov. However, the proposed disclosure statement would be provided on the Regulations.gov comment form, and it is unclear whether similar information would be made available to commenters submitting comments via email or mail. In contrast to the other selected agencies, according to FCC officials, FCC rules require the submission of the commenter’s name and mailing address, or the name and mailing address of an attorney of record. However, in March 2002, FCC initiated a rulemaking related to the submission of truthful statements to the commission. Among other issues, FCC sought comment on whether rulemaking proceedings should be subject to an already existing rule that prohibited the submission of written misrepresentations or material omissions from entities that are subject to FCC regulation. In its final rule, issued in March 2003, FCC decided to continue to exempt comments to rulemakings from this rule because of the potential that such a requirement would hinder full and robust public participation in such policy-making proceedings by encouraging disputes over the truthfulness of the parties’ statements. According to FCC officials, to comply with APA requirements, the commission tries to minimize barriers that could prevent or discourage commenters from participating in the commenting process, and in practice accepts anonymous comments. See figure 3 for an example of an anonymous comment in ECFS. Additionally, in our survey of program offices with rulemaking responsibilities at selected agencies, 39 of 52 offices reported that they received anonymous comments on some rulemakings for which their office has been responsible since 2013. The remaining 13 offices responded that they did not receive or were unaware of receiving anonymous comments, though most of these offices do not have high levels of rulemaking activity or receive a high volume of comments. Regulations.gov and agency-specific comment websites also collect some information about public users’ interaction with their websites through application event logs and proxy server logs. This information, which can include a public user’s Internet Protocol (IP) address, browser type and operating system, and the time and date of webpage visits, is collected separately from the comment submission process as part of routine information technology management of system security and performance. The APA does not require agencies to collect or verify this type of information as part of the rulemaking process. Regulations.gov collects some information from commenters accessing the website but it is never linked to any specific comment. In Regulations.gov, proxy server logs capture information such as the country from which a user accesses the site, the user’s browser type and operating system, and the time and date of each page visit on the website. According to PMO officials, this information is provided to the eRulemaking PMO in summary statistics that are used to assess what information is of least interest to Regulations.gov visitors, determine technical design specifications of the website, and identify system performance problems. This information is collected about all public users visiting Regulations.gov, regardless of whether they submit a comment. Further, because the PMO receives this information in the form of summary statistics, it cannot be connected to any specific comment. The eRulemaking PMO also monitors IP addresses that interact with Regulations.gov via security firewalls, but, according to PMO officials, the web application firewall (WAF) logs (a type of application event log) have never been connected to specific comments, though in some cases the URL the blocked user was attempting to access may be captured in the log. FCC officials stated that the current ECFS application architecture does not facilitate FCC identifying the source IP address of the submitter of a specific comment filed in ECFS. FCC collects information about public users’ interactions with ECFS through its web-based application proxy server logs, including the IP address from which a user accesses the site and the date and time of the user’s interaction. However, ECFS does not obtain or store IP addresses as part of the comment data it collects when a public user ultimately submits a comment. Within the current architecture, ECFS would require officials to match date and time stamps from the proxy server log to the ECFS comment data to connect a given IP address to a specific comment. SEC officials stated it would be difficult to match the large number of daily hits to their general website to the much smaller number of comments submitted to their rulemaking proceedings. SEC collects information about public users’ interactions with the SEC.gov website through proxy server logs, including the IP address from which a user accesses the website and the user’s date, time, and URL requests. However, according to officials, a public user never directly interacts with the Comment Letter Log, and none of the information from the proxy log is included as part of the data it collects in association with comment submissions. Despite this difficulty, SEC officials stated that linking the proxy log data from the general SEC.gov website to a specific comment in the Comment Letter Log could be done on a case-by-case basis. Seven of 10 selected agencies have documented some internal guidance associated with the identity of commenters during the three phases of the public comment process, but the substance of this guidance varies, reflecting the differences among the agencies and their respective program offices. For example, as shown in table 3, BLM has no internal guidance related to identity information, while CFPB has internal guidance related to the comment intake and response to comments phases. For selected agencies that have guidance associated with the identity of commenters, it most frequently relates to the comment intake or response to comment phases of the public comment process. The guidance for these phases addresses activities such as managing duplicate comments (those with identical or near-identical comment text but varied identity information) or referring to commenters in a final rule. In addition, some agencies have guidance related to the use of identity information during comment analysis. Agencies are not required by the APA to develop internal guidance associated with the public comment process generally, or identity information specifically. For the three selected agencies that did not have identity-related guidance for the public comment process, cognizant officials told us such guidance has not been developed because identity information is not used as part of their rulemaking process. For example, BLM officials stated that the only instance in which identity information would be considered is when threatening comments are referred to law-enforcement agencies. According to our analysis of the internal guidance the selected agencies provided, five of the 10 agencies have documented identity-related guidance associated with the comment intake phase. (See table 4.) Identity-related guidance for the comment intake phase addresses posting comments and their associated identity information to public comment websites. The guidance generally falls into two categories: (1) the treatment of duplicate comments (those comments with identical or near-identical content, but unique identity information) and (2) the management of comments reported to have been submitted using false identity information. Four of the 10 selected agencies have documented guidance on defining and posting duplicate comments, which may also be referred to as mass mail campaigns. However, in accordance with the discretion afforded them under the APA, agency definitions of duplicate comments and recommendations on how to manage them during comment intake vary. Specifically, for EBSA and WHD—the selected agencies within the Department of Labor (DOL)—one comment letter with multiple signers is considered one comment, while the same comment submitted by multiple signers as separate letters is counted separately. In both cases, however, each individual signer may provide unique identity information. In contrast, EPA guidance states that mass mail submissions often include attachments containing either bundled duplicate messages or a single comment with multiple signatures. For EPA, each signature is counted as a duplicate comment submission. As of February 2019, CFPB’s draft guidance does not explicitly define duplicate comments, but it does note that “duplicate identical submissions” are not subject to the agency’s policy of posting all comments. Instead, the official responsible for managing the docket during comment intake may remove duplicate comments from posting or decide not to post them. According to CFPB officials, this policy is only applicable to comments that contain entirely identical comment content and identity information, and does not apply to mass mailing campaigns. Similarly, when DOL agencies receive duplicate comments as part of mass mail campaigns, the agency can choose to post a representative sample of the duplicate comment to Regulations.gov along with the tally of the duplicate or near-duplicate submissions, or post all comments as submitted. EPA guidance states that duplicate comments submitted as part of mass mailings are to be posted as a single primary document in Regulations.gov with a tally of the total number of duplicate comments received from that campaign. However, as discussed later in this report, EPA may post all duplicate comments it receives, depending on the format in which they are submitted. Five of the 10 selected agencies have documented internal guidance on how to manage posting comments that may have been submitted by someone falsely claiming to be the commenter. However, the procedures related to addressing comments with potentially false identity information also vary among agencies. For EBSA and WHD, guidance from DOL states that if a comment was submitted by someone falsely claiming to be the commenter, the identifying information is to be removed from the comment and the comment is treated as anonymous and remains posted. In cases where an individual claims that a comment was submitted to CFPB or SEC using the individual’s identity information without his or her consent, both agencies’ guidance provide staff with discretion to redact, reattribute, or otherwise anonymize the comment letter in question. According to internal guidance from CFPB, EPA, and SEC, if agency officials are able to confirm that a comment was submitted by someone falsely claiming to be the commenter, such as by the agency sending an email to the address associated with the comment, the comment may not be made available to the public. SEC officials stated that although they have discretion to remove the comment from public posting, the typical response is to encourage the individual making the claim to submit another comment correcting the record. Similarly, if a member of the public contacts EPA claiming that a comment was submitted using his or her identity information without consent and agency staff cannot confirm it, EPA guidance directs staff to ask the requester who submitted the claim to submit another comment to the docket explaining that the original comment was submitted without the individual’s consent. Both comments will be included in the docket. According to our analysis of the guidance the selected agencies provided, four of the 10 agencies have identity-related guidance for the comment analysis phase (see table 5). Identity-related guidance for the comment analysis phase includes criteria for coding comments for analysis, including by identifying the type of commenter (such as an individual or interest group). CMS guidance states that, during review, comments should be separated by issue area and tables may be used to assist in the grouping of comments and preparing briefing materials. While this guidance notes that these tables may be used to group commenters based on their identity during review, when summarizing comments later in the process the guidance indicates that CMS officials should avoid identifying commenters by name or organization. FDA training materials address how to prepare comment summaries to help ensure the agency has properly identified all comments regarding an issue. To conduct a quality-control check on the comment review process, FDA sorts the comments by commenter and reviews the comments from a sample of key stakeholders, including interested trade associations and consumer or patient groups, to confirm that relevant issues were identified. For EBSA and WHD, guidance from DOL recommends attaching the “organization name” to comments within a docket to improve transparency and help the agency and public users search for organizations within Regulations.gov. In addition, DOL guidance suggests flagging comments for additional review, including at least one flag based on identity. According to our analysis of the guidance the selected agencies provided, five of the 10 agencies have documented identity-related guidance for responding to comments. (See table 6.) Identity-related guidance for the response to comments phase includes guidance for agency officials on how, if at all, to address identity information related to comments in developing the final rule. As discussed previously, during comment analysis, CMS guidance indicates that officials should avoid identifying commenters by name or organization when summarizing comments. These summaries may then be used as a basis for the agency’s formal comment summary included in the preamble of the final rule. CFPB guidance states that a summary of the rulemaking process should be developed for the preamble of the final rule and include how many comments are received and from which type of commenter. CFPB is to describe both the commenters and comments in general terms rather than identify commenters by name or entity. For example, rather than naming a specific financial institution, CFPB may refer to “industry commenters” in the final rule. For EBSA and WHD, guidance from DOL states that when several commenters suggest the same approach to revising or modifying the proposed rule, the names of specific commenters can be cited as a list in a footnote. When choosing which commenter should appear first in the list, DOL agencies are to select the commenter with the strongest or most detailed discussion on the issue. However, it is not necessary to always identify commenters by name, and, according to DOL officials, the department’s general practice is not to do so. Instead, the agency may use phrases such as “several commenters,” or “comments by the ABC Corporation and others.” DOL agencies may also reference commenters by type rather than name, using terms including “municipal agency, state workforce agency, employer, academic representative, agency, and industry,” among others. FDA training materials recommend that the final rule include a very brief explanation of the number and scope of comments to the proposed rule, including who submitted them. Commenters are not identified as individuals, but rather by commenter type, such as trade associations, farms, or consumer advocacy organizations, among others. Within the discretion afforded by the APA, the 10 selected agencies’ treatment of identity information during the comment intake, comment analysis, and response to comments phases of the public comment process varies. Selected agencies differ in how they treat identity information during the comment intake phase, particularly in terms of how they post duplicate comments, which can lead to identity information being inconsistently presented to public users of comment systems. Selected agencies’ treatment of identity information during the comment analysis phase also varies. Specifically, program offices with responsibility for analyzing comments place varied importance on identity information during the analysis phase. All agencies draft a response to comments with their final rule, but the extent to which the agencies identify commenters or commenter types in their response also varies across the selected agencies. Within the discretion afforded by the APA and E-Government Act, selected agencies vary in how they treat identity information during the comment intake phase, which includes identifying duplicate comments and posting comments to the public website. Further, the way in which the selected agencies treat comments during the comment intake phase results in identity information being inconsistently presented on the public website. Generally, officials told us that their agencies either (1) maintain all comments within the comment system, or (2) maintain some duplicate comment records outside of the comment system, for instance, in email file archives. Specifically, officials from four selected agencies (CMS, FCC, FDA, and WHD) stated that they maintain all submitted comments in the comment system they use. Officials from the other six agencies (BLM, CFPB, EBSA, EPA, FWS, and SEC) stated that their agencies maintain some comment records associated with duplicate comments outside of the comment system. Among the four agencies that maintain all submitted comments within their comment system, our review of comment data showed that practices for posting duplicate comments led to some identity information or comment content being inconsistently presented on the public website. For example, according to CMS officials responsible for comment intake, CMS may post all duplicate comments individually, or post duplicate comments in batches. When duplicate comments are posted in batches, the comment title will include the name of the submitting organization followed by the total number of comments. However, as discussed previously, CMS does not have any documented policies or guidance associated with the comment intake process, and we identified examples where the practices described by CMS officials differed. On one CMS docket, for instance, staff entered more than 37,000 duplicate comments individually, with the commenter’s name and state identified in the comment title. However, the attached document included with each of the posted comments was an identical copy of one specific comment containing a single individual’s identity information. While all the individual names appear to have been retained in the comment titles, and the count of total comments is represented, any additional identity information and any potential modifications made to each duplicate comment submitted have not been retained either online or outside of FDMS, and are not presented on the public website. (See fig. 4.) Similarly, although our analysis of WHD comments did not suggest that any comments were missing from Regulations.gov, on one WHD docket almost 18,000 duplicate comments were associated with a single comment with one individual’s name identified in the comment title. While all of the comments are included within 10 separate attachments, none of the identity information included with these comments can be easily found without opening and searching all 10 attachments, most of which contain approximately 2,000 individual comments. (See fig. 5.) Our review of comment data showed that the selected agencies that maintain some comment records outside of the comment system (six of 10) also follow practices that can inconsistently present some identity information or comment content associated with duplicate comments. For BLM and FWS, agency officials responsible for comment intake stated that all comments received through Regulations.gov are posted, but a single example may be posted when duplicate paper comments are received. As discussed previously, neither BLM nor FWS have internal guidance or policy associated with comment intake. For CFPB, EBSA, EPA, and SEC, the agency may post a single example along with the total count of all duplicate comments, but does not necessarily post all duplicate comments online. Thus, identity information and unique comment contents for all duplicate comments may not be present on the public website. For example, on one CFPB comment, the agency posted an example of a submitted comment containing only the submitter’s illegible signature. None of the other associated identity information for the posted sample, or any of the duplicate comments, is included in the comment data. (See fig. 6.) Similarly, for all duplicate comments received, SEC posts a single example for each set of duplicate comments and indicates the total number of comments received. As a result, the identity information and any unique comment content beyond the first example are not present on the public website. (See fig. 7.) On the basis of the results from our survey, program offices with responsibility for analyzing comments differ in the importance they place on identity information during the analysis phase. Because subject-matter experts are responsible for reviewing public comments and considering whether changes to the proposed rule should be made, program offices generally analyze comments. Officials from all but one of the 52 program offices we surveyed responded that they were responsible, in whole or in part, for analyzing public comments. In our survey of program offices with regulatory responsibilities in the 10 selected agencies, at least one program office in each agency reported that the identity or organizational affiliation of a commenter is at least slightly important to comment analysis. Additionally, five of the 10 selected agencies (CMS, EPA, FCC, FDA, and FWS) had at least one program office that reported that the identity or organizational affiliation of a commenter is not at all important to comment analysis. None of the 52 program offices we surveyed responded that the identity of an individual commenter is extremely important to their analysis, while only one program office responded that the commenter’s organizational affiliation is extremely important to its analysis. (See fig. 8.) According to officials we interviewed from eight of the 10 selected agencies, the substance of the comment is considered during analysis rather than the submitted identity information. Officials from six of these agencies emphasized that because the agency accepts anonymous comments, identity is not relevant to their analysis of comments. However, officials from four of the eight selected agencies stated that, in certain instances, identity information may be noted. In the case of FDA, officials explained that commenters are required to indicate a category to which they belong, such as “individual consumer” or “academia.” According to FDA officials, however, these categories were used to assist in writing the comment response, rather than informing the analysis. Officials from the Department of the Interior’s Office of the Solicitor (responsible for part of the comment process at BLM and FWS) stated that the agency may make particular note of comments submitted by a law firm, as these comments can help the agency understand the position of the law firm and to prepare a defense in the event that a lawsuit is filed. Similarly, officials from EPA stated that they are familiar with many commenters and their positions on certain issues, due to prior legal interactions. In another example of how an agency may consider the identity of a commenter, officials from FWS stated that when scientific data are provided in support of a comment, subject-matter experts will verify the data and their source. All selected agencies draft a response to comments with their final rule, but the extent to which the agencies identify commenters in their response varies. In our survey of program offices with regulatory responsibility, officials from 51 of 52 offices stated that they are responsible in whole or in part for responding to comments. Of those responsible, at least one program office from eight of the 10 agencies (28 of 52 offices) reported that they identified comments by commenter name, organization, or comment ID number in the response to comments for at least some rulemakings since 2013. In the case of WHD, officials we interviewed explained that when they discuss a specific comment in the preamble to the final rule, they provide the name of the organization that submitted the comment so that anyone interested in locating the response to the comment may do so easily. We found that EBSA and FCC also identified commenters by individual or organizational name in their response to comments, while EPA referred to comments by their comment ID number. For example, in a rule finalized in 2018, EPA referred to comment ID numbers in the response to comments: “Two comments: EPA-R06-RCRA-2017-0556-0003 and EPA- R06-RCRA-2017-0556-0005 were submitted in favor of the issuance of the petition.” EPA officials noted that there is variation within the agency in terms of how commenters are identified when the agency is responding to comments, and there may be some situations where the commenter is identified by name. Officials from all program offices within CFPB and BLM responded to the survey that they never identified comments by commenter name, organization, or comment ID in their responses to public comments. In its response to comments in a 2014 final rule, for example, CFPB stated that “industry commenters also emphasized the need to coordinate with the States,” without specifying the organization or specific comments. Similarly, in its response to comments document for a 2016 rule, for example, BLM responded directly to the themes and issues raised by comments while stating that the issue was raised by “one commenter” or “some commenters.” The 10 selected agencies have implemented varied ways of posting identity information during the comment intake process, particularly regarding posting duplicate comments, as allowed by the APA. Our analysis of Regulations.gov and agency-specific comment websites shows that these practices are not always documented or clearly communicated to public users of the websites. Public users are members of the public interested in participating in the rulemaking process via Regulations.gov or agency-specific websites. They may or may not submit a comment. In part to facilitate effective public participation in the rulemaking process, the E-Government Act requires that all public comments and other materials associated with the rulemaking docket should be made “publicly available online to the extent practicable.” There may be situations where it is not practicable to post all submitted items, for example when resource constraints prevent the scanning and uploading of thousands of duplicate paper comments. Because the content of such comments is still reflected in the administrative record, such practices are not prohibited by the APA or the E-Government Act. However, key practices for transparently reporting open government data state that federal government websites—like those used to facilitate the public comment process—should fully describe the data that are made available to the public, including by disclosing data sources and limitations. This helps public users make informed decisions about how to use the data provided. In the case of identity information submitted with public comments, for example, public users may want to analyze identity information to better understand the geographic location from which comments are being submitted, and would need information about the availability of address information to do so. The Administrative Conference of the United States has made several recommendations related to managing electronic rulemaking dockets. These include recommendations that agencies disclose to the public their policies regarding the treatment of materials submitted to rulemaking dockets, such as those associated with protecting sensitive information submitted by the public. As described earlier in this report, the varied practices that selected agencies use with regard to identity information during the public comment process results in the inconsistent presentation of this information on the public websites, particularly when it is associated with duplicate comments. Although the APA and E-Government Act do not include any requirements associated with the collection or disclosure of identity information, we found that the selected agencies we reviewed do not effectively communicate the limitations and inconsistencies in how they post identity information associated with public comments. As a result, public users of the comment websites lack information related to data availability and limitations that could affect their ability to use the comment data and effectively participate in the rulemaking process themselves. Public users of Regulations.gov seeking to submit a comment are provided with a blanket disclosure statement related to how their identity information may be disclosed, and are generally directed to individual agency websites for additional detail about submitting comments. The Regulations.gov disclosure statements and additional agency-specific details are provided on the comment form, and a user seeking to review comments (rather than submit a comment) may not encounter them on Regulations.gov. Regulations.gov provides the following disclosure statement at the bottom of each comment submission form: Any information (e.g., personal or contact) you provide on this comment form or in an attachment may be publicly disclosed and searchable on the Internet and in a paper docket and will be provided to the Department or Agency issuing the notice. To view any additional information for submitting comments, such as anonymous or sensitive submissions, refer to the Privacy Notice and User Notice, the Federal Register notice on which you are commenting, and the Web site of the Department or Agency. Similar information is provided to all public users in the Privacy Notice, User Notice, and Privacy Impact Assessment for Regulations.gov and the eRulemaking Program. While all of these note that any information, personal or otherwise, submitted with comments may be publicly disclosed, public users are not provided any further detail on Regulations.gov regarding what information, including identity information, they should expect find in the comment data. We found that when Regulations.gov provides public users with additional agency-specific information about the comment intake process, including accepting and posting comments, it is typically provided in the context of the comment form and does not provide public users enough detail to determine what comment data will be available for use when searching comments that are already submitted. Specifically, each comment form contains a pop-up box under the heading “Alternate Ways to Comment,” which reflects the language associated with comment submission methods included in the NPRM on which individuals are seeking to comment. Additionally, three participating agencies in our review (EPA, FWS, and WHD) provide additional detail about posting practices on the comment form under the heading “Agency Posting Guidelines.” Both FWS and WHD indicate that the entire comment, including any identifying information, may be made available to the public. Although WHD follows DOL policy associated with posting duplicate comments, which allows some discretion in posting practices, according to a WHD official, without exception, all comments are posted to Regulations.gov. In our review of WHD comment data, we did not identify instances where this practice was not followed. The “Agency Posting Guidelines” provided by EPA inform public users that all versions of duplicate or near-duplicate comments as part of mass mail campaigns may not be posted; rather a representative sample will be provided, with a tally of the total number of duplicate comments received. (See fig. 9.) However, this information does not provide enough detail to help public users determine whether all of the individual comments and associated identity information are posted within this docket, because it indicates that samples are provided for duplicate comments, rather than all of the copies submitted. We found that one EPA docket received more than 350 separate sets of duplicate comments comprising a total of more than 4.3 million comments (as reported by Regulations.gov) but there is variation in how these comments were posted. Specifically, EPA inconsistently presented duplicate comments: 198 of the 350 duplicate comment sets in this docket were submitted via email. Of the duplicate comment sets submitted via email, 45 sets have all comments posted in Regulations.gov, while 153 sets have a sample of the comments posted. According to EPA officials, this inconsistency results from the format in which the comments were submitted. For example, when duplicate comments are compiled into a single document and submitted to EPA through one email, all of the comments will be posted, whereas duplicate comments that are emailed separately will be accounted for in the tally accompanying a sample comment. While the APA and the E-Government Act do not require comments to be posted in any particular way, EPA has established detailed internal guidance for the comment intake process for its Docket Center staff. This document is in draft form, but clearly lays out the processes EPA staff are expected to follow when duplicate comments are submitted in different ways, and what naming conventions will be used in different instances. However, EPA does not provide similar information to public users about the process it uses to determine whether all duplicate comments will be posted, making it challenging for public users to determine whether all comments are available on Regulations.gov. The eRulemaking PMO provides participating agencies with flexibility in how they choose to use FDMS and Regulations.gov, with each department or agency responsible for managing its own data within the website. As a result, Regulations.gov directs public users to participating agencies’ websites for additional information about agency-specific review and posting policies. We found that all of the selected participating agencies provide additional information of some kind about the public comment process on their own websites. However, the provided information usually directs users back to Regulations.gov or to the Federal Register. Further, even when selected participating agencies include details on their website about the agency’s posting practices or treatment of identity information associated with public comments, it does not fully describe data limitations that public users need to make informed decisions about how to use the data provided. Specifically, seven of the eight participating agencies (BLM, CMS, CFPB, EPA, FWS, FDA, and WHD) direct public users back to Regulations.gov and the Federal Register, either on webpages that are about the public comment process in general, or on pages containing information about specific NPRMs. As discussed previously, however, the disclosure statement on Regulations.gov directs public users to the agency website for additional information. Although three of these participating agencies (EPA, FWS, and FDA) do provide public users with information beyond directing them back to Regulations.gov or the Federal Register, only FDA provides users with details about posting practices that are not also made available on Regulations.gov. EPA: The additional information provided on EPA’s website largely replicates the “Agency Posting Guidelines” provided on the Regulations.gov comment form, as shown in figure 9. As discussed previously, however, the way in which EPA posts duplicate comments varies, and the provided information does not include details about the process the agency uses to determine whether all duplicate comments will be posted. FWS: One NPRM-specific web page that we identified communicated to public users that all comments will be posted on Regulations.gov, including any personal information provided through the process. This largely replicates the “Agency Posting Guidelines” provided on the Regulations.gov comment form, as well as language included in the NPRM itself. However, according to an FWS official, when the agency receives hard-copy duplicate comments through the mail, only one sample of the duplicate is posted publicly on Regulations.gov. FWS does not have any policies related to this practice and the information FWS provides to public users does not include details about how the agency determines which comment to post as the sample. FDA: On its general website, FDA includes a webpage titled, “Posting of Comments.” On this page, FDA provides users with a detailed explanation about a policy change the agency made in 2015 related to the posting of public comments submitted to rulemaking proceedings. Specifically, prior to October 2015, FDA did not publicly post comments submitted by individuals in their individual capacity. See figure 10. After October 15, 2015, FDA’s policy is to publicly post all comments to Regulations.gov, to include any identifying information submitted with the comment. In our review of FDA comments submitted to dockets opened since October 15, 2015, we did not identify instances where this policy was not followed. The one participating agency in our scope (EBSA) that does not direct public users back to Regulations.gov instead recreates the entire rulemaking docket on its own website. On the main EBSA webpage related to regulations, public users can find links to various websites related to rulemaking, including a “Public Comments” page, but not Regulations.gov. From the “Public Comments” page, public users can access pages that are specific to NPRMs and other activities for which EBSA is requesting public comments. On the NPRM-specific webpages, the rulemaking docket that can be found on Regulations.gov is duplicated, including individual links to each submitted comment. Certain document links, such as those for the proposed rule or final rule, direct a public user to the Federal Register document, but the comment links do not direct users to Regulations.gov. While EBSA follows DOL guidance associated with posting duplicate comments, which allows some discretion in posting practices, EBSA does not have a policy for how comments are posted to Regulations.gov or its own website, and in the examples we reviewed the content of the docket pages does not always match. According to EBSA officials, the agency began this practice prior to the development of Regulations.gov, and has continued it because internal staff and other stakeholders find the webpages useful. However, we have previously reported that reducing or eliminating duplicative government activities can help agencies provide more efficient and effective services. Further, on EBSA’s “Public Comments” webpage, public users are informed that comments with inappropriate content will be removed, but no other information associated with EBSA’s posting practices is provided on this general page. In one instance on an NPRM-specific webpage, public users are informed that identity information has been removed from certain comments due to the inclusion of personal health information, but most of the NPRM-specific webpages we reviewed did not include this disclosure. Additionally, duplicate comments are posted on the NPRM- specific webpages under the heading “Petitions,” and are posted with a number following the title of the comment. While public users are informed that the number represents the total number of comments submitted, not all links include a copy of each individual comment. This practice aligns with DOL guidance, but as a result, the way in which EBSA posts duplicate comments varies even within dockets, and the provided information does not include details about the process the agency uses to determine whether all duplicate comments will be posted. Additionally, because EBSA recreates rulemaking dockets on its own website without referencing Regulations.gov or explaining the process, public users lack assurance about how EBSA’s data sources relate to one another. Because participating agencies are not required to adhere to standardized posting practices, Regulations.gov directs public users to participating agency websites for additional information about posting practices and potential data limitations. However, the additional information provided on the selected agencies’ websites is rarely different from what is provided on Regulations.gov. Further, it does not describe the limitations associated with the identity information contained in publicly posted comments, and in many cases simply directs users back to Regulations.gov. As allowed for under the APA, all of the participating agencies in our review vary in the way in which they post identity information associated with comments—particularly duplicate comments. However, the lack of accompanying disclosures may potentially lead users to assume, for example, that only one entity has weighed in on an issue when, actually, that comment represents 500 comments. The APA, E-Government Act and relevant Executive Orders establish the importance of public participation in the rulemaking process, to include access to electronic rulemaking dockets in formats that can be easily searched and downloaded. Further, key practices for transparently reporting open government data state that federal government websites— like those used to facilitate the public comment process—should fully describe the data that are made available to the public, including by disclosing data sources and limitations. Without better information about the posting process, the inconsistency in the way in which duplicate comments are presented to public users of Regulations.gov limits public users’ ability to explore and use the data and could lead users to draw inaccurate conclusions about the public comments that were submitted and how agencies considered them during the rulemaking process. Both SEC and FCC use comment systems other than Regulations.gov and follow standardized posting processes associated with public comments submitted to their respective comment systems, but SEC has not clearly communicated these practices to the public. Although it appears to users of the SEC website that the agency follows a consistent process for posting duplicate comments, this practice has not been documented or communicated to public users of its website. As discussed earlier, SEC posts a single example for each set of duplicate comments and indicates the total number of comments received. As a result, the identity information and any unique comment content beyond the first example are not accessible to the public online. According to SEC officials, this practice is not documented in formal policy, and is not explicitly communicated to public users of the SEC’s comment website. Although SEC does provide public users with some information on its “How to Submit Comments” page, this information is limited to informing public users that all comments will be posted publicly, without any edits to personal identifying information, and no other information related to SEC’s posting process is provided. Without clearly communicated policies for posting comments, public users of SEC.gov do not have information related to data sources and limitations needed to determine whether and how they can use the data associated with public comments. In contrast, FCC identifies its policies for posting comments and their associated identity information in a number of places on the FCC.gov website, and on the ECFS web page within the general website. Regarding comments submitted to rulemaking proceedings through ECFS, public users are informed that all information submitted with comments, including identity information, will be made public. According to FCC officials, all comments are posted directly to ECFS as they are submitted, without intervention by FCC staff. Further, according to officials, all duplicate comments remain in ECFS as individual comments, unless an organization submits a Standard filing with an attached file containing multiple comments. Our review of ECFS comment data did not identify discrepancies with this practice. While the public comment process allows interested parties to state their views about prospective rules, the lack of communication with the public about the way in which agencies treat identity information during the posting process, particularly for duplicate comments, may inhibit users’ meaningful participation in the rulemaking process. While the APA does not include requirements for commenters to provide identity information, or for agency officials to include commenter identity as part of their consideration of comments, key practices for transparently reporting open government data state that federal government websites—like those used to facilitate the public comment process—should fully describe the publicly available data, to include disclosing data sources and limitations. Without clearly communicating how comments and their associated identity information are presented in the data, public users could draw inaccurate conclusions about public comments during the rulemaking process, limiting their ability to participate in the rulemaking process. Five selected agencies do not have a policy for posting comments, and the selected agencies generally do not clearly communicate to public users about the way in which they publicly post comments and their associated identity information. In addition, one agency fully duplicates rulemaking dockets on its own website, without informing users that the information may be found in a searchable database on Regulations.gov. Regulations.gov does not provide detailed information about posting policies, and seven of the eight participating agencies in the scope of our review direct public users back to Regulations.gov or the Federal Register on their own websites. Further, the available information is provided on the comment form, so public users seeking to review comment data that had been previously submitted may not encounter it. Because all of the participating agencies in our review vary in the way in which they post identity information associated with comments—particularly duplicate comments—the lack of accompanying disclosures may potentially lead users to reach inaccurate conclusions about who submitted a particular comment, or how many individuals weighed in on an issue. As a result, public users of Regulations.gov do not have information related to data sources and limitations that could affect their ability to effectively use the comment data and, consequently, participate in the rulemaking process. Similarly, users of SEC.gov do not have information related to data sources and limitations needed to determine whether and how they can use the data associated with public comments, because the agency lacks a policy for posting duplicate comments and associated identity information to the public. In short, more clearly communicated information about posting policies, particularly with regard to identity information and duplicate comments, could help public users make informed decisions about how to use the comment data these agencies provide, and how comments may have informed the rulemaking process. We are making the following eight recommendations to the Directors of BLM, CFPB, and FWS; the Administrators of CMS, EPA, and WHD; the Assistant Secretary of Labor for EBSA; and the Chairman of the SEC, respectively: The Director of BLM should create and implement a policy for standard posting requirements regarding comments and their identity information, particularly for duplicate comments, and should clearly communicate this policy to the public on the BLM website. (Recommendation 1) The Administrator of CMS should create and implement a policy for standard posting requirements regarding comments and their identity information, particularly for duplicate comments, and should clearly communicate this policy to the public on the CMS website. (Recommendation 2) The Director of CFPB should finalize its draft policy for posting comments and their identity information, particularly for duplicate comments, and clearly communicate it to the public on the CFPB website. (Recommendation 3) The Assistant Secretary of Labor for EBSA should 1. create and implement a policy for standard posting requirements regarding comments and their identity information, particularly for duplicate comments; 2. clearly communicate this policy to the public on the EBSA website; 3. evaluate the duplicative practice of replicating rulemaking dockets on the EBSA website, to either discontinue the practice or include a reference to Regulations.gov and explanation of how the pages relate to one another. (Recommendation 4) The Administrator of EPA should finalize its draft policy for posting comments and their identity information, particularly for duplicate comments, and clearly communicate it to the public on the EPA website. (Recommendation 5) The Director of FWS should create and implement a policy for standard posting requirements regarding comments and their identity information, particularly for duplicate comments, and should clearly communicate this policy to the public on the FWS website. (Recommendation 6) The Chairman of the SEC should develop a policy for posting duplicate comments and associated identity information and clearly communicate it to the public on the SEC website. (Recommendation 7) The Administrator of WHD should clearly communicate its policy for posting comments and their identity information, particularly for duplicate comments, to the public on the WHD website. (Recommendation 8) We provided drafts of this product for comment to CFPB, EPA, FCC, SEC, the Department of Health and Human Services, the Department of the Interior, and DOL. We received written comments from three of the selected agencies and the three Departments which are reproduced in appendixes V through X. All of the selected agencies generally agreed with the recommendations directed to them and indicated that they intended to take action to more clearly communicate their posting policies to the public. BLM, EBSA, FWS, and SEC also stated that they intend to develop written policies associated with posting comments. In its written comments, the Department of Health and Human Services stated that CMS already has policies for standard posting requirements. However, CMS could not provide us with this policy during the course of our review, and in the accompanying technical comments, officials stated that guidance associated with posting comments has not been formalized in a written document. Given that we found significant variation in the way that CMS posts comments, even within a single docket, we continue to believe that it is important for CMS to develop and implement a standard policy for posting comments and their identity information, in addition to communicating this policy to the public on the CMS website. CFPB and EPA also stated that they intend to finalize their draft policies for posting comments and their associated identity information. In addition, EPA included technical comments in its letter, which we considered and incorporated in this report as appropriate. FCC had no comments on the draft report, but provided technical comments, which we incorporated as appropriate. The remaining selected agencies and departments also provided technical comments, which we considered and incorporated in this report as appropriate. As arranged with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees; the Director of CFPB; the Administrator of EPA; the Chairmen of FCC and SEC; and the Secretaries of Health and Human Services, the Interior, and Labor. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6722 or bagdoyans@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix XI. To determine how selected agencies treat identity information associated with public comments, in October 2018 we surveyed and received responses from 52 program offices within the selected agencies about their practices associated with comment intake (including identifying duplicate comments and posting comments to the public website), comment analysis (including reviewing comments and considering their content), and response to comments. To select the program offices to receive survey questionnaires about the public comment process, we first reviewed agency websites to identify all of the program offices in each of the selected agencies. We then identified program offices with regulatory responsibilities described by the websites and that had issued at least one Notice of Proposed Rulemaking (NPRM) from 2013 through 2017, and provided these lists to the selected agencies for confirmation. Table 7 lists the program offices we surveyed. We developed a draft survey questionnaire in conjunction with another GAO engagement team conducting work on the public comment process, and pretested it with program office officials from four of the selected agencies in August and September 2018. We interviewed these officials to improve the questionnaire and ensure that (1) the questions were clear and unbiased, (2) the information could be feasibly obtained by program office officials, (3) the response options were appropriate and reasonable, and (4) the survey did not create an undue burden on program office officials. The process of developing the survey was iterative, where we used the results of one pretest to modify the questionnaire for the next pretest. We distributed the questionnaires to the program offices as fillable Portable Document Format (PDF) forms, in October 2018 requesting that officials collaborate with others in their office to ensure the responses were reflective of the program office as a whole, rather than one individual’s experience. Two agencies, CMS and SEC, have agency-level administrative offices with centralized responsibilities for certain aspects of the public comment process. For these agencies, the selected program offices were instructed to leave certain questions blank, and we provided separate questionnaires for the administrative offices. All 52 program offices completed the survey, but the results cannot be generalized to program offices outside of the selected agencies. In developing, administering, and analyzing this survey, we took steps to minimize the potential errors that may result from the practical difficulties of conducting any survey. Because we surveyed and received responses from all program offices with regulatory responsibilities in the selected agencies, our results are not subject to sampling or nonresponse error. We pretested and reviewed our questionnaire to minimize measurement error that can arise from differences in how questions are interpreted and the sources of information available to respondents. We also answered questions from program offices during the survey, reviewed completed questionnaires, and conducted follow-up as necessary. On the basis of this follow-up and with agreement from the responding officials, we edited responses as needed. For CMS and SEC, we edited the blank questions in the program office questionnaires with responses from their administrative offices. Comments are submitted to Regulations.gov via an electronic comment form. See figure 11 for an example of a comment form from Regulations.gov. The Federal Communications Commission’s (FCC) Electronic Comment Filing System (ECFS) allows commenters to submit comments to rulemaking proceedings via a Standard filing and Express filing. A Standard filing allows commenters to attach a file to their comment. See figure 12 for an example of a Standard filing. An Express filing does not allow for files to be attached. See figure 13 for an example of an Express filing. One way in which comments are submitted to the Securities and Exchange Commission (SEC) is through an electronic comment form. See figure 14 for an example of a comment form from SEC.gov. In addition to the contact named above, David Bruno (Assistant Director), Elizabeth Kowalewski (Analyst in Charge), Enyinnaya David Aja, Gretel Clarke, Lauren Kirkpatrick, James Murphy, Alexandria Palmer, Carl Ramirez, Shana Wallace, and April Yeaney made key contributions to this report. Other contributors include Tim Bober, Dahlia Darwiche, Colin Fallon, Justin Fisher, James Healy, Katie LeFevre, Barbara Lewis, and Maria McMullen.", "summary": "Federal agencies publish on average 3,700 proposed rules yearly and are generally required to provide interested persons (commenters) an opportunity to comment on these rules. In recent years, some high-profile rulemakings have received extremely large numbers of comments, raising questions about how agencies manage the identity information associated with comments. While the APA does not require the disclosure of identifying information from a commenter, agencies may choose to collect this information. This report examines (1) the identity information collected by Regulations.gov and agency-specific comment websites; (2) the guidance agencies have related to the identity of commenters; (3) how selected agencies treat identity information; and (4) the extent to which selected agencies clearly communicate their practices associated with identity information. GAO selected a nongeneralizable sample of 10 federal agencies on the basis of large comment volume. GAO surveyed 52 program offices within these agencies about their comment process; and reviewed comment websites, agency guidance, and posted comment data. GAO also interviewed relevant agency officials. The Administrative Procedure Act (APA) governs the process by which many federal agencies develop and issue regulations, which includes the public comment process (see figure below). Regulations.gov and agency-specific comment websites collect some identity information—such as name, email, or address—from commenters who choose to provide it during the public comment process. The APA does not require commenters to disclose identity information when submitting comments. In addition, agencies have no obligation under the APA to verify the identity of such parties during the rulemaking process. GAO found that seven of 10 selected agencies have some internal guidance associated with the identity of commenters, but the substance varies, reflecting the differences among the agencies. The guidance most frequently relates to the comment intake or response to comment phases of the public comment process. With the discretion afforded by the APA, selected agencies' treatment of commenters' identity information varies, particularly when posting duplicate comments (identical or near-identical comment text but varied identity information). Generally, officials told GAO that their agencies (1) post all comments within the comment system; or (2) maintain some comments outside of the system, such as in email file archives. For instance, one agency posts a single example of duplicate comments and indicates the total number of comments received. However, within these broad categories, posting practices vary considerably—even within the same agency—and identity information is inconsistently presented on public websites. Selected agencies do not clearly communicate their practices for how comments and identity information are posted. GAO's key practices for transparently reporting government data state that federal government websites should disclose data sources and limitations to help public users make informed decisions about how to use the data. As a result, public users of the comment websites could reach inaccurate conclusions about who submitted a particular comment, or how many individuals commented on an issue. GAO is making a total of eight recommendations to the selected agencies to more clearly communicate to the public their policies for posting comments and associated identity information to Regulations.gov and agency-specific comment websites. The selected agencies generally agreed with the recommendations.", "document_type": "gao"}
{"report": "Local educational agencies (referred to in this report as school districts or districts) receive funding for education primarily from state and local sources. School districts can typically use this funding for a wide range of purposes, including school maintenance and operations. Maintenance may include routine replacement of lighting, filters, or building system parts, as well as emergency repairs to building systems. According to Education, maintenance and operations may also cover care and upkeep of grounds and equipment, vehicles (other than student transportation), and security. When school districts need to construct, renovate, replace, or make major repairs to building systems or features, such as roofing or plumbing, they typically use capital funding, which is separate from funding used for maintenance and operations. School districts use various mechanisms to fund capital projects. The specific funding mechanisms available to a given school district may differ based on state laws or regulations, and may require approval from state or local voters. A common funding mechanism for capital projects is to issue bonds. Bonds are debt securities issued by states, school districts, and other governmental entities and are repaid with interest, often through local property taxes or other types of local revenue. In some states, school districts might also use funding mechanisms called capital reserves and sinking funds to raise funds for school facilities projects. Capital reserves allow districts to hold end-of-year surpluses of general education funding in a capital reserve fund, which typically grows over time and can be used for large-scale projects. Sinking funds are usually generated from local property taxes and allow districts to set aside a percentage of property taxes each year to be used for capital projects. Districts do not pay interest because the funds are not borrowed; however, the funds generated may not be sufficient for large-scale projects. In specific circumstances, some federal funding is available for school facilities. For example, Education administers the Impact Aid program, which compensates local school districts that, among other things, have lost property tax revenue due to federal activities. This may include the presence of tax-exempt federal property, such as a military installation, children in public schools whose parents work and live on federal property, or children living on Indian lands. In fiscal year 2019, Education provided $17.4 million in Impact Aid grants to school districts, specifically for construction, renovation, or repair of school facilities. Additionally, the Federal Emergency Management Agency (FEMA) provides funding for school districts affected by some natural disasters, partly to repair and replace damaged buildings. For example, in 2019, following Hurricane Harvey, FEMA awarded grants to two school districts in Texas to set up a temporary middle school and replace books, equipment, and furniture, among other things. A facilities condition assessment is a systematic inspection of building systems and features using a standardized method for recording observations about condition. For example, one might walk through a building, record the condition of building systems and features, and identify deficiencies. Individuals conducting these assessments may also review documentation on the building systems, conduct interviews with administrators or other stakeholders, and develop cost estimates of physical deficiencies. Facilities condition assessments help districts identify deferred maintenance needs in schools, which can help them plan and budget for facilities. School districts can use data gathered from these assessments to develop a facility condition index (FCI). FCIs provide a point-in-time comparison of the cost of repairing deficiencies in a building with the cost of replacing the building, and can help school districts compare conditions across their facilities. FCIs may also help school districts budget for targeted replacements or improvements of building systems. Based on our nationally representative survey of school districts, we estimate that about half (54 percent) of districts need to update or replace at least two building systems in many of their schools. Further, we estimate about a quarter of districts (26 percent) need to update or replace at least six systems in many of their schools. In terms of specific building systems and features, we estimate that 41 percent of school districts need to update or replace HVAC systems in at least half their schools (about 36,000 schools nationwide). We also estimate about a quarter of districts need to update or replace other building systems, including interior lighting, roofing, safety and security systems, or plumbing in at least half their schools (see fig. 2). We saw similar results among the 55 schools we visited. Of those, 28 had HVAC issues, such as older systems that frequently malfunction or leak and damage flooring or ceiling tiles, according to our observations and discussions with district and school officials. For example, one school we visited in Rhode Island had parts or components of their operating HVAC systems that were nearly 100 years old, according to district officials (see fig. 3). In Michigan, we visited one school that district officials said used an original boiler from the 1920s to heat the building. According to district officials, older boilers are labor-intensive to maintain because city code requires an engineer to be on site when each boiler is operating; without constant monitoring when in operation, the boilers could build up too much pressure and explode. Officials in a New Mexico district said their mechanical systems experience issues because hard water (i.e., water with a high concentration of minerals) damages the systems and causes them to malfunction. Because of the hard water, the district spent $150,000 to replace an 8-year-old boiler that, according to district officials, should have lasted 20 years. District officials said they would like to purchase filtration and water softening systems to address the issue, but that the district cannot afford to do so. If not addressed, HVAC issues can result in health and safety problems. Officials in several school districts we visited said there are serious consequences to not maintaining or updating HVAC systems, including lost educational time due to school closings and the potential for mold and air quality issues (see fig. 4). For example, officials in a Michigan district said about 60 percent of their schools do not have air conditioning, and in 2019, some temporarily adjusted schedules due to extreme heat. Without air conditioning, schools relied on open windows and fans, which were not always effective at cooling buildings to safe temperatures for students and staff, according to district officials. Officials in a Maryland district said the district retrofitted some schools with air conditioning, but did not update pipes and insulation serving the HVAC systems, which has caused moisture and condensation problems in these buildings. Officials were concerned the moisture and condensation could lead to air quality and mold problems, but said that to remedy these issues could cost over $1 million for each building. School districts also reported needing to update or replace other key building systems and features. Based on our school district survey, we estimate that about 30,000 schools need to update or replace interior lighting and about 28,000 schools need to update or replace roofing. Of the 55 schools we visited, some had recently updated or replaced these systems, while others continued to face challenges. For example, 15 schools had installed light emitting diode (LED) systems or incorporated other energy efficient features, such as motion sensors to turn off lights in unused rooms or automatic dimmers that adjust based on the amount of daylight in a given space (see fig. 5). Six schools had not recently updated their interior lighting, but officials expressed a desire to do so in the near future, such as by switching to LED systems. Some district officials said LED systems can reduce energy consumption and utility costs. Of the 55 schools we visited, 18 had problems with their roofing, according to district and school officials. Roofing problems ranged from small leaks to larger issues requiring a costly replacement (see fig. 6). For example, officials in a Rhode Island district said that replacing the roofing at one school would likely cost about $3 million. These officials said, because the district did not have the funds to replace it, they instead planned to spend $20,000 on temporary fixes, with the hope that these fixes would last until funding was available for a full replacement. Based on our survey of school districts, we estimate that 65 percent of districts had conducted a facilities condition assessment of their schools at least once in the last 10 years and about 35 percent had not or did not know if their district had (see fig. 7). Of the districts that had conducted these assessments, almost all did so to evaluate safety and hazards (99.6 percent) and support capital planning, including prioritizing large- scale projects (96.6 percent). Additionally, of these districts, an estimated: 86.2 percent assessed facilities at every school in their district; 68.6 percent evaluated their facilities at least every 5 years; and 39.5 percent hired contractors or professional firms to conduct the assessment. We estimate that at least 53 percent of all students in the nation attended a school that had a facilities condition assessment in the last 5 years. We estimate that 16 percent of districts had not conducted a facilities condition assessment in the last 10 years. In our survey, several districts provided reasons why they had not done so, including a lack of available funding or because they assessed school conditions through other mechanisms, such as informal walkthroughs. In addition to district-level facilities condition assessments, 11 states conducted a state-level facilities condition assessment in the last 10 years, according to our state survey (see fig. 8). Common reasons provided by these states for evaluating school facilities included to assess safety and hazards (9 states) and provide facilities information to the public (9 states). However, most states (38 of 49) either had not conducted or did not know if their state had conducted a state-level facilities condition assessment. Of these 38 states: 15 states reported they required school districts to conduct 21 states reported that they neither conduct statewide assessments nor require school districts to do so; and, Two states did not know if their state had conducted such an assessment. States that had not conducted a statewide facilities condition assessment or required districts to do so frequently said they do not assess school conditions because school districts are primarily responsible for addressing deficiencies with school facilities. Most of the districts we visited said they had conducted a facilities condition assessment. Specifically, of the 16 school districts we visited in six states, officials in 12 districts said they had recently conducted a facilities condition assessment for a variety of reasons, such as to develop facilities master plans or raise support for a bond. For example, officials in one urban California district said they conducted an extensive facilities condition assessment for planning purposes and developed a master plan of issues identified in schools 20 years or older. During the assessment, the district assigned barcodes to certain systems, such as HVAC and water fountains, to track conditions across schools (see fig. 9). District officials said they update facilities data as they complete projects. Officials in a rural Michigan district said they conducted an assessment before asking voters to approve a sinking fund. District staff identified the value, age, cost for repairs, and expected lifecycle of all major systems, which helped them estimate funding needs for the next 10 years. Officials in one Florida district said they do not conduct facilities condition assessments because the district is small and the facilities manager knows the condition of their schools and when facilities’ issues arise. Of the six states we visited, officials from Rhode Island and New Mexico said their states had conducted statewide facilities condition assessments and Florida officials reported requiring school districts to conduct these assessments. Officials in Rhode Island and New Mexico said data from these assessments help determine state funding for districts. For example, according to officials, Rhode Island hired a consulting firm to assess school facilities in order to develop an independent estimate of the statewide funding need; in 2017, that estimate was about $3 billion. Officials in three of the states we visited—Michigan, California, and Maryland—said their states had neither conducted a facilities condition assessment nor reported requiring school districts to do so. Officials in Michigan said their state provides no funding for school facilities nor requires districts to conduct facilities condition assessments because districts are responsible for planning and prioritizing school facilities’ needs. Michigan officials said districts often assess facility conditions before seeking bonds or other local funding to show local voters the level of need. Officials in California similarly said that school districts are primarily responsible for evaluating school conditions and noted that it would be cost-prohibitive for the state to conduct a statewide assessment, given the number of schools in the state. Maryland officials said the state has not had funding to conduct a statewide assessment since 2003, but they are currently planning a future statewide assessment. After this initial assessment, the state plans to assess each school facility every 3 to 4 years, according to these officials. In addition to key building systems such as HVAC, lighting, and roofing, school districts considered the need to ensure schools are free from health hazards, as well as update schools with modern educational spaces and features. Specifically, based on our survey, we estimate that school districts’ high priorities when updating or renovating school facilities are as follows: security (estimated 92 percent), student access to technology (87 percent), monitoring hazards to student and staff health (78 percent), and improving telecommunication features such as wireless internet (74 percent). In comparison, the 100 largest school districts, which serve approximately 10.4 million students, identified security (estimated 99 percent), monitoring health hazards (94 percent), and completing projects to increase physical accessibility for students with disabilities (86 percent) as their high priorities. Overall, in response to our survey, districts ranked the level of priority of each building system or feature on a categorical scale of five levels, which we assigned numerical rankings of 1 (not a priority) to 5 (top priority). Average priority ratings ranged from approximately 4.5 for safety and security to approximately 2.9 for access to natural light (see fig. 10). Similarly, officials in nearly all of the 16 school districts we visited told us that some combination of addressing urgent health hazards, improving security, and upgrading technology were among their top priorities. In addition, district staff told us they were undertaking projects to modernize spaces and improve the learning environment, when possible. Districts implemented these priorities differently based on their needs and resources. Many school district officials said they address facility issues that affect staff and student health with more urgency than many other issues. At schools we visited around the country, officials reported initiatives to address health concerns that ranged from total renovations to temporary mitigation programs (see fig. 11). For example, officials in a district in California told us that in two schools we visited they removed all materials containing lead, as well as replaced all roofs that contained asbestos, in accordance with health and safety regulations. These officials also said staff tests the water quality in all schools per recommended guidelines. In a different district, officials said they had concerns about water quality, but that they did not have the funding to remediate the issue in all schools. Therefore, the district provides bottled water to students in nearly all of its schools, and installs water filtration systems when it constructs or renovates schools. In several schools in five states we visited, officials said there is asbestos in floor or ceiling tiles or other materials that would require abatement during any renovation. Because abatement increases costs, schools may prioritize other projects or find workarounds. For example, at one high school in Florida, the district installed interactive white boards on top of old chalkboards rather than risk disturbing asbestos in the walls by removing the chalkboards. Officials in two districts also told us about addressing potential health hazards related to climate. For example, at a school in Florida, officials said they have to address mold and mildew issues due to frequent flooding and high humidity. During heavy storms, school personnel work to clear drains and place sandbags in an attempt to mitigate water intrusion and flooding. In 13 of the 16 districts we visited, officials told us that security has become a top priority, though the specific measures they took to update their security features varied considerably (see fig. 12). One high school we visited recently experienced a school shooting. District officials said they were implementing a variety of new security initiatives, first at the high school, and then at all other schools in the district. In the high school, officials applied a specialized film to exterior windows to make them bullet resistant. The school has a new security vestibule where visitors wait before entering the school, and staff placed comment boxes throughout the school encouraging students to submit safety tips. In Michigan, we visited a middle school that installed additional barricades on classroom doors, and trained students on how to use them during lockdown drills. In California, we visited an elementary school that added exterior windows to the front office so staff could see visitors approaching, and installed a lockdown alarm button. Officials from some districts we visited said they prioritized security over failing building systems. For example, one district in Rhode Island where we observed problems with key building systems, including ceiling damage from a leaking roof, broken windows, and holes in the walls and foundation of a school building, installed new security features throughout their schools. These included equipping classroom doors with electronic lockdown mechanisms that staff can activate remotely. The district updated the main entrance with heavy, reinforced doors and bulletproof glass. In a district in Florida, we visited an elementary school that updated security systems, including installing new cameras. This was despite the school having major challenges with its HVAC system that require maintenance staff to go up to the roof every day to adjust the air conditioning. In addition, we observed multiple buckets throughout the school to collect water leaking through the roof, and the principal described how it frequently “rained” in her office. District officials said they are seeking state funding to renovate the entire school, but decided to first address security updates because all classrooms have exterior doors, making it difficult to control access to the school. In this same district, officials told us they had recently renovated the middle-high school and ensured that all classrooms had “hard corners”—spaces where students could congregate and not be visible to an active shooter in the hallway. Officials in many school districts we visited said that ensuring adequate access to technology was necessary for students to be successful academically (see fig. 13). All schools we visited had WiFi access, though officials in one rural district in New Mexico described access as spotty. The majority of schools we visited provided a laptop or tablet to all or almost all students or had a goal to do so. Officials in a district in California said their most important project of the past decade was to update their fiber optic capability to have a robust WiFi network. All students in this district receive a laptop or tablet beginning in second grade, and officials said these updates allowed students to easily use devices in school. In some school districts that did not provide individual devices, schools had portable technology carts to store and charge devices, so students could access them as needed. Officials in districts we visited also said they use technology to enhance educational offerings. For example, a high school in Maryland equipped a classroom with cameras and a microphone so students could attend community college classes remotely. When renovating schools, some officials told us they incorporate and anticipate technology needs. For example, a newly renovated school in Florida installed electrical outlets on table surfaces in the media center and microphones in all classrooms so students could hear teachers better. At a newly renovated school in Maryland, officials installed a projector and sound system in the cafeteria for students to watch movies and listen to music during lunch, which they said created calmer lunch breaks. Officials in districts we visited said they chose among other competing facility priorities based on available funding as well as conditions at individual schools, such as the age and condition of buildings, timeframe constraints, public opinion, space constraints, and enrollment projections. In school districts we visited that reported having local taxes or bond funds available for facility projects, officials described both the need to address the condition of basic building systems and the need to renovate schools with modern educational spaces and features. For example, officials in a Rhode Island district said they are using most of the approximately $300 million in their 5-year capital plan to ensure schools are safe, warm, and dry. These district officials estimated their school facilities need over $1 billion in updates and replacements to key building systems, based on a recent assessment. However, they said they are using 25 percent of available capital funds to modernize educational spaces, such as collaborative workspaces, student common areas, and outdoor classrooms (see fig. 14 for examples of school modernizations in districts we visited). Officials said that participants in public forums preferred educational enhancements over facility repairs. In this same district, officials said they prioritized system repairs they can complete over the summer because the district does not have designated swing spaces to accommodate students during the school year. In a district in Florida, officials similarly described using the funding from a $1 billion bond for school facilities to address health and safety concerns, HVAC issues, and roofing. They balanced these building system repairs with projects to modernize buildings, including increasing natural light by replacing the windows, upgrading technology to support engineering and robotics programs, and creating open and collaborative spaces. See textbox for examples of how school officials told us school renovations improved student experiences. Additionally, several districts we visited considered enrollment and building capacity to help prioritize projects, but they faced different challenges. Specifically, some districts experienced space constraints and needed to ensure sufficient space for all students, while others had the opposite challenge of maintaining schools that were under-enrolled (see text box). In a district in California, officials said they built nine schools in the past decade because of the increasing student population. At a high school in Maryland, the principal said his priority was ensuring sufficient space because the school was at capacity and he was struggling to find additional classrooms and furniture. Due to population fluctuations at a nearby military installation, he said he often turns offices and workspaces into classrooms and vice versa. Conversely, in a district we visited in Michigan, officials said they struggled with the inefficiencies of maintaining school facilities with low enrollment because closing schools can be difficult, given how it can affect currently enrolled students and neighborhoods. Based on our survey of school districts, funding for school facilities primarily came from local sources for about half of school districts. Specifically, an estimated 55 percent of districts used local funding as their primary source for school facilities, compared to state (36 percent) and federal (1 percent) funding. Based on our survey analysis, we found significant differences in the primary funding sources for school facilities for high-poverty and low-poverty districts. Specifically, high-poverty districts more commonly relied on state funding to address facility needs than low-poverty districts, whereas low-poverty districts more commonly relied on local funding (see fig. 15). School districts reported using several funding mechanisms to access local funding for school facilities projects. The most common was property taxes, which an estimated 77 percent of all school districts used for school facilities. Other local funding came from grants, bonds, other taxes, and public-private partnerships (see fig. 16). Similar to our findings on the sources of school facilities funding, based on our survey analysis we found significant differences in the local funding mechanisms used by high-poverty and low-poverty districts. Specifically, high-poverty districts used property taxes less commonly than low-poverty districts. As noted above, high-poverty districts instead more commonly relied on state funding to address facility needs. We also analyzed federal data on school district expenditures for school facilities and found differences by poverty level (see text box). Spotlight: Federal Data on School District Expenditures for Capital Construction Each year, Education collects data on school district expenditures for capital construction. In school year 2015-16, this spending totaled $44.6 billion. We analyzed these data by school district characteristics: Poverty: Capital construction expenditures, on average, were about $300 less per student in high-poverty districts ($719 per student) compared to low-poverty districts ($1,016). About 1.5 million more students attended school in high-poverty districts than low-poverty districts in 2015-16. Low-poverty districts spent about $1 billion more on capital construction than high-poverty districts that year. Size: Capital construction expenditures per student were similar in the largest (by number of students enrolled) 100 districts compared to smaller districts. Both groups of districts, on average, spent $837 per student on capital construction in school year 2015-16. Locale: Capital construction expenditures per student were similar, on average, for urban ($838 per student) and rural districts ($834). Officials in school districts we visited described various challenges they faced in securing funding for school facilities and how they have managed with limited funding. For example, officials in a Michigan district said the district had $1.5 billion in outstanding bond repayments and state borrowing related to bond repayments. As a result, the district is unable to issue an additional secured bond to fund new school facilities projects. According to officials, Michigan does not provide state-level funding for school facilities, so the district funded some recent school facilities projects using general education surpluses resulting from staff vacancies. However, as the district hired teachers and other staff, funding for facilities will decline, further limiting the district’s ability to address issues with school facilities. That district has also deferred maintenance in order to handle emergency repairs, according to officials. Officials in a high- poverty district in one state we visited said their tax base generates minimal local revenue for school facilities. According to officials, the district is mostly dependent on state funding. In the past decade, the state established a partnership between various public entities, which provided $1 billion to the district to address school facility needs, according to district officials. Officials said the funding through this partnership was enough to renovate about 25 schools. However, officials estimated the district has about $5 billion in unmet needs, and its 2012 facilities condition assessment recommended it consider replacing 50 schools. We also visited districts that have consistently had access to funding for school facilities. For example, officials in one low-poverty California district said their district is generally able to obtain funds needed for school facilities projects, primarily through local taxes and passing general obligation bonds. Officials said there are currently few challenges with the condition of the district’s school facilities because of routine and preventive maintenance. Though school districts most commonly used local funding to address school facility needs, 36 states provided some level of capital funding to school districts for school construction or renovations, based on our state survey (see fig. 17). In addition, states reported using various criteria to determine funding for capital projects, including the condition of a district’s schools (23 states), type of project, such as HVAC or fire safety (22), and size of the student population (18). Fewer states (17) reported providing districts with funding for maintenance and operations—used for routine upkeep and replacement of building system parts—separate from general education funding. State support for school facilities similarly varied within and among the six states that we visited. Five of the six states we visited reported providing state-level capital funding for school facilities, although the amount and mechanisms differed. For example, according to state officials, New Mexico has a capital fund for schools supported through taxes on the oil and gas industry and bases its state funding on a school’s condition. These officials described how New Mexico assesses and ranks all schools based on the condition of their facilities, and funds projects starting with the highest priority school on the list, until each year’s funds are depleted. The state uses capital funds to match local dollars. The percentage of a project’s cost covered by the state depends on the district’s ability to raise local funds. In one district we visited, the state pays 100 percent. Florida targets funding for school facilities to rural districts and charter schools, both of which have limited access to local funding sources such as property taxes, according to officials. These officials said the state has a specific program to support capital projects in rural districts, and other funding—generated from taxes on landlines and utilities—has in recent years gone to charter schools. In California, districts receive state funding based on the order the state receives eligible applications, until funds are depleted, according to state officials. Michigan officials said the state does not fund school facilities projects at the state level, although the state has a program to review school districts’ local bond measures. The state does not require school districts to submit their bonds for state approval, but doing so allows the district to access the state’s credit rating, which usually lowers the district’s interest rate, among other benefits, according to these officials. In three states we visited, state officials we interviewed told us that financial support for capital projects may fluctuate each year depending on availability of state funding. For example, Rhode Island officials said that after the 2007-2009 recession, the state legislature stopped funding school facilities until 2015. This resulted in deferred maintenance in Rhode Island’s schools that the state and school districts now need to address in addition to any new capital projects, according to officials. Based on our state survey, five states require districts to use a portion of their general education funding for maintenance and operations. Three of these states reported requiring districts to use 3 percent or less of their general education funding for this purpose, one state reported requiring districts to use 6 percent, and one state did not know what percent was required. Officials in Rhode Island said they have a new policy to require districts to set aside a portion of the state funds they receive for maintenance and operations to protect the state’s increasing investment in school facilities, and that the state is phasing in the requirement over 5 years. Officials in New Mexico said that while they do not require this type of set aside, they evaluate how well districts maintain their facilities, and districts that inadequately maintain them may be ineligible for some types of state facilities funding. Many states also reported that they considered state-level priorities for school facilities when providing funding and guidance to school districts. Based on our survey, more than half of states provided financial support, as well as standards and guidance, for specific building systems and features of school facilities (see fig. 18). State funding and guidance related to state-level priorities can affect school district decisions on facilities. For example, Rhode Island approved a $250 million state bond for school facilities in 2018, and will provide higher reimbursements for district expenditures on projects reflecting state priorities, such as health and safety and decreasing overcrowding, according to state officials. In two rural districts within two states, district officials told us they cannot afford to undertake capital projects without state funding, and therefore have to balance state requirements with local needs and preferences for their facilities. For example, one district in New Mexico opted to renovate an existing gym using state matching funds, rather than fully replace it, because this allowed the district to maintain existing square footage. According to officials in that district, the state developed standards for how large a gym can be and still receive state funding for a full replacement, and the district prioritized renovating and maintaining the larger existing space instead. We provided a draft of this report to the Department of Education (Education) for review and comment. We also provided selected draft excerpts to relevant officials we interviewed in state agencies and school districts. Education as well as several state and district officials provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. In this report, we examined: (1) the common facility condition issues school districts identify in public schools and how they have done so and (2) school districts’ highest priorities for their school facility renovations and updates, and how districts and states fund them. To address these objectives, we used the following methodologies, which we describe in detail below: Surveyed all 50 states and the District of Columbia. Surveyed a nationally representative sample of K-12 public school districts. Visited 16 school districts in six states and interviewed state, district, and school staff. Conducted building walkthroughs at 55 schools (including five charter schools) and observed a standard set of building systems and features in each school. Analyzed federal data on district expenditures for capital construction projects. We took several steps to inform each of our methodologies and provide background for our objectives. To better understand the federal role in school facilities, we interviewed officials from the Department of Education’s (Education) National Center for Education Statistics (NCES), as well as Education’s Office of Impact Aid Programs and the Office for Civil Rights. During these interviews, we asked officials about their role in collecting information on the condition of school facilities, as well as providing funding and guidance on school facilities, among other topics. We also interviewed officials from the National Association of Federally Impacted Schools and the National Indian Impacted Schools Association to learn about facility concerns in public school districts that receive federal Impact Aid. We reviewed federal documentation including NCES’s 2014 report, Condition of America’s Public School Facilities: 2012-13 and the Congressional Research Service’s 2015 report on federal programs related to school facilities. In addition, we reviewed guidance from the Environmental Protection Agency on creating and maintaining healthy and environmentally friendly school facilities. To better understand assessments of building conditions, as well as to obtain information on school building systems and features, we reviewed the Standard Guide for Property Condition Assessments: Baseline Property Condition Assessment Process, an international standard for assessing the condition of a building. Additionally, we interviewed officials at the 21st Century School Fund, the American Society of Civil Engineers, the Association for Learning Environments, the Center for Cities and Schools at the University of California, Berkeley, the Center for Green Schools, the Council of Chief State School Officers, and the Education Commission of the States. We used this information to create two lists of building systems and features, which we asked about in our surveys and asked to observe in the schools we visited. Specifically: The first list focused on key systems and features that may be necessary to a school building’s day to day operations; the second list focused on additional or emerging priorities for systems and features that school districts may consider when modernizing school facilities. We validated these lists of systems and features through survey pretests with facilities personnel in six states. Because some modernization priorities are also key to a school building’s day-to-day operations, there are systems and features that appear on both lists (see app. II for a full list of our survey questions, including all systems and features about which we asked school districts). We modified and combined the above lists for our state survey to ask states about their priorities and support for school building systems and features. To address both research questions, we designed and administered a web-based survey to all 50 states and the District of Columbia. We sent the survey to the relevant state agency that oversees school facilities, or to the state superintendent of education to be forwarded to the state official best equipped to answer questions related to the condition of school facilities. We conducted the survey between September and December 2019. To obtain the maximum number of responses to our survey, we contacted nonrespondents via email and phone throughout the period the survey was open. In total, 49 states responded to the survey; Mississippi and Illinois did not respond. Data in this report are based on the 49 states that responded, unless otherwise noted. To ensure the quality and reliability of the survey, we pretested the questionnaire with three states that vary in their level of involvement in school facilities, among other factors. We conducted the pretests to check (1) the clarity and flow of the questions, (2) the appropriateness of the terminology used, (3) if the information could be easily obtained and whether there were concerns about the reliability of data that would be collected, and (4) if the survey was comprehensive and unbiased. We revised the questionnaire based on the pretests. We reviewed responses to assess if they were consistent and contained all of the relevant information. The survey included open-ended and closed-ended questions about: The state’s role in assessing the condition of school facilities and the level of information the state has about the condition of school facilities. The state’s role in providing funding to school districts for school facilities and the factors it considers in determining funding levels. The extent to which the state provides standards, guidance and other non-financial resources to school districts about their facilities. Whether the state collects information or provides additional assistance to school districts that receive federal Impact Aid funds. To address both research questions, we designed and administered a generalizable survey of a stratified random sample of local educational agencies, which we refer to as school districts throughout this report. We sent the survey to school district superintendents to be forwarded to the district official best equipped to answer questions related to the condition of school facilities. The survey included questions about: School districts’ policies and practices regarding whether they conduct facilities condition assessments. How often school districts conduct or update these assessments. How school districts use the information from assessments to make decisions regarding school repairs, renovations, and replacements. The extent to which the school districts were facing issues with the condition of building systems and features within their schools. The funding mechanisms that school districts use to address issues with the physical condition of public schools. We defined our target population to be all school districts in the 50 U.S. states and the District of Columbia that are not under the jurisdiction of the Department of Defense or Bureau of Indian Education. We used the Local Education Agency Universe database from Education’s Common Core of Data (CCD) for the 2016-2017 school year as our sampling frame. For the purpose of our survey, we limited the sampling frame to school districts that: were located in the 50 states or the District of Columbia; had one or more schools and one or more students; and were not closed according to the 2016-2017 School Year or preliminary 2017-18 School Year CCD data available just prior to survey deployment. The resulting sample frame included 17,248 school districts and we selected a stratified random sample of 664 school districts. We stratified the sampling frame into 19 mutually exclusive strata based on urban classification and poverty classification (see table 1). We selected the largest 100 school districts, based on student enrollment, with certainty. To determine the appropriate sample size for the survey, we first determined the minimum sample size needed to achieve precision levels of percentage estimates within plus or minus 10 percentage points, at the 95 percent confidence level, within each of three sub-groups: low, medium, and high-poverty districts. Within each of these poverty sub- groups, we proportionately allocated the sample across the race and urban classification groups. We then increased the sample size within each non-certainty stratum for an expected response rate of 55 percent in order to achieve the necessary number of completed surveys for our desired precision level. We defined the three locale classifications (i.e., city, suburban, and rural) based on the NCES urban-centric locale codes. The rural classification included school districts classified as either rural or town. To build a general measure of the poverty level for each school district we used the proportion of students eligible for free or reduced-price lunch (FRPL) as indicated in the CCD data and classified these into the following three groups: High-poverty: more than 75 percent of students in the school district were eligible for FRPL; Mid-poverty: Between 25.1 and 75.0 percent of students in the school district were eligible for FRPL; and Low-poverty: 25 percent or fewer students in the school district were eligible for FRPL. We assessed the reliability of the CCD data by reviewing existing documentation about the data and performing electronic testing on required data elements and determined they were sufficiently reliable for the purposes of our reporting objectives. We administered the survey from August to October 2019. We identified that 11 of the 664 sampled school districts were closed or had no physical school buildings, so these were removed from the universe and sample. Six of these out of scope sample districts were discovered soon after survey deployment, thus, we were able to replace these six sample districts with the next randomly selected district within the same strata. This resulted in a final in scope population of 17,237 districts and 659 in scope sample districts. We received 378 valid survey responses from this in scope sample resulting in an unweighted response rate of 57 percent and a weighted response rate of 53 percent. We analyzed the response status to our survey to identify potential sources of nonresponse bias in accordance with best practices in survey research and echoed in Office of Management and Budget, Standards and Guidelines for Statistical Surveys (September 2006). We examined the response propensity of the sampled school districts using both bivariate and multivariate logistic regression models, including several demographic characteristics available for respondents and nonrespondents: urban classification, race, poverty, district size (number of schools and number of students in a district), and the stratification variable that combines these characteristics. We detected a significant association between both strata and number of students within a district and the propensity to respond to our survey. We did not detect a significant association between urban classification, race, or poverty and the response propensity. We adjusted for the characteristics significantly associated with response propensity using weighting class adjustments. Specifically, we grouped the predicted response propensity derived from our logistic regression model that includes strata and the number of students using quintiles of the predicted response propensity distribution to form five weighting adjustment groups. We applied nonresponse adjustments to the sampling weights within these groups to form nonresponse adjusted analysis weights used in our survey analyses. Based on the nonresponse bias analysis and resulting nonresponse adjusted analysis weights, we determined that estimates using these weights are generalizable to the population of eligible school districts and are sufficiently reliable for the purposes of our reporting objectives. We took steps to minimize non-sampling errors, including pretesting draft instruments and using a web-based administration system. We pretested the draft instrument from June to July 2019 with officials in five school districts in different states and with varying characteristics such as size of the student population. In the pretests, we asked about the clarity of the questions and the flow and layout of the survey. Based on feedback from the pretests, we revised the survey instrument. To obtain the maximum number of responses to our survey, and to minimize non-sampling error caused by nonresponse, we sent reminder emails to nonrespondents and contacted some nonrespondents over the telephone. We express the precision of our particular sample’s results as a 95 percent confidence interval (for example, plus or minus 10 percentage points). This interval would contain the actual population value for 95 percent of the samples we could have drawn. As a result, we are 95 percent confident that each of the confidence intervals in this report will include the true values in the study population. We compared—as appropriate—weighted survey estimates generated for school districts by the school district strata described above. For each subgroup, we produced percentage estimates and standard errors for each level and used these results to confirm the significance of the differences between weighted survey estimates. To address both research questions, we visited six states—California, Florida, Maryland, Michigan, New Mexico, and Rhode Island—from June to September 2019. We selected these states because they varied in the amount and type of funding they provided to school districts for school facilities, the level of information they collected on the condition of school facilities, and for geographic variation. Within these states, we visited 16 school districts, which we selected based on variation in the size and population density of the district, poverty level, racial and ethnic composition, and the receipt of federal Impact Aid funding (see table 2). Within each district, we visited between two and five schools, depending on the size of the district and logistical considerations. We also visited five charter schools across four states, chosen based on their proximity to a selected school district. In total, we visited 55 schools that varied in grade level, enrollment, physical size, age, and condition. For resource efficiency, we generally interviewed state and district officials via phone in advance of the site visit, and toured schools with district and school officials. States: We interviewed state officials who were knowledgeable about their state’s role in funding, assessing, or providing other resources to school districts for school facilities. We discussed the agency’s roles and responsibilities related to statewide school facilities condition assessments or data collection initiatives, state-level priorities for school facilities, and funding mechanisms within the state for school facilities. School districts: We interviewed school district officials in each district we visited. Similar to our school district survey, we discussed their policies and practices on facilities condition assessments, how often they conduct or update these assessments, and how they make decisions regarding school repairs, renovations, and replacements. We also asked questions about how the districts prioritize upgrades and repairs to school facilities and the funding mechanisms they use to address issues with the physical condition of public schools. School Observations: To select schools in each district, we used CCD data to randomize the list of all schools in the district and selected the first two to four schools with consideration for different grade levels. We then asked district officials to verify that our random selections showed sufficient variety in the age and overall condition of the building. We substituted recommended schools when appropriate to ensure we had appropriate variety in seeing schools of different ages and conditions. When logistically feasible, we visited a nearby charter school as well. We toured schools with a combination of district and school officials. During these visits, we used a data collection instrument to ask officials about school building systems and features that school personnel identified as particularly in need of repair or replacement, as well as new or upgraded systems. We photographed these as appropriate. Information we gathered from these interviews and observations, while not generalizable, provides insight into the conditions present in the states and school districts we visited at the time of our interviews, and may be illustrative of efforts in other states and school districts. To examine expenditures for capital construction by school district characteristics, we analyzed federal data from Education’s Local Education Agency Finance Survey for school year 2015-16, the most recent available at the time of our analysis. Education collects these data annually as part of the CCD. State educational agencies provide these data on behalf of their school districts to NCES and the U.S. Census Bureau’s Economic Reimbursable Surveys Division. In school year 2015- 16, states reported finance data for 96.7 percent of school districts, according to Education’s survey documentation. We analyzed school district data on capital construction expenditures by poverty level, locale, district size, racial demographics, and receipt of federal funding through Impact Aid or Indian education grants. We normalized data across school districts that fell into these different categories by calculating capital construction expenditures per student and per school. We determined these data were sufficiently reliable for the purposes of our reporting objectives by reviewing relevant documentation, interviewing knowledgeable Education officials, and testing for missing data, outliers, and other potential errors. Through discussions with NCES officials, we determined it was necessary to exclude some school districts from our analysis to develop accurate per pupil and per school calculations. Specifically, we excluded school districts for which the state did not report finance data and school districts where the number of students and schools was zero or missing. We conducted this performance audit from February 2019 to June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix contains the closed- and open-ended questions from our surveys of (1) local educational agencies (referred to in this report as school districts or districts) and (2) state educational and school facility agencies. In some cases, respondents received different questions based on their response to a prior question. For example, school districts that conducted a facilities condition assessment in the last 10 years received additional questions about those assessments, however school districts that had not conducted such an assessment received questions to explain the reasons why. For a detailed discussion of our survey methodologies, see appendix I. This appendix summarizes key information on the condition of and funding for school facilities in districts that received Impact Aid. The Department of Education (Education) administers the Impact Aid program to assist school districts that experience a financial burden as a result of certain federal activities being carried out there. For example, federal Indian lands and military installations are exempt from property taxes—a key funding source that school districts use to offer a free public education. Impact Aid is intended to compensate school districts, in part, for the lost tax revenue. As noted in this report, property tax revenue was the most common source of funding school districts used for school facilities—an estimated 77 percent of all districts used property taxes for this purpose, based on our nationally representative survey. Districts with reduced property tax revenue, due to tax-exempt federal property or other reasons, may struggle to raise the funds needed for repairs and renovations to their school facilities. School districts that are eligible to receive Impact Aid might qualify for several types of payments under the program. About 90 percent of all Impact Aid funding falls under the category of Basic Support payments. According to the Congressional Research Service (CRS), school districts generally use these funds for current expenditures, such as administration, instruction, and transportation. However, because Impact Aid Basic Support payments are not limited to specific uses, school districts may also use them for capital expenditures. According to Education’s data, approximately 1,040 school districts (of a total of about 14,000 school districts nationwide) received Basic Support payments in fiscal year 2018 totaling $1.26 billion. The amount of these payments varied considerably by district—ranging from a high of about $55 million to a low of $540. Differences in the payments districts received resulted from several factors, including the number and types of federally-connected students the district served, according to CRS. In fiscal year 2018, there were 28 “heavily-impacted” school districts, meaning they were substantially affected by the presence of federally- connected children. Heavily-impacted districts receive increased Basic Support payments. In addition to Basic Support payments, some school districts are eligible for Impact Aid Construction grants for construction and emergency facility repair and renovation. From fiscal year 2014 to 2019, appropriations for Impact Aid Construction funds have consistently been about $17.4 million each year. According to CRS, appropriations language in recent years has determined whether Impact Aid Construction funds are distributed through formula grants to eligible school districts or competitive grants to a limited number of school districts, and from fiscal year 2013 to 2018, distribution alternated between these two types of grants. Approximately 150 school districts are eligible to receive Impact Aid Construction grants, according to Education officials. In fiscal year 2018, these funds were distributed through competitive grants and eight school districts received grants, ranging from $143,000 to $5.3 million. Sixty-seven school districts that received Impact Aid responded to our survey of school districts. In addition, eight of the 16 districts we visited received Impact Aid Basic Support payments in fiscal year 2018. These districts varied based on their proximity to different tax-exempt federal properties (i.e., military installations and Indian lands), as well as the number and percentage of federally-connected students they educated. Two districts we visited received Impact Aid Construction grants. Overall, on our survey of school districts, responses from the nongeneralizable group of districts that received Impact Aid were similar to the generalizable results for all districts nationwide both in terms of the key school building systems and features districts needed to update or replace and district priorities when updating or renovating school facilities. Table 3 shows the number of school districts receiving Impact Aid payments that reported that at least half of their schools needed updates or replacements to each building system or feature listed. As shown, districts most commonly indicated needing to update or replace heating, ventilation, and air conditioning systems (32); followed by safety and security (27), roofing (25), interior light fixtures (23), and plumbing (23). Based on our school district survey, 51 of 66 districts that received Impact Aid had conducted a facilities condition assessment of their schools at least once in the last 10 years. Of those 51 school districts, 34 reported assessing schools at least every 5 years. Nearly all districts (50 of 51) reported conducting the assessment for capital planning purposes and to assess safety and hazards. Similar to generalizable estimates from our nationally representative survey of school districts, districts that received Impact Aid placed a high priority on safety and security (59 of 66 districts), monitoring environmental conditions (55 of 64), and student access to technology (54 of 65). Overall, more than half of districts that received Impact Aid and responded to our survey (36 of 66) reported that local funding was their primary source for funding school facilities projects. In comparison, 19 districts reported state funding as their primary source, eight districts reported federal funding, and three districts selected the “Other” option or did not know. Similar to generalizable estimates from our survey of school districts, about three-quarters of districts that received Impact Aid and responded to our survey (49 of 66) reported using property tax revenue for school facilities. In addition, about two-thirds of them reported using local bonds and local grants for this purpose. Fewer districts reported using public- private partnerships, sales tax revenue, or other tax revenue for school facilities. As noted above, districts may receive Impact Aid because they have lost property tax revenue due to certain federal activities, including being on or near federal property that is exempt from property taxes. Districts that serve a large proportion of federally-connected students, such as those located on or near federal Indian lands or military installations, may look similar to high-poverty districts in their lack of access to local funding mechanisms for school facilities. However, there is wide variety in the amount of Impact Aid payments districts received. This variety was similarly reflected in the eight school districts we visited that received Impact Aid. For example, the Basic Support payments the districts we visited received in fiscal year 2018 ranged from about $16,000 to about $8.6 million, and the percentage of federally-connected students in the districts we visited ranged from 1 to 100 percent. Officials in one of the districts we visited that received Impact Aid explained that, because the district is located on an Indian reservation, there is no property tax base to levy or bond against. In the absence of these local funding options, officials said the district relied on state funding and some federal Impact Aid funding to address facility needs, and noted that the lack of local funding made it difficult for them to reach their goals for their school facilities. For example, officials said the state does not provide funding for designated classrooms for bilingual education. Because the district does not have the local property tax base to fund these spaces, officials said they must be creative with classes and teacher schedules to provide bilingual education. The location of these classes moves to different parts of the school at different times, meaning that teachers cannot set up a stable classroom that is properly equipped to teach bilingual education to students in the district, according to district officials. None of the officials we interviewed in the eight districts that received Impact Aid said their district used Basic Support payments to address issues with the conditions of school facilities. Officials in two districts we visited described receiving Impact Aid construction grants. Officials in one of these districts explained that when these funds are distributed via formula grant, the amounts are not large enough to support a major capital project. An official in the district that had received a competitive grant in recent years said the district used the funds to build a new combined middle and high school. In addition, representatives from the National Association of Federally Impacted Schools and the National Indian Impacted Schools Association told us they have heard anecdotally about some school districts using their Impact Aid funds as the basis for borrowing funds to pay for school facilities projects. They described this as particularly risky because Impact Aid appropriations levels are not guaranteed to remain consistent each year. The representatives said if funding levels for Impact Aid are reduced in the future, the districts would still have to pay back the borrowed funds before allocating funding for other purposes such as general operations, teacher salaries, educational materials, and other essentials for educating students in the school district. On our state survey, eight states reported providing additional school facilities funding or other assistance to districts in the state that receive Impact Aid. For example, an official in New Mexico told us the state has two programs targeted to school districts that get Impact Aid. One program awarded $10 million to districts in 2019 to help them provide teacher housing, according to state officials. State officials said a second state program in New Mexico awarded $24 million in 2019 to districts that received Impact Aid to assist them with projects that were ineligible for funding through New Mexico’s other programs. For example, these officials said this funding could help schools in need of athletic fields, performing arts centers, or administrative buildings. Charter schools comprise a small but growing group of public schools. We previously reported that, in contrast to most traditional public schools, many charter schools are responsible for financing their own buildings and other facilities, i.e., charter school districts may not have access to the same local funding mechanisms as traditional school districts. As a result, charter schools vary in terms of whether they own their own building or pay rent, and whether they operate in buildings originally designed as a school or in buildings that have been redesigned for educational purposes. Sometimes charter schools may also share space in their building with others, such as non-profit organizations. In addition to differences in facility access and finance, charter school governance also varies. We previously reported that in some states, charter schools function as their own school district, while in other states, charter schools have the option to choose between being their own school district or part of a larger school district. The data presented in this appendix are limited to the nongeneralizable responses of the 52 charter school districts that responded to our survey of school districts, unless otherwise noted. In addition, we visited five charter schools across four states (California, Florida, Maryland, and Rhode Island) as part of our school district site visits. This appendix summarizes key information on the condition of and funding for school facilities in these charter school districts and schools. Responses from the nongeneralizable group of charter school districts were similar to the generalizable results for all districts in the nation for key building updates, as well as priorities for modernizing school facilities, but different for how these districts access funding for school facilities. The highest number of charter school districts (20 of 51) indicated needing to update or replace heating, ventilation, and air conditioning systems in the majority of their schools, followed by windows (16), roofing (15), and interior light fixtures (15). School officials at a charter school we visited told us they were having ongoing issues with several key building features, such as doors and windows. The charter school rents their facility from the traditional school district and has a lease that specifies who is responsible for certain maintenance and repair projects. School officials told us the school has a “utilities-only” lease, meaning they should not be responsible for any repairs, but officials told us they had to take on several projects to make the facility usable. Although the traditional school district—of which this charter school is a part—is responsible for many of these projects, district officials said they have not had the funding to address this. For example, before the school opened, school officials said they had to install door handles on interior doors and re-key the building so that they were able to lock and unlock doors. In addition, school officials told us that teachers have complained that windows are nailed shut and cannot be opened. Based on our school district survey, 24 of 52 charter school districts had conducted a facilities condition assessment of their schools at least once in the last 10 years. Of those 24 school districts, 19 reported assessing schools at least every 5 years. Twenty-three charter school districts reported conducting the assessment to assess safety and hazards. Officials at four of the five charter schools we visited told us they were responsible for maintaining their own facilities. The other charter school we visited was part of a larger network of charter schools, and had regional offices that assisted with facilities and operations. When updating or renovating school facilities, charter school districts responding to our survey ranked security and technology as their highest priorities, similar to the generalizable results for all districts in the nation. The top reported priorities were student access to technology (44 of 52), safety and security (43 of 51 districts), and telecommunication systems such as WiFi (36 of 51). An official at a charter school we visited in Florida said safety and security was one of their main focuses when constructing the school. The school and parking lot are gated, and there is a camera to monitor all cars and people entering the campus. School officials told us that all classrooms and common areas are equipped with phones that can broadcast announcements throughout the campus, and that they have a lightening alert system so that they can move students indoors if a storm is approaching. As previously noted, charter schools may or may not be part of a larger school district, and may not be able to access local funding sources such as property tax revenue. As noted in this report, property tax revenue was the most common source of funding that all school districts reported using for school facilities—an estimated 77 percent of all districts nationwide used property taxes for this purpose. Most charter school districts that responded to our survey indicated that state funding was their primary method of funding school facilities (32 of 49) and fewer (8 of 49) reported local funding as their primary method. The most common local funding mechanism that charter school districts reported using for facilities was grant funding (20 of 46 districts), followed by public-private partnerships (12 of 47 districts). A charter school we visited told us about several areas in their school that they had improved with grants from non-profit organizations. For example, a teacher at the school applied for a grant from a foundation to replace the basketball hoops and paint in the gym, and a separate organization had installed a new playground at the school. Based on our state survey, 26 states provide funding to charter schools for facilities—22 states provide direct funding to charter schools and four states provide funding to non-charter school districts, which would indirectly fund certain charter schools. Of the 26 states, 20 states reported doing so either through a funding formula, or a combination of funding formula, charter school requests, and other methods. The most common factor that states considered when determining levels of facilities funding for charter schools was the size of the student population (12 of 25 states). Of the 26 states that provide funding to charter schools for construction or maintenance and operations of charter school facilities, 19 reported using allocated funding from the state legislature to do so. Jacqueline M. Nowicki, Director, (617) 788-0580 or nowickij@gao.gov. In addition to the contact named above, Bill MacBlane (Assistant Director), David Watsula (Analyst-in-Charge), Liz Spurgeon, and Alexandra Squitieri made key contributions to this report. Mariel Alper, Michael Armes, Susan Aschoff, John Bauckman, Alex Galuten, Alison Grantham, Elizabeth Hartjes, Lara Laufer, Sheila R. McCoy, Jean McSween, John Mingus, Lauren Mosteller, Mimi Nguyen, Jean Recklau, Almeta Spencer, Manuel Valverde, Sonya Vartivarian, and Paul Wright provided additional support.", "summary": "Public school facilities primarily serve an educational role, and they also serve a civic role as voting places and emergency shelters. School districts collectively spend tens of billions of dollars each year on facilities construction needs at the nearly 100,000 K-12 public schools nationwide. The Joint Explanatory Statement accompanying the Department of Defense and Labor, Health and Human Services, and Education Appropriations Act, 2019 and Continuing Appropriations Act, 2019 included a provision for GAO to study the condition of public school facilities. This report examines (1) the common facility condition issues school districts identify in public schools and how they have done so and (2) school districts' highest priorities for their school facility renovations and updates, and how districts and states fund them. GAO conducted a nationally representative survey of school districts and also surveyed 50 states and the District of Columbia; visited 55 schools in 16 districts across six states, selected for geographic variation and other characteristics; analyzed federal data on school district expenditures for capital construction projects; and interviewed federal, state, district, and school officials. About half (an estimated 54 percent) of public school districts need to update or replace multiple building systems or features in their schools, according to GAO's national survey of school districts. For example, an estimated 41 percent of districts need to update or replace heating, ventilation, and air conditioning (HVAC) systems in at least half of their schools, representing about 36,000 schools nationwide that need HVAC updates (see figure). In about half of the 55 schools GAO visited in six states, officials described HVAC-related problems, such as older systems that leaked and damaged flooring or ceiling tiles. If not addressed, such problems can lead to indoor air quality problems and mold, and in some cases caused schools to adjust schedules temporarily. To determine the condition of their school facilities, an estimated two-thirds of districts conducted a facilities condition assessment at least once in the last 10 years. According to GAO's survey of the 50 states and District of Columbia, most states do not conduct statewide assessments to determine school facilities' needs and instead leave this task to school districts. School districts' highest priorities for their school facilities were improving security (an estimated 92 percent), expanding student access to technology (87 percent), and monitoring health hazards (78 percent), according to GAO's school district survey. In school districts GAO visited, officials said they first address health hazards and safety issues. In nearly all districts GAO visited, security also had become a top priority, with some districts prioritizing security updates over replacing building systems, such as HVAC. In about half of districts nationwide, funding for school facilities primarily came from local sources such as property taxes, based on GAO's survey. High-poverty districts more commonly relied on state funding and used property taxes less commonly than low-poverty districts. According to GAO's state survey, 36 states provided capital funding to school districts for school construction or renovations, including five of the six states GAO visited, though the funding amounts and mechanisms differed considerably within and across states.", "document_type": "gao"}
{"report": "CBP is the nation’s largest federal law enforcement agency. CBP’s Border Patrol and AMO are the uniformed law enforcement arms responsible for securing U.S. borders between ports of entry in the air, land, and maritime environments. Border Patrol has primary responsibility for securing U.S. land borders between ports of entry. Its area of responsibility along the northern border is divided among eight sectors: Blaine, Spokane, Havre, Grand Forks, Detroit, Buffalo, Swanton, and Houlton. Each Border Patrol sector is further divided into Border Patrol stations and each station is assigned a certain geographic area of responsibility within a sector. Along the northern border, there are a total of 49 stations or between four to eight stations per sector. For a map of Border Patrol’s northern border sectors, see figure 1. Border Patrol agents secure the border between ports of entry, in part, through patrolling international land borders and waterways to detect and prevent the illegal trafficking of people, narcotics, and contraband into the United States. AMO has primary responsibility for securing U.S. borders in the air, marine, and land domains and its operations along the northern border are divided among three branches: Bellingham Air and Marine Branch in Washington, Great Lakes Air and Marine Branch in Michigan, and Manassas Air Branch in Virginia. Each branch is further divided into units to conduct air or maritime missions and there are a total of seven air units and nine marine units along the northern border. For a map of AMO’s northern border operating locations, see figure 2. AMO Air Interdiction Agents are federal law enforcement agents who pilot aircraft, while Marine Interdiction Agents are federal law enforcement agents who operate vessels. Air and Marine Interdiction Agents secure the air and maritime environments along the border, in part, through conducting surveillance and investigative activities to interdict smuggled narcotics and other contraband. Additional offices within CBP that support the activities of Border Patrol and AMO along the northern border include the Office of Facilities and Asset Management, Office of Information and Technology (OIT), and Office of Intelligence. The Office of Facilities and Asset Management is responsible for oversight and management of CBP’s real and personal property portfolios, including managing CBP’s facilities and motor vehicle fleets. OIT is responsible for managing CBP’s technology infrastructure and information technology (IT) operations. These, according to OIT, enable CBP mission readiness and improve the ability of all employees, including agents in the field, to be proactive and responsive to new threats. OIT manages all IT networks, computers, systems, data, tactical communications, and other resources to support CBP employees. OIT is also to provide day-to-day field support primarily through Field Technology Officers who provide services to CBP’s offices and components, such as repairing equipment, upgrading systems and networks, restoring system outages, responding to cybersecurity incidents, and deploying new technology and equipment. The Office of Intelligence is to develop, coordinate, and implement CBP’s intelligence capabilities into a cohesive intelligence enterprise that supports CBP’s primary mission to secure the borders while facilitating legitimate trade and travel. The Office of Intelligence’s Field Intelligence Division is to provide CBP law enforcement personnel with current and relevant intelligence to inform decision makers and those who respond to border related crimes, threats, and hazards. In this division, there are two field intelligence groups with areas of responsibility along the northern border—the Pacific Northwest Field Intelligence Group in Washington and the Great Lakes Field Intelligence Group in Michigan. In addition, through CBP’s National Border Geospatial Intelligence Strategy, the Office of Intelligence produces geospatial intelligence products for Border Patrol sectors to identify areas of potential illicit cross-border activity. Border Patrol and AMO use a variety of technologies, facilities, and other resources to secure the northern border between ports of entry. Figure 3 illustrates examples of resources used by Border Patrol and AMO, which include the following: surveillance technology, such as Remote Video Surveillance Systems—systems of towers with cameras that transmit information to video monitors at a Border Patrol facility—and unattended ground sensors—remotely monitored sensors placed in or on the ground, or on ground-based platforms, to detect, track, identify, and differentiate humans, animals, and vehicles—used by Border Patrol agents to detect and identify illicit cross-border activity; radar systems to detect and identify aircraft and vessel incursions; IT and communication systems to conduct operations and ensure the safety and security of agents while on duty, including databases and systems for processing detainees, infrastructure to operate surveillance technology, and tactical communication equipment such as land mobile radios; aircraft, including fixed- and rotary-wing aircraft, vehicles, including all- terrain vehicles and snowmobiles, and large and small vessels; tactical infrastructure, including fencing, roads, and border markers and signs; and facilities, including buildings to house workstations and offices for agents and civilian personnel, short-term detention facilities to process and hold individuals arrested by Border Patrol agents, forward operating bases in remote locations to support Border Patrol agent operations, and hangars for aircraft and vessel storage and repair. CBP participates in a variety of collaborative efforts—including task forces, joint operations, and partnerships with federal, state, and local law enforcement agencies—to support its efforts to secure the northern border between ports of entry. According to CBP officials, collaborative efforts involve sharing intelligence and other information that informs and guides the efficient use of agents and resources to conduct enforcement activities. For example, AMO’s Air and Marine Operations Center coordinates with federal, state, local, and international law enforcement agencies to detect, identify, track, and coordinate interdiction of suspect aviation and maritime activity near and at the borders, including the northern border, and within the United States. Moreover, Border Patrol’s Northern Border Coordination Center serves as a centralized coordination center for information sharing among Border Patrol’s eight northern border sectors, as well as with domestic and international law enforcement partners, focusing primarily on counter-terrorism and illicit criminal networks. Border Patrol also collaborates with county, state, tribal, local, and other law enforcement agencies through administration of the Operation Stonegarden Grant Program, a part of the Homeland Security Grant Program, to support border security activities. The grant program provides funding to state, local, and tribal law enforcement agencies to support joint efforts to secure U.S. borders. For example, grantees may receive reimbursement for operational overtime costs associated with law enforcement activities and equipment purchases, such as sensors, in support of border security activities. CBP’s collaborative efforts along the northern border also include participation in various task forces with federal, state, and local law enforcement agencies. Specifically, Border Patrol and AMO agents may be assigned as task force officers to conduct or support casework, investigations, and coordination among federal, state, and local law enforcement agencies. For example, Border Patrol and AMO agents are assigned as task force officers along the northern border on the U.S. Immigration and Customs Enforcement-led Border Enforcement Security Task Force in Washington, Michigan, and New York to identify, investigate, disrupt, and dismantle transnational criminal organizations. According to Border Patrol and AMO officials, task force officers help enhance partnerships, information sharing, and situational awareness along the northern border. CBP also partners with other DHS components to support its efforts to secure the northern border between ports of entry. For example, through the Puget Sound Regional Coordinating Mechanism, CBP—including Border Patrol and AMO—and the U.S. Coast Guard coordinate daily and conduct joint operations along the maritime border between the state of Washington and province of British Columbia. CBP also works with DHS’s Science and Technology Directorate to identify, develop, and evaluate technology to address capability gaps across the northern border. For example, DHS’s Science and Technology Directorate, in collaboration with Swanton Border Patrol sector, deployed land surveillance technology along the northern border. CBP also collaborates with law enforcement agencies within the government of Canada through the Cross-Border Law Enforcement Advisory Committee and Integrated Border Enforcement Team Program. The Cross-Border Law Enforcement Advisory Committee is a national- level committee—comprised of the Royal Canadian Mounted Police, Canada Border Services Agency, U.S. Immigration and Customs Enforcement, CBP, and U.S. Coast Guard—that provides guidance to initiatives involving partnerships between United States and Canadian law enforcement agencies along the shared border. The Integrated Border Enforcement Team Program includes the Royal Canadian Mounted Police, Canada Border Services Agency, U.S. Immigration and Customs Enforcement, CBP, and U.S. Coast Guard. According to CBP, the priority of the program is to seek and identify mutual national security threats and combat illicit cross-border activity. According to CBP and government of Canada officials, program activities may include real-time tactical intelligence sharing between Canadian and U.S. law enforcement agencies and periodic meetings to coordinate operations. These officials stated that the program helps to facilitate timely information sharing in accordance with Canadian and U.S. laws and regulations. For example, through the Integrated Border Enforcement Team Charter, Border Patrol and the Royal Canadian Mounted Police may share information related to cross-border criminal activity—such as suspected or known illegal entries between ports of entry—without delay. According to DHS’s 2017 Northern Border Threat Analysis Report, the most common threat to U.S. public safety along the northern border continues to be contraband smuggling; specifically, the bidirectional flow of illicit drugs. In its fiscal year 2018 intelligence reports for its eight northern border sectors, Border Patrol also reported contraband smuggling as a significant threat along the northern border between ports of entry, including bidirectional drug smuggling. According to Border Patrol data for fiscal years 2013 through 2017, 2 percent of Border Patrol’s total drug seizures occurred along the northern border. Examples of smuggling activities include criminal groups with known ties to or hired by Mexican drug trafficking organizations suspected of smuggling narcotics into Canada and smuggling bulk currency from Canada into the United States between land border ports of entry. Border Patrol, in its intelligence reports, also identified contraband smuggling for the purpose of evading customs duties, involving products such as tobacco, prohibited fruits, and medicinal products. Further, according to Border Patrol, criminal organizations smuggle contraband between ports of entry because certain items such as tobacco, agricultural, and medicinal products are prohibited for import even if properly declared at a port of entry. In 2017, AMO reported contraband smuggling across the northern border both into and out of the United States between ports of entry. In its 2017 Northern Border Non-Commercial General Aviation Threat Overview, AMO’s Air and Marine Operations Center identified illicit activity along the northern border using general aviation aircraft, including aircraft operating in a suspicious manner at low attitude (low-flying aircraft). According to Border Patrol’s annual fiscal year 2018 intelligence report, violations of U.S. immigration and travel controls, which Border Patrol refers to generally as “illegal immigration,” along the northern border is a threat and is frequently bidirectional between the United States and Canada. Additionally, our analysis of Border Patrol data from fiscal years 2013 through 2017 showed that Border Patrol agents apprehended 14,319 potentially removable aliens—foreign nationals who Border Patrol suspected or determined were removable from the United States—along the northern border or approximately 1 percent of its total nationwide apprehensions of potentially removable aliens (1.97 million aliens). According to DHS’s 2017 Northern Border Threat Analysis Report, known illegal crossings between ports of entry by individuals on the northern border conform to established migration patterns between large population centers. Further, the report states that terrain, weather, and distance are factors that constrain migrant travel between ports of entry in remote areas of the border. According to Border Patrol officials, the majority of individuals apprehended along the northern border are suspected or known to have illegally entered the United States across the southwest border and traveled to the northern border region before being detected, while a smaller number of individuals are suspected or known to have illegally entered the United States from Canada between ports of entry. Specifically, of the potentially removable aliens apprehended by Border Patrol along the northern border during this period, we found that 61 percent (8,727) were individuals suspected or known to have illegally entered the United States from Mexico, while 19 percent (2,782) were individuals suspected or known to have illegally entered the United States from Canada. The Swanton Border Patrol sector apprehended the highest percentage of individuals who illegally entered the United States from Canada between ports of entry during this period, 43 percent (1,206 individuals) of the total number across all eight northern border sectors. Border Patrol, in its fiscal year 2018 intelligence reports for its eight northern border sectors, also identified alien smuggling—bringing into, or harboring or transporting within, the United States, foreign nationals in violation of U.S. immigration law—organizations operating along the northern border between ports of entry as a threat. Examples of alien smuggling activities include alien smuggling organizations using private residences along international waterways to provide locations for staging an illegal entry. According to Border Patrol officials, criminal organizations operating along the U.S.-Canada border frequently conduct bidirectional alien smuggling activities between the United States and Canada as agents encounter numerous types of groups being smuggled into Canada. CBP identified staffing and resource challenges to its operations and enforcement activities across the northern border and has identified actions to address them, but faces competing priorities. Border Patrol and AMO officials we met with identified agent staffing challenges along the northern border across all sectors and branches that limit enforcement activities, including Border Patrol agent availability to conduct patrol missions and a limited number and frequency of AMO missions due to AMO agent availability. Border Patrol and AMO officials also identified resource challenges along the northern border across all sectors and branches, including radar and surveillance technology used to surveil the air, maritime, and land environments; IT and communication technology, including network infrastructure and bandwidth that allow agents to access CBP systems and tactical communications, such as land mobile radios for agent communication during border security missions; and infrastructure and facilities, including tactical infrastructure—roads, fencing, and border markers—and facilities used by agents to secure the border. It is unknown whether the staffing and resource challenges identified by CBP to secure the northern border between ports of entry will be addressed due to competing southwest border security priorities. CBP identified actions and ongoing efforts to address agent staffing and resource challenges to secure the northern border between ports of entry. In June 2018, DHS released a Northern Border Strategy to establish actions that are intended to, among other things, improve DHS’s efforts to safeguard the northern border against various threats. DHS is developing an implementation plan for its Northern Border Strategy which will, among other things, identify actions to address gaps in capabilities to secure the northern border between ports of entry. However, it is unknown whether CBP’s northern border staffing and resource challenges will be addressed due to competing priorities with southwest border security. For example, instructions in Executive Order 13767 require DHS to obtain complete operational control—prevention of all unlawful entries into the United States, including entries by terrorists, other unlawful aliens, instruments of terrorism, narcotics, and other contraband—of the southwest border, in part through hiring thousands of agents and constructing a physical barrier. Border Patrol officials identified staffing challenges across the northern border sectors that have affected enforcement activities. Officials from northern border sectors told us that an insufficient number of agents authorized or onboard at its sectors and stations limits their ability to conduct enforcement activities and may, at times, pose risks to agent safety. In addition, Border Patrol officials from northern border sectors stated that agent availability for enforcement activities is further limited by detainee transportation and supervision duties and requests for law enforcement assistance from other agencies. For example, Border Patrol sector officials stated that detainee transportation duties result in agents being unable to conduct enforcement activities for up to 1 day and duties related to supervision of detainees during court proceedings and meetings with federal prosecutors may result in agents being unable to conduct enforcement activities for up to 1 week. Further, responding to local calls for assistance during assaults may result in agents being unable to conduct enforcement operations for multiple hours. Also, Border Patrol officials from northern border sectors stated that vacancies in civilian Law Enforcement Communication Assistant positions affect enforcement activities. Law Enforcement Communication Assistant duties at each northern border sector are dispatching and officer safety checks, monitoring surveillance camera feeds and unattended ground sensor activation, and conducting intelligence research checks for agents on duty across all stations in the sector. Border Patrol officials told us it is difficult to recruit and retain qualified applicants for vacant positions due to the lower General Schedule grade of the position across Border Patrol, which is not competitive with salaries for similar positions offered through state and local law enforcement agencies. In August 2018, Border Patrol officials stated that they created an additional position, the Law Enforcement Information Specialist, with additional duties and responsibilities at a higher General Schedule grade. AMO identified staffing challenges across its northern border branches which, according to AMO officials, have affected the frequency and number of air and maritime missions. Specifically, officials at AMO branches told us that an insufficient number of agents authorized or onboard at its branches and units limits the frequency and number of air and maritime missions AMO is able to conduct along the northern border. For example, AMO officials stated that an insufficient number of Marine Interdiction Agents limits the number of daily and weekly maritime patrol missions. For air missions, AMO officials stated that an insufficient number of Air Interdiction Agents may limit the ability to fulfill immediate or previously unscheduled requests for air support. AMO officials from northern border branches also cited agent recruitment, hiring, and retention as a challenge for filling vacant positions. For example, officials stated that AMO faces competition with commercial airline companies for recruitment and retention of qualified individuals with commercial pilot certificates, including higher salaries, as well as delays from CBP’s lengthy application process. AMO officials from northern branches also stated that agent availability for air and maritime missions is sometimes limited due to temporary duty assignments to support national missions, which can limit local operations along the northern border. AMO officials stated that these temporary duty assignments involve relocation of Air Interdiction Agents, aircraft, and maintenance staff to other operating locations for multiple weeks. For example, in 2017, Air Interdiction Agents flew missions to support recovery efforts after the hurricanes in Texas, Florida, and Puerto Rico. In 2018, Air Interdiction Agents supported security operations during the Super Bowl in Minneapolis, Minnesota. CBP is taking actions to address agent recruitment, hiring, and retention. We reported in June 2018 on CBP’s actions to address challenges for recruitment, hiring, and retention of Border Patrol and AMO agents, such as increased participation in recruitment events and offering relocation opportunities for existing employees. According to CBP’s Fiscal Year 2019 Congressional Budget Justification, newly hired Border Patrol agents will be assigned to the southwest border to allow for the reassignment of more experienced agents to the northern border. As of August 2019, Border Patrol officials expected that all sectors in fiscal year 2019, including the northern border sectors, would receive an increase in the number of authorized agent positions. Border Patrol officials also stated that as of June 2018, they were completing a Personnel Requirements Determination Initiative to analyze agent allocations across its sectors and stations to develop a staffing allocation model to optimally align staff according to workload and area of responsibility conditions. In June 2018, we also reported that AMO had taken steps to address staffing challenges, such as implementing voluntary paid relocation opportunities and pursuing additional human capital flexibilities to address its difficulty in retaining Air Interdiction Agents, including a group retention incentive and a special salary rate. AMO personnel who are non- bargaining unit employees and have served for at least 3 years in their current location are also eligible for noncompetitive paid relocations. According to AMO officials, these opportunities are posted every few months and eligible personnel can apply for transfers to a specific duty station based on the needs of the operational component. In September 2017, AMO submitted an official request for a 10 percent group retention incentive for Air Interdiction Agents staffed to the northern border, among other locations. According to the request, the incentive is intended to help AMO retain qualified pilots in these hard-to-fill locations by raising their salaries to be more competitive with commercial airlines. Border Patrol officials we met with stated that Border Patrol’s Operational Mobility and Resident Agent Programs have helped northern border sectors to address agent staffing challenges. The Operational Mobility Program provides Border Patrol agents with opportunities for a paid relocation to a more desirable location at a lower cost to CBP than an official permanent change of station transfer. Border Patrol officials stated that the use of the Operational Mobility Program resulted in agents electing to relocate to northern border sectors from other duty stations. The Resident Agent Program operates in locations where Border Patrol’s routine presence is extremely limited and is intended to improve situational awareness by the creation of partnerships, expansion of community outreach, and development and dissemination of intelligence. The Resident Agent location is the physical residence of an agent in a location where there is not an official Border Patrol station. Officials from Border Patrol sectors and AMO branches stated that there are gaps in air radar coverage along the northern border, limiting their ability to detect and identify aircraft incursions. CBP has taken actions to address these gaps in air radar coverage. In December 2017, CBP completed an AMO-led assessment of air radar capabilities, which identified coverage gaps and needs across the United States, including at the northern border. In May 2018, AMO officials stated that they began working with the Department of Defense to test technology along the northern border to address gaps in air radar coverage. Officials from Border Patrol sectors and AMO branches stated that there are limited maritime radar capabilities to detect and identify vessel incursions along the northern border. CBP has taken actions to address these gaps in maritime radar capabilities. Border Patrol, through its Maritime Detection Project, plans to deploy additional maritime radar technology in Detroit and Buffalo sectors to expand maritime radar coverage on Lake Erie. Also, in 2017, CBP participated in a 1-year DHS pilot project with the government of Canada to share radar information in an area along the northern border to detect vessel incursions. AMO, through its Multi-Role Enforcement Aircraft, conducts maritime radar patrols along portions of the northern border to address gaps in maritime radar coverage on some of the Great Lakes and parts of the Pacific Northwest, to detect and identify vessel incursions. Border Patrol sector officials stated that there are challenges with land surveillance technology that is used for agents to detect, identify, and respond to illicit cross-border activity along the northern border. Further, Border Patrol headquarters and sector officials stated that there are gaps in surveillance technology coverage along the northern border to detect and identify illicit cross-border activity. In addition, Border Patrol officials also identified challenges with Legacy Remote Video Surveillance Systems. For example, officials we met with identified system outages due to delays in maintenance and replacement of parts, and poor quality video surveillance camera images. In March 2017, CBP completed a Border Patrol-led assessment of land surveillance capabilities to assess gaps, including gaps in surveillance technology coverage across all Border Patrol sectors. Officials from Border Patrol sectors and AMO branches we met with identified inadequate network infrastructure—including network infrastructure and equipment nearing or past its useful life—and bandwidth that have affected enforcement activities and other required tasks along the northern border. For example, Border Patrol officials stated that inadequate network infrastructure and bandwidth has delayed or prevented the processing of detainees at some stations. AMO officials also stated that inadequate bandwidth limits the ability of agents to use BigPipe, a system used to coordinate operations with partner agencies during air and maritime missions. In September 2017, DHS’s Office of Inspector General found that outdated IT infrastructure and equipment contributed to CBP-wide system performance and availability challenges; a considerable portion of IT equipment and infrastructure had reached its useful life; and OIT was unable to replace infrastructure past its useful life because of financial constraints. CBP’s Fiscal Year 2019 Congressional Budget Justification identifies actions to improve network infrastructure and bandwidth, including deploying new workstations and replacing network infrastructure components that are past their useful life to provide reliable operations and address vulnerabilities. OIT officials stated that pilot projects using virtual private network connections are being implemented at CBP locations to address bandwidth challenges and reduce costs. Officials from Border Patrol sectors and AMO branches we met with identified challenges with tactical communications, including gaps in land mobile radio coverage along the northern border. Border Patrol and AMO agents responsible for securing the northern border depend on land mobile radio systems for secure, reliable, and timely exchanges of critical information to effectively carry out their mission. Border Patrol and AMO officials we met with identified lack of coverage in certain areas, which impacts agent communication during enforcement activities. CBP has taken actions to identify coverage gaps and deploy additional equipment to improve communications coverage along the northern border. For example, CBP has deployed additional equipment to improve tactical communication coverage in Border Patrol’s Houlton sector in Maine through its Tactical Communication Modernization Program from fiscal years 2009 through 2017. Border Patrol officials stated that they are deploying repeater tower sites—technology used for retransmitting and extending the range of radio communications—and other technology to mitigate dead spots and gaps in coverage in three sectors. According to CBP’s Fiscal Year 2019 Congressional Budget Justification, updated handheld and mobile radios are being provided to Border Patrol and AMO, including northern border locations, to improve tactical communications and interoperability with law enforcement partners. Border Patrol sector officials identified challenges due to limited tactical infrastructure, such as a lack of barriers to impede vehicle incursions and access to roads along the border that make it difficult to impede illegal entries. For example in one sector, officials stated that a lack of vehicle barriers leads to a gap in Border Patrol’s ability to impede illicit vehicle incursions. In other sectors, officials stated that Border Patrol agents face challenges accessing border areas due to a lack of roads or access to maintained roads. Officials from northern border sectors also stated that agents face challenges preventing illegal entries due to a lack of barriers and a lack of signs or markers indicating the location of the international border. Officials from Border Patrol sectors and AMO branches we met with noted that certain facilities do not have space to accommodate the number of assigned agents and civilian personnel along the northern border. For example, in one sector, officials stated that there is lack of space to accommodate Law Enforcement Communication Assistants to monitor surveillance technology and direct agents to respond to potential illicit activity. Border Patrol officials in other sectors also stated that certain stations in their sectors do not have adequate facilities to process and house detainees. For example, one station lacks a dedicated processing and interview area and detainees are processed in an open location next to agent workstations, which may pose a safety risk to agents, according to officials. In November 2018, Office of Facilities and Asset Management officials identified 20 new and major construction projects planned for the northern border, including replacement of Border Patrol facilities with identified challenges; however, these projects have been deferred due to lack of funding. Further, according to Office of Facilities and Asset Management officials, CBP has insufficient funds to address deferred maintenance projects and a limited number of maintenance staff to repair facilities. Officials from Border Patrol sectors we met with identified aging vehicles that are beyond their expected service life, which affect enforcement activities along the northern border. According to Border Patrol officials, funding is not available to replace aging vehicles across all sectors, but funds are allocated annually to replace a percentage of vehicles in the northern border sectors that are beyond their expected service life. Further, Border Patrol officials stated that the harsh climate along the northern border creates additional burdens on agent vehicles prior to those vehicles reaching the end of their expected service life. Officials from Border Patrol sectors we met with identified agent vehicles that lack the technology needed to complete monthly motor vehicle utilization reports required by the DHS Stop Asset and Vehicle Excess Act. In August 2018, Border Patrol officials stated that CBP was in the process of awarding a contract for installation of vehicle reporting technology in agent vehicles, including across the northern border sectors. In addition to the actions identified above by CBP to address northern border staffing and resource challenges, DHS is developing an implementation plan for its Northern Border Strategy, which includes a goal to enhance border security operations. The strategy states that the implementation plan is intended to outline roles, responsibilities, programs, and timelines for accomplishing the strategy’s goals and objectives for fiscal years 2020 to 2024. According to DHS officials, the department plans to use the strategy and corresponding implementation plan to prioritize departmental resources and achieve the specified outcomes over the 5-year period. According to DHS officials, the implementation plan is expected to be completed in 2019 and will identify actions to address gaps in capabilities to secure the northern border between ports of entry; for example, gaps in domain awareness and associated technology, among other things. It is unknown whether the staffing and resource challenges identified by CBP to secure the northern border between ports of entry will be addressed due to competing southwest border security priorities. According to Border Patrol and AMO headquarters officials, resources are allocated across their operating areas based on threats and volume of illicit activity, which are greatest on the southwest border. Further, Border Patrol and AMO headquarters officials stated that resource allocation is prioritized to the southwest border to also meet instructions in Executive Order 13767 to obtain complete operational control—prevention of all unlawful entries into the United States, including entries by terrorists, other unlawful (i.e. inadmissible) aliens, instruments of terrorism, narcotics, and other contraband—of the southwest border. While DHS is implementing its Northern Border Strategy, including developing an implementation plan, addressing CBP’s northern border staffing and resource challenge will compete with its other enforcement priorities along the southwest border. While CBP has performance measures (strategic and management) that assess certain border security operations or programs, some of which include data from the northern border, it does not have specific measures to assess its effectiveness at securing the northern border between ports of entry. More specifically, Border Patrol has two strategic measures that include data from the northern border, but these measures do not assess Border Patrol’s effectiveness at securing the northern border between ports of entry. The two measures—the percent of recurring border surveillance implemented in remote, low-risk areas between ports of entry and the percent of time Border Patrol meets its goal of responding to potential illegal activity in remote, low-risk areas—are based on information from CBP’s National Border Geospatial Intelligence Strategy. The measures assess Border Patrol’s use of reports developed using geospatial intelligence technology of potential illicit cross-border activity. However, this technology is not applied in maritime environments, so the measures do not include data from two northern border sectors. Further, Border Patrol’s two strategic measures combine data from the southwest and northern borders. Border Patrol has four management measures that also contain data from the northern border. These measures are (1) the number of joint operations conducted along the northern border by Border Patrol agents and Canadian law enforcement; (2) the percent of apprehensions at Border Patrol checkpoints; (3) the percent of Border Patrol agents who are trained and certified to perform enforcement actions; and (4) the percent of Border Patrol equipment assessed as ready to support law enforcement operations. Border Patrol’s four management measures include data from the northern border, but do not assess Border Patrol’s effectiveness at securing the northern border between ports of entry. Although one management measure tracks the number of joint operations conducted along the northern border by Border Patrol agents and Canadian law enforcement personnel, that measure does not assess Border Patrol’s performance in conducting those joint operations or their effectiveness. Border Patrol’s three additional management measures include data from the northern border combined with other areas, such as the southwest border, and therefore are not specific to the northern border. AMO’s one strategic and one management measure include data from the northern border, but do not assess AMO’s effectiveness at securing the northern border between ports of entry in the air and maritime environments. For the strategic measure, AMO reports the percent of detected conventional aircraft incursions resolved. The measure represents the percent of conventional aircraft detected visually or by sensor technology, suspected of illicit cross-border activity, which are brought to a successful resolution by its Air and Marine Operations Center. For the management measure, AMO reports air mission launch rate, which is the percent of all requests made for aircraft to which AMO was able to respond. These two measures include data across all border areas, including the northern border, but are not specific to the northern border. Border Patrol officials stated that they have not developed or implemented performance measures to assess their effectiveness at securing the northern border between ports of entry because of competing priorities related to developing measures for southwest border security. According to Border Patrol officials responsible for developing and implementing performance measures, Border Patrol’s priority is to develop measures to assess the effectiveness of its efforts to secure the southwest border, such as the effort to achieve complete operational control as outlined in the Executive Order 13767 instructions and the fiscal year 2018 DHS agency priority goal. Specifically, Border Patrol is required to implement a measure to assess operational control for all southwest border sectors by the end of fiscal year 2019. Border Patrol defines operational control as its ability to impede or deny illegal border crossings, maintain situational awareness, and apply appropriate, time- bound law enforcement response and resolution between the ports of entry. According to Border Patrol officials, the ongoing efforts to develop measures for the southwest border will eventually be applied to the northern border, but it is unknown how these ongoing efforts will be implemented to assess Border Patrol’s performance at securing the northern border between ports of entry. Border Patrol officials stated that following the implementation of operational control for the southwest border, Border Patrol plans to implement the operational control measure along the northern border in fiscal year 2020. Border Patrol officials stated that they are in the early stages of this process, and could not provide any information on how operational control will be implemented for its operations along the northern border. Further, Border Patrol officials could not provide information on how operational control will be used to assess Border Patrol’s performance for securing the northern border between ports of entry. Additionally, in 2012 we recommended that Border Patrol establish milestones and time frames for developing performance measures to support implementation of its 2012-2016 Strategic Plan, including assessing progress made in securing the northern border between ports of entry and informing resource identification and allocation efforts. DHS concurred with our recommendations, and Border Patrol made progress in developing new performance measures for border security. However, we closed the recommendations as not implemented in September 2017 because the measures identified did not apply to the entire northern or coastal borders, as well as the remaining uncertainty about when Border Patrol would develop a new strategic plan. AMO officials stated that they have not implemented performance measures to assess AMO’s effectiveness at securing the northern border between ports of entry in the air and maritime environments because of difficulties in creating region-specific performance targets. Specifically, AMO officials stated that it is difficult to set performance targets for a specific region, such as the northern border, because the threat environment is constantly changing. Also, the officials stated that AMO is waiting for completion of the Northern Border Strategy implementation plan before developing any performance measures specific to the northern border. Additionally, Border Patrol and AMO have ongoing efforts to develop border security metrics pursuant to the National Defense Authorization Act for Fiscal Year 2017. The act directs DHS to annually report metrics and associated data and methodology, including metrics for border security between ports of entry. Consistent with GPRAMA, agencies should establish a balanced set of performance measures, which reinforces the need for agencies to have a variety of measures across program areas. Furthermore, Standards for Internal Control in the Federal Government state that management should determine whether performance measures for the defined objectives are appropriate for evaluating the entity’s performance using targets and milestones. The standards also state that management should track entity achievements and compare actual performance to planned or expected results using established activities such as comparisons and assessments. Border Patrol and AMO could leverage and use their ongoing efforts to develop and implement performance measures to assess effectiveness at securing the northern border between ports of entry. For example, Border Patrol and AMO could use the metrics developed in accordance with the Fiscal Year 2017 National Defense Authorization Act to help inform the development of northern border performance measures. Developing and implementing such measures could help Border Patrol and AMO better assess the effectiveness of their northern border operations between ports of entry, including challenges due to limited staffing and resources, and take corrective actions, as necessary. The United States and Canada share the longest common non-militarized border between two countries, spanning nearly 4,000 miles; however, CBP has historically focused attention and resources, including resources to develop and implement performance measures, primarily on the nearly 2,000 mile U.S.-Mexico border. While Border Patrol and AMO have performance measures that assess specific border security operations or programs that include data from the northern border, these measures generally combine data with other border regions and collectively the measures do not assess effectiveness at securing the northern border between ports of entry. Without northern border performance measures, Border Patrol and AMO cannot assess their effectiveness at securing the northern border between ports of entry. Developing and implementing northern border performance measures could help Border Patrol and AMO assess its northern border operations and address identified challenges. We are making two recommendations, one to Border Patrol and one to AMO. The Chief of Border Patrol should develop and implement performance measures to assess its effectiveness at securing the northern border between ports of entry (Recommendation 1). The Executive Assistant Commissioner of AMO should develop and implement performance measures to assess its effectiveness at securing the northern border between ports of entry in the air and maritime environments (Recommendation 2). We provided a draft of this report to DHS for review and comment. DHS provided written comments, which are reproduced in full in appendix V, and technical comments, which we incorporated as appropriate. DHS concurred with both recommendations in the report and described actions Border Patrol and AMO plan to take in response. Border Patrol plans to develop and apply a measure of operational control to its northern border sectors; however, to meet the intent of our recommendation, Border Patrol will also need to use its measure of operational control to assess its effectiveness at securing the northern border between ports of entry. AMO plans to develop a performance measure to assess its effectiveness at securing the northern border between ports of entry and seek DHS approval through completion of a Performance Measure Definition Form. These actions, if effectively implemented by AMO, should address the intent of the recommendation. We are sending copies of this report to the appropriate congressional committees and the Acting Secretary of the Department of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or gamblerr@gao.gov. Contacts points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. This report addresses the following questions: 1. What threats has U.S. Customs and Border Protection (CBP) identified along the U.S.-Canada (northern) border between ports of entry? 2. What challenges, if any, has CBP identified in its staffing and resources to secure the northern border between ports of entry, and what actions, if any, has CBP taken to address those challenges? 3. To what extent has CBP developed and implemented performance measures to assess the effectiveness of securing the northern border between ports of entry? To address all three questions, we interviewed Department of Homeland Security (DHS) and CBP officials from headquarters and field locations. Specifically, we met with headquarters officials from DHS’s Office of Strategy, Policy, and Plans; Office of Program Analysis and Evaluation; Science and Technology Directorate; U.S. Coast Guard; and U.S. Immigration and Customs Enforcement. From CBP, we met with headquarters officials from the Air and Marine Operations (AMO), U.S. Border Patrol (Border Patrol), Office of Information and Technology, Office of Intelligence, Office of Facilities and Asset Management, and Office of Accountability/Performance Management and Analysis Division. We also met with officials from the government of Canada to discuss their views on northern border security. For a list of government agencies and entities interviewed in field locations, see table 1. In addition, we conducted site visits in Michigan, New York, Vermont, Virginia, and Washington, as well as the Canadian provinces of British Columbia, Ontario, and Quebec. We chose these locations based on deployment of CBP resources—surveillance technology such as Remote Video Surveillance Systems—and reported levels of illicit cross-border activity by Border Patrol, including arrests of individuals and seizures of narcotics. Findings from our site visits cannot be generalized to all CBP locations along the northern border, but provide valuable insights into our research questions. To address the first question, we reviewed DHS and CBP policies, procedures, reports, and assessments describing threats along the northern border between ports of entry. Specifically, we reviewed DHS’s 2017 Northern Border Threat Analysis Report and the June 2018 Northern Border Strategy. We reviewed Border Patrol policies and procedures related to identifying and documenting threats and intelligence reports, referred to as Intelligence Estimates, completed in each northern border sector for fiscal years 2017 and 2018. In addition, we reviewed Border Patrol’s national intelligence estimates for fiscal years 2017 and 2018. We also reviewed documents describing the results of Border Patrol’s Threats, Targets, and Operations Assessments and Intelligence Preparation for the Operation Environment process completed for northern border sectors from 2014 through 2017. To analyze the number apprehensions and drug seizures along the northern border, we obtained data from the Enforcement Integrated Database for fiscal years 2013 through 2017, a time period for which complete data were available at the time of our review. We assessed the reliability of apprehension and seizure data by performing electronic testing for obvious errors in accuracy and completeness, reviewing existing information about the data and the systems that produced them, and interviewing agency officials knowledgeable about the data. As a result of our data reliability assessment, we determined that Border Patrol’s apprehension and seizure data were sufficiently reliable for our intended use. From AMO, we reviewed the 2017 Northern Border Non- Commercial General Aviation Threat Overview and information collected by the Air and Marine Operations Center on vessel and aircraft border incursions detected along the northern border from fiscal years 2013 through 2017. To address the second question, we reviewed CBP’s Fiscal Year 2019 Congressional Budget Justification. We also reviewed the results from Border Patrol’s capability gap assessment process for all eight northern border sectors completed for fiscal year 2017 and associated operational plans completed in September 2018; Border Patrol’s Surveillance Capability Assessment completed in April 2017; and AMO’s capability gap assessment completed in fiscal year 2016. We reviewed CBP capability analysis reports which included requirements along the northern border. In addition, we reviewed our relevant past work and DHS Office of Inspector General reports on northern border security. To determine the staffing and resource challenges across all eight northern border sectors and three AMO branches, we also met with officials at each sector and branch and reviewed supporting documentation. Specifically, we analyzed responses provided by officials in all eight northern border sectors and three AMO branches and supporting documentation to determine challenges mentioned by officials at two or more locations. We also reviewed supporting documentation, including inventories of assets such as vehicles, vessels, aircraft, radar and land surveillance technology, tactical communication equipment, and facilities information. We obtained Border Patrol, AMO, and Office of Information and Technology staffing information as of September 1, 2018, the most recent data available at the time of our review, including the number of authorized, onboard, and vacant positions. To assess the reliability of this staffing information, we examined the information for any anomalies and interviewed agency officials knowledgeable about the data. We found the staffing information data were sufficiently reliable for our purposes of reporting the number of authorized, onboard, and vacant positions. To address the third question, we reviewed and analyzed documentation that describes DHS and CBP processes for developing and implementing performance measures, including DHS’s Annual Performance Report for Fiscal Years 2017-2019, CBP’s Fiscal Year 2019 Congressional Budget Justification, and Performance Measure Definition Forms for recently developed performance measures. We reviewed reports, assessments, and strategies that describe current DHS and CBP performance measure initiatives. We also reviewed information from CBP’s National Border Geospatial Intelligence Strategy, including information on reports derived from geospatial intelligence technology, used as the basis for two of Border Patrol’s performance measures that contain data from the northern border. Additionally, we reviewed DHS’s most recent border security metrics report. We compared CBP’s actions to develop and implement performance measures to Standards for Internal Control in the Federal Government and the principles outlined in Government Performance and Results Act (GPRA) Modernization Act of 2010. We compiled the descriptive information in the northern Border Patrol sector profiles in appendix II from a variety of sources. We obtained information on each sector’s geography and area of responsibility from Border Patrol documentation. We obtained information on the number of authorized agents from Border Patrol as of September 1, 2018. We obtained information on the major urban areas within each sector and population estimates from the U.S. Census Bureau and the data are current as of July 1, 2017, the most recent estimates available at the time of our review. Finally, we obtained geographic information on the location of each northern Border Patrol sector and its stations from Border Patrol and located the data geographically using MapInfo. To analyze the number of apprehensions and drug seizures for each northern Border Patrol sector, we obtained data from the Enforcement Integrated Database for fiscal years 2013 through 2017, a time period for which complete data were available at the time of our review. The data fields we obtained included the individual’s immigration status at entry and country of citizenship and the drug type and quantity in pounds seized. Our analysis categorizes the sector’s apprehensions by the top four to six countries of citizenship of the individuals apprehended by Border Patrol and their immigration status at entry. Present without admission from Canada indicates the individual was suspected to be inadmissible for illegally entering the United States from Canada; present without admission from Mexico indicates the individual was suspected to be inadmissible for illegally entering the United States from Mexico; and the other category is a combination of all remaining categories, such as lawful permanent residents or other foreign nationals who may or may not be lawfully present in the United States. Our analysis also categorizes the sector’s number of drug seizures by the top three to six types of drugs that Border Patrol seized most frequently, as well as the quantity in pounds of those seizures. We assessed the reliability of apprehension and seizure data by performing electronic testing for obvious errors in accuracy and completeness, reviewing existing information about the data and the systems that produced them, and interviewing agency officials knowledgeable about the data. As a result of our data reliability assessment, we determined that Border Patrol’s apprehension and seizure data were sufficiently reliable for our intended use. We compiled the descriptive information in the northern region AMO branch profiles in appendix III from information provided by each branch and AMO headquarters. We obtained information on staffing for the three northern border branches as of September 2018. We obtained the geographic information on location of each northern region AMO branch and unit from AMO and located the data geographically using MapInfo. For total flight and float hours across all AMO operating locations and regions, we reviewed CBP data on flight and float hours from fiscal years 2013 through 2017, a time period for which complete data were available at the time of our review. For Border Patrol riverine float hours across all locations, we reviewed and analyzed float hour data from fiscal year 2017, the most recent year for which complete data were available at the time of our review. For data on air and marine missions across AMO’s northern region branches and units, we reviewed CBP data on seizures of narcotics, apprehensions, and arrests from fiscal years 2013 through 2017, a time period for which complete data were available at the time of our review. To determine the reliability of CBP’s data on flight and float hours, and mission information for seizures of narcotics, apprehensions, and arrests data, we examined the data for any anomalies, reviewed CBP guidance and documents for data collection and entry, and interviewed CBP officials to understand their methods for collecting, reporting, and validating the data. We found these data were sufficiently reliable for our purposes of reporting summary data across fiscal years 2013 through 2017. To obtain information on irregular northbound migration in appendix IV, we met with DHS and Border Patrol officials—including the three sectors (Blaine, Grand Forks, and Swanton sectors) with the highest reported levels of irregular northbound migration at the time of our review—and reviewed intelligence reports and assessments. We obtained the descriptive information in appendix IV on irregular northbound migration from a variety of sources. For data from the government of Canada on the number of asylum claimants, we downloaded publicly reported summary data on asylum claimants from the government of Canada for 2012 through 2017. For data on the number of individuals illegally entering Canada between ports of entry known to Border Patrol, we collected and reviewed information from Blaine, Grand Forks, and Swanton sectors for calendar years 2012 through 2017. To determine the reliability of data, we interviewed officials at each sector to understand their methods for collecting, reporting, and validating the data. According to Border Patrol officials at Blaine, Grand Forks, and Swanton sectors, the number of individuals illegally entering Canada between ports of entry was tracked through agent reporting and detection by land surveillance technology, such as surveillance cameras and unattended ground sensors. Based on Border Patrol’s methods for collecting, reporting, and validating the data, we found these data were sufficiently reliable for our purposes of reporting summary-level data. The performance audit upon which this report is based was conducted from October 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with DHS from March 2019 to June 2019 to prepare this nonsensitive version of the original sensitive report for public release. To provide a descriptive overview of the northern border sectors, we developed a profile for each of the eight U.S. Border Patrol (Border Patrol) sectors located along the U.S.-Canada (northern) border: Blaine, Washington; Spokane, Washington; Havre, Montana; Grand Forks, North Dakota; Detroit, Michigan; Buffalo, New York; Swanton, Vermont; and Houlton, Maine. These profiles are listed in order from the western-most sector to the eastern-most sector and contain an overview of each sector’s geography and area of responsibility and an analysis of apprehensions and drug seizures from fiscal years 2013 through 2017. Within U.S. Customs and Border Protection (CBP), Air and Marine Operations (AMO) conducts multifaceted missions consisting of direct support to U.S. Border Patrol (Border Patrol) and collaborative efforts with U.S. Immigration and Customs Enforcement’s Homeland Security Investigations and other federal, state, and local partner agencies. This includes, but is not limited to, investigative operations, surveillance missions, warrant service, and criminal apprehensions. AMO conducts missions along the U.S.-Canada (northern) border through three branches: Bellingham Air and Marine Branch in Bellingham, Washington; Great Lakes Air and Marine Branch at Selfridge Air National Guard Base, Michigan; and Manassas Air Branch in Manassas, Virginia. Each branch is further divided into units to conduct air or maritime missions. According to AMO data for fiscal years 2013 through 2017, AMO’s Northern Region accounted for 14 percent and 22 percent of total AMO flight and float hours, respectively, as shown in table 10. AMO implements a requirements determination process for annual aircraft flight and vessel float hours based on known mission requirements, funding levels, available assets, and the needs of law enforcement partners. Further, flight and float hours allocated across AMO’s regions are prioritized through CBP’s Flight and Float Hour Executive Oversight Council, which prioritizes flight and float hour allocations considering Department of Homeland Security and CBP’s strategic objectives and border security requirements, threats, and capacity that will be executed over the course of the upcoming year. In February 2018, CBP also created the Flight and Float Hour Executive Steering Committee comprised of Border Patrol and AMO executive leadership to perform periodic audits of flight hour execution, review changing operational environments, validate planning assumptions, and perform an evaluation on overall return on investment to best ensure that CBP asset utilization is consistently aligned with its priorities and threats. AMO’s Bellingham Air and Marine Branch is located in Bellingham, Washington, and is comprised of the Spokane and Montana Air Units and Port Angeles and Bellingham Marine Units. For a map of those operating locations, see figure 20. As of the end of September 2018, Bellingham Air and Marine Branch had 38 authorized Air Interdiction Agent positions and 20 authorized Marine Interdiction Agent positions. According to data provided by AMO for fiscal years 2013 through 2017, missions completed by Bellingham Air and Marine Branch resulted in: 51 apprehensions of potentially removable aliens; 963 arrests of individuals; and 536 drug seizures, including: 204 methamphetamine seizures (1,033 pounds); 93 cocaine seizures (778 pounds); 155 heroin seizures (305 pounds); 65 marijuana seizures (14,132 pounds); and 19 other drug seizures (608 pounds). AMO’s Great Lakes Air and Marine Branch is located at Selfridge Air National Guard Base, Michigan and is comprised of the Buffalo and Chicago Air Units and the Sault Sainte Marie, Port Huron, Trenton, Sandusky, Erie, Buffalo, and Rochester Marine Units. For a map of those operating locations, see figure 21. As of September 2018, Great Lakes Air and Marine Branch had 27 authorized Air Interdiction Agent positions and 49 authorized Marine Interdiction Agent positions. According to data provided by AMO for fiscal years 2013 through 2017, missions completed by Great Lakes Air and Marine Branch resulted in: 157 apprehensions of potentially removable aliens; 2,571 arrests of individuals; and 1,475 drug seizures, including: 553 marijuana seizures (6,974 pounds); 474 cocaine seizures (4,408 pounds); 296 heroin seizures (425 pounds); 87 methamphetamine seizures (1,347 pounds); and 65 other drug seizures (107 pounds). AMO’s Manassas Air Branch is located in Manassas, Virginia, and is comprised of the New York, Plattsburgh, and Houlton Air Units. For a map of those operating locations, see figure 22. As of September 2018, Manassas Branch had 35 authorized Air Interdiction Agent positions. According to data provided by AMO for fiscal years 2013 through 2017, missions completed by Manassas Air Branch resulted in: 57 apprehensions of potentially removable aliens; 1,347 arrests of individuals; and 472 drug seizures, including: 161 marijuana seizures (12,015 pounds); 141 heroin seizures (141 pounds); 134 cocaine seizures (707 pounds); 25 methamphetamine seizures (39 pounds); and 11 other drug seizures (107 pounds). Irregular northbound migration—northbound movement of foreign nationals from the United States across the northern border into Canada between official ports of entry typically to make an asylum claim— increased in 2017. Specifically, in 2017 the Royal Canadian Mounted Police reported approximately 20,000 irregular northbound migrants intercepted between official ports of entry. The majority of interceptions were reported in the province of Quebec (91 percent) with additional interceptions noted in Manitoba (5 percent) and British Columbia (3 percent). In comparison, from 2012 to 2016 the total number of asylum claimants for all of Canada (including at and between official ports of entry) ranged from approximately 10,000 to 24,000 per year. The total number of asylum claimants for all of Canada (including at and between official ports of entry) increased from approximately 24,000 claimants in 2016 to approximately 50,000 claimants in 2017. According to Border Patrol officials, in 2017 the number of individuals crossing from the United States into Canada, other than those crossing through official ports of entry, increased within 3 of 8 Border Patrol sectors along the northern border: Blaine, Washington; Grand Forks, North Dakota; and Swanton, Vermont. Blaine Border Patrol Sector. The number of individuals entering Canada between official ports of entry in British Columbia, north of Blaine sector’s area of responsibility, known to Border Patrol was approximately 1,200 individuals during the 4-year period from 2012 through 2015, according to sector officials. In 2016, the number of individuals known to Blaine sector increased to approximately 1,100 individuals, and then increased again to approximately 1,400 individuals in 2017. Grand Forks Border Patrol Sector. The number of individuals entering Canada between official ports of entry in Manitoba, north of Grand Forks sector’s area of responsibility, known to Border Patrol was approximately 580 individuals during the 4-year period from 2012 through 2015, according to sector officials. In 2016, the number of individuals known to Grand Forks sector increased to approximately 400 individuals, and then increased to approximately 1,000 individuals in 2017. Swanton Border Patrol Sector. The number of individuals entering Canada between official ports of entry in Quebec, north of Swanton sector’s area of responsibility, known to Border Patrol was approximately 1,000 individuals during the 4-year period from 2012 through 2015, according to sector officials. In 2016, the number of individuals known to Swanton sector increased to approximately 1,100 individuals, and then increased to approximately 16,800 individuals in 2017. According to Swanton Border Patrol Sector officials, the majority of known entries into Canada by irregular northbound migrants between official ports of entry have occurred along Roxham Road in Champlain, New York. For a photo of a facility constructed by the government of Canada to process irregular northbound migrants north of Roxham Road, see figure 23. Department of Homeland Security (DHS) and Border Patrol officials we met with identified a bi-national agreement associated with the increased number of irregular northbound migrants from the United States to Canada from 2016 through 2017. Irregular northbound migrants entering Canada between official ports of entry are not subject to the framework established by the 2002 Safe Third Country Agreement signed by Canada and the United States, which governs the processing of asylum claims along the shared land border and applies only to those individuals entering at an official port of entry, not between ports of entry. Therefore, individuals who enter Canada by land between official ports of entry to make an asylum claim may be allowed to stay in Canada rather than have their claim handled by the United States. Individuals seeking to travel to Canada to make an asylum claim, whether or not they may have a valid asylum claim, are made aware of the potential ability to enter and remain in Canada pending an asylum decision due to wide sharing of this information through social media and reporting in the press. Otherwise, for those attempting to enter Canada through an official land port of entry to claim asylum, claimants may be returned to pursue their asylum claim in the country of last presence, which would be the United States, unless they qualify for one of the exceptions in the agreement. According to DHS officials, Canadian data indicates a large percentage of irregular northbound migrants had previously obtained nonimmigrant visas, primarily B1/B2 visas, which authorized their temporary travel to the United States, and subsequently entered Canada between official ports of entry to claim asylum. DHS, in collaboration with the U.S. Department of State, worked to identify, and as appropriate, revoke visas of individuals seeking to enter Canada between official ports of entry. Border Patrol intelligence reporting in 2017 identified visa fraud concerns because individuals obtained visas to enter the United States, when it appeared that their main intention was to enter Canada other than through a port of entry and claim asylum. Border Patrol officials stated that the widespread perception among irregular northbound migrants they encounter is that Canada’s asylum policies are more welcoming than those of the United States, which has also contributed to the increased trend in irregular northbound migration. These officials cited both U.S. and Canadian reporting on the 2016 U.S. Presidential Election, along with a welcoming statement by the government of Canada, and perceived generosity of benefits upon application for asylum in Canada as reasons that migrants seek to enter Canada between official ports of entry and claim asylum. According to Border Patrol officials, the northbound asylum flows from the United States to Canada could potentially lead to future attempts to enter the United States illegally between ports of entry from Canada by individuals whose asylum claims are rejected by the government of Canada. According to anecdotal reporting to Border Patrol officials, some of the irregular northbound migrants who entered Canada from the United States were unable to gain status in Canada or the process was not what they had anticipated. According to the officials, these individuals subsequently attempted to reenter the United States in an effort to gain legal status in the United States. For example, Swanton Border Patrol Sector reported two incidents in April 2018 in which groups of individuals who were apprehended attempting to illegally enter the United States from Canada stated that they were seeking to reenter the United States after their asylum claims were rejected by the government of Canada. In addition to the contact named above, Christopher Ferencik (Assistant Director), David Alexander, Michele Fejfar, Eric Hauswirth, Grant Holyoak, John Mingus, Sasan J. “Jon” Najmi, Claire Peachey, Carl Potenzieri, and Natalie Swabb made key contributions to this report.", "summary": "The United States and Canada share the longest common non-militarized border between two countries, spanning nearly 4,000 miles of land and maritime borders from the states of Washington to Maine. CBP, within DHS, has primary responsibility for securing U.S. borders at and between ports of entry. GAO was asked to review CBP's efforts to secure the northern border between ports of entry. This report examines, among other things, (1) the staffing and resource challenges that CBP identified and actions it has taken to address those challenges and (2) the extent to which CBP has developed and implemented performance measures to assess its effectiveness at securing the northern border between ports of entry. GAO reviewed agency documentation and met with DHS and CBP officials in headquarters and field locations. This is a public version of a sensitive report that GAO issued in March 2019. Information that DHS deemed sensitive has been omitted. U.S. Customs and Border Protection (CBP) identified staffing and resource challenges affecting its enforcement activities along the U.S.-Canada (northern) border and actions to address them, but faces competing priorities. The U.S. Border Patrol (Border Patrol) and Air and Marine Operations (AMO) are the components within CBP responsible for securing U.S. borders between ports of entry in the land, air, and maritime environments. Border Patrol identified an insufficient number of agents that limited patrol missions along the northern border. AMO identified an insufficient number of agents along the northern border, which limited the number and frequency of air and maritime missions. Border Patrol and AMO also identified a variety of resource challenges along the northern border, such as limited radar and surveillance technology coverage and inadequate facilities to process and temporarily hold apprehended individuals. While the Department of Homeland Security (DHS) and CBP identified actions to address staffing and resource challenges, it is unknown whether these challenges will be addressed. This is primarily because CBP's priority is to secure the U.S.-Mexico (southwest) border. Issued in January 2017, Executive Order 13767 directed DHS to take actions to secure the southwest border by, among other things, constructing physical barriers and hiring thousands of agents. While CBP has performance measures that assess selected border security operations or programs, some of which include data from the northern border, it does not have specific measures to assess its effectiveness at securing the northern border between ports of entry. For example, Border Patrol has performance measures that assess security in remote areas on the northern border, but the measures do not include data from maritime border areas. Developing and implementing such measures could help Border Patrol and AMO better assess the effectiveness of their northern border operations between ports of entry, including addressing challenges due to limited staffing and resources. GAO is making two recommendations, that Border Patrol and AMO each develop and implement performance measures to assess their effectiveness at securing the northern border between ports of entry. DHS concurred with both recommendations.", "document_type": "gao"}
{"report": "FEMA’s Public Assistance program provides grant funding to state, territorial, local, and tribal governments as well as certain types of private nonprofit organizations to assist with responding to and recovering from presidentially-declared major disasters or emergencies. As shown in figure 3, Public Assistance grant funds are categorized broadly as “emergency work” or “permanent work.” Within these broad categories are separate subcategories. In addition to the emergency work and permanent work categories, the program includes category Z, which represents indirect costs, direct administrative costs, and any other administrative expenses associated with a specific project. FEMA’s Public Assistance program also provides grant funding for cost- effective hazard mitigation measures to reduce or eliminate the long-term risk to people and property from future natural and man-made disasters and their effects. For example, a community that had a fire station damaged by a disaster could use Public Assistance grant funding to repair the facility and incorporate additional measures such as installing hurricane shutters over the windows to mitigate the potential for future damage. FEMA, the state or territorial government (the recipient), and local or territorial entities (the subrecipient) work together to develop projects under the Public Assistance program. After a project has completed FEMA’s review process and is approved, FEMA obligates funding for the project by placing money into an account where the recipient has the authority to draw down—or withdraw—funding to pay the subrecipient for eligible work upon completion. The Sandy Recovery Improvement Act of 2013 authorized the use of alternative procedures in administering the Public Assistance program, thereby providing new flexibilities to FEMA, states, territories, and local governments for debris removal, infrastructure repair, and rebuilding projects using funds from this program. Unlike in the standard Public Assistance program where FEMA will fund the actual cost of a project, the Public Assistance alternative procedures allow awards for permanent work projects to be made on the basis of fixed-cost estimates to provide financial incentives for the timely and cost-effective completion of work. Under these procedures, if the actual cost of the project exceeds the fixed-cost estimate agreed upon by FEMA and the recipient, the recipient or subrecipient is responsible for the additional costs at the time of the close-out process. However, if the actual cost of completing eligible work for a project is below the estimate, the recipient may use the remaining funds for additional cost-effective hazard mitigation measures to increase the resilience of public infrastructure. In addition, these funds may also be used for activities that improve the recipient’s or subrecipient’s future Public Assistance operations or planning. In October 2017, Puerto Rico requested, and FEMA approved, the use of the alternative procedures program for all large-project funding for Public Assistance permanent work projects in categories C through G. Although FEMA had approved alternative procedure grants in 30 states as of April 2018, in these cases, alternative procedures were used on a project-by-project basis. Puerto Rico’s recovery from the 2017 hurricanes is the first recovery to use alternative procedures for all large permanent work projects. In addition, in July 2018, FEMA approved a request from the Governor of the USVI to transition to using the Public Assistance alternative procedures program for permanent work in the territory. Unlike in Puerto Rico, the USVI may pursue the alternative procedures on a project-by-project basis. As of April 2019, FEMA had obligated a total of about $7.4 billion in grant funds for Public Assistance projects in both Puerto Rico and the USVI. Specifically, as shown in figure 4, FEMA obligated approximately $5.6 billion for 1,264 Public Assistance projects in Puerto Rico, including approximately $5.1 billion (90 percent) for emergency work (categories A and B) and $377.7 million (7 percent) for permanent work in categories C through G). Puerto Rico had expended approximately $3.5 billion—about 61 percent of total Public Assistance grant obligations in Puerto Rico—as of April 2019. Ninety-six percent of the expended amount went toward emergency work projects in categories A and B while just over one percent went toward permanent work projects. The majority of FEMA’s obligations and the funding Puerto Rico expended as of April 2019 are for emergency work because these projects began soon after the disasters struck and focused on debris removal and providing assistance to address immediate threats to life and property. In contrast, permanent work projects take time to identify, develop, and ultimately complete as they represent the longer-term repair and restoration of public infrastructure. In the USVI, FEMA had obligated approximately $1.8 billion for 583 Public Assistance projects across the territory, as of April 2019. Similar to Public Assistance grant funding in Puerto Rico, the majority of funding FEMA obligated and the USVI expended was in emergency work categories A and B. Specifically, FEMA obligated approximately $1.1 billion (63 percent) for emergency work (categories A and B) and $587.3 million (33 percent) for permanent work (categories C through G) in the territory (see fig. 5). Of the $1.8 billion FEMA obligated for Public Assistance projects, the USVI had expended approximately $982.4 million as of April 2019. Specifically, the USVI had expended about $808.1 million (82 percent) for emergency work projects in categories A and B and $163.1 million (17 percent) for permanent work projects in categories C through G. Emergency work. As of April 2019, FEMA had obligated a total of approximately $6.2 billion for emergency work projects in Puerto Rico and the USVI—including about $5.1 billion in Puerto Rico and $1.1 billion in the USVI. These projects focused on debris removal activities and providing assistance to address immediate threats to life and property. For example, as of April 2019, FEMA had obligated $138.9 million for projects focused on debris removal activities in the USVI under category A. This included $45.9 million to the USVI Department of Public Works for USVI-wide debris removal efforts and $39.1 million to the USVI Water and Power Authority for these activities in St. Croix (see fig. 6). In another example, FEMA obligated more than $140.0 million to the Puerto Rico Aqueducts and Sewer Authority under category B to fund emergency protective measures, including using back-up generators to supply water to the island after Hurricane Maria, among other things. Further, as of April 2019, FEMA had obligated $1.1 billion in Puerto Rico and $278 million in the USVI to fund the Sheltering and Temporary Essential Power pilot program. This program, which is implemented as a subprogram under Public Assistance program category B, is intended to provide essential repairs or restore power to private residences to allow affected individuals to return or remain in their homes, thereby reducing the demand for other shelter options. We are continuing to assess this program as part of our ongoing work on recovery efforts in the USVI. Permanent work. As of April 2019, FEMA had obligated approximately $965.0 million for permanent work projects in Puerto Rico and the USVI— including about $377.7 million in Puerto Rico and $587.3 million in the USVI. These projects focused on the restoration of disaster-damaged infrastructure or systems. For example, under category C, FEMA obligated $137.6 million for projects in Puerto Rico focused on the permanent repair of roads and bridges, such as the severely damaged road shown in figure 7 below. In addition, under category E, FEMA obligated $39.2 million and $67.7 million for projects in Puerto Rico and the USVI, respectively, focused on repairing and rebuilding damaged public buildings and equipment, such as the schools shown in figure 8 below. Further, under category F, FEMA obligated $504.9 million for projects in the USVI to repair damaged utilities. Specifically, FEMA obligated $481.8 million—or 95 percent of this total—through the standard Public Assistance program for projects focused on territory-wide permanent electrical distribution system repairs. This includes replacing damaged wooden utility poles with more resilient composite fiberglass poles that can withstand 200 mile per hour winds as well as power transmission lines and transformers (see fig. 9). As the recipients of federal disaster funding, Puerto Rico and the USVI are responsible for monitoring and overseeing the Public Assistance program to ensure it is implemented in compliance with applicable laws, regulations, and requirements as well as FEMA policies and guidance. To address these responsibilities, Puerto Rico and the USVI established recovery offices to manage recovery activities and funding, including through the Public Assistance program. In March 2019, we reported that Puerto Rico, in accordance with Amendment 5 to the President’s disaster declaration, established the Central Office for Recovery, Reconstruction, and Resilience (COR3) to oversee federal recovery funds. We also reported that COR3 was developing an internal controls plan to help ensure better management and accountability of the funds. According to FEMA officials, FEMA instituted a manual reimbursement process due to Puerto Rico’s financial situation, weaknesses in internal controls, and the large amount of recovery funds, among other things, to mitigate risk and help ensure financial accountability. However, from our ongoing work on Puerto Rico’s disaster recovery efforts, we have learned that, on April 1, 2019, FEMA removed the manual reimbursement process and began a transition to allow the central recovery office to take responsibility for the review and reimbursement approval of federal recovery funds. We have also learned from our ongoing work that, in March 2019, COR3 released the Disaster Recovery Federal Funds Management Guide. Among other things, the guide outlines COR3’s roles and responsibilities and the internal controls COR3 put in place to oversee the recovery. For example, COR3 will identify, procure, and administer all federal, territorial, and private resources available to Puerto Rico related to recovery. In addition, it will provide oversight of subrecipients using risk-based monitoring, offer technical assistance, and advise Puerto Rico’s governmental agencies and municipalities regarding any matter related to recovery. COR3 continues to update its online transparency portal intended to provide a breakdown of FEMA Public Assistance and other federal funding obligated for disaster recovery in Puerto Rico. According to our preliminary observations, in February 2019, the USVI established the new Office of Disaster Recovery. This office serves as the primary territorial agency responsible for overseeing all disaster recovery efforts and funding in the territory, and coordinates across all USVI governmental agencies and other pertinent entities. According to USVI officials, following the 2017 hurricanes, key USVI agencies did not have enough employees with the knowledge and expertise necessary to staff recovery-related positions and effectively manage the implementation of recovery efforts. To address this challenge in the short-term, the USVI government hired two contractors in December 2017—Witt O’Brien’s, LLC and Ernst & Young Puerto Rico, LLC—to assist the territory in planning, developing, implementing, and overseeing Public Assistance program projects, among other responsibilities. The Director of the Office of Disaster Recovery told us that while contractor personnel had been valuable in augmenting the USVI’s management capacity in the short term, the territory’s longer-term vision included the establishment of the Office of Disaster Recovery to centrally manage all aspects of federal recovery in the territory. Among other things, the Office of Disaster Recovery is responsible for taking on the USVI’s monitoring and oversight responsibilities for the Public Assistance program in the long term. This includes tracking and reporting on the progress of projects and overseeing reimbursement requests for completed work to ensure compliance with applicable laws and FEMA policies. As of March 2019, the Director of the Office of Disaster Recovery told us the priority is to quickly hire and train qualified individuals to staff the new agency. FEMA officials in the USVI stated that the establishment of the Office of Disaster Recovery and the USVI’s ongoing efforts to hire local residents into recovery-related positions represented a positive step forward in increasing the territory’s capacity to oversee recovery efforts. We will continue to review the monitoring and oversight of recovery efforts in Puerto Rico and the USVI in our ongoing work. Our prior and ongoing work highlight the challenges with implementing the Public Assistance program—and the alternative procedures—in Puerto Rico and the USVI. In particular, our prior and ongoing work have identified challenges related to (1) the clarity of FEMA’s guidance for the Public Assistance program, (2) the time and resources needed to transition to FEMA’s new Public Assistance program delivery model in Puerto Rico, (3) the implementation of flexibilities provided by the Bipartisan Budget Act of 2018, and (4) developing fixed-cost estimates. FEMA has taken some actions, including issuing additional guidance and developing specific training, among other things, to improve Public Assistance implementation in Puerto Rico and the USVI. However, it is too soon to assess their effectiveness in addressing these issues. Clarity of Guidance. In March 2019, we reported that officials from FEMA, COR3, and municipalities said they experienced initial challenges with the recovery process, including concerns about lack of experience and knowledge of the alternative procedures; and concerns about missing, incomplete, or conflicting guidance from FEMA on the alternative procedures. In addition, in our June 2019 testimony statement we continued to report on these challenges and preliminary observations from our ongoing work indicate that these challenges continue. For example, officials from Puerto Rico’s government agencies told us they did not feel they had sufficient guidance on the FEMA Public Assistance program and where they did, written and verbal FEMA guidance was inconsistent or conflicting. For instance, officials from one agency expressed their desire for more FEMA guidance communicated in writing as FEMA officials would frequently interpret existing guidance differently. Similarly, officials from two agencies described situations where they had initially been directed to follow one interpretation of a policy, only to be directed to follow a different, conflicting interpretation in the subsequent months. Puerto Rico agency officials also stated that the lack of sufficient instruction led to a “back and forth” with FEMA for clarifications, which led to delays in the phases of project development. For example, officials from one Puerto Rico government agency stated that conflicting verbal instructions from several FEMA officials contributed to delays in opening the bidding process for recovery-related contracts. FEMA officials in Puerto Rico stated that the agency has developed specific guidance for disaster recovery in Puerto Rico and that there are various ways, such as in-person meetings, where officials from Puerto Rico can obtain clarification. FEMA officials also reported that they developed additional training for new FEMA employees. We are continuing to examine this issue as part of our ongoing review of Puerto Rico’s recovery. FEMA’s new delivery model in Puerto Rico. In May 2019, FEMA’s Federal Disaster Recovery Coordinator for Puerto Rico announced that FEMA was transitioning to using the new Public Assistance program delivery model in Puerto Rico beginning on June 3, 2019. Among other things, the implementation of the new delivery model establishes a new Consolidated Resource Center in Puerto Rico to support grant development for disaster recovery across all recovery sectors and geographic branches. Following the hurricanes, FEMA implemented a program delivery model developed specifically for Puerto Rico which included, among other things, a sector-based approach which coordinated recovery resources across the federal interagency, private sector, and nongovernmental organizations to identify and complete proposed work. According to FEMA officials, the decision to transition from the initial delivery model to the new delivery model in Puerto Rico was due to improvements made since its nationwide deployment in 2017. In response, COR3 officials raised concerns about the scope of the changes and potential challenges with the amount of time and resources needed to transition to the new delivery model. The Bipartisan Budget Act of 2018. We reported in June 2019 that in both Puerto Rico and the USVI, FEMA and local officials have reported challenges with the implementation of the flexibilities authorized by section 20601 of the Bipartisan Budget Act. This section of the Act allows for the provision of assistance under the Public Assistance alternative procedures to restore disaster-damaged facilities or systems that provide critical services—such as medical and educational facilities— to an industry standard without regard to pre-disaster condition. Officials from Puerto Rico’s central government stated that they disagreed with FEMA’s interpretation of the types of damages covered by section 20601 of the Bipartisan Budget Act of 2018. In response, FEMA officials in Puerto Rico stated they held several briefings with Puerto Rico’s central recovery office to explain FEMA’s interpretation of the section. In addition, FEMA officials in the USVI told us that initially, they had difficulty obtaining clarification from FEMA headquarters regarding how to implement key components of section 20601 of the Act. Further, USVI officials stated that at times, the appropriate process for implementing components of the Act was not clear and that ensuring program participants understood its key components was difficult. However, FEMA officials in the USVI stated that they continue to move forward with developing alternative procedures projects. USVI officials also told us that FEMA had been responsive and helpful in identifying its options for using the new flexibilities the Act provides. Developing Fixed-Cost Estimates. Preliminary observations from our ongoing work indicate that as of May 2019, FEMA had obligated funding for four alternative procedures program projects in Puerto Rico and two projects in the USVI. FEMA officials in Puerto Rico and the USVI stated that the ongoing development of a “cost factor” for use in the fixed-cost estimating process has slowed the pace of FEMA obligations for permanent work projects. Specifically, these factors are intended to ensure that the costs associated with implementing projects in Puerto Rico and the USVI are sufficiently captured when developing the fixed- cost estimates for alternative procedures projects. Since incorporating the cost factor into the fixed-cost estimating process will increase the amount of funding obligated for any given permanent work project, FEMA officials explained that Puerto Rico and the USVI have an incentive to delay the obligation of individual projects until this factor is finalized. For example, FEMA officials in the USVI told us in May 2019 that obligations for permanent work projects in the territory were mostly on hold until the USVI-specific cost factor was finalized. As of June 2019, the cost factors for use in both Puerto Rico and the USVI had not yet been finalized. According to FEMA guidance, the Puerto Rico-specific cost factor is being developed by a third-party center of excellence comprising personnel selected by FEMA and Puerto Rico, through COR3. In March 2019, we reported that while FEMA had identified and chosen personnel, COR3 had not yet finalized its hiring of personnel to staff the center of excellence, which resulted in delaying the cost estimation process. Through our ongoing work we learned that, as of June 2019, COR3 had identified and hired personnel to staff the center; however, FEMA and COR3 have not come to agreement on a cost estimation approach. Further, according to FEMA officials, no timeline has been established for the completion of the center of excellence’s standard operating procedures for developing fixed-cost estimates for permanent work projects in Puerto Rico. In addition, according to FEMA officials, the USVI-specific factor is being developed by an independent contractor. FEMA officials told us that territorial officials disagreed with the initial cost factors this contractor proposed and contended the factors were insufficient in accurately capturing the unique circumstances that influence construction costs in the territory, such as the limited availability of local resources and the need to import materials and labor. As of June 2019, these officials told us the contractor was developing a third and final cost factor for potential incorporation into the fixed-cost estimation process in the USVI. Despite these delays, FEMA officials in the USVI stated that they continue to work with territorial officials to develop alternative procedures projects in the territory. They added that once the cost factor is finalized and incorporated into FEMA’s fixed-cost estimating process, FEMA and the USVI will be well positioned to quickly finalize these projects and obligate funding. However, we reported in June 2019 that the territory plans to take a cautious approach in pursuing permanent work projects using the Public Assistance alternative procedures program. Specifically, USVI officials we interviewed told us that developing fixed-cost estimates for alternative procedures projects that accurately incorporate the future impact of inflation and increases in materials and labor costs for certain projects was difficult. Further, these officials stated that since the territory is financially responsible for any costs that exceed these fixed-cost estimates, the USVI plans to pursue alternative procedures projects that do not include high levels of complexity or uncertainty to reduce the risk of cost overruns, especially given its already difficult financial situation. As established in FEMA guidance, Puerto Rico’s deadline for finalizing fixed-cost estimates for permanent work projects using the alternative procedures—and the Bipartisan Budget Act, as applicable—is October 2019. Since Puerto Rico must use the alternative procedures for all large permanent work, all fixed-cost estimates for Public Assistance program permanent work projects in Puerto Rico must be finalized by this date, or, according to FEMA officials, Puerto Rico must request that FEMA extend this deadline on a project-by-project basis. In contrast, the USVI has the flexibility to pursue either the alternative procedures or the standard procedures on a project-by-project basis. As the USVI’s deadline for finalizing these projects is in March 2020, it is too early gauge the extent to which the alternative procedures will play a role in the USVI’s long-term recovery strategy. We will continue to evaluate these identified challenges and any efforts to address them, as well as other aspects of recovery efforts in the USVI and Puerto Rico, and plan to report our findings in late 2019 and early 2020, respectively. Thank you, Chairman Payne, Ranking Member King, and Members of the Subcommittee. This concludes my prepared statement. I would be happy to respond to any question you may have at this time. If you or your staff has any questions concerning this testimony, please contact Christopher P. Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement were Joel Aldape (Assistant Director), Bryan Bourgault, Aaron Gluck, Taylor Hadfield, Brian Lipman, and Amanda Prichard. In addition, Eric Hauswirth, Susan Hsu, Tracey King, Taylor Matheson, Amanda Miller, Heidi Nielson, and Kevin Reeves made contributions to this statement. Key contributors for the previous work on which this statement is based are listed in each product. 2017 Disaster Relief Oversight: Strategy Needed to Ensure Agencies’ Internal Control Plans Provide Sufficient Information. GAO-19-479 (Washington, D.C.: June 28, 2019). Emergency Management: FEMA Has Made Progress, but Challenges and Future Risks Highlight Imperative for Further Improvements, GAO-19-617T (Washington, D.C.: June 25, 2019). Emergency Management: FEMA Has Made Progress, but Challenges and Future Risks Highlight the Imperative for Further Improvements, GAO-19-594T (Washington, D.C.: June 12, 2019). Disaster Assistance: FEMA Action Needed to Better Support Individuals Who Are Older or Have Disabilities. GAO-19-318 (Washington, D.C.: May 14, 2019). Disaster Contracting: Actions Needed to Improve the Use of Post- Disaster Contracts to Support Response and Recovery. GAO-19-281 (Washington, D.C.: April 24, 2019). 2017 Hurricane Season: Federal Support for Electricity Grid Restoration in the U.S. Virgin Islands and Puerto Rico. GAO-19-296 (Washington, D.C.: April 18, 2019). FEMA Grants Modernization: Improvements Needed to Strengthen Program Management and Cybersecurity. GAO-19-164 (Washington, D.C.: April 9, 2019). Disaster Recovery: Better Monitoring of Block Grant Funds Is Needed. GAO-19-232 (Washington, D.C.: March 25, 2019). Puerto Rico Hurricanes: Status of FEMA Funding, Oversight, and Recovery Challenges. GAO-19-256 (Washington, D.C.: March 14, 2019). Huracanes de Puerto Rico: Estado de Financiamiento de FEMA, Supervisión y Desafíos de Recuperación. GAO-19-331 (Washington, D.C.: March 14, 2019). High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas. GAO-19-157SP (Washington, D.C.: March 6, 2019). U.S. Virgin Islands Recovery: Status of FEMA Public Assistance Funding and Implementation. GAO-19-253 (Washington, D.C.: February 25, 2019). 2017 Disaster Contracting: Action Needed to Better Ensure More Effective Use and Management of Advance Contracts. GAO-19-93 (Washington, D.C.: December 6, 2018). Continuity of Operations: Actions Needed to Strengthen FEMA’s Oversight and Coordination of Executive Branch Readiness. GAO-19-18SU (Washington, D.C.: November 26, 2018). Homeland Security Grant Program: Additional Actions Could Further Enhance FEMA’s Risk-Based Grant Assessment Model. GAO-18-354 (Washington, D.C.: September 6, 2018). 2017 Hurricanes and Wildfires: Initial Observations on the Federal Response and Key Recovery Challenges. GAO-18-472 (Washington, D.C.: September 4, 2018). Federal Disaster Assistance: Individual Assistance Requests Often Granted but FEMA Could Better Document Factors Considered. GAO-18-366 (Washington, D.C.: May 31, 2018). 2017 Disaster Contracting: Observations on Federal Contracting for Response and Recovery Efforts. GAO-18-335 (Washington, D.C.: February 28, 2018). Disaster Recovery: Additional Actions Would Improve Data Quality and Timeliness of FEMA’s Public Assistance Appeals Processing. GAO-18-143 (Washington, D.C.: December 15, 2017). Disaster Assistance: Opportunities to Enhance Implementation of the Redesigned Public Assistance Grant Program. GAO-18-30 (Washington, D.C.: November 8, 2017). Climate Change: Information on Potential Economic Effects Could Help Guide Federal Efforts to Reduce Fiscal Exposure. GAO-17-720 (Washington, D.C.: September 28, 2017). Federal Disaster Assistance: Federal Departments and Agencies Obligated at Least $277.6 Billion during Fiscal Years 2005 through 2014. GAO-16-797 (Washington, D.C.: September 22, 2016). Disaster Recovery: FEMA Needs to Assess Its Effectiveness in Implementing the National Disaster Recovery Framework. GAO-16-476 (Washington, D.C.: May 26, 2016). Disaster Response: FEMA Has Made Progress Implementing Key Programs, but Opportunities for Improvement Exist. GAO-16-87 (Washington, D.C.: February 5, 2016). Hurricane Sandy: An Investment Strategy Could Help the Federal Government Enhance National Resilience for Future Disasters. GAO-15-515 (Washington, D.C.: July 30, 2015). Budgeting for Disasters: Approaches to Budgeting for Disasters in Selected States. GAO-15-424 (Washington, D.C.: March 26, 2015). High-Risk Series: An Update. GAO-15-290 (Washington, D.C.: February 11, 2015). Emergency Preparedness: Opportunities Exist to Strengthen Interagency Assessments and Accountability for Closing Capability Gaps. GAO-15-20 (Washington, D.C.: December 4, 2014). Fiscal Exposures: Improving Cost Recognition in the Federal Budget. GAO-14-28 (Washington, D.C.: October 29, 2013). Federal Disaster Assistance: Improved Criteria Needed to Assess a Jurisdiction’s Capability to Respond and Recover on Its Own. GAO-12-838 (Washington, D.C.: September 12, 2012). Opportunities to Reduce Potential Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue. GAO-11-318SP (Washington, D.C.: March 1, 2011). 1. Review of U.S. Virgin Islands recovery planning and progress; 2. Puerto Rico disaster recovery planning and progress; 3. 2017 wildfire response and recovery; 4. Puerto Rico electricity grid recovery after the 2017 hurricane season; 5. Mass care sheltering and feeding challenges during the 2017 6. Department of Transportation highway and transit emergency relief 7. Drinking water and wastewater utility resilience; 8. Review of disaster death count information in selected states and 9. Department of Health and Human Services disaster response efforts; 10. Disaster and climate change impacts on Superfund sites; 11. FEMA Public Assistance program fraud risk management efforts; 12. Wildland fire fuel reduction efforts; 13. Preparedness challenges and lessons learned from the 2017 14. FEMA workforce management and challenges; 15. Small Business Administration response to 2017 disasters; 16. Development of the GAO disaster resilience framework; 17. FEMA Individuals and Households Program operations and 18. National Flood Insurance Program post-flood enforcement; 19. Emergency alerting capabilities and progress; 20. National Flood Insurance Program buyouts and property acquisitions; 21. Economic costs of large-scale natural disasters and impacts on 22. Community Development Block Grants – disaster recovery; and 23. Disaster Housing Assistance Program. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In September 2017, two major hurricanes—Irma and Maria—struck Puerto Rico and the USVI, causing billions of dollars in damage to infrastructure, housing, and the economy. FEMA—a component of the Department of Homeland Security—is the lead federal agency responsible for assisting Puerto Rico and the USVI to recover from these natural disasters. Among other responsibilities, FEMA is administering the Public Assistance program in partnership with the governments of Puerto Rico and the USVI, providing them grant funding for response and recovery activities, including debris removal efforts, life-saving emergency protective measures, and the repair, replacement, or restoration of public infrastructure. This statement describes (1) the status of FEMA's Public Assistance grant funding in Puerto Rico and the USVI in response to the 2017 hurricanes as of April 2019, (2) the establishment of recovery offices in Puerto Rico and the USVI, and (3) challenges in implementing the Public Assistance program and actions FEMA has taken to address them. This statement is based on GAO reports issued in February, March, and June 2019, and includes preliminary observations from ongoing GAO reviews of FEMA operations. For ongoing work, GAO analyzed program documents and data on obligations and expenditures; interviewed agency officials; and visited disaster-damaged areas in Puerto Rico and the USVI, where GAO also interviewed FEMA and local officials. GAO's prior and ongoing work found that the Federal Emergency Management Agency (FEMA) obligated about $7.4 billion in Public Assistance grant funding to Puerto Rico and the U.S. Virgin Islands (USVI) as of April 2019, in response to the 2017 hurricanes. FEMA obligated about $6.2 billion in Public Assistance grants for emergency work—debris removal activities, power restoration, and other emergency measures—and about $965 million in Public Assistance grants for permanent work—including the repair or replacement of public infrastructure such as roads, electrical utilities, and damaged buildings. Further, FEMA is continuing to work with Puerto Rico and the USVI to develop additional permanent work projects to repair damaged public infrastructure, such as schools and hospitals (see figure). In 2017, Puerto Rico established the Central Office for Recovery, Reconstruction, and Resilience and in 2019 the USVI established the Office of Disaster Recovery to coordinate and oversee federal recovery efforts. Among other things, these recovery offices are responsible for monitoring and overseeing the Public Assistance program and developing internal controls to ensure it is implemented in accordance with applicable laws, regulations, and FEMA requirements. GAO's prior and ongoing work highlighted challenges with the Public Assistance program including concerns about the clarity of FEMA's guidance, and the time and resources needed to transition to a new Public Assistance delivery model in Puerto Rico. Further, Puerto Rico and USVI officials reported difficulties understanding FEMA's implementation of new flexibilities authorized by law as well as delays in jointly developing cost estimates for long-term recovery projects such as the repair or replacement of hospitals, buildings, and other public infrastructure. FEMA has taken some actions to help address these issues, including developing additional guidance and specific training. However, it is too soon to determine the effectiveness of FEMA's actions. GAO will continue to evaluate the Public Assistance program in the USVI and Puerto Rico and plans to report its findings in late 2019 and early 2020, respectively.", "document_type": "gao"}
{"report": "At the federal level, CMS, within the Department of Health and Human Services, is responsible for overseeing the design and operation of states’ Medicaid programs, and states administer their respective Medicaid programs’ day-to-day operations. As a comprehensive health benefit program for vulnerable populations, each state Medicaid program, by law, must cover certain categories of individuals and provide a broad array of benefits. Within these requirements, however, states have significant flexibility to design and implement their programs, resulting in more than 50 distinct state-based programs. These variations in design have implications for program eligibility and services offered, as well as for how expenditures are reported and services delivered. In administering their own programs, states may provide Medicaid services under a fee-for-service delivery model or a managed care service delivery model. Under a fee-for-service model, states make payments directly to providers for services provided, and the federal government generally matches state expenditures for such services on the basis of a statutory formula. Under a managed care model, states pay MCOs a capitation payment, which is a fixed periodic payment per beneficiary enrolled in an MCO—typically, per member per month. MCOs pay health care providers for the services delivered to enrollees. In contrast, ACOs are organizations of health care providers and suppliers that come together voluntarily to provide coordinated care to patients with the goal of reducing spending while improving quality. States vary in terms of the types of managed care arrangements used, the populations enrolled, and the parts of the state covered by managed care. We previously reported that a small share of beneficiaries in each state collectively accounted for a disproportionately large share of total Medicaid expenditures. We found that in fiscal years 2009 through 2011, the most expensive 5 percent of Medicaid beneficiaries consistently accounted for almost half of the expenditures for all Medicaid beneficiaries. (See fig. 1.) Examining beneficiaries who were enrolled only in Medicaid, we also found that the most expensive 5 percent of beneficiaries were much more likely to have certain conditions—such as asthma, diabetes, and behavioral health conditions—than all other beneficiaries enrolled only in Medicaid. Examining 2009 data, we found that about 65 percent of the total expenditures for high-expenditure beneficiaries enrolled only in Medicaid were for hospital services and long-term services and supports, with the remaining 35 percent of expenditures for drugs, payments to managed care organizations and premium assistance, and non-hospital acute care. Other studies have also found similar patterns of service utilization and expenditures within the Medicaid population. For example, a January 2018 report noted that while beneficiaries who are dually eligible for Medicare and Medicaid constituted about 15 percent of Medicaid beneficiaries in 2013, they accounted for nearly one-third of Medicaid spending. A study examining data on children’s use of behavioral health services in Medicaid found that in 2005, about 10 percent of children in Medicaid received behavioral health services, but those services accounted for about 38 percent of spending on the overall Medicaid child population. Care management programs can be used as efforts to manage the cost and quality of health care services delivered to high-expenditure Medicaid populations, with the aim of improving outcomes and achieving cost savings. Generally, care management programs seek to assist consumers manage physical and mental health conditions more effectively, for example, by assessing patient needs and coordinating care across different providers. The general goal of care management is to achieve an optimal level of wellness and improve coordination of care while providing cost effective, non-duplicative services. Specific definitions for care management and other related terms such as care coordination, case management, and disease management vary. For the purpose of this report, we use care management to refer to these activities unless otherwise specified. Officials from most state agencies, MCOs, and the ACO said they used risk scores to identify or predict high-expenditure beneficiaries. Officials from four of the seven selected states, four MCOs, and the ACO said they used software or hired vendors who computed beneficiaries’ risk scores based on Medicaid service utilization data. Washington state officials said that in addition to Medicaid service utilization data, they used utilization data from Medicare Parts A, B, and D to compute risk scores for their dual-eligible population. Officials also discussed using the risk scores they computed in different ways. For example, Washington officials said they considered beneficiaries with a risk score of 1.5 or greater to be high expenditure, and they used that risk score as one of the eligibility criteria that must be met to receive certain care management services. In contrast, officials from an MCO in Nevada said they considered risk scores alongside other contextual information, such as the recent diagnosis of a chronic condition, to predict whether the beneficiary would likely generate high expenditures in the future and should be assigned care management services. Officials from three states, an MCO in South Carolina, and the ACO we interviewed said their software or vendors identified or predicted high-expenditure beneficiaries by using the risk scores they computed to stratify beneficiaries into risk tiers, such as low, medium, and high risk. Officials from South Carolina’s state Medicaid agency and two MCOs from Pennsylvania and Washington said they identified high-expenditure beneficiaries by examining service utilization data to identify statistical outliers or trends. Officials from the two MCOs said they looked for statistical outliers for various types of service utilization, such as emergency department visits, inpatient stays, and pharmacy use. Officials from South Carolina said they built internal software tools to help them easily examine service utilization for various subsets of beneficiaries and services. These officials said they looked for beneficiaries whose utilization appeared to be significantly higher or lower compared with other beneficiaries with similar characteristics, such as among children with Type 1 diabetes or among children in foster care. The officials also said that after they identified those outliers, they examined the reasons for those beneficiaries’ utilization patterns to better understand why those beneficiaries were outliers and to take corrective action if appropriate. The officials explained that they did not simply focus on a discrete list of beneficiaries with the highest overall expenditures, because many of those beneficiaries have medical needs that are inherently expensive and cannot be meaningfully improved through intervention. Officials from three of the seven state Medicaid agencies and four MCOs said they identified high-expenditure beneficiaries based on diagnoses or other group categorization. Officials commonly said they used chronic conditions, such as end-stage renal disease, the human immunodeficiency virus or acquired immune deficiency syndrome, chronic obstructive pulmonary disease, diabetes, or Hepatitis C. Pennsylvania officials said their list was developed based on clinical experience. Officials from South Carolina said their list of diagnoses was based on a review of conditions associated with high expenditures. Officials from two state Medicaid agencies—Indiana and Nevada—and all five MCOs said they identified high-expenditure beneficiaries as beneficiaries who exceed certain service utilization or claims expenditure thresholds. Indiana officials said they used service utilization thresholds, such as visiting the emergency room six or more times in the past 6 months. Nevada officials said one of their programs identified high- expenditure beneficiaries as those whose treatment costs exceeded $100,000 over a 12-month period. Officials from the five MCOs offered varying thresholds, such as claims exceeding $100,000 over a 6-month period; claims exceeding $40,000 during a state fiscal year; or stays in a neonatal intensive care unit exceeding 15 days. Officials from two state Medicaid agencies—Nevada and Pennsylvania— four MCOs, and the ACO said they relied on clinical judgment to decide whether a beneficiary was likely to be high expenditure. Officials from one MCO in Washington said the MCO conducted health assessments of new members to obtain a baseline understanding of their clinical states, which were then used to stratify beneficiaries and identify appropriate staff to address their needs. Similarly, officials from Pennsylvania and three MCOs said clinical reviews of beneficiaries’ needs or histories were triggered by providers, caregivers, or self-referrals for care management or other services. Officials from the ACO said that while risk scores made initial predictions about beneficiaries’ risk for generating high expenditures, those predictions could be overridden by clinical judgment. Officials from all seven selected states, all five MCOs, and the ACO we interviewed said they used care management to manage the costs and quality of care for high-expenditure Medicaid beneficiaries. In addition, some states used other strategies, such as strategies involving coverage policies, payment incentives, and restrictions on the number of providers certain beneficiaries could use. Across states that evaluated these efforts to manage costs and quality of care, results were mixed. Officials from all of the seven state Medicaid agencies we interviewed reported that they provided care management for high-expenditure beneficiaries in their fee-for-service delivery systems, for example, by assessing patient needs and coordinating care across providers, in an attempt to manage costs and ensure quality care. Further, the six selected states with MCOs or ACOs required these organizations to provide care management to high-expenditure beneficiaries enrolled in managed care. Officials also reported barriers to their efforts to provide care management. Officials from all of the seven state Medicaid agencies we interviewed reported that they provided care management for high-expenditure beneficiaries in their fee-for-service delivery system, to manage the cost and quality of their care. The organization and scope of the care management programs they described vary in some cases. For example: Pennsylvania provided care management for beneficiaries in fee-for- service through the state’s “intensive case management” unit, a unit of providers that contact beneficiaries by phone to ensure that they get the care they need. Care management is provided to newly enrolled Medicaid beneficiaries who are identified as high-expenditure until the beneficiary selects a managed care plan, typically within 30 days, and to certain other beneficiaries. State officials said that of the approximately 150,000 beneficiaries in fee-for-service, they provide care management to about 1,000 each month. Nevada implemented mandatory care management services for high- expenditure fee-for-service beneficiaries in rural areas of the state through a contract with a care management organization, which was paid to reach out to high-expenditure beneficiaries, assess their needs, and connect them with their medical providers. The organization delivered care management through regional care teams geographically located in beneficiaries’ communities, which coordinated with the beneficiaries’ providers to implement personalized care plans and manage follow-up appointments and services. High-expenditure beneficiaries were assigned to one of eight care management programs based on the beneficiary’s qualifying condition, such as whether they had cancer, chronic kidney disease, or a mental health diagnosis. South Dakota implemented a health home program in 2013, which paid local primary care clinics, community mental health centers, and Indian Health Service facilities to provide care management to high- expenditure Medicaid beneficiaries. Each clinic or center had a care coordinator who reached out to high-expenditure beneficiaries to initiate care management and connect them with their primary care providers. These beneficiaries were placed in one of four categories indicating the level of care coordination they needed based on the severity of their illness and risk of future costs. The program helped beneficiaries create a care plan, set goals to address their particular care needs, and manage their conditions. In state fiscal year 2018, around 5,800 recipients received services through more than 100 health home clinics in South Dakota. Washington State also implemented a health home program in 2013 in which care management activities were coordinated through “lead” entities, such as Area Agencies on Aging and other community-based organizations. These entities established networks of care coordination organizations representing primary care, mental health, long term care, chemical dependency providers, and specialty providers. The lead entities conducted outreach to high-expenditure beneficiaries to connect them with a care manager, who might be a nurse, physician assistant, social worker, behavioral health professional, or chemical dependency professional. State Medicaid officials who have MCOs and ACOs within their states said that they required these organizations to provide care management to high-expenditure beneficiaries to manage the cost and quality of their care. Examples of states’ care management requirements included steps such as beneficiary and provider outreach, conducting screenings or health assessments, and developing care plans (see sidebar). Some requirements specified the minimum frequency for conducting outreach and what information and data must be reported to the state regarding care management activities (see sidebar). Beneficiaries with excessive utilization or under-utilization for conditions other than those specified diseases in the contract must also be eligible for disease management services. beneficiaries are categorized for different levels of care coordination. (Indiana Medicaid) standard model of care management for high-risk beneficiaries, but each clinical department in the MCO—for example, Obstetrics or Cardiology—established specific plans for care management within their area of care. Care managers in these departments—nurses or social workers—were responsible for coordinating with a beneficiary’s primary care provider to ensure that the beneficiary is appropriately referred to specialists. Care managers can contact beneficiaries by phone, but they are also based in the community, such as at hospitals and state mental health clinics. Officials from the ACO in Vermont said that the ACO paid providers that were part of their network—such as primary care offices, home health agencies, and mental health agencies—to serve as beneficiaries’ care managers. Beneficiaries select one provider to be their “lead care coordinator” based on who they have the strongest relationship and trust with, and this provider receives enhanced payments from the ACO to support coordination with other providers in the beneficiary’s care team. Care team members communicate with each other through a software tool provided by the ACO, which maintains updated information on beneficiaries’ conditions and the care received. Officials we spoke to from the selected states, MCOs, and the ACO identified barriers to implementing care management for some high- expenditure Medicaid beneficiaries, including the inability to contact beneficiaries, the lack of social supports—that are part of what is referred to as “social determinants of health”—and shortages of providers or care management staff in rural areas. Difficulties contacting beneficiaries. The lack of valid contact information can result from missing or outdated information, transiency and homelessness, and beneficiary reliance on cell phones with limited minutes. Officials described efforts they had taken to address this barrier, including asking pharmacies to confirm and get updated information when beneficiaries pick up prescriptions; using e- mail, which officials stated is more consistent than physical addresses; and conducting direct outreach in emergency rooms. Addressing Social Determinants of Health Officials at most of the selected states, managed care organizations (MCO), and the accountable care organization said they took steps to help beneficiaries address social determinants of health, for example, by gathering data to identify which beneficiaries needed help with social supports, helping beneficiaries obtain transportation to medical appointments, assisting beneficiaries in accessing social services, providing short- term housing, and meeting other needs. For example, officials from one MCO described a beneficiary with diabetes, who, despite consistently filling his prescription and adhering to his care plan, regularly visited the emergency department in insulin shock. Through outreach they discovered that the beneficiary could not appropriately store his prescribed insulin, which needed to be refrigerated, because his home did not have running electricity or a refrigerator. The MCO identified resources in the community to provide a refrigerator and restore electricity. Social determinants of health. The effectiveness of care management in addressing the health needs of high-expenditure beneficiaries can be hindered by the lack of social supports. Officials said that in order to help beneficiaries manage their medical needs, care managers sometimes needed to address these social determinants of health, such as lack of transportation to medical appointments, lack of stable housing, and inconsistent access to food and other basic resources (see sidebar). At the same time, states and MCOs can face challenges to addressing social determinants of health, such as lack of data on social determinants of health and a lack of understanding about the effect of social determinants of health on health care utilization, which if available could help bolster program investments in those areas. Staff shortages in rural areas. Efforts to provide care management and medical services can be hindered by staff shortages in rural areas. Officials with one state Medicaid agency’s health home program said there was a shortage of individuals in rural areas willing to provide care management to high-expenditure beneficiaries. MCO officials in another state said their ability to care for beneficiaries in rural areas was also affected by a shortage of care managers. Other strategies, in addition to care management, reported by selected states—South Carolina, Nevada, Pennsylvania, and Indiana—to manage the cost and care for high-expenditure Medicaid beneficiaries included coverage policy changes, payment incentives, and restrictions on the use of providers. Coverage policy changes. South Carolina Medicaid officials said that in certain cases they reviewed their coverage policy to see if changes could reduce costs and improve health outcomes for high-expenditure beneficiaries. For example, according to officials, the state had a small number of high-expenditure beneficiaries with Type 1 diabetes that officials thought could benefit from continuous glucose monitoring, which was not covered by their state Medicaid program. The officials said that they wrote a proposal into their state budget and drafted state plan amendment language to address this, though they noted that the proposal had not been implemented as of January 2019. Payment incentives. Medicaid officials in Nevada and Pennsylvania described efforts to use payment incentives to manage costs for high- expenditure beneficiaries. Nevada officials told us that the state’s arrangement with its care management organization for high-expenditure beneficiaries included payment incentives related to reductions in cost, as well as performance on certain quality measures, such as immunization rates and treatments for specific conditions such as asthma, coronary artery disease, and heart failure. However, state officials said that they faced difficulties measuring these outcomes. The care management organization did not receive incentive payments for the first year of operation of the program (2014-2015) and state officials said they did not have results on incentive payments for subsequent years. Pennsylvania officials told us that in response to the high cost of drugs to treat Hepatitis C, Pennsylvania’s Medicaid agency created a risk-sharing arrangement with MCOs that had high-expenditure beneficiaries with Hepatitis C. According to state officials, the MCOs were required to submit their enrollees’ Hepatitis C test scores to show whether beneficiaries were obtaining treatment and experiencing improvement. The state then allocated additional funds to MCOs that demonstrated positive quality outcomes, thus saving the cost of re-treating beneficiaries who failed to follow through on treatment. The Pennsylvania officials also told us that the state provided payment incentives to MCOs in its Integrated Care Plan Program, in which physical health and behavioral health MCOs coordinate with each other in the care of high-expenditure beneficiaries with persistent serious mental illness, such as schizophrenia, depression, or psychosis. To quality for incentive payments, these MCOs had to create an integrated care plan for each beneficiary with a qualifying condition. The state’s Medicaid agency identified outcome measures that MCOs were held accountable to in calendar year 2018 related to emergency department utilization, inpatient admissions, inpatient readmissions, prescription medication adherence, and engagement in treatment for substance use disorders. As metrics improved, MCOs become eligible for incentives. According to state officials, Pennsylvania allocated $10 million for Integrated Care Plan program incentive payments for calendar year 2018. Restrictions on the use of providers. Indiana Medicaid officials described their program to address over-utilization of services by certain high-expenditure beneficiaries who may be engaged in doctor or pharmacy shopping—a strategy of using multiple providers that results in over-utilization or improper utilization of prescription drugs or other services. According to the officials, if other efforts to address a beneficiary’s over-utilization fail over a 2- to 4-month period, the beneficiary may be enrolled in Indiana’s Right Choices Program. This program restricts, or “locks in,” the beneficiary to a single physician, pharmacy, and hospital. Officials said that this program has helped to ensure that the provider is aware of the beneficiary’s history and has proven effective in getting beneficiaries to change their behavior. In addition to using the program for Medicaid beneficiaries enrolled in fee- for-service, MCOs are provided with a report of their beneficiaries who have high-utilization levels so that the MCO can determine if any of these beneficiaries should be enrolled in the program. While some of the selected state Medicaid agencies reported that their efforts to manage costs and care for high-expenditure beneficiaries showed positive results, officials in other states reported mixed or inconclusive findings. Medicaid officials in four states—Pennsylvania, South Dakota, Vermont, and Washington—said their assessment of efforts to manage costs and care for high-expenditure beneficiaries showed positive results, such as cost savings or reductions in the use of expensive services. Pennsylvania Medicaid officials said that their Integrated Care Plan Program for high-expenditure beneficiaries with persistent serious mental illness resulted in improvements in utilization, including reductions in inpatient hospitalizations and readmissions. South Dakota Medicaid officials found that for 2017, health home participants cost $204 less per month than the comparison group, and experienced an 8 percent decline in emergency room visits from the prior year compared with a 10 percent increase in emergency room visits for the comparison group. The state estimated $7.7 million in costs were avoided. Vermont Medicaid officials analyzed utilization of high-expenditure beneficiaries in care management before and after they enrolled. The state reported in 2018 that the rate of inpatient visits per thousand beneficiaries decreased from 600 to 393, and the annual rate of emergency visits per thousand beneficiaries decreased from 1,536 to 1,003. An independent evaluation of a demonstration program for dually eligible beneficiaries in Washington that incorporated its Health Homes program found $107 million in Medicare cost savings over its first 42 months. As part of the state’s Financial Alignment Initiative, part of those savings went to the state Medicaid program. In contrast with the results reported by the four states, officials from Indiana and Nevada Medicaid agencies reported mixed or inconclusive findings related to the impact on cost or quality of their programs for high- expenditure Medicaid beneficiaries. Officials with Indiana’s Medicaid agency told us that an assessment of the Right Choices Program found relatively low cost savings generally, with the exception of pharmacy costs, where the program curbed excessive drug use among beneficiaries with substance use disorders and led to cost savings. Nevada Medicaid officials said that their fee-for-service care management organization appeared to achieve some cost savings, but had little effect on quality of care after the program was implemented in 2013. They also said that it was difficult to determine the true effect of the program, because the state implemented several other cost savings policies at the same time as the care management organization. Nevada let the program expire in 2018 and is researching other potential ways to manage high-expenditure beneficiaries in the state’s fee-for-service program. CMS offered optional tools, as well as technical assistance and other educational resources that state Medicaid agencies used to identify or better manage high-expenditure beneficiaries. CMS’s optional tools included the Health Home State Plan Option and the Financial Alignment Initiative, though these are not specifically designed for the purpose of identifying and managing high-expenditure beneficiaries. Medicaid officials in two selected states said that these programs improved their efforts to manage care for their high-expenditure beneficiaries. Health Home State Plan Option. The Medicaid Health Home State Plan Option, authorized under the Patient Protection and Affordable Care Act, allowed states to design health home programs to provide comprehensive care coordination for Medicaid beneficiaries with chronic conditions. CMS officials we spoke with said the states who chose the option received access to resources including planning funds and technical assistance from CMS. For example, CMS issued a brief illustrating how states could focus their health home programs on high-expenditure beneficiaries. CMS officials noted that they supported 23 states’ and the District of Columbia’s health home programs. Among the state officials we interviewed, South Dakota Medicaid officials said that when they were establishing their health home program, CMS was helpful in connecting them with other states that had created similar programs so that they could learn from other states’ experiences. South Dakota Medicaid officials stated they would like CMS to continue to bring health home program managers from several states together to discuss their successes, challenges, and innovations. Nevada Medicaid officials stated they were considering establishing a health home program. Financial Alignment Initiative. For the Financial Alignment Initiative, CMS oversaw efforts by states to implement improvements in Medicaid service delivery aimed at achieving savings for both Medicare and Medicaid, with one state we spoke with using the initiative to target high- expenditure beneficiaries. As noted earlier, Washington established its Health Homes demonstration program for dually eligible beneficiaries in association with the Financial Alignment Initiative. Washington targeted the demonstration to high-cost, high-risk Medicare-Medicaid beneficiaries based on the principle that focusing intensive care coordination on beneficiaries with the greatest need provided the greatest potential for improved health outcomes and cost savings. Washington’s Financial Alignment Initiative demonstration was approved through 2020, and Washington officials stated they are hoping to get an extension, because it has yielded cost savings for both Medicaid and Medicare. A feature of the Financial Alignment Initiative is that any cost savings achieved by the program are split between the state Medicaid program and Medicare. CMS also offered state Medicaid agencies access to several resources that, while not designed specifically to target high-expenditure beneficiaries, have been used to support states in identifying or better managing care for this population. These resources included the Medicaid Innovation Accelerator Program, the State Data Resource Center, and the Medicare-Medicaid Data Integration Initiative. Medicaid Innovation Accelerator Program. The Medicaid Innovation Accelerator Program is funded by the Center for Medicare and Medicaid Innovation and run by the Center for Medicaid and CHIP Services, both within CMS. The goals of the program were to improve care for Medicaid beneficiaries and reduce costs by supporting states in their ongoing payment and delivery system reforms through targeted technical support. The program offered participating states targeted technical support to Medicaid agencies in building their data analytic capacity as they design and implement delivery system reforms for high-expenditure beneficiaries, one of the program’s focus areas. The program worked with five states on issues such as identifying and stratifying beneficiaries with complex care needs and high costs, designing effective care management strategies, and incorporating social determinants of health into program design activities. In addition to working directly with five states, the program also offered a national webinar series under the broader topic of Medicaid Beneficiaries with Complex Care Needs and High Costs. The webinar series covered a variety of topics, including a webinar titled “Identification and Stratification of Medicaid Beneficiaries with Complex Care Needs and High Costs,” which provided information about different approaches to targeting and assessing the needs of this population. Vermont Medicaid officials we spoke with said it would be helpful to have more information about how social determinants of health impact beneficiaries’ ability to manage their own care. CMS hosted other webinars on various technical support and data analytics topics for states. Among the state Medicaid officials we interviewed, Nevada officials mentioned participating in the Innovation Accelerator Program. State Data Resource Center. State Medicaid agencies have traditionally been hampered in managing the Medicaid portion of care for dually eligible beneficiaries, because they lacked data on the Medicare services these beneficiaries receive, such as hospitalizations, physician visits, prescription drugs, and skilled nursing facility stays. To address this challenge, CMS established the State Data Resource Center to facilitate state access to and use of Medicare data on dually eligible beneficiaries. Through the program, states had access to technical advisors when working with CMS Medicare data, which have allowed states to better predict and identify high-expenditure dually eligible Medicaid beneficiaries, CMS officials told us. The officials said the State Data Resource Center provided states with learning opportunities through webinars and monthly “Medicare Data Workgroup” calls, during which states shared their data use experiences. CMS officials and CMS contractors we spoke with said 29 states have received Medicare data, including all 10 states that participated in the Financial Alignment Initiative, though not all had projects specifically linked to high- expenditure Medicaid beneficiaries. CMS officials said all states had some contact with the State Data Resource Center, whether through data inquiries or participation in webinars. Medicare-Medicaid Data Integration Initiative. The Medicare-Medicaid Coordination Office and the Center for Medicaid and CHIP Services’ Medicaid Innovation Accelerator Program, both within CMS, jointly sponsored the Medicare-Medicaid Data Integration Initiative. The initiative assisted states with integrating Medicare and Medicaid data in order to enhance care coordination and reduce costs for the dually eligible population, which may have included high-expenditure Medicaid beneficiaries. CMS officials we spoke with said the Medicare-Medicaid Data Integration Initiative had assisted 10 states—five participating in the Financial Alignment Initiative (Colorado, Minnesota, Ohio, Rhode Island, and Virginia) and five participating in the Medicaid Innovation Accelerator Program from October 2015 to March 2019 (Alabama, the District of Columbia, New Hampshire, New Jersey, and Pennsylvania). We provided a draft of this product to the Department of Health and Human Services for review. The department provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or yocomc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the contact named above, Lori Achman (Assistant Director), Mary Giffin (Analyst-in-Charge), Matthew Dobratz, Drew Long, and Brandon Nakawaki made key contributions to this report. Also contributing were Julianne Flowers, Vikki Porter, Jennifer Rudisill, and Eric Wedum.", "summary": "Medicaid, a joint federal-state health care financing program, is one of the nation's largest sources of health care coverage for low-income and medically needy individuals. A 2016 report published by the National Governors Association noted that high-expenditure Medicaid beneficiaries typically have poorly managed chronic conditions and a host of unmet social needs that result in potentially preventable use of costly services, such as emergency department visits. The report also noted that identifying and better managing those beneficiaries are key to reducing costs and improving outcomes. GAO was asked to examine state and federal efforts to manage costs and improve care coordination for high-expenditure Medicaid beneficiaries. This report describes (1) approaches selected states used to identify or predict high-expenditure Medicaid beneficiaries; (2) strategies selected states used to manage beneficiaries' health care costs while ensuring quality of care; and (3) resources CMS provided to states to help them identify, predict, or better manage high-expenditure beneficiaries. GAO interviewed officials from CMS, as well as Medicaid officials from a nongeneralizable sample of seven states (Indiana, Nevada, Pennsylvania, South Carolina, South Dakota, Vermont, and Washington) and five MCOs. States were selected for variation in their total Medicaid enrollment, enrollment in Medicaid managed care, percentage of state population living in rural settings, and percentage of state population with disabilities. MCOs were selected based on state suggestions, and varied in terms of whether they operated nationally or on a state or regional basis. GAO previously reported that in fiscal years 2009 through 2011, the most expensive 5 percent of Medicaid beneficiaries accounted for nearly half of the expenditures for all beneficiaries; others have also found that a small percentage of beneficiaries account for a disproportionately large share of Medicaid program expenditures. These high-expenditure beneficiaries are an extremely diverse population with varying needs. GAO found that the seven selected states identified or predicted high-expenditure Medicaid beneficiaries using statistics and other approaches. For example, states used risk scores, which estimate an individual beneficiary's expected health care expenditures relative to the average expenditures for beneficiaries in the group. Other approaches included examining service utilization data to identify statistical outliers and using diagnoses, service utilization and claims expenditure thresholds, or clinical judgment to identify or predict high-expenditure beneficiaries. To manage costs and ensure quality of care for high-expenditure beneficiaries, the seven selected states used care management and other strategies. Care management . All the selected states provided care management—providing various types of assistance such as coordinating care across different providers to manage physical and mental health conditions more effectively—for beneficiaries in their fee-for-service delivery systems. Five of the states also contracted with managed care organizations (MCO) to deliver services for a fixed payment and required the MCOs to ensure the provision of care management services to high-expenditure beneficiaries. Other strategies . Some of the seven selected states used additional strategies to manage care for high-expenditure beneficiaries. For example, Indiana officials described a program to restrict, or “lock in,” a beneficiary who has demonstrated a pattern of high utilization to a single primary care provider, hospital, and pharmacy, if other efforts to change the beneficiary's high utilization were unsuccessful. The Centers for Medicare & Medicaid Services (CMS), which oversees the Medicaid program at the federal level, offered optional tools and other resources to support states' efforts to identify or better manage high-expenditure beneficiaries. For example, CMS officials said states received access to resources and technical assistance on establishing health home programs—which seek to better coordinate care for those with chronic conditions—including how to focus on high-expenditure beneficiaries. CMS officials noted that they supported 23 states' and the District of Columbia's health home programs. CMS also offered several resources that, while not designed specifically to target high-expenditure beneficiaries, have been used to support states in identifying or better managing their care. For example, CMS's Medicaid Innovation Accelerator Program offered targeted technical support to states' Medicaid agencies in building their data analytic capacity as they designed and implemented delivery system reforms, which could be used to identify high-expenditure beneficiaries. Officials in two selected states reported that these tools were beneficial for managing the health care costs associated with high-expenditure beneficiaries. HHS provided technical comments, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "VA’s mission is to serve America’s veterans and their families, and one of the ways it does so is by providing veterans with medical services. To help meet the health care needs of veterans, VHA is planning to complete approximately 70 new major medical projects between 2020 and 2024. Activation is one of the key steps that must occur before veterans can access care at these facilities. According to VA’s Activation Process Guide, activation typically involves activities such as planning for, purchasing, and installing new furniture, fixtures, and equipment (FF&E), ordering supplies, and hiring staff. For new buildings, the Guide states that activation activities begin when the building is being designed, continue through construction, and end when the facility is fully operational. The expenses associated with activation can reflect either one-time purchases or ongoing expenditures. One time purchases—called non- recurring activation expenses—involve the acquisition of assets such as furniture or equipment, or payment to a contractor for services such as equipment installation. Ongoing expenses, or expenses incurred more than once—called recurring activation expenses—are for staff salaries and consumable supplies, such as gowns and gloves. After a facility opens and begins serving patients, facilities are permitted to treat supplies and the salaries of new staff as activation costs until the site is serving enough patients to receive funding through one of VA’s regular funding processes, known as the Veterans Equitable Resource Allocation (VERA). Figure 1 provides examples of recurring and non-recurring activation expenses. The total cost of activation for major lease and major construction projects can be substantial. The median activation funding that facilities reported spending on major activations from fiscal year 2012 through 2018 was approximately $16 million. The four newest hospitals (in Denver, Las Vegas, Orlando, and New Orleans) spent a cumulative total of more than $1.9 billion for activation during this time period. The types of facilities undergoing activation can vary in size, services provided, and overall purpose within the VHA healthcare system, as shown in figure 2. For example, a community-based outpatient clinic (CBOC) is typically much smaller than a medical center but can provide primary, specialty, subspecialty, mental health, or any combination of delivery services that can be appropriately provided in an outpatient setting. Large medical centers can provide outpatient services as well as a broad range of inpatient services, including emergency services, surgery, and acute psychiatric care. Smaller facilities may refer patients to medical centers for complex treatment. National, regional, and local staffs play different roles in the activation process: National: VHA’s Activations Office—under the Office of Capital Asset Management (OCAM)—historically provided ad-hoc support to sites activating a major lease or construction project, such as providing on-site training related to the activation process and facilitating input from subject matter experts within VHA. The office also determines the base amount of activation funding that sites receive. Officials overseeing the office stated that its role is being reassessed and that the type of support it provides for activations may change in light of an internal reorganization and consideration for VA’s future growth plans. Regional: VHA’s18 regional networks, known as Veterans Integrated Service Networks (VISN), are responsible for the coordination and oversight of all administrative and clinical activities at health care facilities within their specified region. A VISN’s role in activation varies depending on the expertise available at the facility level, but VISNs can help facilities arrange contracts for services (like laundry or hazardous waste removal); review a facility’s budget submissions to VHA; and facilitate discussions with senior management or knowledge-sharing with other sites that have recently completed activation. The VISNs are also responsible for distributing activation funding from VHA. Local: In addition to providing medical services, medical centers function as administrative hubs for services in the area. As a result, the medical center director is ultimately responsible for activating facilities within the center’s administrative boundary. The medical centers can appoint staff to manage the activities required for activation. These staff can include the activations project manager, financial officers, and subject matter experts like interior designers. As a team, the staff are responsible for developing technical requirements, creating risk mitigation strategies, and deciding key acquisition dates, among other tasks. While VHA has not identified standard milestones for activation, based upon our review of VHA documents and interviews with local and regional VHA officials, we found that two events are especially relevant to the planning and execution of activation activities: (1) building acceptance (when VHA formally takes possession of and occupies a building) and (2) providing medical services to the first patient. Figure 3 describes examples of activation activities in relation to these events, although the actual timing of tasks will vary depending on the needs of individual facilities. Officials from selected facilities said that prior to building acceptance, their activation activities typically focus on determining furniture and equipment needs, placing orders, anticipating staffing needs, and hiring new staff. Determining furniture and equipment needs is intertwined with the building design process, according to officials from three VHA facilities, because the design of the physical space can dictate what equipment is purchased. For example, a VHA official from one health-care center said that the activations team showed the medical care providers a mock-up of a treatment room and created cardboard models of furniture to help them select items. The official told us that getting the medical care team’s input early in the planning process can avoid the need to make costly changes in order to make the physical space fit the equipment or furniture requested by the medical care providers. Conversely, the building may be designed to accommodate specific equipment. For example: Officials from one facility shared the specifications of the radiology equipment with the team designing the building in order to leave a proper amount of space for the equipment. Similarly, an annex’s activation staff worked with the resident engineer to design an enclosed area separate from their main building for a mobile MRI machine. This design ensured a new MRI machine could be swapped out in the event of a breakdown without causing a disruption to the facility’s operations. Officials said that they also begin the purchasing process for equipment and furniture prior to building acceptance. VHA officials stated they work backwards from the construction endpoint to determine when to order items. VHA officials told us they need to place orders for certain items— such as high-tech equipment or made-to-order furniture—well in advance of the facility’s opening because the items are known to have long delivery times. For example: Officials from one clinic reported they ordered their facility’s imaging equipment 22 months before they needed it. Similarly, an official from a different clinic said that furniture is often not manufactured until it is ordered, so it can take several months to arrive. In contrast, the official said items like a staff refrigerator could be picked up at a local store within days and do not require substantial advance planning. Facilities also begin planning for their workforce needs prior to building acceptance. For example: An official from one clinic stated that facilities typically identify their staffing needs during this time period by position, title, and pay. Officials from an annex said that before their facility’s construction groundbreaking, they discussed how many staff would move from the old facility to the new facility, and how many new staff they expected to hire. After activation teams determine their staffing needs, facilities hire and begin training new staff. For example: An official from one clinic said that new staff needed to be trained prior to opening day, so it is not uncommon for staff to be hired and brought on-board before the facility begins providing clinical services. In the case of a very large facility, such as a medical center, hiring the required staff can require an extensive search that must commence before the building is finished. An official from a medical center said that a shortage of skilled medical workers required a nationwide search for suitable candidates. Officials said that after a building is accepted as complete, activation typically focuses on tasks associated with moving into the space, such as equipment installation and training staff. For example: One clinic’s project calendar showed in the weeks leading up to opening day that the activations staff planned to install office furniture such as desks and filing cabinets, as well as to perform checks on biomedical equipment to ensure proper functioning. Officials from another clinic coordinated equipment and furniture deliveries between the warehouse (where items were being stored) and the new facility. The extent of staff training after building acceptance depends on the need to familiarize staff with the new facility, and the complexity of services offered. Activation staff might choose to have medical staff become familiar with the new facility by working at the facility prior to new operations. For example: One clinic’s staff started working in the building before their first patient was seen in order to become familiar with the new space. An official from a medical center said that facility staff adjusted to operating newer infrastructure, such as learning to operate a modern computerized boiler system. That official also stated that the medical center might need to conduct extensive training exercises to simulate 24/7 inpatient care. In contrast, outpatient facilities that do not operate around the clock may not have these same training needs. Once a facility begins providing medical care, officials said that activation tasks are typically related to facility operations. These tasks can include on-the-job training in the new space and making necessary adjustments to the facility to ensure it runs properly while concurrently serving new patients. For example: An official at one clinic said that beginning patient care with a decreased workload, known as a “soft opening,” can help facilitate on- the-job training. The same official explained that this approach allows staff to become accustomed to their new facility’s operations and address any issues that may emerge without the demands of operating at full capacity. VHA officials from a health care center said that space adjustments included repositioning exam beds and ordering ergonomic chairs. Officials at several sites stated that they used SharePoint, an internal communication tool, to keep track of needed adjustments. This approach enables staff to monitor ongoing issues during the beginning of new operations, resolve unexpected problems, and track issues as they occur. Several VHA officials also said that some activation tasks —such as hiring staff—may occur after a facility begins serving patients. If a facility plans on a phased opening, in which some services will not be available on the first day, processes that would typically be completed earlier may take place during this time frame instead. For example, a medical center in our review utilized a phased-opening approach, as it expanded its capabilities with new medical services after opening. The facilities included in our review provided most medical services within planned time frames; however, nearly one-third of services were delayed for various reasons. Overall, 59 of the 87 services were offered within planned time frames (69 percent). Of the 28 services that were not provided on time, staffing, equipment size, “commissioning”, and procurement issues contributed to the delays, according to officials. Staffing issues delayed a total of 14 services in two of the seven facilities reviewed. One facility had 13 services with delays that ranged from 4 to 6.5 months. Officials said the delays were due to difficulties recruiting the staff necessary for those services, which included various types of surgery, radiology, and mental health, among others. Similarly, difficulties recruiting a dentist at a second facility delayed dental service 4 months beyond the expected delivery time frame. Equipment at one facility did not fit into some of the rooms and the space needed to be altered in order to accommodate it. Officials said that all 12 services were delayed by approximately 1 month so that the facility could open with all services available, though officials noted that the full extent to which the equipment issues contributed to these delays was unknown (i.e., there could have been other causes that they could not recall.) Commissioning issues delayed women’s healthcare services at one annex by approximately 1 month. Officials said that the air circulation rate—which needed to be higher in rooms where certain procedures are performed—was inadequate. As a result, the air exchange had to be improved before the facility could begin performing the planned clinical procedures. Procurement issues led to delays in providing radiology services at one facility. Officials told us that x-ray services were delayed by 3 months because the equipment was ordered through the centralized purchasing process, which took longer than local officials had anticipated. These delays primarily affected services that were originally planned to be offered within 2 months of building acceptance. While selected facilities planned to offer approximately 92 percent of services within 2 months of building acceptance, as shown in figure 4, 61 percent were actually offered within that time frame. VHA does not provide a guideline for how much time facilities should need after building acceptance to provide clinical services. Officials explained that the appropriate amount of time will vary based upon the scope of the project, including factors such as the number and kinds of services offered and the level of effort associated with installing the equipment (e.g., a replacement hospital will require more effort than a small outpatient clinic). Thus, we did not determine if facilities were allotting appropriate amounts of time to complete activation activities and serve patients. However, VA’s Activation Process Guide provides some information regarding when full services should be available. The Guide states that clinical services can be added for up to 6 months after opening day (i.e., the first day that patients receive any services at the facility). The Guide further states that facilities may expect to offer services gradually—versus all on opening day—when services are new to an area. Of the 87 services offered by the facilities in our review, 86 were offered within 6 months of opening day. The remaining service—a clinic that provides colonoscopy and other related procedures at one facility— opened on schedule approximately 11 months after opening day. This facility was replacing another facility that had not previously offered this service. Officials explained that because the service was new, they needed more time to develop and equip the space as well as hire staff, so they planned on offering this service later than services that were being transferred from the previous facility. VHA lacks processes to develop total cost estimates for major activations. Without total cost estimates, VHA is unable to determine whether actual activation expenses are higher or lower than planned. Furthermore, VHA does not have documentation that defines allowable activation costs, including what facilities can purchase with activation funding and when facilities should cease spending activation funds. As a result, VHA officials lack critical information to support decision-making about resource allocation, and are not well positioned to effectively identify and investigate deviations from planned spending. VHA lacks processes to develop reliable total activation cost estimates for major activation projects and to compare actual costs against these estimates. According to our assessment of information from VHA, the current cost estimation process does not cover the full duration of activation and does not reflect best practices for developing reliable cost estimates. In addition, VHA officials said that until recently, the agency lacked the accounting mechanisms necessary to facilitate comparisons of a project’s total activation costs against estimated costs; however, while VHA now possesses these mechanisms, it has not documented the process for how the new information should be used. The Activations Office and facility activation staff annually develop cost estimates for the upcoming 3 fiscal years using (1) an activation cost model (the model) and (2) a cost template (the template). According to Activations Office officials, the model is managed by the Activations Office and uses inputs such as a facility’s square footage and project schedule. Activations Office officials also said that the template is typically completed by facility activation staff and includes inputs such as planned clinical services as well as estimated staffing, equipment, and supply costs. While the cost estimate is driven primarily by the model, information in the template is also considered before annual activation funds are distributed, according to Activations Office officials. Figure 5 shows the steps for determining and distributing annual activation funds. We determined that the model and template do not estimate costs for the entire duration of a facility’s activation. According to our review of facilities’ spending data, activation spending for a given facility can occur over more than 3 fiscal years. All eight of the facilities in our review, for example, spent activation funds over 4 or 5 fiscal years. Thus, the estimate the Activations Office would have developed at the beginning of these projects would not have reflected total activation costs. Moreover, we did not see evidence that VHA medical facilities independently develop total activation cost estimates that are appropriate to compare against total actual costs. None of the eight selected facilities we reviewed could provide total activation cost estimates appropriate for this use, according to officials at each facility. Officials from five facilities stated that they had not developed such estimates, an official from one facility said that an estimate could not be located and probably had never been done, and officials from two facilities said that such documentation could not be located. VHA officials said that the existing cost estimation tools reflected the budgeting process (i.e., the current fiscal year and two future years) and that they had not previously been required to develop a cost estimate for the entirety of activation. Officials noted that as the Activations Office’s role shifts to include more oversight, it will be important for the Office to have total cost estimates for activation; however, as of September 2019, VHA did not have any specific plans for how to collect estimates for a project’s entire activation cost. We also found that VHA’s current process for developing activation cost estimates does not fully align with best practices for developing cost estimates as established in the GAO Cost Guide (see table 1 below). VHA’s process minimally met 10 and did not meet 2 of the steps—each of which reflects multiple best practices—required to develop reliable cost estimates. A reliable cost estimate is critical to the success of any program, providing the basis for informed decision-making, realistic budget formulation and program resourcing, and accountability for results. VHA officials acknowledged that following these practices would be valuable for the activations process, and explained that the agency did not previously incorporate these practices because they had not assessed the strength of their activation cost estimation process in this manner. Lastly, the Activations Office does not compare existing estimates and actual activation costs. While the Activations Office develops activation cost estimates for the upcoming 3 fiscal years and has some capabilities to track activation costs, to date it has not compared the planned costs to actual expenses. According to Activations Office officials, they have historically been unable to track how activation funding was spent at the facility level, which impeded such comparisons. Starting in fiscal year 2020, officials from the Activations Office plan to use accounting codes associated with each activation project, which will allow them to track expenses at the facility level. An internal review conducted in mid-2019 by the department overseeing the Activations Office concluded that the agency needed to regularly assess the extent to which activations spent funds as planned. As of October 2019, however, officials said the office has not documented the process for how they will deploy their new accounting oversight capabilities, including which personnel would be responsible for conducting such comparisons, the frequency of comparisons, and any follow-up steps that would be considered in the event of significant differences. Without processes for estimating total costs and comparing them against actual expenses, the Activations Office is limited in its ability to improve resource planning, budgeting, and allocation—critical elements that support VA’s stated management priority to enhance data-driven decision-making. Further, guidance from the Office of Management and Budget states that agencies should obtain information on actual project costs and compare them against planned expenses so managers can have a clear understanding of how resources are being used and whether cost goals are being met. Documented processes for cost estimation and comparison would be particularly important in the case of large medical centers, whose activation costs are in the hundreds of millions of dollars. The Activations Office has not clearly defined what officials at local facilities can purchase with activation funding and how long activation funding should continue after opening day. Activations Office officials said that there is a general understanding that some expenses, such as medical equipment for new facilities or services, are activation expenses, and that the Activations Office intends to provide activation funding until the facility begins to receive VERA funding to cover operational expenses. However, there is no policy to inform facility activation staff of what they can purchase with activation funding and when funding will cease. In mid-2019, an internal review conducted by the department overseeing the Activations Office found that the lack of clarity regarding what could or could not be purchased should be remedied; however, as of September 2019, no specific plans have been established to define appropriate purchases. Officials we spoke with—both at the selected medical facilities and VISNs—expressed uncertainty about what expenses they could pay for using activation funding. Officials from two of eight facilities told us that there were times when they did not know if they should charge an expense to activation or another funding source, such as construction accounts. For example, officials at one facility told us that they were unsure whether construction or activation funds would pay for the special window blinds needed for the intensive care units. Officials at four of the VISNs also said that when contacted by medical facility officials for guidance on allowable expenses, there were times when they did not know if facilities should charge an expense to activation or another funding code. In addition, officials from the selected sites held differing views on how long they were eligible to receive activation funding from the Activations Office. Finance officials for one of the selected facilities said that activation funding is provided for up to 5 years, while officials from several other facilities said that activation funding is available until operational expenses are covered by VERA. Activations Office officials said that the latter interpretation is accurate and that this transition to ongoing VERA support should take place within approximately 2 years after opening day. However, an official from the Activations Office said that a few facilities have received funding from the Activations Office for more than 2 years after opening because there was no clear definition for when activation funding should cease. The lack of clear definitions regarding what constitutes allowable activation expenses and when activation funding should end limits VHA’s ability to consistently and accurately estimate and track activation costs. For example, similar facilities could develop varying total cost estimates due to different understandings of what expenses are allowable. VA management priorities include making data-driven decisions to improve resource planning, budgeting, and allocation. In addition, Standards for Internal Control in the Federal Government states that management should use quality information to achieve the entity’s objectives. Clear definitions on what expenses facilities should charge to activation accounts, and for how long, would improve the Activations Office’s ability to monitor activation costs and improve resource stewardship. As VHA undertakes the process of replacing facilities to better reflect its focus on outpatient and specialized care, it is poised to spend hundreds of millions of dollars per year to equip and staff these new sites. However, VHA does not have a clear understanding of total costs and whether individual activation projects are spending funds effectively. Because VHA does not have a process for developing an estimate for the entire activation cost of a project, the agency lacks a critical baseline that can inform future spending decisions. In addition, because VHA lacks a process that describes how officials should compare actual expenses to that estimate, the agency has no mechanism to regularly identify and respond to unplanned differences in activation costs. Furthermore, defining allowable activation expenses would better position VHA to ensure total cost estimates are consistent from facility to facility. Lastly, additional clarification on how to estimate activation costs and compare them against actual expenses would help VHA to more effectively manage the activations process. Without processes and clear definitions associated with activation cost measurement, VHA does not have reasonable assurance that it will be able to effectively manage the resources associated with activation. We are making the following four recommendations to VA: The Assistant Deputy Under Secretary for Health for Administrative Operations should develop and document a process for estimating total activation costs for major medical facility projects. This process should reflect the 12 steps for developing a reliable cost estimate outlined in the GAO Cost Guide. (Recommendation 1) The Assistant Deputy Under Secretary for Health for Administrative Operations should develop and document a process for comparing actual activation costs for major medical facility projects to estimates. This process should identify the personnel responsible for comparing the estimated costs to the actual expenses and document their responsibilities. (Recommendation 2) The Assistant Deputy Under Secretary for Health for Administrative Operations should define and document what items and services officials can purchase with activation funds. (Recommendation 3) The Assistant Deputy Under Secretary for Health for Administrative Operations should define and document when facilities should cease to spend activation funds. (Recommendation 4) We provided a draft of our report to VA for review and comment. VA provided written comments, which are reprinted in appendix II. VA concurred with all of our recommendations. VA further provided information on how it intends to address our recommendations, with target dates for completion in December 2020. We are sending this report to the appropriate congressional committees and to the Secretary of the Department of Veterans Affairs. In addition, this report is available at no charge on the GAO website at http://gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other key contributors to this report are listed in appendix III. To understand the costs of the activations of the eight selected facilities, we asked activation officials at each facility to provide a breakdown of the activation costs by the following categories: (1) Furniture, Fixtures, and Equipment; (2) Staffing; (3) Supplies; (4) Other; and (5) Total Cost. We used these cost categories because these are the categories in the template that facilities complete to estimate activation costs. In addition to the contact above, Heather Halliwell (Assistant Director); Alison Snyder (Analyst-in-Charge); Rose Almoguera; Brian Bothwell; Geoffrey Hamilton; Jason Lee; Terence Lam; Ethan Levy; Josh Ormond; Daniel Setlow; Laurel Voloder; Mary Weiland; and Elizabeth Wood made key contributions to this report.", "summary": "VHA operates one of the nation's largest health care systems with more than 1,200 sites across the country; however, many facilities were built decades ago and do not align with the agency's current emphasis on outpatient and specialized care. Additionally, new or expanded facilities are needed to accommodate veterans returning from recent conflicts. VHA is constructing and leasing new facilities to respond to these needs. GAO was asked to review VHA's efforts to activate new major medical facilities. This report examines the extent to which VHA is able to compare the actual costs of activation against the estimated costs, among other objectives. GAO analyzed VHA's documentation on estimating activation costs. GAO also interviewed officials and analyzed cost information reported by a non-generalizable selection of eight medical facilities. The facilities had more than $1 million in annual rent or $20 million in construction costs, reported finishing activation in fiscal years 2016 and 2017, and were located in various regions. The Veterans Health Administration (VHA) under the Department of Veterans Affairs (VA) is constructing and leasing new medical facilities, such as outpatient clinics, to better serve and meet the changing needs of veterans. VHA equips and staffs these new facilities in a multi-year process called “activation.” From fiscal year 2012 through 2018, VHA channeled more than $4 billion to major medical facilities undergoing activation, which these facilities could use toward furniture, equipment, and new staffing costs, among other start-up expenses. VHA lacks processes and clear definitions for estimating total activation costs and for comparing actual expenses against these estimates. Specifically, VHA's current cost estimation process does not cover the full duration of activation. Headquarters officials have never compared activation costs against estimated costs because until recently, officials said, VHA lacked the accounting mechanisms to facilitate such comparisons; however, while VHA now possesses these mechanisms, it has not documented the process for how the new information should be used. VHA documentation does not clearly define allowable activation expenses or the appropriate spending timeframes. Local and regional officials expressed confusion over what items could be purchased with activation funds. In addition, local officials held inconsistent beliefs regarding how long expenses could qualify as activation-related. VHA management's priorities include data-driven decision-making. Further, the Office of Management and Budget's guidance states that agencies should compare actual project costs against planned expenses so managers can determine if cost goals are being met. Without processes and clear definitions associated with measuring activation costs, VHA does not have reasonable assurance that it will be able to effectively manage the resources associated with activation. GAO recommends that VA (1) develop and document a process for estimating total activation costs, (2) develop and document a process for comparing actual activation costs to the estimates, (3) define allowable activation expenses, and (4) clarify when facilities should cease to classify expenses as activation-related. VA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "In 2017, transit agencies provided over 10 billion rides to people traveling to and from businesses, homes, and other locations throughout the United States, according to FTA. Transit infrastructure for those rides includes railways, roads, bridges, tunnels, and stations. According to the 2017 American Public Transportation Association’s (APTA) Fact Book, more than 6,700 organizations provided public transportation in a variety of modes in 2015. Transit modes include: fixed-route bus services—the most prevalent transit mode in the country—vehicles operate according to regular schedules along prescribed routes with designated stops; rail services—vehicles operating along railways; ferryboat services—vessels carrying passengers and/or vehicles over a body of water; paratransit services—generally, accessible, origin-to-destination transportation services that operate in response to calls or requests from riders; and other demand-response services—sometimes called “dial-a-ride.” Public transportation, or transit, is statutorily defined as regular, continuing shared-ride surface transportation services that are open to the general public or open to a segment of the general public defined by age, disability, or low income. This definition has multiple statutory exclusions including intercity bus service, school bus service, and charter bus service. The transit infrastructure requires appropriately trained personnel to operate, maintain, and oversee services and assets (e.g., vehicles). APTA estimated that, in 2015, U.S. public transit agencies employed more than 430,000 fulltime and part-time personnel, including contractors. The transit workforce consists of a variety of occupations, such as bus operators, train conductors, dispatchers, mechanics, supervisors, and other occupations (see fig. 1 for examples.) FTA supports transit agencies’ workforce development by providing financial and technical assistance, among other things. FTA’s financial assistance efforts include implementing the Innovative Public Transportation Frontline Workforce Development Program, which provides competitive grants to transit agencies (transit workforce grants) to assist with the development of innovative human resources activities. FTA awarded transit workforce grants in fiscal years 2011, 2012, and 2015. Grant-eligible projects included employment training, outreach to increase minority and female employment in transit, research on transit personnel and training needs, and training and assistance for minority business opportunities. Grant recipients included transit agencies, non- profit community groups, schools, and others. Additionally, recipients can use up to one-half of one percent of certain grant funds, such as Urbanized Area Formula Grants, for eligible human resources and training activities with the approval of DOT. FTA also administers the National Transit Database (NTD), which is intended to provide information to assist in transit planning efforts. All recipients and direct beneficiaries of grants from the Urbanized Area Formula Program and Rural Area Formula Program, such as local transit agencies, are required to report certain data to the NTD. FTA also encourages agencies not receiving grants from these programs to report voluntarily so the NTD can be more complete. In 2016, over 2000 transit agencies submitted full or partial reports to the NTD. The NTD stores information from local transit agencies, such as financial and operating data, to inform transit service planning for government agencies and other organizations. Some workforce data, such as the number of full-time and part-time transit agency employees, is reported to the NTD. NTD data provide information on the transit workforce at the time the data are reported, but are not used to project future transit workforce needs. FTA partners with the National Transit Institute (NTI), hosted and staffed by Rutgers University, to provide workforce development educational resources and training. NTI delivers over 300 courses per year nationwide to public transit employees and government transportation agency employees at all levels. These courses focus on compliance with federal regulations and developing skills to operate a transit agency. At the department level, in August 2015, DOT, DOL, and the Department of Education worked with industry stakeholders to project the employment and skill needs of the transportation industry from 2012 to 2022. Primary sources for the report included the DOL’s Bureau of Labor Statistics (BLS) employment projections, current population survey (demographics), and analysis from Economic Modeling Specialists International. The effort developed a variety of transportation workforce statistics and resulted in the Strengthening Skills Training and Career Pathways Across the Transportation Industry: Data Report on Future Transportation Workforce Needs (Transportation Industry Report). Among other things, the report: projected that an additional 4.6 million transportation workers will need to be hired to fill vacancies created by separations (occupational transfers, retirement, and other exits from the workplace), and net job growth from 2012 to 2022; provided data on current worker distribution (at that time) by age and sex for six transportation sectors and by race and ethnicity for selected transportation occupations; and included top job occupations by sector and projected industry and occupational job openings based on separations and job growth. The extent of future transit workforce needs is unclear due to the absence of transit-specific workforce projections, unclear communication on the data that are available, and because the data that are available do not extend past 2022. The best information available on future transportation workforce needs, according to FTA officials is the August 2015 Transportation Industry Report. The Transportation Industry Report does have projections on transportation workforce needs, but the transit industry data are combined with other ground passenger transportation industries such as intercity buses, charter buses, taxis, school buses, and limousines. Thus, the report does not exclusively reflect the transit workforce. According to researchers who wrote the report, the transit and ground passenger transportation data were reported as one industrial sector because it would be a significant undertaking to focus solely on transit workforce data without funding a study for this specific area. The Transportation Industry Report was developed with data, in part, from BLS, which does not exclusively report on occupational projections for public transportation. BLS develops workforce projections of the U.S. labor market by industry, subsector, and occupational codes, including the number of employees and types of employers. BLS officials we interviewed said that industry and occupational data sets do not allow the level of specificity that would be needed to identify only transit agencies. The Transportation Industry Report predicted 1 million job openings in the transit and ground passenger transportation sector from 2012 to 2022 and listed the top 10 projected job openings. However, these projected job openings include a number of occupations within services that are statutorily excluded from the definition of transit. For example, the three largest categories of job openings—comprising about 72 percent of the projected openings—have the following key transit exclusions: School bus and special client bus drivers made up approximately 33 percent of the projected openings (330,699 job openings). However, school bus services are specifically excluded from the statutory definition of transit, as are sightseeing services, charter bus services, courtesy shuttle services for patrons of one or more specific establishments, and intra-terminal or intra-facility shuttle services. Transit and intercity bus drivers made up almost 20 percent of the projected openings (199,727 job openings). However, intercity bus service (for example, Greyhound bus service) is specifically excluded from the statutory definition of transit. Taxi drivers and chauffeurs made up almost 19 percent of the projected openings (188,895 job openings). However, these services may not meet the statutory definition of transit. However, when communicating about the report, FTA has not always made clear that the data in the Transportation Industry Report combine both transit and ground passenger transportation workforce projections, though this was not intentional, according to FTA officials. FTA has presented the combined data as representing the “transit” or “public transit” workforce in recent annual reports to Congress and a few public presentations. For example, one of the findings in the Transportation Industry Report is that the combined sector of transit and ground passenger transportation has the highest percentage of older workers who are at or nearing retirement age. However, in recent reports to Congress, FTA states that, public transit has the highest percentage of older workers at or nearing retirement age. However, this statement did not reflect that the percentages included transit and ground passenger transportation data. We also found similar information involving retirement percentages and job openings in a number of FTA presentations that are available to the public online. In addition, we found two examples of incorrect numbers in recent reports to Congress that FTA officials said were “typos” that have now been corrected in the most recent fiscal year 2017 report. FTA’s characterization of this data could confuse transit stakeholders, including Congress, on needs, retirements, and growth in the transit industry. When FTA identifies combined information as “transit” projections, the audience may not understand the extent to which the data reflect services that do not meet the statutory definition of transit. We found evidence that this may have already occurred to some extent. During our review, we found examples of stakeholders in the industry repeating the same statistics that FTA has presented as “transit” in publications and in our interviews with them, raising questions about whether the industry may have misconceptions about the future transit workforce. FTA officials told us that the combination of transit and ground passenger transportation is appropriate because of similarities in the industries and the common practice of transit agencies hiring contractors from ground passenger transportation to supplement workforce personnel. However, the context of this information has not always been clear in reports to external parties. Further, the Transportation Industry Report’s projections on the future transportation workforce are only estimated through 2022. DOT officials said that they do not have plans to update the report beyond 2022 or to develop a report that focuses solely on transit workforce projections. FTA officials told us that they plan to hire a data scientist to assist them with transit workforce issues. FTA officials also told us in November 2018, that the report was intended to provide trend information and that they do not plan to use the transit numbers from the Transportation Industry Report in future reports and presentations, and considering that the report will soon reach the end of its projections in 2022, we are not making a recommendation about how to communicate the context of the information in the Transportation Industry Report. Opinions on the need for additional transit workforce data and projections varied among transit stakeholders we interviewed. Several stakeholders cited the difficulties of collecting transit-specific projections or other types of data, while others pointed to the need for more data such as data identifying shortages in specific occupations and retirement age of transit employees. However, officials from the Community Transportation Association of America, which represents small and rural transit agencies, said that requiring additional data from transit agencies could be a time- consuming burden for local transit officials. The association officials suggested that transit stakeholders should work together to use existing workforce data from FTA or BLS to develop workforce projection data. Transportation Research Board officials also said that additional transit workforce data would be costly and difficult for transit agencies to provide. However, the Director of Eno’s Center for Transportation Leadership, a research organization, stated that it is extremely difficult to develop national policies, programs, and grants to address systemic workforce problems because of the gaps in transit workforce data. Federal Internal Control Standards highlight the importance of using quality information to make informed decisions and identifying the information requirements needed to do so while considering the expectations of internal and external users. While the views of stakeholders we interviewed varied on the extent to which additional transit workforce data and projections are needed, new transit workforce projections could inform decision-making on transit workforce planning to address potential future shortages or other needs. We have previously reported that agencies should weigh data collection decisions carefully, noting that there is a cost to data collection, and that only needed data should be collected. Working with stakeholders to understand what, if any, additional transit-specific workforce data transit stakeholders need and the related collection costs could enable FTA to weigh the complete costs and benefits of developing future data for the transit industry and to make informed decisions on allocating the appropriate resources toward those efforts. Transit stakeholders we interviewed highlighted actions they are taking to address transit workforce needs but also noted continuing difficulties with recruiting and retaining staff. Examples of actions they have taken to address workforce needs, either with a transit workforce grant or with other funding include: Career enhancement: Los Angeles County Metropolitan Transportation Authority officials told us that they have developed a program that offers growth opportunities by providing “upskilling” resources at all levels of the agency including employee development, management/leadership, and transportation senior leadership, among other things. Courseware development: The Transportation Learning Center organized three industry consortiums to develop national standards- based courseware—Rail Car, Signals, and Elevator/Escalator Technicians, according to the Transportation Learning Center. For example, under the Signals Training Consortium, 25 new courses have been developed covering the inspection, maintenance, and troubleshooting of transit and commuter rail signaling equipment. The curriculum is planned to include both classroom and on-the-job training. Internships: The Conference of Minority Transportation Officials developed a program to prepare college, university, and vocational school students to enter transit and transportation-related fields, according to conference officials. In 2018, this program placed 29 interns nationwide in architectural and engineering firms as well as state and local government agencies. Managerial training: The Eno Center for Transportation provided classes for mid- and senior-level transit executives and has started one for first-line supervisors, according to an Eno official. This training includes lectures, classes, job shadowing, informal mentoring, field trips, and meetings with counterparts. Research: The Community Transportation Association of America reported on a survey it conducted of its members in June 2018 on the salary and benefits for professional transit positions in the industry. The survey’s 236 respondents provided information on hourly and/or salary information, operating budget, available benefits, services provided, and number of employees, among other things. According to the Association, the members asked for this survey because it helps them make staffing and employment decisions within their agencies. Technology education: Jacksonville Transportation Authority officials told us that they have established a “Workforce of the Future” working group whose charge is to prepare the workforce to incorporate emerging technologies as it transitions its aged elevated, automated people-mover system—the “Skyway”—to autonomous vehicle technology. The working group supports a public automated-vehicle test track as well as employee town halls to develop the tools to discuss these issues in their communities. FTA’s transit workforce grants also supported stakeholder actions to address transit workforce issues including: In fiscal year 2011, FTA awarded 12 workforce grants totaling $3 million. For example, the Chicago Transit Authority received a grant to develop and validate a transit-manager competency model to help supervisors recognize and support skills and leadership potential in their staff. In addition, the Florida Department of Transportation received a grant for its Certified Transit Technician Program, which resulted in 13 hires, each receiving the opportunity for additional certification and a college degree. In fiscal year 2012, FTA awarded 17 grants totaling $7.05 million. For example, the Southwest Ohio Regional Transit Authority (Cincinnati) received a grant to develop a program that provided technical training in hybrid engine technology to improve its maintenance program and hybrid bus fleet. The Corporation to Develop Communities of Tampa, Inc. received a grant to recruit, train, and employ up to 30 people in the transit industry, including transit operations and maintenance workers. In fiscal year 2015, FTA awarded 16 grants for $8.1 million. For example, the Bay Area Rapid Transit District was awarded a grant to create a pathway to employment in the transit industry for traditionally under-represented individuals. The Workforce Development Council of Snohomish County, Washington, received a grant to bring together local partners to create a pipeline of skilled workers ready to enter the transit and construction industries. The partners have targeted women, minorities, and native tribes to access apprenticeships, social services, and job placement programs. Notwithstanding these efforts, the transit stakeholders we spoke with identified ongoing recruiting and retention challenges as shown in table 1. FTA has taken a number of actions to assist transit agencies with future workforce needs including: FTA provides technical assistance, standards development, training, and workforce development projects for the transit workforce. In fiscal year 2017, the active projects totaled over $29 million. Almost 13.3 million (approximately 46 percent) of this funding went to technical assistance projects. For example, FTA spent almost $4 million to assist the National Aging and Disability Transportation Center with efforts that include developing online courses and training materials to assist certain FTA grant recipients with providing transportation services to older adults and people with disabilities. The funds were also used to present a podcast, as part of an online course on the Americans with Disabilities Act of 1990 (ADA). The podcast addressed common ADA questions related to customer service, wheelchairs on vehicles, and service animals. Over $8.3 million (approximately 28 percent) funded 17 active transit workforce development projects from the fiscal year 2015 transit workforce grants discussed previously for developing new training curriculums, or seeking to recruit and train specific groups, especially those who are underrepresented in the transit workforce. $5 million (approximately 17 percent) funded NTI development and delivery of training programs for federal, state, and local transportation employees. For example, NTI delivered 270 training courses throughout the U.S. to 7,298 participants in fiscal year 2017. NTI also supported a workshop to help address industry issues as they arise, such as workforce shortages and issues in recruitment and retention. $2.5 million (approximately 9 percent) funded a Transit Standards Development Program at the Center for Urban Transportation Research (University of South Florida) to provide research and analysis on needs and gaps, and recommendations for new standards, or to modify existing standards. For example, one of the reports discussed fatigue management, among other things. According to FTA, the ultimate effects of this program are increased safety and reduced injuries and fatalities. FTA is collaborating with NTI to conduct an industry workforce needs assessment aimed at identifying training, skills, and educational gaps that exist in the industry as well as within current NTI programs. They are conducting this assessment because of transit’s changing workforce, technologies, and operating environment. This assessment is intended to result in a report that provides a road map for the transit workforce’s development and training and may identify some of the gaps in transit workforce needs. As of November 2018, NTI has held focus groups at key transit conferences, which have provided a preliminary picture of the transit industry’s critical need, according to FTA. NTI has also completed a draft of a survey of transit agency needs and FTA is reviewing it, according to an NTI official. The next steps include conducting the survey, mapping NTI’s curriculum and courses to desired key skills, and developing a national transit competency framework. The researchers are planning to use a statistically significant sample that represents the needs of the industry, both for urban and rural agencies, with estimated completion of the survey and analysis in early 2019. FTA is evaluating the effectiveness of the transit workforce grant program. FTA funded an evaluation of the 12 fiscal year 2011 transit workforce grants. The report discussed whether the projects met goals, the effect of the programs, and whether the programs were worth further investment. The report noted a number of outcomes including introducing 2,608 youth to transit, and training 1,527 people. FTA also plans to evaluate transit workforce grants awarded in fiscal years 2012 and 2015 and to create outreach materials from the grant projects for transit stakeholders. The evaluation is also intended to provide important best practices and lessons learned for other transit operators. FTA is researching the potential effects of automation on the transit workforce. In January 2018, FTA released the Strategic Transit Automation Research Plan, which established a research and demonstration framework. One of the research projects in the plan is an assessment of the effect of automation on the transit workforce. Specifically, the assessment is planned to provide a qualitative analysis of labor-related considerations with transit bus automation including potential workforce changes, perspectives of organized labor, statutory and regulatory provisions, and other societal factors. A follow-on project is planned to evaluate changes in staffing levels, job responsibilities, labor hours, and training needs to provide a quantitative approach to estimating automation’s effects on transit employment levels, workforce needs, and wages. Although FTA has assisted transit stakeholders with workforce needs, it lacks key strategic planning practices that could ensure its efforts are as effective as possible. FTA reported in its fiscal years 2016 and 2017 annual reports to Congress that it planned to develop a transit workforce strategic plan. Federal Internal Control Standards indicate that plans, such as strategic plans, should set up the effective and efficient operations necessary to fulfill desired objectives. Effective operations produce the intended results from operational processes, while efficient operations do so in a manner that minimizes the waste of resources. However, FTA does not have a comprehensive strategy showing the operations and processes to be developed to guide FTA’s efforts to assist transit agencies with addressing future transit workforce needs. FTA has had a number of starts and stops in producing a transit workforce strategy since it first reported this intention to Congress in 2016, but no clear action has been taken to develop a strategy so far. In July 2018, FTA officials told us that the reason they had not yet drafted a comprehensive strategy is because they considered developing their strategy as part of an overall DOT strategy, rather than a transit workforce strategy as a stand-alone product. However, DOT does not currently plan to develop a comprehensive department-wide transportation workforce strategy. Nevertheless, DOT has consistently identified addressing the transportation workforce as a priority over time in key DOT documents, including its last three strategic plans and the last two performance plans. DOT officials told us that the strategic plan is not intended to provide such detail; rather, it is designed to be a top-level strategic plan that provides a broad framework for DOT. In November 2018, FTA officials told us that they intend to create a “workforce consortium” and a new technical assistance project that would result in a strategy. However, FTA officials did not provide a time frame for when these actions would be taken. Considering the importance of the transit workforce to efficient operation of the transit infrastructure, a transit workforce strategy would be consistent with internal controls, such as setting up effective operations, whether this strategy is developed as part of a department-wide strategy, as a stand-alone project, or through a workforce consortium. Without a comprehensive strategy to guide FTA’s ongoing activities to assist with transit workforce needs, FTA lacks a roadmap to ensure it is effectively leveraging its resources to help address future transit workforce needs. In addition, it may be difficult for Congress to understand the merits of investing in future transit workforce programs because it may not be clear absent a vision of how individual programs fit within the overall transit workforce strategy. In addition to not having a comprehensive strategy, FTA lacks key tools to demonstrate the extent to which individual workforce development efforts are addressing future transit workforce needs. In particular, FTA has not established clearly defined performance goals and measures for its transit workforce assistance efforts. Establishing clear goals and measuring progress toward them are consistent with the management principles set forth in GPRA, as enhanced by the GPRA Modernization Act of 2010, and our previous work. Setting long-term strategic goals is essential for results-oriented management, because such goals explain with greater specificity the results an agency is intending to achieve, as we have previously reported. FTA discussed the pending development of transit workforce goals at an October 2016 summit with transit stakeholders, but these goals were not finalized. Further, there are no performance goals for transit workforce development efforts in DOT’s current annual performance plan, and none is referenced in the current strategic plan. FTA has developed some performance measures for evaluating the outcomes of transit workforce grants—but not for its transit workforce development efforts at large. In addition, these measures are not tied to performance goals that FTA expects the grants to achieve. We have previously reported that results-oriented organizations first set performance goals to clearly define desired outcomes and then they develop performance measures that are clearly linked to the program goals and demonstrate the degree to which the desired results are achieved. For example, two of the performance measures for transit workforce grants are “total projected cost per direct participant,” and “number of people expected to be trained overall,” but FTA has not set a goal for what the projected cost per participant should be or the number of people who should be trained by grant awards. By establishing performance measures before establishing specific performance goals that FTA seeks to achieve, FTA may not ensure that the data gained from these performance measures are an effective use of resources. DOT officials said that performance goals and measures for FTA’s transit workforce grant program were not finalized because no funding has been identified for a subsequent round of these grants. However, FTA’s efforts to assist with transit workforce issues are larger than one grant program. Performance goals and measures are consistent with effective management practices with or without funding for a specific grant program. Without documented, clearly defined goals and performance measures linked to those goals, FTA is limited in its ability to make informed decisions about transit workforce development efforts. As a result, FTA risks expending resources on efforts that it may not be able to demonstrate are meeting intended goals. Focusing on the intended results of FTA’s transit workforce efforts can promote strategic and disciplined management decisions that are more likely to be effective because managers are better able to target areas most in need of improvement and to select appropriate interventions. Further, agency accountability can be enhanced when both agency management and external stakeholders—such as Congress—can assess an agency’s progress toward meeting its goals. Without performance goals and related performance measures, it will be more difficult for FTA to determine the success of its strategies, adjust its approach when necessary, and remain focused on results. Much is unknown about the workforce needed in the future to operate the nation’s transit systems and to transport people to and from work, school, and other destinations. Disaggregating transit workforce data from other transportation data has proved to be challenging, and the best projections of the future transportation workforce needs will expire in 2022. Additionally, how vehicle automation and other technology advances will affect the future transit workforce is unclear, and FTA’s presentations on the transit workforce projections that do exist may have contributed to the lack of clarity on the future needs of the industry. Whether additional, refined data on transit workforce needs—for example, an updated version of the Transportation Industry Report—would provide greater benefits to the industry than the cost of collecting these data is something FTA can determine when it better understands the information the industry needs to make effective workforce decisions. At that point, FTA can decide what additional data need to be collected strategically, if any, and at what cost, as part of strategic planning efforts. FTA has identified the need to create a transit workforce strategy, and has expressed its intention to create one in a number of different ways, but has taken no clear action yet to ensure that FTA’s intention will be realized. By taking the initiative to develop a strategy to help address future transit workforce needs, FTA will be in a position to better manage its ongoing transit workforce activities. FTA is undertaking a number of efforts that could provide the foundation for sound strategic planning, including sponsoring an assessment of transit workforce needs, hiring a data scientist, starting a workforce consortium, and initiating research on future automation that could provide more clarity regarding a key aspect of future transit that is, as of now, an unknown factor. Further, FTA has already drafted some performance measures for its transit workforce grants that it may be able to use as a foundation for creating goals and measures for transit workforce development at large. However, more specific strategic planning efforts that include developing a strategy and performance goals and measures can better enable FTA to effectively help transit agencies identify, prepare, and provide a sufficient workforce for the future. We are making three recommendations to the FTA Administrator: The FTA Administrator should determine, in collaboration with transit stakeholders, whether additional transit workforce data are needed to identify potential future occupational shortages in the transit industry and whether the benefits of this collection would outweigh the cost of gathering it. (Recommendation 1) The FTA Administrator should develop and document a strategy that outlines how FTA will help address future transit workforce needs. (Recommendation 2) The FTA Administrator should develop and document clearly defined performance goals and measures for its transit workforce development efforts. (Recommendation 3) We sent a copy of this draft report to DOT for review and comment. DOT responded with a letter in which it concurred with our recommendations and discussed the successes of its Innovative Workforce Development Program. The letter is reprinted in appendix II. DOT also provided technical comments, which we incorporated in the report as appropriate. We are sending copies of this report to appropriate congressional committees and to the Secretary of Transportation. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who contributed to this report are listed in appendix II. This report addresses: (1) the extent to which information exists about future transit workforce needs, (2) the actions selected transit stakeholders are taking to address current and future transit workforce needs, and (3) the extent to which the Federal Transit Administration (FTA) assists with identifying and addressing current and future transit workforce needs. For all objectives, we analyzed key Department of Transportation (DOT) and FTA documents such as DOT strategic plans, annual reports to Congress in fiscal years 2017 and 2016, and public presentations, which discussed workforce needs, grant programs, and data involving the transit industry. We also interviewed government officials from DOT, including from FTA, the Office of the Secretary, and the Federal Highway Administration; from the Department of Labor (DOL) and its Bureau of Labor Statistics (BLS); and from the Department of Education. Further, we interviewed three transit stakeholders—the Eno Center for Transportation, the National Transit Institute (NTI), and the American Public Transportation Association (APTA) to understand available data sources, relevant studies, and grant programs focused on the transit workforce. Based on our research and recommendations from those interviews with those three stakeholders and FTA, we selected a non- generalizable sample of an additional eight transit stakeholders for interviews. These selected stakeholders included: two research organizations (Transportation Learning Center and the Transportation Research Board, which includes the Transit Cooperative Research Program and the Transit Research Analysis Committee); two unions (Amalgamated Transit Union and Transport Workers Union of America); two trade groups (Community Transportation Association of America and Conference of Minority Transportation Officials); and one membership association of workforce boards (National Association of Workforce Boards); one transportation subject-matter expert (Mort Downey Consulting, LLC). In addition, we selected and interviewed officials from six transit agencies to understand their perspectives regarding workforce issues such as recruiting and retention challenges, and efforts to address those challenges. We selected and interviewed three urban transit agencies: Los Angeles County Metropolitan Transportation Authority (CA); Regional Transportation District—Denver (CO); and the Jacksonville Transportation Authority (FL). We selected these agencies because (1) each was recommended by more than one transit stakeholder we interviewed for taking specific actions to address workforce issues; (2) as a group, they represented geographic diversity (western, central, and eastern United States); and (3) each was awarded at least one FTA transit workforce grant. In order to find smaller, more rural agencies for balance we selected three transit agencies: Advance Transit (Wilder, Vermont); Cache Valley Transit District (Logan, Utah); and the Ki Bois Area Transit System (Stigler, Oklahoma) based primarily on recommendations from the Community Transportation Association of America, which represents thousands of the rural and tribal transit agencies, and then considered geographic diversity (western, central, and eastern United States) and a mix of services offered. Although the views of these selected officials and stakeholders are not generalizable to those of all transit agencies and stakeholders, they represent a range of perspectives and expertise regarding the transit workforce. To determine the extent to which information exists about the future transit workforce, we evaluated the Transportation Industry Report, which projected the employment and skill needs of the transportation industry from 2012 to 2022. We evaluated the scope, methodology, and limitations of the Transportation Industry Report’s workforce projections, as that report provided the best available information according to FTA officials. We analyzed the report’s projected job openings data in the transit and ground passenger transportation sector to understand the extent to which the data represented transit-specific data. In addition, we reviewed FTA’s annual reports to Congress for fiscal years 2016 and 2017, and public presentations to document communication to the public involving transit workforce data. We also reviewed FTA’s National Transit Database (NTD) 2017 and 2016 calendar year data on transit agency employees. Further, we interviewed BLS officials and reviewed the most recent BLS employment projections from 2016–2026 to understand the extent to which the data could be separated to represent only transit- specific data. We reviewed the Standards for Internal Control in the Federal Government on communicating and preparing quality information and compared FTA actions to this information. To determine what actions selected transit stakeholders are taking to address current and future transit needs, we reviewed efforts to address transit workforce needs taken by stakeholders whom we interviewed. We also reviewed challenges involving recruiting and retaining transit workers discussed during our interviews with those stakeholders. We judgmentally included examples in our report to demonstrate the breadth of actions that are being taken. We analyzed various FTA reports on transit workforce grants awarded in fiscal years 2011, 2012, and 2015 and included examples to demonstrate the variety of projects that the transit workforce grants covered. We also included examples in our report of challenges that transit stakeholders we spoke with generally discussed, grouped under common themes. Themes that we included in the report were cited multiple times by stakeholders we interviewed. To determine the extent to which FTA is assisting transit agencies with identifying and addressing current and future workforce needs, we interviewed officials from FTA and DOT’s Office of the Secretary to document their efforts to identify and address current and future transit workforce needs. We compared FTA’s actions to address transit workforce needs to Federal Internal Control Standards, the Government Performance and Results Act of 1993 (GPRA), the GPRA Modernization Act of 2010, and our previous work. We conducted this performance audit from January 2018 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Heather MacLeod (Assistant Director); Amy Higgins (Analyst-in-Charge); Nelsie Alcoser; Melissa Bodeau; Lacey Coppage; Terence Lam; Josh Ormond; Pamela Vines; and Elizabeth Wood made key contributions to this report.", "summary": "FTA provides more than $12 billion annually to support and expand transit services. The operation of transit systems depends on a skilled, qualified workforce, but impending transit worker retirements and advances in transit technology may create challenges for the transit workforce such as finding eligible applicants for transit jobs and obtaining the technology expertise needed. GAO was asked to review various issues related to the sufficiency of the transit workforce. This report discusses the extent to which: (1) information exists about future transit workforce needs and (2) FTA assists with addressing current and future transit workforce needs, among other things. GAO reviewed DOT and FTA documents, including strategic and performance plans, and interviewed DOT and FTA officials and other transit stakeholders, including representatives of transit agencies, research organizations, and unions. Stakeholders were selected based on recommendations from other transit stakeholders and for geographic diversity, among other factors. The nation's transit infrastructure requires a trained workforce, consisting of a variety of occupations (see figure), to operate, maintain, and oversee it. Information on future transit workforce needs is limited in part by the absence of transit-specific workforce projections. According to Federal Transit Administration (FTA) officials, the best information available is an August 2015 report developed by the Department of Transportation (DOT) and other federal stakeholders to produce transportation job projections. However, the report's transit data are combined with ground passenger transportation data (e.g., school buses, taxis), and many of these services are specifically excluded from the statutory definition of transit. Transit-specific data were not available and would be costly to obtain, according to the researchers who wrote the report. Thus, the report does not exclusively reflect the transit workforce. The views of stakeholders GAO interviewed varied regarding whether additional workforce data were needed. Working with stakeholders to understand what, if any, additional information is needed could enable FTA to weigh the complete costs and benefits of developing future transit workforce data. This approach could also enable FTA to make informed decisions on allocating the appropriate resources toward transit workforce efforts. While FTA assists transit stakeholders with addressing workforce needs—for example, providing about $29 million in workforce development assistance in fiscal year 2017—it lacks key strategic planning practices that could ensure its efforts are effective. FTA first reported to Congress in 2016 that it planned to develop a transit workforce strategic plan; however, no clear action has been taken to develop one so far. Further, FTA does not have clearly defined performance goals and measures—as outlined in the Government Performance and Results Act of 1993 (GPRA) and the GPRA Modernization Act of 2010—for FTA's transit workforce development efforts. Without these key strategic planning practices, FTA is limited in its ability to make informed decisions about effectively leveraging its resources to address future transit workforce needs and in measuring the effectiveness of its efforts. GAO is making three recommendations to FTA: (1) in collaboration with stakeholders, determine whether additional transit workforce data are needed; (2) develop a comprehensive transit workforce strategy; and (3) develop performance goals and measures for FTA's transit workforce development efforts. DOT concurred with our recommendations.", "document_type": "gao"}
{"report": "CMS uses the Medicare Physician Fee Schedule to pay physicians and other providers for services delivered to beneficiaries. Physicians and other providers bill Medicare for their services using various five-digit billing codes based in part on codes developed by an AMA panel. Each year, the panel receives proposals from provider groups and others to revise existing billing codes or create new codes. The panel requires those who submit proposals to develop a clinical vignette that describes the typical patient who would receive the service, the diagnosis and relevant conditions, and estimates of time that physicians might spend in providing the service for the typical patient. The panel applies several criteria in reviewing these proposals. For example, a new code should represent a unique, well-defined procedure or service clearly identified and distinguished from existing procedures and services; should not fragment an existing procedure or service represented by one or more existing codes; should reflect the typical (not extraordinary) circumstances related to the delivery of the service; should be performed by many physicians or other qualified health care professionals across the United States; and should be consistent with current medical practice. CMS pays providers a fixed amount known as the Medicare fee for each code. The fees are based on relative values—estimates of resources for the physician’s work (time, skill, and level of training), and practice expenses (the costs of running a practice such as salaries of non- physician employees, rent, and overhead) required to provide a service relative to all other services. In setting fees, CMS also does not allow certain codes to be billed together if it deems that payment for one code is already included in another. CMS establishes and updates relative values annually. By law, the effect of any changes to its payment rates generally must be budget neutral. That is, if total spending increases by more than $20 million each year, including due to the creation of new billing codes, fees for all services would have to be reduced accordingly. Services billed under the physician fee schedule may be provided in a variety of settings, including physicians’ offices and institutional settings such as hospitals, skilled nursing facilities and hospices. Non-physicians may also bill or be reimbursed by Medicare for services under certain circumstances. For example, some types of non-physicians practicing independently—such as physician assistants and nurse practitioners— may bill Medicare for certain services that they are legally authorized to perform under their respective state laws. In other instances, physicians may bill as if they had furnished services that were provided by non- physician staff that they employ or with whom they have a contractual relationship as long as the physician has an established relationship with the beneficiary, and is on the premises to provide supervision if necessary. Providers and other stakeholders have noted that they care for an elderly population with increasingly complex medical conditions who receive care from multiple providers across different sites of care including physicians’ offices, hospitals, nursing homes, and hospices. As such, the focus of primary care has shifted from treating specific medical conditions to increased care coordination and planning. CMS also noted that a new trend in care planning is the use of shared care plans between the beneficiary and the provider rather than those created solely by the provider. These jointly developed care plans can be particularly important to improving overall beneficiary outcomes for beneficiaries with serious illnesses and also allow other providers involved in the beneficiary’s care access to timely information that supports planned care. However, stakeholders have suggested that Medicare’s payment system does not fully reimburse providers for such care planning services. For example, some note that the E/M billing codes that primary care physicians generally use to bill for their services were developed at a time when care coordination and planning was not part of the standard practice of medicine; as such, these codes do not reflect time spent on activities that do not require a face-to-face encounter with the beneficiary, including medical conferences with other physicians, or telephone calls to coordinate care with other providers. Some primary care physicians have requested that CMS conduct a comprehensive review of existing E/M codes to ensure they account for time spent on these services, or develop new codes that primary care physicians may exclusively use to bill for these services. However, others have noted that E/M codes have been reviewed and valued by the AMA, and the codes account for the time spent on these services. Moreover, they have stated that care coordination and planning services are delivered by multiple specialties, not just primary care physicians. Our analysis identified at least 58 Medicare Physician Fee Schedule billing codes that providers may use to bill for LCCP-type services. These 58 billing codes generally contain components we determined to be equivalent to the five key components of the LCCP service as defined in the 2018 BBA. For example, all 58 codes included a provision for the development of a care plan that addresses the beneficiary’s goals, values, and preferences, and a provision for coordination with other providers, which is equivalent to the LCCP component related to interdisciplinary care. Providers may choose a single code or a combination of these codes to account for the time, skill, and resources needed to deliver the service based on the unique health needs of each patient. (See app. IV for more information on the 58 codes and the LCCP components they contain as defined in the 2018 BBA.) The 58 billing codes for LCCP-type services include 45 longstanding, broadly-defined codes and 13 narrowly-defined codes that were more recently introduced starting in 2013. Broadly-defined codes. Of the 45 broadly-defined codes, 39 are E/M codes that have existed for decades. E/M codes are broadly defined to include services provided to treat a variety of illnesses (for example, treatment of a particular medical complaint), but they may also be used to bill for LCCP-type services. In general, the E/M codes range in complexity from low to high depending on the amount of time the provider spends with a patient as well as the complexity of the medical condition(s) being treated. E/M codes may also be billed if more than 50 percent of the time allotted for the service is spent on counseling and care coordination—for example, explaining treatment options and ways to mitigate the patient’s health risks—which are key components of the LCCP service as defined in the 2018 BBA. The E/M codes we identified as representing LCCP-type services were the more complex codes that had estimates of time that may be spent providing the service to a typical patient ranging from 30 to 120 minutes of physician time and 3 to 71 minutes of non-physician time. While the majority of E/M codes have existed for decades, CMS added six new E/M codes starting in 2008—referred to as “prolonged” E/M codes—allowing payment for additional time for care planning and care management services for complex conditions. Narrowly-defined codes. Starting in 2013, CMS added 13 narrowly- defined LCCP-type codes to better account for the time spent coordinating care for patients with complex treatment needs. CMS implemented these more narrowly-defined care planning codes largely in response to provider complaints that E/M codes did not sufficiently account for extensive care management/coordination of care that was required across multiple providers and settings. Unlike broadly-defined codes, these narrowly-defined codes can only be used for LCCP-type services. The 13 narrowly-defined LCCP-type codes fall into four types: transitional care management (TCM), chronic care management (CCM), advance care planning (ACP), and behavioral health integration (BHI). (See table 1.) While some pertain to patients with specific types of health conditions or in certain settings, others are more general and may be used for a range of health conditions. The estimates of physician and non-physician time that may be spent on the broadly-defined and narrowly-defined codes vary, as does Medicare’s 2019 fees for these billing codes—see examples of commonly used LCCP-type billing codes in table 2 and see appendix IV for related information on all 58 LCCP-type billing codes. For example, some stakeholders told us they might bill a complex E/M code (99214) along with a CCM code (99487). As our analysis shows, this combination could result in the provider spending 66 minutes of physician time and 113 minutes of non-physician time for a typical beneficiary, and receiving total Medicare fees of about $203 in 2019. Overall Medicare spending on LCCP-type services represented by the 58 billing codes we identified increased from $26 billion in 2013 to $29 billion in 2017, an 11 percent increase. The vast majority of this spending— about $28.3 billion in 2017—was on services represented by the 45 broadly-defined codes we identified earlier (henceforth we refer to these services as “broadly-defined services.”) By comparison, Medicare spending on LCCP-type services represented by the 13 narrowly-defined codes (henceforth referred to as “narrowly-defined services”) was about $467 million in 2017. Though smaller in terms of total dollars, spending on narrowly-defined services grew at a higher rate than spending on broadly-defined services, from about $2 per beneficiary in 2013 to $14 per beneficiary in 2017. This higher rate in growth can mostly be attributed to these four new types of services being introduced during this 4 year period. For example, as Table 1 shows, two TCM codes were introduced in 2013 and four CCM codes were introduced from 2015 to 2017. In contrast, spending growth for broadly-defined services was much smaller, increasing from about $785 per beneficiary in 2013 to $844 per beneficiary in 2017. For all other Medicare Physician Fee Schedule services combined, per-beneficiary spending decreased from about $1,488 in 2013 to $1,426 in 2017. Spending on CCM and TCM services accounted for most of the total spending on narrowly-defined LCCP-type services from 2013 to 2017. (See fig. 1.) For example, in 2017, TCM services accounted for almost half ($213 million of the total spending of $467 million), while spending on CCM services accounted for over a third ($162 million of the $467 million). The growth in spending on narrowly-defined services was driven by increased utilization—that can be attributed in part to the development of new codes for these services—rather than increases in Medicare fees for these services. Specifically, utilization of narrowly-defined services increased from about 9 services per 1,000 beneficiaries in 2013 to about 177 services per 1,000 beneficiaries in 2017. Average Medicare fees for these services remained flat during this period. The number of beneficiaries receiving narrowly-defined LCCP-type services increased substantially from 2013 to 2017, as more of these Medicare billing codes were added and began to be utilized during this time. Specifically, in 2017, about 2.5 million beneficiaries received narrowly-defined LCCP-type services, representing an 839 percent increase from about 267,000 beneficiaries in 2013. (See fig. 2.) While the overall number of beneficiaries receiving narrowly-defined services increased, these services were concentrated among a relatively small share of Medicare beneficiaries. Specifically, one-quarter of beneficiaries who received any of the narrowly-defined services in 2017 received 62 percent of the approximately 6 million services that were provided that year. (See fig. 3.) In 2017, of the total 2.5 million beneficiaries that received narrowly- defined services, 90 percent received only one type of narrowly-defined LCCP-type service. (See fig. 4.) In contrast, only 10 percent of beneficiaries received multiple types of narrowly-defined LCCP-type services, the most common combination being CCM and ACP. Mirroring beneficiary trends, the number of Medicare providers billing for narrowly-defined LCCP-type services also increased significantly from 2013 through 2017, as these Medicare billing codes were established and began to be utilized during this time. In 2017, a total of about 100,000 providers billed for narrowly-defined services, representing a 227 percent increase from about 31,000 providers in 2013. (See fig. 5.) As with beneficiary trends, while the overall number of providers billing narrowly-defined services grew from 2013 to 2017, billing for these services was also increasingly concentrated among a small share of providers. Specifically, in 2017, 10 percent of providers who billed for any narrowly-defined services billed about 76 percent of the approximately 6 million services that were provided in that year. (See fig. 6.) Each year from 2013 through 2017, physicians specializing in internal medicine accounted for the largest share of spending on narrowly-defined LCCP-type services. In 2017, internal medicine accounted for 45 percent of the $467 million in total Medicare spending on narrowly-defined services. (See fig. 7.) Family practice and nurse practitioners were the other specialties accounting for the greatest shares of spending. In terms of the setting in which narrowly-defined LCCP-type services were provided, the majority were provided in nonfacility settings such as physicians’ offices. Specifically, in 2017, 94 percent of narrowly-defined services were provided in nonfacility settings. This trend was consistent over each of the 5 years from 2013 to 2017. Six of the 19 stakeholders we interviewed did not support the creation of a new billing code for an LCCP service as defined in the 2018 BBA. Two of these—representing physician specialties that together accounted for almost one-fifth of total spending on LCCP-type services in 2017—stated that the existing billing codes were sufficient for them to provide and bill for the full range of the LCCP service. They stated that billing either a single code or a combination of an E/M code and one or more of the 13 narrowly-defined LCCP-type codes we identified allowed them to account for the full range of the LCCP service as defined in the BBA. As such, the two stakeholders said, there was no need for a new billing code. The remaining four stakeholders expressed concerns about creating a new billing code for an LCCP service. These concerns included the following: Overlap with existing codes that require the development of care plans: While not explicitly stating that existing codes were sufficient, some stakeholders said that if a new billing code were created for the LCCP service as defined in the 2018 BBA, it would overlap with or duplicate existing billing codes. For example, three stakeholders noted potential overlap with existing billing codes, such as the ACP and CCM. Three stakeholders said that the care plan that would be required under the new LCCP code would duplicate existing care plans that are required by law for beneficiaries in hospices or skilled nursing facilities. In addition, two stakeholders noted that providers in their specialty already prepare detailed care plans as a standard practice of care when evaluating their patients and billing for these services using existing E/M billing codes. They stated that these care plans exceed the components of the care plan specified in the 2018 BBA. Stakeholders noted that the existence of multiple overlapping codes that include the development of a care plan could create confusion for providers in choosing the most appropriate billing code. Concerns about code proliferation or code fragmentation: Several stakeholders were concerned that adding another code to Medicare’s billing system could result in increased Medicare spending and less, rather than more, care coordination. Specifically, two stakeholders stated that having multiple billing codes for care planning and care management, respectively, would have the potential to increase spending because multiple providers could start billing the new codes even though one provider may have primary responsibility for the beneficiary. (In contrast, under the existing billing codes a single code that encompassed both types of services could be billed.) For example, one of these stakeholders said that primary care physicians generally referred beneficiaries with complex treatment needs to a surgeon or specialist who then both planned and managed the beneficiary’s care, yet the primary care physician might also bill the care planning billing code. In addition to the potential for increased Medicare spending, three stakeholders said that code fragmentation—splitting existing billing codes into multiple codes for services that were previously bundled together—was contrary to the comprehensive patient-centered model of care that Medicare was moving towards. Specifically, one provider stated that under such a model, rather than billing multiple different codes for care planning and coordination, a primary care practice is paid a monthly management fee to (among other things) improve care coordination for patients who receive most of their primary care services from that practice. While six of the stakeholders we interviewed did not support creating a new LCCP code, the remaining 13 stakeholders told us that such a billing code is needed. According to the stakeholders, a new LCCP code as defined in the 2018 BBA could address several concerns they identified in Medicare’s existing billing codes related to the provision of the LCCP service. However, some of these concerns could be addressed under the current Medicare billing framework, as shown by our analysis of available data. For example, stakeholders identified the following limitations that could be addressed by a new LCCP code: Inadequate reimbursement for time spent on interdisciplinary care: The 13 stakeholders stated that Medicare’s existing billing codes either did not require or did not sufficiently reimburse them for the time spent on interdisciplinary care. They stated there should be a separate code to reimburse this type of care. However, stakeholders representing two specialties told us they had proposed such a code to the AMA but the AMA had rejected their proposals because interdisciplinary care is already accounted for in the existing billing codes. Moreover, as our analysis of the 58 billing codes shows, the majority of these codes include a provision for consultation and coordination among providers that is equivalent to input from an interdisciplinary team. With regard to inadequate reimbursement, as another stakeholder noted, providers may bill a complex E/M service along with a narrowly-defined LCCP-type service such as CCM. The total reimbursement for such a combination of codes would be about $203 as of 2019. (See table 2.) Insufficient physician time for care planning: Six stakeholders representing a mix of primary care and medical specialties stated the existing billing codes (including the more complex E/M codes) had insufficient physician time to provide both care planning and care management, which they maintained are separate and distinct activities. They stated a new code could include the appropriate time needed. One stakeholder said that care planning requires at least 30 minutes of time, and the complex E/M codes do not allow providers to bill for the time it takes to provide both the care management of a complex patient as well as care planning for the patient. According to the stakeholder, for example, if a provider bills a complex E/M code that allows for 40 minutes of physician time, that is insufficient to provide both types of services. While CMS has recently established new prolonged E/M codes (which allow for an additional 60 minutes of time), the stakeholder noted that they do not address the problem of insufficient time because a prolonged E/M code may only be billed with a companion E/M code, and the threshold of time needed to bill the two codes together is now too high—specifically 40 minutes for the complex E/M code plus 60 minutes for the prolonged E/M code. However, our review of CMS guidance on billing of prolonged E/M codes shows that providers do not have to meet the full 60 minutes of time in order to bill a prolonged E/M code; they may bill it as long as the total time spent on the visit exceeds the typical time for the E/M visit plus 30 minutes. Documentation requirements: Three stakeholders, largely representing primary care and medical specialties, stated that burdensome documentation requirements for the more complex E/M codes hampered their ability to bill these codes. They suggested that a new billing code could be structured similar to the new ACP or CCM codes which do not have the same documentation requirements. While these stakeholders expressed concern regarding documentation as a discouraging factor, our analysis of 2017 Medicare claims data showed that certain specialties, including some that had expressed this concern, billed the more complex codes at a significantly higher rate than the average across all specialties. This may indicate that these documentation requirements do not necessarily preclude providers from billing these codes. For example, 83 percent of all the E/M new patient visits billed by geriatricians in 2017 were billed using the more complex E/M codes, compared to 48 percent on average. Similarly, 80 percent of all the E/M established patient visits billed by clinical psychologists in 2017 were billed using the more complex E/M codes compared to 50 percent on average. See appendix V for details on billing patterns for all medical specialties. Inability of non-physician staff to independently bill for care planning: Seven stakeholders expressed concerns that non-physician staff such as nurses and social workers cannot independently bill the existing Medicare billing codes that we identified as being LCCP-type services. As one stakeholder explained, non-physician staff may spend time providing coordination and care planning services separately rather than concurrently with the physician, but they cannot bill for this time independently because the physician was not present. These stakeholders stated that a new LCCP code that could be billed by physicians and non-physicians that participated in the care planning process could address this issue. However, other stakeholders expressed concerns about the effect on Medicare spending if multiple providers billed for an LCCP service. Moreover, reimbursement for non-physicians is built into Medicare fees. Specifically, Medicare’s fee for each billing code includes reimbursement for physician’s time as well as their practice expenses (which cover the costs of non-physician staff), and when the AMA panel develops resources estimates for each billing code (upon which Medicare fees are based), it considers the amount of non-physician time spent on that code. Certain non-physician practitioners, such as nurse practitioners and physician assistants, may also independently bill services under the Medicare Physician Fee Schedule subject to certain requirements, and as specified in their scope of practice under state law. Stakeholders generally concurred that if a new LCCP code were implemented, the definition of interdisciplinary care should be flexible and not require a social worker. Currently, the LCCP billing code as defined in the 2018 BBA requires that the interdisciplinary team providing care planning services include a social worker. However, 13 stakeholders stated that a typical practice did not include a social worker, but rather included a nurse who might perform the functions of a social worker. They stated that smaller office-based medical practices could not afford to hire a social worker. The stakeholders concurred that social workers were generally available in larger integrated practices (such as a single or multiple groups aligning with each other or with a larger hospital system) and in facility settings such as hospitals or skilled nursing facilities. (Stakeholders also provided other comments on the structure of a potential new billing code for the LCCP service should such a code be established by CMS, which we summarize in app. VI.) We provided a draft of this report to HHS for review and comment. HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or farbj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII. Appendix II: Longitudinal Comprehensive Care Planning Service Components in the Balanced Budget Act of 2018 Pub. L. No. 115-123, § 50342(c)(4), 132 Stat. 211. We identified 58 billing codes in Medicare’s physician fee schedule that may be used to bill for LCCP-type services as defined in the Balanced Budget Act of 2018 (2018 BBA). Figure 9 shows relevant information on these billing codes including the short descriptor, beneficiary eligibility criteria, our analysis of whether the billing code’s components are equivalent to the components of an LCCP service as defined in the 2018 BBA, and Medicare’s 2019 fee. Medicare’s physician fee schedule contains evaluation and management (E/M) codes that providers may use to bill for face-to-face visits in their offices or other settings such as hospitals. These codes range in complexity from low to high depending on the amount of time the provider spends with a patient as well as the complexity of the medical decision- making and the medical condition(s) being treated. Table 4 shows the percentage of each specialty’s E/M visits that were billed as complex visits (moderate or high complexity). In general, primary care and medical sub-specialties tended to bill complex visits at a higher rate than the all- specialty average, while surgical specialties tended to bill complex visits at a lower rate than the all-specialty average. We interviewed 19 stakeholders including national umbrella groups of physicians and other providers to obtain their perspectives on the structure of a new billing code for LCCP-type services as defined in the Balanced Budget Act of 2018 (2018 BBA), regardless of whether they supported the creation of a new code. Stakeholders were generally in agreement that a new billing code for LCCP-type services as defined in the 2018 BBA, if implemented, should be broadly defined. Specifically, stakeholders stated that it should not be tied to a specific condition, should allow for both in-person and non-face-to-face services performed when the beneficiary was not present, should be billable more than once, should be available for billing by both primary care physicians and specialists, and should have restrictions to avoid duplicative billing with existing billing that provided overlapping services. However, stakeholders had more mixed views about what these specific restrictions should be. Stakeholder views about the various structural components or restrictions included the following: Applicable medical conditions: The majority of stakeholders (13 of the 17 who responded to this question) stated that the new code should be broadly defined although they differed in their opinions of what broadly-defined meant; one stakeholder cautioned against an overly broad definition, and one suggested pilot testing with a discrete list of conditions. Of the 13 stakeholders in favor of a broad definition, 12 stated that the new billing code should not be tied to any particular specific illness or medical condition but should be flexible in structure. Three stakeholders stated that the extent of beneficiaries’ daily functioning or quality of life should also be considered when defining applicable medical conditions. For example, a beneficiary who is not necessarily suffering from a life-threatening illness but is unable to perform the functions of daily living (such as bathing and eating) needs extensive care planning and should therefore be covered under the new LCCP-type service. Three stakeholders stated that the new code should be billable if a beneficiary’s existing diagnosis of a serious illness changed. Three stakeholders stated that a potential new code could be modeled along the lines of existing billing codes—specifically the advance care planning (ACP) or chronic care management (CCM) codes— which do not specify any particular medical condition. Two stakeholders said that a new code should not be so broad that it could apply to a vast majority of beneficiaries. For example, one stakeholder stated that the American Medical Association would likely not approve a code for a generic serious condition because it would be difficult to differentiate that code from an existing billing code, such as an evaluation and management (E/M) code, which may be used for any medical condition including serious, life-threatening conditions. One stakeholder suggested pilot testing the code with a discrete list of conditions, with the intention of expanding the list afterwards. In-person or non-face-to-face: The majority of stakeholders (12 of the 15 that responded to this question) stated that a potential new code should allow for both in-person and non-face-to-face activities (such as virtual or telehealth—providing clinical care remotely by two-way video, phone calls with the beneficiary or to arrange referrals or coordinate care with other providers when the beneficiary was not present); three said it should only include face-to-face activities. Twelve stakeholders stated that the visit should include both types of activities. For example, one stakeholder said the initial visit for LCCP-type services should be in-person, and follow up activities such as updating a care plan or remote patient monitoring (monitoring of patients outside of conventional settings) could be non-face-to-face. Three stakeholders said it should only include face-to-face activities either because of concerns about the potential for overbilling if the new code included non-face-to-face activities which might be difficult to verify or because other existing codes, such as CCM, already cover non-face-to-face activities. Frequency of billing: All of the 16 stakeholders responding to this question concurred that the new code should be billable more frequently than on a one-time basis, although opinions varied on the exact frequency. Nine stakeholders said the code should be billable on an ongoing basis as the beneficiary’s condition changes. For example, one said that the new code should be on-going because the care planning and treatment would continue to evolve over time as the beneficiary’s condition changes. Seven other stakeholders said that while it should not be an ongoing service, it should be billable more frequently than once. For example, one stakeholder specified that it could be billed once per month or over every three months, but that a target end date must be specified; otherwise, it would be too similar to existing billing codes such as the CCM code that may be billed monthly. Two other stakeholders said it could be billed up to 4-5 times a year. Other billing restrictions: All 12 stakeholders responding to this question indicated that restrictions would be necessary to avoid overlap with existing billing codes. For example, three stakeholders suggested that the new code could be billed along with an E/M code for additional services not covered by the E/M code as long as it does not overlap with other existing codes that account for additional time beyond an E/M visit (such as the prolonged E/M visit billing codes). One suggested that it should not be billed along with any of the existing narrowly-defined LCCP-type codes, including CCM, transitional care management, or the ACP codes. One did not specify any particular code with which the new code should not be billed, but cautioned that care should be taken to ensure that time spent with the beneficiary was reported only once. Providers eligible to bill the code: The majority of stakeholders (13 of the 15 that responded to this question) stated that both primary care physicians and specialists should be eligible to bill the new code. Two of these stakeholders said that there should also be a requirement that the billing physician has an established relationship with the beneficiary. Two stakeholders said that only specialists should bill since they are generally the ones attending to the beneficiary’s serious illness. Two stakeholders stated that non-physicians (including social workers) should also be able to bill the code as long as they are currently allowed to bill separately under the Medicare Physician Fee Schedule. In addition to the contact named above, Karen Doran, Assistant Director; Iola D’Souza, Analyst-in-Charge; Sarah Belford; Krister Friday; John Lalomio; and Daniel Ries made key contributions to this report. Also contributing were George Bogart and Muriel Brown.", "summary": "Medicare's physician fee schedule contains over 8,000 billing codes for office visits, surgical procedures, or other services provided to beneficiaries. Some provider groups have concerns that these codes do not sufficiently account for the LCCP-type services they provide to Medicare beneficiaries with complex medical needs. The BBA included a provision that GAO examine billing codes that may be used for LCCP-type services for beneficiaries with a serious or life-threatening illness. GAO identified, among other things, (1) existing Medicare physician fee schedule billing codes that can be used to bill LCCP-type services; and (2) trends in Medicare spending on these services from 2013 through 2017. GAO reviewed Centers for Medicare & Medicaid Services (CMS) billing code manuals and American Medical Association (AMA) code descriptors to identify existing codes containing key components of LCCP-type services; analyzed Medicare Part B claims data from 2013 to 2017 (the most recent available at the time of GAO's review); and interviewed officials from CMS and 19 stakeholders, including the AMA, national physician groups, and other provider groups that had previously given input on the topic to Congress. GAO provided a draft of this report to the Department of Health and Human Services (HHS). In response, HHS provided technical comments, which GAO incorporated as appropriate. The 2018 Bipartisan Budget Act (BBA) defined longitudinal comprehensive care planning (LCCP) as services involving an interdisciplinary team of providers who develop and communicate a care plan to Medicare beneficiaries diagnosed with a serious or life-threatening illness. GAO identified at least 58 billing codes in Medicare's physician fee schedule that could be used by providers to bill for services that cover some or all of the LCCP service components as defined in the 2018 BBA —referred to by GAO as LCCP-type services. The 58 billing codes may be used individually or in combination, depending on a beneficiary's medical needs. Stakeholders representing providers told GAO their members generally use one or a combination of these codes to bill for LCCP-type services. Forty-five of the 58 codes are broadly-defined longstanding codes that can be used for LCCP-type services as well as other services such as the treatment of a specific medical complaint. The remaining 13 codes are more recent narrowly-defined codes introduced starting in 2013 that only cover LCCP-type services. They include transitional care management services introduced in 2013, chronic care management starting in 2015, advance care planning in 2016, and behavioral health integration in 2017. GAO found that overall Medicare spending on LCCP-type services that were billed to the 58 codes increased from $26 billion in 2013 to almost $29 billion in 2017. While narrowly-defined services accounted for a small share of this total spending ($467 million in 2017), spending on these narrowly-defined services such as chronic care management increased rapidly. Moreover, spending growth on narrowly-defined services was driven by increased use of these services rather than increases in reimbursement rates. From 2013 through 2017, more beneficiaries received and more providers billed for narrowly-defined services. The number of Medicare beneficiaries receiving these services grew from about 267,000 to about 2.5 million. The number of providers billing these services grew from about 31,000 to about 100,000.", "document_type": "gao"}
{"report": "FTC and, most recently, CFPB, are the federal agencies primarily responsible for overseeing CRAs. FTC has authority to investigate most organizations that maintain consumer data and to bring enforcement actions for violations of statutes and regulations that concern the security of data and consumer information. CFPB, created in 2010 by the Dodd- Frank Wall Street Reform and Consumer Protection Act (Dodd-Frank Act), has enforcement authority over all CRAs for violations of certain consumer financial protection laws. In general, it also has the authority to issue regulations and guidance for those laws. CFPB has supervisory authority over larger market participants in the consumer reporting market. In 2012, CFPB defined larger market participant CRAs as those with more than $7 million in annual receipts from consumer reporting. CFPB’s supervision of these companies includes monitoring, inspecting, and examining them for compliance with the requirements of certain federal consumer financial laws and regulations. As discussed below, these laws include most provisions of the Fair Credit Reporting Act (FCRA); several provisions of the Gramm-Leach-Bliley Act (GLBA); and provisions of the Dodd-Frank Act concerning unfair, deceptive, or abusive acts or practices. Although there is no commonly agreed-upon definition of “data breach,” the term generally refers to an unauthorized or unintentional exposure, disclosure, or loss of sensitive information. This information can include personally identifiable information such as Social Security numbers, or financial information such as credit card numbers. A data breach can be inadvertent, such as from the loss of an electronic device; or deliberate, such as the theft of a device or a cyber-based attack by individuals or groups, including an organization’s own employees, foreign nationals, or terrorists. Data breaches have occurred at all types of organizations, including private, nonprofit, and federal and state entities. In the Equifax data breach, Equifax system administrators discovered on July 29, 2017, that intruders had gained unauthorized access via the Internet to a server housing the company’s online dispute portal. The breach compromised the personally identifiable information of at least 145.5 million individuals, and included names, addresses, and birth dates; and credit card, driver’s license, and Social Security numbers. Equifax’s investigation of the breach identified the following factors that led to the breach: software vulnerabilities, failure to detect malicious traffic, failure to isolate databases from each other, and inadequately limiting access to sensitive information such as usernames and passwords. Equifax’s public filings after the breach noted that the company took steps to improve security and notify individuals about the breach. Our August 2018 report provides more information on the breach and Equifax’s response. FTC enforces compliance with consumer protection laws under authorities provided in FCRA, GLBA, and the FTC Act. As we reported in February 2019, according to FTC, in the last 10 years, it has brought 34 enforcement actions for FCRA violations, including 17 against CRAs. In addition, FTC said that it has taken 66 actions against companies (not just in the last 10 years), including CRAs, that allegedly engaged in unfair or deceptive practices relating to data protection. In some circumstances, FTC enforcement authority can include civil money penalties—monetary fines imposed for a violation of a statute or regulation. However, FTC’s civil penalty authority does not extend to initial violations of GLBA’s privacy and safeguarding provisions. These provisions require administrative, physical, and technical safeguards with an emphasis on protection against anticipated threats and unauthorized access to customer records. For violations of GLBA provisions, FTC may seek an injunction to stop a company from violating these provisions and may seek redress (damages to compensate consumers for losses) or disgorgement (requirement for wrongdoers to give up profits or other gains illegally obtained). Determining the appropriate amount of consumer compensation requires FTC to identify the consumers affected and the amount of monetary harm they suffered. In cases involving security or privacy violations resulting from data breaches, assessing monetary harm can be difficult. In addition, consumers may not be aware that their identities have been stolen as a result of a breach and or identity theft, and related harm may occur years in the future. It can also be difficult to trace instances of identity theft to specific data breaches. According to FTC staff, these factors can make it difficult for the agency to identify which individuals were victimized as a result of a particular breach and to what extent they were harmed and then obtain related restitution or disgorgement. Having civil penalty authority for GLBA provisions would allow FTC to fine a company for a violation such as a data breach without needing to prove the monetary harm to individual consumers. FTC staff noted that in the case of a data breach, each consumer record exposed could constitute a violation; as a result, a data breach that involved a large number of consumer records could result in substantial fines. In 2006, we suggested that Congress consider providing FTC with civil penalty authority for its enforcement of GLBA’s privacy and safeguarding provisions. We noted that this authority would give FTC a practical tool to more effectively enforce provisions related to security of data and consumer information. Following the 2008 financial crisis, Congress introduced several bills related to data protection and identity theft, which included giving FTC civil penalty authority for its enforcement of GLBA. However, in the final adoption of these laws, Congress did not provide FTC with this authority. Since that time, data breaches at Equifax and other large organizations have highlighted the need to better protect sensitive personal information. Accordingly, we continue to believe FTC and consumers would benefit if FTC had such authority, and we recommended in our February 2019 report that Congress consider providing FTC with civil penalty authority for the privacy and safeguarding provisions of GLBA to help ensure that the agency has the tools it needs to most effectively act against data privacy and security violations. CFPB enforces compliance with most provisions of FCRA; several provisions of GLBA; and the prohibition of unfair, deceptive, or abusive acts or practices under the Dodd-Frank Act. In our February 2019 report, we noted that since 2015, CFPB has had five public settlements with CRAs. Four of these settlements included alleged violations of FCRA, and three included alleged violations of provisions related to unfair, deceptive, or abusive practices. CFPB also has an ongoing investigation of Equifax’s data breach. Under its existing authority, CFPB has examined several larger market participant CRAs, but may not be identifying all CRAs that meet the $7 million threshold. CFPB staff told us that as of October 2018, they were tracking between 10 and 15 CRAs that might qualify as larger market participants. CFPB staff told us that they believe the CRA market is highly concentrated and there were not likely to be many larger market participants beyond the 10 to 15 they are tracking. However, CFPB staff said that the 10 to 15 CRAs may not comprise the entirety of larger market participants, because CRAs’ receipts form consumer reporting may vary from year to year, and CFPB has limited data to determine whether CRAs meet the threshold. Our January 2009 report on reforming the U.S. financial regulatory structure noted that regulators should be able to identify institutions and products that pose risks to the financial system, and monitor similar institutions consistently. CFPB could identify CRAs that meet the larger market participant threshold by requiring such businesses to register with it, subject to a rulemaking process and cost-benefit analysis of the burden it could impose on the industry. Another method CFPB could use to identify CRAs subject to its oversight would be to leverage information collected by states. We recommended in February 2019 that CFPB identify additional sources of information, such as through registering CRAs or leveraging state information, that would help ensure the agency is tracking all CRAs subject to its authority. CFPB neither agreed nor disagreed with our recommendation. Each year CFPB determines the institutions (for example, banks, credit unions, non-bank mortgage servicers, and CRAs) and the consumer product lines that pose the greatest risk to consumers, and prioritizes these for examinations. CFPB segments the consumer product market into institution product lines, or specific institutions’ offerings of consumer product lines. CFPB then assesses each institution product line’s risk to consumers at the market level and institutional level. To assess risk at the market level, CFPB considers market size and other factors that contribute to market risk. To assess risk at the institution level, CFPB considers an institution’s market share within a product line, as well as field and market intelligence. Field and market intelligence includes quantitative and qualitative information on an institution’s operations for a given product line, including the strength of its compliance management systems, the number of regulatory actions directed at the institution, findings from prior CFPB examinations, and the number and severity of consumer complaints CFPB has received about the institution. CFPB then determines specific areas of compliance to assess by considering sources such as consumer complaints, public filings and reports, and past examination findings related to the same or similar products or institutions. Most recently, CFPB examinations of CRA’s consumer reporting have focused on issues such as data accuracy, dispute processes, compliance management, and permissible purposes. Although CFPB’s examination prioritization incorporates several important factors and sources, the process does not routinely include assessments of data security risk, such as how institutions detect and respond to cyber threats. CFPB staff said the bureau cannot examine for or enforce compliance with the data security standards in provisions of GLBA and FCRA or FTC’s implementing rules, even at larger participant CRAs. After the Equifax breach, however, CFPB used its existing supervisory authority to develop internal guidelines for examining data security, and conducted some CRA data security examinations. CFPB staff said that they do not routinely consider data security risks during their examination prioritization process and have not reassessed the process to determine how to incorporate such risks going forward. Statute requires CFPB to consider risks posed to consumers in the relevant product and geographic markets in its risk-based supervision program. In addition, federal internal control standards state that agencies should identify, analyze, and respond to risks related to achieving defined objectives. This can entail considering all significant internal and external factors to identify risks and their significance, including magnitude of impact, likelihood of occurrence, nature of the risk, and appropriate response. In light of the Equifax breach, as well as CFPB’s acknowledgment of the CRA market as a higher-risk market for consumers, it is important for CFPB to routinely consider factors that could inform the extent of CRA data security risk such as the number of consumers that could be affected by a data security incident and the nature of potential harm resulting from the loss or exposure of information. In our February 2019 report, we recommended that CFPB assess whether its process for prioritizing CRA examinations sufficiently incorporates the data security risks CRAs pose to consumers, and take any needed steps identified by the assessment to more sufficiently incorporate these risks. CFPB neither agreed nor disagreed with our recommendation. In our February 2019 report, we noted that FTC and CFPB provide educational information for consumers on ways to mitigate the risk of identity theft. In addition, after a breach, FTC and CFPB publish information specific to that breach. For example, shortly after Equifax’s announcement of the breach, FTC published information on when the breach occurred, the types of data compromised, and links to additional information on Equifax’s website. Similarly, CFPB released three blog posts and several social media posts that included information on ways that consumers could protect themselves in the wake of the breach and special protections and actions for service members. At any time, consumers can take actions to help mitigate the risk of identity theft. For example, consumers can implement a credit freeze free of charge, which can help prevent new-account fraud by restricting potential creditors from accessing the consumer’s credit report. Similarly, implementing a free fraud alert with a credit bureau can help prevent fraud because it requires a business to verify a consumer’s identity before issuing credit. However, consumers are limited in the direct actions they can take against a CRA in the event of a data breach, for two primary reasons. First, consumers generally cannot determine the source of the data used to commit identity theft. As a result, it can be difficult to link a breach by a CRA (or any other entity) to the harm a consumer suffers from a particular incidence of identity theft, which makes it challenging to prevail in a legal action. Second, unlike with many other products and services, consumers generally cannot exercise choice if they are dissatisfied with a CRA’s privacy or security practices. Specifically, consumers cannot choose which CRAs maintain information on them. In addition, consumers do not have a legal right to delete their records with CRAs, according to CFPB staff, and therefore cannot choose to remove themselves entirely from the CRA market. FTC and CFPB have noted that the level of consumer protection required can depend on the consumer’s ability to exercise choice in a marketplace. For example, when determining whether a practice constitutes an unfair practice, FTC considers whether the practice is one that consumers could choose to avoid. Similarly, according to CFPB staff, the consumer reporting market may pose higher risk to consumers because consumers cannot choose whether or which CRAs possess and sell their information. Chairman Krishnamoorthi, Ranking Member Cloud, and Members of the Subcommittee, this concludes my prepared remarks. I would be happy to answer any questions that you may have. If you or your staff have any questions about this statement, please contact Michael Clements at (202) 512-8678 or clementsm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contact named above, John Forrester (Assistant Director), Winnie Tsen (Analyst-in-Charge), and Rachel Siegel made key contributions to the testimony. Other staff who made key contributions to the report cited in the testimony are identified in the source product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "CRAs collect, maintain, and sell to third parties large amounts of sensitive data about consumers, including Social Security numbers and credit card numbers. Businesses and other entities commonly use these data to determine eligibility for credit, employment, and insurance. In 2017, Equifax, one of the largest CRAs, experienced a breach that compromised the records of at least 145.5 million consumers. This statement is based on GAO's February 2019 report on the CRA oversight roles of FTC and CFPB. This statement summarizes (1) measures FTC has taken to enforce CRA compliance with requirements to protect consumer information, (2) measures CFPB has taken to ensure CRA protection of consumer information, and (3) actions consumers can take after a breach. In its February 2019 report, GAO found that since 2008, the Federal Trade Commission (FTC) has settled 34 enforcement actions against various entities related to consumer reporting violations of the Fair Credit Reporting Act (FCRA), including 17 actions against consumer reporting agencies (CRA). Some of these settlements included civil penalties—fines for wrongdoing that do not require proof of harm—for FCRA violations or violations of consent orders. However, FTC does not have civil penalty authority for violations of requirements under the Gramm-Leach-Bliley Act (GLBA), which, unlike FCRA, includes a provision directing federal regulators and FTC to establish standards for financial institutions to protect against any anticipated threats or hazards to the security of customer records. To obtain monetary redress for these violations, FTC must identify affected consumers and any monetary harm they may have experienced. However, harm resulting from privacy and security violations can be difficult to measure and can occur years in the future, making it difficult to trace a particular harm to a specific breach. As a result, FTC lacks a practical enforcement tool for imposing civil money penalties that could help to deter companies, including CRAs, from violating data security provisions of GLBA and its implementing regulations. Since 2015, the Consumer Financial Protection Bureau (CFPB) has had five public settlements with CRAs. Four of these settlements included alleged violations of FCRA; and three included alleged violations of unfair, deceptive, or abusive practices provisions. CFPB is also responsible for supervising larger CRAs (those with more than $7 million in annual receipts from consumer reporting) but lacks the data needed to ensure identification of all CRAs that meet this threshold. Identifying additional sources of information on these CRAs, such as by requiring them to register with the agency through a rulemaking or leveraging state registration information, could help CFPB ensure that it can comprehensively carry out its supervisory responsibilities. After the Equifax breach, CFPB used its existing supervisory authority to examine the data security of certain CRAs. CFPB's process for prioritizing which CRAs to examine does not routinely include an assessment of companies' data security risks, but doing so could help CFPB better detect such risks and prevent the further exposure or compromise of consumer information. Consumers can take actions to mitigate the risk of identity theft—such as implementing a fraud alert or credit freeze—and can file a complaint with FTC or CFPB. However, consumers are limited in the direct actions they can take against CRAs. Consumers generally cannot exercise choice in the consumer reporting market—such as by choosing which CRAs maintain their information—if they are dissatisfied with a CRA's privacy or security practices. In addition, according to CFPB, consumers cannot remove themselves from the consumer reporting market entirely. In its February 2019 report, GAO recommended that Congress consider giving FTC civil penalty authority to enforce GLBA's safeguarding provisions. GAO also recommended that CFPB (1) identify additional sources of information on larger CRAs, and (2) reassess its prioritization of examinations to address CRA data security. CFPB neither agreed nor disagreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Individual Taxpayer Identification Number (ITIN) An ITIN is a tax processing number issued by IRS to individuals who are required to have a U.S. taxpayer identification number but who do not have and are not eligible to obtain a Social Security number from the Social Security Administration. IRS issues ITINs to help individuals comply with the U.S. tax laws, and to provide a means to efficiently process and account for tax returns and payments for those not eligible for Social Security numbers. Taxpayers may first realize they are victims of employment-related identity fraud when IRS notifies them of discrepancies in the reporting of income earned using their names and SSNs. After filing deadlines have passed, IRS’s Nonfiler and Automated Underreporter (AUR) programs use W-2 information to identify and follow up with taxpayers who appear to owe taxes but either have not filed returns (Nonfiler) or have filed returns but underreported earnings (AUR). Other taxpayers may become aware that their SSNs were used by other people when IRS sends them an Employment-Related Identity Theft (CP01E) notice. IRS sends these notices to taxpayers whose SSNs appear on W-2s that have been attached to tax returns (Forms 1040, U.S. Individual Income Tax Return) that were filed with Individual Taxpayer Identification Numbers (ITIN) (see sidebar). In these cases, IRS marks the taxpayer accounts with an employment-related identity theft indicator. Victims may also notice wages they did not earn appearing on their Social Security earnings record or may be alerted by SSA that their Supplemental Security Income benefits are being reduced or eliminated because of wages earned by someone else using their SSN. The following individuals and agencies are involved in verifying individuals’ eligibility for employment, in processing wage information, or in monitoring identity fraud cases. Employer: Employers are required to complete the Form I-9, Employment Eligibility Verification for new hires. As part of completing the form, employers certify that they have examined documentation demonstrating that new hires are who they say they are, are eligible for employment, and that the documentation appears to be genuine. The employer is required to submit a W-2 to each employee as well as SSA by January 31 each year. Employee: As part of obtaining employment, the employee provides the employer with documentation to authenticate his or her identity. It is at this point that the employee could provide someone else’s SSN or other information. DHS: DHS manages E-Verify, a free, internet-based system that employers can use to verify employees’ employment eligibility. SSA supports DHS in this effort. Federal agencies are required to use E- Verify for federal employees and contractors. Some states also require employers to use E-Verify to verify the eligibility of some or all employees or contracts. According to DHS, by the end of fiscal year 2019, more than 890,000 employers were enrolled in E-Verify. SSA: SSA receives W-2s from employers and uses this information to update earnings records and to make determinations about benefits. After receiving and processing W-2s, SSA sends the W-2 information to IRS as part of the Combined Annual Wage Reporting (CAWR) process. SSA also maintains the Social Security Number Verification Service, a free SSN verification program that registered employers can use to verify that employee names and SSNs match SSA’s records before they submit W-2s to SSA. IRS: IRS uses W-2 information to verify tax return information, such as wages, withholdings, and Employer Identification Numbers (EIN), and to enforce tax law. IRS has legal authority to penalize employers $250 for each inaccurate W-2 they submit up to a maximum of $3 million in total penalties per year. In 2013, the SSA OIG reported that IRS does not routinely penalize employers who consistently submit erroneous or inaccurate wage information. Federal Trade Commission: It collects and reports to the public aggregated data from self-reported victims of identity fraud. Victims can visit www.IdentityTheft.gov to report identity theft and access resources. Our analysis shows that millions of SSNs in NDNH data exhibited risk characteristics associated with employment-related identity fraud in tax year 2016. More than a million of those were also at risk of not meeting all IRS tax return requirements, such as reporting all associated W-2s. However, IRS did not identify all of those noncompliant returns. Further, employment-related identity fraud can diminish tax revenues. IRS’s method for tracking employment-related identity fraud likely understates the extent of the problem. We identified more than 2.9 million SSNs that had risk characteristics associated with SSN misuse, and had evidence of employment activity based on our analysis of NDNH verified quarterly wage records for 122.8 million individuals from August to October 2016. The risk characteristics included: Individuals who had wages reported for three or more employers in the same quarter; Individuals who were deceased; Individuals under age 14; and Individuals over age 84 (see table 1). We previously reported that the existence of three or more wage records in the same time frame for the same individual indicates possible SSN misuse, which could include employment-related identity fraud. We also previously reported, along with the Department of Justice and SSA OIG, that deceased persons, children, and elderly populations are at risk of identity theft (IDT). Fraudsters may target these groups because they believe there is a lower chance the SSNs are being used for legitimate employment. Individuals with three or more employers within the same quarter. Our analysis of NDNH data identified millions of SSNs with three or more wage records from August to October 2016. Specifically, of the 122.8 million SSNs included in the data, we found 2.8 million with three or more wage records in the same quarter. Further, we found almost 10,000 of those SSNs had wages reported by 10 or more employers in the same quarter. It is not uncommon for individuals to have second jobs or to change employers. However, when wages are reported by three or more employers for the same calendar quarter, it can indicate potential misuse of an SSN (see table 2). As an illustrative example of potential SSN misuse, one SSN had wages reported by 15 employers from 14 different states for a 3-month period in 2016 (see figure 1). According to the wage data, on average, each of these employers was paying the employee approximately $26,900 a year. Deceased individuals. We identified several thousand SSNs for deceased individuals included in the NDNH data. Specifically, the NDNH data August-October 2016 showed 13,600 SSNs for individuals SSA identified as deceased prior to May 2016. Of these, 8,400 are reported to have died before 2014. In some cases, we found individuals who had been deceased for a decade. Children. We identified tens of thousands of SSNs for children under the age of 14. Specifically, NDNH data included 33,856 SSNs of individuals who, according to SSA data, were under the age of 14 with earned income reported. One reason children can be at risk of long- term victimization of employment-related identity fraud is because it usually takes children a while before they start working or applying for financial credit. This gives a fraudster ample opportunity to exploit their stolen identities. Still, there are legitimate circumstances for children to be earning wages, such as in the entertainment and advertising industries. Elderly. We identified tens of thousands of wage records from elderly individuals. Specifically, the 2016 NDNH data included 65,823 SSNs with earned income reported that SSA data identified as being over 84. The Federal Trade Commission reported that in 2016, approximately one-fifth of IDT complaints they received involved people age 60 years or older. Further, the elderly have low participation rates in the workforce. The Bureau of Labor Statistics reported that, in 2016, the workforce participation rate for those ages 75 and above was 8.4 percent, compared to a rate of 62.8 percent for the overall workforce. Some SSNs with risk characteristics were sometimes also associated with IRS returns that did not include required W-2 forms. Specifically, more than 1.3 million individuals—of the 2.9 million SSNs we determined to have risk characteristics associated with SSN misuse—had at least one wage record they did not report to IRS. Of these 1.3 million individuals, more than half failed to include at least one W-2 on their tax return, and slightly less than half (43 percent) did not include any W-2s in a tax return (see table 3). IRS has enforcement tools that are intended to detect reporting deficiencies, but these tools did not always detect the reporting issues we identified. IRS can use Automated Underreporter (AUR) and the Nonfiler, as well as seven IDT-related indicators to mark a taxpayer’s account or W-2 if it has determined that the SSN was compromised (see sidebar). We compared data from these enforcement tools and IDT indicators to the 1.3 million individuals identified above and found that IRS did not mark all accounts or W-2s. Action Code 501: closed identity theft cases initiated by a taxpayer. Action Code 506: closed identity theft cases initiated by IRS. Action Code 524: deceased taxpayer. It prevents the use of a deceased taxpayer's identity on a federal income tax return. Action Code 525: mismatch between the identity listed on the W-2 and on the tax return. These are cases where returns filed with an Individual Taxpayer Identification Number include a W-2 with an SSN belonging to another person. Individuals with three or more W-2s for the same period. More than a million individuals with three or more wage records did not declare at least one W-2. Additionally, we found that, in general, the more W-2s an individual had, the less likely it was that all of them would be reported to IRS (see figure 2). For instance, individuals with three W-2s declared all of them 68 percent of time, while individuals with seven declared all of them 29 percent of the time. Using its enforcement tools, IRS identified some of these individuals with three or more W-2s. Of the 1.25 million individuals in our analysis with three or more wage records who did not include all W-2s in tax year 2016, about 600,000 had wages totaling more than $23,200, meaning that they were required to file a tax return. Of these, about 340,000 individuals had at least one of the seven IDT-related indicators or were pursued through AUR or Nonfiler. In addition, IRS pursued—with AUR or Nonfiler—about half of the nearly 100 individuals who had 50 or more W- 2s reported by employers for 2016. In addition, approximately 9,000 individuals with wages totaling more than $23,200 and that did not include all W-2s in tax year 2016 also lived in five or more states (see figure 3 for an illustrative example). Deceased individuals. IRS did not apply IDT-related indicators to some of the accounts of deceased individuals we identified as having employer-reported wages not included on a tax return. Out of the 11,573 deceased individuals who reported earned income, we identified nearly 2,627 who earned at least $23,200, a threshold requiring the filing of a tax return. Of these, about 2,441 had at least one of the seven IDT-related indicators or were pursued under IRS’s AUR or Nonfiler enforcement programs. However, there were still 186 individuals that IRS did not identify. Elderly. Out of the 19,460 elderly individuals who reported earned income, we identified nearly 3,800 who earned enough to be required to file a tax return. Of these, about 1,700 had at least one out of the seven IDT-related indicators on their account or were pursued under IRS’s AUR or Nonfiler enforcement programs. However, there were still about 2,100 individuals that IRS did not identify. Children. For tax year 2016, individuals under age 14 were only required to file taxes if they earned more than $7,850. However, nearly 1,900 met this filing threshold and failed to include at least one W-2 on their tax returns. Of these, nearly 1,000 had at least one of the seven IDT-related indicators applied to their account by IRS or were pursued under IRS’s AUR or Nonfiler enforcement programs. However, there were still about 900 individuals that IRS did not identify. In considering employment-related identity fraud, IRS focuses on only one of the seven IDT-related indicators. Specifically, IRS considers mismatches between the identity listed on the W-2 and the identity on the tax return as a type of employment-related identify fraud. IRS does not consider other characteristics, such as individuals with multiple wage records, in its checks for employment-related identity fraud. Doing so would require the development of new codes or the modifications of existing ones. According to the Fraud Risk Framework, two leading practices for managing fraud risks include (1) identifying specific tools, methods, and sources for gathering information; and (2) designing and implementing control activities such as data-analytics activities to prevent and detect fraud. IRS addressed these leading practices, in part, through the AUR program, Nonfiler program, and seven IDT-related indicators, but there were still individuals in the population we examined that IRS did not identify. By assessing and documenting the feasibility of incorporating additional checks—such as multiple wage records or wage records for children under 14—into its checks of employment-related identity fraud, IRS may be able to develop a method for identifying additional taxpayers at risk of this type of fraud. IRS officials stated that employment-related identity fraud has limited tax consequences, as employees will nonetheless pay required taxes— including federal, state, and payroll taxes—through payroll withholding even if the fraudster fails to file a tax return. However, we found that federal income tax withholding was lower for SSNs that did not declare all the W-2s than for SSNs with all W-2s reported (see table 4). Additionally, we found individuals who did not withhold any federal income taxes across all of their related W-2s in 2016. Specifically, 37,868 individuals had at least one W-2 not declared on a tax return and withheld no federal income tax over the course of the year. Together, these individuals earned approximately $340 million in 2016. Further, 18 W-2s that were not reported on a tax return showed wages earned of more than $100,000 yet had $0 of federal income tax withheld (see figure 4 for example). Of the indicators IRS uses to track IDT, the only action code that directly relates to employment is Action Code 525, “Employment-related Identity Theft.” IRS applies the code to a taxpayer’s account when IRS processes a return filed by an individual with an Individual Taxpayer Identification Number (ITIN), and the return includes a W-2 with an SSN that does not belong to the person identified on the ITIN return. IRS refers to this situation as an ITIN/SSN mismatch. In 2018, IRS marked 818,097 accounts with Action Code 525. IRS officials acknowledged that forms of employment-related identity fraud, other than that captured by Action Code 525, are likely, but they said they do not systematically track these situations for several reasons. First, unless a taxpayer contacts IRS to say he or she did not earn the wages and disclaims them, the agency does not know whether a suspected case is employment-related identity fraud or someone who may not have included legitimate wages on his or her tax return. Second, IRS may be unable to distinguish between employment-related identity fraud and fabricated W-2s for jobs that were not worked (i.e., fake employees of a fake business). Third, while our analysis shows that employment-related identity fraud may be a more widespread problem than the ITIN/SSN mismatch that IRS currently tracks, IRS officials told us that other types of employment-related identity fraud would be identified and addressed through processes the agency applies broadly to all taxpayers, such as the AUR or Nonfiler programs. For example, according to IRS officials, if IRS receives a fraudulent W-2 from an employer using a legitimate taxpayer’s SSN, AUR or the Nonfiler program will detect it as IRS matches W-2s with tax returns. However, our analysis of NDNH and IRS data described earlier in this report shows that there are potential cases that these IRS enforcement programs did not identify. Standards for Internal Control in the Federal Government states that management should use quality information that is appropriate and complete to achieve the entity’s objectives, and that it should communicate quality information externally. However, our analysis of SSNs at risk of employment-related identify fraud indicates that the count of cases that IRS identifies under Action Code 525 likely understates the universe of employment-related identity fraud. By modifying the title of its employment-related IDT action code to more accurately reflect the data covered by the code, IRS can ensure that the agency is appropriately conveying the risk this specific type of employment-related identity fraud poses both to victims and tax administration without suggesting its statistics cover other types of employment-related identity fraud. As illustrated in figure 5, SSA analyzes W-2s to detect inaccuracies. For W-2s determined to be accurate, SSA adds wages to the individual’s record on the Master Earnings File, a database that SSA uses to determine an individual’s eligibility for Social Security benefits and the amount of benefits paid. For W-2s determined to be inaccurate, SSA posts the wage information to the Earnings Suspense File. Inaccurate W- 2s may be attributable to various reasons, including employment-related identity fraud or administrative errors. SSA receives hundreds of millions of W-2s each year. SSA analyzes incoming W-2s to detect inaccuracies and adds inaccurate W-2s to the Earnings Suspense File. Based on SSA data from tax year 2016, SSA added millions of W-2s to the Earning Suspense File. On a daily basis, SSA electronically forwards IRS W-2s that it has analyzed, including both accurate and inaccurate W-2s. SSA monitors the effectiveness of its checks for inaccurate W-2s by testing its software prior to the filing season. Prior to each filing season, SSA creates test data that have characteristics of inaccurate W-2s. SSA then processes these data through the annual wage reporting software to ensure automated checks identify potentially inaccurate W-2s according to SSA’s criteria. SSA also has an electronic reporting system in place that SSA employees can use to identify and document problems for management throughout the year. SSA officials told us they have not identified any problems that have prevented checks from working as intended. This public report omits information that SSA has deemed sensitive related to (1) SSA’s efforts to improve W-2 accuracy checks, and (2) SSA’s challenges in addressing risks associated with employment-related identity fraud. SSA is taking steps to more effectively communicate to both victims and employers information on potentially inaccurate W-2s, including potential employment-related identity fraud W-2s. When SSA detects a potentially inaccurate W-2, SSA may send a letter to the employer or employee listed on the W-2 that notifies them of the potential inaccuracy. SSA first sends letters to employers. Responses can help SSA resolve inaccuracies by identifying correct wage earners. Responses can also support SSA’s efforts to provide taxpayers with correct benefits. SSA sends different letters to employees and employers depending on the type of potential inaccuracy detected: Mismatched name and SSN. In March 2019, SSA resumed sending Educational Correspondence (EDCOR) letters to employers who submitted W-2s electronically, notifying them of the number of W-2s they electronically submitted with mismatched names and SSNs. The letters request that employers use SSA’s Business Services Online portal to view specific names and SSNs that did not match and provide necessary Form W-2C corrections. According to SSA, EDCOR letters are meant to educate employers about mismatches and help SSA post wages to correct earnings records. SSA officials told us that SSA had mailed about 577,000 EDCOR letters for electronically submitted W-2s as of June 2019 since resuming the process. Officials said the agency also began sending EDCOR letters for W-2s submitted on paper beginning in October 2019. SSA previously sent EDCOR notices from 1994 through 2007, but SSA stopped sending these notices in response to litigation surrounding a proposed DHS regulation that would have required employers to follow a prescribed course of action upon learning of an employee name or SSN mismatch. DHS rescinded its proposed rule in October 2009. SSA officials told us the agency decided to resume sending EDCOR notices in 2019 because employers are using Business Services Online to file more W-2s electronically. Therefore, employers may be more familiar with the system used to submit W-2C corrections. SSA has taken action to improve the effectiveness of EDCOR letters since the letters were discontinued in 2007. In 2008, the SSA OIG reported that EDCOR letters were not effective in either communicating wage-reporting problems to employers or identifying correct wage earners. For example, the OIG found that 74 percent of employers who reported W-2s with mismatched names and SSNs did not receive letters. Most employers that did not receive letters submitted 10 or fewer mismatched W-2s whereas SSA only sent letters to employers that submitted more than 10 mismatched W-2s. SSA officials told us that EDCOR letters sent beginning in 2019 are sent to every employer who submits a W-2 with a mismatched name and SSN. Disclaimed wages. When an individual disclaims wages, SSA staff have the option of sending a letter to the employer who paid the wages to attempt to identify the wage earner. In 2008, the SSA OIG found that SSA seldom sent letters to employers, and recommended that SSA consider generating a standard, annual letter to each employer that submitted a W-2, which was later disclaimed. SSA officials told us that, as of May 2019, SSA staff in all SSA region offices routinely send letters to employers notifying them of disclaimed wages. SSA officials reported the agency sent 20,945 letters in fiscal year 2018. IRS uses relevant information to detect inaccurate W-2s, including potentially fraudulent W-2s, and makes this information available to relevant enforcement programs, including Nonfiler, which IRS uses to follow up with individuals who appear to owe taxes but have not filed. IRS detects inaccurate W-2s using the results of SSA’s annual wage reporting checks and its own efforts to reconcile and correct some inaccuracies. As part of this process, IRS receives Earnings Suspense File W-2s that have mismatched names and SSNs from SSA and attempts to locate the wage earner’s correct name and SSN. IRS does so by identifying previously filed tax returns that list the same address as the mismatched W-2s. IRS then compares the names and SSNs listed on W- 2s to those on the tax returns to identify accurate name and SSN combinations. Accurate and inaccurate W-2s are then made accessible to IRS enforcement programs, including Nonfiler. Nonfiler and other programs that support IRS’s efforts to collect taxes owed from wage earners, including potential employment fraudsters, also may deter fraudulent activity by reducing the likelihood fraudsters succeed in not paying taxes owed. In reviewing IRS actions that may help deter employment-related identity fraud, we found that Nonfiler uses W-2 information to identify and follow up with individuals who appear to owe taxes but did not file required returns. However, we also found that IRS’s use of Nonfiler to collect taxes owed by potential employment fraudsters is limited. Nonfiler is capable of addressing cases involving certain types of employment-related identity fraudsters who appear to owe taxes—specifically fraudsters for whom IRS receives W-2s that have mismatched names and SSNs as well as SSNs associated with deceased persons or children. However, the agency has made limited use of Nonfiler to collect taxes owed on such cases and faces the following resource challenges in doing so: Reduced staffing capacity. IRS determines the number of noncompliance cases pursued by its enforcement programs based on available resources. IRS’s budget declined by about $2.1 billion (15.7 percent) from fiscal years 2011 through 2018 after adjusting for inflation, and corresponding staff reductions have been most significant within IRS enforcement programs, such as Nonfiler. In 2018, the Treasury Inspector General for Tax Administration (TIGTA) reported that resource constraints have left IRS with fewer resources to work cases involving individuals who do not respond to nonfiler notices. For example, TIGTA found that IRS created 430,000 new compliance cases in fiscal year 2017 involving individuals who did not respond to nonfiler notices compared to 1.6 million in fiscal year 2013. Competing priorities. IRS is focusing its resources on modernizing its information technology systems and implementing Public Law 115- 97—commonly referred to as the Tax Cuts and Jobs Act. This law was enacted in December 2017 and included significant changes to corporate and individual tax law. Costly follow-up contacts. According to IRS officials, collecting taxes owed by employment-related identity fraudsters typically requires IRS staff to make in-person contact with taxpayers by locating them at their places of work, which is resource intensive. According to IRS, in-person contact is typically required because employment fraudsters are unlikely to provide employers and IRS accurate address information on W-2s; therefore IRS often lacks information needed to reach employment fraudsters through mailed Nonfiler notices. To help reduce the number of nonfilers and underreporters, IRS uses the Withholding Compliance Program (WHC) to pre-emptively identify taxpayers who appear to be substantially underwithholding taxes based on prior year W-2 and other information. Through this program, IRS issues “lock-in letters” to employers of individuals who appear to be underwithholding. Lock-in letters require employers to adjust employees’ withholding amounts to rates specified by IRS rather than the employees. IRS adjusts withholding rates based on the number of withholding allowances IRS determines the taxpayer is entitled to claim. Employees are also sent lock-in letters informing them of changes to their withholding rates. WHC may be a more cost-effective opportunity than Nonfiler for IRS to collect appropriate taxes from those employment-related identity fraudsters who do not otherwise file returns and pay taxes owed. First, WHC lock-in letters would be more likely to reach their intended recipients, making them potentially more effective in obtaining their intended responses. IRS sends lock-in letters to employers, and IRS officials said the agency typically has accurate address information for employers. IRS also sends notices to employees affected by lock-in letters, but these letters do not request or require taxpayer action. Second, businesses that employ employment-related identity fraudsters may be more likely to comply with lock-in letters than fraudsters would to Nonfiler notices. According to a 2018 TIGTA report, compliance with lock- in letters could further be improved if IRS took action against employers who do not comply with the letters and adjust employees’ withholdings accordingly. TIGTA recommended that IRS penalize employers who do not respond. IRS has agreed to consider penalties, and officials told us the agency is evaluating opportunities to do so. Third, we have previously reported that IRS is less likely to collect taxes owed the longer it takes IRS to contact taxpayers. Therefore, it is likely more effective for IRS to use WHC to address potential tax liabilities before they accrue, rather than use Nonfiler to assess and attempt to contact fraudsters and collect taxes owed months after filing deadlines have passed. According to IRS officials, WHC issues lock-in letters to address underwithholding by some employees who use matching names and SSNs; however, the program does not issue lock-in letters for cases involving W-2s with mismatched names and SSNs because of privacy concerns. IRS officials said the agency has an obligation to protect all taxpayers, including potential employment-related identity fraudsters. IRS officials told us that IRS previously sent lock-in letters for cases involving mismatched names and SSNs but stopped in 2012 because the agency wanted to avoid potentially disclosing an employment-related identity fraudster’s identifying information, such as the names of their employers, to those individuals whose SSNs were used to commit employment fraud. However, IRS could also redact personally identifiable information in the lock-in letters as it already does this when mailing tax return transcripts. For example, in response to data privacy concerns, in September 2018 IRS began including just the first four characters of business names on tax return transcripts requested by taxpayers. This approach could also be used for sending lock-in letters to employees to reduce disclosures of personally identifiable information in instances where lock-in letters do not reach their intended recipients. IRS officials told us that WHC’s limited resources prevent the program from addressing all underwithholding cases currently identified by the program. Officials also said that, for that reason, expanding WHC to include cases with mismatched names and SSNs would not result in WHC selecting additional cases. However, by not including cases with mismatched names and SSNs, IRS may be missing an opportunity to identify and select a population of underwithholding cases that could lead to greater revenue collection. This is because some cases with mismatched names and SSNs may have greater underwithholding than those cases that are currently selected by WHC. If IRS were able to allocate more resources toward generating additional lock-in letters in the future, these potential benefits could also increase. In addition, WHC may be more affordable than other enforcement programs to administer on a case-by-case basis because unlike enforcement cases initiated through Nonfiler, WHC does not result in IRS pursuing taxpayers through progressively more costly methods of contact to collect additional revenue. IRS officials acknowledged this possibility and told us the agency has not assessed the potential costs and benefits of expanding WHC to include cases with mismatched names and SSNs. Internal control standards state that federal managers should use quality information to achieve their objectives, communicate relevant information throughout the agency, and both assess and address risks to their mission. Additionally, leading practices in managing fraud risks include considering the benefits and costs of controls for addressing fraud-related risks. Further, IRS’s Strategic Plan has goals to use data analytics to inform decision making and protect the integrity of the tax system. Because IRS has not evaluated and documented the costs and benefits of expanding WHC to address risks posed by employment-related identity fraudsters, the agency cannot determine whether or not expanding WHC to include mismatch cases would enable IRS to collect additional revenue and deter employment fraud. By conducting such an assessment, IRS could determine whether expanding WHC to include mismatch cases would likely enable IRS to collect additional revenue and deter employment fraud. To manage the impacts of employment-related identity fraud on victims, IRS limits the circumstances under which these victims may be selected by enforcement programs. In analyzing IRS data, we found about 3 million taxpayers who have either been identified as “employment-related identity theft” victims by IRS (Action Code 525) or who have identified themselves as victims to IRS (Action Code 501). Automated Underreporter (AUR) programming prevents these taxpayers from being selected due to wage discrepancies. Instead, AUR analyzes these taxpayers for reporting discrepancies for other income types, such as investment income. IRS officials told us excluding these taxpayers from AUR’s W-2 checks helps IRS avoid burdening some victims who may be otherwise selected based on wages earned by a fraudster using the taxpayers’ name and SSN. Selected victims would be required to follow up with IRS to avoid being assessed tax liabilities. Following up would be particularly burdensome for victims whose names and SSNs are used by fraudsters year after year. Taxpayers with IDT action codes on their accounts are eligible for analysis and selection by other enforcement programs based on discrepancies in W-2 reporting; however, these programs’ low selection rates suggest that it is unlikely IRS will follow up with these victims and notify them of these discrepancies. For example, although Nonfiler analyzes these taxpayers for evidence of income indicating a filing requirement, TIGTA found that IRS notified just 25,105 or 14 percent of all 179,878 nonfiler cases identified in fiscal year 2016 of these discrepancies. Likewise, although IDT victims may be selected for examination, IRS data show that the agency examined about 892,000 or 0.6 percent of all individual income tax returns in fiscal year 2018, the most recent year for which data are available. IRS officials acknowledge that some of the approximately three million taxpayers with Action Codes 501 or 525 may underreport their own incomes, and excluding these taxpayers from AUR’s W-2 discrepancy checks creates an enforcement gap, enabling some victims who actually underreported their own wages to avoid enforcement. IRS does not know how many of these taxpayers have underreported wage income. However, some IDT victims excluded from AUR’s wage discrepancy checks may be incentivized to underreport wages and pay less tax than they owe if they learn IRS is unlikely to hold them accountable for paying those taxes. Individuals could learn about this enforcement gap, for example, if they accidentally failed to report wages from an employer and were not later contacted by IRS. In addition other taxpayers may be incentivized to falsely claim they are IDT victims to take advantage of this enforcement gap. In its research into behavioral insights, IRS has found that taxpayers are more likely to be noncompliant when they perceive doing so can yield substantial benefits with minimal costs. We have also previously reported that the extent to which taxpayers misreport income closely aligns with IRS’s ability to detect such noncompliance. In some instances, IRS has information needed to distinguish wages earned by legitimate taxpayers from those potentially earned by employment-related identity fraudsters using that same taxpayer’s name and SSN. For example, IRS can reasonably conclude the legitimate taxpayer earned the wages if they are reported on a current- or prior-year return filed by the taxpayer, as this indicates the taxpayer attests to having worked for the employer who paid the wages. Because IRS excludes IDT victims from AUR’s W-2 discrepancy checks, IRS may not identify or collect taxes owed by some who unintentionally underreport their wages (e.g., by forgetting to include a W-2 from a second employer). In addition, IRS is missing an opportunity to incentivize taxpayers to accurately report their income and avoid intentional underreporting. As previously stated, federal internal control standards call for managers to both use quality information and respond to risks. According to IRS officials, modifying AUR to effectively identify the underreporting of wages actually earned by identity theft victims would require IRS to not only adjust AUR to include wage discrepancy checks for these taxpayers but also to change how AUR identifies wage discrepancies. IRS officials told us that when AUR evaluates a taxpayer’s wage information for discrepancies, the program evaluates taxpayers based on aggregated W- 2 information. AUR is not programmed to evaluate taxpayers by analyzing some of their W-2s but not others, such as potential employment fraud W- 2s. IRS officials told us modifying AUR to include W-2 discrepancy checks of these taxpayers while excluding potentially fraudulent W-2s would not be a cost-effective use of IRS resources at this time. Specifically, officials noted that AUR discrepancy checks are programmed in the legacy assembly language code, a low-level computer language initially used in the 1950s. Although they were unable to provide an estimate for the costs of modifying this code, IRS officials said the effort would be resource intensive. IRS is modernizing outdated information technology systems, and officials said it would be more cost effective for the agency to modify W-2 discrepancy checks once the assembly language is replaced. IRS plans to retire 75 percent of the agency’s legacy assembly language code and Common Business-Oriented Language code legacy by the end of fiscal year 2024. Officials told us the agency does not have a specific timeline in place for updating the assembly code that supports AUR, though doing so is a program goal. Modifying AUR to include wage discrepancy checks for IDT victims as part of IRS’s broader effort to update AUR’s programming code would enable IRS to avoid making costly and redundant changes to legacy coding that IRS plans to replace. It would also be consistent with a goal outlined in IRS’s Strategic Plan to advance the use of data and analytics to inform decision making and could potentially result in IRS collecting additional revenue by enabling IRS to analyze wage information for about three million additional taxpayers to identify any wage reporting discrepancies. Some of these taxpayers may have greater revenue collection potential than cases AUR would otherwise select. SSA and IRS both have responsibility for parts of the Combined Annual Wage Reporting (CAWR) process to exchange W-2 information between the two agencies and to help ensure that taxpayers report and pay the proper amount of taxes on their wages. The CAWR Memorandum of Understanding (MOU), which was signed in 2007, is a key part of their collaborative effort, and SSA and IRS are legally bound to the mutually agreed upon purpose and functions. Specifically, the CAWR MOU covers the collaborative processes through which SSA and IRS share earnings information, including establishing clear roles and responsibilities for this effort, as called for by leading practices for inter- agency collaboration. IRS oversees tax administration, including ensuring compliance with tax laws. SSA acts as an agent to these activities by processing W-2s. As illustrated in figure 6, processes covered by the CAWR MOU include SSA sending accurate and inaccurate W-2s to IRS. Also, if wages are disclaimed through IRS, or IRS is able to correct a Social Security number-name mismatch using tax information, IRS sends this information to SSA. Federal law requires the Commissioner of Social Security and the Secretary of the Treasury to share W-2 information, and permits use of the CAWR MOU to effectuate this process. It also requires that the MOU remain in full force and in effect until modified or otherwise changed by mutual agreement of the heads of each agency. SSA and IRS have taken steps to update the 2007 CAWR MOU, but the effort has been underway for more than 3 years. As we reported in September 2012, continually updating agreements is an important part of the leading practice for written guidance and agreements. SSA and IRS officials told us that discussions about the update began in 2012 and the substantive work of updating the MOU began in August 2016. Since the MOU has not been updated in more than a decade, certain data-exchange materials and provisions in the MOU have become outdated, such as the references to microfilm. According to SSA and IRS officials, the MOU update has been driven by efforts at the staff level with executives briefed on the status. We have previously found that leadership involvement in collaborative efforts is needed to overcome the many barriers to working across agency boundaries. SSA officials noted that having highly involved executives would indicate problems with the MOU update process. IRS officials said that the staff level is the appropriate place to negotiate the MOU update with oversight from executives, as needed. However, at both agencies, officials at the staff level do not have the authority to agree to any updates or modifications of the MOU. SSA and IRS are responsible for ensuring the MOU update process is thorough, complete, and carried out in a timely manner. SSA and IRS officials stated that while the MOU is the cornerstone of SSA-IRS collaboration, completing the update is challenging because there are competing priorities. Additionally, the agencies are not legally required to update the MOU; instead, the MOU is in effect until modified or otherwise changed by mutual agreement of the Commissioner of Social Security and the Secretary of the Treasury (who delegated this authority to the Commissioner of Internal Revenue). In September 2019, SSA and IRS officials told us they plan to complete the update of the MOU in spring 2020, more than 3-and-a-half years after the effort to update the MOU began. Standards for project management call for developing a plan with specific actions and time frames. A plan could also identify the resources, processes, and individuals necessary to carry out the update. SSA and IRS officials acknowledged that they did not develop such a plan for the ongoing effort to update the MOU. By developing a plan for future updates that includes actions, time frames, and responsible individuals, including executive leadership, SSA and IRS would have greater assurance that the MOU would be updated when needed. While SSA and IRS have established joint functions in the CAWR MOU, the agencies do not have shared goals and performance measures to help track progress in implementing these functions and identify potential improvements. As we reported in September 2012, defining short- and long-term outcomes is an important part of the leading practice for outcomes and accountability for collaborative efforts. This includes defining and articulating common goals based on what the group shared in common and developing mechanisms, such as performance goals and measures, to evaluate the results. SSA officials said existing goals and measures in the MOU were sufficiently effective. However, we did not find evidence of goals and measures in the MOU and neither SSA nor IRS officials could provide documentation of specific examples of such. Establishing shared goals and performance measures for the CAWR MOU functions would help SSA and IRS monitor and evaluate its results, as well as identify potential weaknesses and potential improvements. While the MOU lacks goals and measures, it does contain provisions for the agencies to conduct annual studies of the CAWR process and to submit a report to each commissioner on the results. However, the agencies have not consistently implemented these provisions. Monitoring progress is an important part of the leading practice for outcomes and accountability for collaborative efforts. Continually monitoring agreements is an important part of the leading practice for written guidance and agreements. For SSA and IRS, this means monitoring progress toward fulfilling their legal obligation to implement the CAWR MOU. In the 2007 CAWR MOU, SSA and IRS agreed to the following monitoring provisions related to conducting an annual review of the CAWR process. Conduct annual joint studies of the CAWR process. Since the MOU was implemented in 2007, IRS and SSA have not conducted a joint study of the CAWR process. These reviews were intended to assist the required annual review of the MOU and help inform the agencies of potential improvements to the CAWR process. Specifically, the MOU requires that upon completion of the annual review, a joint SSA and IRS report should be sent to each commissioner consisting of the results of the review, a list of any changes that have occurred in the process, and any recommendation for changes. This is intended to serve as an important monitoring function for the MOU. IRS officials said the agencies have been unable to conduct annual joint studies or submit the required annual reports primarily because the MOU is extensive and affects many offices at both agencies. SSA and IRS officials said that they plan to change to a biannual interagency review of the MOU so they can do a better job of keeping the MOU updated and relevant. However, officials did not provide information about any steps they plan to take to ensure that the reviews would occur as required. According to SSA officials, SSA and IRS plan to meet every 3 or 6 months to review existing agreements, including the CAWR MOU. This may be a means of identifying necessary changes to the CAWR process since regular communication can facilitate effective collaboration; however, officials did not provide additional details on these potential new meetings. Conduct annual independent studies of the CAWR process. SSA had no records that it had conducted an independent study of the process in the past 3 years. IRS conducted two independent studies in 2018 on the CAWR process which primarily focused on IRS’s adherence to its policy guidance. Annual independent studies were intended to serve as another feedback mechanism to assist in the review of the MOU. According to SSA and IRS officials, they have not implemented these monitoring provisions because of resource constraints. As previously discussed, the agencies are updating the CAWR MOU and plan to finalize the updated MOU by spring 2020. Officials told us that, similar to the 2007 MOU, the updated MOU will include requirements for periodically reviewing the MOU to identify potential improvements to the CAWR process. However, the time frames may change. Developing and documenting a strategy for implementing the monitoring provisions in the updated MOU would provide greater assurance that SSA and IRS are periodically assessing the CAWR process and identifying opportunities for improvement, as required. As we reported in September 2012, agreeing on common terminology and definitions is an important part of bridging organizational cultures. One way to operate across agency boundaries is to foster open lines of communication. SSA and IRS do this by holding interagency meetings, including quarterly executive-level and monthly technical-level meetings. In addition, officials from SSA and IRS said that the agencies have a strong working relationship and that officials at both agencies have frequent informal communication. The agencies also established a fraud working group, which held introductory meetings in 2018 and 2019. While the group does not have a formal mission statement, the general scope of responsibility for the group is to identify areas of common interest related to mitigating fraud and to collaborating on best practices and efforts to mitigate fraud risks. However, SSA and IRS have developed limited common terminology and definitions related to their CAWR collaboration effort. The agencies have agreed on 10 definitions in their MOU, but these definitions are very limited in scope; for example, two of these definitions simply spell out the agency names and none of the definitions are for the 20 data variables the agencies exchange daily. Both SSA and IRS officials stated that common terminology related to identity fraud would be helpful, and acknowledged that they use different terminology and have to call each other to ask what different terms mean. SSA officials cited the use of different terminology at SSA and IRS as a barrier to collaboration. Because of the absence of common terminology, IRS has been unaware of information it receives from SSA in some cases. For example, through the common format record exchange, SSA shares information with IRS about why SSA determined that a W-2 is inaccurate, but IRS was unaware of this information. SSA told us that it sends a table to IRS annually that includes code combinations for their data transfers and their meanings which explain why the W-2 was accurate or inaccurate. However, SSA officials were unsure of the extent to which IRS officials understood the codes. One reason that SSA determines a W-2 is inaccurate is if earnings with the same name, SSN, and EIN were disclaimed in previous years. SSA communicates this to IRS using codes within the W-2 record that are labeled “invalid due to SWED.” However, SSA and IRS have not defined “SWED” and IRS officials said that they were unaware of receiving information from SSA about previously disclaimed wages. Officials said they interpreted the information to relate to invalid wages due to name and SSN mismatches and spent time trying to resolve the mismatch issue. They said that such information could be useful for future enforcement efforts. Further, IRS officials said that they were also unaware of other code combinations that SSA officials told us they use to inform IRS about accurate and inaccurate wages. IRS attributed its unfamiliarity with the data elements coming from SSA to staff turnover since key IRS officials who were familiar with the data elements retired. However, IRS could have been aware of the meaning of the variables if the agencies had established and documented common definitions for these data elements. In addition, according to IRS officials, they have limited resources for following up on information that SSA is sharing because they have been focused on competing priorities, including implementing the Tax Cuts and Jobs Act of 2017. SSA and IRS officials noted that the next version of the MOU will define additional terminology that was not defined in previous MOU documents. For example, officials said that “EIN” and “TIN” are key IRS terminology that may be defined in the new MOU. Until SSA and IRS clearly define the data elements they exchange as part of the CAWR process, SSA and IRS are at risk of not communicating effectively about CAWR and, thus, missing opportunities to use data more effectively to identify fraudulent or otherwise inaccurate W-2s. This could be done, for example, by developing a shared data dictionary that clearly defines all of the data elements the agencies are exchanging. Employment-related identity fraud can have negative impacts on victims and poses risks to both SSA and IRS. Victims may face IRS enforcement actions or a reduction in benefits for some federal programs based on wages earned by fraudsters. The full scope of this fraud is unknown. In 2018, IRS marked more than 800,000 taxpayer accounts with Action Code 525 “employment-related identity theft.” However, IRS’s use of the term “employment-related identity theft” likely understates the true scope and impact of this type of fraud and may be misleading to both agency decision makers and Congress. Additionally, by assessing the feasibility of incorporating additional compliance checks into its checks of employment-related identity fraud, IRS may be able to develop a method for identifying additional taxpayers at risk of this type of fraud. SSA and IRS rely on accurate W-2 information to carry out their missions and have taken steps to detect the submission of fraudulent W-2s. Evaluating the costs and benefits of expanding IRS’s Withholding Compliance Program (WHC) to include cases with mismatched names and SSNs may provide IRS an opportunity to increase revenue collection. Additionally, while IRS has taken steps to manage the impacts of identity fraud on victims, the agency’s decision to exclude approximately 3 million individuals with IDT action codes from Automated Underreporter’s (AUR) wage discrepancy checks has resulted in a gap in enforcement coverage. IRS plans to update most of the agency’s legacy programming code by the end of fiscal year 2024. Updating AUR’s programming to include these individuals would enable IRS to close this enforcement gap and potentially increase revenue. Further, SSA and IRS’s 2007 CAWR MOU plays an important role in IRS and SSA’s efforts to accurately report wage information and resolve mismatches. While the agencies expect to finalize their first update of the MOU by spring 2020, efforts to update the MOU have been ongoing for more than 3 years. Developing a plan for implementing future updates would provide greater assurance that the MOU would be updated when needed. Additionally, developing performance goals and measures for the MOU as well as a strategy for assuring the studies called for by the MOU are completed within the specified time frames would help ensure that SSA and IRS are periodically assessing the CAWR process, and identifying opportunities for improvement. Moreover, by clearly defining the data elements IRS and SSA exchange as part of the CAWR process, the agencies would be better positioned to effectively use the data to identify fraudulent or otherwise inaccurate W-2s. We are making a total of 12 recommendations, including eight to IRS and four to SSA. The Commissioner of Internal Revenue should modify the title of IRS’s employment-related identity theft action code 525 to reflect the type of employment-related identity fraud encompassed by this action code. (Recommendation 1) The Commissioner of Internal Revenue should assess and document the feasibility of incorporating additional checks into its automated checks of employment-related identity fraud for populations at risk of employment- related identity fraud, such as children, elderly, deceased persons, and individuals associated with multiple wage records. (Recommendation 2) The Commissioner of Internal Revenue should assess and document the costs and benefits of using WHC to address compliance risks posed by potential employment-related identity fraudsters who owe taxes and take appropriate action, as needed. (Recommendation 3) The Commissioner of Internal Revenue should modify AUR to include wage discrepancy checks for victims of employment-related identity fraud once IRS has updated AUR’s legacy programming code. (Recommendation 4) The Commissioner of Internal Revenue should, in collaboration with the Commissioner of Social Security, develop and document a plan for updating future CAWR MOUs. The plan should identify actions, time frames, and responsible parties, including executive leadership. (Recommendation 5) The Commissioner of Internal Revenue should, in collaboration with the Commissioner of Social Security, develop and implement goals and performance measures for the CAWR MOU. (Recommendation 6) The Commissioner of Internal Revenue should, in collaboration with the Commissioner of Social Security, develop and document a strategy for assuring that the reviews required by the updated MOU are completed within the specified time frames. (Recommendation 7) The Commissioner of Internal Revenue should, in collaboration with the Commissioner of Social Security, clearly define data elements they exchange with SSA. (Recommendation 8) The Commissioner of Social Security should, in collaboration with the Commissioner of Internal Revenue, develop and document a plan for updating future CAWR MOUs. The plan should identify actions, time frames, and responsible parties, including executive leadership. (Recommendation 9) The Commissioner of Social Security should, in collaboration with the Commissioner of Internal Revenue, develop and implement goals and performance measures for the CAWR MOU. (Recommendation 10) The Commissioner of Social Security should, in collaboration with the Commissioner of Internal Revenue, develop and document a strategy for assuring that the reviews required by the updated MOU are completed within the specified time frames. (Recommendation 11) The Commissioner of Social Security should, in collaboration with the Commissioner of Internal Revenue, clearly define the data elements they exchange with IRS. (Recommendation 12) We provided a draft of the sensitive version of this report to IRS, SSA, the Federal Trade Commission, the Department of Health and Human Services, and the Department of Homeland Security for comment. In comments reproduced in appendix II, IRS neither agreed nor disagreed with the recommendations. In comments reproduced in appendix III, SSA agreed with the recommendations and noted that SSA and IRS officials are meeting on a recurring basis to complete an updated memorandum of understanding. IRS, SSA, the Department of Homeland Security, and the Federal Trade Commission provided technical comments which were incorporated as appropriate. The Department of Health and Human Services had no comments on the report. We are sending copies of this report to the appropriate congressional committees, the Commissioner of Internal Revenue, Commissioner of Social Security, Chairman of the Federal Trade Commission, Secretary of Health and Human Services, Acting Secretary of Homeland Security, Secretary of the Treasury, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov If you or your staff have any questions about this report, please contact Jessica Lucas-Judy at (202) 512-9110 or LucasJudyJ@gao.gov, or Rebecca Shea at (202) 512-6722 or SheaR@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines (1) the potential scope of employment-related identity fraud, including what the Internal Revenue Service (IRS) knows about this type of fraud and what we could determine by analyzing the Department of Health and Human Services’ National Directory of New Hires (NDNH) and IRS data; (2) Social Security Administration (SSA) actions to detect and deter this fraud as well as notify victims; (3) IRS actions to detect and deter this fraud as well as notify victims; and (4) the extent to which SSA and IRS are collaborating to address the issue. To describe and analyze the potential scope of employment-related identity fraud, we took the following steps: 1. Identified groups at risk of identity theft. We first reviewed Treasury Inspector General for Tax Administration, SSA Office of the Inspector General, and our prior reports on Social Security number (SSN) misuse to determine common characteristics of individuals who are at risk of SSN misuse. These characteristics include being deceased, elderly, a child, or having three or more wage records during the 3-month period of our review. Based on these reports, we defined “elderly” as over age 84 and “children” as under age 14 for the purposes of this review. 2. Identified SSNs at risk of SSN misuse. We used SSA’s full death file for dates of death for deceased individuals, and its Numerical Index File (Numident) for dates of birth for living individuals.We next compared full death file and Numident data to a quarterly extract of NDNH data listing the names and SSNs of individuals who earned wages between August and October 2016. We selected data from this quarter because, at the time of our review, these were the oldest data for which relevant IRS tax data were also available. We used this comparison to identify individuals employed between August and October 2016 who also met at least one of these at-risk characteristics. NDNH is a database of individuals employed in the United States. Data are collected and reported by state workforce agencies and federal agencies, and the database is administered by the Department of Health and Human Services’ Office of Child Support Enforcement. NDNH data are comprised of three types: verified, unverified, and unverifiable. The verified data—used in this analysis— have been checked against SSA records to confirm that the name and SSN match SSA records. Unverified data include data that do not match on name or SSN, and unverifiable data include data that did not include enough information to attempt a match (e.g., when states submit partial or missing name information). According to the Department of Health and Human Services, there were 584,013,484 verified wage records, 18,629,720 unverified, and 91,134,352 unverifiable as of December 31, 2018. Verified data were used in this analysis to make the estimate more conservative since cases of potential synthetic identity theft—where the name and SSN do not match—are excluded from verified data. NDNH is designed to assist state child support agencies in locating parents and taking appropriate, interstate actions concerning child support orders. Some authorized agencies also use NDNH data to help prevent overpayments and detect fraud. For example, IRS has access to NDNH to administer the Earned Income Tax Credit. However, IRS and SSA are not authorized to use NDNH information to detect potential employment-related identity fraud. We were authorized to use NDNH through the GAO Access and Oversight Act of 2017, Pub. L. No. 115-3, 131 Stat. 7. Form W-2, Wage and Tax Statement (W-2) was not reported on a 2016 tax return. When possible, we also limited the analysis to cases where the taxpayer had a known filing requirement. We also identified cases that were consistent with misuse of SSNs for employment-related identity fraud, rather than taxpayer noncompliance. However, we were unable to determine the total extent of taxpayer noncompliance for taxpayers included in this analysis. Our analysis is not intended to be a comprehensive effort to identify all potential cases of employment-related identity fraud. We focused our analysis on cases where matching names and SSNs were used to obtain employment. These cases pose a risk to SSA, IRS, and victims, yet little is known about these cases. 4. Analyzed tax characteristics of potential employment-related identity theft victims and other taxpayers. Last, we used CDW to analyze selected tax characteristics of both individuals we identified as having at least one employer-submitted Form W-2 that was not reported on a 2016 tax return as well as those where employer- submitted Forms W-2 were reported. For example, we analyzed data on wage withholding rates, the prevalence of selected IRS identity theft indicators on taxpayers’ accounts, and IRS enforcement actions taken against these individuals. We assessed IRS procedures against the information gathering and data analytics leading practices in the Framework for Managing Fraud Risks in Federal Programs. We did not conduct a comprehensive fraud risk assessment of the IRS enforcement programs. Our assessment was limited to the control activities surrounding employment-related identity fraud. We assessed the reliability of the full death file, Numident, NDNH quarterly wage data, and selected elements of CDW by reviewing relevant documentation, interviewing knowledgeable agency officials, and performing electronic testing to determine the validity of specific data elements in the data. We determined that the data elements used in our analysis were sufficiently reliable for the purpose of our work to describe and analyze the potential scope of employment-related identity fraud. To assess IRS and SSA actions to detect and prevent employment- related identity fraud as well as notify victims, we reviewed relevant documentation including IRS’s Internal Revenue Manual and SSA’s Policy Operations Manual System. We also interviewed knowledgeable officials from both agencies on SSA and IRS processes for detecting and preventing employment-related identity fraud and notifying victims. We compared IRS’s and SSA’s efforts to relevant federal internal control standards. We also assessed the agencies’ efforts against IRS and SSA’s respective strategic plans as well as select leading practices to combat fraud, as identified in the Framework for Managing Fraud Risks in Federal Programs. To evaluate the extent to which IRS and SSA are effectively collaborating to address employment-related identity fraud, we reviewed relevant agency documents, such as IRS and SSA’s Combined Annual Wage Reporting Memorandum of Understanding, other IRS-SSA legal agreements, meeting minutes from IRS-SSA joint meetings, and policy manuals. Because of its role with assisting victims and collecting statistics on identity theft, we interviewed agency officials from the Federal Trade Commission in addition to knowledgeable officials from IRS and SSA. Because of its role helping employers verify the identities of employees, we interviewed officials at the Department of Homeland Security. We focused our assessment on SSA and IRS because those agencies are most directly involved in the wage reporting process used to detect and resolve employment-related identity fraud. We assessed IRS and SSA’s collaboration efforts against leading practices for collaboration we have identified in our prior work and against standards for project management. We identified key elements of each leading practice and assessed the extent to which SSA and IRS collaboration on employment- related identity theft aligned with leading practices or key elements. The performance audit upon which this report is based was conducted from November 2017 to January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We worked with SSA from October 2019 to May 2020 to prepare this public version of the original sensitive report for public release. This public version was also prepared in accordance with these standards. In addition to the individual named above, the following staff made key contributions to this report: Neil A. Pinney (Assistant Director), Philip D. Reiff (Assistant Director), Melissa L. King (Analyst-in-Charge), Priyanka Sethi Bansal, Heather A. Collins, Ann L. Czapiewski, Celina F. Davidson, Pamela R. Davidson, Julia C. DiPonio, Shannon J. Finnegan, Steven Flint, Robert L. Gebhart, James A. Howard, Grace H. Kwon, Krista Loose, Maria C. McMullen, Kevin C. Metcalfe, J. Daniel Paulk, Lindsay W. Swenson, Sonya Vartivarian, Ariel Vega, and Miranda J. Wickham.", "summary": "Employment-related identity fraud poses risks to IRS's ability to collect taxes owed on wages and to SSA's ability to correctly calculate and manage Social Security benefits. GAO was asked to review employment-related identity fraud. This report examines (1) the potential scope of employment-related identity fraud, including what IRS knows about this type of fraud and what GAO could determine by analyzing Department of Health and Human Services' National Directory of New Hires (NDNH) and IRS data; (2) SSA and IRS actions to detect and deter this fraud as well as notify victims; and (3) SSA and IRS's collaboration on the issue. GAO analyzed 3 months of 2016 NDNH wage data and 2016 IRS taxpayer data to identify potential employment-related identity fraud. GAO also reviewed relevant IRS and SSA documentation and interviewed agency officials. This is a public version of a sensitive report that GAO issued in January 2020. Information that SSA deemed sensitive has been omitted. Employment-related identity fraud occurs when people use a name or Social Security number (SSN) other than their own to get a job. People may do this if they are not authorized to work in the United States or are trying to avoid child support payments, among other reasons. Victims may face Internal Revenue Service (IRS) enforcement actions based on wages earned by fraudsters. IRS identified more than 818,000 cases in 2018, but this included only one form of employment-related identity fraud—mismatches between the identity listed on the Form W-2, Wage and Tax Statement (W-2) and the identity on the tax return. The true scope of employment-related identity fraud is unknown. GAO reviewed additional forms of this fraud and identified 1.3 million SSNs that for 2016 had both (1) characteristics associated with employment-related identity fraud; and (2) wages reported by the employer on a W-2, but not reported by the employee on a tax return. This includes about 9,000 individuals whose employers reported W-2s in five or more states, but who did not include them all on their tax return (see figure). The Social Security Administration (SSA) processes W-2s before sending W-2 data to IRS for enforcement purposes. SSA has developed processes to detect some inaccurate W-2s and notify potential fraud victims. IRS uses W-2 information to deter some potential fraudsters, but has not assessed the costs and benefits of expanding its enforcement efforts to include certain individuals who may underwithhold taxes or not file returns. Doing so could help IRS determine if such an effort would enable the agency to collect additional revenue. SSA and IRS entered into a memorandum of understanding (MOU) to collaborate to exchange wage data. However, they have not established performance goals and measures for the MOU, implemented the MOU's monitoring provisions, or clearly defined the data elements they exchange. GAO is making 12 recommendations to IRS and SSA, including that IRS assess the feasibility of adding checks to its review of employment-related identity fraud, and assess the costs and benefits of expanding enforcement; and that both agencies improve the implementation of their MOU. SSA agreed and IRS neither agreed nor disagreed with the recommendations.", "document_type": "gao"}
{"report": "The military retirement system is a government-funded benefit system that has historically been considered a significant incentive in recruiting and retaining a voluntary, career military force. Until recently, almost all active-duty servicemembers were enrolled in the High-3 (legacy) retirement system. In this system, servicemembers who served at least 20 years earned a DB annuity. Those who were eligible earned 2.5 percentage points per year of service multiplied by the average of their highest 36 months of basic pay, with payments beginning upon retirement from the military and adjusted annually for inflation. Servicemembers also had the option to contribute a portion of their basic pay to a personal TSP account, but DOD provided no contributions. A previous GAO report found that active-duty servicemembers’ rate of reaching 20 years of service varied substantially among the military service branches (see fig. 1). For example, for active-duty servicemembers entering military service in 1992, the estimated probability of reaching 20 years of service was almost 15 percentage points higher—and more than three times higher—for the Air Force than the Marine Corps. Federal law established the Military Compensation and Retirement Modernization Commission (MCRMC) in the NDAA for Fiscal Year 2013 to study the military’s compensation system in detail and make recommendations to modernize servicemembers’ pay and benefits. The MCRMC’s final report, released in January 2015, recommended that Congress revise the military retirement system so DOD could help more servicemembers save for retirement earlier in their careers, leverage the retention power of the legacy retirement system, give the services greater flexibility to retain quality people in demanding career fields, and promote servicemembers’ financial literacy, among other things. The NDAA for Fiscal Year 2016 established BRS to replace the legacy retirement system. As with the legacy retirement system, servicemembers in BRS must serve 20 years to receive a DB annuity. Under BRS, eligible retirees receive a DB monthly benefit equal to 2 percentage points per year of service multiplied by the average of a servicemember’s highest 36 months of basic pay—lower than the 2.5 percentage point multiplier under the legacy retirement system. BRS also provides servicemembers with DC benefits through an employer contribution, which did not exist in the legacy retirement system. For servicemembers who began their service on or after January 1, 2018, DOD automatically contributes 1 percent of a servicemember’s basic pay into the individual’s TSP account after 60 days of service and, after 2 years of service, matches a servicemember’s contributions up to 4 percent of their basic pay, for a maximum military contribution of 5 percent of a servicemember’s basic pay. These servicemembers are automatically enrolled in BRS at a 3 percent default contribution rate. DOD estimates that with automatic enrollment in TSP and the automatic government contribution, 85 percent of new servicemembers covered by BRS will receive at least some retirement benefits when they leave military service. BRS offers servicemembers some additional features and benefits not offered under the legacy retirement system. Servicemembers under BRS are eligible for a one-time continuation payment as a retention incentive at the servicemember’s mid-career point, between 8 and 12 years of service. Servicemembers who accept the continuation benefit incur an additional service obligation. BRS also offers servicemembers who serve 20 years or more the option to convert the present-value equivalent of either 25 or 50 percent of their DB annuity payments for the period from their date of retirement until the date they reach their Social Security full retirement age (FRA) to a lump-sum payment upon retirement from the military. Taking this lump-sum payment would reduce the retiree’s annuity payments only until he or she reaches FRA, after which the annuity payments would revert to the full benefit level (see fig. 2). Active-duty servicemembers with fewer than 12 years of service as of December 31, 2017 were eligible to enroll in BRS until December 31, 2018. The decision to opt in to BRS or remain in the legacy retirement system was irrevocable. Compared to the legacy retirement system, which provided only a DB plan, the BRS’s enhanced DC benefit and reduced DB annuity shifts more of the responsibility for managing servicemembers’ retirement security from DOD to servicemembers. To help ensure that servicemembers have the financial literacy to make sound financial decisions, the NDAA for Fiscal Year 2016 added a requirement for DOD to provide servicemembers ongoing financial literacy training at various career and life stages, including at initial entry, promotions, vesting in the TSP, eligibility for continuation pay, marriage, divorce, and the birth of a first child. GAO’s prior work on financial literacy training compiled testimony from experts from the private sector, federal government agencies, nongovernmental organizations, and academic institutions to: define financial literacy as the ability to use knowledge and skills to manage financial resources effectively for a lifetime of well-being; identify the workplace as a particularly effective venue for providing financial education and helping individuals improve their financial decision making; and summarize the effectiveness of various interventions and how to address the needs of workplace populations traditionally underserved by financial education. DOD developed three courses to help servicemembers make informed decisions about whether to opt in to BRS or remain in the legacy retirement system. The BRS Opt-In Course was available as a 2-hour online or in-person course that servicemembers had to attest they had completed before opting into the new retirement system. DOD reported that 91 percent of an estimated 1.7 million eligible servicemembers attested that they had completed the training during the BRS opt-in period. The course included information on (1) the importance of saving for retirement, (2) the differences between the legacy retirement system and BRS, (3) factors for servicemembers to consider in choosing between the two retirement systems, and (4) tools and resources for servicemembers to consult when making their opt-in decision. DOD developed two additional BRS trainings for key military personnel in an effort to expand the network of in-person resources available to servicemembers eligible to opt into BRS. One course provided installation-level financial management professionals—Personal Financial Managers (PFMs) and Personal Financial Counselors (PFCs)—with more detailed information to reinforce the BRS Opt-In Course curriculum for servicemembers and answer their specific questions about BRS. The other course provided optional training to military supervisors regardless of their eligibility to opt into BRS. DOD officials said it was important to educate military supervisors on BRS since many junior servicemembers discuss personal financial information with their direct supervisors. DOD officials said that the agency released both of these trainings in advance of the BRS Opt-In Course so that PFMs and supervisors would have time to understand the new system and prepare for questions from servicemembers. DOD also developed the BRS New Accession Course for servicemembers who entered the military on or after January 1, 2018 and who are automatically enrolled in BRS. (See fig. 3.) Servicemembers take this course when entering service as part of their mandatory basic training (“boot camp”) or at the first school they attend after basic training. This course explains BRS’s key components, identifies the tools and resources available to help servicemembers save for retirement, and encourages servicemembers to actively manage their TSP accounts. DOD officials said that the New Accession Course is very similar in content to the BRS Opt-In Course but without comparisons to the legacy retirement system. The course facilitator leads servicemembers through a series of short videos on BRS, asks questions at the end of each of the course sections, and is available to answer servicemembers’ questions throughout the course. DOD publicized BRS by creating a central website that links to outreach material in a variety of media formats, including videos available on YouTube, social media content, an interactive online comparison calculator, webinars, and external websites such as Military OneSource and https://www.tsp.gov. For example, DOD’s central BRS website links to its BRS Fact or Fiction video series, which addressed various BRS misconceptions through 20 brief videos. In the video series, DOD introduced the #BlendedRetirement hashtag, then distributed supplementary BRS infographics with this hashtag to link back to additional resources on social media sites. DOD officials said they also are developing a mobile app to provide servicemembers easy access to financial readiness information through tools like calculators and games. Additionally, DOD’s interactive online BRS calculator allowed servicemembers to enter personal financial information, such as their military grade, estimated date of military separation or retirement, and TSP contribution percentage, so those who were eligible to opt into BRS could compare how their retirement savings outcomes might differ under BRS and under the legacy retirement system. DOD’s Office of Financial Readiness also trained financial counselors across the service branches to supplement the information in its BRS trainings as well as to provide servicemembers in-person financial literacy education. DOD officials said that the agency employs at least one PFM at most military installations or uses PFCs, who are government contractors. DOD officials said that PFMs and PFCs travel as needed to provide support at multiple installations. One PFM we interviewed estimated that PFMs provide as many as 10 group presentations per week on retirement issues that they tailor to fit their audiences’ needs. Another said one-on-one counseling sessions allowed servicemembers to share their personal financial situations, receive information germane to their unique circumstances, and explore available tools and resources. DOD officials said that, as outlined in federal statute, the role of PFMs and PFCs is to educate servicemembers about financial options available to them and not to provide financial advice. In addition to the centralized trainings and resources DOD created, the service branches used their internal communication systems for BRS outreach campaigns and created additional training tailored to the needs of their servicemembers (see fig. 4). For example, according to Navy officials, during the final 6 months of the BRS opt-in period, the Navy posted approximately 80 Facebook and Twitter posts to its accounts, with many of these reminding servicemembers of their opt-in choice. The posts linked to additional resources and advertised outreach like the Navy’s Facebook Live event, which utilized social media to provide servicemembers online access to financial experts who could answer their retirement-related questions. Military supervisors also said that most of the service branches sent targeted communications to supervisors to remind eligible servicemembers at regular meetings to complete the BRS Opt-In Course. The service branches also created supplemental BRS trainings tailored to meet their servicemembers’ needs. For example, the Marine Corps developed a classroom-based BRS training that included specific instructions on how to use the Marines’ data systems to make BRS decisions, as well as statistics on the average percentage of Marines that complete 20 years of service. With all incoming servicemembers automatically enrolled in BRS as of January 1, 2018, DOD officials said the agency has shifted its continuing financial literacy training from the opt-in decision to saving for retirement. As with the BRS training, the military provides continuing financial literacy education through both DOD and the service branches. DOD’s Office of Financial Readiness provides policy, education, advocacy, and program oversight to promote servicemembers’ financial readiness. While DOD developed the BRS trainings and conducted outreach, DOD officials said that the service branches have the primary responsibility for developing and providing servicemembers continuing financial literacy education, including on saving for retirement, based on their own resources and their servicemembers’ needs. The service branches use a variety of formats (see fig. 5). DOD is also developing a plan to provide continuing financial literacy education to servicemembers at various career and life stages. DOD officials said the agency plans to improve the consistency of the continuing financial literacy education provided by the service branches and consolidate it so it is delivered at the career and life stages specified by the NDAA for Fiscal Year 2016. DOD’s Office of Financial Readiness released guidance in August 2019 to provide the service branches a common set of learning objectives for financial literacy education aligned with these specific career and life stages. DOD officials told us that the service branches are responsible for delivering the continuing financial literacy education to servicemembers at these stages according to their schedules and resources. We found that DOD’s Blended Retirement System (BRS) trainings met many established financial literacy training effective practices (see sidebar on next page and table 1). However, lack of assessments of some courses affected DOD’s ability to measure how well the courses helped participants and to make any needed changes. Financial education experts have found that financial literacy trainings that meet effective practices can improve employees’ overall financial wellness. These experts identified the workplace as a particularly effective venue for providing financial education and helping individuals improve their financial decision making because employers have the potential to reach large numbers of adults in a cost-effective manner at a place where they make important financial decisions. Effective Financial Literacy Training Practices Information is unbiased: Employers’ financial literacy education programs should provide financial information that avoids even the appearance of conflicts of interest. Links to one-on-one financial help: Programs should provide access to one-on- one financial coaches who can help employees understand and take action on their priorities. Leverages trusted messengers: Programs should use trusted coworkers and other peers to provide or facilitate assistance on financial matters. Assesses employees’ financial literacy to provide assistance and help set priorities: Programs should periodically assess employees’ financial situations and goals to pinpoint how best to provide assistance and help employees set priorities. Enables employees to take action directly from the course: Programs should provide employees the means, for example, through direct links or forms provided in the course, to convert knowledge to financial action. According to DOD officials, servicemembers will make more financial decisions that may impact their ability to successfully save for retirement under BRS than under the legacy retirement system, which makes providing effective financial literacy training to servicemembers particularly important. We found that all of DOD’s BRS trainings met the applicable financial literacy effective practices of presenting unbiased information, directing servicemembers to options for one-on-one financial help, and employing trusted messengers—such as military peers and Personal Financial Managers (PFMs)—to deliver the course information. For example, each of the applicable BRS trainings encouraged servicemembers to work with PFMs to understand how their personal financial circumstances impact saving for retirement. While the BRS trainings met many of the financial literacy effective practices we selected, two of the trainings fell short in assessing servicemembers’ financial literacy, which could allow DOD to better pinpoint how to provide assistance and help servicemembers set priorities. Servicemembers were required to pass a test to complete the BRS Opt-In Course; DOD data show that only 32 percent of servicemembers passed on their first attempt. However, DOD did not revise course material to provide additional information in topic areas where post-test results indicated servicemembers may have needed further training. DOD officials said that the agency consciously avoided making significant changes to its BRS trainings to ensure consistency and course stability throughout the opt-in enrollment period. DOD officials also told us that they were not surprised by the initial low pass rate because they designed the test to be difficult so that servicemembers could demonstrate mastery of the material. DOD’s New Accession Course does not assess individual servicemembers’ understanding of the material, which is information DOD would need to improve its training to provide assistance and help servicemembers set priorities. The course includes a series of knowledge checks, but because the questions are administered to the group as a whole, DOD cannot assess individual servicemembers’ understanding and use this information to revise the course material or to provide servicemembers with additional assistance. DOD officials told us that the agency views the course as successful because it gets students to engage in discussion regarding the basics of BRS and financial readiness. DOD does not have a plan to assess individual servicemembers’ understanding of course material going forward. While servicemember engagement is important, it is not an assessment of their understanding of course material. Servicemembers who do not understand BRS concepts may not save enough for a secure retirement under BRS. Additionally, the BRS Opt-In Course did not meet the financial literacy training effective practice of enabling servicemembers to act on course information directly from the training. For example, the BRS Opt-In Course suggested servicemembers contact PFMs and PFCs if they had further questions about BRS, but the course did not provide direct links for servicemembers to do so. Further, the course did not include forms for servicemembers to enroll in and make contributions to TSP accounts. This standard is considered an effective practice for financial literacy training because research has found that employees who can directly convert their knowledge to immediate action have improved overall financial wellness. DOD addressed this issue in its most recently released training, the BRS New Accession Course, which enables servicemembers to make immediate decisions, such as assigning their initial TSP contribution rates, by providing servicemembers the relevant form within the training. The NDAA for Fiscal Year 2016 included a requirement for DOD to add questions on servicemembers’ financial literacy to its annual survey and use the results as a benchmark to evaluate and update the continuing financial literacy training DOD will provide to servicemembers in the future. The NDAA for Fiscal Year 2016 also requires DOD to develop ongoing financial literacy training for servicemembers to take at key career and life stages. DOD has the opportunity to ensure that individual knowledge assessments are included in the guidance it provides the service branches on the key objectives that must be met in these trainings. Military personnel cited multiple challenges described by servicemembers in taking the BRS Opt-In Course and seeking financial literacy support. In our interviews at five military installations, military supervisors and financial counselors said they believed servicemembers had difficulty (1) understanding the training due to their low financial literacy; (2) taking, and relating to, optional financial literacy training due to mission and short-term life goals; and (3) setting up online access to their TSP accounts. Many military supervisors and Personal Financial Managers (PFMs) we interviewed said that many servicemembers with whom they interacted misunderstood key BRS concepts and lacked the basic knowledge to make sound financial decisions related to BRS even after completing the mandatory BRS Opt-In Course. Providing basic financial education to junior enlisted servicemembers, who can be as young as 17 years old, may be especially challenging due to their limited life and work experience. These servicemembers score the lowest on measures of financial literacy, according to the 2017 Status of Forces Survey, an annual survey of a sample of servicemembers that covers key issues of military life. Some servicemembers said that the training platforms (e.g., computer- based and large group training), while efficient in providing mandatory training to a large group of servicemembers, were not ideal for a group with very limited baseline financial literacy. For example, several military supervisors said some servicemembers advanced through the computer- based BRS Opt-In Course as quickly as possible, and may not have understood the content. One military supervisor said it may be hard for servicemembers to identify the most critical elements in the computer- based training because they could not interact with the material or ask clarifying or personal questions. For example, one group of military supervisors said the current training addresses what TSP is, but there is a need for more training to answer servicemembers’ questions about how to manage and optimize their accounts for retirement savings. In response, DOD officials said that while these topics were not covered in depth in the BRS trainings, servicemembers have access to additional resources, such as PFMs and the TSP website for help with personal questions about managing their savings under BRS. Large group trainings, which could have hundreds of servicemembers in attendance, also may have discouraged servicemembers from asking clarifying questions due to the number of participants. DOD officials acknowledged that servicemembers may need more one-on-one help when making personal financial decisions, which is why the agency trained PFMs and PFCs to address servicemembers’ BRS and financial literacy questions and provide additional support. Some military supervisors said the servicemembers who they directed to optional one- on-one financial counseling sessions asked the PFMs detailed questions their supervisors were not able to answer, ran their own numbers and received personalized information to help them make decisions, and often took action during the session. DOD officials said one challenge to getting servicemembers to seek out more personalized one-on-one financial help is the perception that servicemembers seek PFMs primarily after facing financial hardship. These officials said they are working to shift the military culture so servicemembers seek out PFMs for financial planning purposes similar to how civilians use financial counselors. Military supervisors and PFMs told us that servicemembers had difficulty seeking out financial literacy support because of demanding operational schedules and a focus on short-term life and mission goals. This was especially true for junior servicemembers, who may be uncomfortable requesting time away from their mission duties. Further, some military supervisors said junior servicemembers tended not to recognize the importance of saving for retirement when faced with other, more immediate, financial priorities, such as purchasing a car. One group of military supervisors said that since most junior servicemembers do not seek out retirement advice, they try to find opportunities to weave the topic into other discussions, for example, about how taking out a car loan can impact a junior servicemember’s saving for retirement. Servicemembers can manage their TSP accounts online by viewing current plan information and making or changing contribution allocations; however, setting up an online account depends on servicemembers having a stable mailing address. The Federal Retirement Thrift Investment Board (FRTIB), which administers the TSP, mails participants a time-sensitive TSP password required to access their TSP accounts online. Some military supervisors said that servicemembers reported difficulty receiving their initial TSP password because they relocate often and may lack a permanent mailing address. FRTIB officials acknowledged that this fraud prevention measure might make it more difficult for participants to access their TSP accounts, but noted that they must balance security with ease of use and have not yet found any viable options to address this issue. Federal government internal controls standards state that entities should use appropriate methods to communicate so that information is readily available when needed. Under the Blended Retirement System (BRS), military retirees with 20 or more years of service may choose, when they retire, to convert part of their monthly annuity into a lump sum payment, in exchange for a temporarily lower monthly benefit. The lump-sum payment is partial in two ways: 1) servicemembers may convert either 25 or 50 percent of their annuity payments to a lump-sum payment, and 2) the lump-sum conversion only applies to annuity payments payable prior to the servicemember’s Social Security full retirement age (FRA)—age 67 for those born in 1960 or later. After the service member reaches FRA, the annuity payments revert to the full monthly pension. (See fig. 6.) In its final report, the Military Compensation and Retirement Modernization Commission recommended that the new military retirement system should offer a lump-sum payment option to increase flexibility for retiring servicemembers and remain fiscally sustainable. Since many servicemembers retire from the military at a younger age than most civilians in the workplace, DOD officials said that some military retirees might prefer a lump-sum payment to start a business or buy a house. Personal Discount Rates Personal discount rates can be derived from individuals’ behavior when faced with intertemporal monetary choices. In contrast, more traditional approaches to pension discount rates are based on financial market data or expectations rather than on individual preferences or behavior. In theory, personal discount rates reflect individuals’ valuation of money received today versus in the future. However, behavioral economic research has shown that people do not always make rational choices related to foregoing current benefits for future payoff. given stream of converted pension payments. The NDAA for Fiscal Year 2016 directed the Secretary of Defense to choose a discount rate for BRS lump sums that (1) uses average personal discount rates that take into consideration “applicable and reputable studies of personal discount rates for military personnel and past actuarial experience in the calculation of personal discount rates,” and (2) is in accordance with generally accepted actuarial principles and practices. Researchers have sought to quantify personal discount rates by studying personal choices in a variety of contexts involving the tradeoff of payoffs at different times (see sidebar). Two such studies involved military personnel being offered lump-sum payments in lieu of annuity payments. According to the Institute for Defense Analyses (IDA), the studies computed an estimated average personal discount rate for servicemembers who were presented with the offer, based on the choices by servicemembers to either elect the lump-sum payment or the annuity. DOD officials told us that, to comply with the requirements of the NDAA for Fiscal Year 2016, they considered several factors to set the discount rate for BRS lump-sum calculations. DOD officials said they first contracted with a research organization to estimate a range of personal discount rates based on past studies. They said they then adjusted that range based on differences between the specific features of past lump- sum offers and those of BRS lump sums. They also considered how a lump-sum offer could impact the retention of military personnel, since DOD relies on a percentage of experienced servicemembers to continue serving beyond 20 years. DOD officials told us they wanted to reduce the likelihood a lump-sum payment would lead more people to retire earlier than they would otherwise. Finally, even though past studies had found higher personal discount rates (resulting in smaller lump-sum amounts) for enlisted servicemembers than officers, DOD officials told us it would go against core values of military compensation if the agency did not apply the same discount rate to all lump-sum payments, regardless of the servicemember’s rank. Considering all of these factors, DOD devised a formula for setting what it termed the “Government Discount Rate” (GDR) that would be used in calculating BRS lump-sum amounts. DOD constructed the GDR by starting with a market index of high-quality corporate bond rates and then adding an adjustment factor so that the GDR fell within the range of observed personal discount rates. According to DOD, this current method for setting this rate will be reexamined at least every 4 years. The GDR for 2019 is 6.81 percent, which is a “real” interest rate that does not include an inflation component. To compare the GDR to more common nominal interest rates, an inflation adjustment must be added. For example, if inflation were assumed to be 2.4 percent per year, a GDR of 6.81 percent would be approximately equivalent to a nominal discount rate of 9.37 percent. The method used to determine BRS lump-sum payment amounts is likely to result in a discount rate that is higher—based on recent interest rates, roughly double— than that used to calculate minimum lump-sum distributions from private-sector pension plans, when all other factors are equal. The discount rates for determining minimum lump-sum amounts for private-sector pension plans that offer them are governed by ERISA. The Internal Revenue Service (IRS) publishes the discount rates applicable to minimum lump-sum determinations each month, based on ERISA provisions. For 2018, these rates generally fell in the range of 2.5 to 4.9 percent, on a nominal basis, compared to the GDR, which was about 9 percent, on a nominal basis (depending on assumed inflation). We found, based on recent interest rates, holding age and monthly annuity amounts constant, the higher discount rate applied to BRS lump- sum calculations would significantly reduce servicemembers’ lump-sum payment amounts. Additionally, we found that the percentage difference would be the largest at younger retirement ages, since the difference in discount rates would have an impact over a longer period of time. For a servicemember retiring at age 40, for example, we found BRS lump-sum payments to be about 40 percent smaller, based on recent interest rates, than if calculated following the requirements under ERISA. (See fig. 7.) For more information on ERISA and our methodology for calculating lump-sum payments, as well as sensitivity testing and factors that can affect this comparison at different points in time, please see appendix I. DOD officials told us that the discount rate used for BRS lump-sum payments was different than the rate used in private-sector pension plans for some key reasons. DOD officials said the NDAA for Fiscal Year 2016 required that the BRS discount rate be based on average personal discount rates, which is a different approach to discount rates than that used under ERISA. DOD officials also said the agency relies on maintaining a certain percentage of servicemembers with 20 or more years of service and did not want the offer of a lump-sum offer to entice too large a percentage of servicemembers to leave military service. However, knowledgeable stakeholders expressed some concerns with the higher discount rate used to determine BRS lump-sum payment amounts. For example, the DOD Board of Actuaries stated that a relatively high discount rate, and the lower lump-sum payments that would result, could be perceived as taking advantage of servicemembers. Additionally, the American Academy of Actuaries said those who accept lump-sum payments using higher discount rates are likely to either not understand the financial value of their annuity benefits or have an immediate financial need. On the other hand, stakeholders we interviewed noted that BRS’s lump-sum feature was intended to provide options to servicemembers, which was a central component of implementing BRS. Although current active-duty servicemembers eligible to choose a lump- sum payment are not scheduled to retire until 2026, at the earliest, DOD can take certain steps to help them better understand the tradeoffs associated with the lump-sum option. Decisions about lump-sum options are complicated, and stakeholders knowledgeable about financial literacy have pointed out the importance of providing sufficient information about the tradeoffs involved for those making such decisions. In a 2015 report, we identified key information to help individuals in the private sector make an informed decision when considering a lump-sum payment versus an annuity. (See table 2.) Without this key information, service personnel will be unable to prudently weigh the advantages and disadvantages of the lump-sum option in their retirement decisions. DOD officials said they posted a training video on the BRS lump-sum option to the BRS website in July 2019. Servicemembers also have access to other descriptive material on the BRS website, such as a fact sheet on the BRS lump sum, and the BRS calculator to estimate their lump-sum payment with some assumptions about future pay. The shift from the legacy retirement system to BRS marks a significant change in retirement benefits for an estimated 1.7 million military servicemembers. While more servicemembers will receive retirement benefits under BRS than under the legacy retirement system, BRS will require servicemembers to more actively and continuously manage their retirement decisions throughout their military career and in retirement. As an employer, DOD is well positioned to provide financial literacy training and support to servicemembers as they make retirement decisions. DOD has designed a multi-faceted approach to provide resources over time and in a variety of formats, increasing the likelihood that servicemembers will be able to find guidance when they need it. DOD completed a large undertaking in educating servicemembers about the choice they faced in deciding whether to opt into BRS, but this was only the first step in educating servicemembers about how to maximize and manage their retirement savings under BRS. In educating servicemembers about saving for retirement, DOD would benefit from applying the financial literacy training effective practices identified by experts, especially periodically assessing employees’ financial understanding and using these assessments to revise and tailor ongoing training. Given that young servicemembers are often stationed in multiple locations for short amounts of time and that BRS places increased responsibility on servicemembers to save for retirement through TSP contributions, it is important that servicemembers receive the necessary information to access their TSP accounts online in a timely manner. The current TSP password process has limited some servicemembers’ ability to manage their accounts. It is important for FRTIB to expeditiously address this issue. Of additional concern is how DOD will ensure that servicemembers understand the tradeoffs associated with BRS’s lump-sum feature. BRS lump-sum payments are calculated using a higher discount rate than private-sector pension plans, which results in lower lump-sum payments, by comparison. While the BRS lump sum is limited, and the full annuity amount would resume at servicemembers’ Social Security full retirement age, the reduced annuity paid until then could still have a significant impact on some servicemembers’ financial security. A fundamental element of BRS is the greater responsibility and choice placed on individuals. To work well, such a system requires that sufficient, clear, and accurate information be provided so that servicemembers can make the prudent choices best suited to their personal financial situations. Consistent with this principle, DOD should ensure that the information and tools that it provides to eligible servicemembers about the lump sum clearly lay out the tradeoffs of this decision and allows those eligible to make a well-informed prudent choice that best meets their individual financial circumstances. The Secretary of Defense should evaluate the results of its financial literacy training assessments to determine where gaps in servicemembers’ financial knowledge exist and revise future trainings to address these gaps. (Recommendation 1) The Secretary of Defense should provide servicemembers disclosures that explain key pieces of information about the lump-sum payment, including some measure of its relative value, the potential positive and negative financial ramifications of choosing the lump-sum payment option, and a description of how it was calculated. (Recommendation 2) The Executive Director of the Federal Retirement Thrift Investment Board should work with the Secretary of Defense to explore alternative options (including online resources) for servicemembers to receive their initial Thrift Savings Plan password so that servicemembers can access and manage their online accounts without added delays. (Recommendation 3) We provided a draft of this report to the Secretary of Defense and the Executive Director of the Federal Retirement Thrift Investment Board for review and comment. In its letter, which is reproduced in appendix II, DOD concurred with the report’s recommendations and offered comments on some of our findings. For recommendation 1, regarding the evaluation of its financial literacy training assessments, DOD stated that in 2017 it added questions to its annual Status of Forces Survey to assess the military population’s understanding of basic financial concepts. While these survey results will allow DOD to respond to identified gaps in servicemembers’ financial literacy, Status of Forces survey results have taken years to compile in the past. Assessing servicemembers’ financial literacy as part of mandatory trainings will allow DOD to more promptly identify gaps in servicemembers’ knowledge and adjust trainings to address those gaps. For recommendation 2, regarding the provision of information on the BRS’s lump-sum payment options, DOD stated that it has developed a training course, published information to help educate servicemembers on the BRS’s lump-sum option, and included a lump-sum section in its BRS calculator. While we are encouraged by DOD’s efforts to develop various tools for educating servicemembers on the BRS’s lump-sum option, in this report we identified additional information that is important to include in lump sum disclosures. In its letter, DOD expressed concern that the title of our report focused only on one aspect of our findings. We believe that the title accurately reflects our report’s key findings, conclusions, and recommendations. DOD also said that the agency did not intend for the BRS Opt-In Course to be financial literacy training, and thus were concerned that we evaluated this training based on the effective practice identified in prior GAO work of assessing employees’ financial literacy to provide assistance and help set priorities. However, we believe that our use of this effective practice to evaluate the BRS Opt-in Course is consistent with our prior findings that employers are well-suited to provide financial education and help individuals improve their financial decision making. We compared the BRS Opt-In Course to this effective practice because the course provided DOD an opportunity to assess whether servicemembers understood key aspects of BRS, undoubtedly a key aspect of servicemembers’ financial well-being. In addition, DOD stated that the agency viewed servicemembers’ initial low pass-rate of the BRS Opt-In Course as a positive result because they designed the course to be rigorous and it forced servicemembers to retake the parts of the training where they were failing to comprehend the course material. DOD also stated that revising the training during the 2017 training period was not practical because it would have resulted in some servicemembers receiving disparate training formats and materials. We understand DOD’s concerns; however as DOD continues to develop additional financial literacy training we encourage the agency to consider that low pass rates on post-training tests often indicate a gap in knowledge and a possible need to revise the training. In its final comment, DOD agreed with us that there is a lack of reliable data for comparing the BRS lump-sum feature with those provisions offered by state and local government pension plans. DOD also stated that the BRS lump-sum feature was unique and therefore not comparable to private-sector pension plans governed by ERISA. Although there are differences between BRS and ERISA, the BRS and ERISA lump-sum provisions are the only defined benefit lump sum conversion provisions that are specified under federal law. Further, the lump-sum provisions for both reflect a participant choice that can have important consequences for a participant’s financial security. Our recommendation is premised on the principle that regardless of which particular features a pension plan offers, participants need clear, complete, and accurate information to make prudent decisions regarding their retirement security. The FRTIB also provided comments, reproduced in appendix III, and generally agreed with the report’s findings and conclusions. The FRTIB also concurred with our recommendation regarding the provision of TSP passwords to military personnel and said that they will continue to explore avenues to address how servicemembers receive their initial TSP password while continuing to emphasize the need for security. DOD and FRTIB provided technical comments, which we incorporated into the report as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution of this report until 30 days from the report date. We are sending copies of this report to the Secretary of Defense, the Executive Director of the Federal Retirement Thrift Investment Board, the Director of the Consumer Financial Protection Bureau, and other interested parties. This report is also available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Charles Jeszeck at (202) 512-7215 or jeszeckc@gao.gov or Frank Todisco at (202) 512-2700 or todiscof@gao.gov. Mr. Todisco meets the qualification standards of the American Academy of Actuaries to address the actuarial issues contained in this report. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This appendix discusses in detail our methodology for addressing (1) what actions the Department of Defense (DOD) has taken to help servicemembers understand the Blended Retirement System (BRS) and, more generally, educate servicemembers on saving for retirement; (2) what DOD can learn from financial literacy training effective practices and the implementation of BRS training to continue supporting servicemembers in saving for retirement; and (3) how lump-sum payment amounts are determined under BRS and how they compare to the methods used for private-sector pension plans that offer them. To answer all of these questions, we interviewed officials at DOD, the Federal Retirement Thrift Investment Board (FRTIB), the Consumer Financial Protection Bureau (CFPB), and other organizational stakeholders knowledgeable about the military and retirement. We also reviewed relevant agency documents and federal laws and regulations. To understand how DOD helped servicemembers understand BRS, we reviewed DOD’s centralized training and outreach material. We also conducted group interviews with senior officers and enlisted servicemembers on military installations to learn about some of the informal training and mentorship provided by military leaders. We used the following criteria to select military installations to visit: 1. Sufficient number of BRS-eligible personnel available to participate 2. High number of active-duty servicemembers stationed at the 3. Availability of a Personal Financial Manager (PFM) on the installation 5. Mix of single service versus joint bases 6. Proximity to an urban center 7. Primary mission of the installation is operational (versus training) We selected five military installations to visit: Camp Pendleton (Marine Corps), Fort Sam Houston (Army), Naval Base San Diego (Navy), and Randolph Air Force Base and Scott Air Force Base (Air Force). At each installation, we met with separate groups of 8 to 12 senior enlisted servicemembers and senior officers. These senior servicemembers supervise junior servicemembers who, as a group, were most likely to have had to make a decision on whether to opt into BRS. We also met with the groups’ installation-level financial management professionals— Personal Financial Managers (PFM), Personal Financial Counselors (PFC), or Command Financial Specialists (CFS)—who provide servicemembers additional financial literacy training and one-on-one financial counseling. We asked questions of all group interview participants related to: 1. Information provided to servicemembers about BRS 2. Common needs of servicemembers in making decisions about BRS 3. Common questions servicemembers had about BRS 4. Challenges experienced in providing training and/or support 5. Anticipated future needs for training and/or support These interviews provided insights into senior officers and enlisted servicemembers’ experiences facilitating the rollout of BRS training to junior servicemembers, but did not yield information that was generalizable to all senior officers and enlisted servicemembers. We also reviewed and compared DOD’s financial literacy trainings to financial literacy training effective practices. To identify financial literacy effective practices, we reviewed published articles and reports on the topic. Our review included a March 17, 2015 forum GAO convened with 20 financial literacy leaders and experts focusing on financial education in the workplace, and the subsequent report, Financial Literacy: The Role of the Workplace, GAO-15-639SP (Washington, D.C.: July 2015). The report provided the best single compilation of financial literacy effective practices from a diverse set of experts from the private, non-profit, governmental, and academic sectors. The report summarizes forum participants’ discussions across seven topic areas. Of these seven, we selected two that were most germane to DOD’s BRS training: (1) Employers should address the needs of traditionally underserved workplace populations, and (2) Effective practices can include automatic enrollment in retirement plans, financial health checks, and personalization. Across these two topic areas, we selected the five financial literacy training effective practices that were most relevant to the type of trainings DOD developed for BRS. Specifically, we determined if BRS trainings (1) contain unbiased information, (2) contain links to one- on-one financial help, (3) leverage trusted messengers, (4) assess participants’ financial literacy so DOD can provide assistance and help set priorities, and (5) enable participants to take action directly from the course. To understand how BRS lump-sum payments are determined, we interviewed DOD officials to learn about the issues they considered when designing BRS’s lump-sum feature, how DOD determines the discount rate it uses for lump-sum payments, and how the BRS discount rate used to calculate lump sums relates to personal discount rates. To understand discount rate issues applicable to lump-sum payments in other pension plans, we interviewed stakeholders knowledgeable about other pension plans, consulted with our internal actuarial experts, and reviewed relevant prior work. We also consulted with actuaries at DOD to clarify our technical understanding of the calculation of lump-sum amounts under BRS. We created a lump-sum payment calculator to run simulations of various lump-sum calculations—including those used in private-sector pension plans—to show the effect that varying certain calculation methods and assumptions can have on the value of the lump-sum payment. We calculated and compared illustrative lump-sum amounts under BRS to what those lump-sum amounts would have been under federal laws and regulations applicable to private-sector pension plans. We did not do a similar comparison to public-sector pension plans because of a lack of reliable, generalizable data on the prevalence of lump sums offered by the many state and local government plans and the applicable discount rates used. Some lump-sum options under state and local government plans do not require a discount rate at all because they return employee contributions with interest or are a deferred retirement option provision (DROP) rather than lump sums that involve discounting future promised payments. Different state or local governments might set their own rules regarding any lump sums. In contrast, the lump-sum provisions applicable to both BRS and private-sector pension plans under the Employee Retirement Income Security Act of 1974, as amended (ERISA) are in federal law. The following section provides additional technical detail regarding the methods used to determine the lump-sum discount rate (the Government Discount Rate, or GDR) under BRS; the methods used to determine discount rates for determining minimum lump-sum amounts under ERISA; a discussion of key differences between BRS and ERISA approaches; and the methods and assumptions we used to compare BRS lump-sum amounts to minimum lump sums under ERISA, along with a discussion of how the comparison could vary over time. DOD’s construction of the GDR begins with a 7-year average of estimated high-quality corporate bond real interest rates for maturities of about 23 years, and then adds an add-on factor to bring the discount rate up to a level consistent with applicable studies of personal discount rates, subject to possible adjustments for DOD concerns about retention of servicemembers. DOD officials told us that the 23-year maturity was intended to reflect the average time between a servicemember’s retirement from the military until Social Security full retirement age (FRA). The 7-year averaging is for the purpose of smoothing out short-term fluctuations in interest rates. The add-on for 2018 and 2019 is 4.28 percentage points. The GDR for 2019 is 6.81 percent, which is a “real” discount rate that does not include an inflation component. Interest rates are often regarded, economically, as consisting of two components: a portion to cover expected inflation (the inflation component), plus a portion to provide a return in excess of inflation (the “real” return component). For example, if inflation expectations are 2.50 percent per year, and the interest rate on a bond is 4.50 percent, then the bond is expected to provide a real return (in excess of inflation) of approximately 1.95 percent ( x 100). In this case, 4.50 percent would be referred to as the nominal interest rate and 1.95 percent would be referred to as the real interest rate. In order to convert the GDR into an equivalent nominal discount rate (for comparison to ERISA discount rates), an inflation assumption is needed. We used an inflation assumption of 2.40 percent per year, which is the inflation assumption used by the Congressional Budget Office (CBO) in its 2019 long-term budget outlook. As a result, with this inflation assumption, the nominal discount rate equivalent to the GDR of 6.81 percent is 9.37 percent ( x 100). Military pensions (both under legacy and BRS) are increased each year to fully keep up with inflation. The lump-sum equivalent of such a benefit could be calculated in one of two ways, which mathematically would produce the same result: (1) applying the nominal discount rate (in this example, 9.37 percent) to the projected increasing series of monthly annuity benefits, or (2) applying the real discount rate (in this example, 6.81 percent) to a fixed (not inflation indexed) monthly annuity. For determining minimum lump sums under ERISA, the discount rate is actually a combination of three “segment” rates that reflect bond yields at different maturities: a short-term rate to discount future payments due in the next 5 years, a medium-term rate to discount future payments due between 5 and 20 years out, and a long-term rate to discount future payments due beyond 20 years. These are nominal rates. These rates are published monthly by the Internal Revenue Service (IRS) and are based on an average of high-quality corporate bond rates for the month. Private-sector pension plan sponsors have some flexibility in selecting a method for determining which monthly averages would be used to calculate lump sums offered in a particular plan year. As a result, for a lump sum payable in a particular month, the applicable ERISA segment rates could be those for a month up to 16 months prior to the month of the lump-sum payment, depending on the provisions of the plan. Minimum lump sums under ERISA also include a “mortality discount,” which means that the lump sum is reduced to reflect the fact that for any future scheduled pension payment, there is a probability that the retiree will no longer be alive to receive it. We included this mortality discount in our ERISA calculations. DOD decided not to include a mortality discount in the BRS lump-sum methodology. DOD officials told us that mortality rates from age 44 to age 67 are relatively small, such that the impact of including mortality would be overwhelmed by minor changes in the discount rate. As a result, for simplicity, they decided not to include a mortality discount. Not including a mortality discount has the effect of making the BRS lump sum somewhat more generous than it would be if it included a mortality discount. Thus, key differences in the determination of lump-sum amounts under BRS and for ERISA minimums include the following: The development of the GDR starts with corporate bond rates for a 23-year maturity, whereas the ERISA segment rates are based on corporate bond rates for many maturities that are summarized into three segment rates for three different ranges of maturities. Thus, the comparison at any point in time will be affected by the shape of the yield curve. The development of the GDR starts with a 7-year average of corporate bond rates, whereas the ERISA segment rates are based on more current corporate bond rates. Thus, the comparison at any point in time will be affected by movements in interest rates in the prior 7 years. The GDR includes an add-on, currently 4.28 percentage points, to bring the GDR in line with applicable studies of personal discount rates. According to DOD, the add-on also takes into account considerations of retention of military personnel. Thus, the comparison at any point in time will be affected by any changes DOD makes to the magnitude of the GDR add-on. The determination of the minimum lump sum under ERISA includes a mortality discount; the determination of lump sums under BRS does not. The GDR applies over an entire calendar year, whereas the segment rates change month to month, and the segment rates applicable to a particular month’s lump sum could be the published rates for up to 16 months prior, depending on the plan provisions. For our comparison, we assumed a lump sum payable in June 2019. As noted earlier, the applicable GDR for 2019 is 6.81 percent, and the nominal equivalent rate, based on our inflation assumption of 2.40 percent, is 9.37 percent. For the ERISA minimum lump sum, we used the May 2019 segment rates published by IRS, which are 2.72 percent for the first 5 years’ scheduled payments, 3.76 percent for the next 15 years’ payments, and 4.33 percent for the scheduled payments beyond 20 years. We also included the mortality discount in the ERISA calculation. As noted in the body of this report, the result was that the BRS lump sum was 42 percent smaller than it would have been under ERISA rules for an age-40 retirement, and 32 percent smaller for an age-50 retirement. We also looked at the range of ERISA segment rates over the 16-month period from February 2018 through May 2019 to determine the range of potential results depending on which month’s ERISA rates might apply for a particular plan. The BRS lump sum ranged from 38 percent smaller to 42 percent smaller than on an ERISA basis for an age-40 retirement and from 28 percent smaller to 32 percent smaller for an age-50 retirement. We also calculated sensitivities from varying the inflation assumption. As noted earlier, we used an inflation assumption of 2.4 percent, the inflation assumption used by the CBO in its 2019 long-term budget outlook. If instead we used an inflation assumption of 2.0 percent (and the May 2019 ERISA segment rates), the BRS lump sum would have been 39 percent smaller than on an ERISA basis for an age-40 retirement, and 30 percent smaller for an age-50 retirement. The other key differences, noted earlier, in the determination of lump-sum amounts under BRS and for ERISA minimums could also affect the comparison at any point in time. However, we believe the comparisons presented in this report are a reasonable representation of the general magnitude of the differences in lump-sum amounts under BRS compared to the minimum amount required under ERISA. We conducted this performance audit from March 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contacts named above, Mark M. Glickman (Assistant Director), Anjali Tekchandani (Analyst-in-Charge), Cynthia Nelson, and Stephen C. Yoder made key contributions to this report. Also contributing to this report were Vincent Balloon, Alicia Cackley, Virginia Chanley, Sheila R. McCoy, Mimi Nguyen, Stacy Ouellette, Joseph Silvestri, Adam Wendel, and Seyda Wentworth.", "summary": "DOD's new retirement system, BRS, provides automatic and matching DOD contributions to servicemembers' individual Thrift Savings Plan accounts but reduces the retirement annuity paid to those who serve at least 20 years. BRS also offers servicemembers the option of taking part of their retirement annuity as a lump-sum payment. GAO was asked to describe DOD's financial education efforts under BRS. This report examines (1) actions DOD has taken to help servicemembers understand BRS and saving for retirement, (2) what DOD can learn from financial literacy training effective practices and its implementation of BRS training to continue supporting servicemembers in saving for retirement, and (3) how BRS lump-sum payment amounts are determined. GAO reviewed DOD's efforts to educate servicemembers on retirement decisions, conducted group interviews with senior officers and enlisted servicemembers at five military installations on facilitating the rollout of BRS training to junior servicemembers, and created a lump-sum payment calculator to compare different calculation methods and assumptions on the value of the lump-sum payment. In 2016, the Department of Defense (DOD), along with the military service branches, began a multi-year effort to provide training to help servicemembers make informed decisions about saving for retirement through DOD's new retirement system, the Blended Retirement System (BRS). DOD provided computer-based training to help military supervisors, financial counselors, and eligible servicemembers understand the new retirement system, implemented in 2018, and its impact on saving for retirement. DOD trained financial counselors to provide servicemembers in-person, one-on-one financial counseling and classroom courses on BRS and related topics. In addition, DOD prepared ongoing financial literacy training that servicemembers will take upon reaching specific career and life stages. BRS trainings met many of the effective practices for financial literacy training identified in prior GAO work, but some DOD trainings do not incorporate the practice of assessing servicemembers' financial literacy. DOD could use such assessments to modify course material to bolster training in areas where servicemembers' comprehension was weaker. Without assessing whether its financial literacy training is effectively conveying course information, DOD may be missing opportunities to better support servicemembers' retirement decisions. Servicemembers also reported challenges in taking the Opt-In Course for BRS that may inform ongoing and future DOD training. Examples of Servicemembers' Financial Literacy Challenges on Retirement DOD determines BRS lump-sum payment amounts at retirement by applying an interest rate (or discount rate) to calculate the present value of annuity payments servicemembers forego by taking a lump sum. The BRS discount rate exceeds the rate used by private-sector pension plans, resulting in a lower lump sum than if private-sector rates applied. DOD can take certain steps to help servicemembers understand how to compare the BRS lump-sum payment option with the full annuity option. Without this information, servicemembers may not make informed decisions and potentially risk their retirement savings. GAO recommends 1) DOD assess its course evaluations to improve its financial literacy training on retirement for servicemembers, 2) DOD provide key information on the calculation of retirement lump-sum payments, and 3) Federal Retirement Thrift Investment Board explore alternatives for servicemembers to receive their TSP passwords. Both agencies agreed with their respective recommendations.", "document_type": "gao"}
{"report": "In the United States, authority to regulate elections is shared by federal, state, and local officials. Congressional authority to regulate elections derives from various constitutional sources, depending on the type of election. In addition, Congress has passed legislation in major functional areas of the voting process, such as voter registration and prohibitions against discriminatory voting practices. However, responsibility for the administration of state and federal elections resides at the state level. States regulate various aspects of elections including, for example, registration procedures, absentee and early voting requirements, and Election Day procedures. Within each state, responsibility for managing, planning, and conducting elections is largely a local process, residing with about 10,300 local election jurisdictions nationwide. Some states have mandated statewide election administration guidelines and procedures that foster uniformity in the way their local jurisdictions conduct elections, whereas other states have guidelines that generally permit local election jurisdictions considerable autonomy and discretion in the way they run elections. The result is that elections can be administered differently across states and local jurisdictions. Unless states require otherwise, local jurisdictions generally have discretion over activities such as election officials’ training and, in most states, the selection and purchase of voting technology. Among other things, local election officials register eligible voters; educate voters on how to use voting technology; provide information on the candidates and ballot measures; recruit, train, organize, and mobilize poll workers; prepare and test voting equipment for use; and count ballots. The election process is composed of pre-election, Election Day, and post-election activities: Pre-election activities include providing opportunities for eligible individuals to register to vote, maintaining and updating the voter registration database, recruiting and training poll workers, selecting polling locations, preparing voting materials, testing equipment, qualifying candidates for office, and administering absentee and vote-by-mail voting processes. Election Day activities include opening and closing polling places, setting up voting machines and voting booths, checking in voters and verifying registration status, and providing opportunities for voters to mark and cast ballots. Post-election activities include securing equipment and ballots, transferring physical ballots or records of vote counts to a central location for counting, determining the outcome of the election, publishing unofficial results, certifying official election results, and performing recounts, if required. The election process relies on various assets—such as information technology systems, networks, equipment, and facilities. These assets can be broadly categorized as physical, cyber, and human components of the Election Infrastructure Subsector, as described in table 1. Physical, cyber, and human assets comprising the election infrastructure are susceptible to unintentional and intentional threats. As we have previously reported, unintentional, or nonadversarial, threat sources include equipment failures, software coding errors, or the accidental actions of employees (human errors). Threat sources also include natural disasters and other events that can cause failure within sectors on which the election infrastructure is dependent, such as power grid failures in the energy sector. Intentional, or adversarial, threats can involve targeted and untargeted attacks from a variety of sources, including criminal groups, hackers, disgruntled employees, foreign nations engaged in espionage and information warfare, and terrorists. These adversaries vary in terms of the capabilities of the actors, their willingness to act, and their motives, which can include seeking monetary gain or pursuing an economic, political, or military advantage. Appendix I lists general cybersecurity threat sources that can impact information technology systems that support the election infrastructure. Cyber adversaries may make use of various techniques, tactics, and practices—or exploits—to adversely affect an organization’s computers, software, or networks, or to intercept or steal valuable or sensitive information. These exploits are carried out through various conduits, including websites, email, wireless and cellular communications, internet protocols, portable media, and social media. Further, adversaries can leverage common computer software programs, such as Adobe Acrobat and Microsoft Office, to deliver a threat by embedding malware or other exploits within software files that can be activated when a user opens a file within its corresponding program. DHS and others have identified general cyber and physical threats that are applicable throughout the election process. For example, voting equipment may be susceptible to a supply chain attack in which the malicious actor may use the voting equipment vendor as a pathway to plant malware to modify or compromise ballot definition files before they reach the hands of election officials. Also, the absence of or lack of consistent physical access controls, auditable chain of custody procedures, or vendor installed countermeasures may allow malicious actors or well-placed insiders to manipulate voting equipment and ballots at any stage of the process through unauthorized physical access. In addition, there are certain physical and cyber threats that are applicable to individual assets and stages in the election process. Figure 1 provides examples of threats to various assets and stages in the election process. Additionally, DHS has analyzed and identified common cybersecurity vulnerabilities associated with enterprise networks and voter registration systems supporting election infrastructure that could apply to multiple assets at all stages of the election process. Such vulnerabilities include user susceptibility to malicious email, outdated software patches, the use of default system configurations, passwords that are weak or presented in clear text, and the use of operating systems with known weaknesses that have not been properly addressed. Presidential Policy Directive 21, issued in February 2013, shifted the nation’s focus from protecting critical infrastructure against terrorism to protecting and securing critical infrastructure and increasing its resilience against all hazards, including natural disasters, terrorism, and cyber incidents. The directive identified 16 critical infrastructure sectors and outlined roles and responsibilities for protecting these sectors. Further, the directive established sector specific agencies as the federal entities responsible for providing institutional knowledge and specialized expertise to facilitate or support federal, state, and local governments, as well as private sector entities, in protecting critical infrastructure. The National Infrastructure Protection Plan, updated by DHS in December 2013, further integrates critical infrastructure protection efforts between government and private sectors, among other things. It describes a voluntary partnership model as the primary means of coordinating government and private sector efforts to protect critical infrastructure. As part of the partnership structure, the designated sector-specific agencies serve as the lead coordinators for the security programs of their respective sectors. In accordance with the National Infrastructure Protection Plan, the National Protection and Programs Directorate within DHS was designated the sector-specific agency, or lead federal agency, for the Election Infrastructure Subsector. CISA subsequently assumed the role of sector-specific agency upon its establishment as the successor to the Directorate in November 2018. As the lead agency for the Election Infrastructure Subsector, CISA is responsible for coordinating partnership activities and information sharing and is the primary federal interface with the subsector’s stakeholders with respect to security. The Election Security Initiative, part of CISA’s National Risk Management Center, is responsible for managing the agency’s election subsector partnerships. To implement the voluntary partnership model, the subsector created two complementary coordinating councils—one for governments and one for private sector partners—to facilitate partnerships to support election infrastructure. Specifically, the Election Infrastructure Subsector Government Coordinating Council, created in October 2017, enables federal, state, and local governments to share information and collaborate on best practices to mitigate and counter threats to election infrastructure. The council is composed of 27 members, which include three voting members from the federal government—specifically one from DHS and two from the Election Assistance Commission (EAC)—and 24 from state and local governments. The Federal Bureau of Investigation (FBI), EAC, and National Institute of Standards and Technology (NIST) coordinate with each other, with CISA, and with state and local governments through the Election Infrastructure Subsector Government Coordinating Council. Additionally, the Subsector Coordinating Council was chartered in February 2018 and includes private sector entities whose services, systems, products, or technology are used by or on behalf of state or local governments in administrating the U.S. election process. The Election Infrastructure Subsector Specific Plan outlines actions that CISA, as the sector-specific agency, the Government Coordinating Council, and the Subsector Coordinating Council will take to support election infrastructure. Within the Department of Justice, the FBI supports the Election Infrastructure Subsector by countering foreign influence operations and collecting and processing threat information on election infrastructure. This effort is headed by the FBI’s Foreign Influence Task Force, which integrates the agency’s cyber, counterintelligence, counterterrorism, and criminal law enforcement resources to better understand threats posed by foreign influence operations. Among other things, the task force investigates cyber operations targeting election infrastructure or public officials, and covert influence operations designed to influence public opinion and sow division through disinformation and misinformation on social media. The FBI exchanges threat information with CISA and other federal partners to help states and local jurisdictions detect and prevent operations targeting the election infrastructure. Further, the EAC supports the Election Infrastructure Subsector by carrying out its responsibilities under the Help America Vote Act. Specifically, the EAC develops voluntary voting system guidelines and oversees the testing and certification of voting systems. Under the Help America Vote Act, the EAC works through the Technical Guidelines Development Committee to establish a set of principles, guidelines, and requirements specifying how voting systems are to meet standards of functionality, accessibility, and security. The EAC has also provided states with operational grants to replace voting systems. According to the Acting Executive Director of EAC, states also used the grants to increase the security of election systems, such as voter registration systems, and apply other cybersecurity enhancements. Additionally, the EAC and CISA have collaborated to develop select initiatives—such as web-based training for election officials—to expand outreach to states and local jurisdictions. NIST supports the Election Infrastructure Subsector by conducting research to develop and provide standards, tests, guidelines, best practices, and lab accreditation assistance that EAC and states and local jurisdictions may use at their discretion. The Director of NIST chairs the Technical Guidelines Development Committee. At the request of the Committee, the Director of NIST provides technical support for the Committee to carry out its duties, such as by participating in election and constituency working groups to provide technical leadership in support of the development of voluntary voting system guidelines. NIST also helps election officials identify and prioritize opportunities to improve their cybersecurity posture. For example, it established a joint working group with the Election Infrastructure Subsector Government Coordinating Council and Subsector Coordinating Council to develop a framework of cybersecurity practices tailored to elections. In doing so, NIST works with the election community to identify the resources and outcomes needed to ensure the security of the election infrastructure. As part of this effort, it receives feedback from states and local jurisdictions, as well as from CISA, through the Election Infrastructure Subsector Government Coordinating Council. DHS, through CISA, has taken steps to assist election officials in securing election infrastructure by providing services in three areas: regional support and assistance, education and awareness, and information sharing and analysis among federal, state, and local organizations. Appendix II provides a list of the services that CISA makes available to states and local election jurisdictions. Regional support and assistance. CISA employs personnel with cyber and physical security expertise in its 10 regional offices throughout the country. According to CISA, as of November 2019, these experts included 24 cybersecurity advisors and 100 protective security advisors who perform and coordinate security assessments for the 16 critical infrastructure sectors, including the Election Infrastructure Subsector. A single advisor may be responsible for performing and coordinating assessments for an entire state or region and across multiple critical infrastructure sectors. The cybersecurity advisors and protective security advisors consult with state and local election officials and identify services that CISA can provide on a voluntary, no cost basis. For example, according to CISA Election Security Initiative officials, cybersecurity advisors and protective security advisors have promoted CISA services and assessments, such as an assessment of network security vulnerabilities and an assessment of risks associated with information and communication technology suppliers and service providers. In addition, protective security advisors have conducted physical inspections of the protections over facilities that store election-related equipment such as voting machines or poll books. Protective security advisors told us that they also provide a web-based tool that states or local jurisdictions can use to identify security gaps and preparedness across facilities. CISA officials stated that, although regional personnel promote cybersecurity and physical security services to election officials, personnel based at CISA headquarters conduct the more advanced cybersecurity assessments. For example, the Vulnerability Management Branch provides vulnerability scanning and risk and vulnerability assessments, while the Threat Hunting Branch responds to cyber incidents. In September 2019, officials from the Election Security Initiative told us that, based on the CISA Director’s guidance, the agency gives requests from election infrastructure stakeholders a higher level of priority than requests from the other sectors. The precise length of the wait for service depends on the type of service. For some services, such as vulnerability scanning, there is no wait time, according to CISA officials, because CISA can activate the service within 24 hours. Education and awareness. CISA disseminates educational materials to raise awareness of election security-related issues and services available to state and local election officials. For example, CISA provides a web- based training course to help election officials understand the principles of information technology management and has developed guidance to help states and localities adopt recommended information technology practices to improve their security posture. According to CISA, as of November 2019, 1,201 individuals had completed the online course. Further, CISA conducted two election infrastructure tabletop exercises known as “Tabletop the Vote” in August 2018 and June 2019 to help the Election Infrastructure Subsector community collaborate and identify best practices and areas for improvement in election-related cyber incident planning, identification, response, and recovery. The 2018 tabletop exercise included 44 states, the District of Columbia, 16 federal entities, the National Association of Secretaries of State, and the National Association of State Election Directors. According to CISA officials, the June 2019 exercise included 47 states, the District of Columbia, 15 federal entities, the National Association of Secretaries of State, the National Association of State Election Directors, the National Governors Association, and the National Conference of State Legislatures. CISA officials also noted that CISA personnel, including regional personnel, have presented at numerous national and state meetings of election officials, such as the Election Center’s annual conference in August 2019. In addition, as part of CISA’s Last Mile initiative, the agency collaborates with state and local election officials to create customized posters that highlight efforts to strengthen election security. The purpose of the posters is to describe the state’s or local jurisdiction’s election infrastructure assets and systems, characterize risks, and offer specific measures it should implement to mitigate those risks. Election officials can present the posters to voters, lawmakers, and their own personnel to bolster confidence in the security of their election systems. As of November 2019, CISA reported that it had delivered Last Mile posters to 19 states (including six states since the 2018 election) and 1,202 local election jurisdictions. Information sharing and analysis. CISA collects and analyzes election security-related information—such as threat indicators, incident alerts, and vulnerability data—and shares this information with election officials to help them assess cybersecurity controls, detect threats, and mitigate risks. To further this goal, CISA partnered with the Center for Internet Security and the Election Infrastructure Subsector Government Coordinating Council to create the Election Infrastructure Information Sharing and Analysis Center (EI-ISAC) in February 2018. State and local election offices can join the EI-ISAC at no cost and receive election-focused cyber defense tools and products. According to the Director of the EI-ISAC, as of November 2019, its members included 50 states, the District of Columbia, and 2,267 local jurisdictions. CISA officials stated that the EI-ISAC is the primary mechanism that CISA uses to exchange information throughout the election community. For example, the EI-ISAC produces a quarterly threat report to assist the election community in the analysis of active information security threats. From its inception through September 2019, the EI-ISAC had sent out 263 alerts to its members, including weekly, spotlight, and other emails, according to EI-ISAC officials. EI-ISAC officials added that CISA funds the EI-ISAC to, among other things, deploy an intrusion detection sensor in each state specifically for voter registration systems and other supporting infrastructure to detect malicious activity and provide network security alerts. CISA officials stated that the agency, in coordination with the EI-ISAC, analyzes data from these sensors to identify trends in threats and vulnerabilities across states and local jurisdictions. CISA also manages the National Cybersecurity and Communications Integration Center (NCCIC), which receives reports of suspected malicious cyber activity from state and local officials, analyzes attempts to infiltrate election systems, and shares information about threats and vulnerabilities through the EI-ISAC. The NCCIC has also assisted election officials in responding to incidents, upon request. According to CISA officials, in fiscal years 2018 and 2019, NCCIC’s Hunt and Incident Response Teams provided services to 10 states and 16 local election jurisdictions, such as incident response activities and proactive reviews for malicious activity at the time of service. Table 2 identifies selected services that CISA provided to states and local jurisdictions in 2018 and 2019, as of November 6, 2019. State election officials with whom we spoke were generally satisfied with CISA’s support to secure their election infrastructure. Specifically, officials from seven of the eight states we contacted said that they were very satisfied with CISA’s election-related work, while officials from the eighth state said that they were somewhat satisfied. Officials from five states told us that their relationship with CISA had improved markedly since early 2017, when the elections subsector was established. For example, state officials said that CISA has made progress in this area. The Secretary of Homeland Security’s designation of elections as critical infrastructure was initially controversial among state and local officials. For example, in February 2017, the National Association of Secretaries of State voted to oppose the designation of elections as critical infrastructure, citing the states’ constitutional authority to regulate elections. In addition, CISA officials told us that a lack of trust and communication between DHS and state and local election officials hindered initial efforts to establish the Election Infrastructure Subsector. However, officials from one state told us that, despite initial reservations about DHS’s role in election security, CISA has become a good partner over time. An official from another state expressed appreciation that CISA appears to be honestly and earnestly working to gain states’ trust. Officials representing the National Association of Secretaries of State and the National Association of State Election Directors also stated that CISA had worked to improve its relationships with state election officials. According to these officials, CISA has expanded outreach efforts by attending state association meetings and conferences to present information on CISA’s resources and the threat environment and has impressed election officials with the level of detail provided by CISA’s threat reporting. An official from the Election Center, which represents local election officials, stated that CISA officials attend every cybersecurity and critical infrastructure event hosted by the center. CISA officials stated that, while it is not possible to meet individually with all of the local election jurisdictions nationwide, it can engage with multiple local election jurisdictions at one time at these association conferences. Election officials from selected states and local jurisdictions cited various benefits from CISA’s support to election security. According to officials from six states, CISA’s involvement in election security has increased the officials’ understanding of the threat environment that the election community faces. They also said that CISA’s involvement has helped them to plan for cybersecurity threats and to prioritize their election security efforts. For example, officials from one state said that CISA recommended that the state set priorities and focus on risk assessments and network segmentation. In addition, election officials from five states spoke highly of CISA’s expertise and availability. The officials said, for example, that CISA regional and headquarters personnel were easy to get in touch with and knowledgeable about the election community. As a result, the officials said that they had better access to training opportunities and informal advice. Further, officials from each of the eight states spoke positively about the information that the officials received from the EI-ISAC. For example, state officials said that the EI-ISAC updated them regularly on election security incidents and vulnerabilities nationwide, allowing them to prepare for potential incidents. Officials also stated that the EI-ISAC presented the information in a way that was understandable to election administrators who may not have backgrounds in information technology. For example, in a monthly “spotlight” email, the EI-ISAC defines a key cybersecurity term and explains to the election officials why it should matter to them. One official told us that through membership in the EI-ISAC, the state has learned about election security best practices from other states, and other officials said that EI-ISAC allows them to maintain visibility of nationwide threats and other election security issues. Officials from one state said that their contacts through the EI-ISAC helped them to identify a point of contact at social media companies so that they could inform the companies about election-related misinformation being spread online. In addition, election officials from two states said that they have encouraged or required local election jurisdictions to enroll in the EI-ISAC. According to EI-ISAC officials, the number of local election jurisdictions enrolled in the EI-ISAC increased from 1,384 at the end of 2018 to 2,267 in November 2019, and included all three election jurisdictions that we contacted. Officials from two of the three local jurisdictions said that the EI-ISAC emails were valuable. For example, election officials from one local jurisdiction said that communication from the EI-ISAC is meaningful and targeted to bolster their election security efforts. Election officials from the other jurisdiction said that they use the EI-ISAC information to improve their continuity of operations plans. On the other hand, officials from the third local jurisdiction said that the information provided in EI- ISAC emails was too general and not specific enough to their circumstances. Election officials from five states also spoke positively of the EI-ISAC situational awareness chat rooms, which DHS hosted on its Homeland Security Information Network. These officials stated that they participated in and monitored the chat rooms on Election Day to maintain awareness of any emergent election security issues nationwide. For example, officials from one state said that the chat rooms helped them receive real time notification of issues in other states and possible solutions to those issues. In addition, as previously mentioned, one of CISA’s major efforts was the 3-day tabletop exercises held in August 2018 and June 2019, which state and local officials were able to attend remotely by video teleconference from sites around the country. Elections officials from five states said that the exercises conveyed important information and prompted thoughtful discussions among state and local officials. Election officials from four states said that the exercises helped them build relationships within their states, and election officials from three states also said the exercises helped to build relationships with federal agencies as well. In addition, officials from four states said that they conducted or were planning to conduct tabletop exercises modeled on CISA’s exercises. Election officials from three states said that CISA’s cybersecurity assistance has helped them to assure voters that elections in their states are secure or to promote election security efforts. For example, officials from one state said that, when they get questions from the public about election security, they tell voters that CISA’s assessments have shown that the state’s election systems are free of malicious code. Election officials from another state said that CISA officials’ outspokenness has created opportunities for state officials to discuss the importance of election security issues with local officials. Additionally, officials from five states told us they encourage local election officials to request election security services from CISA to increase the security posture of the local jurisdictions. Even though state and local election officials provided mostly positive feedback on DHS’s election security assistance, officials also identified two challenges linked to DHS’s assistance efforts. First, officials from three states stated that it is challenging to find time to schedule election security services. For example, officials from one state said that their biggest challenge is to find time in their state’s election schedule for receiving CISA services because the state has seven to nine elections in off years (that is, years without congressional or presidential elections). Officials from another state said that they might have requested additional election services from CISA if the state had more time in its election calendar. However, none of the state officials with whom we spoke attributed this difficulty to CISA, as election calendars are outside of CISA’s control. In commenting on this challenge, CISA officials said that they have tried to accommodate states’ and local election jurisdictions’ needs, when possible. For example, CISA started offering remote penetration testing as an alternative to the risk and vulnerability assessment. The officials said that the two services are similar, but the remote penetration testing can be completed in fewer days and does not require CISA personnel to be physically present in the election offices. CISA officials told us that smaller jurisdictions sometimes prefer this option. Election officials also identified an additional challenge related to the intelligence briefings that were provided by DHS’s Office of Intelligence and Analysis for state and local officials with security clearances leading up to the 2018 elections. According to Office of Intelligence and Analysis officials, the briefings allowed state and local officials to become more informed about the national threat picture, which in turn, allowed them to adjust to the threat more effectively. Election officials from two states said that the intelligence briefings had provided helpful contextual information about cyber threats. However, election officials in two other states said that the briefings were not as useful as the election officials had hoped because the briefings only provided information that was already available publicly, and election officials from another state said that they learned about a significant election security issue possibly related to their state through news reports. For example, an election official from a different state said that the state learned about threats from the Department of Justice’s July 2018 indictment against foreign intelligence officers. CISA officials stated that they are aware of this issue and have been trying to improve the communication of intelligence information to state and local election officials. For example, at a October 2019 hearing of the House of Representatives Committee on Homeland Security, a CISA senior cybersecurity advisor testified that DHS has begun working with the Intelligence Community to rapidly declassify relevant intelligence or provide as much intelligence as possible, at the lowest classification level possible, to state and local election officials. CISA officials also told us that the agency has started working with cybersecurity intelligence firms to provide election security information to state and local officials without the need for national security clearances or travel to secure facilities. According to CISA officials, two cybersecurity intelligence firms provided webinars to election officials in September and October 2019. CISA officials said that these firms have sophisticated capabilities that they use to analyze information that is not classified. As a result, the cybersecurity intelligence firms can more easily share information with states and local election jurisdictions. CISA officials said that state and local officials will benefit from these briefings because they provide actionable threat information to election officials without requiring them to have security clearances or travel to secure facilities. According to DHS planning guidance, strategic-level planning provides a framework for guiding homeland security activities and generates the objectives and priorities, which influence the roles, responsibilities, and actions that are detailed in the operational-level plans. Further, subsequent operational-level plans are to identify the tasks and resources needed to execute strategic plans. Prior GAO work has shown that strategic and operations plans can help further define capabilities, including opportunities to leverage resources. Such plans can also provide a roadmap for addressing identified gaps and better position an agency and its components to work collaboratively and strategically with external partners, such as states and local jurisdictions. CISA has begun developing strategic and operations plans for assisting states and local jurisdictions in securing election infrastructure in preparation for the 2020 elections. Specifically, CISA has developed a draft strategic plan for securing election infrastructure, known as the #Protect2020 Strategic Plan. According to the draft, CISA intends for its strategic plan to be used to achieve the high-level goals and outcomes called for in the agency’s August 2019 Strategic Intent. The draft strategic plan focuses on four areas, also referred to as lines of effort: (1) protecting election infrastructure, (2) supporting political campaigns, (3) raising public awareness on foreign influence threats and building resilience, and (4) sharing intelligence and identifying threats. In addition, the draft strategic plan identifies several objectives for each line of effort. For example, it includes three objectives for the protecting election infrastructure line of effort: building stakeholder capacity to manage risks and handle adversaries, through activities such as creating incident response and communication plans and encouraging states to adopt and practice them; providing technology services to stakeholders to monitor and secure their networks, by promoting the use of CISA’s voluntary services and assessments, among other things; and facilitating information sharing between the federal government, private sector, and state and local partners by, among other things, hosting situational awareness chat rooms prior to, during, and after state and federal elections. As another example, the draft strategic plan identifies three objectives for the sharing intelligence and identifying threats line of effort: partnering with private sector firms and vendors to improve cyber threat intelligence, through activities such as developing threat indicators and warnings; cooperating across federal partners—including federal law enforcement and the Intelligence Community—by, among other things, advocating for the creation of a joint memorandum to consolidate and highlight current knowledge on election threat intelligence; and monitoring threat activity through actions such as using network monitoring capabilities to spot malicious activity and reveal key trends. In addition, CISA officials stated that the agency has begun developing a draft operations plan, known as the 2020 Election Security Operations Plan. This plan is to—in conjunction with the strategic plan—describe key organizational functions, processes, and resources employed to carry out the agency’s efforts in support of elections in 2020. CISA officials stated, as of November 2019, that the agency intended to finalize the strategic and operations plans by January 2020. However, as of January 2020, CISA’s plans were not yet complete. According to a CISA official, the plans were not finalized due to an ongoing reorganization within CISA and limited staffing resources within the Election Security Initiative. While CISA has drafted the strategic plan, the agency has not yet completed a draft of its operations plan. CISA officials have noted the importance of the operations plan to help ensure the agency is adequately prepared to support election officials in securing election infrastructure in advance of elections, which begin with presidential primaries in February 2020, as well as subsequent primaries leading up to the November 2020 general election. Further, CISA’s operations plan may not fully address the four lines of effort outlined in its strategic plan when finalized. Specifically, according to CISA officials, the operations plan is expected to identify organizational functions, processes, and resources for certain elements of two of the strategic plan’s lines of effort—protecting election infrastructure and sharing intelligence and identifying threats. However, agency officials did not identify the extent to which the operations plan would address all of the objectives from these lines of effort in the strategic plan. CISA officials also stated that the agency is unlikely to develop additional operations plans for the other two lines of effort—providing security assistance to political campaigns, and raising public awareness on foreign influence threats and building resilience. The officials stated that, given the limited amount of time remaining before election preparation activities commence, the agency decided to prioritize developing a plan for the first line of effort that addresses the primary customers of the agency’s election services. In the absence of completed strategic and operations plans, a CISA official in one region stated in October 2019 that the region is moving forward with its own strategy for assisting states and local jurisdictions because the 2020 election cycle is scheduled to start with state primary elections in the region in March 2020. The lack of finalized plans can affect CISA’s achievement of higher-level objectives that take time to accomplish, such as building stakeholder capacity and public awareness. Until CISA finalizes its strategic and operations plans for supporting elections in 2020 and ensures that the operations plan fully addresses all of the aspects of its strategic plan, CISA will not be well-positioned to execute a nationwide strategy for securing election infrastructure prior to the start of 2020 election activities. DHS’s National Infrastructure Protection Plan, which provides strategic direction for national, critical infrastructure protection efforts, calls for sector-specific agencies to coordinate lessons learned and corrective actions and rapidly incorporate them to improve future efforts. Further, GAO’s Standards for Internal Control calls for management to document corrective action plans to remediate internal control deficiencies in a timely manner following the reporting and evaluation of issues. CISA has identified various challenges related to its election assistance efforts; however, the agency has not yet documented plans that address them. Following the 2018 midterm elections, CISA and the RAND Corporation conducted two reviews of CISA’s efforts supporting the elections, in order to inform strategic planning and strengthen future operations. The first review, conducted by the RAND Corporation under a contract with DHS, assessed election security operations that CISA undertook from January 2017 through the November 2018 midterm elections. The review relied upon input from DHS personnel, the EI- ISAC, associations representing state election officials, and election system vendors to identify lessons learned from CISA’s activities to assist in securing election infrastructure. In the second review, CISA conducted an after action review covering its efforts to assist in securing election infrastructure from September 2018 to December 2018, based on input from personnel within DHS and its federal partners who participated in the agency’s election security operations. Both reviews identified various challenges that CISA needed to address in its planning for 2020. For example, the RAND review cited challenges related to the services and threat briefings CISA provided to states and local jurisdictions. The review noted, among other things, that CISA: lacked an approach for prioritizing its activities based on election security risks, which could limit the agency’s ability to dedicate increased attention and resources to the jurisdictions with the highest risk; did not adequately tailor services, which could have made it more difficult to meet the resource and time constraints of customers such as local election jurisdictions; and did not always provide actionable recommendations in DHS classified threat briefings or make unclassified versions of the briefings available, which may have hindered election officials’ ability to effectively communicate with information technology and other personnel in their agencies who did not have clearances. Additionally, the CISA after action report identified a number of internal operational challenges associated with its election-related efforts in 2018. For example, the report cited: a lack of understanding by CISA headquarters staff of the roles and functions of regional field staff, which led to redundant requests for information from headquarters staff to regional staff; the lack of a single agency-wide platform to maintain an awareness of election threats, which resulted in confusion among CISA personnel about which threat information was accurate and current; and the inability of CISA personnel supporting election security operations to access social media websites from situational awareness rooms, which hindered their collection and analysis of threat information. Further, both reviews cited challenges regarding CISA’s ability to manage incident information and provide Election Day incident response capabilities in the event of a compromise. For example, with regards to the 2018 election, the reviews noted: few capabilities that CISA field staff could quickly provide on Election Day, which could limit the agency’s timeliness in mitigating or responding to an incident; a lack of clarity regarding CISA’s incident response capabilities in the event of a compromise that exhausts state and local resources, which may limit knowledge about agency capabilities that are available; and a lack of outreach and situational reporting on incidents, threats, and trends on Election Day from headquarters to regional staff following the closure of the polls on the East Coast, which hindered CISA’s coordination of such information with state and local officials. While CISA identified challenges related to its prior efforts, it has not developed plans to address them. According to a CISA official, the agency does not intend to develop a separate plan addressing how it will remediate the identified challenges in the RAND report. Rather, CISA officials noted that the agency plans to address the challenges from that report in the strategic plan and operations plan that it is developing. In addition, the officials noted that CISA may address challenges through other actions that the agency expects to take, such as hiring additional staff. However, CISA’s draft strategic plan, as of November 2019, had only addressed three challenges from the RAND report—countering the threat of disinformation, clarifying how CISA is to support political campaigns, and prioritizing outreach to local jurisdictions. The extent to which the strategic plan, when finalized, will address the other outstanding challenges remains unclear. In addition, the extent to which the operations plan will document how the agency is to address challenges in the RAND report remains uncertain as the operations plan has not yet been completed. Further, CISA has not documented how the agency is to address challenges in the RAND report through other actions that it expects to take before the 2020 elections. Similarly, CISA officials stated that the agency intends to address a subset of the challenges from CISA’s after action report in its anticipated operations plan. However, the extent to which the operations plan will document how the agency is to address challenges in the after action report remains unclear, given that the operations plan has not yet been completed. Without documented plans that address prior challenges, CISA will not be well-positioned to effectively address the challenges identified in prior reviews. This includes addressing how CISA will coordinate among its personnel and provide accurate threat information and other capabilities that address the needs of the election infrastructure community in the remaining months ahead of the 2020 elections. With primary elections beginning in February 2020 and culminating in the general election in November 2020, CISA has limited time remaining to help states and local election jurisdictions protect their election infrastructure in advance of these elections. State and local election officials that we contacted have been generally satisfied with CISA’s election security efforts. However, CISA’s unfinished planning means the agency may be limited in its ability to execute a nationwide strategy for securing election infrastructure. In particular, the #Protect2020 Strategic Plan’s higher-level objectives—such as building stakeholder capacity and public awareness—necessarily take time to accomplish. In addition, CISA has not fully assessed and documented how it will address challenges identified in prior assessments, which limits the ability of CISA to address these challenges in its current efforts. We are making three recommendations to the Director of the Cybersecurity and Infrastructure Security Agency: The CISA Director should urgently finalize the strategic plan and the supporting operations plan for securing election infrastructure for the upcoming elections. (Recommendation 1) The CISA Director should ensure that the operations plan fully addresses all lines of effort in the strategic plan for securing election infrastructure for the upcoming elections. (Recommendation 2) The CISA Director should document how the agency intends to address challenges identified in its prior election assistance efforts and incorporate appropriate remedial actions into the agency’s 2020 planning. (Recommendation 3) DHS provided written comments on a draft of this report, which are reprinted in appendix III. In its comments, the department concurred with all three of our recommendations and identified actions that it plans to take to implement each of the recommendations. For example, the department stated that CISA intends to finalize its strategic and operations plans by February 14, 2020. The department noted that these plans are to provide a strategic overview and operational framework in support of the primaries and the general election in 2020. Further, the department stated that the operations plan, when finalized, is to address all lines of effort in the strategic plan. In addition, the department noted that both the strategic and operations plans are to further document DHS’s plans to address challenges identified during the 2017-2018 election cycle. If implemented effectively, the actions that DHS plans to take in response to the recommendations should address the weaknesses that we identified during our review. DHS and CISA officials also provided technical comments, which we incorporated, as appropriate. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, the Director of the Cybersecurity and Infrastructure Security Agency, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact Vijay A. D’Souza, Director, Information Technology and Cybersecurity, at (202) 512-6240 or dsouzav@gao.gov or Rebecca Gambler, Director, Homeland Security and Justice, at (202) 512-8777 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. The election process relies on various assets—such as information technology systems, networks, equipment, and facilities—that can be broadly categorized as physical, cyber, and human components of the Election Infrastructure Subsector. The assets and components of the election infrastructure are susceptible to a variety of unintentional, or nonadversarial, and intentional, or adversarial, threats. The table below identifies sources of cybersecurity threats to election infrastructure. The Department of Homeland Security, through the Cybersecurity and Infrastructure Security Agency (CISA), has taken steps to assist state and local election officials in securing election infrastructure by providing a variety of services on a voluntary, no cost basis. These services include cybersecurity assessments, detection and prevention activities, and information sharing. The table below identifies voluntary services that CISA offers to states and local election jurisdictions. In addition to the contacts named above, Josh Leiling (Assistant Director), Tom Jessor (Assistant Director), Torrey Hardee (Analyst-in-Charge), Roger Bracy, Rebecca Eyler, Richard Hung, Amanda Miller, Heidi Nielson, Monica Perez-Nelson, Jeff Tessin, Eric Warren, and Haley Weller made significant contributions to this report.", "summary": "In January 2017, the Secretary of Homeland Security designated election infrastructure as a critical infrastructure subsector. The designation allowed DHS to prioritize assistance to state and local election officials to protect key election assets, including voter registration databases and voting equipment. The Conference Report (H. Rep. No. 116-9) accompanying the 2019 Consolidated Appropriations Act included a provision for GAO to examine how DHS is implementing key responsibilities to help protect the election infrastructure and the reported benefits and challenges of such efforts. This report addresses (1) DHS's election security efforts and selected election officials' perspectives on them, and (2) DHS's planning for the 2020 elections. GAO reviewed DHS's strategies, plans, and services provided to election officials. GAO also interviewed DHS officials, representatives of the EI-ISAC, a DHS-funded center responsible for sharing threat information nationwide, and election officials from eight states and three local jurisdictions. GAO selected the states and local jurisdictions to provide geographic diversity and variation in election administration, among other factors. The results from these states and localities are not generalizable, but provide insight into election officials' perspectives on DHS's efforts. Since the 2017 designation of election infrastructure as critical infrastructure, the Department of Homeland Security (DHS), through its Cybersecurity and Infrastructure Security Agency (CISA), has assisted state and local election officials in securing election infrastructure through regional support and assistance, education, and information sharing. Such efforts help state and local election officials protect various election assets from threats (see figure). In August 2019, the CISA Director identified election security as one of the agency's top five operational priorities. CISA security advisors, who are located throughout the country, consult with state and local election officials and identify voluntary, no cost services that CISA can provide. According to CISA, as of November 2019, 24 cybersecurity advisors and 100 protective security advisors perform and coordinate cyber and physical security assessments for the 16 critical infrastructure sectors, including the Election Infrastructure Subsector. Technical teams at CISA headquarters generally provide the services, once requested. To further assist state and local election officials, CISA conducted two exercises simulating real-world events and risks facing election infrastructure in August 2018 and June 2019. According to CISA, the 2019 exercise included 47 states and the District of Columbia. In addition, CISA has funded the Election Infrastructure Information Sharing and Analysis Center (EI-ISAC). According to CISA officials, the EI-ISAC is the primary mechanism for exchanging information about threats and vulnerabilities throughout the election community. The EI-ISAC director reported that, as of November 2019, its members included 50 states, the District of Columbia, and 2,267 local election jurisdictions, an increase from 1,384 local jurisdictions that were members in 2018. As a result of its efforts, CISA has provided a variety of services to states and local election jurisdictions in the past 2 years (see table). State election officials with whom GAO spoke were generally satisfied with CISA's support to secure their election infrastructure. Specifically, officials from seven of the eight states GAO contacted said that they were very satisfied with CISA's election-related work. Also, officials from each of the eight states spoke positively about the information that they received from the EI-ISAC. Further, officials from five states told GAO that their relationship with CISA had improved markedly since 2017 and spoke highly of CISA's expertise and availability. To guide its support to states and local election jurisdictions for the 2020 elections, CISA reported that it is developing strategic and operations plans. CISA intended to finalize them by January 2020, but has faced challenges in its planning efforts due to a reorganization within CISA, among other things. In the absence of completed plans, CISA is not well-positioned to execute a nationwide strategy for securing election infrastructure prior to the start of the 2020 election cycle. Further, CISA's operations plan may not fully address all aspects outlined in its strategic plan, when finalized. Specifically, according to CISA officials, the operations plan is expected to identify organizational functions, processes, and resources for certain elements of two of the four strategic plan's lines of effort—protecting election infrastructure, and sharing intelligence and identifying threats. CISA officials stated that CISA was unlikely to develop additional operations plans for the other two lines of effort—providing security assistance to political campaigns, and raising public awareness on foreign influence threats and building resilience. Moreover, CISA has not developed plans for how it will address challenges, such as concerns about incident response, identified in two reviews—one conducted by CISA and the other done by an external entity under contract—of the agency's 2018 election security assistance. Challenges that the reviews identified include: inadequate tailoring of services, which could have made it more difficult for CISA to meet the resource and time constraints of customers such as local election jurisdictions; not always providing actionable recommendations in DHS classified threat briefings or making unclassified versions of the briefings available, which may have hindered election officials' ability to effectively communicate with information technology and other personnel in their agencies who did not have clearances; the inability of CISA personnel supporting election security operations to access social media websites from situational awareness rooms, which hindered their collection and analysis of threat information; few capabilities that CISA field staff could quickly provide on Election Day, which could limit the agency's timeliness in responding to an incident; and a lack of clarity regarding CISA's incident response capabilities in the event of a compromise that exhausts state and local resources, which may limit knowledge about agency capabilities that are available. Although CISA officials said that the challenges identified in the reviews have informed their strategic and operational planning, without finalized plans it is unknown whether CISA will address these challenges. GAO is making three recommendations to the CISA Director to (1) urgently finalize the strategic plan and the supporting operations plan for securing election infrastructure for the upcoming elections, (2) ensure that the operations plan fully addresses all lines of effort in the strategic plan for securing election infrastructure for the upcoming elections, and (3) document how the agency intends to address challenges identified in its prior election assistance efforts and incorporate appropriate remedial actions into the agency's 2020 planning. DHS concurred with all three recommendations and provided estimated dates for implementing each of them.", "document_type": "gao"}
{"report": "Most communities in the nation experience some kind of flooding, which may occur after substantial spring rains, heavy thunderstorms, winter snow thaws, or heavy storms over a large body of water. Flood risk management includes the appropriate use of structures such as levees and floodwalls, as well as nonstructural measures such as land acquisition and structure relocation, to reduce the risk of loss of life, reduce long-term economic damage to the public and private sectors, and improve the natural environment. Flood risk management is one of the Corps’ three primary missions. For fiscal years 2015 through 2017, the Corps requested more than $3 billion for 71 construction projects that fell within its three missions, of which the largest amount—$1.33 billion—was for 33 construction projects in the flood risk management mission. Located within the Department of Defense, the Corps has both military and civilian responsibilities. Through the Civil Works Program, the Corps plans, constructs, operates, and maintains a wide range of water resources development projects such as navigation and flood risk projects. The Assistant Secretary of the Army for Civil Works, appointed by the President and confirmed by the Senate, sets the strategic direction for the program and has principal responsibility for the overall supervision of functions relating to the Army’s Civil Works Program. The Chief of Engineers, a military officer, is responsible for execution of the civil works and military missions. The Civil Works Program is organized into three tiers: headquarters in Washington, D.C.; eight regional divisions; and 38 local district offices. (See fig. 2.) The Corps develops water resource projects, including flood risk management projects, in conjunction with nonfederal sponsors such as state and local governments. According to Corps guidance, the planning process for these projects begins with the nonfederal sponsor identifying a problem and approaching the Corps to help develop a solution. Upon congressional authorization for a study and appropriations to fund it, the Corps and the nonfederal sponsor establish an agreement to conduct a feasibility study for a potential project. The Corps initiates a feasibility study by forming a project team comprised of Corps engineers, economists, planners, and possibly other specialists such as nonfederal consultants to conduct the study. The planning process the Corps uses to carry out feasibility studies is described later in our report. Nonfederal sponsors are to participate in the planning process, as well as remain involved through project design, construction, and post-project operations and maintenance. For example, for projects in which the Corps constructs infrastructure such as a flood wall, the nonfederal sponsor is to assume responsibility for monitoring and maintenance costs associated with the flood wall after its construction. The U.S. Water Resources Council’s Principles and Guidelines outlines the principles and procedures the Corps is to follow for planning water resources development projects, including those with flood risk management objectives. The Principles and Guidelines states that the federal objective of water resources development projects is to contribute to national economic development while protecting the nation’s environment. The Corps implements the planning process outlined in the Principles and Guidelines by conducting feasibility studies for proposed water resources development projects. The Corps’ Planning Guidance provides detailed guidance on how to implement the general process outlined in the Principles and Guidelines for planning water resource projects. The Corps’ National Economic Development manuals provide supplemental guidance for the economic analysis of different types of projects—including flood risk management—and how to evaluate the benefits and costs associated with each type of project. To identify the beneficial and adverse effects of each alternative plan considered for a project, the Corps uses four categories of analysis established in the Principles and Guidelines: (1) National Economic Development, (2) Environmental Quality, (3) Regional Economic Development, and (4) Other Social Effects, as shown in table 1. The Corps’ Planning Guidance states that feasibility studies may evaluate the effects of alternative plans using the four categories of analysis, but the evaluations under two categories—National Economic Development and Environmental Quality—must be presented in each feasibility study. According to the Corps’ Planning Guidance, the National Economic Development category requires an economic analysis of each plan’s potential economic benefits and costs in monetary terms, while the Environmental Quality category evaluates each plan’s potential nonmonetary effects such as effects on habitat quality and quantity. The Planning Guidance states that using these categories of analysis provides a basis for determining which alternative plans should be eliminated from consideration, modified, or selected for further analysis. The Corps’ followed the six-step planning process for water resources development projects outlined in its Planning Guidance to identify and evaluate the beneficial and adverse effects of alternative plans for flood risk management projects and select a recommended plan for the eight feasibility studies we reviewed. In the initial three steps of the planning process, the Corps (1) identified the objectives and other parameters of the project; (2) inventoried and forecasted water and related land resources conditions within the planning area; and (3) formulated alternative plans for further consideration. In the final three steps of the planning process, the Corps (1) evaluated and analyzed each alternative plan for its economic, environmental, and other effects, (2) compared the alternative plans to each other, and (3) selected a recommended plan. Corps officials told us that this six-step process is the basic template for planning water resources development projects across all Corps mission areas. (See fig. 3.) For each of the eight studies we reviewed, the Corps followed this template and addressed each of the six steps in planning the proposed flood risk management project, as we describe below. Each study identified objectives, problems, opportunities, and constraints for the project. According to the Corps’ Planning Guidance, identification of problems and opportunities is the foundation for scoping the planning process and should begin as soon as practicable after the decision to initiate a feasibility study. Planning objectives describe the desired results of the process by solving the problems and taking advantage of the opportunities identified. Constraints are restrictions that limit the planning process and are unique to each study. Such constraints can be, for example, limitations imposed by policy or law. All of the studies we reviewed had the objective of reducing or managing flood risk and damages in response to problems such as historic river or stream flooding in the planning area. The studies identified opportunities, such as improving the community’s understanding of flood risk and resiliency from flood events. The studies also identified constraints, such as the need for the plan to incorporate extensive transportation infrastructure within some of the planning areas. The studies inventoried historic and existing water and related land resource conditions and forecasted future conditions within the planning area relevant to the identified problems and opportunities from step one. According to the Corps’ Planning Guidance, the Corps is to use quantitative and qualitative descriptions of critical resources in the planning area to define existing and future without-project conditions— that is, the conditions if no project is constructed. The defined without- project conditions provide the basis from which the Corps formulates alternative plans and assesses impacts. The studies we reviewed inventoried the existing conditions for the planning area. This inventory included geology, groundwater, surface water, hydrology, water quality, biological resources, cultural resources, land use, recreation, air quality, climate change, transportation, public health and safety, public services, utilities, socioeconomics, and environmental justice. The Corps used these existing conditions to forecast the future without-project conditions, such as increasing flood risk for residential and industrial development, culturally significant communities, or specific infrastructure such as a regional wastewater facility. The studies formulated alternative plans for the project, including a range of structural and nonstructural measures and strategies. According to the Corps’ Planning Guidance, an alternative plan consists of a system of management measures, that is, structural and/or nonstructural measures, strategies, or programs formulated to meet the project objectives subject to the planning constraints. The Corps is to identify a range of alternative plans at the beginning of the planning process, screen the plans, and refine them in subsequent iterations throughout the planning process. The Planning Guidance also states that as the Corps develops the alternative plans, it must consider the criteria of completeness, efficiency, effectiveness, and acceptability. In the eight studies we reviewed, the Corps followed an iterative approach to identify measures and form alternative plans. For example, the studies generally identified an initial array of structural and nonstructural measures for conceptual screening, followed by the grouping of viable measures into alternative plans for screening under the criteria, resulting in an array of plan alternatives for more detailed analysis of the beneficial and adverse effects (monetary and nonmonetary) of each. According to Corps officials, flood risk management studies must consider a minimum of two plans— no action and an alternative—and one of the plans considered must be nonstructural. All eight studies we reviewed adhered to this requirement and considered a variety of alternative plans for each proposed flood risk management project. The studies evaluated each alternative plan—including its beneficial and adverse effects—through a comparison of the with-project and without- project conditions. According to the Corps’ Planning Guidance, evaluation consists of (1) forecasting the most likely with-project (e.g., with the alternative plan constructed) condition expected under each alternative plan; (2) comparing each with-project condition to the without-project condition and documenting the differences between the two; (3) characterizing the beneficial and adverse effects; and (4) identifying the plans that will be further considered in the planning process. The studies we reviewed used the categories established in Corps guidance— the National Economic Development and Regional Economic Development categories for monetary benefits and costs and the Environmental Quality and Other Social Effects categories for nonmonetary (quantitative and qualitative) effects—to evaluate and display the beneficial and adverse effects of plan alternatives. The categories and specific types of monetary benefits and costs and nonmonetary effects that the Corps evaluated varied for each study depending on the planning area conditions and the measures and strategies included in the alternative plans. In the studies we reviewed, the economic analyses of monetary effects generally resulted in an estimated net dollar value of benefits (benefits minus costs) expected with each alternative in place, while the analysis of nonmonetary effects generally resulted in a Corps judgment about the net qualitative effect or net quantitative effect (e.g., net units of habitat created) for each alternative. The studies compared the alternative plans based on the economic analysis of benefits and costs and on the evaluations of environmental and other effects. According to the Corps’ Planning Guidance, the alternative plans (including the no-action plan) are to be compared with each other, with emphasis on the outputs and beneficial and adverse effects that will have the most influence in the decision-making process. Such a comparison is to include monetary and nonmonetary benefits and costs and identify and document trade-offs to support the final recommendation. In the studies we reviewed, the Corps compared project effects in a variety of ways, for example, in a series of narratives describing the beneficial and adverse effects of alternative plans, or a grid for side-by-side comparison of selected effects for plan alternatives. In some studies, this comparison included an incremental process in which the Corps considered incorporating additional measures or approaches into an alternative to further optimize the trade-off between beneficial and adverse effects. The result of this step was a final group of plans that the Corps considered for recommendation. The Corps recommended a plan based on the comparison of the alternative plans. According to the Corps’ Planning Guidance, the Corps should recommend a single alternative plan that must be shown to be preferable to taking no action (if no action is not recommended) or implementing any of the other alternatives considered during the planning process. In the studies we reviewed, the recommended plan and the rationale for its selection were identified in the analyses and underwent internal technical review at the district, division, and headquarters levels. The Chief of Engineers signed and submitted the proposed plan for the project—known as the Chief’s Report—to the Office of the Assistant Secretary for review, and the Secretary submitted the report to Congress for authorization. All eight of the studies we reviewed included step 4 of the Corps’ six-step planning process: an economic analysis of the benefits and costs of each proposed project as well as an Environmental Quality analysis, as called for in the Corps’ Planning Guidance. The inclusion of the other two types of analyses—Regional Economic Development and Other Social Effects—are not required, but six of the studies included them. The Principles and Guidelines provide the Corps with general flexibility to choose which benefit and cost categories to include in these analyses. The Corps’ Planning Guidance states the federal government’s and project’s objectives guide the planning process, which includes benefit and cost category selection. The monetary benefits most commonly included in the economic analyses of the Corps feasibility studies we reviewed were reduced damages and emergency costs avoided, as shown in table 2. The Corps included reduced damage benefits in each of the eight studies we reviewed. Reduced damages result from actions such as performing physical modifications to property designed to reduce the frequency of flood damage, relocating structures, or installing flood warning and preparedness systems. For example, a feasibility study for a proposed project in the New York District outlined a plan to modify channels that line the Mamaroneck and Sheldrake Rivers with the goal of reducing the risk of life and property damage within the Village of Mamaroneck. The Corps also included emergency costs avoided as benefits in four of the eight studies we reviewed. Emergency costs include expenses resulting from a flood that otherwise would not be incurred. For example, some of the emergency costs avoided for this proposed project in the New York District included the costs of evacuation, reoccupation, flood fighting, and increased operations, police, fire, and military patrol. Depending on the potential effects of the plan alternatives considered, some studies included monetary benefits from recreation, reduced maintenance costs, flood insurance administrative savings, or reduced transportation disruptions in their economic analyses, but these were not commonly considered in the studies we reviewed. The Corps considered a variety of monetized costs in its economic analyses for feasibility studies we reviewed, as shown in table 3. Among the most commonly included costs in each of the eight studies were for construction; operation, maintenance, repair, replacement, and rehabilitation (OMRR&R); and real estate. Specifically: Construction costs. These are the direct costs of installing project measures. For example, the Honolulu District study included the costs of constructing six in-stream debris and detention basins above a watershed, floodwalls along a canal, an earthen levee, and two pump stations. OMRR&R costs. These represent the current monetary value of materials, equipment, services, and facilities needed to operate the project and make repairs, rehabilitations, and replacements necessary to maintain project measures in sound operating condition during the period of analysis. For example, the Wilmington District study included OMRR&R costs for conducting visual inspections of the levee, mowing twice a year, and conducting video inspections of pipes and culverts every 5 years. Real estate costs. These include activities such as buying out residential structures and demolishing them. For example, the San Francisco District study included real estate costs to acquire approximately 900 acres of city-owned land for ecosystem restoration and levee, road, and temporary work easements. Depending on the potential effects of the plan alternatives considered, some of the studies we reviewed included environmental costs; relocations; planning, engineering, and design; and the costs for cultural resource preservation, recreation, and flood warning systems. In addition to the required economic analysis of benefits and costs, the Corps included other analyses to evaluate monetary and nonmonetary project effects in the flood risk management feasibility studies we reviewed. These included the Environmental Quality, Regional Economic Development, and Other Social Effects analyses. All the studies we reviewed included the Environmental Quality analysis; six studies included the Regional Economic Development or Other Social Effects analyses, as shown in table 4. Corps officials said the additional analyses were included in studies because the analyses were needed to determine the best project design, help make planning decisions, or respond to local sponsors’ preferences. Examples of some additional analyses conducted in different districts include the following: Regional Economic Development effects. In the Sacramento District study, the Corps considered ways reduced flooding could increase local business revenue and short-term construction employment but reduce employment because of loss of damage to businesses, among other effects. The Corps also considered how its expenditures for various services and products during the project were expected to generate additional economic activity, such as through additional jobs, income, and sales. In this case, the Corps estimated the project might add 18,930 jobs in the region. According to a 2011 Corps handbook, considering Regional Economic Development effects can provide a better understanding of the overall impact to the region. Doing so also examines the potential impacts mainly to the localized or regional economic area, instead of the nation as a whole. Other Social Effects. In the Wilmington District study, the Corps considered security of life, health, and safety; preservation of historic significance; and the impacts to cultural resources. According to a 2009 Corps handbook, considering the Other Social Effects analysis has great potential value for better ensuring that water resources solutions address a broad array of issues and concerns that better meet stakeholder needs and expectations. In most of the studies we reviewed, the Corps recommended the alternative plan with the greatest net economic benefits based on the results of its economic analyses. In some cases, however, the Corps relied on other analyses to address different project objectives or the preferences of the local nonfederal sponsors. The Corps’ Planning Guidance directs that the project alternative with the greatest net economic benefit, consistent with protecting the nation’s environment, be selected for recommendation unless an exception is granted. The Assistant Secretary of the Army for Civil Works has the authority to grant exceptions if federal, state, local, or international concerns exist. The Planning Guidance states that projects may deviate from the alternative plan with the maximum net benefits if requested by the nonfederal sponsor and approved by the Assistant Secretary of the Army for Civil Works. Such plan alternatives are referred to by the Corps as the locally preferred plan, with the nonfederal sponsor responsible for any project costs in excess of the costs of the plan with the highest net benefits. The Corps conducted economic analyses in each of the eight studies we reviewed, resulting in a wide range of monetary benefits and costs for the recommended project plan alternatives. Table 5 shows the monetized benefit and cost information that helped the Corps select recommended plans in the eight studies. The annualized project benefits ranged from approximately $500,000 to $210.6 million, and annualized project costs ranged from about $1 million to $65 million, resulting in annual net benefit estimates ranging from approximately -$500,000 to $146 million. For five of the eight studies we reviewed, the Corps primarily used the results of the economic analysis of benefits and costs to recommend a plan with the greatest net benefits from among the alternatives, in accordance with the Planning Guidance. These five studies were with the New York, Honolulu, Sacramento, Nashville, and Kansas City Corps districts. Three of the eight studies we reviewed relied on other analyses as allowed under the Planning Guidance to address different project objectives or the preferences of the local nonfederal sponsors. Corps officials said they relied on other analyses when needed to determine the best project design, help make decisions, or respond to local nonfederal sponsors’ preferences. Specifically: Chicago District. The Chicago District recommended a project based on two separate analyses. Specifically, the project team recommended an alternative plan based on an economic analysis for the flood risk management objective and separate analyses for an ecosystem restoration objective. A Corps document stated that by doing so, the proposed project would help both manage flood risks and restore ecosystems in the watershed. In addition, the study said the recommended plan attempts to maximize the net benefits and find balance between both objectives. Wilmington District. The Wilmington District study indicated that the Corps recommended the locally preferred alternative plan, after receiving approval to do so, instead of the alternative plan with the greatest net benefits at the request of the nonfederal sponsor. The locally preferred alternative plan was recommended so it could incorporate consideration of potential other social effects, such as life and safety risk, and regional economic development, such as employment created during and after construction. By doing so, the study indicated Corps officials responded to local priorities and the recommendations provided by the President’s Council on the Future of Princeville, North Carolina. According to the study, the Corps considered impacts to community cohesion, cultural and historical values, local per capita and household incomes in comparison to national averages, and other factors not captured in an economic analysis. San Francisco District. The San Francisco District study indicated that the Corps based its alternative plan recommendation on a combination of multiple objectives and local preference. The recommended alternative plan’s objectives included reducing the risk of tidal floods as well as restoring the ecosystem to tidal marsh habitat. The Corps selected the recommended alternative plan because the nonfederal sponsor wanted to provide additional transitional habitat and greater flood risk management for Federal Emergency Management Agency accreditation over the 50-year study period. Specifically, the local preference was to build the levee about 3 feet higher than the plan with the greatest net benefits— thereby potentially reducing public health and safety risks associated with flooding more than the alternative plan with the greatest net benefit. The economic analyses for the eight studies we reviewed generally met three of the five key methodological elements and partly met two key elements—analysis of effects and transparency. Our Assessment Methodology for Economic Analysis (Assessment Methodology) identifies five key methodological elements to the baseline structure of an economic analysis. For the analysis of effects element, the Corps has either taken steps to address certain best practices or indicated the agency is limited in adopting other practices due to statutory requirements. For the transparency element, Corps officials acknowledged that transparency could be improved through its review process. According to our Assessment Methodology, an economic analysis should state the action examined and the justification for the action. In addition, the objective of the analysis should be stated; the scope of the analysis should be designed to address the objective; and the analysis period should be long enough to encompass the important economic effects of the proposed action. We found that all eight analyses generally met this key element. For example, all eight economic analyses indicated that the actions examined included the evaluation of flood risk management improvements for resolving flooding problems. In addition, the analyses provided specific planning objectives, such as to reduce flood risks in the relevant watershed over the 50-year analysis period and to improve the quality of life for local neighborhoods. Furthermore, all eight analyses used a 50- year analysis period to analyze benefits and costs—a period that should be long enough to encompass important economic effects, though several studies assumed that economic conditions would remain the same over that time period. For example, the analysis for the Honolulu District’s flood risk management study assumed that the inventory of homes and businesses in the flood plain would not change over the 50- year analysis period. According to the analysis, the project area includes sites that are underutilized or not fully developed, but uncertainty about how development might proceed made it difficult to project what changes might occur. The study acknowledged that changes in the business and residential makeup of the watershed over the 50-year period would occur but that the exact nature of these changes could not be projected with any degree of certainty. In addition, two of the eight studies involved multipurpose projects and specified additional economic-related objectives for ecosystem restoration. For example, the analysis for the San Francisco District’s feasibility study indicated that it was designed to evaluate and compare the economic justification and cost effectiveness of various measures to reduce flood risk and provide ecosystem restoration in South San Francisco Bay. Similarly, the Chicago District’s study indicated that in developing an ecosystem restoration plan, undeveloped lands throughout the watershed were evaluated to determine whether cost-effective aquatic ecosystem restoration at that site was possible and what measures would provide the lowest incremental cost per unit of habitat output. Our Assessment Methodology recommends that an analysis used to examine economic effects should identify and compare alternatives. In addition, the analysis should consider a range of relevant alternatives and should justify that the economic conditions specified under each alternative considered represent the best assessment of conditions under that alternative. We found that all eight economic analyses generally met this key element. For example, all eight economic analyses examined the economic effect of the proposed flood control actions by comparing a range of alternatives, including various structures such as levees or bridge modifications, as well as nonstructural measures such as floodplain management activities or acquisition of land and removal of people from the flood plain. Moreover, the economic analyses in the studies generally described and justified the economic conditions that would be expected under each alternative. For the two studies that also evaluated ecosystem restoration alternatives, the studies considered alternatives for restoring ecosystems. Our Assessment Methodology recommends that the economic analysis be clearly written, include a plain language summary, and provide clearly labeled tables that describe the data used and the results. Also, the analysis should document that it complies with a robust quality assurance process. We found that all eight economic analyses generally met this key element. For example, all eight economic analyses were generally clearly written and included tables that generally described data and results. In addition, seven of the feasibility studies included a plain language summary. Six of the studies indicated that the analyses complied with a robust quality assurance process, in which the analyses were reviewed at the Corps district and by technical and policy experts in headquarters. Corps guidance indicates that the quality assurance process for feasibility studies involves reviews for technical quality and policy compliance, among other considerations, at the Corps district and in headquarters. Further, three studies indicated that an independent external peer review had been conducted. While one study completed in the Nashville District did not indicate whether the study complied with a quality assurance process, district officials told us a thorough review was conducted that included multiple district quality control reviews, agency technical review and headquarters policy reviews, and an independent external peer review. In addition, a study completed in the Chicago District did not indicate that it had undergone an independent external peer review. Our Assessment Methodology recommends that an economic analysis quantify the important costs and benefits and monetize these quantitative effects using the concept of opportunity cost—the maximum worth of a good or input among possible alternatives. The criterion of net present value, or related outcome measures, should be applied to compare these effects across alternatives. In addition, the analysis should control for inflation and use economically justified discount rates. Where important costs and benefits cannot be quantified, the analysis should show how they affect the comparison of alternatives. We identified areas in which the studies did not fully align with certain best practices for various reasons, such as the Corps’ concerns about the reliability of available methods and statutory requirements regarding the use of discount rates. These best practices included: Quantifying and monetizing important benefits and costs. The economic analyses in all eight studies quantified and monetized important benefits and costs associated with each alternative, such as property damage reductions and construction costs. The Corps’ Planning Guidance indicates that studies should consider analyzing loss of life in the Other Social Effects category, in either monetary, quantitative, or qualitative terms. Project alternatives that reduce the risk of flooding or that relocate people from the flood plain may lower the risk that individuals living or working in a flood plain will drown or become injured during flood events. However, the analyses in the eight studies we reviewed generally did not quantify and monetize the effect of project alternatives on loss of life. One of the studies we reviewed quantified these effects, but only for the recommended plan. Specifically, the Sacramento District’s flood risk management study found that the recommended plan, which involved the improvement of an existing levee system, could reduce fatalities during flood events by about 67 percent. Of the other seven studies that we reviewed, six analyses included a qualitative discussion of the effects of alternatives on loss of life, and one analysis did not include an assessment of these effects. A recent National Academy of Sciences study on coastal storm flooding indicated that the practice of quantifying and valuing reductions in loss of life is widespread in the federal government, allowing these risk reductions to be included in the economic analysis. In July 2017, after the eight studies that GAO reviewed were completed, the Corps issued revised guidance requiring flood risk management studies to include a quantitative assessment of loss of life for each alternative when it is a significant factor. Corps officials said they had not attempted to monetize loss of life because of concerns about the reliability of available valuation methods but are monitoring other agencies’ efforts to value these effects and following economic research in the area. Using net present value criterion. Analyses for seven studies we reviewed compared the flood risk management alternatives and identified the alternative expected to maximize net benefits on a comparable, present-value basis (that is, on an “annualized” basis). However, one economic analysis did not clearly indicate whether the costs associated with the flood risk management alternatives were annualized and therefore comparable to the annualized benefits. Controlling for inflation and use of economically justified discount rates. Although all the economic analyses in all eight Corps studies we reviewed controlled for inflation by expressing benefits and costs in “real” terms, the discount rates that the studies used to convert future benefits and costs to present values were in nominal terms. In general, real and nominal values are not combined in the same analysis. Specifically, discounting real benefits and costs with a nominal discount rate understates present values when holding all else the same. Corps officials said that they are aware of this inconsistency, but they have no latitude to use a real discount rate because the Water Resources Development Act of 1974 requires the Corps to use nominal discount rates. Corps officials acknowledged areas in which the eight Corps studies we reviewed partly met the Analysis of Effects key methodological element. However, as noted, the Corps has taken some steps to address one best practice. Specifically, the Corps’ recently revised guidance, which requires quantification of loss of life effects when significant, should allow the Corps to provide decision makers and stakeholders with more precise information about the relative magnitude of these effects in future economic analyses. In terms of the best practice regarding economically justified discount rates, the Corps has not taken steps because it is required to use the statutorily specified nominal discount rates. Our Assessment Methodology recommends that analyses be transparent with respect to their analytical choices, assumptions, and data used. The methodology further recommends (1) evaluating how plausible adjustments to each choice and assumption may impact the estimates of the cost-and-benefit effects and results of the comparison of alternatives and (2) clearly explaining the implications of the key limitations in the data and models used. Where feasible, to ensure transparency, the analysis is to adequately quantify how the statistical variability of the key data elements underlying the estimates of the economic analysis impacts these estimates and the results of the comparison of alternatives. We found that the studies we reviewed did not fully use some best practices related to transparency. Specifically: Being transparent with respect to analytical choices, assumptions, and data used. The economic analyses in the eight studies described and justified several, but not all of the analytical choices, assumptions, and data. For example, to approximate the amount of damages to structures at different flood depths, the Wilmington District’s feasibility study relied on standardized “depth- damage curves” developed by the Corps’ New Orleans District. Corps guidance indicates that standardized curves can be used in the absence of regionally developed data. According to the study, data for structures in the study area were unavailable, and flooding characteristics were similar in the two areas, with both study areas covering urbanized and rural areas representing a mix of residential, commercial, and industrial development with similar types of construction. However, other data and assumptions used by the studies in our review were not fully described or justified. For example, in presenting its results for an initial screening of several flood risk management alternatives, the Sacramento District’s economic analysis relied on cost estimates from several different sources, including prior studies and private consultants. The analysis, however, did not explain how the estimates were developed or justify why the estimates were sufficiently reliable for evaluating alternatives. Clearly explaining the implications of key limitations in the data and models used. With one exception, the economic analyses we reviewed generally did not discuss the implications of key limitations in the models used in the studies. Specifically, the economic analysis for the Sacramento District’s study indicated that the Corps’ Hydrologic Engineering Center-Flood Damage Analysis computer program can overstate damage reduction benefits because of an inability to account for the reduced floodplain occupancy and reduced value of damageable property following a flood event. According to the analysis, by not taking into account the potential for reduced floodplain occupancy, the estimated damage reduction benefits may be overstated, particularly in areas that experience more frequent or severe flooding. To account for this limitation, the Sacramento District’s study reduced the overall value of properties in the floodplain, lowering the average annual benefits for the recommended alternative by about 29 percent. All the other studies used the same program to estimate damage reduction benefits but did not indicate whether this limitation would affect the estimated benefits of the alternatives evaluated in those studies. In accordance with best practices, the Corps’ Planning Guidance indicates that studies should provide adequate supporting documentation to allow reviewers to understand the models and assumptions used to estimate benefits and costs. Corps officials stated that a project team’s analysis may not document every step it took because these are understood among team members, although they may not be apparent to others. Quantifying the statistical variability underlying the results of the comparison of alternatives. Although the economic analyses for the eight studies analyzed the effects of uncertainty associated with several key inputs in the economic analysis, the studies generally did not report the key estimates (for example, benefits, costs, and net benefits) on a probabilistic basis. For example, the Chicago District’s flood risk management study presented damage reduction benefits for each alternative in terms of its expected values as well as the probability that the benefit estimate would exceed a particular value. However, estimates for costs and net benefits were presented as point estimates, which may imply a greater sense of precision than is warranted. In accordance with best practices, the Corps’ Planning Guidance requires economic analyses to report net benefits and benefit-to-cost ratios both as expected (mean) values and on a probabilistic basis for each alternative; also, for each alternative, the analyses are to present the probability that net benefits are positive and that the benefit-to-cost ratio is at or above one. Corps officials said the analyses generally did not follow this guidance because it may not have been useful in helping to select a project alternative. Nonetheless, Corps guidance states that information about the probability distributions can help decision making by local sponsors, stakeholders, and federal officials by helping to increase their understanding of the uncertainty inherent in each alternative. In addition, for only one Corps study, the economic analyses included a sensitivity analysis on the discount rate, which is used to convert benefits and costs of the alternatives to present values. Generally, when benefits or costs are separated in time from each other, the difference in timing should be accounted for by discounting benefits and costs. In addition, the specific discount rate may affect the comparison of alternatives. Corps officials told us that they are required to use the statutorily designated discount rate, and their guidance does not require a sensitivity analysis using an alternative discount rate. The officials added that the Office of Management and Budget requires the Corps to compute the benefit-to-cost ratios for recommended plans using a 7 percent discount rate, for budgeting purposes. The results, though, are not reported in the studies, and the 7 percent rate is not applied in the assessment of the net benefits of the alternatives, according to these officials. Corps officials stated that in general there is a high level of transparency within the project team and with the nonfederal sponsor, but they acknowledged that transparency may not always exist for those outside the team. For example, a project team’s analysis may not document every step it took or assumption it made because these are understood among team members, although they may not be apparent to others. As a result, Corps officials acknowledged that some inconsistency exists in the transparency of the analyses across feasibility studies. Corps officials told us that teams rely on the Corps’ internal process for reviewing all planning products to help ensure the quality of its feasibility studies and analyses. The officials stated that to improve transparency, the Corps could strengthen its internal review process, for example, by adding steps so that all of the important decisions and assumptions made in the analyses are consistently and clearly described. By conducting future economic analyses for potential flood risk management projects so they are more consistent with best practices for transparency, the Corps can better ensure that decision makers and stakeholders are clearly and fully informed about potential economic effects associated with such projects. The economic analyses included in Corps feasibility studies provide important information about the potential economic effects of flood risk management projects. While the economic analyses the Corps conducted for the eight studies we reviewed were generally consistent with several best practices, the Corps did not fully employ best practices pertaining to transparency. Because the information in the economic analyses can be complex and technical, following best practices for transparency helps ensure that the methods used to develop estimates and conclusions are clearly and fully presented. By conducting future economic analyses for potential flood risk management projects so they are more consistent with transparency best practices, the Corps can better ensure that decision makers and stakeholders are clearly and fully informed about the potential economic effects associated with flood risk management projects. We are making the following recommendation to the Department of Defense: The Assistant Secretary of the Army for Civil Works should direct the Chief of Engineers and the Commanding General of the U.S. Army Corps of Engineers to strengthen the Corps’ internal review process for feasibility studies by including steps to ensure consistency with best practices for transparency, such as verifying that all of the important assumptions and limitations in models and their implications for the economic analysis are consistently, clearly, and fully described. (Recommendation 1) We provided a draft of this report to the Department of Defense for its review and comment. In its written comments, reproduced in appendix I, the Department concurred with our recommendation. The Department further stated that guidance related to ensuring transparency in feasibility studies and reviews already exists, but acknowledged that it can be strengthened and enforced more consistently by specifically identifying transparency as a review criterion. For example, they stated that the Corps plans to establish systematic guidance for meeting the transparency objective in preparing reports, assure transparency through the agency’s quality assurance process, and assess the degree of transparency as part of agency technical review and quality control assessment. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Assistant Secretary of the Army for Civil Works, the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and of Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. In addition to the contact named above, Vondalee R. Hunt (Assistant Director), Brad C. Dobbins (Analyst in Charge), Tim Carr, David Dornisch, Juan Garay, Tim Guinane, Gwen Kirby, Keesha Luebke, Jeanette Soares, Sara Sullivan, and Kiki Theodoropoulos made key contributions to this report.", "summary": "The Corps, among other things, constructs flood risk management projects to reduce flood damage in threatened communities nationwide in collaboration with nonfederal sponsors. The Corps prepares feasibility studies to inform decision makers whether a proposed project warrants federal investment. In the studies, the Corps formulates and evaluates alternative plans for achieving the project's objectives and assesses whether the benefits of constructing it outweigh its costs. GAO was asked to review the methodology the Corps used in feasibility studies. This report examines, for 2015 through 2017, (1) the Corps' process for identifying and evaluating the benefits, costs, and effects of project alternatives; (2) the analyses the Corps used to recommend projects; and (3) the extent to which the Corps' economic analyses of benefits and costs are consistent with best practices. GAO reviewed Corps guidance; examined planning documents and economic analyses in flood risk studies that the Corps had most recently completed from 2015 through 2017 from eight districts; and compared the Corps' economic analyses with best practices in GAO's Assessment Methodology. In the eight flood risk management feasibility studies GAO reviewed (see figure), the U.S. Army Corps of Engineers (Corps) followed a six-step planning process consistent with its guidance to, among other things, identify and evaluate the beneficial and adverse effects of alternative plans for proposed projects. In doing so, the Corps used economic analyses to evaluate project-specific categories of potential monetary benefits and costs of alternative plans, such as flood damage reduction benefits and project construction costs. The studies also used separate analyses to evaluate other effects, such as on wildlife habitat and the health and safety of communities. In the eight studies GAO reviewed, the Corps typically recommended the alternative plan with the greatest net benefit, but also relied on other analyses in certain cases, as allowed under Corps guidance. Corps officials said they relied on other analyses to determine the best project design, help make decisions, or respond to local sponsors' preferences. For example, in one study, the Corps recommended a plan that provided a levee 3 feet higher than the plan with the greatest net benefits, in response to the nonfederal sponsor's request. The Corps' economic analyses in the eight studies were generally consistent with best practices, but did not fully adhere to practices for transparency. For example, most analyses did not discuss the implications of key limitations in the models and data used. Corps officials acknowledged that transparency could be improved through their review process. By having future analyses align with transparency best practices, the Corps can better inform decision makers about potential economic effects of flood risk projects. GAO recommends that the Corps strengthen its feasibility study review process by including steps to ensure consistency with transparency best practices. The agency concurred with the recommendation.", "document_type": "gao"}
{"report": "The federal Health Center Program was established in the mid-1960s in an effort to help low-income individuals gain access to health care services. The Health Center Program, authorized in Section 330 of the Public Health Service Act, is administered by HRSA’s Bureau of Primary Health Care and makes grants—known as Section 330 grants—to four types of health centers that primarily serve low-income populations: 1. Community health centers. These health centers serve the general population with limited access to health care. They are required to provide primary health services to all residents who reside in the center’s service area. More than three-quarters of health centers are community health centers. 2. Health centers for the homeless. These health centers provide primary care services to individuals who lack permanent housing or live in temporary facilities or transitional housing. These centers are required to provide substance abuse services and supportive services targeted to the homeless population. 3. Health centers for residents of public housing. These health centers provide primary health care services to residents of public housing and individuals living in areas immediately accessible to public housing. 4. Migrant health centers. These health centers provide primary care to migratory agricultural workers (individuals whose principal employment is in agriculture and who establish temporary residences for work purposes) and seasonal agricultural workers (individuals whose principal employment is in agriculture on a seasonal basis but do not migrate for the work). HRSA’s Section 330 grants are funded by a combination of discretionary appropriations and, since 2011, mandatory appropriations provided from the CHCF. From fiscal years 2010 through 2017, total funding appropriated for Section 330 grants—which includes funding from discretionary appropriations and the CHCF—increased from about $2.1 billion to $4.9 billion (see fig. 1). According to HRSA data, approximately 70 percent of appropriations for Section 330 awards in fiscal year 2017—or about $3.5 billion—were funded by the CHCF. HRSA officials also told us that the total amount of CHCF appropriations may differ from the total amount of awards funded because, for example, appropriations may be (1) used for administrative costs, (2) reduced because of sequestration, or (3) carried over between fiscal years. Health centers are required to provide comprehensive primary health services, including preventive, diagnostic, treatment, and emergency health services. (See table 1.) All services that health centers provide must be available to patients at the center regardless of patient payment source or ability to pay and must be available (either directly or under a referral arrangement) to patients at all health center service sites. Services are provided by clinical staff—including physicians, nurses, dentists, and mental health and substance abuse professionals—or through contracts or cooperative arrangements with other providers. In addition to the services they provide, health centers are also required to document the unmet health needs of the residents in their service area and to periodically review their service areas to determine whether the services provided are available and accessible to area residents promptly and as appropriate. Health centers also must have a sliding fee scale based on a patient’s ability to pay and to be governed by a community board of which at least 51 percent of the members are patients of the health center. HRSA determines whether health center grantees meet these and other health center program requirements when making award determinations. Our analysis shows that total revenue received by health centers nationwide more than doubled from calendar years 2010 through 2017— from about $12.7 billion to $26.3 billion (see fig. 2). Over the same time period, both the number of health centers and the number of patients served also increased. The number of health centers increased from 1,124 centers in 2010—operating 6,949 sites—to 1,373 centers in 2017— operating 11,056 sites. In addition, the total number of patients served at health centers over the same time period increased by 7.7 million patients, from 19.5 million to 27.2 million. See appendix I for additional information. While the total revenue received by health centers more than doubled from 2010 through 2017, the share of revenue received from grants— including Section 330 grants and other federal and non-federal grants— decreased, from 38.0 percent of total revenue in 2010 to about 30.2 percent in 2017. During the same time period, the share of revenue health centers received from Medicaid, Medicare, and private health insurance increased (see fig. 3). (See app. II for more information on health centers’ revenue from 2010 through 2017.) While the share of health centers’ total revenue coming from all grants decreased from 2010 to 2017, the share of revenue from one type of grant—Section 330 grants—increased. Specifically, the share of revenue health centers received from Section 330 grants—a portion of which are funded by the CHCF—increased from 15.7 percent of health centers’ total revenue in 2010 to 18.0 percent in 2017 (see figure 4). Our analysis also shows that the share of revenue health centers receive from Section 330 grants varies by state. As figure 5 below shows, in 2017, health centers in 2 states received more than 40 percent of their total revenue from Section 330 grants, while health centers in 18 states received less than 20 percent of total revenue from these grants. Our analysis of HRSA data shows that for the 7-year period from fiscal years 2011 through 2017, HRSA provided health centers with about $15.8 billion in Section 330 grants funded by the CHCF. Most of this funding—$12.6 billion, or nearly 80 percent of all grants awarded through the CHCF during this period—was awarded for the purpose of service area funding, which supports ongoing operations and services across the nearly 1,400 health centers nationwide (see fig. 6). The remaining $3.2 billion in CHCF grants were awarded to increase the amount of services provided at existing health centers; to increase the number of health centers and sites; and for other special initiatives, such as initiatives to support health information technology. Service area funding. From fiscal years 2011 through 2017, HRSA used the CHCF to provide health centers with approximately $12.6 billion in grants for service area funding, which supports ongoing operations and service delivery. HRSA officials told us that these CHCF grants are used to fill the gap between what it costs to operate a health center and the amount of revenue a health center receives. As such, the awards are a primary means through which health centers provide health care services that may be uncompensated, including services for patients who are uninsured or services not typically reimbursed by other payers, such as adult dental care, or other services such as transportation and nutritional education. These awards can cover uncompensated care costs for patients with incomes low enough to qualify for sliding fee assistance, which reduces or waives the cost of services for patients based on their ability to pay. In addition, these awards can cover patients who have private insurance but face substantial deductibles and cost-sharing. Officials we interviewed from the Congressional Research Service, George Washington University’s Milken Institute, and the National Association of Community Health Centers similarly noted that CHCF grants support services not typically covered by public health insurance, such as adult dental care services not generally covered by Medicare or Medicaid. Increasing services at existing health centers. From fiscal years 2011 through 2017, HRSA used the CHCF to provide health centers with about $1.2 billion in grants to help increase the amount of services offered at existing health centers that chose to apply for an award. This amount included funding to increase the availability of specific health care services, such as dental care, as well as funding to support health centers’ efforts to extend service hours or increase the number of available providers. Specifically, these grants were awarded for the following: Behavioral and mental health, substance abuse. Three grants totaling about $400.8 million were awarded to expand access to behavioral health, mental health, and substance abuse services. These awards focused primarily on integrating primary care and behavioral health care services and expanding substance use services at existing health centers, such as medication-assisted treatment for opioid-use disorder. Oral health. A grant for about $155.9 million was awarded to increase access to oral health services and improve oral health outcomes by funding new onsite providers and supporting the purchase and installation of dental equipment. Expanding Services. Two grants—one in fiscal year 2014 for $295.6 million and another in fiscal year 2015 for about $349.6 million—were made to increase access to comprehensive primary health care in various ways, at the discretion of individual health centers. At existing sites, health centers may have chosen to expand service hours, increase the number of health care providers, or expand services such as oral health, behavioral health, pharmacy, and vision services. Increasing the number of health centers and sites. From fiscal years 2011 through 2017, HRSA awarded about $1.1 billion—or about 7 percent of total CHCF funds—to organizations to help establish new health centers or new sites at existing health centers. Specifically, HRSA awarded grants for the following purposes: New Access Point (NAP) Awards. Most of the funding to increase access to health centers—about $648.5 million of the $1.1 billion— was provided through what are called NAP awards. According to HRSA officials, there are two primary ways these funds can be used—either to allow a new organization to become a health center (about 30 percent of grant applicants) or for an existing health center to add one or more service sites (about 70 percent of grant applicants). HRSA officials told us that they funded 1,059 NAP awards to new and existing health centers from fiscal year 2011 through 2017 for a combined total of 1,609 proposed new health centers or sites. These awards included 295 awards to new organizations and 764 awards to existing health centers adding one or more service sites. (See table 2 below for more information on the increase in health centers resulting from NAP awards.) Among the 1,609 total proposed new health centers or sites, 686 were in rural areas, including 191 new health centers and 495 additional sites at existing centers. Construction Grants. HRSA awarded construction grants totaling about $411.3 million through the Health Infrastructure Investment Program to help existing health centers alter, renovate, expand, or construct a facility. According to HRSA officials, construction grants may increase the number of health center sites or may result in the consolidation of sites while still expanding access to care. Health Center Planning Grants. HRSA awarded a Health Center Planning grant in fiscal year 2011 for about $10.3 million to support planning and development of comprehensive primary care health centers. Collectively, a total of 5,536 new health center sites were added in the United States from fiscal year 2011 through 2017. Of these new sites, 3,838 were in urban locations and 1,698 were in rural locations. While many of these new health center sites were from NAP awards, as previously described, other grants either funded by the CHCF or by discretionary appropriations may have contributed to the establishment of new health center sites. For example, HRSA officials told us that health center sites may be added through a change of scope to their service area competition award or through other types of grants funded by the CHCF, such as grants to increase adult dental services. However, according to HRSA officials, the data do not allow for directly associating the number of new sites with those grants, as the grants may be used for multiple purposes. Figure 7 below shows the locations of health center sites added during this time period that are active as of February 2019. Other special initiatives. From fiscal years 2011 through 2017, HRSA awarded about $898.9 million of CHCF funds for grants to health centers to support other special initiatives and to address identified priorities or emerging health care needs. Specifically, HRSA awarded grants to those health centers that chose to apply for the following purposes: Health information technology. Three grants totaling about $243.4 million were awarded to advance the adoption and implementation of health information technology. For example, the purpose of one grant—the Health Center Controlled Networks—was to advance the adoption, implementation, and optimization of health information technology. Another grant provided supplemental funding to improve the electronic reporting capabilities of health centers in Beacon Communities. HIV. Two grants totaling about $23.8 million were awarded with the goal to increase access to HIV care and services. One specifically targeted prevention and treatment services in those communities most affected by HIV. Outreach and enrollment. $222.0 million in grant funding was awarded to support health centers in raising awareness of affordable insurance options and providing eligibility and enrollment assistance to uninsured patients of health centers and residents in their approved service areas. Patient-Centered Medical Home. About $84.6 million in grant funding was awarded to support HRSA efforts to expand the number of patient-centered medical homes with a particular focus of improving quality of care, access to services, and reimbursement opportunities. Quality improvement. Approximately $305.1 million in grant funding was awarded to support health centers that displayed high quality performance so they can continue to strengthen quality improvement efforts. Specifically, the funds were to support health centers to further improve the quality, efficiency, and effectiveness of health care delivered to the communities served. Training and technical assistance. Two grants totaling about $14.3 million were awarded to support training and technical assistance for health centers in order to support programmatic, clinical, and financial operations. One grant focused on the delivery of training and technical assistance by national organizations and the other grant was based on statewide and regional needs. Zika. A grant for about $5.7 million was awarded to health centers that chose to apply to expand their existing activities to strengthen the response to the Zika virus in Puerto Rico, the U.S. Virgin Islands, and American Samoa. These activities included outreach, patient education, screening, voluntary family planning services, and/or treatment services. See appendix III for a complete list of all grants awarded through CHCF by category. We provided a draft of this report to HHS. HHS provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further action until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Health and Human Services and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at farbj@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This appendix provides information on health centers and patients served. Specifically, figure 8 illustrates the number and location of health centers in 2017; figure 9 illustrates the growth in health centers and sites since 2010; figure 10 illustrates the growth in patients served at health centers since 2010; and table 3 provides information on how the payer mix for patients served at health centers has changed since 2010. HRSA’s Uniform Data System defines other public insurance as state and/or local government programs, such as Washington’s Basic Health Plan or Massachusetts’ Commonwealth plan, that provide a broad set of benefits for eligible individuals. Other federal grants in HRSA’s Uniform Data System include Medicare and Medicaid Electronic Health Record Incentive grants. HRSA’s Uniform Data System defines non-federal grants and contracts as revenue from contracts that are not tied to the delivery of services and revenue received from state and local indigent care programs. HRSA’s Uniform Data System defines other revenue as non-patient related revenue not reported elsewhere. Examples include revenue from fund-raising, rent from tenants, medical record fees, and vending machines. In addition to the contact named above, Kristi Peterson, Assistant Director; Amy Leone, Analyst-in-Charge; Margot Bolon, Krister Friday, Jeff Tamburello, and Eric Wedum made key contributions to this report. Also contributing were Vikki Porter, Rotimi Adebonojo, Giselle Hicks, and Jennifer Whitworth.", "summary": "In 2017, nearly 1,400 health centers provided care to more than 27 million people, regardless of their ability to pay. Health centers were established to increase the availability of primary and preventive health services for low-income people living in medically underserved areas. Health centers rely on revenue from a variety of public and private sources, including revenue from CHCF grants. HRSA began awarding grants funded by the CHCF in fiscal year 2011. GAO was asked to review the sources and amounts of health center revenue. This report describes (1) trends in health centers' revenue and (2) the purposes for which CHCF grants have been awarded. GAO analyzed HRSA data collected from health centers and compiled in its Uniform Data System to identify the sources and amounts of revenue health centers received from 2010 through 2017, the most recent data at the time of GAO's analysis. GAO also reviewed HRSA grant documentation for grants funded by the CHCF for fiscal years 2011-2017—the most recent data at the time of GAO's analysis—including information on the award amount and purpose of the grant, and reviewed published studies that described the purposes for which CHCF grants have been made. Additionally, GAO interviewed HRSA officials, authors of the published studies, and an association representing health centers. GAO provided a draft of this report to HHS. HHS provided technical comments, which GAO incorporated as appropriate. Health centers' revenue more than doubled from calendar years 2010 through 2017, from $12.7 billion to $26.3 billion. Health centers' revenue comes from a variety of sources, including reimbursements from Medicaid, Medicare, private insurance, and federal and state grants. While total health center revenue increased from 2010 through 2017, the share of revenue from each source changed in different ways. In particular, revenue from federal and state grants decreased from 38.0 percent of total revenue in 2010 to about 30.2 percent of total revenue in 2017 while reimbursements from Medicaid, Medicare, and private insurance increased. Over the same time period, the number of health centers increased from 1,124 centers in 2010 to 1,373 centers in 2017. In addition, the number of patients served over the same time period increased by 7.7 million patients, from 19.5 million to 27.2 million. GAO's analysis of Health Resources and Services Administration (HRSA) data shows that from fiscal years 2011 through 2017, health centers received approximately $15.8 billion in federal grants funded by the Community Health Center Fund (CHCF), which was established by the Patient Protection and Affordable Care Act in 2010. Of this total amount, 79.7 percent—or $12.6 billion—was awarded for the purpose of maintaining operations at existing health centers (see figure). According to HRSA officials, these CHCF grants are used to fill the gap between what it costs to operate a health center and the amount of revenue a health center receives. As such, officials explained, the awards are a primary means through which health centers provide health care services that may be uncompensated, including services for uninsured patients or services not typically reimbursed by other payers, such as adult dental care. The remaining $3.2 billion in CHCF grants were made to increase the amount of services provided at existing health centers; increase the number of health centers and sites; and other special initiatives, such as implementing health information technology.", "document_type": "gao"}
{"report": "SEC is a federal agency responsible for protecting investors, maintaining fair, orderly, and efficient markets, and facilitating capital formation. Among its efforts, SEC requires public companies to disclose meaningful financial and other information to the public, examines firms it regulates, and identifies and investigates potential violations of federal securities laws. Each year, SEC brings hundreds of enforcement actions—including judicial enforcement actions and administrative proceedings—against individuals and companies as a result of its investigations. Examples of actions taken in fiscal year 2018 include charges against a company that allegedly defrauded investors in a Ponzi scheme and charges against a bank for misconduct in its sales practices for certain financial products offered to retail investors. SEC’s responsibilities are divided among five divisions and 24 offices. The Division of Enforcement conducts investigations of potential violations of federal securities laws. Enforcement recommends, when appropriate, that SEC bring enforcement actions, litigates these actions, negotiates settlements on behalf of SEC, and works with other law enforcement agencies to bring criminal cases when warranted. Enforcement is currently led by two co-directors who report to the Chairman. Enforcement staff operate from headquarters in Washington, D.C., and in 11 regional offices. Enforcement maintains a database that tracks enforcement-related activities, including all cases from investigation through litigation, and is the source of statistics used in public reporting (see fig. 1). For tracking purposes, “case” encompasses all stages of a possible enforcement action, beginning either as a matter under inquiry or as an investigation. Some cases advance and become an enforcement action. Enforcement’s database includes all key case data other than data on financial penalties and disgorgements, which SEC’s Office of Financial Management stores and manages. Enforcement’s CMS are responsible for recording key data into the database and conducting quality checks on the data throughout the investigative and litigation processes of a case. There are two groups of CMS: local CMS and national CMS, both of which can be located in SEC regional offices or at SEC headquarters. Local CMS in regional offices report to their regional managers but coordinate with the Enforcement Case Management and Systems Reporting Group. National CMS and local CMS at SEC headquarters report to the Case Management Systems and Reporting Group. National CMS have the responsibility of reviewing and verifying case data input by local CMS. Enforcement has documented procedures for recording and verifying enforcement-related data in its central database. More specifically, the Enforcement database user guide has step-by-step procedures for recording case data and clear descriptions of each data entry field. For example, the guide includes brief descriptions of primary classifications— or categories—used to describe the nature of the enforcement action (such as insider trading or delinquent filing). According to the database user guide and other SEC documentation, local CMS have the primary responsibility for recording most case data used in Enforcement metrics. Local CMS may assist with data recording in the opening of a case as a matter under inquiry or, if it is known the case will advance to the next stage, as an investigation. The user guide also states that local CMS are responsible for recording the advancement of a case from an investigation to an enforcement action. According to the user guide, CMS use information (generally, case documentation) received from the courts or SEC staff responsible for the case to create the action entry in the central database, including the primary classification for the action. CMS also facilitate closing completed cases in the database. Enforcement procedures call for Enforcement staff to perform multiple data reviews for all information in the Enforcement database, according to Enforcement staff and the user guide. According to the user guide, local CMS review the accuracy of key case-related data recorded in the system at certain stages as a case proceeds (see fig. 2). The local CMS add case information by checking any new documentation, such as court filings. In addition to the review by the local CMS, national CMS also are to review newly opened cases, as well as cases that have advanced to an investigation, changed from an investigation to an action, or closed. To do this, national CMS compare information recorded in the system against any primary documents related to the case, such as court documentation. Finally, Enforcement staff told us that they have an informal process whereby a group of attorneys in the Case Management and Systems Reporting Group review all primary classifications for enforcement actions. Enforcement lacks written procedures for generating the Enforcement Annual Report, including for compiling and ensuring the accuracy of the statistics published within. Enforcement staff explained that they follow an informal process to generate the annual report, which includes steps to help ensure reliable reporting and detect and prevent errors (see fig. 3). However, Enforcement was unable to provide documentation of this process or of the implementation of the steps to help ensure accuracy. According to staff, the process for generating the annual report includes selecting what statistics to include and what activities and accomplishments to describe in the report narrative. Specifically, Enforcement staff said that the division’s co-directors hold regular weekly meetings with their staff to discuss management of the division. Staff said program metrics and other measures may be discussed at these meetings, including the types of information and statistics that might be used in the Enforcement Annual Report. According to the staff, at the end of the fiscal year the co-directors determine what information and statistics the division will include in reports. Once decisions have been made about the annual report’s content, Enforcement staff told us a contractor uses software queries of the database to compile statistics for the report based on data parameters defined by Enforcement staff. An Enforcement staff member familiar with the data reviews the queries’ output to verify accuracy, according to Enforcement staff. Staff then add the compiled statistics to the draft annual report. According to staff, the draft report is then sent to the Office of Public Affairs for formatting and publication. Enforcement staff stated that staff familiar with the data perform an additional check to ensure that no data values were mistyped or otherwise edited in the formatting process. Finally, the co-directors of Enforcement are to review the draft report. After they give a final approval, the annual report is published. Control activities such as written procedures help ensure that operational processes are effective and actions are taken to address risks. In particular, federal internal control standards identify documentation— including documentation that demonstrates procedures are being implemented—as a necessary part of an effective internal control system and as a means to help detect and prevent errors. Enforcement staff stated that the division does not have written procedures for generating its annual report or documenting the implementation of review processes because the report is not required by law and is discretionary. The staff said they were confident about the reliability of report data because staff were familiar with enforcement data and the informal processes they currently use to verify accuracy. In contrast, Enforcement uses documented SEC guidelines for reviewing and verifying the data used to support performance metrics in the agency- wide SEC Annual Performance Report. Documenting written procedures for generating both Enforcement’s annual report and the processes it uses to verify published statistics— including documentation that procedures were implemented—would provide Enforcement with greater assurance that staff follow necessary steps to help ensure the reliability and accuracy of reported information. Reliability and accuracy of information are important to maintaining the division’s credibility and public confidence in its efforts. In addition, developing written procedures would better position Enforcement to manage risk associated with staff turnover and help ensure continuity of operations in its public reporting. Since 2009, SEC has made changes to how it reports and presents enforcement or enforcement-related statistics, which are included in a number of reports (see table 1). As previously discussed, we reviewed reports from 2009 through 2018 that included enforcement statistics. More specifically, SEC made the following changes to its public reporting of enforcement statistics, which include the creation of a stand-alone Enforcement Annual Report in 2017. Prior to 2017, Enforcement reported similar statistics in the annual Select SEC and Market Data Report. Definition of enforcement actions. Enforcement staff told us that before 2013, the Select SEC and Market Data Reports changed little from year to year, with the previous year’s report used as a template to create the next one. SEC adjusted its definition of enforcement actions in the 2013 report, and included notes explaining the change and providing what the number of enforcement actions would have been under the previous definition. Presentation of enforcement statistics. Enforcement staff said the Office of the Chief Operating Officer determined changes in presentation (such as the order of enforcement action classifications) in the Select SEC and Market Data Report. In 2015, Enforcement changed how the report presented summary data for enforcement actions. Previously, Enforcement counted enforcement actions as civil actions or administrative proceedings, but the fiscal year 2015 report separately identified and counted the proceedings as stand-alone (initial) or follow-on (after initial action). Enforcement staff said these changes were made possible by better software that allowed for enhanced and expanded presentation of the data. Enforcement Annual Report. As previously mentioned, the Select SEC and Market Data Report was discontinued after the fiscal year 2017 report and the Enforcement Annual Report was first published in November 2017. The annual report included additional data tables of enforcement statistics not previously reported (some comparing statistics to the previous year) and narratives about enforcement priorities and cases. Enforcement staff told us the annual report was created to increase transparency and provide more information and deeper context than previous reporting had. The SEC Division of Enforcement voluntarily issues an annual report that includes statistics and highlights significant enforcement actions and initiatives of the previous fiscal year. Enforcement has documented procedures and has designated staff to input and review enforcement- related data in its case-tracking system. However, the division does not have written procedures for generating its public reporting (currently, the annual report), including for compiling and verifying the report’s statistics, or documenting that procedures were implemented as intended. Written procedures would help Enforcement ensure the reliability and accuracy of reported information, manage risk associated with staff turnover, and promote continuity of operations in its public reporting. The Securities and Exchange Commission’s Co-Directors of Enforcement should develop written procedures for generating Enforcement’s public reports, including procedures for compiling and verifying statistics used in the reports and documenting their implementation. (Recommendation 1) We provided a draft of this report to SEC for review and comment. In written comments (reproduced in appendix I), SEC generally agreed with our findings and concurred with our recommendation. In addition, SEC provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Chairman of the Securities and Exchange Commission, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or clementsm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above Kevin Averyt, John Forrester, (Assistant Directors), Jordan Anderson (Analyst in Charge), Tim Bober, Ryan Braun, Marc Molino, Kirsten Noethen, Barbara Roesmann, and Farrah Stone made key contributions to this report.", "summary": "Enforcement supports SEC's mission by bringing civil and administrative actions against individuals and entities for fraud, financial and accounting irregularities and misstatements, and other misconduct. According to SEC, these enforcement actions serve as a deterrent against future wrongdoing. Since 2017, Enforcement has published an annual report that provides statistics on its enforcement activities and highlights its priorities for the coming year. GAO was asked to examine SEC reporting of enforcement statistics. This report examines (1) the ways that enforcement statistics reporting changed over the last 10 years, and (2) policies and procedures for recording, reviewing, and reporting enforcement statistics. GAO reviewed SEC's internal policies, procedures, and manuals for recording, verifying, and reporting data. GAO also interviewed SEC officials and reviewed past SEC reports containing enforcement statistics. Since 2009, the Division of Enforcement (Enforcement) in the Securites and Exchange Commision (SEC) has made modifications to its reporting of enforcement statistics, including by releasing a stand-alone annual report beginning in fiscal year 2017. The Enforcement Annual Report included additional data on enforcement statistics not previously reported and narratives about enforcement priorities and cases. Enforcement staff told us the annual report was created to increase transparency and provide more information and deeper context than previous reporting had provided. Enforcement has written procedures for recording and verifying enforcement-related data (including on investigations and enforcement actions) in its central database. However, Enforcement does not have written procedures for generating its public reports (currently, the annual report), including for compiling and verifying the enforcement statistics used in the report. To produce the report, Enforcement staff told GAO that staff and officials hold meetings in which they determine which areas and accomplishments to highlight (see figure). Enforcement was not able to provide documentation demonstrating that the process it currently uses to prepare and review the report was implemented as intended. Developing written procedures for generating Enforcement's public reports and documenting their implementation would provide greater assurance that reported information is reliable and accurate, which is important to maintaining the division's credibility and public confidence in its efforts. GAO recommends that SEC's Co-Directors of Enforcement develop written procedures for generating Enforcement's public reports, including procedures for compiling and verifying statistics used in the reports, and documenting their implementation. SEC agreed with the recommendation.", "document_type": "gao"}
{"report": "When Mexico and the United States created the Mérida Initiative in 2007, the Mexican government pledged to tackle crime and corruption and the U.S. government pledged to address domestic drug demand and the illicit trafficking of firearms and bulk currency to Mexico. During the early years of the Mérida Initiative, much of the U.S. funding for the initiative was intended to purchase equipment to support Mexican federal security forces, including about $591 million for aircraft and helicopters from fiscal years 2008 through 2010. In 2011, U.S. and Mexican officials agreed to expand the scope of the initiative to prioritize institution building, and later increased the initiative’s focus on community engagement and human rights efforts. According to State, an Executive Order on TCOs issued in 2017 signaled that the focus of the Mérida Initiative would shift to countering TCOs’ illicit activities, such as drug production and the cross- border movement of drugs, cash, and weapons. State/INL and USAID are the lead U.S. agencies for developing the Mérida Initiative’s programming. In these roles, State/INL and USAID work with GOM officials to help outline Mérida Initiative projects’ plans, objectives, and intended impact. State/INL and USAID both manage and fund the Mérida Initiative with the support of a wide range of project implementers, including DOJ, DHS, DOD, contractors, nongovernmental organizations, and international organizations. All Mérida Initiative projects are currently funded through three appropriations accounts: the International Narcotics Control and Law Enforcement (INCLE) account administered by State/INL, the Economic Support Fund (ESF) account, from which the Mérida Initiative project funding is administered by USAID, and the Development Assistance (DA) account, also from which the Mérida Initiative project funding is administered by USAID. According to State/INL and USAID officials, GOM does not provide direct funding to Mérida Initiative projects. Instead, State/INL considers any GOM funding for justice, national security/defense, and public order and domestic security as indirectly supporting the goals of the Mérida Initiative. In addition, according to USAID data, Mexican nonprofit and private sector entities provided about $23 million in matching funds for USAID-funded Mérida Initiative projects active from fiscal year 2014 through 2018. From fiscal year 2014 through 2018, State/INL and USAID allocated about $723 million for Mérida Initiative projects under the following five U.S. government–wide foreign assistance funding categories: Civil Society, Counternarcotics, Good Governance, Rule of Law and Human Rights, and Transnational Crime. U.S. agencies use these government- wide categories to broadly define foreign assistance programs for planning, budgeting, and reporting, which provides a common language to describe programs across agencies, countries, and regions. Over 80 percent of the funding, or $589 million, went toward Rule of Law and Human Rights, and Counternarcotics efforts. (See fig. 1.) Funding allocated for the Mérida Initiative has decreased over time from $178 million in fiscal year 2014 to $139 million in fiscal year 2018. Of the $723 million, State/INL allocated about $542 million and USAID allocated about $182 million. (See fig. 2.) Four hundred and forty-five Mérida Initiative projects were active from fiscal year 2014 through 2018, with State/INL funding 388 projects and USAID funding 57 generally larger projects. State/INL and USAID each categorized their projects with greater specificity than the broad categories used for overall allocated funding. Both State/INL and USAID funded projects to assist Mexico’s transition to a newly reformed criminal justice system that includes oral arguments and the presumption of innocence, categorized by State/INL as “criminal justice” and by USAID as “rule of law.” In addition to projects related to criminal justice, most funding for State/INL projects was for those that focused on border and port security, professionalizing the police, and counternarcotics. For example, numerous State/INL projects provided training; technical assistance; and equipment—including for drug detection, border surveillance, and forensic drug laboratories—for Mexican law enforcement, border security, justice sector, and military officials. In addition to rule of law projects, most funding for USAID projects was for those that focused on crime and violence prevention, human rights, and transparency and accountability. Similar to State/INL, numerous USAID projects provided technical assistance to Mexican justice sector officials. Other USAID projects were designed to engage with civil society groups to address crime and violence, and corruption, and to promote trust in government. State/INL and USAID implemented these projects primarily through contracts, grants, cooperative agreements, interagency agreements, and agreements with international organizations. State/INL-funded Mérida projects focused on criminal justice, border and port security, professionalizing the police, and counternarcotics. State/INL categorizes its Mérida Initiative projects under priority lines of effort developed by State/INL Mexico specifically for the Mérida Initiative. These lines of effort are defined in State/INL’s Mexico Country Plan: Advance Criminal Justice, Counternarcotics, Disrupt Illicit Finance, Professionalize the Police, and Secure Border and Ports. While State/INL uses these lines of effort to categorize State/INL-funded Mérida Initiative projects, these lines of effort also align with the broader U.S. government–wide foreign assistance funding categories outlined in figure 1 above. See table 1 for a description of State/INL’s lines of effort for the Mérida Initiative, and how these lines of effort align with the U.S. government– wide foreign assistance funding categories. The State/INL projects with the highest percentage of State/INL funding were those focused on Advancing Criminal Justice (28 percent) and Securing Borders and Ports (25 percent). Law enforcement related categories—Counternarcotics and Professionalize the Police—also constituted a substantial proportion (30 percent) of State/INL funding, as shown in figure 3. For a list of State/INL’s highest dollar value projects active from fiscal year 2014 through 2018 by these categories, see appendix I. Below are some examples of State/INL-funded Mérida Initiative projects supporting the agency’s five lines of effort in Mexico: Advance Criminal Justice. These 99 projects, with State/INL funding estimated at $241 million, focused on providing training, technical assistance, and equipment to Mexican justice sector and law enforcement officials as they transition to a new judicial system. These projects also provided tools and guidance to civil society to promote the rule of law and trust in government. For example: One DOJ project supported criminal investigations and prosecutions by providing training to GOM officials to improve their forensic laboratories, and by providing technical assistance to forensic scientists testifying as expert witnesses in criminal cases. Through another project, DOJ developed training materials and instructors to assist the GOM Attorney General’s office with the mechanics of Mexico’s judicial system reforms and to create a culture of professionalization within the Attorney General’s office. Some criminal justice projects engaged with civil society, such as a project that worked to promote a culture of lawfulness among Mexican children who attend elementary school in high-crime areas. Counternarcotics. These 76 projects, with State/INL funding estimated at $115 million, focused on assisting Mexican agencies countering the illicit drug trade in Mexico, primarily through technical assistance and equipment, including for forensic labs, drug detection, and surveillance. For example: Intelligence surveillance and reconnaissance technology has been provided to the Mexican Navy to expand its capacity to conduct counternarcotics operations. The Organization of American States implemented a project that expanded Mexico’s drug treatment courts, which offer rehabilitation services and other nonpunitive alternatives for drug offenders who would otherwise face time in prison. Disrupt Illicit Finance. These nine projects, with State/INL funding estimated at $17 million, provided equipment, training, and a public awareness campaign to assist the GOM in its efforts to address TCO’s money-laundering and other illicit financial activities. For example: One DOJ project provided anti–money laundering training to Mexican prosecutors at the state and federal levels. Another United Nations Office on Drugs and Crime project aims to combat money laundering through a public awareness campaign and complaint call center in Mexico. Professionalize the Police. Many of these 97 projects, with State/INL funding estimated at $144 million, provided training and technical assistance to Mexican law enforcement officials at all levels to improve their effectiveness, accountability, and adherence to the rule of law. An aspect of one project conducted surveys with law enforcement personnel and civil society to better inform effective police practices and to link these practices with levels of citizen trust. Two other projects supported tours to the United States for Mexican officials to study issues related to gender-based violence and women’s access to justice. Secure Borders and Ports. These 68 projects, with State/INL funding estimated at $217 million, focused on various efforts and equipment for GOM border and military officials—including equipment for biometrics, surveillance, and telecommunications—to secure Mexico’s air, land, and sea borders and ports. For example, DHS’s Customs and Border Patrol provided mentors and training to GOM border officials to improve their capacity to stem the northward flow of migrants entering Mexico along its southern border. USAID-funded Mérida Initiative projects focused on crime and violence prevention, rule of law, transparency and accountability, and human rights efforts. USAID categorizes its Mérida Initiative projects under the following development objectives developed by USAID Mexico and outlined in USAID’s Mexico Country Development Cooperation Strategy: Crime and Violence Prevention, Human Rights, Rule of Law, and Transparency and Accountability. Similar to State/INL’s Mérida Initiative lines of effort, USAID Mexico uses its development objectives to categorize USAID-funded Mérida Initiative projects. These objectives also align with the broader U.S. government–wide foreign assistance funding categories outlined in figure 1. See table 2 for a description of USAID’s development objectives for the Mérida Initiative, and how these objectives align with the U.S. government–wide foreign assistance funding categories. The USAID projects with the highest percentage of USAID funding were those focused on Rule of Law (39 percent) or Crime and Violence Prevention (22 percent) with Transparency and Accountability and Human Rights constituting slightly smaller percentages (15 percent and 14 percent, respectively). While funding for USAID projects was concentrated in the Rule of Law category, the number of USAID projects was spread relatively evenly among the categories of Crime and Violence Prevention, Human Rights, and Transparency and Accountability, as shown in figure 4. For a list of USAID’s highest dollar value projects active from fiscal years 2014 through 2018, see appendix II. Below are some examples of USAID-funded Mérida projects supporting the agency’s four development objectives in Mexico for the Mérida Initiative: Crime and Violence Prevention. These 20 projects, with USAID funding estimated at $70 million, worked with civil society, nongovernmental organizations, the private sector, and GOM officials to implement various activities, such as training, workshops, and outreach efforts, to mitigate crime and violence. A number of these projects focused on building the skills and knowledge of at-risk youth, such as those in high-crime areas or at risk of dropping out of school. For example, one project aimed to help at-risk youth in communities and detention centers return to school, gain employment, and improve life skills. Human Rights. These 15 projects, with USAID funding estimated at $46 million, worked to advance human rights through various activities that, for example, focused on protecting journalists and human rights defenders, preventing forced disappearances, and promoting freedom of expression. For example, one project supported the GOM’s efforts to implement its National Human Rights Plan by implementing clear procedures in line with international human rights standards. Rule of Law. These three projects, with USAID funding estimated at $126 million primarily provided technical assistance and outreach to assist Mexican officials as they transitioned to a new judicial system. For example, two large projects—one $68 million project and one $56 million project that has since closed—provided a wide range of technical assistance to GOM judges, public defenders, and attorneys general. Another smaller project worked with law schools to adapt their curricula to the new criminal justice system. Transparency and Accountability. These 15 projects, with USAID funding estimated at $49 million engaged with Mexican officials and civil society to address corruption and promote ethical behavior. Projects helped Mexican officials develop and implement anticorruption policies, strengthen transparency in their procurement processes, and implement GOM’s National Anti-Corruption System. For example, one project aimed to deter corruption and support transparency by improving the quality of investigative and data journalism in Mexico. State/INL and USAID implement Mérida Initiative projects primarily through contracts, grants, and agreements with international organizations, but State/INL also employs agreements with U.S. agencies (DOJ, DHS, and DOD). See tables 3 and 4 for the number of and funding for each type of State/INL and USAID funding mechanism, respectively. We provided a draft of this report to State and USAID for review and comment. State and USAID both provided technical comments, which we incorporated as appropriate. USAID also provided formal comments, which are reproduced in appendix III. In these comments, USAID noted that, with its support, the Mérida Initiative has been instrumental in advancing reforms to the Mexican criminal justice sector, promoting human rights, building strong and resilient communities, and improving integrity and accountability. We are sending copies of this report to the appropriate congressional committees, the Secretary of State, and the USAID Administrator. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7141 or groverj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made contributions to this report are listed in appendix IV. This appendix provides a detailed list of the 10 highest dollar value Department of State, Bureau of International Narcotics and Law Enforcement Affairs (State/INL) Mérida Initiative projects active from fiscal year 2014 through 2018 by State/INL’s five lines of effort—Advance Criminal Justice, Counternarcotics, Disrupt Illicit Finance, Professionalize the Police, and Secure Border and Ports. State/INL provided the details in tables 5 to 9 below. This appendix provides a detailed list of the 10 highest dollar value United States Agency for International Development (USAID) Mérida Initiative projects active from fiscal year 2014 through 2018. USAID provided the details in table 10 below. In addition to the contact named above, James Michels (Assistant Director), Teresa Heger (Analyst-in-Charge), Terry Allen, Ashley Alley, Lilia Chaidez, Martin DeAlteriis, Neil Doherty, Francisco Enriquez, John Hussey, and Andrew Kincare made key contributions to this report.", "summary": "For more than a decade, the activities of transnational criminal organizations have led to increased crime, violence, and lawlessness in parts of Mexico. In October 2007, Mexico and the United States created the Mérida Initiative, a bilateral partnership to address crime and violence and enhance the rule of law in Mexico. State/INL and USAID are the lead U.S. agencies for developing programming for the Mérida Initiative. Both State/INL and USAID also manage and fund the Mérida Initiative with the support of a wide range of project implementers, including the Departments of Defense (DOD), Homeland Security (DHS), and Justice (DOJ); contractors; nongovernmental organizations; and international organizations. GAO was asked to describe funding and projects the United States has provided under the Mérida Initiative. This report describes (1) State/INL and USAID funding for the Mérida Initiative from fiscal year 2014 through 2018 and (2) the number and type of Mérida Initiative projects active during these years. GAO reviewed State and USAID documents and data, and interviewed officials from State, USAID, DOD, DHS, and DOJ in Washington, D.C., and Mexico City. From fiscal year 2014 through 2018, the Department of State's (State) Bureau of International Narcotics and Law Enforcement Affairs (State/INL) and the U.S. Agency for International Development (USAID) allocated about $723 million for the Mérida Initiative, which aims to mitigate the impact of the drug trade on the United States and reduce violence in Mexico. State/INL and USAID allocated this funding under the following government-wide foreign assistance funding categories: Civil Society, Counternarcotics, Good Governance, Rule of Law and Human Rights, and Transnational Crime. U.S. agencies use these categories to broadly define foreign assistance programs for planning, budgeting, and reporting across agencies, countries, and regions. Over 80 percent of the funding went toward Rule of Law and Human Rights, and Counternarcotics efforts. Of the $723 million, State/INL allocated about $542 million and USAID allocated about $182 million. There were 445 State/INL and USAID Mérida Initiative projects active from fiscal year 2014 through 2018. State/INL funded 388 of the projects and USAID funded 57, which tended to be larger with higher funding amounts than State/INL projects. State/INL projects generally focused on providing training and assistance to Mexican officials from the justice sector, border security, military, and law enforcement, as well as equipment, including for forensic drug laboratories, drug detection, and border surveillance. Many USAID projects were intended to engage with Mexican civil society organizations and the public to address corruption, promote trust in government, or prevent crime and violence, such as through skill-building for youth, efforts to advance human rights, or technical support for judicial system development. State/INL and USAID implemented their projects mainly through contracts, grants, and interagency agreements, as well as through agreements with international organizations, such as the United Nations Office on Drugs and Crime and the Organization of American States.", "document_type": "gao"}
{"report": "On September 6, 2017, the eye of Hurricane Irma traveled about 50 nautical miles to the north of the northern shore of Puerto Rico as a category 5 hurricane. Less than two weeks later, Hurricane Maria made landfall as a category 4 hurricane on the main island of Puerto Rico on the morning of September 20, 2017, with wind speeds up to 155 miles per hour. The center of the hurricane moved through southeastern Puerto Rico to the northwest part of the island, as shown in figure 1 below. In response to the request of the governor of Puerto Rico, the president declared a major disaster the day after each hurricane impacted Puerto Rico. Major disaster declarations can trigger a variety of federal response and recovery programs, including assistance through FEMA’s Public Assistance program. Under the National Response Framework, DHS is the federal department with primary responsibility for coordinating disaster response, and within DHS, FEMA has lead responsibility. FEMA’s Public Assistance program provides grant funding to state, territorial, local, and tribal governments, as well as certain types of private nonprofit organizations, to assist them in responding to and recovering from major disasters or emergencies. As shown in figure 2, Public Assistance program funds are categorized broadly as either “emergency work” or “permanent work.” Within those two broad categories are separate sub-categories. In addition to the emergency work and permanent work categories, FEMA’s Public Assistance program includes Category Z, which represents any indirect costs, any direct administrative costs, and any other administrative expense associated with a specific project. Given the immense scale and scope of devastation, disaster recovery in Puerto Rico is a complex and dynamic process involving a large number of entities. As shown in figure 3, implementing the Public Assistance program involves recovery partners from the federal government; the Commonwealth of Puerto Rico; and Puerto Rico government agencies, public corporations, municipalities, and eligible nonprofits in Puerto Rico. These recovery partners play a role in implementing the Public Assistance program by developing projects and providing or receiving grants and sub-grants (subawards). FEMA. FEMA administers the Public Assistance program in partnership with Puerto Rico and makes Public Assistance grant funding available to Puerto Rico. Puerto Rico Central Office of Recovery, Reconstruction and Resilience. Puerto Rico was required, as a condition to receiving Public Assistance grant funding, to establish an oversight authority supported by third-party experts and provide centralized oversight over recovery funds. In October 2017, the governor of Puerto Rico established the Central Office of Recovery, Reconstruction, and Resilience (central recovery office) to be the recipient for all Public Assistance funding consistent with the conditions provided in Amendment 5 to the President’s disaster declaration. The central recovery office is a non-federal entity that provides a subaward to an applicant to carry out part of the federal program. As a recipient of federal funds, the central recovery office must oversee subrecipients to ensure that they are aware of and comply with federal regulations. According to central recovery office officials, the office was also established to ensure coordination with FEMA across the numerous partners in recovery. Commonwealth agencies, local entities, and private non-profits. Puerto Rico’s agencies, such as the Department of Housing, and public corporations, such as the Puerto Rico Electric Power Authority, act as subrecipients. Specifically, they work with FEMA and the central recovery office to identify, develop, and implement Public Assistance projects. Local entities, including Puerto Rico’s 78 municipalities and eligible private non-profits that provide critical services, are also subrecipients of FEMA Public Assistance funding. As subrecipients, these entities receive subawards from the central recovery office to carry out work under the Public Assistance program. According to a November 2017 amendment to Puerto Rico’s major disaster declaration, FEMA must obligate all large project funding for Public Assistance permanent work through alternative procedures due to the extraordinary level of infrastructure damage caused by Hurricane Maria, as well as Puerto Rico’s difficult financial position. To develop projects under the Public Assistance program, FEMA and Puerto Rico officials are to collaborate to identify and document the damage caused by a disaster to a particular facility. These officials are to then use the damage description to formulate the scope of work—or activities required to fix the identified damage—as well as the estimated cost of these activities. Under the standard Public Assistance program, FEMA will fund the actual cost of a large project, and will increase or reduce the amount of funding based on the cost of completed eligible work. In contrast, in Puerto Rico, the alternative procedures require that the central recovery office and subrecipients work collaboratively with FEMA to develop a fixed cost estimate. According to FEMA officials, once this fixed cost estimate is agreed to and obligated, subrecipients have flexibility within that fixed cost estimate to rebuild in the manner that they find most appropriate. Subrecipients could do the actual work used to develop the fixed cost estimate, or they could put funds towards another FEMA approved project. Unlike the standard Public Assistance program, the subrecipient is responsible for actual costs that exceed the fixed cost estimate. If actual costs are less than the fixed cost estimate, the subrecipient may use all or part of excess funds for other eligible purposes, such as for additional cost-effective hazard mitigation measures to increase the resiliency of public infrastructure, as detailed in figure 4 below. Section 20601 of the Bipartisan Budget Act of 2018 authorized FEMA, when using the alternative procedures, to provide assistance to fund the replacement or restoration of disaster-damaged infrastructure that provides critical services—such as medical and educational facilities—to an industry standard without regard to pre-disaster condition. It also allows for restoration of components not damaged by the disaster when necessary to fully effectuate restoration of the disaster-damaged components to restore the function of the facility or system to industry standards. For example, through the Act, FEMA may fund the restoration of a disaster-damaged school building—which provides a critical service—to accepted industry standards applicable to the construction of education facilities. Therefore, according to FEMA policy, if the school building was not up to industry standards, or in poor condition prior to the 2017 hurricanes, the Act allows FEMA to fund the restoration of this building to a better condition than it was in prior to the storms. Further, the Additional Supplemental Appropriations for Disaster Relief Act of 2019 (Supplemental Relief Act), which was signed into law on June 6, 2019, provides additional direction to FEMA in the implementation of section 20601. Following the Supplemental Relief Act, FEMA issued additional guidance in September 2019 that includes information on eligibility and applicable industry standards. Since the 2017 hurricanes, FEMA has obligated nearly $6 billion in Public Assistance program funding for 1,558 projects across Puerto Rico, according to our analysis of FEMA’s data as of September 30, 2019 (see fig. 5). Specifically, FEMA had obligated approximately $5.1 billion for emergency work projects (categories A and B), $487 million for permanent work projects (categories C through G), and $315 million for management costs (Category Z). Of the nearly $6 billion FEMA has obligated, Puerto Rico has expended approximately $3.9 billion as of September 30, 2019—about 65 percent of total Public Assistance program obligations to Puerto Rico—to reimburse subrecipients for completed work. As shown in table 1, Puerto Rico has expended about $3.7 billion for emergency work projects, $39 million for permanent work projects, and $104 million for management costs. The majority of FEMA’s obligations and the funding Puerto Rico expended as of September 30, 2019, are for emergency work because these projects began soon after the disasters struck and focused on debris removal and providing assistance to address immediate threats to life and property. In contrast, permanent work projects take time to identify, develop, and ultimately complete as they represent the longer- term repair and restoration of public infrastructure, such as a sports center in Caguas, Puerto Rico, as shown in figure 6 below. FEMA and Puerto Rico officials identified challenges in developing Public Assistance projects in Puerto Rico. Specifically, they cited: (1) delays in establishing a cost estimating guidance for projects in Puerto Rico, (2) the large number of damaged sites that require finalized fixed cost estimates, and (3) challenges with the implementation of the flexibilities authorized by section 20601 of the Bipartisan Budget Act. Delays in establishing cost estimating guidance. Given the importance of reaching mutual agreement on fixed cost estimates for alternative procedures projects, FEMA and Puerto Rico have taken a deliberative approach to establishing the data and procedures that will be used to develop these fixed cost estimates. This includes, among other things, adapting the way FEMA estimates costs to the specific post- disaster economic conditions in the territory, including developing exceptions to FEMA’s cost estimating guidance. According to FEMA, these exceptions were developed to account for risk, including higher anticipated costs due to increased demand for labor, equipment, and materials in Puerto Rico’s post-disaster economy. To develop these exceptions, FEMA and the central recovery office established a Center of Excellence staffed with mutually agreed upon representatives. FEMA used cost estimators from RAND Corporation (RAND) as their chosen representatives, while the central recovery office hired separate contractors as their representatives. According to FEMA officials, the Center of Excellence was established, among other things, to involve Puerto Rico in developing cost estimating guidance and to ensure that the exceptions made to FEMA’s Cost Estimating Format were agreeable to both parties. However, this approach has been beset by delays. For example, it took nearly one year for Puerto Rico to hire its chosen representatives to the Center of Excellence. According to FEMA, the central recovery office did not select members for the Center of Excellence until February 2019, which delayed progress on the development of finalized fixed cost estimates for permanent work. In July 2019, FEMA leadership signed an agreement establishing the exceptions to FEMA’s cost estimating guidance based on an assessment conducted by a panel of FEMA engineers. These exceptions are intended to address certain costs specific to post-disaster conditions in Puerto Rico, for example adjustments to account for increased labor and material costs. Large number of damaged sites requiring a fixed cost estimate. In addition, FEMA and Puerto Rico officials have cited the large number of sites requiring damage assessments, project development, and mutually agreed-upon fixed cost estimates as a challenge. As of September 30, 2019, FEMA identified a total of 9,344 damaged sites in various stages of development. According to FEMA, 6,304 sites (67.5 percent of total sites identified) have completed damage assessments; 3,021 sites (32.3 percent of total sites identified) are pending the completion of damage assessments to begin project development; and 19 projects (0.2 percent of total sites identified) have finalized fixed cost estimates. According to FEMA guidance, October 11, 2019, was the deadline for completing fixed cost estimates for Public Assistance alternative procedures projects. However, on October 8, 2019, officials from the central recovery office requested an extension to the deadline, which FEMA granted. FEMA officials acknowledged that significant work remains on the part of Puerto Rico, subrecipients, and FEMA towards developing fixed cost estimates for all Public Assistance alternative procedures projects in Puerto Rico. According to FEMA officials, as of October 2019, FEMA and Puerto Rico are working together to establish specific time frames for the completion of fixed cost estimates. Implementation challenges with Section 20601 of the Bipartisan Budget Act of 2018. Puerto Rican government and FEMA officials identified challenges with the implementation of the flexibilities authorized by section 20601 of the Bipartisan Budget Act. As previously discussed, this section of the Act allows for the provision of assistance under the Public Assistance alternative procedures to restore disaster-damaged facilities or systems that provide critical services—such as medical and educational facilities to an industry standard without regard to pre-disaster condition. Officials from Puerto Rico’s central government stated that they disagreed with FEMA’s interpretation of the types of damages covered by section 20601 of the Bipartisan Budget Act of 2018. In response, FEMA officials in Puerto Rico stated they held several briefings with Puerto Rico’s central recovery office to explain FEMA’s interpretation of the section, and released new guidance in September of 2019. It is too soon to assess the impact this guidance may have on current and future projects, but we will continue to examine this in future work. We will continue to monitor the status of FEMA’s cost estimating process, the development of the remaining fixed cost estimates for permanent work and the impact of FEMA’s new guidance on the implementation of section 20601 of the Bipartisan Budget Act. As Puerto Rico is responsible for any costs that exceed fixed cost estimates for large infrastructure projects under the alternative procedures, FEMA has adapted its guidance for estimating costs to ensure that these estimates accurately reflect the total costs of Public Assistance projects. As previously mentioned, FEMA and Puerto Rico established a Center of Excellence to develop proposed exceptions to adapt FEMA’s Cost Estimating Format—the agency’s standard guidance used for Public Assistance cost estimating nationwide—to more accurately estimate costs in Puerto Rico. After consideration of these proposals, FEMA approved two exceptions: (1) a cost factor to account for local labor, equipment, and material costs in Puerto Rico, and (2) a future price factor and price curve to account for anticipated rises in construction costs over time due to the massive influx of disaster recovery funds, coupled with limited material and labor resources in Puerto Rico. Cost Factor: According to FEMA officials, during the development of a cost factor by the Center of Excellence, FEMA learned that Gordian, a company that provides local cost indices called RSMeans which FEMA uses as part of their standard Cost Estimating Format, was developing four localized cost indices to apply to San Juan, urban areas, rural areas, and remote island (the islands of Vieques and Culebra) areas of Puerto Rico. FEMA officials told us that these cost indices compile location-specific construction costs for each of the four areas. In 2019, a panel of FEMA engineers assessed the methodologies proposed by RAND, the Center of Excellence, and the RSMeans localized indices for Puerto Rico. On July 12, 2019, in agreement with the panel’s assessment, FEMA decided to use RSMeans’s localized cost indices to act as the cost factor for fixed cost estimates in Puerto Rico beginning on September 27, 2019. For fixed cost estimates developed before this date, FEMA used a different cost index that RSMeans had previously developed for San Juan. According to FEMA, cost estimates signed before September 27, 2019 using RSMeans’s San Juan cost index as the cost factor are considered final. Future Price Factor and Curve: According to FEMA officials, FEMA began using a future price factor—an economic model based on expected construction conditions to estimate construction costs across ten years—in July 2019 to estimate costs in Puerto Rico. FEMA is using this future price factor along with the cost factor. FEMA has also asked RAND to develop a future price curve, an analysis that will adjust as time goes on to account for changing economic conditions, to eventually replace the future price factor. FEMA estimates that RAND will take until November 2019 to develop the future price curve, and that the future price factor is being used in the meantime. FEMA officials stated that cost estimates produced using the future price factor are considered final and will not be eligible for revisions in the future once FEMA implements the future price curve. According to FEMA officials, the use of the cost factor combined with the future price factor and curve are intended to adapt FEMA’s cost estimating guidance to the specific post-disaster economic conditions in Puerto Rico. FEMA’s cost estimating guidance for Public Assistance fully or substantially met nine of the 12 steps from GAO’s Cost Estimating and Assessment Guide (GAO Cost Guide). However, the guidance partially met two and minimally met one of the remaining cost estimating steps, as shown in figure 7 below. The GAO Cost Guide outlines best practices for cost estimating and presents 12 steps that, when incorporated into an agency’s cost estimating guidance, should result in reliable and valid cost estimates that management can use to make informed decisions. A reliable cost estimate is critical to the success of any construction program. Such an estimate provides the basis for informed decision making, realistic budget formulation and program resourcing, and accountability for results. For example, FEMA, Puerto Rico and subrecipients rely on cost estimates to help ensure that funding is sufficient for the costs of the Public Assistance projects carried out under the fixed cost estimate. Accurate and reliable cost estimating is especially important in Puerto Rico where all large permanent Public Assistance projects are being developed under the alternative procedures, which require a fixed cost estimate that cannot be revised once the award is made. Given Puerto Rico’s financial situation, accurate cost estimates are necessary so that Puerto Rico has adequate funds to complete Public Assistance projects. For example, on the basis of our analysis, we determined that FEMA’s guidance fully met the step to “define the estimate’s purpose” because it describes the estimate’s purpose, level of detail required, and overall scope. In addition, the guidance provides a time frame for which the estimates must be developed and reach agreement. FEMA’s guidance substantially met another step, “identify the ground rules and assumptions”, because it provides measures to ensure assumptions are not arbitrary, are founded on expert judgments, and are documented. However, we rated this step as substantially met instead of fully met because FEMA’s guidance does not address all of GAO’s best practices for ground rules and assumptions. For example, it does not discuss the risk of an assumption being incorrect and the resultant effect on the cost estimate. Additionally, FEMA guidance substantially met the step to “document the estimate” because it contains, among other things, basic information about the project and the estimate; a description of the scope of work; the basis for the estimate; and supporting backup information. However, we assessed this step as substantially met instead of fully met because FEMA policy does not require documentation to include a discussion of high risk areas. Further, we found that FEMA’s guidance for cost estimating does not fully or substantially meet three steps: (1) conduct a sensitivity analysis; (2) obtain the data; and (3) conduct a risk and uncertainty analysis. Sensitivity analysis (Minimally met): We found that FEMA’s cost estimating guidance only minimally met the best practice regarding sensitivity analysis. A sensitivity analysis addresses some of the uncertainty in a cost estimate by testing assumptions and other factors that could change cost. By examining each assumption or factor independently, while holding all others constant, the cost estimator can evaluate the results to discover which assumptions or factors most influence the estimate. A sensitivity analysis also requires estimating the high and low uncertainty ranges for significant cost driver input factors. According to the GAO Cost Guide, when an agency does not identify the effect of uncertainties associated with different assumptions, this increases the chance that decisions will be made without a clear understanding of these impacts on costs. According to FEMA officials, FEMA’s cost estimating guidance accounts for construction, cost, and market risks over time which allows FEMA to plan and estimate costs for unknown or unforeseen circumstances such as cost escalation or overhead. In addition, FEMA officials stated that their use of RSMeans unit costs, a benchmark industry standard based on ongoing iterative analysis of construction costs nationwide, allows FEMA to account for fluctuations and uncertainties in the market. However, we rated this step as minimally met because FEMA guidance does not indicate that cost estimators are to conduct a sensitivity analysis as part of FEMA’s cost estimating process. Specifically, the guidance does not require that an estimator examine the effect of changing assumptions and the effect these changes could have on a cost estimate. Since the guidance does not direct estimators to conduct a sensitivity analysis, estimators may not fully understand which variable most affects the cost estimate and FEMA risks making decisions without a clear understanding of the impact of costs. Obtaining the data (Partially met): We found that FEMA’s cost estimating guidance only partially met the best practice for obtaining data—assembling information to serve as the foundation of a cost estimate. The quality of the data obtained affects a cost estimate’s overall credibility. Depending on the data quality, an estimate can range from a mere guess to a highly defensive cost position. We found that FEMA did not meet some of the best practices for obtaining data. Specifically, FEMA’s guidance did not outline procedures for making sure data was validated using historical data as a benchmark for reasonableness. In addition, FEMA’s guidance did not stipulate that data be normalized to remove the effects of inflation or analyzed with a scatter plot to determine trends and outliers. As mentioned previously, FEMA used a city cost index based on San Juan as an interim measure to estimate costs throughout Puerto Rico until September, 2019 when FEMA began using additional cost indices to target costs in particular regions of Puerto Rico. Similarly, FEMA has been using a static future price factor as an interim measure until a more dynamic and iterative future price curve is finalized. FEMA does not plan to adjust cost estimates developed using these interim measures. Without adjusting these costs when better data becomes available consistent with the obtaining the data step, FEMA risks creating estimates that may not be based on accurate data. According to FEMA, estimates are developed based on historical costs or nationally available industry standard data. In addition, FEMA officials stated that FEMA does not revisit cost estimates to reflect updated market conditions or newly available cost information because FEMA uses an industry standard cost database that is updated quarterly. FEMA officials stated that the interim measures used to estimate costs are intended to enable work to continue and cost estimates to be developed while the future cost curve is being developed. However, we rated the step relating to obtaining data as partially met because without finalizing the future cost curve, and updating estimates to reflect this information, estimates may not be based on accurate data. Additionally, while the use of industry standard cost estimating resources addresses some best practices for this step such as data normalization and data validation, industry data is only one of many sources referenced in FEMA’s guidance. For other data sources identified, FEMA guidance does not describe a process to analyze the data for cost drivers or to adequately document the data. Risk and uncertainty analysis (Partially met): We found that FEMA’s cost estimating guidance does not include best practices consistent with performing a statistical analysis of risk to determine a range of possible costs and the level of confidence in achieving the estimate. By conducting a risk and uncertainty analysis, a cost estimator can model the effect of schedules slipping and missions changing, allowing for a known range of potential costs. Having a range of costs around a point estimate is useful to decision makers because it conveys the level of confidence in achieving the most likely cost and informs estimators about potential risks. We found that FEMA’s cost estimating guidance does not require a statistical analysis of risks to be performed to determine a range of possible costs. While contingencies are accounted for within the guidance, they are not derived from a statistical analysis, nor do they reflect a level of confidence in the estimate. According to FEMA, risks associated with changing costs and conditions over the life of a Public Assistance alternative procedures construction project is not a risk that the federal government takes on. Rather, the risk is transferred to the recipient and subrecipients responsible for executing work using Public Assistance alternative procedures funding. In addition, FEMA officials told us that alternative procedures funding is not always used to restore facilities to pre-disaster condition, and therefore may not represent the final cost of work completed. In addition, the procedures are designed to incentivize subrecipients to manage grants and use excess funds for eligible work, as described earlier. However, GAO’s Cost Guide states that point estimates alone are insufficient for good decision- making. For management to make good decisions, the program estimate must reflect the degree of uncertainty, so that a level of confidence can be given about the estimate regardless of the entity holding the risk. In the case of alternative procedures projects in Puerto Rico, where actual costs that exceed the estimate are borne by the recipient or subrecipient, estimates that accurately reflect the degree of uncertainty are important in establishing a level of confidence about the estimate. While FEMA fully or substantially met nine of the 12 steps in the GAO Cost Guide, FEMA could improve its cost estimating guidance to ensure that all best practices in the 12 steps in the GAO Cost Guide are fully met. In doing so, FEMA could further enhance the reliability of its cost estimating guidance. In response to the complexity of the recovery, as well as the nature of change in a recovery environment, FEMA has developed and issued guidance that is specific to the implementation of the Public Assistance program in Puerto Rico. As previously discussed, disaster recovery in Puerto Rico is a complex and dynamic process that requires the coordination of many entities, including FEMA, the government of Puerto Rico, and numerous subrecipients. Recovery in Puerto Rico also involves the use of Public Assistance structures including alternative procedures and new flexibilities afforded to FEMA under the Bipartisan Budget Act of 2018. FEMA officials told us that many elements of the Public Assistance process in Puerto Rico are the same as in other declared disasters across the United States. Therefore, according to FEMA officials, the standard guidance for the Public Assistance program, Public Assistance Policy and Procedures Guide (Policies and Procedures Guide), generally applies in Puerto Rico. FEMA has also developed policies and guidance to address the specific recovery circumstances in Puerto Rico. For example, in April of 2018 and September of 2019, FEMA published the Public Assistance Alternative Procedures Guide for Permanent Work to clarify how FEMA would implement the program in Puerto Rico. This guidance describes the scope and limitations of the alternative procedures; highlights changes to aspects of the Public Assistance program to which these procedures apply; identifies responsibilities for certain activities; and documents timelines for key actions and decisions. FEMA also issued a policy on the agency’s implementation of section 20601 of the Bipartisan Budget Act as it applies in Puerto Rico in September of 2018, detailing the applicability of the section to specific critical services and outlining eligible industry standards for purposes of authorized projects, among other things. Following the Supplemental Relief Act, FEMA issued guidance in September 2019 that includes additional information on eligibility and applicable industry standards. According to FEMA officials, FEMA has also developed and implemented training specific to recovery in Puerto Rico. This training has included presentations to the central recovery office and subrecipients on the flexibilities of the Bipartisan Budget Act and alternative procedures, among other things. FEMA has iteratively developed, refined, and clarified Public Assistance guidance in Puerto Rico to respond and adapt to changing recovery conditions since the 2017 hurricanes. While iterative and responsive guidance is necessary in a complex and changing recovery, the pace of change necessitates that all involved recovery entities have real-time accessibility to current applicable FEMA guidance. Officials from the central recovery office and four Puerto Rico government agencies we spoke with stated that they did not consistently have the guidance they needed to implement the Public Assistance program. For example, an official from one Puerto Rico agency said that they delayed starting on any large Public Assistance projects through alternative procedures because they were waiting for FEMA to issue additional guidance. Similarly, we reported in March 2019 that four municipal officials stated that they were waiting on additional instruction from FEMA to establish more clear and consistent guidance to begin projects in Puerto Rico. According to FEMA officials, the agency works with Puerto Rico government officials and subrecipients to provide relevant guidance and technical assistance throughout the Public Assistance project development process. However, we found that pertinent guidance may not be shared with key recovery partners. For example, FEMA officials told us that the Standard Operating Procedure for Alternative Procedures (SOP) was available as of March 2019, but remains in draft form as of October 2019, pending finalized information about cost estimating procedures. This SOP provides instruction on specific procedures to implement the Public Assistance alternative procedures guide. In April 2019, FEMA officials described the SOP as a “living document”; they also stated that the draft SOP is in effect and has been sent to the central recovery office for further dissemination to subrecipients. While the SOP document is still in draft, according to FEMA officials, it is operative guidance that FEMA expects the central recovery office to disseminate to subrecipients. However, in June 2019, central recovery office personnel told us they did not view the SOP as being in effect as it was still in draft form. As such, central recovery office officials stated they had not distributed the SOP to subrecipients. FEMA officials stated that they rely on the central recovery office to disseminate at least some FEMA guidance and policy to subrecipients in Puerto Rico, including municipalities and government agencies. As the recipient for all Public Assistance funding in Puerto Rico, the central recovery office is responsible for monitoring and providing technical assistance to subrecipients to ensure that federal funding is used in accordance with federal statutes, regulations, and the requirements of the grant. FEMA officials also stated that subrecipients have an assigned FEMA point of contact to assist them through the project development process, including communicating policy information and updates. However, municipal and Puerto Rico agency officials we spoke to said that confusion persisted in part due to changing points of contact. FEMA’s reliance on the central recovery office or individual FEMA staff to deliver and distribute FEMA guidance poses a risk that the guidance is not made accessible to all partners involved in recovery, including subrecipients. While FEMA officials told us that FEMA assigns a point of contact to subrecipients to provide guidance and other necessary information throughout the project development process, Puerto Rico officials described a significant amount of “back and forth” with FEMA regarding requests for clarification, guidance, or instruction. FEMA officials acknowledge that FEMA has faced difficulties in disseminating information in Puerto Rico. According to FEMA officials, FEMA does not maintain a repository of Public Assistance policies and guidance available to all relevant recovery partners. The accessibility of FEMA guidance is especially important because FEMA releases iterative guidance to respond and adapt to changing recovery circumstances, such as updated legislation, among other things. Misunderstandings across recovery partners about guidance applicability raise concerns that subrecipients do not understand which guidance is currently in effect or how they should proceed in accordance with FEMA policy. Without real-time access to the totality of FEMA’s current applicable guidance, recovery partners risk using guidance that has been revised or replaced. According to FEMA’s National Disaster Recovery Framework, the federal government has the role of ensuring that information is distributed in an accessible manner such that all partners are informed of and aware of the recovery process. Developing a repository of current applicable policy and guidance and making it available to all relevant recovery partners in Puerto Rico, including subrecipients, would improve the accessibility of the information and provide greater assurance that recovery partners are aware of current applicable guidance. Following the 2017 hurricanes, Puerto Rico took several steps to provide management and oversight of the Public Assistance program to ensure the program is implemented in compliance with applicable laws and regulations, as well as FEMA policies and guidance. Specifically, Puerto Rico (1) established a central recovery office to provide management and oversight of recovery funds; (2) developed an administrative plan, as required by FEMA policy; (3) developed an internal controls and recovery management plan; and (4) created a system to oversee and assess subrecipient risk. First, in accordance with Amendment 5 to the President’s disaster declaration, the central recovery office has been supported by third-party experts to help it establish its structure and carry out its management and oversight mission. Specifically, the central recovery office has hired contractors to help perform the following functions: Design a management guide and assess subrecipient risk. According to central recovery office officials, the office hired contractors to develop management protocols and guidance to ensure compliance with federal and state law, regulation, and guidance. The office also tasked these contractors with developing a system to oversee subrecipients using risk-based oversight. Provide technical assistance. Central recovery office officials also hired contractors to provide technical assistance and advise Puerto Rico’s government agencies and municipalities regarding recovery processes. This includes helping subrecipients define the scope of damages, and providing technical assistance to develop Public Assistance projects, among other things. The recovery office also tasked these contractors with overseeing grant accounting and reviewing reimbursement requests from subrecipients for eligible Public Assistance work performed. Develop data systems to track the central recovery office’s work. The central recovery office launched an online transparency portal, with the assistance of contractors, that is intended to provide a breakdown of FEMA Public Assistance and other federal funding made available for disaster recovery in Puerto Rico. According to central recovery office officials, in addition to the development of the online transparency portal, contractor personnel also developed systems to track internal recovery data. Second, to meet FEMA reporting requirements, the central recovery office developed an administrative plan—or FEMA State Agreement—in 2019 for the Public Assistance program following the 2017 hurricanes. This plan outlines the central recovery office’s management and oversight activities as well as the procedures that Puerto Rico must follow in implementing the programs. Puerto Rico is responsible, as required in the FEMA State Agreement, to ensure that subrecipients are in compliance with the conditions of the disaster grant award. For example, the plan emphasizes FEMA’s requirement that Puerto Rico submit quarterly progress and financial reports on the status of projects. Further, the plan describes Puerto Rico’s specific roles and responsibilities for managing and overseeing the program. For example, according to the Puerto Rico 2019 Public Assistance Administrative Plan, the central recovery office is responsible for, among other things, processing requests for time extensions to complete projects and conducting quarterly reviews, site inspections, and audits to ensure program compliance. Third, in addition to the administrative plan, in March 2019, the central recovery office released the Disaster Recovery Federal Funds Management Guide (management guide) that includes an internal controls plan and other policies and procedures for managing recovery funds. The management guide’s 14 chapters outline roles, responsibilities, policies and procedures on various recovery functions including procurement, payment and cash management, and subrecipient management and oversight, among other things. FEMA officials told us that they reviewed portions of the management guide, including sections on the central recovery office’s payment and cash management plan and subrecipient oversight. Further, FEMA worked with the central recovery office to make revisions to the plan, which included, adding clarifying information and correcting instances of duplication in the guidance, among other things. In addition, the central recovery office, with the help of contractors, is taking steps to assist subrecipients in meeting compliance requirements and supplementing their management capacity. FEMA and Puerto Rico government agency officials cited varying levels of capacity to manage federal grant funds, including Public Assistance funding. For example, agency officials at one government agency we spoke with stated that they were performing their own federal grants management and had prior experience managing large federal funds. Other Puerto Rico government officials we interviewed reported that central recovery office contractors have helped augment capacity to oversee federal funds. For example, officials from one subrecipient, a Puerto Rico public corporation, said that their agency did not have prior experience managing federal funds on such a large scale. The official told us that in order to bolster the capacity of the agency to oversee these grant funds, central recovery office contractors work closely with the agency to help them manage Public Assistance funding. Similarly, officials at one Puerto Rico government agency stated that the central recovery office offered help on uploading and validating grant data. Fourth, as detailed in its management guide, the central recovery office has also developed criteria to evaluate subrecipients’ risk of noncompliance with federal laws and regulations, as well as FEMA policy. According to the procedures outlined in the central recovery office’s management guide, each subrecipient is to be assessed annually to determine whether they are at a low, moderate, or high risk for noncompliance. The central recovery office is to place additional award conditions on subrecipients with risk factors identified through the risk assessment process. These may include additional oversight and more frequent on-site visits from the central recovery office. Additionally, central recovery office guidance states that corrective actions are to be taken in cases when deficiencies are found during audits. In March 2019, we reported that FEMA instituted a manual reimbursement process in November 2017 for subrecipients in Puerto Rico for federal funds, including Public Assistance funds, to mitigate fiduciary risk and decrease the risk of misuse of funds. Specifically, FEMA officials stated that they decided to institute this process because the government of Puerto Rico had expended funds prior to submitting complete documentation of work performed. According to FEMA officials, they also decided to institute the manual reimbursement process due to Puerto Rico’s financial situation, weaknesses in internal controls, and the large amount of recovery funds, among other things. The manual reimbursement process required that FEMA review each reimbursement request before providing Public Assistance funds to mitigate risk and help ensure financial accountability. In Puerto Rico, the manual reimbursement process requires that the central recovery office fill out the Office of Management and Budget’s Standard Form 270 and submit supporting documentation to FEMA before obligated funds can be withdrawn by Puerto Rico through the central recovery office and reimbursed to subrecipients. Subsequently, FEMA must review the submitted Standard Form 270 and all project documentation for completeness, compliance, and accuracy before disbursing funds to the recipient. In cases where FEMA requires additional documentation to process a Standard Form 270 request, FEMA will submit requests for information asking the central recovery office to supply the information needed for FEMA to complete the review. On March 25, 2019, FEMA and the government of Puerto Rico, through the central recovery office, signed an agreement allowing the central recovery office to directly access federal grant funds and reimburse subrecipients for Public Assistance work they perform. During FEMA’s review of the central recovery office’s management guide, FEMA asked for revisions to sections, including chapters related to payment and cash management and subrecipient management and monitoring. According to the March 2019 agreement, these policies and procedures were developed in collaboration with FEMA, and comments and concerns provided by FEMA were addressed. FEMA officials also told us that they sampled Public Assistance grant documentation for completeness to ensure that the reimbursement requested was eligible for payment. According to FEMA officials, FEMA communicated minor discrepancies with the central recovery office for resolution, but said that they did not find any significant discrepancies during their completeness review. On April 1, 2019, FEMA removed the manual reimbursement process and began a transition to allow the central recovery office to make direct payments to subrecipients. In July 2019, FEMA announced that it would reinstate the manual reimbursement process due to, “ongoing leadership changes within the Puerto Rican government, combined with continued concern over Puerto Rico’s history of fiscal irregularities and mismanagement.” FEMA said that these additional steps are being taken in order to protect the federal investment in Puerto Rico’s recovery. We previously reported that FEMA and central recovery office officials told us that the manual reimbursement process caused delays in reimbursements, but once FEMA increased the number of personnel devoted to reimbursement reviews, delays decreased. In September 2019, FEMA once again lifted the manual reimbursement process following a meeting between FEMA and Governor Vásquez’s senior leadership. According to FEMA, the agreement to remove the manual reimbursement process is contingent on Puerto Rico’s continued ability to implement the mutually-acceptable internal controls plan. FEMA officials also stated that they are selecting samples from fiscal year 2019 to test Puerto Rico’s internal controls, and plan to move to a quarterly testing routine after testing for fiscal year 2019 is complete. As part of our ongoing review, we will continue to monitor the central recovery office’s management and oversight of Public Assistance funding, as well as of FEMA’s oversight of the federal investment in Puerto Rico’s recovery. After the devastation of the catastrophic 2017 hurricane season, FEMA and Puerto Rico face a recovery of enormous scope. Puerto Rico estimates that $132 billion in funding will be needed to repair and reconstruct the infrastructure damaged by the hurricanes through 2028, and FEMA has identified nearly ten thousand damaged sites in need of Public Assistance funding. FEMA has taken steps to adapt its guidance to estimate costs to post-disaster conditions in Puerto Rico, but strengthening its cost estimating guidance could help FEMA provide greater assurance that its cost estimating guidance for Public Assistance projects is reliable. In addition, given the large number of individuals and entities involved in Puerto Rico’s complex recovery, ensuring that all recovery partners have easy access to the most current applicable policy and guidance could help clarify which FEMA guidance and policies are in effect. We are making the following two recommendations to FEMA: The FEMA administrator should revise FEMA’s cost-estimating guidance for Public Assistance projects to fully align with all 12 steps in the GAO Cost Estimating and Assessment Guide. (Recommendation 1) The FEMA administrator should develop a repository for all current applicable Public Assistance policies and guidance for Puerto Rico and make it available to all recovery partners, including subrecipients. (Recommendation 2) We provided a draft of this product to FEMA, DHS and Puerto Rico’s Central Office of Recovery, Reconstruction, and Resilience (central recovery office) for comment. In its comments, reproduced in appendix III, DHS concurred with our recommendations. FEMA also provided technical comments, which we incorporated as appropriate. DHS concurred with our first recommendation that FEMA revise its cost- estimating guidance for Public Assistance projects to fully align with all 12 steps in the GAO Cost Estimating and Assessment Guide. DHS stated that FEMA will create a quality assurance checklist as an addendum to FEMA’s Cost Estimating Format (CEF) to ensure that cost estimates reflect best practices from the GAO Cost Estimating and Assessment Guide. This action is a positive step to addressing our recommendation and we will monitor FEMA’s efforts to complete this work. In DHS’s concurrence to our second recommendation that FEMA develop a repository for all current applicable Public Assistance policies and guidance for Puerto Rico to be made available to all recovery partners, DHS requested that GAO consider this recommendation resolved and closed as implemented. DHS stated that FEMA maintains Public Assistance policy and guidance documents, including those specific to Puerto Rico, on the agency’s public web site, which FEMA stated it will continue to update. DHS also stated that FEMA maintains non-publicly available reference documents on the agency’s internal web site through the Grants Manager and Grants Portal systems. As we noted in our report, Puerto Rico’s recovery is a complex and dynamic process that requires the coordination of many recovery partners, including numerous municipalities and commonwealth agencies. For this reason, ensuring that information is distributed in an accessible manner would provide greater assurance that all recovery partners are aware of the most current and applicable Public Assistance policies and guidance. We will monitor FEMA’s public and internal web sites, including policy and guidance updates, to assess whether the actions outlined by FEMA meet the intent of our recommendation. COR3 also provided comments to our draft report, which we reproduced in appendix IV. In its comments, COR3 stated that it works with Public Assistance applicants to, among other things, provide technical assistance and training, and to monitor projects. COR3 also stated that it has established joint efforts with FEMA to improve COR3’s technical assistance, as well as compliance and monitoring efforts. We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the Administrator of FEMA, the Puerto Rico government, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you and your staff have any questions, please contact me at (404) 679- 1875 or curriec@gao.gov. GAO staff who made key contributions to this report are listed in appendix V. Since September 2017, the Federal Emergency Management Agency (FEMA) obligated nearly $6 billion in Public Assistance grant funding for 1,558 projects across Puerto Rico as of September 30, 2019. Specifically, FEMA had obligated $5.13 billion for emergency work projects (categories A and B), about $487 million for permanent work projects (categories C through G), and $315 million for management costs, (category Z). As of that date, Puerto Rico expended nearly $3.9 billion—about 65 percent of total Public Assistance obligations to Puerto Rico—to reimburse subrecipients for completed work. Of this, Puerto Rico expended about $3.7 billion (96 percent of all expended funds) for emergency work projects, $38.6 million (1 percent) for permanent work projects, and $104 million (3 percent) for management costs. The majority of FEMA’s obligations and the funding Puerto Rico expended as of September 30, 2019 are for emergency work projects because these projects began soon after hurricanes Irma and Maria struck and focused on debris removal and providing assistance to address immediate threats to life and property. In contrast, permanent work projects take time to identify, develop, and ultimately complete as they represent the longer-term repair and restoration of public infrastructure. While the data in this appendix represent the status of Public Assistance funding as of September, 2019, the amount of grant funding FEMA obligates and Puerto Rico expends will likely increase over time as additional projects are finalized and approved. Emergency Work. As of September 30, 2019, FEMA obligated a total of $5.13 billion for approximately 1,200 emergency work projects across Puerto Rico. These projects focus on debris removal activities and providing assistance to address immediate threats to life and property. Category A: Debris Removal. FEMA obligated $637.0 million and Puerto Rico expended $427.1 million for 331 projects focused on debris removal activities in Puerto Rico under category A. Category B: Emergency Protective Measures. FEMA obligated nearly $4.5 billion for 871 projects under Category B. Of this, Puerto Rico has expended $3.29 billion. For example, FEMA has obligated more than $140 million to the Puerto Rico Aqueducts and Sewer Authority under category B to fund emergency protective measures, including using back-up generators to supply water to the island after Hurricane Maria, among other things. Permanent Work. As of September 30, 2019, FEMA has obligated about $487.3 million for 159 permanent work (Categories C through G) projects in Puerto Rico. These projects focus on the restoration of disaster- damaged infrastructure or systems. Category C: Roads and Bridges. FEMA obligated $140.5 million and Puerto Rico has expended $32.8 million for 20 projects focused on the permanent repair of roads and bridges in Puerto Rico, such as the damage illustrated in figure 8 below. Category D: Water Control Facilities. As of September 30, 2019, FEMA has obligated $435,493 for three projects, of which approximately $150,000 has been expended. This includes work on heavy water control infrastructure, such as berms or levees. Category E: Buildings and Equipment. FEMA obligated $43.5 million and Puerto Rico expended nearly $4 million for 87 projects focused on repairing and rebuilding damaged public buildings and equipment, such as the school shown in figure 9 below. Category F: Utilities. Of the $487 million FEMA obligated for permanent work projects, the largest share, $282 million was obligated for nine projects related to utilities, such as architectural and engineering design services for design work for electricity grid recovery projects. For example, in June 2019, FEMA obligated $111 million for architectural and engineering design services for design work for electricity grid recovery projects. Puerto Rico has expended just over $1 million of the funding obligated for projects related to repairing utilities. Category G: Parks, Recreational and Other Facilities. FEMA obligated approximately $20.9 million and Puerto Rico has expended just over $600,000 across 40 projects focused on repairing parks, playgrounds, and other facilities. GAO’s Cost Estimating and Assessment Guide (GAO Cost Guide) outlines best practices pertaining to cost estimating principles, presenting 12 steps to create high-quality estimates. These steps are generally applicable in a variety of circumstances and range from defining the purpose of the estimate to obtaining data to presenting the estimate to management for approval. Application of these principles should result in reliable and valid cost estimates that management can use to make informed decisions. To assess the extent to which FEMA’s cost estimating policy aligns with these best practices, we compared FEMA’s information to the GAO Cost Guide. Specifically, we reviewed FEMA documents containing cost estimating information pertinent to Public Assistance projects including FEMA’s Public Assistance Alternative Procedures Guide for Permanent Work FEMA-4339-DR-PR (Alternative Procedures Guide) and FEMA’s Cost Estimating Format (CEF) for Large Projects Instructional Guide V2.1 (dated September 2009). We compared FEMA’s guidance for developing cost estimates outlined in these documents against the 12 best practices described in the GAO Cost Guide. We assessed the extent to which these documents aligned with the best practices on a five point scale. Fully met. FEMA provided complete evidence that satisfies the elements of the step. Substantially met. FEMA provided evidence that satisfies a large portion of the elements of the step. Partially met. FEMA provided evidence that satisfies about half of the elements of the step. Minimally met. FEMA provided evidence that satisfies a small portion of the elements of the step. Not met. FEMA provided no evidence that satisfies any of the elements of the step. Taken together, FEMA’s documents provided cost estimating information that either substantially or fully meets nine of the 12 cost estimating steps. Furthermore, the information partially met two of the 12 steps, and minimally met one of the 12 steps. Table 1 summarizes GAO’s assessment of the extent to which FEMA’s information aligns with the 12 steps identified in the GAO cost guide. Chris Currie, (404) 679-1875 or curriec@gao.gov. In addition to the contact named above, Joel Aldape (Assistant Director), Taylor Hadfield (Analyst in Charge), Michelle Bacon, Brian Bothwell, Lorraine Ettaro, Eric Hauswirth, Heidi Nielson, Danielle Pakdaman, Amanda Prichard, Kevin Reeves, and Mary Weiland made key contributions to this report. U.S. Virgin Islands Recovery: Additional Actions Could Strengthen FEMA’s Key Disaster Recovery Efforts. GAO-20-54. Washington, D.C.: November 19, 2019. Disaster Resilience Framework: Principles for Analyzing Federal Efforts to Facilitate and Promote Resilience to Natural Disasters. GAO-20-100SP. Washington, D.C.: October 23, 2019. Disaster Recovery: Recent Disasters Highlight Progress and Challenges. GAO-20-183T. Washington, D.C.: October 22, 2019. Wildfire Disasters: FEMA Could Take Additional Actions to Address Unique Response and Recovery Challenges. GAO-20-5. Washington, D.C.: October 9, 2019. Puerto Rico Electricity Grid Recovery: Better Information and Enhanced Coordination Is Needed to Address Challenges. GAO-20-141. Washington, D.C.: October 8, 2019. Emergency Management: FEMA’s Disaster Recovery Efforts in Puerto Rico and the U.S. Virgin Islands. GAO-19-662T. Washington, D.C.: July 11, 2019. 2017 Disaster Relief Oversight: Strategy Needed to Ensure Agencies’ Internal Control Plans Provide Sufficient Information. GAO-19-479. Washington, D.C.: June 28, 2019. Emergency Management: FEMA Has Made Progress, but Challenges and Future Risks Highlight Imperative for Further Improvements. GAO-19-617T . Washington, D.C.: June 25, 2019. Emergency Management: FEMA Has Made Progress, but Challenges and Future Risks Highlight the Imperative for Further Improvements. GAO-19-594T. Washington, D.C.: June 12, 2019. Disaster Assistance: FEMA Action Needed to Better Support Individuals Who Are Older or Have Disabilities. GAO-19-318. Washington, D.C.: May 14, 2019. Disaster Contracting: Actions Needed to Improve the Use of Post- Disaster Contracts to Support Response and Recovery, GAO-19-281. Washington, D.C.: April 24, 2019. 2017 Hurricane Season: Federal Support for Electricity Grid Restoration in the U.S. Virgin Islands and Puerto Rico. GAO-19-296. Washington, D.C.: April 18, 2019. FEMA Grants Modernization: Improvements Needed to Strengthen Program Management and Cybersecurity. GAO-19-164. Washington, D.C.: April 9, 2019. Disaster Recovery: Better Monitoring of Block Grant Funds Is Needed. GAO-19-232. Washington, D.C.: March 25, 2019. Puerto Rico Hurricanes: Status of FEMA Funding, Oversight, and Recovery Challenges. GAO-19-256. Washington, D.C.: March 14, 2019. Huracanes de Puerto Rico: Estado de Financiamiento de FEMA, Supervisión y Desafíos de Recuperación. GAO-19-331. Washington, D.C.: March 14, 2019. High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas. GAO-19-157SP. Washington, D.C.: March 6, 2019. U.S. Virgin Islands Recovery: Status of FEMA Public Assistance Funding and Implementation. GAO-19-253. Washington, D.C.: February 25, 2019. 2017 Disaster Contracting: Action Needed to Better Ensure More Effective Use and Management of Advance Contracts. GAO-19-93. Washington, D.C.: December 6, 2018. Continuity of Operations: Actions Needed to Strengthen FEMA’s Oversight and Coordination of Executive Branch Readiness. GAO-19-18SU. Washington, D.C.: November 26, 2018. Homeland Security Grant Program: Additional Actions Could Further Enhance FEMA’s Risk-Based Grant Assessment Model. GAO-18-354. Washington, D.C.: September 6, 2018. 2017 Hurricanes and Wildfires: Initial Observations on the Federal Response and Key Recovery Challenges. GAO-18-472. Washington, D.C.: September 4, 2018. Federal Disaster Assistance: Individual Assistance Requests Often Granted but FEMA Could Better Document Factors Considered. GAO-18-366. Washington, D.C.: May 31, 2018. 2017 Disaster Contracting: Observations on Federal Contracting for Response and Recovery Efforts. GAO-18-335. Washington, D.C.: February 28, 2018. Disaster Recovery: Additional Actions Would Improve Data Quality and Timeliness of FEMA’s Public Assistance Appeals Processing. GAO-18-143. Washington, D.C.: December 15, 2017. Disaster Assistance: Opportunities to Enhance Implementation of the Redesigned Public Assistance Grant Program. GAO-18-30. Washington, D.C.: November 8, 2017. Climate Change: Information on Potential Economic Effects Could Help Guide Federal Efforts to Reduce Fiscal Exposure. GAO-17-720. Washington, D.C.: September 28, 2017. Federal Disaster Assistance: Federal Departments and Agencies Obligated at Least $277.6 Billion during Fiscal Years 2005 through 2014. GAO-16-797. Washington, D.C.: September 22, 2016. Disaster Recovery: FEMA Needs to Assess Its Effectiveness in Implementing the National Disaster Recovery Framework. GAO-16-476. Washington, D.C.: May 26, 2016. Disaster Response: FEMA Has Made Progress Implementing Key Programs, but Opportunities for Improvement Exist. GAO-16-87. Washington, D.C.: February 5, 2016. Hurricane Sandy: An Investment Strategy Could Help the Federal Government Enhance National Resilience for Future Disasters. GAO-15-515. Washington, D.C.: July 30, 2015. Budgeting for Disasters: Approaches to Budgeting for Disasters in Selected States. GAO-15-424. Washington, D.C.: March 26, 2015. High-Risk Series: An Update. GAO-15-290. Washington, D.C.: February 11, 2015. Emergency Preparedness: Opportunities Exist to Strengthen Interagency Assessments and Accountability for Closing Capability Gaps. GAO-15-20. Washington, D.C.: December 4, 2014. Fiscal Exposures: Improving Cost Recognition in the Federal Budget. GAO-14-28. Washington, D.C.: October 29, 2013. Federal Disaster Assistance: Improved Criteria Needed to Assess a Jurisdiction’s Capability to Respond and Recover on Its Own. GAO-12-838. Washington, D.C.: September 12, 2012.", "summary": "In September 2017, two major hurricanes—Irma and Maria—struck Puerto Rico, destroying roads and buildings among other things. Puerto Rico estimates that $132 billion will be needed to repair and reconstruct infrastructure and services through 2028. FEMA is the lead federal agency responsible for assisting Puerto Rico to recover from these disasters. FEMA administers the Public Assistance program in partnership with Puerto Rico to provide funds to rebuild damaged infrastructure and restore services. GAO was asked to review federal recovery efforts in Puerto Rico. In this report, GAO examines, among other things, (1) the status of FEMA Public Assistance program funding and any challenges in implementing the program, (2) the extent to which Public Assistance cost estimating guidance addresses conditions in Puerto Rico and aligns with best practices, and (3) the extent to which FEMA has developed policies and guidance for the program and any challenges with these policies and guidance. GAO reviewed FEMA's cost estimation guidance as well as documentation and data on the Public Assistance program through September 2019. GAO conducted site visits to Puerto Rico and interviewed FEMA and Puerto Rico government officials regarding the status of recovery efforts. As of September 30, 2019, the Federal Emeregency Management Agency (FEMA) had obligated nearly $6 billion in Public Assistance grants to Puerto Rico for 1,558 projects since the September 2017 hurricanes. Of this $6 billion, $5.1 billion was obligated for emergency work projects such as debris removal and temporary power restoration. However, FEMA and Puerto Rico faced challenges in developing long-term, permanent work projects under the Public Assistance program. The large number of damaged sites and delays in establishing cost estimation guidance specific to Puerto Rico have also presented challenges to developing projects, according to FEMA and Puerto Rico officials. Both parties must agree to fixed cost estimates for these projects before work can begin. FEMA and Puerto Rico had approved fixed cost estimates for 19 projects as of September 2019, out of 9,344 damaged sites in Puerto Rico, such as schools, hospitals, and roads. FEMA and Puerto Rico have recently taken actions, including extending the deadline for fixed cost estimates, to address these challenges. However, it is too soon to assess the impact of these actions. FEMA has adapted its Public Assistance cost estimating guidance to accurately reflect costs in Puerto Rico but could improve the guidance to further enhance its reliability. GAO found that FEMA's guidance substantially or fully met best practices for nine of 12 steps included in the GAO Cost Estimating and Assessment Guide , such as documenting and defining the purpose of the estimate. However, FEMA could improve the guidance in three areas, including analyzing risks and future uncertainties that could affect these estimates. FEMA has developed Public Assistance policies and guidance to respond to complex recovery conditions in Puerto Rico. However, Puerto Rico government officials GAO spoke with stated that they were not always certain about how to proceed in accordance with FEMA policy because they did not consistently understand what guidance was in effect. Further, FEMA does not maintain a repository of Public Assistance guidance available to all recovery partners that includes current applicable guidance. Without real time access to current applicable guidance, recovery partners risk using guidance that has been revised or replaced. GAO recommends that FEMA (1) revise its cost estimating guidance for Public Assistance to more fully adhere to best practices and, (2) develop a repository of current applicable Public Assistance guidance available to all relevant recovery partners in Puerto Rico. The Department of Homeland Security concurred with these recommendations.", "document_type": "gao"}
{"report": "DOD’s 921 plan identifies eight initiatives across the covered activities and generally addresses most of the elements required under section 921. Specifically, section 921 required the CMO to provide a plan, schedule, and cost estimate for conducting its reforms of the covered activities. DOD’s plan provides a schedule for all eight efforts, and provides a cost estimate for all but one, which OCMO officials indicated was still under development. The plan identifies costs of at least $116.3 million to $116.8 million to implement these initiatives through fiscal year 2021. We discuss DOD’s funding of these costs later in this report. According to DOD’s plan, the eight initiatives have the following objectives: Civilian hiring improvement. Shorten the time needed to hire civilian employees, improve the matching of enterprise needs to employee competencies, and establish standard metrics and reports on performance of an improved hiring process. Human resources regulatory reform. Develop a new proposed legal authority that allows the department to simplify, streamline, and standardize civilian personnel policies. In addition, use regulatory reform to better recruit, compensate, and retain a qualified civilian workforce at DOD. Human resources service delivery. Establish a common human resources business and service delivery model, a standard set of performance measures, and a cost accountability structure that will be applied to all human resources service providers, with a focus on certain defense agencies and field activities. Strategic sourcing of sustainment and commodity procurement. Improve the buying power of the department, increasing data transparency related to sustainment and commodity procurement, and apply best-in-class cost and contract management practices with suppliers to drive higher performance and lower cost. Maintenance work packages and bills of material. Improve the accuracy of depot maintenance work packages and related bills of material and develop recommendations for process improvements. Munitions readiness. Produce an integrated tool capable of providing senior leaders with an effective assessment of all the variables associated with the health and readiness of the munitions inventory and the ability to assess options for correcting negative trends. Service requirements review boards. Expand the use of service requirement review boards—which review, validate, prioritize, and approve contracted services requirements to accurately inform the budget and acquisition process. Category management. Implement best practices for purchasing goods and services, such as consolidating separate requirements into single contracts, allowing DOD to achieve savings from volume discounts and develop tools aimed at focusing spending on contracts that meet certain best practices for management. Several of these initiatives address aspects of our prior recommendations related to the objectives of the initiatives. How findings and recommendations from GAO and agency inspectors general have been addressed in proposed reforms is among the key questions GAO has previously identified for assessing agency reform efforts. We found that DOD’s initiatives address aspects of our findings and recommendations, but in some cases do not fully address them. For example: In September 2018, we reported that at least six organizations within DOD, including three defense agencies and field activities and the three military departments, provide human resources services to other defense agencies or organizations. All perform the same types of human resources services, such as those related to civilian workforce hiring across DOD. We also reported that there is fragmentation and overlap within the defense agencies and field activities that provide human resources services to other defense agencies or organizations within DOD. This fragmentation and overlap has resulted in negative effects, such as inconsistent performance information regarding hiring, fragmented information technology systems, and inefficiencies associated with overhead costs. We recommended, and DOD concurred, that DOD collect consistent performance information and comprehensive overhead cost information as well as establish time frames and deliverables for key reform efforts. DOD’s human resource service delivery initiative is intended, in part, to address our recommendations. This initiative, however, is focused only on the defense agencies and field activities responsible for human resources service delivery, and does not include all human resources service providers we highlighted in our September 2018 report. In June 2016, we reported that the Defense Logistics Agency and the military services have some internal efficiency measures for supply and depot operations; however, they generally have not adopted metrics that measure the accuracy of planning factors that are necessary to plan efficient and effective support of depot maintenance. Additionally, the Defense Logistics Agency and the services do not track the potentially significant costs to supply and depot maintenance operations that are created by backorders. Further, we reported that without relevant metrics on cost and planning factors, DOD, the Defense Logistics Agency, and the services are unable to optimize supply and maintenance operations and may miss opportunities to improve the efficiency and effectiveness of depot maintenance. We recommended, and DOD concurred, that DOD, the Defense Logistics Agency, and the services develop metrics to monitor costs and accuracy of demand planning factors. DOD’s initiative on maintenance work packages and bills of material includes steps that may, in part, address these recommendations. Specifically, the initiative plans to assess the accuracy of bills of material, one of the planning factors we recommended DOD develop and implement metrics for, but does not include assessing the accuracy of other planning factors. In August 2017, we reported that DOD’s service requirement review boards were intended to prioritize and approve contracted services in a comprehensive portfolio-based manner to achieve efficiencies, but the military commands we reviewed did not do so. Instead, commands largely leveraged existing contract review boards that occurred throughout the year and focused on approving individual contracts. As a result, the review boards at these commands had minimal effect on supporting decisions within and across service portfolios or capturing efficiencies that could inform the commands’ programming and budgeting decisions. We recommended, and DOD concurred, that DOD clarify policies concerning the purpose and timing of the review board process. DOD’s initiative on service requirements review boards expands the use of these boards, and indicates that they are timed to inform budgets for the following fiscal year, but does not indicate whether guidance to do so has been provided. In its concurrence, DOD stated it would update the relevant DOD instruction to include this guidance, but, as of June 2019, DOD has not issued an updated instruction that includes this guidance. Although these initiatives intend to address aspects of our prior recommendations, assessing the feasibility of DOD’s reform effort is difficult because many of the planned initiatives entail collecting information that will lay the groundwork for later reforms. For example, the human resources service delivery initiative tasks the reform team to draft a project charter, collect and analyze information on human resources service providers within DOD, and eventually develop recommended courses of action for reform by fiscal year 2020. Similarly, the initiative on maintenance work packages and bills of material tasks the reform team to identify opportunities to improve processes, make recommendations to address deficiencies, improve efficiency, and improve material availability and then to develop an implementation plan for the recommendations by the end of fiscal year 2019, with implementation beginning in fiscal year 2020. OCMO officials told us that DOD is making progress in implementing the 921 plan’s initiatives according to the schedules contained in the plan, and they provided summary documentation stating that progress has been made on five of the eight initiatives. However, OCMO did not provide sufficiently detailed documentation for us to independently assess progress on any of the initiatives. Specifically, OCMO provided us some documentation on the progress of the eight initiatives, but this information varied by initiative and was limited. As a result, we were unable to independently assess and verify DOD’s progress in implementing its initiatives. Specifically: For the human resources regulatory reform, civilian hiring improvement, and human resources service delivery initiatives, OCMO provided briefing materials on the status of each milestone under the initiatives, indicating that those initiatives are progressing according to the schedule in the plan. However, DOD did not provide separate underlying documentation for each milestone. For example, under the plan, the teams conducting these initiatives were to have established by June 2019 a common DOD process and metrics for civilian hiring, prepared drafts of updated DOD policies and fiscal year 2020–2021 talent management guidance, and collected and mapped different human resources service delivery models. However, OCMO did not provide documentation of the common DOD process and metrics for civilian hiring, drafts of updated policies and guidance, or human resources service delivery model maps. For the service requirements review boards initiative, OCMO provided documentation stating that the service requirements review boards had largely been completed on schedule, but did not provide information on the outcomes of these boards. OCMO officials told us that delays in completing 3 of 69 boards had prevented them from fully meeting planned deadlines. For the category management initiative, OCMO officials told us that the first two quarterly “sprints”—reviews of different contracts or categories of goods or services to identify savings—for fiscal year 2019 had been completed and the third was in progress, but did not provide documentation to support this assertion. For example, OCMO did not provide information on the outcomes of the sprints. For the strategic sourcing of sustainment and commodity resources, maintenance work packages and bills of material, and munition readiness initiatives, DOD did not provide any documentation on the progress of the initiatives. While most of DOD’s initiatives included in its plan identify either performance metrics or targets, five of the eight initiatives also state that part of the work of the initiatives will be to establish such metrics or targets. Among our key questions for assessing agency reform efforts is the extent to which the agency has established clear outcome-oriented goals and performance measures for the proposed reforms, and whether the agency has put processes in place to collect the needed data and evidence that will effectively measure the reform’s goals. Identifying and collecting this information can lay the groundwork for further reform efforts. Moreover, we found that objectives for some of the initiatives in DOD’s plan are similar to those presented in prior plans with deadlines that have already passed, suggesting that progress on some initiatives is going more slowly than the department originally anticipated. For example, DOD’s August 2017 report to Congress on restructuring the CMO organization included an initiative to create a single civilian personnel system and rating system for certain employees by the middle of fiscal year 2018. DOD’s 921 plan contains a similar initiative on human resources regulatory reform, which aims to develop standardized civilian personnel policies and processes. Development of the initiative is not scheduled to be completed until the end of fiscal year 2019, and implementation would not occur until fiscal year 2020, at the earliest, compared to the original fiscal year 2018 deadline for the initiative. DOD has stated that its business operations reform efforts—which are not limited to the covered activities under section 921—will produce cost savings; however, DOD did not provide underlying documentation to allow us to independently validate the savings. Specifically, in its budget materials for fiscal year 2020, released in March 2019, DOD reported that its reform efforts had saved $4.7 billion in fiscal years 2017 and 2018, and are expected to save $6.0 billion in fiscal year 2019 and $7.7 billion in fiscal year 2020, the first year of required savings under section 921. Of those $7.7 billion in expected savings for fiscal year 2020, about $2.6 billion were in business process and systems improvements. According to OCMO and Office of the Under Secretary of Defense (OUSD) (Comptroller) officials, the OUSD (Comptroller) has validated these savings and the savings have been programmed or budgeted in the fiscal years reported. Specifically, according to OUSD (Comptroller) officials, all of the savings reported in DOD’s budget materials have been validated against OUSD Comptroller’s own systems that record budget information and decisions that are incorporated into DOD’s programming and budgeting process. OUSD (Comptroller) provided a spreadsheet detailing the various reforms and savings DOD cited in its budget materials, but did not provide the underlying support to allow us to independently validate the savings, such as documentation of budgetary decisions that reflect the savings. Our prior work over the past 7 years has found repeated shortcomings in DOD’s ability to demonstrate that it has achieved its goal for savings from reform efforts. Most recently, in September 2018, we reported that DOD could not demonstrate that it met several cost savings requirements mandated by the NDAA for Fiscal Year 2016, in part because there were no baseline costs established to measure any reductions against and documentation supporting cost savings estimates from other efficiencies was not detailed enough. DOD is taking steps to address this challenge and report on its cost baseline to perform all covered activities by January 1, 2020, as required by section 921. Specifically, in March 2019, we reported that OCMO is taking steps to establish cost baselines for DOD’s major lines of business through the fiscal year 2019–2020 timeframe. According to OCMO officials, they are also regularly adjusting the fiscal year 2019 baseline to reflect savings identified during the fiscal year. As of June 2019, OCMO is reviewing its approach for reporting the savings required by section 921 and plans to complete the review by October 2019. OCMO is coordinating with OUSD (Comptroller) on both establishment of the baseline and reporting of savings. While DOD has already funded some of the initiatives included in its plan through its annual budget request process, it continues to face challenges obtaining funding for others. According to DOD’s plan, four of the eight initiatives had no costs associated with them or the initiative has been funded to date using existing resources through the regular budget process, and DOD does not anticipate any additional costs for the initiatives. Funding needs for the remaining four initiatives have not been fully determined or met. Specifically: 1. Funding needs for the human resources service delivery initiative have not yet been determined. OCMO expects to fund the cost of this initiative as a part of the initial stand-up costs for OCMO’s Office of Fourth Estate Management in fiscal year 2020. OCMO officials told us they are reviewing baseline needs for the office and anticipate realigning resources to support the new office. 2. Funding needs for the human resources regulatory reform initiative have been determined, but OCMO has not confirmed that funding has been obtained. DOD’s plan states that future costs for the initiative may include approximately $500,000 for research and studies. To the extent possible, the plan states, DOD will use funds from the OUSD for Personnel and Readiness for studies, but DOD has not indicated that those funds have been obtained. 3. Funding needs for the strategic sourcing of sustainment and commodity procurement initiative have not been determined. According to OCMO, the Defense Logistics Agency and the military services are developing a detailed cost estimate for this initiative. However, neither the plan nor OCMO officials we spoke with identified where any funding that may be needed will come from once the costs are determined. 4. Funding needs for the plan’s category management initiative to conduct reviews of contracts and categories of goods and services have not been fully met. The initiative includes quarterly “sprints” reviewing different contracts or categories of goods or services to identify savings. According to DOD’s plan, each sprint is assisted by consulting firms and industry analyses and is estimated to cost about $11 million. DOD plans to complete a total of 10 sprints, at a total cost of $110 million. According to OCMO, limited funding has hindered execution of two of the sprints so far. OCMO has requested $12 million in its budget request for fiscal year 2020 to support this effort and expects the remaining sprints to be funded by savings identified through earlier sprints. However, in January 2019, we reported on problems associated with this approach. Specifically, we reported that OCMO officials told us the department initially planned to use available funding from OCMO or the savings generated by reform initiatives to fund development of other initiatives, but has since recognized that additional funding is needed. Among the key questions we previously identified for assessing agency reform efforts is the extent to which the agency has considered how the upfront costs of proposed reforms will be funded. In January 2019, we reported that some reform teams lacked resources to fully implement approved initiatives. We recommended, and DOD concurred, that DOD establish a process to identify and prioritize funding for implementing its cross-functional teams’ business reform initiatives. An OCMO official told us OCMO updated its reform management framework—the process it uses for managing its business reform efforts—in part to address this recommendation. However, in light of the continued challenges related to funding that we identified as part of this review, the effectiveness of changes to this framework at this time is unclear. As a result, we will continue to monitor the extent to which OCMO’s adjustments to its processes have addressed this recommendation as OCMO continues to implement its business reforms. We provided a draft of this report to DOD for review and comment. In response, DOD officials told us they concurred and had no comments on the report. We are sending copies of this report to the appropriate congressional committees and to the Secretary of Defense and Deputy Chief Management Officer. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this are listed in appendix III. Since we last reported on the Department of Defense’s (DOD) business reform efforts in January 2019, the Office of the Chief Management Officer (OCMO) has, among other things, changed the composition of the teams and the framework it is using to manage the efforts. Specifically, OCMO has disestablished the teams on real property management, human resources, and testing and evaluation, split the team on information technology and business systems into two separate teams for information technology and business systems, and made changes to the leadership or composition of each of the remaining teams. See table 1 for a summary of these changes. According to an OCMO official responsible for OCMO’s management of the reform efforts, OCMO has not removed any initiatives from the business reform efforts as a result of the changes to these teams, but some teams’ initiatives were absorbed into other business reform teams or organizations that OCMO believed were more appropriate for leading the initiatives, such as the relevant DOD principal staff assistant. For example, the category management team assumed responsibility for the real property management team’s initiatives. According to the same official, OCMO’s new Fourth Estate Management Office and components of the Office of the Under Secretary of Defense (OUSD) for Personnel and Readiness assumed responsibility for some of the human resources team’s initiatives. In addition, an OCMO official told us OCMO revised its business reform management framework—the process it uses for managing its business reform efforts. According to an overview of the new framework provided by OCMO, the new process is designed to establish a simplified, standardized, and repeatable process for managing these reforms and identifying and prioritizing funding for reform initiatives. An OCMO official told us that one of the goals of the updated process is to improve the uniformity of documentation across business reform teams and initiatives. That official told us the updated process also reduced the number of decision points—through which reform teams receive approval from DOD’s Reform Management Group to proceed with an initiative—from five to two. Further, OCMO introduced new processes for estimating and tracking the costs and potential savings resulting from reform initiatives. Among other things, the updated framework includes input from the OUSD (Comptroller). Specifically, according to OCMO documentation and OUSD (Comptroller) officials, OUSD (Comptroller) officials review estimates of the costs and potential savings recorded in OCMO’s reform management portal—a database OCMO uses to monitor business reform initiatives. OUSD (Comptroller) assigns a confidence score based on the degree to which each initiative has been developed. According to an OUSD (Comptroller) official, initiatives that are less developed will have a lower confidence score because they are further from full implementation and subject to more unknowns than those that are closer to implementation. OUSD (Comptroller) officials told us OUSD (Comptroller) uses confidence scores to adjust estimates of potential savings, and to lower potential savings associated with newer initiatives. According to OUSD (Comptroller) officials, these estimates of potential savings are not included in any savings amounts the department reports externally, such as in DOD budget materials, until they are actually programmed or budgeted. In addition to the contact named above, Margaret Best (Assistant Director), Daniel Ramsey (Analyst-in-Charge), Sierra Hicks, Alexa Kelly, and Richard Powelson made key contributions to this report. Other contributors included Bonnie Anderson, Tracy Barnes, Arkelga Braxton, Timothy J. DiNapoli, Michael Holland, Richard Larsen, Ned Malone, Ron Schwenn, Anne Stevens, John Van Schaik, and Sarah Veale.", "summary": "DOD spends billions of dollars each year to maintain key business operations intended to support the warfighter. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 established requirements for DOD to reform its enterprise business operations. Section 921 of the act required the Secretary of Defense, acting through the Chief Management Officer, to submit to the congressional defense committees by February 1, 2019, a plan, schedule, and cost estimate for reforms of DOD's enterprise business operations to increase effectiveness and efficiency of mission execution. Section 921 also requires GAO to provide a report assessing the feasibility of the plan. GAO's objectives were to assess (1) DOD's 921 plan, including its feasibility in reforming DOD's business operations, and (2) the extent to which DOD has made progress in implementing the plan and its broader reform efforts. GAO reviewed DOD's plan and associated documentation and interviewed DOD officials on efforts to reform business operations of the department, including the development and implementation of the plan. GAO also reviewed its past work on DOD reform efforts and the specific subject areas covered by DOD's reform initiatives. GAO has previously made eight recommendations related to DOD's reform initiatives from three prior reports. DOD concurred with those recommendations and is working to address them, in part through the initiatives GAO discusses. The Department of Defense's (DOD) April 2019 plan for business reform identifies eight initiatives related to civilian resources management, logistics management, services contracting, and real estate management. According to the plan, these initiatives will cost at least $116 million to implement through fiscal year 2021. GAO found that the plan generally contains the elements required under section 921—a schedule and cost estimate—and that several initiatives address aspects of GAO's prior recommendations. However, because many of the planned initiatives entail collecting information that will lay the groundwork for later reforms, assessing the feasibility of DOD's reform effort is difficult. For example, one logistics reform initiative plans to identify opportunities to improve processes, make recommendations, and develop an implementation plan for the recommendations by the end of fiscal year 2019. Although DOD officials told GAO that the department is making progress implementing the plan's initiatives and achieving cost savings on its broader efforts, DOD provided limited documentation of that progress. As a result, GAO could not independently assess and verify this progress. For example: Office of the Chief Management Officer (OCMO) officials provided briefing charts on the status of milestones for DOD's three human resource–related initiatives stating that those initiatives are progressing according to the schedule, but did not provide underlying documentation for each milestone. According to DOD, its broader reform efforts have saved or are expected to save about $18.4 billion between fiscal years 2017 and 2020. According to Under Secretary of Defense (Comptroller) officials, they have validated these savings. However, DOD did not provide any supporting documentation that would allow GAO to independently validate these savings. GAO's prior work has found repeated shortcomings in DOD's ability to demonstrate that it has achieved its goals for savings from reform efforts. DOD is taking steps to address these challenges, including establishing cost baselines for DOD's major lines of business and incorporating Comptroller input into estimates of the costs and potential savings from initiatives as they are developed. Further, according to the plan, DOD has provided funding through its annual budget process for four of the eight initiatives included in its plan. For the four remaining initiatives, OCMO has identified a source of funding but not obtained that funding for two initiatives, is awaiting a cost estimate for one initiative, and has identified only partial funding for one initiative, which is designed to review contracts and categories of goods or services on a quarterly basis to identify savings. OCMO anticipates that savings identified in earlier rounds of this initiative will fully fund later rounds. However, in January 2019, GAO reported that, according to OCMO, DOD initially planned to fund its reform initiatives in part with savings generated by other initiatives, but recognized that this approach did not work because additional funding was needed. GAO recommended that DOD establish a process to identify and prioritize funding for implementing its initiatives. OCMO has updated its processes for managing its reform efforts in part to address this issue, but the effects of this update at this time are unclear.", "document_type": "gao"}
{"report": "The federal government has long recognized the need to protect itself by ensuring contractors have appropriately allocated costs on cost-based contracts. In terms of what is potentially covered by CAS, cost-based contracts include cost-type contracts and certain fixed-price contracts where the contractor’s estimated or actual costs play a role in determining the amount the government pays. The total amount obligated annually by the government on these types of contracts is significant. For example, in fiscal year 2018, the federal government obligated approximately $172 billion on cost-type contracts alone, according to our analysis of Federal Procurement Data System information. In 1968, the House Banking and Currency Committee held hearings to determine whether to renew the Defense Production Act of 1950. A witness at the hearings, U.S. Navy Admiral Hyman G. Rickover, testified that defense suppliers could make excessive profits and disguise them as overhead costs or hide them in other ways in the absence of a set of uniform cost accounting standards. Witnesses at the time testified that it was difficult to compare costs among prospective contractors’ cost estimates or even to assess costs incurred on contracts with the same contractor without a set of uniform and consistent standards. Congress subsequently directed us to study the feasibility of establishing such standards. In January 1970, we reported one of many examples of mischarges involving a contractor that had charged the government for costs above the allowed cost ceiling by moving them under a separate contract cost category. We concluded that then-existing financial reporting standards were neither created nor adequate for contract cost purposes. In addition, we concluded that it was feasible to create a set of cost accounting standards and recommended doing so. In August 1970, Congress created the Board as an independent board within the legislative branch. The Board was initially chaired by the Comptroller General, who appointed four other members. The Board was authorized to promulgate standards designed to achieve uniformity and consistency in cost accounting practices used by federal contractors on defense contracts in excess of $100,000. The Board issued 19 cost accounting standards that went into effect between 1972 and 1980 for applicable DOD contracts. These standards covered areas such as consistency between how actual and estimated costs are calculated and reported, and ensuring that costs are not double- counted. The standards were intended to ensure that incurred costs were appropriately allocated to government contracts. In contrast, GAAP is a set of U.S. accounting standards, conventions, and rules focused on measuring companies’ financial performance. GAAP is meant to establish and improve financial accounting and reporting to provide useful information to investors and other users of financial reports, including measurement and recognition of costs in financial statements. Federal endorsement of generally accepted accounting practices or principles dates back to the Securities Act of 1933. Then, the Securities Exchange Act of 1934 created the Securities and Exchange Commission and gave it authority to oversee accounting and auditing methods for publicly traded companies. Subsequently, various professional accounting groups, with oversight by the Securities and Exchange Commission, began working to establish standards and practices for consistent and accurate financial reporting, which became known as GAAP. In 1973, the Securities and Exchange Commission recognized the Financial Accounting Standards Board (FASB) as the designated accounting standard setter for public companies in the United States, and FASB is responsible for GAAP. In fiscal year 1981, Congress stopped funding the Board. However, after a number of disputes arose as to how to interpret various standards, Congress reestablished the Board in 1988. Congress placed the Board under OFPP, which is part of the Office of Management and Budget (OMB) within the executive branch. Congress also broadened the Board’s authority by applying CAS to all federal contracts—they were previously applicable only to defense contracts. Table 1 provides more information on the differences between CAS and GAAP. The Board met intermittently to address issues associated with interpretations of the standards after it was reestablished in 1988. In the late 2000s, the Board revised two of the standards related to pension contributions by government contractors for their employees. Effective in 2008, Congress changed the minimum contributions required to fund pension plans. This change caused pension contributions to greatly exceed CAS pension costs reflected in contract prices. The Board updated the CAS effective in 2012 to harmonize CAS pension costs with statutory changes to the pension funding requirements. However, the Board’s changes did not address how costs were settled when pension plans were curtailed. In January 2013, we recommended that the Board set a schedule to revise parts of the CAS dealing with settlement of pension plan curtailments. Citing our recommendation, the Board began efforts to resolve this issue in July 2013 and the work is on-going. While Board staff have been working to resolve these pension issues, the Board went several years without holding official meetings of the full board. Figure 1 illustrates the Board’s activities over time. The current Board is comprised of five members. The Administrator of OFFP is a member and serves as Board Chair. The other members include representatives from DOD, the General Services Administration, industry, and another private sector representative with cost accounting expertise. According to OFPP officials, the Board is also assisted by two OFPP staff—one on a full-time basis and one on a part-time basis—and a detailee from DCAA. In addition, OFPP officials said that the Board forms interagency working groups to address specific issues, such as pension harmonization. The Board receives its funding from OMB and does not have a separate funding source. According to OFPP officials, the Board’s main expenses were salary reimbursement for the non-government employees who serve on the Board and publication costs for Federal Register notices. Other federal agencies also have responsibilities to help administer the standards. For example, according to OFPP officials, most CAS-covered contracts are defense related. As such, DCAA reviews federal contractors’ disclosure statements for adequacy and compliance—that is, whether the statements are current, accurate, and complete. Disclosure statements describe the company’s actual or proposed cost accounting practices, including how they distinguish between costs, and how costs are allocated to contracts. DCAA also conducts audits to ensure contractors comply with CAS and with the contractors’ disclosed and established cost accounting practices and procedures. In addition, DCMA monitors contractor performance and the contractor’s business management systems, among other things, to ensure that the contractor is consistently following its cost accounting practices for contracts that are subject to CAS. There is no definitive list of the companies, business segments or units, or contracts that are subject to CAS. Whether a contractor’s business segment is required to comply with the standards on a particular contract depends largely on the value of the government contracts it is awarded during the year that are cost-based. Once a contractor’s business segment exceeds a certain dollar threshold of these “CAS-covered contracts,” the business segment is required to comply with either (1) all 19 standards (termed “full CAS-coverage”) or (2) four standards (termed “modified CAS-coverage”). Full coverage applies to business segments with CAS-covered contracts with a combined value of $50 million or more. Modified coverage may apply to business segments with a single CAS- covered contract of $7.5 million or more, and combined CAS-covered contracts valued at less than $50 million. Table 2 below lists all 19 CAS required under full coverage and the four CAS required under modified coverage. We and congressionally established review panels have previously studied the potential impact of CAS on industry as well as possible changes to the CAS and the Board. For example: In April 1994, we reported that seven of eight companies we reviewed either kept their government contracting work separate from their commercial contracting or assigned additional staff to their government contracting segments due to the increased demands of government contracting, citing, among other things, CAS as a factor in that decision. In January 1997, we reported on DOD’s efforts to address acquisition cost drivers based, in part, on a prior DOD-directed study that identified CAS as one of the 10 largest cost drivers on DOD contracts. In that report, a DCMA official noted that, in his opinion, while the annual cost of maintaining a CAS-compliant system is relatively small, the cost to establish a CAS-compliant system may be significant. Congress asked us to lead a panel of experts to assess the future role of the Board. In April 1999, we issued the panel’s report focused on the Board and CAS in light of acquisition reforms and the evolution of GAAP. The panel concluded that, among other things, the Board should review CAS and its attendant requirements to determine whether standards could be streamlined to reduce unnecessary burden on affected contractors. In addition, the panel made several recommendations, including moving the Board out of OFPP to ensure autonomy. Congress did not act on this recommendation. The panel also recommended reviewing contract applicability and full-coverage thresholds for CAS. Congress subsequently set the modified coverage ceiling at $7.5 million in October 1999. In July 2017, most of the 12 companies we spoke with that had not done business with DOD told us they chose not to do so because it might trigger a large number of contract terms and conditions that would be expensive to implement. One reason provided by the companies for not competing for certain types of DOD contracts was the requirement to establish a government-unique cost accounting system and to disclose and follow cost accounting practices consistently. In June 2018, the Section 809 Panel—having been established to advise Congress on streamlining defense acquisition regulations— released the second of three volumes of its report. In its report, the panel made two recommendations related to CAS, which largely reiterated what the GAO-led panel recommended in 1999. In this regard, the Section 809 panel recommended that the Board should be relocated to the General Services Administration as an independent board with a budget sufficient to support at least three full-time, permanent staff. The panel also recommended raising CAS applicability threshold levels again to further reduce burden on contractors. Subsequently, in 2019, OMB submitted a legislative proposal on raising the CAS applicability threshold from $2 million to $15 million. OMB officials also indicated that they would continue analyzing the effects of additional threshold changes. Congress had not enacted the proposal into law at the time of this report. The CAS Board generally has complied or is in the process of complying with the administrative and reporting requirements prescribed by Section 820 of the National Defense Authorization Act for Fiscal Year 2017, including initial efforts to assess the extent to which CAS can be conformed with GAAP. To do so, the Board is taking steps to follow its statutorily prescribed four-step rulemaking process. The Board’s initial efforts focus on the extent to which two of the 19 standards might be modified or eliminated; however, Board members indicated that these efforts may take several more years to complete. The Board has generally complied with the administrative requirements prescribed under Section 820 thus far, including meeting regularly, generally publishing notices and agendas in advance of meetings, and reviewing disputes involving cost accounting-related matters. According to officials from the Office of Federal Procurement Policy, the Board is working on the first of its annual reports on its efforts, including those associated with efforts to conform the standards with GAAP where practicable. Table 3 highlights the steps the Board is taking to address some of the administrative and reporting requirements mandated by Section 820. The Board has also taken initial steps to address Section 820’s requirement that the Board review the standards and conform them to GAAP, where practicable (see table 4). In carrying out this work, the Board is taking steps to follow a statutorily prescribed four-step rulemaking process for promulgating CAS or interpretations. Figure 2 below outlines these requirements. In line with this process, between March and November 2018, the Board discussed the opportunities and methods available for conforming CAS to GAAP. The Board held informal discussions with its staff, industry representatives, and government agencies, such as DCAA. One of the messages coming from the feedback was for the Board to focus first on those standards that offered the greatest potential for change. By the end of 2018, the Board had completed development of the staff discussion paper. The Board expected to release this document in the Federal Register for public comment in January 2019; however, a partial shutdown of the federal government due to lapsed funding delayed its release until March 2019. The March 2019 staff discussion paper (1) outlined a set of five guiding principles that the Board would use to assess whether proposed CAS changes are necessary and whether those changes would reduce the burden on contractors while protecting the government’s interests, (2) identified a roadmap that prioritized the Board’s proposed review of standards, and 3) included a preliminary comparison of two standards to GAAP. Guiding Principles. The guiding principles outlined in the staff discussion paper describe the elements the Board will consider when determining whether changes to the CAS will reduce burden on contractors while continuing to protect the interests of the federal government. As stated in the staff discussion paper, the Board will: 1) reduce CAS requirements where practicable; 2) consider whether the proposed action would reduce burden on 3) consider whether other CAS or federal rules would protect the government’s interests in case of any gaps created by relying on GAAP; 4) monitor future changes to GAAP and the Federal Acquisition Regulation (FAR) to identify and evaluate their impact on CAS and revise CAS, as necessary; and 5) monitor future significant disputes related to the conformance to GAAP and evaluate whether the Board should address them through clarifying guidance or rulemaking. Prioritization. The Board grouped the 19 CAS into four categories based on the Board’s assessment of which standards are most likely to have overlap with GAAP (see figure 3). The Board plans to focus its initial efforts on the seven standards in the first group, which focus on cost measurement and assigning costs to accounting periods. According to its staff discussion paper, the Board’s proposed approach is to assess the standards by developing side-by-side comparisons of CAS requirements to corresponding GAAP requirements and identifying any gaps between the two. The Board will then evaluate the potential risk of any gaps identified, taking into account coverage by other CAS requirements and related regulations; for example, the FAR. The Board will also assess whether there is a history of compliance issues for those standards. According to OFPP officials, such assessments will help the Board determine whether they need to update guidance related to a particular CAS. Lastly, the Board plans to assess changes that have occurred in GAAP relative to CAS and to evaluate the need to conform CAS to the updated GAAP. For example, the Board has identified two recent changes in GAAP that it states may not align with CAS. Comparison. The Board has begun this effort by looking at two standards focused on measuring and assigning costs (CAS 408 and CAS 409), since it believes that GAAP potentially provided additional coverage compared to when the two CAS were established in 1975. OFPP officials stated that these two standards provided a good opportunity to modify and potentially eliminate duplicative coverage while testing the soundness of the Board’s approach to conform CAS to GAAP where practicable. The Board received seven separate comment letters on the staff discussion paper from five industry organizations, one commercial business, and one private individual. Our review of the comments found that they were largely supportive of the Board’s guiding principles, but some commenters raised concerns regarding the Board’s approach to its conformance effort and questioned whether it would ease the burden on contractors. For example, four respondents commented that the Board should not limit its focus to only revising or eliminating particular CAS when it was clear that GAAP provided adequate coverage. Instead, these industry groups stated that each CAS should be eliminated unless proven to be absolutely necessary due to the barriers to contractors that these groups believe the CAS create. The Board members we met with stated that all options for refining CAS requirements were on the table. However, they also stated that GAAP and CAS are focused on two separate goals—the former on a business’s high-level financial statement, the latter on individual contract costs. Board members, as well as DCAA and DCMA officials, noted that eliminating CAS requirements to rely purely on GAAP standards would limit the government’s ability to compare contract proposals, assess actual costs to avoid overcharges by contractors, and protect its interests. For example, DCMA officials stated that the government has $3.1 billion in pending litigation for identified CAS noncompliances. Recovery of increased costs is accomplished in part through contract clauses that entitle the government to recover specific cost increases on affected CAS-covered contracts. Were CAS and the associated contract clauses eliminated, DCAA and DCMA officials noted that the government’s ability to recover these costs would be greatly reduced. In addition, the Board is concerned that in modifying or perhaps even eliminating certain CAS requirements, and instead using GAAP, there is the risk that future GAAP changes would no longer cover the areas of CAS concern. This would leave the government vulnerable to the issues that the modified or eliminated CAS were originally created to address. Members of the Board and staff we spoke with indicated that the Board is reviewing and assessing the public comments on the staff discussion paper to determine whether the Board needs to make changes to the paper’s guiding principles or methodology going forward. According to the Board members, the Board will issue a Federal Register notice explaining any changes resulting from public input and its own additional deliberations. Additionally, they said the Board will consult with the Financial Accounting Standards Board—which is responsible for GAAP— to answer technical questions and ensure that the Board has an accurate understanding of GAAP coverage as they continue to perform side-by- side comparisons of CAS and GAAP. Further, the Board stated that it will publish a notice to address public comments on CAS to GAAP conformance projects and that additional staff discussion papers and associated notices will be published in the Federal Register for public comment as they are completed. In addition to streamlining or eliminating CAS standards, some of the comments in response to the staff discussion paper pointed to other areas that the Board may want to consider to reduce the burden on government contractors. For example, some comments encouraged the Board to consider increasing CAS full-compliance dollar thresholds. Reassessing the CAS full-compliance threshold aligns with findings from congressionally established panel reports from 1999 and 2018. According to both panels’ findings, increasing compliance thresholds is a way to decrease burden on many government contractors while still protecting the bulk of the government’s contracting dollars. As previously noted, OMB recently submitted a legislative proposal to raise the threshold from $2 million to $15 million. OMB also indicated that it intends to continue studying available data to understand the costs and benefits of CAS threshold changes and whether additional changes to the threshold need to be made. We provided a draft of this report to DOD, the Office of Management and Budget, and the Cost Accounting Standards Board for their review and comment. DOD had no comments on the report. The Office of Management and Budget and the Board provided technical comments, which we incorporated where appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense; the Director, Defense Procurement and Contracting; the Director, Defense Contract Audit Agency; the Director, Defense Contract Management Agency; the Director, Office of Management and Budget; and the Administrator, Office of Federal Procurement Policy. In addition, this report will be available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by e-mail at dinapolit@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Cost Accounting Standards Applicability. In general, a business segment is not subject to Cost Accounting Standards (CAS) until it receives a non-exempt contract of $7.5 million or more from the federal government. Generally, a non-exempt contract is a contract that does not meet any of the exemptions listed below. Typically, once a business segment receives a non-exempt contract of $7.5 million or more, all of its prospective non-exempt contracts or subcontracts over $2 million are considered CAS-covered. Summary of Exemptions. The following categories of contracts and subcontracts are exempt from all CAS requirements: Sealed bid contracts; Negotiated contracts and subcontracts not in excess of the Truth in Negotiations Act (TINA) threshold, as adjusted for inflation (41 U.S.C. 1908 and 41 U.S.C. 1502(b)(1)(B)). For purposes of this exemption, an order issued by one segment to another segment shall be treated as a subcontract; Contracts and subcontracts with small businesses (Federal Acquisition Regulation (FAR) Subpart 19.3 addresses determination of status as a small business.); Contracts and subcontracts with foreign governments or their agents or instrumentalities or, insofar as the requirements of CAS other than CAS 401 and CAS 402 are concerned, any contract or subcontract awarded to a foreign concern; A contract or subcontract where the price is set by law or regulation; A contract or subcontract authorized in FAR § 12.207 for the acquisition of a commercial item; A contract or subcontract with a value of less than $7,500,000 if, at the time of award, the business segment of the contractor or subcontractor that will perform the work has not been awarded at least one contract or subcontract with a value of $7,500,000 or more that is covered by the standards. Subcontracts under the North Atlantic Treaty Organization’s Patrol Missile Hydrofoil Ship programs to be performed outside of the United States by a foreign concern; A firm-fixed price contract or subcontract awarded on the basis of adequate price competition without submission of certified cost or pricing data. In addition, in cases where the prime contract is exempt from CAS under any of the exemptions at 48 C.F.R. § 9903.201-1 any subcontract under that prime is always exempt from CAS. Also, Title 41 of the U.S. Code was amended effective in 2018 to allow executive agency heads can waive CAS requirements for a contract or subcontract with a value of less than $100 million if the business segment is primarily engaged in commercial work and would not otherwise be subject to CAS, or for exceptional circumstances where waiving CAS is necessary to meet agency needs. Compliance. There are two levels of CAS coverage—full and modified. Full coverage applies to business segments with CAS-covered contracts valued at $50 million or more; those business segments must comply with all 19 standards. Modified coverage may apply to business segments with CAS-covered contracts valued less than $50 million. Business segments that have contracts awarded with modified coverage must comply with four of the standards. Business segments with full CAS-covered contracts are also required to submit disclosure statements describing the company’s actual or proposed cost accounting practices and procedures, including how they distinguish direct costs from indirect costs and the basis used for allocating indirect costs. The Defense Contract Audit Agency (DCAA) reviews disclosure statements for adequacy and compliance—that is, whether the statement is current, accurate, and complete—prior to contract award and during contract performance. DCAA may also complete CAS compliance audits at the request of the cognizant federal agency official after contract award. In some circumstances, the Defense Contract Management Agency (DCMA) will review disclosure statements that are not audited by DCAA. According to officials, both DCAA and DCMA provide audit findings to the cognizant federal agency official, who then disposes the audit findings by making the final determination of adequacy and compliance. The purpose of disclosure statement audits is to determine whether the contractor’s disclosed or established practices are in compliance with CAS rules, regulations, and standards, as well as appropriate acquisition regulations. A CAS-related noncompliance may be found if a contractor with a CAS-covered contract proposes a practice that will violate CAS or a government acquisition regulations cost principle, or if the contractor’s actual practices are either inconsistent with their own disclosure statement or noncompliant with the cost standards or principles. For example, in 1970, we, along with DCAA auditors, found instances where contractors charged costs as both direct and indirect costs to the same contract, resulting in the contractors recovering the same charge twice. If an auditor discovers a noncompliance issue, the auditor will submit an advisory report to the cognizant federal agency official who makes the final determination. The consequences of a CAS noncompliance can range from a contract adjustment to litigation. According to the DCMA’s Contract Dispute Resolution Center, there were 15 judicial decisions issued in CAS-related board and court cases in the last five years. In addition to the contact named above, Bruce H. Thomas, Assistant Director; Peter Anderson; Jennifer Baker; Miranda Riemer; Jenny Shinn; Ryan Stott; and Roxanna T. Sun made key contributions to this report.", "summary": "Each year, the federal government obligates billions of dollars on contracts for which the final costs depend, in part, on the amount of overhead and other costs charged to the contract. Congress created the Board in 1970. The standards it created ensure contractors appropriately charge costs to government contracts. In contrast, GAAP is a set of financial reporting principles that commercial firms may use in preparing financial statements and which include the basis for recognizing and measuring costs in such statements . Industry representatives and others have raised concerns that complying with CAS may be burdensome and questioned whether the government could rely on GAAP. In 2016, Congress included a provision in law that the Board, among other things, conform CAS with GAAP, where practicable. Congress also included a provision for GAO to assess Board efforts. This report assesses the extent to which the Board is taking steps to meet legislative requirements and describes the Board's efforts to conform CAS to GAAP. GAO reviewed applicable laws, regulations and guidance, Federal Register notices and other documentation on the Board's activities. GAO also examined the Board's methodology for comparing CAS to GAAP and its preliminary analysis of two of the cost accounting standards. Finally, GAO interviewed Board members and federal procurement officials. The Cost Accounting Standards Board (the Board) is generally meeting recent legislative requirements and has taken initial steps to assess the extent to which the government's Cost Accounting Standards (CAS) can be conformed with a set of 12commercial financial reporting principles known as Generally Accepted Accounting Principles (GAAP). Comprising five members representing the government and industry, the Board issued 19 standards between 1972 and 1980. After that point, the Board met intermittently until 2016. At that time, Congress included a provision in the National Defense Authorization Act for Fiscal Year 2017 to require the Board to meet quarterly, to review CAS-related disputes, to conform CAS with GAAP where practicable, and to report annually to Congress on its efforts, among other things. Since the legislation went into effect, the Board has met regularly, has been briefed on CAS-related disputes, and is preparing its initial report to Congress. The Board has also taken initial steps to assess the extent to which CAS can be conformed with GAAP. The Board summarized its approach in a March 2019 staff discussion paper, which it released for public comment. In it, the Board: outlined a set of five guiding principles to assess whether proposed CAS changes are necessary and whether those changes would reduce the burden on contractors while protecting the government's interests, identified a roadmap that prioritized the Board's proposed review of the standards, and included a preliminary comparison of two of the seven standards identified as having the most overlap with GAAP (see figure). Some comments submitted in response to the discussion paper by industry groups stated that each of the 19 CAS should be eliminated unless proven to be absolutely necessary. Board members told GAO they were considering all options for refining CAS but noted that GAAP and CAS are focused on two separate goals—GAAP on businesses' high-level financial performance, CAS on allocating costs to individual government contracts. The Board and other government officials said that eliminating CAS requirements to rely purely on GAAP would limit the government's ability to protect its interests.", "document_type": "gao"}
{"report": "Selected agencies’ funding for university STEM research. The five federal agencies included in our preliminary analysis provide billions of dollars annually for university research in STEM fields, with NIH providing more than the other four agencies combined. Table 1 details the total amount of research funding provided to universities by each agency in fiscal year 2017. Sexual harassment. As defined in the National Academies of Sciences, Engineering, and Medicine (NASEM) 2018 report, sexual harassment encompasses three types of behavior: Sexual coercion: Favorable treatment conditioned on sexual activity. Unwanted sexual attention: Verbal or physical unwelcome sexual advances, which can include assault. Gender harassment: Sexist hostility and crude behavior. The most common form of sexual harassment is gender harassment, which generally involves hostility, exclusion, or other discrimination based on a person’s gender. The 2018 report found that sexual harassment in academia is significantly more common among female students in engineering and medical majors than in non-STEM fields. According to the report, at least five factors create the conditions under which sexual harassment is likely to occur in STEM programs and departments in academia: Perceived tolerance for sexual harassment Environments where men outnumber women and leadership is male Environments in which the power structure of an organization is hierarchical with strong dependencies on those at higher levels or in which people are geographically isolated Increased focus on symbolic compliance with Title IX Uninformed leadership on campus Title IX of the Education Amendments of 1972. Title IX of the Education Amendments of 1972 is the primary federal law that addresses sex discrimination in all federally funded grant programs at educational institutions. Under Title IX, federal agencies that award grants to educational institutions have enforcement responsibilities to ensure such institutions do not discriminate based on sex. Enforcement responsibilities include issuing regulations, conducting periodic compliance reviews of funding recipients, and investigating timely written complaints of sex discrimination against recipients. DOJ and the Department of Education have other responsibilities for administering Title IX. DOJ is designated by Executive Order No. 12250 to coordinate Title IX compliance across federal agencies, including information sharing. Federal grant awards and grant life cycle. In general, federal agencies administer grants through a common administrative life cycle: pre-award, award, implementation, and closeout. During the pre-award stage, most of the agencies we reviewed require grantees to submit an “assurance of compliance” form as part of their grant application to attest compliance with anti-discrimination laws, including Title IX. For the award stage, the federal agency awarding the grant enters into an agreement with grantees stipulating the terms and conditions for the use of grant funds. During the implementation stage, among other things, the federal agency manages and oversees the grant, including any Title IX compliance reviews. A Title IX compliance review is an agency’s assessment of whether a grantee is complying with the law. Federal agencies may conduct these reviews onsite at an institution (grantee) or via a desk audit. In the closeout stage, the awarding federal agency and grantee bring the grant to its conclusion, once all the work associated with the grant agreement is complete, the grant end date has arrived, or both. Among the federal agencies we reviewed, different offices handle various aspects of grant compliance. Generally, each agency’s civil rights or diversity office conducts Title IX compliance reviews, develops policies and procedures for grantees, and investigates allegations and complaints involving university researchers supported by their agency’s federal STEM grants. The office that awards grants generally creates and modifies grant terms and conditions. Our preliminary analysis indicates that the selected federal agencies’ staff and budget available to address sexual harassment complaints from individuals at grantee universities varies according to the duties and funding for the primary agency offices responsible for addressing the complaints, as well as with the number of complaints received from grantees. Duties and funding for offices responsible for addressing complaints. Our preliminary analysis shows that all five agencies (DOE, HHS, NASA, NSF, and USDA-NIFA) primarily address sexual harassment complaints through their civil rights or diversity offices. However, these offices are responsible for more than just addressing complaints and preventing sexual harassment at grantee universities; they also oversee a number of civil rights, diversity and inclusion efforts for the entire agency. Moreover, most of these offices also address internal employee sexual harassment complaints and other discrimination issues. For example, HHS officials described how staff in their Office for Civil Rights at headquarters and eight regional offices conduct compliance reviews and investigate all complaints alleging sexual harassment and other forms of discrimination against recipients of HHS federal financial assistance, including recipients of NIH grants. USDA- NIFA said their civil rights and diversity office staff are not always available when sexual harassment issues arise because they have other duties and also cover other discrimination issues. In addition, some agencies noted challenges in ensuring adequate staffing levels. For example, USDA-NIFA officials cited the need to fill vacant positions in their civil rights office, and NSF officials described a need to find staff with expertise in this complicated, specialized area. All five agencies fund their civil rights and diversity offices separately from their STEM research funding, and there is little relationship between the two budgets. For more information on selected agencies’ civil rights and diversity office staffing and budgets planned for fiscal year 2019, see table 2. Number of complaints received. Our preliminary analysis of sexual harassment complaint information indicates that four of the five selected agencies received three or fewer complaints from individuals at grantee universities from fiscal year 2015 through 2019. See table 3. Officials from DOE told us that because they receive so few sexual harassment complaints from individuals at grantee universities, they have enough resources to address those that are reported to their civil rights or diversity offices. In addition, officials from HHS told us that because they receive so few complaints, their civil rights office has used other oversight mechanisms, like Title IX compliance reviews, to examine whether sexual harassment is occurring at universities receiving HHS funds, including funds from NIH. However, as agencies continue to strengthen grantee policies or requirements, it may affect the number of complaints an agency receives from individuals at grantee universities, as well as the amount of resources an agency needs to address them. For example, NSF officials described how the number of sexual harassment complaints they receive has increased since the agency implemented new grant terms and conditions that require university grantees to report any sexual harassment findings involving a Principal Investigator or co-Principal Investigator for NSF-funded research. NSF officials also described an increased number of questions and calls about how to report incidents, requests for training and presentations, and meetings with program officers, awardee representatives and other stakeholders, among other items. Based on our preliminary review, all five of the selected agencies have established and communicated their own sexual harassment prevention policies to grantees within the last 2 fiscal years, but agency communication mechanisms and the content of these grantee policies vary. Specifically, our preliminary analysis shows that NASA, NIH, and NSF communicate their policies on sexual harassment in multiple forms, such as grantee policy manuals, best practices documents, and online FAQs. The result is that grantees receive a relatively high level of detail about preventing sexual harassment and mechanisms for reporting complaints. In contrast, Cabinet agencies DOE and USDA-NIFA provide fewer forms of guidance, either through their website or agency director and Secretary-level policy statements and documents, which focus more generally on the broader category of sex discrimination or provide different levels of information on sexual harassment prevention policies for grantees. See table 4 for more information. Regarding the content of the policies, our preliminary analysis shows that DOE, NIH, NSF, and USDA-NIFA updated their definitions of behaviors or actions that qualify as sexual harassment in their grantee policies and procedures, and NASA is in the process of doing so. The definitions are more specific than previous definitions; for example, they include descriptions of gender harassment, the most common form of sexual harassment. The increased specificity may make clear the behaviors or actions grantees are expected to address in their efforts to prevent sexual harassment. The agencies continue to develop and revise policies and communication mechanisms for grantees. Also, NSF and NASA have modified, or are taking steps to modify their grant terms and conditions to strengthen requirements for university grantees to report on findings of sexual harassment to the funding agency. Officials from both agencies told us these modifications will help hold grantees accountable for reporting sexual harassment; the NSF Director states on the agency’s website that these changes are “intended to provide targeted, serious consequences for harassers” while also providing “tools to make harassment stop without disturbing others’ careers and lives.” The requirement also supports the NASEM 2018 report recommendation for institutions to be transparent about reporting sexual harassment findings, which is intended to foster a culture and climate that does not tolerate sexual harassment at universities. Officials from cabinet agencies DOE, NIH (a component of HHS), and USDA-NIFA stated they would need to go through formal rulemaking processes to alter their grant terms and conditions in a similar manner. In addition, our preliminary analysis shows that two of the five selected agencies are taking steps to evaluate the effectiveness of their sexual harassment prevention policies and procedures for grantees. NSF officials told us that they are in the process of determining how best to evaluate the effectiveness of the new sexual harassment reporting requirements and how the requirements have affected grantees. Similarly, DOE is in the process of comparing its policies and procedures against other federal agencies’, according to officials. To date, the other three agencies have not yet evaluated the effectiveness of their policies, but officials at these agencies told us that they focus on ensuring grantees comply with Title IX regulations as a way to measure the effect of their policies. For example, NASA established a goal to promote compliance and encourage best practices among grantees, and the agency measures progress towards this goal through verifying grantee compliance with Title IX. USDA-NIFA officials are also in the process of creating a tool to provide a comprehensive blueprint for civil rights compliance—including Title IX compliance—and are planning to implement the tool in fiscal year 2020. We will continue to examine and assess the selected agencies’ sexual harassment prevention policies for university grantees and steps they are taking to evaluate them in our ongoing work. Based on our preliminary review, all five selected agencies have taken some steps to promote information sharing and collaboration among agencies on Title IX compliance reviews through DOJ’s Title IX STEM working group. According to officials, the group discusses strategies for conducting joint Title IX compliance reviews to leverage limited agency resources and share best practices. For example, DOE and NSF have conducted three joint compliance reviews, and NASA and NSF told us that they are in the process of conducting a joint review. These joint reviews can be helpful, as the selected agencies conduct a small number of compliance reviews (two to four) annually relative to the number of university grantees who must comply with Title IX. Despite this collaboration, all five selected agencies reported challenges in obtaining and sharing information. For example, all five selected agencies told us they rarely discuss sexual harassment cases at DOJ’s Title IX STEM working group meetings unless they are directly related to an ongoing or planned compliance review. In addition, DOE, NASA, NIH, and NSF stated they rarely learn about instances of sexual harassment from voluntary reporting from universities or other federal agencies and instead must rely on other sources, such as news reports. This situation may change at NSF and NASA, which have taken steps to modify their grant terms and conditions to require reporting of sexual harassment findings by grantees. Challenges in obtaining and sharing information on sexual harassment cases may increase the risk of a situation known as “pass the harasser,” in which a researcher with substantiated findings of sexual harassment obtains employment at another university or grants from another funding agency without the university or funding agency being aware of the researcher’s history. Officials from all five selected agencies noted a willingness to participate in an interagency working group to address the culture of sexual harassment in STEM research that moves beyond conducting Title IX compliance reviews. The White House’s Office of Science and Technology Policy (OSTP) has taken steps to establish an interagency working group. In May 2019, OSTP established a joint committee under the National Science and Technology Council to address challenges in the research environment. OSTP, NIH, NSF, DOE, and the National Institute of Standards and Technology Directors were selected as joint committee chairs to engage with the academic and science community for policymaking insight and to convene interagency efforts. According to DOE officials, the committee will address several priorities, including the development of policies and procedures across the federal government regarding sexual harassment in the research environment. Three of the five selected agencies (NSF, NASA, and DOE) stated OSTP would be the appropriate entity to establish uniform sexual harassment policy guidelines to help provide consistency across the federal government. NSF and NASA officials suggested that DOJ or the Department of Education would be the appropriate agencies to collaborate with OSTP on the ongoing monitoring of sexual harassment policy guidelines. All five selected agencies reported taking collaborative steps with universities and federal agencies to address the culture and climate for women in STEM. For example, in 2019, NIH established a working group with university experts to collaborate with other federal agencies to assess the current state of sexual harassment allegation investigation, reporting, remediation, and disciplinary procedures at NIH-funded organizations and advise on oversight, accountability, and reporting measures for grantees, among other things. In addition, all five agencies provided examples of collaborative efforts that would help address the culture of sexual harassment in STEM research. For example, NASA officials told us that it would be helpful to conduct joint meetings with other university grantees across agencies to discuss sexual harassment in science. Lastly, efforts to improve information sharing and collaboration across agencies beyond conducting Title IX compliance reviews are consistent with findings in the 2018 NASEM report, which states, “adherence to legal requirements is necessary but not sufficient to drive the change needed to address sexual harassment.” We will continue to examine and assess selected agencies’ Title IX reviews and efforts to collaborate and share information in our ongoing work. In closing, I note that we are continuing our ongoing work on this topic. Sexual harassment is not only degrading to individual researchers, it undermines the quality and fairness of our nation’s research enterprise. It is therefore important that federal agencies ensure their grantees effectively prevent and address sexual harassment in STEM research. We look forward to continuing our work to determine whether additional federal actions may be warranted to promote this objective. Chairwoman Johnson, Ranking Member Lucas, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact John Neumann, Managing Director, Science, Technology Assessment, and Analytics, at (202) 512-6888 or NeumannJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this testimony include Rob Marek (Assistant Director), Michelle St. Pierre (Assistant Director), Kristy Kennedy (Analyst-In-Charge), Nora Adkins, Caitlin Cusati, Nkenge Gibson, Amanda Postiglione, Janay Sam, and Ben Shouse. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In fiscal year 2017, U.S. universities were awarded over $15 billion in federal grant funding for STEM research. Federal agencies are required to enforce Title IX—a law prohibiting discrimination on the basis of sex in education programs receiving federal financial assistance—including at universities they fund. Sexual harassment is not only degrading and illegal, it has a negative effect on the ability of women to engage in research at the same level as men. GAO was asked to review federal efforts to help prevent sexual harassment by STEM research grantees. This testimony is based on ongoing GAO work and provides preliminary observations on selected agencies: (1) availability of staff and budget to address sexual harassment complaints at universities they fund for STEM research; (2) efforts to establish and communicate policies and procedures for university grantees on preventing sexual harassment; and (3) steps taken to promote information sharing and collaboration among agencies to prevent sexual harassment at universities they fund for STEM research. GAO selected five federal agencies that together funded approximately 80 percent of STEM research from fiscal year 2015 through 2017, the latest data available. GAO reviewed these agencies' relevant regulations and documentation. GAO also interviewed agency officials as part of GAO's ongoing work. Based on preliminary information, the availability of agency staff and budget varies across the five selected agencies for efforts to address sexual harassment complaints at universities that use federal funds for Science, Technology, Engineering, and Mathematics (STEM) research. While four of the five agencies received three or fewer sexual harassment complaints from individuals at grantee universities from 2015 through 2019, changes to agency grantee policies or requirements could impact the number of complaints an agency receives and the amount of resources an agency needs to address them. The five selected agencies have established and communicated sexual harassment prevention policies to university grantees to varying degrees. Agencies vary in how they have: Provided detailed policies to grantees on sexual harassment. Three agencies—the National Aeronautics and Space Administration (NASA), Health and Human Services (HHS) National Institutes of Health (NIH), and the National Science Foundation (NSF)—have communicated relatively detailed policies on sexual harassment by issuing multiple forms of guidance, such as grantee policy manuals and best practices documents. In contrast, the Department of Energy (DOE) and Department of Agriculture (USDA) National Institute of Food and Agriculture (NIFA) communicated through more general documents, including policy statements that do not specifically address grantees. Modified grant terms and conditions . Two agencies are modifying the terms and conditions of grants to require grantees to report sexual harassment. NSF now requires grantees to increase transparency by reporting findings of sexual harassment to NSF, and NASA plans to implement the same requirement. Evaluated effectiveness of grantee policies. To date, the five agencies have not evaluated the effectiveness of their grantee policies and procedures to prevent sexual harassment, although two agencies are in the process of planning such evaluations. Based on our preliminary analysis and interviews, all five selected agencies have taken some steps to promote information sharing and collaboration among agencies on the prevention of sexual harassment. But they also noted challenges to these efforts, such as the lack of information on sexual harassment cases. These challenges may increase the risk that universities or agencies are unknowingly funding researchers with a history of past sexual harassment findings. The White House's Office of Science and Technology Policy has taken steps to create an interagency working group by establishing a joint committee in May 2019 under the National Science and Technology Council with NIH, NSF, DOE, and the National Institute of Standards and Technology Directors. The committee plans to address challenges in the research environment, including the lack of uniform federal sexual harassment policies. GAO is not making recommendations at this time but will consider making them, as appropriate, as it finalizes its work.", "document_type": "gao"}
{"report": "GSA’s existing government-wide telecommunications program is called Networx. As part of this program, in 2007 GSA awarded two sets of Networx contracts, which had an estimated combined value of $20 billion. These sets of contracts had differing characteristics: GSA awarded Networx Universal contracts to AT&T, Verizon Business Services, and Qwest Government Services. Networx Universal offers voice and data services, wireless services, and management and application services, including video and audio conferencing, as well as mobile and fixed satellite services, with national and international coverage. Networx Universal contracts were set to expire in March 2017; however, GSA has twice extended these contracts. According to GSA officials, the most recent extension, which GSA announced in November 2018, is to include one base year and two 1-year options, plus an additional option for the number of months required for the contracts to reach May 31, 2023. If the extension is executed and all options are exercised, the contracts will expire in May 2023. GSA awarded Networx Enterprise contracts to AT&T, Verizon Business Services, Qwest Government Services, Level 3 Communications, and Sprint Nextel. Networx Enterprise offers services similar to those of Networx Universal, with a focus on those that are internet-based. Networx Enterprise requires telecommunications services to be available in a smaller geographic area than Networx Universal. Networx Enterprise contracts were set to expire in May 2017; however, GSA has twice extended these contracts to each participating vendor, except one. According to GSA officials, the most recent extension, which GSA announced in November 2018, is to include one base year and two 1-year options, plus an additional option for the number of months required for the contracts to reach May 31, 2023. If the extension is executed and all options are exercised, the contracts will expire in May 2023. In addition, GSA provides telecommunications services through programs called Washington Interagency Telecommunications System 3 and Regional Local Service Agreements. Washington Interagency Telecommunications System 3: these contracts support a variety of telecommunications services available to all federal agencies in Washington, D.C., and surrounding Maryland and Virginia counties. For example, among other things, these contracts provide data and voice services, as well as cloud services. These contracts were set to expire on or before May 2020. As of December 2019, GSA planned to extend these contracts. GSA officials stated that the extension is to include one base year and two 1-year options, plus an additional option for the number of months required for the contracts to reach May 31, 2023. If the extension is executed and all options are exercised, the contracts will expire in May 2023. Regional Local Service Agreements: these contracts provide local telecommunications services in every state and major city in the United States. According to GSA officials, the expiration dates for these contracts ranged from October 2019 through March 2023. As of December 2019, GSA was in the process of extending these contracts. In particular, GSA officials reported that certain contracts had already been extended to May 2023, and the officials planned to extend the remaining contracts through May 2023, as well. According to data provided by GSA officials, in fiscal year 2019, federal agencies spent approximately $2.5 billion on services acquired through Networx, Washington Interagency Telecommunications System 3, and Regional Local Service Agreements contracts. About $2 billion of this spending was on services acquired through Networx alone. EIS is the replacement for Networx, Washington Interagency Telecommunications System 3, and Regional Local Service Agreements telecommunications contracts. GSA intends for EIS to address federal agencies’ global telecommunications and IT infrastructure requirements. GSA plans for EIS to provide agencies with traditional and emerging services to meet current and future requirements by: simplifying the government’s process of acquiring IT and telecommunications products and services; providing cost savings to each agency through aggregated volume buying and pricing (with generally lower costs for services on EIS compared to the costs for similar services on Networx), and spending visibility; enabling the procurement of integrated solutions; promoting participation by small businesses and fostering competition; offering a flexible and agile suite of services supporting a range of government purchasing patterns into the future; and providing updated and expanded security services to meet current and future government cybersecurity requirements. In addition, GSA has identified several benefits that EIS is expected to provide to the agencies that participate in its telecommunications programs. These projected benefits include streamlined contract administration, a possible 15-year period of performance, simplified pricing, and enhanced management and operations support. On August 1, 2017, GSA announced that it had awarded EIS contracts to 10 vendors. These contracts have a combined value of up to $50 billion and are for a possible period of up to 15 years (one 5-year base period and two 5-year option periods). According to GSA’s plans as of November 2019, the transition to EIS is expected to be completed by May 2023, when the current Networx, Washington Interagency Telecommunications System 3, and Regional Local Service Agreements telecommunications contracts are expected to expire (if all contract options are exercised, as discussed earlier). To help ensure that agencies’ services are fully transitioned to EIS before the current contracts expire, GSA issued guidance that identified several critical milestones that agencies should meet. These milestones include: (1) releasing all planned fair opportunity solicitations to EIS vendors by March 31, 2019; (2) issuing all planned task orders by September 30, 2019; and (3) achieving 100 percent transition of services by September 30, 2022. Figure 1 provides a timeline of the planned transition to EIS, including GSA’s critical milestones, as of November 2019. Central to the successful transition from GSA’s current telecommunications services contracts to EIS are transition planning and execution activities that involve GSA, federal agencies, the incumbent telecommunications contractors, and EIS contractors. GSA serves as the facilitator for all transition management activities. The agency is using contractors to assist in tracking transition activities, in order to avoid delays and other problems that can arise throughout the process. In particular, GSA’s primary responsibility is to provide program management for the current telecommunications programs (Networx, Washington Interagency Telecommunications System 3, and Regional Local Service Agreements) and EIS. As part of this, GSA is responsible for conducting government-wide strategy and project management; providing tailored assistance to agencies for transition planning and help with contractor selection and ordering; tracking and reporting the use of metrics that convey the relative complexity and transition progress; and providing customer support, training, and self-help tools and templates. GSA developed two contracting vehicles to provide transition assistance to agencies: (1) a Transition Coordination Center vehicle that includes assistance with inventory validation, transition planning, and solicitation development; and (2) a Transition Ordering Assistance vehicle that addresses tasks including requirements development and source selection assistance, and proposal evaluation. The Coordination Center vehicle was put in place in January 2016 and the Ordering Assistance vehicle was initially awarded in September 2016, but was not finalized until March 2017, after the conclusion of a bid protest. Agencies have principal responsibility for the transition. They are responsible for coordinating transition efforts with the incumbent contractors and EIS contractors to ensure that existing telecommunications services are disconnected and that new services are ordered under EIS. According to GSA, agencies’ responsibilities under EIS include: identifying key personnel, chiefly a Senior Transition Sponsor, Lead Transition Manager, and Transition Ordering Contracting Officer; engaging expertise from Chief Information Officers, Chief Acquisition Officers, and Chief Financial Officers to build an integrated transition team of telecommunications managers, acquisition experts, and financial staff; developing a financial strategy and budget for transition costs beginning in fiscal year 2017; analyzing and confirming the accuracy of the inventory of active services that must be transitioned; developing a transition plan that describes technological goals, a transition schedule that includes GSA’s major transition milestones (e.g., releasing all fair opportunity solicitations by March 31, 2019, and issuing all task orders by September 30, 2019), a strategy for issuing task orders on EIS for transitioning services, and any constraints or risks; preparing solicitations for task orders; placing task and service orders; coordinating resources to facilitate scheduling and communications for implementing and maintaining services; and reviewing, accepting or rejecting, and paying for services. At the agencies we reviewed, the staff responsible for the transition were part of their agencies’ offices that were headed by the Chief Information Officers. Finally, the incumbent and EIS contractors are responsible for disconnecting existing services under the current contracts and installing new services that agencies order under EIS. They are also to collaborate with GSA and agencies to share transition planning and execution best practices and help resolve issues. We have previously reported on efforts by GSA and agencies to transition from one telecommunications program to another. In a June 2006 report, we identified a range of transition planning practices that can help agencies reduce the risk of experiencing adverse effects of moving from one broad telecommunications contract to another. These planning practices were to: (1) develop an accurate inventory of telecommunications assets and services, (2) perform a strategic analysis of telecommunications requirements, (3) develop a structured transition management approach, (4) identify the resources needed for the transition, and (5) develop a transition plan. In that report, we also noted the progress of six selected agencies in preparing for the transition to Networx and found that the agencies generally had not implemented the practices, but were planning to do so. We recommended, among other things, that two of the agencies take actions to address gaps in their transition planning efforts. Both agencies agreed with the recommendations and implemented them. In addition, in 2008, we reported on the extent to which six selected agencies were following the transition planning practices during the Networx transition. We noted that the agencies were generally implementing the practices, but three of them had not fully implemented some of the key activities of the practices and were not planning to do so. For example, one agency was not planning to clearly define all key transition roles and responsibilities and another agency was not planning to identify local and regional points of contact. We made recommendations focused on addressing the gaps in transition planning to the three agencies that had not implemented key practice activities and did not plan to do so. One of the three agencies agreed with the recommendations and two agencies partially agreed with them. One agency implemented the recommendation we made to it, one implemented one of the two recommendations directed to it, and one agency implemented one of the seven recommendations we made to it. In 2013, we reported on factors that had contributed to the delay in the Networx transition and the consequences of the delay. We pointed out that weak project planning and complex acquisition processes were factors that had contributed to the delay. As a result, we recommended, among other things, that GSA take two actions to improve planning and execution of the next telecommunications transition. GSA agreed with these recommendations. The agency then implemented one of the recommendations and did not implement the other one, which was to examine, in coordination with the Office of Personnel Management, potential government-wide telecommunications expertise shortfalls and use the study to shape the next telecommunications acquisition (now called EIS). More recently, we reported in 2017 that, among other things, the five agencies we selected had yet to fully apply most of the five planning practices. Specifically, we noted that one agency fully implemented one practice, partially implemented three practices, and did not implement another. The other four agencies partially implemented each of the five practices. Accordingly, we recommended, among other things, that the five agencies complete adoption of the planning practices to avoid schedule delays and unnecessary costs. Four of the five agencies agreed with all of our recommendations. The other agency agreed with two recommendations, partially disagreed with one, and disagreed with two recommendations. All five agencies have efforts underway to address our recommendations, but had not yet fully implemented them as of November 2019. The 19 selected agencies have varied plans for transitioning from their current telecommunications contracts to EIS program contracts. As of October 2019, these agencies were also in different stages of their EIS transitions. All of the selected agencies reported that they plan to fully transition their telecommunications services to EIS before the current contracts are set to expire in May 2023. However, over half of the selected agencies did not plan to complete the transition by GSA’s September 30, 2022, milestone. In addition, the majority of selected agencies did not meet GSA’s two critical EIS transition milestones in 2019—to (1) release all fair opportunity solicitations by March 31, 2019, and (2) issue all task orders by September 30, 2019. The 19 selected agencies had various plans for completing their transitions to EIS. In particular, eight of the selected agencies reported that they planned to finish their transitions to EIS by GSA’s September 30, 2022, milestone. The 11 remaining agencies did not plan to complete their transitions by that date. Table 1 identifies the 19 selected agencies’ plans for completing the transition to EIS by GSA’s September 30, 2022, milestone. Officials from the 11 selected agencies that did not plan to finish their transitions to EIS by GSA’s September 30, 2022, milestone—the Departments of Agriculture, Commerce, Energy, HHS, Homeland Security, the Interior, Justice, Transportation, the Treasury, and VA; and the Social Security Administration—reported that they planned to complete the transitions before the current telecommunications contracts are set to expire in May 2023. Specifically, Commerce and the Social Security Administration planned to complete their transitions in December 2022; the Department of Transportation planned to do so in January 2023; the Departments of Agriculture, HHS, Homeland Security, and the Treasury planned to complete their transitions in March 2023; and the Departments of Energy, the Interior, Justice, and VA planned to complete their transitions in May 2023, just before the current telecommunications contracts are set to expire. In addition, the planned scope and amount of effort that is expected to be required to fully transition to EIS varied among the selected agencies. Specifically, agencies varied in the scope of their planned efforts related to two of GSA’s critical transition milestones—to release EIS fair opportunity solicitations and issue EIS task orders. Specifically, Eighteen of the selected agencies planned to release between one and six EIS fair opportunity solicitations, and the final agency—the Department of Defense—planned to release 54 solicitations. Thirteen of the agencies planned to issue between one and five EIS task orders, while the remaining six agencies—the Departments of Defense, Homeland Security, Labor, the Treasury, and VA; and NASA—planned to issue more than five task orders. Table 2 identifies the estimated number of planned EIS fair opportunity solicitations and task orders for the 19 selected agencies, as of November 2019. Further, the selected agencies had different plans for the types of transitions that they would implement. Specifically, as of November 2019, four of the selected agencies planned to implement primarily a like-for-like transition of their services. The remaining 15 agencies planned to conduct a combination of a like-for-like transition and upgrading or transforming services. Table 3 identifies the 19 selected agencies’ plans for the types of transitions to EIS that they will implement, as of November 2019. As of October 2019, the 19 selected agencies were in different stages of their EIS transitions. Eighteen of the agencies were in the acquisition planning and/or acquisition decision phases, during which the agencies release fair opportunity solicitations for vendor proposals and issue task orders to selected vendors, respectively. GSA established two critical milestones for agencies to complete these acquisition activities: (1) release all fair opportunity solicitations by March 31, 2019, and (2) issue all task orders by September 30, 2019. Regarding the first milestone—to release all EIS fair opportunity solicitations by March 31, 2019—five of the 19 selected agencies reported that they released all of their solicitations by this date. The 14 remaining selected agencies reported that they did not release all of their solicitations by this date. Table 4 identifies the 19 selected agencies’ status in meeting GSA’s milestone to release all EIS fair opportunity solicitations by March 31, 2019. Officials from each of the five agencies that met GSA’s milestone to finish releasing all of their planned EIS solicitations by March 31, 2019, reported that their agencies released either one or two solicitations. In particular, officials from GSA and the Departments of Justice and Transportation reported that their agencies each released one solicitation, and Commerce and Social Security Administration officials reported that their agencies each released two solicitations. While eight of the 14 other selected agencies had also planned to release either one or two solicitations in total for their transitions, officials from these agencies reported that they did not finish releasing them by March 31, 2019. These agencies were the Departments of Agriculture, Education, Energy, HHS, Housing and Urban Development, Labor, and State; and the Small Business Administration. We asked officials from the 14 selected agencies that did not release all of their planned EIS solicitations by March 31, 2019, to identify the key factors that contributed to their agencies’ delays in releasing these solicitations. In response, agency officials cited numerous key factors for the delays, including the complexity of their telecommunications requirements, changes to the agency’s or GSA’s contracting strategy, and insufficient staff availability. Figure 2 identifies the key factors that contributed to delays in releasing all EIS solicitations by GSA’s March 31, 2019, milestone, as identified by agency officials. In addition, regarding GSA’s second milestone—to issue all EIS task orders by September 30, 2019—one of the selected agencies (the Small Business Administration) reported that it issued all of its task orders by this date. The 18 other agencies reported that they did not issue all of their EIS task orders by this date. Table 5 identifies the 19 selected agencies’ status in meeting GSA’s milestone to issue all EIS task orders by September 30, 2019. Officials from the Small Business Administration—the only agency that met GSA’s September 30, 2019, milestone—reported that the agency issued its lone task order on September 27, 2019. We asked officials from the 18 agencies that did not issue all of their EIS task orders by September 30, 2019, to identify the key factors that contributed to their agencies’ delays in issuing these task orders. In response, agency officials cited 19 key factors that led to the delays. Nine of the identified factors were the same factors that officials cited for their agencies’ delays in releasing EIS solicitations, including the complexity of requirements and having insufficient staff available. The officials also identified 10 other factors unique to their delays in issuing EIS task orders. For example, officials from two agencies reported that the EIS vendors needed clarification on the agencies’ requests for proposals. In addition, officials from three agencies reported that they needed clarification from the EIS vendors on the proposals that the agencies received. Figure 3 identifies the key factors that contributed to delays in issuing all EIS task orders by GSA’s September 30, 2019, milestone, as identified by agency officials. Several of the identified factors, such as the partial government shutdown and the need for vendors to receive authorities to operate, have subsequently been resolved. For other factors, agencies can leverage GSA’s available EIS training and customer support to help minimize delays in meeting GSA’s transition milestones. However, given that the majority of the selected agencies did not meet these transition milestones in 2019, it will be important for agencies to meet the remaining transition milestones to ensure that they complete the transition before the current telecommunications contracts expire in May 2023. In a June 2006 report, we identified five transition planning practices that can help agencies reduce the risk of experiencing adverse effects of moving from one broad telecommunications contract to another. Implementing these transition planning practices represents a comprehensive and rigorous management approach that can help agencies make the most of the opportunity for change that such a major telecommunications transition provides. Each of the five transition planning practices that we identified consists of various activities that should be implemented to fully address the planning practices. Table 6 identifies the five established transition planning practices and their associated activities. All five selected agencies—Commerce, HHS, NASA, State, and VA—had taken steps to implement the five established transition planning practices. However, none of these agencies had fully implemented any of the practices. The five selected agencies had all partially implemented the first established transition planning practice—to develop an accurate inventory of telecommunications assets and services. In particular, all of the selected agencies had partially implemented the two activities associated with this practice. Table 7 summarizes the extent to which the selected agencies had implemented the transition practice to develop an accurate inventory of telecommunications services. Identify a complete telecommunications inventory at every site, facility, and component. The five selected agencies had all partially implemented this activity. While all of these agencies had developed inventories of their telecommunications assets and services, none of the inventories were complete. Specifically, the inventories that Commerce, NASA, and VA developed included the enterprise-wide assets and services in use at their agencies; however, the inventories did not include all of the assets and services that individual mission offices ordered for their own use. In addition, HHS’s and VA’s inventories did not include their assets and services that were associated with commercial contracts not managed by GSA. Moreover, none of the agencies’ inventories included all of the relevant contractors that were listed on USASpending.gov as having received telecommunications-related contracts from those agencies in fiscal years 2018 or 2019. As such, the inventories also did not include assets and services provided by those contractors. Establish a documented process for updating and maintaining the inventories. All five selected agencies partially implemented this activity by taking steps to document their inventory update and maintenance processes. However, none of the agencies had fully documented these processes. Specifically, Commerce, HHS, NASA, and State had documented and finalized their processes for updating and maintaining certain telecommunications assets and services within their inventories. However, these processes did not apply to all assets and services in use at the agencies. For example, NASA’s inventory maintenance processes applied to the agency’s enterprise- level assets and services, but did not apply to assets and services ordered by individual mission centers. VA had developed draft procedures for updating its inventories when new service requests were submitted, but it had not finalized these processes. In addition, VA had not documented processes for maintaining its inventories (e.g., removing telecommunications services from the inventories when they are disconnected). Officials from three of the selected agencies—Commerce, NASA, and VA—cited the same cause for not having complete inventories or associated inventory maintenance procedures. Specifically, the officials from these agencies—all of whom were responsible for their agencies’ transitions to EIS—stated that they did not track all of the assets and services ordered by the agencies. The officials added that they were not responsible for maintaining inventories of all of their agencies’ assets and services. Further, officials in NASA’s and VA’s offices of the Chief Information Officer did not provide inventories of the assets and services ordered by those agencies’ individual mission offices, or any documentation of their agencies’ associated inventory maintenance processes. Commerce officials acknowledged their lack of a complete telecommunications inventory and stated that they were working to identify the agency’s assets and services associated with individual mission offices. The officials stated that they planned to complete this identification effort by 2023, but this schedule was not documented. State officials said that their telecommunications inventories did not include all of the relevant contractors that were listed on USASpending.gov as having received telecommunications-related contracts from the agency in fiscal years 2018 or 2019 because some of the contracts listed on USASpending.gov were for telecommunications services that State does not plan to purchase from EIS. State officials said that their initial focus for the EIS transition is to replace their current domestic services that are ordered through GSA’s telecommunications contracts before those contracts expire. However, all of the relevant telecommunications contractors used by State and reported at USASpending.gov should be included in State’s telecommunications inventory. The lack of a complete inventory that includes these contractors and their associated services will likely limit State’s ability to fully identify areas for optimization and the sharing of telecommunications resources across the agency. Officials from the one remaining agency—HHS—attributed their agency’s lack of a complete telecommunications inventory and associated maintenance procedures to the agency’s decentralized structure. Specifically, the HHS officials stated that the agency’s components are responsible for managing the services that are unique to them, including those associated with commercial contracts not managed by GSA. However, the officials stated that the agency did not have a policy that required its components to maintain an inventory of telecommunications assets and services that they acquired independently. Without complete and accurate telecommunications inventories, the selected agencies may be unable to avoid unnecessary transition delays related to an inability to plan for services not identified in the inventory. The agencies will also likely be limited in their ability to determine areas for optimization and the sharing of telecommunications and IT resources across the agencies. In addition, without documented processes for maintaining inventories of all of their telecommunications assets and services in use, the agencies may not be able to consistently and accurately incorporate into their telecommunications inventories any changes made during and after the transition (e.g., adding new services or removing disconnected services), thus hindering their ability to ensure that they are billed appropriately by the vendor. All of the selected agencies had partially implemented the second established transition planning practice—to perform a strategic analysis of telecommunications requirements. In particular, of the four activities associated with this practice, NASA had fully implemented three of the activities and partially implemented one activity; HHS and VA had fully implemented two of the activities and partially implemented the other two activities; State had fully implemented one of the activities and partially implemented the other three activities; and Commerce had partially implemented each of the four activities. Table 8 summarizes the extent to which the selected agencies had conducted strategic analyses of their telecommunications requirements. Identify current and future telecommunications needs using an inventory of existing services. All of the selected agencies had partially implemented this activity by identifying certain current and future telecommunications needs. However, as discussed earlier, none of the agencies had a complete inventory of current services. As a result, the agencies could not use such an inventory to fully identify their needs. Identify areas for optimization or sharing of telecommunications and IT resources. Three agencies—HHS, NASA, and VA—had fully implemented this activity by completing strategic analyses to identify areas for optimization or sharing of telecommunications resources. The two remaining agencies—Commerce and State—had partially implemented this activity. Specifically, while Commerce had developed a draft strategic analysis to justify the potential optimization and sharing across the agency of a telecommunications service for how hardware devices connect to the internet, it had not yet finalized this analysis. One Commerce bureau had also conducted a strategic analysis to justify potentially optimizing or sharing multiple telecommunications services and IT resources within that bureau, but Commerce was unable to provide documentation demonstrating that its remaining bureaus had conducted similar analyses. Further, while State had conducted a strategic analysis to identify services that could be optimized across the agency and agency officials had also identified potential areas for sharing of resources, State did not provide a documented analysis to justify the sharing of those resources. Evaluate the costs and benefits of any new technology and alternative options. Four agencies—HHS, NASA, State, and VA— had fully implemented this activity by evaluating the costs and benefits of various technologies and alternative options for telecommunications services that they could implement as part of the transition. The one remaining agency—Commerce—had partially implemented this activity. Specifically, while Commerce demonstrated that it had evaluated the costs and benefits of upgrading one service by which hardware devices connect to the internet, and two Commerce bureaus had analyzed the costs and benefits of implementing another type of service for connecting to networks, the remaining Commerce bureaus did not conduct such analyses. Determine that identified telecommunications needs and opportunities are aligned with the agency’s mission, long-term IT plans, and enterprise architecture plans. One agency—NASA— had fully implemented this activity by determining that its telecommunications needs aligned with its mission and plans. The four remaining agencies had partially implemented this activity. Specifically, HHS had determined that its telecommunications needs aligned with its mission and enterprise architecture, but it did not demonstrate a similar alignment with its long-term IT plans. In addition, State had demonstrated that its needs aligned with its mission, but it did not determine and document that these needs aligned with the agency’s long-term IT plans and enterprise architecture. Further, one Commerce bureau had determined that its needs aligned with its mission, long-term IT plans, and enterprise architecture. However, the remaining Commerce bureaus did not determine and document that their telecommunications needs were aligned with the agency’s long-term IT plans and enterprise architecture. VA also had determined that its identified needs aligned with its mission and enterprise architecture, as they relate to an ongoing telecommunications modernization project. However, while VA officials stated that their telecommunications needs were aligned with the agency’s long-term IT plans, the officials did not provide documentation demonstrating this alignment. Agency officials cited several reasons for not fully implementing the activities associated with this practice. For example, NASA did not use a complete inventory of existing telecommunications assets and services to identify its future telecommunications needs because, as discussed earlier, NASA officials stated that the agency’s telecommunications inventory included only enterprise-level assets and services, and did not include assets and services ordered by individual mission centers. The officials further explained that they were not responsible for maintaining inventories of those mission offices’ telecommunications assets and services and, therefore, did not track all of those assets and services. In addition, Commerce officials stated in May 2019 that the majority of the agency’s bureaus did not conduct cost-benefit analyses that considered implementing new telecommunications technologies because Commerce was planning to transition its services on a like-for-like basis in order to complete the transition before May 2020, which was when the current telecommunications contracts were previously set to expire. As such, the officials stated that the agency was not planning to implement new technologies and, thus, a cost-benefit analysis of such technologies was not necessary. However, in October 2019, Commerce officials stated that the agency’s EIS solicitation included options for vendors to propose the implementation of new technologies. State officials explained that they had not conducted and documented an analysis to identify areas for the sharing of telecommunications resources because they did not believe that there were any additional State telecommunications resources that could be shared. State officials attributed this to the agency’s security requirements and regulations, and noted that services on State’s classified network may not be shared with services on its unclassified network. Nevertheless, while services may not be able to be shared between these networks, State did not provide documentation that demonstrated that the agency had determined that there were no additional resources that could be shared on State’s unclassified network. In November 2019, VA officials stated that they thought their telecommunications needs were aligned with the agency’s long-term IT plans. However, the officials did not provide documentation demonstrating this alignment. HHS officials stated that they intend to align the agency’s telecommunications needs and IT strategic plans after the agency establishes a centralized transition program management office. Specifically, the agency decided to centralize its transition management approach in March 2019 and, as of December 2019, HHS officials expected the office to be fully established by March 2020. However, the officials did not have documented plans for when they would align the agency’s telecommunications needs and IT strategic plans. Agencies that do not use complete inventories of their current telecommunications services to identify their future needs are likely not fully identifying these needs. They may also miss opportunities to optimize or share services by consolidating them on EIS. In addition, by not using a rigorous management approach that includes strategically analyzing, identifying, and documenting areas for optimization and sharing of resources, agencies may miss opportunities to upgrade their telecommunications services or to shift these services to more cost- effective technologies. Further, agencies that do not fully assess the costs and benefits of alternatives for meeting their telecommunications needs may miss the opportunity that the transition provides to optimize their telecommunications services. Moreover, without aligning their telecommunications needs and opportunities with their missions and plans, agencies risk missing opportunities to use the new contract to address their highest priorities, or may make decisions that are not aligned with their long-term goals. All of the selected agencies had partially implemented the third transition planning practice—to develop a structured management approach for the telecommunications transition. Specifically, of the three activities associated with this practice, NASA had fully implemented two activities and partially implemented one activity; HHS and VA had fully implemented one activity and partially implemented the other two activities, and Commerce and State had partially implemented each of the three activities. Table 9 summarizes the extent to which the selected agencies had established a structured management approach. Establish a transition management team and clearly define responsibilities for key transition roles. One agency—VA—had fully implemented this activity by establishing a transition management team and defining all key transition responsibilities for the planning and execution phases of the transition, including for project, asset, human capital, and information security management; and contract and legal expertise. The remaining four agencies had partially implemented this activity by establishing transition management teams, but none had defined all key roles and responsibilities for their transitions. Specifically, NASA had not defined a role and related responsibilities for managing human capital throughout the transition, nor for providing legal expertise during the execution phase of the transition. While Commerce had identified the need for managing human capital and telecommunications assets throughout the planning and execution phases of the transition, and for providing legal expertise during the execution phase of the transition, it had not yet assigned these roles and related responsibilities to staff members. In addition, Commerce, State, and HHS had identified the need for an information security management role during the transition. However, Commerce and State had not yet finalized the responsibilities for this role, and Commerce and HHS had not yet assigned this role to a staff member. State and HHS had also not identified roles and responsibilities for managing telecommunications assets throughout the transition, nor for providing legal expertise during the execution phase of the transition. Moreover, while HHS officials stated that a staff member was providing human capital management-related assistance to the agency’s centralized EIS program management office, the agency had not documented this role for the transition, nor defined specific responsibilities for this role. Develop transition communications plans in order to facilitate information sharing during transition planning and execution. Two agencies—HHS and NASA—had fully implemented this activity by developing transition communications plans and identifying all key parties that need to be involved during the agency’s transition effort. The remaining three agencies—Commerce, State, and VA—partially implemented this activity. For example, each of these agencies identified stakeholders responsible for communicating transition information to other stakeholders. While Commerce and VA also identified the frequency with which transition status updates and meetings are to occur, State did not identify this frequency. In addition, State and one bureau within Commerce did not include a description of how changes and disruptions related to the transition would be communicated to end users. Further, Commerce, State, and VA did not identify the key local and regional agency transition officials responsible for disseminating information about the transition to employees and working with the vendor to facilitate transition activities. While VA had identified a potential list of these officials in a previous version of the agency’s transition communications plan, the agency removed this list from the latest version of the plan. Use established project, configuration, and change management processes in the agency’s transition planning efforts. One agency—NASA—had fully implemented this activity by demonstrating the use of all established management processes called for in the activity. The four remaining agencies—Commerce, HHS, State, and VA—had partially implemented this activity by demonstrating the use of project management processes for their transitions, such as tracking transition costs and developing schedules and risk logs. However, VA did not demonstrate that it was applying approved cost and schedule management processes to its transition. In addition, Commerce, HHS, and State did not demonstrate that they were applying established configuration management processes to their transitions. Further, Commerce and HHS did not demonstrate that they had implemented change management processes for their transitions. Officials from four of the selected agencies—Commerce, HHS, NASA, and VA—generally attributed their lack of full implementation of this practice to the fact that, at the time of our review, the agencies were early in their transition planning processes. For example, NASA officials stated that they had not defined a role or responsibilities related to human capital management because their human capital needs for the transition will depend on the vendors selected (incumbents or new vendors). As such, the officials stated that they had not yet determined whether a human capital management role was needed for the transition. The officials said that they would consider adding such a role after they issue their EIS task orders. However, NASA did not conduct an analysis to determine whether there was a need for a human capital manager during the planning phase of the transition. As a result, NASA is risking delays that could lengthen its transition due to the lack of an assigned staff member to manage its human capital needs during the transition planning phase. In addition, State officials said that they did not identify the key local and regional agency transition officials responsible for working with the vendor to facilitate transition activities because, as part of State’s security processes, vendors must work with State’s bureau-level points-of-contact to be escorted to State facilities, as necessary. The State officials said that their bureau-level points-of-contact would coordinate with the local and regional agency transition officials, as appropriate. VA officials stated that they removed from their transition communications plan the list of key local and regional agency transition officials because, in part, as of November 2019 it was still early in the agency’s transition and they expected the contacts to change as the transition is implemented. As such, VA officials also stated that they only identified key transition positions, rather than individuals, in order to ensure the accuracy of the information in the communications plan. Commerce officials explained that they had not yet implemented all of the key management processes for the transition because they planned to work with their selected EIS vendors to establish those processes. These officials further stated that they planned to implement this activity after they issue their EIS task orders. Moreover, HHS officials attributed their lack of established configuration and change management processes to the agency’s previous decentralized management approach, which did not require HHS’s components to establish such processes for the transition. As discussed earlier, in March 2019, the agency decided to centralize its transition management approach. HHS officials stated that, as part of the centralized approach, they planned to develop change and configuration management processes for the transition. However, they did not have documented time frames for establishing and implementing these processes. While the selected agencies were early in their transition planning processes at the time of our review, the limited time remaining to complete the transition makes it critical that agencies conduct early planning with the information that is available. Agencies that do not define all key roles and related responsibilities for their transition management teams risk extending their transition period as they attempt to assign appropriate personnel and update them on transition progress and issues. Further, without identifying all of the key officials that need to be involved with the transition, including the local and regional agency points of contact, agencies may lack the information that is necessary for comprehensive understanding, accountability, and shared expectations among all those with transition responsibilities. Finally, by not using a rigorous management approach that implements established configuration management and change management processes for the transition, agencies risk additional financial costs, extended timelines, and disruptions to the continuity of their telecommunications systems. The limited time available for agencies to complete the transition makes it more important for them to use rigorous management processes in their transition efforts. All of the selected agencies had partially implemented the fourth established transition planning practice—to identify their transition resource needs. In particular, of the four activities associated with this practice, NASA had fully implemented one of the activities and partially implemented the remaining three activities; and the four other agencies— Commerce, HHS, State, and VA—had partially implemented each of the activities. Table 10 summarizes the extent to which the selected agencies had identified their transition resource needs. Identify the level of funding needed to support transition planning. One of the selected agencies—NASA—had fully implemented this activity by identifying the costs needed to support its transition management team and all years of its transition planning efforts. The four other agencies—Commerce, HHS, State, and VA— had partially implemented this activity. In particular, HHS had developed a cost estimate that partially identified the funding needed for its transition management team, but this estimate did not identify the costs for all transition management staff at each of the agency’s components. Commerce had developed a draft analysis that identified the funding needed for government and contractor staff working on the transition, but this analysis was not approved. In addition, one Commerce bureau had not yet identified the funding needed for all years of transition planning support. Further, while State had partially identified the funding needed to support federal and contractor staff working on the transition, it had not identified the funding needed for all transition staff or for all years of transition planning support. Moreover, while VA officials stated that they had identified the costs needed for the transition, the officials did not provide documentation that identified costs for all years of transition planning support. Identify the organizational need for investments and justify resource requests. The five selected agencies had all partially implemented this activity by identifying the need for investments, including funding to obtain GSA transition assistance; however, none of the agencies had fully justified their resource requests for the transition. Specifically, Commerce, State, and VA had not justified their resource requests related to transition program management staff. In addition, HHS lacked justification for its requests for hardware and software upgrades. Moreover, while NASA had identified anticipated cost savings as part of its justification for resource requests related to hardware and software upgrades, it was unable to provide documentation of an analysis to support these identified savings. NASA also did not justify its resource requests related to transition program management staff. Identify human capital needs for the entire transition effort. All of the selected agencies had partially implemented this activity by identifying the need for certain staff to work on the transition, including government and contractor staff. However, none of the agencies had conducted and documented analyses of their human capital needs, to determine the total number of staff required to support their entire transition efforts. Identify and require training for the transition. All of the agencies had partially implemented this activity by identifying training needed by certain transition management staff. In addition, four of the agencies—Commerce, HHS, NASA, and State—had also provided training to transition support staff. However, Commerce, HHS, NASA, and VA had not conducted and documented analyses to identify all of the training needed for their transitions, including training for staff carrying out the transition or operating and maintaining new equipment or services. In addition, while State had developed a draft analysis to identify training needed by staff carrying out the transition, it had not finalized this analysis. Officials from these agencies cited several reasons for not fully identifying their transition resource needs. In general, Commerce, HHS, and VA officials explained that they were too early in their transition efforts to identify all of the funding, human capital, and training needed for their transitions. NASA and State officials also cited this as the reason for why they had not identified all of their human capital needs. In particular, officials from all five of the agencies stated that they will not be able to determine their complete transition resource needs until after they issue their EIS task orders. For example, officials from all of these agencies explained that their human capital needs will depend on which vendors are selected and what new technology will be implemented, if any. Officials from these agencies also stated that they planned to identify all of their human capital needs after they issue their EIS task orders, but none of the agencies had documented plans for doing so. In addition, Commerce officials said that they did not document a cost- benefit justification for using contractor staff to assist with transition program management because they knew that their existing resources (i.e., government staff) were not sufficient. As such, the officials stated that the agency determined that further analysis for justification of using contractor staff was not necessary. State officials also explained that they had not identified all of the funding needed to support transition planning because, per agency policy, they were not required to do so. In particular, the officials explained that the division responsible for the EIS transition operates under a working capital fund. As part of this, the division provides telecommunications services to State customers and charges those customers for the services provided. In accordance with State policy, the division determines the costs for these services on an annual basis. As such, the officials stated that they were not required by agency policy to determine the total funding needed for the entire transition. However, although State policy does not require the agency to identify all of the funding needed to support transition planning, as part of a comprehensive management approach to the transition State should identify its complete transition funding requirements to ensure that sufficient resources are available when needed during the transition. While these agencies may be early in their transition efforts, there is limited time remaining to complete the transition before the current telecommunications contracts expire. If the agencies do not conduct early planning to identify and justify all of their resources needed for the transition, they may underestimate the complexity and demands of their transition efforts. In addition, without using a rigorous management approach to analyze and document the total number of staff required to support the transition and to identify all of the required training for transition staff, agencies risk having insufficient staff available or may experience gaps in staff competencies. Such gaps may lead to delays and unexpected costs as the agencies try to quickly address the lack of resources during the transition’s limited time frame. All of the selected agencies had partially implemented the fifth established transition planning practice—to develop transition plans. Specifically, of the three activities associated with this practice, three agencies—Commerce, NASA, and State—had fully implemented two activities and partially implemented the remaining activity; and two agencies—HHS and VA—had fully implemented one activity and partially implemented the other two activities. Table 11 summarizes the extent to which the selected agencies had developed transition plans. Identify agency-specific transition objectives and measures of success. Three agencies—Commerce, NASA, and State—had fully implemented this activity by identifying transition objectives and associated measures of success that were based on the transition objectives. The remaining two agencies—HHS and VA—had partially implemented this activity. In particular, while these agencies had identified transition objectives and measures of success, their measures were unable to be used to assess transition progress. Specifically, HHS and VA had identified measures that could be used to determine success at the completion of the transition (e.g., all planned services have been transitioned to EIS). However, the measures did not enable the agencies to compare expected performance with actual results in order to track progress during the course of the transition (e.g., identifying the expected number of services that would be moved to EIS during each year of the transition). Identify risks that could affect transition success, including information security risks, and evaluate the importance of these risks relative to the agency’s mission critical systems and continuity of operations plans. All of the selected agencies— Commerce, HHS, NASA, State, and VA—had fully implemented this activity. Specifically, each of the agencies had identified transition risks and evaluated the importance of those risks relative to the agencies’ mission critical priorities. Clearly define transition preparation tasks and develop a time line that takes into account the agency’s mission critical systems, contingency plans, and identified risks. All of the selected agencies partially implemented this activity by developing time lines with clearly defined transition preparation tasks. However, none of these time lines accounted for all key priorities identified in the activity. Specifically, while a 2016 version of Commerce’s transition time line took into account one of the agency’s identified transition risks, Commerce’s more recent transition time lines did not account for its transition risks or for priorities related to its mission critical systems and contingency plans. In addition, NASA’s time lines took into account its transition risks, but did not account for priorities related to its mission critical systems and contingency plans. State’s and VA’s transition time lines did not account for any of these priorities. Further, while HHS had developed time lines with clearly defined transition preparation tasks for certain components of the agency, it did not develop time lines that defined such tasks for all of its components. The time lines that HHS had developed also did not account for priorities related to all of HHS’s mission critical systems, contingency plans, and identified risks. Agency officials identified several reasons for not yet fully implementing the activities associated with developing a transition plan. For example, HHS officials attributed their lack of transition measures of success that could be used to assess transition progress to the agency’s previous decentralized transition management approach. The HHS officials stated that, as part of their new centralized management approach, they planned to develop such measures by the time the agency issues its EIS task order. However, the officials did not have documented plans for developing these measures. In addition, VA officials stated that they had not identified agency-specific transition measures of success that could be used to assess transition progress because these measures will be dependent on the EIS vendors that the agency selects. The officials stated that they expected to define these measures after they issue their EIS task orders. However, as of November 2019, the officials did not have documented plans for finalizing these measures. Moreover, officials from all of the selected agencies generally said that they had not yet developed complete transition time lines because they were focused on activities associated with the acquisition planning phase of the transition, including developing their EIS solicitations. Officials from all of the agencies said that they planned to develop complete transition time lines after they issue their EIS task orders. While agencies’ lack of issued EIS task orders contributed to delays in developing complete transition plans, the limited time remaining to complete the transition makes it critical that agencies conduct early planning with the information that is available. In addition, agencies that do not identify transition objectives and measures of success that can be used to assess transition progress may find it difficult to provide those involved in their transitions with clear expectations. Without measurable metrics, managers will also lack information that could be used to track progress toward transition objectives and inform management decisions. Further, agencies that do not assess risks relative to their mission critical systems and do not incorporate agency priorities related to those systems and contingency plans into transition time lines, may encounter problems and delays during the transition because they are not adequately prepared to mitigate such risks. Although the 19 selected agencies reported that they plan to fully transition to EIS before the current telecommunications contracts expire in May 2023, over half of the agencies do not plan to complete the transition by GSA’s September 30, 2022, milestone to do so. By waiting until close to the end of the current contracts to finish the transition, these agencies are at risk of experiencing disruptions in service if any issues arise that result in transition delays, such as inadequate human capital resources or the need to transition previously unidentified services. Moreover, given agencies’ poor performance during the last two transitions—which resulted in significant delays and cost increases—and their lack of meeting GSA’s two critical EIS transition milestones for 2019, agencies are again at high risk of experiencing delays during this transition. Further, agencies will miss out on potential cost savings by delaying their transitions to the new contracts, which generally have lower rates for services. The five agencies we reviewed had taken steps to prepare for the transition of their telecommunications services to EIS contracts. However, these agencies’ lack of full implementation of established planning practices increases the risk that they will experience adverse effects— such as schedule delays or cost increases—while transitioning to the new contracts. Several agencies stated that they intend to implement the planning practices after they have issued their EIS task orders. However, limited time remains to complete the transition before the current telecommunications contracts expire. Further, inadequate project planning was a key factor that contributed to delays during the prior transition to Networx. Accordingly, it is critical for agencies to apply a rigorous management approach from the start of the current transition using the information that is currently available, even though changes may be necessary as conditions evolve. Agencies that do not fully adopt the comprehensive approach captured in these planning practices may not make the most of the opportunity for change, and the potential to save costs, that such a major telecommunications transition provides. We are making a total of 25 recommendations to five agencies, which includes five each to Commerce, HHS, NASA, State, and VA. The Secretary of Commerce should ensure that the agency’s Chief Information Officer updates the telecommunications inventory to include all telecommunications assets and services in use at the agency, and updates Commerce’s process for ongoing maintenance of the inventory to include the complete inventory. (Recommendation 1) The Secretary of Commerce should ensure that the agency’s Chief Information Officer completes efforts to identify future telecommunications needs using a complete inventory of existing telecommunications services; conducts and documents a comprehensive strategic analysis at all bureaus to identify areas for optimization and sharing of telecommunications resources; evaluates the costs and benefits of implementing new telecommunications technology and alternative options at all bureaus; and fully aligns Commerce’s telecommunications needs with its long-term IT plans and enterprise architecture. (Recommendation 2) The Secretary of Commerce should ensure that the agency’s Chief Information Officer finalizes the responsibilities related to the information security management role during the telecommunications transition, and assigns the roles for providing legal expertise during the transition, as well as for managing human capital, telecommunications assets, and information security during the transition, to staff members; describes how changes and disruptions related to the transition will be communicated to end users at all bureaus and identifies the key local and regional agency transition officials responsible for disseminating information about the transition to employees and working with the vendor to facilitate transition activities in Commerce’s transition communications plan; and establishes and implements configuration and change management processes for its transition. (Recommendation 3) The Secretary of Commerce should ensure that the agency’s Chief Information Officer identifies all of the funding needed to support the telecommunications transition; justifies requests for resources related to transition program management staff; conducts an analysis to identify staff resources needed for the entire transition effort; and analyzes training needs for staff assisting with the transition. (Recommendation 4) The Secretary of Commerce should ensure that the agency’s Chief Information Officer takes into account the agency’s telecommunications transition risks, mission critical systems, and contingency plans in Commerce’s transition time line. (Recommendation 5) The Secretary of Health and Human Services should ensure that the agency’s Chief Information Officer develops a policy that requires the agency’s components to maintain an inventory of the telecommunications assets and services that they acquire independently from headquarters; updates the telecommunications inventory to include all telecommunications assets and services in use at HHS, and updates the agency’s process for ongoing maintenance of the inventory to include the complete inventory. (Recommendation 6) The Secretary of Health and Human Services should ensure that the agency’s Chief Information Officer completes efforts to identify future telecommunications needs using a complete inventory of existing telecommunications services; and aligns HHS’s telecommunications needs with its long-term IT plans. (Recommendation 7) The Secretary of Health and Human Services should ensure that the agency’s Chief Information Officer identifies and documents telecommunications transition roles and responsibilities related to (1) managing assets and human capital during the planning and execution phases of the transition and (2) providing legal expertise during the execution phase of the transition, and assigns the transition information security management role to a staff member; and establishes and implements configuration and change management processes for HHS’s transition. (Recommendation 8) The Secretary of Health and Human Services should ensure that the agency’s Chief Information Officer identifies all of the funding needed to support the telecommunications transition at each of the agency’s components, justifies requests for transition resources related to hardware and software upgrades, conducts an analysis to identify staff resources needed for the entire transition effort, and analyzes training needs for staff assisting with the transition. (Recommendation 9) The Secretary of Health and Human Services should ensure that the agency’s Chief Information Officer completes efforts to identify telecommunications transition measures of success that can be used to assess transition progress; and takes into account all of the agency’s components, as well as its mission critical systems, contingency plans, and telecommunications transition risks, in HHS’s transition time line. (Recommendation 10) The Secretary of State should ensure that the agency’s Chief Information Officer updates the telecommunications inventory to include all telecommunications assets and services in use at the agency, and updates State’s process for ongoing maintenance of the inventory to include the complete inventory. (Recommendation 11) The Secretary of State should ensure that the agency’s Chief Information Officer completes efforts to identify the agency’s future telecommunications needs using a complete inventory of existing telecommunications services; conducts and documents a strategic analysis to justify the sharing of telecommunications resources; and aligns State’s telecommunications needs with its long-term IT plans and enterprise architecture. (Recommendation 12) The Secretary of State should ensure that the agency’s Chief Information Officer identifies telecommunications transition roles and responsibilities related to (1) managing assets during the planning and execution phases of the transition and (2) providing legal expertise during the execution phase of the transition, and finalizes the responsibilities related to the information security management role for the transition; includes in State’s transition communications plan the frequency with which transition status updates and meetings will occur throughout the transition, a description of how changes and disruptions related to the transition will be communicated to end-users, and the key local and regional agency transition officials responsible for disseminating information about the transition to employees and working with the vendor to facilitate transition activities; and establishes configuration management processes for the agency’s transition. (Recommendation 13) The Secretary of State should ensure that the agency’s Chief Information Officer identifies all of the funding needed to support the telecommunications transition, justifies requests for resources related to transition program management staff, conducts an analysis to identify staff resources needed for the entire transition effort, and finalizes its analysis of training needs for staff assisting with the transition. (Recommendation 14) The Secretary of State should ensure that the agency’s Chief Information Officer takes into account the agency’s telecommunications transition risks, mission critical systems, and contingency plans in State’s transition time line. (Recommendation 15) The Secretary of Veterans Affairs should ensure that the agency’s Chief Information Officer updates the telecommunications inventory to include all telecommunications assets and services in use at the agency, and updates and finalizes VA’s process for ongoing maintenance of the inventory to include the complete inventory. (Recommendation 16) The Secretary of Veterans Affairs should ensure that the agency’s Chief Information Officer completes efforts to identify future telecommunications needs using a complete inventory of existing telecommunications services, and determines and documents that VA’s telecommunications needs are aligned with its long-term IT plans. (Recommendation 17) The Secretary of Veterans Affairs should ensure that the agency’s Chief Information Officer includes in its telecommunications transition communications plan the key local and regional agency officials responsible for disseminating information about the transition to employees and working with the vendor to facilitate transition activities; and establishes and uses cost and schedule management processes in the agency’s transition. (Recommendation 18) The Secretary of Veterans Affairs should ensure that the agency’s Chief Information Officer identifies and documents all of the funding needed to support the telecommunications transition, including costs for all years of transition planning support; justifies requests for transition resources related to program management staff; conducts an analysis to identify staff resources needed for the entire transition effort; and analyzes training needs for staff assisting with the transition. (Recommendation 19) The Secretary of Veterans Affairs should ensure that the agency’s Chief Information Officer completes efforts to identify telecommunications transition measures of success that can be used to assess transition progress; and takes into account the agency’s telecommunications transition risks, mission critical systems, and contingency plans in VA’s transition time line. (Recommendation 20) The Administrator of the National Aeronautics and Space Administration should ensure that the agency’s Chief Information Officer updates the telecommunications inventory to include all telecommunications assets and services in use at the agency, and updates NASA’s process for ongoing maintenance of the inventory to include the complete inventory. (Recommendation 21) The Administrator of the National Aeronautics and Space Administration should ensure that the agency’s Chief Information Officer completes efforts to identify the agency’s future telecommunications needs using a complete inventory of existing telecommunications services. (Recommendation 22) The Administrator of the National Aeronautics and Space Administration should ensure that the agency’s Chief Information Officer identifies telecommunications transition roles and responsibilities related to (1) managing human capital during the planning and execution phases of the transition and (2) providing legal expertise during the execution phase of the transition. (Recommendation 23) The Administrator of the National Aeronautics and Space Administration should ensure that the agency’s Chief Information Officer conducts an analysis to support the anticipated cost savings identified as part of the agency’s justification for its resource requests related to hardware and software upgrades for the telecommunications transition, and justifies its resource requests for transition program management staff; conducts an analysis to identify staff resources needed for the entire transition effort; and analyzes training needs for staff assisting with the transition. (Recommendation 24) The Administrator of the National Aeronautics and Space Administration should ensure that the agency’s Chief Information Officer takes into account the agency’s mission critical systems and contingency plans in NASA’s telecommunications transition time line. (Recommendation 25) We provided a draft of this report to the 19 selected agencies for their review and comment. In response, all five agencies to which we made recommendations (Commerce, HHS, State, VA, and NASA) stated that they concurred with the recommendations. In addition, of the 14 agencies to which we did not make recommendations, one (the Department of the Treasury) provided comments on the report, and one (the Small Business Administration) provided a technical comment via email, which we incorporated into the report, as appropriate. The remaining 12 agencies did not have any comments on the report. The following five agencies concurred with our recommendations: In written comments (reprinted in appendix II), Commerce concurred with our five recommendations to the agency and stated that it will take steps to implement them. In written comments (reprinted in appendix III), HHS concurred with our five recommendations to the agency and described actions it has taken or plans to take to address them. For example, with regard to our recommendation that HHS identify and document key telecommunications transition roles and responsibilities, among other things, the agency stated that it had (1) established an integrated program team to coordinate all telecommunications transition activities, in conjunction with its EIS program management office; (2) assigned two legal counsel staff to support the EIS transition during its current procurement phase, as well as for the transition; and (3) included the agency’s Office of Information Security in reviewing and providing input into its EIS solicitation. The agency also stated that it intends to engage the Office of Information Security throughout the lifecycle of the EIS transition, among other things. HHS also provided general comments in response to the findings in the report. Specifically, the agency described actions that it had taken to improve its management of the EIS transition. For example, the agency stated that the Assistant Secretary for Administration decided to centralize HHS’s EIS transition efforts in March 2019, after it had conducted a study of risks and costs associated with the decentralized transition approach that the agency had been taking since 2017. HHS further stated that it had identified the issues that we brought up during our review and had proactively worked since March 2019 to establish processes and procedures to manage its transition in a comprehensive manner. In particular, the agency stated that it established a fully funded, centralized EIS program management office to support all of HHS’s operating divisions during the transition. Establishing and effectively implementing such management processes will be critical to the agency’s successful transition to EIS. In written comments (reprinted in appendix IV), State concurred with our five recommendations to the agency. In written comments (reprinted in appendix V), VA stated that it agreed with our conclusions and concurred with our five recommendations to the agency. VA also stated that it would provide the actions it plans to take to address the recommendations in its 180- day update to the final report. In written comments (reprinted in appendix VI), NASA concurred with our five recommendations to the agency. It also described actions it has taken or plans to take to address each recommendation. For example, the agency described actions it has taken to address our recommendation calling for NASA to update its telecommunications inventory to include all telecommunications assets and services in use at the agency, among other things. Specifically, the agency stated that the NASA communications contractor, under NASA management oversight, maintains an inventory of telecommunications assets and services. The agency added, nevertheless, that unique mission assets are not included in the inventory, are managed by programs and projects, and are available to the NASA Office of the Chief Information Officer. We agree that NASA has established an inventory of certain telecommunications assets and services in use at the agency. However, as discussed earlier in this report, this inventory includes only the enterprise-wide assets and services in use at the agency; it does not include all of the assets and services that individual mission offices ordered for their own use. During our review, we asked NASA’s Office of the Chief Information Officer to provide an inventory of the assets and services ordered by the agency’s individual mission offices and NASA did not provide such an inventory. We maintain that NASA should have a complete inventory of all of its telecommunications assets and services in order to ensure that it is able to transition all services to EIS, as appropriate, before the current GSA telecommunications contracts expire. A complete inventory is also needed for the agency to be able to strategically plan for the transition, including fully identifying the agency’s future telecommunications needs and opportunities to optimize or share services by consolidating them on EIS. In addition, NASA described actions it has taken to address our recommendation calling for the agency to complete efforts to identify its future telecommunications needs using a complete inventory of existing telecommunications services. Specifically, the agency stated, among other things, that it (1) maintains an inventory of telecommunications services that are within the scope of the EIS program, and (2) continually identifies and plans for future NASA telecommunications needs using this inventory. However, as discussed earlier, NASA’s inventory of telecommunications assets and services is not complete because it does not include the assets and services ordered by the agency’s individual mission offices. Identifying NASA’s future telecommunications needs using a complete inventory of telecommunications services, as we recommended, would help to ensure that the agency fully identifies these needs. It would also reduce the likelihood that the agency may miss opportunities to optimize or share services by consolidating them on EIS. In written comments (reprinted in appendix VII), the Department of the Treasury offered additional information intended to clarify our findings regarding the agency’s compliance with GSA’s milestones to (1) release all EIS fair opportunity solicitations by March 31, 2019; (2) issue all EIS task orders by September 30, 2019; and (3) fully transition to EIS by September 30, 2022. In this regard, the agency stated that it had released four of its six EIS fair opportunity solicitations—which the agency said represented the majority of its telecommunications requirements—prior to GSA’s March 31, 2019, milestone; and had released its two other solicitations in July 2019. issued one of its six EIS task orders in September 2019, prior to GSA’s September 30, 2019, milestone and planned to issue its five remaining EIS task orders in March and April 2020. expected to transition all of its telecommunications services associated with its largest EIS solicitation by GSA’s milestone date of September 30, 2022. The agency stated that this solicitation is to provide enterprise managed services (e.g., voice and data services) for all Treasury bureaus except the Office of the Comptroller of the Currency. The agency also stated that it believes it will meet its transition goals for its other five solicitations. While the Department of the Treasury did not specify in its written comments a date for completing the transition of services associated with these other five solicitations, agency officials stated during our review that they planned to complete the transition to EIS in March 2023. The additional clarifications provided by the Department of the Treasury did not change our findings that the agency did not (1) meet GSA’s March 31, 2019, milestone to release all EIS fair opportunity solicitations; (2) meet GSA’s September 30, 2019, milestone to issue all EIS task orders; and (3) plan to fully transition to EIS by GSA’s September 30, 2022, milestone. Finally, 12 agencies responded that they did not have any comments on the report. Ten of these agencies responded via email: the Departments of Agriculture, Defense, Education, Energy, Homeland Security, the Interior, Justice, Labor, and Transportation; and the General Services Administration. Two agencies (the Department of Housing and Urban Development and the Social Security Administration) provided written responses, which are reprinted in appendices VIII and IX, respectively. We are sending copies of this report to the appropriate congressional committees, the Administrator of the General Services Administration, Administrator of the National Aeronautics and Space Administration, Secretary of Commerce, Secretary of Health and Human Services, Secretary of State, Secretary of Veterans Affairs, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Carol Harris at (202) 512-4456 or HarrisCC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix X. their plans for the transition to EIS, including the total number of fair opportunity solicitations and task orders planned; their planned schedules for transitioning to EIS contracts; and key factors that contributed to delays, if any, in meeting two critical transition milestones that GSA established for 2019—to (1) finish releasing all EIS fair opportunity solicitations by March 31, 2019, and (2) finish issuing all EIS task orders by September 30, 2019. After receiving the agencies’ survey responses, we electronically extracted the survey data and examined the results to identify missing data, inconsistencies, and other indications of error. We then addressed such issues, as necessary, including through follow-up communications with the selected agencies. In addition, due to the open-ended responses related to the key factors for delays, we conducted a content analysis of the responses we received in order to identify categories for the reported factors. We also interviewed relevant agency officials for further information regarding their agencies’ plans for transitioning to EIS. Further, for the seven agencies that reported in their survey responses that they planned to meet GSA’s milestone to finish issuing all EIS task orders by September 30, 2019, we asked those agencies in October 2019 to identify whether they actually met that milestone. One of the seven agencies reported that it met the milestone. For the six other agencies that did not meet the milestone, we asked them to identify the key factors that contributed to their delays in issuing the task orders. In November 2019, we also asked all of the 19 selected agencies to provide updated responses regarding their planned dates for fully transitioning to EIS contracts. To address the second objective, we selected for review a nongeneralizable subset of five agencies included in the first objective and assessed those agencies against activities associated with established transition planning practices. To select these five agencies from the 19 agencies included in our first objective, we first excluded the four Chief Financial Officers Act agencies that were included in our most recent prior review of agencies’ telecommunications transition planning efforts. We then used the telecommunications billing data provided by GSA to categorize the 15 remaining agencies based on the total charges billed to the agencies for fiscal year 2018. Specifically, in order to ensure that we would select agencies with different levels of telecommunications spending, we used the following three cost ranges to categorize the agencies as large, medium, or small: large – $100 million or more, medium – $25 million to less than $100 million, and small – less than $25 million. We also identified whether each agency had a centralized or decentralized structure related to its Chief Information Officer office. Further, we identified the number of fair opportunity EIS solicitations that each agency had released, as of October 31, 2018, and the total number of solicitations each agency planned to release, as reported on GSA’s website for tracking agencies’ EIS transition progress. Based on the above considerations, we selected five agencies that exhibited a variety of sizes and structures, and a range of planned and released fair opportunity EIS solicitations. The selected agencies were Commerce, HHS, NASA, State, and VA. Because we did not review a statistically representative sample of federal agencies, we could not conclude that our results represent the entire federal government’s level of preparation. However, the five cases we studied illustrate the levels of planning that these agencies had put into their transitions to EIS. We then obtained and reviewed relevant transition planning documentation from the agencies and assessed it against the following five telecommunications transition planning practices identified in our prior work: 1. develop an accurate inventory of telecommunications assets and services, 2. perform a strategic analysis of telecommunications requirements, 3. develop a structured transition management approach, 4. identify the resources needed for the transition, and 5. develop a transition plan. Specifically, for each of the agencies, we obtained and analyzed documentation, such as EIS transition plans; telecommunications inventories; telecommunications inventory maintenance documentation; EIS fair opportunity solicitations; documentation of strategic analyses completed while the agencies reviewed their telecommunications requirements (e.g., cost-benefit analyses of new technology and alternative options); program management documentation applicable to the transition, including program management plans, communications plans, cost estimates, integrated master schedules, risk logs, and oversight board briefing slides and meeting minutes; agency staffing plans for the EIS transition; and training completion documentation specific to the EIS transition. We also interviewed agency officials—including those that were responsible for managing their agencies’ transitions to EIS—regarding their agencies’ implementation of the established transition planning practices. Regarding our assessments of the agencies’ implementation of each of the activities associated with the five transition planning practices, we assessed an activity as “fully implemented” if agency officials provided evidence that they had implemented all of the aspects of the practice activity, or the agency had approved plans and related policies to fully implement the practice activity at a later time during the transition. We assessed an activity as “partially implemented” if agency officials provided evidence that they had implemented some, but not all, aspects of the practice activity. To assess the reliability of the fiscal year 2018 telecommunications billing data that we used to select the agencies for review, we reviewed the GSA-provided data to identify outliers, missing data, and other potential errors (e.g., components that were not associated with the correct agency). We also interviewed knowledgeable GSA officials about the reliability of the billing data provided. In addition, to assess the reliability of the agency-reported information we used to support the findings in this report, we reviewed relevant program documentation to substantiate evidence obtained through interviews with agency officials. For computer-processed data, such as the telecommunications inventories, we reviewed the data to identify outliers, missing data, and other potential errors; interviewed agency officials regarding the completeness and accuracy of the data; and reviewed related documentation, where available. For example, regarding the telecommunications inventories, we assessed agency documentation of the quality control procedures and practices related to ensuring the accuracy of the inventories. We also interviewed knowledgeable agency officials about the systems and processes in place to collect and verify the inventory data. Further, to determine if the agencies had established complete telecommunications inventories, we searched the data on USASpending.gov to identify the contractors that received telecommunications-related contracts from the selected agencies in fiscal years 2018 and 2019. We then compared the resulting list of contractors to those identified in the agencies’ inventories and, when the list of contractors identified did not match, we interviewed agency officials about the completeness of their inventories. We determined that the data used to select the agencies for review and to support the findings in this report were sufficiently reliable for the purposes of our reporting objectives, with the exception of agencies’ telecommunications inventories. Specifically, we determined that the inventory information provided by all five of the agencies was not reliable, due to the lack of documented procedures to ensure the completeness and accuracy of the data. This conclusion was considered during our assessment of the agencies’ efforts to implement the planning practice to develop an accurate inventory of telecommunications assets and services. We discuss limitations of these data in the report. We have also made appropriate attribution indicating the sources of the data. We conducted this performance audit from November 2018 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, the following staff made key contributions to this report: James R. Sweetman, Jr. (Assistant Director), Emily Kuhn (Analyst-in-Charge), James Brefo, Chris Businsky, Rebecca Eyler, Javier Irizarry, Amber McCants, and Andrew Stavisky.", "summary": "GSA is responsible for contracts that provide telecommunications services for federal agencies. In preparation for the expiration of current telecommunications programs, including one called Networx, GSA has developed a successor program, known as EIS. GSA and agencies now must carry out the task of successfully transitioning to EIS contracts. Previous contract transitions experienced significant delays. Those delays during the transition to Networx resulted in hundreds of millions of dollars in missed savings. GAO was asked to review agencies' EIS transition preparations. This report discusses (1) selected agencies' plans for, and status in, transitioning to EIS; and (2) the extent to which selected agencies were implementing established transition planning practices. GAO administered a survey to 19 selected agencies that spent at least $10 million on telecommunications in fiscal year 2018 regarding their plans for and status in transitioning to EIS. GAO also selected five of these agencies for further review—Commerce, HHS, NASA, State, and VA—based on, among other things, agency size and structure. For these agencies, GAO evaluated documentation to determine the extent to which they had implemented five planning practices identified in a previous GAO report. As of October 2019, the 19 selected agencies were in different stages of transitioning from their soon-to-be-expiring telecommunications contracts to the new Enterprise Infrastructure Solutions (EIS) program. All of these agencies reported that they plan to fully transition to EIS before current contracts expire in May 2023. However, 11 agencies did not plan to fully transition by the General Services Administration's (GSA) September 30, 2022, milestone. The majority of the selected agencies also did not meet GSA's milestones for completing critical contracting actions in 2019 (see table). While transitioning to EIS is a complex undertaking, delaying this transition will cause agencies to miss potential cost savings that would result from the generally lower rates for services on EIS. Five selected agencies—the Departments of Commerce (Commerce), Health and Human Services (HHS), State (State), and Veterans Affairs (VA); and the National Aeronautics and Space Administration (NASA)—had partially implemented established planning practices that can help agencies successfully transition their telecommunications services to new contracts. These practices are to: (1) develop an accurate inventory of telecommunications services, (2) perform a strategic analysis of telecommunications requirements, (3) develop a structured transition management approach, (4) identify the resources needed for the transition, and (5) develop a transition plan. The agencies provided several reasons for partially implementing the practices. For example, transition officials at Commerce, NASA, and VA said that they were not responsible for tracking all of the telecommunications services in use at their agencies; as such, they were unable to provide complete telecommunications inventories. The agencies also planned to implement certain practices after they issue their EIS task orders. However, the limited time remaining to complete the transition makes it critical that agencies conduct early planning with the information available and fully implement these transition planning practices to reduce the risk that the agencies experience the types of delays that occurred in previous transitions. GAO is making a total of 25 recommendations to Commerce, HHS, NASA, State, and VA, to fully implement the established transition planning practices. These agencies concurred with all of the recommendations.", "document_type": "gao"}
{"report": "State reviews and approves FMS purchases, while DOD is responsible for program implementation. DSCA administers the FMS program for DOD, including exercising financial management responsibilities for the FMS trust fund, and DFAS provides DSCA’s accounting services for FMS. Additionally, various other DOD components have responsibilities related to collecting and expending transportation funds, as shown in figure 1. Several DOD publications provide regulations and guidance for the FMS program, including: DOD Financial Management Regulation (FMR). Managed by the Under Secretary of Defense (Comptroller), the FMR defines financial management requirements for all DOD components, and states that DSCA administers the FMS program and is responsible for monitoring the use of the FMS trust fund. The FMR also states that DOD components should maintain documentation that constitutes a complete audit trail. Defense Transportation Regulation. Managed by the U.S. Transportation Command (TRANSCOM), the Defense Transportation Regulation defines requirements for the transportation of items within the Defense Transportation System, such as the use of unique identifiers for all shipments and how to use and pay commercial carriers, when applicable. Security Assistance Management Manual (SAMM). Managed by DSCA, SAMM provides guidance to the DOD components that manage or implement the FMS program. Foreign partners that purchase items and services through the FMS program may use their own funds or, if provided, U.S. funds, such as grants or loans provided through Foreign Military Financing. In addition, some FMS purchases are made using funds appropriated to DOD, State, or other U.S. government agencies for Building Partner Capacity (BPC) programs. These programs purchase items or services for foreign partners through FMS. The FMS process begins when an eligible entity requests information on defense articles or services for purchase. The responsible DOD component then prepares a Letter of Offer and Acceptance (LOA), which is the legal instrument used by the U.S. government to sell defense articles to a foreign country or international organization under authorities provided in the Arms Export Control Act. The LOA itemizes the defense articles or services offered and, when implemented, becomes an official tender by the U.S. government. Signed LOAs are referred to as “FMS cases,” and the individual items or services included for purchase in the FMS case are referred to as “case lines.” Once the LOA is signed, the DOD component responsible for the FMS case then manages the contracting or requisition of the equipment or services specified in the agreement, which are then delivered to the foreign partner. Foreign partners have different options available to them for transporting items they purchase through FMS. Other than when purchasing certain hazardous or sensitive items that must be transported via the Defense Transportation System, foreign partners have the option to arrange for their own transportation of FMS items they purchase—such as using a freight forwarder—for all or part of the transportation needed to reach the final destination. On the other hand, BPC programs use the Defense Transportation System to move all their FMS purchases. When all items have been delivered, all ordered services have been performed, and no new orders exist or are forthcoming, the DOD component responsible for managing the FMS case may mark it as closed. DOD most commonly calculates the FMS transportation fee using a percentage rate applied to the price of the item. The percentage rate varies depending on the extent of the U.S. government’s responsibility for transporting the items purchased, as agreed to between DOD and the foreign partner in the LOA. DOD first determines the estimated transportation fees for shipping FMS purchases based on the terms agreed upon in the LOA. When an item is shipped, the transportation fee is collected from the FMS purchaser’s account into the FMS transportation accounts. Eight transportation accounts within the FMS trust fund are used to hold transportation fees collected from FMS purchasers’ accounts, and to pay FMS transportation bills. Seven separate accounts hold transportation funds for certain larger BPC programs. These seven BPC accounts allow BPC program transportation fee collections and expenditures to be tracked. In addition, one main account holds transportation funds for all foreign partner purchasers and smaller BPC programs. Individual shipments trigger collections into and expenditures from the FMS transportation accounts. As shown in figure 2, after DOD ships an item and DFAS is notified of that shipment—through a process known as delivery reporting—DFAS moves the amount of the related transportation fee from the FMS country or BPC program account into the main transportation account or corresponding BPC program transportation account and records the amount as a collection. DFAS receives monthly bills that include the costs for FMS transportation, which DFAS pays out of the main transportation account, recording the amount paid as an expenditure. Subsequently, DFAS reviews the transportation bills and associated expenditure transaction data to identify any expenditures associated with the seven BPC programs with dedicated transportation accounts. For any BPC transactions identified, DFAS reimburses the main transportation account for the cost of the expenditure by moving funds from the relevant BPC transportation account into the main transportation account through a process DOD refers to as realignment. DOD and DSCA guidance to DOD components identifies controls over the information used to calculate the fees collected into the FMS transportation accounts. For example, DOD has various codes that identify the percentage rate used to calculate the transportation fee charged to FMS purchasers. DSCA provides guidance to components on how to use those codes, and components are responsible for managing the use of those codes. Both DSCA’s guidance and the FMR require that components maintain documentation, such as documentation of significant events related to delivery transactions and authorized exceptions to normal billing procedures. Additionally, both the FMR and DSCA’s guidance to components identifies that components are responsible for submitting delivery reporting within 30 days of completion, which triggers collection of the transportation fee. DSCA’s guidance to components requires components to perform various case reviews and reconciliations, including annual case reviews to verify the accuracy of information, such as the accuracy of the codes applied to case lines, as well as the timeliness of delivery reporting. DSCA’s guidance to DOD components requires components to review FMS cases at least annually to verify the accuracy of data—including the accuracy of the transportation fee collected and the timeliness of delivery reporting—but DSCA does not have a routine process to oversee those reviews. DSCA’s guidance to components also states that DSCA may request copies of components’ annual case reviews for oversight purposes, and DSCA officials told us that they request copies on an ad hoc basis. Although DSCA has oversight responsibility over collections into the FMS transportation accounts, DSCA officials said they do not have a standard process for selecting and examining components’ annual case reviews, and do not document their reviews. Federal internal control standards state that management should establish and implement activities to monitor internal control systems and evaluate results, and ensure that activities are performed routinely and consistently. Management may use ongoing monitoring, separate evaluations, or a combination of the two to obtain reasonable assurance of the operating effectiveness of the controls in place. Without routine oversight of components’ annual case reviews—which could include a process to select annual case reviews for examination, and guidance on how to perform and document examinations—DSCA increases the risk that components may not complete such reviews consistent with DSCA’s guidance, thereby increasing the likelihood that fees collected may be inaccurate. Additionally, according to DOD, the FMS program is intended to operate on a “no profit, no loss” basis, and inaccuracies in the collection of the FMS transportation fee could lead to over- or under- collecting fees from an FMS purchaser. DSCA officials told us that they have begun to work on an initiative to analyze a sample of annual case reviews on a routine basis. According to DSCA officials, the process will include reviewing and documenting cases based on certain events and is expected to be implemented in April 2020. The successful implementation of this initiative may help DSCA ensure that components’ annual case reviews comply with DSCA guidance. However, until DSCA fully implements this initiative, the risk remains that components may not complete annual case reviews consistent with DSCA’s guidance. DSCA does not have a process to monitor the timeliness of DOD components’ delivery reporting of shipments of items, which triggers collections into the FMS transportation accounts. According to DOD regulations, components are required to submit delivery reporting in their systems within 30 days of shipment. Although DSCA has financial responsibility over collections into the FMS transportation accounts, DSCA officials told us that they do not monitor components’ compliance with this regulation. Federal internal control standards state that management should design internal control activities to achieve control objectives and respond to risks, ensure the accurate and timely recording of transactions, and evaluate and document the results of ongoing monitoring activities. Further, the FMR incorporates the federal accounting standards into DOD accounting and financial reporting policy. The federal accounting standards state that revenue transactions—such as the FMS transportation fee—should be recorded when services are provided. DSCA officials told us that they rely on DFAS to monitor components’ delivery reporting. During the course of our review, DFAS officials told us that they began providing a report to DSCA and other components that detailed information on each components’ delivery reporting, which was based on a prior FMR requirement. Both DSCA and DFAS officials told us that they are working on an agreement that would formalize DFAS’s reporting, but have not finalized this agreement as of February 2020. However, DFAS officials told us that they have not followed up with components to verify the accuracy of the delivery reporting, and are not required to do so. While DSCA officials told us that DFAS’s reporting may help provide transparency, without a process to oversee that reporting, DSCA’s lack of monitoring of components’ delivery reporting raises the risk that such reporting may not be timely. As timeliness is an element of accuracy, untimely component delivery reporting may result in the inaccurate collection of related transportation fees into the FMS transportation accounts. A documented process to review reporting and monitor the timeliness of components’ delivery reporting—which could include DSCA’s review of DFAS’s reporting to identify and follow up on discrepancies—could help reduce the risk that transportation fees may not be collected into the FMS transportation accounts in a timely manner. Further, such oversight could assist components during other required reviews, such as annual case reviews. DSCA has limited financial oversight of expenditures from the FMS transportation accounts. While DSCA established internal guidance related to monthly reviews of expenditures from the accounts, that guidance lacks procedures to review expenditures and is not fully documented. In addition, DSCA has not provided guidance to DFAS on preparing the reports DSCA uses for its monthly review. Also, DFAS’s internal guidance on reviewing and realigning expenditures is inconsistent and lacks key controls and details, such as procedures to provide reasonable assurance that all transportation expenditures are reviewed. As a result, DSCA’s financial oversight of the FMS transportation accounts is insufficient to provide reasonable assurance that expenditures paid from the FMS transportation accounts are allowable and paid from the correct account, which limits DSCA’s ability to help ensure that relevant BPC program expenditures are paid from the related BPC accounts. During the course of our audits of the FMS program, DSCA officials told us that they began developing new internal guidance to address financial oversight of expenditures from the FMS transportation accounts, and expect it to be implemented by May 2020. However, until DSCA finalizes and implements that guidance, the risk remains that DSCA may use the FMS transportation accounts to pay for unallowable costs, or pay transportation costs from the incorrect account. In fiscal year 2016, DSCA established a Managers’ Internal Control Program to oversee the FMS transportation accounts, according to DSCA officials. This internal guidance identified the risk that DSCA may use the FMS transportation accounts to pay for unallowable costs—such as those not related to FMS transportation and that may be a result of misuse—or that DSCA may pay transportation costs from the incorrect account. To address these risks, the guidance identified procedures for DSCA to review expenditures. As shown in figure 3, the procedures state that DSCA will review expenditures from the FMS transportation accounts on a monthly basis to ensure costs are valid and applied to the proper account, and to identify and correct discrepancies. DSCA officials told us that to perform their monthly review they use reports provided by DFAS, and DFAS officials told us they provide those reports to DSCA on a monthly basis based on internal guidance. These reports include information on the FMS transportation account balances, and, in addition, DFAS provides supporting documentation that includes: copies of bills paid from the accounts and detailed analysis of individual transportation expenditures; an analysis of discrepancies DFAS identified for each transaction; and financial transactions DFAS performed to reimburse the main transportation account for specific BPC transportation expenditures, through a process DOD refers to as realignment. Both DSCA and DFAS officials said they use two pieces of information from the reports and analyses: The transportation account code, which identifies the DOD component responsible for a particular expenditure, and may provide information on the country or program associated with the transportation expenditure. The transportation control number, which is a unique 17-character code that is associated with a shipment and used throughout the Defense Transportation System for shipment tracking and payment processing. For FMS shipments, the transportation control number includes information that identifies the DOD component and foreign partner, and may be used to tie a particular transportation expenditure to an FMS case. The transportation account code and the transportation control number are entered by DOD components directly involved in ordering and processing shipments into their individual systems. Figure 4 provides additional details regarding the composition of the transportation control number. DSCA’s internal guidance does not contain procedures explaining how DSCA staff should review transportation expenditures. Federal internal control standards state that management should design control activities to respond to risks, implement activities that address those controls, and identify the information requirements needed to achieve objectives. DSCA’s monthly review of expenditures from the FMS transportation accounts is meant to provide financial oversight of the accounts, and DSCA’s internal guidance establishes that, as part of its monthly review, DSCA should review expenditures to ensure they are allowable and paid from the correct account, and follow up on any discrepancies. However, DSCA’s internal guidance does not explain how to review expenditures, and DSCA officials told us that they do not have internal guidance identifying the data needed to oversee expenditures or explaining how they should evaluate expenditure data, which could include steps such as identifying and correcting discrepancies including mismatched, missing, or incomplete entries. To assess transportation expenditure data reviewed by DFAS and DSCA, we analyzed a nongeneralizable sample of expenditure data for the FMS transportation accounts provided by DFAS for the period from May through July 2019. Over that 3-month period, DFAS reported about 6,200 transportation expenditures totaling approximately $21.6 million. Our review of those expenditures identified discrepancies or missing data such as transactions with: Mismatched DOD component codes. Approximately 19 percent of expenditures we examined—representing around $4 million—had transportation account codes and transportation control numbers identifying different DOD components. According to DFAS officials, if the transportation account code and transportation control number for an expenditure do not identify the same component, the mismatch may be a discrepancy. For example, a mismatch could indicate that staff at a component entered an incorrect transportation account code, or misapplied a transportation account code, which may result in the payment of a non-FMS expenditure from the FMS transportation account. Missing or misformatted control numbers. Approximately 3 percent of the number of expenditure transactions in our sample— representing around $40,000—either lacked transportation control numbers or included transportation control numbers that did not contain 17 digits. Without a valid transportation control number, DSCA may not be able to determine whether an expenditure is allowable or paid from the correct account. Because DSCA’s internal guidance lacks procedures—including those explaining what expenditure data is needed to perform oversight and how to evaluate that data for and address discrepancies—DSCA cannot provide reasonable assurance that it appropriately reviews expenditures from the FMS transportation accounts. Without guidance that addresses the risk that unallowable costs may be paid from the transportation account, DSCA raises the risk of misuse of funds of the FMS transportation accounts. Additionally, without guidance that identifies and addresses discrepancies—such as missing transportation control numbers—DSCA raises the risk that transportation expenditures may not be paid from the correct account. DSCA officials told us that they were developing new internal guidance and collaborating with DFAS on an initiative to follow up on discrepancies, and expect both to be implemented by May 2020. However, until DSCA finalizes and implements that guidance, the risk remains that DSCA may use the FMS transportation accounts to pay for unallowable costs, or pay transportation costs from the incorrect account. DSCA’s internal guidance does not identify how DSCA officials should document their monthly review of expenditures. Additionally, DSCA officials confirmed that they do not document their monthly review of expenditures. DSCA officials told us that while DSCA staff conducted regular monthly reviews, DSCA has not issued specific internal guidance explaining how staff should conduct and document these reviews. Federal internal control standards state that management should develop documentation of its internal control system, document internal control activities such as by documenting that activities occurred, and ensure that activities are performed routinely and consistently. Without internal guidance that identifies how staff should perform and document monthly reviews as well as a process to ensure reviews are documented, DSCA cannot provide reasonable assurance that staff perform monthly reviews consistently. DSCA officials told us that their planned internal guidance should address how the monthly review process is conducted, and should be implemented by July 2020. However, until DSCA finalizes and implements that guidance, DSCA will not have guidance on documenting its monthly review of expenditures consistent with federal internal control standards. DSCA officials have not provided written guidance to DFAS on preparing the reports and analyses DSCA uses for its monthly review, and, as a result, those reports and analyses may be inconsistent and incomplete. Federal internal control standards state that management should design control activities that respond to risks, document internal controls, communicate required information to external parties, and obtain relevant data from external sources based on information requirements. DFAS officials confirmed that they do not have written guidance from DSCA regarding how to generate the reports for DSCA, such as what analyses to perform on expenditure data. We found that DFAS’s analyses vary and lack key procedures. For example, our review of the expenditure data DSCA received from DFAS for May through July 2019 showed that DFAS performed certain analyses—such as verifying the validity of the transportation control number—on some transactions, but not on others. Additionally, DSCA did not provide DFAS with a complete list of transportation account codes to use to identify transactions for review. As a result, DSCA’s review of expenditures based on DFAS’s reports—both for allowability, as well as to ensure those transactions are paid from the correct account—excludes some transactions. Because DSCA has not provided written guidance to DFAS on how to generate the reports needed for its monthly review process—including what analysis to perform on expenditure data—or provided DFAS with the necessary transportation account codes, DSCA cannot provide reasonable assurance that all expenditures from the FMS transportation accounts are allowable and paid from the correct account. Additionally, the lack of consistent identification and review of all transactions by DSCA raises the risk of misuse of funds in the FMS transportation accounts. DSCA officials told us that they are developing guidance in coordination with DFAS, and that it should be implemented by July 2020. However, until DSCA finalizes that guidance to DFAS, DSCA may review inconsistent analyses and may not review all transportation expenditures, and the risk remains that DSCA may use the FMS transportation accounts to pay for unallowable costs, or pay transportation costs from the incorrect account. DFAS established procedures to review FMS transportation expenditures and to realign BPC expenditures to the correct FMS transportation accounts in part based on direction from DSCA. However, these procedures lack key steps to ensure that DFAS reviews all expenditures and identifies discrepancies, as well as to address discrepancies that may limit DFAS’s ability to identify transactions for realignment. Federal internal control standards state that management should design control activities to respond to risks, implement activities that address those controls, and ensure that activities are performed consistently. DFAS maintains a separate set of procedures for each of the three transportation service providers that submit FMS transportation bills. The results of DFAS’s procedures—such as how transportation expenditures were realigned—are included as supporting documentation for the monthly reports provided to DSCA. Our review of DFAS’s realignment procedures determined that the procedures are inconsistent or missing key steps that could help address the risk that expenditures paid from the FMS transportation accounts may be unallowable or paid from the incorrect account. We found that DFAS’s procedures do not ensure that DFAS reviews all transactions, including those that may require realignment. For example, DFAS’s procedures for DOD’s commercial transportation payment system—known as Syncada—do not include a step for reconciling the amount of the payment to the service provider against a list of detailed expenditure transactions, which may provide assurance that the list of transactions is complete. Specifically, for Syncada, DFAS queries the provider’s system using only nine transportation account codes provided by DSCA, which do not include any account codes associated with Navy, and only some associated with Air Force. Conversely, the realignment procedures for the Air Mobility Command and the Surface Deployment and Distribution Command include a step for reconciling the amount of the payment to the service provider against a list of detailed expenditure transactions, which helps to provide assurance that the list of transactions being reviewed is complete. Table 1 shows the results of our review of DFAS’s realignment procedures and analysis. Because DFAS’s procedures do not include steps to identify all Navy and Air Force transportation account codes, DFAS does not have reasonable assurance that all expenditures are reviewed by DFAS for realignment and provided to DSCA with the monthly report, which DSCA subsequently uses to review the validity of expenditures. As a result, any of these transactions that should be paid from a BPC transportation account are instead paid from the main FMS transportation account. As shown in figure 5, our review of Syncada expenditure data for May through July 2019 found that the use of DFAS’s procedures resulted in approximately 15 percent of expenditures not being reviewed. Those transactions represent 11 percent of the dollar value of transportation expenditures for that period, or approximately $392,000. In addition, we found that DFAS’s procedures do not address how to correct or follow up on discrepancies. Specifically, all three sets of realignment procedures state that analyzing the list of detailed transactions may identify transactions with discrepancies in their data, but none of the procedures fully address the types of discrepancies or their implications, such as if the expenditure does not include a transportation control number. Rather, according to DFAS officials, if DFAS identifies such a discrepancy with a specific expenditure, that cost remains as an expenditure from the main transportation account. Because DFAS’s procedures do not include steps to reconcile the amount of payments to all service providers against a list of detailed cost transactions or to identify all transportation expenditure transactions, DFAS may not review all FMS transportation expenditures and may not pay all expenditures from the correct transportation account. Additionally, because the list of transportation account codes provided to DFAS does not include all FMS account codes, neither DFAS nor DSCA review all expenditures from the FMS transportation accounts, which raises the risk of unallowable or unapproved expenditures. As a result, DSCA’s ability to provide reasonable assurance that all transportation expenditures are allowable and paid from the correct account is limited. DSCA officials told us that they are developing guidance in coordination with DFAS to identify and follow up on discrepancies and clarify how DFAS is to perform its analysis, and that the guidance should be implemented by July 2020. However, until DSCA finalizes that guidance to DFAS, DSCA may review inconsistent analyses and may not review all transportation expenditures, and the risk remains that DSCA may use the FMS transportation accounts to pay for unallowable costs, or pay transportation costs from the incorrect account. DSCA has developed financial oversight procedures for overseeing the billions of dollars that are collected into and expended from the FMS transportation accounts, but we found weaknesses in oversight of both collections and expenditures. Regarding collections, gaps in DSCA’s oversight of DOD components’ annual case reviews and delivery reporting increase the risk that transportation fees collected may be inaccurate. Similarly, regarding expenditures, we identified gaps in DSCA’s oversight. Specifically, DSCA has not established procedures for conducting monthly reviews of expenditures and correcting discrepancies, or defined the information it needs from DFAS. Further, DFAS’s procedures to review and realign costs between FMS transportation accounts—which are based on guidance from DSCA—do not ensure that all transactions are included. By improving financial oversight of the FMS transportation accounts, DSCA could better ensure the accuracy of fees collected and help provide reasonable assurance that expenditures are allowable and paid from the correct account. DSCA officials told us that they are developing guidance to address these issues, and plan to implement that guidance in 2020. However, until DSCA finalizes and implements that guidance, the risks remain that DSCA may collect inaccurate transportation fees, use the FMS transportation accounts to pay for unallowable costs, or pay transportation costs from the incorrect account. We are making the following five recommendations to DOD: The Secretary of Defense should ensure that the Director of DSCA implements the planned initiative to routinely examine annual case reviews performed by DOD components to help ensure that fees collected into the FMS transportation accounts are accurate. (Recommendation 1) The Secretary of Defense should ensure that the Director of DSCA works with DFAS and DOD components to establish a written process to monitor the timeliness of components’ delivery reporting to help ensure that fees collected into the FMS transportation accounts are accurate. (Recommendation 2) The Secretary of Defense should ensure that the Director of DSCA finalizes and implements internal guidance on how to conduct and document DSCA’s monthly review of expenditures from the FMS transportation accounts, including what information should be reviewed and how to identify and follow up on discrepancies. (Recommendation 3) The Secretary of Defense should ensure that the Director of DSCA works with DFAS to finalize written guidance to DFAS on how to generate the reports needed for DSCA’s monthly review of expenditures from the FMS transportation accounts, including the type of analysis needed. (Recommendation 4) The Secretary of Defense should ensure that the Director of DSCA works with DFAS and other DOD components to finalize the planned guidance to DFAS for the review and realignment of expenditures from the FMS transportation accounts to ensure reviews are consistent and include all expenditures. (Recommendation 5) We provided a draft of this report to DOD and State for review and comment. DSCA provided written comments on behalf of DOD, which are reprinted in appendix II. DSCA concurred with all of our recommendations, and indicated that it had developed plans to address them and had begun implementing some of those plans. DOD noted that annual case reviews and delivery reporting are not directly related to financial transactions tied to the FMS transportation account. However, both annual case reviews and delivery reporting provide an opportunity for oversight that can help verify the accuracy of data, which affects the accuracy of transportation fees collected from FMS purchasers’ accounts into the FMS transportation accounts. We also received technical comments from DOD, which we incorporated in our report as appropriate. State did not provide any written or technical comments. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of State, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6881 or BairJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines (1) the Defense Security Cooperation Agency’s (DSCA) oversight of Department of Defense (DOD) components’ activities that affect fees collected into the Foreign Military Sales (FMS) transportation accounts, and (2) DSCA’s financial oversight of expenditures from the FMS transportation accounts. To obtain information on both of our objectives, we reviewed DSCA’s guidance related to the FMS program and FMS transportation, and analyzed fiscal years 2007 to 2018 summary collections and expenditures data for the FMS transportation accounts maintained by the Defense Finance and Accounting Service (DFAS) in the Defense Integrated Financial System. We chose to review data from these fiscal years based on data availability. Under a prior review of the management oversight of FMS transportation fees, we assessed the reliability of these data by reviewing for duplicate entries, gaps, and obvious errors, comparing the data to similar data obtained under prior reviews, and interviewing agency officials to clarify questions about how to interpret the data. On the basis of this assessment, we determined these data to be reliable for the purposes of summarizing the total collections into and expenditures from the FMS transportation accounts during fiscal years 2007 to 2018. To examine DSCA’s oversight of DOD components’ activities that affect fees collected into the FMS transportation accounts, we reviewed DOD’s current guidance related to the FMS transportation fee, as well as other documentation and internal guidance developed by DSCA. We interviewed DSCA and DFAS officials on their implementation of oversight procedures. To determine and assess the controls DSCA should be using to manage and oversee the account, we reviewed DOD’s Financial Management Regulation, DSCA’s Security Assistance Management Manual, other internal DSCA guidance, federal accounting standards, federal internal control standards, and our prior report on DSCA’s management oversight of the FMS transportation account balances. To examine DSCA’s financial oversight of expenditures from the FMS transportation accounts, we reviewed DOD’s current regulations related to financial oversight and transportation, including DOD Financial Management Regulation and DOD Defense Transportation Regulation. Additionally, we reviewed DSCA’s Managers’ Internal Control Program procedures for monthly FMS transportation account reviews, and we interviewed DSCA officials responsible for these reviews. We also reviewed DSCA’s Security Assistance Management Manual, which provides guidance to DOD components related to the FMS program, and DFAS’s internal guidance on reviewing and realigning expenditures from the FMS transportation account. We analyzed a nongeneralizable, 3-month sample of expenditure data for the FMS transportation accounts provided by DFAS for the period from May through July 2019, including transportation service provider bills and detailed transaction-level expenditures. These data included the transportation account codes and transportation control numbers that DSCA and DFAS use to verify that individual expenses are allowable, and to realign transportation expenditures to the correct FMS transportation account. We initially obtained 1 month of transportation expenditure data, but decided to expand our analysis to 3 months of data to account for any variability between months. Additionally, we chose to review data from this period because they were the most current at the time of our request, and therefore the data were compiled using DFAS’s current process. We determined this period to be sufficient for our analysis of the data, which DSCA and DFAS use to provide assurance that expenditures from the FMS transportation accounts are allowable and paid from the correct account. To assess the reliability of these data, we reviewed the data for internal consistency by reviewing for duplicate entries, gaps, and obvious errors; compared them to DOD regulations on transportation account code and transportation control number construction; and interviewed DSCA and DFAS officials about their data collection and verification procedures. We found the data to be sufficiently reliable for our purpose of presenting the total number and dollar amount of transportation expenditures reviewed by DSCA and DFAS for each month, and to identify the number and dollar amount associated with expenditure records where we identified discrepancies. We found instances of blank or incorrectly formatted transportation account codes and transportation control numbers, and instances where the first characters of the transportation account code and transportation control number did not match, which DFAS officials identified as possible discrepancies in the data. As we discuss in the report, these instances raise questions about the reliability of the data for financial oversight, since DSCA and DFAS use this information to ensure that expenditures are allowable and paid from the correct FMS transportation accounts. We did not conduct any independent testing of the data to determine whether the amounts reflected correct payments made toward accurate billings. To review DFAS’s internal guidance for reviewing and realigning expenditures from the FMS transportation account, we reviewed copies of the procedures provided by DFAS for each of the three transportation service providers. We identified major steps in the procedures related to reviewing expenditure data, as well as various internal controls related to analyzing data. We reviewed each of the procedures against one another to determine the extent to which they addressed the same elements, and we compared relevant procedures against standards for internal control related to obtaining, evaluating, and correcting data, to determine whether they were sufficient to provide financial oversight. Additionally, we interviewed DFAS officials responsible for these procedures. In order to determine whether DFAS’s procedures included all expenditures, we requested and obtained information from Army, Navy, and Air Force on the transportation account codes that each component used in fiscal year 2019, and compared them to the account codes provided to DFAS by DSCA and used to query the third-party transportation service provider system for relevant FMS transportation expenses. We identified a list of transportation account codes not queried as part of DFAS’s procedures, and we requested that DFAS query the third-party transportation service provider’s system for the period of May through July 2019 using that list. We requested data from this period to be consistent with the expenditure data DFAS initially provided us for the same period. We reviewed the resulting data and compared it to the previously-provided data in order to determine the relative size of each data set for this period. We did not independently test to determine whether the lists of transportation account codes provided to us were complete, and therefore the data reviewed may not include all relevant transportation expenditures. We conducted this performance audit from May 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Jason Bair, (202) 512-6881 or BairJ@gao.gov In addition to the contact named above, Cheryl Goodman (Assistant Director), Benjamin L. Sponholtz (Analyst-in-Charge), Adam Peterson, and Heather Rasmussen made key contributions to this report. Martin de Alteriis, John Hussey, Christopher Keblitis, Heather Latta, and Grace Lui also contributed to this report.", "summary": "From fiscal years 2007 to 2018, DOD collected about $2.3 billion in fees into the FMS transportation accounts and expended about $1.9 billion from the accounts. Foreign partners can pay DOD a fee to cover the costs of DOD transporting items. Fees are collected into transportation accounts in the FMS Trust Fund, and expenditures for related transportation are paid from those accounts. DSCA is responsible for financial oversight of the accounts, and DFAS—a service provider to DSCA—also has some accounting responsibilities related to the accounts. House Report 114-537 and Senate Report 114-255 included provisions that GAO review DSCA's management of FMS fees. This report examines (1) DSCA's oversight of DOD components' activities that affect fees collected into the FMS transportation accounts, and (2) DSCA's financial oversight of expenditures from the FMS transportation accounts. GAO reviewed DOD guidance, analyzed 3 months of DOD expenditure data, and interviewed DOD officials. The Foreign Military Sales (FMS) program is one of the primary ways the U.S. government supports its foreign partners, by annually selling them billions of dollars of military equipment and services. However, gaps in the Defense Security Cooperation Agency's (DSCA) oversight of Department of Defense (DOD) components' activities increase the risk that fees collected into the FMS transportation accounts may be inaccurate. While DSCA requires components to perform annual reviews of FMS cases to verify the accuracy of transportation fees collected, DSCA does not routinely oversee these reviews. Additionally, DSCA lacks oversight of the timeliness of DOD components' reporting of deliveries, which should occur within 30 days. DSCA officials indicated that they are developing guidance and processes to help address these challenges, but had not completed them as of February 2020. DSCA's financial oversight of expenditures from the FMS transportation accounts does not provide reasonable assurance that expenditures are allowable and paid from the correct account. In fiscal year 2016, DSCA established internal guidance for financial oversight of expenditures from the accounts. While that guidance includes a process to review expenditures on a monthly basis, DSCA has not established procedures for conducting that review, including how to analyze expenditure data, or identify and address discrepancies. As a result, DSCA may not review FMS transportation expenditures consistently or identify and address discrepancies. GAO found that approximately 19 percent of expenditures reported to DSCA over a 3-month period in fiscal year 2019 inconsistently identified the DOD component responsible for the transaction. For example, a transaction may indicate that both Navy and Air Force are responsible for the shipment. Further, DSCA has not documented how the Defense Finance and Accounting Service (DFAS) should generate the reports DSCA uses for its review, and DFAS's review of expenditures excludes some expenditures from two DOD components. Without a routine process to review expenditures and correct discrepancies, DSCA cannot provide reasonable assurance that all expenditures are allowable and paid from the correct account, raising the risk of misuse of funds. DSCA officials told GAO that they are developing guidance to help address these challenges, and expect to implement it in 2020. GAO is making five recommendations to DOD to strengthen financial oversight of the FMS transportation accounts, including two recommendations to strengthen DSCA's oversight of fees collected into the accounts, and three recommendations to strengthen DSCA's and DFAS's oversight of expenditures from the transportation accounts. DOD concurred with all of the recommendations and identified actions it plans to take to address them.", "document_type": "gao"}
{"report": "Federal agencies conduct a variety of activities related to animal research. These activities include: funding intramural research conducted by agency personnel at federal facilities; funding extramural research conducted by universities, industrial firms, and other nonfederal entities through contracts, grants, or cooperative agreements; establishing guidelines for regulating products that may have been tested for safety or efficacy using animals; and overseeing the welfare of animals used for research. Table 1 shows key activities related to animal research or use that HHS, USDA and EPA and their component agencies and offices are involved in through their funding of intramural and extramural research and regulation of products. Some agencies within USDA and HHS have roles in overseeing animal welfare at federal and nonfederal research facilities. Under the Animal Welfare Act and its implementing regulations, USDA’s APHIS oversees federal and nonfederal research facilities to ensure the humane treatment of covered species of warm-blooded animals when they are used in research, teaching, testing, or experimentation. These species include dogs, cats, nonhuman primates, guinea pigs, hamsters, rabbits, horses used for research purposes, and (with certain exceptions) other warm- blooded animals. Requirements under the act related to consideration of alternatives to animal research include the following: Under the act, research facilities are to appoint an animal care and use committee to, at least semiannually, review the facility’s program for humane care and use of covered animals, inspect all facilities, and prepare reports of its evaluation. The committee is responsible for reviewing research proposals to determine whether the proposed activities are in accordance with the act. This includes a review of research proposals to determine whether researchers have (1) considered alternatives to procedures that may cause more than momentary or slight pain or distress to covered animals and (2) have provided a written narrative description of the methods and sources they used to determine that alternatives were not available. The committee can ask a researcher to explain why any alternatives found are not used in the researcher’s proposal or withhold approval of the proposal. Facilities that used or intended to use live covered animals in research are to submit a retrospective annual report about those animals to APHIS on or before December 1 of each calendar year. In particular, the annual reports are to include an assurance that each researcher considered alternatives to painful procedures. HHS’s NIH is responsible for establishing guidelines implementing certain provisions of the Health Research Extension Act of 1985. NIH’s responsibilities under the act include reviewing federal and nonfederal research facilities’ vertebrate animal care and use programs to determine whether they meet relevant standards and are thereby eligible to receive funding from HHS agencies covered by the act, including NIH. NIH implements the animal care provisions of the act through its Public Health Service Policy. The policy’s requirements related to research facilities’ consideration of alternatives include the following: Consistent with the act, NIH’s Public Health Service Policy directs facilities to provide for NIH’s approval a document that describes their vertebrate animal care and use program and that provides assurances that the research institution meets applicable standards. Such assurances must include a synopsis of training or instruction in research or testing methods that minimize the number of vertebrate animals required to obtain valid results and minimize animal distress and that the facility offers to scientists, animal technicians, and other personnel involved in animal care, treatment, or use. As a condition of receiving funding for animal research from HHS agencies, facilities must, for the most part, adhere to the eighth edition of the Guide for the Care and Use of Laboratory Animals (Guide). The Guide states that in preparing and reviewing research protocols, researchers and animal care and use committees should consider the availability or appropriateness of using less invasive procedures, other species, isolated organ preparation, cell or tissue culture, or computer simulation. The Guide does not limit this to research that is painful or distressful. NIH conducts site visits at selected research facilities to assess compliance with the act. Whereas the Animal Welfare Act applies to certain warm-blooded animals, the definition of animals used for the purposes of the Health Research Extension Act covers all vertebrates, including the mice, rats, and fish species commonly used in laboratory research. Other laws relevant to the consideration of alternatives include the following: The National Institutes of Health Revitalization Act of 1993 directs the Director of NIH to prepare a plan to conduct or support research into methods of biomedical research and experimentation that do not require the use of animals, that reduce the number of animals used in such research, and that produce less pain and distress in such animals. The act also directs NIH to prepare a plan for establishing the validity and reliability of the new methods it develops, encouraging the scientific community’s acceptance of these methods, and training scientists in using such methods. The act further directs NIH to periodically review this plan and, as appropriate, make revisions and include those revisions in a biennial report. In response to the act, in September 1994 NIH established ICCVAM as an ad hoc committee. The ICCVAM Authorization Act of 2000 directed NIH to establish the ICCVAM as a permanent interagency committee under NIH’s National Toxicology Program Interagency Center for the Evaluation of Alternative Toxicological Methods. ICCVAM is administered by the National Institute of Environmental Health Sciences. The act specifies that ICCVAM be composed of the heads (or their designees) of 15 agencies or subagencies, including EPA, agencies within HHS, and USDA. The National Institutes of Standards and Technology joined voluntarily in 2016. The act directed ICCVAM to, among other things, review and evaluate alternative test methods that may be acceptable for specific regulatory uses and to prepare biennial progress reports. Under the act, an alternative test method is one that reduces the number of animals required; refines procedures to lessen or eliminate pain or distress to animals or enhances animal well-being; or replaces animals with non- animal systems or one animal species with a species presumed to have less ability to feel pain, such as replacing a mammal with an invertebrate. In January 2018, ICCVAM published a strategic roadmap articulating its vision to meet its purpose. The Frank R. Lautenberg Chemical Safety for the 21st Century Act amended the Toxic Substance Control Act in 2016 to include language on the use of alternative methods. The act directs the Administrator of EPA to reduce and replace, to the extent practicable, scientifically justified, and consistent with the policies of the Toxic Substances Control Act, the use of vertebrate animals in the testing of chemical substances or mixtures under the Toxic Substances Control Act. The act also directs the Administrator to develop a strategic plan to promote the development and implementation of alternative test methods and strategies to reduce, refine, or replace vertebrate animal testing. Methods HHS, USDA, and EPA have used to ensure that researchers consider alternatives to animal research include requiring researchers to describe and document their consideration of alternatives. In addition, USDA’s APIHS and HHS’s NIH help ensure that researchers consider alternatives by overseeing research facilities and these facilities’ animal care and use committees, including the committees’ review of animal research protocols. USDA and NIH also provide training to researchers and animal care and use committees to help ensure researchers have considered alternatives. For research that they conduct or fund, component agencies and offices within HHS, USDA, and EPA call for individual researchers to describe their consideration of alternatives to animal research. USDA’s APHIS and HHS’s NIH require research facilities to consider alternatives through the agencies’ implementation of the Animal Welfare Act regulations and Public Health Service Policy, respectively. EPA research is covered by the two laws to the extent that it uses animals covered by the Animal Welfare Act or Health Research Extension Act. Table 3 summarizes the factors that determine whether researchers are required under the acts to consider alternatives. The steps HHS, USDA, and EPA take to help ensure that agency researchers and the researchers that they fund or oversee meet the requirement to consider alternatives include (1) calling for written descriptions of researchers’ consideration of alternatives and (2) prescribing or recommending that researchers use searches, such as of databases of published scientific literature, to identify alternatives. Call for written descriptions. As specified in the Animal Welfare Act regulations and Public Health Service Policy, HHS, USDA, and EPA call for researchers to send written descriptions of research projects involving animals to animal care and use committees for their review and approval. In particular, the Animal Welfare Act regulations require these committees to determine that researchers have provided a written narrative description of the methods and sources they used to determine that alternatives were not available. The Public Health Service Policy requires that researchers’ institutions submit written descriptions of research projects to the committees and for the committees to determine that researchers’ procedures avoid or minimize discomfort, distress, and pain to animals, consistent with sound research design, among other things, and that researchers follow the U.S. Government Principles for the Utilization and Care of Vertebrate Animals Used in Testing, Research, and Training. The principles require the consideration of alternatives. In our review of protocol forms from our sample of 12 research facilities (including HHS, USDA, and EPA facilities), we found that all of the forms requested information on researchers’ consideration of alternatives, though the forms varied in the particular information they requested. The types of information requested included a rationale for involving animals and for the number of animals to be used, assurance that research activities do not unnecessarily duplicate previous experiments, and a description of the methods and sources used to determine that alternatives were not available. Several of the protocol forms required researchers to identify alternatives considered but not adopted. Recommended method for identifying alternatives. In their implementation of the Animal Welfare Act and Health Research Extension Act, respectively, USDA and NIH consider database searches as a best practice for researchers using animals covered by the acts to identify and consider alternatives to animal testing. A database search involves a researcher using keywords related to the planned use of animals to query citations in databases of published scientific literature. From April 1997 through July 2018, USDA maintained a policy, known as Animal Care Policy #12, in its animal care policy manual. In Policy #12, the agency recommended a database search as the most effective and efficient method for demonstrating compliance with the requirement to consider alternatives to painful or distressful procedures. According to USDA’s Deputy Administrator responsible for implementation of the Animal Welfare Act regulations, in July 2018, USDA placed the policy in inoperative status after determining that some research facilities and agency inspectors had misinterpreted the policy as a requirement. Moreover, in response to the 21st Century Cures Act, USDA is reviewing its animal care policy manual, including Policy #12, to ensure the policies in the manual conform with the Animal Welfare Act and its implementing regulations, harmonize with NIH guidance, and reduce researcher burden where possible. According to the Deputy Administrator, as of June 2019, USDA had not decided what, if anything, it would do to revise or replace Policy #12. According to a draft interagency report in response to the 21st Century Cures Act, USDA will make any revised and future policies involving the use of animals available for public comment using regulations.gov or a similar service. However, according to the Deputy Administrator, even though Policy #12 is inoperative, USDA continues to advocate for database searches, particularly through the USDA Animal Welfare Information Center’s provision of information to the scientific community about how to search for alternatives. According to a senior NIH official, NIH requires that agency researchers conduct database searches. Also, in a sample animal study proposal form NIH has provided to animal care and use committees, NIH recommends that researchers at other facilities conduct database searches. Furthermore, 11 of the 12 research facilities we reviewed (including HHS, USDA, and EPA facilities) used research protocol forms that required or recommended that their researchers conduct a database search for alternatives to animal research. Agencies may apply additional requirements to individual researchers at their own facilities or through grants they fund, in addition to applying the requirements of the Animal Welfare Act and Health Research Extension Act. For example, CDC’s Fort Collins, Colorado, facility requires researchers to provide assurance on their protocol forms that the facility’s animal care and use committee’s statistician reviewed the form to determine whether the research would use an appropriate number of animals or explain why a review of the number of animals did not occur. Similarly, the Chairman of APHIS’s National Wildlife Research Center committee told us that its animal care and use committee includes a biostatistician who conducts an analysis to ensure that the numbers of animals to be used will produce statistically significant results. USDA’s National Institute for Food and Agriculture requires applicants for funding from the agency’s Agriculture and Food Research Initiative Competitive Grants Program to use statistical power analysis, when appropriate, to determine the sample sizes of animals to be used in research. USDA officials told us that this type of analysis provides a justification for the number of animals needed to provide valid results and helps prevent the unnecessary use of animals. Agencies may also require information on animal use in proposals submitted by extramural researchers. For example, NIH instructs researchers to describe the use of animals in their work in a section of grant applications, contract proposals, and cooperative agreements. Specifically, when submitting a proposal, researchers must justify to agency officials and other reviewers that the species used is appropriate for the proposed research and explain why research goals cannot be accomplished using an alternative model, such as computational, human, invertebrate, or in vitro models. APHIS and NIH help ensure that researchers consider alternatives through the agencies’ oversight of research facilities and these facilities’ animal care and use committees, including the committees’ review of animal research protocols. In particular, APHIS collects and reviews annual reports from federal and nonfederal research facilities in which the facilities are required to provide an assurance that researchers considered alternatives. The Animal Welfare Act requires APHIS to annually inspect nonfederal research facilities to determine whether the facilities are in compliance with the act. As part of a facility inspection, APHIS inspectors are to examine whether researchers have met the requirement to consider alternatives to any procedure likely to produce pain in or distress to species of animals covered by the act. According to APHIS officials, inspectors examine a sample of approved animal research protocols to check whether the protocol forms include a written narrative on the consideration of alternatives and to ensure that the facility’s animal care and use committee approved the protocol forms. The inspectors may issue citations of noncompliance if they find inadequate documentation that researchers associated with one or more protocols considered alternatives to procedures that may cause more than momentary or slight pain or distress to animals. APHIS provided us with inspection reports for fiscal years 2015 through 2018 in which inspectors issued 57 citations to research facilities for noncompliance with the Animal Welfare Act regulations that require researchers to consider alternatives to animals or issued “teachable moments.” The inspection reports included some citations that, according to APHIS officials, were incorrectly issued because inspectors interpreted the Policy #12 recommendations on database searches as requirements. In addition, NIH’s Office of Laboratory Animal Welfare is responsible for the general administration and coordination of the Public Health Service Policy and provides specific guidance, instruction, and materials to research facilities that receive funding from agencies covered by the act. For all such facilities, NIH is to review the facilities’ assurance documents describing their animal care and use programs. In particular, the Animal Welfare Assurance document is to describe the procedures—including review of animal research protocols—that the animal care and use committees follow to fulfill the directives of the NIH Public Health Service Policy. Further, NIH conducts site visits at a small number of facilities. The Public Health Service Policy states that each awardee institution is subject to review at any time by agency staff and advisors to assess the adequacy and accuracy of the institution’s compliance or expressed compliance with the policy, and this review may include a site visit. According to NIH officials, when agency staff conduct site visits, they examine the facility’s protocol form to confirm that its animal care and use committee requests information from researchers about their consideration of alternatives. NIH officials may also examine a sample of approved protocol forms during a site visit. According to NIH officials, the Office of Laboratory Animal Welfare conducted 38 site visits in fiscal years 2015 through 2018 and found one deficiency related to the consideration of alternatives. USDA and NIH have provided training to researchers and animal care and use committee members on the requirements of the Animal Welfare Act and the Health Research Extension Act. The training has addressed, among other things, the requirement to consider alternatives and has included advice on how to search for alternatives. Through its Animal Welfare Information Center, USDA provides training on how to conduct database searches for alternatives to animal research and assists individual researchers with their literature searches. According to USDA staff, the information center provides three workshops per year on meeting the requirements of the Animal Welfare Act, each lasting a day and a half. The workshops are open to anyone working with animals in research, including scientists, veterinarians, librarians, and animal care and use committee members. The center also gives workshops upon request at specific facilities. Additionally, the center’s website contains resources for conducting literature searches, and, according to a senior information center official, the center plans to put workshops into an online format that will be available upon demand. According to the official, the center conducted 137 database searches upon request in fiscal years 2014 through 2018. NIH has also provided training on the consideration of alternatives to help researchers meet their requirements under the Health Research Extension Act. For example, in 2014 NIH presented a webinar on searches for alternatives. The webinar, titled Meeting Requirements for Alternatives Searches and available on NIH’s website, provides advice on how to conduct database searches. For example, the webinar provided advice on the timing of the search, the search strategy, and particular databases to use. In September 2015, NIH presented a webinar demonstrating how to use NIH’s database of research projects to find researchers, projects, and publications that may help replace, reduce, and refine the use of animals in research. The NIH Office of Laboratory Animal Welfare provides on its website a sample animal study protocol form that emphasizes database searches for any procedures that cause more than momentary or slight pain or distress to the animals. In addition, according to a senior official from the NIH office overseeing the agencies’ intramural research using animals, researchers at NIH must complete an online course regarding animal use every 3 years. The course includes a section on replacing, reducing, and refining animal use and outlines how researchers are to report literature searches in order to show they considered alternatives. EPA, HHS, and USDA have facilitated the development, use, and promotion of alternative research methods through individual and collaborative efforts, including strategies for promoting the use of alternative methods and development of policies and guidance on alternative methods. The three agencies have also developed alternative research methods that rely on non-animal models and procedures to test how various products would affect humans. Additionally, the agencies have worked collaboratively with each other and with nonfederal stakeholders to promote alternative methods, in particular through ICCVAM, which is required to report to the public on its progress. However, ICCVAM and its member agencies have not routinely developed or reported metrics for assessing the effect that their efforts are having on animal use. EPA has issued a strategic plan and FDA has issued a roadmap for the use of methods that may reduce animal use in assessments of the safety and efficacy of various products. Both agencies and others within HHS and USDA have also issued guidance on using alternatives to animal research in particular contexts, such as vaccine testing. In June 2018, EPA’s Office of Chemical Safety and Pollution Prevention issued a strategic plan for the reduction of vertebrate animal testing for toxic chemicals regulated under the Toxic Substances Control Act. The office developed and issued this strategic plan to implement a provision in the Frank R. Lautenberg Chemical Safety for the 21st Century Act (Lautenberg Act) calling for such a plan. The strategic plan describes a multi-year process with incremental steps for adopting and integrating methods that do not use vertebrate animals in evaluating chemicals regulated by the Toxic Substances Control Act for their effect on human health and the environment. The strategic plan states that the agency’s long-term goal is to reduce and eventually eliminate vertebrate animal testing for chemicals regulated under the act. Pursuant to the strategic plan, in June 2018 EPA published a list of methods the agency had identified that require no vertebrate testing and that are capable of providing information of equivalent or better scientific reliability and quality than that which would be obtained from vertebrate animal testing. According to EPA, the agency plans to update the list at least once a year. EPA’s strategic plan calls for other near-term activities such as retrospectively identifying and evaluating the studies that it has requested and received for both new and existing chemicals. The plan states that EPA will complete this analysis in 2019 and use the results to support the future development of alternative methods to fit the agency’s needs. In May 2011, EPA’s Office of Chemical Safety and Pollution Prevention issued a strategic plan in response to a 2007 National Academies report calling for a more efficient and informative risk assessment process to predict and characterize potential human health and environmental hazards from exposures to pesticides. The office’s strategic plan envisions using a combination of computational and predictive modeling approaches, in vitro techniques, and targeted in vivo testing to supplement or replace the existing toxicity tests required in federal regulations for pesticide registration under the Federal Insecticide, Fungicide, and Rodenticide Act. Pursuant to this strategic plan, EPA’s Office of Chemical Safety and Pollution Prevention has issued guidance on data requirements for assessing pesticide safety that may reduce animal use. For example, EPA issued guidance in May 2013 on the data that the agency needs in order to adequately assess pesticide risks. The guidance also provided information to manufacturers on how to request waivers from the data requirements, which would enable the manufacturers to reduce animal use. EPA also issued guidance in November 2016 that allows pesticide manufacturers to request a waiver from the requirement to provide data on acute toxicity tests and that contains a policy statement waiving all acute lethality dermal studies for formulated pesticide products; such waivers can reduce the need for pesticide manufacturers to conduct tests using animals. For example, EPA reported granting a total of 223 waivers in fiscal years 2016 and 2017, pursuant to the agency’s May 2013 guidance for toxicity studies, which the agency estimated avoided the use of 85,000 animals and saved pesticide manufacturers $26.4 million in conducting toxicity studies. In February 2016, EPA announced an effort to evaluate and implement alternative methods for tests involving acute oral, dermal, and inhalation toxicity; skin and eye irritation; and skin sensitization. As part of this effort, in April 2018, EPA issued a draft policy to reduce the use of animals in testing chemicals to evaluate whether they cause an allergic reaction, inflammation, or sensitization of the skin. EPA’s policy describes conditions under which the Office of Chemical Safety and Pollution Prevention will accept alternative approaches to laboratory animal studies for identifying skin sensitization hazards. In December 2017, in response to direction from the FDA Commissioner, FDA developed a roadmap to foster the development and evaluation of emerging tools and methods that can improve toxicology methods for assessing the safety of FDA-regulated products. The roadmap does not have an explicit goal to replace, reduce, or refine animal testing but states that new methods may have the potential to do so. In that regard, the roadmap states that FDA will encourage medical product sponsors to submit a scientifically valid approach for using a new method early in the regulatory process and to engage in frequent communication with the agency about the suitability of that method. In addition, the roadmap recommended that FDA establish an organizing committee; conduct training; foster communication and collaboration with stakeholders, such as industry and academia; engage in research; and track and report annually on its progress. In June 2019, FDA posted its first annual report on its progress in implementing the roadmap. Previously, FDA had taken steps to reduce animal use by issuing guidance to members of industry seeking approval for FDA-regulated products. In general, FDA guidance states that industry may choose to use an approach—such as a non-animal testing method—other than one set forth in guidance as long as it complies with relevant statutes and regulations. FDA has also taken more specific steps to modify guidance to promote the use of alternative methods. For example, in 2012, FDA issued guidance to industry that states that firms may use non-animal alternative methods to test the toxicological safety of pharmaceutical drugs if the methods are appropriate or scientifically justified. In 2013, FDA issued guidance that, among other things, allowed industry to use in vitro assays rather than mice to detect toxins in shellfish meant for human consumption; this guidance subsequently played a role in the adoption of additional methods that do not employ animal use. Other HHS and USDA agencies within our scope do not have strategic plans or roadmaps that promote a comprehensive strategy for alternative research methods, but some of the agencies have issued guidance to their own researchers or to regulated entities that may reduce animal use. For example, in 2017, APHIS updated its guidance to allow manufacturers of animal vaccines, inactivated bacterial products, and antibody products to request an exemption to animal safety testing if the products have a documented history of acceptable safety results and controlled manufacturing processes that ensure batch consistency and sterility. APHIS also issued a notice in 2017 of a testing option that can reduce by up to 50 percent the number of hamsters required for potency testing of vaccines for the bacterial disease leptospirosis, according to the notice. In addition, APHIS issued memorandums in 2013 and 2015 that provide guidance on in vitro techniques that researchers may use instead of animals to test the potency of vaccines. Some agencies have also adopted alternative methods for their researchers without issuing specific guidance to do so. For example, in September 2018, CDC began routine use of an in vitro procedure developed by the agency that allows its laboratories to test for botulism in human serum specimens without using mice. CDC officials stated that this method is fast and inexpensive and would reduce the need for hundreds of mice. CDC officials told us that their researchers plan to expand use of the in vitro procedure to test other types of specimens, further reducing the use of animals. HHS, USDA, and EPA have made multiple efforts to develop alternative research methods that, according to the agencies, have reduced animal use or have the potential to do so. Some of these efforts target reducing the use of animals in a particular research context while others have broader applications in toxicology and computer modeling. In some cases, agency officials provided estimates of how their targeted efforts have reduced or may reduce animal use. Examples of targeted efforts include the following: CDC researchers told us they have evaluated a method that reduces the number of animals and time needed to produce kits that are distributed worldwide to identify influenza virus subtypes and thereby aid in strain selection for the influenza vaccine each season. Under the original method, antibodies for the kits were generated from blood samples in sheep that were later euthanized. The CDC researchers concluded that an alternative automated method that draws antibody-rich plasma from goats instead of blood from sheep could reduce the time needed to produce the kits and require fewer animals. According to FDA officials, FDA is collaborating with others on the development of an in vitro assay that will be used to test the potency of human rabies vaccines that manufacturers submit to FDA for approval. This new method will replace the animal-based assay that is part of the current license to manufacture rabies vaccine. The officials said that the animal-based assay uses 600 mice, on average, for each batch of vaccine submitted by a manufacturer. According to an ARS research paper, ARS worked with academic researchers to develop an in vitro method for feeding blood to ticks. According to ARS officials, the method allows researchers to reduce the number of animals used when studying disease transmission in animals via tick-borne pathogens. APHIS currently holds federal pesticide registrations with EPA for active ingredients formulated into end-use products, such as rodenticides, that APHIS uses to prevent damage to agriculture, endangered species, or critical habitats. According to APHIS officials, the agency uses an EPA- approved method for testing the risks to human health from new pesticide products that substantially reduces animal use. The method generally involves progressively increasing the pesticide dose on a relatively small number of animals compared to the previous method and waiting to observe whether the dose causes mortality before deciding whether to increase the dose in further testing. APHIS officials said the new method reduces animal use by 50 percent or more per test. EPA’s Endocrine Disruptor Screening Program Uses Alternatives to Animals Led by its Office of Science Coordination and Policy, EPA established the Endocrine Disruptor Screening Program in 1998 to fulfill a congressional mandate in the 1996 Food Quality Protection Act to develop a program to screen for certain chemicals (e.g., pesticides) that affect human hormones. EPA expanded the scope of the program to include screening the effects of chemicals on the human thyroid system and wildlife. The program began using automated, large-scale screening methods and computational models to evaluate and screen chemicals and, according to EPA, allows EPA to screen more chemicals in less time, use fewer animals, and reduce cost. Agencies’ broader efforts include the integration of advances in biology, chemistry, and computer science into areas of research, such as toxicology, that currently rely heavily on animal use. For example, EPA launched the Toxicity Forecaster in 2007 as an effort to use automated technologies to expose living cells or isolated proteins to chemicals and screen the cells or proteins when exposed to chemicals for changes in biological activity that suggest potential toxic effects. According to EPA documents, these methods could limit the number of required laboratory animal-based toxicity tests while quickly and efficiently screening large numbers of chemicals. According to EPA documents, in the first phase of this effort, which the agency completed in 2009, EPA evaluated more than 300 well-studied chemicals (primarily pesticides) that had extensive data from traditional animal-based toxicity testing; the agency then compared results from automated screening technologies with the results from the traditional animal tests. As of 2018, EPA had developed and made publicly available a library of toxicity data on more than 4,500 chemicals. The availability of the Toxicity Forecaster data has enabled EPA to reduce the need for animal testing in its Endocrine Disruptor Screening Program for identifying chemicals that may affect human hormone systems (see sidebar). Similarly, FDA has initiated a broad effort to incorporate greater use of computer modeling and simulation into its decision-making on FDA- regulated products. For example, FDA formed an agency working group on modeling and simulation in 2017. According to the Chair of the working group, it does not have an explicit objective to reduce animal testing, but such reduction is a potential benefit of the testing approaches the group is advancing. For example, the Chair said that modeling and simulation can help refine questions about products submitted for FDA approval and therefore could reduce the number of animal studies needed before clinical trials. EPA and NIH have provided funding to extramural researchers to develop alternative research methods. For example, from 2013 through 2018, EPA provided $24 million in funding for research on 3-D models containing human cells (these devices are also known as tissue chips) that can be used for tests that otherwise might be conducted using animals. In 2018, EPA also announced $4.25 million in funding for research to promote the development and use of alternative methods that reduce, refine, or replace vertebrate animal use for toxicity testing. Similarly, in 2017, NIH awarded a $962,000 grant to a research facility to conduct studies of an in vitro human bronchial tissue model for predicting the toxicity of inhaled chemicals. Additionally, while USDA’s National Institute for Food and Agriculture did not set aside a specific amount of funding, in May 2019 the agency made clear to applicants for its Welfare and Well-being of Agricultural Animals grant program that proposals that study ways to reduce the need for animals in research are eligible for funding in fiscal years 2019 and 2020. HHS, USDA, and EPA have joined partnerships to develop, use, and promote alternative testing methods. For example, the agencies participate in ICCVAM, which states that its mission is to facilitate the development, validation, and regulatory acceptance of test methods that replace, reduce, or refine the use of animals. ICCVAM itself does not conduct research or validation studies on alternative methods. Instead, it relies on stakeholders including federal agencies that generate, require, or use toxicological data; companies that develop toxicological tests; and animal welfare organizations. According to committee guidelines, stakeholders can submit the results of their research to ICCVAM, and the committee then conducts evaluations and makes recommendations on submissions for regulatory uses that align with the needs and priorities of member agencies. Zebrafish Are Used as Alternatives to Other Animals in Research A zebrafish is a freshwater, tropical vertebrate fish that is widely used in pharmaceutical development and medical and scientific research due to certain qualities of its morphology and development as well as its inexpensive cost to use and maintain. Some of these qualities include genetic and structural similarities to other vertebrates that mimic human responses to certain genes involved in human diseases and its transparent embryonic development that enables researchers to use it as an alternative model for toxicity screening of drugs and chemicals. These qualities have led to the use of zebrafish embryonic models for automated, large-scale screening programs by the National Toxicology Program and the Environmental Protection Agency, among others. ICCVAM’s website contains information on current ICCVAM- recommended protocols for specific test methods, such as methods to test for eye corrosion and irritation and skin sensitization, and on events organized by NIH and others that are relevant to the replacement, reduction, or refinement of animal use in research. For example, the website has a link to a page on NIH’s website that has the slide presentations given at six webinars from 2017 through 2018 on the use of zebrafish in toxicology testing. Researchers may use zebrafish and their embryos in particular as a replacement for other animals, such as mice (see sidebar). ICCVAM maintains on its website a list of 108 alternative methods that, as of June 2019, had been accepted by one or more federal agencies. These include methods that ICCVAM and its member agencies contributed to developing or validating. However, according to ICCVAM’s strategic roadmap issued in January 2018, the committee concluded that its evaluations of new methods during its first 15 years were lengthy, inefficient, and resource intensive. ICCVAM concluded that researchers and test method developers often initiated the development of alternative methods with little input from federal agencies or regulated industries and, therefore, these methods did not always meet the needs of federal agencies. Consequently, these methods were either not accepted by federal agencies or were accepted by the agencies but not used by the regulated community. Recognizing these limitations, ICCVAM initiated a strategic shift in 2013 aimed at adjusting the validation of new test methods to be more responsive to the needs of federal agencies and other stakeholders. Accordingly, ICCVAM’s 2018 strategic roadmap set new objectives for reducing animal use, including the following: Connect the developers of alternative methods with the regulatory agencies and the regulated industries that would ultimately use the new technologies to increase the likelihood of the methods being successfully developed and implemented. Foster the use of efficient and flexible practices, such as public-private partnerships to promote communication and cooperation, to establish confidence in new methods. Encourage the adoption and use of new methods and approaches by federal agencies and regulated industries, such as through training programs on the use of new methods. ICCVAM has established workgroups to develop detailed implementation plans to address roadmap goals. According to the strategic roadmap, the implementation plans will include four key elements: (1) definition of testing needs; (2) identification of any available alternative tests and computer models; (3) a plan to develop integrated approaches to testing and assessment and defined approaches for interpreting data; and (4) a plan to address both scientific and nonscientific challenges, including regulatory challenges, such as international harmonization. As of June 2019, workgroups on acute systemic toxicity, eye and skin irritation, and skin sensitization had posted information concerning these elements on ICCVAM’s website. For example, each workgroup authored an article published in a peer-reviewed journal and posted on the ICCVAM website about the testing needs of regulatory agencies and information about available alternatives. Another interagency effort that has a goal of promoting the use of alternative methods is the Toxicology in the 21st Century (Tox21) Program. Formed in 2008, the program is a collaborative effort among NIH, FDA, and EPA to characterize the potential toxicity of chemicals by using cells and isolated molecular targets instead of laboratory animals. A central component of the program is its focus on developing and evaluating automated in vitro screening methods to assess the hazards of chemical substances. As of February 2018, the program had used this method to assess approximately 10,000 chemicals for their potential impacts on biological systems. According to NIH’s Tox21 website, these automated methods have yielded high-quality toxicity data on environmental substances in a fraction of the time that would have been required with traditional animal testing. To address key challenges in toxicology testing, the program’s federal partners developed a strategic and operational plan in March 2018 that expanded the focus of Tox21’s research activities to include developing alternative test systems that predict chemical toxicity in humans and addressing the technical limitations of and strengthening scientific confidence in current in vitro test systems. According to NIH, activities under the plan will lead to better predicting chemical toxicity to humans through using non-animal alternatives such as stem cells and computational models. Since 2011, federal agencies have also collaborated to develop devices containing human cells that can be used for tests that otherwise might be conducted using animals. See figure 1 for an example of such devices, known as tissue chips or human microphysiological systems. According to NIH officials, this evolving technology may reduce animal testing and produce results more relevant to human health. The interagency effort was initiated in September 2011 when the President announced the formation of a collaborative project between NIH, the Defense Advanced Research Projects Agency, and FDA to develop tissue chips loaded with living human cells to screen the efficacy, safety, and toxicity of drugs, vaccines, or biological products for humans. Subsequently, in July 2012, NIH launched the Tissue Chip for Drug Screening program, which provided 19 grants to research facilities to develop tissue chips that accurately model the structure and function of the human lung, liver, heart, and more. In September 2014, NIH announced a second phase of the program in which researchers would refine existing tissue chips and combine them into an integrated system that can mimic the complex functions of the human body. In one example of this collaboration, two project teams funded by the Defense Advanced Research Projects Agency—the Massachusetts Institute of Technology and the Wyss Institute at Harvard University—are working with NIH-funded researchers to develop platforms that integrate 10 tissue chips that each represent a separate human organ. Federal agencies have also collaborated with nongovernmental organizations on training to promote the use of alternative methods. For example, NIH and the People for the Ethical Treatment of Animals (PETA) International Science Consortium offered a webinar series from March through September 2016 on alternative approaches for assessing acute inhalation toxicity. Webinar presenters described alternative approaches for identifying substances likely to cause acute systemic toxicity through inhalation. Similarly, EPA collaborated with the PETA International Science Consortium and the Physicians Committee for Responsible Medicine on webinars in November 2018, February 2019, and April 2019 that addressed alternative methods for testing the effect of chemicals on skin, for predicting the effect of inhaled substances, and for identifying substances that cause irritation or inflammation in human respiratory systems. ICCVAM’s strategic roadmap calls for its members to identify appropriate metrics for prioritizing activities, monitoring progress, and measuring success toward the goals described in the roadmap. However, ICCVAM and its member agencies have not routinely developed metrics that they could report to the public to demonstrate how their individual or collective efforts to encourage the use of alternative methods have affected or will affect animal use. HHS, USDA, and EPA officials, as well as ICCVAM’s roadmap, have cited challenges to measuring the results of ICCVAM and its member agencies’ efforts. For example, according to agency officials, differences in the regulatory contexts in which agencies use data generated through animal research—for example, in regulation of pesticides versus human or animal drugs—limit agencies’ ability to develop metrics that can be applied across multiple agencies. Furthermore, the ICCVAM roadmap states that measuring the actual impact of encouraging the adoption and use of new methods is difficult in the United States due to the limited ability to quantify animals used for toxicity testing. In particular, the Animal Welfare Act does not cover several species commonly used in research, including mice, rats, and birds bred for research and cold-blooded species such as fish. Therefore, research facilities are not required under the act to report their use of those species to APHIS, and the data APHIS receives from research facilities can only be used to track a subset of the total number of animals used for research in the United States. Although ICCVAM and its member agencies face challenges in developing metrics, the roadmap also states that agency-specific mechanisms to measure progress may exist, such as tracking the number of waivers granted for a particular animal test. For example, as discussed above, EPA has estimated the extent to which its granting of data waivers to pesticide manufacturers has reduced animal use and research costs. Additionally, some agencies have estimated the effect that a new alternative method could have on animal use. Moreover, officials from FDA and EPA said that their agencies are able to accept non-animal test data in lieu of animal test data if the data meet their regulatory needs. Measuring the frequency with which the agencies receive non-animal test data instead of animal data could be another mechanism for estimating changes in animal use. In addition, the ICCVAM Authorization Act of 2000 requires ICCVAM to prepare biennial public reports on its progress under the act—including its efforts to ensure that new and revised test methods are validated to meet the needs of federal agencies and to reduce, refine, or replace the use of animals in testing, among other things. ICCVAM, with support from NIH’s National Institute of Environmental Health Sciences, has issued the required biennial progress reports since 2001, including the most recent report issued in July 2018 that covers 2016 and 2017. However, the committee’s biennial progress reports, including the July 2018 report, provide few quantitative or qualitative assessments of the progress the member agencies have made, individually or collectively, toward reducing, refining, or replacing animal use in testing. ICCVAM’s strategic roadmap states that it envisions that workgroups will play a key role in implementing the goals of the strategic roadmap, but ICCVAM has not designated a workgroup to address the challenges related to metrics, similar to other workgroups that the committee has established to address the roadmap’s goals. According to officials from NIH’s National Institute of Environmental Health Sciences, which manages the committee, the strategic roadmap is a work in progress and developing metrics is the third of three roadmap goals. The ICCVAM Authorization Act of 2000 does not provide the National Institute of Environmental Health Sciences with authority to direct agencies to develop and report metrics. However, agency officials agreed that ICCVAM could facilitate the establishment or designation of a workgroup of member agencies to identify a range of potential quantitative and qualitative metrics that member agencies could use to assess their progress toward reducing, refining, or replacing animal use. By establishing or designating such a workgroup to develop metrics that the agencies could use to assess their individual or collective progress toward reducing, refining, or replacing animal use in testing and by incorporating those metrics in ICCVAM’s biennial progress reports, ICCVAM and its member agencies could better monitor progress across the range of the committee’s efforts and report the members’ progress to the public. HHS, USDA, and EPA use a variety of methods to ensure that researchers—whether employed by or receiving research funding from these agencies—consider alternative methods to animal research. The agencies also have engaged in multiple efforts to expand the range of available alternatives. Under one of these efforts, ICCVAM’s strategic roadmap calls for its members to identify appropriate metrics for prioritizing activities, monitoring progress, and measuring success. The roadmap envisions that workgroups will play a key role in implementing the goals of the strategic roadmap. However, ICCVAM has not designated a workgroup to address the challenges related to developing and reporting metrics. In addition, ICCVAM has issued the required biennial progress reports since 2001, but the reports provide few quantitative or qualitative assessments of the progress member agencies have made, individually or collectively, toward reducing, refining, or replacing animal use in testing. By establishing or designating a workgroup to develop metrics to assess the progress member agencies have made, individually or collectively, toward reducing, refining, or replacing animal use in testing and by incorporating those metrics in ICCVAM’s biennial progress reports, ICCVAM and its member agencies could better monitor progress across the range of the committee’s efforts and report the members’ progress to the public. The Director of the NIH’s National Institute of Environmental Health Sciences should (1) facilitate the establishment or designation of a workgroup of representatives of ICCVAM member agencies to develop metrics that the agencies could use to assess the progress they have individually or collectively made toward reducing, refining, or replacing animal use in testing and (2) incorporate those metrics into the committee’s biennial progress reports. (Recommendation 1) We provided a draft of this report to HHS, USDA, and EPA. HHS provided written comments on the draft, which are presented in appendix I. In its written comments, HHS stated that NIH concurred with our recommendation. NIH further commented that ICCVAM’s activities in support of promoting alternatives for animal use in testing do not extend to animal use in any other context, such as research or training. NIH explained that our use of the terms research and researcher to refer more generally to research, testing, teaching, or experimentation could cause misunderstanding. We understand that ICCVAM’s activities are focused on animal use in product testing. In addition, we intended our recommendation that the Director of the National Institute of Environmental Health Sciences facilitate the establishment or designation of an ICCVAM workgroup to be focused on product testing rather than on other types of animal research. However, for editorial reasons, we did not modify our report’s use of the terms research or researcher. HHS and EPA also provided technical comments, which we incorporated as appropriate. Among those comments, HHS’s FDA officials stated that the agency encourages the use of alternatives to animal testing and supports the principles of replacement, reduction, and refinement, but if no alternative exists, animal testing may be the most appropriate way to meet certain regulatory requirements to ensure the safety and efficacy of medical products. In its technical comments, EPA cited a September 2019, memorandum EPA’s Administrator issued after we sent our draft report to the agencies for comment. The memorandum commits the agency to take several steps to reduce, replace, and refine animal testing requirements. For example, the Administrator committed EPA to reducing its requests for, and funding of, whole and live mammal studies by 30 percent by 2025 and eliminating all mammal study requests by 2035. We acknowledge EPA’s announcement but did not assess it in our review of federal efforts to facilitate the use of alternative research methods. We are sending copies of this report to the appropriate congressional committees, the Secretary of Agriculture, the Secretary of Health and Human Services, the Administrator of the Environmental Protection Agency, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact us at (202) 512-3841 or morriss@gao.gov or neumannj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the individuals named above, Joseph Cook (Assistant Director), Rodney Bacigalupo, Kevin Bray, Ross Campbell (Analyst-in- Charge), Tara Congdon, Hayden Huang, Amber Sinclair, and Kari Terrio made key contributions to this report.", "summary": "methods and computer modeling. HHS, USDA, and EPA conduct and fund animal research and regulate products tested on animals. HHS and USDA also oversee federal and nonfederal research facilities including researchers' consideration of alternatives to animal use. GAO was asked to review issues related to alternatives to animal research. This report (1) describes how HHS, USDA, and EPA ensure researchers consider the use of alternatives to animals and (2) examines the steps the agencies have taken to facilitate the use of alternative research methods and to assess the effect of their efforts on animal use. GAO reviewed documents, such as agency policies and practices relevant to the consideration of alternatives and interviewed agency officials. GAO also interviewed representatives of a nongeneralizable sample of 12 federal and nonfederal research facilities randomly selected across agencies and facilities. The Department of Health and Human Services (HHS), U.S. Department of Agriculture (USDA), and Environmental Protection Agency (EPA) use a variety of methods to ensure researchers consider alternatives to animal use in research (see figure). Two of these methods are (1) requiring researchers to obtain approval of their research protocols, including their consideration of alternatives, from their institutions, and (2) calling for or recommending researchers to use database searches to identify alternatives. HHS and USDA also help ensure that researchers consider alternatives through the agencies' oversight of research facilities. For example, USDA is to conduct annual inspections of nonfederal research facilities. Futhermore, the agencies have provided training to researchers on the consideration of alternatives. HHS, USDA, and EPA have facilitated the development and use of alternatives to animal use in research through individual and collaborative efforts. These efforts include agency strategies and policies for promoting the use of alternative methods and the development of testing methods that rely on non-animal models. Additionally, the agencies are members of the Interagency Coordinating Committee on the Validation of Alternative Methods, which is managed by HHS's National Institute of Environmental Health Sciences. The committee promotes testing methods that protect human health and the environment while reducing animal use. The interagency committee's 2018 strategic roadmap calls for it to identify appropriate metrics for monitoring progress and measuring success in adopting alternatives. However, the committee and its member agencies have not routinely developed or reported metrics that demonstrate how their efforts to encourage the use of alternative methods affect animal use. They have also not designated an interagency workgroup to address the challenges related to developing and reporting such metrics. Facilitating the establishment of such a workgroup would help the committee and its member agencies better monitor their progress across the range of their efforts to reduce animal use and report members' progress to the public. GAO recommends that HHS's National Institute of Environmental Health Sciences facilitate the establishment of an interagency workgroup to develop metrics for assessing progress on the development and promotion of alternatives to animal use and incorporate those metrics into public reports. HHS agreed with our recommendation.", "document_type": "gao"}
{"report": "The ISM Code was established to provide an international standard for the safe management and operation of ships and for pollution prevention. The code establishes safety management objectives, such as preventing human injury or loss of life, and identifies a framework of key elements required to be considered for inclusion in an SMS. According to the ISM Code, each vessel operator should develop, implement, and maintain an SMS that is to include functional requirements, such as procedures to prepare for and respond to emergency situations. An SMS is typically not a single plan and can take different forms. It is up to the vessel operator to determine how best to operationalize these requirements. The SMS plan documents generally contain proprietary information and are not retained by the Coast Guard or the ROs performing services on the Coast Guard’s behalf. There are three key entities involved in vessel SMS activities—vessel operators, ROs, and the U.S. Coast Guard. These entities’ SMS responsibilities are described below. Vessel operators are responsible for developing an SMS in accordance with ISM Code requirements if they operate U.S-flagged vessels that are subject to the ISM Code, such as a vessel engaged in a foreign voyage that is carrying more than 12 passengers, or a tanker or freight vessel of at least 500 gross tons, among other vessel types. Vessel operators are required to perform an internal audit of their company’s SMS each year to ensure it is being implemented effectively. Vessel operators are also responsible for obtaining the requisite evidence that the company and each of its applicable vessels are in compliance with the ISM Code. In practice, this means that the vessel operators obtain certification from ROs, which are described below. According to the Coast Guard, there were approximately 1,170 U.S.-flagged vessels that maintained SMS certifications in 2019. An RO refers to an international classification society authorized by the Coast Guard to conduct applicable vessel oversight and certification services on its behalf. The Coast Guard has authorized several ROs to conduct SMS audits and issue applicable certificates, but over 95 percent of these vessel oversight and compliance activities are conducted by a single RO, the American Bureau of Shipping. ROs have to meet specific requirements for authorization, such as making information about vessel class and inspections available to the Coast Guard. In order to be authorized, the RO needs to have been an international classification society for 30 years and have a history of taking appropriate corrective actions in addressing, among other things, vessel deficiencies. ROs are to conduct the following SMS activities on the Coast Guard’s behalf: review SMS documents and conduct initial company and vessel audits to verify compliance with the ISM Code and applicable national and international requirements; issue a Document of Compliance to the vessel operator and a Safety Management Certificate for the vessel, which is valid for up to 5 years; conduct annual SMS compliance audits of the vessel operator; conduct an intermediate SMS compliance audit for the vessel at least once during the 5-year period; and conduct renewal SMS compliance audits of vessel operator and vessel(s) prior to expiration of the 5-year certificate. The U.S. Coast Guard is ultimately responsible for guaranteeing the effectiveness of SMS compliance activities and audits that ROs perform on its behalf. The Coast Guard’s oversight activities of ROs are conducted by the Office of Commercial Vessel Compliance. This office oversees a range of different activities to help ensure SMS compliance with the ISM Code and applicable federal regulations. Such activities include managing the commercial vessel inspection program, developing related guidance, and overseeing SMS audits and related activities performed by ROs. In addition to oversight provided by officials at Coast Guard headquarters, marine inspectors within local Coast Guard field units are also responsible for conducting vessel inspections, which routinely include assessing SMS effectiveness for applicable vessels. The Coast Guard verifies SMS compliance as part of its overall vessel compliance activities, such as conducting annual inspections of applicable U.S.-flagged vessels. According to the Coast Guard, recurrent vessel inspections are important opportunities for its marine inspectors to verify the effectiveness of the vessels’ SMS, even if SMS oversight is not the primary purpose of the vessel inspections. When conducting an annual vessel inspection, Coast Guard marine inspectors are to look for material deficiencies, such as poor condition of vessel structures, missing or defective equipment, or hazardous conditions that could indicate a potential SMS nonconformity. According to Coast Guard officials, marine inspectors routinely review the Coast Guard’s internal database for a record of any past deficiencies and are to inspect the vessel’s SMS documentation to determine if the Safety Management Certificate is up- to-date and the drill logs are current, among other things. The Coast Guard advises vessel operators to self-report or, in other words, proactively manage their vessels and report any deficiencies identified by the vessel’s crew and report them at the beginning of any Coast Guard inspection. When conducting an annual vessel inspection, Coast Guard marine inspectors are to follow a five-step process to identify any SMS-related deficiencies, determine if there are clear grounds for an expanded vessel inspection, and specify any applicable compliance options. The process requires distinguishing between normal wear and tear to the vessel and deficiencies that could be the result of failures to implement an effective SMS. (See appendix II for further details on this five-step process.) A more in-depth inspection, if warranted, may include a review of maintenance schedules and records, crew training records and certifications, emergency procedures, and associated interviews with the vessel master and crew. Marine inspectors are to record any identified deficiencies on a Form 835V, which specifies the time frames and procedures required to address the identified deficiencies. See figure 1 for a blank copy of the Form 835V. The Coast Guard uses a range of options for addressing SMS-related deficiencies. Some deficiencies, such as improperly secured wiring or missing documentation, can sometimes be corrected by the vessel’s crew during the course of a Coast Guard inspection. According to Coast Guard guidance, if marine inspectors identify serious deficiencies that could indicate broader SMS failures, such as an absence of required equipment or failure by the company to notify the Coast Guard of reportable marine casualties and hazards, the inspectors record an SMS-related deficiency and require an internal SMS audit. An internal SMS audit is for technical or operational deficiencies that individually or collectively do not warrant the detention of the vessel but indicate a failure or lack of effectiveness of the SMS. The internal SMS audit and any corrective actions are to be completed by the vessel operator within three months from the date of the Coast Guard vessel inspection. If during the course of a vessel inspection Coast Guard inspectors observe more serious deficiencies or failures, such as defective or missing fire-fighting or life-saving equipment, the vessel is to be detained and an external audit is to be performed by the RO prior to the vessel being released from detention. Figure 2 shows the Coast Guard’s process for ensuring SMS compliance during vessel inspections. In addition to the annual vessel inspections it conducts, the Coast Guard also maintains a list of vessels that require additional oversight, referred to as the “fleet risk index.” The Coast Guard Office of Commercial Vessel Compliance evaluates vessels enrolled in the Alternate Compliance Program and the Maritime Security Program to develop the fleet risk index using modeling that considers and weighs multiple risk factors to assign each vessel a risk score. This list is used internally by Coast Guard inspectors when prioritizing vessels for additional oversight and more frequent inspections. Assessed risk factors include vessel detentions, marine violations/enforcement actions, vessel deficiencies, vessel type, and vessel age, among others. According to Coast Guard officials, the Coast Guard uses the fleet risk index to identify approximately 50 vessels each year that are subject to inspections every 6 months rather than annually. In 2018, the Coast Guard stipulated that traveling inspectors would accompany the local inspection team to conduct all inspections aboard vessels designated for additional oversight. According to Coast Guard officials, traveling inspectors have additional training and inspection expertise, including supplemental coursework in auditing and quality management systems, and they routinely conduct additional background research on these vessels prior to participating in the inspections. Based, in part, on recommendations in the EL FARO investigative report, in 2018 the Coast Guard took steps to improve its management of the Alternate Compliance Program, including efforts to improve data reporting. For example, the Coast Guard revised its form for documenting deficiencies during annual vessel inspections. In particular, since March 2018, the Form 835V has included a checkbox to indicate if a deficiency is related to an SMS. According to the Coast Guard, this revision will allow for enhanced annual reporting of safety-related deficiencies identified during compliance activities. The Coast Guard reported it conducts approximately 1,200 inspections each year of vessels that are either required to maintain a Safety Management Certificate, or do so voluntarily. According to the Coast Guard, in calendar year 2018, the Coast Guard issued between 70 and 130 SMS-related deficiencies (reporting available for April through December only), and for calendar year 2019, the Coast Guard issued between 183 and 212 SMS-related deficiencies. Given the limited data and time frames available, we were not able to identify any trends regarding SMS deficiencies. However, we noted that the highest number of safety-related deficiencies cited in 2019 were related to maintenance of vessels and equipment—43 of the 212 annual deficiencies. The second-highest number of deficiencies addressed issues related to emergency preparedness—37 of the 212 annual deficiencies. Some specific examples in this category relate to the posting of applicable emergency instructions and providing updated records of emergency drills. According to Coast Guard headquarters officials, the Coast Guard plans to review and assess the SMS deficiency data to provide feedback to inspectors, vessel operators, and ROs. The officials also stated that SMS deficiencies will be included in future risk-based vessel inspection programs, including the fleet risk index discussed earlier. Following the investigative reports of the EL FARO sinking, the Coast Guard initiated several efforts in 2018 to enhance oversight of the ROs that perform SMS-related services and certifications on its behalf. These efforts were largely driven by actions identified by the Commandant of the Coast Guard in December 2017 in response to EL FARO investigative report recommendations. In particular, the Coast Guard established a new group to monitor ROs, developed new SMS-related guidance and associated work instructions, increased direct observations of ROs, developed key performance indicators, and developed guidance to request internal investigations for certain RO deficiencies. It is too early for us to assess the overall effectiveness of these Coast Guard efforts; however, we believe they are positive steps toward enhancing oversight of ROs. Further information on each of these efforts is provided in the sections that follow. Established a new group within the Office of Commercial Vessel Compliance. The Coast Guard established a new group within its Office of Commercial Vessel Compliance in 2018 to help monitor the global performance of the U.S.-flagged fleet, provide enhanced oversight of ROs performing vessel safety management functions, and implement any necessary changes to related roles and responsibilities. Developed SMS-related guidance and work instructions. The Office of Commercial Vessel Compliance developed several new work instructions to help inform mariners, the public, the Coast Guard, and other federal and state regulators in applying SMS-related statutory and regulatory requirements. The following are examples of applicable guidance issued since 2018: CVC-WI-003(1): USCG Oversight of Safety Management Systems on U.S. Flag Vessels (March 23, 2018). This document contains guidance for assessing the effectiveness of the SMS on U.S.-flagged vessels, including directions for evaluating potential deficiencies and compliance options during the course of a vessel inspection. CVC-WI-004(1): U.S. Flag Interpretations on the ISM Code (April 16, 2018). This document provides guidance regarding the Coast Guard’s interpretations on the application and implementation of the ISM Code. Increased the number of Coast Guard direct observations of ROs performing vessel and company audits. The Coast Guard reported it has increased the number of direct observations of ROs conducting vessel and company SMS audits since 2018. According to the Coast Guard, audit observations aboard vessels are routinely performed by traveling inspectors. Additionally, staff from the new Commercial Vessel Compliance group are observing an increased number of company audits. This group has eight staff available for direct observations of ROs, all of whom have received training in international auditing and safety management standards. The Coast Guard reported that the number of audit observations attended by the Commercial Vessel Compliance staff increased from three in 2018 to 21 in 2019. According to the Coast Guard, these additional observations serve as a mechanism to provide increased oversight of the ROs and the companies or vessels being audited, as well as to verify that the services provided by ROs are effectively executed in accordance with established requirements. Developed key performance indicators for assessing ROs. In mid- 2018, Coast Guard officials identified 10 key performance indicators to be used to evaluate the performance of ROs. Due, in part, to challenges with collecting and synthesizing the requested data from the different ROs, the Coast Guard reported on limited performance information in the 2018 Domestic Annual Report. According to Coast Guard officials, the Coast Guard is working with each of the ROs and the International Association of Classification Societies to standardize the key performance indicator data to better integrate the data into the Coast Guard’s data system. The Coast Guard said that it plans to include a subset of the key performance indicators in its 2019 annual report, which is scheduled for issuance in April 2020. See appendix III for more information on these key performance indicators. Developed guidance for ROs on “quality cases.” In May 2018, the Coast Guard also issued guidance that describes a new oversight mechanism, referred to as a “quality case.” If a Coast Guard marine inspector observes evidence during the course of a vessel inspection that an RO is not adequately performing its required SMS-related functions, the Coast Guard can request that the RO conduct a root-cause analysis to help identify the underlying issue(s). This analysis would generally involve the RO evaluating its quality management system and reporting findings and corrective actions to the Coast Guard. From May 2018 to November 2019, the Coast Guard reported it initiated 13 quality cases; one of which was SMS-related. Each of the 12 SMS plans (or plan excerpts) for U.S.-flagged vessels that we reviewed identify potential shipboard emergencies and applicable response procedures, but they do not address the full range of emergency scenarios included in Coast Guard guidance. While the 12 SMS plans do not address all potential emergencies included in Coast Guard guidance, the plans do address the broad, functional requirement to identify potential shipboard emergencies and applicable response procedures to address them, as required by the ISM Code and applicable federal regulations. In reviewing the 12 SMS plans, we also found variation among the specific scope and formats of the emergency preparedness sections. Four of the 12 SMS plans are large documents spanning hundreds of pages that incorporate various component manuals. For example, one vessel operator provided a comprehensive SMS plan document of nearly 600 pages that includes six different procedural manuals covering the following issues: Management, Vessel, Safety, Environmental, Cargo Operations, and Emergency Response. For the other eight SMS plans we reviewed, the vessel operators provided us with either a stand-alone manual specifically addressing shipboard emergency preparedness and response procedures, or individual chapters and excerpts that included this information. According to Coast Guard and RO officials, the ISM Code does not require a specific format or level of detail for SMS plans and, rather, allows vessel operators flexibility to choose how they will implement and document SMS requirements based on their specific operations and business processes. In addition to reviewing the SMS plans for content and format, we also reviewed each of the 12 SMS plans (or excerpts) to determine the extent to which they address 21 different potential shipboard emergencies identified in 2018 Coast Guard guidance related to the application and implementation of the ISM Code (see table 1). The number of unique, potential shipboard emergency scenarios addressed in the SMS plan documents we reviewed generally range from five to 16. Ship routing procedures related to heavy weather, which is an emergency scenario highlighted in the EL FARO investigative report, is clearly identified in five of the 12 SMS plans reviewed. However, one additional SMS plan makes reference to a separate heavy weather plan that was not included in the primary SMS plan documents that we reviewed. The most frequently addressed shipboard emergency scenarios—that are addressed in at least 10 of the 12 SMS plans we reviewed—are Fire, Collision, Grounding, Abandon Ship, and Man Overboard. In addition, 10 of the 12 SMS plans we reviewed also identify additional potential emergency shipboard scenarios not included in the 2018 Coast Guard guidance, such as breakaway from dock, emergency towing, or confined space rescue. While none of the SMS plans that we reviewed specifically address all 21 potential shipboard emergencies identified in the 2018 Coast Guard guidance, the guidance states that it is not a substitute for applicable legal requirements, nor is it itself a rule. According to officials from the two ROs with whom we discussed this program, their auditors are provided the 2018 Coast Guard guidance to use as part of their SMS audit criteria. The officials noted, however, that their auditors may be limited to issuing an “observation” to the vessel operator if any potential shipboard emergency listed in Coast Guard guidance is not addressed in SMS plan documents. Under the ISM Code, an “observation” is not the same as an SMS “nonconformity,” which would require specific corrective action. Officials from one RO noted that any nonconformities identified would need to be based on specified mandatory requirements, such as ISM Code provisions, U.S. statutes, or applicable U.S. or international regulations, and not solely on the 2018 Coast Guard guidance. In addition to the fact that the emergencies listed in the guidance are not required to be included in SMS plans, there are other factors to explain why the SMS plans we reviewed may not address all 21 potential shipboard emergency scenarios identified in the 2018 Coast Guard guidance. Such factors include the following: Size and nature of vessel operations. According to RO and Coast Guard officials, not all of the 21 potential shipboard emergency scenarios contained in the 2018 Coast Guard guidance are applicable for each type of vessel or for all geographical operating areas. For example, specific emergency procedures related to piracy or terrorism, cargo-related accidents, helicopter rescue operations, or loss of key personnel may not be necessary for towing vessels, given the nature of their operations, their limited size, and the reduced number of crew required to operate that type of vessel. Similarly, icing conditions would not be expected to be included in the SMS plans for those vessels that operate solely in temperate waters. Additional time may be needed to incorporate expanded potential shipboard emergency scenarios into existing SMS plans. Although the Coast Guard guidance identifying the 21 potential shipboard emergency scenarios was issued in April 2018, vessel operators may still be in the process of revising their SMS plans to include additional potential shipboard emergency scenarios and applicable emergency response procedures. For example, we observed that six of the 21 scenarios included in the 2018 Coast Guard guidance are not listed in related guidance provided by the International Association of Classification Societies. These six scenarios are among those observed with the lowest frequency during our review of SMS plans. It is feasible that information related to these scenarios—such as loss of key personnel, or loss of communications with a vessel—may exist elsewhere in vessel operators’ SMS documents or in other vessel plans, but not incorporated as potential shipboard emergency response scenarios as proposed in the 2018 Coast Guard guidance. Along these lines, officials from the ROs with whom we spoke also noted that, in accordance with the ISM Code, they routinely use a sampling approach when conducting annual company SMS audits, and would generally not review the entire scope of an SMS plan each year. As a result of the sampling process, the annual audits occurring since April 2018 may not have addressed any potential “observations” related to the expanded scope of potential shipboard emergencies included in the Coast Guard guidance for SMS plans. As noted previously, the ISM Code and corresponding U.S. regulations and Coast Guard guidance allow vessel operators flexibility in how they address SMS functional requirements, including the documentation of potential shipboard emergencies and applicable response procedures in their SMS plans. Following the EL FARO incident, in 2018 the Coast Guard developed guidance to help inform vessel operators and ROs of potential shipboard emergency scenarios to consider. However, similar to the SMS-compliance and oversight practices used by comparable agencies in other developed countries, we found that the Coast Guard does not have a direct role in reviewing or approving vessel SMS plan documents, including response procedures for potential shipboard emergency scenarios. Rather, as described earlier, the Coast Guard relies on periodic vessel inspections and oversight of ROs that perform more rigorous ISM audits on the Coast Guard’s behalf. Although the Coast Guard has taken positive steps since 2018 to develop additional guidance and increase the number of observations of RO audits and inspections, the extent to which these efforts will result in any specific changes to the content of SMS plans by vessel operators in the future is yet to be determined. We requested comments on a draft of this report from DHS and the Coast Guard. Officials from the Coast Guard provided technical comments, which we incorporated into the report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the U.S. Coast Guard, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (206) 287-4804 or AndersonN@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. Federal regulations allow the Commandant of the Coast Guard to delegate certain functions to authorized classification societies. In order for a classification society to be recognized by the Coast Guard and receive statutory authority to carry out delegated functions as a Recognized Organization (RO), the classification society must meet certain requirements, including having functioned as an international classification society for at least 30 years and having established a history of appropriate corrective actions in addressing vessel casualties and deficiencies, among other things. With respect to safety management systems (SMS), ROs—once authorized by the Coast Guard—are able to perform SMS-related audits and issue SMS-related certifications and documentation. The following information summarizes the key roles and responsibilities of ROs related to International Safety Management (ISM) Code certification services and the key activities that ROs perform to fulfill their delegated SMS compliance functions on behalf of the Coast Guard. Interim verification. When a new company (i.e., vessel owner/operator) is established, or an existing company wants to add a new vessel type to its current Document of Compliance, the RO is to first verify that the company has an SMS that complies with ISM Code requirements. If the RO determines that the company is in compliance, it issues the company an interim Document of Compliance (which applies to the entire company) that is valid for up to 12 months. Initial verification. After receiving an interim Document of Compliance, a company applies for ISM Code certification, and an RO conducts an SMS audit of the company’s shoreside management system that is to include a visit to the company’s physical offices. Following the satisfactory completion of the audit and verification that the company’s SMS has been in operation for at least 3 months, the RO would issue the company a Document of Compliance that is valid for 5 years. After the RO issues the Document of Compliance, the RO is to verify that the company’s SMS has been functioning effectively for at least 3 months for each of the vessels for which the company is seeking a Safety Management Certificate. A Safety Management Certificate is vessel- specific and may only be issued to a vessel if the company holds a valid Document of Compliance. To perform the initial verification, the RO is to assess each vessel to determine if the company’s SMS is being employed effectively on that vessel. Annual or intermediate verification. The RO is responsible for verifying a company’s Document of Compliance every year and for verifying the company’s Safety Management Certificates at least once during the 5- year period covered by the issued certificates. ROs generally verify Safety Management Certificates between 2 and 3 years after their issuance. Annual and intermediate verifications are opportunities for the RO to verify whether the company has taken appropriate actions to sufficiently address any deficiencies the RO may have identified during previous audits. Renewal verification. Up to 3 months before a company’s Document of Compliance or a vessel’s Safety Management Certificate expires, the RO is to conduct a renewal verification. The renewal verification is to address all elements of the SMS, including activities required under the ISM code. Additional Verification. The Coast Guard may also require additional verification to ensure that an SMS is functioning effectively—for example, to make sure that the company has sufficiently implemented appropriate corrective actions to address any identified deficiencies. This appendix provides summary information on the Coast Guard’s process for evaluating safety management system (SMS) deficiencies and corrective action options if a Coast Guard marine inspector identifies any SMS-related deficiencies during a vessel inspection. In mid-2018, Coast Guard officials identified 10 key performance indicators to be used to evaluate the performance of Recognized Organizations (RO). Information on these 10 performance indicators is summarized below. 1: Number of RO-issued statutory findings divided by the number of statutory surveys conducted (e.g., 100 findings / 10 surveys = 10 Key Performance Indicators). 2: Number of RO Safety Management Certificate audit findings divided by the number of Safety Management Certificate audits conducted 3: Number of RO Document of Compliance audit findings divided by the number of Document of Compliance audits conducted (includes all types of Document of Compliance audits). 4: Number of RO associations to Port State Control Detentions under the Paris and Tokyo Memoranda of Understanding, and Coast Guard Port State Control programs. 5: Number of International Association of Classification Societies Procedural Requirement-17s (IACS PR-17) issued divided by the total number of RO applicable surveys conducted. 6: Total number of U.S. commercial vessel casualties divided by the total number of commercial vessels in the U.S. fleet of responsibility. 7: Total number of RO nonconformities issued by the Coast Guard divided by the number of statutory surveys and International Safety Management (ISM) audits conducted. 8: Total number of Coast Guard-issued deficiencies related to statutory certificates divided by the total number of Coast Guard inspections conducted. 9: Total number of RO-associated Flag State Detentions divided by the total number of statutory surveys and audits performed. 10: Number of Coast Guard-issued ISM-related deficiencies divided by the total number of Coast Guard inspections completed. Nathan Anderson, (206) 287-4804 or AndersonN@gao.gov In addition to the contact named above, Christopher Conrad (Assistant Director), Ryan Lambert (Analyst-in-Charge), Ben Nelson, Elizabeth Dretsch, Tracey King, Kevin Reeves, and Benjamin Crossley made key contributions to this report.", "summary": "In October 2015, the U.S cargo vessel EL FARO sank after encountering heavy seas and winds from Hurricane Joaquin, killing all 33 crew members. Subsequent investigations cited deficiencies in the vessel's SMS plans as a factor that may have contributed to the vessel's sinking. Some in Congress have raised questions about the effectiveness of vessel SMS plans and the Coast Guard's oversight of third parties responsible for ensuring vessels comply with international standards and federal regulations. The Hamm Alert Maritime Safety Act of 2018 included a provision for GAO to review Coast Guard oversight and enforcement of vessel SMS plans. Accordingly, this report addresses (1) how the Coast Guard (a) verifies domestic commercial vessels' SMS plans comply with federal regulations and (b) conducts oversight of ROs, and (2) the extent to which domestic vessels' SMS plans identify potential shipboard emergencies and include applicable response procedures. To address these objectives, GAO reviewed Coast Guard regulations and guidance, accompanied marine inspectors on vessel inspections and audits, and analyzed available data on identified vessel deficiencies. GAO also reviewed the format and content of a nongeneralizable sample of 12 SMS plans representing various types of vessels and interviewed relevant Coast Guard and RO officials. The Coast Guard verifies that domestic commercial vessels comply with safety management system (SMS) requirements through activities that include conducting annual inspections of applicable U.S.-flagged vessels. In practice, the Coast Guard delegates primary vessel SMS compliance activities to third party entities, called Recognized Organizations (ROs). Among their responsibilities, ROs coordinate with vessel operators to review SMS plans, issue applicable vessel certificates, and conduct SMS compliance audits at the company level and aboard each vessel. Because the Coast Guard relies on ROs to perform SMS certification services on its behalf, it has initiated a series of efforts to enhance its oversight of ROs since 2018. The efforts include: establishing a new group within the Coast Guard to monitor ROs, developing new SMS-related guidance and work instructions, increasing direct observations of ROs performing SMS audits, developing key performance indicators for assessing ROs, and requesting internal investigations for certain RO deficiencies. It is too soon to assess the effectiveness of these efforts; however, GAO believes these are positive steps toward enhancing the Coast Guard's oversight of ROs. Each of the 12 domestic vessel SMS plans GAO reviewed include potential shipboard emergencies and applicable response procedures to address them. None of the plans address all 21 potential shipboard emergencies included in 2018 Coast Guard guidance. However, these 21 potential emergencies are not required to be included in SMS plans; rather, they are suggested as part of the 2018 guidance. Further, GAO found that the SMS plans may not address all potential shipboard emergencies because not all emergency scenarios are applicable for each type of vessel or geographical operating area. Also, vessel operators may still be in the process of revising their SMS plans to include additional emergency scenarios and applicable response procedures.", "document_type": "gao"}
{"report": "The Secret Service’s primary responsibility is to physically protect the President, Vice President, their immediate families, and visiting foreign dignitaries as well as the White House complex. The Office of Protective Operations is the principal office responsible for providing protection. Within the Office of Protective Operations, agents may be assigned to a number of divisions, such as PPD, VPD, or one of the other divisions that is responsible for protecting former presidents and visiting heads of state or heads of government. The Uniformed Division, which is also part of the Office of Protective Operations, is charged with protecting facilities and venues for Secret Service protectees. Uniformed Division officers control access to the White House complex—which includes the White House itself, the Eisenhower Executive Office Building, and the Department of the Treasury building—and the Vice President’s residence. The Secret Service’s secondary responsibility is to conduct criminal investigations in areas such as financial crimes, identity theft, counterfeiting of U.S. currency, computer fraud, and computer-based attacks on banking, financial, and telecommunications infrastructure. Over time, its investigative mission has grown to encompass a wide range of financial and cybercrimes. In addition to investigating financial and electronic crimes, special agents conduct protective intelligence— investigating threats against protected persons, including the President, and protected facilities, such as protectee residences. These activities are conducted through the Office of Investigations, which oversees the agency’s field office structure. Agents assigned to the field offices also support protective operations by, for example, providing physical protection with the assistance of federal, state, and local law enforcement entities when a protectee travels. Because agents are trained to conduct both criminal investigations and to provide protection, agents assigned to investigations in field offices can contribute to protective operations when needed. The Secret Service has experienced a number of protection-related security incidents on the White House complex since April 2012. Incidents have included attempts to gain access to the White House complex by foot, car, and air. These incidents, among others, highlight some of the many challenges the Secret Service confronts while providing protection. See figure 1 for a description of selected security incidents. The PMP’s 2014 report made 19 recommendations regarding (1) training and personnel; (2) technology, perimeter security, and operations; and (3) leadership. We found that the Secret Service fully implemented 11 recommendations and is in the process of implementing the remaining eight recommendations. Table 1 summarizes the progress that the agency has made implementing each recommendation. Appendix I provides further details on the actions the Secret Service has taken to address each recommendation. Following the PMP’s recommendation that the Secret Service should “clearly communicate agency priorities, give effect to those priorities through its actions, and align its operations with its priorities,” the Secret Service took steps toward communicating internally and externally the precedence of protection. For example, the agency hired a Senior Executive Director of Communications in 2016 and formed the Office of Communications and Media Relations in 2018 to manage the agency’s public affairs efforts and to oversee internal agency communication. Additionally, each year the Director of the Secret Service issues a strategic priority memorandum that identifies priority areas for the upcoming fiscal year budget. Further, from fiscal year 2014 through fiscal year 2018, special agents across the entire agency worked more hours on protection assignments and fewer hours on investigations. Specifically, in fiscal year 2014, agents spent 4.3 million hours (54 percent) on protection and 2.8 million hours (36 percent) on investigations, whereas in fiscal year 2018, agents spent 4.9 million hours (59 percent) on protection and 2.2 million hours (26 percent) on investigations. The number and percentage of hours spent on protection peaked in fiscal year 2016, but was higher in fiscal year 2018 than in fiscal year 2014. Figure 2 shows the distribution of agent work hours. The Secret Service has identified protection as its priority, and the Secret Service has identified training as an essential component of protection. In its December 2014 report, the PMP found that the security incident at the White House on September 19, 2014, arose from a “catastrophic failure of training”. The PMP therefore recommended, and Secret Service agreed at the time, that special agents assigned to PPD and VPD train for 25 percent of their work time. This was to be accomplished by allowing agents time to train during the designated training shift, known as the “fourth shift”. However, while training for special agents agencywide increased to 10 percent by 2018—more than triple the amount from fiscal year 2014— training for those assigned to PPD and VPD did not increase accordingly. Specifically, special agents assigned to PPD and VPD reported attending training for about 5.9 percent and 2.9 percent of their regular work hours in fiscal year 2018, respectively, compared with 3.3 percent and 1.9 percent in fiscal year 2014. (See figure 3.) According to our analysis of Secret Service self-reported data, in fiscal year 2018, special agents assigned to PPD and VPD missed achieving the 25- percent training target by 76 and 88 percent each. Figure 3 shows the share of regular work hours that agents assigned to PPD and VPD spent in training in fiscal years 2014 through 2018 compared to the annual target. The Secret Service established the 25 percent training target for agents assigned to PPD and VPD, and senior officials reaffirmed the target in March 2019. However, according to a senior Office of Protective Operations official, the fast operational tempo (i.e., heavy workload) hampered agents’ ability to participate in training. This official told us that the amount of protection that the Secret Service provides dictates how often agents are assigned protection assignments during the training shift. Senior Secret Service officials further added that the number of protectees and the amount of travel for the current protectees is higher for the current administration than for prior administrations, which reduces the time agents have available for training. According to Secret Service officials, the Secret Service’s ability to meet the PPD and VPD training targets is dependent on increased staffing levels. The Secret Service outlined its plans to increase staffing levels in the Secret Service FY 2018–FY 2025 Human Capital Strategic Plan, which was published in May 2017. The plan describes, among other things, the agency’s human capital strategic goals, the process used to determine staffing needs, and annual hiring targets. By the end of fiscal year 2025, the agency plans to employ 9,595 individuals overall, including 4,807 special agents—an increase of 1,193 special agents from the end of fiscal year 2018. To meet the special agent target, the Secret Service assumes an average net growth of about 182 special agents per year between fiscal years 2019 through 2025. However, the Secret Service’s human capital strategy does not address the immediate need to help PPD and VPD meet training targets. Even though the special agent staffing level increased from fiscal years 2014 to 2018 by 332 agents, training levels for agents assigned to PPD and VPD remained below the 25- percent target at 6 and 3 percent, respectively, in fiscal year 2018. Because of the agency’s zero-fail responsibility to protect the President and Vice President, the PMP concluded that it is imperative that the Secret Service strive to address training deficits as soon as possible. In addition, according to leading management practices related to training and development efforts, adequate planning allows agencies to establish priorities and determine the best ways to leverage investments to improve performance. As part of the planning, agencies may need to compare various training strategies in terms of, among other things, the availability of resources and risk of unfavorable consequences if training investments are not made. The agency has focused on increasing training as part of its eight-year human capital strategy, but the Secret Service has not developed a plan to ensure that it meets near-term protection-related training targets. One way that the agency could address PPD and VPD training needs in the short term is to shift agents from investigations to protection assignments. Because all agents are trained to provide protection and to conduct investigations, they can be moved between investigations and protection when dictated by operational circumstances. For example, in fiscal year 2016 agents worked about 548,000 fewer investigative hours in order to support protection than they did in fiscal year 2015. This shift was made to accommodate increased protection demand from candidates in the November 2016 presidential election. See figure 5. In fiscal year 2018, agents across the agency spent nearly 2.2 million hours on investigations. By comparison, agents assigned to PPD and VPD would have needed an additional 136,000 hours and 66,000 hours of training, respectively, in fiscal year 2018 to reach the training targets. Shifting agents from investigations to protection would reduce field offices’ capacity to complete investigations. However, the agency’s stated priority is protection, and training was identified by the PMP as a key component of protection. Increasing staffing levels, as planned, over the long term may adequately support the protective and the investigative priorities at the levels defined by the agency. However, the Secret Service is relying on hiring goals alone to achieve its training-related targets, and it may not be able to achieve its hiring goals because of, among other things, uncertainty about whether enough funding will be requested and appropriated to expand the agency at planned levels. For example, an increase of 89 special agents was requested in the fiscal year 2020 budget submitted to Congress, 88 special agents short of the 177 planned for in the Secret Service FY 2018–FY 2025 Human Capital Strategic Plan. Further, the Secret Service has not developed a plan specifically for meeting training targets for agents assigned to PPD and VPD given current and planned staffing levels. While reviewing a draft of this report and after further consideration of the resources required to achieve the PMP-recommended training targets, in May 2019 the Secret Service stated that it no longer agrees with the training target recommended by the PMP and plans to reevaluate it. Developing and implementing a plan for meeting established training targets given current and planned staffing levels will help ensure that protection-related training targets are met in the near term and that agents assigned to PPD and VPD are prepared to carry out Secret Service’s priority—protection. The PMP recommended, and Secret Service agreed, that Uniformed Division officers—who provide protection at the White House—train for 10 percent of their work time. However, the Secret Service cannot fully assess progress towards achieving the 10-percent training target because it lacks complete and appropriate protection-related training data for Uniformed Division officers. Standards for Internal Control in the Federal Government state that management should design control activities to achieve objectives and respond to risks, and that management should implement control activities through policies. Appropriate types of control activities include, for example, the accurate and timely recording of transactions. Internal control standards also state that management should use quality information to achieve the entity’s objective. Quality information is appropriate, current, complete, accurate, accessible, and provided on a timely basis. According to Secret Service officials, training data for Uniformed Division officers are collected through various means and systems. For example, officials stated that they use a database called ePerson to capture certain types of training, such as firearms and physical training requalification sessions. In addition, officials report that Secret Service separately uses DHS’s Performance and Learning Management System (PALMS), which DHS designed to consolidate existing learning management systems for each of DHS’s agencies. PALMS collects data on computer-based training and training provided at the James J. Rowley Training Center automatically, but requires manual entry for training provided at offsite locations. According to Secret Service officials, there are a significant amount of internal on-the-job training instances that do not get recorded. As a result, training data collected on Uniformed Division training hours are incomplete. Further, we reviewed the training data that the Uniformed Division provided to us and identified a number of data quality issues affecting the data’s completeness and appropriateness. For example, certain training was identified by location only or lacked descriptions to clearly link the training to the skills Uniformed Division officers require while working at the White House. Additionally, Secret Service counted training unrelated to protection, such as training on electronic travel vouchers and retirement planning, towards achievement of the 10-percent protection- training target. This occurred because the Secret Service lacks a policy with a documented process identifying how to capture Uniformed Division training information and the type of training to be captured. According to Uniformed Division management, the Secret Service initiated a process in 2017 to enhance the collection and compilation of Uniformed Division training information. Specifically, each Uniformed Division branch training coordinator is to send a list of completed training to Uniformed Division management every 2 weeks. That training information, along with information captured in other systems such as PALMS, is then to be manually compiled by a Uniformed Division staff person at Secret Service’s headquarters. However, the Secret Service has not consistently employed the new process since initiating it in 2017. For example, according to Secret Service officials, the individual responsible for compiling the data was absent from the position for 3 months, and they did not know whether the data for that period were compiled at that time. As personnel—such as the branch training coordinators or the individual responsible for compiling the data—change positions, it is important that the Secret Service have a policy with a documented process to ensure that data collection continues over time and given staff changes. Further, the process does not include information on how or whether to capture internal on-the-job training instances, or instruction on the type of training to be captured to demonstrate that the training is protection-related training. Developing and implementing a policy with a documented process to collect complete and appropriate Uniformed Division officer training data would better position Secret Service to assess Uniform Division officer training data and make informed decisions about whether and how training needs and the 10-percent training are being met. Protecting the White House, the President, and the Vice President, among others, is a zero-fail responsibility. As such, the Secret Service must be prepared to face every evolving threat in a rapidly changing environment. This involves having certain specific security skills and routine training on an ongoing basis. In December 2014, the PMP recommended that the Secret Service align its operations with its priorities, and chief among these is protection. It further recommended, and the Secret Service agreed to, achieving specified training targets. While training alone will not guarantee the safety of the Secret Service’s protectees, developing and implementing a plan for meeting protection- related training targets would better prepare special agents to effectively respond to the security threats faced by the President and other protectees. Further, the Secret Service lacks a documented process for collecting Uniformed Division training data that the agency can use to determine whether officers trained for 10 percent of their work hours, as recommended by the PMP. Implementing such a policy could help the Secret Service make informed decisions about Uniformed Division training. We are making the following two recommendations to the Secret Service: The Director of the Secret Service should develop and implement a plan to ensure that special agents assigned to PPD and VPD reach annual training targets given current and planned staffing levels. (Recommendation 1) The Director of the Secret Service should develop and implement a policy that documents the process for collecting complete Uniformed Division officer training data and establishes the types of information that should be collected. (Recommendation 2) We provided a copy of this report to DHS for review and comment. DHS provided written comments, which are reproduced in appendix II. DHS also provided technical comments, which we incorporated as appropriate. In its comments, Secret Service, through DHS concurred with the two recommendations. However, related to the first recommendation— develop and implement a plan to ensure that special agents assigned to PPD and VPD reach annual training targets given current and planned staffing levels— Secret Service also stated that after further consideration, the agency no longer believes that the annual training target for Presidential and Vice Presidential Protective Divisions should be set at 25 percent of their work time. We incorporated this change in our report. In its comments, the agency stated that the Secret Service Office of Training will work with the Office of Protective Operations to evaluate the training metric for PPD and VPD and develop a plan focusing on increasing capacity at training facilities, achieving staffing growth, and creating efficiencies in protective division scheduling. With respect to the second recommendation—to develop and implement a policy that documents the process for collecting complete Uniformed Division officer training data and establish the types of information that should be collected—Secret Service, through DHS stated that it will develop rigorous and uniform standards for collecting and reporting training data related to the Uniformed Division branch. The agency also stated that it will continue to add training programs to the Performance and Learning Management System and capture informal and on-the-job training hours for the Uniformed Division. DHS stated that the Secret Service expects to review the Enterprise Personnel Schedule System within the next 2 months and anticipates these efforts will result in a more accurate and expansive method for reporting Uniformed Division training. We are sending copies of this report to the appropriate congressional committees and the Acting Secretary of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or AndersonN@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In October 2014, the then-Secretary of Homeland Security established the United States Secret Service Protective Mission Panel (PMP) and tasked the independent panel to review the security incident that occurred at the White House on September 19, 2014 as well as related security issues. The PMP made 19 public recommendations—as well as additional classified recommendations—to the U.S. Secret Service (Secret Service) in three areas: (1) training and personnel; (2) technology, perimeter security, and operations; and (3) leadership. In this appendix, we list the 19 recommendations and accompanying text as published in the Executive Summary to Report from the United States Secret Service Protective Mission Panel to the Secretary of Homeland Security, dated December 15, 2014. The Secret Service Recruitment and Retention Act of 2018 includes a provision that we report on a detailed summary of the Secret Service’s progress implementing the PMP’s recommendations. Specifically, for each recommendation, we provide our assessment of the Secret Service’s progress, describe some of the actions the Secret Service has taken to implement the recommendations, and identify the actions the agency said it plans to complete. Provide a true “Fourth Shift” for training the Presidential and Vice Presidential Protective Divisions, so that they spend two weeks out of every eight in training, and ensure that Uniformed Division officers are in training for no less than 10 percent of their time. According to the PMP, “Only with constant training can all of the teams at the White House perform the coordinated actions needed to effectively respond.” Status: Implementation in progress. Actions Taken by the Secret Service Summary: The Secret Service instituted a fourth and fifth shift for its Presidential Protective Division (PPD) and a fourth shift Vice Presidential Protective Division (VPD). The fourth shift was created to provide agents with time to participate in training educational opportunities, conduct advances, and take leave. Implementation is still in progress because neither PPD nor VPD special agents consistently used this time to train and are missed the training targets established by this recommendation. In commenting on a draft of this report in May 2019, the Secret Service stated that it no longer agrees with the training target and plans to reevaluate it. The Secret Service does not have a documented process for collecting complete and appropriate Uniformed Division training data that the agency can use to determine whether officers trained for 10 percent of their work hours. The Secret Service adopted the PMP goal for agents assigned to PPD and VPD to train for 25 percent of their regular work hours. However, in fiscal year 2018, according to self-reported data, these agents attended training for about 5.9 percent and 2.9 percent of their regular work hours, respectively. Although the fourth shift was developed to provide special agents assigned to PPD and VPD time away from shift work during which they could attend training, agents have largely not attended training during the fourth shift, according to agency officials. Agents are instead assigned to additional advance assignments or use leave, as agents are not allowed to take leave during the three regular shifts except in an emergency. Additional action(s) Secret Service plans to take: In its FY 2018–FY 2025 Human Capital Strategic Plan and FY 2018 – FY 2025 Training Strategic Plan, the Secret Service stated that increasing staffing levels would allow the agency more flexibility with how it schedules shifts and advance assignments, thereby freeing up special agents’ and Uniformed Division officers’ time for training. The agency plans to have 4,807 agents and 1,797 Uniformed Division officers by the end of fiscal year 2025, up from 3,614 agents and 1,559 officers at the end of fiscal year 2018. The Secret Service also plans to reevaluate the training target for special agents assigned to PPD and VPD. Implement integrated training focused on ensuring that all teams at the White House know their roles in responding to specific threats. According to the PMP, “Teams need to train with the full complement of forces with which they will operate in real life, and the training needs to be provided force-wide, not just to those on duty on the day that training is scheduled.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: To provide integrated training between different Secret Service units, the agency conducted 10 live training exercises and six discussion-based tabletop exercises with personnel from different units in fiscal year 2017. About every two weeks, agents assigned to PPD and VPD each conduct drills and training scenarios, some of which incorporate Uniformed Division officers. In addition, the Secret Service began offering Emergency Action and Building Defense training in October 2014 to Uniformed Division personnel. In fiscal year 2017, the Secret Service conducted 10 live training exercises and 6 discussion-based tabletop exercises with personnel from different units. Among the live exercises, the Secret Service conducted a readiness exercise at the White House in December 2017. About every two weeks, agents assigned to the PPD and VPD conduct drills and training scenarios, some of which incorporate Uniformed Division officers, according to a Secret Service official. These drills take place at the White House, the Naval Observatory, the Department of Treasury building, and the Rowley Training Center. Secret Service began offering Emergency Action and Building Defense training in October 2014 to Uniformed Division personnel. Topics addressed in the course include judgment, firearm control, constitutional law, and emergency medicine. The Emergency Action and Building Defense course is part of the training that new Uniformed Division recruits take. In recent years, the Secret Service conducted joint training exercises with local, state, federal, and foreign tactical units. According to Secret Service officials, the agency conducted 53 of these joint training exercises in fiscal year 2015 through 2018. Train in conditions that replicate the physical environment in which they will operate. According to the PMP, “A security team should also be trained so that it is intimately familiar with the space in which it is operating.” Status: Implementation in progress. Actions Taken by the Secret Service Summary: To train in conditions that replicate the White House, the Secret Service secured approval to build a White House Mockup Training Facility at the James J. Rowley Training Center in Beltsville, Maryland. However, the Department of Homeland Security’s (DHS) fiscal year 2019 budget request to Congress did not include funding for the facility. In 2017, the National Capital Planning Commission approved the Secret Service’s revised master plan for the Rowley Training Center, which includes the White House Mockup Training Facility. The fiscal year 2019 Resource Allocation Plan request submitted by the Secret Service to DHS included $77.4 million for the construction project over 5 years. However, the fiscal year 2019 DHS budget request did not include funding for the facility. Some agent and officer training takes place in the operating environment. According to a Secret Service official, agents assigned to the PPD and VPD run drills and training scenarios about every two weeks. Some of the training takes place at the White House, the Naval Observatory, and the Department of the Treasury building, although most training takes place at the Rowley Training Center. These drills and scenarios sometimes also include Uniformed Division officers. In December 2017, the Secret Service conducted a readiness exercise involving multiple units at the White House. Additional action(s) Secret Service plans to take: The Secret Service will proceed with construction of the White House Mockup Training Facility when funding is available. It was not included in DHS’s fiscal year 2019 budget request. Increase the Uniformed Division, as quickly as can be appropriately managed, by an initial 200 positions, and the Presidential Protective Division by 85 positions. Perform additional analyses and, likely, further increases as necessary. According to the PMP, “Both the Uniformed Division and the Presidential Protective Division are currently stretched beyond their limits.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: The Secret Service increased the number of Uniformed Division officers and the number of agents assigned to PPD by 214 and 151 persons, respectively, from the end of fiscal year 2014 to the end of fiscal year 2018. Further, special agent and Uniformed Division officer external attrition rate declined year-over-year from fiscal year 2016 to fiscal year 2018. The Secret Service also conducted additional analyses to determine optimal staffing levels to be reached by the end of fiscal year 2025. The Secret Service met or exceeded its hiring goals for special agents and Uniformed Division officers in fiscal year 2016 and fiscal year 2018. From the end of fiscal year 2014 to the end of fiscal year 2018, the Secret Service increased the number of agents assigned to PPD from 248 to 399—a net increase of 151 agents—and the number of Uniformed Division officers from 1,345 to 1,559—a net increase of 214 officers. The Secret Service conducted additional analyses and set hiring goals in the FY 2018–FY 2025 Human Capital Strategic Plan. Specifically, by the end of fiscal year 2020, the Secret Service aims to have 3,927 special agents, 1,657 Uniformed Division officers, and 2,366 Administrative, Professional, and Technical staff. By the end of fiscal year 2025, the Secret Service aims to have 4,807 special agents, 1,797 Uniformed Division officers, and 2,991 Administrative, Professional, and Technical staff. Additional action(s) Secret Service plans to take: The Secret Service is planning to continually validate the human capital strategic plan to ensure that staffing levels are responsive to changes in the agency’s operational tempo. The Secret Service also plans to fill administrative jobs that are currently filled by Uniformed Division officers with Administrative, Professional, and Technical employees, so that the Uniformed Division personnel are more focused on protection. Reform and professionalize recruiting, hiring, promotion, and rotation process that puts the most talented, capable individuals in place as efficiently as possible. According to the PMP, “The Secret Service must continue efforts to develop a professionalized recruiting and hiring process that finds talented individuals, evaluates candidates rigorously for the Presidential Protective Division, and hires them quickly.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: The Secret Service hired a Chief Human Capital Officer to run the new Office of Human Resources. In addition, the agency implemented several initiatives to strengthen recruiting, expedite hiring, and clarify the promotion process. The Secret Service reorganized the Office of Human Resources as a stand-alone directorate and hired a Chief Human Capital Officer, who is a professional administrator. The Secret Service developed a National Recruitment Strategy for FY 2016–FY 2020 and the Recruitment and Outreach Plan for FY 2018 to help ensure that the agency is able to meet its staffing requirements through effective and targeted recruitment strategies. Recruitment strategies include increasing the agency’s social media presence, improving the training of its recruiters, and expanding cooperation with the Department of Defense to recruit service members departing the military. The Secret Service developed a number of hiring initiatives, which according to agency officials, reduced time-to-hire for special agents and Uniformed Division officers from an average of 395 days in fiscal year 2016 to 285 days in fiscal year 2018. For example, the agency created Entry Level Assessment Centers that allow applicants to complete several application steps in one week, including the entrance examination, the Special Agent and Uniformed Division Pre- employment Review, and the security interview. The agency also established the Applicant Coordination Center to track applicant processing. In particular, the Applicant Coordination Center brings together a polygraph examiner, a nurse, a security clearance adjudicator, and a human resources specialist to usher candidates through the hiring steps. The agency also began using the web-based Applicant Lifecycle Information System to view applicant materials, process security investigations, send conditional job offers, and track candidates’ progress in one place. The Secret Service published special agent Career Progression guidelines in September 2015 and published a revision to the special agent Merit Promotion Process for agents in May 2017. Additional action(s) Secret Service plans to take: According to agency officials, the Secret Service is currently revising the Uniformed Division Merit Promotion Process and is implementing the Administrative, Professional, and Technical Career Progression Plan. The agency also plans to update and consolidate internal policies for agent and Uniformed Division officer recruitment and hiring. Ensure that the Office of Technical Development and Mission Support proactively reviews and refreshes the Service’s technological footprint. The Service should receive dedicated funds for technology, both within its own budget and within DHS Science and Technology’s budget, to accomplish these tasks. According to the PMP, “Technology systems used on the complex must always remain on the cutting edge, and the Secret Service must invest in technology, including becoming a driver of research and development that may assist in its mission.” Status: Implementation in progress. Actions Taken by the Secret Service Summary: To address current technical capabilities and future needs, Secret Service officials stated that, as of October 2018, the Office of Technical Development and Mission Support was drafting a strategic investment plan. According to Secret Service officials, the agency is continuing to explore new technology to enhance its technological capabilities to mitigate threats, including threats to airspace. According to an agency official, as of October 2018, the Office of Technical Development and Mission Support is drafting a five-year strategic investment plan. The plan is to address current technical capabilities as well as needs into the future. According to Secret Service officials, for more than 10 years, the Secret Service’s Science and Technology Review Committee has met quarterly to discuss protection-related technology requirements. The committee is chaired by the Chief Technology Officer, overseen by the Enterprise Governance Council, and open to representatives from all Secret Service directorates. The Enterprise Governance Council is composed of Deputy Assistant Directors from several Secret Service offices and is responsible for overseeing the agency’s investments in science and technology, information technology, and other capital assets. Also according to agency officials, the Secret Service works with the DHS Science and Technology Directorate, partner agencies, and external stakeholders on technological issues. In particular, the Science and Technology Directorate develops pilot programs based on the Secret Service’s technical requirements. According to an agency official, the Secret Service conducts performance reviews of different technology systems each month with the aim of evaluating the performance of every deployed system at least once per year. In fiscal year 2017, Congress appropriated $2.5 million to the Secret Service for research and development. In addition, the Act appropriated $1.8 billion to the Secret Service for Operations and Support, and the Secret Service allocated $98.2 million to the Secret Service for the Operational Mission Support, which helps to protect the President, Vice President, and others from emerging explosive, chemical, biological, radiological, and cyber threats. The funding for the Operational Mission Support program is divided between technology operations and support; procurement, construction, and improvements; and research and development. According to agency officials, the Secret Service does not receive dedicated technology funds through the DHS Science and Technology Directorate. Additional action(s) Secret Service plans to take: Complete and execute the Office of Technical Development and Mission Support’s 5- year strategic investment plan. The plan is intended to address research and development regarding, among other things, ways to mitigate emerging physical and technical threats and identify additional threats. Replace the outer fence that surrounds the 18 acres of the White House complex to give Secret Service personnel more time to react to intrusions. According to the PMP, “The current seven-and-a-half-foot fence, not just along Pennsylvania Avenue but around the compound’s entire perimeter, must be replaced as quickly as possible.” Status: Implementation in progress. Actions Taken by the Secret Service Summary: As of March 2019, the Secret Service was planning to begin construction of the first phase of the new White House fence in May 2019. As a temporary measure, in 2015, the Secret Service and the National Park Service installed bike-rack barricades about 12 feet in front of the permanent fence. As a temporary measure, in 2015 the Secret Service and the National Park Service installed bike rack barricades about 12 feet in front of the permanent White House fence. According to Secret Service officials, the bike-rack barricades give Secret Service personnel more time to respond to fence-jumpers. The Secret Service additionally installed several interim countermeasures to the existing fences, including additional spikes. The Secret Service is preparing to break ground and begin construction on the Phase I sections of the White House fence in May or June 2019, according to an agency official in December 2018. Phase I includes a fence surrounding the White House and its immediate grounds. The Commission of Fine Arts and the National Capitol Planning Commission approved the Phase I project in January and February 2017, respectively. Additional action(s) Secret Service plans to take: Phase I fence construction is scheduled to begin in May or June 2019. Phase II planning and construction. In its fiscal year 2019 budget request, the Secret Service requested $3 million for preliminary design development of Phase II of the White House fence project. Phase II is to expand the new fence to the Treasury Building and the Eisenhower Executive Office Building. Clearly communicate agency priorities, give effect to those priorities through its actions, and align its operations with its priorities. According to the PMP, “Secret Service’s leadership must make those choices in a manner to ensure that its core protective mission remains first priority.” Status: Implementation in progress. Actions Taken by the Secret Service Summary: The Secret Service has taken steps to communicate that protection is its priority. The agency reiterated its priorities in its 2018– 2022 Strategic Plan and hired a Director of Communications in 2016 to manage the agency’s public affairs efforts and to oversee internal agency communication. However, the Secret Service has not fully aligned its operations with its priorities. For example, in response to the PMP’s 2014 report identifying that the security incident of September 2014 arose from a “catastrophic failure of training,” the Secret Service agreed to having its Presidential and Vice Presidential Protective Divisions train 25 percent of their work hours. Implementation of this recommendation is in progress because its operations do not fully align with the stated priorities, as these divisions are not training at agreed-upon levels. To implement this recommendation, the Secret Service sought to improve internal and external communication efforts. The agency did so by hiring a senior executive Director of Communications in 2016 and forming the Office of Communications and Media Relations in 2017 to manage the agency’s public affairs efforts and to oversee internal agency communication. In addition, in October 2015, the Secret Service developed an internal agency communication platform known as Spark!. The Spark! platform allows all employees to share ideas and submit suggestions on how to improve the agency’s performance and efficiency, thereby improving communication within the agency. The DHS Office of Policy reviewed the dual missions of the Secret Service and issued a report that emphasized the importance of the protective and investigative missions (January 2017). Secret Service officials cited this report as evidence that the agency evaluated its priorities and resource allocation decisions. Secret Service data show that agents increased the share of work hours spent on protection compared with investigation from fiscal year 2014 to fiscal year 2018. Overall, based on our analysis of Secret Service data, 59 percent of special agent hours in fiscal year 2018 were spent on protection and 26 percent on investigations. This is in contrast to 54 percent protection and 36 percent investigations in fiscal year 2014. However, agents worked an average of 2.2 million hours annually on investigations over that period, even though agents assigned to the PPD and VPD did not meet training targets during that time. Despite shifting resources toward protection, the Secret Service’s operations and associated resource allocation do not fully align with its stated priority, protection, because training is an essential component of agents’ protection assignments. Promote specialized expertise in its budget, workforce, and technology functions. According to the PMP, “Filling important administrative functions with agents rather than professional administrators may not be optimal.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: To promote specialized expertise in certain business functions, the Secret Service hired a Chief Operating Officer to run the agency’s business functions and elevated senior civilian (non-agent) executives, including the Chief Technology Officer and the Chief Financial Officer, to lead key staff offices. The agency also hired new professional administrators, instead of promoting special agents, to serve in other senior positions, such as Chief Human Capital Officer and Chief Strategy Officer. The Secret Service reorganized the agency to promote specialized expertise in certain functions. In 2015, the Secret Service established the position of Chief Operating Officer. This principal administrator, who is equivalent in rank to the Deputy Director, directs the agency’s business and programmatic activities, with a focus on improving performance, hiring and retaining personnel, and aligning budgetary and strategic planning efforts. The Secret Service professionalized the leadership of several directorates by elevating or hiring civilian senior executives, instead of placing special agents in these specialized positions. For example, in 2015, the civilian who was serving as Chief Financial Officer was placed in charge of the newly created Office of the Chief Financial Officer. Similarly, the Chief Information Officer was placed in charge of the newly created Office of the Chief Information Officer. Also, in 2015, the civilian Chief Technology Officer was placed in charge of the Office of Technical Development and Mission Support, and the Secret Service hired a civilian from outside of the agency to become Chief Human Capital Officer—a position that was formerly held by a special agent. In 2016, the agency created two additional senior, civilian positions: Chief Strategy Officer and Director of Communications. Secret Service officials stated that the agency is currently developing a training course to instruct senior special agents in the agency budget process. Present a zero-based or mission-based budget that will provide sufficient resources to accomplish its mission, beginning immediately by working within DHS to adopt a workforce staffing model. According to the PMP, “The Service must build a new budget from the ground up by defining its mission, determining what it will take to achieve it, and asking for that. The mission is important enough to justify that approach.” Status: Implementation in progress. Actions Taken by the Secret Service Summary: The Secret Service has incorporated principles of mission- based budgeting in its budget formulation process. According to Secret Service officials, modeling staffing needs is a key part of mission-based budgeting, and personnel costs accounted for about 71 percent of the Secret Service’s fiscal year 2018 budget. The Secret Service developed—and continues to refine—four staffing models that use internal and external data to establish the optimal staffing levels across the agency. Under mission-based budgeting, also known as zero-base budgeting, the agency is to rebuild the budget by clearly defining its mission and desired outcomes and determining what funding level is needed to obtain those outcomes. This process is in contrast to making incremental changes from the prior year’s budget. budgeting process into its overall budget formulation. The Director annually issues priority memos to guide the development of the Secret Service Resource Allocation Plan submissions to DHS. In The Secret Service has worked to incorporate a mission‐based 2016, the Office of the Chief Financial Officer introduced a mission‐ based budgeting approach for developing the FY 2018 – FY 2022 Resource Allocation Plan submission. Further, the Secret Service’s Resource Allocation Plan prioritizes the agency’s needs for inclusion in DHS’s annual budget request to Congress. Part of the Secret Service’s mission-based budgeting approach involved assessing human capital needs. The Secret Service developed four workforce staffing models that provided a basis to identify valid baseline staffing levels for the agency, a key component to the mission-based budgeting process. According to Secret Service officials, these staffing models are designed to ensure that the agency is staffed in such a way that its personnel are properly trained, overtime is minimized, and proper support personnel are in place so that it is fully prepared to meet mission demands. The Secret Service used the results of the staffing models to develop the Secret Service FY 2018–FY 2025 Human Capital Strategic Plan, published in May 2017, which detailed the agency’s plan to increase the workforce to 9,595 total employees by the end of fiscal year 2025. Secret Service officials acknowledge that they still have to strengthen their budget processes. Specifically, they would like to make the budget process more analytical and data-driven. For example, agency officials want to make better use of budget data to support planning and budget requests, such as by combining financial data with programmatic information to better inform budget decisions. Agency officials want to hire one or more individuals who can better interpret and use those data. Additional action(s) Secret Service plans to take: Secret Service officials said they plan to continue to hone the staffing models. For example, the agency plans to include annual leave and increased training levels in the next iterations of the models. Create more opportunities for officers and agents to provide input on their mission and train its mid- and lower-level managers to encourage, value and respond to such feedback. According to the PMP, “Leadership and, even more critically, mid- and lower-level managers, need to make clear that their mission requires that they get things right—and thus that the agency values information out of sync with the status quo or the leadership’s views.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: To improve communication between the workforce and senior leaders, the Secret Service created a platform on its intranet known as Spark! that encourages employees to submit ideas to senior leaders on how to improve the agency’s performance. In October 2015, the Secret Service deployed the Spark! platform, which allows all employees to share ideas and submit suggestions on how to improve the agency’s performance and efficiency. The platform allows two-way communication between leadership and the workforce. The Secret Service reported that in 2017, 96 percent of employees have contributed to a discussion on the site. The agency also reported that, as of June 2018, 51 workforce-generated ideas had been implemented or were being implemented. According to Secret Service officials, several ideas that originated from employees have prompted changes at the agency. These developments include the formation of a new category of employees, known as Technical Law Enforcement in 2018; the introduction of a chaplaincy program in 2017; and the development of the Administrative, Technical, and Professional Career Track. Since 2015, the Secret Service offered a training course on workplace communication called “Enhancing Workplace Communication”. According to agency data, 72 employees took the course in fiscal year 2017 and 2018. From November 2014 to December 2015, the Secret Service contracted with Eagle Hill Consulting to conduct an independent assessment of quality-of-life issues at the Secret Service. The agency workforce was able to provide input through 47 focus groups and an agency-wide anonymous survey. In its final report dated August 22, 2016, Eagle Hill Consulting provided Secret Service management with 22 recommendations to improve quality of life for agency employees. Additional action(s) Secret Service plans to take: The Secret Service plans to introduce additional leadership courses for personnel at all levels. It also plans to remove potential barriers to communication between employees and supervisors by revising merit promotion processes for Uniformed Division officers and for special agents. Lead the federal protective force community. According to the PMP, “Collaboration with protective forces like the Federal Protective Service, the Pentagon Force Protection Agency, the FBI Police, and the State Department’s Bureau of Diplomatic Security and other agencies, especially on technology, could significantly increase opportunities for innovation.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: The Secret Service has engaged with other protective forces, across the federal government, such as the Federal Protective Service, through various mechanisms, including a leadership symposium and technology sharing efforts. According to Secret Service officials, the agency has provided assessments and assistance to other government partners in this area. In 2015, the Secret Service chaired a leadership symposium with other federal agencies to discuss roles, responsibilities, and procedures in the event of a critical incident in the National Capital Region and specifically at the White House. Secret Service officials stated that the agency often consults with federal peers to benchmark capabilities and organizational structures to identify possibilities for improvement. In addition, officials stated that the Secret Service partners with other agencies, leading to technological developments. Developments to date include deployment of a fixed-site sub-sonic detection capability with the Metropolitan Police Department of the District of Columbia; a fixed- site super-sonic detection capability with the U.S. Army; and a vehicle motorcade detection capability with the Department of Defense. The Secret Service’s Protective Intelligence and Assessment Division participates in the International Security Events Group when a Secret Service protectee will be traveling to a high-profile international event, such as the Olympics Games. The International Security Events Group is a working group for over 20 federal security and law enforcement agencies and is managed by the Department of State. According to Secret Service officials, at the request of the Office of Management and Budget, the Secret Service has shared expertise with other agencies, including the U.S. Marshals Service, to help to standardize the protection of cabinet-level government officials. Also according to agency officials, the Secret Service leads the federal force protective community in many areas. For example, the agency’s Hazardous Agent Mitigation Medical Emergency Response team has the capability to detect, mitigate, and response to chemical, biological, radiological, and nuclear attacks on protectees, and the team consults with other agencies on these issues, according to agency officials. Additional action(s) Secret Service plans to take: None. Receive periodic, outside assessments of the threats to and strategies for protecting the White House compound. According to the PMP, “The Secret Service should engage other federal agencies to evaluate the threats that the agency faces and its ways of doing business.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: The Secret Service regularly engages with outside partner(s) to have the threats to and strategies for protecting the White House complex assessed. The Secret Service has a memorandum of agreement in place with partner agencies to ensure that the outside assessments continue. Between 2015 and 2018, non-Secret Service partner(s) assessed the threats to and strategies for protecting the White house compound between two and four times per year. Resume participation in international fora with comparable protective services of friendly nations. According to the PMP, “While most national protective forces do not compare to the Secret Service, those of certain nations are much more similar than they are different.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: The Secret Service, through the Office of Investigations and the Office of Strategic Intelligence and Information, maintains relationships with international partners to share information. In 2018, the Secret Service, among other things, provided training at International Law Enforcement Academies, which are administered by the Department of State, and provided threat assessments to European partners. The Office of Investigations’ International Programs Division provides training to foreign law enforcement organizations through the Department of State’s International Law Enforcement Academies. The Secret Service provides training at all five of the academy’s locations: Bangkok, Thailand; Budapest, Hungary; Gaborone, Botswana; San Salvador, El Salvador; and Roswell, New Mexico. According to agency officials, the Secret Service provided instruction on the agency’s protection methods to over 900 personnel in 2018. The Office of Strategic Intelligence and Information maintains international partnerships to enable the sharing of information and best practices. For example, in 2018, agency officials presented threat assessments to European partners on at least two occasions. Secret Service officials noted that the agency hosts groups of foreign law enforcement personnel for dignitary protection seminars. The seminars are intended to, among other things, encourage future cooperation. Additional action(s) Secret Service plans to take: The Secret Service plans to establish a process for developing proposals to enhance intelligence and operational activities with foreign partners. The agency also plans to formalize a process for senior executive approval of these proposals. Give leadership’s priorities and reform the organization’s sustained attention and hold the agency accountable through to their completion. According to the PMP, “Following through on reforms and recommendations has been an issue for the Secret Service in the past.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: To ensure that the agency implemented the PMP’s recommendations, the Office of Strategic Planning and Policy (OSP) was tasked with overseeing and tracking the PMP’s recommendations. This office also coordinated the development of key strategy documents to guide the agency’s efforts. Secret Service executive leaders tasked OSP to oversee and track the implementation of the PMP’s recommendations. The Office of Strategic Planning and Policy coordinated the development of three key strategy documents: the FY 2018 – FY 2022 Strategic Plan, the FY 2018 – FY 2025 Training Strategic Plan, and the FY 2018 – FY 2025 Human Capital Strategic Plan. Additionally, the OSP refined existing performance measures and is developing additional mechanisms to enhance reporting on performance goals to senior leadership. This includes monthly and quarterly reports on key performance metrics and indicators. In January 2017, the agency revived its Enterprise Governance Council, a deliberative body made up of the deputies from each Secret Service directorate. The Enterprise Governance Council oversees agency-wide priorities, including managing the Resource Allocation Plan process, which prioritizes the Secret Service’s needs for inclusion in the annual budget request. Implement a disciplinary system in a consistent manner that demonstrates zero tolerance for failures that are incompatible with its zero-failure mission. According to the PMP, “It is clear that the rank-and-file—and even very senior current and former members of the Secret Service—do not have confidence that discipline is imposed in a fair and consistent manner.” Status: Implemented; ongoing work may be required to ensure recommendation is sustained. Actions Taken by the Secret Service Summary: The Secret Service established the Office of Integrity in 2013 to centralize and standardize the disciplinary system across the Secret Service. According to agency officials, for each substantiated incident of employee misconduct, the Chief Integrity Officer and Deputy Chief Integrity Officer determine what formal disciplinary action, if any, is warranted. Further, the Discipline Review Board, composed of senior representatives from each directorate, oversees the discipline system and hears appeals from most personnel. (A separate process is in place for members of the Senior Executive Service.) Disciplinary outcomes are detailed in an annual report so as to increase transparency within the agency. The Office of Integrity was established in 2013 to centralize the disciplinary system across the agency. Previously, Special Agents in Charge of field offices had the responsibility of addressing employee misconduct and determining the penalty, according to an agency official. In 2015, the Office of Integrity began publishing an annual discipline report which provides an overview of disciplinary actions taken by deciding officials and analyzes misconduct trends. Additional action(s) Secret Service plans to take: The Secret Service plans to conduct a formal review and periodic analysis of the Office of Integrity to ensure that it is fulfilling its intended purpose. Hold forces accountable for performance by using front-line supervisors to constantly test readiness. According to the PMP, “To be ready for a job where quick reactions and reflexes are critical, supervisors need to drive home to their officers and agents that the front line is constantly being tested.” Status: Implementation in progress. Actions Taken by the Secret Service Summary: The Secret Service introduced new policies and plans to study whether to introduce a “random check” program to test employees’ readiness. In addition, according to Secret Service officials, special agents assigned to PPD and VPD run biweekly drills and training scenarios at the White House complex and at the Vice President’s Residence, while Uniformed Division officers are briefed on emergency actions and responsibilities at shift changes. This recommendation is in progress because it requires additional tests for readiness. In December 2017, the Secret Service conducted scenario readiness exercise involving multiple units at the White House. According to Secret Service officials, the agency runs drills and training scenarios about every 2 weeks. These drills have taken place at the White House, the Naval Observatory (the Vice President’s Residence), and at the Department of the Treasury building. Agents assigned to PPD and VPD are the primary training participants, but Uniformed Division officers are often involved as well. More recently, the training scenarios have included other groups if the practice incident is off, or near the edge, of the White House complex. For example, training scenarios may include U.S. Park Police or Secret Service personnel assigned to the Washington Field Office. The Uniformed Division conducts daily personnel shift briefs, which cover emergency actions and responsibilities. The Office of Protective Operations instituted training classes for personnel assigned to the Joint Operations Center. The Joint Operations Center is located away from the White House and is responsible for managing day-to-day Secret Service operations and coordinating emergency response. Additional action(s) Secret Service plans to take: The Secret Service plans to study whether to introduce a “random check” program to test employees on their responsibilities at operational posts. The next director of the Secret Service should be a strong leader from outside the agency who has a protective, law enforcement, or military background and who can drive cultural change in the organization and move the Secret Service forward into a new era. According to the PMP, “The need to change, reinvigorate, and question long-held assumptions—from within the agency itself—is too critical right now for the next director to be an insider.” Status: Implemented. Actions Taken by the Secret Service Summary: Randolph “Tex” Alles was appointed the Secret Service director in 2017 and was in that position until 2019. He had not worked at the Secret Service prior to taking on this role, but he served previously as Acting Deputy Commissioner of U.S. Customs and Border Protection and in the U.S. Marine Corps. This recommendation was implemented through presidential action, as the Secret Service does not select its own director. Establish a leadership development system that identifies and trains the agency’s future managers and leaders. According to the PMP, “To promote from within and move the agency forward, however, the Secret Service needs to do a better job of identifying future leaders and preparing them for the role.” Status: Implementation in progress. Actions Taken by the Secret Service Summary: The Secret Service has taken steps to improve its leadership development system and has provided leadership training at different levels with a focus on first-line supervisors. The agency is also developing the “Framework for Leadership Development,” which is to identify courses and training opportunities to promote leadership skills at all levels. The Secret Service adopted the DHS Leadership Development Program in 2015 to encourage leadership development at all levels. The agency has provided leadership training at different levels. For example, 409 employees have attended the Seminar for First Line Supervisors since 2015. Since 2016, 178 employees attended the Antietam Leadership Experience, which is a course for supervisors, managers, and senior team leads that focuses on leadership development and capabilities. Since 2018, 41 employees attended the Building Leaders Training Course, which is designed for non- supervisory team leads. The Office of Human Resources, in coordination with the Rowley Training Center, has begun to develop a “Framework for Leadership Development” program to craft effective courses and training requirements tailored to individuals throughout their careers. The Secret Service put on a number of events to emphasize the importance of leadership within the agency in fiscal year 2018 as part of the DHS’s “Leadership Year,” a department-wide effort to promote leadership skills. An intranet site on Leadership Year Resources was also created to centralize information on leadership development resources. In addition, the Director of the Secret Service recorded a video message on leadership that was posted on the agency’s intranet. The agency also established a peer-to-peer award recognition program. The agency established the Leadership Development Council in March 2018, with representatives from each of the four occupation groups (special agents; Uniformed Division officers; Administrative, Professional, and Technical staff; and Technical Law Enforcement staff) and across all grade levels. Additional action(s) Secret Service plans to take: Finalize the Framework for Leadership Development and roll out the program to the agency. Complete development of the Strategic Leadership Course for Managers, which is designated to be a two-week course to promote leadership and strategic planning. Nathan Anderson, (202) 512-3841 or andersonn@gao.gov. In addition to the contact above, Joseph P. Cruz (Assistant Director), Kisha Clark (Analyst-in-Charge), Willie Commons III, Elizabeth Dretsch, Eric Hauswirth, and Eric Warren made key contributions to this report.", "summary": "The Secret Service, a component of the Department of Homeland Security (DHS), is responsible for protecting the President, the Vice President, and their families, as well as the White House complex. In October 2014, following several security lapses, the Secretary of Homeland Security established the Panel, an independent panel of experts, to review White House security and other aspects of Secret Service operations. The Secret Service Recruitment and Retention Act of 2018 contains a provision for GAO to report on the progress made by the Secret Service in implementing the Panel's recommendations. This report addresses the extent to which the Secret Service has implemented the recommendations in the Panel's 2014 report. GAO reviewed Secret Service documents, analyzed agency training and labor-distribution data from fiscal years 2014 through 2018, and interviewed agency officials and Panel members. The U.S. Secret Service (Secret Service) has made progress implementing the 19 recommendations related to training and personnel; technology, perimeter security, and operations; and leadership made by the U.S. Secret Service Protective Mission Panel (Panel). The Secret Service fully implemented 11 of the recommendations. For example, the agency increased the number of agents and officers in the divisions that protect the President and White House and secured approval to build a new fence around the White House complex. The Secret Service is in the process of implementing the remaining eight recommendations. The Panel found that the security incident of September 19, 2014, when an intruder jumped the north fence and entered the White House, arose from a “catastrophic failure of training.” The Panel recommended, and the Secret Service agreed, that the Presidential and Vice Presidential Protective Divisions train for 25 percent of their work time. However, the Secret Service has not met this target and lacks a plan for achieving it. In fiscal year 2018, special agents assigned to these divisions trained for about 6 percent and 3 percent, respectively, of their regular work hours (see figure). In commenting on a draft of this report in May 2019, the Secret Service stated that it no longer agrees with the training target and plans to reevaluate it. Developing and implementing a plan for ensuring that the established training target is met given current and planned staffing levels would better ensure that agents assigned to the Presidential and Vice Presidential Protective Divisions are prepared to carry out Secret Service's protection priority. In addition, the Secret Service does not have a policy with a documented process for collecting complete and appropriate (i.e., protection-related) training hour data for Uniformed Division officers. Implementing such a policy will better position the Secret Service to assess the training data and make informed decisions about whether and how training needs are being met. GAO is making recommendations to the Secret Service: (1) develop and implement a plan to ensure that special agents assigned to the Presidential and Vice Presidential Protective Divisions reach annual training targets, and (2) develop and implement a policy that documents the process for collecting complete and appropriate data on Uniformed Division officer training. DHS concurred with the two recommendations.", "document_type": "gao"}
{"report": "In February of each year, the President submits the budget request for VHA health care, which includes requested funding for the upcoming fiscal year as well as an advance appropriation request to Congress. VHA allocates funds by the beginning of each fiscal year—October 1—to VISNs and medical centers based on the amount VA received in the advance appropriation. Once appropriations are enacted for the upcoming fiscal year, VHA updates the allocated funding levels for VISNs and medical centers. For example, VHA’s appropriations for fiscal year 2018 included advance appropriations for fiscal year 2019. VHA allocated funds by October 1, 2018 (the beginning of fiscal year 2019), based on the advance appropriation for fiscal year 2019 and updated the funding levels once appropriations for fiscal year 2019 were enacted. VHA allocates general purpose funds to its 18 VISNs through the Veterans Equitable Resource Allocation (VERA) model. It uses a separate model, called the Medical Center Allocation System (MCAS), to allocate each VISN’s general purpose funds—as determined by the VERA model—to the medical centers within each network. VHA guidance permits VISNs to make adjustments to the general purpose funding levels determined by MCAS for each medical center. VHA uses other methods to allocate specific purpose funds to VHA program offices that manage various health care programs, such as those for community care, prosthetics, and homelessness. The program offices, in turn, typically allocate funds for these programs directly to medical centers. (See fig. 1.) Once funds are allocated and distributed to VISNs and medical centers, these funds may be redistributed in accordance with law across VA’s health care system. These redistributions can help address unfunded needs or surpluses that may arise. For example, according to officials, a medical center may need additional funds to provide care for veterans when natural disasters occur. From fiscal year 2015 to fiscal year 2019, general purpose funds increased by 33 percent- from $37 to $49 billion—while specific purpose funds increased by 24 percent—from $19 to $23 billion. (See fig. 2.) In fiscal year 2019, community care accounted for $10.5 billion—46 percent—of the $23 billion allocated in specific purpose funds. Patient care services, homelessness programs, non-recurring maintenance, and medical residency programs also accounted for large portions of specific purpose funds. As of fiscal year 2017, VA primarily receives appropriated funds for all health care it provides or purchases through four appropriation accounts. The amount of funds in each appropriation account is determined by VA’s annual appropriation. VHA allocates both general and specific purpose funds from these appropriation accounts. These accounts include the following: Medical Services: health care services provided to eligible veterans and other beneficiaries in VA facilities and non-VA facilities, among other things. Medical Community Care: health care services that VA authorizes for veterans and other beneficiaries to receive from community providers. Medical Support and Compliance: the administration of the medical, hospital, nursing home, domiciliary, supply, and research activities authorized under VA’s health care system, among other things. Medical Facilities: the operation and maintenance of VHA’s capital infrastructure, such as the costs associated with nonrecurring maintenance, leases, utilities, facility repair, laundry services, and groundskeeping, among other things. Separate from VA’s health care appropriation accounts, the Veterans Access, Choice, and Accountability Act of 2014 established the Veterans Choice Fund and provided $10 billion in funds for the Veterans Choice Program (Choice Program), which was implemented in early fiscal year 2015 and authorized until funds were exhausted or through August 7, 2017, whichever occurred first. The Choice Program allowed veterans to elect to receive care from community providers when the services needed were not available at a VA medical center, were not available within VHA’s wait-time goals, or when veterans did not reside near a VA medical facility with a full-time primary care provider. Eligible veterans could also elect to receive care in the community if they met other eligibility criteria as well. VA received additional authority and funds to maintain the Choice Program through June 6, 2019, when it sunsetted, and the new Veterans Community Care Program (VCCP) went into effect. The VCCP was established by the VA MISSION Act and consolidated the Choice Program along with several other community care programs. The VCCP is primarily funded through specific purpose funds in the Medical Community Care appropriation account. The VCCP is similar to the former Choice Program in allowing veterans to elect to receive care from community providers when certain eligibility criteria are met, including criteria relating to the availability and accessibility of the services at VHA. Under the VCCP, VHA adopted designated access standards for VCCP eligibility determinations that are broader than the eligibility criteria that existed under the Choice Program. VHA’s VERA model uses a national, formula-driven approach that considers the number and type of veterans served and the complexity of care provided—collectively referred to as patient workload—as well as certain geographic factors, such as local labor costs, to determine the amount of general purpose funds each VISN should receive. VHA uses VERA to establish funding levels for each VISN in the following areas: patient care, equipment, education support, and research support, the largest of which is patient care. After determining the amount of funds VISNs should retain for VISN-level initiatives, administrative purposes, and reserves, VHA uses its MCAS model to distribute the remainder of each VISN’s general purpose funds to medical centers within the VISN. MCAS is based on a workload measure developed by VHA, called patient-weighted work (PWW) that accounts for medical center-level factors such as patient volume, case- mix, and specialized services. According to VHA officials, PWW establishes an equitable measure of workload among medical centers that vary significantly in their geographic location, and types and costs of services provided. Furthermore, PWW lessens the impact of cost differences between medical centers, by recognizing the varying costs and levels of resource intensity associated with providing care for each patient at each medical center. For example, PWW would result in more funds being allocated to a medical center that provides more complex care, such as open heart surgery, than a workload measure based solely on a count of each individual patient, which would not account for the additional costs associated with more complex care. Similar to MCAS, VHA’s Office of Community Care uses a patient workload-based model to allocate community care funds—which are specific purpose funds—to medical centers, based on each medical center’s community care patient workload in prior years. To determine the community care funding needs for each medical center, VHA calculates the PWW associated with community care. VHA determines the total amount of funds available for community care based on the amounts appropriated to the Medical Community Care appropriation account and the amount available for community care in the Veterans Choice Fund. VHA distributes the funds to each medical center in proportion to each medical center’s PWW. VHA officials told us that VHA is considering making changes to the methodology for allocating community care funds under the new VCCP, but as of July 2019, updates to the methodology had not been developed or implemented. The other four program offices we reviewed developed other methodologies for allocating other specific purpose funds. In general, these methodologies involve coordination between VISNs and their medical centers on needs for these funds and allocating available funds based on identified needs. For example, the Office of Patient Care Services which manages prosthetics and hepatitis C drugs allocates available funds based on identified needs by each of the medical centers. Appendix 1 provides an overview of the methodologies used by these four program offices to allocate special purpose funds. To allocate funds for an upcoming fiscal year, VHA’s allocation models rely on actual patient workload data from prior fiscal years, but not the most recently completed fiscal year. VHA’s VERA model relies on actual patient workload data from two to six years prior to the upcoming fiscal year, in addition to future workload projections. For example, to allocate funds for fiscal year 2019, the VERA model relied on actual workload data from fiscal years 2013 and 2017, in addition to some future projected workload, but did not take into account actual workload data from fiscal year 2018. VHA’s MCAS and community care models rely on actual patient workload data from two years prior to the upcoming fiscal year. For example, the 2019 MCAS and community care models were based on actual workload data from fiscal year 2017, but did not take into account actual workload data from fiscal year 2018. According to VHA officials, patient workload data from the most recently completed fiscal year are not yet available when VHA runs the preliminary VERA, MCAS, and community care models for each fiscal year in August. However, these officials told us that these data would be available in December of each year and therefore could be incorporated into the final model run after VHA receives its enacted appropriation amount for the upcoming fiscal year. These officials told us that doing so would result in little to no delay in when the final model run and the final distribution of funds takes place, which occurs after the appropriation act is enacted. Specifically, according to officials from VHA’s Office of Finance, if the full fiscal year appropriation is enacted prior to the start of the fiscal year on October 1, VHA will be able to perform the final model runs by mid- November. As a result, incorporating data from the most recent fiscal year would result in a one month delay in the final model run. Should the enactment of a full year appropriation be delayed, the timing of the final model would not be impacted by using data from the most recently completed fiscal year. (See fig. 3.) A VHA Office of Finance official told us that VHA had not previously considered using patient workload data from the most recently completed fiscal year because VHA did not believe that using updated data would have a significant impact on the model. However, the official told us that the implementation of the VCCP in June 2019 may result in more significant year-to-year workload changes due to veterans increasing their use of VHA health care services. As a result, the official told us that using more up to date information would be more useful in informing allocation levels. Federal standards for internal control related to information calls for management to use quality information to achieve the entity’s objectives. Quality information is appropriate, current, complete, accurate, accessible, and provided on a timely basis. Because the VERA, MCAS, and community care models do not use the most up-to-date patient workload data available, the models may not reflect the most recent workload trends affecting medical centers. This may result in funding levels determined by the models that may not be commensurate with medical centers’ actual patient workload. For example, VHA data we reviewed show that some medical centers experienced workload changes in fiscal year 2018—changes that were not captured by the models for fiscal year 2019 allocations. Specifically, from fiscal years 2017 through 2018, while PWW for care provided by VA medical centers grew over 1 percent VHA-wide, 34 medical centers experienced growth of over 3 percent, and 9 experienced a decline of over 3 percent. Similarly, the PWW for care in the community grew over 6 percent VHA-wide from fiscal year 2017 to fiscal year 2018, with 97 medical centers experiencing growth of over 3 percent and 25 experiencing a decline in community care of over 3 percent over this time period. Additionally, officials we interviewed at six VISNs told us that the models have not accounted for recent workload growth their medical centers were experiencing due to an increase in the number of veterans they serve, the addition of new services, or changes in the medical centers’ reliance on community care. Two of these VISNs analyzed recent workload trends at the medical centers within their VISN and allocated additional funds to those medical centers with recent growth not accounted for by MCAS. If VHA were to incorporate the most recent available workload data into its allocation models, the need for such funding changes would likely be reduced. As part of the allocation process, VISNs may make adjustments to the amounts of general purpose funds calculated by MCAS and allocated to medical centers. VHA guidance requires VISNs to provide a written explanation to VHA for any adjustments. However, we found that VHA does not adequately monitor these adjustments and that some of these adjustments may lead to inefficiencies. We found that VHA Office of Finance officials did not adequately review the fiscal year 2019 MCAS adjustments to ensure that adjustments were documented and fully explained. Specifically, VHA did not provide evidence that they sought an explanation for MCAS adjustments made by 2 VISNs that provided no written explanation for their adjustments, even though these explanations are required by VHA guidance. VHA Office of Finance officials said they use informal methods via email to learn about the adjustments and follow-up as needed, but could not provide documentation that follow-up and review had occurred. Additionally, VISN officials we interviewed from all 18 VISNs stated that they had not received questions or other feedback from the VHA Office of Finance on the adjustments they made, even if they had not documented and explained the adjustments. Furthermore, even if VISNs follow the requirement and submit written explanations for the adjustments, they may not provide the type of information VHA needs to adequately monitor the adjustments. This is because VHA guidance does not require VISNs to provide information on how they determined how much and for what reasons they are making the adjustments. For example, we found that 6 VISNs provided limited explanations for their fiscal year 2019 MCAS adjustments, such as stating that they had decided to reallocate funds among medical centers to ensure “continuity of operations,” which is insufficient information to allow VHA to determine if the adjustments were appropriate. Federal standards for internal control related to monitoring state that management should establish and operate monitoring activities to monitor the internal control system and evaluate the results. These monitoring activities could include establishing a formal process to document VHA’s review of VISN adjustments to medical center allocations. Additionally, monitoring activities could include requiring VISNs to provide information on how they determined how much and for what reasons they are making the adjustments and then reviewing such information. As VHA evaluates the adjustments, documenting the results of its monitoring and having the information needed to help determine the appropriateness of the adjustments will help VHA identify areas for improvement in the allocation process. Without adequate monitoring, VHA cannot reasonably ensure these adjustments are justified and align with VA’s strategic plan, which calls for the efficient allocation of funds. Based on interviews with VISN officials, we found that, in fiscal year 2019, seven VISNs adjusted the allocations determined by MCAS to ensure that every medical center within their VISN received either the same level of funding or a minimum funding increase of up to 2 percent relative to the prior year. According to VISN officials, funds were often shifted from medical centers that had received relatively large increases in funds due to growing workload to medical centers that had received a decrease or relatively flat funds compared to the prior year due to either declining or relatively flat workload. According to VISN officials, declining workload may be the result of medical centers serving fewer patients or patients obtaining care from community providers rather than VA providers. When asked about these adjustments, officials at the seven VISNs stated that they were necessary to ensure that affected medical centers could continue to cover the costs for the services they offer and the staff they employ, including providing federally mandated annual salary increases for those staff. Officials from four of these VISNs stated that it is difficult for medical centers to absorb a funding cut or only a small increase in funding from one year to the next due to rising costs they face. While VISNs are allowed to make adjustments to medical centers’ allocated general purpose funds, these adjustments may lead to inefficiencies because medical centers are not required to improve efficiency—such as, adjust the level of services they offer—to account for their decreases in workload. Additionally, officials from VHA’s Allocation Resource Center within the Office of Finance, which is responsible for developing and executing VHA’s allocation models, told us that because allocations made through MCAS are based on medical center workload, VISNs should avoid reallocating funds so that all medical centers receive a minimal increase. These officials said that doing so results in medical centers failing to adjust the level of services to meet workload needs. However, we found that for medical centers with declining workload, VHA guidance on allocation of funds does not require VISNs—in conjunction with these medical centers—to develop and submit approaches to improve the efficiency of medical center operations. Such improvements in efficiency would help lower overall costs. As we have previously stated, VHA’s strategic plan calls for the efficient allocation of funds. In addition, federal internal control standards related to control activities state that management should design control activities to achieve agency objectives. Such an activity could include having guidance on the allocation and adjustment of funds that promotes the efficient use of funds for delivering health care services to veterans. Without requiring VISNs—in conjunction with medical centers—to develop and submit an approach to change how medical centers with decreasing workload should operate, VHA increases the risk that these adjustments will not align with VA’s strategic plan. Once funds are allocated and distributed to VISNs and medical centers, VHA uses multiple mechanisms to monitor the balance of general purpose and specific purpose funds. VA uses these mechanisms to ensure that VISNs and medical centers operate within their allocated funding levels and are in compliance with the Anti-Deficiency Act. VHA’s primary monitoring mechanism is through VA’s financial management system, which is used to track obligations and prevent VISNs and medical centers from obligating amounts that are greater than the funds they have available. VHA also employs additional mechanisms to monitor the use of general and specific purpose funds. These additional mechanisms are described below. VHA’s Office of Finance requires each VISN to prepare an annual operating plan after the initial allocation of general purpose funds for each fiscal year that reflects the total planned obligations for the medical centers they oversee. These operating plans describe the planned obligation of funds throughout the fiscal year for various budget categories, such as personnel, equipment, transportation, and supplies and materials. VHA requires planned obligations reported in operating plans to align with the funding levels available to each VISN, which include allocated funds as well as anticipated collections, reimbursements, and funds carried over from previous years. VISNs are required to revise their operating plans during the fiscal year if major changes are made to their available funding levels, due to, for example, the enactment of a final appropriation bill, which results in final allocations. To monitor VISNs’ use of general purpose funds, VHA uses the operating plans to compare each VISN’s planned obligations with actual obligations on a monthly basis. VHA does not compare planned obligations with actual obligations for each medical center individually; instead, each of the 18 VISNs as well as the five medical centers we reviewed developed their own tools to monitor the use of funds. According to VHA officials, VHA requires each VISN to provide an explanation to VHA’s Office of Finance on a monthly basis about any variances of 5 percent or more above or below the amount between planned obligations in their operating plans and actual obligations. Based on VHA documents we reviewed, all 18 VISNs provided explanations for situations in which their actual obligations were equal to, higher, or lower than 5 percent from their planned obligations in fiscal years 2018 and 2019 and in some cases, also explained the actions they were planning to take to address the variance. VISNs reported several reasons for the variances. For example, some VISNs reported that their actual obligations exceeded planned obligations in some months because contracts or equipment purchases were executed earlier than anticipated in the year. Conversely, some VISNs reported that contracting delays led to actual obligations lagging behind planned obligations reported in their operating plans. An official from the VHA Office of Finance told us that they may contact VISN leadership—including the Director and Chief Financial Officer—if the variations are significant and additional actions needed to be taken. VHA Office of Finance officials also told us that they may review other reports if they become aware of an issue of significant interest to VHA leadership regarding a VISN’s or medical center’s obligations. Based on our review of VHA documents and interviews with program office officials, VHA program offices use various monitoring processes developed by each program office to monitor the use of specific purpose funds. Specifically, officials from the Office of Community Care told us that they monitor the use of community care funds by comparing actual obligations to planned obligations based on authorized community care. According to VHA officials, as of February 2019, VHA was in the process of developing an updated process to monitor the use of community care funds, which—starting in fiscal year 2019—were obligated at the time of claim payment rather than when care in the community was authorized. The other four VHA program offices we reviewed monitor the use of the funds they manage by generating a monthly or quarterly budget status report that compares each medical center’s actual obligations against their planned obligations. For example, officials from the Office of Patient Care Services told us that to monitor the use of funds for prosthetics, they conduct monthly reviews of obligations and ask the VISNs and medical centers to explain deviations between the actual and planned expenditures and provide an action plan. After funds are allocated and distributed to VISNs and medical centers, VHA can redistribute funds across the VA health care system in accordance with law. VHA officials told us these redistributions are done to address unfunded needs and surpluses as they are identified and occur throughout the year. However, we found that VHA does not adequately monitor redistributions. We identified the following instances in which allocated funds are redistributed: VHA officials told us that VHA’s Office of Finance redistributes any surplus general purpose and specific purpose funds to medical centers based on VHA priorities and to address needs identified by VISNs. These officials said that these redistributions typically occur after the middle of the fiscal year. As of June 2019, according to VHA officials, one VISN had identified unfunded needs to VHA, but the VISN was working on addressing the funding needs using its own internal resources. Officials from another VISN told us that the VISN anticipated unfunded needs, but had not made a request to VHA for additional funds as of the end of May 2019. VISNs may also exchange funds with other VISNs. For example, if a VISN has excess medical facilities funds but a shortage of medical services funds, the VISN may exchange these funds with another VISN that has excess medical services funds but a shortage of medical facilities funds. According to VHA officials, VISNs must inform VHA about these exchanges of funds, but are not required to provide an explanation and do not require VHA approval for the exchanges. VHA officials told us that VISNs also have the flexibility to redistribute funds throughout the year from medical centers within their VISN that are experiencing a funding surplus to ones with unfunded needs. However, VISNs are not required to inform VHA about these redistributions and are not required to provide an explanation or get approval from VHA. For example, officials at one medical center told us that in recent years, its VISN redistributed an average of $15 million per year above allocated funding levels to this medical center to address unfunded needs. While the redistribution of funds throughout the year gives VHA flexibility to move funds where they are needed, VHA’s Office of Finance does not adequately monitor these redistributions. Specifically, VHA’s Office of Finance does not require VISNs to identify the reasons why they redistribute funds between VISNs and medical centers, and a VHA Office of Finance official told us VHA does not examine the amount of funds that are redistributed. For example, VHA’s Office of Finance could not provide us the total amount of redistributions that occurred throughout fiscal year 2018. As a result, VHA’s Office of Finance does not know why VISNs redistributed funds and the extent to which redistributions resulted in a deviation from VHA’s workload-based allocation levels. Monitoring the redistributions would provide VHA with information on the effectiveness of the allocation models and how they might be improved. VHA’s actions are inconsistent with federal internal control standards related to monitoring, which state that management should establish and operate monitoring activities to monitor the internal control system and evaluate the results. Without requiring VISNs to provide this information and without requiring VHA to document the results of its review of the redistributions, VHA cannot ensure that these redistributions align with VHA’s workload-based allocation of funds. As a result, VISNs and medical centers may not be efficiently operating within available funding levels, which include allocated funds, collections, reimbursements, and carry over from previous years. VA’s strategic plan calls for the efficient use of funds for delivering health care services to veterans. Accordingly, it is critical that VHA closely monitor and account for how its funds are allocated to VA medical centers and redistributed throughout the year to help ensure the most efficient use of funds. Especially as the number of veterans eligible to receive care from a community provider potentially expands, it will be important for VHA to ensure allocated funding levels accurately reflect individual medical center funding needs. However, VHA has opportunities to strengthen its processes for allocating and monitoring funds distributed across its health care system. VHA could improve how it allocates funds to its VISNs and medical centers if it were to use the most up-to-date workload data available as part of its allocation models. This would allow VHA to account for significant changes in workload from year-to-year. VHA could also improve how it monitors VISN adjustments to medical center allocation levels as well as redistributions that may occur after medical centers receive their allocations. While these adjustments and redistributions afford flexibility and may be appropriate in certain circumstances, VHA does not have the information it needs to monitor these changes to ensure that they are appropriate and consistent with department goals. Specifically, VHA does not require VISNs and medical centers to provide information on how they determined the amount and reasons for adjustments, nor does VHA require VISNs—in conjunction with medical centers— to develop and submit an approach to improve efficiency at medical centers with declining workload, such as adjusting the level of services offered. Additionally, VHA does not require VISNs to identify the reasons why they redistribute funds between VISNs and medical centers after allocations have been made, and VHA does not document its review of these redistributions. As a result, VHA lacks reasonable assurance that adjustments and redistributions align with its strategic goals for efficient use of funds to best serve the needs of veterans across its healthcare system. We are making the following five recommendations to VHA: The VA Under Secretary of Health should use workload data from the most recently completed fiscal year as part of the models that inform VISNs’ and medical centers’ general purpose funding needs, when doing so would not significantly delay the allocation of funds. (Recommendation 1) The VA Under Secretary of Health should establish a formal process to document VHA’s review of VISNs’ adjustments to medical center allocation levels. (Recommendation 2) The VA Under Secretary of Health should revise VHA’s existing guidance to require VISNs to provide information on how they determined how much and for what reasons they made adjustments to medical center allocation levels. (Recommendation 3) The VA Under Secretary of Health should revise its existing guidance to require VISNs—in conjunction with medical centers—to develop and submit approaches to improve efficiency at medical centers with declining workload that received adjusted funding levels. These approaches could include adjusting the level of services offered. (Recommendation 4) The VA Under Secretary of Health should require VISNs to provide explanations on the amount of funds redistributed between VISNs and medical centers and VHA to document its review of these redistributions. (Recommendation 5) We provided a draft of this report to VA for review and comment. In its written comments, reprinted in appendix II, VA concurred with four recommendations and concurred in principle with one recommendation. VA also described steps that it plans to take to implement the recommendations. In addition, VA provided a technical comment, which we incorporated as appropriate. Specifically, VA concurred with the first recommendation, stating that it will re-run the VERA model to allocate funds based on prior year workload data if an enacted budget is passed after the start of the second quarter of the current fiscal year. VA also concurred with the second and third recommendations, stating that it will update guidance to establish a formal process to document the review of VISN adjustments to medical center allocation levels and will require VISNs to provide information on how they determined adjustments prior to processing the adjustments. VA concurred in principle with the fourth recommendation, stating that VHA is conducting market assessments over a multi-year period to increase access and quality of care to veterans. VA said that after completing the market assessments and reviewing information from other VHA efforts, it may consider adjusting the level of services along with other alternatives. VA also concurred with the fifth recommendation, stating it will require review of redistributions between VISNs to ensure adequate explanations are included. According to VA, the department will also run a monthly report identifying redistributions between medical centers in a VISN that exceed 1.5 percent of the VISN’s funding allocation. We are sending copies of this report to the Secretary of Veterans Affairs, appropriate congressional committees, and other interested parties. This report is also available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or silass@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In addition to the Office of Community Care, which managed $10.5 billion, or 46 percent, of specific purpose funds in fiscal year 2019, we contacted four other Veterans Health Administration (VHA) program offices that managed the largest amounts of specific purpose funds in fiscal year 2019—these included funds for patient care services, homelessness programs, non-recurring maintenance, and medical residency programs. These four program offices managed at least $1 billion of funds and collectively managed about 36 percent of all specific purpose funds in fiscal year 2019. The four program offices developed methodologies for allocating specific purpose funds that involve coordinating with VISNs and their medical centers on the purposes for which the funds would be used and allocating available funds based on these needs. See table 1. In addition to the contact named above, Rashmi Agarwal (Assistant Director), Michael Zose (Analyst-in-Charge), and Carmen Rivera-Lowitt made key contributions to this report. Also contributing were Krister Friday, Cathleen Hamann, Jacquelyn Hamilton, and Ethiene Salgado- Rodriguez. VA Health Care: Estimating Resources Needed to Provide Community Care. GAO-19-478. Washington, D.C.: June 12, 2019. VA’s Health Care Budget: In Response to a Projected Funding Gap in Fiscal Year 2015, VA Has Made Efforts to Better Manage Future Budgets. GAO-16-584. Washington, D.C.: June 3, 2016. Veterans’ Health Care Budget: Improvements Made, but Additional Actions Needed to Address Problems Related to Estimates Supporting President’s Request. GAO-13-715. Washington, D.C.: August 8, 2013. Veterans’ Health Care: Improvements Needed to Ensure That Budget Estimates Are Reliable and That Spending for Facility Maintenance Is Consistent with Priorities. GAO-13-220. Washington, D.C.: February 22, 2013. Veterans’ Health Care Budget: Better Labeling of Services and More Detailed Information Could Improve the Congressional Budget Justification. GAO-12-908. Washington, D.C.: September 18, 2012. Veterans’ Health Care Budget: Transparency and Reliability of Some Estimates Supporting President’s Request Could Be Improved. GAO-12-689. Washington, D.C.: June 11, 2012. VA Health Care: Estimates of Available Budget Resources Compared with Actual Amounts. GAO-12-383R. Washington, D.C.: March 30, 2012. VA Health Care: Methodology for Estimating and Process for Tracking Savings Need Improvement. GAO-12-305. Washington, D.C.: February 27, 2012. Veterans’ Health Care Budget Estimate: Changes Were Made in Developing the President’s Budget Request for Fiscal Years 2012 and 2013. GAO-11-622. Washington, D.C.: June 14, 2011. VA Health Care: Need for More Transparency in New Resource Allocation Process and for Written Policies on Monitoring Resources. GAO-11-426. Washington, D.C.: April 29, 2011. Veterans’ Health Care: VA Uses a Projection Model to Develop Most of Its Health Care Budget Estimate to Inform the President’s Budget Request. GAO-11-205. Washington, D.C.: January 31, 2011. VA Health Care: Challenges in Budget Formulation and Issues Surrounding the Proposal for Advance Appropriations. GAO-09-664T. Washington, D.C.: April 29, 2009. VA Health Care: Challenges in Budget Formulation and Execution. GAO-09-459T. Washington, D.C.: March 12, 2009. VA Health Care: Long-Term Care Strategic Planning and Budgeting Need Improvement. GAO-09-145. Washington, D.C.: January 23, 2009.", "summary": "VHA operates one of the largest health care systems in the nation with an estimate of $81 billion for providing care to over 6.9 million veterans in fiscal year 2019. Recently, VHA has repeatedly requested that Congress provide supplemental funding due to higher-than-expected needs for care. GAO was asked to examine how VHA allocates funds and monitors use of these funds. This report examines (1) VHA's processes for allocating general purpose and specific purpose funds to its VISNs and medical centers and (2) the extent to which VHA monitors the use of these funds. GAO reviewed VHA's processes for allocating funds, analyzed data on allocation levels for fiscal years 2015 through 2019, and reviewed documentation on VHA's processes for allocating funds and monitoring. GAO interviewed officials from VHA; all 18 VISNs; and a non-generalizable sample of five medical centers selected based on size, facility complexity, growth in funding, and geographic variation. The Department of Veterans Affairs' (VA) Veterans Health Administration (VHA) has developed processes for allocating health care funds to its regional Veterans Integrated Service Networks (VISN) and medical centers. Each year, VHA allocates about two-thirds of funds for general patient care—known as general purpose funds—using two, main allocation models. The first model allocates general purpose funds to each VISN and a second model then allocates these funds to the medical centers that report to each VISN. These models are based on patient workload—that is, the number and type of veterans served and the complexity of care provided. VHA allocates its remaining one-third of funds—known as specific purpose funds—to program offices that manage various, specific programs, such as community care and prosthetics. Program offices, in turn, allocate these funds directly to medical centers using different methodologies, including a workload-based model for community care. GAO found the following weaknesses in VHA's processes for allocating funds: VHA's allocation models do not use workload data from the most recently completed fiscal year. For example, the fiscal year 2019 allocation levels determined by the models were based on data from fiscal years 2013 through 2017 but did not include data from fiscal year 2018. The models do not use more recent data because officials believed that doing so would not significantly affect allocations. By not using the most recent data available when it makes final allocations, VHA's allocations may not accurately reflect medical centers' funding needs if they experience workload changes. For example, from fiscal years 2017 through 2018, 34 medical centers had patient workload growth of over 3 percent, and 9 experienced a decline of over 3 percent, which was not reflected in the fiscal year 2019 allocations. VISNs are allowed to make adjustments to allocated funding levels determined by the models and must submit written explanations for doing so according to VHA guidance. However, VHA officials did not adequately review adjustments for fiscal year 2019 to ensure adjustments were documented. Specifically, VHA officials did not provide evidence they sought an explanation for adjustments made by two VISNs that provided no written explanation for their adjustments. Furthermore, GAO also found that VHA guidance does not require VISNs to explain how they determined adjustment amounts and why they made them. Without requiring this information, VHA cannot ensure that these adjustments lead to efficient use of funds. Once VISNs have made adjustments to allocated funding levels and funds are distributed to VISNs and medical centers, VHA uses multiple mechanisms to monitor the balance of funds. Throughout the year, VHA redistributes funds across the VA health care system to address unfunded needs and surpluses that are identified. However, GAO found that VHA does not adequately monitor the redistribution of allocated funds between VISNs and medical centers. VHA does not require VISNs to provide explanations for redistributions and does not review the amount redistributed. As a result, VHA does not know the extent to which redistributions deviate from workload-based allocations and if VISNs and medical centers are operating efficiently. GAO is making five recommendations including that VHA use workload data from the most recently completed fiscal year to allocate funds; take steps to review adjustments; revise existing guidance to require VISNs to provide information on adjustment amounts and the reasons for doing so; and require VISNs to provide explanations for redistributions of allocated funds between VISNs and medical centers and then review the amounts redistributed. VA concurred with four recommendations and concurred in principle with one recommendation.", "document_type": "gao"}
{"report": "Congress appropriated $12.8 billion in federal funds under Part B of IDEA for fiscal year 2019. Under IDEA, Education awards funds to state educational agencies (SEA), which provide these funds to local educational agencies (LEA). SEAs also monitor Part B implementation by the school districts. As a condition of receiving IDEA funds, states are required to have policies and procedures in effect that are consistent with IDEA requirements, including requirements related to procedural safeguards and due process procedures. IDEA requires states to make dispute resolution options available, which parents may use to resolve disagreements regarding a school district’s decisions related to the identification, evaluation, and educational placement of their child with a disability, or the provision of a free appropriate public education (FAPE) to the child. These options include: Mediation. Mediation is a confidential, voluntary process in which a trained, qualified, and impartial mediator, paid for by the SEA, works with the parents and school district to try to reach an agreement about the IDEA-related issue in dispute. Mediations can be initiated by either the parent or the school district to resolve any dispute related to IDEA, including matters that arise before filing of a due process complaint. If agreement is reached through the mediation process, the parties must execute a legally binding agreement. Due process complaint. A due process complaint is a request for a formal due process hearing. A due process hearing is conducted before a qualified and impartial hearing officer and involves presentation of evidence, sworn testimony, and cross-examination. It often involves attorneys and expert witnesses, and thus may be more costly than other dispute resolution options for all parties involved. Because a due process hearing is a formal proceeding, it may be more adversarial in nature than other dispute resolution options. Either party can appeal a hearing officer’s decision by bringing a civil action in any state court of competent jurisdiction or in a U.S. district court. Not all due process complaints result in a due process hearing. For example, some due process complaints may be withdrawn by the parents or not meet the requirements for a filing a complaint under IDEA regulations. In addition, in some cases, the parents and school district may resolve the complaint through alternative means, such as mediation. The 2004 IDEA reauthorization added the requirement for a resolution meeting to the due process complaint process to try to resolve the issues in a parent’s due process complaint collaboratively before the parties may proceed to the formal and often costly due process complaint hearing procedure. A resolution meeting must take place within 15 days of a parent filing a due process complaint and before any due process hearing involving a hearing officer, unless both parties agree in writing to waive the meeting or agree to use the IDEA’s mediation process. Settlement agreements reached through resolution meetings must be in writing and are legally binding. State complaint. An individual or an organization, including one from another state, may file a complaint with the SEA alleging that a public agency has violated a requirement of Part B of IDEA or its implementing regulations. Once the SEA receives such a complaint, it must engage in specified procedures to resolve the complaint, including conducting an on-site investigation, if the SEA determines that it is necessary. Generally, the SEA must issue a written decision within 60 calendar days unless exceptional circumstances warrant an extension or the parties agree to extend the timeline to engage in an alternative dispute resolution procedure. The SEA’s written decision must include findings of fact and conclusions and the reasons for the SEA’s final decision. The state’s complaint procedures must include steps for effective implementation of the SEA’s final decision, including any corrective actions to achieve compliance, if needed. IDEA also requires school districts to provide parents with a procedural safeguards notice, which explains all of the procedural safeguards available to them under IDEA. Education’s Office of Special Education Programs (OSEP) administers IDEA, and is responsible for data collection and monitoring, among other responsibilities. Data collection. Under IDEA, SEAs are required to annually report to Education data on the use of mediation and due process procedures. Specifically, SEAs report data to OSEP, including the total number of: mediation requests received, mediation agreements reached (related to a due process complaint or not related to a due process complaint), due process complaints filed, resolution meetings that result in a written settlement agreement, and due process hearings conducted. Each state also reports data on the timely resolution of state complaints and timely adjudication of due process complaints. According to Education officials, all dispute resolution data are aggregated at the state level and Education does not collect dispute resolution data at the school or district level. According to Education officials, Education’s collection of state-level dispute resolution data is consistent with the manner in which grant awards are made for Part B of IDEA. Because states are the grantees, it is the states that report data to Education. Education’s monitoring. IDEA requires Education to monitor SEAs to ensure they meet program requirements. According to Education officials, Education uses multiple methods to monitor states’ implementation of IDEA, including reviewing data submitted by the states in their state performance plans and annual performance reports, conducting on-site monitoring visits to some states each year, and following up on concerns raised via customer calls and letters. Based on its monitoring and review of state dispute resolution data, among other information, Education is required under IDEA to annually determine whether each state meets the IDEA requirements or needs assistance or intervention. Education’s technical assistance. In addition to providing technical assistance to states, Education provides technical assistance to parents and the general public through its Parent Training and Information Centers (PTI) and CADRE. PTIs are designed to help parents of children with disabilities participate effectively in their children’s education. Education’s technical assistance covers a range of topics, including IDEA dispute resolution options. States’ responsibilities. While Education monitors states, IDEA requires states to monitor and conduct enforcement activities in their school districts. States are also responsible for investigating state complaints and producing reports with the results of their investigation, as well as providing mediators as needed to mediate disputes between school districts and parents. States may also provide other support and direct services such as training and technical assistance among other activities. For the 6.8 million students from ages 3 to 21 who were served under IDEA Part B in school year 2016-17, there were a total of 35,142 mediation requests, due process complaints filed, and state complaints filed nationwide. Over about the last decade, this total decreased by about 2 percent, according to data from the Center for Appropriate Dispute Resolution in Special Education (CADRE). In addition, the mix of dispute resolution options used has changed. Since school year 2004-05, the number of due process complaints declined, while the number of mediation requests increased. However, due process complaints still made up more than half the total number of dispute resolution options used in school year 2016-17 (see fig. 1). Due process complaints. While the overall number of due process complaints has declined since school year 2004-05 (from 21,118 to 18,490) the percentage of fully adjudicated due process hearings (i.e., due process complaints that went all the way through the hearing process and a hearing officer rendered a decision) has declined more sharply. In school year 2004-05, about 35 percent of all due process complaints were fully adjudicated; in school year 2016-17, 11 percent were fully adjudicated. Due process complaints may not be fully adjudicated for several reasons. For example, complaints may be withdrawn by the filer, dismissed by the hearing officer, or resolved through other means, such as a resolution meeting or an agreement to try to resolve the dispute through mediation. CADRE’s data show that resolution meetings were held less than half the time due process complaints were filed in 6 of the 12 school years between 2005-06, the first year resolution meetings were used, and 2016- 17. When resolution meetings did occur, they resulted in resolution agreements less than 30 percent of the time in 10 of these 12 years. Mediation. According to CADRE, mediation is viewed as less adversarial than due process hearings, in part, because parties work together to try to reach an agreement. CADRE also reports that mediation is generally believed to be less costly than due process hearings because it typically requires less time and may require less involvement from attorneys and other experts. The number of mediation requests increased from school year 2004-05 to 2016-17 as Education and the states encouraged dispute resolution options that stakeholders told us were less costly and confrontational. In school year 2016-17, there were 11,413 mediations requested, the largest number of requests from school year 2004-05 to 2016-17. In addition, mediation requests resulted in mediation meetings at least 60 percent of the time in each of these school years. Those meetings resulted in agreements at least two-thirds of the time in every year but one (see fig. 2). Furthermore, more than half of the mediation meetings held stemmed from due process complaints that had been filed, which suggests that parties involved in the complaints may have been using mediation meetings to try to avoid a due process hearing. State complaints. State complaints were the least commonly used dispute resolution option. There were 5,239 state complaints filed in school year 2016-17, down from 6,201 in school year 2004-05 (see fig. 3). On average, from school year 2004-05 to 2016-17, approximately two-thirds of complaints filed resulted in the state issuing a report, and about two-thirds of those reports included findings of noncompliance with some aspect of IDEA on the part of the school district. According to state officials we spoke with, a state that receives a complaint will issue a report unless the filer withdraws the complaint, the state determines that the complaint is not about an issue covered under IDEA, or the complaint is resolved through other means. The rate at which all three dispute resolution options were used varied widely across states. Some states and territories had much higher rates of dispute resolution activity than others. In school year 2016-17, due process complaints were generally used at a higher rate nationwide than mediation requests and state complaints, according to CADRE data (27.2, 16.8, and 7.7 per 10,000 IDEA students served, respectively). However, the rate of due process complaints filed in states ranged from a high of 252.1 in the District of Columbia to a low of fewer than 1 per 10,000 IDEA students served in Nebraska, respectively. Similarly, some states had much higher rates of mediation requests and state complaints filed than others. Within states, the mix of dispute resolution options used also varied. In some states, due process complaints were used much more frequently than mediation requests and state complaints, while other states saw mediation requests or state complaints used most frequently. According to state officials, Parent Training and Information Center (PTI) staff, Protection and Advocacy (P&A) agency staff, and other stakeholders we interviewed, parents most commonly engage in IDEA dispute resolution because of concerns they have about the evaluations, placement, services and supports, and discipline related to the educational services their child receives. For example, a dispute related to placement may arise if a parent wants their child to spend more time in a regular education classroom as opposed to a self-contained classroom with only special education students. A parent might also object if a school district wants to place their child in an alternative school. On the other hand, some parents may seek an out-of-district placement for their child if they feel that more services will be available. A dispute over services may center on a parent asking for services for their child that the school district refuses to provide, or a parent believing that the school district is not providing services that are included in their child’s individualized education program. Research we reviewed generally supported what stakeholders told us were the main causes of disputes, although discipline issues were not reported as frequently. Other issues that led to disputes less frequently, according to those we spoke with, included, lack of progress on the part of the student, parental participation in decision making, transition services, and other accommodations for students. When we analyzed five states’ dispute resolution data we found that dispute resolution activity varied based on districts’ income levels. In general, a greater proportion of very high-income districts had dispute resolution activity, and these districts also had higher rates of dispute resolution activity than very low-income districts (see fig. 4.) This pattern was mostly consistent for all three types of dispute resolution options. Specifically, Mediation requests and due process complaints: In all five states, a greater proportion of very high-income districts tended to have mediation or due process activity than very low-income districts. Similarly, very high-income districts generally had a higher rate of such activity than very low-income districts. (See app. III for data on the individual states.) State complaints: A greater proportion of very high-income districts had state complaint activity in four of the five states. In addition, very high-income districts also had a higher rate of state complaints compared to very low-income districts in three of the five states. (See app. III for data on the individual states.) When we looked at districts’ racial and/or ethnic characteristics in our five states, we found that a smaller proportion of very high-minority districts had dispute resolution activity than very low-minority districts, but generally had higher rates of activity (see fig. 5, and app. III for data by state). We also analyzed the results of initiated disputes by districts’ income level and racial and/or ethnic characteristics—meaning the percentage of disputes that resulted in a meeting or an agreement for mediation requests, adjudication for due process complaints, and a report with findings for state complaints. As shown in tables 1-3, there was no consistent pattern in the results of dispute activity for all three types of disputes across districts with different income levels and racial/ethnic characteristics. Stakeholders we interviewed identified several types of challenges parents may face in using IDEA dispute resolution options, such as the cost of attorneys for due process hearings. While parents may hire an attorney to help with dispute resolution, stakeholders consistently told us the cost of attorneys and expert witnesses was a significant barrier to parents’ ability to use the due process complaint option in particular—especially low-income parents. Parents are not required to use an attorney at a due process hearing, but stakeholders told us that prevailing is difficult without legal representation and expert witnesses to testify on the parents’ behalf. An Education official told us that school districts may provide a list of free and low-cost attorneys to parents. According to stakeholders we interviewed, in some cases, Protection and Advocacy agencies (P&A)— which are funded by the Department of Health and Human Services (HHS)—provide legal services to parents at no cost, or refer clients to other attorneys. In general, however, very few attorneys will work on a pro-bono basis to handle IDEA dispute cases, according to stakeholders. Further, under IDEA, a court may award parents reasonable attorney’s fees and costs if they prevail in a due process hearing; however, parents cannot recoup expert witness costs regardless of the outcome. Also, if parents do not prevail at a due process hearing, they may be responsible for the school district’s legal costs in addition to their own, which can be a disincentive to going through a hearing. Education regulations allow parents to be accompanied and advised in due process hearings by individuals with special knowledge about children with disabilities, and according to IDEA regulations, whether those individuals can legally represent them is determined by state law. According to Education officials, bringing non-attorneys to support them may help reduce costs. However, the school district is likely to still have legal representation. The amount of direct legal services P&As provide varies across, and even within, states. P&A staff we interviewed in one state told us that their attorneys in one city spend most of their time assessing parents’ cases, reviewing documentation, giving advice, answering questions, and conducting training for parents, but little time participating in actual hearings. In contrast, the P&A attorneys we spoke with in another city in the same state said that 50 to 70 percent of their work is direct representation at hearings. Staff at other P&As we spoke with work primarily on cases that fall within their priority areas or cases they believe will have wide-reaching or systemic effects. The availability of attorneys can also be a challenge. According to stakeholders we interviewed, some areas, particularly rural ones, may have fewer available attorneys. However, Education officials told us that school districts in rural or sparsely populated areas may be more likely to have an incentive to resolve a dispute before it goes to a due process hearing because smaller school districts are unlikely to have in-house attorneys, and hiring an attorney is expensive. According to stakeholders, many parents feel they are at a disadvantage in a conflict with the school district due to an imbalance of power and so may be reluctant to engage in dispute resolution and take on the associated costs when they feel they are unlikely to prevail. Stakeholders also said that some parents who live in less populated and more rural areas may be reluctant to initiate dispute resolution out of concern for their privacy and because, for example, in these communities they and their children are more likely to see the teachers, principals, and district officials at the grocery store or at church, which may be awkward. Furthermore, these families may have no other educational options in the area to turn to if the dispute becomes too contentious. In some cultures, according to stakeholders, it is less common to challenge an authority figure, such as a school district official or teacher. In addition, according to stakeholders, parents may fear the school district will retaliate against their children or them if the parents initiate a dispute, such as by threatening to stop providing services. Stakeholders also told us that they are aware of cases in which the school district has called the state’s child protective services agency in what they believe was retaliation for parents bringing a dispute against the district, and that parents who are undocumented may fear that raising a dispute might result in unwanted attention from immigration officials. Further, according to stakeholders, some parents face other challenges, such as language barriers, difficulty obtaining time off from work, transportation, or internet access that could affect their use of IDEA dispute resolution and their ability to take advantage of resources, such as IDEA dispute resolution training, workshops, and online information. Education and SEAs provide technical assistance to support parents’ understanding of their rights under IDEA and to facilitate their use of dispute resolution options. According to stakeholders we interviewed, the area of special education in general and the federal law, IDEA, are complicated, and parents often do not understand the IDEA dispute resolution process. Education supports several efforts to help parents understand and use dispute resolution options afforded to them under IDEA. Procedural safeguards notice. To receive IDEA funds, states must ensure school districts notify parents of their rights under IDEA, including the right to initiate dispute resolution about the educational services provided to their child. School districts must provide a notice, referred to as a procedural safeguards notice, to parents that explains their rights under IDEA. According to Education officials, to help states meet their IDEA requirements, the agency developed a model notice, which states can, but are not required to, have school districts use to notify parents of their rights under IDEA. States may also develop their own procedural safeguards notice as long as it includes all the information required under IDEA. Technical assistance. Education established and funds different types of technical assistance centers that provide information, training, workshops, and advocate services, and collect and disseminate data on dispute resolution, among other activities. Specifically, Education officials reported that Education provided about $21 million to the network of Parent Training and Information Centers (PTI), about $2.9 million to the network of Community Parent Resource Centers, and $750,000 to CADRE in fiscal year 2019. In addition, Education’s technical assistance centers collaborate with P&As in some cases. Further, P&A staff we interviewed in some of our selected states told us they conduct trainings for advocates to attend meetings with parents, other attorneys working on special education issues, community organizations and agencies, and parents. Education officials told us that, in the past, the agency has facilitated meetings between PTIs and P&As, to improve collaboration between these organizations. According to Education officials, these meetings resulted in informal agreements between PTIs and P&As. In addition, Education’s Center for Parent Information and Resources, the national technical assistance center to the PTIs, provides resources on its website to help parents learn about their rights and the procedural safeguards notice they receive from schools. For example, the center’s website contains an explanation of the procedural safeguards notice and online training on procedural safeguards, among other issues. The website also provides contact information for the PTI(s) in each state. Further, CADRE, part of Education’s technical assistance and dissemination network, has developed concise, easy-to-read materials that it distributes to parent centers and others to help them understand the procedural safeguards and how to resolve disputes with school districts. Stakeholders we interviewed told us that parents often do not understand IDEA dispute resolution procedures, but that PTI staff are available to explain them, discuss the procedural safeguards notice, and offer other assistance at no cost to the parents. According to stakeholders, the IDEA procedural safeguards notice is usually a lengthy document that uses complex, legal language and that parents say the notice is hard to understand. Education officials told us their model notice is complex in part because it must reflect all the applicable provisions of the IDEA statute and regulations. To help parents understand the notice and their dispute resolution options, the PTIs in our selected states offer a variety of assistance, such as staffing telephone helplines, meeting with parents in person, offering workshops and training for parents, and developing or making available easy-to-read documents and other resources. PTI staff can also attend mediation meetings with parents and help parents write state complaints, including parents for whom English is not their first language. In addition, PTI staff told us they try to help specific populations, including parents who are not native English speakers, understand and navigate the dispute process. In some cases, PTI staff will attend mediation meetings with or provide interpreters for non-English speaking parents. PTI staff are also available to help parents who have lower levels of formal education or who have disabilities, which stakeholders identified as other factors that could affect parents’ use of dispute resolution options. Our five selected states provide technical assistance and training to help parents understand and use dispute resolution options, including how to file a state complaint. State officials in some of our selected states said they make available plain language documents that can supplement the legally required procedural safeguards notice. For example, all of the states created a parents’ rights handbook and several have one- or two- page documents describing the IDEA dispute resolution processes that they make available on the state’s public website (see fig. 6 for an example of such a document). In addition, the states we contacted post information about IDEA on their websites in multiple languages. For example, one state’s parents’ rights handbook is available in English and 11 other languages. Regarding the cost of due process hearings discussed earlier, one state we contacted provides information about free and low-cost services along with the state’s parents’ rights booklet, and several states include contact information for the PTIs and sometimes P&As in their booklet. State officials we interviewed also said their states offer telephone helplines that parents can call with questions about their dispute resolution options and the processes involved. Some state officials told us they have staff available by phone to explain the dispute options to parents, including to parents who do not speak English or have lower levels of formal education. One state has a phone line that connects parents to an early resolution specialist who will try to help parents resolve the dispute before a formal complaint becomes necessary. Officials in one state told us that the state has installed voice interpretation technology for its helpline so that parents who need assistance with hearing or speaking can communicate with staff. Some states also employ staff who can serve as interpreters to better assist non-English speaking parents. Officials in some states told us that staff answering the helpline are available to answer questions about dispute resolution documents for parents who have difficulty reading. In addition, some of the states we contacted said they made requesting mediation and/or filing state complaints easier by posting the required initiation forms on their websites. According to staff from one state, after the state posted its state complaint form online, the number of complaints doubled in 5 years. Further, some of our selected states provide training and technical assistance to school districts, parent advocate groups, and parents related to accessing IDEA dispute options. One of our selected states uses 16 regional support teams to provide training and technical assistance to school districts. Another state conducts parent training jointly with the Education-funded PTI in the state. We have previously reported on other efforts some states have taken to help parents understand their dispute rights and reduce the need for parents to initiate formal disputes. For example, some states have offered conflict resolution skills training to school district staff and parents, and support facilitated IEP meetings, among other initiatives. We provided a draft of this product to the Department of Education for review and comment. We received written comments from Education, which are reproduced in appendix I. Education also provided technical comments that we have incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines the use of dispute resolution options available under the Individuals with Disabilities Education Act (IDEA). In particular, this report examines (1) how often IDEA dispute resolution options are used, and whether use in selected states varies across school district-level socioeconomic or demographic characteristics; and (2) what challenges parents face in using IDEA dispute resolution options and how Education and selected states help facilitate parents’ use of these options. To address our first objective, we obtained publicly available dispute resolution data at the national and state levels and collected and analyzed data on the number and types of dispute resolution options used from selected states at the school district level. To address how often dispute resolution options are used, we reviewed and analyzed publicly available data from the Center for Appropriate Dispute Resolution in Special Education (CADRE) from school years 2004-05 to 2016-17, the most recent data available when we conducted our analysis. We assessed the reliability of these data by interviewing knowledgeable CADRE staff and comparing CADRE data to other publicly available data. In addition, we interviewed staff at Parent Training and Information Centers (PTI) funded by the Department of Education (Education) and Protection and Advocacy (P&A) agencies funded by the Department of Health and Human Services, as well as state educational agency (SEA) officials in our five selected states to determine the reasons parents use dispute resolution. We also interviewed various national organizations that advocate for parents and local educational agencies (LEA) and SEAs. To determine whether the use of dispute resolution options varied by socioeconomic or racial and/or ethnic characteristics, we analyzed dispute resolution data we collected at the LEA level from five states for school year 2017-18, the most recent data available at the time of our analysis. We selected these states—Massachusetts, Michigan, New Jersey, Ohio, and Pennsylvania—based on a combination of criteria including the amount of dispute activity within the state (that is, the number of mediations, due process complaints, and state complaints); the large number of LEAs in the state with highly homogenous student populations to allow us to compare across LEAs with different student populations; the large number of IDEA-eligible students in the state; and the states’ ability to provide reliable LEA level data on disputes. We used Education’s Common Core of Data (CCD) to categorize each LEA in our selected states based on (1) income level, as measured by the percentage of students eligible for free or reduced-price school lunch; (2) racial and/or ethnic makeup, as measured by the percentage of Black and/or Hispanic students; and (3) population density, as categorized by CCD. We used Education’s school year 2016-17 CCD data, which was the most recent data available at the time of our analysis. In some cases, states had not reported 2016-17 free or reduced-price school lunch data to CCD so we used CCD data from a previous year. We assessed the reliability of the CCD data by (1) reviewing existing information about the data and the system that produced them and (2) reviewing data reliability assessments of the data from other recent GAO reports. We assessed the reliability of dispute resolution data provided by the states by (1) performing electronic testing of required data elements, (2) conducting interviews with knowledgeable agency officials and reviewing written responses to data reliability questions, and (3) reviewing existing information about the data and systems that produced them, where available. We determined that the CCD and data collected from the states were sufficiently reliable for the purposes of this report. We matched the LEA-level dispute data provided by our states to the LEA-level socioeconomic, race/ethnicity, and population density data from CCD to determine whether the frequency of use of dispute resolution options or the types of options used varied across LEAs with different characteristics. Because our analyses are at the LEA level, and not the individual student or family level, it is impossible to know with certainty whether the families using the dispute resolution options in our school districts match the categorization of the districts themselves. To address this concern to the greatest extent possible, we report on LEAs that are highly homogenous. These districts are those in which: 90 percent or more of the students were eligible for free or reduced- price school lunch (very low-income districts) compared to districts in which 10 percent or fewer of the students were eligible (very high- income districts), and 90 percent or more of the students were Black and/or Hispanic (very high-minority districts) compared to districts in which 10 percent or fewer of the students were Black and/or Hispanic (very low-minority districts). We conducted two separate analyses on the combined data. We analyzed and compared: 1. the percentage of all the “very low” districts in our data that had dispute resolution activity to the percentage of all the “very high” districts in our data with dispute resolution activity, as measured by whether the district had one or more mediation requests, due process complaints, or state complaints. We also conducted this analysis to compare the percentages of urban, suburban, and rural districts with dispute resolution activity. 2. the rate of dispute resolution activity in our “very low” districts and our “very high” districts, as measured by the number of mediation requests, due process complaints, and state complaints per 10,000 students served under IDEA. We also conducted this analysis for urban, suburban, and rural districts. This first analysis compared the percentages of school districts with different income and racial and/or ethnic characteristics that had at least one mediation request, due process complaint, or state complaint. In essence, it shows the differences in whether there is any dispute resolution activity in districts with different income and racial and/or ethnic characteristics, in our selected states. Because our analysis counts districts in which a single dispute resolution was initiated in the same manner as those with more activity, it is not potentially skewed by individual districts that may have unusually high or low levels of dispute resolution activity. To supplement this analysis, our second analysis compares the rate of dispute activity in these types of districts, which shows the magnitude of the various types of dispute resolution activity. Although we use this 90-10 threshold in the body of the report, we also conducted these analyses for districts where 75 percent or more of students were eligible for free or reduced-price lunch and 25 percent or fewer were not eligible. Similarly, we conducted our race/ethnicity analyses at this same level as well. These additional analyses can be found in appendix III. The results from our five states are not generalizable to all states. To address both research objectives, we reviewed relevant federal laws and regulations. We also reviewed Education documents, including its model Notice of Procedural Safeguards, PTI and CADRE documents, and relevant literature related to challenges parents face using dispute resolution. In addition, we interviewed Education officials about challenges families face in using dispute resolution options and Education’s efforts to assist families. We also interviewed PTI, P&A, and advocacy organization staff, and SEA officials from the five states from which we collected data. We conducted this performance audit from June 2018 to November 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix contains tables that show data based on analyses we conducted using dispute resolution data collected from five states– Massachusetts, Michigan, New Jersey, Ohio, and Pennsylvania–for school year 2017-18, and the Department of Education’s Common Core of Data for school year 2016-17. In some cases, states did not report free or reduced-price school lunch data for school year 2016-17. In those cases, we used the most recent year for which the state reported those data. The total number of local educational agencies and the total number of students served in our income analysis and our race/ethnicity analysis are slightly different. Jaqueline M. Nowicki, Director, (617) 788-0580 or nowickij@gao.gov. In addition to the contact named above, Bill MacBlane (Assistant Director), David Barish (Analyst-in-Charge), and Linda Siegel made key contributions to this report. In addition, key support was provided by James Bennett, Deborah Bland, Holly Dye, Sheila R. McCoy, Jean McSween, John Mingus, Amy Moran Lowe, Moon Parks, James Rebbe, Kelly Snow, Joy Solmonson, and Greg Whitney.", "summary": "Almost 7 million children aged 3 to 21 received special education services under Part B of the Individuals with Disabilities Education Act (IDEA) in school year 2016-17. IDEA contains options parents and school districts may use to address disputes that arise related to the education of a student with a disability. These options include mediation and due process complaints, which can be used by parents and school districts; and state complaints, which can be used by any organization or individual, including the child's parent, alleging an IDEA violation. GAO was asked to review parents' use of IDEA dispute resolution options. This report examines (1) how often IDEA dispute resolution options are used, and whether use in selected states varies across school district-level socioeconomic or demographic characteristics; and (2) what challenges parents face in using IDEA dispute resolution options and how Education and selected states help facilitate parents' use of these options. GAO reviewed publicly available data on dispute resolution at the state level and collected data at the school district level from five states—Massachusetts, Michigan, New Jersey, Ohio, and Pennsylvania—selected based on the number of disputes initiated and school district characteristics, among other factors. GAO also reviewed relevant federal laws, regulations, and Education and state documents; and interviewed Education officials, state officials, staff from organizations providing technical assistance in these five states, and other national advocacy organizations. In school year 2016-17, 35,142 special education disputes were filed nationwide, and in five selected states GAO reviewed, dispute resolution options varied across school districts with different socioeconomic and demographic characteristics. The Individuals with Disabilities Education Act (IDEA) provides parents several ways to file and resolve disputes about plans and services that school districts provide to students with disabilities. A greater proportion of very high-income school districts had dispute resolution activity as well as higher rates of dispute activity than very low-income districts in most of the five states GAO reviewed. GAO also found that in most of these states, a smaller proportion of predominately Black and/or Hispanic districts had dispute resolution activity compared to districts with fewer minority students; however, predominately Black and/or Hispanic districts generally had higher rates of such activity. Technical assistance providers and others told GAO that parents used dispute resolution most often for issues related to school decisions about evaluations, placement, services and supports, and discipline of their children. Note: “Very high-income” districts are those in which 10 percent or fewer of students are eligible for free or reduced-price school lunch (FRPL). In “Very low-income” districts, 90 percent or more of students are eligible for FRPL. Parents may face a variety of challenges in using IDEA dispute resolution, and the Department of Education and states provide several kinds of support that, in part, may address some of these challenges. Stakeholders cited challenges such as paying for attorneys and expert witnesses at a due process hearing, parents' reluctance to initiate disputes because they feel disadvantaged by the school district's knowledge and financial resources, and parents' lack of time off from work to attend due process hearings. Education and state agencies provide technical assistance to support parents' understanding of their rights under IDEA and to facilitate their use of dispute resolution options, for example, by providing informational documents and phone help lines to parents.", "document_type": "gao"}
{"report": "Since 1990, generally every 2 years at the start of a new Congress, we call attention to agencies and program areas that are high risk due to their vulnerability to mismanagement or that are most in need of transformation. Our high-risk program is intended to help inform the congressional oversight agenda and to improve government performance. Since 1990, a total of 62 different areas have appeared on the High-Risk List. Of these, 26 areas have been removed, and 2 areas have been consolidated. On average, the high-risk areas that were removed from the list had been on it for 9 years after they were initially added. Our experience with the High-Risk List over the past 29 years has shown that the key elements needed to make progress in high-risk areas are top-level attention by the administration and agency leaders grounded in the five criteria for removing high-risk designations, which we reported on in November 2000. When legislative and agency actions, including those in response to our recommendations, result in our finding significant progress toward resolving a high-risk problem, we will remove the high- risk designation. However, implementing our recommendations alone will not result in the removal of the designation, because the condition that led to the recommendations is symptomatic of systemic management weaknesses. In cases in which we remove the high-risk designation, we continue to closely monitor the areas. If significant problems again arise, we will consider reapplying the high-risk designation. The five criteria for removing high-risk designations are as follows: Leadership commitment. Demonstrated strong commitment and top leadership support to address the risks. Capacity. Agency has the capacity (i.e., people and other resources) to resolve the risk(s). Action plan. A corrective action plan that defines the root causes, identifies solutions, and provides for substantially completing corrective measures in the near term, including steps necessary to implement solutions we recommended. Monitoring. A program has been instituted to monitor and independently validate the effectiveness and sustainability of corrective measures. Demonstrated progress. Ability to demonstrate progress in implementing corrective measures and in resolving the high-risk area. These five criteria form a road map for efforts to improve and ultimately address high-risk issues. Addressing some of the criteria leads to progress, and satisfying all of the criteria is central to removal from the list. Figure 1 shows the five criteria for removal for a designated high-risk area and examples of agency actions leading to progress toward removal. Importantly, the actions listed are not “stand alone” efforts taken in isolation of other actions to address high-risk issues. That is, actions taken under one criterion may be important to meeting other criteria as well. For example, top leadership can demonstrate its commitment by establishing a corrective action plan, including long-term priorities and goals to address the high-risk issue and by using data to gauge progress—actions that are also vital to addressing the action plan and monitoring criteria. When an agency meets all five of these criteria, we can remove the agency from the High-Risk List. We rate agency progress on the criteria using the following definitions: Met. Actions have been taken that meet the criterion. There are no significant actions that need to be taken to further address this criterion. Partially met. Some, but not all, actions necessary to meet the criterion have been taken. Not met. Few, if any, actions toward meeting the criterion have been taken. Officials from Indian Affairs, BIE, BIA, and IHS expressed their commitment to addressing the issues that led to the high-risk designation for federal management of programs that serve tribes and their members. Since we last testified before this committee on June 13, 2018, we met with agency leaders and worked with each agency to identify actions the agencies took or plan to take to address the concerns that contributed to the designation. We determined that Indian Affairs, BIE, BIA, and IHS demonstrated some progress to partially meet each of the criteria for removing a high-risk designation. However, additional progress is needed for the agencies to fully address the criteria and related management weaknesses. As we reported in the March 2019 high-risk report, when we applied the five criteria for High-Risk List removal to each of the three segments— education, energy, and health care—we determined that Indian Affairs, BIE, BIA, and IHS have each demonstrated some progress. Overall, the agencies have partially met the leadership commitment, capacity, action plan, monitoring, and demonstrated progress criteria for the education, health care, and energy areas. However, the agencies continue to face challenges, particularly in retaining permanent leadership and a sufficient workforce. The following is a summary of the progress that Indian Affairs, BIE, BIA, and IHS have made in addressing the five criteria for removal from the High-Risk List. To meet the leadership commitment criterion for removal of a high-risk designation, an agency needs to have demonstrated strong commitment and top leadership support to address management weaknesses. The following examples show actions Indian Affairs, BIE, BIA, and IHS took to partially meet the leadership commitment criterion. Education. Indian Affairs’ leaders have demonstrated commitment to addressing key weaknesses in the management of BIE schools in several ways. For example, the BIE Director formed an internal working group, convened meetings with other senior leaders within Indian Affairs, and publicly stated that his agency is committed to ensuring implementation of our recommendations on Indian education. In addition, the BIE Director and other Indian Affairs leaders and senior managers have met with us frequently to discuss outstanding recommendations, actions they have taken to address these recommendations, and additional actions they could take. We also met with the new Assistant Secretary-Indian Affairs, who expressed her commitment to supporting the agency’s efforts to address weaknesses in the management of BIE schools. However, it is important that Indian Affairs leaders be able to sustain this level of commitment to solving problems in Indian education. Since 2012, there have been seven Assistant-Secretaries of Indian Affairs and five BIE Directors. There has also been leadership turnover in other key offices responsible for implementing our recommendations on Indian education. We have previously reported that leadership turnover hampered Indian Affairs’ efforts to make improvements to Indian education. We believe that ensuring stable leadership and a sustained focus on needed changes is vital to the successful management of BIE schools. Energy. BIA officials demonstrated leadership commitment by, among other things, meeting with us to discuss the agency’s progress in addressing our recommendations. In June 2018, a permanent Assistant Secretary for Indian Affairs was confirmed. This action provided an opportunity to improve Indian Affair’s oversight of federal actions associated with energy development. According to the BIA Acting Director and the Acting Director for Trust Services, BIA held a number of meetings with the Assistant Secretary to discuss agency action plans for our recommendations. However, BIA does not have a permanent Director, and BIA’s Office of Trust Service—which has significant responsibility over Indian energy activities—does not have a permanent Director or Deputy Director. We have seen turnover in these leadership positions as officials have been brought in to temporarily fill these roles. As officials are brought in temporarily, previously identified plans and time frames for completing some activities have changed, and BIA has found itself starting over on the process to identify or implement corrective actions. Health Care. IHS officials demonstrated leadership commitment by regularly meeting with us to discuss the agency’s progress in addressing our recommendations. In addition, IHS has chartered a policy advisory council that will focus on issues related to strategic direction, recommended policy, and organizational adjustments. According to IHS, this advisory council will, among other things, serve as a liaison among IHS leadership for issues involving strategic direction and policy, as well as monitor and facilitate related policy workgroups. However, IHS still does not have permanent leadership—including a Director of IHS—which is necessary for the agency to demonstrate its commitment to improvement. Additionally, since 2012, there have been five IHS Acting Directors, and there has been leadership turnover in other key positions, such as area directors. To fully meet the leadership commitment criterion, all agencies will need, among other things, stable, permanent leadership that has assigned the tasks needed to address weaknesses and that holds those assigned accountable for progress. For a timeline of senior leadership turnover in Indian Affairs, BIE, BIA, and IHS from 2012 through March 2019, see Figure 3. To meet the capacity criterion, an agency needs to demonstrate that it has the capacity (i.e., people and other resources) to resolve its management weaknesses. Indian Affairs, BIE, BIA, and IHS each made some progress in identifying capacity and resources to implement some of our recommendations, but BIE and IHS continue to face significant workforce challenges. The following examples show actions Indian Affairs, BIE, BIA, and IHS took to partially meet the capacity criterion. Education. BIE and other Indian Affairs offices that support BIE schools have made some progress in demonstrating capacity to address risks to Indian education. For example, BIE hired a full-time program analyst to coordinate its working group and help oversee the implementation of our recommendations on Indian education. This official has played a key role in coordinating the agency’s implementation efforts and has provided us with regular updates on the status of these efforts. BIE has also conducted hiring in various offices in recent years as part of a 2014 Secretarial Order to reorganize the bureau. For example, it has hired school safety officers and personnel in offices supporting the oversight of school spending. However, about 50 percent of all BIE positions have not been filled, including new positions that have been added as a result of the agency’s restructuring, according to recent BIE documentation. Moreover, the agency reported that it has not filled the position of Chief Academic Officer, a top-level BIE manager responsible for providing leadership and direction to BIE’s academic programs. Furthermore, BIE has not completed a strategic workforce plan to address staffing and training gaps with key staff, which we previously recommended. Such a plan is important to allow BIE and other Indian Affairs offices to better understand workforce needs and leverage resources to meet them. In February 2019, BIE drafted a strategic workforce plan and reported it is currently gathering feedback on the plan from internal stakeholders. BIE officials indicated they are planning to finalize and implement the plan in 2019. Energy. In November 2016, we recommended that BIA establish a documented process for assessing the workforce at its agency offices. BIA has taken a number of actions, such as conducting an internal survey to identify general workforce needs related to oil and gas development. This survey information supported staffing decisions for the recently created Indian Energy Service Center. In February 2019, BIA officials told us they have drafted a long-range workforce plan to ensure BIA has staff in place to meet its organizational needs. We will review the plan to determine if the planned actions will help BIA identify critical skills and competencies related to energy development and identify potential gaps. Health Care. IHS has made some progress in demonstrating it has the capacity and resources necessary to address the program risks we identified in our reports. For example, among other actions, IHS officials stated that the agency is expanding the role of internal audit staff within its enterprise risk management program to augment internal audits and complement audits by the HHS Inspector General and GAO. In addition, IHS has developed a new Office of Quality, which is expected to develop and monitor agency-wide quality of care standards. However, IHS officials told us there are still vacancies in several key positions, including the Director of the Office of Resource Access and Partnerships, and the Office of Finance and Accounting. Additionally, our August 2018 report found that IHS’s overall vacancy rate for clinical care providers was 25 percent. To fully meet the capacity criterion, all of the agencies need to assess tradeoffs between these and other administration priorities in terms of people and resources, and the agencies should provide to decision makers key information on resources needed to address management weaknesses. To meet the action plan criterion, an agency needs to have a corrective action plan that defines the root causes, identifies solutions, and provides for substantially completing corrective measures in the near term, including steps necessary to implement the solutions we recommended. The following examples show actions Indian Affairs, BIE, BIA, and IHS took to partially meet the action plan criterion. Education. Among other actions, BIE implemented a new action plan for overseeing BIE school spending, including written procedures and risk criteria, which fully addressed two priority recommendations. Also, BIE completed a strategic plan in August 2018, which we recommended in September 2013. The plan provides the agency with goals and strategies for improving its management and oversight of Indian education, and establishes detailed actions and milestones for the implementation. However, Indian Affairs has not provided documentation that it has completed action plans on other important issues, such as a comprehensive, long-term capital asset plan to inform its allocation of school facility funds, which we recommended in May 2017. Energy. In meetings, BIA officials identified actions they have taken towards implementing our recommendations. For instance, BIA officials told us they have recently completed modifications to BIA’s database for recording and maintaining historical and current data on ownership and leasing of Indian land and mineral resources—the Trust Asset and Accounting Management System (TAAMS). The officials said that the modifications incorporate the key identifiers and data fields needed to track and monitor review and response times for oil and gas leases and agreements. BIA officials we met with have demonstrated an understanding that addressing long-standing management weaknesses is not accomplished through a single action but through comprehensive planning and continued movement toward a goal. However, the agency does not have a comprehensive action plan to identify the root causes of all identified management weaknesses and address the problems. Health Care. In February 2019, IHS finalized its strategic plan for fiscal years 2019 through 2023, and is developing a related work plan to address certain root causes of management challenges and define solutions and corrective measures for the agency. The strategic plan divides these challenges into three categories: (1) access to care, (2) quality of care, and (3) program management and operations. We will examine the strategic plan and IHS’s work plan, once issued, to determine whether they contain the needed elements of an action plan. To fully meet the action plan criterion, a comprehensive plan that identifies actions to address the root causes of its management shortcomings would have to come from top leadership with a commitment to provide sufficient capacity and resources to take the necessary actions to address management shortcomings and risks. To meet the monitoring criterion, an agency needs to demonstrate that a program has been instituted to monitor and independently validate the effectiveness and sustainability of corrective measures. We have been working with the agencies to help clarify the need to establish a framework for monitoring progress that includes goals and performance measures to track their efforts and ultimately verify the effectiveness of their efforts. The following examples show actions Indian Affairs, BIE, BIA, and IHS took to partially meet the monitoring criterion. Education. Indian Affairs, in consultation with Department of Interior’s Office of Occupational Safety and Health, has taken actions to monitor corrective measures that address weaknesses with the agency’s safety program—which covers safety at BIE schools. However, the agency has not yet demonstrated that it is monitoring several other areas, such as whether relevant employees are being held to the agency’s required performance standards for safety inspections. Energy. BIA has taken steps to improve monitoring by holding frequent meetings to assess its progress in implementing our recommendations. However, BIA has not yet taken needed steps to monitor its progress in addressing the root causes of management weaknesses. Health Care. IHS has taken some steps toward monitoring the agency’s progress in addressing the root causes of their management weaknesses. In addition to developing its new Office of Quality, IHS has taken steps to develop a patient experience of care survey, as well as standards for tracking patient wait times. These efforts should be reflected in the agency’s corrective plan, as part of an overall framework for monitoring progress that includes goals and performance measures to track their efforts and ultimately verify the effectiveness of their efforts. To fully meet the monitoring criterion, the agencies need to establish goals and performance measures as they develop action plans and take further actions to monitor the effectiveness of actions to address root causes of identified management shortcomings. To meet the demonstrated progress criterion, an agency needs to demonstrate progress in implementing corrective measures and in resolving the high-risk area. The following examples show actions Indian Affairs, BIA, and IHS took to partially meet the demonstrated progress criterion. Education. As of February 2019, Indian Affairs had addressed 11 of the 23 outstanding education recommendations we identified in our September 2017 testimony. Three of these recommendations were closed after the June 2018 hearing, including a recommendation from our 2013 report for BIE to develop a strategic plan and two recommendations from our 2017 report on improving the oversight and accountability for BIE school safety inspections. Overall, Indian Affairs’ efforts since we issued our High-Risk List update in February 2017 represent a significant increase in activity implementing our recommendations. Substantial work, however, remains to address our outstanding recommendations in several key areas, such as in accountability for BIE school safety and school construction projects. For example, Indian Affairs has not provided documentation that the inspection information its personnel collect on the safety of BIE schools is complete and accurate. As of late February 2019, 12 recommendations related to this high-risk area remain open and Indian Affairs concurred with all 12 recommendations. For a full description of the status of these open recommendations, see in table 1 in appendix I. Energy. BIA has shown significant progress developing data collection instruments and processes needed to track and review response times for a number of different actions associated with energy development. For example, in our June 2015 report, we recommended that BIA take steps to improve its geographic information system (GIS) capabilities to ensure it can verify ownership in a timely manner. We closed this recommendation as BIA has made significant progress in enhancing its GIS capabilities by integrating map-viewing technology and capabilities into its land management data system. In addition, we recommended that BIA take steps to identify cadastral survey needs. BIA’s enhanced map- viewing technology allows the bureau to identify land boundary discrepancies, which can then be researched and corrected. To address the recommendation, BIA identified unmet survey needs that were contained within the defunct cadastral request system. BIA developed a new mechanism for its regions and agency offices to make survey requests and a new database to maintain survey requests. In fall 2018, BIA completed enhancements to TAAMS that will allow the agency to track time frames and status of oil and gas revenue-sharing agreements-called communitization agreements (CA) through the review process. BIA held training on the enhancements in November 2018 and requested staff input information on any newly submitted CAs in the system. In a meeting on February 25, 2019, the Acting Director of BIA said that BIA had also completed efforts to modify TAAMS, incorporating the key identifiers and data fields needed to track and monitor review and response times for oil and gas leases and agreements. We believe these actions show significant progress in addressing management weaknesses associated with outdated technology and data limitations for tracking and monitoring the review and approval of energy related documents. However, BIA needs to collect data from its updated system, develop time frames, and monitor agency performance to close open recommendations. For a full description of the status of the agency’s open recommendations, see in table 2 in appendix II. Health Care. IHS has made progress in implementing corrective actions related to the management of health care programs. Specifically, since our 2017 High-Risk Report, IHS implemented four of our 13 open recommendations. For example, in response to our April 2013 recommendation, to ensure that IHS’s payment rates for contracted services do not impede patient access to physician and other nonhospital care, IHS developed an online tool that enables the agency to track providers that do not accept IHS’s payment rates. As of March 2019, six out of the 13 recommendations in our 2017 High- Risk Report remain open, and we have added one additional recommendation—for a total of seven open recommendations related to this high-risk area. IHS officials told us that they plan to complete the implementation of additional recommendations in 2019. For a full description of the status of the agency’s open recommendations, see in table 3 in appendix III. To fully meet the demonstrating progress criterion, agencies need to continue taking actions to ensure sustained progress and show that management shortcomings are being effectively managed and root causes are being addressed. In conclusion, we see some progress in meeting all of the criteria, at all agencies, especially related to education programs. However, permanent leadership that provides continuing oversight and accountability is needed. We also see varying levels of progress at all of the agencies in understanding what they need to do to be removed from the High-Risk List, and identifying steps that can be incorporated into corrective action plans. We look forward to working with the agencies to track their progress in implementing a framework for monitoring and validating the effectiveness of planned corrective actions. Among the greatest continuing challenges for the agencies is developing sufficient capacity, including demonstrating that they have the people and other resources required to address the deficiencies in their programs and activities. This challenge cannot be overcome by the agencies without a commitment from their leadership and the administration to prioritize fixing management weaknesses in programs and activities that serve tribes and their members. Sustained congressional attention to these issues will help ensure that the agencies continue to achieve progress in these areas. Chairman Hoeven, Vice Chairman Udall, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staff have any questions about health care issues in this testimony or the related reports, please contact Jessica Farb at (202) 512-7114 or farbj@gao.gov. For questions about education, please contact Melissa Emrey-Arras at (617) 788-0534 or emreyarrasm@gao.gov. For questions about energy resource development, please contact Frank Rusco at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement include Kelly DeMots, (Assistant Director), Christina Ritchie (Analyst-in-Charge), Edward Bodine, Christine Kehr, Elizabeth Sirois, and Leigh White. As of late February 2019, 12 of the 23 recommendations to the Department of the Interior on Indian education we identified in our September 13, 2017, testimony remain open. As of February 2019, 12 of the 14 recommendations to the Department of Interior’s Bureau of Indian Affairs cited in our 2017 High-Risk Report remain open. As of March 2019, six out of the 13 recommendations in our 2017 High- Risk Report remain open, and we have added one additional recommendation—for a total of seven open recommendations related to this high-risk area.", "summary": "GAO's High-Risk List identifies federal program areas that are high risk due to their vulnerability to mismanagement, among other things. GAO added the federal management of programs that serve Indian tribes and their members to its February 2017 biennial update of high-risk areas in response to management weaknesses at Interior and HHS. GAO's recommendations identified in this high-risk area are neither reflective of the performance of programs administered by tribes nor directed at any tribally operated programs and activities. This testimony, which is based on GAO's March 2019 High Risk report, provides examples of actions taken and progress made by these agencies to address the five criteria GAO uses for determining whether to remove a high-risk designation. For this statement, GAO also drew on findings from its reports issued from September 2011 through August 2018 and updated that work by reviewing agency documentation and interviewing agency officials. GAO designated the federal management of programs that serve tribes and their members as high risk in 2017. Officials from the Department of the Interior's Office of the Assistant Secretary-Indian Affairs (Indian Affairs), the Bureau of Indian Education (BIE), the Bureau of Indian Affairs (BIA), and the Department of Health and Human Services' (HHS) Indian Health Service (IHS) have expressed their commitment to addressing the issues that led to the designation. Since GAO last testified before this committee on June 13, 2018, Indian Affairs, BIE, BIA, and IHS have demonstrated progress to partially meet each of the five criteria for removing a high-risk designation (leadership commitment, capacity, action plan, monitoring, and demonstrated progress). However, additional progress is needed to fully address management weaknesses—particularly in the areas of retaining permanent leadership and a sufficient workforce. For example, to meet the capacity criterion, an agency needs to demonstrate that it has the capacity (i.e., people and other resources) to resolve its management weaknesses. While Indian Affairs, BIE, BIA, and IHS each made progress identifying capacity and resources to partially meet this criterion, BIE and IHS continue to face significant workforce challenges. Specifically, although BIE has conducted hiring in recent years as part of an effort to reorganize the bureau, about 50 percent of all BIE positions have not been filled according to recent BIE documentation. IHS also faces workforce challenges—GAO's August 2018 report found that IHS's overall vacancy rate for clinical care providers was 25 percent. GAO has identified varying levels of progress at the agencies in understanding what they need to do to be removed from the list and will continue to closely monitor their progress. GAO has made more than 50 recommendations related to this high-risk area to improve management weaknesses at some Interior and HHS agencies—specifically BIE, BIA, and IHS—of which 31 recommendations are still open. Sustained focus by Interior and HHS in fully implementing these recommendations and continued oversight by Congress are essential to achieving progress in these areas.", "document_type": "gao"}
{"report": "In developing our June 2018 report to assist the Congress, OMB, and agencies in assessing agency reform plans, we reviewed our prior work and leading practices on organizational transformations; collaboration; government streamlining and efficiency; fragmentation, overlap, and duplication; and high-risk and other long-standing agency management challenges. The resulting June 2018 report includes 58 key questions to aid in assessing reform efforts. These questions are organized into four broad categories and 12 subcategories. We determined that the questions most relevant to the current implementation stage of State’s reform efforts are found in two subcategories: (1) Leadership Focus and Attention and (2) Managing and Monitoring. Table 1 lists the key questions in these subcategories. In response to the March 2017 Executive Order 13781 and the ensuing OMB memo, State launched a “listening tour” intended to gather ideas and feedback from State and USAID employees. As a key component of this outreach effort, State hired a contractor to design and administer a confidential online survey, which was sent to all State and USAID employees in May 2017. According to the contractor’s report, the survey had a 43 percent response rate, with 27,837 State employees and 6,142 USAID employees responding to the survey. The contractor also conducted in-person interviews with a randomly selected cross section of personnel, which included 175 employees from State and 94 from USAID. The contractor’s report on the results of the survey and the interviews highlighted five areas for State reforms. In July 2017, the Deputy Secretary of State created five planning teams to develop multiple projects in those five areas. The Deputy Secretary also established an Executive Steering Committee composed of senior State and USAID officials to guide the five planning teams and provide direction during the reform process. Led jointly by State and USAID, each planning team comprised participants from a cross section of overseas and domestic workforces. The planning teams were tasked with gathering information and conducting analysis as described below: Foreign Assistance Programs: Analyze current foreign assistance programs at State and USAID to develop a future vision, ensuring alignment with national priorities. Overseas Alignment and Approach: Assess key diplomatic activities and identify required platforms, including the balance of work between headquarters and the field. Human Capital Planning: Identify ways to promote an agile and empowered workforce as part of an overarching talent map. Management Support: Identify opportunities to streamline administrative support functions at the bureau and agency levels to ensure front line effectiveness. Information Technology (IT) Platform Planning: Focus on improving the employee experience through increased use of cutting- edge technology and streamlining duplicative systems and processes. Figure 1 shows a timeline of key events in State’s initial reform efforts. The planning teams developed specific reform projects, listed below in table 2, which State described in the fiscal year 2019 budget justification it submitted to Congress in February 2018. According to implementing officials, all these projects predated the Executive Order and OMB memo issued in the spring of 2017. They also noted, however, that the administration’s reform-related directives helped advance State’s preexisting efforts by focusing management attention and agency resources on these projects. As of April 2019, according to State officials and status reports, State had completed one of its 17 reform projects; 13 projects were continuing; two projects were stalled pending future decisions or actions; and one project was discontinued. Table 3 provides additional details on each project and a summary of the results of our analysis. As State shifted into the implementation phase of its reform efforts in early 2018, multiple transitions within the agency contributed to a loss of leadership focus on the efforts, resulting in uncertainty about leadership’s support for some reform projects. In February 2018, State reported to Congress in its fiscal year 2019 budget justification that it was pursuing the reform projects we described above. In March 2018, the first transition affecting the implementation of those projects occurred when the President removed the then Secretary of State and nominated the then CIA director to replace him; in April 2018, the Senate confirmed the current Secretary. According to senior State officials, when the new Secretary took office, his top priority was ending the hiring freeze and restarting a concerted recruitment effort because vacancies in key positions and a general staffing shortfall would otherwise have led to what one senior official described as a “cataclysmic failure” at State. These senior officials noted that the new Secretary decided some of the existing reform projects were not well designed and that he wanted greater emphasis on cybersecurity and data analytics. They said he also wanted to pursue other initiatives, including a new proposal to create a Global Public Affairs Bureau by merging two existing bureaus. The senior officials told us that the Secretary authorized responsible bureaus and offices to determine whether to continue, revise, or terminate existing reform efforts or launch new initiatives. However, State did not formally communicate other changes in its reform priorities to Congress, such as its plan to no longer combine State and USAID’s real property offices. State initiated another transition in leadership of the reform efforts in April 2018 when it disbanded the dedicated planning teams overseeing the reform efforts and delegated responsibility for implementing the reform projects to relevant bureaus and offices. As the planning teams finished working on their particular reform efforts and prepared to transfer these projects to the bureaus, some planning teams provided memos and reports on the status of their efforts and offered recommendations for the bureaus to consider when determining next steps in implementing the projects. Some implementing officials, however, reported that they received little or no direction regarding their projects or any other indication of continued interest in their project from department or bureau leadership aside from the initial notification that the project had been assigned to them. For example, in separate discussions with implementing officials responsible for three different projects, the officials reported that they had not received any direction or other guidance related to their assigned project since it was delegated to them in April 2018. In one case, this lack of communication continued for nearly a year. In addition, although implementing officials said that they have managed to incorporate reform-related work into their daily responsibilities, they noted that there were multiple benefits from having had dedicated planning teams to lead earlier phases of State’s reform efforts. For example, they said that the dedicated teams included senior officials and the regular involvement of high-level leadership facilitated by these teams had helped advance the reform efforts. These dedicated teams also required staff to set aside time to focus on reform initiatives, which allowed them to develop holistic solutions to reform-related challenges. Conversely, implementing officials reported negative implications of not having dedicated teams. For example, one implementing official described how positive work initiated under the leadership of these dedicated teams—including efforts to eliminate redundancies and identify opportunities for consolidation—ended when the teams were disbanded because the staff and resources needed to continue these efforts were no longer available. Various State officials noted that the prolonged absence of Senate- confirmed leadership in key positions posed additional challenges. We have previously testified that it is more difficult to obtain buy-in on long- term plans and efforts that are underway when an agency has leaders in acting positions because federal employees are historically skeptical of whether the latest efforts to make improvements are going to be sustained over a period of time. For example, State did not have a Senate-confirmed Under Secretary for Management from January 2017 to May 2019. In November 2018, the Deputy Secretary of State told us that the lack of a confirmed Under Secretary for Management was hindering State’s ability to conduct business and implement reforms. The bureaus and offices responsible for 12 of State’s 13 continuing reform projects reported directly to an Acting Undersecretary for Management from January 2017 through May 2019. Moreover, State officials told us that both projects that we determined to be stalled were, among other things, awaiting the confirmation of an Under Secretary for Management to make key decisions. Furthermore, some implementing officials told us that the lack of confirmed officials in leadership positions within the bureaus responsible for implementing the projects added to a lack of leadership focus on implementing some of State’s reform projects. According to State officials, as of April 2019, although 13 of the reform projects described in the fiscal year 2019 Congressional Budget Justification were considered by State to be continuing, some had been scaled back, slowed down, or both as a result of senior leadership’s shifting priorities and attention. For example, one of State’s initial reform projects was related to better management of real property. However, State ultimately scaled back this project, effectively splitting it into two projects: One project focused on real property process improvements is continuing, but State has discontinued the other project to consolidate its and USAID’s real property function. Implementing officials told us in November 2018 that they were still pursuing the internal real property process improvements. They said then that they expected this reform project would likely progress at a slower pace without the dedicated team that previously had provided direct access and frequent interaction with senior department leadership. However, these officials recently informed us that the pace of progress on this project actually increased under the leadership of the bureau’s Senate-confirmed Director. The bureau was led by acting directors from January 2017 through September 2018. We have identified leadership focus and attention as practices vital to successfully implementing reform efforts. These practices include communicating clear and compelling reasons for the reforms, having a dedicated implementation team to manage the transformation process, and designating leaders responsible for implementing reforms and holding them accountable. Dedicating a strong and stable implementation team responsible for a transformation’s day-to-day management is important to ensuring that reforms receive the focused, full-time attention needed to be sustained and successful. One of the key responsibilities of a dedicated team is communication, particularly answering questions about the reform process from employees and other stakeholders. An implementation team is also important to ensuring that reform efforts are implemented in a coherent and integrated way. Because an agency’s transformation process is a large undertaking, we have found that an implementation team must have direct access to and be accountable to top leadership. In turn, top leadership must vest the team with the necessary authority and resources to set priorities, make timely decisions, and move quickly to implement top leadership’s decisions regarding the transformation. In addition, we previously reported that the single most important element of successful improvement initiatives is the demonstrated commitment of top leaders. This commitment is most prominently demonstrated through top leaders’ personal involvement in developing and directing reform efforts. Federal standards for internal control in the federal government also emphasize the importance of maintaining leadership continuity in order to achieve agency objectives. As a result, in other reports, we have recognized that agency reform efforts can take years to implement and that the time frame required for change typically takes longer than the tenures of political leaders. Similarly, the time it takes to nominate and confirm officials for senior management positions can also hamper efforts to initiate reforms or sustain momentum needed to successfully implement reform initiatives. For these reasons, and others, we have highlighted the need to ensure that top leadership drives the transformation and establishes dedicated teams to manage the transformation process. Taken together, the leadership transitions at State had two significant effects on State’s reform efforts. First, the transition of departmental leadership and lack of direction and communication about subsequent changes in leadership’s priorities contributed to uncertainty among implementing officials about the future of individual reform projects. Second, according to implementing officials, the transition of project responsibility from dedicated teams to bureau-level implementing officials resulted in fewer resources and a lack of senior leadership involvement and attention for some projects. Absent leadership decisions, implementing officials will continue to struggle with understanding leadership priorities with regard to State’s reform efforts. Similarly, for any projects that are determined to be leadership priorities, day-to-day implementation activities will continue to be hampered by the lack of a dedicated team to guide and manage the agency’s overall reform effort. Although uncertainty exists about the leadership priorities regarding reform efforts, the bureaus and offices responsible for implementing State’s reform projects have taken steps to manage and monitor their reform projects. Our previous work has identified monitoring as another important practice when implementing reform efforts, including, among other things, developing implementation plans and ensuring transparency by publicly reporting on progress toward milestones. These practices are also incorporated into State’s Foreign Affairs Manual and other department policies. We found that the relevant bureaus and offices responsible for implementing reform projects had developed implementation plans and that these plans identified milestones and deliverables for the projects. For example the Human Resources Services Delivery project had an implementation plan with milestones and deliverables, such as identifying programs and functions for consolidation in 2019 and reducing human resource delivery costs by 14 percent by 2022. Similarly, we found that the implementation plan for the IT Modernization project incorporated milestones that including, among other things, implementing a comprehensive enterprise IT risk management program by fiscal year 2020; reducing average deployment time for new IT capabilities by 10 percent annually from fiscal year 2019 through fiscal year 2021; and increasing workforce access to cloud-based email and business data from 10 percent to 100 percent by September 30, 2019. With regard to monitoring, while there is no centralized mechanism for reporting progress on all projects, we found that each of the ongoing projects currently has some form of progress reporting. For example, State reports progress on projects with IT components—such as Real- Time Collaboration and Work Anytime, Anywhere and Improve Enterprise-Wide Data Accessibility—as part of its quarterly reporting on IT Modernization under the Government Performance and Results Modernization Act of 2010. As a result, these projects have continued within a formal monitoring structure that involves regular web-based status updates and progress reporting. Other reform efforts—such as human capital and real property projects—are monitored against milestones established in State’s Joint Strategic Plan and progress is reported in State’s Annual Performance Reports. Progress for certain projects is also monitored and reported in other reports, such as State’s joint strategic plan, IT strategy, or human capital plan. Finally, other reform projects, such as State’s acquisition reform efforts, are reported at the government-wide level as part of the Cross-Agency Priority Goals outlined in the President’s Management Agenda. State collects data and evidence in order to measure progress in achieving outcome-oriented goals it sets for these projects. State reports these goals and relevant performance data in its annual performance plans and reports. For example, State uses the U.S. General Services Administration’s Customer Satisfaction Survey to measure and report the performance of its Human Capital Delivery Services reform efforts. State also uses data collected through the Office of Personnel Management’s Federal Employee Viewpoint Survey to measure employee satisfaction, which State established as a performance indicator for this project. Effectively implementing major reforms can span several years and must be closely managed. In 2017, State began a reform effort that led to 17 reform projects, most of which are unimplemented but still continuing. State notified both OMB and the Congress of these projects. Nevertheless, State leadership has not provided the focus necessary to support the officials responsible for implementing all these reform projects. When a new Secretary of State took charge in March 2018, he transferred responsibility for implementing the reform efforts from dedicated teams led by senior department leadership to bureaus and offices. In addition, key political appointee positions remained filled by officials in an acting capacity until only recently. These transitions at State have had an effect on its reform efforts. Without explicit direction from senior leadership, some implementing officials involved in the reform efforts remain unclear about whether their projects are an agency priority. Further, for the reform efforts that remain an agency priority, a dedicated team to oversee implementation could help accelerate State’s efforts to improve the efficiency and effectiveness of its operations. We are making the following two recommendations to State: The Secretary of State should determine which of the unimplemented reform projects included in its fiscal year 2019 Congressional Budget Justification, if any, should be implemented and communicate this determination to Congress and appropriate State personnel. (Recommendation 1) The Secretary of State should establish a single dedicated team to manage the implementation of all reform efforts that the Secretary decides to pursue. (Recommendation 2) We provided a draft of this report to State, USAID, and OMB for review and comment. We received comments from State and USAID, which are reprinted in appendixes II and III, respectively. In response to our recommendation that State determine which reform projects should be implemented and communicate that information to Congress and appropriate State personnel, State indicated that it concurred but suggested it should inform OMB instead of Congress. While we agree that it is important for State to share information regarding its reform efforts with OMB, we remain concerned about State’s lack of communication with Congress regarding the status of the projects State initially reported in its fiscal year 2019 Congressional Budget Justification. Congress is a key stakeholder in State’s reform efforts and should be informed of changes in State’s priorities and the status of these projects to help ensure successful implementation. In response to our recommendation that State establish a dedicated team to manage the implementation of all reform projects, State suggested that leadership of its reform projects should be decided on a case-by-case basis with the latitude to determine whether projects will be assigned to a higher level or within individual bureaus. We stand by our recommendation that State should establish a single dedicated team to manage the implementation of all its reform efforts. This is a key practice for implementing agency reforms identified in previous GAO reports, as well as in State’s Foreign Affairs Manual (1 FAM 014.2), which calls for State to “dedicate an implementation team to manage the transformation process” for major reorganizations of bureaus or offices. Because reform efforts can span several years, dedicating a strong and stable team is important to ensure that the transformation receives the needed attention to be sustained and successful. In its comments, USAID expressed several concerns about the leadership of State’s reform efforts and State’s coordination with USAID. OMB did not provide written comments on the report. We also received technical comments from State and USAID, which we incorporated throughout our report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of State, the Administrator of USAID, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-6881 or BairJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. We prepared this report under the authority of the Comptroller General to conduct work to assist Congress with its oversight responsibilities. This report examines (1) the status of the reform efforts that the Department of State (State) reported to Congress in its fiscal year 2019 Congressional Budget Justification and (2) the extent to which State addressed key practices we previously identified as critical to the successful implementation of agency reform efforts. For the purposes of this review, we use the term “reform efforts” to refer to all reform-related projects, proposals, plans, activities, and documents related to the 16 projects identified in State’s fiscal year 2019 Congressional Budget Justification. The term “projects” refers specifically to the 16 reform projects identified in State’s fiscal year 2019 Congressional Budget Justification. State subsequently split one of these 16 projects into two separate projects; thus, we refer to 17 reform projects throughout the report. For both objectives, we reviewed State’s reform plans, proposals, and related documents. We also interviewed four senior officials—generally at or above the assistant secretary level—that had responsibility for the reform efforts as a whole, as well as all implementing officials responsible for each of the continuing reform projects. To determine the status of State’s reform efforts, we reviewed documents and reports related to each of the reform projects described in State’s fiscal year 2019 Congressional Budget Justification. To determine the extent to which State addressed key practices for implementing agency reforms, we assessed State’s reform efforts against key questions identified in the implementation category of our June 2018 report. Specifically, we assessed State’s implementation efforts against key questions from the two implementation-related subcategories of our 2018 report: (1) Leadership Focus and Attention and (2) Managing and Monitoring. We considered the nature of each of State’s reform projects and the efforts taken to implement them, reviewed project-specific reports and other relevant State documents, interviewed State officials responsible for implementing each project, and then made qualitative determinations about the extent to which State’s overall reform efforts addressed these criteria. A second analyst then independently reviewed and validated each determination. Subsequently, other GAO staff reviewed and concurred with these determinations. We only applied criteria from our June 2018 report that we determined were relevant to the scope of our review, which was limited to the implementation phase of State’s reform efforts—from April 2018 to the present—to avoid duplicating the reviews of earlier phases of State’s reform efforts conducted by State’s and the U.S. Agency for International Development’s Offices of Inspector General (OIG). Because State’s OIG was also reviewing State’s reform efforts, we coordinated regularly with State’s OIG to avoid duplication. We did not consider criteria from the first two categories of our June 2018 report—(1) Goals and Outcomes and (2) Process for Developing Reforms—because these applied to the initial phases of State’s reform efforts, which were outside the scope of our work and central to the broader historical review that State’s OIG was conducting at the time of our review. We also did not apply criteria from the final category of our June 2018 report—Strategically Managing the Federal Workforce—to avoid duplicating work State’s OIG recently conducted on State’s workforce management. For the two sub- categories that we selected, we considered the key questions in the report in light of their relevance to State reforms efforts, and also employed other relevant criteria, where appropriate, most notably criteria for leadership from federal internal control standards. We conducted this performance audit from October 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Jason Bair, (202) 512-6881, or bairj@gao.gov. In addition to the contact named above, Thomas Costa (Assistant Director), Joshua Akery (Analyst in Charge), Peter Beck, David Dayton, Martin de Alteriis, Emily Gupta, Patrick Hickey, Chris Keblitis, Sarah Veale, and Alex Welsh made key contributions to this report.", "summary": "In 2017, State initiated a series of reform efforts in response to an executive order by the President and guidance issued by the Office of Management and Budget aimed at reorganizing and streamlining the government. GAO's prior work has shown that successful agency reform efforts follow key implementation practices, such as establishing a dedicated team to manage the implementation of reforms, and ensuring transparency by setting public goals and milestones to monitor progress. This report examines (1) the status of the reform efforts that State reported to Congress in February 2018 and (2) the extent to which State addressed key practices critical to the successful implementation of agency reform efforts. GAO reviewed State's reform plans, proposals, and related documents; met with officials involved in State's reform efforts; and assessed implementation of the reform efforts against relevant key practices identified in GAO's prior work. The Department of State (State) is implementing most of the 17 reform projects it reported to Congress in February 2018, but a few are stalled or discontinued. State completed one project streamlining policy formulation, and continues working to implement 13 projects on topics including human resources, information technology, and data analytics. Progress on two projects related to overseas presence has stalled, and State has discontinued a project to consolidate real property management. State has not addressed certain key practices related to leadership focus and attention in implementing its reform efforts. Multiple transitions in State's leadership and changing priorities contributed to uncertainty about leadership support for reform projects.Top leadership is expected to drive any needed transformation by clarifying priorities and communicating direction to employees and stakeholders. In March 2018, the President replaced the Secretary of State, a transition that created uncertainty within the agency regarding the future of ongoing reform projects. While some officials stated that the new Secretary had expressed support for data analytics and cyber security reform efforts, other officials said they were unclear as to whether their projects remained a priority. According to senior officials, the current Secretary has focused on critical needs, such as ending the hiring freeze and increasing recruitment, and on launching new initiatives. In April 2018, State disbanded the dedicated teams overseeing its reform efforts and shifted responsibility to bureaus and offices. In some cases, officials assigned to lead reform projects reported receiving little or no direction from department leadership. GAO's prior work has highlighted the benefits of having a dedicated team to manage agency transformations. In addition, State officials indicated that the challenges posed by these transitions were compounded by a lack of Senate-confirmed leadership in key positions. Specifically, during the first 2 years of State's reform efforts, bureaus and offices responsible for implementing 12 of State's 13 continuing reform projects reported directly to one or more officials serving in an acting capacity. For example, State did not have a Senate-confirmed Under Secretary for Management from January 2017 to May 2019, which, according to senior officials, hindered State's reform efforts. According to State officials, taken together these leadership transitions led to several projects being scaled back, slowed down, or both. Although uncertainties exist about leadership priorities regarding the reform efforts, the bureaus and offices responsible for implementing reform projects have taken steps to manage and monitor them, consistent with key practices. Each of the continuing projects has implementation plans that include milestones and deliverables, and some report their progress publicly. For example, State reports on the progress of some projects in its annual performance plans and reports. The lack of a dedicated team to manage the reform process, however, could slow State's overall efforts. The Secretary of State should (1) determine which unimplemented reform projects, if any, should be implemented and communicate this determination to Congress and appropriate State personnel, and (2) establish a single dedicated team to manage the implementation of all reform efforts that the Secretary decides to pursue. State generally concurred with the recommendations.", "document_type": "gao"}
{"report": "Data from CDC’s 2015 NISVS indicate that about 43.6 million women (36.4 percent) and 37.3 million men (33.6 percent) in the United States have experienced sexual violence, physical violence, and stalking by an intimate partner. Approximately 21.4 percent of women and 14.9 percent of men in the United States experienced severe physical violence by an intimate partner. About 30 million women (25.1 percent) and 12 million men (10.9 percent) reported experiencing some effect from the violence. (See fig. 1 for the most commonly reported effects of intimate partner violence, as reported by NISVS.) Intimate partner violence can also result in death. Data from U.S. crime reports suggest that 16 percent of homicide victims (about one in six) are killed by an intimate partner. Strangulation victims, in particular, are at greater risk for being killed, according to the Training Institute on Strangulation Prevention. Research has shown that certain factors increase the risk that someone may experience intimate partner violence. For example, a review of research on risk factors for women who experience intimate partner violence identified younger age, less education, unemployment, pregnancy, childhood victimization, and mental illness as being associated with higher rates of intimate partner violence. Exposure to intimate partner violence between a child’s parents or caregivers is also associated with a greater risk of intimate partner violence in adulthood, according to CDC. Adults with disabilities are also at a higher risk of violence than those without disabilities. However, research indicates that victims of intimate partner violence may be less likely than others to obtain medical or other services. Even when services are obtained, victims may be less likely than others to identify the source or extent of their injuries out of fear for their safety or reprisal. Brain injuries, including those that may result from intimate partner violence, can have several causes, including physical trauma and strangulation; range in severity; and can result in a number of health consequences. TBI refers to a brain injury caused by external physical force, such as a blow to the head or shaking of the brain. Anoxic (a complete disruption of oxygen to the brain) or hypoxic (a partial disruption of oxygen to the brain) brain injury may result from strangulation or other pressure applied to the neck that restricts blood flow and air passage. TBIs and anoxic or hypoxic brain injuries may result in irreversible psychological and physical harm. Specifically, people who suffer from TBI and anoxic or hypoxic brain injuries may experience cognitive symptoms, including depression and memory loss, as well as behavioral symptoms, such as changes in mood, or difficulty sleeping, among others. The symptoms individuals experience can also vary. The signs and symptoms of an anoxic or hypoxic brain injury from strangulation can be similar to those of mild TBI, which is often referred to as a concussion. (See fig. 2.) According to the Brain Injury Association of America, a severe brain injury can be clearly identified by reviewing an individual’s symptoms, but when the brain injury is mild or moderate, providers may need to conduct further assessments or screening to diagnose the brain injury. According to NIH, providers have several options for assessing brain injury that can help determine the severity of the injury. For example, providers may evaluate a person’s level of consciousness and the severity of brain injury by attempting to elicit body movements, opening of the eyes, and verbal responses. Providers may also evaluate an individual’s speech and language skills or cognitive capabilities. Both HHS and DOJ support activities for individuals affected by intimate partner violence through several of their agencies. Within HHS, for example, ACF provides federal funding to support emergency shelter and services for the victims of domestic violence and their dependents, as well as the National Domestic Violence Hotline. CDC provides grants to state and local entities to develop programs aimed at preventing intimate partner violence. Additionally, HRSA—which provides funding to federally qualified health centers—provides funding to develop educational materials for health care workers, in partnership with ACF, to increase the number of individuals screened for intimate partner violence and referred to treatment services, among other things. DOJ, through its Office of Justice Programs and Office on Violence Against Women, conducts research and provides funding to help states, local governments, and nonprofit organizations’ develop programs to reduce violence against women. Many DOJ programs aim to strengthen responses at the local, state, tribal, and federal levels to domestic violence, dating violence, sexual assault, and stalking. Further, the Violence Against Women Reauthorization Act of 2013 amended federal laws to establish criminal penalties for strangulation or suffocation. Additionally, DOJ increased its support of activities focused on training to recognize and prosecute strangulation. HHS agencies also conduct work related to recognizing and responding to TBI. For example, NIH funds research aimed at developing knowledge about the brain and nervous system in order to reduce the effect of brain- related diseases on individuals. In addition, CDC conducts research on the prevention of TBIs, and ACL provides grants to states to help them to support individuals with brain injuries and to promote the rights of, and provide advocacy support to, those living with TBI. We identified 12 initiatives led by non-federal entities that focused on (1) education on brain injuries resulting from intimate partner violence by developing materials or offering training; (2) screening victims of intimate partner violence for potential brain injuries; or (3) treatment involving individuals with brain injuries resulting from intimate partner violence. Our list represents initiatives identified during the course of our review and may not be exhaustive. Some of these initiatives focus on only TBI or strangulation, while others focused on both. See appendix II for additional information on the initiatives. Training for Domestic Violence Program Staff Domestic violence program advocates we spoke to told us that before they participated in the Ohio Domestic Violence Network training, they knew their clients were having a hard time remembering things or getting their thoughts across; however, they did not know this could be the result of a brain injury. The training helped advocates identify signs and symptoms in their clients and make others aware of these symptoms. For example, advocates told us they may inform a prosecutor that a client may have a brain injury and may have difficulty remembering or sharing their experiences. The Ohio Domestic Violence Network—as a part of its Connect, Acknowledge, Respond, Evaluate (CARE) initiative—trained staff at five domestic violence programs on brain injuries, and developed educational materials for shelter staff to share with intimate partner violence victims, according to network officials. For example, we spoke to staff at a domestic violence program in Ohio who told us how the education they received from the network helped them identify the signs and symptoms of brain injury in their clients. Staff from another domestic violence program in Ohio told us as a result of CARE training they now suggest strategies to clients to assist them with their memory issues, such as writing appointment information on a whiteboard or in a planner. The Swedish Hospital Violence Prevention Program, in Illinois, provided education to physicians, medical residents, and hospital staff to increase health care provider and staff awareness of and ability to respond to brain injuries among victims of intimate partner violence, according to officials with the initiative. The Safe Futures initiative, in Connecticut, developed strangulation training materials for emergency medical personnel, law enforcement, prosecutors, and providers, as well as hosted trainings throughout Connecticut on intimate partner violence and brain injuries, according to officials with the initiative. Screening. Six of the 12 initiatives used screening tools to identify potential brain injuries among intimate partner violence victims, according to officials. Based on our review of documentation from these initiatives, we found that the screening tools generally had a series of questions about injuries to the head, the loss of consciousness, or behavior changes—symptoms that may indicate a potential brain injury. For example: Officials from three initiatives that screened victims for potential brain injuries reported using a version of the HELPS screening tool. (See fig. 4 for an example of a modified version of this screening tool used by one initiative.) Officials from one initiative told us that screening typically occurred at domestic violence shelters where staff and advocates receive training on how to screen intimate partner violence victims. Officials from the other three initiatives told us they developed their own screening methods. For example, staff at the Maricopa County Collaboration on Concussions in Domestic Violence in Arizona screen victims using a tool that measures near point of convergence, which refers to an individual’s ability to focus both eyes on a target, an approach that can be used to detect a concussion. Police officers from two participating departments in Arizona have used this tool to screen individuals when they respond to a domestic violence call, according to officials with the collaboration. Treatment. Two of the 12 initiatives included a treatment component. Officials with the Barrow Concussion and Brain Injury Center in Arizona and the Northside Hospital Duluth Concussion Institute in Georgia told us they provided treatment to victims who were referred by local domestic violence shelters. Providers affiliated with one of these initiatives told us that treatment for brain injuries resulting from intimate partner violence does not differ from treatment for other brain injuries. A provider with one of these initiatives said that treatment could include exercises and movements that decrease dizziness, vertigo, and imbalance; occupational, physical, or speech therapies; or treatment for pain management. An Intimate Partner Violence Victim’s Brain Injury Treatment Jane Doe was abused by her partner. An advocate at a domestic violence shelter screened Jane for a brain injury and referred her for assessment. She was diagnosed and began treatment for a brain injury. Jane Doe told us that the treatments she received, which included nerve blockers—often used by neurologists to lessen chronic pain—helped to relieve the persistent headaches and debilitating migraines she experienced in the aftermath of her abuse. She told us that as a result of the treatment she received, she feels better able to function. Officials from the Barrow Concussion and Brain Injury Center told us that individuals with brain injuries resulting from intimate partner violence may face a longer period of recovery compared to others with brain injuries, in part, because of living in unsafe home environments. As a result, special considerations are sometimes needed due to additional barriers faced by domestic violence victims. For example: Victims may need safety planning and housing. As a part of the Barrow Concussion and Brain Injury Center’s domestic violence initiative, a social worker will help ensure that victims’ other needs are met. Officials from the Northside Hospital Duluth Concussion Institute noted that transportation could also be a barrier for victims of intimate partner violence. As such, the Georgia Department of Public Health’s Injury Prevention Program, which partnered with the Northside Hospital Duluth Concussion Institute, planned to use CDC grant funding to provide domestic violence victims transportation from area shelters to the concussion institute for treatment. Officials from the Barrow Concussion and Brain Injury Center also told us about other considerations, such as the need to have a flexible appointment policy to account for the possibility of victims missing or canceling appointments. Of the 12 initiatives we identified, eight received federal grants from HHS or DOJ, while officials from the other four initiatives told us they were funded with state, local, or private dollars. According to HHS and DOJ officials, the grants did not have specific requirements to address the intersection of brain injuries and intimate partner violence. However, based on our review of documentation, the eight initiatives used the federal funds to focus on the intersection of these two issues. Six of these eight initiatives received funding from HHS. Of them, four were funded by HRSA or ACL grants that focused on TBI-related services and activities, and two were funded by CDC grants focused on injury and violence prevention activities. The other two initiatives were funded by DOJ’s Office of Justice Programs through grants that provide funds to support victims of crime. In addition to the federal funding received by some of the 12 initiatives, we identified other efforts and grants funded by HHS and DOJ. These efforts made educational materials on intimate partner violence and brain injuries accessible online, made ad-hoc or internal trainings available to external parties, or provided education that touched on the connection between intimate partner violence and brain injury, according to HHS and DOJ officials. For example: ACF has funded the National Resource Center on Domestic Violence and Futures Without Violence’s National Health Resource Center on Domestic Violence, which provide information related to intimate partner violence and brain injuries via websites. ACF, in collaboration with HRSA, funded an effort led by Futures Without Violence, which includes some information on TBI and strangulation in trainings for select state leadership teams working to address intersections of health, intimate partner violence, and human trafficking. DOJ’s Office on Violence Against Women provided grant funds to support the Training Institute on Strangulation Prevention, which offers training to individuals and outside entities to help them understand, recognize, and appropriately serve strangulation victims, as well as investigate and prosecute strangulation cases. DOJ’s Office on Violence Against Women has also provided grant funds used by local organizations, such as police departments, to provide ad-hoc or internal training activities on brain injuries and to serve victims with brain injuries, including those caused by strangulation. Based on our review of the literature, as well as interviews with HHS officials and other non-federal stakeholders, we found that data on the overall prevalence of brain injuries resulting from intimate partner violence are limited. Specifically, available data do not provide an overall estimate of the prevalence of brain injuries resulting from intimate partner violence nationwide. While there are studies that estimate the prevalence of these injuries, these studies are also limited. Specifically, among the 28 articles we reviewed, six included an objective to estimate the prevalence of brain injuries resulting from intimate partner violence, while the remaining 22 articles examined other areas, such as health effects or awareness of brain injuries resulting from intimate partner violence, but did not have an objective to estimate prevalence. The six articles are also specific to a certain subpopulation or certain geographic locations and used different approaches to identify individuals with brain injuries. As a result, the range of reported prevalence rates on victims of intimate partner violence with brain injuries (brain injuries caused by trauma or strangulation) varied greatly (from 11 percent to about 79 percent) and were based on a range of sample sizes, from 95 people to about 1,000 people. HHS agencies also have some data collection and research efforts related to this issue; however, these efforts are limited as well. For example, CDC and NIH have efforts that may assist in better understanding the connection between brain injuries and intimate partner violence, but CDC’s efforts do not account for all causes of brain injuries and NIH has only one study focused on this connection. Further, HHS agencies treat brain injuries and intimate partner violence as separate public health issues and pursue their efforts separately—which limits their ability to better understand the connection between the issues and the overall prevalence of brain injuries that result from domestic violence. CDC officials told us that the agency’s data on the connection between brain injuries and intimate partner violence are limited, but the agency plans to address some of the limitations. For example, the officials said CDC analyzes health care claims data from emergency department visits to determine the causes of TBI. However, CDC officials told us that these data likely underestimate TBI among victims of intimate partner violence, because many do not seek medical care; for domestic violence victims who seek care, providers are unlikely to designate the individual as a victim of intimate partner violence. CDC also collects data on intimate partner violence through its NISVS. According to CDC reports, NISVS data are a key source of information on intimate partner violence, but the survey does not collect data on all types of brain injuries related to intimate partner violence. For example, the NISVS estimates the prevalence of victims of intimate partner violence who have been “knocked out after getting hit, slammed against something, or choked.” However, published estimates are based on responses to a survey question that asks individuals about being “knocked out,” which is a colloquial term commonly used to indicate a loss of consciousness. CDC officials stated that in most known incidents of mild brain injury, people do not lose consciousness. As a result, NISVS data likely understate the number of intimate partner violence victims who may have brain injuries. In order to better estimate TBIs resulting from intimate partner violence, CDC officials told us they plan to add a survey question to the NISVS to ask respondents about whether they have experienced a concussion—a common term for mild forms of TBI—due to a current or ex-partner. CDC officials told us that they have begun initial testing on several aspects of the survey, including on the additional question with the goal to begin data collection by the end of 2022, plans which are pending approval. Once the NISVS data are collected and analyzed, CDC officials said the data could help them provide a nationally representative prevalence estimate of intimate partner violence victims’ who experienced a TBI in their lifetimes. However, adding the question to the NISVS may not ensure that these data can provide a comprehensive estimate of the prevalence of brain injuries resulting from intimate partner violence. In particular, The NISVS question will focus on TBIs, and will not account for individuals with brain injuries caused by strangulation. According to educational materials developed by the Training Institute of Strangulation Prevention and used by HRSA in the training of providers and advocates, more than two-thirds of intimate partner violence victims are strangled at least once. CDC officials told us that they are able to measure acts of choking or suffocation through the NISVS, but this measure cannot be used to account for brain injuries resulting from strangulation. Additionally, CDC officials told us that the agency’s priority is to focus on TBI specifically rather than accounting for other brain injuries. Despite the focus on TBIs, CDC officials told us the NISVS data are not designed to examine whether intimate partner violence is a leading cause of TBI in comparison with other causes, such as sports or motor vehicle crashes. CDC officials said that some research and NISVS data suggest that intimate partner violence is not as large a contributor of TBIs when compared to other contributors. However, they noted that they do not have data on the proportion of TBIs resulting from intimate partner violence. Absent the ability to compare intimate partner violence as a cause of TBI against other contributors through the NISVS or other representative studies, CDC officials will continue to lack an understanding of the full scope of TBIs, their primary causes, and who is affected by them. NIH officials identified two agency efforts that could help improve what is known about the connection between brain injuries and intimate partner violence. NIH began funding a study in September 2019 that will use advanced brain imaging, blood analyses, and cognitive and psychological testing to study the effects of multiple brain injuries on women subjected to intimate partner violence. The objectives of the study are not to measure prevalence, but to examine the health effects of brain injuries resulting from intimate partner violence. NIH officials told us that this is the first study funded by NIH using brain images to investigate brain injuries resulting from intimate partner violence. NIH is also developing blood biomarkers—which are clinical diagnosis tools—for identifying mild TBI. Currently, mild TBI is generally diagnosed by taking an inventory of symptoms, but symptoms can lead to misdiagnoses, including for mental illness or a substance use disorder. NIH officials said they are in the initial stages of developing these biomarkers, which could take the place of screening tools in diagnosing a brain injury. While this effort was not initiated to better understand brain injuries among victims of intimate partner violence, biomarkers have the potential to improve the identification of TBIs, provided they are applied to domestic violence victims. Two other HHS agencies—ACL and HRSA—also have efforts that address brain injuries or intimate partner violence. However, these agencies’ efforts are generally not focused on the connection of the two issues, so they are not likely to result in more complete data on the prevalence of brain injuries resulting from intimate partner violence. Specifically: ACL provides grants to states to establish support services for individuals with brain injuries through its TBI State Partnership Program. As part of these efforts, ACL officials told us that they have begun to gather information to determine how many TBI grant recipients are using the funds to support particular populations, including individuals with TBI resulting from intimate partner violence. As of December 2019, ACL officials told us that two states (Idaho and Iowa) have used the grants to focus on individuals with TBI as a result of intimate partner violence. HRSA has proposed an effort to collect data that may assist in further understanding the health consequences of intimate partner violence. As part of its strategy to address intimate partner violence, HRSA officials recently began requiring federally qualified health centers to capture International Classification of Diseases-10 codes for intimate partner violence on health care claims beginning in 2020. This effort is not aimed at the intersection of intimate partner violence and brain injuries; the purpose of this data collection is to better understand the effect of intimate partner violence on victims’ health outcomes. While these data may currently underestimate the number of individuals affected by intimate partner violence, HRSA officials told us that their goal in collecting these data is to underscore the significance of intimate partner violence and help position providers to assist victims. Further, knowing the prevalence of brain injuries resulting from intimate partner violence and using these data could help officials further target education campaigns to providers on the potential injuries associated with intimate partner violence. Officials from HHS agencies acknowledge that the lack of prevalence data on brain injuries resulting from intimate partner violence is a challenge in addressing the intersection of these issues. However, HHS and its agencies do not have a plan for how they would collect better prevalence data, including a plan that specifies the extent to which HHS agencies should collaborate on data collection efforts. Although HHS agencies have some efforts underway, these efforts are limited or do not examine the connection between the issues. For example, CDC is working to add a question to NISVS to improve what is known about the prevalence of TBIs among victims of intimate partner violence; however, this effort overlooks brain injuries resulting from strangulation—which HRSA reports is often also experienced by these victims—because CDC’s priorities are to focus on TBIs specifically. Further, the newly funded NIH study is not intended to estimate the overall prevalence of brain injuries resulting from intimate partner violence. Having complete data on the prevalence of brain injuries resulting from intimate partner violence could strengthen HHS’s efforts to address this public health issue. HHS and its agencies acknowledge that enhancing the health and well-being of Americans is critical to their public health mission and intimate partner violence and TBIs are both prominent injury and violence issues. As part of this mission, CDC uses its Public Health Approach to guide its public health related efforts. The first step of this approach is to define the problem, which includes collecting prevalence data to understand the magnitude of the problem, where the problem exists, and whom it affects. According to CDC, such data are critical to ensuring that resources are focused on the individuals most in need. Collecting data on the prevalence of brain injuries resulting from intimate partner violence is a critical first step. With better data comes a better understanding of the overall prevalence of brain injuries resulting from intimate partner violence. This would give HHS and its agencies the information necessary to inform their efforts and allocate resources, including grant funding, to address victims of brain injuries resulting from intimate partner violence. Intimate partner violence affects over 30 percent of women and men in the United States, and research has raised concerns about brain injuries sustained by these domestic violence victims. Officials from HHS agencies acknowledge the lack of overall prevalence data on brain injuries resulting from intimate partner violence and the adverse effect this lack of data has on understanding the intersection of these two issues. While HHS agencies have some efforts underway to address brain injuries and intimate partner violence, they are limited and address these issues separately. Therefore, HHS and its agencies have missed an opportunity to improve their public health efforts to address this issue, particularly the prevalence of the problem, where the problem exists, and whom it affects. By working together, HHS and its agencies can identify ways that each agency’s efforts could result in better prevalence data and a better overall understanding of brain injuries resulting from intimate partner violence. Improved data, in turn, could also help ensure that federal resources are allocated to the appropriate areas and used as efficiently and effectively as possible to address this public health issue. We are making the following recommendation to HHS: The Secretary of HHS should develop and implement a plan to improve data collected on the prevalence of brain injuries resulting from intimate partner violence and use these data to inform HHS’s allocation of resources to address the issue. (Recommendation 1) We provided a draft of this report to HHS and DOJ for review and comment. In its written comments (reproduced in app. III), HHS concurred with our recommendation and noted that it is coordinating a plan amongst its relevant agencies to augment data collection on the prevalence of brain injuries resulting from intimate partner violence. HHS noted that these data will continue to inform the needs of this vulnerable population. HHS and DOJ also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, Secretary of Health and Human Services, Attorney General, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or yocomc@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. We identified articles for our literature review through a search of bibliographic databases, including Harvard Library Think Tank Search, MEDLINE, and Scopus, using terms such as “intimate partner violence,” “domestic violence,” “traumatic brain injury,” and “strangulation.” We determined there were 57 relevant articles from 2009 through August 2019 discussing brain injuries resulting from intimate partner violence. We reviewed the 57 articles to examine brain injuries resulting from intimate partner violence, including background information on the concerns of brain injuries resulting from intimate partner violence and challenges that researchers may have identified in conducting this work. Of the 57 articles, we identified 28 that had conducted their own data analyses. We analyzed these 28 articles to examine data on prevalence rates, as well as research on health effects, treatment, and screening tools for identifying brain injuries resulting from intimate partner violence. The following articles are based on an original analysis of data. Brown, Joshua, Dessie Clark, and Apryl E. Pooley. “Exploring the Use of Neurofeedback Therapy in Mitigating Symptoms of Traumatic Brain Injury in Survivors of Intimate Partner Violence.” Journal of Aggression, Maltreatment & Trauma, vol. 28, no. 6 (2019): 764-783. Campbell, Andrew M., Ralph A. Hicks, Shannon L. Thompson, and Sarah E. Wiehe. “Characteristics of Intimate Partner Violence Incidents and the Environments in Which They Occur: Victim Reports to Responding Law Enforcement Officers.” Journal of Interpersonal Violence (2017): 1-24. Campbell, Jacquelyn C., Jocelyn C. Anderson, Akosoa McFadgion, Jessica Gill, Elizabeth Zink, Michelle Patch, Gloria Callwood, and Doris Campbell. “The Effects of Intimate Partner Violence and Probable Traumatic Brain Injury on Central Nervous System Symptoms.” Journal of Women’s Health, vol. 27, no. 6 (2018): 761-767. Cimono, Andrea, N., Grace Yi, Michelle Patch, Yasmin Alter, Jacquelyn C. Campbell, Kristin K. Gunderson, Judy T. Tang, Kiyomi Tsuyuki, and Jamila K. Stockman. “The Effect of Intimate Partner Violence and Probable Traumatic Brain Injury on Mental Health Outcomes for Black Women.” Journal of Aggression, Maltreatment & Trauma, vol. 28, no. 6 (2019): 714-731. Crowe, Allison, Christine E. Murray, Patrick R. Mullen, Kristine Lundgren, Gwen Hunnicutt, and Loreen Olson. “Help-Seeking Behaviors and Intimate Partner Violence-Related Traumatic Brain Injury.” Violence and Gender, vol. 6, no. 1 (2019): 64-71. Gagnon, Kelly L., and Anne P. DePrince. “Head Injury Screening and Intimate Partner Violence: A Brief Report.” Journal of Trauma & Dissociation, vol. 18, no. 4 (2017): 635-644. Higbee, Mark, Jon Eliason, Hilary Weinberg, Jonathan Lifshitz, and Hirsch Handmaker. “Involving Police Departments in Early Awareness of Concussion Symptoms during Domestic Violence Calls.” Journal of Aggression, Maltreatment & Trauma, vol. 28, no. 7 (2019): 826-837. Hunnicutt, Gwen, Christine Murray, Kristine Lundgren, Allison Crowe, and Loreen Olson. “Exploring Correlates of Probable Traumatic Brain Injury among Intimate Partner Violence Survivors.” Journal of Aggression, Maltreatment & Trauma, vol. 28, no. 6 (2019): 677-694. Hux, Karen, Trish Schneider, and Keri Bennett. “Screening for traumatic brain injury.” Brain Injury, vol. 23, no. 1 (2009): 8-14. Joshi, Manisha, Kristie A. Thomas, and Susan B. Sorenson. “‘I Didn’t Know I Could Turn Colors’: Health Problems and Health Care Experiences of Women Strangled by an Intimate Partner.” Social Work in Health Care, vol. 51, no. 9 (2012): 798-814. Linton, Kristen Faye. “Interpersonal violence and traumatic brain injuries among Native Americans and women.” Brain Injury, vol. 29, no. 5 (2015): 639-643. Messing, Jill T., Kristie A. Thomas, Allison L. Ward-Lasher, and Nathan Q. Brewer. “A Comparison of Intimate Partner Violence Strangulation Between Same-Sex and Different-Sex Couples.” Journal of Interpersonal Violence, vol. 00, no. 0 (2018): 1-19. Messing, Jill T., Michelle Patch, Janet S. Wilson, Gabor D. Kelen, Jacquelyn Campbell. “Differentiating among Attempted, Completed, and Multiple Nonfatal Strangulation in Women Experiencing Intimate Partner Violence.” Women’s Health Issues, vol. 28, no. 1 (2018): 104-111. Mittal, Mona, Kathryn Resch, Corey Nichols-Hadeed, Jennifer Thompson Stone, Kelly Thevenet-Morrison, Catherine Faurot, and Catherine Cerulli. “Examining Associations between Strangulation and Depressive Symptoms in Women with Intimate Partner Violence Histories.” Violence and Victims, vol. 33, no. 6 (2019): 1072-1087. Nemeth, Julianna M., Cecilia Mengo, Emily Kulow, Alexandra Brown, and Rachel Ramirez. “Provider Perceptions and Domestic Violence (DV) Survivor Experiences of Traumatic and Anoxic-Hypoxic Brain Injury: Implications for DV Advocacy Service Provision.” Journal of Aggression, Maltreatment & Trauma, vol. 28, no. 6 (2019): 744-763. Pritchard, Adam J., Amy Reckdenwald, Chelsea Nordham, and Jessie Holton. “Improving Identification of Strangulation Injuries in Domestic Violence: Pilot Data From a Researcher-Practitioner Collaboration.” Feminist Criminology, vol. 12, no. 2 (2018): 160-181. Ralston, Bridget., Jill Rable, Todd Larson, Hirsch Handmaker, and Jonathan Lifshitz. “Forensic Nursing Examination to Screen for Traumatic Brain Injury following Intimate Partner Violence.” Journal of Aggression, Maltreatment & Trauma, vol. 28, no. 6 (2019): 732-743. Reckdenwald, Amy, Ketty Fernandez, and Chelsea L. Mandes. “Improving law enforcement’s response to non-fatal strangulation.” Policing: An International Journal (2019): 1-15. Reckdenwald, Amy, Chelsea Nordham, Adam Pritchard, and Brielle Francis. “Identification of Nonfatal Strangulation by 911 Dispatchers: Suggestions for Advances Toward Evidence-Based Prosecution.” Violence and Victims, vol. 32, no. 3 (2017): 506-520. Shields, Lisa B.E., Tracey S. Corey, Barbara Weakley-Jones, and Donna Stewart. “Living Victims of Strangulation.” American Journal of Forensic Medicine and Pathology, vol. 31, no. 4 (2010): 320-325. St. Ivany, Amanda, Linda Bullock, Donna Schminkey, Kristen Wells, Phyllis Sharps, and Susan Kools. “Living in Fear and Prioritizing Safety: Exploring Women’s Lives After Traumatic Brain Injury From Intimate Partner Violence.” Qualitative Health Research, vol. 28, no. 11 (2018): 1708-1718. St. Ivany, Amanda, Susan Kools, Phyllis Sharps, and Linda Bullock. “Extreme Control and Instability: Insight Into Head Injury From Intimate Partner Violence.” International Association of Forensic Nursing, vol. 14, no. 4 (2018): 198-205. St. Ivany, Amanda, and Donna Schminkey. “Rethinking Traumatic Brain Injury from Intimate Partner Violence: A Theoretical Model of the Cycle of Transmission.” Journal of Aggression, Maltreatment & Trauma, vol. 28, no. 7 (2019): 1-23. Sullivan, Karen A, and Christina Wade. “Assault-Related Mild Traumatic Brain Injury, Expectations of Injury Outcome, and the Effects of Different Perpetrators: A Vignette Study.” Applied Neuropsychology: Adult, vol. 26, no. 1 (2019): 58-64. Sullivan, Karen A, and Christina Wade. “Does the Cause of the Mild Traumatic Brain Injury Affect the Expectation of Persistent Postconcussion Symptoms and Psychological Trauma?” Journal of Clinical and Experimental Neuropsychology, vol. 39, no. 4 (2017): 408- 418. Valera, Eve M., Aihua Cao, Ofer Pasternak, Martha E. Shenton, Marek Kubicki, , Nikos Makris, and Noor Adra. “White Matter Correlates of Mild Traumatic Brain Injuries in Women Subjected to Intimate-Partner Violence: A Preliminary Study.” Journal of Neurotrauma, vol. 36 (2019): 661-668. Valera, Eve, and Aaron Kucyi. “Brain Injury in Women Experiencing Intimate Partner-Violence: Neural Mechanistic Evidence of an “Invisible” Trauma.” Brain Imaging and Behavior, vol. 11 (2017): 1664-1677. Zieman, Glynnis, Ashley Bridwell, and Javier F. Cardenas. “Traumatic Brain Injury in Domestic Violence Victims: A Retrospective Study at the Barrow Neurological Institute.” Journal of Neurotrauma, vol. 33, (2016): 1- 5. The following table provides a brief overview of each of the 12 initiatives we identified based on information provided by the Department of Health and Human Services, the Department of Justice, and other stakeholders. These initiatives engage in various efforts to address intimate partner violence and brain injuries, including traumatic brain injury and anoxic injuries resulting from strangulation. Our list includes those efforts identified during the course of our review and may not be exhaustive. The descriptions of initiatives are based on our review of documentation and information obtained from interviews with officials. In addition to the contact named above, Shannon Slawter Legeer (Assistant Director), Danielle Bernstein (Analyst-in-Charge), and Ashley Dixon made key contributions to this report. Also contributing were Leia Dickerson, Kaitlin Farquharson, Drew Long, and Ethiene Salgado- Rodriguez.", "summary": "Research has found brain injuries to be common among victims of intimate partner violence, and that such injuries are under-diagnosed and under-treated. House Report 115-952 included a provision for GAO to report on the relationship between intimate partner violence and brain injuries. GAO (1) describes efforts to provide education, screen for, or treat brain injuries resulting from intimate partner violence; and (2) examines what is known about the prevalence of brain injuries resulting from intimate partner violence, including HHS efforts to determine prevalence. GAO reviewed peer-reviewed literature, federal websites, and documentation from HHS and DOJ. GAO also interviewed officials from HHS, DOJ, and 11 non-federal stakeholders, such as domestic violence organizations. GAO identified 12 initiatives, though this list may not be exhaustive, and conducted site visits to three of them. According to the Centers for Disease Control and Prevention (CDC), one in three adults have experienced domestic violence, also known as intimate partner violence. Intimate partner violence includes physical violence, sexual violence, stalking, and psychological aggression. Victims of intimate partner violence may experience brain injury, resulting from blows to the head or strangulation. To address this issue, the Department of Health and Human Services (HHS) and the Department of Justice (DOJ) provide grants to state and local entities that work with victims. GAO identified 12 non-federal initiatives that provide education, screen for, or treat brain injuries resulting from intimate partner violence. All 12 developed and distributed education and training materials to domestic violence shelter staff, victims, health care providers, and others. Six of the 12 initiatives used screening tools to identify potential brain injuries among intimate partner violence victims, and two included a treatment component. Additionally, eight of the 12 initiatives received HHS or DOJ grant funding, although agency officials told us the funding had no specific requirements to address brain injuries resulting from intimate partner violence. Based on its review of the literature, as well as interviews with HHS officials and other non-federal stakeholders, GAO found that data on the overall prevalence of brain injuries resulting from intimate partner violence are limited. HHS officials acknowledged that the lack of data on the prevalence of these issues is a challenge in addressing the intersection of the issues. However, HHS does not have a plan for how it would collect better prevalence data. HHS agencies have some related efforts underway; however, the efforts are limited and generally do not examine the connection between brain injuries and intimate partner violence. Enhancing the health and well-being of Americans is critical to HHS's public health mission. As part of this mission, CDC, within HHS, uses its Public Health Approach, which includes collecting prevalence data to understand the magnitude of public health issues. With better data comes a better understanding of the overall prevalence of brain injuries resulting from intimate partner violence. This, in turn, could help ensure that federal resources are allocated to the appropriate areas and used as efficiently and effectively as possible to address this public health issue. HHS should develop and implement a plan to improve data collected on the prevalence of brain injuries resulting from intimate partner violence and use these data to inform its allocation of resources to address the issue. HHS concurred with our recommendation and is coordinating with its agencies to augment data collection.", "document_type": "gao"}
{"report": "In most cases, a noncitizen must be a LPR to enlist in the U.S. Armed Forces. Special provisions of the INA authorize the naturalization of current and recently discharged service members. Qualifying military service includes active or reserve service in the U.S. Army, Navy, Marine Corps, Air Force, Coast Guard, or service in a National Guard unit. A person who has served honorably in the U.S. Armed Forces for 1 year during peacetime may be eligible to apply for naturalization. In addition, during designated periods of hostilities, such as World War I and World War II and the current global war on terrorism, members of the U.S. Armed Forces who serve honorably in an active duty status, or as members of the Selected Reserve of the Ready Reserve, are eligible to apply for naturalization without meeting any minimum required period of service. DOD determines if a service member meets the qualifying service requirement by certifying Form N-426, Request for Certification of Military or Naval Service, or by issuing Forms DD-214, Certificate of Release or Discharge from Active Duty, NGB-22, National Guard Report of Separation and Record of Service, or an equivalent discharge document. The information provided in those forms determines whether or not the service member completed all requirements for honorable service, including whether the service member served honorably and, if he or she has separated from service, whether his or her separation was under honorable conditions. In order to naturalize, a member of the U.S. Armed Forces must also meet the requirements and statutory qualifications to become a citizen. Specifically, he or she must demonstrate good moral character and have sufficient knowledge of the English language, U.S. government, and history. Additionally, an applicant must show attachment to the principles of the Constitution and favorable disposition toward the good order and happiness of the United States. However, qualified members of the U.S. Armed Forces are exempt from other naturalization requirements, including application fees and requirements for continuous residence and physical presence in the United States. DOD also has authority to expand military recruiting to certain nonimmigrants and other lawfully present aliens. Beginning in December 2008, the Military Accessions Vital to the National Interest (MAVNI) program allowed certain U.S. nonimmigrant visa holders, asylees, refugees, and individuals with Temporary Protected Status to enlist in the military if they possessed medical, language, and other types of skills deemed vital for military operations. DOD ended the MAVNI program in fiscal year 2016, citing counterintelligence concerns. Between 2008 and 2016, 10,400 individuals enlisted in the U.S. military through the MAVNI program, according to DOD data. DHS is responsible for arresting, detaining, litigating charges of removability against, and removing foreign nationals who are suspected and determined to be in the United States in violation of U.S. immigration laws. Trial attorneys from ICE’s OPLA represent the U.S. government as civil prosecutors in immigration court removal proceedings. ICE’s ERO is responsible for arresting and detaining potentially removable foreign nationals pending the outcome of their immigration court cases and removing individuals subject to an immigration judge’s final order of removal. ICE’s HSI is responsible for investigating a range of domestic and international activities arising from the illegal movement of people and goods into, within, and out of the United States. Individuals may be subject to removal for a wide variety of reasons, including entering the United States illegally, staying longer than their authorized period of admission, being convicted of certain crimes, or engaging in terrorist activity. LPRs are foreign nationals under U.S. immigration law and therefore may be subject to immigration enforcement and removal from the United States for reasons such as controlled substance violations or conviction of an aggravated felony. Both HSI agents and ERO officers may encounter potentially removable individuals and are to decide whether to issue them a charging document, known as a NTA, ordering the individual to appear before an immigration judge to respond to removal charges. If the judge finds that the respondent is removable and not otherwise eligible for relief, the judge will issue an order of removal, subjecting the respondent to removal by ERO once the order is administratively final. The VA is responsible for administering benefits and services, such as health care and disability compensation, to veterans in the United States and abroad, including veterans who have been removed from the United States. VA pays monthly disability compensation to veterans for disabilities caused or aggravated by military service, known as service- connected disabilities. Veterans with service-connected disabilities may also be eligible for other VA benefits and services, such as job training. VA staff in regional offices process disability compensation claims. After a veteran submits a disability claim to VA, a VA Veterans Service Representative reviews the claim and assists the veteran with gathering relevant evidence, such as military service records, medical examinations, and treatment records from VA medical facilities and private providers. If necessary to provide support to substantiate the claim, VA will provide a medical examination, known as a Compensation and Pension (C&P) exam, to obtain evidence of the veteran’s disabilities and their connection to military service. Within the United States, medical providers who work for the Veterans Health Administration often conduct these exams. VA also contracts with private firms to perform these exams. Outside the United States, VA contracts with private firms to perform exams in 33 countries. In countries where VA contractors do not perform exams, VA coordinates with State staff at embassies and consulates to schedule exams with private providers. Once VA receives the claim evidence, a Rating Veterans Service Representative evaluates the claim and determines whether the veteran is eligible for benefits, and if so, assigns a percentage rating. After a rating is assigned, VA provides VSO staff assisting a veteran with a claim up to 48 hours to review the claim decision prior to finalizing the decision. A Veterans Service Representative then determines the amount of the award, if any, and drafts a decision notice. A senior Veterans Service Representative then reviews and authorizes the award for release to the veteran. See figure 1 for details on the 5 phases of VA’s disability compensation claims process. From fiscal years 2013 through 2018, VA received over 8.9 million disability compensation claims from over 3.9 million veterans and awarded over $20.2 billion in benefits, according to VA data. ICE has developed policies that govern the handling of cases involving potentially removable veterans. When HSI agents and ERO officers learn that they have encountered a veteran, these policies require they conduct additional assessments, create additional documentation, and obtain management approval in order to proceed with the case. Specifically, in June 2004, ICE’s Acting Director of Investigations issued a memo giving the HSI Special Agent in Charge (SAC) in each field office the authority to approve issuance of a NTA in cases involving current service members or veterans. Similarly, in September 2004, ICE’s Acting Director of Detention and Removal Operations issued a memo giving the ERO Field Office Director (FOD) in each field office the authority to approve issuance of a NTA in cases involving current service members or veterans. In order to issue a NTA to a veteran, the SAC and FOD must consider, at a minimum, the veteran’s overall criminal history, evidence of rehabilitation, family and financial ties to the United States, employment history, health, and community service. The SAC and FOD must also consider factors related to the veteran’s military service, such as duty status (active or reserve), assignment to a war zone, number of years in service, and decorations awarded. To authorize issuance of the NTA, the SAC and FOD are to complete a memo to include in the veteran’s alien file and update ICE’s EARM database with a brief overview of the facts considered. Additionally, in November 2015, ICE’s Director issued a policy establishing ICE’s procedures for investigating the potential U.S. citizenship of individuals encountered by ICE. The policy states that prior military service is one of several indicators that an individual could be a U.S. citizen. Therefore, before issuing a NTA to a veteran or anyone with an indicator of potential U.S. citizenship, the ICE component that first encounters the individual (either HSI or ERO) is to conduct a factual examination, legal analysis, and a check of all available DHS systems, such as USCIS’s Person-Centric Query Service, to assess whether the individual is a U.S. citizen. ERO or HSI (whichever conducted the factual examination) and OPLA’s Office of Chief Counsel must jointly prepare a memorandum that assesses the individual’s citizenship status and recommends a course of action, then submit that memorandum for review and approval by ICE headquarters. The policy also requires that a copy of the memorandum be placed in the individual’s alien file. Our analysis of removed veterans’ alien files found that ICE does not consistently follow these policies. Specifically, ICE policies require agents and officers to document the decision to issue a NTA to a veteran, but do not require agents and officers to identify and document veteran status when interviewing potentially removable individuals. Our analysis found that ICE did not satisfy the 2004 requirement for FOD approval in 18 of 87 (21 percent) cases that OPLA’s check box indicated involved veterans who were placed into removal proceedings and ERO data indicated had been removed from fiscal years 2013 through 2018. Our analysis also found that ICE did not meet the requirements of the 2015 policy requiring elevation to headquarters in 26 of the 37 cases (70 percent) of the cases for which the policy applied. Further, in December 2018 HSI officials told us that HSI has not been adhering to either the 2004 or the 2015 policies because they were unaware of the policies prior to our review. HSI officials stated that they do not distinguish between veterans and nonveterans when conducting administrative or criminal investigations or when deciding whether to issue a NTA. ERO officials stated that the absence of documentation in the alien file does not necessarily indicate that officers did not adhere to the policies; however, as noted above, the policies specifically require ICE to add documentation to the alien file. Because ICE did not consistently follow these policies, some veterans who were removed may not have received the level of review and approval that ICE has determined is appropriate for cases involving veterans. Taking action to ensure consistent implementation of its policies for handling cases of potentially removable veterans, such as issuing guidance or providing training, would help ICE better ensure that potentially removable veterans receive appropriate levels of review and consideration prior to the initiation of removal proceedings. ICE has not developed a policy to identify and document all military veterans it encounters. According to ERO officials, when ERO officers encounter an individual, they interview that individual and complete the Form I-213, “Record of Deportable/Inadmissible Alien,” which documents information on, among other things, the individual’s country of citizenship and most recent employer. Officials stated that ERO officers would generally learn about the individual’s veteran status during that interview. However, ICE does not have a policy requiring agents and officers to specifically ask about and document veteran status. According to ERO officials, ERO does not need such a policy because ERO’s training for new officers, the Basic Immigration Enforcement Training Program, instructs officers to ask about veteran status when interviewing potentially removable aliens. The Basic Immigration Enforcement Training Program includes one lesson plan and one practice exercise stating that the I-213 “Record of Deportable/Inadmissible Alien” should include information on military service, as applicable. The lesson plan also includes a list of mandatory questions that ERO officers must ask in every encounter with an alien; however, that list of mandatory questions does not include any questions about military service. Further, the I-213 “Record of Deportable/Inadmissible Alien” does not have a specific field to indicate veteran status, and ERO’s cover sheet that supervisors use to review the legal sufficiency of NTAs does not contain information about veteran status. For cases processed by HSI, HSI officials stated that agents would generally learn about the individual’s veteran status through the initial interview or through background checks or other information obtained in the course of an HSI investigation. However, during the course of our review, HSI officials stated that there was no policy requiring agents to ask about or document veteran status because, as discussed above, HSI does not handle veterans’ cases differently from other cases. Without mechanisms in place to identify and document veterans, ICE is not positioned to determine whether or not individuals it encounters are potentially veterans and for which individuals the 2004 and 2015 policies discussed above for handling cases of potentially removable veterans should be applied. Standards for Internal Control in the Federal Government state that management should design control activities—that is, the policies, procedures, techniques, and mechanisms that enforce management’s directives to achieve the entity’s objectives. ICE officials told us that the 2004 and 2015 policies are intended to provide guidance and direction to ICE agents and officers for handling cases of potentially removable veterans. ICE officials believe that these policies could be updated with additional guidance to agents and officers to ask about and document veteran status during interviews of potentially removable individuals. Without developing and implementing a new policy or revising its 2004 and 2015 policies to require agents and officers to ask about and document veteran status, ICE has no way of knowing whether it has identified all of the veterans it has encountered and, therefore, does not have reasonable assurance that it is consistently implementing its policies and procedures for handling veterans’ cases. Because ICE has not developed a policy to identify and document all military veterans it encounters, ICE does not maintain complete electronic data on veterans who have been placed in removal proceedings or removed. In the instances in which ICE agents and officers learn that they have encountered a veteran, none of the three ICE components who encounter veterans—ERO, OPLA, and HSI—maintain complete electronic data on the veterans they identify. ERO does not have a specific field for tracking veterans in its database, EARM. According to ERO officials, ERO officers can note veteran status on the Form I-213, “Record of Deportable/Inadmissible Alien,” but ERO does not have the ability to electronically search those notes to identify all of the veterans it has encountered. ERO officials stated that they do not maintain data on veteran status because they do not specifically target veterans for enforcement operations. OPLA has a check box tracking veteran status in its database, PLAnet, but the field is not mandatory. PLAnet also includes a case notes section, where an OPLA attorney may choose to document veteran status information. OPLA officials stated that the reliability of the veteran status box and case notes depends on the diligence of the attorney inputting the case information into PLAnet. HSI officials stated that they do not track veteran status at all because, as discussed above, veteran status does not affect their handling of cases. Our analysis of removed veterans’ alien files identified limitations with the only electronic data on veteran status ICE maintains—OPLA’s check box in the PLAnet database. Specifically, though OPLA’s check box indicated that all 87 of the aliens whose files we reviewed were veterans, we found that 8 of the 87 individuals (9 percent) did not serve in the U.S. Armed Forces, according to the information in their alien files. After reviewing these cases, OPLA officials stated that the individuals were incorrectly designated as veterans due to human error. OPLA officials stated that OPLA does not require attorneys to systematically track veteran status information in PLAnet because the database is not intended to be a data repository, but rather serves as a case management system for OPLA attorneys. OPLA officials stated that the official record of the alien’s case is the paper alien file. Because ICE does not maintain complete electronic data on potentially removable veterans it encounters, ICE does not know exactly how many veterans have been placed in removal proceedings or removed, or if their cases have been handled according to ICE’s policies. Standards for Internal Control in the Federal Government state that management uses quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks. Quality information is appropriate, current, complete, accurate, accessible, and provided on a timely basis. While tracking veteran status in the paper alien file may allow ICE to review whether a specific individual is a veteran, it does not provide the type of complete and accessible electronic data that would allow the agency to systematically evaluate its performance in adhering to its policies. Maintaining complete electronic data on veterans it encounters would assist ICE in determining the extent to which the agency has adhered to its policies for handling cases involving potentially removable veterans. For example, ICE could obtain quality information through a mandatory field, such as a check box to track veteran status. Based on the limited information available in OPLA’s PLAnet database, approximately 250 veterans were placed in removal proceedings or removed from the United States from fiscal years 2013 through 2018. As noted above, ICE does not maintain complete electronic data on veterans it encounters. While OPLA’s PLAnet includes some data on veterans who have been placed in removal proceedings, because the entry of veteran status data in PLAnet is not mandatory, there could be additional veterans who were placed in removal proceedings or removed during the timeframe of our review who were not noted in PLAnet or included in our analysis, as discussed below. We reviewed the data that were included in PLAnet on veterans who were placed in removal proceedings from fiscal years 2013 through 2018 and identified approximately 250 military veterans. This includes those individuals for whom the check box indicating veteran status was checked in PLAnet but, as noted above, does not represent complete data on all possible veterans placed in removal proceedings during the time period we reviewed. Among the approximately 250 individuals who were noted in PLAnet as veterans in removal proceedings, the most common countries of nationality were Mexico (about 40), Jamaica (about 30), El Salvador (about 10), Trinidad and Tobago (about 10), Germany (about 10), and Guatemala (about 10). At the end of fiscal year 2018, about 115 had been ordered removed, about 25 had been granted relief or protection from removal by an immigration judge, and about 5 had their cases administratively closed. The remainder of the cases were still open as of November 2018. From fiscal year 2013 through 2018, ERO had removed 92 of the approximately 250 military veterans from the United States, of which 90 were foreign nationals with one or more criminal convictions, according to ERO data. Nine of the removed veterans had service-connected disabilities recognized by VA, including four removed veterans who had service-connected post-traumatic stress disorder. Based on our review of the alien files of 87 of the individuals that OPLA’s check box indicated were veterans and ERO indicated had been removed, we identified the following characteristics: 26 veterans (30 percent) received an honorable discharge; 26 (30 percent) received a general discharge under honorable conditions; 13 (15 percent) received an other than honorable discharge; 8 (9 percent) received an uncharacterized discharge; 3 (3 percent) received a bad conduct discharge; 2 (2 percent) received a dishonorable discharge; 8 (9 percent) had no evidence of military service in their alien file; and 1 (1 percent) did not have a discharge characterization listed in the alien file. 74 veterans (85 percent) were LPRs, 6 (7 percent) were citizens of the Marshall Islands, the Federated States of Micronesia, and Palau who enlisted under the Compact of Free Association, 6 (7 percent) did not have evidence of lawful status, and 1 (1 percent) was a recipient of Deferred Action for Childhood Arrivals. 26 veterans (30 percent) had previously applied for naturalization with USCIS; 3 of whom submitted multiple applications. Seventeen of those naturalization applications were denied by USCIS, 9 were administratively closed, and 2 were withdrawn. 68 veterans (78 percent) were ordered removed because of at least one aggravated felony conviction, while the remaining 19 (22 percent) were ordered removed for non-aggravated felony convictions. Of the convictions ICE cited on the 87 veterans’ NTAs: 32 veterans had drug-related convictions; 20 had convictions related to sexual abuse, of which 18 involved minors; 21 had convictions related to homicide, assault, or attempted homicides or assaults; 16 had theft-related convictions; and 9 had convictions related to firearms, explosives, or explosive material. USCIS and DOD have policies facilitating the naturalization of noncitizen service members and veterans, and both agencies provide informational resources to noncitizen service members seeking naturalization. USCIS facilitates the application and naturalization process for current and recently discharged members of the U.S. Armed Forces through a dedicated Military Naturalization Unit, which processes military naturalization applications and assists field officers with administrative naturalization tasks overseas, among other things. USCIS interviews and naturalizes active-duty service members abroad at certain U.S. embassies, consulates, and military installations. To provide information to noncitizen service members and veterans, USCIS maintains a toll-free “Military Help Line” and an e-mail box exclusively for members of the military and their families and publishes an “Immigration 101” presentation for relevant stakeholders, including DOD personnel on military bases. In addition, USCIS provides DOD with a checklist of required documents for military naturalization applications and communication guidelines for naturalization application inquiries, according to USCIS officials. DOD determines whether a service member meets the qualifying service requirement for naturalization by certifying whether the service member has served “honorably,” and if he or she has separated from service, whether their separation was under honorable conditions. Additionally, according to DOD officials, every military installation generally designates a naturalization advisor within its Legal Services Office. The advisor, among other things, assists service members with preparation of their naturalization application packets and serves as an intermediary with USCIS staff. For example, at many Army installations, the Army Community Services Office typically performs this function. Although USCIS approved military naturalization applications at a fairly consistent rate from fiscal years 2013 through 2018, the number of applications received declined sharply from fiscal years 2017 to 2018, resulting in a decrease in the number of service members approved for naturalization in fiscal year 2018. From fiscal years 2013 through 2018, USCIS received 54,617 military naturalization applications; USCIS approved 46,835 (86 percent) and denied 3,410 (6 percent). Applicants’ most common countries of nationality were the Philippines (6,267 or 11 percent), Mexico (5,760 or 11 percent), Jamaica (3,510 or 6 percent), China (3,213 or 6 percent), and the Republic of Korea (2,982 or 5 percent). While the number of military naturalization applications was relatively stable between fiscal years 2013 and 2017, applications declined by 72 percent from fiscal year 2017 to fiscal year 2018, from 11,812 in fiscal year 2017 to 3,291 in fiscal year 2018, as shown in figure 2. As a result of this decline in applications, the number of service members approved for naturalization also declined, from 7,303 in fiscal year 2017 to 4,309 in fiscal year 2018. USCIS and DOD officials attributed the decline in military naturalization applications to several DOD policy changes. First, DOD suspended the MAVNI program in September 2016, which reduced the number of noncitizens joining the military. According to DOD officials, due to counterintelligence concerns, DOD suspended the program at the end of fiscal year 2016 and decided not to renew the program in fiscal year 2017. Second, in October 2017, DOD issued policies expanding background check requirements for LPR and MAVNI recruits. The policies specify that LPRs must complete a background check and receive a favorable military service suitability determination prior to entering any component of the U.S. Armed Forces. According to DOD officials, due to backlogs in the background check process, these new recruits were delayed in beginning their service, and officials stated that it may take DOD up to a year to complete enhanced requirements for certain recruits. DOD officials stated that they believe background check backlogs will decrease by the end of fiscal year 2019 and, as a result, the number of noncitizen service members eligible to apply for naturalization will increase. Third, in October 2017, DOD increased the amount of time noncitizens must serve before DOD will certify their honorable service for naturalization purposes. Under the new policy, noncitizens must complete security screening, basic military training, and serve 180 days for a characterization of service determination. Previously, DOD granted that determination in as little as a few days of service. USCIS made several changes to its military naturalization processes in response to or in tandem with DOD’s policy changes. First, in July 2017, USCIS determined that the completion of DOD background checks was relevant to MAVNI recruits’ eligibility for naturalization. USCIS thus began requiring currently-serving MAVNI recruits seeking military naturalization to complete all required DOD background checks before USCIS interviewed them, approved their applications, or administered the Oath of Allegiance to naturalize them. Second, in January 2018, USCIS ended its initiative to naturalize new enlistees at basic training sites. This initiative, known as the “Naturalization at Basic Training Initiative”, began in August 2009 as an effort to conduct outreach to new enlistees at the Army’s five basic training sites and provide noncitizen enlistees an opportunity to naturalize prior to completion of basic training. Because of DOD’s October 2017 policy change increasing the amount of time noncitizens must serve before they are eligible for a characterization of service determination, noncitizen service members no longer meet the requirements for naturalization while they are completing basic training. As a result, USCIS closed naturalization offices in Fort Sill, Fort Benning, and Fort Jackson. USCIS’s processing time for military naturalizations also increased, from an average of 5.4 months in fiscal year 2017 to 12.5 months in fiscal year 2018, according to USCIS data. USCIS officials attributed this increase to the backlog in DOD background checks for MAVNI recruits, as well as an increased volume of naturalization applications from non-military applicants. Citizenship status, including immigration enforcement or removal history, does not affect a veteran’s eligibility for VA benefits and services, according to VA officials. As a result, veterans who have been removed by ICE are entitled to the same VA benefits and services as any other veteran living abroad. Although being removed for violation of immigration law does not in and of itself affect eligibility for VA benefits and services, living abroad affects eligibility for certain benefits and services, as shown in table 1. These differences pertain to all veterans living abroad, including both veterans who have been removed by ICE and veterans who choose to reside abroad. Removed veterans may face additional obstacles in receiving certain benefits for which they are otherwise eligible because they may be barred from traveling to the United States. For example, a removed veteran may not be able to attend a hearing to appeal a VA disability rating decision because VA conducts those hearings exclusively in the United States. Additionally, a removed veteran may not be able to obtain certain Vocational Rehabilitation and Employment services if the veteran is unable to travel to the United States for medical referrals and case management. Veterans living abroad, including removed veterans, may experience challenges accessing certain benefits and services, including slower disability claim processing and Foreign Medical Program (FMP) claim reimbursement, difficulties related to the scheduling and quality of C&P exams, and difficulties communicating with VA. According to VA officials, VA’s processing time for disability compensation claims for veterans living abroad (foreign claims) has improved since fiscal year 2013. For example, in fiscal year 2013, VA processed foreign claims in an average of 521 days and in fiscal year 2018, VA’s processing time for foreign claims decreased to an average of 131 days. However, as of September 2018, VA was not meeting its timeliness goal of 125 days for processing foreign claims and VA took an average of 29 days longer to process foreign claims than domestic claims. VA officials attributed the longer processing times for foreign claims to unreliable foreign mail systems and issues with retrieving and translating foreign records, among other things. From fiscal years 2013 through 2018, VA received disability compensation claims from 26,858 veterans living abroad and awarded over $85 million in benefits, according to VA data. According to VA officials, VA’s processing time for health care claims reimbursements to veterans or their medical providers for treatment of service-connected conditions through FMP has also improved. For example, in October 2018, FMP was processing 53.8 percent of claims in 40 days compared to 70 percent of claims in 40 days in March 2019. However, as of March 2019, VA was not meeting its timeliness goal to process 90 percent of claims reimbursements through FMP in 40 days. FMP officials attributed these delays to the loss of four staff positions in April 2017, as well as FMP assuming responsibility for claims from the Philippines in October 2017. To improve FMP’s processing timeliness, FMP officials stated that VA funded three new full-time equivalent positions for fiscal year 2019. From fiscal years 2013 through 2018, VA reported receiving 373,916 claims reimbursements from veterans and providers living abroad and awarding over $169 million in claims reimbursements. According to both VA and VSO officials, veterans living abroad, including removed veterans, face challenges related to the scheduling and quality of C&P exams. As previously noted, veterans living abroad do not receive C&P exams from VA medical providers, but may receive exams from either a VA contractor or, in countries where VA does not have contractors, from a private provider scheduled by the U.S. embassy or consulate. From fiscal years 2013 through 2018, VA completed over 27,000 exams abroad through contractors and 6,800 exams through U.S. embassies and consulates, according to VA data. For contract exams, as of March 2019, VA had contractors in 33 countries and U.S. territories. This included Mexico, Germany, Belize, Canada, the Dominican Republic, the Federated States of Micronesia, the United Kingdom, the Philippines, Thailand, Costa Rica, Korea, and Poland, which were among the most common countries of nationality for removed veterans in our analysis. VA officials stated that contract C&P exam locations are determined by historical and pending claims data. Moreover, VA contractors abroad are generally located near military installations or areas in which VA determined there is a large veteran population. For embassy-scheduled exams, both VA and VSO officials told us that the effectiveness of coordination between VA and the embassies varies by country. For example, VA staff told us that they have been unable to schedule exams through embassies in Iraq or Afghanistan. State officials told us that processes for scheduling C&P exams and communicating with VA vary depending on the location, activity, and size of the embassy or consulate. State officials also told us that access to specialized providers to conduct exams, including mental health or audio exams, depends on the location of the embassy or consulate. In addition, both VA and VSO officials told us that veterans who receive embassy-scheduled exams from private providers abroad may receive lower-quality exams than veterans who live in the United States. For example, providers abroad may misinterpret VA exam requirements due to language barriers or unfamiliarity with U.S. medical terminology. These providers also do not have access to veterans’ service records, and therefore cannot assess whether a particular condition is service- connected. For these reasons, VA officials told us that VA staff submit C&P exams completed by private providers abroad to the VA Medical Center in Pittsburgh, Pennsylvania for an additional medical opinion. According to VA officials, VA is improving the scheduling and quality of C&P exams by expanding the number of countries where veterans may receive exams from VA contractors. According to VA and VSO officials, veterans living abroad experience challenges communicating with the VA. For example, staff from all four VSOs we interviewed stated that unreliable foreign mail systems and differences in time zones make it challenging for veterans to communicate with the VA, particularly because VA uses paper mail to communicate with veterans living abroad. In addition, VA and VSO officials also told us that veterans living abroad may face challenges applying for and managing their benefits through an online portal maintained by VA and DOD, eBenefits. VA requires veterans to register for a “premium account” in order to access all of the functions of eBenefits, such as applying for benefits online and checking the status of a claim, among other things. To be eligible for a “premium account,” veterans must first verify their identity with DOD. If the veteran provides valid government identification (e.g. driver’s license) and documentation of a financial account (e.g. checking account), DOD may be able to verify the veteran’s identity through an online registration process and VA may be able to verify the veteran’s identity by telephone. If a veteran is unable to verify their identity in this manner, the veteran must verify their identity in-person at a VA regional office in the United States. Therefore, removed veterans who cannot travel to the United States would not be able to obtain a “premium account” if they had not previously registered prior to their removal. VA officials stated that these processes are intended to ensure compliance with National Institute of Standards and Technology guidance for online credentialing. Throughout U.S. history, noncitizens have contributed to the United States through service in the Armed Forces. Through its policies, ICE has established that these noncitizen veterans warrant special consideration in the event that they become subject to immigration enforcement and removal from the United States. However, because ICE did not consistently adhere to these policies, some veterans who were removed may not have received the level of review and approval that ICE has determined is appropriate for cases involving veterans. Moreover, without developing and implementing a new policy or revising its 2004 and 2015 policies to require ICE agents and officers to ask about and document veteran status while interviewing potentially removable individuals, ICE has no way of knowing whether it has identified all of the veterans it has encountered and, therefore, does not have reasonable assurance that it is consistently implementing its policies and procedures for handling veterans’ cases. Further, maintaining complete electronic data on veterans it encounters would also allow ICE to better assess whether ICE has adhered to its policies for handling cases involving potentially removable veterans. We are making the following three recommendations to ICE: The Director of ICE should take action to ensure consistent implementation of ICE’s policies for handling cases of potentially removable veterans. (Recommendation 1) The Director of ICE should develop and implement a policy or revise its current polices to ensure that ICE agents and officers identify and document veteran status when interviewing potentially removable individuals. (Recommendation 2) The Director of ICE should collect and maintain complete and electronic data on veterans in removal proceedings or who have been removed. (Recommendation 3) We provided a copy of this report to DHS, VA, DOD, and State for review and comment. DHS provided written comments, which are reproduced in full in appendix I and discussed below. DHS, VA, and DOD also provided technical comments, which we incorporated as appropriate. State indicated that it did not have any comments on the draft report. In its comments, DHS concurred with our three recommendations and described actions planned to address them. With respect to our first recommendation that ICE should ensure consistent implementation of its policies for handling potentially removable veterans, DHS concurred stating that ICE plans, among other things, to update its guidance and training materials to include information about military service. With respect to our second recommendation that ICE should develop and implement a policy or revise its current policies to ensure agents and officers identify and document veteran status when interviewing potentially removable individuals, DHS concurred, stating that ICE plans to review and clarify existing guidance on the issuance of NTAs to veterans. DHS also concurred with our third recommendation that ICE collect and maintain complete and electronic data on veterans in removal proceedings or who have been removed. Specifically, DHS stated that ICE plans to add data elements for veteran status to its existing systems. The actions described above, if implemented effectively, should address the intent of our recommendations. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, the Secretary of Veterans Affairs, the Acting Secretary of Defense, the Secretary of State, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at 202-512-8777 or gamblerr@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of our report. GAO staff who made key contributions to this report are listed in appendix II. Appendix II: GAO Contacts and Staff Acknowledgements Error! No text of specified style in document. In addition to the contact named above, Meg Ullengren (Assistant Director), Ashley Davis, Eric Hauswirth, Khaki LaRiviere, Sasan J. “Jon” Najmi, Claire Peachey, Mike Silver, Natalie Swabb, and James Whitcomb made key contributions to this report.", "summary": "Throughout U.S. history, noncitizens have served in the U.S. Armed Forces. Although the Immigration and Nationality Act allows noncitizen service members to acquire citizenship, some veterans may not apply or may not satisfy all eligibility criteria. If the Department of Homeland Security (DHS) determines that a noncitizen veteran is potentially removable, the veteran may be subject to administrative immigration enforcement and removal. ICE, among other things, is responsible for identifying and removing aliens who violate U.S. immigration law. GAO was asked to review issues related to the removal of noncitizen veterans. This report examines (1) the extent to which ICE has developed and implemented policies for handling and tracking cases of potentially removable veterans; (2) how federal agencies facilitate the naturalization of noncitizen service members and veterans, and what is known about the number who have applied for naturalization; and (3) how removal affects veterans' eligibility for and access to VA benefits and services. GAO reviewed documentation, met with agency officials, analyzed available data on veterans placed in removal proceedings, and conducted a review of removed veterans' alien files. GAO also analyzed data on military naturalization applications. U.S. Immigration and Customs Enforcement (ICE) has developed policies for handling cases of noncitizen veterans who may be subject to removal from the United States, but does not consistently adhere to those policies, and does not consistently identify and track such veterans. When ICE agents and officers learn they have encountered a potentially removable veteran, ICE policies require them to take additional steps to proceed with the case. GAO found that ICE did not consistently follow its policies involving veterans who were placed in removal proceedings from fiscal years 2013 through 2018. Consistent implementation of its policies would help ICE better ensure that veterans receive appropriate levels of review before they are placed in removal proceedings. Additionally, ICE has not developed a policy to identify and document all military veterans it encounters during interviews, and in cases when agents and officers do learn they have encountered a veteran, ICE does not maintain complete electronic data. Therefore, ICE does not have reasonable assurance that it is consistently implementing its policies for handling veterans' cases. U.S. Citizenship and Immigration Services (USCIS) and the Department of Defense (DOD) have policies facilitating the naturalization of noncitizen service members and veterans, and provide informational resources to noncitizen service members seeking naturalization. The number of military naturalization applications received by USCIS declined sharply from fiscal years 2017 to 2018, resulting in a decreased number of applications approved in fiscal year 2018. USCIS and DOD officials attributed this decline to several DOD policy changes that reduced the number of noncitizens joining the military . Citizenship status, including removal history, does not affect a veteran's eligibility for Department of Veterans Affairs (VA) benefits and services. However, living abroad affects eligibility for certain VA benefits and services. Veterans living abroad may also experience challenges accessing certain benefits and services, such as slower disability claim processing. GAO recommends that ICE (1) ensure consistent implementation of its existing policies for handling veterans' cases; (2) develop a policy or revise its current policies to identify and document veterans; and (3) collect and maintain complete data on veterans in removal proceedings or who have been removed. DHS concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "The purposes of TRIA are to (1) protect consumers by addressing market disruptions and ensuring the continued widespread availability and affordability of commercial property/casualty insurance for terrorism risk; and (2) allow for a transitional period for the private markets to stabilize, resume pricing of such insurance, and build capacity to absorb any future losses, while preserving state insurance regulation and consumer protections. TRIA only applies to certain commercial property/casualty lines of insurance (we refer to them as TRIA-eligible lines) and excludes lines such as health and life insurance. While the law requires insurers to make terrorism risk coverage available to commercial policyholders, commercial policyholders are not required to buy it. Additionally, the law requires an insurer to make coverage for terrorism losses available that does not differ materially from the terms, amounts, and other coverage limitations applicable to losses arising from events other than acts of terrorism. For example, an insurer offering $100 million in commercial property coverage must offer $100 million in coverage that is not materially different for property damage from a certified terrorist event. Insurers may charge a separate premium to cover their terrorism risk, although some include the coverage in their base rates for all-risk policies. The majority of terrorism risk insurance is purchased as part of these embedded policies. The remainder is purchased as stand-alone coverage. Neither insurers nor the federal government charge for the federal coverage of terrorism risk under TRIA, but the government must recoup at least some of its losses following a terrorist event. For eligible commercial lines of property/casualty insurance, TRIA covers insured losses resulting from an act of terrorism, which is defined, in part, as a “violent act or an act that is dangerous” to human life, property, or infrastructure. TRIA is silent about losses from attacks with nuclear, biological, chemical, or radiological (NBCR) weapons. Although TRIA and its reauthorizations do not specifically include cyber risk insurance as a TRIA-eligible line, Treasury issued guidance about such coverage in 2016. The guidance stated that TRIA provisions apply to cyber risk insurance written under an embedded or stand-alone policy in TRIA- eligible lines. A cyberterrorism event could cause minor-to-severe business disruption and physical damage to property. In this report, we include losses resulting from cyberterrorism events with conventional terrorism events. Any catastrophic terrorist event presents both explicit and implicit fiscal exposure for the federal government. TRIA alleviates some of the implicit exposure through loss sharing. Under TRIA, the federal government is legally required to make payments (reimbursements to insurers): this represents an explicit fiscal exposure. Even without a loss-sharing program, the federal government also faces implicit fiscal exposure through a potential expectation to provide policyholders or insurers assistance to address long-term effects after a terrorist event. We have defined implicit fiscal exposures as situations that create expectations for future federal spending based on current policy, past practices, or other factors. Under TRIA, the government and insurers share losses in the event of a certified act of terrorism with insured losses above the program trigger of $200 million and below the program cap of $100 billion. According to the statute, Treasury cannot certify an event as an act of terrorism under TRIA if the aggregate property/casualty “insurance losses” resulting from the event are less than $5 million. Additionally, TRIA is not triggered unless the aggregate property/casualty “insured losses” resulting from one or more certified acts in a particular calendar year reach $200 million. Annual coverage for losses is limited (capped) so that neither the private insurers nor the federal government are responsible for paying aggregate insured losses in excess of $100 billion. Specifically: “Insured losses” are defined in statute and regulation as any losses resulting from an act of terrorism (including an act of war in the case of worker’s compensation) generally occurring in the United States that are covered by primary or excess property/casualty insurance issued by an insurer. TRIA refers to insured losses in defining the program trigger and program cap. “Insurance losses” are not defined in statute or regulation, but TRIA refers to insurance losses in defining the event certification amount. TRIA’s loss-sharing structure requires that insurers pay claims on covered terrorism losses and that Treasury reimburse individual insurers for losses that exceed a specified amount. According to the coshare provision, Treasury reimburses the insurer for a certain percentage (80 percent) of its losses above the insurer deductible, and the insurer is responsible for the remaining portion (20 percent). The policyholder also may retain losses from a terrorist event in the form of an insurance deductible or self-insurance retention. According to Treasury, losses retained by the policyholder are not considered to be “insured losses” under TRIA and do not count toward losses included in the $200 million program trigger or $100 billion program cap. However, this retention may be counted toward the $5 million event certification threshold because it is calculated based on “insurance losses.” TRIA provides for two types of recoupment—mandatory and discretionary—of the federal share of losses after a terrorism event. Figure 1 shows the claim and recoupment processes after a terrorism event resulting in losses covered by insurers and the government. Mandatory recoupment. TRIA requires recoupment of at least a portion of the federal share of losses if the aggregate sum of all insurers’ deductibles and coshares are below an amount prescribed by TRIA—known as the industry aggregate retention amount. Under mandatory recoupment, the insurers must impose and remit to Treasury a premium surcharge on all policies in TRIA-eligible lines until total industry payments reach 140 percent of any mandatory recoupment amount. TRIA specifies the collection time frame (from 1 year and 9 months to about 6.5 years, based on the date of the event). Treasury considers the collection time frame when establishing the amount of the mandatory recoupment surcharge. Discretionary recoupment. TRIA permits discretionary recoupment when the federal share of losses exceeds the mandatory recoupment amount. Under the discretionary recoupment provision, Treasury may consider a number of factors, such as economic conditions in the commercial marketplace, in determining the amount to recoup. To help in its decision-making, Treasury may issue a data call for insurer deductible and insured loss information. Treasury also sets the surcharge for discretionary recoupment, but the increase to TRIA- eligible premiums must not exceed 3 percent per calendar year. TRIA does not specify a collection time frame for discretionary recoupment. Since the 2015 reauthorization, insurers have been required to submit information to Treasury about their coverage of terrorism risk, including the lines of insurance with exposure to such risk, the premiums earned on such coverage, and the participation rate for such coverage. Treasury’s 2017–2019 data calls included loss scenarios in which insurers estimate and report expected losses given a defined terrorist attack at a specified location, date, and time. Treasury’s defined loss scenarios were located in New York City in 2016, Chicago in 2017, and San Francisco in 2018. Treasury’s reporting requirements for insurers vary, based on the following categories: Small insurers. Insurance companies that had both policyholder surplus and prior year TRIA-eligible direct-earned premium of less than five times the program trigger. For example, TRIA’s program trigger in 2020 is $200 million. Thus, Treasury would categorize insurers with less than $1 billion in 2019 direct-earned premiums and less than $1 billion in policyholder surplus as small. Treasury does not require small insurers to report on a number of items, including the loss scenarios. Nonsmall insurers. Insurance companies with either policyholder surplus or prior year TRIA-eligible direct-earned premium greater than the small insurer thresholds. Captive insurers. Special-purpose insurance companies set up by commercial businesses to self-insure risks arising from the owners’ business activities. Alien surplus lines insurers. Insurance companies headquartered in a foreign country that have been qualified to do business in the United States through an NAIC-administered process, which assesses the financial stability and trustworthiness of the insurer. Each reauthorization of TRIA through 2015 reduced the magnitude of the government’s explicit fiscal exposure. Since 2003, changes to TRIA provisions have increased insurers’ share of losses and thus decreased explicit federal fiscal exposure in the event of certified acts of terrorism (see fig. 2). For example, the program trigger rose over time, from $5 million in 2003 to $200 million in 2020. These changes reduced explicit fiscal exposure because they increased the amount of insured losses required before the government would share in the losses. The insurer deductible increased from 7 percent in 2003 to 20 percent for 2020, also reducing the federal share of payments. The 2015 reauthorization required incremental reductions in the federal share of losses over 5 years. The 2019 reauthorization extended the program until 2027, but did not make any changes to the program parameters. See appendix II for more details of the changes in the reauthorizations. Currently, following a certified act of terrorism Treasury pays insurer claims for 80 percent of insurers’ losses above their individual deductibles once losses in a calendar year exceed the program trigger of $200 million (see fig. 3). For example, based on the scenario from Treasury’s 2019 data call, the federal government could have an explicit exposure of about $4.4 billion in reimbursements to insurers. Specifically, insurers estimated that a hypothetical 2018 terrorist event in San Francisco could generate $39.7 billion dollars in overall losses, of which insurers could pay about $17 billion in claims to policyholders and the government could pay about $4.4 billion in reimbursements to insurers after a policyholder retention amount. According to our analysis of Treasury data on insurer direct-earned premiums, federal losses following a terrorist event under the loss-sharing provision in effect in 2020 would be smaller than they would have been for a similar event under the loss-sharing provision in effect in 2015, across all event sizes and subsets of insurers. In addition, more of the federal losses would be recovered through mandatory recoupment (see fig. 4). As we found in 2017, the amount of federal losses depends on event size and how many and which insurers were affected. Additionally, the government share depends on the aggregate TRIA- eligible direct-earned premium of the insurers with losses. Specifically, the federal share of losses is smaller when losses are shared among insurers with larger aggregate premium bases. As the share of losses for which insurers are responsible has increased under TRIA, the ability of insurers to absorb the extra exposure also has increased. Insurers use risk-mitigation strategies to reduce or offset their exposures. These can include purchasing reinsurance—insurance for insurers—to cover their deductibles or coshare payments, or diversifying their portfolios (for instance, reducing concentrations of risk in certain locations or lines of insurance). Insurers told us that they considered the potential effect of program changes in each reauthorization and modified risk-mitigation strategies, as needed. Furthermore, other industry stakeholders, including a broker and an industry association, told us that because program changes have been gradual and expected, insurance companies have been able to adjust their coverage accordingly. Available evidence indicates that TRIA has been largely effective in meeting its statutory objectives of stabilizing the terrorism risk insurance market. First, terrorism risk insurance is available in the market for a relatively low cost and is purchased by the majority of commercial policyholders in the United States, according to industry reports. Second, private reinsurance capacity for terrorism risk insurance increased since the creation of the program, according to Treasury. Third, our analysis of Treasury data suggests there is market stability. Insurers in all of Treasury’s reporting categories largely remained in the market. Furthermore, the market share and number of insurers in the reporting categories generally remained stable. For example, using data on direct- earned premiums of insurers from 2016 to 2018, nonsmall insurers (92 insurers in 2018) held about 80 percent of the TRIA-eligible insurance market. Small insurers (186 insurers in 2018) held about 10 percent. Captive and alien surplus lines insurers (598 and 98 insurers, respectively, in 2018) each held 4 or 5 percent (see fig. 5). Our interviews indicate that some insurers’ interpretation of whether policyholder retention amounts count toward the program threshold, trigger, and cap may differ from Treasury’s. Some large policyholders may retain large amounts of loss in the form of a deductible or self- insurance retention following a terrorist event. Treasury officials said policyholder retention amounts are not counted toward the program’s $200 million trigger or its $100 billion cap, but could be counted toward the $5 million threshold for event certification. They stated that the law utilizes “insured losses” when referring to the program trigger and cap, and “insurance losses” when referring to the certification threshold. If Treasury counted policyholder retention amounts toward the program trigger, the program would be triggered and capped with a smaller amount of overall losses. For example: To illustrate, we use a hypothetical terrorist event resulting in $290 million in overall losses, of which $100 million would be retained by policyholders. Using Treasury’s interpretation that excludes policyholder retention amounts, “insured losses” would be $190 million, which is below the program trigger of $200 million. As a result, the government would not be required to pay insurers a coshare. In contrast, if Treasury included policyholder retention amounts, “insured losses” would be $290 million, which exceeds the program trigger. In this case, the government would pay $112 million to insurers in coshares. In either case, losses would not reach the program cap of $100 billion. We asked 12 industry stakeholders about their understanding of how Treasury would use policyholder retention amounts to calculate the program threshold, trigger, and cap. The distinction between “insured losses” and “insurance losses” in Treasury’s explanation was clear to one insurer. However, some aspect of this distinction was unclear to six industry stakeholders, including an insurer association and three insurers, potentially resulting in uncertainty about how Treasury would calculate losses in the aftermath of a terrorist event. For example, representatives of an insurer association and two insurers told us they interpreted insured losses as including the policyholder retention amounts because insurers could be responsible for paying this amount. This is because some insurers pay the entire claim, including all or a portion of the policyholder retention, up front and then seek reimbursement from the policyholder. In addition, if a policyholder cannot pay its retention, the insurance company is responsible for it. Differences in interpretation could lead to disputes between insurers and Treasury following a terrorist event. We previously found that insurers are concerned about the long-term consequences of disputes related to terrorist events. One purpose of TRIA is to stabilize the market following a terrorist event. Furthermore, federal internal control standards state that management should externally communicate the necessary quality information so that external parties can help the entity achieve its objectives and address related risks. Treasury’s program regulations define “insured losses” and do not define “insurance losses.” Furthermore, the regulations do not explain how such losses are calculated and therefore how the policyholder retention amount does or does not count toward the program threshold, trigger, or cap, as applicable. Treasury uses different methods to communicate program information and clarify program details to stakeholders, such as program regulations and interpretive letters, but has not clarified this issue using these or other methods because officials believe the distinction is understood in the industry. By closing the information gap of how it would calculate losses for the program threshold, trigger, and cap, Treasury would create a common understanding of a critical feature of the program. Furthermore, Treasury may prevent uncertainty in the insurance market and potential litigation following a terrorist event that could delay insurance payments and economic recovery. TRIA explicitly limits federal exposure following a terrorist event, but the federal government could be expected to provide assistance beyond what is explicitly outlined in TRIA. Expectations for the government to provide assistance through its recoupment decisions and to policyholders and insurers, as described below, represent implicit fiscal exposures. Although the government may not act on these expectations, to the extent that it does, the implicit exposure would become an explicit exposure. In certain circumstances under mandatory and discretionary recoupment, such as potential effects on market stability, public expectation may lead the federal government to cancel recoupment or reduce the amount of funds recouped. Any portion of the federal coshare not recouped represents an implicit fiscal exposure. Some mandatory recoupment scenarios may or may not be perceived as burdensome to policyholders, prompting an expectation of federal assistance to ease the burden. Treasury determines mandatory recoupment surcharges based on the statutory deadlines for collecting mandatory recoupment. If a large terrorism act occurs in a year in which the statute requires the collection of mandatory recoupment in a short time frame, Treasury may need to set a high surcharge percentage on premiums for policies with TRIA-eligible lines. In this case, some policyholders may find it difficult to pay the surcharge, making collection of the mandatory recoupment amount burdensome. Large recoupment amounts or surcharges could prompt public expectation, and political will, for reducing or cancelling recoupment to alleviate this burden. Because mandatory recoupment time frames are based in statute, reducing or canceling this recoupment would require congressional action. One insurer told us that they are skeptical that Congress would allow Treasury to collect mandatory recoupment after a large event. Under current program provisions, the maximum mandatory recoupment amount will increase if the TRIA-eligible direct-earned premium increases. Industry stakeholders told us that, with this change, eventually all recoupment could be mandatory. While the amount to be recouped may increase, the recoupment time frame remains unchanged. Therefore, over time, surcharge amounts could increase, which may increase burden on policyholders and increase the expectation for Congress to cancel recoupment. If Congress were to cancel the collection of mandatory recoupment, the explicit fiscal exposure would include both the federal share of losses paid to insurers and the decreased corporate tax receipts from deductions for the recoupment charges policyholders may claim (such deductions otherwise were intended to be offset by the 140 percent recoupment). Changes in legislation following premium rate increases in the National Flood Insurance Program (NFIP) provide an example of Congress changing a law to ease policyholder burden. Congress enacted the Biggert-Waters Flood Insurance Reform Act of 2012, which was intended to strengthen the future financial solvency and administrative efficiency of NFIP by implementing provisions to reduce and eventually eliminate most subsidized premium rates. However, after public outcry claiming negative effects on home values, Congress enacted the Homeowner Flood Insurance Affordability Act of 2014, which repealed some of the premium rate increases in the 2012 act. Discretionary recoupment presents an implicit fiscal exposure because Treasury may decide not to collect the full discretionary recoupment amount. Under TRIA, Treasury decides whether and how much of the discretionary portion of the federal share of losses to recoup. Treasury may recoup some or all nonmandatory funds, or cancel discretionary recoupment. For any amount that Treasury chose not to collect, the fiscal exposure would be the dollar-for-dollar amount of the federal share of losses paid to insurers. As defined in statute, Treasury may consider several factors when determining whether to collect discretionary recoupment, in full or partially, or cancel recoupment. These factors include ultimate cost to taxpayers of no additional recoupment; the economic conditions of the commercial marketplace; the affordability of commercial insurance for small and medium-sized businesses; and other factors Treasury deems appropriate. According to agency officials, decisions regarding discretionary recoupment would be based on the parameters of the specific terrorism act, such as the size of the federal share of losses, location of the event, and length of the collection period. In our analysis of explicit exposure, we found that under some scenarios, the discretionary recoupment amount resulting from a terrorist event could exceed $50 billion. Depending on Treasury’s analysis of these factors, some or all of the discretionary recoupment amount may not be collected. For example, currently, much of the recoupment amount resulting from the most catastrophic losses would be considered discretionary under TRIA’s provisions. Because TRIA mandates an annual 3 percent cap on the increase of premium rates in TRIA-eligible lines for discretionary recoupment, in extreme cases Treasury might need to collect a premium surcharge for a protracted period of time to fully recoup the discretionary portion of losses. The effects of a protracted period of premium surcharges could be a factor in Treasury’s determination to cancel discretionary recoupment. Based on previous federal action following natural disasters or financial market crises, there may be an expectation that the government would provide financial assistance to businesses for uninsured or underinsured losses related to a terrorist event, regardless of whether a loss-sharing program existed. For example, the federal government uses the Disaster Relief Fund to provide compensation for property damage or financial losses to victims of Presidentially declared major disasters and emergencies. In fiscal years 2005–2018, the federal government designated $138 billion in supplemental appropriations to this fund for extreme weather events. And following the financial crisis of 2007– 2009, the federal government provided financial assistance directly to General Motors Company and Chrysler Holdings to help stabilize the U.S. automobile industry and to avoid economic disruptions. Treasury officials and industry stakeholders described several terrorist event scenarios that could produce a large amount of uninsured or underinsured losses that affected businesses might not be able to absorb and that might lead to the expectation of federal assistance. NBCR events present an implicit exposure. Historically, insurance coverage for losses related to a NBCR terrorist event has been limited or unavailable. Stakeholders told us that there likely would be an expectation of federal financial assistance for businesses with uninsured losses related to such an event. Treasury officials and stakeholders we interviewed agreed that primary and reinsurance coverage for NBCR events is limited, resulting in many businesses having limited or no coverage. Stakeholders also told us that, without TRIA, insurers would no longer offer the limited amount of NBCR coverage currently available. Stakeholders attribute the limitations to the potentially catastrophic losses associated with NBCR events and the difficulty in modeling and underwriting such events. Representatives of a policyholder association whose members purchase NBCR coverage stated that available coverage likely was insufficient to cover expected losses. Furthermore, they said some policyholders forgo NBCR coverage because of its limited availability, high cost, and the low perceived risk of a NBCR event. As a result, many businesses may be exposed to high loss. Treasury officials and industry stakeholders described possible NBCR terrorism events in which a significant amount of losses could be uninsured or underinsured. Treasury’s 2019 Small Insurers Report found that a NBCR terrorism event likely posed the greatest risk of total catastrophic terrorism losses, far outpacing a conventional attack. Although modeling these types of losses is difficult, NAIC’s Center for Insurance Policy and Research estimated, taking into account the program cap, that a NBCR event in New York City could generate nearly $60 billion of uninsured loss, 38 percent of the total loss. It also found that a larger NBCR event could create $850 billion in uninsured loss, or 90 percent of total losses. Furthermore, this research estimated large uninsured losses in other cities, such as Houston, where a large nuclear event was estimated to generate $67 billion in uninsured losses, or 40 percent of total losses. Such catastrophic losses could create a strong public expectation of federal financial assistance for uninsured losses. The expectation of financial assistance to policyholders if insured losses exceeded the program cap also creates implicit fiscal exposure. By law, insurers that have met their individual deductible and the federal government are not responsible for losses exceeding the TRIA program cap. However, industry stakeholders we interviewed expected that, in the event the program cap were exceeded, the federal government would provide some form of assistance to those who experienced loss. For example, losses from a NBCR event could be over $1 trillion, with TRIA- insured losses exceeding the $100 billion program cap. While a single conventional attack would be unlikely to exceed the program cap, according to stakeholders, a series of conventional attacks could. Although determining the frequency of terrorist events is difficult, one modeling firm with which we spoke estimates losses great enough to exceed the program cap in a conventional attack to have a 0.0005 percent (or 1/2,000) chance of occurring in a single year. This is less likely than severe natural catastrophes, such as Hurricanes Sandy and Harvey. As we found in a 2019 report on fiscal exposures, Congress demonstrated its willingness to fund the implicit exposure of policyholder claims that exceeded the amount NFIP was authorized to borrow from Treasury. In October 2017, when NFIP was about to exhaust its borrowing authority, Congress passed a supplemental appropriation, which the President signed into law, that cancelled $16 billion of NFIP debt to Treasury. Implicit fiscal exposure also exists in the expectation that the federal government would assist policyholders unable to pay their retained losses. Policyholders with very large retained losses may face financial insolvency after a terrorist attack, which may create an expectation of government assistance. Stakeholders told us that policyholders may choose larger retention amounts to reduce premiums or because insurance for high-risk, high-value properties is unavailable. Insurers we interviewed said that most businesses have small deductibles, but some large businesses may choose higher retention amounts to reduce the high insurance cost in locations considered to be higher-risk and for high- profile properties. For example as shown in figure 6, Treasury’s 2017– 2019 data call scenarios explored estimated losses in locations with high- profile properties such as Rockefeller Center in New York City (2017), Willis Tower and O’Hare International Airport in Chicago (2018), and Embarcadero Center and Union Square in San Francisco (2019). Although the actual amounts may be lower than the estimates insurers reported, the aggregated policyholder retention could exceed losses paid through the program. This demonstrates the potential for large losses that could create an expectation of government assistance if policyholders with large retention amounts were unable to absorb the losses. An implicit fiscal exposure exists from the potential expectation that the government might help stabilize markets by assisting insurers with (1) losses that do not trigger the program’s loss sharing, or (2) losses from lines ineligible for TRIA. While these risks exist, the current market has some protections in place and stakeholders viewed this exposure as unlikely. Loss sharing not triggered. If the total losses from a certified act of terrorism were below the program trigger (currently $200 million), insurers with deductibles below the program trigger could sustain losses larger than their deductible without receiving any federal coshare. Because the amount of the program trigger has increased over time, more insurers potentially face this scenario. Stakeholders told us that small insurers and those that offer workers’ compensation insurance are most affected by changes to the program trigger. Our analysis of Treasury data shows about 97 percent of insurers have deductibles lower than the $200 million program trigger and thus could receive no coshare following a certified act of terrorism. This includes all small, captive, and alien surplus lines insurers, and more than half of the nonsmall insurers. Furthermore, our analysis of 2016–2018 Treasury data shows that these insurers are sometimes concentrated in certain insurance lines. For example, small insurers may be concentrated in commercial multiple peril lines. Additionally, market shares of small insurers and captive insurers increased in the aircraft (all perils) line. Such concentrations could destabilize specific insurance lines following a terrorist event. Although the market is currently stable, some insurers may leave the market to mitigate their risks if their losses are unlikely to trigger the program’s loss sharing, which could reduce the availability of insurance in certain markets. A reduction in the availability of insurance could lead to more uninsured losses in the event of a subsequent terrorist event and could result in an increased expectation for losses to be covered through federal assistance. In its 2019 report, Treasury recognized the potential for small insurers to not provide insurance in certain markets. Additionally, the report cautions that if the program trigger increased, the number of insurers that would face the possibility of a gap between their deductible and the program trigger also would increase. However, analysis of Treasury data indicate that, to date, insurers in this situation largely have not left the market. As previously noted, because the changes to program parameters were gradual, insurers have had time to adjust to the changes. Insurers facing this scenario also can mitigate their risk by purchasing reinsurance to cover the difference between their deductible and the program trigger. However, it is not clear that the reinsurance market can absorb all of this risk. For example, industry stakeholders told us that reinsurers are sensitive to accumulation of exposure and reinsurance is limited in perceived high-risk areas such as New York City, Washington D.C., Los Angeles, and Philadelphia where there are large concentrations of people and high-value properties. In addition, reinsurance premiums may be too costly for some insurers. According to state insurance commissioners, high reinsurance costs may not qualify as a reason for insurers to increase premiums in some states. As a result, insurers would be unable to pass reinsurance costs on to policyholders. Furthermore, limited availability and affordability of reinsurance for high-risk areas and NBCR could be exacerbated following a terrorist event. Alternative forms of reinsurance, such as catastrophe bonds, are not widely used for terrorism risk. Treasury officials said that widespread market instability and an expectation for federal assistance may be unlikely in the case of a certified act of terrorism that produces significant losses for insurers that do not reach the program trigger. They said any market effects would be localized because a smaller terrorist event likely would affect a small number of insurers with a gap between their deductibles and the program trigger. Industry stakeholders said that it is possible that such an event could cause insurers to reduce coverage offered, but other stakeholders said that small insurers were well aware of the risk and their reactions would not create market instability. Additionally, state guarantee funds may provide support to policyholders in case of insurer insolvency before expectations for federal financial assistance arose. Insurance lines not eligible for TRIA. Federal assistance may be expected for insurers if large terrorism losses occur in insurance lines— such as life or health insurance—that are not eligible for TRIA, according to industry stakeholders. A 2019 Insurance Information Institute report found life insurance losses resulting from the attacks of September 11, 2001, were $1.4 billion (about 3 percent of total insurance losses from the attacks), an amount that far exceeds the TRIA program trigger. The report notes that standard homeowner, condo or co-op, standard renters, automobile insurance, and travel insurance policies also could be affected by terrorism and are not covered under TRIA. Adding group life insurance coverage under TRIA has been proposed, but never passed into law. According to some perspectives in congressional debate, other insurance lines were not viewed as needing explicit federal assistance. Insurers have adjusted to changing TRIA program parameters that increased the share of the losses for which they would be responsible. However, some insurers may not clearly understand whether Treasury would include policyholder retention amounts in calculating losses to certify a terrorist event, trigger loss sharing, or determine when the program cap has been reached. External communication to develop a clear understanding of how Treasury calculates “insured losses” and “insurance losses,” specifically as they relate to policyholder retention amounts, would help insurers understand when the program would be activated or capped ahead of any terrorist event. Furthermore, communicating this explanation could help Treasury alleviate uncertainty in the insurance market following a terrorist event. The Director of the Federal Insurance Office should communicate to insurers in writing how it would utilize policyholder retention amounts in calculating “insurance losses” and “insured losses” in determining the program certification threshold, trigger, and cap, as applicable. (Recommendation 1) We provided a draft of this report to Treasury and NAIC for review and comment. Treasury provided written comments through the Federal Insurance Office, which are reproduced in appendix III. Additionally, Treasury provided technical comments, which we have incorporated, as appropriate. NAIC did not provide technical comments. In Treasury’s written response, the Federal Insurance Office agreed that limiting uncertainty in the insurance market following a certified act of terrorism was an important goal. The office accepted the recommendation and stated that it would work to implement it in the coming months. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Treasury, the Chief Executive Officer of NAIC, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or garciadiazd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. The objectives of our report were to (1) examine the changes in explicit fiscal exposure under the Terrorism Risk Insurance Act (TRIA) and how insurers have adjusted to the changes; and (2) describe situations in which implicit fiscal exposures may arise and might become explicit. To address these objectives, we reviewed the Terrorism Risk Insurance Act of 2002, Terrorism Risk Insurance Extension Act of 2005, Terrorism Risk Insurance Program Reauthorization Acts of 2007, 2015, and 2019, implementing regulations, and congressional records. We also reviewed prior GAO work on TRIA and federal fiscal exposures. We reviewed reports from the Department of the Treasury (Treasury), the Congressional Budget Office, and Congressional Research Service. We also reviewed relevant reports from academic researchers and industry stakeholders. We used Treasury’s data calls from 2017 to 2019 for aggregated direct- earned premiums, numbers of insurers, and hypothetical loss scenarios throughout this report. We evaluated the reliability of the data by performing electronic tests and interviewing staff from Treasury and its data contractor, and industry stakeholders. We found the data sufficiently reliable for the purposes described below. We interviewed Treasury officials and representatives from the National Association of Insurance Commissioners (NAIC) and industry stakeholders, including insurers, insurance trade associations, a rating agency, risk modelers, and an insurance broker. We selected a nongeneralizable sample of five insurers to interview. These insurers were selected because they provide terrorism coverage to businesses and reflect a mix of sizes and types of insurance. In interviews, we asked about aspects of the program and the insurance market, and risks that could lead to implicit exposure. To describe the potential explicit fiscal exposure to the federal government, we reviewed the relevant laws and analyzed the changes made in each reauthorization. To quantify and compare the federal explicit exposure from potential terrorist events in 2015 and 2020, we estimated the TRIA-eligible direct-earned premiums for those years and used simulated loss scenarios similar to those we previously developed. To estimate the TRIA-eligible direct-earned premiums for 2015 and 2020, we used aggregated direct-earned premiums for 2016–2018 from Treasury’s data calls. Specifically, first we calculated the annual percentage change in direct-earned premiums for 2016–2017 and 2017–2018. Second, we found the average change to be an increase of about 0.4 percent. This estimate was smaller than the percentage change we used in the 2017 report, but we found the estimate reasonable because our current estimate was based on Treasury’s data, specific to TRIA-eligible lines of insurance. Our 2017 estimate was based on annual estimates of terrorism risk revenue. Third, we estimated the 2015 and 2020 direct-earned premiums using the 0.4 percent average annual change and the reported 2016 and 2018 direct-earned premiums, respectively. To check for reliability, we used the same method to estimate direct-earned premiums for 2014. Our 2014 estimate matched the 2014 data used in our 2017 report. We calculated the change in both nominal and real dollars and decided to use nominal dollars, which is consistent with how Treasury reports direct-earned premiums. We found the data to be sufficiently reliable for estimating the TRIA-eligible direct-earned premiums for 2015 and 2020. To compare events occurring under the program provisions in effect in 2015 and 2020, we analyzed how losses would be shared between the government and insurers by modeling terrorist events with insured losses of $5 billion, $25 billion, $50 billion, $75 billion, and the maximum terrorism event size ($100 billion in insured losses) in 2015 and 2020. In each case, we modeled the affected insurers to have an aggregate direct-earned premium base of 25, 35, 55, and 100 percent of all TRIA-eligible premiums. To demonstrate how the program trigger may affect insurers, we also modeled a smaller terrorist event with $290 million in losses and assumed that the affected insurers had an aggregate direct-earned premium base of $750 million. In all cases, we estimated the portion of federal losses that would be subject to mandatory and discretionary recoupment. To determine how insurers have adjusted to changes in TRIA and measure market stability, we reviewed Treasury and past GAO reports and relevant literature, analyzed Treasury data, and interviewed industry stakeholders. We conducted a literature search to determine how changes in the TRIA program parameters affected insurers and we summarized relevant findings from our review of the literature we identified. Additionally, we summarized Treasury and industry stakeholders’ views on insurers’ ability to cover their share of losses following an event and their willingness and ability to continue providing coverage after a large event without access to the federal share of losses to cover claims from the event. We also analyzed Treasury data to determine whether insurance lines have experienced changes in premiums or coverage availability since 2016. Specifically, we computed market shares by insurer category for calendar years 2016–2018, using direct-earned premiums from Treasury’s data calls. Using Treasury’s insurer categories (alien surplus lines, captive, nonsmall, and small), we computed market shares overall, and for TRIA-eligible lines of coverage in the U.S. terrorism risk insurance market, which includes all the states, the District of Columbia, U.S. territories, and other areas. We did not use number of policies because number of policies was not available in Treasury’s data for all insurer categories for all years. In the course of assessing data reliability for Treasury’s scenario data, we found some lack of clarity regarding two key terms under TRIA: “insured losses” and “insurance losses.” The external communication component of internal control—that management should externally communicate the necessary quality information to achieve the entity’s objectives—was significant to this objective, along with the related principle that management communicate quality information externally through reporting lines so that external parties can help the entity achieve its objectives and address related risks. We assessed Treasury’s external communications about these terms. Specifically, we reviewed the content of TRIA statutory language and program regulations and guidance. We obtained an interpretation of the terms from Treasury officials and also obtained industry stakeholders’ views on them. We determined Treasury’s scenario data to be reliable for the purpose of reporting the loss sharing as it was reported and for illustrating loss sharing examples. To identify potential situations in which implicit federal fiscal exposure may arise, we analyzed TRIA’s program design and reviewed our prior work for sources of implicit fiscal exposures, such as those faced by other disaster insurance programs. To ensure the reasonableness and completeness of our list of identified sources, we consulted with industry stakeholders and made modifications as appropriate. We grouped the sources of implicit fiscal exposures into three broad categories: (1) any unrecouped program expenditures; (2) federal assistance for uninsured or underinsured terrorism losses (including uninsured losses in a nuclear, biological, chemical, or radiological event, losses in excess of the $100 billion program cap, and policyholders’ retained losses); and (3) federal assistance to stabilize the insurance market for insurers that may be unable to access the loss-sharing feature of the program or for lines of insurance not included under TRIA. To describe the potential exposure resulting from not executing the recoupment of program expenditures, we summarized prior GAO reports and industry stakeholder views on the risks and challenges of collecting mandatory and discretionary recoupment funds, including the associated collection time frames. To describe the potential exposure resulting from federal assistance for uninsured or underinsured losses, we reviewed and summarized findings in prior GAO reports on past instances in which the federal government provided disaster assistance (such as following the September 11 terrorist attacks, the financial crisis, and large natural disasters). We summarized Treasury, NAIC, and industry stakeholder views on the likelihood of terrorist events reaching the program cap, their expectations for federal intervention in that case, and the importance of the cap to the terrorism risk insurance market. To describe the potential exposure resulting from federal assistance to stabilize the insurance market following a terrorist event, we interviewed industry stakeholders about scenarios that could produce an expectation for government assistance (implicit exposure), the stability of the terrorism insurance industry, how program details could affect different insurer groups, and insurers’ options for covering their share of losses. We used Treasury’s data to quantify the number and share of insurers that might not be able to access TRIA’s loss-sharing provision and to illustrate estimated loss sharing in Treasury’s scenarios of losses that would have occurred in three hypothetical events: New York City (2016), Chicago (2017), and San Francisco (2018). We also interviewed various insurer associations. We conducted this performance audit from July 2019 to April 2020, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. As shown in table 1, Terrorism Risk Insurance Act (TRIA) reauthorizations through 2015 changed several loss-sharing provisions, which decreased the federal share and increased the insurer share of losses. The 2015 reauthorization required incremental decreases in the federal share of losses over 5 years. The 2019 reauthorization extended the program until the year 2027, but did not make any changes to the program parameters discussed below. In addition to the contact named above, Jill Naamane (Assistant Director), Karen Jarzynka-Hernandez (Analyst in Charge), Rudy Chatlos, Giselle Cubillos-Moraga, Kaitlan Doying, Lijia Guo, John Karikari, Barbara Roesmann, Jessica Sandler, Jena Sinkfield, Frank Todisco, and Rachel Whitaker made key contributions to this report.", "summary": "Congress enacted TRIA to help ensure the availability and affordability of commercial property/casualty insurance for terrorism risk and to address potential effects on the economy in the absence of such coverage. Under the TRIA program, which is set to expire December 31, 2027, the government and insurers share losses following a certified act of terrorism. TRIA creates explicit fiscal exposure because the government is legally required to make payments to insurers after such an event, but there also may be some implicit exposure from an expectation of federal spending. To date, Treasury has not certified any acts of terrorism. GAO was asked to examine federal fiscal exposure under the TRIA program. This report (1) examines changes in explicit fiscal exposure under TRIA and how insurers have adjusted to the changes, and (2) describes situations in which implicit fiscal exposures may arise and might become explicit. To conduct this work, GAO reviewed the TRIA statute and related studies, analyzed Treasury data, and interviewed a nongeneralizable sample of insurers of different sizes providing various types of insurance. Terrorism Risk Insurance Act (TRIA) reauthorizations through 2015 have decreased federal fiscal exposure, and insurers have adjusted by managing their increased risk. Changes in the TRIA program that the Department of the Treasury (Treasury) administers—particularly incremental changes since 2015—reduced the government's explicit fiscal exposure from a certified act of terrorism (see figure). For example, by increasing the program trigger—minimum amount of industry-insured losses needed to activate the program—Congress potentially reduced the number of events that qualify for federal payments. As explicit federal fiscal exposure has decreased, insurer exposure has increased. Nevertheless, the market for terrorism risk has remained stable. However, some insurers are uncertain how Treasury defines insured losses for the purposes of calculating whether the program's $200 million trigger or $100 billion cap have been reached. For example, some insurers interpreted insured losses to include the portion of losses policyholders retain, which was different from Treasury's interpretation. Differences in interpretations could lead to disputes between insurers and Treasury following a terrorist event. One purpose of TRIA is to stabilize the market following a terrorist event. Communicating how it would calculate losses toward these program amounts could help Treasury alleviate uncertainty in the insurance market following a terrorist event. The government also has implicit fiscal exposure following a terrorist event, arising from expectations based on current policy or past practices that it may provide assistance, even when it is not legally required to do so. Although the government may not act on these expectations, the government's implicit exposure might become explicit if it chooses not to recoup the full federal share of losses from property/casualty policies, as allowed under TRIA, to prevent further stresses on the insurance market after a major terrorist event; assists companies with uninsured or underinsured losses after a terrorist event or when losses exceed the program cap; covers uninsured losses from a nuclear, biological, chemical, or radiological terrorism event; or assists insurers with losses that did not meet TRIA's trigger for loss sharing, or that were incurred in excluded lines of coverage, such as life and health insurance. GAO recommends that Treasury communicate how it would calculate losses, as they relate to policyholder retention amounts, in determining the TRIA program trigger and cap. Treasury agreed with the recommendation.", "document_type": "gao"}
{"report": "When complete, JWST will be a large, deployable space telescope, optimized for infrared observations. It is the scientific successor to the aging Hubble Space Telescope launched 29 years ago. JWST is being designed for a 5-year mission to find the first stars, study planets in other solar systems, search for the building blocks of life elsewhere in the universe, and trace the evolution of galaxies from their beginning to their current formation. JWST is intended to operate in an orbit approximately 1.5 million kilometers—or 1 million miles—from Earth. With a 6.5-meter (21.3 foot) diameter mirror, JWST is expected to operate at about 100 times the sensitivity of the Hubble Space Telescope. Its science instruments are designed to observe faint infrared sources and therefore are required to operate at extremely cold temperatures. To help keep these instruments cold, the JWST project will rely on a multi-layered, tennis court-sized sunshield to protect the mirrors and instruments from the sun’s heat. The JWST project is divided into three major segments: observatory, ground, and launch. When complete, the observatory segment will include several elements (Optical Telescope Element, Integrated Science Instrument Module, and spacecraft) and major subsystems (sunshield and cryocooler). Additionally, JWST is dependent on software to deploy and control various components of the telescope, and to collect and transmit data back to Earth. The elements, major subsystems, and software are being developed through a mixture of NASA, contractor, and international partner efforts. See figure 1 for the elements and major subsystems of JWST and appendix I for more details, including a description of the elements, major subsystems, and instruments. JWST depends on more deployment events—steps after launch that configure the observatory for its mission and place it in orbit—than a typical science mission. Due to the observatory’s large size, it is nearly impossible to perform deployment tests of the fully assembled observatory in a thermal vacuum chamber to simulate the space environment, so the verification of deployment elements is accomplished by a combination of lower level component tests in flight-simulated environments; ambient deployment tests for subsystem, element, and observatory levels; and detailed analysis and simulations at various levels of assembly. Figure 1 shows the multiple layers of integration and testing for major components of the JWST observatory. For the majority of work remaining, the JWST project is relying on two contractors: Northrop Grumman and the Association of Universities for Research in Astronomy’s Space Telescope Science Institute. Northrop Grumman plays the largest role, developing the sunshield, the Optical Telescope Element, the spacecraft, and the Mid-Infrared Instrument’s cryocooler, in addition to integrating and testing the observatory. Space Telescope Science Institute’s role includes soliciting and evaluating research proposals from the scientific community, and receiving and storing the scientific data collected, both of which are services that the Institute currently provides for the Hubble Space Telescope. Additionally, the Institute is developing the ground system that manages and controls the telescope’s observations and will operate the observatory on behalf of NASA. JWST will be launched on an Ariane 5 rocket, provided by the European Space Agency. The JWST program has a history of significant schedule delays and project cost increases, which resulted in both the 2011 and 2018 replans. Prior to approving the project’s development, cost estimates for JWST ranged from $1 billion to $3.5 billion, with expected launch dates ranging from 2007 to 2011. Due to early technical and management challenges, contractor performance issues, and low levels of cost reserve, the JWST program experienced schedule overruns, launch delays, and cost growth. The program underwent a replan in September 2011 and then a rebaseline; further, Congress placed an $8 billion cap on the formulation and development costs for the project. However, in June 2018, after a series of launch delay announcements due to technical and workmanship issues identified during spacecraft element integration, NASA notified Congress that it had revised the JWST program’s cost and schedule estimates again. NASA estimated that it would now require $828 million in additional resources over the program’s lifecycle and 29 more months beyond the estimates agreed to in the 2011 rebaseline to complete the project. Since the project’s costs and schedule were baselined in 2009, costs have increased by 95 percent and its launch date has been delayed by over 6.5 years. Prior to this more recent replan, NASA established an Independent Review Board (IRB) in April 2018, comprised of technical experts from outside the JWST program to evaluate all factors that may affect the successful completion of remaining mission steps. The board released its final report in May 2018 in which it made 32 recommendations that address a range of technical, organizational, and other factors. The IRB took into account varying technical and workmanship errors, human mistakes, adequacy of integration and test staff, and other considerations when it analyzed the project’s organizational and technical issues. The IRB recommended, among other actions, that the project conduct an audit to identify potential embedded design flaws; establish corrective actions to detect and correct human mistakes during integration and testing; establish a coherent, agreed-upon, and factual narrative on project status and communicate that status regularly across all relevant stakeholders; and, finally, augment integration and test staff to ensure adequate long-term staffing and improve employee morale. These recommendations also included reconsidering the proposed launch date. In March 2019, we found that NASA had considered many of the program’s risks while developing its 2018 replan schedule and cost baseline but recommended that additional analysis be completed to provide NASA and Congress with better insight into project resourcing and affordability. A Joint Cost and Schedule Confidence Level is an integrated analysis of a project’s cost, schedule, risk, and uncertainty, the result of which indicates a project’s likelihood of meeting cost and schedule targets. The project did not complete such an analysis as part of its second replan. NASA policy says this tool may be used to inform planning. Though not required by NASA policy, we recommended that one be conducted given the long history of program challenges and the significant and complex integration events that still needed to be completed. NASA agreed with our recommendation and completed this analysis in October 2019. GAO plans to conduct a separate, more detailed engagement on this analysis and its findings in the future. See appendix II for more information on this and other GAO recommendations. The JWST project, like other complex development efforts we have reviewed, faces numerous risks and potential unforeseen technical challenges, which often become apparent during integration and testing. To accommodate unanticipated challenges and manage risk, projects include extra time in their schedules, referred to as schedule reserve, and extra funds in their budgets, referred to as cost reserve. Schedule reserve is allocated to specific activities, elements, and major subsystems in the event of delays or to address unforeseen risks. Each JWST element and major subsystem has been allocated schedule reserve. When an element or major subsystem exhausts schedule reserve, it may affect schedule reserve on other elements or major subsystems whose progress is dependent on prior work being finished. Cost reserve is additional funding within the project manager’s budget that can be used to address and mitigate unanticipated issues for any element or major subsystem. Goddard—the NASA center with responsibility for managing JWST— issued procedures detailing the cost and schedule reserve requirements for formulating and executing spaceflight programs. When NASA constructed its 2018 replan for the JWST project, it took into account the remaining integration and test activities planned prior to launch, known technology challenges that presented risks to schedule, as well as potential future risks. The project’s replan reflected a planned schedule reserve above the level indicated by Goddard policy, which would have been approximately 5 months at that time. Instead, the new schedule included a total of 293 days or 9.6 months of schedule reserve, with approximately 6 months of this reserve to be managed at the project level and the remainder held by the program at NASA headquarters. Following the replan, the project and the contractor worked toward a launch date in November 2020, which would have required none of the schedule reserve managed at the NASA headquarters level. However, the committed launch date under the replan, where all available schedule reserve is utilized, is now March 2021. NASA’s cost-plus-award-fee contract with Northrop Grumman has spanned approximately 17 years, during which time there have been significant variances in performance. These types of contracts are suitable when uncertainties in scope of work or cost of services prevent the use of contract types where prices are fixed. Award fee contracts provide contractors the opportunity to obtain additional fee beyond the costs charged to the government for enhanced levels of performance in areas identified in the contract’s award fee plan. Award fees may be used when key elements of performance cannot be defined objectively, and, as such, require the project officials’ judgment to assess contractor performance. For JWST’s contract with Northrop Grumman, these areas include cost, schedule, technical, and business management and are established in the contract’s award fee plan, which allows for the award of a scaled fee based on assessed performance. This plan has been revised over the life of the contract to incentivize performance in certain areas, but it has always required Northrop Grumman to meet a minimum standard to receive any award fee. Over the course of the JWST contract, nearly $250 million dollars will have been available to Northrop Grumman through this incentive. We have found that when NASA and the contractor have made revisions to fee evaluation criteria to focus on certain aspects of performance, the contractor has been responsive to the new criteria during its work on the JWST project. Though the JWST project has made significant progress since our last report in March 2019, technical challenges have required the use of most of the project’s available schedule reserve. According to NASA officials, the contractor has found ways to replenish reserve, but NASA is still reviewing some of these methods and the project continues to work through significant integration and testing events with less than a quarter of the schedule reserve allotted to it in June 2018. The technical challenges have resulted in prolonged employment of the contractor workforce, which is the primary driver for increased costs. Following the June 2018 replan, the project has achieved a number of integration and testing milestones and has taken steps to address previously identified technical challenges. Since our March 2019 JWST report, the program has completed testing on the individual component elements of the observatory and has integrated them to start observatory level testing, the last of five phases of integration and testing. Leading up to observatory integration, the project completed thermal vacuum testing of the spacecraft element in May 2019. This testing helped to ensure that JWST hardware will function properly in the vacuum of space and withstand significant temperature variations during deployment and operation, and provided data to corroborate modelling on which the observatory’s mission is based. Further, the project completed the last major testing milestone for optical telescope and science instrumentation elements—deployment of the secondary mirror assembly—in August 2019. This secondary mirror focuses the light collected by the 18 hexagonal primary mirrors of JWST into a beam and directs it toward scientific equipment aboard the observatory. Integration of the observatory components was completed in August 2019, and the program has deployed the sunshield as part of observatory integration and testing. NASA has also taken steps to address challenges noted in our previous reports. For example, In February 2018, we found that Northrop Grumman planned to modify the design of the sunshield’s membrane tensioning system in response to a risk of a cable snagging during deployment. NASA approved this redesign in May 2019 and employed a new approach to cable management that involves modification and replacement of certain cable clips and routing cables differently to manage slack that could cause snags. We found in March 2019 that the project office identified concerns that trapped air in the folded sunshield membrane could put too much stress on the observatory when the launch vehicle fairing depressurizes—the fairing is the part of the rocket that encapsulates JWST during flight. NASA, Northrop Grumman, European Space Agency, and European vendors responsible for operating and producing the launch vehicle have worked together to study this issue and have designed vents for the fairing that will mitigate the risk of damage to JWST. The new fairing vent design is expected to be tested aboard a rocket planned to launch in the spring of 2020. Despite the major accomplishments of the past year, the program has identified new technical issues that present risk for meeting the 2018 replan’s schedule requirements. Multiple technical issues have contributed to the use of schedule reserve since the June 2018 replan, but two identified in March and April 2019 have had the most significant effect. The program identified two significant anomalies during pre-testing events for the spacecraft element’s thermal vacuum testing, which first delayed thermal vacuum testing and then required additional time for investigation and implementation of solutions. Specifically, a traveling wave tube amplifier and a command and telemetry processor had errant powering issues during testing. These are important components of the observatory’s communication systems that enable JWST to send large amounts of science data and telemetry to the ground segment at high speed. Though the anomalies occurred at the same time and were both power- related, NASA does not believe they are related and has initiated separate review boards to determine solutions. The amplifier failure is attributed to workmanship issues on the part of a subcontractor. As of October 2019, the exact cause of the processor anomaly remained under investigation, but the electrical problem had been isolated to faults within specific circuit cards. NASA has taken steps to address the risks presented by both anomalies: it has received replacement amplifiers and has upgraded and tested an engineering model processor to replace the faulty one aboard the observatory if necessary. As a result of technical issues discovered since the June 2018 replan, the JWST program has had to use significantly more schedule reserve than it planned to and has been working towards the replan’s formally committed launch date of March 2021. As of October 2019, the project had used 224 days of schedule reserve, or about 76 percent of the total project and program-held schedule reserve incorporated into the June 2018 replan. All project-held schedule margin was used by March 2019, a point at which the project would have retained approximately 4 months of reserve according to its original plan. At one point since our March 2019 report, the project had as little as 18 percent of its total schedule reserve left, but contractor-led corrective action plans regained time through found efficiencies. As a result of these challenges, the project’s reserve fell below what is indicated by Goddard policy. NASA determined in May 2019 that the November 2020 launch date that the project had hoped to achieve was no longer feasible, and switched focus to meeting the committed launch date of March 2021. Figure 2 shows the level of planned reserve for JWST, reserve indicated by Goddard policy, and the project’s actual use of schedule reserve. Since then, however, the JWST project has determined that the March 2021 launch readiness date may not be feasible either, based on a detailed assessment of risks, costs, and schedule. In October 2019, the project completed a joint cost and schedule confidence level analysis in response to a GAO recommendation made in a previous report on the JWST program. Because of schedule delays resulting from technical challenges coupled with remaining risks faced by the project, the analysis assessed only a 12 percent confidence level for the project’s ability to meet the March 2021 launch readiness date. NASA typically establishes its cost and schedule baseline commitments at 70 percent confidence level. According to the analysis, this 70 percent baseline confidence level is associated with a July 2021 launch date. The project does not currently intend to change the launch readiness date in response to this analysis alone, but plans to assess the feasibility of the launch readiness date again in spring 2020 after significant technical tasks are completed. NASA and Northrop Grumman have a plan to recover schedule reserve but certain portions of the plan remain under technical review. Following the amplifier and processor anomalies, Northrop Grumman developed a corrective action plan to recover schedule reserve, and the contractor and NASA continue to look for ways to gain efficiencies. In June 2019, Northrop Grumman suggested a number of potential schedule optimization steps that were reviewed by NASA management. Northrop Grumman has begun to be implement some of these steps. If all steps are taken, the contractor estimates 65 days of project schedule will be recovered, nearly doubling the amount of reserve available to the project when the anomalies were discovered. Among the efforts described in this corrective action plan are to streamline aspects of vibration testing and to modify build and repair schedules so that a major panel on the spacecraft will only have to be opened once. Combined, these two steps would save an estimated 46 days. However, officials noted that for the plan for a single panel opening to remain viable, corrective actions for the amplifier and processor replacements would need to remain on schedule. The project continues to review some of Northrop Grumman’s proposed efficiencies, but more than half of these schedule savings have already been incorporated into the schedule reserve forecast. The project also continues to identify and monitor risks that could potentially result in further use of schedule margin. As suggested by the IRB, the project has led a number of audits looking for embedded risks. As of November, most of the audits planned have been completed and NASA identified some new risks. The following are some of the risks the project is monitoring that could affect schedule: The project found that certain bolts, determined to be deficient on another Northrop Grumman program, were used during the construction of the observatory. A study of this issue found that the bolts used did not meet specifications and could pose a mechanical strength risk. The unused bolts have been identified and isolated, but 501 were installed in the observatory. NASA is performing strength testing to determine if the bolts are strong enough, but some of the deficient bolts may need to be replaced, pending the findings of these tests. The project reported in August 2019 that grounding straps on the spacecraft’s momentum flap came loose during vibration testing. This flap will act as balance against solar pressure that could cause unwanted movement of the observatory while in orbit. Observatory- level vibration testing cannot begin until the flap is removed, repaired, and replaced aboard the spacecraft. In September 2019, the project found that a non-explosive actuator on one of its membrane retention devices did not fire as planned. These devices, which help to unfurl the sunshield of the spacecraft, are supposed to be electrically redundant, but only one of the two mechanisms used to fire the actuator worked during the test. The program reports that there are approximately 180 actuators on the JWST and the failure of any one of these actuators could result in the total loss of JWST science mission objectives. If the redundancy for the actuators is reduced, it would have a major impact on system reliability. The project is evaluating whether it needs to replace certain membrane retention devices that may not be able to withstand the coupled pressure placed upon them by the launch and newly designed fairing ventilation. Testing in the past did not account for all aspects of the pressures placed upon this hardware during launch and spaceflight. The project indicated that it is completing an analysis to determine if stronger devices need to be installed. The JWST project office reviews and reports on these and other risks monthly. As of October 2019, the project is tracking 50 risks—three more than when we last reported on JWST—of which 12 continue to be assessed as moderate concerns. Of the 50, 23 have been assessed to be at acceptable levels of risk but continue to be monitored should changes affect their status. For example, the risk associated with cabling within the sunshield was elevated in October 2019 when the project found that further testing was needed to ensure slack did not present an unacceptable threat to the spacecraft during deployment. Finally, nine of the 50 risks currently tracked by the project are related to the more than 300 single points of failure aboard the observatory. The project must conduct significant integration and testing activities in the coming months that could present further challenges. Our previous work on major NASA acquisition programs found that integration and testing is the phase when challenges are most likely to be found and schedules can slip. The science elements and the spacecraft have only recently been integrated. NASA will have to manage seven top-level integration and testing steps between October 2019 and December 2020 to include observatory-level vibration testing, sunshield deployment and stow, and electrical testing and repairs. Currently, this will all have to be completed with a diminished amount of schedule reserve. Northrop Grumman and NASA officials we interviewed agreed that no other major complication, such as those on the scale of the traveling wave tube amplifier and command and telemetry processor anomalies, can happen without putting the March 2021 launch date in jeopardy. As we found in March 2019, changes to JWST project’s life-cycle cost estimate are principally driven by schedule extension, which requires keeping the contractor’s workforce longer than expected to complete integration and testing. We also found that NASA’s cost estimate for the 2018 replan was based on a more gradual workforce reduction schedule than previously used by the Northrop Grumman. NASA continues to forecast an overall reduction in contractor and government workforce following the project’s launch readiness date with continued, steady support by the Space Telescope Science Institute during remaining development and post-launch phases of the program (see figure 3). The program reports that cost reserve is generally sufficient for planned work but technical challenges could cause workforce costs to increase. The cost and schedule analysis completed by the project in October 2019 indicated that the project will not exceed the cost commitment established in the 2018 replan even if launch is delayed further by a few months. According to officials, funding is sufficient to continue work even if the launch date slips 3-4 months past the March 2021 launch date. However, the technical issues identified during integration and testing activities have required the contractor workforce to remain engaged, instead of drawing down as planned. Rather than see a temporary drop in contractor work hours as hardware deliveries were completed ahead of observatory-level testing and integration activities, the project has maintained contractor workforce levels to address the issues described above. The contractor now forecasts approximately 15 percent more workforce hours between 2019 and 2022, the year following launch (see figure 4). Approximately $133 million in cost reserve funding will be used by the project over the next 2 fiscal years to accommodate increasing workforce retention costs. Since the June 2018 replan, NASA has taken steps to improve the JWST project by implementing Independent Review Board (IRB) recommendations, pursuing other oversight initiatives, and continuing to incentivize contractor performance through the use of award fees. NASA addressed all IRB recommendations even though the agency did not always agree with the IRB on the specific steps needed to address the recommendations. Further, NASA has sustained, and in some cases expanded, the oversight initiatives that were started prior to our last report. The cost plus award fee contract used for JWST development efforts provides the project with a means to incentivize contractor performance related to cost, schedule, technical, and business management goals. Since the 2018 replan, Northrop Grumman’s award fee evaluations have improved but remain below its average for the contract. NASA assessed all IRB recommendations as closed in October 2019. The IRB made 32 recommendations covering a range of topics from improving communication with stakeholders to identifying embedded problems. NASA implemented its recommendation to establish March 2021 as the committed launch date for JWST through the June 2018 replan. Responsibility for implementing the remaining 31 recommendations was split among headquarters, the program office, and the project. The 10 headquarters- and program-level recommendations covered high-level recommendations dealing with entities outside of the project or communication between upper-level NASA management and the project. The remaining 21 recommendations were implemented at the project level and included lower-level actions related to assessing, preparing for, and improving day-to-day work. NASA assessed most recommendations as implemented prior to an IRB follow-up assessment, but the IRB found that more work was required for some to completely align with the board’s intent. In February 2019, the IRB found that the steps NASA took for approximately one-third of its recommendations were either inadequate or needed additional work, with the remainder found to be appropriate. Specifically, the IRB categorized 21 of the recommendation responses as appropriate, eight responses as appropriate with additional work needed, and three responses as inadequate. The IRB’s monitoring of the JWST project ended with the February 2019 follow up (see figure 5). Though NASA agreed with the intent of all the IRB recommendations, it took a different approach than described in the IRB report when implementing the three recommendations where the agency’s response was assessed to be inadequate. NASA conducted additional work for the majority of recommendations assessed by the IRB to be incomplete. However, NASA determined that a few of the IRB recommendations managed at the headquarters level should not be implemented the way they were delineated in the IRB report. Specifically, The IRB found that the JWST reporting structure was complex, confusing, and ineffective. The IRB made two recommendations for NASA to update its reporting chain. The IRB believed the Science Mission Directorate Associate Administrator should have responsibility of the entire JWST program and the Goddard Space Flight Center Director should be responsible for all aspects of the JWST project. The IRB asserted that restricting the involvement of the Goddard director will reduce the probability of JWST success. NASA agreed that it is important to have clear organizational roles and responsibilities but had a difference of opinion about the best course of action. In November 2018 and July 2019, NASA announced updates to the JWST reporting structure. However, both times it reduced the role of the Goddard director in favor of more direct line of accountability from the JWST program to the Science Mission Directorate Associate Administrator and the NASA Associate Administrator. NASA asserts that these changes will provide more clear accountability for program performance and allow for expedited decision making. The IRB recommended that NASA’s Launch Services Program should have accountability for the JWST launch. NASA has taken actions to increase the involvement of the Launch Services Program but NASA maintains that it is not prudent or possible for the Launch Services Program to be accountable for the launch because the European Space Agency is contributing the launch vehicle and managing the launch. The IRB recognized the unique circumstances of using an international launch vehicle but continued to assert the importance of Launch Services Program accountability. A minority of IRB members were of the opinion that NASA took appropriate action. NASA has sustained increased oversight and involvement with Northrop Grumman following the announcement of an anticipated cost cap breach in March 2018. Previously reported improvements included both the implementation of IRB recommendations and the pursuit of self-initiated activities, like greater NASA on-site coverage and Northrop Grumman’s culture change campaign designed to shift focus toward quality assurance. Our March 2019 report, provided examples of these changes and initiatives. Table 1 below provides a summary of our previous report findings and the current status of the changes NASA and Northrop Grumman made in providing oversight and ensuring quality. Since we last reported in March 2019, NASA has made additional oversight changes to further enhance communication with and oversight of the contractor. Most of these changes emphasize greater involvement of NASA specialists in meetings and reviews. NASA officials reported that its increased presence with the contractor has had positive effects for both ensuring project outcomes and increasing morale of the government and contractor workforce. For example, NASA integration and testing leadership is present and embedded in Northrop Grumman’s meetings— directly participating in planning sessions, reporting, and reviews of failures and anomalies. As a result, the project was able to plan for early integration of the observatory and completed key integration activities without being the primary driver of the project’s schedule. According to officials, expanded participation has helped to ensure more realistic exercises that include procedural concerns as well as engineering considerations. NASA officials said that the increased participation has allowed NASA input to be incorporated early—potentially reducing issues in the future. Further, NASA officials believe that the consistent presence of NASA personnel has improved morale—an item highlighted by the IRB—and helped foster greater unity of effort between government and contractor workforces. NASA has regularly assessed contractor performance through award fee assessments since the beginning of the contract in 2002. Award fee documentation over the course of the Northrop Grumman contract indicates that contractor performance was assessed as below its average before periods of significant cost and schedule growth. On average, Northrop Grumman has been rated as very good with about three-fourths of evaluations assessing its performance as either excellent or very good. For the award fee evaluations that fall below Northrop Grumman’s average score, cost performance has contributed to the majority of these dips and schedule performance has contributed to almost half. In particular, schedule performance has reduced the contractor’s overall evaluation for all award fee periods since April 2017. The latest dip below the contractor’s average preceded lifecycle cost growth of $828 million and schedule growth of nearly 2.5 years (see figure 6). Since our March 2019 report, Northrop Grumman’s ratings have improved but remain below its average. For the award fee period from October 2017 through March 2018, Northrop Grumman received an unsatisfactory rating, which resulted in the contractor receiving no award fee for the first and only time in the life of the contract. The unacceptable rating was driven by cost and schedule performance—including the anticipation of breaching the $8 billion congressional cost cap established in response to the 2011 rebaseline. In the following two periods, Northrop Grumman has improved its evaluation, but schedule performance remains a concern. During the last award fee period assessed, NASA was internally managing to a November 2020 launch date. Shortly after the award fee period ended, the project found it could no longer support the November 2020 date and began managing to the March 2021 launch date. We are not making recommendations in this report. We provided a draft of this report to NASA for comment. NASA provided technical comments that, among other things, clarified implementation of schedule recovery steps and updated progress on observatory repairs. We incorporated suggested technical changes as appropriate. We are sending copies of this report to the appropriate congressional committees, the NASA Administrator, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. In its previous reports on the James Webb Space Telescope (JWST), the GAO has made several recommendations. These recommendations are listed below. Comments reflect the status of the program at the time GAO closed the recommendation. Appendix III: List of Independent Review Board (IRB) Recommendations Telescope (JWST) launch success at the same level of responsibility they have for U.S. launches, or the National Aeronautics and Space Administration (NASA) should contract with Aerospace Corporation for similar accountability. 2 The Goddard Space Flight Center (Goddard) and Northrop Grumman Project Offices should be established as consistent and factual source of all JWST mission status 3 Communications of status and details appropriate for stakeholders need to be presented clearly and frequently. 4 NASA headquarters should be responsible for developing a “communication plan” (messaging strategy) for JWST. 5 Communicating complexity, risk, and science return for JWST is critically important. 6 Use the same criticality and assessment charts for all JWST reporting. 7 NASA should implement a JWST reporting structure where the Science Mission Directorate Associate Administrator has responsibility for the entire JWST program and the Goddard Space Flight Center Director is responsible for all aspects of the project. 8 NASA should revise NASA policy directives to be consistent with the recommendation. 9 Assure consistent, sustained and meaningful engagement of the Science Working Group (SWG). 10 Appoint an executive committee of NASA-selected members of the SWG to act as conduits to the broader community on mission challenges. 11 NASA should designate a Commission Manager. 12 NASA should implement sunshield hardware and simulation elements to aid in sunshield anomaly identification and resolution. 13 Northrop Grumman should establish corrective actions in1) processes, 2) training, 3) personnel certification, 4) discipline to ensure individual accountability and 5) a failure-proof “safety net” through a robust testing, analysis, and inspection process. 14 Goddard and Northrop Grumman should conduct an audit including forensic engineering, hardware pedigree assessment, drawing checks, etc., to identify potential embedded problems. 15 Goddard should conduct an audit of the JWST project residual risk, reviewing the objective evidence of (a) the completed Test As You Fly and Single Point Failures mitigation plans, and (b) failure corrective action effectiveness to determine the “as built” residual risk. 16 The project should reconcile the “as built” residual risk with the expected “as designed” residual risk. 17 NASA should define security requirements and plan for JWST transport to launch site. 18 Develop contingency operations and sparing plan for spacecraft/launch site operations. 19 Develop “pathfinder” JWST simulator and contamination protection systems for integration “dry runs.” 20 Assess shipping vessel contamination environment and develop contingency plans for off-nominal shipping operation. 21 It is critically important that Goddard JWST Project Office maintain responsibility and provide adequate support to ensure Space Telescope Science Institute (STScI) mission operations readiness 22 The Project should review all simulators/testbeds and required usage against pre-launch tests and rehearsals, post-launch deployment anomaly resolution, fault isolation, and correction. 23 The Goddard JWST Project Office should develop a staffing plan that meets the needs of integration and test and operational readiness. 24 The project should develop and approve a transition plan that defines the level of mission operations responsibility for STScI as a function of time with independent gate reviews at transition points. 25 Management should unambiguously emphasize the priority of mission success to “working level” personnel. 26 Employees must feel empowered to stop or slow down if the pace or procedures can jeopardize mission success. 27 NASA should assess “top ten” mission success enhancements and implement where appropriate. 28 Responsible Design Engineers should be involved and responsible for their element through the successful commissioning of the observatory. 29 The project should augment integration and test staff; this is critically important to execute the integration and test program. 30 Augment integration and test staff to achieve more realistic work schedules. 31 Implement strategies for improving team morale, such as periodic science lectures for Northrop Grumman personnel and families. 32 The Webb IRB recommends the launch date be established as March 2021 (based upon the Project’s 5/18 assessment of the impact of the membrane cover assembly acoustic anomaly). In addition to the contact named above, Raj Chitikila (Assistant Director), Christina Cota-Robles, Carrie Rogers, Ajani Skeete, Jay Tallon, and Thomas Twambly made key contributions to this report. Assistance was also provided by Hannah Brookhart, Brian Bothwell, Lorraine Ettaro, Emile Ettedgui, Laura Greifner, Kaelin Kuhn, Christine Pecora, Roxanna Sun, and Alyssa Weir.", "summary": "JWST, a large, deployable telescope, is one of NASA's most complex projects and top priorities. Problems discovered during integration and testing caused multiple delays that led NASA to replan the project in June 2018. Now estimated to cost $9.7 billion, the project's costs have increased by 95 percent and its launch date has been delayed by over 6.5 years since its cost and schedule baselines were established in 2009. Prior to the replanning process, an independent review board assessed the project and made recommendations to improve performance and oversight. Conference Report No. 112-284 included a provision for GAO to assess the project annually and report on its progress. This is GAO's eighth report. This report assesses the extent to which (1) the project is executing within its revised cost and schedule targets and (2) NASA has implemented and sustained key improvements to performance and oversight established following the June 2018 replan. GAO reviewed relevant NASA policies, analyzed NASA and contractor data, and interviewed NASA and contractor officials. The National Aeronautics and Space Administration's (NASA) James Webb Space Telescope (JWST) project has made significant progress since GAO's last report in March 2019, such as completing testing of the observatory's individual elements and integrating them together in August 2019. However, new technical challenges have required the project to use more schedule reserve—extra time set aside in the project's schedule to accommodate unforeseen risks or delays—than planned. As of October 2019, the project had used about 76 percent of its available schedule reserve and no longer plans to launch in November 2020 (see figure). The project is now managing to a March 2021 launch date but estimates only a 12 percent likelihood that this date will be achieved. NASA plans to reassess the launch date in the spring of 2020. The project used much of the schedule reserve in April 2019 to address issues with two components needed to transmit science data to ground control. The contractor has been able to mitigate some of the schedule loss and continues to look for new efficiencies. Technical challenges also resulted in longer employment of the contractor workforce than planned, which could result in additional cost increases. NASA continues to monitor multiple, other risks that could place further schedule and cost strains on the project. Since NASA replanned the project again in June 2018, the agency has taken steps meant to improve performance and oversight. NASA has addressed all recommendations from an independent review board, but in doing so sometimes took actions that differed from those outlined in the board's report. NASA has sustained, and in some cases expanded, oversight initiatives following the revised cost and schedule commitments that, in many cases, were designed to enhance communication between the government and the contractor. GAO is not making any new recommendations at this time. GAO has made several recommendations to NASA on the management of this project in previous reports and NASA has agreed with and taken action on many of them. Most recently, in March 2019, GAO recommended that NASA complete a joint cost and schedule confidence level analysis for JWST. NASA concurred and completed the analysis in October 2019 to support a key project review.", "document_type": "gao"}
{"report": "FAA is responsible for overseeing and authorizing any flight operations in the national airspace system for both manned and unmanned aircraft. FAA’s UAS Integration Office, located in the Office of Aviation Safety, seeks to integrate UAS operations into the national airspace system while ensuring the safety of the public and integrity of the airspace. In July 2018, FAA released the 2018 UAS Integration Roadmap, a second edition of the agency’s 5-year plan outlining its most current phased approach for integration, with each step toward full integration allowing UAS operations of increasing complexity. FAA’s vision for fully integrating UAS into the national airspace system entails UAS operating safely and routinely—i.e., without requiring prior approval for UAS flights—in the same airspace as manned aircraft. While safety is FAA’s paramount concern, the integration of UAS is important because of the potential economic benefits that progress in UAS integration could bring, including more investment in uses such as large passenger operations, as well as the potential safety benefits, such as more effective firefighting and other disaster response efforts. Currently, FAA only allows certain routine UAS operations under specific conditions while authorizing other UAS operations on a case-by-case basis. For example, since August 2016, operators of small UAS—defined as those UAS weighing less than 55 pounds, including any attachments— who have obtained a remote pilot certificate have generally been allowed to operate without prior FAA approval in certain airspace during the day, under 400 feet, and not over people or beyond an operator’s line of sight, among other requirements under FAA’s Part 107 rule. Small UAS operators may seek a waiver of certain FAA operational requirements (referred to as a Part 107 waiver) from the agency on a case-by-case basis, such as a waiver that would allow an operator to fly drones above 400 feet. In contrast, no routine operations—meaning those that can occur without any prior authorization—are currently allowed for large UAS (55 pounds and over) for any purpose (see fig. 1 for examples of small and large UAS). Rather, operators of large UAS must seek authorization from FAA to fly the aircraft on a case-by-case basis, and the processes for accessing the airspace vary. More specifically, civil large UAS operators must, in most cases, obtain a Certificate of Waiver or Authorization (COA) that demonstrates FAA’s approval of airspace access, and may also require approval for the aircraft itself. A COA allows any certificate holder to fly UAS outside of generally allowable operations, such as at certain altitudes, locations, or airspace classes (e.g., near airports). FAA grants this approval to an entity for a specific activity and time period, and sometimes for a specific make and model of UAS. Public entities—which include federal, state and local governments, public academic institutions, and law enforcement agencies—may apply for a COA in order to obtain authorized access to fly in the national airspace for when they are conducting governmental operations, as defined by statute. In such cases, the COA allows for the certificate holder to operate UAS in ways that would otherwise not comply with airspace requirements, such as operating the drone beyond the pilot’s line of sight. In its 2018 UAS Integration Roadmap, FAA outlined some key topics and operational capabilities to be researched that are associated with specific UAS integration phases (see fig. 2). For example, both government and industry entities have research and testing of technologies underway to provide UAS the capability to detect obstacles in midair, such as other aircraft, and automatically maneuver to avoid collision; this capability is commonly referred to as “detect and avoid.” FAA officials have stated that this key capability is necessary before allowing certain UAS operations on a routine basis, such as flights beyond the operator’s line of sight. According to FAA, the agency plans to use data from several UAS research programs—including the test site program—and from other sources to inform its future decisions regarding UAS integration. In 2012, FAA was required by statute to establish a program to integrate UAS into the national airspace system and to establish six UAS test sites in order to develop a process for allowing research to occur at these test sites, among other requirements. In response to Congress’ mandate, in 2013 FAA selected six public entities to be designated as test sites based on a number of factors, including geography, climate, and the respective institutions’ expertise, and added another entity in response to legislation in 2016 for a total of seven designated test sites. According to FAA officials, the test site program was intended to enable industry stakeholders to test complex UAS operations and conduct research on the corresponding technologies. Each test site is a public entity, such as a public academic institution or branch of the state government, which FAA authorizes to conduct various UAS operations through the COA process. UAS stakeholders, including manufacturers or entities seeking to use UAS for various purposes, can pay to work with any of the seven FAA- designated test sites to conduct test flights or receive training on UAS operations and regulations, among other activities, based on the test site staff’s expertise. FAA has not directly funded the test sites’ general operations, so the sites have had to rely on other funding sources, such as revenues generated from users, state funds, federal research grants, and commercial investment. Congress recently appropriated $6 million to FAA to provide matching funds to qualified commercial entities seeking to test UAS technologies at FAA designated test sites. FAA manages the test site program using formal agreements and by providing support to test site staff. The test sites signed individual Other Transaction Agreements (OTA) with FAA that establish their agreement to meet specific requirements aimed to support FAA’s UAS integration efforts. For example, these agreements lay out that test sites must follow safety processes and data procedures, as well as provide certain deliverables to FAA. Specifically, the agreements outline that the test sites will provide FAA certain operations and safety-related data for specific test flights, which FAA stores in a database it created specifically for test site data. In addition, once the test sites were operational, FAA designated an official to serve as the test site program manager for all seven sites who, among other duties, facilitates regular meetings with test site representatives to discuss ongoing issues and regularly communicates with other FAA lines of business to keep them informed about key efforts underway at test sites. According to FAA, the designated test sites have the equipment and infrastructure to support UAS flight testing, such as UAS pilots, launch pads, command centers, and, if required, chase aircraft (see fig. 3). Test site staff can facilitate UAS flight operations under a test site’s COA or by complying with the Part 107 rule. Since 2015, the test sites have held a “blanket” COA that allows them to conduct government functions for small UAS in Class G (uncontrolled) airspace anywhere in the United States except within restricted or prohibited areas. In addition, test sites have applied for and been granted COAs to operate UAS of different sizes in locations (referred to as “test ranges”) outside their state, and in a variety of airspaces at various elevations (see fig. 4 for a sample of test site COAs). For example, as of October 2019, the Alaska test site had COAs for test ranges in many states including Alaska, Hawaii, Tennessee, and Oregon—one of which allows operations up to 15,000 feet above mean sea level within three classes of airspace around Pendleton, Oregon. Some test ranges are located at airports, such as Griffiss International Airport in New York, which can help facilitate the testing of UAS that may require runways for take-off and landing, as well as testing of UAS flying in areas with manned aircraft. However, stakeholders, such as UAS manufacturers or companies interested in using UAS for various purposes, are not required to use an FAA-designated test site for UAS flight testing. In addition to seeking authorization directly from FAA to conduct their own flights or flying according to current rules such as Part 107, UAS stakeholders can work with other entities—such as military airports, public academic institutions or other public test sites—to which FAA has granted COAs to conduct complex UAS operations. For many stakeholders, however, working with a designated test site may provide quicker access to testing than seeking their own authorization from the FAA. For example, a UAS manufacturer might work with a test site to test the company’s UAS prototype at a certain elevation under a test site’s existing COA (following all applicable COA guidelines, such as performing a government function with the operation) because the test site already had that authorization in place. Additionally, it may be beneficial for a UAS manufacturer or operator to work with a test site because the test site has experience in obtaining authorizations or waivers from FAA for similar types of operations or aircraft. According to FAA’s MLS data, the test sites facilitated about 15,000 total UAS test flights occurring under test site COAs from April 2015 through December 2018 (see table 1). However, according to test site representatives, staff at these sites facilitated more UAS flights during this time frame than is reflected in the MLS data, because additional flights were conducted using different allowances than COAs, such as under the Part 107 rule that allows certain routine small UAS operations. According to FAA officials, the decrease—starting in 2017—in the annual number of reported test flights by the test sites, as reflected in table 1 above, is due in part to a change in regulations. Specifically, when FAA’s Part 107 rule took effect in August 2016, it provided a new avenue for small UAS operators, including test site staff and other airspace users, to test certain small UAS operations without requiring a COA or other authorization, effectively reducing the number of test flights logged into FAA’s MLS. Agency officials also told us that Part 107 changed the type of research users request from the test sites, which may have reduced the number of test flights facilitated through the test sites. While there have been fewer flight tests, according to some test site representatives and users we spoke to, recent testing has been for more complex research. For example, one test site representative stated that now the site’s users have bigger, more extensive research projects involving more tasks than just test flights, such as developing the operational models, performing testing on various technologies, and installing equipment to support complex UAS operations. Research conducted at the test sites has provided information to FAA that, according to agency officials, supports its efforts to integrate UAS into the national airspace system. Test site representatives told us that they have supported over 440 public and private users to conduct research and development on UAS to be used for a variety of UAS activities. While FAA officials told us that they cannot direct specific types of research to be conducted at the test sites unless the agency funds that research, we found that users have nevertheless conducted UAS research and development activities that FAA has identified as important for UAS integration. For example, users conducted research on the safety risks of UAS, such as concussion collision studies, and have tested UAS capabilities, such as the ability to carry loads of varying weights. Also, based on our analysis, we found that users have tested UAS technologies at the test sites that align with some of the key capabilities identified by FAA as necessary for the upcoming phases of UAS integration (see table 2). Test site users also reported benefits from working with test sites. According to the users we interviewed, the test sites have provided them an opportunity to explore and improve UAS technologies, and to learn more about how they could use UAS for their own purposes in the national airspace. For example, one user of the New York test site had tested communication equipment and detect-and-avoid capabilities on large UAS that they manufacture and sell to other entities for conducting surveillance activities, such as drug interdiction. Many of the test site users (11 of 18) we spoke to stated that using a test site provided a significant benefit for advancing their entity’s UAS research and development efforts. In addition, according to 9 of the 18 users we spoke to, test sites provided them with direct and immediate access to tools that helped them test their technologies. For example, users stated that it was beneficial that test sites have specific authorities from FAA for certain types of testing under a COA as well as infrastructure to allow for advanced UAS research. Some activities the test site users we spoke to plan to conduct with UAS are already regularly occurring—meaning FAA either allows these to occur on a routine basis or has allowed them to occur through additional authorization on a regular basis. Others are not yet occurring on a regular or routine basis due either to legal restrictions, such as restrictions on operating UAS beyond the operator’s visual line of sight or needed technological advancements, but FAA expects them to occur routinely in the future (see table 3). Some users we spoke to have also worked with a test site to conduct extensive hazard and risk mitigation testing to build safety cases and get approval from FAA to conduct complex UAS operations. FAA generally requires safety cases when a user is seeking approval to deviate significantly from current UAS requirements, such as when seeking to conduct beyond-visual-line-of-sight operations using a small UAS. For example, according to representatives from an insurance company we spoke to, they worked with the Virginia test site for over a year to build a safety case to prove that the company could safely operate its small UAS beyond the operator’s line of sight and over people. According to test site representatives, this risk mitigation testing entailed dozens of experiments, including how to address the risk of an UAS abruptly losing power. For instance, if a UAS operating over a house for an insurance inspection loses power, it could fall, potentially causing damage to the building as well as injuring someone standing on the ground below. In November 2018, FAA granted approval for the company to fly its fleet of UAS over people and beyond the operator’s line of sight in sparsely populated communities nationwide for insurance claim inspections. All test sites have competed for and were selected by federal agencies to participate, to varying degrees, in additional UAS research efforts designed to inform aspects of FAA’s integration plans. The projects include: The Department of Transportation’s (DOT) UAS Integration Pilot Program (IPP): In May 2018, DOT selected 10 project teams—which included the Alaska, North Dakota, and Virginia test sites—to participate in this program aimed at evaluating different concepts for certain UAS operations in specific communities. According to DOT, the IPP is an opportunity for state, local, and tribal government agencies to partner with private sector entities, such as UAS operators or manufacturers, to, among other things, accelerate the approval of operations that currently require case-by-case authorizations. Two key intended outcomes of the IPP are to assess the respective communities’ acceptance of low-altitude UAS operations, and to balance national and local interests in furthering UAS integration. For example, the Alaska test site is a member of the University of Alaska Fairbanks IPP team, with a primary focus of enabling complex UAS technology for pipeline inspections in the area’s harsh climatic conditions through testing technologies, such as using detect and avoid technology at night. While project awardees do not receive any federal funding for this program, FAA officials told us they are collecting data from IPP efforts to inform future decision- making. FAA’s Center of Excellence for UAS: In May 2015, FAA selected a team of 15 research institutions, including the Alaska and New Mexico test sites, called the Alliance for System Safety of UAS through Research Excellence (ASSURE), to serve as FAA’s Center of Excellence for Unmanned Aircraft Systems and to conduct academic research critical to safe and successful UAS integration. Congress has appropriated funds to ASSURE since fiscal year 2014 to pay for operational expenses and research, and according to FAA officials, ASSURE institutions are eligible to receive grant funding from FAA’s Research, Engineering, and Development appropriations. ASSURE institutions receive federal grants to conduct research to assess specific technologies or risks with the intent to inform FAA regulations and policies. For example, ASSURE institutions have received grants from FAA to study UAS noise certification, ground and airborne collision severity and impacts, and UAS detect and avoid technologies. According to FAA, funding from non-federal entities, such as international civil aviation authorities can be applied to ASSURE. Some of ASSURE’s research has been peer reviewed and published. According to an ASSURE representative we spoke to, all of the research conducted through ASSURE is in alignment with FAA’s plans for UAS integration as outlined in the 2018 UAS Integration Roadmap. FAA’s and NASA’s UAS Traffic Management (UTM): The UTM program is a collaborative effort of FAA and NASA to design a system with a similar concept as FAA’s air-traffic-control system for manned aviation that would enable small UAS to operate safely at low altitudes around other aircraft. NASA is leading the research, development, and testing of various technologies that would comprise the system, and plans to transfer the results of the research to FAA to determine next steps. NASA selected six test sites—Alaska, Nevada, New York, North Dakota, Texas, and Virginia—to participate, to varying degrees, in the four different phases of this project. NASA has provided funding to the six test sites through contracts for their participation in testing the system. UTM research is divided into four phases, called technology capability levels, each with specific technical goals. For example, technology capability level three entailed testing technologies that maintain a safe distance between two UAS flying over moderately populated areas. All six sites participated in the first three phases, which according to NASA officials brought in about 35 industry partners for this research effort. The Nevada and Texas test sites are currently participating in the fourth and final phase, which— as of October 2019— NASA expected to complete in 2019. In addition, FAA selected the North Dakota, Nevada, and Virginia test sites to participate in its UTM Pilot Program. The program’s goals are to develop, demonstrate, and provide services that will support the implementation of UTM operations. NASA’s UAS Integration in the National Airspace System: Beginning in 2015, NASA provided funding to the New York and Virginia test sites, among other entities, for this project, which is intended to demonstrate solutions to technical challenges to inform FAA’s development of operational standards for UAS. For example, through this project, NASA intends to test detect and avoid technologies by assessing UAS performance during a variety of scenarios, and then by recommending a minimum set of performance standards to FAA for consideration. According to NASA officials, the agency has completed work at the New York test site related to developing standards for routine operations by large UAS. As of October 2019, NASA had ongoing research at the Virginia test site on command and control communications that officials expected to complete in 2019. FAA’s UAS Detection at Airports: According to FAA, six test sites— Nevada, New Mexico, New York, North Dakota, Texas, and Virginia— participated in this program alongside various industry partners to evaluate technologies that can be used to safely detect UAS near airports. Funded by FAA, this research project included evaluating the capabilities of various UAS detection technologies by different manufacturers at four U.S. airports in 2016 and 2017. This research was used to inform minimum performance standards for UAS detection systems deployed at airports. All test site representatives stated that FAA has improved both its management of the UAS test sites and collaboration with representatives in recent years as the program has matured. According to test site representatives, initially, as the program began, there was considerable turnover among FAA test site managers, which made it more difficult for the staff at the test sites to collaborate with FAA officials to undertake research efforts. FAA officials acknowledged that because they had not established test sites before, it took time to determine the best approach for managing this program. However, according to most representatives, in the last few years, FAA has begun to better collaborate with the test sites. Specifically, FAA has solicited input from test site representatives on various issues related to UAS integration and helped facilitate information sharing between the test sites and various FAA lines of business. For example, agency officials told us that they invited air traffic specialists from a regional FAA office to participate in a recent UAS Test Site program semi-annual meeting. Through this meeting, these FAA regional staff learned about the test sites’ initiatives and about unique aspects of the test sites’ COAs, which, as previously noted, they use to conduct flight tests. According to FAA officials, with the better understanding about test sites’ operations gained at the meeting, these regional FAA staff will be able to process the test sites’ COA requests more efficiently. Most test site representatives also told us that FAA’s current UAS test site program manager and other FAA staff are responsive to, for example, questions or requests for guidance on a particular issue. Further, based on our interviews with test site representatives and our analysis of test sites’ reports submitted to FAA, the agency has taken steps to address some challenges from the past. In our March 2015 testimony and July 2015 report on FAA’s progress in integrating UAS into the national airspace, we outlined initial challenges that stakeholders most frequently cited as affecting test sites’ ability to attract users and to generate sufficient revenue to remain in operation during their first year. Since 2015, FAA has taken several steps to address these challenges, by providing additional guidance, streamlining the COA process for test sites, and improving the agency’s collaboration with and management of the test sites (see table 4). However, based on our analysis of interviews conducted for this review with test site representatives and users, these previously identified challenges persist. Lack of FAA guidance on priority research: Most test site representatives reported that while FAA has improved its management of the program, available FAA guidance still lacks the needed detail about research areas to prioritize in order to promote overall UAS integration efforts. For example, some test site representatives told us that the 2018 UAS Integration Roadmap should provide more information about the agency’s planned timeframes for implementing various steps to achieve full UAS integration, such as how and when FAA plans to integrate large UAS. Without such details, representatives say they cannot fully inform potential users when it might be possible to routinely use some complex UAS operations that are in demand by industry but currently only allowed on a case-by-case basis, such as the ability to fly small UAS beyond the operator’s line of sight or over people. Several representatives told us they are concerned that some potential test site users may postpone their research or conduct it abroad because of this lack of detail on when FAA plans to routinely allow such complex UAS operations. According to FAA officials and as noted in table 4 above, the agency has issued strategic plans and provided briefings to test site representatives and stakeholders on relevant research needed to achieve UAS integration. However, FAA officials told us that there are limitations on how much guidance they can provide the test sites. They said that the Anti-Deficiency Act prevents FAA from directing specific test site activities and obtaining research data, other than the operations and safety data required by the COA, without providing compensation. Officials also noted that until standards and regulations are developed—an effort for which the agency has not set a targeted completion date—a case-by-case approval basis will be needed for allowing complex UAS operations. With regard to the concern that some potential test site users may be conducting research abroad, FAA officials told us that testing abroad will not provide these stakeholders the same experience as testing in the United States, given that the U.S. national airspace system is more complex than those abroad in terms of traffic and congestion. Complex and lengthy COA process: Most test site representatives and users we interviewed told us that FAA should implement a less complex and time-consuming COA process for the test sites. According to test site representatives, FAA’s actions have decreased the time it takes to obtain simple COAs and Part 107 waivers, but for applications to conduct more complex research activities, FAA’s process remains lengthy and uncertain. This challenge makes it more difficult for test sites to meet users’ needs, according to representatives, and can subsequently lead companies to conduct UAS research in other countries. For example, some representatives told us that one test site’s request for a waiver to fly UAS beyond visual line of sight had taken 3 years for FAA to approve, and they could not understand why. Representatives also told us that for COA applications involving requests to research complex UAS operations, it was not always clear why FAA denied their requests, leading to uncertainty. According to FAA officials, the waiver that took 3 years to approve was an outlier and the agency’s processing of such waivers usually takes 90 days or less. However, in January 2018, DOT’s OIG similarly reported that FAA has had difficulty keeping pace with the volume of Part 107 waiver requests received and, in particular, has been slow to approve complex UAS waivers—such as requests to operate beyond the operator’s visual line of sight. In this report, the DOT OIG made recommendations related to improving the waiver process, which FAA is working to address. Generating sufficient revenue to maintain test site operations: Most test site representatives told us that securing sufficient funding to develop future capabilities and infrastructure in order to attract industry users and partners, remains a major challenge that they predict will continue. Some test site representatives told us that their respective contracts with NASA for projects such as UTM have been their largest single revenue source. Another representative mentioned that the U.S. Coast Guard has been a test site user, which has helped the site to generate revenue. Test sites have attempted to generate revenue in other ways, for example by obtaining state and local government funds to build infrastructure to attract users, applying for competitively awarded research contracts, and consulting and conducting research with potential users in different locations. FAA officials acknowledged that the test sites will need to continue to generate sufficient revenues to support their operations, but noted that, whenever possible, the agency provides the test sites with opportunities to compete to participate in funded research efforts, such as those related to the UTM program. Most test site representatives and users we interviewed also identified technology-related challenges affecting test sites’ ability to conduct research as continuing issues. These mostly relate to technology-related capabilities that will be vital for achieving full UAS integration, but which are currently still in development (see fig. 5). As we have previously reported, integrating UAS into the national airspace will require FAA to address key technology-related challenges to enable routine UAS operations with manned aircraft. For example, in our July 2015 report, we identified such challenges affecting test sites, in addition to the management-related challenges discussed above. According to test site representatives and FAA officials, these key technology challenges and concerns could affect broader UAS integration and research efforts, and thus impact the pace of or stop the progress toward full integration into the national airspace system. Such key technology-related challenges and related efforts to address them include: Availability of dedicated radio-frequency spectrum: Radio- frequency spectrum provides communication links between a UAS and its control station or operator. According to FAA, dedicated radio- frequency spectrum is important to ensure UAS safety and security in order to operate in the national airspace. For example, radio- frequency spectrum is needed for command and control, detect and avoid, and beyond visual-line-of-sight capabilities of UAS. Without a dedicated radio-frequency spectrum, the intentional or unintended interference of radio transmissions could sever the UAS means of control because other consumer products also use radio frequencies that could cause interference. FAA officials and test site representatives told us this spectrum-availability problem is the one challenge that has the potential to bring UAS research efforts to a halt if not addressed. Representatives from five of seven test sites indicated that availability of spectrum affects their ability to conduct their research operations and, more broadly, also affects the progress of other efforts contributing to UAS integration. Similarly, some test site users told us that when deciding on a potential test site to contract with for conducting their research, they asked about whether the test site faced any radio frequency interference. According to FAA officials, the agency is assisting test sites in addressing this challenge by collaborating with the Federal Communications Commission (FCC), which is responsible for allocating spectrum to nonfederal users for various purposes and assigning spectrum licenses. FAA’s Spectrum Office is a participant in the regularly occurring meetings between FAA officials and test site representatives. These representatives said they have been communicating with FAA to clarify guidance on the different frequency bands to use at various operating altitudes related to an FCC rule. Nevertheless, according to FAA officials, in the near future, more issues will likely surface related to spectrum because of the industry’s interest in conducting flights beyond visual line of sight for both small and large UAS. FAA officials told us spectrum reserved for aviation safety communications are limited. Therefore, the officials are investigating how to get the maximum UAS capacity in the national airspace by efficient management of the current allocated spectrum. Furthermore, FAA is preparing a report for Congress that covers the use of spectrum allocated for possible UAS activities. FAA officials told us that the report will not delay or prohibit the use of any licensed spectrum for UAS. FAA expects to submit its report to Congress in April 2020. Limitations to conducting counter-UAS detection and research: Counter-UAS activities involve using technology to help detect, track, and defend against illegal or unauthorized activities. Pursuant to federal law, it is illegal to damage or destroy aircraft, and this statute may apply to UAS. Other provisions of federal law may prohibit the use of certain detection systems and mitigation systems. FAA does not support the use of counter-UAS systems, which includes interdiction capabilities, by any entities other than the federal agencies with explicit statutory authority to use these technologies, including for the testing and evaluation of such systems. In addition, FAA has limited authority for testing UAS detection and mitigation systems at airports. Federal agencies with the authority to mitigate risks of UAS under certain circumstances are the Departments of Defense, Energy, Justice, and Homeland Security. According to one test site representative, industry’s ability to conduct research on counter-UAS technologies is limited because it requires the participation of one of the four agencies listed above. FAA officials told us that these federal agencies have the authority to conduct counter-UAS operations. These agency officials noted that the test sites could support counter-UAS research activities, for example, by providing the expertise and any infrastructure needed for the test flights, such as a chase aircraft. Some test site representatives and users we spoke to suggested that it would be helpful if more counter-UAS research were allowed. For example, they said that further research is needed to understand how to address counter-UAS threats—such as someone illegally trying to interfere with the radio frequency of a UAS delivering a package. One test site representative told us that multiple users want to fly swarms of UAS (where one operator flies multiple UAS simultaneously in proximity) to conduct counter-UAS operation research, but it is a challenge to support users’ desired research because of current restrictions. However, some stakeholders pointed out that the available technology for conducting such research, such as detect and avoid technology, is not developed enough yet to allow for effective research in this area. FAA regularly gathers information from the test sites in the following ways: Meeting with test site representatives: In the previously described regular meetings between FAA and test sites—monthly by teleconference and semi-annually in-person—participants share information on experiences conducting research and challenges faced. According to FAA officials, the meetings are helpful in informing the agency about the types of UAS research that users are pursuing, among other things. Representatives of all seven test sites agreed that these meetings are helpful. For example, some representatives noted that such meetings facilitate information sharing about, for example, the status of other FAA-affiliated UAS research efforts— such as UTM and the IPP—and the status of other FAA initiatives underway, such as UAS rulemakings. Collecting data from test sites: Test sites have provided several types of data to FAA since 2015, including: Entering data on flight tests into the MLS—the system that FAA established for this purpose. MLS data include details about flight tests, such as duration, whether the test involved complex operations such as beyond the operator’s line of sight, and any accidents or incidences that occurred. According to FAA officials, MLS is used for collecting test site data—which will be used to, among other things, inform the final report to Congress that is required by statute. Submitting data into FAA’s aforementioned COA application processing system, which FAA uses to process COAs. Submitting quarterly and annual reports to FAA, which summarize activities completed by each test site, including research and development efforts for users, milestones met and the key challenges faced in undertaking activities. According to FAA officials, their efforts related to the UAS test site program have been primarily focused on meeting requirements such as those related to test sites outlined in the 2012 Act. Among other things, the 2012 Act required FAA to: Establish test sites to provide a way to access airspace to conduct research and development. Develop standards and requirements for UAS flight operations at test sites. At the end of the test site pilot program, submit a final report to Congress with findings and conclusions about projects facilitated through the program. In response to the 2012 Act’s requirements, as previously noted, FAA established the test sites and developed requirements for how test sites should conduct UAS flight testing. As FAA has been focused on collaborating with the test sites and meeting the 2012 Act’s and other requirements, agency officials have not prioritized determining how to use data gathered from the sites to advance UAS integration. To date, FAA has only used data from test sites in a few cases to directly inform the agency’s UAS integration efforts. For example, in one case, FAA used data from an ASSURE project conducted at a test site to develop a noise certification standard; these data were not from MLS. In another example, FAA officials told us that— as of February 2019—they were planning to use MLS and other test site data to make a decision about an applicant that had submitted a request to conduct UAS package delivery operations. According to officials, FAA intends to use the data collected from test sites to a greater extent in the future to further integration, such as in the following ways: In November 2018, FAA asked ASSURE to review test site data to identify data FAA could use to approve safety cases. As previously noted, FAA generally requires safety cases to be submitted as part of any application to use a UAS operation that is not yet routinely allowed in the national airspace due to risk, such as flights beyond the operator’s line of sight. Safety cases include evidence of how the applicant will address any risks that the new complex UAS operation would introduce into the airspace, such as the risks of the UAS abruptly losing power. According to FAA officials, this research was initiated in December 2018 with a plan to complete it by March 2020. According to these officials, the results of this research should help the overall UAS integration effort. Specifically, the results may help FAA officials to more clearly define the information UAS operators should submit to demonstrate how the safety risks associated with their proposed operation will be mitigated. Officials indicated that FAA also intends to use MLS and other test site data to continue developing, evaluating, and validating the aforementioned UTM system. FAA officials told us that while they have not fully leveraged test site data, they are using other information from the test sites—such as information shared in meetings—to support the agency’s efforts to integrate UAS into the national airspace. According to FAA officials, the test site program supports UAS integration not only by providing industry stakeholders with an avenue for testing complex UAS operations and concepts, but also by helping FAA officials stay informed about issues related to integration. Specifically, these officials told us that the informal information sharing that occurs in regular meetings between FAA officials and test site representatives has been valuable. Through such informal exchanges, FAA officials keep abreast of the various types of research being requested by industry stakeholders and challenges faced by such stakeholders pursuing such research. For example, as noted previously, test site representatives have used these meetings to discuss challenges—such as related to dedicated spectrum—with FAA officials. In addition, based on what FAA officials have observed at test sites, the agency has been able to grant other airspace users more flexible authorizations, for example COAs covering larger geographical areas. Specifically, these agency officials told us that because they observed that the test sites have been able to maintain an acceptable level of safety after being allowed more flexibility in their aforementioned nationwide blanket COAs, the agency felt confident enough to give more flexibility to other airspace users with COAs for using complex UAS operations. FAA’s UAS integration plans specify the importance of not only collecting data but also using the data to inform strategic planning efforts. FAA’s publicly available plans state that FAA intended to use information from the test site program to inform its UAS integration efforts. Specifically, according to the 2018 UAS Integration Roadmap, the test site program plays a critical role in UAS integration as one of the program’s goals is to provide information so that FAA can determine technical and operational trends that could support safety-related decision making for integration, and develop policy and standards required to address new and novel aspects of UAS flight operations. In addition, FAA’s Unmanned Aircraft Systems Test Site Data Collection and Analysis document issued in 2016, indicates that by September 2016, FAA planned to analyze the data to determine operational trends, communicate them via dashboards, and share the collected and analyzed data with stakeholders. Further, federal internal control standards state that agencies should use quality information to achieve the agency’s objectives and support informed decisions. Specifically, agencies should first identify what data are needed to achieve the entity’s objectives, then obtain the needed data from internal and external sources in a timely manner, and finally process and evaluate the obtained data into quality information that supports the entity’s objectives. While FAA has indicated plans to analyze and use test site data in the future, it has not yet developed a data analysis plan to do so. FAA officials told us that having an analysis plan for MLS data could be useful and that—as of September 2019— they were considering creating such as plan but had not taken steps to do so. According to FAA officials and some test site representatives, and based on our review, some currently collected data could be useful in informing integration efforts. Specifically, FAA officials and two test site representatives told us that some MLS data—for example on accidents and lost control links—could be useful. For example, data on accidents and lost communication links could be combined with other MLS data on the respective test flights—such as the time of day, type of UAS being flown, and other factors—to determine whether certain conditions or UAS models are at a greater risk of a crash or other incident. According to FAA officials, this combined data could theoretically help the agency to measure risk and to determine if there are any factors that contribute to lost control links between the UAS and the remote pilot in the flight testing environment. The results of such a data analysis could help inform integration efforts, such as in developing operational standards for UAS. Without a plan for analyzing the data, FAA could miss opportunities to leverage what was intended to be a cornerstone of the test site program—information to help FAA move UAS further toward full integration into the national airspace. Having such an analysis plan could help FAA articulate how the agency will use test site data more in the future and identify other data that are within the agency’s authority to request from test sites that would help inform integration. Representatives from three test sites told us that their staff currently collects other data that FAA is not collecting but which could help to inform the agency’s UAS integration efforts. Based on our review of test sites’ annual reports to FAA, for instance, all test sites have been involved in facilitating test flights of UAS operations beyond the operator’s line of sight. FAA may be able to use data from such flight tests as it develops standards for allowing these types of UAS operations on a routine basis in the national airspace. Further, the National Academy of Sciences reported in 2018 that FAA has underutilized the test sites because it has not determined which test site data could inform the agency’s risk assessments for UAS (which FAA conducts before allowing any new complex UAS operation to be used on a routine basis) nor collected that specific data from test sites. FAA provides limited information to the public, including stakeholders and test site users, about how the research being conducted at test sites helps to inform FAA’s UAS integration efforts. FAA officials point to two main public efforts related to the test sites program: FAA’s 2018 UAS Integration Roadmap, described earlier, includes a high-level overview of how the test sites program informs the agency’s integration efforts. For example, it states that test sites provide information that FAA can use to determine technical and operational trends that could support safety-related decision making. However, it does not provide any information about, for example, how the research at test sites directly relates to FAA’s next planned phases of integration. FAA’s UAS Test Sites website is the agency’s main public outreach effort, and provides information such as links to the websites of the test sites. However, in examining the website, we found little description of how this program relates to FAA’s broader integration plans and no discussion of desired outcomes from the research under way at test sites. In contrast, the websites for two other UAS research efforts that FAA is involved in—the UTM program and DOT’s IPP— have program descriptions that include the purpose of the program, and some intended research outcomes. These two program descriptions make it relatively easy for the reader to understand how those programs fit into FAA’s broader UAS integration efforts. See figure 6, which shows the program descriptions on FAA’s respective websites for the test site program and the IPP. FAA also compiles some information on test sites that is not publicly available. For example, FAA staff annually compile information about the types of research conducted at test sites and present it in the Test Sites Fact Book, which links the information to key capabilities needed for the incremental integration of UAS into the national airspace. However, this document is only available to FAA staff and, according to officials, contains some data that test site users could deem proprietary. FAA officials told us that they also plan to submit the aforementioned final report to Congress on the test site program, which is currently due in late 2023. According to these officials, however, this report is not intended to be made public. All test site representatives and many users in our review (13 of 18) reported that publicly available information on research efforts underway at test sites is limited. Many users we spoke to (11 of 18) stated that FAA should include more information about the test sites on its website, and in FAA’s planning documents, such as the 2018 UAS Integration Roadmap. These representatives and users also told us improved FAA communication could increase the UAS stakeholders’ awareness of test sites’ capabilities, expertise, and services, and their understanding about how the program fits into FAA’s broader integration efforts. According to FAA, collaboration and cooperation across industry and government is important for UAS integration—a complex endeavor involving multiple stakeholders from different sectors. As FAA’s 2018 UAS Integration Roadmap states, given the large scale of the UAS integration effort, FAA must rely on crucial relationships across government and industry to ensure its integration efforts are harmonized and consistent. It further states that all the work needed to resolve collective challenges requires collaboration between partners at local, state, tribal, and national levels as well as with partners across the UAS stakeholder community. In addition, federal internal control standards and leading practices for reporting on research and development activities emphasize the importance of making the status of such activities transparent to stakeholders. Specifically the federal internal control standard for communicating information calls on federal agencies to externally communicate quality information so that external parties can help the entity achieve its respective objectives. Further, this standard suggests that agencies should select appropriate methods to communicate externally, taking into consideration factors such as the intended audience and the availability and ease of access to the information. In addition, as we have reported, leading practices for reporting on research and development efforts include clearly communicating the status of such efforts to the public and stakeholders. For example, in a 2017 report about FAA’s management of its aviation research and development portfolio—which includes UAS research efforts—we found that FAA could more fully adhere to leading practices if it provided more information for Congress and other stakeholders, such as on the status of various research and development activities. We noted that with more complete and transparent information, Congress and industry and other stakeholders are better able to make informed decisions. In another example, in several reports on FAA’s implementation of the Next Generation Air Transportation System— another complex endeavor involving coordination with industry and other stakeholders—we emphasized the importance of sharing information about the status of various projects with stakeholders whose participation will be essential to the progress of the overall effort. FAA officials told us that they were wary of providing more public information about the test sites, based on concerns about potentially being perceived to be promoting the designated test sites and concerns about sharing data that could be proprietary. For example, officials told us that when potential test site clients approach FAA, they simply direct these potential clients to the FAA’s UAS Test Sites website. The officials told us that they do not wish to be seen as promoting or advertising one of the FAA-designated UAS test sites over the others, because such promotion would conflict with FAA’s role as a regulator. They also said that FAA wants to avoid suggesting that operators seeking to research complex UAS operations are required to contract with a designated test site. They noted that the decision about whether or not to use a designated test site should be left to the potential client. In addition, FAA officials expressed concerns about sharing any information that the test site users could deem to be proprietary, such as information about their research projects currently underway. For example, the officials noted that some test site users do not want to be identified as such. In our assessment, however, it would be possible for FAA to share more information publicly about how the test site program fits into the agency’s broader UAS integration effort without promoting any particular test site or sharing any proprietary information. For example, some context in the Test Sites Fact Book could be informative because it links research underway at test sites to FAA’s integration plans. This book includes a section on current test site research with examples that, if shared, could help increase stakeholders’ understanding of how FAA could use the research being conducted at test sites to inform its decisions. This section indicates that test sites are involved in research aimed at, for example: Advancing UAS standardization, meaning the FAA and all the test sites working together to advance the industry from a systems perspective to develop standardized UAS training, maintenance, and safety risk mitigation. Data from such research could help inform FAA decisions such as, for example, setting standards for drone spacing and mitigating risks. Using UAS for wildfire operations, including test sites and users— such as emergency response agencies—finding effective ways to use UAS to respond to such situations. Data from such research could help FAA improve, for example, its response time when an emergency COA is requested by such agencies. Such additional information, if shared, could help FAA to clearly demonstrate to the wider audience of UAS stakeholders that the agency is fostering research through test sites that directly relates to its UAS integration plans. As noted above, the test site users we interviewed told us they were conducting research at test sites related to FAA’s upcoming phases of its UAS integration plan, including research on large cargo and passenger operations. Although some UAS stakeholders—such as users of test sites—may currently be aware of the research underway at test sites, the audience for UAS integration is larger and includes others such as those from the information technology and agricultural industries, and local government agencies whose stakeholders may be less familiar with FAA’s efforts. Further, with more accessible information on how research at the test sites relates to FAA’s UAS integration efforts, more stakeholders may choose to use a test site to conduct their own research. Given that one of the primary goals of the test site program is to provide information to FAA to help the agency develop the policies and standards required to address new and novel aspects of UAS flight operations, having more test site users could help the agency achieve this goal by making more data available to FAA. As noted previously, many selected users we interviewed told us that using a test site provided a significant benefit for advancing their entity’s UAS research and development efforts. However, some UAS stakeholders who could benefit from a test site’s assistance— such as those outside of the aviation industry seeking to submit a safety case to FAA for approval of complex UAS operations—may not currently be aware of the option for conducting research through a test site. For instance, a stakeholder interested in conducting research involving, for example, using UAS for small package delivery, may be unaware that test sites have already helped to facilitate such research for their users. FAA officials told us that stakeholders outside of the aviation industry can particularly benefit from a test site’s expertise since they may be less familiar with FAA’s processes for approving UAS operations on a case- by-case basis. All test site representatives and some users in our review told us that if FAA communicated more clearly about the role of the test site program in the overall UAS integration effort, more stakeholders would likely leverage the test sites. FAA’s designated UAS test sites provide significant benefits to the UAS industry, offering their users a variety of services, with minimal operating investment from FAA. Many users in our review told us that their decision to work with a test site proved invaluable in helping achieve their respective goals. As FAA proceeds with its plans to incrementally integrate UAS into the national airspace—a large effort requiring collaboration with many stakeholders—the agency could benefit from better leveraging all of its available resources. According to FAA, additional research and development work—including data on UAS operations—is needed to inform its decisions as it allows for more complex UAS operations to be routinely used in the national airspace. UAS stakeholders working with FAA test sites are testing complex UAS operations and various capabilities identified by FAA as needed to inform integration policies and rules moving forward. However, without a plan for analyzing the test site data, FAA could miss opportunities to better use the data to inform the overall UAS integration effort, such as by applying the data to inform UAS operational standards. Having such an analysis plan could help FAA articulate how the agency will use test site data more in the future and identify other data that are within the agency’s authority to request from test sites that would help inform integration. In addition, by sharing more information about how the program relates to FAA’s integration efforts, the broader community of UAS stakeholders may have a greater awareness of the types of research and testing being conducted at test sites and thus be better able to participate in the effort. Further, without more accessible information, such as examples of how research underway at test sites aligns with FAA’s planned phases of UAS integration, some UAS stakeholders may not be aware of their options for pursuing research through a test site, thus potentially limiting the usefulness of the test site program for UAS stakeholders and for FAA. We are making the following two recommendations to FAA: The Administrator of FAA should develop a plan for analyzing currently-collected UAS test site data to determine how they could be used to advance UAS integration, and whether the collection of any additional test site data, within the agency’s authority to request, could be useful for informing integration. (Recommendation 1) The Administrator of FAA should publicly share more information on how the test site program informs integration while continuing to protect information deemed proprietary. This information could be shared, for example, on the agency’s UAS Test Sites website. (Recommendation 2) We provided a draft of this report to DOT and NASA for their review and comment. In its written comments, reproduced in appendix II, DOT partially agreed with the first recommendation and agreed with the second recommendation. FAA also provided technical comments, which we incorporated as appropriate. NASA officials reviewed our draft, but did not have any comments. FAA partially agreed with the first recommendation to develop a plan for analyzing test site data, noting a concern about using such a plan to determine if the collection of any additional test site data could be useful for informing integration. Specifically, FAA noted that the agency cannot require test sites to share data from their privately contracted users, other than the data required for the test sites’ COAs or for their OTAs with FAA. FAA also noted a concern that our draft report incorrectly assumes that the data collected through the test site program are adequate to meet FAA’s UAS integration needs when this program is limited in the data that can be collected. However, our report states that the test site program is only one of several sources of data to inform FAA’s future decisions regarding UAS integration, and that a data analysis plan could help FAA determine whether any additional data could be useful for informing integration. To address FAA’s comments, we added language to our recommendation to clarify that the consideration of potential additional data would be for data that are within the agency’s authority to request from test sites, such as through the OTAs. We continue to believe that implementing this recommendation would enable the agency to better leverage test site research and data to inform its decisions related to UAS integration. FAA agreed with our second recommendation to share more information on how the test site program informs the agency’s UAS integration effort. However, FAA stated that the agency’s integration plans and Test Site Fact Book cannot be made publicly available due to future rulemaking and proprietary information contained in these documents. We acknowledge in our report that these documents could include information that test site users deem proprietary. We include in our recommendation that FAA should continue to protect any information deemed proprietary while making information about the test site program’s contribution to UAS integration publicly available. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or krauseh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Entity Griffiss International Airport (New York) - Northeast UAS Airspace Integration Research Alliance New Mexico State University – Physical Science Laboratory North Dakota Department of Commerce – Northern Plains UAS Test Site State of Nevada – Nevada Institute for Autonomous Systems Texas A&M University-Corpus Christi – Lone Star UAS Center of Excellence and Innovation University of Alaska Fairbanks – Alaska Center for UAS Integration Virginia Polytechnic Institute and State University – Mid-Atlantic Aviation Partnership Agricultural Research Service, United States Department of Agriculture Alliance for System Safety of UAS through Research Excellence, Mississippi State University Association for Unmanned Vehicle Systems International JHW Unmanned Solutions, LLC Massachusetts Institute of Technology Lincoln Lab National Emergency Response and Recovery Training Center, Texas A&M Engineering Extension Service Project Vahana, Airbus A3 Vanilla Aircraft (now Vanilla Unmanned) Heather Krause at (202) 512-2834 or krauseh@gao.gov. In addition to the individual named above, Vashun Cole (Assistant Director); Jessica Bryant-Bertail (Analyst-in-Charge); Jon Felbinger; Camilo Flores; Richard Hung; Josh Ormond; Amy Rosewarne; Alexandra Rouse; Kelly Rubin; Marc Schwartz; and Larry Thomas made key contributions to this report.", "summary": "UAS could provide significant economic and social benefits, for example by delivering packages or aiding in search and rescue missions. FAA is conducting a phased approach to incrementally integrate UAS safely into the national airspace. As directed by statute, FAA established UAS test sites to allow industry to assess the safety and feasibility of complex UAS operations, such as flying beyond an operator's line of sight. FAA has stated that this program provides research results and other data needed to reach full integration. GAO was asked to review FAA's management of the test sites. This report examines, among other things: (1) the research conducted at FAA's designated UAS test sites, and (2) how FAA is leveraging and sharing information from the test site program to advance integration. GAO reviewed relevant statutes and regulations, reports, and FAA guidance; analyzed test sites' efforts, including flight test data submitted to FAA from 2015 through 2018; and interviewed FAA officials, test site representatives from all 7 test sites, and 18 test site users, selected to include a range of perspectives. The Federal Aviation Administration's (FAA) seven designated test sites for unmanned aircraft systems (UAS) have facilitated about 15,000 UAS flight tests since 2015 and supported a wide range of research. Both public and private entities have used the test sites to test technologies in preparation for varied UAS activities, from inspecting utilities to carrying passengers. Research conducted at test sites provides data on the performance of various UAS capabilities and technologies; such data could support FAA's integration efforts. While FAA collects this data from test sites, it has not fully leveraged the data or the program to advance UAS integration. According to FAA's 2018 Roadmap for UAS Integration a key goal of this program is to provide data to support FAA's decisions on drone integration. FAA officials said the agency intends to use the data to a greater extent in the future to advance integration. Without an analysis plan, however, FAA could miss opportunities to better use the data to inform the overall integration effort, such as to inform UAS operational standards. Also, FAA reports limited public information about how test sites' research relates to the agency's integration plans. Agency officials told GAO they were wary of sharing more information about the test sites, citing concerns about, among other things, protecting test site users' proprietary data. All test site representatives and most users GAO interviewed, however, said that more information on test sites' research would be helpful for UAS stakeholders' research efforts. According to FAA plans, the agency must rely on relationships with stakeholders across government and industry to ensure that integration efforts are harmonized. By sharing more information publicly, FAA could demonstrate to such stakeholders how the agency is fostering and using research to inform and advance integration. Further, with more information, more stakeholders may opt to use a test site to conduct their own research, thus potentially increasing data available to FAA to inform its integration decisions. GAO recommends that FAA (1) develop a data analysis plan for test site data and 2) share more information on how this program informs integration, while protecting proprietary data. FAA partially agreed with the first recommendation and agreed with the second. GAO added language to the first recommendation to address the issue that FAA raised, as discussed in this report.", "document_type": "gao"}
{"report": "The U.S. government supports various types of democracy assistance, which State and USAID categorize under their democracy, human rights, and governance portfolios. State and USAID use the Updated Foreign Assistance Standardized Program Structure and Definitions to categorize democracy assistance activities in six program areas: rule of law, good governance, political competition and consensus building, civil society, independent media and free flow of information, and human rights. Table 1 shows these six program areas and their elements. State bureaus and offices—in particular, DRL and INL—and USAID provide funding for democracy assistance. State. State’s democracy assistance is provided by DRL, INL, and other State bureaus and offices. DRL. As the U.S. government’s primary foreign policy entity advocating for democracy globally, DRL funds programs in every region of the world to promote human rights, democracy, and transparent and accountable governance. INL. INL provides funding for programs that combat crime and narcotics trafficking, including democracy assistance to promote the rule of law, combat corruption, and promote good governance. Other bureaus and offices. Other State bureaus and offices, such as the regional bureaus and the Bureau of International Organization Affairs, provide democracy assistance related to their geographic or functional areas. USAID. As the lead U.S. government agency for international development, USAID considers democracy, human rights, and governance to be central to its core mission. USAID missions overseas play a primary role in providing democracy assistance, and the regional bureaus in Washington, D.C., provide oversight of this assistance. USAID’s Bureau for Democracy, Conflict, and Humanitarian Assistance, headquartered in Washington, D.C., consists of several offices, including two that support the bureau’s mission to promote democratic and resilient societies: the Center of Excellence on Democracy, Human Rights, and Governance and the Office of Transition Initiatives. State and USAID allocated a total of more than $8.8 billion for democracy assistance in fiscal years 2015 through 2018. State allocated 33 percent of this amount—a total of $2.9 billion, averaging approximately $727 million annually—to DRL, INL, and other bureaus to provide democracy assistance. USAID allocated the remaining 67 percent—$5.9 billion, averaging approximately $1.5 billion annually. Figure 1 shows the total amounts that State and USAID allocated for democracy assistance in fiscal years 2015 through 2018. In fiscal years 2015 through 2018, DRL, INL, and USAID directed allocations for democracy assistance to many of the same countries, although the program areas they supported varied. DRL, INL, and USAID directed democracy assistance allocations to a combined total of 100 countries, including 33 countries where all three entities provided such assistance (see fig. 2). DRL directed democracy assistance allocations to 67 countries; INL, to 45 countries; and USAID, to 84 countries. State officials said that, because the countries have serious democracy-related challenges, the agencies providing this assistance may address these challenges from different perspectives and with different objectives. Although DRL and USAID directed democracy assistance allocations to many of the same countries, DRL focused a greater percentage of its funding in countries where citizens enjoy fewer democratic freedoms. DRL directed 70 percent of its allocations for democracy assistance in fiscal years 2015 through 2018 to less democratic countries—those rated as “not free” by Freedom House’s 2018 “Freedom in the World” survey. In contrast, USAID directed about half of its allocations for democracy assistance during this period to “not free” countries. Similarly, although DRL, INL, and USAID directed their allocations for democracy assistance to many of the same countries, the entities concentrated funding in different program areas. In fiscal years 2017 and 2018, DRL and INL directed the largest percentages of democracy assistance allocations to encouraging human rights and promoting the rule of law, respectively, while USAID directed about half of its democracy assistance allocations to promoting good governance (see fig. 3). As figure 3 shows: DRL directed 36 percent (about $203 million) of democracy assistance allocations to projects supporting human rights, 19 percent (about $107 million) to projects supporting civil society, and 14 percent (about $76.4 million) to projects supporting independent media and free flow of information. DRL directed the smallest amounts to projects supporting rule of law, political competition and consensus building, and good governance. INL directed more than 98 percent (about $580 million) of democracy assistance allocations to promote the rule of law. USAID directed 49 percent (about $1.5 billion) of its democracy assistance allocations to projects promoting good governance and 19 percent (about $600 million) to projects supporting civil society. USAID distributed the remainder across the other four democracy assistance program areas, allocating the smallest amounts to projects supporting human rights and independent media and free flow of information. State’s DRL and INL and USAID have strategies that define their roles in democracy assistance, and their funding obligations in the selected countries in fiscal years 2015 through 2018 generally aligned with these roles. DRL and INL strategies identify various program areas as aspects of the bureaus’ respective roles in providing democracy assistance. For example, DRL supports a range of democracy program areas and emphasizes human rights, while INL focuses on the rule of law. In fiscal years 2015 through 2018, DRL’s and INL’s funding obligations for democracy assistance in the countries we selected for our review—the DRC, Nigeria, Tunisia, and Ukraine—generally aligned with the roles defined in bureau strategies and described by bureau officials. USAID plays the leading role in U.S. development assistance overseas, including democracy assistance, according to its 2013 strategy on democracy, human rights, and governance. We found that USAID’s democracy assistance in the four selected countries generally aligned with its strategic goal of supporting democratic change to achieve broader development goals. DRL’s 2018 bureau strategy states that the bureau’s mission is to “champion American ideals as a means of combating the spread of authoritarianism, terrorism, and subversion of sovereign democracies.” According to the strategy, DRL works through diplomatic channels to support democracy-related areas; support human rights, labor, and democracy defenders; and publish reports on human rights in all countries, among other activities. In a 2015 report to Congress, State noted that 90 percent of DRL’s programs operate in restrictive or challenging environments. Although the report did not define restrictive or challenging environments, DRL officials said that the bureau’s assistance focuses on building civil society and supporting diplomatic initiatives to improve governance, particularly in repressive and closed societies. According to the officials, the bureau supports democracy and human rights globally, including in areas where such programs face threats from host governments, and is not constrained to working in countries with a U.S. presence. DRL designs and manages all of its democracy assistance projects from Washington, D.C. DRL officials noted that DRL projects typically receive total allocations of at least $500,000, have a duration of 1 to 5 years, and are implemented by U.S.- based or other large organizations. INL’s most recent bureau strategy states that INL is at the forefront of responding to international security challenges and that INL promotes U.S. leadership by advancing rule-of-law principles. INL officials said that the bureau conducts democracy assistance work to support its provision of security assistance and that INL programming helps governments provide accountability to their citizens. According to agency officials, INL’s funding for democracy assistance generally supports host-country governments through bilateral agreements and is not always project based. INL programs can be managed by INL staff at State’s headquarters in Washington, D.C., and at embassies overseas. INL’s democracy assistance is implemented by its own staff, other U.S. agencies, and U.S.-based or international organizations. DRL and INL officials told us that they ensure consistency between their democracy-related strategic goals and the goals in overarching strategies, such as the government-wide National Security Strategy and State and USAID’s Joint Strategic Plan. The most recent Joint Strategic Plan notes that State and USAID will work to “counter instability, transnational crime, and violence that threaten U.S. interests by strengthening citizen-responsive governance, security, democracy, human rights, and the rule of law.” The Joint Strategic Plan also notes that State and USAID will focus on places that pose the greatest threat to U.S. interests. DRL’s and INL’s total obligations of funding for democracy assistance in the four selected countries for fiscal years 2015 through 2018 generally reflected their defined roles. DRL’s obligations for projects in the selected countries generally reflected the bureau’s focus on supporting democracy and human rights, as defined in DRL’s bureau strategy and described by officials. Overall, the majority of DRL obligations in the four selected countries focused on projects supporting civil society, human rights, and independent media and the free flow of information. In fiscal years 2015 through 2018, 60 to 100 percent of project-level funding was dedicated to these program areas. DRL obligations for democracy assistance projects in the selected countries averaged more than $800,000 for 2 years. Consistent with its stated role of protecting human rights globally, DRL obligated at least a quarter of this funding in three of the four countries to projects that supported human rights (see fig. 4). Similarly, INL’s democracy assistance obligations in the selected countries during the same period generally reflected the bureau’s focus on supporting the rule of law, as defined in its bureau strategy and described by officials. Data for the four countries show that INL obligated $3.2 million in the DRC, $12.5 million in Nigeria, $$3.9 million in Tunisia, and $5 million in Ukraine for democracy assistance for fiscal years 2015 through 2018. In Nigeria, Tunisia, and Ukraine, 100 percent of INL’s democracy-related obligations supported the rule of law. In the DRC, 92 percent of INL’s democracy-related obligations supported the rule of law and the remaining 8 percent supported good governance. (See apps. III through VI for more information on State’s democracy assistance in the DRC, Nigeria, Tunisia, and Ukraine, respectively.) The 2013 USAID Strategy on Democracy, Human Rights and Governance states that USAID plays the leading role in U.S. development assistance overseas, including democracy assistance. The strategy explains that support for democracy, human rights, and governance is essential to achieving the agency’s broader social and economic development goals, which, USAID has noted, contribute to self-reliance. USAID officials told us that, to support democracy from a development perspective, USAID generally funds multiyear, multimillion-dollar democracy assistance projects that are implemented by U.S.-based or international organizations. USAID’s democracy strategy also identifies the roles of various USAID units involved in implementing U.S. democracy assistance. For example, according to the strategy, USAID missions are to play the primary role in implementing it by both designing and managing democracy-focused programs, while USAID’s Center of Excellence on Democracy, Human Rights, and Governance is to provide technical and other assistance to the missions and manage some mechanisms to support programs, among other things. Further, the strategy clarifies relationships in terms of leading and supporting units in areas of democracy assistance and identifies roles of various other agencies, including State. In all four selected countries, USAID’s democracy assistance, as reflected in country-level strategies and projects, generally aligned with the Joint Strategic Plan and with the agency’s democracy strategy to support democratic change in order to achieve broader development goals. We found that the USAID country development cooperation strategy for each of the selected countries articulated democracy assistance objectives to support the country’s overall development. According to USAID officials, these strategies guide the type of democracy assistance provided in a particular country on the basis of the country’s needs and generally focus on supporting sectoral change, such as through policy reform or institution building. For example, the 2016 USAID strategy for Tunisia included a development objective to promote social cohesion through democratic consolidation. Objectives for selected USAID projects in the four countries also reflected the agency’s goal of effecting long-term, development-based change through democracy assistance. For instance, consistent with its country strategy for Tunisia, USAID obligated nearly $22 million in fiscal years 2017 and 2018 for a project designed to improve the relationship between Tunisians and their civic and government institutions, in part by enhancing the responsiveness of government institutions (see fig. 5). Other characteristics of USAID’s democracy assistance projects in the selected countries also reflected the agency’s defined role. In each of the four countries, a democracy office in USAID’s mission in the country managed democracy assistance, consistent with USAID’s democracy strategy. Overall, USAID’s democracy assistance projects in the selected countries demonstrated that the agency implemented multiyear, multimillion-dollar projects, consistent with what USAID officials told us was needed to support long-term development. Data for the four countries showed that USAID’s total obligations for democracy assistance ranged from $49.5 million to $126 million for fiscal years 2015 through 2018 (see fig. 6). Per project, USAID’s obligations in the four countries averaged about $7.2 million, with each project’s implementation period averaging just over 4 years. USAID’s implementing partners were, for the most part, U.S.-based or international organizations. Although USAID democracy assistance obligations in the selected countries covered a variety of program areas, they concentrated on political competition and consensus building, good governance, and civil society. As figure 6 shows, USAID’s obligations for rule-of-law and human rights projects made up less than a quarter of total project-level funding obligated in each country in fiscal years 2015 through 2018. See appendixes III through VI for more information about USAID’s democracy assistance projects in the DRC, Nigeria, Tunisia, and Ukraine, respectively. State and USAID use various mechanisms to coordinate democracy assistance at the headquarters level, such as interagency roundtable discussions of budget allocations. Officials at embassies in the selected countries described interagency coordination efforts at the country level, such as working groups, and provided examples of how coordination helped avoid duplication and improved the effectiveness of democracy assistance efforts. Despite the use of these mechanisms and other steps that DRL takes to coordinate with embassies, embassy officials in all four selected countries reported having incomplete information about DRL’s projects in those countries. State and USAID use various mechanisms, including budget roundtables and proposal review panels, to coordinate democracy assistance between the agencies at headquarters. For instance, State’s Office of U.S. Foreign Assistance Resources manages the annual allocations budget process, which facilitates interagency coordination through structured conversations about democracy assistance and various bureaus’ priorities, according to State and USAID officials. These annual democracy discussions also enable the participants to identify policy changes and share lessons learned. USAID officials added that USAID’s Center of Excellence on Democracy, Human Rights, and Governance serves as the technical lead on democracy assistance issues during these interagency budget discussions. INL officials told us that they take the lead in democracy assistance discussions concerning security sector assistance. In addition, some of State’s regional bureaus, including the Bureaus of Near Eastern Affairs and of European and Eurasian Affairs, maintain assistance coordination offices to coordinate U.S. foreign assistance to countries in those regions, including through strategic planning and budget formulation processes. These offices, based in Washington, D.C., coordinate with embassies, other State bureaus, and USAID at various stages of strategic planning and budget formulation. For example, country coordinators from the Bureau of Near Eastern Affairs’ assistance coordination office are to lead roundtable discussions at least annually to share information among U.S. government agencies and contribute to improved planning and implementation. Some U.S. embassies in these regions, including those in Tunisia and Ukraine, have an assistance coordination unit to coordinate all U.S. foreign assistance in the country, and these units work with State regional bureaus’ Washington, D.C.– based offices. Further, when considering potential democracy assistance projects, DRL coordinates with State and USAID counterparts both in Washington, D.C., and overseas through its proposal review process. DRL proposal review panels include representatives from USAID, State regional bureaus, and other agencies that may have relevant expertise. State and USAID also use various interagency mechanisms to coordinate democracy assistance at the country level within embassies overseas. Examples of coordination mechanisms include the following. Working groups. According to State and USAID officials in the four selected countries, interagency working groups facilitate formal discussions about democracy assistance projects and provide opportunities to identify areas where agencies’ projects might complement or duplicate one another. Working groups at each embassy vary in number, theme, and meeting frequency, depending on the country context and U.S. government priorities. For example, the U.S. embassy in Ukraine has about 10 democracy-related working groups, focused on themes including elections, anticorruption, human rights, and the justice sector. At the U.S. embassies in the DRC and Nigeria, agency officials told us they convened working groups on elections, given the U.S. government’s interest in the countries’ recent and upcoming elections. In Tunisia, where USAID reestablished a presence in 2012 and a mission in June 2019, an interagency development assistance working group that addresses democracy issues, among other things, began meeting in September 2018, according to agency officials. The officials also said that a security assistance working group coordinated assistance related to rule-of- law issues. These working groups meet bimonthly, monthly, or weekly, according to officials. State and USAID officials generally said that they found the working groups were effective in helping to coordinate democracy assistance. Assistance coordination units. U.S. embassies in Tunisia and Ukraine have assistance coordination units designed to coordinate U.S. foreign assistance, including democracy assistance. Unlike the assistance coordination unit in Ukraine, State’s Foreign Assistance Unit in Tunisia managed democracy assistance projects in fiscal years 2015 through 2018 while also coordinating other State and USAID assistance in the country (see app. V for more information about democracy assistance in Tunisia during this period). According to a State document, the assistance coordinator at an embassy in Europe or Eurasia can be a “touch point” for agencies at the embassy to work together on assistance issues and communicate effectively with Washington. The assistance coordination units in both Tunisia and Ukraine have established mechanisms to coordinate U.S. foreign assistance within the embassies, according to officials. For instance, the foreign assistance unit in Tunisia formalized a process by which the ambassador’s office approves all State and USAID assistance projects in the country. Additionally, in both countries, the assistance coordinator participates in working groups and is involved in the design or review of all assistance projects, according to officials. USAID and State officials in these countries expressed varying opinions about the units’ usefulness for coordination. State and USAID officials in the selected countries provided the following additional examples of coordination that, according to the officials, helped avoid duplication and improved the effectiveness of democracy assistance efforts. According to State and USAID, informal coordination and information sharing among agency officials at the embassies occur during regularly scheduled meetings, such as weekly meetings of USAID staff, State’s political unit staff, or embassy senior staff, and through daily interaction. State has developed a tool kit to help embassies with strategic planning, including the development of action plans to document units’ roles. For example, agencies at the U.S. embassy in Nigeria created an action plan that identified the various units supporting assistance for elections to help prevent duplication of efforts. (Fig. 7 shows citizens participating in Nigeria’s elections.) State and USAID officials at embassies described other coordination of the agencies’ democracy assistance. For example, in Nigeria, USAID does not fund any rule-of-law projects because, according to USAID officials, they and INL officials have decided on a clear division of labor: INL manages all rule-of-law projects, including judicial strengthening, judicial reforms, and anticorruption, while USAID manages all other aspects of democracy assistance. In Ukraine, USAID and INL developed a concept paper to guide their collaboration to help the government establish the country’s High Anti- Corruption Court. The concept paper outlined the key roles of USAID and INL and designed complementary projects based on each agency’s strengths. For example, USAID was responsible for developing training programs for judges and INL was responsible for vetting potential judges. Officials told us that this concept paper helped agencies maximize the potential impact of their limited resources. Although DRL takes steps to coordinate with embassies in countries where it funds democracy assistance projects, embassy officials in all four selected countries reported having incomplete information about DRL’s projects in those countries. DRL has various practices and processes to coordinate with embassies. For example, DRL established a standard operating procedure to clarify methods for coordination between itself and State’s regional bureaus, which includes defined steps on engaging with embassies. The procedure outlines steps in DRL’s annual planning process, during which priorities and program strategies are set; in the process for submitting proposed projects and awards; and in the process for proposal review panels. DRL officials in Washington, D.C., also pointed to various methods that they use to coordinate with embassies. Such methods include distributing a description of DRL’s projects by country on an annual basis, training new Foreign Service officers in DRL’s funding mechanisms and awards process, and providing contact information for DRL staff at headquarters to embassy personnel. Additionally, DRL officials said that embassy officials have at least four opportunities to provide official input during the approximately 18-month process of designing and awarding a project. According to DRL officials, embassy personnel designated as human rights officers serve as DRL’s overseas points of contact. However, at the embassies in all four countries, human rights officers or other officials from the political units told us that they were not actively engaged in DRL’s projects and generally lacked updated information about DRL projects in their countries, including descriptions and funding amounts. Embassy officials also said that, although DRL sought their input during the process of selecting proposed democracy assistance projects, DRL did not subsequently communicate its final selection of projects. DRL officials said that sharing complete information can be difficult because of the sensitivity of some DRL projects and the need to safeguard the identities of some local partners. In addition, DRL officials said that managing projects from Washington, D.C., instead of overseas may affect their ability to collaborate with embassy officials. DRL officials commented that embassy personnel’s colocation facilitates their collaborating with one another and that the political and other State officers who may function as in-country DRL points of contact have numerous other duties, with limited capacity to focus on DRL projects. DRL officials also said that frequent turnover among State personnel makes it challenging to maintain embassy officials’ awareness of DRL’s in-country projects. In addition, they said that DRL is sometimes unaware of democracy assistance projects that embassies may be funding. Moreover, we found that existing information-sharing mechanisms, including data systems and strategies, do not consistently address embassy personnel’s information gaps. DRL and other State officials said that embassy personnel may not be able to use State’s data systems to retrieve information on projects, partly because some personnel lack sufficient training or the permissions to access project data in certain systems. Furthermore, the Office of Management and Budget has found the quality of State’s publicly reported data to be low in terms of completeness and accuracy. State’s Office of Inspector General found that, while State has standardized and centralized its foreign assistance budget planning and request processes, State’s inability to provide authoritative foreign assistance financial information is a program management challenge. In addition, the integrated country strategies for the four selected countries for fiscal years 2015 through 2018 do not mention DRL’s projects or general goals when discussing U.S. government democracy-related objectives for each country. Overseas officials’ lack of complete information about DRL’s projects could lead to potential duplication in U.S. democracy assistance and may inhibit State’s efforts to coordinate with other agencies, implementing partners, and other donors. We have previously found that it is helpful when participants in a collaborative effort have full knowledge about the relevant resources available and have the appropriate knowledge, skills, and abilities to contribute. Since 2015, Congress has made available to agencies at least $2 billion annually for democracy assistance programs abroad. State’s DRL and INL, as well as USAID, have articulated their roles in democracy assistance through strategies that include specific democracy-related goals. Although State and USAID use various mechanisms to coordinate democracy assistance at headquarters and in the field, we found that relevant embassy officials in each of the four selected countries did not have ready access to information about DRL projects. As a result, embassy officials lacked an understanding of the full scope of U.S. democracy assistance in their countries. Ensuring access to information about DRL projects could improve State’s overseas coordination, both internally and with other U.S. agencies, implementing partners, and donors, as well as State’s ability to achieve important democracy assistance goals. The Secretary of State should direct the Assistant Secretary of State for Democracy, Human Rights, and Labor to develop a mechanism to facilitate the active sharing of information about democracy assistance projects between DRL and relevant staff at embassies. We provided a draft of this report to State, USAID, and NED for their review and comment. In its written comments, reproduced in appendix VII, State agreed with our recommendation and noted steps that it plans to take to implement it. USAID also provided written comments, which are reproduced in appendix VIII, as well as technical comments that we incorporated as appropriate. NED officials reviewed our draft but did not provide any comments. We are sending copies of this report to the appropriate congressional committees and to the Secretary of State, the Administrator of USAID, the President of NED, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IX. The National Endowment for Democracy (NED) is a private, nonprofit, nongovernmental organization based in Washington, D.C., whose stated purpose is to encourage democracy throughout the world by supporting nongovernmental organizations and actors that are working for democratic goals. NED is funded through a grant from the Department of State (State) pursuant to an annual congressional appropriation and receives additional funding from State to support congressionally directed or discretionary programs. In addition to providing grants to local organizations in other countries, NED provides grants to its four affiliated organizations known as the “core institutes”: the Center for International Private Enterprise, the International Republican Institute, the National Democratic Institute, and the Solidarity Center. In fiscal years 2015 through 2018, NED allocated a total of about $541 million for democracy assistance projects in 100 countries— approximately $114 million in fiscal year 2015, $141 million in fiscal year 2016, $144 million in fiscal year 2017, and $142 million in fiscal year 2018. During this period, NED directed 55 percent of its funding for local organizations to groups in countries rated “not free” by Freedom House’s 2018 “Freedom in the World” survey. Figure 8 shows the countries where NED allocated funding for democracy assistance in fiscal years 2015 through 2018. As figure 9 shows, in fiscal years 2017 and 2018, NED directed funding to projects in six democracy assistance program areas. NED allocated the largest amount during that period—about $100 million (36 percent)—to promote good governance and allocated the next largest amount—about $72.5 million (26 percent)—to promote political competition and consensus building. NED allocated the smallest amount—about $8.5 million (3 percent)—to support the rule of law. According to NED’s 2012 strategy, the organization focuses on providing grants to grassroots activists in response to local needs and “seeks out newly-emerging groups in both democratizing and authoritarian countries around the world, helping to empower the most effective grassroots activists.” The strategy notes that NED is guided by its founding legislation, which established NED as an independent institution whose mission is to promote democracy through grants to nongovernmental organizations. These include the core institutes, whose key roles NED’s strategy also defines. NED officials said that the organization focuses on building the institutional capacity of local civil society organizations, which contributes to building democratic societies. Such capacity building can include institutional support, including funding for basic functions such as operational costs, and management assistance such as budget training, which other donors tend not to provide. NED officials commented that the organization is “demand driven” and responds to funding requests for projects proposed by nongovernmental organizations. According to NED documents, it supports approximately 1,500 organizations in 90 countries with grants averaging $50,000. NED officials noted other elements that distinguish NED’s support from that of U.S. agencies, including continuity in its staff composition; the significant linguistic ability of its staff, enabling close ties with local organizations in other countries; and the relative stability of its mission and priorities, which facilitates long-term engagement on countries’ democratic issues. In addition, NED’s nongovernmental status allows it to provide democracy assistance in difficult environments, where, according to NED officials, staff of local grantees face risks as a result of their work in challenging the government and status quo. The officials said that such risks range from detention and harassment to being killed or “disappeared.” NED’s democracy assistance projects in the countries we selected for our review—the Democratic Republic of the Congo (DRC), Nigeria, Tunisia, and Ukraine—generally aligned with the organization’s strategy of supporting democracy by providing funds for indigenous civil society organizations. (Fig. 10 shows examples of NED’s democracy assistance projects in the DRC and Ukraine.) Consistent with NED’s strategy of providing grants to grassroots activists, data for projects in the four selected countries show that NED provided grants primarily to local civil society organizations in addition to its core institutes. NED grants to civil society organizations in the selected countries averaged approximately $46,000 for year-long projects, and NED renewed support for nearly all organizations on an annual basis, reflecting the long-term support that officials said was necessary to strengthen civil society. Grantees in the DRC told us that NED worked closely with local partners to identify needs and design programs and that this helped to build the partners’ organizational capacity. Consistent with NED’s mission to support democracy in general, grantees in the selected countries worked on projects that included all democracy assistance program areas. NED primarily supported projects to promote political competition and consensus building and good governance, obligating an average of 40 percent and 36 percent of its funding for these two program areas, respectively, across the four countries (see fig. 11). NED’s country priorities are articulated in country summaries that it updates each year on the basis of each country’s political context and democratic challenges. For example, NED’s 2018 Tunisia summary included a priority of supporting civil society to promote effective, democratic governance and advocate for the transparency and accountability of public institutions. The NED project that we reviewed in Tunisia aimed to “enhance the capacity of civil society to advocate for transparency, good governance, and promote social accountability in the six southern governorates of Tunisia.” See appendixes III through VI for more information about NED’s democracy assistance projects in the DRC, Nigeria, Tunisia, and Ukraine. NED’s annual planning documents, which generally outline objectives for each country where NED provides funding, include some statements about coordination and collaboration with other donors. NED officials said that NED senior leaders typically have standing relationships with senior leaders at State’s Bureau of Democracy, Human Rights, and Labor (DRL) because NED receives funding from DRL for particular countries. NED officials also told us that the U.S. Agency for International Development (USAID) has reached out to them to strategically coordinate, although NED does not receive funds from USAID. NED officials added that coordination and collaboration on specific countries largely occur between officials at the regional and country levels. For example, officials said that NED consults with counterparts at State and USAID in the regional bureaus and DRL and shares its list of grantees with DRL. Furthermore, officials said that NED is aware of funding that its grantees receive from State or USAID, because NED obtains information from potential grantees about other funding sources during the grant proposal process. According to NED, State, and USAID officials, additional collaboration occurs between headquarters and overseas officials. NED, which does not have staff overseas, manages its grants in Washington, D.C., but collaborates with overseas counterparts. NED, State, and USAID officials told us that when NED officials conduct site visits, which occur at least annually, they often meet with State and USAID officials at embassies to share information. This report examines (1) the Department of State’s (State) and the U.S. Agency for International Development’s (USAID) allocations of funding for democracy assistance in fiscal years 2015 through 2018, (2) State’s and USAID’s roles in providing democracy assistance and the extent to which their projects in selected countries during this period were consistent with defined roles, and (3) the extent to which State and USAID coordinate in providing democracy assistance. In addition, appendix I provides information about the National Endowment for Democracy’s (NED) democracy assistance allocations, role, and coordination. To examine aspects of State’s, USAID’s, and NED’s democracy assistance roles and coordination efforts, we selected a nongeneralizable sample of four countries—the Democratic Republic of the Congo (DRC), Nigeria, Tunisia, and Ukraine—where the three entities provided democracy assistance in fiscal years 2015 through 2018. In selecting these countries as illustrative examples, we considered the following factors, among others: (1) countries to which all three entities allocated or obligated democracy assistance funding in fiscal years 2015 through 2017, the most recent period for which data were available; (2) democracy assistance allocation amounts that were in the top quartile for each entity for the same period for USAID and State, according to data from State’s Office of U.S. Foreign Assistance, and for NED; (3) democracy assistance obligation amounts that were in the top half of such obligations for the same period for State’s Bureau of Democracy, Human Rights, and Labor (DRL), according to data from USAID’s Foreign Aid Explorer; (4) democracy assistance obligations data that confirmed the presence of the Bureau of International Narcotics and Law Enforcement Affairs (INL) in those countries for the same period; (5) geographical dispersion of the countries; (6) ratings that countries received from Freedom House’s 2018 “Freedom in the World” survey; and (7) suggestions from State, USAID, and NED officials as well as others with relevant expertise. We excluded countries where we had recently reviewed U.S. democracy assistance for other reports. We traveled to the DRC in May 2019, where we met with officials from State, USAID, nongovernmental organizations that had implemented U.S.-funded democracy assistance projects, and the United Kingdom’s Department of Foreign and International Development regarding its coordination with U.S. agencies. We conducted interviews with State and USAID officials who were knowledgeable about democracy assistance, interviewing officials at the embassies in Nigeria, Tunisia, and Ukraine by phone and interviewing officials in Washington, D.C., in person. To examine allocations for democracy assistance, we analyzed State, USAID, and NED global democracy assistance data for fiscal years 2015 through 2018, including the total allocations, the allocations for specific program areas, and the countries for which funding was allocated. We used the six democracy assistance program areas included in USAID’s and State’s Updated Foreign Assistance Standardized Program Structure and Definitions—rule of law, good governance, political competition and consensus building, civil society, independent media and free flow of information, and human rights. Because NED categorizes its democracy assistance using its own program definitions, we cross-referenced NED’s democracy assistance awards with the U.S. government’s six program areas, using information that NED provided. We assessed the reliability of State’s, USAID’s, and NED’s data and determined the data to be sufficiently reliable for reporting the total amount of democracy assistance allocated by each entity as well as the program areas and countries for which the funding was allocated. We also compared funding allocations with the country’s ratings in Freedom House’s 2018 “Freedom in the World” survey to determine the amount of funding that the entities allocated to countries rated as free, partly free, or not free. To identify State’s, USAID’s, and NED’s roles in providing democracy assistance and the extent to which their projects in the selected countries were consistent with their defined roles, we reviewed documents, assessed information on democracy assistance projects, and interviewed officials. While State’s regional bureaus provide some democracy assistance, we focused on State’s democracy assistance roles and projects for DRL and INL, both of which State has identified as leading the provision of its democracy assistance. See appendixes III through VI for regional bureaus’ obligations data for the four selected countries. We reviewed State’s and USAID’s Joint Strategic Plan, FY2018-2022; functional bureau strategies for DRL and INL; the 2013 USAID Strategy on Democracy, Human Rights, and Governance; integrated country strategies and country development cooperation strategies for the four selected countries; and NED’s 2012 Strategy Document. We also reviewed other documents that described aspects of State’s and USAID’s roles, including agencies’ democracy-related reports to Congress and standard operating procedures. We assessed these documents for clarity of roles and responsibilities, based on leading collaboration practices that we have previously identified, and we reviewed agencies’ overarching goals related to democracy and governance. We reviewed information about State, USAID, and NED democracy assistance projects in the DRC, Nigeria, Tunisia, and Ukraine. We reviewed project documents, including award agreements, for selected State, USAID, and NED projects that supported a variety of democracy program areas, among other factors. We assessed State, USAID, and NED obligations data for projects that they funded in the selected countries in fiscal years 2015 through 2018. We determined that these data were sufficiently reliable for reporting the total obligations, by entity and country, for fiscal years 2015 through 2018 and for reporting types of democracy assistance. We also determined these data to be sufficiently reliable for reporting the number of active projects during this time period; the average award amount or average annualized award amount; and the average duration of projects for DRL, USAID, and NED. Because INL’s democracy assistance generally supports the host government through bilateral agreements and is not always project based, we were unable to report these project characteristics for INL. In prior work, we have recommended that State identify and address factors that affect the reliability of INL’s democracy assistance data. State reported that as of July 2019, INL was continuing efforts to improve data reliability; however, because of missing data, we determined that data for INL democracy assistance in the selected countries were unreliable for reporting project characteristics. We also determined that because of missing data, such as project end dates, the data from State’s Bureau of African Affairs were unreliable for reporting some project information for Nigeria; however, the bureau’s project data for the DRC were sufficiently reliable for reporting on democracy assistance and obligations in that country. In addition, we determined the data from the Bureaus of European and Eurasian Affairs and Near Eastern Affairs were sufficiently reliable for reporting on State’s democracy assistance obligations and projects in Ukraine and Tunisia. We interviewed officials in Washington, D.C., and in the four selected countries regarding State’s, USAID’s, and NED’s roles defined in strategies and other documents and regarding democracy assistance projects. In addition, we interviewed agency officials regarding democracy assistance program areas; implementation methods (such as managing programs from headquarters or overseas as well as types of implementing partners); and other features, including typical scale of project funding. To examine the extent to which the agencies coordinated their democracy assistance, we reviewed relevant documents, such as State’s and USAID’s standard operating procedures, to identify the agencies’ mechanisms and practices for coordinating democracy assistance. We drew on our prior work identifying key practices that can enhance and sustain collaboration at federal agencies. We interviewed officials in Washington, D.C., and in the four selected countries to describe any mechanisms that agencies use to coordinate democracy assistance. We conducted this performance audit from September 2018 to January 2020, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Democratic Republic of the Congo (DRC) has experienced more than 2 decades of violence and war, exacerbated by the failure of President Joseph Kabila to hold elections when his term ended in 2016. In this context, the U.S. government’s key policy priority was to encourage the DRC’s government to support credible and peaceful elections in December 2018, according to the Department of State (State). U.S. government democracy assistance projects aimed to build the capacity of the DRC government, political parties, civil society, armed forces, civilian law enforcement, and justice systems to support credible elections and improve governance. (Fig. 12 shows examples of U.S.- funded government assistance to support the DRC’s 2018 elections.) Other U.S. government democracy-related priorities included promoting the rule of law and fighting corruption. The National Endowment for Democracy’s (NED) 2018 country summary for the DRC noted that NED should support DRC civil society’s ability to retain its independence and to continue advocating for a peaceful and democratic transition of power. The summary states that NED’s 2018 priorities for the DRC included supporting civil society’s engagement in elections and ability to promote freedom of information before, during, and after the elections. In fiscal years 2015 through 2018, State, the U.S. Agency for International Assistance (USAID), and NED obligated over $73 million for democracy assistance in the DRC. State’s Bureau of Democracy, Human Rights, and Labor obligated $5.5 million (8 percent) of this assistance, while State’s Bureau of International Narcotics and Law Enforcement Affairs obligated $3.2 million (4 percent). State’s Bureau of African Affairs also obligated about $500,000, for one project, through the Africa Women Peace Security Initiative. USAID obligated the majority of U.S. democracy assistance—$54.7 million (74 percent). In addition, NED obligated $9.6 million (13 percent). Figure 13 shows State’s, USAID’s, and NED’s total obligations, by program area, in the DRC during this period. Table 2 shows characteristics of projects funded by State’s Bureau of African Affairs, DRL, USAID, and NED. Three of DRL’s five projects were implemented by organizations that also implemented USAID projects, and the Bureau of African Affairs’ project was implemented by an organization that also implemented USAID and DRL projects. Table 3 shows examples of democracy assistance projects funded by State, USAID, and NED in the DRC in fiscal years 2015 through 2018. While Nigeria has made important gains in democracy and institution building, those gains are fragile, according to the U.S. Department of State (State). The U.S. government’s recent priorities with regard to Nigeria have included helping to strengthen the country’s democratic governance. Challenges to democratic governance in Nigeria include widespread intercommunal violence, terrorism, poverty, and corruption. At the same time, Nigeria has a free press and a political environment that is largely committed to civilian leadership, and the 2015 elections resulted in the first peaceful transfer of power to an opposition party. In this context, the U.S. government’s goals include strengthening Nigerian democratic institutions, governance, and respect for human rights, such as by assisting Nigerians to conduct credible national elections in 2019. To achieve this goal, the U.S. government’s objectives are to (1) strengthen good governance; (2) strengthen democratic institutions, including rule of law, respect for human rights, and transparency and accountability in government; and (3) reduce corruption at all levels of government. Similarly, the National Endowment for Democracy’s (NED) 2018 country summary for Nigeria notes the success of the country’s 2015 elections while also acknowledging challenges including corruption, economic stagnation, insecurity, and the political marginalization of minority groups. NED’s 2018 priorities in Nigeria were to expand political inclusion and strengthen rule of law by supporting NED’s core institutes and local organizations. In fiscal years 2015 through 2018, State, the U.S. Agency for International Development (USAID), and NED obligated nearly $95 million for democracy assistance projects in Nigeria. State’s Bureau of International Narcotics and Law Enforcement Affairs obligated $12.5 million (13 percent), while State’s Bureau of Democracy, Human Rights, and Labor obligated $5.4 million (6 percent). State’s Bureau of African Affairs also obligated $1.8 million for six projects. According to officials, the Bureau of African Affairs funded these projects through the Africa Regional Democracy Fund and the Trans-Sahara Counterterrorism Partnership program. USAID obligated the majority of U.S. democracy assistance— $66.6 million (70 percent). In addition, NED obligated $8.2 million (9 percent). Figure 14 shows State’s, USAID’s, and NED’s total obligations for democracy assistance, by program area, in Nigeria during this period. Table 4 shows characteristics of projects funded by the Bureau of African Affairs, DRL, USAID, and NED in Nigeria during fiscal years 2015 through 2018. Table 5 shows examples of democracy assistance projects funded by State, USAID, and NED in Nigeria during fiscal years 2015 through 2018. Since its 2011 revolution, Tunisia has been on a steady path toward consolidating its democratic transition, but it still needs to establish critical institutions, advance human rights, counter corruption, and improve government transparency, according to the U.S. Department of State (State). In this context, the U.S. government’s goals include helping Tunisia consolidate and advance its democracy. To achieve this goal, the U.S. government’s objectives are to (1) assist Tunisian government institutions to become more transparent, accountable, and responsive to citizens; (2) help Tunisian citizens understand and exercise their rights and responsibilities in a democratic system; and (3) promote social cohesion through democratic consolidation. The National Endowment for Democracy’s (NED) 2018 country summary for Tunisia similarly notes the country’s democratic progress since the 2011 revolution and adds that Tunisian civil society has been developing quickly and freely and seeks to engage with elected officials as they continue to consolidate democracy. NED’s 2018 priorities in Tunisia were to (1) support civil society to promote effective, democratic governance and advocate for transparency and accountability; (2) encourage citizens to influence policymaking; (3) foster political inclusion of marginalized groups; and (4) enhance the role of independent media. In fiscal years 2015 through 2018, State, the U.S. Agency for International Development (USAID), and NED obligated over $90 million for democracy assistance projects in Tunisia. State’s Bureau of Near Eastern Affairs obligated $20.7 million (23 percent) of these funds; the Bureau of Democracy, Human Rights, and Labor obligated $9.1 million (10 percent); and the Bureau of International Narcotics and Law Enforcement Affairs obligated $3.9 million (4 percent). USAID obligated the majority of U.S. democracy assistance—$49.5 million (54 percent). In addition, NED obligated $8.7 million (9 percent). Figure 15 shows State’s, USAID’s, and NED’s total obligations for democracy assistance, by program area, in Tunisia in fiscal years 2015 through 2018. State’s Bureau of Near Eastern Affairs provided the majority of its democracy assistance through the U.S.–Middle East Partnership Initiative, which generally aims to improve governance and economic opportunity. Many of the 11 projects funded by the bureau supported objectives that were similar to those typically supported by DRL, INL, and USAID projects, including promoting human rights, supporting anticorruption institutions, and strengthening political parties. The Bureau of Near Eastern Affair’s Foreign Assistance Unit at the embassy managed these projects. Table 6 shows information on the characteristics of the projects funded by DRL, the Bureau of Near Eastern Affairs, USAID, and NED. Table 7 shows examples of democracy assistance projects funded by State, USAID, and NED in Tunisia during fiscal years 2015 through 2018. Ukraine’s various democratic challenges include overcoming the legacy of Soviet authoritarian rule, addressing mismanagement, and responding to Russian aggression, according to the Department of State (State). In this context, the U.S. government aims to support Ukraine’s democracy by helping the country combat corruption, advance justice reforms, bolster civil society, create responsive government, and encourage independent media. Overall, the U.S. government seeks to help Ukraine advance its political reforms with more transparent, responsive, and accountable governance, becoming less corrupt and more democratic. U.S. objectives to accomplish this goal include enhancing anticorruption and rule-of-law processes and improving governance processes and outcomes. The National Endowment for Democracy’s (NED) 2018 country summary for Ukraine noted similar challenges to the country’s democracy— Russian aggression, corruption, and a government that is not responsive to its citizens. NED’s 2018 priorities in Ukraine included strengthening the capacity of civil society groups, promoting reconciliation, and fostering the development of new media. In fiscal years 2015 through 2018, State, the U.S. Agency for International Development (USAID), and NED obligated more than $170 million for democracy assistance projects in Ukraine. State’s Bureau of European and Eurasian Affairs obligated $16.7 million (10 percent) of this assistance; the Bureau of Democracy, Human Rights, and Labor obligated $9.6 million (6 percent); and the Bureau of International Narcotics and Law Enforcement Affairs obligated $5.0 million (3 percent). USAID obligated the majority of U.S. democracy assistance—$126 million (73 percent). In addition, NED obligated $16.3 million (9 percent). Figure 16 shows State’s, USAID’s, and NED’s total obligations for democracy assistance, by program area, in Ukraine during fiscal years 2015 through 2018. State’s public affairs unit at the embassy in Ukraine obligated funding for, and managed, all but one of the 613 democracy assistance projects supported by funds from the Bureau of European and Eurasian Affairs. State’s public affairs unit awarded the projects through funding mechanisms that were intended to support civil society and independent media and were specifically designed for locally based implementing organizations. Table 8 shows characteristics of democracy assistance projects funded by DRL, State’s Bureau of European and Eurasian Affairs, USAID, and NED. Table 9 shows examples of democracy assistance projects funded by State, USAID, and NED in Ukraine during fiscal years 2015 through 2018. In addition to the contact named above, Mona Sehgal (Assistant Director), Farhanaz Kermalli (Analyst-in-Charge), Daniela Rudstein, Tom Zingale, Neil Doherty, Reid Lowe, and Alex Welsh made key contributions to this report. Justin Fisher and Sarah Veale provided technical assistance.", "summary": "Congress made at least $2 billion available to agencies annually for democracy assistance abroad in fiscal years 2015 through 2018. State and USAID are the primary U.S. agencies funding democracy assistance. This assistance supports activities related to enhancing rule of law, good governance, political competition and consensus building, civil society, independent media, and human rights. Congress included a provision in the Joint Explanatory Statement accompanying the fiscal year 2015 Continuing Appropriations Act for GAO to review agencies' roles and responsibilities in promoting democracy abroad. This report examines (1) State's and USAID's democracy assistance allocations, (2) State's and USAID's roles in providing democracy assistance and the extent to which their projects in selected countries are consistent with their defined roles, and (3) how State and USAID coordinate on democracy assistance. GAO reviewed State and USAID data and documents for fiscal years 2015 through 2018 and interviewed officials in Washington, D.C., and in the DRC, Nigeria, Tunisia, and Ukraine. GAO selected these countries because they received relatively high amounts of democracy assistance funding from State and USAID, among other factors. The Department of State (State) and U.S. Agency for International Development (USAID) allocated more than $8.8 billion for democracy assistance in fiscal years 2015 through 2018. a According to agency officials, language in the 2015 appropriations act permitted State and USAID to allocate less than the full amount directed to democracy programs by the act. State and USAID have defined roles for democracy assistance and have obligated funding for projects in selected countries accordingly. State has identified its Bureau of Democracy, Human Rights, and Labor (DRL) as the U.S. lead for promoting democracy and protecting human rights abroad and has identified its Bureau of International Narcotics and Law Enforcement Affairs (INL) as the lead for promoting the rule of law. In fiscal years 2015 through 2018, DRL's and INL's obligated funding for democracy assistance in the countries GAO reviewed—the Democratic Republic of the Congo (DRC), Nigeria, Tunisia, and Ukraine—generally reflected their defined roles. For example, 24 to 77 percent of DRL's obligated funding in these countries supported human rights, and at least 90 percent of INL's obligated funding for democracy assistance in the countries supported the rule of law. USAID's democracy assistance strategy states that USAID has the leading role in U.S. development assistance. USAID's obligations for democracy assistance in the four countries supported multiyear, multimillion-dollar projects, consistent with what USAID officials told GAO was needed for long-term development. State and USAID coordinate on democracy assistance in various ways, but embassy officials reported gaps in information about DRL assistance. Examples of coordination mechanisms include budget allocation discussions at headquarters and working groups at embassies to help avoid project duplication. However, State officials in all four selected countries said they generally lacked information about DRL democracy assistance projects, including project descriptions and funding amounts. State's existing information-sharing mechanisms, including data systems and strategies, do not consistently address these gaps. Overseas officials' lack of complete information about DRL's projects may inhibit State's efforts to coordinate with other agencies, implementing partners, and other donors. GAO recommends that the Secretary of State direct DRL to develop a mechanism for the sharing of democracy assistance project information between DRL and relevant embassy staff. State concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "Like all DOD working capital funds, the DWWCF received its initial working capital through an appropriation or a transfer of amounts from existing appropriations to finance the initial cost of products or services. Ongoing DWWCF operations and maintenance of a minimum cash balance are funded through reimbursements to the DWWCF comprised of customer payments made to DFAS, DISA, and DLA. The flow of funding and provision of goods and services between the DWWCF agencies, their customers, and the DWWCF is shown in figure 1. DFAS, DISA, and DLA use funds from the DWWCF to provide goods and services across six activity groups, as shown in table 1. Activities of the DWWCF agencies operate on a break-even basis. As part of the annual budget submission for each upcoming fiscal year, rates are required to be established at levels estimated to recover the budgeted costs of goods and services, including all general and administrative overhead costs, prior period gains and losses, and applicable surcharges. Predetermined or “stabilized” rates developed during the budget process are applied to orders received from DWWCF customers during the fiscal year. The Office of the Under Secretary of Defense (Comptroller) is responsible for reviewing, coordinating, and publishing reimbursable rates for DOD. Where feasible, the Office of the Under Secretary of Defense (Comptroller) publishes applicable reimbursable rates prior to the beginning of each new fiscal year. The military departments are the primary consumers of goods and services provided by the DWWCF agencies. In fiscal year 2018, the reported total dollar value of goods and services ordered from DFAS, DISA, and DLA was approximately $49.4 billion, with the military departments collectively ordering about $36.3 billion (or 74 percent of the total dollar value of orders in fiscal year 2018) in goods and services. Most of the goods and services they purchased fell under two activity groups—supply chain management and energy management. Specifically, approximately $29.4 billion (60 percent of the total dollar value of orders in fiscal year 2018) were for supply chain management and approximately $10.8 billion (22 percent of the total dollar value of orders in fiscal year 2018) were for energy management, as shown in figure 2. Prior to the beginning of the fiscal year, Annual Operating Budgets are issued for each DWWCF activity group managed by DFAS, DISA, and DLA. Budget formulation for a particular fiscal year begins approximately 18 months prior to the beginning of that fiscal year. Each activity’s Annual Operating Budget identifies total budgetary resources authorized for use during the fiscal year. In addition, each DWWCF agency is responsible for maintaining positive cash balances sufficient to allow their operations to continue uninterrupted. As of the end of fiscal year 2017, the DWWCF as a whole held a reported cash balance of about $3.0 billion, which decreased to $2.6 billion by the end of fiscal year 2018. According to DWWCF agency officials, it can be challenging to maintain an appropriate cash balance within the DWWCF because setting accurate rates that reflect their agencies’ actual costs is difficult. If rates are set too low during the budget formulation process, higher-than-expected costs or lower-than-expected sales of goods or services during the fiscal year may result in losses for the DWWCF, which in turn may lead to insufficient balances to meet the minimum current or future financing operational requirements. Similarly, if rates are set too high, lower-than-expected costs or higher-than-expected customer sales during the fiscal year may generate excessive gains for the DWWCF. Excessive gains may be transferred out of the DWWCF into other appropriation accounts or rescinded by Congress. In 2017, we described the DWWCF’s reported monthly cash balances and the extent to which they fell within targeted upper and lower cash requirements. We found that the DWWCF’s reported monthly cash balances were outside the targeted upper and lower cash requirements for 87 of 120 months during that time frame. This was caused by DLA charging its customers more or less than it cost to purchase, refine, transport, and store fuel and by DOD transferring funds into or out of the DWWCF to pay for combat fuel losses or other higher priorities, among other things. As we noted in our report, DOD revised its cash management policy to require a positive cash balance throughout the year and an adequate ending balance to support continuing operations into the subsequent year. According to this revised policy, in setting the cash requirement goals, DOD working capital funds are to consider four elements: (1) the rate of disbursement, which is the average amount disbursed between collection cycles; (2) the range of operation, or the difference between the highest and lowest expected cash levels based on budget assumptions and past experience; (3) risk mitigation, which requires some amount of cash beyond the range of operations to mitigate the inherent risk of unplanned and uncontrollable events; and (4) reserves, which are cash amounts held for known future requirements. DOD officials said they are in the process of adding additional guidance to the DOD Financial Management Regulation about when DOD managers should use available tools to help ensure that monthly cash balances are within the targeted upper and lower cash requirements, as we recommended in our 2017 report. For this report, we updated the 2017 analysis to include the DWWCF’s reported monthly cash balances and targeted upper and lower cash requirements for fiscal years 2017 and 2018, as shown in figure 3. For fiscal year 2017, the monthly cash balances were above and below the targeted upper and lower cash requirements one time each, and for fiscal year 2018 the targeted upper cash requirement was raised and the monthly cash balances were all within the revised targeted upper and lower cash requirements. In our prior work, we identified four key operating principles to guide the management of working capital funds. These key operating principles call for (1) working capital fund self-sufficiency, which includes establishing transparent pricing; (2) clearly delineated roles and responsibilities; (3) performance measurement; and (4) built-in flexibility to obtain customer input and meet customer needs. As we describe later in this report, each of these key operating principles has three underlying components describing specific actions agencies should take to adhere to the principle. For further information about each key principle and its components, see appendix I. We found that DFAS, DISA, and DLA have applied two of the three components of the key operating principle for working capital fund self- sufficiency by setting rates that are designed to cover actual costs and establishing a management review for rate setting. However, despite taking steps intended to allocate costs equitably among their customers, DFAS, DISA, and DLA have not fully applied the third component of the key operating principle by establishing pricing methodologies that are transparent to their customers. DFAS, DISA, and DLA each develop budget proposals annually that are designed to recover their projected costs, while also accounting for any gain or loss from previous years. The three DWWCF agencies generally set rates that are intended to mitigate prior year gains or recover all prior year losses in the current fiscal year of execution, although they can spread the return actions over several fiscal years to minimize the impact on customers from rate fluctuations. The rate-setting processes used by DFAS, DISA, and DLA include management reviews. Each agency’s management reviews and approves the budget proposals for its DWWCF activities during the budget formulation process. The agencies then send the budget proposals to the Office of the Under Secretary of Defense (Comptroller) for further review and approval, and the Office of the Under Secretary of Defense (Comptroller) issues a memo finalizing the rates to be charged by DFAS, DISA, and DLA during the fiscal year. DFAS, DISA, and DLA each use multiple approaches to set rates on an annual basis and adjust these rates, as appropriate, during DOD’s programming and budget development process. All three agencies include direct and indirect costs in their rates, and these costs vary due to differences in the agencies’ missions, including the goods and services they provide. In general, the DWWCF agencies describe direct costs as those costs that can be directly attributed to an output and a customer. For example, DFAS officials told us that DFAS includes the cost of the labor that supports civilian pay services for a customer as a direct cost. DISA officials said that DISA includes the cost of servers as a direct cost. DLA officials indicated that materiel costs, such as the cost to acquire fuel or a spare part, are considered direct costs. Alternately, the three agencies describe indirect costs as costs that cannot be attributed to one specific output and customer. For example, costs for information technology systems that support multiple customers, supervisory staff that support more than one customer, and general and administrative (overhead) costs such as a DWWCF agency’s general counsel services or physical facility maintenance are all indirect costs. DFAS, DISA, and DLA set rates during DOD’s annual programming and budget development process. The three DWWCF agencies begin the process of setting rates approximately 18 months prior to the fiscal year in which the rates will be applied. Setting rates in advance helps ensure that adequate resources are requested in the customers’ fund accounts to pay the established rates and prices. The Office of the Under Secretary of Defense (Comptroller) reviews and approves finalized rates for a particular fiscal year in a rate memo circulated during the prior fiscal year. DFAS, DISA, and DLA use a combination of the following three approaches when setting the rates. Table 2 shows instances in which each of the three DWWCF agencies use the following rate-setting approaches. Per Unit: Determines a specific dollar rate per unit that, when multiplied by the projected workload, will produce revenue sufficient to recover the full costs, including direct and indirect costs, of providing the good or service. Portion of Total Costs: Charges a portion of the agency’s total costs (both direct and indirect costs) of providing a service based either on the proportion of total workload projected for a specific customer or a uniform percentage across all customers. Percentage Markup on Direct Costs: Adds a fee based on a percentage of the direct costs of a service as a proxy for expected indirect costs. DFAS, DISA, and DLA officials described their approaches to allocating costs when setting rates. DFAS allocates costs to each of the services it provides and to each customer using those services through 29 predetermined business rules. DISA groups its services by the costs associated with providing them and allocates these costs to the services in each group based on factors such as the cost of equipment used to provide each service. Similarly, DLA uses various methods to allocate indirect costs to some or all of DLA’s goods or services based on factors such as the number of employees supporting the provision of a given good or service and the total sales of that good or service. In each case, these costs are then included in the rates DFAS, DISA, and DLA charge customers for each good or service. See appendixes II through IV for more information on each agency’s rate-setting approach. We found that DFAS, DISA, and DLA have taken steps intended to establish an equitable pricing methodology. However, customers from the military departments told us, and our review of related documentation provided at rate briefings and cost summits found, that the information they receive regarding the pricing of goods and services is not transparent. Officials from all three DWWCF agencies described efforts to more equitably allocate costs associated with a given good or service to the customers who use that good or service, as described below. DFAS, for fiscal year 2019, changed its method for allocating the costs of its facilities to its customers in an effort to more equitably allocate these costs. Previously, the costs for each DFAS facility were charged directly to the customers whose work was performed in that facility. Since the costs of facilities differ and customers do not choose the location that provides their service, DFAS changed this methodology so that customers now pay a uniform percentage of their direct costs to cover the total cost of DFAS facilities. DISA, for fiscal year 2017, changed the pricing structure for Defense Information System Network Infrastructure Services in response to recommendations from two DOD internal reviews. The structure changed from one designed to encourage adoption of the network across DOD to a consumption-based model that aligns customer billing with consumption so that customers have greater control over their costs. According to DISA officials, this change has enhanced collaboration between DISA and its customers, providing customers more frequent inventories of the services they require so that the customers can determine that the bills for those services reflect their requirements. DLA is implementing two changes to the pricing methodology it uses for distribution services, part of its Supply Chain Management activity group. The first change, which DLA refers to as distribution price equitability, was implemented during fiscal year 2017. This pricing methodology allocates overhead costs to reimbursable distribution services (special services not included in DLA’s standard rate structure). Previously, only rate-driven distribution services were charged for overhead. DLA proposed the second change, market basket pricing, for implementation in fiscal year 2020. Market basket pricing changes this pricing from being based solely on the weight of the items being distributed to a method that considers the level of effort required by DLA to distribute the items. For example, bulky, fragile, and hazardous items will be charged higher rates than small, easy-to-ship items. While we found that DFAS, DISA, and DLA have taken steps intended to establish an equitable pricing methodology, military department officials from the offices we contacted said that they lack visibility into the factors that determine their costs at one or more of the three defense agencies. Specifically, they said they had a limited understanding of the types of indirect costs that are included in the rates they are charged and how those costs are allocated, the specific changes that have been made to the methods used to set rates, or how changes in the customer’s use of the services, which would also change an agency’s workload, would affect overall costs. DFAS, DISA, and DLA have produced documentation for their customers to explain their rates and have developed ways to communicate with their customers—for instance, through the use of customer liaisons. However, officials from the military departments told us this documentation does not contain the level of detail they need to fully understand the rates. For example, DFAS. Navy and Army officials we spoke with regarding DFAS said that their departments lack visibility into how DFAS’s rates and bills are calculated because DFAS informational briefings do not describe the types of costs included in rates and how those costs are calculated and allocated to customers. As a result, officials said they are confused by why declines in their use of DFAS’s services have not resulted in reduced costs. These officials said that this information would make it easier for them to determine how to manage their costs and verify that costs are equitably allocated and reflect usage. DISA. Air Force officials we interviewed regarding DISA told us that DISA does not provide sufficient pricing transparency because, although DISA has provided some documentation of its rates, this documentation does not explain the methodology used to calculate the rates and the costs included in those calculations. Although Army officials who discussed DISA said that DISA rate briefings provide the level of information necessary for customers, the Air Force officials said that this lack of information on how DISA calculates rates makes it difficult for the Air Force to determine how it can manage its costs with DISA or whether the rates it pays reflect the costs of the services it uses. DLA. Navy and Air Force officials we interviewed regarding DLA told us that DLA does not provide sufficient pricing transparency despite the rate briefings DLA conducts for its customers. Although the Army officials who discussed DLA said that the rate briefings provide sufficient information for customers, the Navy officials told us that the lack of detailed information on the costs included in DLA’s rates makes it difficult for the Navy to determine how to lower its costs. They also said this lack of information prevents them from determining whether the rates they pay actually reflect the costs of the services they use, as intended. Similarly, the Air Force officials told us that DLA’s communication regarding its market basket pricing initiative, discussed during the DLA briefings, was confusing and did not include all the information they needed to prepare their budget, such as when the change would be implemented and how the initiative would affect the Air Force’s costs. Officials noted that, despite initially being told by DLA that the Air Force would experience a reduction in its distribution costs as a result of this initiative, they subsequently learned through a Resource Management Decision that the Air Force’s costs would increase instead. DFAS, DISA, and DLA officials told us they make efforts to communicate with their customers and to improve the transparency of their rates. For example, DFAS officials noted that they have one-on-one discussions with each of their customers during the customer rate briefings. DISA officials said that they respond to customer questions regarding rates and share information on the costs included in those rates. DLA officials said they discuss rates at a variety of customer forums and share documentation of changes to their rate-setting methodologies, such as market basket pricing, with customers. Officials from the military departments acknowledged these efforts by the DWWCF agencies to share information. However, as described in the examples above, officials told us that one or more of the agencies have not provided them with the information needed to fully understand their costs, to have assurance that costs are being allocated fairly, or to identify actions they could take to affect their overall bills, in some cases, despite requests for more detailed cost information. In addition, DFAS, DISA, and DLA provided us copies of documents that they present at rate briefings and cost summits to share information about their pricing methodologies with their customers. In our review of those documents, we found that they contained high-level information, such as the rates themselves and the estimated workloads, and did not contain detailed information about the types of costs included in the rates and how those costs are calculated. For example, although DLA provides its cost recovery rate for the materiel supply chains in its rate briefing documentation, the documentation does not provide information on the specific costs that go into that rate. As a result, based on these documents, we also were not able to fully understand the agencies’ costs and how those costs are allocated among their customers. By providing more complete and transparent information on methodologies used to calculate rates, the costs used in those calculations, and how changes in workload affect a customer’s rates, DFAS, DISA, and DLA could improve their communication with their customers and allow their customers to better understand and make decisions to help them manage the costs of the goods and services that they obtain. Such information would also better inform customers of any changes to the assumptions underlying rates and the impact those changes might have on their future costs. We found that DFAS, DISA, and DLA have applied all of the components of the three remaining key operating principles for effective management of working capital funds. These principles relate to delineating roles and responsibilities, measuring performance, and assessing resource requirements and customer needs. By implementing these principles, the DWWCF agencies are better positioned to: Promote a clear understanding of who will be held accountable for specific tasks or duties, reduce the risk of mismanaged funds and tasks or functions “falling through the cracks,” and educate customers about whom to contact if they have questions. Measure their operational performance, assess their performance against strategic goals, and identify opportunities to improve performance. Enable customers to provide input about working capital fund services or voice concerns about their needs, enable agencies to prioritize customer demand, and enable agencies to use resources most effectively. We found that all three DWWCF agencies have fully applied the three components of the principle for clearly delineating roles and responsibilities in that they define key areas of authority and responsibility, segregate duties to reduce fraud, and have a management review and approval process. Define key areas of authority and responsibility. DFAS, DISA, and DLA define key areas of authority and responsibility and provide customers with clear information on who to contact if they encounter issues or have questions. DFAS defines the responsibilities of key offices, such as those responsible for tracking revenue, in its Doing Business with DFAS catalog of services. This document lists points of contact, specific to each customer, who can provide support and address customers’ questions. DFAS also maintains service level and audit agreements with its customers, called mission work agreements, to document the specific level of effort and service it will provide. DISA’s instructions define the roles and responsibilities for key officials and offices involved in managing the agency’s DWWCF activities. DISA also provides contact information for its customer account representatives in its DWWCF Rate Book and on its website. DLA’s customer assistance handbook explains the roles of different offices within DLA and contains contact information for each of DLA’s activity groups and for customer-specific representatives. DLA also defines roles and responsibilities of interagency groups in the performance-based agreements it signs with the military departments and services. For example, an agreement between the Department of the Army and DLA defines roles and responsibilities for the Partnership Agreement Council, the organization that addresses and prioritizes issues related to improving logistics coordination between DLA and the Army. Segregate duties to reduce error or fraud. DFAS, DISA, and DLA segregate duties across their organizations and document this segregation. DFAS documents the segregation of responsibilities for tracking and recording transactions for each of its service offerings in its Doing Business with DFAS catalog of services. For example, DFAS’s Retired and Annuitant Pay section tracks the number of individuals serviced under those pay systems, and DFAS’s Central Revenue Office records these transactions in DFAS’s billing system. DISA describes its processes for segregating duties in the documentation of its working capital fund disbursements and collections processes. For example, when DISA charges a customer agency, an accounts receivable technician records the billing information and a certifying officer verifies and certifies the transaction. DLA documents its segregation of key roles and responsibilities for authorizing, processing, and reviewing transactions according to DOD and DLA guidance. For example, the DOD manual that outlines sales accountability and documentation processes for energy commodities assigns responsibility to DLA Energy for ensuring that DLA customers meet the criteria or have received waivers to purchase fuel through DLA, while DLA Transaction Services provides activity codes to authorized customers to manage their transactions. Additionally, individual fuel handlers at DLA Energy stock points are required to record customer data for sales and credits on a source document. Establish a management review and approval process. DFAS, DISA, and DLA have established management review and approval processes to promote the appropriate tracking and use of funds. DFAS documents its processes for tracking and reviewing transactions in its Doing Business with DFAS document. DISA describes the review of transactions in its documentation of its funds disbursements and collections processes. Each of DLA’s activity groups tracks transactions and funding using DLA’s accounting system of record—the Enterprise Business System. However, each group uses its own unique order validation process that is documented for each DLA activity group. We found that all three DWWCF agencies are applying the three components of the key operating principle for measuring performance in that they have established performance measures and goals, aligned performance measures with strategic goals, and established a management review of DWWCF performance. Establish performance measures and goals. DFAS, DISA, and DLA have each established performance measures and goals. DFAS uses financial and mission-focused performance measures, called business models, which include metrics for service timeliness and accuracy, among others. DISA has operational performance metrics, such as service downtime, and collects customer feedback on service provision through its Mission Partner Engagement Office. DLA establishes performance measures and corresponding thresholds in the performance- based agreements it signs with its customers from the military departments and services. These performance measures include materiel availability and backorders, among other measures. Align performance measures with strategic goals. DFAS, DISA, and DLA have performance measures that are aligned with their strategic goals. Each DWWCF agency is responsible for maintaining positive cash balances sufficient to allow their operations to continue uninterrupted. To achieve this, all three agencies monitor their monthly cash balances and whether each of its activity groups is experiencing gains or losses. DFAS, DISA, and DLA also have aligned operational performance measures with their strategic goals as illustrated by the following examples. DFAS’s Fiscal Year 2017-2021 Strategic Plan identifies achieving cost, schedule, and performance targets that support delivery of best value services. DFAS monitors the timeliness and accuracy of its services reflecting this strategic outcome in its performance measurement. For example, DFAS measures the percentage of commercial payments it processes accurately and the percentage of military pay problem cases that it resolves in a timely manner. DISA’s Strategic Plan 2019-2022 states that optimizing enterprise services and capabilities to minimize costs while delivering high availability, among other benefits, is a strategic objective. To that end, DISA monitors data center and equipment availability through performance measures such as the average number of minutes of facility downtime per fiscal year. DLA’s Strategic Plan 2018-2026 identifies strengthening readiness and lethality as its highest priority line of effort. DLA monitors how its own performance affects readiness of critical weapon systems using its Service Readiness Dashboard, which includes a measure for the number of weapon systems that are non-mission capable due to DLA supply items being unavailable. Defense-Wide Working Capital Fund Midyear Rate Changes Agency officials said that rates are generally fixed for the entire fiscal year, but the Office of the Under Secretary of Defense (Comptroller) can approve midyear rate changes if required. For example, we previously reported that the Defense Logistics Agency (DLA) collected about $3.7 billion more from the sale of fuel than it cost in fiscal year 2015 because of lower fuel prices. Conversely, during fiscal year 2018, DLA’s cost of procuring fuel increased significantly due to increases in the price of the fuel procured from the market. As a result, in April 2018, DOD increased the rate from $90.30 per barrel to $115.92 per barrel to cover its costs. Establish management review of working capital fund performance. DFAS, DISA, and DLA regularly monitor and have management reviews of agency performance against these financial and mission-related performance measures. Officials from each agency said they review the financial performance of the agencies’ activities throughout the year the programs and budgets will be executed to identify how differences between budgeted rates and actual costs affect the fund’s gains, losses, and cash balances. This allows them to coordinate with the Office of the Under Secretary of Defense (Comptroller) to propose price changes when needed, although officials said that a midyear rate change is rare (see sidebar). The agencies also regularly review their non-financial performance based on the previously described measures to identify areas for improvement. We found that all three DWWCF agencies are applying the three components of the key operating principle of building in flexibility to obtain customer input and meet customer needs by communicating with customers regularly and in a timely manner, developing processes to assess resource needs, and establishing processes to prioritize requests for service. Communicate with customers regularly and in a timely manner. DFAS, DISA, and DLA each routinely communicate with customers through annual rate briefings, customer forums, surveys, and other meetings. These meetings enable these agencies to provide high-level rate information to their customers and discuss the goods and services that their customers will need in the coming budget year. For example, DFAS communicates with the military services’ budget offices through an annual briefing at a meeting hosted by the Office of the Under Secretary of Defense (Comptroller) and surveys finance officers and end-user customers on their satisfaction with military pay services. Similarly, DISA holds routine meetings at the working group and senior official levels to discuss service offerings, among other things. In addition to its biannual cost summits where DLA discusses its pricing strategies with representatives from the military services, the Office of the Under Secretary of Defense (Comptroller), and the Office of the Under Secretary of Defense for Acquisition and Sustainment, DLA also holds an annual demand planning summit with the military services to discuss their projected requirements for the upcoming budget year. Develop process to assess resources needed to meet changes in customer demand. DFAS, DISA, and DLA each take steps to communicate with customers regarding future demand and requirements. All three agencies have customer-specific representatives that obtain information on future requirements and facilitate communication between the agencies and their customers. DFAS uses client executive liaisons to resolve issues and collect information about customer needs for its goods and services. DISA uses its Mission Partner Engagement Office to address customer concerns and conduct surveys about customer needs. DLA has national account managers that represent each military service and facilitate DLA’s engagement with the services regarding requirements and customer service representatives with select customers to meet their specific needs for DLA’s goods and services. Establish process to prioritize requests for services. DFAS, DISA, and DLA each have processes to adjust resources in response to the needs of their customers. This primarily occurs during the budget formulation process. DFAS officials told us that labor accounts for about 75 percent of the agency’s costs, and management can decide to adjust its workforce resources depending on customer needs, often by shifting personnel and workload among customers, temporarily hiring additional staff, or reducing staffing levels through attrition. DISA’s Strategic Resourcing Council is responsible for addressing issues such as resourcing strategies for existing and emerging programs. DLA uses its Enterprise Operations Planning Council, a group of DLA executives responsible for actively balancing customer needs and supply chain constraints, to ensure that resources are aligned with customer requirements during the budget formulation process. The agencies whose operations are financed through the Defense-Wide Working Capital Fund have applied all but one of the components of the key operating principles for effective management of working capital funds—establishing a transparent and equitable pricing methodology, a component of the principle of ensuring self-sufficiency by recovering the agency’s actual costs. Transparent pricing helps ensure that customers understand their costs and can make choices to manage these costs. Officials from the military departments—the largest customers of DFAS, DISA, and DLA—said they lack visibility into the types of costs included in their rates and some do not understand how changes to rate-setting methodologies or defense agency workload can affect their overall costs. By providing this information to customers, DFAS, DISA, and DLA would better equip them to reduce their costs and improve efficiency. Further, DOD would have greater assurance that the DWWCF was operating as intended. We are making the following three recommendations to DOD: The Secretary of Defense should ensure that the Director of the Defense Finance and Accounting Service provides customers with more complete information on the agency’s rate-setting methodologies in rate documentation, briefings, and other forums where rates are discussed, including the costs included in rates, how those costs are calculated, and how changes in DFAS’s workload affect customers’ overall costs. (Recommendation 1) The Secretary of Defense should ensure that the Director of the Defense Information Systems Agency provides customers with more complete information on the agency’s rate-setting methodologies in rate documentation, briefings, and other forums where rates are discussed, including the costs included in rates, how those costs are calculated, and how changes in DISA’s workload affect customers’ overall costs. (Recommendation 2) The Secretary of Defense should ensure that the Director of the Defense Logistics Agency provides customers with more complete information on the agency’s rate-setting methodologies in rate documentation, briefings, and other forums where rates are discussed, including the costs included in its rates, how it calculates those costs, and how and when proposed changes to its rate-setting methodologies will affect customers’ overall costs. (Recommendation 3) We provided a draft of this report to DOD for review and comment. In an email accompanying its written comments, DOD concurred with our recommendations. In the department’s written comments, DFAS, DISA, and DLA stated that they intend to take steps to provide their clients with additional information on rates. These steps include reaching out to customers to better understand their information needs and providing additional information on potential pricing methodology changes. DOD’s comments are reprinted in appendix V. DOD also provided technical comments during this review, which we incorporated, as appropriate. We are sending copies of this report to the appropriate congressional addressees and the Secretary of Defense. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Elizabeth Field at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Examples of evidence supporting principle the agency’s actual costs Transparent and equitable pricing methodologies allow agencies to ensure that rates charged recover agencies’ actual costs and reflect customers’ service usage. If customers understand how rates are determined or changed including the assumptions used, customers can better anticipate potential changes to those assumptions, identify their effect on costs, and incorporate that information into budget plans. A management review process can help to ensure the methodology is applied consistently over time and provides a forum to inform customers of decisions and discuss as needed. Published price sheets for services are readily available. Documentation of pricing formulas supports equitable distribution of costs. Pricing methodology and accompanying process ensures that, in aggregate, charges recover the actual costs of operations. Management review process allows fund managers to receive and incorporate feedback from customers. Discussions with customers confirm an understanding of the charges and that they are viewed as transparent and equitable. Written roles and responsibilities specify how key duties and responsibilities are divided across multiple individuals/offices and are subject to a process of checks and balances. This should include separating responsibilities for authorizing transactions, processing and recording them, and reviewing the transactions. Written description of all WCF roles and responsibilities in an accessible format such as a fund manual. Discussions with providers and clients confirm a clear understanding. A routine review process exists to ensure proper execution of transactions and events. Performance goals and measures are important management tools applicable to all operations of an agency, including the program, project, or activity level. Performance measures and goals could include targets that assess fund managers’ responsiveness to customer inquiries, the consistency in the application of the funds’ rate-setting methodology, the reliability of cost information, and the billing error rates. Performance measures that are aligned with strategic goals can be used to evaluate whether, and if so how, WCF activities are contributing to the achievement of agency goals. A management review process comparing expected to actual performance allows agencies to review progress towards goals and potentially identify ways to improve performance. Performance indicators and metrics for WCF management (not just for the services provided) are documented. Indicators or metrics to measure outputs and outcomes are aligned with strategic goals and WCF priorities. WCF managers regularly compare actual performance with planned or expected results and make improvements as appropriate. In addition, performance results are periodically benchmarked against standards or “best in class” in a specific activity. input and meet customer needs. Opportunities for customers to provide input about WCF services, or voice concerns about needs, in a timely manner enable agencies to regularly assess whether customer needs are being met or have changed. This also enables agencies to prioritize customer demands and use resources most effectively, enabling them to adjust WCF capacity up or down as business rises or falls. Established forum, routine meetings, and/or surveys solicit information on customer needs and satisfaction with WCF performance. Established communication channels regularly and actively seek information on changes in customer demand and assess the resources needed to accommodate those changes. Established management review process that allows for trade-off decisions to prioritize and shift limited resources needed to accommodate changes in demand across the organization. DFAS reported receiving total Defense-Wide Working Capital Fund orders for services valued at approximately $1.4 billion in fiscal year 2018. DFAS employs around 12,000 civilian personnel and provides services to DOD and other federal entities through a single activity group—Finance and Accounting Services. Approach to Allocating Costs: The Defense Finance and Accounting Service (DFAS) establishes rates for each of the services it provides. DFAS first links direct costs to each service and to each customer benefitting from that service. Then, DFAS applies 29 predetermined business rules to allocate indirect costs, which include mission-related indirect costs and general and administrative costs. These business rules identify costs associated with specific combinations of mission-related indirect costs necessary to provide a service and then apply a “fair-share” percentage of general and administrative indirect costs, which allows DFAS to determine the rates it needs to charge to recover all costs. General and administrative costs associated with supporting the entire DFAS organization are allocated at a uniform percentage within DFAS’s rates for all systems, services, and customers based on total direct costs. Services Provided: DFAS provides centralized finance, accounting, human resources, and financial systems management services. DFAS categorizes its services into three types: rate-based services (which include military and civilian pay services and accounting services), direct systems reimbursements (i.e., legacy accounting systems), and support- to-others (i.e., reimbursable services that are outside of DFAS’s core mission or reflect emerging mission workload). Indirect Costs: DFAS differentiates between two types of indirect costs: (1) indirect costs that are necessary to support DFAS’s direct mission but are not direct costs because they support multiple types of work or customers (e.g., information technology network infrastructure, senior operations management, and facilities costs) and (2) general and administrative costs that support DFAS as a whole and are not linked to specific services (e.g., costs for DFAS’s internal review office and other headquarters-related costs). rate for each service that includes direct and indirect costs as allocated by its predetermined business rules. For civilian pay services, the number of units sold is based on the number of active civilian pay accounts in a given month (e.g., the number of civilian leave and earnings statements generated). For accounting services, the number of units sold is based on the number of labor hours DFAS employees recorded supporting a given task and customer. 2. Portion of Total Costs: DFAS charges a portion of the agency’s total legacy systems costs (direct costs and both mission-related and general and administrative indirect costs) of providing a service based on the proportion of total workload projected for a specific customer. DFAS uses this approach for its direct systems reimbursement services, as described below. Direct Systems Reimbursement. DFAS charges customers a percentage of the total costs—including direct and both types of indirect costs—of each legacy accounting system based on each customer’s portion of total system usage. 3. Percentage Markup on Direct Costs: DFAS adds a percentage markup to its direct costs in support of non-core or emerging mission workload to recover general and administrative indirect costs of the associated support. DFAS uses this approach for its support-to-others services, as described below. Support-to-Others. DFAS charges customers the actual direct cost of providing a support-to-others service plus the general and administrative percentage markup. DISA reported receiving total Defense-Wide Working Capital Fund (DWWCF) orders for services valued at approximately $7.5 billion in fiscal year 2018. DISA employs around 8,700 military and civilian personnel and provides its services through two activity groups: Computing Services and Telecommunications Services and Enterprise Acquisition Services. Services Provided: Computing Services operates the DISA Data Centers, which provide mainframe and server processing operations, data storage, and other information technology services and support across the Department of Defense (DOD). Telecommunications Services provides secure telecommunications services, including the Defense Information Systems Network. Enterprise Acquisitions Services provides contracting services for information technology and telecommunications acquisitions from the commercial sector and contracting support to the Defense Information Systems Network programs and other customers through DISA’s Defense Information Technology Contracting Organization. Approach to Allocating Costs: The Defense Information Systems Agency (DISA) groups its services by the costs associated with providing them. These costs are specific to the service being provided and are influenced by factors such as the cost of equipment used to provide the service. Computing Services has a large collection of billing rates, tailored to the services provided to a customer, such as mainframe and server processing; storage; and other services. Approximately half of DISA’s business in Telecommunications Services is for the Defense Information Systems Network, for which DISA sets a standard rate to recover costs. The remaining half is for reimbursable services that cover services such as commercial satellite phones, instant message services, global videoconferencing services, and support for secure portable electronic devices (both smartphones and tablets). The commercial satellite communications program recovers costs through a management fee that is added to the direct contract costs. According to DISA officials, cost reimbursable services are those services that are not included in DISA’s standard offerings and thus do not have a standard rate. DISA recovers the cost for these services, including direct, indirect (overhead), and general and administrative costs, and the total cost is negotiated with customers up front. Approaches Used to Calculate Rates for DISA Services: DISA uses three approaches for calculating rates: 1. Per Unit: DISA determines a specific dollar rate per unit that, when multiplied by the projected workload, will produce revenue sufficient to recover the full costs, including direct and indirect costs, of providing the good or service. DISA uses this approach for most Computing Services and some Telecommunications Services. Computing Services. DISA calculates most of its Computing Services rates by dividing the total costs of providing a service by total projected units. Total costs of a service comprise direct and indirect costs, including general and administrative costs. Indirect Costs: These include costs that are associated with a particular service, such as facilities costs, and those that are associated with support provided to all services, such as personnel support. Contract management costs are included for all services but are recovered differently. Telecommunications Services. DISA’s mobility program, which provides support for portable electronic devices, recovers costs by charging a rate per device per month. Similarly, DISA’s cross- domain services, which provide the ability for customers to transfer information across different security domains (unclassified and classified systems) at a price for each filter supported. 2. Portion of Total Costs: DISA charges a portion of its total costs, including direct and indirect costs, of providing a service based on the proportion of total workload projected for a specific customer. DISA uses this approach for several of its Telecommunications Services. DISA’s Telecommunications Services charges customers a portion of the total costs for the Defense Information Systems Network based on each customer’s portion of total network usage. Total costs includes bandwidth, circuits, maintenance, sustainment costs, network support and operations labor, outage monitoring, and contract management, among others. This approach is also used for DISA’s Global Video Services (video teleconferencing capabilities) and Organizational Messaging Services (command and control messaging). 3. Percentage Markup on Direct Costs: DISA adds a percentage markup on its direct costs as a proxy for indirect costs. DISA uses this approach to calculate some rates for its Computing Services and for its Telecommunications Services and Enterprise Acquisition Services activity groups. Computing Services.There are some services within the Computing Services activity group which DISA charges on a reimbursable basis, such that customers pay the direct cost of the service provided plus an additional percentage of the direct cost to cover general and administrative costs. Telecommunications Services and Enterprise Acquisition Services. DISA charges the customer for the full cost of the contract plus an additional percentage of the direct costs to cover DISA’s indirect costs associated with contract management through the Defense Information Technology Contracting Organization. This fee ranges from 1.75 to 2.5 percent of the contract amount and is based on the expected support costs for associated information technology systems, billing support personnel and systems, financial management, and space and facility costs. This standard contracting fee may change from year to year, but it remains fixed within any given year. DLA reported receiving total Defense-Wide Working Capital Fund (DWWCF) orders for goods and services valued at approximately $40.6 billion in fiscal year 2018. DLA employs around 26,000 military and civilian personnel. DLA provides its services through three activity groups: Energy Management, Supply Chain Management, and Document Services. Approach to Allocating Costs: The Defense Logistics Agency (DLA) allocates direct costs to the individual good or service for which the costs were incurred. For indirect costs, DLA determines whether each cost is associated with providing specific goods or services (such as labor that supports a specific materiel supply chain) or is associated with supporting DLA as a whole (such as the DLA general counsel). DLA uses various methods to allocate these indirect costs, taking into account factors such as the number of employees supporting the provision of a given good or service and the total sales of that good or service. Services Provided: DLA provides fuel and other energy commodities through its Energy Management activity group; consumable materiel (i.e., supplies and parts), distribution services for this materiel, and disposition services for excess property through its Supply Chain Management activity group; and printing, electronic document management and invoicing, and other document services through its Document Services activity group. Indirect Costs: These include costs for information technology systems, facilities, and labor that support the provision of multiple goods and services. Costs for information technology systems and labor that provide enterprise-level support to all of DLA (such as DLA’s accounting system and headquarters staff), among other costs, are also included. The indirect costs included in rates vary among the different goods and services that DLA provides. rates that are calculated by dividing the total processing costs (excluding transportation costs) for items in each weight category by the projected number of units shipped for each category.a uniform percentage across all customers. DLA uses this approach for disposition and some document services, as described below. Disposition. DLA charges each customer a portion of the total direct and indirect costs of providing disposition services based on the customer's portion of total disposition service usage. When applicable, DLA subtracts the revenue it generates through the sale of excess property, reimbursements it receives from customers for hazardous waste management, and funding it receives for Overseas Contingency Operations from the total costs before assigning costs to customers. Document Services Electronic Document Access and Wide Area Workflow (Invoicing). For Electronic Document Access, DLA charges all customers a uniform percentage of its total costs for providing that service. For Wide Area Workflow, DLA charges each customer a portion of the total costs based on the customer’s portion of total system usage. 3. Percentage Markup on Direct Costs: DLA adds a percentage markup on the cost to acquire each good (i.e., the product cost) as a proxy for non-aqcuisition costs associated with that good (i.e., non- product costs). DLA uses this approach for its weapons systems and troop support materiel supply chains. Materiel Supply Chains.To calculate the cost recovery percentage, DLA divides the projected non-product costs for each materiel supply chain by the projected product costs of that materiel supply chain. The rate charged is the sum of the product cost of the good and an additional percentage of this product cost corresponding to the markup percentage. In addition to the contact named above, Alex Winograd (Assistant Director), Karyn Angulo, Martin de Alteriis, Garrick Donnelly, Christopher Gezon, Felicia Lopez, Keith McDaniel, Susan Murphy, Suzanne Perkins, Carol Petersen, Richard Powelson, Lauren Shaman, Kevin Walsh, and Doris Yanger made key contributions to this report. Document Services: DOD Should Take Actions to Achieve Further Efficiencies. GAO-19-71. Washington, D.C.: October 11, 2018. Defense-Wide Working Capital Fund: Action Needed to Maintain Cash Balances within Required Levels. GAO-17-465. Washington, D.C.: June 30, 2017. Bulk Fuel: Actions Needed to Improve DOD’s Fuel Consumption Budget Data. GAO-16-644. Washington, D.C.: September 12, 2016. Department of Justice: Working Capital Fund Adheres to Some Key Operating Principles but Could Better Measure Performance and Communicate with Customers. GAO-12-289. Washington, D.C.: January 20, 2012. Intragovernmental Revolving Funds: Commerce Departmental and Census Working Capital Funds Should Better Reflect Key Operating Principles. GAO-12-56. Washington, D.C.: November 18, 2011. Federal User Fees: A Design Guide. GAO-08-386SP. Washington, D.C.: May 29, 2008.", "summary": "As DOD continues to focus its resources on improving military readiness and modernizing its forces, it seeks to minimize costs associated with its business operations. DFAS, DISA, and DLA are financed through the Defense-Wide Working Capital Fund (DWWCF). Collectively, they provide shared services and goods to their customers, including finance and accounting services; information technology services; and fuel provision and inventory management. Senate Report 115-262, accompanying a bill for the National Defense Authorization Act for Fiscal Year 2019, includes a provision that GAO evaluate the activities DWWCF agencies fund through overhead charges and fees collected from customers. This report evaluates the extent to which DFAS, DISA, and DLA (1) have a process for setting rates to recover costs and provide transparent pricing to customers and (2) clearly delineate roles and responsibilities, measure performance, and assess resource requirements and customer needs. GAO reviewed relevant sections of DOD's Financial Management Regulation and agency documentation and interviewed officials from DFAS, DISA, and DLA and the military departments in comparing the agencies' management practices to the key operating principles for effective management of working capital funds. The Defense Finance and Accounting Service (DFAS), Defense Information Systems Agency (DISA), and Defense Logistics Agency (DLA) use a combination of approaches to set rates that are intended to recover their costs and equitably allocate costs to customers. However, DFAS, DISA, and DLA have not provided transparent pricing to the military departments, which are their largest customers. Each agency annually develops budget proposals designed to recover projected costs and account for gains or losses from prior years. DFAS, DISA, and DLA have taken steps intended to establish an equitable pricing methodology. For example, DLA changed its pricing method for distribution services to align the rates customers pay with DLA's costs of providing the service. However, customers from the military departments said they lack visibility into the factors that determine their overall costs at one or more of the three defense agencies, including how indirect costs are allocated and included in the rates they are charged. GAO's review of cost and rate documentation provided to the military departments also found that they provide high-level information, such as the rates and estimated workloads, and did not include details about the types of costs included or how they are calculated. Specifically, (1) DFAS informational briefings do not describe the types of costs included in rates and how those costs are calculated and allocated. As a result, customers from the Army and Navy said they were confused about why declines in their use of DFAS's services have not resulted in reduced costs. (2) DISA does not include in its documentation the methodology it uses to calculate its rates, making it difficult for officials from the Air Force to determine how they can manage their costs with DISA. (3) DLA does not provide detailed information on the costs included in its rates, making it difficult for customers from the Navy and Air Force to determine how to lower their costs or, in the case of the Air Force, understand the cost implications of DLA's newly announced pricing initiative. Because DFAS, DISA, and DLA share only high-level information on their rate-setting methodologies, the military departments have been limited in their abilities to understand and manage the costs they pay for the services they obtain. By providing more complete information on rate setting, including the calculation and use of costs, DFAS , DISA , and DLA could help their customers better manage their costs and make more informed budgeting decisions. Improved transparency could also help customers anticipate how potential changes to the assumptions underlying rates could affect future costs. GAO also found that DFAS, DISA, and DLA clearly delineate roles and responsibilities, measure performance, and assess resource requirements and customer needs for goods and services, as called for by the three remaining key operating principles for effective working capital fund management. As a result, these agencies are positioned to promote a clear understanding of who will be held accountable for specific tasks or duties, reduce the risk of mismanaged funds, measure their operational performance and identify opportunities to improve performance, and use resources most effectively. GAO recommends that DFAS, DISA, and DLA provide more complete information to customers on their rate-setting methodologies. DOD concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Antibiotics are drugs that work by killing bacteria or slowing their growth. However, some bacteria have developed ways to resist the effects of antibiotics, for example, by preventing antibiotics from entering the cell or pumping them out after the antibiotic enters. Bacteria that are able to survive in the presence of antibiotics will multiply and pass on their new genetic material that confers resistance to future generations of bacteria and, in some cases, to other types of bacteria. Resistance can arise in bacteria in humans, animals, and the environment, including in health care settings, and can spread through contact with infected people or animals, contact with contaminated water, soil or surfaces, or consumption of contaminated food. The spread of antibiotic resistance threatens not only the ability to fight bacterial infections but also threatens to reverse some significant medical gains. For example, in addition to treating infections, antibiotics have allowed for numerous medical procedures, such as joint replacements, caesarian sections, organ transplants, chemotherapy, and dialysis—all of which would be significantly riskier without effective antibiotics. Antibiotic resistance also poses a significant economic burden resulting from the direct costs of treating those with resistant infections and the loss of economic productivity from those who get sick or die. In the 2013 Threats Report, CDC identified 17 bacterial pathogens that the agency considers to be “urgent,” “serious,” or “concerning” because they have developed enough resistance to antibiotics to be considered a threat to human health. (See fig. 1.) According to CDC, certain types of bacteria, called gram-negative bacteria, are particularly worrisome because they are becoming resistant to nearly all drugs that would be considered for treatment. The most serious gram-negative infections can be acquired in hospitals or other health care settings and can cause pneumonia, bloodstream infections, wound or surgical site infections, and meningitis. Nine of the 17 bacterial threats on CDC’s threat list are gram-negative. One of the bacteria CDC considers to be an urgent threat—Clostridioides difficile (C. difficile)—is classified as a threat not because it is resistant to antibiotics, but because it is caused by the same factors that drive antibiotic resistance, such as antibiotic use. CDC estimates that C. difficile alone accounted for 12,800 deaths in U.S. hospitals in 2017. CDC’s 2013 Threats Report also identified one type of fungus—Candida auris—that it considered to be a serious threat (see text box). Candida auris Is a Resistant Fungal Threat Candida auris (C. auris) is an emerging infectious fungus that, according to the Centers for Disease Control and Prevention (CDC), presents a global health threat in part because it is highly resistant to anti-fungal drugs and is challenging to address. C. auris was first identified in Japan in 2009. CDC reported 806 confirmed cases in the United States, as of August 31, 2019. According to CDC, C. auris is highly transmissible and some commonly used hospital surface disinfectants appear to be less effective against C. auris. A CDC official told us C. auris is a good example of an emerging threat that requires more research and associated efforts to properly address. Addressing C. auris is challenging for reasons including the rise of resistance and limitations in diagnostic tests. According to CDC, there are three classes of antifungals available to treat C. auris. However, CDC has identified strains that are resistant to all three classes. A CDC official noted that getting new antifungals to market is challenging because, among other things, the demand for antifungals, relative to antibiotics, is low. Additionally, according to FDA, although reliable tests for identifying C. auris exist, commonly used laboratory tests may misidentify this fungus, posing a barrier to correct diagnosis. In 2018, the Food and Drug Administration (FDA) cleared a test based on mass spectrometry to identify C. auris, but this test cannot characterize resistance. FDA officials told us there are three FDA-cleared tests available for testing for other Candida species’ resistance to fluconazole. However, none of these tests can provide rapid results, such as within an hour. Finally, interpretation of culture-based diagnostic tests, which examine how well bacteria grow in the presence of an antibiotic, is challenging due to the lack of established interpretive criteria for C. auris, by both the Clinical and Laboratory Standards Institute, which promotes the development and use of voluntary laboratory consensus standards and guidelines within the health care community, and by FDA. U.S. spending on antibiotics in health care from 2010 through 2015 was estimated in one study to be nearly $56 billion, ranging from $8.4 billion to $10.6 billion annually. While CDC states that antibiotic prescribing improved nationally with a 5 percent decrease from 2011 to 2016, the agency estimated in 2017 that at least 30 percent of antibiotics used across both outpatient and inpatient settings are still prescribed unnecessarily or incorrectly and, therefore, are considered inappropriate. According to CDC, approximately 85 to 95 percent of the nation’s antibiotic use, by volume, occurred in outpatient settings from 2010 through 2015; and roughly 270 million antibiotic prescriptions— equivalent to 836 per 1,000 persons in the United States—were written in these settings in 2016. (For more information on antibiotic use in the United States, see text box.) Antibiotic Use in the United States A 2017 Centers for Disease Control and Prevention (CDC) report estimates that about 30 percent of antibiotics used in U.S. hospitals are inappropriate (unnecessary or prescribed incorrectly), and as much as 50 percent of antibiotics prescribed in outpatient settings—such as physicians’ offices, emergency departments, urgent care centers, and retail clinics—may be inappropriate. For example, CDC reports that each year, an estimated 47 million unnecessary antibiotic prescriptions are written in physicians’ offices and emergency departments. Most of these unnecessary prescriptions are for respiratory conditions most commonly caused by viruses— including common colds, viral sore throats, and bronchitis—that do not respond to antibiotics, or for bacterial infections that do not always need antibiotics, like many sinus and ear infections. Furthermore, CDC reports that even when antibiotics are needed, prescribers often favor drugs that may be less effective and may carry more risk over more targeted, “first-line” drugs recommended by nationally recognized antibiotic prescribing guidelines. (First-line drugs are the drugs generally recommended for initial treatment for a given diagnosis, often combining the best efficacy with the best safety profile or the lowest cost.) According to CDC, antibiotics are among the most frequently prescribed medications in nursing homes, with up to 70 percent of residents receiving one or more courses of systemic (non-topical) antibiotics in a year; CDC also cites studies showing that 40 to 75 percent of antibiotics prescribed in nursing homes may be inappropriate. CDC further reports that harms from antibiotic overuse include the risk of serious diarrheal infections from C. difficile, increased adverse drug events and drug interactions, and increased risk of infection with antibiotic-resistant organisms. According to CDC officials, “unnecessary” antibiotic use means the antibiotic was prescribed when no antibiotic was needed, based on clinical practice guidelines. “Inappropriate” antibiotic use includes both unnecessary antibiotic use, as well as inappropriate antibiotic selection, dosing, or duration when antibiotics are indicated. CDC officials also told us they consider “misuse” and “inappropriate use” to be synonymous terms. Vaccines Can Also Help Prevent Antibiotic Resistance While we did not include vaccines in the scope of this report, vaccines play a role in helping combat antibiotic resistance because they are designed to prevent infections, including resistant infections. In addition, by preventing infections from occurring, they can reduce the need to use antibiotics, which in turn, can slow the development of antibiotic resistance. For example, according to the Centers for Disease Control and Prevention (CDC), since introduction of the pneumococcal conjugate vaccine among children in 2000, rates of antibiotic-resistant infections caused by certain Streptococcus pneumoniae strains decreased by 97 percent among children under 5 and by more than 60 percent among adults. However, few vaccines are available that target antibiotic-resistant bacteria on CDC’s threat list. In September 2014, the President signed Executive Order No. 13676 (Executive Order), which directed that several federal actions be initiated related to antibiotic resistance. For example, the Executive Order directed the creation of the National Action Plan, which the White House released in 2015, to provide a roadmap for federal agencies to respond to the threat of antibiotic resistance. The National Action Plan set five major goals over 5 years related to (1) slowing the emergence of resistant bacteria and preventing the spread of resistant infections; (2) strengthening national One-Health surveillance efforts to combat resistance; (3) advancing the development and use of rapid and innovative diagnostic tests for the identification and characterization of resistant bacteria; (4) accelerating basic and applied R&D for new antibiotics, other therapeutics, and vaccines; and (5) improving international collaboration and capacities related to the first four goals. In addition, the National Action Plan discusses the importance of preventing and controlling infections, such as through rapid detection, to combat antibiotic resistance domestically and globally (see text box). Within each of these five goals, the National Action Plan contains numerous objectives, sub-objectives, agency-specific milestones, and other performance targets called significant outcomes. For example, the National Action Plan set a significant outcome of reducing inappropriate antibiotic use by 50 percent in outpatient settings and by 20 percent in inpatient settings by 2020. According to the World Health Organization (WHO), effective infection prevention and control measures are a practical and scientific approach to reduce health care- associated infections in patients and health care workers, and help combat antibiotic resistance. Infection prevention and control measures serve as the cornerstone of actions needed to address epidemics, pandemics, and antibiotic resistance. Such measures include implementing hand hygiene practices, providing vaccinations, cleaning and disinfecting hospital rooms, isolating patients with infectious diseases, decontaminating and sterilizing medical equipment, and tracking data about emerging infectious diseases. WHO states that health care-associated infections are a global challenge from which no country or health care facility is immune. The Centers for Disease Control and Prevention (CDC) has taken actions to address and track health care-associated infections, including antibiotic-resistant infections. For example, in 2009, CDC issued guidance for infection control targeting Enterobacteriaceae that may be resistant to carbapenem, a class of antibiotics. In 2018, CDC published a study suggesting that a tracked decline in the proportion of resistant bacteria, including carbapenem-resistant Enterobacteriaceae, observed in some health care settings, could be attributable—at least in part—to actions such as those outlined in its 2009 guidance. In addition, CDC has reported that U.S. hospitals have made major progress since 2005 in declining rates of methicillin-resistant Staphylococcus aureus (MRSA) bacteremia because of infection prevention measures. The interagency CARB Task Force, which was created by the Executive Order to issue and monitor the implementation of the National Action Plan, is co-chaired by the Secretaries of Defense, Agriculture, and HHS, and is additionally comprised of representatives from VA and several other agencies. Representatives from HHS agencies—including BARDA, CDC, CMS, FDA, and NIH—make up nearly two-thirds of the task force’s participants (see table 1). According to the HHS Assistant Secretary for Planning and Evaluation officials who coordinate it, the task force is developing a new National Action Plan that will span the years 2020 through 2025. To provide additional advice to the CARB Task Force and the Secretary of HHS, the Executive Order also created the Presidential Advisory Council on Combating Antibiotic-Resistant Bacteria (PACCARB), which is composed of 15 non-governmental members. The Executive Order also charged the CARB Task Force with providing annual updates to the President regarding progress made in implementing the National Action Plan, plans to address any barriers preventing its full implementation, and recommendations for any new or modified actions, taking federal government resources into consideration. Since 2015, the CARB Task Force has produced four progress reports, which summarize agency actions toward meeting the goals and milestones laid out in the National Action Plan; these reports were provided to the President and are publicly available. Since the National Action Plan was released in 2015, CDC has made progress in expanding surveillance for antibiotic resistance in the United States and abroad. However, the magnitude of the problem and its trends over time remain unknown, in part because of challenges in three areas: (1) tracking antibiotic resistance across all health care settings, (2) reporting complete and timely information on magnitude and trends of antibiotic resistance, and (3) tracking and assessing the global antibiotic resistance threat. To better assess the full extent of antibiotic resistance, CDC has expanded its surveillance of priority bacteria in the United States in order to better assess the full extent of antibiotic resistance since the 2015 National Action Plan was released. CDC tracks antibiotic resistance through several infectious disease surveillance systems in collaboration with state and local health officials, health care providers and facilities, and laboratories. Rather than establishing a single surveillance system for antibiotic resistance, CDC generally incorporates tracking of antibiotic resistance into broader surveillance systems, according to agency officials. The surveillance systems are spread across various divisions within CDC that specialize in specific types of infection or certain settings. (See table 2 for a description of each system and the resistant bacteria it tracks.) According to CDC and other officials and documents we reviewed, including the National Action Plan Year 3 Progress Report, CDC has taken the following actions, among others, to expand surveillance in order to better assess the scope of antibiotic resistance: Established the Antibiotic Resistance Laboratory Network in 2016 to improve testing capacity to better identify antibiotic resistance in the United States. The network consists of 55 state and local (including Puerto Rico), and seven regional, public health laboratories and the National Tuberculosis Molecular Surveillance Center. The network is improving and expanding laboratory capacity response at public health laboratories around the country, as well as at regional centers, according to representatives from two national professional organizations of state and local health officials and epidemiologists. Expanded antibiotic resistance-related efforts in its Emerging Infections Program (EIP), a network that seeks to monitor, prevent, and control emerging infectious diseases. For example, since 2015, more of the existing 10 EIP sites are conducting surveillance for invasive Staphylococcus aureus infections, carbapenem-resistant Enterobacteriaceae, and C. difficile, among others. Separately, the National Action Plan had included a goal for CDC to expand EIP by adding up to 10 sites within 3 years. However, CDC officials told us that in light of resource limitations, they chose to instead increase the number of pathogens reported at existing EIP sites. They told us they determined this was a better use of the limited funds, and that existing EIP sites are sufficient for current EIP efforts related to antibiotic resistance. Updated the domestic tuberculosis surveillance system by incorporating advanced drug susceptibility testing and reporting and by developing capacity for state surveillance systems to report their tuberculosis test data electronically to CDC laboratories. Supported state and local health departments to better track, investigate, and prevent resistant foodborne disease, among other things, through the National Antimicrobial Resistance Monitoring System for Enteric Bacteria (NARMS). For example, the system can now carry out whole genome sequencing for all the pathogens it tracks, which enhances its detection and response capabilities, such as by expanding CDC’s ability to detect new and emerging resistance, according to CDC officials. Launched the Enhanced Gonococcal Isolate Surveillance Program (eGISP), which augments the main Gonococcal Isolate Surveillance Program (GISP). Whereas GISP only collects samples from the urethras of men with symptoms of gonorrhea, in select sexually transmitted disease clinics, eGISP also collects samples from women and from other sites on the body, such as the throat. The specimens are sent to regional laboratories for resistance testing. CDC has also worked with international partners to expand surveillance of antibiotic resistance abroad. These efforts involved CDC collaborations with WHO, the European Center for Disease Prevention and Control, the government of the United Kingdom, other governments, and other multi- country efforts, such as the Surveillance and Epidemiology of Drug- Resistant Infections Consortium and the Transatlantic Taskforce on Antimicrobial Resistance (TATFAR). The collaborations aimed to develop technical guidance to help improve surveillance in other nations and to organize an international forum. CDC also launched its Antibiotic Resistance (AR) Solutions Initiative, which invests in national and international infrastructure to address resistant infections across health care settings and communities and from food. CDC faces three general challenges in tracking and reporting trends in antibiotic resistance. First, it faces limitations in data reporting and resistance testing from hospitals, as well as challenges ensuring that its resistant gonorrhea surveillance system is representative of the U.S. population. Second, CDC faces challenges in reporting complete and timely information on the magnitude of and trends in antibiotic resistance. Finally, CDC faces challenges to detecting resistance threats abroad. The first challenge CDC faces in tracking trends in resistance is addressing low hospital participation in a new option of CDC’s National Healthcare Safety Network (NHSN) system intended to address some limitations in NHSN. NHSN is, among other things, an online system for tracking health care-associated infections. It provides facilities, states, regions, and the nation with data needed to identify problem areas, measure the progress of prevention efforts, and ultimately eliminate health care-associated infections, according to CDC. Patients in settings such as hospitals and long-term care facilities (e.g., nursing homes) in many cases already have a weakened immune system or an underlying illness, making an antibiotic-resistant infection especially dangerous, according to the Centers for Disease Control and Prevention (CDC). A high proportion of the morbidity and mortality associated with antibiotic resistance is seen in health care-associated infections. Tracking resistance in health care settings is therefore critical to national surveillance efforts. CDC established three modules within NHSN that allow hospitals to report select antibiotic-resistant infections, among other things, which include reporting required by states or by CMS, according to agency officials. Two modules track patients who have an infection associated with a medical device or resulting from a surgical procedure. Hospitals only report on resistance in these modules for specific combinations of antibiotics and bacteria, such as carbapenem-resistant Enterobacteriaceae. The third module tracks certain hospital patients who test positive for certain multidrug-resistant infections, including methicillin- resistant Staphylococcus aureus (MRSA)—a type of bacteria found on people’s skin that is usually harmless but can cause serious infections, according to CDC. However, according to CDC, many antibiotic- resistant infections detected during hospital care do not fall into one of these three modules and therefore would not be captured in NHSN, limiting CDC’s ability to identify important new resistances or trends. In 2014, to help address this limitation, CDC officials told us they introduced a new option for hospitals to report data on antibiotic resistance—the Antimicrobial Resistance Option (AR Option). This option allows for reporting of data on antibiotic resistance for certain bacteria, regardless of whether the patient has a health care-associated infection. In contrast to the other three modules, reporting to the AR Option is voluntary. As a result, while about 86 percent of the 17,529 eligible U.S. health care facilities participate in at least one of the older three antibiotic-resistance reporting modules, only about 10 percent of the 6,836 eligible hospitals participate in the newer, voluntary AR Option, according to our analysis of NHSN hospital participation data as of January 2020. The hospital participation rate among U.S. states and territories ranged from no participation (in nine states and territories) to about 27 percent. Representatives from a national association of state public health officials we interviewed said that this low rate limits the value of the data, a view that echoed the findings of a 2018 report by the Joint Public Health Informatics Task Force. CDC officials acknowledged that participation in the AR Option is low and cited reasons for this, including hospital resource limitations, and—in many cases because participation is voluntary—because hospitals do not prioritize submitting data to the AR Option. According to CDC officials, it is particularly challenging for many smaller hospitals and Indian Health Service facilities with resource constraints to participate, as it requires significant information technology investment. The Joint Public Health Informatics Task Force report noted two other common challenges: low capacity for information technologies needed to support data submission to the AR Option, and a lack of motivated leadership, such as a facility “champion,” to oversee the development and maintenance of needed reporting infrastructure. For example, the maintenance of reporting infrastructure could address changes to electronic medical records that are not immediately compatible with the AR Option reporting format. CDC officials told us the agency is taking some steps to increase participation in the AR Option. For example, it is encouraging the over 1,500 hospitals (as of December 31, 2019) that are participating in a related reporting effort—known as the Antimicrobial Use Option (AU Option)—but not in the AR Option to participate in both. In addition, the agency is working with vendors of equipment and electronic health record software to make it easier for hospitals to participate in the AR Option. One of CDC’s goals for the AR Option is to use reported data to conduct regional and national assessments of resistance. To help meet this goal, officials said they would like participation by all eligible hospitals in the AR Option, but they have not determined the needed participation rates or appropriate distribution of participating hospitals. Our past work has shown that leading practices for federal strategic planning include articulating specific goals, establishing a method to assess progress toward these goals, and aligning the plans and goals with the agency’s mission. By taking steps to determine the participation rates and distribution of participation hospitals needed for CDC to meet its goal of conducting regional and national assessments of antibiotic resistance of public health importance, CDC would have more reasonable assurance that it can achieve its goal. The second challenge CDC faces is ensuring representativeness of its resistant gonorrhea surveillance system. CDC has classified resistant gonorrhea as one of the most urgent antibiotic-resistance threats in the nation, affecting over half a million patients annually. According to the agency, resistant gonorrhea warrants this designation because of the limited remaining treatment options, the high number of gonorrhea infections, potential adverse outcomes (such as increased transmission of HIV), and the prospect that gonorrhea may become incurable if new resistance arises and spreads. The Urgent Threat of Resistant Gonorrhea According to the Centers for Disease Control and Prevention (CDC), gonorrhea is the second most commonly reported notifiable disease in the United States, with over 500,000 infections reported in 2017. However, CDC estimates that the true number could be as many as 820,000 each year. In addition to being a very common infection, gonorrhea is developing resistance to treatment options. As recently as 2006, CDC had five recommended options, but it estimates that nearly half of U.S. infections are now resistant to available antibiotics, including combinations. Consequently, it now recommends only one regimen. In 2014, a case of dual-therapy failure was reported in the United Kingdom, and in February 2018, a similar case in the United Kingdom was reported that also failed to respond to the last-resort therapy, spectinomycin, resulting in treatment failure. As of June 2019, CDC reported that it had not received any reports of verified clinical treatment failures to any cephalosporin in the United States. It is not clear, however, that GISP data are representative of the general U.S. population because GISP draws on a limited sample of that population. Specifically, GISP collects culture specimens—-called isolates—and accompanying epidemiologic data from only the first 25 men with inflammation of the urethra consistent with gonorrhea visiting each participating sexually transmitted disease clinic each month. It does not collect culture specimens from women. In addition, the number of participating clinics each year has varied from 21 to 30 (see fig. 2 for the current sites). CDC estimates that the cases of gonorrhea identified through GISP surveillance represent only about 1 to 2 percent of all reported cases of gonorrhea in the United States each year. Further, the GISP sample design also over-represents cases in the western United States, where antibiotic-resistant gonorrhea has tended to initially emerge, according to CDC. According to CDC, this design allows for more rapid detection of emerging resistance by ensuring a sufficient sample size from the western United States because resistance tends to emerge from that area. CDC has two projects—Strengthening the United States Response to Resistant Gonorrhea (SURRG) and eGISP— intended to, among other things, enhance domestic gonorrhea surveillance and learn more about the representativeness of GISP through limited testing of women and of body sites other than urethras, respectively. However, CDC’s current methodology may limit its ability to establish a representative trend. According to CDC officials, GISP could improve its representativeness by adding clinics or covering more of the population at its current sites. However, efforts to expand GISP would be difficult due to limited local capacity (see text box). Barriers to Expanding the Gonococcal Isolate Surveillance Program (GISP) GISP currently tracks a limited sample of the U.S. population. According to Centers for Disease Control and Prevention (CDC) officials, a more thorough expansion of GISP would be more difficult because of limited local capacity to conduct culture-based testing for resistance in gonorrhea. Specifically, laboratories increasingly use newer gonorrhea testing technology that gives more rapid results but cannot currently be used to test for resistance. This trend has contributed to the reduced capability of many laboratories to perform the gonorrhea culture-based testing for antibiotic susceptibility testing, to the point that many clinics cannot collect specimens for testing, according to CDC officials. Furthermore, officials said that adding new clinics to GISP would require financial and other resources for, among other things, establishing culture testing for resistance and information technology needed to report data to the system. Most gonorrhea cases are diagnosed outside sexually transmitted disease clinics. However, expanding GISP to non-sexually transmitted disease clinic sites could be particularly costly and inefficient, officials said, because these sites tend to see many fewer gonorrhea cases per year compared to sexually transmitted disease clinics; therefore they may not be able to contribute significant data to GISP. Through the Strengthening the United States Response to Resistant Gonorrhea (SURRG) project, CDC is currently exploring options to work with states to enhance gonorrhea testing capacity. This program was established in 2016 but has not received the funding needed to expand capacity to the extent CDC had planned. In addition, physicians and other providers have limited time to devote to data collection and reporting needed to participate in GISP. CDC officials also told us the reimbursement rates for providers for these services are inadequate. CDC has taken some steps to assess the representativeness of the current GISP design, but it has not conducted a comprehensive study to assess the representativeness of the trends identified in GISP. A 2015 CDC evaluation concluded that the representativeness of GISP was “good” on a scale of fair, good, or great. However, the evaluation covered only part of fiscal year 2014 and consisted of a limited comparison of selected demographic characteristics captured in gonorrhea cases identified in GISP to those captured through the National Notifiable Diseases Surveillance System, according to CDC officials, and which has its own limitations. Further, the results of this evaluation have not resulted in any changes to the GISP design. CDC officials told us they hope to learn more about the representativeness of GISP urethral isolates from testing women, patients in non-sexually transmitted disease clinic sites in the SURRG project and eGISP, and testing at other body sites, and then comparing some of these results to those of GISP. However, these efforts overall were not specifically designed to fully assess the representativeness of GISP and may not provide a sufficient assessment for impacting changes to the GISP design. CDC’s guidelines of efficient and effective public health surveillance systems state that, in order to be representative, the data from a public health surveillance system should accurately reflect the characteristics of the health-related outcome—such as resistant gonorrhea—under surveillance. A more precise evaluation of the representativeness of the surveillance system can be done via carefully designed studies to obtain complete and accurate data for the health event in question—namely, the urgent threat of antibiotic-resistant gonorrhea. By evaluating the surveillance system for resistant gonorrhea to ensure that it includes measures of its representativeness, such as by comparing the trends in the sample population with those in the overall U.S. population, using specially designed studies if needed, CDC would have better assurance that the trends detected in GISP accurately reflect the characteristics of the health-related outcome the system is designed to monitor. In addition to the limited design of GISP, CDC faces the challenge of competing priorities under reduced funding that precluded it from completing its plans to expand the SURRG project. The SURRG expansion was designed to address a National Action Plan goal of controlling resistant gonorrhea, among other things, but also affects surveillance, as CDC officials told us SURRG was established to address some limitations in GISP surveillance. Specifically, one of the plan’s milestones assigned to CDC is to maintain advanced capacity for rapid response to antibiotic-resistant gonorrhea for at least 20 state health departments. Such capacity includes detection, diagnosis, and investigation of suspected resistant cases within their state or region and assistance for health care providers in appropriately treating infected patients. CDC officials told us that because they received about half of the appropriations they had requested, CDC had to make cuts in some of their projects, and SURRG was one of those that CDC chose to reduce. Eight SURRG sites, rather than the 20 recommended by the National Action Plan, collect and analyze data. However, in its progress reports covering the first 4 years of the National Action Plan’s implementation, the CARB Task Force did not identify plans to address barriers related to expanding the SURRG project. The CARB Task Force coordinators told us that the progress reports have not identified plans to address barriers largely because the task force focused on reporting the agencies’ accomplishments in implementing the National Action Plan. The coordinators also said that, in response to our inquiries during this review, the task force intends to identify agencies’ plans for addressing barriers in the progress report to be published in fall 2020. The Executive Order directs the CARB Task Force to provide annual updates to the President on federal government actions to combat antibiotic resistance, including progress made in implementing the National Action Plan, plans for addressing any barriers preventing its full implementation, and recommendations for any new or modified actions, taking federal government resources into consideration. Without reporting its plans to address such barriers, the CARB Task Force has not provided all the information required by the Executive Order and has not fully carried out its role to facilitate and monitor implementation of the National Action Plan, which may reduce the effectiveness of federal efforts to combat antibiotic resistance. The third challenge CDC faces tracking antibiotic resistance is addressing limitations to the use of test results in surveillance in health care settings. For example, some health care facilities are not using the most up-to-date testing methods for determining whether the bacteria causing an infection are resistant to certain antibiotics, according to CDC officials and a report from the Antibiotic Resistance Surveillance Task Force. In addition, laboratories may only report an interpretation of the test result to CDC (e.g., whether the bacteria is resistant or susceptible to an antibiotic) and not the quantitative results (e.g., measures of the growth of bacteria in the presence of the antibiotic). This presents a challenge for comparing data from different laboratories, since they may not be using consistent testing thresholds for determining antibiotic resistance. Another limitation is that some test equipment may be designed to give limited results for the purposes of guiding treatment recommendations and stewardship efforts, which may also limit the information available to CDC. For example, the test may inform the user that the infection is susceptible to one antibiotic but “suppress” information on susceptibility to other antibiotics, in order to guide the user toward treatment with the preferred first-line treatment. The Antibiotic Resistance Surveillance Task Force report noted that some suppression is done by the testing equipment itself and some by software systems that record, manage, and store data for clinical laboratories. CDC officials told us they are working with some diagnostic test manufacturers to explore these issues and develop solutions to address them. The Antibiotic Resistance Surveillance Task Force is also working to address the diagnostic test challenges related to antibiotic resistance surveillance. CDC also faces challenges in reporting timely and complete information on the magnitude of and trends in antibiotic resistance in the agency’s Threats Reports. One challenge is in providing information in these reports on the uncertainties in reported numbers of deaths from antibiotic- resistant infections. Another challenge is in issuing such reports in regular, timely intervals. As a result of these challenges, among others, the true magnitude of, and trends in, antibiotic resistance over time are unknown, including trends in various places and among people with various characteristics. Surveillance for antibiotic resistance is complex and costly, according to experts at our meeting, CDC officials, and literature we reviewed. Experts told us such surveillance encompasses diverse pathogens, diseases, and health care settings and requires a variety of data sources and collection efforts. Furthermore, experts from our meeting told us the fundamental data required—such as data on the number of illnesses and deaths attributable to resistance and data on related health care costs—are currently insufficient. One expert added that there is a lack of real-time monitoring data, such as data that are available within hours or days of being generated. The data gaps are especially large for infections acquired in the community, as opposed to in a health care setting, because there is very limited tracking of such infections and whether they are resistant. As a result, CDC officials said, it is challenging to provide ranges of uncertainty, a critical component of any effort to measure and report on magnitude and trends. Neither the 2013 Threats Report nor the 2019 Threats Report provided quantitative measures of uncertainty, such as confidence intervals, for CDC’s estimates of morbidity and mortality resulting from antibiotic- resistant infections. For example, the report stated that there are at least 23,000 deaths a year as a direct result of antibiotic-resistant infections, but it did not include an upper limit or a single point estimate for this number. Similarly, the 2019 Threats Report stated that there are at least 35,900 deaths a year, without an upper limit or a single point estimate. A recent re-estimate by a group of scientists has put the likely minimum number of deaths annually in the United States at approximately 153,000, or about four times the 2019 CDC minimum estimate. CDC officials told us that because of several limitations, its estimates were the best that could be derived from the data available. For example, for the 2013 Threats Report, CDC only had data from a national hospital survey intended to produce estimates of all health care-associated infections and indirect estimates of the proportion of infections that were resistant. These data did allow CDC to calculate confidence intervals for infections by specific pathogens, but this information was not disclosed in the Threats Reports. Because the data sources were not intended for this purpose, the 2013 intervals were wide, from approximately 26 percent to 380 percent of the point estimates for each pathogen. CDC officials told us they elected not to include these ranges of uncertainties to avoid confusion in the 2013 Threats Report, because the report was intended for a variety of audiences, including the general public. Officials told us they planned to provide confidence intervals in an appendix of the 2019 Threats Report, but they did not. CDC officials explained that they elected not to include confidence intervals in the 2019 Threats Report because several publications are pending that provide more granular data for many of the estimates included in the report. It is thus unclear whether CDC plans to include any measures of uncertainties in future Threats Reports. Federal standards for agency dissemination of information it produces stipulate that when information products are disseminated, error estimates are calculated and disseminated to support assessment of the appropriateness of the uses of the estimates or projections. Providing measures of uncertainties in antibiotic resistance estimates, such as standard errors or confidence intervals, as appropriate, in its Threats Reports would help CDC and others compare information within and across reporting efforts, without having to consult multiple documents over time. CDC and others could use this information to draw appropriate conclusions about the characteristics of antibiotic resistance in the United States, including limitations associated with reported findings and conclusions. Additionally, CDC does not have a plan for timely, regular issuance of their Threats Reports. It took CDC over 6 years to update the 2013 Threats report. CDC officials told us this length of time between reports was in part because, following issuance of the 2013 Threats Report, the agency was focused on implementing priority actions to improve antibiotic resistance surveillance data, including those efforts prescribed by the National Action Plan. In some cases, implementing these actions involved new data collection efforts that took time to establish, including that it can take up to 2 years to get new surveillance variables cleared by the Office of Management and Budget (OMB), CDC officials told us. In addition, CDC officials said it is time consuming to coordinate across the decentralized structure of antibiotic-resistance tracking at CDC to compile a consolidated report. However, lack of timely, regular updates may affect the information available to the public as well as policy-makers. For example, the 2013 Threats Report stated that there are at least 23,000 deaths a year as a direct result of antibiotic-resistant infections. The 2019 report stated the number of deaths each year to be at least 35,900 deaths a year. This report also revised the 2013 estimate from 23,000 to 44,000 deaths a year, suggesting a nearly two-fold revision to the initial 2013 estimate. CDC officials told us they would like to publish the report more frequently than every 6 years, and that it is reasonable they would develop such a plan for frequency of publication following the 2019 report. However, they said the agency does not currently have a plan for how often it will release future consolidated reports. CDC’s attributes of efficient and effective public health surveillance systems include timely data dissemination for planning, implementing, and evaluating public health policies and programs. By developing a plan for more frequent dissemination of consolidated reporting on priority pathogens at regular intervals, CDC would have more timely trend data and other information necessary for users of the data, including policymakers, to prioritize, plan, implement, and evaluate public health actions to address antibiotic resistance. In October 2015, the World Health Organization (WHO) launched the Global Antimicrobial Resistance Surveillance System (GLASS). The objectives of GLASS are to foster national surveillance systems and harmonized global standards and estimate the extent and burden of antimicrobial resistance globally by selected indicators, among other things. As of November 2019, 86 countries were enrolled in GLASS, a 25 percent increase over 2018. Participants were in various stages of economic development (13 lower-income countries, 23 lower-middle- income countries, 17 upper-middle-income countries, and 33 high-income countries) and in all WHO regions. Seventy-five countries provided descriptive information on their surveillance systems for tracking antimicrobial resistance, and 57 countries provided resistance data for 2018. antibiotic resistance from the national surveillance systems of some countries are incomplete because of a lack of capability and resources for implementing standardized protocols, according to WHO officials. Moreover, most information on antibiotic-resistant infections is limited to laboratory test data and does not include epidemiological data, such as data on the patient and location, which could provide additional insight about the circumstances around the resistant infection. Also, a lack of a sampling strategy for the detection of cases that are antibiotic-resistant may bias the representativeness of the data and interpretation of results. Specifically, when case identification is done only on the population of patients that seeks medical care and is tested, or when testing of the population varies such as across health care settings, the incidence and trends determined from this population may not represent the total population of concern. Aggregated data reporting. Some countries report aggregated, rather than isolate, or infection-level, data to the WHO’s Global Antimicrobial Resistance Surveillance System (GLASS), a practice that WHO officials stated creates a challenge for data analysis and results interpretation. According to officials, such aggregation limits statistical analysis that can be performed and limits analysis of factors such as the specific antibiotic-resistant bacteria, or the age or gender of the patient, among other things. Surveillance is a complex function. Many different health care and public health professionals are involved in the multistep process for generating data, according to a WHO report on GLASS. According to WHO officials, obtaining the staff commitment and training needed to ensure high-quality data can pose a challenge to public health agencies and health care organizations. As we noted above, CDC has worked with, and continues to work with, international partners to expand surveillance of antibiotic resistance abroad, including through U.S. participation in GLASS. For example, CDC has helped develop technical guidance for surveillance programs in other countries and has organized international forums for surveillance. CDC officials also told us portions of domestic surveillance systems data collection include collection of patient travel history. Federal agencies have helped advance the development of new FDA- authorized tests and the use of existing tests for diagnosing antibiotic- resistant infections, but these efforts have limitations. Specifically, HHS and DOD have funded studies and taken other steps to advance testing, but they have not defined leadership, roles, and responsibilities to address a key barrier to the use of tests: a lack of clinical outcome studies. FDA has taken additional steps to advance testing; however, it has not regularly monitored test updates. HHS and DOD have awarded grants and contracts for the development of new FDA-authorized tests for diagnosing antibiotic-resistant infections. Some of these awards address specific needs in the current availability of FDA-authorized tests, while others support more general research and development efforts. In addition, these agencies have taken steps to help reduce the chances of duplicative funding. According to experts, tests for antibiotic resistance not only help clinicians decide what antibiotics to use, they also provide important information for surveillance, including the number of cases of resistant infections in a population and the mechanisms of resistance. to the 2013 Threats Report. Differentiate between viral and bacterial infections. Such a test would be useful primarily in preventing use of antibiotics for viral infections, which can contribute to the development of resistance in bacteria, among other things. HHS and DOD have awarded funding to address these needs. For example: CARB-X—a program supported by NIH and BARDA within HHS—has awarded funding to a company to develop a rapid test to both diagnose gonorrhea and test for antibiotic resistance. CARB-X is funding other companies to, among other things, develop rapid testing for identification of and resistance in bloodstream infections, including for some priority bacteria. In September 2016, NIH and BARDA announced the Antimicrobial Resistance Rapid, Point-of-Need Diagnostic Test Challenge. As of December 2019, there were five finalists, working on such projects as developing a rapid test to differentiate viral from bacterial infections and developing a test that can identify or detect antibiotic-resistant bacteria, including antibiotic-resistant gonorrhea. Within DOD, the Defense Advanced Research Projects Agency officials told us that the agency used fiscal year 2015 funding on contracts for the development of rapid molecular tests for resistant gonorrhea and to distinguish between viral and bacterial infections. Federal agencies have also funded more general research and development efforts related to resistance testing. For example: NIH officials told us their agency has supported extramural projects related to the development of tests for antibiotic resistance by issuing grants and entering into contracts since fiscal year 2015. Separately from the Antimicrobial Resistance Diagnostic Challenge, BARDA entered into contracts with three organizations to develop tests focusing on the advanced stages of test development, including clinical trials, according to BARDA officials. Within DOD, the Defense Threat Reduction Agency is funding three projects using Other Transaction Authority or direct funding to a DOD Service laboratory, for developing tests. Federal agencies have also taken steps to help reduce the chances of duplicative funding, including working with some international efforts to develop tests, according to agency officials. For example, NIH reviews current and pending support of key project personnel prior to issuing of any research award, to help ensure NIH support complements support from other agencies and organizations. Similarly, officials from HHS’s Office of Global Affairs worked during the creation and launch of the NIH- BARDA challenge and an analogous United Kingdom innovation foundation competition called the Longitude Prize to help ensure these programs were designed to support different aspects of needed diagnostics. HHS has funded some studies to assess the extent to which testing patients to identify whether they have antibiotic-resistant infections leads to improved clinical outcomes, such as more effective treatment for patients or more judicious use of antibiotics. However, HHS has not identified relevant leadership, roles, and responsibilities among the HHS agencies that could fund such studies. Clinical outcome studies are important for encouraging the use of diagnostic tests for antibiotic resistance, among other things, because such studies can demonstrate the benefits of those tests. According to PACCARB, there is very limited information on why clinicians sometimes forgo diagnostic testing, but one possible explanation is that there may be limited data demonstrating the value of such testing. In the absence of such data, a clinician may choose to treat the patient immediately rather than using a test for antibiotic resistance that has unknown value. Research into the clinical outcomes associated with such testing could therefore be used to help promote the use of those resistance tests that are found to be beneficial. As a result, patient care could be improved and clinicians could be guided towards appropriate antibiotics to prescribe. Two HHS agencies have awarded grants for studies on the clinical outcomes of resistance testing, according to agency officials. For example, NIH provided grant support for a study that found, among other things, that using a rapid blood test for a range of potential bacteria and antibiotic resistance led to more judicious use of antibiotics. Similarly, officials from the Agency for Healthcare Research and Quality (AHRQ) stated that the agency is funding investigator-initiated grant studies to assess the impact of tests on antibiotic stewardship. However, agency officials only mentioned these and a few other examples of studies they have funded on clinical outcomes. International Needs for Diagnostic Tests for Antibiotic Resistance To better understand international needs for antibiotic resistance tests, we interviewed officials from international organizations and the Office of Global Affairs within the Department of Health and Human Services (HHS). A Public Health England official told us that United Kingdom users are not confident that these tests will have a clinical impact or be cost effective. Similarly, an official from a trade organization of British medical test manufacturers told us that the value of tests for antibiotic resistance needs to be captured and disclosed, especially because people are more willing to pay for treatment than for tests. However, other factors could also be important in determining which tests will be useful internationally. World Health Organization officials told us that they are working to determine what characteristics health care providers worldwide identify as key to making tests useful, so industry can develop such tests. They noted that tests designed for use in the United States may not be suitable for use in other countries. They also noted that laboratories in developing countries may not have the capacity to culture bacteria, so many need to use culture- independent tests. Office of Global Affairs officials told us that a big challenge is developing accessible tests for use internationally. Their ideal test would be inexpensive, rapid, and capable of point- of-care use. They noted that cost and usability are the barriers to test use, not technology, and that use of existing tests remains limited, including within the United States. Agency officials and experts agree that more needs to be done to evaluate clinical outcomes associated with use of diagnostic tests for antibiotic resistance. For example, in 2017, PACCARB reported that “there is a lack of clinical and economic outcome studies showing that any diagnostic test could prevent the emergence of antibiotic-resistant bacteria and would be cost effective.” Officials we interviewed from AHRQ, BARDA, CDC, FDA, and NIH all agreed with that PACCARB statement. Additionally, experts told us that such studies are lacking but important for advancing the use of tests. For example, one health care organization official told us the decision to adopt a test is based at least in part on whether there will be a clinical benefit. An infectious disease expert noted that to provide incentives for test use there needs to be some evidence that tests affect and improve care, but that most tests do not come with any evaluation of how they perform in practice. International organizations expressed similar opinions. One reason for the relatively low number of studies is that those agencies that could conduct or fund diagnostic outcome studies have not clearly identified leadership, roles, and responsibilities for doing so. Although they agree that more such studies are needed, they have not identified which agency or agencies should take the lead, and what the roles of the other agencies should be. Instead, agencies have offered differing views on what each agency could do. For example, BARDA officials told us their agency has not funded such studies because it generally does not play a role in test adoption. BARDA officials, as well as officials from DOD and NIH, said that CDC should play a role in funding or conducting the studies. However, CDC officials told us that a lack of resources has prevented their agency from doing so, and that the responsibility should fall at least partly on BARDA. Our previous work shows that key practices for interagency collaboration include identifying a lead agency (or, if leadership is shared, clearly identifying roles and responsibilities among the lead agencies), as well as clarifying the roles and responsibilities of all participating agencies. By taking these actions, agencies—including AHRQ, BARDA, CDC, FDA, and NIH—could more effectively address the need for clinical outcome studies. Those studies, in turn, could help demonstrate the value of diagnostic tests for antibiotic resistance, potentially increasing their use, improving patient care, and enhancing stewardship efforts. CMS and FDA have taken some steps to advance the use of tests, including those to identify antibiotic-resistant bacteria. For example, FDA established a Payor Communication Task Force, which helps facilitate communication between test manufacturers and payors. Such communication is important because payors decide whether tests will be covered by insurance, among other things. According to an FDA web page, by communicating with payors, test manufacturers could, for example, learn what data payors need to approve a test for coverage and then use this information to design clinical trials to provide that information. This process could reduce the time between when a test is cleared or approved by FDA and when it is covered. A similar step FDA and CMS took to advance the use of tests was to extend the Parallel Review program indefinitely, a move they announced in 2016. This program established a mechanism for FDA and CMS to simultaneously review clinical data, with the aim of reducing the time between FDA’s approval and CMS’s decision on whether to pay for the test. Experts told us challenges remain with test payments that may result in lower test use. For example, a PACCARB report states that “currently, for many diagnostic tests is not aligned with the value of the test,” and noted that supplementing payments for tests could drive test development and use. BARDA officials also told us that a major factor affecting adoption of new tests is the cost of the test relative to reimbursement. Additionally, experts, including those at our meeting, told us that test payments remain insufficient to encourage broad test use. For example, two experts from our meeting said that there is not always a clear link between the medical value of a test and the payment level for that test. One of these experts added that their laboratory decided not to adopt a test because low payment levels relative to costs made doing so a money-losing proposition. Three other experts we interviewed agreed that disparities between cost and payment can discourage test adoption. Regarding federal payments for tests involving CMS and their payments through Medicaid and Medicare, there are limits to CMS’s ability to address any disparities. For example, CMS officials told us the payments for some tests are based on a weighted, median, private-payor rates pursuant to the Protecting Access to Medicare Act of 2014, so CMS cannot specify the methodology used to set those rates. Further, for inpatient tests, Medicare pays hospitals a single, bundled payment per patient stay, which is based on multiple factors, including the patient’s diagnosis and treatment strategy, rather than on a specific service. As such, a separate payment for individual tests is not made under Medicare. FDA has taken steps toward the development of FDA-authorized tests for resistance for newly approved antibiotics—a process that currently can take months to years, according to experts and agency officials. The delay stems in part from the need for a critical testing threshold known as a breakpoint—the threshold that is used to help a clinician decide whether or not a pathogen is resistant to the antibiotic (see text box). The breakpoint of a new antibiotic is generally finalized only when FDA has approved the antibiotic. This means that breakpoints may often not be available for test manufacturers until after a new antibiotic is FDA- approved. As a result, test manufacturers generally may not be able to complete developing FDA-authorized culture-based tests for resistance to a specific antibiotic until after the antibiotic is commercially available. The result is that the development of such culture-based tests may be generally delayed even after the new antibiotic is approved by FDA. This delay could affect the ability of clinicians to treat patients. For example, according to an expert, such a delay could lead to underuse of a newly available antibiotic, among other things, because a clinician may not be willing to prescribe the antibiotic without test results to guide treatment. How Breakpoints Are Used to Interpret Tests According to officials from the Food and Drug Administration (FDA), breakpoints, also referred to as “susceptibility test interpretive criteria,” are used to define susceptibility and resistance to antibiotics to help guide patient care. Culture-based tests rely on breakpoints to provide a determination of resistance to clinicians. In the United States, breakpoints (based on clinical or microbiological data) are established by standards- development organizations such as the Clinical and Laboratory Standards Institute (CLSI) and FDA. One example of how breakpoints are used involves the Kirby-Bauer disk diffusion test. This test is conducted by spreading bacteria on a laboratory agar plate containing bacterial nutrients, and then placing paper disks containing a known amount of antibiotics on the “lawn” of bacteria. Plates are observed after overnight incubation to determine the extent of bacterial growth. Closer to the disk, there is a higher concentration of antibiotic, and the concentration declines with distance. Around most disks, there is a “zone of inhibition,” where the concentration of antibiotic is too high for bacteria to grow. After allowing the bacteria to grow for a defined period of time, the diameter of the zone of inhibition is measured in millimeters. Procedure for Assessing Antibiotic Resistance Using Breakpoints If the diameter is larger than or equal to the breakpoint, then the strain of bacteria is considered susceptible to the antibiotic, suggesting that the antibiotic can be used to treat infections caused by that strain. If the diameter is smaller than the breakpoint, then the strain is considered resistant, suggesting that the antibiotic should not be used. According to FDA, in most cases, there is a range of “intermediate” or “susceptible dose-dependent” diameters for which treatment might be effective. Other types of culture-based diagnostic tests for resistance have analogous breakpoints for interpreting the test. For example, the minimum inhibitory concentration—the lowest concentration of an antibiotic that prevents growth of bacteria—can be compared to a breakpoint to establish whether the bacteria are considered resistant. In addition to antibiotic developers waiting until FDA approves an antibiotic before a breakpoint is finalized, there are technical hurdles in developing a test for some new antibiotics, according to FDA officials. For example, it may be challenging for certain automated test manufacturers to address unique growth properties of certain bacteria in the presence of specific antibiotics or combinations of antibiotics. According to a test manufacturer, these hurdles include the need for additional studies, and such studies may not be straightforward because of the need to determine what clinical data FDA requires. In addition, in the case of automated tests, a representative from a test manufacturer association told us the software used to run and interpret a new test needs to be revised, which can be time consuming. The delay between approval of an antibiotic and the availability of a test for resistance could result in suboptimal treatment and increase burdens on the health care system. For example, one expert stated that during this delay, laboratories need to create or modify tests and then validate those tests instead of using a FDA-authorized test, which increases the time required and places demands on facility personnel and budgets. This expert added that to conduct validation studies, the laboratories need a variety of samples for testing, called “isolates,” which may not be available. A second expert said that the delay leads to both overuse and underuse of the new antibiotic: in the absence of a test, some clinicians will prescribe the antibiotic when it may be inappropriate, leading to overuse; some other clinicians refrain from prescribing the antibiotic, even if appropriate, leading to underuse. To help address this delay, FDA has created a process known as coordinated development, whereby test manufacturers can submit a coordinated development plan to FDA describing the test manufacturer’s intent to coordinate with the antibiotic manufacturer. The plan is submitted prior to, or shortly after, submission of an application to market a new drug. Under the coordinated development program, FDA shares breakpoint information from the antibiotic manufacturer with a prospective test manufacturer. It then reviews the test application at the same time as the antibiotic application and takes other steps to facilitate more timely clearance of the test. FDA officials told us this process has significantly reduced the delay between approval of the antibiotic and clearance of the test. Another FDA step to help test manufacturers speed development of tests is the establishment, in collaboration with CDC, of a centralized repository of bacterial strains with well-characterized antibiotic resistance profiles. These strains are available to test manufacturers and others to help them design, validate, and evaluate tests by checking that they give the correct results for bacteria whose profile of antibiotic resistance is known. Finally, FDA officials also said that they offer pre-submission advice, whereby a test manufacturer can ask for initial guidance on the design of clinical studies for their tests. In the United States, breakpoints are established and updated by organizations such as the Clinical and Laboratory Standards Institute (CLSI). After CLSI establishes a breakpoint, FDA may review and recognize the breakpoint, according to FDA officials. Test manufacturers rely on breakpoints recognized by FDA to support marketing authorization of their tests. An expert who works for CLSI identified more than 50 breakpoints that have not been recognized by FDA, and for which CLSI considers FDA recognition important in order to help make FDA-authorized tests available. Experts, including one from our meeting, cited the following examples of breakpoints needing recognition: CDC recommends a dual therapy of antibiotics—azithromycin and ceftriaxone—to be taken together to treat gonorrhea. However, FDA does not recognize any azithromycin breakpoints for N. gonorrhoeae, which an expert from our meeting told us could be a barrier to developing FDA-authorized culture-based tests for N. gonorrhoeae resistance to the recommended dual therapy. Colistin is an antibiotic used in hospitals because of its efficacy against carbapenem-resistant bacteria, according to one manufacturer of a test for colistin resistance. This manufacturer markets its test in many countries but not in the United States, because FDA does not recognize colistin breakpoints. FDA has taken some steps to address unrecognized breakpoints, which are a potential barrier to developing some tests for antibiotic resistance. For example, FDA officials told us that the agency conducts regular internal reviews of breakpoints. According to FDA officials, the agency reviewed the 2019 CLSI breakpoint standards and updated FDA’s website with changes to recognized breakpoints as of June 2019. FDA has been posting such updates since December 13, 2017. FDA also accepts public comments requesting the recognition of new breakpoints, according to agency officials. However, we found there was some confusion between CLSI officials and experts and the FDA involving the number of comments FDA could review each year, which FDA later clarified on its website. One expert at our meeting later told us that CLSI adjusts its process for submitting comments based in part on their understanding of FDA’s communication. This expert added that FDA making a public commitment to a specific number of comments they would review would help CLSI improve its planning. FDA officials told us there is no legal requirement for FDA to communicate the number of comments the agency can review, but that in previously published notices of opportunities for public comments, there was nothing that indicated there would be limits. However, after we informed FDA officials of concerns by experts regarding the number of comments FDA could review, FDA updated their webpage to clarify that they will review all submitted comments. FDA has taken limited steps to monitor whether FDA-authorized tests are using new breakpoints after these breakpoints are updated and accepted by FDA. Because bacteria can develop increasing resistance to antibiotics, it is sometimes important to change the breakpoints used for determining whether or not bacteria are resistant to a given antibiotic. Using tests with out-of-date breakpoints could result in misidentifying a resistant infection as non-resistant, which can lead to treating a patient with an ineffective antibiotic and the further spread of the infection. FDA officials told us the agency has taken limited steps to monitor the status of breakpoint updates, and that out-of-date breakpoints being used in tests should be a rare occurrence. In contrast, a CDC official told us that keeping tests updated is a significant concern. This official cited the example of carbapenem- resistant Enterobacteriaceae infection, which triggers specific procedures to limit the spread of these bacteria. If the test breakpoint is out of date, the infection may not be detected in a timely manner, and the pathogen could spread broadly as a result. A recent study looking at hypothetical scenarios in one U.S. county estimated that a 32-month delay in updating tests to match CLSI breakpoints for carbapenem-resistant Enterobacteriaceae would have resulted in an average of almost 2,000 additional carriers of these bacteria county-wide. Additionally, an expert told us that use of out-of-date breakpoints could lead to improper patient care, improper surveillance reporting, and slower detection of emerging resistance. However, the true impact of this issue is challenging to discern (see text box). The Extent of Any Negative Effects of Out-of-Date Breakpoints on Public Health Is Unclear Experts and agency officials voiced a range of opinions on the public health effects of tests with out-of-date breakpoints. For example, one Centers for Disease Control and Prevention (CDC) official told us that despite the lack of breakpoint updates, cases of a type of carbapenem-resistant Enterobacteriaceae were likely ultimately caught by hospitals because a second test was used by all but a small number of hospitals. One expert stated that how quickly test breakpoints are updated is less important when deciding what test to adopt than other factors, such as ease of use. However, another expert noted that laboratories addressing emerging threats may feel the need to use non-Food and Drug Administration (FDA) cleared tests, because they are aware that FDA-cleared tests may not be updated as quickly as needed. Test updates may be an issue for smaller laboratories, which do not have dedicated personnel keeping track of breakpoint revisions, Department of Veterans Affairs officials told us. FDA officials told us that because manufacturers are strongly motivated to keep their tests current, only a few tests have out-of-date breakpoints. However, the only confirmation FDA officials offered for this statement was to mention an unofficial internal survey of FDA’s database of existing tests, conducted in March 2019, which concluded that all FDA-authorized tests had implemented breakpoint updates made since December 13, 2017. They said this survey is not conducted regularly. They also stated that it is possible that some tests have not been updated to reflect breakpoint updates made prior to December 13, 2017, but that FDA is unaware of any such tests that also pose a public health threat. To assess the extent to which there are FDA-authorized tests using out- of-date breakpoints, we spoke with experts and stakeholders and reviewed studies they identified. We identified several FDA-authorized tests with breakpoints that were changed nearly a decade ago. Some of these tests could be used for diagnosing infection with carbapenem- resistant Enterobacteriaceae, which CDC identified as an urgent threat. One manufacturer told us that one of their tests has not been updated with new breakpoints nearly 10 years after a breakpoint revision. FDA officials acknowledged it is possible some FDA-authorized tests might continue to rely on outdated breakpoints. Further, in 2019, a scientific article listed four different test manufacturers offering tests that have not been fully updated to reflect revised breakpoints, including some affecting antibiotics for some types of carbapenem-resistant Enterobacteriaceae. Finally, CDC officials told us they asked hospital laboratories in a survey for 2017 and 2018 if they had updated their tests to reflect revisions in breakpoints for carbapenem-resistant Enterobacteriaceae that were implemented in 2010. According to CDC, nearly 1,000 of over 5,000 responding hospital laboratories had not implemented the revised breakpoints, and, of these, over 85 percent were using FDA-authorized tests. One CDC official stated that there is significant concern for patient safety associated with out-of-date breakpoints, and another said that there are few justifications for failing to update the tests after 8 years. FDA officials told us they have not received reports of suspected device-associated deaths, serious injuries, or malfunctions that are specific to out-of-date carbapenem-resistant Enterobacteriaceae breakpoints in FDA-authorized tests using such breakpoints. The officials added that it is possible to detect carbapenem-resistant Enterobacteriaceae under certain situations, even if the test had an out-of-date breakpoint for a given antibiotic against these bacteria. However, FDA does not know the actual negative effect, if any, of out-of- date breakpoints because it does not know how many FDA-authorized tests rely on such breakpoints. Since December 2017, FDA has conducted one unofficial survey of tests to assess breakpoint updates that was limited in scope and is not a regular event. Other than that, FDA is relying on market incentives to drive manufacturers to make sure their devices are updated. According to FDA and others, the extent of the problem is not clear. However, PACCARB identified updating test breakpoints as an important issue in a 2017 report. Additionally, one of the sub-objectives in the National Action Plan notes that rapid updating of breakpoints is essential to provide accurate information to guide appropriate drug treatment. Finally, the Standards for Internal Control in the Federal Government directs management to establish and operate monitoring activities to monitor its internal control systems and evaluate the results. In this case, monitoring and evaluation of the status of breakpoint updates in FDA-authorized tests could help FDA identify and address the National Action Plan sub-objective, as well as a strategic priority in the mission statement of its Center for Devices and Radiological Health: “FDA assures that patients and providers have timely and continued access to safe, effective and high-quality medical devices.” FDA officials said they do not believe the issue is a significant problem, but the agency has also not regularly evaluated any effects of using tests for antibiotic resistance with out-of-date breakpoints. FDA officials stated that there may be resource constraints to their ability to conduct regular monitoring and evaluation. By regularly monitoring and evaluating FDA-authorized tests, FDA would be better positioned to determine the extent of tests relying on out-of-date breakpoints and may be better positioned to provide assurance that patients and providers have timely access to safe and effective tests. Furthermore, by regular monitoring, FDA would be able to determine whether test manufacturers are updating breakpoints as needed, and help ensure that patient care and infection control efforts are effective. Experts, federal officials, and antibiotic developers have identified economic and other challenges to developing new antibiotics. Federal agencies, including HHS and DOD, have engaged in efforts to address some of the challenges; however, experts said these efforts are not sufficient and that additional federal incentives are needed to encourage the development of new antibiotics. Experts are concerned about a void in the discovery of new antibiotic classes and the current pipeline of antibiotics in development. According to The Pew Charitable Trusts, a nonprofit public policy organization that tracks the pipeline of antibiotics, no new classes of antibiotics approved for human use have been discovered since 1984. In addition, experts are concerned that the number of antibiotics in clinical development is insufficient to meet the threat of antibiotic resistance. For example, according to The Pew Charitable Trusts, only 42 antibiotics were in clinical development globally—meaning clinical trials were being conducted to test their safety and efficacy in humans—as of June 2019, and only 24 of them targeted bacteria on CDC’s or WHO’s priority lists. According to a recently published analysis, the authors found that the pipeline of antibiotics that target gram-negative bacteria is dominated by derivatives of existing classes of antibiotics and “does not sufficiently address the problem of extensively drug-resistant gram-negative bacteria”. For example, one study estimated the average cost per new molecular compound that received FDA approval between 2005 and 2013 to be $1.4 billion. See J. A. DiMasi, H. G. Grabowski, and R. W. Hansen, “Innovation in the Pharmaceutical Industry: New Estimates of R&D Costs,” Journal of Health Economics, vol. 47 (2016): pp. 20-33. Other studies suggest lower development costs. For example, another study estimated a median cost to develop cancer drugs of $0.6 billion. See V. Prasad and S. Mailankody, “Research and Development Spending to Bring Single Cancer Drug to Market and Revenues After Approval,” JAMA Internal Medicine, vol. 177, no. 11 (2017): pp. 1,569-1,575. for antibiotic-resistant infections have a narrow set of patients for whom the treatment would be appropriate. As a result of the perceived poor return on investment, many large pharmaceutical companies have discontinued their antibiotic development in recent years. In 2018, according to The Pew Charitable Trusts and other published sources, four large pharmaceutical companies worldwide had antibiotics in clinical development globally compared to 1990, when 18 were involved in antibiotic R&D. Two antibiotic companies declared bankruptcy in 2019; in the case of one, the company filed for bankruptcy only 10 months after its antibiotic, which targets resistant bacteria, received FDA approval. The majority of antibiotics in the development pipeline are being developed by smaller companies that do not have other drugs on the market to help cover their R&D costs. However, representatives from three small antibiotic developers we spoke with noted that their field is struggling because it is difficult to raise funds from private investors due to the low return on investment potential. with bacterial infections into certain clinical trials prior to initiating treatment can be difficult due to a lack of available rapid diagnostic tests to identify the type of infection and the urgent need to begin treatment immediately for acute infections. According to FDA officials, this is problematic for clinical trials because any prior treatment could obscure the true efficacy of the drug under investigation. Recognizing this often unavoidable issue, FDA has issued guidance giving antibiotic developers additional, but limited, flexibility with their clinical trial protocols in certain cases. Superiority trials, which aim to show that the drug being investigated is more effective than an existing drug. Non-inferiority trials, which aim to demonstrate that the difference between the effectiveness of the drug being investigated and an existing drug is small enough to show that the drug being studied is also effective. Typically, there are three phases of clinical trials, with the sizes of the trials increasing with each phase. FDA generally prefers that when conducting clinical trials, developers demonstrate the effectiveness of a new drug by showing its impact on a clinical endpoint— a direct measure of how a patient feels, functions, or survives. FDA also accepts surrogate endpoints, which are laboratory measures or physical signs used as a substitute for a clinical endpoint that reasonably predict a clinical benefit. developers told us that, for most antibiotics, it is difficult to conduct superiority clinical trials and more feasible to conduct non-inferiority trials, because the latter allows for smaller enrollment. (See side bar for an explanation of clinical trial types.) They told us that the inability to demonstrate their drug’s superiority limits their ability to market the drug, because it can be difficult to convince purchasers (e.g., hospitals) to choose the newly approved antibiotic over existing antibiotics, especially when the new antibiotic is significantly more expensive. Gaining approval for multiple indications. FDA generally approves drugs for a specific indication; therefore, antibiotic developers told us they tend to design their clinical trials around common infection types, largely because of the relative ease of enrolling patients. However, some antibiotics can treat infections in multiple parts of the body, which may not have been studied in a clinical trial. While providers are able to prescribe drugs for “off-label” use—that is, for a condition or patient population for which the drug has not been approved—they may lack information on the safety and efficacy of the drug for such use. In addition, such off-label use may not be reimbursed by the patient’s insurance. According to The Pew Charitable Trusts, there were 29 nontraditional antibacterial products in clinical development for the U.S. market in June 2019. Among the 29 products in the pipeline, nine were antibodies, seven were vaccines, seven were live biotherapeutic products, and six were other types of products. No bacteriophages were in clinical development. More than half of these products are for the treatment of Clostridioides difficile or Staphylococcus aureus infections. Experts, antibiotic developers, and federal officials also said it is scientifically challenging to develop new antibiotics that can overcome existing mechanisms of resistance. One expert at our meeting explained that it is necessary to develop an antibiotic that works differently than existing antibiotics so that bacteria are not resistant to it. In particular, experts and federal officials have noted that it is challenging to develop antibiotics that can kill certain types of bacteria, called gram-negative bacteria, largely due to their double membrane that makes it difficult for antibiotics to enter the bacterial cell, and to pumps that can remove the drug once it enters. Three antibiotic developers we spoke to explained that as bacteria continue to evolve new ways to resist antibiotics, it is difficult for scientists to keep pace by developing new treatments that can overcome those mechanisms. In addition, experts noted that scientists have already discovered most of the antibiotics from known sources, such as soil. As a result, scientists are now exploring new sources of chemicals with antibiotic properties, such as insects. As the rate of antibiotic discovery has slowed, scientists have also begun to explore alternatives to traditional antibiotics—which we call “nontraditional products” in this report. Many types of nontraditional products are currently being researched and developed to treat antibiotic- resistant infections, including, among others, live biotherapeutic products, antibodies, and bacteriophages. For example, one type of nontraditional product in use for the treatment of recurrent C. difficile- associated disease—which causes diarrhea, abdominal cramps, and an estimated 15,000 deaths in the United States each year, according to CDC—is fecal microbiota for transplantation, more commonly known as fecal transplants. (See text box.) However, scientists and companies researching and developing certain types of nontraditional products face development challenges. For example, according to a paper written by BARDA officials and others, certain types of nontraditional products target only one or a few types of bacteria, which makes enrollment of patients in clinical trials difficult and potentially cost-prohibitive. The authors also stated that additional research is needed to evaluate side effects and measure the efficacy of some types of nontraditional products. According to another published paper, more than half of the nontraditional products in development are intended to be used concurrently with a traditional antibiotic, and it can be difficult to demonstrate the additional clinical benefit of adjunctive therapies in clinical trials. The authors also noted that additional clinical trial endpoints still need to be developed and validated for such nontraditional products. Fecal Transplants The goal of a fecal transplant—which involves collecting stool from healthy donors and transferring it to patients via enema, oral capsule, or another modality—is to restore a healthy gut microbiome for recipients. According to the National Institutes of Health (NIH), multiple research studies have indicated that these transplants are effective, but their long-term safety has not been established. Questions remain about the Food and Drug Administration’s (FDA) policy regarding stool banks that collect, prepare, and distribute fecal transplant products. FDA issued guidance in 2013 indicating its intention to exercise enforcement discretion regarding Investigational New Drug requirements for the use of fecal transplants to treat Clostridioides difficile infections, provided that the treating physician obtained adequate consent from the patient or his or her legally authorized representative. In other words, FDA’s guidance indicated it would not require fecal transplant products to satisfy the Investigational New Drug requirements— which refer to the requirements for FDA’s approval before beginning clinical trials to test a product on humans. [FDA, Enforcement Policy Regarding Investigational New Drug Requirements for Use of Fecal Microbiota for Transplantation To Treat Clostridium difficile Infection Not Responsive to Standard Therapies; Guidance for Industry; Availability, 78 Fed. Reg. 42965 (Jul. 18, 2013).] However, FDA later issued draft guidance in 2016 stating that FDA did not intend to extend enforcement discretion with respect to the Investigational New Drug requirements applicable to stool banks distributing fecal products. [FDA, Enforcement Policy Regarding Investigational New Drug Requirements for Use of Fecal Microbiota for Transplantation To Treat Clostridium difficile Infection Not Responsive to Standard Therapies; Draft Guidance for Industry; Availability, 81 Fed. Reg. 10632 (Mar. 1, 2016).] FDA has not finalized the 2016 draft guidance, which leaves the final guidance from 2013 as the current policy. According to FDA, the agency received many comments from patients and industry groups in response to the 2016 draft guidance expressing concern about the effect that the requirement for clinical trials would have on access to these products. In March 2019, FDA officials told us they were still reviewing comments to the 2016 draft guidance and were unable to say whether or not it would be finalized. In November 2019, FDA held a public hearing to obtain further input on the use of fecal transplants to treat C. difficile infection not responsive to standard therapies and to better understand the effect of FDA’s enforcement policy on product development. Multiple federal agencies have supported the development of new antibiotic treatments, including providing funding for antibiotic R&D, issuing guidance related to antibiotic clinical trials, and implementing Medicare payment mechanisms. Agencies have made available both “push” incentives, which directly support antibiotic R&D, and “pull” incentives, which offer financial benefit, either directly or indirectly, to developers of successful antibiotics after they reach the market. Federal funding for antibiotic R&D. Several federal agencies award grants or contracts, create public-private partnerships, or use other approaches to provide researchers the funding for R&D of new treatments for antibiotic-resistant infections (see table 3). This type of pre- market R&D support is considered a “push incentive.” See appendix III for additional examples of efforts to support antibiotic R&D by NIH and DOD. Among the products in the CARB-X portfolio, 12 would represent a new antibiotic class (if approved) and 14 target a novel molecular bacterial target. Awardees were based in six countries. Issued guidance to support clinical trials. FDA has implemented programs and issued guidance that help address some regulatory challenges and encourage antibiotic development. In 2012, through the Generating Antibiotic Incentives Now provisions of the Food and Drug Administration Safety and Innovation Act, Congress created the Qualified Infectious Disease Product (QIDP) designation. Drugs that FDA designates as QIDPs, which include antibiotics and antifungals, may qualify for 5 years of additional exclusivity and fast-track or priority review designation during the FDA review process. The additional exclusivity conferred to QIDP designees is a type of “pull incentive,” because it offers the potential for enhanced financial gain after a drug receives FDA approval and reaches the market. According to FDA officials, as of September 2019, FDA had granted 192 QIDP designations, 24 of which it has approved for marketing. Also in response to the Generating Antibiotic Incentives Now Act, FDA released final guidance in August 2017 to streamline clinical development of antibiotics for patients with an unmet medical need—that is, those with a serious bacterial disease that has few or no treatment options. FDA explains in this guidance that it may consider drugs for these patients that have higher risks than would be acceptable for a broad patient population and provides information on types of antibiotics that could be eligible for approval based on smaller, shorter, or fewer—as few as only one— clinical trials. The 21st Century Cures Act required FDA to establish a Limited Population Pathway for Antibacterial and Antifungal Drugs (LPAD). In June 2018, FDA issued draft LPAD guidance, as required by the Act. Under LPAD, eligible products—which are drugs and biologics intended to treat a serious or life-threatening infection in a limited population of patients with unmet needs—may follow a streamlined development program, similar to the approaches described in its earlier unmet medical need guidance. A biotechnology association noted in its public comments to the draft LPAD guidance the need for FDA to issue additional guidance to clarify its expectations for acceptable types of efficacy data when clinical trials are small and to clarify its interpretation of a “limited population of patients” for the purpose of the LPAD pathway. An expert who attended our meeting later told us there is a great need to address how to develop narrow-spectrum antibiotics—those designed to treat a single or small number of bacterial pathogens—using LPAD. FDA held a public meeting in July 2019 to solicit stakeholder comments on the draft LPAD pathway guidance, and FDA officials told us they expect to finalize the guidance by February 2020. However, as of March 17, 2020, FDA had not yet issued final guidance. In addition to issuing guidance, and to help inform future guidance, FDA engages with industry stakeholders to discuss and identify possible solutions to challenges related to the clinical development of antibiotics and nontraditional products. For example, FDA has held multiple public workshops, including one in November 2019 with experts from NIH’s National Institute of Allergy and Infectious Diseases, the Infectious Disease Society of America, and The Pew Charitable Trusts to better understand the current state of antibiotic clinical trials in the United States, and how to enhance enrollment and research in these trials. FDA officials told us they believe it is too early to issue guidance that would be broadly applicable and useful to nontraditional product developers. They explained that for certain types of nontraditional products, the approaches and specifics of product development are varied and evolving quickly. Instead, FDA’s Center for Biologics Evaluation and Research has a program in place that allows developers to meet with FDA prior to beginning clinical trials to obtain advice on a wide range of development-related topics. Implemented Medicare payment mechanisms. CMS uses Medicare payment mechanisms to help increase reimbursement to hospitals for certain antibiotics. For qualifying antibiotics, these payments are a form of indirect pull incentive because they have the potential to increase the demand for the new antibiotics after they reach the market, which could in turn improve their financial performance. Beginning in fiscal year 2020, CMS updated how it will pay hospitals for treating Medicare patients who have an antibiotic-resistant infection. Specifically, CMS changed the eligibility criteria and payment amount for antibiotics that qualify for “new technology add-on payments” and how it pays hospitals for treating Medicare patients with antibiotic-resistant infections. These payment changes are: Revised eligibility criteria for and amount of add-on payments. New technology add-on payments provide hospitals with additional compensation for a period of 2 or 3 years when they use qualifying new technologies or drugs that offer substantially improved clinical treatment, and when regular Medicare payments for the hospital stay are inadequate to cover the cost of the new technology or drug. Generally, medical services and technologies must be new and must demonstrate a substantial clinical improvement over existing services or technologies to receive the additional payment. However, CMS has acknowledged the difficulty antibiotic developers face in demonstrating such substantial clinical improvement due to manufacturers seeking FDA approval for most antibiotics on the basis of noninferiority clinical trials, as described above. To make it easier for antibiotics to qualify for the additional payments, under the revisions to the CMS payment policy beginning in fiscal year 2021, CMS will consider all antibiotics with a QIDP designation from FDA to be “new” for purposes of the add-on payment, and these antibiotics will not have to meet the substantial clinical improvement criteria. In addition, CMS has increased the amount of the temporary add-on payment for qualifying antibiotics. Prior to this change, the add-on payments for qualifying antibiotics were limited to 50 percent of the cost of the drug. Under the new policy, the payment percentage increased to a maximum of 75 percent of the cost of the drug. CMS has specified that two antibiotics are eligible for new technology add- on payments in fiscal year 2020. Increased payment for hospital stays. CMS changed the severity level designation for certain antibiotic resistance-related diagnosis codes, in recognition of the added clinical complexity and cost of treating patients with antibiotic resistance. This change in severity level can result in higher payments to hospitals when treating patients diagnosed with antibiotic resistance, which, according to the Administrator of CMS in an August 2019 blog post, will create “financial flexibility for physicians to prescribe the appropriate new antibiotics.” The Administrator also noted that CMS made this policy change because it recognized that new technology add-on payments are temporary and “further action was needed to realign financial incentives for antibiotics for the long-term.” See appendix III for additional examples of efforts to support antibiotic R&D by these and other federal agencies. Experts and antibiotic developers told us that the economic challenges have remained despite the available federal push and pull incentives for antibiotic R&D. Currently available premarket push incentives include grants and awards from NIH and BARDA that fund antibiotic R&D; currently available postmarket pull incentives include the additional market exclusivity available through QIDP designation and Medicare add- on payments for antibiotics. (See fig. 3.) Both of the antibiotic companies that declared bankruptcy in 2019 had received push incentives from BARDA and pull incentives through Medicare New Technology Add-on Payments and the QIDP 5-year extension of market exclusivity. While experts at our meeting and antibiotic developers told us that push incentives have been helpful, they also said push incentives alone are not sufficient to sustain antibiotic development. For example, two antibiotic developers we spoke with explained that push incentives have provided needed funding for conducting R&D, but said that push incentives will not help cover the costs they will incur after their drug reaches the market— for example, to manufacture and market their product. Experts and antibiotic developers have indicated that the effects of the existing pull incentives, QIDP market exclusivity, and Medicare add-on payments on stimulating development of new antibiotics have been limited for the following reasons: QIDP and market exclusivity. As we previously reported, several pharmaceutical companies told us that the market exclusivity incentive may not stimulate the development of new antibiotics, because the extension is unlikely to extend past the typical patent life of a new drug. In addition, a representative from The Pew Charitable Trusts said that, while the passage of the Generating Antibiotic Incentives Now Act initially bolstered private investments in antibiotics, it did not ultimately stabilize the pipeline of antibiotics in development, noting that since then, several large pharmaceutical companies have discontinued their antibiotics R&D programs. Medicare updates to hospital payments. While CMS recently increased new technology add-on payments for certain antibiotics beginning in fiscal year 2020 to help improve access to antibiotics, these payments are limited to antibiotics used to treat Medicare patients. In addition, although Medicare increased the add-on payment amount to up to 75 percent of the estimated costs of qualifying antibiotics in excess of the regular Medicare payment, hospitals could still face costs for providing these drugs that are not covered by the Medicare payment. Furthermore, representatives from an antibiotic company and a biotechnology trade association told us the add-on payments do not directly incentivize hospital pharmacies to purchase the drug, because the add-on payment may not flow back to the pharmacy department’s budget. For these reasons, it remains to be seen whether the Medicare new technology add-on payments to hospitals for inpatient antibiotics will help improve the return on investment for antibiotic developers and further stimulate the antibiotic development pipeline. Similarly, it remains to be seen how CMS’s policy change that provides increased payments for hospital stays when Medicare patients have been diagnosed with certain types of antibiotic-resistant infections will affect hospitals’ use of new antibiotics. In light of the limitation of existing incentives for antibiotic development, experts, federal officials, and antibiotics developers have called for additional postmarket pull incentives to reinvigorate the pipeline of antibiotics under development. For example, PACCARB issued recommendations to the Secretary of HHS in September 2017 and July 2019 for the adoption of pull incentives, calling for the development of market entry rewards and options for plausible business models. In addition TATFAR—of which officials from BARDA, CDC, FDA, and NIH are members—reported that it is critical to develop a pull incentive strategy now to ensure that enough antibiotics are available in the future. Former FDA Commissioner Dr. Scott Gottlieb also stated in 2018 that he was “deeply concerned that without stronger pull incentives that encourage more R&D, we’ll see a far less robust pipeline of products than we need to address antimicrobial resistance.” Eight of the antibiotic developers we interviewed told us they think additional financial incentives are needed. For example, one developer said that sales revenues from antibiotics will never be sufficient to justify R&D investments, and another noted that financial incentives are needed during the first few years after a new antibiotic reaches the market to cover not only these costs, but also to conduct additional clinical trials to help expand the drug’s possible market. Finally, several experts at our expert meeting noted that, without pull incentives, most of the small companies currently developing antibiotics are unlikely to survive, and large pharmaceutical companies will likely continue to exit the antibiotic market. Advisory groups and others have identified multiple options for how postmarket pull incentives could be designed, including market entry rewards—either in the form of lump sum payments or transferable vouchers that could be sold to confer additional market exclusivity to other pharmaceutical drugs—or reimbursement reform, such as licensing arrangements or add-on payments for hospital-administered antibiotics. (See fig. 4.) The four advisory groups whose papers we reviewed each recommended market entry rewards as effective pull incentive options. While Commissioner of the FDA, Dr. Scott Gottlieb proposed an antibiotics licensing arrangement, which he called a subscription model, in a 2018 speech. Views on the utility of reimbursement reform as a pull incentive strategy are mixed. For example, representatives from The Pew Charitable Trusts stated their view that, while CMS’s recent changes to Medicare payment for antibiotics will likely be helpful to some degree, no reimbursement policy on its own would be able to increase antibiotic sales revenues sufficiently to transform the business model for antibiotics. An antibiotic developer we spoke to also told us that reimbursement policies would not be sufficient to support their business model because of low sales volumes for new antibiotics. The developer explained that it can take 2 or 3 years of antibiotic sales to recoup their R&D costs and finance their ongoing business operations, and that while larger pharmaceutical companies can rely on other profitable drugs to offset those costs, they could not because they did not have other drugs on the market. However, a representative from a biotechnology trade association told us that increasing reimbursement could help alleviate some of the economic challenges faced by developers of antibiotics that are already on or about to reach the market while policy makers explore longer-term pull incentive strategies. TATFAR cautioned that simply increasing reimbursement for antibiotics could potentially limit patient access, particularly for patients without health insurance—including those in low-and middle-income countries—and it could incentivize only antibiotics for common types of infections with a large market potential, rather than for rare, yet dangerous, types of pathogens. Advisory groups and others have evaluated potential market entry reward models, taking into consideration factors such as format, value, funding sources, and eligibility criteria. Some have proposed that receipt of a market entry reward should be delinked, fully or partially, from sales revenues—that is, the developer would have to forgo some or all sales revenue as a condition of receiving the reward. Proponents of delinkage believe that separating revenues from antibiotics sales volumes would discourage aggressive sales that could lead to overuse. An expert who attended our meeting later told us that policies to incentivize use of new antibiotics must be balanced with policies to monitor prescribing of new drugs to prevent inappropriate use. Generally, advisory groups stipulate that to maximize the public health benefit, only antibiotics that treat what are deemed to be high priority bacteria should be eligible for a reward. Specific recommendations and conclusions included the following: TATFAR concluded in 2017 that a partially delinked market entry reward of approximately $500 million would be the least disruptive option but noted that additional assessment would be necessary to select the most appropriate model and determine governance and other design elements. PACCARB expressed support for a delinked model, in which a company accepting a market entry reward would be required to forgo marketing activities and profits based on sales volume. In addition, they suggested the establishment of an antibiotic incentive fund supported by an antibiotic usage fee or the sale or auction of transferable exclusivity vouchers as plausible options for financing pull incentives. The Duke University Margolis Center for Health Policy recommended in 2017 a delinked, public-private market entry reward model. This model was comprised of publicly funded market entry rewards for qualifying antibiotics for the first 5 or 6 years, followed by privately funded “value-based” contracts between antibiotic developers and health care payors, in which the payor could agree, for example, to pay a predetermined amount for full access to the antibiotics for a given population. The Duke-Margolis Center proposal did not specify a funding source, but it noted multiple options for consideration, including general government funds, antibiotic use taxes, or the sale of transferable exclusivity vouchers. The European DRIVE-AB project recommended in 2018 an internationally funded, partially delinked market entry reward valued at approximately $1 billion per antibiotic, paid over the course of 5 or more years. Recipients of a market entry reward would be allowed to sell their drug on the private market, but they would agree to certain marketing restrictions to discourage inappropriate use. HHS may need to request authority and appropriations to create and implement certain types of market entry rewards. For example, HHS does not currently have authority to offer transferable exclusivity vouchers to antibiotic developers, since that would require a change in statute. Advisory groups also noted that the various pull incentive approaches would require additional public or private expenditures and offered possible sources of funding. For example, in addition to general fund revenues, PACCARB suggested that pull incentives could be funded through antibiotic usage fees, the auctioning of transferable exclusivity vouchers, or by allowing developers of new antibiotics to earn a transferable exclusivity voucher. The Duke-Margolis Center suggestions included funding market entry rewards through a yearly per-member fee for all health insurance plans. Transferable exclusivity vouchers may not require an independent funding source, because the value of the reward is based on the sale of the voucher to another drug developer. However, vouchers would still increase public and private health care expenditures, because expenditures would likely increase for drugs for which the extra period of exclusivity was purchased due to the delayed entry of lower- priced generics. Finally, reimbursement reform could increase health care expenditures for health care payors, including Medicare and private health insurance carriers. Although PACCARB, TATFAR, and other experts have called for additional postmarket pull incentives to increase the antibiotic pipeline, as of January 2020 HHS has not developed a strategy for creating these incentives. HHS officials told us that the department created an interagency workgroup within HHS in spring 2019 to identify possible pull incentive options, among other things. The recently convened HHS interagency workgroup is a step in the right direction toward exploring options for new antibiotic development incentives. Through this workgroup, HHS has an opportunity to determine which types of postmarket incentives it believes would most effectively incentivize the development of new treatments for antibiotic-resistant infections. However, it is unclear whether the HHS interagency workgroup’s efforts will include consideration of such incentives because, according to HHS officials in January 2020, the interagency workgroup was still considering possible recommendations for HHS leadership and had not produced any specific documents to share with us. The Government Performance and Results Act of 1993 (GPRA) and the GPRA Modernization Act of 2010, which significantly enhanced agencies’ responsibilities under GPRA, include principles for federal agencies to consider related to developing strategies for achieving results, among other principles. We have previously reported that these principles can serve as leading practices for planning at lower levels within agencies, such as individual programs or initiatives. Our past work has shown that strategic frameworks can serve as a basis for guiding policy makers, including congressional decision makers and agency officials, when making decisions about resources, programs and activities, particularly in relation to issues that are national in scope, such as antibiotic development. Developing a strategic framework that outlines new postmarket pull incentives and their key design elements—such as monetary value, eligibility criteria, and guidelines to prevent overuse— would be a first step toward identifying potential authorities and resources that may be needed to create the incentives, and toward determining agency roles for implementation and oversight of the incentives. Until such incentives are developed, more drug companies may exit the antibiotic development sector, and the pipeline of new treatments for antibiotic-resistant infections may continue to decrease. Furthermore, the current significant federal investment in push incentives to support antibiotic R&D will remain a high-risk enterprise, if companies receiving large R&D grants are unable to sustain their business once their treatment reaches the market. Federal agencies have undertaken several efforts to promote the appropriate use of antibiotics through stewardship programs and activities. However, four key challenges remain that have limited this progress. To promote the appropriate use of antibiotics across health care settings through antibiotic stewardship programs and activities, federal agencies have undertaken several efforts that aim to reduce inappropriate antibiotic use, reduce health care costs, improve patient outcomes, and combat antibiotic resistance. Selected examples of these efforts are discussed below. (For more detailed information on agencies’ efforts to promote the appropriate use of antibiotics, see app. IV.) Federal agencies require certain types of health care facilities to implement antibiotic stewardship programs, as follows: CMS. In September 2019, CMS finalized new health and safety requirements for hospitals and critical access hospitals to implement antibiotic stewardship programs by March 30, 2020, as a condition of their participation in the Medicare and Medicaid programs. Under these requirements, hospitals and critical access hospitals are required, among other things, to implement these programs facility- wide (which includes emergency departments) and to adhere to nationally recognized antibiotic prescribing guidelines. Nearly 3 years prior, CMS published similar requirements for nursing homes and skilled nursing facilities—collectively known as long-term care facilities—to establish antibiotic stewardship programs by December 4, 2017. Experts, including those at our meeting and the PACCARB, credit these requirements with being a powerful lever for promoting the appropriate use of antibiotics; Medicare comprises a significant portion of the nation’s health care expenditures—$741 billion in 2018, covering 59.9 million beneficiaries. DOD. DOD published a policy, effective October 2017, requiring the establishment of antibiotic stewardship programs within its military medical treatment facilities and, one year later, issued guidance for implementation. Among other things, the policy specified that these facilities’ antibiotic stewardship programs include components such as (1) leadership commitment by each facility; (2) accountability; (3) pharmacy expertise, including antibiotic prescribing and use evaluation; (4) implementation of action for change that would demonstrate commitment to the program; and (5) training for clinicians regarding antibiotic resistance and prescribing practices. DOD officials told us that all of these facilities (both inpatient and outpatient) were in different stages of implementing the antibiotic stewardship policy. VA. In January 2019, VA updated its 2014 policy directive for the implementation and maintenance of antibiotic stewardship programs in its health care facilities, which provide both inpatient and outpatient services to veterans. This policy directive includes requirements for its facilities to develop a written policy, conduct an annual evaluation of stewardship activities, ensure that adequate staff and resources are in place, and identify medical and pharmacy personnel as stewardship “champions.” According to department officials, VA has successfully implemented antibiotic stewardship programs in all of its health care facilities. CMS has developed incentives for eligible clinicians in any type of health care facility to improve antibiotic use and stewardship, as part of the agency’s broader efforts to improve care for Medicare patients. Through the Merit-based Incentive Payment System (MIPS) launched in 2017, CMS offers hundreds of quality measures and nearly 100 “improvement activities” on a wide range of topics—including the appropriate use of antibiotics—on which eligible clinicians can choose to report their performance to the agency. CMS then adjusts payments higher for clinicians who report data and achieve a performance-based, final score above a certain threshold—and penalizes clinicians who do not achieve that threshold with lower payments. Federal agencies have published guidance for health care facilities on how to implement antibiotic stewardship, as follows: AHRQ. Through a 5-year nationwide project, the AHRQ Safety Program for Improving Antibiotic Use has provided technical assistance and CDC’s guidance to hospitals, long-term care settings, and physicians’ offices to promote implementation of antibiotic stewardship activities and help clinicians select optimal antibiotic treatment regimens. In December 2018, AHRQ completed implementation of this guidance in more than 400 hospitals, which included six DOD facilities and 79 critical access hospitals, according to AHRQ officials. CDC. Since 2014, CDC has published a series of guidance documents—called the Core Elements of Antibiotic Stewardship (Core Elements)—to promote the appropriate use of antibiotics in health care. The Core Elements are tailored to hospitals, nursing homes, outpatient settings, small and critical access hospitals, and low- and middle-income countries with limited resources. Common elements in these guidance documents include (1) leadership commitment, (2) implementation of policies and interventions to improve antibiotic use, (3) tracking and reporting antibiotic use, and (4) education to providers on appropriate antibiotic use. For more information on MIPS, see GAO, Health Care Quality: HHS Should Set Priorities and Comprehensively Plan Its Efforts to Better Align Health Quality Measures, GAO-17-5 (Washington, D.C.: Oct. 13, 2016) and Medicare: Small and Rural Practices’ Experiences in Previous Programs and Expected Performance in the Merit-based Incentive Payment System, GAO-18-428 (Washington, D.C.: May 31, 2018). other sources. In particular, CDC has focused its efforts to expand antibiotic use data collection from hospitals, where an estimated one in two patients receives an antibiotic for at least one day during an average hospital stay. CDC launched its AU Option in 2011 as a voluntary, electronic reporting tool added on to the pre-existing NHSN. The AU Option allows the nation’s 6,849 hospitals that are already reporting to the NHSN to submit their antibiotic use data in a standardized format. CDC then aggregates such data to calculate national benchmarks and allows hospitals to compare their actual antibiotic use against those benchmarks. In addition, CDC has periodically conducted prevalence surveys through the EIP to gather data on health care-associated infections and antibiotic use in about 200 hospitals and 161 nursing homes in 10 states. With regard to outpatient settings, CDC has acquired, through a proprietary source, 8 years of pharmacy data on antibiotic prescriptions since 2011, which the agency is using to better characterize patterns in outpatient prescribing and to develop targeted interventions for high-prescribing areas. Federal agencies have developed training on antibiotic stewardship, as follows: CDC. In 2018, CDC launched a free, online training course for various types of clinicians—including physicians, dentists, pharmacists, physician assistants, and nurses—to inform them about proper antibiotic prescribing and strategies for communicating with patients. Clinicians can receive credit for partial completion (at least 50 percent) or full completion of this training as improvement activities under MIPS in 2019. CMS. CMS has provided training, technical assistance, and other learning opportunities to more than 4,000 hospitals, 2,400 nursing homes, and 7,600 outpatient settings on best practices for antibiotic stewardship and guidance on C. difficile prevention. In addition, CMS and CDC have developed and launched free, online training to help nursing homes implement antibiotic stewardship and prevent and manage C. difficile infections. DOD and VA. These departments have also offered antibiotic stewardship training to their health care facilities through webinars, workshops, or briefings. Federal agencies have funded research on antibiotic stewardship, as follows: AHRQ. Since 2015, AHRQ has increased its support for research to develop improved methods to combat antibiotic resistance and promote antibiotic stewardship, including through grants for research that will total more than $57 million, according to AHRQ officials. This research includes studies on the role of diagnostic tools in improving antibiotic use and reducing antibiotic resistance. AHRQ has also published numerous research studies on antibiotic or antimicrobial stewardship that the agency funded or authored. CDC. CDC supports research to identify, develop, and implement practices to stop the spread of resistance and to promote appropriate use of antibiotics in health care. CDC also supports research to fill gaps in knowledge related to aspects of antibiotic use and resistance that have public health impact. According to agency officials, CDC has provided approximately $110 million since 2016 to support this research through cooperative agreements and contracts. In 2017, CDC revised a national campaign to promote public awareness about appropriate antibiotic use. The campaign, called “Be Antibiotics Aware: Smart Use, Best Care,” is aimed at both health care providers and the general public and refines the message from CDC’s earlier campaign (“Get Smart: Know When Antibiotics Work”). HHS’s Office of Global Affairs has collaborated with other countries, including those participating in the TATFAR program, to promote the appropriate use of antibiotics internationally. In addition, CDC and the Office of Global Affairs launched the Antimicrobial Resistance Challenge at the United Nations General Assembly in September 2018 to catalyze global action against antibiotic resistance. A year later, CDC announced this challenge had resulted in nearly 350 commitments from government health officials, pharmaceutical and health insurance companies, and others from 33 countries to make formal commitments that further the progress against antimicrobial resistance, such as by improving appropriate antibiotic use. We identified four key challenges that have limited progress in federal efforts to promote the appropriate use of antibiotics, based on our analysis of documents, interviews with agency officials and experts, and other information. First, federal requirements for antibiotic stewardship programs apply only to certain types of health care facilities, and federal incentives for clinicians to adopt antibiotic stewardship activities are optional, limiting implementation of antibiotic stewardship across the health care spectrum. Second, CDC faces challenges in collecting complete antibiotic use data, limiting the agency’s ability to monitor and improve antibiotic use. Third, the CARB Task Force has not identified and reported on agencies’ plans to address the challenges related to expanding antibiotic stewardship programs and antibiotic use data collection across health care settings, so these plans are not publicly known. Fourth, antibiotic stewardship training for health care providers may have limited success in improving antibiotic prescribing behavior, and federal agencies indicate that it is challenging to evaluate the effectiveness of such training. Federal requirements for antibiotic stewardship programs are limited to certain types of health care facilities, and federal incentives for antibiotic stewardship activities are optional and limited to eligible Medicare clinicians, such as physicians. Federal requirements for antibiotic stewardship programs are limited to certain types of health care facilities. As previously noted, federal requirements for antibiotic stewardship programs are currently limited to hospitals and critical access hospitals, long-term care facilities such as nursing homes, and DOD and VA health care facilities. However, CMS has not yet developed requirements for ambulatory surgery centers or dialysis centers to implement antibiotic stewardship programs, which the National Action Plan called for being implemented by March 2018. CMS officials told us that the agency would develop those requirements once the rule for hospitals and critical access hospitals—which was delayed—was finalized. In addition, CMS’s health and safety requirements do not extend to other types of outpatient settings—such as physicians’ offices, retail clinics, and urgent care centers—where inappropriate antibiotic use has been found to be high. In the absence of regulatory levers, CDC and AHRQ encourage those types of facilities to establish antibiotic stewardship programs on a voluntary basis. Experts, including those at our meeting, indicate that expansion of antibiotic stewardship across the health care spectrum is likely to remain limited without additional federal requirements or other meaningful incentives—thus hindering the nation from fully achieving the benefits of appropriate antibiotic use. Such benefits include better patient outcomes, lower health care costs, and slower growth of antibiotic resistance. CMS incentives for clinicians to improve antibiotic use are optional, and implementation has been limited. The MIPS program’s effect on incentivizing appropriate use of antibiotics is limited, in part, because the incentives are available only to clinicians who meet MIPS eligibility criteria and because eligible clinicians can choose not to report data to CMS. In addition, participating clinicians have a wide range and number of quality measures and improvement activities, beyond those related to antibiotics, from which the clinicians can choose to report data to CMS to meet program requirements; thus, the likelihood that clinicians will choose to report on antibiotics- related measures or activities may remain low. For example, in 2017, MIPS-eligible clinicians were generally required to select and submit data to CMS on six out of 271 available quality measures; we identified nine of those measures as being related to antibiotics. MIPS-eligible clinicians were also generally required to select and submit data that year for up to four out of 93 available improvement activities; we identified one such activity as being related to antibiotics. Our analysis of CMS data on MIPS participation in 2017, the program’s first performance year and the most recently available data, indicates that implementation of the antibiotics-related quality measures and improvement activities was limited. According to a CMS report, a total of 1,057,824 clinicians were eligible for MIPS in 2017, of which 1,006,319 clinicians, or 95 percent, reported data. Based on our analysis of data contained in the CMS report’s appendix, the number of 2017 MIPS-participating clinicians who reported to CMS on the nine antibiotics-related quality measures ranged from 844 clinicians to 33,631 clinicians; the measure on appropriate treatment for children with an upper respiratory infection was the most reported antibiotics-related measure. By contrast, the most frequently reported quality measures overall in 2017 were controlling high blood pressure (510,723 clinicians), preventive care and screening for tobacco use (492,357), and breast cancer screening (473,819). CMS’s data also show that for the 2017 MIPS improvement activities, 47,645 of the 1,006,319 participating clinicians reported on the one improvement activity related to antibiotics that year: implementation of an antibiotic stewardship program. Specifically, this activity referred to implementation of an antibiotic stewardship program that measured the appropriate use of antibiotics for several different conditions (upper respiratory infections in children, pharyngitis, and bronchitis in adults), according to clinical guidelines for diagnostics and therapeutics. CDC’s ability to monitor and improve appropriate antibiotic use is limited by challenges it faces in collecting complete antibiotic use data across health care settings. According to CDC, experts we interviewed, and documents we reviewed, more data are needed to identify the extent of antibiotic use, including inappropriate use. In turn, CDC and experts say that more antibiotic use data would enable health care providers, federal agencies, and others to identify and target areas for improvement, track results over time, and adjust antibiotic stewardship activities as needed. We have also previously reported that monitoring antibiotic use over time in both inpatient and outpatient settings is important for understanding patterns in antibiotic resistance and for targeting stewardship activities. In addition, WHO notes that data on global antibiotic use is essential for obtaining a comprehensive picture of antibiotic resistance and for identifying areas where actions are needed. Despite progress in collecting antibiotic use data (as previously discussed), CDC faces several challenges in its efforts to collect complete antibiotic use data. For example, health care providers across various inpatient and outpatient settings do not record such data in one centralized, electronic database. In addition, CDC officials told us that there are no uniform requirements at the federal level (with the exception of DOD and VA hospitals) for providers to report their antibiotic use data to a centralized database such as the NHSN AU Option, and, according to CDC officials and experts we interviewed, data collection can be costly for CDC and health care providers. Because of these and other challenges, CDC relies on data voluntarily reported by hospitals through the AU Option, and the agency collects its own data or purchases proprietary pharmacy data to estimate antibiotic use—and, to some degree, to assess appropriateness of use—across health care settings. However, these data are incomplete owing to several limitations, as described by type of setting below. Hospitals. Our analysis of CDC data shows that although the number of hospitals participating in the AU Option has gradually risen since its launch in 2011, participation remains limited, with 1,561, or 23 percent, of the 6,849 eligible hospitals reporting at least one month of antibiotic use data as of January 1, 2020. (See fig. 5 for a map showing the percentage of U.S. hospitals reporting antibiotic use data to the AU Option, by state, plus the District of Columbia and Puerto Rico, as of August 2019.) While CDC officials told us they considered this level of participation to be an accomplishment given that participation is voluntary, the National Action Plan set 95 percent participation in the AU Option by 2020 as a significant outcome to support the plan’s goal to strengthen national surveillance efforts to combat resistance. Experts, including those at our meeting, cite multiple challenges that CDC faces in collecting hospitals’ antibiotic use data through the AU Option. For example, The Pew Charitable Trusts has stated that current, voluntary data are limited and that mandatory reporting would provide the data needed to establish a more accurate baseline of antibiotic use, identify stewardship interventions that would be most effective, and measure progress toward reducing inappropriate prescribing. An expert who attended our meeting later suggested that CMS could implement a pay-for-reporting program to incentivize hospitals to report data to the AU Option, and that the program could transition to a pay-for-performance program over time. In addition, experts we interviewed told us that a participating hospital must be willing to spend as much as tens of thousands of dollars for a vendor to customize software for their electronic health record systems to use the AU Option, in addition to investing time training staff on how to use it. CDC officials also told us that the agency lacks the authority to require hospitals to report their antibiotic use data, and that there is currently no federal funding available to assist hospitals with the investment needed to participate in the AU Option. Furthermore, hospitals’ voluntary participation in the AU Option may remain limited until CDC’s benchmark measures are adequately risk- adjusted for different locations and patient populations. For example, one expert we interviewed said that because the AU Option currently aggregates data on the volume of antibiotics used without adequate risk adjustment, a hospital with a patient population that might warrant higher use of antibiotics may be reluctant to report its antibiotic use data to avoid looking like an unnecessarily high prescriber. Regarding another data source for antibiotic use in hospitals, CDC’s EIP provides more granular data at the patient level that allows CDC to assess the appropriateness of antibiotic use. However, CDC officials told us that the agency has been unable to repeat its hospital prevalence survey since 2015 due to insufficient resources (the next survey is expected in 2020) and that the survey encompasses a limited number of hospitals, patients, and states. Nursing homes. According to CDC, nursing homes may be the most challenging health care setting from which the agency collects antibiotic use data; CDC officials stated that this is because electronic health record systems, from which data could be easily accessed, are less common in nursing homes. In addition, CDC officials stated that the agency’s collection of antibiotic use data through the EIP nursing homes prevalence survey has been limited in scope and frequency due to insufficient resources. Outpatient settings. Collecting data for outpatient settings, such as retail pharmacies, is also challenging. For example, CDC officials stated that one proprietary source from which CDC purchases data reflects the volume of pharmacy antibiotic prescriptions, but the data do not contain diagnostic information, preventing the agency from evaluating the appropriateness of those prescriptions. Other CDC or proprietary data sources from which the agency collects or purchases antibiotic use data are limited by the frequency with which those sources release such data, the age range of patients included in the data (i.e., whether they are over or under 65 years), or other characteristics. As previously noted, approximately 85 to 95 percent of the nation’s antibiotic use, by volume, occurred in outpatient settings from 2010 through 2015. The National Action Plan calls for strengthening antibiotic stewardship and for the timely reporting of antibiotic use data across health care settings. Executive Order No. 13676, as previously noted, directs the CARB Task Force to provide annual updates to the President on federal government actions to combat antibiotic resistance, including progress made in implementing the National Action Plan and plans for addressing any barriers preventing its full implementation. These annual updates are to include specific goals, milestones, and metrics for proposed actions and recommendations, taking into consideration federal resources. However, in its progress reports covering the first four years of the National Action Plan’s implementation—which were provided to the President and the public—the CARB Task Force has not identified plans to address barriers that agencies face in expanding antibiotic stewardship programs across health care settings. For example, the task force did not include in the progress reports CMS’s plans to address barriers to expanding its requirements for antibiotic stewardship programs in hospitals, which were delayed, or in certain other types of health care facilities. In addition, in its progress reports to date, the CARB Task Force has not identified plans to address the barriers to expanding the collection of antibiotic use data across health care settings. For example, the task force did not include in the progress reports CDC’s plans to address barriers to achieving the significant outcome of 95 percent of eligible hospitals participating in the AU Option by 2020, although participation was only 23 percent as of January 1, 2020. The CARB Task Force coordinators said, in response to our inquiries during this review, that the task force intends to identify agencies’ plans for addressing barriers in the Year 5 progress report to be published in fall 2020. However, the coordinators also stated that the progress reports to date have not identified plans to address barriers largely because the task force focused on reporting the agencies’ accomplishments in implementing the National Action Plan. Until the CARB Task Force identifies and reports on agencies’ plans to address barriers related to the expansion of antibiotic stewardship programs and the collection of antibiotic use data across health care settings to the extent feasible, the federal government will not have reasonable assurance that it is fully implementing the National Action Plan and addressing antibiotic resistance. While training is recognized as one component of an antibiotic stewardship program, such training may have limited success in improving antibiotic prescribing behavior, and federal agencies indicate that it is challenging to evaluate the training’s effectiveness. CDC officials and experts say that inappropriate antibiotic use could be improved through stewardship training, but it is challenging because antibiotic prescribing behavior is driven by multiple factors and can be difficult to change. For example, a PACCARB report stated that prescribers often feel pressure to prescribe antibiotics—even when antibiotics may not be warranted—because of their perception that a patient is demanding such a prescription, or a patient’s actual demand. In addition, CDC notes that antibiotics are frequently prescribed for respiratory conditions most commonly caused by viruses such as the common cold, against which antibiotics are ineffective. Other factors that drive antibiotic prescribing behavior, as cited by experts, include habit, which may stem from what physicians and other prescribers learn during their residencies or observe in the workplace; the time it takes to explain to a patient why an antibiotic is inappropriate; and “decision fatigue” caused by tiredness or hunger. (See table 4 for examples of factors that drive or deter antibiotic prescribing behavior.) Nevertheless, federal agencies plan to evaluate the effectiveness of their antibiotic stewardship training programs to some extent, although the National Action Plan does not require the agencies to do so. For example, CDC officials told us that their online training course for various types of clinicians allows participants to fill out an evaluation that includes questions about whether the participant will be able to apply knowledge gained from the course, which the agency will use to refine and update the course. In addition, for the antibiotic stewardship training for nursing homes that CDC and CMS jointly developed, CDC officials told us that participants will be asked 6 months after the training whether participants implemented stewardship practices—and whether there have been reductions in antibiotic use—as a result of the training. However, CMS, DOD, and VA officials noted that it is difficult to isolate and measure the effectiveness of antibiotic stewardship training specifically on antibiotic prescribing behavior—compared to other, concurrent federal efforts, such as requirements and guidance to promote appropriate antibiotic use. For example, DOD officials told us that their department has looked at antibiotic use data from DOD health care facilities as a “surrogate” to evaluate whether antibiotic stewardship in general has been effective—but noted that is an imperfect measure since there are many factors that affect antibiotic prescribing behavior, and training is only one of several interventions aimed at reducing inappropriate antibiotic use. Antibiotic resistance has been characterized as one of the greatest public health threats the world faces. A concerted effort involving coordination of multiple stakeholders and countries and across health fields is critical to helping ensure that bacterial infections remain treatable. Steps by federal agencies to expand surveillance, facilitate the development and use of new diagnostic tests, fund R&D for the development of new treatments, and issue requirements and guidance for antibiotic stewardship programs are important efforts toward addressing the problem of antibiotic resistance and implementing the National Action Plan. Significant challenges to conducting surveillance remain. For example, CDC has not determined the participation rates or appropriate distribution of participating hospitals needed by the voluntary antibiotic-resistance reporting option to achieve CDC’s goal of conducting regional and national assessments of resistance. By taking steps to determine the participation rates and distribution needed for this option, CDC would have more reasonable assurance that it can achieve its goal. CDC classified gonorrhea as one of the most urgent resistant threats in the nation, but collects limited specimens—representing an estimated 1 to 2 percent of the reported cases in the United States—for GISP, its primary surveillance system for resistant gonorrhea. However, CDC has not fully evaluated the representativeness of the trends identified by this surveillance system. By evaluating GISP to ensure that it includes measures of its representativeness, such as comparing the trends in the sample population with those in the overall U.S. population, using specially designed studies if needed, CDC would have better assurance that the trends detected in GISP accurately reflect the characteristics of the health-related outcome the system is designed to monitor. Further, neither the 2013 nor the 2019 Threats Reports provided quantitative measures of uncertainty for CDC’s estimates of morbidity and mortality resulting from antibiotic-resistant infections. Providing such measures, such as standard errors or confidence intervals, as appropriate, in its Threats Reports would help CDC and others compare information within and across reporting efforts, and draw appropriate conclusions about the characteristics of antibiotic resistance in the United States, including limitations associated with reported findings and conclusions. Finally, there has been a 6-year interval between CDC’s reports on antibiotic resistance threats. By developing a plan for more frequent dissemination of consolidated reporting on priority pathogens at regular intervals, CDC would have more timely trend data and other information necessary for users of the data, including policymakers, to prioritize, plan, implement, and evaluate public health actions to address antibiotic resistance. HHS has funded some studies to assess whether certain tests for antibiotic resistance lead to improved clinical outcomes, including more effective treatment for patients or more judicious use of antibiotics. However, HHS agencies that are in a position to conduct or fund such studies have not identified leadership, roles, and responsibilities to help further such efforts. By taking steps to identify leadership, roles, and responsibilities, agencies could more effectively address the need for clinical outcomes studies, potentially increasing test use, improving patient care, and enhancing stewardship efforts. In addition, for its part, FDA has not regularly monitored tests for antibiotic resistance to assess breakpoint updates or evaluated any effects of using tests for antibiotic resistance with out-of-date breakpoints. By regularly monitoring and evaluating FDA-authorized tests that rely on breakpoints, FDA would be able to determine whether test manufacturers are updating breakpoints as needed and help ensure that patient care and infection control efforts are effective. While government push incentives to support antibiotic R&D have been helpful, experts and antibiotic developers have indicated that push incentives alone are not sufficient to sustain antibiotic development. PACCARB, TATFAR, and other experts have called for additional postmarket pull incentives to increase the antibiotic pipeline, but HHS does not have a strategy for doing so. Developing a strategic framework that outlines key design elements of new incentives would be a first step toward identifying potential authorities and resources that may be needed and determining agency roles for implementation and oversight of the incentives. Until such incentives are developed, more drug companies may exit the antibiotic development sector, and the pipeline of new treatments may continue to decrease. Finally, in its progress reports covering the first four years of the National Action Plan’s implementation, the CARB Task Force did not identify plans, as required by the Executive Order, to address barriers that agencies face in fully implementing the National Action Plan, such as expanding (1) a CDC program designed to strengthen the U.S. response to resistant gonorrhea; (2) antibiotic stewardship programs across health care settings; and (3) antibiotic use data collection, to the extent feasible. Without identifying plans to address these and other challenges, the federal government cannot assure that the country is prepared to overcome the urgent health consequences of antibiotic resistance. Until the CARB Task Force, which is coordinated by HHS officials, identifies and reports on agencies’ plans to address barriers preventing full implementation of the National Action Plan, the federal government will not have reasonable assurance that it is fully implementing the National Action Plan and addressing antibiotic resistance. We are making a total of eight recommendations, including four to CDC, three to HHS, and one to FDA. Specifically: The Director of CDC should take steps to determine participation rates and distribution needed in the AR Option of the National Healthcare Safety Network for conducting regional and national assessments of antibiotic resistance of public health importance. (Recommendation 1) The Director of CDC should ensure that CDC’s evaluation of its surveillance system for antibiotic-resistant gonorrhea includes measures of its representativeness, such as comparison of the trends in the sample population with those in the overall U.S. population, using specially designed studies if needed. (Recommendation 2) The Director of CDC should provide information on uncertainties for antibiotic resistance estimates in its consolidated Threats Reports, including standard errors or confidence intervals, as appropriate. (Recommendation 3) The Director of CDC should develop a plan for timely, consolidated reports of antibiotic resistance in priority pathogens at regular intervals. (Recommendation 4) The Secretary of HHS should identify leadership and clarify roles and responsibilities among HHS agencies to assess the clinical outcomes of diagnostic testing for identifying antibiotic-resistant bacteria. (Recommendation 5) The Commissioner of FDA should direct the Center for Devices and Radiological Health to conduct additional monitoring and evaluation of the status of FDA-authorized tests that rely on breakpoints, on a regular basis, to determine whether test manufacturers are updating breakpoints, seeking additional resources as needed. (Recommendation 6) The Secretary of HHS should develop a strategic framework to further incentivize the development of new treatments for antibiotic-resistant infections, including through the use of postmarket financial incentives, and, if appropriate, make recommendations to Congress for necessary authority. (Recommendation 7) The Secretary of HHS should direct the CARB Task Force to include in its annual updates to the President plans for addressing any barriers preventing full implementation of the National Action Plan and, as appropriate, make recommendations for new or modified actions. Specifically, the CARB Task Force should identify plans to address barriers, such as those related to expanding (1) a CDC program designed to strengthen the U.S. response to resistant gonorrhea; (2) antibiotic stewardship programs across health care settings; and (3) antibiotic use data collection across health care settings, to the extent feasible. (Recommendation 8) We provided a draft of this report to DOD, VA, and HHS for review and comment. DOD and VA did not provide formal comments but generally agreed with our report. In its comments, reproduced in appendix V, HHS generally concurred with our findings and seven of our recommendations, and did not concur with one of our recommendations, as discussed below. HHS identified several actions it intends to take to address our recommendations. DOD and HHS also provided technical comments, which we incorporated as appropriate. In response to our first recommendation, HHS concurred and CDC stated it is working with public health partners to promote the voluntary use of the AR Option, providing technical support to states that may be considering a state or local mandate to require AR and AU reporting, and developing pilot programs to assess AR Option data and other data sources for certain types of antibiotic resistance. While these actions are helpful, we believe taking additional steps, such as determining goals for participation rates and distribution for AR Option reporting, would give CDC more reasonable assurance that hit can conduct regional and national assessments of resistance. In response to our second recommendation, HHS concurred and CDC stated it is taking additional efforts to examine the representativeness of data collected through its primary surveillance system for resistant gonorrhea, including working to develop laboratory methods to reduce dependence on cultured isolates. CDC stated that steps to refine and improve collection of resistant gonorrhea data require additional resources. We believe that CDC requesting such resources would help ensure that such data are representative of the overall U.S. population. HHS generally concurred with our third recommendation. CDC stated it feels that it is critical to publish the data after peer review and then plans to link the publications back to online resources of the 2019 Threats Report. We believe that peer-reviewed publication is important, but it is also important for CDC to take additional steps to establish and report uncertainties for the national estimates or summary data that would help CDC and others draw appropriate conclusions about the characteristics of antibiotic resistance in the United States. In response to our fourth recommendation, HHS concurred and CDC stated it has plans to update its enterprise-wide AR Threats Report every three years, and that it also issues regular reports on specific groups of pathogens. In response to our fifth recommendation, HHS concurred and stated that the CARB Task Force leadership will work with relevant HHS agencies to clarify roles and responsibilities and identify leadership, if appropriate, for supporting research on clinical outcomes delated to diagnostic tests. HHS concurred with our sixth recommendation, and FDA concurred with conducting additional monitoring and evaluation of tests relying on breakpoints when FDA identifies or recognizes new breakpoints. FDA stated that it has taken major steps to help address challenges associated with updating such tests to reflect the most current breakpoints. We believe that in addition to these steps, monitoring and evaluation of current FDA-authorized tests that may still be using out-of- date breakpoints will enhance FDA’s ability to provide assurance that patient care and infection control efforts are effective. HHS did not concur with our seventh recommendation that HHS should develop a strategic framework to further incentivize the development of new treatments for antibiotic-resistant infections, including through the use of postmarket financial incentives. HHS noted that, while it agrees that additional incentives are needed to address the limited pipeline for novel and innovative treatments to combat antibiotic resistance, it is still conducting analyses to understand whether postmarket incentives should be included as a component of its forthcoming strategic framework to further incentivize the development of new treatments. However, HHS did not specify when its framework would be released. We support HHS’s efforts to develop such a framework, as this is a complex issue with multiple factors to consider. However, we believe our recommendation is still warranted. Antibiotic resistance is one of the greatest global public health threats, and experts, including the WHO, have warned that the pipeline of new antibiotics in development is insufficient to combat the threat. Without an adequate arsenal of treatments, we are likely to see increasing mortality caused by these deadly infections. As we reported, experts, advisory groups, federal officials, and antibiotic developers have all called for additional postmarket incentives to reinvigorate the pipeline of antibiotics under development. The current significant federal investment in push incentives to support antibiotic R&D is helpful but will ultimately be ineffective if companies receiving this investment are unable to sustain their business once their treatment reaches the market. Therefore, we maintain that it is important that HHS not delay the development of a strategic framework that includes postmarket incentives, which is just an initial step toward the creation of these incentives. Until additional postmarket incentives are developed, more drug companies may exit the antibiotic development sector, and the pipeline of new treatments for antibiotic-resistant infections may continue to decrease. In response to our eighth recommendation, HHS concurred and stated that beginning in 2020 and continuing annually thereafter, the CARB Task Force’s progress reports will include discussion of any barriers preventing full implementation of the National Action Plan, including, as appropriate, barriers that GAO has identified. We emphasize that the CARB Task Force should also identify plans to address such barriers—and, as appropriate, make recommendations for new or modified actions—in future progress reports, in accordance with Executive Order No. 13676. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to appropriate congressional committees; the Secretaries of DOD, HHS, and VA; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact us at (202) 512-6888 or personst@gao.gov, or (202) 512-7114 or deniganmacauleym@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. This report examines: (1) the Centers for Disease Control and Prevention’s (CDC) efforts to address surveillance of antibiotic resistance and any challenges to these efforts; (2) federal efforts to advance the development and use of diagnostic tests for identification and characterization of resistant bacteria and to address barriers to the development of diagnostic tests; (3) challenges to developing new treatments for antibiotic-resistant infections and federal efforts to address the challenges; and (4) federal efforts to promote the appropriate use of antibiotics and any challenges that remain. We focused our review primarily on agency actions since 2015, when the National Action Plan for Combating Antibiotic-Resistant Bacteria (National Action Plan) was published. We also focused our review on human health, as we have reported on federal efforts to address the use of antibiotics in food animals and recommended actions to improve these efforts for more than 20 years. Additionally, we focused our review on antibiotic-resistant bacteria. We generally excluded federal efforts related to infection prevention and control in human health care, on which we have previously reported. To address all four objectives, we reviewed relevant agency reports and documents, such as CDC’s report, Antibiotic Resistance Threats in the United States, 2013 (2013 Threats Report); conducted interviews with officials from federal agencies, experts, and stakeholder organizations; and we reviewed relevant literature, policy papers, and GAO reports. We interviewed officials from federal agencies responsible for implementing the aspects of the National Action Plan related to our research objectives: the Department of Health and Human Services’ (HHS) Office of the Assistant Secretary for Planning and Evaluation, the Biomedical Advanced Research and Development Authority (BARDA), CDC, the Centers for Medicare & Medicaid Services (CMS), the Food and Drug Administration (FDA), the National Institutes of Health (NIH), and the Office of Global Affairs; as well as the Department of Defense (DOD) and the Department of Veterans Affairs. We also interviewed experts and representatives from organizations involved in public health and epidemiology, infectious diseases and microbiology, antibiotic research and development (R&D), antibiotic stewardship, and other issues relating to antibiotic resistance. Because antibiotic resistance is a global problem, we also interviewed officials from the World Health Organization (WHO), the European Centre for Disease Prevention and Control, the European Medicines Agency, the Wellcome Trust, Public Health England, and the Surveillance and Epidemiology of Drug-Resistant Infections Consortium about various aspects of our review; and we reviewed relevant documents from these entities. We identified experts and organizations through literature and other documents we reviewed and through referrals from agency officials and other experts we interviewed. In addition, we attended several meetings and reviewed summaries of meetings held by the Presidential Advisory Council on Combating Antibiotic-Resistant Bacteria (PACCARB). Furthermore, we attended two conferences related to antibiotic resistance: the World Anti-Microbial Resistance Congress and the Gordon Research Conference on chemical and biological threat defense, the latter of which had a session devoted to antibiotics and antibiotic resistance. For each of our objectives, we identified and reported on actions taken by federal agencies and key challenges that the agencies face in addressing antibiotic resistance. We evaluated the actions taken by federal agencies against relevant criteria, as applicable. In addition, in September 2018, we convened a meeting of experts in antibiotic resistance epidemiology, diagnostic testing, antibiotic development, and antibiotic stewardship. This meeting of experts was planned and convened with the assistance of the National Academy of Sciences to better ensure that a breadth of expertise was brought to bear in its preparation; however, all final decisions regarding meeting substance and expert participation are the responsibility of GAO. Any conclusions and recommendations in GAO reports are solely those of the GAO. The Board on Population Health and Public Health Practice within the National Academy of Sciences solicited expert nominations from academia, public health laboratories, industry, and other organizations working in topics relating to antibiotic resistance. From their list of 51 nominees, and additional nominees we independently identified, we convened a meeting of 18 experts selected for their knowledge and expertise related to antibiotic resistance epidemiology, diagnostic testing, antibiotic development, and antibiotic stewardship. Eleven of the 18 experts who participated in our meeting also reviewed and provided comments on a draft of our report. We refer to such experts in this report as “experts at our meeting;” appendix II contains a list of the expert participants. To examine CDC’s efforts to address surveillance for antibiotic resistance and any challenges to these efforts, we reviewed documentation and conducted interviews with agency officials and other key stakeholders on each of the surveillance systems across CDC that track antibiotic resistance and reviewed CDC’s 2013 Threats Report and CDC’s Antibiotic Resistance Threats in the United States, 2019 data. We further focused our review on the 17 priority disease-causing bacteria listed in CDC’s 2013 Threats Report. The CDC surveillance systems included: Antibiotic Resistance Laboratory Network Emerging Infections Program (EIP) Gonococcal Isolate Surveillance Program (GISP) National Antimicrobial Resistance Monitoring System (NARMS) National Healthcare Safety Network (NHSN) National Notifiable Diseases Surveillance System National Tuberculosis Surveillance System For NHSN, we also assessed health care facility participation data by state and territory. We assessed the reliability of these data by reviewing them for any outliers or anomalies and by inquiring with agency officials about their source and any known reliability issues. We determined that these data were sufficiently reliable for assessing facility participation rates by U.S. state and territory. Stakeholder organizations we interviewed represented state and territorial epidemiologists and other public health officials (the Council of State and Territorial Epidemiologists and the Association of State and Territorial Health Officials) and an international consortium to address challenges in surveillance of antibiotic resistance (the Surveillance and Epidemiology of Drug-resistant Infections Consortium). We also reviewed reports on antibiotic resistance surveillance challenges from the Public Health Informatics Task Force and the Antibiotic Resistance Surveillance Task Force. We also reviewed documents from WHO’s global surveillance system and interviewed WHO and CDC officials to identify challenges that limit CDC’s ability to assess threats from abroad. We evaluated challenges and steps CDC has taken against CDC’s “Updated Guidelines for Evaluating Public Health Surveillance Systems;” Standards for Internal Control in the Federal Government; prior GAO work; the Government Performance and Results Act of 1993 (GPRA) and the GPRA Modernization Act of 2010; the Office of Management and Budget Circular No. A-11 and Standards and Guidelines for Statistical Surveys; relevant National Action Plan objectives, aims, and milestones; and Executive Order No. 13676, September 2014. To examine federal efforts to advance the development and use of diagnostic tests, we also interviewed representatives from a nongeneralizable selection of six diagnostic test manufacturers to identify challenges they face in developing tests for antibiotic resistance and challenges in increasing user adoption of their tests. We further focused our review on the 17 priority disease-causing bacteria listed in CDC’s 2013 Threats Report. The six manufacturers we interviewed were Accelerate Diagnostics, Beckman Coulter, BioFire and its parent company, BioMerieux, Bruker, Cepheid, and Roche Diagnostics. We identified these manufacturers by compiling a list based on previous work we conducted, interviews with select experts, and internet search. We selected six manufacturers that were identified by more than one source while encompassing different types of tests (culture and genotypic). We limited our scope to FDA-authorized tests—that is, tests that have been reviewed and cleared by FDA for marketing in the United States—that can identify resistance in at least one type of bacteria categorized as priority bacteria in CDC’s 2013 Threats Report. Some of these tests are called antibiotic susceptibility tests, but we refer to the entire class of such tests as “tests.” We included in our scope tests that can differentiate between viral and bacterial infection because these types of tests are included in the National Action Plan. We evaluated the actions taken by federal agencies against the Standards for Internal Control in the Federal Government, relevant National Action Plan objectives, aims, and milestones under Goal 3, and relevant sections in the PACCARB Recommendations for Incentivizing the Development of Vaccines, Diagnostics, and Therapeutics to Combat Antibiotic Resistance. We also evaluated federal agency actions against the “leadership” and “clarity of roles and responsibilities” leading practices from GAO’s Managing for Results: Key Considerations for Implementing Interagency Collaborative Mechanisms. We focused on these key practices when there was a lack of specifically assigned roles in either the National Action Plan or the PACCARB report for key activities. To identify challenges to developing new treatments for antibiotic- resistant infections and examine federal efforts to address these challenges, we also interviewed 11 randomly selected companies that conduct research and development on new treatments for bacterial infections. We included companies that are researching or developing both traditional antibiotics and alternatives to antibiotics—which we call “nontraditional products” in this report—and we included companies that had and had not received funding from the Combating Antibiotic- Resistant Bacteria Biopharmaceutical Accelerator (CARB-X) and companies that do and do not have existing FDA-approved drugs on the market. We asked company representatives about challenges in developing new antibiotics they have identified, support they may have received from federal agencies, how effective the support has been to them, and their views on additional incentives that would promote the development of new antibiotics. We also interviewed experts on the topic of antibiotic development and industry stakeholders, specifically The Pew Charitable Trusts and the Biotechnology Innovation Organization. We interviewed federal officials from BARDA, CMS, DOD, FDA, and NIH to learn about their programs and actions to support the development of treatments for antibiotic-resistant infections and requested information about funding for antibiotic R&D from BARDA, DOD, and NIH. We included relevant agency actions that began before the National Action Plan was issued in 2015 if they continued after 2015. Finally, we reviewed literature related to antibiotic development and reports about antibiotic pull incentives written by health policy advisory groups, including the PACCARB, the Transatlantic Taskforce on Antimicrobial Resistance (TATFAR), the DRIVE-AB project, and the Duke Margolis Center for Health Policy. We evaluated the actions taken by federal agencies to help address the challenges to developing new treatments against experts’ and advisory groups’ views on additional actions needed and against the principles related to developing strategies outlined in GPRA and the GPRA Modernization Act of 2010. We did not assess challenges to developing products designed to prevent infections, such as vaccines, nor federal actions related to these types of products. To examine federal agency efforts to promote the appropriate use of antibiotics and any challenges that remain, we also analyzed CMS data and related documentation on the quality measures and improvement activities related to antibiotics as part of CMS’s Merit-based Incentive Payment System (MIPS) in 2017. Specifically, we identified CMS’s antibiotics-related quality measures and improvement activities by conducting a search for the words “antibiotic,” “antimicrobial,” “bacteria,” “resistance,” and “resistant” on CMS’s MIPS website. We then reviewed CMS’s data on the number of MIPS-eligible clinicians who selected and reported on these measures and activities in 2017, the most recently available data. In 2017, there were nine MIPS quality measures related to antibiotics, as follows: (1) acute otitis externa: systemic antimicrobial therapy - avoidance of inappropriate use; (2) adult sinusitis: antibiotic prescribed for acute sinusitis (overuse); (3) adult sinusitis: appropriate choice of antibiotic: amoxicillin with or without Clavulanate prescribed for patients with acute bacterial sinusitis (appropriate use); (4) appropriate testing for children with pharyngitis; (5) appropriate treatment for children with upper respiratory infection; (6) appropriate treatment of Methicillin-sensitive Staphylococcus aureus bacteremia; (7) avoidance of antibiotic treatment in adults with acute bronchitis; (8) perioperative care: selection of prophylactic antibiotic – first- or second-generation Cephalosporin; and (9) total knee replacement: preoperative antibiotic infusion with proximal tourniquet. In addition, there was one MIPS improvement activity related to antibiotics in 2017: implementation of antibiotic stewardship program. We reviewed the MIPS data for any obvious outliers or anomalies, and we determined that these data were sufficiently reliable for reporting the number of clinicians who reported implementing these quality measures and improvement activities. In addition, we reviewed aggregated data from CDC on the total number of eligible U.S. hospitals voluntarily reporting their antibiotic use data to a CDC system (the NHSN’s Antimicrobial Use Option); we then calculated the percentage of eligible hospitals reporting such data as of January 1, 2020. We assessed the reliability of the aggregated data by reviewing them for any obvious errors or missing data totals and inquiring with CDC officials about their source and any known reliability issues. We determined that these data were sufficiently reliable for reporting hospital participation rates in the system. We also reviewed selected articles on antibiotic use and stewardship— compiled from a variety of sources, including CDC documents and experts we interviewed—published in literature. In addition, we interviewed experts on antibiotic use and stewardship, including representatives from PACCARB, Emory University’s School of Medicine, the University of Minnesota’s Center for Infectious Disease Research and Policy, The Joint Commission, the Society of Infectious Diseases Pharmacists, The Pew Charitable Trusts, and the Association for Professionals in Infection Control and Epidemiology. We evaluated federal efforts and challenges against relevant National Action Plan objectives and milestones and Executive Order No. 13676. We focused on antibiotic use in the United States, rather than global antibiotic use. We conducted this performance audit from February 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We collaborated with the National Academy of Sciences to convene a two-day meeting of experts to inform our work on federal efforts to address antibiotic resistance; the meeting was held on September 17 and 18, 2018. The experts who participated in this meeting are listed below. Many of these experts gave us additional assistance throughout our work, including by providing additional technical expertise and answering questions, and 10 of these experts reviewed and provided comments on our draft report for technical accuracy. This appendix contains additional examples of efforts by agencies within the Departments of Health and Human Services, Defense, and Energy to provide support for antibiotic research and development beyond those mentioned in the report. These examples do not comprise the full extent of agencies’ efforts. This appendix contains more detailed information on federal efforts to promote the appropriate use of antibiotics in health care through antibiotic stewardship programs and activities, organized by agency. These examples do not comprise the full extent of agencies’ efforts. In addition to the contacts named above, John Neumann (Managing Director); Will Hadley, Anne K. Johnson, and Sushil K. Sharma, PhD, DrPH (Assistant Directors); Josey Ballenger, Hayden Huang, PhD, and Laura Tabellion (Analysts-in-Charge); and Amber Sinclair, PhD, made key contributions to this report. Nora Adkins, George Bogart, Jehan Chase, Anika McMillon, Laurie Pachter, Eric Peterson, Sarah Sheehan, Ben Shouse, Sara Sullivan, Walter Vance, Harris Weisz, and Emma Williams also made important contributions.", "summary": "Bacterial infections have become more difficult, and sometimes impossible, to treat due to antibiotic resistance, which occurs when bacteria develop the ability to defeat the available drugs designed to kill them. Concerns about rising rates of resistance to available treatment options prompted the federal government to create the 5-year National Action Plan in 2015. The plan called for federal agencies to strengthen surveillance, advance the development of diagnostic tests and new antibiotics, and slow the emergence of resistant bacteria, among other things. GAO was asked to review federal efforts to address antibiotic resistance. This report examines federal efforts and challenges related to (1) surveillance of antibiotic resistance, (2) the development and use of diagnostic testing to identify antibiotic resistance, (3) the development of treatments for resistant infections, and (4) appropriate antibiotic use. GAO reviewed literature and agency documents; interviewed agency officials and health care industry, drug industry, and other stakeholders; and held a meeting of international and U.S. experts to obtain their views. The precise magnitude of the problem of antibiotic resistance is unknown. The Centers for Disease Control and Prevention (CDC) has made progress in expanding surveillance of infections from certain antibiotic-resistant bacteria in the United States and abroad but faces several challenges. Note: This figure tracks a type of carbapenem-resistant Enterobacteriaceae (CRE), which, according to CDC, is a “nightmare bacteria” resistant to nearly all available antibiotics. Shading indicates CDC confirmed the presence of these bacteria within that state in that year or a previous one. CDC faces challenges in conducting surveillance for antibiotic resistance due to the limited data it is able to collect through various surveillance systems. For example, CDC's primary surveillance system for gonorrhea—which CDC classified as an urgent antibiotic resistance threat affecting over half a million patients annually—currently tracks only an estimated 1 to 2 percent of all U.S. cases and only in males. CDC has not fully evaluated the representativeness of the gonorrhea surveillance system's results. However, it could do so, for example, by comparing the trends in their limited sample population with trends it can establish in the overall U.S. population via additional studies. Such an evaluation could give CDC more confidence that the system's data accurately reflect national trends. Federal agencies have taken steps to advance the development and use of diagnostic tests to identify antibiotic-resistant bacterial infections, but these efforts have limitations. For example, agencies have conducted some studies to establish whether testing can lead to positive health care outcomes, such as reduced rates of antibiotic-resistant infections. However, more such studies are needed, according to experts and agency officials. Without information to guide test usage, clinicians may not be able to select appropriate treatments for their patients. One reason for the insufficient number of studies is that Department of Health and Human Services (HHS) agencies that are in a position to conduct or fund such studies—such as CDC and the Biomedical Advanced Research and Development Authority—disagree about what each agency should do. By clarifying roles and responsibilities, HHS agencies could more effectively address the need for more studies. The resulting studies could help demonstrate the value of diagnostic tests for antibiotic resistance, potentially increasing their use and improving patient care. Experts warn that the current pipeline of antibiotics in development is insufficient to meet the threat of resistance. Several challenges impede the development of new treatments for resistant infections, notably inadequate return on investment for drug companies largely due to low prices and a limited patient population for whom these treatments would be appropriate. While HHS and Department of Defense agencies have provided financial premarket incentives to support antibiotic research and development, experts, federal officials and antibiotic developers agree that more postmarket incentives are needed to overcome the economic challenges. Advisory groups, including a presidential advisory council, and others have called for new postmarket incentives and identified multiple options for their design, including market entry rewards and reimbursement reform (see figure). However, HHS has not developed a strategy to further incentivize development of new treatments for antibiotic-resistant infections, and it may need to request authority and appropriations to create and implement certain types of incentives. Until such incentives are developed, more drug companies may exit the antibiotic development sector, and the pipeline of new treatments may continue to decrease. Federal agencies have made several efforts to promote the appropriate use of antibiotics across health care settings through antibiotic stewardship—giving patients the right antibiotic at the right time, in the right dose, and for the right duration. However, key challenges remain. For example, federal agencies require only certain types of health care facilities to implement stewardship programs. In addition, CDC is limited in its ability to monitor and improve appropriate antibiotic use, in part because providers are not generally required to report antibiotic use data to a centralized database. The 5-year National Action Plan for Combating Antibiotic-Resistant Bacteria (National Action Plan) calls for strengthening antibiotic stewardship and for the timely reporting of antibiotic use data across health care settings. An executive order directs an interagency task force—the Combating Antibiotic-Resistant Bacteria (CARB) Task Force, coordinated by HHS—to provide annual updates to the President on, among other things, plans for addressing any barriers to full implementation of the National Action Plan. However, in its progress reports covering the first 4 years of the National Action Plan's implementation, the task force did not identify plans to address barriers to expanding antibiotic stewardship programs or the collection of antibiotic use data. Until it does so, the government will not have reasonable assurance that it is fully implementing the National Action Plan and addressing antibiotic resistance. GAO is making eight recommendations to strengthen the federal response to combating antibiotic resistance. HHS concurred with seven recommendations and did not concur with one. More details are provided on the next page. In response to the findings presented in this Highlights, GAO recommends that: CDC ensure that its evaluation of its surveillance system for antibiotic-resistant gonorrhea includes measures of the system's representativeness of the U.S. population; HHS identify leadership and clarify roles and responsibilities to assess the clinical outcomes of diagnostic testing; HHS develop a strategy to further incentivize the development of new treatments for antibiotic-resistant infections, including through the use of postmarket financial incentives; HHS direct the CARB Task Force to include in its annual updates to the President plans for addressing any barriers preventing full implementation of the National Action Plan. In addition, GAO is making four recommendations to address other CDC efforts in surveillance and reporting and to address FDA efforts in monitoring diagnostic tests. HHS did not concur with the recommendation that it develop a strategy that includes the use of postmarket financial incentives to encourage the development of new treatments for antibiotic-resistant infections, citing its ongoing analysis to understand whether postmarket incentives should be included in such a strategy. GAO recognizes the complexity of this issue and maintains that this recommendation is warranted given that experts and others have called for additional postmarket incentives and the insufficiency of the current pipeline of new treatments for antibiotic-resistant infections. or Mary Denigan-Macauley at (202) 512-7114 or deniganmacauleym@gao.gov .", "document_type": "gao"}
{"report": "Federal law and regulations require that contractors receiving a contract with a value greater than the simplified acquisition threshold must ensure that small businesses have the “maximum practical opportunity” to receive subcontracting work. In addition, a prospective contractor generally must submit a subcontracting plan for each solicitation or contract modification with a value of more than $700,000—or $1.5 million for construction contracts—whenever subcontracting opportunities exist. Contractors with federal contracts typically use one of three types of subcontracting plans: Individual subcontracting plan, which applies to a specific contract, covers the entire contract period including option periods, and contains subcontracting goals; Commercial subcontracting plan, which covers the company’s fiscal year and the entire production of commercial items sold by either the entire company or a portion of it (such as a division, plant, or product line) and contains subcontracting goals; and Comprehensive subcontracting plan, which is similar to a commercial subcontracting plan and applies only to DOD contracts. Each company reports on subcontracting goals and achievements for a specific fiscal year on a plant, division, or corporate-wide basis. A comprehensive plan may cover a large number of individual contracts. Federal contractors use these plans to document subcontracting goals as a specific dollar amount planned for small business awards and as a percentage of total subcontracting dollars available to small businesses and socioeconomic categories of small businesses. Contractors also may establish, for specific facilities, a master subcontracting plan that contains all the required elements of an individual plan, except the subcontracting goals. Because a master plan does not include specific subcontracting goals, an individual subcontracting plan or an addendum typically provides the goals for specific contracts associated with the master subcontracting plan. After a contract is awarded, the contractor must periodically submit to the government a subcontracting report that describes progress towards meeting these goals. Individual subcontracting plans require reporting on a single contract and commercial and comprehensive subcontracting plans allow for consolidated reporting of multiple contracts on a division- or company-wide basis. Contractors must report their subcontracting achievements through eSRS, a web-based government-wide system that both contractors and agency contracting officers can access. The FAR requires contractors to submit individual subcontracting reports (ISR) and summary subcontract reports (SSR) (see table 1). These reports show contractors’ progress toward meeting their small business subcontracting goals. Several regulations, processes, and procedures dictate contracting officers’ responsibilities for oversight of subcontracting plans during the pre-award and post-award phases of the acquisitions process. Before making an award, the FAR requires that contracting officers review the subcontracting plan to help ensure that the required information, goals, and assurances—such as a contractor committing to submit periodic reports to the government to determine the extent of compliance with the subcontracting plan—are included. Additionally, the FAR requires contracting officers to provide the SBA Procurement Center Representative (PCR)—SBA staff whose responsibility includes supporting agency contracting opportunities for small businesses—with an opportunity to review the proposed contract, including the subcontracting plan and supporting documentation. After a contract or contract modification containing a subcontracting plan is awarded or an existing subcontracting plan is amended, the FAR requires that contracting officers monitor the prime contractor’s compliance with its subcontracting plan. In carrying out their post-award oversight responsibilities, the FAR requires contracting officers to (1) ensure contractors file their subcontracting reports in eSRS within 30 days of the close of each reporting period (a report is also required for each contract within 30 days of contract completion); (2) review ISRs, and where applicable SSRs, in eSRS within 60 days of the reporting end date; and (3) acknowledge receipt of, accept, or reject the reports in eSRS (see fig.1). The FAR requires agencies to perform annual evaluations of and report on a contractor’s performance when work under the contract has been completed. Small business subcontracting is one evaluation area for which agencies rate a contractor’s performance. Agencies use the Contractor Performance Assessment Reporting System to collect and manage the library of Contractor Performance Assessment Reports. Agency contracting officers are to consider information on a contractor’s past performance from these reports when making future contract award decisions, including a contractor’s actions for previously awarded contracts that had a small business subcontracting plan. The FAR also requires contractors to comply in good faith with the agreed-upon subcontracting plan goals and requirements. When a contractor fails to meet the small business goals in the subcontracting plan, the contractor must provide a rationale for not being able to meet the goals. In determining whether a contractor failed to make a good-faith effort, a contracting officer must look at the totality of the contractor’s actions, consistent with the information and assurances provided in its subcontracting plan, and consider the rationale the contractor provided. The contractor’s failure to meet its subcontracting goals does not, in and of itself, constitute failure to make a good-faith effort. Failure to submit required subcontracting reports as required by the FAR also may factor into contracting officers’ determinations. If a contracting officer determined that a contractor failed to make a good-faith effort, the FAR requires the contracting officer to assess liquidated damages (monetary assessments for breaching the contract) against the contractor. SBA’s Office of Government Contracting administers SBA’s subcontracting assistance program. In this office, headquarters and field staff have responsibilities to assist small businesses in meeting requirements to receive government contracts as subcontractors. SBA staff have related responsibilities in both the pre- and post-award acquisition phases. For example, SBA’s PCRs and Commercial Market Representatives (CMR) play a role in helping to ensure that small businesses gain access to subcontracting opportunities. In particular, a PCR’s key responsibilities include reviewing proposed agency contracts and making recommendations to agency contracting officers. PCRs also review proposed subcontracting plans and provide advice and recommendations on them to contracting officers. Key responsibilities of CMRs include counseling small businesses on obtaining subcontracts and conducting reviews, including compliance reviews, of large prime contractors with subcontracting plans. SBA’s standard operating procedure (SOP) for the subcontracting assistance program provides guidance for how CMRs conduct reviews. Although SBA conducts monitoring activities, the awarding federal agency remains responsible for overseeing and enforcing compliance with a subcontracting plan throughout the life of the contract. In the case of DOD, in addition to the components within the agency that award and monitor contracts, the Defense Contract Management Agency (DCMA) also provides contract administration services for certain DOD contracts. SBA and DCMA may conduct compliance reviews jointly to evaluate prime contractor subcontracting programs supporting specific DOD contracts administered by DCMA. SBA is also authorized to enter into agreements with other federal agencies to conduct compliance reviews and further the objectives of the subcontracting program. We discuss SBA oversight in more detail later in the report. Annually, SBA negotiates with agencies to establish individual small business subcontracting goals based on recent subcontracting achievement levels by each agency. Agencies awarding contracts with small business subcontracting plans aim to provide opportunities to promote the use of small businesses, veteran-owned small businesses, service-disabled veteran-owned small businesses, Historically Underutilized Business Zone small businesses, small disadvantaged small businesses, and women-owned small businesses. These efforts can help agencies achieve their individual small business subcontracting goals. The four agencies we reviewed—DLA, GSA, Navy, and NASA— demonstrated that contracting officers reviewed and approved subcontracting plans in most of the contracts in our sample. However, they could not demonstrate they followed procedures for making a determination of subcontracting possibilities for all of the contracts we reviewed without a subcontracting plan. Agencies also could not demonstrate they followed procedures related to PCR reviews in about half of the contracts reviewed. The four agencies provided documentation to show that contracting officers reviewed and approved subcontracting plans in most of the 26 contracts that had subcontracting plans. FAR §§ 19.705-4 and 19.705-5 contain contracting officer responsibilities related to reviewing a proposed subcontracting plan and determining its acceptability. For 25 of the 26 contracts we reviewed with a subcontracting plan, the agencies provided documentation showing the contracting officer reviewed the subcontracting plan. In some instances, we also found specific agency guidance for, and checklists or memorandums documenting the reviews of, subcontracting plans. For example: GSA has guidance for its contracting officers when reviewing subcontracting plans. Specifically, GSA’s Acquisition Manual includes a checklist for reviewing subcontracting plans and ensuring the plans meet FAR requirements. Contracting officers used the checklist in their reviews for five of the six GSA contracts we reviewed with a subcontracting plan. The checklist also documents whether the total planned subcontracting dollars and percentages, the method for developing these goals, and information about supplies or services that will be subcontracted are acceptable to the contracting officer. DOD’s guidance on subcontracting program business rules and processes contains a specific DOD checklist for subcontracting plan reviews. Contracting officers used the DOD checklist for three of 14 DLA and Navy contracts with a subcontracting plan that we reviewed. In addition to documenting the extent to which a subcontracting plan meets FAR and Defense Federal Acquisition Regulation Supplement requirements, the checklist also reflects certain requirements related to master and commercial subcontracting plans. The checklist is optional for contracting officers to use when reviewing subcontracting plans. NASA also has guidance that includes steps contracting officers should take when conducting subcontracting plan reviews. For two of the six NASA contracts with a subcontracting plan that we reviewed, we found a checklist that the contracting officer used or a memorandum the contracting officer prepared that detailed the subcontracting plan review, including proposed subcontracting goals. For almost all the contracts we reviewed that did not have a specific checklist or memorandum to document the contracting officer’s review, we found other evidence, such as a contracting officer’s signature on the subcontracting plan, acknowledging review of the plan. Additionally, for one Navy contract with a contract award value of more than $13 million and with an individual subcontracting plan, we found evidence that, after reviewing the subcontracting plan, the contracting officer requested that the contractor make corrections to it. For one DLA contract we reviewed, based on the limited documentation provided, we were unable to determine the extent to which the subcontracting plan was reviewed. DLA officials stated at the time of our review that they were unable to determine if the subcontracting plan was reviewed. We also obtained documentation that demonstrated the subcontracting plan was approved for most of the contracts—21 of 26—we reviewed with a subcontracting plan. For example, we obtained documentation with the contracting officer’s signature on the subcontracting plan (approving the plan), the contracting officer’s signature approving the contract (which included the subcontracting plan), or a signed memorandum that documented approval of the plan. However, we identified five contracts across DLA, Navy, and GSA that had limited documentation (three contracts) for approval of the subcontracting plan, or for which we could not determine whether the subcontracting plan was approved (two contracts). For one DLA contract with an award amount of $15 million and with an individual subcontracting plan, we were unable to determine if the subcontracting plan was approved. Documentation we reviewed, including DLA emails, did not indicate whether the subcontracting plan was approved. In our review of the subcontracting plan, the section of the plan documenting its approval was not completed. Additionally, according to DLA officials, the contract file does not contain any record of the contracting officer’s signature on the subcontracting plan. For two Navy contracts with award amounts of about $17 million and about $32 million and both with individual subcontracting plans, we found limited documentation demonstrating approval of the subcontracting plan for the first contract and, based on the lack of documentation, were unable to determine if the second contract was approved. For the first contract, we found a checklist with signatures demonstrating review of the subcontracting plan by the contracting officer and other officials. However, the subcontracting plan was not signed by the contracting officer as the approval/signature field in the subcontracting plan was empty. For the other contract, Navy officials could not provide any documentation showing approval of the subcontracting plan. The subcontracting plan was not signed by a Navy contracting officer or other Navy staff, and according to Navy officials, they were unable to find a signed subcontracting plan in the pre-award contract file. For two GSA contracts with individual subcontracting plans, we also found limited documentation approving the subcontracting plan. Similar to one of the Navy contracts discussed above, we found checklists with signatures demonstrating reviews of the subcontracting plan by the contracting officer and other officials. However, in both of these instances, the contracting officer did not sign the approval section of the subcontracting plan. Additionally, for one DLA contract we reviewed with an individual subcontracting plan and contract award amount of about $18 million, while we found documentation indicating that the contract had been approved, DLA could not provide documentation for a DOD requirement related to a socioeconomic subcontracting goal. Specifically, the subcontracting plan for this contract listed the small disadvantaged business goal at less than 1 percent. According to Defense Federal Acquisition Regulations Supplement § 219.705-4, a small disadvantaged business goal of less than 5 percent must be approved one level above the contracting officer. In our review of this contract, DLA could not provide documentation specifically showing a higher-level approval for the goal of less than 1 percent. DLA provided an interoffice record and a signed price negotiation memorandum approval document, but these documents did not reference the small disadvantaged business subcontracting goal of less than 1 percent. As a result, we were unable to determine that this subcontracting goal was approved at the appropriate level. In addition to the 26 contracts with subcontracting plans, we also reviewed another six contracts that initially appeared to require a subcontracting plan (based on data in FPDS-NG) but did not have one. For three of the six contracts, the contracting officer or relevant official did not document why a subcontracting plan had no subcontracting possibilities, or prepared the required documentation years after the contract award. For contracts over $700,000, the FAR generally requires contracting officers to award the contract with a subcontracting plan or to make a determination that no subcontracting possibilities exist. If the contracting officer determines that there are no subcontracting possibilities, the determination should include a detailed rationale, be approved at one level above the contracting officer, and be in the contract file. GSA accounted for one of the three contracts and NASA for the remaining two. A subcontracting plan was not included in a GSA construction contract with an award amount of about $7 million (which met requirements for a small business subcontracting plan based on the award amount and type of contract). GSA did not have any documentation and could not tell us why the contract did not require a subcontracting plan or had no subcontracting possibilities, or why a subcontracting plan was not included in the contract. Specifically, GSA provided a response explaining the agency did not have documentation to support why the contracting officer (who is no longer with the specific contracting center that awarded the contract) determined there were no subcontracting possibilities. For two NASA contracts, NASA officials provided documentation signed by one level above the contracting officer, but the documentation was prepared years after the contract award. For the first contract, with an award value of almost $8 million and awarded in March 2016, the determination providing the rationale for no subcontracting possibilities was created and signed in March 2019, about 3 years after the contract was awarded instead of when the award was made. For the second NASA contract, awarded in September 2017 with a contract award amount of about $2 million, NASA officials explained that in 2017, the initial procurement was estimated at a dollar amount below the threshold for a subcontracting plan and therefore no subcontracting plan was required in the solicitation. The contract value was later changed to add two option periods, which put the estimate over the subcontracting plan threshold. NASA officials said the contracting officer’s documentation to determine the need for a subcontracting plan was inadvertently omitted from the file. As a result of our document request, the reviewing contracting officer noted that the file did not properly address the issue of the increased estimate relative to subcontracting plan requirements. NASA then conducted a review to determine if the award met the requirements for a subcontracting plan or if it would have been waived in 2017. Based on the recent review, NASA officials determined that a requirement for a subcontracting plan would have been waived in 2017 based on, among other factors, the specific product purchased through the contract and the structure of the contract, and they prepared a memorandum (in July 2019) documenting this review and conclusion. A 2018 DOD OIG report on small business subcontracting at two Army contracting command locations found similar issues. Specifically, the report found that of 50 contracts the DOD OIG reviewed, the two contracting command locations awarded six contracts, valued at $330.7 million, without a subcontracting plan or a contracting officer’s determination that no subcontracting possibilities existed. The three other contracts we reviewed—two at DLA and one at GSA— had appropriate documentation directly explaining or a rationale supporting why no subcontracting plan was in place. For example, for one contract, DLA officials provided a memorandum signed at one level above the contracting officer that documented the specific nature of the contract for a particular type of metal, the work required, and ability of the contractor to perform the work in-house. For the second contract, DLA officials provided information that the contract was awarded through the AbilityOne Program—which does not require a subcontracting plan. The GSA contract was an automotive contract in which the vendor initially represented itself as a large business and had submitted a subcontracting plan. However, after the contract award, GSA documented a modification to the contract that reclassified the vendor as a small business, based on size standards for the North American Industry Classification System codes for the specific acquisition. Therefore, the subcontracting plan was no longer required. For half of the contracts we reviewed with a small business subcontracting plan (individual or commercial), the agencies could not demonstrate that procedures related to PCR reviews were followed for one or more contracts. According to FAR § 19.705-5(a)(3), when an agency is making a contract award that includes a subcontracting plan, contracting officers should notify the appropriate PCR of the opportunity to review the proposed contract, including the associated subcontracting plan and supporting documentation. More specifically, for 12 of 24 contracts we reviewed with an individual or commercial subcontracting plan, the agencies could not provide documentation or we were unable to determine from the documentation provided whether the contracting officer gave the SBA PCR a review opportunity and whether the PCR may have conducted a review. Of these 12 contracts, DLA and Navy accounted for 10, while GSA and NASA accounted for one each. Five of the six DLA contracts we reviewed did not have any documentation or lacked sufficient documentation to determine if the contracting officer or other official provided the PCR with an opportunity to review the contract, and whether a PCR review occurred. More specifically, DLA was unable to provide any documentation related to the PCR review process for three contracts with a subcontracting plan and told us they could not locate such documentation in the contract file. For one of these three contracts, DLA referred us to DCMA for additional documentation, but the documentation DCMA provided did not confirm whether the PCR had an opportunity to review the contract. For the remaining two of five contracts, DLA provided documentation, including a review by DCMA’s Small Business Office for one of the contracts, but this documentation did not demonstrate the contract was provided to an SBA PCR for review. Five of six Navy contracts we reviewed that had individual subcontracting plans also lacked this documentation. Specifically, Navy was unable to provide documentation specific to the PCR review process for three contracts. For two other contracts, Navy provided documentation of various internal reviews. For example, Navy provided a checklist for one contract showing that the contract was reviewed and signed by the contracting officer and a small business specialist. However, the section of the checklist where the PCR would sign indicating review of the contract and subcontracting plan was left blank. For the other contract, Navy provided documentation that an Assistant Deputy Director for the procuring contracting command center had reviewed and signed the subcontracting plan, but the PCR signature field was blank. In both cases, no other documentation indicated whether the contract was sent to the PCR for review. Therefore, we were unable to determine if a PCR reviewed the plan or was provided the opportunity to review the plan. GSA and NASA each had one contract (of the six we reviewed for each) for which they could not provide any documentation related to the PCR review process. Both of these contracts had an individual subcontracting plan. For the remaining 12 contracts across the four agencies, the agencies provided documentation demonstrating that the PCR was given the opportunity to and had reviewed the contract and associated subcontracting plan. For these contracts, we obtained documentation such as a memorandum, checklist, or email showing the PCR had reviewed and provided concurrence with the subcontracting plan, or commented on the proposed goals in the plan. According to officials from three of the four agencies we reviewed, contracting officers have a large workload with responsibility for a large number of processes and reviews, which may result in a specific process or task—such as coordinating the PCR review—being missed. Additionally, according to NASA officials, the PCR review process may occur but not be documented for some NASA contracts. The selected agencies provide some training to contracting officers on monitoring subcontracting plans. But, for most of the 26 contracts we reviewed with a subcontracting plan, contracting officers did not ensure contractors met their subcontracting reporting requirements. Contracting officers also accepted subcontracting report submissions with erroneous subcontracting goal information for several contracts. For more than half of the 26 contracts, contractors reported that they met or were meeting their small business subcontracting goal. Officials from all four agencies told us that they provide periodic training to contracting officers related to monitoring subcontracting plans, as illustrated in the following examples: NASA: According to a NASA official, NASA conducted training at the Kennedy Space Center in October 2018 and October 2019 that focused on whether contracting officers should accept or reject an ISR, and how to assign a Compliance Performance Assessment Report rating. The agency also conducted training at the Goddard Space Flight Center in October 2018. GSA: GSA’s Office of Small Business Utilization provided a refresher on eSRS reporting, including how to review the report in eSRS, for contracting officers in May 2018. They also provided training to contracting officers in October 2019 on reviewing ISRs and SSRs, including understanding how to review an ISR and ensuring timely submissions of SSRs. DLA: According to DLA staff with the DLA Contracting Services Office, when a contract requires a subcontracting plan, the office’s eSRS coordinator recommends that contracting personnel responsible for administering subcontracting plans take the Defense Acquisition University online course about eSRS. Navy: According to a Navy official, DOD has conducted extensive training to address eSRS known issues and data collection and guidance on the proper review of ISRs. Additionally, Navy contracting officers can enroll in a 5-day course on subcontracting offered by the Defense Acquisition University. According to Defense Acquisition University staff, in addition to the 5-day classroom course, the university also offers other training online related to subcontracting. For more than half of the 26 contracts we reviewed with a subcontracting plan, agency contracting officers did not ensure contractors met their reporting requirements. Specifically, 14 of 26 contracts with subcontracting plans did not have all required ISR or SSR submissions. Three of the four agencies—DLA, NASA, and Navy—accounted for the 14 contracts without all the required submissions. For the remaining 12 contracts we reviewed, the agencies provided documentation showing that contractors submitted all required ISR or SSR submissions for these contracts. FAR § 19.705-6(f) requires contracting officers to monitor the prime contractor’s compliance with subcontracting plans to ensure that subcontracting reports (ISRs and, where applicable, SSRs) are submitted in eSRS in the required time frames. The contracting officer is also to review the reports in the required time frames, acknowledge receipt of, and accept or reject the reports. Our review of 26 contracts with subcontracting plans found limited monitoring of contractor report submissions. Specifically, we found the following for each agency (see table 2): DLA. Five of the six DLA contracts we reviewed did not have all of the required ISR or SSR contractor submissions. For example, for a $6.6 million contract, with a commercial subcontracting plan that was awarded in fiscal year 2016, we could not locate any SSRs in eSRS. Based on limited documentation DLA provided, the contractor submitted only one SSR for the duration of the contract and did so by email to the contracting officer in November 2018. This document was not an official SSR and it did not include required information such as the vendor’s number, information on who submitted the report from the contractor, a self-certification statement attesting to the accuracy of the report, or acceptance or sign off by a DLA official. Four other DLA contracts with individual subcontracting plans had multiple missing submissions. For two of these contracts, the agency could not explain why the reports were missing, and for the other two contracts, the contractors were not aware of the SSR reporting requirement, according to a DLA official. NASA. Similar to DLA, five of the six NASA contracts we reviewed did not have all of the required ISR or SSR submissions. For example, for a $4.6 million contract with an individual subcontracting plan awarded in fiscal year 2016, the contractor submitted ISRs for 2016 and 2017 and the SSR for 2016. However, according to information we reviewed in eSRS and a NASA official, the contractor did not submit any ISRs for 2018 and 2019, and did not submit any SSRs for 2017 or 2018. The official stated that there was contracting officer turnover during this contract, and the contracting officer monitoring the contract at the time of our review could not find any documented explanation for the reports not being submitted. The same agency official explained that for another contract, the contractor experienced issues submitting documents in the electronic system initially and that there were personnel changes around the time the missing report was due. Additionally, for another contract awarded in 2017 for $3.8 million, the contractor did not submit any SSRs. We discuss the two remaining NASA contracts in our discussion of contracts with subcontracting report submissions that were submitted well past their due dates. Navy. Four of the eight Navy contracts we reviewed did not have all the required report submissions. For example, for one contract awarded for $16.6 million, the contractor submitted the first two required ISRs and an SSR for fiscal year 2016, the year in which the contract was awarded. However, we did not locate any other required submissions in eSRS for subcontracting activity in fiscal year 2017, the year in which the contract ended. A Navy official told us it is not unusual for information related to monitoring and compliance of subcontracting plans to be missing from the contract files. Three remaining contracts with individual subcontracting plans also had missing SSRs. However, the agency did not explain why these submissions were missing. GSA. The six GSA contracts all had the required report submissions. Additionally, contractors submitted ISRs or SSRs well past their required due dates for at least four contracts. For example, for one Navy contract and one DLA contract, we found that the contractors submitted an ISR more than 125 days late, and almost 50 days late, respectively. For two NASA contracts, contractors submitted reports after they were due. For one of these NASA contracts, we found that the March 2016 and September 2016 ISRs were submitted well past their due dates—more than 400 days and more than 150 days, respectively. For the second NASA contract, the contractor did not submit any of the required reports during the life of the contract and only submitted one final ISR when the contract ended. This contract was awarded in fiscal year 2016 and ended in August 2018. According to a NASA official, failure to submit the required subcontract report was an error by the contractor and insufficient contracting officer oversight. Additionally, the contractor did not submit any SSRs for this contract as required by the FAR. In another four instances, contractors began submitting the required reports (ISRs and SSRs) after we inquired about the specific contracts with the respective agencies. For example, the contractor for one NASA contract, which also had some missing subcontracting reports, submitted its 2017 SSR more than 600 days after it was due, and after we inquired with NASA about the SSR. We also found that while contractors for two DLA contracts submitted the required ISRs, they did not submit the required SSRs. In one of these two instances, an agency official told us that the contracting officer was unaware of the need for the contractor to submit both an ISR and SSR, and did not inform the contractor of this requirement. For this contract, which was awarded in fiscal year 2017, the contractor submitted its first SSR in October 2019, after we inquired with DLA officials about the lack of SSR submissions. For the second of these two contracts, which also was awarded in fiscal year 2017, the contractor informed the agency that they had not submitted SSR reports in the past because they were unaware of this requirement, and did not submit an SSR until October 2019. Finally, for one other DLA contract, the only ISR we found in eSRS was submitted by the contractor in October 2019, after we inquired about the ISR and more than 2 years after the contract was awarded. This contractor submitted reports outside of eSRS for two of the four prior reporting periods. These reports did not have acceptance or sign off by the accepting DLA official. In addition, while a DCMA staff member told us that the contractor did not submit its September 2017 and March 2018 ISR reports, the staff member did not provide an explanation why these reports were not submitted. Additionally, officials from all four agencies told us they conduct some type of periodic review related to oversight of subcontracting plans, which can include determining compliance with the subcontracting plan and related reporting requirements. In some of these reviews, the agencies had similar findings to ours. For example, NASA: According to an agency official, NASA’s Office of Small Business Programs conducts procurement management reviews of subcontracting plans every 2–3 years. The official told us that these reviews serve to monitor whether (1) prime contractors submitted the required ISRs and (2) contracting officers assessed the subcontracting plans and reviewed the ISRs, among other things. The results of a review conducted in May 2017 identified missing ISRs and reports that were accepted with incomplete information. Navy: According to a Navy official, the Navy Office of Small Business Programs conducts Procurement Performance Management Assessment Program reviews. The official stated that these reviews are conducted every 3 years at each of Navy’s command centers that conduct buying activities. If a command center receives an unsatisfactory or marginal rating, then the Deputy Assistant Secretary of the Navy for Acquisition and Procurement will perform follow-up reviews every 6–12 months until the issues are addressed. As part of the review process, Navy reviews subcontracting plans and data in eSRS to determine how subcontracting plans are monitored and evaluated. A review conducted in June 2018 concluded that monitoring of prime contractor’s subcontract reporting and compliance was inadequate. GSA: According to agency officials, GSA’s Office of Small Business Utilization, in conjunction with GSA’s Procurement Management Review team, conducts Small Business Compliance Reviews. Annually, the agency selects 4–6 regions from which to select a sample of contracts to review for both pre-award and post-award compliance. According to agency officials, these reviews are designed to help determine if subcontracting goals were met, among other subcontracting-related requirements. A review GSA conducted in March 2019 for one contract noted that the subcontracting plan could not be located in the contract file and that there was a lack of post- award subcontracting plan oversight, including contractor reports on subcontracting activities. DLA: According to a DLA official, various DLA offices, including the DOD Office of Small Business Programs, monitor eSRS regularly to ensure contracting officers are reviewing and processing contractor submissions through the system. The official stated that these reviews happen at various times throughout the year. For example, the Small Business Director at DLA Distribution—an organization within DLA— checks eSRS on a biweekly basis and DLA Aviation—another organization within DLA—conducts semi-annual reviews of eSRS. The DOD OIG had similar findings regarding oversight of contractor compliance with subcontracting plan requirements, including contractor reporting requirements. For example, in 2018 the DOD OIG reported that contracting officers at two Army contracting commands did not monitor prime contractors’ compliance with subcontracting plans. The DOD OIG made three recommendations to address the findings, which have been implemented according to the DOD OIG. As previously mentioned, contracting officers are responsible for a large number of processes and reviews, which may result in a specific process or task being missed. According to officials from Navy and NASA, other factors also contributed to the existence of limited documentation for certain post-award requirements for the contracts we reviewed. For example, the agency officials stated that contracting officers focus more on the award process than on contract administration and fail to properly consider the requirement that subcontracting plans become a material part of the contract on award, resulting in a lack of due diligence after the award. Officials from NASA and Navy also cited eSRS not providing notifications to contracting officers and contractors when reports are not submitted, among other things, as a contributing factor in missing ISR reports. Additionally, according to NASA officials, eSRS does not generate a list of prime contractors who are delinquent in submitting their SSRs. For the 26 contracts we reviewed with a subcontracting plan, contracting officers accepted several report submissions containing incorrect information about subcontracting goals. According to FAR § 19.705-6(j), after a contract containing a subcontracting plan is awarded, the contracting officer must reject a contractor’s subcontracting report submission if it is not properly completed—for example, if it has errors, omissions, or incomplete data. In fulfilling their responsibilities related to FAR § 19.705-6(j), contracting officers can identify omissions that a contractor may need to address. For example, in reviews of ISRs for a $31.8 million Navy contract awarded in fiscal year 2017, the contracting officer noted concerns about the contractor not meeting its socioeconomic goals and asked the contractor to provide an explanation for why the goal was not being met. The contracting officer rejected the September 2018 ISR and later rejected the September 2019 ISR twice because the contractor either did not provide an explanation for not meeting certain socioeconomic goals or failed to describe good-faith efforts to do so. The contractor submitted a revised ISR in December 2019, which included a description of its good-faith efforts to meet the socioeconomic goals. Upon review, the contracting officer accepted the submission stating that it seemed clear from the information provided that the contractor put forth a good-faith effort to meet the goals. However, for the 21 contracts we reviewed in total that required contractor ISR submissions (which provide information on approved subcontracting goals and achievements towards them), we found that for nine contracts, the contracting officers accepted one or more submissions with errors or unexplained conflicting information related to subcontracting plan goals (see table 3). Specifically, all nine contracts lacked explanations of the discrepancies in the ISR or other documentation we reviewed. We discuss the nine contracts in more detail below: NASA: Contracting officers accepted multiple ISRs with errors or unexplained conflicting information for three NASA contracts. In one of the three contracts, awarded in fiscal year 2017 for $3.8 million, the contractor combined small business subcontracting goals (listed as whole dollars and percent of total subcontracting dollars) from two different subcontracting plans associated with the contract into one ISR. However, the dollar amount reported in the ISR as the subcontracting goal—about $177,000—reflected the small business goal from only one of the subcontracting plans, rather than the two subcontracting plans, which would have been a total of about $309,000. As a result, the actual percentage of subcontracting to small businesses of total subcontracting and of the total amount of the contract value was incorrect. In the second contract, awarded in 2016 for $4.6 million with a planned small business subcontracting total of about $2 million, the contractor listed an overall small business subcontracting goal different from the approved subcontracting goal in three ISRs, and there was no documentation explaining the difference. For the third contract, awarded in fiscal year 2016 for $45.2 million with a planned small business subcontracting goal of 10 percent of total subcontracting dollars, the contractor listed this goal incorrectly in two ISRs. According to a NASA official, at the time of our review, the contracting officer was working with the contractor to correct the error. DLA: For one contract awarded in 2017 for $34.1 million with a planned subcontracting total of about $11 million, a DLA contracting officer accepted a September 2019 ISR that listed the small business goal at 90 percent of the total subcontracting dollars for the contract instead of the 87.4 percent (base) or 87.6 percent (option years) in the contract addendum. The actual cumulative subcontracting percentage reported in the ISR was 88.1 percent, which met the goal in the addendum, but not the 90 percent goal in the accepted September 2019 ISR. We could not identify any information in the ISR explaining the conflicting information. Additionally, when calculating the amount of cumulative dollars awarded to small business concerns, the contractor appeared to have excluded about $54,000 in subcontracting, which was included in a separate line item in the ISR for women-owned small business concerns. As a result, we were unable to determine whether this contractor had been meeting its small business goal. For a second contract also awarded in 2017 for $74.9 million with a planned subcontracting total of about $23 million, the contractor reported the approved small business goal of 96 percent of total subcontracting dollars in the March 2018 and September 2018 reports. However, in March 2019 and September 2019 ISR submissions for this contract, the contractor reported a small business goal of 98.5 percent and 74.8 percent, respectively. We found no documentation explaining why the contractor reported goals in the 2019 ISRs that were different from the approved 96 percent goal. Navy: For one Navy contract, which was awarded for $13.5 million in fiscal year 2018 with a planned subcontracting total of $2.7 million, the contracting officer notified the contractor in the September 2018 and March 2019 ISRs that the small disadvantaged business goal of 0 percent of total subcontracting dollars in these submissions did not match the 25 percent goal in the approved subcontracting plan. The contractor corrected the error and the contracting officer accepted the revised reports. In the September 2019 submission, the contractor once again reported that particular goal as 0 percent, but the contracting officer did not note the recurring error in this submission. For another contract, awarded for $16.6 million in fiscal year 2016 with a planned subcontracting total of about $5.9 million, the March 2016 ISR listed a small business goal of 693 percent (the goal in the approved subcontracting plan was 69.3 percent) of total subcontracting dollars. The contracting officer did not address the incorrect percentage. Moreover, in the September 2016 submission, the goal was reduced to 61.8 percent, which was less than the goal in the approved subcontracting plan. There was no explanation for the discrepancies in either submission. GSA: For one GSA Public Building Service contract, which was awarded in fiscal year 2018 for $7.5 million, we found discrepancies between the goals listed in multiple accepted ISRs and the approved subcontracting plan. This contract involved janitorial services performed at two locations. Each location had a different approved small business goal—96 percent and 87 percent of total subcontracting dollars. However, the contractor reported only one small business goal in the three ISRs submitted for September 2018, March 2019, and September 2019, and this reported goal varied from 89 to 97 percent in the three ISRs. According to a GSA official, the contractor submitted one ISR in each reporting period to convey the combined progress toward meeting its subcontracting goals for both locations, but the small business goal the contractor reported in each ISR did not accurately reflect the combined goals for both locations. The GSA official told us the combined goal the contractor should have reported for this contract was about 91 percent. According to the GSA official, these submissions contained data entry errors by the contractor, perhaps due to the contractor not knowing how to properly report its subcontracting data. For one GSA Federal Acquisition Service contract awarded in fiscal year 2017 for $3.6 million, we found a discrepancy between the small business goal reported in multiple ISR submissions—5 percent of total subcontracting dollars—and the 25 percent goal of total subcontracting dollars in the approved subcontracting plan, and we notified the agency of the discrepancy. However, none of these submissions included an explanation for the discrepancy and the agency’s reviewing official accepted the submissions without addressing the conflicting information. We also found one instance involving unclear oversight responsibilities among the 26 contracts we reviewed. We were unable to determine which agency actively monitored one DLA contract, which was awarded in fiscal year 2017 for $23.3 million. According to DLA staff, DCMA is responsible for monitoring, evaluating, and documenting performance of the contractor for the associated small business subcontracting plan. However, DCMA officials provided responses that DLA is the entity that should be conducting oversight of the subcontracting plan. If oversight responsibility of contracts involving two agencies is not apparent, it is unlikely that the contractor’s compliance with their subcontracting plans is being properly monitored. According to agency officials, several factors contributed to contracting officers accepting subcontracting reports with erroneous information. For example, as previously stated, agency officials told us that contracting officers’ large workload and focus on the award process (rather than on contract administration) can contribute to not always considering subcontracting plans as material parts of contracts and, thus, not conducting related due diligence after the contract award. GSA officials also noted that contracting officers may not have read or understood FAR requirements for oversight of contracts. For 16 of the 26 contracts we reviewed with a subcontracting plan, contractors reported that they met their small business subcontracting goal or were meeting the goal in situations where the contract had not yet ended. For the remaining 10 contracts, three ended without the contractor meeting the small business goal, five were not meeting the small business goal but the contract had not yet ended, and two had limited documentation available and we were unable to determine whether the goal was met. For the three contracts that ended without the contractor meeting the small business goal, two contracts had documentation that included a rationale for why the goal was not met. For one NASA contract, the contracting officer documented in a memorandum that a decision was made that there was no longer any subcontracting possibilities. The other instance involved a GSA Federal Acquisition Service contract, in which the assessing official documented in the final Compliance Performance Assessment Report that the low goal achievement was due to the nature of the automotive manufacturing industry. We could not identify a rationale for one Navy contract for why the small business subcontracting goal was not met and the agency could not provide documentation explaining why the goal was not met. The FAR requires contracting officers to assess liquidated damages against a contractor if a contracting officer determined the contractor failed to make a good-faith effort to comply with the subcontracting plan. However, a contractor’s failure to meet its subcontracting plan goals does not, in and of itself, constitute a failure to make a good-faith effort. Of the three contracts we reviewed that did not meet their small business subcontracting goal, we found no instances in which a contracting officer pursued liquidated damages or other actions against a contractor. As previously mentioned, two of these three contracts had a documented rationale for not meeting the small business subcontracting goal. Agency officials told us that contracting officers rely on Compliance Performance Assessment Reports or other performance assessment measures to rate a contractor’s performance relative to their subcontracting goals. Officials from three of the four agencies also told us a contractor’s past performance could affect their future ability to obtain government contracts, which can incentivize contractors to take steps to meet their subcontracting goals. SBA provides training to federal agencies’ contracting officers and contractors to assist in complying with small business subcontracting plan requirements. As part of its Small Business Subcontracting Program, SBA conducts certain reviews to assess overall effectiveness of small business subcontracting, including compliance reviews that are designed to assess contractor compliance with small business subcontracting plans. However, SBA could only provide limited documentation on compliance reviews it conducted from fiscal years 2016 through 2018, and limited information on compliance reviews conducted in fiscal year 2019. SBA provides training for contracting officers yearly to assist them in their reviews of subcontracting plans, including training related to pre-and post- award subcontracting activities for contracting officers. Beginning in 2017, SBA made available annual training for contracting officers to assist them in reviewing subcontracting plans. SBA also provides training to contractors, which provides them with information on meeting subcontracting plan requirements. If a prime contractor receives a less than satisfactory rating on a compliance review, the prime contractor must attend a mandatory training to address the issues found in the initial rating. According to SBA officials, the agency also has been developing new electronic-based training to coincide with new compliance review processes. According to the officials, the training is intended to educate prime business contractors with a subcontracting plan and federal agencies awarding contracts with a subcontracting plan on how to comply with post-award subcontract program requirements. SBA plans to make this training available in July 2020 in an electronic format that will provide information and require the participant to answer a series of questions to ensure they comprehend and retain the information. In addition to providing training, SBA’s CMRs conduct reviews related to SBA’s Small Business Subcontracting Program. In particular, SBA’s Standard Operating Procedure (SOP) 60 03 6, which was effective from December 4, 2006 through July 17, 2018, identified CMR responsibilities and included guidance for conducting reviews related to the Small Business Subcontracting Program. According to this SOP, CMRs were to conduct different types of reviews: In Performance Reviews (also referred to as desk reviews), CMRs were to review ISRs and SSRs that contractors submitted to determine which large business contractors in their portfolios they should visit, and what type of compliance review would be most effective. In Small Business Program Compliance Reviews (compliance reviews), CMRs were to evaluate a contractor’s compliance with subcontracting program procedures and goals in a contractor’s small business subcontracting plan. CMRs also were to conduct follow-up compliance reviews on areas found deficient during a compliance review or previous follow-up review. SOP 60 03 6 also described some orientation or outreach activities as reviews. In Subcontracting Orientation and Assistance Reviews, CMRs were to visit a large business contractor’s facility or telephone the contractor to introduce them to the Small Business Subcontracting Program and provide an overview of the roles and responsibilities of a prime contractor. According to SBA, the agency conducted 417 of these reviews from fiscal years 2016–2018. According to SBA, the agency’s CMRs conducted hundreds of various reviews in fiscal years 2016 through 2018, and a total of 118 compliance reviews specifically during that period (see table 4). SBA staff said SBA also conducts surveillance reviews to evaluate the overall effectiveness of an agency procurement center’s small business program by reviewing contract files and procedures. According to SBA documentation, these reviews allow SBA to recommend changes to improve small business participation at procurement centers. A surveillance review also examines the procurement center’s subcontracting program. SBA staff examine subcontracting files to determine if procurement center staff routinely perform subcontracting plan reviews, route the subcontracting plans to the PCR for review during the contract award process, incorporate approved subcontracting plans into contracts, and ensure that prime contractors submit the subcontracting plan ISRs into eSRS. For example, in a 2019 surveillance review (for which we obtained a copy) SBA found the center that conducted the procurements did not have a subcontracting plan in the file for two contracts and the subcontracting plan was not sent to the appropriate SBA Area Director for four contracts. In July 2018, SBA issued a new SOP entitled Subcontracting Assistance Program Post Award, which revised SBA’s compliance review process. According to SBA officials and a high-level outline SBA provided, SBA intends to have the following three phases for the new review processes that will implement the new SOP: 1. Subcontract Reporting Compliance – In this phase, CMRs are to review and rate a prime contractor’s compliance with subcontracting reporting requirements (that is, the contractor’s ISR and SSR reporting requirements). According to SBA officials, SBA also intends to inform contract awarding and administering agencies of their findings. 2. Subcontracting Plan Goal Attainment Compliance – In this phase, CMRs are to review whether a prime contractor has met or is on track to meet the goals listed in the subcontracting plan. 3. Subcontract Regulation Compliance – In this phase, CMRs are to review the prime contractor’s actions in adhering to all the elements in the subcontracting plan and meeting subcontracting plan goals, among other related actions. According to SBA officials, the new compliance review process is intended to standardize compliance reviews based on the new SOP. SBA developed a broad outline of the three-phase compliance review process, and to implement this process, developed a CMR portfolio tracking document, in the form of a spreadsheet, and a draft compliance review guidance document, both of which SBA is currently using for the first phase of the process. However, SBA officials told us they could not provide detailed procedures for implementing the second and third phases and they continue to refine the compliance review spreadsheet in conjunction with the compliance review guidance. As of mid-March 2020, they stated that they intend to complete phase 2 guidance by July 30, 2020, and phase 3 guidance by October 30, 2020. SBA could not provide us with requested information and almost no documentation on the compliance reviews its CMRs conducted in fiscal years 2016–2018. SBA could not provide basic information such as the list of contractors reviewed, the specific type of compliance reviews (such as reviews conducted individually or conducted jointly with another agency), which agencies may have assisted in the reviews (in the case of any joint reviews), and contractor ratings resulting from the reviews. SBA could only provide one CMR compliance review and two follow-up compliance reviews for this time frame, and all three were conducted in fiscal year 2017. The one CMR compliance review SBA provided included general observations from the review, specific findings, follow-up actions required, best practices for the contractor, and the rating provided to the contractor. The follow-up compliance reviews from fiscal year 2017 identified steps that contractors took to address deficiencies found in the initial compliance review and steps to enhance their subcontracting program. According to SBA officials, the agency’s CMRs conducted 680 compliance reviews in fiscal year 2019 and SBA was able to provide some documentation related to these reviews. To conduct these reviews, SBA officials explained that they selected about 4,000 prime contracts from FPDS-NG with individual subcontracting plans that ended in fiscal year 2019 or later. From these approximately 4,000 contracts, SBA officials told us that CMRs randomly selected 680 for review during fiscal year 2019. The CMRs assessed the selected sample of contracts against the first phase of the new compliance review process—the extent to which contractors complied with their reporting requirements. In our review of the documentation SBA provided, we could not clearly identify how many reviews they conducted. For example, the summary information from the reviews was not documented or maintained in a single document, but was in multiple spreadsheets with some inconsistencies, making it difficult to determine how reviews were counted. Additionally, one spreadsheet contained a summary tab for many contracts, but a count of the unique contracts did not add up to 680. Other spreadsheets did not have a summary tab, and contained information on the reviewed contracts in tabs organized by contractor. According to its latest SOP, SBA conducts compliance reviews to determine whether prime contractors that are not small businesses complied with their post-award subcontracting responsibilities outlined in the subcontracting plan to ensure small business subcontracts are being properly awarded and reported. However, based on our review of the limited documentation provided, SBA lacks specific guidance in its SOP on how CMRs should maintain information for compliance reviews they conduct. SBA has draft guidance on the new compliance review process, including some specific information regarding what CMRs are to record as part of the compliance review. However, SBA does not have clearly documented and maintained records on the first phase of these compliance reviews. Requirements for small business subcontracting plans in certain contracts enhance opportunities for small businesses to participate in federal contracting. However, weaknesses in selected agencies’ oversight of subcontracting plans—such as not following all procedures and not reviewing contractor submissions for errors or omissions—can reduce those opportunities and limit agencies’ knowledge about the extent to which contractors fulfill obligations to small businesses. The frequency with which issues arose in our sample suggests agencies can do more to improve oversight. For contracts we reviewed which used checklists or memorandums to document the PCR review process, we found that those contracts generally demonstrated compliance with the requirement for the opportunity for a PCR review. Taking steps to ensure that contracting officers provide PCRs the opportunity to review contracts with subcontracting plans would help agencies identify subcontracting opportunities and benefit from suggestions for increasing small business participation. In turn, such efforts could help agencies achieve their small business subcontracting goals. Similarly, improved monitoring of submitted contractor reports on subcontracting activities would identify errors in the submissions and increase agencies’ ability to assess contractor performance. Without complete and accurate information on a contractor’s subcontracting goals, agencies cannot adequately assess a contractor’s performance in meeting its subcontracting plan responsibilities. Given the many responsibilities of contracting officers, steps to ensure that contractor report submissions on meeting subcontracting goals are accurate would assist agencies’ oversight efforts. SBA also has opportunities to significantly enhance oversight related to its subcontracting program. It lacks documentation for almost all compliance reviews conducted in three of the four fiscal years from 2016 through 2019, has not fully implemented revisions to the compliance review process, and has not yet developed procedures for ensuring clear and consistent records of all compliance reviews are documented and maintained. By having clear and consistent documentation for compliance reviews and maintaining those records, SBA would better position itself to track contractor compliance for contracts it reviews and would be able to use this information to inform subsequent reviews. Additionally, contracting agencies would be able to leverage the information from SBA for their own reviews of contractor performance and subcontracting plans. We are making a total of 10 recommendations to five agencies (three to DLA, one to GSA, two to NASA, three to Navy, and one to SBA): The Director of DLA should include a step for the opportunity for PCR review of the proposed contract and subcontracting plan in agency procedures and memorandums, and develop a mechanism for documenting whether the opportunity for PCR review was provided. (Recommendation 1) The Secretary of the Navy should include a step for the opportunity for PCR review of the proposed contract and subcontracting plan in agency procedures and memorandums, and develop a mechanism for documenting whether the opportunity for PCR review was provided. (Recommendation 2) The Director of DLA should take steps to fulfill the requirement that contracting officers ensure that subcontracting reports are submitted by contractors in a timely manner. For example, the agency could require contracting officers to verify that prior reports were submitted when reviewing current submissions. (Recommendation 3) The NASA Administrator should take steps to fulfill the requirement that contracting officers ensure that subcontracting reports are submitted by contractors in a timely manner. For example, the agency could require contracting officers to verify that prior reports were submitted when reviewing current submissions. (Recommendation 4) The Secretary of the Navy should take steps to fulfill the requirement that contracting officers ensure that subcontracting reports are submitted by contractors in a timely manner. For example, the agency could require contracting officers to verify that prior reports were submitted when reviewing current submissions. (Recommendation 5) The Director of DLA should take steps to ensure contracting officers compare subcontracting goals in contractor report submissions to goals in the approved subcontracting plan and address any discrepancies. (Recommendation 6) The Administrator of the GSA should take steps to ensure contracting officers compare subcontracting goals in contractor report submissions to goals in the approved subcontracting plan and address any discrepancies. (Recommendation 7) The NASA Administrator should take steps to ensure contracting officers compare subcontracting goals in contractor report submissions to goals in the approved subcontracting plan and address any discrepancies. (Recommendation 8) The Secretary of the Navy should take steps to ensure contracting officers compare subcontracting goals in contractor report submissions to goals in the approved subcontracting plan and address any discrepancies. (Recommendation 9) The SBA Administrator should ensure Commercial Market Representatives clearly and consistently document compliance reviews and maintain these records. (Recommendation 10) We provided a draft of this report to DOD, GSA, NASA, and SBA for review and comment. DOD provided a written response, reproduced in appendix II, in which it concurred with our recommendations. DOD described steps that DLA and Navy intend to take to address the recommendations, including actions to remind contracting officers or to provide additional guidance related to giving the PCR an opportunity to review the proposed contract and subcontracting plan. DOD also described actions that DLA and Navy intend to take to remind contracting officers of the requirement to ensure that subcontracting reports are submitted in a timely manner and to remind contracting officers to compare subcontracting goals in contractor report submissions to goals in the approved subcontracting plan and address any discrepancies. GSA provided a written response, reproduced in appendix III, in which it concurred with our recommendation. NASA provided a written response, reproduced in appendix IV, in which it concurred with our recommendations. NASA described steps it intends to take, such as requiring procurement offices to monitor contracting officer reviews of contractor report submissions and comparisons of subcontracting goals for consistency with the subcontracting plan. NASA also provided technical comments on the draft report that we incorporated where appropriate. SBA provided a written response, reproduced in appendix V, in which the agency partially concurred with our recommendation. SBA also asked us to consider rewording a few statements that it considered to have appeared for the first time in the draft report. In the draft report we sent to SBA, we provided additional information about how we could not clearly identify how many reviews the CMRs conducted. SBA stated in its written response that it has comprehensive documents and records for fiscal year 2019 compliance reviews and while its CMRs maintain a separate workbook of spreadsheets for reviews they conduct, the agency maintains a summary document that combines the compliance reviews performed collectively by its CMRs. During our audit and as part of its written response to our draft report, SBA did not provide a summary document that showed all reviews conducted by its CMRs for fiscal year 2019. SBA also acknowledged in its written response that it could not provide requested documentation for compliance reviews conducted during fiscal years 2016 through 2018. SBA stated it has developed detailed procedures for maintaining consistent records for compliance reviews and that while CMRs are using these procedures currently, the agency intends to finalize the procedures on May 29, 2020 to ensure that SBA continues to fully document its compliance reviews. Based on the documentation we reviewed and analyzed during our audit, we maintain that SBA does not have clearly documented and maintained records of compliance reviews and should clearly and consistently document its compliance reviews and maintain these records. We will review any additional documentation of records of compliance reviews when SBA provides it in response to this recommendation. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to appropriate congressional committees and members, the Secretary of DOD, the Administrator of GSA, the Administrator of NASA, the Administrator of SBA, and other interested parties. This report will also be available at no charge on our website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. Our objectives in this report were to examine (1) the extent to which select agencies conduct oversight related to small business subcontracting plans in the pre-award phase of the federal contracting process; (2) the extent to which select agencies conduct oversight of such subcontracting plans in the post-award phase; and (3) steps the Small Business Administration (SBA) has taken to encourage agencies to conduct oversight activities related to small business subcontracting plans. To address the first two objectives, we reviewed the Federal Acquisition Regulation (FAR) and agency-specific procedures. We also reviewed requirements for contractor submissions on subcontracting activity related to subcontracting plans, and corresponding agency oversight requirements for the submissions. We reviewed documentation on agency training for contracting officers related to subcontracting plans and requirements. We judgmentally selected two military agencies—the Defense Logistics Agency (DLA) and the Department of the Navy (Navy)—and two civilian agencies—the General Services Administration (GSA) and the National Aeronautics and Space Administration (NASA)— to review based on our analysis of Federal Procurement Data System- Next Generation (FPDS-NG) data and other factors. More specifically, we selected the agencies because they (1) included a mix of military and civilian agencies, (2) had relatively high dollar amounts of federal contracts awarded in fiscal years 2016–2018, and (3) included a range of performance related to subcontracting based on SBA’s annual procurement scorecard. We also reviewed documentation for a nongeneralizable sample of 32 contracts—eight per agency—awarded in fiscal years 2016–2018 across the four agencies. We randomly selected these 32 contracts from a set of contracts that met several criteria. Specifically, the criteria were contracts with dollar amounts above $1.5 million, that had a mix of subcontracting plans (individual, commercial, and comprehensive) or reasons for not including subcontracting plans in a contract (such as no subcontracting possibilities for the contract or the contract not requiring a subcontracting plan), and a mix of their current status at the time of our selection (completed or active). We selected contracts as follows: We first randomly selected six contracts per agency (total of 24) that had a small business subcontracting plan at the time of award. To do this, we used a random number generator for the universe of contracts meeting the above criteria and selected contracts in the order of the random number generator, but skipped a contract if it was too similar to already-selected contracts (for example, same type of subcontracting plan or similar dollar amount). We then selected another set of contracts—two per agency (total of eight)—that seemed to meet criteria for requiring small business subcontracting plans, such as exceeding the dollar threshold, but were coded in FPDS-NG as not having a plan in place. We also obtained reports on contractor submissions on small business subcontracting activity, where applicable, and agency reviews of the submissions from the Electronic Subcontracting Reporting System (eSRS). Specifically, we searched eSRS for any contractor-submitted individual subcontracting reports (ISR) or summary subcontract reports (SSR), where applicable, for each contract with a subcontracting plan and reviewed the reports along with agency contracting officer comments, approvals, or rejections related to the reports. If we were unable to locate any ISRs or SSRs in eSRS, we asked the procuring agency to provide copies of the reports. We also requested agency documentation for any actions contracting officers took, if applicable, for each contract where the contractor had not met the small business subcontracting goal. We also interviewed officials from each agency about their efforts related to oversight of small business subcontracting plans and these contractor submissions. We assessed the reliability of FPDS-NG data by reviewing available documentation and prior GAO data reliability assessments and by electronically testing for missing data, outliers, and inconsistent coding. We found the data to be reliable for the purposes of selecting agencies and contracts to review. We assessed the reliability of eSRS by reviewing available documentation and verifying information with agencies. We found the information in eSRS to be reliable for purposes of assessing the extent to which agencies conduct oversight related to contractor submission reports in the system. To address the third objective, we reviewed documentation on several types of SBA reviews, including compliance reviews, related to contractor compliance with and agencies’ oversight of subcontracting plans. Specifically, we reviewed documentation on reviews SBA conducted related to its subcontracting program during fiscal years 2016–2019. We also reviewed SBA’s standard operating procedures for the subcontracting program, documentation on processes implementing the new procedures, and documentation on SBA training programs for the small business subcontracting program. We interviewed SBA officials regarding steps the agency takes to encourage agency oversight of subcontracting plans. For all the objectives, we reviewed relevant federal laws and regulations and reviewed previous GAO reports and reports from the Department of Defense Office of Inspector General (DOD OIG). We also interviewed officials from the DOD OIG to obtain an understanding of their work on DOD’s oversight of subcontracting plans at selected DOD components and command centers. We conducted this performance audit from January 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Andrew Pauline (Assistant Director), Tarek Mahmassani (Analyst in Charge), Suellen Foth, Jonathan Harmatz, Julia Kennon, Jill Lacey, Yola Lewis, John McGrail, Marc Molino, and Barbara Roesmann made key contributions to this report.", "summary": "Certain federal contracts must have a small business subcontracting plan if subcontracting opportunities exist. But recent Department of Defense Inspector General reports raised concerns about agency oversight of subcontracting requirements. GAO was asked to review oversight of subcontracting plans. Among its objectives, this report discusses (1) the extent to which selected agencies (DLA, GSA, NASA, and Navy) oversee small business subcontracting plans, and (2) how SBA encourages agency compliance with subcontracting plan requirements. GAO reviewed data and documentation for a non-generalizable sample of 32 federal contracts (including 26 contracts with a subcontracting plan) at four agencies, selected to include contracts over $1.5 million at both civilian and military agencies awarded in fiscal years 2016–2018. GAO also reviewed the Federal Acquisition Regulation, SBA and selected agency documentation, and interviewed agency officials. GAO found selected agencies did not consistently follow all required procedures for oversight of small business subcontracting plans, both before and after contracts were awarded. GAO reviewed 26 contracts with a subcontracting plan at four agencies—Defense Logistics Agency (DLA), General Services Administration (GSA), National Aeronautics and Space Administration (NASA), and the Department of the Navy (Navy). For about half of the 26 contracts, agencies could not demonstrate that procedures for Procurement Center Representative (PCR) reviews were followed. These representatives may review small business subcontracting plans and provide recommendations for improving small business participation. When an agency is awarding a contract that includes a subcontracting plan, contracting officers are required to notify these representatives of the opportunity to review the proposed contract. Without taking steps to ensure these opportunities are provided, agencies may not receive and benefit from suggestions for increasing small business participation. For 14 of the 26 contracts, contracting officers did not ensure contractors submitted required subcontracting reports. After a contract is awarded, contracting officers must review reports contractors submit that describe their progress towards meeting approved small business subcontracting goals. In some cases, contracting officers accepted reports with subcontracting goals different from those in the approved subcontracting plans, with no documentation explaining the difference. Without complete and accurate information about a contractor's subcontracting goals, an agency cannot adequately assess a contractor's performance in meeting its subcontracting plan responsibilities. The Small Business Administration (SBA) encourages agency compliance with small business subcontracting plan requirements by providing training to contracting officers and contractors, and by conducting reviews. For instance, SBA Commercial Market Representatives conduct compliance reviews to evaluate a large prime contractor's compliance with subcontracting program procedures and goal achievement. However, SBA could not provide documentation or information on almost all compliance reviews conducted in fiscal years 2016–2018. SBA has developed new procedures for conducting compliance reviews, but as of mid-March 2020, had yet to fully implement them. SBA has conducted fiscal year 2019 compliance reviews that reflect a first phase of their new procedures. SBA has draft guidance on the new compliance review process, including some specific information regarding what Commercial Market Representatives are to record as part of the compliance review. SBA has begun to conduct compliance reviews in accordance with the guidance, but does not have clearly documented and maintained records for the first phase of these reviews. Without consistent, clear documentation and records that will be maintained going forward, SBA's ability to track contractor compliance and agency oversight efforts will be limited. GAO is making 10 recommendations for ensuring procedures for PCR reviews are followed, contractor subcontracting reports are monitored and reviewed for accuracy, and SBA compliance reviews are clearly documented and maintained. DLA, GSA, NASA, and Navy concurred with our recommendations. SBA partially concurred with our recommendation. GAO maintains that its recommendation is warranted.", "document_type": "gao"}
{"report": "VA, through VHA, operates the nation’s largest integrated health care system. At the local level, VA has 172 VA medical centers that are organized into 18 Veterans Integrated Service Networks (VISN). At the national level, VHA’s central office includes approximately 75 national program offices as of October 2019, which perform a range of clinical or administrative functions. For example, some program offices are responsible for specific clinical areas, such as spinal cord injury or mental health care, and may develop policy for those areas. To support VA’s health care delivery system, VA’s intramural research program aims to improve veterans’ health by funding research on issues that affect veterans, developing effective treatments for veterans, and recruiting and retaining VA researchers. VA’s medical and prosthetic research appropriation—$722 million in fiscal year 2018—funds VA’s intramural research program. VA also uses funding from its other appropriation accounts—$544 million in fiscal year 2018—to support VA’s intramural research by paying some costs associated with this research, such as equipment maintenance. According to VA, more than 60 percent of VA researchers are also clinicians who provide direct patient care, which helps translate VA research into clinical practice. ORD manages VA’s intramural research program. Within ORD, there are four research and development services that are responsible for administering and supporting research; each research service has a specific focus, such as biomedical research and rehabilitation research. Each of the four research services is led by a director and has scientific program managers who are responsible for specific research portfolios (or topic areas) within their service. In addition to the four research services, ORD has a Cooperative Studies Program that is responsible for large-scale clinical trials and epidemiological studies within VA. (See Table 1.) All five of these ORD components support research by funding VA research projects. See appendix II for details on research funding and awards. Organizationally, ORD falls within the Office of Discovery, Education and Affiliates Networks, which was created in November 2018 to foster collaboration in addressing veterans’ health concerns. ORD funds VA intramural research in a number of ways, such as the following: Merit Review Program. This program supports VA research projects that are typically led by one VA researcher at one VA facility. ORD’s four research services administer this program and are responsible for soliciting, reviewing, selecting, and funding research proposals submitted by VA researchers. Researchers may submit proposals either in response to a request for applications on a specific topic (sometimes called targeted or focused requests) or to a general request for applications, for which researchers can propose projects on a wider range on topics. To be considered for funding, research proposals must be veteran-centric and meet other requirements. Each ORD research service typically evaluates Merit Review Program research proposals in two review cycles per year. Selected projects are funded for a set number of years and have a maximum budget— typically for four years with a maximum amount of $1.2 million. The Merit Review Program accounts for the majority of VA-funded research studies that ORD funds. Cooperative Studies Program. This program funds larger-scale, multi-site clinical trials and epidemiological research studies on diseases that affect veterans. VA researchers can submit proposals at any time during the year. ORD’s Chief Research and Development Officer and Cooperative Studies Program leadership evaluate proposals in two review cycles per year. The time frame and budget for selected studies varies, depending on features of specific studies. Career Development Program awards. The research services’ Career Development Program provides funding to support, train and mentor individuals early in their career as VA researchers, which can include funding for specific research projects. Funding to support VA research centers and entities. In addition to funding individual research projects and researchers, ORD funds research centers and entities that focus on specific research areas. For example, the rehabilitation research service provides “core funding” to support 12 research centers focusing on areas such as limb loss, spinal cord injury, vision loss, and auditory disorders. This type of funding is competitive, and VA researchers must recompete in 5-year cycles. Once research has been completed, the findings can inform additional research. For example, research findings from a study on tissue could be used to inform a study done on humans in a controlled clinical setting, such as a clinical trial, which could in turn inform research that tests the effectiveness of a particular intervention in a less-controlled community setting. This concept is referred to as the “research pipeline.” Research findings leading to broad changes in clinical practice that affect public health is considered the end of the pipeline. Within VA, research findings can be translated into clinical practice in a number of ways, such as by implementing a new diagnostic tool, changing the treatment protocol for a particular disease, adding a new prompt for providers in the electronic health record, or developing a new clinical policy or a clinical practice guideline. The specific outcome might vary depending on the study or type of research. For example, one body of VA research has confirmed the utility of an intimate partner violence screening tool for female veterans in primary care settings. These findings will be used in developing national guidelines for screening for, and responding to, intimate partner violence. In addition, VA research contributed to the development of a clinical practice guideline for the management of upper extremity amputation rehabilitation. This guideline is a tool to assist clinicians and health care professionals with their decision-making when caring for individuals with upper extremity amputation. ORD leadership sets VA’s national research priorities based on input from internal and external stakeholders and other factors. The directors of ORD’s four research services, in turn, set their own service-level research priorities based on the national priorities, veterans’ specific needs, and other considerations. Once these research priorities are set, ORD officials use a range of approaches to incorporate them when funding research, such as by funding collaborative research efforts focused on specific priorities. ORD officials said in funding research, they also consider other clinical and research needs that are not identified as priorities but are still important to veterans’ health, such as encouraging researchers to test new ideas. At the national level, ORD leadership sets VA’s overall research priorities based on input from internal and external stakeholders and other factors. ORD sets these priorities annually, and its priority-setting process involves discussions with stakeholders and reviews of relevant VA data, according to ORD officials. For example, ORD sets priorities using the following methods: Input from internal VA stakeholders. The directors of ORD’s research services provide input to the Chief Research and Development Officer on issues they see as priorities. Officials outside of ORD, such as the leadership of VA’s new Office of Discovery, Education, and Affiliated Networks, of which ORD is a part, and other VA leaders also provide input, according to ORD officials. Input from external stakeholders. ORD officials said they also consider input from the National Research Advisory Council, a 12- member federal advisory committee that provides advice to VA on its research and development efforts, such as recommending which topics to include among the agency’s research priorities. According to ORD officials, the office also obtains input on research priorities by meeting with veterans service organizations and by hearing from veterans through veterans’ engagement opportunities. In addition, Congress provides direction and input on topics that VA should study, such as through legislation or committee reports. Other factors. ORD officials said, for example, they set research priorities using VHA data on the prevalence of health conditions among veterans and Veterans Benefits Administration data on military deployment-related conditions. Based on stakeholder input and other factors, ORD established three types of national research priorities: strategic, cross-cutting clinical, and other priorities. (Figure 1 shows these research priorities for fiscal years 2019 and 2020.) As of October 2019, ORD officials told us they were determining what these priorities will be for fiscal year 2021. Strategic priorities are broad long-term priorities that focus on VA’s research capability, resources, and operations, rather than on specific clinical conditions, according to ORD officials. For example, one of VA’s strategic priorities is to “put VA data to work for veterans.” As part of this priority, VA aims to improve its ability to leverage the agency’s medical data to improve veterans’ care. ORD officials said they plan to revisit these priorities about every 5 years, though the specific initiatives that fall within each priority can change annually. As an example of how the initiatives within the strategic priorities can change, ORD officials said that given VA’s plans to implement a new electronic health records system, the National Research Advisory Council advised ORD in 2019 to focus on mitigating any unintended consequences of this transition on research, as part of the strategic priority on data. ORD officials said that as a result of this input, they are increasing the intensity and scope of their efforts pertaining to this transition. Cross-cutting clinical priorities, in contrast, focus on predominant clinical conditions seen in veterans and can change yearly, according to ORD officials. For example, one current cross-cutting clinical priority is PTSD. As part of this priority, VA supports research to better understand the underlying biology of PTSD, refine approaches for diagnosing this condition, and develop and test new treatments. ORD officials said they plan to add precision oncology as a new cross- cutting clinical research priority for fiscal year 2021, based on input from VA leadership. ORD officials also said they plan to broaden the Gulf War illness cross-cutting clinical priority to include the effects of military service-related toxic exposures, more generally. This planned change is based in part on veterans service organization and Gulf War veteran input. Other priorities are those that VA will focus on in the near term, based primarily on input from Congress, veterans service organizations, and other non-ORD stakeholders, according to ORD officials. For example, several of these priorities for fiscal year 2020— such as addressing the prosthetic needs of women veterans and exploring ways to use an “exoskeleton” for veterans who have experienced strokes or traumatic brain injury (TBI)—were identified by Congress as research needs. In addition to national priorities for research funding, ORD permits its four research services to set their own service-level research priorities, which are based in part on the national priorities. According to ORD officials, the directors of the four ORD research services—the biomedical laboratory, clinical science, health services, and rehabilitation research and development services—have latitude to set their own priorities, given their expertise in, and the particular focus of, their respective research areas. These directors told us they may consider a range of internal and external factors when setting priorities, including: Internal factors. The four research service directors told us they take into account VA’s national research priorities when determining their priorities. For example, the director of the biomedical laboratory service identified VA’s five cross-cutting clinical research priorities as priorities for this research service. Also helping shape research priorities are VA stakeholders, such as the Secretary of VA and scientific program managers—the ORD staff who are responsible for specific topic areas within their services. In addition, service directors told us they use VA data on veterans’ health conditions when setting research priorities. The leadership of ORD’s rehabilitation research service, for example, told us they review VA data on the top service- connected conditions for which veterans are receiving disability benefits, and take that factor into consideration, along with less prevalent conditions such as spinal cord injuries, that also have a significant impact on veterans’ function and independence when setting priorities. (See text box for examples of research projects on spinal cord injury, which is one of the rehabilitation research service’s priorities.) Exoskeleton-assisted walking Exoskeletons are motorized prostheses that are worn outside a person’s clothes and provide powered hip and knee motion, to help veterans with spinal cord injuries stand and walk. In this photograph, a research participant with a spinal cord injury uses an exoskeleton at the Bronx VA medical center. Department of Veterans Affairs (VA) Research on Spinal Cord Injuries VA provides care for about 27,000 veterans with spinal cord injuries. Veterans with spinal cord injuries may have secondary bone loss, muscle atrophy, and other conditions. They also have an increased prevalence of diabetes, heart disease, stroke, bowel and bladder incontinence, chronic pain, and reduced quality of life, according to VA. VA’s National Center for the Medical Consequences of Spinal Cord Injury, located at the Bronx VA medical center, is one of the VA’s rehabilitation research service’s research centers. The Center’s mission is to improve quality of life and increase longevity in individuals with spinal cord injuries by identifying and intervening to reduce and prevent the secondary consequences of spinal cord injuries. Examples of the Center’s VA-funded research include: Studying the safety and efficacy of exoskeleton-assisted walking in rehabilitation settings and in home and community environments, Developing and testing innovative approaches to improve bowel function, Studying the impact of low blood pressure and developing approaches to help individuals maintain normal blood pressure, Studying individuals’ difficulties regulating their body temperature, and developing interventions to address this problem, and Using magnetic and electrical stimulation to enhance arm and leg function. External factors. Congress can play a role in shaping the research services’ priorities. For example, the health services research service has identified research on policies and programs included in recently enacted legislation, including the VA MISSION Act of 2018 and the Comprehensive Addiction and Recovery Act of 2016, as a priority. Input from other federal partners, such as the National Institutes of Health (NIH) or the Department of Defense (DOD), also can influence the priorities of ORD’s research services. Officials with the rehabilitation research service, for example, said they meet with DOD officials about research efforts and that input from DOD on the health issues seen among active-duty service members can help them anticipate what health issues those service members might face when they transition to veteran status. They can then use that information when deciding which clinical areas to prioritize. In addition, one director said that input from veterans service organizations can shape research priorities, while another director obtains input from veterans through a VA veteran engagement group that provides information on the needs of veterans. ORD’s service-level research priorities cover a wide range of areas, such as service-connected conditions and conditions that veterans may experience as they age. As of October 2019, the services had each identified between 10 and 20 research priorities. (See the box below for examples.) Examples of Office of Research and Development’s Service-Level Research Priorities, as of October 2019 Traumatic brain injury (TBI) is a research priority for all the services. For example, the effect of prolonged opioid use on TBI outcomes is a priority for the rehabilitation research service. Post-traumatic stress disorder (PTSD) is a research priority for several services. For example, PTSD and the conditions that commonly co-occur with this condition is a priority for the clinical science research service. Pain is a research priority for all the services. For example, pain mechanisms and treatments, including alternatives to opioids, is a priority for the clinical science research service. Spinal cord injuries are a research priority for some services. For example, disability—including spinal cord injury and TBI— is a priority for the health services research service. Suicide prevention is a research priority for multiple services. The biomedical laboratory service, for example, has an emphasis on biological markers of suicide. Aging-related issues are a research priority for several services. For example, “long-term care, aging, and caregiver support” is a priority for the health services research service. Once priorities are set, ORD officials told us they use a range of approaches to incorporate those priorities when funding research projects, such as: encouraging researchers to study priority topics, considering priorities when deciding which projects to fund, and funding collaborative research efforts that are focused on specific priorities. In addition to the research priorities, ORD officials said they also consider other clinical and research needs when funding VA research, such as encouraging researchers to test new ideas in clinical areas that are not identified as priorities but are still important to veterans’ health. ORD officials’ approaches to incorporating priorities when funding research included the following examples: Encouraging researchers to study priority topics. ORD’s research services highlight their research priorities in their requests for research proposals. In some cases, they use targeted requests for research proposals solely on priority topics. In fiscal year 2019, ORD issued targeted requests for proposals linked to priorities such as suicide prevention, TBI, and the VA MISSION Act. In other cases, the research services highlight their research priorities in general requests for applications, which permit VA researchers to submit proposals on both priority and non-priority topics. For example, in 2019, the rehabilitation research service issued a general request for research proposals, stating that four research priorities—the prosthetic needs of women veterans, exoskeleton research related to patients with stroke and TBI, non-pharmacological interventions for chronic pain, and the effects of prolonged opioid use on long-term outcomes from TBI—were of particular interest for that funding cycle. (See text box for examples of VA research projects on priority topics.) Department of Veterans Affairs (VA) Research on Traumatic Brain Injury (TBI) and Stress Disorders Traumatic brain injury (TBI), a common injury among veterans of conflicts in Iraq and Afghanistan, can lead to a number of physical, cognitive, and emotional problems, such as memory and attention issues. These veterans may also experience post-traumatic stress disorder (PTSD), which can lead to anger, irritability, depression, substance abuse, and other symptoms, according to VA. VA’s Translational Research Center for TBI and Stress Disorders, located at the Jamaica Plain (Boston, Mass.) VA medical center, is one of VA’s rehabilitation research service’s research centers. The Center seeks to better understand the complex cognitive and emotional problems faced by these returning veterans, with the goal of developing better treatment options. The Center runs a longitudinal cohort study that collects imaging, genetic, and other data on returning veterans. Examples of the Center’s VA-funded research projects include: A study to assess the efficacy of the STEP-Home program, a 12-week workshop to help veterans who have served in Iraq or Afghanistan. The program aims to strengthen behavioral and emotional skills so that veterans are better equipped to rejoin their families and civilian communities. Studies to identify sub-types of PTSD, and to assess the long-term effects of PTSD and mild TBI. Research on the use of non-invasive brain stimulation to help patients with PTSD. Development of the Boston Assessment of Traumatic Brain Injury- Lifetime tool, a clinical interview to characterize head injuries and diagnose TBI throughout a patient’s lifespan. Considering priorities when deciding which projects to fund. Directors from all of ORD’s four research services stated that the scientific merit of research proposals—based on the proposals’ significance to veterans’ health, feasibility, and other criteria—is a key factor in funding decisions. Several directors said they may decide to fund a meritorious project that addresses one of their research priorities in lieu of another project that was ranked similarly or higher but does not address a priority. Some of the directors noted that this only applies to a small share of funded projects, but is part of how they align research projects with priorities. Funding collaborative research efforts. The biomedical laboratory service has funded field-based meetings to plan collaborative multi- site research programs to speed the development of treatments for service-related illnesses and injuries. The director of this research service said that in 2019, these research-planning meetings focused on ORD’s national research priorities, such as TBI, PTSD, and pain and opioids, among other topics. Also, starting in 2019, the health services research service is providing funding for its research centers to collaborate with other VA researchers on three of its priority areas: suicide prevention, opioid reduction and pain, and access to care. Officials from this research service said they also hold “State of the Art” conferences that can help VA make progress on priority areas. For instance, VA officials held a September 2019 conference on managing pain and addiction, specifically focusing on strategies to improve opioid safety. VA officials said this conference involved a wide range of VA staff and resulted in recommendations about research priorities, including areas where more research is needed. In addition to the research priorities, ORD officials said they consider other clinical and research needs when determining which health care research efforts to fund. Rehabilitation research service officials specifically noted that if they did not fund research in non-priority clinical areas, it would hinder their goal of encouraging researchers to test new ideas in other areas that are important to veterans’ health, which the officials say can lead to discoveries. The importance of innovation was echoed by other ORD officials, as well. Some ORD research service directors said that while new research needs emerge over time—as stakeholders highlight particular clinical needs, or VA leadership changes—it is important for VA research efforts to anticipate veterans’ longer-term needs and focus on more enduring issues, too. Officials from one research service said, for example, that they encourage researchers to focus on issues that will still be important in several years, such as women veterans’ care, because research can take years to yield results. In addition, some ORD research service directors said that although service-connected conditions are key parts of their research portfolios, they also work to address other conditions. Clinical science service officials said, for example, that they have a broad charge to support research into any disease or condition that affects veterans’ health. One of their priorities is researching diseases with a high health care burden among veterans, which may or may not be related to veterans’ military service. Rehabilitation service officials noted that their work focuses on veterans’ disabilities and impairments incurred through military service but is not limited to service-connected conditions. The officials said that because VA provides lifetime care to veterans, their research portfolio addresses events that cause impairment and disability throughout a veteran’s lifespan, including the aging process. For example, their research portfolio includes research on medical conditions that veterans may experience as they age, such as stroke, and chronic conditions like diabetes and kidney disease. As part of their efforts to consider multiple clinical and research needs, officials from the health services research service told us they are analyzing their overall research portfolio to determine where more or less research funding may be warranted. The officials explained that as part of their strategic planning efforts, they are identifying any areas they have “under-invested” in, and any areas that have received significant funding in the past but might no longer need that degree of investment. Among other things, they are considering the extent to which research on health conditions is already being done by other research organizations, such as NIH. They noted that while addressing chronic diseases is important to VA and its veteran population, it is possible that research on certain diseases is being covered by other research partners. In contrast, they said, there are areas where VA may have a unique ability to contribute to research because of its nationwide health care system or because it is ahead of the curve in health care trends, such as in telehealth and in integrating mental health care into primary care settings. Officials said their portfolio analysis could result in some “resetting” of research priorities and funding after the analysis is completed in 2020. Looking forward, ORD officials shared examples of approaches they are taking to boost the agency’s ability to address its research priorities. For example, ORD officials said there are a limited number of VA researchers working in certain priority areas, such as suicide prevention, which the officials said can hinder their efforts to fund new research projects. Among the efforts to boost the number of researchers working on priority areas, officials from one research service said they recently began incorporating their research priorities into the service’s Career Development Program funding awards. In addition, in 2019, ORD implemented a new method to spur and track ORD progress in addressing priorities. According to ORD leadership, as part of this method, ORD staff will identify the actions and resources needed to address specific priorities, and meet quarterly with the Chief Research and Development Officer to review their progress and identify next steps. VA has a variety of efforts to facilitate translating research findings into clinical practice to improve the care veterans receive. These efforts include those undertaken by ORD’s QUERI program, its health services research service, and VA’s Diffusion of Excellence Initiative, as discussed below. ORD’s QUERI provides a link between the research program, VA program offices, and VA providers. According to the QUERI director, QUERI serves as the center of VA’s efforts to translate research into clinical practice. QUERI’s overall mission is to improve veteran health by rapidly implementing research findings and interventions into clinical practice. QUERI is housed within ORD, but funded separately by non- research dollars. QUERI facilitates research implementation through activities such as the National Network of QUERI programs. According to the director of QUERI, these programs are partnered with VA national program offices, and they take various practices—often identified or developed through VA studies—and implement them at the regional or national level. For example, through its “Bridging the Care Continuum” QUERI investigators focus on improving the health of vulnerable veteran populations, such as homeless veterans, by implementing a co-occurring mental health and substance use treatment within multiple VA medical centers. (See text box for an example of implementation through a QUERI National Program.) In addition, QUERI funds resource centers with technical experts who can help promote and review best practices for implementation. Specifically, one resource center—the Center for Evaluation and Implementation Resources in Ann Arbor, Mich.—is available to VA researchers for consulting on strategies to translate research. Example of Quality Enhancement Research Initiative (QUERI) Research Translated into Clinical Practice: Telemedicine Outreach for Post-Traumatic Stress Disorder (PTSD) in Small Rural Community-Based Outpatient Clinics The goal of the Virtual Specialty Care QUERI National Program is to implement and evaluate promising clinical practices that incorporate technologies to improve access to specialty care for veterans in rural settings. One example of its efforts is the telemedicine program, based on VA-funded research demonstrating the effectiveness of using telemedicine outreach for veterans with PTSD. The Office of Rural Health and the Virtual Specialty Care QUERI partnered to implement this telemedicine program which provides evidence-based psychotherapy for veterans with PTSD via interactive video either from their homes or at community-based outpatient clinics, and connects veterans with care managers to coordinate their treatment. According to VA, as of June 2019 the telemedicine program is being implemented in six states and 1,073 “hard to reach” veterans have been engaged via the program. In 2019, QUERI published the “Implementation Roadmap,” a new resource—intended for a variety of users, including researchers, clinicians, and leadership—to advance research translation at VA and provide information on how to identify, implement, and sustain evidence- based practices to improve the quality of care for veterans. The Roadmap outlines the different stages of research implementation, specifically delineating when research is ready to be implemented into clinical practice. The QUERI director told us staff created the Roadmap as a teaching tool to provide guidance on how to implement research at VA and when to collaborate with QUERI. In addition, the director of QUERI told us the Roadmap demonstrates the cyclical nature of research and how implementation is part of a continuous scientific process, not an “end game.” QUERI officials said that throughout the process of implementation, new research questions might be generated, which QUERI can use to inform further investigation or follow up studies. ORD’s health services research service funds studies that focus on direct application of research in clinical practice. ORD’s health services research service supports research translation by funding studies focused on how interventions work in “real world” settings and on implementing VA research findings into clinical practice. For example, little is known about the quality of non-VA care for sex-specific services such as mammography, according to VA, despite increasing numbers of women veterans relying on such care due to limited availability within VA. One study funded by this research service looked at strategies for provision, coordination, and quality of oversight of non-VA care for women, and assessed perceptions and experiences with non-VA care among women veterans. Among other things, the study found VA sites providing mammography were more likely to notify women more quickly of abnormal results than non-VA sites, but non-VA sites were more likely to meet guidelines for timely follow-up. In addition to funding individual research studies through merit review, the health services research service funds 18 Centers of Innovation (COIN), each of which focus on one or more areas of research that address questions significant to clinical and operational partners. For example, officials from the dual-site COIN in Seattle and Denver, which focuses on veteran-centered and value-driven care, told us they are participating in a study co-funded by VA, NIH, and DOD evaluating non-pharmacological options to treat pain and co-occurring mental health conditions in veterans with chronic pain; the study will be overseen by VA’s Office of Patient-Centered Care and Cultural Transformation. According to the research service officials, the COINs are designed to bring researchers from multi-disciplinary research teams together to engage in research and establish partnerships that can affect VA policies, practices, and health care outcomes. (See text box for an example of research funded by the health services research service that has been translated into clinical practice.) Example of Research Translated into Clinical Practice across VA: Reducing Catheter-Associated Urinary Tract Infections The Center for Innovation (COIN) in Ann Arbor, Mich., partners with VA clinical, policy, and operations leaders to implement and evaluate ways to make health care safer, more effective, and affordable for veterans. For example, an investigator from this COIN, funded in part through a Career Development Award, conducted research on enhancing patient safety by reducing catheter-related infections. Then, in partnership with another VA researcher, this investigator conducted a study funded by the health services research service and created a “bundle” of activities to implement in VA hospitals throughout Michigan. This included removing catheters as soon as possible and increasing the use of recommended infection control practices. VA researchers assisted the National Center for Patient Safety in implementing the practices, and the success of the “bundle” resulted in its national implementation in more than 1,000 hospitals. VA reported that catheter-associated urinary tract infection rates decreased by 32 percent in participating general medical and surgical units. According to officials from the health services research service, in addition to funding studies and COINs, the service maintains four resource centers, which provide support to VA researchers in several areas, including data, health economics, and dissemination. For example, the Center for Information Dissemination and Education Resources circulates research findings through VA newsletters, cyberseminars, and publications and educates clinicians and researchers on sharing findings. In addition, the center coordinates meetings and conferences—such as the service’s joint national conference with QUERI—which provides an opportunity for VA researchers to present scientific findings and discuss the implementation of findings into practice. In 2017, the conference focused on accelerating the adoption and spread of practices and improving VA’s ability to utilize healthcare data to enhance care for veterans. Officials from the health services research service told us that, starting in 2019, they began implementing two new strategies to increase the impact of VA research on veterans’ health care. First, the service began a new effort to bring together and fund consortiums of researchers from multiple COINs each with a particular focus on implementing evidence-based practices in a given priority area. VA officials told us that as of October 2019, the health services research service established two consortiums of researchers to focus on suicide prevention and opioids and pain management. The service is planning to add two additional consortiums in 2020 to focus on access to care and telehealth and connected care. Second, in 2019 the research service provided additional funding opportunities for COINs to submit research proposals that include five- year goals for the impact of their research, such as VA policy changes or spreading the research to additional sites, and yearly milestones for achieving those goals. Per the request for applications, applicants’ proposals must provide information on how the COINs plan to apply health services research methods, including implementation research. According to VA, as of October 2019, 20 proposals had been selected to receive funding through this new strategy. Diffusion of Excellence Initiative aims to encourage practitioner implementation of research-based practices outside of ORD. VA’s Diffusion of Excellence Initiative, created in 2015, established an annual competition—known as VA’s “Shark Tank”—to engage employees in implementing innovative practices that will positively impact veterans. According to officials from the Diffusion of Excellence Initiative, many of these practices are based on evidence-based research. Under the competition, “investors” (directors from VA medical centers and VISNs) make offers on practices that have been successfully implemented in at least one VA medical center, and the winning investor receives facilitated implementation support so that the practice can be implemented at the investor’s medical center. The officials told us that several of these practices have been identified as exemplary practices and are now being used nationally across multiple VA health care settings. For example, they described one such practice, a tooth-brushing routine implemented for hospitalized veterans to decrease the risk of oral bacteria getting into the veterans’ lungs, which research had shown could increase their risk of pneumonia. According to officials from the Diffusion of Excellence Initiative, the practice decreased hospital-acquired pneumonia by 90 percent at the pilot site, and is being implemented in other VA health care settings. Other VA efforts to facilitate research translation into clinical practice. In addition to the changes to the existing efforts for facilitating research translation, VA has recently taken other actions to help ensure findings from VA research are integrated into practice. In response to ORD’s current strategic priority to “increase the substantial real-world impact of VA research,” the director of ORD established a workgroup to create “The Research Lifecycle,” which was published in October 2019. The lifecycle is a resource that specifies processes to help move research to direct application in routine clinical care. It describes the research and implementation process from identifying innovations that align with clinical priorities to ensuring practices are sustained in clinical care, beyond research and implementation. For example, one phase of the process involves evaluating interventions to determine if they are ready to be implemented into clinical practice. The director of QUERI told us that the information in the publication is broadly applicable across all ORD research and that like the QUERI Implementation Roadmap, the publication reiterates that research is a continuous process rather than a straight line with an endpoint. In addition to the research lifecycle, an official from the agency’s Cooperative Studies Program—which funds large, multi-center clinical trials—told us the program established a new requirement in 2019 that research proposals include an implementation plan. The goal of this change is to encourage researchers to think about research translation from the beginning—and how their work might be translated into clinical practice, according to the program official. Researchers planning to conduct these types of clinical trials will have the opportunity to consult with internal implementation experts to develop plans to translate the research into clinical practice, according to ORD officials. VA officials from both ORD and the national program offices we spoke with described their experiences coordinating on research. Coordination can help both to inform research priorities to make them most useful and applicable, and to encourage the translation of research into clinical practice, which can help VA meet its broader goal of ensuring its research is benefiting veterans’ health. National program offices—such as those for clinical specialties including mental health or spinal cord injury care—provide input to ORD both on research priorities and on efforts to translate research findings into clinical practice within their respective issue areas. For example, officials from the Office of Mental Health and Suicide Prevention told us that their lead staff for suicide prevention participated in strategic planning efforts with ORD to determine a “road map” for current and future research in this area. This VA program office official described working with ORD to provide clinical perspective on gaps in research and clinical care related to suicide prevention, among other things. Given the disproportionately higher rate of suicide among veterans compared with the civilian population, such coordination can help maximize VA’s efforts both in research and in clinical care. Among other things, the road map identifies remaining questions related to suicide prevention to be addressed by VA and other researchers, categorized by type of research (e.g., epidemiological or intervention). Coordination between the research program and national program offices also can facilitate the conduct of the research itself, encouraging research that is viable and relevant to be conducted and translated into practice. For example, ORD leadership told us that program office buy-in on VA research priorities and efforts can lead to VA clinicians being more willing and able to participate in VA research. ORD leadership also told us that ORD has recently begun requesting that researchers engage and collaborate with relevant program offices during the planning process for large multi-site clinical trials, including seeking input from program offices on research proposals. Potential questions for researchers to ask include: does the relevant program office think the proposal’s topic is clinically important; is the research proposal feasible; and will it answer a question that is important from a clinical perspective? According to ORD officials, because VA funds a small number of these types of trials—which are intended to provide a definitive answer to a clinical question—researchers want to be sure the studies are relevant to the needs of the program offices. One specific example both ORD and program office officials provided was related to their coordination on research on osseointegration—a medical procedure through which a metal rod is inserted into the bone at the site of an amputation, allowing a prosthetic limb to be attached through the skin directly to the remaining bone of the amputated limb. Officials from the rehabilitation research service told us that they have been working with program office officials to consider aspects of implementation prior to beginning a clinical trial, including the availability of the surgical procedure throughout VA and the types of post-operative care patients would need. These officials told us that their goal is to ensure the clinical trial is designed for translation. In addition, because national program offices establish policies that affect the provision of care across VA, program office officials told us that collaboration with ORD can help them to incorporate evidence-based practices in developing and rolling out these policies. For example, an official from the Spinal Cord Injury and Disorders System of Care program office told us that it incorporated research findings when it revised its national policy—including a new requirement for all spinal cord injury centers to have vocational rehabilitation counselors on staff. A program office official told us that the addition of this requirement resulted from VA research—led by a researcher clinician—that found that veterans with spinal cord injuries who received specialized vocational support services had the best chance of success for job placement and continued employment. In another example, ORD officials told us that VA researchers were serving as subject matter experts to the national program office developing the protocol and clinical guidelines for implementing intranasal ketamine as a new treatment for certain veterans with treatment-resistant depression. ORD and program office officials described using both informal and formal approaches to coordinate on research priorities and translation. For example, program office officials told us about occasional participation of ORD staff in their regular meetings and calls, as well as relationships between program office staff and individual researchers. ORD officials from one service told us that their scientific portfolio managers serve as a sort of liaison between researchers and clinical program office partners. These officials told us that because VA’s research program is intramural, there is an ongoing discussion with researchers and others within VA in setting research priorities. Although VA officials were mostly positive in describing coordination between the research program and program offices, some officials noted opportunities for improvement, as well. Specifically, officials from the three national program offices we spoke with said it would be beneficial to have a more formal or systematic approach for coordination with ORD. An official from one national program office said a more systematic process would be helpful, so that collaboration is not so dependent on individual relationships or personalities. Officials from another national program office noted that they considered having one staff person as a dedicated resource to liaise with ORD, but lacked resources to do so. Given limited time and competing priorities for researchers and program office officials, ORD officials told us that it would be best to focus on strategic coordination, and noted that some such efforts are underway. Specifically, ORD leadership acknowledged that it would be helpful for ORD and program offices to engage more in general—particularly related to ORD’s research priorities. However, because ORD leadership said it would not be efficient to have to go “door to door” to each individual program office or VISN to have those discussions, it would be more helpful to find more strategic ways to engage. For example, ORD leadership said that ORD’s inclusion in larger annual VHA strategic planning sessions could be a way to facilitate strategic coordination. Similarly, health services research service officials told us that service’s new effort to build consortiums of researchers focused together on a particular priority area may also facilitate coordination between researchers and clinical program offices, particularly on key topics for VA, such as suicide prevention and opioids. In addition, ORD leadership told us that ORD is focusing its efforts on “big ticket items”—such as larger studies or clinical trials through the Cooperative Studies Program—where there can be a big impact through collaboration with program offices, because a single study generally does not lead to changes in clinical practice. Another mechanism available to facilitate strategic coordination between the research program and national program offices is ORD’s QUERI— particularly its Partnered Evaluation Initiatives—through which researchers partner with national program offices to evaluate specific initiatives with potentially high impact on VA national policy. For example, QUERI investigators have partnered with the Office of Mental Health and Suicide Prevention to evaluate an upcoming initiative to send “caring letters” to veterans who have called the Veterans’ Crisis Line and have been engaged in VA care recently. Caring letters, letters noting that the veteran is cared about and matters, are an intervention shown to be effective in reducing suicides in various at-risk populations. In addition, VA program office officials told us that while most VA program offices have their own internal program evaluation services, they do not have sufficient resources to evaluate the effectiveness of all of their programs and policies, motivating them instead to work with QUERI and ORD. For example, the Office of Mental Health and Suicide Prevention has also partnered with QUERI on the STAR VA program, to examine non- pharmacological approaches to treating agitation and other issues in veterans with dementia. Program office officials told us that they can use the results of these evaluations to influence policy, standard operating procedures, and treatment in the field. In summary, VA is uniquely positioned to implement research into clinical practice because of the research program’s adjacency to such a large, integrated health care system. As we have noted, coordination between the research program and partner entities could help ensure VA-funded research results in the spread and adoption of evidence-based practices. VA recognizes the importance of this coordination and continues to actively pursue effective coordination strategies. We provided a draft of this report to VA for review and comment. VA provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of the Department of Veterans Affairs and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or farbj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In fiscal year 2018, VA’s appropriation for its intramural research program totaled $722 million. Of this amount, $558 million was for awards made by the Office of Research and Development’s (ORD) four research services and the Cooperative Studies Program. Table 3 below presents data on VA’s intramural research program funding and awards for fiscal year 2018. In addition to the contact named above, Raymond Sendejas (Assistant Director), Julie T. Stewart (Analyst-in-Charge), Lauren Anderson, Jennie F. Apter, Robin Burke, and Taylor German made key contributions to this report. Also contributing were Jacquelyn Hamilton and Vikki Porter.", "summary": "In addition to providing health care services, VA funds research on veterans' health conditions, including chronic conditions (such as diabetes) as well as illnesses and injuries resulting from military service (such as TBI). VA's ORD manages the agency's research program, including its intramural research. In fiscal year 2018, VA resources for its intramural research program included an appropriation of $722 million. GAO was asked to review aspects of VA's research program. In this report, which focuses on VA's intramural research, GAO describes 1) how VA sets priorities for funding research, 2) VA efforts to facilitate translation of research into clinical practice, and 3) coordination between VA's research program and other VA entities. To perform this work, GAO reviewed VA policies, reports, and other documents about VA research efforts. GAO also interviewed officials from ORD, three VA national clinical program offices, and two VA offices that focus on implementing evidence-based practices. In addition, GAO conducted site visits with four VA medical centers. GAO selected those locations because they house VA-funded research centers that focus on a range of topics and ORD programs that focus on disseminating and translating research. At each location, GAO interviewed medical center officials and VA researchers. GAO also reviewed VA summary data on research projects and funding for fiscal year 2018. VA provided technical comments on a draft of this report, which GAO incorporated as appropriate. The Department of Veterans Affairs (VA) uses stakeholder input and other information to set priorities for funding research projects. VA's Office of Research and Development (ORD) manages VA's intramural research program—that is, research funded by and conducted within VA, by VA researchers. To set priorities, ORD considers input from VA and non-VA stakeholders (such as agency leaders and a federal research advisory council, respectively) and data on veterans' health conditions. ORD encourages VA researchers to study—and collaborate with other VA researchers on—priority topics, such as post-traumatic stress disorder (PTSD) and traumatic brain injury (TBI). ORD's Quality Enhancement Research Initiative (QUERI) and other VA entities facilitate translating research findings into clinical practice to improve care for veterans. QUERI is VA's central point of focus for research translation and provides a link between ORD, VA program offices, and providers. For example, one QUERI program is studying delivery of an evidence-based treatment for PTSD using telemedicine, specifically, by providing psychotherapy via video to veterans in rural areas. Another program recently adopted a new research translation strategy by establishing a requirement that research proposals for large, multi-center clinical trials include an implementation plan. VA officials said the goal of the new requirement is to encourage researchers to think about research translation from the beginning of a study—and how their work might be translated into practice. VA officials from both ORD and the national program offices GAO spoke with described a variety of efforts coordinating on research. Such coordination can help inform research priorities and help program offices incorporate evidence-based practices in developing and rolling out national policies. For example, ORD officials said that VA researchers were serving as subject matter experts to the national program office developing a protocol and clinical guidelines for a new treatment for certain veterans with depression that is resistant to existing treatments.", "document_type": "gao"}
{"report": "States have three options for designing their CHIP programs: Medicaid expansion CHIP, separate CHIP, and combination CHIP. Medicaid expansion CHIP. States may operate CHIP as an extension of their Medicaid programs. Under Medicaid expansion CHIP, states expand income eligibility levels for children beyond those of the state’s Medicaid program. Medicaid expansion CHIP programs must follow Medicaid rules, including providing all Medicaid covered benefits to enrolled children. Separate CHIP. States may operate their CHIP programs separate from their Medicaid programs. In so doing, the states are not required to follow the same rules as Medicaid; thus, these states have some additional flexibility in designing CHIP, such as determining which benefits to offer and how, if at all, to charge premiums. Combination CHIP. States may have a combination program, where they operate a separate CHIP program, as well as a Medicaid expansion CHIP program, each for a different population of children. For example, some states that operate combination CHIP programs apply different age or income eligibility requirements for their Medicaid expansion CHIP and separate CHIP programs. Similar to Medicaid, CHIP program expenditures are shared between the states and the federal government, but federal matching rates for CHIP are higher than for Medicaid and federal funding for CHIP is capped, with states receiving annual CHIP allotments. The type of CHIP program a state designs may affect the amount of federal funding available to that state in the event the state exhausts available CHIP funding for the year. A state with a Medicaid expansion CHIP program that exhausts available CHIP funding may apply Medicaid funds at the Medicaid matching rate to remaining expenses for enrolled children for that year. However, a state with a separate CHIP program that exhausts available funding would not have access to such funding. In general, states administer CHIP under broad federal requirements that permit flexibility in how they design their programs, including in the services they cover, their upper income eligibility limits, and the fees they charge to participate. In terms of income eligibility, as of January 2019, 19 states, including the District of Columbia, had CHIP upper income eligibility limits of 300 percent of the FPL or higher compared with 32 states whose CHIP upper income eligibility limits were below 300 percent of the FPL. (See fig. 1.) In addition, states can charge beneficiaries fees for CHIP coverage. These fees can vary depending on whether they are enrollment fees, premiums, or other types of cost sharing. Among the states that charge CHIP premiums, the premiums can vary based on family income and the number of children in CHIP. (See table 1.) Although states may charge premiums or have other cost sharing, according to CMS, CHIP provides more affordable coverage than is generally available in the private health insurance market. CHIP crowd-out may occur when employers modify or decide not to offer health insurance to their employees or to their dependents, because of CHIP availability. For example, employers who are aware of CHIP may decide not to offer health insurance to employees or their dependents due to concerns about the costs of providing insurance, especially for smaller sized firms, or as a result of changes in federal or state policies, such as requirements resulting from PPACA. Crowd-out may also occur when employees drop or decide not to enroll in insurance offered by their employers and enroll their children in CHIP, because of CHIP availability. As we have identified in prior work, assessments of the potential for crowd-out must take into account an understanding of the extent to which private health insurance is available and affordable to low-income families who qualify for CHIP. National survey results show that private health insurance is the most prevalent source of insurance for children; however, there is substantial variation across states in coverage rates. Additionally, the extent to which employers offered individuals insurance varies by family income. For additional information on factors that may affect crowd-out, see appendix I. For information on sources of health insurance for children under age 19, including CHIP and employer sponsored insurance, see appendix II. The type of CHIP program a state designs affects its responsibilities for monitoring and mitigating the potential for CHIP crowd-out. The 42 states with separate CHIP programs—including those in combination CHIP states—are required to submit CHIP plans that describe reasonable procedures to prevent crowd-out and to report annually to CMS on certain crowd-out related indicators, such as the number of CHIP applicants with access to private health insurance; however, CMS provides states flexibility to decide which crowd-out prevention procedures to use. For example, states can require CHIP applicants to undergo a period of uninsurance prior to enrollment, known as a waiting period, to deter families that have access to private health insurance from dropping that insurance to enroll in CHIP. In contrast, states are not required to take steps to prevent crowd-out for their Medicaid expansion CHIP programs and may only do so if consistent with the Medicaid statute, or if under an approved section 1115 demonstration, which allows states to implement policies that waive certain Medicaid requirements. For states with separate and combination CHIP programs, CMS provides general guidance for minimizing crowd-out, which the agency has modified over time. (See table 2 for a description of the crowd-out related responsibilities.) For example, in 2013, CMS issued regulations to align with a PPACA provision for health plans and health insurance issuers that limited waiting periods to a maximum of 90 days, and established mandatory waiting period exemptions. The regulations also eliminated the application of a CHIP policy requiring that states with separate CHIP programs have different crowd-out prevention procedures in place for children at different income levels. In making this change, CMS noted that available research called into question the prevalence of crowd-out. CMS indicated that its policy still required states to monitor crowd-out and, if a high rate of crowd-out were to occur, states should consider implementing prevention procedures, such as public outreach about other health care options available in the state. In response to crowd-out related recommendations we made in 2009, CMS modified its guidance to collect additional information from states in their 2009 through 2013 annual reports on how they assess the availability and affordability of private health insurance for CHIP applicants. For example, from 2009 through 2013, states were required to report to CMS if the state’s CHIP application asked if applicants had access to private health insurance. Additionally, states that operated a waiting period without affordability exceptions were asked if the state collected data on the cost of health insurance for an individual or family. However, CMS officials stated that the agency eliminated the questions regarding affordability of private health insurance in 2013, as part of efforts to update the electronic system states use to submit their CHIP annual reports to reflect PPACA enrollment simplification and coordination requirements. CMS officials said some of the questions were duplicative of other state reporting requirements and other questions were deemed irrelevant in light of the establishment of affordability exceptions to waiting periods. States reported indicators of potential crowd-out to CMS in their annual reports, although some do not report on these indicators and those that do may calculate them differently. The states also varied in the extent to which they have processes for directly estimating crowd-out; however, CMS officials and officials in selected states told us they understand the occurrence of crowd-out to be low. Further, we identified few published research studies that directly estimated crowd-out; each used different methodologies, resulting in varied estimates. States with separate CHIP programs—including those in combination states—are required to annually report indicators of potential crowd-out; states must also describe in their CHIP plans other indicators of potential crowd-out they collect. CMS’s 2017 CHIP annual report asks these states to report on crowd-out related questions, including two indicators of crowd-out: (1) the percentage of individuals who enrolled in CHIP that have access to private health insurance, and (2) the percentage of CHIP applicants who cannot be enrolled, because they have private health insurance—an indicator of potential crowd-out averted. However, not all states with separate CHIP programs track and report information related to these two indicators of potential crowd-out, and those that do may calculate these indicators differently. For example, of the 42 states with separate CHIP programs, the 2017 annual reports showed the following: Four of the 42 states reported that they tracked the number of individuals who have access to private health insurance; the remaining 38 states either did not report tracking this information or did not respond to this question. Of the four states tracking this information, the percentages reported ranged between 0.5 percent and 7 percent of CHIP applicants who have access to private health insurance. Twenty-one of the 42 states reported that they tracked the percentage of applicants who could not be enrolled in CHIP because they were enrolled in private health insurance; the remaining 21 states did not report this percentage to CMS. This is a measure of crowd-out averted due to state oversight of its enrollment process. The percentages reported by the 21 states tracking this information ranged from 0 percent in several states to 18 percent in one state. Among the states that reported they do not track individuals with access to private insurance and did not provide a percentage of applicants not enrolled in CHIP because of enrollment in private health insurance, five states indicated that either their electronic eligibility systems did not allow them to capture this information or the data to report this information were not available. CMS officials acknowledged that not all states report on these indicators; however, they noted that states operating separate CHIPs have other processes in place to prevent children with other health insurance from enrolling in CHIP. Further, some states that operate separate CHIP programs describe approaches for directly estimating crowd-out in their CHIP plan amendments. The results of these estimates are not reported to CMS unless they reach a threshold defined by each state. In 2013, CMS required separate CHIP states to submit state plan amendments to CMS to update their eligibility-related policies, including their crowd-out prevention procedures. In response, 17 of the 42 states submitted these amendments and described approaches they would use to directly measure crowd-out. For example: Colorado reported conducting a biennial survey to estimate the percentage of enrollees who dropped group health insurance without good cause to gain eligibility for CHIP, according to its CHIP plan. Connecticut reported comparing the number of children denied CHIP enrollment because they were enrolled in private health insurance to those same applicants who reapplied for CHIP 6 months later, but did not have private health insurance. The crowd-out threshold defined by Colorado and Connecticut is 10 percent; therefore, if these states’ crowd-out estimates were to exceed 10 percent, each state would collaborate with CMS to identify other procedures to reduce crowd-out. According to CMS officials, no state using this approach to estimate crowd-out has exceeded the percentages established or expressed concerns with crowd-out. States we interviewed varied in the extent to which they estimate crowd- out; however, most states did not view crowd-out to be of concern. Among our six selected states with separate CHIP programs, one state— New York—directly measures crowd-out. New York asks applicants that dropped their private insurance in the last three months the reasons why they dropped this coverage, which includes responses such as the family’s preference for the child to have CHIP benefits over their previously held private health insurance. New York state officials told us they consider instances of crowd-out to include when individuals drop private insurance because CHIP costs and benefits are more favorable. For the last 9 months of 2014, the officials estimated crowd-out in New York to be about 1.9 percent. If New York estimates crowd-out to be higher than 8 percent, state officials told us they will report this to CMS and work with CMS on implementing additional crowd-out prevention procedures. Officials from the other five selected states said they do not actively measure crowd-out, some of them citing limited resources and difficulties developing estimates, and noted that crowd-out was not a high priority for them, because they did not think crowd-out was prevalent in their states. For example, officials from two states said they had not heard any concerns regarding crowd-out from their state legislature, state insurance agencies, or others. CMS officials also told us that no state had reported concerns about crowd-out. Our review identified few research studies that directly estimated CHIP crowd-out. Specifically, we identified three research studies published from 2013 to 2018; each used different methods and arrived at varying estimates of crowd out. One study estimated crowd-out across 15 states that expanded their CHIP income eligibility requirements between 2008 and 2012 by examining health insurance enrollment changes in a sample of children after they became newly eligible for CHIP. This study estimated that public insurance among children under age 19 increased about 2.9 percentage points during this period, and private insurance decreased by 1.8 percentage points. The study reported that 63 percent of the 2.9 percentage point increase in public insurance was due to crowd-out. The researchers also produced state-level estimates for the effects of CHIP income eligibility expansions on insurance coverage in newly eligible children. These estimates varied by state, suggesting that crowd-out also varies by state. In particular, three states had an increase in public insurance ranging from about 4 to 12 percentage points, and three states had a decrease in private insurance that ranged from about 7 to 14 percentage points. The researchers noted they did not account for factors that may have caused privately insured individuals to increase their use of public insurance, such as changes in the affordability of private health insurance. Another study estimated the effect of CHIP income eligibility expansions on crowd-out in Illinois. This study examined the differences in public and private health insurance between children in Illinois, where CHIP income eligibility was expanded, and children from a combination of other states that did not expand CHIP—and were chosen to resemble the demographic characteristics and health insurance profile of Illinois. This study found a 6.5 percentage point increase in CHIP enrollment in 2010 among families between 200 percent and 300 percent of the FPL, and estimated that 35 percent of this increase in CHIP enrollment was due to crowd-out. At other income levels higher than 300 percent of the FPL, the study found either no net effect on private health insurance, or an increase. The third study estimated public and private insurance under different CHIP income eligibility thresholds and different premium schedules. While the study estimated that a CHIP expansion from 200 to 400 percent of the FPL with no premium contribution and a 4 month waiting period increased CHIP enrollment by about 4.5 percentage points and decreased private coverage by about 2.2 percentage points, these estimates do not provide evidence of crowd-out, because the differences in these percentage point estimates were not statistically significant. Although not reporting direct estimates of CHIP crowd-out, we identified other studies that provide related information. For example: In one study, researchers surveyed the parents of current and former CHIP enrollees in 10 states to examine access to private coverage for children enrolled in CHIP. This study found that about 13 percent of new CHIP enrollees had private health insurance in the year before enrolling in CHIP. Among the 13 percent, about 18 percent reported that they dropped their private health insurance, because CHIP was more affordable, and about 5 percent dropped their private health insurance, due to a preference for CHIP. The authors noted that access to private coverage among CHIP enrollees is low and when access is available, affordability is a serious concern for parents. The authors concluded that this suggests limited potential for crowd-out. A study published in 2015 that surveyed the parents of about 4,100 new CHIP enrollees to understand why children enrolled in CHIP, among other things, found that 35 percent of these parents reported applying for CHIP, because it was more affordable than the other health insurance options they could obtain for their children. Representatives from national organizations, researchers, and CMS officials we interviewed noted some of the challenges measuring the extent of CHIP crowd-out, including the limitations of available data sources; however, they did not consider crowd-out to be prevalent. For example: Some data sources do not separately collect or categorize CHIP information. For example, the ACS does not specifically ask respondents if their children have health insurance through CHIP; thus, researchers have to manipulate the data to separate CHIP coverage from other forms of public health insurance, such as Medicaid. The methodologies available to separate CHIP from Medicaid respondents have many limitations, according to researchers and U.S. Census Bureau officials we contacted. Accurate crowd-out estimates require researchers to account for the reasons why someone dropped his or her health insurance and enrolled in CHIP, and this information is not captured by national surveys. Researchers may also vary in what they consider to be crowd-out; for example, some may not consider dropping private health insurance and enrolling a child in CHIP because of a job loss or change in employment to constitute crowd-out. Others do not consider it to be CHIP crowd-out when parents drop their private health insurance and enroll in CHIP, because CHIP is more affordable. CMS officials also noted complexities in measuring crowd-out—such as variation in definitions of crowd-out and methodologies for measuring it—and they said that the agency has not conducted or commissioned its own evaluation. However, CMS officials reiterated that no state has reported concerns with crowd-out and based on their review of studies conducted by researchers understand that its prevalence is likely low. CMS monitors states’ CHIP crowd-out prevention procedures and offers technical assistance, while states ask CHIP applicants about other sources of health care coverage, and use waiting periods and cost- sharing procedures, such as enrollment fees and premiums. Several state officials we interviewed told us that their crowd-out prevention procedures are effective; however, they could not speak to the effectiveness of any particular procedure and few studies have examined the issue. CMS officials told us that they track the information states submit about their CHIP crowd-out prevention procedures as part of their annual report review process to identify any inconsistencies between the information contained in their state plans and the information submitted in states’ annual reports, among other reasons. When CMS officials identify any noticeable differences in the information reported by states from year-to- year in the annual reports—such as the percentage of CHIP applicants with access to private insurance—they told us they follow-up with the state to obtain additional information about these differences, and, if needed, advise states on ways they can prevent crowd-out. CMS officials also told us they provide technical assistance, when requested, to assist states in developing crowd-out prevention procedures. For example, CMS officials said they provided states with technical assistance after issuing regulations in 2013 on the use of waiting periods that also required states to update their state plan amendments. CMS officials said they have no plans to develop additional strategies for collecting states’ crowd-out information, because states have not reported crowd-out to be a concern, and there is no need to re-examine states’ oversight if prevalence as measured in research is likely low. All 42 states with separate CHIP programs reported to CMS that they had implemented at least one of the following six types of procedures to prevent crowd-out: (1) asking about other health insurance and denying CHIP coverage if other sources of health insurance are identified; (2) implementing cost sharing for CHIP coverage; (3) conducting database checks for other health insurance; (4) implementing a waiting period for CHIP coverage; (5) measuring crowd-out and taking steps if certain thresholds are exceeded; and (6) offering premium assistance for private health insurance. The majority of these states (36 of the 42 states with separate CHIP programs) implemented at least three crowd-out procedures. All 42 states with separate CHIP programs asked applicants about other insurance coverage on their CHIP applications to deny applicants CHIP coverage if private insurance coverage was found, and CMS officials told us that 35 of those states required CHIP enrollees to pay premiums or make other financial contributions to the cost of the coverage. (See table 3.) Among our six selected states with separate CHIP programs, there were differences in how some crowd-out procedures were implemented. For example, three states conducted database checks to see if applicants had other sources of health insurance; however, one state checked prior to enrollment, another checked at enrollment and during application renewal, and one state ran weekly checks. Among our six selected states with separate CHIP programs, none planned to change procedures to prevent potential crowd-out. Among the 42 states with separate CHIP programs, some crowd-out prevention procedures vary or have changed over time. For example, while many states use a private company to collect state and national health insurance coverage information to conduct database checks, another state developed a database that contains information on insurance coverage available through over 40,000 employers in the state. Additionally, prior to 2014, 36 states imposed waiting periods, during which applicants could not have health insurance for a specified time before CHIP enrollment, to prevent crowd-out. In 2017, 14 states used waiting periods. Prior to PPACA and the implementation of CMS regulations that limited waiting periods to 90 days, waiting periods could range from 1 to 12 months. After CMS updated its regulation, 21 states eliminated their waiting periods and five states shortened them. Among our four selected states with separate CHIP programs that shortened or eliminated their waiting periods, none of the state officials expressed concerns that this change contributed to CHIP crowd-out. Administering a waiting period may involve the state tracking or determining whether the applicant meets any of the state and federal waiting period exemptions, the number of months for the waiting period before the applicant can be enrolled in CHIP, and informing the federally facilitated exchange if an exemption to the waiting period applies to the applicant. As a result, some officials noted that reducing waiting periods eased their state’s administrative burdens, as well as eliminated gaps in children’s health insurance. Among the four selected states, officials from New York said they eliminated their waiting periods because, after undergoing the various administrative steps to verify each application and apply the waiting period, the majority of the CHIP applicants met at least one waiting period exemption. However, three of the selected states with separate CHIP programs maintained waiting periods, and state officials from Texas told us that few individuals met the waiting period exemptions. Some state officials told us they attributed waiting periods—which require children to go uninsured for a period of time—to gaps in health care, and their states eliminated the waiting period in an attempt to provide continuity in children’s access to health care. Although not required by law, officials from two of our selected states with Medicaid expansion CHIP programs told us their states previously had approved 1115 demonstration waivers permitting their states to use a CHIP waiting period, but eliminated them in 2013 and 2014 to close gaps in children’s health insurance coverage. Currently, these states use similar procedures as separate CHIP states to prevent crowd-out, according to state officials. Of our three selected states with Medicaid expansion CHIP programs, one state monitors CHIP enrollment trends; a second state requires its managed care organizations to check CHIP enrollees for other sources of insurance as part of their claim processing activities; and one state conducts database checks for other health insurance at the time of enrollment and re-enrollment. The effect of some of the states’ procedures on preventing CHIP crowd- out is unclear and, according to selected state officials and stakeholders, some crowd-out prevention procedures may have unintended consequences. For example, state officials and stakeholders told us waiting periods result in coverage gaps, which, as one stakeholder noted, could be catastrophic for a family with a sick child who would not have coverage during the waiting period. Several CHIP officials we interviewed believed their procedures are effective in preventing crowd- out; however, they either had not studied the effectiveness of their procedures or could not speak to the effectiveness of any particular procedure. Relatively few of the studies we reviewed examined the effectiveness of state procedures for preventing crowd-out. Specifically, two studies looked at this issue. Both studies concluded that cost-sharing procedures, such as premiums, can reduce the potential for crowd-out among higher- income CHIP-eligible families. A 2014 study used CHIP-related data from 2003 and found that CHIP premiums discourage individuals with private health insurance from dropping their insurance to enroll in CHIP. The study compared health insurance outcomes across 19 states for children with incomes slightly above states’ CHIP income eligibility thresholds with children in families with incomes slightly below the thresholds. The results indicated that there is an association between CHIP premiums and private insurance coverage; that is, a $1 increase in the CHIP premium above the income cut-off is associated with a 2.2 percentage point higher probability of the child being privately insured for families within 15 percent of the upper income level, and a 1.7 percentage point higher probability for families within 25 percent of the upper income level. These findings suggest that private health insurance may be a preferable alternative for CHIP eligible families at higher income levels who face higher CHIP premiums. A 2013 study used survey data from 50 states and the District of Columbia from 2002 to 2009 to estimate the effect CHIP premium contributions have on enrollment in CHIP, private insurance, and rates of uninsurance among children in families with income eligibility levels of 200 to 400 percent of the FPL. The study found that if CHIP programs expand eligibility to those at higher income levels and charge those families a higher premium, the families may be more likely to choose private health insurance, nullifying the effects of CHIP expansion among higher income families. We provided a draft of this report to HHS for review and comment. The department provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services, the Administrator of CMS, appropriate congressional committees, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or at yocomc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to this report are listed in appendix III. Crowd-out may occur when employers modify or decide not to offer health insurance to their employees or to their dependents because of Children’s Health Insurance Program (CHIP) availability. For example, employers who are aware of CHIP may decide not to offer health insurance to employees due to concerns about the costs of providing insurance, especially for smaller sized firms, or as a result of changes in federal or state policies, such as requirements resulting from the Patient Protection and Affordable Care Act (PPACA). For example, PPACA required employers with a certain number of employees to offer their full-time employees a health insurance option meeting certain criteria, including affordability, or face tax penalties. Some researchers and policymakers expressed concern that this requirement may encourage employers to change how they offer insurance to employees, such as no longer offering family and dependent coverage, instead only offering health insurance to the employees, thereby causing employees with children to seek public insurance or insurance through health insurance exchanges. Other researchers and organizations point to PPACA increasing the availability of private health insurance offered by employers and through health insurance exchanges, particularly in areas and among populations where employer sponsored health insurance may not be as readily available. Crowd-out may also occur when employees drop or decide not to enroll in insurance offered by their employers and enroll their children in CHIP because of CHIP availability; however, as we have reported in the past, assessments of crowd-out should consider the affordability and availability of the employer sponsored insurance. For example, families with access to employer sponsored insurance may find CHIP more affordable or find CHIP benefits more comprehensive than employer sponsored insurance. Alternatively, they may find that CHIP provides better access to services specific to their child’s health care needs. For example, an evaluation of CHIP published in 2014 found that CHIP enrollees had better access to dental benefits than children with private insurance, although they were less likely to have a regular source of medical care and nighttime or weekend access to a provider. As we have identified in prior work, assessments of the potential for crowd-out must take into account an understanding of the extent to which private health insurance is available and affordable to low-income families who qualify for CHIP. American Community Survey (ACS) data showed that for 2013 through 2017, the most prevalent source of insurance for children in the United States under the age of 19 was private health insurance available through a parent’s employer or union. (See fig. 2.) Although private health insurance is the most prevalent source of insurance for children, there is substantial variation across states in coverage rates. (See fig. 3.) For example, in eight states, fewer than 40 percent of children were insured through an employer in 2017. In contrast, in Utah, more than 60 percent of families with children were insured by an employer in 2017. Medical Expenditure Panel Survey (MEPS) data show that the extent to which employers offered individuals insurance in 2013 through 2015 varied by family income. For example, MEPS Household Component data—which includes information on whether individuals were offered insurance by their employers—show that over 90 percent of families with incomes greater than 400 percent of the federal poverty level (FPL) were offered insurance by their employers from 2013 through 2015. The percentage of families offered insurance by their employers ranged from about 35 percent for families with incomes less than or equal to 138 percent of the FPL to about 85 percent for families with incomes above 300 and less than 400 percent of the FPL. (See fig. 4.) An Agency for Healthcare Research Quality (AHRQ) analysis of MEPS Insurance Component data—which includes information on whether employers offered insurance to their employees and the cost of that insurance— shows that in 2017, 24.2 percent of small employers (less than 50 employees) with a predominately lower-wage workforce offered their employees health insurance compared with 57.6 percent for small employers with a higher-wage workforce. In contrast, in 2017, offer rates at larger employers—that is, employers with more than 50 employees— was 94 percent for those with predominately lower-wage employees and 98.7 percent for large employers with predominately higher wage employees. With regard to affordability, the MEPS Insurance Component data show that average employee premium contributions for family coverage from 2013 through 2017 increased. Over this period, employees who work for employers with a predominantly lower-wage workforce—that is, employers that paid 50 percent or more of their workforce $12 or less per hour—contributed a larger amount and percentage of premiums to their employer-sponsored insurance than did employees who work for non- low-wage employers. (See fig. 5.) MEPS Insurance Component data also show that employees who work at establishments with a predominately lower-wage workforce enroll in insurance offered by their employers at a lower rate than employees of other establishments, though it is not known if this is due to affordability reasons. Finally, MEPS Insurance Component data show that the percentage of employees with deductibles and the amount of the deductibles have increased from 2004 to 2017. Between 2013 and 2017, average family deductibles increased about 36 percent, from $2,491 in 2013 to $3,396 in 2017. In addition, research published in 2018 on high deductible health insurance plans showed both increasing enrollment in these plans and that larger employers (1,000 or more employees) contributed more toward health insurance premiums for these plans than smaller employers (less than 25 employees). For example, according to this study: From 2006 to 2016, there was a 35 percentage point increase (11.4 percent to 46.5 percent) in enrollees in high-deductible health plans, with enrollees from smaller employers more likely to be enrolled in these plans compared with enrollees from larger employers (56.4 percent of enrollees from small firms compared with 42 percent of enrollees from large firms). A lower percentage of enrollees from the smaller firms had a plan with an employer-funded account, which defray health care costs, compared with enrollees from larger firms. For example, in 2016, only about one-third of enrollees in high-deductible health insurance plans from the smallest employers had an employer funded account to help pay for medical expenses compared with 89.3 percent of enrollees from the largest employers. High-deductible health insurance plan enrollees of the smallest employers were also more likely to not have the choice of an alternative plan type compared with enrollees from the largest employers. Although private health insurance is the most prevalent source of insurance for children, there is substantial variation across states in coverage rates. Figure 6 provides information on the percentage of children under age 19 insured through employer sponsored insurance, Medicaid, and the Children’s Health Insurance Program, as well as those who were uninsured in 2017. Carolyn L. Yocom, (202) 512-7114 or yocomc@gao.gov. In addition to the contact named above, individuals making key contributions to this report include Shannon Legeer (Assistant Director), Toni Harrison (Analyst-in-Charge), Mollie Lemon, and Courtney Liesener. Also contributing were Alison Binkowski, George Bogart, Jill Center, Leia Dickerson, Giselle Hicks, Drew Long, Kristeen McLain, Yesook Merrill, Jasleen Modi, Vikki Porter, Lisa Rogers, and Merrile Sing.", "summary": "CHIP is a public insurance program established in 1997 that finances health care for over 9 million low-income children whose household incomes do not qualify them for Medicaid. States have flexibility in structuring their CHIP programs under broad federal requirements, and their income eligibility limits vary. Policymakers have had concerns that some states' inclusion of children from families with higher income levels could result in some families substituting CHIP for private insurance (i.e., crowd-out). Crowd-out may occur when, because of CHIP availability, (1) employers make decisions about offering health insurance; or (2) employees make decisions about enrolling in employer-sponsored health insurance. GAO was asked to examine CHIP crowd-out. This report describes (1) the information on potential indicators of crowd-out reported by states and estimates of crowd-out; and (2) the procedures CMS and states use to address potential crowd-out. GAO reviewed federal laws and guidance and state CHIP documentation, including their 2017 annual reports (the latest available at the time of GAO's review); conducted a literature review of studies published between 2013 and 2018; and interviewed CMS officials, stakeholders from national health policy organizations, and researchers. GAO also interviewed a non-generalizable selection of officials from nine states chosen to obtain variation in CHIP programs, such as income eligibility levels and geography. HHS provided technical comments on a draft of this report, which GAO incorporated as appropriate. Limited information exists about Children's Health Insurance Program (CHIP) crowd-out—that is, substituting CHIP for private health insurance. The Centers for Medicare & Medicaid Services (CMS), within the Department of Health and Human Services (HHS), asked the 42 states that have separate CHIP programs to report on two crowd-out indicators for the 2017 annual reports: (1) the percentage of individuals who are enrolled in CHIP that have access to private health insurance and (2) the percentage of CHIP applicants who cannot be enrolled because they have private health insurance. The 2017 reports showed that: 4 states reported 0.5 percent to 7 percent of CHIP applicants had access to private health insurance; and 21 states reported denying CHIP enrollment to 0 percent to 18 percent of applicants because they had private insurance. Not all of these 42 states reported on these indicators and GAO found that those that do may calculate them differently. CMS officials acknowledged that not all states report on these indicators; however, they noted that states operating separate CHIPs have other processes in place to prevent children with other health insurance from enrolling in CHIP. Further, some states may have other processes for directly measuring CHIP crowd-out. GAO also identified three studies published between 2013 and 2018 that estimated CHIP crowd-out. However, these studies used different methods to calculate crowd-out, and as a result produced varied estimates. For example, one study attributed a portion of increased enrollment in CHIP and other public insurance to crowd-out, while another study found no evidence of crowd-out. According to CMS's 2017 annual reports and other information, the 42 states with separate CHIP programs reported implementing at least one of six types of crowd-out prevention procedures. Source: GAO analysis of information from the Centers for Medicare & Medicaid Services, state Children's Health Insurance Programs (CHIP), and a Kaiser Family Foundation and Georgetown Center for Children and Families survey on Medicaid and CHIP programs. │GA O-20-12", "document_type": "gao"}
{"report": "Explosives include high explosives, propellants, and pyrotechnics. Propellants and pyrotechnics are sometimes referred to as low explosives. All three types of explosives serve essential functions in nuclear weapons. Figure 1 illustrates key explosive-containing components found in a generic nuclear weapon as well as the types of explosives these components contain. High explosives are the most common explosive by volume in nuclear weapons. There are two classes of high explosives used in nuclear weapons: insensitive high explosives (IHE) and conventional high explosives (CHE). An IHE is less susceptible to accidental detonation than a CHE and less violent upon accidental ignition, therefore it is safer to handle. NNSA places a premium on safety throughout all phases of explosives activities, including research and development, testing, production, and storage, because handling any explosive material is inherently dangerous, according to NNSA officials and contractor representatives. Producing a high explosive material generally follows four steps, as shown in figure 2: (1) synthesis—producing raw explosive molecules; (2) formulation—mixing raw explosive molecules with binding ingredients to form an explosive mixture; (3) pressing—compacting formulated explosives into shapes of the required density; and (4) machining— cutting away excess material to achieve the final shape. Analytical, mechanical, safety, and performance testing are to occur after each step. During synthesis, technicians use chemicals to produce fine, powder-like raw explosives. During formulation, technicians combine the explosive powder with plastic binder ingredients to produce a mixture that exhibits the physical and performance properties desired. Formulated explosives used by NNSA often appear like small, irregularly shaped pebbles, known as prills, as shown in figure 3. During pressing, the third step, technicians compact formulated explosives into a solid form. During machining, the fourth step, technicians use computer-controlled equipment to cut and shape the explosive into its final shape. After the explosive has been machined, technicians join explosive and non-explosive parts into functional components during subassembly. Small-scale synthesis and formulation and production-scale pressing, machining, and subassembly activities are carried out at multiple NNSA sites. After each step of the production process, NNSA’s sites conduct tests to ensure that explosives meet NNSA’s safety and performance requirements. During safety testing, scientists conduct a variety of tests to ensure that explosives meet DOE’s safety requirements. Regarding performance testing, scientists conduct other tests that require specialized equipment. For example, scientists use scanning equipment, like heat flow sensors, for thermal testing on formulated explosive material. Scientists also conduct tests using X-ray imaging equipment to evaluate weapon characteristics by detonating a “mockup.” The mockup uses a high explosive main charge—the explosive material that surrounds the nuclear core, known as the pit—and a nonfissile surrogate material that has similar physical properties to plutonium. The mock implosion is called a hydrodynamic test because the surrogate material and other components become hot enough to flow like fluid. High explosive molecules used in U.S. nuclear weapons include but are not limited to high melting explosive (HMX), pentaerythritol tetranitrate (PETN) and triaminotrinitrobenzene (TATB). First fielded in conventional weapons in World War II, HMX and PETN were later introduced into several components in the U.S. nuclear weapons stockpile and are still used in them today. DOE first introduced TATB into the nuclear stockpile in 1979, and it is still the only molecule that DOE considers to be an IHE (see sidebar). In all U.S. nuclear weapons, the main charge is made of formulations of HMX or TATB. DOD also uses HMX and TATB in certain conventional weapons. TATB: NNSA’s Key Insensitive High Explosive Triaminotrinitrobenzene (TATB) is a key insensitive high explosive that is currently used in National Nuclear Security Administration (NNSA) and Department of Defense (DOD) military applications, including nuclear and conventional weapons. Scientists first synthesized TATB in 1888 but did not initially recognize it as an explosive. In 1966, Los Alamos National Laboratory developed the industrial method for synthesizing TATB. From the late 1970s to the late 1980s, two domestic manufacturers supplied TATB to DOD and NNSA. However, when the Cold War ended and a U.S. nuclear test moratorium began, the demand for TATB declined, and both manufacturers ceased production by 1993. DOD then acquired TATB from a U.K.-based firm until its plant closed in 2005. Beginning in 2007, DOD and NNSA collaborated to re-establish a manufacturing capability for TATB in the United States. Specifically, DOD’s Holston Army Ammunition Plant (Holston), which is located in Kingsport, Tennessee, began producing TATB in 2014. DOD has qualified the Holston-produced TATB for use in conventional weapons but NNSA has not yet qualified it for use in nuclear weapons because the material properties of the formulated material are not yet up to NNSA standards, according to NNSA documentation. available binding ingredient to create a plastic bonded explosive. Each explosive formulation is designed for a specific application. The performance requirements for explosive formulations in nuclear weapons are more stringent than those for conventional weapons for DOD formulations to ensure both performance and safety. Explosives scientists commonly use the term “recipe” to describe the ingredients and many variables in the process—such as the temperature, mixing speed, or container size—used to make explosive molecules and formulations that meet specific performance requirements. In December 2018, NNSA completed the last production unit for the W76- 1 LEP, marking the completion of warhead production for the first LEP in which NNSA undertook full-scale design activities for weapon systems since 1982. Five other LEPs and stockpile modernization efforts were ongoing as of January 2019, as shown in table 1. As we concluded in an April 2017 report, this is a particularly challenging time for NNSA, as the agency plans to simultaneously execute LEPs and modernization efforts along with major construction projects, such as efforts to modernize NNSA’s uranium and plutonium manufacturing capabilities. NNSA’s nuclear security enterprise consists of eight government-owned sites managed and operated by seven contractors. Five of these sites conduct explosives activities: Livermore, Los Alamos, Sandia, Pantex, and Nevada. In addition to these sites, NNSA relies on several third-party suppliers of explosive materials and related equipment. The largest of these is Holston, which is a government-owned, contractor-operated facility that primarily produces explosives for DOD. Holston is NNSA’s sole supplier of explosives used in main charges. The infrastructure that supports NNSA’s explosives activities consists of thousands of real property assets, which are to be tracked in FIMS. The database is managed for NNSA missions by its Office of Safety, Infrastructure and Operations. According to NNSA officials and DOE documents, FIMS helps managers understand the current state of NNSA infrastructure and inform infrastructure modernization funding decisions. We have previously reported on concerns about the accuracy of the FIMS database with respect to certain data fields that were not assessed as part of this review. DOE has taken sufficient steps to address recommendations we have previously made about FIMS. Workforce levels for explosives activities have generally increased in recent years, which contractor representatives attribute to the increase in workload because of LEP and modernization efforts. Table 2 shows NNSA contractor representatives’ estimates for actual full-time equivalents (FTE) and percentages of FTEs engaged in explosives activities at each of the five sites over the last 5 fiscal years. The Government Performance and Results Act of 1993 as amended (GPRA) requires, among other things, that federal agencies develop strategic plans. The Office of Management and Budget (OMB) provides guidance to federal executive branch agencies on how to prepare their agency-wide strategic plans in accordance with GPRA requirements, as updated and expanded by the GPRA Modernization Act of 2010. We have reported that these requirements also can serve as leading practices for strategic planning at lower levels within federal agencies, such as planning for individual divisions, programs, or initiatives. In addition, we have reported in the past on federal agencies’ strategic planning efforts and have identified additional useful practices to enhance agencies’ strategic plans. The leading practices in federal strategic planning that we selected are: (1) involving stakeholders, such as federal agencies, state governments, or others, in defining the mission and desired outcomes, which helps ensure that their expectations and interests are met and that resources and efforts are targeted at the program’s highest priorities; (2) assessing external and internal forces, which helps managers anticipate future challenges and make adjustments before potential problems become crises; and (3) covering at least a 4-year period while making adjustments as needed to reflect the operating environment. Further, our past work has shown that effective strategic plans should include several specific elements. These elements include: (1) a comprehensive mission statement that explains why a program exists and what it does; (2) long-term goals and objectives that specify how an agency will carry out its mission and explain what results are expected from the program; (3) strategies to achieve the goals and objectives that are specific enough to allow an agency to assess whether the strategies will help achieve those goals; (4) a description of how performance measures will be used to assess progress toward long-term goals; and (5) the identification of external factors that could significantly affect achievement of the strategic goals. NNSA’s five sites involved in explosives conduct interdependent activities to design and produce explosives and about 100 different nuclear weapon components that contain explosive materials. Each of the sites assumes primary responsibility for certain explosives activities—such as Livermore conducts design, research, and development of new IHE main charge formulations; Pantex produces all main charges; Los Alamos conducts design and production of main charge detonators as well as explosives research and development; Sandia conducts design and production of nonnuclear explosive components; and Nevada conducts large experimental explosive shots to support design activities. However, most of these activities require the participation of multiple sites. The following examples illustrate some of the collaborative, interdependent activities that NNSA’s sites and their suppliers undertake to design and produce explosive components found in nuclear weapons. Main charge for the W80-4 LEP. Livermore manages design activities for the W80-4 LEP, including for its main charge. The main charge used in the W80-4 warhead will consist of newly synthesized TATB, formulated with a new binding ingredient, according to contractor representatives. As NNSA officials and contractor representatives explained during our site visits to Livermore and Pantex, Livermore scientists redeveloped the specific process for TATB synthesis and formulation that is being used in the W80-4 LEP, first in small test batches and then in larger amounts. Next, Livermore sent its specifications for synthesis and formulation to Holston, which has produced successively larger batches. As the design and cost study phase of the W80-4 LEP continues, Livermore and Pantex continue to receive and test these batches of formulated explosive and work with Holston to ensure that production lots meet NNSA specifications. In coordination with Livermore, Pantex will press and machine the finished main charges for the W80-4 when the LEP reaches the production phase. Pantex will receive formulated TATB from Holston and conduct its own tests to ensure the quality of the initial production lots and pressing, machining, and subassembly processes. Detonators. The design and production of main charge detonators involves several NNSA sites and their suppliers. According to contractor representatives, Livermore and Los Alamos share the responsibility for designing the main charge detonators, and Los Alamos will produce all the detonators. As part of production, Los Alamos reprocesses the PETN used in detonators from a stockpile of DOD-grade material purchased 30 years ago. Other detonator parts come from third-party suppliers and from NNSA’s Kansas City National Security Campus, another NNSA site that does not have a role in designing or producing explosives, according to contractor representatives. Los Alamos produces and tests completed detonators and then sends them to Pantex for weapon assembly, according to contractor representatives. Spin rocket motors. Sandia plays the primary role in designing spin rocket motors. Spin rocket motors use pyrotechnics and propellants and are a key component in the B61 and B83 bombs. Contractor representatives at Sandia said that they supply the explosives to third- party suppliers, who produce the motors. The completed spin rocket motors are sent to Sandia for inspection and testing, and after Sandia approves the components, they are shipped to Pantex for weapon assembly, according to contractor representatives. Component manufacturing research. In addition to designing and producing components for LEPs and modernization efforts, NNSA sites also collaborate on other explosives research and development programs, such as on component manufacturing processes. For example, Los Alamos, Livermore, Sandia, and Pantex are collaborating on additive manufacturing processes for explosives. Additive manufacturing differs from traditional manufacturing in that it builds components by depositing material rather than by cutting material away during machining. This research effort seeks to introduce additive manufacturing into the explosives production process, producing explosive parts with highly complex geometries while meeting NNSA’s safety and performance requirements, according to a contractor representative. In May 2018, according to NNSA documentation, NNSA began implementing a new enterprise-wide approach to improve the management and coordination of explosives activities across its sites. In the past, each program that used explosives—such as an LEP or a research and development program—developed or procured them independently of other programs, without formal coordination to ensure each program’s awareness of other programs’ requirements or time frames. Under the new enterprise-wide approach, NNSA has taken several steps to centralize management at an enterprise level and to coordinate explosives activities across its sites. Specifically: In May 2018, NNSA established the Energetic Materials Enterprise Manager (enterprise manager) position to help coordinate NNSA’s explosives activities. The agency issued a May 2018 memorandum formally establishing the position, signed by the Acting Deputy Administrator for Defense Programs. The memorandum specified that the enterprise manager should encourage collaboration among the sites and programs that conduct explosives activities. In September 2018, the enterprise manager established NNSA’s Energetics Coordinating Committee (coordinating committee) to identify coordination challenges across the enterprise and emerging needs for critical explosive materials, among other purposes. The coordinating committee is composed of NNSA officials and contractor representatives from NNSA’s sites, is chaired by the enterprise manager, and is expected to meet at least once a year. According to NNSA documents, the coordinating committee met twice in 2018 and identified a number of future actions requiring input from the sites, such as defining future needs associated with the production of main charge explosive materials. In December 2018, NNSA issued the strategic plan for energetic materials. This strategic plan states that it will help NNSA organize its efforts to meet weapon delivery schedules for the overall energetics community. Prior to the strategic plan’s final issuance, the enterprise manager provided a draft to coordinating committee members to solicit their comments. However, more recent action taken by NNSA indicates that the enterprise approach to managing high explosives is continuing to evolve. First, according to NNSA officials, in 2019 NNSA is planning to reorganize the Office of Defense Programs—which is responsible for all stockpile activities. This reorganization could affect the approach to managing high explosives activities. Specifically, officials said part of this reorganization is the creation of a new organization for production activities, which is expected to divide production activities into several groups oriented around different weapons components. It is currently unclear under which production group explosives activities will fall because there are production activities associated with explosives for both nuclear and nonnuclear components, according to NNSA officials. Second, in December 2018, NNSA officials indicated that they are considering elevating high explosives to a “strategic material” and managing it more similarly to NNSA’s existing approach for managing other strategic materials, such as plutonium. NNSA’s strategic materials managers are overseen by a senior NNSA official and appointed to manage each material as a program, with a budget and dedicated staff, according to NNSA documentation. NNSA does not consider the high explosives enterprise manager to be managing a program; therefore, the enterprise manager does not have an explosives budget or dedicated staff, according to NNSA officials. NNSA officials said they anticipate issuing an analysis of alternatives study in spring 2019 that will contain a recommendation to the NNSA Administrator on how explosives activities should be managed going forward, which could reflect a shift toward managing high explosives as a strategic material. NNSA officials and contractor representatives have identified a number of challenges related to NNSA’s supply of explosive materials, infrastructure, and staff recruitment and training. First, NNSA’s supply of certain highly specialized explosive materials is dwindling, and NNSA officials and contractor representatives stated that it is challenging to reproduce or procure these materials. Second, officials and contractor representatives identified infrastructure that is aging and deteriorating, inaccurate information on that infrastructure, and storage areas filled to near capacity as challenges. Finally, according to NNSA contractor representatives, there are difficulties in recruiting and training qualified staff. NNSA has taken some actions to address these challenges, such as starting to recreate “recipes” for specialized materials and modernize aging infrastructure, according to NNSA officials and contractor representatives. However, taking additional steps to improve the quality of information about its explosives infrastructure would give the agency more reasonable assurance that officials, contractor representatives, and the enterprise manager have the quality information necessary to support management decisions. NNSA’s Challenges Producing Fogbank The National Nuclear Security Administration (NNSA) has had challenges in the past producing materials other than explosives that are essential to the successful operation of nuclear weapons. In 2000, NNSA began a life extension program (LEP) to replace or modernize components for W76 warheads, which are delivered by submarine-launched ballistic missile systems. NNSA had to delay production of the refurbished warheads when it encountered problems in manufacturing an important material that NNSA refers to as “Fogbank.” In March 2009, we reported that NNSA had lost knowledge of how to manufacture the material because it had kept few records of the process when the material was made in the 1980s, and almost all staff with expertise on production had retired or left the agency, leaving the production process for Fogbank dormant for about 25 years. As we reported, NNSA’s loss of the technical knowledge and expertise to manufacture Fogbank resulted in a 1-year delay in the W76-1 LEP and an unexpected cost increase of nearly $70 million. According to NNSA officials, production challenges with Fogbank have since been resolved, and the last production unit for the W76-1 LEP was completed in December 2018. NNSA’s supply of certain highly specialized explosive materials is dwindling. These materials have specific chemical and physical characteristics that fulfill precise performance requirements in nuclear weapons, such as detonation within nanoseconds, according to contractor representatives. One such material, titanium sub-hydride potassium perchlorate (THKP), is used in actuators to open valves, among other things, according to contractor representatives. TATB, the IHE molecule used in main charges, is another such material, according to contractor representatives. In some cases, contractor representatives said that only one container or production lot of specialized material was ever produced that met NNSA’s specifications. The inventories of these materials have dwindled as ongoing LEPs, modernization efforts, and research and development activities draw on them. For example, only a small container of THKP remains. Additional inventory will be required to meet the needs of four of the five ongoing LEPs and modernization efforts, as well as for any future needs, according to contractor representatives from Sandia. Similarly, although Pantex has a stockpile of legacy TATB for the B61-12 LEP, contractor representatives said that new material will be needed to meet the requirements of planned and future LEPs and modernization efforts. NNSA officials stated that reproducing and procuring these highly specialized materials presents challenges for the agency. According to NNSA documents and officials, lost recipes and a fragile supplier base contribute to these challenges (see sidebar). Some specialized materials were created decades ago, and the knowledge base to successfully produce them is now gone. According to NNSA documents, technical knowledge of material production processes can be lost when long intervals occur between production orders. In some cases, processes were not well documented or were infrequently practiced and proven. Thus, NNSA sites must spend considerable effort to recreate the recipes and techniques for producing these materials. Sandia representatives explained that sometimes a single company or even an individual created these materials and has since ceased production or is now deceased. For example, THKP was produced exclusively for Sandia by DOE’s Mound Site near Dayton, Ohio, which closed in 1994. The THKP production process was designed by an individual at the Mound Site who is now deceased. In some cases, according to contractor representatives, a single container of explosives (or a single production lot) met anticipated future needs for quality and quantity when it was originally produced, so production was discontinued. Contractor representatives explained that replicating the material exactly is nearly impossible because of the large number of variables, such as the mixing speed and temperature, that must be controlled for, even if the ingredients are identical to those used many years ago, which is not often the case. To address the challenge of lost recipes, Los Alamos, Sandia, Livermore, and Pantex are all working to reproduce materials with performance and physical properties similar to those of legacy materials and prepare for their full-scale production. For example, Livermore scientists said they are conducting research to synthesize new TATB that is uniquely suited to NNSA’s needs. According to NNSA contractor representatives, the synthesis process will be refined until it can be replicated by Holston for the W80-4 LEP. Additionally, Los Alamos scientists are researching the formulation process with legacy TATB for the B61-12 main charges. The chemical formulation of binder material used in the past has slightly changed, affecting the structural strength of formulated TATB. Without the proper strength, this formulated explosive cannot be pressed effectively, according to contractor representatives. Sandia is also working to re-establish the THKP production process. NNSA is also working to address the challenge of lost recipes by developing a comprehensive master list for explosive materials. The list tracks information such as the suppliers involved and specific production challenges. According to NNSA and contractor officials, collecting and sharing such information across the sites related to explosive production processes, specifications, and performance will help prevent lost recipes in the future. Even if the sites can replicate lost recipes for explosive materials, NNSA’s supplier base for those materials is fragile. As previously reported and according to NNSA documentation, finding suppliers willing and able to provide required parts and materials can be difficult. Contractor representatives told us that this difficulty arises because of the small quantities of explosive parts and materials NNSA procures, the irregular nature of NNSA’s procurements, and the agency’s exacting performance requirements. For example, neutron generators contain explosive parts that Sandia orders irregularly, in batches numbering only in the hundreds. These parts have such exacting requirements for size and timing that they are hand-made under microscopes. Sandia contractor representatives explained that sometimes the laboratory’s part and material orders may represent only 1 to 3 percent of a company’s total production. To address this challenge, NNSA is working to purchase materials more consistently to ensure that companies can rely on NNSA as a steady customer and be comfortable working to meet NNSA’s exacting requirements. Contractor representatives said that ensuring consistency in production can help maintain the expertise needed to avoid having to reconstitute a specialized process, which can be costly. For example, the effort to restart TATB synthesis and formulation cost approximately $13 million and added 3-1/2 years to the original TATB production schedule, according to Los Alamos contractor representatives. Contractor representatives at Pantex and Los Alamos said that they plan to support continuous production of synthesized TATB and formulated explosives at Holston in the future to avoid delays in restarting production (see sidebar). A Fragile Supplier Base for Other Material The National Nuclear Security Administration (NNSA) has identified challenges with a fragile supplier base for other specialized materials that are used in explosives-related experiments and research. For example, Los Alamos National Laboratory (Los Alamos) in New Mexico requires highly specialized test vessels to conduct essential nuclear weapons research. Specifically, Los Alamos’s Dual-Axis Radiographic Hydrodynamic Test Facility (DARHT) uses X-ray machines to record three-dimensional interior images of mock nuclear materials that are imploded using explosives. The exploding components are contained in steel vessels. This facility is unique because it is the world’s most powerful X-ray machine for analysis of these implosions (called hydrodynamic tests). The vessels used at DARHT are made of specialized steel that does not need to be heat-treated during repairs, allowing the laboratory to easily repair them after explosive testing. There is currently a small supplier base (domestic and international) for manufacturing these vessels. Los Alamos contractor representatives are concerned with vendor availability, capability, and willingness to produce vessels because of the small number the laboratory has purchased in the past—they currently have seven operational vessels. Also, contractor representatives said they are concerned that the workforce which knows how to create this specialized steel is nearing retirement. To help ensure a continued future supply of the vessels, Los Alamos is working with Lawrence Livermore National Laboratory in California and the Nevada National Security Site, which use similar vessels, to develop a multi-year procurement plan to encourage suppliers to continue to produce the specialized steel used in their manufacture. NNSA supplier challenges are complicated further when a supplier is foreign or there is only one domestic supplier. According to NNSA documentation, using a foreign supplier may leave NNSA vulnerable to a potential national security risk. Even when the only supplier is domestic, single-point failure is a concern should that supplier delay or cease production, according to contractor representatives. NNSA officials provided an example involving Holston, NNSA’s sole supplier of TATB. According to NNSA officials and contractor representatives, Holston also serves DOD customers that order far larger quantities of explosives, and Holston is required to prioritize those customers’ orders using DOD procurement priority ratings, which may mean that NNSA orders are delayed. For example, Livermore placed an order for the W80-4 main charge explosives at Holston that was to be fulfilled by March 2019, but that order was delayed while the plant worked to finish a DOD order with a higher-priority rating. In addition to this delay, Livermore’s order will be further delayed because Holston had an explosive incident in January 2019 and ceased operations for 3 weeks, according to Livermore and DOD contractor representatives. As a result of both these delays, the W80-4 LEP will have to postpone a hydrodynamic test and other studies, complicating an already tight design and development schedule. This will delay the W80-4 LEP at least 2 months, according to Livermore officials. To minimize the potential for future production delays at Holston, NNSA is working to elevate the priority of all its orders for explosives. Some DOD nuclear weapon delivery platforms have the highest-priority DOD rating, and NNSA officials said they have received permission from DOD to apply this rating to the DOE explosives orders for the nuclear warheads associated with those delivery platforms, including explosive orders for the B61-12 LEP. NNSA officials said they cannot currently use the highest-priority rating for orders associated with the W80-4 LEP because the delivery platform for that LEP does not have the highest-priority rating. NNSA officials are working with DOD and DOE attorneys to obtain permission for using DOD’s highest-priority rating. A contractor representative at Livermore said that in addition to NNSA’s efforts, the Air Force is working separately to obtain permission to use the highest- priority rating for this delivery platform. If the Air Force is successful, NNSA could use that delivery platform’s new high-priority rating for its W80-4 LEP orders. The Livermore contractor representative said that they believe the Air Force will receive permission to use the highest- priority rating before NNSA does. In situations where a supplier cannot or will not produce a specialized material or related component, NNSA is exploring options for producing those materials or components itself. NNSA officials said that they are conducting an analysis of alternatives to meet synthesis, formulation, and production requirements to be completed by the spring of 2019. The analysis will include an option for in-house production of TATB at Pantex. NNSA documentation indicates that Pantex could independently produce the TATB needed for current and future LEPs and modernization efforts with a substantial investment, exact figures for which may be reported upon completion of the analysis of alternatives. Similarly, contractor representatives from Sandia said that in the absence of qualified suppliers, they are working to produce explosive materials, such as THKP as discussed above. NNSA has also identified challenges with its explosives infrastructure, infrastructure data, and workforce. Specifically, NNSA’s infrastructure is aging and deteriorating, some infrastructure data are inaccurate, and some storage areas are near capacity. In addition, recruiting and training qualified staff have presented a challenge to NNSA. As we have previously reported, these challenges are shared across the nuclear security enterprise and are not confined to explosives activities. NNSA is taking several actions to address these challenges, as described below, but data inaccuracies remain related to NNSA’s explosives-related assets. According to NNSA documentation, no mission risk is greater than the state of the agency’s aging infrastructure. The NNSA 2019 Master Asset Plan states that 40 percent of the explosives infrastructure of NNSA’s sites is insufficient to meet mission needs, which can lead to contamination of explosive products or limit the use of facilities. Contractor representatives told us that such contamination has occurred. For example, Pantex contractor representatives said that batches of explosives have been contaminated in its main formulation building by rust falling from the rafters and grass blowing through cracks in the walls. Similarly, Los Alamos contractor representatives said that detonator subassemblies have been rejected at the laboratory because of contamination from foreign debris, such as dust particles that enter through cracks in exterior doors. In addition, older facilities were not built to modern safety standards and pose risks to explosives activities and employees, according to contractor representatives and NNSA documents. At Los Alamos, the design of several older facilities is insufficient to meet current needs, which negatively affects both productivity and safety. For example, the Los Alamos’s High Explosives Chemistry Laboratory is a 1950s era building that is difficult to adapt to modern instrumentation, and electrical and other system failures cause approximately 20 percent downtime, according to contractor representatives. This building is also under a state of continuous limited operation because the laboratory must work under a decreased net explosive limit to keep employees safe while handling explosive materials because the facility lacks adequate blowout walls, according to contractor representatives. Contractor representatives at Los Alamos said that the decreased explosive limits in this facility have hampered their productivity levels. Contractor representatives at Pantex stated that the intrusion of water in key facilities poses electrocution risks, can damage expensive equipment, and can affect production because of downtime when explosives activities must be suspended because of severe weather. Further, we observed facilities at Pantex with water leaks in the roof and floor; some of these facilities house expensive equipment that must be stored under plastic sheeting to prevent water damage. One such facility is Pantex’s Analytics and Chemistry Laboratory, built in 1943 and shown in figures 4 and 5. NNSA and its sites have taken some actions to address this infrastructure challenge. For example, Los Alamos plans to replace its High Explosives Chemistry Laboratory by 2026, and Pantex recently constructed a new building to replace an aging pressing facility and has plans to begin construction on a new analytical laboratory and a formulation building in the 2020s. NNSA documentation states that the new pressing facility will improve operational safety and security thereby enhancing the quality and efficiency of operations. Pantex’s planned analytical laboratory and formulation buildings, however, will not be completed in time to support the currently scheduled B61-12 LEP and W88 alteration modernization effort. Further, according to NNSA officials and contractor representatives, site infrastructure modernization plans are budget dependent and funding for infrastructure modernization efforts is not always certain. Contractor representatives told us and we observed during site visits that some of the data on explosives-related assets in the FIMS real property database were inaccurate and out of date. NNSA policy and the FIMS user’s guide state that NNSA and sites should review and update the capabilities, or programmatic mission(s) associated with each asset, such as being explosives-related, every 5 years, or more frequently if mission requirements change or there are changes in an asset’s physical condition or use. However, 8 of the 22 randomly selected assets from the four sites that we observed contained data in FIMS that were inaccurate because either the information on an asset was out of date or the asset should never have been listed as explosives-related. Some contractor representatives told us that they did not understand why some of their sites’ assets had been characterized as primary assets related to the high explosives mission. For example, an inert storage closet at Pantex and a tool shed at Livermore were labeled as primary explosives-related assets, but according to contractor representatives, they can no longer be used to store explosives because they do not meet appropriate safety standards. Figure 6 illustrates the inert storage at Pantex, which officials said had not been used for any explosives operations for at least 20 years, despite “explosives storage” labeling on the door, but was still characterized as a primary explosives-related asset. However, according to NNSA officials, NNSA was, at the time of our review, in the process of revising guidance on how to associate capabilities with assets. The contractor representatives may not have been aware of the initial guidance the asset was characterized under or of the change underway at the time of our site visit. In other cases, contractor representatives told us that the asset name did not indicate its current use. For example, FIMS data on explosives-related assets at Los Alamos has a “plastics building” that had not been used for manufacturing and assembling plastics for 20 years. Although it currently houses explosives-related work, the building’s name in FIMS had not been updated. Additionally, Los Alamos’s FIMS data indicated that the site had a “day room” that to contractor representatives’ knowledge had never been used for any explosives activities although its purpose has changed over time. We found additional inaccuracies related to various measures of explosives-related assets reported in FIMS. For example, we found at least 94 erroneous entries on the gross square footage of the 1,266 assets identified as having some type of explosives-related capability. For example, FIMS data indicated that a road at Livermore, a bunker at Sandia, and an asset named “recreational/fitness” at Pantex were 3, 1, and 2 gross square feet, respectively. The data listed replacement values of at least $1 million for each of these assets. Los Alamos’s data contained similar errors, such as electrical cables recorded as measuring zero square feet. NNSA officials and contractor representatives identified potential causes for inaccuracies in the FIMS data. For example, contractor representatives who work on explosives activities do not enter explosives- related asset information in FIMS, according to NNSA officials and contractor representatives. Instead, FIMS administrators, who manage information on infrastructure across NNSA sites, said they update FIMS using information that subject matter experts or building managers provide to them, typically in an annual data call. FIMS administrators may therefore not be aware of information that is dated or otherwise incorrect for explosives-related assets. In addition, entering information in certain data fields in FIMS was difficult for assets that were not buildings, according to one FIMS administrator. For example, piping and other utilities may be replaced or updated in sections over time, and it can be difficult to know which date to record for age in FIMS. Because our review included only a limited sample of explosives-related assets, we could not determine the full extent of the FIMS data inaccuracies. NNSA managers use data from FIMS for planning purposes on infrastructure modernization decisions. According to NNSA officials, data from FIMS feeds into other databases that are used to inform infrastructure funding decisions, such as developing the Integrated Priority List that helps NNSA determine the most critical infrastructure modernization projects. While NNSA relies on these data to make planning and funding decisions, our observations of explosives-related assets shows that these data may not be useful in informing the agencies’ infrastructure modernization decisions. Federal internal control standards state that managers should make decisions using quality information that is appropriate, current, complete, accurate, accessible, and provided on a timely basis. By taking steps to improve the accuracy of FIMS data— such as by reviewing and updating information about associations of assets with their primary and secondary programmatic missions, ensuring that those who provide asset information to FIMS administrators understand the data they need to provide, and clarifying how to accurately enter information in FIMS for assets that are not buildings— NNSA would have more reasonable assurance that officials, contractor representatives, and the enterprise manager have the quality information necessary to support management decisions on explosives-related activities. DOE’s requirements for explosives storage limit the amount and type of explosives that can be stored in a single location, since certain explosives may react when stored together. Explosives must be properly stored throughout their life cycles, from the synthesis of raw explosives to their use in weapon assembly or testing. According to a senior NNSA official and site contractor representatives, some sites are running out of space where they can safely store explosives. As contractor representatives from Pantex told us and we observed on our site visit, bunkers for storing explosives are filled to or near capacity, especially for storage in high- security areas. According to contractor representatives, this is problematic because Pantex has the greatest need of all NNSA sites for explosives storage because of its role in producing explosives, receiving and holding explosive parts from across the nuclear security enterprise prior to weapon assembly, and assembling and disassembling weapons. Contractor representatives from Los Alamos also voiced concern about being near their capacity to store detonator cable assemblies and other explosives awaiting shipment for installation in weapons or for testing. NNSA officials and contractor representatives said that they are tracking the shortage of sufficient explosives storage and in some cases have plans to expand current capacity. Los Alamos contractor representatives also said that they are moving forward with constructing a small staging facility that will be collocated with their detonator production facility. It is expected to cost less than $5 million so it will not affect larger line item infrastructure projects. Contractor representatives at Pantex explained that although some storage areas have been identified for replacement, they are, as yet, unfunded projects. In the near term, contractor representatives said that they have other, more pressing infrastructure modernization project needs than explosives storage. They said that they are closely monitoring their storage capacity and expect ongoing modernization efforts to free up some storage space as weapons are assembled. According to NNSA documents and contractor representatives, the contractor workforce at NNSA sites needs to grow to meet the demands of ongoing and future explosives work, but contractors face difficulty recruiting and training qualified new staff to perform this specialized work, which often requires a security clearance. In 2018, Pantex estimated that it needed 211 FTE contractor staff to adequately carry out the site’s explosives activities. However, Pantex contractor representatives indicated that as of November 2018, they had 172 FTEs on board. A major recruitment challenge is competition from industry. Contractor representatives at multiple sites told us that they often compete with large corporations and industries in the local area that offer well-paying jobs for qualified new staff, such as for engineers. For example, site contractor representatives told us that Los Alamos and Sandia compete with Facebook in Albuquerque to attract qualified staff; and Pantex competes with various oil and gas companies in Texas. To address this challenge, contractor representatives from Pantex have recently expanded outreach to local colleges and universities, and NNSA has held job fairs to attract new staff. Lengthy training and clearance processes that are required for specialized explosives work present another challenge. Pantex representatives said recent graduates are required to undergo on-the-job training that can take years before they are ready to safely engage in certain explosives activities. NNSA officials and contractor representatives said that this training challenge is exacerbated by the delays in processing security clearances. NNSA contractor representatives said that some new hires have waited more than a year, and some more than 2 years, to receive clearances to conduct required work or training. In December 2017, we identified delays in obtaining personnel security clearances as a government-wide risk. We also added this issue to our March 2019 High-Risk List. To mitigate this challenge, contractor representatives from Pantex said that they are hiring students before they finish college so that security clearances can be granted by the time students are ready to begin their first day on the job or at least closer to that time. Los Alamos has decided to hire and train individuals without clearances, who must wear red vests and be escorted at all times while their clearances are finalized. We observed numerous workers in this temporary and escorted status during our site visit. Contractor representatives at Livermore said that they also use escorts for new staff without clearances. However, contractor representatives said that requiring additional staff as escorts is costly, can decrease productivity, and has safety impacts because additional staff must be present during activities involving high explosives. NNSA’s 2018 strategic plan for energetic materials describes some identified explosives-related challenges discussed above, as well as further actions to address these challenges, but does not describe other challenges NNSA officials and contractor representatives identified. This strategic plan incorporates some leading practices for strategic planning. However, some of the strategic plan’s elements have not been fully developed consistent with selected leading practices for strategic planning. The strategic plan for energetic materials, which includes comments from coordinating committee members, describes some of the challenges that NNSA officials and contractor representatives identified in conducting explosives activities, which we discussed above. Specifically, it describes some challenges related to the supply of explosive materials and to infrastructure modernization, including the following: Supply of explosive materials. The strategic plan describes both the supply of explosive materials as well as the supply of pre-cursor ingredients as a challenge facing NNSA. The strategic plan also identifies a number of actions NNSA is taking to bolster the supply chain, such as re-establishing the capability to manufacture THKP. Infrastructure modernization. The strategic plan notes that explosives-related “facilities require recapitalization to support LEP activities, improve efficiencies, reduce downtime, and maintain baseline capabilities.” It also identifies several interrelated actions NNSA is taking to address infrastructure challenges, such as re- purposing some facilities and eliminating others that are inadequate, too costly to maintain, or no longer needed. In addition, the strategic plan describes the challenge of adequate storage for explosives and includes actions to annually monitor and track storage conditions at the sites as well as provide long-term, low-temperature, moisture-free storage for explosives. However, based on our review of the strategic plan, it does not discuss three of the challenges that NNSA officials and contractor representatives had identified: the quality of data on infrastructure information, workforce levels, and safety. First, the data quality challenge related to infrastructure information, such as inaccuracies in FIMS, is not discussed in the strategic plan, although NNSA officials and contractor representatives we interviewed identified it as a challenge that may affect its planning and decision-making related to explosives activities. Second, the strategic plan does not discuss workforce challenges. While the strategic plan states that NNSA “recognize(s) that staffing is an important aspect for supporting energetics, it assumes that ongoing efforts across the nuclear security enterprise related to workforce are successful.” Since the enterprise manager does not track workforce levels across the enterprise, as previously noted, it is unclear how NNSA can determine if its contractors’ workforce efforts across the enterprise are successful and whether levels are adequate to achieve the goals of the strategic plan for explosives over time. Third, outside of infrastructure improvement, the strategic plan also does not directly discuss the challenge of safety, although it affects all explosives activities and challenges that NNSA has identified. Because of the inherent danger of explosives activities, safety is important, and even when protocols are followed, unintended events can occur that affect human safety—as illustrated by a safety incident last year. The incident occurred at a Los Alamos facility in April 2018 when a small explosive pellet deflagrated during pressing, causing two people to incur short-term hearing loss. One of those people was an escort and was only required to be present because of the delay in security clearance processing, a challenge discussed above. According to a December 2018 Los Alamos document, pressing operations had resumed at the facility. Although the cause of the incident is still unclear, it provided an opportunity to make safety improvements in the facility at Los Alamos, according to contractor representatives. According to a Los Alamos document about the incident, a key lesson learned was that safety records like maintenance logs, blast calculations, and materials safety testing results need to be archived and readily accessible to staff before the start of any work activities. The inherent challenge of safety in explosives and key lessons learned, such as this one, are not discussed in the strategic plan. NNSA officials said that they are planning to revise the strategic plan for energetic materials in 2020 but did not state that they would include the challenges of data quality, workforce, or safety. All three of these challenges may impede NNSA’s ability to achieve the goals described in the plan for explosives activities. We have previously identified selected leading practices in strategic planning. These practices specify that agencies should define strategies that address management challenges that threaten an agency’s ability to meet its long-term strategic goals. As NNSA revises its strategic plan for energetic materials, by discussing the data, workforce, and safety challenges it faces and the actions it plans to address them, as appropriate, or documenting the rationale for why the challenges were not included, NNSA would have better assurance that its strategies address these challenges. In developing its strategic plan for energetic materials, NNSA followed several key leading practices in strategic planning that we have identified in our past work, including the following: Involving stakeholders, such as federal agencies, state governments, or others, in defining the mission and desired outcomes helps ensure that their expectations and interests are met and that resources and efforts are targeted at the program’s highest priorities. When developing the strategic plan, NNSA shared a draft with members of the coordinating committee and incorporated their comments to ensure that their interests and expectations were met. Assessing external and internal forces helps managers anticipate future challenges and make adjustments before potential problems become crises. For example, external forces (e.g., emerging technological trends and new statutory requirements) and internal forces (e.g., culture, management practices, and business processes) may influence the program’s ability to achieve its goals. When developing the strategic plan, NNSA officials and coordinating committee members considered external and internal forces. For example, the officials and members discussed the availability of explosives from external suppliers, such as Holston, compared to the potential costs or challenges related to internal NNSA production of explosives. Covering at least a 4-year period, while making adjustments as needed to reflect significant changes to the operating environment, is also a key strategic planning practice. The strategic plan covers more than 4 years of explosives activities. For example, there is a performance goal to re-establish a reliable THKP supply by 2024. In addition, NNSA officials have discussed their intention to update the plan as their operating environment changes. Our past work has also shown that effective strategic plans should include specific elements. We reviewed NNSA’s Defense Programs Strategic Plan for Energetic Materials and found that the strategic plan includes most of these elements, but we also found that some of the strategic plan’s elements have not been fully developed. Specifically: Mission statement. According to leading federal strategic planning practices, a comprehensive mission statement should explain why a program exists and what it does. The strategic plan does not clearly identify a mission statement but includes an overarching “strategy to ensure the availability of energetic materials and products for the stockpile.” When asked to identify the energetics mission statement, the two contractor representatives who led the development of the strategic plan told us that they consider this “strategy” to be the energetics mission. However, a strategy cannot be a mission, since a strategy is how a mission may be achieved. Long-term strategic goals and objectives, strategies, and performance goals. There are several interrelated elements on long- term strategic goals, objectives, strategies, and performance goals, according to leading strategic planning practices. These include that long-term strategic goals and objectives should specify how an agency will carry out its mission and explain what results are expected from the program. The strategic plan includes four long-term strategic goals for meeting its mission, some strategies for achieving its goals, and some performance goals to assess progress related to ensuring the availability of explosives. They are also logically linked to each other. For example, the strategic plan’s goal to sustain and modernize the energetics infrastructure relates to the strategic plan’s strategy to eliminate facilities that are inadequate, too costly to maintain, or no longer needed. However, we found that responsibilities for achieving the strategic plan’s four goals are not clearly assigned within NNSA, and the four goals are not consistently quantifiable. For example, the third goal is to “manage the energetics supply chain,” but the strategic plan does not specify who is responsible for achieving this goal within NNSA. Further, this long-term strategic goal is not quantifiable because it describes a general process and does not define the expected results, which may make it difficult for NNSA to assess progress in meeting the goal. According to leading strategic planning practices, strategies should be specific enough to allow an assessment of whether they will help achieve those goals, such as by describing the resources needed, including the staff responsible to achieve a program’s goals and objectives. We found that the strategic plan contains several strategies for achieving goals, but some of them are not specific enough to clearly identify the types of resources required, such as the parties responsible for achieving them. For example, under the goal of managing the energetics supply chain, there is a strategy to “plan, track and assess the energetics strategic posture,” but the strategic plan does not specify what is meant by the energetics strategic posture or who is responsible for undertaking these actions. This strategy is also limited because it does not describe the resources needed to achieve the broader goal. According to leading strategic planning practices, performance goals should be used to assess progress toward long-term goals and should include (1) the specific activities within the program that will be assessed for performance and (2) the level of performance to be achieved for each measure. We found that the strategic plan has 50 performance goals, most of which were quantifiable—or able to be assessed for performance or progress. However, some were not quantifiable, such as to “enhance or advance energetics formulations for additive manufacturing.” This performance goal also does not set milestones, such as a time frame for completion, or staff assigned to achieve it, contrary to leading strategic planning practices. Further, the level of performance for some goals was not fully developed. For example, the performance goal “to reduce substandard mission- critical facilities below 10 percent” does not clarify whether the goal is to reduce the current number of inadequate and substandard facilities by 10 percent (a change of about 50 facilities) or reduce the total number of inadequate and substandard facilities to be less than 10 percent of all facilities (a change of about 500 facilities). This performance goal also does not set time frames for measuring performance or list responsible parties associated with it. Another performance goal that was not fully developed is to “manage the energetics supply chain,” which falls under the long-term strategic goal of “sustaining and modernizing the infrastructure.” In addition, this performance goal is identical to a long-term strategic goal in the strategic plan titled “manage the energetics supply chain.” A performance goal should not replicate a strategic goal, since long- term strategic goals are broader in nature than performance goals. Moreover, this particular performance goal is not quantifiable, does not set a time frame for completion, and does not list a responsible party to carry out specific activities to achieve the goal. External factors. According to leading strategic planning practices, external factors that could significantly affect achievement of the strategic goals, such as economic trends or actions by Congress, state and federal agencies, or other entities, should be identified. The strategic plan identifies some external factors that could significantly affect the achievement of strategic goals. Specifically, the strategic plan notes that DOD’s demand for explosives from Holston could affect NNSA’s ability to achieve its goals. According to NNSA officials and documents, DOD’s demand for explosives is increasing, and Holston is already struggling to meet DOD’s needs. According to Holston contractor representatives, DOD is expanding Holston’s production capabilities for HMX, research development explosive (RDX), and insensitive munitions explosive (IMX) which when completed will relieve pressure on TATB production. In addition, the strategic plan identifies challenges to its supplier base, such as the difficulty of sourcing explosive materials from non-U.S. suppliers and that the small size of NNSA’s orders provides limited economic incentive for commercial vendors. However, the strategic plan does not identify other external factors that could significantly affect the achievement of strategic goals, such as actions taken or not taken by Congress. Specifically, modernizing the explosives infrastructure will require appropriations for the significant capital investment needed, but the uncertainty of future appropriations in a challenging fiscal environment is an external factor not identified in the strategic plan. In addition, the strategic plan does not acknowledge that other NNSA programs may compete for funds or affect infrastructure modernization priorities at a given site. NNSA officials, including the enterprise manager, stated that they are aware that the strategic plan for energetic materials has limitations, such as performance goals that are not specific or are difficult to quantify. NNSA officials said that they released the strategic plan quickly as it was the first of its kind for explosives activities, and they believed the explosives community would receive the most benefit it if it was published as soon as possible, though it was not fully complete. Further, they said that they intend to revise the strategic plan in the next year or so. As NNSA revises its strategic plan for energetic materials, including fully developed elements of an effective plan—such as a clear mission statement and quantifiable performance goals that set time frames for completion and list responsible parties who will carry out specific activities for all strategic goals—would help NNSA make the strategic plan useful in measuring goal achievement and assessing accountability. NNSA is undertaking an extensive, multifaceted effort to sustain and modernize U.S. nuclear weapons, and explosives are essential to the functioning of these weapons. Five NNSA sites conduct a range of interdependent activities to design and produce explosives. NNSA has identified several challenges in carrying out these activities and is taking actions to address them. For example, NNSA officials and contractor representatives identified challenges related to producing highly specialized materials and are working to re-establish their supply. However, NNSA managers may be relying on inaccurate FIMS data on infrastructure related to explosives activities to make modernization decisions, because we found a number of inaccuracies in FIMS data on explosives activities. NNSA officials and contractor representatives identified a few potential causes for these inaccuracies; however, because our review included only a limited sample of explosives-related assets, we could not determine the full extent of the FIMS data inaccuracies. According to NNSA officials, NNSA has taken some initial steps to revise guidance, which we find encouraging as these revisions may help improve accuracy of FIMS data. By taking additional steps to improve the accuracy of FIMS data—such as reviewing and updating information about associations of assets with their primary and secondary programmatic missions, ensuring that those who provide asset information to FIMS administrators understand the data they need to provide, and clarifying how to accurately enter information in FIMS for assets that are not buildings—NNSA would have more reasonable assurance that officials, contractor representatives, and the enterprise manager have the quality information necessary to support management decisions on explosives-related activities. In addition, the strategic plan for energetic materials—which represents a positive step toward managing explosives in a forward-looking, enterprise-wide approach—does not discuss three of the significant challenges that NNSA officials and contractor representatives identified related to explosives activities. NNSA officials said that they are planning to revise the strategic plan in 2020 but did not state that they would incorporate data quality, workforce, or safety challenges. As the agency revises its strategic plan for energetic materials, by discussing these challenges and actions planned to address them, as appropriate, or documenting the rationale for why the challenges were not included, NNSA would have better assurance that it is effectively managing challenges that present risks to achieving its objectives. The strategic plan for energetic materials also does not contain fully developed elements that we have previously reported that effective strategic plans should include, such as a fully developed mission statement and performance goals that are quantifiable, set time frames for completion, and list responsible parties to carry out specific activities. NNSA officials said that they intend to revise the strategic plan in 2020. As NNSA revises its strategic plan for energetic materials, by including fully developed elements of an effective strategic plan—such as a fully developed and clearly identified mission statement and performance goals that are quantifiable, have time frames for completion, and list responsible parties to carry out specific activities for all strategic goals— NNSA would help make the strategic plan more useful in measuring goal achievement and assessing accountability. We are making the following three recommendations to NNSA: NNSA’s Energetic Materials Enterprise Manager and relevant NNSA officials and contractor representatives at NNSA sites should take steps to improve the accuracy of FIMS data related to NNSA’s infrastructure supporting explosives activities. These steps should include reviewing and updating information about associations of assets with primary and secondary explosives missions; ensuring that those who provide asset information to FIMS administrators understand the data they need to provide; and clarifying how to accurately enter information in FIMS for explosives assets that are not buildings. (Recommendation 1) NNSA’s Energetic Materials Enterprise Manager, in consultation with members of NNSA’s Energetics Coordinating Committee, should, as the agency revises its Defense Programs Strategic Plan for Energetic Materials, include discussion of identified challenges related to explosives activities, such as data quality, workforce levels, and safety as well as any actions to address them, as appropriate, or document the rationale for why identified challenges were not included. (Recommendation 2) NNSA’s Energetic Materials Enterprise Manager, in consultation with members of NNSA’s Energetics Coordinating Committee, should, as the agency revises its Defense Programs Strategic Plan for Energetic Materials, include fully developed elements of an effective strategic plan, such as a clearly identified mission statement and performance goals that are quantifiable, set time frames for completion, and list responsible parties to carry out specific activities for all strategic goals. (Recommendation 3) We provided a draft of this report to DOE and NNSA for review and comment. In its written comments, which are summarized below and reproduced in full in appendix I, NNSA concurred with the report’s recommendations and described actions that it intends to take in response to our recommendations. NNSA also provided technical comments, which we considered and incorporated in our report as appropriate. DOE did not comment on our findings and recommendations. In response to our first recommendation, NNSA stated that it recognizes the need to improve infrastructure data consistency and accuracy and intends to complete several actions by March 31, 2020 to improve its infrastructure data. For example, DOE’s Infrastructure Executive Committee is conducting a comprehensive review of the existing 178 data elements in FIMS and has proposed deleting or adjusting 66, which it believes will sharpen its focus on data quality for the remaining data elements. In addition, among other actions, NNSA stated it is implementing the Mission Dependency Index 2.0 initiative, which is expected to provide greater consistency and accuracy on reporting asset capability and determining consequence to mission. In response to our second and third recommendations, NNSA stated that it is planning to revise its Strategic Plan for Energetic Materials by October 31, 2019. NNSA stated that the update to its plan will include a discussion of the identified challenges to explosives activities as well as fully developed elements of an effective strategic plan. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or at bawdena@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Jonathan Gill (Assistant Director), Eric Bachhuber (Analyst in Charge), Natalie Block, Scott Fletcher, Ellen Fried, Rob Grace, and Dennis Mayo made key contributions to this report. Also contributing to this report were Cindy Gilbert, Penney Harwell Caramia, Dan C. Royer, Jeanette Soares, Kiki Theodoropoulos, and Khristi Wilkins.", "summary": "NNSA is responsible for the management and security of the U.S. nuclear stockpile. NNSA has ongoing and planned efforts to modernize nearly all of the weapons in the stockpile, which require new explosive components. The production of some key explosives ceased in the early 1990s, and much of the infrastructure supporting this work is aging, making it expensive and difficult to maintain. The Senate Report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2018 included a provision for GAO to review NNSA's high explosive capabilities specific to nuclear weapons. This report examines (1) explosives activities that NNSA and its sites conduct and how NNSA manages them; (2) challenges NNSA officials and contractor representatives identified in conducting these activities and the extent to which NNSA has taken actions to address them; and (3) the extent to which NNSA's strategic plan for explosives activities describes further actions, if any, to address the challenges identified and follows leading practices for strategic planning. GAO reviewed NNSA documents related to explosives activities, including its strategic plan; compared the plan with leading practices; and interviewed NNSA officials and site representatives. Five National Nuclear Security Administration (NNSA) contractor-operated sites conduct activities to design and produce explosive materials. There are about 100 different nuclear weapon components that contain explosive materials (see figure). Each site assumes primary responsibility for certain activities, but most activities require collaboration by multiple sites, according to NNSA officials and contractor representatives. In 2018, NNSA began adopting a centralized approach to managing these activities and coordinating them across its sites. NNSA officials and contractor representatives identified several challenges related to explosives activities, such as the agency's dwindling supply of explosive materials, aging and deteriorating infrastructure, and difficulty recruiting and training qualified staff. For example, only a single container of one specialized material remains. NNSA officials and contractor representatives indicated that the agency is taking some actions to address these challenges, such as working to replenish the supply of dwindling, highly specialized materials. NNSA's strategic plan for explosives activities addresses some of the challenges agency officials and contractor representatives have identified, and NNSA followed several key leading practices in developing its strategic plan. However, some of the plan's elements have not been fully developed consistent with selected leading practices. For instance, the plan does not include a fully developed mission statement, and some performance goals are not quantifiable. NNSA officials stated that they are aware of the strategic plan's limitations and that they released it quickly to ensure that the explosives community could use it as soon as possible. NNSA officials said that they intend to revise the strategic plan in the next year or so. As NNSA revises its strategic plan, by including fully developed elements of an effective strategic plan, NNSA would help make the strategic plan more useful in measuring goal achievement and assessing accountability. GAO is making three recommendations, including that NNSA, as it revises its strategic plan for explosives activities, include fully developed elements of an effective strategic plan. NNSA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Our experience has shown that the key elements needed to make progress in high-risk areas are top-level attention by the administration and agency leaders grounded in the five criteria for removal from the High-Risk List, as well as any needed congressional action. The five criteria for removal that we issued in November 2000 are as follows: Leadership commitment. Demonstrated strong commitment and top leadership support. Capacity. Agency has the capacity (i.e., people and resources) to resolve the risk(s). Action plan. A corrective action plan exists that defines the root cause, solutions, and provides for substantially completing corrective measures, including steps necessary to implement solutions we recommended. Monitoring. A program has been instituted to monitor and independently validate the effectiveness and sustainability of corrective measures. Demonstrated progress. Ability to demonstrate progress in implementing corrective measures and in resolving the high-risk area. Starting in our 2015 update, we added clarity and specificity to our assessments by rating each high-risk area’s progress on the five criteria and used the following definitions: Met. Actions have been taken that meet the criterion. There are no significant actions that need to be taken to further address this criterion. Partially met. Some, but not all, actions necessary to meet the criterion have been taken. Not met. Few, if any, actions towards meeting the criterion have been taken. We are removing two areas—DOD Supply Chain Management and Mitigating Gaps in Weather Satellite Data—from the list due to the progress that was made in addressing the high-risk issues. As we have with areas previously removed from the High-Risk List, we will continue to monitor these areas to ensure that the improvements we have noted are sustained. If significant problems again arise, we will consider reapplying the high-risk designation. We added two areas to the High-Risk List since our 2017 update—Government-Wide Personnel Security Clearance Process and VA Acquisition Management. We are removing the area of DOD Supply Chain Management from the High-Risk List because, since 2017, DOD has addressed the remaining two criteria (monitoring and demonstrated progress) for the asset visibility and materiel distribution segments. Congressional attention, DOD leadership commitment, and our collaboration contributed to the successful outcome for this high-risk area, which had been on GAO’s High-Risk List since 1990. DOD’s actions for the asset visibility segment of this high-risk area included (1) providing guidance for the military components to consider key attributes of successful performance measures during metric development for their improvement initiatives; (2) incorporating into after- action reports, information relating to performance measures; and (3) demonstrating sustained progress by, for example, increasing its visibility of assets through radio-frequency identification (RFID), an automated data-capture technology that can be used to electronically identify, track, and store information contained on a tag. According to DOD, the use of RFID tags to provide visibility of sustainment cargo at the tactical leg (i.e., the last segment of the distribution system) resulted in $1.4 million annual cost savings. DOD’s actions for the materiel distribution segment of this high-risk area included (1) making progress in developing its suite of distribution performance metrics; (2) incorporating distribution metrics, as appropriate, on the performance of all legs of the distribution system, including the tactical leg; (3) making progress in refining its Materiel Distribution Improvement Plan and incorporating additional actions based on interim progress and results; and (4) improving its capability to comprehensively measure distribution performance, identifying distribution problems and root cause, and implementing solutions. According to DOD, initiatives focused on distribution process and operational improvements have resulted in at least $1.56 billion in distribution cost avoidances to date. As we have with areas previously removed from the High-Risk List, we will continue to monitor this area to ensure that the improvements we have noted are sustained. Appendix I provides additional information on this high-risk area. We are removing the area of Mitigating Gaps in Weather Satellite Data from the High-Risk List because—with strong congressional support and oversight—the National Oceanic and Atmospheric Administration (NOAA) and DOD have made significant progress since 2017 in establishing and implementing plans to mitigate potential gaps in weather satellite data. The United States relies on polar-orbiting satellites to provide a global perspective on weather every morning and afternoon. NOAA is responsible for the polar satellite program that crosses the equator in the afternoon while DOD is responsible for the polar satellite program that crosses the equator in the early morning orbit. NOAA’s actions for polar- orbiting weather satellites that addressed the remaining criteria of action plan and demonstrated progress included (1) issuing three updates to its gap mitigation plan between January 2016 and February 2017 to address shortfalls we had identified previously; and (2) successfully launching the NOAA-20 satellite in November 2017, which is currently operational and is being used to provide advanced weather data and forecasts. Moreover, NOAA is also working to build and launch the next satellites in the polar satellite program. DOD’s actions for polar-orbiting weather satellites, pursuant to statutes and accompanying congressional direction, included DOD leadership (1) developing and implementing plans to acquire satellites as part of a family of systems to replace its aging legacy weather satellites, including awarding a contract for its Weather System Follow-on–Microwave program, planned for launch in 2022; (2) establishing plans to meet its highest-priority weather monitoring data collection needs that will not be covered by the Weather System Follow-on–Microwave program, including by acquiring and launching the Electro-Optical/Infrared Weather Systems satellite in 2024; and (3) monitoring the Weather System Follow-on- Microwave satellite program’s progress toward addressing critical needs and assessing its operations and sustainment costs. As we have with areas previously removed from the High-Risk List, we will continue to monitor this area to ensure that the improvements we have noted are sustained. Appendix I provides additional information on this high-risk area. Executive branch agencies are not meeting investigation timeliness objectives, and these processing delays have contributed to a significant backlog that the National Background Investigations Bureau (NBIB)—the agency responsible for personnel security clearance investigations— reported to be approximately 565,000 investigations as of February 2019. In addition, the executive branch has not finalized performance measures to ensure the quality of background investigations and some long- standing key reform initiatives remain incomplete. Further, information technology (IT) security concerns may delay planned milestones for the development of a new background investigation IT system. We included the DOD program on our High-Risk List in 2005 and removed it in 2011 because of improvements in the timeliness of investigations and adjudications, and steps toward measuring the quality of the process. We put the government-wide personnel security clearance process on our High-Risk List in January 2018 because of significant challenges related to the timely processing of security clearances and completing the development of quality measures. In addition, the government’s effort to reform the personnel security clearance process, starting with the enactment of the Intelligence Reform and Terrorism Prevention Act of 2004, has had mixed progress, and key reform efforts have not been implemented government-wide. Since adding this area to the High-Risk List, the Security Clearance, Suitability, and Credentialing Performance Accountability Council (PAC), including its four principal members—the Deputy Director for Management of the Office of Management and Budget (OMB), the Director of National Intelligence (DNI); the Under Secretary of Defense for Intelligence; and the Director of the Office of Personnel Management (OPM)—have not fully met the five criteria for high-risk removal. Several issues contribute to the risks facing the government-wide personnel security clearance process: Clearance processing delays. Executive branch agencies are not meeting most investigation timeliness objectives. The percentage of executive branch agencies meeting established timeliness objectives for initial secret clearances, initial top secret clearances, and periodic reinvestigations decreased each year from fiscal years 2012 through 2018. For example, 97 percent of the executive branch agencies we reviewed did not meet the timeliness objectives for initial secret clearance investigations in fiscal year 2018. Lack of quality measures. While the executive branch has taken steps to establish government-wide performance measures for the quality of background investigations—including establishing quality assessment standards and a quality assessment reporting tool—it is unclear when this effort will be completed. Security clearance reform delays. The executive branch has reformed many parts of the personnel security clearance process— such as updating adjudicative guidelines to establish common adjudicative criteria for security clearances; however, some long- standing key initiatives remain incomplete—such as completing plans to fully implement and monitor continuous evaluation. IT security. DOD is responsible for developing a new system to support background investigation processes, and DOD officials expressed concerns about the security of connecting to OPM’s legacy systems since a 2015 data breach compromised OPM’s background investigation systems and files for 21.5 million individuals. As of December 2018, OPM has not fully taken action on our priority recommendations to update its security plans, evaluate its security control assessments, and implement additional training opportunities. However, since we added this area to our High-Risk List, the PAC has demonstrated progress in some areas. For example, NBIB reported that the backlog of background investigations decreased from almost 715,000 cases in January 2018 to approximately 565,000 cases in February 2019. NBIB officials credit an Executive Memorandum—issued jointly in June 2018 by the DNI and the Director of OPM and containing measures to reduce the investigation backlog—as a driver in backlog reduction. Further, in response to a requirement in the Securely Expediting Clearances Through Reporting Transparency (SECRET) Act of 2018, in September 2018, NBIB reported to Congress, for each clearance level, (1) the size of the investigation backlog, (2) the average length of time to conduct an initial investigation and a periodic reinvestigation, and (3) a discussion of the factors contributing to investigation timeliness. The PAC is also reporting publicly on the progress of key reforms through www.performance.gov, and for fiscal year 2018, the website contains quarterly action plans and progress updates, which present figures on the average timeliness of initial investigations and periodic reinvestigations for the executive branch as a whole, investigation workload and backlog, and investigator headcounts. We have made numerous recommendations to PAC members to address risks associated with the personnel security clearance process between 2011—when we removed DOD’s personnel security clearance program from the High-Risk List, and 2018—when we placed the government-wide personnel security clearance process on the High-Risk List. We consider 27 of these recommendations key to addressing the high-risk designation. Eight recommendations key to the high-risk designation have been implemented, including three since January 2018. Nineteen of these key recommendations remain open—including recommendations that the principal members of the PAC (1) conduct an evidence-based review of investigation and adjudication timeliness objectives, (2) develop and report to Congress on investigation quality measures, (3) prioritize the timely completion of efforts to modernize and secure IT systems that affect clearance holders government-wide, and (4) develop and implement a comprehensive workforce plan that identifies the workforce needed to meet current and future demand for background investigations services and to reduce the investigations backlog. See page 170 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. VA spends tens of billions of dollars to procure a wide range of goods and services—including medical supplies, IT, and construction of hospitals, clinics, and other facilities—to meet its mission of providing health care and other benefits to millions of veterans. VA has one of the most significant acquisition functions in the federal government, both in obligations and number of contract actions. The Veterans Health Administration (VHA) provides medical care to veterans and is by far the largest administration in the VA. Since we began focusing on VA’s acquisition management activities in 2015, we have reported numerous challenges in this area. Since 2015, we have made 31 recommendations, 21 of which remain open, that cover a range of areas to address challenges in VA’s acquisition management. In fiscal year 2019, VA received the largest discretionary budget in its history—$86.5 billion, about $20 billion higher than in 2015. About a third of VA’s discretionary budget in fiscal year 2017, or $26 billion, has been used to contract for goods and services. VA’s acquisition management continues to face challenges including (1) outdated acquisition regulations and policies; (2) lack of an effective medical supplies procurement strategy; (3) inadequate acquisition training; (4) contracting officer workload challenges; (5) lack of reliable data systems; (6) limited contract oversight and incomplete contract file documentation; and (7) leadership instability. In light of these challenges and given the significant taxpayer investment, it is imperative that VA show sustained leadership commitment to take steps to improve the performance of its procurement function so that it can use its funding in the most efficient manner possible to meet the needs of those who served our country. This area has been added to the High-Risk List for the following reasons in particular: Outdated acquisition regulations and policies. VA’s procurement policies have historically been outdated, disjointed, and difficult for contracting officers to use. In September 2016, we reported that the acquisition regulations contracting officers currently follow have not been fully updated since 2008 and that VA had been working on completing a comprehensive revision of its acquisition regulations since 2011. VA’s delay in updating this fundamental source of policy has impeded the ability of contracting officers to effectively carry out their duties. We recommended in September 2016 that VA identify measures to expedite the revision of its acquisition regulations and clarify what policies are currently in effect. VA concurred with this recommendation but has not yet fully implemented it. Lack of an effective medical supplies procurement strategy. VA’s Medical Surgical Prime Vendor-Next Generation (MSPV-NG) program for purchasing medical supplies to meet the needs of about 9 million veterans at 172 medical centers has not been effectively executed, nor is it in line with practices at leading hospitals that have launched similar programs. We reported in November 2017 that VA’s approach to developing its catalog of supplies was rushed and lacked key stakeholder involvement and buy-in. As a result, VA was not able to accomplish some of the key efficiencies the program was intended to achieve, such as streamlining the purchase of medical supplies and saving money. We recommended in November 2017 that VA develop, document, and communicate to stakeholders an overarching strategy for the program. VA concurred with this recommendation and reported that it would develop a new strategy by March 2019. Contracting officer workload challenges. The majority of our reviews since 2015 have highlighted workload as a contributing factor to the challenges that contracting officers face. Most recently, in September 2018, we reported that about 54 percent of surveyed VA contracting officers said their workload was not reasonable. In addition, in September 2016, we reported that VHA contracting officers processed a large number of emergency procurements of routine medical supplies, which accounted for approximately 20 percent of VHA’s overall contract actions in fiscal year 2016, with obligations totaling about $1.9 billion. Contracting officers told us that these frequent and urgent small-dollar transactions reduce contracting officers’ efficiency and ability to take a strategic view of procurement needs. We recommended in November 2017 that VHA network contracting offices work with medical centers to identify opportunities to more strategically purchase goods and services frequently purchased on an emergency basis. VA concurred with this recommendation and reported in December 2018 that it is utilizing a supply chain dashboard to track items purchased on an emergency basis and determine which of those items to include on the catalog. VA noted that it added 13,300 items to the catalog from June 2018 to December 2018, including items often purchased on an emergency basis. We requested documentation showing which items added to the catalog were previously purchased on an emergency basis, but as of January 2019, VA had not yet provided it. Among other things, VA should implement our 21 open recommendations and specifically needs to take the following steps to demonstrate greater leadership commitment and strategic planning to ensure efficient use of its acquisition funding and staffing resources: Prioritize completing the revision of its acquisition regulations, which has been in process since 2011. Develop, document, and communicate to stakeholders a strategy for the Medical Surgical Prime Vendor program to achieve overall program goals. Identify opportunities to strategically purchase goods and services that are frequently purchased on an emergency basis. See page 210 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. In addition to specific areas that we have designated as high risk, other important challenges facing our nation merit continuing close attention. One of these is the use of illicit drugs and the misuse of prescription drugs and the ways they affect individuals, their families, and the communities in which they live. Over 70,000 people died from drug overdoses in 2017—about 191 people every day—according to the Centers for Disease Control and Prevention, with the largest portion of these deaths attributed to opioids. Further, drug overdoses are the leading cause of death due to injuries in the United States. They are currently at their highest ever recorded level and, since 2011, have outnumbered deaths by firearms, motor vehicle crashes, suicide, and homicide, according to the Drug Enforcement Administration. The Council of Economic Advisors estimates that in 2015, the economic cost of the opioid crisis alone was more than $500 billion when considering the value of lives lost due to opioid-related overdose. Federal drug control efforts spanning prevention, treatment, interdiction, international operations, and law enforcement represent a considerable federal investment. According to the President’s fiscal year 2019 budget, federal drug control funding for fiscal year 2017 was $28.8 billion. Multiple federal agencies have ongoing efforts to respond to this crisis, including efforts to reduce the supply and demand for illicit drugs, to prevent misuse of prescription drugs, and to treat substance use disorders. However, we previously found that many efforts lacked measures to gauge the success of the federal response. Further, we have long advocated an approach to decision-making based on risk management. Such an approach would (1) link agencies’ plans and budgets to achieving their strategic goals, (2) assess values and risks of various courses of actions to help set priorities and allocate resources, and (3) provide for the use of performance measures to assess progress. The Office of National Drug Control Policy (ONDCP) is responsible for overseeing and coordinating the implementation of U.S. drug policy, including developing the National Drug Control Strategy (Strategy). ONDCP released the 2019 Strategy on January 31, 2019. The Strategy focuses on approaches related to prevention, treatment and recovery, and steps to reduce the availability of illicit drugs in the United States. We will continue to monitor the extent to which ONDCP and other federal agencies are employing a risk management and coordinated approach to their efforts to limit drug misuse. In particular, we have ongoing and planned work to assess ONDCP’s operations, including its (1) leadership and coordination of efforts across the federal government; (2) the effects of the drug crisis on labor force participation and productivity and on people with disabilities and other vulnerable populations; (3) key federal efforts to reduce the availability of illicit drugs; and (4) agency efforts around drug education and prevention. We will determine whether this issue should be added to the High-Risk List once we have completed this ongoing and planned work. Agencies can show progress by addressing our five criteria for removal from the list: leadership commitment, capacity, action plan, monitoring, and demonstrated progress. As shown in table 1, 24 high-risk areas, or about two-thirds of all the areas, have met or partially met all five criteria for removal from our High-Risk List; 20 of these areas fully met at least one criterion. Compared with our last assessment, 7 high-risk areas showed progress in one or more of the five criteria without regressing in any of the criteria. Ten high-risk areas have neither met nor partially met one or more criteria. Two areas showed mixed progress by increasing in at least one criterion and also declining in at least one criterion. Three areas declined since 2017. These changes are indicated by the up and down arrows in table 1. Figure 1 shows that since our 2017 update, the most progress was made on the action plan criterion—four high-risk areas received higher ratings. We rated two areas lower on leadership commitment and two areas lower on monitoring. Table 2 shows that 17 of the 34 high-risk areas we rated have met the leadership commitment criterion while two high-risk area ratings regressed on leadership commitment from met to partially met since our last report. Leadership commitment is the critical element for initiating and sustaining progress, and leaders provide needed support and accountability for managing risks. Leadership commitment is needed to make progress on the other four high-risk criteria. Table 2 shows that only three high-risk areas met the criterion for capacity, six met the criterion for action plan, and two met the criterion for demonstrated progress. One high-risk area—U.S. Government’s Environmental Liability—has partially met only one criterion since we added the area to our list in 2017 and the rest are not met. As noted, seven areas showed improvement in one or more criterion without regressing in any criteria. Two areas showed sufficient progress to be removed from the High-Risk List. The other five high-risk areas remaining on the 2019 list demonstrated improvement and are described below. Three of these five improving high-risk areas are the responsibility of the Department of Defense (DOD)—DOD Support Infrastructure Management, DOD Financial Management, and DOD Business Systems Modernization. The two other improving areas are Department of Energy’s (DOE's) Contract Management for the National Nuclear Security Administration and Office of Environmental Management, and Medicare Program & Improper Payments. DOD Support Infrastructure Management: DOD manages a portfolio of real property assets that, as of fiscal year 2017, reportedly included about 586,000 facilities—including barracks, maintenance depots, commissaries, and office buildings. The combined replacement value of this portfolio is almost $1.2 trillion and includes about 27 million acres of land at nearly 4,800 sites worldwide. This infrastructure is critical to maintaining military readiness, and the cost to build and maintain it represents a significant financial commitment. Since our 2017 High-Risk Report, DOD’s rating for two criteria—leadership commitment and action plan—improved from partially met to met. DOD has demonstrated leadership commitment by stating its commitment to addressing key recommendations we have made by, for example, (1) better forecasting the initial Base Realignment and Closure (BRAC) costs for military construction, IT, and relocating military personnel and equipment; (2) better aligning infrastructure to DOD force structure needs by, for example, improving the accuracy and sufficiency of its excess capacity estimates; and (3) pursuing an effort to consolidate and standardize leases, which includes analyzing whether it is feasible to relocate functions from commercial leased space to existing space on an installation, thereby reducing leases and better utilizing excess space. DOD has developed action plans to better identify excess infrastructure and thus be positioned to dispose of it. For example, in the 2017 High- Risk Report, we stated that DOD’s Real Property Efficiency Plan includes DOD’s goals for reducing the footprint of its real property inventory and metrics to gauge progress, to be implemented by the end of 2020. We also found in 2018 that DOD was achieving cost savings and cost avoidances as it had begun using intergovernmental support agreements between military installations and local governments to obtain installation services, such as waste removal, grounds maintenance, and stray animal control. As a result of these and other actions, DOD now meets the action plan criterion for this high-risk area. As of December 2018, 23 recommendations related to this high-risk area remain open. DOD continues to partially meet the criteria for capacity, monitoring, and demonstrated progress. See page 158 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. DOD Financial Management: Since our 2017 High-Risk Report, ratings for the DOD Financial Management high-risk area improved for the criteria of leadership commitment and monitoring. For the leadership commitment criterion, the high-risk area rating improved from partially met to met in 2019 due to several DOD leadership actions. For example, in 2018, DOD leadership met the goal of undergoing an agency-wide financial statement audit and established a process to remediate any audit findings—ultimately to improve the quality of financial information that is most valuable in managing the department’s day-to-day operations. In addition, according to a DOD official, audit remediation efforts have produced benefits in certain inventory processes that have led to operational improvements. DOD leadership demonstrated its commitment to making needed improvements by developing a database that tracks hundreds of findings and recommendations that came out of the audits. In addition, senior leadership has been meeting bimonthly with military services’ leadership for updates on the status of corrective action plans to address audit findings and recommendations, and the Under Secretary of Defense (Comptroller) has been meeting frequently with the Secretary of Defense to review the plans. These same DOD actions also led to the high-risk area’s rating for the criterion of monitoring to improve from not met to partially met. For example, the database mentioned above is intended to capture, prioritize, and assign responsibility for auditor findings and related corrective action plans, which are meant to be used to measure progress towards achieving a clean audit opinion. Further, DOD leadership has held frequent meetings to discuss the status of corrective action plans. In addition, DOD also established councils in certain areas (e.g., financial reporting) to review the status of audit remediation activities and challenges. All of these actions demonstrate an improvement in DOD’s monitoring activities for its financial management function. However, DOD’s efforts to improve its financial management continue to be impaired by long-standing issues—including its decentralized environment; cultural resistance to change; lack of skilled financial management staff; ineffective processes, systems, and controls; incomplete corrective action plans; and the need for more effective monitoring and reporting. DOD remains one of the few federal entities that cannot accurately account for and report on its spending or assets. As of December 2018, 53 recommendations for this high-risk area are open. The DOD Financial Management high-risk area continues to partially meet the capacity and action plan criteria and not meet the demonstrated progress criterion. See page 147 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. DOD Business Systems Modernization: DOD spends billions of dollars each year to acquire modernized systems, including systems that address key areas such as personnel, financial management, health care, and logistics. This high-risk area includes three critical challenges facing DOD: (1) improving business system acquisition management, (2) improving business system investment management, and (3) leveraging DOD’s federated business enterprise architecture. DOD’s capacity for modernizing its business systems has improved over time and, since our 2017 High-Risk Report, DOD’s overall rating for the criterion of action plan improved from not met to partially met in 2019. DOD established a plan for improving its federated business enterprise architecture (i.e., description of DOD’s current and future business environment and a plan for transitioning to the future environment). Specifically, the rating improved for DOD’s federated business enterprise architecture segment of the high-risk area because DOD’s assistant deputy chief management officer approved a business architecture improvement plan in January 2017. Since 2017, we have made 10 recommendations related to this high-risk issue. As of December 2018, 27 recommendations are open. The leadership, capacity, monitoring, and demonstrated progress criteria remain partially met as in 2017. See page 152 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. DOE's Contract Management for the National Nuclear Security Administration and Office of Environmental Management: DOE oversees a broad range of programs related to nuclear security, science, energy, and waste cleanup, among other areas. As the largest civilian contracting agency in the federal government, DOE relies primarily on contractors to carry out its programs. For instance, DOE spends about 90 percent of its annual budget on contracts and acquiring capital assets. In fiscal year 2018, DOE’s budget was $34.5 billion. The high-risk area focuses on contracts, as well as major projects—those with an estimated cost of $750 million or greater—managed by DOE’s National Nuclear Security Administration (NNSA) and Office of Environmental Management (EM). Since our 2017 High-Risk Report, DOE has made progress by improving from a not met to a partially met rating for the demonstrated progress criterion. Specifically, through its Office of Cost Estimating and Program Evaluation, NNSA has enhanced its capability to estimate costs and schedules, and to assess alternatives for programs and projects, among other things. NNSA also made progress by adopting best practices in several areas, such as those for estimating costs and schedules in nuclear weapons refurbishment activities and capital asset acquisitions. For example, we determined that DOE’s revised cost estimate of $17.2 billion to construct a Mixed Oxide Fuel Fabrication Facility to dispose of surplus, weapons-grade plutonium substantially met best practices— providing assurance that the estimated costs could be considered reliable. This finding contributed to DOE’s reevaluation of the project and ultimate termination, in October 2018, in favor of a potentially less costly disposal approach. Fifty-one of our recommendations were open as of December 2018; 15 recommendations were made since the last high-risk update in February 2017. DOE continues to meet the criterion of leadership commitment, partially meet the criteria for action plan and monitoring, and not meet the criterion for capacity. See page 217 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Medicare Program & Improper Payments: In calendar year 2017, Medicare, which is overseen by the Centers for Medicare & Medicaid Services (CMS), financed $702 billion worth of health services for approximately 58 million elderly and disabled beneficiaries. Medicare faces a significant risk with improper payments—payments that either were made in an incorrect amount or should not have been made at all— which reached an estimated $48 billion in fiscal year 2018. Since our 2017 High-Risk Report, estimated improper payment rates declined more than one percent across the Medicare program. In addition, CMS’ rating for the capacity criterion of the improper payments segment improved from partially met to met in 2019 due to several actions. First, the Center for Program Integrity’s (CPI) budget and resources have increased over time and the agency has established work groups and interagency collaborations to extend its capacity. For example, CMS allocated more staff to CPI after Congress provided additional funding. CPI’s full-time equivalent positions increased from 177 in 2011 to 419 in 2017. Additionally, in August 2017, we reported that CMS’s Fraud Prevention System, which analyzes claims to identify health care providers with suspect billing patterns, helped speed up certain fraud investigation processes. Further, the Healthcare Fraud Prevention Partnership helped improve information sharing among payers inside and outside of the government. Since 1990, when we added Medicare to our High-Risk List, we have made many recommendations related to the Medicare program, 28 of which were made since the last high-risk update in February 2017. As of December 2018, more than 80 recommendations remain open. CMS continues to meet the criterion of leadership commitment and to partially meet the remaining three criteria of action plan, monitoring, and demonstrated progress. See page 241 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Congress enacted several laws since our last report in February 2017 to help make progress on high-risk issues. Table 3 lists selected examples of congressional actions taken on high-risk areas. Congressional oversight also plays a vital role in addressing high-risk issues. For example, at a May 2018 hearing, we testified that the Census Bureau’s (Bureau) cost estimate was not reliable, and that the actual cost could be higher than planned. Further, the Secretary of Commerce created a dedicated team to provide oversight and guidance to the Bureau on cost estimation. In addition to its instrumental role in supporting progress in individual high-risk areas, Congress also enacted the following statutes that, if implemented effectively, will help foster progress on high-risk issues government-wide: Fraud Reduction and Data Analytics Act of 2015 (FRDAA):FRDAA is intended to strengthen federal antifraud controls. OMFRDAA requires OMB to use our Fraud Risk Framework to create guidelines for federal agencies to identify and assess fraud risks, and then design and implement control activities to prevent, detect, and respond to fraud. Agencies, as part of their annual financial reports beginning in fiscal year 2017, are further required to report on their fraud risks and their implementation of fraud reduction strategies, which should help Congress monitor agencies’ progress in addressing and reducing fraud risks. To aid federal agencies in better analyzing fraud risks, FRDAA requires OMB to establish a working group tasked with developing a plan for creating an interagency library of data analytics and data sets to facilitate the detection of fraud and the recovery of improper payments. This working group and the library should help agencies coordinate their fraud detection efforts and improve their ability to use data analytics to monitor databases for potential improper payments. The billions of dollars in improper payments, some of which may be a result of fraud, are a central part of the Medicare Program, Medicaid Program, and Enforcement of Tax Laws (Earned Income Tax Credit) high-risk areas. We reported in 2018 that, among other things, OMB did not involve all agencies subject to the act as required by FRDAA or hold the required minimum number of working-group meetings in 2017. As shown in figure 2, a majority of the 72 agencies surveyed indicated a lack of involvement with and information from the working group as challenges in implementing FRDAA. We made three recommendations, including that OMB ensure the working group meets FRDAA’s requirements to involve all agencies that are subject to the act and ensure that mechanisms to share controls, best practices, and data-analytics techniques are in place. OMB did not concur with our recommendations. We continue to believe the recommendations are valid, as discussed in the 2018 report. IT Acquisition Reform, statutory provisions known as the Federal Information Technology Acquisition Reform Act (FITARA): FITARA, enacted in December 2014, was intended to improve how agencies acquire IT and better enable Congress to monitor agencies’ progress in reducing duplication and achieving cost savings. Since the enactment of these provisions, OMB and federal agencies have paid greater attention to IT acquisition and operation, resulting in improvements to the government-wide management of this significant annual investment. These efforts have been motivated in part by sustained congressional support for improving implementation of this law, as highlighted in agencies’ FITARA implementation scores issued biannually by the House Committee on Oversight and Reform. This continuing oversight has produced positive results. For example, in the committee’s December 2018 FITARA implementation scorecard, 18 of the 24 major federal agencies received the highest possible rating for their efforts to improve the management of software licenses, of which we have found there are thousands annually across the government. Seven months earlier, in the prior scorecard, only eight agencies had achieved this rating. Moreover, federal agencies have taken actions to address 106 of the 136 related recommendations that we have made in this area since 2014. FITARA includes specific requirements related to seven areas: the federal data center consolidation initiative, enhanced transparency and improved risk management, agency Chief Information Officer authority enhancements, portfolio review, expansion of training and use of IT acquisition cadres, government-wide software purchasing, and maximizing the benefit of the federal strategic sourcing initiative. In November 2017, Congress extended or removed the sunset dates of several of these statutory requirements that were originally to end in 2018 and 2019. While all of the 24 federal agencies covered by this law have developed FITARA implementation plans, the agencies need to effectively execute these plans. Successfully addressing FITARA requirements is central to making progress in Improving the Management of IT Acquisitions and Operations, which has been on our High-Risk List since 2015. Program Management Improvement Accountability Act (PMIAA): Enacted in December 2016, the act is intended to improve program and project management in certain larger federal agencies. Among other things, the act requires the Deputy Director for Management of OMB to adopt and oversee implementation of government-wide standards, policies, and guidelines for program and project management in executive agencies. The act also requires the Deputy Director to conduct portfolio reviews to address programs we identify as high-risk. It further creates a Program Management Policy Council to act as the principal interagency forum for improving practices related to program and project management. The council is to review programs identified as high-risk and make recommendations to the Deputy Director or designee. OMB has produced a general strategy for implementing the law through 2022 and met some initial milestones required by PMIAA. For example, in June 2018, OMB issued OMB Memorandum M- 18-19, which includes: (1) agency guidance for implementing PMIAA, (2) a five-year strategic outline for improving program and project management, and (3) initial program management standards and principles. Further, agencies have designated Program Management Improvement Officers to guide their implementation of PMIAA. According to OMB, it began implementing PMIAA’s requirement to conduct portfolio reviews on high-risk areas by requiring relevant agencies to provide several items for discussion during the 2018 Strategic Review meetings. These annual meetings are to consist primarily of a discussion of agency progress towards each of the strategic objectives outlined in their strategic plans, but also cover other management topics such as enterprise risk management and high-risk area progress. According to OMB documents, in advance of these meetings, OMB required agencies to provide a high-level summary of (1) any disagreements with our recommendations, (2) progress barriers, and (3) actions needed by OMB, other agencies, or Congress to help the agency achieve progress towards removal from our High-Risk List. OMB officials told us their 2018 Strategic Review meetings did not address each high-risk area but did address government-wide high-risk areas, such as cybersecurity, information technology, and strategic human capital as they related to the President’s Management Agenda. In the past, senior management officials from OMB, applicable agencies, and our agency have met to address areas where additional management attention could be beneficial to high-risk issues. These trilateral meetings, beginning in 2007 and pre- dating PMIAA’s 2016 enactment, have continued across administrations. However, OMB has organized only one of these high-risk meetings since the last high-risk update in 2017, on the Government-wide Personnel Security Clearance Process. In November 2018, OMB told us of plans to hold additional meetings on priority high-risk areas, including the 2020 Decennial Census, Strategic Human Capital Management, Ensuring the Cybersecurity of the Nation, National Aeronautics and Space Administration (NASA) Acquisition Management, and Managing Federal Real Property. Effective implementation of PMIAA provides an important opportunity to enhance progress on high-risk areas by focusing leadership attention through the portfolio reviews and trilateral meetings. Further, a number of high-risk areas have longstanding or significant program and project management concerns, including the acquisition-related high-risk areas for DOD, DOE, NASA, and VA. These and other programs can benefit from improving program and project management. In December 2019, we will report on OMB’s progress in implementing PMIAA, including what further steps it has taken to use the portfolio review process required in PMIAA to address issues on our High-Risk List. Agency leaders took actions to implement our recommendations. These resulted in numerous improvements to programs and operation and improved service. Further, these actions to implement our recommendations resulted in significant financial benefits. Table 4 shows some examples of the financial benefits achieved since our last High-Risk Report. In the 2 years since our last High-Risk Report, three areas—NASA Acquisition Management, Transforming EPA's Process for Assessing and Controlling Toxic Chemicals, and Limiting the Federal Government's Fiscal Exposure By Better Managing Climate Change Risks—have regressed in their ratings against our criteria for removal from the High- Risk List. In addition, while progress is needed across all high-risk areas, we have identified nine additional areas that require significant attention to address imminent, longstanding, or particularly broad issues affecting the nation. NASA plans to invest billions of dollars in the coming years to explore space, improve its understanding of the Earth’s environment, and conduct aeronautics research, among other things. We designated NASA’s acquisition management as high risk in 1990 in view of NASA’s history of persistent cost growth and schedule delays in the majority of its major projects. Following several years of continuing a generally positive trend of limiting cost growth and schedule delays for its portfolio of major projects, we found that NASA’s average launch delay increased from 7 to 12 months between May 2017 and May 2018. Further, the overall development cost growth increased from 15.6 percent to at least 18.8 percent over the same time period. NASA’s largest science project, the James Webb Space Telescope, has experienced schedule delays of 81 months and cost growth of 95 percent since the project’s cost and schedule baseline was first established in 2009. NASA is at risk for continued cost growth and schedule delays in its portfolio of major projects. Since our 2017 high-risk update, we have lowered NASA acquisition management from meeting the rating to partially meeting the rating in two criteria—leadership commitment and monitoring. The other three criteria ratings remained the same as in 2017. Ratings for capacity and demonstrated progress remain partially met and the rating for action plan remains met. Over the next several years, NASA plans to add new, large, and complex projects to the portfolio, including a lunar Gateway—currently being discussed as a platform in a lunar orbit to mature deep space exploration capabilities. In addition, many of NASA’s current major projects, including some of the most expensive ones, are in the phase of their life cycles when cost growth and schedule delays are most likely. NASA acquisition management requires significant attention for the following reasons: NASA leadership has approved risky programmatic decisions for complex major projects, which compounded technical challenges. For example, leadership has approved some programs to proceed (1) with low cost and schedule reserves, (2) with overly aggressive schedules, and (3) without following best practices for establishing reliable cost and schedule baselines. NASA leadership has also not been transparent about cost and schedule estimates for some of its most expensive projects. Without transparency into these estimates, both NASA and Congress have limited data to inform decision making. NASA has not yet instituted a program for monitoring and independently validating the effectiveness and sustainability of the corrective action measures in its new action plan, which NASA finalized in December 2018. In addition, while NASA has taken some steps to build capacity to help reduce acquisition risk, including updating tools aimed at improving cost and schedule estimates, other areas still require attention. For example, we reported in May 2018 that several major NASA projects experienced workforce challenges, including not having enough staff or staff with the right skills. NASA has also identified capability gaps in areas such as scheduling, earned value management, and cost estimating, and has efforts underway to try to improve capacity in these areas. Since 2017, we have made 9 recommendations on this high-risk area, and as of December 2018, 15 recommendations remain open. These recommendations include that NASA needs to improve transparency of major project cost and schedule estimates, especially for its human spaceflight programs, as well as continue to build capacity to reduce acquisition risk. NASA will also need to implement its new action plan and track progress against it. See page 222 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The Environmental Protection Agency’s (EPA’s) ability to effectively implement its mission of protecting public health and the environment is dependent on it assessing the risks posed by chemicals in a credible and timely manner. Such assessments are the cornerstone of scientifically sound environmental decisions, policies, and regulations under a variety of statutes. Based on our work since our 2017 High-Risk Report, the overall rating for leadership commitment decreased from met to partially met due to limited information for completing chemical assessments and proposed budget cuts in the Integrated Risk Information System (IRIS) Program. The ratings for the remaining four criteria remain unchanged and are partially met. The EPA Acting Administrator indicated his commitment to fulfill the agency’s obligations under the Toxic Substances Control Act (TSCA) as amended by the 2016 Frank R. Lautenberg Chemical Safety for the 21 Century Act (Lautenberg Act) and ensure chemicals in the marketplace are safe for human health and the environment. Nonetheless, EPA needs to give more attention to several areas to fully realize the benefits of the new law, and to demonstrate additional progress in the IRIS Program, such as: While EPA released a document in late December 2018 called the IRIS Program Outlook, the Outlook fails to list the projected date for most of the assessments and includes no information regarding assessment prioritization—including how these assessments will meet program and regional office needs. The Lautenberg Act increases both EPA’s responsibility for regulating chemicals and its workload. EPA recently issued a rule under the act to collect fees from certain companies to defray a portion of the implementation costs, but it is unclear whether the fees collected will be sufficient to support relevant parts of the program. EPA issued a First Year Implementation Plan in June 2016 noting that this document is intended to be a roadmap of major activities EPA will focus on during the initial year of implementation. As of mid-February 2019 the plan has not been updated, according to publically available information, although EPA had indicated that it is a living document that will be further developed over time. EPA needs to ensure that the people and resources dedicated to the IRIS Program and TSCA implementation are sufficient. Our March 2019 report on chemical assessments provides information on what remains to be done to address challenges in the IRIS program and implement the Lautenberg Act. Since we added this area to our High-Risk List in 2009, we have made 12 recommendations to EPA related to IRIS and TSCA. As of February 2019, seven recommendations remain open. See page 204 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Numerous studies have concluded that climate change poses risks to many environmental and economic systems and creates a significant fiscal risk to the federal government. The rising number of natural disasters and increasing reliance on the federal government for assistance is a key source of federal fiscal exposure. As of December 2018, total federal funding for disaster assistance since 2005 is approaching half a trillion dollars (about $430 billion), most recently for catastrophic hurricanes, flooding, wildfires, and other losses in 2017 and 2018. The costliness of disasters is projected to increase as extreme weather events become more frequent and intense due to climate change. There are five areas where government-wide action is needed to reduce federal fiscal exposure, including, but not limited to, the federal government’s role as (1) the insurer of property and crops; (2) the provider of disaster aid; (3) the owner or operator of infrastructure; (4) the leader of a strategic plan that coordinates federal efforts and informs state, local, and private-sector action; and (5) the provider of data and technical assistance to decision makers. Neither global efforts to mitigate climate change causes nor regional adaptation efforts currently approach the scales needed to avoid substantial damages to the U.S. economy, environment, and human health over the coming decades, according to the November 2018 Fourth National Climate Assessment. Government-wide action is needed to improve the nation’s resilience to natural hazards and reduce federal fiscal exposure to climate change impacts. Congress continues to show its commitment to progress on this high-risk issue by enacting legislation. For example, in October 2018, the Disaster Recovery Reform Act was enacted, which, among other things, allows the President to set aside, with respect to each major disaster, a percentage of certain grants to use for pre-disaster hazard mitigation. In addition, the National Defense Authorization Act of 2018, required, among other things, DOD to report on climate impacts to its installations. However, the federal government has not made measurable progress since 2017 to reduce its fiscal exposure to climate change, and in some cases, has revoked prior policies designed to do so. Specifically, since 2017, the ratings for four criteria remain unchanged—three at partially met and one at not met. The rating for one criterion—monitoring—regressed to not met. Limiting the federal government’s fiscal exposure to climate change requires significant attention because the federal government has revoked prior policies that had partially addressed this high-risk area and has not implemented several of our recommendations that could help reduce federal fiscal exposure. For example, since our 2017 high-risk update, the federal government: revoked Executive Order 13690, which had established a government-wide federal flood risk management standard to improve the resilience of communities and federal assets against the impacts of flooding. This action could increase federal fiscal exposure, as taxpayer-funded projects may not last as long as intended because they are not required to account for future changes in climate-related risk. rescinded its guidance directing agencies to consider climate change in their National Environmental Policy Act of 1969 reviews for certain types of federal projects. has not implemented our July 2015 recommendation to establish a comprehensive investment strategy identifying, prioritizing, and implementing federal disaster resilience investments that could reduce federal fiscal exposure to climate change. has not implemented our November 2015 recommendations to create a national climate information system providing authoritative, accessible information useful for state, local, and private-sector decision making. We have made 62 recommendations related to this high-risk area, 12 of which were made since our February 2017 high-risk update. As of December 2018, 25 remain open. The federal government needs a cohesive strategic approach with strong leadership and the authority to manage climate change risks across the entire range of federal activities. See page 110 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Federal agencies and the nation’s critical infrastructures—such as energy, transportation systems, communications, and financial services— are dependent on information technology systems to carry out operations. The security of these systems and the data they use is vital to public confidence and national security, prosperity, and well-being. The risks to systems underpinning the nation’s critical infrastructure are increasing as security threats evolve and become more sophisticated. We first designated information security as a government-wide high-risk area in 1997. This was expanded to include protecting cyber critical infrastructure in 2003 and protecting the privacy of personally identifiable information in 2015. In 2018, we updated this high-risk area to reflect the lack of a comprehensive cybersecurity strategy for the federal government. Since 2010, we have made over 3,000 recommendations to agencies aimed at addressing cybersecurity shortcomings, including protecting cyber critical infrastructure, managing the cybersecurity workforce, and responding to cybersecurity incidents. Of those 3,000 recommendations, 448 were made since our last high-risk update in February 2017. Although many recommendations have been addressed, about 700 have not yet been implemented. Despite the number of unimplemented recommendations, since our 2017 High-Risk Report, the administration has made progress in this high-risk area as it continues to meet the leadership commitment criterion through various actions. These include the President issuing (1) an executive order in May 2017 requiring federal agencies to take a variety of actions, including better managing their cybersecurity risks and coordinating to meet reporting requirements related to cybersecurity of federal networks and critical infrastructure and (2) a National Security Strategy in December 2017 citing cybersecurity as a national priority and identifying needed actions. Further, the administration issued a government-wide reform plan and reorganization recommendations in June 2018 with, among other things, proposals for solving the federal cybersecurity workforce shortage. Additionally, the administration released a National Cyber Strategy in September 2018 outlining activities such as securing critical infrastructure, federal networks, and associated information. However, additional actions are needed. We have identified four major cybersecurity challenges facing the nation: (1) establishing a comprehensive cybersecurity strategy and performing effective oversight, (2) securing federal systems and information, (3) protecting cyber critical infrastructure, and (4) protecting privacy and sensitive data. To address the four major cybersecurity challenges, we identified 10 critical actions the federal government and other entities need to take. These critical actions include, for example, developing and executing a more comprehensive federal strategy for national cybersecurity and global cyberspace; addressing cybersecurity workforce management challenges; and strengthening the federal role in protecting the cybersecurity of critical infrastructure (see figure 3). Until these shortcomings are addressed, federal agencies’ information and systems will be increasingly susceptible to the multitude of cyber- related threats that exist. See page 178 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The expanded federal role in housing finance that began during the 2007–2009 financial crisis has substantially increased the government’s exposure to potential mortgage losses. Federally supported mortgages include those backed by the Federal National Mortgage Association (Fannie Mae) and the Federal Home Loan Mortgage Corporation (Freddie Mac)—collectively, the enterprises—which the Federal Housing Finance Agency (FHFA) placed into government conservatorships in 2008. Federal support also occurs through Federal Housing Administration (FHA) mortgage insurance and Government National Mortgage Association (Ginnie Mae) guarantees on mortgage-backed securities. The substantial financial assistance the enterprises required during and after the crisis, coupled with the large fiscal exposure they and other federal mortgage entities represent today, underscore the need to reform the federal role in housing finance. Delay in resolving the federal role in housing finance poses considerable risks. Through the enterprises, FHA, and Ginnie Mae, the federal government is exposed to potential losses on several trillion dollars in mortgage debt. A severe economic downturn could trigger significant taxpayer assistance to one or more of these entities. Congress and federal agencies have taken some steps to facilitate the transition to a revised federal role, such as holding hearings, introducing legislation, issuing regulations, and developing market monitoring tools. For example, in 2013 and 2014, housing and regulatory agencies finalized rules designed to prevent a recurrence of risky practices in originating and securing mortgages that contributed to the financial crisis. Additionally, FHFA and the Consumer Financial Protection Bureau have developed a representative database of mortgage information that could be useful for examining the effect of mortgage market reforms. However, overall progress on resolving the federal role will be difficult to achieve until Congress provides further direction by enacting changes to the housing finance system. Several issues contribute to the risks facing federal housing finance, including the following: More than 10 years after entering federal conservatorships, the enterprises’ futures remain uncertain and billions of taxpayer dollars remain at risk. Under agreements with the Department of the Treasury (Treasury), the enterprises have received $191.4 billion in capital support as of the end of fiscal year 2018 and have paid dividends to the department exceeding that amount. If they were to incur major additional losses, they would draw required amounts from their remaining $254.1 billion in Treasury commitments. In addition, prolonged conservatorships could hinder development of the broader mortgage securities market by creating uncertainty and crowding out private investment. Nonbanks (lenders and loan servicers that are not depository institutions) have played an increasingly large role in the mortgage market in recent years. While nonbanks have helped provide access to mortgage credit, they also may pose additional risks, in part because they are not federally regulated for safety and soundness. However, FHFA lacks statutory authority to examine nonbank mortgage servicers and other third parties who do business with and pose potential risks to the enterprises. The statutory 2 percent capital requirement for FHA’s $1.26 trillion mortgage insurance fund is not based on a specified risk threshold, such as the economic conditions the fund would be expected to withstand. As a result, it may not provide an adequate financial cushion under scenarios in which Congress may anticipate the fund would be self-sufficient. During the last housing downturn, the fund’s capital ratio fell below the required level and remained there for 6 consecutive years. At the end of fiscal year 2013, the fund required supplemental funds—about $1.7 billion—for the first time in its history. Six of our federal housing recommendations remain open, including those we made in June 2015 on assessing the effects of mortgage reforms already in place. Further, as we previously recommended in November 2016 and January 2019, Congress should consider housing finance reform legislation that: establishes objectives for the future federal role in housing finance, including the role and structure of the enterprises within the housing finance system; provides a transition plan to a reformed system that enables the enterprises to exit federal conservatorship; and addresses all relevant federal entities, including FHA and Ginnie Mae. As we recommended in March 2016 and November 2017, respectively, Congress also should consider granting FHFA explicit authority to examine nonbank servicers and other third parties that do business with the enterprises, and specifying the economic conditions FHA’s insurance fund would be expected to withstand without a substantial risk of requiring supplemental funds. See page 95 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Due to the significance and risk associated with Resolving the Federal Role in Housing Finance, we are separating it from the high-risk area of Modernizing the U.S. Financial Regulatory System. These areas were combined in our 2017 High-Risk report. See page 95 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The Pension Benefit Guaranty Corporation (PBGC) is responsible for insuring the defined benefit pension plans for nearly 37 million American workers and retirees, who participate in about 24,800 private sector plans. PBGC faces an uncertain financial future due, in part, to a long- term decline in the number of traditional defined benefit plans and the collective financial risk of the many underfunded pension plans that PBGC insures. PBGC’s financial portfolio is one of the largest of all federal government corporations. While PBGC’s single employer program had a net surplus of about $2.4 billion at the end of fiscal year 2018, its multiemployer program had a net deficit of about $54 billion—or a combined net accumulated financial deficit of over $51 billion. Its deficit has increased by nearly 45 percent since fiscal year 2013. PBGC has estimated that, without additional funding, its multiemployer insurance program will likely be exhausted by 2025 as a result of current and projected pension plan insolvencies. The agency’s single-employer insurance program is also at risk due to the continuing decline of traditional defined benefit pension plans, as well as premiums that are not well aligned to the financial risk presented by the plans it insures. While Congress and PBGC have taken significant and positive steps to strengthen the agency in the past 5 years, challenges related to PBGC’s funding and governance structure remain. Congress established a temporary Joint Select Committee on multiemployer pension plans in 2018—with the goal of improving the solvency of the multiemployer program. However, the committee did not release draft legislation. Addressing the significant financial risk and governance challenges that PBGC faces will require additional congressional action. Over the years since we added PBGC to the High-Risk List, we have suggested a number of matters for congressional consideration, including: (1) authorizing a redesign of PBGC’s single employer program premium structure to better align premium rates with sponsor risk; (2) adopting additional changes to PBGC’s governance structure—in particular, expanding the composition of its board of directors; (3) strengthening funding requirements for plan sponsors as appropriate given national economic conditions; (4) working with PBGC to develop a strategy for funding PBGC claims over the long term as the defined benefit pension system continues to decline; and (5) enacting additional structural reforms to reinforce and stabilize the multiemployer system, and balance the needs and potential sacrifices of contributing employers, participants, and the federal government. Absent additional steps to improve PBGC’s finances, the long-term financial stability of the agency remains uncertain, and the retirement benefits of millions of American workers and retirees could be at risk of dramatic reductions. See page 267 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. VA operates one of the largest health care delivery systems in the nation through its Veterans Health Administration (VHA), with 172 medical centers and more than 1,000 outpatient facilities organized into regional networks. VA has faced a growing demand by veterans for its health care services—due, in part, to the needs of an aging veteran population—and that trend is expected to continue. The total number of veterans enrolled in VA’s health care system rose from 7.9 million to more than 9 million from fiscal year 2006 through fiscal year 2017. Over that same period, VHA’s total budgetary resources have more than doubled, from $37.8 billion in fiscal year 2006 to $92.3 billion in fiscal year 2017. Given the importance of VHA’s mission, coupled with its lack of progress in addressing its high-risk designation, we continue to be concerned about VHA’s ability to ensure its resources are being used effectively and efficiently to improve veterans’ timely access to safe and high-quality health care. We have identified five areas of concern: (1) ambiguous policies and inconsistent processes; (2) inadequate oversight and accountability; (3) IT challenges; (4) inadequate training for VA staff; and (5) unclear resource needs and allocation priorities. VHA has begun to address each of these areas but, prior to Secretary Robert Wilkie’s July 2018 confirmation, its efforts were impeded by leadership instability. Since taking office, Secretary Wilkie has demonstrated his commitment to addressing the department’s high-risk designation by, among other things, creating an office to direct an integrated, focused high-risk approach and communicating to VA leaders the importance of addressing our recommendations. While VHA completed root cause analyses for each area of concern and developed an action plan in response, the plan lacks milestones and metrics needed to effectively monitor its implementation and demonstrate progress made in addressing the high-risk designation. Additionally, many of VHA’s capacity-building initiatives are either in the initial stages of development or are lacking necessary funding and resources. As such, VHA has not made sufficient progress since our 2017 update to improve its overall ratings, as two high-risk criteria remain partially met and three criteria remain unmet. We remain concerned about VHA’s ability to oversee its programs, hold its workforce accountable, and avoid ambiguous policies and inconsistent processes that jeopardize its ability to provide safe, high-quality care to veterans: In November 2017, we reported that, due in part to misinterpretation or lack of awareness of VHA policy, VA medical center officials did not always document or conduct timely required reviews of providers when allegations were made against them. As a result, we concluded that VA medical center officials may have lacked necessary information to reasonably ensure that their providers were competent to provide safe, high-quality care to veterans and to grant approvals about these providers’ privileges to perform specific clinical services at VA medical centers. We made four recommendations related to this and other findings, all of which remain open. In June 2018, we reported that VHA could not systematically monitor the timeliness of veterans’ access to Veterans Choice Program (VCP) care because it lacked complete, reliable data to do so. We also found that veterans, who were referred to the VCP for routine care because health care services were not available in a timely manner, could potentially wait for care up to 70 calendar days if the maximum amount of time allowed by VA processes is used. This wait time exceeds the statutory requirement that veterans receive VCP care within 30 days of the dates their VA health care providers indicated they should receive appointments, or if no such date existed, within 30 days of the veteran’s preferred date. We made 10 recommendations related to this and other findings, all of which remain open. Similarly, in July 2018, we reported that VA collected data related to employee misconduct and disciplinary actions, but data fragmentation and reliability issues impeded department-wide analysis of those data. Additionally, we found that VA did not consistently ensure that allegations of misconduct involving senior officials were reviewed according to its investigative standards or ensure these officials were held accountable. We made 16 recommendations related to this and other findings, all of which remain open. In November 2018, we reported that VHA’s suicide prevention media outreach activities declined in recent years due to leadership turnover and reorganization. Additionally, we found that VHA did not assign key leadership responsibilities or establish clear lines of reporting for its suicide prevention media outreach campaign, which hindered its ability to oversee the campaign. Consequently, we concluded that VHA may not be maximizing its reach with suicide prevention media content to veterans, especially those who are at-risk. This is inconsistent with VHA’s efforts to reduce veteran suicides, which is VA’s highest clinical priority. We made two recommendations related to this and other findings, both of which remain open. VA needs to further develop its capacity-building initiatives and establish metrics to monitor and measure its progress addressing the high-risk areas of concern. It is also important that our recommendations continue to be implemented. The department has implemented 209 of the 353 recommendations related to VA health care that we made from January 1, 2010 through December 2018, but more than 125 recommendations remain open as of December 2018. This includes 17 that are older than 3 years. In addition to addressing our recommendations, VA needs to make systemic change to department management and oversight in order to fully address the high-risk issues and improve the health care provided to our nation’s veterans. See page 275 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Mission-critical skills gaps both within federal agencies and across the federal workforce impede the government from cost-effectively serving the public and achieving results. For example, the difficulties in recruiting and retaining skilled health care providers and human resource staff at VHA’s medical centers make it difficult to meet the health care needs of more than 9 million veterans. As a result, VHA’s 168 medical centers have large staffing shortages, including physicians, registered nurses, physician assistants, psychologists, physical therapists, as well as human resource specialists and assistants. OPM continues to demonstrate top leadership commitment through its numerous efforts to assist agencies’ in addressing mission-critical skills gaps within their workforces. This includes providing guidance, training and on-going support for agencies on the use of comprehensive data analytic methods for identifying skills gaps and the development of strategies to address these gaps. However, since we first added strategic human capital management to our High-Risk List in 2001, we have reported on the need for agencies to address their workforce skills gaps. As of December 2018, OPM had not fully implemented 29 of our recommendations made since 2012 relating to this high-risk area. Staffing shortages and the lack of skills among current staff not only affect individual agencies but also cut across the entire federal workforce in areas such as cybersecurity and acquisition management. Skills gaps caused by insufficient number of staff, inadequate workforce planning, and a lack of training in critical skills are contributing to our designating other areas as high-risk. As table 5 shows, of the 34 other high-risk areas covered in this report, skills gaps played a significant role in 16 of the areas. Over the years since we added this area to our High-Risk List, in addition to recommendations to address critical skills gaps in individual high-risk areas, we have made numerous recommendations to OPM related to this high-risk issue, 29 of which remain open. Agencies also need to take action to address mission-critical skills gaps within their own workforces – a root cause of many high-risk areas. See page 75 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The 2010 Census was the costliest in history at about $12.3 billion; as of October 2017, the 2020 Census is projected to cost about $15.6 billion, a 27 percent increase. For the 2020 Census, the U.S. Census Bureau (Bureau) plans to implement several innovations, including new IT systems. Implementing these innovations, along with other challenges, puts the Bureau’s ability to conduct a cost-effective census at risk. The decennial census is mandated by the U.S. Constitution and provides vital data for the nation. Census data are used, among other purposes, to apportion seats in the Congress and allocate billions of dollars in federal assistance to state and local governments. To ensure its success, this complicated and costly undertaking requires careful planning, risk management, and oversight. Census activities, some of which are new for the 2020 cycle, must be carried out on schedule to deliver the state apportionment counts to the President by December 31, 2020. The Bureau and the Department of Commerce (Commerce) have strengthened leadership commitment with executive-level oversight of the 2020 Census by holding regular meetings on the status of IT systems and other risk areas. In addition, in 2017 Commerce designated a team to assist senior Bureau management with cost estimation challenges. These examples demonstrate both the Bureau’s and Commerce’s strong leadership commitment to implementing the 2020 Census. One of the Bureau’s major challenges is to control any further cost growth and develop cost estimates that are reliable and reflect best practices for the 2020 Census. According to the Bureau, the total cost of the 2020 Census is now estimated to be approximately $15.6 billion, more than $3 billion higher than previously estimated by the Bureau. The higher estimated life-cycle cost is due, in part, to the Bureau’s failure to previously include all cost associated with the decennial census. The Bureau’s schedule for developing IT systems has experienced delays that have compressed the time available for system testing, integration testing, and security assessments. These schedule delays have contributed to systems experiencing problems after deployment, as well as cybersecurity challenges. For example, as of December 2018, the Bureau had identified nearly 1,100 system security weaknesses that needed to be addressed. Continued schedule management challenges may compress the time available for the remaining system testing and security assessments, and increase the risk that deployed systems will either not function as intended, have security vulnerabilities, or both. As of January 2019, 30 of our recommendations related to this high-risk area had not been implemented. To make continued progress, the Bureau needs to ensure that its approach to strategic planning, IT management, cybersecurity, human capital management, internal collaboration, knowledge sharing, as well as risk and change management are all aligned toward delivering more cost-effective outcomes. Among other things, the Bureau needs to ensure cost growth is controlled and that the development and testing of key systems is completed and fully integrated with all census operations before the 2020 Census. In addition, the Bureau needs to address cybersecurity weaknesses in a timely manner and ensure that security risks are at an acceptable level before systems are deployed. See page 134 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. An improper payment is any payment that should not have been made or that was made in an incorrect amount (including overpayments and underpayments) under statutory, contractual, administrative, or other legally applicable requirements. Reducing improper payments—such as payments to ineligible recipients or duplicate payments—is critical to safeguarding federal funds. However, the federal government has consistently been unable to determine the full extent of improper payments and reasonably assure that appropriate actions are taken to reduce them. Since 2003—when certain agencies were required by statute to begin reporting improper payments—cumulative improper payment estimates have totaled about $1.5 trillion. As shown in figure 4, for fiscal year 2018, federal entities estimated about $151 billion in improper payments. Medicare and Medicaid improper payments and the Earned Income Tax Credit (EITC) improper payments—a part of the Enforcement of Tax Laws high-risk area—accounted for about 68.5 percent of this total. Federal spending for Medicare programs and Medicaid is expected to significantly increase in the coming years, so it is especially critical to take appropriate measures to reduce improper payments in these programs. Internal Revenue Service estimates also show that the EITC has consistently had a high improper payment rate. OMB has designated Medicare programs, Medicaid, and EITC as high-priority programs for improper payments, indicating they are amongst the highest-risk programs where the government can achieve the greatest return on investment for the taxpayer by ensuring that improper payments are eliminated. Our work has identified a number of strategic and specific actions agencies can take to reduce improper payments, which could yield significant savings, and help ensure that taxpayer funds are adequately safeguarded. Continued agency attention is needed to (1) identify susceptible programs, (2) develop reliable methodologies for estimating improper payments, (3) report as required by statute, and (4) implement effective corrective actions based on root cause analysis. Absent such continued efforts, the federal government cannot be assured that taxpayer funds are adequately safeguarded. See pages 241, 250, and 235 of the report (respectively) for additional detail on the Medicare Program & Improper Payments, Strengthening Medicaid Program Integrity, and Enforcement of Tax Laws high-risk areas, including more details on actions that need to be taken. The Internal Revenue Service (IRS) continues to face two pressing challenges in enforcing tax laws: addressing the tax gap—amounting to hundreds of billions of dollars each year when some taxpayers fail to pay the taxes that they owe—and combatting identity theft (IDT) refund fraud. Enforcement of Tax Laws has been on GAO’s high risk list since 1990. IRS enforcement of tax laws helps fund the U.S. government by collecting revenue from noncompliant taxpayers and, perhaps more importantly, promoting voluntary compliance by giving taxpayers confidence that others are paying their fair share. In 2016, IRS estimated that the average annual net tax gap, the difference between taxes owed and taxes paid on time, was $406 billion, on average, for tax years 2008-2010. While IRS continues to demonstrate top leadership support to address the tax gap, IRS’s capacity to implement new initiatives and improve ongoing enforcement and taxpayer service programs remains a challenge. For example, IRS’s strategic plan includes a goal to facilitate voluntary compliance and deter noncompliance that could address the tax gap. However, IRS could do more to identify specific efforts for improving compliance in its strategic plan, measure the effects of compliance programs—such as those used for large partnerships—and develop specific quantitative goals to reduce the tax gap. Such efforts would help IRS make more effective use of its resources and gauge the success of its strategies. The second challenge facing IRS is IDT refund fraud, which occurs when an identity thief files a fraudulent tax return using a legitimate taxpayer’s identifying information and claims a refund. IRS estimates that at least $12.2 billion in individual IDT tax refund fraud was attempted in 2016, of which it prevented at least $10.5 billion (86 percent). Of the amount attempted, IRS estimated that at least $1.6 billion (14 percent) was paid. IRS’s ability to combat IDT fraud continues to be challenged as more personally identifiable information has become readily available as a result of large-scale cyberattacks on various entities. This makes it more difficult for IRS to distinguish between fraudsters and legitimate taxpayers. While IRS has demonstrated some progress by developing tools and programs to further detect and prevent IDT refund fraud, it has not completed updating its authentication procedures to be in compliance with new government standards. As a result, IRS may be missing an opportunity to implement the most secure, robust technologies to protect taxpayers. As of December 2018, 189 GAO recommendations related to this high- risk area had not been implemented. To make continued progress on closing the tax gap, IRS needs to re-establish goals for improving voluntary compliance and develop and document a strategy that outlines how it will use its data to help address this issue. Reducing the tax gap will also require targeted legislative actions, including additional third- party information reporting, enhanced electronic filing, expanded math error authority (also referred to as correctible error authority), and paid preparer regulation. To help stay on top of IDT refund fraud, IRS should develop a comprehensive process to evaluate alternative options for improving taxpayer authentication. Given that IDT refund fraud continues to be a challenge, targeted legislative action, such as requiring a scannable code on returns prepared electronically but filed on paper could help IRS address such fraud. See page 235 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The federal government currently invests more than $90 billion annually in IT, and OMB has implemented several key initiatives intended to help better manage this investment. Additionally, enactment of FITARA, in conjunction with greater attention paid to the acquisition and operation of IT, has helped further improve the government-wide management of this significant annual investment. OMB’s current level of top leadership support and commitment to ensure that agencies successfully execute its guidance on implementing FITARA and related IT initiatives has helped this high-risk area meet the leadership commitment high-risk criteria. Additional positive government-wide actions have enabled this high-risk area to partially meet the four remaining high-risk criteria. For example, OMB has established an IT Dashboard—a public website that provides detailed information on major IT investments at 26 federal agencies—and agencies’ data center consolidation efforts have resulted in a total savings of slightly more than 80 percent of the agencies’ planned $5.7 billion in savings since 2011. However, major federal agencies have yet to fully address the requirements of FITARA and realize billions of dollars in planned or possible savings and improved government performance through more efficient budgeting and management of IT. As government-wide spending on IT increases every year, the need for appropriate stewardship of that investment increases as well. However, OMB and federal agencies have not made significant progress since 2017 in taking the steps needed to improve how these financial resources are budgeted and utilized. While OMB has continued to demonstrate its leadership commitment through guidance and sponsorship of key initiatives, agencies still have not fully implemented all requirements of FITARA, such as putting into place authorities the law requires for chief information officers (CIO). Additionally, while the President’s Management Agenda has a goal to improve IT spending transparency, agencies are underreporting IT contract obligations by billions of dollars. OMB and the agencies also have not yet implemented hundreds of our recommendations on improving shortcomings in IT acquisitions and operations. In an August 2018 review of the 24 federal agencies covered by FITARA, none had IT management policies that fully addressed the role of their CIOs consistent with federal laws and guidance. Specifically, the majority of the agencies only minimally addressed, or did not address, their CIO’s role in assessing agency IT workforce needs and developing strategies and plans for meeting those needs. Correspondingly, the majority of the 24 CIOs acknowledged that they were not fully effective at implementing IT management responsibilities, such as IT strategic planning and investment management. Further, in January 2018, we reported that the majority of 22 agencies did not identify all of their IT acquisition contracts, totaling about $4.5 billion in IT-related contract obligations beyond those reported by agencies. In addition, in November 2018 we reported that four selected agencies lacked quality assurance processes for ensuring that billions of dollars requested in their IT budgets were informed by reliable cost information. Until agencies properly identify IT contracts and establish processes for ensuring the quality of cost data used to inform their budgets, agency CIOs are at risk of not having appropriate oversight of IT acquisitions and may lack adequate transparency into IT spending to make informed budget decisions. As of December 2018, OMB and federal agencies had fully implemented only 59 percent of the recommendations we have made since fiscal year 2010 to address shortcomings in IT acquisitions and operations. OMB and agencies should work toward implementing our remaining 456 open recommendations related to this high-risk area. These remaining recommendations include 12 priority recommendations to agencies to, among other things, report all data center consolidation cost savings to OMB, plan to modernize or replace obsolete systems as needed, and improve their implementation of PortfolioStat—an initiative that is to consolidate and eliminate duplicative systems. OMB and agencies need to take additional actions to (1) implement at least 80 percent of our open recommendations related to the management of IT acquisitions and operations, (2) ensure that a minimum of 80 percent of the government’s major IT acquisitions deliver functionality every 12 months, and (3) achieve at least 80 percent of the over $6 billion in planned PortfolioStat savings. See page 123 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Our high-risk program continues to be a top priority at GAO and we will maintain our emphasis on identifying high-risk issues across government and on providing recommendations and sustained attention to help address them, by working collaboratively with Congress, agency leaders, and OMB. As part of this effort, we hope to continue to participate in regular meetings with the OMB Deputy Director for Management and with top agency leaders to discuss progress in addressing high-risk areas. Such efforts have been critical for the progress that has been made. This high-risk update is intended to help inform the oversight agenda for the 116th Congress and to guide efforts of the administration and agencies to improve government performance and reduce waste and risks. Thank you, Chairman Cummings, Ranking Member Jordan, and Members of the Committee. This concludes my testimony. I would be pleased to answer any questions. For further information on this testimony, please contact J. Christopher Mihm at (202) 512-6806 or MihmJ@gao.gov. Contact points for the individual high-risk areas are listed in the report and on our high-risk website. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. The following pages provide overviews of the two areas removed from the High-Risk List. Each overview discusses (1) why the area was high risk, and (2) why the area is being removed from the list. Each of these high- risk areas is also described on our High-Risk List website, http://www.gao.gov/highrisk/overview. Since our 2017 High-Risk Report, DOD has continued to meet the criteria of leadership commitment, capacity, and action plan for asset visibility. Further, DOD has fully addressed the three remaining actions and outcomes we outlined in 2017 in order to mitigate or resolve long-standing weaknesses in asset visibility. Consequently, DOD has met the monitoring and demonstrated progress criteria for asset visibility to remove this area from our High-Risk List. Leadership commitment: met. Senior leaders have continued to demonstrate commitment through their involvement in groups such as the Supply Chain Executive Steering Committee—senior-level officials responsible for overseeing asset visibility improvement efforts—and through the Asset Visibility Working Group, which identifies opportunities for improvement and monitors the implementation of initiatives by issuing its Strategy for Improving DOD Asset Visibility (Strategy) in 2014, 2015, and 2017. Capacity: met. DOD continues to demonstrate that it has the capacity— personnel and resources—to improve asset visibility. For example, DOD’s 2015 and 2017 Strategies advise the components to consider items such as staffing, materiel, and sustainment costs when documenting cost estimates for the initiatives in the Strategy, as we recommended in January 2015. Action plan: met. A provision in the National Defense Authorization Act for Fiscal Year 2014 required DOD to submit to Congress a comprehensive strategy and implementation plans for improving asset tracking and in-transit visibility. In January 2014, DOD issued the Strategy and accompanying implementation plans, which outlined initiatives intended to improve asset visibility. DOD updated its 2014 Strategy in October 2015 and in August 2017. Importantly, since 2017 DOD addressed the three remaining actions and outcomes related to the monitoring and demonstrated progress criteria through updates to and implementation of the Strategies (see table 6). Monitoring: met. DOD provided guidance in its 2017 update to the Strategy for the military components to consider key attributes of successful performance measures during metric development for their improvement initiatives. As appropriate, the military components have followed the guidance and provided high-level summary metrics updates to the Asset Visibility Working Group. In addition, DOD has taken steps to monitor asset visibility by incorporating into after-action reports, as appropriate, information relating to performance measures. These after- action reports serve as closure documents and permanent records of each initiative’s accomplishments. Demonstrated progress: met. DOD has demonstrated sustained progress by completing 34 of the 39 initiatives to improve asset visibility and continues to monitor the remaining 5 initiatives. These initiatives have supported DOD’s goals and objectives, which include: (1) improving visibility efficiencies of physical inventories, receipt processing, cargo tracking, and unit moves; (2) ensuring asset visibility data are discoverable, accessible, and understandable to support informed decision-making across the enterprise; and (3) increasing efficiencies for delivery accuracy and cycle times. Also, the Asset Visibility Working Group meets regularly to identify opportunities to further improve asset visibility within DOD. DOD has taken the following actions to demonstrate sustained progress: (1) created an integrated single portal system providing 7,500 users access to near-real-time, in-transit visibility of eight million lines of items of supply and transportation data; and (2) increased its visibility of assets through radio-frequency identification (RFID), an automated data-capture technology that can be used to electronically identify, track, and store information contained on a tag. There are two main types of RFID tags, passive and active, which show whether assets are in-storage, in-transit, in-process, or in-use. Passive tags, such as mass transit passes, do not contain their own power source and cannot initiate communication with a reader; while active tags, such as an “E-Z pass,” contain a power source and a transmitter, and send a continuous signal over longer distances. DOD closed nine initiatives from its Strategies by implementing RFID technology. For example, the Marine Corps implemented long-range passive RFID for visibility and accountability of items, resulting in improvements that include an increased range for “reading” an item— from 30 feet to 240 feet—and reduced inventory cycle times from 12 days to 10 hours. Also, the Navy reported that the use of passive RFID technology to support the overhaul of its nuclear-powered attack submarines enabled the Navy to better track parts, resulting in 98 percent fewer missing components and an average cost avoidance of $1.3 million per boat. Additionally, according to DOD, the use of RFID tags to provide visibility of sustainment cargo at the tactical leg resulted in $1.4 million annual cost savings. Further, DOD reported that the migration of the active RFID enterprise from a proprietary communication standard to a competitive multivendor environment reduced the cost of active RFID tags by half, resulting in an estimated $5.7 million annual reduction in costs. Since our 2017 High-Risk Report, DOD has continued to meet the criteria of leadership commitment, capacity, and action plan for materiel distribution. Further, DOD has fully addressed the four remaining actions and outcomes we outlined in 2017 in order to mitigate or resolve long-standing weaknesses in materiel distribution. Consequently, DOD has met the monitoring and demonstrated progress criteria for materiel distribution to remove this area from our High-Risk List. Leadership commitment: met. Senior leaders continue to demonstrate commitment through their involvement in groups such as the Supply Chain Executive Steering Committee—senior-level officials responsible for overseeing materiel distribution corrective actions—and through the Distribution Working Group, which helped develop the Materiel Distribution Improvement Plan (Improvement Plan) in 2016. Capacity: met. DOD has continued to demonstrate that it has the personnel and resources, such as key organizations and the associated governance structure, to improve materiel distribution. The Improvement Plan recognizes that additional resources will be required to accomplish its corrective actions and close any identified performance gaps within the time frame specified. Action plan: met. In 2016, DOD developed its corrective action plan to address the department’s materiel distribution challenges. The Improvement Plan details specific goals and actions to better measure the end-to-end distribution process, ensure the accuracy of underlying data, and strengthen and integrate distribution policies and the governance structure. Importantly, since 2017, DOD has fully addressed the four remaining actions and outcomes related to monitoring and demonstrated progress to mitigate or resolve long-standing weaknesses in materiel distribution (see table 7). Monitoring: met. DOD has monitored materiel distribution by making progress in developing its suite of distribution performance metrics, improving the quality of their underlying data, and sharing metrics information with stakeholders. For example, in January 2017, DOD developed a suite of performance metrics that provides a comprehensive picture of the distribution process, including whether supplies are delivered on time and at sufficient quantity and quality. Also, DOD implemented checklists to assess the quality of data underlying each performance metric based on relevance, accuracy, comparability, and interpretability. The checklists and their standards assist in identifying root causes and addressing areas where performance data quality may be lacking. DOD has also incorporated internal control requirements in its supply chain management guidance to increase confidence in the performance data. Additionally, DOD has revised its policy documents to require stakeholders to routinely capture and share distribution performance metrics, including cost data, and the department maintains websites to provide current performance information to distribution stakeholders. DOD has also incorporated distribution metrics, as appropriate, on the performance of all legs of the distribution system, including the tactical leg (i.e., the last segment of the distribution system). We previously reported on DOD’s deficiencies to accurately assess its distribution performance at the tactical leg, such as missing delivery dates for shipments in Afghanistan. Since that time, the geographic combatant commands have been tracking metrics at the tactical leg, including required delivery dates, to determine the movement and causes of delays for shipments, and have been sharing distribution performance information with the U.S. Transportation Command (TRANSCOM) through their deployment and distribution operations centers. DOD is implementing a cost framework to incorporate transportation costs for all legs of the distribution system, which will provide an additional metric for distribution stakeholders to assess the efficiency of the system. The first phase of the cost framework began operating in August 2018 and is expected to be fully implemented in 2019. DOD is making progress in refining its Improvement Plan and is incorporating additional actions based on interim progress and results. Since DOD issued the Improvement Plan in September 2016, the agency has (1) documented the results and monitored the status of each corrective action, (2) revised completion dates as needed, and (3) periodically provided decision makers with summary action charts, plans, and milestones. DOD is also updating its instruction on management and oversight of the distribution enterprise to clarify the roles and responsibilities of all distribution stakeholders. DOD officials have not determined a date for when this instruction will be issued. Demonstrated progress: met. DOD has demonstrated sustained progress in improving its capability to comprehensively measure distribution performance, identify distribution problems and root causes, and implement solutions. DOD has implemented 10 of 18 corrective actions in its Improvement Plan and is on track to implement the remaining 8 by September 2019. Because of this progress, DOD’s monthly shipment reports have assessed performance against enhanced metrics across the distribution system. For example, in December 2017, TRANSCOM investigated performance standards for truck deliveries from its Defense Logistics Agency warehouses in Bahrain to customers in Kuwait due to frequent delays in shipments. TRANSCOM determined that inadequate time for clearing customs in Kuwait resulted in an unrealistic delivery standard. TRANSCOM, in coordination with distribution stakeholders, adjusted the delivery standard to adequately account for the in-theater customs process. In addition, TRANSCOM, in partnership with the Defense Logistics Agency and the General Services Administration, developed and implemented initiatives focused on distribution process and operational improvements to reduce costs and improve distribution services to the warfighter. According to DOD, these efforts have resulted in at least $1.56 billion in distribution cost avoidances to date. DOD has demonstrated commendable, sustained progress improving its supply chain management. This does not mean DOD has addressed all risk within this area. It remains imperative that senior leaders continue their efforts to implement initiatives and corrective actions to maintain visibility of supplies, track cargo movements, meet delivery standards, and maintain delivery data for shipments. Continued oversight and attention are also warranted given the recent reorganization of the Office of the Under Secretary of Defense for Acquisition and Sustainment and the resulting change in the oversight structure of Supply Chain Management. We will therefore continue to conduct oversight of supply chain management at DOD. Defense Logistics: Improved Performance Measures and Information Needed for Assessing Asset Visibility Initiatives. GAO-17-183. Washington, D.C.: Mar. 16, 2017. Defense Logistics: DOD Has Addressed Most Reporting Requirements and Continues to Refine its Asset Visibility Strategy. GAO-16-88. Washington, D.C.: Dec. 22, 2015. Defense Logistics: Improvements Needed to Accurately Assess the Performance of DOD’s Materiel Distribution Pipeline. GAO-15-226. Washington, D.C.: Feb. 26, 2015. Since our last high-risk update in 2017, NOAA continues to meet the criteria of leadership commitment, capacity, and monitoring and now also meets the criteria of action plan and demonstrated progress. Leadership commitment: met. NOAA program officials met the leadership commitment criteria in 2015 and have continued to sustain their strong leadership commitment to mitigating potential satellite data gaps since that time. For example, NOAA issued and frequently updated its polar satellite gap mitigation plan, which identifies the specific technical, programmatic, and management steps the agency is taking to ensure that satellite mitigation options are viable. In addition, NOAA executives continue to oversee the acquisition of polar-orbiting satellites through monthly briefings on the cost, schedule, and risks affecting the satellites’ development. Capacity: met. NOAA continues to meet the criterion of improving its capacity to address the risk of a satellite data gap. In December 2014, we recommended that NOAA investigate ways to prioritize the gap mitigation projects with the greatest potential benefit to weather forecasting, such as by improving its high-performance computing capacity. NOAA agreed with this recommendation and implemented it. For example, NOAA upgraded its high-performance computers, which allowed the agency to move forward on multiple other mitigation activities, including experimenting with other data sources and assimilating these data into its weather models. Action plan: met. NOAA now meets the criterion for having a plan to address the risk of a polar satellite data gap, which is an increase over its rating in 2017. In June 2012, we reported that, while NOAA officials communicated publicly and often about the risk of a polar satellite data gap, the agency had not established plans to mitigate the gap. We recommended that NOAA establish a gap mitigation plan, and the agency did so in February 2014. However, in December 2014, we recommended that NOAA revise its plan to address shortfalls, including (1) adding recovery time objectives for key products, (2) identifying opportunities for accelerating the calibration and validation of satellite data products, (3) providing an assessment of available alternatives based on their costs and impacts, and (4) establishing a schedule with meaningful timelines and linkages among mitigation activities. mitigation plan between January 2016 and February 2017. With the last of the updates, the agency addressed the shortfalls we had identified. Monitoring: met. NOAA met this criterion in 2017, and continues to meet it now, by implementing our recommendations to more consistently and comprehensively monitor its progress on gap mitigation activities. For example, all three NOAA organizations responsible for gap mitigation projects regularly brief senior management on their progress. Demonstrated progress: met. NOAA now meets the criterion for demonstrated progress, which is an increase over its prior rating. In our 2017 High-Risk Report, we noted that NOAA had identified 35 different gap mitigation projects and was making progress in implementing them. These projects fell into three general categories: (1) understanding the likelihood and impact of a gap, (2) reducing the likelihood of a gap, and (3) reducing the impact of a gap. Nevertheless, one of the most important steps in reducing the likelihood of a gap—keeping the launch of the next polar satellite on schedule—had encountered problems. Specifically, agency officials decided to delay the launch due to challenges in developing the ground system and a critical instrument on the spacecraft. This delay exacerbated the probability of a satellite data gap. More recently, however, NOAA was able to demonstrate progress by successfully launching the satellite in November 2017. That satellite, now called NOAA-20, is currently operational and is being used to provide advanced weather data and forecasts. Moreover, the agency is also working to build and launch the next satellites in the polar satellite program. Since our last high-risk update in 2017, DOD now meets all five high-risk criteria. Leadership commitment: met. With strong congressional oversight, DOD now meets this criterion. Pursuant to enactment of the Carl Levin and Howard P. ’Buck’ McKeon National Defense Authorization Act for Fiscal Year 2015 (NDAA for FY 2015), the National Defense Authorization Act for Fiscal Year 2016 (NDAA for FY 2016), and the Consolidated Appropriations Act, 2016, DOD leadership committed to developing and implementing plans to address its weather satellite requirements. For example, in late 2017, the department awarded a contract for its Weather System Follow-on— Microwave satellite to fulfill core weather requirements. Capacity: met. With strong congressional oversight, DOD now meets the capacity criterion. Specifically, the NDAA for FY 2015 restricted the availability of 50 percent of the FY 2015 funds authorized for the Weather Satellite Follow-on System (now called the Weather System Follow-on— Microwave satellite program) until DOD submitted to the congressional defense committees a plan to meet weather monitoring data collection requirements. In addition, the explanatory statement that accompanied the Consolidated Appropriations Act, 2016, recommended that the Air Force focus on ensuring that the next generation of weather satellites meet the full spectrum of requirements and work with civil stakeholders to leverage appropriate civil or international weather assets. As called for in the law and the explanatory statement, DOD established plans to meet weather monitoring data collection needs, including by acquiring satellites as part of a family of systems to replace its aging legacy weather satellites. Additionally, DOD formally coordinated with NOAA on weather monitoring data collection efforts. In January 2017, the Air Force and NOAA signed a memorandum of agreement, and in November 2017, signed an annex to that agreement, to allow for the exchange of information and collaboration on a plan for collecting weather monitoring data. The Air Force and NOAA are now developing plans to relocate a residual NOAA satellite over the Indian Ocean, an area of concern for cloud characterization and area-specific weather imagery coverage. Action plan: met. In our 2017 High-Risk Report, we reported that DOD was slow to establish plans for its Weather System Follow-on–Microwave program and had made little progress in determining how it would meet weather satellite requirements for cloud characterization and area-specific weather imagery. Pursuant to the NDAA for FY 2015, the NDAA for FY 2016, and the explanatory statement that accompanied the Consolidated Appropriations Act, 2016, the department developed and began implementing plans to address its weather satellite requirements. As mentioned above, in late 2017, the department awarded a contract for its Weather System Follow-on–Microwave satellite to fulfill core weather requirements. Under this program, the department may launch a demonstration satellite in 2021 and plans to launch an operational satellite in 2022. capabilities. DOD plans to launch Operationally Responsive Space-8 as early as 2022. Monitoring: met. DOD now meets the monitoring criterion as evidenced by its actions to initiate a major acquisition program, the Weather System Follow-on–Microwave, and award a contract for the first satellite. In addition, program officials stated that they plan to monitor the program’s progress toward addressing critical needs and assess its operations and sustainment costs. Demonstrated progress: met. DOD now meets the demonstrated progress criterion because it has developed plans and taken actions to address gaps in weather data through its plans to launch the Weather System Follow-on–Microwave satellite in 2022. The department also plans to launch the Electro-Optical/Infrared Weather Systems satellite in 2024 and provide interim capabilities beginning as early as 2022. By developing these plans, DOD has reduced the risk of a gap in weather satellite data and addressed the concerns about a lack of planning that we identified in our 2017 High-Risk Report. DOD’s effective implementation of its plans will be key to further reducing the risks of gaps in weather satellite data in the future. Moving forward, we will continue to monitor both NOAA and DOD efforts to develop and launch the next satellites in their respective weather satellite programs. NOAA plans to launch its next geostationary weather satellite in 2021 and to launch its next polar weather satellite in 2022. DOD plans satellite launches in 2021 (potentially), 2022, and 2024. In addition, we will continue to monitor DOD’s efforts to develop long-term plans to meet its weather satellite requirements. Weapon Systems Annual Assessment: Knowledge Gaps Pose Risks to Sustaining Recent Positive Trends. GAO-18-360SP. Washington, D.C.: Apr. 25, 2018. Satellite Acquisitions: Agencies May Recover a Limited Portion of Contract Value When Satellites Fail. GAO-17-490. Washington, D.C.: June 9, 2017. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-17-333SP. Washington, D.C.: Mar. 30, 2017. Defense Weather Satellites: DOD Faces Acquisition Challenges for Addressing Capability Needs. GAO-16-769T. Washington, D.C.: July 7, 2016. Polar Satellites: NOAA Faces Challenges and Uncertainties that Could Affect the Availability of Critical Weather Data. GAO-16-773T. Washington, D.C.: July 7, 2016. Polar Weather Satellites: NOAA Is Working to Ensure Continuity but Needs to Quickly Address Information Security Weaknesses and Future Program Uncertainties. GAO-16-359. Washington, D.C.: May 17, 2016. Defense Weather Satellites: Analysis of Alternatives Is Useful for Certain Capabilities, but Ineffective Coordination Limited Assessment of Two Capabilities. GAO-16-252R. Washington, D.C.: Mar. 10, 2016. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The federal government is one of the world's largest and most complex entities; about $4.1 trillion in outlays in fiscal year 2018 funded a broad array of programs and operations. GAO's high-risk program identifies government operations with vulnerabilities to fraud, waste, abuse, and mismanagement, or in need of transformation to address economy, efficiency, or effectiveness challenges. This biennial update describes the status of high-risk areas, outlines actions that are still needed to assure further progress, and identifies two new high-risk areas needing attention by the executive branch and Congress. Solutions to high-risk problems save billions of dollars, improve service to the public, and would strengthen government performance and accountability. GAO uses five criteria to assess progress in addressing high-risk areas: (1) leadership commitment, (2) agency capacity, (3) an action plan, (4) monitoring efforts, and (5) demonstrated progress. The ratings for more than half of the 35 areas on the 2019 High-Risk List remain largely unchanged. Since GAO's last update in 2017, seven areas improved, three regressed, and two showed mixed progress by improving in some criteria but declining in others. Where there has been improvement in high-risk areas, congressional actions have been critical in spurring progress in addition to actions by executive agencies. GAO is removing two of the seven areas with improved ratings from the High-Risk List because they met all of GAO's five criteria for removal. The first area, Department of Defense (DOD) Supply Chain Management, made progress on seven actions and outcomes related to monitoring and demonstrated progress that GAO recommended for improving supply chain management. For example, DOD improved the visibility of physical inventories, receipt processing, cargo tracking, and unit moves. Improvements in asset visibility have saved millions of dollars and allow DOD to better meet mission needs by providing assets where and when needed. The second area, Mitigating Gaps in Weather Satellite Data, made significant progress in establishing and implementing plans to mitigate potential gaps. For example, the National Oceanic and Atmospheric Administration successfully launched a satellite, now called NOAA-20, in November 2017. NOAA-20 is operational and provides advanced weather data and forecasts. DOD developed plans and has taken actions to address gaps in weather data through its plans to launch the Weather System Follow-on–Microwave satellite in 2022. There are two new areas on the High-Risk List since 2017. Added in 2018 outside of GAO's biennial high-risk update cycle, the Government-Wide Personnel Security Clearance Process faces significant challenges related to processing clearances in a timely fashion, measuring investigation quality, and ensuring information technology security. The second area, added in 2019, is Department of Veterans Affairs (VA) Acquisition Management. VA has one of the most significant acquisition functions in the federal government, both in obligations and number of contract actions. GAO identified seven contracting challenges for VA, such as outdated acquisition regulations and policies, lack of an effective medical supplies procurement strategy, and inadequate acquisition training. Overall, 24 high-risk areas have either met or partially met all five criteria for removal from the list; 20 of these areas fully met at least one criterion. Ten high-risk areas have neither met nor partially met one or more criteria. While progress is needed across all high-risk areas, GAO has identified nine that need especially focused executive and congressional attention, including Ensuring the Cybersecurity of the Nation, Resolving the Federal Role in Housing Finance, addressing Pension Benefit Guaranty Corporation Insurance Programs, Managing Risks and Improving VA Health Care, and ensuring an effective 2020 Decennial Census. Beyond these specific areas, focused attention is needed to address mission-critical skills gaps in 16 high-risk areas, confront three high-risk areas concerning health care and tax law enforcement that include billions of dollars in improper payments each year, and focus on a yawning tax gap. This statement describes GAO's views on progress made and what remains to be done to bring about lasting solutions for each high-risk area. Substantial efforts are needed by the executive branch to achieve progress on high-risk areas. Addressing GAO's hundreds of open recommendations across the high-risk areas and continued congressional oversight and action are essential to achieving greater progress.", "document_type": "gao"}
{"report": "DOD describes military commissions as a form of military tribunal convened to try individuals for unlawful conduct associated with war. According to DOD, military commissions—as they came to be known in the 19th century—were preceded by military tribunals during previous conflicts, beginning from the Revolutionary War. After the September 11, 2001 terrorist attacks on the United States, the President issued an order, directing the Secretary of Defense to establish commissions to try certain individuals for violations of the laws of war and other offenses. In 2006, the United States Supreme Court invalidated the military commissions established under the President’s order. In response to the court’s ruling, Congress passed the Military Commissions Act of 2006. In 2009, the President ordered a review of military commissions and detention at NSGB which led to a halt in all pending military commissions’ proceedings. In 2009, Congress passed the Military Commissions Act of 2009 which replaced the Military Commissions Act of 2006 and led to the reinstatement of criminal proceedings against certain detainees. Held on NSGB, Cuba, current commissions’ proceedings include alleged terrorists accused of engaging in attacks against the United States, such as the USS Cole attack in which 17 people were killed and the September 11, 2001 attack in which 2,976 people were killed. The Military Commissions Act of 2009 establishes procedures governing the use of military commissions to try alien unprivileged enemy belligerents engaged in hostilities against the United States for violations of the law of war and other offenses triable by military commission. The Act defines an alien unprivileged enemy belligerent as a person who has engaged in hostilities against the United States or its coalition partners; has purposefully and materially supported hostilities against the United States or its coalition partners; or was a part of al Qaeda at the time of the alleged offense. While the Military Commissions Act of 2009 also provides protections for the accused individuals undergoing trial (the accused) similar to rights afforded to defendants in a federal criminal trial, the Act is more closely aligned with military court-martial practice. For example, the Act states that procedures for military commissions are based upon the procedures for trial by general courts-martial under the Uniform Code of Military Justice, Chapter 47 of the U.S. Code, except for certain provisions such as provisions related to speedy trial and pretrial investigations. Article 36 of the Uniform Code of Military Justice states that the President may prescribe regulations for pretrial, trial and post-trial procedures for cases triable in courts-martial and military commissions which shall, so far as the President considers practicable, apply the principles of law and the rules of evidence generally recognized in the trial of criminal cases in the United States district court but which may not be contrary to other provisions of the Uniform Code of Military Justice. Article 36 also states that all rules and regulations prescribed by the President or the Secretary of Defense as his designee shall be uniform insofar as practicable. In addition to relevant law, commissions’ proceedings are conducted in accordance with certain DOD manuals and regulations and rulings by military judges who preside over the proceedings. There are a number of DOD organizations responsible for conducting the commissions’ proceedings included in the scope of our review. Each has separate functions and responsibilities, as shown in figure 1. The Convening Authority is responsible for the overall management of the commissions’ process and is empowered to convene the commissions, refer charges to trial, negotiate pre-trial agreements, review records of trial, and maintain the public website, among other responsibilities. The Office of the Chief Prosecutor includes attorneys, paralegals, and support staff from each branch of the United States Armed Forces, DOD, and attorneys from the Department of Justice. These attorneys coordinate investigative efforts, prepare charges, and represent the United States government in commissions’ proceedings. Located in the Office of the Chief Prosecutor, DOD’s Victim and Witness Assistance Program provides services to approximately 2,000 victims and their family members. The Military Commissions Defense Organization maintains a structure separate from the structure of OMC, to help ensure fairness and independence of the commissions’ legal system. Defense attorneys representing the accused can be military and/or civilian, either employed by DOD and/or a civilian attorney retained by the accused at their own expense. These attorneys are appointed by the Chief Defense Counsel to represent the accused. In capital cases, i.e. those cases in which the United States government is seeking the death penalty for the accused, the Military Commissions’ Defense Organization will also appoint a “learned counsel”—that is, an attorney with specialized training and experience in trials involving the death penalty. The Military Commissions’ Trial Judiciary consists of military judges nominated by the Judge Advocate Generals of the military departments to preside over trials. The Trial Judiciary also includes the judges’ support staff that, among other responsibilities, manages court documents—such as legal motions and judges’ rulings—that are part of the commissions’ process. According to OMC officials, the Trial Judiciary has also established certain practices—followed by OMC— for the review of these documents before they are posted on OMC’s public website. The Expeditionary Legal Complex at NSGB was completed in January 2008 and consists of various facilities, including a courtroom in which classified and unclassified proceedings may be conducted, office space and equipment for court administration employees as well as the prosecution and defense legal teams, and expeditionary lodging capable of housing up to 300 personnel, according to an OMC official. Key elements of this complex are highlighted below. The courtroom, shown in figure 2, is a multi-defendant courtroom capable of trying up to six defendants jointly. The courtroom can accommodate a case with the possibility of the death penalty, and has unique features that permit the use of highly-classified information at the Top Secret/Sensitive Compartmented Information level or below during closed proceedings. The courtroom within the Expeditionary Legal Complex has a viewing gallery (gallery), as shown in figure 3, where selected members of the public may view commissions’ proceedings, through soundproof glass. This is because the gallery was designed to permit public viewing of the proceedings even in the event that classified information is inadvertently disclosed. Specifically, according a DOD official, the gallery has video display monitors that play a closed-circuit television feed of the proceedings, on a 40-second delay between live action in the courtroom and the video transmitted to the gallery. This system provides United States government officials with time to prevent any inadvertent disclosure of classified information from being disseminated to the public. If victims or family members are present in the gallery, they enter last and are seated nearest to the exit. A curtain is available to separate the victims and family members from other members of the public, if they desire privacy. Commissions’ proceedings that are open to the public are transmitted by closed-circuit television to the media operations center located outside of, but nearby, the Expeditionary Legal Complex courtroom. The media operations center, shown in figure 4, also includes telephone and computer support, which enables up to 60 members of the media to simultaneously watch the proceedings, with the 40-second delay to prevent the inadvertent disclosure of classified information, while they work. The center also has a room for conducting press briefings. DOD has taken various steps to facilitate public access to commissions’ proceedings, using four primary methods to do so. Rule 806 of DOD’s Manual for Military Commissions specifies that, except in certain instances, such as to protect national security, that military commissions shall be publicly held. In accordance with this guidance, DOD facilitates public access to commissions’ proceedings by (1) communicating directly with victims and their family members about the status of scheduled hearings and other administrative matters; (2) enabling selected members of the public to view proceedings in-person at NSGB; (3) providing CCTV sites within the United States for viewing proceedings remotely; and (4) making information, such as court documents that will be used during proceedings, available to the public on the commissions’ website. In figure 5, we summarize key DOD efforts to facilitate public access to commissions’ proceedings, followed by a description of each method. According to officials, DOD established its Victim and Witness Assistance Program in June 2004 to provide support services to the approximately 2,000 victims and their family members who opted to participate in the program. The program, which falls within the Office of the Chief Prosecutor, provides updates to victims and their family members on pending military commission cases, notifies them of scheduled hearings, and assists with the logistics associated with viewing proceedings at NSGB or a CCTV site. In our survey of victims and family members, we asked about their perspectives on communication originating from the prosecution team and found that a majority of those who responded (72 percent) were satisfied or very satisfied with DOD’s efforts. Due to space limitations, DOD is currently able to allot 52 seats for selected members of the public to view “open” commissions’ proceedings in-person from the courtroom gallery on NSGB. DOD is responsible for selecting these individuals who generally fall into three categories: (1) victims and their family members, (2) non-government stakeholders, and (3) the general public. DOD provides air transportation to and from NSGB for all individuals approved to view the proceedings in-person. Further details about DOD’s selection process and seating allocation, by category, are provided below. Victims and their family members: Per DOD policy, up to 5 victims or their family members and the person accompanying them to provide support are allotted seating in the courtroom gallery. There are also seats reserved for a grief counselor and an escort from the Victim and Witness Assistance Program for a total of 12 seats. Due to the limited number of total seats and lodging currently available, DOD asks the approximately 1,140 victims and family members who have expressed an interest in attending proceedings to identify the proceedings they would prefer to attend. DOD then uses these preferences to select victims and family members to travel to NSGB for each week that proceedings are held. According to DOD officials, this procedure works better than the lottery system that the Victim and Witness Assistance Program previously used because it provides victims and their family members more flexibility with their travel dates. Non-government stakeholders: This category includes individuals who represent 25 non-governmental organizations pre-approved by DOD to view proceedings in-person, as well as members of the media. DOD currently allots 12 seats in the courtroom gallery to representatives of approved non-governmental and civic organizations and 10 seats to the media. All individuals within this category who are approved for travel to NSGB are required to sign a list of “ground rules” developed by DOD and to be escorted by military personnel while on the base. General public: The remaining 18 seats are filled on a “first come, first served” basis by members of the public who live on NSGB or who have been cleared by the Navy to visit the base. In 2012, DOD established five CCTV sites on the East Coast of the United States where individuals may view commissions’ proceedings remotely. Specifically, four CCTV sites are reserved for victims and their family members, and are located at Fort Hamilton, New York; Fort Devens, Massachusetts; Joint Base Dix/McGuire/Lakehurst, New Jersey; and Naval Station Norfolk, Virginia. The fifth CCTV site is located at Fort Meade, Maryland, and is open to victims and their family members, non- government stakeholders, and the general public. According to officials, at these sites, large video display monitors display the same video feed that appears on monitors in the viewing gallery at NSGB, with the same 40-second delay to prevent the inadvertent disclosure of classified information. This feed is delivered to CCTV sites by both fiber optic cable and satellite transmission. According to court documents, these sites are the result of DOD acknowledging both the importance of the public’s physical access to proceedings held at NSGB and the limited ability of the general public to do so. According to our analysis of available data from DOD on attendance at NSGB and the CCTV sites, there have been a total of 2,304 recorded visitors, beginning in 2011. It is important to note that DOD did not record the number of visitors from the general public at NSGB until approximately September 2018. Also, according to officials, DOD did not begin recording visitors from the general public at the Fort Meade CCTV site until September 2018, and did not record data on non-government stakeholder visitors to the Fort Meade CCTV site from 2012 to 2015. However, our review of available data indicates that of the recorded visitors, the majority—64 percent—were non-government stakeholders, while victims and family members made up 34 percent of attendees, and the general public made up 2 percent. Table 1 summarizes available DOD data on attendance at NSGB and CCTV sites, from November 2011 to September 2018. According to a DOD official, DOD established the Office of Military Commissions’ website as an online resource for the public in March 2005 to provide a variety of information about OMC’s organization, its facilities and services on NSGB, active and inactive cases, and court documents approved for public dissemination, among other things. Court documents may include legal motions (motions) filed by the prosecution and defense, docket-related documents (e.g., documents that list motions to be argued during a specific hearing), judges’ rulings on motions, and transcripts of hearings. According to officials, DOD updated the website in 2011 and 2014, which government and non-government stakeholders told us made it easier to use and provided additional information, thereby facilitating public access to information about the commissions’ proceedings. In addition, DOD officials told us the website has the only official, public calendar of scheduled hearings. The public faces a number of challenges in gaining access to commissions’ proceedings or obtaining information about them. These challenges can be categorized into two groups: (1) those that DOD has limited ability to address, and (2) those that DOD has greater ability to address. During our review, we identified several aspects of commissions’ proceedings that constrain the extent of public access that DOD is able to provide. Specifically, DOD has limited ability to address these challenges because they result, in part, from factors that are not under the department’s control. As confirmed by DOD officials, these challenges are (1) the location of proceedings, (2) the prevalence of classified information associated with them, and (3) the duration of time awaiting trial—each of which are discussed in more detail below. We also identified other public access challenges that DOD has a greater ability to address because the challenges result largely from factors under DOD’s control. As confirmed by DOD officials, these challenges to public access of military commissions’ proceedings involve limitations related to in-person viewing of proceedings at NSGB, remote viewing of proceedings, and the timeliness with which key information is posted on the commissions’ website. DOD policy and processes, the size of the gallery DOD built, and the limited logistical support DOD provides to non-government stakeholders substantially constrain the public’s ability to view commissions’ proceedings at NSGB. As discussed previously, DOD policy and the size of the courtroom gallery on NSGB currently limit in-person attendance to a total of 52 seats for each week of hearings—12 of which are reserved for victims or their family members, as well as the support people and DOD escorts accompanying them. The relatively limited number of seats means that—in the 10 years since victims and their family members were permitted to travel to NSGB—according to a DOD official, fewer than half have been selected to do so. According to our review of DOD data on total attendance at NSGB since 2011, victims and family members comprise 21 percent of attendees. The limited weekly attendance for all visitors to commissions’ proceedings is in contrast to United States district court that conducts federal criminal trials and can generally accommodate a new set of attendees each day, if those attendees are in the local area or can travel to the court house. However, as discussed previously, DOD provides air transportation to and from NSGB, the department must approve all individuals who fly to NSGB to view the proceedings in-person, and the seats available to the general public in the gallery are filled on a “first come, first served” basis by members of the public who live on NSGB or who have been cleared by the Navy to visit the base. These constraints do not exist at federal courthouses. Thus, the portion of the general public that can attend commissions’ proceedings is substantially smaller than the portion of the public that can attend federal criminal trials. In addition, according to non-government stakeholders, DOD provides limited logistical support for their work at NSGB, which constrains their ability to provide the public with access to information about the commissions’ proceedings. Based on discussions with non-government stakeholders, the logistics of traveling to NSGB and the inherent limitations of working in a challenging environment made it difficult for some of these non-government stakeholders to be able to view proceedings in-person with the frequency that they believe is needed. For example, one national security policy expert told us that they “cannot afford the time required to attend another hearing.” This is because “…hearings are frequently cancelled or closed to the public,” and as a result, attendees “…typically spend at least a week there to see maybe two days of hearings.” We also spoke with a legal expert who explained that the lack of reliable internet and phone service while on NSGB presented challenges in maintaining contact with the individual’s law practice, thus limiting their ability to travel to NSGB and view proceedings in-person. Similarly, a member of the media told us that the conditions of reporting the commissions’ proceedings are “an extreme hindrance.” This member of the media noted that while at NSGB, visitors have access to limited and unreliable internet and telephone service. This has made covering the trials “extremely difficult,” according to the freelance journalist because the cost, lack of resources and unreliable schedule make it increasingly difficult to take a week away from reporting on other events “in order to cover only a couple of days of open hearings.” For many of the non-government stakeholders included in our review, their role as observers, scholars, or reporters on the commissions’ proceedings is not their full-time job. Instead, they do so as one part of their professional responsibilities or as volunteers. In this context, they told us generally that the time required to travel to NSGB due to infrequent flights, the difficulty of working there, and the frequent closings or cancellations of hearings discourage non-government oversight and reporting on the proceedings. This, in turn, reduces the amount and quality of the information that they can provide to the public. Also, while selected victims and family members and non-government stakeholders are able to view proceedings in-person on NSGB, the vast majority of the general public cannot, due to DOD policy. The exceptions are—according to a DOD official—civilians traveling to NSGB on official business and those who have a sponsor living at NSGB. DOD’s decision to locate all CCTV sites on military bases on the East Coast of the United States has resulted in several challenges that limit the current usefulness of CCTV sites in facilitating public access to commissions’ proceedings. First, all five CCTV sites are concentrated within a 600 mile span on the East Coast of the United States. However, victims and their family members—the primary intended users of these sites—are located throughout the world or are concentrated in areas of the United States that are a significant distance from one of these five locations. According to our survey of victims and their family members, a majority of those who responded (71 percent) said that it was important to have the location of the hearings close to where they live. For example, the victim and family member population served by DOD’s Victim and Witness Assistance Program has a significant presence in California and Florida, as well as smaller populations in eight other countries. Further, survey respondents from Texas, Florida, and the United Kingdom noted that it was impractical for them to travel to the current CCTV sites, especially considering the unpredictable hearing schedule. Figure 7 shows the location of the CCTV sites along with the dispersion of victims and their family members served by DOD’s Victim and Witness Assistance Program. The logistics of traveling to the CCTV site at Fort Meade, Maryland—the only location open to non-government stakeholders and the general public—is also a factor that limits the public’s access to information about commissions’ proceedings. For example, non-government stakeholders who observe the commissions’ proceedings and were included in our review explained that the majority of their organizations are located in cities that either do not have a CCTV site, or are not near a site to which they have access. Examples include Los Angeles, California, and New York City, New York. Non-government stakeholders also expressed that there are challenges associated with the amount of time and travel it takes to get to Fort Meade, which can be difficult especially when hearings are often cancelled or closed with little or no notification, according to these stakeholders. Further, although the CCTV site at Fort Meade is open to the general public, DOD officials acknowledged that there is no practical way for the department to advertise the availability of the opportunity to view proceedings at the CCTV site on Fort Meade. In addition to the challenges of traveling to CCTV sites, some victims and family members and non-governmental stakeholders noted challenges regarding their ability to access military bases that host these sites. For example, some victims and family members told us that they or their relatives had been denied access to certain CCTV sites because, according to DOD, they did not meet the department’s definition of a victim or family member. Further, non-government stakeholders who are foreign nationals are required to be escorted while on Fort Meade, per DOD policy. However, DOD officials told us that Fort Meade does not always have the personnel necessary to escort these individuals, which could preclude certain non-government stakeholders from being able to access the site. Further, a senior DOD official acknowledged that by locating CCTV sites on military bases, DOD is running the risk that—in certain scenarios—no member of the public would be able to access the sites. This is because, in the event of a threat to base security, it may be closed to civilians who do not live or work on the installation. As discussed previously, OMC’s website is a key enabler of public access to information about commissions’ proceedings because it provides the public with a way to retrieve unclassified court documents related to the commissions’ proceedings, such as legal motions and transcripts, and a schedule of the proceedings’ hearings. According to DOD’s Regulation for Trial by Military Commission (Regulation), court documents are provided by OMC to an inter-agency review team, which examines them and removes any classified or protected information that is identified. Once this examination is completed, the inter-agency review team returns the document to OMC to be posted to its website. DOD’s Regulation’s sets a timeliness standard for reviewing and posting court documents—noting that the entire process generally should take no longer than 15 business days. However, based on our analysis of available data, we determined that DOD has generally not met this standard for the timely posting of documents, which substantially limits public access to information about proceedings. Specifically, we obtained and analyzed data on when court documents were filed with OMC and the date on which the inter-agency review team returned them to OMC for posting and found that from October 2011 to October 2018, DOD frequently missed the timeliness standard laid out in its Regulation. For example, since 2011, we found that 8 percent of court documents reviewed by the inter-agency review team were returned to OMC after the 15 business day standard. Further, we found that—since 2015—DOD missed its timeliness standard with greater frequency. For example, approximately 7 percent of documents reviewed in 2015 were returned to OMC after the 15 business day standard whereas in 2018, more than 50 percent of the documents submitted for review missed the timeliness standard. Our analysis of data from the inter-agency review team is summarized in table 2. In addition to the data provided by the inter-agency review team, we independently collected and analyzed data from the commissions’ website on the filing and posting dates for more than 11,000 court documents filed between June 19 and November 19, 2018. Our analyses of these data further demonstrate DOD’s challenges with timely posting of court documents. For only one category of court documents— unofficial, unauthenticated transcripts from open hearings—our analysis of data collected from the website from June to November 2018 show that these transcripts were posted in a timely manner. For the remainder, over a five month period, nearly 1,300 court documents either remained unposted or were posted to OMC’s website after the 15 day business standard. Furthermore, the total for the median number of business days these documents were filed after the 15 business day standard ranged from 90-103.5 days—that is, from almost four months to more than five months past DOD’s timeliness standard. Table 3 summarizes our analysis for the five cases in the scope of our review. We reviewed relevant case studies in federal criminal proceedings involving both terrorism charges and certain matters related to commissions’ cases, and identified instances in which federal judges adopted processes for review and release of classified documents that are similar to processes specified in DOD’s regulation. However, we also identified differences, such as shorter timeframes in the federal court systems for the government’s review and public release of documents with the potential for classified information. For example, in one case, court security experts had 48 hours—and in another, 72 hours—to complete this process. According to various non-government stakeholders, DOD’s inability to post court documents in a timely manner has negatively impacted their ability to perform their role in facilitating public access to information about commissions’ proceedings. For example, according to our analysis, DOD posted legal motions filed by the prosecution and defense teams a median of 97 business days past DOD’s timeliness standard; military judges’ rulings were posted a median of 69 days past DOD’s timeliness standard. One member of the media explained that DOD’s delayed posting of court documents limits their access to information needed to justify travel to NSGB. They further explained that not being able to travel to NSGB impedes their ability to conduct interviews and research about the proceedings, which are needed to inform the general public. Similarly, other stakeholders told us that they believe the delays in posting docket- related documents have made it difficult for them to assess the proceedings and communicate their assessments to the public. According to our analysis, DOD posted these documents a median of 99 business days past DOD’s timeliness standard. Further, for hearings held between June 19, 2018 and November 19, 2018, we found that of the 74 docket- related documents filed with the court, three were posted in advance of the hearing. We also found that the hearing schedule posted on the commissions’ website—the only official, publicly-accessible schedule of proceedings, according to DOD officials—frequently is not updated in a timely manner to reflect schedule changes. According to DOD officials, this is because information on schedule changes is often not provided to the webmaster for timely updates, as the inter-agency review team is examining it; much like the inter-agency review team does with court documents. As a result, several non-governmental stakeholders told us that it is difficult to justify the time and costs of traveling to Fort Meade, Maryland—the only CCTV site open to them—given the risk of arriving only to learn that the scheduled hearing has been canceled or closed to the public. We observed the effect of these cancellations on public access firsthand during our review. For example, we attempted to attend hearings at Fort Meade on various occasions. On several of those occasions, the hearing was canceled. While we learned this information directly from our DOD contact, none of these changes were reflected on the website’s schedule. Also, when we asked for updates on scheduled hearings, multiple DOD officials told us that we should not bother checking the website’s hearing schedule. Instead, they recommended that we check the Twitter feed of a certain reporter who spends a lot of time at NSGB and routinely provides updates on hearings. In addition, according to DOD officials, victims and family members who attempt to access the website from certain locations outside of the United States are sometimes unable to do so. OMC officials are aware of this issue and an OMC information technology expert told us that while OMC has tried to fix this issue several times, it is based on security for the website. In addition, according to DOD officials, victims and family members who attempt to access the website from certain locations outside of the United States are sometimes unable to do so. OMC officials are aware of this issue and an OMC information technology expert told us that while OMC has tried to fix this issue several times, restricting access from certain locations outside of the United States is based on security for the website. DOD officials acknowledged that they are regularly not meeting their timeliness standard for posting court documents to OMC’s website— something that they largely attribute to the volume of documents submitted and the government-wide security classification review process to which they are subjected. Specifically, in this process for the military commissions’ proceedings, there are two DOD and two non-DOD intelligence agencies with the chief responsibility for conducting the security classification review of court documents filed for commissions’ proceedings. The Defense Intelligence Agency (DIA) is responsible for coordinating the process and all four agencies may be required to review a document depending on the type of information it contains. In accordance with DOD’s Regulation and the interests of national security, a review of certain documents submitted must be conducted to confirm that such filings are in publicly releasable form. Due to the multiple levels of review and depending on the amount and complexity of classified information involved, intelligence agency officials told us that— in the course of the inter-agency review team’s efforts—it can take anywhere from one day to several weeks to review a single document. These officials also told us that it is very difficult to hire personnel with the requisite expertise and experience to serve as reviewers, given that classified information that may be in these documents can be complex, esoteric, and decades old. Thus, it is unlikely that a significant number of new reviewers could be hired to help expedite the review team’s processes. According to our review of available information from intelligence agency officials, the agencies have a relatively small number of personnel reviewing large numbers of documents. Further, those personnel responsible for reviewing OMC-related documents spend only a portion of their time reviewing court documents for the purpose of posting them on the commissions’ website. This is because inter-agency review team personnel are also responsible for reviewing documents not released on the commissions’ website. According to a senior official from the review team, it has been tasked with competing requests for document reviews that have impacted the team’s ability to review court documents for posting on OMC’s website. For example, the official explained that—from May 2017 to February 2018—the review team completed seven of these large-scale, time- sensitive tasks, involving about 31,400 pages of document review, according to the official’s estimate. Table 4 summarizes available information about the agencies’ review of court documents to be posted on the website. Based on our discussions with officials from DOD and the inter-agency review team, factors such as—the complexity of documents, relative scarcity of qualified reviewers, and other document review tasks unrelated to web posting—are somewhat out of DOD’s control. For example, a senior official from the inter-agency review team explained, the complexity of court documents is the responsibility of the prosecution and defense teams that write them; the other document review tasks are often driven by the schedule of individual cases or military judges’ rulings. However, there is a key factor driving the timeliness challenge that may be in the department’s control. According to our discussions with DOD officials, they attributed document posting delays to a policy decision by the department to subject the extremely large volume of court documents filed—including schedule changes—to the same type of security review. Through our review of agency documentation and discussions with DOD officials, victims and family members, and non-government stakeholders, we identified a variety of potential options for expanding access to commissions’ proceedings. We have organized these options into three categories, as shown in table 5. The majority of both victims and family members who responded to our survey and non-government stakeholders who responded to our questionnaire support most potential options for expansion of public access. Specifically, the majority of victims and family members who responded to our survey supported six of the seven potential options about which we asked. The majority of non-government stakeholders supported seven of the ten potential options. There was general agreement between these two groups on the potential options they supported. This information is summarized in figures 8 and 9. Options exist that may potentially help DOD address the challenges the public faces attending hearings at NSGB. Specifically, a physical expansion of the courtroom viewing gallery that increases the number of seats open to the public, along with a change in DOD policy to allow more visitors, would enable NSGB to accommodate more people wishing to view proceedings in-person. An OMC official responsible for management of the office’s infrastructure at NSGB acknowledged that an expansion of the NSGB gallery and the number of the people it can accommodate is theoretically possible, potentially in the context of an ongoing project to renovate the complex of buildings that contains the courtroom, gallery, and other facilities that support the commissions’ proceedings. DOD officials expressed a number of concerns with this option. First, an OMC official cautioned that expanding the gallery’s capacity would likely increase the cost of the current $14 million expansion project, though the official was unable to estimate by how much. Second, an increase in the number of visitors would require a commensurate increase in logistical support—for example, more lodging and utilities—which an OMC official said may not be supported by the current level of resources. Third, according to an OMC official, an expansion of the gallery would require it to be temporarily closed, thus delaying commissions’ proceedings. This is because, the official explained, the current courtroom is the only venue at NSGB that can accommodate a multi-defendant trial and any highly classified evidence required for the proceedings. Further, according to a senior DOD official, renovation of the gallery will require it to be re- accredited before DOD could resume discussing highly classified evidence in the adjoining court room. This could result in a substantial increase in both the period of time in which the gallery and court room are unavailable, as well as the cost of a renovation. In our review of DOD documents and discussions with department officials, we learned that there may be ways to address some of these concerns. For example, DOD is planning to accommodate at least some additional visitors to NSGB. According to OMC documentation, it is planning to support about 350 total attendees per week of hearings during the trial phase of Khalid Shaikh Mohammad et al (2). This is an increase of about 260 percent, compared to the average total number of visitors for a week of pre-trial hearings in this case. Based on our review of relevant court documents and discussions with DOD officials and stakeholders, we identified two broad categories of potential options that may help DOD address the public access challenges associated with CCTV sites: (1) adding or changing the locations of CCTV sites and (2) broadcasting video from NSGB using other technologies, such as the internet. CCTV sites: Additional CCTV sites—that are more evenly distributed across the country—could potentially be established for the general public or for use solely by victims and their family members. DOD officials acknowledged that most military bases have the requisite technology and physical infrastructure to host a CCTV site and that expanding the number of CCTV sites would require a relatively small outlay of resources. Further, they also acknowledged that there may be opportunities to establish CCTV sites at locations other than military bases, such as federal courthouses, which may help address the public access challenges posed by bases’ security procedures, such as foreign nationals’ difficulty when serving as observers or reporters. DOD officials noted however, that expanding CCTV sites would require approval by the Secretary of Defense or a military judge, because— according to DOD’s Manual for Military Commissions—the broadcasting of proceedings in the court room, to include video and audio recording or television broadcasting, shall not be permitted. The military judge, however, may permit contemporaneous closed-circuit video or audio transmission. For example, the prosecution requested this permission in 2012 and the military judge authorized the transmission of all open proceedings, by CCTV, to several sites. Similarly, based on our review of relevant selected case studies of terrorism trials in U.S. federal court, there are previous examples of federal terrorism trials using CCTV sites for the benefit of the public, victims and family members. Further, DOD officials were hesitant to support such an expansion based on their perception that relatively few people have utilized the current CCTV sites, but they were unable to provide complete or fully accurate and reliable data on attendance of certain groups, such as the media and general public. In addition, according to DOD officials, the demand for public access during the current cases’ decade-long pre-trial phase likely does not represent the magnitude of future public interest, which DOD officials believe will increase significantly once the trial phase begins. Television and internet broadcast: Broadcasting video of hearings via other technologies, such as the television or internet would increase opportunities for the general public to view commissions’ proceedings remotely. An OMC information technology expert told us that it would be relatively simple and inexpensive to transmit the existing video feed from the proceedings on NSGB to either television stations, such as C-SPAN, or through the internet using the same cyber security protocols used for CCTV sites. Further, internet broadcasts could either be password- protected so that they could be viewed only by a specific group, such as victims and family members, or they could be made available to the general public. This option raised mixed views from the experts and officials we interviewed. According to Rule 806(c) of the Manual for Military Commissions, television or internet broadcasting would require express authorization by the Secretary of Defense—and as previously noted—this rule is consistent with federal criminal practice which prohibits the broadcasting of judicial proceedings from the courtroom. Legal experts who we contacted had varying perspectives on this issue. For example, officials from the Office of the Chief Prosecutor had concerns that parallel those of the Judicial Conference of the United States—the national policy- making body for the federal courts—on the negative impact of cameras in the courtroom on jurors and witnesses, among other reasons. Specifically, the Judicial Conference cited concerns such as publicity that could threaten jurors’ privacy and witnesses that could be by subjected to distractions, intrusions or influences. In contrast, a senior official in the Military Commissions’ Defense Organizations generally supported television and internet broadcasting of proceedings. This perspective was shared by the American Bar Association, which stated that it would support adoption of such an initiative in the future to protect the integrity of the military commissions’ process and better educate the public about these proceedings. Also, in our discussions with DOD officials, they too expressed mixed perspectives regarding internet or television broadcast of proceedings from NSGB. On one hand, according to an OMC information technology expert, broadcasting is technologically possible and could use certain existing security procedures. Specifically, because safeguarding classified information is critical, any television or internet broadcast of proceedings would use the same video feed currently transmitted to the NSGB gallery and CCTV sites, and thus would use the same safeguards provided by the 40-second delay previously discussed. Further, DOD information technology experts suggested that using a limited internet broadcast, it could be possible for DOD to create temporary viewing sites almost anywhere they are needed; for instance, in a hotel conference room. On the other hand, senior DOD officials expressed several concerns regarding the security implications of broadcasting video outside of the current CCTV framework. For example, they highlighted the potential for adversaries of the United States to copy and alter the video feed from an unsecured broadcast—thus creating a new and inaccurate record of proceedings that could be used as propaganda. Further, while internet broadcasts could be password-protected for victims and their family members only, DOD officials were concerned that the size of the group may make it more likely that the password would be shared with people outside of the group. In regard to these concerns, DOD’s technology experts suggested that they could potentially be addressed, at least in part, by using security procedures already in place at the NSGB gallery and CCTV sites. Specifically, at the temporary viewing sites they proposed, DOD officials would not allow recording of the video feed, following the rules currently in place at NSGB and CCTV sites. However, regarding this proposal, senior DOD officials conveyed force protection concerns for government personnel and any attendees. For example, an official noted that there have been investigations into allegations that OMC personnel have been surveilled by unknown persons, both in the United States and overseas, when on official travel. Also, in a relatively unsecure civilian location like a hotel, DOD would not be able to enforce the rules of the commissions. For instance, according to this official, if someone wanted to attend a temporary viewing site but refused to relinquish their electronic recording devices, per rules currently in place at NSGB and CCTV sites, DOD’s only recourse would be to call local law enforcement authorities. DOD’s Regulation suggests two possible approaches the department can take when reviewing court documents, prior to posting on the website, and one of these could help the department post court documents in a timelier manner. The first approach would allow for an OMC security classification expert to independently determine whether a court document may contain classified information. If it is determined that the document does not contain classified information, the document is to be posted within 1 business day of it being filed. In contrast, according to OMC officials, the second approach provided by the Regulation—and since at least 2014—has been interpreted as directing that every document filed must undergo a security review before it is posted to the OMC website. As discussed previously, DOD officials told us that they attributed the department’s document posting delays to DOD’s policy decision to subject the extremely large volume of court documents filed, including schedule changes, to the same type of security review. DOD’s practice has resulted in nearly every document filed with the commission undergoing a security review before it could be posted to the OMC website. However, at the end of our review, a military judge’s ruling on a pre-trial motion in the case of U.S. v. Khalid Shaikh Mohammad et al.(2) is expected to substantially change DOD’s previous practice of submitting every document for security review prior to posting to the OMC website. Specifically, in December 2018, a military judge found that DOD’s practice, based on the interpretation of the relevant provisions of the Regulation by the previously assigned military judge and the office of the convening authority, resulted in all pleadings—classified or not— undergoing a more laborious classification review intended for classified (or arguably classified) filings. As a result, the military judge found that compliance with DOD’s timeliness standard has, since at least 2017, been the exception rather than the rule. In this ruling, the military judge ordered that commencing on January 16, 2019, the OMC Trial Judiciary’s Chief Clerk will instead send all filings that do not require a classification security review directly to the OMC Webmaster for posting within one business day of filing. Further, per the regulation, filings requiring a classification security review will be sent to the inter-agency review team to coordinate the classification review. Implementation of the military judge’s ruling is expected to reduce the volume of documents submitted for security classification review and thus may improve the timeliness of posting information to OMC’s website. Current law and DOD guidance establish a framework in which DOD and military judges are to weigh the interests of public access to commissions’ proceedings against other considerations, including national security. For example, paralleling the statutory requirement for public access found in the Military Commissions Act of 2009, DOD’s Regulation for Trial by Military Commission states that its goal is to make commissions’ proceedings accessible to the public to the maximum extent possible, consistent with the interests of national security, the rights of the accused, and other interests protected by law. Standards for Internal Control in the Federal Government state that agencies should identify and analyze risks related to achieving their defined goals. These standards also maintain that—based on an agency’s assessment of risks—it should design specific actions as part of their response to the risks. However, DOD has not yet assessed the tradeoffs made by maintaining its current approach in pursuit of its goal of maximizing public access to the extent possible versus expanding public access by implementing other options. This is because the department has not yet identified these options and analyzed the risks associated with them for expanding public access. For example, we spoke to senior DOD officials who expressed strong support for public access to commissions’ proceedings. While they were not necessarily opposed to the concept of expanding public access, they did express concerns about the potential risks and challenges associated with how it may be achieved. Specifically, according to the former Acting Convening Authority, open and transparent commissions’ proceedings are “very important,” adding that public access must be weighed against the need to protect the proceedings’ large amounts of classified information. Similarly, the current Chief Prosecutor for Military Commissions stated that public access to commissions’ proceedings is “hugely important” and that they are “owned by the American People,” but also noted the importance of protecting classified information, especially the sources and methods of the intelligence community. Further, the current head of the Military Commissions Defense Organization, while acknowledging the necessity of processes to protect classified information, stated that “nothing is more important” than public access to the proceedings, calling them “the most important cases of our lifetime.” While these officials generally acknowledge that there are tradeoffs to be made, for example, in facilitating public access while protecting classified information, they have not identified how this could be accomplished or assessed the extent of the tradeoffs associated with any potential options for expanding public access to proceedings. As discussed previously, there are a number of potential options for expanding public access—well supported by victims, their family members that we surveyed, and non-government stakeholders. However, DOD officials have cited various tradeoffs, in the form of concerns over resources and national security, among others. While DOD officials’ concerns may be warranted, until it fully assesses these tradeoffs by identifying and analyzing the potential risks and challenges, it may be missing an opportunity to expand public access. For example, DOD officials have expressed concern with the potential cost and logistical challenge of expanding the viewing gallery on NGSB. However, DOD officials have not assessed such options for increasing public access to proceedings at NSGB while weighing the risks of doing so—such as cost or potentially delaying hearings—and not doing so—such as the current situation, with hundreds of victims and family members who have not been able to attend hearings. Our prior work on leading practices for effective strategic planning has also shown that agencies should define strategies that address management challenges and identify resources needed to achieve their goals. However, according to DOD officials, the department has not developed a strategy that explains how DOD will achieve its goal of maximizing public access to the military commissions’ proceedings in the context of public access challenges and the expected increase in demand for public access, once the cases’ trial phases begin. For example, DOD officials acknowledged that there are large populations of victims and family members who are “underserved” by the current number and locations of CCTV sites and that they need to be expanded. Further, the former acting convening authority noted that there would be a substantial amount of time required to plan for additional sites. Some DOD officials estimate that there will likely be 12-24 months advance notice before trials are held and therefore believe that this will provide sufficient time to develop a strategy that addresses challenges with opening additional sites. However, based on our discussions with DOD officials, this may not be enough time given the substantial planning and coordination that will need to take place within and outside the department on such efforts and the lengthy lead time typically needed to secure additional resources through DOD’s budget process. For example, DOD officials told us that they do not have many facilities anymore in urban communities, which necessitates that they have partners in these areas to facilitate additional CCTV sites. DOD officials said that they have tried working with government officials in New York City—a city with a high concentration of victims and family members—to identify ways to expand options for remote viewing of proceedings. However, DOD officials said that the coordination has been challenging, given management challenges—such as finding adequate space that is accessible for victims, family members, and the media—and required resources—such as reimbursing the City of New York for required security. In addition, while other agencies’ facilities could potentially be used, DOD officials noted that they have not begun coordinating with other agencies because the trial dates are currently unknown. But, given the logistical constraints and budget challenges, if DOD waits until the announcement of a trial dates, the department runs the risk of not having adequate time to plan and budget for a new CCTV site in New York City or any other appropriate location. This example illustrates the complexities of addressing public access, the usefulness of assessing the tradeoffs between DOD’s current approach to public access and options for expanding access, and a strategy that addresses management challenges and identifies needed resources. Until DOD comprehensively identifies and analyzes the risks of maintaining its current approach compared with those posed by potential options for expanding public access, it cannot be assured that it has met its objective of maximizing public access to the extent possible. Furthermore, until DOD develops a strategy, as appropriate, to deal with potential options and describes how the department plans to achieve its public access goals, it cannot ensure that it is well-positioned for the substantial increase in demand for public access that is anticipated when the commissions’ proceedings move into the trial phase. With the responsibility to carry out military commissions’ proceedings for cases that many believe to be the most consequential in United States history, DOD also has—according to its guidance—the responsibility to provide the public with as much access as possible, consistent with national security interests. Although this is a complex set of responsibilities, DOD has facilitated public access to commissions’ proceedings in a variety of ways. These complexities and constraints notwithstanding, there are a number of challenges posed to the public’s ability to access commissions’ proceedings and obtain information about the proceedings. While there are potential options to address these challenges, there are also potential risks that need to be assessed. Whether or not DOD should expand public access—as outlined by these potential options—is a determination the department must make. Given that the public’s demand for access will most likely increase substantially when the commissions’ enter into their trial phases, the longer DOD waits to determine its strategy, the greater the risk of not fully meeting the demand from victims and family members, non-government stakeholders, and the general public. The Secretary of Defense should ensure that the Deputy Secretary of Defense assesses the tradeoffs of potential options for expanding public access to military commissions’ proceedings by identifying and analyzing associated risks, and, as appropriate, developing a strategy to implement any viable options. We provided a draft of this report to the DOD, Department of Justice, and relevant intelligence agencies for review and comment. In written comments provided by DOD (reproduced in appendix IV), DOD concurred with our recommendation, noting planned actions to address it. DOD and certain intelligence agencies also provided technical comments, which we incorporated in the report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Defense, the Office of Military Commissions, Department of Justice, and four relevant intelligence agencies. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3604 or FarrellB@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report describes (1) how the Department of Defense (DOD) currently facilitates public access to military commissions’ proceedings; (2) the challenges, if any, that the public faces in gaining access to or obtaining information on these proceedings; and (3) what is known about potential options to address public access challenges, including any related tradeoffs. Specifically, the military commissions’ cases included in our review are 9/11: Khalid Shaikh Mohammad et al (2), USS Cole: Abd al-Rahim Hussein Muhammed Abdu Al-Nashiri, Majid Shoukat Khan, Abd al Hadi al-Iraqi, and Ahmed Mohammed Ahmed Haza al Darbi. To address our first objective, we reviewed relevant guidance, policies, and regulations related to public access to military commissions’ proceedings. We attended military commissions’ proceedings at U.S. Naval Station Guantanamo Bay (NSGB) from April 30, 2018, to May 4, 2018 to observe how the public accessed and viewed the proceedings, in person. During this visit we also visited the facilities relevant to public access. For example, in the Expeditionary Legal Complex, where proceedings are held, we inspected the courtroom where hearings occur, discussing the equipment used to facilitate the 40-second delay used to ensure that classified information is not transmitted to Closed Circuit Television (CCTV) sites during open hearings. We also inspected the gallery, from which the public watches hearings. In addition, we visited facilities where certain of the accused are detained, discussing with DOD officials the access granted by the department to visiting victims and family members and non-government stakeholders. We also discussed key issues with DOD officials, such as the Chief Prosecutor and the Military Commissions Defense Organization. To observe how the public utilized remote viewing sites we viewed military commissions’ proceedings remotely at one CCTV site, and visited another. These include Fort Meade, Maryland, which is a site for victims and their family members, as well as being the site for use by the media, non-governmental organizations, and members of the general public. In addition, we visited the Norfolk Naval Station, Virginia CCTV site, which is open to victims and family members only. In addition to watching the hearings, we spoke with Office of Military Commissions (OMC) representatives at the sites regarding their responsibilities and they provided us with an overview of how the sites operate. In addition, to determine what information was available on OMC’s public website and how it is organized, we reviewed its content, including the portion of the site reserved for victims and their family members. Further, to obtain information on how public access is provided in federal criminal courts, we conducted interviews with officials from the Department of Justice and the Administrative Offices of the U.S. Courts, also discussing with these organizations whether they provided support to DOD’s public access procedures for the commissions’ proceedings. To address our second objective, we reviewed applicable sections of the U.S. Constitution, relevant case law, executive orders, DOD guidance and reports from experts on public access to military commissions’ proceedings to understand the role that current laws, policies, and judicial precedence play in decisions about public access to military commissions’ proceedings. We then took selected examples of public access issues at military commission proceedings and compared them to the access afforded to the public at terrorism trials held in U.S. federal courts. To identify and understand any challenges facing public access, we obtained the perspectives of both victims and their family members and other non-government stakeholders on any challenges associated with public access to commissions’ proceedings. We developed a non- generalizable survey to obtain perspectives on public access from a sample population of victims and their family members associated with terrorist attacks being adjudicated by military commissions’ proceedings, such as the attacks on the USS Cole and September 11, 2001. See appendix II for further details regarding our survey of victims and family members. We also developed a standardized set of 10 questions that was used to obtain the perspectives of 55 selected non-government stakeholders on challenges to public access to military commissions’ proceedings. The questions were delivered to these stakeholders in the form of a self- administered questionnaire. To identify the non-government stakeholders included in our review, we first obtained a list of the non- governmental organizations that DOD has approved to observe military commissions’ proceedings in-person at NSGB. These organizations include victim advocacy groups, universities, civic organizations, and independent professional associations. During the course of our review, we identified additional individuals with relevant expertise, such as legal and national security policy experts and members of the media whom we also asked to complete our self-administered questionnaire. We pre-tested the self-administered questionnaire with four non- government stakeholders to ensure functionality and ease of understanding—after which we distributed the questionnaires via email to the remaining non-government stakeholders included in our review. Of the 55 non-government stakeholders who received our questionnaire, 25 completed it. The analysis was conducted by two analysts who reviewed and coded responses according to a pre-determined coding scheme. A third analyst was used to reconcile any conflicting conclusions from the first two analysts. The results of our analysis were used to describe non- government stakeholders’ perspectives in the report, as appropriate. We supplemented data obtained through our survey and self-administered questionnaire with interviews of victims and their family members, DOD officials, and observers from non-governmental organizations to better understand their perspectives. To assess the timeliness of information posted on OMC’s website, we gathered and analyzed data from an inter-agency review team that reviews documents to be posted on OMC’s website, as well as the website itself. In regard to data from the inter-agency review team, we obtained and analyzed data on when court documents were filed with OMC and the date on which the inter-agency review team returned them to OMC for posting; comparing that amount of time to a timeliness standard laid out in DOD’s Regulation for Trial by Military Commission (Regulation). According to the Regulation, DOD is supposed to post documents to the OMC website generally no later than 15 business days after documents have been filed with OMC’s Trial Judiciary, known as the “file date.” In regard to our analysis of data from OMC’s website, we collected this information using a “web-scraping tool” that we developed to regularly visit OMC’s website and capture data about a court document’s file date and the date on which it was posted on OMC’s website. We selected these two dates because they allowed us to compare the time DOD took to post court documents to the department’s timeliness standard. Using our analysis of data from the review team and OMC’s website, we determined the extent to which DOD posted court documents in a timely manner. Please refer to appendix III for additional details on the scope and methodology for our collection of data using the web-scraping tool and our analysis of these data. For data provided by DOD, we performed a number of assessments. As a result of discussions with the Defense Intelligence Agency about the timeframes and completeness of available data, the agency clarified timeframes and explained why the data are not fully complete. As a result of these assessments, we determined that data from DOD on timeliness of information posted to the commissions’ website are sufficiently reliable to serve as one of several sources of information used to determine that DOD faces challenges in the timeliness with which it posts court documents to the commissions’ website. In addition, through discussions with OMC officials about the way information is added to the commissions’ website, we determined that the data we independently collected and analyzed from the website are sufficiently reliable to serve as another source of information used in our determination of challenges that DOD faces. To address our third objective, we reviewed relevant reports to identify potential options for expanding public access to commissions’ proceedings and any concerns associated with doing so. To determine potential options for expanding public access to the commissions’ proceedings, we obtained the perspectives of victims and their family members, other non-government stakeholders, and DOD officials on (1) what potential options for expansion or improvement exist, and (2) any associated concerns with potential options for expansion or improvement. We conducted a survey of victims and their family members to determine the extent to which respondents support various options for expanding public access and their views on the timeliness of court document postings to OMC’s website. Similarly, we provided standardized question sets to non-government stakeholders and analyzed responses from the completed questionnaires to determine the extent to which respondents support various options for expanding public access as well as their views on other issues, such as the timeliness with which court document are posted to OMC’s website. Further, to examine the potential risks associated with these options for expansion—and ways to mitigate those risks—we discussed these potential options with DOD officials. Finally, we asked OMC officials to identify any DOD-led efforts to assess the current level of public access to commissions’ proceedings. We then compared any related efforts with Standards for Internal Control in the Federal Government, which state that agencies should identify and analyze risks related to achieving its defined objectives, and to develop leading practices for sound strategic management planning. Further, we compared any related DOD efforts to leading practices of effective federal strategic planning, which we derived in part from the Government Performance and Results Act (GPRA), as updated by the GPRA Modernization Act of 2010, associated guidance, and our prior work. To assess the extent to which DOD has applied selected principles of effective federal strategic planning in its facilitation of public access to military commissions’ proceedings, we compared actions DOD has taken to address challenges that it faced with meeting its goal of maximizing public access, consistent with the interests of national security, to these leading practices of effective federal strategic planning. We conducted this performance audit from January 2018 to February 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. To obtain information about the perspectives of victims and their family members on public access to military commissions’ proceedings, we administered a survey to the memberships of three victim’s organizations. In the survey questionnaire, we asked victims and their family members to provide their perspectives on the different ways they access information about, or participate in viewing military commissions’ proceedings. We administered the survey from July to September 2018. A reproduction of the questions and answers in the questionnaire and aggregate responses from the survey are included in this appendix. We informed our methodology approach and survey development through interviews and other communications with representatives from eight victim’s organizations. From these interviews we gathered information from the organizations about their membership, such as, the number of members, criteria for becoming a member, and how information about the members was recorded and stored. We also ascertained their willingness to share contact information for their membership with us for the sole purpose of administering the survey. We defined and identified the survey’s target population of victims and family members through interviews with victims’ organizations whose memberships were impacted by the attack on the USS Cole, the events of 9/11, or other terrorist attacks for which there are military commissions cases being tried or that have been completed. Our survey population was composed of the memberships of the Department of Defense’s (DOD) Victim and Witness Assistance Program (VWAP) (1,928 eligible members), which includes victims who were impacted by the attack on the USS Cole, the events of 9/11, or other terrorist attacks, for which Hadi Al-Iraqi is accused, as well as Massachusetts 9/11 Fund, Inc. (470 eligible members), and 9/11 Families for Peaceful Tomorrows (200 eligible members). Membership in these organizations, and inclusion in our survey population, was limited to those family members or surviving victims who chose to join one or more of these organizations. In addition, we added 42 other qualifying victims and family members (who may not have been members of the three organizations) that we identified in answers to a survey question that respondents were asked. Our survey’s population totaled 2,640 victims and family members, and we attempted to contact each one in our survey. Our survey population was limited to the memberships of these organizations because of concerns from some other victims’ organizations about the applicability of their data. However, many more people were significantly impacted by the events of 9/11 than are represented in our survey population. For example, according to the World Trade Center Health Program there are 88,484 individuals who have received medical treatment for 9/11 related injuries or illnesses. Thus, the survey results presented in the body of this report represent the views of only those responding, and are not generalizable to any broader population because it is difficult to determine with certainty the total population that was impacted by the events of 9/11 and would therefore have an interest in access to military commissions’ proceedings. We informed the development of our methodological approach and the actual questionnaire through four meetings with eight victims and their family members during our visit to Naval Station Guantanamo Bay (NSGB). In these meetings, we piloted an interviewer administered questionnaire that included items that (1) related to their views on various topics related to the military commissions’ proceedings, and (2) solicited input on the best approaches for gathering the views of victims and family members. These meetings confirmed that a survey would be a valuable method for gathering the views of a broad range of victims and family members and informed the development of a draft instrument for further pre-testing. In developing, administering, and analyzing this survey, we took steps to minimize the five types of potential errors that the practical difficulties of conducting any survey may introduce. Because we surveyed all members of the population we identified, there was no statistical uncertainty in our estimates due to sampling error. A different issue, measurement error, can result from differences in how a particular question is interpreted, and the sources of information available to respondents. We conducted 4 pre- tests of the draft questionnaire with 4 victim family members and made revisions to (1) ensure that survey questions were clear, (2) obtain any suggestions for clarification, (3) determine whether victims and their family members would be able to provide responses to questions with minimal burden, and (4) ensure that the survey was comprehensive and unbiased. We also provided GAO contact information in our communications for respondents who had questions about the survey or experienced technical problems. To minimize the effects of coverage error—the exclusion of some eligible members of the population, duplicate responses, or inclusion of ineligible members—we consulted the three victims’ organizations to determine the coverage of their membership lists and what survey methodology options for contacting them existed based on their willingness to provide us with contact information for their membership. All three of the organizations preferred to retain their member contact information citing privacy concerns, but agreed to send their membership unique usernames and passwords provided by GAO via email that their members could use to access the survey. Additionally, DOD VWAP also agreed to send postal mail questionnaires provided by GAO to approximately 500 of their members who did not have email addresses on record. GAO also provided an introductory email or letter, and postal questionnaires. Survey respondents received the email and used their associated username and password to access the survey website, and before opening their questionnaire, were required to change their password to further prevent unauthorized access to their responses. Those respondents who received postal mail questionnaires were given the option to complete the paper questionnaire or to log into and complete the web-based version. Because we did not obtain contact information from the organizations we worked with we were unable to determine if more than one survey was sent to any of the respondents. For example, if a respondent was a member of both 9/11 Families for Peaceful Tomorrows and DOD VWAP it is possible that they would have received two sets of unique usernames and passwords. However, we did include statements in the introductory email that directed respondents to disregard the email if they had already received a copy of the survey. Non-response error can result when a survey fails to capture information from all population members selected into the survey. To encourage survey response, for emails that were undeliverable, their respective organizations contacted them via telephone and attempted to obtain new email addresses. We were also able to send reminder emails out to respondents who were members of the two private victims’ organizations. However, DOD VWAP preferred not to send reminder emails to its members because of concerns of being overly intrusive. In an effort to increase the number of respondents to the survey we included a question asking respondents if they wanted to provide contact information for any other victims and family members who might be eligible to respond to the survey, and we administered the survey to them as well. We received 248 responses to the 2,640 questionnaires that were sent out, which after removing two ineligible population members confirmed to have died, resulted in a response rate of 9.4 percent. We anticipated a fairly low response rate because in our discussions with the leadership of each of the victims’ organizations they had pointed out that this population was quite private. In addition, the issues were sensitive, and not all organization members may wish to engage in discussions or surveys regarding activities related to the terrorist events. There were 70 responses by mail and the remaining 178 responses were to the web based survey. Also, there were 11 partial, but usable responses and 22 partial, but not usable responses. Finally, to limit the possibility of processing error, survey responses were checked for invalid or illogical answer patterns, and edits were made as necessary. All analysis programming was verified by a separate data analyst. Reproduced below are the questionnaire text and question and answer wording presented to victims and family members in our survey. The percentage of responses for each answer to a question is displayed. Not all 248 respondents to the survey answered each question—some questions were only asked of a subset of respondents giving a qualifying answer to an earlier question, and not all qualifying respondents may have answered a particular question. Percentages may not sum to 100 percent due to rounding. Narrative answers to open-ended text questions are not displayed for brevity and to limit the possibility of identification of individual respondents. This survey is being done by the Government Accountability Office, or GAO. GAO is sometimes called the Congressional Watchdog because it reviews federal programs for the United States Congress. Congress directed us to consider if it’s possible, and a good idea, to expand the public’s access to Military Commission proceedings (usually referred to as hearings) that are open to the public. As part of this effort, Congress also asked us to speak with those affected by terrorism and their families. We are very appreciative of your willingness to respond to this survey. We will combine your answers with those of many others, and we will not publish any information that could identify you. We will not share any identifiable information from this survey unless required by law or a member of Congress. If you have any questions about this survey, or the GAO study, please contact ________, an analyst on this study, at proceedings@gao.gov ________. 1. To better understand your perspective on the events of 9/11 or the attack on the USS Cole, which one of the following best describes you? Family member of a victim (parent, sibling, daughter, son) Family member of a victim (aunt, uncle, niece, nephew, grandparent) We used these data collected from the commissions’ website in three analyses, as discussed below. Analysis, as of June 19, 2018: According to our research, the first document recorded as being filed with the Trial Judiciary, and included in our scope, on the current OMC website has a file date in April of 2011. On June 19, 2018, we began data collection using the web-scraping tool, as described above. While the website provides a file date for all documents, the website does not provide a date when documents are uploaded. Thus, for documents uploaded before June 19, 2018, we were not able to assess the Department of Defense’s (DOD) timeliness performance with data from the web-scraping tool. However, our analysis as of June 19, 2018, allowed us to asses other aspects of performance. Specifically, we determined the following: On June 19, 2018, the number of documents that had been filed with the Trial Judiciary, number that had been uploaded, or number that had yet to be uploaded. On June 19, 2018, the number of documents that had not been uploaded within 15 business days of the file date. We refer to these documents as having missed DOD’s 15 business day timeliness standard. On June 19, 2018, for documents that missed the 15 business day standard, the median number of days that they were uploaded after the timeliness standard. On June 19, 2018, DOD’s performance in these parameters, for five different types of court documents: motions, rulings, transcripts from open hearings, transcripts from closed hearings, and docket-related documents. Recent performance analysis, June 19 to November 19, 2018: While the website does not provide a date when documents are uploaded, our web-scraping tool provided this information for each document uploaded on or after June 19, 2018. Thus, for the five months we used the tool, we were able to assess DOD’s timeliness performance for each document filed with the Trial Judiciary or uploaded. For these documents, we determined the following: The number and percentage of documents that were uploaded after DOD’s 15 business day timeliness standard. For documents uploaded after the 15 business day standard, the median number of days that the standard was missed. DOD’s performance in these parameters, for five different types of court documents: motions, rulings, transcripts from open hearings, transcripts from closed hearings, and docket-related documents. Docket availability analysis, June 19 to November 19, 2018: According to DOD guidance and an OMC official, there is a set of documents that list the legal motions on which the military judge plans to hear arguments from the prosecution and defense during a specific hearing. We refer to these documents as docket-related documents. This set of documents includes dockets and amended dockets, among others, that are a sub-category of all the court documents that we discuss in this report. For hearings that occurred during the five months in which we used the web-scraping tool, we reviewed the commissions’ public website to identify hearings that occurred during this timeframe, cross-referencing the hearings with the posted court documents to identify docket-related documents related to these hearings. Because docket-related documents for a specific hearing share an alphanumeric designation, we were able to use this information to determine DOD’s timeliness performance for posting docket-related documents for these five hearings. For these documents, we determined the following: For each hearing that occurred from June 19, 2018, whether the relevant docket-related documents for a hearing were posted at least one day before the hearing for which those docket-related documents list the motions to be argued in the hearing. In addition to the contact named above, Kimberly Mayo, Assistant Director; Tracy Barnes; Kathryn Bassion; Steven Campbell; Signe Janoska-Bedi; Jill Lacey; Ronald La Due Lake; Amie Lesser; Ying Long; Ned Malone; Samuel Moore; Christina Murphy; Samuel Portnow; Carl Ramirez; Clarice Ransom; Paul Seely; Chris Turner; and John Van Schaik made key contributions to this report.", "summary": "DOD is in the pre-trial phase of the military commissions' proceedings it is conducting to try the alleged perpetrators of terrorist attacks on the USS Cole and September 11, 2001. The Military Commissions Act of 2009 specifies that proceedings shall be publicly held unless the judge makes findings that justify a closed session, such as national security concerns. The National Defense Authorization Act for Fiscal Year 2018 included a provision for GAO to study the feasibility and advisability of expanding access to commissions' proceedings that are open to the public. This report describes (1) how DOD currently facilitates public access to proceedings; (2) challenges the public faces in gaining access to or obtaining information on proceedings; and (3) what is known about potential options to address public access challenges, including any related tradeoffs. GAO analyzed relevant laws and guidance; conducted a non-generalizable survey that received responses from 248 victims of terrorist attacks and their family members; collected data from DOD's website to analyze timeliness of court document postings; and interviewed relevant DOD officials and other government and non-government stakeholders. The Department of Defense (DOD) currently facilitates public access to and information about military commissions' proceedings at Naval Station Guantanamo Bay (NSGB) in Cuba by: communicating directly with victims and their family members about hearings; enabling selected members of the public to view proceedings in-person; providing five sites in the United States to view proceedings remotely via closed circuit television (CCTV); and making information such as court documents available on the Office of Military Commissions' website. The public faces various challenges in gaining access to military commissions' proceedings or obtaining information about them. First, some aspects of the proceedings limit public access, but addressing them is largely outside of DOD's control. For example, proceedings, by law, are held on NSGB—a location that is largely inaccessible to the general public. Further, cases currently before the military commissions have spent 4-10 years in pre-trial hearings with trials yet to be scheduled, which some suggest has lessened media coverage and public visibility. Second, there are other challenges that DOD officials have acknowledged that they have a greater ability to address. For example, the courtroom gallery is limited to 52 seats for those permitted to travel to NSGB. Additionally, all five CCTV sites are located within a span of 600 miles on the East Coast of the United States. However, victims and their family members—the primary intended users of these sites—often live a significant distance from these locations. A number of options may potentially address some of the public access challenges identified. DOD could potentially expand the viewing gallery to accommodate more people as part of an ongoing project to renovate the NSGB courtroom. However, DOD officials cautioned that it would require a commensurate increase in the lodging needed to house more visitors, which may not be supported by current levels of resources. Further, DOD has two potential options for addressing challenges with the remote viewing of proceedings. First, DOD could potentially increase the number and geographic dispersion of CCTV sites. Second, DOD could potentially maximize public access by broadcasting proceedings via the television or internet. DOD officials acknowledged that both options are possible and likely would require a relatively small outlay of resources. However, broadcasting proceedings via the television or internet is currently prohibited by DOD's regulation, and DOD officials were especially concerned with the security implications of this option. DOD has not assessed the tradeoffs nor identified or analyzed the risks of options for expanding public access to military commissions' proceedings. Consequently, DOD has not developed a strategy to address challenges or identified the resources needed to achieve its public access goals. Until DOD does so, it cannot be sure that it is meeting its goal of maximizing public access and may not be prepared for the potential increased demand for public access that is anticipated when proceedings move into the trial phase. GAO recommends that DOD identify and analyze the risks associated with potential options for expanding public access to proceedings, and develop a strategy, as appropriate, for how it will meet its public access goals with the expected increase in public interest. DOD concurred with the recommendation.", "document_type": "gao"}
{"report": "Overall, component agencies reviewed proposals and notified awardees within the required time for 12,890 of the 15,453 SBIR and STTR awards that we reviewed (84 percent), for fiscal years 2016 through 2018. The Small Business Act and SBIR/STTR policy directive require most agencies notify applicants of the agency’s decision within 90 calendar days and require NIH and NSF do so in 1 year. Agencies notified awardees after the required time period for 2,533 of 15,453 awards (16 percent). Three of the 28 component agencies met the notification requirement for every award in the data we reviewed, and nine additional component agencies did so for at least 90 percent of their awards. The remaining 16 component agencies met the notification requirement for less than 90 percent of their awards. Table 2 lists the mean and median notification times and the percentage of awardees notified within the required time period for each component agency. Some notifications occurred within days after the required time period, while others occurred months later. For example, all of the notifications by the Department of Education from fiscal year 2016 through 2018 that took longer than 90 days occurred in 91 days. Department officials attributed the one day difference to interpreting the 90-day requirement as a 3- month requirement. Similarly, all of the notifications for Army STTR awards that occurred after the 90-day requirement occurred within 92 days. Of the 2,533 awards with notifications after the required time, notifications occurred on average about 1 ½ months later. During the 3 fiscal years that we reviewed, some component agencies had substantial changes from year to year in the percentage of awardees that they notified within the required time period, while other component agencies consistently notified about the same percentage of awardees. For example, the Department of Energy’s Office of Science and the Army SBIR program each had a single fiscal year during which they notified less than 50 percent of awardees within the required time period, substantially less than during the other fiscal years we examined. Table 3 describes the percent of awardees notified within the required time by each component agency for each of the 3 fiscal years we examined. Agency officials described several factors that affect the time spent reviewing proposals and notifying awardees, including (1) the availability of reviewers, (2) the number of proposals to review, and (3) other agency- specific factors. Availability of reviewers. Officials from some component agencies we interviewed said the availability of agency staff or external reviewers affected the time they spent reviewing proposals. For example, USDA officials told us that the agency cannot notify awardees within 90 days because they need additional time to identify and recruit experts for their external peer review system. USDA officials compared their review process to that of the NSF and NIH, the two agencies that are directed to complete proposal review and notification within 1 year. Similarly, Navy officials said that the availability of reviewers was the biggest variable in completing their proposal review and notification process. These reviewers are Navy employees who contribute part of their time to reviewing SBIR and STTR proposals while continuing to perform their regular duties. According to Navy officials, although they give reviewers deadlines based on the number of proposals they have to review, conflicts with their regular duties or higher priority tasks may cause reviewers to miss their deadlines. In contrast, Department of Education officials said that they identify and train reviewers before the agency receives proposals so that the reviews may begin as soon as possible. Other agencies, however, may not know what areas of expertise reviewers will need until the agency has examined the proposals it received. Number of proposals. Officials from some component agencies we interviewed said the number of proposals they receive affected the time spent reviewing proposals and notifying awardees. For example, officials from the Department of Transportation said that the number of proposals they receive can range between two and 40, which makes it difficult to predict the workload of agency evaluators who perform the proposal reviews. Similarly, National Institute of Standards and Technology officials said that the number of proposals they receive fluctuates from year to year. Because agencies must review all proposals that meet the minimum requirements, an increase in the number of proposals directly increases the workload of proposal reviewers. Other agency-specific factors. Some component agency officials identified factors specific to their agency or process as factors affecting the time needed. For example: Two component agencies within the Department of Health and Human Services—the Centers for Disease Control and Prevention (CDC) and Food and Drug Administration (FDA)—notified none of their awardees within the required time in fiscal years 2016 through 2018. CDC and FDA participate in the solicitation and review process led by the NIH. However, while the NIH has 1 year to notify awardees, these agencies are required to notify awardees within 90 calendar days. CDC officials said that participating in the longer NIH program is more efficient than creating their own review process and allows them to leverage additional programs at NIH that support small business awardees. Environmental Protection Agency officials told us that their review process includes three consecutive reviews, which leads the agency to regularly request waivers to exceed the 90-day notification requirement. These reviews include an administrative review for responsiveness to the solicitation, an external peer review process, and an internal review by the SBIR program office. Some agency officials also identified continuing resolutions, sequestration, or government shutdowns as factors that could slow proposal review. Proposal review and notification activities could be affected because the availability or amount of funds for agency activities is uncertain in these instances. For example, a Defense Microelectronics Activity official told us that their agency generally completes its proposal review process within 90 days, but does not notify awardees until it has determined funding availability for awards later in the fiscal year. National Institute of Standards and Technology officials described a delay notifying one awardee, a replacement awardee, due to the initial awardee being determined ineligible during a pre-award assessment. The agency made a replacement selection immediately, but this replacement awardee was notified approximately 20 days after the 90-day requirement. Overall, component agencies issued 11,710 of the 15,453 awards we reviewed (76 percent) within the recommended time period, for fiscal years 2016 through 2018. The SBIR/STTR policy directive recommends that most agencies issue an award within 180 days and recommends that NIH and NSF do so in 15 months. Agencies issued 3,743 of the 15,453 awards (24 percent) after the recommended time period. Three of 28 component agencies issued every award in the data we reviewed within the recommended time, and five additional component agencies did so for at least 90 percent of their awards. The remaining 20 component agencies issued less than 90 percent of their awards within the recommended time period. For the 3,743 awards that agencies issued after the recommended time period, the average award was issued about two and a half months after the recommended time. Table 4 lists the mean and median award issuance times and the percent of awards issued within the recommended time for each component agency. During the 3 fiscal years that we reviewed, some component agencies had substantial changes from year to year in the percentage of awards they issued within the recommended time period, while other component agencies consistently issued about the same percentage of awards within the recommended time period. For example, the Department of Energy’s Advanced Research Projects Agency-Energy issued no awards within the recommended time in each of the three years we examined. Table 5 describes the percent of awards issued within the recommended time period by each component agency for each of the 3 fiscal years we examined. Agency officials described several factors that increased the time spent issuing awards, including (1) additional time needed to issue certain types of contracts, (2) the availability of grants and contracting officers, (3) delays coordinating among agency officials, (4) the responsiveness of awardees, and (5) the availability of funding for the awards. Cost reimbursement contracts. Officials from some component agencies we interviewed said that the contract type was a factor that affected the time needed to issue SBIR and STTR awards. Specifically, officials said cost reimbursement contracts took longer to issue because of the need to review the awardee’s accounting system in accordance with federal acquisition regulations. For example, officials from the Defense Advanced Research Projects Agency (DARPA) said cost reimbursement contracts routinely take more time to award than fixed- price contracts because of this accounting system review. According to DARPA officials, this review can add 45 days or more to the awards process. In February 2019, we found that the Department of Defense does not have a mechanism to monitor and ensure that contractor business system reviews and audits are conducted in a timely manner and recommended that the department develop such a mechanism. Our analysis of the SBIR and STTR award data confirmed that component agencies spent more time issuing awards identified as cost reimbursement contracts than issuing fixed price contracts. We found that SBIR and STTR awards identified as cost reimbursement contracts in the fiscal year 2016 through 2018 data took significantly longer to issue than those identified as fixed-price, as shown in figure 1. Fixed-price contracts took on average 152 days and cost reimbursement contracts took 231 days (79 days longer). Cost reimbursement contracts also took on average 40 days longer than contracts that were not specified as fixed or cost reimbursement. Availability of grants or contracting officers. The availability or experience of agency staff to negotiate the contract or grant can be a factor, according to some component agency officials. First, some officials said limited availability of grants or contracting officers was a factor in the time to issue awards and may result in delays. For example, officials from both Army program offices said that the workload for contracting officers is high, and SBIR and STTR awards are part of a larger contracting backlog. Similarly, officials from the National Institute of Standards and Technology and National Oceanic and Atmospheric Administration also said that the availability of grants and contracting officers is a pervasive issue for federal agencies that can affect award timeliness. Second, officials from some component agencies said that the contracting officer’s level of experience with small business awards affects the time needed to issue SBIR and STTR awards. Coordination among agency officials. Air Force officials said that the need for coordination among agency officials, such as between the contracting officer and proposal evaluators, can create delays. Because the proposal review and award process can require coordination among multiple officials who are not always immediately available, delays may occur as one official waits for input or information from another. Beginning in fiscal year 2018, the Air Force made changes to its proposal review and award process for a subset of awards that included scheduling dedicated time for reviewers, contracting officers, and other agency officials to jointly evaluate proposals and process awards. This change guaranteed the availability of agency officials and reduced the time needed for coordination among these officials. Overall, it allowed the agency to issue awards within a few days or weeks. According to agency officials, the Air Force awarded about 150 awards in 2018 through this process and they expect about one-third of Air Force awards in fiscal year 2019 and half of awards in fiscal year 2020 will use this expedited process. Responsiveness of awardees. Some component agency officials said that the responsiveness of the small business was a factor in delays. For example, officials from USDA said that the majority of SBIR grantees at USDA are first-time grantees who have never worked with the federal government, and this can extend the time it takes to issue the award. In order to receive an SBIR or STTR award, the small business must, among other things, submit a certification that it meets size, ownership, and other requirements. Delays in providing these certifications or other information required by the awarding agency can therefore delay award issuance. In our July 2018 report that reviewed DOD’s weapon-systems- related contracts awarded from fiscal year 2014 through fiscal year 2016, contracting officials stated that quicker contractor responses to requests for additional information could help reduce the time between when a solicitation is issued to when a contract is awarded. Availability of funding. Some component agency officials said that delays in determining the amount of funding available for small business awards due to continuing resolutions or delays in intradepartmental fund transfers may delay the issuance of awards. For example, NASA officials said that they estimate the agency’s R&D budget at the start of the fiscal year to calculate the amount required for SBIR and STTR awards. According to these officials, if NASA is operating under a continuing resolution at the start of the fiscal year, the estimate may be smaller than the final appropriated amounts. In this case, NASA would go back to its proposals to make additional awards from the pool of proposals that were rejected under the original estimate, and this would lead to longer issuance times for some awards. We provided a draft of this report to SBA and the 11 agencies that participated in the SBIR and STTR programs in fiscal years 2016 through 2018 for their review and comment. The SBA, Department of Defense, and Department of Education provided written comments that are reproduced in Appendix II, III, and IV. In addition, the Department of Energy, the NIH within the Department of Health and Human Services, Department of Transportation, and the National Institute of Standards and Technology within the Department of Commerce provided technical comments, which we incorporated as appropriate. The remaining agencies told us they had no comment. In its formal comments, the Department of Education stated that it has taken steps to ensure that future awardees will be notified within the required period. In their comments, SBA and the Department of Defense suggested phase I and II awards should be evaluated separately in future reports. In this report, we combined phase I and II awards because we did not find a statistically significant difference in notification time between phase I and II awards in the fiscal year 2016 through 2018 data that we examined. However, some analyses showed that phase II awards took longer to issue. We may further examine differences between phase I and phase II awards in subsequent reports. SBA also described the importance of minimizing delays between phase I and phase II awards. We did not evaluate the time between phase I and subsequent phase II awards in this report, but agree that the time between awards may be of interest in future reports because, as noted by SBA, the time between awards may affect small businesses' ability to retain key personnel. SBA also sought explanations for various dates and figures used in our analysis and we updated the report to include the definitions used when collecting award data and to describe our figures in more detail. The Department of Defense also stated that the SBIR and STTR policy directive does not explicitly include phase II awards in its 90 and 180-day timeliness requirements. However, we confirmed with SBA—the agency that issues the directive—that the 90-day requirement for notification of selection and the 180-day recommendation for award issuance apply to both phase I and phase II awards. The Department of Defense further stated that subsequent phase II awards could occur several years after the end of the initial phase II award and should not be included in the analysis of phase II awards. In this report, we took steps to eliminate these outliers from the data. We are sending copies of this report to the appropriate congressional committees, the Acting Administrator of the SBA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6888 or neumannj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This appendix describes the awards made by agencies participating in the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs, based on the data provided to GAO for fiscal years 2016 through 2018. These data include figures showing the (1) proposal review and notification time, (2) award issuance time, and (3) distribution of awards by fiscal year and phase. The fiscal year and phase figure describes the number of phase I and phase II awards issued in fiscal years 2016 through 2018 and is based on the first year of the award activities. For example, if an agency obligated funding to a phase II award in fiscal years 2017 and 2018, the award is counted among the fiscal year 2017 phase II awards. NIST participated in SBIR only. NOAA participated in SBIR only. Air Force participated in SBIR and STTR. Small Business Award Timeliness (Fiscal Year 2016-2018 Awards) Small Business Award Timeliness (Fiscal Year 2016-2018 Awards) Navy participated in SBIR and STTR. MDA participated in SBIR and STTR. DARPA participated in SBIR and STTR. DHA participated in SBIR and STTR. SOCOM participated in SBIR and STTR. DLA participated in SBIR and STTR. DTRA participated in SBIR and STTR. CBD participated in SBIR and STTR. NGA participated in SBIR and STTR. DMEA participated in SBIR and STTR. Education participated in SBIR only. Office of Science participated in SBIR and STTR. ARPA-E participated in SBIR and STTR. NIH participated in SBIR and STTR. CDC participated in SBIR only. FDA participated in SBIR only. DHS S&T participated in SBIR only. DNDO participated in SBIR only. DOT participated in SBIR only. EPA participated in SBIR only. NASA participated in SBIR and STTR. NSF participated in SBIR and STTR. USDA participated in SBIR only. In addition to the contact named above, Rob Marek (Assistant Director), Tind Shepper Ryen (Analyst-in-Charge), Nora Adkins, David Aja, Jenny Chanley, Robert Letzler, Anika McMillon, Amanda Postiglione, and Ben Shouse made key contributions to this report.", "summary": "Since the SBIR and STTR programs began in 1982 and 1992, respectively, federal agencies have awarded at least 162,000 contracts and grants totaling around $46 billion to help small businesses develop and commercialize new technologies. Eleven agencies participate in the SBIR program and five of them also participate in the STTR program. Each agency issues a solicitation requesting proposals at least once a year. Agencies then review proposal submissions and issue awards using grants or contracts. The SBIR and STTR policy directive recommends that most agencies issue awards no more than 180 calendar days from solicitation close. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to report on the timeliness of agencies' SBIR and STTR proposal review and award issuance. This report examines the time agencies spend issuing SBIR and STTR awards and the factors that affect the time spent, among other things. Within the 11 agencies, GAO reviewed 28 component agencies that participate in these programs. GAO analyzed agency-provided award data from fiscal years 2016 to 2018 for 15,453 awards and interviewed officials from the Small Business Administration and 26 of the component agencies. In fiscal years 2016 through 2018, agencies issued 11,710 of the 15,453 Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) awards we reviewed (76 percent) within the recommended time period. However, component agencies varied in the percentage of awards that they issued within the recommended time (see figure). Agency officials described a number of factors that can affect award issuance timelines, including: Some agencies use cost reimbursement contracts, which require additional agency review under federal acquisition regulations. Some contracting officers have limited expertise in issuing SBIR and STTR awards and their overall workloads can be heavy. Small businesses may be slow to respond to agency requests for information, such as requests for information needed to meet government contracting requirements.", "document_type": "gao"}
{"report": "States have flexibility within broad federal requirements to design and implement their Medicaid programs. For example, while states must cover certain mandatory groups and benefits, they have the option to cover certain other groups of individuals and benefits. States’ Medicaid plans outline the services provided, the populations covered by their programs, and how they implement and comply with other federal requirements. States share responsibility for oversight of Medicaid eligibility with CMS. States are primarily responsible for assessing applicants’ eligibility for, and enrolling eligible individuals into, Medicaid. These responsibilities include verifying individuals’ eligibility at the time of application, performing redeterminations of eligibility, and promptly disenrolling individuals who are no longer eligible. In verifying individuals’ eligibility, states must assess specified financial and nonfinancial information. Financial: Individuals applying for Medicaid generally must have an income below a certain limit. PPACA requires states to calculate the income for most nondisabled, nonelderly applicants using a uniform method based on modified adjusted gross income (MAGI), which is derived from a federal tax-based definition of income. States have more flexibility in determining how to calculate incomes for individuals whose eligibility is determined on the basis of age or disability, because their income is not calculated using MAGI-based methods. For example, states may disregard certain types or amounts of income for these MAGI-exempt populations. Additionally, individuals eligible on the basis of age or disability generally must also have assets—cash or real or personal property that are owned and can be converted to cash—below specified standards that vary by state. Nonfinancial: Individuals applying for Medicaid must also satisfy certain nonfinancial criteria. For example, to be eligible for Medicaid individuals generally must be residents of the state in which they are applying and must be either citizens of the United States or certain noncitizens, such as lawful permanent residents. States generally have flexibility in the sources of information they use to verify applicants’ financial eligibility and citizenship or immigration status. However, to the extent practicable, states must use third party sources of data for these verifications prior to requesting documentation from the applicant. When data from reliable third party sources are inconsistent with information from an application, the state must have processes in place to resolve these inconsistencies, such as through requesting additional documentation or accepting the applicant’s attestation. Additionally, states may accept self-attestation for some eligibility criteria, such as residency in the state and household composition (which is used in determining if applicants’ income is below the limit). Once a state determines that an individual meets relevant financial and nonfinancial eligibility criteria, the state enrolls the individual into Medicaid under one basis of eligibility. Examples of bases of eligibility include those applicable to children, pregnant women, individuals eligible for Supplemental Security Income (SSI)—a program that provides cash assistance to low-income adults and children with disabilities—and other low-income adults under age 65 in states that expanded their Medicaid populations under PPACA. (See table 1.) Since individuals may meet the criteria for more than one category and eligibility group, they could have more than one basis of eligibility. For example, a child who is pregnant could meet the criteria applicable to children and those applicable to pregnant women. However, a state would enroll each individual under one basis of eligibility. CMS regulations specify that when states determine applicants eligible based on MAGI criteria, they must notify these individuals of the benefits and services available through any MAGI-exempt bases of eligibility for which they may qualify, in order to provide the individual information about whether to request a MAGI-exempt eligibility determination. However, CMS officials explained that they advise states that they do not need to inform applicants of benefits and services under other eligibility groups if there is no meaningful difference in the benefits or cost-sharing that the individual would receive under one basis compared to another. CMS officials also noted that they have provided further guidance to states on assigning bases of eligibility, including that if an individual meets the criteria for more than one basis, the state should enroll the person into the most beneficial coverage in terms of factors such as the benefit package and out-of-pocket costs. In 2014, CMS issued a framework based on federal rules for states to use in developing their systems to assess individuals’ bases of eligibility. The framework describes a hierarchy for states to use in developing their eligibility systems that begins with bases related to receipt of other federal benefits, such as SSI and federally funded foster care and adoption assistance, which often result in automatic eligibility for Medicaid. Following these bases of eligibility, states are to assess eligibility for bases subject to MAGI-based income rules, and should first evaluate for mandatory coverage before evaluating for optional coverage. Federal rules allow for some exceptions to this sequence, such as when an individual who may be eligible for bases subject to MAGI-based income rules requests consideration under a MAGI-exempt basis to access certain additional benefits, such as long-term services and supports. CMS has historically operated two distinct, but complementary programs to oversee states’ eligibility determinations in the Medicaid program. The Medicaid Eligibility Quality Control (MEQC) program, which is implemented by states and overseen by CMS, was created in 1978 to monitor the accuracy and timeliness of Medicaid eligibility determinations in order to avoid inappropriate payments and eligibility decision delays. MEQC was also designed to identify methods to reduce and prevent errors related to incorrect eligibility determinations by having states review sample cases to independently verify eligibility criteria and then report the results to CMS. The Payment Error Rate Measurement (PERM) program is CMS’s process to estimate the national Medicaid improper payment rate in accordance with the Improper Payments Information Act of 2002, as amended, and Office of Management and Budget guidance. To calculate the Medicaid improper payment rate through PERM, CMS computes an annual rolling average of improper payment rates across all states based on a 3-year rotation cycle of 17 states each year. PERM is comprised of three components, including one that measures errors in state determinations of Medicaid eligibility. For fiscal years 2015 through 2018, CMS suspended MEQC and the eligibility component of PERM to provide states with time to adjust to eligibility process changes in PPACA; in its place, CMS required states to implement pilots to assess the accuracy of their eligibility determinations. As a result, CMS did not publish an updated national estimate of improper payments due to Medicaid eligibility errors for fiscal years 2015 through 2018. Eligibility reviews under PERM, which are conducted by a federal contractor, resumed in July 2017 for fiscal year 2019. In November 2019, CMS released an updated national estimate of Medicaid eligibility errors, which reflected results of the first 17 states reviewed under the new PERM process. Going forward, states are to resume MEQC reviews in the 2 years between their PERM reviews. The MEQC reviews will focus, in part, on specific areas of improvement for each state. For example, states might choose to focus on specific populations, such as whether pregnant women were assigned to the appropriate eligibility group, or specific processes, such as asset verification. The Medicaid statute includes a provision for CMS to recoup, or disallow, federal funds related to erroneous payments for ineligible individuals and overpayments for eligible individuals. The provision generally requires CMS to recoup funds from states for eligibility-related improper payments if the state’s eligibility error rate exceeds 3 percent. CMS has general authority to recoup funds from states when it determines that an expenditure of federal funding is not an allowable expense; according to CMS, however, this general authority does not apply to eligibility-related errors, given the separate specific statutory authority. Therefore, it is the view of the agency that CMS cannot recoup funds from states whose eligibility-related improper payment rate is below the 3 percent threshold. In addition to the PERM and MEQC oversight, state auditors review Medicaid eligibility determinations, including through audits conducted at the auditors’ initiative and as part of audits required by provisions of the Single Audit Act of 1984. To guide auditors in performing reviews under the Single Audit Act, the Office of Management and Budget issues a document referred to as the Compliance Supplement, which identifies important compliance requirements that the federal government expects to be considered as part of such an audit. Beginning in fiscal year 2014, the Compliance Supplement directed auditors to forgo review of eligibility determinations for individuals whose income is calculated based on MAGI. The supplement noted that testing was being performed under Medicaid eligibility review pilots, which would serve as CMS’s oversight during the initial years of PPACA implementation when the MEQC and the eligibility component of PERM were suspended. In June 2019, the Office of Management and Budget issued the 2019 Compliance Supplement that included changes to permit state auditors to test eligibility determinations of both MAGI and MAGI-exempt populations to ensure enrollees qualify for the Medicaid program and are in the appropriate enrollment category. The five selected states in our review considered similar factors when ranking the bases of eligibility to which individuals are assigned—such as bases related to children, pregnant women, or disabled individuals—but the resulting basis of eligibility in which individuals were placed could vary. Each of the five states ranked bases of eligibility by comparing how beneficial they were for enrollees across several key factors, and ordered the bases into a hierarchy starting with the most beneficial, according to officials. The states’ eligibility systems were programmed to apply these hierarchies in deciding each individual’s basis of eligibility; when an individual was potentially eligible for more than one basis of eligibility, the system would assign them to the basis highest in the ranking. The key factors the selected states considered in ranking the bases of eligibility, according to state officials, included (1) whether eligibility was related to the receipt of benefits from other programs, (2) the services provided through the benefit package, and (3) the financial implications for the individual. Eligibility related to other programs. The selected states ranked bases of eligibility associated with enrollment in other federal and state assistance programs at or near the top of their hierarchies. For instance, eligibility associated with receipt of SSI was generally at the top of the states’ hierarchies, and eligibility associated with receipt of federal foster care and adoption assistance benefits was ranked above other bases for which a child might be eligible. Services included in the benefit package. Bases of eligibility that conveyed additional benefits, such as long-term services and supports, were ranked higher. Similarly, bases that offered limited benefits, such as only covering family planning services or assistance with cost-sharing for Medicare beneficiaries (i.e., the Medicare Savings Program) were ranked lower in the selected states’ hierarchies. Financial impact. The selected states ranked bases of eligibility lower if they were associated with additional financial requirements for the individual, such as asset tests as a condition of eligibility, or out- of-pocket costs once enrolled. For example, bases of eligibility that required applicants to make copayments to receive certain services, or to pay a monthly premium, were ranked lower than those without such costs. Although the selected states considered similar factors when deciding an individual’s basis of eligibility for Medicaid, a similarly situated individual could be enrolled under a different basis of eligibility in one state versus another state. Decisions varied across states, in part, because of differences in (1) how states factored in the length of the enrollment period; and (2) the degree to which states’ eligibility systems and processes were integrated. Length of the enrollment period. Officials in selected states considered the length of the enrollment period when deciding bases of eligibility for certain populations, such as pregnant mothers (pregnant women who were also eligible as caretakers of dependent children). Pregnant women who are eligible for Medicaid have continuous eligibility, which guarantees enrollment through at least 60 days postpartum regardless of income changes. For this reason, Oklahoma enrolled pregnant mothers under a basis of eligibility applicable to pregnant women. In contrast, Virginia enrolled pregnant mothers under a basis of eligibility applicable to caretakers, because it has a 12-month enrollment period. However, pregnant women have continuous eligibility through at least 60 days postpartum regardless of income changes or whether they are enrolled as caretakers or on some other basis. As such, if a pregnant woman enrolled as a caretaker no longer met the income standard for a caretaker, for example, she could still remain eligible through her postpartum period. Alternatively, a woman enrolled under a pregnancy-related basis of eligibility would be redetermined for eligibility at the end of her postpartum period and could continue enrollment as a caretaker if she continued to meet the financial and other eligibility criteria. CMS noted that such variations in eligibility policies are allowable and expected among state Medicaid programs. Eligibility system integration. Differences in the degree to which selected states integrated their eligibility systems affected how individuals were assessed for potential bases of eligibility and potentially resulted in different eligibility determinations. Officials in four of our five selected states—New Mexico, Oklahoma, Tennessee, and Virginia—reported operating unified or integrated eligibility systems through which individuals could be considered for both MAGI and MAGI-exempt bases of eligibility. The fifth state, Maryland, had separate eligibility systems for MAGI and MAGI-exempt bases of eligibility, so an individual would need to apply through both systems to have all potential bases of eligibility considered. As such, an individual who is over age 65 and a caretaker of a dependent child would have to submit two separate applications to be assessed for all potential bases of eligibility in Maryland. Depending on the system to which he or she applied, that individual could be enrolled in a less beneficial basis of eligibility or denied eligibility for Medicaid. For example, the individual might be determined ineligible for full Medicaid benefits and enrolled in a Medicare Savings Program, in which Medicaid covers out-of-pocket costs related to Medicare benefits. (See fig. 1.) Our review of 47 state and federal audits across 21 states identified multiple issues affecting the accuracy of states’ Medicaid eligibility determinations. The accuracy issues identified in the audits we reviewed generally fell into nine broad categories, such as eligibility determinations made with incorrect or incomplete income or asset information, unresolved discrepancies between what applicants reported as their income and electronic data sources, and unidentified or unaddressed changes in circumstances, such as changes in household income or size. (See table 2.) Within these nine broad categories, the audits identified several specific accuracy issues, including states that were not conducting income checks for individuals reporting no income; not terminating the enrollment of individuals who had moved out of enrolling individuals who did not provide required information (such as proof of citizenship) on a timely basis; months or years behind schedule in conducting required eligibility not acting on—or not having adequate systems in place to detect— changes in enrollees’ circumstances that could affect eligibility, such as changes in income or household composition. See table 3 for examples of audit findings related to each of the accuracy issue categories, and appendix I for an overview of the key findings for each audit we reviewed. In some cases, the accuracy issues identified by auditors resulted in errors in eligibility determinations, such as instances when applicants were determined eligible even though their incomes were above the applicable limit, or instances in which the state did not enroll eligible individuals. However, in other instances, the accuracy issues identified by auditors did not result in erroneous eligibility determinations. For example, in some cases the audit found that a state determined that an applicant was eligible based on incorrect or incomplete financial information; however, auditors found that the applicant would have still been eligible for Medicaid even after reviewing additional financial information. In other cases, auditors found that eligibility determinations complied with state policies and federal requirements, but that changes in state policies— such as using additional data sources or checking sources more frequently—could provide more information that could be used to improve eligibility determinations. For example, audits in three states found that the quarterly wage data the states used to verify income did not detect certain nonwage income; that income could have been identified had the states chosen to use state or federal tax data as a verification source. Auditors in one of these states (Louisiana) also found that checking income data during individuals’ coverage period, such as on a quarterly basis, could have saved the state tens of millions of dollars in managed care fees for individuals whose incomes exceeded eligibility thresholds during their enrollment period. The selected states we reviewed reported having processes in place that were designed to avoid or address many, but not all, of the accuracy issues identified. The following are examples of the states’ processes related to specific accuracy issues. Incorrect or incomplete income or asset information. All five selected states we reviewed reported checking electronic data sources to verify income, including for individuals who report $0 in income. Officials from some states noted, however, that the electronic sources they have chosen to use do not include all relevant types of income, such as self- employment income. The five states also reported having electronic asset verification systems to verify financial assets, such as bank and retirement accounts for applicants subject to asset limits. One state (New Mexico) reported that it recently implemented an asset verification system that includes information from financial institutions, property ownership records, and vehicle licensing. Eligibility redeterminations not made in a timely manner. To help ensure that redeterminations are made in a timely manner, all five selected states reported conducting automatic redeterminations for at least some MAGI enrollees using electronic data sources to confirm continued eligibility. The proportion of MAGI enrollees whose eligibility was automatically redetermined ranged from about 10 to 80 percent. Officials from Virginia, which was cited by auditors in 2015 as having significant delays in conducting redeterminations, reported that automatic redeterminations have helped improve timeliness. Where automatic eligibility redeterminations are not conducted—such as for enrollees whose incomes could not be confirmed through electronic sources or who are eligible on a MAGI-exempt basis—the five selected states reported having systems in place to generate a redetermination packet or notice to be sent to enrollees prior to the end of their eligibility period. Officials reported that enrollees who do not complete their redetermination would be disenrolled, with states varying in how quickly they would take such action. For example, Oklahoma officials reported that the state automatically terminates enrollment for individuals who do not reply with the required information by the end of their coverage period. In contrast, Virginia officials reported that redeterminations for which no response was provided are kept open, pending eligibility worker action; the state’s systems do not automatically terminate enrollment. Unresolved income discrepancies. Officials in the five selected states reported that their eligibility systems automatically identify income discrepancies. For example, Oklahoma officials indicated that if there is more than a 5 percent difference in the income reported on the application and the income from electronic data sources, their system either alerts eligibility workers or automatically sends a request for additional information to the enrollee. Individuals enrolled in incorrect basis of eligibility. According to state officials, their eligibility systems have automated checks to reassess the eligibility for individuals reaching certain milestones, such as the maximum age for their basis of eligibility (i.e., children reaching adulthood and adults reaching age 65) and pregnant women who are approaching the end of their 60-day postpartum period. For example, to help ensure individuals are correctly assigned to the appropriate basis of eligibility, officials in Maryland noted that they apply system edits that preclude individuals who are pregnant, age 65 or older, or enrolled in Medicare from being incorrectly assigned to the new adult group. Unidentified or unaddressed changes in circumstances. Officials from the five selected states indicated that they generally had systems in place to identify if an enrollee had died or moved out of state. For example, officials from the five selected states reported conducting periodic checks of residency through the Public Assistance Reporting Information System—a federal data source that identifies individuals receiving benefits in other states—and following up with identified enrollees to see if they still reside in the state. None of the selected states conducted regular reviews to identify changes in MAGI enrollees’ incomes during the enrollment period, although one state—Oklahoma— planned to implement interim checks of income in response to a recent change in state law. Oklahoma also reported that it conducted quarterly checks of wage data for MAGI-exempt enrollees. Use of incomplete or incorrect information on household composition. The selected states generally did not have processes in place to detect accuracy issues related to household composition, although officials in four of the five states—Maryland, New Mexico, Oklahoma, and Virginia—noted that eligibility information from other benefit programs may be compared with Medicaid files to detect changes or discrepancies in household membership. In 1983, CMS implemented its statutory requirement to recoup funds associated with Medicaid eligibility-related improper payments for states with an eligibility error rate above 3 percent through its MEQC program. The MEQC program required states to randomly sample Medicaid enrollees to verify eligibility. Claims related to enrollees determined ineligible were tallied and compared with total claims for the sample universe to calculate an error rate. Following federal validation, states were subject to recoupment of all or part of the federal funds expended related to erroneous state payments over the 3 percent error rate threshold. However, in 1992, HHS’s Departmental Appeals Board—the department’s final level of administrative review—concluded that the MEQC error rate was not sufficiently accurate to provide reliable evidence to support recoupment of funds due to the small sample size from which the error rate was calculated. Consequently, the appeals board stated that it was “impossible to conclude with a reasonable certainty that the States failed to meet their target rates….” As a result of this opinion, CMS provided states the option, beginning in 1994, to either continue operating a traditional MEQC program or to conduct what CMS referred to as “MEQC pilots,” which focused on prospective improvements in eligibility determinations, rather than calculation of error rates. Since the “MEQC pilots” did not produce an error rate, CMS could not recoup federal funds expended due to erroneous eligibility determinations for states participating in the pilots. Between 2012, the earliest year for which CMS has maintained records, and 2014 when CMS suspended the MEQC program, 39 states participated in these “MEQC pilots” exempting them from possible recoupment of funds due to eligibility errors. While the other 12 states that continued to operate traditional MEQC programs could still be subject to recoupment of funds, CMS officials reported that no recoupments related to eligibility errors had occurred since the 1992 appeals board ruling, because none of these states had an error rate exceeding the 3 percent threshold. Thus, CMS has not recouped federal funds due to eligibility errors in decades. However, the agency has introduced new procedures through which it can, under certain circumstances, begin to recoup funds based on eligibility errors in fiscal year 2022. Specifically, in July 2017, CMS issued new regulations that included changes to its PERM process to satisfy the statutory requirements for recouping funds that MEQC was previously designed to operationalize. Under the revised PERM rules, CMS calculated an eligibility error improper payment rate beginning with the cohort of states under review for the fiscal year 2019 reporting period. However, it will not recoup funding from states with error rates exceeding the 3 percent threshold until states have a second review under the revised PERM rules, which will occur for the first cohort of states in fiscal year 2022. This allows each state the opportunity to implement improvements based on its initial PERM review and the MEQC review it will conduct in the off-cycle years to reduce the error rate or demonstrate a “good faith effort” to do so. CMS officials recognize the benefits of using state and federal audits, such as audits we reviewed for this report, as part of a broader strategy to improve program integrity and oversee states’ eligibility determination processes. However, CMS officials told us they do not have the authority to recoup federal funds related to eligibility errors identified outside of the PERM process, such as through state single audits. According to CMS officials, this is because of the specific statutory instruction limiting recoupments to instances when eligibility-related errors exceed the 3 percent error rate threshold, and because PERM is the process that CMS uses to calculate that error rate. The President’s fiscal year 2020 budget request included a legislative proposal to expand HHS’s authority to issue disallowances for eligibility errors. Specifically, the proposal requests legislative authority to permit HHS to issue disallowances outside of PERM and allow HHS, including the HHS-OIG, to extrapolate findings on beneficiary eligibility to ensure federal recovery of incorrect eligibility determinations; and eliminate the current 3 percent threshold for states’ eligibility-related improper payments. In place of the current 3 percent disregard, HHS would issue rulemaking specifying criteria for the recoupment of funds, including limiting them to instances of monetary loss, such as cases in which ineligible individuals received benefits. Determining whether individuals are eligible for Medicaid is a complex process that is vulnerable to error. The processes used to measure the extent of eligibility errors have been, and will continue to be, in a state of transition over the next several years as CMS implements its new PERM procedures and states implement improvements after their initial PERM reviews under these new procedures. Because CMS has not had a complete national estimate of improper payments due to eligibility errors since 2014, policymakers and other stakeholders have had an incomplete picture of the extent of eligibility errors in the Medicaid program nationally. This state of flux will make the findings from federal and state audits an even more important source of information on the accuracy of states’ eligibility determinations. As we have previously reported, oversight of the Medicaid program could be further improved through leveraging and coordinating program integrity efforts with state auditors to further improve the integrity of the Medicaid program. We provided a draft of this report to HHS for review and comment. HHS provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of HHS, the Administrator of the CMS, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or yocomc@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Table 4 provides a summary of key findings from the 47 federal and state audits that discussed the accuracy of states’ Medicaid eligibility determinations, published from 2014 through 2018, which we identified and reviewed. In addition to the contact named above, Michelle Rosenberg (Assistant Director), Perry Parsons (Analyst-in-Charge), and Heather Tompkins made key contributions to this report. Also contributing were Drew Long, Vikki Porter, and Jenny Rudisill.", "summary": "In fiscal year 2018, Medicaid covered approximately 75 million individuals at an estimated cost of $629 billion, $393 billion of which were federal funds. Medicaid eligibility is governed by a network of federal and state laws and regulations. In assessing eligibility for Medicaid, states must determine whether applicants meet eligibility criteria, such as financial and citizenship requirements. The accuracy of eligibility decisions has implications for federal and state spending. The Patient Protection and Affordable Care Act made significant changes to Medicaid eligibility rules beginning in 2014, including new ways of calculating income and new requirements related to electronically verifying applicants' information. Yet, little is known about the accuracy of states' Medicaid eligibility determinations since these changes were implemented. GAO was asked to review Medicaid eligibility determinations. This report describes, among other things, what is known about the accuracy of Medicaid eligibility determinations, and CMS's efforts to recoup funds related to eligibility errors. GAO reviewed 47 state and federal audits of Medicaid eligibility determinations across 21 states published between 2014 and 2018. GAO also reviewed relevant federal laws and regulations, and interviewed CMS officials. The Department of Health and Human Services provided technical comments on a draft of this report, which GAO incorporated as appropriate. States are responsible for determining applicants' eligibility for Medicaid, including verifying eligibility at application, redetermining eligibility, and disenrolling individuals who are no longer eligible. The Centers for Medicare & Medicaid Services (CMS) oversees states' Medicaid eligibility determinations. CMS did not publish an updated national Medicaid eligibility improper payment rate from 2015 through 2018 as states implemented the Patient Protection and Affordable Care Act. CMS released an updated rate in November 2019 that reflected new information on eligibility errors from 17 states. In lieu of complete and updated data, GAO reviewed 47 state and federal audits published between 2014 and 2018 related to 21 states' eligibility determinations. The identified accuracy issues did not always result in erroneous eligibility determinations. For example, some audits found applicants were determined eligible based on incomplete financial information, but when the audits reviewed additional information they found that the applicants still would have been eligible for Medicaid; and eligibility determinations complied with state policies and federal requirements, but noted that changes in state practices—such as using additional data sources to verify applicant information or checking sources more frequently—could improve eligibility determinations. While CMS is generally required to disallow, or recoup, federal funds from states for eligibility-related improper payments if the state's eligibility error rate exceeds 3 percent, it has not done so for decades, because the method it used for calculating eligibility error rates was found to be insufficient for that purpose. To address this, in July 2017, CMS issued revised procedures through which it can recoup funds for eligibility errors, beginning in fiscal year 2022. In addition, the President's fiscal year 2020 budget request includes a legislative proposal to expand the agency's authority to recoup funds related to eligibility errors. During this period of transition, federal and state audits will continue to provide important information about the accuracy of states' eligibility determinations.", "document_type": "gao"}
{"report": "The MIECHV program provides voluntary, evidence-based home visiting services for at-risk eligible families with children up to kindergarten entry. HRSA allocates MIECHV program formula grant funds to states based partly on the proportion of children under age 5 living in poverty in each state, among other factors. In fiscal year 2018, states received an average of $6.9 million in MIECHV program formula grant funding, ranging from $1.2 million provided to North Dakota to $21.4 million to California (see appendix I for a list of all states and their fiscal year 2016 through 2018 funding). Generally, the state’s public health or social services department is the lead agency that receives and administers the funds. States target MIECHV program resources to at-risk communities and have the flexibility to tailor the program to serve the specific needs of their communities. States are generally required to provide home visiting services using an HHS-approved evidence-based program model. Currently, HHS has determined through its Home Visiting Evidence of Effectiveness review that 18 evidence-based home visiting models meet HHS-established criteria for evidence of effectiveness, and are therefore eligible for MIECHV funding. States may select programs to implement from the models that have been approved by HHS, or states may choose to implement a home visiting service delivery model that qualifies as a promising approach, as defined in the statute. In MIECHV-funded home visiting programs, professionals meet regularly with families and provide services tailored to the families’ specific needs, such as teaching parenting skills, promoting early learning in the home, or conducting screenings and providing referrals to address caregiver depression, substance abuse, and family violence. According to HHS, the MIECHV program builds upon decades of scientific research showing that home visits by a nurse, social worker, or early childhood educator during pregnancy and early childhood have the potential to improve the lives of children and families. From fiscal years 2013 through 2018, the number of families served and number of home visits conducted nearly doubled (see table 1). The MIECHV program is the primary federal program focusing exclusively on evidence-based home visiting, according to HHS. However, in addition to administering the MIECHV program, states may have other home visiting programs that may be supported by funds from other federal programs, such as Temporary Assistance for Needy Families and the Maternal and Child Health Services Block Grant. These home visiting programs may provide services that differ from those provided under the MIECHV program. For example, states may provide home visiting services through these programs that use program models that are different from the MIECHV program models approved by HHS. The MOE requirement in the MIECHV program’s authorizing statute provides that funds provided to an eligible entity receiving a MIECHV grant “shall supplement, and not supplant, funds from other sources for early childhood home visitation programs or initiatives.” To demonstrate their compliance with this statutory requirement, states are required by HRSA to report in their annual grant applications their MOE spending for the prior fiscal year. HRSA provides guidance to states on how to report their MOE spending in the annual NOFOs. For example, since fiscal year 2013, the MOE guidance in the NOFOs generally has directed states to only report spending that meets the following criteria: paid for with state general funds, spent in the prior fiscal year on HHS approved evidence-based programs that include home visiting as a primary service delivery strategy, implemented in response to findings from the most current statewide needs assessment, and offered on a voluntary basis to pregnant women or caregivers of children from birth to kindergarten entry. Over time, HRSA has clarified the MOE guidance provided in the NOFOs to help address questions received from states, according to HRSA officials. We previously reported that certain grant design features affect the likelihood that states will use federal funds to supplement, rather than supplant (or replace), their own spending. One such design feature requires grant recipients to contribute their own funds in order to obtain grant funds. Requiring grant recipients to contribute their own funds can take the form of a match or MOE requirement. According to our prior report, matching grants typically contain either a single rate (e.g., 50 percent) or a range of rates (e.g., 50 to 80 percent) at which the federal government will match state spending on a particular program. An MOE requirement, in contrast, requires states to maintain existing levels of state spending on a particular program as a condition of receiving federal funds. Depending on the specific program and its MOE requirement, if a state did not previously spend any state funds on covered activities, then the state could be allowed to maintain MOE spending of $0. The MOE requirement is one of many MIECHV program requirements that HRSA is responsible for monitoring. HRSA also monitors MIECHV’s programmatic and technical requirements, such as evidence-based model implementation, policies and procedures, data collection, and organizational structure and capacity. HRSA also monitors fiscal and administrative requirements, such as those related to accounts payable and cash flow, accounting systems, and cost allocations. From fiscal years 2016 through 2018, state-reported MOE spending varied from $0 to more than $25 million, according to our review of MIECHV program grant applications (see fig. 1). For example, 28 states reported MOE spending of $0 in fiscal year 2018. Most of the 23 states that reported MOE spending greater than $0 in fiscal year 2018 reported spending less than $3 million, while three states reported spending more than $9 million. See appendix II for each state’s reported MOE spending for fiscal years 2016 through 2018. State-reported MOE spending does not necessarily reflect all state spending on all home visiting services. When states report their prior year’s MOE spending on their MIECHV grant applications, they are only required to include home visiting spending if it meets the criteria specified by HRSA in the NOFO. In addition to reporting their MOE spending in grant applications, some states also noted that they spent funds on home visiting services that did not meet those criteria. In fiscal year 2017, for example, one state reported that it had spent funds on home visiting services for a non-evidence-based model (i.e., a model not approved by HHS), and the state also funded an evidence-based program with funds other than state general funds. However, the state did not include either in its reported MOE spending because that spending did not meet the criteria for MOE spending in the NOFO. An update to the MIECHV program’s MOE guidance in the NOFO for fiscal year 2018 further impacted some state reported MOE spending. The update clarified the MOE guidance, stating that states should only report MOE spending by the recipient entity administering the MIECHV grant, and not report spending by other state agencies. According to HRSA officials—because the states were now directed to exclude some previously reported home visiting spending—five states decreased their reported MOE spending to $0. In addition, three other states reported a decrease in their MOE spending ranging from about $1.2 million to about $9.3 million because of this change (see table 2). HRSA determined that states generally met the MIECHV program’s MOE requirement because there was no supplantation of federal funds, including in states that reported no MOE spending and those that reported decreased MOE spending from the prior fiscal year. States may be permitted to report $0 in MOE spending if the non-federal spending on home visiting does not meet the criteria in the MOE guidance in the NOFO. For example, if the state had not previously funded home visiting programs that met HRSA’s MOE criteria for the MIECHV program, then the state could maintain state spending of $0, according to HRSA officials. States may report MOE spending of $0 if state general funds were spent on a home visiting model that was not approved by HRSA, if the state supports an evidence-based home visiting program with funds other than state general funds, or if the state did not support a home visiting program prior to implementation of MIECHV. HRSA determined that state-reported year-to-year decreases in MOE spending did not constitute supplantation (or replacement) of state funds with federal funds, because as described more fully below, HRSA determined there were valid reasons for the decreased MOE spending, according to agency officials. Based on our analysis of grant applications, 15 states reported decreases in MOE spending from fiscal years 2016 through 2018 (see table 3). These decreases ranged from $75,000 to $71,539 in one state, and $25,207,294 to $0 in another state. According to HRSA officials, there were three different reasons why states might have reported a decrease in MOE spending compared to the prior year: 1. The state made a technical error in its MOE calculation that subsequently was corrected. For example, some states reported a decrease in MOE spending compared to the prior year because the state previously included erroneous funding sources, such as funding for a home visiting program that did not meet the MIECHV program’s MOE criteria. 2. Circumstances outside of the state agency’s control contributed to the state reporting decreased funding, such as when a state legislature authorized budget cuts that affected home visiting funding or failed to pass a budget. For example, according to HRSA officials, one state experienced state budget challenges in fiscal years 2016 and 2017, which resulted in decreased funding for some home visiting services. The officials said this funding would have been included in the state’s reported MOE spending and these budget reductions resulted in a reduction to the reported MOE spending from the prior year. 3. The clarification to the MOE guidance that HRSA made in the fiscal year 2018 NOFO limited the spending states should report, as previously discussed. HRSA uses several methods to monitor the MIECHV program and the program’s MOE requirement is addressed to some extent as part of each, according to our review of HRSA grants monitoring documentation and interviews with HRSA officials. These monitoring methods include grant application reviews, site visits, and financial assessments, among others. The monitoring methods vary in terms of the extent to which the MOE requirement is specifically examined, who conducts the monitoring, and the frequency of monitoring (see table 4). The primary mechanism for monitoring the MIECHV program’s MOE requirement is the review of grant applications, according to HRSA officials. HRSA project officers review the MOE chart in states’ grant applications for 2 fiscal years to compare state reported MOE spending— actual non-federal expenditures—and determine if states maintained their level of spending (see table 5). If there is a missing MOE chart or potentially inaccurate MOE spending information, project officers work with states to resolve the issue. While HRSA primarily relies on its review of grant applications to monitor state compliance with the MIECHV program’s MOE requirement, the agency supplements these reviews with other monitoring techniques, and some of these techniques have identified issues with state-reported MOE spending. For example, operational site visits provide HRSA an opportunity to ask detailed questions about state-reported MOE spending and obtain supporting documentation. As a result of operational site visits, HRSA identified inaccurate state-reported MOE spending in some states. We reviewed four completed site visit reports from 2017—the most recently completed reports at the time of our review—and two of these reports had findings related to inaccurate state-reported MOE spending. For example, one site visit report noted that the state incorrectly included home visiting spending that did not use an evidence-based model in its reported MOE spending. HRSA also found some deficiencies with states’ reported MOE spending through the agency’s review of state single audits. According to HRSA officials, there were five state single audits with MIECHV MOE findings from fiscal years 2014 through 2017. We found that four of these audits identified deficiencies with how states monitored and accounted for their MOE spending. For example, one audit found that the state did not have internal controls in place to ensure that state spending met the minimum MOE requirement. In three of the four single audits that identified deficiencies, the state agencies concurred with the findings and prepared corrective action plans to address the deficiencies. As of June 2019, HRSA officials said they have taken steps, or are planning steps, to modify or provide additional guidance related to how the agency monitors the MOE requirement for the MIECHV program. Specifically: HRSA officials told us that beginning with the formula grant NOFO for fiscal year 2019, HRSA added an additional column to the MOE chart for states to provide the expenditures for the 2 years prior to the current fiscal year of the application. According to HRSA officials, this will streamline HRSA’s process to compare state-reported MOE spending across 2 prior fiscal years without having to go back to the previous year’s grant application. In February 2019, HRSA published an internal grants policy bulletin that specifically addressed MOE requirements and the agency’s monitoring of those requirements for all HRSA programs. HRSA is currently working on MIECHV program standard operating procedures that are intended to clarify staff monitoring roles and responsibilities across the agency. Completion of this resource is targeted for the end of fiscal year 2019. HRSA is also planning to add the MOE table to future MIECHV program Final Reports submitted by grantees, beginning with the fiscal year 2017 Final Report, which is due to HRSA in December 2019. According to officials, this will allow for a formal resubmission of MOE spending if there have been any changes since the submission of the most recent grant application. We provided a draft of this report to HHS for review and comment. HHS provided technical comments that we have incorporated in the report as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of the Department of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Washington, D.C. In addition to the contact named above, Elizabeth Morrison (Assistant Director), Andrea Dawson (Analyst in Charge), David Reed, and Kelly Snow made key contributions to this report. In addition, key support was provided by Jennifer Cook, Sarah Cornetto, Thomas James, Jean McSween, Mimi Nguyen, Stacy Ouellette, Michelle Sager, Almeta Spencer, and Matthew Valenta.", "summary": "The MIECHV program provides grants to states to support evidence-based home visiting services for at-risk pregnant women and parents with young children. HHS was appropriated $400 million per year for the MIECHV grant program for fiscal years 2018 through 2022. Families volunteer to participate in the MIECHV program and are provided regular home visits and support services from a nurse, social worker, or other professional. According to HHS, the program builds upon decades of scientific research showing that home visits during pregnancy and early childhood can improve the lives of children and families. States began receiving federal MIECHV program funds in fiscal year 2010, but many states provided home visiting services prior to the MIECHV program using state or other funds. To meet the program's MOE requirement, states are required to maintain home visiting spending that meets MIECHV program criteria. GAO was asked to review the MIECHV program's MOE requirement. GAO examined (1) what is known about the MOE spending reported by states that receive federal MIECHV program funds and (2) how HHS monitors states to ensure the MOE requirement is met. GAO reviewed MIECHV program notices of funding opportunity for fiscal years 2013 through 2018 and state grant applications for fiscal years 2016 through 2018, the most recent three years available. GAO also reviewed HHS grants monitoring documentation and interviewed HHS officials. From fiscal years 2016 through 2018, state reported maintenance of effort (MOE) spending varied from $0 to more than $25 million for the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program, according to GAO's review of MIECHV program grant applications. The program's authorizing statute requires states to meet an MOE requirement. MOE requirements in federal programs generally require grantees to maintain a certain level of spending to ensure grantee dollars are not replaced with federal dollars. To demonstrate their compliance with the MIECHV program's MOE requirement, states report in their annual grant applications their MOE spending for the prior fiscal year. HHS determined that states generally met the MIECHV program's MOE requirement because states did not replace state funds with federal funds, including states that reported no MOE spending or decreased MOE spending. States may be permitted to report $0 in MOE spending in certain circumstances; for example, if a state's only home visiting spending was on programs that did not meet MIECHV program criteria. According to HHS officials, state-reported decreases in MOE spending were due to errors in calculations that were subsequently corrected, clarifications to HHS's MOE guidance, or because of circumstances outside of the state agency's control. HHS uses multiple methods to monitor state compliance with the MOE requirement, according to GAO's review of HHS documentation and interviews with HHS officials. The agency's monitoring strategy includes reviews of grant applications, reviews of state single audits, and operational site visits, among other techniques. According to HHS officials, grant application reviews are the primary mechanism used to monitor state compliance, through which HHS compares state-reported MOE spending in grant applications across two fiscal years to determine if states maintained their level of spending. In addition, HHS identifies and resolves issues with state-reported MOE spending through its operational site visits and the agency's review of state single audits.", "document_type": "gao"}
{"report": "NNSA is responsible for managing national nuclear security missions: ensuring a safe, secure, and reliable nuclear deterrent; supplying nuclear fuel to the Navy; and supporting the nation’s nuclear nonproliferation efforts. NNSA largely relies on management and operating contractors to carry out these missions and to manage the day-to-day operations at eight sites collectively known as NNSA’s nuclear security enterprise. The Y-12 National Security Complex in Tennessee is the primary site among these with enriched uranium capabilities. Y-12’s primary mission is processing and storing uranium, processing uranium for naval reactors for the Navy, and developing associated technologies, including technologies to produce uranium-related components for nuclear warheads and bombs. According to NNSA documents, Y-12’s enriched uranium operations have key shortcomings, including an inefficient workflow, continually rising operations and maintenance costs stemming from facility age, and hazardous processes that could expose workers to radiological contamination. To address these shortcomings, NNSA developed plans to replace aging infrastructure at Y-12 and relocate key processing equipment without jeopardizing uranium production operations. In 2004, NNSA initially proposed relocating Y-12’s main uranium processing equipment into a new facility referred to as the UPF. NNSA planned to construct this single, consolidated facility that would reduce the overall size of existing uranium processing facilities, reduce operating costs by using modern equipment, and increase worker and environmental health and safety. NNSA estimated in 2007 that the UPF would cost approximately $1.4 billion to $3.5 billion to design and construct. In June 2012, the Deputy Secretary of Energy approved an updated cost estimate range for the UPF of $4.2 billion to $6.5 billion, with the latter being the project’s maximum allowable cost. However, by August 2012, the UPF contractor concluded that the UPF as designed would not provide enough space to house all of the uranium processing and other equipment. In October 2013, an external review estimated that the UPF project could cost as much as $11 billion. In 2014, because of the high cost and scheduling concerns of a solution focused solely on constructing new buildings, NNSA established its uranium program within its Office of Defense Programs. NNSA also prepared a high-level strategic plan based on its objectives of 1) completing the UPF project with a reduced scope within the cost and schedule limits established for the original UPF project and 2) phasing out mission dependency on Building 9212. Under NNSA’s revised approach, the agency plans to transition production operations out of Building 9212 and into the re-scoped UPF or existing buildings at Y-12 after they have been upgraded as described in further detail below. Building 9212. Constructed in 1945, the building’s design predates modern nuclear safety codes. It consists of a number of interconnected buildings that contain capabilities for uranium purification and casting, among other things. One of NNSA’s key goals is to shut down the Building 9212 operations that have the highest nuclear safety risks. Because of these risks, NNSA is implementing a four-phase exit strategy to systematically phase out mission dependency on Building 9212. According to NNSA’s September 2018 implementation plan for the exit strategy, the first three phases focus on reducing inventory, system isolation and clean out, and relocating capabilities from Building 9212 to other existing Y- 12 facilities or to the UPF once startup is complete. Building 9212 will then enter a phase of post-operational clean out, during which operations will be limited to simple processing, recovery, and inventory accountability. By about 2035, management of the building will transition to DOE’s Office of Environmental Management for decontamination and decommissioning activities. Building 9215. Constructed in the 1950s, the building’s design predates modern nuclear safety codes. It consists of three main structures, and its current primary function is fabrication, which involves metal machining operations for enriched uranium. As part of the Building 9212 exit strategy, NNSA plans to move capabilities into Building 9215, such as the uranium purification and the processing of uranium metal scraps resulting from machining operations. The uranium program is managing the development and deployment of new technologies to increase the efficiency and effectiveness of these capabilities. NNSA initially intended to house these two capabilities in the UPF before re-scoping the project to meet its cost and schedule goals. According to NNSA documents, NNSA is identifying and prioritizing infrastructure investments for Building 9215 that are to ensure its reliability through the 2040s. Building 9995. Constructed in the mid-1950s, this building’s design predates modern nuclear safety codes. It consists of a laboratory with capabilities for analytical chemistry operations, which can sample enriched uranium for material assay, chemistry content, and metallography in support of production. NNSA initially intended to house the analytical chemistry capabilities to support enriched uranium processing and material characterization in the UPF before re-scoping the project to meet its cost and schedule goals. According to NNSA documents, NNSA is identifying and prioritizing infrastructure investments for Building 9995 that are to ensure its reliability through the 2040s and its continued analytical chemistry support for the UPF and Y-12 more broadly. Building 9204-2E. Constructed in the late 1960s, this building’s design predates modern nuclear safety codes. It consists of a three- story, reinforced concrete frame structure that includes capabilities for assembly and disassembly of enriched uranium components with other materials. According to NNSA officials, the agency installed its radiography capability in Building 9204-2E in April 2017. According to NNSA documents, NNSA is identifying and prioritizing infrastructure investments for Building 9204-2E that are to ensure its reliability through the 2040s. Highly Enriched Uranium Materials Facility (HEUMF) (also called Building 9720-82). Beginning operations in January 2010, this building was built to modern nuclear safety codes. It is a reinforced concrete and steel structure that provides long-term storage of enriched uranium materials and accepts the transfer of some legacy enriched uranium from older facilities. HEUMF is the central repository for highly enriched uranium. Figure 1 shows NNSA’s planned relocation of uranium processing capabilities out of Building 9212 and into the re-scoped UPF and existing Y-12 facilities. The figure also indicates which existing facilities will require infrastructure investments to support enriched uranium operations. Under the new approach, the re-scoped UPF will be smaller than the UPF project’s original design and will house capabilities for casting, oxide production, and salvage and accountability of enriched uranium. NNSA has stated that the re-scoped UPF is to be built for no more than $6.5 billion by the end of 2025 through seven subprojects, described below. Site Readiness. This subproject included work to relocate an existing road, construct a new bridge, and extend an existing haul road. Site Infrastructure and Services. This subproject included demolition, excavation, and construction of a parking lot, security portal, concrete batch plant, and support building. Substation. This subproject included construction of an electrical power substation to provide power to the UPF and Y-12, replacing an existing substation at Y-12. Process Support Facilities. This subproject includes work to provide chilled water and storage of chemical and gas supplies for the UPF. Salvage and Accountability Building. This subproject includes construction of a nuclear facility for the decontamination of wastes and recovery of chemicals associated with uranium processing. Main Process Building. This subproject includes construction of the main nuclear facility to contain casting and special oxide production capabilities and a secure connecting portal to the HEUMF. Mechanical Electrical Building. This subproject includes construction of a building to house mechanical, electrical, heating, ventilation, air conditioning, and utility equipment for the Salvage and Accountability Building and Main Process Building. NNSA is required to manage construction of capital asset projects with a total project cost of greater than $50 million, such as the UPF, in accordance with DOE Order 413.3B. NNSA’s Office of Acquisition and Project Management manages the UPF project under DOE Order 413.3B with funding from NNSA’s Office of Defense Programs through the uranium program. DOE Order 413.3B requires that the project go through five management reviews and approvals, called “critical decisions” (CD), as the project moves from planning and design to construction and operation. (See fig. 2.) DOE Order 413.3B also requires that, before project completion (CD-4), NNSA issue a transition-to- operations plan, which is to ensure efficient and effective management as a project becomes operational and provide a basis for attaining initial and full operational capability. For projects likely to have an extended period of transition to the start of operations, an August 2016 memorandum from DOE requires that NNSA develop a more detailed plan to attain full operational capability. The plan must be developed earlier in the project management process— before start of construction (CD-3). In addition, NNSA must provide quarterly updates to DOE’s Project Management Risk Committee after completing construction until full operational capability is attained. The memorandum notes that DOE’s complex nuclear facilities can have significant risks that continue after project completion. These ongoing risks may impact achievement of full operational capability and thus require more efficient management. In September 2019, we reported that DOE officials stated that the August 2016 memorandum was largely created in response to experience with the Integrated Waste Treatment Unit facility at Idaho National Laboratory. This facility, which is intended to treat two forms of nuclear waste, is not operating as expected approximately 7 years after the completion of its construction. DOE Order 413.3B also states that projects with a total estimated cost of more than $100 million should have an independent cost estimate and external independent review prior to approval of the project’s performance baselines for cost and schedule (CD-2). Further, appropriations acts since fiscal year 2012 have included a limitation that prohibits the use of funds to approve CD-2 (approval of the project’s performance baselines for cost and schedule) or CD-3 (approval to start construction) for capital asset projects where total project costs exceed $100 million until a separate independent cost estimate has been developed. According to DOE’s standard operating procedure for conducting independent cost estimates, an independent cost estimate is prepared by an organization independent of the project sponsor—DOE-PM, in this case—using the same detailed technical and procurement information that was used to make the initial project estimate. The purpose of the estimate is to validate the project’s performance baselines—which include cost and schedule estimates—to determine these estimates’ accuracy and reasonableness. DOE-PM may use the independent cost estimate as supporting information in developing the external independent review. The external independent review is a broader analysis of the project to provide an unbiased assessment of whether NNSA can execute the project within the proposed scope, schedule, and cost commitments while meeting key performance requirements and fulfilling the mission need. Many of the federal government’s more costly and complex capital asset projects, including the UPF, require the development of cutting-edge technologies and integration of those technologies into large and complex systems. For example, DOE and NNSA use a systematic approach for assessing how far a technology has matured to evaluate the technology’s readiness to be integrated into a system—Technology Readiness Levels (TRL). This approach is intended to ensure that new technologies are sufficiently mature in time to be used successfully when a project is completed. TRLs progress from the least mature level, in which the basic technology principles are observed (TRL-1), to the highest maturity level, in which the total system is used successfully in project operations (TRL- 9). DOE Order 413.3B requires that each critical technology item or system on which a project depends must be demonstrated as a prototype in an operational environment (TRL-7) before the project’s performance baselines are approved (CD-2). According to our guide on evaluating technology readiness, assessing technology readiness does not eliminate the risk of relying on new technology but can identify concerns and serve as the basis for realistic discussions on how to mitigate potential risks associated with the project’s scope, for example. According to the Project Management Institute, Inc. (PMI), effective program management, in addition to effective project management, is important to the success of efforts such as NNSA’s uranium program. According to PMI’s standard for program management, effective program management helps ensure that a group of related projects and program activities are managed in a coordinated way to obtain benefits not available from managing them individually. Program management involves aligning multiple components to achieve the program’s goals. Other general standards relevant to program management for the uranium program include our cost-estimating guide and schedule assessment guide. In March 2009, we issued our cost-estimating guide to provide a consistent methodology that is based on cost-estimating best practices and that can be used across the federal government for developing, managing, and evaluating program cost estimates. The methodology outlined in the guide is a compilation of best practices that federal cost-estimating organizations and industry use to develop and maintain reliable cost estimates throughout the life of a government acquisition program. According to the guide, developing accurate life- cycle cost estimates has become a high priority for agencies in properly managing their portfolios of capital assets and in decision-making throughout the process. A life-cycle cost estimate provides an exhaustive and structured accounting of all resources and associated cost elements required to develop, produce, deploy, and sustain a particular program. The guide also states that a reliable cost estimate reflects all costs associated with a program—meaning that the estimate must be based on a complete scope of work—and the estimate should be updated to reflect changes in requirements (which may affect the scope of work). In December 2015, we issued our schedule guide, which develops the scheduling concepts introduced in our cost-estimating guide and presents them as best practices associated with developing and maintaining a reliable, high-quality schedule. According to the schedule guide, a well- planned schedule is a fundamental management tool that can help government programs use funds effectively by specifying when work will be performed and by measuring program performance against an approved plan. An integrated master schedule integrates all of the planned work in the program, the resources necessary to accomplish that work, and the associated budget, and it should be the focal point for program management. This schedule can show, for example, the completion dates for all activities leading up to major events or milestones, which can help determine if the program’s parameters are realistic and achievable. An integrated master schedule may consist of several or several hundred individual project or other activity schedules that represent the various efforts within a program. It should include the entire known scope of work, including the effort necessary from all government, contractor, and other key parties for a program’s successful execution. In addition, NNSA has various program management policies and guidance that apply to uranium program efforts that are not capital asset projects and that fall outside of DOE Order 413.3B. For example: NNSA issued a program management policy in January 2017 that defines general roles and responsibilities for the program managers for all of its strategic materials, such as uranium. This policy broadly outlines the managers’ authority and responsibilities for managing the strategic materials; these responsibilities include developing program documentation and managing risk. NNSA issued a program management policy in February 2019 that states program managers should establish and document the requirements for scope, schedule, and cost management using a tailored approach to their program. These requirements include the development of schedule and cost estimates that cover the life cycle of a program where appropriate, among other things. NNSA’s program guidance—applicable to the uranium program and others that fall under the Office of Defense Programs—recommends the development of an integrated master schedule and states that having one supports effective management of a program’s scope, risk, and day-to-day activities. Specifically, the guidance states that during the initial phases of a program, an integrated master schedule provides an early understanding of the required scope of work, key events, accomplishment criteria, and the likely program structure by depicting the progression of work through the remaining phases. The guidance allows for tailoring of the agency’s management approach based on the particular program being managed. According to NNSA documents and officials, the UPF project is on schedule and within budget, and NNSA has developed a plan to receive start-up authorization for UPF operations in 2025 and attain full operational capability in 2026. NNSA documents and officials reported that the UPF project is on track to meet its cost and schedule baseline estimates, and thus is expected to be constructed for $6.5 billion by the end of 2025. According to DOE’s project report and NNSA officials, three of the seven UPF subprojects are complete and four are ongoing as of December 2019. When we last reported in September 2017, NNSA had completed the Site Readiness subproject. In February 2018, NNSA completed the Site Infrastructure and Services subproject—about 2 months early and about $18 million under budget. In December 2019, NNSA completed the Substation subproject—about 6 months early and $13 million under budget. As shown in table 1, by March 2018 all UPF subprojects’ formal scopes of work and cost and schedule baseline estimates were approved (CD-2), and NNSA gained approval to start construction on them (CD-3). Since establishing these cost and schedule baseline estimates, NNSA officials stated that they have not made any significant changes that would require DOE executive-level approval. According to DOE policy, changes that affect the project’s ability to satisfy the mission need or that increase costs by the lesser of $100 million or half the project costs must be approved by the DOE Deputy Secretary as DOE’s Chief Executive for Project Management. According to DOE’s project report and NNSA officials, the four ongoing subprojects were progressing on schedule and within budget as of December 2019. NNSA officials stated that they expect these subprojects to meet their respective cost and schedule performance baselines and that the overall UPF project will be constructed for $6.5 billion by the end of 2025. (See fig. 3 for photograph of Main Process Building and Salvage and Accountability Building’s construction progress as of September 2019.) NNSA and its contractor for Y-12 have developed a plan to receive start- up authorization for UPF operations in 2025 and then will likely attain full operational capability for the UPF in 2026, according to NNSA officials and contractor representatives. DOE and NNSA approved this plan, which is required by DOE policy, in February 2018. This plan outlines three major risks associated with the UPF project that NNSA will need to address so that the project can attain full operational capability: 1. Capabilities and systems integration within the UPF. Addressing this risk includes actions to ensure that all of the UPF’s systems, and the capabilities that those systems provide (e.g., casting, oxide production), can function together as designed through testing. 2. Process prove-in and design authority qualification. Addressing this risk includes actions to ensure that the UPF’s systems meet certain metrics and are qualified for mission work. Aspects of this include laboratory analysis, statistical validation of repeatability, and engineering evaluations. 3. Integration of UPF with other facilities. Addressing this risk includes actions to ensure that the UPF systems can interface with other facilities’ systems (e.g., those in Buildings 9215, 9204-2E, and 9995) as designed and that all systems are able to support full-scale operations. NNSA officials estimated that construction of the UPF will be completed in 2022. According to the plan, the UPF will then go through various preoperational testing and operational readiness reviews to demonstrate the capabilities using nonhazardous surrogate material. Following testing and readiness reviews, the UPF will gain startup authorization, go through additional testing and first use, and then attain full operational capability— also referred to as “operational release.” NNSA officials and contractor representatives stated in June 2019 that the UPF should receive startup authorization sometime in 2025, before the project’s estimated completion (CD-4) date of December 2025. These officials and representatives estimated that the UPF would attain full operational capability about a year from receiving that startup authorization—that is, sometime in 2026. (See fig. 4.) NNSA officials stated in October 2019 that in fiscal year 2020 they will update the plan to attain full operational capability to include a schedule with more specific time frames for startup authorization, hot functional testing, first use, and operational release, among other things. According to NNSA’s plan, attaining full operational capability for the UPF is the final step that will ultimately lead to and enable the cessation of uranium operations in Building 9212, which could then be turned over to DOE Office of Environmental Management for final disposition in 2035. NNSA followed requirements to obtain independent cost estimates for the UPF (i.e., the four largest UPF subprojects) whose total estimated costs exceeded $100 million. NNSA then used those estimates to help negotiate with contractors and inform baseline estimates. NNSA obtained independent cost estimates from DOE-PM for the four UPF subprojects for which total costs exceeded $100 million. As noted above, projects with total costs that exceed $100 million are subject to an appropriations limitation unless independent cost estimates are obtained, and DOE policy requires such estimates for such projects. DOE-PM, an office independent from NNSA and its management of the UPF project, conducted the independent cost estimates for the four larger subprojects: the Mechanical Electrical Building, Process Support Facilities, Salvage and Accountability Building, and Main Process Building subprojects. In addition, NNSA officials stated that they obtained independent reviews for the three subprojects for which costs did not exceed $100 million. DOE policy does not require independent cost estimates for projects whose total estimated costs are less than the $100 million threshold. However, a NNSA policy states that NNSA should obtain an independent cost estimate or independent cost review to validate a project’s cost baselines for those projects for which estimated costs are between $20 million and $100 million. NNSA organized the independent cost estimates for the four larger subprojects so that some of the independent cost estimates included work for more than one subproject. Specifically, DOE-PM completed two estimates—one in March 2016 and one in December 2016—that included site preparation work and long lead procurements for the Salvage and Accountability Building and Main Process Building subprojects. In November 2016, DOE-PM completed the independent cost estimate for the Mechanical Electrical Building, which was the only estimate to include a single UPF subproject. NNSA officials explained that they handled the estimate for this subproject differently because work for the Mechanical Electrical Building could be separated easily from the other subprojects, and it was largely designed as a commercial-grade building. Lastly, in November 2017, DOE-PM completed the independent cost estimate for the majority of the work for the Process Support Facilities, Salvage and Accountability Building, and Main Process Building subprojects. NNSA officials stated they organized the independent cost estimates in this way to meet DOE requirements and appropriations limitations but still be able to begin work on the aspects of the overall UPF project that need to be completed earliest. DOE-PM conducted the four UPF subprojects’ independent cost and schedule estimates using our cost estimating and scheduling best practices, according to DOE-PM’s independent cost estimate reports. DOE-PM reviewed the project’s key cost drivers—elements whose sensitivity significantly affects the total project cost. The DOE-PM team then established independent estimates for those cost drivers, which may include vendor quotes for major equipment and detailed estimates for other materials, labor, and subcontracts. The team also prepared an independently generated resource-loaded schedule that allowed them to check for adequate funding compared with the project’s funding profile developed by the project team. DOE-PM’s analyses are based on their review of the UPF project’s work breakdown structure and associated documents, which include all of the activities that make up the project’s scope. DOE-PM also compared the UPF project estimates with our cost estimating and scheduling best practices, according to DOE-PM’s independent cost estimate reports. For example, DOE-PM’s November 2017 report found that the three larger UPF subproject’s cost and schedule estimates partially met the best practices and recommended some changes to the contractor to address those estimates that did not. DOE-PM reconciled the results of its independent cost estimates with the initial project estimates, as required by DOE’s standard operating procedure and NNSA’s business operating procedure for conducting independent cost estimates. During the reconciliation, DOE-PM worked with the UPF project team to adjust both the initial project estimates and its own independent cost estimates to correct any errors or misinterpretations of project requirements, according to the independent cost estimate reports. Under DOE’s and NNSA’s independent cost estimate procedures and according to DOE-PM officials, any remaining differences should be identified and explained, but estimates should not be changed. DOE-PM drew from the independent cost estimates for the Mechanical Electrical Building subproject to complete an external independent review of that subproject in November 2016. Then, DOE-PM drew from the independent cost estimates that included work for the Main Process Building, Salvage and Accountability Building, and Process Support Facilities subprojects to complete its external independent review of the UPF project in March 2018. NNSA officials stated that they used information from DOE-PM’s independent cost estimate and external independent review reports to help negotiate remaining work with the contractor and finalize the overall UPF project’s baseline estimates before starting construction. In June 2018, NNSA prepared a strategy to guide its negotiation of the remaining UPF project work that had not yet been priced with the contractor. Based on our review of NNSA’s negotiation strategy, we found that NNSA used DOE-PM’s independent cost estimate and external independent review reports to negotiate at least 14 of the 22 major and minor issues identified for discussion. These 14 issues included, for example, reducing concrete and freight direct costs, reducing the margin added to cover any increase in design scope, reducing subcontractor indirect costs, and increasing accuracy of other cost and schedule estimates. DOE approved NNSA’s cost and schedule baseline estimates (CD-2) and start of construction (CD-3) in March 2018 for three UPF subprojects. (See table 2 for the recommended cost and schedule baselines from the external independent review report and the final cost and schedule baseline estimates for all UPF subprojects.) In five of the seven subprojects, the final cost baseline estimates were close to or below the recommended baselines from DOE-PM’s external independent review. Also, in four of the seven subprojects, the final schedule baseline estimates were close to the recommended baselines. According to NNSA officials, the UPF project final baseline cost estimate includes cost contingency, and the December 2025 final schedule baseline estimate includes a year of schedule contingency. NNSA officials stated that, if necessary, they could use available funds to expedite the schedule. NNSA officials also expressed confidence that the UPF project will meet its goal of construction for $6.5 billion by the end of 2025. Since we last reported in September 2017, NNSA identified and made progress in implementing the uranium program’s scope of work and developed an integrated master schedule and life-cycle cost estimate— key management information for the program. The uranium program’s integrated master schedule extends through fiscal year 2035, and the life- cycle cost estimate includes the $7.4 billion in program costs from fiscal years 2016 through 2026. Since we last reported in September 2017, NNSA identified the uranium program’s scope of work and made progress in carrying out key activities. Specifically, NNSA identified the uranium program’s scope of work as required under NNSA program management policy and which we identified as a leading practice in our cost estimating and schedule guides. According to NNSA documents we reviewed and officials we interviewed, NNSA developed the uranium program’s scope of work in a work breakdown structure, which defines in detail the work or activities necessary to accomplish the program’s objectives. NNSA officials stated that the uranium program’s scope of work includes the UPF project as well as the capabilities and other activities necessary for the overall modernization effort that are not part of the UPF project. NNSA made progress implementing the following three main areas of the uranium program’s scope of work: Process Technology Development. Since we last reported in September 2017, NNSA’s uranium program has made progress in three of the four process technology projects that it manages to develop new uranium processing capabilities. According to NNSA officials, these capabilities are not included in the UPF project but are necessary to complete the suite of uranium capabilities required to meet weapons program needs. NNSA approved the electrorefining project’s cost and schedule performance baselines and start of construction (CD-2/3) in February 2019. This project, along with the direct chip melt projects discussed further below, are designed to provide a capability that was scoped out of the UPF project. Specifically, the electrorefining project is to provide the capability to purify uranium metal. NNSA officials stated that the calciner project will have its cost and schedule baselines and start of construction approved (CD- 2/3) in May 2020. This project is to provide the capability to convert uranium-bearing solutions to uranium oxide (a dry solid) so that it can be stored pending further processing in the future. The project will be located in Building 9212 and supports the exit of that building by enabling the processing of certain uranium- bearing solutions (such as the solutions resulting from cleaning out the building’s pipes and vessels) into a dry solid oxide that can be stored pending further processing. According to NNSA officials, the direct chip melt projects include two related efforts—a front-loading furnace and a bottom- loading furnace—that will provide the capability to process uranium scrap metal. Officials stated that the front-loading furnace direct chip melt project received approval to start work in September 2019 and has an estimated project completion of May 2021. This will provide near-term capability to process uranium scrap metal until the bottom-loading furnaces are designed and constructed. Officials said NNSA initiated the bottom-loading furnace direct chip melt project in July 2019 and expects to start construction in January 2021. Because the direct chip melt projects fall below the $50 million threshold for management under DOE Order 413.3B, they do not have CD dates. However, NNSA officials stated they will manage and oversee the bottom- loading furnace project under the Office of Defense Programs’ authorization-to-proceed memorandum and follow the sound project management principles outlined in the order. NNSA officials stated that the agency requires an oxide-to-metal conversion capability. In June 2019, NNSA issued a Notice of Intent to enter into a sole-source contract to provide the uranium oxide to metal conversion capability. According to NNSA officials, this potential sole-source contract is a near-term strategy that could cover any gap caused by phasing out operations in Building 9212. According to NNSA, under this contract the contractor could provide conversion services in 2023, effectively covering any gap caused by phasing out conversion operations in Building 9212. NNSA officials stated that the agency intends to continue pursuing the direct electrolytic reduction technology to provide the oxide-to-metal conversion capability after the sole-source contract, but the technology has not progressed since we last reported in 2017. Extended Life Programs. In December 2017, NNSA developed the implementation plan for the extended life programs for Buildings 9215 and 9204-2E. NNSA also developed an extended life program for Building 9995 in November 2017 and the implementation plan for that program in September 2018. NNSA updated both of these implementation plans in September 2019. Further, in September 2018, NNSA developed an implementation plan for its strategy to stop operations in Building 9212 and begin post-operations clean-out activities. These implementation plans identify a specific scope of work, and the necessary funding, that NNSA must execute in order to extend the operational lives of Buildings 9215, 9204-2E, and 9995 through the 2040s. Reducing Material at Risk in Older Buildings. Since we last reported in September 2017, NNSA has made progress in its efforts to move uranium materials out of older facilities and into the HEUMF. Specifically, NNSA officials said in November 2019 that they were about 77 percent done with this effort and had moved more than 50 metric tons of uranium out of older facilities and into the HEUMF since fiscal year 2015. In June 2019, NNSA officials said that their current strategy focuses on incorporating near-just-in-time inventory practices and further reducing material at risk by 2023. According to NNSA officials, this strategy is to further minimize the amount of material that is staged in Y-12’s older buildings. Also, according to NNSA officials, NNSA achieved a target working inventory of material in Building 9215 in 2016 and in Building 9204-2E in 2019. NNSA officials stated that, as of November 2019, they were on schedule to complete the remaining efforts by their estimated time frames. NNSA officials stated that the program’s scope of work includes elements for which additional analyses may be required and that any additional program work identified by those analyses will be incorporated into the scope of work, as appropriate. For example, NNSA identified the additional environmental and seismic analyses necessary to develop the scope of work for addressing certain structural deficiencies in Buildings 9215 and 9204-2E. NNSA is under a court order to complete additional environmental and seismic risk analyses following a 2014 update in the seismic hazard map for the area, which showed a greater risk than the previous version. According to Defense Nuclear Facilities Safety Board officials, in response to its 2015 report, NNSA identified their approach for re-evaluating the facilities’ conditions and risks and addressing some of the board’s seismic-related concerns. According to board officials, NNSA plans to start the re-evaluation of these structures in early fiscal year 2020. NNSA officials stated that if the additional analyses identify additional necessary work for the uranium program, NNSA will update the scope of work and revise the extended life program implementation plans to include that work. In December 2019, NNSA developed an integrated master schedule through fiscal year 2035 and a life-cycle cost estimate for the program through fiscal year 2026 that includes over $850 million in costs in addition to the UPF project. Successful management of federal acquisition programs, such as NNSA’s uranium program, partly depends on developing this key management information, as stated in our cost estimating and schedule guides. In September 2017, we found that NNSA had not yet developed an integrated master schedule or life-cycle cost estimate for the uranium program and recommended that NNSA set a time frame for doing so. NNSA agreed with this recommendation and has made progress in implementing it. A complete scope of work is required to develop an integrated master schedule and life-cycle cost estimate. (See fig. 5.) In December 2019, NNSA developed an integrated master schedule based on the uranium program’s scope of work to help manage its uranium program, as recommended in NNSA’s program guidance as well as our schedule guide and other best practices. According to PMI’s Program Management Standard, a program-integrated master schedule is the top-level planning document that includes individual program elements’ schedules and defines their dependencies among those required to achieve the program’s goals. According to NNSA officials, NNSA included all of the uranium program’s capabilities and elements that make up its scope of work, as well as other work that may affect the program, through fiscal year 2035. NNSA officials stated that the schedule includes the key milestones for each uranium program capability and element, such as project completion (CD-4) and operational release, since these key milestones are important for tracking the uranium program’s critical path of activities and for overall program management. NNSA officials stated that they will start reporting the uranium program’s progress against this integrated master schedule beginning in 2020. NNSA officials stated that they expect the integrated master schedule to be iterative and that they will update it to capture any changes or additions to the program’s scope of work. In December 2019, NNSA developed a life-cycle cost estimate through fiscal year 2026 for the uranium program, as called for in our cost estimating guide and other best practices. NNSA estimated that the uranium program will spend a total of approximately $7.4 billion from fiscal years 2016 through 2026 to support its uranium processing modernization efforts. Specifically, NNSA officials stated that the life-cycle cost estimate includes $6.5 billion in UPF project costs and over $850 million in program costs that include developing the uranium processing capabilities that are not part of the UPF project, integrating those capabilities with the UPF, improving the infrastructure of existing buildings, and transitioning out of Building 9212. NNSA officials stated that they estimated uranium program life-cycle costs from fiscal years 2016 through 2026 because they could not accurately estimate some of the activities in the program’s scope of work that are enduring for the nuclear security enterprise rather than specific projects with finite schedules for construction. According to our cost- estimating guide, a reliable cost estimate reflects all costs associated with a program’s scope of work, and the estimate should be updated to reflect any changes in requirements—that is, a life-cycle cost estimate can be iterative. NNSA officials stated that they expect to update the life-cycle cost estimate with additional program costs, once known, and will include any additional future scope added to the program. Schedule milestones and cost estimates included in NNSA’s integrated master schedule and life-cycle cost estimate for the uranium program are summarized in table 3. We are encouraged that NNSA may be able to better manage the day-to- day activities of the uranium program and mitigate any risks associated with integrating the UPF project with other aspects of the program through its development of key program management information—a scope of work, an integrated master schedule, and a life-cycle cost estimate. Successful program management through the life of a program depends in part on all of these efforts and may provide decision makers such as Congress with needed information on the program’s complete scope of work, key events, and expected long-term program costs. We provided DOE and NNSA with a draft of this report for review and comment. NNSA provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, the Administrator of the National Nuclear Security Administration, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the individual mentioned above, Jonathan Gill (Assistant Director), Elizabeth Luke (Analyst in Charge), Danny Baez, John Bauckman, Brian Bothwell, Juaná Collymore, Jennifer Echard, Justin Fisher, Juan Garay, William Gerard, Cynthia Norris, Dan Royer, and Kiki Theodoropoulos made key contributions to this report.", "summary": "A supply of enriched uranium is crucial to support the nation's nuclear weapons stockpile and the U.S. Navy, but the infrastructure of several U.S. uranium-processing facilities is outdated. In 2014, NNSA began plans to meet the nation's uranium needs by redirecting processing capabilities to the UPF and to other existing buildings NNSA plans to upgrade at Y-12 in Oak Ridge, Tennessee. The National Defense Authorization Act for Fiscal Year 2013, as amended, includes a provision for GAO to periodically review the UPF. Also, a Senate report accompanying the National Defense Authorization Act bill for fiscal year 2012 provides for GAO to review the independent cost estimates for the UPF. This report, which is GAO's sixth on the UPF, examines (1) the status of the UPF project and plans for starting UPF operations; (2) the extent to which NNSA has followed requirements to obtain independent cost estimates for the UPF, and how NNSA has used information from those estimates; and (3) the extent to which NNSA has made progress in developing uranium program management information since GAO's September 2017 report. GAO reviewed project and program documents on planning, schedule, cost, and implementation, and interviewed program officials. National Nuclear Security Administration (NNSA) documents and officials reported that the new Uranium Processing Facility (UPF) is on schedule and within budget. As of December 2019, three of the seven UPF subprojects were complete, and four were ongoing. NNSA officials told GAO they estimate that construction of the UPF will be complete in 2022 and that they expect to meet NNSA's goal of completing the UPF project for $6.5 billion by the end of 2025. As required, NNSA and its contractor developed a plan for starting operations at the UPF, which officials stated will likely occur in 2026. According to NNSA's plan, attaining full UPF operational capability will be the final step to enable NNSA to stop certain operations in Building 9212—the oldest building with the highest nuclear safety risk at the Y-12 National Security Complex (Y-12)—and turn it over to the Department of Energy (DOE) for final disposition by 2035. In managing the UPF project, NNSA obtained independent cost estimates for the four largest UPF subprojects whose total estimated costs exceeded $100 million. Such estimates are required by DOE policy and to satisfy limitations in appropriations laws. Moreover, based on its review of NNSA documents, GAO found NNSA used those estimates to help inform the UPF's approved cost and schedule baseline estimates. NNSA officials stated that they used information from the independent cost estimate and other sources to help negotiate remaining work with the contractor and finalize the overall UPF's baseline estimates before starting construction. Since GAO last reported on NNSA's broader uranium program in September 2017, NNSA identified and made progress in implementing the uranium program's scope of work that includes capabilities and other activities that are not part of the UPF project but are needed for weapons program. Specifically, NNSA made progress in the following areas: 1. developing process technologies that are expected to increase the efficiency and effectiveness of certain uranium processing capabilities; 2. investing in infrastructure to extend the operational lives of older uranium facilities; and 3. reducing the amount of uranium stored and used in these older uranium facilities. NNSA has also made progress in implementing GAO's 2017 recommendation to develop key management information for the uranium program. Specifically, NNSA developed an integrated master schedule covering the scope of work for the program through fiscal year 2035 and a life-cycle cost estimate that includes program costs through fiscal year 2026. NNSA estimated that, in addition to completing the UPF project for $6.5 billion, the uranium program will spend over $850 million from fiscal years 2016 through 2026 to support modernizing other needed uranium processing capabilities and transitioning out of Building 9212.", "document_type": "gao"}
{"report": "Priority and Standard Review For a priority review, the Food and Drug Administration (FDA) directs its resources to applications for new drugs that prevent, diagnose, or treat a serious condition and, if approved, would provide significant improvements in safety or effectiveness compared to available drugs. A drug may also receive priority review if the drug sponsor redeems a priority review voucher, among other things. FDA’s goal is to complete the review of a priority application within 6 months. Drugs that do not receive priority review receive standard review. FDA’s goal is to complete the review of a standard application within 10 months. FDA, an agency within the Department of Health and Human Services (HHS), is responsible for overseeing the safety and efficacy of drugs and biological products, such as vaccines, sold in the United States. Before a drug sponsor can market a new drug, it generally must submit evidence of the drug’s safety and effectiveness to FDA in a new drug application or biologics license application. While FDA reviews most drug applications using its standard review process, FDA’s priority review designation is intended to reduce the review time needed to bring a drug to market for certain drugs that treat serious conditions. A drug application typically receives a priority review designation if the drug would provide a significant improvement in the safety or effectiveness of the prevention, diagnosis, or treatment of a serious condition when compared to available drugs, among other things (see sidebar). FDA reviews all applications to determine if they qualify for priority review. FDA is also responsible for the implementation of the three PRV programs, which are intended to encourage development of drugs for tropical diseases, rare pediatric diseases, and medical countermeasures. Qualifying diseases and conditions for the tropical disease PRV program and criteria for the rare pediatric disease PRV program are set forth in statute—though the list of eligible tropical diseases can be updated by order of the Secretary of HHS. For the medical countermeasure PRV program, HHS publishes a list of high- priority threats that qualify for a PRV, including those that the Department of Homeland Security determines to pose a material threat sufficient to affect national security. (See table 1 for the types of drugs eligible for a PRV.) In order to be awarded a PRV, drug applications must meet additional criteria. For example, for all three PRV programs, the drug application must be eligible for priority review and a drug may be disqualified if its active ingredient has been previously approved by FDA in another drug application. If a drug application meets the eligibility criteria for one of the PRV programs, the drug sponsor can include a request for a PRV in its application, including supporting documentation demonstrating how the application meets the PRV eligibility criteria. Once FDA receives a sponsor’s drug application and PRV request, it reviews the information and considers whether the drug should be approved. If FDA approves the drug application, it includes its decision regarding whether to award a PRV in its approval letter. Once FDA awards a PRV to a drug sponsor, the sponsor can redeem the PRV with the submission of a future drug application for a drug intended to treat any disease or condition, shortening FDA’s targeted review time from the 10-month standard review to 6 months, even if the drug in that future application would not qualify for priority review on its own merits. The drug sponsor also has the option of selling or transferring the PRV to another drug sponsor, which may then choose to use it or similarly sell or transfer it. PRVs may be transferred any number of times before they are used. When the drug sponsor possessing the PRV ultimately decides to redeem it, the sponsor must notify FDA at least 90 days in advance of submitting its drug application that is using the PRV. Figure 1 provides a general overview of the PRV programs. The drug sponsor redeeming a PRV must also pay a PRV user fee (about $2.5 million in fiscal year 2019), in addition to other user fees required for all drug applications. Because drug applications submitted to FDA with a PRV would not otherwise qualify for priority review, PRV user fees are intended to cover FDA’s additional costs incurred when reviewing new drug applications with a PRV. When a drug sponsor notifies FDA of its intent to redeem a PRV, its notification serves as a legally binding commitment to pay the PRV user fee. Of the three PRV programs, two—the rare pediatric disease and the medical countermeasure PRV programs—are set to expire in the coming years, unless they are reauthorized by Congress. The rare pediatric disease PRV program will begin to expire on September 30, 2020, and the program will end in September 2022. The medical countermeasure PRV program will expire on October 1, 2023. After these end dates, FDA could no longer award a PRV for a rare pediatric disease or a medical countermeasure; however, the expiration dates do not affect PRV redemptions, as drug sponsors may redeem PRVs earned at any point in the future. As of September 30, 2019, FDA awarded 31 PRVs across the three PRV programs, with the majority being awarded through the rare pediatric disease PRV program (see fig. 2). According to FDA, all PRVs were awarded for drugs that met unmet medical needs. The 31 PRVs were awarded to 26 different drug sponsors; three sponsors were awarded two PRVs each and one sponsor was awarded three PRVs. FDA awarded the 31 PRVs for drugs that treat 27 different diseases. For five diseases— malaria, tuberculosis, smallpox, spinal muscular atrophy, and Duchenne muscular dystrophy—FDA awarded PRVs to two different drugs for their treatment, and FDA awarded one PRV for a drug that prevents two different diseases. (See appendix I for more information about the drugs for which FDA awarded PRVs.) The first PRV was awarded in fiscal year 2009, 2 years after the start of the tropical disease PRV program, and none were awarded in fiscal years 2010 through 2012. The first rare pediatric disease PRV was awarded in fiscal year 2014—about 2 years after that PRV program was authorized— and, beginning in fiscal year 2015, the majority of PRVs awarded were for rare pediatric diseases. In fiscal year 2018, FDA awarded eight PRVs, including the first medical countermeasure PRV, the most awarded in a single fiscal year (see fig. 3). Of the 31 PRVs that FDA awarded to drug sponsors, available data indicate 17 PRVs were subsequently sold to another drug sponsor, providing revenue to the sponsor selling the PRV. For 14 of these 17 PRVs, we were able to determine a sales price, which ranged from $67.5 million for a PRV sold in fiscal year 2014 to $350 million for a PRV sold in fiscal year 2015. However, the available sales prices of the PRVs sold since February 2017 have varied less than those sold previously, ranging from $80 to $130 million (see fig. 4). Because drug sponsors are only required to notify FDA of sales of rare pediatric disease PRVs at the time the sale occurs, additional transfers or sales of PRVs may have occurred. The drug sponsors, stakeholders, and researchers we interviewed noted that several factors could influence whether a drug sponsor keeps a PRV for future use, sells the PRV to another drug sponsor, or purchases a PRV to use on a drug that would not otherwise qualify for priority review. The PRV programs allow PRVs to be transferred multiple times, and according to stakeholders and drug sponsors we spoke with, the revenue gained from such sales may be a motivating factor for drug sponsors to sell them. For example, three stakeholders we interviewed said they believe drug sponsors consider the drugs in their development pipeline when deciding to keep, sell, or purchase a PRV, and one stated that drug sponsors need to determine if they would benefit more from using the PRV or the money they could make from selling it. One researcher commented that price variation for PRVs can affect how a drug sponsor perceives the incentive and that low prices for PRVs may signify the need for additional incentives for drug development. However, two drug sponsors told us that they would continue to pursue PRVs as long as they were available and useful for a particular drug in their pipeline. As of September 30, 2019, drug sponsors redeemed 16 of the 31 PRVs— that is, they submitted the PRV to obtain priority review for a drug application for a drug that would not otherwise qualify for a priority review. The drugs for which the PRVs were redeemed treat or prevent a variety of conditions and diseases, including human immunodeficiency virus (HIV), type 2 diabetes, and different forms of arthritis. (See appendix II for a complete list of PRV redemptions.) The first PRV was redeemed in fiscal year 2011, about 2 years after the first PRV was awarded, and the second PRV was redeemed in fiscal year 2015. Since 2017, drug sponsors have redeemed between three and six PRVs each year (see fig. 5). The 16 PRVs were redeemed by 10 different drug sponsors. Twelve of the 16 redeemed PRVs were purchased and redeemed by a drug sponsor different from the original PRV awardee. All 16 redeemed PRVs were redeemed within 4 years of FDA awarding them (see fig. 6). Of the 15 PRVs that were not redeemed as of September 30, 2019, 12 were awarded in fiscal years 2018 or 2019, and one was awarded in early fiscal year 2016. (See fig. 7.) Drug sponsors we contacted told us that decisions on when to redeem PRVs are largely strategic and take into consideration their drug development pipeline and market competition. For example, three of the drug sponsors told us they might choose to redeem a PRV to help a drug reach the market faster than a competitor’s drug, and two drug sponsors told us they may hold a PRV to use to obtain priority review for a particular drug that is in development. Another drug sponsor told us it considers the likelihood of a drug receiving approval from FDA when deciding when to use a PRV (since the PRV only affects the time frames for FDA’s review and does not guarantee approval), and if a drug in its pipeline could receive priority review from FDA on its own merit. Almost half of the awarded PRVs had not been redeemed as of the end of fiscal year 2019, which may affect FDA’s ability to forecast resources needed in the future. In 2016, we reported that FDA told us that the rare pediatric disease PRV program placed a substantial strain on its workload, explaining that performing a priority review on a drug that would otherwise merit a standard review requires the agency to conduct significant work in a compressed time frame. Between fiscal years 2011 and 2018, PRV redemptions have accounted for less than 1 percent of FDA’s reviews in any given year, according to FDA. While FDA receives 90 days’ notice of a PRV redemption, the notice period may not be enough time to ensure the appropriate staff are available to review a drug application that the agency does not consider to be a public health priority, according to FDA. However, one researcher noted that this uncertainty exists for all drug applications, as FDA cannot know in a given year how many drug applications will be submitted in any particular therapeutic area or how many of these applications will qualify for priority review. Furthermore, two drug sponsors, one researcher, and one stakeholder we spoke with noted that FDA collects additional user fees for PRV redemptions specifically to support the priority review for a drug that would not normally qualify for one. Since fiscal year 2011, FDA has collected almost $44 million in PRV user fees for the 16 redeemed PRVs. FDA does not track the resources it uses specifically for the PRV programs, so the agency cannot determine if the PRV user fees paid when PRVs are redeemed cover the associated costs. According to FDA, the agency cannot anticipate the therapeutic area for which a PRV will be redeemed, so PRV user fees may not ameliorate the effect of PRV redemptions on the review divisions or provide for rapid hiring of additional review staff with relevant experience and technical expertise. FDA officials told us that each new PRV program—and changes made to existing PRV programs—requires additional resources to implement. The agency reports that the services of over 11 offices within FDA are required to work on some aspect of the PRV programs, which may at times require FDA to shift resources from its public health priorities. According to FDA, the PRV programs also expend and divert agency resources to draft and revise PRV-related guidance; update webpages; research, draft, and publish notices and orders to add or decline to add diseases to the list of eligible tropical diseases; respond to inquiries from sponsors, potential sponsors, investors, attorneys, and other interested individuals; and respond to requests for a rare pediatric disease designation. Our literature review found three studies—one for each of the PRV programs—that examined and drew conclusions about how PRV programs affect drug development; of these, one study found evidence of an effect of a PRV program on drug development. Specifically, it found that drugs to treat rare pediatric diseases, which could be eligible for a rare pediatric disease PRV, were more likely to advance from phase I to phase II clinical trials when compared to rare adult disease drugs. The studies examining the other two PRV programs did not find an effect on drug development. Rare pediatric disease PRV program. A 2019 study found that the rare pediatric disease PRV program was not associated with an increase in the number or rate of new pediatric disease drugs that started or completed clinical trials. However, the study found that, after the creation of the rare pediatric disease PRV program, drugs the study authors determined could be eligible for a rare pediatric disease PRV were more likely to advance from phase I to phase II clinical trials compared to rare adult disease drugs, which are not eligible for a PRV under this program. Additionally, the study found the time it took for drugs to progress to the next stage of development was shorter among drugs eligible for a rare pediatric disease PRV compared to drugs for rare adult diseases, across all three phases of clinical development. Tropical diseases PRV program. A 2017 study found that this PRV program was not associated with an increase in tropical disease drugs starting clinical testing. The study found the proportion of tropical disease drugs among all drugs in development decreased slightly after the PRV program was created. Study authors suggested the relatively small number of approved tropical disease products in the last decade indicates the PRV program did not serve as a stimulus for completing late-stage drug development. Medical countermeasure PRV program. A 2018 study reported that 25 of 26 medical countermeasures undergoing clinical trials received direct or indirect public support, such as funding from the Department of Defense. Authors stated that, given the extent to which development of medical countermeasures already occurs via direct or indirect federal funding, alternatives other than the PRV program could better stimulate development of medical countermeasures. While the few studies of the PRV program found little to no effect on drug development, the seven drug sponsors we contacted told us the PRV programs were an incentive—that is, a factor in their decisionmaking—for drug development. In contrast, the seven researchers and seven stakeholders we contacted reported mixed views of the PRV programs as an incentive for drug development. Drug sponsors. All seven drug sponsors told us the PRV programs were a factor in drug development decisions—six sponsors said it was one of a number of factors, and one sponsor said it was pivotal in its development of a drug. For example, three drug sponsors told us PRVs were important to help fund drug development and one of these drug sponsors told us the PRV program supported its decision to move a drug already under development to market. Four drug sponsors told us PRV programs may be a more significant incentive for small drug sponsors, with one small, nonprofit drug sponsor noting that it entirely relied on the profits from the sale of its PRV to ensure its drug would become available to those who need it. Additional factors drug sponsors reported considering included whether the sponsor has a drug in their development pipeline that could particularly benefit from a PRV, and whether its drug development program has public financial support, such as direct federal funding. Researchers. The seven researchers reported mixed views of the PRV programs as an incentive for drug development, and their perceptions of the three programs varied. For example, when asked to describe the incentive for drug development provided by the tropical disease PRV program, two researchers described it as “not significant,” and two researchers described it as “somewhat significant.” However, one of these researchers told us the tropical disease PRV program encouraged drug development, particularly for diseases such as tuberculosis and malaria for which a drug is potentially more commercially viable. Regarding the rare pediatric disease PRV program, three researchers told us they have heard anecdotally that the program is an incentive to develop or continue development of rare pediatric disease drugs. In contrast, one researcher told us many drug sponsors have received a rare pediatric disease PRV for drugs they would have produced anyway, and another told us he did not believe the rare pediatric disease PRV provided an adequate incentive for adding new drugs into a drug sponsor’s pipeline. Finally, four researchers told us it was too early to evaluate the medical countermeasure PRV program as an incentive. Stakeholders. The seven stakeholders also reported mixed views on the PRV programs as an incentive for drug development. For example, one stakeholder told us that drug sponsors have entered particular drug development areas because of the PRV programs, and the PRV program has been pivotal to the financial planning of small drug sponsors working in the medical countermeasures and rare pediatric disease spaces. In contrast, two other stakeholders told us the PRV programs are an incentive to obtain FDA approval for a drug that has already been developed and marketed outside of the United States but are not an incentive for developing new drugs. One of these stakeholders and an additional stakeholder also noted that PRVs are often a source of additional revenue to drug sponsors that would have developed their PRV drug anyway and did not need the PRV to finance drug development. The number of PRVs awarded by FDA could influence the effectiveness of the PRV programs as incentives, according to several drug sponsors, researchers, and stakeholders we contacted. Specifically, some indicated that the potential revenue from the sale of a PRV could decline if more PRVs are awarded, and there is an increased supply of PRVs available for sale. Specific comments included the following: One drug sponsor told us that, while the number of PRVs on the market was a concern, they have remained valuable. Another drug sponsor told us it was not concerned with the relative value of PRVs, because it did not plan to sell its remaining PRVs and would purchase more in the future if PRVs would benefit drugs in its pipeline. One researcher told us lower prices for PRVs merited concern, because the PRV alone might not be sufficient to motivate drug development. The researcher indicated that a drug would also need either sufficient sales or additional government incentives. Two stakeholders told us the sales prices of PRVs (and potential revenue from selling them) might be more of a concern for small drug sponsors than large drug sponsors, as these stakeholders told us small drug sponsors are more likely to sell their PRV instead of using it for another drug in their portfolio. Drug sponsors, researchers, and stakeholders we contacted also reported mixed views on whether the rare pediatric disease and medical countermeasure PRV programs—set to expire by 2022 and 2023, respectively—should be reauthorized. While FDA officials reported that, as of April 2019, the agency does not have a position on the reauthorization of these two PRV programs, drug sponsors generally indicated support for their reauthorizations, with some noting that PRV program expirations may negatively affect overall drug development and the willingness of drug sponsors to work in these areas. The researchers we contacted offered mixed opinions on reauthorization. For example, one recommended reauthorizing both PRV programs, but indicated that his opinion could change if a better incentive was developed. In contrast, another researcher supported the expiration of these two programs, noting that their expiration could ultimately raise the potential revenue from the sale of an available PRV and could also make the tropical disease PRV program, which does not require reauthorization, more popular to encourage drug development. Most stakeholders we contacted did not offer a clear opinion on reauthorization; those that did generally supported reauthorization. Drug sponsors, researchers, and stakeholders we contacted suggested several improvements to the PRV programs, including those described below. Require innovation for PRV-eligible drugs. Two researchers and two stakeholders noted that the PRV programs, particularly the tropical disease PRV program, have been criticized for not providing incentives for innovation and suggested PRV awards be limited to drugs new to the global market. Currently, drug sponsors can receive a PRV for a drug that has already been developed and marketed outside of the United States, but which qualifies for a PRV because the drug has not been approved for marketing in the United States. One researcher suggested the federal government should not provide an incentive, like a PRV, for drugs already in existence outside of the United States, for which most research and development was already completed. However, one drug sponsor told us that requiring a tropical disease drug to be approved first in the United States to qualify for a PRV would delay entry of the drug into the international markets that need it the most. Additionally, two stakeholders told us that drugs that have already been developed may have significant benefits to patients when combined or used to treat other diseases. Require drug sponsors to guarantee access to PRV-eligible drugs. One researcher and two stakeholders suggested drug sponsors submit an access plan to help ensure the drug reaches the populations in need of the treatment, and one drug sponsor suggested they supply at cost the drugs for which the PRV was awarded. One of these stakeholders noted that a weakness of the PRV program is that drug sponsors awarded a PRV have no obligation to make the approved drug available at an affordable price. It suggested that requiring an access plan may result in drugs for which a PRV was awarded being more available and accessible to the populations that need them. However, three stakeholders noted that FDA may not have the resources or authority to enforce such access commitments. Limit PRVs to drug sponsors with financial need. One drug sponsor and one researcher suggested awarding a PRV only to drug sponsors that financially require it to develop their drug, such as a nonprofit organization that must leverage potential revenue from the PRV to help offset drug development costs. Make administrative changes. One drug sponsor told us FDA’s process for determining the list of tropical diseases eligible for a PRV was not transparent and wanted clarification on FDA’s timeline for editing this list. Another drug sponsor told us it wanted clarification on whether a drug would merit priority review on its own, so the sponsor could determine whether to redeem a PRV for that drug. In addition to suggesting improvements to the PRV programs, drug sponsors, researchers, and stakeholders we contacted, as well as our literature review, identified potential alternatives to the PRV programs that provide incentives for drug development (see table 2). We provided a draft of this report to HHS for review and comment. HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on GAO’s website at http://www.gao.gov/. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix IV. Sales price (dollars in millions) Gilead Sciences, Inc. Gilead Sciences, Inc. Novartis Pharmaceuticals Corporation Dengvaxia Sanofi Pretomanid The Global Alliance for TB Drug Development (TB Alliance) BioMarin Pharmaceutical Inc. Treatment of mucopolysaccharidosis type IVA (MPS IVA; Morquio A syndrome) PRV Sale date purchaser August 2015 AbbVie Inc. Drug name Drug sponsor Unituxin United Therapeutics Corporation Cholbam Asklepion Pharmaceuticals, LLC Xuriden Wellstat Therapeutics Corporation Strensiq Alexion Pharmaceuticals Inc. Kanuma Alexion Pharmaceuticals Inc. Exondys 51 Sarepta Therapeutics, Inc. Teva Pharmaceutical USA, Inc. Treatment of hypophosphatasia (HPP) Spinraza Biogen, Inc. Gilead Sciences, Inc. Treatment of Duchenne muscular dystrophy (DMD) Treatment of tripeptidyl peptidase 1 (TPP1) deficiency (Batten disease) Marathon Pharmaceuticals, LLC Brineura BioMarin Pharmaceutical Inc. Kymriah Novartis Pharmaceuticals Corporation Mepsevii Ultragenyx Pharmaceutical Inc. Luxturna Spark Therapeutics, Inc. Sales price (dollars in millions) Vertex Pharmaceuticals Inc. Crysvita Ultragenyx Pharmaceutical Inc. Epidiolex GW Research, Ltd. Treatment of X-linked hypophosphatemia (XLH) Inc. Biohaven Pharmaceuticals, Inc. Leadiant Biosciences, Inc. Gamifant Novimmune S.A. Zolgensma Avexis, Inc. Treatment of seizures associated with Lennox Gastaut-Syndrome and Dravet syndrome Treatment of adenosine deaminase-severe combined immunodeficiency (ADA- SCID) Treatment of primary hemophagocytic lymphohistiocytosis (HLH) Treatment of pediatric patients with spinal muscular atrophy (SMA) Legend: ✓ = transferred from original drug sponsor; ✗ = no public announcement of transfer; — = not applicable. Appendix II: Redeemed Priority Review Vouchers (PRV) Appendix II: Redeemed Priority Review Vouchers (PRV) Appendix III: Key Milestones of the Priority Review Voucher Programs FDA may not award any rare pediatric disease priority review vouchers after September 30, 2020, unless the drug has received a rare pediatric disease designation by that date, and FDA has approved the drug application by September 30, 2022. In addition to the contact named above, Kim Yamane (Assistant Director), Erin C. Henderson (Analyst-in-Charge), Kaitlin Farquharson, Laurie Pachter, Vikki Porter, Helen Sauer, Meghan Shrewsbury, and Merrile Sing made key contributions to this report. Also contributing were Leia Dickerson, Hayden Huang, and Yesook Merrill.", "summary": "Few drugs are currently available to treat certain tropical and rare pediatric diseases and to use as medical countermeasures, given their small market or potentially limited profitability. To help provide incentives for the development of such drugs, Congress created three PRV programs, which award PRVs to drug sponsors that develop drugs for tropical diseases, rare pediatric diseases, and medical countermeasures (e.g., drugs to mitigate harm from biological, chemical, radiological, or nuclear agents). FDA, an agency within the Department of Health and Human Services (HHS), administers these programs. The 21st Century Cures Act included a provision for GAO to study the PRV programs. GAO examined the number of PRVs awarded and redeemed and the drugs for which they were awarded or redeemed, and what is known about the extent to which the PRVs provide incentives for developing drugs to meet unmet needs. GAO analyzed FDA data on awarded and redeemed PRVs for fiscal years 2009 through 2019 and other publicly available information on their transfers and sales. GAO conducted a literature review of peer-reviewed articles published from January 2009 through May 2019 that examined the PRV programs and interviewed FDA officials. GAO also interviewed seven stakeholder groups, seven academic researchers, and seven drug sponsors selected based on factors such as familiarity with PRV programs or drug development. HHS provided technical comments on a draft of this report, which were incorporated as appropriate. The Food and Drug Administration (FDA) awards priority review vouchers (PRV) to drug sponsors that develop drugs for tropical diseases or rare pediatric diseases or to use as medical countermeasures. The PRV—which can be sold to another drug sponsor—may be redeemed later to receive priority review from FDA with a targeted review time of 6 months, rather than the 10-month standard review, for a drug application of the PRV holder's choice. The potential for additional revenue from either marketing a drug about 4 months sooner or from selling the PRV could provide an incentive for drug sponsors to develop drugs for these diseases or conditions. From fiscal year 2009, when the first PRV was awarded, through fiscal year 2019, FDA awarded 31 PRVs, mostly for drugs to treat rare pediatric diseases. Of the 31 PRVs awarded by FDA,17 were sold to another drug sponsor for prices ranging from about $67 million to $350 million, according to available data. As of September 30, 2019, available data show that drug sponsors had redeemed 16 of the 31 PRVs to obtain a shorter FDA review time for drugs to treat conditions and diseases such as human immunodeficiency virus (HIV), type 2 diabetes, and different forms of arthritis. These drug applications may not otherwise qualify for priority review. GAO found few studies that examined the PRV programs, and those that did found the programs had little or no effect on drug development. However, all seven drug sponsors GAO spoke with stated that PRVs were a factor in drug development decisions—six sponsors said they were one of a number of factors, while one sponsor said they were pivotal in its development of a drug. Some academic researchers and stakeholders expressed concerns about the PRVs as incentives for drug development, including the potential for the expected revenue from the sale of a PRV to decline as more are awarded and available for sale.", "document_type": "gao"}
{"report": "The federal government and states share responsibility for the financing and administration of the Medicaid program. With regard to financing, Medicaid is funded jointly by the federal government and states, with FMAP rates ranging from a statutory minimum of 50 percent to a statutory maximum of 83 percent. Under PPACA, expenditures for Medicaid expansion enrollees are matched at 90 percent for fiscal year 2020. Program administrative responsibilities are shared between states and the federal government. State administrative responsibilities include, among other things, determining eligibility, enrolling beneficiaries, and adjudicating claims. With regard to eligibility, states are primarily responsible for verifying eligibility and enrolling Medicaid beneficiaries. These responsibilities include verifying and validating individuals’ eligibility at the time of application and periodically thereafter, accurately assigning enrollees to the appropriate eligibility group, and promptly disenrolling individuals who are not eligible. PPACA requires states to use third-party sources of data to verify eligibility to the extent practicable. Consequently, states have had to make changes to their eligibility systems, including implementing electronic systems for eligibility determination and coordinating systems to share information. In addition, states have had to make changes to reflect new sources of documentation and income used for verification. In certain circumstances, states may delegate responsibility to the federal government to make eligibility determinations. At the federal level, CMS is responsible for overseeing states’ design and operation of their Medicaid programs and ensuring that federal funds are appropriately spent. CMS oversees state enrollment of beneficiaries and reporting of expenditures. For example: CMS reviews and approves states’ Medicaid eligibility verification plans, which rely primarily on information available through data sources—including federal data sources such as the Social Security Administration and the Internal Revenue Services, or state data sources such as state tax records or unemployment information— rather than paper documentation from families. CMS has various review processes in place to ensure that expenditures reported by states are supported and consistent with Medicaid requirements. The agency also has processes to check whether the correct federal matching rates were applied only to expenditures receiving a higher than standard federal matching rate, which can include certain types of services and populations. CMS estimates Medicaid improper payments, including improper payments due to erroneous beneficiary eligibility determinations. Although CMS has not calculated the improper payments related to beneficiary eligibility determinations since 2014, it plans to begin reporting this estimate in November 2019. Our previous work has identified gaps in CMS oversight of Medicaid eligibility determinations, which affect the federal matching rate. An accurate determination of eligibility is critical to ensuring that only eligible individuals are enrolled, that they are enrolled in the correct eligibility group, and that states’ expenditures are appropriately matched with federal funds for Medicaid enrollees. The implications of inaccurate eligibility determinations can be significant, especially given the growth in enrollment and spending of the expansion population, which represented nearly one quarter of program enrollment and federal expenditures in fiscal year 2017. (See fig. 1.) In September 2016, we reported on our undercover testing for determining Medicaid eligibility and the vulnerabilities we found. We found weaknesses that led to inaccurate eligibility determinations. For example, three of eight fictitious applications we submitted to federal and state marketplaces were approved for Medicaid, despite having identity information that did not match Social Security Administration records. These results, while illustrative of the challenges of assuring accurate eligibility determinations, cannot be generalized. With respect to CMS’s reviews of eligibility determinations, in 2015, we also found that CMS did not review federal Medicaid eligibility determinations in the states that delegated such authority to the federal government. Based on our findings, we made the following recommendations. CMS should use information obtained from state and federal eligibility reviews to inform the agency’s review of expenditures for different eligibility groups in order to ensure that expenditures are reported correctly and matched appropriately. In February 2019, we considered this recommendation implemented, as CMS confirmed that it was sharing information between its eligibility reviews and quarterly expenditure reviews regarding Medicaid expansion enrollees. CMS should conduct reviews of federal Medicaid eligibility determinations to ascertain their accuracy and institute corrective action plans where necessary. CMS has taken some action to review federal eligibility determinations; however, until the review results are publicly reported, which CMS expects to occur in November 2019, this recommendation is not fully implemented. We will continue to monitor CMS’s implementation of this recommendation. In August 2018, we reported that improvements in oversight of state expenditures could help CMS ensure that individuals are enrolled in the correct Medicaid eligibility group. CMS processes for reviewing expenditures reported by states and FMAP rates collectively have had a considerable federal financial benefit, with CMS resolving errors that reduced federal spending by over $5.1 billion in fiscal years 2014 through 2017. However, we identified weaknesses in how CMS targets its resources to address risks when reviewing whether states’ expenditures are supported and consistent with Medicaid requirements. For example: CMS devotes similar levels of staff resources to review expenditures despite differing levels of risk across states. For example, the number of staff reviewing California’s expenditures—which represent 15 percent of federal Medicaid spending—is similar to the number reviewing Arkansas’ expenditures, which represents 1 percent of federal Medicaid spending. Additionally, CMS reviews a sample of claims for expansion enrollees to examine Medicaid expansion expenditures, but the sample size does not account for previously identified risks in a state’s program. Specifically, as we noted in a 2015 report, CMS’s sampling review of expansion expenditures was not linked to or informed by reviews of eligibility determinations conducted by CMS, some of which identified high levels of eligibility determination errors. To address these weaknesses, we made three recommendations, including that the Administrator of CMS revise the sampling methodology for reviewing expenditures for the Medicaid expansion population to better target reviews to areas of high risk. CMS concurred with this recommendation, but in November 2018, CMS officials indicated that given the agency’s resources, they believe the current sampling methodology is sufficient and have no plans to revise it. However, we continue to believe action is needed to better target areas of high risk and this recommendation remains unimplemented. Our examination of Medicaid eligibility determinations will continue as we have work underway that will describe how selected states decide the basis of eligibility for individuals who may qualify for Medicaid under more than one category of eligibility, such as a low-income individual with a disability; what is known about the accuracy of Medicaid eligibility determinations and selected states’ processes to improve the accuracy of determinations; and CMS efforts to recoup funds related to eligibility errors. We expect to complete this work early next year. Improvements in Medicaid data could benefit program oversight, including ensuring that only eligible beneficiaries are enrolled. CMS has acknowledged the need for improved Medicaid data and the Transformed Medicaid Statistical Information System (T-MSIS) initiative is the agency’s primary effort—conducted jointly with states—to improve its collection of Medicaid expenditure and utilization data. According to CMS officials, aspects of T-MSIS are designed to broaden the scope and improve the quality of state-reported data, as well as the data’s usefulness for states. T-MSIS also includes automated quality checks that should improve the quality of data that states report. In addition, T-MSIS is designed to capture significantly more data from states than was previously reported. For example, T-MSIS will include a beneficiary eligibility file that will have expanded information on enrollees, such as their citizenship, immigration, and disability status; and expanded diagnosis and procedure codes associated with their treatments. T-MSIS also is intended to benefit states by reducing the number of reports CMS requires them to submit, and by improving program efficiency by allowing states to compare their data with other states’ data in the national repository or with information in other CMS repositories, including Medicare data. With the continued implementation of T-MSIS, CMS has taken an important step toward developing a reliable national repository for Medicaid data. While recognizing CMS’s progress, we have made several recommendations aimed at improving the quality and usefulness of T- MSIS data. For example, we recommended in 2017 that CMS refine its T- MSIS data priority areas to identify those that are critical for reducing improper payments and expedite efforts to assess and ensure their quality. CMS has implemented this recommendation, yet other recommendations that CMS concurred with related to T-MSIS have not been fully implemented, including outlining a specific plan and associated time frames for using T-MSIS data for oversight. We have previously reported that oversight of the Medicaid program could be further improved through leveraging and coordinating program integrity efforts with state agencies, state auditors, and other partners. CMS has engaged state agencies and other partners to promote program integrity through the Medicaid Integrity Institute, a national training program for states, and other partnerships to combat Medicaid fraud. These efforts have created more opportunities for program integrity professionals to collaborate, share best practices, and ultimately increase the effectiveness of their oversight activities. We have also testified that state auditors are uniquely positioned to help CMS in its oversight of state Medicaid programs, because of their roles and responsibilities—which can include carrying out or overseeing their state’s single audits. Through their program integrity reviews, state auditors have identified improper payments in the Medicaid program and deficiencies in the processes used to identify them. For example, state auditors have found that in some cases their state Medicaid agencies’ eligibility determinations did not identify or address beneficiaries’ changes in circumstances, and in other cases relied on incorrect or incomplete income or asset information. A 2018 audit of New Jersey’s Medicaid program found the state was not identifying and disenrolling some deceased individuals. When state auditors conducted a data match to a Social Security number verification service, they found managed care payments of $510,834 and fee-for-service claims of $217,913 for 41 individuals after their reported date of death. Auditors recommended that the eligibility system be reconciled with a Social Security number validation service on a periodic basis to better identify deceased individuals. In 2017, state auditors in North Carolina found that most of the 10 sample county departments of social services did not consistently provide adequate oversight or controls for the eligibility determination of new applications and re-certifications. For new applications, the auditors showed accuracy error rates ranging from 1 percent to nearly 19 percent; for redeterminations of eligibility, accuracy error rates ranged from 1 percent to 23 percent. Based on information from an independent verification service, state auditors in New York found, during a 9-month period in 2014, that 354 Medicaid enrollees were actually deceased, and that the state made $325,030 in Medicaid payments for a subset of these individuals. Auditors noted that the state’s eligibility system did not have a standard process to periodically verify the life status of all enrollees and end coverage for deceased individuals. In April 2019, the Comptroller General and representatives from the National State Auditors Association sent a letter to CMS requesting changes to the Compliance Supplement to leverage state auditors’ ability to examine key areas of Medicaid, including improvements in the oversight of Medicaid eligibility processes. The Compliance Supplement—which is issued by the OMB based on agency input and direction—is used by state auditors during their annual audit of state entities that administer federal financial assistance programs, including Medicaid. In June 2019, OMB issued the 2019 Compliance Supplement, which included changes related to overseeing testing of eligibility determinations that GAO and the state auditors had proposed. Specifically, the supplement now permits state auditors to test eligibility determinations to ensure that beneficiaries qualify for the Medicaid program and are in the appropriate enrollment category. The supplement also notes a requirement for states to coordinate with other state and federal insurance affordability programs, including the federally facilitated exchanges. These changes to the Compliance Supplement will better enable state auditors to audit states’ eligibility determinations to ensure beneficiaries qualify for the Medicaid program and are enrolled in the correct eligibility group. Such eligibility determinations will supplement CMS’s eligibility determination reviews and may yield insights into program weaknesses that CMS could learn from and potentially address nationally. We continue to believe that CMS could help improve program integrity by further providing state auditors with a substantive and ongoing role in auditing their state Medicaid programs. Chairman Toomey, Ranking Member Stabenow, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions you may have. If you or your staff members have any questions concerning this testimony, please contact me at (202) 512-7114 or yocomc@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals who made key contributions to this testimony include Leslie V. Gordon (Assistant Director), Kristin Ekelund (Analyst-in-Charge), Michael Erhardt, Arushi Kumar, and Drew Long. Also contributing were Susan Anthony, Vikki Porter, and Emily Wilson. Take immediate steps to assess and improve the data available for Medicaid program oversight, including, but not limited to, the Transformed Medicaid Statistical Information System (T-MSIS). Such steps could include (1) refining the overall data priority areas in T- MSIS to better identify those variables that are most critical for reducing improper payments, and (2) expediting efforts to assess and ensure the quality of these T-MSIS data. (GAO-17-173) Recommendation implemented; no action needed. Take additional steps to expedite the use of data for program oversight. Such steps should include, but are not limited to, efforts to (1) obtain complete information from all states on unreported T-MSIS data elements and their plans to report applicable data elements; (2) identify and share information across states on known T-MSIS data limitations to improve data comparability; and (3) implement mechanisms, such as the Learning Collaborative, by which states can collaborate on an ongoing basis to improve the completeness, comparability, and utility of T-MSIS data. (GAO-18-70) Status of recommendation; actions needed to implement recommendations Not fully implemented. Continue taking steps to make T-MSIS data usable for Medicaid program oversight, such as (1) obtaining information on the completeness and comparability of T-MSIS data, (2) notifying states of their compliance status and obtaining corrective action plans, and (3) establishing mechanisms for ongoing feedback and collaboration across states. Articulate a specific plan and associated time frames for using T-MSIS data for oversight. (GAO-18-70) Not fully implemented. Outline a specific plan and associate time frames for using T- MSIS data for oversight. GAO, Medicaid: Additional Efforts Needed to Ensure that State Spending is Appropriately Matched with Federal Funds, GAO-16-53 (Washington, D.C.: Oct. 16, 2015). GAO, Medicaid: Further Action Needed to Expedite Use of National Data for Program Oversight, GAO-18-70 (Washington, D.C.: Dec. 8, 2017). Medicaid: CMS Has Taken Steps to Address Program Risks but Further Actions Needed to Strengthen Program Integrity. GAO-18-687T. Washington, D.C.: August 21, 2018. Medicaid: CMS Needs to Better Target Risks to Improve Oversight of Expenditures. GAO-18-564. Washington, D.C.: August 6, 2018. Medicaid: Actions Needed to Mitigate Billions in Improper Payments and Program Integrity Risks. GAO-18-598T. Washington, D.C.: June 27, 2018. Medicaid: Opportunities for Improving Program Oversight. GAO-18-444T. Washington, D.C.: April 12, 2018. Federal Health-Insurance Marketplace: Analysis of Plan Year 2015 Application, Enrollment, and Eligibility-Verification Process. GAO-18-169. Washington, D.C.: December 21, 2017. Medicaid: Further Action Needed to Expedite Use of National Data for Program Oversight. GAO-18-70. Washington, D.C.: December 8, 2017. Improper Payments: Improvements Needed in CMS and IRS Controls over Health Insurance Premium Tax Credit. GAO-17-467. Washington, D.C.: July 13, 2017. Medicaid: Program Oversight Hampered by Data Challenges, Underscoring Need for Continued Improvements. GAO-17-173. Washington, D.C.: January 6, 2017. Patient Protection and Affordable Care Act: Results of Enrollment Testing for the 2016 Special Enrollment Period. GAO-17-78. Washington, D.C.: November 17, 2016. Health Care: Results of Recent Undercover Testing for Patient Protection and Affordable Care Act Coverage, and Review of Market Concentration in the Private Insurance Markets. GAO-16-882T. Washington, D.C.: September 14, 2016. Patient Protection and Affordable Care Act: Results of Undercover Enrollment Testing for the Federal Marketplace and a Selected State Marketplace for the 2016 Coverage Year. GAO-16-784. Washington, D.C.: September 12, 2016. Patient Protection and Affordable Care Act: Final Results of Undercover Testing of the Federal Marketplace and Selected State Marketplaces for Coverage Year 2015. GAO-16-792. Washington, D.C.: September 9, 2016. Patient Protection and Affordable Care Act: CMS Should Act to Strengthen Enrollment Controls and Manage Fraud Risk. GAO-16-29. Washington, D.C.: February 23, 2016. Medicaid: Additional Efforts Needed to Ensure that State Spending is Appropriately Matched with Federal Funds. GAO-16-53. Washington, D.C.: October 16, 2015. Medicaid: Additional Actions Needed to Help Improve Provider and Beneficiary Fraud Controls. GAO-15-313. Washington, D.C.: May 14, 2015. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Medicaid, a joint federal-state health care program, is one of the nation's largest sources of funding for medical and other health-related services for tens of millions of low income and medically needy individuals. In fiscal year 2018, estimated federal and state expenditures for Medicaid were $629 billion. The size and complexity of Medicaid make the program particularly vulnerable to improper payments—including payments made for people not eligible for Medicaid. States have significant flexibility to design and implement their Medicaid programs based on their unique needs. These programs are administered at the state level, overseen at the federal level by CMS, and jointly funded by the states and federal government. The federal government matches most state expenditures for Medicaid services based on a statutory formula. Under the Patient Protection and Affordable Care Act, states have the option to expand their Medicaid programs to cover nearly all adults with incomes at or below 133 percent of the federal poverty level. States that choose to expand their programs receive a higher federal matching rate for the Medicaid expansion enrollees. This testimony will cover improvements needed to ensure accurate eligibility determinations and focuses on (1) CMS's oversight of Medicaid eligibility and related expenditures; (2) CMS's efforts to improve Medicaid data; and (3) other opportunities to improve oversight and ensure appropriate enrollment. This testimony is generally based on GAO findings and recommendations on the Medicaid program issued from 2015 through 2018, and steps taken to address them through September 2019. The Centers for Medicare & Medicaid Services (CMS) has taken steps to improve its oversight of the Medicaid program; however, GAO has identified areas where additional actions could improve program oversight and ensure that only eligible individuals are enrolled in the Medicaid program. These actions include closing gaps in oversight of eligibility determinations and related expenses, improving data, and furthering federal-state collaboration. Gaps in oversight of Medicaid eligibility determinations and related expenses. Since 2014, CMS has not estimated improper payments due to erroneous eligibility determinations; it plans to report these estimates in November 2019. GAO found that for fiscal year 2017 Medicaid expansion enrollees accounted for nearly a quarter of all Medicaid enrollees and federal Medicaid expenditures. GAO's prior work has identified gaps in CMS oversight, which affects the federal match. An accurate determination of eligibility is critical to ensuring that only eligible individuals are enrolled, that they are enrolled in the correct eligibility group, and that states' expenditures are appropriately matched with federal funds for Medicaid enrollees. GAO recommended that CMS conduct reviews of federal Medicaid eligibility determinations to ascertain their accuracy and institute corrective actions where necessary, and revise the sampling methodology for reviewing expenditures for the expansion population. CMS concurred with these recommendations, though has since indicated that it will not revise the sampling methodology. We continue to believe that additional steps are needed to fully implement these recommendations. Better Medicaid data. Improvements in Medicaid data could aid program oversight to ensure that only eligible beneficiaries are enrolled. CMS officials acknowledged the need for improved data and cited the Transformed Medicaid Statistical Information System (T-MSIS) initiative as its primary effort—conducted jointly with states—to improve the collection of Medicaid expenditure and utilization data. According to CMS officials, aspects of T-MSIS are designed to broaden the scope and improve the quality of state-reported data, as well as the data's usefulness to states. GAO made a series of recommendations related to T-MSIS. CMS concurred with the recommendations, but some have not been fully implemented, including expediting the use of T-MSIS data for oversight, and outlining a plan and associated time frames for using the data for oversight. Further federal-state collaboration needed for oversight and appropriate enrollment. GAO has previously reported that collaborative activities between the federal government and the states are important to improving oversight of the Medicaid program. CMS has ongoing efforts to engage state agencies and others through a national Medicaid training program for state officials and partnerships to combat Medicaid fraud. Recently, steps were taken to better enable state auditors to audit states' eligibility determinations to ensure beneficiaries qualify for the Medicaid program and are enrolled in the correct eligibility group. GAO has previously suggested that CMS could leverage the unique qualifications of state auditors and help improve program integrity by further providing state auditors with a substantive and ongoing role in auditing state Medicaid programs.", "document_type": "gao"}
{"report": "VA provides or pays for long-term care—ranging from assistance with dressing and bathing to clinical care for spinal injuries or dementia— through three institutional and 11 noninstitutional programs. (See fig. 1 for a list of VA’s institutional and noninstitutional long-term care programs and app. I for brief descriptions of these programs.) VA’s long-term care programs serve over 500,000 veterans with a wide range of characteristics and needs. Further, certain Community Nursing Homes, Adult Day Health Care, and Hospice and Respite Care programs have specially trained staff to serve veterans with dementia, and the Spinal Cord Injury and Disability Home Care program and certain VA Community Living Centers are equipped to serve veterans needing ventilator care. All veterans enrolled in the VA health care system are eligible for VA’s basic medical benefits package, which includes coverage for certain institutional and noninstitutional long-term care services. A veteran’s eligibility for fully or partially covered nursing home care is determined by the veteran’s priority for care, which is generally based on the veteran’s service-connected disability status. VA must cover the full cost of nursing home care for veterans who need this care for a service-connected disability and for veterans with service-connected disabilities rated at 70 percent or more. Veterans’ placement into particular long-term care programs may depend on their clinical needs, disability ratings, preferences, and the availability of VA programs. When funds are limited, the agency may prioritize program placement based on veterans’ service- connected disability ratings. Decisions about which long-term care programs may be the best fit are made at the VA medical center (VAMC) level between VA providers, veterans, and their families. As we reported in February 2020, VA data shows that utilization of and spending for VA long-term care programs generally increased from fiscal years 2014 through 2018. Specifically, the number of veterans receiving care in VA’s long-term care programs increased 14 percent from fiscal years 2014 through 2018, from 464,071 to 530,327 veterans, while spending grew 33 percent from $6.8 billion to $9.1 billion. Further, we found that VA projects utilization and expenditures for long-term care to increase for most of the programs included in VA’s EHCPM from fiscal years 2017 through 2037. Specifically, over that time period VA’s model projects the following: Utilization of long-term care—in terms of various VA workload units— is projected to grow in one of the two institutional programs and nine of the 10 noninstitutional programs included in the EHCPM from fiscal years 2017 through 2037. Spending, which VA reports as expenditures, is projected to more than double from fiscal years 2017 through 2037, increasing from $6.9 billion to $14.3 billion. (See fig. 2.) VA also projects that the proportion of expenditures for institutional long-term care will decrease from 63 percent to 53 percent while the proportion of noninstitutional program expenditures is projected to grow from 37 percent to 47 percent in that same time period. According to VA officials, these projected increases are due to a variety of factors, including that VA plans to continue expanding the availability of noninstitutional care, and plans on providing care to an increasing number of aging veterans and veterans rated in the highest service- connected disability groups. Officials also noted that expanding veterans’ access to noninstitutional care programs is less costly than institutional care, and veterans prefer to delay or reduce the amount of institutional care they receive. VA’s strategies to meet the growing demand for long- term care are operationalized by GEC at the program level and implemented at the regional and VAMC level. In our February 2020 report, we found that VA faces a number of key challenges in meeting veterans’ growing demand for long-term care: workforce shortages, geographic alignment of care, and difficulty meeting veterans’ needs for specialty care. While GEC recognizes and has taken some steps to address the challenges it faces, it has not established measurable goals for its efforts to address these three key challenges: GEC has not established measurable goals to address workforce shortages, such as staffing targets to address the waitlist for the Home-Based Primary Care program. GEC has not established measurable goals for its efforts to address the geographic alignment of care, such as specific targets for providing long-term care within the Home Telehealth and Veteran Directed Care programs. GEC has not established measurable goals for its efforts to address difficulties meeting veterans’ needs for specialty care, such as specific targets for the number of available ventilators or the number of caregivers educated to help veterans with dementia. As we noted in our report, without measurable goals, VA is limited in its ability to better plan for and understand progress towards addressing the challenges it faces meeting veterans’ long-term care needs. To address this issue, we recommended that GEC develop measurable goals for its efforts to address these key long-term care challenges. VA concurred with this recommendation. In our February 2020 report we also found that VA had identified, but had not planned to take steps to fully address, challenges at the VAMC level that affect VA’s ability to meet veterans’ long-term care needs: VA identified that VAMCs do not have a consistent approach to managing VA’s 14 long-term care programs. At VAMCs where there are not GEC staff, long-term care programs could be run by one or more departments within the VAMC, for example the Nursing department or the Social Work department. GEC officials told us that this fragmentation hinders standardization and the ability to get veterans the appropriate care. VA also identified that VAMCs use different approaches to assess the amount of noninstitutional long-term care services veterans need. While GEC has developed a tool to improve the consistency in these determinations, VA has not required the tool be used in all VAMCs, as of October 2019. As a result, decisions about the amount of services veterans receive may vary by VAMC. To address these issues, we recommended that GEC leadership set time frames for and implement (1) a consistent GEC structure at the VAMC level and (2) VAMC-wide standardization of the tool for assessing noninstitutional program needs of veterans. VA concurred with our recommendations. Chairwoman Brownley, Ranking Member Dunn, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact A. Nicole Clowers, Managing Director, Health Care at (202) 512- 7114 or clowersa@gao.gov or Sharon Silas, Director, Health Care, at (202) 512-7114 or silass@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contacts named above, key contributors to this statement were Karin Wallestad (Assistant Director), Luke Baron (Analyst-in-Charge), Summar C. Corley, and Laurie Pachter. Also contributing to the underlying report for this statement were Kye Briesath, Vikki Porter, Corinne Quinones, and Jennifer Rudisill. Appendix I: Department of Veterans Affairs’ (VA) Institutional and Noninstitutional Long- Term Care Program Descriptions This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Department of Veterans Affairs (VA) provides or purchases long-term care for eligible veterans through 14 long-term care programs in institutional settings like nursing homes and noninstitutional settings like veterans' homes. From fiscal years 2014 through 2018, VA data show that the number of veterans receiving long-term care in these programs increased 14 percent (from 464,071 to 530,327 veterans), and obligations for the programs increased 33 percent (from $6.8 to $9.1 billion). VA projects demand for long-term care will continue to increase, driven in part by growing numbers of aging veterans and veterans with service-connected disabilities. Expenditures for long-term care are projected to double by 2037, as shown below. According to VA officials, VA plans to expand veterans' access to noninstitutional programs, when appropriate, to prevent or delay nursing home care and to reduce costs. VA currently faces three key challenges meeting the growing demand for long-term care: workforce shortages, geographic alignment of care (particularly for veterans in rural areas), and difficulty meeting veterans' needs for specialty care. VA's Geriatrics and Extended Care office (GEC) recognizes these challenges and has developed some plans to address them. However, GEC has not established measurable goals for these efforts, such as specific staffing targets for programs with waitlists or specific targets for providing telehealth to veterans in rural areas. Without measurable goals, VA is limited in its ability to address the challenges it faces meeting veterans' long-term care needs.", "document_type": "gao"}
{"report": "IHS was established within the Public Health Service in 1955 in order to meet federal treaty obligations to provide health services to members of federally recognized AI/AN tribes primarily in rural areas on or near reservations. IHS oversees its provision of health care services through a decentralized system of 12 area offices, which are led by area directors and located in 12 geographic areas. IHS’s headquarters office is responsible for setting national health care policy, ensuring the delivery of quality comprehensive health services, and advocating for the health needs and concerns of AI/AN people. The area offices are responsible for monitoring federally operated IHS facilities’ operations and finances, and providing guidance and technical assistance. IHS’s 12 area offices oversee 168 service units which provide care at the local level through a total of 742 federally operated and tribally operated hospitals, health centers, and other health facilities. The types of services offered by these facilities vary, but most commonly include primary care and emergency care, as well as some ancillary and specialty services. Table 1 displays the number of federally operated and tribally operated facilities as of February 2019. If federally operated or tribally operated facilities are unable to provide needed care, they may contract for health services from private providers through the PRC program. Patients must meet certain eligibility and administrative requirements in order to qualify for this care—including having exhausted all other health care resources available to them and living on a federally recognized Indian reservation or within a designated PRC delivery area. The PRC program is funded through the annual appropriations process and administered at the local level by individual PRC programs that are often affiliated with local facilities. Individual PRC programs may be federally or tribally administered, and as of fiscal year 2018, IHS administered 39 percent of PRC appropriations, and tribes administered the remaining 61 percent. PRC funding is limited and has traditionally been reserved for the most critical cases. IHS has established five medical priority levels. Funds permitting, federally administered PRC programs first pay for all of the highest priority services, and then all or some of the lower priority services. IHS’s five PRC medical priority levels are 1. Emergent and acutely urgent care services, which include treatment for threats to life, limb, or senses; 2. Preventive care services, which include prenatal care and 3. Primary and secondary care services, which include scheduled ambulatory services for nonemergent conditions, and specialty consultations; 4. Chronic tertiary and extended care services, which include rehabilitation care, skilled nursing facility care, and organ transplants; and 5. Excluded services, which include cosmetic and experimental procedures. Beginning in 2014, PPACA allowed states to expand Medicaid eligibility to non-elderly, non-pregnant adults who are not eligible for Medicare and whose income does not exceed 133 percent of the federal poverty level. As of September 2018, there were 32 “expansion states”—those states including the District of Columbia that chose to expand Medicaid eligibility to this additional adult population—and 19 “non-expansion states”—those that had not expanded Medicaid eligibility. PPACA also required the establishment of health insurance exchanges in 2014—marketplaces where individuals may compare and select among health insurance plans offered by participating private insurers. PPACA included a number of provisions that reduced these plans’ costs— including premiums and cost-sharing, such as deductibles and copayments—for eligible enrollees, including certain AI/AN. Our analysis of IHS data shows that from fiscal year 2013 through fiscal year 2018, the percent of patients at 73 federally operated IHS hospitals and health centers who reported having health insurance coverage increased an average of 14 percentage points, from 64 percent in fiscal year 2013 to 78 percent in fiscal year 2018. The majority of coverage gains occurred in fiscal years 2014 through 2016 (see fig. 1). Patients at federally operated IHS facilities reported obtaining health insurance coverage from several sources. The largest increase in coverage occurred among those reporting Medicaid coverage. On average, 41 percent of IHS patients in fiscal year 2013 reported they had coverage through Medicaid at some point during the year; this number increased to 53 percent in fiscal year 2018. In comparison, the percent of patients at each facility who reported having Medicare and the percent who reported having private insurance at some point during the year each increased an average of two percentage points from fiscal years 2013 to 2018. (See fig. 2.) While the average percent of patients reporting health care coverage increased across all federally operated IHS facilities, our analysis of IHS data showed substantial variation in the magnitude of these increases. Specifically, from fiscal year 2013 through fiscal year 2018, increases at each of the 73 facilities ranged from a low of 2 to a high of 31 percentage points. Forty-four federally operated IHS facilities experienced an increase in the percent of patients with coverage over this time period of more than 10 percentage points (see fig. 3). Our analysis of IHS data shows that federally operated IHS facilities in states that expanded Medicaid had larger increases in health insurance coverage compared with such facilities in states that had not expanded Medicaid. Specifically, federally operated IHS facilities in Medicaid expansion states experienced an average 17 percentage point increase in patients reporting any form of health coverage, compared with an average 8 percentage point increase at federally operated IHS facilities in states that did not expand Medicaid. However, these increases in coverage were not spread evenly among the facilities. (See fig. 4.) IHS officials we interviewed also reported that a variety of factors in addition to Medicaid expansion likely affected the number of patients at federally operated IHS facilities who reported having health insurance coverage. Specifically, officials we interviewed at all of the 11 selected federally operated IHS facilities cited efforts at their facilities that helped increase coverage, such as increasing the number of onsite patient benefits coordinators to help enroll patients in all forms of health coverage and enhancing efforts to ensure that all patients were screened for coverage. For example, one federally operated IHS facility reported renovating its office to, among other things, move the patient benefits coordinator near the waiting room, which allowed patients to be immediately screened after walking in for an appointment. Officials we interviewed at nearly all of the selected federally operated IHS facilities also noted that their outreach and education efforts about the importance of health insurance coverage may have helped to increase enrollment. Officials we interviewed at all of the selected federally operated IHS facilities said they were engaged in such activities which included broadcasting public service announcements, posting newspaper advertisements, and promoting insurance during community events. Officials from most of the 12 IHS area offices also reported collaborating with tribes to conduct outreach and education to enhance enrollment. Officials at many IHS area offices also noted that external factors may have also played a role in increasing coverage levels, such as improvements in the local economy, which officials said led to increases in the number of patients with private health insurance. Additionally, entities outside of IHS also implemented initiatives to increase coverage for patients at federally operated IHS facilities. For example, IHS officials stated that some patients obtained health insurance through the health insurance exchanges, and in some cases, the tribe paid all premiums, coinsurance, and deductibles for these plans. In addition, a number of area Indian health boards worked together to develop a train-the-trainer program to disseminate information and resources to encourage enrollment and share information on the benefits of having health coverage. Third-party collections across all federally operated IHS facilities increased 51 percent from fiscal year 2013 through fiscal year 2018, according to our analysis of IHS data. Specifically, total third-party collections increased from $708 million in fiscal year 2013 to about $1.07 billion in fiscal year 2018 while the number of patients seeking care remained constant. Medicaid collections accounted for 65 percent of the total $360 million increase, though collections from Medicare, private insurance, and Veterans Affairs also increased during this period. For example, Medicaid collections grew 47 percent, from $496 million in fiscal year 2013 to $729 million in fiscal year 2018. (See fig. 5.) While third-party collections at federally operated IHS facilities collectively increased from fiscal year 2013 through 2018, there was significant variation in changes for individual facilities. IHS officials we interviewed noted several reasons why third-party collections may vary over time and by location, including the size of the facility and any changes in the number of providers, patients, or business office staff that process billing and collections; the ability to collect payment from certain tribal health insurance, which may opt to not pay for services provided to enrolled members; and the number of patients enrolled in Medicaid managed care plans, which may identify IHS facilities as out-of-network providers and not pay for covered services. IHS and federally operated facility officials we interviewed noted that gains in health insurance coverage during this time period contributed to increases in collections. In addition, officials we interviewed from most of the 12 area offices and 11 selected federally operated IHS facilities described steps they took to enhance collections. More specifically, officials from seven area offices discussed initiating steps to improve billing and collections functions for federally operated IHS facilities in their area; at one area office this involved creating a new area-level position focused on revenue enhancement at federally operated IHS facilities. Additionally, officials we interviewed at six federally operated IHS facilities identified steps they took to enhance the accuracy and efficiency of facilities’ collections, noting efforts such as improving training related to coding and billing. For example, officials at one of these facilities described convening a team to review why all claims related to a specific service were being rejected. The team then instituted changes to their billing procedures that resulted in the facility collecting payments for these services. Officials we interviewed at selected tribally operated facilities and tribal organizations—including national tribal organizations and area Indian health boards—described increases in health insurance coverage and collections at some tribally operated facilities that occurred from 2013 through 2018. Specifically, some tribal organization officials reported increases in coverage at facilities located in states that had expanded their Medicaid programs, compared with those that had not. For example, officials at one tribally operated facility noted that the percent of their patients with health coverage increased by 10 percentage points from 2013 to 2018. Similar to federally operated IHS facilities, officials we interviewed from some tribally operated facilities said they focused on screening patients for coverage at the time of service, including by increasing the number of patient benefits coordinators and always having staff available to help enroll patients in coverage. These officials also noted that they conducted outreach and enrollment activities to inform patients of the importance of having coverage and benefitting from outreach and education activities conducted directly by local tribes, including through messages that emphasized the importance of coverage for the tribe and tribally operated facility. Officials from a national tribal organization told us that one tribally operated facility placed stickers on all equipment purchased with third- party collections as a way to educate patients about the benefits of having health insurance coverage and to encourage further enrollment in coverage. Officials we interviewed at selected tribally operated facilities and national tribal organizations also described increases in third-party collections that occurred from 2013 through 2018 at many tribally operated facilities— particularly those located in Medicaid expansion states. For example, officials from one tribally operated facility told us that they anticipated that their third-party collections for 2018 would be more than twice the amount they collected for 2013. Similar to federally operated IHS facilities, officials we interviewed from some tribally operated facilities noted that their facilities had enhanced collections by making improvements to their billing processes and taking steps to increase patient volume. For example, officials at one tribally operated facility said they recently began allowing non-tribal members to receive care at their facility—an option available to tribally operated facilities but not to federally operated IHS facilities—as a way to increase third-party collections and bolster the facility’s long-term sustainability. Some officials also noted that not all tribally operated facilities experienced increases in collections, in part because of decreases or limitations in the number of providers, patients, or business office staff that process billing and collections. Similar to federally operated IHS facilities, officials from tribally operated facilities noted that the enrollment of patients in Medicaid managed care plans also reduced their ability to collect payment for covered services because these plans often identify the facilities as out-of-network providers and therefore do not pay for covered services provided onsite. Officials we interviewed from selected federally operated and tribally operated facilities stated that increases in coverage and third-party collections helped them to (1) continue their facilities’ operations, (2) expand the services they offer onsite at their facilities, and (3) expand the services they cover offsite through IHS’s PRC program. Officials we interviewed from all 17 selected federally operated and tribally operated facilities noted that they used increased third-party collections to fund their continued operations. Even as officials we interviewed from nearly all of the 11 selected federally operated IHS facilities reported that their facilities’ third-party collections had grown from fiscal years 2013 to 2018, officials from most of these facilities also said they relied more heavily on these collections to support their continued operations. Officials we interviewed from all of the IHS area offices told us that third-party collections provide a vital source of funding for federally operated IHS facilities in their area. These collections allowed them to maintain a level of operations that would otherwise be challenging, for reasons such as increasing costs of payroll and of maintaining an aging infrastructure. In addition, officials we interviewed from most of the selected federally operated IHS facilities reported using third-party collections to fund a substantial and increasing portion of their payroll costs. Officials at many of the IHS area offices and most of the selected federally operated IHS facilities we interviewed also reported using third-party collections to ensure that their facility met all required standards, including those required for ongoing accreditation, or to undertake any needed maintenance such as by repairing roofs and heating systems. Some of these officials also reported using third-party collections to repair or replace medical equipment that was broken or had exceeded its intended lifespan. Table 2 displays examples of how selected federally operated and tribally operated facilities reported using third-party collections. Officials we interviewed from most of the 17 selected federally operated and tribally operated facilities told us they used increased third-party collections to expand the volume or scope of services they offered onsite as a way to better meet patients’ medical needs. With respect to increasing the volume of services provided, officials at most of these facilities said they added providers and medical equipment to provide patients with more timely access to services. In one example, officials from a federally operated IHS hospital said they added about 30 additional nurses from 2013 to 2018 as a result of increased third-party collections. As a result of increases in the number of providers at their facilities, officials we interviewed from several federally operated IHS facilities said they were able to schedule appointments for patients more quickly, which reduced wait times for an appointment—including two facilities that reported being able to newly offer same-day appointments. Officials from facilities that expanded the scope of services provided said they did so by adding new specialties, such as behavioral health and dentistry, purchasing new medical equipment such as hospital beds, dental chairs, and magnetic resonance imaging machines, and funding health promotion and education activities such as those related to diabetes education. (See fig. 6.) To support efforts to expand services and bolster their sustainability, officials from most of the 17 federally operated and tribally operated facilities said they used third-party collections to offer more competitive salaries and bonuses for providers. In addition, officials from a few of the 12 IHS area offices told us that federally operated facilities in their area used third-party collections to fund projects to construct nearby housing for providers. In another example, officials from a national tribal organization noted that the use of third-party collections to enhance provider salaries at one facility led to a decrease in provider turnover from about 40 percent prior to 2014 to 14 percent in 2018. In addition, officials from many of the IHS area offices told us that some federally operated facilities in their area reported using third-party collections accumulated over multiple years to make investments in expanding their facilities to provide the space necessary to support these additional services. For example, according to IHS officials, one federally operated IHS facility reported using $7 million in third- party collections to fund an over 11,000 square foot expansion to house an expanded emergency room and a new urgent care clinic; two federally operated IHS facilities reported using third-party collections to purchase modular buildings to provide medical services such as audiology, behavioral health, and dental services; and one federally operated IHS facility reported saving third-party collections for six years to fund the construction of a new 23,000 square foot building to provide additional space for an increased volume of services, including dental, optometry and physical therapy services, and to pay for the new medical equipment to support these services (see fig. 7). Officials from some IHS area offices stated that the extent to which federally operated IHS facilities in their area invested in expanding onsite services largely depended on the level of facilities’ third-party collections. Specifically, facilities experiencing larger increases in collections, such as larger facilities or those located in Medicaid expansion states, were able to invest more heavily in an expansion of onsite services compared to those that had lower increases in collections, according to these officials. To identify their facilities’ needs, officials from federally operated and tribally operated facilities reported using a variety of approaches. For example, officials from three IHS area offices and one tribally operated facility said they analyzed PRC data to identify the services that patients were obtaining through that program, and worked to bring those services onsite. Officials from two federally operated IHS facilities also noted that they incorporated local tribal input as they identified local needs and projects to fund. For example, these officials told us that their facilities were in the process of adding new specialty services onsite, including acupuncture, chiropractor, and eye clinic services, at the request of their local tribes. The recent growth in third-party collections has made it possible for many federally operated IHS facilities to consider funding a range of projects, and IHS officials said they relied on established procedures to fund these projects. According to IHS officials, local facility officials draft annual spending proposals to identify the resources, including third-party collections, that they would like to use to address their facilities’ needs. These proposals are provided to each facility’s governing board for review; the governing board is comprised of area office and facility officials whose top priority is maintaining accreditation and ensuring patient safety at each facility, according to IHS officials. Once these basic needs are met, IHS officials told us that facilities may begin to identify and fund projects to expand access to health services. Officials from IHS, as well as some of the 17 selected federally operated and tribally operated facilities, told us that increased coverage and collections allowed for an expansion in the complexity of services provided offsite through the PRC program. Specifically, officials reported that an increase in the percent of patients with health insurance, coupled with facilities’ enhanced onsite services, has led PRC programs to be able to expand the level of care that they can offer. For example, they stated that increases in the health insurance coverage of patients have led to a smaller percent of patients needing to access PRC, since patients may use their coverage to obtain needed services directly from other private providers. In addition, an expansion of available services onsite at federally operated and tribally operated facilities resulting from increased collections reduced the need for some patients to use PRC. From 2013 through 2018, most IHS-administered PRC programs moved from covering only the most acute and emergent cases to funding nearly all types of care covered through the PRC program, according to our analysis of IHS data and interviews with agency officials. Specifically, IHS officials we interviewed told us that prior to 2014, most PRC programs administered by the agency were only able to fund care for the most acute and emergent cases—referred to as priority level 1. Our analysis of IHS data showed that these PRC programs were increasingly able to fund additional medical priority levels of care each year from fiscal year 2015— the first year that such data were available—through fiscal year 2018, with most IHS-administered programs funding care through priority level 4 in fiscal year 2018. (See fig. 8.) Officials we interviewed at some of the 17 selected federally operated and tribally operated facilities that had been able to both expand services onsite and offsite through PRC funds told us that these changes have made a large impact on patients’ health and quality of life. For example, officials at some federally operated IHS facilities reported that having more providers onsite has allowed them to offer patients more rapid access to care, and officials from some tribally operated facilities reported that an expansion of onsite services has allowed them to serve more patients. Officials at some of the selected federally operated and tribally operated facilities reported that an expansion of onsite services has also reduced the need for some patients to travel long distances to obtain diagnostic services and specialty care through the PRC program. In addition, officials from two IHS area offices noted that PRC has been able to pay for services such as patients’ long-awaited knee and hip replacements, which have enabled patients to return to normal activities of life and reduce their need for pain management. We provided a draft of this report for review and comment to the Secretary of Health and Human Services. The Department did not have any comments on the draft report. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of the Department of Health and Human Services and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or farbj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In the years since the Patient Protection and Affordable Care Act (PPACA) authorized states to expand access to Medicaid and offer health insurance through the exchanges in 2014, the percent of American Indian and Alaska Native (AI/AN) in the general population with health insurance has increased. Specifically, according to an analysis of U.S. Census Bureau’s American Community Survey data, the percent of nonelderly AI/ANs with health insurance coverage increased from 70 percent in 2013 to 78 percent in 2017. (See fig. 9.) While the estimated percent of AI/AN nationwide reporting health insurance coverage increased from 2013 to 2017, these increases in coverage were not evenly distributed among the states, according to an analysis of U.S. Census Bureau’s American Community Survey data. The estimated percent of AI/AN reporting health insurance increased more in states that expanded Medicaid compared to those that did not. (See fig. 10.) In addition to the contact named above, Kristi Peterson, Assistant Director; Patricia Roy, Analyst-in-Charge; Michelle Duren; and Lisa Rogers made key contributions to this report. Also contributing were Todd Anderson, Krister Friday, Ethiene Salgado-Rodriguez, and Emily Wilson Schwark. Tribal Consultation: Additional Federal Actions Needed for Infrastructure Projects. GAO-19-22. Washington, D.C.: March 20, 2019. Indian Health Service: Spending Levels and Characteristics of IHS and Three Other Federal Health Care Programs. GAO-19-74R. Washington, D.C.: December 10, 2018. Indian Health Service: Considerations Related to Providing Advance Appropriation Authority. GAO-18-652. Washington, D.C.: September 13, 2018. Indian Health Service: Agency Faces Ongoing Challenges Filling Provider Vacancies. GAO-18-580. Washington, D.C.: August 15, 2018. High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others. GAO-17-317. Washington, D.C.: February 15, 2017. Indian Health Service: Actions Needed to Improve Oversight of Quality of Care. GAO-17-181. Washington, D.C.: January 9, 2017. Indian Health Service: Actions Needed to Improve Oversight of Patient Wait Times. GAO-16-333. Washington, D.C.: March 29, 2016. Indian Health Service: Opportunities May Exist to Improve the Contract Health Services Program. GAO-14-57. Washington, D.C.: December 11, 2013. Indian Health Service: Most American Indians and Alaska Natives Potentially Eligible for Expanded Health Coverage, but Action Needed to Increase Enrollment. GAO-13-553. Washington, D.C.: September 5, 2013. Indian Health Service: Increased Oversight Needed to Ensure Accuracy of Data Used for Estimating Contract Health Service Need. GAO-11-767. Washington, D.C.: September 23, 2011. Indian Health Service: Updated Policies and Procedures and Increased Oversight Needed for Billings and Collections from Private Insurers. GAO-10-42R. Washington, D.C.: October 22, 2009. Indian Health Service: Health Care Services Are Not Always Available to Native Americans. GAO-05-789. Washington, D.C.: August 31, 2005.", "summary": "IHS provides care to American Indians and Alaska Natives through a system of health care facilities. The Patient Protection and Affordable Care Act (PPACA) provided states with the option to expand their Medicaid programs, and created new coverage options beginning in 2014, including for American Indians and Alaska Natives. GAO was asked to review how PPACA has affected health care coverage and services for American Indians and Alaska Natives. In this report, GAO describes (1) trends in health insurance coverage and third-party collections at federally operated and tribally operated facilities from fiscal years 2013 through 2018, and (2) the effects of any changes in coverage and collections on these facilities. To address these objectives, GAO analyzed IHS data on coverage, third-party collections, and PRC. GAO interviewed IHS officials from headquarters and all 12 area offices, as well as from 17 facilities selected to include a mix of federally operated and tribally operated hospitals and health centers in states that both had and had not expanded their Medicaid programs as of September 2018. GAO interviewed officials from 11 federally operated IHS facilities and 6 tribally operated facilities. GAO provided a draft of this report to the Secretary of Health and Human Services for comment. The Department did not have any comments on the draft report. GAO's analysis of Indian Health Service (IHS) data shows that from fiscal years 2013 through 2018, the percent of patients at federally operated IHS hospitals and health centers that reported having health insurance coverage increased an average of 14 percentage points. While all federally operated IHS facilities reported coverage increases, the magnitude of these changes differed by facility, with those located in states that expanded access to Medicaid experiencing the largest increases. Federally operated IHS facilities' third-party collections—that is, payments for enrollees' medical care from public programs such as Medicaid and Medicare, or from private insurers—totaled $1.07 billion in fiscal year 2018, increasing 51 percent from fiscal year 2013. Although exact figures were not available, tribally operated facilities, which include hospitals and health centers not run by IHS, also experienced increases in coverage and collections over this period, according to officials from selected facilities and national tribal organizations. Increases in health insurance coverage and third-party collections helped federally operated and tribally operated facilities continue their operations and expand the services offered, according to officials from 17 selected facilities. These officials told GAO that their facilities have been increasingly relying on third-party collections to pay for ongoing operations including staff payroll and facility maintenance. Officials at most facilities with increases in third-party collections also stated that they expanded their onsite services, including increasing the volume or scope of services offered by, for example, adding new providers or purchasing medical equipment. Increased coverage and collections also allowed for an expansion in the complexity of services provided offsite through the Purchased/Referred Care (PRC) program, which enables patients to obtain needed care from private providers if the patients meet certain requirements and funding is available. According to IHS and facility officials, increases in coverage have allowed some patients to access care offsite using their coverage, and an expansion of onsite services has reduced the need for some patients to access PRC. Officials GAO interviewed from federally operated and tribally operated facilities stated that facilities' expansion of onsite and offsite services has led to enhancements in patients' access to care in some instances.", "document_type": "gao"}
{"report": "In 2015, the President signed Executive Order 13693, “Planning for Federal Sustainability in the Next Decade.” This executive order called for federal agencies to, among other things, advance waste prevention and pollution prevention in federal facilities, operations, and vehicles by diverting at least 50 percent of nonhazardous solid waste, including food and compostable material, but not construction and demolition debris, in their internal operations annually, and pursue opportunities for net-zero waste or additional diversion opportunities. In May 2018, Executive Order 13693 was revoked and replaced by Executive Order 13834, “Efficient Federal Operations,” which directed federal agencies to implement waste prevention and recycling measures but no longer included the specific direction to divert at least 50 percent of nonhazardous solid waste, including food and compostable material, annually or to pursue opportunities for net-zero waste or additional diversion opportunities. CEQ and OMB are responsible for implementing and tracking progress for these executive orders. Among its duties, EPA oversees municipal solid-waste management. For example, EPA regulates the management of household, industrial, and manufacturing solid and hazardous wastes under the Resource Conservation and Recovery Act. The objectives of the act include protecting the United States from the hazards of waste disposal, to conserve energy and natural resources by recycling and recovery, and to minimize the generation of hazardous waste. However, the management of nonhazardous solid waste, such as food waste, is left primarily to the states and local governments. Under the act, EPA established solid- waste management guidelines for municipalities that encouraged recycling, including composting food and yard waste. EPA’s Sustainable Materials Management Program, including the Sustainable Management of Food strategic priority area, seeks to reduce the environmental impact of materials through their entire life cycle. Furthermore, available landfill space is decreasing in various parts of the United States; EPA’s FLW activities may help extend the life of those existing landfills and provide opportunities for energy generation. According to USDA officials, USDA has developed a broad range of programs and policies to reduce FLW as a means to support its overarching objectives related to reducing food insecurity, improving food safety, increasing market efficiencies, and enhancing farmer income and rural development. USDA also conducts education and outreach through its network of state and local offices, the Cooperative Extension Service, state Departments of Agriculture, land-grant university partners, and nongovernmental, nonprofit, community, and faith-based organizations. Additionally, the 2018 Farm Bill requires the Secretary of Agriculture to take a number of actions to address FLW: (1) create a FLW Reduction Liaison to coordinate federal, state, local, and nongovernmental programs, and other efforts, to measure and reduce the incidence of FLW; (2) conduct a study on food waste in consultation with the FLW Reduction Liaison and report data collected on food waste and efforts to reduce and prevent such waste; (3) issue guidance outlining the best practices to minimize food waste for donated commodities; (4) enter into cooperative agreements with local or municipal governments to develop and test strategies for planning and implementing municipal compost plans and food waste reduction plans; (5) establish a milk donation program to encourage the donation of milk products produced and processed in the United States, assist individuals in low-income groups, and reduce food waste; and (6) establish a Local Agriculture Market Program to, among other things, promote new business opportunities and marketing strategies to reduce on-farm food waste. Finally, FDA, which is responsible for, among other things, overseeing the safety of about 80 percent of the nation’s food supply, has a limited mission related to FLW. FDA was not involved with establishing the national FLW reduction goal in 2015 but, according to agency officials, has become more engaged in consumer education and outreach to the food industry, hunger relief and food rescue organizations, state and local governments, academia, and other stakeholders on issues related to FLW. By signing the 2018 formal agreement on collaboration and coordination with EPA and USDA, FDA has committed to taking further actions to reduce FLW. Definitions of FLW vary among the various organizations, including federal agencies, working in this area, which inform different methodologies for measuring and reporting FLW. For example, consistent with its focus on advancing the sustainable use of materials, including food, throughout their life cycle to minimize waste and environmental impacts, EPA uses the term “wasted food” instead of “food waste” for food that is not used for its intended purpose because it conveys that a resource with value is being wasted, whereas “food waste” implies that the food no longer has value and needs to be managed as waste. EPA states that “wasted food” is managed in a variety of ways, including through donations to food banks, conversion to animal feed, composting, anaerobic digestion, or sending it to landfills. In contrast, USDA’s Economic Research Service (ERS) defines food loss as edible food that is available for human consumption but that is not eaten. According to ERS, food losses may occur for any number of reasons, including cooking loss and natural shrinkage; loss from mold, pests, or inadequate climate control; and plate waste, which refers to edible food that is served but discarded. In addition, ERS defines food waste as a component of food loss that refers to food discarded by retailers and consumers due to quality concerns, such as blemished food. ERS takes this approach in support of its effort to estimate the nation’s available food supplies, which it adjusts to account for nonedible parts of foods and losses throughout the food supply chain. USDA has noted that definitions of FLW vary worldwide. For example, FAO differentiates food loss from food waste based on the stage of the food supply chain in which the amount of edible food decreased. FAO refers to food loss as the decrease in edible food that occurs throughout the production and processing stages of the food supply chain, whereas food waste occurs at retail and consumer stages of the food supply chain. These varying definitions have led to different methodologies for measuring and reporting FLW. For example, EPA estimates the amount of food from residences, commercial establishments (such as grocery stores), and institutional establishments (such as schools) that is disposed of through landfills or converted to energy, while ERS estimates the amount, value, and calories of postharvest food losses at the retail and consumer stages of the food supply chain. In addition, ERS and FAO FLW estimates do not include inedible parts, whereas, with its focus on materials management, EPA’s estimates do. FLW can occur across the entire food supply chain, occur at more than one stage (e.g., spoilage), or be unique to a specific stage, as seen in figure 1 below. However, the share of total FLW due to each of these causes is currently unknown, according to a USDA report. EPA’s Food Recovery Hierarchy, shown in figure 2 below, focuses on different options for reducing FLW. According to EPA, the top levels of the hierarchy are the best ways to reduce FLW because they create the most benefits for the environment, society, and the economy. Source reduction is the preferred option for reducing FLW because it provides the greatest benefits in terms of environmental sustainability. This is because growing food requires resources, such as land, water, fertilizer, and pesticides. In contrast, food that is sent to landfills generates greenhouse gases, such as methane. Prevention refers to reducing the amount of surplus food generated at any stage of the food supply chain. For example, businesses, such as restaurants, may prevent FLW through better planning and food preparation techniques. Diversion includes recovering food by donating edible food to feed hungry people or sending food scraps to feed animals. Diversion also includes recycling food scraps for industrial uses, such as waste-to- energy generation or anaerobic digestion, or for composting. Disposal refers to food that is sent to landfills, incinerators, or washed into sewers. According to USDA, some FLW is inevitable and, therefore, entirely eliminating FLW is unrealistic. For example, USDA ERS reports that there is a practical limit to how much FLW in the United States could be reduced, given different factors, such as food safety concerns, the perishability of foods, storage and temperature considerations, and risk management for production and marketing uncertainties, and resource constraints to recover uneaten food for another use, among others. According to USDA officials, to be successful, FLW reduction strategies should consider the economic incentives and disincentives faced by stakeholders across the food supply chain. Nonfederal stakeholders we interviewed cited various challenges that exist to reducing FLW in the United States. Through our analysis of those interviews, we identified three key areas: (1) limited data and information about the amounts and causes of FLW; (2) a lack of awareness and education about FLW; and (3) limited infrastructure and capacity, which can hamper efforts to reduce FLW. In some instances, the nonfederal stakeholders also provided their views for ways federal agencies could potentially address the identified challenge areas. Through interviews with nonfederal stakeholders, we identified limited data and information about the amounts and causes of FLW as a challenge to reducing FLW in the United States. For example, several stakeholders told us that data gaps associated with the different food supply-chain stages make it challenging to estimate FLW. For example: An international organization published a study in 2011 that included estimates of FLW by different regions and different stages of the food supply chain. The organization reported in its study that there were major data gaps in the knowledge of global FLW, such as the causes of FLW. Representatives of this organization told us that a challenge to measuring and estimating FLW is the lack of data on the various stages of the food supply chain. They are proposing a new methodology intended to help countries measure FLW along the supply chain in a cost-effective manner and monitor progress in reducing FLW. Researchers from two academic institutions told us there are challenges to estimating farm-production food losses. For example, researchers from one of these academic institutions told us that farm- production losses fluctuate from year to year based on changes in markets and growing conditions, such as weather, which can make estimating FLW more challenging. In addition, these researchers told us more information is needed about how different economic factors, such as the existence of secondary (alternative) markets to sell excess food, or changes in farming costs such as increases in labor costs, may influence FLW. One nonprofit organization reported that data at the farm-production stage of the food supply chain are limited, including data on what happens to some food at that stage. For example, there are limited data about whether produce that goes unsold is tilled back into the farmland, composted, or sent to a landfill. This nonprofit organization reported the limitations of its estimate of FLW across the food supply chain in the United States. For example, the nonprofit organization documented in its FLW estimate methodology that its farm-production FLW data analysis focused on estimating imperfect-produce rates, but noted that FLW may occur at this stage for a variety of reasons, including inclement weather, pests, or overproduction. It also documented that future research efforts could assess actual produce imperfection and loss rates for each produce type using geographical differences to improve estimate accuracy. Representatives from another nonprofit organization that has published an estimate of FLW told us there are data gaps about FLW along the food supply chain. For example, this nonprofit organization reported in 2017 that improved research is needed regarding farm- production data and FLW estimates of the consumer stage of the food supply stage. In addition, this nonprofit organization reported that one challenge is the absence of standardized measurement methodologies and common metrics to help entities representing all food supply-chain stages accurately estimate FLW, develop strategies to reduce FLW, and measure progress. In this report, they noted that federal agencies’ efforts to develop a mechanism to aggregate and disseminate FLW information as it is gathered by businesses and institutions, among others, would be beneficial to all stakeholders. Representatives of a third nonprofit organization stated that FLW measurement methodologies need to be tailored to the particular stages of the food supply chain and that the strategies to reduce FLW need to respond to the conditions associated with specific foods. Nonfederal stakeholders identified a lack of education and awareness about FLW as a challenge to reducing FLW. For example, an official from one state told us that there is a lack of awareness among various organizations about the benefits of preventing FLW. Specifically: One state official told us that there is a lack of awareness among food producers, businesses, and consumers about the benefits of preventing FLW. According to this official, to address this challenge the state developed a strategic action plan that prioritizes focusing upstream in the food supply chain to prevent FLW, as opposed to the more traditional focus on increasing FLW diversion, such as through composting. This official also told us that implementing organic waste (e.g., food waste or other plant and animal materials) bans, which prohibit specified waste generators from sending food waste to landfills, as several states are doing to reduce FLW, tends to promote FLW reduction activities further down on EPA’s Food Recovery Hierarchy. As a result, organic waste bans may contribute relatively little to reducing FLW or maximizing the benefits of such reductions. This official emphasized that additional steps are needed to increase awareness about the benefits of prioritizing prevention through shifts in supply chains, purchasing, and consumption patterns to reduce FLW. Officials from two states told us there is a lack of resources to support efforts to educate consumers about FLW. For example, one state official told us that the state agency has insufficient staff resources to do effective outreach regarding FLW, and current staff members do not yet have the expertise to fully educate and assist consumers and businesses about all options available to reduce FLW. Another state official told us that the state would like to do a state-wide social marketing campaign to disseminate education and information about FLW to household consumers, but the state lacks sufficient resources to launch such an effort. Representatives of a nonprofit food donation organization identified a lack of education and awareness about date labeling as one of the challenges to reducing FLW. For example, the representatives told us that consumer confusion about date labels may be an impediment to reducing FLW among consumers. However, in these representatives’ view, reducing date labeling confusion is unlikely to lead to additional food donations. In addition, an academic institution representative, in collaboration with other authors, reported that a driver of household FLW is consumer confusion over date labels and conducted a survey to gain information about consumer perceptions of date labels. They concluded from their research that increasing consumer education on the meaning of date labels can help to reduce FLW. A representative of a nonprofit food donation organization told us that education and awareness about liability protections and compliance are lacking for various potential food donors and may hinder some food producers from donating food and, by extension, reducing FLW. Through interviews with nonfederal stakeholders, we identified that limited infrastructure and capacity is a challenge that can hamper efforts to reduce FLW. For example: Representatives of a nonprofit food donation organization that receives food donations cited a lack of sufficient capacity and logistical support to collect and distribute available food. For example, representatives told us that food pantries may not have a sufficient volunteer workforce or enough food storage capacity to be able to distribute all donated food to needy people. Food industry representatives told us that businesses have infrastructure limitations, such as a lack of transportation options to deliver excess food to food pantries or composting facilities. For example, representatives told us that if such facilities were available, food scraps, such as produce peels, could be used as animal feed or composted. However, if the infrastructure to utilize these options is not available, the companies generating the FLW may opt to send it to landfills instead. An official from one state told us that the state does not have access to the infrastructure and capacity needed to separate contaminants in order to be able to divert FLW for other uses, such as animal feed, composting, or anaerobic digestion. For example, this state official told us that the state does not have access to the necessary equipment to separate plastic and other packaging materials from food waste in order to be able to process FLW through anaerobic digesters. Officials of another state provided a study stating that removing packaging from food waste can be an obstacle to successful FLW diversion and that separation of food waste for composting or other diversion can be costly. In addition, a representative of one international organization told us that federal agencies could facilitate a collaborative approach with industry stakeholders to develop voluntary industry standards on food packaging materials and food portion sizes to help reduce FLW in the United States. Officials from another state told us that a lack of food recycling infrastructure limits their ability to enforce the state’s organic waste ban and reduce FLW. A state official told us that the state has one anaerobic digester facility to process food waste, but additional recycling infrastructure would be needed statewide to enable food waste generators, such as hospitals or schools, to recycle their food waste instead of sending it to landfills. Since announcing the national FLW reduction goal in 2015, EPA and USDA have taken initial actions to address challenges in the three key areas that nonfederal stakeholders identified to reduce FLW. For example, EPA and USDA have taken actions to provide improved data and information about FLW in the United States; educate and increase awareness of FLW along the food supply chain; and expand the infrastructure and capacity to support efforts to reduce FLW. EPA and USDA have provided some data and information about FLW in the United States. Specifically: In a 2018 report, EPA published trends of food waste materials generation, among other materials, and provided updated information about municipal solid waste being generated, recycled or composted, landfilled, and combusted with energy recovery using 2015 data from residential, commercial, and institutional sources. According to EPA, food waste represents the largest percentage of landfilled material in municipal solid waste, as seen in figure 3 below. EPA relies on gathering these data on food waste generation and management from studies conducted by other organizations, such as state and local governments and food waste generators. EPA measures certain FLW diversion activities (i.e., divert food to a destination other than landfill or incineration). For example, in September 2018 EPA completed an effort to quantify the number and capacity of anaerobic digestion facilities in the United States. EPA also aggregates and publishes data submitted by EPA’s Food Recovery Challenge program participants on recycling fats, oils, and grease, which may otherwise be disposed of through wastewater. For example, participating restaurants may submit data on the amount of fats from their fryer grease containers that they send for recycling through rendering, conversion to biofuels, or to an anaerobic digester. In addition, EPA develops estimates of food waste composting based on a review of state environmental agency websites, as well as published reports. EPA updates its FLW estimates annually. EPA officials stated that these annual estimates are the most comprehensive annual estimates of generated and managed FLW and that EPA plans to use these estimates to track progress. However, EPA officials acknowledged certain limitations in using these estimates to track annual progress against the 2030 goal. For example, EPA officials stated that data challenges include limited studies available for some sectors and the lack of geographic coverage, among others. EPA is taking steps to improve its FLW estimates. For example, officials stated that in 2017 EPA embarked on an effort to improve its food measurement methodology to reflect all potential FLW generating sectors for which there are data, and to characterize how food is being managed beyond composting and landfill. USDA has also provided some data and information about FLW at various stages of the food supply chain in the United States since 2015. Specifically: ERS is working on initiatives to refine and improve its data system in order to support its ongoing efforts to estimate FLW at the retail and consumer stages of the food supply chain. For example, USDA officials told us they are developing a proposal for an external expert panel to analyze food loss estimates at the consumer stage of the food supply chain and make recommendations for data updates. In addition, USDA officials told us that work is under way to update the retail-level loss estimates of selected foods. In addition, in December 2017, ERS initiated work on a study to identify gaps in information about farm-level FLW. According to ERS officials, as part of the study, ERS will describe the existing data- collection challenges and address the economic factors that influence farmers’ decisions as they relate to FLW at the farm level. For example, one factor could involve a farmer deciding to plow excess produce into the fields instead of harvesting or processing the crop if the potential additional labor or operations costs exceed the potential revenue. One senior ERS official told us that ERS expects to issue the study by the end of calendar year 2019. Additionally, ERS officials told us that USDA could use the final study to inform USDA’s policy approaches to reducing FLW. For example, the report may inform USDA’s efforts to assist farmers in implementing best practices in reducing FLW and expanding market opportunities for imperfect fruits and vegetables or excess harvest. USDA’s National Institute for Food and Agriculture has provided grant funding to projects related to FLW. For example, the institute awarded a grant in 2018 to an academic institution to study the effect of secondary markets as alternative channels for usable food. To advance the research mission of the agency, among other reasons, USDA has a memorandum of understanding with the Foundation for Food and Agriculture Research, an organization that Congress authorized as part of the 2014 Farm Bill. The Foundation for Food and Agriculture Research conducts research in six defined challenge areas, including one area that focuses research on inefficiencies in the food system, such as FLW. EPA and USDA have taken some actions to educate and build awareness about FLW in the United States since announcing the national FLW reduction goal in 2015. For example, EPA published its Sustainable Materials Management Program Strategic Plan, Fiscal Years 2017-2022 in October 2015. One of the plan’s three strategic priority areas is Sustainable Food Management, which includes an action area of promoting opportunities to reduce wasted food and the food’s associated effects over the entire food supply-chain life cycle with a preference for using approaches that are higher on the agency’s Food Recovery Hierarchy. EPA’s strategic plan describes delivering tools and education; working with states and local communities to help provide regional or sector-based support; and sharing best practices on wasted-food reduction efforts. In addition to the planned actions identified in the Sustainable Food Management area, EPA has also provided the following FLW education and awareness tools, among others: Food: Too Good to Waste. This community-based social marketing campaign, implementation guide, and toolkit aim to reduce wasteful household food management practices and keep FLW out of landfills. The toolkit is designed for community organizations, local governments, households, and others interested in reducing wasteful household food management practices. The implementation guide is designed to teach local governments and community organizations how to implement a Food Too Good to Waste campaign in their community using the toolkit. In a 2016 report, EPA listed 17 communities in various states, including Rhode Island and Vermont, that had implemented Food Too Good to Waste campaigns and, as part of this implementation, could use outreach and engagement tools adaptable to the needs of their communities based on their available resources. The campaigns focused on helping households make small shifts in how they shop, prepare, and store food to prevent it from being wasted. Waste Reduction Model. According to the agency’s website, EPA created this tool to help solid-waste planners and organizations track greenhouse gas emissions reductions from several different waste- management practices, including source reduction, recycling, anaerobic digestion, combustion, composting, and landfilling. For example, a food service establishment can use the tool to create an estimate of the greenhouse gas savings associated with decreasing the amount of bread and produce landfilled. Tip sheets. EPA developed tip sheets about reducing FLW for different sectors involved in the food supply chain, including manufacturers and restaurants, to emphasize FLW prevention options. EPA officials told us that they make these tip sheets available online on the agency’s website and attend conferences to disseminate information. For example, the officials said that they attended the Midwest Food Recovery Summit in September 2018 and provided these tip sheets at the EPA information booth during the conference. In addition, USDA has been involved in the following FLW reduction efforts to raise awareness and educate various stakeholders along the food supply chain: FLW roundtable meeting. In May 2018, the Secretary of Agriculture hosted a roundtable meeting with members of Congress, food industry representatives, and nonprofit groups to raise awareness about FLW and discuss potential solutions. FoodKeeper application. In 2015, USDA, in partnership with Cornell University and the Food Marketing Institute, launched the FoodKeeper application, a tool to provide consumers with specific storage advice, including storage timelines for the refrigerator and freezer for food and beverage items. USDA officials stated that the agency updated the application in October 2018 to include various features including searching for food and beverages in Spanish and increasing the number of food items with storage information. USDA has continued to highlight the FoodKeeper application as part of USDA and EPA’s Food Waste Challenge effort to help educate consumers to reduce FLW. Infographic. Also in 2015, the USDA Center for Nutrition Policy and Promotion issued an infographic, “Let’s Talk Trash,” to help inform American consumers about the benefits of reducing FLW, as shown in figure 4. USDA made the infographic available on its www.choosemyplate.gov website, which includes additional resources to help consumers think about the amount of FLW at home. Strategies for schools. In 2015, USDA’s Food and Nutrition Service issued a summary of strategies for schools to reduce FLW that included a list of resources to encourage FLW diversion by donating uneaten food to nonprofit institutions and information about composting. The Food and Nutrition Service also recommended that schools introduce “share tables” into cafeterias so that students could exchange unwanted but otherwise edible food items. In June 2016, USDA issued a memorandum to remind states’ Child Nutrition Program directors of the opportunities to use share tables to reduce FLW in a number of Child Nutrition Programs, such as the National School Lunch Program. In July 2016, the Food and Nutrition Service issued guidance directed at school staff members and students, among others, with tips to prevent FLW, including encouraging students to use share tables. To further provide information, raise awareness, and educate different stakeholders along the food supply chain, EPA and USDA have collaborated on the following FLW reduction efforts: A Guide to Conducting Student Food Waste Audits. In 2017, EPA, USDA, and the University of Arkansas collaborated to create this guide for students and school personnel about the amount of FLW in their cafeterias. The guide provides information on why and how to do a food waste audit and what to do with the data collected. It also offers FLW prevention ideas. Public/private partnerships. EPA and USDA support public/private partnerships to provide key information, solutions, and best practices to reduce FLW across the food supply chain. For example, EPA and USDA established the U.S. Food Loss and Waste 2030 Champions initiative in November 2016 as a way to increase efforts to meet the national FLW reduction goal. This 2030 Champions initiative recognizes organizations that have committed to cutting FLW in their own operations in half by 2030 and encourages Champions to report on their progress. In May 2018, EPA hosted a public webinar to highlight the actions of three 2030 Champions to share best practices, tools, and resources these organizations created to prevent food from going to waste. In March 2019, USDA officials told us that eight additional businesses have joined the 15 Champions involved in the initiative since its launch. In addition, EPA and USDA also support Further With Food, an online hub developed by EPA, USDA, and 10 other organizations that provides information and solutions to raise public awareness and reduce FLW. Participation in external conferences. EPA and USDA have conducted outreach, including through participation in conferences and seminars, and have disseminated resources related to FLW. For example, EPA and USDA each sent an official to attend and present at the National Academies of Science’s Reducing Food Loss and Waste: A Workshop on Impacts in October 2018. USDA officials told us they helped fund this workshop and helped develop the workshop’s objectives, which were to explore the effects of reducing FLW on food availability and other factors; to examine the role of governments, nongovernmental organizations, and the private sector in adopting best practices to improve the benefits and reduce the costs of reducing FLW; and to discuss opportunities for partnerships to address FLW. USDA has also collaborated with FDA to address FLW. For example, USDA and FDA are both on the Executive Board of the Conference for Food Protection, an organization that brings together representatives from the food industry, government, academia, and consumer organizations to identify and address emerging problems of food safety. In April 2016, this group released a Comprehensive Resource for Food Recovery Programs to reduce FLW through the recovery of consumable food. This report is intended to assist stakeholders involved in the recovery, distribution, or service of food to people who are food insecure. The report references the national food standards at the retail level, as expressed in the FDA Food Code, to minimize the occurrence of risk factors that contribute to foodborne illness. FDA contributed to the submission of an issue to the 2018 Biennial Meeting of the Conference for Food Protection that sought to promote uniformity in the way in which state and local governments regulate food donation and recovery operations in retail and foodservice establishments. In addition, FDA has disseminated information to the public about strategies to reduce FLW while maintaining food safety and has referred to USDA’s FoodKeeper application as a resource for learning how to store perishable food and employ safe storage practices. EPA and USDA have each taken some actions to increase infrastructure and capacity to support efforts to reduce FLW in the United States. EPA has taken some actions to increase infrastructure and capacity to reduce FLW in the United States. For example: Technical assistance. EPA provides technical assistance to state and local governments in developing anaerobic digestion projects, a technology to process wasted food that is more desirable than landfilling or incineration, according to EPA’s Food Recovery Hierarchy. Excess Food Opportunities Map. EPA’s Excess Food Opportunities Map displays the locations of more than 500,000 industrial, commercial, and institutional food generators that may potentially produce excess food and more than 4,000 potential recipients of that excess food. The map also provides information at the specific establishment–level, including estimates of excess food generation that may help users identify alternatives to sending excess food to landfills. The map helps users identify potential infrastructure gaps for managing excess food, inform FLW management decisions at the local level, and identify potential sources of food for rescue and reuse, among other purposes. An EPA official told us that the communication plan for the launch of the Excess Food Opportunities Map included a webinar announcing the map in July 2018 and providing presentations about the map at various conferences, including during the National Academies Reducing Food Loss and Waste Workshop in October 2018. The official also stated that emails about the map were sent to over 13,000 people and approximately 700 people attended the webinar EPA hosted in July 2018. Recycling infrastructure. EPA’s Sustainable Materials Management program’s strategic plan describes EPA’s role in providing states, businesses, and other stakeholders with, among other things, tools, guidelines, and technical support to more effectively manage waste, including by helping increase recycling infrastructure. In May 2018, EPA cohosted a recycling infrastructure workshop to identify solutions for creating infrastructure for anaerobic digestion and composting. In addition, EPA officials told us that the agency is in the process of updating its recycling guide for state and local governments and they anticipate completing it by the end of 2020. USDA also has taken some actions to increase infrastructure and capacity to reduce FLW in the United States. For example: Food programs. USDA officials told us that USDA food programs, such as The Emergency Food Assistance Program, support efforts to feed people and to provide access to affordable and nutritious food. For example, food donation organizations that are recipients of program funds may use these funds to pay the direct expenses associated with the distribution of USDA foods, such as fruits, vegetables, and beans. New FLW-reduction technologies. USDA’s Agricultural Research Service has various research programs, including one to enhance the quality and utilization of agricultural products. Potential benefits listed as part of this research program are minimizing food product losses and reducing FLW through the development of farm production technologies, such as the development of an apple-sorting system that will help reduce apple harvest losses. According to USDA officials, most of the innovations of this research program involve creating value-added products from “ugly produce” or from food processing byproducts, such as orange peel or mushroom-stalk waste, or creating new technologies to prolong the shelf life of food products. Meat and poultry donation rules. USDA’s Food Safety and Inspection Service issued a directive that outlines procedures for donating certain meat and poultry products to nonprofit organizations. The Food Safety and Inspection Service has also begun, under certain circumstances, to recognize food banks as “retail-type” establishments, which allows food banks to break down bulk shipments of federally inspected meat or poultry products, wrap or rewrap those products, and label the products for distribution to consumers. In one case, this recognition enabled a nonprofit organization engaged in food donations to gain 2.6 million pounds of food donations from manufacturers in 2016, according to USDA documents. Grant funding. USDA’s Rural Utilities Service has provided some funding to support FLW reduction infrastructure in rural communities. For example, USDA awarded a 2016 USDA Rural Utilities Service Solid Waste Management grant to the University of Iowa’s Waste Reduction Center, which has worked toward addressing the issue of FLW disposal. More recently, in 2018, USDA awarded a solid-waste management grant to the Center for EcoTechnology, a nonprofit that provides technical assistance to implement FLW diversion programs. Low-interest loans. USDA’s Farm Storage Facility Loan Program provides low-interest loans for producers to store, handle, and transport the food they produce. The loans are designed to assist a diverse range of farming operations, including small and midsized businesses and operations supplying local food and farmers markets. The program helps keep food from being damaged by pests or inclement weather, among other things, so that more food can reach store shelves. Funding for renewable energy systems. USDA’s Rural Energy for America Program provides grants and loan guarantees to farmers, ranchers, and eligible small businesses to install renewable energy and energy-efficiency systems. For example, according to a Rural Energy for America Program Fact Sheet, funds may be used for the purchase, installation, and construction of renewable energy systems, such as anaerobic digesters. In a 2016 USDA Rural Development report, USDA provided examples of anaerobic digesters that use FLW to produce a biogas that is converted into energy. EPA and USDA have each taken some actions to plan and organize their efforts toward achieving the national FLW reduction goal, such as issuing strategic plans and establishing working groups. Additionally, EPA, USDA, and FDA signed a joint agency formal agreement in October 2018 aimed at increasing collaboration and coordination among the agencies on FLW reduction efforts. EPA, USDA, and FDA only recently initiated their interagency collaboration on FLW reduction efforts toward achieving the national FLW reduction goal, but have not yet taken certain steps that align with key practices for interagency collaboration. EPA has taken actions to guide its own efforts toward achieving the national FLW reduction goal. For example, in 2015, EPA issued a strategic plan that included a strategic priority area of sustainable food management. Subsequently, EPA developed an internal planning document (U.S. EPA Sustainable Management of Food Strategy, Fiscal Year 2018-2022). This planning document established action areas, goals, and activities for reducing FLW to achieve the national FLW reduction goal. For example, the plan identified five action areas, including addressing data and measurement issues, collaboration and partnerships, technical assistance, infrastructure and capacity, and communication and outreach. According to EPA officials, the agency intends to use the plan to track its progress and measure results towards the national FLW reduction goal. USDA has also taken actions to guide its own efforts toward achieving the national FLW reduction goal. For example, according to USDA officials, the department established a FLW working group in 2015 that currently meets on a monthly basis. According to officials from the Office of the Chief Economist and ERS, the department also designated an individual within the Office of the Chief Economist to guide USDA’s FLW efforts. In addition, in March 2016, the National Institute for Food and Agriculture’s Pilot Science Outcome Committee on Environmental Sustainability identified FLW as a top science priority area to address environmental sustainability. According to the committee, FLW is an integral component of environmental sustainability, and mitigating FLW has the potential to create economic, environmental, and social benefits while contributing to food security, resource conservation, and the mitigation of climate change. Furthermore, EPA and USDA have contributed to the work of the Commission for Environmental Cooperation, an intergovernmental organization established by the governments of Canada, Mexico, and the United States to facilitate effective cooperation on the conservation, protection, and enhancement of the environment in their territories. The organization has an initiative to identify challenges, opportunities, and solutions related to increasing organic waste diversion and processing capacity in North America. This organization issued a report in 2017 about, among other things, the management of organic waste and best practices for reducing FLW and diverting other organic waste materials away from landfills. EPA is on the steering committee for this effort. According to an EPA announcement in March 2019, the commission issued a practical guide and technical report on FLW measurement. Moreover, in October 2018, the Secretary of Agriculture hosted a public meeting to promote FLW reduction. During this meeting, EPA, USDA, and FDA signed a formal interagency agreement referred to by the agencies as the Winning on Reducing Food Waste initiative. Under this 2-year agreement, the agencies committed to developing an interagency strategic plan to increase collaboration and coordination among the agencies on their FLW reduction efforts. According to the agreement, this additional collaboration is intended to strategically align each agency’s efforts to better educate Americans on the impacts of reducing FLW. The agencies also agreed to, where appropriate, educate actors throughout the supply chain on the best practices to reduce FLW in the growing, manufacturing, transporting, selling, and disposing of food and the handling, preparation, and storage of food, as well as creating new uses for excess food. The formal agreement mentions public-private partnerships and, according to EPA officials, the agencies intend to use the views of stakeholders in the public, private, and nonprofit sectors to inform their strategic plan. According to EPA officials, the agencies intend to discuss common goals and to identify additional initiatives as appropriate to achieve the national FLW reduction goal. In announcing this initiative, the Secretary of Agriculture affirmed the importance of reducing FLW by saying that “an unacceptable percentage of our food supply is lost or wasted” and that “as the world’s population continues to grow and the food systems continue to evolve, now is the time for action to educate consumers and businesses alike on the need for food waste reduction.” In addition, the FDA Commissioner stated that “by taking steps to address obstacles that food donation and recovery programs may face in giving unsold foods a second opportunity and helping food producers find ways to recondition their products so that they can be safely sold or donated, our aim is to both reduce food waste and nourish Americans in need.” In April 2019, the agencies held a public event to announce their Winning on Reducing Food Waste Federal Interagency Strategy. This strategic plan identified six prioritized action areas for activities to reduce FLW. For example, the agencies plan to, among other things, increase consumer education and outreach efforts; increase coordination and guidance on FLW measurement; and clarify and communicate information on food safety, food date labels, and food donations. In addition, the agencies signed a formal agreement with ReFED to, among other things, better evaluate and improve upon strategies to reduce FLW. For example, according to the 2019 agreement, the agencies and ReFED intend to leverage existing partnerships to advance data-collection and measurement activities related to FLW. Finally, EPA announced that it had selected three recipients to receive EPA funding to support infrastructure projects to help reduce FLW and divert FLW from landfills. In our prior work, we have found that key practices to enhance and sustain interagency collaboration include agreeing on roles and responsibilities and developing mechanisms to monitor, evaluate, and report on results. In addition, we have found that key practices for agency collaboration call for clearly defining short- and long-term outcomes. Furthermore, such interagency efforts benefit from identifying how leadership commitment will be sustained. Lastly, we identified a key practice that calls for ensuring that the relevant stakeholders have been included in the collaborative effort. This collaboration can include other federal agencies, state and local entities, and private and nonprofit organizations. According to the strategic plan, the agencies built on information from several sources, including prior GAO work on implementing interagency collaborative mechanisms, to develop the Winning on Reducing Food Waste Federal Interagency Strategy. However, this strategic plan does not align with certain key practices for interagency collaboration. For example, the first priority area identified in the strategic plan is to enhance interagency collaboration, and the strategic plan states that an interagency, collaborative mechanism will be established to reduce programmatic redundancies and leverage complementary activities. However, the strategic plan does not identify how this mechanism will be used to monitor, evaluate, or report on results, establish a time frame for developing this collaborative mechanism, or describe how the agencies will engage relevant stakeholders, such as other federal, state, and local agencies, nonprofit organizations, academic institutions, food industry entities, international organizations, and tribal organizations. In addition, several of the strategic plan’s priority areas address specific aspects of reducing FLW, such as encouraging FLW reduction by federal agencies in their respective facilities. However, the strategic plan does not identify the roles and responsibilities of the respective agencies for taking action in these areas and it does not clearly define what specific short- and long- term outcomes the agencies intend to achieve. Furthermore, the agencies have not identified how they intend to sustain leadership commitment to this goal. For example, the Winning on Reducing Food Waste formal interagency collaborative agreement is a 2-year agreement among the agencies, but the national FLW reduction goal calls for reducing FLW by half by 2030, which falls well beyond this 2-year time frame. According to a USDA official, the agencies do not have plans for how they will continue their interagency collaboration beyond the life of the current agreement. This official noted that the agencies do not intend to update the strategic plan for the duration of the 2-year agreement and that the agencies will release more information to the public about specific actions and timelines as it becomes available. By incorporating leading practices for interagency collaboration as they implement their interagency strategic plan, EPA, USDA, and FDA would have better assurance that they are effectively collaborating toward achieving the national FLW reduction goal. Achieving the national FLW reduction goal could provide significant economic, environmental, and social benefits to the United States, such as helping to lower consumer expenses, reducing harmful greenhouse gas emissions, and providing additional meals to feed food-insecure people through increased food donations. This is an important issue that requires action across the food supply chain and collaboration among federal agencies and nonfederal stakeholders, such as states and businesses. EPA and USDA have taken steps to develop programs and policies that aim to reduce FLW and to collaborate on their various initiatives. In addition, EPA, USDA, and FDA have taken some actions to plan and organize their efforts toward achieving the national goal of reducing FLW by half by 2030, including announcing an interagency strategic plan to reduce FLW. However, this strategic plan does not align with key practices in interagency collaboration that we have identified, such as agreeing on roles and responsibilities; developing mechanisms to monitor, evaluate, and report on results; clearly defining short- and long- term outcomes; identifying how leadership commitment will be sustained; and ensuring that the relevant stakeholders have been included in the collaborative effort. By incorporating such leading practices for interagency collaboration as they implement their interagency strategic plan, EPA, USDA, and FDA would have better assurance that they are effectively collaborating toward achieving the national FLW reduction goal. We are making three recommendations to the agencies in our review. Specifically: The Administrator of EPA should work with the Commissioner of FDA and Secretary of Agriculture to incorporate leading collaboration practices as they implement their interagency FLW reduction strategic plan, to include (1) agreeing on roles and responsibilities; (2) developing mechanisms to monitor, evaluate, and report on results; (3) clearly defining short- and long-term outcomes; (4) identifying how leadership commitment will be sustained; and (5) ensuring that the relevant stakeholders have been included in the collaborative effort. (Recommendation 1) The Commissioner of FDA should work with the Administrator of EPA and Secretary of Agriculture to incorporate leading collaboration practices as they implement their interagency FLW reduction strategic plan, to include (1) agreeing on roles and responsibilities; (2) developing mechanisms to monitor, evaluate, and report on results; (3) clearly defining short- and long-term outcomes; (4) identifying how leadership commitment will be sustained; and (5) ensuring that the relevant stakeholders have been included in the collaborative effort. (Recommendation 2) The Secretary of Agriculture should work with Administrator of EPA and Commissioner of FDA to incorporate leading collaboration practices as they implement their interagency FLW reduction strategic plan, to include (1) agreeing on roles and responsibilities; (2) developing mechanisms to monitor, evaluate, and report on results; (3) clearly defining short- and long-term outcomes; (4) identifying how leadership commitment will be sustained; and (5) ensuring that the relevant stakeholders have been included in the collaborative effort. (Recommendation 3) We provided a draft of this report to EPA, USDA, and the Department of Health and Human Services for review and comment. We also provided CEQ and OMB a draft of this report for review. In its comments, reproduced in appendix I, EPA agreed with our recommendation to the agency and described current and future actions to implement the recommendation. Similarly, in its comments, reproduced in appendix II, USDA agreed with our recommendation to it and described current and future actions to implement the recommendation. In addition, in its comments, reproduced in appendix III, the Department of Health and Human Services concurred with our recommendation to it and described current and future actions to implement the recommendation. USDA and CEQ provided technical comments, which we incorporated as appropriate. In response to our recommendations, EPA, USDA, and the Department of Health and Human Services said that they will work with each other to incorporate leading collaboration practices as they implement the interagency FLW reduction strategic plan. Both EPA and USDA also stated that they intend to complete implementation of their respective recommendations by October 2020, to align with the duration of the 2- year formal agreement between EPA, USDA, and FDA. The Department of Health and Human Services stated that FDA issued a letter to the food industry supporting the industry’s efforts to standardize voluntary quality date labeling. We are sending copies of this report to the appropriate congressional committees, the Administrator of EPA, the Secretary of Agriculture, the Secretary of Health and Human Services, the Director of OMB, the Chair of CEQ, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions concerning this report, please contact me at (202) 512-3841 or morriss@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. In addition to the contact named above, Anne K. Johnson (Assistant Director), Joseph Capuano (Analyst in Charge), David Bennett, Carol Bray, Tara Congdon, Juan Garay, Serena Lo, Greg Marchand, Jordan Mettica, Oliver Richard, Dan Royer, Marie Suding, Kiki Theodoropoulos, and Sarah Veale made key contributions to this report.", "summary": "The Natural Resources Defense Council reported that in the United States up to 40 percent of the food supply goes uneaten. FLW has significant economic, environmental, and social effects on various stakeholders, including businesses and consumers. In 2015, EPA and USDA announced a national goal to reduce FLW in the United States by half by 2030. In 2018, FDA joined EPA and USDA in these efforts. GAO was asked to examine efforts by federal agencies to reduce FLW. This report (1) describes nonfederal stakeholder views on key challenges to reducing FLW in the United States, (2) describes actions EPA and USDA have taken to address key challenges to reducing FLW in the United States, and (3) examines federal planning efforts toward achieving the national FLW reduction goal. GAO reviewed federal reports on FLW; analyzed agency documents; interviewed officials from EPA, FDA, USDA, and states and representatives of nonfederal stakeholders, such as academic institutions, industry, international organizations, nonprofit organizations, and a tribal organization, based on their demonstrated expertise on FLW; and attended conferences on FLW. GAO identified three key areas in which challenges exist to reducing food loss and waste (FLW) in the United States: (1) limited data and information about FLW; (2) a lack of awareness and education about FLW; and (3) limited infrastructure and capacity. For example, the causes of FLW vary across the stages of the food supply chain (see figure), but the share of total FLW due to each of these causes is currently unknown, according to a U.S. Department of Agriculture (USDA) report. GAO identified these challenges through interviews with selected stakeholders. The Environmental Protection Agency (EPA) and USDA have taken initial actions to address key challenges to reducing FLW in the United States since announcing a national FLW reduction goal in 2015. These actions include conducting a study to identify gaps in information about farm-level FLW and building public awareness about ways to reduce FLW. EPA, USDA, and the U.S. Department of Health and Human Services' Food and Drug Administration (FDA) have taken some actions to plan and organize their efforts toward achieving the national FLW reduction goal. For example, EPA developed an internal plan that established action areas, goals, and activities for reducing FLW, and USDA designated an individual to guide USDA's FLW efforts. In October 2018, EPA, USDA, and FDA signed an interagency agreement committing them to developing a strategic plan to improve their collaboration and coordination in reducing FLW. In April 2019, the agencies announced an interagency strategic plan with prioritized action areas to reduce FLW, but this strategic plan does not address how it will incorporate key practices for interagency collaboration that GAO identified, including (1) agreeing on roles and responsibilities; (2) developing mechanisms to monitor, evaluate, and report on results; (3) clearly defining short- and long-term outcomes; (4) identifying how leadership commitment will be sustained; and (5) ensuring that the relevant stakeholders have been included in the collaborative effort. By incorporating such practices as they implement their interagency strategic plan, EPA, USDA, and FDA would have better assurance that they were effectively collaborating toward achieving the national FLW reduction goal. GAO is making three recommendations in this report. GAO is recommending that EPA, FDA, and USDA incorporate leading collaboration practices as they implement their interagency strategic plan to reduce FLW.", "document_type": "gao"}
{"report": "VA has been providing veterans educational assistance benefits since 1944. We previously reported that these benefits have been put in place over time to compensate for compulsory service, encourage voluntary service, avoid unemployment, provide equitable benefits to all who served, and promote military retention. The Post-9/11 GI Bill, which took effect on August 1, 2009, is now VA’s largest educational program. This program generally provides benefits to veterans who served on active duty for at least 90 days beginning on or after September 11, 2001. Full benefits are generally available to those who served on active duty for 36 months, for which VA will pay the net cost for in-state tuition and fees at public schools and up to an annual maximum amount at nonprofit and for- profit schools ($24,477 in academic year 2019-2020). VA pays schools directly for tuition and fees and sends additional payments for housing and books directly to veterans who are eligible for these payments. To receive education benefits through the Post-9/11 GI Bill, students submit applications to VA, schools certify enrollments, and VA processes claims and payments. For help covering the costs of their postsecondary education, veterans may also be eligible for grants and loans available from federal student aid programs administered by Education, such as Pell Grants and Direct Loans. According to Education data, an estimated 32 percent of student veterans had received Pell Grants and 28 percent had taken out Direct Loans, during school year 2015-16. VA education payments, such as Post-9/11 GI Bill benefits, are not considered when calculating eligibility for federal student aid and do not affect the amount of aid a veteran can receive from Education. Student veterans may also be eligible for state and institutional aid (scholarships from state governments or schools, for example). Nearly 700,000 student veterans received Post-9/11 GI Bill tuition and fee benefits to attend almost 6,000 schools in fiscal year 2017. VA paid about 40 percent of the Post-9/11 GI Bill tuition and fee payments to public schools, 30 percent to nonprofits, and 30 percent to for-profits (see fig. 1). Most student veterans used Post-9/11 GI Bill tuition and fee payments to attend schools that provided 4-year undergraduate programs (see fig. 2). Veterans may also use Post-9/11 GI Bill benefits for training opportunities at schools that do not offer college degrees, including training in areas such as driving, emergency medical training, and barber or beautician skills. These programs received about $360 million Post-9/11 GI bill tuition and fee payments in fiscal year 2017. A relatively small number of schools received a large share of Post-9/11 GI Bill tuition and fee payments. In fiscal year 2017, the 50 schools that received the highest total amount of Post-9/11 GI Bill tuition and fee payments accounted for over 30 percent of all such benefits, collectively receiving $1.4 billion for over 190,000 beneficiaries. These 50 schools consisted of 14 public, 16 nonprofit, and 20 for-profit schools (see fig. 3). In fiscal year 2017, the 50 schools received between $11 million and $191 million each in tuition and fee payments and enrolled between around 350 and 28,000 Post-9/11 GI Bill beneficiaries. In contrast, among all schools receiving Post-9/11 GI Bill benefits in fiscal year 2017, the majority of them enrolled fewer than 15 veterans. Student outcomes at the 50 schools that received the most Post-9/11 GI Bill tuition and fee payments were, on average, generally comparable to the national average, but varied more widely across sectors. Since available data on student veteran outcomes is currently limited, we analyzed common outcome measures for the broader student populations at each school: 4-year program graduation rates: the percent of first-time full-time students who completed a 4-year program within 6 years. Full- and part-time retention rates: the percent of first-time students who enrolled in one fall and either successfully completed their program or re-enrolled in the next fall. When examined as a whole, the average student outcomes for the 50 schools that received the most Post-9/11 GI Bill tuition and fee payments were generally comparable to the national average. For example, the average 4-year program graduation rate at the top 50 schools was 61— the same as the national average. For one of the outcome measures— full-time retention rate—the average was higher for the top 50 schools (83 percent) than the national average (75 percent). Within the 50 schools that received the most Post-9/11 GI Bill tuition and fee payments, student outcomes varied across schools in different sectors (see fig. 4). For-profit schools had lower 4-year program graduation and retention rates compared to public and nonprofit schools among these 50 schools, although there was wide variation among schools in each sector. Although a relatively small number of schools close each year, these closures can affect thousands of student veterans. In 2017 we reported that about 95 schools closed in school year 2015-16, according to Education data, which was higher than in previous years, primarily due to a rise in for-profit school closures (see fig. 5). Schools can close in different manners and for a variety of reasons, including declining enrollments, financial problems, loss of accreditation, and legal actions. When a school ceases operations in an orderly process over several months it gives students time to complete the current school term and make arrangements to transfer and continue their education at another school. The effect of school closures is often worse when the closures occur abruptly with little or no advance warning, because these schools generally do not have time to establish transfer arrangements that allow students to easily continue their education at another school. Abrupt closures of large schools, although infrequent, can affect thousands of student veterans and result in large financial losses for the federal government and taxpayers. For example, Corinthian Colleges Inc. (Corinthian) enrolled more than 72,000 students before its closure in April 2015. The following year, ITT Educational Services Inc. (ITT), another large for-profit provider of higher education, closed all of its 136 campuses in September 2016, affecting more than 35,000 students. More than 7,000 Post-9/11 GI Bill students were pursuing educational programs at schools operated by ITT and Corinthian at the time of their closures, according to VA. More recently, closures at Education Corporation of America in 2018 and Dream Center Education Holdings in 2019, which operated schools under multiple brands, including Argosy University and several campuses of The Art Institutes, affected tens of thousands of students, including thousands of Post-9/11 GI Bill recipients. Student veterans attending a school that closes may be eligible to have some or all of their Post-9/11 GI Bill benefits restored. As a result of the Harry W. Colmery Veterans Educational Assistance Act of 2017, VA restores GI Bill entitlements to eligible beneficiaries affected by recent and future school closures. Student veterans may also be entitled to a discharge on eligible federal student loans they may have received from Education or to have their Pell Grant eligibility restored if they are unable to complete a program because their school closed. Despite these options for having benefits restored and loans discharged, school closures can still create hardships for veterans. As we have previously reported, college students in general can face challenges transferring credits and continuing their education at a new school under any circumstances. Students who transferred lost, on average, an estimated 43 percent of their credits, and credit loss varied depending on the transfer path, based on data from 2004 to 2009. For example, students who transferred between public schools—the majority of transfer students—lost an estimated 37 percent of their credits. In comparison, students who transferred from for-profit schools to public schools—which happens less frequently—lost an estimated 94 percent of their credits. Even if a student’s credits transfer, they may not apply toward fulfilling degree requirements for their intended major. In these cases, a student will likely have to take additional courses at their new school, which could potentially delay graduation and result in additional costs to pay for repeated courses. Further, some student veterans with credits that do not transfer may exhaust their Post-9/11 GI Bill benefits before completing their degree. School closures can also exacerbate other challenges veterans may face pursuing their education. As we have previously reported, many student veterans already cope with challenges transitioning from the military to an academic environment. For example, they can face challenges navigating the academic bureaucracy, whether in attempting to receive transfer credit for previous college courses or in determining what other sources of financial aid may be available to them. Many student veterans are also trying to balance school with family and work obligations or dealing with the effects of combat-related physical and psychological injuries. When a school closes, the burden of finding and enrolling in a new school may be especially difficult for these veterans. Closures can also pose a financial risk for the government and taxpayers to the extent that Post-9/11 GI benefits are restored and federal student loans are discharged. For example, in 2017 the Congressional Budget Office estimated that restoring Post-9/11 GI Bill benefits and other VA education benefits to student veterans who attend schools that closed will increase direct spending by $320 million over the 10 year period from 2018 to 2027. School closures can also result in hundreds of millions of dollars in financial losses for the federal government and taxpayers due to discharged federal student loans. In conclusion, the Post-9/11 GI Bill has provided valuable education benefits to millions of veterans who attend a wide range of schools. However, when schools abruptly shut their doors, it can leave student veterans—who already face unique challenges in an academic environment—without a clear path to continuing their education and can force taxpayers to cover the cost of restoring their benefits and discharged student loans. Student veterans who continue their education at another school may also find that many of the credits they earned will not ultimately help them after they transfer, delaying their degrees and resulting in additional costs. As the number of school closures has increased in recent years, the risks and challenges associated with such closures are particularly salient for student veterans, their families, and the federal government. Chairman Levin, Ranking Member Bilirakis, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Melissa Emrey-Arras, Director, Education, Workforce, and Income Security Issues at (617) 788-0534 or emreyarrasm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony include Will Colvin (Assistant Director), Brian Schwartz (Analyst-in-Charge), and Jeffrey G. Miller. In addition, key support was provided by James Bennett, Deborah Bland, Benjamin DeYoung, Alex Galuten, Theresa Lo, John Mingus, Corinna Nicolaou, and Michelle St. Pierre. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Post-9/11 GI Bill is VA's largest educational program. It provides payments for eligible veterans to cover tuition and fees, housing and other costs while they pursue a higher education. However, for some veterans this pursuit is interrupted when the school they attend unexpectedly closes. This testimony addresses (1) the distribution of Post-9/11 GI Bill tuition and fee payments among schools, (2) outcomes of students at schools that receive the most Post-9/11 GI Bill payments, and (3) how school closures can affect student veterans. To address these topics, GAO reviewed VA data on Post-9/11 GI Bill tuition and fee payments to schools for fiscal year 2017, the most recent school-level data available. GAO analyzed student outcome measures for these schools using Department of Education data reported for school year 2017-2018. GAO also reviewed its prior reports issued between 2013 and 2017 on school closures, credit transfers, and related challenges faced by student veterans. In fiscal year 2017, nearly 700,000 student veterans used their Post-9/11 GI Bill benefits from the Department of Veterans Affairs (VA) to attend programs at almost 6,000 schools. Of the almost $4.5 billion in Post-9/11 GI Bill tuition and fee payments VA made to schools in fiscal year 2017, about 40 percent went to public schools, 30 percent to nonprofits, and 30 percent to for-profits. A small number of schools received a large share of the tuition and fees paid, with 30 percent of payments totaling $1.4 billion going to 50 schools that enrolled over 190,000 veterans in fiscal year 2017. The average student outcomes at the 50 schools that received the highest total amount of Post-9/11 GI Bill tuition and fee payments in fiscal year 2017 were generally comparable to the national averages, but varied widely when examined by school sector. For example, the average 4-year program graduation rate for the top 50 schools was the same as the national average (61 percent). Within the top 50 schools, average graduation rates varied between public (73 percent), nonprofit (66 percent) and for-profit schools (22 percent). Although a relatively small number of schools close each year, these closures can affect thousands of student veterans. School closures, which have increased in recent years, are particularly harmful when they involve large schools that close abruptly with little or no advance warning. For example, more than 7,000 veterans receiving Post-9/11 GI Bill benefits were attending schools operated by Corinthian Colleges and ITT Educational Services when they abruptly closed in 2015 and 2016, respectively. Although veterans affected by school closures may qualify to have their GI Bill benefits restored, these closures can create hardships for veterans and significant costs for taxpayers. For example, veterans can face challenges transferring credits and continuing their education at a new school. This may make it more difficult for veterans to complete their degrees before exhausting their eligibility for Post-9/11 GI Bill benefits. School closures also pose a financial risk for the government and taxpayers due to the costs associated with restoring benefits.", "document_type": "gao"}
{"report": "The FSS program is directed and managed by GSA and provides federal agencies with a simplified process for obtaining commercial supplies and services at prices associated with volume buying. Schedules are catalogs of related products and services, from pre-approved vendors, with established pricing that can be used by federal agencies to obtain goods and services, ranging from office furniture to medical equipment and supplies. Since 1960, GSA has delegated authority to VA to manage health care related schedules, currently totaling nine schedules. (Throughout this report, we use the term “FSS” to refer to VA’s FSS program, unless otherwise noted.) These nine VA schedules, as shown in figure 1, are designed to provide FSS users at VA and other agencies with a menu of items—including medical equipment, supplies, and services—they can order from in a streamlined manner. Sales to VA on the pharmaceutical schedule were $33.5 billion from fiscal years 2014 through 2018. We omitted the pharmaceutical schedule from our review because it differs substantially from the other eight schedules, particularly in its use of a prime vendor. VA’s FSS program is managed by the National Acquisition Center (NAC), a VA-wide contracting organization which is also responsible for procuring items like high-tech medical equipment for medical centers. NAC is part of VA’s Office of Procurement, Acquisition and Logistics, which is overseen by VA’s Office of Acquisition, Logistics, and Construction. Within NAC, the FSS Service, which is comprised of about 80 staff, is divided into teams of contracting staff who are responsible for individual schedules. Another team, NAC’s Program Management and Resource Support, manages functions such as issuing guidance and providing training. In 2016, NAC issued a Procedural Guideline that generally sets a goal for its contracting staff to complete their review of and award decision for vendor-submitted FSS offers within 180 calendar days. Like GSA’s FSS program, users of the VA FSS program are charged a fee on the price of their FSS purchases, called the Industrial Funding Fee (IFF). VA’s FSS fee is 1 percent for services and 0.5 percent for goods. Fees generated by VA FSS fund its operations and other VA procurement operations. NAC facilitates collection of the IFF from vendors that sell products or services under VA FSS contracts, and vendors remit the IFF to VA’s Supply Fund, a self-supporting revolving fund. The Supply Fund, in turn, is used to provide funding to NAC for the operation of the FSS Service, the office that manages VA’s FSS program. VA’s schedules are used by organizations across the federal government, including the Department of Defense, Department of Health and Human Services, and the Department of Homeland Security. In this review, we focus on how the Veterans Health Administration (VHA) uses VA’s schedules. VHA, the only VA administration that uses the VA schedules, provides medical care to about 9 million veterans at 170 medical centers. These medical centers are organized into 18 Veterans Integrated Service Networks (VISN), organizations that manage medical centers and associated clinics across a given geographic area. Each VISN is served by a corresponding Network Contracting Office (NCO), which is responsible for awarding contracts for medical goods and services that the medical centers need. Two primary groups of VHA staff place FSS orders: 1. VHA contracting officers, who are authorized to enter into contracts on behalf of the government, may place orders against the schedules. They handle purchases over the micro-purchase threshold, which is generally $10,000. 2. Certain VHA medical center logistics staff are authorized to make smaller purchases at or below the micro-purchase threshold, including placing FSS orders. Many of these staff are in the medical centers’ logistics offices, which are responsible for managing the supply chain for VHA’s medical centers. Figure 2 provides an overview of VA’s procurement structure and FSS users. In addition to purchasing goods and services through FSS, VHA logistics staff at VA’s medical centers can also buy them through the MSPV-NG program. In this program, VA medical centers use contractors called medical-surgical prime vendors to obtain many of the supplies they use on a daily basis, such as bandages and scalpels. These prime vendors operate local warehouses and deliver supplies ordered by medical centers. The prices for these medical supplies are established by separate contracts or agreements that are awarded by contracting officers within VA’s Strategic Acquisition Center (SAC). The MSPV-NG program is managed by SAC and VHA. As we reported in 2018, for over a decade, each medical center used VHA’s legacy MSPV program to order medical supplies. Many of those items were purchased using VA’s FSS, which provided medical centers with a great deal of flexibility to order from a catalog containing hundreds of thousands of items. However, this flexibility prevented VHA from standardizing the items used across its medical centers and also affected its ability to leverage its buying power to achieve greater cost avoidance. In December 2016, VHA transitioned to a new iteration of this program called MSPV-NG, which has a narrower catalog of medical supplies than the legacy program, which offered hundreds of thousands of items. As of September 2019, the catalog offers about 22,000 supply items to medical centers. In June 2016, a Supreme Court decision clarified that VA must apply the Veterans First Contracting Program preference before contracting with a non-veteran-owned business, including purchases made through FSS. This program, referred to in this report as Veterans First, provides preference within VA for contracting with veteran-owned small businesses. Specifically, VA contracting officers must apply the “VA Rule of Two,” meaning they must conduct market research to determine whether there is a reasonable expectation that two or more veteran- owned small businesses will submit offers for a particular good or service at a fair and reasonable price that offers best value to the government. If two or more such businesses are found, contracting officers must set aside the procurement for the veteran-owned small businesses. VHA used NAC’s FSS to purchase billions of dollars in medical supplies and services over the past 5 years. Sales for the eight non- pharmaceutical schedules, however, have been largely flat, as compared to the rise in VHA’s total health care spending. Though we found vendor- submitted sales reports to be sufficiently reliable for describing overall trends, we found that NAC does not have controls in place to ensure that vendors are providing complete data—used to calculate fees that finance the program. In an attempt to assess data completeness, NAC recently began comparing vendor data to Federal Procurement Data System-Next Generation (FPDS-NG) data for verification. However, because agencies do not report micro-purchases made via purchase card in FPDS-NG, this approach alone is not effective. We also found that NAC does not analyze existing data on the number of veteran-owned small businesses that hold FSS contracts, the types of goods and services they offer, or which schedules have the most or least participation by these businesses. This information is important because VHA contracting officers must apply the Veterans First preference to contracts. The existence or lack of veteran-owned small businesses on FSS affects whether these staff can use FSS to fulfill their needs. Finally, we found that NAC has limited visibility into the FSS user experience. Those insights could help NAC identify potential areas for improvement. VHA obligated $291 billion from fiscal years 2014 to 2018 for health care services provided at its medical facilities—$12 billion of which was for medical supplies and services obligated using the eight non- pharmaceutical VA schedules. In contrast to VHA obligations for health care at its medical centers, which increased nearly 20 percent during this 5-year period, VA FSS purchases on these schedules were flat. See figure 3. We found that the VHA sales trends varied among our three selected schedules during this period, as shown in figure 4. Specifically, VHA sales on the FSS for Medical Equipment and Supplies, the largest of the three, were generally flat. VHA sales on the FSS for Patient Mobility Devices, which includes items such as wheelchairs, increased nearly 50 percent, while sales on the Healthcare Staffing schedule fell by more than 30 percent. NAC does not have controls in place to ensure that vendors provide complete data in their sales reports. These sales reports—required per an FSS contract clause—are NAC’s only means of tracking FSS sales and related fees that finance the FSS program. In fiscal year 2018, vendors on VA’s nine schedules remitted $82 million in IFF from customers to VA’s Supply Fund. Figure 5 provides an overview of key steps in VA’s FSS vendor sales report and IFF collection process. Like NAC, GSA also depends on vendor-reported data to track its FSS sales. However, we found that GSA takes additional steps for its FSS program to ensure the completeness of vendor-reported data. GSA has internal controls to ensure data completeness, including a staff of 43 Industrial Operations Analysts who implement GSA procedures to ensure that, among other things, vendors have sound sales data reporting processes. We did not evaluate GSA’s use of these analysts, but, according to GSA FSS officials, these analysts review vendor sales data, educate vendors about GSA’s requirements, and conduct checks on vendor internal controls and compliance with GSA policies. Having internal controls in place is essential to ensure completeness of vendor-reported sales data. In February 2019, NAC officials told us they had tried to use obligation data reported in FPDS-NG to verify the completeness of vendor sales data. However, they found that FPDS-NG did not contain a substantial portion of vendor sales. These officials stated that the difference between the vendor sales data and the obligations in FPDS-NG was likely due in large part to purchases under the micro-purchase threshold (currently $10,000) that agencies do not report to FPDS-NG. Because agencies do not report micro-purchases made via purchase cards to FPDS-NG, per the Federal Acquisition Regulation, this approach alone is not effective to ensure data completeness. To estimate what portion of vendor-reported sales were below the micro- purchase threshold, we compared sales data that vendors reported to VA to the obligations included in FPDS-NG data for fiscal year 2018 for the eight schedules we reviewed. We found that 54 percent of VA FSS sales as reported in vendor data were not included in FPDS-NG. VA procurement officials we interviewed told us that the 54 percent were likely micro-purchases made by medical center logistics staff using their government purchase cards. For a more detailed view, we also reviewed the percentage of fiscal year 2018 sales included in FPDS-NG for our three selected VA schedules, and found that 65 percent and 71 percent of FSS sales for the Medical Equipment and Supplies and the Patient Mobility Devices schedules, respectively, were not included in FPDS-NG, as shown in figure 6. Finally, we compared FPDS-NG data to FSS sales reports for non-VA agencies. We found instances where obligations in FPDS-NG reported by these agencies exceeded those reported by vendors in FSS sales reports—sometimes significantly. Specifically, from fiscal years 2014 to 2018, FPDS-NG reflected a cumulative $533 million more than the sales that vendors reported to NAC for non-VA agencies for the Healthcare Staffing schedule. This difference between reported sales and obligations indicates a risk that vendors are under-reporting VA FSS sales to other agencies. Standards for Internal Control in the Federal Government require that management have adequate controls for ensuring the quality of data. NAC FSS officials told us that they would like to do more to mitigate the risk that vendors may not be reporting complete VA FSS sales data, given the differences we found and that NAC also found between reported sales and obligations. Additional internal controls, similar to the process used by GSA, would help NAC to ensure vendor-reported sales are complete, and that the appropriate IFF sales fees that finance the FSS program are collected. We found that NAC does not assess data on the participation of, and items and services offered by, veteran-owned small businesses in NAC’s FSS program. This information is important because VHA contracting staff must apply the “VA Rule of Two” preference before contracting with a non-veteran-owned business. This preference for veteran-owned small businesses under the Veterans First program—which VA implemented more expansively after the 2016 Supreme Court decision—has had a major impact on VA procurement, as we reported in 2018. Thus, the availability of information about goods and services offered by veteran- owned small businesses on FSS affects whether contracting officers can use FSS as a simplified means of making purchases. However, NAC officials do not track the types of goods and services offered by veteran- owned small businesses holding FSS contracts, or which schedules have the most or least participation by these businesses. In interviews for this review and from our prior work, 10 contracting officers told us they use FSS less often than in the past. They cited the Veterans First requirement and instances where their market research showed a lack of veteran- owned small businesses holding FSS contracts. Instead, these contracting officers said, they found the goods and services they needed from veteran-owned small businesses on the open market. In addition, officials with VA’s Office of Small and Disadvantaged Business Utilization told us that that they do not analyze existing data on veteran-owned small businesses to assess these businesses’ participation in the FSS program. To conduct our own analysis, we looked at March 2019 data for the three selected VA schedules and found that goods and services provided by veteran-owned small businesses ranged from 11 to 23 percent of all the line items offered on these schedules, as shown in figure 7. According to the Standards for Internal Control in the Federal Government, program officials need quality information on how well their programs are serving end users—in this case, VHA contracting officers. Without analyzing veteran-owned small business participation in its FSS program, NAC cannot assess whether its program is meeting the needs of its users in light of the Veterans First preference, which requires contracting officers to apply the VA Rule of Two before purchasing through a non-veteran-owned business. Because the number of veteran- owned small businesses available on FSS directly affects how often contracting officers are able to use FSS, taking steps to better understand these data would enable NAC to, if necessary, make adjustments to its program to ensure contracting officers can use FSS as a regular, reliable, and simplified source for obtaining goods and services. NAC is not consistently obtaining and analyzing feedback from FSS users on their experience with the FSS program, despite having some tools in place to gather such information. To provide support to users, NAC’s website provides links to the FSS Help Desk and to a customer survey, among other contact information. However, while these tools could be used to gather feedback on whether the products and services offered on VA’s schedules meet user needs, NAC has received minimal user feedback via these tools. FSS leadership acknowledged the importance of user feedback, and stated they would like to develop a more comprehensive feedback mechanism, such as email surveys sent to users on a periodic basis with questions specific to their FSS program experience. Without such a feedback mechanism, NAC officials lack information on users’ experience with the program that could provide insights on areas for improvement. These insights, including whether program improvements are needed, are especially important given that FSS sales did not keep pace with the increased VHA spending over the past 5 years. Standards for Internal Control in the Federal Government state that, in order to formulate a strategy and achieve program objectives, management needs quality information to make informed decisions and evaluate performance. The NAC FSS program office faces numerous challenges—some of which are VA-wide issues we have identified in prior reports—including inadequate training and leadership instability. For example, NAC FSS guidance and training for contracting staff is not comprehensive, which poses a risk of inefficient use of contracting staff. Further, limited collaboration between FSS leadership at both NAC and GSA has resulted in missed opportunities to share tools and practices. These and other challenges faced by NAC were further exacerbated by a 3-year leadership gap in the FSS program; these positions have since been filled. The Federal Acquisition Regulation, along with GSA and VA’s FSS regulations and policies, form the basis for NAC’s management of the VA FSS program. NAC issues additional guidance to operationalize these higher-level policies into NAC’s FSS work processes. This internal guidance takes several forms, including Procedural Guidelines, FSS Bulletins, and Standard Operating Procedures. We found that NAC’s internal guidance does not provide contracting staff with a comprehensive overview of key aspects of their jobs, creating confusion for the staff that implements the guidance. For example, NAC contracting staff members we interviewed stated that, for offers from resellers and distributors without significant commercial sales, assessing price reasonableness was a challenge. They said NAC’s standard processes assume that commercial sales data would be available to form the foundation of price analysis, but no NAC guidance outlines how to approach this analysis for distributors and resellers that lack significant commercial sales. Additionally, several contracting staff we interviewed told us that FSS team chiefs and supervisors provide them guidance informally, which can create confusion and variation in applying requirements across the VA FSS teams. For example, one member of the contracting staff told us that some teams require vendors to submit new commercial sales data when exercising an option to extend an FSS contract. But he told us this is not the case across all of VA’s FSS teams. Standards for Internal Control in the Federal Government state that management should periodically review policies, procedures, and related control activities for continued relevance and effectiveness in achieving objectives or addressing related risks. Reviewing NAC FSS policies and procedures, including those given informally, will allow NAC FSS leadership to provide more comprehensive guidance to all FSS contracting staff on these basic, but critical, steps to help ensure an even application of the FSS offer review process. We found that training for NAC FSS contracting staff is not comprehensive, posing another challenge to NAC’s management of its FSS program; we also cited inadequate training when adding VA Acquisition Management to GAO’s High Risk list. NAC contracting staff attend training at VA’s Acquisition Academy, and several staff members we interviewed said they found it helpful. However, according to NAC officials, and based on our review of course materials and VA-wide training, the Academy does not provide FSS-specific training. There are differences between awarding and managing FSS contracts and other types of federal contracts. For instance, before awarding an FSS contract, NAC FSS contracting staff must take steps that in some cases are specific to the evaluation of FSS contract offers, such as seeking to obtain discounts from a vendor’s commercial pricelist that are equal to or greater than the discounts offered to the vendor’s most favored customer. In addition, while NAC has provided some FSS-specific training to its contracting staff, such training is not offered on a regular basis, and no overall FSS training program or curriculum exists to guide NAC training for contracting staff. According to the NAC FSS official responsible for training, the last comprehensive office-wide FSS training provided to staff was about 3 years ago (since then, 16 new contracting staff have joined NAC FSS). We interviewed 16 other NAC contracting staff, and six of them expressed the need for more extensive training on FSS-specific contracting. In mid-2018, the NAC FSS Director created Team Lead positions to help train and mentor FSS contracting staff. This effort is still in development. The NAC official responsible for training, among other things, stated that other training and mentoring efforts are underway since he joined NAC in November 2018, specifically among individual NAC schedule teams, which sometimes provide informal training to their contracting staff. However, some of these informal training efforts are not offered on a consistent basis. For example, a senior member from the contracting staff on the Medical Equipment and Supplies schedule team provided a series of training sessions to his team on evaluating offers from distributors and resellers without significant commercial sales to the general public. However, this training was not provided to all NAC FSS contracting staff, even though contracting staff working on most of the eight schedules must review offers from distributors and resellers. NAC officials are in the early stages of exploring ways to improve training for their contracting staff. In the summer of 2019, NAC officials told us they plan to post materials from all prior training on NAC’s intranet. They also developed new orientation training for the nine new contracting staff who joined NAC in July 2019, covering topics such as FSS policies and where to find them, as well as an introduction to contract systems. The NAC FSS official responsible for training emphasized that these topics were covered at a basic level and said he would like to develop more comprehensive training that would be offered on a consistent basis. Standards for Internal Controls in the Federal Government state that well- trained staff, among other things, are essential to effective program execution. In addition, GAO’s guide for assessing federal government training states that in order to ensure training is effective, training programs should be guided by an overall strategy, informed by assessing priorities and evaluating results. While NAC has taken some steps to improve training for FSS contracting staff, NAC has yet to implement a comprehensive and consistently offered FSS training curriculum. Doing so could enable NAC to provide its staff with the tools and clarity needed to perform their roles and increase efficiency. We found that NAC faces challenges effectively collaborating with GSA, the agency that oversees all FSS for the federal government. GSA has longstanding processes and established tools—such as its use of analysts to review vendors’ internal controls, as well as its automated offer-intake system—stemming from its decades of experience running an FSS program that is larger than VA’s program. However, collaboration and knowledge-sharing between NAC and GSA is limited. For example, in the past, NAC and GSA held meetings quarterly, but since 2015, these meetings have been held on an ad hoc basis. Neither organization took action until recently to ensure that meetings continued at regular intervals. GSA officials told us that during 2016 through 2018, they met with NAC a number of times in response to questions from NAC FSS officials. However, these meetings covered general policy questions, and according to NAC, did not focus on discussing cross-agency roles and responsibilities or on sharing practices for managing the FSS program. Separately, VA Office of the Inspector General’s Office of Contract Review officials told us of a 2010 working group formed to collaboratively discuss revisions to GSA’s regulations, which included representatives from GSA, NAC, the VA Office of the Inspector General, and others. According to these VA Inspector General officials, this group was disbanded about a year after it began due to disagreements among the participants. Upon NAC’s FSS Director’s request, in February 2019, NAC and GSA resumed quarterly meetings. However, NAC officials noted instances where collaboration is still limited. For example, GSA did not provide NAC officials with advance notice about the publication of a final rule establishing changes to GSA’s FSS regulations that were relevant to NAC’s administration of its FSS program. NAC discovered the final rule had gone into effect after it was published, independent of any communication from GSA. NAC officials stated they would like to have additional opportunities for input on GSA regulations that affect the VA FSS program. Our prior work has found that clearly defining roles and responsibilities is a key practice for cross-agency collaboration. Without a clear and shared understanding of their respective responsibilities, and processes to ensure they share tools and practices, NAC will not have the opportunity to learn from GSA’s experience or have timely input on GSA actions that affect the VA FSS program. GSA and VA are also missing a document—namely, GSA’s updated delegation of authority to VA—that could guide their collaboration efforts. This delegation should state what authority is granted to VA, and cite the limitations on that authority. We found a January 2008 Federal Register notice mentioned a 2004 update to the delegation, but the GSA Director of Policy for the Federal Acquisition Service was unable to locate or provide this update. VA was also unable to locate a copy of the 2004 update. Instead, GSA and VA gave us a number of documents, including memorandums and other communications that spanned from the 1960s to the 1990s. The documents were fragmented and outdated. Further, many of these older documents referred to outdated laws, regulations, or organizations, raising questions about their current applicability. Standards for Internal Control in the Federal Government state that the roles of those responsible for carrying out programs should be clearly outlined in policy, and GAO has also reported that written guidance and agreements on collaboration are key features of successful cross-agency collaboration. The lack of current documentation related to GSA’s delegation to VA, alongside the limitations in NAC and GSA communication, undermine a firm foundation on which to build collaboration. Without a clear delineation of roles and responsibilities—and effective overall coordination—NAC and GSA risk misunderstandings and missed opportunities to share information and tools that could improve NAC’s management of the VA FSS program. From 2015 until 2018, senior VA FSS leadership positions were vacant, which affected VA’s FSS program management and directly contributed to many of the challenges we identified above. Namely, the FSS program director position was vacant for over 2 years and the role of FSS Program Management and Resource Support team chief was vacant for about 19 months. During that time, chiefs of individual VA schedules held the Director or Chief positions on an acting and rotational basis. During these rotations, these chiefs were dual-hatted as they maintained responsibility for their primary job role. The Associate Executive Director of the NAC stated that he was reluctant to make long-term, strategic policy decisions while the FSS Director position was vacant. In late 2017 and late 2018, respectively, NAC permanently filled these two FSS program positions. However, by then, broader changes had taken place within VA contracting that affected the VA FSS program: namely, the Supreme Court ruled in 2016 that before VA may contract with a non-veteran-owned business, VA must apply the “VA Rule of Two,” including instances when VA makes purchases through FSS. Also in late 2016, VA launched the MSPV-NG program, which offers items similar to those items offered on two VA schedules. Figure 8 provides a timeline of these FSS leadership vacancies and events. Although both of these leadership positions have since been filled, the effect of the gaps is still evident in some cases. According to NAC FSS officials, hiring to fill open FSS contracting staff positions was slowed by the leadership gaps, which added to workload pressures; more contract offers were received than completed in fiscal years 2015 and 2016, creating a backlog. Contracting staff workload is a VA-wide issue we previously identified, and is one of the areas of concern we cited in adding VA Acquisition Management to GAO’s High Risk List in 2019. In late 2018, the FSS Director sought approval for 10 additional contracting staff; 9 of these positions were filled in July 2019. NAC leadership stated that these positions should help address some of the backlog faced by FSS contracting staff. NAC has experienced major delays in awarding vendor contracts and missed its timeliness goal for contract award 75 percent of the time from fiscal years 2014 through 2018. NAC’s inefficient offer intake system and fragmented vendor guidance likely contributed to these delays. NAC FSS leadership has acknowledged these challenges and is working to address some of them. Assessing the appropriateness of these timeliness goals and taking steps to comprehensively identify and address barriers to achieving them will better position NAC’s contracting workforce to improve contract award timeliness. Our analysis shows that from fiscal years 2014 through 2018, NAC did not meet its timeliness goals for 75 percent of its FSS contract awards. To do this analysis, we compared NAC FSS contract award data for the eight non-pharmaceutical schedules against the timeliness goal of 180 calendar days for contracting staff reviews and decisions on vendor contract awards, as set forth in a NAC Procedural Guideline. Specifically, we found that 319 of the 803 FSS contract awards took at least double the 180-day goal. In addition, 12 of them exceeded the goal by more than 1,080 days—six times the goal. During fiscal years 2015 and 2016, the program accumulated a backlog of FSS offers, which coincided with a vacancy in the FSS Director position starting in October 2015. While NAC staff made some progress on mitigating this backlog, timeliness remains an issue. For example, during fiscal year 2018, NAC missed its timeliness goal 73 percent of the time. Figure 9 portrays NAC’s timeliness of FSS contract awards over this 5-year period. We also analyzed the timeliness of awards for the three selected VA schedules from fiscal years 2014 through 2018, as shown in figure 10. This analysis shows that NAC consistently missed its timeliness goals across the three different schedules. However, for contract modifications—typically changes to contract items or prices—NAC met its timeliness goal—set at 60 calendar days—80 percent of the time over this 5-year period. We reviewed data on about 14,000 modifications executed by NAC for the eight non-pharmaceutical schedules from fiscal years 2014 to 2018. About 2,300 of these modifications were executed to add new items to existing contracts. For these, NAC met the timeliness goal only 54 percent of the time. The ability to quickly add new items to FSS contracts is important to ensure that agency users have access to up-to-date medical supplies and services through the VA FSS program. These timeliness goals apply across all of NAC’s eight non- pharmaceutical schedules, regardless of how complex a contract award or modification might be. Various factors can affect contracting staff’s ability to meet these goals, including the complexity of the award, staff’s workload, and whether or not vendor documentation is complete. NAC FSS leadership has acknowledged these challenges and is working to address some of them. However, NAC has not assessed if the current timeliness goals are appropriate, or performed a comprehensive assessment of the barriers that prevent FSS contracting staff from achieving timeliness goals. Standards for Internal Control in the Federal Government state the importance of management making well-informed decisions and conducting meaningful evaluations of their organization’s performance. Assessing the appropriateness of current timeliness goals and taking steps to comprehensively identify and address barriers to achieving them will better position NAC contracting staff to meet these goals. Moreover, timelier contract awards enable medical centers to obtain needed goods and services and, as a result, help FSS to remain useful to medical centers. When seeking a VA schedule contract award, NAC processes require vendors to submit an offer and required documents. Because NAC’s offer intake system is not automated, NAC officials must manually check a general FSS email inbox for vendor submissions and manually review the vendor’s offer and required documents to determine if all information is included. Further, there are no automated checks for completeness of vendor documentation. We analyzed a non-generalizable sample of 26 selected FSS contracts awarded beginning in fiscal year 2014 through January 2019 on three schedules—Medical Equipment and Supplies, Patient Mobility, and Healthcare Staffing—and found that in 14 instances, VA contracting staff identified incomplete documentation and had to follow up with the vendors to receive revisions. NAC officials told us that tracking vendor offers and associated documents from email is cumbersome and time consuming because they have to sort through several separate vendor email messages to splice together vendor offer submissions, due to file size limitations. The inefficient offer intake process also led to delays in assigning offers to contracting staff. NAC did not always assign offers to contracting staff immediately after vendor submission, delaying the start of work. Our review of 26 selected VA FSS contract files identified 10 instances where NAC took more than 20 days after receipt to assign the offer to contracting staff. NAC officials told us that these delays were caused by both the non-automated offer-intake process as well as the team chiefs’ lack of time to assign these offers to contracting staff for their review. According to the FSS Director, in mid-2018 he created a team lead for each NAC schedule team to assign offers to contracting staff and monitor these offers to better ensure timeliness. Figure 11 summarizes key steps in NAC’s FSS manual offer intake and award process, as described by FSS contracting officials. In contrast to VA’s manual system, since 2004, GSA has used an online system called eOffer to manage its FSS offer intake process. This system includes automated system checks to ensure documentation is complete before it is submitted by vendors. Once offers are submitted, supervisors in GSA FSS offices review offers in the system and assign them to contracting staff for review. We have not evaluated whether the eOffer system increases efficiency or reduces errors in submitted offers, but, according to GSA officials, eOffer achieves efficiency and accuracy due to the automated checks that will not let vendors submit an incomplete offer package. NAC contracting staff told us that they could benefit from a more efficient system to accept offers from vendors. In late 2018, NAC and GSA discussed the possibility of adopting GSA’s eOffer online system, as well as its companion eMod, which is used to process modifications. In November 2018, GSA’s estimate to add VA to the system was about $9 million for the first year, and nearly $8 million annually thereafter, which, according to NAC’s FSS Director, is cost-prohibitive. GSA officials told us that to determine this cost, they compared the number of NAC FSS contracts to the number of GSA FSS contracts, and apportioned 10 percent of the overall system development and operation cost to VA. Despite the cost of GSA’s systems, if VA does not address limitations in its own manual offer-intake process, such as implementing a system that can provide automated checks for completeness, delays in assigning offers to contracting staff will continue. Further, FSS contracting staff will continue to spend additional resources and time to gather and complete offer documentation before they can determine whether to award the contract. We also reviewed NAC’s website and found that guidance for vendors was fragmented. Pieces of guidance were spread out among a number of documents as opposed to being located in one document or section of the website for vendors to easily locate. This also contributes to vendors submitting incomplete offer documentation, which, in turn, contributes to delays in contract awards. Incomplete documentation for pricing and sales data is particularly common—namely, information that enables contracting staff to compare the prices vendors offer the government and commercial customers for the same goods. As stated in Standards for Internal Control in the Federal Government, clear communication with outside parties, like vendors, is essential to ensuring that NAC is able to help achieve its program objectives. Clearer guidance would provide vendors with a reminder of program requirements that could reduce VA FSS contracting staff review time and improve their efficiency in reviewing contract offers. Over the past few years, the FSS and MSPV programs have transitioned from functioning together to existing as separate programs serving similar VA medical center needs. However, VA leaders have not assessed if the overlapping offerings are a necessary and effective use of resources, or allow VA to fully leverage its buying power—a stated goal of both the MSPV-NG and FSS programs. As we reported in November 2017, for over a decade, VA’s medical centers used VHA’s legacy MSPV program to order medical supplies—many of which were purchased using NAC’s FSS program. When VHA transitioned to its MSPV-NG program in late 2016, it significantly narrowed the catalog to 6,000 items, and VHA contracting officials told us that they modified the contracting approach in March 2018 to have the prime vendor supply the items directly, separate from FSS. At the outset, VHA set goals for the MSPV-NG program, including standardization of requirements for supply items and cost avoidance by leveraging VA’s substantial buying power. However, the MSPV-NG program recently revised its goals from focusing on standardization to increasing the number of catalog items available for medical centers’ use—the catalog contains more than 20,000 items as of September 2019. We compared the MSPV-NG catalog to the VA Medical Equipment and Supplies schedule to determine whether they offered similar products, and found overlap. For example, we found that as of June 2019, about two-thirds (139 of 206) of the MSPV-NG catalog suppliers also offered items on the Medical Equipment and Supplies schedule. Also, in March 2019, NAC FSS leadership provided analysis to the MSPV-NG program office showing that 41 percent of items that the MSPV-NG program planned to include in an update of the MSPV catalog were already available under VA FSS contracts. This duplication could result in inefficiencies whereby different sets of contracting staff within the FSS and MSPV-NG programs award, modify, and manage contracts for the same or similar medical supplies for VA medical center use. The MSPV-NG program office is currently developing the next iteration of the program, called MSPV 2.0, which it plans to roll out in February 2021. In April of 2019, a senior VHA procurement official announced at a vendor forum that the VA FSS program would be used as a source for its MSPV 2.0 supply catalog. However, in June of 2019, MSPV-NG program officials and VHA procurement leadership told us they decided against using FSS for this purpose and provided several reasons for this decision. First, these officials stated that FSS was not comprehensive enough to fulfill the MSPV 2.0 catalog; as noted above, FSS could provide about 40 percent of needed items. These officials also stated that the effort needed to create new FSS contracts or add new items to existing VA FSS contracts to fulfill the remaining 60 percent of the required MSPV 2.0 catalog would be too time consuming. Further, these officials also stated that there were not enough veteran-owned small businesses that offer items on VA FSS to support the MSPV 2.0 requirements. They stated this could result in extra time and resources to solicit both within FSS and the open market to ensure that they meet the VA Rule of Two. NAC FSS leaders told us that they communicated their willingness to support the MSPV 2.0 program by offering to work with vendors to quickly add the needed items; however, the MSPV-NG program office did not involve them in their final decision not to use FSS as a source for the MSPV 2.0 program. VA procurement leaders have informally discussed the future of the FSS program, according to a senior VHA procurement official. According to this official, VA has not determined whether it will change the strategy for FSS, or if the duplication between the FSS and MSPV-NG programs is a necessary and efficient use of resources. VA’s Strategic Plan for Fiscal Years 2018-2024 calls for related efforts to be coordinated with each other to achieve cross-organizational unity of purpose. When adding VA Acquisition Management to our High Risk List in March of 2019, we reported that VA lacks an effective medical supplies procurement strategy. While this finding stemmed from our review of VA’s MSPV-NG program and was related to the recommendation that VA develop an overarching strategy for this program, the same applies for VA’s FSS program in that VA does not have a strategic approach for its procurement of medical supplies through these two programs. Further, we reviewed VHA’s Modernization Campaign Plan, dated March 2019, and VHA’s Modernization Plan briefing slides, dated October 2019, which describe several modernization initiatives. One of these initiatives is to transform the supply chain through modernization. This modernization plan includes the planned MSPV 2.0 program and VA’s planned changes to its supply chain management system; however, it does not include FSS. As VA is undertaking these efforts, it is unclear how and whether FSS fits into VA’s vision of a modernized supply chain. Taking steps to assess VA FSS and MSPV program duplication will allow VA to determine if it is efficiently using its contracting staff. Moreover, communicating its decision to managers of these two programs will allow these managers to focus and coordinate their resources accordingly. This assessment will also help VA determine if it is leveraging its buying power to improve the effectiveness and efficiency of services for veterans and their families. The continued utility of parts of VA’s FSS program is in question amid flat sales in recent years and competing programs available to medical center staff for supplies, such as MSPV-NG. VA has the opportunity to improve its FSS program by ensuring that it has complete vendor sales data and better information on participation by veteran-owned small business and user experiences. Obtaining such information would enable NAC to ensure it is collecting all fees it is owed which support program operations, and ensure that FSS can remain a regular, reliable and simplified source for contracting officers to obtain goods and services on behalf of the medical centers. Other steps are necessary, however, to address challenges VA faces with its FSS program. Specifically, the FSS program needs to provide comprehensive guidance and training to its contracting officers, and assess timeliness goals and barriers to achieving these goals to ensure the program remains useful to customers—namely medical centers that rely on the goods and services provided by FSS. In working to improve its FSS program, NAC has the opportunity to gain insights and experience from GSA on how it manages its much larger schedules program. However, lack of collaboration between GSA and NAC has resulted in missing opportunities for such information sharing. Finally, both VA’s FSS and MSPV-NG programs support VA’s overall medical supply chain, yet VA has not assessed whether duplication between them is a necessary and effective use of resources. Without this assessment, VA could be missing opportunities to leverage buying power and improve efficiency in procuring goods and supplies for its medical centers. We are making a total of 11 recommendations, including nine to VA and two to GSA: The Secretary of Veterans Affairs should ensure that the Associate Executive Director of VA’s NAC puts controls in place to better ensure the completeness of vendor FSS sales reporting. (Recommendation 1) The Secretary of Veterans Affairs should ensure that the Associate Executive Director of VA’s NAC assesses data on the participation of and items and services offered by veteran-owned small businesses in NAC’s FSS program, in order to determine whether their program is meeting the needs of VHA contracting officers who use it given the Veterans First requirements they must meet. (Recommendation 2) The Secretary of Veterans Affairs should ensure that the Associate Executive Director of VA’s NAC directs the FSS Director to develop a mechanism to consistently obtain and analyze VHA user feedback on the FSS program. (Recommendation 3) The Secretary of Veterans Affairs should ensure that the Associate Executive Director of VA’s NAC provides FSS contracting staff with comprehensive FSS guidance. (Recommendation 4) The Secretary of Veterans Affairs should ensure that the Associate Executive Director of VA’s NAC develops an FSS-specific training program to include essential skills and processes to meet ongoing training needs for new and existing contracting staff. (Recommendation 5) The Administrator of GSA should work with the Secretary of VA to develop a memorandum of understanding outlining the roles and responsibilities of GSA and NAC for collaborating under GSA’s delegation of authority to VA for the healthcare-related Federal Supply Schedules, including the processes through which the two organizations will coordinate and share useful tools and practices. (Recommendation 6) The Secretary of Veterans Affairs should work with the Administrator of GSA to develop a memorandum of understanding outlining the roles and responsibilities of GSA and NAC in collaborating under GSA’s delegation of authority to VA for the healthcare-related Federal Supply Schedules, including the processes through which the two organizations will coordinate and share useful tools and practices. (Recommendation 7) The Administrator of GSA should take steps to document its delegation of authority for the healthcare-related Federal Supply Schedules to VA. (Recommendation 8) The Secretary of Veterans Affairs should ensure that the Associate Executive Director of VA’s NAC assesses the appropriateness of NAC’s current timeliness goals for FSS contract awards and takes steps to comprehensively identify and address barriers to achieving them. (Recommendation 9) The Secretary of Veterans Affairs should ensure that the Associate Executive Director of VA’s National Acquisition Center takes measures to ensure greater efficiency in the offer-intake process, such as providing additional guidance for vendors or by adopting a system that includes checks for completeness of required vendor documentation. (Recommendation 10) The Secretary of Veterans Affairs should take steps to assess duplication between VA’s FSS and MSPV programs, to determine if this duplication is necessary or if efficiencies can be gained. (Recommendation 11) We provided a draft of this report to the Department of Veterans Affairs and to the General Services Administration for review and comment. In VA’s comments, reproduced in appendix II, it concurred with all of our nine recommendations. In GSA’s comments, reproduced in appendix III, it concurred with our two recommendations. We are sending copies of this report to the appropriate congressional committees. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of the Department of Veterans Affairs and the Administrator of the General Services Administration. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by email at oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report assesses: (1) what is known about the Department of Veterans Affairs’ (VA) use of its Federal Supply Schedule (FSS) program for fiscal years 2014 through 2018; (2) challenges the National Acquisition Center (NAC) faces in effectively managing the FSS program, (3) the extent to which NAC awarded FSS contracts in a timely manner from fiscal years 2014 through 2018, and (4) the extent to which FSS and the Medical-Surgical Prime Vendor-Next Generation (MSPV-NG) programs provide overlapping or duplicative offerings. To assess what is known about VA’s use of its FSS program from fiscal years 2014 through 2018, we analyzed quarterly vendor sales report data provided by NAC. To assess the reliability of these data, we reviewed documentation and other information on the internal controls of the data systems used by NAC to collect and verify VA FSS sales reports. We found these data to be sufficiently reliable for analyzing overall trends in sales on VA’s FSS for this time period. We also obtained contracting data from the Federal Procurement Data System-Next Generation (FPDS-NG) for fiscal years 2014 through 2018, as well as a listing of VA FSS contracts active during that period from NAC. We used these data to analyze contract obligations on VA FSS contracts over this time period. Further, we compared total obligations based on FPDS-NG data to vendor-reported sales data provided by NAC by schedule and fiscal year. We found the FPDS-NG data sufficiently reliable for our purpose of comparing reported obligations to vendor sales data. We also used information obtained from a prior GAO review of VA’s Veterans First Program when discussing FSS use and used Standards for Internal Control in the Federal Government as criteria to assess this use. We selected the Medical Equipment and Supply (65IIA), Patient Mobility (65IIF), and Healthcare Staffing (621I) schedules as the focus of our review, based on total number of active contracts; they collectively represented about two-thirds of the approximately 1,700 active VA FSS contracts at the time we began our review. We excluded the pharmaceutical schedule from our review because, unlike the other schedules, orders are placed almost exclusively through the pharmaceutical prime vendor, and participation is a statutory requirement. We also reviewed and compared VA, Veterans Health Administration (VHA), and General Services Administration (GSA) policies, guidance, and memorandums related to the program and interviewed VHA- and VA- wide procurement officials regarding factors that affect use of the VA schedules, including the Veterans First program. We also analyzed data on items offered by vendors on the three selected schedules to determine the percent of items offered by veteran-owned small businesses. We interviewed VHA contracting staff and supply chain logistics staff at VA medical centers—users of the program—on factors that affect their use of FSS. We conducted site visits at a non-generalizable selection of two Veterans Integrated Service Networks (VISNs), visiting one medical center within each. Additionally, we interviewed Network Contracting Office (NCO) officials within each selected VISN, either in person or via telephone: VISN 12: VA Great Lakes Health Care System Clement J. Zablocki VA Medical Center (Milwaukee, Wisc.) Great Lakes Acquisition Center, NCO 12 (Milwaukee, Wisc.) VISN 10: VA Healthcare System Cincinnati, Ohio VA Medical Center NCO 10 (via telephone) We selected VISNs and medical centers primarily based on geographical proximity to NAC and GAO offices, as well for higher total obligations in fiscal year 2018. At each selected medical center, we interviewed the Facility Chief Supply Chain Officer and other members of the logistics staff. At each selected NCO, we interviewed leadership, branch chiefs, and contracting officers on teams that cover goods and services included on the three VA schedules we selected. Separately, we also spoke with representatives of the Coalition for Government Procurement, a group representing a number of FSS vendors, and attended a conference for vendors organized by NAC. To assess challenges NAC faces in effectively managing the FSS program, we reviewed GSA and VA procurement regulations, policies, and guidance as well as NAC FSS guidance. We analyzed the content of training offered by the Veterans Affairs Acquisition Academy and by NAC FSS. We also reviewed systems and processes used by NAC FSS staff to accept and review FSS offers and award contracts. We also interviewed NAC FSS leadership, contracting staff, and other staff regarding management of the FSS program during a site visit to NAC. We obtained and analyzed information on NAC FSS staffing, including leadership vacancies. We obtained documentation on analogous GSA practices for managing its FSS program, as well as documents delegating management of healthcare-related schedules to VA. We also interviewed officials in GSA’s Federal Acquisition Service who are responsible for overseeing its FSS program. To determine the extent to which NAC met its timeliness goal for processing FSS offers and modifications, we analyzed timeliness data collected by NAC for fiscal years 2014 through 2018, for the eight non- pharmaceutical schedules; we also performed limited analysis of timeliness for the pharmaceutical schedule, and additional analysis for our three selected schedules. We focused our analysis of timeliness on offers resulting in a contract award, because these are the cases that are relevant to users of the FSS program. To provide context for overall workload, we also analyzed timeliness for offers that were withdrawn, or where contracting staff decided not to make an award. We excluded offers that were reviewed by the VA Inspector General, Office of Contract Review, from our overall timeliness analysis because NAC policy does not count the time required for these reviews against its timeliness goal. To assess the reliability of timeliness data, we collected information on the system and processes used to maintain the data, performed electronic testing, and compared reported dates to source documents for selected contracts. We found these data sufficiently reliable for the purpose of assessing overall performance and trends in NAC FSS timeliness. From the three selected schedules, we selected a non-generalizable sample of 26 NAC FSS contracts awarded in fiscal years 2014 through January 2019. Eight of the contracts were randomly selected from all active contracts on the three schedules as of January 2019, while the remaining 18 contracts were selected by stratified random sample of contracts awarded in fiscal year 2018, focusing on those which exceeded the 180-day timeliness goal and omitting those with few or no sales. Thirteen of the contracts were under the Medical Equipment and Supplies schedule, seven of the contracts were under the Healthcare Staffing schedule, and the remaining six were under the Patient Mobility schedule. For each selected contract, we reviewed documents in the contract file; we also interviewed cognizant members of the contracting staff for 16 of the contracts. We selected this non-generalizable sample to provide illustrative examples of process steps and factors affecting timeliness; it was not the sole source of our findings on factors contributing to timeliness, which also included analysis of policies, guidance, and data, and interviews with NAC officials. To assess the extent to which the FSS and MSPV-NG programs provide overlapping or duplicative offerings, we reviewed policy and guidance related to both programs and interviewed VHA- and VA-wide procurement leaders. To assess the extent of overlap between the MSPV-NG and VA FSS catalogs, we also analyzed data on the items available through MSPV-NG and VA FSS, as well as the vendors participating in each, to assess extent of duplication. We interviewed VA officials from NAC, the Office of Acquisition and Logistics, the Strategic Acquisition Center, and VHA regarding the relationship between MSPV-NG and the FSS program. We analyzed policies related to these programs that affect management and use of VA FSS, and interviewed VA officials about their impact. We also reviewed documents and interviews with MSPV program office staff from an ongoing GAO review of the MSPV program. We used information obtained from an ongoing GAO review as well as published GAO reports on VA’s MSPV-NG program. Finally, we reviewed documents, including VA’s 2018-2024 Strategic Plan and VHA supply chain modernization plans. We conducted this performance audit from November 2018 to January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Shelby S. Oakley, 202-512-4841 or oakleys@gao.gov. In addition to the individual named above, Lisa Gardner, Assistant Director; Teague Lyons, Analyst-in-Charge; Erin Stockdale; Sarah Amer; Maurice Robinson; Emily Bond; Rashmi Agarwal; Andrew Burton; Virginia Chanley; Matthew T. Crosby; Susan Ditto; Lori Fields; Gina Flacco; Suellen Foth; and Alyssa Weir made key contributions to this report.", "summary": "Through the FSS program, VA manages nine healthcare-related schedules—groups of contracts used to order medical supplies and services—under authority delegated by GSA. VA's FSS program management, including the speed with which it adds new contracts, affects VA medical centers' ability to use it to easily obtain goods and services. Further, recent changes in VA's medical procurement have also raised questions about the future role of the program. GAO was asked to examine VA's management and use of its FSS program. This report assesses (1) what is known about VA use of its FSS program for fiscal years 2014-2018; (2) program management challenges faced by NAC; (3) the extent to which NAC awarded FSS contracts to vendors in a timely manner from fiscal years 2014-2018; and (4) the extent to which the FSS and MSPV-NG programs provide overlapping or duplicative offerings. GAO reviewed eight VA schedules, excluding pharmaceutical due to the use of a prime vendor, among other things. GAO also analyzed three of these schedules representing about two-thirds of VA's FSS contracts; analyzed policies, guidance, and processes; and interviewed senior VA procurement, contracting, and supply chain logistics staff at NAC and two medical centers. Over the past 5 years, Department of Veterans Affairs (VA) medical spending increased, but spending on its eight non-pharmaceutical Federal Supply Schedules (FSS) was flat. GAO found the vendor-submitted sales reports to be sufficiently reliable for describing these trends. However, GAO found that VA's National Acquisition Center (NAC)—the VA-wide contracting organization responsible for FSS—lacks controls to ensure the completeness of vendor sales data, which is used to calculate fees that finance the program. The FSS program faces numerous challenges. For instance, NAC FSS guidance and training are not comprehensive, posing a risk of inefficiency and uneven application of requirements by contracting staff. Limited collaboration between FSS leadership at both NAC and the General Services Administration (GSA) also resulted in missed opportunities to share tools and practices. A 3-year FSS leadership gap further exacerbated challenges; these positions are now filled. NAC also failed to meet its 180-day timeliness goal for 75 percent of the non-pharmaceutical FSS contracts it awarded from fiscal years 2014 through 2018 (see figure), though NAC met its goal for contract modifications 80 percent of the time. By assessing timeliness goals and identifying barriers to achieving them, NAC leadership can take steps to better enable its contracting workforce to provide an efficient and reliable means to obtain needed goods and services through FSS. Moreover, VA's procurement leaders have not assessed, and communicated to program managers, whether the duplication between FSS and the Medical Surgical Prime Vendor-Next Generation (MSPV-NG) program is a necessary and effective use of resources. These two programs feature many of the same items, and different contracting staff manage different contracts for the provision of the same or similar medical supplies for VA medical centers. Without assessing duplication between these two programs, VA is at risk of inefficient use of its contracting workforce, and may be unable to fully leverage its buying power. GAO is making 11 recommendations: nine to VA and two to GSA; including that VA provide comprehensive guidance and FSS-specific training, improve NAC and GSA collaboration, evaluate timeliness goals and barriers, and assess FSS and MSPV-NG program duplication. VA and GSA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Since 2008, both the number of pedestrian fatalities and the share of pedestrian fatalities as a percentage of overall highway fatalities have increased (see fig. 1). In 2008, pedestrian fatalities represented about 12 percent of overall highway fatalities, while in 2018 they represented about 17 percent. In addition to fatalities, the estimated number of pedestrians injured in crashes has increased from about 71,000 in 2008 to about 79,800 in 2018. A range of factors can influence pedestrian fatalities including exposure of pedestrians to crashes, roadway characteristics, and driver and pedestrian behavior. According to DOT officials, there is little nationwide information about pedestrian exposure to potential crashes and that data may be more available on the state or local level. Some national data, however, shows that there may have been some change in people walking. For example, the U.S. Census Bureau’s American Community Survey estimated that in 2018, 4 million people reported walking to work compared with an estimated 3.8 million people in 2010. Regarding roadways, in 2018 the National Transportation Safety Board (NTSB) reported that most pedestrian fatalities occur in urban areas on principal arterial roads that carry high volumes of traffic, traveling at the highest speeds. In 2015, we noted that behavior such as distracted driving, walking, and cycling may contribute to pedestrian and cyclist fatalities. When drivers and pedestrians use cell phones or are otherwise distracted, they may be less aware of their surroundings and more likely to be involved in a crash. Finally, NHTSA data shows that most pedestrian fatalities occurred after dark and at places other than intersections. Specifically, in 2018, of the 6,300 reported fatalities, over 4,700 pedestrians (about 75 percent) were killed after dark and about 4,600 pedestrians (about 73 percent) were killed at non-intersection locations. See appendix II for additional information on pedestrian fatalities from 2008 through 2018. Automakers have developed vehicle features intended to avoid pedestrian crashes and mitigate the extent of injury to pedestrians. Crash avoidance features (also known as “active” safety features) rely on cameras, radar, and other devices to detect a pedestrian and then act to alert a driver to take action, or automatically apply a vehicle’s brakes to slow or stop the vehicle to avoid striking a pedestrian (see fig. 2). One pedestrian crash avoidance system is referred to as pedestrian automatic emergency braking, which uses a camera, radar, or a combination, to automatically apply brakes to avoid a collision. Crash mitigation features (also known as “passive” safety features) generally involve the use of pedestrian-friendly vehicle components that are designed to reduce the severity of injuries should a pedestrian be hit. Passive safety features can include energy absorbing bumper material, hoods that provide space between the hood and the hard components in the engine compartment, and contoured vehicle front-ends intended to reduce harm to pedestrians (see fig. 3). In executing its mission, NHTSA administers NCAP and issues federal motor vehicle safety standards (FMVSS) and the federal bumper standard, among other things. In general, NCAP tests supplement safety standards established in law or regulation. NCAP. Created in 1978, this program tests new vehicles to determine how well they protect drivers and passengers during a crash (front and side) and how well vehicles resist rollovers. NHTSA tests and rates vehicles using a five-star safety rating system with five stars being the highest safety rating and one star the lowest. NHTSA communicates the results of its vehicle tests through window labels on new vehicles and on its website. In 2010, NHTSA also began recommending various safety technologies for consumers to consider when purchasing vehicles. Recommended technologies include such things as forward collision warning (an alert that warns drivers to brake or steer to avoid a crash if they are too close to a car in front of them); lane departure warning (an alert that warns drivers of unintentional lane shifts); and automatic emergency braking, which can automatically activate a vehicle’s brakes if a driver takes no action to avoid an imminent crash with a preceding vehicle. NHTSA has not yet included pedestrian automatic emergency braking systems as recommended technologies. Recommended technologies are not included in star ratings, but rather are features NHTSA believes consumers may wish to look for in new vehicles. Pedestrian safety tests are not currently part of NCAP. FMVSS. These are minimum performance standards established in regulation for new motor vehicles and items of motor vehicle equipment. According to NHTSA officials, FMVSS have test procedures and performance criteria with minimum thresholds for motor vehicles and motor vehicle equipment, such as minimum light intensity requirements for headlamps. Bumper standard. In addition, while not in the FMVSS, NHTSA’s bumper standard prescribes performance requirements in regulation for passenger cars in low-speed front-end and rear collisions. According to NHTSA officials, the bumper standard is intended to prevent damage to the car body and safety related equipment at speeds equivalent to a 5 miles-per-hour (mph) crash into a parked vehicle of the same weight. The standard applies to front and rear bumpers on passenger cars, but not to other multipurpose passenger vehicles, such as SUVs, minivans, or pickup trucks. The United States is also involved with pedestrian safety internationally. In June 1998, the United States signed an international agreement administered by the United Nations concerning the establishment of global technical regulations for motor and other wheeled vehicles. The purpose of the agreement was to establish a global process for jointly developing technical regulations regarding such things as safety, environmental protection, and energy efficiency of vehicles. As part of this agreement, in 2008, Global Technical Regulation No. 9 was established to improve pedestrian safety by requiring vehicle hoods and bumpers to absorb energy more efficiently when impacted in a vehicle-to-pedestrian collision. This international standard has two sets of performance criteria: head impact requirements that ensure vehicle hoods provide protection to a pedestrian’s head when impacted; and leg protection requirements for the front bumper that would require bumpers to subject pedestrians to lower impact forces. According to NHTSA, as a signatory to the 1998 agreement, the United States is obligated to consider adopting global technical regulations, but is not obligated to adopt them. NHTSA officials told us the agency has not yet initiated the rulemaking process for Global Technical Regulation No. 9. Although pedestrian safety testing is not currently a part of the U.S. NCAP, it is a part of similarly established new car assessment programs in other countries. For example, since 2016 both the European New Car Assessment Programme (Euro NCAP) and a program in Japan (known as the Japan New Car Assessment Program (JNCAP)) have tested vehicle pedestrian crash avoidance systems using a variety of scenarios and vehicle speeds. Euro NCAP tests include an adult dummy walking or running perpendicular to a test vehicle and walking parallel to a vehicle. Tests are also conducted with a child dummy running out from parked cars (see fig. 4). Euro NCAP tests are also conducted in daylight and at night. In the United States, two nongovernmental organizations have also conducted pedestrian safety testing. IIHS began a program to test pedestrian crash avoidance systems on 2018 and 2019 vehicles, and in 2020 began using the results to help determine its Top Safety Pick awards. The American Automobile Association (AAA) also recently conducted tests of crash avoidance systems. Moreover, crash mitigation tests that measure the potential for head and leg injuries resulting from pedestrian-motor vehicle crashes have been in place for many years in Europe and Japan. Euro NCAP began head and leg testing in 1997 and Japan began pedestrian head protection testing in 2003 and pedestrian leg protection testing in 2011. In general, these tests launch projectiles designed to simulate a person’s legs or head into various locations on a vehicle’s hood and bumper to assess the effectiveness in limiting pedestrian injury (see fig. 5). We found that several vehicle characteristics including the age and body type of the vehicle and the speed at which the vehicle was being driven at the time of the crash are associated with the increase in pedestrian fatalities from 2008 through 2018. However, NHTSA lacks complete data on the relationship between vehicle characteristics and pedestrian injuries, including detailed information on injury type and severity. Although NHTSA initiated a pilot program to improve its data collection protocol for pedestrian injuries, NHTSA lacks a plan for this program to evaluate its results and determine whether and how it should be expanded. Through FARS, NHTSA annually collects and analyzes data on all crashes involving pedestrian fatalities, including vehicle-related characteristics. Based on these data and relevant research, we analyzed the relationship between pedestrian fatalities and the age, body type, and speed of vehicles. Our analysis of FARS data shows that from 2008 through 2018, the number of pedestrian fatalities increased more for crashes involving vehicles that were: 11 years old or older (123 percent increase) compared with newer vehicles (9 percent increase); SUVs (68 percent increase) compared with other light trucks (25 percent increase), and passenger cars (47 percent increase); and traveling at reported speeds 31 mph and above (45 percent increase), compared to vehicles traveling at lower speeds (28 percent increase). The number of pedestrians struck and killed by vehicles 11 years old or older (older vehicles) increased more relative to the number of pedestrians struck and killed by vehicles 10 years old or newer (newer vehicles). In 2008, 1,139 pedestrian fatalities involved older vehicles, which represented about a quarter (26 percent) of reported pedestrian fatalities (see fig. 6). By 2018, that number more than doubled to 2,537 pedestrian fatalities, or 40 percent of reported pedestrian fatalities. Over that same time period, the number of pedestrian fatalities involving newer vehicles also increased from 2,800 in 2008 to 3,044 in 2018. However, this increase was less than fatalities involving older vehicles, and the overall share of pedestrian fatalities involving newer vehicles decreased from 63 to 48 percent over that period. The rise in the number of older vehicles involved in pedestrian fatalities may reflect the rise in the average age of vehicles in operation. According to data from DOT’s Bureau of Transportation Statistics, the average age of all vehicles in operation in the United States increased by about 1.5 years from 10.1 years old in 2008 to 11.7 years old in 2018. In comparison, the average age of passenger vehicles that struck and killed pedestrians increased by roughly 2 years from 8.1 years in 2008 to 10 years in 2018. Another possible contributing factor to the increased share of pedestrian fatalities resulting from crashes with older vehicles may be the prevalence of safety features in newer vehicles compared with older vehicles. As discussed below, vehicle manufacturers are offering new vehicles with pedestrian safety features such as pedestrian crash avoidance and crash mitigation systems, which may reduce pedestrian injuries and fatalities. The number of pedestrian fatalities where passenger cars, SUVs, or other light trucks were reported as striking vehicles all increased from 2008 to 2018 (see table 1). However, the number of SUVs involved in fatal pedestrian crashes increased by a higher percentage than passenger cars and other light trucks. As table 1 shows, pedestrian fatalities involving SUVs increased by about 68 percent, while pedestrian fatalities involving passenger cars increased by 47 percent and light trucks and vans increased by 25 percent. Additionally, although the number of SUVs involved in pedestrian fatalities increased the most in this timeframe, passenger cars still accounted for the largest share of fatalities. Data on the growth of SUVs within the U.S. vehicle fleet and academic research identify potential contributing factors as to why the number of SUVs involved in pedestrian fatalities increased between 2008 and 2018: Increasing SUV market share. SUVs represent a growing share of the total U.S. vehicle fleet. According to the Highway Loss Data Institute, the share of new vehicles in the United States that were SUVs grew from 30 percent in model year 2008 to 48 percent in model year 2018. In addition, 11 of the 13 auto manufacturers we interviewed stated that SUV sales, either market-wide or at their company, increased relative to passenger car sales in the United States since 2008. Increased risk of injuries based on vehicle size and weight. Research suggests that if a pedestrian is struck by a vehicle with greater mass the crash is more likely to result in serious injuries or a fatality than if the pedestrian is struck by a lower-mass vehicle. For example, one study we reviewed that cited work from five other studies found that the chief determinants for the severity of injuries in motor vehicle collisions are vehicle size and weight. According to one NHTSA-funded study, which used information from NHTSA’s Pedestrian Crash Data Study, researchers found that the probability of death for pedestrians struck by light trucks (including SUVs) was 3.4 times higher than for pedestrians struck by passenger cars. Between 2008 and 2018, the number of pedestrian fatalities involving higher speeds (31 mph and above) at the time of the crash increased more sharply than the number involving lower speeds (30 mph and below). Although vehicle speed was missing or not reported for 62 percent of pedestrian fatalities (as discussed below), our analysis of FARS data showed that when speed data are recorded, the number of pedestrian fatalities involving vehicles reportedly travelling at higher speeds increased from 1,315 to 1,912 (45 percent) between 2008 and 2018 (see fig. 7). The number of pedestrian fatalities involving vehicles reportedly traveling at lower speeds also increased, but at a smaller percentage (28 percent) than vehicles at higher-speed. During this time period, about 79 percent of pedestrian fatalities involved vehicles travelling 31 mph and above, and about 21 percent involved vehicles traveling at lower speeds. Multiple studies have found that when vehicles travel at higher speeds and strike pedestrians, they are more likely to kill or severely injure the pedestrian. For example, the NTSB reported in 2018 that the relationship between speed and the severity of injuries is consistent and direct— higher crash speeds result in injuries that are more severe. The NTSB added that the effect of speed is especially critical for pedestrians because they lack protection. In addition, according to a 2019 report from the National Cooperative Highway Research Program, a pedestrian’s risk of fatality is 90 percent when struck by vehicles travelling between 54 and 63 mph compared with a 10 percent risk of fatality between 24 and 33 mph. We also found that between 2008 and 2018, the speed of the striking vehicle was not reported for about 62 percent of pedestrian fatalities. This omission is likely because it is difficult for police officers to determine a vehicle’s speed after a crash occurs. Further, some organizations we spoke with told us that low speed collisions were typically underreported. According to NHTSA officials, the speed recorded is generally up to the discretion of the responding police officer. NHTSA officials and other stakeholders we interviewed identified limitations in NHTSA’s data on the relationship between vehicle characteristics and pedestrian injuries. These include (1) incomplete and inconsistent injury designations, (2) crash and vehicle information not linked to medical data, and (3) outdated pedestrian crash investigation data. Incomplete and inconsistent injury information. Within CRSS, NHTSA relies on information provided in police reports to determine national estimates of injured pedestrians. According to NHTSA officials, data from the police reports are typically after-the-fact descriptions of events and NHTSA conducts little, or no, follow up investigations of these reports. As a result, CRSS data may not include the cause of crashes or pedestrian injuries, and for some crashes it may be missing detailed information on specific characteristics of the striking vehicle. In addition, there may be inconsistencies in pedestrian injury information. NHTSA’s injury severity data rely on reporting from states and localities, which may define injury severity differently, year-to-year. As we have reported, NHTSA standardized the injury severity definitions nationally in April 2019; however, it will take time for states to adopt this standard. Crash and vehicle data are not linked to medical records. According to NTSB and some researchers we spoke with, the five point injury severity scale used on police crash reports does not effectively capture injury severity or actual injury outcomes because NHTSA does not link crash data with medical and hospital records. Without crash and vehicle information linked to medical records, researchers cannot crosscheck injury severity designations with actual injury outcomes or identify specific injury types. NHTSA previously sponsored a program to help link crash data with injury data contained in medical records, but federal funding for the program was discontinued in 2013. Outdated pedestrian crash investigation data. NHTSA last collected detailed data on pedestrian crash and injury characteristics from 1994 to 1998. The Pedestrian Crash Data Study collected information from over 500 pedestrian crashes, including data on pedestrian injury types, severity, and potential causation. The study also reported the vehicle’s type and the part of the vehicle that caused the injury, such as the front bumper. In its 2018 report, NTSB stated that while this study was the most complete set of pedestrian crash data available in the United States, the data are over 20 years old. NTSB recommended that NHTSA develop a detailed and current pedestrian crash data set for local and state analysis and to model and simulate pedestrian collision avoidance systems. As of February 2020, however, NHTSA had not fully implemented the recommendation. Some automakers and equipment suppliers we spoke with noted that improved real world injury data would help them better develop pedestrian safety features. NHTSA has recognized that it needs to collect more detailed and complete data on pedestrian injuries. For example, in a 2011 report to Congress on the agency’s data gaps, NHTSA noted that internal stakeholders (those within NHTSA) requested an updated Pedestrian Crash Data Study with crashes involving late-model-year vehicles and detailed injury data on the body region impacted rather than the vehicle’s point of contact. Further, in its 2016 to 2020 strategic plan, NHTSA stated that it would work to improve the quality, timeliness and relevance of safety data collected. In 2018, NHTSA initiated a pilot program to evaluate existing and new protocols for collecting pedestrian crash and injury data as part of its Crash Injury Research and Engineering Network (CIREN). The purpose of this pilot program is to develop a data collection protocol and collect preliminary data for pedestrian-motor vehicle crashes, including analysis on injury causation. Further, NHTSA stated that it intends to use this protocol and data as the foundation for subsequent pedestrian crash studies such as research related to injury trends and testing tools. NHTSA officials also told us that the pilot will help update and build upon the data collection and analysis protocols for pedestrian-motor vehicle crashes used in the 1990s in the Pedestrian Crash Data Study. According to NHTSA officials, the pilot will collect data on nine cases from two hospitals. A third hospital will provide engineering support. NHTSA officials stated that they limited the pilot study to nine cases so they would be able to act quickly on the pilot to determine if a full project was worth pursuing and to avoid delays. According to NHTSA officials, they expect initial results to be available by fall 2020. We have reported that a well-developed and documented pilot program can help ensure that agency assessments produce information needed to make effective program and policy decisions. Well-designed pilot programs use five leading practices including: 1. establishing clear, appropriate, and measurable objectives; 2. articulating an assessment methodology and data gathering strategy; 3. developing a data analysis and evaluation plan to track pilot 4. identifying criteria for determining whether and how to scale the pilot and integrate it into overall efforts; and 5. ensuring two-way stakeholder communication through the pilot program. Through our review of the CIREN pedestrian pilot program documentation, we determined that NHTSA met most of the criteria for a well-developed pilot program, but not all. Specifically, NHTSA documented clear, appropriate, and measureable project objectives; identified an assessment methodology and data gathering strategy; developed a data analysis plan; and communicated with stakeholders. NHTSA, however, did not establish an evaluation plan that includes criteria to determine if the pilot program’s data collection and analysis protocol should or could be continued or expanded, once the data have been collected from the nine cases. Although NHTSA officials reported that they had a plan to review and evaluate individual cases, NHTSA does not have an evaluation plan for the pilot program that includes criteria or standards for identifying lessons learned or determining whether the new data collection and analysis procedures would satisfy data needs related to pedestrian’s injuries. NHTSA officials told us that they did not develop an evaluation plan or criteria for determining the success or scalability of the pedestrian pilot program because they were not required to create one. They also said they did not have enough information to tell if the pilot program should be integrated into overall efforts, although they expect the tools developed by the pilot to be incorporated into later efforts to increase the number of pedestrian crashes reviewed under the CIREN program. Pedestrian crash avoidance and crash mitigation safety features are commonly available on many 2019 model year vehicles offered in the United States, according to the 13 automakers we interviewed. As previously discussed, crash avoidance features rely on cameras or radar or both to detect a pedestrian and take action to avoid a crash. Crash mitigation generally involves use of pedestrian-friendly vehicle components (such as energy absorbing bumper components or hoods) that are designed to reduce the severity of injuries should a pedestrian be hit. The 13 automakers we interviewed responded that they, collectively, offered 262 model year 2019 vehicles for sale in the United States. Of those vehicle models, almost 60 percent included pedestrian automatic emergency braking as either a standard or an optional feature (see fig. 8). About 62 percent of their model year 2019 vehicles had some type of standard pedestrian crash mitigation feature. In total, 12 of 13 automakers that we interviewed responded that they offered one or more 2019 model year vehicles with pedestrian automatic emergency braking as either a standard or optional feature; similarly, 12 of 13 automakers told us they offered crash mitigation features in at least one of their 2019 model year vehicles. Some stakeholders we interviewed told us that a combination of crash avoidance and crash mitigation features can be effective in minimizing pedestrian injury. For example, NHTSA officials told us that crash avoidance features, such as pedestrian automatic emergency braking can slow a vehicle to a speed where it will be less damaging to a pedestrian once struck, and if the vehicle also has crash mitigation features the impact of the crash can be further mitigated. We found that almost half of 2019 vehicle models had some combination of both pedestrian automatic emergency braking and crash mitigation features. For example, about 47 percent of 2019 vehicle models had pedestrian automatic emergency braking as either standard or optional equipment along with crash mitigation features, such as softer hoods. However, 24 percent of vehicle models had neither of these (see fig. 9). Officials from the 13 automakers we interviewed identified a variety of factors that influenced their decisions to offer vehicles with pedestrian safety features in the United States. These include a desire to achieve high safety ratings for their vehicles, as well as the following: New car assessment programs: New car assessment programs in the United States and other countries also influence why automakers may offer pedestrian safety features. For example, officials from nine of 10 automakers that responded to this question in our interview replied that Euro NCAP was a major factor to them in providing pedestrian safety features, while seven of 10 automakers responded that JNCAP was a major factor. In contrast, three of 11 automakers responded that the U.S. NCAP was a major factor in their decisions to offer vehicles with pedestrian safety features. As previously discussed, the United States, unlike the European Union and Japan, does not incorporate pedestrian safety tests into its NCAP. Independent safety testing: Independent safety testing was also a factor in why automakers may offer pedestrian safety features on vehicles. For example, officials from five automakers said that they considered IIHS safety ratings to be a major factor in their company’s decision to offer pedestrian safety features on vehicles sold in the United States. As previously discussed, IIHS began testing pedestrian crash avoidance systems on 2018 and 2019 vehicles. These tests are known as pedestrian automatic emergency braking tests and in 2020 IIHS began using the results to help determine their Top Safety Pick awards. Officials from two automakers said a company’s goal is to earn an IIHS top safety-pick rating for each of their models. Cost: Cost appeared to be less of a factor influencing whether pedestrian safety features were offered on vehicles. Officials from seven of eight automakers who responded to this question replied that costs either were a minor factor, or did not apply, in their decisions to offer vehicles with pedestrian safety features. However, officials from four automakers told us that, in general, while customers want safer vehicles, automakers have to consider what safety features could be included without increasing the overall cost. Further, one automaker’s representative said that as more manufacturers and customers are buying crash avoidance systems the costs are decreasing. The future availability of crash avoidance features may depend on several factors. Specifically, in 2016, 20 automakers voluntarily committed to making automatic emergency braking systems standard in vehicles sold in the United States by 2022. Officials from three automakers said that they planned to incorporate pedestrian automatic emergency braking into their vehicles’ automatic emergency braking systems as part of this commitment. Another factor is customer demand. One automaker said that the number of models that include pedestrian safety features in the future would depend on consumer demand or changes in regulation. Officials from another automaker said their customers often ask for features they see in Europe and ask why such features are available there but not in the United States. The auto industry officials we interviewed identified benefits and challenges with commonly available pedestrian safety features. Benefits of crash avoidance systems include the potential of eliminating or reducing car-to-pedestrian accidents. For example, officials from six automakers said that crash avoidance features were more effective than crash mitigation features because the purpose of crash avoidance features is to prevent the collision from occurring in the first place. Almost half of the automakers we interviewed (six of 13), however, reported that a primary challenge with a camera-based pedestrian automatic emergency braking system was the camera’s ability to work in low lighting or poor weather. Recently issued research has raised questions about the overall effectiveness of crash avoidance systems. In October 2019, AAA reported that based on its own assessment, some vehicles’ pedestrian safety systems were inconsistent at either slowing down or stopping a vehicle to avoid hitting a pedestrian. Specifically, the association reported that none of the crash avoidance systems on the four vehicles they tested worked in dark conditions. Auto industry officials also identified benefits and challenges with pedestrian crash mitigation features. For example, 12 of 13 automakers reported that crash mitigation features have the overall benefit of reducing the risk or severity of pedestrian injuries. Officials from eight automakers, however, said that the current federal bumper standard created challenges to offering softer, more pedestrian-friendly bumpers in the United States. Officials from the eight automakers said they offered softer bumpers in Europe or elsewhere—where there is no similar bumper standard—but do not offer softer bumpers in the United States. NHTSA officials told us the current bumper standard is primarily a cost savings standard in that it is intended to reduce repair costs and not necessarily to offer safety protection for vehicle occupants. NHTSA officials told us trade-offs are required to establish a bumper standard that addresses pedestrian safety, yet minimizes bumper damage and repair costs. NHTSA officials told us they are in the process of reevaluating the bumper damageability standard, as part of a Notice of Proposed Rulemaking, which they expect to publish in 2020. Appendix III discusses the benefits and challenges of commonly available pedestrian safety features. NHTSA has considered pedestrian safety for many years by conducting research, considering implementation of global regulations for pedestrian crash mitigation tests, and proposing pedestrian crash avoidance and mitigation tests for NCAP (see fig. 10). NHTSA’s last substantial update of NCAP was in July 2008 (with changes effective for model year 2011 vehicles). This update established additional crash tests and technical standards to protect vehicle occupants, but did not include pedestrian safety tests. In the past 10 years, NHTSA has considered but has not yet initiated a rulemaking process related to international standards for crash mitigation tests, among other actions. For example, in 2008, the United States along with other countries approved a United Nation’s international standard for pedestrian crash mitigation tests. This international standard, if implemented in the United States in a domestic regulation, would require U.S. vehicles to meet minimum performance requirements in pedestrian crash mitigation tests. The United States approved the international standard in 2008; however, NHTSA has yet to initiate a rulemaking to implement it either as part of the FMVSS or adopt it as a testing protocol through NCAP. According to NHTSA officials, implementation of the standard would require NHTSA to initiate a regulatory proceeding. Although the United States formally agreed to the standard more than 10 years ago, NHTSA officials told us that the rulemaking initiative is classified as a long-term action and that there is no timeline for such a rulemaking to implement pedestrian crash mitigation requirements. NHTSA has also conducted a range of research on pedestrian crash avoidance and mitigation tests. Specifically, NHTSA has published, contributed to, or sponsored over 55 studies and presentations on pedestrian safety issues since 2008, and NHTSA officials provided information stating that NHTSA has spent over $8.4 million to research pedestrian safety, including pedestrian automatic emergency braking and passive safety features from 2008 through 2019. In addition, officials stated that NHTSA has conducted a number of additional studies related to pedestrian safety, studies that NHTSA is currently reviewing for final publication, though officials did not provide expected publication dates. NHTSA officials told us this research serves as a body of work that supports and facilitates agency decisions and policies with respect to pedestrian safety. NHTSA’s pedestrian safety research has focused on several key issues, including developing objective test protocols and reliable test instruments for inclusion in NCAP and assessing the potential safety benefits. NHTSA officials told us there are three important elements associated with any safety tests (including pedestrian safety tests). These elements are (1) creating test protocols that measure a vehicle’s safety performance objectively, (2) validating test instruments that measure human injury, and (3) estimating the potential safety benefit of the tests. NHTSA’s pedestrian safety research includes work related to all three of these elements, as follows: Objective Test Protocols. One NHTSA study developed objective test protocols to evaluate the effectiveness of pedestrian crash avoidance systems based on analyses of crash scenarios from real- world crash data. Another NHTSA study applied pedestrian crash mitigation test protocols used by Euro NCAP to the U.S. vehicle fleet. NHTSA found that the European protocols could be used to assess the pedestrian safety performance of vehicles in the United States, but that the performance of different U.S. vehicle types could vary. Specifically, NHTSA found that “global platform” vehicles (i.e., models that include a U.S. and European variant of the same vehicle) offered more pedestrian safety than vehicles that are only marketed in the United States. Valid Test Instruments. NHTSA has been a key contributor in the development of pedestrian test instruments. For example, NHTSA has presented information on mannequins for evaluating the repeatability and accuracy of pedestrian crash avoidance systems, concluding that mannequins should be durable, realistic, and comparable in size and movement to humans. In addition, NHTSA found there are instruments that produce repeatable and reproducible measurements of pedestrian head, upper leg, and lower leg injuries on tests. Potential Safety Benefits. NHTSA has studied the potential benefits of pedestrian crash avoidance, and estimated that these technologies could reduce the number of annual vehicle-pedestrian crashes by between 620 and 5,000, and reduce the number of annual fatal vehicle-pedestrian crashes by between 110 and 810. NHTSA has also reported that Europe and Japan have responded to the high proportion of pedestrian fatalities compared to all traffic fatalities by including pedestrian protection in their respective NCAPs and requiring pedestrian protection through regulation. According to NHTSA, these actions have likely contributed to a downward trend in pedestrian fatalities in Europe and Japan. Further, an international study found that including pedestrian safety testing in consumer testing programs has real world benefits by reducing pedestrian fatalities and injuries. For example, a European study concluded that vehicles that score well in Euro NCAP pedestrian crash mitigation tests are less likely to severely injure pedestrians. As previously noted, Euro NCAP and JNCAP have included pedestrian crash mitigation tests since 1997 and 2003, respectively, and both Euro NCAP and JNCAP incorporated pedestrian crash avoidance tests in 2016. In December 2015, NHTSA proposed pedestrian crash avoidance and mitigation safety tests for NCAP by publishing a Request for Comments notice in the Federal Register. In the 2015 Request for Comments, NHTSA indicated that including these tests in NCAP could lead to a decrease in vehicle-pedestrian crashes and resulting pedestrian injuries and fatalities. In this request, NHTSA also reported that it believed the greatest gains in highway safety in coming years would result from widespread application of crash avoidance technologies and that its proposed safety tests for crash avoidance technologies, including pedestrian detection and automatic emergency braking, met NHTSA’s four prerequisites for updating NCAP. Those four prerequisites include that: a safety need is known or capable of being estimated; vehicle and equipment designs exist (or are anticipated in prototype design) that are capable of mitigating the safety need; a safety benefit is estimated based on the anticipated performance of the existing or prototype design; and a performance-based, objective test procedure exists to measure the ability of the vehicle technology to mitigate the safety issue. With regard to crash mitigation tests, NHTSA reported that it intended to use the Euro NCAP test procedures rather than those used in Japan because the European fleet make-up, including vehicle sizes and classes, is more similar to the U.S. fleet. NHTSA also reported in its 2015 Request for Comments that including pedestrian crash mitigation tests in NCAP is necessary to stimulate improvements in pedestrian crashworthiness in new vehicles sold in the United States. NHTSA, however, did not state in its 2015 Request for Comments whether the proposed crash mitigation tests met NHTSA’s prerequisites for updating NCAP, as it had for the crash avoidance tests. The proposed changes in the 2015 Request for Comment were to take effect for model year 2019 vehicles. In response to the 2015 Request for Comment, NHTSA officials told us they received 290 comments, 31 of which addressed pedestrian safety. According to the officials, the comments received were generally supportive of including pedestrian safety testing in NCAP, and commenters proposed that the U.S. tests should be consistent, or harmonized, with the tests already conducted by Euro NCAP. NHTSA officials also noted that some commenters expressed concern with test tools and proposed test scenarios. Since the 2015 proposal to include pedestrian tests in NCAP, NHTSA has continued to solicit updated information in additional Requests for Comments. Most recently, in October 2019 NHTSA announced it would seek comment on NCAP updates in 2020, and in November 2019, NHTSA requested comments on draft research test procedures for forward and rear pedestrian crash avoidance, among other technologies. However, NHTSA stated that its draft test procedures were developed for research purposes only, and the fact that it was soliciting comments on these procedures was not an indication that it would then, or at any time in the future, initiate a rulemaking related to that technology or include that technology in NCAP. NHTSA officials told us there are many actions that go into their decision- making on whether to update NCAP and that this decision-making process can take years. These actions include such things as reviewing data, ensuring the reliability and repeatability of proposed tests by validating protocols at multiple independent test laboratories, and conducting market research to obtain consumer input. In addition, NHTSA officials told us that it also uses its four prerequisites for updating NCAP, and while these prerequisites are not required by law, they represent good governance practices and are in consumers’ best interest. However, since NCAP is considered a consumer testing information program and not a regulation, there are no particular requirements for when or how final decisions would be made as to whether pedestrian safety should or should not be included in NCAP. NHTSA officials told us that ultimately the NHTSA Administrator decides whether to go forward with changes to NCAP. Although NHTSA officials told us NCAP is not a regulation, they said NHTSA generally follows the processes in the Administrative Procedure Act for informal rulemaking to update NCAP. This process includes a notice, comment, and decision process in the Federal Register for transparency. NHTSA, however, has not used this process to communicate to stakeholders the additional steps that it must take before it can make its decision on NCAP testing. In addition, although NHTSA requested and received numerous comments on including pedestrian safety tests in NCAP in 2015, as of April 2020, it has yet to respond to those comments. Leading practices for program management emphasize the importance of milestones and decision points, documentation, and clearly communicating to external stakeholders. The Project Management Institute, Inc., The Standard for Program Management stresses the importance of program management plans that align with organizational goals and objectives. Elements of such plans are to provide a roadmap that identifies such things as milestones and decision points to guide programs forward. In addition, Standards for Internal Control in the Federal Government, state that entities should externally communicate the necessary quality information to achieve the entity’s objectives. In particular, entities should communicate to external stakeholders significant matters related to risks or changes. These standards also state that documentation is necessary for design, implementation, and operating effectiveness. Compared to these leading practices, NHTSA’s process does not provide documentation of the process, decision points, or milestones to guide the program. For example, NHTSA officials could not provide us with documentation as to how it determined that the pedestrian crash avoidance tests proposed in 2015 met the four prerequisites, or how the proposed crash mitigation tests compared to the prerequisites. Other NCAPs have used various methods for documenting their process for updating their testing. For example, Euro NCAP uses a roadmap to communicate to stakeholders the planned changes for NCAP tests, the timeline of steps toward the changes, and when those changes will be effective. Officials from Euro NCAP told us the test and assessment protocols are developed in conjunction with working groups made up of automakers, equipment suppliers, test facilities, and Euro NCAP member organizations. Further, officials told us the working groups and roadmaps provide automakers with the opportunity to provide real-time input and obtain information to support their investment decisions. The lack of a documented overall process for updating NCAP affects NHTSA’s ability to achieve NCAP’s goals to provide manufacturers an incentive to improve the safety performance of new vehicles and to assist consumers with their vehicle purchasing decisions. Specifically, without a transparent process for NHTSA’s decision-making on NCAP, automakers lack information on NHTSA’s progress in evaluating proposed changes— such as those offered in the 2015 Request for Comment—and the timing of the implementation of any specific testing procedures. This is particularly important because automakers need quality information to make investments to support the development and deployment of new technologies and equipment in their product lines to meet testing requirements. For example, representatives from one automaker told us that vehicle design is a 6 to 8 year product cycle and that if NHTSA decides to implement certain tests in the middle of that cycle, it would be difficult and costly to make changes. Without a clearly documented process for making changes to NCAP, including established criteria and milestones for decisions, automakers and the public lack clarity on NHTSA’s plans for improving vehicle safety to inform investment and purchasing decisions. NHTSA has yet to make or communicate a decision as to whether it intends to include pedestrian safety tests in NCAP. As discussed above, NHTSA has conducted extensive research and requested comments on pedestrian crash avoidance and mitigation tests in 2013 and 2015. Although NHTSA reported in 2015 that these tests could lead to a decrease in vehicle-pedestrian crashes and resulting pedestrian injuries and fatalities, it has yet to make or communicate a decision about the future of NCAP in relation to pedestrian safety to stakeholders. Nine of 13 automakers we interviewed told us that a lack of communication from NHTSA on its plan for addressing pedestrian safety issues has presented a challenge to them, often because they require long lead times to develop, test, and launch new technologies. Leading practices for program management also stress the importance of communication with stakeholders and that effective stakeholder communications are key to executing program endeavors, addressing risks, and, ultimately, delivering benefits. Specifically: The Project Management Institute, Inc., The Standard for Program Management stresses the importance of managing external communications, stating that communication provides critical links for successful decision making. It also stresses the importance of providing decision-making stakeholders with adequate information to make the right decisions at the right time in order to move programs forward. Standards for Internal Control in the Federal Government states that entities should identify, analyze, and respond to risks related to achieving the defined objectives and should externally communicate the necessary quality information to achieve the entity’s objectives. As discussed above, these standards also state that management should externally communicate quality information to external stakeholders significant matters related to risks or changes. Further, the statute underlying NCAP requires NHTSA to communicate certain vehicle safety information to the public. Specifically, DOT is to provide the public with information on crash avoidance, crashworthiness, and damage susceptibility. Such information is to be provided in a simple and understandable form to allow comparison among vehicles to assist a consumer in buying a new car. NHTSA officials told us that it has not made or communicated a decision as to whether it will include pedestrian safety testing in NCAP because administration priorities have shifted since publication of the 2015 Request for Comments. Specifically, NHTSA officials told us that the agency drafted technical specifications and testing protocols for pedestrian safety tests for NCAP and posted those tests to its public web site in January 2017. After the administration changed, however, those specifications were withdrawn and not published in the Federal Register. NHTSA officials told us that, since that time, the agency has sought to conduct additional review before final decisions could be made. Although the policy decision as to whether to include pedestrian safety tests in NCAP ultimately resides within NHTSA’s discretion, NHTSA’s lack of a decision and its related rationale limits NHTSA’s ability to address emerging safety risks and to meet its strategic objectives. Specifically, in the Department of Transportation’s Enterprise Risk Profile for 2019, NHTSA recognized that increases in roadway fatalities in general—and pedestrian fatalities in particular—represent one of the top strategic risk areas for the Department. The document states that to meet its objectives, NHTSA must focus on areas where there have been increases in road deaths, including pedestrian fatalities, and advance crash avoidance and mitigation technology to prevent crashes from occurring. NHTSA also recognized the importance of using a data-driven and systematic approach that is timely and complete when making decisions. In the absence of a decision on whether to include pedestrian safety testing in NCAP, and the rationale for that decision, stakeholders lack clarity on whether NHTSA is using all of the policy tools at its disposal to address emerging safety risks and to achieve its strategic objectives. The design of vehicles and the safety features they offer can play an important role in reducing the frequency and severity of pedestrian crashes. NHTSA’s pedestrian pilot program is an important step toward addressing data gaps on the relationship between vehicle characteristics and pedestrian injuries. Without an evaluation plan that includes criteria for assessing the scalability of the pilot program, however, NHTSA lacks the tools necessary to assess whether and how the pilot should be expanded into a more robust effort to inform NHTSA’s understanding of pedestrian injury mitigation efforts. Although NHTSA has recognized that the increase in pedestrian fatalities presents a risk to the safety of the nation’s roadways, it is not well positioned to address this risk through NCAP because NHTSA does not have a clear process for making changes to the program. Documenting and communicating the process for updating NCAP, with clear criteria and decision points, would enhance NHTSA’s accountability to key stakeholders—including Congress, automakers, and consumers—and help NHTSA communicate the important policy decision as to whether to include pedestrian safety tests in NCAP. Making and communicating a decision regarding pedestrian safety testing would give automakers clarity on whether NHTSA intends to establish performance standards and tests to evaluate the pedestrian safety features that are commonly available on new vehicle models. Communicating a decision and the rationale for that decision would provide transparency and accountability to the public. We are making the following three recommendations to NHTSA. The Administrator of NHTSA should document an evaluation plan for the Crash Injury Research and Engineering Network pedestrian pilot program that includes criteria for determining whether and how to scale the pilot program to ensure that the piloted data-collection and analysis procedures will address NHTSA’s data needs related to pedestrian injuries and vehicle characteristics. (Recommendation 1) The Administrator of NHTSA should document the overall process for making changes to NCAP, including established criteria and milestones for decisions, and share this process with external stakeholders. (Recommendation 2) The Administrator of NHTSA should decide whether to include pedestrian safety tests in NCAP and NHTSA should communicate this decision and rationale to relevant stakeholders and the public. (Recommendation 3) We provided a draft of this report to the Department of Transportation for review and comment. The department provided a written response, which is reprinted in appendix IV, and technical comments that we incorporated as appropriate in the report. The department concurred with all three of our recommendations. It described various activities NHTSA has underway related to pedestrian safety, including the CIREN study, a special study initiated this year to gather detailed data on a selection of fatal pedestrian crashes, and continuing research on pedestrian crash test procedures. Regarding Recommendation 2, the department stated that it has made its procedures to change NCAP transparent and inclusive of the public. Specifically, the department stated it has published and requested comment on its proposals in the Federal Register, as we described in this report. However, the department agreed that documenting the overall process on its website would generate increased public awareness of NCAP as a consumer safety tool. While such a step could increase public awareness of NCAP, we continue to believe that any steps taken to document the overall process for making changes to NCAP should also include established criteria and milestones for decisions to enhance NHTSA’s accountability to Congress, automakers, and consumers. We are sending copies of this report to the appropriate congressional committees, the Secretary of Transportation, and the Administrator of the National Highway Traffic Safety Administration. In addition, we will make copies available to others upon request, and the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834, or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report: (1) examines what is known about the relationship between motor vehicle characteristics and pedestrian injuries and fatalities, (2) describes approaches automakers have taken to address pedestrians’ safety and discusses stakeholders’ perspectives on these approaches, and (3) evaluates actions the National Highway Traffic Safety Administration (NHTSA) has taken to assess whether pedestrian safety testing should be included in its New Car Assessment Program (NCAP). For all of our objectives we reviewed pertinent federal statutes and regulations and applicable program documents. Our work covered the 2008 through 2018 timeframe, with 2018 being the most recent data available at the time of our analysis. We focused on motor vehicles as opposed to infrastructure (e.g., roadway design, highway lighting) or driver/pedestrian behavior. Although infrastructure and behavior may also contribute to pedestrian fatalities and injuries, the scope of this report was to assess motor vehicles and their role in pedestrian safety. We defined motor vehicles as passenger cars, sport utility vehicles, and light trucks and vans that were offered for sale in the United States. We excluded commercial vehicles, motorcycles, and buses. The intent was to include those vehicles that a typical consumer would purchase and the pedestrian safety features that may or may not be offered on such vehicles. Our scope also included gaining an understanding of pedestrian safety testing activities in Europe (European New Car Assessment Programme (Euro NCAP)) and Japan (Japan New Car Assessment Program (JNCAP)). We selected these programs since pedestrian safety testing is part of their NCAPs and some auto industry stakeholders identified them as being in the forefront of this type of testing. Both Europe and Japan began testing crash avoidance systems as part of their NCAPs in 2016. We interviewed officials with Euro NCAP, received a written response to questions from JNCAP, and obtained information on pedestrian safety testing from both organizations. To examine what is known about the relationship between vehicle characteristics and pedestrian injuries and fatalities, we analyzed data from three NHTSA databases for the period of 2008 through 2018: (1) Fatality Analysis Reporting System (FARS); (2) Crash Report Sampling Systems (CRSS); and (3) National Automotive Sampling System/General Estimates System (NASS/GES). To ensure the accuracy of our analysis we reviewed agency technical documentation related to these databases and ensured that our figures matched publicly available injury and fatality data contained in NHTSA publications such as its annual Traffic Safety Fact Sheets. FARS data are derived from a census of all fatal motor vehicle traffic crashes within the 50 states, Puerto Rico, and the District of Columbia and provide uniformly coded, national data on police reported fatalities. We analyzed FARS data to determine the total number of pedestrian fatalities each year as well as the number of pedestrian fatalities by vehicle age, vehicle body type, and vehicle travelling speed (speed just prior to the crash). These variables were selected based on our interviews of NHTSA officials and a review of relevant research about the relationship between pedestrian fatalities and motor vehicle characteristics. We also analyzed FARS data on the number of pedestrian fatalities by environmental characteristics such as type of roadway, light condition, and relationship to intersection, selecting these characteristics based on our interviews and research. CRSS is a sample of police reported motor vehicle crashes involving all types of motor vehicles, pedestrians, and cyclists that is used to develop national estimates of the number of injuries associated with motor vehicle crashes, among other things. The CRSS police crash report sample is selected in multiple stages to produce a nationally representative probability sample, and the target annual sample size is 50,000 police accident reports. We analyzed CRSS data from 2016 through 2018, the only years CRSS data were available, to better understand the estimated total number of pedestrian crashes as well as the estimated number of pedestrian crashes by vehicle age, vehicle body type, vehicle speed, and level of pedestrian injury severity. Similar to our analysis of FARS data, these variables were selected based on our interviews with NHTSA officials and a review of relevant research about the relationship between motor vehicle characteristics and pedestrian crashes. NASS/GES preceded CRSS and obtained its data from a nationally representative probability sample of police accident reports. We analyzed NASS/GES data from 2008 through 2015, the most recent years available within the database, to better understand historical trend data on the variables we analyzed in CRSS. Although NHTSA collected similar variables in CRSS and NASS/GES, differences in the sampling methodologies of each may contribute to differences in the estimated number of pedestrian crashes between 2008 through 2015 and 2016 through 2018 timeframes. We used agency technical documentation for CRSS and NASS/GES as well as guidance from NHTSA statisticians to estimate the sampling error associated with our estimates derived from CRSS and NASS/GES data. We express confidence levels of estimates derived from CRSS and NASS/GES data at the 95 percent confidence interval. This level means that we are 95 percent confident that the actual population values are within this interval. Additionally, for our analysis, we used CRSS and NASS/GES variables that included imputed values for items missing data on the estimated number of pedestrian crashes by vehicle age, vehicle body type, and pedestrian injury severity. We reviewed and assessed NHTSA technical documentation for their statistical imputation methodology and determined it was sufficiently reliable for us to make use of the vehicle age, body type, and injury severity variables with imputed data. In addition to analyzing NHTSA databases, we analyzed data from the Highway Loss Data Institute (HLDI), an organization affiliated with the Insurance Institute for Highway Safety (IIHS), to better understand how the U.S. vehicle fleet has changed, specifically between 2008 and 2018. HLDI collected and decoded vehicle identification numbers (VINs) for each model year between 1983 and 2018. For HLDI’s analysis, it used VINs from its member companies, among other sources, and information encoded in the VIN to determine the body styles for these VINs. According to HLDI, passenger cars include regular two-door models, regular four-door models, station wagons, minivans, sports models and luxury models, while SUVs are vehicles with conventional front-end constructions and large passenger and cargo areas which can be built on either heavy-duty chassis capable of off-road use or passenger car platforms. HLDI definitions for vehicle body type classifications differ from those used by NHTSA. According to HLDI officials, however, the classifications are comparable. For our analysis, we used these data to calculate the proportion of vehicles that were passenger cars, light trucks, or SUVs from 2008 through 2018. We also conducted interviews with federal government and non- governmental organizations about the relationship between vehicle related characteristics and pedestrian injuries and fatalities, as well as issues related to NHTSA’s pedestrian safety data and potential data gaps and limitations. To discuss NHTSA’s pedestrian safety data, we spoke with NHTSA officials from the Data Reporting and Information Division, Mathematical Analysis Division, and Vehicle Research and Test Center. We also spoke with officials from the National Transportation Safety Board, which conducts independent accident investigations and advocates for safety improvements, including those related to pedestrian safety and motor vehicles. Non-governmental organizations we spoke with included IIHS and major auto industry trade associations, such as the Alliance of Automobile Manufacturers, Association of Global Automakers, the Motor and Equipment Manufacturers Association, and the Automotive Safety Council. We also spoke with vehicle safety advocates, such as the Governors Highway Safety Association. These organizations were selected based on their relationship to the auto industry, referrals from other interviewees, and recent publications on pedestrian-motor vehicle safety issues. We also identified and reviewed studies either published or referenced by these organizations to better understand research related to pedestrian injuries and fatalities and motor vehicle characteristics. Where appropriate, we conducted a methodological review of these studies. Further, we spoke with academic researchers from six research centers across four universities with expertise in human-vehicle interaction and pedestrian-motor vehicle safety, including injury biomechanics and auto industry data analysis. These researchers were selected based on referrals from other interviewees and reviews of their organization’s websites to ensure that their research would be informative for our purposes. Although these organizations had, or have, relationships with NHTSA or the auto industry, we included them based on their expertise with issues related to our work. Based on these criteria we interviewed officials at the University of Virginia (Center for Applied Biomechanics); the Ohio State University (Center for Automotive Research; Injury Biomechanics Research Center); the University of North Carolina (Highway Safety Research Center), and the University of Michigan (University of Michigan Transportation Research Institute; International Center for Automotive Medicine). We conducted interviews with these researchers to better understand general information on the relationship between vehicle-related characteristics and pedestrian injuries and fatalities, uses and limitations of NHTSA data, and potential areas for further research. Results of our interviews are not generalizable to the universe of non-governmental organizations or researchers studying pedestrian-motor vehicle safety. We also spoke with automakers and equipment suppliers about pedestrian safety and data needs. The automakers and equipment suppliers were the same as those contacted about how automakers are addressing pedestrian safety (discussed below). Finally, we reviewed documents and interviewed NHTSA officials about the Crash Injury Research and Engineering Network (CIREN) and the associated CIREN pedestrian pilot program NHTSA recently initiated. This pilot will assess data collection approaches and methodologies for pedestrian injuries resulting from motor vehicle crashes. Specifically, we reviewed CIREN contract and methodology documents such as the 2016 CIREN Request for Proposal, 2018 CIREN Pedestrian Pilot Study Request for Proposal, Task Orders for CIREN centers participating in the pedestrian pilot study, CIREN Pedestrian Crash Process and Coding Manual, and the Pedestrian Crash Inclusion Criteria. We also interviewed NHTSA officials responsible for managing the CIREN program and the pedestrian pilot study. We assessed this program using criteria for designing successful pilot programs developed during prior GAO work. To describe the approaches automakers have taken to address pedestrian-motor vehicle safety and discuss stakeholder perspectives on these approaches, we contacted automakers that sell new vehicles in the United States. Specifically, NHTSA provided us with a list of 17 automakers that participated in the 2018 New Car Assessment Program. NHTSA officials told us they do not necessarily include automakers with low sales volumes in NCAP testing. As a result, to better ensure that we had a complete list of automakers that sell vehicles in the United States we compared the names on NHTSA’s listing to the membership lists of the Alliance for Automobile Manufacturers and the Association of Global Automakers—two major trade associations of the auto industry. Officials told us that between the two organizations we would account for most, if not all, of the automakers that sell new vehicles in the United States. Finally, we compared our list with 2018 market share data from Ward’s Automotive to identify the automakers with the highest U.S. sales. Based on our analysis, we identified 17 automakers to include in our work. However, during our contacts with automakers, we determined that one of the 17 automakers—Porsche—was part of the Volkswagen Group. Thus, our final review resulted in a total of 16 automakers to contact as part of our study (see table 2). Thirteen of the 16 automakers responded to our request for information. We developed a semi-structured interview instrument to collect information from the automakers. This instrument focused on the approaches that automakers took to address pedestrian-motor vehicle safety. The semi-structured interview instrument was peer reviewed by an independent survey specialist and pretested with two automakers before we began collecting data. Based upon on their responses, we revised and clarified the semi-structured interview instrument. In total, 13 of 16 automakers completed and submitted the semi-structured interview instrument. Those 13 automakers represented approximately 70 percent of new vehicle sales in the United States for 2018. The interview instrument asked automakers to identify pedestrian safety features on their 2019 model year vehicles, as these vehicles would have the most recent pedestrian safety features available at the time of our work. Although 12 of the 13 automakers did not respond in full to all the questions on the semi-structured interview instrument, we obtained additional information through telephone and in-person interviews conducted from May 2019 through October 2019. The results of these interviews are not generalizable to the universe of automakers that may sell vehicles in the United States. Upon completion of all the interviews, a GAO methodologist compiled the individual responses from each of the 13 automakers into a database. We used this database to perform a qualitative content analysis to identify common themes and the frequency with which the automakers identified certain issues related to pedestrian safety. A GAO analyst independently verified the themes and certain other information we received from the automakers to ensure accuracy and completeness. We also used semi-structured interview instruments to obtain information on stakeholders’ perspectives on the approaches automakers have taken to pedestrian safety. For purposes of this report, we define stakeholders as automakers, auto equipment suppliers, and auto industry trade associations. These organizations develop or deploy pedestrian safety technology in motor vehicles, or, in the case of the trade associations, are knowledgeable about the legal and regulatory issues related to pedestrian safety and the auto industry. In addition to interviewing 13 automakers, we interviewed officials from five auto equipment suppliers and four auto industry trade associations (see table 3). The five auto equipment suppliers included in our work were identified with the assistance of the Motor and Equipment Manufacturers Association, a trade association for auto industry suppliers. The organization provided us the names of seven equipment suppliers, five of which agreed to participate in our semi- structured interviews. In general, these equipment suppliers develop or produce equipment used in motor vehicle crash avoidance or crash mitigation systems. The semi-structured interview instrument asked questions about such things as crash avoidance and crash mitigation technology and the benefits and challenges of this technology. We did not assess the effectiveness of these features. Additionally, we interviewed officials from four auto industry trade associations. We conducted telephone and in-person interviews with these stakeholders from March 2019 through September 2019. In addition to automakers, equipment suppliers, and auto industry trade associations, we also interviewed NHTSA and IIHS about crash avoidance and crash mitigation technology and reviewed applicable federal regulations related to pedestrian safety. These include federal headlight and bumper standards. We also reviewed an October 2018 Notice of Proposed Rulemaking in which NHTSA agreed to evaluate proposed amendments to current federal motor vehicle headlight requirements. We discussed with NHTSA the federal headlight and bumper standards and how these relate to pedestrian safety, as well as any potential changes to these standards to better accommodate pedestrian safety. Lastly, we reviewed an October 2019 IIHS press release and an October 2019 American Automobile Association study discussing the results of pedestrian crash avoidance tests each organization performed. To assess NHTSA’s actions related to pedestrian safety and NCAP, we reviewed applicable federal laws and regulations related to vehicle safety as well as documents published in the Federal Register, such as Requests for Comments, soliciting comments on proposed NCAP changes related to pedestrian safety. NHTSA provided a high-level summary of comments received from Requests for Comments issued in 2015 and 2018 that we reviewed. We reviewed selected comments and supporting documents submitted to NHTSA as part of the docket in support of the Requests for Comment, such as those provided by auto industry trade associations, automakers, and auto equipment suppliers. We also reviewed program documents discussing how NHTSA assesses new car safety, performs NCAP safety tests, and reports the results to the public. Further, we reviewed over 55 studies and presentations on the agency’s work related to pedestrian safety. NHTSA highlighted 22 of these reports and presentations as being representative of the body of research that supported and facilitated agency decisions and policies with respect to pedestrian safety, including the 2015 and 2018 Requests for Comments. We reviewed the 22 reports and presentations and determined that 14 met our inclusion criteria, in that the reports and presentations were focused on potential pedestrian safety tests and their applicability to the U.S. vehicle fleet, the use of various test instruments, and the potential safety effects associated with technologies intended to avoid and mitigate crashes. Where appropriate, we conducted a methodological review of these studies. In addition, NHTSA officials provided additional studies after our interviews, which we also reviewed. To better understand pedestrian safety testing and issues related to incorporating such testing into NCAP, we visited NHTSA’s Vehicle Research and Test Center in East Liberty, Ohio. We interviewed officials there about NHTSA pedestrian safety research and how it supported NHTSA’s proposed pedestrian safety tests for NCAP. We also discussed the applicability of pedestrian safety tests to the U.S. vehicle fleet, including tests used by Euro NCAP. During our visit, we observed examples of a pedestrian crash mitigation test for lower leg injury and a rear-facing pedestrian crash avoidance test. We reviewed NHTSA’s budget documentation on pedestrian safety research from fiscal year 2008 to 2019, the most recent year for which data were available. We also visited and discussed pedestrian safety issues with officials of IIHS’ Vehicle Research Center in Ruckersville, Virginia. We observed a forward-facing pedestrian crash avoidance test. Further, we interviewed NHTSA officials about such things as the process for making changes to NCAP and activities associated with this process, documentation of this process, how NCAP changes are communicated to stakeholders, and NHTSA plans for determining whether to incorporate pedestrian safety tests in NCAP. Finally, we interviewed automakers, auto industry equipment suppliers, and IIHS about incorporating pedestrian safety tests into NCAP. To understand how other NCAPs address pedestrian safety, we interviewed officials from Euro NCAP and received written responses from JNCAP to a set of questions we sent them. We also reviewed supporting documents from both Euro NCAP and JNCAP on pedestrian crash avoidance and mitigation tests they perform and how such tests are scored when determining star ratings. Further, we discussed with Euro NCAP how it works with the auto industry to test vehicles and to develop future changes to Euro NCAP, including the Euro NCAP roadmap. We also reviewed selected international studies related to the real-world benefits of pedestrian safety testing performed by Euro NCAP. We determined those studies to be sufficiently reliable for our purposes. To assess how NHTSA’s process for making changes to NCAP compares to leading practices, we reviewed the Project Management Institute, Inc., The Standard for Program Management, and GAO’s Standards for Internal Control in the Federal Government. The Project Management Institute’s standards are utilized worldwide and provide guidance on how to manage various aspects of projects, programs, and portfolios. In particular, according to The Standard for Program Management, this standard provides guidance that is generally recognized to support good program-management practices for most programs, most of the time. We conducted our work from February 2019 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix contains additional information on pedestrian fatalities and the estimated number of pedestrians injured from 2008 through 2018. Although we included much of our pedestrian fatality analysis in the report, this appendix includes data on the number of pedestrian fatalities involving particular light conditions and relationships to intersections— environmental factors relevant to pedestrian crashes—as well as data on vehicle body types (see fig. 11, 12, and 13). We used data from the National Highway Traffic Safety Administration’s (NHTSA) Fatality Analysis Reporting System (FARS) to compile information on pedestrian fatalities. The following figures show information about the estimated number of pedestrians injured from 2008 through 2018 (see figs. 14, 15, 16, 17, and 18). These figures show pedestrians injured by age of the striking vehicle, body type of vehicle, reported speed of the vehicles, and estimated number of pedestrians with serious or fatal injuries. We used data from NHTSA’s Crash Report Sampling System (CRSS) for years 2016 through 2018, and National Automotive Sampling System/General Estimates Survey (NASS/GES) for years 2008 through 2015 to compile information on pedestrians injured. Within CRSS and NASS/GES databases, we specifically analyzed data on pedestrians injured by vehicle related characteristics such as the age, body type, and speed of vehicles that struck and injured pedestrians, as well as the estimated number of severe and fatal pedestrians injured. As part of our analysis on how automakers are addressing pedestrian safety through crash avoidance and crash mitigation technologies, we obtained the views of 13 automakers and five auto equipment suppliers. As discussed below, auto industry officials provided their views on the benefits and challenges of commonly available crash avoidance and crash mitigation technologies. Automaker and auto equipment supplier officials identified various benefits and challenges with pedestrian crash avoidance features. For example, 12 of 13 automakers reported and two of five auto equipment suppliers said that crash avoidance features have the overall potential benefit of eliminating or reducing car-to-pedestrian accidents. The Highway Loss Data Institute reported in 2017 that one automaker’s pedestrian automatic emergency braking system reduced pedestrian- related bodily injury liability claims by 35 percent compared to other vehicles manufactured by that automaker. In addition, the automaker itself found that, in Japan, its vehicles equipped with the system experienced 60 percent fewer accidents with injury compared to its vehicles without the system. Officials from automakers and auto equipment suppliers we interviewed also identified challenges with pedestrian crash avoidance technologies. Specifically, stakeholders cited some distinctions between the performance of camera-based and radar-based pedestrian automatic emergency breaking systems. Almost half of the automakers we interviewed (six of 13) reported that a primary challenge with a camera- based pedestrian automatic emergency braking system was the camera’s ability to work in low lighting and poor weather. As previously noted in this report, about 75 percent of all reported pedestrian fatalities occurred in 2018 after dark. In contrast, several automakers stated that radar based pedestrian detection systems are not dependent on light to function, but that they are less effective at identifying pedestrians than camera-based systems. Officials from another automaker said manufacturers have attempted to offset the challenges of cameras and radar by developing “fusion” systems (combination of camera and radar). These officials said, however, these systems add complexity and processing time to the technology because the system must manage two separate functions that must be processed together to identify a pedestrian. Officials from automakers said that a challenge affecting both camera- and radar-based systems was limiting the occurrence of false positives, or the activation of these systems when they are not required. Recently issued research has raised questions about the overall effectiveness of crash avoidance systems. In October 2019, the American Automobile Association (AAA) reported that, based on its own assessment, some vehicles’ pedestrian safety systems were inconsistent at either slowing down or stopping a vehicle to avoid hitting a pedestrian. For example, AAA reported that dark conditions could affect the effectiveness of available pedestrian detection systems and that none of the crash avoidance systems on the four vehicles they tested worked in dark conditions. Automaker officials told us that the performance of crash avoidance systems could be improved through updates to current vehicle headlight standards. Specifically, officials from four automakers indicated that the National Highway Traffic Safety Administration (NHTSA) should update federal standards for headlights to permit the use of adaptive driving beam headlights on new vehicles. Adaptive driving beam headlights are currently in use in European and other countries, and are different from the combination high- and low-beam systems used in the United States. In general, adaptive driving beam headlights use advanced sensors and computing technology to shape the headlamp beams to provide enhanced illumination of unoccupied portions of the road and avoid glaring other vehicles. In October 2018, NHTSA published a Notice of Proposed Rulemaking in which it tentatively concluded that federal standards for headlights do not permit adaptive driving beam systems because those systems would not comply with some of the standards. NHTSA, however, has said adaptive driving beam headlights have the potential to create significant safety benefits in avoiding collisions with pedestrians, cyclists, animals, and roadside objects by providing additional front-end illumination. Five automakers we interviewed said that they offer adaptive driving beam headlamps as a crash avoidance technology on their vehicles sold in other countries. In its October 2018 Notice of Proposed Rulemaking, NHTSA sought public comment on amending federal standards to allow the use of adaptive driving beam systems in response to a petition from an automaker. NHTSA officials said that it is in the process of developing a final rule but did not have a period for when it would be issued. Another challenge for crash avoidance systems is the federal standard for bumpers. As previously discussed in this report, this standard requires that vehicles, including their bumpers, meet specified damage criteria when bumpers are hit at 2.5 miles-per-hour (mph). Officials from five automakers said that this standard presented challenges with the placement of crash avoidance sensors. On some vehicles, crash avoidance sensors are placed in the same area where the vehicles are tested for compliance with the bumper standard. As a result, the test could damage or destroy the crash avoidance sensor. Two automaker officials told us that they have addressed this challenge by relocating the sensors to another part of the vehicle to avoid conflicts with the bumper standard. NHTSA officials told us they are in the process of reevaluating the bumper damageability standard, including the placement of sensors, as part of a Notice of Proposed Rulemaking, which they expect to publish in early 2020. Officials from automakers and auto equipment suppliers we interviewed identified benefits and challenges for pedestrian crash mitigation features. For example, 12 of 13 automakers reported and one of five auto equipment suppliers said that pedestrian crash mitigation features have the overall benefit of reducing the risk or severity of pedestrian injuries. Officials from four automakers, however, said that crash mitigation features do not protect pedestrians from the secondary impact of an accident, such as the residual injuries from hitting the pavement. Additionally, officials from six automakers said that crash avoidance features were more effective than crash mitigation features because the purpose of crash avoidance features is to prevent the collision from occurring in the first place. Similar to crash avoidance, the federal bumper standard may also affect crash mitigation systems. Officials from eight automakers said that the bumper standard created challenges to offering additional crash mitigation features in the United States, such as softer, more pedestrian friendly bumpers. Officials from the eight automakers said they offered softer bumpers in Europe or elsewhere—where there is no similar bumper standard—but do not offer softer bumpers in the United States. Some stakeholders told us the current bumper standard runs counter to pedestrian safety, and softer bumpers would help mitigate the severity of pedestrian injuries. Similarly, NHTSA officials told us the current bumper standard is primarily a cost savings standard in that it is intended to reduce repair costs and not necessarily to offer safety protection for vehicle occupants. NHTSA officials told us that establishing a bumper standard that addresses pedestrian safety, yet minimizes bumper damage and repair costs requires tradeoffs. The officials told us as part of the Notice of Proposed Rulemaking it is reviewing the broader damageability requirement. Andrew Von Ah, Director, (202) 512- 2834 or vonaha@gao.gov In addition to the contact named above, Matt Barranca (Assistant Director), Richard Jorgenson (Analyst-in-Charge), Carl Barden, Namita Bhatia-Sabharwal, Melissa Bodeau, Breanne Cave, Michelle Everett, Susan Fleming, Geoff Hamilton, Hannah Laufe, Regina Morrison, Joshua Ormond, Terry Richardson, and Michael Steinberg made significant contributions to this report.", "summary": "In 2018, about 6,300 pedestrians—17 per day—died in collisions with motor vehicles in the United States, up from about 4,400 in 2008. Many factors influence pedestrian fatalities, including driver and pedestrian behavior. Vehicle characteristics are also a factor. NHTSA tests and rates new vehicles for safety and reports the results to the public through its NCAP. Currently, pedestrian safety tests are not included in NCAP. This report examines: (1) what is known about the relationship between vehicle characteristics and pedestrian fatalities and injuries, (2) approaches automakers have taken to address pedestrian safety, and (3) actions NHTSA has taken to assess whether pedestrian safety tests should be included in NCAP. GAO analyzed data on pedestrian fatalities and injuries from 2008 through 2018 (the most recent available data); reviewed NHTSA reports; and interviewed NHTSA officials. GAO also obtained information about pedestrian safety features from 13 automakers that represented about 70 percent of new vehicle sales in the United States in 2018, and compared NHTSA's actions with leading program management practices. National Highway Traffic Safety Administration (NHTSA) data show that certain vehicle characteristics related to age, body type, and the speed of the vehicle at the time of the crash are associated with increases in pedestrian fatalities from 2008 to 2018. Specifically, the number of pedestrian fatalities during this time period increased more for crashes involving vehicles that were: 11 years old or older compared to newer vehicles, sport utility vehicles compared to other passenger vehicles, and traveling over 30 miles per hour compared to vehicles traveling at lower speeds. GAO also found that NHTSA does not consistently collect detailed data on the type and severity of pedestrian injuries, but began a pilot program in 2018 to improve its data collection efforts. NHTSA, however, lacks an evaluation plan with criteria to assess whether to expand the pilot program, as called for in leading practices. As a result, NHTSA lacks information to determine how and whether it should expand the pilot to meet the agency's data needs. Automakers offer a range of approaches to address pedestrian safety. For example, pedestrian crash avoidance technologies use cameras or radar to detect an imminent crash with a pedestrian and engage a vehicle's brakes to avoid a crash. GAO found that about 60 percent of the model year 2019 vehicles offered in the United States by 13 automakers had pedestrian crash avoidance technologies as standard or optional equipment. In 2015 NHTSA proposed pedestrian safety tests for its New Car Assessment Program (NCAP), but NHTSA has not decided whether it will include such tests in the program. NHTSA has reported that crash avoidance technologies could lead to a decrease in pedestrian fatalities. Nine automakers that GAO interviewed reported that NHTSA's lack of communication about pedestrian safety tests creates challenges for new product development. NHTSA has also not documented a clear process for updating NCAP with milestones for decisions. NHTSA officials said that updating NCAP involves many actions and can take years. However, absent a final decision on whether to include pedestrian safety tests in NCAP and a documented process for making such decisions, the public lacks clarity on NHTSA's efforts to address safety risks. GAO is recommending that NHTSA (1) develop an evaluation plan with criteria for expanding its pilot program, (2) make and communicate a decision about whether to include pedestrian safety tests in NCAP, and (3) document the process for making changes to NCAP. The Department of Transportation concurred with our recommendations.", "document_type": "gao"}
{"report": "Our experience has shown that the key elements needed to make progress in high-risk areas are top-level attention by the administration and agency leaders grounded in the five criteria for removal from the High-Risk List, as well as any needed congressional action. The five criteria for removal that we issued in November 2000 are as follows: Leadership commitment. Demonstrated strong commitment and top leadership support. Capacity. Agency has the capacity (i.e., people and resources) to resolve the risk(s). Action plan. A corrective action plan exists that defines the root cause, solutions, and provides for substantially completing corrective measures, including steps necessary to implement solutions we recommended. Monitoring. A program has been instituted to monitor and independently validate the effectiveness and sustainability of corrective measures. Demonstrated progress. Ability to demonstrate progress in implementing corrective measures and in resolving the high-risk area. Starting in our 2015 update, we added clarity and specificity to our assessments by rating each high-risk area’s progress on the five criteria and used the following definitions: Met. Actions have been taken that meet the criterion. There are no significant actions that need to be taken to further address this criterion. Partially met. Some, but not all, actions necessary to meet the criterion have been taken. Not met. Few, if any, actions towards meeting the criterion have been taken. We are removing two areas—DOD Supply Chain Management and Mitigating Gaps in Weather Satellite Data—from the list due to the progress that was made in addressing the high-risk issues. As we have with areas previously removed from the High-Risk List, we will continue to monitor these areas to ensure that the improvements we have noted are sustained. If significant problems again arise, we will consider reapplying the high-risk designation. We added two areas to the High-Risk List since our 2017 update—Government-Wide Personnel Security Clearance Process and VA Acquisition Management. We are removing the area of DOD Supply Chain Management from the High-Risk List because, since 2017, DOD has addressed the remaining two criteria (monitoring and demonstrated progress) for the asset visibility and materiel distribution segments. Congressional attention, DOD leadership commitment, and our collaboration contributed to the successful outcome for this high-risk area, which had been on GAO’s High-Risk List since 1990. DOD’s actions for the asset visibility segment of this high-risk area included (1) providing guidance for the military components to consider key attributes of successful performance measures during metric development for their improvement initiatives; (2) incorporating into after- action reports, information relating to performance measures; and (3) demonstrating sustained progress by, for example, increasing its visibility of assets through radio-frequency identification (RFID), an automated data-capture technology that can be used to electronically identify, track, and store information contained on a tag. According to DOD, the use of RFID tags to provide visibility of sustainment cargo at the tactical leg (i.e., the last segment of the distribution system) resulted in $1.4 million annual cost savings. DOD’s actions for the materiel distribution segment of this high-risk area included (1) making progress in developing its suite of distribution performance metrics; (2) incorporating distribution metrics, as appropriate, on the performance of all legs of the distribution system, including the tactical leg; (3) making progress in refining its Materiel Distribution Improvement Plan and incorporating additional actions based on interim progress and results; and (4) improving its capability to comprehensively measure distribution performance, identifying distribution problems and root cause, and implementing solutions. According to DOD, initiatives focused on distribution process and operational improvements have resulted in at least $1.56 billion in distribution cost avoidances to date. As we have with areas previously removed from the High-Risk List, we will continue to monitor this area to ensure that the improvements we have noted are sustained. Appendix II provides additional information on this high-risk area. We are removing the area of Mitigating Gaps in Weather Satellite Data from the High-Risk List because—with strong congressional support and oversight—the National Oceanic and Atmospheric Administration (NOAA) and DOD have made significant progress since 2017 in establishing and implementing plans to mitigate potential gaps in weather satellite data. The United States relies on polar-orbiting satellites to provide a global perspective on weather every morning and afternoon. NOAA is responsible for the polar satellite program that crosses the equator in the afternoon while DOD is responsible for the polar satellite program that crosses the equator in the early morning orbit. NOAA’s actions for polar- orbiting weather satellites that addressed the remaining criteria of action plan and demonstrated progress included (1) issuing three updates to its gap mitigation plan between January 2016 and February 2017 to address shortfalls we had identified previously; and (2) successfully launching the NOAA-20 satellite in November 2017, which is currently operational and is being used to provide advanced weather data and forecasts. Moreover, NOAA is also working to build and launch the next satellites in the polar satellite program. DOD’s actions for polar-orbiting weather satellites, pursuant to statutes and accompanying congressional direction, included DOD leadership (1) developing and implementing plans to acquire satellites as part of a family of systems to replace its aging legacy weather satellites, including awarding a contract for its Weather System Follow-on–Microwave program, planned for launch in 2022; (2) establishing plans to meet its highest-priority weather monitoring data collection needs that will not be covered by the Weather System Follow-on–Microwave program, including by acquiring and launching the Electro-Optical/Infrared Weather Systems satellite in 2024; and (3) monitoring the Weather System Follow-on- Microwave satellite program’s progress toward addressing critical needs and assessing its operations and sustainment costs. As we have with areas previously removed from the High-Risk List, we will continue to monitor this area to ensure that the improvements we have noted are sustained. Appendix II provides additional information on this high-risk area. Executive branch agencies are not meeting investigation timeliness objectives, and these processing delays have contributed to a significant backlog that the National Background Investigations Bureau (NBIB)—the agency responsible for personnel security clearance investigations— reported to be approximately 565,000 investigations as of February 2019. In addition, the executive branch has not finalized performance measures to ensure the quality of background investigations and some long- standing key reform initiatives remain incomplete. Further, information technology (IT) security concerns may delay planned milestones for the development of a new background investigation IT system. We included the DOD program on our High-Risk List in 2005 and removed it in 2011 because of improvements in the timeliness of investigations and adjudications, and steps toward measuring the quality of the process. We put the government-wide personnel security clearance process on our High-Risk List in January 2018 because of significant challenges related to the timely processing of security clearances and completing the development of quality measures. In addition, thITe government’s effort to reform the personnel security clearance process, starting with the enactment of the Intelligence Reform and Terrorism Prevention Act of 2004, has had mixed progress, and key reform efforts have not been implemented government-wide. Since adding this area to the High-Risk List, the Security Clearance, Suitability, and Credentialing Performance Accountability Council (PAC), including its four principal members—the Deputy Director for Management of the Office of Management and Budget (OMB), the Director of National Intelligence (DNI); the Under Secretary of Defense for Intelligence; and the Director of the Office of Personnel Management (OPM)—have not fully met the five criteria for high-risk removal. Several issues contribute to the risks facing the government-wide personnel security clearance process: Clearance processing delays. Executive branch agencies are not meeting most investigation timeliness objectives. The percentage of executive branch agencies meeting established timeliness objectives for initial secret clearances, initial top secret clearances, and periodic reinvestigations decreased each year from fiscal years 2012 through 2018. For example, 97 percent of the executive branch agencies we reviewed did not meet the timeliness objectives for initial secret clearance investigations in fiscal year 2018. Lack of quality measures. While the executive branch has taken steps to establish government-wide performance measures for the quality of background investigations—including establishing quality assessment standards and a quality assessment reporting tool—it is unclear when this effort will be completed. Security clearance reform delays. The executive branch has reformed many parts of the personnel security clearance process— such as updating adjudicative guidelines to establish common adjudicative criteria for security clearances; however, some long- standing key initiatives remain incomplete—such as completing plans to fully implement and monitor continuous evaluation. IT security. DOD is responsible for developing a new system to support background investigation processes, and DOD officials expressed concerns about the security of connecting to OPM’s legacy systems since a 2015 data breach compromised OPM’s background investigation systems and files for 21.5 million individuals. As of December 2018, OPM has not fully taken action on our priority recommendations to update its security plans, evaluate its security control assessments, and implement additional training opportunities. However, since we added this area to our High-Risk List, the PAC has demonstrated progress in some areas. For example, NBIB reported that the backlog of background investigations decreased from almost 715,000 cases in January 2018 to approximately 565,000 cases in February 2019. NBIB officials credit an Executive Memorandum—issued jointly in June 2018 by the DNI and the Director of OPM and containing measures to reduce the investigation backlog—as a driver in backlog reduction. Further, in response to a requirement in the Securely Expediting Clearances Through Reporting Transparency (SECRET) Act of 2018, in September 2018, NBIB reported to Congress, for each clearance level, (1) the size of the investigation backlog, (2) the average length of time to conduct an initial investigation and a periodic reinvestigation, and (3) a discussion of the factors contributing to investigation timeliness. The PAC is also reporting publicly on the progress of key reforms through www.performance.gov, and for fiscal year 2018, the website contains quarterly action plans and progress updates, which present figures on the average timeliness of initial investigations and periodic reinvestigations for the executive branch as a whole, investigation workload and backlog, and investigator headcounts. We have made numerous recommendations to PAC members to address risks associated with the personnel security clearance process between 2011—when we removed DOD’s personnel security clearance program from the High-Risk List, and 2018—when we placed the government-wide personnel security clearance process on the High-Risk List. We consider 27 of these recommendations key to addressing the high-risk designation. Eight recommendations key to the high-risk designation have been implemented, including three since January 2018. Nineteen of these key recommendations remain open—including recommendations that the principal members of the PAC (1) conduct an evidence-based review of investigation and adjudication timeliness objectives, (2) develop and report to Congress on investigation quality measures, (3) prioritize the timely completion of efforts to modernize and secure IT systems that affect clearance holders government-wide, and (4) develop and implement a comprehensive workforce plan that identifies the workforce needed to meet current and future demand for background investigations services and to reduce the investigations backlog. See page 170 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. VA spends tens of billions of dollars to procure a wide range of goods and services—including medical supplies, IT, and construction of hospitals, clinics, and other facilities—to meet its mission of providing health care and other benefits to millions of veterans. VA has one of the most significant acquisition functions in the federal government, both in obligations and number of contract actions. The Veterans Health Administration (VHA) provides medical care to veterans and is by far the largest administration in the VA. Since we began focusing on VA’s acquisition management activities in 2015, we have reported numerous challenges in this area. Since 2015, we have made 31 recommendations, 21 of which remain open, that cover a range of areas to address challenges in VA’s acquisition management. In fiscal year 2019, VA received the largest discretionary budget in its history—$86.5 billion, about $20 billion higher than in 2015. About a third of VA’s discretionary budget in fiscal year 2017, or $26 billion, has been used to contract for goods and services. VA’s acquisition management continues to face challenges including (1) outdated acquisition regulations and policies; (2) lack of an effective medical supplies procurement strategy; (3) inadequate acquisition training; (4) contracting officer workload challenges; (5) lack of reliable data systems; (6) limited contract oversight and incomplete contract file documentation; and (7) leadership instability. In light of these challenges and given the significant taxpayer investment, it is imperative that VA show sustained leadership commitment to take steps to improve the performance of its procurement function so that it can use its funding in the most efficient manner possible to meet the needs of those who served our country. This area has been added to the High-Risk List for the following reasons in particular: Outdated acquisition regulations and policies. VA’s procurement policies have historically been outdated, disjointed, and difficult for contracting officers to use. In September 2016, we reported that the acquisition regulations contracting officers currently follow have not been fully updated since 2008 and that VA had been working on completing a comprehensive revision of its acquisition regulations since 2011. VA’s delay in updating this fundamental source of policy has impeded the ability of contracting officers to effectively carry out their duties. We recommended in September 2016 that VA identify measures to expedite the revision of its acquisition regulations and clarify what policies are currently in effect. VA concurred with this recommendation but has not yet fully implemented it. Lack of an effective medical supplies procurement strategy. VA’s Medical Surgical Prime Vendor-Next Generation (MSPV-NG) program for purchasing medical supplies to meet the needs of about 9 million veterans at 172 medical centers has not been effectively executed, nor is it in line with practices at leading hospitals that have launched similar programs. We reported in November 2017 that VA’s approach to developing its catalog of supplies was rushed and lacked key stakeholder involvement and buy-in. As a result, VA was not able to accomplish some of the key efficiencies the program was intended to achieve, such as streamlining the purchase of medical supplies and saving money. We recommended in November 2017 that VA develop, document, and communicate to stakeholders an overarching strategy for the program. VA concurred with this recommendation and reported that it would develop a new strategy by March 2019. Contracting officer workload challenges. The majority of our reviews since 2015 have highlighted workload as a contributing factor to the challenges that contracting officers face. Most recently, in September 2018, we reported that about 54 percent of surveyed VA contracting officers said their workload was not reasonable. In addition, in September 2016, we reported that VHA contracting officers processed a large number of emergency procurements of routine medical supplies, which accounted for approximately 20 percent of VHA’s overall contract actions in fiscal year 2016, with obligations totaling about $1.9 billion. Contracting officers told us that these frequent and urgent small-dollar transactions reduce contracting officers’ efficiency and ability to take a strategic view of procurement needs. We recommended in November 2017 that VHA network contracting offices work with medical centers to identify opportunities to more strategically purchase goods and services frequently purchased on an emergency basis. VA concurred with this recommendation and reported in December 2018 that it is utilizing a supply chain dashboard to track items purchased on an emergency basis and determine which of those items to include on the catalog. VA noted that it added 13,300 items to the catalog from June 2018 to December 2018, including items often purchased on an emergency basis. We requested documentation showing which items added to the catalog were previously purchased on an emergency basis, but as of January 2019, VA had not yet provided it. Among other things, VA should implement our 21 open recommendations and specifically needs to take the following steps to demonstrate greater leadership commitment and strategic planning to ensure efficient use of its acquisition funding and staffing resources: Prioritize completing the revision of its acquisition regulations, which has been in process since 2011. Develop, document, and communicate to stakeholders a strategy for the Medical Surgical Prime Vendor program to achieve overall program goals. Identify opportunities to strategically purchase goods and services that are frequently purchased on an emergency basis. See page 210 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. In addition to specific areas that we have designated as high-risk, other important challenges facing our nation merit continuing close attention. One of these is the use of illicit drugs and the misuse of prescription drugs and the ways they affect individuals, their families, and the communities in which they live. Over 70,000 people died from drug overdoses in 2017—about 191 people every day—according to the Centers for Disease Control and Prevention, with the largest portion of these deaths attributed to opioids. Further, drug overdoses are the leading cause of death due to injuries in the United States. They are currently at their highest ever recorded level and, since 2011, have outnumbered deaths by firearms, motor vehicle crashes, suicide, and homicide, according to the Drug Enforcement Administration. The Council of Economic Advisors estimates that in 2015, the economic cost of the opioid crisis alone was more than $500 billion when considering the value of lives lost due to opioid-related overdose. Federal drug control efforts spanning prevention, treatment, interdiction, international operations, and law enforcement represent a considerable federal investment. According to the President’s fiscal year 2019 budget, federal drug control funding for fiscal year 2017 was $28.8 billion. Multiple federal agencies have ongoing efforts to respond to this crisis, including efforts to reduce the supply and demand for illicit drugs, to prevent misuse of prescription drugs, and to treat substance use disorders. However, we previously found that many efforts lacked measures to gauge the success of the federal response. Further, we have long advocated an approach to decision-making based on risk management. Such an approach would (1) link agencies’ plans and budgets to achieving their strategic goals, (2) assess values and risks of various courses of actions to help set priorities and allocate resources, and (3) provide for the use of performance measures to assess progress. The Office of National Drug Control Policy (ONDCP) is responsible for overseeing and coordinating the implementation of U.S. drug policy, including developing the National Drug Control Strategy (Strategy). ONDCP released the 2019 Strategy on January 31, 2019. The Strategy focuses on approaches related to prevention, treatment and recovery, and steps to reduce the availability of illicit drugs in the United States. We will continue to monitor the extent to which ONDCP and other federal agencies are employing a risk management and coordinated approach to their efforts to limit drug misuse. In particular, we have ongoing and planned work to assess ONDCP’s operations, including its (1) leadership and coordination of efforts across the federal government; (2) the effects of the drug crisis on labor force participation and productivity and on people with disabilities and other vulnerable populations; (3) key federal efforts to reduce the availability of illicit drugs; and (4) agency efforts around drug education and prevention. We will determine whether this issue should be added to the High-Risk List once we have completed this ongoing and planned work. Agencies can show progress by addressing our five criteria for removal from the list: leadership commitment, capacity, action plan, monitoring, and demonstrated progress. As shown in table 1, 24 high-risk areas, or about two-thirds of all the areas, have met or partially met all five criteria for removal from our High-Risk List; 20 of these areas fully met at least one criterion. Compared with our last assessment, 7 high-risk areas showed progress in one or more of the five criteria without regressing in any of the criteria. Ten high-risk areas have neither met nor partially met one or more criteria. Two areas showed mixed progress by increasing in at least one criterion and also declining in at least one criterion. Three areas declined since 2017. These changes are indicated by the up and down arrows in table 1. Figure 1 shows that since our 2017 update, the most progress was made on the action plan criterion—four high-risk areas received higher ratings. We rated two areas lower on leadership commitment and two areas lower on monitoring. Table 2 shows that 17 of the 34 high-risk areas we rated have met the leadership commitment criterion while two high-risk area ratings regressed on leadership commitment from met to partially met since our last report. Leadership commitment is the critical element for initiating and sustaining progress, and leaders provide needed support and accountability for managing risks. Leadership commitment is needed to make progress on the other four high-risk criteria. Table 2 shows that only three high-risk areas met the criterion for capacity, six met the criterion for action plan, and two met the criterion for demonstrated progress. One high-risk area—U.S. Government’s Environmental Liability—has partially met only one criterion since we added the area to our list in 2017 and the rest are not met. As noted, seven areas showed improvement in one or more criterion without regressing in any criteria. Two areas showed sufficient progress to be removed from the High-Risk List. The other five high-risk areas remaining on the 2019 list demonstrated improvement and are described below. Three of these five improving high-risk areas are the responsibility of the Department of Defense (DOD)—DOD Support Infrastructure Management, DOD Financial Management, and DOD Business Systems Modernization. The two other improving areas are Department of Energy’s (DOE's) Contract Management for the National Nuclear Security Administration and Office of Environmental Management, and Medicare Program & Improper Payments. DOD Support Infrastructure Management: DOD manages a portfolio of real property assets that, as of fiscal year 2017, reportedly included about 586,000 facilities—including barracks, maintenance depots, commissaries, and office buildings. The combined replacement value of this portfolio is almost $1.2 trillion and includes about 27 million acres of land at nearly 4,800 sites worldwide. This infrastructure is critical to maintaining military readiness, and the cost to build and maintain it represents a significant financial commitment. Since our 2017 High-Risk Report, DOD’s rating for two criteria—leadership commitment and action plan—improved from partially met to met. DOD has demonstrated leadership commitment by stating its commitment to addressing key recommendations we have made by, for example, (1) better forecasting the initial Base Realignment and Closure (BRAC) costs for military construction, IT, and relocating military personnel and equipment; (2) better aligning infrastructure to DOD force structure needs by, for example, improving the accuracy and sufficiency of its excess capacity estimates; and (3) pursuing an effort to consolidate and standardize leases, which includes analyzing whether it is feasible to relocate functions from commercial leased space to existing space on an installation, thereby reducing leases and better utilizing excess space. DOD has developed action plans to better identify excess infrastructure and thus be positioned to dispose of it. For example, in the 2017 High- Risk Report, we stated that DOD’s Real Property Efficiency Plan includes DOD’s goals for reducing the footprint of its real property inventory and metrics to gauge progress, to be implemented by the end of 2020. We also found in 2018 that DOD was achieving cost savings and cost avoidances as it had begun using intergovernmental support agreements between military installations and local governments to obtain installation services, such as waste removal, grounds maintenance, and stray animal control. As a result of these and other actions, DOD now meets the action plan criterion for this high-risk area. As of December 2018, 23 recommendations related to this high-risk area remain open. DOD continues to partially meet the criteria for capacity, monitoring, and demonstrated progress. See page 158 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. DOD Financial Management: Since our 2017 High-Risk Report, ratings for the DOD Financial Management high-risk area improved for the criteria of leadership commitment and monitoring. For the leadership commitment criterion, the high-risk area rating improved from partially met to met in 2019 due to several DOD leadership actions. For example, in 2018, DOD leadership met the goal of undergoing an agency-wide financial statement audit and established a process to remediate any audit findings—ultimately to improve the quality of financial information that is most valuable in managing the department’s day-to-day operations. In addition, according to a DOD official, audit remediation efforts have produced benefits in certain inventory processes that have led to operational improvements. DOD leadership demonstrated its commitment to making needed improvements by developing a database that tracks hundreds of findings and recommendations that came out of the audits. In addition, senior leadership has been meeting bimonthly with military services’ leadership for updates on the status of corrective action plans to address audit findings and recommendations, and the Under Secretary of Defense (Comptroller) has been meeting frequently with the Secretary of Defense to review the plans. These same DOD actions also led to the high-risk area’s rating for the criterion of monitoring to improve from not met to partially met. For example, the database mentioned above is intended to capture, prioritize, and assign responsibility for auditor findings and related corrective action plans, which are meant to be used to measure progress towards achieving a clean audit opinion. Further, DOD leadership has held frequent meetings to discuss the status of corrective action plans. In addition, DOD also established councils in certain areas (e.g., financial reporting) to review the status of audit remediation activities and challenges. All of these actions demonstrate an improvement in DOD’s monitoring activities for its financial management function. However, DOD’s efforts to improve its financial management continue to be impaired by long-standing issues—including its decentralized environment; cultural resistance to change; lack of skilled financial management staff; ineffective processes, systems, and controls; incomplete corrective action plans; and the need for more effective monitoring and reporting. DOD remains one of the few federal entities that cannot accurately account for and report on its spending or assets. As of December 2018, 53 recommendations for this high-risk area are open. The DOD Financial Management high-risk area continues to partially meet the capacity and action plan criteria and not meet the demonstrated progress criterion. See page 147 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. DOD Business Systems Modernization: DOD spends billions of dollars each year to acquire modernized systems, including systems that address key areas such as personnel, financial management, health care, and logistics. This high-risk area includes three critical challenges facing DOD: (1) improving business system acquisition management, (2) improving business system investment management, and (3) leveraging DOD’s federated business enterprise architecture. DOD’s capacity for modernizing its business systems has improved over time and, since our 2017 High-Risk Report, DOD’s overall rating for the criterion of action plan improved from not met to partially met in 2019. DOD established a plan for improving its federated business enterprise architecture (i.e., description of DOD’s current and future business environment and a plan for transitioning to the future environment). Specifically, the rating improved for DOD’s federated business enterprise architecture segment of the high-risk area because DOD’s assistant deputy chief management officer approved a business architecture improvement plan in January 2017. Since 2017, we have made 10 recommendations related to this high-risk issue. As of December 2018, 27 recommendations are open. The leadership, capacity, monitoring, and demonstrated progress criteria remain partially met as in 2017. See page 152 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. . DOE's Contract Management for the National Nuclear Security Administration and Office of Environmental Management: DOE oversees a broad range of programs related to nuclear security, science, energy, and waste cleanup, among other areas. As the largest civilian contracting agency in the federal government, DOE relies primarily on contractors to carry out its programs. For instance, DOE spends about 90 percent of its annual budget on contracts and acquiring capital assets. In fiscal year 2018, DOE’s budget was $34.5 billion. The high-risk area focuses on contracts, as well as major projects—those with an estimated cost of $750 million or greater—managed by DOE’s National Nuclear Security Administration (NNSA) and Office of Environmental Management (EM). Since our 2017 High-Risk Report, DOE has made progress by improving from a not met to a partially met rating for the demonstrated progress criterion. Specifically, through its Office of Cost Estimating and Program Evaluation, NNSA has enhanced its capability to estimate costs and schedules, and to assess alternatives for programs and projects, among other things. NNSA also made progress by adopting best practices in several areas, such as those for estimating costs and schedules in nuclear weapons refurbishment activities and capital asset acquisitions. For example, we determined that DOE’s revised cost estimate of $17.2 billion to construct a Mixed Oxide Fuel Fabrication Facility to dispose of surplus, weapons-grade plutonium substantially met best practices— providing assurance that the estimated costs could be considered reliable. This finding contributed to DOE’s reevaluation of the project and ultimate termination, in October 2018, in favor of a potentially less costly disposal approach. Fifty-one of our recommendations were open as of December 2018; 15 recommendations were made since the last high-risk update in February 2017. DOE continues to meet the criterion of leadership commitment, partially meet the criteria for action plan and monitoring, and not meet the criterion for capacity. See page 217 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. . Medicare Program & Improper Payments: In calendar year 2017, Medicare, which is overseen by the Centers for Medicare & Medicaid Services (CMS), financed $702 billion worth of health services for approximately 58 million elderly and disabled beneficiaries. Medicare faces a significant risk with improper payments—payments that either were made in an incorrect amount or should not have been made at all— which reached an estimated $48 billion in fiscal year 2018. Since our 2017 High-Risk Report, estimated improper payment rates declined more than one percent across the Medicare program. In addition, CMS’ rating for the capacity criterion of the improper payments segment improved from partially met to met in 2019 due to several actions. First, the Center for Program Integrity’s (CPI) budget and resources have increased over time and the agency has established work groups and interagency collaborations to extend its capacity. For example, CMS allocated more staff to CPI after Congress provided additional funding. CPI’s full-time equivalent positions increased from 177 in 2011 to 419 in 2017. Additionally, in August 2017, we reported that CMS’s Fraud Prevention System, which analyzes claims to identify health care providers with suspect billing patterns, helped speed up certain fraud investigation processes. Further, the Healthcare Fraud Prevention Partnership helped improve information sharing among payers inside and outside of the government. Since 1990, when we added Medicare to our High-Risk List, we have made many recommendations related to the Medicare program, 28 of which were made since the last high-risk update in February 2017. As of December 2018, more than 80 recommendations remain open. CMS continues to meet the criterion of leadership commitment and to partially meet the remaining three criteria of action plan, monitoring, and demonstrated progress. See page 241 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Congress enacted several laws since our last report in February 2017 to help make progress on high-risk issues. Table 3 lists selected examples of congressional actions taken on high-risk areas. Congressional oversight also plays a vital role in addressing high-risk issues. For example, at a May 2018 hearing, we testified that the Census Bureau’s (Bureau) cost estimate was not reliable, and that the actual cost could be higher than planned. Further, the Secretary of Commerce created a dedicated team to provide oversight and guidance to the Bureau on cost estimation. In addition to its instrumental role in supporting progress in individual high-risk areas, Congress also enacted the following statutes that, if implemented effectively, will help foster progress on high-risk issues government-wide: Fraud Reduction and Data Analytics Act of 2015 (FRDAA):FRDAA is intended to strengthen federal antifraud controls. FRDAA requires OMB to use our Fraud Risk Framework to create guidelines for federal agencies to identify and assess fraud risks, and then design and implement control activities to prevent, detect, and respond to fraud. Agencies, as part of their annual financial reports beginning in fiscal year 2017, are further required to report on their fraud risks and their implementation of fraud reduction strategies, which should help Congress monitor agencies’ progress in addressing and reducing fraud risks. To aid federal agencies in better analyzing fraud risks, FRDAA requires OMB to establish a working group tasked with developing a plan for creating an interagency library of data analytics and data sets to facilitate the detection of fraud and the recovery of improper payments. This working group and the library should help agencies coordinate their fraud detection efforts and improve their ability to use data analytics to monitor databases for potential improper payments. The billions of dollars in improper payments, some of which may be a result of fraud, are a central part of the Medicare Program, Medicaid Program, and Enforcement of Tax Laws (Earned Income Tax Credit) high-risk areas. We reported in 2018 that, among other things, OMB did not involve all agencies subject to the act as required by FRDAA or hold the required minimum number of working-group meetings in 2017. As shown in figure 2, a majority of the 72 agencies surveyed indicated a lack of involvement with and information from the working group as challenges in implementing FRDAA. We made three recommendations, including that OMB ensure the working group meets FRDAA’s requirements to involve all agencies that are subject to the act and ensure that mechanisms to share controls, best practices, and data-analytics techniques are in place. OMB did not concur with our recommendations. We continue to believe the recommendations are valid, as discussed in the 2018 report. IT Acquisition Reform, statutory provisions known as the Federal Information Technology Acquisition Reform Act (FITARA): FITARA, enacted in December 2014, was intended to improve how agencies acquire IT and better enable Congress to monitor agencies’ progress in reducing duplication and achieving cost savings. Since the enactment of these provisions, OMB and federal agencies have paid greater attention to IT acquisition and operation, resulting in improvements to the government-wide management of this significant annual investment. These efforts have been motivated in part by sustained congressional support for improving implementation of this law, as highlighted in agencies’ FITARA implementation scores issued biannually by the House Committee on Oversight and Reform. This continuing oversight has produced positive results. For example, in the committee’s December 2018 FITARA implementation scorecard, 18 of the 24 major federal agencies received the highest possible rating for their efforts to improve the management of software licenses, of which we have found there are thousands annually across the government. Seven months earlier, in the prior scorecard, only eight agencies had achieved this rating. Moreover, federal agencies have taken actions to address 106 of the 136 related recommendations that we have made in this area since 2014. FITARA includes specific requirements related to seven areas: the federal data center consolidation initiative, enhanced transparency and improved risk management, agency Chief Information Officer authority enhancements, portfolio review, expansion of training and use of IT acquisition cadres, government-wide software purchasing, and maximizing the benefit of the federal strategic sourcing initiative. In November 2017, Congress extended or removed the sunset dates of several of these statutory requirements that were originally to end in 2018 and 2019. While all of the 24 federal agencies covered by this law have developed FITARA implementation plans, the agencies need to effectively execute these plans. Successfully addressing FITARA requirements is central to making progress in Improving the Management of IT Acquisitions and Operations, which has been on our High-Risk List since 2015. Program Management Improvement Accountability Act (PMIAA): Enacted in December 2016, the act is intended to improve program and project management in certain larger federal agencies. Among other things, the act requires the Deputy Director for Management of OMB to adopt and oversee implementation of government-wide standards, policies, and guidelines for program and project management in executive agencies. The act also requires the Deputy Director to conduct portfolio reviews to address programs we identify as high-risk. It further creates a Program Management Policy Council to act as the principal interagency forum for improving practices related to program and project management. The council is to review programs identified as high-risk and make recommendations to the Deputy Director or designee. OMB has produced a general strategy for implementing the law through 2022 and met some initial milestones required by PMIAA. For example, in June 2018, OMB issued OMB Memorandum M- 18-19, which includes: (1) agency guidance for implementing PMIAA, (2) a five-year strategic outline for improving program and project management, and (3) initial program management standards and principles. Further, agencies have designated Program Management Improvement Officers to guide their implementation of PMIAA. According to OMB, it began implementing PMIAA’s requirement to conduct portfolio reviews on high-risk areas by requiring relevant agencies to provide several items for discussion during the 2018 Strategic Review meetings. These annual meetings are to consist primarily of a discussion of agency progress towards each of the strategic objectives outlined in their strategic plans, but also cover other management topics such as enterprise risk management and high-risk area progress. According to OMB documents, in advance of these meetings, OMB required agencies to provide a high-level summary of (1) any disagreements with our recommendations, (2) progress barriers, and (3) actions needed by OMB, other agencies, or Congress to help the agency achieve progress towards removal from our High-Risk List. OMB officials told us their 2018 Strategic Review meetings did not address each high-risk area but did address government-wide high-risk areas, such as cybersecurity, information technology, and strategic human capital as they related to the President’s Management Agenda. In the past, senior management officials from OMB, applicable agencies, and our agency have met to address areas where additional management attention could be beneficial to high-risk issues. These trilateral meetings, beginning in 2007 and pre- dating PMIAA’s 2016 enactment, have continued across administrations. However, OMB has organized only one of these high-risk meetings since the last high-risk update in 2017, on the Government-wide Personnel Security Clearance Process. In November 2018, OMB told us of plans to hold additional meetings on priority high-risk areas, including the 2020 Decennial Census, Strategic Human Capital Management, Ensuring the Cybersecurity of the Nation, National Aeronautics and Space Administration (NASA) Acquisition Management, and Managing Federal Real Property. Effective implementation of PMIAA provides an important opportunity to enhance progress on high-risk areas by focusing leadership attention through the portfolio reviews and trilateral meetings. Further, a number of high-risk areas have longstanding or significant program and project management concerns, including the acquisition-related high-risk areas for DOD, DOE, NASA, and VA. These and other programs can benefit from improving program and project management. In December 2019, we will report on OMB’s progress in implementing PMIAA, including what further steps it has taken to use the portfolio review process required in PMIAA to address issues on our High-Risk List. Agency leaders took actions to implement our recommendations. These resulted in numerous improvements to programs and operation and improved service. Further, these actions to implement our recommendations resulted in significant financial benefits. Table 4 shows some examples of the financial benefits achieved since our last High-Risk Report. In the 2 years since our last High-Risk Report, three areas—NASA Acquisition Management, Transforming EPA's Process for Assessing and Controlling Toxic Chemicals, and Limiting the Federal Government's Fiscal Exposure By Better Managing Climate Change Risks—have regressed in their ratings against our criteria for removal from the High- Risk List. In addition, while progress is needed across all high-risk areas, we have identified nine additional areas that require significant attention to address imminent, longstanding, or particularly broad issues affecting the nation. NASA plans to invest billions of dollars in the coming years to explore space, improve its understanding of the Earth’s environment, and conduct aeronautics research, among other things. We designated NASA’s acquisition management as high risk in 1990 in view of NASA’s history of persistent cost growth and schedule delays in the majority of its major projects. Following several years of continuing a generally positive trend of limiting cost growth and schedule delays for its portfolio of major projects, we found that NASA’s average launch delay increased from 7 to 12 months between May 2017 and May 2018. Further, the overall development cost growth increased from 15.6 percent to at least 18.8 percent over the same time period. NASA’s largest science project, the James Webb Space Telescope, has experienced schedule delays of 81 months and cost growth of 95 percent since the project’s cost and schedule baseline was first established in 2009. NASA is at risk for continued cost growth and schedule delays in its portfolio of major projects. Since our 2017 high-risk update, we have lowered NASA acquisition management from meeting the rating to partially meeting the rating in two criteria—leadership commitment and monitoring. The other three criteria ratings remained the same as in 2017. Ratings for capacity and demonstrated progress remain partially met and the rating for action plan remains met. Over the next several years, NASA plans to add new, large, and complex projects to the portfolio, including a lunar Gateway—currently being discussed as a platform in a lunar orbit to mature deep space exploration capabilities. In addition, many of NASA’s current major projects, including some of the most expensive ones, are in the phase of their life cycles when cost growth and schedule delays are most likely. NASA acquisition management requires significant attention for the following reasons: NASA leadership has approved risky programmatic decisions for complex major projects, which compounded technical challenges. For example, leadership has approved some programs to proceed (1) with low cost and schedule reserves, (2) with overly aggressive schedules, and (3) without following best practices for establishing reliable cost and schedule baselines. NASA leadership has also not been transparent about cost and schedule estimates for some of its most expensive projects. Without transparency into these estimates, both NASA and Congress have limited data to inform decision making. NASA has not yet instituted a program for monitoring and independently validating the effectiveness and sustainability of the corrective action measures in its new action plan, which NASA finalized in December 2018. In addition, while NASA has taken some steps to build capacity to help reduce acquisition risk, including updating tools aimed at improving cost and schedule estimates, other areas still require attention. For example, we reported in May 2018 that several major NASA projects experienced workforce challenges, including not having enough staff or staff with the right skills. NASA has also identified capability gaps in areas such as scheduling, earned value management, and cost estimating, and has efforts underway to try to improve capacity in these areas. Since 2017, we have made 9 recommendations on this high-risk area, and as of December 2018, 15 recommendations remain open. These recommendations include that NASA needs to improve transparency of major project cost and schedule estimates, especially for its human spaceflight programs, as well as continue to build capacity to reduce acquisition risk. NASA will also need to implement its new action plan and track progress against it. See page 222 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The Environmental Protection Agency’s (EPA’s) ability to effectively implement its mission of protecting public health and the environment is dependent on it assessing the risks posed by chemicals in a credible and timely manner. Such assessments are the cornerstone of scientifically sound environmental decisions, policies, and regulations under a variety of statutes. Based on our work since our 2017 High-Risk Report, the overall rating for leadership commitment decreased from met to partially met due to limited information for completing chemical assessments and proposed budget cuts in the Integrated Risk Information System (IRIS) Program. The ratings for the remaining four criteria remain unchanged and are partially met. The EPA Acting Administrator indicated his commitment to fulfill the agency’s obligations under the Toxic Substances Control Act (TSCA) as amended by the 2016 Frank R. Lautenberg Chemical Safety for the 21 Century Act (Lautenberg Act) and ensure chemicals in the marketplace are safe for human health and the environment. Nonetheless, EPA needs to give more attention to several areas to fully realize the benefits of the new law, and to demonstrate additional progress in the IRIS Program, such as: While EPA released a document in late December 2018 called the IRIS Program Outlook, the Outlook fails to list the projected date for most of the assessments and includes no information regarding assessment prioritization—including how these assessments will meet program and regional office needs. The Lautenberg Act increases both EPA’s responsibility for regulating chemicals and its workload. EPA recently issued a rule under the act to collect fees from certain companies to defray a portion of the implementation costs, but it is unclear whether the fees collected will be sufficient to support relevant parts of the program. EPA issued a First Year Implementation Plan in June 2016 noting that this document is intended to be a roadmap of major activities EPA will focus on during the initial year of implementation. As of mid- February 2019 the plan has not been updated, according to publically available information, although EPA had indicated that it is a living document that will be further developed over time. EPA needs to ensure that the people and resources dedicated to the IRIS Program and TSCA implementation are sufficient. Our March 2019 report on chemical assessments provides information on what remains to be done to address challenges in the IRIS program and implement the Lautenberg Act. Since we added this area to our High-Risk List in 2009, we have made 12 recommendations to EPA related to IRIS and TSCA. As of February 2019, seven recommendations remain open. See page 204 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Numerous studies have concluded that climate change poses risks to many environmental and economic systems and creates a significant fiscal risk to the federal government. The rising number of natural disasters and increasing reliance on the federal government for assistance is a key source of federal fiscal exposure. As of December 2018, total federal funding for disaster assistance since 2005 is approaching half a trillion dollars (about $430 billion), most recently for catastrophic hurricanes, flooding, wildfires, and other losses in 2017 and 2018. The costliness of disasters is projected to increase as extreme weather events become more frequent and intense due to climate change. There are five areas where government-wide action is needed to reduce federal fiscal exposure, including, but not limited to, the federal government’s role as (1) the insurer of property and crops; (2) the provider of disaster aid; (3) the owner or operator of infrastructure; (4) the leader of a strategic plan that coordinates federal efforts and informs state, local, and private-sector action; and (5) the provider of data and technical assistance to decision makers. Neither global efforts to mitigate climate change causes nor regional adaptation efforts currently approach the scales needed to avoid substantial damages to the U.S. economy, environment, and human health over the coming decades, according to the November 2018 Fourth National Climate Assessment. Government-wide action is needed to improve the nation’s resilience to natural hazards and reduce federal fiscal exposure to climate change impacts. Congress continues to show its commitment to progress on this high-risk issue by enacting legislation. For example, in October 2018, the Disaster Recovery Reform Act was enacted, which, among other things, allows the President to set aside, with respect to each major disaster, a percentage of certain grants to use for pre-disaster hazard mitigation. In addition, the National Defense Authorization Act of 2018, required, among other things, DOD to report on climate impacts to its installations. However, the federal government has not made measurable progress since 2017 to reduce its fiscal exposure to climate change, and in some cases, has revoked prior policies designed to do so. Specifically, since 2017, the ratings for four criteria remain unchanged—three at partially met and one at not met. The rating for one criterion—monitoring—regressed to not met. Limiting the federal government’s fiscal exposure to climate change requires significant attention because the federal government has revoked prior policies that had partially addressed this high-risk area and has not implemented several of our recommendations that could help reduce federal fiscal exposure. For example, since our 2017 high-risk update, the federal government: revoked Executive Order 13690, which had established a government-wide federal flood risk management standard to improve the resilience of communities and federal assets against the impacts of flooding. This action could increase federal fiscal exposure, as taxpayer-funded projects may not last as long as intended because they are not required to account for future changes in climate-related risk. rescinded its guidance directing agencies to consider climate change in their National Environmental Policy Act of 1969 reviews for certain types of federal projects. has not implemented our July 2015 recommendation to establish a comprehensive investment strategy identifying, prioritizing, and implementing federal disaster resilience investments that could reduce federal fiscal exposure to climate change. has not implemented our November 2015 recommendations to create a national climate information system providing authoritative, accessible information useful for state, local, and private-sector decision making. We have made 62 recommendations related to this high-risk area, 12 of which were made since our February 2017 high-risk update. As of December 2018, 25 remain open. The federal government needs a cohesive strategic approach with strong leadership and the authority to manage climate change risks across the entire range of federal activities. See page 110 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Federal agencies and the nation’s critical infrastructures—such as energy, transportation systems, communications, and financial services— are dependent on information technology systems to carry out operations. The security of these systems and the data they use is vital to public confidence and national security, prosperity, and well-being. The risks to systems underpinning the nation’s critical infrastructure are increasing as security threats evolve and become more sophisticated. We first designated information security as a government-wide high-risk area in 1997. This was expanded to include protecting cyber critical infrastructure in 2003 and protecting the privacy of personally identifiable information in 2015. In 2018, we updated this high-risk area to reflect the lack of a comprehensive cybersecurity strategy for the federal government. Since 2010, we have made over 3,000 recommendations to agencies aimed at addressing cybersecurity shortcomings, including protecting cyber critical infrastructure, managing the cybersecurity workforce, and responding to cybersecurity incidents. Of those 3,000 recommendations, 448 were made since our last high-risk update in February 2017. Although many recommendations have been addressed, about 700 have not yet been implemented. Despite the number of unimplemented recommendations, since our 2017 High-Risk Report, the administration has made progress in this high-risk area as it continues to meet the leadership commitment criterion through various actions. These include the President issuing (1) an executive order in May 2017 requiring federal agencies to take a variety of actions, including better managing their cybersecurity risks and coordinating to meet reporting requirements related to cybersecurity of federal networks and critical infrastructure and (2) a National Security Strategy in December 2017 citing cybersecurity as a national priority and identifying needed actions. Further, the administration issued a government-wide reform plan and reorganization recommendations in June 2018 with, among other things, proposals for solving the federal cybersecurity workforce shortage. Additionally, the administration released a National Cyber Strategy in September 2018 outlining activities such as securing critical infrastructure, federal networks, and associated information. However, additional actions are needed. We have identified four major cybersecurity challenges facing the nation: (1) establishing a comprehensive cybersecurity strategy and performing effective oversight, (2) securing federal systems and information, (3) protecting cyber critical infrastructure, and (4) protecting privacy and sensitive data. To address the four major cybersecurity challenges, we identified 10 critical actions the federal government and other entities need to take. These critical actions include, for example, developing and executing a more comprehensive federal strategy for national cybersecurity and global cyberspace; addressing cybersecurity workforce management challenges; and strengthening the federal role in protecting the cybersecurity of critical infrastructure (see figure 3). Until these shortcomings are addressed, federal agencies’ information and systems will be increasingly susceptible to the multitude of cyber- related threats that exist. See page 178 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The expanded federal role in housing finance that began during the 2007–2009 financial crisis has substantially increased the government’s exposure to potential mortgage losses. Federally supported mortgages include those backed by the Federal National Mortgage Association (Fannie Mae) and the Federal Home Loan Mortgage Corporation (Freddie Mac)—collectively, the enterprises—which the Federal Housing Finance Agency (FHFA) placed into government conservatorships in 2008. Federal support also occurs through Federal Housing Administration (FHA) mortgage insurance and Government National Mortgage Association (Ginnie Mae) guarantees on mortgage-backed securities. The substantial financial assistance the enterprises required during and after the crisis, coupled with the large fiscal exposure they and other federal mortgage entities represent today, underscore the need to reform the federal role in housing finance. Delay in resolving the federal role in housing finance poses considerable risks. Through the enterprises, FHA, and Ginnie Mae, the federal government is exposed to potential losses on several trillion dollars in mortgage debt. A severe economic downturn could trigger significant taxpayer assistance to one or more of these entities. Congress and federal agencies have taken some steps to facilitate the transition to a revised federal role, such as holding hearings, introducing legislation, issuing regulations, and developing market monitoring tools. For example, in 2013 and 2014, housing and regulatory agencies finalized rules designed to prevent a recurrence of risky practices in originating and securing mortgages that contributed to the financial crisis. Additionally, FHFA and the Consumer Financial Protection Bureau have developed a representative database of mortgage information that could be useful for examining the effect of mortgage market reforms. However, overall progress on resolving the federal role will be difficult to achieve until Congress provides further direction by enacting changes to the housing finance system. Several issues contribute to the risks facing federal housing finance, including the following: More than 10 years after entering federal conservatorships, the enterprises’ futures remain uncertain and billions of taxpayer dollars remain at risk. Under agreements with the Department of the Treasury (Treasury), the enterprises have received $191.4 billion in capital support as of the end of fiscal year 2018 and have paid dividends to the department exceeding that amount. If they were to incur major additional losses, they would draw required amounts from their remaining $254.1 billion in Treasury commitments. In addition, prolonged conservatorships could hinder development of the broader mortgage securities market by creating uncertainty and crowding out private investment. Nonbanks (lenders and loan servicers that are not depository institutions) have played an increasingly large role in the mortgage market in recent years. While nonbanks have helped provide access to mortgage credit, they also may pose additional risks, in part because they are not federally regulated for safety and soundness. However, FHFA lacks statutory authority to examine nonbank mortgage servicers and other third parties who do business with and pose potential risks to the enterprises. The statutory 2 percent capital requirement for FHA’s $1.26 trillion mortgage insurance fund is not based on a specified risk threshold, such as the economic conditions the fund would be expected to withstand. As a result, it may not provide an adequate financial cushion under scenarios in which Congress may anticipate the fund would be self-sufficient. During the last housing downturn, the fund’s capital ratio fell below the required level and remained there for 6 consecutive years. At the end of fiscal year 2013, the fund required supplemental funds—about $1.7 billion—for the first time in its history. Six of our federal housing recommendations remain open, including those we made in June 2015 on assessing the effects of mortgage reforms already in place. Further, as we previously recommended in November 2016 and January 2019, Congress should consider housing finance reform legislation that: establishes objectives for the future federal role in housing finance, including the role and structure of the enterprises within the housing finance system; provides a transition plan to a reformed system that enables the enterprises to exit federal conservatorship; and addresses all relevant federal entities, including FHA and Ginnie Mae. As we recommended in March 2016 and November 2017, respectively, Congress also should consider granting FHFA explicit authority to examine nonbank servicers and other third parties that do business with the enterprises, and specifying the economic conditions FHA’s insurance fund would be expected to withstand without a substantial risk of requiring supplemental funds. See page 95 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Due to the significance and risk associated with Resolving the Federal Role in Housing Finance, we are separating it from the high-risk area of Modernizing the U.S. Financial Regulatory System. These areas were combined in our 2017 High-Risk report. See page 95 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The Pension Benefit Guaranty Corporation (PBGC) is responsible for insuring the defined benefit pension plans for nearly 37 million American workers and retirees, who participate in about 24,800 private sector plans. PBGC faces an uncertain financial future due, in part, to a long- term decline in the number of traditional defined benefit plans and the collective financial risk of the many underfunded pension plans that PBGC insures. PBGC’s financial portfolio is one of the largest of all federal government corporations. While PBGC’s single employer program had a net surplus of about $2.4 billion at the end of fiscal year 2018, its multiemployer program had a net deficit of about $54 billion—or a combined net accumulated financial deficit of over $51 billion. Its deficit has increased by nearly 45 percent since fiscal year 2013. PBGC has estimated that, without additional funding, its multiemployer insurance program will likely be exhausted by 2025 as a result of current and projected pension plan insolvencies. The agency’s single-employer insurance program is also at risk due to the continuing decline of traditional defined benefit pension plans, as well as premiums that are not well aligned to the financial risk presented by the plans it insures. While Congress and PBGC have taken significant and positive steps to strengthen the agency in the past 5 years, challenges related to PBGC’s funding and governance structure remain. Congress established a temporary Joint Select Committee on multiemployer pension plans in 2018—with the goal of improving the solvency of the multiemployer program. However, the committee did not release draft legislation. Addressing the significant financial risk and governance challenges that PBGC faces will require additional congressional action. Over the years since we added PBGC to the High-Risk List, we have suggested a number of matters for congressional consideration, including: (1) authorizing a redesign of PBGC’s single employer program premium structure to better align premium rates with sponsor risk; (2) adopting additional changes to PBGC’s governance structure—in particular, expanding the composition of its board of directors; (3) strengthening funding requirements for plan sponsors as appropriate given national economic conditions; (4) working with PBGC to develop a strategy for funding PBGC claims over the long term as the defined benefit pension system continues to decline; and (5) enacting additional structural reforms to reinforce and stabilize the multiemployer system, and balance the needs and potential sacrifices of contributing employers, participants, and the federal government. Absent additional steps to improve PBGC’s finances, the long-term financial stability of the agency remains uncertain, and the retirement benefits of millions of American workers and retirees could be at risk of dramatic reductions. See page 267 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. VA operates one of the largest health care delivery systems in the nation through its Veterans Health Administration (VHA), with 172 medical centers and more than 1,000 outpatient facilities organized into regional networks. VA has faced a growing demand by veterans for its health care services—due, in part, to the needs of an aging veteran population—and that trend is expected to continue. The total number of veterans enrolled in VA’s health care system rose from 7.9 million to more than 9 million from fiscal year 2006 through fiscal year 2017. Over that same period, VHA’s total budgetary resources have more than doubled, from $37.8 billion in fiscal year 2006 to $92.3 billion in fiscal year 2017. Given the importance of VHA’s mission, coupled with its lack of progress in addressing its high-risk designation, we continue to be concerned about VHA’s ability to ensure its resources are being used effectively and efficiently to improve veterans’ timely access to safe and high-quality health care. We have identified five areas of concern: (1) ambiguous policies and inconsistent processes; (2) inadequate oversight and accountability; (3) IT challenges; (4) inadequate training for VA staff; and (5) unclear resource needs and allocation priorities. VHA has begun to address each of these areas but, prior to Secretary Robert Wilkie’s July 2018 confirmation, its efforts were impeded by leadership instability. Since taking office, Secretary Wilkie has demonstrated his commitment to addressing the department’s high-risk designation by, among other things, creating an office to direct an integrated, focused high-risk approach and communicating to VA leaders the importance of addressing our recommendations. While VHA completed root cause analyses for each area of concern and developed an action plan in response, the plan lacks milestones and metrics needed to effectively monitor its implementation and demonstrate progress made in addressing the high-risk designation. Additionally, many of VHA’s capacity-building initiatives are either in the initial stages of development or are lacking necessary funding and resources. As such, VHA has not made sufficient progress since our 2017 update to improve its overall ratings, as two high-risk criteria remain partially met and three criteria remain unmet. We remain concerned about VHA’s ability to oversee its programs, hold its workforce accountable, and avoid ambiguous policies and inconsistent processes that jeopardize its ability to provide safe, high-quality care to veterans: In November 2017, we reported that, due in part to misinterpretation or lack of awareness of VHA policy, VA medical center officials did not always document or conduct timely required reviews of providers when allegations were made against them. As a result, we concluded that VA medical center officials may have lacked necessary information to reasonably ensure that their providers were competent to provide safe, high-quality care to veterans and to grant approvals about these providers’ privileges to perform specific clinical services at VA medical centers. We made four recommendations related to this and other findings, all of which remain open. In June 2018, we reported that VHA could not systematically monitor the timeliness of veterans’ access to Veterans Choice Program (VCP) care because it lacked complete, reliable data to do so. We also found that veterans, who were referred to the VCP for routine care because health care services were not available in a timely manner, could potentially wait for care up to 70 calendar days if the maximum amount of time allowed by VA processes is used. This wait time exceeds the statutory requirement that veterans receive VCP care within 30 days of the dates their VA health care providers indicated they should receive appointments, or if no such date existed, within 30 days of the veteran’s preferred date. We made 10 recommendations related to this and other findings, all of which remain open. Similarly, in July 2018, we reported that VA collected data related to employee misconduct and disciplinary actions, but data fragmentation and reliability issues impeded department-wide analysis of those data. Additionally, we found that VA did not consistently ensure that allegations of misconduct involving senior officials were reviewed according to its investigative standards or ensure these officials were held accountable. We made 16 recommendations related to this and other findings, all of which remain open. In November 2018, we reported that VHA’s suicide prevention media outreach activities declined in recent years due to leadership turnover and reorganization. Additionally, we found that VHA did not assign key leadership responsibilities or establish clear lines of reporting for its suicide prevention media outreach campaign, which hindered its ability to oversee the campaign. Consequently, we concluded that VHA may not be maximizing its reach with suicide prevention media content to veterans, especially those who are at-risk. This is inconsistent with VHA’s efforts to reduce veteran suicides, which is VA’s highest clinical priority. We made two recommendations related to this and other findings, both of which remain open. VA needs to further develop its capacity-building initiatives and establish metrics to monitor and measure its progress addressing the high-risk areas of concern. It is also important that our recommendations continue to be implemented. The department has implemented 209 of the 353 recommendations related to VA health care that we made from January 1, 2010 through December 2018, but more than 125 recommendations remain open as of December 2018. This includes 17 that are older than 3 years. In addition to addressing our recommendations, VA needs to make systemic change to department management and oversight in order to fully address the high-risk issues and improve the health care provided to our nation’s veterans. See page 275 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Mission-critical skills gaps both within federal agencies and across the federal workforce impede the government from cost-effectively serving the public and achieving results. For example, the difficulties in recruiting and retaining skilled health care providers and human resource staff at VHA’s medical centers make it difficult to meet the health care needs of more than 9 million veterans. As a result, VHA’s 168 medical centers have large staffing shortages, including physicians, registered nurses, physician assistants, psychologists, physical therapists, as well as human resource specialists and assistants. OPM continues to demonstrate top leadership commitment through its numerous efforts to assist agencies’ in addressing mission-critical skills gaps within their workforces. This includes providing guidance, training and on-going support for agencies on the use of comprehensive data analytic methods for identifying skills gaps and the development of strategies to address these gaps. However, since we first added strategic human capital management to our High-Risk List in 2001, we have reported on the need for agencies to address their workforce skills gaps. As of December 2018, OPM had not fully implemented 29 of our recommendations made since 2012 relating to this high-risk area. Staffing shortages and the lack of skills among current staff not only affect individual agencies but also cut across the entire federal workforce in areas such as cybersecurity and acquisition management. Skills gaps caused by insufficient number of staff, inadequate workforce planning, and a lack of training in critical skills are contributing to our designating other areas as high-risk. As table 5 shows, of the 34 other high-risk areas covered in this report, skills gaps played a significant role in 16 of the areas. Over the years since we added this area to our High-Risk List, in addition to recommendations to address critical skills gaps in individual high-risk areas, we have made numerous recommendations to OPM related to this high-risk issue, 29 of which remain open. Agencies also need to take action to address mission-critical skills gaps within their own workforces – a root cause of many high-risk areas. See page 75 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The 2010 Census was the costliest in history at about $12.3 billion; as of October 2017, the 2020 Census is projected to cost about $15.6 billion, a 27 percent increase. For the 2020 Census, the U.S. Census Bureau (Bureau) plans to implement several innovations, including new IT systems. Implementing these innovations, along with other challenges, puts the Bureau’s ability to conduct a cost-effective census at risk. The decennial census is mandated by the U.S. Constitution and provides vital data for the nation. Census data are used, among other purposes, to apportion seats in the Congress and allocate billions of dollars in federal assistance to state and local governments. To ensure its success, this complicated and costly undertaking requires careful planning, risk management, and oversight. Census activities, some of which are new for the 2020 cycle, must be carried out on schedule to deliver the state apportionment counts to the President by December 31, 2020. The Bureau and the Department of Commerce (Commerce) have strengthened leadership commitment with executive-level oversight of the 2020 Census by holding regular meetings on the status of IT systems and other risk areas. In addition, in 2017 Commerce designated a team to assist senior Bureau management with cost estimation challenges. These examples demonstrate both the Bureau’s and Commerce’s strong leadership commitment to implementing the 2020 Census. One of the Bureau’s major challenges is to control any further cost growth and develop cost estimates that are reliable and reflect best practices for the 2020 Census. According to the Bureau, the total cost of the 2020 Census is now estimated to be approximately $15.6 billion, more than $3 billion higher than previously estimated by the Bureau. The higher estimated life-cycle cost is due, in part, to the Bureau’s failure to previously include all cost associated with the decennial census. The Bureau’s schedule for developing IT systems has experienced delays that have compressed the time available for system testing, integration testing, and security assessments. These schedule delays have contributed to systems experiencing problems after deployment, as well as cybersecurity challenges. For example, as of December 2018, the Bureau had identified nearly 1,100 system security weaknesses that needed to be addressed. Continued schedule management challenges may compress the time available for the remaining system testing and security assessments, and increase the risk that deployed systems will either not function as intended, have security vulnerabilities, or both. As of January 2019, 30 of our recommendations related to this high-risk area had not been implemented. To make continued progress, the Bureau needs to ensure that its approach to strategic planning, IT management, cybersecurity, human capital management, internal collaboration, knowledge sharing, as well as risk and change management are all aligned toward delivering more cost-effective outcomes. Among other things, the Bureau needs to ensure cost growth is controlled and that the development and testing of key systems is completed and fully integrated with all census operations before the 2020 Census. In addition, the Bureau needs to address cybersecurity weaknesses in a timely manner and ensure that security risks are at an acceptable level before systems are deployed. See page 134 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. An improper payment is any payment that should not have been made or that was made in an incorrect amount (including overpayments and underpayments) under statutory, contractual, administrative, or other legally applicable requirements. Reducing improper payments—such as payments to ineligible recipients or duplicate payments—is critical to safeguarding federal funds. However, the federal government has consistently been unable to determine the full extent of improper payments and reasonably assure that appropriate actions are taken to reduce them. Since 2003—when certain agencies were required by statute to begin reporting improper payments—cumulative improper payment estimates have totaled about $1.5 trillion. As shown in figure 4, for fiscal year 2018, federal entities estimated about $151 billion in improper payments. Medicare and Medicaid improper payments and the Earned Income Tax Credit (EITC) improper payments—a part of the Enforcement of Tax Laws high-risk area—accounted for about 68.5 percent of this total. Federal spending for Medicare programs and Medicaid is expected to significantly increase in the coming years, so it is especially critical to take appropriate measures to reduce improper payments in these programs. Internal Revenue Service estimates also show that the EITC has consistently had a high improper payment rate. OMB has designated Medicare programs, Medicaid, and EITC as high-priority programs for improper payments, indicating they are amongst the highest-risk programs where the government can achieve the greatest return on investment for the taxpayer by ensuring that improper payments are eliminated. Our work has identified a number of strategic and specific actions agencies can take to reduce improper payments, which could yield significant savings, and help ensure that taxpayer funds are adequately safeguarded. Continued agency attention is needed to (1) identify susceptible programs, (2) develop reliable methodologies for estimating improper payments, (3) report as required by statute, and (4) implement effective corrective actions based on root cause analysis. Absent such continued efforts, the federal government cannot be assured that taxpayer funds are adequately safeguarded. See pages 241, 250, and 235 of the report (respectively) for additional detail on the Medicare Program & Improper Payments, Strengthening Medicaid Program Integrity, and Enforcement of Tax Laws high-risk areas, including more details on actions that need to be taken. The Internal Revenue Service (IRS) continues to face two pressing challenges in enforcing tax laws: addressing the tax gap—amounting to hundreds of billions of dollars each year when some taxpayers fail to pay the taxes that they owe—and combatting identity theft (IDT) refund fraud. Enforcement of Tax Laws has been on GAO’s high risk list since 1990. IRS enforcement of tax laws helps fund the U.S. government by collecting revenue from noncompliant taxpayers and, perhaps more importantly, promoting voluntary compliance by giving taxpayers confidence that others are paying their fair share. In 2016, IRS estimated that the average annual net tax gap, the difference between taxes owed and taxes paid on time, was $406 billion, on average, for tax years 2008-2010. While IRS continues to demonstrate top leadership support to address the tax gap, IRS’s capacity to implement new initiatives and improve ongoing enforcement and taxpayer service programs remains a challenge. For example, IRS’s strategic plan includes a goal to facilitate voluntary compliance and deter noncompliance that could address the tax gap. However, IRS could do more to identify specific efforts for improving compliance in its strategic plan, measure the effects of compliance programs—such as those used for large partnerships—and develop specific quantitative goals to reduce the tax gap. Such efforts would help IRS make more effective use of its resources and gauge the success of its strategies. The second challenge facing IRS is IDT refund fraud, which occurs when an identity thief files a fraudulent tax return using a legitimate taxpayer’s identifying information and claims a refund. IRS estimates that at least $12.2 billion in individual IDT tax refund fraud was attempted in 2016, of which it prevented at least $10.5 billion (86 percent). Of the amount attempted, IRS estimated that at least $1.6 billion (14 percent) was paid. IRS’s ability to combat IDT fraud continues to be challenged as more personally identifiable information has become readily available as a result of large-scale cyberattacks on various entities. This makes it more difficult for IRS to distinguish between fraudsters and legitimate taxpayers. While IRS has demonstrated some progress by developing tools and programs to further detect and prevent IDT refund fraud, it has not completed updating its authentication procedures to be in compliance with new government standards. As a result, IRS may be missing an opportunity to implement the most secure, robust technologies to protect taxpayers. As of December 2018, 189 GAO recommendations related to this high- risk area had not been implemented. To make continued progress on closing the tax gap, IRS needs to re-establish goals for improving voluntary compliance and develop and document a strategy that outlines how it will use its data to help address this issue. Reducing the tax gap will also require targeted legislative actions, including additional third- party information reporting, enhanced electronic filing, expanded math error authority (also referred to as correctible error authority), and paid preparer regulation. To help stay on top of IDT refund fraud, IRS should develop a comprehensive process to evaluate alternative options for improving taxpayer authentication. Given that IDT refund fraud continues to be a challenge, targeted legislative action, such as requiring a scannable code on returns prepared electronically but filed on paper could help IRS address such fraud. See page 235 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. The federal government currently invests more than $90 billion annually in IT, and OMB has implemented several key initiatives intended to help better manage this investment. Additionally, enactment of FITARA, in conjunction with greater attention paid to the acquisition and operation of IT, has helped further improve the government-wide management of this significant annual investment. OMB’s current level of top leadership support and commitment to ensure that agencies successfully execute its guidance on implementing FITARA and related IT initiatives has helped this high-risk area meet the leadership commitment high-risk criteria. Additional positive government-wide actions have enabled this high-risk area to partially meet the four remaining high-risk criteria. For example, OMB has established an IT Dashboard—a public website that provides detailed information on major IT investments at 26 federal agencies—and agencies’ data center consolidation efforts have resulted in a total savings of slightly more than 80 percent of the agencies’ planned $5.7 billion in savings since 2011. However, major federal agencies have yet to fully address the requirements of FITARA and realize billions of dollars in planned or possible savings and improved government performance through more efficient budgeting and management of IT. As government-wide spending on IT increases every year, the need for appropriate stewardship of that investment increases as well. However, OMB and federal agencies have not made significant progress since 2017 in taking the steps needed to improve how these financial resources are budgeted and utilized. While OMB has continued to demonstrate its leadership commitment through guidance and sponsorship of key initiatives, agencies still have not fully implemented all requirements of FITARA, such as putting into place authorities the law requires for chief information officers (CIO). Additionally, while the President’s Management Agenda has a goal to improve IT spending transparency, agencies are underreporting IT contract obligations by billions of dollars. OMB and the agencies also have not yet implemented hundreds of our recommendations on improving shortcomings in IT acquisitions and operations. In an August 2018 review of the 24 federal agencies covered by FITARA, none had IT management policies that fully addressed the role of their CIOs consistent with federal laws and guidance. Specifically, the majority of the agencies only minimally addressed, or did not address, their CIO’s role in assessing agency IT workforce needs and developing strategies and plans for meeting those needs. Correspondingly, the majority of the 24 CIOs acknowledged that they were not fully effective at implementing IT management responsibilities, such as IT strategic planning and investment management. Further, in January 2018, we reported that the majority of 22 agencies did not identify all of their IT acquisition contracts, totaling about $4.5 billion in IT-related contract obligations beyond those reported by agencies. In addition, in November 2018 we reported that four selected agencies lacked quality assurance processes for ensuring that billions of dollars requested in their IT budgets were informed by reliable cost information. Until agencies properly identify IT contracts and establish processes for ensuring the quality of cost data used to inform their budgets, agency CIOs are at risk of not having appropriate oversight of IT acquisitions and may lack adequate transparency into IT spending to make informed budget decisions. As of December 2018, OMB and federal agencies had fully implemented only 59 percent of the recommendations we have made since fiscal year 2010 to address shortcomings in IT acquisitions and operations. OMB and agencies should work toward implementing our remaining 456 open recommendations related to this high-risk area. These remaining recommendations include 12 priority recommendations to agencies to, among other things, report all data center consolidation cost savings to OMB, plan to modernize or replace obsolete systems as needed, and improve their implementation of PortfolioStat—an initiative that is to consolidate and eliminate duplicative systems. OMB and agencies need to take additional actions to (1) implement at least 80 percent of our open recommendations related to the management of IT acquisitions and operations, (2) ensure that a minimum of 80 percent of the government’s major IT acquisitions deliver functionality every 12 months, and (3) achieve at least 80 percent of the over $6 billion in planned PortfolioStat savings. See page 123 of the report for additional detail on this high-risk area, including more details on actions that need to be taken. Our high-risk program continues to be a top priority at GAO and we will maintain our emphasis on identifying high-risk issues across government and on providing recommendations and sustained attention to help address them, by working collaboratively with Congress, agency leaders, and OMB. As part of this effort, we hope to continue to participate in regular meetings with the OMB Deputy Director for Management and with top agency leaders to discuss progress in addressing high-risk areas. Such efforts have been critical for the progress that has been made. This high-risk update is intended to help inform the oversight agenda for the 116th Congress and to guide efforts of the administration and agencies to improve government performance and reduce waste and risks. Thank you, Chairman Johnson, Ranking Member Peters, and Members of the Committee. This concludes my testimony. I would be pleased to answer any questions. For further information on this testimony, please contact J. Christopher Mihm at (202) 512-6806 or MihmJ@gao.gov. Contact points for the individual high-risk areas are listed in the report and on our high-risk website. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. The following pages provide summaries of selected high-risk areas. These summaries are included in our High-Risk Report and are also available on our High-Risk List website, http://www.gao.gov/highrisk/overview. Embassy Construction: Pace is Slower Than Projected, and State Could Make Program Improvements. GAO-18-653. Washington, D.C.: September 25, 2018. Tax Administration: Opportunities Exist to Improve Monitoring and Transparency of Appeal Resolution Timeliness. GAO-18-659. Washington, D.C.: September 21, 2018. Information Technology: IRS Needs to Take Actions to Address Significant Risks to Tax Processing. GAO-18-298. Washington, D.C.: June 28, 2018. Cybersecurity Workforce: Agencies Need to Improve Baseline Assessments and Procedures for Coding Positions. GAO-18-466. Washington, D.C.: June 14, 2018. Defense Acquisition Workforce: Opportunities Exist to Improve Practices for Developing Program Managers. GAO-18-217. Washington, D.C.: February 15, 2018. Cybersecurity Workforce: Urgent Need for DHS to Take Actions to Identify Its Position and Critical Skill Requirements. GAO-18-175. Washington, D.C.: February 6, 2018. Bureau of Prisons: Better Planning and Evaluation Could Help Ensure Effective Use of Retention Incentives. GAO-18-147. Washington, D.C.: December 7, 2017. National Weather Service: Actions Have Been Taken to Fill Increasing Vacancies, but Opportunities Exist to Improve and Evaluate Hiring. GAO-17-364. Washington, D.C.: May 24, 2017. Strategic Human Capital Management: NRC Could Better Manage the Size and Composition of Its Workforce by Further Incorporating Leading Practices. GAO-17-233. Washington, D.C.: April 27, 2017. Veterans Health Administration: Actions Needed to Better Recruit and Retain Clinical and Administrative Staff. GAO-17-475T. Washington, D.C.: March 22, 2017. The ratings for capacity and action plan improved since our 2017 High-Risk Report and the remaining three criteria remain unchanged. Leadership commitment: met. OMB and GSA continue to take action to reduce costly leasing. For example, OMB proposed the creation of a capital revolving fund designed to facilitate ownership over operating leases for large-dollar buildings, although no action has been taken to implement it. An OMB staff member said that the legislative proposal to establish a capital fund was similar to an option we identified in a 2014 report. Additionally, GSA has developed a strategy to reduce leasing costs by a projected $4.7 billion by fiscal year 2023, through steps that include focusing resources on high-value lease renewals. Capacity: partially met. GSA made improvements and now partially meets the capacity criterion. Specifically, GSA implemented our September 2013 recommendation to develop a strategy to increase ownership investments for a prioritized list of high-value leases. These leases are for properties where it would be less expensive in the long run to own. GSA plans to purchase at least one leased building in 2019. In addition, as noted in our 2017 high-risk update, GSA could potentially help tenant agencies save millions of dollars from some leases by loaning them funds to improve newly leased spaces instead of agencies financing these costs with private-sector owners at private-sector interest rates. While GSA officials agreed that doing so would save money in interest fees, it has not yet developed a legislative proposal to obtain the needed authority, as we recommended in 2016. Action plan: met. GSA has made improvements and now meets the action plan criterion. GSA created an action plan to purchase buildings when it is more cost-effective than leasing by establishing criteria to rank and prioritize leased spaces that would benefit from federal ownership as discussed above. Additionally, GSA is implementing strategies to better manage leases that include avoiding short-term extensions and identifying opportunities to enter into long-term and lower cost leases. Monitoring: partially met. GSA continues to partially meet this criterion through implementation of the National Strategy, as noted in our 2017 high-risk update. However, GSA should also implement our recommendations to reduce the costs to tenants by exploring strategies to enhance competition for GSA leases and reducing unneeded fees. Additionally, GSA has identified actions to better monitor leases at different points along the process in order to minimize the need to enter into short-term, costly lease extensions. Demonstrated progress: partially met. GSA has made some progress in reducing the long-term costs of leasing by stemming the growth in leasing according to GSA data and committing to further reducing leasing costs. However, GSA must follow through on its plans to purchase leased buildings and reduce costs. GSA could also further reduce costs by loaning tenant agencies the funds needed to improve newly leased spaces but still needs to develop a legislative proposal to obtain authority to do so. GSA should develop a legislative proposal to obtain authority to loan agencies funds needed to improve newly leased spaces, as we recommended in 2016. Ratings for one criterion improved since our 2017 High-Risk Report and the other four criteria remain unchanged. Leadership commitment: met. In December 2017, GSA continued efforts to improve data reliability by completing a major effort to make the Federal Real Property Profile (FRPP) public. Also, as we reported in our 2017 High-Risk Report, GSA issued its Federal Real Property Data Validation and Verification (V&V) Guidance in May 2016 and required agencies to address 13,257 data anomalies it found in fiscal year 2016 data. Capacity: met. OMB and GSA continue to help agencies’ increase their capacity to submit accurate data. For example, GSA revised certain data elements’ definitions in 2016 and incorporated them in the 2018 FRPP Data Dictionary. In addition, OMB and GSA have further increased the capacity of FRPP to act as a government-wide database since additional agencies are required to report. Action plan: met. GSA has made progress by developing an action plan in 2017 for federal agencies to develop processes to assess, address, and track FRPP data quality. Specifically, this plan identifies data elements to appropriately indicate data quality, identifies best practices and other methods that help agencies measure and assess improvements, and enables federal agencies to develop performance metrics. Monitoring: partially met. While GSA required agencies to research the anomalies it found in its V&V process, only some agencies have identified and committed to correct mistakes. Further, of the 13,257 anomalies GSA identified in the fiscal year 2016 data, agencies overall acknowledged that less than 8 percent of the anomalies (1,004 anomalies) represented erroneous data to be corrected, while indicating that the others were correct. Furthermore, some agencies acknowledged less than 1 percent of the anomalies represented erroneous data. In addition, we found in 2018 that DOD did not correct discrepancies identified by its own V&V process. Demonstrated progress: partially met. While GSA and some agencies have taken action to correct data, serious data reliability challenges remain with some individual agencies that undermine the reliability of the FRPP. In 2018, we found that DOD’s real property data continue to be inaccurate and incomplete, and that DOD lacks a plan for making the necessary improvements. OMB and GSA should continue working with federal agencies to improve the reliability of their real property data through V&V efforts and encouraging agencies to implement action plans to better assess, address, and track data quality, as discussed in the above action plan. In particular, DOD should take steps to ensure that DOD improves the reliability of its real property data, as we recommended in 2018. Ratings for this segment remain unchanged since our 2017 High-Risk Report. Leadership commitment: met. DHS’s Federal Protective Service (FPS) continues to take action to address our recommendations. The Interagency Security Committee (ISC), an organization chaired by DHS that sets standards for physical security for federal nonmilitary facilities, also continues to implement the updated Risk Management Process—a consolidated set of standards for physical security at federal facilities. In addition, in 2018, GSA, the Administrative Office of the U. S. Courts (AOUSC), the U.S. Marshals Service, and FPS implemented our 2017 recommendation to establish a national-level working forum for courthouse security, known as the Interagency Judicial Security Council. Capacity: partially met. FPS has taken several actions to address identified physical security issues since our 2017 High-Risk Report. For example, in 2018 FPS improved its risk assessment tool to incorporate all necessary elements recommended by the ISC, which has now certified it. In 2018, FPS also addressed our recommendation related to improving training for instructors and identified actions to address our recommendations associated with tracking guard training. Finally, in 2018, FPS also implemented several actions associated with our recommendation to develop human capital-related performance measures to evaluate progress towards agency goals. Some agencies may not have the capacity to conduct adequate risk assessments because their processes do not fully align with the ISC Risk Management Process. To improve their capacity, the U.S. Customs and Border Protection, Federal Aviation Administration, and the Department of Veterans’ Affairs still need to complete an assessment of their policies against the ISC’s standards in response to our 2017 and 2018 recommendations. Action plan: partially met. In September 2018, FPS and GSA signed a memorandum of agreement (MOA) clarifying their respective roles and responsibilities for federal facility security. However, FPS, GSA, and the Department of Justice have not yet addressed our 2011 recommendation to address a number of courthouse security challenges. Specifically, FPS, the U.S. Marshals Service, AOUSC, and GSA are still working to finalize the draft MOA on courthouse security. Monitoring: partially met. FPS continues to develop a system that will allow FPS to verify independently that FPS’s contract guards are current on all training and certification requirements, and are taking steps to close this recommendation as implemented. FPS expects that system to be in place in 2019. In 2018, we also found that actions were needed to better address various emerging security threats to federal facilities. Demonstrated progress: not met. The federal government has not demonstrated progress to improve physical security. Although agencies have taken some actions, time is needed for agencies to demonstrate the results of these actions. Additionally, agencies need to complete other actions. For example, once FPS, the U.S. Marshals Service, AOUSC, and GSA sign their MOA on courthouse security, they will be able to better protect federal facilities. Further, once FPS fully implements its guard management system and it interacts with its training system, FPS will be able to obtain information to assess its guards’ capability to address physical security risks across its portfolio. To improve the physical security of federal buildings, the following steps are necessary: Clarify roles and responsibilities for the protection of federal facilities by finalizing the MOA for federal courthouse security between GSA, FPS, the U.S. Marshals, and AOUSC, as we recommended in 2011. FPS must validate training information being entered to ensure that guards are getting critical training, as we recommended in 2012. Implement our recommendations for agencies to improve their monitoring of collaborative efforts to protect federal facilities, as we recommended in 2015. Take actions to better address emerging security threats to federal facilities, as we recommended in 2018. Federal Facility Security: Actions Needed to Better Address Various Emerging Threats. GAO-19-32SU. Washington, D.C.: October 17, 2018. Defense Real Property: DOD Needs to Take Additional Actions to Improve Management of Its Inventory Data. GAO-19-73. Washington, D.C.: November 13, 2018. Federal Buildings: More Consideration of Operations and Maintenance Costs Could Better Inform the Design Excellence Program. GAO-18-420. Washington, D.C.: May 22, 2018. Federal Real Property: Agencies Make Some Use of Telework in Space Planning but Need Additional Guidance. GAO-18-319. Washington, D.C.: March 22, 2018. Federal Buildings: Agencies Focus on Space Utilization As They Reduce Office and Warehouse Space. GAO-18-304. Washington, D.C.: March 8, 2018. VA Facility Security: Policy Review and Improved Oversight Strategy Needed. GAO-18-201. Washington, D.C.: January 11, 2018. Federal Facility Security: Selected Agencies Should Improve Methods for Assessing and Monitoring Risk. GAO-18-72. Washington, D.C.: October 26, 2017. Federal Real Property: GSA Should Inform Tenant Agencies When Leasing High-Security Space from Foreign Owners. GAO-17-195. Washington, D.C.: January 3, 2017. Postal Retiree Health Benefits: Unsustainable Finances Need to Be Addressed. GAO-18-602. Washington, D.C: August 31, 2018. U.S. Postal Service: Projected Capital Spending and Processes for Addressing Uncertainties and Risks. GAO-18-515. Washington, D.C.: June 28, 2018. International Mail: Information on Changes and Alternatives to the Terminal Dues System. GAO-18-112. Washington, D.C.: October 12, 2017. U.S. Postal Service: Key Considerations for Potential Changes to USPS’s Monopolies. GAO-17-543. Washington, D.C.: June 22, 2017. U.S. Postal Service: Key Considerations for Restoring Fiscal Sustainability. GAO-17-404T. Washington, D.C.: February 7, 2017. U.S. Postal Service: Continuing Financial Challenges and the Need for Postal Reform. GAO-16-651T. Washington, D.C.: May 11, 2016. U.S. Postal Service: Financial Challenges Continue. GAO-16-268T. Washington, D.C.: January 21, 2016. Information Technology: Departments Need to Improve Chief Information Officers’ Review and Approval of IT Budgets. GAO-19-49. Washington, D.C.: November 13, 2018. Federal Chief Information Officers: Critical Actions Needed to Address Shortcomings and Challenges in Implementing Responsibilities. GAO-18-93. Washington, D.C.: August 2, 2018. Data Center Optimization: Continued Agency Actions Needed to Meet Goals and Address Prior Recommendations. GAO-18-264. Washington, D.C.: May 23, 2018. Information Technology: Agencies Need to Involve Chief Information Officers in Reviewing Billions of Dollars in Acquisitions. GAO-18-42. Washington, D.C.: January 10, 2018. Information Technology: OMB Needs to Report On and Improve Its Oversight of the Highest Priority Programs. GAO-18-51. Washington, D.C.: November 21, 2017. Information Technology Reform: Agencies Need to Improve Certification of Incremental Development. GAO-18-148. Washington, D.C.: November 7, 2017. Data Center Optimization: Agencies Need to Address Challenges and Improve Progress to Achieve Cost Savings Goal. GAO-17-448. Washington, D.C.: August 15, 2017. Data Center Optimization: Agencies Need to Complete Plans to Address Inconsistencies in Reported Savings. GAO-17-388. Washington, D.C.: May 18, 2017. Information Technology: Opportunities for Improving Acquisitions and Operations. GAO-17-251SP. Washington, D.C.: April 11, 2017. 2020 Census: Additional Steps Needed to Finalize Readiness for Peak Field Operations, GAO-19-140. Washington, D.C.: December 10, 2018. 2020 Census: Continued Management Attention Needed to Address Challenges and Risks with Developing, Testing, and Securing IT Systems, GAO-18-655. Washington, D.C.: August 30, 2018. 2020 Census: Census Bureau Improved the Quality of Its Cost Estimation but Additional Steps Are Needed to Ensure Reliability, GAO-18-635. Washington, D.C.: August 17, 2018. 2020 Census: Bureau Has Made Progress with Its Scheduling, but Further Improvement Will Help Inform Management Decisions, GAO-18-589. Washington, D.C.: July 26, 2018. 2020 Census: Actions Needed to Address Challenges to Enumerating Hard-to-Count Groups, GAO-18-599. Washington, D.C.: July 26, 2018. 2020 Census: Actions Needed to Improve In-Field Address Canvassing Operation, GAO-18-414. Washington, D.C.: June 14, 2018. 2020 Census: Actions Needed to Mitigate Key Risks Jeopardizing a Cost- Effective and Secure Enumeration, GAO-18-543T. Washington, D.C.: May 8, 2018. 2020 Census: Continued Management Attention Needed to Mitigate Key Risks Jeopardizing a Cost-Effective and Secure Enumeration, GAO-18-416T. Washington, D.C.: April 18, 2018. 2020 Census: Actions Needed to Mitigate Key Risks Jeopardizing a Cost- Effective Enumeration, GAO-18-215T. Washington, D.C.: October 31, 2017. 2020 Census: Continued Management Attention Needed to Oversee Innovations, Develop and Secure IT Systems, and Improve Cost Estimation, GAO-18-141T. Washington, D.C.: October 12, 2017. 2020 Census: Bureau Is Taking Steps to Address Limitations of Administrative Records, GAO-17-664. Washington, D.C.: July 26, 2017. Personnel Security Clearances: Additional Actions Needed to Implement Key Reforms and Improve Timely Processing of Investigations. GAO-18- 431T. Washington, D.C.: March 7, 2018. Personnel Security Clearances: Additional Actions Needed to Ensure Quality, Address Timeliness, and Reduce Investigation Backlog. GAO-18- 29. Washington, D.C.: December 12, 2017. Personnel Security Clearances: Plans Needed to Fully Implement and Oversee Continuous Evaluation of Clearance Holders. GAO-18-117. Washington, D.C.: November 21, 2017. Information Security: OPM Has Improved Controls, but Further Efforts Are Needed. GAO-17-614. Washington, D.C: August 3, 2017. Information Security: Agencies Need to Improve Controls over Selected High-Impact Systems. GAO-16-501. Washington, D.C.: May 18, 2016. Personnel Security Clearances: Funding Estimates and Government-wide Metrics Are Needed to Implement Long-Standing Reform Efforts. GAO- 15-179SU. Washington, D.C.: April 23, 2015. Information Security: OPM Has Implemented Many of GAO’s 80 Recommendations, but Over One-Third Remain Open. GAO-19-143R. Washington, D.C.: November 13, 2018. Cybersecurity: Office of Federal Student Aid Should Take Additional Steps to Oversee Non-School Partners’ Protection of Borrower Information. GAO-18-518. Washington, D.C.: September 17, 2018. High-Risk Series: Urgent Actions Are Needed to Address Cybersecurity Challenges Facing the Nation. GAO-18-622. Washington, D.C.: September 6, 2018. Information Security: IRS Needs to Rectify Control Deficiencies That Limit Its Effectiveness in Protecting Sensitive Financial and Taxpayer Data. GAO-18-391. Washington, D.C.: July 31, 2018. Data Protection: Actions Taken by Equifax and Federal Agencies in Response to the 2017 Breach. GAO-18-559. Washington, D.C.: August 30, 2018. High-Risk Series: Urgent Actions Are Needed to Address Cybersecurity Challenges Facing the Nation. GAO-18-645T. Washington, D.C.: July 25, 2018. Information Security: Supply Chain Risks Affecting Federal Agencies. GAO-18-667T. Washington, D.C.: July 12, 2018. Electronic Health Information: CMS Oversight of Medicare Beneficiary Data Security Needs Improvement. GAO-18-210. Washington, D.C.: March 6, 2018. Critical Infrastructure Protection: Additional Actions Are Essential for Assessing Cybersecurity Framework Adoption. GAO-18-211. Washington, D.C.: February 15, 2018. Cybersecurity Workforce: Urgent Need for DHS to Take Actions to Identify Its Position and Critical Skill Requirements. GAO-18-175. Washington, D.C.: February 6, 2018. DHS Acquisitions: Additional Practices Could Help Components Better Develop Operational Requirements. GAO-18-550 Washington, D.C.: August 8, 2018. Homeland Security Acquisitions: Leveraging Programs’ Results Could Further DHS’s Progress to Improve Portfolio Management. GAO-18-339SP Washington, D.C.: May 17, 2018. Cybersecurity Workforce: Urgent Need for DHS to Take Actions to Identify Its Position and Critical Skill Requirements. GAO-18-175, Washington, D.C.: February 6, 2018. DHS Financial Management: Better Use of Best Practices Could Help Manage System Modernization Project Risks. GAO-17-799 Washington, D.C.: September 26, 2017. Homeland Security: Progress Made to Implement IT Reform, but Additional Chief Information Officer Involvement Needed. GAO-17-284 Washington, D.C.: May 18, 2017. Homeland Security Acquisitions: Identifying All Non-Major Acquisitions Would Advance Ongoing Efforts to Improve Management. GAO-17-396 Washington, D.C.: April 13, 2017. Homeland Security Acquisitions: Earlier Requirements Definition and Clear Documentation of Key Decisions Could Facilitate Ongoing Progress. GAO-17-346SP Washington, D.C.: April 6, 2017. Error! No text of specified style in document. The following pages provide overviews of the two areas removed from the High-Risk List. Each overview discusses (1) why the area was high risk, and (2) why the area is being removed from the list. Each of these high- risk areas is also described on our High-Risk List website, http://www.gao.gov/highrisk/overview. Since our 2017 High-Risk Report, DOD has continued to meet the criteria of leadership commitment, capacity, and action plan for asset visibility. Further, DOD has fully addressed the three remaining actions and outcomes we outlined in 2017 in order to mitigate or resolve long-standing weaknesses in asset visibility. Consequently, DOD has met the monitoring and demonstrated progress criteria for asset visibility to remove this area from our High-Risk List. Leadership commitment: met. Senior leaders have continued to demonstrate commitment through their involvement in groups such as the Supply Chain Executive Steering Committee—senior-level officials responsible for overseeing asset visibility improvement efforts—and through the Asset Visibility Working Group, which identifies opportunities for improvement and monitors the implementation of initiatives by issuing its Strategy for Improving DOD Asset Visibility (Strategy) in 2014, 2015, and 2017. Capacity: met. DOD continues to demonstrate that it has the capacity— personnel and resources—to improve asset visibility. For example, DOD’s 2015 and 2017 Strategies advise the components to consider items such as staffing, materiel, and sustainment costs when documenting cost estimates for the initiatives in the Strategy, as we recommended in January 2015. Action plan: met. A provision in the National Defense Authorization Act for Fiscal Year 2014 required DOD to submit to Congress a comprehensive strategy and implementation plans for improving asset tracking and in-transit visibility. In January 2014, DOD issued the Strategy and accompanying implementation plans, which outlined initiatives intended to improve asset visibility. DOD updated its 2014 Strategy in October 2015 and in August 2017. Importantly, since 2017 DOD addressed the three remaining actions and outcomes related to the monitoring and demonstrated progress criteria through updates to and implementation of the Strategies (see table 7). Monitoring: met. DOD provided guidance in its 2017 update to the Strategy for the military components to consider key attributes of successful performance measures during metric development for their improvement initiatives. As appropriate, the military components have followed the guidance and provided high-level summary metrics updates to the Asset Visibility Working Group. In addition, DOD has taken steps to monitor asset visibility by incorporating into after-action reports, as appropriate, information relating to performance measures. These after- action reports serve as closure documents and permanent records of each initiative’s accomplishments. Demonstrated progress: met. DOD has demonstrated sustained progress by completing 34 of the 39 initiatives to improve asset visibility and continues to monitor the remaining 5 initiatives. These initiatives have supported DOD’s goals and objectives, which include: (1) improving visibility efficiencies of physical inventories, receipt processing, cargo tracking, and unit moves; (2) ensuring asset visibility data are discoverable, accessible, and understandable to support informed decision-making across the enterprise; and (3) increasing efficiencies for delivery accuracy and cycle times. Also, the Asset Visibility Working Group meets regularly to identify opportunities to further improve asset visibility within DOD. DOD has taken the following actions to demonstrate sustained progress: (1) created an integrated single portal system providing 7,500 users access to near-real-time, in-transit visibility of eight million lines of items of supply and transportation data; and (2) increased its visibility of assets through radio-frequency identification (RFID), an automated data-capture technology that can be used to electronically identify, track, and store information contained on a tag. There are two main types of RFID tags, passive and active, which show whether assets are in-storage, in-transit, in-process, or in-use. Passive tags, such as mass transit passes, do not contain their own power source and cannot initiate communication with a reader; while active tags, such as an “E-Z pass,” contain a power source and a transmitter, and send a continuous signal over longer distances. DOD closed nine initiatives from its Strategies by implementing RFID technology. For example, the Marine Corps implemented long-range passive RFID for visibility and accountability of items, resulting in improvements that include an increased range for “reading” an item— from 30 feet to 240 feet—and reduced inventory cycle times from 12 days to 10 hours. Also, the Navy reported that the use of passive RFID technology to support the overhaul of its nuclear-powered attack submarines enabled the Navy to better track parts, resulting in 98 percent fewer missing components and an average cost avoidance of $1.3 million per boat. Additionally, according to DOD, the use of RFID tags to provide visibility of sustainment cargo at the tactical leg resulted in $1.4 million annual cost savings. Further, DOD reported that the migration of the active RFID enterprise from a proprietary communication standard to a competitive multivendor environment reduced the cost of active RFID tags by half, resulting in an estimated $5.7 million annual reduction in costs. Since our 2017 High-Risk Report, DOD has continued to meet the criteria of leadership commitment, capacity, and action plan for materiel distribution. Further, DOD has fully addressed the four remaining actions and outcomes we outlined in 2017 in order to mitigate or resolve long-standing weaknesses in materiel distribution. Consequently, DOD has met the monitoring and demonstrated progress criteria for materiel distribution to remove this area from our High-Risk List. Leadership commitment: met. Senior leaders continue to demonstrate commitment through their involvement in groups such as the Supply Chain Executive Steering Committee—senior-level officials responsible for overseeing materiel distribution corrective actions—and through the Distribution Working Group, which helped develop the Materiel Distribution Improvement Plan (Improvement Plan) in 2016. Capacity: met. DOD has continued to demonstrate that it has the personnel and resources, such as key organizations and the associated governance structure, to improve materiel distribution. The Improvement Plan recognizes that additional resources will be required to accomplish its corrective actions and close any identified performance gaps within the time frame specified. Action plan: met. In 2016, DOD developed its corrective action plan to address the department’s materiel distribution challenges. The Improvement Plan details specific goals and actions to better measure the end-to-end distribution process, ensure the accuracy of underlying data, and strengthen and integrate distribution policies and the governance structure. Importantly, since 2017, DOD has fully addressed the four remaining actions and outcomes related to monitoring and demonstrated progress to mitigate or resolve long-standing weaknesses in materiel distribution (see table 8). Monitoring: met. DOD has monitored materiel distribution by making progress in developing its suite of distribution performance metrics, improving the quality of their underlying data, and sharing metrics information with stakeholders. For example, in January 2017, DOD developed a suite of performance metrics that provides a comprehensive picture of the distribution process, including whether supplies are delivered on time and at sufficient quantity and quality. Also, DOD implemented checklists to assess the quality of data underlying each performance metric based on relevance, accuracy, comparability, and interpretability. The checklists and their standards assist in identifying root causes and addressing areas where performance data quality may be lacking. DOD has also incorporated internal control requirements in its supply chain management guidance to increase confidence in the performance data. Additionally, DOD has revised its policy documents to require stakeholders to routinely capture and share distribution performance metrics, including cost data, and the department maintains websites to provide current performance information to distribution stakeholders. DOD has also incorporated distribution metrics, as appropriate, on the performance of all legs of the distribution system, including the tactical leg (i.e., the last segment of the distribution system). We previously reported on DOD’s deficiencies to accurately assess its distribution performance at the tactical leg, such as missing delivery dates for shipments in Afghanistan. Since that time, the geographic combatant commands have been tracking metrics at the tactical leg, including required delivery dates, to determine the movement and causes of delays for shipments, and have been sharing distribution performance information with the U.S. Transportation Command (TRANSCOM) through their deployment and distribution operations centers. DOD is implementing a cost framework to incorporate transportation costs for all legs of the distribution system, which will provide an additional metric for distribution stakeholders to assess the efficiency of the system. The first phase of the cost framework began operating in August 2018 and is expected to be fully implemented in 2019. DOD is making progress in refining its Improvement Plan and is incorporating additional actions based on interim progress and results. Since DOD issued the Improvement Plan in September 2016, the agency has (1) documented the results and monitored the status of each corrective action, (2) revised completion dates as needed, and (3) periodically provided decision makers with summary action charts, plans, and milestones. DOD is also updating its instruction on management and oversight of the distribution enterprise to clarify the roles and responsibilities of all distribution stakeholders. DOD officials have not determined a date for when this instruction will be issued. Demonstrated progress: met. DOD has demonstrated sustained progress in improving its capability to comprehensively measure distribution performance, identify distribution problems and root causes, and implement solutions. DOD has implemented 10 of 18 corrective actions in its Improvement Plan and is on track to implement the remaining 8 by September 2019. Because of this progress, DOD’s monthly shipment reports have assessed performance against enhanced metrics across the distribution system. For example, in December 2017, TRANSCOM investigated performance standards for truck deliveries from its Defense Logistics Agency warehouses in Bahrain to customers in Kuwait due to frequent delays in shipments. TRANSCOM determined that inadequate time for clearing customs in Kuwait resulted in an unrealistic delivery standard. TRANSCOM, in coordination with distribution stakeholders, adjusted the delivery standard to adequately account for the in-theater customs process. In addition, TRANSCOM, in partnership with the Defense Logistics Agency and the General Services Administration, developed and implemented initiatives focused on distribution process and operational improvements to reduce costs and improve distribution services to the warfighter. According to DOD, these efforts have resulted in at least $1.56 billion in distribution cost avoidances to date. DOD has demonstrated commendable, sustained progress improving its supply chain management. This does not mean DOD has addressed all risk within this area. It remains imperative that senior leaders continue their efforts to implement initiatives and corrective actions to maintain visibility of supplies, track cargo movements, meet delivery standards, and maintain delivery data for shipments. Continued oversight and attention are also warranted given the recent reorganization of the Office of the Under Secretary of Defense for Acquisition and Sustainment and the resulting change in the oversight structure of Supply Chain Management. We will therefore continue to conduct oversight of supply chain management at DOD. Defense Logistics: Improved Performance Measures and Information Needed for Assessing Asset Visibility Initiatives. GAO-17-183. Washington, D.C.: Mar. 16, 2017. Defense Logistics: DOD Has Addressed Most Reporting Requirements and Continues to Refine its Asset Visibility Strategy. GAO-16-88. Washington, D.C.: Dec. 22, 2015. Defense Logistics: Improvements Needed to Accurately Assess the Performance of DOD’s Materiel Distribution Pipeline. GAO-15-226. Washington, D.C.: Feb. 26, 2015. Since our last high-risk update in 2017, NOAA continues to meet the criteria of leadership commitment, capacity, and monitoring and now also meets the criteria of action plan and demonstrated progress. Leadership commitment: met. NOAA program officials met the leadership commitment criteria in 2015 and have continued to sustain their strong leadership commitment to mitigating potential satellite data gaps since that time. For example, NOAA issued and frequently updated its polar satellite gap mitigation plan, which identifies the specific technical, programmatic, and management steps the agency is taking to ensure that satellite mitigation options are viable. In addition, NOAA executives continue to oversee the acquisition of polar-orbiting satellites through monthly briefings on the cost, schedule, and risks affecting the satellites’ development. Capacity: met. NOAA continues to meet the criterion of improving its capacity to address the risk of a satellite data gap. In December 2014, we recommended that NOAA investigate ways to prioritize the gap mitigation projects with the greatest potential benefit to weather forecasting, such as by improving its high-performance computing capacity. NOAA agreed with this recommendation and implemented it. For example, NOAA upgraded its high-performance computers, which allowed the agency to move forward on multiple other mitigation activities, including experimenting with other data sources and assimilating these data into its weather models. and impacts, and (4) establishing a schedule with meaningful timelines and linkages among mitigation activities. The agency agreed with the recommendation and subsequently addressed it. Specifically, NOAA issued three updates to its gap mitigation plan between January 2016 and February 2017. With the last of the updates, the agency addressed the shortfalls we had identified. Monitoring: met. NOAA met this criterion in 2017, and continues to meet it now, by implementing our recommendations to more consistently and comprehensively monitor its progress on gap mitigation activities. For example, all three NOAA organizations responsible for gap mitigation projects regularly brief senior management on their progress. Demonstrated progress: met. NOAA now meets the criterion for demonstrated progress, which is an increase over its prior rating. In our 2017 High-Risk Report, we noted that NOAA had identified 35 different gap mitigation projects and was making progress in implementing them. These projects fell into three general categories: (1) understanding the likelihood and impact of a gap, (2) reducing the likelihood of a gap, and (3) reducing the impact of a gap. Nevertheless, one of the most important steps in reducing the likelihood of a gap—keeping the launch of the next polar satellite on schedule—had encountered problems. Specifically, agency officials decided to delay the launch due to challenges in developing the ground system and a critical instrument on the spacecraft. This delay exacerbated the probability of a satellite data gap. More recently, however, NOAA was able to demonstrate progress by successfully launching the satellite in November 2017. That satellite, now called NOAA-20, is currently operational and is being used to provide advanced weather data and forecasts. Moreover, the agency is also working to build and launch the next satellites in the polar satellite program. Since our last high-risk update in 2017, DOD now meets all five high-risk criteria. Fiscal Year 2015 (NDAA for FY 2015), the National Defense Authorization Act for Fiscal Year 2016 (NDAA for FY 2016), and the Consolidated Appropriations Act, 2016, DOD leadership committed to developing and implementing plans to address its weather satellite requirements. For example, in late 2017, the department awarded a contract for its Weather System Follow-on—Microwave satellite to fulfill core weather requirements. Capacity: met. With strong congressional oversight, DOD now meets the capacity criterion. Specifically, the NDAA for FY 2015 restricted the availability of 50 percent of the FY 2015 funds authorized for the Weather Satellite Follow-on System (now called the Weather System Follow-on— Microwave satellite program) until DOD submitted to the congressional defense committees a plan to meet weather monitoring data collection requirements. In addition, the explanatory statement that accompanied the Consolidated Appropriations Act, 2016, recommended that the Air Force focus on ensuring that the next generation of weather satellites meet the full spectrum of requirements and work with civil stakeholders to leverage appropriate civil or international weather assets. As called for in the law and the explanatory statement, DOD established plans to meet weather monitoring data collection needs, including by acquiring satellites as part of a family of systems to replace its aging legacy weather satellites. Additionally, DOD formally coordinated with NOAA on weather monitoring data collection efforts. In January 2017, the Air Force and NOAA signed a memorandum of agreement, and in November 2017, signed an annex to that agreement, to allow for the exchange of information and collaboration on a plan for collecting weather monitoring data. The Air Force and NOAA are now developing plans to relocate a residual NOAA satellite over the Indian Ocean, an area of concern for cloud characterization and area-specific weather imagery coverage. requirements. Under this program, the department may launch a demonstration satellite in 2021 and plans to launch an operational satellite in 2022. DOD also developed plans for providing its two highest-priority capabilities—cloud characterization and area-specific weather imagery data collection—that will not be covered by the Weather System Follow- on–Microwave satellite program. The department is planning a longer- term solution, called the Electro-Optical/Infrared Weather Systems program, to meet these needs, with a planned satellite launch in 2024. Meanwhile, DOD is in the process of acquiring a small prototype satellite, called the Operationally Responsive Space-8 satellite, to provide interim capabilities. DOD plans to launch Operationally Responsive Space-8 as early as 2022. Monitoring: met. DOD now meets the monitoring criterion as evidenced by its actions to initiate a major acquisition program, the Weather System Follow-on–Microwave, and award a contract for the first satellite. In addition, program officials stated that they plan to monitor the program’s progress toward addressing critical needs and assess its operations and sustainment costs. Demonstrated progress: met. DOD now meets the demonstrated progress criterion because it has developed plans and taken actions to address gaps in weather data through its plans to launch the Weather System Follow-on–Microwave satellite in 2022. The department also plans to launch the Electro-Optical/Infrared Weather Systems satellite in 2024 and provide interim capabilities beginning as early as 2022. By developing these plans, DOD has reduced the risk of a gap in weather satellite data and addressed the concerns about a lack of planning that we identified in our 2017 High-Risk Report. DOD’s effective implementation of its plans will be key to further reducing the risks of gaps in weather satellite data in the future. Moving forward, we will continue to monitor both NOAA and DOD efforts to develop and launch the next satellites in their respective weather satellite programs. NOAA plans to launch its next geostationary weather satellite in 2021 and to launch its next polar weather satellite in 2022. DOD plans satellite launches in 2021 (potentially), 2022, and 2024. In addition, we will continue to monitor DOD’s efforts to develop long-term plans to meet its weather satellite requirements. Weapon Systems Annual Assessment: Knowledge Gaps Pose Risks to Sustaining Recent Positive Trends. GAO-18-360SP. Washington, D.C.: Apr. 25, 2018. Satellite Acquisitions: Agencies May Recover a Limited Portion of Contract Value When Satellites Fail. GAO-17-490. Washington, D.C.: June 9, 2017. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-17-333SP. Washington, D.C.: Mar. 30, 2017. Defense Weather Satellites: DOD Faces Acquisition Challenges for Addressing Capability Needs. GAO-16-769T. Washington, D.C.: July 7, 2016. Polar Satellites: NOAA Faces Challenges and Uncertainties that Could Affect the Availability of Critical Weather Data. GAO-16-773T. Washington, D.C.: July 7, 2016. Polar Weather Satellites: NOAA Is Working to Ensure Continuity but Needs to Quickly Address Information Security Weaknesses and Future Program Uncertainties. GAO-16-359. Washington, D.C.: May 17, 2016. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The federal government is one of the world's largest and most complex entities; about $4.1 trillion in outlays in fiscal year 2018 funded a broad array of programs and operations. GAO's high-risk program identifies government operations with vulnerabilities to fraud, waste, abuse, and mismanagement, or in need of transformation to address economy, efficiency, or effectiveness challenges. This biennial update describes the status of high-risk areas, outlines actions that are still needed to assure further progress, and identifies two new high-risk areas needing attention by the executive branch and Congress. Solutions to high-risk problems save billions of dollars, improve service to the public, and would strengthen government performance and accountability. GAO uses five criteria to assess progress in addressing high-risk areas: (1) leadership commitment, (2) agency capacity, (3) an action plan, (4) monitoring efforts, and (5) demonstrated progress. The ratings for more than half of the 35 areas on the 2019 High-Risk List remain largely unchanged. Since GAO's last update in 2017, seven areas improved, three regressed, and two showed mixed progress by improving in some criteria but declining in others. Where there has been improvement in high-risk areas, congressional actions have been critical in spurring progress in addition to actions by executive agencies. GAO is removing two of the seven areas with improved ratings from the High-Risk List because they met all of GAO's five criteria for removal. The first area, Department of Defense (DOD) Supply Chain Management, made progress on seven actions and outcomes related to monitoring and demonstrated progress that GAO recommended for improving supply chain management. For example, DOD improved the visibility of physical inventories, receipt processing, cargo tracking, and unit moves. Improvements in asset visibility have saved millions of dollars and allow DOD to better meet mission needs by providing assets where and when needed. The second area, Mitigating Gaps in Weather Satellite Data, made significant progress in establishing and implementing plans to mitigate potential gaps. For example, the National Oceanic and Atmospheric Administration successfully launched a satellite, now called NOAA-20, in November 2017. NOAA-20 is operational and provides advanced weather data and forecasts. DOD developed plans and has taken actions to address gaps in weather data through its plans to launch the Weather System Follow-on–Microwave satellite in 2022. There are two new areas on the High-Risk List since 2017. Added in 2018 outside of GAO's biennial high-risk update cycle, the Government-Wide Personnel Security Clearance Process faces significant challenges related to processing clearances in a timely fashion, measuring investigation quality, and ensuring information technology security. The second area, added in 2019, is Department of Veterans Affairs (VA) Acquisition Management. VA has one of the most significant acquisition functions in the federal government, both in obligations and number of contract actions. GAO identified seven contracting challenges for VA, such as outdated acquisition regulations and policies, lack of an effective medical supplies procurement strategy, and inadequate acquisition training. Overall, 24 high-risk areas have either met or partially met all five criteria for removal from the list; 20 of these areas fully met at least one criterion. Ten high-risk areas have neither met nor partially met one or more criteria. While progress is needed across all high-risk areas, GAO has identified nine that need especially focused executive and congressional attention, including Ensuring the Cybersecurity of the Nation, Resolving the Federal Role in Housing Finance, addressing Pension Benefit Guaranty Corporation Insurance Programs, Managing Risks and Improving VA Health Care, and ensuring an effective 2020 Decennial Census. Beyond these specific areas, focused attention is needed to address mission-critical skills gaps in 16 high-risk areas, confront three high-risk areas concerning health care and tax law enforcement that include billions of dollars in improper payments each year, and focus on a yawning tax gap. This statement describes GAO's views on progress made and what remains to be done to bring about lasting solutions for each high-risk area. Substantial efforts are needed by the executive branch to achieve progress on high-risk areas. Addressing GAO's hundreds of open recommendations across the high-risk areas and continued congressional oversight and action are essential to achieving greater progress.", "document_type": "gao"}
{"report": "In 2012, the Secretary of Homeland Security issued a memorandum directing TSA to take a number of actions in response to allegations of profiling by behavior detection officers. These actions included, among others, working with the DHS Office of Civil Rights and Civil Liberties to (1) review, and revise as necessary, behavior detection officer training policies, training curriculum, and supervisory guidance to ensure they adequately address and train against profiling; (2) enhance data collection to facilitate appropriate supervision and monitoring of behavior detection activities; and (3) ensure passengers are aware of complaint mechanisms and ensure complaints are appropriately handled. TSA has taken some actions to address these directives. For example, TSA has revised its standard operating procedures and training materials to more clearly instruct personnel trained in behavior detection and other TSA personnel on how to avoid unlawful profiling; initiated a study to collect data on the race and national origin of passengers referred for behavior detection screening and examine whether disparities exist in the referral trends, and if so, whether these differences suggest discrimination or bias in the referral process; and issued a Management Directive establishing TSA policy and procedures for receiving, documenting, and referring passenger screening complaints resulting from the application of TSA security screening policies and procedures, including processes for all involved offices in headquarters and the field that handle passenger complaints. The Aviation and Transportation Security Act established TSA as the federal agency with primary responsibility for securing the nation’s civil aviation system, which includes the screening of all passengers and property transported by commercial passenger aircraft. At the approximately 440 TSA-regulated airports in the United States, all passengers, their accessible property, and their checked baggage are screened prior to boarding an aircraft or entering the sterile area of an airport pursuant to statutory and regulatory requirements and TSA-established standard operating procedures. TSA began using behavior detection in 2006 as an added layer of security to identify potentially high- risk passengers. Through the end of fiscal year 2016, TSA’s behavior detection screening process was a stand-alone program that used specially trained behavior detection officers to observe passengers at the screening checkpoint and engage them in brief verbal exchanges. During this period, behavior detection officers had brief interactions with passengers in the queue leading up to the screening checkpoint. If the behavior detection officers determined during this interaction that a passenger exhibited a certain number of behavioral indicators, the behavior detection officer was to refer the passenger for additional screening or, if circumstances warranted, contact a law enforcement officer. According to TSA procedures, if a passenger was referred for additional screening, one behavior detection officer conducted a pat-down of the passenger and search of his or her personal property while another checked documents and conversed with the passenger, attempting to understand why the behavioral indicators were being displayed and continuing to look for additional behavioral indicators. If a passenger did not exhibit a certain number of additional indicators, he or she was allowed to proceed to the boarding gate. If the passenger did exhibit a certain number of additional indicators, or other events occurred, such as the discovery of a fraudulent document, the behavior detection officer was to call a law enforcement officer. The law enforcement officer then would determine next steps, which could include questioning the passenger or conducting a criminal background check. The law enforcement officer then determined whether to release the passenger, refer the passenger to another law enforcement agency, or arrest him or her. In fiscal year 2017, consistent with the Aviation Security Act of 2016, TSA eliminated the stand-alone behavior detection officer position. TSA transferred the former behavior detection officers to serve as part of the screener workforce and began assigning them to the checkpoint to screen passengers. According to TSA officials, when screeners trained in behavior detection are assigned to a position, TSA policies and procedures permit them to use behavior detection when applicable. Furthermore, TSA’s checkpoint standard operating procedures do not currently include the use of behavior detection, as behavior detection’s use continues to be guided by its own policies established in 2016. However, some screeners trained in behavior detection continue to use behavior detection to support passenger screening canine teams as part of expedited screening. As part of this process, screeners trained in behavior detection work in conjunction with canine teams to observe passenger behavior and identify passenger behaviors that may indicate that a passenger poses a higher risk to the aviation system. The Training and Development Division (Training Division), within TSA headquarters, oversees the development, delivery, and evaluation of training programs for TSA employees. The National Training Plan, developed annually by the Training Division and Security Operations, contains the core curriculum for screeners to meet their yearly training requirements. In addition, Security Operations works with the Traveler Engagement Division to develop and deliver specific training on topics such as disability profiling, racial profiling, and screening transgender persons. In August 2017, TSA began training screeners on its new behavioral indicators. TSA revised the behavioral indicators by eliminating and combining some of the indicators used to observe passenger behavior, which TSA refers to as Optimized Behavior Detection. According to TSA officials, Optimized Behavior Detection includes 36 revised behavioral indicators—which TSA pared down from a list of 96 indicators. As of January 2019, TSA officials told us out of the approximately 43,000 screeners nationwide, a total of 2,541 screeners had been trained at 117 airports in Optimized Behavior Detection. Screeners must be trained in passenger and accessible property screening before they are eligible to attend Optimized Behavior Detection training. Upon successful completion of Optimized Behavior Detection training, screeners are permitted to utilize behavior detection in accordance with the standard operating procedures, such as when operating in conjunction with canine teams or screening airport and airline workers. In addition, screeners must complete all requirements in the National Training Plan which includes elements of training on TSA’s mechanisms for preventing unlawful profiling. TSA’s Security Operations is responsible for overseeing the use of behavior detection. TSA’s behavior detection policies and procedures prohibit screeners from selecting passengers for additional screening based on race, ethnicity, religion, and other factors, whether through behavior detection or other security measures. This responsibility includes overseeing officers trained in behavior detection to ensure they conduct behavior detection without regard to race/ethnicity, color, gender/sex, gender identity, religion, national origin, sexual orientation, or disability, in accordance with constitutional, statutory, regulatory, and other legal and DHS policy requirements to protect the civil rights and civil liberties of individuals. Although the stand-alone behavior detection officer position was eliminated and the program ended in 2017, the requirement to conduct oversight and verify compliance with TSA policies still applies when behavior detection is used, such as when behavior detection is used in conjunction with passenger screening canine teams. According to TSA’s policies and procedures, supervisors must conduct oversight observations of behavior detection activities a minimum of 8 hours every 14 days to verify and document compliance with behavior detection policies, standard operating procedures, the handbook, and training, among other things, and submit a compliance checklist documenting the review to TSA Security Operations. The TSA Contact Center (TCC) is the primary point of contact for collecting, documenting, and responding to public questions, concerns, or complaints regarding passengers’ screening experience; reports and claims of lost, stolen, or damaged items; and complaints submitted by TSA employees. The TCC may refer screening complaints for resolution to other TSA headquarters offices, depending on the specific allegation. For example, complete complaints alleging violations of civil rights and civil liberties, which include allegations implicating color, race, ethnicity, gender, genetic information, national origin, religion, sexual orientation, and parental status, must be referred to the Multicultural Branch. Figure 1 describes the TCC’s complaint review process. TSA’s Multicultural Branch is responsible for collecting, monitoring, and adjudicating passenger complaints alleging civil rights and civil liberties violations at the passenger screening checkpoint, including complaints alleging unlawful profiling and discrimination, among other things. The Multicultural Branch receives complaints alleging civil rights and civil liberties violations from several sources within TSA including the TCC. When TCC officials determine a complete complaint involves a potential civil rights or civil liberties violation, they are to forward the complaint to the Multicultural Branch where staff are to input the complaint into a database and track the resolution of each complaint they receive. The Multicultural Branch, in consultation with Security Operations, determines whether a screener followed standard operating procedures while screening the complainant by reviewing available video of an incident or interviewing witnesses who saw the incident. Depending on the nature and severity of the allegation, TSA airport staff may also elevate the complaint and evidence to the airport’s Assistant Federal Security Director (FSD) for Screening. If the investigation finds fault with the screener, the screener’s supervisor or manager is to determine the corrective action to be taken. Corrective actions specified in TSA’s guidelines for disciplinary actions to address misconduct range from mandating that the screener take additional training to correct the behavior to terminating the screener’s employment for multiple repeat offenses or a single egregious action. Following the outcome of the complaint review and any resulting corrective actions, the TSA headquarters unit or the TSA customer support manager at the airport is to communicate the status of the resolution, if any, to the complainant— such as by using a template letter that explains TSA’s policies and procedures or issuing an apology. According to Multicultural Branch protocols for reviewing passenger complaints, complaints may be resolved in three ways: Closed-Administratively: If the complainant does not respond within 10 days to the Multicultural Branch’s first contact for additional information, such as a request for additional information on the alleged civil rights and civil liberties violation, the complaint is to be closed. Closed-No Jurisdiction: Complaints that are not within the Multicultural Branch’s jurisdiction, such as complaints involving rude and unprofessional conduct that are not related to allegations of civil rights and civil liberties violations, are to be closed and referred to other TSA offices or the TSA designated point of contact at the airport for further handling. Closed-Resolved: Following the outcome of the investigation, the Multicultural Branch is to send a letter to the complainant summarizing the allegations reviewed, explaining whether TSA procedures were followed, and in some cases, issuing an apology or informing the complainant of the type of training offered to the screener(s). The Multicultural Branch may recommend training and provide refresher training materials for distribution at the airport to the screener(s) involved, if identified, or for all screeners at the airport’s checkpoint at which the complaint originated. According to TSA officials, the Multicultural Branch recommends training when standard operating procedures for screening were not followed or when it determines that the proactive measure of refresher training would be useful. According to TSA, the designated TSA point of contact at the airport is required to verify when the training is completed. Before screeners are eligible to conduct any behavior detection activities, they must first complete a 5-day Optimized Behavior Detection Basic Training course, and undergo on-the-job training at their local airport. This course includes an overview of DHS and TSA policies that prohibit unlawful profiling, and trains screeners to apply behavioral indicators to passengers without regard to race/ethnicity, color, gender/sex, gender identity, religion, national origin, sexual orientation, or disability. Participants must complete the Optimized Behavior Detection Basic Training course and pass a 40 question job knowledge test at the end of the class, in addition to completing 32 hours of on-the-job training under the supervision of an officer already trained in behavior detection. If a participant fails the job knowledge test, he or she is to receive 1 hour of remedial training before retaking the test. Screeners must pass the test in two attempts to be eligible to conduct behavior detection activities. In the four Optimized Behavior Detection Basic Training courses we attended, the training instructors covered TSA’s policies on prohibiting unlawful profiling on day one of the course, and explained that profiling passengers based on discernible traits was not only illegal, but that such practices are ineffective at identifying potentially high-risk passengers. In addition, the course manual included a copy of DHS’s 2013 memorandum defining racial profiling, which all participants were required to review. To test their understanding of TSA policy and the Optimized Behavior Detection Standard Operating Procedures, the instructors presented various scenarios to engage participants in practicing how they would apply behavior detection at the checkpoint. The 2018 National Training Plan required behavior detection–trained screeners to complete four recurrent technical training courses related to behavior detection, including two that contain material reinforcing DHS’s and TSA’s policies prohibiting unlawful profiling. Screeners participate in each of the four interactive training courses using a computer and the courses contain knowledge checks that the participant must answer correctly before completing the training. Table 1 describes the training courses screeners trained in behavior detection are required to complete and appendix I includes a list of additional training related to unlawful profiling. TSA determines the effectiveness of particular training programs using the Kirkpatrick Evaluation Model, a commonly accepted training evaluation model endorsed by the Office of Personnel Management and used throughout the federal government. In May 2018, TSA updated its training standards based on the ADDIE model, a methodology comprising five phases: Analysis, Design, Development, Implementation, and Evaluation (ADDIE). TSA uses the Kirkpatrick model as part of the evaluation stage of ADDIE. The Kirkpatrick model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things. TSA conducts Levels 1 and 2 evaluations on selected training courses. Table 2 provides an overview of the Kirkpatrick model and the evaluation levels for courses related to behavior detection and unlawful profiling. TSA officials told us they will continue to evaluate the Optimized Behavior Detection Basic Training course and Level 3 evaluations are under development, as they roll out their training evaluation process. According to TSA’s Training Standards, a review team determines the frequency of curriculum review, which should occur at least once every 5 years. As part of this review, TSA plans to leverage data reported in evaluations at Kirkpatrick Levels 1 through 3. TSA’s 2016 Optimized Behavior Detection Program Handbook and Operational Oversight Compliance Guidance require supervisors to conduct routine checks of behavior detection operations to monitor compliance with standard operating procedures. TSA’s behavior detection Operational Oversight Compliance Guidance outlines seven specific assessments of behavior detection operations and includes a checklist for each assessment for managers to document completion of these routine oversight tasks. According to TSA officials, these assessments should occur when screeners use behavior detection in conjunction with canine operations and while screening airline and airport workers, among other activities. When conducting these assessments, supervisors are to conduct 1-hour observations and use detailed checklists to document how screeners trained in behavior detection perform the behavior detection in practice. For example, one checklist requires supervisors to observe how screeners trained in behavior detection monitor passenger flow and communicate with passengers while observing for behavioral indicators, such as ensuring screeners using behavior detection do not ask passengers intrusive or offensive questions, among other activities related to the use of behavior detection. However, our review of the oversight checklists found that they do not specifically instruct supervisors to monitor for compliance with procedures intended to prohibit unlawful profiling. According to TSA officials, TSA’s guidance and checklists do not include this type of monitoring for unlawful profiling because officials believe that the training screeners receive, adherence to the standard operating procedures, and the general supervisory oversight in place are sufficient to prevent unlawful profiling and could alert supervisors to situations where unlawful profiling happens. However, the 2013 DHS memorandum on DHS’s policy on unlawful profiling states that each component, including TSA, should both implement specific policy and procedures on racial profiling, and ensure all personnel are trained and held accountable for meeting the standards set forth in DHS policy. In addition, Standards for Internal Control in the Federal Government states that management should establish and implement activities to monitor the internal control system and evaluate the results, as well as remediate identified internal control deficiencies. Such a mechanism could be an item added to a checklist for supervisors to document, based on their observations, whether screeners selected individuals for additional scrutiny in a manner consistent with policies and procedures. Another oversight mechanism, as noted in DOJ’s guidance on the use of race and other factors, could be studying the implementation of policies and procedures that prohibit unlawful profiling through targeted, data- driven research projects. As previously discussed, in 2013, TSA initiated a study and collected data through October 2017 on passengers referred for secondary screening to monitor compliance with policies that prohibit unlawful profiling. TSA discontinued the study and did not analyze the data collected because the stand-alone behavior detection program ended in November 2017. As a result of not conducting the analysis, TSA does not know what the data would have shown regarding compliance with policies that prohibit unlawful profiling. TSA officials said they plan to update the behavior detection and checkpoint screening policies, procedures, and guidance during fiscal year 2019. As a part of this update, TSA officials told us they plan to include language in the standard operating procedures reinforcing the use of behavior detection simultaneously with other checkpoint duties, such as the document checker position. However, TSA officials told us they are not planning to add an oversight mechanism specific to profiling as part of the updates because, as previously noted, they believe screener training, adherence to the standard operating procedures, and general supervisory oversight are sufficient. Developing a specific oversight mechanism, such as a checklist or a data-driven study, to monitor screeners’ compliance with policies that prohibit unlawful profiling would provide TSA with greater assurance that its personnel are adhering to these policies when using behavior detection, and better position TSA to identify potential incidents of unlawful profiling. The TCC received 3,663 complaints related to passenger screening alleging violations of civil rights and civil liberties from October 2015 through February 2018. These complaints are not specific to behavior detection activities and generally reflect alleged conduct occurring at the screening checkpoint through the application of screening measures. We analyzed the 3,663 complaints and found that the majority (2,251 of 3,663) of the complaints alleged discrimination or profiling based on personal attributes and characteristics. For example, the TCC received complaints alleging discrimination that involved assertions by passengers that they had been selected for pat-downs based on race and ethnicity, among other reasons, when the passengers believed they did not trigger an alarm prompting the pat-downs. The TCC also received complaints related to passengers’ transgender identity alleging selection for additional screening because of their transgender status. Additionally, the TCC received passenger complaints alleging that screening procedures were aggressive or inappropriate for senior citizens. Table 3 provides a list of complaint types based on our analysis. In addition, appendix II provides additional detail about our content analysis of complaints alleging civil rights and civil liberties violations, and appendix III provides a list of 10 airports most often identified in the complaints. As TSA’s primary point of contact for passenger complaints, the TCC is responsible for the initial review and referral of all complaints that involve allegations of civil rights and civil liberties violations to the Multicultural Branch. According to the TCC standard operating procedures, TCC analysts review the complaints to ensure that they contain the necessary information to be considered complete, including the airport, passenger’s name, date of the incident, and description of the alleged civil rights and civil liberties violation. In addition, complaints reported over the phone or made on behalf of another person without the person’s consent are initially considered incomplete. For complaints that are not complete, the TCC sends the passenger a document request for information when the passenger has provided correct contact information. According to TCC officials, passengers often do not provide the correct contact information or do not respond with the necessary information to complete the complaint. TCC officials said that incomplete complaints are typically sent to the Multicultural Branch for informational purposes. Multicultural Branch officials told us that they consider information from incomplete complaints to inform its policy and training initiatives, and to improve how TSA engages with the public. From October 2015 through February 2018, the TCC referred 51 percent (1,865) of the 3,663 complaints it received to the Multicultural Branch for review. The TCC reported that 48 percent (1,764) of the 3,663 complaints did not have complete information necessary for further review, such as the airport and date of the incident. According to TSA officials, these complaints were sent to the Multicultural Branch for informational purposes. TCC’s passenger complaint data show that the remaining 1 percent (34) of the complaints were from TSA employees and were referred to other TSA offices for review. TSA’s Multicultural Branch receives and reviews complete complaints related to allegations of violations of civil rights and civil liberties that are referred to it from the TCC, DHS’s Office of Civil Rights and Civil Liberties, TSA’s Disability Branch, and TSA personnel at airports. From October 2015 through February 2018, the Multicultural Branch received 2,059 complaints alleging violations of civil rights and civil liberties, as shown in figure 2. Multicultural Branch officials stated that the majority of these complaints were referred from the TCC. As shown in figure 2, for 1,066 (52 percent) of the complaints, Multicultural Branch staff found indications of potential discrimination, such as instances of rude or unprofessional conduct that included the use of race or other protected characteristics. According to Multicultural Branch staff, to resolve the 1,066 complaints, they recommended a range of refresher training. Multicultural Branch staff explained that when issues are identified, their policy is to address the issues through screener training. Multicultural Branch officials reported that these trainings were provided through National Shift Briefings, which were circulated across TSA, or through training provided at a particular airport. For example: In one of the complaint cases we reviewed, a passenger alleged profiling based on headwear. Multicultural Branch officials used camera recordings and statements from officers involved in the encounter to substantiate that screening procedure violations had occurred. As a result, Multicultural Branch officials recommended refresher training to the airport on headwear screening protocols for all screeners at the airport to review. In another complaint case we reviewed, a passenger alleged profiling based on the use of a tribal-issued photo identification card. In response, Multicultural Branch officials sent refresher training on verifying tribal identification and the screening of Native American passengers to the TSA designated point of contact at the airport involved for distribution to TSA personnel identified in the complaint. In a third complaint reviewed, a passenger alleged being profiled at the screening checkpoint, without including any additional details. According to TSA officials, based on the particular allegations of the complaint and the lack of details, TSA was unable to substantiate the allegations made in the complaint. As a result, Multicultural Branch sent National Shift Briefings on TSA’s policies and procedures that prohibit unlawful profiling and inappropriate comments to the TSA designated point of contact at the airport involved for distribution to TSA personnel identified in the complaint. As shown in figure 2, there were 993 complaints that the Multicultural Branch reviewed but did not address through training. The Multicultural Branch closed 121 of these complaints because it determined that the complainant did not provide sufficient information about the alleged civil rights and civil liberties violation for Multicultural Branch review and the complainant did not respond with additional information requested by the Multicultural Branch within 10 days. The Multicultural Branch determined that the remaining 872 complaints were not substantiated based on its review of the camera recording of the alleged incident, or were not within its jurisdiction. For the complaints not within its jurisdiction, the Multicultural Branch referred them to other TSA offices, to TSA officials at the airport or airports identified in the complaints for review, or to other federal agencies (e.g., U.S. Customs and Border Protection, Department of Transportation, or the Federal Aviation Administration) as appropriate. These complaints involved allegations of unprofessional conduct and other issues that did not involve allegations of civil rights and civil liberties violations. According to Multicultural Branch guidance, the designated TSA point of contact at the airport along with the Multicultural Branch analyst are to determine appropriate next steps for resolving complaints, such as preparing a briefing for screeners that is tailored to address the concerns raised by the complainant. TSA officials stated that resolutions to the complainant are tailored to reflect the allegation, type of inquiry conducted, and investigation of the facts and evidence underlying the complaint. TSA’s responses to the complainant include, but are not limited to, apologizing for the screening experience or informing the complainant about the next steps such as the agency’s plans to address the complaint or underlying conduct that gave rise to the complaint. For example, in a letter we reviewed, TSA apologized for the “unprofessional and inappropriate personal questions” the passenger experienced during screening, and stated that refresher training would be distributed to screeners at the airport involved. According to documentation we reviewed related to this complaint, the Multicultural Branch sent refresher training materials on avoiding inappropriate comments to the designated TSA point of contact at the airport involved. In addition, TSA’s office of Human Capital Employee Relations reported that it took a range of disciplinary actions—from letters of reprimand to termination—for 100 screeners from October 2015 through February 2018, in part in response to passenger complaints alleging civil rights and civil liberties violations. TSA’s Multicultural Branch regularly collects and analyzes data on passenger civil rights and civil liberties and discrimination complaints and their resolution status, and shares this information with TSA executive leadership, TSA airport customer service managers, and screeners in the field, among others. Multicultural Branch officials told us their staff are assigned to specific airports based on geographic region, and they continually analyze passenger complaints referred to their office from the TCC to identify trends. Staff members meet weekly to discuss trends in complaints for their geographic regions, and they review weekly, quarterly, and annual reports on the number and category of complaints referred to their office by the TCC. In addition, Multicultural Branch officials track the resolution of the cases for which they have jurisdiction and submit this information to their senior leadership each week. Specifically, the Multicultural Branch uses a database to track complaints by type, airport, submission date, and resolution status, such as how many cases are open, closed, or whether they have been resolved. Multicultural Branch officials share trends in complaints throughout TSA in several ways, including conference calls, monthly briefings, reporting metrics to TSA executive leadership, and on-site training events at airports each year. For example, Multicultural Branch officials hold monthly conference calls with customer service managers at airports to review complaint trends, upcoming on-site airport trainings, and job aids they have developed to help screeners understand issues, such as screening passengers wearing religious headwear. Multicultural Branch officials stated they also share information with screeners and supervisors through National Shift Briefings that are distributed at all airports, and focus on bringing awareness to screeners on events they need to be aware of when screening passengers, such as religious observances occurring that month. According to TSA officials, the Multicultural Branch uses its analysis of passenger complaints and the results of complaint investigations to develop training aids and materials on areas where they determine screeners need more training, such as multicultural awareness or screening of transgender passengers. For example, the Multicultural Branch has developed briefings focusing on unlawful profiling and unconscious bias which reiterated that unlawful profiling is against TSA policy, defined unconscious bias, and provided scenario-based examples. Additionally, members from the Multicultural Branch hold on-site training for screeners at selected airports each year based on complaint data analysis and other factors. These training sessions last three days, include topics stemming from complaint data TSA has analyzed, and can include webinars, role-playing, and other forms of instruction. DHS and TSA have policies prohibiting unlawful profiling—using race, ethnicity, gender, or other protected characteristics to identify passengers for additional screening—when using behavior detection, as well as other screening measures. While TSA has oversight guidance and checklists to monitor screeners’ use of behavior detection, these policies and procedures do not include a specific mechanism to monitor whether screeners may be using behavior detection to unlawfully profile passengers. Although TSA officials report that they are working to update the standard operating procedures in 2019, they currently have no plan to add a specific mechanism to monitor compliance with policies that prohibit unlawful profiling. Developing a specific oversight mechanism would provide TSA with greater assurance that screeners are adhering to such policies and help TSA identify any potential incidents of unlawful profiling. We are making the following recommendation to TSA. The TSA Administrator should direct Security Operations to develop a specific oversight mechanism to monitor the use of behavior detection activities for compliance with DHS and TSA policies that prohibit unlawful profiling. (Recommendation 1) We provided a draft of this report to DHS for review and comment. DHS provided written comments which are reproduced in appendix IV. In its comments, DHS concurred with our recommendation and described actions planned to address it. Security Operations, TCC, and the Multicultural Branch also provided technical comments, which we incorporated as appropriate. DHS correctly noted in its letter that GAO’s analysis of civil rights and civil liberties complaints related to every aspect of TSA’s passenger and baggage screening and is not specific to behavior detection. We agree with DHS’s observation, as this analysis provides information on what passengers alleged in their complaints and how TSA addressed them. It is important to note that the complaint data provided by TSA did not preclude behavior detection activities as a potential contributing factor to any number of the complaints submitted. With regard to our recommendation, that the TSA Administrator should direct Security Operations to develop a specific oversight mechanism to monitor the use of behavior detection activities for compliance with DHS and TSA policies that prohibit unlawful profiling, DHS stated that TSA plans to take additional steps to continue to ensure behavior detection activities adhere to polices that prohibit unlawful profiling. In fiscal year 2019, TSA plans to modify existing oversight checklists used by managers and supervisors to include specific terminology for monitoring unlawful profiling. DHS estimated that this effort would be completed by September 30, 2019. This action, if fully implemented, should address the intent of the recommendation. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees and the Secretary of Homeland Security. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact William Russell at (202) 512-8777 or RussellW@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. The Transportation Security Administration (TSA) provided examples of refresher training materials that are provided to screeners on TSA’s prohibition on the use of unlawful profiling at the passenger screening checkpoint. Table 4 provides information on these materials, including the methods used to distribute the materials to screeners. From October 2015 through February 2018, the Transportation Security Administration (TSA) Contact Center (TCC) received 3,663 complaints that it classified as alleging violations of civil rights and civil liberties. Of the 3,663 complaints, the TCC received 707 complaints, or about 19 percent, by phone. Table 5 summarizes our analysis of the complaints the TCC received. From October 2015 through February 2018, the Transportation Security Administration (TSA) Contact Center (TCC) received 3,663 complaints that it classified as alleging violations of civil rights and civil liberties. The TCC received 707 of these complaints, or about 19 percent, by phone. Of the 3,663 complaints, Los Angeles International Airport was identified most often in the complaint data. Table 6 lists the 10 airports most often identified in these complaints. William Russell, (202) 512-8777 or RussellW@gao.gov. In addition to the contact named above, Ellen Wolfe (Assistant Director), Natalie Maddox (Analyst in Charge), Saida Hussain, and Brendan Kretzschmar made key contributions to this report. Also contributing to the report were Alyssa Bertoni, David Dornisch, Ben Emmel, Eric Hauswirth, Susan Hsu, Tom Lombardi, Amanda Miller, Sam Portnow, Rachel Stoiko, and Adam Vogt.", "summary": "In 2016, TSA began using behavior detection in a more limited way to identify potentially high-risk passengers who exhibit certain behaviors it asserts are indicative of stress, fear, or deception, and refer them for additional screening or, when warranted, to law enforcement. TSA's policies and procedures prohibit unlawful profiling, i.e., screeners are prohibited from selecting passengers for additional screening based on race, ethnicity, or other factors. Allegations of racial profiling have raised questions about TSA's use of behavior detection. GAO was asked to review TSA's measures to prevent behavior detection activities from resulting in unlawful profiling. This report examines, among other things, (1) TSA's oversight of behavior detection activities and (2) the number of complaints alleging violations of civil rights and civil liberties related to passenger screening and actions taken by TSA to address them. GAO reviewed TSA policies and procedures; analyzed passenger complaint data received by TSA from October 2015 through February 2018 and actions taken to address them; and interviewed TSA officials. Complaint data we analyzed alleged conduct that occurred at the screening checkpoint and was not specific to behavior detection activities. Transportation Security Administration (TSA) policy requires managers to ensure behavior detection is conducted without regard to race or ethnicity, among other factors. TSA uses seven oversight checklists to assess whether behavior detection activities are conducted in accordance with TSA policy, such as monitoring whether screeners trained in behavior detection observe and engage passengers correctly. However, these checklists do not instruct supervisors to monitor for indications of profiling. TSA officials stated that the training screeners receive, adherence to operating procedures, and general supervisory oversight are sufficient to alert supervisors to situations when unlawful profiling may occur. However, developing a specific mechanism to monitor behavior detection activities for compliance with policies prohibiting unlawful profiling would provide TSA with greater assurance that screeners are adhering to such policies. From October 2015 through February 2018, TSA received about 3,700 complaints alleging civil rights and civil liberties violations related to passenger screening. These complaints are not specific to behavior detection activities. The TSA Contact Center (TCC), the office that initially receives these complaints, reported that about half of the complaints did not have complete information from passengers necessary for further review, such as the airport and date of the incident. According to TCC officials, they attempt to obtain the additional information from passengers, but often the complaint does not include the correct contact information or the passenger does not respond to the TCC's request for additional information. The TCC complaint data show that the remaining 51 percent (about 1,900) of complaints were referred to the TSA Multicultural Branch, the office responsible for reviewing complaints alleging civil rights and civil liberties violations. The Multicultural Branch reported reviewing 2,059 complaints, including approximately 1,900 complaints from TCC, as well as complaints referred from other TSA offices. For about half of the complaints (1,066) the Multicultural Branch reviewed, it found indications of potential discrimination and unprofessional conduct that involved race or other factors and recommended a range of refresher training across airports or for screeners at individual airports identified in the complaints. TSA's Multicultural Branch Reviewed 2,059 Complaints Alleging Violations of Civil Rights and Civil Liberties from October 2015 through February 2018 TSA should develop a specific oversight mechanism to monitor behavior detection activities for compliance with policies that prohibit unlawful profiling. DHS concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "In its fiscal year 2018 financial statement, DOE reported an estimated environmental liability of $494 billion. The majority of this liability was for cleanup work overseen by EM. We reported in January 2019 that in recent years, EM’s environmental liability has grown annually at a level that has outpaced the department’s annual spending on cleanup activities, and its liability may continue to grow. In its fiscal year 2018 financial statement, DOE reported its estimated environmental liability at $494 billion. In the financial statement, EM accounted for $377 billion (over 75 percent) of DOE’s total liability. In developing its environmental liability estimate, EM estimates the costs of storing, treating, or disposing of a variety of waste types. Storing and treating radioactive tank waste account for the largest portion of EM’s costs. For example, in January 2019 we reported that, in fiscal year 2017 (the most recent year for which these data were available at the time of our review), EM’s responsibilities to store and treat radioactive waste stored in underground tanks accounted for nearly half of EM’s total environmental liability, and its responsibilities for addressing contaminated facilities and remediating soil and groundwater contamination accounted for about one-quarter. Figure 2 shows the percentage and dollar amount of EM’s environmental liability by cleanup activity for fiscal year 2017. In January 2019, we also found that, of the 16 sites across the United States at which EM has cleanup responsibilities, two sites accounted for more than 70 percent of EM’s environmental liability in fiscal year 2017: the Hanford site and the Savannah River site (see fig. 3). These sites also include the majority of EM’s radioactive tank waste and the majority of radioactive contamination, which is particularly costly and complicated to treat. The Hanford site has 177 tanks containing 55 million gallons of waste, and the Savannah River site has 43 tanks containing 36 million gallons of waste. We reported in January 2019 that in recent years, EM’s environmental liability has grown annually at a level that has outpaced the department’s annual spending on cleanup activities. This growth has occurred at the same time as the number of contaminated sites has decreased. In fiscal years 2011 through 2018, EM spent over $48 billion, primarily to address radioactive tank waste as well as treat and dispose of other nuclear and hazardous materials. Nonetheless, since 2011, EM’s environmental liability grew by $214 billion, from $163 billion to $377 billion, according to our analysis of DOE financial data and documents (see fig. 4). EM’s environmental liability may continue to grow because its currently estimated environmental liability does not include the costs of all cleanup activities for which the agency will likely be responsible in the future and because the cost of addressing some of EM’s largest projects is still underestimated. First, not all of the cleanup activities EM must undertake are captured in the current liability because, according to federal accounting standards, only work that is probable and reasonably estimable is required to be reported in an agency’s liability. For example, EM has not yet developed a cleanup plan or cost estimate for the Nevada National Security site and, as a result, the cost of future cleanup of this site was not included in EM’s reported environmental liability. The nearly 1,400-square-mile site has been used for hundreds of nuclear weapons tests since 1951. These activities have resulted in more than 45 million cubic feet of radioactive waste at the site, but the costs for the cleanup of this waste are excluded from EM’s annually reported environmental liability. Second, the current cost associated with some of EM’s cleanup efforts may be underestimated. For example, as of April 2018, EM and its contractor had still not negotiated a cost for completing the WTP—DOE’s largest and most complex construction project. Further, although EM typically spends about $6 billion per year on cleanup activities, a large amount of its cleanup budget does not support actual cleanup activities. Instead, this funding goes toward recurring activities necessary to maintain the sites rather than toward reducing the environmental liability. EM refers to these activities as “minimum safety” work. According to EM officials, examples of such work include physical security, health and radiation protection, or critical facility and infrastructure maintenance for safe conditions. These officials said that minimum safety work constitutes 30 to 60 percent of individual sites’ budgets, for a total of at least $2.7 billion of EM’s fiscal year 2018 budget, as we reported in February 2019. The Assistant Secretary for EM noted in September 2018 that much of DOE’s environmental liability is associated with managing minimum safety work and that significant potential cost savings could result from reducing minimum safety work. Accordingly, she stated that EM planned an initiative in fiscal year 2019 to examine how EM can reduce this work. EM has undertaken several ad hoc studies and initiatives to address the growing costs in its cleanup program. However, EM has not conducted a formal root cause analysis to identify the causes for the growth in its environmental liabilities. Specifically, EM headquarters officials we interviewed said they were aware of the increases to the environmental liability from year to year, as well as the areas in which the liability changed; however, they said they had not done a detailed analysis of the root causes of the growth. A leading practice for program management is monitoring and controlling the program, which includes conducting root cause analyses and developing corrective action plans. However, in February 2019, we found that EM’s cleanup policy does not follow this leading practice because it does not include any such requirements. We recommended that EM review and revise its policy to include program management leading practices in its requirements, including for monitoring and controlling the program. DOE agreed with our recommendation and stated that it plans to revise its policy. EM has not resolved long standing management challenges that affect its cleanup program and contracts. In March 2019, we issued our 2019 High- Risk Series, which included updates related to DOE’s environmental liability and its contract management. While officials at EM have taken some steps toward management improvements aimed at reducing its environmental liabilities, we found that EM has not demonstrated progress toward resolving these challenges. We have identified several unresolved issues including the following: EM does not have a program-wide cleanup strategy. We reported in January 2019 that EM relies primarily on individual sites to locally negotiate cleanup activities and establish priorities. Our analysis of DOE documents identified instances of decisions involving billions of dollars where such an approach did not always balance overall risks and costs. For example, we reiterated what we found in May 2017 that two EM sites had plans to treat similar radioactive tank waste differently, and the estimated costs for treating the waste at one site— Hanford—may be tens of billions more than those at the other site— Savannah River. In addition, EM sites generally do not consider other sites’ risks and priorities when making cleanup decisions. We reported in January 2019 that this approach is not consistent with recommendations we and others have made over the last 2 decades that EM develop national priorities to balance risks and costs across and within its sites. Moreover, EM has not developed a program- wide strategy that sets such priorities and describes how EM will address its greatest risks. Instead, according to agency officials, it continues to prioritize and fund cleanup activities by individual site. We recommended in January 2019 that EM develop a program-wide strategy that outlines how EM will direct available resources to address human health and environmental risks across and within sites. DOE agreed with our recommendation and has since said it is working toward this goal. EM manages most of its cleanup work as operations activities, under less stringent requirements than capital asset projects. In February 2019, we reported that EM manages its cleanup work under different requirements, depending on whether it classifies the work as a capital asset project or an operations activity. EM currently manages most of its work as operations activities. In its fiscal year 2019 budget, operations activities accounted for 77 percent of EM’s budget (about $5.5 billion), and capital asset projects accounted for 18 percent (about $1.3 billion). Operations activities have less stringent requirements. For example, unlike capital asset projects, operations activities are not required to go through a thorough upfront planning process to determine the scope of work to be completed. In addition, under EM cleanup requirements, operations activities are not subject to independent oversight by entities outside EM. As a result, DOE management does not have information on how EM manages operations activities and cannot hold EM accountable for cost- effective and timely completion of this cleanup work. Since 2015, experts in DOE’s Office of Project Management have raised concerns that some operations activities, such as cleanup of radioactive tank waste, should be classified as capital asset projects. In February 2019, we recommended that EM work with DOE’s Office of Project Management—which is responsible for providing DOE-wide leadership and assistance pertaining to project management—to establish requirements for classifying cleanup work as capital asset projects or operations activities and then work together to asses EM’s ongoing operations activities to determine if they should be reclassified as capital asset projects based on the newly established requirements. DOE generally agreed with our recommendations and committed to review and revise its requirements as appropriate. EM’s cleanup policy does not follow program and project management leading practices. In February 2019, we also found that EM’s 2017 cleanup policy, which outlines procedures that govern the EM program and its operations activities, does not follow most selected leading practices for program and project management. Specifically, we found that EM’s 2017 cleanup policy does not follow any of 9 selected program management leading practices related to scope, cost, schedule performance, and independent reviews. For example, the policy does not require the program management leading practice of monitoring and controlling the program, including conducting root cause analyses and developing corrective action plans. Further, EM’s 2017 cleanup policy follows only 3 of 12 selected project management leading practices related to these areas. For example, EM’s 2017 cleanup policy does not require any independent reviews of its operations activities by anybody outside of EM. We recommended that DOE review and revise EM’s cleanup policy to include program and project management leading practices related to scope, cost, schedule performance, and independent reviews. DOE agreed with our recommendations. In addition, broader DOE management challenges affect EM and have implications for EM’s ability to effectively manage its cleanup work and begin reducing its environmental liability. EM, like DOE, executes its program activities primarily through the use of contracts. We have reported that about 90 percent of DOE’s budget is spent on contractors that manage the laboratories and carry out DOE’s programs. DOE’s contract management, however, is one of the areas we have identified as posing a high risk of fraud, waste, abuse, and mismanagement because of DOE’s record of inadequate management and oversight of contractors. As a result, DOE’s contract and project management has been on our High Risk List since 1990. Most recently, we found in March 2019 that DOE did not always ensure that contractors audited subcontractors’ incurred costs as required in their contracts. We identified more than $3.4 billion in subcontract costs incurred over a 10-year period that had not been audited as required, and some subcontracts remained unaudited or unassessed for more than 6 years. Completing audits in a timely manner is important because of a 6-year statute of limitations to recover unallowable costs that could be identified through such audits. We recommended that DOE develop procedures that require local offices to monitor contractors to ensure timely completion of required subcontract audits. DOE partially concurred with this recommendation and stated that it plans to review existing requirements and guidance and to consider the extent to which it requires monitoring of contractors’ progress in completing required subcontract audits. As we noted in the March 2019 report, we believe that DOE’s plans to further examine the issues raised in our report is a positive step toward resolving the issues; however, we continue to believe that the actions called for in our recommendations remain valid and that DOE could more efficiently resolve the issues by proceeding to implement those actions. Accurate and reliable information on the status and progress in a program is essential for effective management and to ensure key stakeholders are provided the information they need to fulfill their oversight, advisory, and other essential roles. However, EM’s performance measures for operations activities do not provide a clear picture of overall performance, and EM has not followed best practices in implementing its performance reporting systems. In addition, EM has historically not provided all of the statutorily required information about the status of its cleanup effort, and the information EM has reported has been incomplete or inaccurate. Finally, in its recent budget materials, EM did not include the funding it says it needs to meet its schedule cleanup milestones. In February 2019, we found that EM’s performance measures for operations activities—which constitute most of its cleanup activities—do not provide a clear picture of overall performance. According to EM documentation and officials, EM uses three tools to measure the overall performance of operations activities: earned value management (EVM), performance metrics, and cleanup milestones. We found problems with EM’s use of each of these tools. Figure 5 summarizes our findings on these three performance measures and how they affect EM’s ability to effectively manage the cleanup effort. First, we found in February 2019 that EM does not always ensure that its EVM data are comprehensive or reliable. EVM measures the value of work accomplished in a given period and compares it with the planned value of work scheduled for the period and with the actual cost of the work accomplished. EM relies primarily on EVM data to measure the overall performance of its operations activities. EM relies on contractors’ EVM systems to measure the performance of its contractors’ operations activities. We reviewed all 20 EM contracts covering operations activities and found that EM requires its contractors to maintain EVM systems for 17 of all 20 contracts. We also found that EM paid its contractors to maintain these systems and provide EVM reports to EM. However, we found that EM has not followed best practices to ensure that these systems are (1) comprehensive, (2) provide reliable data, and (3) are used by EM leadership for decision-making—which are the three characteristics of a reliable EVM system. For example, only about half of the EVM systems met the best practices for conducting integrated baseline reviews and performing ongoing surveillance. Among those, many of the reviews were not rigorous enough to ensure that the performance measurement baseline captured all of the work. We found that EM officials were not performing thorough surveillance reviews to ensure that EVM systems were in alignment with EVM guidelines and that the data being reported by the EVM systems were reliable. In addition, the EVM data for contracts covering operations activities contained numerous, unexplained anomalies in all the months we reviewed, including missing or negative values for some of the completed work to date. Even though EM requires most of its contractors for operations activities to maintain EVM systems and pays them for doing so, EM’s 2017 cleanup policy generally does not require that EVM systems be maintained and used in a way that follow EVM best practices. The use of EVM as a management tool is considered an industry standard and a best practice for conducting cost and schedule performance analysis for projects. EVM data can alert project managers to potential problems sooner than expenditures alone can. Because EM does not follow best practices in administering its EVM systems, EM leadership may not have access to reliable performance data to make informed decisions in managing billions of dollars’ worth of cleanup work every year and to provide to Congress and other stakeholders. We recommended that EM update its cleanup policy to require that EVM systems be maintained and used in a way that follows EVM best practices. DOE agreed with this recommendation, and said it would implement it. Second, we found that EM’s performance metrics do not link performance to cost. EM collects performance metrics from the sites monthly to measure progress toward completing the scope of work for the contract and against a goal set at the beginning of each year. We found in February 2019 that EM’s performance metrics do not link that work to the cost of completing it. For example, EM reported that it eliminated 72,000 gallons of radioactive liquid waste out of a target of 342,000 gallons for fiscal year 2017 at the Savannah River site and disposed of 1,734 cubic meters of low-level waste out of a target of 360 cubic meters at the Idaho site. However, in neither case did EM indicate how much that work cost to accomplish, such as whether those costs were above or below what had been planned. Because EM’s metrics do not link performance to cost, the performance information EM has provided to Congress does not indicate whether EM received good value from the contractor since it does not show how much that work cost to accomplish. We recommended that EM integrate EVM data into EM’s performance metrics for operations activities. DOE agreed with this recommendation and said it would implement it. Finally, we found in February 2019 that sites regularly renegotiate cleanup milestones they are at risk of missing, and EM does not track data on the history of postponed milestones or the reasons why milestones were postponed. As a result, milestones have limited value as a means of tracking cleanup progress since EM does not track the original (or any previously revised) milestone dates, which could provide some data to measure the progress of cleanup activities. We recommended that EM track and report original milestones dates as well as changes to its cleanup milestones. DOE agreed with our recommendation and said it is already tracking this information at the site level. In response, we reiterated the importance of tracking these changes and reporting that information at the headquarters level to help inform Congress. We reported in January 2019 that EM has not submitted congressionally mandated reports on its cleanup program and the information EM has reported has been incomplete or inaccurate. These reports are intended to provide Congress with information on the progress, challenges, and expected future costs of the EM cleanup program. Under the fiscal year 2011 National Defense Authorization Act, EM must annually develop and report to Congress a Future-Years Defense Environmental Management Plan that reflects estimated expenditures and proposed appropriations included in the DOE budget for defense environmental cleanup activities. It must do so at or about the same time that it submits its budget request. The plan is to cover the fiscal year for which the budget is submitted and at least the 4 succeeding fiscal years. The plan is required to describe the cleanup activities to be carried out during the period specified by the plan, estimated expenditures and proposed appropriations necessary to support them, and each milestone in an enforceable agreement governing the cleanup activity. For each milestone, EM is to identify whether the milestone will be met and, if not, explain why not and provide the date by which EM expects to meet it. EM submitted the required plan in fiscal year 2012 but did not submit plans from fiscal year 2013 through fiscal year 2016, as we found in January 2019, or in fiscal year 2018. EM’s most recent Future-Years Defense Environmental Management Plan, which DOE submitted to Congress in August 2017, included little of the information required by the fiscal year 2011 National Defense Authorization Act . Table 1 shows our assessment of the information EM provided in its 2017 Future-Years Defense Environmental Management Plan against the reporting requirements. We also found in February 2019 that the forecast completion dates for milestones listed in the 2012 and 2017 plans may not present an accurate picture of the status of the milestones and EM’s cleanup efforts. For example, in the 2012 plan, DOE reported that only four out of 218 milestones were at risk of missing their planned completion date, while the rest were on schedule. When comparing these milestones to the 2017 plan, we found that at least 14 of them had been postponed. Similarly, the 2017 plan listed only one milestone out of 154 as forecast to miss its due date. However, because EM does not have a historical record of the changes made to the milestones, it is unclear how many of these milestones were recently revised or actually represented their original due dates because the report does not include this information. Because DOE is not consistently and comprehensively submitting complete information about the status of its cleanup, Congress and other stakeholders may not have access to reliable information to make informed decisions about billions of dollars of cleanup work. We recommended that DOE submit in EM’s annually required Future-Years Defense Environmental Management Plan all mandated requirements, as well as information on annual growth in environmental liability estimates by site, the key factors causing that growth, and an explanation of significant differences between environmental liability estimates and life cycle cost estimates. DOE agreed with our recommendation and has since said it is working toward this goal. In addition to the Future-Years Defense Environmental Management Plan, DOE is to submit a budget request each fiscal year to Congress along with an explanation of what EM cleanup activities the funding will accomplish. However, in January 2019 we found that the information EM provided to Congress in its fiscal years 2016, 2017, and 2018 budget requests did not reflect the funding some senior DOE officials said EM needs to meet its scheduled cleanup milestones. We reported that in a 2015 presentation on cleanup priorities, EM’s Deputy Assistant Secretary noted that EM’s anticipated long-term funding needs for the full costs of cleanup far exceeded the office’s annual budget requests and noted that in fiscal years 2016, 2017, and 2018, EM anticipated that it needed nearly $8 billion annually to meet scheduled milestones called for in compliance agreements. However, DOE’s budget requests for those fiscal years were $5.8 billion, $6.1 billion, and $6.5 billion, respectively—a shortfall of at least $1.5 billion per year. The Deputy Assistant Secretary also noted that if EM continued to receive about $6 billion per year in the coming 2 decades, it would face a funding shortfall of about $28 billion. He also said that the time frame for EM’s cleanup mission would likely be extended for years, thereby increasing cleanup costs and raising the environmental liability. Similarly, we reported that in a 2017 site cleanup meeting, EM’s Associate Principle Deputy Assistant Secretary for Field Operations said that in order for EM to meet all of the cleanup requirements reflected in agreements with federal and state regulators, EM would need a much larger budget than was requested in fiscal year 2018. For example, this official said that EM’s Hanford site, which received about $2.5 billion in fiscal year 2018, needed more than $4 billion per year to meet scheduled milestones to construct and operate the WTP—one of many cleanup activities at the site—for the duration of its planned mission. The official added that EM’s annual budget will not cover all needs, particularly because infrastructure maintenance, repair, and replacement needs are growing and extending the completion of cleanup further into the future. We recommended that DOE disclose the funding EM needs to meet all of its scheduled milestones. DOE agreed with this recommendation and said it plans to request the funding needed to meet its cleanup agreements. Chair DeGette, Ranking Member Guthrie, and Members of the Subcommittee, this concludes my prepared remarks. I would be happy to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact David C. Trimble, Director, Natural Resources and Environment, at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony. GAO staff who made key contributions to this testimony are Amanda Kolling (Assistant Director), Chad Clady, Kelly Friedman, Cristian Ion, Jeff Larson, Cynthia Norris, Dan Royer, and Kiki Theodoropoulos. Department of Energy Contracting: Actions Needed to Strengthen Subcontract Oversight. GAO-19-107. Washington, D.C.: March 12, 2019. High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas. GAO-19-157SP. Washington, D.C.: March 6, 2019. Nuclear Waste Cleanup: DOE Could Improve Program and Project Management by Better Classifying Work and Following Leading Practices. GAO-19-223. Washington, D.C.: February 19, 2019. Nuclear Waste: DOE Should Take Actions to Improve Oversight of Cleanup Milestones. GAO-19-207. Washington, D.C.: February 14, 2019. Department of Energy: Program-Wide Strategy and Better Reporting Needed to Address Growing Environmental Cleanup Liability. GAO-19-28. Washington, D.C.: January 29, 2019. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "EM's cleanup responsibilities include remediating contaminated soil and groundwater, deactivating and decommissioning contaminated facilities, and treating millions of gallons of radioactive waste that resulted from nuclear weapons produced during World War II and the Cold War. GAO has reported on a wide range of challenges facing EM, including management challenges and the office's increasing environmental liability. In 2017, GAO added the U.S. government's environmental liability to the list of program areas that are at high risk for fraud, waste, abuse, and mismanagement or in need of transformation. DOE is responsible for over 80 percent of the federal government's environmental liability. This testimony discusses (1) the status of DOE's environmental liability, (2) management challenges at EM, and (3) EM's reporting on its cleanup efforts. It is based on five GAO reports issued from January to March 2019, updated with information from DOE's recent Fiscal Year 2018 Agency Financial Report and 2020 budget request. In fiscal year 2018, the Department of Energy's (DOE) estimated environmental liability—that is, its estimated probable costs of future environmental cleanup—was $494 billion. Of this amount, DOE's Office of Environmental Management (EM)—which is responsible for most of DOE's cleanup activities—accounted for $377 billion. EM's portion of the liability reflects cleanup estimates for 16 sites across the United States. Two of these, the Hanford site in Washington and Savannah River site in South Carolina, have most of EM's nuclear waste stored in tanks, which is particularly costly and complicated to treat. EM's environmental liability grew by $214 billion in fiscal years 2011 through 2018, even though EM spent over $48 billion on cleanup. GAO found that this liability may continue to grow for several reasons: EM's environmental liability does not include the costs of all future cleanup responsibilities. For example, as of April 2018, DOE and its contractor had not negotiated a cost for completing a large waste treatment facility, called the Waste Treatment and Immobilization Plant, at the Hanford site. About 30 to 60 percent of EM's cleanup budget goes toward recurring activities necessary to maintain the sites—such as physical security and infrastructure maintenance—rather than toward reducing EM's environmental liability. EM officials have not analyzed the root causes of the cost growth. GAO found that EM has not resolved long-standing management challenges. First, EM does not have a program-wide cleanup strategy and relies primarily on individual sites to locally negotiate cleanup activities and establish priorities, which does not always balance overall risks and costs. For example, the Hanford and Savannah River sites plan to treat similar radioactive tank waste differently, with Hanford's efforts possibly costing tens of billions more than Savannah River's. In addition, EM manages most of its cleanup work as operations activities, under less stringent requirements than other environmental remediation projects. For example, operations activities are not subject to independent oversight outside EM, and therefore DOE cannot hold EM accountable for its performance. GAO also found that EM has not consistently reported to Congress on its cleanup efforts as required, and the information EM has reported has been incomplete or inaccurate. Under the National Defense Authorization Act for Fiscal Year 2011, EM must annually report estimated costs and detailed funding needs for future cleanup activities. EM's fiscal year 2017 submission to Congress was only the second one since fiscal year 2011, and it did not include a detailed list of upcoming activities or funding needed to meet those activities. Finally, GAO found that information provided in EM's fiscal year 2016 to 2018 budget requests did not reflect the funding some DOE officials said it needs to meet its milestones. Budget requests for those years were for at least $1.5 billion less than the $8 billion a senior EM official said EM anticipated was needed annually to meet milestones called for in legally enforceable agreements. Since January 2019, GAO has made 20 recommendations to DOE to address the growing environmental liability and management challenges and will continue to monitor DOE's implementation of these recommendations. DOE has generally agreed with all but one of these recommendations and has noted plans to implement many of the recommendations.", "document_type": "gao"}
{"report": "The Bureau is charged with counting every person in the decennial census once, only once, and in the right place. To ensure fairness and consistency in where people are counted, for the first decennial in 1790, Congress established the concept of counting people where they usually reside. The Bureau has relied on that concept ever since. Building on the concept of usual residence, the Bureau subsequently established criteria, which it refers to as residence criteria, to determine where people should be counted during each decennial (see text box). Residence criteria 1. Count people at their usual residence, which is the place where they live and sleep most of the time. 2. People in certain types of group facilities on Census Day are counted at the group facility. 3. People who do not have a usual residence, or who cannot determine a usual residence, are counted where they are on Census Day. For most people, applying the concept of usual residence and the Bureau’s associated residence criteria is straightforward. For others who may be more mobile, like members of the military, college students, migrant farm workers, and people living in group quarters, determining where to count them can be more complicated. Therefore, for each decennial the Bureau issues guidance describing how the criteria should be applied to certain complex living situations for which people commonly request clarification. The guidance is intended to inform the public about how to respond and to assist enumerators and other Bureau staff in administering a proper count. In addition to counting the nation’s population accurately, the Bureau must complete the count and tabulate it against a backdrop of immutable deadlines. The Bureau is required by law to count the population as of April 1, 2020 (Census Day); deliver state apportionment counts to the President by December 31, 2020; and provide redistricting data to the states by April 1, 2021. To meet these deadlines and ensure an accurate count, the Bureau carries out thousands of interrelated activities before, during, and after data collection (see figure 1 for a timeline of selected key activities). The Bureau plans to use its concept of usual residence and its associated residence criteria to determine where to count people in the 2020 Census generally as it did in 2010, but in 2018 the Bureau updated its guidance on how to apply that concept to count people in six complex living situations (see figure 2 for an overview of these changes). In developing the guidance for 2020, the Bureau sought input from external stakeholders on needed changes and solicited public comments on the draft guidance through the Federal Register. In response, the Bureau received input and comments from a variety of entities including federal, state, local, and tribal governments, as well as civil rights and other advocacy organizations. Military and civilian employees of the United States deployed overseas. In 2010, overseas military and civilian employees of the United States who were U.S. citizens were counted at their home state of record for apportionment purposes only. For 2020, the Bureau decided to count these personnel differently depending on whether their permanent duty station was in the United States. Personnel assigned or stationed overseas will continue to be counted as they were in 2010. Personnel stationed in the United States while deployed overseas, however, will instead be counted at their usual home address in the United States for both apportionment and redistricting purposes. According to Bureau documentation, this change resulted from Bureau analysis of data from the Department of Defense which found that personnel deployed overseas were there for shorter periods and were likely to return to their prior usual place of residence, whereas personnel assigned or stationed overseas generally remained overseas for greater periods and often did not return to their prior stateside location. Military and civilian employees of the United States deployed, stationed, or assigned overseas who are legal U.S. residents but not citizens. For 2020, the Bureau plans to count this population the same way it counts U.S. citizens working for the federal government overseas, as described above. According to a Bureau assessment of how it counted personnel overseas in 2010, its guidance for federal agencies that provide the Bureau with data on overseas personnel was unclear on the treatment of non-citizens. According to Bureau officials, it is therefore likely that other federal agencies following that guidance generally excluded non-citizens from the 2010 count. Based on the Bureau’s assessment, the Bureau plans to make clear in its 2020 guidance that U.S.-resident non-citizens working for the federal government overseas are to be counted the same way as U.S. citizens. Bureau officials stated that this change should ensure that U.S.-resident non-citizens are counted more consistently with other U.S. residents. Crews of U.S. maritime and merchant vessels sailing between a U.S. and a foreign port. In 2010, if a U.S. maritime or merchant vessel was sailing between a U.S. and a foreign port on Census Day, then the crewmembers were not counted. For 2020, the Bureau plans to count these crewmembers at their onshore usual residence in the United States or, if they have none, then at the vessel’s U.S. port of departure or arrival. This matches how the Bureau counts crewmembers if their vessel is at a U.S. port or sailing between two U.S. ports. According to Bureau documentation, this change resulted from Bureau analysis and consultation with stakeholders (including the Maritime Administration) which found that crewmembers in each of these situations usually retain an onshore residence in the United States where they live and sleep most of the time so they should be counted in the same way. Juveniles in non-correctional residential treatment centers. For 2020, the Bureau plans to count this population at the U.S. residence where they live and sleep most of the time or, if they have no usual home address, then at the facility. In 2010, they were counted at the facility. The Bureau made this change after concluding that these juveniles typically only stay at residential treatment center facilities temporarily and generally have a usual home elsewhere to which they return after treatment is completed. Religious group quarters residents. For the 2020 Census, the Bureau will count this population at the religious group quarters facility. In 2010, this population was counted at their usual home address or, if they had no usual home address, then at the facility. The Bureau made this change after concluding that this population typically does not have a place of usual residence elsewhere. According to Bureau officials, the Bureau expects the updated guidance will provide greater clarity and result in more informed responses and, thus, higher quality data. Among other things, Bureau officials stated that the data will more accurately reflect the composition of local communities. For the 2020 Census, the Bureau did not change its guidance regarding where to count people in other complex living situations. For example, the Bureau did not make changes to where it will count college students, who will continue to be counted at their parents’ or guardians’ home if they live and sleep there most of the time or, if they live away from their parents’ or guardians’ home, then at their on- or off-campus residence. See table 1 for an overview of where the Bureau will count people in complex living situations the same as it did in 2010. In addition, the Bureau’s guidance includes examples of situations in which people should not be counted in the census, such as the following: people living outside the United States on Census Day who are not military or civilian employees of the U.S. government and are not dependents living with military or civilian employees of the U.S. government; babies born after Census Day or people who die before Census Day; college students living at and attending college outside the United States; and citizens of foreign countries visiting the United States, such as on vacation or a business trip. To help census respondents understand who and where to count household members and others, the Bureau is translating key terms from its census form for 2020 into 59 languages and making it available to community partners and others who may be in a position to help linguistically isolated groups provide accurate responses. It is translating scripted responses to questions about complex living situations into 12 foreign languages to be used by staff who will help answer questions about and take responses over the telephone. The Bureau reports that stakeholder feedback on where to count prisoners largely urged the Bureau to count them at their pre- incarceration addresses to avoid shifting political power to the prison locations at the expense of the prisoners’ home communities. However, the Bureau concluded that counting prisoners anywhere other than the correctional facility would be less consistent with the concept of usual residence, since the majority of people in prisons live and sleep there most of the time. Therefore, for 2020, the Bureau decided that it will continue to count prisoners at the correctional facility as it did in 2010. However, the Bureau will make available to states two tools to allow them to “move” their prisoner population to the prisoners’ pre-incarceration addresses for redistricting purposes. The tools are intended to support such movement within but not across state boundaries. The Bureau is providing states with an online tool that will identify the census geographical block that the population would be tabulated in for any state-provided addresses. If a state wants to “move” the tabulation of specific prisoners within its boundaries, this information will let state officials know which block tabulations to adjust. On November 4, 2019, the Bureau launched the web page that will support states in using this tool. The Bureau plans to update it with 2020 Census geographic data in February 2021, before the Bureau is required to provide redistricting data to the states. The Bureau also plans to provide states with data on group quarters, which will contain a separate count of their prisoner populations, as part of each state’s redistricting file. By including group quarters data in the redistricting file, the Bureau plans to provide these group quarters data to users 1 to 2 months earlier than it did in 2010 when it provided group quarters data separately from the redistricting file. According to the Bureau, this earlier release will benefit many users, including state officials who must consider whether to include or exclude certain populations when redrawing boundaries as a result of state legislation. Once the Bureau has completed its decennial data collection efforts, it generally finds that a small proportion of responses have data quality issues that were not resolved during preceding operations. In these cases, (1) some addresses have multiple responses, (2) some households are missing responses altogether, or (3) some responses include answers that are incomplete or conflict with one another. The Bureau has a variety of plans to resolve these issues (see figure 3). The Bureau may receive multiple census responses from a household for various reasons. For example, different members of the same household could each respond by mail or over the internet, or one member could mail a response and another answer a census worker’s questions in person during the Bureau’s non-response follow-up operation. The Bureau assessed its response processing in 2010, and identified about 14 million responses for households that already had another response (roughly 10 percent of the total number of households included in the final 2010 count). The widespread option to respond over the internet is new for 2020, and while having included it in multiple census tests, Bureau officials have not set expectations on the extent to which it may increase the number of addresses at which it gets multiple responses. To guard against overcounting, as it has in prior decennials, for 2020 the Bureau plans to use an automated routine—referred to as the Primary Selection Algorithm—to determine whom to count at addresses for which it has received multiple responses once data collection is complete. According to Bureau officials, in making this determination, the algorithm takes into account a wide range of information, including results from its fraud detection efforts. We did not examine the algorithm for this review. In addition, to help ensure the integrity of these determinations, the Bureau does not disclose the details of the algorithm publicly and permits only Bureau officials with an operational need to know to access the algorithm. According to Bureau documentation, the Bureau has updated the algorithm for 2020 based on its review of various response scenarios and data from past censuses and census tests. When the Bureau, after its data collection efforts are completed, has been unable to reach anyone able and willing to respond at a particular address or to obtain information about the address and its potential occupants in other ways—such as through neighbors or a building manager—it may be left not knowing whether a housing unit even exists at the address or whether it is occupied, and, if so, by how many people. As it did in 2010, for 2020 the Bureau plans to use a statistical technique it refers to as count imputation to fill in missing data about the existence and number of people living at an address in question. Count imputation has three types. Residence status. This is used when the Bureau does not know whether an address is a real and livable residence. In contrast, it could be a business or in such disrepair that no one could live there. Occupancy status. This is used when the Bureau knows that an address is a real housing unit, but not whether it is vacant or occupied. Household size. This is used when the Bureau knows an address is a real, occupied home, but not how many people live there. To carry out each of these types of count imputation, the Bureau uses a technique referred to as hot-deck imputation which employs continually updated census data from similar nearby households as the basis for filling in the missing statuses and household size. The Bureau has been using some form of hot-deck imputation since at least the 1960 Census. According to Bureau reporting, in 2010, about 500,000 of 137 million addresses counted in the decennial (0.4 percent) were missing an entire response and the Bureau therefore used count imputation to determine a combination of their residence and occupancy status and household size. The Bureau’s count imputation in 2010 added about 1.2 million people to the final census count. For 2020, however, some of the missing responses which otherwise would have required count imputation will instead be resolved through the use of administrative records in conjunction with the Bureau’s door-to-door non-response follow-up effort. Specifically, the Bureau plans to draw on relevant data from records of sufficient quality thereby reducing the amount of follow-up field work needed and the number of households in need of count imputation. According to Bureau officials, they plan to finalize a decision memorandum in December 2019 specifying the Bureau’s thresholds for determining whether administrative records are of sufficient quality for such uses. According to Bureau officials, tests and evaluations performed during the preceding decade demonstrate that these uses of administrative records will provide more accurate results than traditional methods of seeking information about the address from neighbors and others or from count imputation alone. Bureau testing and evaluation also identified improvements to its count imputation technique for 2020, including enhanced use of administrative records in its hot-deck imputation, which it expects will generate results better aligned with actual data for those addresses where it had been missing. Once the Bureau has determined the total number of households and people as of Census Day for apportionment purposes, it resolves incomplete and conflicting information within individual household responses for redistricting and final tabulation purposes. Specifically, for 2020, the Bureau plans to ensure that each household response includes complete and consistent information regarding, for occupied housing units, the age, date of birth, sex, race, and ethnicity (Hispanic or non- Hispanic origin) of each household resident; their relationship to the householder; whether the housing unit is rented or owned by a member of the household; for group quarters, the type of such quarters, such as federal detention center or in-patient hospice facility; and, if the unit is vacant, why (see figure 4). These characteristics (which the Bureau refers to as person and housing characteristics) may be incomplete or conflicting for various reasons, including intentional or accidental omissions or errors by the person filling out the form. According to Bureau data, in 2010, responses for 13 percent of the people counted in the decennial (about 40 million of the about 300 million counted) contained incomplete or conflicting person characteristics that the Bureau had to resolve. For 2020, as it did for 2010, the Bureau plans to use a technique it refers to as edit and characteristic imputation to fill in incomplete and reconcile conflicting information in individual household responses. As summarized in table 2 and described below, it does so using one of three methods, depending on which characteristics are incomplete or conflicting and on what other information the Bureau has about those characteristics. The Bureau has been using some form of characteristic imputation since at least the 1940 Census. Use existing information about same person or household. This method is used when some, but not all, person and household characteristics are incomplete or conflicting and those characteristics can be filled in or reconciled using other information about the same person or household reported within the 2020 response or in prior census or other administrative records. For example, if a person’s date of birth is reported but not his or her age, the Bureau will fill in the age based on the date of birth. If neither age nor date of birth is reported, the Bureau will look to the 2010 Census to fill in both characteristics, adjusting for the intervening years. Use existing information about other people or households. As with the prior method, this method is used when some, but not all, person and household characteristics are incomplete or conflicting. However, unlike the prior method, the incomplete or conflicting characteristics cannot be filled in or reconciled using other information about the same person or household reported within the 2020 response or in prior census or other administrative records. Therefore, the Bureau will look instead to information about other people included in the same 2020 response or to nearby people or households from other 2020 responses. For example, if race is reported for a parent but not a child, the Bureau will fill in the child’s race using the race provided for the parent. If there is no information within the household response that can be used to fill in the child’s race, the Bureau will use a hot-deck imputation method. As discussed earlier, this method employs continually updated census data from similar nearby households as the basis for filling in the needed information. Use existing information about same or other households. This method is used when all person characteristics are incomplete. In this instance, the Bureau will first look to prior census and other administrative records. If the household size reflected in those records matches the household size reflected in the 2020 response, the Bureau will use those records to fill in all person characteristics available in previous census and administrative records. Remaining characteristics not filled in by previous census and other administrative records will be filled in using the methods discussed above. If the household size totals do not match, the Bureau will use a hot-deck imputation method, drawing the missing information from continually updated census data from similar nearby households. As with other areas of the 2020 Census, the key change in the Bureau’s plan for resolving incomplete and conflicting information is the enhanced use of prior census and other administrative records. Specifically, in 2010 the Bureau relied on such records to fill in only race and ethnicity. In contrast, as discussed above, for 2020 the Bureau plans to use prior census and other administrative records as an integral part of its edit and characteristic imputation methods. The Bureau believes this will result in improved data quality and more accurate results. We provided a copy of this draft report to the Department of Commerce. The Census Bureau provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Commerce, the Director of the U.S. Census Bureau, and the appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report please contact me at (202) 512-2757 or goldenkoff@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the contact named above, Ty Mitchell (Assistant Director), Karen Cassidy and Emmy Rhine Paule (Analysts-in-Charge), Mark Abraham, Joy Booth, Ann Czapiewski, Brenda S. Farrell, Robert Gebhart, Gretta Goodwin, Amalia Konstas, Lisa Pearson, Cynthia Saunders, Andrea Starosciak, Jon Ticehurst, and Peter Verchinski made significant contributions to this report. The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is through our website. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. You can also subscribe to GAO’s email updates to receive notification of newly posted products. The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts. Visit GAO on the web at https://www.gao.gov. James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 Please Print on Recycled Paper.", "summary": "The decennial census produces data vital to the nation. The data are used for congressional apportionment and redistricting; to allocate billions each year in federal funds; and to provide a social, demographic, and economic profile of the nation to guide policy decisions at all levels of government. Given census data's importance, it is incumbent upon the Bureau to ensure their quality. If people are counted in the wrong place, some states and localities may unduly lose or gain political power through apportionment and redistricting disproportionate to their actual population. Poor outcomes can also result if some households are over counted due to multiple responses, not counted due to missing responses, or miscounted due to incomplete or conflicting responses. GAO was asked to describe the Bureau's plans for the 2020 Census to resolve multiple, missing, incomplete, and conflicting responses. This report describes how, for 2020, the Bureau plans to (1) determine where to count people, including those in complex living situations, and how this differs from 2010; and (2) resolve multiple, missing, incomplete, and conflicting responses after data collection, and how this differs from 2010. GAO reviewed relevant Bureau documents and interviewed officials responsible for the 2020 Census. GAO provided a draft of this report to the Bureau. The Bureau provided technical comments, which were incorporated as appropriate. To determine where people should be counted during each decennial census, the Census Bureau (Bureau) has established residence criteria (see figure). For most people, applying these criteria is straightforward. For others who may be more mobile, like members of the military, college students, migrant farm workers, and people living in group quarters such as federal detention centers or in-patient hospice facilities, it can be more complicated. Therefore, for each decennial the Bureau issues guidance describing how the criteria should be applied to certain complex living situations. For the 2020 Census, the Bureau has updated its guidance on where to count people in six complex living situations, such as U.S. employees deployed overseas. The Bureau plans to count people in other living situations in the same manner as it did in 2010. As in 2010, the Bureau will count prisoners at the correctional facility where they are housed, but also plans to make other resources available to states that want to use prisoners' in-state, pre-incarceration addresses for redistricting purposes instead of their prison addresses. To resolve multiple responses for a single address, for 2020 the Bureau plans to use a longstanding automated routine—its Primary Selection Algorithm—to determine who to count at the address. For 2020, Bureau documents indicate it updated the algorithm after reviewing various response scenarios and data from past censuses and tests. To resolve missing household responses following data collection, as it did in 2010, the Bureau plans to use for 2020 a technique it refers to as count imputation, which draws data from similar nearby households to determine whether a housing unit exists, whether it is occupied, and, if so, by how many people. However, for 2020, the Bureau will also try to reduce the number of households which otherwise would have required count imputation and help reduce follow-up field work by drawing on relevant data from administrative records of sufficient quality in conjunction with its non-response follow-up field work. To resolve incomplete and conflicting information within a household response, the Bureau plans to use a technique it refers to as edit and characteristic imputation. This technique involves drawing data from the same household response, prior census and other administrative records or similar nearby households, which the Bureau believes will improve data quality and produce more accurate results.", "document_type": "gao"}
{"report": "According to the 2015 report ordered by the Secretary of Defense and issued by the Military Justice Review Group, the military justice system is designed to ensure discipline and order in the armed forces, since crimes committed by servicemembers have the potential to destroy the bonds of trust, seriously damage unit cohesion, and compromise military operations. The jurisdiction of the UCMJ extends to all places and applies to all active-duty servicemembers. UCMJ jurisdiction applies to other individuals as well, such as members of the National Guard or reserves who are performing active-duty service; retired members who are entitled to pay or are receiving hospitalization in a military hospital; prisoners of war in custody of the armed forces; persons serving with or accompanying the armed forces in the field in time of declared war or contingency operations, such as contractors; and members of organizations such as the National Oceanic and Atmospheric Administration and the Public Health Service when assigned to and serving with the armed forces. In creating the military justice system, Congress established three types of military courts, called courts-martial: summary, special, and general. Each of these types respectively is intended to deal with progressively more serious offenses, and each court-martial type may adjudicate more severe maximum punishments as prescribed under the UCMJ. In addition, an accused servicemember can receive nonjudicial punishment under Article 15 of the UCMJ, by which a commander can punish a servicemember without going through the court-martial process. Table 1 provides an overview of nonjudicial punishments and the three different types of courts-martial. The Military Justice Act of 2016 enacted significant reforms to the UCMJ, most of the provisions of which became effective on January 1, 2019. These reforms included changes such as limitations on the types of punishments permitted with nonjudicial punishments, changes to required size of the panel, or jury, and changes to what judicial outcomes are subject to automatic appeal. There are some areas where individual services supplement but remain consistent with the UCMJ. For example, the Air Force provides a right to counsel in certain forums where the services are not required to do so. In addition to the reforms affecting the UCMJ, the Military Justice Act of 2016 also directed changes to military justice data collection and accessibility. Specifically, section 5504 of the Military Justice Act of 2016 directed the Secretary of Defense to prescribe uniform standards and criteria pertaining to case management, data collection, and accessibility of information in the military justice system. As a result, the DOD Office of General Counsel authorized the establishment of the Article 140A Implementation Subcommittee of the Joint Service Committee on Military Justice to, among other things, assess each service’s case management system, recommend what data fields the services should collect, propose uniform definitions for the data fields the services should collect, and recommend standardized methods and data field definitions to improve the collection of data concerning race and ethnicity of individuals involved in the military justice system. The subcommittee conducted a study and submitted its recommendations to the Joint Service Committee Voting Group on July 2, 2018, and the Voting Group submitted a report and its agreed upon recommendations to the DOD Office of General Counsel on August 24, 2018. The Military Justice Act of 2016 provides that the Secretary of Defense was to carry out this mandate by December 23, 2018, and that the Secretary’s decisions shall take effect no later than December 23, 2020. On December 17, 2018, the General Counsel of the Department of Defense issued uniform standards and criteria, which directed that each military justice case processing and management system be capable of collecting uniform data concerning race and ethnicity. From fiscal years 2013 through 2017, more than 258,000 active-duty servicemembers were disciplined for a violation of the UCMJ, out of more than 2.3 million unique active-duty servicemembers who served across all of the military services during this period. Figure 1 shows the number of cases of each type of court-martial and of nonjudicial punishments in each of the military services. There are several steps in the discipline of a servicemember who allegedly commits a crime under the UCMJ, which are summarized in figure 2 below. The military justice process begins once an offense is alleged and an initial report is made, typically to law enforcement, an investigative entity, or the suspect’s chain of command. Policies for initiating criminal investigations by military criminal investigative organizations (MCIO) and procedures for investigating criminal allegations are set forth in DOD and service guidance. At this time, the commanding officer or law enforcement will conduct an inquiry or investigation into the accusations and gather all reasonably available evidence. MCIOs have the authority and independent discretion to assume investigative jurisdiction, and do not require approval from any authority outside of the MCIO to conduct such an investigation—commanders outside of the organization are not to impede or interfere with such decisions or investigations by the MCIO. If an MCIO is involved in the inquiry, the investigative entity is to gather all reasonably available evidence and provide the commanding officer with unbiased findings that reflect impartiality as required by DOD instruction. According to service officials, during the conduct of the criminal investigation, the subject of the investigation has the right to obtain legal counsel at any time. After an investigation, the first step toward initiation of a court-martial is when the accused is presented with a list of charges signed by the accuser under oath, which is called preferral of charges; the accuser who prefers the charges may be anyone subject to the UCMJ. After charges are preferred, the charges are forwarded to an officer with sufficient legal authority to convene a court-martial, also known as the “convening authority.” The convening authority in receipt of preferred charges may, among other actions and depending on the nature of the charges and the level of the convening authority, refer the case to its own court or forward the case to a superior commander for disposition, for example, to a general court-martial convening authority. The general court-martial convening authority would have similar options: to dismiss the charges, refer them to a general or special court-martial, or take some lesser action. Before any case is referred to a general court-martial, the case must proceed through a preliminary hearing under Article 32 of the UCMJ, unless waived by the accused. The Article 32 hearing is presided over by an impartial judge advocate, or another individual with statutory authority, who is appointed by the convening authority and makes a recommendation to the convening authority. We analyzed general and special courts-martial that were preceded by investigations recorded in databases maintained by MCIOs, which we refer to as recorded investigations, and general and special courts-martial that did not have a record within an MCIO database. As shown in figure 3 below, the majority of general and special courts-martial, ranging from 53 percent to 74 percent across the services, had a recorded investigation, while the remaining cases would have been investigated by other sources, such as local civilian law enforcement, command investigations, or in the case of the Air Force, their military law enforcement forces. Once referred to a general or special court-martial, an accused servicemember may be tried by a military judge alone or by a military judge with a military jury, referred to as members of the court-martial. If the accused servicemember is tried by a military jury, the members of the court-martial determine whether the accused is proven guilty and, if the accused requests sentencing by the members, adjudicate a sentence. Otherwise, the military judge adjudicates the sentence. If the accused is tried by a military judge alone, the judge determines guilt and any sentence. In a summary court-martial, a single commissioned officer who is not a military judge adjudicates minor offenses and a sentence. Convictions at the general and special court-martial level are subject to a post-trial process and may be appealed to higher courts in cases where the sentence reaches a certain threshold. For example, depending on the forum and the adjudged sentence, the accused may be entitled to appellate review by the service Court of Criminal Appeals, and may be able to request or waive assignment of appellate defense counsel, or waive appellate review entirely. Depending, again, on forum and sentence, some cases that do not qualify for appellate review will receive review by a judge advocate to, among other things, determine that the court had jurisdiction and that the sentence was lawful. Some cases may then be further reviewed by the Court of Appeals for the Armed Forces, as well as by the U.S. Supreme Court at their discretion, if the case was reviewed by the Court of Appeals for the Armed Forces. The military justice system, like the civilian criminal justice system, provides avenues for accused servicemembers to raise allegations of discrimination, improprieties in investigations, improprieties in disposition, and improprieties in the selection of panel members at the court-martial proceeding, before a military judge and on appellate review. The Military Justice Act of 2016 requires that legal training be provided to all officers, with additional training for commanders with authority to take disciplinary actions under the UCMJ. The Office of Management and Budget (OMB) has established standards for collecting, maintaining, and presenting data on race and ethnicity for all federal reporting purposes. These standards were developed in cooperation with federal agencies to provide consistent data on race and ethnicity throughout the federal government. OMB standards establish the following five categories of race: American Indian or Alaska Native: A person having origins in any of the original peoples of North and South America (including Central America), and who maintains tribal affiliation or community attachment. Asian: A person having origins in any of the original peoples of the Far East, Southeast Asia, or the Indian subcontinent including, for example, Cambodia, China, India, Japan, Korea, Malaysia, Pakistan, the Philippine Islands, Thailand, and Vietnam. Black or African American: A person having origins in any of the black racial groups in Africa. Native Hawaiian or Other Pacific Islander: A person having origins in any of the original peoples of Hawaii, Guam, Samoa, or other Pacific Islands. White: A person having origins in any of the original peoples of Europe, the Middle East, or North Africa. The OMB standards also establish two categories of ethnicity. Hispanic or Latino: A person of Cuban, Mexican, Puerto Rican, South or Central American, or other Spanish culture or origin, regardless of race. Not Hispanic or Latino: A person not having the above attributes. In addition to defining race and ethnicity for federal administrative reporting and record keeping requirements, OMB standards provide two methods for federal agencies to follow regarding the collection of data on race and ethnicity. 1. Separate questions shall be used for collecting information about race and ethnicity wherever feasible. In this case, there are 5 categories of race noted above which individuals can select, and individuals can identify with more than one category of race. In addition to race, individuals can select one of the two ethnicity categories above. 2. If necessary, a single question or combined format can be used to collect information about race and ethnicity, where the following categories are provided for individuals: American Indian or Alaska Native, Asian, Black or African American, Hispanic or Latino, Native Hawaiian or other Pacific Islander, and White. In this instance, individuals can also select more than one category. Information collected on servicemembers’ gender is governed by DOD guidance. DOD Instruction 1336.05 provides that information collected on a servicemember’s gender is based on reproductive function. It provides that there are three options that can be selected when inputting a servicemember’s gender: male, female, or unknown. Racial and gender disparities in the civilian criminal justice system have been the subject of several studies in the past decade. While the civilian and military justice systems differ from each other, we reviewed information about racial and gender disparities in the civilian criminal justice system to enhance our understanding of the complexities of the issues, including how others had attempted to measure disparities. Some studies have assessed the rates at which minority groups are policed. For example, a Department of Justice study of data from the Bureau of Justice Statistics’ 2011 Police-Public Contact survey found that Black drivers were more likely than White or Hispanic drivers to be pulled over in a traffic stop; specifically, the study found that 10 percent of White drivers and 10 percent of Hispanic drivers were pulled over in a traffic stop, compared to 13 percent of Black drivers. This study also found that Black and Hispanic drivers were more likely to be searched once they were pulled over by the police; specifically, the study found that 2 percent of White drivers stopped by police were searched, compared to 6 percent of Black drivers and 7 percent of Hispanic drivers. In addition, U.S. government data shows that racial disparities exist among individuals who are arrested. For example, data from the Federal Bureau of Investigation’s Uniform Crime Reporting Program, which compiles data from law enforcement agencies across the country, indicates that in 2016, Black individuals represented 26.9 percent of total arrests nationwide, but comprised 13.4 percent of the U.S. population according to U.S. census data estimates as of July 1, 2017. This data also shows that 69.6 percent of all arrested individuals were White, while White individuals comprised 76.6 percent of the U.S. population. Studies have also identified racial and gender disparities in civilian justice sentencing. In 2010 and 2017, the U.S. Sentencing Commission reported that Black male offenders received longer sentences than similarly situated White male offenders. Specifically, in 2017, the Commission analyzed federal sentencing data and reported that Black male offenders received sentences that on average were 19.1 percent longer than similarly situated White males for fiscal years 2012 to 2016. This analysis controlled for factors such as type of offense, race, gender, citizenship, age, education level, and criminal history. This study also found that female offenders of all races received shorter sentences than White male offenders. Similarly, the Commission’s 2010 report found that Black offenders received sentences that were 10 percent longer than those imposed on White offenders from December 2007 through September 2009, and male offenders received sentences that were 17.7 percent longer than female offenders, after controlling for the same factors as noted for the 2017 study, among others. Finally, racial and gender disparities have been identified among incarcerated populations. According to data from the Bureau of Justice Statistics, for prisoners with sentences of 1 year or more under the jurisdiction of state or federal correctional officials in 2016, Black males were six times more likely to be imprisoned than White males, and Hispanic males were 2.7 times more likely to be imprisoned than White males. The racial disparities were more pronounced for younger males, where Black males aged 18 to 19 were approximately 11.8 times more likely than White males of the same age to be imprisoned. The Bureau also reported that Black females were imprisoned at approximately twice the rate of White females. We did not assess the methodologies used in any of these studies or the reliability of the data cited in the studies; these studies are discussed here to provide broader context for the discussion about racial and gender disparities in the military justice system. The military services collect and maintain gender information, but they do not collect and maintain consistent information about race and ethnicity in their investigations, military justice, and personnel databases. This limits the military services’ ability to collectively or comparatively assess these demographic data to identify any racial or ethnic disparities in the military justice system within and across the services. The military services use different databases to collect and maintain information for investigations, courts-martial, and nonjudicial punishments. All of the databases collect and maintain gender information, but the Coast Guard’s military justice database does not have the capability to query or report on gender data. While the military services’ databases collect and maintain complete data for race and ethnicity, the information collected and maintained about race and ethnicity is not consistent among the different databases within and across the services. Moreover, the Coast Guard, the Navy, and the Marine Corps do not collect and maintain complete and consistent servicemember identification data, such as social security number or employee identification number, in their respective military justice databases, although DOD leadership recently directed improvements in this area. Finally, the military services do not report data that provides visibility into disparities in the military justice system, and DOD and the services lack guidance about when potential racial, ethnic, or gender disparities should be further reviewed, and what steps should be taken to conduct such a review if needed. Each military service uses a different database to collect and maintain information on investigations and courts-martials, and, in some services, nonjudicial punishments, as shown in figure 4. For three of the military services—the Army, the Navy, and the Coast Guard—the databases listed in figure 4 include information about some, but not all, of their nonjudicial punishment cases. Additionally, the nature of the information collected by each of the services’ databases varies, as noted below. Investigations. The Army collects and maintains information on investigations conducted by the Army Criminal Investigation Command in the Army Law Enforcement Reporting and Tracking System database. According to Army officials, the Office of the Provost Marshal General and the Army Criminal Investigation Command developed this database to replace a 2003 system, the Army Criminal Investigation and Intelligence System, and a significant part of the military police’s 2002 system, the Centralized Operations Police Suite. The officials said that the Army Law Enforcement Reporting and Tracking System has been operational since 2015, and has become the primary case management system for all Army law enforcement professionals. However, Army officials said that cases involving commander-led investigations are unlikely to be recorded in this database. Courts-martial and nonjudicial punishments. The Army uses Military Justice Online and the Army Courts-Martial Information System to collect data on court-martial cases. According to Army officials, Military Justice Online, created in 2008, is a document- generating system that primarily is used by the Army’s judge advocate general corps and promotes uniformity in case processing among the Army’s staff judge advocate offices. Military Justice Online includes information about courts-martial, some nonjudicial punishments, administrative separations, and administrative reprimands of servicemembers. Army officials said that the Army Courts-Martial Information System, which has been used since 1989, serves as the Army trial judiciary’s case tracking system and is used by the Army’s trial judiciary to track court-martial cases. Investigations. The Air Force military criminal investigative organization, the Office of Special Investigations, uses a system called the Investigative Information Management System to collect and maintain information related to investigations. According to Air Force officials, the Investigative Information Management System has been in use since 2001. Courts-martial and nonjudicial punishments. The Air Force uses the Automated Military Justice Analysis and Management System, which is designed to be a case management system to collect comprehensive information for both court-martial cases and nonjudicial punishments. According to Air Force officials, the Automated Military Justice Analysis and Management System has been in use since 1974. Investigations. According to Navy officials, the Navy and Marine Corps’ joint system for maintaining and collecting information related to investigations is the Consolidated Law Enforcement Operations Center, which has been in use since 2004. Navy officials said that this database initially contained information regarding Navy and Marine Corps law enforcement incidents and criminal investigations, but began to include investigations conducted by the Naval Criminal Investigative Service in 2012. Courts-martial. The Navy and the Marine Corps both use the Case Management System to collect and maintain information about military justice matters with involvement by a Navy or Marine Corps legal office, including special and general court-martial cases. This system was initially developed by the Marine Corps to track information about legal services provided by their legal offices. According to Navy and Marine Corps officials, the system has been in use by the Marine Corps since 2010 and by the Navy since 2013. Officials from the Marine Corps said that although the Case Management System has been in use since 2010, the system was not widely used until 2012. Nonjudicial punishments. The Marine Corps Total Force System, the Marine Corps personnel database, collects and maintains information on summary courts-martial and nonjudicial punishments for cases where there was a conviction or punishment. According to Marine Corps officials, this system has been in use since 1995. Navy officials said that their personnel database records information about nonjudicial punishments if the punishment involved a change in pay or grade. The services’ military justice Case Management System includes information on some nonjudicial punishment cases in the Navy and the Marine Corps, which Navy and Marine Corps officials said was for those cases that had involvement by their legal offices. Investigations. The Coast Guard Investigative Service uses the Field Activity Case Tracking System to collect and maintain information on servicemembers investigated for violations of the UCMJ. According to Coast Guard officials, this system has been in use since July 2014. Courts-martial. According to Coast Guard officials, the Coast Guard uses Law Manager to collect and maintain administrative information on court-martial cases. Law Manager has been in use since 2000, but was not used for court-martial data until 2003. Nonjudicial punishments. Coast Guard officials said that their military justice database contains records of nonjudicial punishments if a case involved their legal offices. In addition, according to Coast Guard officials, Direct Access, the Coast Guard’s personnel database, also collects and maintains information about some court-martial cases and nonjudicial punishments if the punishment resulted in a change in rank or pay or an administrative action against the accused servicemember. All of the military services collect and maintain gender information in their investigations, military justice, and personnel databases, but are inconsistent in whether they allow an unknown or unspecified gender, and the Coast Guard’s military justice database does not allow Coast Guard officials to query or report on gender data. Table 2 below summarizes how data regarding the servicemember’s gender is entered into the services’ databases and the number of potential gender options. Each database identifies at least two potential options—male and female—for data related to the servicemember’s gender, while about half of the databases (8 of 15) provide a third option to indicate that the gender is either unknown or not specified. Each of the military services’ investigations, military justice, and personnel databases maintained gender data for almost 100 percent of servicemembers, except we were unable to determine this completion rate for the Coast Guard’s military justice database. We could not determine the completeness of the Coast Guard’s gender data in its military justice database because, as previously noted, its military justice database does not have the capability to query on gender data. Standards for Internal Control in the Federal Government states that management should use quality information and obtain data on a timely basis so they can be used for effective monitoring. However, the Coast Guard does not have visibility over the gender of servicemembers prosecuted for UCMJ violations without merging data from multiple databases, which can be a labor-intensive and time-consuming process. According to Coast Guard officials, information regarding the gender of servicemembers prosecuted for UCMJ violations can be recorded in its military justice database, but gender is not a field that can be searched on or included in the reports they run using information from their military justice database, because of the way the military justice module in the database was designed. Coast Guard officials told us that the military justice database—Law Manager—was designed to determine the status of court-martial cases, and captures attributes that are generated by relevant UCMJ documents. Those official documents do not require the annotation of demographics such as gender, so this information is not used in Law Manager. A Coast Guard official indicated that it would be feasible to modify Law Manager to make it easier to run reports and queries that include gender information. The ability to query and report on the gender of servicemembers in its military justice database would provide the Coast Guard with more readily available data to identify or assess any gender disparities that may exist in the investigation and trial of military justice cases. Each of the military services’ databases collect and maintain complete data for race and ethnicity, but the military services do not collect and maintain consistent information regarding race and ethnicity in their investigations, military justice, and personnel databases. Additionally, the military services have not developed a mechanism to aggregate the data into consistent categories of race and ethnicity to allow for efficient analysis and reporting of consistent demographic data. The number of potential responses for race and ethnicity within the 15 databases across the military services ranges from 5 to 32 options for race and 2 to 25 options for ethnicity, which can complicate cross-service assessments. For example, the Army’s personnel database maintains 6 options for race and 23 options for ethnicity, whereas the Coast Guard’s personnel database maintains 7 options for race and 3 for ethnicity. Table 3 summarizes how the databases used by the military services vary in how the servicemember’s race is entered and the number of potential race options. Table 4 shows that the military services’ databases also vary in how information about servicemembers’ ethnicity is entered into the databases and the number of potential ethnicity options that are collected. Although the data collected and maintained was not consistent within and across the military services, each of the military services’ databases maintained race and ethnicity data for at least 99 percent of the servicemembers, with the exception of the Coast Guard. The Coast Guard does not track information about race or ethnicity in its military justice database. Coast Guard officials stated that this is because Law Manager was designed to determine the status of court-martial cases, and captures attributes that are needed to generate relevant UCMJ documents, such as court pleadings. Demographic information such as race and ethnicity is not included in these official documents, so this information is not input into Law Manager. Further, four of the databases we reviewed—including both of the Army’s military justice databases, and the Navy and the Marine Corps’ military justice databases—collect information on race and ethnicity in a combined data field as shown in table 4, whereas the other databases collect and maintain race and ethnicity information in two separate fields. Standards for Internal Control in the Federal Government states that management should use quality information to achieve the entity’s objectives. Among other things, attributes of this internal control principle call for management to identify information requirements; obtain relevant data from reliable sources that are reasonably free from error; ensure that the data it receives is timely and reliable; and process the data obtained into quality information— information that is appropriate, current, complete, and accurate. In addition, federal internal control standards call for management to design the entity’s information system and related control activities to achieve objectives and respond to risks, thereby enabling information to become available to the entity on a timelier basis. Further, the Military Justice Act of 2016 required the Secretary of Defense to prescribe uniform standards and criteria for various items, including data collection and analysis for case management at all stages of the military justice system, including pretrial, trial, post-trial, and appellate processes, by December 2018. On December 17, 2018, the General Counsel of the Department of Defense issued the uniform standards and criteria required by article 140a of the Military Justice Act of 2016. As part of these uniform standards, the services were directed to collect data related to race and ethnicity in their military justice databases, and to collect racial and ethnic data in separate data fields. The standards provide that the services may have their military justice databases capture expanded ethnic or racial categories; however, for reporting purposes, expanded categories will aggregate to those categories listed in the standards. For race, the services will choose from six designations: (1) American Indian/Alaska Native, (2) Asian, (3) Black or African American, (4) Native Hawaiian or Other Pacific Islander, (5) White, or (6) Other. For ethnicity, the services will choose from two options: (1) Hispanic or Latino, or (2) Not Hispanic or Latino. These categories are consistent with the OMB standards for collecting and presenting such data. The military services are to implement the Secretary’s direction no later than December 23, 2020. However, DOD has applied these newly issued standards only to the military justice databases and not to the investigations and personnel databases. DOD officials stated that the investigations and personnel databases do not fall under the charter of the DOD General Counsel, which issued the standards for the military justice databases. Hence, these uniform standards do not apply to the military services’ investigations and personnel databases. We were able to analyze data across the investigations, military justice, and personnel databases by merging data from these databases, but this took multiple, detailed steps and would not be an efficient approach for routine analyses. Taking steps to develop the capability to present the race and ethnicity data in the military services’ personnel and investigations databases using the same categories included in the December 2018 standards for the military justice databases would allow for more efficient analysis of consistent demographic data. This could be done through either collecting and maintaining race and ethnicity data in the investigations and personnel databases using the December 2018 uniform standards or developing a capability to aggregate the data into the race and ethnicity categories included in the standards. The Navy, the Marine Corps, and the Coast Guard did not collect and maintain complete servicemember identification data, such as social security number or employee identification number, in their military justice or investigations databases; however, DOD recently directed them to do so. In the course of conducting our analysis, in some instances, we could not match personnel records with military justice records because the social security number or employee identification number in the military justice database did not match the information in the personnel database. In other instances, we could not match personnel records with military justice records because the military justice records did not contain a social security number or employee identification number to match with information found in their personnel record. As shown in table 5, we initially were unable to match 5 percent of Navy military justice cases, 12 percent of Marine Corps military justice cases, 18 percent of Coast Guard investigation cases, and 6 percent of Coast Guard military justice cases. On December 17, 2018, the General Counsel of the Department of Defense issued the uniform standards and criteria required by article 140a of the Military Justice Act of 2016. As part of these uniform standards, the services were directed to collect either the social security number or DOD identification number in their military justice databases. The military services are to implement the Secretary’s direction no later than December 23, 2020. Although some military services report demographic information about the subjects of military justice actions internally, the military services do not externally report data that provides visibility into, or would enable an analysis of, the extent of racial, ethnic, or gender disparities in the military justice system. Service officials from all of the military services told us that they compile internal quarterly or monthly staff judge advocate reports, which include the total number of each type of court-martial handled by their legal offices and of nonjudicial punishments. According to service officials, in the Air Force and the Army these reports include demographic information about servicemembers involved in these cases, such as the total number of each type of case broken out by the subject’s race, ethnicity, or gender, but the Navy, Marine Corps, and Coast Guard reports do not include this demographic information, and there is no requirement to do so. Regarding external reporting, the UCMJ directs the Court of Appeals for the Armed Forces, the Judge Advocates General, and the Staff Judge Advocate to the Commandant of the Marine Corps to submit annual reports on the military justice system to the Congressional Armed Services Committees, the Secretary of Defense, the secretaries of the military departments, and the Secretary of Homeland Security. These reports are to include information on the number and status of pending cases handled in the preceding fiscal year, among other information. The annual reports include the total number of cases each service handled for each type of court-martial and for nonjudicial punishments. However, these annual reports do not include demographic information about servicemembers who experienced a military justice action, such as breakdowns by race or gender, because the reporting requirement does not direct the services to include such information. A DOD official expressed concern about expanding the reporting requirement to have public dissemination of race, ethnicity, and gender information due to the potential for misinterpretation, but stated that such reporting requirements for internal use would be beneficial. However, Congress and members of the public have expressed an interest in this information. Standards for Internal Control in the Federal Government state that management should externally communicate the necessary quality information to achieve the entity’s objectives. Furthermore, these standards state that management should use quality information to make informed decisions and evaluate the entity’s performance. According to DOD guidance, the Joint Service Committee on Military Justice, a committee comprised of representatives from each service’s legal office, is responsible for reviewing the Manual for Courts-Martial and the UCMJ on an annual basis. The Joint Service Committee can consider suggested changes to the UCMJ or the Manual for Courts-Martial or its supplementary materials from the services or from the general public. The Joint Service Committee then determines whether to propose any desired amendments to the UCMJ, or the Manual for Courts-Martial or its supplementary materials. If the Joint Service Committee finds that an amendment to either the Manual for Courts-Martial or the UCMJ is required, the committee will provide the General Counsel of DOD with a draft executive order containing the recommended amendments or will forward a legislative proposal to amend the UCMJ. While it is unclear whether the committee has ever considered or proposed an amendment to the UCMJ or Manual for Courts-Martial that would require the external reporting on an annual basis of demographic information about the race, ethnicity, and gender of servicemembers charged with violations of the UCMJ, no such change has been made. Reporting this information would provide servicemembers and the public with greater visibility into potential disparities and help build confidence that DOD is committed to a military justice system that is fair and just. Furthermore, DOD has not issued guidance that establishes criteria to specify when any data indicating possible racial, ethnic, or gender disparities in the investigations, trials, or outcomes of cases in the military justice system should be further reviewed, and to describe what steps should be taken to conduct such a review if it were needed. GAO’s Standards for Internal Control in the Federal Government provides that an agency needs to establish a baseline in order to perform monitoring activities. The baseline helps the agency understand and address deficiencies in its operations. While equal employment opportunity enforcement is a very different context than the military justice system, other federal agencies have developed such criteria in the equal employment opportunity context that can indicate when disparities should be examined further. For example, the Department of Justice, the Department of Labor, the Equal Employment Opportunity Commission, and the Office of Personnel Management use a “four-fifths” test to determine when differences between subgroups in the selection rates for hiring, promotion, or other employment decisions are significant. These criteria, though inexact, provide an example of the type of criteria that DOD could consider using as a basis for determining when disparities among racial or gender groups in the military justice process could require further review or analysis. By issuing guidance that establishes criteria for determining when data indicating possible racial and gender disparities in the investigations, trials, or outcomes of cases in the military justice system should be further examined, and describes the steps that should be taken to conduct such further examination, DOD and the services would be better positioned to monitor the military justice system to help ensure that it is fair and just, a key principle of the UCMJ. Racial and gender disparities exist in investigations, disciplinary actions, and punishment of servicemembers in the military justice system, and gender disparities exist in convictions in the Marine Corps. Our analysis of available data from fiscal years 2013 through 2017, which controlled for attributes such as race, gender, rank, education, and years of service, found racial and gender disparities were more likely in actions that first brought servicemembers into the military justice system. Specifically, we found that: Black, Hispanic, and male servicemembers were more likely than White and female servicemembers to be the subjects of recorded investigations in all of the military services, and were more likely to be tried in general and special courts-martial in the Army, the Navy, the Marine Corps, and the Air Force. There were fewer statistically significant racial and gender disparities in most military services in general and special courts-martial that were preceded by a recorded investigation than in general and special courts-martial overall. We also found that statistically significant racial and gender disparities in general and special courts-martial that did not follow a recorded investigation were similar to those we identified for general and special courts-martial overall. Black and male servicemembers were more likely than White and female servicemembers to be tried in summary courts-martial and to be subjects of nonjudicial punishment in the Air Force and the Marine Corps. The Army and the Navy did not maintain complete data, and the Coast Guard had too few summary courts-martial for us to analyze, and did not maintain complete nonjudicial punishment data. We identified fewer statistically significant racial or gender disparities in case outcomes—convictions and punishment severity. Specifically: Race was not a statistically significant factor in the likelihood of conviction in general and special courts-martial in the Army, the Navy, the Marine Corps, and the Air Force, but gender was a statistically significant factor in the Marine Corps. Black servicemembers were less likely to receive a more severe punishment in general and special courts-martial compared to White servicemembers in the Navy but there was no statistically significant difference for Black servicemembers in the Marine Corps, the Army, and the Air Force. Additionally, there were no statistically significant differences for Hispanic servicemembers in the Navy, the Marine Corps, the Army, or the Air Force; and males were more likely than females to receive a more severe punishment in the Marine Corps, the Army, and the Air Force. Finally, DOD and the military services have taken some steps to study racial and gender disparities in the military justice system over the last several decades, but they have not comprehensively studied the extent or causes of any disparities. Black, Hispanic, and male servicemembers were more likely than White or female servicemembers to be the subjects of recorded investigations in all of the military services, after controlling for other attributes, as shown in figure 5. Servicemembers in the Other race category were more likely than White servicemembers to be the subjects of recorded investigations in the Navy, but were less likely in the Army. Our analyses did not identify any statistically significant differences for servicemembers in the Other race category from the Air Force, the Marine Corps, or the Coast Guard. For the Army, the Navy, the Marine Corps, and the Air Force, Black, Hispanic, and male servicemembers were more likely than White and female servicemembers to be tried in general and special courts-martial after controlling for other attributes, as shown in figure 6 below. Servicemembers in the Other race category were more likely than White servicemembers to be tried in general and special courts-martial in the Navy, but we found no statistically significant differences in the likelihood of servicemembers in the Other race category in the Army, the Marine Corps, and the Air Force to be tried in general and special courts-martial compared to White servicemembers. We could not analyze Coast Guard cases due to the small number of general and special courts- martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. When separating general and special court-martial cases into those that either were or were not preceded by an investigation recorded in an MCIO database, we found fewer statistically significant racial and gender disparities in most of the military services in general and special courts- martial that were preceded by a recorded investigation. However, statistically significant racial and gender disparities were also present in general and special courts-martial that did not follow a recorded investigation in all services included in this analysis, which would include cases where the investigation was performed by the servicemember’s command. Specifically, as shown in figure 7 below, we found that Black, Hispanic, Other, and male servicemembers in the Army, Hispanic servicemembers in the Marine Corps, and males in the Air Force were more likely than White or female servicemembers to be tried in general and special courts- martial following a recorded investigation, after controlling for other attributes. We found no statistically significant differences in the likelihood of any other racial or gender groups to be tried in general and special courts-martial following a recorded investigation in any other services. Our analyses of general and special courts-martial with a recorded investigation generally found fewer statistically significant differences compared to the results of our analyses for all special and general courts martial. We also found that Black and male servicemembers in all of the military services were more likely than White and female servicemembers to be tried in general and special courts-martial without a recorded investigation after controlling for other attributes, as shown in figure 8 below. Further, Hispanic servicemembers in the Army were more likely than White servicemembers to be tried in general and special courts-martial without a recorded investigation, but we found no statistically significant differences in the likelihood of Hispanic servicemembers to be tried in general and special courts-martial without a recorded investigation in the Marine Corps, the Navy, or the Air Force. We found no statistically significant differences in the likelihood of servicemembers in the Other race category to be tried in general and special courts-martial compared to White servicemembers in all of the military services. Our findings of racial and gender disparities in general and special courts-martial without a recorded investigation found statistically significant differences for Black and male servicemembers consistent with the differences we identified for general and special courts-martial overall, as shown in figure 6 above. Black and male servicemembers were more likely than White or female servicemembers to be tried in summary courts-martial in the Air Force and the Marine Corps after controlling for other attributes, as shown in figure 9 below. We did not identify any statistically significant differences in summary courts-martial rates for servicemembers who identified as Hispanic or in the Other race category in either the Air Force or the Marine Corps. We could not determine whether there were racial or gender disparities for summary courts-martial in the Army, the Navy, and the Coast Guard due to data limitations. We could not analyze Coast Guard cases due to the small number of summary courts-martial adjudicated in the Coast Guard from 2013 through 2017. We could not determine whether disparities existed among servicemembers tried in summary courts-martial in the Army and the Navy because the Army and the Navy did not collect complete summary courts-martial data in their investigations, military justice, or personnel databases. Specifically, as part of our data reliability checks, we identified the total number of summary courts-martial that the Army and the Navy reported in the Court of Appeals for the Armed Forces annual reports for fiscal years 2013 through 2017, and compared these totals to the number of cases we identified in their military justice databases. While our comparisons are not exact, due to differences in the dates we used to count the number of cases, we found that approximately 60 percent of the Army’s reported summary courts-martial cases and less than 50 percent of the Navy’s reported summary courts-martial cases were included in their military justice databases. Army and Navy officials cited several reasons why complete summary courts-martial information was not collected. First, they said that the services are not required to collect and maintain complete data on summary courts-martial because these cases result in non-criminal convictions under the UCMJ. Summary courts-martial are typically used for minor offenses, and the accused is not guaranteed the right to be represented by a military attorney. As a result, military attorneys may not be involved in summary courts-martial. Army and Navy officials said that if military attorneys are not involved in the case, there is not likely to be a record of the case in their service’s military justice database. In contrast, Air Force officials said that they provide a military attorney to represent the accused in summary courts-martial; as a result, Air Force officials said their attorneys create records for these cases in the Air Force’s military justice database. The Marine Corps does not maintain summary court- martial data in its military justice database but tracks summary courts- martial in its personnel database. Officials in the Navy and the Army told us that the lack of complete summary court-martial data in their military justice databases is also in part because these systems were not designed to serve as repositories for complete military justice data. Instead, the officials said that the military justice databases were primarily created to assist attorneys in generating trial documents, meeting timeframes, and other aspects of case management. Nevertheless, Army officials said they plan to start collecting more complete summary court-martial information. Specifically, Army officials said that the Army is encouraging their judge advocate general staff to create records for all summary courts-martial in the service’s military justice database. The absence of complete summary court-martial data in the military justice databases of the Army and the Navy limits these services’ visibility into any disparities that may exist among servicemembers involved in these types of military justice proceedings. On December 17, 2018, the General Counsel of the Department of Defense issued the uniform standards and criteria required by article 140A of the Military Justice Act of 2016. As part of these uniform standards, the services were directed to collect certain information about all cases in their military justice databases, which a DOD official said includes summary courts-martial cases. The military services are to implement the Secretary’s direction no later than December 23, 2020. Black and male servicemembers were more likely than White or female servicemembers to be subject to nonjudicial punishments in the Air Force and the Marine Corps, after controlling for other attributes, as shown in figure 10 below. In the Air Force, we found that Hispanic servicemembers were more likely than White servicemembers to receive nonjudicial punishments, while we observed no statistically significant differences in nonjudicial punishment rates for Hispanic servicemembers in the Marine Corps. Servicemembers in the Other race category in the Marine Corps were less likely to receive nonjudicial punishments, but we observed no statistically significant differences in nonjudicial punishment rates for servicemembers in the Other race category in the Air Force. However, we could not determine whether there were racial or gender disparities among servicemembers subject to nonjudicial punishments in the Army, the Navy, and the Coast Guard because these services do not collect complete nonjudicial punishment data, such as data on the servicemember’s race, ethnicity, gender, offense, and punishment, in any of their databases. As part of our data reliability checks, we identified the total number of nonjudicial punishments that the Army, the Navy, and the Coast Guard reported in the Court of Appeals for the Armed Forces annual reports for fiscal years 2013 through 2017, and compared these totals to the number of cases we identified in their military justice and personnel databases. As shown in figure 11 below, we found that 65 percent of the Army’s reported nonjudicial punishments, 8 percent of the Navy’s reported nonjudicial punishments, and 82 percent of the Coast Guard’s reported nonjudicial punishments were recorded in their military justice databases. Officials from these services cited several reasons why they did not have complete information about all nonjudicial punishments. First, they said that the services are not required to track nonjudicial punishment cases because they are non-criminal punishments that are typically imposed for less serious offenses. Army and Navy officials noted that complete records of these punishments are not recorded at least in part because nonjudicial punishments are not meant to follow servicemembers throughout their career, but instead are intended to incentivize servicemembers to correct their behavior. Because nonjudicial punishments are not criminal punishments, the process afforded to servicemembers in nonjudicial punishment proceedings differs as well. For example, the servicemember is not guaranteed the right to representation by a military attorney. Army and Navy officials noted that their military justice databases contain records of nonjudicial punishments if there was legal involvement by the Judge Advocate General’s Corps in the case. Similarly, Coast Guard officials said that their military justice database contains records of nonjudicial punishment if a case originated as a criminal case involving a judge advocate, for example, if charges were preferred. According to Air Force and Marine Corps officials, the Air Force maintains complete nonjudicial punishment data in its military justice database, and the Marine Corps maintains complete nonjudicial punishment data in its personnel database. Standards for Internal Control in the Federal Government state that management should use quality information to achieve an entity’s objectives. Additionally, management should identify information requirements; ensure that the data it receives are timely and reliable; and process the data obtained into quality information. Officials from the Army, the Navy, and the Coast Guard expressed concerns regarding the feasibility of collecting and maintaining information about all nonjudicial punishments. Army officials stated that the collection and maintenance of all nonjudicial punishment data would be a substantial administrative burden due to the number of nonjudicial punishments awarded to servicemembers every week. Navy officials also stated that it would be a significant challenge to collect and maintain information about all nonjudicial punishments in either the Navy’s military justice database or its personnel database. They stated that there are few individuals who have access and can input data into the military justice database, and to expand the scope of criminal justice data collected in that manner, more people would have to be hired or assigned to assist with data entry. Similarly, Coast Guard officials said that tracking all nonjudicial punishment cases would be a difficult addition to their current data collection and maintenance workload. Coast Guard officials further stated that in addition to providing commanders with an essential means of providing good order and discipline, nonjudicial punishment also may promote positive change. Some Coast Guard officials stated concerns that recording all nonjudicial punishments in a database may inhibit the rehabilitative component of nonjudicial punishment. While the Army, Navy, and Coast Guard officials expressed these concerns, none of these military services had formally assessed the feasibility of collecting data on nonjudicial punishments. The absence of complete nonjudicial punishment data limits the military services’ visibility into the vast majority of legal punishments imposed on servicemembers under the UCMJ every year. Without such data, these three services will remain limited in their ability to assess or identify disparities among populations subject to this type of punishment. Among the servicemembers convicted in general and special courts- martials, we found no statistically significant differences regarding the likelihood of conviction among racial groups in the Army, the Navy, the Marine Corps, and the Air Force, while controlling for other attributes, as shown in figure 12 below. In the Marine Corps, male servicemembers were more likely to be convicted compared to female servicemembers. We found no statistically significant differences in the likelihood of convictions between males and females in the Army, the Air Force, and the Navy. In the military services that maintained complete punishment data—the Army, the Navy, the Marine Corps, and the Air Force—we found that minority servicemembers were either less likely to receive a more severe punishment in general and special courts-martial compared to White servicemembers, or there were no statistically significant differences in punishments among racial groups. Our findings regarding gender varied among the services. Male servicemembers were more likely to receive a more severe punishment compared to females in the Marine Corps, the Army, and the Air Force; for the Navy, we found there were no statistically significant differences in punishments between males and females. Navy and Marine Corps: Among servicemembers that were convicted in general and special courts-martial in the Marine Corps, we found no statistically significant differences regarding minority servicemembers being more likely or less likely to receive a dismissal or discharge punishment versus some other punishment, while controlling for other attributes, as shown in figure 13 below. In the Navy, among servicemembers that were convicted in general and special courts- martial, Black servicemembers were less likely than White servicemembers to receive a discharge or dismissal. We found no statistically significant differences regarding Hispanic servicemembers or those of Other races in the Navy. In the Marine Corps, among servicemembers that were convicted in general and special courts-martial, male servicemembers were more likely than female servicemembers to receive a discharge or dismissal. In the Navy, there were no statistically significant differences in punishments between males and females. Army and Air Force: We found no statistically significant differences regarding Black or Hispanic servicemembers being more likely or less likely to receive a more severe punishment in the Air Force or the Army, while controlling for other attributes, as shown in figure 14 below. We also found that servicemembers in the Other race group were less likely to receive a more severe punishment compared to White servicemembers in the Army, but punishment results for servicemembers in the Other race group in the Air Force were not statistically significant. Additionally, we found that male servicemembers were more likely to receive a more severe punishment compared to female servicemembers in the Army and the Air Force. We could not determine disparities in case outcomes—convictions and punishment severity—in the Coast Guard’s general and special courts- martial for fiscal years 2013 through 2017 because the Coast Guard did not collect and maintain complete conviction and punishment data in its military justice database. Specifically, 16 percent of all Coast Guard cases were missing conviction and punishment data. When broken down by court-martial type, 20 percent of general court-martial cases, 15 percent of special court-martial cases, and 4 percent of summary court- martial cases were missing conviction and punishment data. Coast Guard officials acknowledged that incomplete conviction and punishment data entry is a consistent problem. They said that data entry had improved recently. On December 17, 2018, the General Counsel of the Department of Defense issued the uniform standards and criteria required by article 140a of the Military Justice Act of 2016. As part of these uniform standards, the services were directed to collect information about the findings for each offense charged, and the sentence or punishment imposed. The military services are to implement the Secretary’s direction no later than December 23, 2020. DOD and the military services have conducted some assessments of disparities in the military justice system. We previously reported in 1995 on DOD studies on discrimination and equal opportunity, and found DOD and the services conducted seven reviews of racial disparities in discipline rates between 1974 and 1993. Since our 1995 report through 2016, DOD and service assessments of military justice disparities have been limited. Officials in the Office of Diversity, Equity and Inclusion (ODEI) noted DOD has not conducted any department-wide assessments of racial or gender disparities in military justice during this period. The military services’ diversity offices also were not able to identify any service-specific reviews of disparities in military justice. However, the military services have some initiatives to examine and address disparities in military justice. For example, Air Force officials said that in May 2016, the Air Force conducted a servicewide data call to solicit information about cases involving a challenge to a member of a court-martial based on race or a motion for selective prosecution. The officials said that a thorough review revealed no evidence of selective prosecution in Air Force courts-martial. In addition, the Air Force has conducted analyses of its own military justice data. Specifically, the Air Force routinely analyzes military justice data using a rates-per-thousand analysis to identify whether certain demographic groups are tried by court-martial or subject to nonjudicial punishments at higher rates than others. These Air Force analyses found that Black and male servicemembers were more likely than White and female servicemembers to be subject to courts-martial and nonjudicial punishments from fiscal years 2013 through 2017, which is consistent with what we found. However, the other services do not routinely conduct such analyses. Moreover, DOD has conducted climate surveys to address servicemembers’ perceptions of bias. In 2013, for example, DOD conducted service-wide equal opportunity surveys that queried servicemembers on whether they believed they received nonjudicial punishment or a court martial they should not have, and whether they believed their race or ethnicity was a factor. The survey responses showed that 1.3 percent of servicemembers indicated experiencing a perceived undue punishment, a result that was unchanged from the 2009 survey. Minority members were more likely to indicate experiencing perceived undue punishment than White members, but there were no significant differences between racial or ethnic groups who indicated experiencing undue punishment. ODEI officials told us that their office did not make any recommendations related to military justice as a result of these 2013 survey results because the findings were too small to warrant such steps. Moreover, ODEI officials said that while they have not completed their analysis of the 2017 survey data, the question about receiving nonjudicial punishment or court-martial had been removed from the 2017 survey. ODEI officials explained that the question was removed because the perception of unfair punishment was not the goal of the survey, although they said that the question could be reinstated for future surveys if the goals for the survey change. In June 2017, ODEI initiated a review of the military justice system following the publication of a report by a non-profit organization that found racial disparities in military justice actions. According to ODEI officials, their review assesses disparities in the military justice system using a similar analysis to that in the non-profit organization’s report, which analyzed rates of military justice actions per thousand servicemembers. ODEI officials told us they also observed racial and gender disparities among servicemembers involved in the military justice system in their own analysis of the service data. The officials said that the report on the results of their review will not directly address the issue of whether bias exists in the military justice process or the causes of any disparities, but will serve as a precursor to a future research study that looks more comprehensively into the issue of whether bias exists in the military justice system. ODEI officials said that their report should be issued in 2019. Standards for Internal Control in the Federal Government state that management uses quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks. The standards further provide that management should evaluate issues identified through monitoring activities and determine appropriate corrective actions. Officials from DOD and the military services acknowledged that they do not know the cause of the racial and gender disparities that have been identified in the military justice system. This is because they have not conducted a comprehensive evaluation to identify potential causes of these disparities and make recommendations about any appropriate corrective actions to remediate the cause(s) of the disparities. By conducting a comprehensive analysis into the causes of disparities in the military justice system, DOD and the military services would be better positioned to identify actions to address disparities, and thus help ensure that the military justice system is fair and just, a key principle of the UCMJ. The single overarching principle of the UCMJ is that a system of military law can foster a highly disciplined force if it is fair and just, and is recognized as such by both members of the armed forces and by the American public. DOD and the military services collect and maintain data on the race, ethnicity, and gender of all servicemembers. However, these data vary within and across the services, limiting the ability to collectively or comparatively assess military justice data to identify any disparities. DOD has recently taken steps to address this issue by directing the military services to, no later than December 23, 2020: collect uniform race and ethnicity data in their military justice databases, or aggregate any expanded ethnic or racial categories to the categories listed in the standards; collect either the social security number or DOD identification number in their military justice databases; and collect complete summary courts-martial information. It will be important for the military services to complete these actions to allow for efficient analysis and reporting of consistent military justice data. However, the newly issued standards apply only to the military justice databases and not to the investigations and personnel databases. The ability to query and report on the gender of servicemembers in its military justice database would provide the Coast Guard with more readily available data to identify or assess any gender disparities that may exist in the investigation and trial of military justice cases without merging data from multiple databases. Moreover, taking steps to develop the capability to present the race and ethnicity data from the military services’ personnel and investigations databases using the same categories included in the December 2018 standards for the military justice databases would enable DOD and the military services to more easily and efficiently assess the extent to which there are any racial or ethnic disparities throughout the military justice process. Further, DOD’s annual reports about the number and status of pending military justice cases do not include demographic information, such as breakdowns by race or gender, about servicemembers who experienced a military justice action. Reporting this information would provide servicemembers and the public with greater visibility into potential disparities and help build confidence that DOD is committed to a military justice system that is fair and just. Moreover, DOD does not have guidance that establishes criteria to determine when data indicating possible disparities among racial, ethnic, or gender groups in the investigations, trials, or outcomes of cases in the military justice system should be further reviewed, or describes the steps that should be taken to conduct such further review. By establishing such criteria, DOD and the services would be better positioned to monitor the military justice system to help ensure that it is fair and just, a key principle of the UCMJ. Our analysis of available data identified racial and gender disparities in all of the military services for servicemembers with recorded investigations, and for four of the military services for trials in special and general courts- martial, but these disparities generally were not present in the convictions or punishments of cases. These findings suggest disparities may be limited to particular stages of the military justice process for the period covered by our analysis. However, we were unable to determine whether there were disparities among servicemembers subject to nonjudicial punishments in the Army, the Navy, and the Coast Guard because these services do not collect complete nonjudicial punishment data, such as data on the servicemember’s race, ethnicity, gender, offense, and punishment for all nonjudicial punishments, in any of their databases. The absence of complete nonjudicial punishment data in the Army, the Navy, and the Coast Guard limits their visibility into the vast majority of legal punishments imposed on servicemembers under the UCMJ every year. Without such data, these three services will remain limited in their ability to assess or identify disparities among populations subject to this type of punishment. Finally, DOD recently conducted a study of racial and gender disparities in the military justice system, and expects to complete its report in 2019. However, this study will not assess the causes of the racial and gender disparities identified in the military justice system. Our findings of racial and gender disparities, taken alone, do not establish whether unlawful discrimination has occurred, as that is a legal determination that would involve other corroborating information along with supporting statistics. By conducting a comprehensive evaluation of the causes of these disparities, DOD and the military services would be better positioned to identify actions to address disparities, and thus help ensure that the military justice system is fair and just, a key principle of the UCMJ. We are making a total of 11 recommendations, including 3 to the Secretary of Homeland Security, 3 to the Secretary of Defense, 2 to the Secretary of the Army, 2 to the Secretary of the Navy, and 1 to the Secretary of the Air Force. The Secretary of Homeland Security should ensure that the Commandant of the Coast Guard modifies the Coast Guard’s military justice database so that it can query and report on gender information. (Recommendation 1) The Secretary of the Army should develop the capability to present servicemembers’ race and ethnicity data in its investigations and personnel databases using the same categories of race and ethnicity established in the December 2018 uniform standards for the military justice databases, either by (1) modifying the Army’s investigations and personnel databases to collect and maintain the data in accordance with the uniform standards, (2) developing the capability to aggregate the data into the race and ethnicity categories included in the uniform standards, or (3) implementing another method identified by the Army. (Recommendation 2) The Secretary of the Air Force should develop the capability to present servicemembers’ race and ethnicity data in its investigations and personnel databases using the same categories of race and ethnicity established in the December 2018 uniform standards for the military justice databases, either by (1) modifying the Air Force’s investigations and personnel databases to collect and maintain the data in accordance with the uniform standards, (2) developing the capability to aggregate the data into the race and ethnicity categories included in the uniform standards, or (3) implementing another method identified by the Air Force. (Recommendation 3) The Secretary of the Navy should develop the capability to present servicemembers’ race and ethnicity data in its investigations and personnel databases using the same categories of race and ethnicity established in the December 2018 uniform standards for the military justice databases, either by (1) modifying the Navy’s investigations and personnel databases to collect and maintain the data in accordance with the uniform standards, (2) developing the capability to aggregate the data into the race and ethnicity categories included in the uniform standards, or (3) implementing another method identified by the Navy. (Recommendation 4) The Secretary of Homeland Security should ensure that the Commandant of the Coast Guard develops the capability to present servicemembers’ race and ethnicity data in its investigations and personnel databases using the same categories of race and ethnicity established in the December 2018 uniform standards for the military justice databases, either by (1) modifying the Coast Guard’s investigations and personnel databases to collect and maintain the data in accordance with the uniform standards, (2) developing the capability to aggregate the data into the race and ethnicity categories included in the uniform standards, or (3) implementing another method identified by the Coast Guard. (Recommendation 5) The Secretary of Defense should ensure that the Joint Service Committee on Military Justice, in its annual review of the UCMJ, considers an amendment to the UCMJ’s annual military justice reporting requirements to require the military services to include demographic information, including race, ethnicity, and gender, for all types of courts-martial. (Recommendation 6) The Secretary of Defense, in collaboration with the Secretaries of the military services and the Secretary of Homeland Security, should issue guidance that establishes criteria to specify when data indicating possible racial, ethnic, or gender disparities in the military justice process should be further reviewed, and that describes the steps that should be taken to conduct such a review. (Recommendation 7) The Secretary of the Army should consider the feasibility, to include the benefits and drawbacks, of collecting and maintaining complete information for all nonjudicial punishment cases in one of the Army’s databases, such as information on the servicemembers’ race, ethnicity, gender, offense, and punishment imposed. (Recommendation 8) The Secretary of the Navy should consider the feasibility, to include the benefits and drawbacks, of collecting and maintaining complete information for all nonjudicial punishment cases in one of the Navy’s databases, such as information on the servicemembers’ race, ethnicity, gender, offense, and punishment imposed. (Recommendation 9) The Secretary of Homeland Security should ensure that the Commandant of the Coast Guard considers the feasibility, to include the benefits and drawbacks, of collecting and maintaining complete information for all nonjudicial punishment cases in one of the Coast Guard’s databases, such as information on the servicemembers’ race, ethnicity, gender, offense, and punishment imposed. (Recommendation 10) The Secretary of Defense, in collaboration with the Secretaries of the military services and the Secretary of Homeland Security, should conduct an evaluation to identify the causes of any disparities in the military justice system, and take steps to address the causes of these disparities as appropriate. (Recommendation 11) We provided a draft of this report to DOD and the Department of Homeland Security for review and comment. Written comments from DOD and the Department of Homeland Security are reprinted in their entirety in appendixes X and XI, respectively. DOD and the Department of Homeland Security provided additional technical comments, which we incorporated in the report, as appropriate. In written comments, DOD concurred with six recommendations, and partially concurred with two recommendations that were directed to the Secretary of Defense. The Department of Homeland Security concurred with the three recommendations directed to the Secretary of Homeland Security. DOD concurred with our six recommendations to present servicemembers’ race and ethnicity data in each of the military services’ respective investigations and personnel databases using the same categories of race and ethnicity established for their military justice databases; consider an amendment to the UCMJ’s annual military justice reporting requirements to require the military services to include demographic information for all types of courts-martial; and consider the feasibility of collecting and maintaining complete information for all nonjudicial punishment cases. DOD partially concurred with two of our recommendations, agreeing with the content, but requesting that we modify the recommendations to direct them to more appropriate entities. Specifically, DOD concurred with our recommendations that guidance should be issued to establish criteria specifying when data indicating possible racial, ethnic, or gender disparities require further review and the steps that will be taken to conduct the review; and to conduct an evaluation to identify the causes of any racial or gender disparities in the military justice system and, if necessary, take remedial steps to address the causes of these disparities. For both recommendations, DOD suggested that the Secretary of Homeland Security be added, and that we remove the DOD Office for Diversity, Equity and Inclusion and the Commandant of the Coast Guard, as they fall under the Secretary of Defense and the Secretary of Homeland Security, respectively. We agree with DOD’s suggestions, and we have modified both recommendations accordingly. In an email correspondence, the Department of Homeland Security and the Coast Guard concurred with the updates. In its written comments, the Department of Homeland Security concurred with our three recommendations to modify the Coast Guard’s military justice database so that it can query and report on gender information, to present servicemembers’ race and ethnicity data in its investigations and personnel databases using the same categories of race and ethnicity established for the military justice database, and to consider the feasibility of collecting and maintaining complete information for all nonjudicial punishment cases. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Defense, and the Acting Secretary of Homeland Security. In addition, this report will also be available at no charge on the GAO website at http://www.gao.gov. If you or your members of your staff have any questions regarding this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in Appendix XII. The objectives of this report were to assess the extent to which (1) the military services collect and maintain information about the race, ethnicity, and gender of servicemembers investigated and disciplined for violations of the Uniform Code of Military Justice (UCMJ) that can be used to assess disparities; and (2) there are racial and gender disparities in investigations, disciplinary actions, and case outcomes in the military justice system, and whether the Department of Defense (DOD) and the military services have taken steps to study any identified disparities. To address both of our objectives, we analyzed data collection, data maintenance, and military justice disciplinary actions involving active-duty servicemembers in the Army, the Navy, the Marine Corps, the Air Force, and the Coast Guard. Although the Coast Guard is part of the Department of Homeland Security, the Coast Guard is a military service and a branch of the armed forces at all times. We analyzed military justice actions initiated and recorded in service investigations and military justice databases between fiscal years 2013 through 2017. We chose this time period because it provided the most recent history of available military justice data at the time of our review. We requested record-level data from each of the military services’ personnel, investigations, and military justice databases, which resulted in a total of 15 data requests. Table 6 below provides an overview of the databases included in our review, broken out by database type. We sent individual data requests that were tailored based on our conversations with service officials and our own analysis of the availability of data. In addition to requesting the race, ethnicity, and gender of servicemembers subject to military justice actions, we also requested other demographic and administrative attribute data—such as rank, age, years of service, duty station, and occupation—from the services’ personnel databases to include in our statistical models. We identified these attributes by reviewing relevant literature and interviewing agency officials. Personnel databases. We requested and received monthly snapshots with record-level data on all active-duty servicemembers in each of the military services from fiscal years 2013 through 2017. Specifically, we requested demographic and administrative data, including race, ethnicity, gender, rank, education, age or date of birth, years of service, occupation, location or duty station, deployed status, administrative or disciplinary actions and dates, character of service separation, and servicemembers’ unique identifiers (social security number and employee identification number). Investigations databases. We requested and received record-level data on all investigations recorded in a military service military criminal investigative organization (MCIO) database that were initiated from fiscal years 2013 through 2017, where the subject of the investigation was an active-duty servicemember. For each case, we requested certain attribute data on the investigation subject, including race, ethnicity, gender, rank, age or date of birth, service and component, offense(s) investigated, case initiation date, investigation source, investigating entity, investigation outcome and date, incident location, and the subject’s unique identifier, such as social security number or employee identification number. In some services not all of these attributes were available or requested. For example, since the Air Force database only included investigations conducted by the Air Force Office of Special Investigations, we did not request information about the investigating entity. In addition, the Navy Criminal Investigative Service provided us with data about and we analyzed closed cases only, whereas the Army and the Air Force MCIOs provided us with data about and we analyzed all cases in their database during the period of our review. Military justice databases. We requested and received record-level data on all cases where a servicemember was subject to disciplinary proceedings under the Uniform Code of Military Justice (UCMJ) from fiscal years 2013 through 2017. For each case where charges were preferred against a servicemember during this period, we requested demographic and administrative data on the servicemember as well as key information related to their case, including race, ethnicity, gender, rank, age or date of birth, component, case type and forum, offense(s) charged, case disposition and date, appeals status, case outcome or sentence, disciplinary action taken, date charges were first preferred, and the servicemember’s unique identifier, such as social security number or employee identification number. We received general and special courts- martial data from all of the services from their military justice databases. For the Army, in addition to data from their military justice database, Military Justice Online, we also received courts-martial data from a separate database, called the Army Court-Martial Information System (ACMIS), which is used by the service’s trial judiciary to track courts- martial. For summary courts-martial and nonjudicial punishments, the services varied in the extent that and the location where they collected and maintained complete data for these two military justice actions, as is discussed further earlier in this report. In the Air Force, summary courts-martial and nonjudicial punishment data is maintained in the service’s military justice database, the Automated Military Justice Analysis and Management System. The Marine Corps did not collect and maintain complete data about summary courts-martial or nonjudicial punishments in its military justice database, however, its personnel database included information about all summary courts-martial and nonjudicial punishments imposed on servicemembers during the period of our review. The Army and the Navy did not collect and maintain complete data about summary courts-martial or nonjudicial punishments in their military justice databases, or other databases. In these services, summary courts-martial and nonjudicial punishments were recorded in their military justice databases if these actions had involvement by the services’ legal offices. Further, summary courts-martial and nonjudicial punishments were recorded in the personnel databases used by these services only if these actions resulted in an administrative action against the accused, such as a forfeiture of pay or reduction in grade. The Coast Guard did not collect and maintain complete data about nonjudicial punishments in its military justice database or other databases; nonjudicial punishments were recorded in its military justice database if a legal office was involved in the action. Further, nonjudicial punishments were recorded in the Coast Guard’s personnel database if they resulted in an administrative action against the accused, such as a forfeiture of pay or reduction in grade. To evaluate the extent to which the military services collect and maintain race, ethnicity, and gender data about servicemembers investigated and disciplined for violations of the UCMJ, we first reviewed service guidance, user manuals, and other documents related to the services’ investigations, military justice, and personnel databases. We reviewed these documents to determine: the types of data officials are required to collect and maintain; and the internal procedures the services follow in inputting information about race, ethnicity, and gender data into each type of database. For example, we determined whether collection of this information was mandatory, and how this information was entered into and recorded in each database. Specifically, we determined whether information about race, ethnicity, and gender was entered into each database manually, using a drop-down menu, or was auto-populated from another database. Further, we identified the number of possible response options that each database contained for each of these demographic fields. Second, we interviewed service officials who manage and use the military justice, investigations, and personnel databases to discuss: which fields in each database track the race, ethnicity, and gender of how these data are input and their insights regarding the reliability of these data. Specifically, we interviewed officials from the legal branches of the military services, including the Army Office of the Judge Advocate General, the Navy Judge Advocate General’s Corps, the Marine Corps’ Judge Advocate Division, the Air Force Judge Advocate General’s Corps, and the Coast Guard Office of the Judge Advocate General. In addition, we spoke with officials in the military criminal investigative organizations (MCIO), including the Army Criminal Investigation Command, the Naval Criminal Investigative Service, the Air Force Office of Special Investigations, and the Coast Guard Investigative Service. We also interviewed officials from the manpower and personnel offices of the services with responsibility for the services’ personnel databases, including the Army’s Human Resources Command and the Office of the Deputy Chief of Staff; the Navy’s Personnel Command; the Marine Corps Manpower and Reserve Affairs Manpower Information Systems Branch; the Air Force Personnel Center; and the Coast Guard’s Personnel Service Center. Finally, we analyzed the data we received from the investigations, military justice, and personnel databases to determine the completeness of the race, ethnicity, and gender information that was recorded in each of the databases. We assessed the military services’ systems and procedures for collecting data against DOD and service guidance and relevant federal internal control standards. To evaluate the extent to which there are racial, ethnic, and gender disparities in investigations, disciplinary actions, and case outcomes, we analyzed data from the military services’ investigations, military justice, and personnel databases to determine summary statistics and we then conducted bivariate and multivariate regression analyses. Investigations. We focused on alleged violations of the UCMJ that were recorded in databases used by service-specific MCIOs. Investigations are recorded in the MCIO databases when a servicemember is the subject of a criminal allegation made by another person; for purposes of this report, we say the servicemember had a “recorded investigation” to describe these cases. We analyzed investigation information from the databases used by each of the military services’ MCIOs. Specifically, we analyzed data from the Army’s Criminal Investigation Command, which included cases investigated by military police and Criminal Investigation Command; the Navy and Marine Corps’ Naval Criminal Investigative Service, which included cases investigated by the Naval Criminal Investigative Service and military police; the Air Force’s Office of Special Investigations, which included only Office of Special Investigations cases; and the Coast Guard Investigative Service, which included only Coast Guard Investigative Service cases. Our analysis of recorded investigations data did not include investigations conducted by a servicemember’s command, because those investigations are not recorded in the MCIO databases. Military Justice Discipline. We included in our definition of servicemembers disciplined for a violation of the UCMJ those servicemembers with cases that resulted in a trial in any type of court- martial (general, special, and summary), or servicemembers who were subject to a nonjudicial punishment from fiscal years 2013 through 2017. We analyzed data for trials in general and special courts-martial separately from trials in summary courts-martial because general and special courts-martial result in a criminal conviction if the servicemember is found guilty, while summary courts-martial are not a criminal forum and do not result in a criminal conviction. We analyzed general and special courts-martial cases together due to the small number of cases for some racial or gender groups. In addition, we also separated general and special courts-martial into cases that either were or were not preceded by an investigation recorded in an MCIO database. Our analysis of general and special courts-martial cases without a recorded investigation included those general and special courts-martial that were investigated by a servicemember’s command or other law enforcement entities. We used the preferral date, or the date when an accused servicemember was first charged with a violation, to count the number of courts-martial that occurred in a given fiscal year. However, each military service uses the date in which the court-martial judgment was given when reporting the number of each type of court-martial in their annual reports to the Court of Appeals for the Armed Forces. As a result, the number of court-martial cases in a given year analyzed for our review differs from what was reported in the annual reports. In discussions with officials after we had completed our preliminary analyses, they recommended that we use the referral date instead of the preferral date, so that our total number of cases would be more consistent with the number of cases that they reported. However, changing the date for grouping cases would have required us to request new military justice data from each of the military services, and conduct additional work. Above all, using the preferral date would not impact the findings of racial and gender disparities. In addition, our analyses only counted cases that were ultimately tried at general, special, or summary courts-martial, and excluded those cases where charges were dismissed, withdrawn, or subject to some alternate resolution. For nonjudicial punishments, we used the date that the punishment was imposed. To prepare the data for our analyses and ensure that we had consistent profiles for the race, ethnicity, and gender of the servicemembers, we merged records from the military services’ investigations, military justice, and personnel databases. We merged records using servicemembers’ unique identifiers, such as social security number or employee identification number, that were common among a particular service’s databases. In some instances—a small proportion of cases—we could not match personnel records with military justice records because the social security number or employee identification number in the military justice database did not match the information in the personnel database. In other instances, we could not match personnel records with military justice records because the military justice records did not contain a social security number or employee identification number to match with information found in their personnel record. We first tried to match these cases using the servicemembers’ name and date of birth; however, in some cases we were unable to match personnel records with investigations or military justice cases. As a result, we compiled lists of those cases we were unable to match, and we provided the services with lists of these cases. Service officials manually looked up this data and provided us with the missing social security numbers or employee identification numbers for these cases so that we could complete our analyses. These manual look up efforts increased our match rates so that we had a data set that we determined was sufficiently complete to perform our analyses. For servicemembers who were the subjects of military justice actions, we used the attribute data that was available in the personnel database at the time an investigation or disciplinary action was initiated (the preferral date for courts-martial). For our total service populations, which included servicemembers who were not the subject of a military justice action, we used their attribute data from the “median” snapshot of the five fiscal years of personnel data we received. Based on discussions with service officials, we treated the personnel databases as the authoritative sources for servicemembers’ demographic and administrative data. For some services when needed, if we identified a discrepancy in the race or gender value for a servicemember between the data in the personnel and military justice databases, we used the value recorded in the personnel database because service officials had told us that the personnel databases were the official sources for demographic data such as race and gender, and would be more likely to contain more reliable data for these fields than the investigations or military justice databases. For some services where there were cases where an attribute value was missing in the personnel database, we used the military justice or investigative database as a secondary source for this information. In merging the records from the personnel, military justice, and investigations databases, we created a single data file for each service that contained attribute data for all active-duty servicemembers, as well as complete information on the investigation and discipline of servicemembers who were the subject of a military justice action from fiscal years 2013 through 2017. In using this methodology to merge the records, the total number of servicemembers we use in our report when discussing the total service populations for each service is greater than the total active-duty force end strength of that service in any given fiscal year. This is because our total service populations represent the number of unique individuals who served on active duty from fiscal years 2013 through 2017. In addition, as part of our data preparation, we consolidated the various race and ethnicity values in the service personnel databases to the five groups for race and the two groups for ethnicity established by Office of Management and Budget (OMB) standards for maintaining, collecting, and presenting data on race and ethnicity for all federal reporting purposes. The five race groups in the standards are American Indian or Alaska Native; Asian; Black or African American; Native Hawaiian or Other Pacific Islander; and White. The two ethnic groups are Hispanic or Latino and Not Hispanic or Latino. First, we collapsed race and ethnicity data into a single combined field. Specifically, we grouped individuals of Hispanic ethnicity together, regardless of their racial identification, so that we could compare those of Hispanic ethnicity to other racial groups. We did this in part because of the ways in which some of the services record these data in their databases. For example, the Navy’s and the Marine Corps’ military justice databases do not have separate fields for race and ethnicity; instead, the values are tracked in a single field. Throughout the discussion for objective 2 of this report, we refer to the combined race and ethnicity values as race. We then consolidated races to the five racial groups in the OMB standards. When military service personnel databases included different or additional possible options for race and ethnicity than the groups established by the OMB standards, we consolidated the options in accordance with the definitions for each race and ethnicity listed in the OMB standards. Given the small number of cases in some racial groups, we collapsed certain racial groups into an “Other” group in order to report statistically reliable results. The “Other” group includes individuals who identified as Asian, Native Hawaiian/Other Pacific Islander, American Indian/Alaska Native, and multiple races. Summary statistics. We analyzed data from the military services’ investigations, military justice, and personnel databases to determine the extent to which racial and gender groups were the subjects of recorded investigations, tried in courts-martial, and subject to nonjudicial punishments (for Army and Marine Corps, services for which we had complete data) at higher rates or lower rates than each racial and gender group’s proportion of the overall service populations. Other than our analysis of recorded investigations, we did not analyze Coast Guard cases due to the small number of general and special courts-martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. To conduct this analysis, we used data on all active-duty servicemembers to identify what proportion each racial group (White, Black, Hispanic, and Other) and gender group (male, female) made up of the overall service population from fiscal years 2013 through 2017. We then used data from the services’ military justice or personnel databases to calculate the representation of each racial and gender group as a percent of the population subjected to each type of military justice action. We also examined the rates at which certain racial and gender groups were charged with drug offenses (Article 112a) and sexual assault offenses (Article 120) compared to their proportions of the overall service populations. See Appendix III for information regarding recorded investigations and general and special courts-martial of drug and sexual assault offenses. We analyzed these two specific UCMJ offenses because officials from some services told us that an investigation into these offenses may frequently be mandatory, and thus could potentially mitigate the risk of bias. To conduct this analysis, we used offense data from the services’ military justice databases to determine each racial and gender group’s representation in the population that was the subject of a military justice action for a drug, sexual assault, or other offense type. Bivariate and Multivariate Regression Analyses. We developed a logistic regression model using the data we received from the services’ investigations and military justice databases to determine the extent that certain attributes were associated with higher rates of investigation or discipline of servicemembers. We conducted bivariate logit analyses to estimate the association between select attribute factors (or independent variables) and the outcome variables (the dependent variable) in a binary format, except for the two offense outcome variables. Table 7 below lists all of the dependent and independent variables we used in our analyses. To conduct our statistical analyses, we created groups for each demographic and administrative attribute (independent variable) that we tested in our regression model. We created these groups based on input and guidance from service officials. While the modeling subgroups we created are largely consistent across services, some values are different for certain services. Table 8 summarizes the modeling groups we constructed for each service for each attribute included in our regression analyses. When analyzing the severity of punishments, we developed two groups for the Navy and the Marine Corps, and three groups for the Air Force and the Army, as shown in table 9 below. We did not create a third punishment group for confinement without dismissal or discharge for the Navy and the Marine Corps because of the small number of cases with confinement that did not also include some sort of discharge. Based on discussions with service officials, we determined that a sentence resulting in a dismissal or discharge was the most severe punishment outcome. Typically, a logistic regression model is appropriate when the model outcome is a binary (yes/no) response. Because the punishment groups for the Army and the Air Force were not binary, they could not be analyzed using a multivariate logistic regression. Instead, we used an ordered logit model, also called an ordered logistic regression model, to analyze punishment severity in the Army and the Air Force. An ordered logistic regression is an extension of the logistic regression model that applies to dependent variables where there are more than two response categories. This model allowed us to examine the degree to which a racial or gender group was more likely or less likely than another group to receive a more severe punishment in general and special courts-martial, while controlling for other attributes, such as gender, education, rank, composition of panel, and offense type. To conduct this analysis, we reviewed outcome data from the services’ personnel, investigations, and military justice databases. Based on our bivariate analyses, we determined which variables were significantly associated with military justice actions, and that appeared to be statistically significant predictors of an individual’s likelihood to be subject to a military justice action. Appendix IX includes a summary of those indicators for each of the services. We also examined correlation matrices of the independent variables to determine where there were high correlations between two variables. Where variables were highly correlated, we chose one variable over the others or created a hybrid variable combining those two variables. Specifically, we excluded age and years of service for most of the military services, due to high correlation with the rank variable. Based on our discussions with service officials, they indicated that rank would be the preferred variable to include in our analyses if selecting only one variable among rank, age, and years of service. However, for the Air Force, based on discussion with Air Force officials, we did control for years of service among the lower enlisted ranks (E1-E4). In addition, we could not include education for the Army due to variability and overlapping values in the data. Further, we chose not to model attributes such as occupation and location due to the great variability in these data and the difficulty in creating groups and reaching agreement about those groups with service officials. Based on these results, we then conducted a series of multivariate logistic regression models. Multivariate logistic regression modeling is a statistical method that examines several variables simultaneously to estimate whether each of these variables are more likely or less likely to be associated with a certain outcome. A multivariate regression analysis analyzes the potential influence of each individual factor on the likelihood of a binary outcome (e.g., a specific military justice action) while simultaneously accounting for the potential influence of the other factors. This type of modeling allowed us to test the association between servicemember characteristics, such as race or gender, and the odds of a military justice action (shown as the outcome variables in table 7 above), while holding other servicemember attributes constant (such as gender, rank, and education, shown as the independent variables in table 7 above). We conducted a separate regression for each of the military justice actions listed as an outcome variable. We selected this type of model because it could account for the attributes simultaneously. For the purposes of consistency, in our multivariate regression analyses, we made all racial comparisons with White servicemembers as the reference category. Similarly, we made all gender comparisons with female servicemembers as the reference category. A logistic regression model provides an estimated odds ratio, where a value greater than one indicates a higher or positive association; in this case, between the race, ethnicity, or gender of a servicemember (the independent variables) and the likelihood of being the subject of a military justice action (the dependent, or outcome, variable). An estimated odds ratio less than one indicates lower odds or likelihood of being the subject of a military justice action when a factor—here, a specific demographic or administrative attribute—is present. The statistical significance of the logistic regression model results is determined by a p-value of less than 0.05. As a result, in our report we state that odds ratios that are statistically significant and greater than 1.00 or lower than 1.00 indicate that individuals with that characteristic are more likely or less likely, respectively, to be the subject of a particular outcome or military justice action. In cases where the p-value was greater than 0.05, we report that we could not identify any statistically significant differences, which means that we could not conclude that there was an association between race or gender and the likelihood of a military justice action. We report the results from our regression models as odds ratios. We generally report multivariate results from testing associations between key attributes—including race, ethnicity, gender, rank, and education—on a servicemember’s likelihood of being investigated and disciplined for a UCMJ violation. In the body of this report, we focused on race and gender disparities among servicemembers investigated and disciplined for violations of the UCMJ, while holding other factors constant; however, our analyses of recorded investigations and general and special courts- martial for drug and sexual assault offenses are discussed in Appendix III. In all of these analyses for the Air Force, we also controlled for years of service among the lower enlisted ranks (E1-E4). In the analyses we conducted for the Army, we could not control for education, but we were able to control for age. All regression models are subject to limitations. For our analyses, the limitations included: Results of our analyses are associational and do not imply a causal relationship. We did not identify the causes of any racial or gender disparities, and the results of our work alone should not be used to make conclusions about the military justice process. Our analyses of these data in finding the presence or absence of racial or gender disparities, taken alone, do not establish the presence or absence of unlawful discrimination, as that is a legal determination that would involve other corroborating information along with supporting statistics. We could not assess some attributes that potentially could be related to a servicemember’s likelihood of facing a military justice action in the data analyzed for this review. For example, a servicemember’s socioeconomic background or receipt of a waiver upon entering the service could potentially be related to the likelihood of being investigated, tried in a court-martial, or subject to a nonjudicial punishment. However, we were unable to test these associations because most services indicated they did not have information about socioeconomic status or waivers in the databases that we requested data from. Furthermore, while some other attributes may have been available—such as marital status of the subject or the number of dependent children—we did not include these attributes in our data requests because we prioritized analyzing other demographic factors based on our background research and conversations with service officials. As outlined above, we incorporated input from service officials to the extent possible as we prepared our modeling groups for the demographic and administrative attributes we tested, such as rank, education, and years in service. However, this process was necessarily imprecise. Our modeling results may have been impacted by our discretionary decisions to include certain values in the groups we created for these variables. Data reliability. We conducted data reliability assessments on the datasets we received from the databases in our review. We examined the documentation officials provided to us on each database and conducted electronic tests on the data we received to check for completeness and accuracy. We also sent data reliability questionnaires to database managers about how the data are collected and their appropriate uses, and had discussions with database managers to discuss the reliability of the data in their databases. When we determined that particular fields were not sufficiently reliable, we excluded them from our analysis. For example, we did not use data in our analysis where a substantial number of values were missing. We also checked to see that the values for variables were internally consistent and that results were not affected unduly by outlier values that might suggest miscoded values. For the purposes of our analysis, we found the variables we ultimately reported on to be sufficiently reliable. Furthermore, due to the sensitivity of the information analyzed in this report, we did not include information in instances where the number of servicemembers subjected to a particular military justice action was fewer than 20, to protect privacy. Literature review. To assess the extent to which disparities in the military justice system and the civilian justice system had been previously assessed, we conducted a literature review. To identify relevant publications about disparities in the military justice system and the civilian justice system, we performed a literature search of a number of bibliographic databases, including ProQuest Academic, ProQuest Dialog, Scopus, EBSCO, and HeinOnline. We also searched two think tank search engines: Policy File and the Think Tank Search (from the Harvard Kennedy School). We received the following types of publications: scholarly/peer reviewed material, dissertations, and association/think tank/nonprofit publications. To identify publications by DOD and the services related to the military justice system, we reviewed prior GAO reports and asked officials at the DOD Office of Diversity, Equity and Inclusion, and in the services’ respective diversity and inclusion offices to identify relevant publications. We concluded our searches in October 2018. We also asked the service Judge Advocate General offices for publications relevant to disparities in military justice. We also identified publications in our own background information search. We reviewed those publications that assessed racial, ethnic, or gender disparities among servicemembers in the military justice system. While the civilian and military justice systems differ from each other, we selected a few nationwide studies examining disparities in the civilian justice system to summarize in the background section of our report, in order to enhance our understanding of the complexities of the issues, including how others have attempted to measure disparities. We did not assess the methodologies used in any of these studies or the reliability of the data cited in the studies; the studies related to the civilian justice system are discussed in our report to provide broader context for the discussion about racial and gender disparities in the military justice system. We conducted this performance audit from November 2017 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform an audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. As shown in figure 15 below, our analysis of data contained in the military services’ military criminal investigations databases found that Black servicemembers were subjects of recorded investigations at a higher rate compared to their proportion of the overall service population in all of the military services. Hispanic servicemembers were the subjects of recorded investigations at a higher rate compared to their proportion of the overall service population in the Navy and the Air Force, at a lower rate in the Marine Corps, and at the same rate in the Army. Additionally, we found that males were the subjects of recorded investigations at higher rates than their share of the general service population in all of the military services. In addition, figure 15 above also shows the results of our bivariate analyses, which calculated the degree to which one racial or gender group was more likely or less likely than another racial or gender group to be the subject of recorded investigations. Our bivariate analyses found that Black and male servicemembers in all of the military services were statistically significantly more likely to be the subjects of recorded investigations for alleged UCMJ violations than servicemembers of all other races or females. Hispanic servicemembers were statistically significantly more likely in the Navy, the Air Force, and the Coast Guard, and were statistically significantly less likely in the Army to be the subjects of recorded investigations than servicemembers of all other races. Servicemembers in the Other race category were statistically significantly less likely than servicemembers of all other races to be the subjects of recorded investigations in the Army and the Marine Corps. Our bivariate analyses did not show any statistically significant differences for servicemembers in the Other race category in the Navy, the Air Force, or the Coast Guard, or Hispanic servicemembers in the Marine Corps. As shown in figure 16 below, Black, Hispanic, and male servicemembers in all of the military services included in this analysis were represented at a higher rate than their proportions of the overall service population. White and female servicemembers in all of the military services were represented at a lower rate than their proportions of the overall service population. Servicemembers in the Other race category were represented at a higher rate in the Navy, at a lower rate in the Army and the Air Force, and at the same rate in the Marine Corps compared to their proportion of the overall service population. We could not analyze Coast Guard cases due to the small number of general and special courts-martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. The bivariate regression analysis results in figure 16 above calculate the degree to which one racial or gender group was more likely or less likely than servicemembers of all other races and genders to be tried in general and special courts-martial. We found that Black and male servicemembers in all of the military services were more likely to be tried in general and special courts-martial than servicemembers of all other races or females. Our bivariate analyses found that Hispanic servicemembers in the Army were more likely to be tried in general and special courts-martial than servicemembers of all other races. We found no statistically significant differences in the likelihood of Hispanic servicemembers to be tried in general and special courts-martial compared to servicemembers of all other races in the Navy, the Marine Corps, and the Air Force. White and female servicemembers in all of the military services were less likely to be tried in general and special courts- martial than servicemembers of other races or males. Furthermore, servicemembers in the Other race category were more likely in the Navy and less likely in the Army to be tried in general and special courts-martial than servicemembers of other races. We found no statistically significant differences in the likelihood of servicemembers in the Other race category to be tried in general and special courts-martial in the Marine Corps and the Air Force compared to servicemembers of other races. As shown in figure 17 below, for trials in general and special courts- martial that followed a recorded investigation, Black servicemembers were represented at a lower rate in the Army, the Navy, and the Marine Corps, and at the same rate in the Air Force compared to their proportions of the service population that had recorded investigations. Hispanic servicemembers in trials of general and special courts-martial following a recorded investigation were represented at a higher rate than their proportion of the overall service population that had recorded investigations in the Army and the Marine Corps, and at the same rate in the Navy and the Air Force. White servicemembers were represented at a lower rate in the Army, the Navy, and the Marine Corps, and at the same rate in the Air Force compared to their proportions of the service population with recorded investigations. Servicemembers in the Other race category were represented at a higher rate in the Army, the Navy, and the Marine Corps, and at the same rate in the Air Force compared to their proportions of the overall service population with recorded investigations. We could not analyze Coast Guard cases due to the small number of general and special courts-martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. Male servicemembers with trials in general and special courts-martial that followed a recorded investigation were represented at a higher rate in all of the military services compared to their proportions of the service population that had recorded investigations. Females were represented at a lower rate in all of the military services compared to their proportions of the service population that had recorded investigations. As shown in figure 17 above, our bivariate regression analyses showed that, in the Army, White servicemembers were statistically significantly less likely to be tried in general and special courts-martial following a recorded investigation than servicemembers of all other races, whereas Hispanic servicemembers were statistically significantly more likely to be tried following a recorded investigation. In the Navy, servicemembers in the Other race category were statistically significantly more likely to be tried in general and special courts-martial following a recorded investigation than servicemembers of all other races. Males were more likely, and females were less likely, to be tried in general and special courts-martial following a recorded investigation in the Army and the Air Force. The remaining odds ratios shown in figure 17 above were not statistically significant. We identified racial and gender disparities in the rate and likelihood of trial in general and special courts-martial in cases without a recorded investigation in all of the military services. Specifically, as shown in figure 18 below, for trials in general and special courts-martial without a recorded investigation, Black and male servicemembers in all of the military services were represented at a higher rate than their proportion of the service population that did not have a recorded investigation. Hispanic servicemembers were represented at a higher rate in the Army and the Marine Corps, and at the same rate in the Navy and the Air Force compared to their proportions of the service population that did not have a recorded investigation. Servicemembers in the Other race category were represented at a lower rate in the Marine Corps and the Air Force, and at the same rate in the Army and the Navy compared to their proportion of the overall service population that did not have a recorded investigation. White and female servicemembers in all of the military services were represented at a lower rate than their proportions of the overall service population without a recorded investigation. We could not analyze Coast Guard cases due to the small number of general and special courts- martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. The bivariate regression analysis results in figure 18 above calculate the degree to which one racial or gender group was more likely or less likely than servicemembers of all other races and genders to be tried in general and special courts-martial without a recorded investigation. We found that Black and male servicemembers in all of the military services were more likely to be tried at special and general courts-martial that were not preceded by a recorded investigation than servicemembers of all other races or females. White and female servicemembers in all of the military services were less likely to be tried at special and general courts-martial that were not preceded by a recorded investigation than servicemembers of all other races and males. We found no statistically significant differences in the likelihood of Hispanic servicemembers or servicemembers in the Other race category in any of the military services being tried in general and special courts-martial without a recorded investigation compared to servicemembers of all other races. We identified racial and gender disparities in the rate and likelihood of trial in summary courts-martial in the Air Force and the Marine Corps. Specifically, as shown in figure 19 below, Black and male servicemembers were tried in summary courts-martial for UCMJ violations at higher rates than their share of the overall service population in the Air Force and the Marine Corps. White and Hispanic servicemembers were tried in summary courts-martial at lower rates than their share of the overall service population in both services. Servicemembers that were included in the Other race category were tried at higher rates in the Air Force, and at lower rates in the Marine Corps. We could not determine whether there were any racial or gender disparities for summary courts-martial in the Army and the Navy because these services did not collect complete summary court-martial data— information about all summary court-martial cases, to include demographic information about the subject—in their investigative, military justice, or personnel databases, as discussed above in the report. We could not analyze Coast Guard cases due to the small number of summary courts-martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. The bivariate regression analysis results in figure 19 above calculate the degree to which one racial or gender group was more likely or less likely than servicemembers of all other races and genders to be tried in summary courts-martial. We found that Black servicemembers in the Marine Corps and the Air Force were more likely to be tried in summary courts-martial than servicemembers of all other races. We also found that male servicemembers were more likely than their female counterparts to be tried in summary courts-martial in the Marine Corps and the Air Force. We observed no statistically significant differences in summary court- martial rates for servicemembers in the Other race category in either the Marine Corps or the Air Force, or for Hispanic servicemembers in the Marine Corps. As shown in figure 20 below, we found that Black and male servicemembers were subject to nonjudicial punishment for UCMJ violations at a higher rate than their share of the overall service population in the Marine Corps and the Air Force. White servicemembers were subject to nonjudicial punishments at lower rates than their share of the overall service population in both services, and Hispanic servicemembers were subject to nonjudicial punishments in a proportion equal to their share of the general service population in both services. Servicemembers that were included in the Other race category were subject to nonjudicial punishment at lower rates than their share of the overall service population in the Marine Corps and the Air Force. We could not analyze nonjudicial punishments in the Army, the Navy, and the Coast Guard because these services do not collect complete nonjudicial punishment information. The bivariate regression analyses in figure 20 above calculate the degree to which one racial or gender group was more likely or less likely than another racial or gender group to be subject to nonjudicial punishment. We found that Black and male servicemembers were more likely than servicemembers of all other races or female servicemembers to receive nonjudicial punishments in the Marine Corps and the Air Force. We also found that Hispanic servicemembers in the Air Force were less likely to be subject to nonjudicial punishment, but we observed no statistically significant difference for Hispanic servicemembers in the Marine Corps. Servicemembers in the Other race category were less likely to be subject to nonjudicial punishment than servicemembers of all other races in the Marine Corps and the Air Force. As shown in figure 21 below, we found that Black servicemembers were convicted in general and special courts-martial at a lower rate in the Army and the Air Force, and at an equal rate in the Navy and the Marine Corps compared to their proportion of the overall general and special courts- martial population. In the Army, the Navy, and the Marine Corps, Hispanic servicemembers were convicted in general and special courts- martial at an equal rate compared to their proportion of the overall general and special courts-martial population. Compared to their proportion of the overall general and special courts-martial population, Hispanic servicemembers were convicted at a lower rate in the Air Force. We could not analyze Coast Guard cases due to the small number of general and special courts-martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. As shown in figure 21 above, bivariate regression analyses found that, in the Army, White servicemembers were statistically significantly more likely to be convicted, whereas Black servicemembers were statistically significantly less likely to be convicted in general and special courts- martial compared to all other servicemembers. White servicemembers in the Air Force were also statistically significantly more likely to be convicted in general and special courts-martial compared to all other servicemembers. In the Marine Corps, we found that males were more likely to be convicted than females, whereas in the Air Force, males were less likely to be convicted than females. The remaining odds ratios shown in figure 21 above were not statistically significant. As shown in figures 22 and 23 below, we found that Black servicemembers received a more severe punishment at a lower rate compared to their share of the convicted service population in the Army, the Navy, and the Air Force. We also found that Hispanic servicemembers received a more severe punishment at a lower rate compared to their share of the convicted service population in the Air Force, but at a higher rate in the Marine Corps. We found that male servicemembers in the Marine Corps and the Air Force received a more severe punishment at a higher rate, and at the same rate in the Army and the Navy, compared to their share of the convicted service population. Females received a more severe punishment at a lower rate in the Air Force and the Marine Corps, and at the same rate in the Army and the Navy, compared to their share of the convicted service population. We could not analyze Coast Guard cases due to the small number of general and special courts-martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. The bivariate regression analyses in Figures 22 and 23 above calculated the degree to which one racial or gender group was more likely or less likely than another racial or gender group to be dismissed or discharged after a conviction in general and special courts-martial. In the Navy, we found that Black servicemembers were statistically significantly less likely to be dismissed or discharged after conviction in general and special courts-martial compared to all other servicemembers. We found no statistically significant differences regarding minority servicemembers being more likely or less likely to be dismissed or discharged after conviction in general and special courts-martial in the Marine Corps, or to receive a more severe punishment in the Army or the Air Force. We found that males in the Marine Corps and the Air Force were more likely to be dismissed or discharged or receive a more severe punishment after conviction than females, but we did not find any statistically significant differences regarding male servicemembers in the Army or the Navy. This appendix contains several figures that show the underlying data related to drug and sexual assault offenses from fiscal years 2013 through 2017 for the Army, the Navy, the Marine Corps, and the Air Force. Across most military services, Black, Hispanic, and male servicemembers were the subjects of recorded investigations and tried in general and special courts-martial at higher rates than their shares of the overall service population for drug offenses, sexual assault offenses, and all other offenses. We found that the likelihood of conviction varied among the services for these two offenses. We analyzed these two specific Uniform Code of Military Justice (UCMJ) offenses separately from all other offenses because service officials told us that an investigation into these offenses may frequently be mandatory, and thus could potentially mitigate the risk of bias. We analyzed data for these offenses for recorded investigations, trials in general and special courts-martial, and convictions from fiscal years 2013 through 2017 to assess the extent to which racial and gender disparities may exist. Our analyses of the services’ investigation, military justice, and personnel databases, as reflected in these figures, taken alone, do not establish the presence or absence of unlawful discrimination. We identified racial and gender differences in recorded investigation rates for drug offenses, sexual assault offenses, and all other offenses compared with the total service populations. Our analysis focused on alleged UCMJ violations for these offenses that were recorded in the Military Criminal Investigative Organization (MCIO) investigations databases. Other investigations conducted within the military, such as command investigations, were not considered in this analysis. For example, as shown in figure 24 below, Black servicemembers were the subjects of recorded investigations for drug offenses, sexual assault offenses, and all other offenses at a higher rate than their share of the overall service population across all military services. Hispanic servicemembers were the subjects of recorded investigations for drug offenses, sexual assault offenses, and all other offenses at a higher rate than their share of the overall service population in the Air Force, but were the subjects of recorded investigations for drug offenses at a lower rate than their share of the overall service population in both the Army and the Marine Corps. Male servicemembers were the subjects of recorded investigations for drug offenses and sexual assault offenses at a higher rate than their share of the overall service population across all of the military services. We found that White servicemembers were tried for drug offenses, sexual assault offenses, and all other offenses in general and special courts- martial at lower rates than their share of the overall service population across all of the military services. Black servicemembers were tried for drug offenses, sexual assault offenses, and all other offenses in general and special courts-martial at a higher rate than their share of the overall service population in all of the military services. Hispanic servicemembers were tried for drug offenses in general and special courts-martial at a lower rate in the Navy and the Marine Corps, and at a higher rate in the Air Force, compared to their share of the overall service population. Hispanic servicemembers were tried for sexual assault offenses at a higher rate than their proportion of the overall service population in all of the military services. Female servicemembers were tried for drug offenses, sexual assault offenses, and all other offenses in general and special courts-martial at lower rates than their share of the general service population in the Army, the Navy, and the Air Force, and were tried for sexual assault offenses and all other offenses at lower rates than their share of the overall service population in the Marine Corps. Figure 25 below shows the gender and racial composition of general and special court-martial trials for drug offenses, sexual assault offenses, and all other offenses. We could not analyze Coast Guard cases due to the small number of general and special courts-martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. We conducted multivariate regression analyses to calculate the degree to which servicemembers charged with drug offenses and sexual assault offenses were more likely or less likely than a composite variable comprised of all other offenses to be convicted in general and special courts-martial, while controlling for other attributes, such as race, gender, education, and rank. As shown in figure 26 below, we did not identify any statistically significant difference in conviction rates for drug offenses compared to all other offenses in the Army, the Navy, the Marine Corps, and the Air Force. Sexual assault offenses were less likely to result in a conviction in the Army, the Navy, and the Air Force, and there was no statistically significant difference for the Marine Corps. We could not analyze Coast Guard cases due to the small number of general and special courts-martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. This appendix contains several tables that show the underlying data and analyses used throughout this report relating to Army personnel and military justice disciplinary actions from fiscal years 2013 through 2017. We did not include populations that contained fewer than 20 servicemembers in the total populations presented in these tables to ensure the protection of sensitive information. As a result, the total populations presented in this appendix may vary among the different tables and may vary from the total populations presented in the body of the report. Our analyses of the Army’s investigations, military justice, and personnel databases, as reflected in these tables, taken alone, do not establish the presence or absence of unlawful discrimination. The multivariate results listed below in table 17 show the odds ratios for the multivariate regression analyses of the Army data. We used logistic regression to assess the relationship between the independent variables, such as race, education, rank, or gender, with the probability of being subject to a military justice action. Logistic regression allows for the coefficients to be converted into odds ratios. Odds ratios that are statistically significant and greater than 1.00 indicate that individuals with that characteristic are more likely to be subject to a military justice action. For example, an odds ratio of 1.55 for Black servicemembers would mean that they are 1.55 times more likely to be subject to a military justice action compared to White servicemembers. Odds ratios that are statistically significant and lower than 1.00 indicate that individuals with that characteristic are less likely to be subject to a military justice action. We excluded years of service from the Army analyses due to high correlation with the rank variable. This appendix contains several tables that show the underlying data and analyses used throughout this report relating to Navy personnel and military justice disciplinary actions from fiscal years 2013 through 2017. We did not include populations that contained fewer than 20 servicemembers in the populations presented in these tables to ensure the protection of sensitive information. As a result, the populations presented in this appendix may vary among the different tables and may vary from the populations presented in other places in this report. Our analyses of the Navy’s investigations, military justice, and personnel databases, as reflected in these tables, taken alone, do not establish the presence or absence of unlawful discrimination. The multivariate results listed below in table 26 show the odds ratios for the multivariate regression analyses of Navy data. We used logistic regression to assess the relationship between the independent variables, such as race, education, rank, or gender, with the probability of being subject to a military justice action. Logistic regression allows for the coefficients to be converted into odds ratios. Odds ratios that are statistically significant and greater than 1.00 indicate that individuals with that characteristic are more likely to be subject to a military justice action. For example, an odds ratio of 1.55 for Black servicemembers would mean that they are 1.55 times more likely to be subject to a military justice action compared to White servicemembers. Odds ratios that are statistically significant and lower than 1.00 indicate that individuals with that characteristic are less likely to be subject to a military justice action. We excluded age and years of service from the Navy multivariate regression analyses due to high correlation with the rank variable. This appendix contains several tables that show the underlying data and analyses used throughout this report relating to Marine Corps personnel and military justice disciplinary actions from fiscal years 2013 through 2017. We did not include populations that contained fewer than 20 servicemembers in the populations presented in these tables to ensure the protection of sensitive information. As a result, the populations presented in this appendix may vary among the different tables and may vary from the populations presented in other places in this report. Our analyses of the Marine Corps investigations, military justice, and personnel databases, as reflected in these tables, taken alone, do not establish the presence or absence of unlawful discrimination. The multivariate results listed below in table 35 show the odds ratios for the multivariate regression analyses of Marine Corps data. We used logistic regression to assess the relationship between the independent variables, such as race, education, rank, or gender, with the probability of being subject to a military justice action. Logistic regression allows for the coefficients to be converted into odds ratios. Odds ratios that are statistically significant and greater than 1.00 indicate that individuals with that characteristic are more likely to be subject to a military justice action. For example, an odds ratio of 1.55 for Black servicemembers would mean that they are 1.55 times more likely to be subject to a military justice action compared to White servicemembers. Odds ratios that are statistically significant and lower than 1.00 indicate that individuals with that characteristic are less likely to be subject to a military justice action. We excluded age and years of service from the Marine Corps multivariate regression analyses due to high correlation with the rank variable. This appendix contains several tables that show the underlying data and analyses used throughout this report relating to Air Force personnel and military justice disciplinary actions from fiscal years 2013 through 2017. We did not include populations that contained fewer than 20 servicemembers in the populations presented in these tables to ensure the protection of sensitive information. As a result, the populations presented in this appendix may vary among the different tables and may vary from the populations presented in other places in this report. Our analyses of the Air Force’s investigations, military justice, and personnel databases, as reflected in these tables, taken alone, do not establish the presence or absence of unlawful discrimination. The multivariate results listed below in table 45 show the odds ratios for the multivariate regression analyses of Air Force data. We used logistic regression to assess the relationship between the independent variables, such as race, education, rank, or gender, with the probability of being subject to a military justice action. Logistic regression allows for the coefficients to be converted into odds ratios. Odds ratios that are statistically significant and greater than 1.00 indicate that individuals with that characteristic are more likely to be subject to a military justice action. For example, an odds ratio of 1.55 for Black servicemembers would mean that they are 1.55 times more likely to be subject to a military justice action compared to White servicemembers. Odds ratios that are statistically significant and lower than 1.00 indicate that individuals with that characteristic are less likely to be subject to a military justice action. We controlled for years of service among the lower enlisted ranks (E1- E4), but excluded age from the Air Force multivariate regression analyses due to high correlation with the rank and years of service variables. This appendix contains several tables that show the underlying data and analyses used throughout this report relating to Coast Guard personnel and military justice disciplinary actions from fiscal years 2013 through 2017. We did not include populations that contained fewer than 20 servicemembers in the populations presented in these tables to ensure the protection of sensitive information. As a result, the populations presented in this appendix may vary among the different tables and may vary from the populations presented in other places in this report. Our analyses of the Coast Guard’s investigations, military justice, and personnel databases, as reflected in these tables, taken alone, do not establish the presence or absence of unlawful discrimination. The multivariate results listed below in table 52 show the odds ratios for the multivariate regression analyses of Coast Guard data. We used logistic regression to assess the relationship between the independent variables, such as race, education, rank, or gender, with the probability of being subject to a military justice action. Logistic regression allows for the coefficients to be converted into odds ratios. Odds ratios that are statistically significant and greater than 1.00 indicate that individuals with that characteristic are more likely to be subject to a military justice action. For example, an odds ratio of 1.55 for Black servicemembers would mean that they are 1.55 times more likely to be subject to a military justice action compared to White servicemembers. Odds ratios that are statistically significant and lower than 1.00 indicate that individuals with that characteristic are less likely to be subject to a military justice action. We excluded age and years of service from the Coast Guard analyses due to high correlation with the rank variable. We found that age, rank, length of service, and education were indicators of a servicemember’s likelihood of being the subject of a recorded investigation, court-martial, or nonjudicial punishment across the military services. To analyze age, rank, length of service, and education, we used bivariate regression analyses to determine which sub-population of each attribute was most likely to be subject to a recorded investigation, court-martial, or nonjudicial punishment. This appendix contains several tables that show the rank, education, length of service, and age groups most likely to be subject to a recorded investigation, tried in general and special courts-martial, tried in summary court-martial, and receive a nonjudicial punishment for all services from fiscal years 2013 through 2017. For the Coast Guard, we could not analyze age, rank, length of service, and education as indicators for courts-martial or nonjudicial punishment due to the small number of recorded military justice cases from fiscal years 2013 through 2017. Our analyses of the services’ investigations, military justice, and personnel databases, as reflected in these tables, taken alone, do not establish the presence or absence of unlawful discrimination. Brenda S. Farrell, (202) 512-3604 or farrellb@gao.gov. In addition to the contact named above, key contributors to this report were Kimberly C. Seay, Assistant Director; Parul Aggarwal; Christopher Allison; Renee S. Brown; Vincent M. Buquicchio; Won (Danny) Lee; Amie M. Lesser; Serena C. Lo; Dae B. Park; Samuel J. Portnow; Clarice Ransom; Christy D. Smith; Preston Timms; and Schuyler Vanorsdale.", "summary": "The Uniform Code of Military Justice (UCMJ) was established to provide a statutory framework that promotes fair administration of military justice. Every active-duty servicemember is subject to the UCMJ, with more than 258,000 individuals disciplined from fiscal years 2013-2017, out of more than 2.3 million unique active-duty servicemembers. A key principle of the UCMJ is that a fair and just system of military law can foster a highly disciplined force. House Report 115-200, accompanying a bill for the National Defense Authorization Act for Fiscal Year 2018, included a provision for GAO to assess the extent that disparities may exist in the military justice system. This report assesses the extent to which (1) the military services collect and maintain consistent race, ethnicity, and gender information for servicemembers investigated and disciplined for UCMJ violations that can be used to assess disparities, and (2) there are racial and gender disparities in the military justice system, and whether disparities have been studied by DOD. GAO analyzed data from the investigations, military justice, and personnel databases from the military services, including the Coast Guard, from fiscal years 2013-2017 and interviewed agency officials. The military services collect gender information, but they do not collect and maintain consistent information about race and ethnicity in their investigations, military justice, and personnel databases. This limits their ability to collectively or comparatively assess these data to identify any disparities (i.e., instances in which a racial, ethnic, or gender group was overrepresented) in the military justice system within and across the services. For example, the number of potential responses for race and ethnicity across the military services' databases ranges from five to 32 options for race and two to 25 options for ethnicity, which can complicate cross-service assessments. The services also are not required to and, thus, do not report demographic information in their annual military justice reports—information that would provide greater visibility into potential disparities. GAO's analysis of available data found that Black, Hispanic, and male servicemembers were more likely than White or female members to be the subjects of investigations recorded in databases used by the military criminal investigative organizations, and to be tried in general and special courts-martial in all of the military services when controlling for attributes such as rank and education. GAO also found that race and gender were not statistically significant factors in the likelihood of conviction in general and special courts-martial for most services, and minority servicemembers were either less likely to receive a more severe punishment than White servicemembers or there was no difference among racial groups; thus, disparities may be limited to particular stages of the process. The Department of Defense (DOD) has taken some steps to study disparities, but has not comprehensively evaluated the causes of racial or gender disparities in the military justice system. Doing so would better position DOD to identify actions to address disparities and help ensure the military justice system is fair and just. Note: These analyses, taken alone, should not be used to make conclusions about the presence or absence of unlawful discrimination. These multivariate regression analysis results estimate whether a racial or gender group is more likely or less likely to be the subject of an investigation or a trial in general or special courts-martial after controlling for race, gender, rank, and education, and in the Air Force, years of service. GAO made all racial comparisons to White servicemembers and all gender comparisons to females. GAO grouped individuals of Hispanic ethnicity together, regardless of race. GAO is making 11 recommendations, including that the services develop the capability to present consistent race and ethnicity data, and DOD include demographic information in military justice annual reports and evaluates the causes of disparities in the military justice system. DOD and the Coast Guard generally concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Our past work has identified five key challenges related to the nation’s ability to detect and respond to biological events that transcend what any one agency can address on its own. They include: (1) enterprise-wide threat determination, (2) situational awareness and data integration, (3) biodetection technologies, (4) biological laboratory safety and security, and (5) emerging infectious disease surveillance. The complexity and fragmentation of roles and responsibilities across numerous federal and nonfederal entities presents challenges to ensuring efficiency and effectiveness across the entire biodefense enterprise. In September 2018, the White House issued the National Biodefense Strategy and through NSPM-14 established a governance structure to guide its implementation. The activities and responsibilities assigned to the interagency governance body by the strategy and NSPM-14 may create new opportunities to make progress on these longstanding and complex issues. However, because implementation of the Strategy and NSPM-14 are in early stages, it remains to be seen how or to what extent they are able to do so. We have ongoing work assessing the strategy and early efforts to implement it. We plan to report in fall 2019. We reported in October 2017 that opportunities remain to enhance threat awareness across the entire biodefense enterprise, leverage shared resources, and inform budgetary tradeoffs among various threats and agency programs. As depicted in figure 1, we reported in October 2017 that key biodefense agencies, including DHS, DOD, HHS, USDA, and EPA, rely on intelligence and global surveillance information, scientific study of disease agent characteristics, and analysis to better understand threats and help make decisions about biodefense investments. These activities are often conducted to support the agencies’ mission or to understand a specific threat. Additionally, to facilitate collaboration among government partners, federal agencies with key roles in biodefense share biological threat information through many different mechanisms including interagency bodies, working groups at the agency and executive level, formalized agreements, colocation, joint projects and funding efforts, and shared expertise (see figure 2). The collaborative mechanisms in which the key agencies in our October 2017 review participated may facilitate information sharing in support of specific federal activities and in individual programs, or in response to specific biological events after they begin to unfold. However, as we reported in October 2017, there was no existing mechanism that could leverage threat awareness information to direct resources and set budgetary priorities across all agencies for biodefense. The nation faces many biological threats, including naturally occurring diseases that affect human, animal, and plant health, and biological weapons used by state or nonstate actors. Without a mechanism that is able to assess the relative risk from biological threats across all sources and domains, the nation may be limited in its ability to prioritize resources, defenses, and countermeasures against the most pressing threats. The Strategy and NSPM-14 outline requirements for participating agencies that lay the ground work for a more systematic, cross- government examination of existing programs. The effort offers the potential for the nation to progress toward more integrated and enterprise-wide threat awareness and to use that information to identify opportunities to leverage resources, but this will take time and entails a change in the way participating agencies have traditionally operated. Because implementation of the strategy is in its early stages, it is too soon to assess how, if at all, it might address this challenge. We have reported that DHS’s National Biosurveillance Integration Center (NBIC), which was created to integrate data across the federal government with the aim of enhancing detection and situational awareness of biological events, has suffered from long-standing issues related to its clarity of purpose. In 2009, we reported that some of NBIC’s partners were not convinced of the value that working with NBIC provided because NBIC’s mission was not clearly articulated. We also reported that NBIC was not fully equipped to carry out its mission because it lacked key resources—data and personnel—from its partner agencies, which may have been at least partially the result of collaboration challenges it faced. In the 2009 report, we recommended that NBIC develop a strategy for addressing barriers to collaboration and develop accountability mechanisms to monitor these efforts. DHS agreed, and in August 2012 NBIC issued the NBIC Strategic Plan, to provide its strategic vision, clarify the center’s mission and purpose, and articulate the value that NBIC seeks to provide to its partners, among other things. In September 2015, we reported that despite NBIC’s efforts to collaborate with interagency partners to create and issue a strategic plan that would clarify its mission and efforts, a variety of challenges remained. We identified options for policy or structural changes that could help a federal data integrator like NBIC better fulfill its mission, given the complexity and difficulty inherent in achieving truly integrated situational awareness that makes new meaning out of disparate data, but we did not make specific recommendations. The National Biodefense Strategy identified biosurveillance data integration among several information sharing activities that need to be enhanced. Interagency attention to the goals, opportunities, and challenges of enterprise-wide data integration offers the potential for the nation to better define what kind of integrated situational awareness is possible, what it will take to effectively and efficiently achieve it, and what value it has. However, it remains to be seen how or whether the interagency efforts to implement the Strategy will be able to address ongoing situational awareness and data integration challenges. Since 2012, we have reported that DHS has faced challenges in clearly justifying the need for the BioWatch program and its ability to reliably address that need (to detect aerosolized biological attacks). In September 2012, we found that DHS approved a next-generation BioWatch acquisition in October 2009 without fully developing knowledge that would help ensure sound investment decision making and pursuit of optimal solutions. We recommended that before continuing the acquisition, DHS reevaluate the mission need and possible alternatives based on cost-benefit and risk information. DHS concurred and in April 2014, canceled the acquisition because an alternatives analysis did not confirm an overwhelming benefit to justify the cost. DHS continues to rely on the currently-deployed BioWatch system for early detection of an aerosolized biological attack, but in 2015 we found that DHS lacked reliable information about the current system’s technical capabilities to detect a biological attack, in part because DHS had not developed technical performance requirements for the system. We reported in September 2015 that DHS commissioned tests of the current system’s technical performance characteristics, but without performance requirements, DHS could not interpret the test results and draw conclusions about the system’s ability to detect attacks. At the time of our report in October 2015, DHS was considering upgrades to the Gen-2 system, but we recommended that DHS not pursue upgrades until it establishes technical performance requirements to meet a clearly defined operational objective and assesses the system against these performance requirements. DHS concurred and reported it was working to address the recommendation. DHS has since begun to acquire a different type of biodetection system, BioDetection 21 (or BD21), intended to replace BioWatch. BD21 is currently in a pilot phase; therefore we cannot yet determine how it will be implemented in the future or what decisions DHS will ultimately make regarding the existing BioWatch system. In August 2017, we reported that from a homeland security and public health perspective, threats of bioterrorism, such as anthrax attacks, and high-profile disease outbreaks, such as Ebola and emerging viruses like dengue, chikungunya, and Zika, highlight the continued need for diagnostic tests that provide early detection and warning about biological threats to humans. Multiplex point-of-care technologies are technologies that can simultaneously test for more than one type of human infectious disease pathogen from a single patient sample (such as blood, urine, or sputum) in one run at or near the site of a patient. Multiplex point-of-care technologies can be used for diagnosing different diseases, including more common diseases such as influenza, emerging infectious diseases, or diseases caused by select agents in minutes to a few hours. We further reported that, while potential benefits of these technologies include more appropriate use of antibiotics and improved ability to limit the spread of disease, among others, developers and users disagreed on the strength of evidence showing the extent of multiplex point-of-care technologies’ improvement on patient outcomes and identified the need for more clinical studies to establish the benefits of these technologies. Additionally, implementation challenges include lack of familiarity with such technologies, cost considerations, false positive results for rare diseases, and the challenges related to the regulatory review process for developers to get approval or clearance to market their technologies. The National Biodefense Strategy and its interagency governing leadership offer the potential for the nation to better define the role of detection technologies in a layered national biodefense capability to help those that pursue these technologies better articulate the mission needs and align requirements and concepts of operation accordingly. Because implementation of the strategy is in its early stages, it remains to be seen how or whether the interagency will be able engage on this issue in a way that helps to drive informed investment tradeoff decisions about technology alternatives. We—along with Congress and various federal committees—have, for many years, identified challenges and areas for improvement related to the safety, security, and oversight of high-containment laboratories. These laboratories conduct research on hazardous pathogens—such as the Ebola virus and the bacteria that causes anthrax—and toxins that may pose a serious threat to humans, animals, or plants. In 2008 and 2009, we found a proliferation of high-containment laboratories across the United States, with the number of such laboratories in the government, academic, and private sectors increasing since 2001. We recommended that the National Security Advisor name an entity charged with government-wide strategic evaluation of high-containment laboratories. National Security Staff disagreed with this recommendation. After reporting on these issues again in 2013, the Office of Science and Technology Policy implemented this recommendation. In January 2013, we also found that, for the subset of these laboratories subject to federal oversight, the oversight was duplicative, fragmented, and dependent on self-policing. We recommended that HHS’s Centers for Disease Control and Prevention and USDA’s Animal and Plant Health Inspection Service work with DHS and DOD to coordinate inspections and ensure consistent application of inspection standards; the departments generally agreed with our recommendations and noted various actions they had already taken, or planned to take, to coordinate inspection efforts, such as conducting joint inspections. More recently, in response to reported lapses in laboratory safety at HHS and DOD in 2014 and 2015, we examined how federal departments oversee their high-containment laboratories. In March 2016, we found that most of the 8 departments and 15 agencies that we reviewed had policies that were not comprehensive or were not up to date. Also, while the departments and agencies we reviewed primarily used inspections to oversee their high-containment laboratories, some of them were not routinely reporting inspection results, laboratory incidents, and other oversight activities to senior officials. We made 33 recommendations in total, including that departments develop and update policies to include missing elements and ensure that oversight activity results are reported to senior officials. To date, 12 of the 33 recommendations have been implemented—including updating policies and reporting requirements. We continue to monitor agency progress in implementing the 21 that remain open. In response to several incidents involving the shipment of improperly inactivated pathogens, in August 2016 we reported on issues related to the inactivation of pathogens in high-containment laboratories and found that both the science and the federal guidance around pathogen inactivation are limited and inconsistently implemented. Additionally, we found that federal officials did not know how many incomplete inactivation incidents have occurred because laboratories do not have to identify them in incident reports, and are only required to report incidents involving certain pathogens. We made 11 recommendations to HHS and USDA that they improve the oversight of inactivation by revising reporting forms, improving guidance for development and validation of inactivation protocols, and developing consistent criteria for enforcement of incidents involving incomplete inactivation. To date, 6 of the 11 recommendations have been addressed and we continue to monitor the 5 that remain open. Safety lapses continued to occur at laboratories in the United States that conduct research on hazardous pathogens, raising concern about the efficacy of federal oversight. In October 2017, we found that the Federal Select Agent Program—jointly managed by HHS and USDA—oversees laboratories’ handling of certain hazardous pathogens known as select agents, but the program does not fully meet all key elements of effective oversight. For example, the Federal Select Agent Program was not independent from all laboratories it oversees, and it had not assessed risks posed by its current structure or the effectiveness of its mechanisms to reduce organizational conflicts of interest. We made 11 recommendations for the Federal Select Agent Program, including to (1) assess risks from its current structure and the effectiveness of its mechanisms to reduce conflicts of interest and address risks as needed, (2) assess the risk of activities it oversees and target reviews to high-risk activities, and (3) develop a joint workforce plan; to-date, 5 of 11 recommendations have been addressed and we continue to monitor the progress for the 6 that remain open. In September 2018 we found that DOD had made progress by taking a number of actions to address the 35 recommendations from the Army’s 2015 investigation report on the inadvertent shipment of live anthrax; however, DOD had not yet developed an approach to measure the effectiveness of these actions. Additionally, we reported that although DOD had implemented a Biological Select Agents and Toxins Biosafety and Biosecurity Program to improve management, coordination, safety, and quality assurance, DOD had not developed a strategy and implementation plan for managing the program. Also, we found that the Army had not fully institutionalized measures to ensure that its biological test and evaluation mission remains independent from its biological research and development mission so that its test and evaluation procedures are objective and reliable. Finally, DOD had not completed a required study and evaluation of its Biological Select Agents and Toxins infrastructure that will affect the future infrastructure of the Biological Select Agents and Toxins Biosafety and Biosecurity Program. DOD officials had no estimated time frames for when DOD will complete the study and evaluation. We recommended that DOD develop an approach to assess the effectiveness of the recommendations, a strategy and implementation plan for its Biological Select Agents and Toxins Biosafety and Biosecurity Program, measures to ensure independence, and time frames to complete a study. To date, all of these recommendations remain open. In agency comments, DOD concurred with all four of our recommendations and discussed the actions the department intended to take to address them, including finalizing the development of a long-term strategy and implementation plan by September 1, 2019. The National Biodefense Strategy highlights the need for continuous improvement of biosafety and biosecurity for laboratories and other facilities. However, it is not yet known how, if at all, the strategy will drive interagency partners to develop additional oversight or other practices to mitigate the risk of bioincidents at high containment laboratories, because implementation of the strategy is in its early stages. We have reported that establishing and sustaining biosurveillance capabilities can be difficult for a myriad of reasons. For example, maintaining expertise in a rapidly changing field is difficult, as is the challenge of accurately recognizing the signs and symptoms of rare or emerging diseases. Additionally, we reported in October 2011 that funding targeted for specific diseases does not allow for focus on a broad range of causes of morbidity and mortality, and federal officials have said that the disease-specific nature of funding is a challenge to states’ ability to invest in core biosurveillance capabilities. Further, we reported in May 2018 that although the awards funded by supplemental appropriations have allowed state and local public health departments, laboratories, and hospitals to surge during a threat—for example, the H1N1influenza and Zika viruses—most of the 10 non-federal stakeholders we interviewed, as well as HHS officials said that the timing of these awards can result in challenges to carrying out preparedness and response activities during infectious disease threats. An effective medical response to a biological event depends in part on the ability of individual clinicians and other professionals to identify, accurately diagnose, and effectively treat diseases, including many that may be uncommon. For example, in May 2017, we reported that because Zika virus disease was a newly emerging disease threat in the United States and relatively little was known about the virus prior to 2016, HHS and state and local public health agencies were not fully equipped with information and resources needed for a rapid response at the outset of the recent outbreaks. They faced challenges establishing and implementing surveillance systems for Zika virus disease and infection and its associated health outcomes. Additionally, in March 2019, we reported that USDA would likely face surveillance challenges that could delay detection of the first cases in a foot-and-mouth disease outbreak in livestock, which could have a devastating impact on our economy and trade agreements. For example, foot-and-mouth disease can spread without detection as signs can be difficult to notice in some species, take up to 4 days to manifest after an animal is infected, and infection in wild animals could go undetected and continue to spread the virus. In 2011, while reporting on nonfederal biosurveillance efforts, we found state and local agriculture, public health, and wildlife departments were completely or largely dependent on federal funding for biosurveillance- related activities. At that time, we also reported that the common federal approach of disease-specific funding—for example, West Nile virus— limited nonfederal efforts to develop core capabilities that could provide surveillance capacity that cut across health threats and for emerging- disease threats. According to federal, state, and local officials, early detection of potentially serious disease indications nearly always occurs first at the local level, making the personnel, training, systems, and equipment that support detection at the state and local level a cornerstone of our nation’s biodefense posture. In May 2018, we reported that officials from HHS told us that their grant awards funded by annual appropriations are intended to establish and strengthen emergency preparedness and capacity building, but may not fully support the need for surge capacity that states and other jurisdictions require in order to respond to an infectious disease threat. We reported that during recent infectious disease threats, HHS received supplemental appropriations to respond to Zika in 2016, Ebola in 2014, and H1N1 pandemic influenza in 2009. However, as mentioned above, officials also said that the timing of these awards can result in challenges to carrying out preparedness and response activities during infectious disease threats. HHS officials, as well as all 10 selected non-federal stakeholders, also noted in May 2018 that a funding mechanism to fund rapid response activities when additional support is needed would be beneficial and could help address timing challenges. However, we reported that concerns were also raised about (1) when such a mechanism for funding infectious disease threats should be used, and (2) that any type of emergency fund should not be used to make up for a lack in investments at all levels of government for current preparedness and capacity-building activities. We did not make recommendation as part of this work. However, part of our May 2018 reporting included perspectives from various stakeholders on such a fund. Stakeholders cited six factors that may be considered for a new emergency response fund: (1) who determines when to use an emergency fund, (2) what factors would trigger the use of an emergency fund, (3) methods to determine the amount of available funding, (4) activities to fund with an emergency fund, (5), accountability for use of an emergency fund, and (6) whether an emergency fund would be specific to infectious disease threats. The National Biodefense Strategy and its interagency governance structure offer the opportunity to design new approaches to identifying and building a core set of surveillance and response capabilities for emerging infectious diseases. However, it is too early into implementation to determine how effective, if at all, the new strategy will be in addressing this challenge. How and to what extent implementation of the Strategy is able to efficiently leverage and effectively sustain capacity across both nonfederal and federal stakeholders will affect how prepared the nation is to more quickly gear up for whatever challenges emerge when outbreaks of previously non-endemic diseases threaten the nation. Thank you, Chairman Lynch, Ranking Member Hice, and Members of the Subcommittee. This concludes my prepared statement. I would be happy to respond to any question you may have at this time. If you or your staff has any questions concerning this testimony, please contact Christopher P. Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Kathryn Godfrey (Assistant Director), Susanna Kuebler (Analyst-In-Charge), Nick Bartine, Jeffrey Cirillo, Michele Fejfar, Eric Hauswirth, Tracey King, Dawn Locke, and Adam Vogt. Key contributors for the previous work that this testimony is based on are listed in each product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Catastrophic biological events have the potential to cause loss of life, and sustained damage to the economy, societal stability, and global security. The biodefense enterprise is the whole combination of systems at every level of government and the private sector that contribute to protecting the nation and its citizens from potentially catastrophic effects of a biological event. Since 2009, GAO has identified cross-cutting issues in federal leadership, coordination, and collaboration that arise from working across the complex interagency, intergovernmental, and intersectoral biodefense enterprise. In 2011, GAO reported that there was no broad, integrated national strategy that encompassed all stakeholders with biodefense responsibilities and called for the development of a national biodefense strategy. In September 2018, the White House released a National Biodefense Strategy. This statement discusses GAO reports issued from December 2009 through March 2019 on various biological threats and biodefense efforts, and selected updates to BioWatch recommendations made in 2015. To conduct prior work, GAO reviewed biodefense reports, relevant presidential directives, laws, regulations, policies, strategic plans; surveyed states; and interviewed federal, state, and industry officials, among others. GAO's past work has identified a number of challenges related to the nation's ability to detect and respond to biological events that transcend what any one federal department or agency can address on its own. They include, among others: Assessing enterprise-wide threats. In October 2017, GAO found there was no existing mechanism across the federal government that could leverage threat awareness information to direct resources and set budgetary priorities across all agencies for biodefense. GAO said at the time that the pending biodefense strategy may address this. Situational awareness and data integration. GAO reported in 2009 and 2015 that the Department of Homeland Security's (DHS) National Biosurveillance Integration Center (NBIC)—created to integrate data across the federal government to enhance detection and situational awareness of biological events—has suffered from longstanding challenges related to its clarity of purpose and collaboration with other agencies. DHS implemented GAO's 2009 recommendation to develop a strategy, but in 2015 GAO found NBIC continued to face challenges, such as limited partner participation in the center's activities. Biodetection technologies. DHS has faced challenges in clearly justifying the need for and establishing the capabilities of the BioWatch program—a system designed to detect an aerosolized biological terrorist attack. In October 2015, GAO recommended that DHS not pursue upgrades until it takes steps to establish BioWatch's technical capabilites. While DHS agreed and described a series of tests to establish capabilities, it continued to pursue upgrades. Biological laboratory safety and security. Since 2008, GAO has identified challenges and areas for improvement related to the safety, security, and oversight of high-containment laboratories, which, among other things, conduct research on hazardous pathogens—such as the Ebola virus. GAO recommended that agencies take actions to avoid safety and security lapses at laboratories, such as better assessing risks, coordinating inspections, and reporting inspection results. Many recommendations have been addressed, but others remain open, such as finalizing guidance on documenting the shipment of dangerous biological material. In September 2018, the White House issued the National Biodefense Strategy and associated plans, which could help to address some of the ongoing challenges GAO has previously identified. However, because implementation of the strategy is in early stages, it remains to be seen how or to what extent the agencies responsible for implementation will institutionalize mechanisms to help the nation make the best use of limited biodefense resources. GAO is currently reviewing the strategy and will report out later this year. GAO has made numerous agency recommendations in its prior reports designed to address the challenges discussed in this statement. As of June 2019, agencies have taken steps to address many of these, and GAO is monitoring ongoing efforts.", "document_type": "gao"}
{"report": "Over the next 10 years, the Navy plans to continue developing critical technologies, complete detail design, and begin construction of the lead Columbia class submarine. In December 2017, we found that the schedule to deliver the lead submarine was aggressive, with extensive overlap—or concurrency—between development, design, and construction, as shown in figure 1. Our prior work reviewing shipbuilding programs has shown that the programs with the greatest amount of overlap between shipbuilding phases often have the highest cost and schedule growth, as well as quality and performance issues. The National Defense Authorization Act for Fiscal Year 2018 included reporting requirements for the Columbia class program. As part of these annual reporting requirements, the Navy must submit to Congress matrices that identify (1) key milestones, events, and performance goals for the design and construction of the Columbia class program; and (2) costs associated with the design and construction period of the Columbia class program. The Navy submitted its initial matrices to Congress in February 2018 and an update to the matrices in October 2018. The next matrices update is due in March 2019 and annually, thereafter, until the lead Columbia submarine is delivered. The Navy is developing a number of new technologies related to submarine propulsion, missile tubes, and survivability that are planned to ensure that the Columbia class will remain operationally relevant throughout its planned 42.5-year service life, as shown in figure 2. In 2015, as part of its technology readiness assessment, the Navy identified two technologies—the advanced carbon dioxide removal unit and the stern area system—as critical technology elements. However, as we found in 2017, several Columbia class technologies that met GAO’s definition of a critical technology element were not identified by the Navy as critical technologies. In addition, several of these were immature, with technology readiness levels (TRL)—used to describe the maturity of critical technologies—of less than 7. See appendix II for a description of TRLs. As part of its matrices to Congress, the Navy is required to report on the TRLs of the integrated power system, nuclear reactor, propulsor, coordinated stern features, stern area system, and common missile compartment—which are the critical technologies we identified in our prior report. Table 1 lists each GAO-identified critical technology and its TRL as of October 2018, as reported by the Navy. Two shipbuilders—General Dynamics Electric Boat (Electric Boat) and Huntington Ingalls Industries Newport News (Newport News)—design and build nuclear submarines. Electric Boat is the prime contractor for both design and construction of the Columbia class program, with Newport News serving as a subcontractor. Similar to the Virginia class program, each shipbuilder will construct segments of the submarine, but Electric Boat will complete final outfitting and deliver the submarines to the Navy. The Navy awarded a detail design contract in September 2017 to Electric Boat for work including completion of the submarine’s design, component and technology development, and prototyping efforts. The detail design process for the Columbia class program encompasses three activities, which began after the Navy set the technical requirements for the submarine in 2016: Arrangements outline the steel structure and routes distributive systems—such as electrical or piping systems—throughout the submarine. At this time, the shipbuilder generates a three-dimensional computer-aided design model for the area. Disclosures complete the design work for even the lowest-level items of the submarine, including material information. After these are completed, the shipbuilder can begin ordering material and long lead items for the submarine. Work instructions are three-dimensional electronic products that shipyard workers use to construct the submarine. Figure 3 illustrates the design phases for the Columbia class program. The shipbuilder will design and construct Columbia class submarines in six large hull segments, referred to as super modules, a method also used to construct most of the Virginia class submarines. During construction, the modules will largely be outfitted with systems and connections prior to being attached together during final assembly. According to the shipbuilder, this method is more efficient than outfitting the hull after it is constructed because more workspace is available to install equipment. Figure 4 illustrates the super modules within the submarine. A reliable cost estimate is critical to program success. It provides the basis for informed investment decision making, realistic budget formulation and program funding, meaningful progress measurement, proactive course correction when warranted, and accountability for results. GAO’s Cost Estimating and Assessment Guide states that reliable cost estimates reflect four characteristics, which encompass 19 best practices. These characteristics—comprehensive, well documented, accurate, and credible—are shown in table 2. For Navy shipbuilding programs, including the Columbia class, several different entities are involved in cost estimating: The Naval Sea Systems Command (NAVSEA) Cost Engineering and Industrial Analysis Group develops the program life-cycle cost estimate, which is an estimate accounting for the total cost to the government of acquisition and ownership of a system over its full life. NCCA develops an independent cost assessment for certain Navy programs, such as the Columbia class program, at milestone events in the defense acquisition system. This assessment is not a separate estimate, but rather a review of the NAVSEA program life-cycle cost estimate. A cost review board, comprised of multiple Navy offices, establishes a service cost position based on their review of the program life-cycle cost estimate and the independent cost assessment. The Office of the Secretary of Defense’s CAPE conducts or approves independent cost estimates for major defense acquisition programs. Independent cost estimates are statutorily required for major defense acquisition programs at milestone events. The milestone decision authority, which in the case of the Columbia class program is the Under Secretary of Defense for Acquisition and Sustainment, reviews the service cost position and independent cost estimate and selects the cost estimate to baseline and fund the program. The most recent milestone event for the Columbia program was the Milestone B decision in January 2017, where the program received approval to proceed to the next acquisition phase—engineering and manufacturing development, which includes detail design of the lead submarine. In a memo documenting that decision, the milestone decision authority noted that significant development risks remain for the Columbia program and cost control must remain a priority. To limit program cost growth, the milestone decision authority established an affordability cap: the average submarine procurement cost should not exceed $8.0 billion in constant year 2017 dollars. Figure 5 summarizes the cost estimating process for the Columbia class program’s Milestone B review. The Navy is attempting to mitigate an aggressive schedule for lead submarine construction by (1) setting a goal to mature a significant amount of the submarine’s design prior to the start of construction and (2) beginning advance construction of submarine modules prior to October 2020. The shipbuilder is working to improve design performance and would have to maintain this increased pace to achieve its design goal, which is necessary to mitigate schedule risk associated with constructing the lead submarine. This may prove challenging as it must complete an increasingly higher volume and complexity of design products. At the same time, the Navy is continuing to develop several critical technologies and recent manufacturing defects with the integrated power system and missile tubes are among the challenges that the Navy is facing in ensuring timely delivery of critical components to the shipyard. Finally, to achieve Columbia’s aggressive construction schedule, while simultaneously building Virginia class submarines, the shipbuilder is working to ensure that it has sufficient shipyard capacity—including new facilities, additional suppliers, and an increased workforce. The shipbuilder has failed to achieve its planned rates for completing design arrangements and disclosures to meet its design maturity goal in recent months—hampered by implementation of a new design software tool and an insufficient number of designers to meet monthly design completion rates. As we reported in December 2017, the Navy’s priority is to complete a high level of design—specifically, 100 percent of design arrangements and 83 percent of design disclosures—by the start of lead submarine construction in October 2020. By maturing the design before beginning construction on the lead submarine, the Navy is attempting to mitigate the risk of costly rework from design changes and subsequent delays to the Columbia class program’s 84-month construction schedule, which the Navy has acknowledged is aggressive. The Navy established the design maturity goal for Columbia based on lessons learned from the Virginia class program, when the shipbuilder began constructing the lead submarine with only 76 percent of arrangements and 43 percent of disclosures completed and, subsequently, realized 21 percent cost growth. Since the shipbuilder began work on the detail design, it has generally met its overall goal of completing the arrangements on schedule. As detail design continues, however, the shipbuilder is transitioning from relatively simple designs for the hull to the more complex designs for the submarine’s internal systems, increasing the pace needed to complete the remaining designs, as shown in figure 6. Navy officials stated that design disclosures are generally considered the most challenging phase of design work, where the shipbuilder specifies the lowest-level items and defines all aspects of the submarine. The shipbuilder has to maintain this increased pace in order to achieve the design maturity goal by the start of lead submarine construction. However, the shipbuilder’s design progress in completing disclosure products has fallen short of its plan in recent months as the planned pace and complexity of the design has increased. Using data from the program’s cost performance reports, we analyzed the shipbuilder’s monthly design progress according to a schedule performance index that measures the value of the work completed against the work scheduled. For example, if the schedule performance index is less than 1.00, then the shipbuilder has completed less than a dollar’s worth of work for each dollar that was scheduled. As shown in figure 7, since January 2018, schedule performance has consistently fallen below 1.00. Both DOD and Navy officials attributed the shipbuilder’s design delays to challenges adapting to a new design software tool. Beginning with the Columbia class program, the shipbuilder transitioned to a new customized software tool for design and construction because its prior software was no longer supported by the original developer. However, the shipbuilder has experienced problems developing the tool, which has resulted in slower progress to complete both design arrangements and disclosures, as certain aspects of the software’s functionality were delayed. Navy officials stated that, as of June 2018, they believe that design software functionality was performing at a level that no longer impeded design progress. While the designers have gained proficiency with the new design tool to complete arrangements and disclosures, according to Navy officials, the shipbuilder is now facing similar challenges using the tool to generate work instructions. Navy program officials also stated that the shipbuilder has not delivered some of the software functionality needed to produce work instructions as scheduled. Further, Navy officials noted that the process to create work instructions from completed disclosures takes longer with the new design software so the shipbuilder has begun generating work instructions earlier. According to Navy officials and shipbuilder representatives, the shipbuilder hired 150 additional designers in an effort to recover its design schedule and meet future monthly design goals. However, adding designers to recover and maintain the shipbuilder’s design schedule ultimately increases the program’s design costs. Similar to the schedule analysis above, we used data from cost performance reports to analyze the shipbuilder’s monthly design progress according to a cost performance index that measures the budgeted value of the work completed against what it actually costs to complete it. For example, if the cost performance index is less than 1.00, then less than a dollar’s worth of work has been completed for each dollar spent. As shown in figure 8, the shipbuilder’s cost performance has consistently fallen below 1.00 since December 2017. If the shipbuilder cannot address challenges associated with using the software tool to generate work instructions discussed above, it will likely need additional design hours in the future, resulting in higher costs in order to mature the design on schedule. Navy officials and shipbuilder representatives expect to mitigate risks associated with the Columbia construction schedule by accelerating the building of certain components more than a year in advance of the formal start of construction. They anticipate that this advance construction strategy will allow them to gain 2 months of schedule margin for final assembly and testing prior to delivery of the lead submarine. Starting in December 2018, the shipbuilder will begin constructing modules of the submarine as part of its advance construction effort. In 2017, we reported that the Navy had planned to begin advance construction for four of the submarine’s six super modules, but since our report was issued, it now plans to begin construction on all six super modules including building components like the stabilizers, impulse tanks, and others. Figure 9 shows the start of advance construction for each super module. Navy officials estimate that the current advance construction efforts will require approximately 631,000 labor hours. In addition, advance construction efforts would require that the Navy accelerate delivery of equipment provided to the shipbuilder for installation on the submarine, such as pumps and valves. Shipbuilder representatives stated that a lesson learned from the Virginia class program was that construction of certain complex components should begin as early as possible if capability requirements and designs are stable. However, based on its plan, the shipbuilder will begin advance construction having completed less than 40 percent of the total design disclosures for the Columbia class submarine, as shown in figure 10. The number of disclosures completed at the start of advance construction is less than half of those the shipbuilder plans to complete by the start of lead submarine construction in October 2020. Navy officials stated that they believe the risk associated with beginning construction with a less mature overall design is mitigated because the program selected components for advance construction that are well understood and unlikely to be affected by design changes, like ballast tanks, decking, and hull segments. In addition, Navy officials stated that they will not begin construction on the component or hull unless the arrangements associated with the structure of that area of the submarine are complete. However, based on the shipbuilder’s design plans, the arrangements and disclosures of adjoining areas of the super module may not be complete, which could negatively affect construction. Specifically, the shipbuilder’s design plans indicate that it will have completed 100 percent of disclosures for only one super module at the start of advance construction. As we have found in our prior work, proceeding with construction despite having completed fewer designs than planned increases the likelihood of design changes later that may, in turn, require costly and time-intensive re-work to change components that have already been built. Shipbuilder representatives acknowledged that there is risk in starting construction of some components prior to completing the design for individual super modules or the entire submarine. However, shipbuilder representatives stated that they believe this risk is reduced by only starting construction on components for which the disclosures are complete. While ship design is underway, the Navy is continuing to develop and mature the critical technologies related to the Columbia class program. While these critical technologies are not required at the shipyard for several years, recent challenges have eroded available schedule margin, as illustrated below: Integrated Power System: In 2017, we reported that the Navy experienced manufacturing problems associated with the integrated power system. We found that the Navy continues to experience problems with the electric drive of the integrated power system that could potentially affect construction of the lead submarine. A manufacturing defect that affected the system’s first production- representative propulsion motor required extensive repair that consumed 9 months of schedule margin at the land-based test facility. The Navy now plans to test the motor at the same time it had originally scheduled to make any final design changes before starting production. This could constrain opportunities to implement timely, corrective actions if problems are discovered during testing. Common Missile Compartment: Navy officials stated that, in July 2018, the shipbuilder identified substantial weld defects in missile tubes from one of three tube suppliers and resulted in investigations of the missile tubes from all suppliers. These defects were discovered after seven tubes in various stages of outfitting had already been delivered to the shipyard and five additional tubes under production have been affected. Navy program officials stated defects occurred because inexperienced welders performed the complex work and inspectors at the supplier’s facility subsequently failed to identify the defects. While the Navy and shipbuilder are still determining the cost and schedule impacts of the weld defects, program officials estimated that addressing this issue will consume up to 15 of the 23-month schedule margin for these components. In addition, program officials stated that the Navy likely will be responsible for some of the cost associated with investigating the root cause of the defects and risk mitigation efforts going forward. Given the erosion of available schedule margin, there is less time available to address issues without resulting in schedule delays. For example, the shipbuilder’s construction plans for two super modules do not include schedule margin to accommodate any delays that may occur as the technologies are matured and detail design is completed. One of these, the stern super module contains three technologies that are not fully mature—the integrated power system, stern area system, and advanced propulsor bearing. The integrated power system is not expected to reach full maturity until October 2019 and the remaining two technologies will not be mature until after the shipbuilder begins construction on the lead submarine, not including those components that begin advance construction years earlier. Without schedule margin to accommodate any changes or issues, any delays in delivering equipment to the shipyard on time could disrupt the shipbuilder’s construction sequence for the lead submarine. To meet the Navy’s aggressive construction schedule for the lead submarine, the shipbuilder has to ensure that it has the capacity to meet a substantially higher workload and effectively balance Columbia and Virginia class construction. At the same time as construction on Columbia begins in 2020, the shipbuilder will also have begun constructing two modified Virginia class attack submarines per year. To accommodate the construction of both submarine classes, the shipbuilder is planning an extensive expansion of its facilities, including new buildings, a pier, an ocean transport barge, and a floating dry dock. The anticipated increases in workload at the shipyard will also require the shipbuilder to manage a higher volume of build materials and an expansion of its workforce. While construction of new facilities is progressing on schedule, according to shipbuilder representatives, it faces other challenges preparing for Columbia class construction. Achieving the planned construction schedule will require the Navy and shipbuilder to ensure that materials arrive on time and meet quality expectations, but according to Navy officials, supplier oversight has been a challenge for this shipbuilder in the past. Both Navy officials and shipbuilder representatives stated that they are concerned about the capacity of its suppliers to meet the demand for high-quality components given an industrial base that has diminished significantly since previous major submarine construction efforts in the 1980s. Many of the parts and equipment on Columbia class are common with those used on Virginia class submarines but, in other instances, suppliers are producing components for the first time after a considerable break, such as missile tubes that have not been produced since the early 1990s. Navy program officials and shipbuilder representatives stated that they monitor supplier capacity and quality—among other areas—and they have several methods to intervene if a supplier is not able to perform as needed. The shipbuilder and the Navy have formed a group to assess the three primary areas of supplier performance: Capability: includes the uniqueness of the supplier’s product on the market, challenges in shifting to a different supplier due to intellectual property rights or technical knowledge, and the ability for the supplier to sustain their own supply base. Capacity: includes the supplier’s ability to increase production without decreasing quality, maintain that capacity over the program’s production, their financial dependence on Navy programs for revenue, lead time needed to meet new orders, and the capacity of their own suppliers. Cost: includes the costs of increasing production spread out across demand from Navy programs. In 2017, the shipbuilder assessed its supplier base using these areas, identified the criticality and risk of each supplier based on their potential impact to the program and potential alternate suppliers, and conducted a gap analysis comparing the supplier’s current performance to the program’s desired performance. Based on the results of the analysis, the shipbuilder identified and is monitoring at-risk suppliers in coordination with the Navy to determine if immediate intervention is needed, such as investing in new facilities for the supplier, improving manufacturing workflow, or finding new sources of material from that supplier. Despite these efforts, supplier oversight remains an issue, because—in the instance of the missile tube welds mentioned above—the shipbuilder focused on managing certain anticipated risks, as opposed to actively managing the supplier’s quality and performance with on-site independent inspections, according to Navy officials. In response to the missile tube issues, the shipbuilder has proposed additional supplier oversight by assessing the need for on-site inspection teams depending on the risk each supplier poses to the program. Navy officials stated that they have begun some assessments but, as of March 2019, had yet to determine who will pay for this additional oversight. We plan to more fully assess the Navy and shipbuilder’s oversight of its suppliers for the Columbia class program in future work. According to shipbuilder representatives, the start of lead submarine construction for the Columbia class, combined with expanding Virginia class construction, increases the demand for hiring and retaining skilled workers at levels not seen at this shipyard since the 1980s. Navy officials expressed concerns about the risk of adding large numbers of new workers, including an influx of inexperienced welders and inspectors— issues that also contributed to the defects in missile tubes discussed above. To support growing workload from both the Columbia and Virginia submarine programs, the shipbuilder plans to increase workforce at its two facilities over the next decade: by 66 percent at Quonset Point, Rhode Island—where the components and individual submarine modules will be constructed—and 174 percent at Groton, Connecticut—where the super modules will undergo final outfitting and assembly. To meet this increased demand in a skilled workforce, the shipbuilder assessed future demographic trends in the area surrounding its facilities and found that, while sufficient labor will likely be available, more training will be necessary. Consequently, the shipbuilder established internal and external training programs and partnerships with educational institutions in the area to grow the qualified workforce in time to begin lead submarine construction in October 2020. The influx of inexperienced workers can temporarily decrease construction efficiency as compared to a current, more experienced workforce. For example, when the Virginia class program expanded its workforce to build a second submarine each year, the addition of new staff contributed to an 8 percent decrease in cost efficiency for the program. Shipbuilder representatives at one production facility have already reported reduced efficiency following increased hiring of new workers. The shipbuilder’s goal is to maintain an average of 8 years of experience for workers in core trades, such as welding. However, the shipbuilder’s projections show that the new workforce ramp-up at the Groton facility will reduce workers’ average experience from 13.1 years to a low of 5.6 years in 2028—just after the shipbuilder plans to deliver the lead Columbia class submarine. If workforce growth or efficiency assumptions are not met, the shipbuilder may resort to scheduling overtime work or outsourcing some activities to meet the program’s construction schedule, which would have cost impacts for the program. The Navy’s procurement cost estimate of $115 billion to construct Columbia class submarines is not reliable because it does not reflect likely program costs and risks. We assessed the Columbia class cost estimate by comparing it with the best practices identified in GAO’s Cost Estimating and Assessment Guide. We found that it substantially met the criteria for the comprehensive characteristic of a reliable cost estimate, and partially met the criteria for the remaining characteristics, including accurate and credible. In particular, we found that the cost estimate does not accurately reflect program costs because it is based on overly optimistic labor hour assumptions, and is not fully credible because while the Navy conducted risk and sensitivity analyses to test the likelihood of achieving its assumptions, it selected a specific cost estimate that informs the program’s budget which does not include any margin in case those assumptions are not achieved. In addition, the cost estimates and assessments conducted by other entities produced a range of results, indicating that there is a high degree of uncertainty regarding program costs. See appendix III for the full results of our assessment of the Navy’s cost estimate. Navy officials stated they plan to update the Columbia class cost estimate in support of DOD’s decision to authorize construction of the lead submarine and this decision is expected to occur in summer 2020. Navy officials also stated that they expect that the cost estimate will be complete by the end of fiscal year 2019, followed by an independent cost assessment to support the authorization decision. However, this timeframe does not provide assurance that both the update and the independent assessment will be complete before the Navy requests funding from Congress for lead submarine construction, as part of its fiscal year 2021 budget request, which could be submitted as early as February 2020. If so, decision makers may be basing their decisions on outdated or incomplete information. The Columbia class cost estimate relies on optimistic program assumptions and does not reflect the likely labor hour costs that the Navy will incur to construct the submarines. As part of our assessment of the Columbia program cost estimate, we found that it did not fully meet the best practices for an accurate estimate. A cost estimate is considered accurate when it is based on an assessment of the most likely costs—that is, it is neither overly conservative nor overly optimistic. The Navy estimates that it will need $115 billion to design and construct 12 submarines and NAVSEA cost estimators identified labor costs as a primary source of cost risk. As discussed below, if the program’s optimistic assumptions are not realized, the program may require more funding than originally planned to construct the Columbia class. The Navy anticipates that it will need 12 million labor hours to directly construct the lead submarine—referred to as touch labor. This represents 17 percent fewer labor hours than what was needed for the lead Virginia class submarine, when adjusted for weight differences. To develop this estimate, NAVSEA estimators relied heavily on historical touch labor hour data from the construction of the lead Virginia class submarine and cost data from the Ohio class submarine program for unique ballistic submarine components, such as missiles. NAVSEA estimators took the following steps to develop the Columbia lead submarine estimate: In general, heavier ships cost more to construct, so NAVSEA cost estimators calculated a weight-adjusted estimate based on Virginia class labor hours to account for the heavier weight of the Columbia class. This resulted in an initial estimate of 14.5 million touch labor hours for the lead submarine. NAVSEA cost estimators then made numerous adjustments in the cost estimate that reduced the expected number of labor hours based on multiple assumptions that differences in the design and construction process would lead to more efficient construction of Columbia class submarines than previous submarine classes. These adjustments subsequently decreased the estimate to 12 million touch labor hours for the lead submarine. NAVSEA cost estimators then used the lead Columbia submarine estimate as the basis to calculate labor hours for follow-on submarines, estimating an average of 8.9 million touch labor hours. Figure 11 illustrates NAVSEA’s touch labor hour calculation for the lead submarine. However, the touch labor hour estimate is overly optimistic—with assumptions on construction efficiencies that are either unsubstantiated or unprecedented compared to Virginia class and other shipbuilding historical data. Compared to the Navy’s estimate, Columbia’s estimated touch labor hours, as calculated by other organizations, are more conservative. For example, CBO questioned the Navy’s assumption that ballistic submarines are less expensive to build than attack submarines, after accounting for weight differences and estimated that for the overall class, including the lead and follow-on submarines, the Navy would more likely realize an 8 percent reduction rather than the 19 percent reduction estimated by the Navy. While the shipbuilder will likely realize some efficiencies from initiatives to improve design and construction processes, our analysis of the Navy’s assumptions used to develop the cost estimate indicates that they likely overstate the labor hour reduction the shipbuilder can realistically achieve. These assumptions include that the program (1) achieves its design goals at the start of construction; (2) is constructed more efficiently than Virginia class submarines; and (3) successively reduces the number of hours needed to construct follow-on submarines. If these assumptions are not realized, overall program costs could be higher than the Navy’s procurement estimate of $115 billion. Navy officials stated that they believe that these assumptions are valid and that the cost estimate is achievable. However, our assessment indicates that the assumptions for the cost estimate are overly optimistic, as discussed below. The Navy’s cost estimate does not reflect the risk that the shipbuilder may not achieve its planned design completion goals. As we reported above, design performance to date has slowed and the shipbuilder has had to hire additional designers in an effort to mature its design on schedule. NAVSEA cost estimators stated that they recognize that an incomplete design at the start of ship construction was a significant driver of cost growth on other shipbuilding programs. For the Columbia class, NAVSEA cost estimators assumed that achieving the design maturity goal would eliminate 2 million labor hours by reducing costs associated with rework and out of sequence work. In October 2018, NCCA officials stated that they recently reviewed shipbuilder data and the expected design completion at construction start continues to range between 55 and 75 percent complete—the same range that they estimated in their independent assessment. While this lower rate would be an improvement over the Virginia class program, it would still fall short of the 84 percent assumption built into the cost estimate. If the shipbuilder does not complete the design at its planned rate and begins construction with a less mature design, it may need additional labor hours to construct the ship, resulting in increased program costs. The Navy’s cost estimate includes assumptions that reduce Columbia’s estimated touch labor hours due to efficiencies from constructing Columbia and Virginia class submarines concurrently, an assumption with which the shipbuilder does not agree. NAVSEA cost estimators calculated a 1.1 million-labor hour reduction, attributing the decrease to efficiencies gained from constructing multiple submarines at the same time, basing their assessment on shipbuilder estimates of the Virginia class. However, it is unclear how increased shipyard production would result in fewer labor hours to construct each submarine. Shipbuilder representatives stated that rather than a reduction in touch labor hours, they expect to realize efficiencies from increased production primarily from reduced overhead rates and material costs. Further, the Navy’s independent assessment analyzed labor hour data for Virginia class construction and found that there was no correlation between the number of submarines constructed at a time and the total number of labor hours. However, increasing shipyard production to include both Virginia and Columbia class construction may increase schedule risk for the shipbuilder, which could result in additional costs if the shipbuilder does not achieve planned increases in its workforce and facility upgrades. When the number of Virginia class submarines under construction increased, both shipyards experienced inefficiencies due to poorly planned ramp-up requirements. In addition, DOD officials stated that problems encountered on one program could affect the other as the shipbuilder is relying on the same workforce and vendor base for both programs. The Navy’s cost estimate also assumed construction efficiencies— because the Columbia class submarine will be less dense than the Virginia class submarine—another assumption with which the shipbuilder does not agree. Navy officials stated that less dense submarines are less costly to construct as the additional space within the hull allows for faster and more efficient work. However, the shipbuilder conducted analysis to compare the density of various areas of the Virginia class and Columbia class submarines and found that areas had very similar density. Specifically, there was only a 1 percent and 3 percent difference, between the forward compartments and aft compartments, respectively—some of the more complex areas of the submarine. If the shipbuilder does not realize these construction efficiencies, more total labor hours would be required to construct the submarine, resulting in increased cost. The Navy’s cost estimate assumes that the costs for follow-on Columbia class submarines will decrease at a rate that may overstate the improvements the shipbuilder can realistically achieve. The Navy expects the number of labor hours to construct Columbia class follow-on submarines to decrease based on an assumed learning curve rate. Learning occurs when construction is consistent and continuous and the shipbuilder learns how to do repetitive tasks more efficiently. The decrease in the number of expected labor hours is expressed as a learning curve rate, where a lower percentage indicates that less labor is required for follow-on units. NAVSEA cost estimators calculated a learning curve of 88.9 percent for Columbia class submarines. A learning curve indicates that as the number of units doubles, unit cost decreases by a constant percentage. In this case, the cost estimate assumed that the fourth submarine would require only 88.9 percent the amount of labor to build the second submarine. NAVSEA’s assumption may overstate the potential learning rate that Columbia can expect to achieve. The first four Virginia class submarines, hull numbers SSN 774 through 777, incorporated modular construction techniques where submarines were built in 10 modules. The next six Virginia class submarines, hull numbers SSN 778 through 783, were constructed using four modules. As a result of the improvements in the modular construction process, construction across the first ten submarines was not consistent, which is a condition that is necessary to determine the learning curve rate. Therefore, there is no way to determine what share of the labor hour reduction on later submarines was due to learning or process improvements. Rather, SSN 778, the first Virginia class submarine to use the four modular construction approach is a better starting point to determine the shipbuilder’s capacity for learning. The Navy’s independent assessment included a separate learning curve analysis for Virginia class submarine hulls SSN 778 through 791 and calculated a potential learning curve rate of 93.9 percent. A learning curve assumption applies to all follow-on submarines and has a cumulative effect on the number of labor hours and, ultimately, the cost of these submarines. In the case of the Columbia program, the rate will apply to the second through twelfth submarines. Figure 12 shows how the difference in the learning curve rate can affect the estimated labor hours for follow-on submarines. Therefore, a small change in the assumed learning curve rate can have a significant effect on the cost estimate for follow-on submarines. For example, the Navy’s independent assessment of the cost estimate calculated that production costs could increase by $3.59 billion in constant year 2010 dollars if a learning curve of 93.9 percent was realized, rather than the 88.9 percent rate estimate. Our previous work on Navy shipbuilding performance has shown that the Navy has consistently underestimated the costs for follow-on ships, with costs for Virginia class submarines underestimated by close to 40 percent. The Columbia program cost estimate did not fully meet the best practice criteria to be considered credible because, in part, Navy program management did not sufficiently account for program risks when it selected the final estimate. To determine the estimate’s credibility, we examined the extent to which NAVSEA cost estimators tested, among other things, the sensitivity of key cost elements such as labor hours and conducted uncertainty analyses to quantify risks; and an independent cost estimate and assessment were conducted by groups outside the acquiring organization (specifically, CAPE and NCCA) to determine whether other estimating methods produced similar results. We found that while the Navy program management’s $115 billion procurement cost estimate for the Columbia class is overly optimistic in some of its assumptions, the estimate does not reflect any contingency to offset the likely effects of not meeting the assumptions, which is a best practice. In addition, the independent cost estimates and assessments conducted by other organizations had varying results, indicating the high level of uncertainty regarding Columbia program costs. We further address these issues below. Navy leadership’s decision to select $115 billion as the program cost estimate means that there is no margin in the program budget to cover likely program costs if risks are realized. The best practices identified in GAO’s cost estimating guide state that the results of a risk analysis should be used to select a cost estimate that is sufficient to manage program risks. NAVSEA cost estimators conducted a risk analysis to identify and quantify program risks, and determined the effects of changing key cost driver assumptions and factors—important steps in creating a high quality estimate. However, while NAVSEA cost estimators identified 54 risk parameters for construction costs, we found that some of the inputs for these ranges resulted in a cost estimate that understates the potential impact of program cost risks. For example, the risk ranges do not sufficiently account for the issues we identified above, including that increased shipyard construction could result in similar inefficiencies that occurred in the production of the Virginia class, requiring more labor hours than estimated; and shipbuilder workforce ramp-up could result in decreased efficiency and quality due to the influx of new workers even greater than the issues observed on the Virginia class when shipyard construction increased. For other risk parameters, such as cost of material provided by the shipbuilder, the cost estimate documentation was not sufficient for us to analyze whether the risk ranges included in the estimate were reasonable (i.e., not overly optimistic or pessimistic). As a result, we could not determine whether the risk analysis sufficiently captures the risk of program cost growth, or what the probability is of achieving the $115 billion procurement cost estimate. Further, Columbia’s program management and the milestone decision authority selected $115 billion as the program’s procurement cost estimate, without adjusting for the likelihood of cost growth in the design or construction of Columbia class submarines identified in the risk analysis. As we reported in December 2017, the risk analysis developed by NAVSEA indicated that there is only a 45 percent probability that the overall program cost estimate will be sufficient to cover program costs. The cost estimating best practices identified in our cost estimating guide state that a risk-adjusted cost estimate helps ensure that sufficient funding will be available for the expected program costs. Additionally, a risk-adjusted cost estimate is consistent with federal internal control standards, which indicate that risk mitigation efforts should be selected to sufficiently respond to risks. However, Columbia program officials stated that they believe program risks can be managed within the current cost estimate—which they consider to be conservative—as it does not account for all of the program’s potential cost savings. Specifically, the Navy anticipates that the program will realize up to $1.9 billion in additional cost savings from use of authorities associated with the National Sea-Based Deterrence Fund (the Fund), such as the authority to purchase components for multiple submarines—which we discuss later in this report. As a result, the program office estimate represents the program manager’s cost goal for the Columbia program, rather than the risk- adjusted estimate. Even if the Navy were to achieve the full anticipated $1.9 billion savings, these savings represent only 1.5 percent of program costs. Such cost savings are unlikely to cover program cost overruns for a high-risk program, such as Columbia, given that historically shipbuilding programs experience 27 percent cost growth. As the current estimate does not include any reserves for cost overruns, program management is relying on these potential savings to help mitigate likely cost growth. Several entities have conducted independent reviews of the Columbia program cost estimate, with varying results. CAPE conducted an independent cost estimate and NCCA conducted an independent cost assessment of the program estimate in support of the Columbia class program’s Milestone B review. CAPE’s independent cost estimate was 3 percent lower than the Navy’s service cost position, which it stated was due to CAPE’s use of lower shipyard labor rates. However, NCCA’s assessment did not produce similar results as the program cost estimate and concluded that the program is at risk of up to $6.14 billion in cost growth. The program manager reviewed the recommendations in the independent cost assessment and determined that the program office estimate appropriately weighs program risks. Navy leadership selected the program office estimate to serve as the Navy’s service cost position because program officials stated that they believe program risks can be managed within the program cost estimate. CBO also conducted a cost estimate and projected that procurement of 12 submarines would be 6 percent higher than the program estimated. The results of these cost estimates and NCCA’s assessment are summarized in table 3. As part of the Milestone B review, the milestone decision authority reviewed the service cost position and CAPE’s independent cost estimate. The independent cost assessment was reviewed by Navy leadership as part of the service cost position process and, therefore, was not briefed as part of the milestone review. The milestone decision authority accepted the Navy service cost position and directed the Navy to use this estimate as the basis of its fiscal year 2018 budget request. It also established an $8 billion affordability cap for the average procurement cost of all 12 submarines to control future program costs. Navy officials stated that they plan to update the cost estimate for the lead submarine in support of a planned Defense Acquisition Board review, in the third quarter of fiscal year 2020. At that point, the Navy will be seeking approval from the milestone decision authority to award the contract for construction of the lead submarine. However, the Navy and DOD’s general timeframes do not provide assurance that the planned update of the cost estimate would be completed prior to the fiscal year 2021 budget request, which will include funding for lead submarine construction, as shown in figure 13 below. The milestone decision authority has directed CAPE, with assistance from NCCA, to assess the lead submarine cost estimate to support the decision to authorize the Navy to award the contract for lead submarine construction. Since this assessment will occur after the Navy has updated the lead submarine cost estimate, it is even less likely that the program budget request will reflect the results from the independent cost assessment. Additionally, the current program cost estimate the Navy developed for the Milestone B review does not reflect the program’s current strategy to use authorities associated with the Fund to achieve cost savings, as discussed further below. The best practices identified in GAO’s cost estimating guide state that cost estimates should be regularly updated and reflect the program acquisition baseline. Updating the cost estimate and risk analysis to include these anticipated savings and current program data would improve its reliability and help ensure that budget requests are sufficient to execute the Columbia program as planned. After we provided our draft report to DOD for comment, Navy officials briefed us on the changes they had made to the program’s estimate to date, stating that they updated the cost risk analysis as part of an internal program review. While the Navy plans to update the lead submarine cost estimate again by the end of fiscal year 2019 to support the Defense Acquisition Board review in the summer of 2020, it has yet to provide specific details on the steps it will take to update this estimate to ensure that it would include likely program costs and risks, such as the cost data it plans to include or the assumptions it may reassess. Further, since the Navy will likely submit its budget request to Congress as early as February 2020, Congress may be asked to authorize and fund lead submarine construction without the benefit of any changes to the estimate that may occur as a result of recommendations stemming from an independent review of the update. Further, although the Navy reports Columbia program cost information to Congress through annual matrices submissions, updates to the program cost estimate will not be reflected in these reports. For example, the Navy plans to report program manager and contractor cost estimates for individual submarines in the matrices once the submarines are under construction. Since these estimates are based on shipbuilder contract performance, they are initially calculated only after construction of each submarine is 15 percent complete, when sufficient data are available to show performance trends. While the Navy plans to award the contract for the lead submarine in October 2020, limited contractor performance data will be available in time for the February 2021 matrix submission. As a result, the earliest opportunity to report on the cost of the lead submarine would be the Navy’s next submission in February 2022, at which point the Navy will have already requested funding for the second and third Columbia submarine. In 2014, Congress created a National Sea-Based Deterrence Fund (the Fund) that provides DOD with greater discretion to fund the design, construction, purchase, alteration, and conversion of the Columbia class. Since then, Congress has provided the Navy with enhanced acquisition authorities to buy and construct submarines and certain key components early, in bulk, and continuously, when using these funds. The Navy anticipates saving over $1.9 billion through use of these authorities, but these savings, which were not included in the Columbia class program’s cost estimate, may be overestimated. Since its inception in 2014, Congress has expanded the special acquisition authorities under the Fund, in part, to allow the Columbia class program to gain economic efficiencies and realize cost savings. The timeline of the establishment of the Fund and legislative changes are shown in figure 14. The following authorities have been included as part of the use of the Fund: Economic order quantity: Permits awarding of contracts that provide a quantity of supplies that will result in a total cost and unit cost most advantageous to the government by achieving economic efficiencies based on production economies. Advance construction: Allows for manufacturing and fabrication efforts prior to ship authorization. Multiyear procurement authority: Permits a single contract for more than one year of critical components. Incremental funding authority: Facilitates the purchase of long lead items through partial funding of a contract with the expectation that full funding will be provided later. Using the Fund’s associated authorities, the Navy is able to purchase significant components and start advance construction prior to receiving Congress’s authorization of and funding to purchase each submarine. In total, the Navy will have requested and received $8.6 billion in funding, including 33 percent of funding for the lead submarine, before it receives authorization and funding to begin construction of the lead submarine in October 2020. At that point, the Navy will also have already requested funding for the propulsor and advance construction for the second submarine. Under law, the Navy is required to deposit all appropriations for the Columbia class construction and design into the Fund. To date, the Navy has made three deposits from the Shipbuilding and Conversion, Navy account into the Fund, totaling over $1.6 billion. The Navy is using initial deposits of $773 million in fiscal year 2017 and $862 million in fiscal year 2018 for detail design and continuous production of missile tube components. The Navy Comptroller initiates all deposits into the Fund, which are approved by the DOD Comptroller as internal reprogramming actions, as shown in figure 15. The Navy anticipates achieving over $1.9 billion in savings through the use of the Fund’s associated authorities, but the Navy did not evaluate these savings when it developed the program office cost estimate. Table 4 provides a description of each authority and the Navy’s plans and estimated potential savings resulting from use of the authorities. Overall, while we were unable to fully assess the methodology and assumptions the Navy used to estimate anticipated savings, the information we reviewed indicated that the Navy may have overestimated some of the savings the program can realistically achieve through use of the Fund’s associated authorities. While the Navy provided some documentation of the cost estimate methodologies, we could not fully validate that the estimated savings were realistic because, in general, the documentation provided by the Navy did not include a detailed description of how the estimates were calculated or how historical data were used to develop the estimate—a best practice identified in GAO’s cost estimating guide. In some cases, such as for individual critical components, the total value of the component costs was not documented. For other savings, such as advance construction, the Navy could not provide documentation of the calculations or a rationale for the estimated savings. In addition, the Navy assumes a higher rate for Columbia multiyear procurement savings than what has been typically achieved for other programs. The Navy has generally used multiyear procurement contracts after production has begun and some units have already been purchased. For example, according to the Navy, it did not receive multiyear procurement authority for the DDG 51 Arleigh Burke-class destroyer program—until 1998—more than 10 years after the contract for the lead ship was awarded and 38 ships had been purchased. We have reported that DOD typically overestimates savings from multiyear procurement authority. Further, in a 2017 presentation to Congress, the Navy stated that multiyear procurement savings are historically 10 to 12 percent. When the Navy requested multiyear procurement authority for the DDG 51 program in fiscal year 2013, it estimated achieving a savings of 8.7 percent. Similarly, when planning material purchases for the Virginia class submarine, the shipbuilder estimated that it would achieve 10 to 15 percent savings through the use of multiyear procurement authority. However, the Navy estimates that the Columbia class program will realize savings of 15 to 20 percent using multiyear procurement authority. A realistic estimate of savings is essential because program management is essentially relying on these savings as the only cost reserve to address any issues that arise during design and construction of the submarines. Updating the cost estimate to reflect these savings will provide program management with a more realistic assessment of the margin available and resources needed to achieve their costs. The Columbia class program is driven by the continued and pressing need to meet the Navy’s nuclear deterrent requirements as the legacy submarine fleet cannot extend its life any longer. From the outset this has translated into an aggressive and concurrent schedule for lead submarine construction. To counterbalance this schedule risk, the program plans to complete a substantial amount of the design before starting construction, which may prove challenging as the shipbuilder must complete an increasingly higher volume and complexity of disclosures. This, coupled with failures in missile tubes already delivered to the shipyard, highlight the potential for management challenges ahead. This is not to suggest that in a program of this size and complexity that some issues are not to be expected. Rather, the challenge for the Columbia class program is that the Navy has a limited ability to slow the pace of the program given the mission imperatives. At present, the need for additional resources appears likely because the Navy’s margin to mitigate any cost growth from issues that develop during design and construction relies on overestimated savings from use of the Fund’s associated authorities. The steps that the Navy takes between now and the fiscal year 2021 budget request to understand and plan for likely program costs will determine whether sufficient funding is in place to cover potential cost growth. The Navy plans to update the lead submarine cost estimate to reflect its current acquisition strategy and, in doing so, the Navy has the opportunity to incorporate more realistic information into the risk analysis and lead submarine cost estimate. In addition, a realistic and well-documented estimate of savings from use of the Fund’s associated authorities would help ensure that the Navy has allocated the necessary resources to address any issues that emerge during design or construction of the lead submarine. Such steps will likely improve the reliability of the lead submarine cost estimate and would position the Navy to better align its fiscal year 2021 budget request with funding it will likely need to construct the lead submarine—the next key decision point in the Columbia class program. Without an updated cost estimate with more realistic assumptions, Congress will be asked to commit billions of dollars for the lead submarine without knowing the full potential cost of construction and the possible effect on other shipbuilding programs. We are making three recommendations to the Secretary of the Navy: The Secretary of the Navy should direct NAVSEA to incorporate current cost and program data and an updated cost risk analysis in its planned update of the Columbia class lead submarine cost estimate. (Recommendation 1) The Secretary of the Navy should direct NAVSEA to develop a realistic and well-documented estimate of savings from use of the authorities associated with the Fund and incorporate the savings associated with the lead submarine into the Columbia lead submarine cost estimate. (Recommendation 2) The Secretary of the Navy should direct the Columbia class program office to update the lead submarine cost estimate and cost risk analysis prior to requesting funds for lead submarine construction. (Recommendation 3) We provided a draft of the sensitive report to DOD for comment. DOD’s written comments on the sensitive report are reprinted in appendix IV and summarized below. DOD concurred and described the actions they have taken or plan to take in response to all three of our recommendations. Regarding our recommendations to update its cost estimate update prior to requesting funds for lead submarine construction, the Navy has stated that it incorporated current cost and program data and an updated risk analysis into its cost estimate for the lead submarine in 2018, as part of an annual review. The Navy also stated that it will continue to update the lead submarine cost estimate with current data prior to requesting funding for lead submarine construction in fiscal year 2021. Until the updated estimate is independently validated—an essential cost estimating step— we cannot determine that the updated estimate is credible. Further, in response to our recommendation regarding the development of a realistic and well-documented estimate of savings from use of the Fund’s associated authorities, the Navy stated that it incorporated savings in its updated cost estimate. However, it has not provided any additional evidence to demonstrate that estimated savings from use of the Fund’s associated authorities are realistic and well-documented. Based on documentation that the Navy provided to us, it did not include a detailed description of how the estimates were calculated or how historical data were used to develop the estimate. Until these estimates are independently validated, the Navy cannot be confident that the program will achieve the planned amount of savings. The Navy also provided technical comments, which we incorporated as appropriate. DOD also raised a number of issues related to our assessment of the cost estimate, advance construction, and technology development, which we address in appendix IV. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Defense, the Secretary of the Navy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. Should you or your staff have questions, please contact me at (202) 512- 4841 or oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report evaluates the Navy’s Columbia class submarine program. Specifically, we assessed (1) the Navy’s progress and challenges, if any, associated with meeting design goals and preparing for lead submarine construction; (2) the reliability of the Navy’s cost estimate for the Columbia class submarine program; and (3) how the Navy is implementing the National Sea-Based Deterrence Fund (the Fund) and associated authorities to construct Columbia class submarines. This report is a public version of a sensitive report that we issued in March 2019. The Department of Defense (DOD) deemed some of the information in our March report to be sensitive, which must be protected from public disclosure. Therefore, this report omits sensitive information about the Navy’s development of critical technologies for the Columbia class program, including specific details about the technologies. Although the information provided in this report is more limited, the report addresses the same objectives as the sensitive report and uses the same methodology. To assess the Navy’s progress and what challenges, if any, are associated with meeting design goals and preparing for lead submarine construction, we reviewed Navy and shipbuilder documents, including program briefings, schedules, and contract status reports to assess the schedule and performance risks of the Columbia class program. To evaluate the shipbuilder’s progress in maturing the Columbia class design, we reviewed the Navy’s plans for design management and completion, evaluated the shipbuilder’s design schedule, and compared them against design progress reports to identify any delays. To evaluate the Navy’s plans for advance construction, we analyzed metrics reported in Navy and shipbuilder documents, briefing slides, and other documentation including key dates and estimated construction plans. We compared design knowledge on the Columbia class program to our prior work on shipbuilding best practices. We reviewed ongoing development efforts and schedules for the Columbia class program’s critical technologies to determine remaining risks to their development and integration. We also reviewed the matrices submitted by the Navy to Congress in February and October 2018, to determine the status of the program and identify any changes to the Navy’s design and construction goals for the program since our last report in December 2017. We also analyzed available documentation related to the status of the nuclear reactor and integrated power system. We reviewed the shipbuilder’s construction plans for its new facilities and its workforce hiring plans. We also reviewed the shipbuilder’s and Navy’s process for evaluating its suppliers. To corroborate documentary evidence and gather additional information in support of our review, we met with officials from the Navy’s Columbia class submarine program office; Naval Nuclear Propulsion Directorate; Naval Surface Warfare Center Philadelphia; Office of the Chief of Naval Operations; Supervisor of Shipbuilding, Groton; the Office of the Deputy Assistant Secretary of Defense for Systems Engineering; and the Office of Undersecretary of Defense for Acquisition and Sustainment. Additionally, we met with shipbuilding representatives from General Dynamics Electric Boat—the prime contractor—as well as their main subcontractor, Huntington Ingalls Industries Newport News Shipbuilding to understand their role in Columbia class design and construction. To assess the reliability of the Navy’s cost estimate for the Columbia class submarine program, we determined the extent to which the estimate met best practices as identified in GAO’s Cost Estimating and Assessment Guide. We examined cost estimate documentation, such as the Columbia class program life-cycle cost estimate, briefs, memoranda, and other documents that contain cost, schedule, and risk information. We also examined the independent cost estimate conducted by the Office of the Secretary of Defense’s Office of Cost Assessment and Program Evaluation (CAPE), the independent cost assessment conducted by the Naval Center for Cost Analysis (NCCA), and the cost estimate conducted by the Congressional Budget Office, to determine what methodologies and assumptions differed from the program cost estimate. We met with Navy officials who were responsible for developing the cost estimate to understand the processes used by the cost estimators, to clarify information, and to allow the Navy to provide additional documentation on the data and methodologies used in the estimate. We also observed portions of the Columbia class program’s cost model during a presentation and discussion with Navy cost estimators. We also reviewed the matrices submitted by the Navy to Congress to identify any changes to the Navy’s cost goals and reported information. To further corroborate documentary evidence and gather additional information in support of our review, we conducted interviews with relevant DOD and Navy officials responsible for developing, updating, and assessing the Columbia class program cost estimate, including CAPE; NCCA; the Naval Sea Systems Command’s (NAVSEA) Cost Engineering and Industrial Analysis Group; and the Columbia class program office. To evaluate how the Navy is implementing the Fund and associated authorities to construct Columbia class submarines, we reviewed the legislation establishing and modifying the Fund, program budget request documents, and DOD reprogramming approvals. We also reviewed the Navy’s basis of estimate for the savings it plans to achieve from these authorities. To further corroborate documentary evidence and gather additional information in support of our review, we met with officials from the Office of the Assistant Secretary of the Navy for Financial Management and Comptroller; Office of the Under Secretary of Defense (Comptroller); and the Columbia class program office to discuss the Navy’s plans to use and execute the Fund and DOD’s role in approving transfers into the Fund. The performance audit upon which this report is based was conducted from December 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with DOD from February 2019 to April 2019 to prepare this unclassified version of the original sensitive report for public release. This public version was also prepared in accordance with these standards. Appendix II: Technology Readiness Levels Description Lowest level of technology readiness. Scientific research begins to be translated into applied research and development. Examples might include paper studies of a technology’s basic properties. Invention begins. Once basic principles are observed, practical applications can be invented. The application is speculative and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies. Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative. Basic technological components are integrated to establish that the pieces will work together. This is relatively “low fidelity” compared to the eventual system. Examples include integration of “ad hoc” hardware in a laboratory. Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in a simulated environment. Examples include “high-fidelity” laboratory integration of components. Representative model or prototype system, which is well beyond the breadboard tested for TRL 5, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high-fidelity laboratory environment or in a simulated realistic environment. Prototype near or at planned operational system. Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in a realistic environment, such as an aircraft, vehicle, or space. Examples include testing the prototype in a test bed aircraft. Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of the true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications. Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluations. In almost all cases, this is the end of the last “bug fixing” aspects of true system development. Examples include using the system under operational mission conditions. To assess the reliability of the Navy’s cost estimate, we determined the extent to which the estimate was consistent with cost estimating best practices as identified in GAO’s Cost Estimating and Assessment Guide. This guide groups the best practices into four general characteristics: well documented, comprehensive, accurate, and credible. We reviewed documentation the Navy submitted for its cost estimate including limited portions of the Navy’s cost model, conducted numerous interviews, and reviewed relevant sources. We determined that the Columbia class cost estimate substantially met one, and partially met three of the four characteristics of a reliable cost estimate, shown in figure 16. We determined the overall assessment rating by assigning each individual rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3, Substantially Met = 4, and Met = 5. Then, we calculated the average of the individual assessment ratings to determine the overall rating for each of the four characteristics as follows: Not Met = 1.0 to 1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4, Substantially Met = 3.5 to 4.4, and Met = 4.5 to 5.0. We consider a cost estimate to be reliable if the overall assessment ratings for each of the four characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and is not considered reliable. In addition to responding to our recommendations, DOD also provided observations on a number of issues related to our assessment of the cost estimate, advance construction, and technology development. Our response to DOD’s observations is as follows. In paragraph 4, page 1 of the letter above, the Navy did not agree with our assessment of the accuracy of the cost estimate and stated that the life cycle cost estimate includes accurate calculations, proper inflation tables, and updates to requirements. DOD also stated that GAO or other stakeholders did not identify any errors. This is incorrect. While the Navy allowed us to observe the model, we did not independently check the accuracy of the calculations because Navy officials stated that the cost model, which contains the cost calculations, could not be released. We informed the Navy that this would affect parts of our assessment. After we provided a draft of the report, the Navy provided a briefing summarizing the results of a program office cost checkpoint conducted in September 2018. At the briefing, we received information on updates that the Navy made to the program cost estimate. As a result, we updated our assessment to reflect that the Navy substantially met the best practice to regularly update the cost estimate to reflect significant changes. However, the additional information provided by the Navy did not change our assessment of the accuracy and, therefore, our overall assessment of the Columbia cost estimate remains valid. In paragraph 1, page 2, the Navy did not agree with our assessment of the credibility of the cost estimate and stated that the life cycle cost estimate includes analyses that address sensitivity, risks, and uncertainty within the estimate. As we point out in the report, the estimate is based, in part, on optimistic assumptions regarding the number of labor hours needed to construct Columbia class submarines. The Navy has made updates to the program cost estimate based on a 2018 checkpoint review and stated that the cost risk analysis has been updated and program costs are less than originally estimated. The Navy provided us with a high-level brief of these updates. However, due to the timing of this report, we were not able to fully assess the update to the cost model. Given the size and complexity of the Columbia class program, we continue to believe that the program’s cost estimate does not adequately account for program risks. In paragraph 3, page 1, DOD stated that our findings were largely informed by an assessment conducted by the Naval Center for Cost Analysis (NCCA). However, our process for assessing program cost estimates is based on the extent to which the estimate met best practices outlined in GAO’s Cost Estimating and Assessment Guide. In conducting our assessment, we examined multiple sources of information, including the Columbia class program life cycle cost estimate, NCCA’s independent cost assessment, DOD’s Office for Cost Assessment and Program Evaluation’s (CAPE) independent cost estimate, and the cost estimate conducted by the Congressional Budget Office (CBO), to determine what methodologies and assumptions differed from the program cost estimate. We also relied on prior experience examining and reporting on the cost performance of Navy shipbuilding programs, issuing 26 reports over the past 10 years. We found, for example, that the cost estimate is based on optimistic labor assumptions which, while in agreement with NCCA’s assessment and CBO’s estimate, results from our independent assessment of the evidence we reviewed and on our prior work. In paragraph 2, page 2, the Navy stated that it identified super modules and selected components where acceleration would reduce construction schedule risk. We acknowledge in the report that the design for these components will be complete prior to starting construction. However, we continue to believe that starting construction for components of the lead submarine before the arrangements for the submarine are complete increases design and construction risk. Even if the components included in advance construction are fully designed, risk remains for the adjoining and interfacing components within the module that may have ongoing design work, potentially requiring costly and time-intensive rework. In paragraph 4, page 2, the Navy notes that fully maturing all of the key technologies identified in our 2017 report—such as the advanced propulsor bearing—would require substantial investments in money and time. However, we continue to reinforce that a tenet of achieving design maturity is based on demonstrating a prototype in its final form, fit, and function in a realistic environment—which requires a design resembling the final configuration. In paragraph 6, page 2, the Navy stated that it does not agree with our characterization that the Navy is continuing to experience manufacturing problems with the electric drive of the integrated power system. DOD stated that while the vendor experienced delays in manufacturing the prototype motor, it has taken proactive measures to deliver the motor to the shipyard, as scheduled. However, the Navy’s plan to concurrently test and finalize the design increases risk that any issues identified in testing could delay the delivery of the system to the shipyard. As a result, we continue to identify this as a key risk to the program. Additional details on this system are classified. Shelby S. Oakley, (202) 512-4841 or oakleys@gao.gov. In addition to the contact above, the following staff members made key contributions to this report: Diana Moldafsky, Assistant Director; Laura Jezewski; Jessica Karnis; and Nathaniel Vaught. Other contributions were made by Brian Bothwell; Daniel Glickstein; Kurt Gurka; Stephanie Gustafson; and Robin Wilson.", "summary": "The Navy has identified the Columbia class submarine program as its top acquisition priority. It plans to invest over $100 billion to develop and purchase 12 nuclear-powered ballistic missile submarines to replace aging Ohio class submarines by 2031. The National Defense Authorization Act for Fiscal Year 2018 and House Report 115-200 included provisions that GAO review the status of the program. This report examines (1) the Navy's progress and challenges, if any, in meeting design goals and preparing for lead submarine construction; (2) the reliability of the Navy's cost estimate; and (3) how the Navy is implementing a special fund and associated authorities to construct Columbia class submarines. GAO reviewed Navy and shipbuilder progress reports, program schedules, and construction plans. GAO assessed the Navy's cost estimate and compared it to best practices for cost estimating. GAO also reviewed certain Navy funding and acquisition authorities and interviewed program officials. This is a public version of a sensitive report that GAO issued in March 2019. Information that the Department of Defense (DOD) deemed sensitive has been omitted. The Navy's goal is to complete a significant amount of the Columbia class submarine's design—83 percent—before lead submarine construction begins in October 2020. The Navy established this goal based on lessons learned from another submarine program in an effort to help mitigate its aggressive construction schedule. Achieving this goal may prove to be challenging as the shipbuilder has to use a new design tool to complete an increasingly higher volume of complex design products (see figure). The shipbuilder has hired additional designers to improve its design progress. The Navy also plans to start advance construction of components in each major section of the submarine, beginning in fiscal year 2019, when less of the design will be complete. The Navy's $115 billion procurement cost estimate is not reliable partly because it is based on overly optimistic assumptions about the labor hours needed to construct the submarines. While the Navy analyzed cost risks, it did not include margin in its estimate for likely cost overruns. The Navy told us it will continue to update its lead submarine cost estimate, but an independent assessment of the estimate may not be complete in time to inform the Navy's 2021 budget request to Congress to purchase the lead submarine. Without these reviews, the cost estimate—and, consequently, the budget—may be unrealistic. A reliable cost estimate is especially important for a program of this size and complexity to help ensure that its budget is sufficient to execute the program as planned. The Navy is using the congressionally-authorized National Sea-Based Deterrence Fund to construct the Columbia class. The Fund allows the Navy to purchase material and start construction early on multiple submarines prior to receiving congressional authorization and funding for submarine construction. The Navy anticipates achieving savings through use of the Fund, such as buying certain components early and in bulk, but did not include the savings in its cost estimate. The Navy may have overestimated its savings as higher than those historically achieved by other such programs. Without an updated cost estimate and cost risk analysis, including a realistic estimate of savings, the fiscal year 2021 budget request may not reflect funding needed to construct the submarine. GAO is making three recommendations: that the Navy update the lead submarine cost estimate with cost risk analysis using current cost data, develop a realistic estimate of savings from use of the Fund's authorities, and use this updated cost estimate to inform its budget request for lead submarine construction. DOD concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Federal land management agencies have law enforcement divisions that protect their employees and secure their facilities across nearly 700 million acres of federal lands (see fig. 1). To do so, the four agencies’ law enforcement divisions employ uniformed law enforcement officers who patrol federal lands, respond to illegal activities, conduct routine investigations, and, depending on the agency, may also provide expertise in assessing facilities’ security. Each agency also maintains a law enforcement data system in which law enforcement officers record and track incidents of suspected illegal activity on federal lands. These systems can be used in conducting investigations, identifying trends in crime data, and assisting with decision making regarding staffing, resource allocations, and budgetary needs. BLM. BLM’s Office of Law Enforcement and Security is charged with promoting the safety and security of employees and visitors, as well as environmental protection, across approximately 245 million acres of BLM lands in 12 states. At the end of fiscal year 2018, BLM had 194 field law enforcement officers engaged in such duties. According to agency documentation, these law enforcement officers also coordinate with state agencies and county law enforcement officers on large-scale recreational events, such as Burning Man. These field law enforcement officers may also be tasked with conducting facility security assessments. FWS. FWS’s division of Refuge Law Enforcement helps ensure the safety and security of visitors, employees, government property, and wildlife and their habitats on approximately 150 million acres of land. At the end of fiscal year 2018, FWS had 231 field law enforcement officers on the agency’s 567 wildlife refuges. According to agency documents, FWS law enforcement officers serve as ambassadors by providing important services to the public beyond law enforcement, such as providing visitors with information and guidance regarding fishing, hunting, hiking, and wildlife viewing opportunities. These field law enforcement officers may also be tasked with conducting facility security assessments. Forest Service. The Forest Service’s Law Enforcement and Investigations division is charged with protecting natural resources, employees, and visitors on approximately 193 million acres of National Forest System lands in 44 states. At the end of fiscal year 2018, the Forest Service had 417 field law enforcement officers. Additionally, law enforcement officers may be tasked with conducting facility security assessments. Park Service. The Park Service’s division of Law Enforcement, Security, and Emergency Services is charged with protecting resources, managing public use, and promoting public safety and visitor enjoyment across the agency’s 85 million acres, 418 park units, 23 national scenic and national historic trails, and 60 wild and scenic rivers. At the end of fiscal year 2018, the Park Service had 1,329 field law enforcement officers stationed at 240 of the Park Service’s units. Field law enforcement officers may also be tasked with conducting facility security assessments. The ISC Standard applies to all facilities in the United States occupied by federal employees for nonmilitary purposes, including federal land management agencies’ facilities. This includes existing facilities, new construction, or major modernizations; facilities owned, to be purchased, or leased; stand-alone facilities; special-use facilities; and facilities on federal campuses. Among other things, the ISC Standard requires agencies to assess the risks faced by each of their facilities. According to Department of Homeland Security officials, since 2010, executive departments and agencies responsible for protecting their own facilities have been required to conduct facility security risk assessments as part of the ISC Standard’s risk management process. The ISC Standard states that risk is a measure of potential harm from an undesirable event that encompasses threat, vulnerability, and consequence. The ISC Standard then defines these terms as follows: Undesirable event: An incident, such as vandalism, active shooters, and explosive devices that has an adverse impact on the facility occupants or visitors, operation of the facility, or mission of the agency. Threat: The intention and capability of an adversary to initiate an undesirable event. Vulnerability: A weakness in the design or operation of a facility that an adversary can exploit. Consequence: The level, duration, and nature of the loss resulting from an undesirable event. Based on the assessed level of risk, the ISC Standard provides a method for agencies to identify which countermeasures, such as security cameras or security gates, should be implemented to protect the facility against each of the undesirable events. According to the ISC Standard, once an initial assessment is completed, facility security reassessments should be conducted at least once every 3 to 5 years, depending on the facility’s security level, to reassess whether existing countermeasures remain adequate for mitigating risks. Beginning in fiscal year 2020, the ISC will require departments and agencies to report their compliance with the requirement to conduct facility security assessments on occupied facilities. Figure 2 shows the steps of the ISC Risk Management Process, and figure 3 shows some examples of facility countermeasures. Because facility security assessments are a key component of the ISC’s risk management framework, the ISC Standard includes requirements for agencies’ risk assessment methodologies. Specifically, among other things, the ISC Standard requires that agencies use facility security assessment methodologies that (1) consider all 33 of the undesirable events identified in the ISC Standard, and (2) evaluate the three factors of risk (threat, vulnerability, and consequence) to each undesirable event. During facility security assessments, ratings are assigned to the threat, vulnerability, and consequence of an undesirable event, and the combined ratings produce an overall measurement of risk. In our hypothetical facility security assessment example shown in figure 4, each component of risk is assigned a rating of between 1 (very low) and 5 (very high) based on the facility’s conditions. These ratings are then multiplied to produce an overall estimate of risk for each undesirable event. Agencies can use this and other information resulting from a facility security assessment to make security-related decisions and direct resources to implement countermeasures to address unmitigated risk. Available federal law enforcement data show a range of threats and assaults against the four federal land management agencies’ employees in fiscal years 2013 through 2017. For example, incidents ranged from threats conveyed by telephone to attempted murder against federal land management agency employees. Additionally, FBI data on its investigations into potential domestic terror threats to land management agencies show a wide variety of statutes and regulations that may have been violated. However, not all incidents are captured in the federal land management agencies’ data because not all incidents are reported to the agencies’ law enforcement officials. Additionally, some incidents are investigated by state or local law enforcement and recorded in their data systems rather than in land management agencies’ systems. As a result, the number of actual threats and assaults is unclear and may be higher than what is represented in available data. Our analysis of data from each of the four land management agencies and the FBI showed the following: BLM. BLM data for fiscal years 2013 through 2017 included 88 incidents of threats and assaults against BLM employees and cited eight different statutes or regulations. A federal law prohibiting people from assaulting, resisting, or impeding certain federal officers or employees, 18 U.S.C. § 111, was the statute most frequently cited in BLM’s data. Examples of incidents that identified this statute include an individual harassing a BLM law enforcement officer by repeatedly swerving and cutting off the officer on the highway, an individual making threats against a BLM employee on Facebook and YouTube, and an incident during which an employee was stabbed outside a federal building. Twenty-one of the 88 incidents occurred in fiscal year 2013, when BLM categorized incidents using uniform crime reporting codes rather than federal statutes, regulations, or state laws. These incidents include, for example, an incident in which an individual attempted to murder a law enforcement officer with a firearm. Table 1 provides additional information on threats and assaults against BLM employees for fiscal years 2013 through 2017. FWS. FWS data for fiscal years 2013 through 2017 included 66 incidents of threats and assaults against FWS employees and cited nine different statutes and regulations. A federal law prohibiting people from assaulting, resisting, or impeding certain federal officers or employees, 18 U.S.C. § 111, was the statute most frequently cited in FWS’s data and included a variety of incidents, such as a law enforcement officer who was assaulted with a tree branch during a suspected drug trafficking incident at the border. According to FWS officials, when law enforcement officers cite violations of state statutes, they enter the violation into the law enforcement data system under a generic description such as “Assault: simple, on officer,” and then manually enter the relevant state statute. Of the total FWS incidents, 26 were recorded under unspecified state statutes. These incidents included, for example, an officer who was assaulted while arresting an individual driving under the influence and an officer who received a death threat during an arrest. Table 2 provides additional information on threats and assaults against FWS employees for fiscal years 2013 through 2017. Forest Service. Forest Service data for fiscal years 2013 through 2017 included 177 incidents of threats and assaults against Forest Service employees and cited seven different statutes or regulations. Officials said that the data provided to us generally included only the most serious offense that occurred during an incident, due to limitations on linking records in Forest Service’s data system. For example, if both a verbal threat and physical assault occurred during an incident, only the physical assault would be included in the data. Therefore, potential violations of some statutes or regulations that occurred during incidents of threats and assaults may not be recorded in the data. About half of the Forest Service incidents involved potential violations of 36 C.F.R. § 261.3(a), which includes interfering with a forest officer, among other things. Such incidents included: an individual telling a Forest Service employee that his dog would “rip her head off” if she approached his camp; threatening graffiti written on a law enforcement officer’s personal residence; and a death threat to a law enforcement officer. Table 3 provides additional information on threats and assaults against Forest Service employees for fiscal years 2013 through 2017. Park Service. Park Service data for fiscal years 2013 through 2017 included 29 incidents of threats and assaults against Park Service employees and cited six different offense descriptions. According to a Park Service official, some incident records cite a statute or regulation. However, all agency incident records include offense codes that are unique to the Park Service and are associated with the type of violation, such as assault or disorderly conduct. Unlike with statutes and regulations, a perpetrator does not need to be identified for the law enforcement officer to cite an offense code. Three of the six Park Service offense codes relate to assault. Incidents that cited these codes included an individual ramming an employee’s patrol vehicle and a death threat left on an employee’s personal cell phone. Table 4 provides additional information on threats and assaults against Park Service employees for fiscal years 2013 through 2017. FBI. FBI data for fiscal years 2013 through 2017 show that the FBI initiated under 100 domestic terrorism investigations into potential threats to federal land management agencies, and that these investigations most frequently cited eight specific statutes. Investigations can either be initiated by the FBI or referred to the FBI by land management agencies. Land management agency officials said they refer only the most serious incidents to the FBI—such as the armed occupation of Malheur National Wildlife Refuge. The FBI receives information from a variety of sources, including from confidential human sources; public tips; and state, local, tribal, and federal partners. According to FBI officials, an investigation into a domestic terrorism threat may only be initiated if there is information indicating potential violent criminal activity committed in furtherance of ideology. Our analysis of FBI data showed that the majority of the domestic terrorism investigations involved BLM, and the majority involved individuals motivated by anti-government ideologies. Most of the domestic terrorism investigations cited more than one statute or regulation as having been potentially violated, and the severity of the threat varied. For example, some investigations involved written threats and threats conveyed by telephone to government officials. In one example, the investigation involved a subject posting a BLM law enforcement officer’s personal information on Twitter, which resulted in over 500 harassing phone calls and several death threats. Table 5 provides information on the percentage of FBI investigations citing various statutes and regulations related to threats to federal land management agencies for fiscal years 2013 through 2017. incidents of threats. According to officials at all four agencies, employees do not always report threats to agency law enforcement. For example, some field unit employees said that in certain circumstances, they consider receiving threats a normal part of their job. Specifically, field unit employees we interviewed at three land management agencies cited incidents in which they were yelled at, for example, by hunters, permittees, or attendees of public planning meetings. While this behavior may be threatening, some employees told us it was “a part of the job,” and they did not report such incidents. In addition, some officials described being threatened while off-duty, such as by being harassed in local stores or being monitored at their home, which officials said in some cases they did not report because it was a common occurrence. Additionally, according to agency officials, threats are subject to interpretation, so employees may be reluctant to report an incident unless it involves an explicit threat of physical harm or death. During an incident, some threats and assaults may not be recorded in agency data systems by agency law enforcement officers. BLM and Forest Service officials told us that when a single incident involves multiple offenses, the less serious offenses are unlikely to be recorded in the data system. Therefore, the entirety of what occurred during the incident may not be captured in the data system. For example, according to one BLM official we interviewed, if an incident involved a verbal threat and a physical assault, it would likely be recorded into the data system as an assault. there were trucks regularly parked outside their homes, with individuals holding anti- government beliefs, who appeared to be monitoring them and their families. One official stated that “They were holding us hostage in our own homes.” Some incidents are investigated by state or local law enforcement and recorded in their data systems, rather than in land management agencies’ systems. Some incidents of threats and assaults to federal employees may be investigated by state or local law enforcement entities. Specifically, during our site visits, officials from all four land management agencies stated that their employees are instructed to call 911 in the case of an emergency, such as a threat or assault, and that, generally, a local law enforcement officer—such as a county sheriff’s deputy—will respond to the call. Land management agency officials said that when state or local law enforcement respond to an incident, even those that occur on federal lands, the incident would be recorded in those entities’ data systems and may not be entered into the land management agency’s law enforcement data system. Additionally, according to agency officials at all four land management agencies, due to resource constraints, many of their field units do not have any law enforcement officers or have a limited law enforcement presence, which limits the agencies’ ability to respond to and therefore record incidents of threats and assaults. For example, according to agency officials, as of October 2018, 178 of 418 Park Service units had no law enforcement presence. Furthermore, even when field units had dedicated law enforcement officers, the officers might not have been available to immediately respond to incidents, so employees might instead have contacted local law enforcement. Given these reasons, the actual number of incidents of threats and assaults is unclear and may be greater than the number reported and entered in the land management agencies’ law enforcement data systems, according to federal land management agency officials. Federal land management agencies use various approaches to protect their employees from threat and assaults, including building relationships with external law enforcement entities and the public; receiving, collecting, and disseminating intelligence; and offering training to agency employees. Agency officials we interviewed cited four factors that can affect their ability to protect employees, including that employees often work in remote locations. Federal land management agencies use various approaches to protect their employees from threats and assaults. Specifically: Agencies deploy their law enforcement officers to protect employees and resources. All four federal land management agencies have their own law enforcement divisions with law enforcement officers who are tasked with protecting employees and resources in the field. According to agency officials we interviewed, where available, agency law enforcement officers respond to incidents, including threats and assaults against employees. When necessary, agencies also deploy additional law enforcement officers to assist local officers. For example, during the armed occupation of the Malheur National Wildlife Refuge, FWS officials said the agency deployed FWS law enforcement officers from around the country to field units in western states to provide additional security for FWS employees. Similarly, according to BLM documents, BLM officers are sometimes deployed from their home field units for various reasons, such as assisting with large-scale recreational events and supporting fire investigations and natural disaster recovery. Agencies build relationships with local, state, and other federal agency law enforcement entities, as well as the public. Federal land management agencies build relationships with local, state, and other federal agency law enforcement entities to help protect employees and resources in the field and to assist with coordinating law enforcement responses, according to agency officials. These officials said such relationships are important because not all field units have a law enforcement officer, and those that do often rely on local law enforcement for assistance with incidents of threats or assaults against agency employees. For example, officials at one field unit in Nevada stated that during a high-profile court case involving the agency, the Las Vegas Metropolitan Police Department kept a patrol car outside the field unit for several days to help ensure the safety of the field unit’s employees. Agency field officials said that building relationships with the public—both visitors and local citizens—can help keep their employees safe by cultivating trust and reducing potential tension over federal land management practices. For example, officials at one field unit drafted talking points for employees in the event that visitors asked them about a high-profile incident of anti-government behavior directed at a federal land management agency. The talking points outlined the agency’s responsibilities and authorities and, according to agency officials, were aimed at dispelling misunderstandings about federal land management policies. Additionally, officials at several field units we visited stated that their law enforcement officers are focused on educating, rather than policing, visitors. Agencies receive, collect, and disseminate intelligence information. To varying degrees, federal land management agencies receive, collect, and disseminate intelligence information, which helps them anticipate, prepare for, and react to threats against employees and facilities. For example, officials we interviewed from all four agencies said that they receive intelligence information from various sources, including Interior’s Office of Law Enforcement and Security, the Department of Homeland Security, FBI, Federal Protective Service, and Joint Terrorism Task Forces. Additionally, after the armed occupation of Malheur National Wildlife Refuge, FWS created a new risk and threat assessment coordination unit to collect intelligence, inform decision-making, and improve coordination with other Interior bureaus. Agency officials said they disseminate intelligence information about potential threats to their field units so that field personnel can respond appropriately to the threat—including encouraging employees to telework, directing employees to temporarily stop field work, or temporarily closing their field unit. Agencies have developed plans and guidance to promote employee safety. Agency officials have developed a variety of written plans and guidance to promote employee safety. For example, agencies are required to develop occupant emergency plans for most occupied facilities. Occupant emergency plans we obtained covered employee safety, including what to do in the event of a bomb threat or active shooter event. Additionally, some field units developed other documents that outlined actions employees are to take to remain safe, such as plans to address critical incidents or protests at their field unit. Agencies offer various types of safety training. All four federal land management agencies offer a variety of training to help protect employees and promote their safety, according to agency documents and officials. Examples of topics addressed in agencies’ training include understanding anti-government ideologies, communicating techniques for de-escalating conflicts, and responding to an active shooter event. Agency officials cited four factors that can affect agencies’ efforts to protect their employees: Agency employees work with the public and are often easily recognizable. Agency officials said their employees are required to interact with the public as part of their official duties, which can put them at risk of being threatened or assaulted. FWS officials said they temporarily closed field units in an adjacent state during the beginning of the armed occupation of the Malheur National Wildlife Refuge to reduce the likelihood that their employees would interact with members of the public who were traveling to Malheur to participate in the occupation. FWS and Park Service officials stated that their employees are easily recognizable because they typically wear uniforms, which may put them at greater risk of being harassed or threatened by individuals who hold anti-government beliefs. (See figure 5 for examples of uniforms.) In response, on certain occasions, some agency officials direct their employees to wear street clothes instead of their uniforms. Officials we interviewed indicated that whenever they are concerned about a potential safety issue at their field unit, such as a protest, they may encourage eligible employees to telework from home instead of reporting to their work station. Employees often work in remote locations to fulfill agency missions. Agency officials stated that it can be difficult to protect employees because, as part of their field work, employees may be dispersed across hundreds of miles of federal lands and may be located hours or days away from the nearest agency law enforcement officer. (See figure 6 for an example of a remote location.) As a result, some agency officials said they sometimes direct employees to postpone fieldwork if there is a known or anticipated risk of threats or assaults. In addition, according to officials, various field units have developed check-in and check-out procedures to keep track of employees when they are in the field and to help verify that they report back to the office after concluding their fieldwork. Additionally, some field units have purchased satellite communication devices that operate when cell or radio signals are not available, so that employees conducting remote field work can call for help if needed. The number of agency field law enforcement officers has declined. As of the end of fiscal year 2018, the overall number of field law enforcement officers at each of the four land management agencies had declined from fiscal year 2013, which agency officials noted as a factor straining their efforts to protect employees. For example, the Park Service had the lowest decrease of 7 percent, whereas the Forest Service had the greatest decrease of 22 percent. (See table 6.) Figure 7 shows the total number of acres for which federal land management agencies are responsible, the number of field law enforcement officers they had as of the end of fiscal year 2018, and the ratio of officers to acres of federal land. In addition, field officials from the three Interior agencies stated that as a result of various requirements to send law enforcement officers to support border protection efforts, their law enforcement officers are occasionally absent from their field units when deployed 14 days or more to the border. To help address the effects of border deployments, some agency officials told us that they seek opportunities to share law enforcement resources among field units and with other land management agencies and that they typically deploy law enforcement officers from field offices across the agency to minimize the effects on any one unit. Anti-government sentiment can be unpredictable, difficult to respond to, and disruptive. Agency officials we interviewed said that the risk to employee safety posed by individuals holding anti- government sentiments can be unpredictable and that incidents of threats and assaults against employees by such individuals are generally sporadic. For example, BLM, FWS, and Forest Service officials said it would have been difficult to predict that armed individuals would occupy FWS’s Malheur National Wildlife Refuge, since they were protesting BLM actions. BLM and FWS agency officials said they believed that the occupiers chose Malheur National Wildlife Refuge because it was an easier target. In addition, some agency field unit officials told us that incidents of threats and assault from individuals holding anti-government beliefs generally occur when agency personnel are conducting normal operating activities, such as during routine traffic stops or when they are collecting park entrance fees, making them difficult to predict. Officials from one field unit also noted that while their agency wants to ensure employee safety, it is contrary to their mission to close a field unit every time there is a potential anti-government threat—such as threats made on social media. However, during the armed occupation of the Malheur National Wildlife Refuge, refuges in an adjacent state were closed out of caution, and FWS employees turned away visitors who had driven hundreds of miles to view wildlife, according to FWS officials. To help address the potential disruption posed by unpredictable anti- government threats, some agencies and field units developed plans and guidance that prescribed various actions field units and their employees could take to help ensure employees’ safety while also counteracting the disruptive effects of threats and attacks on a facility’s operations. The four federal land management agencies have completed some but not all of the facility security assessments on their occupied federal facilities as required by the ISC Standard and three do not have a plan for doing so. Furthermore, the Forest Service has a facility security assessment methodology that complies with key requirements described in the ISC Standard, but BLM, FWS, and the Park Service do not. The ISC Standard requires that agencies complete facility security assessments on all occupied facilities and suggests that agencies establish annual objectives for conducting assessments. As suggested in the ISC Standard, to do so, agencies may need to consider several things, such as: the number and locations of needed facility security assessments, by establishing which facilities in the agency’s inventory are occupied and grouping them into campuses, if desired; the agency’s organizational structure, to determine entities responsible for conducting the assessments; training needs of entities responsible for conducting the assessments; which facilities or campuses should be prioritized for assessments, if a schedule for completing the assessments, given the agency’s available resources and priorities. The four land management agencies have not completed facility security assessments on all occupied facilities, and agency officials cited various reasons for not doing so. FWS has a plan to complete its assessments, but BLM, the Forest Service, and the Park Service do not. Specifically: FWS. FWS has conducted five facility security assessments on its approximately 465 occupied facilities and has a plan for completing the remaining assessments. According to FWS headquarters officials, FWS employees have limited physical security expertise to conduct facility security assessments; therefore, the agency has developed a plan to meet the ISC Standard’s requirement using contractors. Specifically, in May 2019, FWS hired a project manager to implement a new facility security assessment program and, according to agency documentation, the new program will, among other things, employ contracted assessors to conduct facility security assessments agency-wide. Agency officials said FWS will hire the assessors after the project manager and other agency officials complete preliminary tasks such as developing ISC-compliant policies and procedures, establishing the number and locations of facility security assessments needed, and developing an electronic tracking system for the assessors to use while conducting assessments. Once these tasks are completed—which could take up to 1 year, according to officials—FWS is to develop a schedule for assessors to complete the remaining assessments. BLM. BLM has conducted 21 facility security assessments on its approximately 280 occupied facilities, but officials do not know when they will complete the remaining assessments and do not have a plan to do so. BLM headquarters officials we interviewed said that the agency is decentralized and its state offices are responsible for the security of facilities in their states, including scheduling and conducting facility security assessments. However, some BLM state and field officials we interviewed said they do not have the resources or expertise to conduct the assessments, and BLM does not offer relevant training. In June 2019, the agency issued a hiring announcement for a headquarters-level security manager. According to officials, once hired, the security manager is to establish training for field employees to conduct facility security assessments and monitor state offices’ compliance with the requirement to conduct assessments. Headquarters officials noted that state offices will remain responsible for scheduling and conducting their own assessments. However, as of June 2019, the agency had not developed a plan for how the security manager would implement agency-wide training given available resources, or ensure state offices’ compliance with the requirement to conduct assessments. Forest Service. The Forest Service has conducted at least 135 facility security assessments on its approximately 1,135 occupied facilities, but officials do not know when they will complete the remaining assessments and do not have a plan for doing so. Forest Service headquarters officials we interviewed said that the agency is decentralized and its regional offices are responsible for the security of facilities in their regions, including scheduling and conducting facility security assessments. However, some regional officials we interviewed said they do not have resources or sufficient staff expertise to conduct the assessments. Forest Service headquarters officials stated that they have partnered with the U.S. Department of Agriculture’s Office of Homeland Security to offer facility security assessment training to Forest Service regional employees. Additionally, Forest Service headquarters officials stated that with the assistance of the U.S. Department of Agriculture’s Office of Homeland Security, they were restructuring their physical security program. Under the new structure, headquarters will oversee compliance at a national level and each region will have a team responsible for facility security assessments in their region, which agency officials said will establish lines of authority to account for the agency’s decentralized structure. However, the Forest Service headquarters official responsible for leading this effort said that, due in part to staff turnover, restructuring the physical security program has been difficult. As of June 2019, the Forest Service does not have a documented plan for how the restructured program will operate, how to ensure sufficient staff are trained to complete the assessments given available resources, or how and when regions will complete all of their assessments. Park Service. The Park Service has conducted at least 148 facility security assessments on its approximately 1,505 occupied facilities, but officials do not know when they will complete the remaining assessments and do not have a plan to do so. Park Service headquarters officials we interviewed said that the agency is decentralized and the superintendents of its 418 park units are responsible for the security of facilities within their parks, including scheduling and conducting facility security assessments. However, some park unit officials we interviewed said they do not have the resources or sufficient staff with expertise to conduct the assessments. Park Service headquarters officials stated that they have developed a program to offer facility security assessment training to park employees. In February 2019, according to agency officials, the Park Service hired a security manager who will standardize the agency’s facility security assessment practices, expand facility security assessment training opportunities, and monitor parks’ compliance with the requirement to conduct assessments. Headquarters officials noted that park units will remain responsible for scheduling and conducting their own assessments. However, as of June 2019, the agency had not developed a documented plan for how to ensure sufficient staff are trained to complete the assessments given available resources, or how the security manager would ensure park units’ compliance with the requirement to conduct assessments. Not complying with the ISC Standard’s requirement to complete facility security assessments on all occupied facilities could leave federal agencies exposed to risks in protecting their employees and facilities. Specifically, without conducting all of the required assessments, agencies may not identify the degree to which undesirable events can impact their facilities or identify the countermeasures they could implement to mitigate the risks of those events. Officials from BLM, the Forest Service, and the Park Service acknowledged that completing the remaining facility security assessments is important and that developing an agency-wide plan to do so may help them as they work towards compliance with this ISC Standard requirement. In the process of developing their plans, the agencies could take into consideration their organizational structure, available resources, and training needs, all of which may affect how quickly they can complete their assessments. Furthermore, developing a plan for completing facility security assessments will require agencies to identify the number and locations of their required assessments, which may help them fulfill the fiscal year 2020 ISC compliance reporting requirement. Three of the four federal land management agencies have not developed a facility security assessment methodology that complies with two key requirements in the ISC Standard. Specifically, according to the ISC Standard, methodologies must, among other things, (1) consider all 33 of the undesirable events identified in the Standard, such as active shooters, vandalism, and explosive devices; and (2) evaluate the three factors of risk—threat, vulnerability, and consequence—for each undesirable event. According to our analysis of agency documentation and interviews with agency officials, the extent to which each agency’s facility security assessment methodology complied with the two key ISC Standard requirements we evaluated varied. As of June 2019, the Forest Service’s facility security assessment methodology met the two key ISC Standard requirements we evaluated, and the Park Service’s methodology partially met the requirements. BLM and FWS did not have established facility security assessment methodologies as of June 2019. Specifically: Forest Service. The Forest Service utilizes an ISC-compliant facility security assessment methodology developed by the U.S. Department of Agriculture. The methodology adheres to the two key ISC Standard requirements that we evaluated. Park Service. The Park Service developed a risk assessment methodology, but it only partially adheres to the two key ISC Standard requirements we evaluated. Specifically, the Park Service’s risk assessment methodology does not include a step to assess the consequences of specific undesirable events, as required by the ISC Standard. Park Service officials indicated the agency’s commitment to conducting facility security assessments using an ISC-compliant methodology and said that they plan to submit the Park Service’s risk assessment methodology to the ISC to be certified as compliant with requirements in the ISC Standard. A Park Service official acknowledged, however, that the agency needs to update its methodology to include a step to assess the consequences of specific undesirable events, and the official stated that the agency does not plan to submit the methodology to the ISC until those changes have been made. As of June 2019, officials did not have a timeframe for doing so. BLM. BLM officials said that, as of June 2019, the agency did not have an established methodology for conducting facility security assessments. Officials told us that, once hired, the new BLM security manager will develop an assessment methodology and that the agency intends to employ a methodology that complies with the ISC Standard. However, BLM officials do not know when the security manager will be hired, and the agency has not documented requirements for the security manager to adhere to the ISC Standard’s requirements. FWS. FWS officials said that, as of June 2019, the agency did not have an established methodology for conducting facility security assessments., Officials told us that the agency intends to employ a methodology that complies with the ISC Standard and provided a high-level description of what they expect the methodology to include. However, this description did not indicate that the agency would evaluate consequences of specific undesirable events, as required by the ISC Standard. According to FWS officials, because staff do not have the expertise to conduct facility security assessments, in 2011, the agency developed physical security survey checklists as an interim solution for assessing facilities. These checklists allowed staff to document the presence or absence of countermeasures identified in the ISC Standard. However, FWS headquarters officials acknowledged that these checklists were not an ISC- compliant risk assessment methodology since they do not consider undesirable events or measure risk, as required by the ISC Standard. By not using a methodology that fully complies with the ISC Standard, agencies could face adverse effects, such as an inability to make informed resource allocation decisions for their physical security needs and providing facilities—and the facilities’ occupants—with an inappropriate or insufficient level of protection. Specifically, according to the ISC Standard, when agencies do not use methodologies that comply with risk assessment requirements in the ISC Standard, facilities may have either less protection than needed, resulting in unmitigated risks, or more protection than needed, resulting in wasted resources. To carry out their critical missions to manage the resources on over 700 million acres of federal lands, BLM, FWS, Forest Service, and Park Service officials and facilities are often the most visible and vulnerable representatives of the federal government in remote areas and have been subject to a range of threats and assaults. One way for these agencies to address the safety risks posed by unpredictable anti-government sentiment or other threats is to follow the ISC Standard requirements for conducting facility security assessments. However, BLM, FWS, the Forest Service, and the Park Service have not conducted all required facility security assessments, and BLM, the Forest Service, and the Park Service do not have a plan for doing so. Agency officials stated that this is due, in part, to decentralized organizational structures, limited available resources, and insufficient training. Without a plan for conducting all of the remaining assessments, agencies may not identify the degree to which undesirable events can impact their facilities or identify countermeasures they could implement to mitigate the risks of those events. In addition, as of June 2019, BLM, FWS, and the Park Service do not have facility security assessment methodologies that fully comply with two key requirements in the ISC Standard—namely, to consider the 33 undesirable events identified in the Standard and to evaluate risk factors for each of these events. Without using a methodology that complies with the ISC Standard, the agencies could face adverse effects, including an inability to make informed resource allocation decisions for their physical security needs and providing facilities—and the facilities’ occupants—with an inappropriate or insufficient level of protection. We are making a total of six recommendations, including two to BLM, one to FWS, one to the Forest Service, and two to the Park Service. Specifically: The Director of BLM should develop a plan to conduct all required facility security assessments agency-wide, taking into consideration the agency’s organizational structure, available resources, and training needs. (Recommendation 1) The Chief of the Forest Service should develop a plan to conduct all required facility security assessments agency-wide, taking into consideration the agency’s organizational structure, available resources, and training needs. (Recommendation 2) The Director of the Park Service should develop a plan to conduct all required facility security assessments agency-wide, taking into consideration the agency’s organizational structure, available resources, and training needs. (Recommendation 3) The Director of the Park Service should update the agency’s facility security assessment methodology to comply with requirements in the ISC Standard, including a step to consider the consequence of each undesirable event. (Recommendation 4) The Director of BLM should develop a facility security assessment methodology that complies with requirements in the ISC Standard to assess all undesirable events and consider all three factors of risk for each undesirable event. (Recommendation 5) The Director of FWS should develop a facility security assessment methodology that complies with requirements in the ISC Standard to assess all undesirable events and consider all three factors of risk for each undesirable event. (Recommendation 6) We provided a draft of this report to the Departments of Agriculture, Homeland Security, Interior, and Justice for their review and comment. The Forest Service, responding on behalf of the U.S. Department of Agriculture, generally agreed with the report and our recommendation and cited its efforts to develop a plan to complete required facility security assessments. The Forest Service’s written comments are reproduced in appendix III. Interior, responding on behalf of BLM, FWS, and the Park Service, concurred with our recommendations and provided examples of actions the three agencies planned to take. Specifically, regarding our recommendation that BLM and the Park Service develop a plan to conduct facility security requirements agency-wide, BLM intends to revise its policy and develop such a plan, and the Park Service intends to develop a plan that includes training and tools so that park unit staff can conduct the required assessments. Regarding our recommendation that BLM, FWS, and the Park Service develop methodologies that comply with requirements in the ISC Standard, the agencies cited various efforts to do so, including revising policies and developing new tools, training, and data system modules. Interior’s written comments are reproduced in appendix IV. The Department of Homeland Security provided a technical comment that we incorporated. The Department of Justice told us that they had no comments. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees; the Attorney General; and the Secretaries of Agriculture, Homeland Security, and the Interior. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix V. Our objectives were to examine for the four federal land management agencies, (1) what is known about the number of threats and assaults against their employees, (2) the approaches the agencies used to protect their employees from threats and assaults and any factors affecting their ability to do so, and (3) the extent to which agencies met facility security assessment requirements. For the first objective, we obtained and analyzed data on threats and assaults against land management agency employees from the law enforcement databases of the Forest Service within the U.S. Department of Agriculture and the Bureau of Land Management (BLM), Fish and Wildlife (FWS), and National Park Service (Park Service) within the Department of the Interior for fiscal years 2013 through 2017. These data were the most recent available at the time we began our review. We also obtained and analyzed data from the Federal Bureau of Investigation (FBI) regarding its investigations into potential domestic terror threats to land management agencies. Each land management agency’s law enforcement division records data on threats and assaults to employees, as part of its broader mission to enforce laws that safeguard employees and protect resources. The data systems, however, were not specifically designed for reporting threats and assaults against employees, and they do not include the suspect’s motivation for a crime, such as anti-government extremist ideologies. Since each agency collects and maintains data in a different data system and has agency-specific reporting requirements for incidents, the data differ in how they were originally recorded by field law enforcement officers and how they were queried and reported by headquarters officials responding to our request for data. As such, if data were not entered, or not entered correctly, they would not have been captured in agency queries. According to agency officials at the four land management agencies, they queried their data systems to identify records of incidents that pertained to threats and assaults against employees. BLM, Forest Service, and Park Service officials then conducted record-level reviews and removed records that they determined were not threats or assaults, contained errors, were duplicative, or did not contain sufficient information to make a conclusive determination. We did not systematically review the records they removed. Information about each agency’s data system and limitations related to the agency’s data are as follows: BLM. BLM maintains its data in the Incident Management, Analysis, and Reporting System (IMARS). IMARS is used by most Interior bureaus for incident management and reporting and to prevent, detect, and investigate known and suspected criminal activity. Each bureau uses a different, customized version of IMARS. BLM officials said that beginning in fiscal year 2014, BLM began collecting data on violations of federal statutes, regulations, and state laws during incidents. Prior to that, BLM used a generic description of each offense. Officials also said that when multiple offenses occur during an incident, the less serious offenses are unlikely to be entered into the system. Therefore, some offenses that occurred during incidents of threats and assaults may be excluded from these data. FWS. FWS maintains its data the agency’s Law Enforcement Management Information System (LEMIS). According to FWS documents, LEMIS is used to process and store investigations, intelligence, and other records. FWS officials said the agency changed data systems during our reporting time frame. Specifically, FWS originally stored fiscal year 2013 and 2014 data in the Law Enforcement-Information Management and Gathering System and imported the data into LEMIS in July 2014. We assessed the data across the two systems by comparing incidents per year and types of violations that occurred, and we found that the data were comparable. According to agency officials, they did not review the incidents before providing them to us; therefore, some incidents may not have been actual threats or assaults. Forest Service. The Forest Service maintains its data in the Law Enforcement and Investigations Management Attainment Reporting System (LEIMARS). Forest Service officials said LEIMARS is used to record criminal and claims activity in the national forests, which include verified violations of criminal statutes and agency policy, as well as incidents that may result in civil claims for or against the government. Incidents are recorded in LEIMARS in one of three types of law enforcement report categories: (1) an incident report, which records when an offense occurred but the perpetrator was unknown; (2) a warning notice, which is issued when an offense occurred but the law enforcement officer determined that the offense was inadvertent or committed due to lack of understanding or misinformation; and (3) a violation notice, which is issued for an offense that violates the U.S. Code or Forest Service regulations and the perpetrator was known. We present these three types of reports as incidents. A Forest Service official identified 125 incidents for which the agency could not determine whether a threat or assault to an employee occurred. We excluded these 125 incidents from our analysis. Officials told us that they only provided data on the most serious offense occurring during an incident due to limitations on linking records in the Forest Service’s data system; they also told us that there may be a minor amount of overlap between violation notices and incident reports. Park Service. As with BLM, Park Service data is maintained in the IMARS data system. According to a Park Service official, some but not all Park Service incident records cite a federal statute or regulation. However, all Park Service incident records include offense codes—which are unique to the Park Service—that are associated with the type of violation, such as assault or disorderly conduct. Unlike with the statutes and regulations, a perpetrator does not need to be identified for the law enforcement officer to cite an offense code. Therefore, the Park Service provided data to us by offense code, and we were not able to present the data by the statute or regulation that was potentially violated. We also obtained data from the FBI on investigations into potential domestic terror threats to land management agencies. FBI investigation data is maintained in the Sentinel data system, which is FBI’s case management system. FBI officials provided data from the FBI’s domestic terrorism program on three types of investigations: assessments, preliminary investigations, and full investigations. We reported data on the full investigations because of the limited information available on assessments and preliminary investigations. Before providing the data to us, an FBI official reviewed the record of each domestic terrorism investigation initiated in fiscal years 2013 through 2017 to determine whether the investigation was relevant to threats to BLM, FWS, the Forest Service or the Park Service. These data represent all potential violations known at the time the FBI agent first opened the case and therefore include various potential violations beyond threats and assaults against federal employees. According to agency officials, in some cases, the FBI agent opening the case may not have been able to fully identify all relevant subsections of the statute or regulation that was potentially violated. To account for this, we report FBI’s data at the statute or regulation level. Since we relied on the professional judgement of agency officials to review and interpret incident data, we may be unable to replicate the final data selection drawn from each agency’s database, even if we retrieved the data using the same method and search criteria. We independently assessed the reliability of each agency’s data by (1) reviewing related documentation about the data system; (2) conducting manual reviews of the data for missing data, outliers, and obvious errors; (3) reviewing related internal controls; and (4) interviewing agency officials knowledgeable about the data, among other things. In our interviews, we asked agency officials about data entry practices, data system capabilities and limitations, and circumstances whereby incidents of threats and assaults might not appear in the database. Based on our review, we determined that the data were sufficiently reliable for the purposes of reporting descriptive summary information on the number of threats and assaults against federal land management employees during fiscal years 2013 through 2017. To address our second objective, we examined policies and requirements regarding federal land management agencies’ responsibilities for protecting employees against threats and assaults. We also interviewed headquarters and selected field unit officials about the agencies’ approaches to protecting their employees from threats and assaults and factors that may affect their ability to do so, and we obtained supporting documentation where available. We conducted site visits from March through July 2018 to a nongeneralizable sample of 11 of the 35 regional or state offices and 14 field units across the federal land management agencies. We selected sites in Colorado, Nevada, Oregon, and Utah, since the majority of federal lands are located in the West and some field units in these states had been affected by actions of individuals motivated by anti-government ideologies. Specifically, we conducted site visits to five BLM field units, nine FWS field units, seven Forest Service field units, and four Park Service field units. The number of field units we interviewed varied on several factors, including how many field units regional and state offices invited to the meeting. Findings from the interviews we conducted at our site visits provide useful insights but cannot be generalized to those units we did not include in our review. Based on our site visit interviews, we identified four primary factors affecting agencies’ abilities to protect their employees from threats and assaults. We also collected information from each agency on the number of field law enforcement officers they had at the end of fiscal years 2013 and 2018, the most recent year for which data were available—to analyze any changes in resources. We took steps to assess the reliability of these data, including comparing the data to agency budget justifications and interviewing agency officials, and found them to be sufficiently reliable for the purpose of reporting the number of field law enforcement officers agencies had in fiscal years 2013 and 2018. For the third objective, we examined government-wide requirements promulgated by the Interagency Security Committee (ISC) and documented in ISC’s Risk Management Process for Federal Facilities, which we refer to in this report as the ISC Standard, and its related appendixes. We interviewed ISC officials to learn more about the development of the requirements in the ISC Standard and variations, if any, in how agencies are expected to implement them. To determine whether agencies met the requirement to conduct facility security assessments on all of their occupied facilities, we obtained documents on the agencies’ inventories of occupied facilities and assessed whether the agencies had conducted security assessments on those facilities. We interviewed headquarters and field officials about their inventories and their plans, if any, for completing the remaining assessments. We also examined the extent to which agencies’ facility security risk assessment methodologies complied with two key requirements in the ISC Standard. These included that methodologies must: (1) consider all 33 of the undesirable events identified in the Standard and (2) evaluate the three factors of risk—threat, vulnerability, and consequence—for each undesirable event. We analyzed the agencies’ methodologies and compared them against requirements in the ISC Standard. We also interviewed agency officials about the methodologies. We conducted this performance audit from November 2017 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Anne-Marie Fennell, (202) 512-3841 or fennella@gao.gov. In addition to the individual named above, Casey L. Brown (Assistant Director), Tanya Doriss (Analyst in Charge), Charles W. Bausell, Charles A. Culverwell, John W. Delicath, Emily E. Eischen, Cindy K. Gilbert, Richard P. Johnson, Vanessa E. Obetz, Dan C. Royer, and Breanna M. Trexler made key contributions to this report.", "summary": "A 2014 government report predicted that the rate of violent domestic extremist incidents would increase. In recent years, some high-profile incidents have occurred on federal lands, such as the armed occupation of a FWS wildlife refuge in 2016. Federal land management agencies manage nearly 700 million acres of federal lands and have law enforcement divisions that protect their employees and secure their facilities. GAO was asked to review how land management agencies protect their employees and secure their facilities. For the four federal land management agencies, this report examines, among other things, (1) what is known about the number of threats and assaults against their employees and (2) the extent to which agencies met federal facility security assessment requirements. GAO analyzed available government data on threats and assaults; examined agencies' policies, procedures, and documentation on facility security assessments; compared the agencies' methodologies against ISC requirements; and interviewed land management agency, ISC, and FBI officials. Data from the four federal land management agencies—the Forest Service within the U.S. Department of Agriculture and the Bureau of Land Management (BLM), Fish and Wildlife (FWS), and National Park Service (Park Service) within the Department of the Interior—showed a range of threats and assaults against agency employees in fiscal years 2013 through 2017. For example, incidents ranged from telephone threats to attempted murder against federal land management employees. However, the number of actual threats and assaults is unclear and may be higher than what is captured in available data for various reasons. For example, employees may not always report threats because they consider them a part of the job. Federal Bureau of Investigation (FBI) data for fiscal years 2013 through 2017 also showed that the FBI initiated under 100 domestic terrorism investigations into potential threats against federal land management agencies. The majority of these investigations involved BLM and individuals motivated by anti-government ideologies. The four federal land management agencies have completed some but not all of the facility security assessments on their occupied federal facilities as required by the Interagency Security Committee (ISC). Officials at the four agencies said that either they do not have the resources, expertise, or training to conduct assessments agency-wide. FWS has a plan to complete its assessments, but BLM, the Forest Service, and the Park Service do not. Such a plan could help these agencies address the factors that have affected their ability to complete assessments. The ISC also requires that agencies conduct assessments using a methodology that meets, among other things, two key requirements: (1) consider all of the undesirable events (e.g., arson and vandalism) identified as possible risks to facilities, and (2) assess the threat, vulnerability, and consequence for each of these events. The Forest Service's methodology meets these two requirements and the Park Service's methodology partially meets the requirements, but BLM and FWS have not yet established methodologies for conducting facility security assessments. Without developing a plan for conducting all of the remaining facility security assessments and using a methodology that complies with ISC requirements, agencies may not identify the risks their facilities face or identify the countermeasures—such as security cameras or security gates—they could implement to mitigate those risks. GAO is making six recommendations: that BLM, the Forest Service, and the Park Service develop a plan for completing facility security assessments and that BLM, FWS, and the Park Service take action to ensure their facility security assessment methodologies comply with ISC requirements. The agencies generally concurred with the recommendations.", "document_type": "gao"}
{"report": "The 340B Program was created in 1992 following the creation of the Medicaid Drug Rebate Program and gives 340B covered entities—certain eligible hospitals, clinics, and other entities—discounts on covered outpatient drugs comparable to those made available to state Medicaid agencies. According to HRSA, which administers and oversees the 340B Program, the program’s purpose is to enable participating hospitals and other providers to stretch scarce federal resources to reach more eligible patients and provide more comprehensive services. In addition to realizing substantial savings through 340B Program price discounts— which HRSA estimates as 25 to 50 percent of the cost of drugs—covered entities can generate revenue through their participation in the 340B Program. For example, they can purchase covered outpatient drugs at the 340B Program price for all eligible patients regardless of the patients’ income or insurance status and generate revenue by receiving reimbursement from patients’ insurance that may exceed the 340B prices paid for the drugs. Entities are generally eligible for the 340B Program—that is, are covered entities—if they receive one of 10 federal grants or are one of six types of hospital. Hospitals must also meet additional requirements, such as being owned or operated by a state or local government, being formally granted governmental powers, or being nongovernmental. The 340B statute requires nongovernmental hospitals to be nonprofit and to have contracts with state or local governments to provide health care services to the 340B-specified low-income population. However, the requirement does not specify criteria for these contracts, such as the amount or type of services to be provided to these low-income individuals. Generally, hospitals must also meet other requirements to participate, such as treating a disproportionate number of low-income Medicare and Medicaid patients. Hospital participation in the 340B Program has more than tripled over the last decade, due, in part, to the enactment of the Patient Protection and Affordable Care Act in 2010, which expanded the types of hospitals that could qualify for the program. According to data from HRSA, in 2009, prior to the law’s enactment, there were more than 800 340B-participating hospitals, compared to more than 2,500 in 2019. The majority of participating hospitals are nongovernmental hospitals. Specifically, 1,690, or 67 percent, of the hospitals participating as of January 1, 2019 were nongovernmental hospitals. (See figure 1.) HRSA is responsible for verifying hospitals’ and other covered entities’ eligibility to participate in the 340B Program. HRSA reviews nongovernmental hospitals’ eligibility for the 340B Program at registration, recertification, and through audits. Registration. Prior to participation in the 340B Program, hospitals must register with HRSA, at which point they must self-attest to meeting the program’s eligibility requirements. Additionally, HRSA’s hospital registration instructions specify that, at the time of registration, a nongovernmental hospital must have documentation that shows it is nonprofit (such as copies of Internal Revenue Service documentation) and a copy of its contract with a state or local government to serve the 340B-specified low-income population. This documentation must be provided to HRSA upon request. During each quarterly registration period, HRSA conducts contract integrity checks for a random sample of 20 percent of newly registering nongovernmental hospitals. For the selected hospitals, HRSA requests a copy of the hospital’s contract with the state or local government, which it reviews to verify that the contract is signed by officials from both organizations, is in effect, and does not expire before program participation would begin. HRSA policy states that a hospital that cannot provide a state or local government contract when selected for a contract integrity check at registration will not be registered for the 340B Program. Recertification. To remain in the 340B Program, hospitals must annually recertify their eligibility. During recertification, hospitals are to ensure that their information (e.g. name, address, point of contact) is correct in HRSA’s internal 340B Program database and self-attest that the hospital still meets program requirements. HRSA collects documentation from the hospital if it reports changes to its name, classification (i.e., whether it is government owned or operated, delegated governmental powers, or nongovernmental), or nonprofit status. Audits. HRSA audits 200 covered entities—a combination of hospitals and federal grantees—per year. HRSA’s audits include covered entities (including hospitals) that are selected based on risk-based criteria (approximately 90 percent of the audits conducted each year), and entities that are targeted based on, for example, stakeholder allegations of noncompliance (10 percent of audits conducted). The criteria for risk- based audits include a covered entity’s changes in the volume of 340B Program drug purchases, time in the program, complexity of its program, and history of violations or allegations of noncompliance. Among other things, HRSA’s audits include assessments of each hospital’s 340B eligibility status. For a nongovernmental hospital, HRSA’s guidance indicates that auditors are expected to review the hospital’s contract with the state or local government to ensure that it is for serving the 340B-specified low-income population and is signed by both a hospital and state or local government official. Auditors are also expected to review the contract’s start and end dates to ensure that it is effective during a specific period of review. HRSA defines the audit’s period of review as the time frame beginning the first day of the audit’s sample period—a six-month period that predates and is not contiguous with the beginning of the onsite audit—and ending on the last day of the onsite audit. For example, a hospital with an onsite audit in March of 2017 may have a sample period from July 1, 2016 through December 31, 2016, which means that auditors should verify that the hospital’s contract was in effect from at least July 1, 2016 through the end of the March 2017 onsite audit. If HRSA identifies deficiencies in hospitals’ contracts, the agency may issue (1) findings of noncompliance, which are made public on HRSA’s website, or (2) areas for improvement, which are not made public. When an audit results in a finding of noncompliance, the hospital is required to submit a corrective action plan within 60 days of the audit report being finalized for HRSA’s approval. HRSA closes the audit once the hospital attests that the corrective action plan has been fully implemented, and any necessary repayments have been made to affected manufacturers. For example, if a nongovernmental hospital were unable to demonstrate that it had a contract with a state or local government when audited, HRSA policy states that the hospital would be issued a finding of noncompliance and may be subject to termination from the 340B Program for not meeting eligibility criteria. In addition, the hospital may be responsible for repayment to manufacturers for discounts it received during the period it lacked a contract. Most of the contracts we reviewed between nongovernmental hospitals and state or local governments obligated the hospitals to provide health care services to low-income individuals, but they included few details about those obligations. The 340B statute requires participating nongovernmental hospitals to have state or local government contracts to provide health care services to the 340B-specified low-income population, but does not otherwise specify details for the content of these contracts. Of the 240 contracts we reviewed, 224 (93 percent) required the hospital to provide services to low-income individuals. Of these 224 contracts, 169 (75 percent) specifically mentioned providing services to the 340B-specified low-income population (low-income individuals not eligible for Medicaid or Medicare). 55 (25 percent) specified a more general obligation to provide services to individuals who are likely low-income, uninsured, or underinsured, such as enrollees in a county program for the medically indigent, inmates at a local detention center, or individuals receiving treatment through a county mental health program. Less than one-third of the contracts we reviewed defined “low-income” or included detailed requirements for the amount or type of services to be provided. Of the 224 contracts that contained an obligation to provide services to low-income individuals, 14 (6 percent) specified what was considered low income. Of these contracts, the specific income threshold varied, generally ranging from 100 percent to 400 percent of the federal poverty level. 71 (32 percent) specified the amount of services the hospitals were to provide to low-income individuals. The contracts generally defined the amount of services as a range in the cost of care the hospital was expected to provide; the amount varied by contract. For example, one contract specified that the hospital would provide $60,000 to $100,000 of services per year, while another included a range of $62 million to $85 million per year. Contracts that did not specify dollar amounts included, for example, provisions regarding the number of staff available to provide services or requirements to provide services at certain times. One such contract required a hospital to provide at least one full-time-equivalent behavioral health provider for specified sites, while another required a hospital to administer influenza vaccines at two clinics on two Fridays each year during influenza season. 53 (24 percent) identified specific types of services that hospitals were to provide, often specifying multiple categories of services. For example, one contract required a hospital, among other things, to provide inpatient and outpatient services, obstetrics, and cardiovascular surgery. Other contracts only identified a single category of service that the hospital was required to provide. For example, nine of the 53 contracts specified that the hospitals were required to provide emergency services, although hospitals that operate emergency departments are already required, as a condition of participating in Medicare, to screen, and if necessary stabilize patients who seek emergency care, regardless of their ability to pay. Additionally, four of the contracts reviewed required the provision of behavioral health services, two specified the provision of vaccinations, and one was for the evaluation and treatment of tuberculosis. 46 of the 224 contracts (21 percent) specified that state or local governments would pay hospitals for the services provided. In some cases, the contracts specified that the hospitals would be paid to provide care for low-income individuals at rates established under other programs–such as the state’s Medicaid program. Others established rates specifically for services provided to the population covered under the contract. Finally, approximately one-third of the contracts reviewed included provisions that would allow the state and local governments contracting with hospitals to ensure that the contractually required services are being provided. Specifically, 68 of the 224 contracts (30 percent) included provisions for reporting, monitoring or enforcement, as shown below in Figure 2; some contracts included more than one type of provision. Of the 68 contracts, 56 required hospitals to report information to the state or local government. Of these 56, 40 required reporting on the services provided under the contract, including types, dollar amounts, or number of services provided to certain populations, such as “medically indigent,” “uninsured persons,” and “underinsured persons.” For example, one contract required the hospital to provide the government with an annual report containing information about the value of free care provided to indigent persons, the total value of discounted care provided to uninsured patients, and the number of declined requests for free or discounted care. The remaining 16 contracts included more general reporting requirements, such as to provide copies of any reports requested by state or federal licensing, regulatory, or accrediting entities, to the state or local government. 29 contracts included provisions for governments to monitor the hospitals’ provision of care. Specifically, 10 contracts required regular reviews, with some of those at specific time intervals (e.g. annually, quarterly), while 19 contracts required that hospitals be available for periodic audits or to provide the government access, upon request, to medical records and documents which could be used to review or evaluate the services being provided under the contract. 34 contracts included enforcement mechanisms for the government to apply consequences if the hospital did not meet the terms of the agreement. For example, one contract allowed the state government to terminate the contract 90 days after providing notice of the state’s determination that the hospital was not providing sufficient services to low-income individuals. Contracts for eight hospitals provided for monetary fines or withholding of funds if hospitals were found to be in breach of the contract. HRSA uses self-reported data to determine whether hospitals are nonprofit without assessing whether the data are reliable for that purpose. Additionally, HRSA relies primarily on nongovernmental hospitals’ self- attestations to verify the existence of state or local government contracts, and weaknesses in the reviews of contracts it does conduct hamper the identification of potential eligibility issues. HRSA uses Medicare cost report data from CMS to determine whether hospitals are nonprofit, but these data may not be sufficiently reliable for this purpose. Specifically, HRSA relies on self-reported information from cost reports on whether hospitals operate as nonprofit, proprietary, or governmental organizations. HRSA reviews this information at registration to check that hospitals have indicated that they are nonprofit organizations. Additionally, in April 2019, HRSA began conducting quarterly checks of cost report data to identify hospitals that list themselves as proprietary for further review, as this designation could be used by for-profit, rather than nonprofit, hospitals, contrary to 340B Program eligibility requirements. HRSA officials told us that the agency has not independently evaluated the reliability of the cost report data for determining nonprofit status. Additionally, a CMS official responsible for oversight of Medicare cost reports told us that CMS does not have any formal processes to assess the reliability of the data on whether a hospital is nonprofit, proprietary, or governmental, because these data do not affect Medicare reimbursement. The official added that the question on the cost report used to collect these data was not intended to assess nonprofit status, is not clearly defined, and may not be reported accurately. For example, the cost report instructions do not include definitions of nonprofit and proprietary for providers to refer to when they are completing their cost reports. HRSA requires hospitals to maintain additional documentation, such as Internal Revenue Service forms for tax-exempt organizations or documents from the state, to demonstrate their nonprofit status, but does not collect or review this documentation if hospitals indicate that they are nonprofit on their cost reports. In August 2019, HRSA submitted a proposal to the Office of Management and Budget to require hospitals registering for the 340B Program to submit documentation supporting the hospital classification that they select during registration, which would include requiring nongovernmental hospitals to submit documentation of their nonprofit status. However, this requirement, if it goes into effect, would apply only to newly registering hospitals and would not affect the nearly 1,700 nongovernmental hospitals currently participating in the 340B Program. For those hospitals, HRSA would continue to rely on the Medicare cost report data. Relying on the self-reported data from Medicare cost reports is inconsistent with federal internal control standards related to information and communication, which state that management should use quality information to achieve the entity’s objectives, such as by obtaining relevant data, based on identified information requirements, that are reasonably free from error and bias, and that management should evaluate the data for reliability. Without ensuring that the information it uses on hospitals’ nonprofit status is reliable, HRSA cannot effectively determine if nongovernmental hospitals participating, or seeking to participate, in the 340B Program meet the statutory eligibility requirements, creating a risk that for-profit hospitals could receive discounted pricing for which they are not eligible. HRSA primarily relies on self-attestations from nongovernmental hospitals to verify that they have contracts in place with state or local governments as required to participate in the 340B Program. Specifically, HRSA relies on the attestations that hospitals are required to make during registration and recertification that they meet the program’s eligibility requirements. Although HRSA requires nongovernmental hospitals to have copies of their contracts, and to provide them upon request, it does not require most hospitals to submit those contracts at either registration or recertification. Additionally, while HRSA previously required each nongovernmental hospital to submit a certification of contract form during registration that was signed by a government official and attested to the existence of a contract to serve the 340B-specified low-income population, officials said the agency stopped requiring submission of this form in July 2014. At that time, officials said HRSA initiated a process of contacting government officials directly through an online certification process to confirm that newly registering hospitals had contracts in place. However, that process was eliminated in September 2017, and HRSA no longer has a process that requires state and local government officials to confirm the existence of contracts with nongovernmental hospitals. HRSA does collect and review contracts with state or local governments for a sample of nongovernmental hospitals through its audit and contract integrity check processes, but these reviews are currently limited in number and scope. Specifically, in fiscal years 2017 and 2018, HRSA audited about 7 percent of nongovernmental hospitals per year (108 and 109 hospitals in fiscal years 2017 and 2018, respectively). Additionally, at the time of our review, HRSA conducted contract integrity checks for 20 percent of newly registering hospitals; this equated to 41 hospitals in calendar years 2017 and 2018 combined. HRSA’s August 2019 proposed information collection request, if approved, would require all newly registering nongovernmental hospitals to submit their state or local government contracts at registration. However, as previously mentioned, this new requirement would only affect newly registering hospitals and not those already participating. Consequently, for the large majority of nongovernmental hospitals already registered for the 340B Program, self-attestations made electronically at registration and recertification would remain HRSA’s sole method of verifying that hospitals have state or local government contracts as required by the 340B statute. Additionally, HRSA officials told us that when the agency does collect documents from nongovernmental hospitals through its audits or contract integrity checks, they do not review them to determine if they are contracts (i.e., mutually binding agreements to provide services or supplies in exchange for something of value). Based on our review of documentation submitted to HRSA from 258 hospitals, 18 hospitals submitted documents that did not appear to meet this common definition of a contract; examples included certification of contract forms without accompanying contracts, articles of incorporation, and descriptions of community programs. Nevertheless, these hospitals were permitted to participate in the 340B Program. HRSA’s reliance on hospitals to attest that the required contracts are in place is contrary to federal internal control standards related to information and communication, which state that management should use quality information to achieve the entity’s objectives, such as by obtaining relevant data from external sources in a timely manner based on the identified information requirements. Without a process to verify that all nongovernmental hospitals have contracts in place, HRSA does not have reasonable assurance that nongovernmental hospitals participating in, or seeking to participate in, the 340B Program have contracts with state and local governments. Consequently, this increases the risk that nongovernmental hospitals that do not have the statutorily required contracts and are thus ineligible may register for, and participate in, the program. In addition to not determining whether the documentation provided by nongovernmental hospitals during contract integrity checks and audits are contracts, weaknesses in HRSA’s reviews hamper its ability to identify and address issues that affect the hospitals’ eligibility for the 340B Program. Specifically, we identified three weaknesses: (1) contract integrity checks do not assess whether contracts require hospitals to serve the 340B-specified low-income population; (2) guidance for auditors’ review of contracts has not been consistently documented and lacks detail; and (3) HRSA allows hospitals to avoid audit findings by entering into new contracts with state and local governments while audits are being conducted. HRSA’s contract integrity checks for newly registering hospitals do not assess whether the contracts require the provision of services to the 340B-specified low-income population. HRSA’s contract integrity checks for newly registering nongovernmental hospitals are limited to verifying that contracts clearly list the names of the hospital and unit of government and have appropriate signatures and dates; procedures for conducting these checks do not instruct staff to review whether the contracts require hospitals to provide health care services to the 340B-specified low-income population, as required to participate in the 340B Program. Of the 38 contracts submitted to HRSA for contract integrity checks in 2017 and 2018, two (5 percent) did not appear to require the hospitals to serve the 340B-specified low-income population, yet HRSA allowed the hospitals to begin participating in the 340B Program. Specifically, one hospital submitted a contract with a state government that was limited to providing services to beneficiaries of the state’s Medicaid program, although nongovernmental hospitals are to have contracts to provide services to individuals who are not entitled to Medicaid benefits. The other hospital submitted an agreement with a nonprofit company for management services, including accounting and payroll services, for their hospital and nursing home facilities. To participate in the 340B Program, nongovernmental hospitals must have a contract with a state or local government to provide health care services to the 340B-specified low-income population. Thus, allowing hospitals to participate when the state or local government contracts they submitted for review do not require them to serve this population is inconsistent with HRSA’s responsibilities for oversight of the 340B Program, including ensuring that participating hospitals meet the statutory eligibility requirements. Without amending its contract integrity checks to include verifying that newly registering hospitals have contracts that meet statutory eligibility requirements, HRSA risks allowing hospitals that are not eligible, and which may not be providing services to the 340B- specified low-income population, to participate in the 340B Program. Guidance for contract reviews during audits has not been consistently documented over time and lacks detailed instructions. Although HRSA officials told us they have always expected auditors to look for a contract through which a nongovernmental hospital would be eligible for the 340B Program, we found that HRSA’s guidance for auditors has not clearly documented these expectations and lacks detailed instructions. HRSA did not document key elements to look for— signatures, dates, and a requirement to serve the 340B-specified low- income population—in its guidance for auditors until August 2018. Further, the agency has made frequent changes to its guidance and procedures. For example, between November 2017 and July 2019, HRSA modified its guidance for auditors at least six times. In addition, HRSA’s guidance states that auditors are expected to perform a “simple logic test” to determine whether contracts require the hospital to serve the 340B- specified low-income population, but HRSA has not provided any additional information about how auditors are expected to conduct such a test. The guidance also advises auditors not to “dive too deep” when reviewing contracts. Of the 202 contracts submitted by hospitals as part of HRSA’s audits that we reviewed, 11 contracts (5 percent) did not appear to require hospitals to provide care to the 340B-specified low- income population, yet HRSA allowed the hospitals to continue their participation in the program. One such contract was a consent order that stated that the state’s attorney general would defer enforcement action based on the hospital’s agreement to abide by certain medical debt collection practices, such as adopting a zero tolerance policy for abusive, harassing, oppressive, false, deceptive, or misleading language or collections conduct. Furthermore, HRSA’s procedures for audits do not require auditors to separately affirm and record their review of the dates, signatures, and services required in the contracts. Thus, HRSA has no way of knowing whether auditors have checked and verified each of these elements. In addition to the 11 contracts that did not appear to obligate the hospitals to provide health care services to the 340B-specified low-income population, our review of 202 contracts submitted to HRSA by audited hospitals found 16 contracts (8 percent) were missing one or both signatures; 15 contracts (7 percent) were missing effective dates or were expired; at least 8 contracts had dates that did not cover the audit’s period of review, which includes a 6-month sample period before the start of the audit. For at least some of these contracts, HRSA was unaware of the issues we identified; HRSA did not issue audit findings in response to any of these contracts. HRSA has taken steps to address expired contracts. Specifically, in May 2019, HRSA revised its procedures for hospital registration and contract integrity checks to include language specifying that a hospital should not be approved for registration unless a contract is currently in place and that the contract must not expire before the participation start date. In addition, HRSA officials told us that in January 2020 the agency plans to implement a quarterly check of its 340B database to identify hospitals with expired state or local government contracts. However, these efforts do not address other date-related issues such as missing effective dates, or the issues with signatures or contract service requirements. Federal internal control standards related to control activities and enforcing accountability state that agencies should (1) implement control activities through policies, such as by documenting policies in the appropriate level of detail to allow management to effectively monitor the control activity; and (2) evaluate performance and hold individuals accountable for their internal control responsibilities, such as by communicating with the service organizations contracted to perform roles about the agency’s objectives and related risks, assigned responsibilities and authorities, and the expectations of competence to enable the service organization to perform its responsibilities. Without more specific guidance for auditors’ review of contracts, and procedures requiring auditors to separately document their review of each contract element, HRSA lacks reasonable assurance that the audits are appropriately identifying deficiencies in nongovernmental hospitals’ contracts with state or local governments. As a result, some hospitals appear to be participating in the 340B Program based on contracts that are inconsistent with program requirements or HRSA’s guidance. HRSA allows audited hospitals to avoid audit findings by entering into new contracts with state and local governments while audits are being conducted. As previously noted, our review of contracts submitted to HRSA by audited hospitals found that eight hospitals provided contracts that did not appear to cover the audit’s period of review. Three of the eight hospitals entered into the contracts while the audit was ongoing. According to HRSA policy, a hospital that does not demonstrate that it had a contract for the entire audit period should be issued a finding of noncompliance and held responsible for repayment to manufacturers for any discounts received improperly during the period for which it did not have a contract. However, HRSA did not issue such findings or penalties for any of the hospitals we identified with contracts that did not cover the audit’s period of review. For example, in one case, officials said HRSA had included a finding in its draft audit report, but withdrew it when presented with a new contract with an effective date made retroactive to cover the audit’s entire period of review. in another case, a hospital that had been government-owned was sold to a private company in 2013, but did not switch its classification to nongovernmental until 2015, and did not sign a contract with a state or local government until it was audited in fiscal year 2018. The hospital’s contract, signed in 2018, included a retrospective attestation that the hospital had been providing care for the 340B-specified population since 2013. HRSA officials told us that they accept such retroactive documentation in conjunction with current, valid contracts on a case-by-case basis. As such, a hospital may avoid findings, and potential repayments to manufacturers, by asserting that it had been providing care even when a contract was not in place. To participate in the 340B Program, a nongovernmental hospital is required by statute and HRSA policy to have a contract with state or local government to serve the 340B-specified low-income population. Allowing hospitals to submit retroactive contracts after they have already begun participation in the program is inconsistent with HRSA’s responsibilities for oversight of the 340B Program, including ensuring that participating hospitals meet the statutory eligibility requirements. Further, allowing hospitals that are unable to demonstrate that they have contracts in place that cover the audit’s period of review to continue to participate in the 340B Program without consequence undermines the effectiveness of HRSA’s audit process and increases the risk that ineligible hospitals will receive discounts under the program. The 340B Program allows hospitals and certain other providers to stretch federal resources to reach more eligible patients and provide more comprehensive services. Participation in the 340B Program also can be beneficial for hospitals and other covered entities as they can realize substantial savings on covered outpatient drugs and generate revenue on those drugs. Hospital participation in the 340B Program, and hospital purchases of discounted drugs through the 340B Program, has risen rapidly over time. However, HRSA’s current processes and procedures do not provide reasonable assurance that nongovernmental hospitals seeking to participate and benefit from the 340B Program meet the program’s eligibility requirements. Given the weaknesses in HRSA’s oversight, some hospitals that do not appear to meet the statutory requirements for program eligibility are participating in the 340B Program and receiving discounted prices for drugs for which they may not be eligible. Although HRSA has initiated some efforts to strengthen its processes for assessing hospitals’ eligibility, continued growth in the number of participating hospitals and 340B- purchased drugs highlights the need for HRSA to improve its oversight processes. This is critical to safeguarding the integrity of the 340B Program. We are making the following six recommendations to HRSA: The Administrator of HRSA should ensure that the information it uses to verify nonprofit status for all nongovernmental hospitals that participate in the 340B Program is reliable—for example, by requiring and reviewing the submission of official documentation hospitals must already maintain or by ensuring the reliability of the data the agency uses. (Recommendation 1) The Administrator of HRSA should implement a process to verify that every nongovernmental hospital that participates in the 340B Program has a contract with a state or local government as required by statute. (Recommendation 2) The Administrator of HRSA should amend its contract integrity check procedures for the 340B Program to include a review of whether hospitals’ contracts with state and local governments require the provision of health care services to low-income individuals not eligible for Medicaid or Medicare as required by statute, and should provide guidance for staff to conduct these reviews. (Recommendation 3) The Administrator of HRSA should provide more specific guidance for 340B Program auditors on how to determine if nongovernmental hospitals’ contracts with state and local governments require the provision of health care services to low-income individuals not eligible for Medicaid or Medicare. (Recommendation 4) The Administrator of HRSA should revise its 340B Program audit procedures to require auditors to document their assessments of whether nongovernmental hospitals’ contracts with state and local governments are appropriately signed, cover the time periods under review, and require hospitals to serve low-income individuals not eligible for Medicaid or Medicare, such as by requiring auditors to separately affirm and record their review of each of these elements. (Recommendation 5) The Administrator of HRSA should require nongovernmental hospitals participating in the 340B Program to demonstrate that they have contracts with state or local governments in effect prior to the beginning of their audits’ periods of review and should apply consistent and appropriate consequences for hospitals that are unable to do so. (Recommendation 6) HHS provided written comments on a draft of this report, which are reproduced in appendix II, and technical comments, which we have incorporated as appropriate. In its written comments, HHS concurred with five of our six recommendations; it did not concur with one of them. In concurring with five of our recommendations, HHS stated that HRSA is evaluating its audit process and other program integrity efforts, and noted that HRSA has made improvements to strengthen its program integrity efforts that align with some of our recommendations. With respect to our recommendation to require auditors to document their assessments of the required elements of contracts, HHS concurred and noted that HRSA updated its audit procedures. Specifically, HRSA’s draft procedures for fiscal year 2020 audits require auditors to specify if the hospital provided a contract that includes the names and signatures for both the hospital and government agency, effective dates that cover the entire audit period, and that requires the provision of services to the 340B-specified low- income population. We are pleased that HRSA has already taken this step to implement our recommendation. To fully implement this recommendation, HRSA should incorporate these changes into its final audit procedures for fiscal year 2020. HHS also concurred with our recommendation to require nongovernmental hospitals to demonstrate that they have contracts in effect prior to the beginning of the audits’ periods of review, and to apply consistent and appropriate consequences if they do not. Also, as noted above, HRSA has updated its draft audit procedures to specify that auditors should look for effective dates that cover the entire audit period. While this is an important step, HRSA must also show that it has applied consistent and appropriate consequences when auditors find that nongovernmental hospitals did not have contracts in effect prior to the beginning of their audit periods. On a related issue, HHS expressed concern over and disagreed with our finding that HRSA allows hospitals to avoid audit findings by entering into new contracts while audits are being conducted, noting that HRSA assesses potential audit findings on a case-by-case basis to ensure that any necessary steps are taken to address issues. However, as we reported, HRSA officials have indicated that they accept retroactive contract documentation on a case-by-case basis; we continue to believe that this practice—accepting new contracts that are retroactive— effectively allows hospitals to avoid audit findings. In addition, while we agree that working with hospitals to address noncompliance is appropriate, we continue to believe that such efforts should be in addition to, not instead of, documenting noncompliance by issuing findings and applying appropriate consequences, in accordance with HRSA’s audit policies and procedures. To do otherwise undermines the integrity of HRSA’s audits, and increases the risk that ineligible hospitals will receive discounts under the program. HHS also concurred with our recommendation to ensure that the information HRSA uses to verify nonprofit status is reliable, but stated that HRSA believes that the information it uses from hospitals’ Medicare cost reports is reliable, because hospital administrators attest to the accuracy of their cost reports. However, as discussed in our report, neither HRSA nor CMS has evaluated the reliability of the cost report data for verifying nonprofit status, and a CMS official responsible for oversight of the cost reports told us that the question on the cost report is not clearly defined and may not be reported accurately. As such, we continue to believe that HRSA needs to assess the reliability of the Medicare cost report data should it continue to use those data for determining hospitals’ nonprofit status. Alternatively, HRSA could require hospitals to submit documentation of their nonprofit status, such as Internal Revenue Service documents, which HRSA acknowledged hospitals are required to maintain as part of their auditable records. HHS did not concur with our recommendation to implement a process to verify that every nongovernmental hospital that participates in the 340B Program has the statutorily required contract with a state or local government. HHS noted that it has requested authority to require hospitals registering for the 340B Program to submit documentation supporting the hospital classification that they select during registration. According to HHS, if approved, HRSA would begin collecting and reviewing contracts from all newly registering nongovernmental hospitals. However, HHS stated that HRSA does not have the resources to collect, review, and verify that every participating nongovernmental hospital has a contract with a state or local government. While we understand that verifying the existence of contracts for all participating nongovernmental hospitals would require additional effort on HRSA’s part, our review found that relying on hospitals’ attestations is not sufficient to ensure hospitals’ eligibility. Additionally, implementing a process to verify the existence of a contract does not necessarily require that HRSA collect and review contracts from every hospital. There are other potential options, such as obtaining confirmation from the state or local government that they indeed have a contract with the hospital to provide services to the 340B-specified low-income population. HHS also commented that implementing our recommendation would create a significant burden on covered entities. However, as we noted in our report, HRSA already requires hospitals to maintain copies of their state or local government contracts. Therefore, it is unclear how implementing a process to verify the existence of those contracts would represent a significant burden for nongovernmental hospitals already registered for the program. Ensuring the eligibility of covered entities that participate in the 340B Program is essential for program integrity. As such, we continue to believe that HRSA needs to take action, beyond relying on hospitals’ self-attestations, to verify that all participating nongovernmental hospitals have contracts with state or local governments that meet the statutory requirements of the program. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of HHS, the Administrator of HRSA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or draperd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other major contributors to this report are listed in appendix III. Table 1 provides information about the 240 contracts between hospitals and state or local governments that were included in our review, including information about the type of hospital and the level of government that were parties to the contract. In at least two cases, the hospitals contracted with other health care providers who were themselves 340B Program participants, such as a community health center operated by a local health department. Officials signing on behalf of state and local governments included individuals with executive positions, such as the heads of state agencies, mayors, and county executives, but also included a city alderman, a vice-chancellor for finance at a state university health system, and a juvenile court judge. In addition to the contact named above, Michelle Rosenberg, Assistant Director; Hannah Locke, Analyst-in-Charge; Jennie Apter; George Bogart; Kaitlin Farquharson; Matthew Green; Vikki Porter; Daniel Ries; Brienne Tierney; and William T. Woods made key contributions to this report.", "summary": "The number of nongovernmental hospitals participating in the 340B Program has grown over time. HRSA requires participating hospitals to register and annually recertify their eligibility. HRSA also reviews hospitals' eligibility through audits of a small sample each year. GAO was asked to provide information on 340B-participating hospitals' contracts with state and local governments. This report (1) describes any obligations in selected nongovernmental hospitals' contracts to serve low-income individuals, and (2) examines HRSA's processes to assess nongovernmental hospitals' eligibility. GAO examined contract documentation from all 258 nongovernmental hospitals HRSA reviewed in 2017 and 2018; and HRSA's policies, procedures, and guidance related to 340B hospital eligibility. GAO also interviewed HRSA officials. Under the 340B Drug Pricing Program (340B Program), administered by the U.S. Department of Health and Human Services' (HHS) Health Resources and Services Administration (HRSA), drug manufacturers provide discounted prices on outpatient drugs to certain hospitals and other entities. About two-thirds of hospitals participating in the 340B Program (approximately 1,700) are nongovernmental hospitals (private, nonprofit hospitals), which qualify for the program, in part, based on having contracts with state or local governments to provide health care services to the 340B-specified low-income population—low-income individuals not eligible for Medicaid or Medicare. GAO's review of contract documentation for 258 nongovernmental hospitals found that most contracts obligated these hospitals to provide health care services to low-income individuals. However, few of the contracts reviewed included details about those obligations, such as the amount or type of care hospitals were required to provide. The statute does not require the contracts to contain such details. GAO found that HRSA's processes do not provide reasonable assurance that participating nongovernmental hospitals meet eligibility requirements. For example, HRSA primarily relies on hospitals' self-attestations to verify the existence of contracts with state and local governments. The agency reviewed contract documentation for less than 10 percent of nongovernmental hospitals per year in 2017 and 2018. GAO also identified several weaknesses in HRSA's review of the nongovernmental hospital contracts: HRSA does not conduct reviews to determine whether the documents submitted by nongovernmental hospitals are actual contracts, namely that they are mutually binding agreements to provide services or supplies in exchange for something of value. GAO found that 18 of the 258 hospitals reviewed submitted documents that did not appear to be contracts, such as descriptions of community programs, yet all of these hospitals were permitted to participate in the program. When audits have identified hospitals that did not have contracts in place throughout the audits' periods of review, HRSA has allowed hospitals to avoid audit findings by, for example, entering into new contracts with retroactive start dates. This practice undermines the integrity of HRSA's audits. HRSA's contract reviews do not always include assessments of whether contracts are consistent with the statutory requirement to provide health care services to the 340B-specified low-income population and HRSA's guidance for conducting such assessments, when required, lacks detailed instructions. As a result, GAO found that contracts for 13 hospitals reviewed did not appear to require hospitals to serve the 340B-specified low-income population. Despite this, these 13 hospitals were permitted to participate in the program. Given these weaknesses, some nongovernmental hospitals that do not appear to meet the statutory requirements for program eligibility are participating in the 340B Program and receiving discounted prices for drugs for which they may not be eligible. GAO is making six recommendations, including that HRSA implement a process to verify that all nongovernmental hospitals have contracts in place, including throughout hospitals' audit periods; amend its contract reviews to include an assessment of whether contracts meet statutory requirements; and provide better guidance on contract reviews. HHS concurred with all of the recommendations except the one to implement a process to verify that all nongovernmental hospitals have contracts. GAO continues to believe this action is needed to ensure that only eligible hospitals are allowed to participate in the 340B Program.", "document_type": "gao"}
{"report": "As shown in figure 2, a number of activities led up to OMB publishing the reform plan in June 2018, and subsequently OMB has provided updates on the proposals in the reform plan. In March 2017, the President issued an executive order requiring comprehensive reorganization plans for executive branch agencies. In April 2017, OMB provided guidance to federal agencies for developing their respective reform plans. According to this guidance, the government-wide reform plan was to have been based on the agency reform plans, OMB-coordinated crosscutting proposals, and public input. In addition, OMB’s guidance indicated that OMB would track the progress of the reforms in coordination with the President’s Management Council. OMB’s guidance also stated that it would track progress of the reforms by leveraging the federal performance planning and reporting framework that was originally put into place by the Government Performance and Results Act of 1993 (GPRA), and significantly enhanced by the GPRA Modernization Act of 2010 (GPRAMA). Accordingly, OMB’s guidance explained that progress would be tracked through the use of cross- agency priority (CAP) goals, agency priority goals, and Performance.gov. In March 2018, OMB released the President’s Management Agenda, which identified a set of CAP goals, required under GPRAMA. The CAP goals target areas where multiple agencies must collaborate to effect change, and agencies must report CAP goal progress in a manner the public can easily track. In June 2018, the administration released its government-wide reform plan, Delivering Government Solutions in the 21st Century: Reform Plan and Reorganization Recommendations (reform plan). In July 2019, the administration reported on the first year of progress toward its reform proposals. According to the 1-year update, the President’s Fiscal Year 2020 Budget included 18 of the proposed reform proposals in whole, or in part, and also described administrative actions by agencies to implement more than 20 of its 32 proposals. Of these proposals, the administration reported progress toward four of the five reforms we selected for review: (1) moving personnel security clearance background investigations from OPM to DOD; (2) solving the cybersecurity workforce shortage; (3) establishing the GEAR Center; and (4) reorganizing OPM. OMB officials said that they are not planning to move forward with the customer experience improvement capability reform during fiscal year 2020 because they are pursuing other customer experience activities, such as those included in the CAP goal for Improving Customer Experience with Federal Services. We added the government-wide personnel security clearance process to our High-Risk List in January 2018 because it continues to face challenges in the timely processing of clearances, measuring the quality of investigations, and ensuring the security of related information technology (IT) systems. The National Defense Authorization Act (NDAA) for Fiscal Year 2018 included provisions that resulted in the transfer of background investigations from OPM’s National Background Investigations Bureau (NBIB) to DOD for certain DOD personnel, which represented approximately 70 percent of all federal background investigations performed by NBIB. Subsequently, the selected reform proposal recommended moving the remaining 30 percent of investigations to DOD. According to the reform plan, this transfer would provide an opportunity to conduct the background investigations more efficiently and economically than having them be performed by separate agencies. In January 2019, DOD formally established the Personnel Vetting Transformation Office (PVTO) to implement and oversee activities related to the transfer of NBIB functions. In April 2019, the President issued Executive Order 13869 which generally provided for the transfer of the remaining background investigation operations from OPM to DOD. The executive order also called on the Secretary of Defense to enter into an agreement with the Director of OPM to set forth expectations and designate the appropriate support functions for the transfer. As directed, in June 2019, OPM and DOD signed an interagency memorandum that set forth expectations for activities necessary for the transfer of functions of NBIB and associated employees and resources from OPM to DOD, including measurable deliverables, key considerations for executing deliverables, and processes for coordination and governance. According to documents we received from DOD, the transfer of NBIB functions to DOD occurred by September 30, 2019, as required by the April executive order. As shown in figure 3, OMB, OPM, and DOD have generally addressed most key reform practices in implementing the transfer of background investigations from OPM to DOD. According to DOD, more than 99 percent of NBIB employees, totaling 2,979 individuals, accepted positions transferring them to DOD’s Defense Counterintelligence and Security Agency (DCSA) by September 30, 2019. According to the DOD’s PVTO Director, 17 individuals chose not to transfer, and instead retired as permitted. Going forward, we will continue to monitor the government- wide personnel security clearance process as part of our work to identify and assess high-risk issues across the government. OMB, OPM, and DOD have generally addressed key practices related to establishing goals and outcomes. The NDAA for Fiscal Year 2018 and Executive Order 13869 established a goal and related requirements for the transfer of OPM’s NBIB personnel, resources, and functions to DOD. Specifically, the executive order established a goal to complete the transfer of all NBIB administrative and operational functions to DOD by September 30, 2019. The executive order also outlined a series of deliverables and objectives for OMB, OPM, and DOD to achieve during the transfer. For example, the executive order required DOD to execute a written agreement with OPM to establish expectations for the transition period related to detailing personnel, safeguarding information technology, contracting, and funding background investigations, among others. OPM and DOD achieved their intended goal, and as of September 30, 2019, DOD is the primary provider of national security background investigations for the federal government. As directed, OPM and DOD signed an interagency agreement in June 2019 to address expectations, including governance, information technology, contracting, and funding issues, among others. According to documents provided by DOD, the Transfer Tollgate group and the Executive Steering Committee provided interagency leadership including an executive-level decision venue for implementation, resourcing, and other decisions. According to DOD, these interagency groups also provided accountability for implementation milestones. Under the leadership of these two groups, DOD officials shared with us that they worked with OPM to resolve a host of issues such as the transfer of personnel, funding for transfer costs, transfer of information technology assets, financial management issues, and acquisition concerns, among other critical issues. To help address differences in the financial management and funding of background investigations between OPM and DOD, the agreement required DOD to establish a Working Capital Fund to fund DCSA’s background investigation mission by September 1, 2019. According to DOD officials and the agency’s Transfer Status Dashboard, the Working Capital Fund was established prior to the September 1, 2019, deadline; and, as of October 7, 2019, the fund had a balance of approximately $1 billion. Neither the NDAA for Fiscal Year 2018 nor the executive order outlined measurable outcomes related to the efficient and effective delivery of background investigations, but rather goals and deliverables related to transferring NBIB functions to DOD, among other things. According to DOD officials we spoke with, the reform’s objective was the timely transfer of background investigation functions and coordination between affected agencies and stakeholders. DOD officials explained that following completion of the transfer, on October 1, 2019, PVTO, in coordination with other DOD components and federal stakeholders, began work transforming DCSA’s processes and procedures, including the background investigation process, to improve outcomes. OMB, OPM, and DOD generally addressed key practices related to involving employees and key stakeholders. OPM and DOD generally communicated with affected employees and key stakeholders and involved them in the implementation of the transfer of NBIB functions to DOD. Agencies’ communication included email correspondence to affected staff from agency leaders including OPM’s Acting Director, and the Director of NBIB. These emails provided NBIB staff regular updates on the status and details surrounding the transfer of NBIB functions to DOD. Based on documents provided by OPM, communication to affected staff began in June 2017, informing staff that Congress was considering a legislative proposal to move certain NBIB functions to DOD. According to documents we received, communication with staff has continued regularly since this time, including a July 29, 2019, message to affected staff with an official notice that NBIB employees would be offered an appointment to DOD’s DCSA effective September 29, 2019. This notice explained that OPM’s NBIB employees accepting this appointment would transfer to DOD without changes to their duty stations, grades, or benefits. In addition to email communication, in-person town hall meetings were held between agency leaders and affected staff to provide updates on the status of the transfer and answer questions. According to OPM’s then NBIB Director, a July 2017 town hall was held addressing the congressional proposal to move the majority of NBIB staff to DOD. The Director also reported that OPM and DOD had worked via meetings, information exchanges, site visits, and communication at all levels in the organization to assemble information on the implication of the transfer and its potential impacts. OPM officials testified at a number of hearings in 2018 and 2019 related to the transfer, and OPM officials told us that they joined DOD in providing quarterly briefings required by the NDAA for Fiscal Year 2018, on the status and progress of the transfer. DOD and OPM also developed a Joint Transfer Plan that described strategic communication activities with affected employees, contractors, and other stakeholders including public media outlets, ourselves, and state and local law enforcement agencies, among others. DOD officials at the PVTO explained that they developed a more detailed communication plan in March 2019 that was implemented prior to the transfer. OMB, OPM, and DOD have partially addressed key practices related to addressing high-risk areas and longstanding management challenges. As previously mentioned, we placed the government-wide personnel security clearance process on our High-Risk List because of continuing challenges in the timely processing of clearances, measuring the quality of investigations, and ensuring the security of related IT systems. While OMB and lead agencies have considered our related high-risk work, the reform proposal and implementation plans do not demonstrate how the transfer and delegation of background investigation functions from OPM to DOD will address these challenges. Moreover, in November 2019, OPM’s Inspector General identified the background investigation legacy information systems as an ongoing top management challenge that will need to be addressed by both OPM and DOD moving forward. The Director of the PVTO told us that the office’s initial goal was to ensure a smooth and timely transition of functions from OPM’s NBIB to DOD by the beginning of fiscal year 2020. The Director also told us that after the transfer occurred, the office would shift its focus to address our high-risk area by, among other things, transforming these security clearance services to optimize processes government-wide. Specifically, the PVTO charter established a goal to “identify efficiencies to be gained, areas where the organizational structure and business services may be incomplete, maximize synergy where possible, and propose mitigation strategies to address gaps and shortfalls.” We will continue to monitor the government’s progress toward addressing security clearance challenges as part of our work to track high-risk issues across the government. OMB, OPM, and DOD have generally addressed key practices related to leadership focus and attention. In particular, Executive Order 13869 outlined the roles and responsibilities of OMB, OPM, and DOD, and authorized a new office (PVTO) to assist in the execution of the transfer. The executive order also clarified agencies’ roles and requirements for coordinating the transfer, delegation, and other activities. Specifically, the executive order directed the Secretary of Defense and the OPM Director, in consultation with the OMB Director and the Security Executive Agent, to provide for the transfer of the bulk of OPM’s investigative functions to the DCSA, along with any appropriate OPM-associated personnel and resources, including infrastructure and certain investigation-related support functions. With regard to a dedicated implementation team, PVTO was responsible for ensuring coordination and resource alignment during the transfer, as well as ensuring that personnel security background investigations continued without disruption during the transfer. The PVTO Director told us in July 2019 that his team reports directly to the Office of the Under Secretary of Defense for Intelligence. The Director stated that he has experience in the areas of acquisitions, mergers, and reorganizations, and has support from experts and top leadership throughout the department. In addition, the PVTO charter states that the office be composed of employees with extensive experience and expertise in personnel vetting processes and reform efforts, as well as business and technology innovation, program evaluation, acquisitions and mergers, and organization and change management. OMB, OPM, and DOD have generally addressed key practices related to managing and monitoring. Specifically, PVTO developed a joint transfer plan outlining critical assumptions for the transfer, major activities, and time frames across nine functional areas, including personnel, training, information technology, financial management, acquisitions, strategic communications, and security, among others. For each functional area, the transfer plan provided a summary of the functional area’s objective and a set of recommended major activities. For example, the functional area for IT had an objective to provide secure, current hardware and software in compliance with DOD and federal standards, and promote the unique requirements of a highly mobile, geographically dispersed workforce managing significant volumes of personally identifiable information and other sensitive data. Major activities included: (1) the transfer of IT infrastructure, (2) the completion of a gap analysis to determine which NBIB systems and hardware are transferrable or require new acquisitions, and (3) the provision of secure devices that support mobile operations. The PVTO Director also showed us a detailed implementation plan organized around the nine functional areas identified in the broader joint transfer plan. The implementation plan tracked thousands of activities and provided a detailed timeline for completion. The Director also provided us a dashboard that his team used to track implementation progress. The Director told us that his office used the dashboard to manage and monitor the transfer daily. The dashboard allowed the implementation team to identify areas where attention was needed using red, yellow, and green stoplight indicators signaling the status of major objectives. The annual assessments of timeliness and quarterly briefings required by the NDAA for Fiscal Year 2018 also serve as mechanisms for Congress and the executive branch to monitor timeliness, costs, and continuous evaluation, among other things. OMB also publishes quarterly milestone progress and metrics on the related Security Clearance, Suitability, and Credentialing Reform cross-agency priority goal on Performance.gov. OMB, OPM, and DOD have generally addressed key practices related to employee engagement. In addition to the communication and outreach activities described above, OPM and DOD have undertaken additional efforts to engage affected employees and monitor levels of employee engagement at both agencies. For example, according to DOD officials, to engage and communicate with affected employees the agency held several town hall meetings to provide information and answer questions. They also said that DOD leadership regularly emailed affected staff providing updates on the status of the transfer and held separate question-and-answer sessions to keep staff informed and engaged. According to PVTO planning documents, the office also developed a strategy to achieve stakeholder buy-in through empowering leaders and through efforts to build a coalition of stakeholders around a common vision for the future of the background investigation function at DOD. In April 2019, OPM also conducted an internal survey of agency staff to collect information on employees’ perceptions of the transition to DOD, personal work experiences, satisfaction with their job, and any intent to leave DOD and reasons for leaving. The survey asked NBIB employees the extent to which they felt informed about the upcoming transition to DOD. According to the roughly one-third of staff who responded, 35 percent felt extremely or moderately informed, 32 percent felt somewhat informed, and 33 percent felt slightly or not at all informed. Approximately 75 percent of the survey respondents reported that they had enough information to do their job well, and 74 percent reported that they were proud to tell others they worked at their organization. When asked about satisfaction with involvement with decisions that affect their work, 38 percent of respondents were positive, 34 percent were neutral, and 28 percent were negative. OPM officials told us that they continued to monitor engagement of NBIB staff throughout the transition. OPM and DOD have partially addressed key practices related to strategic workforce planning. In March 2019, we reported that, to make progress on removing the Government-wide Personnel Security Clearance Process from our High-Risk List, OPM and DOD should develop and implement a comprehensive strategic workforce plan that identifies the workforce needed to meet the current and future demand for its services, as well as reduce the current backlog to a manageable level. OPM completed this action in September 2019 with the release of the NBIB Strategic Workforce Plan for the Background Investigation Mission. The strategic workforce plan includes initiatives to strengthen investigative workforce capacity and training, promote the use of different hiring authorities, and provide succession planning, among other initiatives. According to the plan, senior leadership will build upon the strategic workforce plan to create an implementation strategy. While OPM has taken action, DOD has yet to complete its workforce plan. As of October 2019, DOD’s strategic workforce plan for the new DCSA enterprise was under development. In response to our request for information, OPM, DOD, and OMB provided information regarding the authorities they are using to implement the reform proposal to move all background investigations from OPM to DOD. According to OPM and OMB, the legal authorities by which NBIB moved to DOD consisted of section 925 of the National Defense Authorization Act for Fiscal Year 2018 (NDAA 2018) and Executive Order 13869, issued in April 2019, which re-designated DOD’s DCSA as the primary investigative service provider for national security investigations. OPM also cited 5 U.S.C. § 1104, which permits OPM to delegate certain personnel management functions to other agencies. Section 925 of the NDAA 2018 authorized DOD to conduct its own background investigations and required DOD to begin carrying out an implementation plan required under the National Defense Authorization Act for Fiscal Year 2017 (NDAA 2017) by October 1, 2020. The NDAA 2018 also required the Secretary of Defense, in consultation with the OPM Director, to provide for a phased transition of DOD background investigations from OPM to DOD. According to OPM, the DOD background investigations, consisting of investigations for civil service, military, contract, and non-appropriated fund personnel, constitute approximately 70 percent of the work performed by NBIB. Executive Order 13869 provided for the transfer of the primary responsibility for conducting national security background investigations, government-wide, from OPM to DOD. The executive order designated DOD, rather than OPM, as the agency to serve as the primary entity for conducting background investigations for national security adjudications, pursuant to and consistent with the NDAA 2018 and section 3001(c) of the Intelligence Reform and Terrorism Prevention Act of 2004 (IRTPA). According to OPM, this has the effect of moving the remaining national security investigations, not already transferred by section 925 of the NDAA 2018, to DOD. The Executive Order also acknowledged that OPM will delegate, pursuant to 5 U.S.C. § 1104, other background investigation functions to DOD for non-DOD personnel, such as investigations performed to enable the adjudication of the subject’s suitability or fitness for federal employment, eligibility for logical or physical access to systems and facilities, fitness to perform work for a federal agency under a government contract, and fitness to work as a nonappropriated fund employee. In accordance with the executive order, DOD and OPM signed an agreement on June 25, 2019 that set forth the expectations for necessary activities for the transfer of functions of the NBIB from OPM to DOD. The agreement provided that the period of transition was from June 24, 2019, through September 30, 2019. The agreement covered such areas as personnel, information technology, facilities and property, contracting, administrative support, records access, claims, and funding. This reform proposal directs OMB and DHS, in coordination with other agencies, to prioritize and accelerate efforts to recruit, evaluate, hire, pay, and distribute cybersecurity talent across the federal government. Ensuring the cybersecurity of the nation is a longstanding challenge that has been on our High-Risk List for more than two decades. Efforts to solve the cybersecurity workforce shortage will help to address a number of high-risk issues we have previously identified. To accomplish the objective of filling cybersecurity vacancies, the reform lays out a series of projects and activities intended to identify and close workforce skills gaps and develop a standardized approach to hiring, training, and retaining qualified cybersecurity professionals. Specifically, the proposal calls for: identifying and categorizing the federal cybersecurity workforce using the National Initiative for Cybersecurity Education Cybersecurity Workforce Framework (NICE framework), implementing DHS’s Cyber Talent Management System (CTMS) with options to expand the capability across the government, rationalizing and expediting the security clearance process, standardizing training for cybersecurity employees, increasing the mobility of cybersecurity positions, developing plans to establish a cybersecurity reservist program to provide needed surge capacity, reskilling federal employees to fill critical cyber positions, and rationalizing the size and scope of federal cybersecurity education programs. As shown in table 2, OMB, DHS, and other federal agencies have made progress in implementing certain projects and activities included in the reform proposal. Although OMB and DHS have several projects and activities underway related to this reform, they did not provide us with information about the government-wide goals or implementation plans for the proposal. In November 2019, OMB staff told us that they did not have additional information to share regarding their application of key reform practices because they are still developing this reform. As a result, we found that most of the key reform practices were partially met (see figure 4). We did obtain information showing that OMB and DHS addressed key practices for some of the projects and activities included in the reform proposal, but the extent to which these practices were being applied to the reform proposal as a whole, or being coordinated government-wide, was unclear. OMB and DHS have partially addressed key practices related to establishing goals and outcomes. We found that this reform established an objective to solve the cybersecurity workforce shortage across the government, and DHS established outcome-oriented goals and performance measures for certain agency-specific projects that are part of the reform. For example, as shown in table 2, DHS established a measure to hire at least 150 cybersecurity professionals at the agency during fiscal year 2020 using its new Cyber Talent Management System. In addition, DHS provided us its 2017 Comprehensive Cybersecurity Workforce Update, which includes an array of data and analysis, including cybersecurity workforce trends, metrics on DHS components’ vacancies, attrition, capacity gaps, hiring, and other information describing the status of the agency’s cybersecurity workforce. The administration also released a National Cyber Strategy in September 2018 outlining broad activities related to the government-wide reform such as building a talent pipeline, reskilling employees, and improving the process of recruiting and retaining qualified cybersecurity professionals. These documents may provide a first step toward developing clear outcome-oriented goals and performance measures for the reform as a whole. However, OMB and DHS have not yet established measurable outcome- oriented goals for the government-wide projects and activities outlined in the reform proposal. For example, there are not government-wide measurable goals for hiring cybersecurity professional across the government, reductions to attrition, training, or other aspects of the reform. As shown in table 2, OMB and agencies have made progress on a number of areas related to the reform; however, without establishing government-wide measurable goals and outcomes, OMB and DHS will not be able to determine whether progress is being made across the federal government to solve the cybersecurity workforce shortage. OMB and DHS have partially addressed key practices related to involving employees and key stakeholders. We obtained information on targeted outreach to employees and stakeholders for certain projects and activities outlined in the reform, but as of November 2019, OMB and DHS did not have information on how they were addressing these key practices for the reform as a whole. For example, DHS’s Cybersecurity and Infrastructure Security Agency (CISA) officials told us that they participate in interagency coordination activities related to the NICE framework with OMB, the Department of Commerce National Institute of Standards and Technology (NIST), the Federal Chief Information Officers Council, and outside stakeholders. CISA officials said they worked with government and industry stakeholders to develop the NICE framework, and are working with educators and certification vendors to help build a pipeline of cybersecurity talent. Additionally, in March 2019, we reported that OPM and NIST coordinated with academia and the private sector to develop a cybersecurity coding structure that aligns with the work roles identified in the NICE framework. While OMB and DHS conducted outreach for certain projects and activities included in the reform proposal, it is unclear what, if any, outreach occurred for other projects. However, without a government- wide or project-by-project plan for communicating with and involving employees and stakeholders across the government, OMB and lead agencies will not know if certain agencies or employee groups are being adequately involved and informed. We have previously reported that creating an effective, ongoing communication strategy is essential to implementing a government-wide reform. The most effective strategies involve communicating early and often, ensuring consistency of message, encouraging two-way communication, and providing information to meet the specific needs of affected employees. This reform will be more likely to achieve its intended objective if OMB and DHS establish effective lines of communication with affected federal employees and the broader cybersecurity community. OMB and DHS have partially addressed key practices related to addressing high-risk areas and longstanding management challenges. We have designated information security as a government-wide high-risk area since 1997. We expanded this high-risk area in 2003 to include protection of critical cyber infrastructure and, in 2015, to include protecting the privacy of personally identifiable information. OMB and DHS generally considered areas that we previously identified as high-risk. OMB staff and DHS officials told us that they considered our high-risk reports when developing reform proposals, and have provided some documentation of these considerations. For example, OMB’s former Deputy Director for Management stated that, when developing the Solving the Cybersecurity Workforce Shortage reform, OMB used our 2017 High-Risk Series and noted that of more than 2,500 past recommendations, about 1,000 still needed to be implemented. OMB also identified several of our reports that touch on cybersecurity workforce issues. Although OMB and DHS have considered our prior work, as of November 2019, they had not demonstrated how the projects and activities outlined in the reform proposal would address our related high-risk issues and open recommendations. Without more detailed information describing how our high-risk issues are being addressed across the reform projects and activities, it is unclear which issues and recommendations are being targeted, and which are outside of the scope of this reform. OMB and DHS have partially addressed key practices related to leadership focus and attention. In May 2019, the President issued Executive Order 13870 requiring federal agencies to take a variety of actions related to cybersecurity, including efforts to enhance the mobility of cybersecurity practitioners, support the development of cybersecurity skills, and create organizational and technological tools to maximize cybersecurity talents and capabilities. Many of the actions outlined in this executive order align with the stated objectives and components outlined in the reform proposal. However, neither OMB nor DHS have created a dedicated team with necessary resources to manage and implement this reform on a government-wide scale. Moreover, DHS staff we spoke with told us that OMB was the government-wide lead for this reform, and their agency was responsible for a subset of the projects and activities outlined in the reform proposal. OMB staff did not provide us with any plans or other documents regarding the individuals or team responsible for implementation across the government. OMB staff explained that DHS’s CISA and the Federal Chief Information Security Officer Council have some responsibility for federal cybersecurity workforce issues; however, they did not clarify which organization, team, or individuals were responsible for coordinating and implementing the reform government- wide. Our prior work has shown that establishing a strong and stable team that will be responsible for the transformation’s day-to-day management is important to ensuring that it receives the resources and attention needed to be successful. A dedicated leadership team responsible for overseeing and implementing the reform can also help ensure that various change initiatives are sequenced and implemented in a coherent and integrated way. OMB and DHS have partially addressed key practices related to managing and monitoring. DHS has developed some agency-specific implementation plans and mechanisms to monitor progress. For example, DHS provides progress updates to Congress related to its continued efforts to code cybersecurity positions and to review the readiness of the cybersecurity workforce to meet DHS mission requirements, among other agency-specific assessments. However, OMB and DHS have not yet developed a government-wide implementation plan with goals, timelines, key milestones, and deliverables for the reform proposal as a whole. As previously discussed, OMB staff told us that they did not yet have a government-wide reform plan because they are still developing this reform. Without a government-wide implementation plan to track and communicate implementation progress, OMB and DHS will be unable to determine whether the reform is achieving its intended objectives, or whether unanticipated challenges or negative workforce trends are impeding efforts to close the cybersecurity workforce gaps across the government. OMB and DHS have not addressed key practices related to employee engagement. In February 2019, DHS officials told us that the agency had not yet reached the stage of implementation for its projects and activities where they were considering employee engagement in this reform. According to DHS officials, they have started collecting data on employees, but have not interacted with individual employees on specific reform initiatives. As of November 2019, OMB had not provided information on its efforts to engage affected employees across the government on this reform. We have reported that employee engagement affects attrition, absenteeism, and productivity. Moreover, we have found that failure to adequately address a wide variety of people and cultural issues, including employee engagement, can also lead to unsuccessful change. We identified six key drivers of engagement based on our analysis of selected questions in the Federal Employee Viewpoint Survey, such as communication from management. Given that the objective of this reform is to address a critical workforce skills gap, it is important that OMB and DHS remain attentive to the engagement levels of cybersecurity employees across the government to ensure that productivity and morale are not adversely affected. As previously discussed, OMB and DHS lack a government-wide or project-by-project plan for communicating with and involving employees across the government. Such a communications strategy could be used to inform and, as appropriate, involve employees on implementation of the reform. OMB and DHS have partially addressed key practices related to strategic workforce planning. As set forth in the Cybersecurity Workforce Assessment Act, DHS developed and published its Cybersecurity Workforce Strategy for 2019 through 2023. DHS’s strategy contains a 5- year implementation plan and a set of goals and objectives. Goals and objectives include an analysis of DHS’s cybersecurity workforce needs, a multi-phase recruitment strategy, professional and technical development opportunities, and plans to develop a talent management system, among others. OMB and DHS have yet to develop a government-wide cybersecurity strategic workforce plan that addresses the needs of all federal agencies. However, because this reform is focused on addressing a government- wide workforce shortage, it is particularly important that OMB and DHS complete their efforts to develop a strategic workforce plan for cybersecurity professionals that takes into account existing workforce capabilities, workforce trends, and shortages across the government. Without this information, DHS and OMB will not be able to determine if they are making progress or when they have addressed the government’s cybersecurity workforce shortage. DHS identified some existing legal authority for implementing aspects of the reform proposal, but neither DHS nor OMB provided us with a legal analysis for full implementation of the reform. OMB’s General Counsel stated, in a November 2019 letter to us, that OMB continues to collaborate with DHS and other federal agencies on a wide range of measures to address the cybersecurity workforce shortage. OMB stated that efforts had been within the confines of various current laws and appropriations, and that new legislation had not been required for any of these efforts. OMB did not provide additional details on the existing legal authorities on which it is relying. OMB also stated that the administration would seek legislation for any efforts beyond the scope of what is permitted under current law. DHS identified activities it is currently implementing related to the reform proposal that were previously authorized or required by law. For example, the CISA Chief Counsel identified DHS’s effort in establishing the forthcoming Cyber Talent Management System (CTMS) as a reform activity authorized by statute. The Chief Counsel noted DHS was authorized to establish this new personnel system for recruitment and retention of cybersecurity workers by the Border Patrol Agent Pay Reform Act of 2014. Under the act, DHS may establish cybersecurity positions; appoint personnel; fix rates of pay; and provide additional compensation, incentives, and allowances, subject to certain restrictions. The authority to implement this new system, however, is limited to DHS, and DHS officials acknowledged that CTMS cannot be implemented government-wide without statutory authorization. Additionally, DHS officials identified work being conducted at DHS to identify and categorize cybersecurity workforce positions, another activity related to the reform proposal and required by statute. Specifically, DHS was required by the Homeland Security Cyber Workforce Assessment Act of 2014, to: identify all cybersecurity workforce positions, determine the cybersecurity work category and specialty area of such assign data element codes developed by OPM in alignment with the NICE framework for each position. Furthermore, the Federal Cybersecurity Workforce Assessment Act of 2015 required OPM, in consultation with DHS, to identify critical needs for the IT, cybersecurity, or cyber-related workforce across federal agencies and to report to Congress on the identification of IT, cybersecurity, or cyber-related work roles of critical need. DHS officials also explained that, consistent with the 2015 act, it is currently working with other agencies and with industry to catalogue the federal cybersecurity workforce. The CISA Chief Counsel also identified DHS authorities that, under subchapter II of chapter 35 of Title 44 of the United States Code, CISA could leverage when implementing reform activities having government- wide or interagency impacts. Under these authorities, CISA (in consultation with OMB) administers the implementation of agency information security policies and practices, assists OMB with carrying out its responsibilities for overseeing agency information security policies and practices, and coordinates government-wide efforts on information security policies and practices. The Chief Counsel added that CISA “continues to consider, new, more specific…statutory authority aligned to specific reform responsibilities.” The administration is working toward establishing the GEAR Center, which it described in the reform plan as a vehicle for applied research that would help improve government operations and decision-making. OMB staff stated that the GEAR Center would be administered as a public- private partnership, and that the administration spent about $3 million for it in fiscal years 2018 through 2020 from available appropriations (see table 3). On Performance.gov, OMB provided options for the GEAR Center’s structure; it could be housed in a physical location, composed of a network of researchers working in multiple locations, or follow a different model. The administration does not envision that the GEAR Center will require government funds to conduct all of its initiatives in the long term. Instead, OMB staff said that the private sector would help fund its work after an initial stand-up period. According to the reform plan, GEAR Center research could help inform, for example, how the government responds to technological advances, how to provide better customer service experiences, and how to better leverage government data. In March 2019, OMB staff told us that they planned to establish the GEAR Center in fiscal year 2019, but as of February 2020, the center had not been formally established. To date, OMB staff have conducted preparation activities for establishing the GEAR Center, such as gathering stakeholder input through a Request for Information and a GSA- administered GEAR Center challenge competition to learn more about the types of projects a GEAR Center could facilitate. Through the challenge, GSA requested ideas on possible research projects, as well as related materials such as a project plan and ways to measure success. The challenge competition judges, which included OMB staff, selected three winning project plans with a prize of $300,000 each (for a total of $900,000). GSA specified that the cash prizes were for high potential project plans, and not grants to execute work on behalf of the government. In September 2019, GSA announced and awarded the winners of the GEAR Center challenge. The grand prize winners submitted 1-year project plans to: (1) help solve the federal cybersecurity workforce shortage by involving neurodiverse individuals, such as those with autism; (2) integrate currently disparate datasets to measure the impact of a federally funded program; and (3) train federal employees on how to better use their data for decision-making and accountability. In addition to the challenge competition, OMB contracted with the Center for Enterprise Modernization, a Federally Funded Research and Development Center operated by the MITRE Corporation, to examine options for operating the GEAR Center, in two projects. The first project— conducted from July 2019 through September 2019—was to explore a number of options for operating the GEAR Center. Following the first project, OMB staff laid out three tasks to accomplish during calendar year 2020 that they said would help them establish the GEAR Center: (1) establish a central coordinating function for the GEAR Center, (2) build the GEAR Center’s network of research partners, and (3) develop a draft government-wide learning agenda with input from federal agencies to inform the GEAR Center’s research and piloting activities. For the second project—which began in September 2019 and is scheduled to be completed in July 2020—the contractor is to provide additional detail on options for operating the GEAR Center, including on creating a network of research partners to support the GEAR Center. Table 3 provides details on these expenditures. As shown in figure 5, OMB has generally addressed most of our relevant key reform practices, and partially addressed the others. OMB staff generally addressed key practices related to determining the appropriate role of the federal government for the GEAR Center. While OMB staff have not developed a detailed governance structure for the GEAR Center, they have determined, with input from the private sector, that the GEAR Center will be a public-private partnership. Specifically, OMB staff considered the private sector’s ability or likelihood to invest its own resources in the initiatives the GEAR Center undertakes and otherwise contribute to the GEAR Center’s work. OMB did this by formally seeking the private sector’s input on these topics first through a Request for Information, and subsequently through a challenge competition. OMB has partially addressed key practices related to establishing goals and outcomes. Specifically, OMB has initiated a process for developing outcome-oriented goals and performance measures for the GEAR Center, but has not finalized them. The GEAR Center challenge competition asked respondents to provide short- and long-term outcome- focused measures of success for the proposed projects in their submissions. However, as of November 2019, OMB staff told us they have not finalized these goals and measures for the GEAR Center. They stated that this is because they have not yet analyzed the results of the progress made by the challenge competition’s grand prize winners, and because they believe the purpose, or broad goal, of the GEAR Center is sufficient for their purposes at this stage of implementation. OMB staff told us that while they acknowledge that grand prize winners are not required to complete the projects they proposed, they anticipate the winners will carry them out to some extent, and they plan to monitor their work to inform GEAR Center planning activities. As OMB moves forward with establishing the GEAR Center, OMB staff should complete their efforts to develop goals and measures, because they will be necessary to track and communicate the GEAR Center’s progress over time. In addition, OMB staff have not yet fully assessed the costs and benefits of the various options OMB is considering for operating the GEAR Center. As previously discussed, OMB has stated that the GEAR Center could be housed in a physical location, composed of a network of researchers working in multiple locations, or follow a different model. Also, as previously stated, MITRE is currently exploring details of options for operating the center, and plans to provide them to OMB in July 2020. However, OMB has not yet conducted an analysis of the costs and benefits of the options for operating the center. In July 2018, OMB’s then Deputy Director for Management said that defining costs and benefits is dependent on refining and finalizing implementation plans. As of November 2019, OMB had not developed an implementation plan for establishing the GEAR Center. As OMB moves forward with establishing the center, assessing the costs and benefits of the various options for operating it will enable OMB to communicate the value of each option to Congress and other stakeholders. This assessment can help build a business case for OMB’s ultimate choice of how to operate the GEAR Center that presents facts and supporting details among competing alternatives. OMB has generally addressed key practices related to involving employees and relevant stakeholders. Specifically, OMB has coordinated with internal government stakeholders, sought input from the private sector, and publicly communicated GEAR Center progress. For example, OMB staff said that they have worked to develop the GEAR Center with the President’s Management Council, the National Science Foundation, and DOD’s National Security Technology Accelerator group. Also, OMB held a Virtual Stakeholder Forum to provide information about the GEAR Center and to gather stakeholder input. During the forum, OMB sought attendees’ input through live polls, and announced that attendees could ask questions and provide additional input by sending messages to an OMB email account. OMB also sought stakeholder input through the GEAR Center Request for Information and challenge competition. Finally, as shown in figure 6, OMB has publicly reported on the GEAR Center’s progress on Performance.gov. OMB has generally addressed key practices related to leadership focus and attention. To accomplish this, OMB has designated leaders, including OMB’s former Deputy Director for Management, a member of OMB’s Performance Team, and other staff to be responsible for implementing the reform. OMB has partially addressed key practices related to managing and monitoring the reform to establish the GEAR Center. Specifically, OMB has gathered input from stakeholders on what research it could pursue, and from both stakeholders and a contractor on how the GEAR Center could be operated. OMB has done some analysis of that input, but has neither determined how the GEAR Center will operate nor developed an implementation plan. For example, OMB’s analysis of Request for Information responses shows that OMB is considering several options for how to execute the GEAR Center’s public-private partnership—a network of researchers, a physical location, etc.—but has not decided on one. In addition, as discussed previously, OMB contracted with MITRE to further assist with determining how the GEAR Center will operate. As OMB moves forward with establishing the GEAR Center, it will be able to track the GEAR Center’s progress, and communicate these results to Congress and key stakeholders, by developing and communicating an implementation plan with key milestones and deliverables. In response to our request to identify the legal authority OMB will need to implement this reform, OMB’s General Counsel stated in a November 2019 letter to us that it and its agency partners have relied upon existing legal authorities and available appropriations to develop the Request for Information, obtain external submissions for ideas to develop the GEAR Center, and issue the prize challenge. OMB stated that in conducting any future implementation activities, it would seek new legislative authority, if necessary. While planning and implementation progress has been made since the administration’s government-wide reform plan was released in June 2018, important details surrounding the implementation of certain reform proposals have not been developed or communicated. OMB has a central role in overseeing and prioritizing these reforms for implementation, with support from lead agencies. In our previous work on government reorganization and reforms, we have found that there are key practices that, if followed, can help manage the risk of reduced productivity and effectiveness that often occurs as a result of major change initiatives. Important practices such as engaging and communicating with Congress, employees, and key stakeholders; dedicating a senior leadership team; and developing implementation plans, can help to ensure the successful implementation of reorganizations and reforms. OMB and DHS partially addressed most of the leading practices through their efforts to implement several projects related to the cybersecurity workforce reform, including efforts to reskill employees to fill vacant cybersecurity positions, establish a cybersecurity reservist program to provide needed surge capacity, and streamline relevant hiring processes. However, OMB, in coordination with DHS, has not yet followed relevant key practices to implement its reforms government-wide. Specifically, OMB and DHS have not yet developed a communications strategy to involve Congress, employees, and other stakeholders; established a dedicated government-wide leadership team; or developed a government-wide implementation plan with outcome-oriented goals, timelines, key milestones, deliverables, and processes to monitor implementation progress. In addition, OMB and DHS have not demonstrated how the projects and activities outlined in the reform proposal would address our related high-risk issues and major management challenges, or developed workforce plans that assess the effects of the proposal on the current and future workforce. If OMB, in coordination with DHS, applied key reform practices government-wide, they would be better positioned to manage the reform, and track progress across all agencies facing cybersecurity workforce shortages. OMB has taken steps toward determining how the GEAR Center will operate, such as, by determining the appropriate role of the federal government; providing leadership focus and attention; and collecting input from the public, academia, and industry on how the center could operate and on ideas for possible research projects. However, OMB has neither assessed the costs and benefits of the options it is considering for operating the center, nor developed an implementation plan with outcome-oriented goals and performance measures for it. As OMB moves forward with establishing the GEAR Center, completing these two activities can help OMB (1) make a case for why OMB’s ultimate decisions on how to operate the center are the most optimal, and (2) provide greater transparency to the public and private partners involved in its development, help build momentum, and demonstrate the center’s value. We are making a total of seven recommendations to OMB. The Director of OMB, working with DHS, should develop a government- wide communications strategy to inform and, as appropriate, involve Congress, employees, and other stakeholders in implementation of the reform proposal to solve the cybersecurity workforce shortage. (Recommendation 1) The Director of OMB, working with DHS, should establish a dedicated government-wide leadership team with responsibility for implementing the reform proposal to solve the cybersecurity workforce shortage. (Recommendation 2) The Director of OMB, working with DHS, should develop a government- wide implementation plan with goals, timelines, key milestones, and deliverables to track and communicate implementation progress of the reform proposal to solve the cybersecurity workforce shortage. (Recommendation 3) The Director of OMB, working with DHS, should provide additional information to describe how the projects and activities associated with the reform proposal to solve the cybersecurity workforce shortage will address our high-risk issues related to ensuring the cybersecurity of the nation. (Recommendation 4) The Director of OMB, working with DHS, should develop a government- wide workforce plan that assesses the effects of the reform proposal to solve the cybersecurity workforce shortage on the current and future federal workforce. (Recommendation 5) The Director of OMB should assess the costs and benefits of options for operating the GEAR Center. (Recommendation 6) The Director of OMB should develop an implementation plan that includes outcome-oriented goals, timelines, key milestones, and deliverables to track and communicate implementation progress of the reform proposal to establish the GEAR Center. (Recommendation 7) We provided a draft of this report for review and comment to the Directors of OMB and OPM, the Secretary of DOD, the Acting Secretary of DHS, and the Administrator of GSA. OMB did not comment on the report. DHS and DOD provided technical clarifications, which we incorporated as appropriate. OPM and GSA responded that they did not have comments on the report. We are sending copies of this report to the Director of OMB and the heads of the agencies we reviewed as well as appropriate congressional committees and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or Mcneilt@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. In a 2018 report, we developed key questions based on our prior work on key practices that can help assess agency reform efforts. The 58 questions are organized into four broad categories and 12 subcategories, as shown in table 4. For the purpose of this review, we selected those subcategories and key questions that were most relevant to the selected reforms based on the information contained in the reform proposals, agency documentation, and interviews with the Office of Management and Budget and lead agencies for each of the reforms. The administration’s proposal to reorganize the Office of Personnel Management (OPM) evolved from June 2018 through November 2019, and was effectively halted by Congress in December 2019. In the June 2018 government-wide reform plan, the administration proposed: (1) moving OPM’s policy functions to a new office in the Executive Office of the President, which would also provide a government-wide view of human capital policy issues, (2) merging a number of OPM’s responsibilities with the General Service Administration’s (GSA) or other government entities’ to be determined at a later date, and (3) renaming GSA as the Government Services Agency. The goals of this proposal were to help elevate the importance of these functions, improve efficiency of operations, and save money, according to the reform plan. Specifically, the administration suggested integrating the following duties into the Government Services Agency or other government entities: administration of healthcare and insurance programs, Human Resources Solutions (HRS), which provides products and services to other federal agencies on a reimbursable basis, and information technology services. In addition, the reform plan contained another proposal to move all of OPM’s national security background investigation functions to the Department of Defense (DOD). The President’s Fiscal Year 2020 Budget, published in March 2019, expanded and modified the original OPM reorganization proposal. It proposed that all of OPM’s functions beyond those moving to the Executive Office of the President and DOD be transferred to GSA, rather than merging a portion of them into a newly formed Government Services Agency. It also called for creating a new GSA service area to house certain functions, and for moving OPM’s Office of the Inspector General to GSA. In May 2019, the administration submitted a legislative proposal to Congress requesting new authority to implement aspects of the OPM reorganization reform proposal. As of December 2019, this proposal had not been introduced in Congress. In May 2019, we testified on issues to consider in the proposed reorganization of OPM. We found that the Office of Management and Budget (OMB) and the two lead agencies (OPM and GSA) had generally not addressed key practices for reforms, such as establishing outcome- oriented goals, assessing costs and benefits, or developing an implementation plan, and had not fully involved or communicated their efforts with Congress, employees, and other key stakeholders. We also found that OMB, OPM, and GSA had not shown how they would address management challenges that may affect their ability to successfully reorganize the government’s central human capital functions. Between May and September 2019, OPM provided us with additional information, which contributed to our assessment of the extent to which OMB, OPM, and GSA addressed key practices for this reform (see figure 7). In October and November 2019, OMB staff and OPM and GSA officials provided us with updates on the status of the OPM reorganization reform proposal. OMB staff and OPM and GSA officials told us that the transfer of major functions from OPM to GSA, such as retirement services and HRS, was on hold until Congress, through legislation, provided the necessary authority to move these functions. They also told us that they were working together on moving the following functions from OPM to GSA through their existing authorities: (1) administrative responsibilities for the Chief Human Capital Officers (CHCO) Council; (2) the Program Management Office for the Security, Suitability, and Credentialing Performance Accountability Council (PAC); and (3) management of two OPM office buildings—the Theodore Roosevelt Building, which houses OPM’s headquarters in Washington, D.C., and the Federal Executive Institute located in Charlottesville, Virginia. OMB staff and OPM and GSA officials stated that the primary purpose of these moves was to achieve greater efficiency of operations, and that these transfers were not components of the OPM reorganization reform proposal. In November 2019, OPM’s Inspector General expressed concern over ongoing efforts to merge these functions with GSA, noting that the specific details of the full merger continued to evolve, and every iteration of the proposed reorganization would fundamentally alter how agency functions and duties are performed. In the National Defense Authorization Act (NDAA) for Fiscal Year 2020, signed into law in December 2019, Congress effectively halted actions to reorganize OPM pending the completion of reports by the National Academy of Public Administration (NAPA) and OPM. The law directed OPM to enter into a contract with NAPA to conduct a study to identify challenges associated with OPM’s execution of its functions and make recommendations for addressing them, including a cost-benefit analysis of proposed changes, and the identification of statutory or regulatory changes needed to execute recommended actions, among other things. Approximately 6 months after the NAPA report, OPM must submit a report providing its views on the NAPA report and its recommendations for changes to its functions. OPM is also to include a business case analysis associated with such changes and a proposal for legislative and regulatory action required to effect the changes. Many of these requirements reflect the issues we raised in our May 2019 testimony on the extent to which the proposal to reorganize OPM was consistent with our key reform practices. According to the President’s fiscal year 2021 budget request, the administration continues to pursue implementation of OPM’s reorganization. Specifically, it proposes to transfer the functions of OPM to GSA, contingent upon enactment of authorizing legislation. OMB, OPM, and GSA partially addressed the key practices related to establishing goals and outcomes. First, OMB, OPM, and GSA considered how the upfront costs of the reform would be funded by, for example, requesting funds through the President’s Fiscal Year 2020 Budget. However, OMB, OPM, and GSA did not fully address other aspects of the key practices. Specifically, since our May 2019 testimony, OPM provided us information on additional draft goals and measures for some portions of the reform. For example, according to a document we received from OPM in August 2019, a team leading the reform effort was developing “critical to quality” metrics in areas such as cost reduction, employee engagement, and flexible operations. However, these metrics did not have targets and had not been finalized. In November 2019, OMB staff told us that metrics were not yet final because they were still working with Congress to develop a legislative proposal authorizing the reform, and implementation of the merger was not yet underway. The NDAA for Fiscal Year 2020 requires NAPA and OPM to make recommendations for changes to OPM’s structure, functions, responsibilities and authorities, which may differ from those the administration proposed. We have also previously reported that major change initiatives should be based on either a clearly presented business case or analysis of costs and benefits grounded in accurate and reliable data, both of which can show stakeholders why a particular initiative is being considered and the range of alternatives considered. While OPM officials had some information on the costs and benefits they planned to achieve by merging functions with other agencies, they did not have an analysis or underlying data supporting their conclusions. Specifically, OPM provided us with its rationale for the reform in several documents, including: a summary of the agency’s financial and management challenges, a qualitative business case, a list of state and foreign governments’ administrative models where human resources and administrative functions are merged, and a presentation providing OPM’s estimate of the annual savings that could be realized by “fully integrating OPM’s operations into GSA.” However, the information that OPM provided did not include measurable performance or outcome metrics, or quantify benefits relative to costs, to provide a complete assessment of the costs and benefits and any alternative solutions to the reform proposal. OPM’s Office of Inspector General also found, in its fiscal year 2020 top management challenges report, that OPM had not developed a thorough analysis of costs and benefits. Shortly after OMB published the reform plan, OMB’s then Deputy Director for Management, who also served as OPM’s Acting Director, said that defining costs and benefits was dependent on refining and finalizing implementation plans. Since then, in the NDAA for Fiscal Year 2020, Congress required that NAPA’s study include an analysis of the benefits, costs, and feasibility of each recommendation, and a timetable for implementing these options. In addition, the law requires that OPM’s report include a business case analysis that describes the operational efficiencies and cost savings (both short- and long-term) associated with its recommendations. OMB, OPM, and GSA partially addressed the key practices related to involving employees and key stakeholders. Specifically, since our May 2019 testimony, OPM officials provided us with documents to demonstrate that the agency took additional actions in this area, as discussed in more detail below. However, we found that OPM’s early outreach efforts to employees and stakeholders were insufficient, the agency did not have a plan for incorporating employee and stakeholder feedback, and it did not share relevant implementation details that may have affected employees and stakeholders. For example, OPM provided us with a communications tracker that listed meetings and correspondence with Congress, staff, and employee groups from OPM’s Acting Director, Deputy Director, and Deputy Chief of Staff. While this document listed a number of meetings and calls, it showed that most of OPM’s efforts to involve Congress, employees, and employee groups began in April 2019, more than 9 months after OMB published the reform plan, and more than 8 months after OPM’s Director and GSA’s Administrator testified before Congress about their plans for carrying out the reform proposal. In addition, both members of Congress and employee groups expressed dissatisfaction with initial outreach from OMB, OPM, and GSA, including lack of transparency. For example, during a House Committee on Oversight and Reform Subcommittee on Government Operations hearing on May 21, 2019, members of Congress and employee groups testified that they felt insufficiently involved in the reform. Both groups stated that OPM officials communicated with them on few occasions, and members of Congress said that they had not received key documents they requested from OPM, including an implementation plan. In August 2019, OPM provided us with a strategic communications plan that included high level messages and strategies for reaching out to Congress, employees, and the public. This and other OPM documents demonstrated that OPM communicated with employees and key stakeholders, and provided opportunities for its employees to ask questions and provide comments about the reform, activities consistent with our key practices. However, the documents did not indicate how senior OPM officials planned to use the feedback they received from their employees. Similarly, neither OMB nor GSA described how they planned to use employee feedback to inform their reform efforts. The NAPA study required by the NDAA for Fiscal Year 2020 must include methods for involving, engaging with, and receiving input from other federal agencies, departments, and entities potentially affected by any change in OPM that NAPA recommends. The study must also incorporate the views of stakeholders. OMB, OPM, and GSA partially addressed the key practices related to addressing high-risk areas and longstanding management challenges, consistent with our assessment in May 2019. Since then, OPM provided additional documents related to (1) our relevant high-risk area of strategic human capital management, as well as (2) longstanding challenges at OPM we and OPM’s Inspector General have reported. However, OMB, OPM, and GSA did not explain how the OPM reorganization reform proposal would address our high-risk issue or mitigate major management challenges, and did not have plans to monitor the potential effects of the reform on these issues. As a result, OMB, OPM, and GSA did not fully consider the potential risks of transferring OPM systems with longstanding weaknesses to GSA, and of GSA taking on duties in areas such as information technology, where it faces major management challenges. They also lacked a means of monitoring the reform’s potential effects on our strategic human capital management high-risk area and on major management challenges. Moreover, in November 2019, OPM’s Office of the Inspector General continued to identify the proposed merger of OPM with GSA as a top management challenge because the proposal did not include an implementation plan, and created a burden for the agency to fully study, plan, and execute reorganization activities. In November 2019, OMB staff told us that, because the proposed merger was a long-term effort and plans were still under development, they had not yet determined how our high-risk and other management challenges would be addressed. The NDAA for Fiscal Year 2020 requires the NAPA study to include analyses of OPM’s challenges and a recommended course of action for resolving them. OMB, OPM, and GSA generally addressed the key practices related to leadership focus and attention. Specifically, since our May 2019 testimony, OPM officials provided us with documents demonstrating that OMB, OPM, and GSA made progress in this area. For example, OPM documents showed that OPM, OMB, and GSA leaders approved a governance structure for leading reform efforts that included: an executive steering committee that provided guidance and made decisions. Its members included the OMB Deputy Director for Management (serves as executive sponsor and chair), the OPM Director (serves as a vice-chair), and the GSA Administrator (serves as a vice-chair). The group used the Lean Six Sigma management approach to make decisions related to planning and implementing the reform during Tollgate meetings. an interagency task force that led activities to implement the reform, and that raised issues to the Executive Steering Committee as needed. Its members included leaders from OMB, OPM, and GSA. interagency teams, which provided subject matter expertise and performed tasks and activities to implement the reform. Their members were OPM and GSA officials. From August 2018 through April 2019, our analysis of OPM documents showed that these groups met and communicated frequently—from every few days to every few weeks, depending on the group. From May 2019— when the administration transmitted their legislative proposal to Congress to reorganize OPM—to November 2019, these groups met less frequently, according to OMB staff and GSA officials. OMB, OPM, and GSA partially addressed the key practices related to managing and monitoring. Since our May 2019 testimony, OPM provided us with documents that demonstrated improvements in this area, but as of November 2019, had yet to finalize an implementation plan. Specifically, the documents showed that OMB, OPM, and GSA held leadership meetings and systematically tracked various aspects of the reform. For example, OPM officials tracked the status of certain activities associated with the reform, such as progress on developing a plan for communicating with employees and stakeholders, through leadership meetings. Also, OPM had a document identifying risks associated with the reform, such as ensuring continuity of services, as well as mitigation strategies, such as including provisions in OPM-GSA interagency agreements. The document also specified individual agency officials responsible for each risk. OMB, OPM, and GSA did not develop an implementation plan for the OPM reorganization reform that included key milestones and deliverables. In November 2019, OMB staff told us that their plans were still being developed because they were waiting for Congress to pass the administration’s legislative proposal authorizing the reform. The NDAA for Fiscal Year 2020 requires NAPA and OPM to make recommendations for changes to OPM’s structure, functions, responsibilities, and authorities, which may differ from those the administration proposed. OMB, OPM, and GSA partially addressed the key practices related to employee engagement. Specifically, while OMB and agencies undertook activities to measure employee engagement, such as surveying and communicating with employees, they did not develop a comprehensive strategy for sustaining and strengthening employee engagement during and after the reform. For example, GSA officials told us that they established a GSA-OPM change management and communications workgroup, which developed a change management and communications plan that included employee engagement activities. Also, in April 2019, OPM conducted an internal survey of agency staff to measure employee engagement, among other factors. OPM officials also identified employee morale issues as a risk in a document identifying risks associated with the reform and risk mitigation strategies. To address employee dissatisfaction and low morale, OPM officials, including OPM’s then-Acting Director, shared the survey results with employees, held listening sessions to determine employees’ preferences for communications about the OPM reorganization reform proposal, and developed a communications strategy. However, OPM officials did not determine how they planned to use these communications to sustain and strengthen employee engagement. In November 2019, OMB staff told us that because they were still in the planning stages of the reorganization, the proposed reform had not yet involved major changes for employees, so they put employee engagement efforts on hold. The NAPA study required by the NDAA for Fiscal Year 2020 is to include methods for involving, engaging with, and receiving input from other federal agencies, departments, and entities potentially affected by any change in OPM that NAPA recommends. The study is to also incorporate the views of stakeholders. OMB, OPM, and GSA did not address the key practices related to strategic workforce planning. OPM and GSA officials told us that they were conducting workforce planning activities associated with the OPM reorganization reform. Also, the President’s Fiscal Year 2020 Budget provided some information about staff levels at OPM and GSA. However, OMB, OPM, and GSA did not produce strategic workforce plans for OPM and GSA employees. OPM and GSA officials stated that they had not provided us with these plans because they were under development. In November 2019, GSA officials added that they were waiting for congressional authorization to carry out the reform proposal, so they had put their efforts to develop a workforce plan on hold. The NDAA for Fiscal Year 2020 requires that NAPA and OPM make recommendations on changes to OPM, which may differ from the administration’s proposed reorganization of OPM. As part of our review, our Office of General Counsel sent letters to the Offices of General Counsel at OPM, GSA, and OMB requesting they provide us with a description of the legal authorities they were using to support the proposed OPM reorganization. OMB, OPM, and GSA provided responses to our letter, but did not identify which aspects of the OPM reorganization could be carried out under existing law and which would require legislative authority. GSA, OPM, and OMB officials stated that they had not yet finalized their legal analysis, and that they were still determining which legal authorities they could use to implement elements of the reform. OMB General Counsel stated that to implement the administration’s proposed reorganization, both legislative and administrative actions would be necessary and dependent on each other “in the long run.” In May 2019, the administration submitted a legislative proposal requesting authority to transfer OPM functions—such as Human Resources Solutions, Information Technology, Retirement, and Health and Insurance Services—to GSA. As of December 2019, the proposal had not been introduced in Congress. OMB staff told us that the legislative proposal was an effort to communicate transparently about the extent to which new authorities would be required. As discussed earlier, in December 2019, Congress passed the NDAA for Fiscal Year 2020. In the NAPA study required under the NDAA, NAPA is to provide a comprehensive assessment and analysis of the statutory or regulatory changes needed to implement any recommended course of action, and to submit this report to Congress and the Director of OPM. The Director of OPM is then to submit a report to Congress that lays out OPM’s views on the findings and recommendations of the NAPA study, along with OPM’s recommendations for change. Any recommendation submitted by OPM for change is to include a business case analysis that sets forward the efficiencies and cost savings (both short- and long-term) associated with the change, and a proposal for legislative or administrative action required to effect the change. The statutory provisions in the act generally provide that no aspect of the agency that is assigned in law to OPM may be moved to GSA, OMB, or the Executive Office of the President until 180 days after OPM’s report is submitted to congressional committees, and subject to the enactment of any required legislation. This reform proposal aims to modernize and streamline the way citizens interact with the federal government, and to raise customer experience to a level comparable with leading private sector organizations. With support from the United States Digital Service (USDS) and GSA’s Technology Transformation Service, OMB has stated that it will lead an effort to establish a government-wide capability that will enable agencies to identify their customers, map their interactions (or journeys) with federal programs or services, and leverage digital tools and services to improve their experiences and overall satisfaction. For example, as reported in the reform plan, the U.S. Department of Agriculture created a “digital front door,” accessible at Farmers.gov, that is organized around the user experience rather than the government’s structure. The reform plan further explains that the improved capability provided by USDS and GSA would also provide for a government-wide resource to manage organizational change, including improved project planning, facilitating interagency collaboration, and sharing best practices on change management. OMB staff told us in early 2019 that they have delayed implementation of this reform, and instead will focus on other customer experience activities, such as those outlined in the related Cross-Agency Priority (CAP) goal. Upon release of the President’s Fiscal Year 2020 budget in March 2019, we confirmed that this reform was not included in the administration’s reorganization priorities, and OMB confirmed that no funding was requested for its implementation. OMB and agencies are also pursuing a related but distinct CAP goal under the President’s Management Agenda–Improving Customer Experience with Federal Services–with the aim of providing a modern, streamlined, and responsive customer experience across government, comparable to leading private-sector organizations. According to OMB, the reform proposal is meant to stand up a central capacity, or office, within GSA to manage customer experience government-wide; whereas, the CAP goal is intended to support capacity growth and accountability within agencies to develop and manage their own customers’ experience and satisfaction. Because OMB has not yet begun to implement this reform, and no actions are planned for fiscal year 2020, we are not able to assess the extent to which the reform is adhering to key reform practices. When the administration moves forward with implementing this reform, it will be better positioned for its successful implementation if the key reform practices are followed. In response to our request to identify the legal authority OMB will need to implement this reform, OMB’s General Counsel responded in a November 2019 letter that the initiative will not require new legislation. OMB stated the reform can be implemented within current law and available appropriations. Triana McNeil at (202) 512-6806 or Mcneilt@gao.gov. In addition to the contact named above, Sarah E. Veale (Assistant Director, Strategic Issues), Peter Beck (Analyst-in-Charge, Strategic Issues), Colenn Berracasa, Karin Fangman, Steven Putansu, Janet Temko-Blinder, Peter Verchinski, and Kellen Wartnow made key contributions to this report. Timothy Carr, Jacqueline Chapin, Tom Costa, Sara Cradic, Brenda Farrell, Patrick Hickey, Shirley Jones, Tammi Kalugdan, Brian Mazanec, Kimberly Seay, Gregory Wilshusen, and Alicia White also contributed to this report.", "summary": "In June 2018, the administration released its government-wide reform plan, which included 32 proposals aimed at achieving management improvements and organizational efficiencies, among other things. OMB has a central role in overseeing these reform proposals, with support from various lead agencies. In July 2018, GAO reported on key questions to consider when developing and implementing reforms. GAO was asked to examine reform implementation. This report discusses three selected reforms that the administration prioritized: (1) moving background investigations from OPM to DOD, (2) solving the cybersecurity workforce shortage, and (3) establishing the GEAR Center. For each selected reform, GAO determined the extent to which OMB and the lead agencies addressed key practices for effectively implementing reforms, among other issues. GAO reviewed relevant documentation and interviewed OMB staff and agency officials. GAO assessed OMB's and lead agencies' efforts against relevant key practices for effective reforms. In working to implement three selected government-wide reforms that GAO reviewed, the Office of Management and Budget (OMB) and lead agencies followed some, but not all, of the key practices associated with effective reforms. Following key practices, such as those reflected in the questions below, would better position OMB and lead agencies to effectively implement such major change initiatives and achieve their intended objectives. Moving background investigations from the Office of Personnel Management (OPM) to the Department of Defense (DOD) : As required, the transfer of background investigations took place by September 30, 2019. OMB, OPM, and DOD generally addressed most key reform practices in this transfer, including involving employees and stakeholders, establishing an implementation team, and developing implementation plans. With the transfer complete, DOD officials told GAO they are shifting focus toward addressing GAO's high-risk area on the government-wide personnel security clearance process. Solving the cybersecurity workforce shortage : OMB and the Department of Homeland Security (DHS) partially addressed most leading practices through their efforts to implement several projects, such as reskilling employees to fill vacant cybersecurity positions, and streamlining hiring processes. However, GAO found that OMB and DHS have not established a dedicated implementation team, or a government-wide implementation plan, among other practices. Without these practices in place, OMB and DHS may not be able to monitor implementation activities and determine whether progress is being made toward solving the cybersecurity workforce shortage. Establishing the Government Effectiveness Advanced Research (GEAR) Center : According to OMB, the GEAR Center will bring together researchers from private and public sectors to inform and develop ways to improve government services and operations. OMB is working toward establishing the GEAR Center by collecting input from the public, academia, and industry for how the Center could be structured and ideas for possible research projects. However, OMB has not yet developed an implementation plan with key milestones and deliverables to track its progress. Developing and communicating an implementation plan will help OMB track the GEAR Center's progress and communicate its results. GAO is making 7 recommendations to OMB to follow certain key practices to help solve the cybersecurity workforce shortage and to establish the GEAR Center. OMB did not comment on the report.", "document_type": "gao"}
{"report": "In the Santa Cruz River Basin and Tijuana River Valley watersheds, which straddle portions of the 1,954-mile U.S.-Mexico border, water flows north from higher elevations in Mexico into the United States. Both countries have infrastructure along the border to manage, divert, and treat wastewater, including sewers, pipelines, and treatment plants, in addition to the two international wastewater treatment plants in the United States. The Nogales and South Bay plants are located in the middle and lower end of the Santa Cruz River Basin and Tijuana River Valley, respectively. Figure 1 shows the location of the international wastewater treatment plants along the border. In 2018, USIBWC treated more than a combined 14 billion gallons of sewage at the Nogales and South Bay international wastewater treatment plants. At the Nogales plant, USIBWC treated 4.5 billion gallons of sewage—an average of 12.45 million gallons per day from the city of Nogales in Sonora, Mexico. In addition, the plant treats an average of 2 million to 2.5 million gallons per day of sewage from the Arizona cities of Nogales and Rio Rico. The Nogales plant discharges treated wastewater into the Santa Cruz River. At the South Bay plant, USIBWC treated 9 billion gallons of sewage in 2018—an average of 24.8 million gallons per day from the City of Tijuana in Baja California, Mexico. The South Bay plant discharges treated wastewater though a pipeline, called the South Bay Ocean Outfall, into the Pacific Ocean. Both watersheds are located in arid regions characterized by infrequent but sometimes intense precipitation that forms short-lived streams or washes that fill with water during such events but may be dry at other times. These high-precipitation events lead to high levels of stormwater runoff. Urban stormwater is a major contributor to pollution in the nation’s waterbodies, including rivers and oceans, and can contribute to disease outbreaks and beach closings, as well as flooding. IBWC’s mission is to provide binational solutions to issues that arise during the application of U.S.-Mexico treaties regarding, among other things, water quality and flood control in the border region including constructing and operating wastewater treatment plants, as directed by Congress. The U.S. and Mexican governments established IBWC (then the International Boundary Commission) in 1889, initially to resolve boundary-related differences arising along the border. Various agreements between the United States and Mexico added water distribution and flood management in the transboundary rivers to IBWC’s responsibilities, including management of the border reaches of the Rio Grande and Colorado rivers. In the 1944 treaty, Utilization of Waters of the Colorado and Tijuana Rivers and of the Rio Grande, the United States and Mexico agreed to apportion their shared waters, distributing the waters of the Colorado River and the Rio Grande between both countries. As part of the 1944 treaty, the United States agreed to annually provide a guaranteed amount of water from the Colorado River to Mexico—unless deliveries were limited by extraordinary drought—and to allocate the waters of the Rio Grande between the two countries, as well as authorizing jointly built and operated dams, reservoirs, and hydroelectric plants to manage water from the Rio Grande River. USIBWC manages this infrastructure and ensures annual compliance with the 1944 treaty water delivery requirements. As part of its flood control efforts, IBWC maintains and manages over 500 miles of levees for flood protection. The 1944 treaty established the key organizational components of IBWC and its two sections—USIBWC and the Mexican Section—which are federal agencies of their respective governments. Under the treaty, USIBWC and the Mexican Section are each headed by a commissioner who is an engineer. The treaty allows each commissioner to employ engineers, legal advisers, and assistants as needed and established certain positions—two principal engineers, legal counsel, and a secretary (that is, a foreign officer)—as entitled to diplomatic status in the other country’s territory. USIBWC is headquartered in El Paso, Texas, and the Mexican Section is headquartered in the adjoining city of Ciudad Juarez, Chihuahua, Mexico. USIBWC and the Mexican Section also have their own field offices along the border that operate and oversee joint work. USIBWC operates under the foreign policy guidance of the Department of State and implements treaties between the United States and Mexico related to boundary preservation and water management, including border sanitation and flood control in the border region. USIBWC is headed by the U.S. Commissioner and is made up of six executive offices and three departments, with about 240 full-time equivalent employees as of fiscal year 2017, the most recent data available at the time of our review. The six offices include the Foreign Affairs Office and a Legal Affairs Office; the former houses the foreign officer responsible for diplomatic communications and provides advice for the interpretation of treaties and minutes, and the latter houses legal counsel. The three departments in USIBWC are the Administrative Department, which supports all agency functions through acquisitions, budget, finance, accounting, and information management; Engineering Department, which is headed by a Principal Engineer of Engineering who provides technical and policy advice to the U.S. Commissioner and technical support in planning, engineering, environmental management, and construction management; and Operations Department, which is headed by the Principal Engineer of Operations who through the agency’s field offices oversees the maintenance and operations of the two international wastewater treatment plants as well as more than 100 hydrologic gaging stations, 500 miles of levees, four diversion dams, two international storage dams and associated hydroelectric power plants, more than 600 hydraulic structures, and one-half of all international boundary monuments and markers on the U.S.-Mexico land border and at international ports of entry. USIBWC’s annual budget, which has averaged $75 million per year since fiscal year 2010, is submitted to Congress as part of the Department of State’s overall budget. Under State’s budget process, USIBWC submits a budget request 2 years in advance of the funding to be spent. Once the department’s leadership approves USIBWC’s budget, it is incorporated into the overall departmental budget request for review by OMB. After OMB’s review, the budget is included as part of the President’s annual budget request to Congress. The agency receives its appropriated funding in two budget line items: (1) Salaries and Expenses and (2) Construction. As shown in figure 2, USIBWC funding has declined, when considering inflation (fiscal year 2018 dollars). According to USIBWC officials, the agency’s funding has increased about 1.1 percent per year since fiscal year 2010 and has been relatively flat since fiscal year 2017. According to a USIBWC budget official, the agency’s costs are increasing at an average inflation rate of nearly 3 percent per year. USIBWC’s budget from fiscal years 2010 through 2019, however, was more than double its budget from fiscal years 2003 through 2007. According to the official, starting in fiscal year 2010, the agency received an increase in its construction appropriations to fund dam and levee improvements along the border. Before that, in fiscal year 2008, USIBWC received additional appropriations of $55.6 million to pay for levee repairs; and in fiscal year 2009, under the American Recovery and Restoration Act, received $220 million for construction projects. In addition to USIBWC, other federal agencies that manage or collaborate on water infrastructure projects in communities along the U.S.-Mexico border include the following: U.S. Army Corps of Engineers (Corps). The Corps provides assistance for flood control, wastewater treatment, drinking water, and water supply projects in communities across the United States, as directed by Congress. To provide flood control assistance, the Corps’ Emergency Streambank and Shoreline Protection program plans, designs, and constructs erosion control projects that protect public infrastructure. It conducts these directly or under contract with other federal agencies, such as USIBWC. Congress has also authorized the Corps to provide assistance to nonfederal interests for carrying out water-related environmental infrastructure and resource protection and development projects, including waste water treatment and related facilities. In addition, the Corps’ Planning Assistance to States program helps states, local governments, and tribes with preparing comprehensive plans for the development and conservation of water and related land resources. The Corps has worked on various projects along the U.S.-Mexico border. For example, to address stormwater that flows downhill from Nogales, Sonora into Nogales, Arizona, near USIBWC’s Nogales plant, USIBWC requested an evaluation by the Corps on possible flood protection improvements in Mexico, which was completed in 2004. Based on the Corps’ recommendations, the local and federal governments in Mexico constructed several dams and detention basins. To address flooding of the Tijuana River in southern California, USIBWC contracted with the Corps to implement the U.S. portion of the Tijuana Flood Control Project in 1978. For this project, the Corps prepared construction plans and supervised the construction of a quarter-mile concrete channel in the United States that extends downstream from the U.S.-Mexico border. EPA. In 1983, the United States and Mexico signed the Agreement on Cooperation for the Protection and Improvement of the Environment in the Border Area (the La Paz Agreement). In the agreement, the United States and Mexico agreed to coordinate their efforts to address problems of air, land, and water pollution in the border area, defined as the area situated within 100 kilometers (approximately 62 miles) of either side of the border. The agreement names EPA as the national coordinator responsible for its implementation and provides EPA with a formal means of working with its federal counterpart in Mexico on binational programs. In addition, EPA and its Mexican counterpart created a binational program to fund environmental improvement projects for communities along the border, called the U.S.-Mexico Border Water Infrastructure Program. The most recent plan developed under the agreement—U.S.- Mexico Border 2020—is an 8-year cooperative program initiated in 2013 that identified five goals to protect the environment and public health in both countries. The second goal—to improve access to clean and safe water—includes protecting and restoring binational watersheds by addressing the inadequate collection and treatment of wastewater. Under the program, EPA works with federal agencies—including USIBWC—and state and local agencies to build grant-funded projects to improve water quality in the border area, including wastewater infrastructure projects that connect to or are related to USIBWC’s two international wastewater treatment plants in Arizona and California. North American Development Bank (NADB). In 1993, another agreement between the United States and Mexico led to the creation of two entities—NADB and the Border Environmental Cooperation Commission—to develop the environmental infrastructure of the U.S.- Mexico border region. NADB’s supervisory board includes representatives from EPA and the Departments of State and Treasury. NADB also established the Border Environment Infrastructure Fund to administer grant funds provided by EPA, for the implementation of high- priority municipal water and wastewater infrastructure projects located within 62 miles north of the U.S.-Mexico border, as well as 187 miles south of the border. NADB funds wastewater and sewer projects in communities along the border, including projects at USIBWC’s two international wastewater treatment plants. IBWC’s two wastewater treatment plants are required to meet water quality standards under the Clean Water Act. The act establishes the basic structure for regulating surface water quality, including regulation of discharges of such pollutants as E. coli bacteria and heavy metals, such as arsenic and lead, into the waters of the United States. The act requires states to establish water quality standards that protect public health and the environment and consider aquatic wildlife and human consumption and recreation, among other uses. The act also requires EPA to maintain and improve water quality by assisting and overseeing states’ efforts, among other responsibilities. The states are required to monitor and assess the conditions of water bodies, and those that do not meet state water quality standards are considered impaired. Other provisions of the Clean Water Act include the following: NPDES permits. The Clean Water Act prohibits the discharge of pollutants from point sources (sources of pollution, such as wastewater treatment plants and industrial facilities) into waters of the United States without a permit from EPA or an authorized state. Under the act, EPA and authorized states issue NPDES permits for point sources of pollution, which among other things regulate the amount of pollutants that can be discharged. Another component of the NPDES program is the pretreatment program, to prevent the introduction of pollutants into a publicly owned wastewater treatment plant that will interfere with its operation. According to EPA, by reducing or eliminating waste from industries, fewer toxic pollutants are discharged to and treated by the publicly owned wastewater treatment plant, providing benefits to both these plants and the industrial users. EPA has authorized most states, including Arizona and California, to administer clean water discharge permits. The Arizona Department of Environmental Quality administers the NPDES permit for the Nogales plant. The plant is also subject to state permits, such as an aquifer permit required in Arizona to limit the impact of the plant’s discharge on groundwater in the vicinity. The San Diego Regional Water Quality Control Board administers the NPDES permit for the South Bay plant. Stormwater runoff. Stormwater runoff is generated from rain and snowmelt events that flow over land or impervious surfaces—such as paved streets, parking lots, and building rooftops—and does not soak into the ground. The NPDES stormwater program regulates some stormwater discharges from three potential sources: certain municipal storm sewer systems, construction activities, and industrial activities. Operators of these sources might be required to obtain an NPDES permit before they can discharge stormwater. This permitting mechanism is designed to prevent stormwater runoff from washing harmful pollutants into local surface waters. Total Maximum Daily Load. Under the Clean Water Act, states must establish water quality standards; for waters that do not meet these standards, states must develop Total Maximum Daily Loads (TMDLs), which EPA approves. TMDLs set targeted limits for pollutants but are not self-implementing; EPA and states help reduce pollutants by issuing permits for point sources, whereas they provide voluntary incentives to reduce nonpoint source pollution (pollution that cannot be traced to a single source). Wastewater and stormwater utilities in the United States and Mexico are managed, for the most part, by local municipal governments. In the United States, local governments own and operate the majority of drinking water and wastewater utilities and charge users for their service through water rates. In Mexico, local and state governments, including Nogales and Tijuana, own and operate drinking water and wastewater utilities. Each city has its own sewer and wastewater infrastructure, including wastewater treatment plants. For example, the state of Tijuana Public Service Commission of Tijuana is responsible for the operation and maintenance of wastewater collection and treatment infrastructure, and of the drinking water distribution system. In the United States and Mexico, stormwater may be managed by a wastewater utility or a local municipality. Asset management is a widely recognized tool used across a variety of infrastructure sectors to manage physical assets, such as highways, machinery, and buildings. In the case of water infrastructure, those assets include pipelines, tanks, pumps, sewers, and other facilities. In a March 2004 report, we found that water utilities may benefit from implementing asset management practices to better identify and manage their infrastructure needs. To assist water utilities in adopting asset management, in 2003, EPA developed an asset management framework for water utilities. In 2008, EPA incorporated this framework into a best practices guide for water utilities based on similar frameworks used by water utilities in Australia and New Zealand. In a March 2004 report, we reported that federal law does not require water utilities to use asset management, but large water utilities may be more likely to use asset management than small water utilities. In a January 2016 report, we identified leading asset management practices for wastewater utilities that include identifying key assets—such as pipelines, treatment plants, and other facilities—and assessing their life-cycle costs. We have also previously identified key capital planning principles that apply to large capital acquisitions, such as infrastructure. For example, in a February 2007 report, we identified five key planning principles in OMB guidance on capital programming contained in OMB Circular A-11. These include developing links between strategic goals and infrastructure; developing a needs assessment and identifying gaps in infrastructure; evaluating alternatives; using a review and approval framework with criteria for selecting capital investments; and developing a long-term capital investment plan. Further, OMB’s capital planning guidance states that each capital asset should have an operations and maintenance plan that outlines the procedures and responsibilities for scheduled preventive and regular or routine corrective maintenance. In addition, in November 2019, OMB issued a memorandum to federal agencies that reinforced the need to implement the capital programming guidance in OMB Circular A-11 that agencies develop, document, and implement a capital planning process. We have also previously found that economic guidance generally states investment decisions such as those made for infrastructure should be informed by a consideration of both benefits and costs of relevant alternatives. For example, OMB has issued guidance on estimating costs and benefits to help federal agencies efficiently allocate resources through well-informed decision making about activities. This guidance includes OMB Circular A-94, which we have previously identified as providing leading practices for economic analysis. OMB Circular A-94 directs agencies to follow certain economic guidelines for estimating costs and conducting cost-effectiveness analyses of federal programs or policies to promote efficient resource allocation through well-informed decision making in certain circumstances. The guidance applies to federal agencies and programs, but we have previously found that it provides leading practices for economic analysis of investment decisions. Under OMB Circular A-94, a cost estimate is to include a comprehensive assessment of the costs. Under the 1944 Treaty, USIBWC and the Mexican Section have negotiated minutes laying out the countries’ roles and responsibilities in managing and operating the two wastewater treatment plants in the United States. Under this authority, both sections have also established cost-sharing agreements for the ongoing operation and maintenance of each plant. The 1944 treaty establishes the jurisdiction, structure, and functions of IBWC under the treaty, largely establishing IBWC’s present form and processes. Specifically, IBWC is authorized to jointly study, investigate, and develop solutions to transboundary problems related to water and the international boundary. Under the treaty, when a new or anticipated boundary or water problem is identified, USIBWC and the Mexican Section are to discuss solutions and make recommendations to their respective governments for its resolution before negotiating a formal solution through a minute. The early detection and evaluation of the problem, followed by the development of measures for resolution, are a part of IBWC’s mission, according to USIBWC’s website. The proposal for a new IBWC project may be initiated by one or both governments, or by state or local authorities in either country through their respective IBWC section. The project is then to be jointly investigated. If the findings of the IBWC joint investigations show that a cooperative project is feasible and is justified as a binational project, USIBWC and the Mexican Section may endorse the findings in a minute and recommend the project to the United States and Mexico governments. Under the 1944 Treaty, IBWC is also authorized to resolve disputes between the two countries arising from the interpretation or application of the treaty. In ratifying the treaty, the U.S. Senate resolution specified that USIBWC should only conduct work related to the eight projects identified in the treaty and not undertake any other construction projects without congressional authorization. As a result, USIBWC has received separate authorizations from Congress for projects implemented through treaty minutes, including the two international wastewater treatment plants. Specifically, USIBWC constructed the Nogales and South Bay plants under a series of statutory authorizations enacted in several Congresses from the 1930s to the 2000s. USIBWC officials said that IBWC can develop documents that are an alternative to a minute but serve the purpose of gaining consensus between the two sections. Alternatives include an exchange of letters, a signed term of reference, and a joint report drafted by principal engineers from USIBWC and the Mexican Section. A letter exchange would provide the approval of an activity from both the U.S. and Mexican Commissioners, such as flood operations criteria in any given year or emergency notification protocols for communities along the border. A term of reference would provide the scope of work for a project or protocol, describe the work that the two sections will do, and how they will do it. A joint report of the principal engineers is a technical document that can describe ongoing activities or that can commit IBWC to a new activity. These reports can be adopted as a minute, or, if the activity is already under way, not adopted. As a diplomatic agency under the Department of State, USIBWC can negotiate agreements with Mexico on its own, but State gets involved in certain situations, such as the negotiation and conclusion of an IBWC minute, or with respect to large and costly projects, according to State officials. For example, under the Department of State’s Circular 175 procedure, authorization to negotiate and conclude binding international agreements is obtained via approval of a memorandum by the Secretary of State or another Department of State senior official. The Department of State may also provide diplomatic support in a variety of ways. For example, State may draw attention to an issue by sending a diplomatic note to the Mexican Embassy to formally request the need for action to resolve a problem. Further, in coordination with USIBWC, the U.S. Embassy and Consulates may engage with Mexican government officials to advocate actions to address problems, such as water quality problems, including during meetings with Mexican federal and local officials, according to State officials. The IBWC commissioners and staff from both USIBWC and the Mexican Section work together in formal and informal ways, according to officials from both sections. The commissioners meet on a regular basis to discuss ongoing and, if appropriate, new, projects to carry out the treaty. Between meetings, the commissioners exchange information through formal channels with letters. In addition, according to USIBWC officials, the two sections’ staff are in frequent contact, through formal and informal communication. For example, USIBWC officials said staff from both sections will exchange daily emails and telephone calls to discuss information and collaboration on various IBWC projects. Under Article 3 of the treaty, the joint use of international waters “is subject to any sanitary measures or works which may be mutually agreed upon by the two Governments, which hereby agree to give preferential attention to the solution of all border sanitation problems.” Under this article and the articles authorizing joint investigations and solutions, IBWC has negotiated a series of minutes related to sanitation issues, one of which dealt with the issue broadly and others of which dealt with specific geographic locations. Each minute is pursuant to various statutory authorizations in the United States. In 1979, IBWC signed Minute 261, which provides that that the two countries should take timely measures to prevent any border sanitation problem. The minute also provides that for each border sanitation problem, IBWC would prepare a minute that would identify the problem, the course of action for resolution, and a specific time schedule for implementation. Other minutes were executed for sanitation issues in Nogales and Tijuana, pursuant to various statutory authorizations in the United States. For the Nogales plant, Minute 206, signed in 1958, approved a jointly operated and maintained wastewater plant in Arizona based on a Joint Report by the principal engineers. Minute 227, signed in 1967, provided for the relocation of the plant to its current location and expanded the treatment capacity of the plant. Minute 276, signed in 1988, approved a further increase in the capacity of the plant. For the South Bay plant, Minute 283, signed in 1990, approved the construction of the South Bay plant in San Ysidro, California. This minute described the water quality situation, discussed alternatives to fix the problem, and recommended a plan to fix it. The recommended plan included the building of the international wastewater treatment plant, as well as completion of Mexico’s sewage system for Tijuana, and other steps. The Nogales plant provides secondary treatment for wastewater generated in both Nogales, Arizona, and Nogales, Sonora, Mexico. USIBWC and the City of Nogales, Arizona, own the plant, which began operating in 1972. In 1945, IBWC recommended that a plant be built 1.5-miles north of the border with a treatment capacity of 1.6 million gallons per day. The plant was completed in 1951. An underground pipeline, referred to as the “trunkline,” was also constructed to transport the sewage under the border from Mexico 1.5-miles to the plant for treatment. As the population grew in both cities, the communities recognized the need for a larger plant. At the request of the City of Nogales, Arizona, the new plant—with a treatment capacity of 8.2 million gallons per day according to USIBWC documents—was constructed 9-miles north of the border, at the confluence of the Nogales Wash with the Santa Cruz River (see fig. 3). Construction on the new plant began in 1970 and was completed in 1972. In 1988, IBWC signed a minute upgrading the plant, adding additional treatment capacity for Mexican wastewater. Then to comply with more stringent federal and state regulations, the plant was upgraded in 1992 and 2009. At present, USIBWC manages the plant, which has treatment capacity for up to 17.2 million gallons of wastewater from Mexico and the United States per day according to USIBWC documents. Wastewater treatment plants collect sewage from residences and businesses and treat it to remove pollutants such as sediment, bacteria, and other materials. There are three types of treatment at wastewater treatment plants in the United States. States are required to meet standards for two of them. Primary treatment involves physical processes such as screening and sedimentation to remove a portion of pollutants that settle or float. Secondary treatment augments physical treatment with biological processes to remove organic matter. The treatment involves the use of bacteria to consume waste material. Secondary treatment, combined with disinfecting chemicals, such as chlorine, can reduce about 85 percent of pollutants. Tertiary treatment involves specialized or advanced treatment that is specific to the pollutant. For example, some treatment plants try to reduce nutrients such as nitrogen and phosphorus. Tertiary treatment can include additional filtration, reverse osmosis, or additional chemical or biological processes. The South Bay plant provides secondary treatment for wastewater generated in Tijuana, Mexico. USIBWC operates the plant. In the decades, before the plant was built in 1997, untreated sewage reached the Tijuana River, which flows north from Mexico to San Diego, California. The river transported raw sewage to the Pacific coast at Imperial Beach, California, creating a nuisance and public health risk in the United States. To address the problem, IBWC signed Minute 283 in 1990, which provided the framework for a project to treat wastewater from Tijuana, Mexico, at a plant located in the United States. Construction began in 1994. In 1997, the South Bay plant opened with discharge through an emergency connection to the City of San Diego’s wastewater treatment facility. The South Bay plant became fully operational in 1999, providing advanced primary treatment for 25 million gallons of sewage coming from Mexico daily and discharging treated wastewater 3-miles offshore in the Pacific Ocean through the South Bay Ocean Outfall, which is a 3.5-mile-long pipe, according to USIBWC documents. The plant was upgraded with secondary treatment facilities in 2010. It is designed to treat up to 25 million gallons per day of Tijuana’s sewage, with the ability to treat up to 50 million gallons per day for a short period of time, according to USIBWC officials. The City of Tijuana also operates five wastewater treatment plants in Mexico to treat its remaining sewage, though these plants are not always fully operational. The South Bay plant’s facilities include five canyon collectors located along the border in five of the six cross-border canyons. During normal operations, smaller amounts or “low-flows” of urban runoff and wastewater from Mexico are diverted by these canyon collectors and conveyed to the plant through underground pipelines (see fig. 5). IBWC minutes, with the approval of the U.S. and Mexican governments, establish each country’s roles and responsibilities, outline the costs of the Nogales and South Bay plants, and describe the cost-sharing arrangements between the United States and Mexico for operating and maintaining the plants. Minutes for each plant specify the cost-sharing arrangement for construction. See appendix II for details of the IBWC minutes that authorize the construction, management, and operation of the two plants. For the original Nogales plant, the U.S. government authorized the funding in the Department of State, Justice, Commerce, and the Judiciary Appropriation Act for 1947 and provided funding with certain conditions, including that the City of Nogales agreed to furnish the lands or easements free of cost and that the city operate and maintain the project once it was completed. Under Minute 227, signed in 1967, Mexico agreed to participate in funding the expansion of the capacity of facilities at the Nogales plant. This Minute also authorized the relocation of the plant; however, the Minute provided that Mexico’s share of the construction costs of enlarging the international sewage treatment facilities would not change if the United States for domestic reasons constructed the enlarged treatment plant north of its existing site. Mexico conditioned its approval of the relocation on the agreement that Mexico not bear any costs associated with the extension of the IOI pipeline necessary for the relocation, according to USIBWC officials. Further, under Minute 227, the United States, Mexico, and the City of Nogales, Arizona, shared the construction costs of the treatment plant. During the relocation of the plant and resulting extension of the IOI, the City of Nogales acquired all easements in land or the land necessary for the relocation and contributed $791,000 for the expanded plant and IOI, according to USIBWC officials. Mexico’s share was based on the costs of enlarging the treatment plant at the site used for the initial 1951 plant. Since the City of Nogales, Arizona, wanted the plant to be located away from the city limits, the additional IOI costs were not borne by Mexico. The second plant was upgraded in 1988, 1992, and 2009. In 1988, Mexico provided $1 million to pay for the additional capacity built at the plant, as the total capacity allotted to Mexico after the upgrade was 9.9 million gallons per day, and the City of Nogales, Arizona, was allotted a total capacity of 4.84 million gallons per day. The United States and the City of Nogales, Arizona, shared the costs for the 2009 upgrade to the facility. During the 2009 upgrade, EPA provided a $65 million grant to the City of Nogales, Arizona; the City of Nogales, Arizona, contributed $700,000; and USIBWC provided an additional $2 million for the construction of an ultraviolet disinfection system according to USIBWC documents. For the South Bay plant, the United States and Mexico agreed to construct the plant under Minute 283 and to share the costs for construction, operation, and maintenance for the plant under Minute 296. Congress authorized USIBWC’s participation in 1987 amendments to the Clean Water Act. The construction cost for the plant was $241.1 million. The United States contributed $224.6 million—specifically, EPA provided $127.4 million to USIBWC for costs associated with the construction of the plant and related infrastructure, $89.2 million to the City of San Diego and the Corps to construct the South Bay Ocean Outfall, and $8 million to the Corps for additional environmental work. Mexico contributed $16.8 million, which was the amount that it would have had to pay to construct and maintain a plant in Mexico. As part of Minute 283, IBWC also built a diversion infrastructure just south of the border to capture low-volume, dry-weather flows in the Tijuana River to prevent northbound transboundary flows into the United States. This diversion system is operated by Mexican entities and includes pumps and pipelines that send wastewater to the South Bay plant. Minutes also specify cost-sharing arrangements for the ongoing operation and maintenance of the plants. Under the cost-sharing agreements in relevant minutes, the Mexican government generally reimburses USIBWC annually for a portion of the treatment costs at each plant. The reimbursement rate is annually adjusted based on what it would cost to treat a similar amount of wastewater in Mexico according to USIBWC officials. In addition, USIBWC has a separate agreement with the City of Nogales, Arizona, for the Nogales plant that stipulates reimbursements for their sewage treatment. These minutes and cost-sharing arrangements are as follows: Cost-Sharing Agreements for the Nogales plant. Under Minute 206, Mexico agreed to pay for some operations and maintenance costs, based on its proportion of wastewater flows to the Nogales plant for treatment, at a discounted rate for a predetermined amount of sewage. IBWC commissioners periodically review this discounted rate. Specifically, USIBWC assesses the percentage of sewage (up to 9.9 million gallons per day) Mexico sends to the Nogales plant and adjusts the rate to what it would cost to perform the same service in Mexico, according to USIBWC officials. Furthermore, the Mexican government has agreed to pay full U.S. cost for any flow above the treaty-allotted 9.9 million gallons per day, according to these officials. Meters located at three sites along the U.S-Mexico border continuously measure the sewage flow, and if the amount of sewage treated by the plant exceeds the 9.9 million gallons per day, Mexico is billed by USIBWC for the full cost of sewage treatment, according to USIBWC officials. Separately, USIBWC charges a rate for treatment of the city’s sewage under a Memorandum of Agreement with the City of Nogales, Arizona. Cost-Sharing Agreements for the South Bay plant. Under Minute 296, Mexico agreed to pay for operations and maintenance costs for the plant based on the treatment of up to 25 million gallons per day. The pump that diverts Tijuana’s wastewater into the South Bay plant can pump as much as 29 million gallons per day, and the plant can treat more than 25 million gallons per day if needed. Similar to the Nogales plant, USIBWC, on a quarterly basis, bills the Mexican government a prorated amount for the treatment services based on the amount of flow. For example, in fiscal year 2018 Mexico paid USIBWC about $2.4 million for treatment of its wastewater. In fiscal year 2018, the plants’ operational and maintenance costs totaled $4.5 million for the Nogales plant and $15 million for the South Bay plant, and in that fiscal year, the Mexican government reimbursed USIBWC $4.4 million for both plants, according to USIBWC documents. In addition, according to USIBWC officials, the City of Nogales, Arizona is behind in its payments for the Nogales plant by $3 million, and Mexico owes $3 million, according to officials. USIBWC initially pays for the operations and maintenance costs at all its facilities, including the two wastewater treatment plants, and then seeks reimbursement from Mexico and the City of Nogales, Arizona, for their portions of the operation and maintenance costs. The operation and maintenance costs for each plant include the plant’s employees, such as water operators and skilled technical employees, who manage nonstop operations such as running the equipment, controlling the processes, and monitoring the facilities. The Nogales plant employed 17 people as of 2019. USIBWC has used a third-party contractor (Veolia Water Operating Services) to conduct operational and maintenance activities at the South Bay plant since 1998, according to officials. USIBWC Salaries and Expenses budget line item includes funding for each plant’s operation and maintenance. According to federal officials in the United States and Mexico, the operations and maintenance of wastewater infrastructure in Mexico is an ongoing challenge. According to these officials, Mexican wastewater utilities do not have the resources or the long-term technical expertise to address equipment maintenance problems in a timely manner to prevent spills. Although NADB has provided financing to wastewater infrastructure utilities that send wastewater to USIBWC’s Nogales and South Bay plants, utilities often the lack the resources necessary to adequately maintain the infrastructure and equipment after the construction loan ends, according to NADB officials. NADB could condition financing for every wastewater infrastructure project on capacity to adequately manage operations and maintenance of the infrastructure, as it has for a few projects, according to USIBWC and EPA officials. In the United States, as we reported in January 2016 the U.S. Department of Agriculture includes as one of its loan conditions the capacity of the wastewater utility to pay for operations and maintenance of infrastructure. USIBWC has identified numerous projects related to operating the plants or building new infrastructure that remain unfunded under the agency’s current appropriations level, according to an agency document. USIBWC’s Budget Office, as part of its Fiscal Year Year-end Budget Procedures and Guidance, annually sends its staff a report with the projected balances for the Salaries and Expenses and Construction line items for the remainder of that fiscal year. The guidance directs that each department—Engineering, Operations, and Administration—identify work or projects for which they need funding. Each projected balance is to be calculated by subtracting expenses from each group’s allocated funding for the year. The departments are to identify any outstanding requirements and associated costs for the remainder of the fiscal year. For fiscal year 2018, USIBWC identified $9 million in potential operations and maintenance work and $2.8 million for potential construction projects, based on agency documents. For example, USIBWC identified the need for $149,000 for new pumps and motors for pump stations at the South Bay plant but deferred the purchase due to other funding needs, according to an official. Several factors can affect each plant’s operations. IBWC and others have taken some actions to address the factors affecting each plant’s operation, including initiating an informal binational rapid response team to address breaking and failing wastewater infrastructure along the border. However, IBWC has not taken the necessary steps to formalize this rapid response team, and raw sewage continues to periodically spill into the Santa Cruz River Basin and Tijuana River Valley watersheds. USIBWC’s Nogales and South Bay plants are subject to NPDES permits issued by the states of Arizona and California, respectively, which generally prohibit the discharge of pollutants from the plants unless specifically allowed under the permit. Generally, a NPDES permit is issued for a term of 5 years to a single facility and reflects site-specific conditions of that facility. The Nogales plant’s NPDES permit requirements are based on a maximum monthly average of 17.2 million gallons per day to be treated and discharged into the Santa Cruz River. The permit allows the discharge of certain pollutants within specified limits, including some heavy metals, such as mercury and copper. Under the permit, USIBWC must also meet several monitoring requirements, including monitoring the pollutants in the water coming into the plant from the IOI, the amount of treated wastewater discharged into the Santa Cruz River, and the presence of pollutants named in the permit. USIBWC is to submit this information to Arizona Department of Environmental Quality (ADEQ) for monthly or annual review. The Nogales plant permit also requires USIBWC to remove sludge produced as part of the treatment process and dispose of it at an offsite location that is certified to receive that type of byproduct. Since 2014, ADEQ has issued four Notices of Violation to USIBWC for the Nogales plant’s permit. The notices cited the exceedances of certain substances above permit limits, including some heavy metals in the discharge (in 2019); the presence of pollutants toxic to human, animals, plants, or other organisms (in 2018); untreated sewage spilled into a tributary of the Santa Cruz River (in 2017); and USIBWC’s failure to accurately monitor and report specific substances to ADEQ as outlined in the permit (in 2014). Each notice outlined actions that USIBWC was required to take to improve the water quality problem identified within a specific time frame. The South Bay plant has not received any Notices of Violation under its current NPDES permit, which was issued in 2014, according to USIBWC officials. The current permit covers the South Bay plant and other infrastructure including five canyon collectors and the South Bay Ocean Outfall. The permit sets a discharge limit of 25 million gallons per day of treated wastewater, on a monthly average, to the Pacific Ocean through the South Bay Ocean Outfall. The permit limits the pollutants that can be discharged, such as zinc and mercury. Under the permit, USIBWC and the City of San Diego conduct a joint monitoring program of the wastewater discharge at the South Bay Ocean Outfall and are required to submit the data collected from this joint monitoring effort to the San Diego Water Board. The permit also includes monitoring requirements for other parameters, including heavy metals and organic chemicals that are considered harmful to the environment and public health. The South Bay plant has not violated the permit’s discharge limits through the South Bay Ocean Outfall since secondary treatment began in 2010, according to USIBWC officials. During rainstorms or wet weather in Tijuana and when pipelines or pumps break, the plant does not treat all the water flowing from Mexico. During these events, water flows to the Tijuana River and canyons and mixes with unknown amounts of urban runoff, treated effluent from the Tijuana River, and wastewater in Mexico and then flows into the Tijuana River Valley watershed in the United States. During dry weather, the runoff is largely groundwater and some untreated discharge from illegal connections (dry-weather flows); during storms, this runoff mixes with large amounts of rainfall (wet-weather flows). There are several factors that can affect the operation of the Nogales plant. Lack of heavy metal pretreatment in Mexico. In Mexico, metal treatment and plating facilities operate in Nogales, Sonora and directly discharge wastewater that contains heavy metals into the city’s sewer systems, which end up at the Nogales plant for treatment. While a municipal pretreatment program exists in Nogales, Sonora, it is designed to meet Mexico’s minimum federal requirements and is insufficient to detect and respond to the dumping of industrial contaminants when they occur, according to ADEQ documentation and officials. Deteriorating sewage infrastructure in Mexico. Sewage infrastructure in the City of Nogales, Sonora, is not adequately maintained, according to USIBWC officials. As a result, the city of Nogales, Sonora, sends wastewater amounts to the plant in excess of the amount agreed upon in the minute between USIBWC and the Mexican section. Due to the proximity of the plants to the U.S.- Mexico border, USIBWC’s international wastewater treatment plants in southern California and southern Arizona are located in areas patrolled by Customs and Border Protection (CBP) agents. In southern California, the waterways in which sewage pipelines connect to the South Bay International Wastewater Treatment Plant provide a natural crossing point at the border, which CBP has blocked with gates. In southern Arizona, drug smugglers use the International Outfall Interceptor pipeline— which transports sewage from Mexico to the Nogales International Wastewater Treatment Plant—to transport drugs. According to CBP officials, smugglers in Mexico drop drug bundles into manholes that connect to the pipe, and smugglers in the United States cut into the pipe to retrieve the bundles. These holes in the pipe can cause sanitary sewer spills in Nogales, Arizona. CBP agents patrol along the pipeline to catch smugglers and retrieve the drug bundles, according to CBP officials. Deteriorating infrastructure in the United States. In the United States, the deteriorating condition of the IOI causes untreated sewage to periodically spill into the Santa Cruz River watershed and Nogales Wash. The deterioration is due to the age of the pipe, as well as ongoing corrosion and erosion of the pipeline (see fig. 6). See appendix III for more details on the factors that affect the operations of the Nogales plant. One key factor affects the operation of the South Bay plant: insufficient sewage infrastructure in Mexico contributes to transboundary sewage flows that, if not diverted, can reach the plant and disrupt its operations. According to a 2019 study, Tijuana has not built sufficient sewage infrastructure to serve the area’s exponential population growth and urbanization. When problems arise with Tijuana’s treatment facilities, the city diverts a portion of its wastewater for treatment at the South Bay plant. In these instances, the Mexican utility may also shut down Pump Station CILA, a main pump located in the Tijuana River that diverts the river to the treatment plant. If the South Bay plant is not notified and does not shut down its pump and canyon collectors, it may receive additional flows. While the plant can treat additional wastewater and has not violated its NPDES permit, the plant is experiencing an increase in the number of days that it treats above capacity, according to USIBWC officials. In addition, USIBWC officials stated that the South Bay plant is not designed and operated to address some of the wastewater that flows into the Tijuana River Valley watershed. These wastewater flows are due to: Limited Tijuana Basin diversion infrastructure. The Tijuana Basin diversion system consists of the Mexican-operated Pump Station CILA and the South Bay plant’s canyon collectors. This system captures dry-weather flows for treatment at the South Bay plant or for a wastewater treatment plant in Mexico. However, it is not designed to capture high flows that result from pipe breaks or pump failures. To avoid affecting the South Bay plant’s wastewater treatment operations, during incidents of high flows, Pump Station CILA and the five canyon collectors are shut off. During these events, the water bypasses the South Bay wastewater treatment plant and flows untreated into the Tijuana River and watershed. For example, a February 2017 spill from a broken pipeline in Mexico released 143 million gallons of sewage-contaminated water into the Tijuana River that bypassed the South Bay plant and was not treated. Lack of maintenance for existing sewage infrastructure in Mexico. A lack of maintenance for Tijuana’s existing sewage infrastructure causes excess wastewater flows into the Tijuana River according to USIBWC officials. For example, in August 2019, USIBWC reported that on June 19, 2019, 1.9 million gallons of wastewater were released into the Tijuana River because of trash buildup at one of Tijuana’s pumps that caused the pump to fail. A 2019 study also reported that the poor condition of critical wastewater infrastructure in Mexico results in approximately 30 percent of Tijuana’s wastewater entering the Tijuana River or Pacific Ocean without treatment. See appendix III for more details on these factors that affect the operations of the South Bay plant. USIBWC and the Mexican Section have taken some actions to address the factors that can impede plant operations. However, raw sewage is still released from Mexico into the Santa Cruz River Basin and Tijuana River Valley watersheds and continues to have significant public health and environmental impacts. USIBWC and others have taken various actions to address the factors that affect Nogales plant operations, including the following: Sending letters to heavy metal dischargers. To address the presence of heavy metals in the wastewater stream, in October 2018, USIBWC, ADEQ, and EPA sent joint letters to four American companies affiliated with the metal treatment and plating facilities in Nogales, Sonora, Mexico. The letters asked for the companies’ cooperation in addressing the issue and offered to meet with each company to discuss possible solutions. According to USIBWC officials, they received a response from one company, but not the other three. However, in continued monitoring, USIBWC has seen fewer instances of heavy metals in the wastewater that it treats at the Nogales plant according to agency officials. Maintaining treatment capacity in Nogales, Sonora. To address the inadequate wastewater infrastructure in Nogales, Sonora, IBWC has collaborated with other stakeholders to maintain wastewater treatment capacity in Mexico. For example, the U.S. State Department sent a diplomatic note to the Mexican government in February 2019 regarding the failing pumps and asked the Mexican government to quickly respond and eliminate the discharges that end up at the Nogales plant. USIBWC officials stated that the Mexican Section of the IBWC purchased two new pumps, which were expected to arrive at the pump station in Nogales, Sonora, in late 2019. The Mexican Section also plans to work with the local utility to install equipment to remove grit from the wastewater and prevent degradation of the new pumps. Upgrading infrastructure in the United States. In 2005, USIBWC proposed a five-phase plan to rehabilitate the IOI’s pipe that uses a process referred to as “cured-in-place pipe.” In this process, a polyester tube is inserted into the pipe and inflated, which then hardens to become a pipe within a pipe. This process has an estimated cost of $50 million. As of November 2019, the rehabilitation had not started due to funding disagreements between USIBWC and the state of Arizona. According to USIBWC officials, the agency does not want to fund the entire project but has secured $28.1 million for it. According to USIBWC officials, the City of Nogales will not contribute any funding without a change to the current cost-sharing agreement on reimbursements between the city and USIBWC for sewage treatment. The cured-in-place pipe process will address some of the IOI’s deferred maintenance issues but will not resolve ongoing disagreements about which entity is responsible for funding annual maintenance and operations. According to USIBWC officials, the annual maintenance needs include more than the work to repair the IOI. For example, lateral pipelines that connect City of Nogales sewers to the IOI also need to be maintained; occasional breaches in the pipeline need to be repaired; and vegetation management along the pipeline is necessary to prevent root intrusion into the pipeline. USIBWC officials estimated the annual cost for operations and maintenance, including infrastructure repair and personnel costs, at about $1.5 million to $2 million. IBWC and others have also taken actions to address the pump failures and pipeline breaks in Tijuana that send polluted flows downstream, affecting the Tijuana Basin diversion infrastructure and subsequently the South Bay plant. These actions include the following: Negotiating a Binational Tijuana River Spill Notification Protocol. In August 2017, IBWC negotiated a notification protocol for raw sewage discharges into the Tijuana River that may enter the United States. The protocol was prompted by the February 2017 spill from a broken pipeline in Mexico of 143 million gallons of sewage- contaminated water that flowed into the Tijuana River. The initial protocol stated that a formal memorandum of understanding would be developed at a later date to formalize the protocol; however, the initial protocol remains in place. According to an USIBWC official, Mexico has since adhered to the protocol twice by warning USIBWC of imminent raw sewage flows when pipelines in Tijuana, Mexico, ruptured. However, in August 2019, USIBWC reported that most of the transboundary flows were detected by an automated alert system on the U.S. side of the border that was deployed by USIBWC in October 2018 to better monitor and detect any transboundary flows. The system relies on river gage data recorded at the Tijuana River that is also posted to the USIBWC website. Upgrading infrastructure in Mexico. In April 2018, the Department of State sent a diplomatic note to the Mexican government after failures in Tijuana’s sanitation infrastructure led to sewage flows on multiple days in 2017 and 2018. The diplomatic note requested that the Mexican government take appropriate measures (as outlined in Minute 283) to stop sewage flows from crossing into the United States, including making short-term repairs and longer-term upgrades. According to USIBWC officials, Mexico does not have much funding for its infrastructure. However, in March 2019, Mexico and EPA, through NADB, funded the replacement of three segments of the Poniente Collector in Tijuana, Mexico, to eliminate a key source of untreated discharges into the Tijuana River in the United States. Participating in the Tijuana River Diversion Study. In 2019, NADB funded the study of alternatives to expand or adapt the diversion infrastructure in the Tijuana River to identify potential infrastructure projects (and associated costs) to divert dry-weather flows and possibly some flows that result from wet weather mixed with wastewater and raw sewage. The study developed project alternatives in Mexico, the United States, or both countries that would reduce the number of days that transboundary flows occur, including by diverting more wastewater through the South Bay plant to prevent its release in the United States. The alternatives range in cost from $8 million to $236 million. USIBWC, the Mexican Section, the EPA, the Mexican National Water Commission, and the Tijuana water utility also coordinated on the study, which was completed in July 2019. Even with the efforts of IBWC and others, raw sewage continues to be released in both watersheds due to deteriorating and insufficiently maintained sewage infrastructure primarily in Mexico, with the exception of the IOI in the United States. In the Santa Cruz River, the presence of raw sewage in Nogales Wash and the river continues to threaten public health and the survival of fish and wildlife, including endangered species, according to representatives of Friends of the Santa Cruz River, a local nonprofit organization. Similarly, raw sewage containing E. coli and other pathogens continues to flow into the Tijuana River and watershed primarily during storm events or breaks in infrastructure in Tijuana, contributing to public health concerns and beach closures in southern California. To address the continuing release of raw sewage due to pipe breaks and pump failures, at an IBWC meeting in spring 2019, USIBWC proposed the development of a rapid response team comprised of technical experts from both countries that could immediately respond to infrastructure problems, such as pipe breaks and pump failures. This team would take actions to mitigate sewage leaks along the border such as those in Nogales, Sonora, and Tijuana. For example, the team would respond immediately to situations in which a pipe break in Mexico causes wastewater to flow into the United States and would put in place appropriate diversions and equipment to repair the break. Members of the team would come from both countries, and funding for their deployment would come primarily from the United States. USIBWC has not estimated the cost to form and annually support the binational team. The principal engineers from both USIBWC and the Mexican Section have agreed to start building the team with their respective staff, according to a USIBWC official. However, this agreement is informal, and IBWC has not taken the necessary steps to formalize the team. Such steps could include preparing a minute. Specifically, Minute 261 states that for each border sanitation problem, IBWC is to prepare a minute identifying: (1) the problem; (2) the conditions which require solution; (3) specify quality standards that should be applied; (4) the course of action that should be followed for its solution; and (5) the specific time schedule for its implementation. According to IBWC officials, the benefit of a minute is that it functions as a formal agreement between the respective governments, encouraging them to provide greater support through funding and other resources to ensure the solutions and projects are implemented. According to USIBWC officials, they also have alternatives to negotiating a minute, such as issuing a joint report, and a minute may not be necessary for the countries to formalize their commitment. For example, IBWC could exchange formal letters signifying their intent to form the team or issue a joint report written by each IBWC section’s principal engineers. By formalizing a binational rapid response team to address sewage infrastructure failures along the U.S.-Mexico border, including the watersheds around the Nogales and South Bay plants, USIBWC would have better assurance that it is able to more effectively address the urgent and recurring sewer breaks and pump failures in Mexico that contribute to raw sewage spills. USIBWC has taken some actions to address water quality problems at both plants, but USIBWC and the Mexican Section have not taken actions to address unmanaged stormwater flows and their associated water quality problems. USIBWC officials stated that the agency does not have the authority to manage stormwater problems in the Santa Cruz River Basin or Tijuana River Valley watersheds without direction by Congress. Further, USIBWC has not fully incorporated key planning principles for long-term capital planning that would help it identify alternative approaches for resolving the ongoing water quality problems along the border. USIBWC and others have taken some actions to address the water quality problems that exist in the two watersheds, but USIBWC has not taken actions that include identifying alternatives to address stormwater and stormwater quality in the Santa Cruz River Basin watershed or in the Tijuana River Valley watershed. As a result, unmanaged stormwater flows largely untreated downhill from Mexico, carrying bacteria, trash, and sediment into the lower portions of the Santa Cruz River Basin and Tijuana River Valley watersheds where the Nogales and South Bay plants are located, threatening key infrastructure and complicating water quality management in the watersheds. The stormwater carries the pollutants across the border, depositing them in the river channel, shorelines, nearby wetlands, and—in the case of the Tijuana River—ultimately the ocean, causing public health and environmental concerns in the United States. In addition, stormwater can damage plant infrastructure. The Nogales Wash is the main drainage for the cities of Nogales, Sonora, and Nogales, Arizona. Stormwater from the upper watershed flows into the wash and crosses the border, carrying bacteria and sediment into the United States. According to IBWC officials, because Nogales, Sonora, does not have adequate stormwater sewers, Mexican citizens remove manhole covers to allow stormwater to drain from the streets into the sanitary sewers during heavy rainstorms. The IOI essentially becomes a combined sewage system—one in which wastewater and stormwater flow in the same pipelines—even though it was not designed as such, according to USIBWC officials. The excess stormwater causes increased pressure in the IOI that is released when the manholes in the United States overflow, sending sewage into the streets of Nogales, Arizona. In July 2018, ADEQ documentation noted that Nogales, Sonora, experiences frequent flooding during heavy rain events in the summer and uses the IOI to mitigate flood events, which results in releases of untreated sewage into the residential and business neighborhoods in the City of Nogales, Arizona and the Santa Cruz River watershed. For example, in 2017, Santa Cruz County Health Services and the Arizona Department of Health Services released public health advisories for elevated E. coli for the City of Nogales, Arizona, due to untreated sewage leaking from the IOI. According to one of these advisories, stormflows are typically high in pollutants that can be harmful to human health such as bacteria and pathogens. Unmanaged stormwater flowing into the Nogales Wash can destabilize the IOI, which runs inside or below the wash, from the border to the Nogales plant. Stormwater rushing down the wash erodes and removes natural and manmade materials covering the pipeline, such as the cement panels lining the middle portion of the wash (see fig. 7). For example, in July 2017, flooding in the Nogales Wash eroded the soil around a manhole in the IOI, partially shearing the pipe and causing untreated wastewater to flood into the wash and into the streets of Nogales, Arizona, resulting in elevated levels of E. coli in the wash and Santa Cruz River. As a result, the Arizona Governor’s Office declared a State of Emergency in Santa Cruz County and sent a notice of the Nogales plant’s permit violation to USIBWC. To date, USIBWC’s actions have focused on emergency repairs and cleanup when untreated sewage has leaked from the IOI into the Nogales Wash and Santa Cruz River. During the July 2017 event, for example, to prevent further contamination of the wash due to the release of raw sewage leaking from the broken section of the IOI, USIBWC hired a contractor to install a bypass system to divert the raw sewage spilling into the wash to the Nogales plant for treatment. Other stakeholders also took action. For example, at the request of the Arizona governor’s office, the Corps stabilized earthen banks along the Nogales Wash that had eroded. The Arizona Army National Guard and Arizona Department of Transportation also took part in similar efforts. In general, the Nogales Wash is not regularly maintained to stabilize the earthen banks and concrete panels to prevent erosion. According to USIBWC officials, operations and maintenance of the Nogales Wash and management of stormwater in the Nogales Wash is a municipal responsibility and not the responsibility of the IBWC. As a result, USIBWC has not taken action to manage the Wash to prevent stormwater damage to the IOI. Instead, it has—as with the example above—sought to bring in other federal agencies that USIBWC says have authority over domestic water management. However, Nogales city managers do not accept responsibility for managing the wash, stating that it is IBWC’s responsibility. USIBWC and other federal agencies have conducted some studies in Mexico to address stormwater management in the watershed. For example, USIBWC contracted the Corps to conduct an evaluation to develop measures to reduce the threat of flooding and alternatives to reduce potential flood damage in Nogales, Sonora. The study was completed in 2004. Based on the recommendations in the evaluation, Nogales, Sonora, and the Mexican federal water agency, constructed 14 dams and detention basins from 2008 through 2015. However, according to USIBWC officials, the basins that are in Mexico and maintained by the local utility are full of sediment and have not been cleared because the local Mexican utility does not have funds to maintain them. In addition, USIBWC and the U.S. Geological Survey have collaborated on joint studies of the watershed surrounding Nogales, Sonora, for many years according to USIBWC officials. For example, one study completed in 2016 was to be the basis of further work to identify stormwater management projects, but that work has not been planned or conducted. (See app. IV for details of additional studies.) In the absence of an entity that regularly maintains the wash, the IOI is still threatened when stormwater runs through the wash. IBWC has not contracted for or conducted a study to identify long-term solutions to the stormwater quality problems in the watershed, like was done with the Tijuana River Diversion Study. Instead, since 2005, USIBWC has responded to events that threaten the IOI as they occur at the request of the City of Nogales, Arizona, and used an emergency response authority that is applicable to the U.S.-Mexico border, according to USIBWC officials. The Mexican Section also has not addressed maintenance of the already insufficient stormwater conveyance infrastructure in Nogales, Sonora. Without resolution, the unmaintained wash and inadequate stormwater infrastructure in Mexico threaten the stability of the IOI with additional stormwater damage. Stormwater carries trash into the canyons that cross the border area, as well as bacteria from illegal sewer connections and infrastructure breaks in Tijuana, and sediment that erodes from the steep hills of Tijuana. As part of routine operation and maintenance, USIBWC annually removes trash and clears sediment from the grates in the South Bay plant’s five canyon collectors according to agency officials (see fig. 8). The pollutants carried in the transboundary stormwater also cause ongoing degradation to the riparian and estuarine habitats within the lower Tijuana River Valley, impacting ecological diversity, wildlife, and ceremonial and recreational use of the area. For example, from 2003 through 2017, the City of Imperial Beach, California, closed public beaches for at least one-quarter of the year and half the year in some years due to bacterial contamination in the Tijuana River, according to city officials. Although the parties dispute the source of pollution causing the closures, the raw sewage that enters into the Tijuana River Valley and flows with stormwater into the ocean is a likely source of pollution. In response to the bacteria and trash problems caused by flows from Mexico into the Tijuana River Valley, the California Regional Water Quality Control Board, San Diego Region, initiated the development of two TMDLs—for bacteria and trash—for the Tijuana River. If the TMDLs are applied, USIBWC would be responsible for meeting the TMDL requirements, according to a California state official; USIBWC disagrees. If it were subject to a TMDL, USIBWC would be expected to oversee the trash collection and removal in the United States even if the trash originated in Mexico. USIBWC maintains that its ownership of the Tijuana Flood Control Project does not make it responsible for the quality of water flowing in that project from Mexico under the Clean Water Act. As of November 2019, the issue of whether USIBWC should take action to resolve these pollutant problems is in litigation. In 2015, IBWC also negotiated a minute to address stormwater effects in the Tijuana River Valley, Minute 320, General Framework for Binational Cooperation on Transboundary Issues in the Tijuana River Basin. According to USIBWC officials, the minute was developed after local stakeholders in California asked Mexico to take action to address stormwater problems in the United States. Mexico responded that it participates in binational solutions to issues through IBWC. Under Minute 320, the United States and Mexico acknowledged that binational coordination is required to address stormwater flows that carry bacteria, trash, and sediment, as well as other pollutants that threaten the Tijuana River Basin. growth of aquatic vegetation and decrease spawning areas and habitats for fish and other organisms. rubber, and construction material—settles on the bottom of waterways, affecting bottom feeding organisms. In response, IBWC formed three binational working groups composed of local, state, and non-governmental stakeholders to conduct studies to identify the sources of bacteria, trash, and sediment that stormwater flows carry into the Tijuana River Valley. The working groups are tasked with recommending solutions to the problems based on the studies’ findings. However, Minute 320 did not set a timeline for completion of the studies nor did it identify sources of funding for potential projects recommended by the working groups. According to USIBWC officials, Minute 320 anticipates that there may be variety of sponsors and funding resources for projects recommended by the working groups. The three groups stopped meeting in 2017. In June 2019, the water quality and sediment groups resumed meetings, but as of September 2019, the trash working group had not reconvened. According to USIBWC officials, as of November 2019, IBWC is convening a meeting of a reconstituted Minute 320 Binational Core Group, following up on stakeholder recommendations to re-establish and strengthen the Minute 320 process: Water quality working group. The water quality group is working on an ongoing binational water quality monitoring program that began in December 2018 and was to end in November 2019. The group is sampling sewage and other flows at various locations in the United States and Mexico to establish baseline data for pollutants in the waters of the Tijuana River watershed according to USIBWC officials. Sediment working group. The sediment group is working on an ongoing sediment detention feasibility study funded by USIBWC to identify the most effective means of sediment management within the Tijuana River channel. The sediment working group had recommended the study. USIBWC estimates the cost of removing sediment at $15 million per year, based on an estimated 492,000 tons of sediment entering the river each year and about three-quarters of it being removed. According to USIBWC officials, the sediment working group expects to complete the study in early 2020. Trash working group. The working group has developed the scope of work for a binational study of trash booms in different sites along the Tijuana River. It is waiting on funds to perform a feasibility study. USIBWC and several state and local agencies have taken further actions to address these water quality problems in the Tijuana River Valley Watershed, including the following: Constructing a temporary earthen berm in the Tijuana River Channel. In 2018, USIBWC constructed a temporary earthen berm in the U.S. section of the concrete channel of the Tijuana River, close to the border. The purpose of the berm was to hold back low-volume, dry-weather flows contaminated with untreated sewage; however, some sediment and sewage still enters the Tijuana River Valley during high-volume flows or storm events because those flows permeate the berm according to USIBWC documents. USIBWC officials said the berm is just a temporary measure to capture low- volume flows of sediment and trash during dry weather and is not intended as a long-term solution for the river channel. Monitoring water quality in the Pacific Ocean. To understand the sources of beach pollution, USIBWC contracts with the City of San Diego to regularly monitor water quality in the Pacific Ocean, in particular around the discharge points for the city’s wastewater treatment plant and the South Bay plant. Starting in 2018, the City of San Diego and USIBWC began a joint program to track the extent of dispersion of sediment into the Pacific Ocean where the Tijuana River empties into the ocean according to USIBWC and City of San Diego officials. Collecting and disposing of trash and sediment. Several state and local agencies collect and remove sediment from their land parcels in the Tijuana River Valley. For example, California State Parks placed a boom across the floor of one of the five canyons to collect trash and sediment from stormwater flows (see fig. 9). Since 2015, California State Park employees annually collect and remove trash and sediment from the rack and disposes of it at a local landfill and quarry, at a cost of $1.8 million per year. In addition, the U.S. Customs and Border Protection agency also removes trash and debris from grates associated with four of the five cross-border canyons as often as necessary to protect the health of agents conducting daily patrol operations. Identifying projects to reduce sewage, trash, and sediment, in the Tijuana River Valley. The County of San Diego is funding an assessment to identify and prioritize potential projects that could be implemented in the United States to improve the water quality in the Tijuana River Valley by addressing transboundary flows of sewage, trash, and sediment. The county expects the assessment to be completed in March 2020, and intends to work with partners in the region to identify funding and other resources necessary to implement the highest priority projects, according to San Diego County officials. (See app. IV for additional studies.) As of October 2019, USIBWC officials said they were reviewing alternatives outlined in the 2019 study of alternatives to expand or adapt the diversion structure for the South Bay plant to address transboundary sewage flows. In December 2019, local government officials in California passed a resolution supporting a set of projects to be built on the U.S. side of the border to resolve the water quality problems. The mayors of several California municipalities endorsed EPA to receive funding to construct projects on the U.S. side of the border to help resolve water quality problems in the Tijuana River basin. In January 2020, a large trash buildup in a storm drain on the border caused putrid water to back up in Tijuana, highlighting the nature of the trash and sediment problem in the upper watershed, which also affects the lower watershed. In December 2019, a congressional committee identified the need for EPA to lead the efforts to resolve these problems. According to USIBWC officials, while the most cost-effective solutions are in Mexico, the Mexican government lacks resources to make all of the infrastructure improvements. However, officials told us the proposed solutions on the U.S. side of the border may be more expensive or difficult to implement in part due to other constraints to the United States. For example, one of the alternatives would divert untreated sewage to the South Bay Ocean Outfall for direct discharge into the Pacific Ocean, but the discharge likely would not meet Clean Water Act standards. According to USIBWC officials, solutions that lead to violations of Clean Water Act standards would not be acceptable to USIBWC, EPA, or other U.S. stakeholders. According to EPA officials, USIBWC has expertise in operating and managing water and wastewater infrastructure, while EPA has expertise in addressing water pollution. In addition, EPA officials stated that USIBWC’s binational presence and ability to work across the border is important to deal with operations and maintenance issues, such as clearing stormwater channels. EPA officials stated that their role in coordinating with USIBWC is important for identifying and addressing specific water quality problems. For example, joint efforts by both agencies through the Mexicali Binational Sanitation Observation and Technical Committee led to successful solutions to wastewater pollution and trash problems through joint monitoring and site visits, according to EPA officials. USIBWC officials stated that the agency does not have the specific authority to manage stormwater problems in the Santa Cruz Basin or Tijuana River Valley watersheds without the direction of Congress. Minute 261 states that IBWC shall “give permanent attention to border sanitation problems and give currently existing problems immediate and priority attention.” In addition, OMB Circular A-94 calls for agencies to assess the benefits and costs of alternative projects. Although IBWC, USIBWC, and others have taken some actions to address stormwater quality problems in the Santa Cruz River Basin and Tijuana River Valley watersheds, such as conducting studies of stormwater and building some retention basins, the problems have nevertheless continued to occur over many years, and no entity has taken action to identify alternatives, cost estimates, funding sources, or time frames for implementing them. USIBWC officials stated that feasibility studies and analyses are necessary steps in justifying requests for funding a project and investigating the cost and technical feasibility of a project. While USIBWC has conducted some feasibility studies on different individual solutions, it has not done a comprehensive study to recommend any overall solutions to address the transboundary stormwater problems of bacteria, trash, and sediment in either watershed. According to USIBWC officials, previous projects it has built in Nogales and South Bay were developed with federal, state, and local partnerships and with congressional approval. In particular, USIBWC officials stated that the agency does not have specific authorization for stormwater management in the watersheds surrounding the Nogales and South Bay plants because the 1944 Treaty and accompanying legislation did not authorize that the agency carry out projects for stormwater management along the border. USIBWC’s role in addressing certain transboundary stormwater flows and associated water quality problems is in dispute in ongoing litigation involving the Santa Cruz and Tijuana River basins, and USIBWC officials stated that they would not take action to resolve the stormwater quality problems without congressional direction. Yet without action, the long-standing environmental and health problems associated with transboundary stormwater flows in the watersheds of both rivers will continue. Under these circumstances, Congress has the opportunity to provide direction and specific authorization for USIBWC to take action. Such action would include identifying alternatives, cost estimates, funding, and time frames. USIBWC has not fully incorporated key capital planning principles that would help identify alternative approaches to address water quality problems in the Santa Cruz or Tijuana River Valley watersheds. In 2019, OMB issued a Capital Programming Guide that supplements Circular A- 11, which provides guidance on capital programming, including key capital planning principles (see table 1). In February 2007, we reported that OMB’s guidance on capital planning requires long-range planning and a disciplined decision-making process as the basis for managing assets to achieve an agency’s goals and objectives. We also reported that the planning phase is the most important for the capital decision-making process and that it links capital asset investments to an organization’s overall mission and long-term strategic goals. We emphasized that agencies should evaluate a full range of alternatives to bridge any performance gap and recommended that Congress require agencies to develop long-term capital plans and submit them for review. Furthermore, in January 2016, we reported that asset management planning for water utilities includes key components such as assessing the current state of their assets (for example pipelines and treatment plants), incorporating life-cycle costs, and developing a strategy for the long-term funding of the repair and replacement of their assets. In our review of documents and interviews with USIBWC officials, we found that the agency incorporates aspects of the key planning principles in its capital planning and budgeting but has not fully incorporated the principles. For example, the agency has a strategic plan that identifies its goals, including a goal to improve the quality of water along the border. We have stated, along with OMB, the importance of linking capital asset investments to an organization’s overall mission and long-term strategic goals. However, in its capital planning and budget process, USIBWC does not fully assess or identify future needs, as called for in OMB’s key capital planning principles. Those principles state that a needs assessment identifies the resources needed to fulfill both immediate requirements and anticipated future needs, based on the agency’s goals and objectives. According to USIBWC officials, the agency conducts and funds capital planning on a project-by-project basis because it uses year-end money to fund studies or evaluations to identify project needs or alternatives. Specifically, USIBWC engineers identify the need for a project, and the agency identifies year-end appropriations to pay for a study of that project. For example, in one case described by a USIBWC official, the agency contracted with the Corps of Engineers to conduct a study of USIBWC flood control levees and their condition. The agency used year- end funds in its Salaries and Expenses budget line item to pay for the study, and USIBWC officials stated that the study has since been the basis for its request for levee repair and replacement projects. In addition, we found that USIBWC conducts alternative evaluations of potential projects, as directed by OMB’s guidance that states an evaluation should be conducted of a wide range of alternative approaches to determine how to bridge performance gaps in capital infrastructure. According to USIBWC officials, the agency is considering a range of alternatives and plans to conduct an analysis of costs associated with the projects, as leading practices for benefit-cost analysis and alternative comparison suggest. For example, the contractor for the 2019 study of alternatives to expand or adapt the diversion infrastructure for the South Bay project has assessed alternatives and costs to reduce the number of days that transboundary flows cross the border bringing bacteria, trash, and sediment into the United States. However, this was done for one part of the water quality problems created by transboundary flows and will not solve the problems associated with water quality problems created by all stormwater flows. USIBWC has not evaluated alternative approaches or costs for managing stormwater and associated water quality problems in the Santa Cruz River Basin and Tijuana River Valley watersheds that will continue to impact water quality along the border, and states it has no responsibility to do so. USIBWC also has not developed a comprehensive, long-term capital plan to help achieve its strategic goal for water quality. Instead USIBWC has elements of a plan, such as asset management documents for each of its two wastewater plants that identify key equipment replacement costs and schedules. The Nogales plant manager provides USIBWC officials with 10-year cost projections for major equipment, which include information on cyclic maintenance and life-cycle replacements. The operator of the South Bay plant prepares a 5-year plan that assesses the condition of equipment and recommends repair and replacement. However, neither plan identifies gaps in infrastructure needed to resolve water quality problems that are separate from the plants and their normal operation, such as stormwater problems that destabilize the Nogales plant’s IOI pipeline and cause polluted water to be diverted around the South Bay plant. USIBWC states it has no responsibility to do so. As noted above, USIBWC’s role in addressing certain transboundary stormwater flows and associated water quality problems is in dispute in ongoing litigation involving the Santa Cruz and Tijuana River Valley basins. Under OMB’s capital planning principles, conducting long-term capital planning should enable USIBWC to more systematically assess its long- term needs, including its future needs and identify alternative approaches and costs to address stormwater problems in the watersheds. Furthermore, a long-term capital plan should identify the capital projects that USIBWC needs to achieve the strategic goal it seeks to accomplish—in this case, improvement of water quality along the border. In February 2007, we reported that a long-term capital plan can include elements such as (1) a baseline assessment and identification of performance gaps; (2) justification of spending on proposed new assets; (3) the basis for selecting proposed assets; and (4) cost schedules and performance goals. In addition, OMB’s capital planning guidance states that each capital asset should have an operations and maintenance plan that outlines the procedures and responsibilities for scheduled preventive and regular or routine corrective maintenance. Currently, USIBWC has not comprehensively developed this information into a long-term capital plan. A long-term capital plan would help USIBWC budget for capital projects and investments in the watersheds and provide justification for funds requested for capital investment in future water quality projects. OMB Circular A-11 also encourages agencies to use a summary of the capital plan for budget justification to OMB, congressional authorizations of projects, and justification for congressional appropriations. In November 2019, OMB issued a memorandum to federal agencies that reinforced the need to implement the capital programming guidance in OMB Circular A-11 that agencies develop, document, and implement a capital planning process. In its budget process, USIBWC requests funding for individual capital projects for the budget year in which the projects are needed. Specifically, for each annual budget cycle, USIBWC’s Principal Engineers provide information to agency budget officials on the projects they have identified and funds needed for each plant. USIBWC budget officials use this information to prepare budget requests that are then reviewed by the State Department and OMB, and, ultimately, Congress. According to USIBWC’s Administrative Officer, the agency previously provided capital needs in an attachment to the budget requested in OMB Circular A-11. The official told us preparing the information was time-intensive yet helpful. For example, the information helped the agency understand the scope and life-cycle costs of a project. However, when OMB no longer collected agencies’ information, USIBWC stopped providing the information in its budget. According to Department of State budget examiners, USIBWC notifies them of potential infrastructure projects and funding needs; however, this information is not included in the agency’s budget request and is therefore not available to identify funding needs. According to USIBWC officials, they do not provide the information in a budget request to State because they are told to conduct agency operations within a flat budget. By conducting long-term capital planning for the Santa Cruz River Basin and Tijuana River Valley watersheds, following the principles in OMB Circular A-11, USIBWC would have more information to address the water quality problems resulting from unmanaged stormwater in either the Santa Cruz River Basin or Tijuana River Valley watersheds; and could provide the information to State, OMB, or Congress as part of annual budget deliberations. USIBWC and the Mexican Section of IBWC have successfully developed binational solutions to water quality issues along the U.S.-Mexico border, including constructing two international wastewater treatment plants to treat raw sewage that would otherwise flow into the United States. Nonetheless, in the decades since construction of the plants, the communities along the border have experienced exponential growth in populations and development that has, exacerbated by aging and deteriorating infrastructure, resulted in ongoing transboundary flows of raw sewage, trash, and sediment. USIBWC and the Mexican Section have discussed some alternatives to deal with ongoing water quality problems at both plants and in both watersheds. However, water quality problems, including unmanaged and untreated stormwater, bring bacteria, trash, and sediment into the lower watersheds in the United States. To date, USIBWC and the Mexican Section have only studied or monitored the problems; they have not taken actions to resolve the problems by proposing and analyzing alternatives, analyzing costs, identifying solutions, or establishing time frames. The long-standing environmental and health problems associated with transboundary stormwater flows in the watersheds of both rivers continue. USIBWC officials’ statement that it lacks the authority to resolve the problems suggests that congressional direction may be needed to specifically authorize USIBWC to take action. This action could include identifying alternatives, cost estimates, funding, and time frames. Such action would help address the environmental and health problems associated with transboundary stormwater flows in the Santa Cruz River Basin and the Tijuana River Valley watersheds. To help address some of the infrastructure problems in Mexico that cause the transboundary flows—such as pipe breaks and pump failures— USIBWC has proposed the development of a binational rapid response team comprised of technical experts in both countries that would immediately respond to infrastructure problems. However, it has not taken the necessary steps to formalize the team within IBWC. By formalizing the binational rapid response team to address sewage infrastructure failures along the U.S.-Mexico border, USIBWC would have better assurance that it is able to more effectively address the urgent and recurring sewer breaks and pump failures in Mexico that contribute to raw sewage spills. In addition, USIBWC has not fully incorporated key capital planning principles that would help identify alternative approaches for the agency to address stormwater problems in the Santa Cruz River Basin or Tijuana River Valley watersheds. By conducting long-term capital planning in the Santa Cruz River Basin and Tijuana River Valley watersheds, following the principles in OMB Circular A-11, USIBWC would have better information to address the water quality problems resulting from unmanaged stormwater in either the Santa Cruz River Basin or Tijuana River Valley watersheds. USIBWC would also have capital planning information available to provide to State, OMB, and Congress, as part of the budget process, as directed in the 2019 OMB memorandum. Congress should consider providing direction and specific authorization for USIBWC to take action to resolve the long-standing water quality problems associated with transboundary stormwater flows in the Santa Cruz River Basin watershed, including identifying alternatives, cost estimates, funding sources, and time frames, in coordination with federal, state, and local partners. (Matter for Consideration 1) Congress should consider providing direction and specific authorization for USIBWC to take action to resolve the long-standing water quality problems associated with transboundary stormwater flows in the Tijuana River Valley watershed, including identifying alternatives to include cost estimates, funding sources, and time frames, in coordination with federal, state, and local partners. (Matter for Consideration 2) We are making the following two recommendations to the U.S. Commissioner of the IBWC. The U.S. Commissioner of the IBWC should work with the Mexican Commissioner to formalize a binational rapid response team to address sewage infrastructure failures along the U.S.-Mexico border, including the Nogales and South Bay wastewater treatment plants. (Recommendation 1) The U.S. Commissioner of the IBWC should direct USIBWC staff to conduct long-term capital planning for the Santa Cruz River Basin and Tijuana River Valley watersheds, following the principles in OMB Circular A-11. (Recommendation 2) We provided a draft of this report to USIBWC, EPA, and the Departments of State and Homeland Security for comment. USIBWC provided written comments, which are reproduced in appendix V. The other three agencies did not provide written comments on our draft report; however, they provided technical comments that we incorporated as appropriate. In its written comments, USIBWC concurred with our first recommendation that it work to formalize a binational rapid response team to address sewage infrastructure failures along the U.S.-Mexico border. The agency noted that it has held extensive consultations with the Mexican Section of the IBWC, and once there is agreement on the designated responsibilities and funding of the team, USIBWC will seek to formalize the arrangement through a written agreement or exchange of letters between the U.S. and Mexican Sections, approaches we outlined in the report. USIBWC also noted that the United States and Mexico have not agreed upon each country’s share of expenses and that the U.S. financial contribution is subject to legislative approval and contributions, including in-kind contributions, from domestic nonfederal entities. USIBWC partly concurred with our second recommendation that the U.S. Commissioner of the IBWC direct staff to conduct long-term capital planning for the Santa Cruz River Basin and Tijuana River Valley watersheds. The agency noted that it provided us the long-term capital planning information previously required by the Office of Management and Budget and that the practice had been useful. However, the agency also noted that to the extent our report envisions USIBWC undertaking long-term capital planning for (1) nonfederal infrastructure; (2) infrastructure that does not yet exist; and/or (3) infrastructure that the USIBWC is not yet authorized to construct or maintain, it does not concur. USIBWC stated that Congress may not view it as the lead agency, and therefore Congress does not need to provide it with the authorization to oversee cross-border pollution matters. Regardless of whether Congress considers USIBWC as the lead agency in resolving transboundary water quality, the agency is a key player in managing water quality on the border and has the infrastructure and organization that will be part of the solution. To date, the agency has been more reactive than proactive in participating in planning efforts and studies to resolve water quality problems and has told us that it does not have the authority to do so. Yet, without the information that USIBWC would generate by comprehensively assessing its long-term needs, such as through long-term capital planning efforts, Congress cannot authorize specific work that needs to be done. We recommended that the agency conduct long-term planning, including for infrastructure that does not exist and for infrastructure that is not yet authorized specifically to address this problem. We continue to believe that USIBWC should recognize its role along the border and, as we recommended, start planning for it, including by undertaking long-term capital planning for existing and potential future infrastructure and identifying alternatives to address the long-standing water quality problems. The agency also commented on our two Matters for Congressional Consideration in which we said that Congress should consider providing direction and specific authorization for USIBWC to take action to resolve long-standing water quality problems associated with transboundary stormwater flows in the watersheds. In its comments, USIBWC stated that it partly concurred with the Matters. USIBWC also stated that the phrasing of the Matters suggests that Congress should assign USIBWC specific duties and responsibilities, including identifying time frames for a comprehensive solution of pollution problems associated with transboundary stormwater flows and binational watershed management. This is correct. In our report, we highlighted the role USIBWC plays along the border and the infrastructure USIBWC manages and operates to address transboundary flows from Mexico. Given the location of the USIBWC’s wastewater treatment plants, along with its expertise and role working with Mexico, the agency would need to be centrally involved in any transboundary solution. However, it is incorrect, as USIBWC’s letter further stated, that our Matters imply that USIBWC would have the lead role in resolving water quality problems along the border. USIBWC’s letter stated that while the Matters acknowledge that USIBWC might coordinate with a wide range of partners, the language implies that Congress would designate USIBWC as the lead agency. Further, the agency stated that such a designation may run counter to past and current congressional intent and reasoning, as evidenced in very recent developments. In our matters, we stated that Congress should authorize USIBWC to take action to resolve water quality problems because it is a central actor in managing water and water quality along the border and because, during the course of our review, USIBWC stated that it needed specific congressional authorization to manage stormwater problems and to construct and maintain new infrastructure. We included the need for USIBWC to coordinate its action with other agencies because USIBWC would not be the sole lead actor. We note that USIBWC did not state what its role would be. Moreover, USIBWC stated that Congress may be in the process of designating EPA as the lead agency in developing major new infrastructure in the Tijuana Valley watershed to mitigate problems resulting from transboundary flows from Mexico. USIBWC also cited a recent bill to show that Congress is considering, consistent with proposals from California stakeholders, an appropriation request for as much as $300 million for the EPA to build this infrastructure. The agency stated that the bill lists USIBWC as one of 11 eligible public entities with which EPA may coordinate its efforts, as opposed to identifying the USIBWC as the lead agency. It also stated that the United States-Mexico-Canada Agreement Implementation Act accompanying this bill explains that EPA’s designation as the lead agency was premised on Congress’s determination that EPA has the expertise and experience necessary to lead and coordinate efforts involving wastewater, stormwater, nonpoint sources of pollution, and related matters in the Tijuana watershed. At a minimum, USIBWC will be a key partner with EPA if it is given the authority to help resolve stormwater quality problems in the Tijuana River watershed. Yet, as discussed in our report, USIBWC stated it needs congressional authorization to participate in addressing stormwater issues along the border. We note that the bill to which USIBWC refers does not specifically address USIBWC’s authority to develop and implement stormwater projects near the border. Our report shows that this authorization is necessary for the agency to take action, whether as a lead agency or as an eligible partner that may coordinate with others. We added a discussion of the bill in our report, as well as about the expertise that EPA and USIBWC have to address transboundary flow problems. Specifically, we described that according to EPA officials, EPA lacks the expertise to construct and maintain water infrastructure projects on its own. Further, EPA officials stated that EPA will need to carry out any work in the area through contracts with other agencies, as EPA does not have expertise in operating and maintaining water infrastructure, as USIBWC does. EPA also noted that USIBWC is one of the only federal agencies that works across the border because it has consistent communication and contacts in Mexico. Finally, USIBWC stated in its comments that the reasoning for designating EPA in the bill and the accompanying act as the lead agency for pollution reduction for the Tijuana River watershed—because of EPA’s unique qualifications—also applies in any border area, including the Santa Cruz watershed in Arizona. Again, our report showed that USIBWC is a central actor in managing water and water quality on the border and that congressional authorization is needed for USIBWC to help address transboundary stormwater flows, including identifying alternatives for solutions, in the Santa Cruz watershed. We did not change our Matters, but added a discussion in our report of the proposed congressional legislation to address the water quality problems in Tijuana specifically and the expertise that EPA and USIBWC each bring to addressing transboundary flow problems. We are sending copies of this report to appropriate congressional committees; the Commissioner of the U.S. Section of the International Boundary and Water Commission; the Secretaries of Homeland Security and State; the Administrator of the EPA; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI. This report (1) describes the authorities and roles involved in developing, managing, and sharing costs for the International Boundary and Water Commission’s (IBWC) two international wastewater plants in the United States; (2) examines factors that affect the operation of the two plants and steps IBWC has taken to address these; and (3) examines the extent to which the U.S. Section of the International Boundary and Water Commission (USIBWC) has taken steps to address water quality problems in the two watersheds, including through the use of key capital planning principles. To address these three objectives, we visited the Nogales International Wastewater Treatment Plant (Nogales plant) in Arizona and the South Bay International Wastewater Treatment Plant (South Bay plant) in California. At each facility, we interviewed USIBWC officials and toured each wastewater treatment plant and its associated infrastructure. We also met with other federal, state, and local government officials and representatives of non-governmental organizations to discuss USIBWC’s management and operations of the plant. Specifically, in Arizona we met with officials from the Department of Homeland Security’s Custom and Border Protection (CBP), the City of Nogales, the Arizona Department of Environmental Quality, the County of Santa Cruz, and the nonprofit Friends of the Santa Cruz River. In California, we met with officials from CBP; Environment Protection Agency Region 9; the California Water Quality Regional Control Board; the City of San Diego; the County of San Diego; the California State Parks; the City of Imperial Beach; and Surfrider Foundation San Diego Chapter, Wildcoast, and 4Walls International (all nongovernmental organizations). We visited USIBWC Headquarters in El Paso, Texas, to meet with agency officials, including the U.S. Commissioner and budget, engineering, and general counsel staff. We also met with the Mexican Commissioner of the IBWC in Ciudad Juarez, Chihuahua, Mexico. To describe authorities and roles involved in developing, managing, and sharing the costs of USIBWC’s two international wastewater plants in the United States, we reviewed the 1944 treaty between the United States and Mexico, Treaty Relating to the Utilization of Waters of the Colorado and Tijuana Rivers and of the Rio Grande, and associated IBWC minutes. For cost-sharing of operational and maintenance expenses at the plants, we reviewed minutes between USIBWC and the Mexican Section and a memorandum of agreement between USIBWC and the City of Nogales, Arizona. We reviewed USIBWC’s budget for fiscal years 2003 through 2019, including appropriated funding information for fiscal years 2003 through 2019. We also met with budget officials at USIBWC and the Department of State. To determine if these data are reliable, we interviewed a USIBWC official about the source of the data and reviewed documentation to determine that the data were sufficiently reliable for the purposes of discussing USIBWC budget and project costs. To examine factors, if any, that affect the operation of the two plants and steps IBWC has taken to address these factors, we reviewed each plant’s permit from the National Pollutant Discharge Elimination System, violation notices, and USIBWC documentation, such as plans for projects to resolve the violations. We interviewed USIBWC officials about their plans and projects to resolve any water quality problems at the plants. We also interviewed Arizona and California state environmental officials responsible for developing and enforcing the permits, to discuss permit violations and water quality problems at the plants and actions to resolve them. To examine the extent to which USIBWC has taken steps to address water quality problems in the two watersheds, including using key capital planning principles, we reviewed and analyzed IBWC minutes, USIBWC’s annual financial reports for fiscal years 2015 through 2019, USIBWC’s most recent strategic plan covering fiscal years 2011 through 2016, the South Bay plant’s 5-year and Nogales plant’s 10-year equipment investment plans, and documentation from USIBWC’s citizen forums in each location. In addition, we reviewed prior GAO reports on federal agency capital planning and asset management, the Office of Management and Budget’s (OMB) Capital Programming Guide (Version 3.0) Supplement to OMB Circular No. A-11, and OMB’s 2019 guidance on implementing agency-wide real property capital planning. We compared USIBWC’s capital planning efforts against OMB’s Capital Programming Guide and past GAO reports on capital planning leading practices. Further, we interviewed USIBWC officials and stakeholders at each plant, including local government officials and environmental group representatives, about the water quality problems and solutions they have discussed. We also reviewed studies conducted in the two watersheds. We conducted this performance audit from September 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix provides information about some of the factors that can affect the operations of the Nogales and South Bay plants. For the Nogales plant, the factors that can affect the operations include: Lack of heavy metal pretreatment. Numerous metal treatment and plating facilities operate in Nogales, Sonora, Mexico. These facilities directly discharge their wastewater, which contains heavy metals such as chromium, zinc, and nickel, into the city’s sewer system. The heavy metals are comingled with the other sewage and sent to the Nogales plant (in Arizona) for treatment. In the United States, similar types of facilities would be required to pretreat the wastewater to remove metals and other pollutants before discharging it into the public sewer system. In the United States, the mechanism used to limit industrial discharges into a sewer system is a pretreatment program that can ultimately cause dischargers to be shut off from the system or fined if they do not limit the industrial contaminants in their discharges to the system. While a municipal pretreatment program exists in Nogales, Sonora, it is designed to meet Mexico’s minimum federal requirements and is insufficient to detect and respond to the dumping of industrial contaminants when they occur, according to the Arizona Department of Environmental Quality documentation and officials. According to U.S. Section of the International Boundary and Water Commission (USIBWC) officials, the Nogales plant is not designed to separate out heavy metals during its treatment processes, and as a result the heavy metals contaminate the plant’s sludge. Furthermore, due to the presence of heavy metals, USIBWC disposes of the sludge at a municipal landfill, a process that is more expensive than other disposal options, which has led to increased operational costs for the plant. According to USIBWC officials, it would cost about $60 million to update the Nogales plant to a tertiary treatment system that could remove the heavy metals from the sludge. Deteriorating sewage infrastructure in Mexico. Sewage infrastructure in the City of Nogales, Sonora, is not adequately maintained, according to USIBWC officials. As a result, USIBWC officials told us that the amount of wastewater Nogales, Sonora, sends exceeds the amount agreed upon in a minute between the two sections. Although Nogales, Sonora, built a new plant—the Los Alisos Plant—that can treat 5.5 million gallons per day, wastewater has to be pumped uphill from Nogales, Sonora, into the plant. After the first year of operation, the Mexican government could not maintain the plant due to funding constraints, according to USIBWC officials. The pumps responsible for delivering the wastewater uphill to the Los Alisos plant continually break or fail. For example, as of July 2019, only one of the five pumps at the Los Alisos plant was operational, according to USIBWC officials. When these pumps fail, Mexico releases the 2 million to 4 million gallons per day of wastewater—which normally would have been intercepted and sent to the Los Alisos plant— through the International Outfall Interceptor (IOI) to the Nogales plant. Deteriorating infrastructure in the United States. The deteriorating condition of the IOI has caused untreated sewage to periodically spill into the Santa Cruz watershed and Nogales Wash. The IOI is over 45 years old, and according to USIBWC officials, the typical lifespan of a similar pipeline is 50 years. Maintenance has been deferred because of continuing disagreement between USIBWC and the City of Nogales, Arizona, regarding which entity owns the pipeline and is therefore responsible for its maintenance, according to USIBWC officials. The IOI’s condition continues to worsen and requires a significant amount of rehabilitation to address structural damage. Erosion and corrosion are continuously occurring, according to a 2005 assessment of the IOI prepared for the City of Nogales, Arizona. Specifically, gases released by the sewage corrode the pipeline, and root intrusion and groundwater cause erosion. According to the 2005 assessment, half of the thickness of the pipe had been eroded and corroded For the South Bay plant, the factor that may affect the operations is: Insufficient sewage infrastructure in Mexico. According to the 2019 study of alternatives to expand or adapt diversion infrastructure, Tijuana has not built sufficient sewage infrastructure to serve the area’s increasing population and urbanization, contributing to transboundary sewage flows. According to USIBWC officials, the city of Tijuana does not prioritize wastewater issues and is experiencing exponential population growth and urbanization. As a result, areas of Tijuana are not connected to the city’s sewer system. A 2017 study prepared by a Mexican state agency estimated that over $340 million would be required to fix and develop adequate wastewater treatment and reuse systems for the city of Tijuana. When there are problems with Tijuana’s treatment facilities, Tijuana diverts a portion of its wastewater to be treated at the South Bay plant. If the South Bay plant is not notified and does not shut down the pump and canyon collectors, it may receive additional flows. While treating the excess wastewater does not violate the plant’s National Pollutant Discharge Elimination System permit, the plant is experiencing an increase in the number of days that it treats flows above capacity, according to USIBWC officials. This could eventually cause violations to occur as the plant is not supposed to operate above capacity for prolonged periods. In addition, USIBWC officials stated that the South Bay plant is not designed and operated to address some of the wastewater that flows into the Tijuana River Valley watershed. These wastewater flows are due to: Limited Tijuana Basin diversion infrastructure. The Tijuana Basin diversion system is comprised of Mexican-operated Pump Station CILA and the South Bay plant’s five canyon collectors. This system captures dry-weather flows for treatment at the South Bay plant or a wastewater treatment plant in Mexico. However, it is not designed to capture high flows that result from pipe breaks or pump failures. Specifically, the system has a peak capacity of 29 million gallons per day, while Pump Station CILA can only operate at 23 million gallons per day. To avoid affecting the South Bay plant’s wastewater treatment operations, during incidents of high flows, Pump Station CILA and the canyon collectors are shut off. During these events, the water bypasses the South Bay wastewater treatment plant and flows untreated into the Tijuana River and watershed. The Senate committee report accompanying the Department of State, Foreign Operations, and Related Programs Appropriations Act, 2019 required the USIBWC to submit a report quantifying the total annual volume of transboundary flows entering the United States from Mexico in the Tijuana River watershed. USIBWC issued this report in August 2019. Lack of maintenance for existing sewage infrastructure in Mexico. A lack of maintenance for Tijuana’s existing sewage infrastructure causes excess wastewater flows into the Tijuana River according to USIBWC officials. For example, in August 2019, USIBWC reported that on June 19, 2019, nearly 1.9 million gallons of wastewater were released into the Tijuana River because trash buildup at one of Tijuana’s pumps caused the pump to fail. In the last 2 decades, according to a 2019 study, the local Mexican utility that operates and manages the city’s sewage infrastructure has invested in expanding the city’s wastewater collection infrastructure to address direct dischargers or inadequate disposal practices, according to USIBWC officials. However, the overall system has not kept pace with the region’s rapid growth, nor has the existing infrastructure in Mexico received sufficient maintenance. In addition, the local utility that manages and operates Tijuana’s wastewater system has a limited number of personnel. The study also reported that existing personnel were “very knowledgeable, dedicated, and creative in their efforts” to maintain and operate the sewage infrastructure. Nonetheless, Tijuana’s existing sewage pipes consistently break and its pump stations fail. Another 2019 study also reported that the poor condition of critical wastewater infrastructure in Mexico results in approximately 30 percent of Tijuana’s wastewater enters the Tijuana River or Pacific Ocean without treatment. In addition to the contact named above, Susan Iott (Assistant Director), Heather Dowey (Analyst-in-Charge), Farah Angersola, Mark Braza, Chuck Bausell, Tara Congdon, Carol Henn, Richard P. Johnson, Anika McMillon, Sara Sullivan, and Kiki Theodoropoulos made key contributions to this report.", "summary": "Ongoing sewage spills and stormwater runoff carrying trash, sediment, and other pollutants in the Santa Cruz River Basin and Tijuana River Valley watersheds along the U.S.-Mexico border have affected public health, the environment, and local economies. Under the 1944 treaty, the United States and Mexico agreed to work together through IBWC to address these water quality problems. As part of this effort, USIBWC manages two wastewater treatment plants in Arizona and California. In 2018, the plants treated more than 14 billion gallons of sewage from Mexico. This report (1) describes the authorities and roles for developing and managing the plants and sharing their costs; (2) examines factors affecting the operation of each plant and steps taken to address them; and (3) examines the extent to which USIBWC has taken actions to address water quality problems in the watersheds. GAO reviewed U.S-Mexico treaties, IBWC minutes and permits, and planning and budget data for USIBWC. GAO also interviewed officials from IBWC and other federal agencies, local and state governments, and non-governmental groups. A 1944 treaty designated the International Boundary and Water Commission (IBWC) and authorized it to resolve water and boundary issues along the U.S.-Mexico border, including providing wastewater treatment. IBWC's two sections—the U.S. Section (USIBWC) and the Mexican Section, negotiated agreements to construct, manage, and operate two wastewater plants in Nogales, Arizona, and San Ysidro (South Bay), California, to resolve ongoing water quality problems stemming from sewage flowing downhill from Mexico into the United States (see figure). Several of these agreements describe each country's roles, such as sharing costs for the operation and maintenance of each plant. Several factors can affect the plants' operations, including deteriorating infrastructure in Mexico and the United States that results in raw sewage spills around the plants. USIBWC has taken steps to resolve some of these factors. For example, USIBWC proposed a binational rapid response team to address broken pipes and failing pumps that can send sewage from Mexico into the United States; however, the team has not been formalized to ensure its long-term operation. By taking steps to formalize the team, USIBWC would have assurance it can more effectively address recurring infrastructure failures contributing to sewage spills. USIBWC and others have taken some actions to address stormwater problems, such as studying stormwater flows in the Tijuana River Valley watershed and building some retention basins. However, USIBWC has not taken action, in coordination with federal, state, and local partners, to identify alternatives, cost estimates, funding sources, and time frames for implementing solutions in either watershed. USIBWC officials said without direction from Congress, it does not have specific authorization for stormwater management in the watersheds because the 1944 treaty and accompanying legislation did not authorize it to carry out such projects. The long-standing stormwater quality problems and their associated environmental and health effects suggest congressional direction is needed to authorize USIBWC to take action. Such action would include identifying alternatives, cost estimates, funding sources, and time frames. GAO believes that Congress should consider providing direction and specific authorization to USIBWC to take action to resolve stormwater quality problems in the Santa Cruz River Basin and Tijuana River Valley watersheds. GAO is also making two recommendations to USIBWC, including that it formalize the rapid response team. USIBWC concurred with that recommendation and partly concurred with the other.", "document_type": "gao"}
{"report": "A reverse mortgage is a nonrecourse loan against home equity that does not require mortgage payments as long as the borrower meets certain conditions. In contrast to traditional forward mortgages, reverse mortgages typically are “rising debt, falling equity” loans (see fig. 1). As the borrower receives payments from the lender, the lender adds the principal and interest to the loan balance, reducing the borrower’s home equity. Also unlike traditional forward mortgages, reverse mortgages have no fixed term. Prospective borrowers must meet a number of requirements to be eligible for a HECM (see sidebar). The amount of money a borrower can receive from a HECM—called the principal limit—depends on three things: (1) the age of the youngest borrower or eligible nonborrowing spouse, (2) the lesser of the appraised value of the home or the FHA mortgage limit as of the date of loan closing (for calendar year 2019, $726,525), and (3) the expected average interest rate. The borrower can receive funds in a variety of ways—for example, as monthly payments, a line of credit, a combination of the two, or a single lump sum. A large majority of borrowers choose the line of credit option. The interest rate lenders charge is typically an adjustable rate, although the lump sum option can be chosen at a fixed interest rate. session given by a HECM counselor approved by the Department of Housing and Urban Development (HUD) make timely payment of ongoing property charges (e.g., taxes and insurance) HECMs that are due and payable because the borrower has not paid property charges, met occupancy requirements, or maintained the home. HECM borrowers (or their heirs) satisfy the debt by (1) paying the loan balance using their own funds, (2) selling the home and using the proceeds to pay off the loan balance, (3) providing a deed-in-lieu of foreclosure (which transfers title for the property to the lender to satisfy the debt), or (4) selling the home for at least the lesser of the loan balance or 95 percent of the property’s appraised value (also known as a short sale). According to FHA regulations, the borrowers or their heirs generally have 30 days after being notified that the loan is due and payable to satisfy the debt or bring the loan out of due and payable status. Servicers generally have 6 months to take first legal action to initiate foreclosure from the date that they, as applicable, notified, should have notified, or received approval from FHA that the HECM is due and payable. According to FHA regulations, the borrower is generally allowed to correct the condition that resulted in the due and payable loan status and reinstate the loan, even after foreclosure proceedings have begun. Figure 2 illustrates the reasons why HECMs terminate and how borrowers typically satisfy the debt under various termination scenarios. If the servicer experiences a loss because the loan balance exceeds the recovery from selling the property, the lender can file a claim with FHA for the difference. Additionally, when the loan balance reaches 98 percent of the maximum claim amount (the lesser of the appraised value of the home at origination or FHA’s loan limit), the lender can “assign” the loan to FHA and file a claim for the full amount of the loan balance, up to the maximum claim amount. Lenders can only assign HECMs in good standing to FHA (that is, assignments can only be for HECMs not in a due and payable status). FHA continues to service the assigned loans using a contractor until the loans become due, either due to the death of the borrower or for other reasons. Additionally, the FHA insurance guarantees borrowers will be able to access their loan funds, even if the loan balance exceeds the current value of the home or if the lender experiences financial difficulty. Further, if the borrower or heir sells the home to repay the loan, he or she will not be responsible for any loan amount above the value of the home. As of the end of fiscal year 2018, FHA had insured over 1 million HECMs. According to FHA data, these include an active HECM portfolio of approximately 551,000 loans serviced by various FHA-approved servicers, 79,000 FHA-assigned loans serviced by an FHA contractor, and about 468,000 terminated loans (see fig. 3). HECM terminations have exceeded new originations every year since fiscal year 2016, and the number of HECMs assigned to FHA has grown substantially since fiscal year 2014. As of the end of fiscal year 2018, FHA’s total insurance-in-force for HECMs (total insured mortgage balances outstanding) was roughly $100 billion. HECMs are held in two FHA insurance funds. HECMs originated prior to fiscal year 2009 are in the General Insurance and Special Risk Insurance Fund (roughly 27 percent of all HECMs), and those originated in fiscal year 2009 and later are in the Mutual Mortgage Insurance Fund (roughly 73 percent of all HECMs). When the post-2008 HECM portfolio became part of FHA’s Mutual Mortgage Insurance Fund, it also was included in the fund’s capital ratio assessment and became subject to annual actuarial reviews. As we found in a November 2017 report, subjecting HECMs to the annual actuarial review requirements has improved the transparency of the program’s financial condition and has highlighted the financial risks of the HECM portfolio to FHA. According to FHA, the financial performance of the HECM portfolio has been historically volatile, largely due to uncertainty in future home prices, interest rates, and other factors. In recent years, FHA has responded with several policy changes to help strengthen the portfolio’s financial performance and mitigate risks. Because FHA’s projected losses on HECMs depend on factors such as maximum claim amount, the length of time the borrower stays in the home, changes in home prices, and interest rates, most of FHA’s policy changes have been aimed at better aligning expected revenues (charging borrowers premiums) with expected costs (cash outflows due to paying insurance claims). For example, FHA has made changes to insurance premiums and principal limits, the most recent of which took effect in fiscal year 2018. Effective in fiscal year 2019, FHA also revised property appraisal practices for new HECMs to guard against inflated property valuations. According to agency officials, FHA made this change to address appraisal bias concerns identified in research by an economist in HUD’s Office of Policy Development and Research. The HECM market includes various participants. After a lender originates a HECM, the loan must be serviced until it terminates. HECM lenders and servicers must be FHA-approved and can be the same entity but often are not. HECM lenders often sell the mortgage to another entity, which FHA refers to as an investor, and this entity has the right to enforce the mortgage agreement. HECM servicers are typically third parties that contract with lenders or investors but do not have ownership in the loans they service. As previously discussed, HECM servicers perform a number of functions, such as making payments to the borrowers and providing monthly account statements. Servicers also must monitor borrower compliance with various mortgage conditions and, if necessary, communicate with borrowers about any violation of these conditions (defaults) and, as appropriate, ways they can avoid being foreclosed on. HECM servicers also transfer up-front and annual insurance premiums to FHA each month and file claims with FHA for losses on insured HECMs. In carrying out these duties, servicers are responsible for complying with various requirements, including FHA regulations, policies, and procedures, as well as federal consumer financial laws. Historically, commercial banks, thrifts, and credit unions were the primary lenders and servicers of mortgage loans. Following the 2007–2009 financial crisis and subsequent revisions to regulatory bank capital requirements, banks reevaluated the benefits and costs of being in the mortgage lending market, as well as retaining mortgages and the right to service them. Since the financial crisis, some banks have exited or reduced their mortgage lending and servicing businesses. This development, among others, created an opportunity for nonbank servicers to increase their presence in the mortgage market. Nonbank issuers such as mortgage originators and servicers are not subject to the same comprehensive federal safety and soundness standards as banks. While banks offer a variety of financial products to consumers, nonbank servicers are generally involved only in mortgage-related activities and do not take deposits from consumers. Almost all HECMs are originated, owned, and serviced by nonbank entities: Lenders. According to FHA, in fiscal year 2018, 54 lenders originated HECMs, including 49 nonbank entities and five banks. Investors. As of the end of fiscal year 2018, six investors (all nonbank entities) and the Federal National Mortgage Association (Fannie Mae) owned roughly 92 percent of the privately owned (non-FHA-assigned) HECM portfolio, while the remaining 8 percent was owned by a mixture of bank and nonbank entities. Servicers. Five nonbank entities serviced over 99 percent of the privately owned HECM portfolio as of the end of fiscal year 2018. As previously noted, FHA has a contractor (also a nonbank entity) that services FHA-assigned HECMs. A number of federal agencies have roles in overseeing the reverse mortgage market, including the following: FHA. Insures HECMs and administers the HECM program, including issuing program regulations and enforcing program requirements. FHA supplements regulations through additional policies, procedures, and other written communications for the HECM program. For example, FHA officials said the agency utilizes its Single Family Housing Handbook, HECM handbook, and mortgagee letters to communicate changes about the HECM program. In 2013, Congress enacted a law that allowed FHA to make changes to HECM program requirements by notice or mortgagee letter in addition to regulation. Since then, FHA has made several policy changes to the HECM program through mortgagee letters. CFPB. Supervises nonbank reverse mortgage lenders and servicers for compliance with, and enforces violations of, federal consumer financial protection laws. CFPB can also issue regulations under the federal consumer protection laws addressed specifically to protecting consumers considering reverse mortgages. Additionally, CFPB examines entities for compliance with federal consumer financial laws to obtain information about an institution’s compliance management systems and procedures and to detect and assess risks to consumers and markets. Further, CFPB collects consumer complaints regarding consumer financial products or services (including reverse mortgages) and educates consumers about their rights under federal consumer financial protection laws. Federal depository institution regulators. These regulators monitor compliance with relevant laws and regulations, such as provisions of the Federal Trade Commission Act and the Truth in Lending Act, primarily through periodic examinations, for federally regulated lenders that originate HECMs. Several features and requirements of the HECM program provide consumer protections to borrowers. For example, borrowers must undergo preloan counseling, the program limits costs and fees lenders can charge, and lenders must provide certain disclosures. In addition, FHA has made several changes to the HECM program in recent years to help borrowers who have defaulted due to unpaid property charges. As previously discussed, if a HECM borrower does not pay his or her property charges, FHA regulations generally require the servicer to pay the property charges on the borrower’s behalf to help avoid a tax foreclosure by the local authority and protect the investor’s and FHA’s interest in the home. FHA regulations also allow servicers to charge certain fees once a loan is called due and payable. These are typically amounts related to attorney or trustee fees, property preservation, and appraisal fees during the foreclosure process. The payments and fees that servicers make on behalf of borrowers—referred to as servicer advances in this report—are added to the loan balance and accrue interest. In 2010, HUD’s Office of Inspector General reported that HUD was not tracking borrower defaults or servicer advances for the HECM program and made several recommendations to FHA. To address these recommendations, FHA took several steps. For example, in 2011, FHA stopped the practice of allowing servicers to defer foreclosing on loans that were in default due to unpaid property changes and issued a mortgagee letter addressing how to handle these loans. Additionally, in September 2012, FHA announced the launch of a new data system for the HECM program, the Home Equity Reverse Mortgage Information Technology (HERMIT) system which would be used starting in October 2012. With this new system, FHA combined former legacy systems that had been used to collect insurance premiums, service FHA-assigned loans, and process claims. According to FHA, adopting the HERMIT system allowed FHA to better monitor and track the HECM portfolio in real time and to automate insurance claim processing. Finally, FHA modified program features to help minimize potential borrower defaults and help strengthen borrower eligibility requirements. For example, in 2013, FHA reduced the amount of equity borrowers could generally withdraw during the first year from 100 to 60 percent of the principal limit. According to FHA, this change was designed to encourage borrowers to access their equity slowly over time rather than all at once to reduce risks to borrowers and FHA’s insurance fund. In 2015, the financial requirements for HECMs changed to include a financial assessment of the prospective borrower prior to loan approval. FHA began requiring HECM lenders to look at the prospective borrower’s credit history, income, assets, and financial obligations. Based on the results of the financial assessment, the lender may require a set-aside for the payment of property charges. Additionally, FHA made several program changes to help distressed HECM borrowers by allowing servicers to offer options to help borrowers delay or in some cases avoid foreclosure if they are behind on paying property charges. These foreclosure prevention options include repayment plans, at-risk extensions, and extensions for low-balance arrearage, as described later in this report. FHA also has taken steps to help nonborrowing spouses stay in their homes after a borrowing spouse dies by deferring repayment of the HECM as long as the nonborrowing spouse fulfills certain conditions. In these cases, the servicer can assign the HECM to FHA under what FHA refers to as the mortgagee optional election assignment process. Our analysis of FHA data found that about 272,155 HECMs terminated from fiscal years 2014 through 2018. Over that period, the number of terminations rose from about 24,000 in fiscal year 2014 to a peak of roughly 82,000 in fiscal year 2016, before declining to about 60,000 in fiscal year 2018, as previously shown in figure 3. As shown in figure 4, death of the borrower was the most common reported reason for HECM terminations, followed by borrower defaults. The relative size of each termination category varied from fiscal years 2014 through 2018, with borrower defaults accounting for an increasing proportion of terminations in recent years. In fiscal year 2018, borrower defaults made up 18 percent of terminations. Specific results for all major termination categories over the 5-year period were as follows:  Death. About 34 percent of terminations (approximately 87,000 loans) were due to the death of the borrower. Borrower deaths ranged from roughly 29 percent to 40 percent of annual terminations over the 5- year period.  Default. About 15 percent of terminations (approximately 40,000 loans) were due to borrower defaults. As discussed in appendix IV, this percentage varied widely by location and was highest in Michigan (36 percent) and lowest in the District of Columbia (1 percent). About 29,000 defaults were for noncompliance with occupancy or residency requirements, about 11,000 were for nonpayment of property charges, and about 200 were for not keeping the property in good repair. The borrowers of these loans likely lost their homes through foreclosure or a deed-in-lieu of foreclosure. However, it is possible that some of these borrowers would have ultimately lost their homes even if they had not taken out a HECM. For example, as noted in CFPB’s 2012 report to Congress on reverse mortgages, some borrowers may have taken out a HECM to help pay off their traditional mortgage rather than as a way to pay for everyday expenses. In these cases, the money borrowers received from their HECMs may have helped them temporarily but may ultimately have been prolonging an unsustainable financial situation. In addition, some borrowers who did not meet occupancy or residency requirements may have permanently moved out of their homes—for example, to assisted living or nursing home facilities. Borrower defaults as a percentage of annual HECM terminations grew from 2 percent of terminations in fiscal year 2014 to 18 percent in fiscal year 2018. Noncompliance with occupancy requirements was the primary cause of defaults each year, but unpaid property charges represented a growing share. From fiscal years 2014 through 2018, property charge defaults as a percentage of all defaults grew from 26 percent to 45 percent, and property charge defaults as a percentage of all terminations grew from less than 1 percent to 8 percent. Loan balance repaid. About 9 percent of terminations (approximately 23,000 loans) were due to the borrower repaying the loan balance. This category accounted for a declining share of terminations over the 5-year period, falling from 24 percent in fiscal year 2014 to 4 percent in 2018. Refinanced. About 8 percent of terminations (about 20,000 loans) were due to the borrower refinancing into a new HECM. This category remained relatively stable over the 5-year period, accounting for about 5 percent to 10 percent of terminations each year. Borrower moved or conveyed title. About 3 percent of terminations (approximately 8,000 loans) were due to the borrower either moving or conveying title to the property to someone else. The percentage of terminations in this category declined from 6 percent in fiscal year 2014 to 2 percent in fiscal year 2018. Unknown. For about 30 percent of terminations (roughly 78,000 loans), we were unable to readily determine a termination reason from FHA’s data. Over the 5-year period, this category accounted for over 25 percent of terminations each year and reached a high of 39 percent in fiscal year 2018. We discuss challenges related to determining termination reasons later in this report. For HECMs that terminated in fiscal years 2014 through 2018, servicers advanced almost $3 billion on behalf of borrowers for unpaid property charges or various other costs that are charged once a loan becomes due and payable. The advances increased from $508 million in fiscal year 2014 to a peak of $731 million in fiscal year 2016, before declining to $453 million in fiscal year 2018 (see fig. 5). This pattern aligns with the overall trend in terminations, which also peaked in fiscal year 2016. Over the 5-year period, advances for property charges made up 58 percent of the total. The remaining 42 percent consisted of advances for other costs, many of them foreclosure-related, such as attorney fees and appraisal costs. From fiscal years 2014 through 2018, HECM servicers advanced a total of $567 million on behalf of living borrowers who defaulted on their HECMs due to unpaid property charges. For these loans, the median advance was $7,007. From April 2015 (the effective date of FHA’s current repayment plan policy) through the end of fiscal year 2018, 22 percent of HECM borrowers with overdue property charges had received repayment plans, and FHA’s information on the use of other foreclosure prevention options was limited. As previously noted, property charge defaults and issues surrounding nonborrowing spouses not being included on the mortgage have been long-standing problems in the HECM program. Since 2015, FHA has made program changes to allow servicers to offer different types of foreclosure prevention options to distressed HECM borrowers and nonborrowing spouses of deceased borrowers (see table 1). These options can help delay and, in some cases, avoid foreclosure. According to officials from HUD’s Office of General Counsel, HUD does not have the statutory authority to require servicers to provide HECM borrowers foreclosure prevention options. Our analysis of FHA data found that servicers’ use of selected foreclosure prevention options for HECM borrowers was limited or that FHA did not have readily available information to assess the extent of use, as follows: Mortgagee optional election assignments. According to information generated by FHA, HECM servicers submitted 1,445 requests for mortgagee optional election assignments from June 2015 (when FHA made this option available) through September 30, 2018 (see table 2). In total, FHA approved roughly 70 percent (1,013) of the requests and denied the remaining 30 percent (432). According to FHA officials, the top two reasons for denying mortgagee optional election assignments were HECM servicers not meeting the deadline for electing to pursue the assignment and not meeting the deadline to initiate the assignment. FHA officials told us the third most common reason for denial was a nonborrowing spouse not submitting evidence of marketable title to the property or the legal right to remain in the property for life within required time frames. With respect to the 432 denials, FHA provided information indicating that as of May 31, 2019, 79 percent (342) of the associated loans had not terminated; 14 percent (62 loans) terminated because the loan balance had been paid off; and the remaining 7 percent ended in foreclosure (22 loans), deed-in-lieu of foreclosure (four loans), or short sale (two loans). Estimating the universe of HECMs potentially eligible for mortgagee optional election assignments is difficult because nonborrowing spouses were not listed on loan documentation for HECMs originated prior to August 4, 2014. As a result, FHA does not know how many eligible nonborrowing spouses could have, but did not, apply for the mortgagee optional election assignment, or how many are potentially eligible to apply for it in the future. FHA officials told us they have relied on an industry association and HECM servicers to estimate how many nonborrowing spouses may be associated with pre-August 2014 HECMs. FHA officials told us they sent letters to borrowers with FHA-assigned HECMs that were originated prior to August 4, 2014, to inform them of the mortgagee optional election process and ask them to self-identify whether there was a nonborrowing spouse associated with their loan. FHA officials also noted they were drafting a similar letter for servicers to send to borrowers with HECMs not assigned to FHA. As of August 2019, the servicer letter was still in draft form, pending completion of an ongoing internal review of FHA’s mortgagee optional election assignment processes and the related time frames. FHA officials said once the ongoing review is complete, they anticipated that FHA would issue a new mortgagee letter with revised time frames that would afford both HECM servicers and borrowers more time to meet FHA requirements for mortgagee optional election assignments. Repayment plans. Our analysis of FHA data showed that 22 percent of borrowers with property charge defaults were granted a repayment plan from April 2015 (the effective date of FHA’s current repayment plan policy) through the end of fiscal year 2018. All five legal aid organizations we interviewed said the availability of repayment plans was a top concern. For example, for some of their clients, repayment plans were unavailable because the borrowers did not meet certain financial requirements. In contrast, representatives of the top five HECM servicers told us they generally do offer repayment plans when feasible to help borrowers delay or avoid foreclosure. Servicers we interviewed noted that while repayment plans can delay or avoid foreclosure, they are rarely successful in the long-run and borrowers in such plans often miss payments. Servicers said the same reasons that typically contribute to initial defaults also explain why repayment plans are rarely successful. For example, borrowers on limited incomes may struggle to pay increasing property tax and insurance costs or may fall behind on property charges when the death of a spouse reduces their income. At-risk extensions. Our analysis of FHA data found that from April 2015 (the effective date of FHA’s at-risk extension policy) through the end of fiscal year 2018, about 2 percent of borrowers with property charge defaults received an at-risk extension. To grant an at-risk extension, FHA requires HECM servicers to provide valid documentation that the youngest living borrower is at least 80 years of age and has critical circumstances such as a terminal illness, long-term physical disability, or a unique occupancy need (for example, terminal illness of family member living in the home). Representatives from one legal aid organization told us that some HECM servicers have straightforward requirements for the documentation borrowers must submit to obtain an at-risk extension, while others do not. Representatives from another legal aid organization said that meeting FHA’s annual renewal requirement for at-risk extensions was challenging for some borrowers because they have to submit documentation to HECM servicers every year as they age and continue to struggle with serious health issues or disabilities. Low-balance extensions. FHA officials told us they do not track how often HECM servicers use the option to delay calling a loan due and payable if the borrower has unpaid property charges of less than $2,000. Our analysis of FHA data on servicer advances found that approximately 8,800 HECMs that terminated in fiscal years 2014 through 2018 had unpaid property charges of less than $2,000 at the time of termination. Some of these HECMs may have been eligible for a low-balance extension when they terminated. Representatives from one legal aid organization said they represented a HECM borrower who was at risk of foreclosure for having 27 cents in unpaid property charges. HECM servicers told us they use the low-balance extension option to varying degrees. For example, representatives from one servicer said the servicer follows instructions from the entity that owns the HECM and, in some cases, the owners of the loan do not want to offer the low-balance extension to the borrower. In these cases the servicer calls the loan due and payable for any amount in unpaid property charges and initiates the foreclosure process in accordance with FHA regulations. Another HECM servicer told us it tries to use the low-balance extension every time a borrower owes less than $2,000 in unpaid property charges. Since fiscal year 2013, FHA has used the HERMIT system to collect data on the servicing of HECMs, but the system does not contain comprehensive and accurate data about the reasons why HECMs terminate, a key servicing event. According to the HERMIT User Guide, servicers should provide a reason in HERMIT when they terminate a HECM. However, as noted previously in figure 4, for about 30 percent of HECM terminations from fiscal years 2014 through 2018 (roughly 78,000 loans), we were unable to determine the reason for termination. Specifically, for these loans we could not identify in HERMIT any associated borrower death or default, or evidence that the borrower repaid, refinanced, moved, or conveyed title. Instead, these loans were coded as terminating for “other reasons” or coded based on how the debt was satisfied rather than an actual termination reason. The HERMIT User Guide provides a list of termination codes available in the system, but the list and guide have shortcomings that limit analysis of HECM terminations. First, the list includes codes servicers can use to indicate that a loan terminated for “other reasons,” but the guide does not specify what these other reasons are. However, servicers have been using the “other reasons” code increasingly over the past 5 years. We asked servicers how they used the “other reasons” code and found inconsistency in and uncertainty about its use. For example, servicers’ responses ranged from not using it at all, to using it when they did not intend to file an insurance claim with FHA, to not being sure under what circumstances they used it. Second, the list of termination codes consists of both reasons for termination and descriptions of how the debt was satisfied. As a result, the final status code of some loans in HERMIT shows only the way in which the debt was satisfied—for instance, a deed-in-lieu of foreclosure, foreclosure, or short sale. These codes could apply to terminations resulting from the borrower dying, defaulting, or moving and do not ultimately provide a specific reason for loan termination. FHA officials were unaware of any proxy variables that we could use to help identify the underlying termination reasons for these loans. The officials said the termination reasons are available on an individual loan basis in the HERMIT system but not in an extractable form. As discussed later in this report, FHA does not regularly track and report on HECM termination reasons, due partly to this system limitation. The limitations in FHA’s data are inconsistent with federal internal control standards, which require agencies to use quality information to achieve their objectives. To meet this internal control standard, agencies can obtain relevant data that are reasonably free from error and bias and evaluate sources of data for reliability. FHA’s annual report to Congress states that the HECM program helps seniors remain in their homes and age in place. However, without comprehensive and accurate data on HECM terminations, FHA does not have a full understanding of loan outcomes—information FHA and Congress need in order to know how well the HECM program and FHA’s policies are working to help seniors age in place. While FHA has taken steps to improve the performance of the HECM program in recent years, it has not incorporated key elements of performance assessment into its management of the program. We have previously reported that a program performance assessment contains three key elements: program goals, performance metrics, and program evaluations. Performance assessment can provide important information about whether, and why, a program is working well or not. Additionally, OMB Circular A-129 states that agencies must establish appropriate performance indicators for federal credit programs, such as the HECM program, and that such indicators should be reviewed periodically. It states further that agency management structures should clearly delineate accountability and responsibility for defining performance indicators and monitoring and assessing program performance. We found limitations in FHA’s performance assessment of the HECM program, specifically a lack of performance indicators and recent program evaluations: Lack of HECM performance indicators. According to HUD’s strategic plan for fiscal years 2018–2022 and the agency’s most recent annual performance report, the HECM program falls under the strategic goal of advancing economic opportunity and the strategic objective of supporting fair, sustainable homeownership and financial viability. The strategic plan and annual performance report include some strategies for achieving this objective, such as modernizing FHA underwriting guidelines, lending standards, and servicing protocols to serve the needs of borrowers, protect taxpayers, and ensure the sustainability of FHA’s program. However, none of the six performance indicators associated with this strategic objective and discussed in the strategic plan or corresponding performance report are HECM-specific. Four of the indicators focus on FHA-insured forward mortgages. Another indicator focuses on construction of manufactured housing. The remaining indicator— maintaining a capital reserve ratio for FHA’s Mutual Mortgage Insurance Fund that meets or exceeds the statutory minimum requirement— encompasses both forward mortgages and HECMs but does not provide specific information about HECM loan outcomes, risk factors, or loan characteristics. Additionally, FHA’s annual reports to Congress on the financial status of the insurance fund contain multiple tables of HECM data but limited information on loan outcomes. For example, among other things, the fiscal year 2018 report provides the number of new HECM originations, the average age of new borrowers, the amount of HECM insurance claims paid, and estimates of the HECM portfolio’s capital position. However, the report does not include other information that would provide insight into loan outcomes, such as the percentage of HECM terminations due to borrower defaults, the proportion of active HECMs with delinquent property charges, or the percentage of distressed HECM borrowers who have received foreclosure prevention options. Limited program evaluations. The last comprehensive evaluations of the HECM program were done in 2000 and 1995. Officials said they were in the planning phase for a new evaluation of the HECM program but had not set a start date and did not expect the evaluation to include an analysis of reasons for HECM terminations or the use of foreclosure prevention options for borrowers in default. Instead, the officials told us the evaluation would focus on the impact of an FHA policy change implemented in 2015 that requires prospective HECM borrowers to undergo a financial assessment to evaluate their ability to pay ongoing property charges. While financial assessments could help reduce tax and insurance defaults, and ultimately foreclosures, they only apply to new HECMs issued on or after the effective date of the policy (April 27, 2015) and are not relevant to other HECMs within the portfolio. Therefore, for most of the HECM portfolio, an equally important consideration is the impact of FHA’s policy changes that created foreclosure prevention options for distressed borrowers. As previously noted, borrower defaults have accounted for an increasing proportion of terminations in recent years, and in fiscal year 2018, borrower defaults made up 18 percent of terminations. Expanding the program evaluation to include the impact of foreclosure prevention options would provide a more complete picture of how well FHA is reducing defaults in the HECM portfolio and helping HECM borrowers. FHA officials acknowledged the need for more performance assessment of the HECM program. The officials said their recent focus has been on financial aspects of the program, in particular losses associated with insurance claims. According to the FHA Commissioner, a key challenge for the HECM program is that FHA has historically administered it without a designated program head. The 2000 program evaluation noted that lenders and servicers found it frustrating that FHA did not have one person with responsibility for the HECM program. Further, the 2000 program evaluation noted that the division of responsibility for the program fell across many offices and that it was hard to find senior managers with a sense of ownership for the HECM program. In January 2019, an economist from HUD’s Office of Policy Development and Research transferred to the Office of Housing (which includes FHA) to serve as a Senior Advisor to the Deputy Assistant Secretary for Single Family Programs, with a focus on the HECM program. Without more comprehensive performance indicators and program evaluations, FHA lacks information that could be useful for monitoring the effects of recent policy changes and may be missing evidence of the need for further program improvements. Additionally, in the absence of performance indicators and reporting, FHA and Congress lack insight into how well the HECM program is helping senior homeowners. According to OMB Circular A-129, agencies must have monitoring, analysis, and reporting mechanisms in place to provide a clear understanding of a program’s performance. The circular says these mechanisms should be sufficiently flexible to perform any analysis needed to respond to developing issues in the loan portfolio. However, we found shortcomings in FHA’s internal reporting. We also found that FHA had not analyzed the implications of its foreclosure prioritization process for FHA-assigned loans. Internal reporting for the HECM program is limited. Although FHA adopted the HERMIT system to improve oversight of the HECM portfolio, it has not used program data to regularly report key loan performance information—for example, HECM termination reasons, servicer advances, and use of foreclosure prevention options. FHA officials said they have been more focused on the analysis and reporting of claims and other financial data for the HECM program. However, according to OMB Circular A-129, effective reporting provides accurate, timely information on program performance, early warnings of issues that may arise, and analytics to drive decision-making. FHA has generated some reports from HERMIT to help oversee the HECM portfolio, but it has been slow to develop regular and comprehensive reporting mechanisms. FHA officials told us that while data on defaults and use of foreclosure prevention options have generally been available in HERMIT since 2015, FHA was unable to obtain reports on these topics until the summer of 2018 because of contract funding limitations. FHA officials said that starting in September 2018, FHA began receiving regular reports from its HERMIT system contractor on issues such as HECMs assigned to FHA; HECM origination, assignment, and termination activity by month; summary information on the number and dollar amount of HECMs originated each year; and HECMs with a default date. Additionally, around the same time, FHA requested and received ad hoc reports (one-time reports created for specific purposes) from the contractor that included spreadsheets of all active HECMs with a repayment plan and all active HECMs for which there was an identified nonborrowing spouse. FHA officials said the purpose of the reports generated from HERMIT is to help FHA better manage HECM program performance. However, our review of these regular and ad hoc reports found that many are lists of loans that meet certain criteria and do not provide summary statistics that could be used to readily identify patterns or trends in metrics, such as the number of or reasons for HECM terminations or use of different foreclosure prevention options. The reports require additional analysis to generate meaningful management information. According to OMB Circular A-129, graphics, tables, and trend analysis that compare performance over time and against expectations and other information can provide critical context for understanding program performance. Further, the circular says dashboards (easy-to-comprehend summaries of key quantitative and qualitative information) and watch lists are tools that can help all levels of the organization receive appropriate information to inform proactive portfolio management and ensure program decisions are informed by robust analytics. FHA’s lack of analysis and internal reporting on HECM termination reasons hampered the agency’s ability to respond to a 2017 Freedom of Information Act request about the number of and reasons for HECM foreclosures. FHA’s response contained data showing that over 99 percent of HECM foreclosures occurring from April 2009 through December 2016 resulted from the death of the borrower. However, FHA officials told us they subsequently looked more closely into the issue and redid the analysis using more reliable and updated information from January 2013 through December 2017. The revised analysis showed that 61 percent of foreclosures over that period were due to borrower deaths, 37 percent were due to borrower defaults, and 2 percent were due to conveyance of title. If FHA had regular and meaningful management information about HECM terminations, it could have initially responded to the 2017 request with more reliable information. FHA officials told us that HERMIT is an accounting system to process HECM claims and has limitations as a broader portfolio monitoring tool. However, our analysis of HERMIT data and reports generated by FHA’s HERMIT contractor suggest that the system can be used for this broader purpose. Without more robust program analysis and internal reporting, FHA is not well positioned to detect and respond to any emerging issues and trends in the HECM portfolio. As previously discussed, these trends include growing numbers of HECMs entering default and an increasing number of loans being assigned to FHA. FHA has not evaluated its foreclosure prioritization process for FHA- assigned HECMs. As previously noted, FHA-assigned loans are a growing part of the HECM portfolio. According to FHA officials, the agency generally does not foreclose on borrowers whose HECMs have been assigned to FHA and who are in default due to unpaid property charges. According to FHA, the properties associated with these loans are typically occupied. FHA officials said the agency prioritizes processing foreclosures on assigned HECMs for which the property is vacant (because the borrower passed away, for example). FHA officials said that prioritizing foreclosure processing for those loans and delays by the Department of Justice in completing those foreclosures has effectively resulted in few foreclosures on assigned loans with property charge defaults. However, FHA regulations state that servicers generally must initiate foreclosure within 6 months of calling a loan due and payable due to a death or default (if the borrower or heirs have not yet paid the debt off). FHA’s prioritization of processing vacant properties for foreclosure and generally not foreclosing on FHA-assigned HECMs with a property charge default raises issues and potential risks that FHA has not fully analyzed. First, defaulted borrowers whose loans are privately owned (that is, have not been assigned to FHA) face a greater risk of foreclosure than defaulted borrowers with FHA-assigned loans. According to a representative from one HECM servicer we interviewed, FHA’s practice is unfair because it treats HECM borrowers inconsistently. Second, FHA’s foreclosure prioritization processing may create a financial incentive for HECM borrowers with assigned loans to not pay their property charges, which, in turn, can have negative financial consequences for FHA, localities, and taxpayers. For example, because FHA does not foreclose on assigned loans in tax and insurance default, FHA advances tax and insurance payments on behalf of the borrower and adds them to the loan balance to secure and maintain its first-lien position on the mortgaged property. This makes it more likely that the loan balance will increase to a point that it exceeds the value of the home. When the borrower dies or vacates such a property, FHA may not be able to recoup the loan balance in a foreclosure sale, resulting in a loss to the insurance fund. As of August 2019, FHA had not evaluated the various risks of generally not foreclosing on assigned HECMs with property charge defaults. As a result, FHA does not know how its process for prioritizing foreclosures for assigned loans affects the HECM portfolio, HECM borrowers, neighborhoods, and FHA’s insurance fund. FHA’s oversight of HECM servicers is limited. FHA requires HECM servicers, among other things, to inform borrowers of their loan status, including any conditions resulting in a loan becoming due and payable; to notify struggling borrowers of the availability of housing counseling and foreclosure prevention options; to inform surviving nonborrowing spouses of conditions and requirements for the deferral period; and to manage the transfer of loan servicing from one entity to another. These requirements are identified in FHA regulations, handbooks, and mortgagee letters. If properly implemented, these requirements can help ensure that HECM borrowers and nonborrowing spouses are aware of their mortgage responsibilities, options for resolving situations that can result in foreclosure, and who to contact with loan servicing questions. FHA officials said they maintain communication with HECM servicers, including through an industry working group, about their compliance with FHA requirements. The officials also noted that FHA conducts reviews of due and payable requests and insurance claims, which can include checks for some of the requirements discussed above. However, FHA has not performed comprehensive on-site reviews of HECM servicers’ compliance with program requirements since fiscal year 2013 and does not have current procedures for conducting these reviews. The lack of on-site reviews of HECM servicers is inconsistent with OMB requirements for managing federal credit programs. OMB Circular A- 129 states that agencies should conduct on-site lender and servicer reviews biennially where possible and annually for lenders and servicers with substantial loan volumes or those with other risk indicators such as deterioration in their credit portfolio, default rates above acceptable levels, or an abnormally high number of reduced or rejected claims. The purpose of these reviews is to evaluate and enforce lender and servicer performance and identify any noncompliance with program requirements. The circular encourages agencies to develop a risk-rating system for lenders and servicers to help establish priorities for on-site reviews and to monitor the effectiveness of required corrective actions. The circular also says that agencies should summarize review findings in written reports with recommended corrective actions. FHA previously conducted on-site reviews of HECM servicers. However, according to agency data, FHA has not performed on-site reviews since fiscal year 2013. From fiscal years 2010 through 2013, FHA’s Quality Assurance Division (a component of the Office of Lender Activities and Program Compliance) conducted 14 on-site reviews of HECM servicers (see table 3). These reviews examined compliance with FHA servicing requirements and included detailed reviews of samples of loans. FHA provided us three examples of HECM servicing reviews conducted in fiscal year 2013. While not representative of all reviews, the three reviews identified multiple violations of FHA requirements, as follows: Quality control plans. Two of the three reviews found that the servicers’ quality control plans—an internal control mechanism to help ensure compliance with FHA requirements—were missing required elements. For example, one review found that the servicer’s plan lacked 13 required elements, including those intended to ensure compliance with fair lending laws and immediate reporting of fraud or other serious violations. Another review found deficiencies with the servicer’s plan, including in the areas of customer service, servicing transfers, and fees and charges. Communication with borrowers. In these same two reviews, FHA found that the servicers did not always provide borrowers with a designated contact person or timely and accurate information about their loan status. For both servicers, FHA’s reviews of files for a sample of active loans found no evidence that the servicer had provided the borrower a contact person to handle inquiries. FHA requires servicers to designate for borrowers a contact person knowledgeable about servicing and provide the name of the person annually and whenever the contact person changes. Additionally, both reviews found that the servicers’ annual loan statements to borrowers were missing critical information, such as the net principal limit (total loan funds available), and that the servicers did not provide borrowers with statements after every loan disbursement, as required. Filing claims. In two of the three reviews, FHA found deficiencies in the servicers’ filing of insurance claims. For example, in one review, FHA identified multiple cases where the servicer submitted claims that were greater than the amounts warranted, including excess attorney and appraisal fees, property preservation and protection expenses, and interest costs. In another review, FHA found numerous instances where the servicer missed various deadlines—including for submitting claims, commencing foreclosure, and obtaining appraisals—and therefore was not entitled to the full claim amounts it received. Loan disbursements. One of the three reviews found numerous instances in which the servicer did not respond to borrowers’ requests for payment plan changes within the required time frame of 5 business days, and therefore did not make timely loan disbursements to borrowers. FHA required these servicers to take corrective actions, including updating quality control plans, revising policies and procedures, reimbursing FHA for unwarranted claim amounts, indemnifying FHA for losses on a loan, and paying late charges to borrowers who did not receive timely loan disbursements. FHA has the option of referring violations of FHA requirements to HUD’s Mortgagee Review Board, which can take administrative actions such as issuing letters of reprimand, suspending or withdrawing approval to participate in FHA programs, entering into settlement agreements to bring an entity into compliance, and imposing civil money penalties. FHA officials said they had not referred any HECM servicers to the board as a result of findings from on- site reviews. According to FHA’s current Director of the Quality Assurance Division, under previous leadership, the division suspended on-site reviews of HECM servicers after fiscal year 2016 because of servicers’ concerns about the clarity and consistency with which FHA was conducting the reviews and applying enforcement remedies. He said the Quality Assurance Division had intended to revise its guidance for conducting the reviews and then resume them, but the effort had stalled during a change in leadership. The current Director said he was not aware that HECM servicing reviews had been suspended until the fall of 2017, when the division began targeting on-site reviews for fiscal year 2018, and noticed that HECM servicers were not included in the prior year’s targeting methodology. The lack of recent HECM servicer reviews is problematic for a number of reasons. First, as previously noted, the number of HECM borrowers defaulting on their loans has grown in recent years. As a result, knowing whether servicers are providing borrowers with accurate and timely communications about their mortgage obligations and the status of their loans has become increasingly critical. Second, FHA has recently made program changes and implemented foreclosure prevention options, such as at-risk extensions and mortgagee optional election assignments, to help struggling borrowers and nonborrowing spouses delay or avoid foreclosure. But FHA does not know how effectively servicers inform borrowers of these options and use these tools due to its lack of oversight. Third, as discussed earlier, the majority of HECM servicers are nonbank entities that may pose risks because they are not subject to the same comprehensive federal safety and soundness regulations as banks and rely on funding sources, such as lines of credit, that may be less stable than deposits. The Director of the Quality Assurance Division said FHA plans to begin conducting HECM servicer reviews in fiscal year 2020 but will first need to revise its procedures for reviewing HECM servicers, which were last updated in 2009. However, the Director told us the division decided not to develop criteria for selecting HECM servicers for review. Instead, he said FHA plans to review all HECM servicers with significant portfolios at least once every 3 years, starting with the three servicers that account for 96 percent of the HECM portfolio. While FHA’s plan to review HECM servicers with significant portfolios captures one aspect of portfolio risk (loan volume), it does not account for other risk indicators that OMB Circular A-129 says agencies should consider. The circular also encourages agencies to develop risk-rating systems that incorporate these indicators. While the current HECM servicing market is dominated by a small number of companies, the ability to prioritize on-site reviews based on risk ratings will be important if the market becomes less concentrated in the future. Additionally, some HECM servicers may warrant review more frequently than once every 3 years if their business volume or performance poses substantial risks to FHA or to borrowers. FHA’s plans do not account for these contingencies. CFPB oversees reverse mortgage servicers through examinations designed, among other things, to identify whether servicers engage in acts or practices that violate federal consumer financial laws. CFPB issued its Reverse Mortgage Examination Procedures in 2016 and began conducting examinations in 2017. CFPB’s procedures include reviewing servicers’ compliance with the Real Estate Settlement Procedures Act of 1974 and its implementing regulations (which, among other things, contain requirements for notifying borrowers of servicing transfers, responding to borrowers’ written information requests and notices of error, and disclosures relating to force-placed insurance); the Truth In Lending Act and its implementing regulations (which impose requirements on servicers governing the use of late fees and delinquency charges, provisions for payoff statements, and disclosures regarding rate changes for adjustable-rate mortgages); and other consumer protection laws. Additionally, CFPB’s procedures include a review of whether a HECM servicer is following selected elements of FHA’s HECM program requirements. For example, CFPB’s examiners are directed to determine whether information provided to the borrowers about life expectancy set- aside accounts (an FHA requirement) is clear, prominent, and readily understandable, and whether the borrower incurred penalties or unnecessary charges in the event the servicer failed to make disbursements of set-aside funds for insurance, taxes, and other charges with respect to the property in a timely manner. CFPB examiners also are directed to determine whether the servicer referred a HECM to foreclosure improperly after the death of a borrower, such as when an eligible nonborrowing spouse still occupies the home. If CFPB’s reverse mortgage examinations identify violations, CFPB may require the examined entity to take corrective actions, which are recorded in the examination results as matters requiring attention. CFPB examinations of reverse mortgage servicers have found deficiencies in monitoring of servicing actions, compliance with consumer protection laws, and communications with consumers. For example, CFPB reported in the March 2019 edition of its Supervisory Highlights that one or more reverse mortgage servicing examinations found cases where the servicer did not provide the heirs of deceased borrowers a complete list of the documents needed to evaluate their case for a foreclosure extension. (Extensions can give heirs additional time to sell or purchase the property and delay or avoid foreclosure.) As a result, in some instances, one or more servicers foreclosed rather than seeking a foreclosure extension from FHA. According to CFPB, in response to the examinations, one or more servicers planned to improve communications with borrowers’ heirs, including specifying the documents needed for a foreclosure extension and the relevant deadlines. CFPB officials said they plan to continue examining reverse mortgage servicers. In addition to conducting examinations and issuing matters requiring attention, CFPB officials said the bureau has other options—including issuing warning letters and taking enforcement actions—to stop unlawful practices or promote future compliance by supervised entities. Warning letters advise companies that certain practices may violate federal consumer financial law. Enforcement actions are legal actions against an entity initiated through federal district court or by an administrative adjudication proceeding. CFPB officials told us the bureau had not issued any warning letters or enforcement actions against HECM servicers as of August 2019. While CFPB has examined reverse mortgage servicers and plans to continue doing so, CFPB officials said the bureau and FHA do not have an agreement in place to share supervisory information, which inhibits sharing of examination results. Information-sharing agreements may address topics such as what and how information will be shared and handling of sensitive information. CFPB officials said that an agreement with FHA would be needed to ensure that supervisory information in the bureau’s examinations is kept confidential. Under the Dodd-Frank Wall Street Reform and Consumer Protection Act, CFPB must share results of the examination of a supervised entity with another federal agency that has jurisdiction over that entity, provided that CFPB received from the agency reasonable assurances as to the confidentiality of the information disclosed. In addition, in previously issued work, we noted that interagency collaboration can serve a number of purposes, including, among other things, policy development, oversight and monitoring, and information sharing and communication. CFPB officials said CFPB and FHA had taken initial steps in 2017 toward developing an information-sharing agreement. However, as of August 2019, an information-sharing agreement had not been completed. CFPB officials told us there were existing ways for the two agencies to share examination findings, but that an information-sharing agreement would facilitate the process. CFPB officials said developing information- sharing agreements can be a lengthy process and that both agencies had other competing priorities. However, because of the limited information sharing between CFPB and FHA, FHA is not benefiting from oversight findings about servicers it could rely on to help implement the HECM program. Having this information is particularly important given that FHA does not comprehensively review HECM servicers itself and CFPB’s examinations address a number of FHA requirements. Access to CFPB’s examination results could enhance FHA’s oversight of HECM servicers and potentially help it respond to consumer protection issues facing HECM borrowers. CFPB collects, analyzes, and reports on consumer complaints related to reverse mortgages. The bureau began collecting reverse mortgage consumer complaints in December 2011 and has collected about 3,600 complaints since then. CFPB collects complaints through an online forum on its website called the Consumer Complaint Database, as well as via email, mail, phone, fax, or referral from another agency. CFPB’s authority to collect complaints comes from the Dodd-Frank Wall Street Reform and Consumer Protection Act, which states that one of the bureau’s primary functions is collecting, investigating, and responding to consumer complaints. CFPB officials told us the bureau uses consumer complaints as part of its criteria for selecting entities to examine, including reverse mortgage servicers, and to inform its educational publications. For example, in June 2015, CFPB released a report on reverse mortgage advertising and consumer risks. In August 2017, CFPB released an issue brief on the costs and risks of using a reverse mortgage to delay collecting Social Security benefits. In February 2015, CFPB issued a report on reverse mortgage consumer complaints it received from December 2011 through December 2014. CFPB found that consumer complaints indicated frustration and confusion over the terms and requirements of reverse mortgages. CFPB also found that many complaints were about problems with loan servicing. For example, some consumers complained that they were at risk of foreclosure due to nonpayment of property taxes or homeowners insurance and that they faced obstacles when trying to prevent a foreclosure. CFPB officials told us they did not currently have plans to publish additional reports on reverse mortgage complaints, but that CFPB would continue to produce educational materials on reverse mortgages and internally review the data on a routine basis. For this report, we performed a high-level analysis of roughly 2,500 reverse mortgage complaints received by CFPB from calendar years 2015 through 2018. We analyzed patterns in the number of complaints by year, state, submission method, and company. By year. Complaint volumes varied across the 4 years, with the most complaints received in 2016 and the least received in 2018 (see table 4). By state. The states with the most complaints were California (accounting for 16 percent of reverse mortgage complaints), Florida (11 percent), New York (8 percent), and Texas (7 percent). These states are among the most populous, and three of them (California, Florida, and Texas) also had the greatest numbers of HECMs. By submission method. A majority of reverse mortgage complaints (56 percent) were submitted through CFPB’s website. The remaining complaints were submitted through referrals to CFPB from other agencies (22 percent), by phone (12 percent), by postal mail (7 percent), and by fax (3 percent). Compared to the percentage of all types of mortgage complaints filed during the 4-year period, the percentage of reverse mortgage complaints filed through the website (56 percent) was lower than the corresponding percentage for complaints about all types of mortgages (67 percent). Representatives from legal aid organizations representing HECM borrowers said that reverse mortgage consumers may be less likely to file a complaint through a website because of limitations sometimes related to aging— for example, lack of internet access or computer skills. Additionally, representatives from three of the five organizations said seniors may suffer from health or capacity issues, such as hearing, vision, or memory loss, that may make it difficult for them to file or follow up on a complaint. For these reasons, seniors may not be submitting complaints through CFPB’s website and seniors’ complaints about reverse mortgages may be underreported in general. By company. Companies that were the subject of reverse mortgage complaints included both lenders and servicers. From 2015 through 2018, five companies were the subject of more than 100 complaints each, ranging from a low of 116 to a high of 506. Together, these five companies accounted for 61 percent (1,509) of the reverse mortgage complaints CFPB received. Additionally, one company received the most complaints in 4 out of the 5 years reviewed. We also conducted a more detailed analysis of a random, generalizable sample of 100 consumer complaint narratives from among the 2,472 total reverse mortgage complaints CFPB received in calendar years 2015 through 2018. The purpose of this analysis was to identify patterns in consumer-described issues about reverse mortgages. We created issue categories by reading the consumer narratives. Figure 6 shows the estimated percentage of reverse mortgage complaints received by CFPB over the 4-year period by consumer-described issue categories, based on our sample of 100 complaint narratives. Among the largest consumer-described issue categories were foreclosures; poor communication from lenders or servicers; problems at loan origination; estate management; and unfair interest rates, fees, or costs. Being at risk of foreclosure or in foreclosure. The largest consumer-described issue category (47 percent) involved consumers (or someone complaining on behalf of the consumer) who said they were at risk of foreclosure or in the foreclosure process. For example, some consumers said they or the borrower they represent had received a notice of default, were in due and payable status, or were at risk of foreclosure. Some consumers sought help in preventing foreclosure or felt they were wrongly being foreclosed on. In 16 of the 47 complaints about being at risk of or in foreclosure, consumers also cited concerns about property taxes, insurance, or other property charges. Poor communication on a servicing or lending issue. The second largest consumer-described issue category (42 percent) involved complaints about poor communication on a reverse mortgage servicing or lending issue. These complaints included concerns about a lack of communication or communications that were unclear or unresponsive to the consumer’s needs. Complaints in this category often overlapped with those about being at risk of or in foreclosure. For example, some of these complaints included consumers’ concerns that they had not received information about the status of or reason for a possible foreclosure from their servicer or did not get responses to their inquiries. Loan origination issues. The third largest complaint category involved problems occurring at loan origination (29 percent). These complaints included consumers’ concerns that the amount of funds available from their reverse mortgage was less than expected or that interest rates or fees were not disclosed or explained to them. The complaints also included cases where the adult children of borrowers said they felt the lender took advantage of their parents. Estate-management issues. Twenty-seven percent of consumer complaints were about estate management issues. Complaints involving estate-management were often submitted by deceased borrowers’ families or heirs. In some cases, heirs said that they were unable to get information about the status of the reverse mortgage. In other cases, the heirs said that because of the reverse mortgage, they were at risk of losing the home, which was also their place of residence. Unfair interest rates, fees, or costs. Twenty-seven percent of consumer complaints were about being charged higher-than-expected costs, fees, or interest. For example, in a few complaints, consumers said that their servicers required them to pay for insurance products (for example, flood insurance) that they felt were not needed. According to CFPB officials, the bureau (1) refers consumer complaints about financial products and services to the companies the complaints are about or other federal regulators with supervisory jurisdiction over those companies or (2) makes complaint information available to other federal agencies with jurisdiction over the relevant product or service. CFPB officials said the bureau does not currently refer reverse mortgage complaints to FHA; however, they told us reverse mortgage complaints are available to FHA through CFPB’s public website and through a secure portal FHA can access that has more data available than on the public website. FHA collects and records inquiries and complaints about HECMs and, as previously mentioned, has access to CFPB data on reverse mortgage complaints. However, FHA does not use its inquiry and complaint data to help inform HECM program policies and oversight, and the way data are collected does not produce quality information for these purposes. Additionally, FHA has not leveraged CFPB’s complaint data for HECM program oversight. Federal internal control standards state that agencies should use quality information to achieve the entity’s objectives, including using relevant data from reliable internal and external sources. Additionally, in prior work we identified practices to enhance collaboration across agencies, including leveraging agency resources. According to agency officials, FHA’s two main methods for collecting customer inquiries and complaints are hotlines operated by the agency’s National Servicing Center and the FHA Resource Center. Historically, the National Servicing Center was FHA’s primary method for collecting inquiries and complaints about the HECM program. From calendar years 2015 through 2018, the National Servicing Center received about 105,000 HECM-related calls. During this same period, the FHA Resource Center received 147 HECM-related calls. In April 2019, the FHA Resource Center became the primary entity for collecting, recording, and responding to all HECM-related calls. FHA officials told us they transferred these responsibilities from the National Servicing Center to the FHA Resource Center to help improve call management. While this change could help improve customer service, it would not fully resolve limitations we found in FHA’s approach to collecting and recording HECM inquiries and complaints that diminish the usefulness of the information for program oversight. These limitations include the following: Information is not suitable for thematic analysis. Both the National Servicing Center and the FHA Resource Center do not collect call information in a way that would allow FHA to readily analyze the data for themes. For example, both centers do not reliably differentiate between inquiries and complaints—a potentially important distinction for determining appropriate agency-level responses (for example, creating informational materials to address frequently asked questions from borrowers or investigating problematic servicing practices after repeated complaints). Additionally, while both the centers collect data on the reason for calls, neither did so in a systematic way that would allow FHA to readily determine how frequently issues are being raised. For example, neither centers’ data systems contained standardized categories or menus with options for recording reasons for calls. As a result, the FHA Resource Center’s data from 2015 through 2018 contained more than 100 separate reasons for 147 HECM-related calls. Some of the reasons the center recorded were too specific (for example, a property address or a case number) to be useful for identifying themes, while others were so similar that they do not provide meaningful distinctions (but could be combined into fewer, potentially more useful categories). We noted similar limitations in the National Servicing Center’s data, which included ambiguous call reasons such as “history” and “documents,” and categories that could be collapsed, which hinders thematic analysis. Customer type is not recorded. The National Servicing Center, which received the large majority of HECM-related calls to FHA, did not record information on the type of customer that made the call. National Servicing Center guidance for staff says customers include borrowers, nonprofit organizations, government entities, real estate brokers and agents, title companies, attorneys, lenders and servicers, and HUD employees, but its data system does not include these categories. Information on customer type could be useful in identifying issues facing different populations of callers and could help FHA tailor strategies for addressing their concerns. In contrast, the FHA Resource Center’s data system does include categories for customer type for the smaller number of HECM-related inquiries and complaints it received. Because the FHA Resource Center’s system is now FHA’s primary repository for new HECM-related calls, information on customer type should be available for future inquiries and complaints. However, this information is not available for the bulk of HECM-related calls FHA received in prior years. FHA officials said the agency uses customer complaint and inquiry data to improve customer service. For example, FHA officials said the National Servicing Center monitors calls on a daily basis to ensure that prompt responses are provided. Similarly, FHA officials said they review call data monthly to identify training needs of servicers or contractors and potential process changes to improve customer experience with the call process. However, FHA does not analyze data for other purposes that could enhance program oversight, such as determining which HECM servicers and lenders receive the most complaints, targeting entities for on-site reviews, or identifying topics that may need additional borrower education. FHA also does not use CFPB’s consumer complaint data to inform management and oversight of the HECM program, even though some of the information could be useful to the agency. For example, according to CFPB’s complaint data for 2015 through 2018, approximately 6 percent of reverse mortgage complaints were about FHA’s servicing contractor. FHA officials said they do not review CFPB’s complaint data because they believe the data are too limited to be useful and because they have concerns about CFPB’s controls over data integrity. However, as our analysis shows, CFPB’s data can be used to identify consumer concerns—such as difficulties avoiding or navigating foreclosure or problems communicating with servicers—that may merit additional attention by FHA. Additionally, CFPB’s Office of Inspector General recently reviewed CFPB’s management controls for the Consumer Complaint Database and did not identify major data integrity issues that would preclude use of the data for general oversight purposes. Periodically analyzing CFPB consumer complaint data and internally collected consumer complaint data could help FHA to detect and respond to consumer protection issues regarding HECMs. Since 2000, the take-up rate—the ratio of HECM originations to eligible senior homeowners—has been limited (see fig. 7). This rate, which provides an indication of how popular HECMs are among the population of senior homeowners, has not reached 1 percent and has fallen in recent years. In addition, since calendar year 2010, the volume of HECM originations has declined and is about half of what originations had been at their peak. For example, in calendar years 2007–2009, more than 100,000 new HECMs were originated each year, compared with roughly 42,000 in calendar year 2018. The relatively high homeownership rate and low retirement savings of U.S. seniors suggest that reverse mortgages could be a way for many older Americans to tap their home equity and supplement retirement income. However, the popularity of reverse mortgages has declined in recent years for a number of possible reasons. We developed an econometric model to examine the relationship between HECM take-up rates and a number of explanatory variables. For additional information and detailed results from our econometric model of factors associated with HECM take-up rates, see appendix II. Among other factors, our model results indicate that house price changes, home equity, and prior use of other home equity lending products were statistically significant (at the 1 percent level) in explaining the decrease in HECM take-up rates since 2010. Changes in house prices. The decline in take-up rates may reflect lower house prices, which have limited the number of households with sufficient home equity (as a percentage of home value) to benefit from a HECM. Our model estimated that, controlling for other factors, take- up rates were higher when house price growth was large and there was a history of house price volatility compared to either relatively low house price appreciation or stable house prices. This result is consistent with senior homeowners using reverse mortgages to insure against house price declines. For example, researchers have noted that in states where house prices are volatile and the current level is above the long-term norm, seniors anticipate future reductions in house prices and lock in their home equity gains by obtaining a reverse mortgage. Home equity and prior home equity borrowing. Additionally, we found that controlling for other factors, take-up rates were higher where home equity (house value minus any mortgage debt) was high. In these cases, senior homeowners tap into their high home equity to help supplement income with proceeds from the HECM. Further, we found that among seniors who had previously used other home equity lending products, such as home equity loans, take-up rates were high. This result is consistent with seniors using HECMs to pay off these loans. Academics and industry experts have also noted possible reasons why the popularity of reverse mortgages is limited. For example, senior homeowners can tap their home equity by other means, such as home equity loans, home equity lines of credit, and cash-out refinancing. Some of these options may be less expensive than reverse mortgages. Seniors can also downsize––sell their current home and buy or rent a less expensive one—and keep the difference to supplement retirement savings. Seniors have other ways to supplement their retirement income and age in place—for example, one academic noted that some seniors rent out rooms in their homes, potentially using online marketplaces such as Airbnb. Additionally, our literature review and interviews with academics identified other factors that have may have contributed to limited interest in reverse mortgages, including the following: Exit of large bank lenders. As previously noted, banks, thrifts, and credit unions were historically the primary lenders and servicers of mortgage loans. Following the 2007–2009 financial crisis and subsequent revisions to regulatory bank capital requirements, banks reevaluated the benefits and costs of being in the mortgage lending market, as well as retaining mortgages and the right to service them. Today, the reverse mortgage market is dominated by a relatively small number of nonbank entities. The exit of large, well-known lenders, such as Bank of America and Wells Fargo, from the HECM market created opportunities for smaller nonbank lenders to enter the market. According to an academic we spoke with, in addition to new capital requirements, large banks may have exited the market partly out of concern that they risked damage to their reputations if they foreclosed on seniors who defaulted on their HECMs. Additionally, a 2018 survey of lenders found a variety of reasons why lenders have stopped originating HECMs, including potential reputation risk and concerns about HECMs being a distraction from their forward mortgage business. Although the HECM market is currently served by several nonbank lenders, their smaller scale, limited access to capital, and limited name recognition may limit their ability to reach more potential borrowers. FHA policy changes to the HECM program. FHA has made several policy changes in recent years to help stabilize the financial performance of the HECM portfolio and strengthen financial criteria for HECM borrowers. Although many of the HECM policy changes introduced since 2010 were intended to minimize program losses, they also may have reduced take-up rates. For example, in 2010 FHA reduced the amount of money a borrower can get from a HECM. Some academics we interviewed said reductions in the loan amounts that borrowers can receive likely reduced demand for HECMs. In 2015, FHA changed financial requirements for HECMs to include a financial assessment of the prospective borrower prior to loan approval. Some academics said these changes made other home equity extraction options that already had similar requirements more competitive with HECMs. Consumers’ misunderstanding and product complexity. A 2013 survey of U.S. homeowners aged 58 and older revealed a lack of knowledge of reverse mortgages. The survey found that awareness of reverse mortgages is high, but knowledge of mortgage terms is limited. Additionally, the survey found that respondents perceived reverse mortgages to be fairly complex. Consumers’ perception of the product. Academics we spoke with told us that consumers’ negative perception of reverse mortgages likely has a negative influence on take-up rates. For example, three academics elaborated that consumers build their perception of the product based on the industry’s marketing and advertising, which includes television commercials with celebrity spokespeople that may appeal to individuals facing economic hardship. Additionally, a 2016 survey of Americans aged 55 to 75 found that many respondents had reservations about reverse mortgages, including that they are often considered a financial tool of last resort. For example, only 27 percent of survey respondents stated that, in general, it was better to use a reverse mortgage earlier in retirement as opposed to using it as a last resort. Relatively high origination costs and fees. HECMs also may be unpopular with borrowers because they can be more expensive than other home equity lending products, such as home equity lines of credit. For example, HECM borrowers are charged various fees, such as the up-front insurance premiums that FHA charges as compensation for its insurance guarantee and origination fees lenders charge. The up-front insurance premium is 2 percent of the mortgage’s maximum claim amount. Also, for origination fees, lenders can charge the greater of $2,500 or 2 percent of the first $200,000 of the mortgage’s maximum claim amount plus 1 percent of the maximum claim amount over $200,000. However, origination fees are currently capped at $6,000. Further, because borrowers do not make monthly payments on the loans, the interest will accumulate over time, and compounding the interest, the loan balance can rise quickly. Seniors’ attitudes toward debt and desire to leave a bequest. Some academics have noted that seniors tend to be financially conservative and avoid debt in old age––behavior driven by their desire to leave a bequest or save for emergency expenses or long- term care costs. For example, academics have noted that some impediments to home equity extraction are behavioral and have to do with seniors’ long-held values, beliefs, and attitudes, such as to maximize wealth transfer to heirs by leaving a bequest. As a result, they may be reluctant to take out a HECM, even if it could help pay for some future expenses. HECMs allow seniors to tap a portion of their home equity to supplement their retirement income, but these loans can present risks to borrowers and their spouses. The growing number of borrowers who have defaulted on their HECMs and faced foreclosure in recent years highlights the importance of monitoring loan outcomes and overseeing loan servicing policies and practices in the HECM program. FHA has taken some steps to enhance the data it receives from servicers and has created foreclosure prevention options for distressed borrowers. However, FHA could significantly improve its monitoring of loan outcomes and oversight of servicing in the HECM program in the following areas: FHA’s lack of comprehensive termination data limits understanding of the reasons why HECMs end, how the debt is satisfied, and how well the program is helping seniors age in place. By, for example, updating and providing more guidance to servicers on how to record termination reasons, FHA could improve the completeness and accuracy of HECM termination data. FHA has not effectively assessed the performance of the HECM program. By establishing performance indicators and periodically assessing them, FHA could better oversee the program and communicate information on program performance to Congress. Further, FHA could use the performance data to help make informed decisions about any needed program changes in the future. FHA’s internal monitoring and reporting on loan outcomes has been limited. Adopting analytic tools could better position FHA to evaluate loan outcomes and help ensure senior officials have information needed to make key decisions. FHA has not fully analyzed the implications of how it prioritizes foreclosures for assigned HECMs. FHA’s current process generally results in no foreclosures on assigned loans with property charge defaults. Analyzing the implications of this process could help FHA optimize how it services assigned loans. Because FHA does not currently perform on-site reviews of HECM servicers, it lacks assurance that servicers are complying with rules and program requirements. While FHA plans to begin reviewing HECM servicers in fiscal year 2020, its plan does not include development of a risk-rating system to prioritize reviews and identify servicers that should be reviewed more frequently. CFPB does not share the results of its examinations of HECM servicers with FHA, in part because the two agencies have not completed a formal information-sharing agreement. Sharing these results could aid FHA’s oversight of HECM servicers by providing additional information about the servicers’ performance and operations. FHA’s collection and use of consumer complaint data could be improved. More organized collection of complaints and better monitoring of internal and external complaint data could help FHA detect and respond to emerging consumer protection issues regarding HECMs. By addressing these issues, FHA could help ensure that the HECM program achieves program goals, effectively oversees servicers, and provides appropriate borrower protections. We are making a total of nine recommendations, eight to FHA and one to CFPB: The FHA Commissioner should take steps to improve the quality and accuracy of HECM termination data. These steps may include updating the termination reasons in the HERMIT system or updating the HERMIT User Guide to more clearly instruct servicers how to record termination reasons. (Recommendation 1) The FHA Commissioner should establish, periodically review, and report on performance indicators for the HECM program—such as the percentage of terminations due to borrower defaults, the proportion of active HECMs with delinquent property charges, the amount of servicer advances, and the percentage of distressed borrowers who have received foreclosure prevention options—and examine the impact of foreclosure prevention options in the forthcoming HECM program evaluation. (Recommendation 2) The FHA Commissioner should develop analytic tools, such as dashboards or watch lists, to better monitor outcomes for the HECM portfolio, such as reasons for HECM terminations, defaults, use of foreclosure prevention options, or advances paid by servicers on behalf of HECM borrowers. (Recommendation 3) The FHA Commissioner should evaluate FHA’s foreclosure prioritization process for FHA-assigned loans. Such an analysis should include the implications that the process may have for HECM borrowers, neighborhoods, and FHA’s insurance fund. (Recommendation 4) The FHA Commissioner should develop and implement procedures for conducting on-site reviews of HECM servicers, including a risk-rating system for prioritizing and determining the frequency of reviews. (Recommendation 5) The FHA Commissioner should work with CFPB to complete an agreement for sharing the results of CFPB examinations of HECM servicers with FHA. (Recommendation 6) The CFPB Director should work with FHA to complete an agreement for sharing the results of CFPB examinations of HECM servicers with FHA. (Recommendation 7) The FHA Commissioner should collect and record consumer inquiries and complaints in a manner that facilitates analysis of the type and frequency of the issues raised. (Recommendation 8) The FHA Commissioner should periodically analyze available internal and external consumer complaint data about reverse mortgages to help inform management and oversight of the HECM program. (Recommendation 9) We provided HUD and CFPB with a draft of this report for review and comment. HUD provided written comments, which are reproduced in appendix V, that communicate FHA’s response to the report. CFPB’s written comments are reproduced in appendix VI. CFPB said that it did not object to our recommendation to complete an agreement for sharing the results of CFPB examinations of HECM servicers with FHA (recommendation 7) and that it would work to complete such an agreement with FHA. FHA agreed with six of our eight recommendations and neither agreed nor disagreed with the remaining two. Recommendation 1. FHA agreed with our recommendation to improve HECM termination data and said it would convene a working group to update the HERMIT system and User Guide and develop clear directions for HECM servicers to record termination reasons in HERMIT. Recommendation 2. Regarding our recommendation on HECM performance indicators and program evaluation, FHA agreed that periodic review and reporting of HECM performance indicators is critically important and said it would work to expand its reporting to include the level of foreclosure prevention activity. However, FHA added that there were no HECM metrics for early default or delinquency rates, as those measures are linked to the amortizing nature of forward mortgages. We agree that early default and delinquency rates are not suitable metrics for HECMs, and our draft report did not suggest that they are. Our report focuses on metrics that would be pertinent to HECMs and that would provide additional insight into HECM loan performance. These include the percentage of HECM terminations due to borrower defaults, the proportion of active HECMs with delinquent property charges, and the amount of funds servicers have advanced on behalf of borrowers. We revised the recommendation in our final report to more specifically describe the types of performance indicators that FHA should establish and report on. In addition, FHA disagreed with a statement in our draft report that its evaluation of the HECM program has been limited. FHA said it engages in robust HECM program evaluation and cited an example that led to recent changes in FHA’s appraisal practices for HECMs. While our draft report described the change in FHA’s appraisal practices, we updated our final report to include reference to the FHA study that prompted the appraisal change. However, we maintain that FHA’s evaluation of the HECM program has been limited because the last comprehensive program evaluation was completed 19 years ago and FHA has not assessed the impact of HECM foreclosure prevention options. Recommendations 3 and 4. FHA agreed with our recommendations to develop analytic tools for monitoring HECM loan outcomes and to evaluate its foreclosure prioritization process for FHA-assigned loans. Regarding the latter, FHA said that it is evaluating alternative disposition options to reduce the number of loans that must go through foreclosure and that it would take steps to evaluate the impact of its prioritization process to assist in future decision-making. Recommendation 5. FHA agreed with our recommendation to develop and implement procedures for conducting on-site reviews of HECM servicers, including a risk-rating system for prioritizing and determining the frequency of reviews. As noted in our draft report, FHA said it is in the process of updating procedures for on-site reviews and plans to implement them in fiscal year 2020. FHA disagreed with a statement in our draft report that FHA’s oversight of HECM servicers is limited. FHA said the HECM servicing community is small, which allows the agency to maintain regular communication with HECM servicers, including through training sessions and industry working group meetings. Our draft report acknowledged FHA’s communications with servicers, but these activities are not a substitute for in-depth compliance reviews of servicers’ operations. As our draft report stated, FHA has not conducted on-site HECM servicer reviews since fiscal year 2013. Given the 5-year lapse in FHA’s use of this key oversight tool, we maintain that FHA’s oversight of HECM servicers has been limited. Recommendation 9. FHA agreed with our recommendation to periodically analyze internal and external consumer complaint data about reverse mortgages. FHA said it is expanding its data and reporting capabilities as part of an information technology modernization initiative. FHA also said that routing consumer inquiries through the FHA Resource Center should improve data collection and analysis. FHA did not explicitly agree or disagree with our recommendations to work with CFPB to complete an agreement for sharing examination results and to collect and record consumer inquiries and complaints in a manner that facilitates analysis (recommendations 6 and 8, respectively). FHA said it would explore opportunities to coordinate with CFPB where appropriate. FHA also said that routing inquiries through the FHA Resource Center would help identify common issues, track servicer performance, and inform policy decisions. Fully implementing our recommendations will help ensure that FHA has the information it needs to effectively oversee the HECM program. We are sending copies of this report to the Secretary of the Department of Housing and Urban Development, the Director of the Consumer Financial Protection Bureau, appropriate congressional committees, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or CackleyA@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. This report examines issues related to reverse mortgages made under the Home Equity Conversion Mortgage (HECM) program administered by the Department of Housing and Urban Development’s (HUD) Federal Housing Administration (FHA). The Consumer Financial Protection Bureau (CFPB) also plays a role in overseeing reverse mortgages, including HECMs. Our objectives were to examine (1) what FHA data show about HECM terminations, servicer advances, and the use of foreclosure prevention options; (2) FHA’s assessment and monitoring of HECM portfolio performance, servicer advances, and foreclosure prevention options; (3) FHA’s and CFPB’s oversight of HECM servicers; (4) how FHA and CFPB collect, analyze, and respond to consumer complaints about HECMs; and (5) how and why the market for HECMs has changed in recent years. To address all of our objectives, we reviewed relevant laws, regulations, and requirements, such as HECM authorizing legislation, the Reverse Mortgage Stabilization Act of 2013, FHA regulations, and mortgagee letters governing the HECM program. We also interviewed FHA and CFPB officials and staff from other relevant HUD offices such as the Office of Policy Development and Research and the Office of General Counsel. We reviewed FHA’s annual reports to Congress on the financial status of the Mutual Mortgage Insurance Fund, actuarial reports on the HECM portfolio, and FHA’s annual management reports. We also reviewed our prior reports and reports by HUD’s Office of Inspector General about the HECM program. Additionally, we identified the largest HECM servicers using FHA data on the number of loans serviced as of the end of fiscal year 2018. We found that five companies serviced more than 99 percent of the HECM portfolio (excluding loans assigned to FHA, which are serviced by an FHA contractor) as of the end of fiscal year 2018. We developed a questionnaire to solicit information applicable to our objectives from these five servicers. We took steps to verify the information gathered in the questionnaire. We reviewed responses for completeness and held teleconferences with each HECM servicer to discuss, clarify, and amend responses. Where possible, we corroborated servicers’ responses with information or analysis from other sources, such as our analysis of FHA loan-level data or FHA documents. We use summary statements and illustrative examples from these questionnaires and our interviews with the five servicers throughout the report. We also interviewed representatives from five legal aid organizations representing HECM borrowers in the states of California, Florida, New York, Texas, and Washington. We selected these states because they had the highest number of HECM originations in the past decade and because they provided some geographic diversity; the five states span the West (California), Northwest (Washington), Northeast (New York), Southeast (Florida), and South (Texas). We selected the specific legal aid organizations within those states because they represented a large number of HECM borrowers, according to organization representatives. We conducted semistructured interviews with organization representatives that included questions on the top consumer protection issues facing HECM borrowers, how recent HECM program changes may have helped borrowers delay or avoid foreclosure, and characteristics of HECM borrowers that may affect their ability to file consumer complaints. We use summary statements and illustrative examples from these interviews throughout the report. To address the first objective, we analyzed FHA data to determine the number of and reasons for HECM terminations, the amounts of servicer advances, and the number of borrowers approved for selected foreclosure prevention options (for example, repayment plans). We used data from the Home Equity Reverse Mortgage Information Technology (HERMIT) system, which FHA adopted in fiscal year 2013. FHA provided us a HERMIT case detail table from its Single Family Data Warehouse that contained loan-level information as of the end of fiscal year 2018. We separately obtained several ad hoc HERMIT reports from FHA’s HERMIT system contractor, as described below. For some of our analyses, we merged data from the ad hoc reports with data from the case detail table using the unique FHA case number for each HECM. Unless otherwise noted, we analyzed data for the 5-year period spanning fiscal years 2014–2018. We assessed the reliability of data from the HERMIT system by reviewing FHA documentation about the data system and data elements. For example, we reviewed the HERMIT User Guide and notes on HERMIT system updates. Additionally, we interviewed FHA and contractor staff knowledgeable about the HERMIT system and data to discuss interpretations of data fields and trends we observed in our analyses. We also conducted electronic testing, including checks for duplicate loans, outliers, missing data fields, and erroneous values. Where appropriate, we removed from our analyses any loans missing an endorsement (insurance approval) date as well as cases with erroneous values. When possible, we corroborated our analyses with external reports such as FHA’s annual reports to Congress, management reports, and production reports. Based on these steps, we determined the data we used from the HERMIT system were sufficiently reliable for summarizing trends and generating descriptive statistics for HECM terminations, servicer advances, and selected foreclosure prevention options over the 5-year period. We analyzed FHA loan-level data from the HERMIT system to determine the total number of terminated HECMs and reasons for terminations by fiscal year. We first identified terminations occurring in fiscal years 2014– 2018 using data fields for case status and termination date (see table 5). We then removed any terminated loans that had previously been assigned to FHA (16,008) using the data field that records the date FHA accepted assignment of the loan. We removed these loans because FHA officials told us the agency generally does not foreclose on FHA-assigned HECMs that default and keeping them in the analysis would have resulted in understating the proportion of terminations stemming from defaults. Accordingly, the denominator for our terminations analysis was 256,147 loans (272,155 total terminations minus the 16,008 loans previously assigned to FHA). We then identified the reported termination reasons for the 256,147 loans. We analyzed loan-level data from the HERMIT system to identify the number of loans that fell into various termination reason categories. To identify terminations stemming from a HECM becoming due and payable, we used data from two reports that we obtained from FHA’s HERMIT system contractor: the Default Key Dates Report and the Due and Payable Delinquency Report. From these reports, we identified the number of terminations due to a borrower’s death, conveyance of title, default due to unpaid property charges, default due to failure to meet occupancy or residency requirements, and default due to failure to keep the home in good repair. To identify terminations stemming from repayment, refinancing, moving, or other (undetermined) reasons, we used information on case substatus from the HERMIT case detail table from the Single Family Data Warehouse. Our undetermined reasons category included loans for which the case substatus either was labeled “terminate-other” or showed how the debt was satisfied (such as through a deed-in-lieu of foreclosure, foreclosure, or short sale) rather than providing a termination reason. For the full results of our terminations analysis, see appendix III. We analyzed servicer advances to HECM borrowers using data from an ad hoc HERMIT report we requested from FHA’s HERMIT system contractor. We analyzed the data to determine the amounts and types of servicer advances in fiscal years 2014 through 2018 for terminated HECMs. We distinguished between servicer advances for unpaid property charges and servicer advances for other costs. Examples of the latter are attorney, trustee, and appraisal fees typically incurred during the foreclosure process. For each year and for the 5-year period as a whole, we calculated total servicer advances and the amount and percentage of advances for property charges and for other costs. Additionally, we distinguished between servicer advances for unpaid property charges before and after a HECM borrower’s death using the date of death of the last surviving borrower in HERMIT. This allowed us to determine the amount of servicer advances for unpaid property charges on behalf of living borrowers. We calculated the total amount of these advances over the 5-year period as well as the mean, median, and 25th and 75th percentile values. We also calculated the number and percentage of loans for which property charge advances on behalf of living borrowers were less than $2,000 (the threshold for one of FHA’s foreclosure prevention options). We analyzed data from HERMIT on the use of selected foreclosure prevention options—repayment plans and at-risk extensions—for borrowers who defaulted because of unpaid property charges. We analyzed data from April 2015 (the effective date of FHA’s current repayment plan and at-risk extension policies) through fiscal year 2018. To conduct the analysis of repayment plans, we used the HERMIT Due and Payable Delinquency Report noted previously, which includes data fields for loan default status and the dates borrowers were approved for a repayment plan. We calculated the percentage of borrowers with property charge defaults who were approved for repayment plans during the period examined. To conduct the analysis of at-risk extensions, we requested an ad hoc report from FHA’s HERMIT system contractor showing whether and when borrowers had been approved for at-risk extensions and appended it to the default status within the Due and Payable Delinquency Report using FHA case numbers. We calculated the percentage of borrowers with property charge defaults who were approved for at-risk extensions during the period examined. We also reviewed and summarized information that FHA provided us from HERMIT on nonborrowing spouses who applied for mortgagee optional election assignments from June 2015 (the effective date of FHA’s mortgagee optional election assignment policy) through fiscal year 2018. FHA provided information on the number of requested, approved, and denied mortgagee optional election assignments during that period. We also reviewed documentation from FHA and interviewed agency officials about the mortgagee optional election assignment process and reasons for denials. For the denied mortgagee optional election assignments, we reviewed information that FHA provided us from HERMIT on the current status of the associated loans as of May 31, 2019. For example, for the denied mortgagee optional election assignments, FHA determined whether the loan had been terminated as of that date. For those that had terminated, we summarized whether the debt was paid off or whether the debt was satisfied because of a foreclosure, deed-in-lieu of foreclosure, or short sale. To address the second objective, we reviewed agency reports and interviewed agency officials to determine how the agency assesses the performance of the HECM program, including the use of any performance indicators or program evaluations. For example, we reviewed HUD’s strategic plan for fiscal years 2018–2022 and its most recent annual performance report to identify any goals and performance indicators related to the HECM program. Additionally, we reviewed program evaluations completed for the HECM program. We also interviewed FHA and HUD Office of Policy Development and Research officials about previous program evaluations and HUD’s plans for forthcoming evaluations of the HECM program. We compared FHA’s practices against leading practices identified in our previous work on assessing program performance and against Office of Management and Budget (OMB) policies and procedures on managing federal credit programs (OMB Circular A-129). Additionally, we reviewed FHA documents and interviewed FHA officials concerning the agency’s internal reporting and analysis of the HECM portfolio. For example, we reviewed examples of regular and ad hoc reports FHA received from its HERMIT system contractor. These internal reports contained information on HECM origination, assignment, and termination activity and HECM defaults. We interviewed FHA officials to understand the purpose of the reports, when they were developed, and how agency management uses them. We compared FHA’s internal reporting practices to OMB Circular A-129 on reporting mechanisms and formats for federal credit programs. To address the third objective, we reviewed FHA and CFPB policies and procedures for overseeing HECM servicers and interviewed agency staff with oversight responsibilities. To assess the extent to which FHA oversees HECM servicers’ compliance with servicing requirements, we requested information on the number of on-site monitoring reviews of HECM servicers that FHA completed from fiscal years 2010 through 2019. We also reviewed corrective actions FHA can take to address noncompliance. We reviewed and summarized a nongeneralizable sample of three reports from on-site servicer reviews FHA conducted in fiscal year 2013, the most recent year in which FHA had completed a review. Additionally, we interviewed the director of FHA’s Quality Assurance Division, which is responsible for conducting on-site reviews of FHA-approved lenders and servicers, about the division’s past practices for reviewing HECM servicers and plans for future reviews. We compared FHA’s practices and plans to criteria in OMB Circular A-129 regarding the frequency, targeting methodology, and other aspects of on-site lender and servicer reviews. Further, we interviewed FHA officials about the extent of information sharing between FHA and CFPB on HECM servicer oversight. To examine CFPB’s oversight of HECM servicers, we reviewed CFPB’s reverse mortgage examination procedures and the examinations completed under those procedures as of fiscal year 2018. We also reviewed CFPB’s methodology for selecting reverse mortgage servicers for examination and documentation on its plans for future examinations. We reviewed CFPB’s examination findings and corrective actions as of August 2019. We interviewed CFPB officials about the examination process and agency efforts to share examination results with FHA. We reviewed statutes and regulations related to CFPB’s authority to share the results of its examinations, and we compared CFPB’s information-sharing efforts with FHA against practices for interagency collaboration we identified in previous work. To address our fourth objective, we analyzed CFPB data on reverse mortgage consumer complaints from the bureau’s online website, called the Consumer Complaint Database. The database includes information provided by consumers on their location (state), the company they are complaining about, and the nature of their complaint. For example, consumers can submit narratives describing their complaints about reverse mortgage lenders or servicers. Because CFPB had published an analysis of reverse mortgage consumer complaints using data from calendar years 2011 through 2014, we analyzed reverse mortgage complaints and narratives received by the bureau from calendar years 2015 through 2018. We analyzed all 2,472 complaints filed in those 4 years to determine the number of complaints by year, state, submission method (for example, internet, phone, or email), and company. For the analysis by submission method, we compared the results to those for complaints about all types of mortgages filed during the same period. To identify patterns in consumer-described issues about reverse mortgages, we reviewed a generalizable sample of 100 complaint narratives and categorized these complaints by topic. For this analysis, two independent reviewers read the complaints and categorized them into predetermined topics based on their content. We used nine complaint issue topics, including complaints where the consumer (or someone complaining on behalf of the consumer) said he or she (1) was at risk of foreclosure or in foreclosure; (2) was charged unfair interest rates, fees, or costs; (3) experienced problems after the loan was transferred to a new servicer; (4) had issues with, or defaulted as a result of, property taxes, insurance, or other property charges; (5) experienced poor communication on a servicing or lending issue; (6) had an issue involving occupancy requirements; (7) had concerns or issues involving the management of the estate after the borrower died or left the property; (8) had difficulties gaining approval for a mortgagee optional election assignment or recognition of a nonborrowing spouse; or (9) experienced problems during loan origination. If a complaint narrative in our sample did not contain enough information or was not clear enough to determine a complaint topic, we replaced it with another randomly selected complaint narrative. In cases where the two reviewers categorized a complaint differently, a third independent analyst read the complaint narrative and adjudicated the difference to place the complaint in a topic category. We calculated confidence intervals for these categories at the 95 percent confidence level. We determined that the CFPB data were sufficiently reliable for the purposes described above by reviewing CFPB documentation and reports from CFPB’s Office of Inspector General on CFPB’s consumer complaint database and by interviewing CFPB officials about our interpretation of data fields. Also, we interviewed CFPB officials about their collection, analysis, and use of the consumer complaint data. To determine the extent to which FHA collects consumer inquiries and complaints about HECMs, we reviewed the HECM-related calls received by FHA’s National Servicing Center and the FHA Resource Center from calendar years 2015 through 2018. We calculated the total number of HECM-related calls each center received over that period. The data from both centers included fields to capture a description of the issue raised by the caller. However, unlike CFPB’s consumer complaint data, the information in the issue description was recorded by FHA customer service staff (rather than the complainants themselves) and did not differentiate between inquiries and complaints. We determined there was not enough information in these descriptions to perform an analysis similar to the one we performed on CFPB’s consumer complaints. Both the National Servicing Center and the FHA Resource Center record the reasons for calls. However, neither entity records this information in a consistent or standardized way that would allow for analysis. For example, the data we reviewed from the National Servicing Center included about 100 reasons. Additionally, we reviewed CFPB and FHA policies and procedures for collecting and addressing consumer complaints and interviewed officials on how consumer complaints were incorporated into their oversight of HECM servicers. We interviewed officials from both agencies about their collection and use of customer complaint data. We also interviewed CFPB and FHA officials about the extent to which they share consumer complaint data or access and use the other agency’s data. Finally, we compared CFPB’s and FHA’s efforts against federal internal control standards for using quality information and against approaches we identified in prior work for enhancing collaboration across agencies. To address our fifth objective, we analyzed FHA data on HECM originations from calendar years 1989 through 2018 to identify any trends in HECM program activity. Additionally, using FHA and Census Bureau data, we calculated HECM take-up rates—the ratio of HECM originations to eligible senior homeowners—from calendar years 2000 through 2017. We also developed an econometric model to examine, to the extent possible, factors affecting HECM take-up rates from calendar years 2000 through 2016 (the last year we could include in the model due to data constraints). Following the existing research literature, we hypothesized that HECM loan originations could be affected by several demand- and supply-related factors that could be represented by demographic and socioeconomic characteristics, housing market conditions, and product features. Accordingly, our model used a variety of data from FHA, the Census Bureau, the Federal Housing Finance Agency, and other sources. For a detailed description of our econometric model—including the model specification, factors used, data sources, and results—and a list of selected studies we consulted to develop the model, see appendix II. We also reviewed relevant literature and interviewed academic and HUD economists about FHA policy changes and behavioral and structural factors (for example, consumers’ perception of reverse mortgages) that we could not account for in our econometric model but that may influence HECM take-up rates. These individuals included three academic economists who have conducted extensive research on reverse mortgages and economists from FHA and HUD’s Office of Policy Development and Research. We present summary information about these factors in this report. We conducted this performance audit from July 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We developed an econometric model to examine, to the extent possible, factors associated with Home Equity Conversion Mortgage (HECM) take- up rates—the ratio of HECM originations to eligible senior homeowners— using data from calendar year 2000 through 2016. Take-up rates provide an indication of how popular HECMs are among the population of senior homeowners. A number of factors may have affected the take-up rates over this period. For instance, it has been asserted that demand for HECMs would be high for elderly people that are house-rich but cash- poor, but behavioral factors such as their desire to leave a bequest could limit demand. Also, the limited number of large, well-known lenders could constrain supply of HECMs. Furthermore, several FHA policy changes to the HECM program may have affected the number of loan originations. Following the existing literature, we hypothesized that HECM loan originations could be affected by several demand- and supply-related factors that could be represented by demographic and socioeconomic characteristics, housing market conditions, product features, program policy changes, and behavioral and structural factors. The general specification of the model we used, which is a quasi-reduced form of the net effect of demand and supply factors on HECM take-up rates, is as follows: Yit = θ + α + γ + Xit β + εit. Y is the dependent variable, the take-up rate, representing the ratio of HECM originations to eligible senior homeowners in state (i) in year (t). An eligible senior homeowner is an owner-occupied householder aged 65 or older. Both α and γ control, respectively, for state-specific (but time-invariant) and year-specific (but state-invariant) observable and unobserved factors. They help to minimize omitted variable bias that could be caused by excluding time-invariant or state-invariant variables. The latter, which are year fixed-effects (that is, variables that change over time but are constant across the states), would pick up average differences in take-up rates over the years. These factors would include changes in HECM program policies and market conditions over time, such as the exit of large, well-known HECM lenders or investors. In general, using the year fixed-effects precluded the estimation of the impact of variables that are state-invariant (for example, interest rates).The state fixed-effects are used to control for average differences in take-up rates across the states (that is, variables that differ across the states but are constant over time). These effects would include regulatory variations across states. The vector X captures measured variables represented by demographic and socioeconomic characteristics, housing market conditions, and product features that vary across states and over time. Given that the time-invariant and state-invariant factors would be accounted for by the state fixed-effects and year fixed-effects, respectively, the measured variables capture how changes in these variables within states (that is, over time) could affect take-up rates. θ is the constant term. ε, the regression error term, represents random and other unobserved factors that could vary across the states and over time, such as random changes in risk behavior of HECM borrowers and lenders. It also captures errors due to misspecification and measurement. The data sources for our analysis are as follows: Census Bureau. The data include demographic, socioeconomic, and housing characteristics in geographic areas. The data are from the Integrated Public Use Microdata Series National Historical Geographic Information System (IPUMS NHGIS) for 2000; 1-year American Community Survey data from the American FactFinder for 2005–2009, and 1-year American Community Survey data from IPUMS NHGIS for 2010–2016. We interpolated the data for 2001– 2004 using all data available for the other years: 2000 and 2005– 2016. All the data are for seniors aged 65 years or older and at the state level. Federal Housing Finance Agency. House price indexes at the state level, 2000–2016. Federal Reserve Bank of New York. Federal Reserve Bank of New York Consumer Credit Panel/Equifax: Mortgage debt balances of seniors 62 years or older, state level, 2003–2016. Survey of Consumer Finances: Triennial data on family net worth, national level, 2000–2016. Federal Reserve Bank of St. Louis’s Federal Reserve Economic Data. Consumer price index for all urban consumers, national level, 2000–2016. Effective federal funds rate, national level, 2000–2016. Federal Housing Administration (FHA). HECM loan-level data from the Single Family Data Warehouse, available yearly, 2000–2017. The data include when the loan was endorsed by FHA, property location, appraised home value, and maximum claim amount. The list of potential explanatory variables we used in the model is provided below. The data are measured at the state level and are available from 2000 through 2016 (unless indicated otherwise). Also, the variables are for senior householders, aged 65 or older (unless indicated otherwise). All monetary values are in 2016 dollars using the Consumer Price Index for All Urban Consumers. The data sources are indicated in brackets (see the data sources above for details). Demographic and socioeconomic characteristics [Census Bureau]. Fraction 75 years or older in occupied housing units. Fraction of senior householders who are married or those who are unmarried females. Fraction African American or Hispanic. Fraction of individuals 65 years or older with high school education or some college education, or with college, graduate, or professional degree. Fraction in the labor force: the ratio of the labor force (the employed and the unemployed) to civilian noninstitutionalized adult population (65 years or older). Fraction in poverty. Median household income (natural logarithm). Ratio of family net worth of individuals 65 years or older to house value. Net worth is measured as the difference between families’ gross assets and liabilities using triennial data at the national level . House value is measured as the ratio of aggregate house value to number of owner-occupied housing units. Housing market conditions. House price changes : o House price growth: 5-year intervals prior to the observation. o House price volatility: standard deviation of annual house price percent change in the 5 years prior to the observation. Effective federal funds rate (percent). [Federal Reserve Bank of Home equity per senior homeowner (natural logarithm), 1-year lag. Home equity is measured as the aggregate house value of owner-occupied housing units minus total mortgage debt. Total mortgage debt comprises aggregate mortgage, home equity loan, and home equity line of credit balances of individuals 62 or older (2003–2016). [Census Bureau; Federal Reserve Bank of New York/Equifax] Ratio of individuals aged 62 or older with home equity loan to senior homeowners, 1-year lag (2003–2016). [Federal Reserve Bank of New York/Equifax; Census Bureau] Ratio of individuals aged 62 or older with home equity line of credit to senior homeowners, 1-year lag (2003–2016). [Federal Reserve Bank of New York/Equifax; Census Bureau] Fraction of owner-occupied housing units with ratio of selected monthly housing costs to household income greater than or equal to 35 percent. Product features. FHA loan limit: proportion of HECM loans in a state and year for which the appraised home value is more than the maximum claim amount; that is, the FHA loan limit is binding. The maximum claim amount equals the minimum of the appraised home value and the FHA loan limit. Although we did not directly include other variables that could affect HECM take-up rates in our model partly due to lack of data, we included year fixed-effects and state-fixed effects to minimize omitted variables problem associated with state-invariant variables and time-invariant variables, respectively. These included several FHA policy changes to the HECM program and behavioral and structural factors, as discussed earlier in this report. We used a state as the geographic area instead of a smaller area, such as ZIP code. The data on HECM originations are available at the household (or family) level from FHA. However, the factors used in the model (demographic and socioeconomic characteristics and housing market conditions) are generally available at the state level or at the ZIP code level from the Census Bureau and other sources. There are advantages and disadvantages to using state-level or ZIP-code-level data. Given the low HECM take-up rates (see fig. 7 earlier in this report), using ZIP-code data would generally imply very low, if not zero, take-up rates across a large number of ZIP codes, which would make it harder to identify effects from our model. Also, not all of the data for the factors used in the model are available for every ZIP code with a HECM origination—including the home equity extraction variables—which would lead to exclusion of some areas, resulting in potential sample-selection bias. On the other hand, using ZIP code-level data could allow for more heterogeneity in certain states, and certain variables such as house price changes when measured at the ZIP code level could be closer to what the homeowner experiences. We decided to use state-level data because of our concern for potential sample-selection bias and the quality of data at the ZIP code level, although using state-level data could limit heterogeneity in the data across geographic areas. We estimated panel data of state-year observations of the model specified above using fixed-effects estimation. Because of data limitations with some of the key variables—home equity and home equity extraction via loans or lines of credit—and because we used a 1-year lag of these variables, we estimated the model from 2004 through 2016. We also excluded the District of Columbia, which was an outlier, with a take- up rate that was 4.5 times the national average. The list of the variables we used and the estimation results are provided in tables 6 and 7, respectively, at the end of this appendix. The standard fixed-effects estimates are reported in column 1 (the base model) of table 7. We also report fixed-effects estimates that account for spatial and temporal dependence in columns 2 through 4—column 2 estimates the base model, column 3 excludes the variables for home equity extraction from the base model, and column 4 excludes the year fixed-effects from the base model. We focused on these estimates because spatial correlations may be present as states are likely to be subject to both observable and unobservable common disturbances, and failure to account for these would yield inconsistent estimates of the standard errors. Our econometric estimates indicated that several demographic and socioeconomic characteristics and housing conditions are associated with take-up rates, using data across states from 2004 through 2016. The results discussed below, which are based primarily on the estimates in column 2 of table 7, are statistically significant at the 10, 5, or 1 percent levels or lower. Because the fixed-effects technique controls for the effects of both observable and unobservable factors that vary across states (but are time-invariant), the estimates of the measured effects are for only within-state variations and the results are interpreted accordingly. House price changes. The interaction term for house price growth and house price volatility is positive and significant at the 1 percent level. This implies that within states, take-up rates were higher when house price growth was large and when there was a history of house price volatility compared to either relatively low house price appreciation or stable house prices. This result is consistent with senior homeowners using reverse mortgages to insure against house price declines, which is supported by the positive and significant effects of the house price volatility by itself. On the other hand, the weak significance of house price growth by itself (at the 10 percent level) provides only modest support for senior homeowners using reverse mortgages purely to extract home equity. Home equity. Within states, take-up rates were higher when home equity of senior homeowners was high, significant at the 1 percent level. Fractions of senior homeowners with a home equity loan or home equity line of credit. Within states, take-up rates were higher when the fractions of senior homeowners with a home equity loan or home equity line of credit were high, significant at the 1 percent and 10 percent levels, respectively. Because these loans were outstanding as of the prior year, it is likely that borrowers used HECMs to pay them off. Fraction of owner-occupied housing units with ratio of housing costs to household income greater than or equal to 35 percent. Within states, take-up rates were higher when the ratio of housing costs to household income was high, significant at the 1 percent level. Fractions of seniors with high school or college education. Within states, take-up rates were higher when the fractions of seniors with high school or college education were high, significant at the 1 percent and 10 percent levels, respectively. Median household income. Within states, take-up rates were higher when incomes of senior households were high, significant at the 5 percent level. Fraction of senior households who were married. Within states, take-up rates were lower when the fraction of married senior households was high, significant at the 10 percent level. Fraction of homes in states with binding FHA loan limit. Although the effect was generally not statistically significant, the effect of the FHA loan limit on take-up rates was negative. We estimated other specifications of our model to test the robustness and reasonableness of our results. The alternative specifications, described below, yielded estimates similar to those of our original model. We estimated the model excluding the variables for home equity loans and home equity lines of credit, which are alternative channels of home equity extraction, because they could be endogenous (see column 3 of table 7). We estimated the model excluding the year fixed-effects (see column 4 of table 7). We estimated the model using the number of senior housing units (instead of senior homeowners) within a state to normalize the number of HECMs in order to account for nonhomeowners who might become homeowners. We note the following caveats and limitations of our study: We were not able to include some factors that could affect HECM take-up rates, including FHA program policy changes and behavioral and structural factors previously discussed in this report. Some of our estimates could be different if we used areas smaller than a state as the units of observation, such as ZIP codes or counties. The estimates represent the average effects for all states and for all periods we analyzed, but the effects could differ for specific states or specific periods. Our analysis pertains to the period that we analyzed and may not be generalizable to other periods. To help develop our HECM take-up rate model, we consulted the following studies. 1. Banks, James, Richard Blundell, Zoe Oldfield, and James P. Smith. “Housing Price Volatility and Downsizing in Later Life.” National Bureau of Economic Research Working Paper 13496. Cambridge, Mass.: National Bureau of Economic Research, October 2007. Accessed April 30, 2019. http://www.nber.org/papers/w13496. 2. Chatterjee, Swarn. “Reverse Mortgage Participation in the United States: Evidence from a National Study.” International Journal of Financial Studies, vol. 4, no. 5 (2016): pp. 1–10. 3. Consumer Financial Protection Bureau. Reverse Mortgages: Report to Congress. Washington, D.C.: June 28, 2012. 4. Davidoff, Thomas. Reverse Mortgage Demographics and Collateral Performance. February 25, 2014. Accessed November 19, 2018. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2399942. 5. Davidoff, Thomas. “Supply Constraints Are Not Valid Instrumental Variables for Home Prices Because They Are Correlated With Many Demand Factors.” Critical Finance Review, vol. 5, no. 2 (2016): pp. 177–206. 6. Davidoff, Thomas, Patrick Gerhard, and Thomas Post. Reverse Mortgages: What Homeowners (Don’t) Know and How It Matters. October 24, 2016. Accessed November 19, 2018, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2528944. 7. Driscoll, John C., and Aart C. Kraay. “Consistent Covariance Matrix Estimation with Spatially Dependent Panel Data.” Review of Economics and Statistics, vol. 80 (1998): pp. 549–560. 8. Golding, Edward, and Laurie Goodman, “To Better Assess the Risk of FHA Programs, Separate Reverse and Forward Mortgages.” Urban Wire (blog), Urban Institute. November 29, 2017. Accessed August 14, 2019. https://www.urban.org/urban-wire/better-assess-risk-fha- programs-separate-reverse-and-forward-mortgages. 9. Goodman, Laurie, Karan Kaul, and Jun Zhu. What the 2016 Survey of Consumer Finances Tells Us about Senior Homeowners. Washington, D.C.: Urban Institute, November 2017. 10. Haurin, Donald, Chao Ma, Stephanie Moulton, Maximilian Schmeiser, Jason Seligman, and Wei Shi. “Spatial Variation in Reverse Mortgages Usage: House Price Dynamics and Consumer Selection.” Journal of Real Estate Finance and Economics, vol. 53 (2016): pp. 392–417. 11. Integrated Financial Engineering, Inc. “Appendix E: HECM Demand Model” in HECM Demand Model Actuarial Review of the Federal Housing Administration Mutual Mortgage Insurance Fund HECM Loans For Fiscal Year 2015. Prepared at the request of the Department of Housing and Urban Development. November 16, 2015. 12. Kaul, Karan, and Laurie Goodman. Seniors’ Access to Home Equity: Identifying Existing Mechanisms and Impediments to Broader Adoption. Washington, D.C.: Urban Institute, February 2017. 13. Lucas, Deborah. “Hacking Reverse Mortgages.” (Working paper, October 26, 2015). Accessed June 14, 2019. http://mitsloan.mit.edu/shared/ods/documents/?DocumentID=4596. 14. Mayer, Christopher J., and Katerina V. Simons. “Reverse Mortgages and the Liquidity of Housing Wealth.” Journal of the American Real Estate and Urban Economics Association, vol. 22, no. 2 (1994): pp. 235–255. 15. Mummolo, Jonathan, and Erik Peterson. “Improving the Interpretation of Fixed Effects Regression Results.” Political Science Research and Methods, vol. 6 (2018): pp. 1–7. 16. Moulton, Stephanie, Donald R. Haurin, and Wei Shi. “An Analysis of Default Risk in the Home Equity Conversion Mortgage (HECM) Program.” Journal of Urban Economics, vol. 90 (2015): pp. 17–34. 17. Moulton, Stephanie, Cazilia Loibl, and Donald Haurin. “Reverse Mortgage Motivations and Outcomes: Insights from Survey Data.” Cityscape: A Journal of Policy Development and Research, vol. 19, no. 1 (2017): pp. 73–97. 18. Moulton, Stephanie, Samuel Dodini, Donald Haurin, and Maximilian Schmeiser. “Seniors’ Home Equity Extraction: Credit Constraints and Borrowing Channels.” May 20, 2019. Accessed August 12, 2019. https://ssrn.com/abstract=2727204. 19. Nakajima, Makoto, and Irina A. Telyukova. “Reverse Mortgage Loans: A Quantitative Analysis.” The Journal of Finance, vol. 72, no. 2 (2017): pp. 911–949. 20. Redfoot, Donald L., Ken Scholen, and S. Kathi Brown. Reverse Mortgages: Niche Product or Mainstream Solution? Report on the 2006 AARP National Survey of Reverse Mortgage Shoppers. Washington, D.C.: December 2007. 21. Shan, Hui. “Reversing the Trend: The Recent Expansion of the Reverse Mortgage Market.” Real Estate Economics, vol. 39, no. 4 (2011): pp. 743–768. 22. Warshawsky, Mark J. “Retire on the House: The Possible Use of Reverse Mortgages to Enhance Retirement Security.” The Journal of Retirement, vol. 5, no. 3 (2018): pp. 10–31. In addition to the contact named above, Steve Westley (Assistant Director), Beth Faraguna (Analyst in Charge), Steven Campbell, William Chatlos, Holly Hobbs, John Karikari, Matthew Levie, Risto Laboski, Marc Molino, Jennifer Schwartz, Tyler Spunaugle, and Khristi Wilkins made key contributions to this report.", "summary": "Reverse mortgages allow seniors to convert part of their home equity into payments from a lender while still living in their homes. Most reverse mortgages are made under FHA's HECM program, which insures lenders against losses on these loans. HECMs terminate when a borrower repays or refinances the loan or the loan becomes due because the borrower died, moved, or defaulted. Defaults occur when borrowers fail to meet mortgage conditions such as paying property taxes. These borrowers risk foreclosure and loss of their homes. FHA allows HECM servicers to offer borrowers foreclosure prevention options. Most HECM servicers are supervised by CFPB. GAO was asked to review HECM loan outcomes and servicing and related federal oversight efforts. Among other objectives, this report examines (1) what FHA data show about HECM terminations and the use of foreclosure prevention options, (2) the extent to which FHA assesses and monitors the HECM portfolio, and (3) the extent to which FHA and CFPB oversee HECM servicers. GAO analyzed FHA loan data and FHA and CFPB documents on HECM servicer oversight. GAO also interviewed agency officials, the five largest HECM servicers (representing 99 percent of the market), and legal aid groups representing HECM borrowers. The vast majority of reverse mortgages are made under the Federal Housing Administration's (FHA) Home Equity Conversion Mortgage (HECM) program. In recent years, a growing percentage of HECMs insured by FHA have ended because borrowers defaulted on their loans. While death of the borrower is the most commonly reported reason why HECMs terminate, the percentage of terminations due to borrower defaults increased from 2 percent in fiscal year 2014 to 18 percent in fiscal year 2018 (see figure). Most HECM defaults are due to borrowers not meeting occupancy requirements or failing to pay property charges, such as property taxes or homeowners insurance. Since 2015, FHA has allowed HECM servicers to put borrowers who are behind on property charges onto repayment plans to help prevent foreclosures, but as of fiscal year-end 2018, only about 22 percent of these borrowers had received this option. FHA's monitoring, performance assessment, and reporting for the HECM program have weaknesses. FHA loan data do not currently capture the reason for about 30 percent of HECM terminations (see figure). FHA also has not established comprehensive performance indicators for the HECM portfolio and has not regularly tracked key performance metrics, such as reasons for HECM terminations and the number of distressed borrowers who have received foreclosure prevention options. Additionally, FHA has not developed internal reports to comprehensively monitor patterns and trends in loan outcomes. As a result, FHA does not know how well the HECM program is serving its purpose of helping meet the financial needs of elderly homeowners. FHA has not conducted on-site reviews of HECM servicers since fiscal year 2013 and has not benefited from oversight efforts by the Consumer Financial Protection Bureau (CFPB). FHA officials said they planned to resume the reviews in fiscal year 2020, starting with three servicers that account for most of the market. However, as of August 2019, FHA had not developed updated review procedures and did not have a risk-based method for prioritizing reviews. CFPB conducts examinations of reverse mortgage servicers but does not provide the results to FHA because the agencies do not have an agreement for sharing confidential supervisory information. Without better oversight and information sharing, FHA lacks assurance that servicers are following requirements, including those designed to help protect borrowers. GAO makes eight recommendations to FHA to, among other things, improve its monitoring and assessment of the HECM portfolio and oversight of HECM servicers, and one recommendation to CFPB to share HECM servicer examination information with FHA. FHA and CFPB generally agreed with the recommendations.", "document_type": "gao"}
{"report": "TSA began implementing its Secure Flight program in 2009 to identify passengers who may pose security risks before boarding an aircraft. The program requires U.S. and foreign commercial aircraft operators traveling to, from, within or overflying the United States, as well as U.S. commercial aircraft operators with international point-to-point flights, to collect information from passengers and transmit it electronically to TSA. This information includes personally identifiable information, such as full name, gender, date of birth, passport information (if available), and certain non-personally identifiable information, such as itinerary information and the unique number associated with a travel record (record number locator). The Secure Flight program matches the passenger-provided personally identifiable information against federal government watchlists and other information to determine if passengers may pose a security risk and to assign them a risk category. Since January 2009, Secure Flight has matched passengers to two subsets of the Terrorist Screening Database—the No Fly List, composed of individuals who should be precluded from boarding an aircraft or entering the sterile area of a U.S. airport, and the Selectee List, composed of individuals who should receive enhanced screening prior to boarding an aircraft or entering an airport sterile area. The risk categories are not specifically communicated to the air carriers, but for each passenger Secure Flight provides responses to air carriers commensurate with the risk levels identified (e.g., an air carrier will receive a response of “inhibited” if the passenger was identified as being in the highest-risk category, or the boarding pass printed for a high-risk passenger will identify that passenger as a selectee for enhanced screening at the security checkpoint). In April 2011, in response to the December 25, 2009 attempted attack, TSA also began matching passengers to a third subset of the Terrorist Screening Database—the Expanded Selectee List—to designate known or suspected terrorists not otherwise included on the No Fly or Selectee Lists as selectees for enhanced screening. The Expanded Selectee List, in general, includes all records in the Terrorist Screening Database with a full name (first name and surname) and full date of birth not otherwise included on the No Fly or Selectee Lists. The Secure Flight system, which also screens passengers against the Silent Partner and Quiet Skies Lists, among others, results in passengers receiving one of four prescreening outcomes: Low risk (expedited screening). Passengers who are eligible for expedited screening, such as those with TSA Pre®, Unknown Risk (standard screening). Passengers who warrant standard screening, High Risk (enhanced screening). Passengers who receive enhanced screening such as a pat down and explosives trace detection, because they have been identified as matches to government watchlists, including the Selectee, Expanded Selectee, Silent Partner and Quiet Skies Lists, or Highest Risk (denied boarding). Passengers who are not permitted to board a commercial aircraft, such as passengers who are on the No Fly List or the Centers for Disease Control and Prevention Do Not Board List (see fig. 1). Secure Flight also randomly identifies passengers for enhanced screening. Although subject to the same screening measures as high risk passengers, they have not been determined to be high risk. Similarly, individuals included on the Silent Partner and Quiet Skies Lists have not been determined to be of high risk, but rather have been identified using rules based on current intelligence and other factors that may indicate an elevated risk. TSA leverages CBP information and targeting capabilities to create the Silent Partner List. Specifically, TSA leverages (1) data CBP collects regarding passengers traveling internationally (such as citizenship, passport country of issuance, and address information), and (2) CBP’s Automated Targeting System. CBP uses the Automated Targeting System to identify potentially high risk passengers arriving or departing the United States by comparing passenger information with law enforcement, intelligence, and other enforcement data using risk-based targeting scenarios and assessments. Analysts within TSA I&A’s Threat Analysis Division review intelligence to identify factors that may indicate elevated passenger risk. TSA works with CBP to create Silent Partner and Quiet Skies rules in the Automated Targeting System based on these factors. The system returns information on passengers who match with the rules and are scheduled to fly on U.S.- bound flights. TSA then omits any individuals on the Silent Partner cleared list (i.e. travelers exempted from further enhanced screening based on a specific rule) before placing the remaining passengers on the Silent Partner List. The Secure Flight program designates passengers who are on the Silent Partner List as selectees for enhanced screening for a particular international flight. In April 2012, TSA’s Quiet Skies List became fully operational. The Quiet Skies List is a subset of passengers on the Silent Partner List. Specifically, TSA identifies certain Silent Partner rules that warrant continued enhanced screening for passengers’ subsequent domestic or outbound travel after arriving in the United States. Passengers identified via these rules—the Quiet Skies rules—comprise the Quiet Skies List. Passengers matched to the Quiet Skies List are designated as selectees and receive enhanced screening on any subsequent domestic flights for a designated period of time, or for a designated number of flights, whichever comes first. After the designated time period has elapsed (or number of flights is flown), passengers’ names and identifying information are moved to a cleared list. Pursuant to the TSA Modernization Act, TSA I&A is to identify and review its Silent Partner and Quiet Skies screening rules, in coordination with DHS and TSA stakeholders, every 120 days and provide notification to these stakeholders no later than two days after making a change to a rule. Table 1 lists the DHS and TSA stakeholders TSA I&A must coordinate with under the Act. According to DHS and TSA officials, TSA has coordinated quarterly rule review meetings with DHS and TSA stakeholders since the inception of the Silent Partner and Quiet Skies programs. We reviewed documentation of the reviews that occurred from December 2018 through March 2019. The quarterly review meetings are called for in DHS’s Automated Rule Review SOP and its Quiet Skies Implementation Plan. Pursuant to the TSA Modernization Act, TSA I&A is to identify and review its screening rules in coordination with DHS and TSA stakeholders every 120 days—or at least three times a year. TSA I&A officials stated that they plan to continue convening four times a year because, given the difficulty of scheduling these large meetings, it will help them ensure they meet the 120 day requirement. Since October 2018, TSA I&A has also included representatives of DHS’s Traveler Redress Inquiry Program and the Federal Air Marshal Service in these quarterly review meetings, as required by the Act. Officials from these offices told us in August 2019 that they are still determining their role in the rule review process, but expect the coordination to be beneficial. DHS and TSA SOPs set forth the process for the quarterly review meetings. TSA I&A and stakeholder officials stated that the process generally happens as described in the SOP. Two weeks prior to the meeting, TSA I&A sends out materials including a list of new rules, rule changes, archived (discontinued) rules, and the rationale and links to the underlying intelligence supporting each rule change. According to TSA officials, TSA and DHS stakeholders review the rules from their particular areas of expertise. For example, TSA Chief Counsel officials reported that they review rules and the supporting intelligence to ensure that the rules meet legal sufficiency standards. A TSA Privacy official stated that they review rules and the supporting intelligence to ensure rules do not violate passengers’ rights. All stakeholders review the rules to ensure they are based on current intelligence that identify specific threats. If a stakeholder finds that there is insufficient current intelligence to support the rule, TSA I&A officials stated that they would modify it to ensure it is tailored to current intelligence or archive a rule when the intelligence- based threat is no longer relevant. For example, during the March 2019 quarterly review meeting TSA I&A officials discussed archiving a Silent Partner rule due to insufficient current intelligence to support it. According to TSA I&A officials, the rule was archived in April 2019. TSA I&A officials and stakeholders generally agreed that the quarterly reviews provide a good mechanism for oversight of both programs. Stakeholders told us these meetings provide a forum to discuss the scope of the rules and whether or not they were supported by current intelligence or if they are sufficiently specific. For example, a TSA stakeholder questioned the basis for a rule that identified a particular travel pattern as a high risk factor. As a result, TSA I&A officials reviewed the intelligence and revised the rule. TSA I&A officials stated that since enactment of the TSA Modernization Act in October 2018, they have also notified DHS and TSA stakeholders within two days of making changes to a rule. We reviewed the eight notifications that TSA I&A sent to stakeholders regarding rule changes during the period from October 2018 through May 2019. These notifications detailed changes to rules, new rules, and rules that were archived. DHS and TSA stakeholders we spoke with said that the two day notifications are helpful in keeping them informed in between quarterly meetings. In addition, stakeholders said it allowed them to proactively reach out to TSA I&A to ask questions and share more timely feedback about rule changes. TSA I&A has implemented the two day notifications and other steps required in the TSA Modernization Act, but TSA I&A’s Standard Operating Procedures have not yet been updated to reflect these changes. TSA I&A officials stated that they have plans to do so in fall 2019. TSA I&A’s standard operating procedures establish two situation- dependent processes for reviewing and approving rule changes, as shown in figure 2. First, under standard circumstances, TSA I&A’s standard operating procedures detail a four-part vetting process by which TSA I&A drafts support for the rule change and it is subsequently approved by TSA Chief Counsel, the TSA I&A Assistant Administrator, and ultimately TSA senior leadership. TSA procedures specify that in standard circumstances, all rule changes are to be supported and approved in writing prior to implementation. Specifically, TSA I&A is to draft a memo with the nature of the threat and how all components of the rule address the concerns from intelligence reporting. The memo, along with all pertinent intelligence sources, is then required to be routed through TSA Chief Counsel and TSA leadership for intelligence, legal, and policy review. TSA’s April 2012 Quiet Skies Implementation Plan specified that the Chief Counsel’s review is to ensure that the proposed rule targets the threat presented in the assessment, the assessment properly documents the reasons for the recommendation, and the recommendation is in compliance with relevant legal authorities, regulations, and DHS policies. Upon approval, the memo is referred to TSA senior leadership—the TSA Administrator or TSA Deputy Administrator—for final written approval. Following this, the rule change can be implemented. A second process, called exigent, is also briefly described in the SOPs. In exigent circumstances—circumstances requiring immediate action—the TSA I&A Assistant Administrator or his or her designee may direct that the rule be implemented immediately without a signed decision memo. The signed memo is still required, but can be drafted, reviewed, and approved after the change is implemented. TSA I&A officials stated that the exigent process entails verbal direction to implement a rule. It is unclear if TSA I&A has followed the exigent rule review process in standard circumstances because the SOP is unclear on the criteria for each process. TSA’s SOP states that the exigent review process may be used “if TSA determines that exigent circumstances require immediate implementation of a Silent Partner rule.” However, the SOP does not clarify who or which office within TSA makes this determination or what types of circumstances would be appropriately characterized as exigent. TSA I&A officials told us that exigent circumstances were very rare. They estimated that in the last 3 years exigent circumstances had occurred once. Yet, the same officials also estimated that they implemented approximately 90 percent of the rule changes following verbal approval from either TSA or I&A leadership and drafted the required memos after the fact. This indicates that TSA I&A officials have not followed the standard review process when implementing rule changes in circumstances they regard as standard, and the process followed appears to be closer to what would occur in exigent circumstances. These TSA I&A officials explained that drafting and processing the approval memo after they implement a rule change allows them to more quickly respond to changing intelligence. TSA’s SOP provides flexibility for this in exigent circumstances. However, given the absence of clarity in the SOP about when the exigent process is to be used and who is to make that decision, it is unclear whether or not TSA I&A used the exigent review process—a process which is not, initially, contingent upon TSA’s legal review or I&A’s written support—in circumstances that DHS and TSA leadership who oversee the program would regard as standard. According to Standards for Internal Control in the Federal Government, management should implement control activities through policies by, for example, documenting responsibilities in policies and periodically reviewing policies and procedures for continued relevance and effectiveness. As TSA I&A updates its Silent Partner and Quiet Skies SOPs in fall 2019, clarifying the criteria for standard and exigent rule review procedures would provide greater assurance that screening rule changes are reviewed as intended. TSA I&A officials further told us that they do not document or otherwise have a way of determining what proportion of rule changes have been reviewed in accordance with the standard process versus the exigent process because they had not identified a need to do so. According to the 2012 TSA memo establishing Quiet Skies as a permanent program, at the program’s outset a working group of DHS and TSA stakeholders identified the need for transparency as the first of seven key areas of consensus. Further, DHS’s Integrated Risk Management Framework establishes transparency and documentation as important characteristics of homeland security risk management. Documenting which review process TSA I&A uses for each rule change could improve transparency. TSA I&A monitors some operational data on its passenger screening rules. For example, TSA I&A officials track the number of individuals on the Silent Partner and Quiet Skies Lists, and the number of Silent Partner and Quiet Skies rules triggered by the passengers’ travel. TSA I&A officials stated that rule matches and list size are helpful for oversight purposes because they allow TSA I&A to monitor for Secure Flight system errors. Officials identified one example in which a Secure Flight software update created a system error that prevented 808 passengers from being moved to the Quiet Skies cleared list after a designated number of flights. According to the officials, monitoring list size and the number of rules triggered by passengers’ travel allowed them to identify and correct this error within 10 days of identifying the system error. TSA I&A has not identified a means to comprehensively measure rule effectiveness. TSA I&A officials explained that they would find it helpful to demonstrate the effectiveness of the program, but had not yet done so because it was difficult to measure. TSA I&A officials reported that the approach they have used was to count the number of Quiet Skies passengers who were later identified as a known or suspected terrorist and added to the Terrorist Screening Database. TSA I&A officials reported that in January 2019 they reviewed all Quiet Skies passengers from January 2014 through July 2018 to determine how many were subsequently added to the Terrorist Screening Database. However, because it included Quiet Skies only, this analysis excluded about 93 percent of the rules. TSA officials reported that it is not feasible to do a similar analysis for Silent Partner rules because of the higher numbers of rules and matches and the difficulty matching Silent Partner rules to data in the Terrorist Screening Database. Further, TSA officials noted that without comparable information on the rate that non-Quiet Skies passengers were added to the Terrorist Screening Database during that time period, it is difficult to interpret what the results indicate about rule effectiveness. TSA’s April 2012 Quiet Skies Implementation Plan established that TSA would continually evaluate the performance of the rules in the Silent Partner and Quiet Skies programs. Further, GAO and the Office of Management and Budget have previously identified useful practices to enhance performance management and measurement processes. GAO has previously reported that measuring performance allows organizations to track the progress they are making toward their goals and gives managers critical information on which to base decisions for improving their progress. Office of Management and Budget guidance has also focused specifically on common challenges associated with measuring effectiveness, including data availability and identifying measurable outcomes for a program. This guidance suggests using a variety of approaches such as outlining short-term milestones, identifying target outcomes, and using proxy measures to assess these programs. Assessing the effectiveness of Silent Partner and Quiet Skies rules may be difficult, but I&A could explore using other data sources to assess program effectiveness in addition to further developing their consideration of Terrorist Screening Database additions. For example, TSA I&A could consider analyzing TSA data on the outcomes of the enhanced screening of Silent Partner and Quiet Skies passengers at passenger security checkpoints. CBP officials said that they review secondary inspection results to help them assess CBP’s rules-based program. TSA I&A officials noted that they were considering this measure and would need to determine what comparison group would make sense, and if they want to focus on specific screening outcomes versus all outcomes. TSA I&A could also consider using the results of air marshals’ monitoring of Quiet Skies passengers. According to senior Federal Air Marshal Service officials, the service—with a budget of approximately $780 million for fiscal year 2019—began deploying air marshals on as many flights as possible with Quiet Skies passengers in March 2018. According to TSA’s Privacy Impact Assessment for Silent Partner and Quiet Skies and a Federal Air Marshal Service official, after air marshals complete a flight with a Quiet Skies List match, they file a report saying either “nothing to report” or, if they observe that the individual was involved in a security incident or suspicious activity, they will describe this in an after-action report. TSA I&A officials told us that while they have seen individual after-action reports, they do not review them regularly. These after-action reports are another source of information TSA I&A could consider using to gauge program effectiveness. Given the TSA resources being devoted to the enhanced screening and in-flight monitoring of many passengers matching the Silent Partner and Quiet Skies Lists, and the burden on the traveling public, it is important that TSA understand the value of its screening rules programs. Exploring additional data sources—such as checkpoint screening results and Federal Air Marshal Service after-action reports—could help TSA refine and supplement their existing efforts to measure program effectiveness. The attempted attack of December 25, 2009, highlighted the unknown threats to U.S. civil aviation. TSA has created the Silent Partner and Quiet Skies Lists to help address these unknown threats by ensuring that certain potentially higher risk passengers receive enhanced screening when traveling to, from, or within the United States. TSA created an oversight process that was further bolstered by the TSA Modernization Act, and DHS and TSA officials we met with generally regard the process as effective. However, TSA SOPs are not clear about when it is appropriate for TSA to use an expedited review process and they do not document which review process they used. The lack of clear SOPs inhibits program oversight. By establishing clear criteria for and documentation of each review process, TSA could increase transparency and ensure rule changes are reviewed as intended. Moreover, TSA has not identified a means to comprehensively measure the effectiveness of its Silent Partner and Quiet Skies rules. Exploring additional data sources—such as checkpoint screening results and Federal Air Marshal Service after-action reports—could help TSA refine and supplement their existing efforts to measure program effectiveness. We are making the following three recommendations to TSA: The Administrator of TSA should clarify the criteria for exigent circumstances and standard rule review procedures; (Recommendation 1) The Administrator of TSA should document which rule review process TSA I&A uses (exigent or standard) for each new rule or rule change; (Recommendation 2) The Administrator of TSA should explore additional data sources measuring the effectiveness of Silent Partner and Quiet Skies rules. (Recommendation 3) We provided a draft of our report to DHS for comment. In written comments, which are included in appendix I, DHS concurred with our three recommendations and described steps they plan to take to address them. DHS also provided technical comments, which we have incorporated, as appropriate. We are sending copies of this report to the appropriate congressional committees and to the Acting Secretary of Homeland Security. In addition, this report is available at no charge on the GAO website at http://gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in Appendix II. Appendix II: GAO Contacts and Staff Acknowledgments Error! No text of specified style in document. William Russell, (202) 512-8777 or russellw@gao.gov. In addition to the contact named above Claudia Becker, Assistant Director; Imoni Hampton, Analyst-in-Charge; Melissa Greenaway, John De Ferrari, Michele Fejfar, Eric Hauswirth, Tom Lombardi, and Kevin Reeves made key contributions to this work.", "summary": "On December 25, 2009, while on a flight from Amsterdam to Detroit, a person attempted to detonate explosives hidden in their underwear. This person was not included in the government's consolidated database of known or suspected terrorists at the time. In response, in 2010, TSA began identifying passengers who are not known or suspected terrorists, but who TSA determined should receive enhanced screening. Specifically, TSA identifies passengers for enhanced screening through the application of screening rules, which TSA develops by considering current intelligence and other factors. TSA refers to these rules and lists as Silent Partner and Quiet Skies. The TSA Modernization Act includes a provision for GAO to review the current oversight mechanisms and effectiveness of Silent Partner and Quiet Skies. This report examines the extent to which TSA has (1) coordinated with relevant DHS and TSA stakeholders to review passenger screening rules; and (2) assessed the effectiveness of these rules. GAO analyzed TSA documents, including standard operating procedures, and interviewed senior DHS and TSA officials involved in managing and overseeing the programs. The Transportation Security Administration (TSA) coordinates reviews of its intelligence-based screening rules known as Silent Partner and Quiet Skies. Specifically, TSA's Intelligence and Analysis office (I&A) coordinates quarterly rule reviews and notifies Department of Homeland Security (DHS) and TSA stakeholders of rule changes. According to stakeholders, these review processes provide a good mechanism for program oversight. TSA has established guidance for rule changes that involve TSA stakeholders reviewing rules in advance of their implementation. In some instances, TSA uses an alternate process, allowed by guidance in exigent circumstances, where rule changes go into effect before some stakeholders review them. However, agency guidance does not define the conditions for using the standard or exigent processes. Further, TSA officials do not document which review process—standard or exigent—they use for each rule change. Clarifying guidance and documenting which review process is used could improve transparency and better ensure screening rule changes are adequately reviewed. TSA tracks some data on rule implementation, but has not identified a means to comprehensively measure rule effectiveness. TSA officials explained that they had not yet fully assessed the rules' effectiveness because it was difficult to measure. Silent Partner rules identify passengers for enhanced screening on inbound flights to the United States. Quiet Skies rules—a subset of the Silent Partner rules—identify passengers for enhanced screening on subsequent domestic and outbound flights. TSA officials said that the one method they had used to assess effectiveness was to count Quiet Skies passengers who were later added to the government's watchlist of known or suspected terrorists. However, because this analysis was limited to Quiet Skies, it excluded 93 percent of the screening rules, making it difficult to interpret what the results indicate about effectiveness. TSA has access to data, such as the outcomes of enhanced screening of Silent Partner and Quiet Skies passengers, that could be explored to better assess rule effectiveness. Exploring additional data sources could help TSA refine and supplement their existing efforts to measure program effectiveness. GAO is making three recommendations. DHS should (1) clarify the criteria for exigent and standard rule review procedures; (2) document which review process is used for each new rule or rule change; and (3) explore additional data sources for measuring rule effectiveness. DHS concurred with these recommendations.", "document_type": "gao"}
{"report": "In 1992, the Prescription Drug User Fee Act (PDUFA) was enacted, in part, to provide additional funds for FDA to support the process of reviewing NDAs. PDUFA authorized FDA to collect user fees from drug sponsors to supplement its annual appropriation for salaries and expenses. PDUFA has been reauthorized every 5 years since 1992; most recently PDUFA VI reauthorized the prescription drug user fee program from fiscal year 2018 through fiscal year 2022. As part of each reauthorization process, FDA identifies goals in a commitment letter to Congress. In general, these goals identify a percentage of certain types of applications that FDA is expected to review within specified time frames, including goals for the time the agency takes to complete reviews of different types of NDAs upon initial submission and resubmission. For example, in its commitment letters for PDUFA V and VI, FDA committed to completing its initial review of 90 percent of priority NDAs that involve previously marketed or approved active ingredients within 6 months of receipt. As previously noted, four key features of NDAs are linked to drug development and review processes. For initial NDA reviews, the time frames for FDA’s review that would meet its PDUFA V and VI commitments—its PDUFA goals—vary and are linked to three key features of the NDA. (See table 1.) The target time frame for the initial review of any specific NDA under these user fee commitments reflects the goals associated with all three of the key features. The fourth key feature of NDAs is whether they qualify for one of FDA’s expedited programs. Whether designated as priority or standard, FDA may determine that NDAs for drugs intended to treat serious or life- threatening conditions qualify for development and review under one or more expedited programs. These programs confer specific benefits with the potential to help reduce the development or review time needed to bring a drug to market. For example, some expedited programs provide for more intensive drug development guidance from FDA officials or allow the applicant to submit completed sections of the NDA for review before submitting the entire application. FDA’s expedited programs include accelerated approval, breakthrough therapy designation, and fast track designation. (See table 2.) NDAs must include substantial evidence of a drug’s effectiveness, which is typically drawn from clinical trials. In traditional clinical trials, patients receiving a new drug are often compared with patients receiving a placebo or a different drug. To maximize data quality, these clinical trials are usually randomized (patients are randomly assigned to either the group receiving the new drug or a comparison group) and double-blinded (neither the patients nor the investigators know who is receiving a particular treatment). According to FDA, although this type of study design is often the most powerful tool for evaluating the safety and effectiveness of new drugs, many traditional clinical trials are becoming more costly and complex to administer. Additionally, according to FDA, many new drugs are not easily evaluated using traditional approaches. For example, drugs intended for patients with rare diseases are difficult to evaluate due to the limited number of patients affected by the disease and available for study. The Cures Act was enacted on December 13, 2016, to accelerate the discovery, development and delivery of new treatments—including drugs—for patients. Among other things, the Cures Act includes provisions for FDA to evaluate and facilitate the use of evidence from sources other than traditional clinical trials to support safety and effectiveness determinations for new drugs. For example, FDA was directed to evaluate the potential use of evidence based on data that is routinely collected outside of traditional clinical trials from sources such as electronic health records, medical claims data, and disease registries; evidence from such data sources is referred to as real-world evidence. In the commitment letter associated with PDUFA VI, which was enacted on August 18, 2017, the agency agreed to certain goals relating to the use of real-world evidence in regulatory decision-making and also agreed to certain activities intended to facilitate the development and application of an additional source of evidence known as model-informed drug development. Although these nontraditional sources of evidence were included in NDAs prior to the enactment of the Cures Act and PDUFA VI, at the time this legislation was enacted, most of them were not widely used. For example, according to FDA officials, the NDAs that included real-world evidence were generally for drugs to treat oncology diseases or rare diseases. Our analysis of the 637 original NDAs submitted from fiscal years 2014 through 2018 indicates that divisions differed in the proportions of NDAs they reviewed that had any one of three key features that are linked to time frames for initial review under FDA’s PDUFA goals. As examples: 6 percent of the NDAs reviewed by the dermatology and dental division had a priority review designation, while 56 percent of the NDAs reviewed by the anti-infective division had a priority review designation; 4 percent of the NDAs reviewed by the anesthesia, analgesia, and addiction division involved a new molecular entity, while 52 percent of the NDAs reviewed by the neurology division involved one; and None of the NDAs reviewed by the transplant and ophthalmology division involved a major amendment, while 36 percent of the applications reviewed by the gastroenterology and inborn errors division involved one. (See fig. 1. App. IV provides more detailed information about differences between divisions in the number and proportion of NDAs with these key features.) We also found differences between divisions in the proportion of NDAs that they reviewed under an expedited program—the fourth key feature of NDAs. For example, none of the NDAs reviewed by the metabolism and endocrinology division qualified for one or more expedited programs, while 52 percent of the NDAs reviewed by the antiviral division qualified for one or more expedited programs. (See fig. 2. App. V provides more detailed information about differences between divisions in the number and proportion of NDAs that qualified for one or more expedited programs.) It is not unexpected that divisions differ in the proportion of their applications with key features linked to FDA’s time frames for review or qualification for expedited programs because the divisions are responsible for different products. For example, some divisions, such as the oncology divisions, regulate products for conditions that are more likely to be serious or life-threatening, and therefore the NDAs reviewed by these divisions are more likely to qualify for priority review designation and expedited programs, compared with other divisions, such as the dermatology and dental division. Our analysis of review times for the 637 original NDAs submitted from fiscal years 2014 through 2018 shows that FDA divisions differed in the number of days they took to complete their initial reviews. For example, the median time taken to complete an initial review of an NDA by the anti- infective division was about 2 months faster than the median time taken by the gastroenterology and inborn errors division. (For more information about initial review times, see app. VI.) We found, however, that these differences in initial review times largely reflected key features of the NDAs reviewed by the divisions, particularly those features linked to FDA’s time frames for review under its PDUFA goals. We analyzed initial review times using a statistical regression with two variables reflecting key features of the NDAs—target time frame for review of the application under FDA’s PDUFA goals (in days, from FDA’s receipt of the NDA to FDA’s targeted date for completion of the initial review) and number of expedited programs (0, 1, or 2 or more)—along with division as independent variables. We found that each of these variables was a significant determinant of initial review times. Specifically, our regression analysis shows that on average The shorter the target time frame for initial review of the NDA under FDA’s PDUFA goals, the shorter the initial review, and this target time frame was responsible for the majority of variation in initial review times. The greater the number of expedited programs for which the NDA qualified, the shorter the time FDA took to complete the initial review. Controlling for the effects of these key NDA features, however, we found that most of the divisions’ average review times were similar to (within 2 weeks of) each other. In contrast, the hematology and oncology divisions reviewed applications a bit more rapidly—about 2 or 3 weeks faster—than other divisions. Figure 3 illustrates the results of our analyses. The panel on the left shows the variation in the divisions’ actual average review times. The panel on the right shows the estimated average review times, after accounting for key application features, that is, what the review times would have been if each division had reviewed equal numbers of applications with these key features. We asked FDA officials what might contribute to somewhat faster review times by the hematology and oncology divisions, and FDA officials told us that a number of variables could have contributed to these differences. For example, the officials told us that applicants differ in their level of experience, which can affect the quality of the NDA or the speed of response to FDA’s requests for information; applications differ in complexity; and the oncology and hematology divisions could differ from others in their risk/benefit considerations. As previously noted, some divisions, such as the oncology divisions, regulate products for conditions that are more likely to be serious or life-threatening compared with other divisions, such as the dermatology and dental division, and risk/benefit considerations can differ across conditions that vary in how serious or life- threatening they are. For example, the potential benefits of drugs that carry substantial risks for dangerous side effects would likely be weighed differently if the drug is intended to address a life-threatening illness for which there is no other treatment than if the drug is intended to address an illness that is not life-threatening or for which there is an alternative treatment. FDA has several initiatives underway to evaluate and facilitate FDA review divisions’ and drug sponsors’ use of evidence derived from sources other than traditional clinical trials to support NDAs. (See table 3 for a description of these different evidence sources and each initiative.) According to FDA officials, implementing these initiatives can help ensure that when drug sponsors utilize these sources of evidence in NDAs, the evidence is of sufficient quality to be used in regulatory decision-making and that there is consistency across FDA review divisions in their evaluation of the evidence. FDA officials also said that although complex innovative trial designs might replace traditional clinical trials as evidence in NDAs, real-world evidence is more likely to be used to supplement clinical trial data. Although the initiatives are not restricted to any particular type of disease or patient population, according to FDA officials, some initiatives may be more relevant for certain types of diseases or patient populations than others. For example, according to FDA officials: real-world evidence may be most relevant for diseases that have outcomes that are consistently collected in the health care system. clinical outcome assessments (one aspect of patient-focused drug development) may be most relevant for diseases that are chronic, symptomatic, or affect functioning and activities of daily living. complex innovative trial designs may be most relevant for situations in which the population size is small or limited, such as pediatric populations, or where there is an unmet medical need, such as rare diseases. Our review of FDA documentation and interviews with FDA officials show that FDA has taken steps to implement each of these five initiatives. These steps include conducting public workshops with key stakeholders, issuing guidance for industry and FDA staff, initiating pilot programs, and developing FDA staff capacity, including by providing training and other educational resources. (See table 4 for examples of key activities by initiative.) These and future planned activities—including issuing additional guidance and revising relevant FDA policies and procedures— are intended to address deliverables for FDA to accomplish through 2021 that are outlined in the Cures Act and the PDUFA VI commitment letter. According to FDA officials, the agency intends to meet these deliverables, though, according to these officials, some of the activities implemented under the initiatives, such as certain pilot programs, will likely extend beyond 2021. Although implementation is still in progress for all of the initiatives, FDA officials reported some outcomes. For example, since the launch of the model-informed drug development pilot program, the agency has received two NDA supplements that incorporated model-informed drug development concepts discussed during pilot program meetings. Additionally, officials told us there has been a recent increase in investigational new drug submissions utilizing complex innovative trial designs. FDA officials also reported an increase in biomarker submissions under the drug development tool qualification program, and continued growth of the clinical outcome assessment qualification program. FDA expects that fully implementing the initiatives will lead to further increases in the use of evidence from sources other than traditional clinical trials. We provided a draft of this report to the Department of Health and Human Services for review and comment. The department provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of the Department of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. To determine (1) how Food and Drug Administration (FDA) divisions differ in the proportion of new drug applications (NDA) they review with key features linked to review time goals and expedited programs and (2) how FDA review divisions differ in the time taken to complete initial reviews and the extent to which key features of NDAs contribute to those differences, we analyzed data from FDA. We also interviewed FDA officials about the data and their review processes. We obtained data regarding all NDAs submitted to FDA’s Center for Drug Evaluation and Research (CDER) from fiscal years 2014 through 2018. These data included information about features that distinguish NDAs from one another, including which division was responsible for the review. The data also included information through March 31, 2019, about the dates when FDA received and completed a review of each NDA, along with the target dates for completion of review under FDA’s goals in commitment letters associated with the Prescription Drug User Fee Act (PDUFA) reauthorizations for fiscal years 2013 through 2017 (PDUFA V) and fiscal years 2018 through 2022 (PDUFA VI). To ensure meaningful analysis of review times, we excluded NDAs for which FDA had not completed an initial cycle of review. Of 686 NDAs submitted in fiscal years 2014 through 2018, the applicant withdrew 10 NDAs prior to completion of FDA’s initial review and 39 NDAs were still under FDA review as of March 31, 2019, leaving 637 NDAs for which FDA had completed an initial review. To assess the reliability of these data, we conducted a series of electronic and logic tests to identify missing data or other anomalies. These analyses were informed by our review of relevant documentation and interviews with knowledgeable FDA officials. As part of our assessment of reliability, we worked with FDA to identify and correct information about certain NDAs in a small number of instances in which we identified discrepancies. Using these methods, we determined that the remaining data were sufficiently reliable for our purposes. Unless otherwise specified, the results we present are statistically significant at the 0.05 level. To determine how FDA divisions differ in the proportion of NDAs they review with key features linked to FDA’s time frames for initial reviews and expedited programs, we conducted a series of chi-square tests comparing the distributions of the 637 NDAs with and without specific features across divisions. These key features included: whether the NDA had a priority review designation (a designation applied by FDA if the product would provide a significant therapeutic improvement in the safety and effectiveness of the prevention, diagnosis, or treatment of a serious condition when compared to available drugs) or instead had a standard designation; whether the NDA did or did not involve a new molecular entity—an active ingredient that had not previously been marketed or approved for use as a drug in the United States, whether the NDA did or did not involve a major amendment (a submission, while a pending NDA is under FDA review, of additional information that may include a major new clinical safety or efficacy study report or major new analyses of studies, among other things); and whether the NDA did or did not qualify for an expedited program (accelerated approval, breakthrough therapy designation, or fast track designation), programs intended to help reduce the time involved in developing or reviewing certain drugs that have the potential to treat serious or life-threatening conditions. (See table 5 for relevant statistics from these chi-square tests.) To determine how FDA review divisions differ in the time taken to complete initial reviews, we conducted a preliminary regression analysis of 637 NDAs with the number of days an FDA division took to complete its initial review as the dependent variable and division as a single independent variable. We defined the time to complete a review as the number of days from FDA’s receipt of the NDA to the agency’s completion of the initial review by taking regulatory action. To determine the extent to which key NDA features contributed to differences between divisions in the time taken to complete initial reviews, we conducted a multiple regression analysis of the number of days FDA took to complete its initial review with division as an independent variable, along with two other independent variables to control for the key NDA features: Target time frame for initial review of the NDA under FDA’s PDUFA goals. Three key NDA features are linked to time frames for FDA’s initial review under its PDUFA goals—whether the NDA was priority or standard, did or did not involve a new molecular entity, and did or did not involve a major amendment. To control for these three features simultaneously, we counted the number of days from FDA’s receipt of the NDA until FDA’s target date for completion of the initial review under FDA’s PDUFA goals, and used that variable—the target time frame for review under FDA’s PDUFA goals—as an independent variable. We identified five NDAs for which FDA’s review time was exceptionally long in comparison to the target time frame for review under its PDUFA goals, and we asked FDA officials about them. FDA officials stated that these reviews were substantially delayed because of complicated manufacturing site issues, complicated legal and regulatory issues, or emerging public health issues requiring last minute advisory committee meetings—conditions that we deemed sufficiently unusual to exclude these five NDAs from further statistical analyses of review times. Number of expedited programs for which the NDA qualified. Another key NDA feature is whether it qualified for one or more expedited programs, programs with the potential to help reduce the development or review time needed to bring a drug to market. We controlled for this feature by including number of expedited programs (0, 1, or 2 or more) as an independent variable in our multiple regression analysis. Thus, we tested the effect of division on initial review times for 632 NDAs while controlling for the target time frame for review under FDA’s PDUFA goals and qualification for expedited programs. (See tables 6 and 7 for relevant statistics from this multiple regression analysis.) Our multiple regression analysis allowed us to test a specific hypothesis about the effect of division on review times, namely, whether divisions differed in their review times after controlling for the key features of NDAs. This regression analysis did not test a model of review times—that is, we did not attempt to identify all variables that affect review times, nor did we seek to identify the specific set or combination of variables within our data that had maximum explanatory power. Our analyses indicated that variation remained in initial review times, even after we controlled for these variables. It is important to note that an array of factors might be expected to influence review times, including not just those factors that were captured in our analysis, but also factors such as state of the science and quality of the application. With data from 632 NDAs distributed unevenly across 15 divisions, meaningful tests of additional variables or their interactions were not possible. Nonetheless, we conducted exploratory analyses that included other potentially relevant variables in addition to the target time frame for review under FDA’s PDUFA goals, number of expedited programs, and division. In separate regression analyses, we examined (a) the fiscal year in which FDA received the NDA and (b) whether the application was a BLA, an NDA based on information from studies conducted by the applicant, or an NDA based on at least some information from studies not conducted by or for the applicant. We did not find evidence of a consistent effect of either of these additional factors on review times, but in light of the number of NDAs, we cannot exclude the possibility that one or more of these factors affects review times. In a third exploratory analysis, we examined the outcome of the initial review—(a) approval; (b) tentative approval, which FDA grants if the NDA meets requirements for approval, but cannot be approved due to a patent or exclusivity period for a listed drug; or (c) issuance of a letter to the applicant called a complete response letter, in which FDA describes the specific deficiencies the agency identified and recommends ways to make the application viable for approval. This analysis suggested that NDAs that were approved for marketing at the end of the initial cycle of review were reviewed slightly faster on average than other NDAs, but this result should be viewed with caution because a small number of NDAs with certain initial review outcomes were distributed unequally. For example, very few of the NDAs (11) reviewed through one or more expedited programs resulted in tentative approval. The Food and Drug Administration’s (FDA) Center for Drug Evaluation and Research (CDER) divisions differed in the total number of days they took to complete reviews of 637 new drug applications (NDA) submitted from fiscal years 2014 through 2018 and completed by March 31, 2019. (See fig. 4.) Importantly, these times reflect differences associated with the number of completed review cycles, FDA’s target time frames for review under its goals in commitment letters associated with the Prescription Drug User Fee Act (PDUFA) reauthorizations for fiscal years 2013 through 2017 (PDUFA V) and fiscal years 2018 through 2022 (PDUFA VI), and number of expedited programs. Number of review cycles. The number of cycles of review to which the NDAs we examined were subject was largely dependent on factors that were not under FDA’s control, namely, the applicant’s actions and timing. When a cycle of review ends with an FDA action, that action can be (a) approval, which allows the applicant to market the drug, (b) tentative approval, which FDA grants if the NDA meets requirements for approval, but cannot be approved due to a patent or exclusivity period for a listed drug, or (c) issuance of a letter to the applicant called a complete response letter, in which FDA describes the specific deficiencies the agency identified and recommends ways to make the application viable for approval. The applicant may respond to either tentative approval or a complete response letter by resubmitting a revised application, triggering a new cycle of review; it is up to the applicant to decide whether to resubmit the application. In addition, NDAs that were submitted earlier in time would have a greater chance of being resubmitted and reviewed by March 31, 2019, than applications submitted later in time. The number of completed review cycles ranged from one to four cycles: 637 NDAs went through a completed first (initial) cycle review; 99 of those 637 NDAs went through a completed second cycle review; 20 of those 99 NDAs went through a completed third cycle review; 3 of those 20 NDAs went through a completed fourth cycle review. Target time frames for review. Review times reflect differences in time frames for review under FDA’s PDUFA goals. The target time frames for review ranged from less than 6 months to 15 months for the first cycle and from less than 2 months to 9 months for later cycles of review. Number of expedited programs. These review times also reflect differences associated with the number of FDA’s expedited programs for which NDAs qualified. In general, these expedited programs are designed to help reduce the development or review time needed for drugs intended to treat serious or life-threatening conditions. Two of the Food and Drug Administration’s (FDA) expedited programs for new drugs intended to treat serious or life-threatening conditions— breakthrough therapy designation and fast track designation—must be requested by the drug sponsor. These programs are intended to help reduce the development or review time needed to bring a drug to market by offering benefits such as more intensive drug development guidance from FDA officials or by allowing the applicant to submit completed sections of the NDA for review before submitting the entire application. The request is normally made while the drug sponsor is conducting clinical trials or when seeking FDA’s permission to collect clinical trial data, although the request may also be made when submitting a new drug application (NDA) or while the NDA is under review. FDA’s Center for Drug Evaluation and Research (CDER) divisions are responsible for determining whether requests qualify for these expedited programs based on evidence the drug sponsors provide in support of the requests. To qualify for breakthrough therapy designation, the drug sponsor must present preliminary clinical evidence involving one or more clinically significant endpoints that indicate that the drug may demonstrate substantial improvement over available therapies. To qualify for fast track designation, the drug sponsor must either provide evidence demonstrating the drug’s potential to address unmet need or document that the drug is designated as a qualified infectious disease product. FDA may grant or deny the request, or the drug sponsor may withdraw the request before FDA renders a decision. If FDA grants the designation, the drug sponsor may subsequently withdraw from the designation, or FDA may rescind either designation if the drug no longer meets the qualifying criteria. We obtained data regarding all requests for breakthrough therapy and fast track designations submitted to CDER from fiscal years 2013 through 2018. These data included information about which division was responsible for the review and the outcome of the request—whether it was granted or denied or whether the drug sponsor withdrew the request before FDA reached a decision. To assess the reliability of these data, we conducted a series of electronic and logic tests to identify missing data or other anomalies. These analyses were informed by our review of relevant documentation and interviews with knowledgeable FDA officials. Using these methods, we determined that the data were sufficiently reliable for our purposes. We examined these data to determine whether there were any material differences between divisions in the frequency of possible outcomes. Our analyses focused on the outcomes and did not allow us to determine whether divisions differed in their application of the stated criteria. Breakthrough therapy designation. We found few differences across divisions in the frequency of the possible outcomes of requests for breakthrough therapy designation: Of 634 requests for breakthrough therapy designation (including nine requests submitted with or after the NDA submission), 39 percent were granted, 48 percent were denied, and 13 percent were withdrawn by the drug sponsor before FDA reached a decision. Divisions differed widely in the number of requests for breakthrough therapy designation they received, from 0 for the nonprescription drug division to 102 for one of FDA’s two oncology divisions. With two exceptions, the numbers of these requests that were granted, denied, or withdrawn for each division were similar to what would be expected based on the overall frequency of the possible outcomes. Requests to the hematology division were withdrawn more frequently than requests to other divisions (32 percent) and that division denied requests less frequently (17 percent) than other divisions. The neurology division denied more (81 percent), and granted fewer (13 percent), requests for breakthrough therapy designation than other divisions. Within the time period we studied, the drug sponsor withdrew from breakthrough therapy designation after it was granted in six cases and FDA rescinded the designation in 14 cases. Fast track designation. Similarly, we found few differences across divisions in the frequency of the possible outcomes of requests for fast track designation: Of 965 requests for fast track designation (including 35 requests submitted with or after the NDA submission), 71 percent were granted, 24 percent were denied, and 5 percent were withdrawn by the drug sponsor before FDA reached a decision. Again, divisions differed widely in the number of requests for fast track designation they received, from 2 for the nonprescription drug division to 133 for the neurology division. The numbers of these requests that were granted, denied, or withdrawn for each division were generally similar to what would be expected based on the overall frequency of the possible outcomes, although the anti-infective division granted more (91 percent), and denied fewer (6 percent), requests for fast track designation than other divisions. Within the time period we studied, no drug sponsor withdrew from fast track designation after it was granted, nor did FDA rescind any such designation. Pursuant to the Prescription Drug User Fee Act (PDUFA) and its subsequent reauthorizations, the Food and Drug Administration (FDA) collects user fees from drug sponsors to supplement its annual appropriation for salaries and expenses. As part of each reauthorization process, FDA identifies goals in a commitment letter to Congress, including goals for the time the agency takes to complete reviews of different types of drug applications upon initial submission and resubmission. In general, these goals identify a percentage of certain types applications that FDA is expected to review within specified target time frames. For initial NDA reviews—reviews of the NDA as originally submitted—FDA’s target time frames for review that would meet its PDUFA goals vary and are linked to three key NDA features that reflect the drug or the applicant’s action: (1) whether or not the application receives priority review designation, which indicates that the drug could provide significant therapeutic improvements in the safety and effectiveness of the prevention, diagnosis, or treatment of a serious condition when compared to available drugs; (2) whether or not the application involves a new molecular entity—an active ingredient that has not been previously marketed or approved for use in the United States; and (3) whether or not the applicant submitted a major amendment while the NDA was pending, that is, while under FDA’s review. The target time frame for review for any specific NDA reflects all three of these features. Reviews are conducted by one of the agency’s Center for Drug Evaluation and Research (CDER) divisions, each of which specialize in a specific group of drug products, such as hematology or neurology. As shown in table 8, divisions differed in the numbers and proportions of NDAs they reviewed that had the features linked to time frames for review under FDA’s PDUFA goals. The Food and Drug Administration (FDA) may determine that NDAs for drugs intended to treat serious or life-threatening conditions qualify for one or more expedited programs. These programs confer specific benefits with the potential to help reduce the development or review time needed to bring a drug to market, for example, some expedited programs provide for more intensive drug development guidance from FDA officials or allow the applicant to submit completed sections of the NDA for review before submitting the entire application. FDA’s expedited programs include accelerated approval, breakthrough therapy designation, and fast track designation. Reviews are conducted by one of the agency’s Center for Drug Evaluation and Research (CDER) divisions, each of which specialize in a specific group of drug products, such as hematology or neurology. As shown in table 9, divisions differed in the proportions of NDAs they reviewed that qualified for expedited programs. The Food and Drug Administration’s (FDA) Center for Drug Evaluation and Research (CDER) divisions differed in the total number of days they took to complete initial reviews of new drug applications (NDA) received from fiscal years 2014 through 2018 and completed by March 31, 2019. (See fig. 5.) These review times reflect differences associated with FDA’s target time frames for initial review under its goals in commitment letters associated with the Prescription Drug User Fee Act (PDUFA) reauthorizations for fiscal years 2013 through 2017 (PDUFA V) and fiscal years 2018 through 2022 (PDUFA VI). These target time frames for review are linked to specific features of the NDA and ranged from less than 6 months to 15 months for the initial review. These review times also reflect differences associated with the number of expedited programs for which NDAs qualified. John E. Dicken, (202) 512-7114 or dickenj@gao.gov. In addition to the contact named above, William Hadley (Assistant Director), Geri Redican-Bigott (Assistant Director), Aubrey Naffis (Analyst- in-Charge), and Kristen Joan Anderson made key contributions to this report. Also contributing were Sam Amrhein, Todd D. Anderson, Leia Dickerson, Kaitlin Farquharson, Rich Lipinski, and Ethiene Salgado- Rodriguez.", "summary": "Before a drug can be marketed in the United States, FDA must determine that the drug is safe and effective for its intended use through a review of evidence that a drug sponsor—the entity seeking to market the drug—submits in an NDA. The review is conducted by one of FDA's divisions (17, at the time of GAO's review) that each specialize in a specific group of drug products, such as hematology products. NDA reviews are complex, and may involve not only an initial review, but also reviews of resubmissions if the initial review does not result in approval. Under FDA's PDUFA commitments, FDA's goal is to complete reviews of 90 percent of NDAs within specific time frames linked to key features of the NDAs. GAO was asked to examine NDA review times across FDA's divisions. In this report, GAO examines (among other things) differences between FDA divisions in the key features of the NDAs they review and initial review times, as well as the extent to which key NDA features contribute to these differences. GAO analyzed data from FDA's Center for Drug Evaluation and Research regarding 637 NDAs submitted from fiscal years 2014 through 2018. These data also included biologic license applications submitted to the center. GAO excluded NDAs that were withdrawn by the applicant before FDA completed a review, as well as NDAs for which FDA had not completed a review by March 31, 2019. GAO also interviewed FDA officials about the agency's review process and these review times. The Department of Health and Human Services provided technical comments on a draft of this report, which GAO incorporated as appropriate. Four key features of new drug applications (NDA) are linked to the time the Food and Drug Administration (FDA) takes to complete initial reviews of NDAs. Three key NDA features determine the time frames for initial review that would meet FDA's goals under the Prescription Drug User Fee Act (PDUFA) and its reauthorizations, which authorize FDA to collect user fees from drug sponsors: Whether or not the NDA qualifies for the priority review program, which is generally an expedited program for drugs that provide significant therapeutic improvements in the prevention, diagnosis, or treatment of a serious condition when compared to available drugs. The PDUFA goal for review of a priority NDA is 4 months less than for an otherwise similar standard NDA, for which the goal is to complete the review in 10 months. Whether or not the NDA involves a new molecular entity (an active ingredient that has not been previously marketed or approved in the United States). The PDUFA goal for review of an NDA with a new molecular entity is 2 months longer than for an NDA without one. Whether or not the applicant submits a major amendment (additional or new information, such as a major new clinical study) while the NDA is under review. The PDUFA goal for a review of an NDA may be extended by 3 months if the applicant submits a major amendment. The fourth key NDA feature is whether or not it qualified for one or more of three other expedited programs for drugs intended to treat serious or life-threatening conditions. GAO's analysis of 637 NDAs submitted from fiscal years 2014 through 2018 indicated that the proportion of NDAs with these key features differed among FDA review divisions. For example, 6 percent of the NDAs reviewed by the dermatology and dental division had a priority designation, compared to 56 percent for the anti-infective division. FDA has reported that some divisions, such as the oncology divisions, generally regulate products for conditions that are more likely to be serious or life-threatening, and, therefore, those products may be more likely to qualify for priority designation and other expedited programs. GAO found that FDA's divisions differed in the average number of days they took to complete an initial review of NDAs, and these differences largely reflected the key features of the NDAs they reviewed. GAO's analysis shows that the time FDA took to complete an initial review of NDAs was affected by (1) the target time frame for completion of the review under the agency's PDUFA goals, (2) the number of expedited programs for which the NDA qualified, and (3) the division performing the review. GAO also found that the target time frame for review was largely responsible for differences in initial review times. Specifically, NDAs with key features that resulted in shorter target time frames for review under FDA's PDUFA goals had shorter initial review times. Controlling for the effects of these target time frames and the number of expedited programs for which the NDA qualified, GAO found that most of the divisions' average review times were similar to (within 2 weeks of) each other.", "document_type": "gao"}
{"report": "Federal agencies are dependent on information technology (IT) systems and electronic data to carry out operations and to process, maintain, and report essential information. These systems are highly complex and dynamic, technologically diverse, and often geographically dispersed. However, the IT systems supporting federal agencies and our nation’s critical infrastructures are at risk. Information and systems are subject to serious threats that can have adverse impacts on organizational operations and assets, individuals, other organizations, and the nation. These threats can include purposeful attacks, environmental disruptions, and human/machine errors, and may result in harm to the national and economic security interests of the United States. In recognition of the growing threat, we designated information security as a government-wide high-risk area since 1997. In 2003, we expanded the information security high-risk area to include the protection of critical cyber infrastructure. We further expanded the information security high- risk area in 2015 to include protecting the privacy of personally identifiable information. Cybersecurity incidents continue to impact federal agencies, as well as entities across various critical infrastructure sectors. In fiscal year 2017, federal executive branch civilian agencies reported 35,277 incidents to the U.S. Computer Emergency Readiness Team. These incidents included web-based attacks, phishing, and the loss or theft of computing equipment. These incidents and others like them can pose a serious challenge to economic and national security and personal privacy. The following examples highlight the impact of such incidents: In January 2019, the Department of Justice (Justice) announced that it had indicted two Ukrainian men for their roles in a large-scale, international conspiracy to hack into the Securities and Exchange Commission’s computer systems and profit by trading on critical information they stole. The indictment alleges that the two hacked into the Commission’s Electronic Data Gathering, Analysis, and Retrieval system and stole thousands of files, including annual and quarterly earnings reports containing confidential, non-public, financial information, which publicly traded companies are required to disclose to the Commission. In March 2018, a joint alert from DHS and the Federal Bureau of Investigation stated that Russian government actors had been targeting the systems of multiple U.S. government entities and critical infrastructure sectors since at least March 2016. These Russian government actors had affected multiple organizations in various sectors, to include energy, nuclear, water, aviation, construction, and critical manufacturing. DHS and the Federal Bureau of Investigation characterized this activity as a multi-stage intrusion campaign by Russian government cyber actors who targeted small commercial facilities’ networks where they staged malware, conducted spear phishing, and gained remote access into energy sector networks. In June 2015, the Office of Personnel Management (OPM) reported that an intrusion into its systems had affected the personnel records of about 4.2 million current and former federal employees. Then, in July 2015, the agency reported that a separate, but related, incident had compromised its systems and the files related to background investigations for 21.5 million individuals. In total, OPM estimated 22.1 million individuals had some form of personally identifiable information stolen, with 3.6 million being a victim of both breaches. The risks to IT systems supporting the federal government and the nation’s critical infrastructure are increasing as security threats continue to evolve and become more sophisticated. These risks include insider threats from witting or unwitting employees, escalating and emerging threats from around the globe, steady advances in the sophistication of attack technology, and the emergence of new and more destructive attacks. Therefore, it is imperative for agency leaders and managers at all levels to manage the risks associated with the operation and use of information systems that support their missions and business functions. Cybersecurity risk management comprises a full range of activities undertaken to protect IT and data from unauthorized access and other cyber threats; maintain awareness of cyber threats; detect anomalies and incidents adversely affecting IT and data; and mitigate the impact of, respond to, and recover from incidents. Information sharing facilitates and supports all of these activities. Several federal laws, executive orders, and policies establish requirements for protecting federal systems and managing cybersecurity risks. Specifically, FISMA is intended to provide a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support federal operations and assets, as well as the effective oversight of information security risks. The act requires each agency to develop, document, and implement an agency- wide information security program to provide risk-based protections for the information and information systems that support the operations and assets of the agency, including those provided or managed by another entity. FISMA also assigns government-wide responsibilities to key agencies: OMB is responsible for developing and overseeing implementation of policies, principles, standards, and guidelines on information security in federal agencies, except with regard to national security systems. DHS is responsible for certain operational aspects of agencies’ information security policies and practices, including assisting OMB in fulfilling its FISMA authorities, issuing binding operational directives, monitoring agencies’ security policies and practices, and assisting them with implementation. NIST is responsible for developing standards for categorizing information and information systems, security requirements for information and systems, and guidelines for detection and handling of security incidents. More recently, the administration has re-emphasized the importance of improving agencies’ cybersecurity risk management capabilities through the issuance of an executive order. Further, OMB has issued minimum requirements, standards, and guidance to ensure federal managers are effectively managing cybersecurity risks. OMB has also issued policies for enterprise risk management (ERM), which considers all key risks that agencies face and their potential impacts on the agency’s mission. Cybersecurity risk is just one type of risk that agencies consider in their enterprise approach to risk management. Table 1 identifies the administration’s May 2017 executive order and relevant OMB publications and guidance on cybersecurity risk management. In its responsibility for certain operational aspects of agencies’ implementation of cybersecurity practices, DHS is spearheading several initiatives to assist federal agencies in protecting their computer networks and electronic information. Examples of DHS’s initiatives are described in table 2. Implementing effective cybersecurity requires any organization—whether a private sector company; a non-profit entity; or an agency at the state, local, or federal level—to identify, prioritize, and manage cyber risks across its enterprise. Risk management is a comprehensive process that requires organizations to (1) frame risk (i.e., establish the context for risk- based decisions), (2) assess risk, (3) respond to risk once determined, and (4) monitor risk on an ongoing basis using effective organizational communications and a feedback loop for continuous improvement in the risk-related activities of organizations. In accordance with its responsibilities under FISMA, as well as other laws and executive orders, NIST has developed a framework for managing risk to federal information and information assets. This framework calls for a multi-tiered approach to risk management, with activities at the information system (system), business/mission, and organization (e.g., agency) level. Cybersecurity risk management activities at the organization level provide the foundation for activities at the mission/business process and system levels, such as the selection and implementation of security controls and decisions about the operation of systems based on a determination of risk. Figure 1 illustrates an organization-wide approach to cybersecurity risk management. Guidance for federal agencies’ cybersecurity risk management processes is found in a suite of NIST special publications. Table 3 highlights key NIST cybersecurity risk management publications. OMB and NIST guidance identify practices for establishing agency-wide cybersecurity risk management programs. Among other things, these activities are intended to facilitate better communication between senior leaders and executives and system owners and operators; align agency priorities with resource allocation and prioritization at the system level; and convey acceptable limits regarding the selection and implementation of controls within the established organizational risk tolerance. Practices that provide a foundation for an agency’s cybersecurity risk management program are summarized in table 4. Establish the role of a cybersecurity risk executive: In order to ensure that cybersecurity risks are being addressed across the agency, NIST Special Publication 800-39 states that agencies should establish a cybersecurity risk executive. This can take the form of an individual or group that provides agency-wide oversight of cybersecurity risk activities and facilitates collaboration among stakeholders and consistent application of the cybersecurity risk management strategy. The cybersecurity risk executive should ensure that risk-related considerations for information systems are viewed from an agency-wide perspective regarding the strategic goals and objectives. The cybersecurity risk executive also should ensure that cybersecurity risk is managed consistently across the agency, reflects organizational risk tolerance, and is considered along with other types of risk to ensure mission/business success. Develop a cybersecurity risk management strategy: According to NIST Special Publication 800-39 and other guidance, agencies should develop a cybersecurity risk management strategy to provide a foundation for managing risk and delineate the boundaries for risk-based decisions. The strategy should describe the strategic-level decisions and considerations that senior leaders and executives are to use to manage security and privacy risks to agency operations, assets, individuals, other organizations, and the nation. The strategy should also guide and inform how security and privacy risks are framed, assessed, responded to, and monitored. The strategy should include (1) a statement of the agency’s risk tolerance, (2) how it intends to assess risk (e.g., acceptable risk assessment methodologies), (3) acceptable risk response strategies (e.g., acceptance, mitigation, avoidance), and (4) how the agency intends to monitor risk over time. Document risk-based policies: NIST Special Publication 800-37 identifies foundational activities at the agency and information system levels that should be included in policies to help prepare agencies to manage security and privacy risks. These activities should be guided by risk-based decisions. Specific elements of such risk-based policies include (1) identifying and assigning individuals with key roles for executing the risk management framework; (2) requiring an agency-wide assessment of cyber risks; (3) identifying and documenting common security controls that can be inherited by multiple information systems; (4) developing an agency-wide strategy for monitoring control effectiveness; (5) requiring system-level risk assessments to be performed and regularly updated; (6) tailoring system security controls based on risk; (7) prioritizing remedial actions to correct vulnerabilities identified in plans of action and milestones (POA&M) based on risk; and (8) using a determination of risk to make decisions about system operation and use. Conduct an agency-wide cybersecurity risk assessment: According to NIST Special Publications 800-39 and 800-37, agencies should assess cybersecurity and privacy risks and update the results on an ongoing basis. Risk assessment at the agency level is based primarily on aggregated information from system-level risk assessment results, continuous monitoring, and any relevant strategic risk considerations. The assessment is intended to help the agency consider the totality of risk derived from the operation and use of its information systems and from information exchanges and connections with other internally and externally owned systems. Such assessments may identify systemic weaknesses or deficiencies discovered in multiple information systems and assess the overall risks that these present to operations, assets, and individuals. Establish coordination between cybersecurity and enterprise risk management: ERM, as a discipline, deals with identifying, assessing, and managing risks. OMB has stated that an effective enterprise risk management program should promote a common understanding for recognizing and describing potential risks that can impact an agency’s mission and the delivery of services to the public. Such risks include strategic, market, cyber, legal, reputational, political, and a broad range of operational risks. Toward this end, OMB Circular A-123 directs agencies to implement a capability for enterprise risk management. Specifically, it encourages agencies to establish a risk management governance structure, such as a risk management council, which may be integrated with existing management structures; develop “risk profiles” that identify risks arising from mission and mission-support operations; and consider those risks as part of the annual strategic review process. Because cybersecurity is a key risk facing virtually every federal agency, it is important for coordination to exist between agencies’ ERM functions and their cybersecurity risk management programs, particularly the cybersecurity risk executive. NIST SP 800-39 states that effective risk management requires an agency’s mission/business processes to explicitly account for information security risk when making operational decisions and that cybersecurity risk information should be shared with key stakeholders throughout the organization. According to NIST, the risk executive should serve as a common risk management resource for senior leaders, mission/business owners, and other organization officials and as a focal point for communicating and sharing information security risk-related information among key stakeholders. OMB has also raised concerns that agencies’ ERM programs do not effectively identify, assess, and prioritize actions to mitigate cybersecurity risks in the context of other enterprise risks. GAO has also emphasized the importance of sharing risk information with stakeholders as part of an effective risk management program. The 23 civilian CFO Act agencies varied in the extent to which they had established key elements of their cybersecurity risk management programs. Specifically, 22 of the 23 agencies established the role of cybersecurity risk executive, and most of the 23 agencies had established policies that include elements to ensure their activities are guided by risk- based decisions. However, fewer than half of the agencies developed an agency-wide cybersecurity risk management strategy or fully established coordination with their enterprise risk management function. Figure 2 summarizes the extent to which the agencies had established these elements as of April 2019. Twenty-two of the 23 civilian CFO Act agencies established a cybersecurity risk executive to provide agency-wide oversight of cybersecurity risk activities. Agencies varied in assigning this responsibility to the chief information officer (CIO), chief information security officer (CISO), or another official or entity. For example: At the Department of Health and Human Services (HHS), the CIO serves as the risk executive for the department, and is responsible for executing the Risk Management Framework tasks outlined in NIST SP 800-37. The United States Agency for International Development (USAID) designated the CISO with responsibility for carrying out the risk executive functions for the agency. Among other things, the CISO is responsible for developing, implementing, and managing an agency- wide security authorization process and a threat awareness program. The Department of the Treasury (Treasury) assigned the function of risk executive to its department CIO Council. The council’s responsibilities include ensuring the cybersecurity program is consistent with the provisions of NIST SP 800-39; providing guidance to and oversight of the organization’s risk management program and developing the cybersecurity risk management strategy; communicating organization-wide threat, vulnerability, and risk-related information; and providing a strategic view for managing cyber risk throughout the organization. One agency, the General Services Administration (GSA), had not defined the role of its cybersecurity risk executive in its policy. Officials in GSA’s Office of the CIO stated that they had not formally designated this role because the agency’s risk executive responsibilities were shared among the CIO, CISO, authorizing officials, and other GSA officials for risk management. However, without clearly defining and documenting the responsibility for the risk executive function, the agency may lack consistent implementation and oversight of cybersecurity risk management activities and an effective agency-wide view for managing risk. Additional details on the 23 agencies’ cyber risk executive positions are provided in appendix II. Among the 23 civilian CFO Act agencies, seven had developed a cybersecurity risk management strategy that fully addressed the four elements called for in the NIST guidance. Specifically, each of the seven agencies (the Department of Commerce (Commerce), the Department of Labor (Labor), the Department of State (State), USAID, GSA, OPM, and the Social Security Administration (SSA)) had developed a strategy to guide how cybersecurity risk is to be framed, assessed, responded to, and monitored. For example, some of the strategies discussed risk tolerance in terms of thresholds based on essential mission functions and the processing of personally identifiable information or system impact levels, types of data processed, and accessibility of systems, among other factors. The strategies also included breakdowns of appropriate risk response strategies and how the agencies intended to assess and monitor risk. In addition, five of the 23 agencies (the Department of Education (Education), Environmental Protection Agency (EPA), National Science Foundation (NSF), the Department of Transportation (Transportation), and the Small Business Administration (SBA)) had partially developed cybersecurity risk management strategies, but their strategies did not address certain required elements. Specifically, while these agencies developed strategic documents, these documents did not include all of the required elements, such as a statement of risk tolerance or acceptable risk mitigation strategies. EPA officials stated that they intended to update their strategy documents to address how the agency intends to assess risk, while Education and NSF officials did not state whether they intended to update their strategy to include a statement of risk tolerance, among other missing elements. Transportation and SBA officials stated that they believed their existing strategy documents addressed all the elements; however, neither agency’s strategy included an expression of departmental risk tolerance and risk mitigation strategies. Further, Transportation’s strategy did not include a description of acceptable risk assessment methodologies. The remaining 11 agencies had not developed an agency-wide cybersecurity risk management strategy. These agencies offered a variety of reasons for not doing so. Seven agencies—the Department of Agriculture (Agriculture), Department of Energy (Energy), HHS, Department of the Interior (Interior), Treasury, the National Aeronautics and Space Administration (NASA), and the Nuclear Regulatory Commission (NRC)—acknowledged that they had not developed a cybersecurity risk management strategy that includes the key elements. According to agency officials, this was due to the federated nature of the agency or difficulty in establishing an agency-wide understanding of risk tolerance, among other factors. Further, these agencies stated that they intended to develop such a strategy or were considering doing so. The other four agencies—DHS, the Department of Housing and Urban Development (HUD), Department of Justice (Justice), and Department of Veterans Affairs (VA)—stated that they believed their existing documents and policies constituted a risk management strategy. However, we determined that these documents did not constitute an integrated strategy that addressed key elements such as risk tolerance and risk mitigation strategies. Without a comprehensive risk management strategy, the agencies may lack an organization-wide understanding of acceptable risk levels and appropriate risk response strategies to protect their systems and data. Additional details regarding the 23 agencies’ establishment of cybersecurity risk management strategies are discussed in appendix III. Most of the 23 agencies had established policies that include elements to ensure their activities are guided by risk-based decisions. However, many agencies had gaps in one or more of these areas. Specifically, six agencies (DHS, Education, Justice, Treasury, NSF, and SSA) addressed all of these areas in their policies and procedures, while the remaining 17 agencies had not addressed at least one area. Table 5 discusses, for each of these elements, which of the 23 agencies had addressed it in their policies. Eleven agencies—Agriculture, Commerce, Energy, HHS, Interior, Labor, EPA, GSA, NASA, NRC, and OPM—generally agreed that their policies lacked identified elements and either stated that they intended to update policies to include them or would consider doing so. The remaining six agencies—HUD, State, Transportation, VA, USAID, and SBA—stated that they believed their policies addressed these elements or that they carried out these activities in practice, but did not provide documentation of policies that addressed them. Without ensuring that their policies include all key risk management activities, the agencies may not be taking the foundational steps needed to effectively identify and prioritize activities to mitigate cybersecurity risks that could result in the loss of sensitive data or compromise of agency systems. Additional details on the agencies’ risk management policies are provided in appendix IV. Twelve of the 23 civilian CFO Act agencies had developed a process or mechanism for conducting an agency-wide cybersecurity risk assessment. Specifically, these agencies (Agriculture, Education, Energy, DHS, HUD, Interior, Justice, Labor, State, Transportation, NSF, and SSA) had developed processes for aggregating system-level data and analyzing them to assess overall cybersecurity risk to agency operations and assets. For example, these 12 agencies developed scorecards or dashboards that provided agency-wide views of key indicators aggregated from system-level information and risk scores for agency components. Officials from seven of these agencies described how these assessments enable them to make enterprise-wide decisions on prioritizing and remediating risks. The remaining 11 agencies (Commerce, GSA, HHS, NASA, NRC, Treasury, VA, EPA, OPM, SBA, and USAID) offered a variety of reasons for why they did not develop a process for assessing cybersecurity risks at the agency level. Five agencies stated that they were still working to develop or acquire tools that will allow them to aggregate system-level data, and three of these noted that they expected further implementation of DHS’s CDM initiative to provide this capability. The other six agencies stated that they did conduct such an assessment in practice, but did not provide sufficient documentation of the process they use. Without a means of aggregating and assessing cybersecurity risks arising from their information systems to the organizational level, these 11 agencies may be missing opportunities to identify trends or prioritize investments in cybersecurity risk mitigation activities in order to target widespread or systemic risks to the systems and organization. Additional details of agencies’ processes for conducting organization-wide cyber risk assessments are contained in appendix V. Ten of the 23 civilian CFO Act agencies provided evidence of having a fully established process for coordination between their cybersecurity risk executive and the entity responsible for overall ERM functions. Five agencies provided evidence of a partially established process, and eight could not provide evidence of such a process. The ten agencies with fully established processes included this coordination as part of their defined and documented ERM governance structure and process. The agencies took steps to ensure such coordination in a variety of ways. For example, eight agencies, including Education and USAID, established a specific body, such as a risk management council, with responsibility for ERM. These agencies included their cybersecurity risk executive in the council’s membership in order to facilitate coordination. Other agencies, such as the National Science Foundation, ensured coordination through regular reporting or briefings between their cybersecurity risk executive and their ERM governance structure. In addition, five agencies partially established an approach to coordination in this area. These agencies provided some evidence of coordination activities, but had not formally defined or documented this coordination as part of their ERM structure or process. Specifically, four of these agencies (Justice, the Department of Transportation (Transportation), the Environmental Protection Agency (EPA), and the Social Security Administration (SSA)), provided evidence of occasional coordination between their cybersecurity risk executive and officials responsible for ERM. However, they did not fully define and document their ERM governance structures and processes, including how coordination with the cybersecurity risk executive was to take place. One agency—GSA—had not formally documented the position or responsibilities of the cybersecurity risk executive in its policy. Thus, the agency could not show that the risk executive was involved in ERM activities, although the agency board responsible for ERM does include the agency CIO as a co-chair. Although they did not provide evidence of a fully documented process, officials from these five agencies stated that they perform this coordination in practice. However, documenting these processes would help ensure a consistent, rather than ad-hoc, approach to communication and coordination. Lastly, eight agencies had not established an approach to coordination in this area. In particular, these agencies (Agriculture, HHS, Interior, VA, DHS, State, Treasury, and NRC) either did not have an ERM governance structure and/or did not provide evidence of a process for coordination between their ERM governance structure and their cybersecurity risk executive. Officials from two of these agencies stated that they were still in the process of formalizing their approach to ERM, while the other six stated that such coordination occurs, even if processes may not be fully documented. However, as noted previously, documenting these processes would help ensure a consistent, rather than ad-hoc, approach to communication and coordination. Without regular coordination between the cybersecurity risk executive and broader ERM entity, senior leadership responsible for ERM may not be fully aware of significant cybersecurity risks and, thus, may not be positioned to address them in the context of other risks and their potential impacts on the mission of the agency. Additional details on agencies’ coordination processes are provided in appendix VI. Officials responsible for cybersecurity risk management at a majority of the 23 civilian CFO Act agencies reported eight challenges in establishing and implementing cybersecurity risk management programs. Most commonly cited were challenges related to hiring and retaining qualified personnel, competing priorities between cybersecurity and agency mission or operations, and establishing and implementing consistent cybersecurity risk management policies and procedures. Figure 3 shows the challenges identified and the number of agencies reporting each challenge. All of the 23 civilian CFO Act agencies reported hiring and retaining personnel to fill key cybersecurity risk management positions as a challenge in establishing a cybersecurity risk management program. In particular, six agencies cited the lengthy federal hiring process, and 14 noted the difficulty in competing with private-sector companies in salary and other benefits. Further, 11 agencies noted that there is a shortfall in candidates with the skills needed for cybersecurity risk management. For example: NASA’s Chief Cyber Risk Officer noted that cybersecurity risk management is a multi-disciplinary field that blends technical cyber expertise with project management principles and a business-focused management background. This official stated that it is difficult to find talent that possesses this multi-disciplinary experience, in part, because current government marketing for cybersecurity skill sets advertise for purely technical skills. The official added that, currently, the government lacks clearly defined roles for cyber risk management as a dedicated job function. HUD’s CIO saw this challenge as part of a larger shortfall of this highly in-demand resource and noted that HUD must compete with tech giants and Silicon Valley startups for qualified personnel. The official stated that the executive order providing direct hiring authorities for cybersecurity positions provides assistance, though the department still needs to be creative in enhancing retention and recruitment efforts through bonuses and other incentives. A key to having a successful cybersecurity program is having a well- trained, highly qualified workforce that is versed in identifying cyber threats and recognizes steps to take once confronted with them. Our work has identified difficulties in recruiting and retaining qualified cybersecurity professionals as a continuing challenge. If agencies are unable to hire and retain qualified cybersecurity risk management personnel, they will be hindered in establishing effective programs for cybersecurity risk management. Nineteen of the 23 civilian CFO Act agencies reported competing priorities between agency mission operations and cybersecurity as a challenge. In particular, 12 agencies noted that cybersecurity requirements are sometimes perceived as impeding mission activities, such as deploying systems, sharing information, or providing public services. In addition, four agencies highlighted the competition for limited resources between cybersecurity risk management activities and operational or mission needs. For example: HHS’s Acting Deputy CISO stated that, due to the federated nature of the agency and the broad spectrum of its missions and business functions, there is often a disconnect between security and operational personnel. As an example, the official stated that Operating Divisions that are research or academics focused will require increased information sharing and flexibility, but this often conflicts with cybersecurity concepts and processes. Interior’s Deputy CIO stated that the need to balance mission priorities with those related to cybersecurity risk management leads to fiscal and operational challenges when making investment, architectural, and operational decisions. NIST emphasizes determining the relative importance of the mission/business functions in order to make the appropriate level of risk management investment. If agencies are unable to establish priorities among cybersecurity and operational needs, they may be challenged in allocating resources appropriately to ensure their systems and information are appropriately secured. Eighteen of the 23 civilian CFO Act agencies reported challenges in establishing and implementing consistent cybersecurity risk management policies and procedures across the organization. Eight agencies cited challenges in this area arising from the difficulty in ensuring consistency across a federated or decentralized organization, while other factors included training staff and making them aware of policies, and the need to integrate cybersecurity policies with missions and operations. For example: EPA’s CISO related that challenges in consistent implementation of policies and procedures include the need to train individuals involved in the risk management process, address different views of risk appetite within the agency, and deal with varying perspectives on the importance of cybersecurity, among other things. OPM’s Deputy CISO highlighted that frequent changes in the agency’s leadership (e.g., having eight CIOs since 2012) had led to challenges with the agency’s ability to implement consistent policies in an ongoing, streamlined manner. As we have previously reported, CIOs and former agency IT executives believed it was necessary for a CIO to stay in office for 3 to 5 years to be effective and 5 to 7 years to fully implement major change initiatives in large public sector organizations. In addition, the Deputy CISO stated that the establishment and implementation of cybersecurity risk management policies and procedures has been viewed as a secondary responsibility, to be accomplished when more pressing and immediate operational concerns do not need attention. NIST has emphasized the importance of a consistent approach in order for cybersecurity risk management to succeed at all levels of an agency. If agencies are unable to establish consistent cybersecurity risk management policies and procedures, they may not be able to effectively prioritize and implement security and privacy activities to protect their most critical assets and systems. Eighteen of the 23 civilian CFO Act agencies reported challenges in establishing and implementing standardized IT capabilities across the organization. Eleven of these agencies noted that decentralized or federated organizations create difficulty in implementing standardized, agency-wide tools and solutions to manage cybersecurity risks. In addition, four agencies cited issues with legacy systems, which may not always be compatible with capabilities intended to be used agency wide. For example: The Department of Commerce’s (Commerce) Deputy CISO stated that, because Commerce is a largely federated agency, with each bureau operating and maintaining its own environment, managing a truly enterprise solution is challenging in numerous areas. For example, the official stated that the department cannot control access at bureaus due to disconnected networks, different security offices and policies, and even different logical access policies. The official added that a change in governance and thinking toward common enterprise tools and solutions requires a shift in management and thinking across the department and its bureaus. Energy’s Acting Deputy CIO for Cybersecurity stated that the department is working, to the degree possible, to implement enterprise solutions for cybersecurity and continuous monitoring; however, because the enterprise is comprised of laboratories and sites with very diverse mission sets, doing so is always challenging. This official added that the department has embraced the DHS CDM initiative, which will be leveraged to standardize some IT cybersecurity capabilities, but it does not have a single standardized solution across the enterprise. OMB recently noted that an agency’s ability to mitigate security vulnerabilities becomes more complex in federated agencies, where there are not standardized procedures or technology across the organization. The challenges in implementing standardized IT capabilities may hinder these agencies in applying a consistent level of protection to their systems and data. Eighteen of the 23 civilian CFO Act agencies reported that they had experienced challenges in receiving quality data (e.g., accurate, timely information on threats and vulnerabilities). Twelve of these agencies expressed challenges in receiving data from all parts of their agencies or stated that they relied on manual reporting from their components, which did not provide real-time visibility into risks. In addition, six agencies cited difficulties in combining data from disparate sources into an agency-wide view of risk. For example: DHS’s Acting Director of Governance and Executive Management noted that the department’s management currently depends on its components to submit timely and accurate information on cybersecurity vulnerabilities instead of having real-time, centralized reporting of data. The official added that DHS expects to address this challenge through implementation of CDM centralized reporting to the DHS Dashboard on a near real-time basis and other tools and processes for enterprise data collection. State’s Enterprise Risk Officer for Cybersecurity reported that threat information is difficult to gather with the specificity needed to make strategic decisions. The official added that, with regard to vulnerability data, sufficient data exist and are gathered on a regular basis; however, it is difficult in a large global enterprise to prioritize actions without credible information on the likelihood of a threat or its impact on the agency’s mission. NIST emphasizes that risk monitoring tools, techniques, and procedures can increase risk awareness and help senior leaders develop a better understanding of the ongoing risk to organizational operations and assets. If the agencies are unable to consistently receive quality, timely data from their entire organizations, they will continue to be challenged in making effective decisions to address organization-wide cybersecurity risks. Sixteen of 23 civilian CFO Act agencies reported the lack of sufficiency, clarity, or usefulness of NIST and/or OMB guidance for cybersecurity risk management as a challenge. Six agencies stated that there was a lack of practical instruction to assist agencies in implementing guidance. Six agencies also stated that various guidance documents are not always consistent or easy to understand. Six agencies also expressed a need for guidance to address new technologies or emerging areas such as the use of cloud providers or establishing cybersecurity risk management programs at all levels of an organization. For example: HHS’s Acting Deputy CISO stated that, for all the positive aspects of the NIST guidance, there is a lack of a centralized document or road map that ties all the documents together from a cybersecurity standpoint. Also, the official stated that the guidance from NIST provides limited direction for producing specific metrics and checklists in support of laws, policies, directives, instructions, and standards. Transportation’s CISO stated that current guidance does not always provide agencies with practical ways to implement requirements. For example, the official noted that current OMB guidance on cyber and privacy risk management does not tell agencies how to practically integrate these disciplines, and that frequent updates to NIST guidance that agencies have to respond to might be better applied to identifying practical implementations. The official added that a lack of practical implementation guidance may lead to duplication of effort and inconsistency of outcomes. OMB and NIST play important roles in issuing policies, standards, and guidelines for agencies’ cybersecurity risk management programs. However, if agencies find guidance unclear or insufficient, they will be challenged in implementing key cybersecurity risk management requirements. Fifteen of the 23 CFO Act agencies reported challenges in developing an agency-wide cybersecurity risk management strategy that includes a statement of risk tolerance and how the agency will assess, respond to, and monitor risks. Ten agencies stated that they faced challenges in establishing an agency-wide risk tolerance statement, while five noted that they faced challenges in implementing a strategy across the agency. For example: Education’s Audit Liaison Officer from its Office of the CIO noted that it was a challenge to develop an enterprise-level statement of risk tolerance and that currently risk tolerance decisions were made at the system level by the authorizing official. EPA’s CISO reported that it was challenge to establish an agency- wide statement of risk tolerance. This is because it was difficult to determine such factors as how much the mission’s operation is worth, how much information resources are worth, and how much negative public perception of the agency costs in terms of money or resources. NIST notes that framing risk through the creation of a cybersecurity risk management strategy establishes a foundation for managing risk and delineates the boundaries for risk-based decisions within an agency. If agencies are challenged in developing cybersecurity risk management strategies, they may be hindered in making consistent decisions for identifying, assessing, and responding to cybersecurity risks. Fourteen of the 23 civilian CFO Act agencies reported that incorporating cyber risks into the enterprise risk management process was a challenge. Nine of these agencies noted challenges related to coordination between cybersecurity and ERM, such as establishing effective channels of communication or developing vocabularies for discussing risk that were understandable by all stakeholders. In addition, five agencies noted that their ERM process was still maturing. For example: GSA’s Associate Chief Information Officer for Enterprise Planning & Governance stated that a process was implemented to assess cyber risks as part of the formalized ERM process; however, this official noted that additional work is still needed to align and incorporate other regular cybersecurity risk management reporting processes and communication channels into the broader ERM framework. Treasury’s Enterprise Cybersecurity Risk Management Officer stated that incorporating cyber risks into ERM is a challenge because cybersecurity risk is not currently quantified in the same way as other risks. The official expressed the need for a standard vocabulary for discussing cyber alongside other risks, adding that this makes it very challenging to integrate cybersecurity risk management into ERM. OMB has stated that an effective enterprise risk management program promotes a common understanding for recognizing and describing potential risks that can impact an agency’s mission and the delivery of services to the public. Such risks include strategic, market, cyber, legal, reputational, political, and a broad range of operational risks. If agencies do not successfully integrate cyber risks into their ERM processes, they may be hindered in making effective decisions about addressing cybersecurity risks in the context of other risks and their potential impact on agency missions. In accordance with a recent executive order, OMB and DHS took steps to assess agencies’ cybersecurity management capabilities. They also identified core actions to be taken, in coordination with agencies, to address cybersecurity risks across the executive branch. Accordingly, OMB and DHS have several initiatives under way to address these risks, and several of these initiatives should help address some of the challenges in establishing cybersecurity risk management programs that the agencies in our review identified. However, these initiatives do not address other challenges identified by a majority of the agencies. EO 13800 on Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure emphasizes the importance of reducing cybersecurity risks while also providing exceptional service to the public. The EO aligns with FISMA by holding agency heads accountable for managing cybersecurity risks. Toward this end, it directed agency heads to provide a risk management report to OMB and DHS that documented the agency’s risk mitigation and acceptance choices as of May 2017 and describe the agency’s action plan to implement the NIST cybersecurity framework. The EO required OMB and DHS to assess each agency’s risk management report and OMB, in coordination with DHS, to develop and deliver a risk determination report to the President on whether the risk mitigation and acceptance choices set forth in the agencies’ reports were appropriate and sufficient to manage the cybersecurity risk to the executive branch as a whole. OMB’s and DHS’s report was also to include an action plan to, among other things, adequately protect the executive branch, should the risk determination identify insufficiencies in agencies’ risk mitigation and acceptance choices; establish a regular process to reassess and, if appropriate, reissue the determination and address future recurring and unmet budgetary needs necessary to manage risk to the executive branch; and if appropriate, clarify, reconcile, and reissue policies, standards, and guidelines issued in furtherance of FISMA and the EO, and align them with the NIST cybersecurity framework. In May 2017, OMB issued guidance to agencies for implementing the provisions in EO 13800 on managing cybersecurity risks. This guidance required agencies to, among other things, report on their cybersecurity risk management capabilities using the metrics established for monitoring FISMA implementation. OMB and DHS used the results of the agencies’ risk management reports and responses to the FISMA reporting metrics to assess agencies’ capabilities and make risk determinations of agencies’ performance (“high risk,” “at risk,” or “managing risk”). OMB and DHS’s process included an assessment of 96 agencies across the executive branch, including the 23 civilian CFO Act agencies in the scope of our review. In May 2018, OMB published the Federal Cybersecurity Risk Determination Report and Action Plan, in which OMB and DHS determined that 74 percent of the federal agencies participating in the risk assessment process had cybersecurity programs that were either “at risk” or “high risk.” The report identified four key findings and actions necessary to address cybersecurity risks across the federal enterprise, as summarized in table 6. The report also described OMB’s plans to work with DHS and other federal entities to implement these actions and reduce cybersecurity risks across the government. OMB and DHS also established a process for reassessing and, if necessary, reissuing the agency risk determinations. Specifically, OMB and DHS use the metrics collected during the FISMA reporting process to update each agency’s risk management assessment on an ongoing basis. At a minimum, CFO Act agencies must update their metrics quarterly. The quarterly risk management assessment process allows for the monitoring of agency-level risks, and OMB issues guidance yearly codifying this process. In addition, OMB staff stated that they plan to incorporate the overall risk determination into the office’s annual FISMA report to Congress, although they noted that this is subject to change. Further, OMB and DHS took steps to align government-wide cybersecurity guidance with the NIST cybersecurity framework. For example, OMB and DHS, in coordination with the federal cybersecurity community, updated the reporting guidance on CIO and Inspector General FISMA metrics to align with the framework. The FISMA metrics leverage the framework as a standard for managing and reducing cybersecurity risks, and the metrics are aligned with the five main functions of the framework to provide agencies with a comprehensive structure for making more informed, risk-based decisions, managing cybersecurity risks across their enterprise, and providing a view of agencies’ capabilities and potential gaps. OMB and DHS have several initiatives under way—some of them also outlined in OMB’s federal cybersecurity report—that can assist agencies in meeting challenges related to hiring and retaining cybersecurity risk management personnel, establishing standardized IT capabilities, receiving quality data, and using NIST and OMB guidance. Workforce education initiatives: In November 2018, OMB announced the launch of the Federal Cyber Reskilling Academy pilot program, which is being sponsored by the CIO Council. This program offers current federal employees who do not work in the IT field the opportunity for hands-on training in cybersecurity for 3 months to help them build foundational skills in cyber defense analysis. In addition, the National Initiative for Cybersecurity Careers and Studies is an online resource for cybersecurity training managed by DHS that connects government employees, students, educators, and industry with cybersecurity training providers throughout the nation. The initiative’s Federal Virtual Training Environment, for example, is an on-demand cybersecurity training system that contains more than 800 hours of training on a variety of topics, including risk management. These initiatives, if effectively implemented, could help address challenges agencies identified in hiring and retaining cybersecurity risk management personnel. Specifically, the Cyber Reskilling Academy has the potential to increase the pool of federal employees with skills that agencies need for cyber risk management. In addition, the Federal Virtual Training Environment can enhance federal employees’ knowledge of and skills in cybersecurity risk management. Continuous Diagnostics and Monitoring (CDM): DHS’s CDM initiative is to provide federal agencies with tools and services that have the intended capability to automate network monitoring, correlate and analyze security-related information, and enhance risk- based decision making at agency and government-wide levels. These tools include sensors that perform automated scans or searches for known cyber vulnerabilities, the results of which can feed into a dashboard that, at an agency level, is intended to alert network managers and enable the agency to allocate resources based on the risk. Summary data from each participating agency’s dashboard is expected to be transmitted to the Federal Dashboard, where the data can be used to inform decisions about cybersecurity risks across the federal government. A DHS CDM program official stated that the department plans to continue to deploy capabilities in fiscal year 2019 for asset management, identity and access management, and monitoring network controls and activity. The CDM initiative, if effectively implemented, has the potential to assist in addressing challenges agencies identified in establishing standardized IT capabilities for cybersecurity risk management and improving the quality of data to provide visibility into cyber risks. In particular, the tools and services offered through the program can provide agencies with standardized capabilities for collecting and analyzing cyber risk information. In addition, automated network monitoring and analysis can help agencies that currently must manually collect data from components based on self-reporting. Such data may be less timely and accurate than those collected through the tools available through CDM. Security operations center (SOC) consolidation and maturation: A SOC defends an organization against unauthorized activity within computer networks, including, at a minimum, detecting, monitoring, and analyzing suspicious activity. According to OMB, CISOs report that these centers do not communicate with each other and that they hoard, rather than share, threat information and intelligence. SOC consolidation focuses on centralizing information sharing across the agency, which is intended to improve the data agencies receive to provide visibility into cybersecurity risks. OMB and DHS are working with agencies to assess and enhance the maturity of their SOCs and streamline security operations across their enterprise. Specifically, agencies are required to develop and submit a Cybersecurity operations maturation plan to OMB and DHS by April 2019. Following submission of the plan, agencies are then required to complete SOC maturation, consolidation, or migration to a SOC-as-a-Service provider by September 2020. Similar to CDM, SOC consolidation and maturation initiatives may help address challenges related to standardizing capabilities and collecting quality data, while enhancing enterprise-wide visibility. Consolidation can provide agencies with a standardized set of SOC services, while maturation can increase the quality of data on risks by establishing a baseline set of expected SOC capabilities for executive branch agencies. Cyber threat framework: OMB and DHS are developing and disseminating a framework, working with the Department of Defense, Office of the Director of National Intelligence, and the National Security Agency, to enable consistent characterization and categorization of cyber threat events. Specifically, the Cyber Threat Framework provides a hierarchical, structured, transparent, and repeatable methodology for characterizing adversarial activities in a standardized way across the federal government. The framework and the related methodology provide for a cybersecurity architecture review that allows an agency to assess its cyber capabilities against its actual threat environment. This includes a gap analysis to determine where agencies may need to enhance their capabilities to defend against key threats. To foster the adoption of the Cyber Threat Framework across the government, DHS—in coordination with OMB and the Department of Defense—intends to develop and implement a solution that will be available for agencies to use by the end of December 2019. The Cyber Threat Framework, if effectively implemented by civilian federal agencies, can also help address agency challenges related to the quality of data about cyber risks. By providing a standardized framework for understanding cyber threats, it is intended to assist agencies to better identify and prioritize risks, as well as the gaps in their capabilities for protecting against such threats. Inter-agency cyber-focused working groups: In coordination with DHS, OMB established CyberStat review sessions to assist agencies in protecting their systems, networks, and data. Specifically, agency cyber professionals, from the working level to the CIO, meet with DHS subject matter experts to participate in working sessions throughout a 4- to 6-week period to overcome barriers to success in specific cybersecurity programs. During a CyberStat review, DHS provides agencies with guidance on best practices and connects them with other subject matter experts who can provide advice on implementing the NIST framework and cybersecurity risk management practices. In addition, the federal CIO Council has recently issued the CISO Handbook, which was created to educate and inform new and existing CISOs about their role in federal cybersecurity. The council is the principal interagency forum for improving agency practices related to the use, sharing, and performance of federal information resources and part of its governing principles are to adopt and share IT management best practices and to manage risk and ensure privacy and security. Within the CIO Council, the CISO Council is specifically tasked with developing IT security policy and sharing best practices to improve the cybersecurity posture of the United States. Among other things, the CISO Handbook includes information on NIST’s cybersecurity framework and how it can be leveraged in conjunction with other NIST risk management publications. CyberStat reviews and the federal CIO Council can provide channels to help agencies in better understanding and implementing guidance from NIST and OMB on cybersecurity risk management. By connecting agencies with best practices and subject matter experts, CyberStat sessions are intended to help agencies, for example, apply the NIST framework and cyber risk management practices. In addition, the CIO Council, through sharing of best practices and issuing publications, can provide guidance on how to more effectively implement federal cybersecurity risk management guidance. Although the initiatives under way could address challenges related to hiring and retaining cybersecurity risk management personnel, developing standardized capabilities, acquiring quality data about cyber risks, and using NIST and OMB guidance, the existing initiatives do not address challenges related to managing competing priorities, establishing consistent policies and procedures, incorporating cyber risks into enterprise risk management, and developing an agency-wide strategy for managing cybersecurity risks. Managing competing priorities between cybersecurity and operations: OMB staff stated that its newly developed risk-based budgeting model could help agencies prioritize their cybersecurity investments. This model is intended to tie agencies’ cybersecurity spending to the FISMA metrics process in order to identify capability and process gaps that pose risks to an agency. OMB plans to disseminate the risk-based budgeting process to enable agency CIOs, CISOs, and Chief Financial Officers to communicate cyber risks effectively across their agencies and to budget strategically for cyber capabilities that address the agency’s most critical cybersecurity needs. OMB anticipates being able to provide agencies with additional details surrounding this model in the cybersecurity section of its upcoming fiscal year 2020 guidance to the President’s budget. However, while this risk-based approach to cybersecurity budgeting should help agencies prioritize their cybersecurity investments, it does not address issues related to prioritizing between cybersecurity and mission or operational needs. The agencies in our review highlighted that mission or operational priorities can conflict with cybersecurity requirements when, for example, components within an agency have differing views about the relative importance of mission and cybersecurity activities. These issues do not relate to prioritizing investments in cybersecurity but to managing conflicts, or potential conflicts, between cybersecurity and mission needs. Implementing consistent cybersecurity risk management policies and procedures: OMB staff stated that several of OMB’s and DHS’s initiatives emphasize driving performance through centralized visibility, authority, and reporting. For example, OMB staff stated CDM is intended to establish agencies’ visibility across the enterprise, as well as government-wide visibility. OMB staff stated the implementation of provisions commonly referred to as the Federal Information Technology Acquisition Reform Act is intended to enhance the role and authority of agency CIOs, particularly with respect to relationships with agency components and accountability for IT costs, performance, and security. Additionally, OMB staff stated the risk management assessment process established in response to EO 13800 emphasizes centralized visibility, authority, and reporting. While these efforts could provide increased visibility and CIO authority, they do not address factors identified by agencies that affected their ability to implement consistent cybersecurity risk management policies and procedures. These include differing views among staff regarding the importance of risks, and frequent changes in leadership, all of which, according to agencies, make consistency difficult to achieve. Incorporating cyber risks into ERM: While existing OMB guidance requires agencies to establish ERM programs and NIST guidance requires agencies to establish cybersecurity risk management programs, this guidance does not address how these efforts should be integrated or coordinated. For example, OMB A-123 outlines agencies’ responsibilities for establishing an ERM capability but does not specifically address how enterprise risk management should incorporate cyber risks. In addition, NIST guidance on cybersecurity risk management recognizes that cybersecurity can be an important component of an organization’s overall risk management and states that its information security risk management guidance should be used as part of a more comprehensive ERM program. However, it does not explicitly discuss how to integrate or coordinate cybersecurity risk management and enterprise risk management. Establishing a cybersecurity risk management strategy: OMB noted that the cyber threat framework will provide a more tangible way for agencies to identify and prioritize cyber risks. However, while this framework will allow agencies to better identify and categorize threats and the capabilities needed to counter them, it does not address key aspects of risk framing such as establishing an agency-wide statement of risk tolerance and acceptable risk mitigation strategies. Several agencies noted that they struggled to define risk tolerance and establish criteria for different risk responses that could provide a consistent, agency-wide approach to risk management. Without additional guidance or other processes to identify successful approaches for addressing these challenges, agencies will continue to be hindered in establishing programs for effectively managing their cybersecurity risks. Given the increasing number and sophistication of cyber threats facing federal agencies, it is critical that agencies are well positioned to make consistent, informed risk-based decisions in protecting their systems and information against these threats. While all the agencies in our review have taken steps to establish cybersecurity risk management programs, they have not fully addressed key practices that are foundational to effectively managing cybersecurity risks. In particular, without developing an agency-wide cybersecurity risk management strategy, agencies may lack a consistent approach to managing cybersecurity risks. In addition, while agencies have documented policies and procedures that include many key practices, gaps remain that may hinder their ability to ensure a consistent implementation of risk-based practices. Further, without a process for an agency-wide cybersecurity risk assessment, agencies may be missing opportunities to identify risks that affect their entire organization, and to implement solutions to address them. Finally, establishing processes for coordinating cybersecurity risk information with the entity responsible for enterprise risk management would help ensure that cyber risks are being considered by senior leadership in the context of other risks facing the agency. This inconsistent establishment of cybersecurity risk management practices can be partially attributed to challenges agencies identified in establishing and implementing their cybersecurity risk management programs. Specifically, agencies noted a variety of challenges such as hiring qualified staff, competing priorities between cybersecurity and mission needs, implementing consistent policies and procedures, incorporating cyber risks into enterprise risk management processes, and developing a cybersecurity risk management strategy. Addressing these challenges will be an important step toward establishing more effective cybersecurity risk management programs across the 23 agencies. OMB and DHS have taken steps to carry out their responsibilities to identify and address weaknesses across the executive branch, including actions that would address many of the challenges identified by agencies. However, without fully addressing challenges related to prioritization between cybersecurity needs and mission priorities, implementing consistent risk management policies and procedures, incorporating cyber risks into enterprise risk management, and establishing a cybersecurity risk management strategy, OMB and DHS are likely to be missing opportunities to assist agencies in these key areas. Clarified or updated guidance, along with sharing successful practices or lessons learned, could help agencies more fully establish their cybersecurity risk management capacity. We are making the following recommendation to OMB: The Director of OMB should, in coordination with the Secretary of Homeland Security, establish guidance or other means to facilitate the sharing of successful approaches for agencies to address challenges in the areas of (1) managing competing priorities between cybersecurity and operations, such as when operational needs appear to conflict with cybersecurity requirements; (2) implementing consistent cybersecurity risk management policies and procedures across an agency; (3) incorporating cyber risks into enterprise risk management, and (4) establishing agencies’ cybersecurity risk management strategies. (Recommendation 1) We are also making a total of 57 recommendations to the 23 civilian CFO Act agencies in our review to fully address key practices in their cybersecurity risk management policies and procedures. Appendix VII contains these recommendations. We requested comments on a draft of this report from OMB and the 23 civilian CFO Act agencies included in our review. All the agencies provided responses, as further discussed. In an email from the office’s GAO audit liaison on July 8, 2019, OMB did not state whether it agreed or disagreed with our recommendations. However, the office provided technical comments, which we incorporated as appropriate. Of the 23 civilian CFO Act agencies, 17 agencies (Education, Energy, DHS, HUD, Interior, Labor, State, Transportation, VA, USAID, GSA, NASA, NSF, NRC, OPM, SBA, and SSA) concurred with our recommendations; one agency (HHS) partially concurred with our recommendations; three agencies (Commerce, Justice, and Treasury) provided comments but did not state whether they agreed or disagreed with our recommendations; and two agencies (Agriculture and EPA) stated that they had no comments on the report. Multiple agencies also provided technical comments, which we incorporated as appropriate. The following 17 agencies concurred with our recommendations and, in most cases, described steps planned or under way to address them: The Department of Education provided written comments in which it concurred with our recommendation and stated that the department will continue its efforts to fully develop a cybersecurity risk management strategy that includes the definition of risk tolerance and acceptable risk response strategies. Education’s comments are reprinted in appendix VIII. The Department of Energy provided written comments in which it concurred with our two recommendations and described steps and time frames for addressing them. In one case, regarding our recommendation to update the department’s policies to address missing elements, Energy stated that, as of May 2019, it had already completed an update of its policies to implement this recommendation. We intend to follow up with the department and obtain and assess evidence to determine its implementation of this recommendation. Energy’s comments are reprinted in appendix IX. In written comments, the Department of Homeland Security stated that it was pleased that our report noted steps that DHS and OMB have taken to improve agencies’ capabilities for managing cyber risks. DHS also concurred with our two recommendations and described steps it intends to take to address them, along with estimated completion dates. DHS’s comments are reprinted in appendix XI. The department also provided technical comments, which we have incorporated as appropriate. The Department of Housing and Urban Development provided written comments in which it thanked GAO for the opportunity to review the report and stated that it concurred with the recommendations. HUD’s comments are reprinted in appendix XII. The Department of the Interior provided written comments in which it concurred with our three recommendations. Interior also described planned steps to address the recommendations, such as developing a cybersecurity risk management strategy that includes the key elements and updating its policies. The department’s comments are reprinted in appendix XIII. In written comments, the Department of Labor concurred with our recommendation. Labor stated that it intends to take necessary steps to update the department’s policies. The department’s comments are reprinted in appendix XIV. The Department of State provided written comments in which it concurred with our two recommendations. State also described steps planned or under way to address the recommendations. For example, State described ongoing policy updates to address control monitoring, system-level risk assessments, and the use of risk assessments to inform control tailoring. It also described ongoing steps to align its cybersecurity risk management activities with its ERM governance structure. State’s comments are reprinted in appendix XV. The Department of Transportation’s Director of Audit Relations & Program Improvement provided comments via email on June 25, 2019, which stated that the department concurs with the findings and recommendations in the draft report. The Department of Veterans Affairs provided written comments in which it concurred with our four recommendations. VA also described actions planned or under way to address the recommendations. Regarding our recommendation to establish and document a process for coordination between its cybersecurity and enterprise risk management functions, the department stated that it had already established such a process and requested closure of the recommendation. We intend to follow up with the department and obtain and assess evidence to determine if its actions fully address our recommendation. VA’s comments are reprinted in appendix XVI. The U.S. Agency for International Development provided written comments in which it agreed with our two recommendations. USAID also described steps it has planned or under way to address the recommendations, such as amending its guidance to address an organization-wide cybersecurity risk assessment. The agency’s comments are reprinted in appendix XVII. In written comments, the General Services Administration stated that it appreciated the opportunity to review the report and concurred with its findings. The agency added that it is implementing an action plan to address the four recommendations. GSA’s comments are reprinted in appendix XVIII. The National Aeronautics and Space Administration provided written comments in which it concurred with our two recommendations. NASA also described planned steps to address the recommendations, such as updating its policies and establishing a process for an organization-wide cybersecurity risk assessment, along with estimated completion dates. The agency’s comments are reprinted in appendix XIX. The National Science Foundation’s GAO liaison provided comments via email on July 3, 2019, which stated that the agency concurred with our recommendation and intends to update its cybersecurity risk management strategy to address the missing elements. The Nuclear Regulatory Commission provided written comments in which it stated that the agency was in general agreement with the findings and recommendations in our draft report. NRC’s comments are reprinted in appendix XX. The Office of Personnel Management provided written comments in which it stated that it concurred with our two recommendations. OPM also described planned steps to address the recommendations, such as updating its policies and establishing a process for an organization- wide cybersecurity risk assessment. The agency’s comments are reprinted in appendix XXI. In written comments, the Small Business Administration concurred with our three recommendations. SBA described steps planned or under way to address the recommendations, such as updating its cybersecurity risk management strategy and policies and establishing a process for an organization-wide cybersecurity risk assessment, along with estimated completion dates. The agency’s comments are reprinted in appendix XXII. In written comments, the Social Security Administration agreed with our recommendation and described planned efforts to further integrate its cybersecurity and enterprise risk management functions. SSA’s comments are reprinted in appendix XXIII. One agency—the Department of Health and Human Services—concurred with three of our recommendations and partially concurred with one recommendation. Specifically, HHS concurred with our recommendations to develop a risk management strategy that includes key elements, establish a process for conducting an agency-wide cybersecurity risk assessment, and establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. Further, HHS described steps planned or under way to address these recommendations. Regarding our recommendation to update department policies to require an organization-wide cybersecurity risk assessment and the use of risk assessments to inform control tailoring, HHS stated that it concurred with the first part of the recommendation, but did not concur with the second part of the recommendation. Specifically, the department described steps it has planned or under way to update its policies to require an organization-wide risk assessment, in accordance with the first part of the recommendation. With respect to the second part of the recommendation, the department pointed to portions of its information security and privacy policy that address the selection of security and privacy controls. However, while these policy statements require adherence to NIST and OMB standards for selecting security controls and require a rationale for tailoring decisions, they do not specifically require the use of risk assessments to inform the tailoring of security controls. As NIST states, organizations apply the tailoring process to align the controls more closely with the specific conditions within the organization and should use risk assessments to inform and guide the tailoring process for organizational information systems and environments of operation. Making this requirement explicit in policy would help HHS ensure that it is applying the appropriate set of controls to its systems; thus, we maintain that our recommendation is still warranted. HHS’s comments are reprinted in appendix X. The department also provided technical comments, which we incorporated as appropriate. We received technical comments via email from the GAO audit liaisons at three agencies—the Department of Commerce (on June 21, 2019), the Department of Justice (on July 8, 2019), and the Department of the Treasury (on July 3, 2019). The agencies did not state whether they agreed or disagreed with our recommendations. We incorporated their technical comments as appropriate. We received emails from Agriculture’s Director of Strategic Planning, Egovernment and Audits on June 19, 2019, and from a Division Director in the Environmental Protection Agency’s Office of Information Security and Privacy on July 8, 2019, which stated that their agencies had no comments on the draft report. We are sending copies of this report to the appropriate congressional committees, the heads of the agencies in our review, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9342 or marinosn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix XXIV. The objectives of our review were to examine (1) the extent to which agencies established key elements of a cybersecurity risk management program; (2) what challenges, if any, agencies identified in developing and implementing cybersecurity risk management programs; and (3) what steps the Office of Management and Budget (OMB) and Department of Homeland Security (DHS) have taken to meet their risk management responsibilities under Executive Order (EO) 13800 and to address any challenges agencies face in implementing cybersecurity risk management practices. In conducting this engagement, we focused on 23 of the 24 agencies covered by the Chief Financial Officers Act of 1990. To address our first objective, we collected agency policies, procedures, and other documentation and compared them to selected key practices from OMB and National Institute of Standards and Technology (NIST) guidance for cybersecurity risk management. To identify the key practices, we reviewed OMB guidance pertaining to cybersecurity risk management, including OMB Circular A-130: Managing Information as a Strategic Resource, as well as Circular A-123: Management’s Responsibility for Enterprise Risk Management and Internal Control, which outlines agency responsibilities for enterprise risk management. We also reviewed NIST guidance, including the Framework for Improving Critical Infrastructure Cybersecurity; Special Publication 800-30: Guide for Conducting Risk Assessments; Special Publication 800-37: Guide for Applying the Risk Management Framework to Federal Information Systems, and Special Publication 800-39: Managing Information Security Risk: Organization, Mission, and Information System View. In selecting the key practices for our assessment, we focused on those practices identified by OMB and NIST as foundational for providing an organization-wide approach to cybersecurity risk management. We collected and analyzed documentation and other information from each agency related to cybersecurity risk management and compared it to the identified key practices. We supplemented our analyses with interviews with relevant agency officials to discuss the development of their policies. We discussed the results of our initial analysis of agency documentation with agency officials to validate our findings, collect additional evidence, and identify causes for any gaps. We then determined whether the evidence provided by the agency addressed each identified criteria element. Specifically, for each criteria element, we determined if the evidence fully addressed the element (“met”), addressed some, but not all, aspects of the element (“partially met”), or did not address any aspects of the element (“not met”). To address the second objective, we administered structured interview questions to the agencies to determine what challenges, if any, they face in developing and implementing policies and procedures for managing cybersecurity risk. We developed a list of potential challenges based on our assessment of agencies’ policies and procedures, a review of OMB’s risk report on agencies’ cybersecurity risk management capabilities, and reviews of prior GAO reports in areas related to cybersecurity risk management. We worked with GAO methodologists to develop a set of structured interview questions that were sent to the agencies and asked them to indicate if they faced each of these, as well as any additional, challenges, and to provide specific examples. We received responses from all 23 agencies in our review and analyzed them to identify those challenges that were indicated by a majority of the agencies. We excluded from our counts agencies that stated they did not have challenges in a particular area. We also identified common themes within the challenge areas. To address the third objective, we reviewed EO 13800 and implementation guidance issued by OMB, as well as relevant reports and other documents, including OMB’s Federal Cybersecurity Risk Determination Report and Action Plan, OMB memos, and supporting documentation for DHS initiatives. We also interviewed OMB and DHS officials with government-wide cybersecurity responsibilities to gain an understanding of initiatives under way to address their responsibilities under the order, and that could help address challenges identified by the agencies. We then compared these initiatives to the responses we received from agencies to determine if there were any gaps between the challenges and the ongoing initiatives. Specifically, for each challenge identified by a majority of the agencies in our review, we determined if any of the initiatives under way would address them based on a review of documentation associated with the initiatives as well as discussions with OMB and DHS officials. We conducted this performance audit from February 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Twenty-two of the 23 civilian Chief Financial Officers Act agencies in our review established and documented the role of the cybersecurity risk executive. Agencies varied in assigning this responsibility to the chief information officer (CIO), chief information security officer (CISO), or another official or entity. Table 7 provides details on our assessment. Of the 23 civilian Chief Financial Officers Act agencies, seven fully established a cybersecurity risk management strategy that included key elements recommended by National Institute of Standards and Technology (NIST) guidance. Specifically, these seven agencies developed strategies to guide how cybersecurity risk is to be framed, assessed, responded to, and monitored. In addition, five of the 23 agencies partially developed a cybersecurity risk management strategy, but their strategies did not address certain required elements. The remaining 11 agencies did not develop an agency-wide cybersecurity risk management strategy. Table 8 provides details on our assessment. The following elements, identified in NIST guidance, should be addressed in policies and procedures to facilitate risk-based decision making in securing information systems and data. Most of the 23 civilian Chief Financial Officers Act agencies addressed the majority of the key practices for incorporating risk-based decision- making in their policies and procedures. However, most of the agencies also had gaps in one or more of these areas. Specifically, six agencies addressed all the elements in their policies and procedures, and the remaining 17 were missing at least one. Table 10 provides details on our assessment of the agencies’ policies. Of the 23 civilian Chief Financial Officers Act agencies, 12 developed a process for an agency-wide cybersecurity risk assessment. Specifically, these agencies developed processes for aggregating system-level data and analyzing them to assess overall cybersecurity risk to agency operations and assets. The remaining 11 agencies did not establish such a process. Table 11 provides details on our assessment. Of the 23 civilian Chief Financial Officers Act agencies, 10 fully established a process or mechanism for coordination between their cybersecurity risk executive and their enterprise risk management (ERM) governance structure, five agencies partially established such a process, and the remaining eight agencies did not provide evidence of coordination. Table 12 provides details on our assessment. We are making a total of 57 recommendations to the 23 civilian Chief Financial Officers Act agencies in our review to fully address key practices in their cybersecurity risk management policies and procedures. The Secretary of Agriculture should take the following three actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 2) Update the department’s policies to require (1) the use of risk assessments to inform security control tailoring and (2) the use of risk assessments to inform plan of actions and milestones (POA&M) prioritization. (Recommendation 3) Establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 4) The Secretary of Commerce should take the following two actions: Update the department’s policies to require (1) an organization-wide cybersecurity risk assessment and (2) the use of risk assessments to inform POA&M prioritization. (Recommendation 5) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 6) The Secretary of Education should take the following action: Fully develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 7) The Secretary of Energy should take the following two actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 8) Update the department’s policies to require (1) an organization-wide cybersecurity risk assessment and (2) the identification of common controls. (Recommendation 9) The Secretary of Health and Human Services should take the following four actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 10) Update the department’s policies to require (1) an organization-wide cybersecurity risk assessment and (2) the use of risk assessments to inform security control tailoring. (Recommendation 11) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 12) Establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 13) The Secretary of Homeland Security should take the following two actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 14) Establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 15) The Secretary of Housing and Urban Developing should take the following two actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 16) Update the department’s policies to require the use of risk assessments to inform POA&M prioritization. (Recommendation 17) The Secretary of the Interior should take the following three actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 18) Update the department’s policies to require an organization-wide cybersecurity risk assessment. (Recommendation 19) Establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 20) The Attorney General should take the following two actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 21) Fully establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 22) The Secretary of Labor should take the following action: Update the department’s policies to require (1) the use of risk assessments to inform control tailoring and (2) the use of risk assessments to inform POA&M prioritization. (Recommendation 23) The Secretary of State should take the following two actions: Update the department’s policies to require (1) an organization-wide risk assessment, (2) an organization-wide strategy for monitoring control effectiveness, (3) system-level risk assessments, (4) the use of risk assessments to inform security control tailoring, and (5) the use of risk assessments to inform POA&M prioritization. (Recommendation 24) Establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 25) The Secretary of Transportation should take the following three actions: Fully develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 26) Update the department’s policies to require an organization-wide risk assessment. (Recommendation 27) Fully establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 28) The Secretary of the Treasury should take the following three actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 29) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 30) Establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 31) The Secretary of Veterans Affairs should take the following four actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 32) Update the department’s policies to require an organization-wide cybersecurity risk assessment. (Recommendation 33) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 34) Establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 35) The Administrator of USAID should take the following two actions: Update the agency’s policies to require (1) an organization-wide cybersecurity risk assessment and (2) the use of risk assessments to inform control tailoring. (Recommendation 36) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 37) The Administrator of EPA should take the following four actions: Fully develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 38) Update the agency’s policies to require an organization-wide cybersecurity risk assessment. (Recommendation 39) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 40) Fully establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 41) The Administrator of General Services should take the following four actions: Designate and document a risk executive function with responsibilities for organization-wide cybersecurity risk management. (Recommendation 42) Update the agency’s policies to require an organization-wide cybersecurity risk assessment. (Recommendation 43) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 44) Fully establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 45) The Administrator of NASA should take the following two actions: Update the agency’s policies to require (1) an organization-wide risk assessment and (2) the use of risk assessments to inform POA&M prioritization. (Recommendation 46) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 47) We are not making a recommendation to NASA to establish a cybersecurity risk management strategy because we previously made such a recommendation, which remains open. The Director of NSF should take the following action: Fully develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 48) The Chairman of NRC should take the following four actions: Develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 49) Update the agency’s policies to require (1) an organization-wide cybersecurity risk assessment and (2) the use of risk assessments to inform POA&M prioritization. (Recommendation 50) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 51) Establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 52) The Director of OPM should take the following two actions: Update the agency’s policies to require (1) an organization-wide cybersecurity risk assessment and (2) the use of risk assessments to inform control tailoring. (Recommendation 53) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 54) The Administrator of SBA should take the following three actions: Fully develop a cybersecurity risk management strategy that includes the key elements identified in this report. (Recommendation 55) Update the agency’s policies to require (1) an organization-wide cybersecurity risk assessment and (2) the use of risk assessments to inform POA&M prioritization. (Recommendation 56) Establish a process for conducting an organization-wide cybersecurity risk assessment. (Recommendation 57) The Commissioner of SSA should take the following action: Fully establish and document a process for coordination between cybersecurity risk management and enterprise risk management functions. (Recommendation 58) In addition to the individual named above, Marisol Cruz Cain (assistant director), Lee McCracken (analyst in charge), Kiana Beshir, Roger Bracy, Chris Businsky, Alan Daigle, John de Ferrari, Nancy Glover, Franklin Jackson, Vernetta Marquis, Carlton Maynard, Scott Pettis, Tomas Ramirez, Andrew Stavisky, and Shaunyce Wallace made significant contributions to this report.", "summary": "Federal agencies face a growing number of cyber threats to their systems and data. To protect against these threats, federal law and policies emphasize that agencies take a risk-based approach to cybersecurity by effectively identifying, prioritizing, and managing their cyber risks. In addition, OMB and DHS play important roles in overseeing and supporting agencies' cybersecurity risk management efforts. GAO was asked to review federal agencies' cybersecurity risk management programs. GAO examined (1) the extent to which agencies established key elements of a cybersecurity risk management program; (2) what challenges, if any, agencies identified in developing and implementing cybersecurity risk management programs; and (3) steps OMB and DHS have taken to meet their risk management responsibilities and address any challenges agencies face. To do this, GAO reviewed policies and procedures from 23 civilian Chief Financial Officers Act of 1990 agencies and compared them to key federal cybersecurity risk management practices, obtained agencies' views on challenges they faced, identified and analyzed actions taken by OMB and DHS to determine whether they address agency challenges, and interviewed responsible agency officials. Key practices for establishing an agency-wide cybersecurity risk management program include designating a cybersecurity risk executive, developing a risk management strategy and policies to facilitate risk-based decisions, assessing cyber risks to the agency, and establishing coordination with the agency's enterprise risk management (ERM) program. Although the 23 agencies GAO reviewed almost always designated a risk executive, they often did not fully incorporate other key practices in their programs: Twenty-two agencies established the role of cybersecurity risk executive, to provide agency-wide management and oversight of risk management. Sixteen agencies have not fully established a cybersecurity risk management strategy to delineate the boundaries for risk-based decisions. Seventeen agencies have not fully established agency- and system-level policies for assessing, responding to, and monitoring risk. Eleven agencies have not fully established a process for assessing agency-wide cybersecurity risks based on an aggregation of system-level risks. Thirteen agencies have not fully established a process for coordinating between their cybersecurity and ERM programs for managing all major risks. Until they address these practices, agencies will face an increased risk of cyber-based incidents that threaten national security and personal privacy. Agencies identified multiple challenges in establishing and implementing cybersecurity risk management programs (see table). GAO is making 57 recommendations to the 23 agencies and one to OMB, in coordination with DHS, to assist agencies in addressing challenges. Seventeen agencies agreed with the recommendations, one partially agreed, and four, including OMB, did not state whether they agreed or disagreed. GAO continues to believe all its recommendations are warranted.", "document_type": "gao"}
{"report": "When patients are seeking access to investigational drugs, their first option is to consider whether they can obtain them through participation in a clinical trial. Clinical trials are a step in the drug development process through which a drug manufacturer assesses the safety and effectiveness of its investigational drug through human testing. A clinical trial can take place in a variety of settings (e.g., research hospitals, universities, and community clinics) and geographic locations, and is led by a principal investigator that is typically a physician. Manufacturers establish clinical trial eligibility criteria to define the patient population to be studied, and only patients who meet those criteria can participate. These criteria can vary depending on the drug being studied and its intended use. Patient eligibility criteria consist of both inclusion and exclusion criteria. Inclusion criteria specify the characteristics of the patient that are required for participation, such as the stage or characteristics of a disease, and typically identify a patient population in which it is expected that the manufacturer can demonstrate the effect of an investigational drug. In comparison, exclusion criteria specify the characteristics that disqualify patients from clinical trial participation and can include factors that could mask the effect of an investigational drug, such as the presence of comorbidities or simultaneous use of other drugs. Certain patient populations, such as children and pregnant women, may also be excluded from clinical trial participation because of ethical reasons. Drug manufacturers, FDA, and IRBs each have responsibilities as part of the clinical trial process. In order to test an investigational drug on human volunteers in clinical trials, a manufacturer must first submit an investigational new drug application (IND) to FDA. FDA is responsible for reviewing the IND, which includes various components such as the clinical trial protocol that describes the patient eligibility criteria, the medications and dosages to be studied, and other details. In turn, an IRB is responsible for reviewing and approving the clinical trial protocol as well as reviewing the informed consent form for the study. In general, clinical trials that involve human volunteers can begin after FDA has reviewed and allowed the IND to proceed and the IRB has given its approval. An investigational drug typically goes through three phases of clinical trials before an application is submitted to FDA for marketing approval. At any point during the clinical trials, FDA could issue a clinical hold on the existing IND that would delay the proposed clinical trials or suspend the ongoing clinical trials. When a proposed or ongoing study is placed on a complete clinical hold, the investigational drug cannot be administered to any human volunteers. Traditionally, the three clinical trial phases are the following: Phase I: This clinical trial phase generally tests the safety of the drug on about 20 to 80 healthy volunteers. The goal of this phase is to determine the drug’s most frequent side effects and how it is metabolized and excreted. If the drug does not show unacceptable toxicity in the phase I clinical trials, it may move on to phase II. Phase II: This clinical trial phase assesses the drug’s safety and effectiveness on people who have a certain disease or condition, and typically the assessment is conducted on a few dozen to hundreds of volunteers. Generally, during this phase some volunteers receive the drug and others receive a control, such as a placebo. If there is evidence that the drug is effective in the phase II clinical trials, it may move on to phase III. Phase III: This clinical trial phase generally involves several hundreds to thousands of volunteers who have a certain disease or condition and gathers more information about the drug’s safety and effectiveness, again while being compared to a control. If phase III clinical trials are successfully completed, the drug may move on to FDA’s review and approval process. When seeking FDA’s approval to market a drug in the United States, the manufacturer submits an application to FDA that includes the data from the safety and efficacy clinical trials for FDA to review. Safety data include clinical trial results about a drug’s toxicity (e.g., the highest tolerable dose) and adverse events that may result from exposure to the drug. Efficacy data include information on whether the drug demonstrated a health benefit over a placebo. FDA reviews the information in the application to either approve or not approve the drug. If a patient seeking access to an investigational drug is not able to participate in the drug’s clinical trial (e.g., because of the study’s eligibility criteria or geographic location), another pathway to potentially obtain access to the drug outside of a clinical trial is through FDA’s expanded access program. Under the program, a licensed physician can submit a request for access to an investigational drug for treatment use on behalf of a patient and may do so during or after phase I, II, or III of clinical trials. To allow access to an investigational drug under the program, FDA must determine that a patient has a serious or immediately life-threatening disease or condition and has no other comparable medical options, among other criteria. FDA’s goals for the program are to facilitate the availability of investigational drugs when appropriate, ensure patient safety, and preserve the clinical trial development process. FDA is responsible for determining whether to allow individual requests to proceed to treatment once the manufacturer has agreed to provide access. If FDA allows the request to proceed, an IRB must approve the clinical treatment plan that is submitted as part of the individual request and review the informed consent form. The licensed physician treating a patient under expanded access would be required to report to FDA any unexpected serious adverse reactions that occur during treatment for which there is a reasonable possibility that the drug caused the reaction. In 2018 the federal RTT Act established another pathway through which patients may potentially obtain access to investigational drugs outside of clinical trials. To be eligible under the law, a patient must have been diagnosed with a life-threatening disease or condition, have exhausted approved treatment options, and be unable to participate in a clinical trial involving the investigational drug. Obtaining access to investigational drugs through the federal RTT Act primarily requires the involvement of the manufacturer and treating physician. Similar to FDA’s expanded access program, treatment can only proceed if the drug manufacturer allows the patient access to its drug. Under the federal RTT Act, the manufacturer is responsible for providing to FDA an annual summary of any use of its drugs under this pathway that includes information on any known serious adverse events. The treating physician is responsible for requesting access to the investigational drug for the patient and for obtaining written informed consent from or on behalf of the patient if the manufacturer agrees to provide access. Eligibility of an investigational drug for patient use through this pathway is based on certain criteria, including that the drug has completed phase I clinical trials, the manufacturer has not discontinued clinical development of the drug, and the drug has not been placed on a clinical hold. Unlike FDA’s expanded access program, the federal RTT Act does not require the FDA or an IRB to review individual requests for access. Figure 1 shows a summary of the three pathways through which patients may obtain access to investigational drugs. Some patients, such as those with compromised liver and kidney function, have traditionally been excluded from clinical trials. FDA has ongoing efforts to help drug manufacturers identify the circumstances under which they could broaden their eligibility criteria to include such patients without compromising study results. These efforts include issuing recent guidance with recommendations for including certain patients in clinical trials for cancer drugs. Officials from one of the 10 drug manufacturers we interviewed told us they had broadened their eligibility criteria and another one was taking steps to do so, but these officials and others noted challenges to broadening eligibility criteria. FDA public workshop on broadening eligibility criteria. In April 2018, FDA held a public workshop with stakeholders—including drug manufacturers, patient advocacy groups, and government agencies—to discuss ways drug manufacturers and other investigators could safely broaden eligibility criteria for clinical trials and to inform FDA guidance on this topic. In July 2018 FDA publicly released a report summarizing the workshop, in accordance with FDARA. According to the report, stakeholders at the meeting emphasized the importance of broadening clinical trial eligibility, when appropriate, to include more patients who will likely use the drug if it is approved. Stakeholders recommended that investigators ensure that the eligibility criteria for each of their clinical trials are scientifically justifiable, rather than, for example, “copying and pasting” a narrow set of criteria from a prior study without considering if the exclusions are valid for scientific reasons. According to the report, this practice can unnecessarily limit eligibility for certain patients. While stakeholders commented that assessing whether eligibility criteria are scientifically justifiable may require additional time and resources, they emphasized it could lead to the removal of unnecessarily restrictive eligibility criteria and thereby increase participation among patient populations that have been typically excluded from clinical trials, such as pediatric patients and patients with compromised liver and kidney function. FDA guidance on eligibility criteria. In March 2019, FDA issued four new draft guidance documents and finalized one guidance document with recommendations for drug manufacturers to broaden clinical trial eligibility criteria for drugs that treat cancer. The guidance recommends that manufacturers include certain patient populations that have typically been excluded from participation. The patient populations are adolescents; pediatrics (children and adolescents); patients with human immunodeficiency virus (HIV), hepatitis B virus (HBV), or hepatitis C virus (HCV) infections; patients with brain metastases (i.e., cancer that has spread to the brain); and patients with compromised kidney, heart, or liver function, or who have a history of (or concurrent) cancer. According to FDA, the guidance documents are intended to help drug manufacturers and other investigators broaden cancer trial eligibility criteria. This will help improve patient access to investigational drugs and ensure that the results from the clinical trials are generalizable to patients likely to use the drugs once they are approved. In addition, FDA officials have noted that including broader patient populations in clinical trials can lead to new information in a drug’s labeling, which will help communicate the safe and effective use of these drugs. Table 1 provides a summary of each of the five guidance documents. In June 2019, FDA issued draft guidance for manufacturers on broadening clinical trial eligibility criteria, in accordance with FDARA. The guidance applies to a wider range of clinical trials beyond cancer trials and includes recommendations to broaden eligibility criteria and considerations for the use of clinical trial designs and other methodologies to help facilitate patient participation. For example, FDA recommends that manufacturers examine each exclusion criterion to determine if it is needed to help assure the safety of trial participants or to achieve the study’s objectives. If not, the manufacturer should consider eliminating or modifying the criterion to expand the study population as well as tailoring the exclusion criteria as narrowly as possible to avoid unnecessary restrictions to the study population. Two manufacturers’ efforts to broaden eligibility criteria. Officials from one of the 10 drug manufacturers we interviewed told us they broadened their clinical trial eligibility criteria and another manufacturer we interviewed reported that it was taking steps to do so. These two manufacturers told us they were taking these steps in part because both believe it will facilitate the drug approval process. Officials from one manufacturer stated that they broadened their eligibility criteria by removing exclusions after determining they were not critical to clinical trial designs, including exclusions related to liver function, infections (e.g., HIV), and the use of other medications (e.g., steroids). The officials explained that, since 2015, they have systematically evaluated their eligibility criteria to ensure that they do not unnecessarily exclude patient populations from their clinical trials. Officials from the second manufacturer told us they have begun evaluating whether to remove certain exclusion criteria that they typically use in clinical trials, and added that their efforts are partially in response to FDA’s 2018 public workshop report, as described above. For example, the manufacturer is reviewing its exclusion of adolescents in prior clinical trials and officials told us they will likely include adolescents in an upcoming study if they determine that patient safety would not be compromised. Officials from both manufacturers stated that broader eligibility criteria will allow more patients to access investigational drugs through clinical trial participation. It can also, officials said, help them obtain FDA approval for a drug that extends to a wider range of patients, if the drug is found to be safe and effective. Further, officials from one of the two manufacturers noted that broader eligibility criteria, such as criteria that include patients with infections, could help streamline the process for conducting clinical trials—for example, by eliminating the need to conduct clinical testing to screen for the presence of infections. Although most drug manufacturers in our review did not report efforts to broaden their eligibility criteria, many noted efforts to address other barriers to clinical trial participation. For example, to address geographic barriers, officials from six of the 10 manufacturers told us they help cover costs for patients to travel to clinical trial sites, such as by reimbursing transportation and hotel costs for patients who travel long distances. In addition, officials from one manufacturer said they completed a pilot clinical trial on diabetes in 2019 that used decentralized trial locations in three states, such as retail health clinics and patients’ homes, to help patients overcome challenges with obtaining transportation to trial sites. Similarly, within the next 2 years, another manufacturer is planning to conduct a pilot clinical trial that is fully remote and expects the design to improve patient participation in rural communities. To address the lack of information about upcoming and ongoing clinical trials that is available to and tailored to patients, two manufacturers launched clinical trial registries in 2015 and 2016, respectively. Officials from one of the manufacturers stated they designed their registry to bridge the gap between the information that patients want about clinical trials (e.g., information targeted to medical conditions that uses basic terminology), and what is available in ClinicalTrials.gov, a federal database that includes information on privately and publicly funded clinical trial studies. Officials explained that ClinicalTrials.gov is, in general, more targeted to physicians. In addition, to address barriers associated with the mistrust of research stemming from historical events among African-Americans and other communities, one manufacturer has several ongoing efforts to increase the participation of racially and ethnically diverse populations in its clinical trials. For example, the manufacturer conducts workshops to train minority investigators who conduct clinical trials and requires certain clinical trial sites to be located in areas with minority patient populations of more than 25 percent. Challenges with broadening eligibility criteria. Officials from four of the 10 drug manufacturers we interviewed—including the two taking steps to broaden their clinical trial eligibility criteria—told us broadening eligibility criteria is challenging. They stated that broader criteria must be carefully balanced with the need to collect evidence from a well-defined population. Officials from one manufacturer explained that removing standard exclusion criteria, such as excluding patients who use other medications, could interfere with the success of their clinical trial if those medications make it difficult to identify the effects of the studied drug. In addition, officials from another manufacturer emphasized that determining whether to remove exclusion criteria takes time and resources because it involves additional study, which could slow down the clinical development of a drug. To facilitate access to investigational drugs outside of clinical trials, FDA has simplified its expanded access program’s IRB review requirements for individual patient requests. FDA made this change in October 2017, in accordance with a provision in FDARA. This provision addressed concerns that FDA’s requirement to convene a full IRB to review an expanded access request could result in delays of approvals because full IRBs may not meet regularly. Under the revised process, FDA now allows for a waiver of the requirement for full IRB review when concurrence is obtained by the IRB chair or another designated member. According to FDA officials, the updated process will help reduce the potential burden for physicians, who are responsible for obtaining IRB approval, while still protecting patients. In addition, to further simplify its expanded access process for individual patient requests, in June 2019 FDA launched a pilot program called Project Facilitate for oncologists and other health care professionals that treat patients with cancer. According to FDA officials, the pilot program is focused on oncology because the agency receives a large number of individual expanded access requests from oncologists. Under the pilot program, FDA established a new call center that provides a single point of contact where FDA staff are available to answer questions, assist in filling out appropriate paperwork, and facilitate the overall process for requesting and obtaining access to investigational drugs. For example, FDA officials told us that FDA staff may assist oncologists in locating an IRB, if needed. As part of the pilot program, FDA will follow up on individual requests and gather data, such as how many patients received investigational drugs, and if not, why the requests were denied by manufacturers. According to FDA, the agency can use these data to determine how the process is benefiting patients. Twenty of the stakeholders we interviewed were familiar with FDA’s simplified IRB review requirements, and of those, 18 told us these updates were helpful for physicians and patients. For example, officials from one drug manufacturer commented that the new IRB review requirements reduce the amount of time it takes for patients to obtain access to investigational drugs, which is especially important for patients who are very sick. In addition, we spoke to 12 stakeholders about FDA’s plans for its pilot program, and of those, nine generally had positive views of the agency’s planned activities. Officials from one manufacturer explained that the pilot program could help reduce the burden on oncologists seeking access to investigational drugs for their patients through the expanded access program. On the other hand, the officials from this same manufacturer raised concerns about the potential for FDA to intentionally or unintentionally pressure companies to make their investigational drugs available to patients, should FDA have increased involvement with drug manufacturers as part of the pilot program. FDA has also taken recent actions to facilitate access to investigational drugs outside of clinical trials by increasing its communication about the expanded access program and the federal RTT Act. FDA’s increased communication about the expanded access program. In November 2018, FDA updated the web pages for its expanded access program in response to findings from an external assessment that the web pages were difficult to navigate and contained unclear information. FDA created separate web pages for patients, physicians, and drug manufacturers, and tailored information about the expanded access process to each of these stakeholders. In addition, FDA added a new web page with information that is commonly requested by physicians and patients, such as the instructions for completing the form for submitting individual requests and definitions of keywords associated with the expanded access process (e.g., IRB, informed consent). In addition, in October 2017, in response to a recommendation in our July 2017 report, FDA clarified its guidance for drug manufacturers on how the agency reviews adverse events that occur under FDA’s expanded access program. In the 2017 report, we found that some drug manufacturers were concerned that use of adverse event data may influence FDA in making final approval decisions, and that this possibility could contribute to a manufacturer deciding not to grant patients access to their drugs through the expanded access program. In response, we recommended that FDA clearly communicate how the agency will use adverse event data from expanded access use when reviewing drugs and biologics for approval. FDA’s updated guidance states that FDA is not aware of instances in which adverse event information prevented the agency from approving a drug, and that it is very rare for FDA to place a clinical hold on an investigational drug due to adverse events observed during expanded access treatment. The guidance also explains that several factors make it difficult for FDA to link an adverse event to the expanded use of a drug being considered for approval. For example, the guidance acknowledges that the use of investigational drugs though the expanded access program generally occurs outside of a controlled clinical trial setting and patients receiving such drugs may be sicker than patients participating in a clinical trial, making it more difficult to determine whether the use of the investigational drug has led to the adverse event. In responding to questions about increased FDA communication about the expanded access program, 19 of the stakeholders we interviewed were familiar with FDA’s updated expanded access web pages, and of those, 16 told us they were an improvement. Officials from one physician organization stated that the updated web pages were easier to navigate than the previous web pages and presented information about the process more clearly. Among the 10 manufacturers we interviewed, we found varying views of FDA’s updated guidance on the use of adverse event data. Officials from seven of the 10 manufacturers viewed the updated guidance as an improvement. Officials from one of the seven explained that it contributed to their company’s decision to allow access to investigational drugs, when appropriate. Officials from two of the 10 manufacturers did not view the guidance as an improvement. Officials from both manufacturers stated that they still had significant concerns about the potential use of adverse event data by FDA to adversely affect the development of their investigational drugs, such as being used to issue a clinical hold. An official from one of the two manufacturers commented that these concerns remained despite FDA’s statement in the guidance that it is difficult for FDA to link expanded access use to a particular adverse event. In addition, officials from two other manufacturers who viewed the guidance as an improvement similarly expressed remaining concerns that adverse events could negatively affect the development of their investigational drugs. One manufacturer was unfamiliar with the updated guidance. Further, officials from four of the 10 drug manufacturers we interviewed, including two who viewed the updated guidance as an improvement, said they believed that manufacturers’ concerns about this issue may never be fully resolved even with additional FDA guidance. In other comments related to FDA’s communication on its use of adverse events data from the expanded access program, some drug manufacturers we interviewed noted the merits of using efficacy and safety data from the expanded access program to inform FDA’s drug approval decisions. Officials from two of the 10 manufacturers told us they believe that FDA’s potential use of adverse event data from expanded access use, but not efficacy data, would be unfair. Officials from one of these two manufacturers cited FDA’s updated guidance on adverse events as contributing to their view, referring to FDA’s statement that it is unlikely that FDA’s program would yield data that is useful to FDA in considering an investigational drug’s effectiveness. However, FDA officials told us that efficacy and safety data from the expanded access program have been used to support drug approvals in several instances. For example, in January 2018 FDA approved the drug Lutathera to treat rare tumors in the pancreas and gastrointestinal tract using efficacy and safety data the manufacturer submitted to FDA from a subset of the roughly 1,200 patients who received the drug through the expanded access program. Officials from four of the 10 manufacturers expressed interest in discussing further with FDA how the agency would evaluate efficacy and safety data from the expanded access program and use these data to help support a drug’s approval and other regulatory decisions. FDA’s communication about the federal RTT Act. In November 2018, FDA launched a new federal RTT web page that outlines both the eligibility requirements for patients interested in seeking access to investigational drugs and the criteria that must be met for an investigational drug to be eligible for use through this pathway. For example, the web page states that patients must be diagnosed with a life- threatening disease or condition to be eligible to access investigational drugs under the federal RTT pathway. Further, the agency plans to issue proposed regulations in September 2019 to implement the federal RTT Act requirement for manufacturers to submit an annual summary to FDA on any use of their investigational drugs under this pathway. The regulations will include a due date for manufacturers to submit the annual summaries as well as information on what they are to contain, according to FDA. Fourteen of the stakeholders we interviewed were familiar with FDA’s new web page on the federal RTT Act, and among those, eight stated that it communicated useful and balanced information for physicians and patients. Officials from the remaining six stakeholders told us they did not find it helpful for physicians or patients. For example, officials from two stakeholders (including one drug manufacturer) commented at the time of our review that the web page could be misleading to some patients if they interpret the federal RTT Act to mean that manufacturers must provide access to their investigational drugs. Both added that FDA should more clearly communicate on the web page that there is no such requirement. In addition, officials from another stakeholder stated at the time of our review that FDA should explain on the web page the agency’s role in implementing the federal RTT Act. In May 2019 FDA clarified on its web page that the federal RTT Act does not require manufacturers to provide patients access to their investigational drugs and that FDA’s role includes posting a consolidated annual summary report on the use of investigational drugs through the federal RTT pathway. Most of the 29 drug manufacturers in our review used their websites to communicate to patients and physicians whether they would consider individual requests for access to their investigational drugs outside of clinical trials. Among those that would consider requests, most also communicated the conditions under which they would review requests and grant access. Manufacturers’ consideration of requests for access. Our review of drug manufacturers’ websites between January 31, 2019, and March 12, 2019, found that 23 of the 29 manufacturers in our review used their websites to communicate whether they considered individual requests for access to investigational drugs outside of clinical trials. In communicating this information, 19 of the 23 manufacturers stated they were willing to consider requests, while the other four stated they were not considering requests. The remaining six of the 29 manufacturers did not communicate information about whether they would consider requests for access to investigational drugs outside of clinical trials at the time of our review, but officials from all six told us they were in the process of developing content on this topic that they intended to post on their websites. Information communicated by manufacturers that consider requests. Among the 19 manufacturers willing to consider requests for access to investigational drugs outside of clinical trials, all communicated on their websites that they required physicians to submit requests on behalf of their patients and provided information on how physicians should submit these requests. In addition, 18 manufacturers communicated an estimated time frame within which they would respond to requests. The manufacturers provided additional information, including the following: Eighteen communicated information about the type of patient for whom they would consider granting access. Eighteen stated that patients must have a serious or life- threatening disease or condition; have no comparable or satisfactory alternative therapies available; and be unable to participate in a clinical trial to be eligible to obtain access. In addition, 17 stated that the treating physician must determine for the patient seeking access that the risk of taking the investigational drug is not greater than the anticipated benefit. Fifteen communicated other factors they would take into account during their review of requests. These factors included the following: Ten stated that the supply of their investigational drugs was a consideration. That is, a manufacturer must have a sufficient supply of the investigational drug to support the drug’s clinical development before granting access to patients outside of clinical trials. Five referred to specific drugs to which they would consider granting access when describing the conditions under which they would consider reviewing requests. For example, one manufacturer stated that it would consider requests to access three of its investigational drugs (intended to treat bladder cancer, influenza, and HIV). One manufacturer communicated that after its initial review of individual requests, it uses an external advisory committee to further evaluate certain requests and ensure they are evaluated in an ethical and fair manner. The committee, which includes bioethical experts, physicians and patient representatives, makes recommendations to the manufacturer about providing access to individual patients. Many of the 19 manufacturers that communicated they were willing to consider individual requests for access stated that after they have approved a request they also required external entities to review the request. These included the following: Thirteen stated they require the relevant regulatory authority to review requests. Of these, six specified that they require FDA to review requests for access in the United States. One of these six explained that it required a review by FDA to ensure all available safety data for the investigational drug were considered, and added that FDA is uniquely aware of such safety data. Five stated they require the review of a research ethics committee or an IRB. Information communicated by manufacturers that do not consider requests. Among the four manufacturers that communicated on their websites they were not considering requests for access to investigational drugs outside of clinical trials at the time of our review, two provided reasons for their decision. Both cited safety concerns; for example, one explained that it wanted to ensure its investigational drugs were administered to patients only through clinical trials where safety could be closely monitored. One also cited limited resources, stating that it chose to focus its resources solely on conducting clinical trials. Both of the manufacturers that provided reasons for not considering requests for access communicated that they will periodically re-evaluate their policies. We provided a draft of this report to HHS for comment and HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. John E. Dicken at (202) 512-7114 or dickenj@gao.gov In addition to the contact named above, Gerardine Brennan, Assistant Director; Pamela Dooley, Analyst-in-Charge; Craig Gertsch; Gay Hee Lee; and Moira Lenox made key contributions to this report. Also contributing were George Bogart, Laurie Pachter, and Ethiene Salgado- Rodriguez.", "summary": "When investigational drugs show promise for treating serious or life-threatening diseases, patients are often interested in obtaining access to them. Congress included a provision in the FDA Reauthorization Act of 2017 for GAO to review actions taken to facilitate access to these drugs. This report describes (1) actions FDA and drug manufacturers have taken to broaden eligibility criteria for clinical trials, (2) actions FDA has taken to facilitate access to investigational drugs outside of clinical trials, and (3) information drug manufacturers have communicated to patients and physicians about access to investigational drugs outside of clinical trials. GAO reviewed laws, regulations, FDA documents, and manufacturer policies and interviewed FDA officials and a non-generalizable selection of 10 manufacturers and 14 other stakeholders (including patient advocacy and physician organizations). The manufacturers were developing drugs to treat serious or life-threatening diseases, and were selected for variation in company size. GAO also reviewed information that a non-generalizable selection of 29 manufacturers communicated through their websites about access to investigational drugs outside of clinical trials. GAO selected manufacturers for variation in the type of serious diseases their investigational drugs were intended to treat, company size, and other factors. HHS provided technical comments on a draft of this report, which GAO incorporated as appropriate. Individuals may access investigational drugs—those not yet approved for marketing in the United States by the Food and Drug Administration (FDA)—by participating in clinical trials conducted by drug manufacturers to test drug effectiveness and safety. FDA has ongoing efforts to help manufacturers identify the circumstances under which they could broaden clinical trial eligibility criteria to include patients who are commonly excluded, such as pediatric patients and patients with impaired liver and kidney function, without compromising study results. FDA issued guidance in March 2019 with recommendations on ways manufacturers could broaden eligibility criteria for cancer clinical trials, when clinically appropriate. In June 2019, FDA issued related guidance that applies to a wider range of clinical trials beyond cancer trials. One of the 10 manufacturers GAO interviewed reported broadening its eligibility criteria to include more patients, such as those with HIV. Another manufacturer has begun reviewing its eligibility criteria and expects to include adolescents, as appropriate, in future studies—a population that has generally been excluded from trials. However, these and two other manufacturers cited challenges in these efforts. One stated that expanding participation to patients who use other medications, for example, could adversely affect a study's ability to identify the effects of the studied drug. Outside of clinical trials, patients with certain medical conditions, who are unable to enroll in a clinical trial, and have no other comparable medical options, may request to obtain access to investigational drugs. This can occur under FDA's expanded access program, or through a 2018 federal law known as “Right to Try.” Under either pathway, a patient can only access the investigational drug if its manufacturer agrees to the request. FDA has taken steps to facilitate access to investigational drugs outside of clinical trials, and most manufacturers in GAO's review communicated information to patients and physicians through their websites about how to access their investigational drugs outside of clinical trials. For example: Since 2017, FDA took steps to simplify its expanded access program to make it easier to participate. In addition, to address concerns raised by manufacturers, FDA clarified guidance on how it would review data resulting from the program. Seven of the 10 manufacturers GAO interviewed viewed the guidance as an improvement. GAO's review of information communicated by 29 manufacturers on their websites found that 23 had policies about accessing investigational drugs outside of clinical trials. At the time of GAO's review, 19 of the 23 stated they would consider individual requests for access, while the other four stated they would not. More than half of the manufacturers stated that if they approve a request, they require additional steps, such as FDA review of the request.", "document_type": "gao"}
{"report": "In fiscal year 2018, OJJDP administered 16 programs, which collectively included 53 grant solicitations and 295 associated awards. As shown in table 1, the 16 programs included the Title II Formula Grant Program, two discretionary grant programs for youth mentoring and missing and exploited children that awarded the most funding, and 13 other discretionary grant programs to support efforts such as helping opioid- affected youth, victims of child abuse, and girls in the juvenile justice system. For a complete list of OJJDP programs and grants in fiscal year 2018, see appendix I. On December 21, 2018, the Juvenile Justice Reform Act of 2018 (“Reform Act”) enacted various requirements to strengthen accountability and oversight in OJJDP grant programs, including grants to states under the Title II Formula Grant Program. The Reform Act expressed the sense of Congress that OJJDP must restore meaningful enforcement, and states must exercise vigilant oversight, to ensure compliance with core requirements of the Title II Formula Grant Program. Among the measures the Reform Act put into place to achieve this goal was a requirement that states maintain an “effective” system of monitoring compliance—a requirement that became effective for fiscal year 2020 awards. In contrast, the requirement prior to fiscal year 2020 was for states to maintain an “adequate” system of monitoring compliance, which was the requirement that applied to the grant performance period reviewed for this report (October 2015 through September 2018). This report is one of many called for by the Reform Act to improve oversight of OJJDP grant programs, including the Title II Formula Grant Program. Other oversight requirements in the Reform Act are addressed to OJJDP; OJP’s Office of Audit, Assessment, and Management; and the DOJ OIG. Appendix III provides information on selected Reform Act requirements that relate to accountability and oversight in the Title II Formula Grant Program, and the status of efforts to implement them. All OJP grantees and awards are subject to “grant monitoring.” Grant monitoring consists of (1) programmatic and (2) financial monitoring, and according to OJP officials, helps ensure the programmatic and financial integrity and accountability of grantees. OJP policy requires programmatic desk reviews on all open awards each fiscal year and “in- depth” monitoring—consisting of enhanced programmatic desk reviews or site visits—on at least 10 percent of the total number and dollar amount of open and active awards annually. In addition, OJP’s Office of the Chief Financial Officer plans to financially monitor at least 10 percent of the award population annually. States awarded Title II Formula Grants are subject to an additional form of monitoring—“compliance auditing”—which is conducted by OJJDP annually to fulfill statutory requirements unique to the program. Specifically, “compliance auditing” refers to OJJDP’s process for (1) auditing the compliance monitoring systems used by states, and (2) evaluating states’ compliance with four core requirements specified in law. See appendix IV for more information on compliance auditing in the Title II Formula Grant Program associated with the performance period we reviewed. Both the grant monitoring and compliance auditing broadly consist of three parts: (1) desk reviews that occur annually, (2) risk assessments that assist officials in determining what additional monitoring or auditing activities to perform or how to prioritize them, and (3) additional monitoring or auditing activities, such as enhanced desk reviews or site visits (see figure 1). Fraud and “fraud risk” are distinct concepts. Fraud—obtaining something of value through willful misrepresentation—is challenging to detect because of its deceptive nature. Fraud risk (which is a function of likelihood and impact) exists when individuals have an opportunity to engage in fraudulent activity, have an incentive or are under pressure to commit fraud, or are able to rationalize committing fraud. Fraud risk management is a process for ensuring program integrity by continuously and strategically mitigating the likelihood and impact of fraud. When fraud risks can be identified and mitigated, fraud may be less likely to occur. Although the occurrence of fraud indicates there is a fraud risk, a fraud risk can exist even if actual fraud has not yet been identified or occurred. According to federal standards and guidance, executive-branch agency managers—including those at DOJ, OJP, and OJJDP—are responsible for managing fraud risks and implementing practices for addressing those risks. Federal internal control standards call for agency management officials to assess the internal and external risks their entities face as they seek to achieve their objectives. The standards state that as part of this overall assessment, management should consider the potential for fraud when identifying, analyzing, and responding to risks. We issued our Fraud Risk Framework in July 2015. The Fraud Risk Framework provides a comprehensive set of leading practices, arranged in four components, which serve as a guide for agency managers developing efforts to combat fraud in a strategic, risk-based manner. The Fraud Risk Framework is also aligned with Principle 8 (“Assess Fraud Risk”) of Standards for Internal Control in the Federal Government. The Fraud Reduction and Data Analytics Act of 2015 requires agencies to establish financial and administrative controls that are aligned with the Fraud Risk Framework’s leading practices. In addition, guidance under Office of Management and Budget Circular A- 123 affirms managers should adhere to the leading practices identified in the Fraud Risk Framework. OJJDP has established goal statements and performance measures for individual programs. Goal statements broadly convey a program’s overall intent, and performance measures assess program outputs or outcomes. Figure 2 below shows current goal statements and performance measures for the Youth Mentoring Program, for example. OJJDP requires grantees to report on performance measures which are listed in grant solicitations. OJJDP has designated 15 of its measures as “core” measures that are generally applicable across most OJJDP programs. OJJDP aggregates data from some of these core measures across all applicable programs collectively to assess progress toward office-level targets. Targets represent outputs or outcomes expressed as a numeric goal. For example, OJJDP has office-level targets for the percent of youth who offend and reoffend, as well as the percent of grantees that have implemented an evidence-based program. OJJDP obtains, reports on, and uses OJJDP program performance data, as shown in figure 3 below. We reviewed a selection of performance data that grantees provided to OJJDP for three programs: Title II Formula Grant Program, Youth Mentoring Program, and Gang Prevention Program. As a result of our review—which covered data from October 2015 through December 2018—we determined that these data were not sufficiently reliable for the purpose of providing examples of performance results in this report. For example, we found that grantees reported inconsistent information, such as different numbers of youth served for different performance measures within the same data collection period. Further, some performance measures double-count youth when presenting data by year. OJJDP collects data from discretionary grantees during two 6-month reporting periods and some measures are not designed to be aggregated across time periods. Nevertheless, we found that OJJDP was aware of most of the issues we identified and is taking steps to improve the reliability of grantee- submitted data. For instance, beginning October 2017 (for formula grant programs) and July 2018 (for discretionary grant programs), OJJDP implemented a process to identify inconsistent or otherwise questionable data and reach out to grantees for verification. Appendix II includes information about the issues we found with the data we reviewed and steps OJJDP is taking to improve the reliability of these data. While OJJDP has established goal statements and performance measures, it has not set numeric targets by which it can assess progress for each individual program. For example, one stated goal of the Title II Formula Grant Program is to prevent youth already in the juvenile justice system from reoffending. OJJDP’s annual Title II Formula Grant Program performance reports state the percentage of youth who reoffended, the performance measure for this goal; however, the reports do not provide a target against which to evaluate whether the result reflects progress toward the stated goal. Further, while OJJDP has set several office-wide targets for all programs collectively, these targets may not be appropriate for assessing the progress of individual programs because programs vary in size. For example, according to grantee-submitted performance data, the Title II Formula Grant Program served approximately 100,000 youth in fiscal year 2018, while the Gang Prevention Program served about 1,000 youth in calendar year 2018. Thus, office-level targets, while useful, may be more representative of the performance of OJJDP’s largest programs and obscure the results of individual programs. Since June 2019, OJJDP has been reviewing individual program goal statements and performance measures as part of an OJP-wide review, but this review does not include setting program-level targets. OJJDP officials said that they have not set program-level targets for two reasons: (1) there was uncertainty about whether OJJDP had the authority to do so (versus OJP’s Office of the Chief Financial Officer), and (2) setting such targets has not been a priority for OJJDP in the past, in part, due to a lack of resources. In October 2019, OJP clarified that the OJJDP Administrator has authority for OJJDP performance measurement, including setting program-level and office-level targets. OJJDP officials also stated that program oversight has recently become a higher priority and they plan to bring on new staff; and they agreed with the need to set program-level targets. Tracking performance measures against established numeric targets is a leading practice in performance management. Numeric targets establish standards against which federal programs can measure progress towards goals because comparisons can be easily made between projected performance and actual results. Thus, updating program goal statements and performance measures would be more effective with related numeric targets. One goal of the OJP-wide review is to increase accountability for achieving results. Setting program-level targets could help OJJDP meet this goal by establishing a clear means by which progress toward goals can be measured. According to OJP officials, some of the tools it uses in grant monitoring and compliance auditing consider fraud risk affecting OJJDP grant programs. As previously discussed, OJP’s grant monitoring and compliance auditing broadly consist of three parts: (1) desk reviews that occur annually, (2) risk assessments that assist officials in determining what additional monitoring or auditing activities to perform or how to prioritize them, and (3) additional monitoring or auditing activities, such as enhanced desk reviews or site visits. To carry out these efforts, OJP relies on various tools to assess the overall risk of grantees and awards. Figure 4 provides additional information on the tools that provide insight on fraud risks, according to OJP officials. According to OJP officials, pre-award and ongoing risk assessment processes that apply to all OJJDP grantees are the primary way in which the office identifies fraud risks. The bullets below describe the nature of the primary tools used during these risk assessment processes for all OJJDP grantees and how they provide insights into potential fraud risks. Financial Capability Questionnaire: This questionnaire includes 28 questions designed to provide insight on the financial systems and internal controls a grantee has in place prior to receiving an award. OJP developed the current version of this questionnaire in part as a response to a 2013 DOJ OIG audit, according to OJP officials. All applicants for OJJDP awards are required to fill out this questionnaire, and new grantees’ pre-award risk scores and corresponding risk levels rely, in part, on the applicants’ responses to it. These include detailed questions related to the capabilities of the applicant’s financial management system, such as whether it has the capability to record expenditures by budget cost categories. According to OJP officials, if an applicant’s accounting system cannot do so, the opportunity for fraud increases because the commingling of funds between budget categories would make it difficult to determine whether federal funds were spent in accordance with the approved budget. The questionnaire also includes items related to procurement, travel policy, and subrecipient management and monitoring. These questions may similarly indicate increased fraud risk depending on how the applicant responds. For instance, according to OJP officials, questions related to procurement are designed to determine whether the applicant employs a fair, transparent, and competitive procurement process. If an applicant’s procurement standards do not meet these criteria, the likelihood of fraudulent activity may increase. Grant Assessment Tool: The Grant Assessment Tool helps assess open/active OJJDP awards and grantees against 38 risk criteria. OJP officials identified 14 of these criteria as being indicators of potential fraud risks, such as the results of recent audits, whether the award has subawards or subcontracts, and whether grantees have completed progress reports on time. Officials explained that progress report delinquencies, for example, may be an indicator that a recipient does not have adequate internal controls to handle federal awards, which may provide a greater opportunity for fraud. The Grant Assessment Tool generates a risk score and corresponding monitoring priority for each open/active grantee and award quarterly. According to OJP officials, Grant Assessment Tool criteria have evolved over time in response to common audit or monitoring findings, as well as ongoing coordination with the DOJ OIG, as discussed later. For the Title II Formula Grant Program, grantees are also assigned risk assessment scores and audited using the compliance auditing tools described in the bottom half of figure 4. OJP officials stated that certain responses to any of the questions in the tools used during compliance auditing may indicate inadequate program management or weak internal controls, which may increase the risk of fraudulent activity. OJP officials use a variety of other tools during grant monitoring to monitor each OJJDP grantee or award (see tools in the top half of figure 4). The tools used differ depending on the type and level of monitoring being performed. According to OJP officials, all of the monitoring and auditing tools include detailed questions that provide insight about the strength of a grantee’s internal controls. In cases where a grantee is unable to provide adequate documentation in response to these questions, OJP officials stated that they may have weak internal controls which may increase the risk of fraudulent activity. OJP regularly coordinates with the DOJ OIG on issues related to fraud risk affecting OJJDP grant programs through meetings, trainings, and reviews of OIG audits. Specifically, according to officials, staff from OJP’s Office of Audit, Assessment, and Management meet with the OIG two to three times per year to discuss fraud allegations and ongoing fraud investigations related to OJJDP grant programs. These discussions assist OJP officials in determining their monitoring or auditing plans, but also provide insights about the types of issues that are being referred to the OIG for investigation, which can then inform needed changes to monitoring and auditing tools, according to OJP officials. The OIG also provides training to OJP staff every other year on how to identify and report potential fraud. Additionally, Office of Audit, Assessment, and Management officials regularly review OIG audit findings pertaining to individual grantees and compare them to grant monitoring findings for the same grantees. According to OJP officials, the purpose of these reviews is to evaluate where OJP may be able to improve its monitoring processes. Officials stated that one example of an outcome from such a review is the previously discussed Financial Capability Questionnaire. According to JMD officials, the department is implementing fraud risk management requirements through an iterative process that will be completed over multiple years, which leverages the department’s overall ERM processes. Leading practices for planning and conducting fraud risk assessments acknowledge that assessing fraud risks is an iterative process. According to JMD officials, as a result of this iterative approach, the fraud risk assessments JMD conducted in fiscal years 2017 and 2018 did not fully align with selected leading practices in the Fraud Risk Framework, which include documenting a fraud risk profile. Although the Department continued to implement additional leading practices from the Fraud Risk Framework by developing a fraud risk profile as part of the fiscal year 2019 fraud risk assessment, the fraud risk profile did not include any consideration of DOJ’s fraud risk tolerance—another leading practice identified by the Fraud Risk Framework. In fiscal year 2017, JMD conducted an assessment of fraud risks that consisted of four facilitated discussion groups with relevant officials. One discussion group was tailored to the specific concerns of grant programs—which included grant programs managed by OJJDP—as recommended by leading practices of the Fraud Risk Framework. During the grant-focused facilitated discussion group, participants identified five fraud scenarios relevant to DOJ grant programs, such as misdirection of funds, which occurs when a recipient deliberately misdirects funds in a manner inconsistent with the purpose outlined in the award agreement. After participants reached consensus on the fraud scenarios, they then ranked the inherent risk of each scenario, in alignment with leading practices of the Fraud Risk Framework. DOJ officials also told us that participants ranked residual risk using a voting tool, which required the participants to understand the various management controls in place to address a particular fraud risk. However, the documented outcomes of these discussions did not identify specific fraud risk controls or the extent to which those controls mitigate specific fraud risks, and as a result, it is unclear how these residual risk values were determined. Further, according to officials responsible for managing the fiscal year 2017 effort, the facilitated discussion group did not determine a specific and measurable fraud risk tolerance for DOJ grant programs generally nor discuss specific fraud risk management activities or controls for any specific grant programs. As discussed in greater detail later in this report, leading practices for fraud risk management state that managers should determine a fraud risk tolerance and Standards for Internal Control in the Federal Government states that managers should define risk tolerances in specific and measurable terms so they are clearly stated and can be measured. The fiscal year 2018 effort to assess fraud risk at DOJ consisted of a brief survey about fraud-related issues that was distributed to all DOJ components and was more limited in nature than the fiscal year 2017 effort. Specifically, the fiscal year 2018 questionnaire did not ask the components to identify inherent or residual risks, as was done in the prior year. Further, for the grants category, JMD included four of the five fraud risk areas that were addressed in fiscal year 2017. However, the fiscal year 2018 questionnaire did specifically ask components to identify key fraud risk management activities designed to prevent, detect, or respond to fraud, which had not been part of the fiscal year 2017 effort. Including information about control activities indicates additional maturation of JMD’s fraud risk management activities, but the questionnaire did not ask components to consider the extent to which these control activities mitigate the likelihood and impact of risk—as recommended by leading practices of the Fraud Risk Framework. The questions asked in fiscal year 2018 are shown in table 2 below. According to JMD officials, the components’ responses to this questionnaire were summarized for internal purposes, but no additional analysis or work, such as defining a fraud risk tolerance and documenting a fraud risk profile, was completed for any of the categories listed in table 2. Further, JMD officials stated that the department does not believe a full-scale fraud risk assessment is warranted annually, and the fiscal year 2018 effort was designed to build on the prior year’s assessment. According to officials, this is part of the department’s iterative approach to implementing fraud risk management requirements, which, consistent with leading practices, may not necessarily incorporate all relevant leading practices in each iteration. In December 2019, JMD officials provided the final summary of the fiscal year 2019 fraud risk assessment, which included a fraud risk profile as recommended by leading practices. According to JMD officials, to conduct the fiscal year 2019 fraud risk assessment, JMD officials first created a fraud risk profile template using information from the 2017 and 2018 assessments. Figure 5 shows an excerpt of JMD’s draft fraud risk profile template, as of September 2019. After senior leadership reviewed the pre-populated template, JMD held a facilitated discussion with representatives from each component to evaluate the risk information presented in the template for each topic area. JMD identified several risks for each topic area, including the same five risks for the grants area that were identified in fiscal year 2017. Based on information provided, the fiscal year 2019 fraud risk assessment and resulting fraud risk profile incorporate many leading practices of the Fraud Risk Framework. These include consideration of inherent fraud risk, current fraud risk controls and their suitability (the extent to which control activities mitigate the likelihood and impact of risk), and residual fraud risk. However, the 2019 fraud risk profile did not determine a measurable fraud risk tolerance, or prioritize residual risk against that tolerance for any of the assessed categories, including grants. Managers’ defined risk tolerance may depend on the circumstances of individual programs and other objectives beyond mitigation of fraud risks. Leading practices for fraud risk management state that managers should define a fraud risk tolerance, examine the suitability of existing fraud controls, and then prioritize residual fraud risks. In doing so, managers should consider the extent to which existing control activities mitigate inherent risks and whether the remaining risks exceed managers’ tolerance. Based on this analysis and the defined risk tolerance, managers then rank residual risks in order of priority, and determine their responses, if any, to mitigate those risks that exceed their risk tolerance. JMD officials stated that they did not yet define the department’s fraud risk tolerance for any of the assessed categories because they view it as the next step in the maturation of DOJ’s fraud risk assessment processes. However, JMD did not provide details or documentation of its plans to develop a specific and measurable fraud risk tolerance for the next iteration of their fraud risk assessments. Although following an iterative approach to fraud risk management is consistent with leading practices, until DOJ defines a measurable fraud risk tolerance for the assessed categories, the department may not effectively allocate limited resources to address fraud risks—including those associated with OJJDP grant programs. Specifically, by determining a measurable fraud risk tolerance for the grants category and assessing identified residual fraud risks against that tolerance to prioritize these risks, the department will help ensure that OJJDP’s grant programs are not vulnerable to greater risks than DOJ is willing to tolerate. Doing so will also provide assurance that OJJDP does not unintentionally over-allocate limited funding to address fraud risks the department is willing to tolerate. According to JMD officials, they are in the process of awarding a contract that will result in an implementation plan for addressing fraud risk management requirements in the future. Specifically, in July 2019, JMD released a Request for Quotes for a Blanket Purchase Agreement in support of DOJ’s implementation of OMB Circular A-123. One of the deliverables JMD expects to order under this agreement is an implementation plan for addressing fraud risk management requirements, which will include developing a plan for conducting regular fraud risk assessments consistent with leading practices for fraud risk management. According to officials, they expect to award the agreement by the end of calendar year 2019, after which the contractor will perform task orders issued by JMD that will include details related to the methodology, timeframes, and staffing associated with each deliverable. Because neither the award nor the task orders were in place at the time of our review, we cannot determine whether DOJ’s planned efforts will fully align with the leading practices of the Fraud Risk Framework, but we will continue to monitor DOJ’s efforts during related ongoing work. In fiscal year 2018, OJJDP made 295 awards totaling over $290 million to support programs intended to ensure youth are held appropriately accountable and empower youth to live productive lives. Performance measurement helps ensure funding achieves such outcomes and fraud risk management helps ensure funding is not improperly diverted from this intended purpose. Both of these management principles facilitate stewardship and accountability for federal funds. Since June 2019, OJJDP has been reviewing and updating goal statements and performance measures for individual programs. While OJJDP has office-level targets, it does not have program-level targets. Program-level targets (specific numeric goals) would help OJJDP assess progress toward individual program goals and increase accountability for achieving positive outcomes. Over the past few years, DOJ has taken steps to consider fraud risk for all DOJ grants, including OJJDP’s. Determining a fraud risk tolerance—and assessing residual fraud risk against that tolerance to prioritize these risks—would help OJP calibrate resources to address grant fraud risk for OJJDP programs, helping ensure that resources are not under- or over- allocated. We are making two recommendations, including one to OJJDP and one to JMD. Specifically: The OJJDP Administrator should set performance targets for individual grant programs. (Recommendation 1) The Assistant Attorney General for Administration should ensure that future department-level fraud risk profiles (1) determine the department’s fraud risk tolerance for DOJ grants—which include OJJDP grant programs, and (2) prioritize residual fraud risks based on an assessment against that tolerance, consistent with leading practices in GAO’s Fraud Risk Framework. (Recommendation 2) We provided a draft of this report to DOJ for review and comment. In an email message, an official within JMD stated that the Department concurred with our recommendations. In written comments provided by OJP, which are reproduced in appendix V, the agency concurred with our recommendation that it set performance targets for individual OJJDP programs. Specifically, OJP stated that the OJJDP Administrator will set performance targets, to the extent practical, for OJJDP's current and new grant programs. Further, the OJJDP Administrator will ensure that the performance targets are reviewed annually. We believe this action, if implemented, would address our recommendation. DOJ also concurred with our second recommendation to include a fraud risk tolerance for DOJ grants in future department-level fraud risk profiles, but did not provide details as to how they will address it. DOJ also provided technical comments, which we have incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Attorney General, and other interested parties. In addition, the report is available at no charge on GAO’s website at https://www.gao.gov. If you or your staff have any questions about this report, please contact Gretta Goodwin at (202) 512-8777 GoodwinG@gao.gov or Rebecca Shea at (202) 512-6722 or SheaR@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. Under the Department of Justice Appropriations Act, 2018, the Office of Juvenile Justice and Delinquency Prevention (OJJDP) administered 16 grant programs in fiscal year 2018, corresponding with a total of 53 grant solicitations and 295 awards. OJJDP issued certain grant solicitations to carry out the purposes of more than one program. Table 3 shows OJJDP’s fiscal year 2018 programs and associated information, including the primary source of authority for each program, as identified by OJJDP. To provide examples of performance results from three of the Office of Juvenile Justice and Delinquency Prevention’s (OJJDP) 16 programs funded in fiscal year 2018, we assessed the reliability of key data elements and methods used to calculate selected performance measures. We determined that data grantees submitted to OJJDP for the programs we selected were not sufficiently reliable for this purpose. Nevertheless, we present rounded numbers of “youth served” for two of the programs we selected (2018 data only), as we found these data to be reliable for the specific purpose of illustrating differences in the relative sizes of the programs. This appendix discusses our approach for selecting programs to review, identifying the timeframe of review, and selecting performance measures to review. It also discusses issues we found with the data we reviewed and steps OJJDP is taking to improve the reliability of data submitted by grantees. We selected three of OJJDP’s 16 programs funded in fiscal year 2018 and listed in appendix I. Specifically, we selected the Title II Formula Grant Program because it is OJJDP’s only formula grant program and because it is a comprehensive nationwide program. We selected the Youth Mentoring and Gang Prevention programs from among the 15 discretionary grant programs as a result of the following process: we ranked programs by total amount awarded during fiscal years 2016 through 2018 and randomly selected one program from the top 50 percent and one from the bottom 50 percent, and we excluded programs that we had recently reviewed in other reports—such as the Tribal Youth and Victims of Child Abuse programs. We obtained performance data for the selected programs from OJJDP from October 2015 through December 2018. We chose this timeframe because 2018 was the latest full calendar year for which data were available, and looking back three years was sufficient to capture variations in the programs’ funding levels. We selected seven of OJJDP’s 15 core performance measures for review. Specifically, we selected measures that focus on youth and that objectively measure short-term outputs or outcomes, as follows: number of program youth served number and percent of youth with whom an evidence-based program number and percent of youth completing program requirements number and percent of program youth who offend (short-term) number and percent of program youth who re-offend (short-term) number and percent of program youth who are victimized (short-term); number and percent of program youth who are re-victimized (short- term) Not all of these performance measures are applicable to all programs. Performance measures that assess the number and percent of program youth who offend, re-offend, and are victimized measure both short-term and long-term outcomes, and we focused only on short-term because of the challenge that grantees face in tracking youth after they exit programs, as explained to us by OJJDP officials. We reviewed relevant performance measures, their definitions, and the methodology for calculating them. We reviewed data cleaning and validation procedures that OJJDP uses to verify data provided by grantees. We also tested grantee-submitted data provided to us by OJJDP for missing data, outliers, and inconsistencies. For example, we tested for illogical values, such as different numbers of youth served within the same time period, or numerators that were higher than the denominators (i.e. more youth successfully exiting the program than the total number exiting). Finally, we interviewed knowledgeable OJJDP officials and contractors on several occasions. We determined that the data were not sufficiently reliable for the purpose of providing examples of program performance results due to several limitations (bulleted below). Although not every limitation applies to every program we reviewed, the number and significance of the collective issues identified led us not to use the data as examples of performance results in this report. Performance measure results for discretionary programs are often double-counted when presenting data by year. OJJDP collects data from discretionary grantees every 6-months; however, some performance measures are not designed to be aggregated across time periods. OJJDP acknowledges that aggregating data from two, 6-month time periods often results in double-counting when presenting annualized data. Some grantees report inconsistent numbers of youth served. For instance, in the Gang Prevention program, the total number of youth served from January through June 2018 was reported as 387 and 267 for different performance measures in the same data set (within one reporting period). We also identified instances where grantees reported inconsistent numbers of youth across reporting periods. Some grantees do not respond to requests to verify questionable data. According to OJJDP officials grantees may lack resources or staff capacity to collect data and track youth’s outcomes. As a result, some grantees may submit incorrect data or submit data after the reporting deadline. Beginning October 2017 (for formula grant programs) and July 2018 (for discretionary grant programs), OJJDP implemented a process whereby contractors flag inconsistent or otherwise questionable data and reach out to grantees for clarification. However, according to officials, grantees do not face any consequences if they do not respond to requests from OJJDP for data verification. For instance, for Youth Mentoring program data submitted by grantees for July through December 2018, OJJDP contractors flagged 122 of 630 grantees as having potentially inaccurate data, but subsequently received responses from only 74 of the 122 grantees. Inconsistent performance measure definitions. We found inconsistencies in the definitions for the following performance measures: (1) number and percent of program youth who offend (short-term), and (2) number and percent of program youth who re- offend (short-term). For both measures, one documented definition states that they apply only to youth who offend or re-offend during the reporting period, and another documented definition states that it also applies to youth who exited the program 0-6 months ago. Reporting on a subset of youth not representative of all program youth. OJJDP uses the number of youth “tracked” as the denominator for several of its performance measures, including the number of youth who offend and reoffend. According to OJJDP, the number of youth tracked for the offend and reoffend measures should ideally be the same number as total youth served by a program. However, the number of youth tracked is usually lower than the number of youth served. As a result, the measure often reflects a subset of youth that, according to one official, may have characteristics that are not well understood—such as youth or families who are more willing to be tracked because they have not offended recently—and thus may skew the results. OJJDP acknowledges there are concerns with the quality of the grantee- reported performance measures data. According to OJJDP officials, the limitations are the result of several challenges which they are addressing: Replacing outdated data collection tool. Officials said their current data collection tool is outdated and can be unwieldy and confusing for grantees. Along these lines, the tool only includes automated error checks for a limited amount of data fields and does not include an auto-populate feature, which would prevent grantees from entering illogical or inconsistent data. However, according to Office of Justice Program (OJP) officials, as of October 2019, OJP is designing a new data collection tool for all OJP components—including OJJDP—that will include automated error checks and an auto-populate feature, and they plan to implement this tool beginning in October 2020. Updating performance measures. Officials said that some performance measures are also outdated, such as those that result in duplication when reported annually. Officials also said that some performance measures are confusing to grantees. Nevertheless, as part of an ongoing OJP-wide review of performance measures, OJJDP is in the process of reviewing and updating all OJJDP performance measures and plans to provide updated definitions and instructions to grantees. Increasing grantee response rate for data verification. To increase grantee response rates to data verification requests, OJJDP reported that it is exploring possible consequences for grantees if they do not respond, such as increased scrutiny by OJJDP staff who oversee awards or temporary withholding of funds until verifications are submitted. According to officials, whatever approach (or approaches) they decide on, they will implement them by March 2020. The Office of Juvenile Justice and Delinquency Prevention (OJJDP) within the Department of Justice is responsible for administering grant programs under the Juvenile Justice and Delinquency Prevent Act of 1974. One of these programs, the Title II Formula Grant Program, authorizes the award of formula grants to states to develop programs for juveniles and improve their juvenile justice systems. On December 21, 2018, the Juvenile Justice Reform Act of 2018 (“Reform Act”) enacted amendments to the Title II Formula Grant Program, including new accountability and oversight requirements for grantees and OJJDP. The amendments were not effective until the fiscal year 2020 grant award cycle and did not apply to the period of performance we evaluated for this report, which was through fiscal year 2018. Table 4 summarizes the accountability and oversight requirements now in effect for the Title II Formula Grant Program and the status of OJJDP’s efforts to implement them. The Reform Act also requires several evaluations and assessments to help strengthen OJJDP’s internal controls and identify fraud, waste or abuse in its programs. Table 5 summarizes selected oversight requirements related to the Title II Formula Grant Program. The Title II Formula Grant Program—so called because it was authorized by Title II of the Juvenile Justice and Delinquency Prevention Act of 1974 (JJDPA)—is a state formula grant program, administered by the Office of Juvenile Justice and Delinquency Prevention (OJJDP). The program has been amended several times since 1974—most recently, by the Juvenile Justice Reform Act of 2018 (“Reform Act”), which also called for this evaluation of OJJDP’s performance. The performance data we reviewed (which covers Title II Formula Grants from October 2015 through September 2018) corresponds with statutory requirements in effect at that time, not the current requirements, as amended by the Reform Act, which apply to grant awards made in fiscal year 2020 and subsequent fiscal years. To be consistent with the data we reviewed, this appendix presents information on program requirements that applied prior to fiscal year 2020. Because these requirements are no longer current, we will differentiate them from those that are by citing the superseded edition of the U.S. Code in which they appear—(2012 & Supp. V 2018)—in comparison to the 2018 Main Edition (2018), which contains the provisions now in force. The term “compliance auditing” refers to OJJDP’s process for (1) auditing the compliance monitoring systems used by states, and (2) evaluating states’ compliance with four core requirements specified in law. During the grant application process, the four core requirements are among several (previously 28, now 33) that a state’s 3-year plan must satisfy for the state to be eligible for award. However, unlike the other eligibility requirements, the four core requirements can trigger a reduction to a state’s grant allocation unless the state maintains compliance during performance. States must provide adequate systems of monitoring their compliance with three of the four core requirements—i.e. those related to when and where juveniles may be detained in detention or correctional facilities—and OJJDP must audit the adequacy of states’ compliance monitoring systems. OJJDP must also determine whether states maintained compliance with each of the four core requirements and, if not, OJJDP must reduce the state’s allocation the following fiscal year by at least 20 percent for each core requirement that the state failed to meet. During the period covered in our review (i.e., prior to fiscal year 2020), the four core requirements subject to compliance auditing were: 1. Deinstitutionalization of status offenders—which prohibits states from using secure detention or correctional facilities to hold juveniles charged with status offenses (except for a listed few). This requirement also applies to juveniles not charged with an offense but who enter the justice system as aliens or as dependent, neglected or abused youths. 2. Separation of juveniles from adult inmates—which prohibits a state from detaining or confining juveniles protected by the deinstitutionalization of status offenders requirement (see above), or juveniles who are alleged or found to be delinquent, in any institution where they have contact with adult inmates. 3. Removal of juveniles from adult jails and lockups—which prohibits a state from detaining or confining juveniles in adult jails or lockups, except in limited circumstances and for specified periods of time, and only if the juvenile has no contact with adult inmates. 4. Addressing disproportionate minority contact—which requires a state to address the disproportionate number of minority youth who come into contact with the juvenile justice system. OJJDP’s compliance auditing process during the majority of the period of our review is set forth in a 2017 OJJDP policy document. According to this policy, OJJDP conducts a comprehensive assessment and makes a determination whether the state is in compliance with each of the four core requirements. The comprehensive assessment includes verification of the data submitted, an analysis of the data submitted by the state to evaluate compliance with each of the four core requirements, and a review to assess the adequacy of internal controls over the state’s compliance monitoring process for collecting and reporting compliance monitoring data. According to this policy, the OJJDP Administrator issues correspondence annually regarding final compliance determinations. These determinations include, as necessary, specific details regarding why a state was determined to be out of compliance with any of the four core requirements or the required compliance monitoring system. Per the policy, a state’s formula grant funding will be reduced by 20 percent for each core requirement with which OJJDP has determined a state to be out of compliance. Additionally, if OJJDP determines that the state has an inadequate system of monitoring, the state may have receipt of its formula grant funding withheld or may be deemed ineligible for a formula grant award. Finally, according to this policy, OJJDP conducts field audits on a rotating schedule. The purpose of the field audits is to confirm state compliance monitoring activity and practices through direct onsite observation and file review, and to identify needed areas for technical assistance. OJJDP anticipates, with available funding and resources, that every state will receive a field audit every three years. In addition to the contacts named above, Tonnyé Conner-White (Assistant Director), Jonathan Oldmixon (Assistant Director), Jeff Jensen (Analyst-in-Charge), James Ashley, Dominick Dale, Christine Davis, Caroline DeCelles, Elizabeth Dretsch, Eric Hauswirth, Elizabeth Kowalewski, Ben Licht, Jan Montgomery, Heidi Nielson, and Abby Volk, made key contributions to this report.", "summary": "OJJDP administers grant programs to improve positive outcomes for juveniles in the justice system. In fiscal year 2018, OJJDP made 295 awards across 16 programs totaling over $290 million. The Juvenile Justice Reform Act of 2018 included a provision for GAO to review OJJDP performance and internal controls intended to prevent fraud, waste, and abuse of grant funds. This report examines the extent to which (1) OJJDP has goals and measures to assess the performance of its programs, and (2) DOJ has considered fraud risks for OJJDP grant programs. GAO reviewed DOJ documentation, such as OJJDP's Performance Measures Manual and OJP's risk management policy. GAO also reviewed performance data from selected OJJDP programs from October 2015 through December 2018, and interviewed DOJ officials. The Department of Justice's (DOJ) Office of Juvenile Justice and Delinquency Prevention (OJJDP) has goal statements and performance measures for each of its programs, but has not established corresponding program-level targets (specific numeric goals). Rather, OJJDP has established several office-level targets to help assess progress across OJJDP grant programs collectively. For example, OJJDP has a target for the percent of youth who offend and reoffend across all applicable grant programs. Such office-level targets, while useful, might obscure the results of individual programs. Setting program-level targets would help OJJDP assess the progress of each program and reach its goal of increasing accountability for achieving results in individual programs. DOJ's Office of Justice Programs (OJP) and DOJ's Justice Management Division (JMD) have taken steps to consider fraud risk affecting OJJDP programs. Specifically, OJP—the grant-making component in which OJJDP resides—has tools it uses to monitor grantee performance and compliance with award terms and conditions. According to OJP, these tools—such as checklists used during desk reviews and site visit audits—provide insight into grant fraud risks. Additionally, JMD—the component that manages fraud risk assessment across all components within DOJ—has taken steps to assess fraud risks affecting OJJDP grant programs. Specifically, JMD conducted department-wide fraud risk assessments in fiscal years 2017, 2018, and 2019. These assessments addressed all DOJ grants, including OJJDP's. DOJ's 2017 assessment identified fraud risk scenarios and assessed their likelihood and impact—leading practices in GAO's Fraud Risk Framework. Building on the 2017 assessment, the 2018 assessment identified key fraud risk management activities, and the 2019 assessment resulted in a fraud risk profile. However, these assessments did not determine a fraud risk tolerance—i.e. managers' willingness to accept a specific level of risk—as it relates to OJJDP grant programs. JMD officials said they view this as the next step in the maturation of DOJ's fraud risk assessment processes, but did not have details or documentation of plans to do so. Determining a fraud risk tolerance—and assessing fraud risks against that tolerance to prioritize them—would help OJP calibrate resources to address grant fraud risk for OJJDP programs, helping ensure that resources are not under- or over-allocated. GAO recommends that (1) OJJDP set performance targets for individual grant programs, and (2) DOJ determine the agency's fraud risk tolerance for all grants—which include OJJDP grant programs—and prioritize fraud risks based on an assessment against that tolerance. DOJ concurred with both recommendations.", "document_type": "gao"}
{"report": "To ensure sound management and long-term stability in their operations, organizations track their financial activities (transactions), such as expenses they incur and income they generate. Organizations record their daily transactions, which increase or decrease account balances, in their accounting systems. For example, an organization’s “cash balance” account increases when customers make payments due for goods or services previously provided, while other account balances, such as “accounts receivable” (the amount owed to an organization for goods or services provided), decrease because customers are paying part of what they owed to the organization. At DOD, as seen in figure 1, this daily process of recording transactions in accounting systems occurs at individual DOD components. These components use multiple accounting systems to record and summarize their financial transactions. Each month, quarter, and year, components send summarized financial information to DFAS, the DOD agency that provides accounting support for DOD. DOD’s core financial reporting system consolidates the summarized financial information from individual components into DOD’s department-wide financial information. Financial statements provide information about an organization’s financial position—such as assets (what it owns) and liabilities (what it owes)—as of a certain point in time, in addition to the financial results of its operations—such as revenue (what came in) and expenses (what went out)—over a period of time, such as a fiscal year. Financial statements are prepared based on the summarized, or consolidated, financial information from an organization’s accounting systems. Their reliability depends on there being accurate financial information in the accounting systems. Federal agencies such as DOD combine summarized financial information from their subsidiary organizations (e.g., DOD’s military components—Army, Air Force, Navy, and Marine Corps) to produce consolidated financial statements, as seen in figure 2. Agency management takes steps to ensure that the financial information contained in financial statements is reliable and accurate. Federal agencies submit their financial information to the Department of the Treasury (Treasury), which then combines the information for presentation in the consolidated financial statements of the U.S. government. Reliable and complete financial information is necessary to help agency management and Congress understand the agency’s finances, make informed policy and resource decisions, and hold agency officials accountable for their use of these resources. Accounting adjustments are used to record corrections or adjustments to transactions in an accounting system. They are usually prepared at the end of an accounting period to adjust ending account balances. For example, accounting adjustments can be recorded monthly, quarterly, or annually to record or accrue an activity that is not accounted for in the organization’s accounting systems, such as certain payroll expenses; correct errors identified in processing financial information; record transactions based on the result of reconciliations; record additional information at the request of a subsidiary organization; or record necessary accounting adjustments caused by accounting system limitations or timing differences. Organizations often record such accounting adjustments when preparing financial statements. For example, adjustments to eliminate intragovernmental transactions, such as accounts receivable and sales, may be recorded. These adjustments are particularly necessary to consolidate information from subsidiary organizations and properly present consolidated financial statements. At DOD, components record accounting adjustments within their own accounting systems, and DFAS records adjustments at the consolidated level in DOD’s core financial reporting system. DFAS often has to reformat the summary information it receives from the components’ accounting systems before DOD’s core financial reporting system can accept and process it. To address these or other issues in the financial information it receives as part of the consolidation process, DFAS records accounting adjustments. See figure 3 for an example of where accounting adjustments can be recorded during the consolidation process. DFAS records accounting adjustments both manually and automatically in an accounting system. DFAS personnel record manual adjustments to (1) adjust errors identified during the financial statement compilation process, (2) record necessary accounting adjustments caused by system limitations or timing differences, and (3) prepare required month-end and year-end closing adjustments. System-generated adjustments are automatically recorded in the accounting system without manual involvement. DFAS uses system-generated adjustments when the volume of adjustments needed for a particular purpose is too high and labor-intensive for the adjustments to be recorded manually. For the fourth quarter of fiscal year 2018, DFAS processed 18,521 manual and 181,947 system-generated adjustments at the consolidated level. During the fourth quarter of fiscal year 2018, as noted above, DFAS recorded, at a DOD consolidated level, over 200,000 accounting adjustments in DOD’s core financial reporting system. The large volume of these adjustments is one of the major impediments to DOD maintaining accurate and reliable financial information. While some of these adjustments are expected in the routine course of business, others—such as those DOD records to force account balances to match—are not. We found that DFAS’s lack of reliable business processes and limitations in the source-level accounting systems that DOD components use to process financial information leads them to record adjustments to remove and replace component-submitted financial information in order to force account balances to agree with Treasury balances. The recording of these types of adjustments was identified as a material weakness in DOD’s internal control over financial reporting in its fiscal year 2019 financial statement audit. While DOD has taken steps to address this issue, because of the multitude of contributing factors involved, DOD faces significant challenges in its effort to successfully reconcile its account balances with Treasury and eliminate the need for recording these adjustments. Some manual and system-generated accounting adjustments are expected in the routine course of business and are recurring in nature. For example, elimination adjustments for intragovernmental balances, as previously discussed, are necessary in order to avoid overstating the account balances of subsidiary organizations in the consolidated financial information. The need for these types of adjustments occurs on a regular basis when two or more DOD components enter into business transactions with each other. For example, when the Army purchases weapons from the Defense Logistics Agency, the Army records the transaction as an expense while the Defense Logistics Agency records this transaction as revenue in its accounting system. At the DOD consolidated level, both the revenue and expense reported at the subsidiary level need to be eliminated to avoid overstating revenue and expense for DOD as a whole. These elimination adjustments are routine, expected, and recurring because they must be prepared every time DFAS compiles DOD’s quarterly and annual consolidated financial statements. Other adjustments, such as those DFAS records in order to force account balances to match (forced-balance adjustments) are not expected within the routine course of business. DOD defines a forced-balance adjustment as any amount recorded, usually at a summary level, to eliminate differences between the component’s general ledger balance and Treasury’s control total. Such adjustments, recorded without adequate supporting documentation at the transaction level, are commonly referred to by the accounting community as plugs. Fund Balance with Treasury (FBWT) adjustments are one example of forced-balance adjustments DFAS records to eliminate differences between its cash balances and the amounts Treasury reported. In the federal government, Treasury acts as the government’s bank and keeps an official record of the remaining spending authority for each agency. Consequently, reconciling an agency’s FBWT account with Treasury- reported amounts is similar to an individual reconciling a checkbook to a bank statement. Treasury requires agencies to reconcile their cash balances each month with the balances reported in Treasury’s records. However, DOD generally records adjustments to make its FBWT agree with Treasury’s records rather than performing proper research to identify what caused the differences. (See fig. 4 for more information on forced- balance adjustments.) The use of forced-balance accounting adjustments affects the reliability of an organization’s financial information and may indicate weaknesses within its systems and processes. Over the years, DOD’s practice of recording forced-balance adjustments has been questioned by GAO and by DOD’s auditors. For example, in an audit of the Army General Fund’s reconciliation process for the FBWT account, DOD’s Office of Inspector General (OIG) stated that the Army and personnel in DFAS’s Indianapolis office “make forced-balance adjustments, which are unsupported manual and system-generated adjustments.” GAO and an independent accounting firm both reported similar practices at the Navy and Marine Corps. These audits have repeatedly identified limitations within the source-level accounting systems that DOD components use and the multitude of legacy systems (computer systems that are outdated or that can no longer receive support and maintenance but are still essential for an organization) as the main contributing factors for the use of forced- balance adjustments. DOD’s Financial Management Regulation (FMR) states that a forced- balance adjustment does not represent an adequate reconciliation. Instead, DOD components are required to maintain detailed reconciliation documentation to provide an adequate audit trail. Further, according to the FMR, a reconciliation is not complete until all differences are identified, accountability is assigned, differences are explained, and appropriate adjustments are made to records. These activities are needed to establish an adequate audit trail. Despite this policy, during the fourth quarter of fiscal year 2018, we found that DFAS recorded approximately 36,000, or over 17 percent of the total accounting adjustments, to force the FBWT accounts to agree with Treasury. Out of the 242 fiscal year 2018 fourth quarter accounting adjustments we selected for testing, nine were FBWT forced-balance accounting adjustments related to undistributed collections and disbursements. Based on our review of these adjustments, we found that DFAS continues to rely on forced-balance adjustments to correct the differences between amounts DOD recorded and those that Treasury reported without properly investigating and resolving the differences. Specifically, we found that DFAS systematically recorded forced-balance adjustments to replace information that DOD components submitted with the amounts that Treasury reported without reconciling and researching the causes of differences and making any appropriate adjustments. DFAS indicated that it performs reconciliations on the FBWT accounts when compiling financial statements and researches the causes of any differences arising from these reconciliations after it records the forced-balance adjustments. However, for our sample of nine FBWT forced-balance adjustments related to undistributed collections and disbursements, DFAS was unable to provide evidence that these reconciliations were performed or that the causes of differences were researched or resolved. Rather, DFAS provided a general description of the reconciliation process it expects each of the three DFAS sites to perform. According to DOD officials, DOD has identified some key causes of the long-standing challenges in reconciling its account balances with Treasury. As noted above, many of these challenges are caused by timing issues, limitations in the source-level accounting systems that DOD components use, or the multitude of legacy systems that different DOD components use. To address these challenges, DOD is currently implementing enterprise resource planning (ERP) systems in the military services, such as the Defense Enterprise Accounting and Management System that the Air Force uses. These systems will replace the current legacy systems across DOD with the expectation of a full transition to ERP systems at all military services by 2025, at which point DOD expects the need to record forced-balance adjustments to decrease. OUSD (Comptroller) has a plan for implementing ERP systems at smaller DOD components that also use legacy systems. However, the challenges that legacy systems cause are likely to continue until the ERP transitions are completed and ERP systems are fully implemented at the military services and smaller DOD components. A DFAS official stated that until DOD fully implements the ERP systems, DFAS does not have any plans to modify the current financial management environment to eliminate the recording of these types of adjustments. As noted earlier, as part of the routine course of business certain adjustments will still need to be made following the full implementation of the ERP systems. Along with DOD’s implementing of the ERP systems, DOD officials stated that some DFAS sites, in coordination with various DOD components, have implemented tools to help them reconcile FBWT balances and research the causes of any differences arising during these reconciliations. According to DOD officials, these tools have the ability to produce supporting documentation for management and auditors to use when reviewing FBWT accounts. However, DFAS did not provide supporting documentation for these reconciliations in order for us to verify that they had been performed. Until DOD consistently performs and documents the required reconciliations to identify the causes for these types of adjustments and takes a holistic approach to resolving them, DOD’s financial management issues—such as those associated with FBWT—are likely to continue, resulting in a continued inability to produce reliable and auditable consolidated financial statements. Establishing clear policies and procedures for recording accounting adjustments is crucial for (1) ensuring that accounting adjustments are properly recorded and adequately supported with documentation; (2) identifying the underlying causes for the recording of adjustments; and (3) developing, implementing, and monitoring action plans to reduce the need for accounting adjustments. We found that DOD and DFAS policies and procedures for recording accounting adjustments were insufficient, outdated, and not consistently implemented. Additionally, we found that DOD and DFAS lacked policies and procedures in certain key areas, such as performing cause analyses and developing action plans to reduce the need for accounting adjustments. By not ensuring that policies and procedures are up-to-date and consistently implemented, DOD faces an increased risk that inaccurate, invalid, or unapproved adjustments will be recorded in its core financial reporting system, resulting in misstatements in its consolidated financial statements. System-generated accounting adjustments are recorded automatically in an accounting system and have unique characteristics and processes that differ from those applicable to manual accounting adjustments. Unlike manual adjustments, which are initiated, recorded, and approved in the accounting system by a person, system-generated adjustments are guided by business rules embedded in an accounting system. These business rules drive the accounting adjustment process and are configured to record the adjustment when certain conditions are met. We found that DFAS lacked documentation to support the business rules, such as documentation of programming logic that creates the system- generated adjustments. Based on our review of accounting adjustments at DOD, we found that system-generated adjustments are recorded in large numbers and account for the majority of the accounting adjustments that DFAS recorded. For example, for the fourth quarter of fiscal year 2018, system- generated accounting adjustments accounted for over 90 percent of the total volume of adjustments recorded in DOD’s core financial reporting system at the consolidated level. Given the magnitude and unique characteristics of system-generated adjustments, developing and maintaining adequate supporting documentation are critical. According to the FMR, adjustments to the accounting records should be supported with sufficiently detailed written documentation to provide an audit trail to the source transaction that requires the adjustment. Further, the FMR requires supporting documentation to include information such as the reason for the adjustment, calculation of the adjustment amount, and evidence of managerial review and approval of the adjustment. To support certain types of recurring system-generated adjustments, DFAS developed eight standardized narratives that include the reasons for the adjustments and the documentation DFAS considers necessary to support the adjustments. Other recurring system-generated adjustments are recorded based on System Change Requests, which are proposals to modify information in an accounting system such as revising programming logic and coding changes. In the fourth quarter of fiscal year 2018, DOD determined that 74 percent of the recorded system- generated adjustments related to four of the eight standardized narratives. Most of the System Change Requests we tested related to financial information migration from a legacy system to a new responsible work area for the U.S. Army Corps of Engineers. DFAS annually selects and reviews a random sample of 40 system- generated adjustments related to each type of narrative for which supporting packages are prepared. According to DFAS officials, the supporting package preparation for the selected sample involves verifying that the adjustments impacted the intended accounts. If no issues are identified, DFAS concludes that the core financial reporting system recorded the adjustments as intended, the desired results were achieved, and the adjustments were supported. Within a given year, if DFAS sample testing demonstrates that a certain type of system-generated adjustment was supported, DFAS categorizes all the accounting adjustments that relate to this particular type as supported for the rest of the year. We found that other than the supporting packages created specifically for the periodic random samples, DFAS maintains no other documentation to support the system-generated adjustments related to each of the eight narratives. As part of our audit, we selected for testing 242 accounting adjustments that impacted the financial statements for the fourth quarter of fiscal year 2018, of which 93 were system-generated accounting adjustments. Of these 93 adjustments, DFAS categorized 42 as unsupported and 51 as supported. DFAS categorized adjustments as either supported or unsupported depending upon the circumstances. The circumstances considered include whether it relates to one of the eight narratives or to specific System Change Requests. When the eight narratives are tested, if no issues are identified as part of the testing, the transactions linked to that narrative are considered supported. However, we determined that the corresponding narratives and System Change Requests were insufficient support for the 51 adjustments categorized as supported because we were unable to verify the validity and accuracy of the adjustments with supporting documentation. In addition, we found that DFAS did not maintain evidence demonstrating the review and approval of the programming of predefined business rules in the systems that recorded the adjustments. When our results are projected to the fiscal year 2018 fourth quarter population of 181,947 system-generated adjustments, we estimate that at least 96 percent of the system-generated accounting adjustments were recorded without adequate supporting documentation, which is required by DOD’s policy and procedures and federal internal control standards. In 2018, DFAS’s auditor issued a finding identifying similar issues with DFAS’s system- generated adjustments related to the scope of the eight narratives. As of November 2019, this finding was still open. Standards for Internal Control in the Federal Government requires that management design control activities to achieve objectives and respond to risks, such as designing controls to help ensure accurate and timely recording and maintenance of appropriate transaction documentation. Although the FMR has guidance on supporting documentation requirements for accounting adjustments, we found that DOD’s FMR does not clearly define or include examples of what constitutes adequate supporting documentation of system-generated accounting adjustments. Specifically, the FMR does not differentiate between documentation requirements for manual and system-generated accounting adjustments. Rather, it states that reporting organizations must maintain adequate documentation, audit trails, and internal controls, and that the documentation must be made available upon request. Because system- generated accounting adjustments consist of summary-level financial information, DFAS officials stated that maintaining documentation at a detailed level would be impractical given the large volume of transactions at DOD. However, without adequate supporting documentation for the business rules driving the recording of these adjustments, such as documentation of the programming logic for these adjustments, management and others cannot determine whether an adjustment was recorded for a valid reason or for the correct amount. The March 2002 version of the DOD’s FMR, volume 6A, chapter 2, established 10 category codes that are used to identify the circumstances under which accounting adjustments may be recorded. For example, DFAS uses category A for reversing entries for a prior reporting period and category B for data call adjustments. Additionally, the FMR specifies the required documentation needed to support each category. For example, for category A adjustments, adequate documentation includes information on the original entry and a statement that the adjustment is a reversing entry, whereas for category B adjustments, documentation requirements include information on the summarized amount and identification of the source or location of the transaction-level detail for the adjustment. DOD’s core financial reporting system is designed to allow a DFAS accountant to select one of these 10 codes when recording an adjustment. Since these codes are used to identify the required documentation to support the adjustments, it is important that the codes are periodically reviewed to ensure that they are still relevant in DOD’s current financial reporting environment and that supporting documentation requirements are appropriate. We found that some of the category codes were rarely used and new codes may need to be added to reflect the current financial reporting environment. For example, we found that of the 18,521 manual adjustments recorded in DOD’s core financial reporting system during the fourth quarter of fiscal year 2018, category F (supply management inventory) was used only four times. According to DFAS officials, this code was primarily used to adjust the purchase cost of certain supplies, but those adjustments are now rarely needed. DFAS officials stated that the codes had not been reviewed for continued relevance since they were first established and expressed a need to revisit the current categorization scheme to determine whether the codes should be redefined. The most recent update of FMR, volume 6A, chapter 2, in June 2019 included the addition of category M, the first code added since 2002. According to DFAS officials, this code was not added based on a thorough review of the existing codes but because there was already a substantial volume of data call adjustments taking place in the Data Collection Module. Further, a DOD official suggested that an additional code may be needed for tie-point adjustments, which DFAS accountants frequently record as a result of tie-point reconciliation. We found that 34 of the 149 manual accounting adjustments we tested related to tie-point adjustments. Because there is not a designated category code for tie-point adjustments, we found that accountants used various other category codes when recording the 34 tie-point adjustments, including D (Recognition of Undistributed Disbursements and Collections), E (Reconciliation of Trial Balance and Budget Execution Reports), G (Reclassification of Accounts), H (Identified Errors and Reasonableness Checks), I (Adjustment to Balance Reports Internally), or at times no category code. As a result, there may be inconsistency in the documentation maintained to support tie-point adjustments. Having a single category code for tie-point adjustments could standardize recording by accountants, enabling DFAS to identify the frequency with which tie- point adjustments are recorded and ensure that it maintains adequate supporting documentation. Standards for Internal Control in the Federal Government requires that management implement control activities through policies. To do this, management periodically reviews policies, procedures, and related control activities for continued relevance and effectiveness in achieving the organization’s objectives or addressing related risks. The DOD FMR Revision Standard Operating Procedures indicates that the FMR is reviewed every 2 years. However, based on our discussion with DOD officials, a thorough review of the category codes has not been performed and is needed to ensure the ongoing applicability of current category codes or the need for additional codes to reflect the current financial reporting environment. Without category codes for accounting adjustments that reflect current business needs, there is an increased risk that the reasons for recording these adjustments will not be properly captured and adequate supporting documentation will not be specified or maintained, hindering DFAS’s ability to provide DOD management or auditors with reliable information about recorded accounting adjustments. The FMR identifies critical elements that need to be included as part of the supporting documentation package when recording manual accounting adjustments. These elements include (1) correct appropriation and accounting information, (2) balanced adjustments, (3) approvals, (4) supporting documentation, and (5) valid U.S. Standard General Ledger (USSGL) account numbers. The FMR further states that supporting documentation included in the package must include, among other things, elements to enable the assessment of the (1) accuracy and completeness of financial information recorded, (2) applicable criteria to support the reason for recording the adjustment, (3) specific expenditure or receipt accounts used, and (4) calculation of the dollar amount of the adjustment. For the fourth quarter of fiscal year 2018, we selected a sample of manual and system-generated adjustments to determine if the supporting documentation for these adjustments included the critical elements described in DOD’s FMR. We found that DFAS accountants did not consistently follow the DOD FMR and DFAS’s policies and procedures for some of these critical elements, resulting in (1) the failure to maintain adequate supporting documentation, (2) the recording of out-of-balance accounting adjustments, and (3) the use of account numbers that do not comply with the USSGL. Inadequate supporting documentation: We found that 51 of the 87 manual adjustments we reviewed that DFAS categorized as supported did not contain supporting documentation required by the DOD FMR. For example, we found instances where supporting documentation packages were missing information to support the reason for recording the adjustment or detailed worksheets to support the calculation of the adjustment amount. DFAS officials explained that 30 of the 51 adjustments resulted from a major change in how DFAS processes U.S. Army Corps of Engineers’ financial information. Because of time sensitivity and based on a risk analysis, management decided to process over 3,000 manual adjustments for the fourth quarter of fiscal year 2018, including the 30 that were selected for our sample, without preparing supporting documentation for these individual adjustments. We also found that not all supporting documentation packages included a DFAS Form 9339, DFAS Journal Voucher Catalog and Checklist, as required by DFAS’s Interim Policy Memorandum. This memorandum requires that all manual accounting adjustments, whether classified as supported or unsupported by DFAS, include a Form 9339 to help ensure the inclusion of the appropriate supporting documentation. We reviewed 149 manual adjustments and found that 94 included a Form 9339. Our review of these 94 accounting adjustments found that 28 lacked the required information. For example, nine packages did not include one or more of the necessary elements required by Form 9339. We found instances where source information, customer coordination, document labeling, before and after trial balances, narratives, or a combination of these were missing. For six of the nine packages, DFAS agreed that some of the necessary data elements were missing; for the remaining three packages, DFAS’s response did not fully address the reasons for the missing documentation. DFAS officials stated that one of the reasons why these errors may have occurred was because the implementation of the Interim Policy Memorandum was in its beginning stages when we selected our fourth quarter fiscal year 2018 sample. The policy memorandum was dated June 11, 2018, and was effective immediately. We also found that 19 adjustments lacked the required root cause indicator code on the Form 9339. According to DFAS officials, DFAS Cleveland prepared those 19 forms. DFAS Cleveland officials explained that in collaboration with the Navy Financial Management Office, DFAS Cleveland’s senior leadership decided to deviate from the Interim Policy Memorandum and not include the root cause indicator code in the Form 9339 when recording Navy’s related accounting adjustments. Rather, DFAS Cleveland developed its own system to identify root causes by using a unique identifier code. However, the Interim Policy Memorandum does not exclude any DFAS site from adhering to the requirement, and DFAS was unable to provide documentation to demonstrate that DFAS Cleveland had authorization to deviate from this policy. According to DFAS officials, DFAS Cleveland will begin including the root cause indicator code on the Form 9339, starting second quarter of fiscal year 2020. Out-of-balance adjustments: We found that about 2,800 manual adjustments, or approximately 15 percent of all manual adjustments recorded at the DOD consolidated level for the fourth quarter of fiscal year 2018, were out-of-balance. For example, we identified one adjustment in which DFAS decreased its FBWT account by $14,232,000 without recording a change to a corresponding account. DOD’s FMR requires that all recorded accounting adjustments be balanced. Additionally, we found that the FMR does not identify any situations where an out-of-balance adjustment is allowable, despite DFAS officials stating that out-of-balance adjustments are sometimes necessary. Auditors of the military services found that reasons for out-of-balance financial information include (1) service-level general ledger systems are not effectively designed to prevent incomplete transactions from being recorded and (2) controls are not in place at the service level to detect these errors in a timely manner. According to DFAS officials, out-of-balance adjustments are recorded to correct out-of-balance financial information received from DOD components’ accounting systems. Use of non-USSGL-compliant accounts: During our review of fourth quarter fiscal year 2018 manual and system-generated adjustments, we found that over 13,000 adjustments (over 6 percent) recorded at the DOD consolidated level used non-USSGL-compliant accounts, which are not allowed by the Treasury Financial Manual or DOD FMR. The Federal Financial Management Improvement Act of 1996 requires certain federal agencies, such as DOD, to use the specific and standardized set of accounts referred to as the USSGL in their financial reporting systems. Treasury maintains this set of accounts annually to help ensure the comparability of financial information across the federal government. DFAS officials stated that these noncompliant accounts are referred to as memo accounts and were primarily used for management planning purposes. They further explained that DOD had controls in place to prevent financial information recorded in memo accounts in DOD’s core financial reporting system from being transferred into the financial reporting systems, which Treasury uses to compile the U.S. government consolidated financial statements. DOD OIG also reported issues related to DOD’s use of noncompliant accounts, which it identified in fiscal year 2018. In addition, in DOD’s Fiscal Year 2018 Agency Financial Report, management acknowledged that DOD’s financial management systems did not comply with the USSGL at the transaction level. In response, the DOD Deputy Chief Financial Officer issued a memorandum on March 15, 2019, acknowledging that DOD components must use the established USSGL accounts identified in the Treasury Financial Manual for financial reporting purposes. Additionally, the memorandum stated that supporting documentation must be maintained for any accounting adjustments recorded using memo accounts. However, for the seven non-USSGL-compliant adjustments included within our sample of manual and system-generated adjustments, we found that documentation had not been maintained. Proper recording of adjustments is crucial for ensuring that the financial information accurately reflects the financial transactions that have occurred. This includes maintaining adequate supporting documentation and implementing review procedures to help ensure controls are in place to detect errors in a timely manner. Failure to fully adhere to established procedures increases the risk that inaccurate accounting adjustments will be recorded, thereby reducing the reliability of reported financial information and potentially causing misstatements in the DOD consolidated financial statements. Organizations use root cause analysis as a tool to identify and evaluate systems, processes, or both that prompted the recording of an accounting adjustment. For certain adjustments, it may be determined that a root cause analysis or action plan is not necessary—for instance, if the adjustment is onetime or nonroutine. However, information obtained through a root cause analysis may be used to make system or process changes within a specific program, thus reducing the need to record adjustments. Once a root cause has been identified and analyzed, an organization should create an action plan that describes the steps to be taken to address the root cause and monitors the effectiveness of the actions taken. Figure 5 illustrates the identification, implementation, and monitoring of accounting adjustment root causes and related action plans. We found that although DFAS headquarters and its individual sites perform root cause analysis and develop and take some actions to address the identified causes, neither DOD nor DFAS has established policies and procedures that require staff to perform root cause analysis; develop and implement action plans for issues that DFAS staff identified; or monitor the effectiveness of action plans in eliminating the need for accounting adjustments. DFAS officials acknowledged that there are no policies in place requiring DFAS to perform root cause analyses that would permit them to compare root causes for accounting adjustments at a consolidated level across the DFAS sites. DFAS and its sites identify root causes for individual accounting adjustments when accountants select a root cause indicator code when recording the adjustment. DFAS staff also identify root causes for accounting adjustments at an aggregate level when preparing summary metrics on adjustment types. For example, we found that in addition to the requirements previously discussed, Form 9339 requires accountants across all DFAS sites to prepare “white papers/narratives (white papers) each time a manual adjustment is recorded.” Our review of these white papers identified that some DFAS sites use the white papers to document the root cause analyses while others do not because DFAS has not provided a template that identifies the minimum required information to be included in the white papers. As a result, we found that individual DFAS sites do not use standardized white paper templates and that the information included in the white papers was not always consistent between and within the DFAS sites. Inconsistency between DFAS sites: Our review of 52 white papers found that DFAS Indianapolis was the only site to include information such as scope, source system, and financial statement impact of the accounting adjustment in its white papers, while DFAS Columbus was the only site that included corrective actions taken. (See fig. 6.) Inconsistency within a DFAS site: Our review of 52 white papers found that DFAS Indianapolis did not consistently include background, purpose of the adjustment, a description of the root cause, posting logic, financial statement impact, pending action, source system, and scope in all white papers it prepared. Some white papers had these elements and others did not. We found similar issues with the white papers prepared by DFAS Columbus and DFAS Cleveland. Figure 7 illustrates information included in white papers that was inconsistent within a DFAS site. We also found that DFAS does not have policies and procedures requiring the identification and implementation of action plans to address the cause of and need for accounting adjustments that staff identified internally. This resulted in inconsistencies in how the different DFAS sites developed and implemented action plans. For example, we found that 73 of the 98 manual and system-generated unsupported adjustment packages that included a root cause analysis that we reviewed did not include an action plan to address the root cause. Based on this testing, we estimate that at least 88 percent of fourth quarter fiscal year 2018 unsupported adjustments for which a root cause analysis was performed did not have a documented action plan. For the remaining 25 packages with documented action plans, we found that only two included steps documenting how the action plan was to be implemented. The remaining 23 packages with documented action plans lacked implementation details, and DFAS officials stated that they were waiting for resolution from the relevant DOD components. DFAS officials stated that they prepare and document action plans for issues that affect multiple accounting adjustments but not for issues that affect only one adjustment (unique root cause). Unique root causes do not necessitate action plans and are resolved the following month through DFAS working with the affected DOD components. According to DFAS officials, many of the action plans are discussed in biweekly and monthly meetings, but these action plans are not documented. Finally, we found that DOD and DFAS do not have policies and procedures requiring management to monitor the results of action plans that individual DFAS sites prepared or to measure whether implemented action plans are effective in addressing the causes for accounting adjustments. DFAS management activities were limited to periodically reviewing summary metrics on the numbers and types of accounting adjustments recorded. These metrics did not contain detailed information, such as the causes of accounting adjustments to be addressed, accountable officials responsible for implementing action plans, expected time frames for the implementation of action plans, or specific steps to be performed to address the causes. Additionally, the metrics did not include any information on action plans to address system-generated accounting adjustments, which account for the majority of the adjustments. This type of detailed information is critical to DOD management and DOD external stakeholders for evaluating the department’s progress in correcting the issues. GAO has previously reported that a lack of comprehensive information on corrective action plans limits DOD’s and Congress’s ability to evaluate DOD’s progress toward fully, timely, and efficiently correcting its long-standing financial management deficiencies. Office of Management and Budget Circular A-123 requires agencies to perform cause analysis of deficiencies identified to ensure that subsequent strategies and plans address the causes of the problem and not just the symptoms. Additionally, Standards for Internal Control in the Federal Government requires that management implement control activities through policies. To do this, management documents in its policies the internal control responsibilities of the organization. In addition, management periodically reviews policies, procedures, and related control activities for continued relevance and effectiveness in achieving the entity’s objectives or addressing related risks. Without policies that require consistent processes across DOD for identifying and addressing the causes of accounting adjustments, from the identification of underlying root causes to the development, implementation, and monitoring of action plans, it is likely that DOD’s efforts to reduce accounting adjustments will be inefficient and ineffective. The large number of accounting adjustments that are recorded in the preparation of DOD’s financial statements presents audit challenges. To address this issue, DOD and DFAS have established multiple initiatives aimed at reducing accounting adjustments. While these initiatives have resulted in fewer adjustments of certain types being recorded at the component and consolidated levels, the current focus has been on reducing the number of adjustments recorded without adequate supporting documentation within their responsible work areas, not on reducing the need for recording accounting adjustments department-wide. Both DFAS and OUSD (Comptroller) have developed department-wide strategies to decrease accounting adjustments; however, neither DFAS nor OUSD (Comptroller) have developed procedures for implementing the department-wide strategies. Without a clear department-wide approach to reducing accounting adjustments across all DOD components, there is a risk that DOD’s effort to reduce accounting adjustments will be unsuccessful, which in turn hinders its ability to produce reliable and auditable consolidated financial statements. To reduce accounting adjustments, OUSD (Comptroller) and DFAS have undertaken many initiatives over the last few years. In fiscal year 2018, OUSD (Comptroller) determined that a large number of accounting adjustments at the consolidated level resulted from data calls. To decrease the need for recording these adjustments at the consolidated level, DOD established the Data Call Journal Voucher (JV) Migration Initiative with the goal of eliminating data call adjustments in its core financial reporting system to the maximum extent possible. The first phase of this initiative moved the recording of adjustments for the Federal Employees’ Compensation Act (FECA) liability to the DOD component level responsible for the underlying transaction. According to OUSD (Comptroller), this initiative resulted in the successful migration of the recording of the Missile Defense Agency’s FECA liability from DOD’s core financial reporting system to Missile Defense Agency accounting systems in the second quarter of fiscal year 2019, and has set the stage for 19 other components using the same accounting system as the agency to follow suit. Although this initiative may not reduce the overall number of accounting adjustments that DOD records, it will reduce the need for data call adjustments to be recorded at the consolidated level. OUSD (Comptroller) expects this initiative will also enhance the quality of the supporting documentation maintained for these types of adjustments because the underlying transaction-level detail for the adjustments will be available in the components’ accounting systems. Individual DFAS sites have also undertaken their own initiatives that eliminate the need for some accounting adjustments. For example, in fiscal year 2018, DFAS Indianapolis found that some financial information from the Army’s accounting systems was improperly recorded, requiring adjustments to correct the errors when the financial information transferred into the DOD’s core financial reporting system. DFAS Indianapolis staff worked with the Army to resolve the issue. As a result, adjustments are no longer needed at the consolidated level when the information transfers from Army’s system into DFAS’s system. According to DFAS Indianapolis officials, this initiative resulted in a significant decrease in accounting adjustments at the consolidated level for fiscal year 2019. Developing and implementing a DOD department-wide strategy to reduce the need for recording accounting adjustments at the consolidated level requires DOD to identify the underlying root causes and risks associated with accounting adjustments and to prioritize efforts to address them. This involves clearly defining what is to be done, who is to do it, how it will be done, and the time frames for achievement. To address DOD’s many financial management issues, including reducing accounting adjustments, OUSD (Comptroller) and DFAS have developed different strategies and business plans. However, these strategies and business plans do not include clearly defined expected outcomes or procedures for achieving stated goals. OUSD (Comptroller) issued the DOD Financial Management Strategy Fiscal Years 2016–2020 (Strategy) to help achieve a simplified, standard, affordable, auditable, and secure financial environment, which includes the reduction of accounting adjustments. The Strategy’s JV initiative states that “The purpose of this initiative is to determine why unsupported JVs occur and resolve them.” However, we found that the Strategy did not provide clear direction to staff on how to achieve the JV initiative and did not call for a department-wide effort to address accounting adjustments recorded at the consolidated level. The Strategy also acknowledged that excessive adjustments can indicate underlying problems, such as weak internal controls, and may indicate that transactions are not captured, reported, or summarized correctly. However, we found that the focus of the Strategy was on reducing the number of accounting adjustments recorded without adequate supporting documentation rather than on reducing the overall need for recording accounting adjustments department-wide. In addition to following the OUSD’s (Comptroller) Strategy, DFAS management has also developed the Fiscal Years 2017—2021Strategic Plan (Strategic Plan) and Fiscal Year 2018 Annual Business Plan (Business Plan), which include goals for reducing accounting adjustments, supplemented by bimonthly Strategy Updates. For example, DFAS’s November 2017 Strategy Update outlined the Business Plan goals for fiscal year 2018 with regard to internal controls and business processes. In that update, DFAS set broad goals, such as executing plans to support or reduce system-generated and manual adjustments. However, we found that similar to the OUSD (Comptroller) Strategy, neither DFAS’s Strategic Plan nor Business Plan included defined outcomes or clear procedures for accomplishing the stated goals. The primary focus of these goals was also to reduce the number of accounting adjustments recorded without adequate supporting documentation. We found that this lack of clear procedures led each DFAS site and DFAS headquarters to focus their initiatives on accounting adjustments that impacted their responsible work areas instead of reducing the need for recording accounting adjustments overall. According to DFAS site officials, in some instances, reducing the number of accounting adjustments recorded without adequate supporting documentation at their individual sites could have an impact on the need to record adjustments at the consolidated level; however, the effect at the consolidated level was not their primary focus. Standards for Internal Control in the Federal Government requires that management implement control activities through policies. To do this, management documents in its policies the internal control responsibilities of the organization. In addition, management periodically reviews policies, procedures, and related control activities for continued relevance and effectiveness in achieving the entity’s objectives or addressing related risks. Without detailed documented policies and procedures for implementing its initiatives, and a complete understanding of the issues contributing to the recording of accounting adjustments (both supported and unsupported) across DOD, there is an increased risk that management efforts to reduce accounting adjustments at the consolidated level will be ineffective. In the routine course of business, organizations often record accounting adjustments on a monthly, quarterly, and annual basis. Some adjustments are necessary so that financial information is presented meaningfully and accurately. However, an extensive use of accounting adjustments may indicate significant underlying problems. In order to produce reliable financial information that DOD management and Congress can use for decision-making, DOD needs to develop policies and procedures for recording accounting adjustments that are consistently implemented across the department and reflect the current DOD financial reporting environment. DOD also needs to address the issues that contribute to its need to extensively record accounting adjustments by implementing policies and procedures for the consistent identification of the causes for recording adjustments and the development, implementation, and monitoring of action plans to address the identified causes. If DOD does not address these issues, there is an increased risk that its financial information will be misstated and DOD will continue to be unable to prepare reliable and auditable consolidated financial statements. We are making the following eight recommendations to DOD: The Director of DFAS should, in accordance with the FMR, implement procedures to help ensure that FBWT reconciliations are consistently performed and that all DFAS sites review and document research conducted on the causes of any differences arising from these reconciliations. (Recommendation 1) The Under Secretary of Defense (Comptroller) should update the FMR to clearly define the required supporting documentation for system-generated accounting adjustments, including the required documentation of business rules driving the recording of these adjustments, such as documentation of the programming logic. (Recommendation 2) The Under Secretary of Defense (Comptroller) should perform and document a comprehensive review of the FMR accounting adjustment category codes to determine their ongoing applicability or the need for additional codes to reflect the current financial reporting environment. (Recommendation 3) The Under Secretary of Defense (Comptroller) should establish procedures to help ensure the consistent implementation of the requirements of DFAS Form 9339. (Recommendation 4) The Under Secretary of Defense (Comptroller) should update policies and procedures to identify the causes of out-of-balance accounting adjustments and resolve the causes in a timely manner. (Recommendation 5) The Under Secretary of Defense (Comptroller), in conjunction with the Director of DFAS, should develop and implement policies and procedures to help ensure that root cause analyses for accounting adjustments are consistently performed and documented across DOD. (Recommendation 6) The Under Secretary of Defense (Comptroller), in conjunction with the Director of DFAS, should develop and implement policies and procedures to help ensure consistent development, implementation, monitoring, and documentation of action plans across DOD that address accounting adjustment causes that staff identified internally. (Recommendation 7) The Under Secretary of Defense (Comptroller), in conjunction with the Director of DFAS, should develop and implement procedures across DOD that include clearly defined outcomes focused on reducing accounting adjustments (supported and unsupported) with specific actionable steps and procedures for achieving stated goals. (Recommendation 8) We provided a draft of this report to DOD for review and comment. In written comments, DOD concurred with all eight of our recommendations and cited actions to address them. DOD’s comments are reproduced in appendix II. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 11 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense (Comptroller), the Director of the Defense Finance and Accounting Service and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2989 or kociolekk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines (1) accounting adjustments and their effect on the reliability of the Department of Defense’s (DOD) financial information, (2) the extent to which DOD has established and implemented policies and procedures for recording accounting adjustments, and (3) the extent to which DOD has taken actions to reduce accounting adjustments recorded at the consolidated level. To determine the accounting adjustments recorded and their effect on the reliability of DOD’s financial information, we focused our review on the categories of accounting adjustments that DOD recorded at the consolidated level. We reviewed prior audit reports issued by GAO, DOD’s Office of Inspector General (OIG), and independent public accountants for fiscal years 2015 through 2019 to gain an understanding of the types and categories of accounting adjustments. We also reviewed related policies and procedures, such as DOD’s Financial Management Regulation (FMR); performed walk-throughs of the Defense Finance and Accounting Service’s (DFAS) processing of accounting adjustments; and interviewed DOD officials to gain an understanding of the types of accounting adjustments. Additionally, we obtained and analyzed summary information on the adjustments affecting fiscal years 2017 and 2018 by quantity, dollar value, whether they were manual versus system- generated, and unsupported versus supported. To evaluate the extent to which DOD has established and implemented policies and procedures for recording accounting adjustments, we reviewed relevant notices of finding and recommendation that the independent public accountants and DOD OIG issued related to accounting adjustments for fiscal year 2018. We also reviewed DOD and DFAS policies and procedures and interviewed officials from DFAS and the Office of the Under Secretary of Defense (OUSD) (Comptroller) to identify issues surrounding accounting adjustments and the procedures used to process, review, and approve these adjustments in DOD systems. We also inquired about the procedures used to determine the underlying causes of accounting adjustments, if action plans to address the causes had been developed, and the status of these plans. In addition, we assessed DFAS’s efforts to monitor the effectiveness of its action plans. To determine the specific internal controls DOD had in place over its accounting adjustment processes, we interviewed DFAS officials knowledgeable about the accounting adjustment processes and performed walk-throughs of these processes at DFAS. We evaluated the procedures observed during our walk-throughs and those that DOD officials described to determine whether DFAS recorded adjustments in accordance with established policies and procedures. For issues identified, we interviewed DOD officials to confirm our understanding and determined the reasons for the issues identified. To determine if DOD had designed and implemented internal controls over its accounting adjustment processes, we analyzed the information we obtained through the interviews and walk-throughs using relevant criteria, including the DOD FMR, the Treasury Financial Manual, and our Standards for Internal Control in the Federal Government. We also performed tests of controls on a random sample of 242 accounting adjustments from a population of 200,468 adjustments that DFAS recorded at the consolidated level that impacted the financial statements for the fourth quarter of fiscal year 2018. The selected adjustments were recorded in the Defense Departmental Reporting System (DDRS)— Budgetary (DDRS-B), DDRS—Audited Financial Statements (DDRS- AFS), and DDRS-AFS Beginning Balance Adjustment modules. From the DDRS-B and DDRS-AFS modules, we selected a random sample of 225 accounting adjustments, and from the DDRS-AFS Beginning Balance Adjustment module we selected all 17 accounting adjustments. From the three different sets of data, we stratified the selected accounting adjustments into six strata (see table 1). Of the total 242 adjustments, we selected all 17 adjustments in stratum 1, and 45 adjustments each from strata 2 through 6 for testing. We designed the sample to support estimation for all supported accounting adjustments with a margin of error no greater than plus or minus 11.7 percentage points at the 95 percent level of confidence, estimation for all unsupported accounting adjustments with a margin of error no greater than plus or minus 11.8 percentage points at the 95 percent level of confidence, and estimation overall for all accounting adjustments with a margin of error no greater than plus or minus 8.4 percentage points at the 95 percent level of confidence. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval (e.g., plus or minus 8 percentage points). This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. For the accounting adjustments in our sample that DOD considered supported, we reviewed underlying documentation to determine whether the adjustments were properly supported and contained all critical elements required by DOD policy. We then shared the results of testing with DOD and incorporated any applicable additional information DOD officials provided into our analysis, as appropriate. As part of our testing, we also reviewed documentation related to unsupported accounting adjustments selected in our sample and interviewed DFAS officials to determine if DOD had performed root cause analyses, developed action plans to address the identified causes, and taken any actions in response. We then shared the results of testing with DOD and incorporated any applicable additional information DOD officials provided into our analysis, as appropriate. To assess the reliability of the accounting adjustment information we received from DOD, we conducted interviews with relevant agency officials, compared summary-level dollar amounts and quantities to another DOD information source, performed electronic testing of the financial information, and reviewed related internal controls. On the basis of this work, we found the financial information to be sufficiently reliable to project results of our random sample testing to the population of accounting adjustments for the fourth quarter of fiscal year 2018. Margins of error varied depending on the specific stratum being projected and are disclosed with all estimates contained within the report. To determine the extent to which DOD has taken actions to reduce accounting adjustments recorded at the consolidated level that may affect the reliability of its financial information, we interviewed officials from DFAS and the OUSD (Comptroller) to identify initiatives aimed at reducing accounting adjustments. We further inquired about what tools DFAS used to measure its progress and analyzed summary metrics provided from fiscal years 2017 to 2018 to determine the effect of these efforts on the number of accounting adjustments recorded during these periods. We conducted this performance audit from November 2018 to January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Kimberley McGatlin (Assistant Director), Carl Barden, Rathi Bose, Veronica Cadiz-Rodriguez, Virginia Chanley, Benjamin Durfee, Patrick Frey, Maxine Hattery, Jason Kelly, Jason Kirwan, Zhen Li, John Lopez, Samuel Sawhook, Dacia Stewart, and Anne Thomas made key contributions to this report.", "summary": "DOD remains the only major federal agency that has been unable to obtain a financial statement audit opinion. One of the contributing factors is DOD's large volume of nonroutine accounting adjustments, which are used for recording corrections or adjustments in an accounting system. This report examines accounting adjustments and their effect on the reliability of DOD's financial information, the extent to which DOD has established and implemented policies and procedures for recording accounting adjustments, and the extent to which DOD has taken actions to reduce adjustments recorded at the consolidated level. For this report, GAO reviewed DOD and DFAS's policies and procedures, interviewed DOD officials about the adjustment process, and reviewed initiatives to reduce the number of adjustments being recorded. GAO also selected a random sample of 242 adjustments recorded at the DOD consolidated level for the fourth quarter of fiscal year 2018 to determine whether the adjustments were recorded in accordance with established policies. While the use of accounting adjustments is a common practice, the Department of Defense's (DOD) reliance on a large volume of nonroutine adjustments to prepare its financial statements is primarily a result of deficient business processes and limitations in accounting systems that DOD components use to process financial information. For example, the Defense Finance and Accounting Service (DFAS) continues to rely on forced-balance adjustments to replace the financial information that DOD's components submit to force agreement with Department of the Treasury balances without reconciling and researching the cause of differences (see figure). The recording of these adjustments was identified as a material weakness in DOD's internal control over financial reporting in its fiscal year 2018 financial statement audit. GAO found that DOD and DFAS policies and procedures for accounting adjustments are insufficient, outdated, and inconsistently implemented. For example, DOD's current policies do not define what constitutes adequate supporting documentation for system-generated adjustments, nor have DOD and DFAS established policies for identifying the cause of the adjustments, developing and implementing action plans to reduce the need for adjustments, and monitoring the effectiveness of those action plans. Because DOD and DFAS are not ensuring that their policies and procedures are up-to-date and consistently implemented, there is an increased risk that inaccurate, invalid, or unapproved adjustments will be recorded in DOD's core financial reporting system, resulting in a misstatement in DOD's consolidated financial statements. DOD and DFAS have undertaken initiatives to address some of the issues that contribute to the need for adjustments. Both organizations have developed strategies to decrease adjustments; however, neither has developed specific outcomes or detailed procedures for achieving stated goals in the strategies. Without clear procedures on how to implement its initiatives and a complete understanding across DOD of the issues contributing to the need for accounting adjustments, there is an increased risk that management efforts to reduce adjustments at the DOD consolidated level will be inefficient and ineffective. GAO is making eight recommendations to DOD, which include updating and implementing policies and procedures on recording accounting adjustments and identifying steps to reduce the need for recording adjustments across the department. DOD agreed with all eight recommendations and cited actions to address them.", "document_type": "gao"}
{"report": "Within the executive branch, both OMB and GSA provide leadership in managing federal real property. OMB, among other things, issues policies and memorandums, including the RTF policy discussed below. GSA has dual roles with regard to the management of federal real property. First, GSA’s Office of Government-wide Policy supports the implementation of OMB’s real property policies, including RTF, by collecting and analyzing federal real property data and providing agencies with guidance on leading practices. According to GSA officials, GSA and OMB coordinated to develop data analysis methods to monitor agencies’ performance in meeting OMB’s real property policies, and OMB approved the methods that are used. Second, as the federal government’s principal landlord, GSA’s PBS acquires, manages, and disposes of federally-owned real property for which it has custody and control on behalf of agencies that occupy it, and leases commercial space on behalf of agencies. In these cases, GSA manages the lease agreements. We refer to both of these types of properties as GSA-managed space. All of the agencies obtain at least some of their office space through GSA’s PBS; in fact, two-thirds of the 23 agencies’ office space is GSA-managed space. When agencies obtain space through GSA, they enter into occupancy agreements with PBS and pay rent, operations, and maintenance costs to PBS. When GSA obtains space for its own employees, it also enters into occupancy agreements with PBS. PBS maintains a record of agencies’ GSA-managed space, including information on square footage and costs, in its Occupancy Agreement database. We discuss PBS’s role in agencies’ office space decisions later in this report. Some agencies also have independent statutory authority to lease, or acquire and manage their own property, which GSA refers to as directly- leased or directly federally-owned (“directly-owned”) space. Additionally, some agencies may be authorized to directly lease or acquire property when GSA delegates authority to them because doing so is in the government’s best interest. Fourteen of the 23 agencies directly lease or own some of their office space, and about one-third of these agencies’ total office space is made up of directly-leased or owned property. Agencies pay rent to private landlords when directly leasing space and are responsible for operating and maintaining directly-owned property. Agencies must report, among other things, the square footage and costs to rent, operate, or maintain such properties to the FRPP database, which GSA maintains. As previously mentioned, in March 2015, OMB issued the RTF policy to promote the more efficient use of real property assets through improved space utilization and reduction. According to OMB, the policy is intended to provide value to the taxpayer. The RTF policy requires agencies to submit annual Real Property Efficiency Plans (Efficiency Plans) to OMB that: (1) identify annual reduction targets for domestic office and warehouse space for a 5-year period; (2) include a policy that specifies the maximum usable square feet per person, also known as a utilization rate, and (3) refrain from increasing the square footage of domestic office and warehouse space over fiscal year 2015 levels. As part of the Efficiency Plans, agencies must also identify specific projects they will implement to reduce or improve efficient use of their space. Agencies may undertake different types of projects such as renovation, relocation, or consolidation projects to achieve their space reduction or efficiency goals. While OMB oversees the implementation of the RTF policy, GSA tracks and reports key cost performance measures on agencies’ square footage and cost changes, in accordance with analysis methods it developed in coordination with OMB. GSA reports these performance measures on performance.gov and to Congress in annual reports, and provides measures for agencies to use in their Efficiency Plans. According to these data, the 23 civilian agencies reduced more than 6 million square feet of office space from fiscal year 2015 through fiscal year 2018. As shown in figure 1, space changes varied across agencies; 16 of the 23 agencies reduced office space, while 7 increased space. According to publicly available RTF data, some agencies’ space reductions have slowed as the RTF policy approaches its end date in fiscal year 2020 and as according to OMB officials, many of the lower- cost, high financial return projects have been executed. GSA has reported varied results with regard to changes in agencies’ costs since the start of RTF. GSA reported that the federal government has avoided spending millions of dollars as a result of reduced office and warehouse space but also has reported that the average cost per square foot for office space has increased. We discuss the RTF cost performance measures in detail later in this report. The RTF policy is effective through the end of fiscal year 2020. OMB and GSA officials told us that discussions about a real property policy to succeed RTF were underway as of early 2019. However, no policy to succeed RTF has been issued as of September 2019, according to a senior GSA official. GSA tracks and reports two RTF cost performance measures—estimated cost avoidance and average cost per square foot. These measures provide useful information on agencies’ results, but the average cost per square foot performance measure does not use the most accurate information. Regarding estimated cost avoidance, GSA reported that the 24 CFO Act agencies—including DOD—avoided an estimated $166 million in office and warehouse costs as a result of their space reductions since fiscal year 2015. We used GSA’s data and the cost avoidance approach GSA developed with OMB to identify that $114 million of the estimated cost avoidance can be attributed to civilian agencies’ office space reductions since fiscal year 2015. The estimated cost avoidance measure reflects overall federal cost avoidance because it accounts for space that agencies have returned to GSA but that remains unoccupied. Under certain conditions, agencies may vacate GSA-managed space prior to the end of their occupancy agreement and report that as a reduction in their space. However, until this space is reoccupied or GSA disposes of it, the federal government continues to incur costs to operate and maintain the space. Because of these continued costs, GSA accounts for vacant space when it estimates cost avoidance. For example, from fiscal year 2016 to fiscal year 2017 the amount of vacant GSA-managed office space increased more than the amount of space agencies reduced. Since this increase meant that the federal government had not reduced office space overall when the calculation was made, GSA estimated that rather than avoiding costs, costs for civilian office space increased by roughly three-quarters of a million dollars during this period. GSA officials noted that this estimate represents estimated cost avoidance at a single point in time and does not capture fluctuations in agencies’ space or vacant federal space throughout the year. Average fiscal year 2015 cost per square foot for (1) space agencies lease directly, (2) space agencies acquire and manage directly, (3) space GSA leases on behalf of agencies, and (4) federally-owned space GSA manages. In 2018, GAO reported that GSA’s and OMB’s method for estimating the cost avoidance associated with agencies’ real property changes is a reasonable approach given current limitations. OMB officials explained that the estimated cost avoidance is not intended to depict actual cost savings or the net effect of space changes on costs (i.e., investment cost minus savings) because the estimate does not include agencies’ investment costs to renovate, relocate, or dispose of space. Rather, GSA’s and OMB’s method estimates the costs for rent, operations, and maintenance that the federal government did not incur because it no longer occupies space. Further, OMB officials pointed out that because agencies use a variety of methods and systems to track and categorize their renovation, relocation, and disposal costs, agencies’ data on actual investment costs are not consistent across agencies and using these data would limit the accuracy of any estimate purporting to be an actual cost savings measure. Another annual cost measure GSA uses to track agencies’ RTF performance is the average cost per square foot, which is intended to reflect actual changes in agencies’ real property costs. GSA calculates the annual average cost per square foot for different categories based on how the space is managed—directly-owned, directly-leased, and GSA- managed office space. GSA uses the same approach to calculate the measure for all agencies, and for each agency to use in their annual Efficiency Plans. GSA’s performance measure shows an increase in all types of office space costs since fiscal year 2015, and our analysis of FRPP and Occupancy Agreement data similarly found that overall office space costs have increased for the majority of agencies, some by as much as 10 to 15 percent. We found the approach GSA developed with OMB to calculate average cost per square foot for directly-owned and directly-leased office space to be reasonable because GSA used the best available data. However, we found that GSA’s and OMB’s approach for GSA-managed space understated the average cost per square foot. Specifically, we found that GSA understated the overall average cost per square foot for all agencies’ GSA-managed office space by $1.31 (4.7 percent) on average from fiscal years 2015 through 2018. Furthermore, we found that GSA’s and OMB’s method understated the average cost per square foot for 18 of the 23 agencies between 3 percent and 41 percent on average from fiscal years 2015 through 2018. Figure 2 illustrates the range of differences we found between GSA’s and OMB’s method and actual costs. GSA understated the average cost per square foot for GSA-managed space because it did not use readily available data on the actual costs agencies paid to GSA for office space each year. Instead, GSA used the “rental rate”, which reflects the cost per square foot that agencies paid in the month GSA accessed the data—usually September. This rate does not include all agency costs, such as costs for GSA’s fee. Using the monthly rental rate to calculate average cost per square foot can significantly affect the resulting measure because the rental rate can differ from month to month. According to GSA officials, this variation can occur for many reasons including rental incentives, credits, or one-time costs that are reflected in that particular month but do not apply in all months. To identify how GSA’s use of the rental rate affected the cost information GSA used to calculate the measure, we calculated costs using the rental rate (GSA’s method) and compared them to the actual annual costs in GSA’s data. We found that costs calculated using the rental rate were almost always lower than actual annual costs for agencies, sometimes by millions of dollars for a single space. This approach led GSA to exclude an average of $271 million in office space costs per year from its calculations during this time period. Moreover, by using this approach GSA did not include the costs for spaces that did not have a rental rate, even when agencies paid for those spaces during the fiscal year. In fiscal year 2018, GSA’s and OMB’s method excluded 405 GSA-managed office spaces that did not have a rental rate but that had a combined annual rental cost of $24.2 million. Example of the Effect of General Services Administration’s (GSA) and Office of Management and Budget’s (OMB) Method on Cost per Square Foot: National Aeronautics and Space Administration (NASA) GSA used the rental rate even though it tracks and can easily access actual annual costs in its Occupancy Agreement database because, in GSA’s view, the rental rate better reflects the real average cost of an office space. Officials said that the actual annual cost can represent partial year costs and that GSA did not want to skew the averages toward zero- or low-cost spaces. However, as demonstrated by our analysis, GSA’s use of the rental rate, rather than preventing GSA from skewing the average costs toward lower cost office spaces, actually resulted in an understatement of these costs. GSA and OMB’s method excluded about $31 million from the cost per square foot calculation. This difference was largely attributable to one office space, which had a rental rate of $44.67, much lower than the actual annual cost per square foot for that space, which was $97.98. Standards for Internal Control in the Federal Government state that agencies should use and externally communicate quality information— information that is accurate and complete—to achieve their goals. Understating the average cost per square foot for GSA-managed office space, which comprises two-thirds of agencies’ office space, has implications for federal efforts to efficiently manage space. First, using an inaccurate cost performance measure affects stakeholders’ and policymakers’ ability to accurately judge and oversee agencies’ progress toward reducing space costs. Second, because agencies use these data to judge their own performance and make decisions about how to efficiently manage their space, agencies are at risk of taking ineffective steps to manage their costs and achieve their goals. As the government’s principal landlord, GSA’s PBS emphasizes cost savings from a government-wide perspective when working with agencies. To facilitate this approach, PBS has established policies and tools that focus on early planning and cost analysis. For example, according to PBS officials, PBS generally begins planning and cost analysis 5 years ahead of expiring occupancy agreements and leases. As part of this planning, PBS analyzes project costs and cost savings, and considers opportunities to fill vacant federal space and improve a space’s efficiency by, for example, improving the utilization rate. PBS recommends projects—including consolidation, relocation, and renovation projects—to agencies based on its analysis. Though PBS officials said that PBS has the final decision-making authority regarding agencies’ space, they said PBS works closely with agencies to make collaborative decisions about office space changes. Officials also told us that early planning helps ensure that PBS and agencies have time to identify and select the most cost-effective project option. According to PBS officials, PBS developed a tool in 2018 to compare potential space projects based on, among other factors, market and move costs. Officials told us that this tool is a way to ensure that PBS analyzes all projects consistently to identify the most cost-effective option for the federal government. They also told us that they use this tool iteratively throughout the planning process and that the cost analysis becomes more refined as PBS coordinates with agencies and identifies specific spaces as options. For high-cost projects, PBS also performs cost analysis of alternative options, including comparing each alternative’s net present value. However, PBS officials said that there are instances when they do not perform GSA’s standard cost analysis because it is not necessary. Specifically, PBS does not conduct this analysis when there is a space option that clearly aligns with its priorities. For example, PBS did not conduct its standard cost analysis for two of our 13 selected projects because both agencies moved into vacant federally-owned or leased space, moves that presented clear benefits to the federal government, according to PBS officials. However, GSA’s government-wide emphasis may not always result in cost savings for individual agencies, and in some cases, what is most cost-effective for the federal government does not always align with what is most cost-effective for individual agencies. For example, when Education relocated its San Francisco office to vacant federal space in fiscal year 2016, GSA’s analysis of the relocation showed that it cost Education slightly more than one other option, but was the lowest cost option for the federal government because it allowed GSA to fill space the federal government was already paying for. PBS officials told us that they expect their analysis to heavily influence agencies’ office space decisions and do not expect agencies to perform their own cost analysis for these decisions, but said some agencies do conduct such analysis. We found that all five of our selected agencies— Education, GSA in the space that it occupies, IRS, Labor, and NIH— conducted some type of cost analysis to inform office space changes. We found that some agencies include such analysis as part of their routine policies and procedures, while others conducted analysis for specific projects as needed. Percentage Change in Selected Agencies’ Square Footage and Cost, Fiscal Year 2015 through 2018 Cost plays an important role in agencies’ office space decision-making processes, which can influence office space changes over time. From fiscal year 2015 through 2018, our selected agencies’ office space costs and square footage changes varied, and square footage and cost changes did not always have the same trend. Education: A senior Education official said that the Department carries out various cost analyses when making office space decisions. For example, the official told us that Education conducts various cost analyses to identify the most cost-effective options for the Department. To manage agency office space costs, the Education official told us that Education focuses its planning process on expiring leases, high-cost leases, and low-cost projects with a large and rapid return on investment. We found that Education conducted such analysis when carrying out its Washington, D.C., consolidation project in fiscal year 2016. The official told us that Education had to quickly reduce agency costs and decided to do so by reducing space as opposed to furloughing employees. After reviewing their space and conducting rent savings analysis, Education decided to consolidate its staff from three different office buildings in Washington, D.C., into excess space it had in two other buildings. According to Education’s analysis, this consolidation reduced the annual rent for its Washington, D.C., offices by about 19 percent. GSA: GSA considers cost when it identifies and evaluates potential projects for the space it occupies. Specifically, GSA requires its offices to use a project template to routinely collect rent savings and payback period information on almost all potential projects. In fiscal year 2015, GSA also conducted a portfolio-wide review of GSA-occupied space during which it identified potential projects based, in part, on rent savings and payback period analysis. Through this review, GSA identified and recommended 15 projects that would reduce 964,000 rentable square feet, use space more efficiently, and save up to more than $17 million in rent over 5 years. GSA officials told us GSA prioritized its implementation of the recommendations by starting with the projects that had the largest space reduction and rent savings. For example, in fiscal year 2015, GSA decided to consolidate two of its Atlanta offices into one smaller, more efficient space. GSA determined that this consolidation could reduce its rentable square feet by 150,000 square feet (52 percent) and save $4.1 million in annual rent. IRS: IRS has developed multiple tools to analyze the cost of project options based on market data and upfront costs, among other information. For example, IRS developed its Return on Investment Calculator to help determine whether it is most cost-effective to stay in place, downsize, or relocate when a lease or occupancy agreement expires. The tool compares the return on investment for moving versus staying by using cost information, such as market data, travel, furniture, and rent costs. The IRS also uses a project estimating and tracking tool called the Space, Time & Resources Tool to create general cost estimates for a variety of project types, evaluate alternatives, and according to IRS officials, contribute to the development of plans for expiring leases. This tool uses preliminary space and cost estimates for needs such as facilities, security, and information technology to determine each project’s return on investment and potential annual rent savings. IRS officials told us that they may use the analyses from some of the tools to make a case to GSA in support of IRS’s preferred alternative or lowest-cost option, if necessary. Labor: Labor considers cost in its space policies and procedures, and we found that Labor sometimes conducted its own cost analysis to identify opportunities to achieve savings. A senior Labor official told us that Labor conducts an informal, broad review of its space that allows the Department to identify opportunities for cost savings. The official told us Labor looks for opportunities to co-locate staff from multiple agencies, and according to its space management policy, co-location allows Labor agencies to share support spaces which can reduce overall square footage and administrative costs. The official said that through review and analysis, Labor identified an opportunity to consolidate staff from multiple offices in Washington, D.C., into a single space in fiscal year 2016. Labor’s analysis indicated that the consolidation could save the Department an estimated $789,000 in annual rent in fiscal year 2014, the year the project began. The Labor official told us that Labor is focused on early planning to identify opportunities for cost savings and space reductions, and is beginning a new initiative to review and plan for projects up to 6 years in advance of lease or occupancy agreement expirations. NIH: We found that NIH routinely considers costs when it evaluates potential projects. Specifically, when NIH considers potential projects, it collects information on costs, such as long-term budget effects. A senior NIH official also told us that NIH works closely with GSA to conduct cost analysis, including analysis for high-cost projects that NIH submits as part of its funding requests to Congress, such as analysis of rent costs over the full lease term. Through this analysis, NIH has been able to identify lower-cost space options to meet its needs. For example, NIH determined that it could save $3.6 million annually and $53 million over 15 years by locating to office space that was closer to its other offices because it would decrease the time employees spent traveling between spaces. Additionally, the NIH official said one of NIH’s goals includes co-locating agency offices and staff to improve efficiency and reduce costs, and NIH routinely identifies opportunities to co-locate as part of its project selection process. For example, when NIH consolidated staff in Maryland into two buildings on one campus, the official said that NIH chose consolidation because it offered NIH an opportunity to operate more efficiently. We found that while all selected agencies consider cost when making office space decisions, they generally do not make decisions based on cost alone. We have previously reported that cost, mission, and external considerations influence agencies’ efforts to manage, reduce, or change their space. We found that all five of our selected agencies balance these factors, as well as workforce considerations, with cost, and with each other, when making office space decisions. These factors may not always align with each other and the extent to which these factors influence space decisions and their cost implications can vary for each specific office project need. Mission and Goals: We found that all selected agencies considered and balanced their mission or goals with other factors, such as cost, when making office space decisions. Mission: Agencies’ missions are an important factor and can work in tandem, or be in tension, with agencies’ efforts to achieve cost savings. For example, we found that when GSA decided in fiscal year 2014 to renovate and reduce space in its Chicago, IL, office, it considered, among other factors, how this project supported GSA’s government-wide mission to make federal space available to agencies. By reducing space in the existing location by fiscal year 2017, GSA determined it could reduce its annual rent in Chicago by 40 percent and provide more than 50,000 square feet of federal space to other agencies. On the other hand, a senior IRS official told us that because enforcement needs—a central part of IRS’s mission—are constantly shifting to different parts of the country, IRS may not always be able to enter into long-term lease agreements, which are generally more cost-effective. Goals: We found that agencies’ goals could be complementary to or in conflict with their efforts to reduce cost. A senior NIH official told us that NIH’s offices are currently widely dispersed and that NIH has a goal of “making the crumbs into a loaf” by co-locating different offices as leases expire. The NIH official told us that co-locating can facilitate cost savings because it allows NIH to operate more efficiently by, for instance, reducing shuttle services and sharing common areas and services. For example, one NIH project in Bethesda, Maryland will consolidate 11 expiring leases in five locations into three leases in a two-building campus that, according to GSA analysis, will reduce rent by 42.5 percent per year for 15 years. Conversely, some of our selected agencies noted that agency goals do not always align with cost savings. For example, both Labor and IRS officials told us that they may not pursue their space utilization goals if it costs too much to renovate space to meet their desired space per person. Additionally, the senior NIH official told us that NIH has previously moved to office space that did not meet its utilization rate goals because it was able to achieve larger cost savings by moving to a space in an area with lower rent than the area it previously considered. Workforce Impact: We found that all five of the selected agencies considered how office space decisions could impact their workforce, and a couple of agencies told us that they balance this consideration with costs, along with agency mission and goals. Commuting time: Officials from three selected agencies noted that changes in employees’ commuting time can influence what office space to select. For example, Education is scheduled to relocate its Dallas regional office in fiscal year 2020. A senior Education official told us that Education chose a space that has close proximity to the current space, in part, because the relocation will have minimal impact on employees’ commute. The official also said that even if federally- owned office space further away became available, Education may not move there if it would be difficult for staff to get to. Similarly, IRS’s business case to consolidate several offices in the Cincinnati, OH, area into one office starting in fiscal year 2015, analyzed how the project would affect IRS employees, including the impact on employees’ commute, ability to park, and the effect on employees’ income taxes. Employee Morale and Productivity: Several selected agencies noted that reducing the amount of space per person can affect employee morale and productivity. According to GSA’s strategic goals, improving space utilization by, for example, reducing the amount of space per person can help the federal government achieve cost savings. A senior Education official told us that when redesigning Education’s Washington, D.C., offices, which reduced the amount of space per person, leadership engaged in a substantial employee outreach effort to understand how these changes affected employees and to build employee support for the changes. The official also said Education took into account upfront costs for tools to improve employees’ experience. For example, the official said that the Department invested in noise cancelling headphones to improve the employee experience, which was a small cost compared to the cost for office space. To ensure that reductions are not having a negative impact on its employees, GSA developed a survey that it sometimes distributes both before and after making space changes. External Factors: Officials from four of the five selected agencies said external factors, such as federal priorities, statutes, regulations, and policies can influence their office space decisions. In some cases, these factors did not complement efforts to reduce costs. Federal Priorities: Federal goals and priorities can influence agencies’ space decisions, and these requirements may not align with efforts to reduce costs. For example, in fiscal year 2016, GSA relocated its regional office in New York City from federally-owned to federally- leased space in the World Trade Center. Though GSA considered cost, the federal government’s commitment to move into the World Trade Center after the terrorist attacks on September 11, 2001, influenced this decision, which resulted in increased costs for GSA. Statutes, Executive Orders, and Regulations: Some agencies told us that statutory requirements, directives, and regulations can influence their space decisions, and may or may not align with efforts to reduce costs. For example, a senior IRS official told us that a 1978 Executive Order, which requires that agencies with a mission need to locate in an urban area first consider moving to a central business district, might result in IRS moving to higher-cost neighborhoods. A senior official from the Department of Health and Human Services also told us that locating office space in the central business district of urban areas can be more expensive, but that the Department often does so because of the Executive Order. GSA policies: GSA policies on space management can also affect agencies’ office space decisions. A senior Labor official told us that the Department is currently reducing space in its Chicago, IL, regional office but the ability to do so is dependent on whether Labor can return the space to GSA. GSA policy states that agencies occupying space acquired from PBS can return space within a certain time frame if, among other requirements, the space is categorized as cancelable and is in marketable blocks based on the location, usage, and size of the space. If the space does not meet these criteria, an agency can return the space to PBS but is still responsible for paying rent and other costs associated with the space until the occupancy agreement or lease expires. Even as agencies have intensified their focus on better space management in an effort to save taxpayer dollars, overall, the cost for office space continues to rise. Using the best data available to assess space options and trade-offs is critical. GSA’s and OMB’s cost per square foot performance measure could provide agencies a good way to assess their costs and track cost trends, particularly as agencies’ efforts continue to evolve beyond reducing their footprints toward optimizing their space. However, the measure is only as good as the approach and data used in the calculation. Because GSA’s and OMB’s cost per square foot performance measure is not using actual cost information for GSA- managed space, GSA and OMB are understating the average cost per square foot for a significant portion of square footage. This inaccurate information could adversely affect agencies’ and stakeholders’ understanding of RTF results. As the RTF policy ends in fiscal year 2020 and agencies look toward the next initiative, having the most transparent and accurate information on the results of agencies’ efforts to date can inform new strategies and tools to help agencies continue and expand upon their efforts to manage their property more efficiently and ultimately save money. Moreover, having accurate information on agencies’ real property costs will continue to be important in future initiatives to efficiently manage federal real property. The Administrator of the General Services Administration (GSA), in coordination with the Director of the Office of Management and Budget, should ensure that the average cost per square foot performance measure for GSA-managed space is calculated using actual cost information. (Recommendation 1) We provided a draft of this report to GSA, OMB, and the Secretaries of the Departments of Labor, Education, the Treasury, and Health and Human Services for review and comment. In GSA’s written comments, which are reproduced in appendix II, GSA agreed with our recommendation. OMB did not provide comments, but GSA stated in its comments that it is working with OMB to develop a plan to address our recommendation. The Departments of Labor, Education, the Treasury, and Health and Human Services told us that they had no comments on the draft report. We are sending copies of this report to the appropriate congressional committee; the Administrator of GSA; the Director of the OMB, and; the Secretaries of the Departments of Education, Health and Human Services, Labor, and the Treasury. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning the report, please contact me at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in Appendix III. This report discusses: (1) the extent to which Reduce the Footprint performance measures reflect changes in civilian Chief Financial Officers Act agencies’ (CFO Act agencies) office space costs, and (2) how selected agencies considered costs in office space decisions. To obtain background information on both of our objectives, we reviewed literature including the Office of Management and Budget’s (OMB) and General Services Administration’s (GSA) memos and guidance governing the Reduce the Footprint (RTF) policy, the Real Property Efficiency Plans (Efficiency Plans) agencies submit to OMB and GSA annually as part of RTF, and relevant regulations and statutes. We also examined information GSA uses to track RTF progress, including public data on agencies’ square footage changes. We assessed the reliability of these data by conducting electronic testing, reviewing prior GAO assessments of reliability, and interviewing agency officials. Based on this assessment, we determined these data to be reliable for the purposes of describing changes in agencies’ square footage. Additionally, we reviewed previous GAO and GSA Inspector General reports describing the federal government’s efforts to use its property more efficiently and reduce costs. To address our first objective, we analyzed federal data on office space square footage and costs, and reviewed the two RTF cost measures GSA developed with OMB to track and report agency performance: (1) estimated cost avoidance and (2) changes in average cost per square foot. To identify changes in agencies’ office space costs, we analyzed square footage, and rent, operations, and maintenance costs from Federal Real Property Profile (FRPP) data submitted by agencies and GSA’s Occupancy Agreement data. Office space costs in both datasets may contain costs for additional items beyond rent, operations and maintenance, such as tenant improvements, but we determined that the inclusion of these costs did not preclude us from using these data to describe agencies’ costs as the data reflect the total annual costs to agencies. Though agencies may report different types of square footage in FRPP, as specified by GSA’s FRPP reporting guidance, we analyzed rentable square footage where available because it represents the total space an agency pays for. We limited our analysis to the CFO Act agencies because these agencies are subject to RTF requirements, but we excluded the Department of Defense (DOD) from our analysis because of GSA concerns about the reliability of DOD’s data. We analyzed data from fiscal year 2015, the year RTF began, through fiscal year 2018, the most recent year for which data were available. To assess the reliability of these data, we conducted electronic testing, reviewed GSA documentation and prior GAO data reliability assessments, and interviewed GSA officials. Based on our assessment, we determined that both the FRPP and Occupancy Agreement data were reliable for the purposes of describing changes in agencies’ office space costs and square footage. To analyze the extent to which the cost performance measures reflected agencies’ cost changes, we reviewed the methodologies GSA developed with OMB for the cost performance measures and GSA’s calculations, interviewed OMB and GSA officials regarding the measures, and replicated one of the methods. We also reviewed previous GAO assessments of the estimated cost avoidance methodology. To determine how GSA’s and OMB’s approach to calculating the average cost per square foot affected the results for GSA-managed space, we used GSA Occupancy Agreement data to compare the average cost per square foot based on GSA’s and OMB’s method to the average cost per square foot using actual costs. We compared our analysis of the average cost per square foot method to Standards for Internal Control in the Federal Government, which state that agencies should use and communicate quality information—information that is complete and accurate—to inform decisions. To address our second objective, we selected five agencies—the Department of Education (Education), GSA, the Department of the Treasury, the Department of Labor (Labor), and the Department of Health and Human Services—to review in depth. Within the Departments of the Treasury and Health and Human Services, we further selected the Internal Revenue Service (IRS) and National Institutes of Health (NIH) respectively because we determined that real property within these Departments is managed at the agency level. Using FRPP and Occupancy Agreement data on agencies’ costs and square footage, we selected agencies based on factors such as office space portfolio size, whether the agencies obtain office space themselves or through GSA, and changes in portfolio cost and square footage. We selected agencies for variety but weighted our selection toward agencies with larger absolute changes in cost and square footage. Our selection is not representative, and these agencies’ experience is not generalizable to all agencies. To gain insights into how these agencies consider costs when making office space decisions, we reviewed selected agencies’ real property management policies, and interviewed agency officials. We then analyzed this information to identify common themes across selected agencies. To further understand how agencies implemented their policies and the factors agencies considered when making specific office space decisions, we also selected 13 office space projects these agencies undertook from fiscal year 2015 through fiscal year 2018. We identified potential projects based on selected agencies’ annual Efficiency Plans, agency project data, and interviews with agency officials. We selected specific projects based on factors such as cost, location, changes in square footage, and project type. We chose projects with a range of types and locations to better understand agencies’ decision-making process for different kinds of projects. However, we selected only projects with a cost of $1 million or more and with larger changes in square footage because these projects have more effect on overall federal and agency costs and portfolios. Because our intent was to understand the factors selected agencies considered when deciding on projects, our selection includes both completed projects and projects that were ongoing as of spring 2019, when we collected our data. The projects we selected are not representative of all projects or agencies, and are not generalizable. We analyzed project documentation and interviewed agency officials about each project. We also reviewed federal data for some projects to identify the changes in agencies’ square footage and costs before and after projects. To further address our second objective, we reviewed GSA Public Buildings Service (PBS) policies and guidance, and interviewed PBS headquarters officials to understand PBS’s role in agencies’ office space decisions, including how PBS considers costs when helping agencies obtain space. We also reviewed PBS cost analyses, such as net present value alternatives analysis and move-stay analysis, for most of our selected projects. We conducted this performance audit from November 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Maria Edelstein (Assistant Director), Katherine Raymond (Analyst-In-Charge), Eli Albagli, Ricki Gaber, and Minette Richardson made significant contributions to the report. Also contributing to this report were Melissa Bodeau, Josh Ormond, Kelly Rubin, Terence Lam, and Crystal Wesco.", "summary": "The government's RTF policy has intensified federal efforts to reduce office space and save money since 2015. GSA and OMB report key cost performance measures but questions exist about how well these measures reflect agencies' efforts. GAO was asked to review how federal real property costs have changed since 2015. This report examines (1) the extent to which performance measures reflect changes in civilian CFO Act agencies' office space costs and (2) how selected agencies considered cost in their office space decisions. To conduct this work, GAO analyzed federal data on office space square footage and cost changes for the 23 civilian CFO Act agencies from fiscal years 2015 through 2018, and reviewed GSA's and OMB's calculations for cost performance measures. GAO selected five agencies and 13 of their office space projects as non-generalizable case studies based on several factors, including those with larger space and cost changes. GAO reviewed the selected agencies' policies and project documentation, and interviewed agency officials. The Office of Management and Budget (OMB) issued the Reduce the Footprint (RTF) policy in 2015 to promote the more efficent use of federal space. The General Services Administration (GSA) and OMB track and report two RTF cost performance measures: estimated costs avoided and average cost per square foot. GAO found that the method for estimating costs avoided was reasonable. However, the average cost per square foot was not accurate for the federally-owned and leased office space GSA manages for agencies. Specifically, GAO found that from fiscal years 2015 through 2018 the actual average cost per square foot for this space was, on average, $1.31 per square foot higher than the costs GSA and OMB reported for the 23 civilian agencies subject to the Chief Financial Officers (CFO) Act. The actual cost per square foot was higher for 18 out of 23 of these agencies (see figure). Because GSA and OMB did not use readily available actual cost data, their method, which is based on 1 month's data, excluded an average of $271 million per year in costs over this period. Consequently, stakeholders and agencies do not have accurate information to assess agencies' performance or help manage their space decisions. Note: This information covers the 23 Civilian Chief Financial Officers Act agencies. While selected agencies considered costs when making office space decisions, they balanced other factors as well. As the federal government's principal landlord, GSA obtains space for many agencies. In so doing, it emphasizes federal cost savings, which may not lead to agency savings. For example, GSA prioritized filling unoccupied federally-managed space even if it was more costly to an agency than another option. The selected agencies also reported that factors such as mission, workforce needs, and external factors are important to consider and balance as well. For example, a senior official from the Department of Education said that effects on employees' commutes are an important factor in its space decisions, and that it weighs the impact of potential office locations on the Department's workforce against the cost of the space. GAO recommends that GSA coordinate with OMB to use actual cost information to calculate the average cost per square foot performance measure for GSA-managed space. GSA agreed with the recommendation.", "document_type": "gao"}
{"report": "Federal law requires executive agencies to: maintain adequate inventory controls and accountability systems for property under their control; continuously survey property under their control to identify excess; and promptly report excess property to GSA and generally dispose of it in accordance with GSA regulations so that it can be made available to other federal agencies and stakeholders for reuse. In addition, the FPPMA requires agencies to assess accountable property within their control in accordance with guidance from GSA. Property can be accountable or non-accountable. Accountable property is property with a useful life of at least 2 years that an agency determines should be tracked in its property records. Each agency determines what constitutes accountable property for that agency, and for our selected agencies, consideration is given to an item’s acquisition cost and other factors, such as ease of theft and sensitivity. For our three selected agencies, these acquisition cost thresholds ranged from $5,000 to $10,000 in fiscal year 2018. Non-accountable property is property that does not meet the agency’s definition of accountable property and may include items such as office furniture and printers. Agencies typically do not track non-accountable property unless they need to for specific purposes, such as managing inventory levels. Some agency property is located in warehouses. For the purpose of this report, we used the definition of “warehouse” in FRPP guidance: “buildings used for storage, such as ammunition storage, covered sheds, and buildings primarily used for storage of vehicles or materials.” This term encompasses a broad array of property that agencies may classify differently for internal purposes but classify as warehouses for FRPP reporting in the absence of more precise categories. For example, some buildings that DOE reports as warehouses in FRPP are specialized facilities for storing nuclear and nuclear-contaminated material. In the absence of an FRPP category for nuclear storage, these buildings are classified in FRPP as warehouses. In a similar manner, FAA classified as warehouses in FRPP buildings used to house air traffic support systems, such as approach lighting systems, because no other category in FRPP was a better fit. GSA’s role in agencies’ management of property they have acquired, whether in warehouses or elsewhere, is limited until an agency declares that property as excess. Once property is declared excess, it can be transferred to another agency or certain non-federal recipients, donated, sold, abandoned, or destroyed. GSA has issued regulations that govern agencies’ actions in the property disposal process, and it administers a web-based system that facilitates property disposal. However, prior to an agencies’ identifying property as excess, GSA’s authority to issue regulations or formal guidance regarding agencies’ management of property is limited to topics that have been specifically assigned to the GSA Administrator, according to GSA officials. GSA distinguishes between formal guidance and informal guidance, and GSA officials explained this distinction as follows. Formal guidance, such as a Federal Management Regulation bulletin, must be reviewed by GSA management and general counsel officials. For example, in 2017 GSA issued a Federal Management Regulation bulletin on warehousing that summarized industry perspectives from two voluntary consensus standards, which were published by ASTM International’s asset management committee and which GSA had participated in developing. One standard addressed storage of property and the other addressed strategic warehousing. GSA officials stated they were authorized to include content on property management in the bulletin under the authority of GSA’s real-property policy program because the content supported the real property goal of reducing the federal government’s real property footprint. In contrast, informal guidance does not require review by GSA management or general counsel officials but may be published on GSA’s website or disseminated at GSA trainings. Whether formal or informal, agencies are not required to adhere to such guidance. Typically, within agencies, responsibility for managing property is generally shared between property officials and property custodians. Property officials’ primary responsibilities relate to property management. For example, they may be responsible for updating property data systems, providing property lists and instructions for property inventories, resolving issues that arise with property management, and managing the disposal process after a property custodian has determined that an item is no longer needed. Property custodians are generally program managers who are assigned responsibility for specific property items associated with the program they manage as an ancillary duty. For example, property custodians may be required to conduct physical inventories of property assigned to their program and work with a property official to resolve any issues arising during the inventory. Depending on how an agency uses its warehouses and property, property custodians may be responsible for property in a single warehouse, in multiple warehouses, or in a variety of locations. Moreover, multiple property custodians may be responsible for property in a single warehouse, as depicted in figure 1. The three selected agencies had a total of 1,221 warehouses, with over 6.4-million square feet, that contained a broad array of property. Although comprehensive data on property in these agencies’ warehouses were unavailable, interviews, site visits, and agencies’ data on warehouses themselves provided some information on the types of property in them. We found that the agencies had some commonalities in the contents of their warehouses. For example, all three had warehouses that contained material-handling equipment, such as fork lifts, as well as excess property being processed for disposal. However, much of the property in agencies’ warehouses was specific to their missions, according to agency officials and our observations. Table 1 includes information about the agencies’ warehouses and examples of the types of agency-specific property in them. We also visited warehouses at each of the selected agencies to obtain additional information about and view the types of property stored within them, as described below. FAA. FAA had warehouses at four main sites that contained property specific to the sites’ missions, and most of the remaining warehouses were buildings that contained equipment, tools, or materials to maintain aviation support systems or housed support systems, such as approach lighting systems, according to our analysis of FAA warehouse data, FAA officials, and sites we visited. For example, we visited FAA’s warehouses at the Mike Monroney Aeronautical Center, including the Logistics Support Facility, FAA’s largest warehouse and central location for maintaining and repairing aviation support systems deployed throughout the national airspace system. Most items in the warehouse were spare parts, materials, and systems or system components that had been sent to the facility for repair. We also visited FAA’s Staging Area, which supports FAA’s manufacture and assembly of new systems to be deployed throughout the country. Accordingly, much of the property at the two warehouses that comprise this facility was equipment, parts, and material, along with the machines and tools to manufacture and assemble the material. For example, we viewed components of a wind shear alert system that were being prepared to be shipped. FAA’s Mobile Asset Deployment Center stored and maintained FAA’s mobile assets, such as air traffic control towers and housing units that FAA deploys to maintain service during disruptions such as natural disasters. Finally, we visited a 96-square-foot shack—identified in FRPP as a warehouse and pictured below—that housed an approach lighting system. (See fig. 2 for examples of FAA warehouses and property.) Office of Science. Most Office of Science warehouses were located at Office of Science national laboratories. Warehouses at the two national laboratories we visited—Argonne National Laboratory and Fermi National Accelerator Laboratory—contained a broad variety of equipment, including equipment being staged for near-term use and equipment in longer-term storage specifically designated for future projects. For example, one warehouse at Fermi National Accelerator Laboratory contained a cryogenic system acquired by CERN, the European Organization for Nuclear Research, as its contribution to a planned experiment. This cryogenic system will be used for cooling purposes. A warehouse at the same site also contained some decades-old items kept as replacements for items still in use. According to officials, many of these older items would be difficult to obtain in a reasonable time frame for a reasonable price if a replacement were needed. In addition, at Fermi National Accelerator Laboratory, we saw a large, out-of-use calorimeter— a device commonly used in physics experiments—that was being stored for eventual use in an educational display. Warehouses also contained parts, materials, and supplies for laboratory use. (See fig. 3 for examples of Office of Science warehouses and property.) BOP. Most of the BOP warehouses were located at correctional institutions throughout the country, served similar functions, and contained similar types of property for inmate use, according to BOP headquarters officials and our review of BOP real property data. The two correctional institutions we visited each had a warehouse that served as a distribution center, where items arriving at the institution were received, processed, and sent to the appropriate personnel within the institution, and a food service warehouse, where food items used to feed the inmate population were stored. At one institution, non-perishable items for inmate use, such as uniforms, mattresses, soap, and toilet paper were stored at the distribution center, while the other institution we visited stored less property at the distribution center and expedited delivery to the relevant division. Additionally, one institution used a warehouse to store dairy equipment in support of an inmate-run dairy. (See fig. 4 for examples of BOP warehouses and property.) All three selected agencies tracked certain direct costs for owned and leased warehouses, including operations and maintenance costs for owned warehouses and some leased warehouses, and the rental cost for leased warehouses (see table 2). Although the agencies had this cost information, they did not use it to systematically determine how much it costs to store their property in warehouses, whether at an aggregate or per-item level. Two features of how these agencies track property and warehouse costs would make it difficult to do so. First, as mentioned above, selected agencies did not have comprehensive information on items in warehouses, information that would be needed to determine per-item storage costs. Second, selected agencies generally incurred direct costs—rent, operations costs, and maintenance costs—at a warehouse level. However, because a warehouse may have had some of its square footage dedicated to other uses, such as office or laboratory space, it would be difficult to ascertain what percentage of costs would be allocated to storage versus these other uses. Moreover, in some cases, operations costs, such as utilities, were incurred at a multi-building level, making it difficult to determine what portion of the bill is attributable to a single warehouse. Finally, selected agencies generally did not track indirect costs, such as personnel costs for conducting regular inventories and other administrative costs associated with storing property in their warehouses, according to agency officials. While none of the selected agencies systematically tracked property storage costs, we did identify one Office of Science site, one Department of Transportation site, and one Department of Justice site that analyzed the use of specific portions of warehouses for cost allocation purposes. Officials at these agencies said that this approach may create incentives for property custodians to identify excess property in a timelier manner. Argonne National Laboratory, within the DOE’s Office of Science, annually analyzes direct costs for each building, including warehouses, and charges each division within the laboratory for the space it occupies. A report assessing contractor performance at DOE’s Fermi National Accelerator Laboratory noted that implementation of such a system could be an effective way to hold divisions accountable for the number of items they have in storage. The Department of Transportation and the Department of Justice each manage a warehouse near their respective headquarters that they use to store property for various divisions within each department. The departments charge users for the portions of the warehouses they occupy. While these approaches may create incentives to identify unneeded property in a timely manner, they may not be applicable for all circumstances. For example, staff at Fermi National Accelerator Laboratory stated that they explored the cost and benefits of analyzing space use to allocate costs by user but had not found it to be cost- effective or feasible. In addition, allocating costs based on warehouse usage would be challenging if users’ space usage changes regularly. Two of the three agencies we reviewed had policies in place explaining the frequency in which property custodians should assess property for ongoing need. Specifically, the Office of Science and BOP had policies that called for identifying unneeded property beyond the statutory requirement to continually survey property to identify excess. For example, DOE regulations, which cover the Office of Science, require managers to perform walkthroughs at least every 2 years to identify unneeded property. According to officials, these walkthroughs are conducted by contractors that manage national laboratories. Similarly, BOP policy requires that property custodians conduct an annual site inspection to identify unneeded property prior to the annual inventory, and, according to officials, this process is overseen by the institution’s associate warden. In contrast, FAA policy does not set any timeframe for property custodians to identify unneeded property. However, according to one FAA headquarters official, assessing property for ongoing need is inherent to the inventory process, which, according to FAA policy, should occur at least every 3 years for accountable property. In addition, only DOE had specific requirements to determine if property is needed. Specifically, DOE regulation requires written justification for retention of property classified as equipment held for future projects. If equipment is retained for longer than a year, the justification is to be reviewed by a higher level of authority, and retention of such equipment for longer than 3 years requires approval by the head of the DOE field organization. The Office of Science Organizational Property Management Officer—who is responsible for reviewing contractors that manage Office of Science sites—reviewed sites’ adherence to this requirement using metrics, such as acquisition date and time in storage, according to officials. Beyond this particular requirement for DOE, none of the agencies had a systematic way to identify property that may be unneeded. Instead, they primarily relied on professional judgment to determine the ongoing need for property in warehouses in the absence of guidance on how to determine whether property is still needed. For example, FAA officials confirmed that they do not have guidance or metrics on how to identify unneeded property and typically rely on property custodians’ professional judgment. According to officials at one FAA site we visited, property custodians do not use specific criteria for identifying unneeded property because it is obvious when items are no longer needed. Similarly, at the BOP institutions we visited, officials confirmed that they rely on property custodians’ professional judgment, along with the judgment of associate wardens, to identify unneeded property during the annual site inspections, but acknowledged that this has led to different outcomes. For example, at one site we visited, site officials stated that some associate wardens are more inclined than others to require property custodians to identify property as unneeded. While officials at all of the selected agencies said they believed property custodians were able to identify unneeded property in a timely manner using their professional judgement, we identified instances, through our interviews and agency assessments, where agencies had retained unneeded property in storage. While the agencies identified and in most cases addressed these instances, these situations demonstrate the challenges associated with agencies’ existing approaches. Specifically: A 2016 report from DOT’s inspector general found that FAA property custodians allowed obsolete computers to remain on the property records, including computer systems manufactured in 2006 or earlier that were likely no longer in use because of their 3- to 4-year lifecycles. In 2018, a review found that Fermi National Accelerator Laboratory’s contractor was storing IT equipment, which had not been classified as equipment held for future projects, dating back to 1998. The report recommended that the contractor review all IT equipment for continued need and that certain items be removed from the active inventory in their asset management system. BOP headquarters officials told us that, when assisting regional office personnel in training a new property official at an institution, they noticed the institution was storing inmate clothing that exceeded the institution’s needs. According to the officials, they worked with the new property official to transfer the clothing from the institution to another BOP institution that needed it. Selected agencies’ limited guidance on how to identify unneeded property and reliance on professional judgment were not unique to the agencies in our review. For example, in a previous review that examined five agencies—Environmental Protection Agency, Forest Service, GSA, Department of Housing and Urban Development, and Internal Revenue Service—we found that selected agencies did not have policies and processes for identifying unneeded property on a proactive basis and relied on “triggering events,” such as an office move to make excess property decisions. Moreover, the industry and standards-setting groups we interviewed for this review indicated that these approaches were common across the federal government. However, the industry stakeholders and federal agencies that participated in ICPM that we interviewed identified more systematic ways to identify unneeded property. For example: Periodic justification for continued storage. One agency implemented a policy in 2013 requiring written justification to retain certain accountable property for certain time periods, with the time period varying for different types of property. After the initial storage time period, written justification for continued storage must be reviewed and approved by an official who is above the property custodian. According to property officials, this policy has contributed to an estimated 35 to 40 percent reduction of property held in storage. Data analytics. Officials from another agency stated that they use a logistics management application to track and analyze information, such as property age, amount, rate of usage, and warehouse space availability. As a result, the agency has identified and disposed of excess property at various warehouses that otherwise would likely have been retained. For example, according to officials, analysis conducted using this application on idle property in one warehouse informed the decision to identify as unneeded a significant amount of furniture. A previous manager had acquired the furniture for use in staff housing, but the items were not well-suited to available housing in the area. Utilization reviews. Industry groups we interviewed advocated for increased use of data to assess utilization to inform decisions on whether to retain stored property, such as utilization reviews that systematically assess property utilization and continued need. For example, when conducting a utilization review, one stakeholder recommended a process that begins with pinpointing where the inactive population of property items reside. Upon locating anything that has been inactive for a certain period of time, within a certain storage area, those items are identified as candidates for disposal. After the results come in, the property custodians can recommend that a certain amount of items on the overall list are marked for disposal. While some stakeholders identified systematic ways to identify unneeded property in certain circumstances, limited government-wide guidance exists for agencies to use to determine whether property in warehouses is still needed and being used. Specifically, there are two sources for guidance related to assessing property in warehouses for ongoing need: ASTM’s standards for strategic warehousing and storage of property. The standard for strategic warehousing notes that entities often continue a warehouse activity largely because it is easier than going through the effort of dismantling it. It urges that entities consider whether warehousing is needed. Furthermore, the standard asserts that a sound business case should be in place to support storage of property, including a decision of whether the items need to be warehoused. The standard for storage of property notes that entities should deploy an inventory management system to track incoming and outgoing assets; such a system can help in developing performance metrics for stored items. GSA’s federal warehousing bulletin. This bulletin references the two ASTM standards identified above and discusses the importance of critically assessing the need for items in storage, but provides limited information on how to make such assessments. According to GSA officials, the use of voluntary consensus standards, such as ASTM standards, can assist agencies with property management. However, only one agency official we interviewed stated that voluntary consensus standards informed the agency’s policy; the others we interviewed were either unaware of the standards or said the standards were not relevant to agency policy or practice. The FPPMA requires agencies, in accordance with GSA guidance, to inventory and assess property. As part of such assessments, it calls for evaluations of the age and condition of the property and the extent to which the agency uses it. According to officials at the selected agencies, they are waiting for guidance from GSA before taking steps to implement FPPMA. According to GSA officials, they are in the process of developing informal guidance on minimizing and identifying excess property to meet this requirement because FPPMA did not provide GSA additional authority to issue regulations or formal guidance. In particular, GSA developed draft guidance, which incorporated principles from a new ASTM standard on identifying and reducing excess property that GSA officials expect will be issued in early 2020, and provided it to ICPM participants for review and comment in September 2019. According to GSA officials, this informal guidance will be issued in December 2019. GSA officials plan to include the guidance on the GSA website and disseminate it to ICPM participants and may provide it in hard copy at relevant GSA events. The draft guidance we reviewed encourages agencies to designate an individual to manage an agency’s asset management program and use that system to capture and provide information on property age, condition, utilization, and mission dependency on a real-time basis, among other things. The draft guidance also included some criteria agencies could use to identify excess property. However, the guidance did not provide specific approaches or practices agencies could use to assess property utilization, including property stored in warehouses. The draft guidance and an accompanying strategy document indicate that GSA will collect best practices and incorporate them into the guidance, but GSA officials did not specify what types of best practices it plans to include or provide a timeline for doing so. Including additional information on approaches or practices agencies can use to assess property use and ongoing need—such as periodic property justifications, data analytics, and utilization reviews—could assist agencies in fulfilling their FPPMA requirements. Moreover, GSA officials did not provide a documented plan or time frame for communicating the guidance beyond publishing it on GSA’s website and disseminating it to ICPM members, an approach that can limit the reach and awareness of this information to agencies government-wide. As we have previously reported, work by others has shown that inaction on unneeded government property can limit its efficient use. As agencies continue efforts to manage their warehouse space in accordance with government-wide initiatives, improvements to how agencies assess property utilization and identify unneeded property in warehouses could enhance these efforts. The agencies in our review did not systematically assess their property for ongoing need and in some cases, retained unneeded property. More broadly, agencies across the government are operating without the benefit of government-wide guidance that could help assess their property for ongoing need in a systematic manner. With the recent enactment of FPPMA, an opportunity exists for GSA to develop and communicate guidance to help agencies assess property utilization and identify unneeded property in warehouses more efficiently that includes practices GSA identifies as being useful. Such guidance could help agencies avoid retaining property that is no longer needed and, as a result, allow them to better manage the use of their warehouse space. The Administrator of GSA should direct the Office of Government-wide Policy (1) to incorporate into its guidance approaches or practices that agencies could use to assess utilization of and the ongoing need for property—approaches such as recommendations for periodic justifications, data analytics, and utilization reviews—and (2) to develop a plan and timelines for communicating the guidance to agencies government-wide. (Recommendation 1) We provided a draft of this report to GSA, DOT, DOE, and DOJ for review and comment. GSA concurred with our recommendation and provided written comments, which are reprinted in appendix II and summarized below. DOT, DOE, and DOJ each stated in an email that they had no comments on the draft report. In its written comments, GSA agreed with our recommendation and stated that it is further developing its guidance as well as a plan and timeline for dissemination of that guidance to executive agencies. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the GSA Administrator, the Secretary of Energy, the Attorney General, and the Secretary of Transportation, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions regarding this report, please contact Lori Rectanus at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. This report addresses: (1) what is known about property in selected federal agencies’ warehouses and how much they spend to store this property and (2) the extent to which selected agencies assess the ongoing need for property stored in warehouses. To address both objectives, we selected three agencies for analysis—the Federal Aviation Administration (FAA) within the Department of Transportation; the Office of Science within the Department of Energy (DOE); and the Bureau of Prisons (BOP) within the Department of Justice. We limited our scope to civilian agencies because we have already done extensive work on property management within Department of Defense. At the department level, we used Reduce the Footprint data from fiscal year 2017 because they were the most current data available when we conducted the analysis to identify the top 10 departments in terms of warehouse square footage. To obtain variation among these agencies, we categorized these departments as large, medium, or small in terms of warehouse square footage and selected one from each category based on changes in square footage between fiscal years 2015 and 2017 using fiscal year 2017 Reduce the Footprint data and on the proportion of leased warehouse space to owned warehouse space using fiscal year 2017 Federal Real Property Profile (FRPP) data. Because none of the selected agencies manages property at the department level, we then selected a component within each department. For the Department of Transportation and the Department of Justice, we selected the components with the most warehouse square footage according to fiscal year 2017 FRPP data—FAA and BOP, respectively. For DOE, we used the agency’s fiscal year 2017 real property data to identify the components with the most warehouse square footage because DOE reports most information to FRPP at the department level rather than for specific offices, such as the Office of Science. We then selected the Office of Science, which had third highest amount of warehouse square footage, because of security concerns with one of the components with more warehouse square footage and because the other component with more warehouse square footage used a greater proportion of warehouse space to store nuclear and nuclear-related material. To determine what is known about property in selected agencies’ warehouses, we interviewed headquarters-level officials regarding the agencies’ property data, conducted site visits to view and photograph property stored in warehouses, and gathered information in interviews with agency officials. In selecting sites, we selected at least one site per agency that was among the largest in terms of warehouse square feet for that agency and at least one other site that was near one of the large sites, as described below: FAA: Mike Monroney Aeronautical Center in Oklahoma City, Oklahoma; Staging Area and Mobile Asset Deployment Center in Independence, Missouri; and Charles B. Wheeler Downtown airport in Kansas City, Missouri. Office of Science: Argonne National Laboratory and Fermi National Accelerator Laboratory in the Chicago area. BOP: U.S. Penitentiary Leavenworth in Leavenworth, Kansas, and Federal Correctional Institute El Reno in El Reno, Oklahoma. For each agency, we also interviewed officials at the headquarters and regional levels. Information obtained from these sites and regional officials is not generalizable to the selected agencies, and information from these agencies is not generalizable to other agencies. To determine how much selected agencies spend to store property in warehouses, as well as the numbers and square footage of these warehouses, we analyzed FRPP data from fiscal year 2018 for FAA and BOP, and DOE fiscal year 2018 real property data for the Office of Science because DOE reported most data to FRPP at the department level; both sources included information about direct costs such as rent, operations, and maintenance costs. We used FRPP data from fiscal year 2018 because that was the most recent data available when we conducted our analysis and DOE data covering the same period to be consistent. We reviewed documentation related to these data sources, interviewed knowledgeable officials, and determined that these data were sufficiently reliable for providing information about warehouse numbers, square footage, and the costs listed above. To determine the extent to which selected agencies assess the ongoing need for property stored in warehouses, we reviewed statutes, regulations, GSA guidance, our prior work, reports by federal agencies’ Offices of Inspector General, and relevant industry standards related to property storage and warehousing practices. In addition, for selected agencies we analyzed property policies and procedures for identifying and disposing of unneeded property and interviewed headquarters, regional, and site officials. We also interviewed three industry stakeholders—two property-management and one standards-setting organization—to discuss property storage and warehousing processes, practices, and standards that agencies could use to assess the ongoing need for property. We selected these organizations based on their knowledge about property management practices. Furthermore, we interviewed officials from four agencies—Census Bureau, Department of State, Internal Revenue Service, and National Aeronautics and Space Administration—that participate in the Interagency Committee on Property Management (ICPM), a committee chaired by GSA that consists of executive agency representatives interested in federal property. We invited all ICPM participants to speak with us regarding their practices for identifying unneeded property and interviewed all participants who volunteered to participate to understand how other agencies assess property for ongoing need. Finally, we reviewed FPPMA’s requirements and interviewed GSA’s Office of Government-wide Policy officials about GSA’s role in assisting agencies in identifying unneeded federal property, how FPPMA could affect GSA’s roles and responsibilities going forward, and GSA’s progress in implementing FPPMA. We conducted this performance audit from October 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the individual named above, the following individuals made important contributions to this report: Nancy Lueke (Assistant Director), Rebecca Rygg (Analyst-in-Charge), Terence Lam, Malika Rice, Kelly Rubin, Patrick Tierney, Laurel Voloder, and Crystal Wesco.", "summary": "Federal civilian agencies hold and manage billions of dollars in property that is not considered to be real property, such as vehicles, furniture, computers, and scientific instruments. Some of these items are stored in nearly 18,000 warehouses covering more than 90-million square feet. Agencies are required by law to regularly identify and dispose of unneeded items. However, GAO reported in 2018 that agencies often did not do so. The Federal Personal Property Management Act of 2018 requires agencies to use GSA guidance to assess the utilization and ongoing need for property. GAO was asked to review property stored in warehouses. This report examines: (1) what is known about property in selected agencies' warehouses and how much agencies spend to store it, and (2) the extent to which selected agencies assess the ongoing need for property stored in warehouses. GAO reviewed federal statutes, regulations, and GSA's guidance; analyzed policies from three agencies—FAA, Office of Science, and BOP—which were selected based on total warehouse square footage, among other factors; conducted site visits to agencies' warehouses; and interviewed stakeholders such as agency officials and industry groups. GAO found that three selected agencies stored a wide variety of property in their warehouses. For example: Federal Aviation Administration (FAA) warehouses at four main sites contained items used to build and repair aviation support systems, such as wind shear alert systems. Other sites contained tools and equipment to maintain aviation support systems or housed the systems themselves. The Department of Energy's Office of Science warehouses, located primarily at national laboratories, contained items, such as large magnets, for use in scientific experiments. Bureau of Prisons (BOP) warehouses, located mainly at federal correctional institutions, contained items, such as food, uniforms, and soap, for inmates. The above agencies reported spending approximately $50.1 million in fiscal year 2018 on warehouse rent, operations, and maintenance costs. The three selected agencies generally did not systematically assess the ongoing need for property in their warehouses and had limited guidance for doing so. For example, although two of the agencies had policies about when such an assessment should occur, none of the agencies specified how it should occur for most types of property. Instead, agencies primarily relied on agency officials' professional judgment to assess ongoing need. GAO identified instances where agencies retained unneeded property absent relevant guidance. For example, one agency site had stored obsolete computers dating back to the 1990s. While the General Services Administration (GSA) drafted guidance in response to recent legislation, this guidance does not describe approaches or practices stakeholders identified as potentially useful for assessing ongoing need for property, such as periodic retention justifications, use of data analytics, and utilization reviews. Further, while GSA officials intend to put the final guidance on GSA's website and provide it to agencies that participate in a GSA-chaired committee on property management by December 2019, GSA has not provided a documented plan or a timeline for broader dissemination. Guidance that incorporates such approaches could help agencies avoid retaining property that is no longer needed and, as a result, allow them to better manage their property and use of their warehouse space. GAO recommends that GSA incorporate approaches agencies could use to assess the ongoing need for property in GSA's guidance—such as periodic justifications, use of data, and utilization reviews—and develop a plan for communicating the guidance government-wide. GSA concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "In our review of 57 providers selected for our February 2019 report, we found that the responsible VA medical centers took action against some providers with disqualifying information in the NPDB but overlooked others. We found that VA medical centers took administrative or disciplinary actions against some providers, such as removing them from patient care, after becoming aware of adverse information in the NPDB. However, many of these actions were taken following our review and a VHA-wide licensure review, both of which occurred in 2018, rather than at the time of the NPDB report. Specifically, the responsible VA medical centers removed five providers who they determined did not meet VA licensure requirements following our inquiries. For example, one of these five providers had surrendered a license in 2014, while employed at VA, but was not removed by the VA medical center until after our inquiries in 2018. Additionally, another provider was reported to the Drug Enforcement Administration (DEA) by a VA medical center after we inquired about the provider prescribing controlled substances without appropriate registration. We also found that VA medical centers hired or retained some of the 57 providers who they acknowledged had disqualifying adverse information in the NPDB, which is inconsistent with VHA policy. Specifically, these providers had licenses that were revoked or surrendered for cause, but VA medical center officials overlooked or were unaware of this information. However, none of these providers still worked at VHA at the time we completed our review. For example, one VA medical center hired a provider who had a state license revoked for patient neglect and substandard care. VA medical center officials stated that they received the NPDB report about the revoked license at the time the provider was hired in 2014 but it was inadvertently overlooked by multiple staff. This provider voluntarily resigned in 2017. In our February 2019 report, we found that three factors were largely responsible for inconsistent adherence to VHA policies that disqualify providers from employment. First, some medical center officials are not aware of key VHA policies, such as the requirement that a provider who has had a license revoked or surrendered for cause is ineligible for employment unless the license is reinstated. For example, in the case of the provider who surrendered a license in 2014, documentation shows that the medical center staff became aware of the surrendered license in 2015, but VHA staff stated that the removal was stalled due to confusion about policies. This lack of awareness of key policies may be linked to a lack of mandatory training for credentialing staff. Second, gaps in VHA policy allow for inconsistent interpretation. For example, VHA has not issued policies pertaining to employing providers who have had their DEA registration for prescribing controlled substances revoked or surrendered for cause. While the DEA requires registrants, like VHA, to obtain a waiver before employing such providers, VHA policy is silent on the requirement to obtain a waiver; we found that VA medical center officials were unclear on the DEA requirement and had hired providers without obtaining the required DEA employment waiver. Further, we found that two providers inappropriately prescribed controlled substances without a DEA waiver. Third, VHA’s oversight of VA medical centers’ reviews of adverse information is inadequate. Under VHA policy, VISN officials are responsible for reviewing providers with certain adverse licensure actions. However, we found that this review was not always conducted or documented. Further, although VHA-wide reviews of provider licenses have been completed and have identified providers with licensure issues, VHA officials indicated that these types of reviews are not routinely conducted because they are labor intensive. In our February 2019 report, we also found that some VA medical centers had taken steps to improve the credentialing process and identify providers who do not meet the licensure requirements. For example, one medical center completed a periodic review of all licensed providers to identify providers who may have had an expired licensure issue. Another VA medical center updated its policies to require providers with adverse actions to be reviewed by management. However, we found that VHA does not routinely assemble and disseminate information about initiatives that medical centers have undertaken to improve the oversight of providers. In our February 2019 report, we concluded that without consistent adherence to VHA employment policies and adequate oversight, VHA lacks assurance that all VA providers have the appropriate professional qualifications and clinical abilities to care for patients. To address these shortcomings, in our February 2019 report we made seven recommendations to VA. VA concurred with these recommendations. Table 1 summarizes these recommendations and the steps VA has taken to address them. As we reported in November 2017, we found that from October 2013 through March 2017, the five selected VA medical centers required reviews of a total of 148 providers’ clinical care after concerns were raised about their care. However, for almost half of these reviews, officials at these medical centers could not provide documentation to show that the reviews had been conducted. We found that all five VA medical centers lacked at least some documentation of the reviews they told us they conducted, and in some cases, we found that the required reviews were not conducted at all. For example, we found that the medical centers lacked documentation showing they conducted a prospective review of 26 providers. Additionally, VA medical center officials confirmed that they failed to conduct this required review for an additional 21 providers. We also found that the five selected VA medical centers did not always conduct reviews of providers’ clinical care in a timely manner. Specifically, of the 148 providers, the VA medical centers did not initiate reviews of 16 providers for 3 or more months, and in some cases, for multiple years, after concerns had been raised about the providers’ care. For three of these 16 providers, additional concerns about the providers’ clinical care were raised before the reviews began. In our November 2017 report, we found that two factors were largely responsible for the inadequate documentation and untimely provider reviews. First, VHA policy does not require VA medical centers to document all types of reviews of providers’ clinical care, including retrospective reviews, and VHA has not established a timeliness requirement for initiating reviews of providers’ clinical care. Second, VHA’s oversight of the reviews of providers’ clinical care is inadequate. Under VHA policy, VISN officials are responsible for overseeing the credentialing and privileging processes at their respective VA medical centers. While reviews of providers’ clinical care after concerns are raised are a component of credentialing and privileging, we found that none of the VISN officials we spoke with described any routine oversight of such reviews. This may be in part because the standardized tool that VHA requires the VISNs to use during their routine audits does not direct VISN officials to ensure that all reviews of providers’ clinical care have been conducted and documented. Further, some of the VISN officials we interviewed told us they were not using the standardized audit tool as required. In our November 2017 report, we concluded that without adequate documentation and timely completion of reviews of providers’ clinical care, VA medical center officials lack the information they need to make decisions about providers’ privileges, including whether or not to take adverse privileging actions against providers. Furthermore, because of its inadequate oversight, VHA lacks reasonable assurance that VA medical center officials are reviewing all providers about whom clinical care concerns have been raised and are taking adverse privileging actions against the providers when appropriate. To address these shortcomings and improve VA medical center reviews of provider quality and safety concerns, we made three recommendations to VA in our November 2017 report. VA concurred with these recommendations. Table 2 summarizes these recommendations and the steps VA has taken to address them. In our November 2017 report, we found that from October 2013 through March 2017, the five VA medical centers we reviewed had only reported one of nine providers that should have been reported to the NPDB as required by VHA policy. Furthermore, none of these nine providers were reported to state licensing boards as required by VHA policy. These nine providers either had adverse privileging actions taken against them or resigned or retired while under investigation before an adverse privileging action could be taken. The VA medical centers documented that these nine providers had significant clinical deficiencies that sometimes resulted in adverse outcomes for veterans. For example, the documentation shows that one provider’s surgical incompetence resulted in numerous repeat surgeries for veterans. Similarly, the documentation shows that another provider’s opportunity to improve had to be halted and the provider was removed from providing care after only a week due to concerns that continuing the review would potentially harm patients. In addition to these nine providers, one VA medical center terminated the services of four contract providers based on deficiencies in the providers’ clinical performance, but the facility did not follow any of the required steps for reporting providers to the NPDB or relevant state licensing boards. This is concerning, given that the VA medical center documented that one of these providers was terminated for cause related to patient abuse after only 2 weeks of work at the facility. At the time of our review, two of the five VA medical centers we reviewed each reported one provider to the state licensing boards for failing to meet generally accepted standards of clinical practice to the point that it raised concerns for the safety of veterans. However, we found that the medical centers’ reporting to the state licensing boards took over 500 days to complete in both cases, which was significantly longer than the 100 days suggested in VHA policy. Across the five VA medical centers, we found that providers were not reported to the NPDB and state licensing boards as required for two reasons. First, VA medical center officials were generally not familiar with or misinterpreted VHA policies related to NPDB and state licensing board reporting. For example, at one VA medical center, we found that officials failed to report six providers to the NPDB because they were unaware that they were responsible for NPDB reporting. Officials at two other VA medical centers incorrectly told us that VHA cannot report contract providers to the NPDB. Second, VHA policy does not require the VISNs to oversee whether VA medical centers are reporting providers to the NPDB or state licensing boards when warranted. We found, for example, that VISN officials were unaware of situations in which VA medical center officials failed to report providers to the NPDB. As a result of VHA staff misinterpretation of VHA policy and insufficient oversight, we concluded that VHA lacks reasonable assurance that all providers who should be reported to the NPDB and state licensing boards are reported. Consequently, the NPDB and state licensing boards in other states where the providers we identified held licenses were not alerted to concerns about the providers’ clinical practice. We reported that this could allow a provider who delivered substandard care at one VA medical center to obtain privileges at another VA medical center or at hospitals outside of VA’s health care system. In our November 2017 report, we noted several cases of this occurring among the providers who were not reported to the NPDB or state licensing boards by the five VA medical centers we reviewed. For example, We found that two of the four contract providers whose contracts were terminated for clinical deficiencies remained eligible to provide care to veterans outside of that VA medical center. At the time of our review, one of these providers held privileges at another VA medical center, and another participated in the network of providers that can provide care for veterans in the community. We also found that a provider who was not reported as required to the NPDB during the period we reviewed had their privileges revoked 2 years later by a non-VA hospital in the same city for the same reason the provider was under investigation at the VA medical center. Officials at this VA medical center did not report this provider following a settlement agreement under which the provider agreed to resign. A committee within the VA medical center had recommended that the provider’s privileges be revoked prior to the agreement. There was no documentation of the reasons why this provider was not reported to the NPDB. To improve VA medical centers’ reporting of providers to the NPDB and state licensing boards and VHA oversight of these processes, we made one recommendation in our November 2017 report. VA concurred with this recommendation. Table 3 summarizes the recommendation and the steps VA has taken to address it. Chairman Pappas, Ranking Member Bergman, and Members of the Subcommittee, this concludes my statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff members have any questions concerning this testimony, please contact me at (202) 512-7114 (silass@gao.gov). Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals who made key contributions to this testimony include Marcia A. Mann (Assistant Director), Kaitlin M. McConnell (Analyst-in-Charge), Summar C. Corley, Cathy Hamann, Jacquelyn Hamilton, and Vikki Porter. Other contributors include David Bruno, Julia DiPonio, Ranya Elias, Kathryn A. Larin, and Joy Myers. According to Department of Veterans Affairs’ (VA) Veterans Health Administration (VHA) policies, all licensed health care providers must be credentialed before they are permitted to work. Credentialing is the process of screening and evaluating qualifications and other credentials— including licensure, education, and relevant training—that is the first step in the process of determining whether the provider has appropriate clinical abilities and qualifications to provide medical services. Credentialing processes and requirements differ for independent licensed providers, such as doctors—who are permitted by law and the facility to deliver patient care services independently, without supervision—and dependent providers, such as nurses—who deliver patient care under the supervision or direction of an independent provider. Additionally, VHA policy states that only licensed independent providers may be granted clinical privileges. Privileging is a process through which a provider is permitted by a facility to independently provide medical or patient care that is in alignment with the provider’s clinical competence. Figure 1 provides a summary of the VHA credentialing and privileging processes for independent and dependent providers. VHA facilities are also required to monitor providers’ licenses after they are hired to ensure the licenses are current and review any licensure actions, in accordance with VHA policy. Figure 2 provides a summary of VHA’s processes for monitoring independent and dependent providers’ licenses. Veterans Health Administration: Greater Focus on Credentialing Needed to Prevent Disqualified Providers from Delivering Patient Care. GAO-19-6. Washington, D.C.: February 28, 2019. Department of Veterans Affairs: Actions Needed to Address Employee Misconduct Process and Ensure Accountability. GAO-18-137. Washington, D.C.: July 19, 2018. VA Health Care: Improved Oversight Needed for Reviewing and Reporting Providers for Quality and Safety Concerns. GAO-18-260T. Washington, D.C.: November 29, 2017. VA Health Care: Improved Policies and Oversight Needed for Reviewing and Reporting Providers for Quality and Safety Concerns. GAO-18-63. Washington, D.C.: November 15, 2017. Veterans Health Care: Improved Oversight of Community Care Physicians’ Credentials Needed. GAO-16-795. Washington, D.C.: September 19, 2016. VA Health Care: Improvements Needed in Processes Used to Address Providers’ Actions That Contribute to Adverse Events. GAO-14-55. Washington, D.C.: December 3, 2013. Veterans Health Care: Veterans Health Administration Processes for Responding to Reported Adverse Events, GAO-12-827R. Washington, D.C.: August 24, 2012. VA Health Care: Improved Oversight and Compliance Needed for Physician Credentialing and Privileging Processes. GAO-10-26. Washington, D.C.: January 6, 2010. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Nearly 165,000 licensed health care providers, such as physicians and nurses, provide care in VHA's VA medical centers and outpatient facilities. Medical center staff must determine whether to hire and retain health care providers by reviewing and verifying information about their qualifications and practice history. The NPDB is a key source of information about a provider's clinical practice history. Medical center staff must also investigate any concerns that arise about the clinical care their providers deliver. Depending on the findings from these reviews, medical centers may take an adverse privileging action against a provider. VA medical centers are required to report providers to the NPDB and state licensing boards under certain circumstances. Failing to adhere to these requirements can negatively affect patient safety. This testimony is primarily based on GAO's 2019 and 2017 reports on VHA processes for reviewing and reporting quality and safety concerns about VA providers. It addresses VA medical centers' implementation and VHA's oversight of (1) reviews of adverse information about providers in the NPDB; (2) reviews of providers' clinical care after concerns are raised; and (3) reporting of providers to the NPDB and state licensing boards. For the 2019 report, GAO reviewed a nongeneralizable sample of 57 VA providers who had an NPDB report. For the 2017 report, GAO reviewed providers whose clinical care was reviewed after a concern was raised about that care at a nongeneralizable selection of five VA medical centers. The Department of Veterans Affairs (VA) needs to take action to ensure its health care providers have the appropriate qualifications and clinical abilities to deliver high quality, safe care to veterans, as GAO recommended in its February 2019 and November 2017 reports. Specifically, GAO found the following: VA medical centers took action against some providers who did not meet VA licensure requirements, but overlooked others . In its 2019 report, GAO found that some VA medical centers took administrative or disciplinary actions against these providers, such as removing them from employment, after becoming aware of disqualifying information in the National Practitioner Data Bank (NPDB). The NPDB is an electronic repository that contains information on providers who have been disciplined by a state licensing board, among other information. However, in some cases VA medical centers overlooked or were unaware of disqualifying information in the NPDB. For example, officials told GAO they inadvertently overlooked a disqualifying adverse action and hired a provider whose license had been revoked for patient neglect. GAO found three reasons for this inconsistency: lack of mandatory training for key staff, gaps in Veterans Health Administration (VHA) policies, and inadequate oversight. Selected VA medical centers' reviews of providers' clinical care were not always documented . The five selected VA medical centers that GAO included in its 2017 report were required to review 148 providers' clinical care after concerns were raised about their care from October 2013 through March 2017. However, officials at these medical centers could not provide documentation to show that almost half of these reviews had been conducted. GAO found two reasons for inadequate documentation of these reviews: gaps in VHA policies and inadequate oversight of the reviews. Selected VA medical centers did not report providers to the NPDB or to state licensing boards as required . The five selected VA medical centers that GAO included in its 2017 report had reported one of nine providers to the NPDB that they were required to report from October 2013 through March 2017. None of these providers were reported to state licensing boards, as required by VHA policy. These nine providers either had adverse privileging actions taken against them—actions that limit the care providers can deliver at a facility or prevent the providers from delivering care altogether—or resigned or retired while under investigation before such an action could be taken. GAO found two reasons providers were not reported: lack of awareness or understanding of VHA policies and inadequate oversight of this reporting. GAO made 11 recommendations in its 2019 and 2017 reports to address the deficiencies identified. VA implemented two of these 11 recommendations, and provided action plans to address the other nine recommendations.", "document_type": "gao"}
{"report": "DOD’s policy is to ensure that eligible personnel and their families have access to affordable, quality housing facilities and services consistent with grade and dependent status, and that the housing generally reflects contemporary community living standards. From the inception of MHPI, the military departments were provided with various authorities to obtain private-sector financing and management to repair, renovate, construct, and operate military housing in the United States and its territories. Through these authorities, the military departments have entered into a series of agreements with private partners to provide housing to servicemembers and their families. The military departments have flexibility in how they structure their privatized housing projects, but typically the military departments lease land to private developers for 50- year terms and convey existing housing located on the leased land to the developer for the duration of the lease. The developer then becomes responsible for renovating and constructing new housing and for the daily management of these housing units. At the end of fiscal year 2017, 14 private partners were responsible for 79 privatized military family housing projects—34 for the Army, 32 for the Air Force, and 13 for the Navy and Marine Corps—in the United States, each of which includes housing at one or more military installation. The Deputy Assistant Secretary of Defense for Facilities Management, under the authority, direction, and control of the Assistant Secretary of Defense for Sustainment, is responsible for all matters related to MHPI and is the program manager for all DOD housing, whether DOD-owned, DOD-leased, or privatized. In this capacity, the Deputy Assistant Secretary is to provide both guidance and general procedures related to military housing privatization, as well as required annual reports to Congress on the status of privatized military housing projects. However, it is the responsibility of the military departments to execute and manage privatized housing projects, including conducting financial management and monitoring their portfolio of projects. Each military department has issued guidance that outlines its responsibilities for privatized housing, such as which offices are responsible for overseeing privatized housing projects. We have previously reported on DOD’s privatized housing program. Table 1 provides a summary of key findings and recommendations from our prior reports and the implementation status of the recommendations. Each military department conducts a range of oversight activities—some more extensive than others—for its privatized housing projects. For example, among other things, military departments review sample work order requests and inspect housing during the change of occupancy process. DOD guidance requires that the military departments ensure eligible personnel have access to quality housing facilities and services that generally reflect contemporary living standards. Further, DOD’s housing manual states that because privatization creates a long-term governmental interest in privatized housing, it is essential that projects be attentively monitored. Through its guidance, DOD delegates oversight responsibility of the individual privatized housing projects to each of the military departments. In addition, according to documents we reviewed, individual project business agreements set guidelines that convey the management, operation, and maintenance duties to the private partner, with the caveat that the military departments still have the right to access the premises or private partner records to ensure compliance with applicable laws. We determined that OSD and the military departments’ oversight has been limited in key areas. Specifically, our ongoing review showed (1) the scope of oversight of the physical condition of privatized housing has been limited; (2) performance metrics focused on quality of maintenance and resident satisfaction may not provide meaningful information on the condition of privatized housing; (3) there is a lack of reliable or consistent data on the condition of privatized housing; and (4) past DOD reports to Congress on resident satisfaction are unreliable due to inconsistent handling and calculation of the data, and therefore may be misleading. DOD delegates oversight responsibilities of the individual privatized housing projects to each of the military departments, and each military department has subsequently issued guidance outlining oversight roles and responsibilities. Military department oversight activities generally fall into two categories—(1) daily oversight of management and operations and (2) reviews of compliance with each project’s business agreements. Daily oversight of management and operations. Daily oversight of a project’s management and operations is to be conducted by each installation’s military housing office. Military housing officials told us activities to monitor the physical condition of housing units generally include reviewing sample work order requests, following up with a sample of residents to check on their experience with recently completed work, and inspecting housing units during the change of occupancy process. Based on our preliminary observations, the implementation and scope of these activities varies and can be limited. For example, during our site visits conducted from June through August 2019, we identified the following installation-specific practices: Military housing office officials at one Army installation told us that they inspect 100 percent of housing units that have completed change of occupancy maintenance. In contrast, officials from an Air Force installation told us that they inspect 10 to 20 percent of housing units that have completed change of occupancy maintenance. Military housing officials at one Marine Corps installation told us that for one of the two partners that own housing on the base, they had access to only 3 percent of completed work order tickets from the previous month, as reported to them by the private partner. Officials from a Navy installation told us that they had access to the private partner’s maintenance record system and would pull reports on new resident housing occupants who had made 6 or more maintenance calls in a 30-day period. Military housing officials at half of the sites we visited stated that staffing levels limited their ability to carry out oversight duties, such as work order data analysis and housing inspections. Reviews of compliance with each project’s business agreements. Reviews of compliance with a project’s business agreements are a joint effort between the local military housing office, the private partners, military department installation commands, and other echelons of command. These reviews can include neighborhood tours to view project amenities such as community centers, playgrounds, and pools, all of which are owned, maintained, and operated by the private partner companies, as well as exteriors of housing units. However, our preliminary work showed these reviews have been limited in the scope of their assessment of the physical condition of the housing units, as interior walk-throughs may have been limited to just a few housing units at each installation. According to military department officials, each department is currently taking steps to revise guidance and standardize daily oversight activities in an effort to provide consistent oversight across projects and installations, and to increase the focus of oversight on the physical condition of housing. The military departments are taking additional steps, such as increasing staffing levels, improving training for military housing office officials, and ensuring that housing officials have independent access to data. However, each military department is working to implement service-specific initiatives with only limited guidance from OSD on the level of oversight expected of the services as it relates to the condition of the housing. While existing OSD guidance provides objectives to the military departments for oversight of the condition of DOD-owned housing, guidance for privatized housing is focused on the implementation of projects, construction of new housing units, and financial management. The guidance does not include objectives for monitoring the condition of privatized housing projects, such as objectives focused on both ensuring the operation and maintenance of privatized housing to standards that provide safe living conditions for servicemembers and providing authorities to installation commanders to oversee those standards. We will continue to assess any implications of the lack of OSD guidance as part of our ongoing review. The military departments each use a range of project-specific performance metrics to monitor private partner performance, but our ongoing work showed that the metrics designed to focus on resident satisfaction and on the quality of maintenance conducted on housing units may not provide meaningful information or reflect the actual condition of the housing units. Most but not all of the private partners are eligible to receive performance incentive fees based on generally meeting the performance metrics established in each individual project’s business agreement. Private partner performance is measured through a variety of metrics, such as resident satisfaction, maintenance management, project safety, and financial management. To determine how well the private partners are performing under the metrics, military housing office officials told us that they rely on a range of specific indicators established in the project business agreements. However, the indicators themselves may not provide meaningful information on the private partner’s performance in maintaining quality housing units. For example, our preliminary work identified the following: Maintenance management. One indicator of performance of maintenance management that is regularly included in project business agreements measures how often the property manager’s response time to work orders meets required time frames established in the project’s business agreements. While this indicator measures the timeliness of the private partner’s response, it does not measure or take into account the quality of the work that was conducted or whether the resident’s issue was fully addressed. Some projects include indicators that aim to more directly measure quality, such as the number of work orders placed during the first 5 business days of residency, which may indicate the extent to which change of occupancy maintenance was completed. Resident satisfaction. One example of an indicator of resident satisfaction is whether a project has met the target occupancy rates established in the business agreements. An OSD official we spoke with and private partner officials told us they use occupancy as an indicator of satisfaction based on the assumption that residents would move if they were dissatisfied with their housing unit. However, based on our focus groups, this may not be a reliable assumption. Although most residents are not required to live in military housing, residents in each of our 15 focus groups indicated a variety of reasons for choosing to live in privatized housing, many of which did not have to do with their satisfaction with the quality or condition of their homes. For example, residents cited factors influencing their decision to live in privatized housing, such as living in close proximity to military medical or educational services for children or other family members who receive benefits through the military’s Exceptional Family Member Program, access to quality schools, and a lack of safe and affordable housing in the surrounding community. OSD and military department officials we spoke with recognized that the current metrics do not consistently focus on or prioritize the private partners’ performance with maintaining housing units and ensuring resident satisfaction. In October 2019 OSD issued new guidance standardizing the performance incentive fee framework across the military departments. According to OSD and the private partners with whom we spoke, this guidance was developed through a joint effort with the military departments and the private partners; it provides a framework where the metrics for resident satisfaction and maintenance management will account for a majority of the fee, with project safety and financial performance weighted less heavily. However, according to officials from OSD and officials we spoke with from each of the military departments, the specific indicators used to drive the metrics will need to be negotiated with each of the private partners for each project. Performance indicators designed to more directly measure the quality of maintenance conducted on housing units and resident satisfaction will provide military departments more transparency into private partner performance with regard to these two important metrics—metrics that are often directly tied to the performance incentive fees provided to the private partners. The housing projects’ business agreements typically include a requirement for the private partner to maintain a records management system to record, among other things, maintenance work requested and conducted on each housing unit. According to private partner officials, each company uses commercial property management software platforms that are used for activities such as initiating maintenance work orders and dispatching maintenance technicians. Some private partner officials also stated that data from the work order tracking systems were intended to prioritize and triage maintenance work, not to monitor the overall condition of privatized housing units. While data from these work order tracking systems may be useful for point-in-time assessments of work order volume at a given installation, military department officials told us that efforts are underway to monitor work order data to increase the military departments’ oversight and the accountability of the private partners for providing quality housing. However, in our ongoing work we observed that these data are not captured reliably or consistently for use in the ongoing monitoring of the condition of privatized housing units. We received and reviewed data from each of the 14 private partners’ work order tracking systems covering each of the 79 privatized family housing projects. Based on our preliminary analysis of the initial data provided by the private partners, we noted the following: Data anomalies. We identified anomalies in work order data from each of the 14 partners. For example, we identified instances of, among other things, duplicate work orders, work orders with completion dates prior to the dates that a resident had submitted the work order, and work orders still listed as in-progress for more than 18 months. Inconsistent use of terminology. Based on our preliminary review of the data provided by the private partners and discussions with private partner officials, we noted cases where work orders were inconsistently entered into the work order tracking systems with respect to two primary factors—(1) how the request is described by the resident or interpreted by the official entering the data, which can differ for each work order, and (2) the existing range of pre- established service category options in the private partner’s work order tracking system, which differ among the partners. Differing practices for opening and closing work orders. At some installations we visited, private partners noted changes in practices for opening and closing work orders, limiting the usefulness of the data in monitoring the status of work orders over time and thus the condition of privatized housing. According to military department officials, efforts to review data from the private partners’ work order tracking systems has increased, and military department officials told us that they have found similar limitations. However, there are no standard practices currently in place for assessing the accuracy and reliability of the work order data or for setting standard terminology and practices for opening and closing work orders. DOD is statutorily required to provide reports to Congress that include, among other things, information about military housing privatization projects’ financial health and performance and backlog, if any, of maintenance and repairs. These reports have included information on resident satisfaction with privatized housing based on results of the annual military department satisfaction surveys. Based on our preliminary work, we have determined that information on resident satisfaction in these reports to Congress on privatized housing has been unreliable and may be misleading due to variances in the data the military departments collect and provide to OSD and in OSD’s calculation and presentation of these data. In May 2019, OSD issued its report for fiscal year 2017, which stated that overall resident satisfaction for calendar year 2017 was 87 percent. For OSD’s fiscal year 2017 report, the military departments provided data on resident satisfaction based on information from the annual resident satisfaction surveys. Specifically, OSD’s instructions to the military departments required the military departments to report satisfaction based on resident responses to the question that asks: “Would you recommend privatized housing,” with results indicating how many tenants responded “yes,” “no,” or “don’t know.” However, the military departments’ approaches for collecting data in their annual resident satisfaction surveys varies, which limits their ability to assess whether residents would recommend privatized housing. Instead of asking whether residents would recommend privatized housing, the military departments’ annual resident satisfaction survey asks residents the following: “How much do you agree or disagree with the following statement, ‘I would recommend this community to others.’” A resident’s satisfaction with his or her community and inclination to recommend it to others may not be reflective of satisfaction with either the privatized housing unit or privatized housing in general. Residents are then provided the following response categories on a scale of 5 to 0: (5) strongly agree, (4) agree, (3) neither agree nor disagree, (2) disagree, (1) strongly disagree, and (0) not applicable, no opinion, don’t know, or no answer. Through our analysis, we have identified variances in the methods the military departments use to translate the residents’ responses into the “yes,” “no,” or “don’t know” categories. The variances in how the military departments calculate “yes,” “no,” or “don’t know” result in inconsistencies in how resident satisfaction is ultimately reported to Congress. Specifically: For the fiscal years 2015 through 2017 reports, Navy officials told us that they counted responses reported in categories 5 and 4 as “yes,” responses in categories 2 and 1 as “no,” and responses in categories 0 and 3 as “don’t know.” For the same time period, Air Force officials told us that they counted responses in categories 5, 4, and 3 as “yes,” responses in categories 2 and 1 as “no,” and responses in category 0 as “don’t know.” The Army calculated responses differently for the fiscal years 2015, 2016, and 2017 reports. Specifically: For the fiscal year 2015 report, the Army counted responses in categories 5, 4, and 3 as “yes,” responses in categories 2 and1 as “no,” and responses in category 0 as “don’t know.” For the fiscal year 2016 report, the Army counted responses in categories 5 and 4 as “yes,” responses in categories 2, 1, and 0 as “no,” and responses in category 3 as “don’t know.” For the fiscal year 2017 report, the Army counted responses in categories 5 and 4 as “yes,” responses in categories 2 and 1 as “no,” and responses in categories 0 and 3 as “don’t know.” In our ongoing work, we have also identified instances of errors and inaccuracies in how OSD calculates these data and reports on resident satisfaction to Congress. Specifically, we found missing data points and incorrect formulas, among other errors, in OSD’s calculation of the data submitted by the military departments for OSD’s fiscal year 2017 report to Congress. For example: The formula used by OSD to calculate overall resident satisfaction for the fiscal year 2017 report did not include data for several projects, including for four Army projects that, as of September 30, 2017, accounted for over 18 percent of the Army’s total housing inventory. Additionally, we identified that OSD did not include resident satisfaction data for a Navy project in its fiscal year 2017 report to Congress, even though when we reviewed the Navy’s submission to OSD, we found that the Navy had included data for that project. For one Air Force project, OSD reported identical resident satisfaction data for the fiscal year 2015, 2016, and 2017 reports, despite the fact that Air Force officials had noted in their submissions to OSD that the resident satisfaction data were from the annual resident satisfaction survey conducted in December 2013. We also found that presentation of data in OSD’s report to Congress may be misleading because OSD did not explain the methodology it used to calculate the overall resident satisfaction percentage or include caveats to explain limitations to the data presented. Specifically, OSD did not include information on overall response rates to the annual satisfaction survey for each military department, nor did it include response rates by project. Low response rates can create the potential for bias in survey results. For example, in the report for fiscal year 2017, OSD reported that 25 percent of residents living in renovated housing units for one privatized housing project were satisfied with their housing, but we found that only four residents had provided responses to this question. Thus, only one resident reported being satisfied. In addition, we found that OSD did not provide an explanation in the report for why five projects were listed as “not applicable.” According to OSD officials, this error was a quality control issue that they plan to address. According to OSD officials, OSD and the military departments are reviewing the resident satisfaction survey questions and will be identifying and implementing measures to ensure an accurate and reliable process to compile, calculate, report and compare MHPI resident satisfaction by military department and across DOD. Military housing office officials located at each installation are available to provide resources to servicemembers experiencing challenges with their privatized housing, among other services, but these offices have not always effectively communicated this role to residents of privatized housing. The military housing office is to provide new residents with information on their local housing options, to include referral services for housing options. According to some military housing office officials, the military housing office then works with the private partner to identify the eligibility and type of home the servicemember qualifies for, if the resident chooses to live in privatized housing. According to some residents we spoke with in one of our focus groups, beyond this initial interaction, military housing office officials generally do not interact with residents on a regular basis. Additionally, residents who participated in our focus groups noted they were sometimes confused about the military housing offices’ roles and responsibilities with regard to the maintenance of their home; there was a perception that the military housing office was not working independently of the partner in the residents’ best interest; or they did not know the military housing office existed. The military department oversight agencies have acknowledged resident confusion and a lack of awareness regarding the role of the military housing offices as an issue. In May 2019, the Army Inspector General reported to the Secretary of the Army that at 82 percent of Army installations with privatized housing, residents did not know how to escalate issues to either the private partner or the Army housing office. Additionally, the Army Inspector General reported that installation command teams and staff cited multiple circumstances where military housing offices and tenant advocacy roles and responsibilities were unclear. Further, some military housing office officials with whom we spoke during our site visits acknowledged the gap in resident awareness regarding the existence and purpose of the military housing office. Some military housing officials also noted that some residents are unaware of the difference between the military housing office and the private partner office, due in part to their physical co-location and unclear building signage. Each military department has issued information that establishes that its housing offices can assist in the resident dispute resolution process. Specifically, if servicemembers are experiencing a dispute with a private partner, military department guidance establishes varying roles for their respective military housing office officials. For example, Army policy states that each installation should have an official tasked with supporting servicemembers regarding resident issues that cannot be resolved by the private property manager. This individual is also responsible for resolving every resident complaint and the military housing office, if required, can request mediation by the garrison commander. Despite this guidance, according to DOD officials, the military departments had generally decreased their staffing and oversight of daily privatized housing operations since the enactment of MHPI. For example, Army officials we spoke with in January 2019 told us that they typically filled 80 percent of available military housing office positions across their installations. Additionally, officials stated that housing offices were generally staffed with two or three officials responsible for assisting servicemembers with housing needs both on the installation as well as in the local community. Further, the officials told us that the team at one Army installation was decreased from about 15 to 3 positions. According to OSD officials, while housing offices should generally not require the number of personnel that were necessary prior to privatization, reductions following sequestration reduced housing staff below the level necessary to fully perform required privatized housing oversight as it was originally envisioned at the outset of the program. OSD has recognized that the military departments’ communication with residents about their role as a resource for them has been limited. In February 2019, the Assistant Secretary of Defense for Sustainment testified before Congress that a way forward in addressing resident concerns would require focus in three key areas: communication, engagement, and responsiveness. Some military housing office officials told us they have taken steps to increase resident awareness, such as increasing the advertising of the military housing office’s role and contact information, conducting town hall meetings, and rebranding their military housing offices to differentiate them from the private partners. For example, a Marine Corps housing office official stated that the housing office established a document, which is distributed to residents by the private partner, informing residents of housing office contact information and the service’s 3-step dispute resolution process, but efforts have not been standardized across all projects. OSD, the military departments, and the private partners have identified and begun collaborating on a series of initiatives aimed at improving residents’ experiences with privatized housing, but our preliminary work showed that these efforts face challenges. According to an OSD official, a series of initiatives has been identified and are some are currently in various phases of development and implementation. Tri-service working groups, each chaired by a designated military department and comprising officials and legal counsel from each military department as well as private partner representatives, are leading efforts to develop and implement the initiatives. In particular, DOD and the private partners are collaborating on the following key initiatives: Development of a Resident Bill of Rights. The Resident Bill of Rights is to provide clarity to residents on their rights and responsibilities while living in privatized military housing. Development of a common tenant lease. The common lease framework will be binding in all 50 states, but also include addendums to capture state and local laws, as required. The common lease would provide residents of privatized housing with similar terms in their leases, regardless of where they are living and which private partner owns their housing unit. Establishment of a resident advocate position. The resident advocate position, according to an OSD official, will be available to provide independent advice, education, and support to residents. However, an OSD official noted that the military departments have not yet determined whether this individual would be active duty or civilian and where the position would fall organizationally—specifically, whether it would be part of the military housing office. Development of a standardized adjudication process. The military departments and private partners are developing a common dispute resolution process that would apply to all projects. According to OSD, this process would provide residents the right to have housing issues heard and resolved by a neutral third party. DOD and Congress are exploring additional initiatives and legislative proposals. However, both DOD and private partner officials have noted several challenges that could impact their ability to implement some of these initiatives and legislative proposals. Key challenges include the following: The need to collaborate with and obtain input and agreement from the large number of stakeholders involved in privatized housing. Many of the initiatives aimed at improving privatized housing require not only agreement between DOD and the private housing partners, but may also require discussion with and approval by the project bond holders. This requirement could limit the military departments’ legal authority to unilaterally make changes to existing business agreements. The private partners noted that the bond holders may be reluctant to agree to changes to the business agreements that could result in higher project costs. Limited military department resources. The military departments had reduced their involvement in daily privatized military housing operations as part of the overall privatization effort. This included reducing staffing levels at the installations. Each of the military departments has plans to increase the military housing office staffing at each installation to allow for enhanced oversight. The potential for negative financial impacts to the projects that may outweigh the intended benefits of the initiatives. Representatives from many of the private partners we met with expressed concern that some proposed initiatives may result in a financial burden for their projects, such as legal fees associated with the development of a common lease and the various addendums that would be required; unanticipated costs of hiring outside third party inspections; or the potential impact to project revenue that would result from residents withholding rent. Some of the private partners noted that the financial impact of unfunded requirements to projects that are already experiencing financial distress could result in even fewer funds available to reinvest in the physical condition of the housing units. In summary, while the privatization of military housing has resulted in private partners assuming primary responsibility for military housing, DOD maintains responsibility for overseeing privatized housing and ensuring that eligible personnel and their families have access to affordable, quality housing facilities and services. While DOD and the private partners have taken steps to address concerns raised about their ability to adequately maintain and oversee the condition of these housing units and provide quality housing for servicemembers, the extent to which the efforts will be sustained and result in improvements remains unclear. We are continuing our broader review of DOD’s oversight of privatized housing, including the issues addressed in this statement and will make recommendations as appropriate in our final report, which we anticipate issuing in early 2020. Chairman Inhofe, Ranking Member Reed, and Members of the Committee, this concludes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff members have any questions about this testimony, please contact Elizabeth A. Field, Director, Defense Capabilities and Management, at (202) 512-2775 or FieldE1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Kristy Williams (Assistant Director), Tida Reveley (Analyst in Charge), Austin Barvin, Ronnie Bergman, Vincent Buquicchio, William Carpluk, Juliee Conde-Medina, Mae Jones, Jordan Mettica, Kelly Rubin, Monica Savoy, and John Van Schaik. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In 1996, Congress enacted the Military Housing Privatization Initiative in response to DOD concerns about inadequate and poor quality housing for servicemembers. Today, private partners are responsible for the ownership, construction, renovation, maintenance, and repair of about 99 percent of housing units on military bases in the continental United States. DOD's policy requires that the department ensure eligible personnel and their families have access to affordable, quality housing facilities. The Office of the Secretary of Defense is responsible for providing guidance and general procedures related to military housing privatization. The military departments are responsible for executing and managing privatized housing projects. Drawing from ongoing work, GAO discusses (1) DOD's oversight of privatized military housing for servicemembers and their families, (2) efforts of the military departments to communicate their roles and responsibilities to servicemembers and their families, and (3) DOD and private partner development and implementation of initiatives to improve privatized housing. GAO reviewed relevant policies, guidance, and legal documents; visited 10 installations; conducted 15 focus groups; analyzed maintenance work order data; and interviewed relevant DOD and private partner officials. GAO will continue its ongoing work and make recommendations as appropriate in the final report. Each military department conducts a range of oversight activities—some more extensive than others—for its privatized housing projects, but these efforts have been limited in key areas. Specifically, based on GAO's ongoing work: The Department of Defense (DOD) conducts oversight of the physical condition of housing, but some efforts have been limited in scope. Military departments have guidance for conducting oversight of the condition of privatized housing. This oversight generally consists of reviewing a sample of work order requests, visually inspecting housing during change of occupancy, and conducting other point in time assessments. However, GAO found that these efforts are limited in scope. For example, interior walk-throughs may have been limited to just a few homes at each installation. DOD uses performance metrics to assess private partners, but metrics may not provide meaningful information on the condition of housing. The Office of the Secretary of Defense (OSD) has recently issued guidance to ensure consistency in the framework used to measure project performance. However, the specific indicators used to determine if the metrics are being met may not fully reflect private partner performance. For example, a common measure is how quickly the private partner responded to a work order, not whether the issue was actually addressed. DOD and private partners collect maintenance data on homes, but these data are not captured reliably or consistently. DOD is expanding its use of work order data to monitor and track the condition of privatized housing. However, based on GAO's analysis of data provided by all 14 private partners, these data cannot reliably be used for ongoing monitoring of privatized housing because of data anomalies and inconsistent business practices in how these data are collected. DOD provides reports to Congress on the status of privatized housing, but some data in these reports are unreliable and may be misleading. DOD provides periodic reports to Congress on the status of privatized housing, but reported results on resident satisfaction are unreliable due to variances in the data military departments provide to OSD and in how OSD has calculated and reported these data. Military housing offices located at each installation are available to provide resources to servicemembers experiencing challenges with their privatized housing, but GAO's ongoing work showed these offices have not always effectively communicated this role to residents. For example, residents in GAO's focus groups noted confusion over the roles and responsibilities of these offices, and military housing officials have found that residents could not readily differentiate between military and private housing officials. DOD, working with the private partners, has made progress in developing and implementing a series of initiatives. However, both DOD and private partner officials have noted several challenges that could affect implementation, including limitations to DOD's legal authority to unilaterally make changes to the terms of the projects and limited resources to implement increased oversight.", "document_type": "gao"}
{"report": "The platform economy is relatively new. Its rise has been tied to the development of the iPhone and Apple’s app store, which launched in 2008. Smart phones and apps made it easier for people to develop online marketplaces where buyers and sellers could connect to exchange goods. Since then, more platform companies have emerged, expanding into marketplaces for a host of services. This expansion has increased the ways in which people could sell their services or goods and lowered the barriers to entering this type of work. Platform companies generally facilitate the match, transaction, and payment between those seeking goods or services online and those providing them. Platform companies can be divided into marketplaces for services or goods and can be further divided into two types of services— transportation or other types of services—and two types of goods—retail or short-term rental. See figure 1 for a description of the different platform company marketplaces. Transportation platforms comprise the largest sector of the platform economy—by revenue and number of workers—and include companies like Uber and Lyft. Examples of platform companies providing other types of services include Care.com (child or senior care) and Upwork (a range of freelance services). Retail platforms like Etsy and eBay provide an online marketplace for goods. Companies that facilitate short-term rental include Airbnb (property) or Kerb (parking spaces). Platform companies also vary by the number and degree of specialization of services offered. For example, platform companies could be differentiated by the range of services offered, from those that offer a single service, such as transporting people and goods, to those that are a one-stop source to access workers specializing in an array of fields such as legal, information technology, marketing, writing, design, and accounting. Further, companies have adopted different business models. Some set prices while others allow the worker and customer to negotiate a price for the good or service provided. While platform workers can choose whether to accept work, companies are not committed to pay workers for a set number of hours, which allows companies to more easily manage demand fluctuations. Many platform companies classify workers providing services as independent contractors rather than as employees. This classification has implications for companies and workers. For independent contractors, companies are not responsible for paying a minimum wage or overtime and do not typically provide benefits such as paid leave, health insurance, or retirement plans. Furthermore, numerous labor protections that apply to employees do not apply to independent contractors. Companies also do not have to withhold and remit federal income or employment taxes for independent contractors. Instead, the worker is responsible for estimating, saving, and remitting these taxes each quarter to IRS. Furthermore, a platform company that operates as an intermediary between buyers and sellers to transfer funds for transactions—for example, a ride-hailing company that matches drivers and riders and processes those riders’ payments to drivers—may act as a third party settlement organization (TPSO) under IRS regulations, depending on its circumstances. Acting as a TPSO can further reduce the tax reporting requirements for these companies. SB/SE oversees taxpayers filing tax returns as self-employed individuals with business income, such as independent contractors, and businesses with less than $10 million in total assets. SB/SE aims to promote compliance among this population by raising awareness through outreach and education about tax obligations and how to comply. SB/SE also takes enforcement actions such as audits to pursue noncompliance, although with budget cuts, the number of audits has dropped considerably over time. While the nature of the work may be similar, key differences exist among traditional independent contractors, platform workers, and employees. We use the term platform worker for those workers that platform companies classify as independent contractors, which is the focus of our report. A major similarity is that all three types of workers provide services to customers in a business transaction, regardless of whether the customer is another business or an individual such as a homeowner. However, the three types of workers differ in terms of payments, tax responsibilities, and the information reported to IRS and workers. Figure 2 summarizes these differences, which are discussed below. A major difference is how these workers provide their services and receive payments. Traditional independent contractors directly provide their services to and receive payments from a business or an individual customer. Employees provide services through an employer who pays them wages. Unlike the other two types of workers, the platform worker provides services through a third party intermediary—the platform company—that uses online tools to connect workers with customers and facilitates payment between them. Platform workers that are classified as independent contractors have different tax obligations than employees. Independent contractors are generally expected to pay the full amount of their personal federal income and self-employment taxes—which are comprised of Social Security and Medicare taxes—through quarterly estimated tax payments to IRS. For employer-employee relationships, both the employer and the employee each pay roughly half of the Federal Insurance Contribution Act (FICA) taxes—which are comprised of Social Security and Medicare taxes. An employer is required to withhold income taxes and the employee portion of FICA taxes from wage payments made to the employee and pay these taxes to IRS. Additionally the employer is required to pay its own portion of the FICA taxes to IRS. These workers have different requirements for filing annual tax returns. Generally, independent contractors, including those that are platform workers, are to use Schedule C, Profit or Loss from Business (Sole Proprietorship), to report their self-employment revenues and expenses, and to calculate net income; the Schedule C is to be attached to their Form 1040. Employees are to report wage amounts received on their Form 1040. Businesses also differ in information-reporting obligations for workers classified as independent contractors and employees. Generally speaking, when a business pays an independent contractor, the business is required to report annual payments of $600 or more on Form 1099- MISC; for tax year 2020, businesses are to start using Form 1099-NEC to report such payments. Instead of a Form 1099-MISC (or Form 1099- NEC), TPSOs are required to file Form 1099-K to report annual payments made through their payment networks that exceed $20,000 and 200 transactions. When paying employees, businesses are to file a Form W- 2 annually to report all wages paid. A difference between platform workers and traditional independent contractors is that platform companies often have information on workers’ earnings and some expenses that could help workers file tax returns. For example, a ride-hailing company has information on fares, tips, and miles driven for each customer as well as fees and other charges. Traditional independent contractors who provide rides to individuals typically have no equivalent third party to provide such information. The population of platform workers is difficult to count, due in part to the variation in terms and definitions used to describe these workers such as gig, on-demand, sharing economy, contingent, and freelance, among others. As a result of these and other barriers, efforts to measure the size of the platform workforce through household surveys or administrative data have produced estimates that lack certainty; these estimates of the workforce size generally have ranged from around 1.5 million to 2 million for recent years and suggest that the platform workforce may be growing. In 2017, the Department of Labor’s Bureau of Labor Statistics (BLS) collected data on alternative employment. BLS’s survey found there are an estimated 1.6 million “electronically-mediated workers,” defined as those obtaining work through mobile apps and websites that connect workers with customers and arrange payment. However, in January 2019 we reported that several factors may have contributed to an undercount of platform workers. For example, survey questions may not have captured individuals who engage in platform work on a part-time or sporadic basis. BLS has stated that it will not use the same questions again and it is working with stakeholders and the Committee on National Statistics of the National Academies of Science, Engineering, and Medicine to address survey limitations. Despite the difficulties estimating the size of the workforce, two studies and stakeholders we interviewed have concluded that the platform workforce is growing. For example, in a review of customer checking account transactions, the JP Morgan Chase Institute documented a five- fold increase between 2012 and 2018 to about 2.3 million families receiving at least one payment from known platform companies. Two studies have also found that the majority of platform workers work part time, for a short time, for secondary income, and for relatively low earnings. For its sample, the JP Morgan Chase & Co. Institute found that platform work was not the primary source of income for most families that participated in the platform economy between 2012 and 2018. Yet the study also found that platform work can account for roughly 20 percent of workers’ income during months when they are actively engaged in platform work. Also, while the number of platform workers who provide transportation has increased, their average monthly earnings fell steadily between 2014 and 2018. In another study, IRS used tax data to attempt to estimate the population of platform workers, which it termed “gig workers.” Similar to BLS, IRS attempted to count workers who use websites and mobile apps to connect with customers to obtain short-term work and to receive payment through the company that owns the website or mobile app. IRS identified the names and Employer Identification Numbers of companies providing labor services and matched Forms 1099-MISC and 1099-K issued by the companies to workers. IRS estimated the number of workers who received a Form 1099-K or 1099-MISC from known labor platform companies increased from roughly 20,000 in 2012 to 1.9 million in 2016, then dropped to 1.3 million in 2017. However, IRS’s methodology had limitations that make the study an uncertain source for determining the number of platform workers. First, IRS cannot easily identify the universe of platform companies because it is rapidly changing and not well documented. Second, according to IRS, an unknown number of platform workers receive neither a Form 1099- MISC nor a Form 1099-K, for reasons we discuss later in the report. In sum, the data IRS receives do not allow it to accurately count the number of platform workers or determine their tax reporting behaviors. Given the limited data IRS has on the platform workforce, IRS cannot be assured that it knows enough about the size, characteristics, and behaviors of this workforce to better understand how to help workers comply with tax obligations. According to federal standards for internal control, managers should collect and use quality information to achieve an entity’s objectives. IRS officials we interviewed acknowledged that identifying platform workers is challenging given limitations within IRS’s data. IRS does not have a straightforward way for taxpayers to indicate on their tax forms whether they performed platform work. For example, Schedule C—the form that independent contractors are to use to report profit or loss—includes a series of yes/no checkboxes near the top of the form; however, it does not have a checkbox to indicate whether any reported income or expenses are from platform work. Similarly, Schedule C has a six-digit code to indicate the type of business or activity conducted, but it does not include a code for all platform workers. Although this information would be self-reported and imperfect, it would provide more information than IRS has on platform workers. Obtaining more information on the number of platform workers and their tax filing and reporting behaviors could help IRS develop ways to assist platform workers in complying with their tax obligations. For example, if IRS had more data on the types of deductions that certain types of workers were reporting, IRS would be better able to craft guidance on such deductions or do outreach to help those types of workers comply. Further, without changes to Schedule C, IRS cannot cross-check whether workers who self-identify as platform workers are also identified as platform workers on Forms 1099 filed by platform companies; a cross- check would enhance IRS’s understanding of the platform workforce. For example, if the cross-check shows that the number of workers who self- identify as platform workers significantly exceeds the number of Forms 1099 filed, IRS may have to work with platform companies to understand the filing shortfall. Conversely, if the number of filed Forms 1099 is significantly higher, IRS may need to enhance outreach efforts to make platform workers more aware of their tax obligations. We identified three areas of challenges for platform workers’ compliance with their tax obligations. First, according to our stakeholder interviews, platform workers may be less aware of their tax responsibilities than some independent contractors. Independent contractors typically have to advertise or seek referrals to gain customers, develop a network of peers, and learn about rules related to licensing or certifications. These activities can educate them about the basic responsibilities of being self- employed, including their tax obligations. In contrast, entry into platform work can be quick and workers may begin the activity without time to learn how their tax obligations differ from those of employees. For example, someone with a car, a valid driver’s license, and a smart phone can start working as a ride-hailing driver after they register with an app. Likewise, some platform workers may approach platform work as a hobby, an artistic endeavor, or something they do for a short time or in addition to another job. They may not realize that the company is treating them as an independent contractor, that the platform income may affect their taxes owed, and that they may need to track their expenses and make quarterly tax payments. As a second challenge, platform workers may have limited information about the payments they receive for their work. TPSOs are not required to file information returns to report earnings information to workers or IRS if the workers receive $20,000 or less in annual payments or have 200 or fewer transactions. Available tax data from tax year 2016 suggest that only around 30 percent of platform workers who were known to IRS had gross platform-related earnings higher than $5,000. Hence, most platform workers are likely not receiving an information return from the company. As a result, workers may not be aware that their income is taxable and IRS is less able to the check the workers’ tax compliance. A third challenge, according to stakeholders, is the burden associated with the steps platform workers must take to estimate, save, and remit quarterly tax payments. Because earnings of some platform workers may be low and earnings and expenses may fluctuate, they can have difficulty estimating their taxes owed and setting aside money to pay the taxes. They may also find it time consuming or costly to track expenses and determine profit. To the extent these burdens and difficulties confuse workers, they are less likely to pay the estimated tax payments fully and on time and may incur a penalty as a result. One stakeholder from a large tax preparation firm raised the concern that if the penalty or amount owed is more than workers can afford, they are at risk of falling into a cycle of noncompliance. In response to a February 2019 recommendation by the Treasury Inspector General for Tax Administration, IRS formed a team of officials from across the agency to develop a strategy to promote compliance among platform workers. The strategy focuses on two challenges for platform workers: (1) raising awareness about federal tax obligations, and (2) easing burden by identifying improvements to instructions, guidance, or forms that platform workers are likely to use. As part of the strategy, IRS developed a communications plan that includes a redesign of IRS web pages for platform workers and companies, outreach activities to the workers and various stakeholders, and a review of IRS guidance and related forms or instructions. The IRS Communications & Liaison office is managing the communications plan to educate platform workers and companies about their tax and reporting responsibilities. A key component of the plan is to redesign IRS web pages that provide tax information for platform workers and companies. IRS changed the name to the Gig Economy Tax Center (previously it was called the Sharing Economy Tax Center) and launched it in January 2020 before the start of the filing season (see fig. 3). As part of this effort, IRS is working to make web pages more user friendly for platform workers as well as platform companies. IRS’s Online Services (OLS) conducted research on the expectations, behaviors, motivations, and needs of self-employed individuals. According to OLS officials, IRS used insights about platform workers from the research coupled with user testing of the new web pages to inform the redesign effort. IRS’s steps to redesign the web pages align with selected leading practices for improving the online user experience, such as taking steps to make the pages useful and findable (see table 1). It is too soon to know whether the intended users find the new pages useful, usable, findable, and credible, as envisioned by the practices. IRS’s communications plan included outreach activities to raise awareness. For example, to publicize the redesigned web pages, the plan envisions targeting audiences such as platform workers and companies, news media, national tax publications, tax professionals, government agencies, IRS employees, and other groups. IRS created and distributed products such as a national news release about the new web pages and articles for newsletters and other products for tax professionals, small businesses, and payroll providers. IRS created a one-page electronic brochure to inform platform workers about their tax obligations and available tools to help them (see fig. 4). IRS intends to encourage platform companies to provide the brochure to workers. IRS is also using social media such as Twitter and Instagram to target platform workers and direct them to the redesigned web pages through posts such as those seen in figure 5. To assess the effectiveness of its efforts to increase awareness, including whether outreach efforts are driving people to the new web pages, IRS plans to compare data analytics for the redesigned site with the previous site. OLS plans to analyze changes in website traffic volume and taxpayer behavior—such as click patterns and how long users stay on the page—3 months after the launch of the redesigned pages and again 6 months later, and make changes as warranted. IRS’s office of Tax Outreach, Partnership, and Education (TOPE) is supporting awareness efforts by building relationships with partners, such as platform companies and organizations that advise platform workers. The communications plan provides a high-level description of efforts to engage with partners. Goals are to increase partner use of IRS social media, develop more industry-specific content, and increase TOPE involvement in virtual and face-to-face partner events and conferences, among others. IRS also plans to leverage these new partnerships to solicit feedback on its communication efforts to ensure they meet the needs of platform workers and companies. However, the communications plan lacks details about how IRS will monitor feedback from stakeholders. Specifically, IRS does not have a process for documenting and evaluating feedback based on the various communications efforts and products tailored for platform workers. According to federal standards for internal control, management should establish two-way reporting of quality information to achieve its objectives. Management should monitor activities and periodically evaluate the quality of information received to achieve its objective. According to IRS officials, they do not have the time or staff to document all feedback received. Further, they said the value of responding quickly to stakeholder comments supersedes the value of documenting and evaluating feedback. Given limited resources, IRS could choose to do something simple like creating a spreadsheet that captures feedback received, such as stakeholder emails, and document whether it led to changes. IRS has dedicated time and resources to better understand platform workers’ tax- related challenges and has undertaken multifaceted communication efforts to address them. Without a process to monitor feedback, IRS may miss opportunities to find better ways to drive platform workers toward the redesigned web pages, to ensure the redesigned web pages are meeting platform workers’ needs, or to strengthen communication efforts to enhance tax compliance. These opportunities could become more important as the platform worker population grows and evolves. IRS identified changes to forms, guidance, and other publications that could make it easier for platform workers to understand which forms apply to them. According to IRS officials, the Gig Strategy Team reviewed tax forms, publications, instructions, and training materials relevant to independent contractors, including platform workers. The team concluded that it did not need to create new forms or publications, although it identified 10 forms, instructions, guidelines, and publications that could be updated to be more helpful to platform workers. For example, the team suggested that IRS make several changes to update the instructions for Form 1040, such as adding a reference to Form 1099-K and clarifying that platform economy work can be a trade or business. The team also suggested that IRS revise the Form 1099-K instructions for payees to indicate how this form could clarify information on business gross receipts. IRS did not approve the gig strategy team’s requested changes to clarify the instructions for Forms 1040 and 1099-K for 2020. However, IRS is considering making these changes in 2021. According to IRS officials, they want to ensure these changes complement each other and the guidance is clear for taxpayers. According to federal standards for internal controls, management should communicate both internally and externally information necessary to achieve the entity’s objectives. Moreover, research based on behavioral insights has shown that introducing small interventions or removing small obstacles can significantly improve effectiveness.For example, interventions should be attractive (to draw people in) and easy (use simple, plain language). Leading practices suggest that people respond to information that is relevant to them. One way to do this is to include examples that help people recognize when information is relevant. For example, adding a brief reference to the gig or platform economy to the Form-1099 K payee instructions, along with a simple description of what the taxpayer should do, such as “show total payments from a company that facilitated a match, transaction, and payment for goods or services,” could help platform workers understand that the forms apply to them. By not including plain language for the Form 1099-K and 1040 instructions, platform workers are less likely to recognize which information applies to them. Simplifying one aspect of the tax system for platform workers by making the forms easier to understand could lead to enhanced tax compliance. According to our prior work, a good tax system should be equitable, economically efficient, and simple, transparent, and administrable. However, the challenges we have discussed—such as the lack of awareness and information—are complicating the tax system for platform workers and limiting IRS’s ability to more effectively collect taxes. We identified nine options from our literature review and stakeholder interviews to address these challenges and enhance tax compliance for platform workers (see table 2). For each option, we analyzed available data on the potential design and tradeoffs, including the potential costs and benefits. We discuss two options related to reporting and two options related to withholding in the sections below. For five options related to simplifying the reporting and filing processes, we found that the available data and research did not support a full assessment of potential pros and cons. We discuss those five options in appendix II. Many platform workers are not receiving a Form 1099 on their self- employment income, and therefore may be unaware of their tax reporting obligations. It is difficult to estimate how many workers are not receiving these forms because of limitations in available data. However, IRS found that the number of workers receiving Forms 1099-K or 1099-MISC from known labor platform companies dropped more than 30 percent from 2016 to 2017. Such a decline in information reporting for platform workers can be attributed to three factors. First, reporting thresholds for TPSOs were set at a high level to prevent unnecessary information reporting to IRS. When Congress enacted the Housing and Economic Recovery Act of 2008, TPSOs applied mainly to online marketplaces for goods, like eBay, and companies that facilitated payments, like PayPal. The act required information reporting by TPSOs for payments made through their payment networks only if those payments exceed both $20,000 and 200 transactions annually. Individuals who were generally not engaged in business or not producing a profit, such as casual sellers of goods, would likely fall below these thresholds. Second, IRS created a “tie-breaker rule” to avoid duplicative reporting which also led to no Form 1099 being filed in some cases. Businesses, including TPSOs, are in general required to report certain transactions on Form 1099-MISC, while TPSOs are also required to report certain transactions on Form 1099-K. IRS instituted a rule to break the “tie” that exists when a TPSO is required to report the same transactions on both Forms 1099-MISC and 1099-K. Specifically, the rule states that payments made through a TPSO’s payment network are not required to be reported on Form 1099-MISC, subject to an annual $600 threshold. Instead, TPSOs’ payments are required to be reported on Form 1099-K, subject to the annual $20,000 and 200 transactions thresholds. Third, since 2008, new platform companies have emerged that fit the TPSO designation but that facilitate payments for workers providing services rather than goods. These payments often fall below the combined annual $20,000 and 200 transaction thresholds. Options to increase information reporting for platform workers would help raise awareness about their tax obligations while lowering their burden. As a result, platform workers would be more likely to comply with tax obligations. The following sections discuss our analysis of two options. The “tie-breaker rule” could be amended to reverse the rule for payments made through a TPSO’s payment network. This would result in more reporting on Forms 1099-MISC under Internal Revenue Code (IRC) Section 6041, rather than on Forms 1099-K under IRC Section 6050W. Given the much lower threshold for the Form 1099-MISC versus the thresholds for the Form 1099-K, more workers would receive reports on their payments and IRS would receive more reports, too. Information reporting under the regulations related to IRC Section 6041 would generally increase for TPSOs that facilitate the provision of services. If the TPSO makes a payment for a service on behalf of another and performs management or oversight for the payment, then that TPSO would generally be responsible for filing a Form 1099-MISC, subject to the annual $600 threshold. Also, TPSOs that facilitate payments of rentals and also provide management or oversight of those payments would likely be subject to the $600 threshold. Reporting would remain unchanged for some types of companies. Because payments for goods are generally not reportable under IRC Section 6041, reporting would not change for online marketplaces that facilitate the sale of goods only. Because of the management and oversight requirements of IRC Section 6041, TPSOs that only facilitate payments would also generally not be affected by a change to the tie- breaker rule (see fig. 6). IRS Counsel has discussed amending the tiebreaker rule, but has yet to take action. According to IRS, Counsel has had to address other priorities, such as reviewing rules and publishing guidance related to Public Law 115-97, commonly known as the Tax Cuts and Jobs Act of 2017. Without amending this rule, the lack of information-return reporting for many workers complicates their efforts to comply with their tax obligations and IRS’s ability to ensure that these workers are correctly reporting their income. When workers do not receive forms related to their self-employment income, they have more difficulty determining how much money they made for computing taxable income. IRS analysis indicates that taxpayers are more likely to report their income that is subject to some information reporting (an estimated 83 percent compliance) compared to little or no information reporting (an estimated 45 percent). Sending more information returns may add costs for some companies; however, stakeholders had differing views on how significant those costs would be. In addition to amending the tie-breaker rule, stakeholders, as well as our literature review, discussed changing the various reporting thresholds. IRS and the Department of the Treasury (Treasury) have unique access to tax data and could analyze whether the 1099-K and 1099-MISC reporting thresholds are set at levels appropriate for tax administration. The 1099-MISC threshold was enacted in 1954 and the 1099-K reporting threshold was enacted in 2008; neither reflect the development of the platform economy. These changes include the emergence of companies that facilitate workers’ earning income by renting their houses or by providing transportation services, among many other activities. Informed stakeholders suggested that the current 1099-K thresholds may be appropriate for some TPSOs, such as online marketplaces that facilitate the sale of goods or companies with a primary function of facilitating payments. Different thresholds may be more appropriate for other activities facilitated by platform companies, such as renting houses or other assets. For example, a lower dollar threshold and no transaction threshold may be appropriate for home rentals because substantial income could be generated from even one transaction and provider costs may be limited. However, some service providers incur significant costs, such as drivers who must pay to own and operate a vehicle. Some stakeholders suggested that a threshold of $600 was more appropriate than a threshold of $20,000 and 200 transactions. Even so, some stakeholders suggested that typical costs to service providers may justify a threshold higher than the $600 threshold set in 1954. We found no available analysis of tax or other data showing whether the reporting thresholds for Forms 1099-MISC and 1099-K are appropriate for today’s economy or what the thresholds should be. The NEW GIG Act of 2019, which was introduced in Congress in March 2019, would raise the Form 1099-MISC reporting thresholds from $600 to $1,000. It would also lower the Form 1099-K reporting threshold from $20,000 and 200 transactions to $5,000 or 50 transactions for TPSOs making payments to those primarily engaged in the sale of goods, among other actions. Aligning reporting thresholds with today’s economy would support tax administration for IRS, as IRS studies have shown that information reporting increases tax compliance. It would also help reduce compliance burden for workers, since they would have clear information on their earnings. Platform workers can be burdened in estimating, saving, and remitting quarterly payments for income, Social Security, and Medicare taxes. Companies are not allowed to withhold and remit these taxes for platform workers who are treated as independent contractors and who want to participate in voluntary withholding—except where backup withholding is required. Voluntary tax withholding that satisfies quarterly tax payment requirements could reduce the workers’ burden and promote their tax compliance. We identified two actions that IRS could take on voluntary tax withholding. IRS could work with Treasury to implement voluntary withholding on payments to independent contractors for services. Voluntary withholding would be an option where it would be voluntary for companies to participate; for those companies that choose to participate, it would be voluntary for the independent contractor to participate. According to IRS Counsel, IRS has the statutory authority to take this action if the Secretary of the Treasury finds that withholding would be appropriate and would improve tax administration, and if the company and independent contractor agree to such withholding. According to IRS officials, IRS and Treasury have not determined whether they intend to pursue such an action. Voluntary withholding could be implemented by adjusting existing procedures and using the existing requirements for paying estimated taxes each quarter as a foundation. For example, to enable companies to solicit workers’ choices on when and how much to withhold, IRS could create a form similar to the Employee’s Withholding Certificate, Form W- 4, which employees complete so that employers can withhold the correct estimated federal income tax from employees’ wages. Workers could choose to vary the amounts withheld and the frequency of withholding as long as they met the quarterly estimated tax requirements. Alternatively, IRS could create guidance similar to what exists for workers who opt-in to electronically receive tax forms from their companies. Companies could use the existing procedures for tax withholding on wages paid or for backup withholding to withhold and remit the taxes to IRS. Companies would need to develop other design features, such as how they inform workers about participation. If the withholding is voluntary for companies, any burden is limited to those companies that choose to participate. For workers who choose to participate, they would still have the burden of estimating the amount of taxes owed for each quarter, but their burden to save and remit those taxes could be reduced. If some workers find that withholding is not appropriate for them, they could continue to use the existing system for quarterly tax payments. As long as workers choose withholding amounts that satisfy these quarterly estimated tax requirements, IRS would not need to design default percentages or dollar amounts to be withheld. While withholding could potentially help all types of independent contractors, voluntary withholding would specifically address some challenges that platform workers face. For example, platform workers have had challenges meeting the quarterly estimated tax requirements when their work is part time or sporadic. Giving workers and the companies the flexibility to structure tax withholding would better ensure that these challenges are addressed and the taxes are paid. Furthermore, this voluntary withholding regime would help those workers who choose to participate set aside a sufficient amount of money for taxes throughout the year, reducing the likelihood of an unanticipated large tax bill and tax penalty at the end of the year. Tax withholding plays a critical role in supporting voluntary compliance. IRS’s analysis indicates that withholding helps induce higher compliance in reporting income; higher compliance can help to reduce the tax gap—the difference between tax amounts the taxpayers should have paid and what they paid voluntarily and on time. Another option we reviewed involved Congress requiring companies to offer the withholding of taxes for remittance to IRS. That is, the companies must offer withholding; it would still be voluntary for the independent contractor to participate. Given that IRS does not currently allow for voluntary withholding on payments to independent contractors, we could not determine the need for such a requirement. If IRS implements withholding that is voluntary for companies, it would have the opportunity to collect and assess information on the need for such a requirement. For example, IRS could determine whether workers who volunteer to participate in withholding are able to better meet their estimated quarterly tax payments. IRS also could determine whether companies are offering withholding. If IRS finds that voluntary withholding bolsters compliance while reducing burden for workers, but that platform companies are not offering it, IRS could recommend that Congress take action. In considering which platform companies could be required to offer voluntary withholding, a starting point would be TPSOs. Focusing the requirement on TPSOs would likely be less burdensome than on other platform companies. TPSOs already settle payments to multiple parties; withholding and remitting taxes to IRS would only be one more type of payment that the TPSOs would process. This requirement would benefit platform workers who wished to participate in voluntary tax withholding and who work for TPSOs. More workers would have the option to have their taxes withheld. Because worker participation would remain voluntary, the need to determine default percentages or dollar amounts for withholding could be avoided as long as the worker met the quarterly requirement for paying estimated taxes. Some stakeholders suggested the option of mandatory withholding for platform workers to help them avoid a tax penalty. However, mandatory withholding could be burdensome for workers who need to balance cash flow and other spending priorities. Additionally, if participation in withholding is mandated, IRS would need to determine minimum withholding amounts. Because different businesses involve different typical expenses, it may be difficult to create an appropriate withholding rate across all business types. Creating multiple withholding rates for different types of businesses instead could be complicated and burdensome. The platform economy is still relatively new but available evidence suggests that it is growing and presenting tax-related challenges for both workers and IRS. While IRS has addressed some of these challenges, it could do more to promote voluntary compliance among platform workers by further raising awareness and easing taxpayer burden. IRS developed a communications plan to raise awareness among platform workers of their tax responsibilities; however, the plan lacks details about how IRS will monitor stakeholder feedback. Having a process for documenting and evaluating feedback would help assure that IRS’s communications efforts are addressing platform workers’ tax-related challenges, even as the population grows and evolves. Additionally, IRS could better understand the platform workforce if it had a straightforward way to collect data to identify platform workers. IRS has identified changes to forms, guidance, and other publications that could make it easier for platform workers to understand which forms apply to them. However, IRS has not yet taken actions such as adding plain language to instructions and publications that clearly indicate when a form applies to a platform worker. Making instructions and publications easier to understand by adding a simple description could help platform workers comply with their tax obligations. IRS could help ease taxpayer burden by taking steps to increase information reporting for these workers. Platform companies that act as TPSOs do not have to report income information on many workers because reporting thresholds are much higher than what most workers earn. Amending IRS rules to require such reporting at lower thresholds would provide workers with more information to comply with their tax responsibilities and would give IRS additional information to support enforcement efforts. Taxpayer burden could also be eased through a voluntary withholding program for platform workers. IRS and Treasury have the statutory authority to take such an action if the Secretary of the Treasury finds that withholding would be appropriate; however, IRS and Treasury have not determined whether they intend to pursue it. Voluntary withholding could help platform workers save and remit their taxes to IRS. This would address specific challenges platform workers face, such as low or fluctuating income. Once a voluntary withholding program is created, IRS would be able to assess its impacts and, if warranted, work with Treasury on a proposal to Congress to require TPSOs to offer voluntary withholding on payments for platform workers and other independent contractors. We are making the following seven recommendations to IRS: The Commissioner of IRS should change Schedule C or Form 1099-NEC so that taxpayers can identify if they received payment for platform work. (Recommendation 1) The Commissioner of IRS should develop a process for monitoring feedback on its communications efforts and products tailored for platform workers, which should include documenting and evaluating feedback. (Recommendation 2) The Commissioner of IRS should clarify the instructions and publications for Forms 1040 and 1099-K by adding plain language to clearly indicate to platform workers that the forms apply to them. (Recommendation 3) The Commissioner of IRS should work with the Secretary of the Treasury to amend the 6050W “tie-breaker rule” that applies to duplicative reporting requirements so that payments made through a TPSO’s third party payment network are reportable under Section 6041, rather than under Section 6050W. (Recommendation 4) The Commissioner of IRS should work with Treasury to determine what thresholds would be the most appropriate for payment information reporting and, if warranted, recommend that Congress adjust the thresholds. (Recommendation 5) The Commissioner of IRS should work with the Secretary of the Treasury to implement withholding that is voluntary for companies making payments for services to platform workers and other independent contractors who choose to participate. (Recommendation 6) The Commissioner of IRS should assess the impact of withholding that is voluntary for companies, once implemented, and if warranted, work with the Secretary of the Treasury on a proposal to Congress that would require TPSOs to offer tax withholding to platform workers and other independent contractors who choose to participate. (Recommendation 7) We provided a draft of this report to Treasury and IRS for review and comment. IRS provided written comments, which are reproduced in appendix III and summarized below. Of our seven recommendations, IRS agreed to implement or consider two recommendations, said it could not agree with two recommendations due to other priorities, disagreed with two, and said it could not implement a recommendation that flowed from another recommendation. IRS also provided technical comments, which we incorporated as appropriate. IRS agreed to refine its process for monitoring feedback on its communication efforts and products but was silent on documenting and evaluating the feedback (Recommendation 2). The value of monitoring feedback will be minimized unless IRS has a process for documenting and evaluating it over time. In addition, IRS agreed to consider clarifying the language in the instructions and publications for Forms 1040 and 1099-K (Recommendation 3). We revised this recommendation by deleting the word “examples” from the clarifications to make. IRS stated that it could not agree to work with Treasury to (a) amend the 6050W “tie-breaker rule” and (b) determine appropriate thresholds for payment information reporting and, if warranted, recommend that Congress adjust the thresholds (Recommendations 4 and 5). For both recommendations, IRS said it cannot commit to implementation dates because of higher priority guidance projects, especially in light of the many new tax provisions enacted by Congress. We acknowledge IRS’s need to prioritize guidance projects, but we do not understand why it does not agree to address problems that will persist into the future absent corrective actions or why IRS believes it cannot agree unless it commits now to a future implementation date. In fact, IRS stated that it has a long- term guidance project under development for amending the rule to clarify definitions. IRS said it would consider our recommendation on amending the rule as it develops its 2020-2021 plan for guidance priorities. As for our recommendation on determining reporting thresholds, IRS said it is willing to meet with Treasury officials to discuss the need to analyze the current thresholds. IRS disagreed with our recommendation to change Schedule C or Form 1099-NEC so that taxpayers can identify if they received payments for platform work (Recommendation 1). IRS said it has no evidence that platform workers pose a greater compliance risk and that the platform worker designation, by itself, would not be a selection factor for compliance actions. We found that IRS has not measured this risk, in part, because it cannot easily identify the workers. Further, we did not intend that IRS use the data to initiate compliance against the workers; our report discussed using the data to identify workers and their reporting behaviors to improve services and voluntary compliance. IRS concluded that the additional taxpayer burden and its costs were not warranted at this time. IRS did not identify the level of burden or costs, and the burden from checking a box on Schedule C does not seem high. We asked about the costs to revise the forms to add a checkbox but IRS did not provide them. As for IRS’s costs to capture the data, we note that IRS’s costs to transcribe the data would be zero for returns filed electronically. Individual taxpayers (including platform workers) electronically filed 138 million of 156 million (89 percent) tax returns—which includes the Schedule C— during 2019. If IRS intends to help these workers become more aware and reduce their taxpayer burdens to comply, identifying them and their tax behaviors would enhance those efforts. IRS disagreed with our recommendation to work with Treasury to implement withholding that is voluntary for companies making payments for services to workers who choose to participate (Recommendation 6). IRS said that its role is to administer tax law rather than propose tax policy changes. As we discuss in the report, IRS has the authority to take this action if the Secretary of the Treasury agrees that the action would improve tax administration and our recommendation focuses on working with Treasury officials. IRS also stated that it cannot commit to an implementation date for publishing guidance on a voluntary withholding program due to higher priorities, including implementing various COVID- 19 relief programs. As stated above, we do not understand why current higher priorities would prevent IRS from taking future corrective actions or why IRS believes it must commit to an implementation date at this time. We continue to believe that voluntary withholding has potential to improve compliance and reduce taxpayer burden Finally, IRS said it would not assess the impact of such voluntary withholding and thus not work with Treasury on a proposal to Congress that would require third party settlement organizations to withhold taxes for workers who choose to participate (Recommendation 7). IRS’s rationale was that it disagreed with the voluntary withholding recommendation. As stated above, we believe this proposal has potential to improve compliance and an assessment would help inform Congressional deliberations about additional statutory changes that could enhance tax compliance among platform workers. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will copies of this report to the relevant congressional committees, the Secretary of the Treasury, the Commissioner of IRS, and other interested parties. In addition, this report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff has any questions about this report, please contact me at (202) 512-9110 or mctiguej@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff members who made major contributions to this report are listed in appendix IV. You asked us to review issues related to platform workers and tax compliance. This report (1) examines what is known about the platform workforce and what can be done to improve the available Internal Revenue Service (IRS) data, (2) identifies challenges platform workers face complying with federal tax obligations, (3) assesses IRS actions to promote tax compliance among this population, and (4) assesses additional options to promote tax compliance. This sector and its workers are referred to by different names including sharing, on-demand, gig, and platform. We selected the term platform because it was the most comprehensive and relevant to online work arrangements. For example, neither sharing nor gig communicates that the work could be full time and for income. Platform worker also clearly communicates that workers are using an online app or other platform to find and secure work. While platform workforce does not have a standard definition, for the purposes of this report we defined it as workers who provide goods or services to customers through an online platform operated by a company that facilitates the match, transaction, and payment. To identify what is known about platform workers, we conducted a literature review of peer-reviewed material; trade and industry articles; government reports; conference papers; general news; association, nonprofit, and think tank publications; hearings and transcripts; and working papers that described the platform economy in the United States and related tax issues. We searched publication databases such as Proquest and Dialog for keywords like gig, contingent, sharing, or platform, and tax, IRS, or compliance to identify studies that were relevant to our research objectives. We also conducted semistructured interviews with selected stakeholders and reviewed documents provided by them to obtain information on and descriptions of platform workers. Through these interviews, we obtained stakeholder views on (1) the use of the term “platform workforce,” (2) its size and composition, (3) workers’ understanding of and compliance with tax obligations, and (4) federal, state, and private-sector-level efforts and policy proposals to help workers comply with their tax obligations. We conducted close to 30 interviews with knowledgeable individuals that we selected to represent varied areas of expertise and perspectives including Department of the Treasury and IRS officials; academics and other researchers; state government tax and revenue officials from the Massachusetts Department of Revenue, the California Franchise Tax Board, the Maryland Department of Labor & Regulations, and the Vermont Department of Taxation; private-sector and nonprofit tax preparers; tax software developers; and platform company representatives and workers. We interviewed representatives from four companies–Airbnb, Thumbtack, Etsy, and eBay; and two representatives from professional associations, Technet and Internet Association, that include platform companies among their members. Company representatives from Uber, Lyft, Taskrabbit, Upwork, Snag, and Postmates participated in the Technet group interview. We also interviewed representatives from organizations that work directly with platform workers such as the Freelancers Union, Center on Budget and Policy Priorities and Creating Assets, Savings and Hope Campaign of Maryland, National Association of Self-Employed, and the Independent Drivers Guild (IDG). IDG also set up 30-minute phone calls for us with five full-time drivers who use mobile apps to connect with customers. We were also invited to attend a panel from the National Academies of Science on the platform workforce. The panel included experts from the Aspen Institute, JP Morgan Chase, the Federal Reserve Board, the Census Bureau, and various think tanks and universities. We identified potential interviewees through a literature search, and recommendations from our initial interviews. We selected interviewees based on their relevance to the scope of our review. We also aimed for balance between those who could serve as a proxy specifically for low- income platform workers, and those who work with people across the income spectrum. Although the results of these interviews are not generalizable to the views of all stakeholders, they still provide important insight into and illustrative examples of the challenges platform workers face understanding and complying with federal tax obligations. To assess IRS actions to promote tax compliance, we reviewed IRS research on the platform workforce and other documents describing current and planned actions to identify taxpayers who are platform workers and to promote their tax compliance, including the Gig Economy Compliance Strategy. We also interviewed IRS officials from the Small Business/Self-Employed Division; Research, Applied Analytics, and Statistics Division; Communications & Liaison Office, including officials from the Offices of Online Services and Tax Outreach, Partnership, and Education; and the Office of Chief Counsel about their efforts to develop, implement, and assess the impact of the new strategy. We identified criteria for assessing elements of IRS’s strategy such as its research, communication materials, and evaluation plan. We based these criteria on Standards for Internal Control in the Federal Government and leading practices for designing web materials to improve the user experience. The relevant internal control principles focus on information and communication and monitoring. The leading practices posted at usability.gov describe principles for creating a meaningful and valuable online user experience. To assess options to promote tax compliance, we identified options from our literature review and our interviews of stakeholders and selected those that were commonly cited as potential solutions for the challenges. We then sent 39 stakeholders a list of the potential solutions to solicit their views on these options, including whether they supported or opposed the option. Twenty-eight stakeholders, including those from academia, the research sector, government, platform companies, tax preparation firms, and worker advocacy, provided their views. We analyzed their responses to help us identify strengths, weaknesses, and other considerations associated with each option. We also assessed each option using “criteria for a good tax system” described in our prior work. These criteria state that a good tax system should be equitable, economically efficient, and simple, transparent, and administrable. We focused on whether individual options would increase simplicity by reducing compliance burden; enhance transparency by helping taxpayers better understand their tax obligations; and improve tax administration by helping IRS more effectively collect taxes. We found that the available data and research for some options did not support a full assessment of the pros and cons that might be offered by the option. We describe these options in appendix II. We conducted this performance audit from February 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. One set of options we identified focused on simplifying aspects of the tax system to help platform workers comply with their tax obligations. For all of them, sufficient evidence did not exist on how the option would be designed or on the tradeoffs such as costs and benefits. Some of the options require legislative action, and other options could be costly for the Internal Revenue Service (IRS) to implement. Balancing the pros and cons is a policy decision for IRS and the Department of the Treasury (Treasury) or Congress to make and more research would be needed. The following discusses our analyses of these options including what information would be needed to design the option and compare the tradeoffs. According to some stakeholders, properly tracking and deducting expenses from gross income can be burdensome for self-employed workers, including platform workers. A common feature of platform work is using an asset, such as a car or house, for personal and business use; for example, a car owner may use his or her personal car for occasional ride-hailing work. Stakeholders and several articles from our review discussed how the rules for apportioning expenses for personal and business use and determining which ones are deductible can be complicated, especially for part-time or short-term drivers or renters. Some stakeholders as well as some articles mentioned that Congress could consider creating a standard business deduction that platform workers could use. Platform workers, especially those who may not have the necessary knowledge or experience, may not track all eligible business deductions and overpay income and self-employment taxes. They may improperly overstate their expense deductions to offset income, contributing to the tax gap. Creating a standard business deduction could reduce the complexity associated with the current system. However, different businesses involve different expenses, and it may be difficult to create an appropriate standard deduction across all business types. Creating multiple standard deductions for different types of businesses could be complicated and burdensome, and could ultimately reduce tax revenue if the standard deduction exceeds actual expenses for many platform workers. Additionally, a standard business deduction limited to only platform workers would be difficult to design because no regulatory or statutory definition of the term platform worker exists. Further, providing the standard deduction option only for platform workers would raise disparate treatment concerns. Given the limited data on the number and tax reporting characteristics of platform workers, we could not analyze the potential benefits and costs of creating a standard deduction. Because platform workers may be unaware of their federal tax obligations, Congress could consider requiring platform companies to inform workers who are classified as independent contractors about their tax obligations. While platform companies may incur some burden to inform platform workers, some could leverage existing processes to inform their workers. For example, when platform companies hire such workers, they could inform them about their tax obligations. Officials at platform companies we contacted raised concerns about the legal risks for them to develop the information provided to the workers. Platform companies would not necessarily need to develop the information provided. IRS developed a one-page publication that platform companies could provide to workers to inform them about tax obligations (see fig. 4). The publication is available in an electronic format for sharing with workers. However, no statutory or regulatory definition of a platform company exists. Without a definition, IRS could not equitably enforce the requirement. Alternatively, the requirement to provide the information could be limited to third party settlement organizations (TPSO) which have been defined. However, even a well-designed requirement could have limited impact on awareness. IRS is collaborating with partners such as platform companies to better inform workers about their tax obligations. To the extent that this collaboration as well as other IRS actions help make workers more aware, Congress may not need to consider requiring platform companies to inform their workers. Congress could require platform companies to provide available payment and expense information to platform workers who are independent contractors. Multiple stakeholders suggested that sharing this information is a best practice for raising awareness and helping workers comply with tax obligations. Some platform companies already provide a dashboard to workers showing total payments and other data, like miles driven. Platform companies have visibility over the full range of payments and some expenses for workers. While some companies would be providing data or providing online access to data that they already have, requiring information-sharing may increase their administrative burden to at least some extent. However, we did not find information on how much more burden would be created, which could vary based on the type of platform company. Further, given no statutory or regulatory definition of a platform company, a legal definition would need to be created or the requirement to provide the information could be limited to a subset of companies, such as TPSOs. Because several prominent platform companies already provide expense information, Congress may not need to add a legal obligation on the companies. We did not find sufficient evidence to show that this legal requirement would benefit most platform workers. Likewise, IRS considered adding information to its website encouraging platform companies to provide available payment and expense information their workers, but decided not to do so. IRS officials explained that many platform companies are already providing workers with this information. They also expressed concerns that platform workers might rely on this information, and not maintain their own records, which could cause challenges if platform companies do not track all available expenses or income. According to IRS officials, they will continue to update the website to include tips for platform companies to help workers comply with their tax obligations. The Internal Revenue Service Advisory Council (IRSAC) found that the definition of “gross amount” for reporting purposes on Form 1099-K includes items that are not part of the economic transaction between the purchaser and the seller or service provider. This includes refunds, fees, discounts, and other items. IRSAC recently recommended that the definition of “gross amount” for the purposes of reporting on Forms 1099- K should exclude these items, which are not taxable income for the platform worker, and include only payments to the workers for their service. IRS officials have identified some practical drawbacks to implementing this proposal. For example, determining what is includable and what is excludable from the “economic transaction” can vary from industry to industry and from taxpayer to taxpayer. Depending on the industry or activity, the gross transaction amount may not be itemized to specify what is included; for example, many payment settlement entities would not know whether sales tax is included in a transaction amount. IRS officials stated that it is not clear whether this proposal would provide any value to the worker or IRS given the variations. IRS could work with Treasury to allow for electronic delivery of Forms 1099 by default. The default delivery method is mail, and workers must opt-in to receive forms electronically. When a worker receives the form electronically, the worker may receive an email notification that the electronic form is available for download in a secure online account. Platform company officials with whom we spoke said sending forms by mail can be burdensome due to mailing costs and the costs of finding accurate mailing addresses. However, they generally have accurate email addresses because that is how they exchange information with platform workers. Allowing electronic delivery of Forms 1099 by default could reduce the burden for companies, while ensuring that more workers receive their forms. However, new rules would be needed to ensure that workers are easily able to opt-out of electronic delivery, and receive the forms on paper if they wish. We were not able to collect sufficient data on the cost or savings for companies and barriers for workers. James R. McTigue, Jr. (202) 512-9110, mctiguej@gao.gov Key contributors to this report include: Julie Anderson, Rob Gebhart, Sarah Gilliland, Gina Hoover, Jesse Mitchell, Ed Nannenhorn, Jessica Nierenberg, Robert Robinson, Eden Savino, Tom Short, AJ Stephens, and Peter Verchinski.", "summary": "Platform companies typically classify workers offering services as independent contractors and do not withhold taxes from their payments for remittance to IRS. GAO was asked to review issues related to platform workers and tax compliance. This report, among other things, examines (1) what is known about the platform workforce, and (2) options to promote compliance among its workers. GAO reviewed research on the U.S. platform economy and interviewed stakeholders on the tax-related challenges platform workers face; reviewed IRS documents; interviewed IRS officials; and assessed potential impacts of some options that could address platform worker tax-related challenges. The platform economy is an arrangement where workers offering goods or services connect with customers through an app or other online platform. Estimates of the population of platform workers lack certainty, but generally range from around 1.5 million to 2 million workers for recent years and suggest that the platform workforce may be growing. According to stakeholders, such as researchers and tax preparers, platform workers may not realize that a company is treating them as independent contractors rather than employees and that they must comply with different tax requirements. To help address this challenge, the Internal Revenue Service (IRS) developed a communications plan aimed at workers in the platform economy (which IRS calls the gig economy). The communications plan incorporates leading practices for redesigning web pages and improving the online user experience, but lacks a monitoring plan to help assure IRS's efforts address platform workers' tax challenges. GAO found that platform workers may not receive information on their earnings, creating compliance challenges for them and enforcement challenges for IRS. GAO identified actions that could promote compliance. For example, some platform companies only report total annual payments for workers if they exceed $20,000 and 200 transactions—an amount that exceeds the average gross pay from a single company for many platform workers. Amending this rule to lower the reporting thresholds would provide workers with more information to help them comply with their tax obligations. The change could also enhance IRS's ability to ensure that these workers are correctly reporting their income. Additionally, IRS could implement voluntary withholding on payments to independent contractors (including platform workers). IRS data indicate that tax withholding substantially increases the compliance rate. GAO is making seven recommendations to IRS, including actions to enhance its communications plan, increase information reporting for platform workers, and allow voluntary withholding. IRS agreed with the recommendation to enhance its communications plan. For four recommendations related to information reporting and voluntary withholding, IRS either disagreed or said it was unable to agree because it could not commit to an implementation date due to higher priorities. GAO continues to believe that all the recommendations are valid, as discussed in the report.", "document_type": "gao"}
{"report": "Airline customer service agents have a number of duties to assist passengers at the airport (see fig. 1). Customer service agents can check passengers into flights; handle and tag checked bags; and board and deplane passengers on the aircraft; in addition to assisting passengers when service failures occur, such as helping to locate a lost bag. At many airports and airlines, customer service agents are trained to work the ticket counter and the arrival and departure gates. In this role, airline customer service agents’ interactions with passengers can range from pleasant to routine to contentious. For example, if bad weather causes an airline to delay or cancel flights, harried passengers trying to make connecting flights or get to a destination may take their frustration out on a customer service agent. The following entities are responsible for helping to prevent or address passenger assaults: Airlines seek to provide a safe work environment for customer service agents. Among other things, airlines set policies and procedures instructing customer service agents how to handle and report incidents, in addition to how management should respond. Airport law enforcement responds to allegations of violence at airports and enforces state and local laws. According to airport law enforcement, when they respond to incidents, they generally capture information in police reports. Airport management, such as a security director, may be informed of alleged passenger assaults at the airport or support ensuing investigations and prosecutions. Prosecutors at the federal and state level decide whether to charge passengers for offenses that violate laws. No one federal agency is responsible for addressing passenger assaults against customer service agents at the airport. For example, FAA sets policies that airlines and their employees must adhere to for aviation safety, but TSA oversees the security of the nation’s civil aviation system. However, officials from both TSA and FAA told us their responsibilities for passenger assaults at airports are limited. In particular, FAA officials said their primary responsibility is for assaults onboard aircraft as opposed to at the airport. Similarly, TSA officials said they only get involved in assaults of airline customer service agents in the rare instances where incidents affect airport security. Within DOJ, FBI conducts investigations of incidents that are deemed to violate federal law, and federal prosecutors can decide whether to prosecute individuals for alleged incidents that are deemed to violate federal law. Limited data are available to determine the frequency or nature of passenger assaults at airports against airline customer service agents. We reviewed selected data from DOJ, DOT, FAA, FBI, and TSA and found that no dataset can isolate such passenger assaults. For example, while the FBI collects transportation crime data from law enforcement agencies about incidents that occur at air, bus, or train terminals— including information on the victim, offender, and location of the crime— the data cannot isolate passenger assaults against airline customer service agents. While representatives from selected airport law enforcement agencies and airlines we interviewed said they collect information related to passenger assaults for their respective airports or airlines, these data were generally unavailable. In particular, representatives from all six selected airport law enforcement agencies we interviewed said providing data on passenger assaults against airline customer service agents would require manually reviewing all police reports. Results from one selected airport law enforcement agency that had manually reviewed its data for 2018 found that of the 237 assistance calls it received for incidents between customer service agents and passengers, law enforcement completed an incident report for 12 of these calls, and referred two reports to state prosecutors. Representatives from five of the six selected airlines declined to share data with us, saying data were not readily available, or were business proprietary, or business sensitive. Representatives from the remaining airline provided us with data from the third and fourth quarters of 2018; this data indicated that incidents between passengers and customer service agents generally remained constant, with an average of approximately 1.2 disruptive passengers per 1,000 passenger boardings. In the absence of available data, we surveyed a non-generalizable sample of 104 randomly selected customer service agents to understand their experiences performing their jobs over the last year. According to these 104 customer service agents, almost all (96) reported experiencing verbal harassment, such as passengers yelling, cursing, or being argumentative (see fig. 2). Almost half (46) reported experiencing verbal threats, such as passengers threatening to harm the customer service agent. Twenty-two customer service agents reported that a passenger attempted to physically assault them by, for example, attempting to push them. Fewer (12) customer service agents said that passengers actually physically assaulted them. We also found that about one-third (34) of surveyed customer service agents said they experienced “other types of harmful actions,” which agents said included passengers destroying property, taking video of agents, grabbing agents’ identification badges, and stalking agents after work. Stakeholders we interviewed from selected airports, airport law enforcement, and airlines generally agreed that passengers can be verbally disruptive but that physical assaults are less frequent. More specifically, of these 17 stakeholders, most (13) agreed that disruptive passenger behavior is frequent. Most (11) also agreed that physical assaults occur less frequently than verbal threats. Nevertheless, while representatives from two selected unions did not have data on such actions, they emphasized to us that the customer service agents they represent face difficult working conditions. The union representatives also stated that passenger assaults, including verbal threats and physical assaults, are becoming more common. Further, three of the nine stakeholders who provided a perspective said that incidents against customer service agents are increasing. For example, representatives from one airline we interviewed said that over the past 5 years, they have observed an increase in both the frequency and severity of passenger assaults, in addition to other disruptive behavior. A number of factors may contribute to passenger assaults. Selected stakeholders, including those from airlines, airports, airport law enforcement, and other industry associations most commonly cited (24) alcohol consumption at the airport or drug use as a contributing factor. For example, according to representatives from one law enforcement agency, when customer service agents deny boarding to intoxicated passengers, passengers can become verbally or physically aggressive toward customer service agents. Other stakeholders told us that passengers increasingly have more opportunities to consume alcohol while waiting for their flights, thereby increasing alcohol-related incidents. For example, representatives from one airport noted that tablets at the boarding area allow passengers to place orders for alcohol while seated at the gate. Seventeen selected stakeholders we interviewed also told us that airlines’ business practices, such as charging fees for checked and carry-on baggage or policies around delays and cancellations might aggravate or surprise passengers and lead them to be aggressive toward customer service agents. Some stakeholders (10) also said that other factors, such as long lines and large crowds in the airport can increase passengers’ stress levels. Moreover, according to some stakeholders, service failures—such as flight delays, cancellations, or lost baggage—can exacerbate these stressors. Of the 61 surveyed customer service agents who reported experiencing verbal threats, attempted physical assaults, actual physical assaults, or other harmful actions, most (45) said these incidents negatively affected their overall well-being. Similarly, selected union representatives we interviewed also said that these incidents can increase stress and anxiety for customer service agents. Almost all customer service agents (56 of 61) who stated in our survey that they experienced passenger conduct amounting to more than harassment said they reported the conduct to someone. Specifically, 46 customer service agents stated that they contacted their immediate airline manager; 28 stated that they contacted airport law enforcement; and 6 stated that they contacted airport staff or other entities. These actions described by customer service agents we surveyed generally aligned with selected airlines’ procedures for handling passenger assaults. Specifically, representatives from five selected airlines told us that while their respective airline’s policy generally calls for agents to contact management first, agents can also contact airport law enforcement if they feel like their safety is threatened. However, representatives from two selected unions told us that airline managers are sometimes hesitant to inform law enforcement about incidents—or have their agents contact law enforcement—or to elevate incidents internally. According to one union representative, airlines prefer to keep such incidents internal and emphasize providing on-time service to their passengers. Contacting law enforcement could make this difficult to achieve, so when disruptive passenger behavior occurs, airlines may be inclined to allow the passenger onboard the aircraft instead of contacting law enforcement. Of the 56 customer service agents who stated they reported the passenger conduct, over half (33) said that, to their knowledge, representatives from airlines, law enforcement, or airports took action in response. According to our survey results, these representatives generally took a range of actions, including but not limited to, requesting that a passenger stop the disruptive behavior, completing an airline or police report, denying a passenger boarding, or arresting a passenger. Representatives most commonly removed passengers from an area or denied passengers from boarding (18); diffused the situation (7); or arrested the passenger (4). Twenty-six customer service agents said that no action was taken in response to the incident, which left some to not feel supported by airline management. Moreover, according to representatives from one union, in some instances, customer service agents feel that if airline management provides passengers with travel benefits, such as seat upgrades or airline miles, to diffuse these types of situations, it can appear to be condoning or rewarding any passenger misbehavior. The FAA Reauthorization Act of 2018 requires airlines to develop and submit employee assault-prevention and response plans to FAA by January 2019. In these plans, airlines are required to document: reporting protocols for airline customer service agents who have been the victim of a verbal or physical assault; protocols for notifying law enforcement after an incident of verbal or physical assault committed against an airline customer service agent; protocols for informing federal law enforcement about violations of federal law that prohibits interference with security screening personnel; protocols for ensuring that a passenger involved in a violent incident with an airline customer service agent is not allowed to move through airport security or board an aircraft until appropriate law enforcement has an opportunity to assess the incident and take appropriate action; and protocols for informing passengers of federal laws protecting federal, airport, and airline employees who have security duties within an airport. In March 2019, FAA officials said they had not received employee assault-prevention and response plans from all of the 49 U.S. airlines that were required to submit such plans. However, at that time, officials also said they were not concerned about any delays because they believed airlines already have internal policies and procedures for handling these types of incidents. Nevertheless, FAA officials told us they intended to issue a reminder to the airlines. Of the six selected airlines we interviewed, representatives from two airlines said they had submitted their plans to FAA, and representatives from the remaining four airlines said their plans were in development. Further, when we asked airlines to describe their policies for handling assaults, some of the policies that representatives described aligned to some requirements in the Act for the plans. For example, as discussed previously, all six selected airlines told us they had policies for how customer service agents or managers should notify airport law enforcement when assaults occur. Moreover, representatives from all six airlines also described reports that that customer service agents and employees complete when such incidents occur. In July 2019, FAA issued a notification to airlines, reminding them to develop and submit their plans. FAA officials attributed delays in following up with airlines to the government shutdown in early 2019 and multiple competing requirements in the Act. FAA officials also said they were initially hesitant to issue a notification around these plans, since the agency has a limited role and does not promulgate requirements for the training or oversight of customer service agents. Nevertheless, FAA officials said they plan to continue to follow up with the airlines as needed to collect the remaining plans. All selected stakeholders we interviewed representing airlines, airports, airport law enforcement, and prosecutors (23 of 23) who provided a perspective said that current state and local laws sufficiently deter and address passenger assaults. We spoke with seven selected state prosecutors who told us that, among other offenses, they can charge passengers for actions against customer service agents with assault; battery (e.g., intentional causing of bodily harm); disorderly conduct (i.e., acts that are of a nature to outrage the sense of public decency, or affect the peace and quiet of persons who may witness them, or engaging in brawling or fighting); and trespassing. According to these prosecutors, they typically charge passengers for assaults as misdemeanors, which one prosecutor told us generally does not result in passengers’ serving any jail time. While four selected state prosecutors who regularly handle misdemeanor prosecutions did not have data isolating these crimes, three recalled charging passengers for assaults against customer service agents. For example, a representative from one prosecutor’s office estimated that, over the last 5 years, law enforcement had referred 25 to 30 of these incidents to his office and that his office had prosecuted six or seven of these cases. In determining whether to pursue a case, five prosecutors we interviewed told us they weigh a number of factors, such as whether the customer service agent is willing to file charges; whether law enforcement observed the assault; and whether witnesses are available to testify. Nonetheless, according to prosecutors we interviewed, crimes committed at airports present unique challenges. More specifically, according to one prosecutor we spoke with, the transitory nature of airports makes it difficult to get witnesses to testify at a trial, because they are often passing through the airport en route to another destination. Four selected prosecutors also told us that passenger assaults might be charged as felonies if, for example, the crime involves the use of a deadly weapon or causes serious physical injury to the victim. However, these prosecutors told us such instances are infrequent and incidents between passengers and customer service agents rarely rise to the level of severity of a felony charge. To that end, none of the three prosecutors we interviewed who typically prosecute felony cases could remember charging a passenger for an assault of a customer service agent within the last year. Nevertheless, some selected stakeholders told us opportunities exist to strengthen penalties for passenger assaults. More broadly, a few stakeholders that we interviewed—including one airline, one prosecutor, and one union—suggested opportunities exist to pursue harsher penalties. According to selected stakeholders, this could be achieved by, for example, prosecuting passenger assaults as felonies, prosecuting these incidents at the federal level, or seeking a legislative change to classify airline customer service agents as a protected class. For example, under Florida statute, an alleged battery against certain specified protected classes, including elected officials and teachers, are automatically reclassified from a first degree misdemeanor to a third degree felony charge, resulting in potentially harsher penalties. Most selected stakeholders we interviewed who provided a perspective said that their current resources sufficiently deter and address passenger assaults. Specifically, of the 20 selected stakeholders who provided a perspective, 15 said that current resources are sufficient and did not identify other resources that could improve their ability to address or mitigate passenger assaults. The remaining five stakeholders would like to see additional resources directed toward airport’s law enforcement agencies. In particular, four selected stakeholders said they believe that increasing the number and presence of law enforcement in airports would help deter or address passenger assaults. Representatives from one airline told us they hired private security officers to monitor ticketing and baggage areas at the airport to increase their security posture. While the purpose is not to address passenger assaults, representatives told us that these officers can respond to such assaults. The remaining stakeholder suggested law enforcement could receive additional training to improve responses when passenger assaults occur. Some of the selected stakeholders we interviewed who did not identify gaps in resources nonetheless offered suggestions to further deter or mitigate passenger assaults, including: Provide additional training for customer service agents. Three stakeholders told us customer service agents should receive additional training on conflict de-escalation. Increase information sharing and reporting. Three selected stakeholders said that information sharing could be improved among relevant stakeholders—including airlines and airport law enforcement. For example, representatives from one airline said they have limited insight into the outcomes of passenger assaults unless they contact airport law enforcement or prosecutors. Two selected union representatives said that having better data on these incidents could be beneficial to understand the scope of the problem. Increase public education and support for customer service agents. Representatives from two unions would like to see (1) signage at airports saying that assaults by passengers are subject to prosecution, and (2) airlines provide additional support to customer service agents, in the form of legal assistance or time off, to press charges against passengers alleged to have committed such assaults. Moving forward, the FAA Reauthorization Act of 2018 requires airlines to provide initial and recurrent training for all employees on, among other things, de-escalating hostile situations, and, as previously noted, the reporting protocols for these incidents. Providing such training and having additional reporting protocols could provide customer service agents with additional tools for diffusing these incidents and standardize how airlines respond to these incidents, respectively. We provided a draft of this report to DHS, DOJ, and DOT for review and comment. DOJ provided technical comments, which we incorporated as appropriate. DHS and DOT did not have any comments. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, the Attorney General, the Secretary of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff any have questions about this report, please contact me at (202) 512-2834 or VonahA@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In the past year, how many times have experienced the following incidents: Passenger verbally harassed you Passenger verbally threatened you (i.e., said they would do something to you specifically) Passenger attempted to physically assault you (tried to hurt you) Passenger committed other harmful action (please describe) 3. How, if at all, have these incidents affected your overall well-being? a. No effect b. Slightly negative effect c. Very negative effect 4. Now thinking about the most severe incident you have experienced in the past year, which of the following airport officials, if any, did you contact about this incident? a. Immediate airline manager b. Airport law enforcement c. Airport staff d. Other–Please identify ______________________________ e. None 5. Did any airport or airline officials take action because of your most severe incident in the past year? a. No b. Don’t know c. Yes. Please describe the action that was taken. 6. How, if at all, could airlines support customer-service representatives when these incidents happen? Andrew Von Ah, (202) 512-2834 or VonahA@gao.gov. In addition to the individual named above, other key contributors to this report were Jonathan Carver, Assistant Director; Melissa Swearingen, Analyst-in-Charge; Emily Flores; Clara Goldrich; Geoffrey Hamilton; Delwen Jones; Dawn Locke; Malika Rice; Kelly Rubin; and Amy Suntoke.", "summary": "Recent media reports have detailed incidents at airports where passengers have acted disruptively or violently toward airline customer service agents, who assist passengers checking into their flights and boarding aircraft, among other things. While state and local laws generally prohibit these types of actions, some stakeholders have raised questions about these agents' safety. The FAA Reauthorization Act of 2018 included a provision that GAO examine passenger violence against airline customer service agents at airports. This report examines (1) what is known about assaults by passengers against customer service agents and (2) stakeholders' perspectives on the sufficiency of state and local laws and resources to deter and address such incidents. GAO interviewed and reviewed available information from a non-generalizable sample of representatives from five large airports and six large airlines. GAO also interviewed six airport law enforcement agencies, and seven prosecutors' offices. Further, GAO reviewed documents and interviewed two unions representing customer service agents and five federal agencies with airport safety or security responsibilities. GAO developed and administered a brief, non-generalizable survey to 104 customer service agents working at four selected large airports that GAO visited in March and April 2019. Survey results on customer service agents' experiences with passengers cannot be used to make inferences about all customer service agents but nevertheless provide valuable insights. No comprehensive data are available to determine the nature and frequency of passenger assaults—e.g., verbal threats, attempted physical acts, or actual physical acts—against airline customer service agents at airports. This lack of data is due, in part, to the limited federal role in addressing such assaults. GAO's survey of 104 airline customer service agents showed that over half (61) reported experiencing such action in the past year, while almost all reported experiencing verbal harassment. About 10 percent reported experiencing physical assaults. Stakeholders GAO interviewed said that while passengers are often verbally disruptive, physical assaults are less frequent. These stakeholders also said that alcohol consumption, frustration over airlines' business practices (e.g., fees for checked or carry-on baggage), and long lines can contribute to these incidents. Of the stakeholders—i.e., airlines, airports, law enforcement, and prosecutors— GAO interviewed who provided perspectives and have responsibilities for passenger assaults, all 23 said state and local laws sufficiently deter and address such incidents, and 15 (of 20) said current resources are sufficient. One prosecutor told GAO the transitory nature of airports makes it difficult to get witnesses to testify at trial; when prosecuted, passengers generally face misdemeanor charges. While stakeholders GAO interviewed generally did not identify gaps in resources, some said incidents could be further mitigated if, for example, airports made law enforcement's presence more visible or airlines provided conflict de-escalation training to customer service agents. The FAA Reauthorization Act of 2018 required that airlines (1) provide such training to all employees, and (2) submit plans to the Federal Aviation Administration (FAA) by January 2019 detailing how airlines respond to passenger assaults. In July 2019, FAA issued a notification to airlines reminding them to submit their plans; officials said they will continue to follow up with airlines until they receive the plans.", "document_type": "gao"}
{"report": "NNSA is executing and plans to carry out multiple weapon programs and a range of related capital asset projects over the next 2 decades. First, NNSA is currently conducting four weapon modernization programs: the B61-12 LEP, the W88 Alteration 370, the W80-4 LEP, and the W87-1 Modification program. Table 1 provides more information on each of these programs based on our prior work, with selected updates on program schedules, cost estimates, and budgets from the Fiscal Year 2020 Stockpile Stewardship and Management Plan and NNSA testimony. In addition to these four ongoing programs, the 2018 Nuclear Posture Review calls for NNSA to consider additional weapon programs— specifically, a program to develop a modern nuclear-armed sea-launched cruise missile, and another to develop a new submarine-launched ballistic missile warhead (now being referred to as the W93). The Nuclear Posture Review also instructs NNSA to maintain the B83-1 bomb until a suitable replacement can be found. To support and enable ongoing and planned weapon programs, NNSA also plans to spend billions of dollars over the next 2 decades on capital asset projects and other infrastructure risk reduction and recapitalization efforts to modernize the production infrastructure NNSA uses to produce components and materials needed for its weapon programs. Table 2 provides more information on selected NNSA capital asset projects discussed in our recent reports, with selected updates on program schedules and cost estimates from the DOE Office of Project Management’s January 2020 Monthly DOE Project Portfolio Status Report. According to NNSA’s plans, the agency must carry out many of its weapon programs while simultaneously modernizing the very infrastructure on which these weapon programs rely for components and other materials. Therefore, any delays or technical challenges that affect NNSA’s plans for its production facilities may be expected to result in delays and challenges to the weapon programs. Figure 1 shows the estimated schedules for the weapon programs and related capital asset projects described in tables 1 and 2 and reported on in our prior work, with updated information as presented in the Fiscal Year 2020 Stockpile Stewardship and Management Plan. We have reported on the potential effects on NNSA’s weapon programs of delays or technical challenges in modernizing its production facilities. For example: The W87-1 Modification program’s schedule may be particularly vulnerable to production challenges, including pit production challenges, because, as we reported in November 2018, it will require all newly-made components, including pits. In our most recent report on the W87-1 program, a classified report issued in February 2020, we found that NNSA’s past challenges in managing plutonium activities at Los Alamos and in executing projects of this size cast doubt on NNSA’s ability to produce 80 pits per year in 2030. As we note in that report, an independent assessment of NNSA’s pit production strategy in March 2019 concluded that no options evaluated by NNSA could be expected to produce 80 pits per year by 2030. The independent assessment further stated that NNSA had no precedent for major projects costing more than $700 million dollars that had been completed in fewer than 16 years, and that many similar projects were eventually cancelled. Future weapon programs will require newly produced explosives, including some that NNSA has not produced at scale since 1993. As we reported in June 2019, NNSA officials stated that producing these materials will pose challenges that include replicating decades-old recipes for the materials and preparing for their full-scale production in aging facilities. As we noted in that report, similar problems restarting dormant production capabilities have delayed past weapon programs—notably, the W76-1 LEP, which NNSA completed in December 2018. As we reported in March 2009, NNSA had to delay first production of the W76-1 from September 2007 to September 2008 when it encountered problems restarting production of a key material, known as Fogbank. NNSA is working to reconstitute its high explosives capabilities, as we reported in June 2019. Nonnuclear parts and components comprise over 80 percent of the items in a nuclear weapon, and NNSA’s Kansas City National Security Campus procures or produces most of these. In April 2019, we found that work on the B61-12 LEP and W88 Alteration 370 was expected to double at the Kansas City site during fiscal years 2020 through 2022. Our April 2019 report also identified challenges that could complicate work at the site. For example, disruption to the established supply chain for externally supplied parts—which comprise about 65 percent of the nonnuclear parts used at the Kansas City site—could result in production delays, and the site needs hundreds of thousands of additional square feet of manufacturing space to meet workload demands. We have also recently completed work in which we reported on challenges integrating the schedules of NNSA’s weapon programs with the schedules for DOD’s modernized delivery systems. For example, the W87-1 warhead will need to be integrated on a delivery system that is under development, an intercontinental ballistic missile known as the Ground-Based Strategic Deterrent. We have ongoing work examining DOD and DOE plans to modernize and integrate warheads and delivery vehicles and expect to issue a classified report in spring 2020. As we have recently reported, NNSA has made improvements in its management of some weapon modernization programs and enabling capital asset projects. We have concluded that NNSA’s federal program and project management capacity is improving, as are the controls it has developed for program and project performance. For example: We found in January 2018 that NNSA has established and strengthened management requirements for LEPs. Specifically, in January 2016, NNSA’s Office of Defense Programs issued a program management directive that designates risk-based program execution requirements that all programs must follow. The directive places LEPs in one of the highest-risk categories, meaning these programs are required to apply more rigorous management controls specified in the directive, including using earned value management. Further, in January 2017, NNSA issued two directives implementing requirements for NNSA’s Office of Cost Estimating and Program Evaluation to conduct independent cost estimates. In May 2018, we found that the program cost estimate for the B61-12 LEP substantially met the criteria for all four characteristics of a high-quality, reliable cost estimate, in part because it was the first LEP to undergo an independent cost estimate. We reported in our February 2017 high-risk update that DOE demonstrated a strong commitment and top leadership support for improving project management. For example, DOE made changes to its revised project management order, issued in May 2016, in response to recommendations we made in prior years, such as requiring that projects develop cost estimates and analyses of alternatives according to best practices we identified. In September 2017, we found that NNSA had made progress in developing a revised scope of work, cost estimate, and schedule for the Uranium Processing Facility project, which is to modernize uranium production efforts at the Y-12 National Security Complex. We reported at that time that these improvements may help NNSA stabilize escalating project costs and technical risks experienced under the previous strategy. In November 2017, we found that NNSA had established programs to manage strategic materials—specifically, uranium, plutonium, tritium, and lithium—and had defined requirements and managerial roles for program managers. Since that time, NNSA has taken steps to implement a new enterprise-wide approach for managing explosives activities, as we found in our June 2019 report on those activities. However, we have identified additional actions NNSA could take to further improve its management of weapon modernization programs and related projects. As NNSA’s workload increases, additional management rigor will help ensure that programs and projects are executed consistent with cost and schedule estimates, and that risk is effectively managed and communicated. For example: We found in our January 2018 report that NNSA had not adopted the best practice of having an independent team validate its earned value management systems against the national standard for such systems, which could help the agency better manage risk in its LEPs. We also found that NNSA had not established specific benchmarks for technology readiness at LEP decision points, consistent with best practices. We recommended that NNSA require an independent team to validate contractor earned value management systems for LEPs and establish technology readiness requirements at LEP decision points. According to an update NNSA provided to us in September 2019, the agency has not taken action to address these recommendations. We continue to believe that it should do so. We found in our September 2017 report that NNSA had not developed a complete scope of work, a life-cycle cost estimate, or an integrated master schedule for its overall uranium program—of which the Uranium Processing Facility is only one part—and had no time frame for doing so. We recommended that NNSA should set a time frame for when the agency would develop a complete scope of work, a life-cycle cost estimate, and an integrated master schedule for the overall uranium program. NNSA generally agreed with our recommendation and has taken actions to respond to it. We expect to issue a report on the Uranium Processing Facility and NNSA’s plans for its uranium program in March 2020. As we reported in February 2020, the plutonium program has begun to develop a schedule for pit production. However, NNSA allows strategic materials programs such as the plutonium program to tailor their approach to developing schedules and does not require that they meet best practices for schedule estimating. We recommended that NNSA ensure that the plutonium program develop a schedule for pit production consistent with best practices for schedule development. NNSA agreed with our recommendation. Our ongoing work includes reviews of NNSA’s management of other efforts essential to ongoing weapon modernization programs, such as the production of radiation-hardened microelectronics at Sandia National Laboratories in New Mexico and of depleted uranium at the Y-12 National Security Complex in Tennessee. NNSA’s weapon modernization programs and enabling infrastructure efforts have significant interdependencies that require integrated management across the portfolio of programs to effectively manage cost, schedule, and risk. Portfolio management best practices developed by the Project Management Institute state that organizations can optimize their portfolios of programs and projects by assessing their capability and capacity to finance specific portfolio components; determining which portfolio components should receive the highest priority; and identifying components to be suspended, reprioritized, or terminated. In our April 2017 report on NNSA’s budget materials and modernization plans, we found that NNSA did not clearly identify the extent to which its long-range budget estimates for its overall modernization program fell short of specific annual budget requests anticipated in this plan. We concluded that NNSA had not addressed the projected bow wave of future funding needs and the mismatch between those needs and the potential funding available in the years in question. By not addressing the risks associated with the potential funding shortfall, we concluded, NNSA raised questions about its ability to achieve its modernization program goals at cost and on schedule. As a result, as discussed above, we recommended that NNSA include an assessment of the affordability of its portfolio of modernization programs in future versions of the Stockpile Stewardship and Management Plan—for example, by presenting options NNSA could consider to bring its estimates of modernization funding needs into alignment with potential future budgets, such as potentially deferring the start of or canceling specific modernization programs. NNSA did not explicitly agree or disagree with our recommendation. The President’s fiscal year 2021 budget request for NNSA indicates that the bow wave has arrived, requesting an increase of about $3.1 billion over the funding enacted for Weapons Activities in fiscal year 2020—a year-to-year increase of over 25 percent. The Fiscal Year 2020 Stockpile Stewardship and Management Plan, issued in July 2019, includes a new section on affordability analysis and states that the section was added in response to our April 2017 recommendation. However, our review of this section indicates that it does not fully respond to our recommendation because it does not provide information about how potential misalignment between NNSA’s modernization budget estimates and projections of the President’s modernization budgets may be addressed, or about the potential impacts of adjusting program schedules or cost or schedule overruns. Since the issuance of the 2018 Nuclear Posture Review, NNSA’s portfolio of planned programs has only grown more extensive and complex. We continue to believe that NNSA, by assessing its portfolio of modernization programs in future versions of the Stockpile Stewardship and Management Plan—for example, by presenting options NNSA could consider to bring its estimates of modernization funding needs into alignment with potential future budgets, such as potentially deferring the start of or canceling specific modernization programs—could help congressional and NNSA decision makers better understand NNSA’s priorities and trade-offs that it may need to undertake in the future, depending on funding and program performance. Chairman Cooper, Ranking Member Turner, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff members have any questions about this testimony, please contact me at (202) 512-3841 or bawdena@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Jason Holliday, Assistant Director; Antoinette C. Capaccio; Julia Coulter; Rob Grace; John Hocker; Dan Royer; and Kiki Theodoropoulos. The following is a selection of GAO’s recent work assessing the National Nuclear Security Administration’s management of nuclear weapon programs and related capital asset projects: Nuclear Weapons: NNSA Should Further Develop Cost, Schedule, and Risk Information for the W87-1 Warhead Program. GAO-20-207C. Washington, D.C.: February 28, 2020. Nuclear Weapons Sustainment: Improvements Made to Budget Estimates in Fiscal Year 2019 Joint Report, but Opportunities Remain to Enhance Completeness. GAO-20-37R. Washington, D.C.: November 7, 2019. Nuclear Weapons: Additional Actions Could Help Improve Management of Activities Involving Explosive Materials. GAO-19-449. Washington, D.C.: June 17, 2019. Modernizing the Nuclear Security Enterprise: NNSA Is Taking Action to Manage Increased Workload at Kansas City National Security Campus. GAO-19-126. Washington, D.C.: April 12, 2019. High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas. GAO-19-157SP. Washington, D.C.: March 6, 2019. Nuclear Weapons: NNSA Has Taken Steps to Prepare to Restart a Program to Replace the W78 Warhead Capability. GAO-19-84. Washington, D.C.: November 30, 2018. B61-12 Nuclear Bomb: Cost Estimate for Life Extension Incorporated Best Practices, and Steps Being Taken to Manage Remaining Program Risks. GAO-18-456. Washington, D.C.: May 31, 2018. Nuclear Weapons: NNSA Should Clarify Long-Term Uranium Enrichment Mission Needs and Improve Technology Cost Estimates. GAO-18-126. Washington, D.C.: February 16, 2018. Nuclear Weapons: NNSA Should Adopt Additional Best Practices to Better Manage Risk for Life Extension Programs. GAO-18-129. Washington, D.C.: January 30, 2018. Nuclear Weapons: NNSA Needs to Determine Critical Skills and Competencies for Its Strategic Materials Programs. GAO-18-99. Washington, D.C.: November 14, 2017. Modernizing the Nuclear Security Enterprise: A Complete Scope of Work Is Needed to Develop Timely Cost and Schedule Information for the Uranium Program. GAO-17-577. Washington, D.C.: September 8, 2017. National Nuclear Security Administration: Action Needed to Address Affordability of Nuclear Modernization Programs. GAO-17-341. Washington, D.C.: April 26, 2017. High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others. GAO-17-317. Washington, D.C.: February 15, 2017. DOE Project Management: NNSA Needs to Clarify Requirements for Its Plutonium Analysis Project at Los Alamos. GAO-16-585. Washington, D.C.: August 9, 2016. DOE Project Management: NNSA Should Ensure Equal Consideration of Alternatives for Lithium Production. GAO-15-525. Washington, D.C.: July 13, 2015. Nuclear Weapons: NNSA and DOD Need to More Effectively Manage the Stockpile Life Extension Program. GAO-09-385. Washington, D.C.: March 2, 2009. GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs. GAO-09-3SP. Washington, D.C.: March 2009. Nuclear Weapons: Opportunities Exist to Improve the Budgeting, Cost Accounting, and Management Associated with the Stockpile Life Extension Program. GAO-03-583. Washington, D.C.: July 28, 2003. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "NNSA is simultaneously modernizing the nation's nuclear weapon stockpile and the infrastructure on which weapon programs depend. In a 2019 report, NNSA stated that this is the busiest time for the nuclear security enterprise since the Cold War era. GAO's April 2017 review of NNSA nuclear modernization programs concluded that NNSA made optimistic assumptions about future costs. DOD and DOE estimate that nuclear modernization will cost hundreds of billions of dollars over the next decade. This statement is based on 18 GAO reports issued from July 2003 to February 2020 and selected updates. It discusses (1) NNSA's ongoing and planned programs and projects to modernize weapons and related infrastructure and challenges they present; (2) NNSA's improvements in managing these programs and projects, and additional steps NNSA could take to make further improvements; and (3) GAO's prior recommendation to NNSA on assessing the affordability of its portfolio of modernization programs. To conduct the updates, GAO reviewed DOE planning and budget documents. The Department of Energy's (DOE) National Nuclear Security Administration (NNSA) is conducting four programs to modernize nuclear weapons, and the Department of Defense's (DOD) 2018 Nuclear Posture Review calls for NNSA to consider additional programs to refurbish or build new weapons over the next 2 decades. NNSA is also managing numerous, multi-billion-dollar construction projects to modernize the infrastructure it uses to produce components and materials needed for its weapon programs. GAO has reported on challenges NNSA faces in managing these efforts. For example, GAO's February 2020 report on the W87-1 warhead program found that NNSA's past challenges in managing plutonium activities cast doubt on NNSA's ability to produce the required number of plutonium weapon cores on schedule. GAO also found in June 2019 that future weapon programs will require newly produced explosives, including some that NNSA has not produced at scale since 1993. NNSA has improved its management of weapon programs and related projects in some respects. For example, NNSA has established requirements for independent cost estimates in weapon programs and has made progress in revising plans for the Uranium Processing Facility project. However, GAO has identified additional actions that could further improve NNSA's management of weapon programs and projects. For example, in September 2017, GAO reported that NNSA had not developed a complete scope of work, a life-cycle cost estimate, or an integrated master schedule for its overall uranium program. GAO recommended that NNSA set a time frame for developing these plans. GAO expects to issue a report on NNSA's uranium program plans in March 2020. GAO concluded in April 2017 that NNSA had not addressed a potential mismatch between funding needs and funding availability. GAO recommended that NNSA assess its portfolio of modernization programs—for example, by presenting options to align programs to potential future budgets, such as potentially deferring the start of or cancelling specific programs. NNSA did not explicitly agree or disagree with GAO's recommendation. NNSA included an affordability analysis in July 2019 planning documents, but the analysis does not fully respond to GAO's recommendation because it does not state how potential misalignment between program costs and budget projections may be addressed. GAO continues to believe that presenting options to align its portfolio of programs to potential future budgets could help Congress and NNSA better understand NNSA's priorities and trade-offs that may need to be undertaken in the future. GAO has made numerous recommendations to NNSA, including that it assess its portfolio of modernization programs to present options to align programs and budgets. NNSA has taken some action but has not fully responded to this recommendation.", "document_type": "gao"}
{"report": "IRA owners are not permitted to engage in certain prohibited transactions involving IRA assets. Prohibited transactions generally fall into two categories: Transaction involving disqualified persons. An IRA is prohibited from engaging in a transaction with disqualified persons, such as members of the IRA owner’s family or an IRA fiduciary. Transaction involving self-dealing. An IRA owner who is a fiduciary is prohibited from engaging in a transaction with the IRA where the IRA owner personally benefits (other than through the receipt of a distribution). We previously reported that prohibited transactions are more likely to arise when IRA owners make unconventional IRA investments. Unlike conventional IRA investments in publicly traded stocks, bonds, and mutual funds, unconventional investments in real estate, virtual currency, or private equity are more likely to involve the IRA owner, disqualified family members, or other disqualified persons. For example, an IRA invested in rental real estate can leave IRA owners susceptible to a number of prohibited transactions, such as renting to family or paying for repairs with personal funds. IRA owners may face adverse and potentially severe tax consequences if they are found to have engaged in a prohibited transaction. Specifically, the IRA could lose its tax-favored status. The account would then be treated as distributing all its assets to the IRA owner at the fair market value on the first day of the year in which the prohibited transaction occurred. The IRA owner may be subject to additional income taxes because of any early distribution from an IRA. The prohibited transaction may also be subject to excise taxes. The Employee Retirement Income Security Act of 1974 (ERISA), which established IRAs and rules prohibiting certain IRA transactions, assigned IRA oversight roles to both DOL and IRS. To avoid confusion over dual jurisdiction, a 1978 Executive Order further clarified the agencies’ roles and responsibilities regarding prohibited transactions. As a result, the authority to interpret the prohibited transaction rules and grant exemptions to those rules was transferred to DOL. The transfer did not affect IRS’ ability to enforce the excise tax provisions or the tax consequences for IRA owners who are found to have engaged in a prohibited transaction. However, in enforcing such tax consequences, IRS is bound by the regulations, rulings, opinions, and exemptions issued by DOL. DOL has the authority to grant administrative exemptions to the prohibited transaction rules on either an individual or a class basis. DOL can grant prospective exemptions for a transaction that an IRA is considering, as well as retroactive exemptions for transactions that have already occurred. To grant an exemption from prohibited IRA transaction rules, DOL evaluates applications using statutory criteria, and follows administrative procedures codified in regulations. Generally, DOL may not grant an exemption unless it finds the exemption to be: in the interest of the plan and its participants and beneficiaries, and protective of the rights of plan participants and beneficiaries. Before granting an exemption, DOL generally must publish a notice of proposed exemption in the Federal Register inviting interested parties to comment on the proposed exemption. DOL regulations lay out the process for filing and processing prohibited transaction exemptions applications. Among other things, the regulations explain: who may apply, what information must be included with an application, when a conference with DOL can be requested, when a request for reconsideration of a DOL decision can be made, how DOL and the applicant will notify interested persons if DOL decides a tentative approval is warranted. DOL also publishes a booklet that provides an explanation of the regulations and applicable laws, and includes additional information for applicants like examples of common types of exemption requests. IRA owners or their fiduciaries file applications for exemptions with DOL’s Office of Exemption Determinations which is part of EBSA. Applicants can research information about past exemptions granted by the agency on EBSA’s website. As explained in the DOL booklet describing the application requirements, applicants have the burden of demonstrating that they should be granted an exemption. If DOL tentatively denies an application, applicants have options for requesting that the denial be reconsidered. Within 20 days of the tentative denial, applicants can request a conference with DOL, or notify DOL of their intent to submit additional information. If, after a conference has been convened, DOL issues a final denial of the application, DOL will entertain one request for reconsideration if the applicant presents significant new facts or arguments, which, for good reason, could not have been submitted earlier. After DOL publishes a notice of proposed exemption in the Federal Register that describes the pending application, the applicant must notify interested persons of the pending exemption. Often, the contents of the information sent to all interested persons, the manner in which it is sent, and any associated deadlines will have previously been agreed to by DOL and the applicant. DOL may also hold public hearings during the comment period. For example, if the transaction involves potential fiduciary self-dealing or conflicts of interest, any individual potentially adversely affected by the exemption may submit a request for a public hearing to DOL. If granted, DOL publishes information about the exemption in the Federal Register and on its website. Figure 1 provides an overview of the exemption application process. The regulations describe circumstances in which DOL will ordinarily not consider an application. For example, DOL generally will not consider an individual application if DOL already has under consideration a class exemption relating to the same type of transaction. DOL will also not consider an application for transactions subject to DOL or IRS investigations. DOL requires applicants to disclose in their applications whether exemption transactions are, or have been, subject to an investigation or enforcement action by DOL or IRS. In addition, if the applicant or any other party in interest becomes the subject of an investigation or enforcement action, the applicant is required to promptly notify DOL. If applicants find that their prospective transaction is substantially similar to other transactions for which the agency has previously granted exemptions, they can follow an expedited process by submitting an “EXPRO” application. EXPRO applications are required to cite prior exemptions granted by DOL to demonstrate that the proposed IRA transaction is substantially similar to other IRA transactions for which DOL has previously provided an exemption. Specifically, EXPRO applicants must cite as substantially similar, either (1) two individual exemptions granted by DOL within the previous 5 years, or (2) one individual exemption granted within the past 10 years, and one transaction authorized pursuant to the EXPRO class exemption within the past 5 years. The applicant must give notice to all interested persons, and the applicant must resolve all substantive adverse comments provided by interested persons before DOL will grant final approval. The time to complete the exemption process can range from a few months to more than a year. DOL officials told us that the process generally takes about 1 year for an individual IRA application that is relatively simple or routine. EXPRO applications have been processed in as few as 78 days. According to DOL officials, the process can start before an applicant submits a formal application because applicants can, and do, request informal consultations and conferences with DOL. DOL officials explained that sometimes potential applicants decide not to file an application after an informal conference because applicants realize that their application would likely be denied. DOL officials explained that during the review process, they first confirm their understanding and characterization of the proposed exemption through correspondence with the applicants. Then, in response, DOL often sets conditions under which relief from the prohibited transaction rules is contingent, such as on the applicant taking additional actions and remaining in compliance with those conditions. For example, if an applicant wants to sell or purchase an asset in what would be an otherwise prohibited IRA transaction, DOL may stipulate that the applicant first obtain an independent appraisal or valuation assessment to determine a fair-market value of that asset. After applications are formally submitted, many IRA applicants withdraw during DOL’s review process. Over an 11-year period, we found that of the 124 IRA applications, applicants withdrew roughly half (56) before the review process was completed (see table 1). Of the remaining 68 applications that continued with the review process, DOL granted 48, denied 16, and closed four application cases for administrative or other reasons. DOL officials did not dispute the results of our analysis, but they said that it would be misleading to conclude that DOL is more likely to grant than deny applications. Rather, they said that their practice of encouraging applicants to consult with DOL in advance leads some potential applicants to decide not to pursue an exemption. In our review of processed applications, we found that most of the applications involved the sale of IRA assets. We found that 88 of the 124 applications were for transactions involving the sale of IRA assets. Most of these were sales of securities or real property (see appendix I for additional information). The next most common type of transaction was for the purchase of assets (21 applications), and most of those also involved securities or real property. The remaining applications involved other transactions, including leases, loans, and extensions of credit. DOL has not sufficiently documented internal policies and procedures to manage and help ensure effective internal controls of its prohibited transactions exemption process. While DOL regulations and guidance detail the requirements for applicants, DOL generally lacks internal documentation of the steps and actions DOL officials are to follow when processing applications, and the roles and responsibilities of agency officials. DOL officials told us that they use a case tracking system to record and track applications. When an application is received by DOL, the division chief of EBSA’s Office of Exemption Determinations (OED) reviews the application and assigns it to an OED supervisor. Either the division chief or the supervisor enters preliminary information from the application into the system, and classifies the transaction by applying one or multiple subject matter codes. The supervisor then reviews the information in the applicant’s case file and assigns the case to an OED analyst. DOL officials told us that any interim data, such as the publication date for a proposed exemption, is entered by the supervisor in the system. If an application is withdrawn by an applicant, denied, or granted, the supervisor records this information in the system, including the dates of these actions. When a case is closed, the analyst completes a close-out index form and submits it to the supervisor for review, and the supervisor enters a closing code in the system. DOL officials told us that they can use the system to generate management reports, such as on the number of applications filed and the amount of time to process cases. Neither the process described above, nor the different roles and responsibilities of the OED division chief, supervisors, and analysts in that process, were documented in the internal documents that DOL provided. A system reference guide included instructions to system users for how to input and modify case records, generate reports, and add or modify users. The reference guide also included screen prints indicating which fields are required by the system to process a case. However, the reference guide did not contain information about responsibilities and duties for these data entry activities, and how those duties are assigned. The documentation provided is unclear regarding who within OED is ultimately responsible for making final decisions on applications. According to Standards for Internal Control in the Federal Government, documentation of an agency’s policies and procedures is a necessary part of an effective internal control system. Such documentation can appear, for example, in management directives or operating manuals, and it should be readily available for examination. Policies and procedures can also help document internal control responsibilities within the agency. DOL officials told us that OED is a small and compact organization, and as such, its policies and procedures can easily be communicated “person to person” and through onsite training. DOL officials also said that the process for entering data is not difficult, and there are few opportunities for error because nearly all data on applications is prepopulated. The principles of internal control, however, apply to both large and small organizations. The level and nature of documentation may vary based on the size of the organization and the complexity of the processes the organization performs, but documentation is still necessary. By documenting policies and procedures, management will be better positioned to monitor whether the organization’s activities are aligned with those policies and procedures, and assess whether the organization is achieving its objectives. Documenting procedures also would provide greater transparency about how applications are handled, and can reduce the risk of employees carrying out their duties inconsistently. For a small organization like OED, documentation of policies and procedures provides a means to retain organizational knowledge, and can help ensure continuity of and consistency in operations if key personnel leave the organization unexpectedly. Some information sharing takes place between DOL and IRS on applications for IRA prohibited transaction exemptions, but no formal mechanism exists to help guide collaboration between the two agencies. As previously discussed, DOL and IRS share oversight responsibility for prohibited IRA transactions. Based on our review of applications and DOL data as well as interviews with agency officials, we found that interactions between DOL and IRS regarding applications for prohibited transaction exemptions are infrequent and limited in scope. Of the 124 applications we reviewed, only eight were coded in OED’s Case Tracking System as having “external contact with IRS,” and DOL officials confirmed that this accurately reflects the level of interagency coordination. DOL officials stated that they sometimes contact IRS about exemption applications, and IRS officials confirmed to us that they periodically receive communications from DOL. IRS officials also told us that they occasionally contact DOL. Both agencies described to us how their current interaction occurs. DOL officials told us that they coordinate with IRS in the following ways: If, during the application review process, OED staff identify applications that may warrant further review or investigation for tax violations, they refer the case to EBSA’s Office of Enforcement, which may then coordinate or refer the case to IRS. DOL officials said that OED staff review the IRS “Dirty Dozen” list of potentially abusive tax scams and schemes. IRS officials said that when possible prohibited transactions arise during an examination that might require DOL input, IRS examiners reach out to DOL to ensure that IRS understands DOL decisions on those transactions. DOL officials said that, in their view, most requested prohibited IRA transaction exemptions do not require extensive interaction with IRS. They questioned the potential usefulness of information about denied or withdrawn applications that might be shared with IRS, but said that IRS could certainly obtain this information if IRS requested it. IRS officials, however, told us that more information from DOL about prohibited IRA transactions and requested exemptions could be useful in carrying out IRS’s oversight responsibilities. For example, DOL does not share information on denied or withdrawn applications with IRS, information that IRS officials told us would be helpful to them. We found that denial information could be useful to IRS as illustrative examples of prohibited transactions for examiner training and educational outreach to IRA owners. Information about the types of transactions in withdrawn applications could also help IRS identify emerging issues or trends in potential prohibited transactions marketed to IRA owners. Although some limited collaboration between DOL and IRS exists, the agencies have not applied to their oversight of prohibited transactions some key practices we have identified in prior reviews of interagency collaboration. Specifically, developing a mechanism to formalize the sharing of information between DOL and IRS could help support current collaboration activities, and could be useful in helping the agencies identify opportunities for greater collaboration going forward. Furthermore, documentation is a necessary part of an effective internal control system. Documenting the procedures for interagency collaboration would improve internal control over the agencies’ activities. A formal agreement, such as a memorandum of understanding (MOU) or other mechanism, can further help agencies monitor, evaluate, and update interagency collaboration. For example, DOL and IRS have previously formalized their collaboration regarding oversight of a different type of retirement savings vehicle— employer-sponsored retirement plans. DOL and IRS have oversight responsibilities for employer-sponsored retirement plans, such as pensions, and in 2003, DOL and IRS completed an MOU to implement collaboration between the two agencies with regards to investigations of and litigation involving employer-sponsored retirement plans. The employer retirement plan MOU and the implementing guidance contain some features of interagency collaboration mechanisms that we have identified in prior work. For example: The responsibilities of each agency are documented, and responsible agency components and officials are identified. The agencies use collaboration tools (checklists) for determining whether issues presented in an examination or investigation by one agency should be referred to the other. A system and process exists to track referrals, and the agencies reconcile their data about referrals (including pending referrals) quarterly. The employer retirement plan MOU also established a process to periodically monitor its effectiveness, and the MOU was last updated in 2013. Developing a similar mechanism to formalize the sharing of information between DOL and IRS regarding IRA prohibited transaction exemptions could help the agencies better support their current coordination efforts and identify additional opportunities for greater collaboration. IRAs are a key vehicle for individuals to save for retirement. IRA owners’ decisions to invest in unconventional assets can expand their role and responsibilities substantially. The consequences for account owners who make a mistake can be severe. When IRA owners request an exemption from rules on prohibited transactions, DOL evaluates applications using statutory criteria, and follows administrative procedures codified in regulations. However, DOL has not sufficiently documented internal policies and procedures for how to manage its process for granting exemptions. Such documentation is a necessary part of an agency’s effective internal control system. DOL and IRS share oversight responsibility of prohibited IRA transactions. While the two agencies do share some information, they do not have a formal mechanism to guide and monitor their collaboration. By formalizing interagency collaboration, such as through an MOU or other mechanism, DOL and IRS could help reinforce their current information sharing and potentially identify new opportunities to improve their oversight efforts through greater collaboration. Documenting procedures for DOL and IRS collaboration on prohibited IRA transactions would also help introduce better internal control over these activities. We are making a total of three recommendations, including two to DOL and one to IRS. The Secretary of Labor should document internal policies and procedures for managing the IRA prohibited transaction exemption process. (Recommendation 1) The Secretary of Labor, in consultation with the Commissioner of Internal Revenue, should establish a formal means, such as a memorandum of understanding or other mechanism, to support and guide DOL’s and IRS’s collaborative efforts to oversee IRA prohibited transaction exemptions. (Recommendation 2) The Commissioner of Internal Revenue, in consultation with the Secretary of Labor, should establish a formal means, such as a memorandum of understanding or other mechanism, to support and guide DOL’s and IRS’s collaborative efforts to oversee IRA prohibited transaction exemptions. (Recommendation 3) We provided a draft of this report to the Secretary of Labor, the Commissioner of Internal Revenue, and the Secretary of the Treasury for review and comment. In its comments, reproduced in appendix II, DOL generally agreed with our two recommendations directed to it. For recommendation 1, DOL plans to create an internal procedure manual formalizing OED’s administrative case processing procedures to help in passing along institutional knowledge. For recommendation 2, DOL agreed to periodically discuss all IRA exemption cases with IRS and did not elaborate on the formal means for this information sharing. DOL also provided technical comments which we incorporated as appropriate. In its comments, reproduced in appendix III, IRS generally agreed with our recommendation directed to it. For recommendation 3, IRS said it is committed to discussing an appropriate mechanism, including periodic meetings, to formalize collaboration on IRA prohibited transaction exemptions. IRS plans to consider expanding its formal collaboration with DOL as part of the next periodic update of the existing employer plan MOU. IRS also provided technical comments which we incorporated as appropriate. The Department of the Treasury provided technical comments which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Labor, the Secretary of the Treasury, and the Commissioner of Internal Revenue. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact James R. McTigue, Jr. at (202) 512-9110 or Charles A. Jeszeck at (202) 512-7215. You may also reach us by email at mctiguej@gao.gov or jeszeckc@gao.gov. GAO staff making key contributions to this report are listed in appendix IV. In addition to the contacts named above, MaryLynn Sergent and David Lehrer (Assistant Directors), Ted Burik, Susan Chin, Steven Flint, Emily Gruenwald, Mark Kehoe, Jungjin Park, and David Reed made key contributions to this report. James Bennett, Amy Bowser, Jacqueline Chapin, Edward J. Nannenhorn, Andrew J. Stephens, Walter Vance, and Adam Wendel also provided support.", "summary": "IRA owners are able to invest in a wide variety of assets, but they are prohibited from engaging in certain transactions involving IRA assets. IRA owners who engage in prohibited transactions may incur increased income tax liability, additional taxes, and the loss of the tax-advantaged status of their accounts. DOL can grant exemptions from the prohibited transaction rules. IRS enforces tax laws relating to IRAs and can assess additional taxes. GAO was asked to examine (1) DOL's process for granting exemptions for prohibited IRA transactions and outcomes of that process, and (2) the extent to which DOL and IRS collaborate on oversight of prohibited transaction rules for IRAs. GAO reviewed relevant federal laws and regulations; examined agency guidance, exemption process documentation, and application case files; assessed interagency coordination using internal control standards and prior work on interagency collaboration; and interviewed DOL and IRS officials. The Department of Labor (DOL) has a process to grant administrative exemptions for individual retirement account (IRA) transactions that would otherwise be prohibited by law, such as an IRA buying investment property from the IRA owner. DOL evaluates applications using statutory criteria and follows administrative procedures codified in regulations. Applications for proposed transactions that are substantially similar to certain other transactions previously granted exemptions may follow an expedited process. As shown in the figure, GAO found that roughly half (56) of the IRA prohibited transaction exemption applications it reviewed were withdrawn by the applicant before the review process was completed. In reviewing processed applications, GAO found that most of the prohibited transactions for which an exemption was sought involved the sale of IRA assets. With regard to DOL's application review process, GAO found that DOL has not sufficiently documented internal policies and procedures to help ensure effective internal control of its process. Documenting procedures could increase transparency about how applications are handled, reduce the risk of DOL employees carrying out their duties inconsistently, and provide a means to retain organizational knowledge should key personnel leave unexpectedly. Although DOL and the Internal Revenue Service (IRS) share some information as part of their oversight responsibility for prohibited IRA transactions, no formal mechanism exists to help guide collaboration between the agencies. Of the 124 IRA applications GAO reviewed, only eight reflected DOL contact with IRS. GAO found that DOL has information about requested exemptions to prohibited IRA transaction rules that could be useful to IRS in carrying out its oversight responsibilities. For example, DOL does not share information on denials—information that could be useful as prohibited transaction examples for IRS examiner training and educational outreach to IRA owners. In prior work on interagency collaboration, GAO has found that formal agreements, such as a memorandum of understanding, can help agencies monitor, evaluate, and update interagency collaboration. Formalizing the sharing of information between DOL and IRS regarding IRA prohibited transaction exemptions could help the agencies better support their current coordination efforts and identify additional opportunities for greater collaboration. GAO is recommending that DOL and IRS establish a formal means—such as a memorandum of understanding or other mechanism—to collaborate on oversight of prohibited IRA transaction exemptions. GAO is also recommending that DOL document policies and procedures for managing the exemptions process. DOL and IRS generally agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Insurance allows individuals and businesses to manage risk by providing compensation for certain losses or expenses, such as those from car accidents, fires, medical services, or inability to work. According to NAIC, as of December 31, 2017, there were 2,509 property/casualty companies and 852 life insurance companies in the United States and its territories. In 2017, premiums written for the property/casualty sector totaled $602.2 billion in 2017 and premiums written for the life and health sector totaled $683.2 billion. As we have noted in recent reports, advances in technology and widespread use of the internet have brought about significant changes in the financial industry. For example, in recent years technology has changed consumer expectations and preferences, with younger consumers especially being well-versed in new technologies and looking to take a more hands-on approach to managing their finances. Similarly, over the last 5 years, established insurers and insurtech companies have used technology to offer simpler insurance products and streamlined customer experiences. Insurtech companies have been playing a variety of roles in the U.S. insurance market. Key players in insurtech include the following: Insurtech companies (typically startups) that are licensed insurance companies. Insurtech startups offer innovative products and services and are active in all major insurance products and all lines of business, with concentrations in the property/casualty business. For example, according to its website, Lemonade Insurance Company is a property/casualty insurer that sells products exclusively through mobile applications (apps) and its website. It offers renters, condominium, and homeowners insurance in several states. Another example is Root, which describes itself as an automobile insurance company that uses a smartphone app to understand individual driving behavior. Customers can download the Root app to their smartphones, obtain a personalized quote after a 2–3 week test drive, and purchase and manage their policy entirely within the mobile app. Insurtech companies that do not provide insurance themselves, but offer technology solutions for insurers. For example, according to the website for Groundspeed Analytics, they use AI and data science methods to provide information for the commercial property/casualty insurance industry to help identify potential areas of profit and enhance the customer experience. According to the website for Habit Analytics, they use real-time consumer data, sourced from smartphones and connected devices in homes, to create behavioral profiles that enable insurance companies to provide input for their risk models. Many established insurers have been acquiring such companies. Established insurers that use technologies or partner with insurtech companies. For example, the insurer Nationwide notes on its website that it created Nationwide Ventures to invest in startups, pilot new technologies, and test new solutions and business models by exploring topics that range from analytics and automation technology to new insurance and financial services platforms. According to analysis by the Deloitte Center for Financial Services and data collected by research firm Venture Scanner, as of mid-2018 there were more than 1,000 insurtech firms established in more than 60 countries, with more than half of those launched in the United States since 2008. Insurance companies are regulated principally by the states and are licensed under the laws of a single state, known as the state of domicile. Companies may conduct business in multiple states, but the state of domicile serves as an important regulator. State regulators license insurance agents, generally review and approve insurance products and premium rates, and examine insurers’ financial solvency and market conduct. As we have previously reported, state regulators typically conduct financial solvency examinations every 3–5 years, while market conduct examinations are generally done in response to specific consumer complaints or regulatory concerns. To help ensure that policyholders continue to receive coverage if their insurer becomes insolvent or unable to meet its liabilities, states also have guaranty funds (separate for life and property/casualty insurance), which are funded by assessments on insurers doing business in those states. Individuals who wish to sell, solicit, or negotiate insurance in the United States must generally be licensed as producers, a term including insurance agents and insurance brokers. Insurance agents typically represent only one insurance company. Insurance brokers represent multiple insurance companies and are free to offer a wider range of products to their clients. Brokers can search the market and obtain multiple price quotes to fit their clients’ needs. Producers must comply with state laws and regulations governing their activities. NAIC notes that as of September 2018, more than 2 million individuals and more than 200,000 business entities were licensed to provide insurance services across all lines of insurance in the United States. Traditional insurers, sometimes referred to as admitted insurers, can be licensed to sell several lines or types of coverage to individuals or families, including personal lines—such as homeowners, renters, and automobile insurance—and commercial lines—such as general liability, commercial property, and product liability insurance. Admitted insurers can sell insurance in one or more states but, according to NAIC, must be licensed to operate in every state in which they sell coverage. To help ensure adequacy and fairness in pricing and coverage, state regulators oversee the insurance rates and forms of admitted insurers. State regulators also may require admitted insurance companies to maintain specific levels of capital to continue to conduct business. The surplus lines insurance market, also known as the nonadmitted market, can provide insurance coverage for risks that traditional insurers are unwilling or unable to cover. The risks covered can include potentially catastrophic property damage and liability associated with high-hazard products, special events, environmental impairment, and employment practices. In the absence of the surplus lines market, NAIC notes that some insureds in those markets would be unable to secure coverage. In most states, surplus lines insurers cannot write insurance coverage that is available from admitted insurers and only may write coverage rejected by a number of admitted insurers, according to NAIC. Furthermore, in those states, the surplus lines insurance broker must conduct a “diligent search” of the admitted insurance market to determine if comparable coverage is available. The broker can write coverage only if a specified number of admitted insurers have declined to offer such coverage. According to NAIC, new and innovative insurance products for which there is no loss history may be difficult to appropriately price. According to stakeholders we interviewed, the nonadmitted market is therefore a common entry point into the insurance market for insurtech firms that want to sell insurance products. NAIC notes that, after a new coverage has generated sufficient data, the coverage often eventually moves to, and is sold by, insurers in the admitted market. For example, private flood insurance was developed and first offered in the nonadmitted market but now also is offered in the admitted market. The nonadmitted market is generally regulated somewhat differently than the admitted market. According to NAIC, surplus lines insurers are subject to regulatory requirements and are overseen for solvency by their domiciliary state or country, but surplus lines transactions are regulated through the licensing of surplus lines brokers. NAIC states these brokers are responsible for ensuring that the surplus lines insurer meets eligibility criteria to write policies in the state and is financially sound. Furthermore, NAIC notes surplus lines brokers and producers must be licensed to sell surplus lines insurance in each state in which they operate. State insurance departments may have authority to suspend, revoke, or not renew the license of a surplus lines broker or producer. Unlike admitted insurers, surplus lines insurers may not have access to state guaranty funds that are available to help pay claims in the event of an insurer insolvency. In addition, according to NAIC, surplus lines insurers generally have more freedom to change policy coverages and premium rates than admitted insurers. NAIC stated that state regulators require both nonadmitted and admitted insurance companies to maintain specific levels of capital to continue to conduct business. According to NAIC, most state insurance regulators also can use their authorities under state statues such as an unfair trade practices act to ensure consumers are protected (for example, to ensure that claims are paid and insurers or brokers do not misrepresent policy terms) and to remedy other bad conduct. NAIC assists state regulators with various oversight functions. While NAIC does not regulate insurers, it provides services designed to make certain interactions between insurers and regulators more efficient. These services include providing detailed insurance data to help regulators understand insurance sales and practices; maintaining a range of databases useful to regulators; and coordinating regulatory efforts by providing guidance, model laws and regulations, and information-sharing tools. The Federal Insurance Office was established in the Department of the Treasury (Treasury) by the Dodd-Frank Wall Street Reform and Consumer Protection Act. The office is headed by a director appointed by the Secretary of the Treasury. The Federal Insurance Office monitors all aspects of the insurance industry (including by identifying issues or gaps in insurance regulation that could contribute to systemic risk in the insurance industry), and helps develop federal policy on international insurance matters, but is not a regulatory agency itself. The office also serves as an information resource for the federal government and coordinates with federal regulators, state insurance regulators, and NAIC. The Federal Insurance Office also represents the United States in the International Association of Insurance Supervisors and coordinates federal efforts in international insurance matters. In recent years, the insurance industry has begun to adopt several types of technology that are designed to provide a range of benefits to insurers and consumers (policyholders), including improved risk monitoring, reduced costs, and improved underwriting. However, the use of these technologies also can create challenges for insurers and potential risks for consumers, including changed business models, pricing fairness, and privacy issues. Based on our literature review and interviews with stakeholders, we identified six key technologies that have seen increased use in the insurance industry in recent years and one technology (blockchain) that has seen limited adoption and which the industry has been exploring for wider use. Mobile apps. A mobile app is software designed to run on a mobile device, such as a smartphone or tablet computer. Insurance industry stakeholders told us that several insurers have adopted mobile apps to make their products and services available on mobile devices. For example, insurers have adopted mobile apps that allow consumers to purchase products online. An increased number of insurers in recent years also have adopted mobile apps that allow customers to complete tasks online such as submitting insurance claims and turning on-demand insurance coverage on or off. Insurers also have been using mobile apps to capture consumer data and usage patterns (behaviors). AI, algorithms, and machine learning. AI is the development of computer systems to perform tasks and make decisions that historically have required human intelligence to perform. Machine learning is a subset of AI and focuses on the ability of machines to receive a set of data and learn for themselves, changing algorithms as they learn more about the information they process. (Algorithms are sets of rules that a computer or computer program follows to compute an outcome.) In the insurance industry, AI includes applications that provide specific expertise or allow for task completion. For example, AI provides on-line “chatbots” (sometimes called robo-advisory services) that answer questions specific to an insurance product or service. When a consumer communicates with a chatbot, the chatbot takes the information the consumer provided and enters it into an algorithm. Based on protocols outlined in the algorithm, the chatbot provides a response to the consumer’s question. As the conversation moves forward, the chatbot will adapt to answer more questions using machine learning in real-time. According to insurance industry stakeholders, insurers have been using algorithms to analyze information obtained from other technology sources to determine what a consumer’s risk profile is and then determine the consumer’s premium rate based on their risk profile. Big data. Big data are large volumes of data (often aggregated from multiple sources to develop data sets). As we have noted in other work, big data are frequently analyzed using predictive analytics, machine learning, and data mining to identify trends, patterns and characteristics. The insurance industry uses big data in several ways, including analyzing consumer information, identifying risk patterns and pricing risk, and analyzing information related to risk pooling. Insurers also use big data to streamline and more accurately underwrite products. For instance, an insurer may use big data to determine whether consumers are high- or low-risk based on factors identified from extensive datasets such as what they purchase online or how they shop for insurance online. This is similar to lenders’ usage of big data. In a previous report, we noted that lenders were using big data to evaluate risk and make lending decisions using real- time nontraditional information gathered from social media sites. Internet of things. The internet of things refers to semi-autonomous and internet-capable devices (such as machinery, home appliances, thermostats, and smartphones) that have sensors that interact with the physical environment and typically contain elements for processing and communicating information. Some insurers stated that the internet of things could be used in the insurance industry to track and reduce risk, detect problems, and mitigate potential claims. For example, a homeowner could have a smart home thermostat that sends alerts when the power goes off and indoor temperature decreases. With the homeowner able to address the issue in real time, the homeowner could mitigate the risk of frozen pipes bursting and potentially prevent a loss and an insurance claim. According to CBInsights, insurers have partnered with insurtech firms that provide this technology to offer real-time monitoring. Drones. Drones are remotely piloted aircraft systems. Insurers have been using drones for a variety of purposes in the insurance industry. For example, insurers use drones to obtain aerial footage over a disaster area to determine the amount of damage to a house or crop field. Insurance companies also use drones to verify information submitted by a policyholder in a claim or help determine the risk presented by difficult-to-reach areas of a property, such as a roof. Telematics. Telematics combines telecommunications and information processing to send, receive, and store information related to specific items such as automobiles or water heaters. Telematics often uses sensors to relay information such as global positioning system location, speed, and water levels. For example, sensors in an automobile can provide data on a driver’s behavior (such as speed, hard braking, and turning radius). The insurer may use that information to determine the driver’s risk profile and help determine the premium rate for that driver. These technologies can be used together. For example, a telematics device can be used to provide data to a mobile app, which can then send the information to an AI algorithm to determine whether a claim should be paid. See figure 1 for examples of the types of technologies that insurers may use to automate the claims process. Blockchain/ distributed ledger technology and smart contracts. The insurance industry has been studying whether blockchain technology could be used to improve insurance processes. Blockchain refers to a type of distributed ledger technology—in which multiple entities and locations share and synchronize datasets—that facilitates and permanently records virtual transactions. Information is uploaded and recorded in a series of secured blocks; the information uploaded cannot be modified or erased once uploaded into the blockchain (thus providing an accurate history of specific transactions and information). According to insurers, blockchain could be used by the industry to track insurance coverage history, expedite the claims process, provide an audit trail of insurance transactions, and address cybersecurity issues. For instance, a blockchain could expedite the claims process by allowing agents, policyholders, and repair companies immediate, secure access to certain data that are part of the claim only as the data are needed. “Smart contracts” include provisions for contract performance that can be executed by a computer algorithm (for instance, on a blockchain). For example, an insurer stated that a smart contract for homeowners insurance might stipulate that if an earthquake of a specific size occurred in a policyholder’s residential area, a claim payment for damage in a specified dollar amount automatically would be made from the insurer to the policyholder. According to NAIC, adoption of blockchain technology in insurance is limited at this time. According to stakeholders with whom we spoke and literature we reviewed, the use of technology in the insurance industry creates potential benefits but also can create risks for both insurers and consumers. We present stakeholder views on the benefits and challenges technology presents in the primary areas they identified as being affected by technology, which include (1) pricing and risk evaluation, (2) consumer protection, (3) business operations and risk monitoring, and (4) product offerings. See figure 2 for a summary of the potential benefits and challenges we discuss. According to stakeholders we interviewed and literature we reviewed, the use of technology for determining insurance pricing and coverages creates several benefits and risks for insurers and consumers: Increased underwriting accuracy. Insurers and others told us that insurers have been using technologies that provide enhanced analytic capabilities or data from previously unavailable sources to increase the accuracy of underwriting. These technologies allow insurers to make new connections between policyholder characteristics and risk. That is, insurers are using big data, AI, and algorithms to obtain and analyze more information about consumers than they previously had been able to obtain. For instance, a property/casualty insurer could collect data on when consumers set their home alarms and use this and other risk information to refine risk determinations for those individuals. Another example is when insurers use data collected from telematics devices in automobiles to inform the insurer about the policyholder’s risk of being involved in an accident. A better understanding of the risk presented by policyholders can help insurers more accurately and effectively price and manage risks. More individualized pricing. Insurers also have been using technologies to underwrite policies in a way that results in more individualized pricing, which benefits insurers and could benefit some consumers. That is, big data can allow an insurer to use factors for which traditional underwriting typically has not accounted. According to stakeholders we interviewed, doing so allows an insurer to place an individual in a smaller risk pool than if traditional underwriting factors were used and to price coverage for that individual more in line with the risk that individual presents. This can help an insurer better manage its level of risk by offering lower prices to lower-risk customers, charging more for higher-risk customers, or even declining to offer coverage to consumers it considers high-risk. Some stakeholders told us that technologies allow consumers to receive more individualized premium rates, based on their risk characteristics, than had been possible. For example, some insurers have been using telematics devices to obtain information on policyholder driving habits and the risk level they present and adjust premium rates based on this information. As a result, consumers who engage in safer driving practices receive the benefit of lower premiums. Policyholders also could use such information to take actions that will lower their risk level and therefore their premiums. For instance, consumers could seek to reduce specific driving behaviors, such as fast stops or starts, which negatively affect their premium rate. However, consumers with higher-than-average risks could end up paying more or perhaps be declined coverage. Stakeholders including an industry representative and a law firm in the field indicated that insurers also might use data to exclude high-risk consumers from marketing. For example, an insurer might not choose to market to high-risk consumers to discourage them from buying their insurance. This approach, in theory, helps insurers decrease the number of high-risk policyholders they insure but could create difficulties for some seeking coverage. Two industry representatives and an academic in the field indicated that the potential for decreased risk pooling creates a difficult question about the minimum extent of pooling that is socially desirable. For example, these stakeholders stated that when insurance underwriting becomes too individualized, it might no longer serve an insurance function; that is, there is very little pooling of risk. They stated it may be a desirable social benefit to have a certain level of risk pooling to allow more people to effectively manage their risk. In a November 2018 issue paper, the International Association of Insurance Supervisors noted the potential effect of more individualized underwriting on the fairness of consumer outcomes. Among other findings, the paper noted the collection of more data on policyholders may enable a more specific risk categorization that could affect risk pooling principles and lead to issues around affordability of certain insurance products or even availability (the potential for exclusion). The association noted that insurance supervisors should monitor whether such negative consumer impacts become a trend and, if so, raise awareness at the appropriate policy and political level(s). Validating consumer data and models. Insurers and insurtech firms increasingly have been using AI and data collection algorithms to gather data through mobile, wearable, and other internet-connected devices and from online sites. According to two academics in the field, collecting consumer data in large quantities and from multiple disparate sources, including social media, poses challenges for insurers in relation to validating those data. Insurers and insurtech firms also face challenges associated with validating models that use the data. Although AI and machine learning can help insurers and agents underwrite risk more accurately, these stakeholders said that these tools and processes can increase risk because the collected information may be inaccurate or inappropriately used in determining premium rates. For example, while models may indicate that certain factors developed by AI from social media and other sources are associated with increased policyholder risk, it may be difficult or impossible for insurers to validate the accuracy of such data. In addition, it can be a challenge for insurers to ensure that the use of such data and models does not result in the use of prohibited factors in determining premium rates, such as race or sex. For example, several stakeholders told us that certain factors, while not specifically disallowed by insurance regulations, could end up serving as a proxy for a disallowed factor. One example cited by a stakeholder was the use of information on consumer magazine subscriptions, which are not prohibited on their own, but could serve as proxies for factors that are prohibited. Finally, it can be a challenge for insurers to document and explain to regulators how rating models that use AI and machine learning work and provide assurance that the rates produced by the models are not unfairly discriminatory toward policyholders. For example, some industry stakeholders we interviewed said that these models are often developed by data scientists and not actuaries, as had been the case in the past. Unlike actuaries, they said data scientists who develop rating models may not fully understand insurance-specific requirements, such as setting premium rates that are not unfairly discriminatory, and may struggle to measure the impact of new variables used in the models. Furthermore, data scientists may be unfamiliar with insurance rules and regulations and may not understand how to communicate their work to state insurance regulators. One regulator described to us how one insurance company was unable to explain how one of the factors that it entered into its advanced risk model—proximity of a home to a day care center—related to the risk that a consumer posed. An actuarial group suggested a greater collaboration between actuaries and data scientists could provide greater assurance that such rating models meet regulatory requirements. Quality of data used in pricing. According to some stakeholders, insurers’ use of nontraditional data and AI to develop insurance pricing models creates two potential risks for consumers that parallel some of the risks for insurers. First, as previously mentioned, insurer’s use of nontraditional data and AI can create a risk that factors unrelated to the risk presented by a consumer could be used to set his or her premium rate. Stakeholders including a regulator said that algorithms or big data may allow insurers to correlate certain factors with higher claim rates, although the factors do not actually relate to risk and may even act as a proxy for a prohibited factor such as race or sex. As a result, some stakeholders noted that using such information to determine a premium rate could be unfairly discriminatory. Some stakeholders also said that such factors unintentionally could become proxies for prohibited rating factors— such as race. For example, using information on a consumer’s purchase history could serve as a proxy for race. Second, some stakeholders indicated that when insurers use AI to generate information on consumers, it is difficult to ensure these data are accurate. Because the data were not explicitly provided by the consumer, the consumer does not have a chance to correct or dispute the data. For example, if an insurer uses AI to pull data from a consumer’s social media accounts, those data could be incorrect or outdated, but the consumer would not know the data were being used as a factor in determining his or her premium rate. This would prevent the consumer from correcting the information if it was wrong. Some stakeholders indicated that if an insurer has difficulty understanding the factors and algorithms being used to price the insurance product, the consumer most likely will not be able to understand them. According to stakeholders with whom we spoke and literature we reviewed, some uses of technology can pose risks in terms of the protection of consumer data. In addition, the use of the nonadmitted market by insurtech companies and insurers may result in more limited financial protections for consumers. Cost of protecting consumer data. As noted earlier, insurers collect and use consumer data in large quantities and from multiple disparate sources, including social media, posing challenges for protecting those data. For example, according to representatives of one property/casualty industry association we interviewed, it can be expensive to maintain the appropriate level of cybersecurity (including technical and organizational measures) to prevent any unauthorized access or use of the additional volumes and types of customer information used in recent years. Consumer privacy concerns. Stakeholders noted that insurers’ expanded use of consumer data raises concerns about the privacy of such data. For example, an automobile insurer may collect data on a consumer using a telematics device installed in the consumer’s vehicle. While an insurer may use data on the consumer’s driving habits for the purpose of adjusting premium rates, the device also may collect information on where and when a consumer drives. This is information consumers may not wish others to possess. One academic also said there is concern about the ownership of the data collected through telematics and other technologies, such as AI, for the purposes of insurance. For instance, if an insurer obtained data from a policyholder’s automobile with a telematics device, a question exists about whether policyholders would have the right to take those data to another insurer if they switched insurers or whether the data belong to the first insurer. As we have described in other work, this presents a larger privacy issue as it may not be possible for a consumer to know exactly what is collected, or when and how the data are used. This lack of knowledge reduces the consumer’s control over their personal information and limits their ability to track what data belong to them. Some stakeholders mentioned concerns about insurers collecting information from social media and other sources that consumers did not explicitly consent to provide to insurers. The European Union (EU) General Data Protection Regulation, which includes regulations governing consumer consent, had an entry into force and application date of May 25, 2018. According to an industry analyst, the General Data Protection Regulation applies to insurance companies around the world, including those in the United States, that process the personal data of EU residents, regardless of the nationality of the person in question or the location of the company. Furthermore, the analyst notes that the regulation strictly defines legal uses of individuals’ data and requires companies to ensure individuals can explicitly and individually consent to other uses of their data. In prior reports, we also noted data privacy concerns in relation to lender use of financial technology. Consumer protection concerns due to use of the nonadmitted market. The nonadmitted market is a common entry point for insurtech firms because of that market’s usefulness for innovative insurance products with little loss history. However, the sale of consumer insurance through nonadmitted insurers raised concerns among several stakeholders. As we noted in a prior report, nonadmitted insurers may face fewer regulatory constraints than traditional insurers in the prices they can charge and their ability to create and offer new products. While data do not exist on the number of insurtechs using the nonadmitted market, industry representatives told us that because of this greater regulatory freedom, a number of insurtechs choose to operate as nonadmitted insurers or as brokers selling policies through nonadmitted insurers. As described in the Background, when consumers purchase insurance from nonadmitted insurers, they do not have some of the same consumer protections they would have if they purchased coverage from an admitted insurer. For example, regulators conduct limited reviews of the prices charged and the products sold by nonadmitted insurers. And as noted earlier, if nonadmitted insurers became insolvent, state guaranty funds may not be available to help pay policyholder claims. As we previously reported, some regulations serve to push potential policyholders toward the admitted market because of the better financial protections it provides (such as rate approvals and access to state guaranty funds). For example, as noted earlier, a broker placing coverage with a nonadmitted insurer generally must conduct a diligent search for available coverage in the admitted market every time a potential policyholder requests coverage in the nonadmitted market. This helps ensure coverage is purchased from an admitted insurer as often as possible. Stakeholders offered differing assessments on the extent of any related risks to consumers resulting from insurtech use of the nonadmitted market. For example, an industry representative said the nonadmitted market is not appropriate for most consumer products because of the lower consumer protections as compared with the admitted market. Two insurtech firms also have raised questions about the ability of insurtech companies and other market participants to properly comply with diligent search requirements. For example, an industry representative told us it does not seem possible to satisfy the diligent search requirement when products are sold on-demand through a mobile app. Furthermore, the representative raised the question of how a broker could legitimately search the admitted market for coverage in cases in which an insurer offers immediate coverage as soon as consumers complete applications on their smartphones. Conversely, some insurers, regulators, and NAIC said that nonadmitted insurers are appropriately regulated and consumers are not necessarily at any greater risk than when purchasing coverage from admitted insurers. Also, several states have eliminated the diligent search requirements. However, a consumer advocate noted that such deregulation raises further consumer protection issues in a market where less regulation is already a concern for consumers. According to the literature we reviewed and stakeholders we interviewed, insurers have been using various technologies to reduce their operating costs but may face risks that affect their operations and business models. Reduced costs. Stakeholders described how adopting various technologies has led to reduced costs in four operational areas for insurers: Communicating with customers. Insurers have been using mobile apps and chatbots to reduce the cost of providing information to potential customers. For example, a consumer might be shopping online for an insurance policy late in the evening. The insurer can use a chatbot to interact with that consumer and answer questions about insurance products. In the past, this might not have been possible if an agent was not available to work nonstandard business hours or insurers might have needed to hire and retain more agents to work evenings and weekends. Underwriting. Insurers have been using technology to reduce the cost of underwriting insurance. For example, according to two insurtech firms and one industry representative we interviewed, some insurers review multiple sources of data with AI to automatically review the information in a consumer’s insurance application, rather than incurring the costs of hiring staff to do so. Through the industry article review and stakeholder interviews, we found that insurers also use the internet of things to obtain data from smart home alarms to monitor consumer usage of alarm systems and thereby assess consumer risk levels. This reduces the costs associated with determining and analyzing risk factors. Claims processing. According to some stakeholders we interviewed, insurers now have the capability to digitally collect and automatically analyze claim evidence, thereby reducing staffing needs and realizing cost savings. For example, consumers can use their smartphones to take photographs of their vehicles after an accident and send the photographs and other information to their insurers through mobile apps. On receipt of the photographs, insurers can use AI algorithms to verify the damage shown—decisions that historically required human intelligence to perform—and automatically start the claims process for the consumer. Fraud. Insurers are able to detect fraud, or decide which claims need to be investigated further by employees, with information verified using big data, the internet of things, and telematics. For instance, an insurer may verify information provided in a claim against information obtained from a smart device to determine if the information provided by the policyholder was accurate. An insurer also might identify a false burglary claim by verifying whether an alarm was set during the time frame identified in the claim and reviewing video from home security cameras. Connecting to legacy computer systems. Some industry stakeholders and association representatives we interviewed stated that established insurers face significant challenges using new technologies because they first have to replace legacy computer systems or customize their systems to interface with new technologies properly. According to industry stakeholders, legacy computer systems were, in some ways, built around satisfying regulatory requirements rather than enhancing the consumer experience or providing more desirable products. They noted it can be costly and difficult to replace such systems or to modify them to interface with more consumer-centered systems, such as those being developed by insurtech companies. Changing roles for insurers and agents. According to some insurance industry stakeholders, emerging uses of key technologies and innovative business models could lead to changes in insurers’ roles and products. For example, with the advent of self-driving vehicles, the liability for accidents could shift from the driver to the vehicle maker or the company that produced the self-driving system. In such cases, they said insurance coverage primarily would be sold to those entities rather than the consumer, and the demand for and amount of consumer automobile coverage sold could decrease substantially. This could cause a shift in demand for products from consumers to commercial lines, resulting in the potential loss of business for some agents and insurers. Some industry stakeholders we interviewed also told us that as more technologies (such as telematics or other smart devices) were adopted to help consumers mitigate risk, insurers likely would have to shift their business model. That is, they would have to move from a model focused on sales of policies, in which agents play a central role, to a model focused on providing consulting services to consumers to help them prevent and mitigate risk and loss. Risk monitoring. Insurers have been using big data with data aggregation and mining to improve monitoring of insured risks. More specifically, several stakeholders told us that these tools and analytical methods can help insurers quickly analyze volumes of data from many sources in or near real time. For example, several stakeholders gave the example of an insurance company using sensors or other devices to continuously collect verified data on movements of insured ships and their cargo. Such data can be useful to insurers for understanding the risks associated with providing insurance coverage and even can be used to provide the ship carrying the cargo the appropriate insurance documentation required for the port of entry. Several stakeholders also told us that some insurtech companies have been using telematics to collect real-time data on driver behavior, which they combine with other information such as credit scores, to develop a fuller and more accurate picture of the risk presented by a given policyholder. Insurers then can use these risk profiles to determine whether to change a policyholder’s rates or continue to insure them. Several stakeholders indicated that such real-time information is likely more accurate than previous risk- assessment methods. According to stakeholders we interviewed and literature we reviewed, the use of various technologies to create new product offerings has created several benefits for insurers and consumers. Ability to offer on-demand products. Technologies have been helping insurers tailor products to specific consumer needs and expand offerings to niche markets. Some insurtech companies have started offering on-demand insurance (insurance that policyholders can turn on and off as needed). For example, one regulator and an academic said that market research data demonstrated that consumers want to be able to turn on insurance for their drones when the drones are in use and turn it off when the drones are idle. Insurers also have been developing similar on-demand products for drivers working for rideshare companies such as Lyft and Uber and for Airbnb and VRBO rentals (to cover the gaps that traditional homeowners insurance, which generally provides coverage on a long-term basis, might have in relation to short-term rentals of homeowners’ properties). On-demand products allow insurers to diversify their product lines and attract more consumers, which is discussed later in this report. Increased convenience. With some insurers providing mobile apps and chatbots, consumers are able to access insurance products and information 24 hours a day. For example, consumers can use mobile apps to get immediate quotes and underwriting decisions from some insurers. In the past, consumers likely would have had to visit an insurance agent or fill out a lengthy application and wait much longer for an underwriting decision. And as previously discussed, some insurers allow their policyholders to submit claim information and photographs of damage through a mobile app without speaking with an agent. Increased consumer choice. According to NAIC and an insurtech firm, consumers can benefit from the increased choice that comes from insurers using technology to offer additional products and services. For example, consumers obtain the ability to purchase insurance for certain time periods for certain items such as drones and action cameras, home sharing, or mile-based automobile insurance. NAIC and the insurtech firm said that some insurers that offer insurance to rideshare operators allow the policyholders to turn the coverage on when they are working and off when they are not. This can reduce premium rates for policyholders who only occasionally work as rideshare drivers. According to the industry articles we reviewed and the stakeholders with whom we spoke, insurers’ use of technology also has benefitted consumers by leading to the development of aggregator websites that bring together quotes from multiple insurers and allow consumers to comparison shop for insurance products. Some insurers said technology may soon give consumers the added ability to further customize their insurance policies by allowing them to select among various available coverages and terms and essentially create a policy that best suits their needs. NAIC, state regulators, and others have initiated a number of actions intended to monitor and address industry and regulator concerns associated with insurtech, including any insurance rules and regulations that could affect insurers’ adoption of technologies. These actions address challenges in areas including (1) evaluation of underwriting methodologies, (2) approvals for new insurance products, (3) customer notification methods and time frames, (4) anti-rebating laws, (5) cybersecurity, and (6) regulator skillsets and resources. NAIC and state regulators have initiated a number of actions intended to monitor concerns that regulations could affect insurers’ adoption of innovative technologies while maintaining oversight of consumer protection issues. First, to monitor technology developments that may affect the state insurance regulatory framework and to develop regulatory guidance, as appropriate, NAIC created an Innovation and Technology Task Force. According to NAIC, this task force provides a forum for regulator education and discussion of innovation and technology in the insurance sector. For example, the task force has held discussions on the collection and use of data by insurers and state insurance regulators—as well as new products, services, and distribution platforms—to educate the regulators on how these developments affect consumer protection, privacy, insurer and producer oversight, marketplace dynamics, and the state-based insurance regulatory framework. In addition, the task force has held forums on emerging issues related to companies or licensees leveraging new technologies. Areas discussed included developing products for on-demand insurance purposes, reviewing new products and technologies affecting the insurance space, and potential implications for the state-based insurance regulatory structure. In addition, in 2012 the EU-U.S. Insurance Dialogue Project was formed, in which EU and U.S. insurance regulators discuss emerging technology issues in the international insurance industry. During the project’s sixth forum in November 2018, the regulators and representatives from industry and consumer organizations discussed challenges and opportunities relating to issues including cyber risks, the use of big data, and AI. According to a project publication, the dialogue project enhanced mutual understanding of respective regulatory frameworks and initiatives between the United States and European Union, which will help ensure effective coordinated supervision of cross-border insurance groups for the benefit of policyholders. In 2018, the project published an issues paper on big data. The paper discusses data collection, portability, quality, and availability and how insurers and third parties use data in marketing, rating, underwriting, and claims handling. Future work by the project may include discussion of insurers’ use of third-party vendors, disclosures to applicants, and insurers’ use of AI models. NAIC and state regulators have initiated a number of actions intended to address industry and regulator concerns about certain insurance rules and regulations that a number of them said could affect insurers’ adoption of technologies. Stakeholders, including regulators, told us that regulators can face challenges in assessing new underwriting methodologies, such as those that use predictive analytics or AI. Reviewing predictive analytics can be a challenge for regulators because of the amount of data used to develop a model, the complexity of techniques, and limited staff resources (discussed in more detail later in this section). In addition, insurers employ different technological approaches, and their documentation and explanation of the methods and approaches differ. Finally, the data and models insurers use dynamically change and may have to be re- submitted for review even before regulators have an opportunity to review the original submission. One state regulator and an industry stakeholder also told us that while an insurer may know the universe of factors from which an AI system pulls, the insurer may not know, or be able to describe for regulators, how the system uses those factors to determine a premium rate. In turn, this may prevent regulators from understanding the system or validating the insurer’s assertions about the system. For example, one state regulator told us that after presenting a rate scheme based on nontraditional factors, an insurer was unable to provide assurances or explanation to the regulator that the resulting premium rates were not unfairly discriminatory. In 2018, NAIC’s Casualty Actuarial and Statistical Task Force began developing a white paper on best practices state regulators can use when reviewing predictive models and analytics filed by insurers to justify rates and guidance they can use for their review of rate filings based on predictive models. NAIC officials told us the Casualty Actuarial and Statistical Task Force will receive comments on the white paper and then evaluate how to incorporate best practices into the Product Filing Review Handbook and recommend such changes to other NAIC working groups. Insurtech firms and other stakeholders told us that working through other regulatory processes, such as the insurance product filing and approval process, often can be inefficient and time consuming because insurers must file in every state in which they wish to sell a product and state requirements can vary. We have noted such difficulties in the insurance market in general. These challenges can be exacerbated by rapid technological evolution in insurer products and risk models. In addition, some stakeholders noted that a lengthy product approval process can be challenging for technology-oriented products. For instance, an insurtech firm may develop a new product quickly to meet consumer demand but might not be able to get the product to market quickly. Some also said that products might become obsolete before the filing approval process was completed. Some stakeholders told us that such challenges can motivate insurtechs to sell insurance through nonadmitted insurers because such insurers have more freedom in altering and selling new products. As we have noted, doing so can bring risks for consumers. In December 2017, the American Insurance Association proposed the Insurance Innovation Regulatory Variance or Waiver Act (Proposed Model Law) to NAIC. The proposed model law would urge allow regulators to create regulatory “sandboxes,” wherein certain regulatory requirements would be waived for insurers seeking to pilot innovative products. Specifically, the proposed model law would authorize insurance regulators to grant variances, waivers, or no-action letters with respect to statutory or regulatory requirements that make it more difficult to introduce new insurance technologies, products, or services. Under the proposed model law, regulators also would be authorized to attach terms and conditions meant to protect consumers to such variances or waivers. Some stakeholders with whom we spoke believed that regulatory sandboxes would not work in the U.S. state-based regulatory framework. For example, some stakeholders told us it would be inappropriate for a state to change legal or regulatory requirements for some but not all insurers or grant exceptions to laws passed by a state legislature to some insurers and not others, as it would no longer be a level playing field. State regulators generally told us they believe that the current regulatory framework provides state regulators with enough flexibility to allow for technology-based innovation. Accordingly, some states have been promoting the use of innovation in the insurance industry by hosting technology sandboxes, where technology companies meet regularly with state regulators to improve companies’ knowledge of insurance regulations and also educate regulators about how the technologies work. According to stakeholders, these technology sandboxes are not the same as regulatory sandboxes that have been established in other nations, as they do not allow waivers of laws and regulations for insurtech companies to test their products. Insurtech firms we interviewed told us that regulations that require paper notifications and U.S. mail delivery for certain processes can make it difficult or more costly for them to offer products with features such as immediate underwriting or on-demand policies. For example, according to insurers and other industry stakeholders, some state laws require that insurance policy cancellation notices be sent by U.S. mail rather than by email. One insurtech firm told us that it would be very costly to meet requirements for mail delivery of insurance policies and cancellation notices because they would have to set up another delivery mechanism (in addition to their electronic notification system). Industry stakeholders also told us that certain laws and regulations that require a minimum period of time before a consumer-initiated policy cancellation takes effect can present challenges for products designed to allow consumers to immediately turn certain coverage on or off. For instance, if consumers used a mobile app to indicate they wanted to turn their automobile insurance coverage off temporarily, it could be unclear if this constituted an actual policy cancellation. Some stakeholders are concerned that states may require an insurance company to give the policyholder a written notice of cancellation at least 30 days before the end of the policy term. Similarly, industry stakeholders told us that some current state regulations could impede on-demand coverage because policies usually must indicate that coverage begins at 12:01 a.m. on the day after a policy is signed and approved. For instance, for on-demand policies that allow on/off subscription at the consumer’s request, it can be unclear whether they are covered the minute that they initiate the coverage, or if they must wait until the following day for coverage to be effective. According to NAIC, many states have taken steps to work within or modify existing laws and regulations to adapt to the increased use of technology in the insurance industry. For example, to address concerns that insurers are required to provide customers with a written, 30-day notice of a policy cancellation, NAIC conducted an analysis in 2018 that found that many states instead require “adequate” notice and that approximately 44 states allow notices to be provided electronically. However, some stakeholders in the insurance industry told us that state cancellation notice requirements are still a barrier to innovation. According to industry stakeholders, many states have anti-rebating laws that generally prohibit insurers from providing consumers with anything of value as an inducement to purchase insurance. NAIC Model Law 880 states that unless expressly provided by law, no insurer may knowingly pay any rebate or incentive to an insured to induce them to purchase a specific product. Insurers, industry stakeholders, and regulators (including NAIC’s Innovation and Technology Task Force) told us that anti-rebating laws can be a barrier to innovation because they could preclude insurers from offering devices that could be used to help insurers and consumers monitor risk. For example, if an insurer offered a policyholder free use of a telematic device (to help insurers collect real- time data and potentially help the policyholder make driving habits safer), it could be considered an inducement and violate anti-rebating laws. The same possibility exists if an insurer were to provide a policyholder with a device to monitor the operating conditions of a boiler to prevent potential water damage should a problem arise. As a result, anti-rebating laws may make it difficult for insurers to make use of certain technologies that could benefit both insurers and policyholders. In contrast to the consensus on the legitimacy of electronic communications, there is little consensus among states on addressing insurers’ concern that anti-rebating laws are a barrier to innovation. According to NAIC, states vary widely on the types of items insurers are allowed to provide for free to customers, with some states having dollar limits on allowable items or allowing items that are specifically linked in a policy. In other cases, it is unclear what is allowable. At NAIC’s fall 2018 meeting, participants noted that some of the NAIC bulletins related to the anti-rebating model law have not addressed whether technologies such as telematics that provide benefits to consumers are considered rebates. According to NAIC, others noted that states typically have taken the position that if a rebate or incentive reduces risk that is the most important issue for all parties involved. NAIC officials noted during the fall 2018 meeting that they will continue to monitor the issues involved. NAIC adopted a model law and states have passed new laws governing cybersecurity and data protection to safeguard the increasing amount of personal data used by insurers. In 2017, NAIC approved the Insurance Data Security Model Law, which creates a legal framework for requiring insurance companies to operate cybersecurity programs. The law outlines planned cybersecurity testing, creation of an information security program, and incident response plans for breach notification procedures. The NAIC model law is only a guideline until adopted by individual states, but NAIC noted that in 2018 and 2019, Michigan, Ohio, Mississippi, and Alabama adopted laws based on the NAIC model and additional states have pending legislation. In an October 2017 report, Treasury endorsed the model law and recommended that Congress consider preempting the states if the law were not adopted over the next 5 years. At the state level, New York’s Department of Financial Services noted it was the first state agency to establish cybersecurity regulations, which became effective March 1, 2017. In May 2018, South Carolina enacted the South Carolina Department of Insurance Data Security Act, which NAIC has characterized as an adoption of the model law. In December 2018, Michigan adopted a similar law. Separately, in June 2018 California passed a law giving consumers more control over their personal information. California’s law generally requires companies to report to customers, upon their request, the categories of personal information they collected about the customer, the business or commercial purpose for collecting and selling such personal information, and what categories of third parties received it. According to industry and regulatory stakeholders, the complexity and evolving nature of the models and approaches used by insurers may outpace the rate at which regulators can educate themselves on those models and approaches. For example, regulators trained in the current rating models may need to acquire new skills to understand and validate advanced and evolving models. In addition, stakeholders told us that new technologies used by insurers can pose significant challenges to regulators partly because of the resource requirements. For instance, regulators and other stakeholders told us that regulators often do not have enough staff with technical expertise, such as data analytics skills, and find it challenging to hire and retain such staff due to limited resources. NAIC has initiated actions to address concerns that state insurance regulators may not have staff with the knowledge or skill sets to address more complex predictive models. For example, in 2018 NAIC management conducted a survey of states regarding the appropriate skills and potential resources NAIC membership may need to deal with big data. Subsequently, in April 2019, NAIC management made recommendations to its Big Data Working Group to hire a technical staff resource to provide technical support for state insurance regulators in the review of actuarial models; develop a tool for state insurance departments to share information on model reviews; and develop a training and education program. NAIC officials told us they also plan to develop a white paper to provide state regulators with guidance on the use of chatbots and AI in the distribution of insurance and the regulatory supervision of these technologies. As many of the regulatory initiatives that NAIC and states have undertaken to address challenges associated with the implementation of new technologies are under development (or recently developed), the impact of these actions on innovation and consumer protection is unknown. It will be important for NAIC and state insurance regulators, as well as the Federal Insurance Office, to continue monitoring developments in these areas. We provided a draft of this report to Treasury and NAIC for review and comment. Treasury and NAIC provided technical comments that we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Treasury, the Chief Executive Officer of the National Association of Insurance Commissioners, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-8678 or ortiza@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. This report (1) identifies uses of technologies and the benefits and challenges they might present for insurers and their customers, and (2) discusses what stakeholders identified as key challenges that could affect the adoption of new technologies, and actions that have been taken to address those challenges. While insurance technology (insurtech) does not have a standard definition, for the purposes of this report we defined it as the use of emerging technologies by insurance companies. We focused on insurtech activities in the property/casualty and life sectors of the U.S. insurance market, including information on personal and commercial insurance where available. We did not include the health insurance sector in our scope because of significant differences between that sector and the property/casualty and life insurance sectors in terms of the types of products offered and the methods by which they are sold and regulated. To identify technologies being used in the insurance industry and gain insights about their (potential) benefits and challenges for insurers and customers, we conducted a literature review of scholarly and peer- reviewed material, trade and industry articles, government reports, conference papers, general news, association, nonprofit, and think tank publications, hearings and transcripts, and working papers that described these technologies and their uses. We conducted searches of the ProQuest and HeinOnline databases to identify studies published from January 2015 through June 2018 that were relevant to our research objectives. Because insurtech is a fairly new field, we found few academic publications related to our objectives. We also conducted background research for examples of technologies being used in the insurance industry and their associated benefits and challenges. We also conducted semi-structured interviews with cognizant stakeholders and reviewed documents provided by them to obtain information on and descriptions of current, in-development, and potential future uses of existing or new technology in the insurance industry. We also obtained their views on the benefits and challenges experienced or expected by insurance companies as well as the (potential) benefits and challenges for consumers. We conducted more than 35 interviews with representatives of regulatory organizations, including the Federal Insurance Office; National Association of Insurance Commissioners (NAIC); state insurance regulators in Arizona, California, Connecticut, and Michigan; and the National Council of Insurance Legislators. We also interviewed three academics, representatives of one consumer group, 13 traditional insurance and reinsurance providers and industry associations, two actuarial professional associations, four consulting groups, two law firms in the field, and seven insurtech firms. We identified potential interviewees by conducting internet research, reviewing literature search results, and reviewing recommended interviewees from our initial interviews. We selected interviewees based on their relevance to the scope of our review. Based on our literature review and interviews with stakeholders, we identified seven recently used and emerging technologies in the insurance industry: (1) mobile applications; (2) artificial intelligence (AI), algorithms, and machine learning; (3) big data; (4) internet of things; (5) blockchain/ distributed ledger technology and smart contracts; (6) drones; and (7) telematics. To obtain information about challenges that could affect the adoption of innovative technologies, we identified relevant laws and regulations pertaining to insurance technology innovation by reviewing prior GAO reports on financial regulation, interviewing regulators and industry participants, and analyzing relevant documents, including relevant NAIC model laws and state laws and regulations. We also conducted semi- structured interviews with and reviewed documents provided by the key stakeholders identified in the first objective to identify (1) any actions NAIC and selected state insurance regulators were taking on new insurance technologies, and what challenges, if any, insurers’ use of new technologies creates for regulators; (2) what is known about the impact of any actions taken by NAIC and state insurance regulators on innovation among insurance companies and on consumer protection; and (3) stakeholders’ views on the applicability of foreign regulatory actions for U.S. insurtech markets. We conducted this performance audit from April 2018 to June 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Patrick Ward (Assistant Director), Deena Richart (Analyst in Charge), Gina Hoover, Hadley Nobles, Akiko Ohnuma, and Tyler Spunaugle made key contributions to this report. Also contributing were Emei W. Li, Barbara Roesmann, Jena Y. Sinkfield, Frank Todisco, and Helen Tulloch.", "summary": "The innovative use of technology by insurance companies (insurtech) is growing and offers the potential to improve customer experiences while also lowering insurer costs. Some stakeholders have raised questions about how certain uses of insurtech could create both risks for consumers and challenges for regulators, and whether some challenges might slow technological innovation in the insurance sector. GAO was asked to provide information on insurtech activities in the property/casualty and life insurance sectors. This report (1) identifies new uses of technologies and potential benefits and challenges for insurers and their customers; and (2) discusses what stakeholders identified as key challenges that could affect the adoption of new technologies, and actions taken to address those challenges. GAO reviewed available literature; analyzed relevant laws and regulations; and conducted interviews with more than 35 stakeholders, including federal and state regulators, technology companies, insurers, and consumer groups (selected based on literature reviews and recommendations, and for relevance to the scope of GAO's review). GAO is not making any recommendations in this report. Insurtech companies (recently established companies bringing technology-enabled innovations to the insurance industry) as well as established insurers have begun to use technologies, including artificial intelligence (AI) and mobile applications, in an attempt to improve risk assessment and enhance customer experiences. For example: Consumers can purchase insurance products specifically tailored to their situation and needs, such as renters or auto insurance that can be turned on and off as needed using a mobile app. Some insurers have begun to use nontraditional data (such as from social media) to analyze policyholder risk, and use AI and complex algorithms to reduce costs by automating information gathering and risk assessment. However, implementing these technologies can create potential challenges for insurers and risks for consumers, including the following: The use of AI to create underwriting models for determining premium rates can make it challenging for insurers to ensure that factors prohibited by regulation (such as race) are not used in models. Such models are often developed by data scientists who, unlike actuaries, may not fully understand insurance-specific requirements. Insurer collection and use of consumer data not provided by the consumer raise questions about data accuracy, privacy, and ownership. Some insurtechs sell coverage through nonadmitted insurers. As we have previously reported, nonadmitted insurers—unlike traditional insurers—are not required to be licensed in each state in which they sell insurance, and receive less regulatory oversight of their policies and rates. Also, if nonadmitted insurers became insolvent, state guaranty funds would not be available to help pay policyholder claims. Stakeholders with whom GAO spoke identified challenges they said might affect adoption of innovative technologies. These include paper-based documentation requirements that do not accommodate online insurance transactions, and challenges for regulators in the evaluation of complex rating models. The National Association of Insurance Commissioners (NAIC) and state regulators have initiated a number of actions designed to address such concerns. For example: State insurance regulators, through an NAIC task force, have been examining regulatory areas that may pose obstacles for innovation, such as requirements for paper documentation or signatures. NAIC issued draft best practices for states to use when reviewing complex rating models. NAIC adopted a model law that creates a legal framework for states to use to require insurance companies to operate cybersecurity programs and protect consumer data. Because many of these regulatory initiatives are still in development (or recently developed), the effect on innovation and consumer protection is unknown.", "document_type": "gao"}
{"report": "Counting the nation’s approximately 140 million households is an enormous undertaking requiring such essential logistics as the opening of hundreds of area offices to conduct essential field activities, recruiting and hiring hundreds of thousands of temporary workers to carry out those activities, and developing an approach to training those employees. To help control costs while maintaining accuracy, the Bureau is making significant changes in each of these areas compared to prior decennials. According to Bureau planning documents, the Bureau intends to use technology to efficiently and effectively manage the 2020 Census fieldwork, and as a result, reduce the staffing, infrastructure, and brick- and-mortar footprint required for the 2020 Census. The three main components of the reengineered field operations are increased use of technology, increased management and staff productivity, and streamlined office and staffing structure. The Bureau’s 2020 Operational Plan states that 2020 Census field operations will rely heavily on automation. For example, the Bureau plans to provide most listers—temporary staff who verify and update addresses and maps—and enumerators— temporary staff who follow up with households that do not respond to the census questionnaire—with the capability to receive work assignments and perform all administrative and data collection tasks directly from a mobile device allowing them to work remotely. Supervisors will also be able to work remotely from the field and communicate with their staff via these devices—precluding them from needing access to a nearby local office. The Bureau’s 2020 Operational Plan states that these enhanced capabilities will significantly reduce the number of offices required to support 2020 Census fieldwork. In the 2010 Census, the Bureau established 12 RCCs and nearly 500 ACOs. The new design for the 2020 Census field operations includes six RCCs with 248 ACOs. Those 248 will be split into two waves, with 39 of the offices opening for Wave 1 by March 2019 to support early census operations such as in-field address canvassing, and the remaining 209 opening for Wave 2 by September 2019. Recruiting enough workers to fill the hundreds of thousands of temporary positions needed to conduct the 2020 Census is a tremendous challenge. According to Bureau plans before hiring begins, the Bureau needs to assemble an applicant pool in the millions. For the decennial census, Bureau plans indicate the Bureau will need a large and diverse workforce to ensure the accuracy of its maps and address list, and to follow up by phone or in person with households that do not respond to the questionnaire. Making these efforts even more difficult are external factors beyond the Bureau’s control, such as low unemployment rate, which can make it harder to recruit. According to Bureau plans, recruiting of potential employees will be conducted throughout the ACOs’ geographic area, based on projected operational workloads and staffing models developed for 2020 Census operations. Selected candidates will be invited to be fingerprinted and submit selected appointment paperwork prior to attending classroom training. The candidates will be sworn in and hired during the first day of training. The ACO staff model is as follows: one ACO Manager, one Lead Census Field Manager, one Administrative Manager, one Recruiting Manager, one Information Technology (IT) Manager, and Office Operations Supervisors, Clerks, and Recruiting Assistants. For data collection, it is: multiple Census Field Managers, Census Field Supervisors, and Enumerators; specific numbers based on workload; and supervisory ratios to be determined (see fig. 1). According to Bureau plans, the 2010 Census approach to training was predominantly instructor-led training with some hands-on training. This primarily consisted of instructors standing in front of a room of trainees and reading training materials to them from a prepared script. For 2020, the Bureau has developed training materials that use a blended training approach including instructor-led training, computer-based training, and hands-on training. This approach is intended to maximize trainee learning and on-the-job performance during the 2020 Census. According to the Bureau’s Detailed Operational Plan for the Field Infrastructure and Decennial Logistics Management Operations, it has developed training materials based on the lessons learned from previous censuses, such as the need to provide computer-based training. The Bureau’s Detailed Operational Plan for the Field Infrastructure and Decennial Logistics Management Operations also states that this innovation to training combines multiple modes of training delivery designed to maximize training outcomes for various types of learning styles: visual, auditory, and hands-on, blending online training methods, instructor-led classroom training, and on-the-job training or role-playing to prepare field staff to effectively fulfill their duties. Blended training is intended to: Provide a standardization of training, limiting the impact of instructor interpretation. Allow for easily updateable training materials in the case of errors or operational changes, minimizing the burden of errata materials. Provide automated assessment tools to enable a more consistent and reliable way to measure learner understanding of concepts. Provide post-training support through easily accessible online manuals and job aids. Training materials are designed to maximize self-paced learning. These accompanying training materials are developed to provide the most up-to- date methodologies for recruiting, onboarding, and training-the-trainer to carry out field data collection activities. For the 2020 Census, the Bureau plans to open 248 ACOs. Similar to the 2010 Census, the total number of ACOs for 2020 was derived from the projected workload for field operations based on the number of enumerators needed for nonresponse follow-up. The Bureau allotted a specific number of ACOs to each of its six regional offices. Regions then developed boundaries for the ACO based on seven mandatory criteria that are described in a program memorandum, including that every state have at least one ACO; federally-recognized American Indian areas and military bases (regardless of county, state, or regional boundaries) will be managed by only one ACO; and ACO areas of responsibility will not cross state boundaries (with the exception of Indian reservations and military bases). See figure 2 below for the location of the 248 offices. In addition to the criteria used to delineate boundaries for its ACOs, the Bureau also had requirements for the ACO leased space. These requirements, for example, included that the ACO have a certain amount of contiguous square footage depending on the ACO type, and that an ACO not be co-located in a building that also houses agencies with law enforcement responsibilities because of privacy and confidentiality concerns. The Bureau also designated an “area of consideration” for each of its ACOs. According to Bureau officials, the area of consideration, which is a smaller geographic range where they would like to house the office, was based on such factors as access to public transit, general centrality within the ACO work boundaries, and proximity to eating establishments. In some cases, the Bureau had to deviate from its requirements for leased space or initial area of consideration. The decision to deviate from requirements usually arose from a lack of viable options in the real-estate market coupled with the Bureau’s need to meet its time frames. According to RCC staff, any deviations from requirements were presented at weekly staff meetings and then subsequently approved by the Regional Director, and in some cases such as co-location with law enforcement Bureau Headquarters approval was needed. According to Bureau officials, co-location with law enforcement is sensitive because of concerns that census data may be shared with others. Census data are kept confidential for 72 years. However, Bureau officials told us either the law enforcement offices were deemed innocuous, for example, the office housed a public defender or the law enforcement offices operated undercover, whereby no one entering the building would have been aware of their presence. In another case, Bureau officials told us that the Philadelphia region was struggling to find space for its ACO in Frederick, Maryland. When the General Services Administration (GSA) proposed a space in Hagerstown, Maryland, 30 miles away, the Bureau accepted it, though it was outside the initial area of consideration. According to officials at the regional office, the Bureau saved time and money by using a readily available cost-effective option by choosing Hagerstown, Maryland. The Bureau also had to expand the area of consideration for more than 31 percent or 77 of its 248 ACOs. According to Bureau officials, designating an area of consideration was an iterative process based on market availability, and having to expand the area was often necessary to secure space (see table 1). In select cases, the Bureau co-located ACOs in the same building. For example, instead of having one office in North Philadelphia and one in South Philadelphia, Bureau officials in the Philadelphia Region Census Center agreed to accept space in the same building located within the boundaries of the South Philadelphia ACO. The Bureau hired staff for each ACO from the original designated areas and kept the two offices completely separated. Bureau officials provided documentation indicating that this compromise came with considerable cost savings. The Bureau also abandoned other planned requirements in a number of cases to secure space, such as access to loading docks, assigned parking, and freight elevators. When we reviewed selected ACO files at the regional offices to determine whether the files included support for when deviations from space requirements and initial areas of consideration were documented, we did not find documentation. Instead, documentation was in staff emails. Files included a checklist of documents required, such as the signed lease and design intent drawings; however, there was not a requirement that documentation of deviations from space requirements or initial areas of consideration be maintained. Bureau officials at the regional level said that all procedures for handling waivers and expansions of the area of consideration were driven by the RCCs as well as informal guidance that was not documented. Standards for Internal Control in the Federal Government calls for documentation and records to be properly managed and maintained. Based on our suggestion that the Bureau develop a procedure for documenting these deviations in ACO areas of consideration or requirements, Bureau officials sent an email requiring that staff keep documentation (electronic or paper) on deviations in ACO areas of consideration or requirements in the ACO’s lease file folders. In cases where decisions are made via telephone or email, Bureau officials asked staff to write notes and scan emails, and add them to the ACO files. Maintaining this documentation will help ensure the transparency and integrity of Bureau decision-making, and ensure the information is readily available. The Bureau experienced some early delays when regions were trying to find space and acquire leases. The Bureau attributed some of these delays to the use of the GSA’s Automated Advanced Acquisition Program (AAAP) process. This procurement process provides building owners and their authorized representatives with the opportunity to offer general purpose office space for lease to the federal government. The AAAP process accepts bids the first week of each monthly cycle. Then the remaining three weeks of the month are used to evaluate submitted offers and identify a potential lessor. According to GSA documents, in tight real estate markets, the first cycle did not always yield a suitable lessor due to lack of available inventory, and the short lease term the Bureau was seeking. Therefore, the Bureau had to wait three weeks until the start of the next cycle to re-open the bidding process. Bureau officials stated that during these 3 weeks, the Bureau regions would conduct additional market outreach and communicate outreach efforts with GSA to find a lessor. According to GSA, they agreed that too much time was elapsing in Wave I trying to receive offers without making any changes to the requirements or areas of consideration. To address this issue for Wave 2, the Bureau stated that GSA provided additional training to the Bureau’s regional staff, increased market outreach which included dedicated support from GSA’s national office, and the development of a strategy to use all of GSA’s tools, such as using GSA’s contract brokers in regions with the greatest number of Wave 2 ACOs. Bureau regional staff also told us they were able to meet leasing milestones in part because of flexibility in their requirements and in the areas of consideration. As of June 2019, there were signed leases for 247 of 248 offices. However, during our review, the Bureau reported that it had missed several construction (meaning renovations such as new electrical layouts, heating, ventilation, and air conditioning) and deployment deadlines. According to Bureau documents, for Wave 1 offices, nine of 39 offices had missed the February 28, 2019 deadline for having furniture and IT equipment; and for Wave 2 offices, 49 of 209 offices missed the February 20, 2019 deadline for having construction drawings complete. According to Bureau officials they are managing each of these delays on an office- by-office basis, and headquarters officials meet weekly with the RCCs to discuss the status of each office. They are also actively communicating with GSA on how to best work with the landlord to meet deadlines. Agency officials also indicated that the schedule deadlines for the later phases of construction allow for more time than may be necessary, allowing them to make up time lost from early delays. For example, at the Concord, New Hampshire ACO, the Bureau plans to make up lost time in construction with actions such as using a fence to divide two office areas instead of adding a wall, and using a “cage” for badging instead of constructing a separate room inside the space. As of June 3, 2019, 38 of 39 Wave 1 offices are ready for business. Seven of 209 Wave 2 offices are still working to finish the milestone of completing construction drawings, which had an original deadline of February 20, 2019. According to Bureau officials, the seven offices without completed construction drawings are being given priority attention by both GSA and the Bureau. We will continue to monitor the opening of ACOs in ongoing work. According to Bureau reporting documents, as of June 2019, the Bureau is exceeding its recruiting goals for early operations. This includes field staff for in-field address canvassing where census staff verify address and map information for housing units in selected areas of the country, office staff at the 39 Wave 1 ACOs, recruiting assistants, and partnership specialists. The Bureau had a goal of recruiting approximately 205,000 individuals for its 2020 early operations efforts by the end of June 2019, and plans to recruit between 2.4 million and 2.6 million applicants for all field operations. By comparison, in 2010, the Bureau recruited about 3.9 million applicants. As of June 17, 2019, the Bureau had processed job applications and assessments for approximately 428,000 applicants which represent about 208 percent of its roughly 205,000 recruiting goal. For the 2020 Census, the Bureau plans to hire nearly 400,000 temporary field staff from its applicant pool for two key operations: in-field address canvassing and nonresponse follow-up, where census staff visit households that do not return census forms to collect data in person. In 2010, the Bureau hired approximately 628,000 temporary workers to conduct the address canvassing and nonresponse follow-up field operations. Below is the recruiting and hiring timeline for the in-field address canvassing and nonresponse follow-up operations (see fig. 3). According to Bureau officials, they are recruiting and hiring fewer temporary staff in 2020 compared to 2010, in part, because automation has made field operations more efficient. For example, there is less paper to manage and process as daily payroll records and daily field work assignments are electronic. As a result, productivity has increased and mileage and labor costs have decreased because census field staff do not meet daily with their supervisors, as was the case in 2010. Moreover, the automation of assignment routing to housing units has optimized the time spent by enumerators driving to housing units. During the 2018 End- to-End Test, the Bureau found the productivity for in-field address canvassing had exceeded its goal at all three test sites (see table 2). The Bureau attributes these efficiencies to the automation of work assignments. For the 2020 Census, the Bureau plans to use some of the same strategies it used to recruit and hire all temporary workers as during the 2010 Census—because those strategies were successful—while also leveraging technology and social media. For example, according to the Bureau, the overarching strategy for hiring enumerators is to hire people who will work in the communities where they live. This strategy provides the Bureau with enumerators who are familiar with the areas where they will be working and who speak the languages of the local community. To recruit staff, recruiting assistants are to work with local partnership staff and use paid advertisements and earned media (e.g., publicity gained through promotional efforts, news reports, etc.). The Bureau plans to also continue to use its recruiting website, http://www.2020census.gov/jobs, which provides information about the various positions, local pay rates, application materials, and job qualifications. Moreover, Bureau officials stated that a diverse multilingual workforce is needed and that the Bureau has tailored its approach to that end. For example, the website includes Spanish language pages and recruitment materials (see fig. 4). Bureau documentation indicates that similar to 2010, the Bureau will continue to use waivers and hiring exemptions to enable well-qualified individuals to work on the 2020 Census who otherwise might not have applied for jobs, particularly in hard-to-recruit areas. These waivers allow the Bureau to temporarily hire federal retirees and individuals receiving public assistance without impacting their benefits, and to hire current federal employees without impacting their job status or salary. As of February 27, 2019, the Office of Personnel Management (OPM) had given the Bureau approval to hire 44 re-employed annuitants for the 2020 Census. The Bureau also had dual employment agreements with 28 federal agencies and commissions. For the 2010 Census, the Bureau had these agreements with a total of 81 federal agencies. To obtain waivers for individuals on public assistance, the Bureau is partnering with the Office of Management and Budget and working with Health and Human Services to obtain waivers for Temporary Assistance for Needy Families and Supplemental Nutrition Assistance Program recipients. The Bureau is also working with tribal governments to acquire similar waivers. In addition to these previously used strategies, the Bureau is planning to leverage technology in its recruiting strategy for 2020. This technology includes the Bureau-developed Response Outreach Area Mapper (ROAM) application, a publicly available online mapping tool that Bureau staff can use to better understand the sociodemographic makeup of their assigned areas. The Bureau plans to use ROAM to identify areas where recruiting could be hard and to develop recruitment strategies such as hiring staff with specific language skills. The new technology also includes the MOJO Recruiting Dashboard (also referred to as MOJO Recruit), which is software for Census recruiting personnel to plan and manage recruiting activities and track recruiting progress. For example, MOJO Recruit includes an interactive mapping feature which lets the Bureau plan recruiting activities and track recruiting status for each census tract. The map draws attention to areas that may be experiencing recruiting problems (see fig. 5). Red indicates areas where the Bureau is less than 50 percent of the way toward meeting its recruiting goal. Yellow indicates areas where the Bureau is 50 to 79 percent of the way toward meetings its goal. Green indicates areas where the Bureau is 80 percent or higher of the way toward meetings its goal. Bureau officials also stated that that they plan to increase the use of social media platforms such as Facebook, Twitter, and Instagram to promote and advertise 2020 Census job opportunities. For example, the Bureau’s 2020 Census Recruitment Toolkit includes social media guidelines, tips, sample posts, and sample email messages to assist recruiting staff in providing information about 2020 Census job opportunities. It also assists recruiting staff with responding to questions and concerns or directing people to the appropriate location for more information about jobs. For the 2020 Census, the Bureau revised its application and assessment process to ease the burden on job applicants and to better assist the Bureau in identifying qualified applicants. Job candidates are to apply and take a skill assessment online, as opposed to attending recruiting sessions in person and taking a written test. The Bureau has also streamlined both the application and assessment process by asking fewer questions and requiring only one assessment for all nonsupervisory positions. According to Bureau officials, the 2020 Census job application should take 10 minutes to complete, by comparison the 2010 Census job application took 30 minutes to complete. Moreover, for prior censuses, applicants had to complete one of two 45-minute assessments to determine the appropriate skill set for either working in the office or in the field. For 2020, OPM has approved the Bureau giving one assessment for all five short-term census positions: Recruiting Assistant, Clerk, Office Operations Supervisor, Enumerator, and Census Field Supervisor, thereby eliminating the need to give separate assessments for the office and field positions. Finally, for those considering a supervisor position, a separate supervisory assessment is required. For 2020, this consists of nine questions compared to 29 questions in 2010. According to Bureau officials, this supervisory assessment should take an additional 10 minutes to complete instead of 1 hour, as it did on 2010. For 2020, the Bureau has also changed the assessment questions it asks applicants from situational-judgment questions to biodata and personality questions. In making this decision, during the 2018 End-to-End Test, the Bureau asked situational-judgment questions in the assessment questionnaire, and then administered a set of biodata and personality questions after hiring. The Bureau conducted an analysis of both types of questions and concluded that the biodata and personality questions were a better predictor of job success. Bureau officials told us they will be evaluating their new job assessment processes for 2020, including the use of biodata. The Bureau has identified challenges that exist in some areas, such as: (1) delayed background checks; (2) low unemployment; and (3) language barriers. Employment with the Bureau is contingent upon successfully completing a background check. The Bureau found that the process for four positions (recruiting assistants, office operation supervisors, clerks, and partnership specialists) was taking longer than it expected. These positions require a full background check because employees will have access to the Bureau’s network, they will be issued expensive equipment (e.g., laptops and desktops), and their employment will likely last more than 6 months. For the full background checks, applicants must complete two security background forms—Standard Form 85: Questionnaire for Nonsensitive positions (SF85) through the Electronic Questionnaires for Investigations Processing system (e-QIP) and Optional Form 306: Declaration for Federal Employment (OF306)—and must have their fingerprints processed, in which the Federal Bureau of Investigations conducts a review for any prior arrest or convictions. Once completed, the forms are reviewed by the Census Investigative Services (CIS) where OPM-trained staff make either a favorable, unfavorable, or inconclusive precheck employment determination. According to Bureau officials, certain crimes, for example violent crimes, automatically exclude the applicant from further consideration. If the determination is inconclusive, then CIS is to send the form to the Office of Employee Relations to make a favorable or unfavorable determination. All favorable determinations are then sent to OPM for adjudication with a full background check (see fig. 6). According to Bureau officials, in December of 2018, they began to encounter a backlog of pre-employment background checks Bureau-wide as they began hiring some 800 recruiting assistants and about 1,970 office staff for the first wave of 39 ACO openings. As of March 21, 2019, Bureau officials told us that Bureau-wide there were 7,092 background clearances pending, of which, 4,900 were for field positions. In response to the backlog, Bureau leadership said it created a team to determine the cause of the backlog and started having weekly meetings to prioritize which job positions needed to be cleared first. Bureau officials stated that the delays arose, in part, because a significant number of applicants did not completely or correctly fill out the e-QIP form. This, they said, coupled with the increase in required pre- employment background checks, resulted in a growing backlog of clearances for which the Bureau did not have the resources to clear. In response, in February 2019, the Bureau began to bring on, through a combination of new hires and reassignments, about 130 temporary staff. New staff was assigned to either review the forms for accuracy and completeness prior to being submitted to the CIS office, or help the CIS offices conduct the pre-employment background checks. Additionally, Bureau officials told us that they meet weekly to reprioritize job positions for the clearance process. The CIS office is to process background checks for all Census employees requiring them, including decennial census field staff, decennial census contractors, and staff needed for nondecennial census surveys in headquarters and in the field. According to Bureau officials, the decennial census takes precedence and within the decennial census positions are also prioritized. For example, in January 2019, the 800 recruiting assistants were given priority and now the hiring of 1,501 partnership specialists has been given priority. Bureau officials told us that in December 2018 they were processing 110 background checks a week, and have set a goal that each CIS analyst process 25 pre-employment packages a week. There are 40 analysts on board, giving the Bureau the ability to process 1,000 pre-employment background check packages a week. Bureau officials also told us that they anticipate the clearance process for the positions of enumerator/lister and census field supervisor will not experience the same delays because these positions only require fingerprint processing, which is quicker. According to Bureau officials, these results can be made available within 3 hours. Moreover, although the Bureau has taken steps to address the backlog, the bulk of pre-employment background clearances has yet to be processed and Bureau officials told us that they remain concerned. In the coming months, the Bureau will need to conduct background checks for an additional 3,991 recruiting assistants and about 10,300 office staff for the remaining 209 offices. We will continue to monitor the backlog of background clearances through our ongoing work. Although the Bureau has exceeded its recruiting goals for early operations, recruiting a sufficient number of job applicants for the job of partnership specialist is a challenge. Bureau officials told us that a robust economy and low unemployment rate have resulted in a smaller pool of applicants for that position. For example, as part of its 2020 Census efforts, the Bureau had planned to hire 1,181 partnership specialists by May 1, 2019 and 1,501 partnership specialists by June 30, 2019, to help increase awareness and participation in the 2020 Census in minority communities and hard-to-reach populations. The Bureau did not meet its goal to hire 1,181 partnership specialists by May 1, 2019. To hire sufficient partnership staff, Bureau officials told us they have an “open and continuous” posting for partnership specialist positions instead of discrete individual job postings, and they are selecting two candidates from each certification list of qualified applicants. Moreover, Census leadership tracks the weekly progress of the partnership specialist positions. As of July 9, 2019, the Bureau’s latest biweekly reporting indicated that it had hired 813 partnership specialists as of June 22, 2019. Moreover, as of July 10, 2019 Bureau officials told us that another 830 applicants were waiting to have their background checks completed. According to Bureau officials hiring data are based on payroll dates generated biweekly, while background check data are tracked internally. Therefore, according to Bureau officials, more current hiring data were not available as of July 10, 2019 to indicate whether the Bureau had met its June 30 hiring goal. Hiring partnership specialists in a timely manner is key to the Bureau’s ability to carry out its planned outreach efforts, especially for hard-to- count communities. In addition, several RCC officials said the pay rate and the low unemployment rate in some ACO locations initially affected their ability to recruit well-qualified staff for office positions. Atlanta RCC officials stated it was challenging to recruit managers in the Gainesville, Florida, area. According to Bureau officials, the pay rate was too low and potential recruits were seeking employment elsewhere. The Bureau increased the managers’ pay rate to be more competitive for the area. Philadelphia RCC officials stated that in rural ACO locations the pay rate is lower and potential recruits would rather travel to the metro areas to get the higher pay rates offered there. The Denver RCC reported that low unemployment rates throughout the regions make recruiting difficult, and that Census enumerators jobs are not as competitive with many other wages offered in the region. The Los Angeles RCC reported having difficulty recruiting local applicants in high-cost areas like Beverly Hills, the San Francisco Bay Area, and Silicon Valley. Bureau headquarters officials acknowledge that some ACO locations have experienced some recruiting challenges, but said that the RCCs were ultimately able to fill the office positions. Headquarters officials stated that their pay rates either match or exceed the competitive pay rate in the majority of the ACO locations. According to Bureau headquarters officials, regional offices that may be experiencing challenges recruiting staff must demonstrate or prove that the pay rate for a specific ACO is causing difficulty recruiting. The Field Division is responsible for approving or denying the request to adjust pay. For the 2010 Census, the Bureau reported 124 requests for pay rate adjustments, of which 64 were approved. The Bureau stated that it will continue to monitor how low unemployment affects its ability to recruit and hire. The Bureau reports that the demographic and cultural makeup of the United States continues to increase in complexity, including a growing number of households and individuals whose proficiency in English is limited. Language barriers could make it difficult to enumerate these households, whose members may have varying levels of comfort with government involvement. Several RCC officials also mentioned that language barriers could impact their recruiting efforts: Both the Los Angeles and New York RCCs reported it is hard to recruit in immigrant communities where residents speak a foreign language or dialects, and often have no organizational infrastructure (such as associations of individuals of the same national origin, print news media, or radio). The New York RCC reported challenges in locating applicants who are bilingual in English and other languages such as Chinese, Russian, Arabic, Korean, Creole, Polish, Portuguese, Bengali, Urdu, Punjabi, Gujarati, Hindi, and Hebrew, as well as Yiddish and African languages. The Atlanta RCC reported challenges related to the diverse language needs (e.g., Spanish, Chinese, Vietnamese, Creole, Portuguese, etc.) in south and central Florida. The Chicago RCC reported recruiting outreach challenges in urban areas, including Chicago, Indianapolis, Detroit, Minneapolis/St. Paul, St. Louis, and Kansas City, that have higher minority and immigrant populations as well as in rural areas with increasing diversity. Bureau officials responded that later this fall, in preparation for their peak operations effort, they will begin to focus recruiting efforts on foreign language recruiting. Specifically, partnership and recruiting staff plan to work with partners and advertise jobs locally (at the grassroots level) in places where persons with these skills are likely to look to ensure they are meeting recruiting goals in those areas. For the 2020 Census, the Bureau is following its plans to use a blended training approach combining technology-assisted training with classroom instruction. According to Bureau planning documents, on the first day of in-person classroom training, the Bureau will provide orientation information and issue devices that trainees will use to conduct census operations. The Bureau plans to use local institutions such as schools, libraries, churches, and fire halls to host training. ACO staff are to coordinate the training location setup, device deliveries to training sites, and manage other logistics for large-scale field staff training. After the first day of training, field staff will spend the next 4 to 6 days (depending on the operation) completing at-home training online using their own personal device at their own pace. This training will include, for example, operation-specific skills, use of the data collection device (smart phone or tablet), and general field processes. Trainees who complete the online portion of the training program will return to the classroom to practice what they learned through role-playing, mock interviews, or live cases (for listing operations) facilitated by managers or supervisors. According to Bureau officials, employees will also have access to just-in- time training materials on their devices for use in the field. The Bureau encountered a number of challenges in implementing and testing its blended training approach, but is taking steps to mitigate those challenges. Specifically, during the 2018 End-to-End Test, the Bureau (1) experienced problems with the proper recording of online training scores for census staff, (2) was unable to test online training for one of its operations because the operation was added late, and (3) encountered challenges with census staff not always having access to the internet, which is required to complete the training. The 2018 End-to-End Test of address canvassing and nonresponse follow-up training revealed some technical challenges in using the Learning Management System. The Learning Management System is the online training system for the 2020 Census; it contains online training modules and tracks final assessment scores and training certifications. In February 2019, the Department of Commerce (Commerce) Office of the Inspector General (OIG) noted that during the address canvassing operation there was no final assessment scores recorded for 23 trained listers. The Bureau was also unable to provide documentation that another three lister trainees who failed the final assessment had been observed by their supervisor before being permitted to work. Bureau officials said they provided an action plan to the Commerce OIG in April 2019. According to Bureau officials, the action plan has not been finalized because they are incorporating changes to the action plan based on Commerce OIG comments. In December 2018, we reported that roughly 100 enumerator trainees in the nonresponse follow-up operation were unable to transmit their final test scores because the Learning Management System had an erroneous setting. According to Bureau officials, this problem delayed the start of unsupervised work for these otherwise-qualified enumerator trainees by an average of 2 days per trainee, and resulted in the attrition of some who were able to quickly find other work. Bureau officials reported that they have fixed the system setting. Moreover, according to Bureau officials, they have also developed an alternative means to certify training by incorporating the employee final assessment into the final day of classroom training. According to Bureau officials, Update Leave online training was not tested during the 2018 End-to-End Test due to the late addition of the operation to the 2020 Census design. Officials told us that the Update Leave operation was approved in May 2017, leaving just 10 months for the development team to create and implement software and the systems to support this field operation for the End-to-End Test. This left no time to develop online training that would be ready for the End-to-End Test in March 2018. Therefore, the Bureau classroom-trained headquarters staff instead of temporary field staff for the operation. According to the Bureau’s risk register, the utilization of Bureau headquarters staff did not properly simulate training conditions or staff characteristics in which new employees have no prior knowledge of census operations. Therefore, the 2018 End-to-End Test did not allow for proper training feedback or the capture of lessons learned with regard to temporary staff or the mode of training. According to Bureau officials, the Bureau plans to conduct scheduled dry runs of training in September 2019 to collect feedback and, if necessary, make changes to Update Leave-specific training. In June 2018, we reported that some listers had difficulty accessing the internet to take online training for address canvassing. According to the Bureau, in addition to the Bureau-provided laptop, listers also needed a personal home computer or laptop and internet access at their home to complete the training. However, while the Bureau reported that listers had access to a personal computer to complete the training, we found some listers did not have access to the internet at their home and had to find workarounds to access the training. We recommended that the Bureau finalize plans for alternate training locations in areas where internet access is a barrier to completing training. The Bureau took action and in March 2019 finalized its plans for identifying alternate training locations in areas where internet access is a barrier to completing training. Specifically, Bureau officials told us that in areas of known low connectivity rates, regional staff will identify sites that trainees can access to complete online components of the training. In addition, the Bureau provided us with a training module for identifying training field staff locations that emphasized training sites need to be located in areas with a good cellular connection and also have access to the internet. Effective training can enhance the Bureau’s ability to attract and retain employees with the skills and competencies needed to conduct the 2020 Census. Our Guide for Assessing Strategic Training and Development Efforts in the Federal Government describes components for developing effective training in the federal government. Our strategic training guide identifies four phases of the training—planning, design/development, implementation, and evaluation. We assessed the Bureau’s training approach and found that it generally aligned with selected leading practices. This report includes the design/development and evaluation phases of training. We did not assess the implementation phase because field staff training had not yet begun during our audit, and we did not assess the planning phase because practices in that phase are more applicable to agency-wide rather than program-specific training development. The design/development phase involves identifying specific training and development initiatives that the agency will use, along with other strategies, to improve individual and agency performance. According to the guide, well-designed training and development programs are linked to agency goals and to the organizational, occupational, and individual skills and competencies needed for the agency to perform effectively. Moreover, in response to emerging demands and the increasing availability of new technologies, agencies, including the Bureau for the 2020 Census, are faced with the challenge of choosing the optimal mix for the specific purpose and situation from a wide range of mechanisms, including classroom and online learning as well as structured on-the-job experiences (see fig. 7). In developing its training approach we found the Bureau met all five selected leading practices related to design/development. Specifically, Bureau training aligned with achieving results for the Bureau’s re- engineered field operations. Specifically, the Bureau has a formal online training program that uses the Learning Management System as a control mechanism to provide and record training results for all 2020 Census field staff who take online training. The Bureau’s training program is integrated with other strategies to improve performance such as building team relationships. For example, the training includes modules for supervisors that focus on guiding and motivating employees, communicating effectively, and resolving conduct issues. To ensure the training is properly integrated with device issuance, for larger scale operations, the Bureau plans to stagger training sessions to help ensure there is the necessary support during the first day of training when census field staff receive their devices. The Bureau also plans to use different training delivery mechanisms. For example, the Bureau will use a blended training approach which includes a mix of computer-based and instructor-led classroom training. The Bureau has measures of effectiveness in its course design. The Bureau relied on an in-house training development team that worked with the data collection operations staff to develop learning objectives. We found that that the Bureau has procedures to incorporate feedback. Specifically, the Bureau incorporated lessons learned from previous census tests, such as the refinement of procedures for reassigning work in the field and emphasizing the importance of knocking on doors to find a proxy respondent during the nonresponse follow-up operation. Finally, the Bureau’s training documents contained goals for achieving results for its new training approach. Specifically, the Operational Assessment Study Plan for Recruiting, Onboarding, and Training for the 2018 End-to-End Test contained the following measures of success for training—reduce cost and increase efficiency over what was reported in 2010. In developing its evaluation phase for training, the Bureau met five of six selected leading practices and partially met one leading practice. The evaluation phase involves assessing the extent to which training and development efforts contribute to improved performance and results. We have previously found that it is increasingly important for agencies to be able to evaluate their training and development programs, and demonstrate how these efforts help develop employees and improve the agencies’ performance (see fig. 8). Overall, we found that the Bureau has a robust evaluation plan for the 2020 Census that gathers data from multiple sources. For example, The Bureau has a plan to evaluate the effectiveness of training for the 2020 Census. Specifically, operational and assessment study plans set priorities for evaluations and cover the methods, timing, and responsibilities for data collection, including assessment questions, metrics, data sources and expected delivery dates, and division responsibilities. The Bureau has an analytical approach to assess training programs. For example, the Field Decennial Data Collection Training Branch has developed three separate training evaluation surveys which will be administered to field staff through the Learning Management System. The three evaluations provide training feedback after the completion of the online training; after the completion of the classroom training; and near the completion of the operation. According to the Bureau, these assessments will help determine the effectiveness of training. The Bureau incorporated evaluation feedback into planning and design of training. For example, the Bureau held debrief sessions with census workers during the 2018 End-to-End Test and told us they were also incorporating recommendations made by a training vendor. Feedback from the 2018 End-to-End Test is being used to inform training for the 2020 Census. The Bureau incorporates different perspectives in assessing the impact of training. Bureau officials stated that they incorporated feedback from a variety of stakeholders when evaluating the effectiveness of its training during testing, including participant debriefs and evaluations from vendors. As previously discussed, the Bureau used three different surveys at different points in time to evaluate training, and relied on debrief sessions with census managers and staff in the field. Bureau officials said they considered the training methods of another organization. For example, Bureau officials told us they used training vendors that followed requirements, including e-learning content developed by the Department of Defense. However, we found that the Bureau does not have performance goals or measures for training in its corresponding study plan for the 2020 Census. Specifically, we found that in the Detailed Operational Plan for the Field Infrastructure and Decennial Logistics Management Operations for the 2020 Census, the Bureau had planned to include the following success measures: Process Measures that indicate how well the process works, typically including measures related to completion dates, rates, and productivity rates. Cost Measures that drive the cost of the operation and comparisons of actual costs to planned budgets. Costs can include workload as well as different types of resource costs. Quality Measures, such as, the results of the operation, typically including rework rates, and error rates. However, according to Bureau officials they decided not to include the measures from the study plan for training because the study plan was intended to provide descriptive information about operations rather than evaluate them. We have previously reported that a fundamental element in an organization’s efforts to manage for results is its ability to set meaningful goals for performance and to measure progress toward those goals. Thus, without specific performance goals and measures for its new blended training approach that considers cost and benefits when compared to 2010, the Bureau will not be able to determine whether its blended training approach reduced costs or increased efficiency. Moreover, not having goals and measures in place could inhibit the Bureau’s ability to develop meaningful lessons learned from the 2020 Census. Bureau officials agreed and stated they will consider including goals and measures on cost and efficiency in its plans; however, the Bureau has not yet provided us with documentation to reflect the goals and measures it will use to evaluate training, and has no time frame for doing so. Training for in-field address canvassing operation will begin in July 2019. Having performance goals and measures will help the Bureau assess the impact of its new training approach on cost, quality, and resources expended. Successfully carrying out the thousands of activities needed to complete an accurate, cost-effective head count on schedule is an enormous and challenging task. However, for those activities we examined, the Bureau appears to be positioned to carry them out as planned, if implemented properly. While Bureau officials acknowledged there were some early delays when regions were trying to find office space and acquire leases, they said that the deadlines for the later phases of construction allow extra time—giving them a chance to make up lost time. Regarding recruiting and hiring, the Bureau was exceeding its recruiting goals for early operations, but identified challenges in areas such as promptly completing background checks, hiring in a time of low unemployment, and overcoming language barriers. Moreover, although the Bureau has exceeded its recruiting goal for early operations, recruiting a sufficient number of job applicants for partnership specialist is a challenge. The Bureau’s continued response to and management of these challenges will be important as it begins recruiting for its peak operation efforts later this fall. The Bureau has generally followed its training plans for 2020, but has opportunities to improve its ability to evaluate training efforts. The Bureau notes that the blended training approach is intended to maximize trainee learning and on the job performance during the 2020 Census. However, 2020 Census documents do not contain performance goals or measures for determining the cost and benefits of the training when compared to 2010. Revising plans to include goals and measures will better position the Bureau to determine how its blended training approach will impact the cost, quality, and resources expended on the 2020 Census. We recommend that the Secretary of Commerce direct the U.S. Census Bureau to revise plans to include goals and measures for assessing the cost and benefits of the Bureau’s new blended training approach. These measures might include, but are not limited to, measures of cost, quality, and resources associated with training when compared to 2010. (Recommendation 1) We provided a draft of this report to the Secretary of Commerce. In its written comments, reproduced in appendix II, the Department of Commerce agreed with our findings and recommendation and said it would develop an action plan to address our recommendation. The Census Bureau also provided technical comments, which we incorporated. We are sending copies of this report to the Secretary of Commerce, the Under Secretary of Economic Affairs, the Director of the U.S. Census Bureau, and interested congressional committees. The report also is available at no charge on GAO’s website at http://www.gao.gov. If you have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. This report assesses the extent to which the Census Bureau (Bureau) is following its plans for space acquisition, recruiting and hiring, and training. For all of our objectives, we reviewed current Bureau planning documents and schedules, and interviewed Bureau officials including officials at the Bureau’s six regional offices. To assess the Bureau’s progress in opening area census offices (ACO), we obtained and reviewed current Bureau leasing agreement information and construction (meaning renovations such as new electrical layouts, heating, ventilation, and air conditioning) and deployment information. We also gathered information on the General Services Administration’s role in obtaining office space. To determine whether the Bureau is on track, we compared the current status of opening, construction, and deployment of ACOs to the Bureau’s plans, schedules, and timelines, and identified differences for follow-up with Bureau officials. We also reviewed a randomly selected nongeneralizable sample of ACO files at the Philadelphia RCC to determine whether justification was included when changes to ACO locations occurred. To determine the extent to which the Bureau is following its field hiring and recruiting strategy for the 2020 Census, we reviewed Bureau documentation regarding its strategy for recruiting and hiring temporary field staff for the 2020 Census. We also reviewed output and analysis from relevant Bureau human resources systems/databases, such as MOJO Recruit. We interviewed Bureau officials in both headquarters and the field who are knowledgeable about and responsible for recruiting and hiring temporary field staff to determine the extent to which the Bureau is meeting its recruiting and hiring goals, to describe their perspectives on any challenges facing the 2020 Census, and to understand the Bureau’s actions to mitigate any challenges. To understand changes from 2010, we compared the 2010 Census recruiting and hiring plans to those of the 2020 Census to determine differences, and interviewed Bureau officials to discuss what drove these changes. Finally, to determine the extent to which the Bureau has followed its plans for training field staff, and whether this training approach is consistent with selected leading practices, we examined relevant documents and interviewed Bureau officials to determine the Bureau’s planned approach for training, lessons learned from prior Census tests, the extent to which the Bureau is incorporating lessons learned as a result of its own testing, and what changes to training need to be made before the start of 2020 field operations. Additionally, we interviewed Bureau officials responsible for developing training curriculum to understand how training was developed (e.g. what courses to develop, challenges to using technology, etc.). We also reviewed federal guidance and our prior reports, and selected 11 leading practices from GAO’s Guide for Assessing Strategic Training and Development Efforts in the Federal Government (GAO-04-546G) as leading practices for training. Our strategic training guide identifies four phases of the training development process (planning/analysis, design/development, implementation, and evaluation). We assessed the approach against leading practices in two of these phases: design/development and evaluation. We did not assess the implementation phase because field staff training for the 2020 Census had not yet begun during our audit, and we did not assess the planning/analysis phase because practices in that phase are more applicable to agency-wide rather than program-specific training development, and focus on full-time permanent employees rather than temporary employees. Moreover, within the design/development phase and evaluation phase, we did not assess all best practices because some of those best practices were also more applicable to agency-wide rather than program-specific training development, or we had already evaluated such practices as cost. Moreover, this report primarily focuses on training for the address canvassing and nonresponse follow-up operations. We then compared the Bureau’s training approach to those leading practices and identified practices being followed and any differences. We conducted this performance audit from August 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Other key contributors to this report include Lisa Pearson, Assistant Director; Timothy Wexler, Analyst-in-Charge; Mark Abraham; Michael Bechetti; Jessica Blackband; James Cook; Rob Gebhart; Kerstin Hudon; Kayla Robinson; and Cynthia Saunders.", "summary": "The decennial census is a crucial, constitutionally mandated activity with immutable deadlines. To meet these statutory deadlines, the Bureau carries out thousands of activities that need to be successfully completed on schedule for an accurate, cost-effective head count. These activities include opening area census offices, recruiting and hiring a large temporary workforce, and training that workforce. GAO was asked to review the Bureau's plans for critical logistical support activities. This report (1) assesses the Bureau's progress in opening area census offices; (2) determines the extent to which the Bureau is following its field hiring and recruiting strategy for the 2020 Census; and (3) determines the extent to which the Bureau has followed its plans for training field staff, and whether this training approach is consistent with selected leading practices. To assess the extent to which the Bureau is following its plans for opening area census offices, recruiting and hiring, and training, GAO reviewed current Bureau planning documents and schedules, and interviewed Bureau officials, including officials at the Bureau's six regional offices. GAO used its guide to training ( GAO-04-546G ) as criteria for selected leading practices. To help control the cost of the 2020 Census while maintaining accuracy, the Census Bureau (Bureau) is making significant changes in three areas—office space, recruiting and hiring, and training—compared to prior decennials. The Bureau is reducing its use of office space, hiring fewer census field staff, and adopting a blended training approach of instructor-led, computer-based, and hands-on training (see figure). GAO found that the the Bureau generally appears to be positioned to carry out these activities as planned, if implemented properly. Opening offices. While the Bureau experienced early delays when regions were trying to find office space and acquire leases, Bureau officials said that the deadlines for the later phases of renovations will allow them to make up time lost. As of June 2019, there were signed leases for 247 of 248 offices. Recruiting and hiring. As of June 2019, the Bureau was exceeding its recruiting goals for early operations, but identified challenges in areas such as completing background checks and hiring during low unemployment, especially for partnership specialist positions. GAO will continue to monitor these challenges, as recruiting and hiring for the census continues. Training. The Bureau generally followed its training plans for 2020 and generally followed selected leading practices for its training approach. However, GAO found that the Bureau does not have goals and performance measures for evaluating its new training approach. Without goals and performance measures the Bureau will not be able to accurately assess the cost and benefits of its new training approach. The Secretary of Commerce should direct the U.S. Census Bureau to revise plans to include goals and performance measures for evaluating its new training approach. The Department of Commerce agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "In 2017, three sequential hurricanes—Harvey, Irma, and Maria—created an unprecedented demand for federal disaster response and recovery resources. According to the Federal Emergency Management Agency (FEMA), these hurricanes ranked among the top five costliest on record, costing $125 billion (Harvey); $90 billion (Maria); and $50 billion (Irma). As a result of these storms, Florida, Texas, and Puerto Rico faced hardships, including devastation to infrastructure, such as highways and bridges. The island of Puerto Rico in particular was severely affected, which created multiple challenges for federal response efforts. Specifically, within a 2-week period Puerto Rico was hit by both hurricanes Irma and Maria, resulting in power outages that lasted up to 11 months and the need for commodities, such as food and water, and requiring one of the largest recovery efforts in history. The federal response was complicated by several factors, including the remoteness of the island, limited local preparedness, outdated infrastructure, and workforce capacity constraints. The Emergency Relief Program provides assistance to repair or reconstruct highways and bridges on federal-aid highways and roads and bridges on federally owned public lands that have sustained serious damage from natural disasters or catastrophic failures. FEMA is responsible for providing funds to repair and replace roadways damaged as a result of disasters that are not eligible for federal-aid highway funding. For natural disasters or other events to be eligible for emergency relief funding, the President must declare the event to be an “emergency” or a “major disaster” under the Robert T. Stafford Disaster Relief and Emergency Assistance Act, or the governor must declare an emergency with the concurrence of the Secretary of Transportation. Damage to highways must be severe, occur over a wide area, and result in unusually high expenses to the highway agency. Congress has provided funds for highway emergency relief since at least 1938 and, since 1972, has authorized $100 million annually in “contract authority” for the Emergency Relief Program to be paid from the Highway Trust Fund. Accordingly, FHWA may obligate up to $100 million of funds from the Highway Trust Fund in any one fiscal year for the program. Congress also regularly provides funds to the Emergency Relief Program from general revenues through supplemental appropriations. Most recently, Congress passed the Bipartisan Budget Act of 2018 in February 2018, and the Additional Supplemental Appropriations for Disaster Relief Act, 2019 in June 2019, which included more than $3 billion for the FHWA Emergency Relief Program to repair damages caused by a number of natural disasters. According to FHWA officials, these funds will be used to address damaged related to the 2017 hurricanes. FHWA’s Emergency Relief Program regulations further define policies for the program and the eligibility requirements for selecting projects. These regulations state that emergency relief funds are not intended to correct preexisting deficiencies or duplicate assistance available under another federal program or compensation from insurance or other sources. Emergency relief projects are to be promptly constructed, and construction funds must be obligated within two years (i.e., by the end of the second fiscal year following the disaster) unless suitable justification is provided to FHWA. Emergency relief regulations specify the activities that emergency relief funds may be used for as well as those activities they may not be used for, such as reconstruction of facilities affected by long-term, predictable developing situations or deficient bridges scheduled for replacement with other funds. Because the statute and its regulations are, by necessity, fairly broad, FHWA publishes guidance to further assist the agency in administering the Emergency Relief Program. The Emergency Relief Manual, updated in 2013, is a guide for FHWA and state and local agency personnel for requesting, obtaining, and administering emergency relief funds. The manual provides additional information and examples of the types of activities and projects that are both eligible and ineligible for funding, the process for states to apply for emergency relief funding, and the documents and reports that are required to be prepared. FHWA’s Emergency Relief Order, issued in 2016, further defines the application and review process and the roles and responsibilities of FHWA and state personnel. As with other federal-aid highway programs, the Emergency Relief Program is a partnership in which states plan and execute projects to complete necessary repairs, and FHWA provides assistance to states in applying for funds and conducts oversight to determine eligibility and ensure that federal requirements are met. States and territories are required to conduct damage inspections, submit documentation to their respective FHWA division office to determine if repairs are eligible for federal funds, enter into project agreements, and complete final project inspections. The FHWA division office is responsible for reviewing damage inspections to determine whether proposed projects are eligible for emergency relief funds. FHWA headquarters officials use the information collected from these inspections to allocate funds to each state or territory for particular events; division offices obligate those funds and ultimately reimburses the states for allowable expenses. The Emergency Relief Program’s authorizing statute and FHWA’s regulations and guidance distinguish between federal share payable for emergency and permanent repairs. Specifically, according to FHWA regulations, emergency repairs are undertaken during or immediately after a disaster to restore essential traffic, minimize the extent of damage, or protect the remaining facilities. Emergency repairs are eligible to receive 100 percent federal reimbursement if they are accomplished within 180 days of the disaster. By statute, this deadline may be extended taking into consideration any delay in the ability of the state to access damaged facilities to evaluate damage and the cost of repair. FHWA and federal regulations also state that emergency repairs can be completed by state and local maintenance workforces, and qualify for categorical exclusions from the National Environmental and Policy Act’s (NEPA) requirements. FHWA’s Emergency Relief Manual further characterizes emergency repairs as repairs that can be completed relatively quickly, may be temporary in nature, and typically require little preliminary engineering or design effort, e.g., erecting barricades and detour signs. States and local transportation agencies may begin emergency repairs without prior FHWA authorization. Permanent repairs are undertaken after the occurrence of a disaster to restore the highway to its pre-disaster conditions. Permanent repairs receive a federal share, between 80 and 90 percent, depending on the type of roadway being repaired. However, in response to the level of devastation in Puerto Rico, Congress provided a 100 percent federal share for all emergency relief projects, including permanent repairs, necessary to address damage caused by hurricanes Irma and Maria in Puerto Rico. FHWA’s regulations state that permanent repairs are to be done through a competitively bid contract, unless the state demonstrates that another method is more cost effective (e.g., the use of abbreviated plans or a shortened advertisement period). In addition, many, but not all, permanent repairs meet the criteria for categorical exclusions from NEPA’s requirements. FHWA’s Emergency Relief Manual indicates that typically permanent repairs (1) should have obligated funds for construction within 2 years, (2) require the development of plans, specifications, and estimates, and (3) must receive prior FHWA authorization. Our prior work has raised concerns about FHWA’s management and oversight of the Emergency Relief Program. In 2007 we reported on the expanding scope of eligible activities funded by the Emergency Relief Program over time, resulting in projects that went beyond the original intent of the program. We recommended to FHWA and suggested that Congress consider tightening the program’s eligibility standards, but this recommendation has not been implemented and FHWA does not plan to do so. In 2012, we raised concerns about FHWA’s partnership relationship with the states, particularly its oversight of the Emergency Relief Program, which we first reported in November 2011. For example, we were unable to determine the basis of FHWA’s eligibility decisions on 81 emergency relief projects representing $193 million in federal funds because of missing or incomplete documentation. In addition, we identified cases where FHWA showed a lack of independence in decisions, placing its partners’ interests above federal interests. For example, FHWA allowed two states to retain unused Emergency Relief Program allocations to fund new emergencies, despite FHWA’s policy that these funds are made available to other states with potentially higher- priority emergencies. We concluded that while FHWA’s partnership relationship with the states yields benefits such as proactively identifying issues before they become problems, it also poses risks. Thus we recommended that FHWA develop a strategy to mitigate these risks. In March 2014, FHWA announced it had established an enhanced risk- based oversight approach that, while not targeting the specific risks we identified related to state partnerships, addressed the intent of our recommendation to increase transparency and consistency. Following hurricanes Harvey, Irma, and Maria, state and local officials prepared damage assessments that identified more than 2,500 projects eligible for emergency relief funds costing approximately $1 billion. Projects range in size and cost from replacing signage and traffic signals to multi-million dollar bridge and highway repairs (see fig. 1). Following a number of natural disasters in 2017—including hurricanes Harvey, Irma, and Maria—Congress appropriated more than $1 billion to the Emergency Relief Program in February 2018 to help states repair and rebuild federal-aid highways. As of September 2019, FHWA has allocated $634 million to repair hurricane-related damage in Florida, Texas, and Puerto Rico. Specifically, immediately following the hurricanes in August, September, and November 2017 FHWA allocated $122.5 million in quick release funding to Florida, Texas, and Puerto Rico. In April 2018, FHWA allocated an additional $242 million to Florida, Puerto Rico, and Texas. Further, on February 6, 2019 FHWA allocated $130 million more to Puerto Rico for damages caused by hurricanes Irma and Maria (see fig. 2). FHWA subsequently de-allocated $69 million from Florida on February 27, 2019, because state officials determined the funds were no longer necessary for hurricane-related repairs. Most recently, in September 2019, FHWA allocated an additional $208 million to Puerto Rico. While the estimated repair costs exceed the amount of funds allocated by FHWA, officials stated that additional emergency relief funds are allocated and reimbursed approximately every 6 months and states and territories will be reimbursed for all eligible expenses related to hurricanes Harvey, Irma, and Maria as they are completed. These funding decisions are to be made as FHWA continues to review and approve projects and Congress appropriates additional funds. As we have noted in prior work, the $100 million in annual authorized funding has not been enough to meet the needs of the program. Therefore, states have relied on supplemental appropriations to fund repairs caused by natural disasters and catastrophic events. We identified a number of cases in which FHWA did not document decisions to classify emergency relief projects as emergency repairs (those necessary to restore essential traffic, undertaken during or immediately after a disaster and generally accomplished within 180 days) as opposed to permanent repairs (those undertaken to restore a facility to pre-disaster conditions). Specifically, 22 out of 25 emergency repair projects we reviewed—which account for approximately $50 million in emergency relief funds—did not include a documented justification for classifying repairs as an emergency repair instead of a permanent repair. In addition, out of approximately 1,200 eligible projects in Puerto Rico, FHWA officials reported undertaking 34 more than 180 days after the hurricanes and continuing to classify them as emergency repairs without documenting the basis for doing so. Without documentation it is not possible to definitively determine the justification for why projects were classified as emergency repairs and we identified at least three projects that may have been inappropriately classified because they (1) may not have been necessary to restore essential traffic, or (2) were not undertaken during or immediately after the disaster. For example: The Lynchburg Ferry ($10.7 million project in Texas). This project rebuilt the ferry docks and landings, which are used to transport up to 10 passenger vehicles at a time across the Houston Ship Channel (1,100 feet). FHWA classified the project as an emergency repair to restore essential traffic but did not document the basis for this decision. When asked, FHWA officials from the Texas Division stated that engineers used their professional judgment to determine that the ferry route provided essential traffic. It is not clear, however, that the ferry was necessary to restore traffic as several alternative routes were available immediately following the disaster on existing highways that service the same locations and typically result in faster travel times than the ferry (see fig. 3). According to officials, engineers did not assess these alternative routes and there is no requirement for them to do so. This project was a significant commitment of emergency relief funds, representing approximately 11 percent of the emergency relief funding Texas received in the aftermath of Hurricane Harvey. Because the project was classified as an emergency repair, Texas was permitted to use a non-competitive bidding process to solicit and hire contractors to complete the work, instead of a competitive bidding process designed to achieve the best possible price and quality of work. The project was completed within the required 180 day time frame required to receive 100 percent federal reimbursement. FHWA’s oversight of this project raises issues we cited in past work concerning its partnership with the states, namely putting the partner’s interest above federal interests. Had FHWA classified this project as a permanent repair instead of an emergency repair, state and local agencies would have been responsible for paying approximately $2.1 million in matching funds on the $10.7 million project. Moreover, prior to the hurricane, the ferry docks and landings were in poor condition and local officials were in the initial stages of planning a project to replace it, including hiring a consultant to identify potential sources of federal funds. Because substantive planning and design work had not yet been completed, this project was eligible for emergency relief funds, which, according to officials, resulted in a new, state-of-the-art facility. Ciales Bridge ($4.9 million project in Puerto Rico). This project will install a temporary 80 meter long bridge over the Rio Grande de Manati River. FHWA classified this project an emergency repair to restore essential traffic and extended the project beyond 180 days but did not document the basis for either decision, as described below. FHWA officials said that they were not aware of another route to carry essential traffic at the time they approved the emergency repair. However, we identified an alternative route on a nearby roadway that uses another bridge less than a mile away. When we asked officials about this nearby route, they said that it is not sufficient for essential traffic, because it is too narrow to safely accommodate two-way traffic, has load limitations, and lacks lighting and pavement markings. Officials stated that the temporary bridge was necessary to quickly restore essential traffic until a new permanent bridge could be built. However, construction on the temporary bridge is not planned for completion until October 2019—more than 2 years after Hurricane Maria hit, raising questions about whether an emergency situation exists and the project is needed to quickly restore essential traffic. FHWA also continued to classify this project as an emergency repair even though the contract for the project was not signed within 180 days after the emergency occurred and FHWA did not document the rationale for doing so. By statute, emergency repair projects must be accomplished within 180 days to receive a 100 percent federal share, but may be extended taking into consideration any delay in the ability of the state to access damaged facilities. According to FHWA officials in Puerto Rico, while division offices should document decisions regarding emergency repair projects, the statutory provision that projects can only be extended beyond 180 days if the damaged facilities are inaccessible does not apply to Puerto Rico because it is funded at a 100 percent federal share, and therefore, such a determination and documentation was not necessary. There are, however, statutory and regulatory provisions other than the percentage of costs covered by the federal government that apply to emergency projects, including contracting and environmental requirements. Because this project was classified as an emergency repair, officials used a bidding technique—called short-list bid—that limited the number of firms which were permitted to submit proposals. This project also received a categorical exclusion for emergencies and was not subject to further environmental review under NEPA. However, although these projects went forward, FHWA’s policy regarding time limits on the use of expedited contracting and environmental procedures is not clear. After we raised this and similar issues on other projects with FHWA, officials stated that the administration’s position was that emergency repair projects using expedited contracting and environmental procedures are only permitted within the first 180 days of a disaster. According to these officials, as a matter of policy, 180 days after the disaster is a “pencils down” moment when projects should be subject to permanent repair requirements, including environmental and contracting requirements. Officials acknowledged this policy is not well documented, and stated they planned to address this gap in future updates to program guidance. These updates—initially planned for 2019—have taken more time than anticipated and are currently planned for 2020, but officials were unable to provide a specific timeline. The classification of the project as an emergency repair raises questions about whether the project was an efficient use of federal funds. The $4.9 million temporary bridge involves considerable construction such as building footings with 5-million pounds of concrete and reinforced steel (see fig. 4) and, as stated previously, is not planned for completion until October 2019. FHWA officials stated this structure will be torn down within a couple of years and replaced by a $6.4 million permanent structure. PR-14 Bridge ($1.4 million project in Puerto Rico). This project will construct a temporary bridge across one of a few main routes on the south-central side of the island that is located in one of Puerto Rico’s mountainous municipalities that is rural and relatively sparsely populated. FHWA officials classified the temporary bridge as an emergency repair to restore essential traffic, including the transportation of people and commercial goods but did not document the basis for this decision. According to officials, this bridge was necessary to restore essential traffic because damage caused by the hurricane led to a reduction in the vehicle load limit from 5 tons to 3 tons. However, the basis for this determination is not clear since the bridge was never closed to traffic and a reduced load limit from 5 to 3 tons would not significantly affect the type of vehicle traffic able to safely cross the bridge. For example, the pre-existing 5-ton limit would have already prevented most types of ambulances and commercial trucks from using the bridge, and the 3-ton limit still permits most passenger vehicles and some types of light-duty trucks. In addition, according to officials, one of the reasons for installing a temporary bridge instead of waiting on the planned installation of a permanent bridge was to quickly restore traffic. However, the temporary bridge will not be completed until February 2020—almost 2 and a half years after the hurricanes, which raises questions about whether or not the project was necessary to quickly restore essential traffic. As with the Ciales Bridge, FHWA did not document the basis for classifying this project as an emergency repair even though it was undertaken more than 180 days after the emergency occurred. The project was contracted using a pre-existing contract and not competitively bid and received a categorical exclusion from NEPA requirements. Similar to the Ciales Bridge, this $1.4 million temporary bridge will be torn down within a couple of years and replaced by a $4.2 million permanent structure. While officials did not document decisions to classify emergency relief projects as emergency repairs, FHWA did improve the documentation of emergency relief projects in some areas since the last time we examined the program in 2011. Specifically, we found more consistent documentation of the onsite damage inspections, cost estimates, and FHWA oversight of eligibility determinations. For example, 39 out of 39 emergency relief projects we reviewed included photographs of the damage and a repair cost estimate; whereas, only 24 out of 83 projects we examined in 2011 included this information. According to Federal Internal Control Standards, to achieve objectives and identify and respond to risks, management should clearly document all transactions and significant events, and define objectives clearly, including specific terms so that they can easily be understood. FHWA did not clearly document transactions and significant events because: (1) in the case of classifying projects as emergency repairs, there is no requirement to do so, and (2) in the case of extending emergency repair projects in Puerto Rico, existing requirements did not apply. FHWA officials stated that these decisions were made as part of an ongoing dialogue between FHWA, the states, and Puerto Rico that is done through emails and in-person and telephone meetings. However, by not documenting emergency repair decisions, such as whether alternative strategies or repairs were considered and the rationale for classifying projects as emergency repairs after the emergency has passed, FHWA lacks definitive explanations for its decisions. This, in turn, raises questions as to whether those decisions were appropriate. When questioned about individual projects, including the examples in Texas and Puerto Rico previously discussed, officials often could not provide concrete rationales for these decisions. In addition, because guidance in the Emergency Relief Manual is intentionally flexible and written to apply to a wide range of circumstances, key terms are not clearly defined and easily understood and applied. This is particularly true for the term “essential traffic,” which is being broadly applied to provide support for repairs necessary to restore any type of traffic without fully considering potential alternatives. While FHWA’s manual generally describes projects to restore essential traffic (e.g., detours that relieve excess traffic directly attributable to the disaster), it does not discuss how to determine whether a project will relieve excess traffic or require officials to evaluate alternative routes. Moreover, FHWA’s guidance and policy are not clear on the time frames for when emergency repair projects must adhere to contracting and environmental requirements. This lack of clearly defined and easily understood terms in emergency relief guidance could result in FHWA inappropriately classifying projects as emergency repairs, which affects: the federal fiscal exposure in a disaster, the level of FHWA oversight because projects may begin without prior authorization, the extent to which projects must be competitively bid, and potentially the level of environmental review accorded a project. Moreover, unclear guidance increases the chances that program guidance could be inconsistently applied, potentially giving access to emergency relief funds to one state and not another. We identified several instances in which officials in one Division Office made emergency repair decisions that differed from another division office. For example, FHWA officials in Florida did not include highway finishes, such as pavement markings, as part of emergency repair projects, while officials in the Puerto Rico Division did. FHWA officials in Puerto Rico also reported that FHWA officials from different division offices who came to assist in the aftermath of the 2017 hurricanes had substantively different interpretations of emergency relief guidance, including how to define emergency repairs and what was and was not essential traffic. For many years, FHWA’s Emergency Relief Program has provided crucial funding to states and territories to rebuild transportation infrastructure, including in the aftermath of hurricanes Harvey, Irma, and Maria. The consecutive timing and scale of these disasters overwhelmed local, state, and territorial governments, and Puerto Rico was hit particularly hard. Given the level of devastation, it was imperative for the federal response to be quick and effective, and that essential services be quickly restored to help people rebuild and recover. However, it is not clear that emergency relief funds are always being used for the purposes intended or put to the highest use. In the absence of well-documented rationales for classifying projects, more clearly defined terms and circumstances for making these decisions, and time frames for accomplishing them, FHWA may have inappropriately classified projects as emergency repairs. While this represent a small percentage of projects undertaken in response to the 2017 hurricanes, FHWA’s actions may have resulted in the federal government forgoing millions of dollars in state contributions, thus increasing the federal fiscal exposure in disasters. Moreover, permitting projects to proceed under expedited contracting requirements many months after the disaster deprived the federal government of a valuable tool intended to ensure the best price for services it receives. Finally, in an environment where needs outweigh funding, multi-million dollar bridge projects are being constructed that will be torn down in a couple of years to make way for other multi-million dollar bridge projects. FHWA’s decision-making invites questions we have raised before about the partnership relationship between FHWA and the states. In high stress and politically sensitive situations like natural disasters in particular, the relationship could lead FHWA to put states’ interests before federal ones or give the appearance of having done so. If FHWA’s decisions are, in fact, appropriate, documentation and clearer guidance could reduce unnecessary skepticism, enhance transparency, and result in more effective use of limited resources. We are making the following two recommendations to FHWA: The Administrator of FHWA should require FHWA division offices to document the rationale for classifying projects as emergency repairs, such as a description of why an emergency repair is necessary and which alternative strategies or repairs were considered, and to more clearly define the circumstances under which projects are classified as emergency repairs, including what constitutes restoration of essential traffic. (Recommendation 1) The Administrator of FHWA should identify a specific timeline for clarifying the policy on the acceptable time frames for accomplishing emergency repair projects undertaken under expedited contracting and environmental requirements, and require FHWA division offices to document the rationale for decisions to extend projects beyond these time frames. (Recommendation 2) We provided a draft of this report to DOT for review and comment. In comments, reproduced in appendix II, DOT concurred with our recommendations. DOT also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretary of the Department of Transportation. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In 2011, we reported on how Federal Highway Administration (FHWA) officials applied Emergency Relief Program guidance to a selected group of projects that received funding. In that review, we selected a nongeneralizable sample of eligible Emergency Relief Program projects in three states—New York, Texas, and Washington—that matched criteria such as receiving more than $1 million in obligated federal funds and approval by FHWA between fiscal years 2007 through 2010. We reviewed those projects’ files to determine whether they included required or recommended documentation cited in federal statute, regulations, and FHWA program guidance. In our 2011 report, we found many instances of missing or incomplete documentation, such as required repair cost estimates, because FHWA lacked clear requirements for how states submitted and FHWA approved key project documentation, leading to FHWA division offices applying eligibility criteria differently. We recommended that FHWA standardize their procedures for reviewing emergency relief documentation and making eligibility decisions, including retaining damage inspection reports with detailed repair cost estimates. In response, FHWA issued an Order in February 2016 that included procedures to ensure that FHWA makes eligibility determinations consistently and transparently that we determined addressed our recommendation. To evaluate how FHWA officials applied Emergency Relief Program guidance to selected projects in recent emergency events and whether documentation had improved since our 2011 report, we conducted a file review of 39 nongeneralizable emergency relief projects—25 of which included emergency repairs—in Texas, Florida, and Puerto Rico. These projects, which FHWA determined were eligible for Emergency Relief Program funding, were necessary to repair damage caused by three 2017 hurricanes: Harvey, Irma, and Maria. The purpose of this review was to determine whether each project file included information showing the project met eligibility requirements or information required or recommended in federal statute, regulations, and FHWA program guidance. To select these 39 project files (13 projects each from Texas, Florida, and Puerto Rico), we used the following criteria: We reviewed those with the highest estimated cost to ensure the inclusion of projects likely to receive the most federal funds. The 39 project files we selected represented over 38 percent of Emergency Relief funds allocated to those three states for the 2017 hurricanes, as of February 2019. We selected a mix of road and bridge projects to ensure we reviewed a selection of projects that could include different types or amounts of documentation. States typically have more data and oversight processes in place for bridges than other roads, as most bridges are required to be inspected at least every 2 years. We selected a mix of a state and local agency projects to ensure we reviewed a selection of projects that may have been prepared with different levels of detail. Though state agencies ultimately submit all Emergency Relief Program requests to FHWA, local agencies prepare some of the paperwork for projects within their jurisdictions and could provide a different level of detail in their project files than state agencies. For each of the 39 projects in our review, the FHWA division offices in Texas, Florida, and Puerto Rico provided associated project files. Through discussions with state officials, we determined that FHWA’s Mobile Solution for Assessment and Reporting (MSAR) was sufficiently reliable for our purposes of obtaining documentation for file reviews for projects located in Texas. For Puerto Rico, because state officials acknowledged some files were not included in MSAR, we asked for state officials to directly send us additional documentation as needed. As Florida does not use MSAR to record project information or documentation, we asked for state officials to send us relevant project documentation directly. Project files from these locations included information on project type and estimated costs as well as other relevant documents, such as engineering reports, bridge inspection reports, or photographs of the damage. Two analysts reviewed those files for information that is required or recommended by statute or FHWA guidance. This information included much of the same information we had previously evaluated in our 2011 review. To conduct the review, one analyst reviewed the documentation provided by FHWA’s division offices and completed a data collection instrument, then a second analyst reviewed the same documentation to verify the results of that review. Afterwards, the two analysts discussed and resolved any discrepancies and questions. The analysts then analyzed and summarized the results for the 39 eligible projects of this review to determine whether each file included documentation for damage and cost information, emergency repair requirements, and eligibility determination, as detailed below: Damage and cost information: We reviewed whether the project file included a complete detailed damage inspection report (DDIR), which documents an on-site inspection of the damage. FHWA’s Emergency Relief Manual states that a complete DDIR should include a number of details including: the type of federal-aid highway, such as an interstate, freeway, or expressway; the average daily traffic or the typical traffic volume in a location over a 24-hour period; the nature or type of damage, such as a bridge collapse or landslide, and extent or amount of damage, such as fully or partially collapsed; a field site sketch or drawing that shows details of the damage site such as the width of the road or bridge; a total estimated cost for repair; and documentation related to an environmental review recommendation, which would include the potential effects of repairs on nearby species or waterways. For the 39 projects we included in our file review, we found that DDIR documentation generally improved compared to the 2011 review. For instance, each of the 39 projects included a DDIR, photographs of the damage, and the repair’s cost estimate; only 24 of the 83 eligible projects we reviewed in 2011 included each of those pieces of information. However, we found other recommended DDIR documentation to be lacking. For example, of the 39 projects in our review, 36 did not include Average Daily Traffic and 22 did not include the type of federal-aid highway. Figure 3 represents the results of our review of damage and cost information. Emergency repair requirements: We reviewed whether eligible emergency repair projects included a documented rationale or justification for classifying the project as an emergency repair instead of a permanent repair. As discussed in the body of this report, by statute, emergency repairs are repairs undertaken during or immediately after a disaster specifically to restore essential traffic, to minimize the extent of damage, or to protect the remaining facilities. As discussed in the body of this report, classifying a project as an emergency repair affects the percentage of costs covered by federal funds, level of FHWA oversight, and the extent to which environmental and contracting requirements apply. We found that of the 25 project files that included an emergency repair (out of the 39 in our review), 22 did not include a documented rationale or justification for classifying the project as an emergency repair instead of a permanent repair. Eligibility determination: We reviewed whether a representative of FHWA signed and recommended eligibility for Emergency Relief funding and whether the applicant or state representative signed and agreed with FHWA’s recommendation. The Emergency Relief Manual states that documentation should include an eligibility recommendation by an FHWA representative and acknowledgement of that recommendation by the applicant. For the 39 projects we included in our file review, we found that documentations of FHWA and applicant signatures generally improved compared to the 2011 review. In our current review, we found that the FHWA and applicant or state representatives signed each of the 39 eligible project files; in our 2011 review, only 36 of the 83 eligible projects included a signature from an FHWA representative and 47 of the 83 eligible projects included a signature from the applicant or state representative. In addition to the contact named above, Steve Cohen (Assistant Director), Matthew Cook (Analyst in Charge), Pedro Almoguera, Aditi Archer, Danielle Ellingston, Lauren Friedman, Kathryn Godfrey, Hannah Laufe, Leslie Locke, Cheryl Peterson, Malika Rice, Amy Rosewarne, and Elizabeth Wood made key contributions to this report.", "summary": "In 2017, hurricanes in Texas, Florida, and Puerto Rico caused $1 billion in estimated damage. FHWA's Emergency Relief Program provides funding for states to repair or reconstruct federal-aid highways damaged or destroyed by natural disasters, including funding for emergency and permanent repairs. As of September 2019, FHWA has allocated $634 million in federal funds to the two states and Puerto Rico. By statute, emergency repairs are undertaken during or immediately following a disaster to quickly restore essential traffic and minimize further damage. These repairs receive 100 percent federal reimbursement if accomplished within 180 days and may proceed under expedited contracting and environmental procedures. GAO was asked to evaluate the federal response to the 2017 disasters. This report assesses how FHWA applied program guidance to classify selected emergency relief projects, among other objectives. GAO visited 33 out of approximately 2,500 projects in Texas, Florida, and Puerto Rico; analyzed 25 emergency repair project files; and interviewed FHWA, state, and local government officials. GAO found that the Federal Highway Administration (FHWA) did not document the bases for decisions to classify projects as emergency repairs in 22 of the 25 project files reviewed. Without such documentation, it is not possible to definitively determine the justification for these decisions; GAO identified at least three projects that may have been inappropriately classified. For example, FHWA classified a $10.7 million ferry project in Lynchburg, Texas as an emergency repair to restore essential traffic. Several highways, however, were available immediately following the disaster that service the same locations and result in faster travel times than the ferry. FHWA guidance does not require officials to document decisions to classify projects as emergency repairs or clearly define what constitutes restoration of essential traffic. Designating projects as emergency repairs can increase the federal fiscal exposure in disasters. Had FHWA classified the ferry project as a permanent repair—instead of an emergency repair—the state would have been responsible for paying approximately $2.1 million in matching funds. GAO also identified two temporary bridge projects in Puerto Rico classified as emergency repairs even though (1) work did not start within180 days of a disaster, as generally required; (2) the bridges are not to be completed until late 2019 and early 2020; and (3) both are to be replaced by permanent bridges within a couple of years. Out of approximately 1,200 eligible projects in Puerto Rico, FHWA officials reported undertaking 34, including the two bridges GAO identified, after 180 days. Officials also stated they did not document the basis for continuing to classify these projects as emergency repairs. FHWA officials in Puerto Rico stated they were not required to complete repairs within the 180 day limit established in law because Congress exempted Puerto Rico from federal matching share requirements. Further, emergency repair projects are allowed to expedite contracting and environmental procedures. After GAO raised this issue with FHWA, the agency stated that emergency repair projects are only permitted to use these expedited procedures within the first 180 days. While officials stated they plan to update guidance to include this policy, there is no specific timeline for doing so. FHWA should (1) document decisions to classify projects as emergency repairs and more clearly define what constitutes restoration of essential traffic, and (2) identify a specific timeline for clarifying the policy on when expedited contracting and environmental procedures are permitted. DOT concurred with GAO's recommendations and provided technical comments that GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "The ASD(HA) serves as the principal advisor for all DOD health policies and programs. The ASD(HA) has the authority to issue DOD instructions, publications, and memorandums that implement policy approved by the Secretary of Defense or the Under Secretary of Defense for Personnel and Readiness and govern the management of DOD medical programs. The ASD(HA) also exercises authority, direction, and control over the President of the Uniformed Services University of the Health Sciences. Further, the ASD(HA) sets the maximum special and incentive pay amounts for all military physicians and dentists. The Army, the Navy, and the Air Force have the authority to recruit, train, and retain physicians and dentists. Currently, there is no joint DOD unit or process dedicated to recruiting medical students and accessing medical officers because recruiting and retention are the responsibility of the military departments. Each military department has its own organizational structure, responsibilities, and varying degrees of personnel resources for accessing physicians and dentists. The departments’ recruiting commands recruit medical and dental students into the scholarship program. In a separate process, the University recruits and admits a set number of medical students each year. Figure 1 shows the organizational structure of the Military Health System as it relates to the recruitment and retention of military physicians and dentists. DOD has two primary sources of recruitment for military physicians: the scholarship program and the University. DOD recruits most military dentists through the scholarship program. Participants in DOD’s scholarship program and the University accrue an active-duty service obligation in return for a tuition-free medical or dental education and certain financial benefits. Specifically, scholarship program participants enrolled in a civilian medical or dental school receive paid tuition, books and fees, and a monthly stipend. In some cases, participants are also offered an accession bonus. In exchange, scholarship program participants incur a 6- month active-duty service obligation for each 6 months of benefits received, with a 2-year minimum service obligation. Students at the University are enrolled in the DOD-sponsored medical school at no cost, enter active-duty service as medical students and receive the pay and benefits of an officer at the O-1 pay grade. In exchange, University medical students accrue a 7-year service obligation. Career paths for medical and dental school graduates can differ. For example, Army and Air Force medical school graduates typically become specialized before practicing medicine, while 55 percent of Navy physicians complete a General Medical Officer tour before becoming specialized, according to department officials. Moreover, dental school graduates typically practice as general dentists after completing licensure requirements before choosing to specialize. To become specialized, medical and dental school graduates apply to a medical or dental residency training program, which may require or include a 1-year internship, depending on the program or specialty. After residency, some physicians or dentists may decide to pursue further training, known as “fellowships,” in order to become subspecialists. For example, to become a cardiologist, a physician must complete an internal medicine residency followed by a cardiology fellowship. Residency training typically requires 3 to 7 years for physicians and 1 to 6 years for dentists. Fellowship training typically is 1 or more years in length for physicians and dentists. The required number of years depends on the specialty or subspecialty. After residency or fellowship training—hereafter referred to collectively as residency training—physicians and dentists become credentialed and privileged to practice the specialty or subspecialty that they trained in, and they are also eligible for board certification. Figure 2 portrays possible paths to becoming a military physician or dentist. As noted earlier, scholarship program medical and dental students incur 6 months of an active-duty service obligation for each 6 months of benefits received, with a 2-year minimum service obligation; University medical students accrue a 7-year service obligation. While training in a military residency program, residents receive the pay and benefits of an officer at the O-3 pay grade or higher, depending on prior years of service, and earn creditable years of service toward retirement. In exchange, participants incur an additional 6 months of an active-duty service obligation for each 6-months of residency training, with a minimum 2-year service obligation. However, according to DOD officials, the first year of postgraduate training (i.e., internship or 1 year of advanced education in general dentistry and general practice residency) does not accrue a service obligation and is considered obligation neutral. Currently, the two sets of obligations—the obligation for medical or dental school and the obligation for military residency training—are served concurrently, or at the same time, effectively resulting in the servicemember serving the longer of the two obligations. For example, a student who accepts a 4- year scholarship, trains in a 1-year internship, and then trains in a 4-year residency program will serve a total of 9 years. The first 5 years would be spent in internship and residency, and the final 4 years of this service would be spent discharging the active-duty service obligations concurrently (see figure 3, scenario 2). Depending on career path, years of active-duty service after completion of medical and dental school will vary (see figure 3). DOD’s measure of cash compensation, known as regular military compensation, includes the sum of basic pay, basic allowance for housing, basic allowance for subsistence, and the federal income tax advantage that accrues from the non-taxable nature of the allowances. For example, according to DOD, in 2017 the average married military officer at the pay grade of O-3 received annual regular military compensation of around $99,000. Specifically, this average officer received around $67,000 for basic pay, $24,000 for the basic allowance for housing, $3,000 for the basic allowance for subsistence, and a federal income tax advantage of $5,000. In addition to regular military compensation, physicians and dentists may be eligible for various special and incentive pays which vary depending upon their status as residents, their service obligations, and their specialty. During residency training, physicians and dentists are eligible for select medical or dental corps incentive pays. Upon completion of residency training, they become eligible for higher rates of incentive pay and, if they become board certified, for Board Certification Pay. After fulfilling their active-duty service obligations from medical or dental school and residency training, in addition to special and incentive pays already received, physicians and dentists become eligible for a multi-year retention bonus. Cash compensation for active-duty military physicians and dentists was generally less than the median compensation for private sector civilians in calendar year 2017 for most specialties we reviewed, including at key retention points. However, a substantial portion of the costs of DOD’s overall compensation package is comprised of deferred and noncash benefits provided to active-duty personnel, such as a pension in retirement and tuition-free medical and dental education, but the extent to which servicemembers value these benefits is difficult to determine. Cash compensation for active-duty military physicians and dentists varied depending on pay grade, specialty, and decisions to accept retention bonuses or other special and incentive pays, but was generally less than the median compensation for private sector civilians in calendar year 2017 for most specialties. Although we could not make direct comparisons of military and private sector civilian cash compensation by years of service or experience, we estimated the minimum and maximum military cash compensation for specialized active-duty physicians and dentists in pay grades O-3 to O-6, which represented more than 99 percent of military physicians and dentists in fiscal year 2018. Specifically, we found that the minimum military cash compensation for all 21 physician and 5 of 6 dental specialties we reviewed was less than the civilian median for all pay grades; and the maximum military cash compensation for 16 of 21 physician (see figure 4 below) and 5 of 6 dental specialties (see figure 5 below) we reviewed was also less than the civilian median for all pay grades. Therefore, for many of these specialties, even the most senior military physician and dentists (i.e., pay grade O-5 or higher) at the top of the pay range were estimated to receive cash compensation below the private sector civilian median. The minimum and maximum of total military cash compensation, by specialty and pay grade, and how these compare to reported private sector civilian cash compensation are presented in appendix II. Cash compensation for military physicians and dentists is generally less than private sector civilian compensation at key retention points. Specifically, we calculated 2017 cash compensation for military medical officers who completed their residency directly after medical school across 21 medical specialties and found that at their first unobligated year of service—after they fulfill their initial active-duty service obligations accrued from medical school and military residency training—all 21 specialties had cash compensation below the private sector civilian median. In addition, we found that all but one specialty (psychiatry) was less than the 20th percentile for private sector civilian compensation. Notably, nine specialties that DOD identified as critical trauma-related wartime specialties in 2019 were less than the 20th percentile. According to senior military department medical corps officials, the first unobligated year of service is a key point of retention for military physicians. A 2012 study of military physicians found that compensation had a large impact on the decision to remain in the military in the first unobligated year of service and just a small impact on retention in the years afterward. For DOD’s scholarship program participants, which constitute the majority of recruited military physicians, we estimate that initial service obligation fulfillment typically occurs about 4 years after successful completion of their residency, or at about 9 years of service. We also calculated cash compensation for military medical officers who (a) completed a 3-year General Medical Officer tour prior to specializing in a residency, or (b) attended the University and accrued a 7-year active- duty service obligation and found that all but three specialties (pediatrics, family medicine, and psychiatry) had cash compensation less than the 20th percentile for private sector civilian compensation, and all specialties were compensated below the median. We reviewed 2017 cash compensation for typical military dental officers across six dental specialties and found that at each of these retention points, military cash compensation was less than the median private sector civilian compensation, three of which were below the 25th percentile (orthodontics, endodontics, and periodontics). According to senior military department dental corps officials, two key points of retention for military dentists are (1) after they fulfill their scholarship service obligation by practicing as a general dentist for several years, and (2) after they have completed residency training for a dental specialty, such as orthodontics, and fulfill their residency service obligation. Unlike their physician counterparts, dental students typically do not begin residency immediately after graduation. According to military department dental corps Chiefs, dental student graduates generally complete a 1- year advanced education in general dentistry certificate, which does not incur a service obligation, then fulfill their dental school active-duty service obligation as general dentists before taking a general dentist’s retention bonus and beginning residency training. Cash compensation is just one factor that servicemembers may consider when making the decision to stay with or separate from the military. According to DOD medical and dental corps officials, other factors that may influence this decision include number and frequency of deployments, ability to function at full scope of practice for training, additional nonphysician duties and administrative requirements placed on active-duty physicians that their private sector counterparts do not experience, family considerations associated with permanent change of station orders, nonselection to residency of choice, nonselection for promotion, and retirement eligibility. Similarly, data from the 2017 DOD Status of Forces Survey show that among all officers, the most important factors that would be considered in a decision of whether to stay on active-duty were the military retirement system and personal choice/freedoms (e.g. control of where to work), as well as factors such as opportunities to be assigned to station of choice, family concerns, and pay and allowances. Moreover, a 2019 study of Army physician service obligations showed that military physicians who were most likely to continue serving after completion of their obligation and ultimately retire were those who had the most years of service accumulated when obligations were completed. That is, those who were close to retirement after completing their service obligations were more likely to stay to receive their retirement benefit. In addition to cash compensation, DOD offers substantial deferred benefits, such as retirement pensions and benefits, and noncash benefits, such as tuition-free medical school education and health care, to its military physicians and dentists. In its report on military compensation, DOD noted that nearly half of military compensation is made up of deferred and noncash benefits, and that this proportion is considerably higher than in civilian compensation. Additionally, in 2011 we identified military personnel costs as an area where DOD could recognize long- term cost avoidance by using a total compensation approach to manage military compensation in a holistic manner that considers deferred and noncash benefits alongside cash compensation. Studies of military compensation highlight that assigning a value to deferred and noncash benefits and comparing them to the civilian private sector proves more difficult than for cash compensation because servicemembers value or use these benefits differently, various assumptions have to be made to assign value, and access to such benefits is not universal among private sector civilian workers. Additionally, it is difficult to measure the extent to which servicemembers discount the value of future benefits. We previously reported that it is generally accepted that some deferred benefits, such as a pension in retirement, are not valued as highly by servicemembers as current cash compensation. However, a recent study found that servicemembers, particularly military officers, may value deferred benefits more highly than was previously reported. For these reasons we did not compare the value of military deferred and noncash benefits to similar benefits in the civilian private sector; however, we describe certain types of deferred and noncash benefits available to physicians and dentists and provide estimates of their value where possible. DOD provides access to two primary types of deferred benefits: its employer-sponsored retirement plans and retiree health and dental care. As mentioned previously, the likelihood of benefiting from DOD’s military retirement system is a factor that officers consider when deciding to stay on active duty. Retirement plans. In DOD’s traditional retirement system, known as the “High-Three System,” servicemembers are eligible to receive a defined benefit annuity based on their pay grade and years of service after a minimum 20 years of active-duty service, with no benefits provided to those who separate before then. This system was closed to new entrants at the end of 2017. Based on our estimates, under the High-Three System, the defined benefit for a physician or dentist who retires with 20 years of service in 2035 was estimated to be $2,457,253 (present value). New servicemembers from 2018 onwards were enrolled in the Blended Retirement System (BRS). BRS is a hybrid retirement system that includes a revised defined benefit plan requiring 20 years of active-duty service, a defined contribution plan with agency matching contributions, and a one-time direct cash payout—called continuation pay—distributed at the midcareer point (between 8-12 years of service). Based on our estimates, under the BRS, the defined benefit for a physician or dentist who retires with 20 years of service in 2035 was estimated to be $1,965,802 (present value). The defined contribution plan offers government automatic and matching contributions of up to 5 percent of basic pay to the servicemember’s Thrift Savings Plan, and vested servicemembers who separate before 20 years of active-duty service retain ownership of these contributions. The BRS was implemented in 2018 to modernize the military retirement system. As the Military Compensation and Retirement Modernization Commission reported in 2015, roughly 51 percent of military officers exited service before 20 years, meaning that most left without any retirement benefits under the High-Three System. The BRS is expected to provide retirement benefits for the majority of servicemembers, including those who serve fewer than 20 years, according to DOD. In our interviews, some DOD officials expressed concern about the effects of BRS on retention of military physicians and dentists, because, for example, they believed the opportunity to separate with defined contributions will reduce their incentive to remain for a longer period of active duty. Other DOD officials we interviewed stated that it is too soon to determine the effects of the BRS on retention, and noted that the inclusion of continuation pay as part of the BRS was designed to encourage servicemembers to continue serving at the mid-career point. Retiree health and dental care. Servicemembers retiring from active duty are eligible to enroll in TRICARE. Specifically, retired servicemembers and their eligible dependents are able to participate in TRICARE Prime which is comparable to a health maintenance organization (HMO) program, and TRICARE Select, which is comparable to a preferred provider organization (PPO) program. After they are eligible for Medicare, retired servicemembers and their eligible dependents with Medicare Part A and B can enroll in TRICARE for Life, which provides Medicare-wraparound coverage. Eligible retired servicemembers may also receive benefits from the Department of Veterans Affairs health care system. Specifically, active-duty servicemembers who served 24 continuous months or the full period for which they were called to active duty are eligible for Veteran Affairs’ health care. DOD provides access to a wide variety of noncash benefits, some of which are uncommon in the civilian sector, and may offset some of the discrepancies in military and private sector civilian cash compensation. However, limited information exists on the extent to which noncash benefits are used by military physician and dentists. Therefore, we have highlighted select benefits that may be used by military physicians and dentists. Tuition-free medical and dental school. Military physicians and dentists benefit from DOD’s scholarship program and the University, through which prospective medical and dental students receive tuition-free education in exchange for commitment to a number of years in active- duty service. This benefit allows physicians and dentists to avoid thousands of dollars of student debt. For example, according to the Association of American Medical Colleges, the average first-year medical student paid $36,755 for tuition, fees, and health insurance to attend a public medical school during the 2018-19 academic year, and the average first-year student attending a private medical school paid $59,076. Medical and dental care. DOD offers comprehensive health coverage to military personnel and their dependents through TRICARE, a managed care program. Care is provided in more than 650 military treatment facilities worldwide, supplemented by civilian providers. TRICARE offers two health care options to non-Medicare-eligible beneficiaries: TRICARE Prime and TRICARE Select. All active-duty servicemembers are automatically enrolled in TRICARE Prime, which is comparable to a private health maintenance organization plan. Under this program, active- duty servicemembers have no premium costs, deductibles, or out-of- pocket costs for servicemembers and no or low costs for dependents. Medical Expenditure Panel Survey data indicate that the average private sector civilian employee spent over $5,000 in health insurance employee contributions for family coverage in 2018. The TRICARE Active Duty Dental Program supplements the dental services available to active-duty servicemembers at military treatment facilities when necessary care is not available or the servicemember does not have ready access to a military treatment facility. Active-duty servicemembers do not pay premiums for this dental care, do not share in the costs of the care, and do not face any annual or lifetime maximums on the cost of care. Financial benefits during education and training. Medical and dental scholarship students receive O-1 pay and allowances for 45 days of active duty for annual training performed for each year the scholarship is awarded. Participants may also be eligible for a $20,000 signing bonus. During their education, medical and dental scholarship students receive a monthly stipend, and medical students at the University receive officer salary and benefits at grade O-1. After medical school, medical and dental residents receive officer pay and benefits at grade O-3 or higher, according to DOD officials. Based on our analysis of DOD’s incentives to recruit and retain military physicians and dentists, DOD generally (1) clearly defined the criteria used to determine when to offer incentives, (2) identified and incorporated opportunities for improvement, (3) identified and evaluated unique staffing situations, and (4) made investments to attract and retain top talent. However, we found that DOD did not consistently collect information on (1) replacement costs, (2) current and historical retention efforts, and (3) comparable civilian wages to help inform investment decisions in its package of incentives to recruit and retain military physicians and dentists. Fully applying these seven key principles of effective human capital management in its approach to recruit and retain military physicians and dentists is important to making fully informed investment decisions. We found that DOD generally applied effective human capital management principles related to clearly defined criteria on when to use incentives, making investments based on expected improvement in agency results, identifying and evaluating unique staffing situations, and identifying and incorporating opportunities for improvement. To support its operational needs, DOD uses educational, training, and monetary incentives to recruit and retain physicians and dentists. Specifically, DOD’s package of incentives includes, among other things, a tuition-free medical school education through the scholarship program and the University, pay as an O-3 officer or higher during medical or dental residency, the opportunity for further training via a fellowship, and a series of special and incentive pays for fully trained physicians and dentists. According to DOD’s report on military compensation, special and incentive pay authorities provide the services with greater flexibility to target additional compensation where needed to address emerging staffing shortfalls and maintain staffing in critical or hard-to-fill skills. We found that DOD generally applied four of the seven key principles, as described below: Relied on clearly defined, well-documented, consistently applied, and transparent criteria. DOD and the military departments have established rules-based pay plans with clear eligibility criteria for special and incentive pays and recruitment and retention bonuses. Key principles for human capital management state that agencies should consider making targeted investments in specific human capital approaches, and that these approaches should have clearly defined, well-documented, transparent, and consistently applied criteria for making these investments. Identified opportunities for improvement and incorporated these opportunities into the next planning cycle. The services and officials from the Office of the ASD(HA) participate in the Health Professions Incentives Working Group to review recruitment and retention special pay and incentives and recommend adjustment to amounts offered as necessary. For example, as a result of working group discussions, DOD officials stated that they established a new 6- year retention bonus in the fiscal year 2019 pay plan for select medical and dental specialties, in part to ensure greater stability in the numbers of physicians and dentists within these specialties. Military department officials stated they plan to identify potential impacts and determine adjustments, if any, that need to be made. DOD’s report on military compensation advises officials to identify opportunities for improvement using analytical tools to model how changes in compensation might alter the force or career profile. It further states that taking a structured approach to determining both incentive pay eligibility criteria and amounts helps force managers optimize their limited special and incentive pay budgets. Such an approach also provides a mechanism to periodically conduct a rigorous assessment of such pays to ensure that they keep pace with changing conditions. Identified and evaluated unique staffing issues. According to military department officials, medical corps and dental corps community managers, specialty leaders, consultants, and others actively discuss military physicians’ and dentists’ career plans to help inform future staffing needs. Moreover, to attract physicians and dentists in specialties which DOD has identified as a critically short wartime specialty, DOD offers a Critical Wartime Skills Accession Bonus. However, as we reported in 2018, military department officials cited a number of challenges that make it difficult to attract and retain military physicians and dentists, including national shortages and competition with the private sector. Incentive pay and retention bonus amounts are specific to each specialty. DOD’s report on military compensation states that evaluation of unique staffing issues identified by community managers should be a core part of a systematic approach to assessing the application of a special or incentive pay. Similarly, key principles for human capital management note that agencies should tailor human capital strategies to meet their specific mission needs. Targeted investments to attract and retain top talent. The services are authorized to offer targeted monetary incentives in the form of special and incentive pays and recruitment and retention bonuses to eligible physicians and dentists who are in good standing. Moreover, military department officials stated that DOD offers Board Certification Pay to physicians and dentists who achieve and maintain this accreditation because it reflects that the physician or dentist is maintaining skills and qualifications and allows the department to better reflect the high level of the quality of care that is provided by the military health system. Similarly, we reported in 2018 that DOD and the military departments had established a set of minimum qualifications for medical school applicants applying to the scholarship program and the University. Key principles for human capital management state that targeted investments in human capital approaches should help the agency attract, develop, retain, and deploy the best talent and then elicit the best performance for mission accomplishment. The principles further state that decisions regarding these investments should be based largely on the expected improvement in agency results. Similarly, DOD’s Diversity and Inclusion Strategic Plan 2012-2017 notes that retaining top talent is essential to sustaining mission readiness that is adaptable and responsive. In three key areas of effective human capital management related to data on replacement costs, recruitment and retention, and civilian wages, DOD does not consistently collect information to help inform investment decisions in its package of incentives to recruit and retain military physicians and dentists, as described below: Did not identify replacement costs. Military departments do not consistently collect information on replacement costs of military physicians and dentists. Specifically, no military department was able to provide us with a comprehensive assessment of the replacement cost for military physicians and dentists. Replacement cost assessments can be found in other occupations within DOD. For example, in 2017, we reported that the Navy considers the high replacement costs of its nuclear propulsion personnel—up to $986,000 per trainee—in justifying a strategy that prioritizes investment in retention initiatives over new accessions or recruits. Moreover, DOD requires that the training investment and replacement cost for those qualified in the skill be considered when justifying the need for the critical skills retention bonus. DOD’s report on military compensation identified replacement costs and training costs as a factor in assessing incentive pay appropriateness. In 2018, we recommended that the ASD(HA) require that the University develop a reliable method to accurately determine the cost to educate its medical students. DOD partially concurred with our recommendation. In response to our recommendation, the University contracted with the Institute for Defense Analyses to determine the costs to educate University medical students. In its October 2019 final draft report, the Institute for Defense Analyses estimated total accession costs for a fully trained physicians through both the scholarship program and the University; specifically, the report estimated the total cost for a fully trained physician who completes 4 years of medical school and a 3-year military residency to be $878,000 for scholarship medical students and approximately $1.5 million for University medical students. In another similar ongoing effort, Navy officials stated that they have commissioned a Life Cycle Cost study with the Center for Naval Analyses. We are encouraged by these initiatives, which will provide the Office of the ASD(HA) and the military departments a foundation for formalizing the process of collecting information on replacement costs. With the benefit of this information, DOD can make more informed decisions regarding its packages of recruitment and retention incentives. Did not collect current and historical retention information. Military departments do not consistently collect and use current and historical retention information to help inform decisions about investment in retention incentives. Specifically, Navy and Air Force officials told us that they do not have readily available information to determine the percentage of those who accepted a retention bonus among the eligible population, and Army officials noted they do not have a framework in place to use retention information to determine the effectiveness of retention bonuses. Using retention data to measure effectiveness of retention incentives is performed by other communities within DOD. For example, in 2018 we reported that officials from the Navy, Marine Corps, and Air Force measured the effectiveness of aviation retention bonuses by monitoring bonus acceptance rates. DOD’s report on military compensation stated that a review of current and historical data on retention should be a core part of a systematic approach to assessing the application of a special or incentive pay. Further, key principles for human capital management note that periodic measurement of an agency’s progress toward human capital goals and the extent that human capital activities contributed to achieving programmatic goals provides information for effective oversight by identifying performance shortfalls and appropriate corrective actions. Without information on the acceptance rate among those eligible, the military departments cannot assess the effectiveness of the performance of their investment in retention bonuses. Did not assess private sector civilian wages. DOD does not consistently collect and use private sector wage information to help inform investment decisions in its special and incentive pays for physicians and dentists. Based on our review of the minutes of meetings of the Health Professions Incentives Working Group, which recommends changes to the rate and term of special and incentive pays, private sector compensation was occasionally raised as a challenge. However, it was not collected and used to help inform investment decisions on a consistent basis. According to officials from the Office of the ASD(HA) and the military departments, an assessment of civilian wages is not a driving factor when considering adjustments to special and incentive pays, in part because DOD cannot always match civilian sector compensation for military physicians and dentists. Officials from the Office of the ASD(HA) and the military departments acknowledged the disparity between military and civilian cash compensation varies by specialty; however, incentive pay and retention bonus amounts have largely remained the same for over a decade. DOD’s Ninth Quadrennial Review of Military Compensation states that pay at around the 70th percentile of comparably educated civilians is necessary to enable the military to recruit and retain the quantity and quality of personnel it requires. Based on our comparison of military and civilian cash compensation pay previously discussed, we found that the gap between military and private sector civilian varies by specialty and many fall below the civilian private sector median. Moreover, based on our review of cash compensation for medical officers who completed their residency directly after medical school across 21 medical specialties, we found that at their first unobligated year of service, all 21 specialties had cash compensation below the private sector civilian median. Additionally, all but one specialty (psychiatry) were compensated at less than the 20th percentile of private sector civilian compensation. Use of assessments of private sector civilian compensation can be found in other communities within DOD. For example, in 2017, we reported that the Navy justified its use of selective reenlistment bonuses for cyber-related occupations by noting the specific level of starting salaries for comparable civilians. DOD’s report on military compensation states that reviewing civilian wages is a key element in assessing the application of a special or incentive pay. Further, it states that periodic reviews, which should include the use of an analytical tool or model, will ensure that resources are directed at the most pressing staffing needs. For example, professions that consistently command higher pay in the civilian sector—such as the medical professions—may merit predictable pays over the long term. Yet in other areas, evolving mission needs, changing conditions in the civilian market, and other factors may call for increasing an incentive or, in some cases, may show that additional pay can be reduced or eliminated. According to a former Under Secretary of Defense for Personnel and Readiness and a noted expert on defense personnel issues, DOD would benefit from analysis to determine the point at which cash compensation for military physicians, including special and incentive pays, reaches a minimum threshold of attractiveness compared to the private sector. Assessing civilian wages could help DOD understand the relationship of any military and civilian pay discrepancies to its ability to fill particular specialties. For example, we found that in fiscal year 2018, all but three of the specialties we reviewed were below 90 percent of authorization by at least one of the services’ active components. By consistently collecting civilian wage information and using it to inform its package of incentives, DOD will be better positioned to make the most effective use of its recruitment and retention incentives. DOD officials stated that their approach to managing the package of incentives to recruit and retain military physicians and dentists is driven by a number of considerations. Specifically, DOD officials stated that the rates of special and incentive pays represent amounts that are affordable and that the military departments generally believe have allowed them to meet their personnel needs. Further, military department officials stated that budget considerations and statutory limitations hinder their ability to change the rate of special and incentive pays. Current statutory limits to the amount of the retention bonus, incentive pay, and board certification pay are $75,000, $100,000, and $6,000, respectively; there is currently no statutory limit on the critical skills retention bonus for health professionals, which can be paid in addition to other pays. While we believe these are valid considerations, collecting information on replacement costs, retention, and civilian wages would allow the Office of the ASD(HA) and the military departments to provide greater stewardship of available funding by ensuring its efficient application. Specifically, Standards for Internal Control in the Federal Government state that management should use quality information to achieve the entity’s objectives. For example, further analysis of replacement costs could reveal that retention of fully trained physicians is highly economical for DOD, and provide strong support for changes to retention incentives to safeguard significant investment in physicians and dentists. By collecting and using this information to inform its decision-making, DOD and the military departments would be better positioned to assess the effectiveness of their incentives to recruit and retain military physicians and dentists and make sound investment decisions for the future. Our surveys of medical students, focus groups with medical residents, and interviews with DOD officials showed there was a general perception that lengthening active-duty service obligations, such as through a system of serving obligations from medical school and residency training consecutively, could negatively affect recruitment and retention of military physicians. Moreover, DOD is considering reductions to the overall number of active-duty physicians, including targeted reductions to certain specialties, and participants in all eight focus groups with residents had concerns about the proposed reductions to authorizations for certain medical specialties. In our surveys of medical students, we found that they generally would not have accepted the scholarship or attended the University if the service obligations from medical education and residency training were served consecutively. Specifically, an estimated 61 percent of scholarship recipients and an estimated 51 percent of University students in our representative survey responded that they would not have accepted DOD’s scholarship program or attended its University had they been required to fulfill these service obligations consecutively. However, our survey results indicated that students are willing to accept some additional active-duty service obligation for their current programs. Specifically, 68 percent of the University students and almost half (46 percent) of the scholarship students would be willing to accept an additional year of active-duty service obligation. Notably, a lower percentage of medical students would accept 2 additional years of active- duty service obligations—specifically 34 percent of University students and 16 percent of scholarship recipients. Our survey results found that medical students would be more willing to accept longer service obligations if accompanied by additional cash incentives. For example, 80 percent of University students and more than half of scholarship recipients (63 percent) would be willing to accept an additional year of service obligation if accompanied by additional cash incentives. (See figure 6 and appendix III for specific estimates and confidence intervals.) Similar to the survey responses, participants in all eight focus groups with medical residents also would not have accepted the scholarship or attended the University under a system of consecutive active-duty service obligations. However, participants in seven out of eight focus groups we conducted stated that they would be more willing to accept longer service obligations if accompanied by additional cash incentives, such as a larger accession bonus. Lengthening service obligations may also have unintended consequences without other changes to DOD policy. Specifically, participants in five out of eight of our focus groups with medical residents and DOD officials we interviewed expressed concern that lengthening service obligations would delay physicians’ eligibility for retention bonuses, resulting in a reduction of cash compensation over the course of a career. For example, under current policy, a physician who accepted a 4-year scholarship, completed a 1-year internship, and then trained in a 4-year residency training program would be eligible for a retention bonus after 9 years of service. Under a consecutive service obligation model, that same physician would be eligible for a retention bonus after 13 years of service (see figure 7). Further, as previously reported, cash compensation for military physicians is generally less than private sector civilian compensation, and participants in seven out of eight of our focus groups with residents expressed that lengthening service obligations would extend the amount of time they would not be paid comparably to their private sector civilian counterparts. Residents in our focus groups stated that lengthening active-duty service obligations would make residency training in a military hospital less attractive and would likely affect their decision to continue military service. Specifically, medical residents in most focus groups we held noted that lengthening service obligations would make them more likely to: fulfill their medical school active-duty service obligation by serving one or more tours as a General Medical Officer and then separate from the military in order to train in a civilian residency program; decline to participate in further medical training and specialization via a fellowship program within the military; and separate from the military sooner than planned, in part because a longer active-duty service obligation would delay their eligibility for certain special and incentive pays. Military department officials we interviewed expressed concern that lengthening active-duty service obligations, such as through a system of serving obligations consecutively, could encourage potential medical residents to choose shorter residency training programs over longer ones. However, participants in all eight focus groups we held with medical residents stated that the ability to train in a chosen medical specialty is more important than the length of the residency program, and a longer active-duty service obligation would not influence their chosen medical specialty. Further, residents who participated in our focus groups stated that the proposed reductions in authorizations—that is, funded positions—for certain medical specialties and associated reductions in residency program spots could negatively affect the attractiveness of residency training in a military setting. DOD has reduced authorizations for certain specialties based on our analysis of DOD’s Health Manpower Personnel Data System information and is considering additional reductions to the overall number of active-duty physicians as part of its budgeting process for fiscal years 2020-2024, including targeted reductions to certain specialties. For example, DOD reduced authorizations for the general pediatrics specialty by 40 percent from fiscal year 2015 through fiscal year 2018, and based on our surveys of medical students, 12 percent of scholarship recipients and 16 percent of University students in the clinical stage of medical school responded that they are interested in practicing the pediatrics specialty after they have completed all required training. Participants in all eight of our focus groups with residents commented that the ability to specialize in their medical specialty of choice was important when deciding to accept the scholarship or attend the University, and narrowing such opportunities would negatively affect the attractiveness of either program for future prospective participants. When reflecting on the proposal to reduce the range of available specialties, residents questioned their ongoing ability to practice their preferred specialty as an active-duty servicemember. In our focus groups, some residents expressed that this issue could play a role in their future decision to continue military service or separate and pursue civilian medical practice. DOD’s ability to recruit and retain the right numbers and types of physicians and dentists depends in part on the effectiveness of the package of incentives in which the department invests. To initially recruit these physicians and dentists, DOD relies on its scholarship program and University, which come with active-duty service obligations. Changes to the structure of its active-duty service obligations could affect recruitment and retention of physicians and dentists. Given that DOD spends millions of dollars annually to train medical and dental students to become fully trained physicians and dentists and that almost half of DOD’s special pay budget is dedicated to retaining them, consistently collecting information to help inform investment decisions is critical to ensuring the efficiency of these significant resources. For example, information on the replacement costs of physicians and dentists would help DOD make decisions about whether it is more cost effective to train or retain these personnel. Further, consistent collection of information on the extent to which eligible physicians and dentists accept retention bonuses will help DOD monitor the effectiveness of an incentive that represents a significant investment by DOD. Our comparison of military to private sector cash compensation highlighted that military physicians and dentists generally receive less cash compensation than their private sector civilian counterparts for most specialties we reviewed. This differential, according to DOD officials, is one factor that servicemembers consider in deciding whether to continue service in the military. However, while DOD and military department officials stated that they are aware of how prevailing private sector civilian wages for medical and dental specialties compare to military cash compensation, they do not consistently collect information on this matter and that its role in setting military cash compensation is limited. By collecting and using such information to inform investment decisions, DOD will have better information to efficiently and effectively meet its mission of providing health care during times of war and peace. We are making the following three recommendations to DOD: The Secretary of Defense should ensure that the Assistant Secretary of Defense for Health Affairs, in coordination with the military departments, collect consistent information on the replacement costs of military physicians and dentists and use this information to inform investment decisions in the package of incentives to recruit and retain military physicians and dentists. (Recommendation 1) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Health Affairs, in coordination with the military departments, collect consistent information on current and historical retention data, to include data on the percentage of eligible physicians and dentists who accept retention bonuses, and use this information to inform investment decisions in the package of incentives to retain military physicians and dentists. (Recommendation 2) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Health Affairs, in coordination with the military departments, collect consistent information on private sector civilian wages and use this information to help inform investment decisions in the package of incentives to recruit and retain military physicians and dentists. (Recommendation 3) We provided a draft of this report to DOD for review and comment. DOD concurred with all three recommendations and noted that it will take actions to incorporate them into policy within the next two years. DOD’s comments are reprinted in appendix IV. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Office of the Assistant Secretary of Health Affairs, the Secretaries of the Army, the Navy, the Air Force, and the President of the Uniformed Services University of the Health Sciences. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3604 or FarrellB@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report addresses the following objectives: 1. how compensation for military physicians and dentists compares to private sector civilians with comparable skills in 2017; 2. the extent to which the Department of Defense (DOD) has developed an approach to recruit and retain military physicians and dentists through a package of incentives that reflect key principles of effective human capital management; and 3. the perceptions of military medical students, residents, and DOD officials regarding active-duty service obligations, including their effect on recruitment and retention. For our first objective, we compared cash compensation for military physicians and dentists to comparable private sector civilian specialties, described the deferred and noncash benefits available to military physicians and dentists, and created estimates of the value of DOD’s retirement benefit for officers with varying current years of service. To compare cash compensation for military physicians and dentists to comparable private sector civilian specialties, we estimated military cash compensation and compared that to civilian compensation data reported in surveys by the American Medical Group Association and American Dental Association. Specialty selection. To select DOD physician and dental specialties that have private sector civilian equivalents, we began with the list of 44 physician and 11 dental specialties in DOD’s Fiscal Year 2018 Health Manpower Personnel Data System report. We selected 21 physician specialties in consideration of the following factors: a comparable private sector civilian specialty existed; the majority of the physician workforce was represented; deploying specialties were included; a balance of procedural, surgical, and other specialties was included, and; specialties identified as critically-short, trauma-related wartime specialties were included. We selected six dental specialties in consideration of the following factors: a comparable private sector civilian specialty existed, and private sector civilian compensation information was available. Estimates of military cash compensation. To estimate cash compensation for military physicians and dentists for our selected specialties, we reviewed DOD policy and guidance and relevant statutes to identify any current measures of cash compensation and other key elements of cash compensation for physicians and dentists. DOD’s measure of cash compensation, known as regular military compensation, includes the sum of basic pay, average basic allowance for housing, basic allowance for subsistence, and the federal income tax advantage that accrues from the nontaxable nature of the allowances. Another key element of cash compensation is the special and incentive pays that DOD offers to eligible military physicians and dentists, such as incentive pay, Board Certification Pay, and retention bonuses. We collected information on basic pay, basic allowance for housing, and basic allowance for subsistence for married personnel from DOD’s fiscal year 2017 Greenbook publication, and information on incentive pays, Board Certification Pay, and retention bonuses from DOD’s fiscal year 2017 Health Professions Officer Special and Incentive Pay Plan. We selected fiscal year 2017 because it was the most recent year of available data amongst all of our sources, and we selected married personnel because according to a DOD report, the majority of officers in the pay grades O-4 to O-6 are married, which largely aligns with DOD's population of physicians and dentists. We estimated a range—the minimum and maximum—of military cash compensation by specialty for pay grades O-3 to O-6. The minimum and maximum are based on two scenarios that represent a range of pay that specialized physicians and dentists can expect to receive, considering only the special and incentive pays listed in the Health Professions Officer Special and Incentive Pay Plan. The minimum includes regular military compensation, Board Certification Pay, and incentive pay. The maximum includes the regular military compensation, Board Certification Pay, and incentive pay at a higher amount in conjunction with a 4-year retention bonus. Our estimates represent the sum of basic pay, average basic allowance for housing, basic allowance for subsistence, special and incentive pays, and the federal tax advantage that accrues from the nontaxable nature of the allowances. To calculate the federal tax advantage, we used the 2018 federal tax tables and applied the 2018 federal tax standard deduction and then converted the calculated federal tax advantage to 2017 dollars. According to a senior DOD dental corps official, most general dentists are not board-certified and do not receive Board Certification Pay; we therefore omitted Board Certification Pay in our estimates for the minimum and maximum military cash compensation of general dentists. Private sector civilian cash compensation information. To identify private sector civilian cash compensation for physicians in comparable specialties, we chose the American Medical Group Association’s 2018 Medical Group Compensation and Productivity Survey—2018 Report Based on 2017 Data because (1) it included all the specialties we selected to review, and (2) it contained information on physicians who practiced in settings that were similar to those in which federal physicians practiced.The survey data provided compensation amounts for each specialty by 20th percentile, median, and 80th percentile. The data excluded the value of any employer-provided malpractice insurance, but some physicians may incur costs for this coverage. Military physicians generally do not need to purchase malpractice insurance. To identify private sector civilian cash compensation for dentists in comparable specialties, we chose the American Dental Association’s Health Policy Institute, Income, Gross Billings, and Expenses: Selected 2017 Results from the Survey of Dental Practice because (1) it included all the specialties we selected to review, and (2) included the net income of dentists and specialists in private practice, which is comparable to military dentists who generally do not need to purchase malpractice insurance. We obtained net income information for full-time practitioners—those who reported working 35 hours a week or more— from the American Dental Association. The survey data provided compensation amounts for each specialty by 25th percentile, median, and 75th percentile. Both surveys represent salaries for 2017. To help determine the reliability and accuracy of private sector civilian compensation information, we checked these data for reasonableness and the presence of any obvious or potential errors in accuracy and completeness. We believe the data are sufficiently reliable for the purpose of this report. Comparisons of military and private sector civilian cash compensation. We compared our estimates of the ranges of military cash compensation by specialty and pay grade to the ranges of private sector civilian cash compensation by specialty from our selected surveys. As we could not make direct comparisons of military and civilian cash compensation by years of service or experience due to data limitations, we compared and presented the ranges of compensation as appropriate. We also compared military cash compensation at the first unobligated year of service to the range of private sector civilian cash compensation, by specialty. We estimated military cash compensation at the first unobligated year of service based on the length of each residency and, if applicable, fellowship—among other assumptions. We identified physician residency and fellowship length information by using the Accreditation Council for Graduate Medical Education’s Data Resource Book for Academic Year 2017-2018, and we requested information on military residency lengths from military department officials to confirm that residency lengths generally aligned with this information. We identified dentist residency and fellowship length information by requesting it from military department officials. For each specialty, we estimated the officers’ pay grade using the following assumptions: (1) no creditable service before medical or dental school; (2) a 4-year medical or dental school duration; (3) participants were commissioned at the O-3 pay grade after medical or dental school completion with 4 years of constructive credit—in accordance with entry grade credit guidance outlined in DOD Instruction 6000.13; (4) the first year of post-graduate medical or dental education does not accrue an active-duty service obligation, and; (5) were promoted to O-4 at 6 years of service, and to O-5 at 12 years of service—in accordance with DOD’s promotion schedule outlined in DOD Instruction 1320.13. The entry grade credit and promotion schedule practices were confirmed by DOD officials. For physicians, we assumed that the active-duty service obligations for medical school and residency were served concurrently, in other words we assumed immediate entry into a residency program. We performed our calculations twice, first assuming no tour as a General Medical Officer and second assuming that physicians completed a 3-year tour as a General Medical Officer—adding 3 years to their years of service at service obligation fulfillment. According to Navy medical corps officials, 55 percent of Navy physicians perform such a tour. When assuming no General Medical Officer tour, the majority of physicians reached this decision point at the O-4 pay grade with the exception of neurosurgeons and cardiac/thoracic surgeons, who were at the O-5 pay grade due to longer residency and fellowship lengths. When assuming a 3-year General Medical Officer tour, physicians in 12 specialties reached this point at the O-5 pay grade, with the remaining nine specialties at the O-4 pay grade. We also conducted this analysis for Uniformed Services University of the Health Sciences (University) students who accrued a 7- year active-duty service obligation. We found that assuming a 7-year obligation for University students produced the same results as assuming a 3-year tour as a General Medical Officer for Health Professions Scholarship Program (scholarship) participants. For dentists, we assumed that the dental school and residency obligations were not served concurrently because, according to the military department Dental Corps Chiefs, dental student graduates typically complete a 1-year advanced education in general dentistry certificate, which does not incur a service obligation, then fulfill their dental school active-duty service obligation as general dentists before taking a general dentist’s retention bonus and beginning residency training. We completed an analysis to understand how the pay grade at the first year of unobligated service may vary for general dentists who worked as a general dentist immediately after completing dental school or completed a 1-year advanced education residency. We found that general dentists generally reached this decision point at the O-3 pay grade; endodontists, orthodontists, pedodontists, and periodontists reached it at the O-4 pay grade, and; oral and maxillofacial surgeons reached it at the O-5 pay grade. Estimates of retirement benefit. To develop estimates of the value of the defined benefit portion for DOD’s two retirement benefit programs— the Blended Retirement System (BRS) and the High-Three—we developed two scenarios for a hypothetical officer who either chose to remain in the High-Three System or to opt into the BRS. We used DOD’s publically-available, online retirement calculators to generate an estimate for each scenario, which Office of the Under Secretary for Personnel and Readiness officials described as the best available tools to determine the value of military retirement benefits. Specifically, the estimates were for a physician or dentist who was commissioned as an O-3 officer in 2015 and assumed separation from service at 20 years. For these scenarios, we developed reasonable estimates to enter into the calculators. For example, in the personal information section of the calculators we estimated the pay entry base date assuming that the officer began earning creditable years of service toward retirement after medical or dental school and that they began active-duty service as an officer at the O-3 pay grade in the month of June after the completion of medical or dental school. The calculators produced an estimate of the present value estimated retirement benefit at 20 years of service, which is when the defined benefit portion becomes effective. Estimates were as of August 2019 and included a specific value for the defined benefit. DOD’s publically available retirement calculators use a discount rate of 5 percent per year, as of July 2018. We also consulted with a senior DOD official from the Office of the Under Secretary for Personnel and Readiness to corroborate the reasonableness of our approach. To help determine the reliability and accuracy of DOD’s retirement calculators, we checked the data for reasonableness and the presence of any obvious or potential errors in accuracy and completeness and interviewed DOD officials knowledgeable about the data. We believe the data are sufficiently reliable for the purpose of this report. Description of deferred and noncash benefits. To describe deferred and noncash benefits available to military physicians and dentists, we reviewed our prior reports, other relevant research, and publically available reports and information from DOD. We interviewed cognizant DOD officials to understand which benefits military physicians and dentists were most likely to utilize. For our second objective, we reviewed pay plans, policies, and other documents developed by the Office of the Assistant Secretary of Defense for Health Affairs (OASD(HA)) and the respective military departments concerning DOD’s approach to recruitment and retention of military physicians and dentists. We also interviewed officials from OASD(HA) and the military departments concerning their decision-making processes in managing this package of incentives. We compared this information with seven key principles of effective human capital management, which was reported in our February 2017 report on military compensation. As we reported in that report, to identify key principles of effective human capital management, we reviewed a compilation of our body of work on human capital management, DOD’s Report of the Eleventh Quadrennial Review of Military Compensation, and the DOD Diversity and Inclusion Strategic Plan 2012 - 2017. The seven key principles of effective human capital management include (1) criteria for making human capital investments are clearly defined, well-documented, consistently applied, and transparent; (2) replacement costs of personnel are considered when deciding to invest in recruitment and retention programs; (3) decisions regarding human capital investments are based largely on expected improvement in agency results and implemented in a manner that fosters top talent; (4) unique staffing issues are identified and evaluated as part of establishing the incentive structure; (5) opportunities for improvement are identified and incorporated into the next planning cycle; (6) current and historical retention data are collected and reviewed as part of efforts to evaluate effects and performance of human capital investments; and (7) civilian wages are assessed and plans are updated as needed. In addition to using the key principles, we also compared aspects of DOD’s approach to recruitment and retention of military physicians and dentists with federal internal control standards, which state management should use quality information to achieve an entity’s objectives, and highlighted areas where DOD’s approach differed from these principles. For our third objective, to obtain perceptions of (1) military medical students, (2) residents, and (3) DOD officials regarding active-duty service obligations, including their effect on recruitment and retention, we utilized, respectively, (1) web-based surveys of military medical students, (2) focus groups with military medical residents, and (3) interviews with knowledgeable officials. Surveys. For our third objective, to obtain perceptions of military medical students regarding active-duty service obligations, including their effect on recruitment and retention, we conducted two web-based surveys with a generalizable sample of current scholarship and University medical students to obtain information on the students’ knowledge of the current program and willingness to accept different lengths of service obligations or a change to a consecutive service obligation model (see table 1). One survey was administered to current scholarship medical students, while the other was administered to current University medical students. The questions in both surveys were largely the same. The main differences reflected the different pay and benefits from accepting a scholarship or attending the University and the differences in length of active-duty service obligation. For example, scholarship students receive a monthly stipend and, sometimes, an accession bonus, while University students receive the pay and allowances for commissioned officers in the O-1 pay grade. Scholarship participants incur 6 months of an active-duty service obligation for each 6 months of scholarship benefits they receive, with a 2-year, minimum service obligation, while University medical students accrue a 7-year active-duty service obligation. A full listing of survey questions is provided in appendix III. We worked with our social science survey specialists to develop our survey questionnaires, applying generally accepted survey design standards. We conducted pretests of the survey with scholarship and University students who varied by number of years in medical school and military service. Pretesting is necessary to ensure common understanding of terms used and to minimize errors that might occur from respondents interpreting the questions differently than we intended. During each pretest, the subject was not provided the draft survey in advance, but instead was either provided the draft survey at the meeting, or the survey was emailed to the subject at the beginning of the teleconference. After the pretester completed the survey, we discussed all survey questions and response options with the pretester to ensure clarity. We revised the survey instruments based on the feedback we received during each of the pretests until clarity issues were reasonably addressed. We determined fourth-year medical students were less likely to participate in the survey for three reasons: (1) they were close to graduating from medical school at the time the survey instrument was launched; (2) they lose their school email addresses shortly after graduation; and (3) once they are out of medical school, they are further removed from the decision point about either accepting the scholarship or attending the University. Therefore, we excluded them from the sample population. Dental students were also excluded from the sample population because they generally practice as general dentists after graduating from dental school and before training in a residency program, which differs significantly from the career paths of scholarship and University medical students. We defined our target population to be all medical students in their first, second, or third school year under the scholarship program or enrolled at the University. By stratifying, as shown in table 1, the sample allowed us to estimate any population figure across the service with a predetermined statistical precision. We determined the target sample size needed to achieve precision levels of plus or minus 10 percentage points or fewer, at the 95 percent confidence level. We then increased the sample size within each stratum for an expected response rate of 25 percent. The resulting sample frame included 2,972 students, and we selected a stratified random sample of 1,355. We stratified the sampling frame into four mutually exclusive strata based on medical program and service. One survey was administered to current scholarship medical students from June 26, 2019 through August 26, 2019; the survey of current University medical students was administered from June 25, 2019 through August 6, 2019. We created two administrative email accounts, one for scholarship medical students and one for the University medical students, through which we sent an announcement email to the medical students in our sample population. We administered the survey through a web-based application and sent an email from the administrative email accounts stating that the survey was ready to complete. When we received bounce-back messages, we used secondary email addresses if available or called students to request updated contact information. To maximize our response rate, we sent two reminder emails and contacted nonrespondents by telephone to encourage them to complete the survey. Also, we took steps in the development of the survey, data collection, and data analysis to minimize nonsampling errors and help ensure the accuracy of the answers that were obtained. For example, a social- science survey specialist helped to design the questionnaire, in collaboration with analysts having subject-matter expertise. Then, as noted earlier, the draft questionnaire was pretested to ensure that questions were relevant, clearly stated, and easy to comprehend. Our unweighted survey response rate was 60.5 percent for scholarship students and 80 percent for University students, with 624 and 259 respondents, respectively. Per Office of Management and Budget (OMB) Standards and Guidelines for Statistical Surveys, a nonresponse bias analysis should be conducted for a survey with a response rate less than 80 percent (Guideline 3.2.9). The response rate for the survey of University students met this threshold, and we did not assess the potential for nonresponse bias. With respect to scholarship students, after conducting an analysis of propensity of responding to the survey to identify potential sources of nonresponse bias, we identified differential student response patterns by military department and marital status. We developed sampling weights based on the population size, divided by the number of sample students within each stratum. Weights were adjusted for overall nonresponse in University students and nonresponse by military department and marital status among scholarship students so that statistical estimates for survey response percentages are generalizable to the population of students. We expressed the precision of our particular sample’s survey responses as a 95 percent confidence interval. This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. As a result, we were 95 percent confident that each of the confidence intervals in this report included the true percentages of survey responses in the study population. All survey response percentage estimates presented in this report from this survey had a margin of error of plus or minus 6 percentage points or fewer, unless otherwise noted. Focus groups. We also conducted eight focus group meetings with a nongeneralizable sample of 79 military medical residents at three military treatment facilities to obtain the perspectives of military medical residents on issues related to: (1) the nature of active-duty service obligations, including their willingness to accept different lengths of active-duty service obligations; (2) the relative importance of the service obligations in relation to other factors at different decision points, including accepting the scholarship or attending the University; (3) participating in a military residency program, and; (4) choosing a medical specialty to pursue. These meetings involved structured small-group discussions designed to gain more in-depth information about specific issues that cannot easily be obtained from single or serial interviews. Consistent with typical focus group methodologies, our design included multiple groups with varying characteristics but some similarity in experience and responsibility. To identify focus group participants, we considered gender, number of residents who had accepted the scholarship or attended the University, medical specialties, military department affiliation, number of years in a military residency training program, and prior service as a General Medical Officer. The focus groups involved a range of seven to 15 participants during each meeting. We did not select participants using a statistically-representative sampling method, so the information collected from the focus groups is not generalizable and, therefore, cannot be projected across DOD, a military department, or any single military treatment facility we visited. The eight focus group sessions included two pilot focus groups at Walter Reed National Military Medical Center and two sessions for each of the three military departments (Army, Navy, and Air Force). To identify the focus group locations, we selected military treatment facilities that included a diverse mix of medical specialties and a large pool of residents from which to select participants in order to ensure sufficient participation in the focus groups. We traveled to military treatment facilities in Bethesda, Portsmouth, and San Antonio to conduct the focus groups. Table 2 illustrates the total number of focus group participants categorized by military treatment facility, military department, and whether they accepted the scholarship or attended the University. To conduct the focus groups, one of our trained facilitators moderated each of the sessions, following a protocol that included discussion guidelines and a set of eight questions (see table 3). The focus group protocol was validated by one of our methodologists with a social science background and knowledge of small group methods. The same focus group protocol was used at all military treatment facilities the engagement team visited, with some minor modifications made after the pilot sessions at Walter Reed National Military Medical Center. We assured participants that their names would not be directly linked to their responses, and that the results would generally be reported in the aggregate. Because of the limitations on the use of data derived from the focus group meetings, including the nongeneralizable sample and results reported in the aggregate, we did not rely entirely on focus groups, but rather used several different methodologies to corroborate and support our conclusions, including web-based surveys with medical students who either accepted the scholarship or attended the University, and interviews with DOD officials. We performed a content analysis on the responses to identify common themes from across the responses to determine their frequencies. For the qualitative analysis, we developed a standard coding scheme to identify common themes and determine their frequencies. We also identified other themes that we determined to be important based on our surveys with scholarship and University medical students and interviews with DOD officials. To obtain information concerning military dental residents’ views, perceptions, and feelings on issues related to the nature of active-duty service obligations, including their willingness to accept different lengths of service obligations and a change from a concurrent to a consecutive model of service obligation fulfillment, we conducted two focus group sessions with 20 Air Force dental residents who were in training at the Air Force Postgraduate Dental School, Joint Base San Antonio. The focus group participants had previously accepted the scholarship and varied by gender, rank, prior military service, dental specialty, and number of years in dental residency training. These discussions were conducted using a method and protocol that was similar to the approach for the medical students. After analyzing the results of these two focus groups with military dental residents and taking into consideration the interviews we conducted with DOD officials, we determined it was not necessary to conduct further focus groups with military dental residents or include dental students in our survey of current scholarship students. Dental students’ career paths differ in significant ways from medical students’ career paths. According to DOD officials and residents in the dental focus groups, military dentists are generally already serving consecutive service obligations by fulfilling their active-duty service obligation from dental school while serving as general dentists before training in a military residency program. As a result, a change from a concurrent to a consecutive service obligation model may not affect military dentists in a similar way that it would military physicians. Interviews. In addition, we conducted interviews with relevant DOD officials to understand their position on the effect of the length of active- duty service obligations on recruitment and retention of military physicians and dentists. Specifically, we interviewed officials from the Office of the Assistant Secretary of Defense for Health Affairs; the Office of the Under Secretary of Defense for Personnel and Readiness; the Defense Health Agency, and; various areas within the military departments with responsibilities related to medical or dental corps recruitment, retention, and education, such as the Offices of the Surgeons General, Manpower and Reserve affairs, and medical and dental corps or commands. We conducted this performance audit from September 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Cash Compensation of Specialized Military Physicians and Dentists Compared to Private Sector Civilians, 2017 percentile) percentile) Below 20th percentile Below 20th percentile median percentile) percentile) Below 20th percentile Below 20th percentile median percentile) percentile) Below 20th percentile Below 20th percentile median percentile) percentile) Radiology- Diagnostic (Interventional) Radiology- Diagnostic (Non-Interventional) The American Medical Group Compensation and Productivity Survey information represents the total annual compensation of the physician, including base and variable compensation plus all voluntary salary reductions. Examples of total compensation would include, but are not limited to, the following: compensation paid as salary or production-based compensation plans, any type of additional bonuses or incentives, clinically-related medical directorships, call coverage, and ancillary or advanced practice clinical supervision stipends. Compensation excludes any fringe benefits and employer payments to any type of retirement, pension, Supplemental Executive Retirement Plan, or tax-deferred profit-sharing plan. compensation (25th percentile, median, 75th percentile) percentile) Below 25th percentile Below 25th percentile below median compensation (25th percentile, median, 75th percentile) percentile) The American Dental Association, Health Policy Institute, Survey of Dental Practice information represents the reported annual net income of specialists in private practice, 2017. We obtained net income information for full-time practitioners—who reported working 35 hours a week or more—from the American Dental Association (ADA). Payments toward a retirement plan are included in net income. We conducted two web-based surveys with a generalizable sample of current Health Professions Scholarship Program (scholarship) and Uniformed Services University of the Health Sciences (University) medical students to obtain information about the students’ knowledge of the current program and willingness to accept different lengths of service obligations or a change to a consecutive service obligation model. One survey was administered to current scholarship medical students from June 26, 2019 through August 26, 2019; the survey of current University medical students was administered from June 25, 2019 through August 6, 2019. The questions in both surveys were largely the same. The main differences reflected the different pay and benefits from accepting a scholarship or attending the University and the differences in length of active-duty service obligation. The survey provided to scholarship students also included questions about whether students considered attending the University, while the survey provided to University students did not include a question about whether they considered accepting the scholarship. As a result, the scholarship survey had additional questions than the University survey. Responses to selected questions we asked in the surveys that were directly applicable to the research objectives in this report are shown below. The surveys consisted of closed- and open- ended questions, including demographic questions that were used in our analyses of the students’ responses. In this appendix, we did not provide information on responses provided to the open-ended or the demographic questions. See appendix I for a full description of the survey and estimation methodologies. The U.S. Government Accountability Office (GAO), an agency of the United States Congress, is studying the active-duty service obligation associated with the Armed Forces Health Professions Scholarship Program (HPSP) and the Uniformed Services University of the Health Sciences (USUHS). As a part of this study, GAO is conducting a nationwide survey of medical students who are participating in the HPSP or attending USUHS. We appreciate your insights, as it is important for GAO to provide student views of the current program to the Congress. Question 1 - How much did the following factors contribute to your decision to accept the HPSP scholarship? 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval - upper bound (percentage) NA e. Desire to provide care to military personnel, dependents, and retirees f. Desire to provide medical care while deployed g. Other (please specify below) Generally, participants in the HPSP incur a 1-year active-duty service obligation for each year of HPSP scholarship accepted. Similarly, a military residency may also result in an active- duty service obligation of 1 year for each year of residency. Currently, these two sets of obligationsare served at the same time, so a servicemember will effectively serve the longer of the two obligations. Residencies vary in length, and result in different service obligations. One example would be that a service member accepts 4 years of HPSP funding, requiring a 4-year active-duty service obligation, AND completes a 4-year military residency, which requires a 3-year active-duty service obligation. A 4-year military residency only requires a 3-year active-duty service obligation because the intern year or first year of residency does not result in a service obligation. Under the current system, this servicemember would serve both obligations (4 years and 3 years) at the same time. Completion of the first 3 years would satisfy the residency obligation and 3 of the 4 years of HPSP obligation; the final 1 year would satisfy the remaining HPSP obligation. Question 3 - When you decided to accept an HPSP scholarship, how familiar were you, if at all, with the active-duty service obligation requirements for HPSP and for completing a military residency? 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval - upper bound (percentage) a. An additional 1-year service obligation for 4 years of HPSP (1.25 years of commitment for each year of funding)? (No change in the service obligation for the medical residency.) (CHECK ONLY ONE ANSWER) No Don’t Know b. An additional 2-year service obligation for 4 years of HPSP (1.5 years of commitment for each year of funding)? (No change in the service obligation for the medical residency.) (CHECK ONLY ONE ANSWER) Additional service obligations and incentives c. An additional 1-year service obligation for 4 years of HPSP AND additional cash incentives? (No change in the service obligation for the medical residency.) (CHECK ONLY ONE ANSWER) 28.3 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval - upper bound (percentage) d. An additional 2-year service obligation for 4 years of HPSP AND additional cash incentives? (No change in the service obligation for the medical residency.) (CHECK ONLY ONE ANSWER) e. Service obligations served one after the other? For example, a service obligation for 4 years of medical school with the HPSP scholarship and a 4 year military residency have two service obligations – 4 years for HPSP and 3 years for the residency. Service obligations served one after the other in this example would result in a term of 7 years. (CHECK ONLY ONE ANSWER) f. A 4-year active-duty commitment AND a 2-year selected reserve commitment? Currently, HPSP participants may be subject to an individual ready reserve commitment after the completion of their active-duty service obligation. With a selected reserve commitment, reservists typically drill about 1 weekend a month and 2 weeks a year, and may be activated in support of military operations. (CHECK ONLY ONE ANSWER) If ‘No’ or ‘Don’t Know’ to Questions 5a, 5b, 5c, 5d, 5e, or 5f Which of the following funding options, if any, would you have pursued instead of accepting the HPSP scholarship? (CHECK ALL THAT APPLY) Personal or family resources Yes National Health Service Corps Scholarship Program None - would not have attended medical school Other (please specify) The U.S. Government Accountability Office (GAO), an agency of the United States Congress, is studying the active-duty service obligation associated with the Armed Forces Health Professions Scholarship Program (HPSP) and the Uniformed Services University of the Health Sciences (USUHS). As a part of this study, GAO is conducting a nationwide survey of medical students who are participating in the HPSP or attending USUHS. We appreciate your insights, as it is important for GAO to provide student views of the current program to the Congress. Question 1 - How much did the following factors contribute to your decision to attend USUHS? 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval – upper bound (percentage) a. Desire to avoid or reduce medical school debt b. Officer pay while in school c. Desire to serve your country in the armed forces 34.1 95 Confidence Interval – lower bound (percentage) Very Great Contribution Substantial Contribution Some Contribution Little or No Contribution h. Other (please specify below) The active-duty service obligation for completing the 4-year program at USUHS is 7 years. A military residency also results in an active-duty service obligation of 1 year for each year of residency, with the exception of the first year or intern year, which does not result in an active duty service obligation. Currently, these obligations are served at the same time, so a servicemember will serve the longer of the two obligations. Residencies vary in length and result in different service obligations. An example would be that a servicemember completes medical school at USUHS, which requires a 7-year active-duty service obligation, AND completes a 4-year military residency, which requires a 3-year active- duty service obligation. A 4-year military residency only requires a 3-year active-duty service obligation because the intern year or first year of residency does not result in a service obligation. Under the current system, this servicemember would serve both obligations (7 years and 3 years) at the same time. Completion of the first 3 years would satisfy the residency obligation and 3 of the 7 years of USUHS obligation; the next 4 years would satisfy the remaining USUHS obligation. Question 2 - When you decided to attend USUHS, how familiar were you, if at all, with the active- duty service obligation requirements for attending USUHS and for completing a military residency? 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval – upper bound (percentage) Question 3 - When you decided to attend USUHS, how familiar were you with the fact that the medical school and military residency service obligations are served at the same time? (CHECK ONLY ONE ANSWER) 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval – upper bound (percentage) a. An additional 1-year service obligation for attending USUHS? (No change in the service obligation for the medical residency.) 71.8 23.3 b. An additional 2-year service obligation for attending USUHS? (No change in the service obligation for the medical residency.) No Don’t Know Additional service obligations and incentives c. An additional 1-year service obligation for attending USUHS AND additional cash incentives? (No change in the service obligation for the medical residency.) NA 19.4 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval – upper bound (percentage) d. An additional 2-year service obligation for attending USUHS AND additional cash incentives? (No change in the service obligation for the medical residency.) No Don’t Know e. Service obligations served one after the other? For example, a service obligation of 7 years for attending USUHS and a 4- year military residency has two service obligations – 7 years for USUHS and 3 years for the residency. Service obligations served one after the other in this example would result in a term of 10 years. f. A 7-year active-duty commitment FOLLOWED BY a 2-year selected reserve commitment? Currently, USUHS graduates may be subject to an individual ready reserve commitment after the completion of their active-duty service obligation. With a selected reserve commitment, reservists typically drill about 1 weekend a month and 2 weeks a year, and may be activated in support of military operations. (No change in the service obligation for the medical residency) If ‘No’ or ‘Don’t Know’ to Questions 4a, 4b, 4c, 4d, 4e, or 4f Which of the following funding options, if any, would you have pursued instead of attending USUHS? (CHECK ALL THAT APPLY) National Health Service Corps Scholarship Program 19.0 95 Confidence Interval – lower bound (percentage) 95 Confidence Interval – upper bound (percentage) In addition to the contact named above, Lori Atkinson (Assistant Director), Adam Howell-Smith (Analyst in Charge), Taylor Bright, Timothy Carr, Breanne Cave, Alexandra Gonzalez, Caitlin Jackson, Ronald La Due Lake, Won (Danny) Lee, Kirsten Leikem, Amie Lesser, Amanda Miller, Dae B. Park, Stephanie Santoso, and Lillian Yob made key contributions to this report. Defense Health Care: DOD's Proposed Plan for Oversight of Graduate Medical Education Programs. GAO-19-338. Washington, D.C.: March 28, 2019. Defense Health Care: Actions Needed to Determine the Required Size and Readiness of Operational Medical and Dental Forces. GAO-19-206. Washington, D.C.: February 21, 2019. Military Personnel: DOD Needs to Improve Dental Clinic Staffing Models and Evaluate Recruitment and Retention Programs. GAO-19-50. Washington, D.C.: December 13, 2018. Military Personnel: Additional Actions Needed to Address Gaps in Military Physician Specialties. GAO-18-77. Washington, D.C.: February 28, 2018. Defense Health Reform: Steps Taken to Plan the Transfer of the Administration of the Military Treatment Facilities to the Defense Health Agency, but Work Remains to Finalize the Plan. GAO-17-791R. Washington, D.C.: September 29, 2017. Military Compensation: Additional Actions Are Needed to Better Manage Special and Incentive Pay Programs. GAO-17-39. Washington, D.C.: February 3, 2017. Defense Health Care Reform: DOD Needs Further Analysis of the Size, Readiness, and Efficiency of the Medical Force. GAO-16-820. Washington, D.C.: September 21, 2016. Defense Health Care: Actions Needed to Help Ensure Full Compliance and Complete Documentation for Physician Credentialing and Privileging. GAO-12-31. Washington, D.C.: December 15, 2011. Military Cash Incentives: DOD Should Coordinate and Monitor Its Efforts to Achieve Cost-Effective Bonuses and Special Pays. GAO-11-631. Washington, D.C.: June 21, 2011. Military Personnel: Status of Accession, Retention, and End Strength for Military Medical Officers and Preliminary Observations Regarding Accession and Retention Challenges. GAO-09-469R. Washington, D.C.: April 16, 2009.", "summary": "DOD invests in a number of incentives to recruit and retain its nearly 15,000 military physicians and dentists, such as providing a tuition-free education to medical and dental students who in return agree to serve as military physicians or dentists for a specific amount of time. Section 597 of the John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to review military physicians' and dentists' compensation, among other things. This report addresses, among other objectives, (1) how compensation for military physicians and dentists compared to private sector civilians with comparable skills in 2017, and (2) the extent to which DOD has developed an approach to recruit and retain military physicians and dentists through a package of incentives that reflect key principles of effective human capital management. GAO compared military and civilian cash compensation for 2017—the most recent year of data amongst data sources, assessed incentive packages against key principles of human capital management, and conducted surveys and held focus groups to obtain the perspectives of current military medical students and residents regarding military service obligations. In 2017, cash compensation for military physicians and dentists in most of the 27 medical and dental specialties GAO reviewed was generally less than the median compensation of private sector civilians, but the Department of Defense (DOD) provides substantial deferred and noncash benefits, such as retirement pensions and tuition-free education, whose value to servicemembers is difficult to determine. GAO found that for 21 of the 27 physician and dental specialties, the maximum cash compensation was less than the private sector civilian median within four officer pay grades (O-3 to O-6) (see figure for number of physician specialties by pay grade). Moreover, cash compensation for military physicians and dentists was less than the private sector civilian median at key retention points, such as after physicians and dentists fulfill their initial active-duty service obligations. DOD recruits and retains physicians and dentists through a package of incentives, including tuition-free medical or dental school and special and incentive pays, such as multi-year retention bonuses. However, DOD does not consistently collect information related to the following three key principles of effective human capital management to help inform investment decisions in its package of recruitment and retention incentives: Replacement costs . DOD does not consistently collect information on replacement costs of military physicians and dentists. However, DOD has previously identified replacement costs as a factor in assessing the appropriateness of incentive pays. Current and historical retention information . DOD does not consistently collect information on retention of physicians and dentists, specifically acceptance rates for retention bonuses, to help assess the effectiveness of these bonuses. Private sector civilian wages. DOD does not consistently collect information on private sector civilian wages. Officials stated that civilian wages are not a driving factor when considering adjustments to special and incentive pays, in part because DOD cannot always match civilian sector compensation for military physicians and dentists. By collecting and using this information to help inform its decision-making, DOD would be better positioned to assess the effectiveness of its incentives to recruit and retain military physicians and dentists and make sound investment decisions for the future. GAO recommends that DOD should collect and use information on (1) replacement costs of military physicians and dentists, (2) retention, and (3) private sector civilian wages to inform its investment decisions. In commenting on a draft of this report, DOD concurred with these recommendations.", "document_type": "gao"}
{"report": "There are a number of organizations within DOD with responsibility for preventing, responding to, and resolving incidents of child abuse, including child-on-child abuse, as described below. Under Secretary of Defense for Personnel and Readiness. The Under Secretary of Defense for Personnel and Readiness collaborates with DOD component heads to establish programs and guidance to implement the FAP, among other things; it also programs, budgets, and allocates funds and other resources for the FAP. The Assistant Secretary of Defense for Manpower and Reserve Affairs, under the authority of the Under Secretary of Defense for Personnel and Readiness, provides policy, direction, and oversight to the FAP. The Assistant Secretary of Defense for Manpower and Reserve Affairs, through the Deputy Assistant Secretary of Defense for Military Community and Family Policy, is also responsible for collaborating with service Secretaries to monitor compliance with FAP standards. The Defense State Liaison Office, located within the Office of the Deputy Assistant Secretary of Defense for Military Community and Family Policy, is responsible for assisting with the passage of state bills that affect key issues within the department, such as the reporting of child abuse. DOD Family Advocacy Program. DOD FAP serves as the policy proponent for, and a key element of, DOD’s coordinated community response system to prevent and respond to reports of child abuse, domestic abuse, and problematic sexual behavior in children and youth in military families. The FAP, among other things, provides trauma- informed assessment, rehabilitation, and treatment to persons who are involved in alleged incidents of child abuse, domestic abuse, and problematic sexual behavior in children and youth who are eligible to receive treatment in a military treatment facility. To execute these responsibilities, DOD funds over 2,000 positions in the department to deliver FAP services, including credentialed and licensed clinical providers. The department prescribes uniform standards for all service FAPs through DOD Manual 6400.01, Volume 1, FAP Standards. DOD uses these standards to promote public awareness; aid in prevention, early identification, reporting, and coordinated, comprehensive intervention and assessment; and to support victims of child abuse and domestic abuse. DOD revised these standards in July 2019 to include the same support and services for children exhibiting or affected by problematic sexual behavior. Military Service Family Advocacy Programs. Each military department Secretary is responsible for developing service-wide FAP policy that addresses any unique requirements for their respective installation FAPs. The department Secretaries are also responsible for requiring that all installation personnel receive the appropriate training to implement the FAP standards. In addition, each service has a FAP headquarters entity that develops and issues implementing guidance for the installation FAPs for which they provide oversight. At the installations, commanders are to establish an installation Family Advocacy Committee with a chairperson that serves as the policy implementing, coordinating, and advisory body to address child abuse and domestic abuse at the installation. Military Criminal Investigative Organizations and Military Police. The Department of Defense Inspector General establishes policy, provides guidance, and monitors and evaluates program performance for all DOD activities relating to criminal investigations and military law enforcement programs, including coordination with DOJ. Military law enforcement organizations include both military police and military criminal investigative organizations. Each military department has established a military criminal investigative organization that may initiate investigations on incidents with a DOD nexus, such as if a crime occurred on a military installation or involved military personnel or dependents. The military departments’ military criminal investigative organizations are the Army Criminal Investigation Command, Naval Criminal Investigative Service, and Air Force Office of Special Investigations. Each military criminal investigative organization provides an element of DOD’s special victim investigation and prosecution capability. DOD defines special victims as adults or children who are sexually assaulted or suffer aggravated assault with grievous bodily harm. A special victim investigation and prosecution designation allows the military criminal investigative organizations to assign specially trained investigators who work collaboratively with other relevant trained personnel, such as Judge Advocates and FAP managers, to provide services to the victim. While military criminal investigative organizations can investigate any crime with a DOD nexus—within their investigative purview—officials from each organization stated that they primarily investigate serious felony-level offenses and any type of sexual offense. Military police that provide services at military installations primarily serve as first responders to incidents and will notify a military criminal investigative organization for more serious incidents requiring an investigation, according to service officials. DOD Office of the General Counsel and Service Judge Advocates. The DOD Office of General Counsel provides advice to the Secretary of Defense regarding all legal matters and services performed within, or involving, DOD. The DOD Office of General Counsel also provides for the coordination of significant legal issues, including litigation involving DOD and other matters before DOJ. Each military department also has a Judge Advocate General’s Corps that establishes legal offices (Offices of the Staff Judge Advocate) which, among other things, serve as prosecutors and defense counsel at courts-martial; provide legal assistance to eligible personnel on personal, civil, and legal matters; advise commanders on military justice and disciplinary matters; and provide legal advice to military investigative agencies. In addition, any person identified as the victim of an offense under the Uniform Code of Military Justice (or in violation of the law of another jurisdiction if any portion of the investigation is conducted primarily by the DOD components) is to be notified of their rights under DOD’s Victim and Witness Assistance Program, informed about the military justice process, and provided other services to support the victim or witness and their family. DOD Education Activity. DODEA operates as a DOD field activity under the Office of the Under Secretary of Defense for Personnel and Readiness. It is a federally-operated school system that is responsible for planning, directing, and managing prekindergarten through 12th grade educational programs for DOD. All DODEA personnel are designated as mandatory reporters of child abuse and are required to participate in the early identification of child abuse and the protection of children, including the prompt reporting of alleged child abuse or any information that gives reason to suspect child abuse. FAP is responsible for several child abuse prevention programs across the services. For example, the New Parent Support Program offers intensive home visiting services on a voluntary basis to expectant parents and parents with young children. Officials target the program toward families who display some indicators of being at risk for child abuse or who have been assessed and determined as at risk for child abuse. All FAP personnel are mandated reporters to state child welfare service agencies for all allegations of child abuse. In addition, the service FAPs, at every military installation where families are located, work with the other entities within the coordinated community response, including civilian social services agencies and law enforcement, to provide comprehensive prevention and response to maltreatment. According to service FAP officials, while each service FAP has a domestic abuse victim advocate program that serves domestic abuse victims as well as non-offending parents in child abuse incidents, specific prevention efforts vary across installations and services. For example, the Air Force FAP is taking steps to track the effectiveness of FAP treatment programs to strengthen prevention efforts. Through the Navy FAP’s victim advocate program, non-offending parents are connected with resources from initial referral to case closure—or until the non-offending parent no longer desires services—that include potential prevention techniques, such as establishing a strong support system. The Marine Corps initiated evaluation of prevention programs and uses evidence-informed curricula to provide parenting education and support, according to Marine Corps officials. The Army has begun to operationalize combined parent-child cognitive behavior therapy to address the needs of children and families at risk for child physical abuse through child interventions, parent strategies to address child trauma, and family interventions. At one Army installation, a FAP official described a puppet show aimed at teaching children about appropriate and inappropriate behaviors as part of prevention efforts related to problematic sexual behavior in children and youth. Other DOD organizations also have roles related to prevention. For example, child development centers located on installations have a number of child abuse prevention measures, including visual access throughout activity rooms used for care, closed circuit television, identification checks and badges for all visitors, and a system to indicate which staff members are cleared to be alone with children, such as a system of colored smocks. In addition, all personnel on military installations who work with children, including those at DODEA schools, child development centers, and child and youth centers, must pass a background check as a condition of employment, among other things. Each military installation with a FAP has an Incident Determination Committee (IDC) that reviews reported incidents of child abuse and domestic abuse to determine whether they meet DOD’s criteria for abuse. Per DOD guidance, every reported incident of abuse or neglect must be presented to the IDC unless there is no possibility that the incident could meet any of the criteria for abuse or neglect. Physical abuse, emotional abuse, and neglect each have two primary associated criteria: (a) an act or failure to act, and (b) physical injury or harm, or the reasonable potential for physical injury or harm; psychological harm, or the reasonable potential for psychological harm; or stress-related somatic symptoms resulting from such act or failure to act. Any act of child sexual abuse that is found to have occurred under part (a) is automatically considered to have had a significant impact on the child, which is the criterion for part (b); therefore, the IDC only considers part (a) for incidents of child sexual abuse, and if the IDC determines the act occurred, then the incident is found to have met criteria. Voting members of the IDC include: the deputy to the installation commander (Chair); the senior noncommissioned officer advisor to the installation commander; representatives from the servicemember’s command, the Staff Judge Advocate’s office, and military police; and the FAP manager or FAP supervisor of clinical services. According to DOD policy, the IDC may request that additional personnel, such as medical personnel and military criminal investigative organizations, attend the IDC when necessary to provide input on incidents and to answer any questions about the results of a medical examination or an investigation. IDC members review what is known about the incident, and then the voting members vote to determine if an incident meets each of DOD’s criteria for abuse. The final incident determination is made by a simple majority vote, and the IDC Chair serves as the tiebreaker in the event of a tie. The IDC’s decision is communicated to the servicemember via the servicemember’s command. IDC determinations may be reconsidered. The appeal request and response processes vary by service. In August 2016, DOD issued guidance standardizing the IDC process across the services. According to DOD officials, prior to this, each service had a similar but distinct process for determining whether abuse occurred. According to a DOD report, the IDC is to be a clinical, not a disciplinary, process. The IDC is separate and distinct from any law enforcement or military criminal investigative organization process. Each incident that is presented to the IDC is also discussed at a clinical case staff meeting, which is made up of personnel from the FAP, among others. During the clinical case staff meeting—which can occur before or after the IDC makes its determination, according to DOD officials— attendees generate clinical recommendations for support services and treatment for victims and offenders of child abuse who are eligible for treatment at a military medical treatment facility, and ongoing coordinated case management. DOD FAP officials stated that treatment is not dependent on an IDC’s determination, meaning that the FAP may still provide support services to the family even if the IDC finds that a reported incident does not meet DOD’s criteria for abuse. The Executive Office for United States Attorneys provides general executive assistance and supervision to the Offices of the United States Attorneys, including evaluating their performance, making appropriate reports and inspections, and taking corrective action when needed. The Executive Office for United States Attorneys also serves as a liaison between DOJ and the 93 United States Attorneys located across the 50 states, the District of Columbia, and some U.S. territories. United States Attorneys serve as the nation’s principal litigators and work under the direction of the Attorney General to prosecute crimes, including some crimes that occur on some military installations. When cases from military installations are referred to a United States Attorney’s office for prosecution, they can be accepted, referred, or declined. The case can be declined for prosecution for several reasons: (1) it may not constitute a federal offense, (2) there is insufficient evidence to obtain a conviction, (3) prosecution would not serve a substantial federal interest, (4) the individual may be prosecuted in another jurisdiction, or (5) there is another adequate noncriminal alternative to prosecution. DOJ’s Criminal Division comprises multiple sections, including the Child Exploitation and Obscenity Section and the Human Rights and Special Prosecutions Section, both of which have responsibility for resolving crimes occurring on overseas military installations. The mission of the Child Exploitation and Obscenity Section is to protect child welfare and communities by enforcing federal criminal statutes relating to the exploitation of children and obscenity. The Human Rights and Special Prosecutions Section primarily investigates and prosecutes cases against human rights violators and other international criminals. The Office of Juvenile Justice and Delinquency Prevention within DOJ’s Office of Justice Programs provides national leadership, coordination, and resources to prevent and respond to juvenile delinquency and victimization. The Office supports the efforts of states, tribes, and communities to develop and implement effective and equitable juvenile justice systems that enhance public safety, ensure youth are held appropriately accountable to both crime victims and communities, and empower youth to live productive, law-abiding lives. In addition to DOD and DOJ, there are also community partners that assist in responding to and resolving incidents of child abuse, including child-on-child abuse. Depending on the military installation, there may be local memorandums of agreement or understanding between the installation and community partners, such as CACs, child welfare agencies, and civilian law enforcement that help guide the response to and reporting of these incidents. The National Children’s Alliance and Children’s Advocacy Centers. The National Children’s Alliance is the national association and accrediting body for a network of approximately 900 CACs with locations in all 50 states and the District of Columbia. CACs provide a child-focused environment to conduct child forensic interviews and medical exams, which are then reviewed by a multi-disciplinary team that includes medical personnel, law enforcement, mental health personnel, legal personnel, victim advocates, and state child welfare agencies. The purpose of the multi-disciplinary team is to determine how to best support the child, such as through therapy, courtroom preparation, and victim advocacy. State and local child welfare agencies and civilian law enforcement. Each state or locality has a public child welfare agency that is responsible for receiving and investigating reports of child abuse, as well as assessing the needs of children and their families. This could include removing a child from an abusive home or providing support services to families in need. These agencies are governed by state laws that define child protection roles and processes. The administrative framework for child welfare services and programs vary by state, but all are responsible for compliance with state and applicable federal requirements. For example, states that accept federal funding under the Child Abuse Prevention and Treatment Act must meet the statutory requirements of the Act. Civilian law enforcement organizations are also key to ensuring the welfare of children. In general, civilian law enforcement organizations act as first responders to incidents and may provide a variety of services from reporting the abuse to the appropriate child welfare agency to conducting an investigation of the incident. As of 2018, DOD occupied varying legislative jurisdictions throughout the 26.9 million acres of land at 4,775 sites worldwide for which it is responsible. Military installations may consist of one or more sites. In the United States, military installations have one of four types of legislative jurisdiction—or, depending on the installation, multiple types of jurisdiction—that, among other things, helps determine the proper adjudication venue for any criminal offenses committed on the property of the installation. The four types of jurisdiction are described below. Exclusive federal jurisdiction gives the federal government sole authority to adjudicate criminal misconduct. Exclusive federal jurisdiction exists when the federal government elected to reserve authority at the time the real property was granted to the state, or when the state transferred real property to the federal government and failed to reserve jurisdictional authority as part of the transfer. Concurrent jurisdiction applies when both the state and the federal governments retain all authority to adjudicate criminal misconduct. In the event of a conflict, the federal government prevails under the Supremacy Clause of the Constitution. Partial jurisdiction applies when both the state and the federal government have some legislative authority, but neither one has absolute power. The sharing of authority is not exclusive to adjudication of criminal misconduct and federal supremacy applies in the event of a conflict. Proprietary jurisdiction applies to instances where the federal government has virtually no legislative authority. The only federal laws that apply are those that do not rely upon federal jurisdiction, such as espionage, bank robbery, tax fraud, and counterfeiting; the federal government maintains immunity and supremacy for inherently governmental functions. An installation commander can exclude civilians from the area pursuant to his or her inherent authority. The installation’s jurisdiction as well as the status of the alleged offender (civilian or servicemember) determines which venue will adjudicate the incident. For example, if a servicemember commits a crime in exclusive federal jurisdiction, the adjudication would likely fall under the Uniform Code of Military Justice. If a civilian commits a crime in exclusive federal jurisdiction, he or she may be prosecuted under federal law through the appropriate United States Attorney’s Office. However, if a civilian commits a crime in concurrent or proprietary jurisdiction, he or she may be prosecuted by the state. The age of the accused is also an important consideration because the intent of federal laws concerning juveniles is to help ensure that state and local authorities will deal with juvenile offenders whenever possible. Exclusive federal jurisdiction may be relinquished in part or completely to a state, and this action is referred to as the retrocession of jurisdiction. The conference report accompanying the John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for the Secretaries of the military departments to seek to relinquish jurisdiction, such that the state, commonwealth, territory, or possession would have concurrent jurisdiction over offenses committed on military installations by individuals not subject to the Uniform Code of Military Justice, such as civilian dependents and children. The conference report also directed the Secretaries of the military departments to report to the defense committees on these efforts 15 months after the enactment of the Act. In June 2019, the Acting Deputy Secretary of Defense issued a memorandum directing each military department to seek to establish concurrent jurisdiction with the respective states for offenses committed by juveniles in areas on military installations that are currently exclusive federal jurisdiction. This action seeks to provide ways for the department to address actions by children in areas of exclusive federal jurisdiction that may constitute a crime, such as some instances of problematic sexual behavior in children and youth, since, absent unusual circumstances, children and other civilians are not subject to the Uniform Code of Military Justice. According to Army and department officials, states—whose juvenile courts are rehabilitative in nature—are much better equipped to deal with suspected crimes committed by children than the federal government, which does not have a juvenile justice system. These officials also noted that federal prosecution is usually declined for such cases. There are various laws and agreements in place regarding crimes committed on U.S. military installations or involving servicemembers or military dependents overseas. These laws include U.S. criminal laws that may be applied extraterritorially, the Military Extraterritorial Jurisdiction Act, the Uniform Code of Military Justice, and host nation laws. Whether a particular law provides extraterritorial jurisdiction over such crimes depends on the specific facts of the incident, such as the nature and location of the alleged crime, the status of the alleged offender (servicemember or civilian), and the nationalities of the alleged offender and the victim. Status of forces agreements between the United States and the host nation may also clarify how these circumstances should be considered in determining venue. Three primary issues limit DOD’s visibility over reported incidents of child abuse and child-on-child abuse—standalone databases, information sharing challenges, and installation discretion. The military services use standalone databases to track the reporting, response to, and resolution of each reported incident of child abuse, which limits the department’s visibility over these incidents. While DOD is developing a new database to track problematic sexual behavior in children and youth, it has not yet made key decisions about its development and implementation, which could further affect visibility. In addition, challenges related to information sharing limit visibility over child abuse incidents within and across the military services. Further, Family Advocacy Program (FAP) installation personnel are given considerable discretion in deciding how reported incidents of child abuse are tracked and reported, as are DODEA school personnel with regard to incidents of child-on-child abuse, which also hinders the department’s visibility over these incidents. Each military service maintains multiple standalone databases that separately track the reporting, response to, and resolution of each reported incident of child abuse, which limits DOD’s visibility over the extent to which children have been affected by abuse on military installations or as military dependents and its visibility over incident outcomes. Depending on the reported incident, information regarding the alleged abuse may be retained in multiple databases or only one database. Specifically, each service’s FAP has a database—referred to as the “central registry”—where it tracks the total number of reported incidents of child abuse (by a parent or someone in a caregiving role) and detailed information, such as information about the offender, victim, and type of abuse, for incidents that met DOD’s criteria for abuse. Incidents of abuse where the alleged offender was not in a caregiving role are not tracked in the FAPs’ central registries and would only be tracked as incidents of abuse if they were investigated by military law enforcement. Information associated with investigations of these incidents by any military criminal investigative organization is tracked in a separate database maintained by each investigative organization. If the alleged offender was a servicemember, information related to the adjudication or case resolution is tracked in the relevant service’s military justice database maintained by the services’ legal offices. Figure 1 shows the department’s databases for tracking the abuse of children and how they differ depending on the circumstances of the incident. Because of DOD’s multiple standalone data systems, it is difficult to know the extent to which children have been affected by abuse on military installations or as military dependents. From fiscal years 2014 through 2018, the military service FAPs’ central registries recorded more than 69,000 reported incidents of child abuse, of which 48 percent met DOD’s criteria for abuse. Over this same time period, the military criminal investigative organizations conducted approximately 9,500 investigations involving a child victim, some but not all of which may have also been recorded in the service FAPs’ central registries. Figures 2 and 3 show the number of incidents of child abuse reported to the military service FAPs and the number of military investigations involving a child victim from fiscal years 2014 through 2018, respectively. However, the number of incidents tracked by both organizations cannot simply be added together because, as previously discussed, there is some overlap between them. For example, an incident of child sexual abuse inflicted by a servicemember parent or a teacher would likely be in both databases. Moreover, neither the service FAPs nor the military criminal investigative organizations individually track all reported incidents of abuse. Specifically, the FAP only tracks information related to abuse inflicted by a parent, guardian, or someone in a caregiving role. It does not capture incidents of abuse inflicted by, for example, a neighbor who was not babysitting at the time of the incident. While the services’ military criminal investigative organizations track any abuse of a child that rises to their level of investigation, such as a felony or sexual offense—regardless of the relationship between the alleged offender and the victim—they only investigate certain crimes. For example, an incident of child neglect would likely only be in the FAP’s central registry because incidents of neglect do not typically rise to the level of a military criminal investigative organization investigation. Similarly, an August 2019 report by the Defense Health Board found that it is difficult to establish the true incidence of child abuse across the department due to challenges associated with the underreporting of cases and unreliable capture of data. Standalone databases also limit DOD’s visibility over incident outcomes. Depending on the reported incident of abuse—for example, child sexual abuse inflicted by a servicemember parent—to get the most complete picture of how the incident was reported, responded to, and resolved, service officials would need to query three databases: the FAP, military criminal investigative organization, and military justice databases. Navy legal officials stated that a centralized database for all child abuse incidents—that tracks the FAP’s determination about whether the incident met DOD’s criteria for abuse, the investigation, and resolution—would be beneficial because it is currently very difficult to track an incident from the initial report to its final outcome in order to easily determine what happened in a particular case. These officials further stated that such a database would benefit commanders’ oversight of cases for which they are responsible. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision directing DOD to establish and maintain a centralized database on each incident of problematic sexual behavior in children and youth reviewed by an installation FAP. Specifically, per the statute, for each substantiated and unsubstantiated incident of problematic sexual behavior, the database is to track a description of the allegation, whether or not a FAP review of the case has been completed, the status and results of any related law enforcement investigation, and the nature of any action taken. Officials responsible for the development of the database—which is supposed to begin in fiscal year 2020—stated that it will maintain information related solely to cases of problematic sexual behavior and will not include other types of child-on-child abuse, such as physical assaults not of a sexual nature. Additionally, these officials stated that they do not have plans to expand the scope of the database to include any adult-on-child inflicted abuse. As a result, even once the centralized database on problematic sexual behavior in children and youth is implemented, DOD will still lack a centralized mechanism to track the reporting, response to, and resolution of other incidents of abuse involving children that were reported to the FAP or investigated by a military law enforcement organization— specifically, any abuse or neglect inflicted by an adult or physical abuse inflicted by another child. DOD officials responsible for the development of the database stated that they do not plan to expand the scope of the centralized database because they do not want to conflate the processes for responding to incidents of adult-inflicted child abuse and incidents of problematic sexual behavior. While the response process differs between incidents of adult-inflicted child abuse and incidents of problematic sexual behavior, DOD officials acknowledged that the organizations involved in the response process and the primary data sources are the same. Additionally, DOD FAP officials stated the scope of the centralized database was defined in statute and that they foresee additional privacy and data-safeguarding issues if they were to expand its scope. While the statute indicated what must be included in the database, it did not limit the scope of the database to those required elements. DOD not only lacks visibility over incidents of problematic sexual behavior, but over any reported abuse of a child and could therefore benefit from a centralized tracking mechanism for all such incidents. With regard to privacy and data-safeguarding concerns, according to DOD, data-safeguarding precautions were taken when developing the Defense Sexual Assault Incident Database, which the department successfully implemented. While the Defense Sexual Assault Incident Database does not contain information pertaining to children, it contains sensitive information that the department has taken steps to protect. Specifically, according to DOD, the Defense Sexual Assault Incident Database is reviewed annually to ensure all security controls are maintained and it is secured using physical, technical, and administrative controls, such as role-based permissions, to maintain the privacy of personal information. DOD FAP officials also expressed concerns about maintaining information about both adults and children in the centralized database. However, information about both adults and children is included in the service FAPs’ central registries and the military criminal investigative organizations’ databases. DOD officials responsible for developing the database noted that the department already plans to take precautions when developing the database due to the collection and retention of information about children. Standards for Internal Control in the Federal Government states that management should use quality information to achieve the entity’s objectives. Specifically, quality information is appropriate, current, complete, accurate, accessible, and provided on a timely basis. In addition, management should design control activities to achieve objectives, such as clearly documenting significant events in a manner that allows the documentation to be readily available for examination. Without a centralized database that tracks all incidents of abuse involving children that were reported to the FAP or investigated by a military law enforcement organization, DOD and Congress will not know the extent to which children have been affected by abuse on military installations or as military dependents, or how such incidents have been responded to and resolved—making it difficult to identify and address trends that could lead to further prevention efforts. While DOD is in the early stages of developing a centralized database to track incidents of problematic sexual behavior in children and youth, it has not yet made key decisions about its development and implementation, which could further affect visibility over such incidents. Specifically, DOD has not yet identified all information requirements, developed a plan for how it will use the data it collects, or established a schedule for development and implementation. DOD officials responsible for developing the database stated that they are still in the process of selecting a vendor to develop the system and that once a contract has been awarded and is underway, they can make such decisions. Our prior work has found that inadequate acquisition planning, including poorly defined requirements and unrealistic cost estimates, can increase the risk that the government may receive services that cost more than anticipated, are delivered late, and are of unacceptable quality. Given that DOD officials stated they plan to select a vendor in early fiscal year 2020 and move quickly with development—expecting to complete the bulk of it in fiscal year 2020—it is an appropriate time to make these decisions. First, DOD has not yet identified all of the information it will track in the database. DOD officials responsible for the development of the centralized database stated that they have not yet identified all of the information the database will track—other than the information required by statute and some information related to the response process— because they are still in the early stages of the development process. However, as previously discussed, DOD officials expect to complete the bulk of the development this fiscal year. In November 2006, we found that establishing a valid need and translating that into a service acquisition requirement is essential for obtaining the right outcome. Without this, an organization increases the risk that it will pay too much for the services provided, acquire services that do not meet its needs, or enter too quickly into a sensitive arrangement that exposes the organization to financial, performance, or other risks. Additionally, Standards for Internal Control in the Federal Government states that management should use quality information to achieve the entity’s objectives, which includes identifying information requirements that consider the expectations of both internal and external users. As DOD progresses in its development of the centralized database, identifying and defining the elements that each responsible organization, such as the FAP and military law enforcement, must track would help to ensure that the data collected are useful, accurate, and complete, and that the data collected ultimately increase the department’s visibility over these incidents. Second, DOD has not yet determined how it will use the data it collects from the database to increase visibility. DOD officials stated that because they have not yet finalized the information requirements for the database, they have not yet developed a plan for how the collected data will be used. GAO-identified leading practices for results-oriented management have shown that data-driven decision making leads to better results. Further, agencies can use performance information to identify problems or weaknesses in programs, to try to identify factors causing the problems, and to modify a service or process to try to address problems. As DOD progresses in the development of its database, developing a plan for data-driven decision making—that details how the department will use the data to help inform program development and increase visibility— would help DOD to assess its processes and procedures for responding to and resolving incidents of problematic sexual behavior in children and youth, identify any needed changes, and modify them as appropriate. Finally, DOD has not yet established a completion date for the database or developed a schedule to guide its development and implementation. According to DOD officials responsible for the development of the database, while they do not have a planned completion date for the database or any associated milestones, they plan to select a vendor for the development in early fiscal year 2020 and they anticipate the majority of the development will take place the same year. These officials stated that they have not yet set a completion date, in part, because of the sensitivity of the information being collected and because the department does not have a comparable database that collects and maintains information on children. In addition, while these officials stated that they had identified resources for the development of the database through fiscal year 2020, they had not yet identified funding for future years. GAO-identified practices for developing and maintaining a reliable schedule include: (1) capturing all key activities, (2) sequencing all key activities, (3) assigning resources to all key activities, (4) integrating all key activities horizontally and vertically, (5) establishing the duration of all key activities, (6) establishing the critical path for all key activities, (7) identifying float—the amount of time a task can slip before affecting the critical path—between key activities, (8) conducting a schedule risk analysis, and (9) updating the schedule using logic and durations to determine the dates for all key activities. Given that DOD is in the early stages of development, establishing a reliable schedule for the development and implementation of the centralized database—including key activities and the timeframes and resources needed to execute them—would provide the means to gauge progress, identify and address potential problems, and promote accountability. Until the database is implemented, DOD will continue to have limited visibility over incidents of problematic sexual behavior in children and youth. Information sharing challenges limit visibility within each military service— specifically, as it relates to required notifications between a service’s installation FAP office and military law enforcement about reported incidents of child abuse inflicted by a parent or someone in a caregiving role. DOD policy states that the Secretaries of the military departments are to ensure that installation commanders or service-equivalent senior commanders ensure that the installation FAPs immediately report any allegations of child abuse and any criminal allegations to the appropriate law enforcement authority. Similarly, service guidance states that military law enforcement is responsible for notifying the installation FAP office of reported or suspected incidents of child abuse. However, officials at four installations in our review described notification challenges between these organizations. For example, officials at one installation described a child abuse incident that had been investigated by military law enforcement for 2 to 3 months, but the investigating organization had not notified the installation’s FAP office. Legal officials at another installation stated that over the past year, there had been five incidents of child abuse that were reported to the installation FAP office, but that the FAP had not reported to military law enforcement. These officials stated that the lack of notifications can be frustrating for commanders who need complete information about these incidents to determine whether they need to take any action. In addition, DODEA policy states that, among other things, DODEA personnel are to promptly report all suspected or alleged incidents of child abuse to the installation FAP office and the relevant child welfare agency, if available. The policy does not require them to also report the suspected abuse to law enforcement, but the FAP is to report the incident to law enforcement. However, a senior DODEA official stated that one of its regions has instituted a procedure for all child abuse incidents to be reported to the FAP and law enforcement because the region had experienced challenges with the FAP not consistently notifying law enforcement. The extent of these notification challenges is unknown because service FAP and military law enforcement officials stated that they do not document in their central registries or military criminal investigative organization databases whether each notified the other. Service FAP and military law enforcement officials stated that they can add fields to their databases to track new information if provided with the direction and resources to do so. Officials from these organizations also noted that any notification to the other entity may instead be documented in any case notes or in the case file. However, in April 2019, the DOD Office of Inspector General evaluated military law enforcement incident reports and found similar notification challenges related to FAP and military law enforcement notifications for domestic violence incidents. Specifically, the DOD Office of Inspector General evaluated 212 military law enforcement domestic violence reports in which a FAP notification was required and for 23 percent of the incidents (49 incidents) the military law enforcement organization had not notified the FAP as required. Standards for Internal Control in the Federal Government states that management should internally communicate information to achieve the entity’s objectives. For example, information is communicated down, across, and up reporting lines to all levels of the entity. In addition, the oversight body receives quality information that flows up the reporting lines from management and personnel. Without directing the service FAPs and military law enforcement organizations to document in their respective databases the date that they notified each other, these entities’ headquarters will remain limited in their oversight abilities to ensure that these notifications occur and to take appropriate actions in response. Even if notifications are documented in case files, there is no mechanism for the headquarters entities to efficiently determine whether a notification was made. Without ensuring that notifications are made to both organizations, which play critical roles in addressing incidents of child abuse, it is possible that an incident may not be fully assessed by the FAP or investigated by military law enforcement. Notification delays could result in at-risk children remaining in an unsafe environment or could delay time-critical portions of an investigation, such as forensic interviews or sexual assault exams. Information sharing challenges limit visibility across the military services, specifically as it relates to sharing child abuse incident determinations. Installation officials stated that the lead service for any installation is responsible for the installation’s FAP. They stated that even though the Incident Determination Committee (IDC) will hear cases about the other services’ members and dependents, all information is recorded in the lead service’s central registry. For example, if an Air Force servicemember is involved in a reported incident of child abuse while on an Army installation, the Army FAP will record information about the incident in its central registry. Of the Air Force FAP’s more than 3,000 reported incidents that met criteria for child abuse from fiscal years 2014 through 2018 and had a servicemember offender, 22 percent of those offenders were from one of the other three services. For the Army, the Navy, and the Marine Corps, 2 percent, 9 percent, and 5 percent, respectively, of their records were associated with servicemembers from another service. Table 1 shows the number of child abuse incidents that met DOD’s criteria for child abuse and involved a servicemember offender from fiscal years 2014 through 2018, by the service that recorded the incident and servicemember affiliation. Since FAP personnel at the installations do not share access to the other service’s central registries or the DOD Central Registry, according to DOD FAP officials, they have established a process to share information about child abuse allegations and determinations across the services. Per DOD guidance, the service FAPs are to submit data from their central registries on a quarterly basis for consolidation into DOD’s Central Registry. According to DOD FAP officials, after the service FAPs submit their data, the Defense Manpower and Data Center reviews the data and identifies any child abuse incidents that met DOD’s criteria for abuse and were recorded by a service FAP that is not the service to which the servicemember is assigned. According to these officials, the Center then forwards those relevant incidents to the services to which the servicemembers are assigned with the expectation that they will incorporate them into their central registries. According to Air Force, Navy, and Marine Corps FAP officials, they regularly incorporate the data received from the Center into their central registries so that they can be searched by FAP personnel at the installations. However, DOD does not have guidance that describes how the service FAPs should receive information from the Center about child abuse allegations and determinations that involve their personnel, but were recorded by another service’s installation FAP, or how they should incorporate such information into their central registries once received. Further, according to DOD FAP officials, DOD does not have a process to monitor that the service FAPs are consistently incorporating the information they receive from the Center into their central registries. Standards for Internal Control in the Federal Government states that management should internally communicate information to achieve the entity’s objectives. In addition, management should implement control activities through policies and establish and operate monitoring activities and evaluate the results. Specifically, ongoing monitoring is built into the entity’s operations, performed continually, and responsive to change. For example, one of the required fields in the service FAPs’ central registries is whether the offender was previously known to the service’s central registry—meaning that the offender was involved in a previous incident of child abuse or domestic abuse that was presented to the service FAP and was determined to meet DOD’s criteria for abuse. However, if the incident of abuse occurred on another service’s installation, and was therefore recorded in that other service’s central registry—and the service to which the servicemember is assigned was either not informed or did not input the information into its central registry—the servicemember’s FAP may not be aware of the prior case and therefore may not record the offender as previously known. Issuing guidance that describes the process through which the service FAPs are to receive and incorporate information into their central registries regarding child abuse allegations and determinations involving their servicemembers and dependents—that also includes a mechanism to monitor that the process is consistently occurring—would provide better assurance that the services have complete and up-to-date information about their personnel and their dependents, which ultimately affects their visibility over such incidents. FAP personnel at all seven installations in our review stated that they screen reported incidents of child abuse to determine whether to present them to the IDC. DOD guidance states that every reported incident of child abuse must be presented to the IDC for a determination unless there is no possibility that the incident could meet any of the criteria for child abuse or neglect. However, installation personnel described reported incidents of child abuse that had been screened out that, per DOD guidance, should have been presented to the IDC. For example, FAP officials at one installation stated that they screen out reports of spanking by a parent if there is no mark. Since DOD’s list of actions considered to be nonaccidental physical force includes spanking, it meets at least one of DOD’s criteria for child abuse and should be presented to the IDC for a determination. The IDC would then determine whether there was a significant impact on the child, such as a welt or a more than superficial bruise, or the reasonable potential for a more than inconsequential physical injury or fear reaction—to determine whether the reported incident meets all of DOD’s criteria for child physical abuse. Officials from three of the services’ FAPs stated that if spanking is used as a discipline technique—without information of injury or potential for injury or psychological harm—then it should not be opened as an incident and presented to the IDC. However, this is in conflict with DOD guidance as confirmed by DOD FAP officials. At another installation, child development center officials described an incident where a staff member was speaking harshly with a child. These officials stated that the supervisor at the center considered the action to be child abuse—berating the child, which per DOD guidance is an act of emotional abuse—and contacted the installation FAP. However, they stated that the FAP personnel that received the report stated, without any assessment of the incident, that it was not emotional abuse and that the center should handle it administratively. According to center officials, the incident was never presented to the IDC, but they considered the incident to be significant enough that the center terminated the staff member’s employment. FAP officials at a different installation stated that the medical clinics were not previously reporting suspected abuse to the FAP, but are now doing so. Because of this change, the FAP personnel said they believe the clinics are over-reporting, which has led to the FAP personnel screening out some of the clinic’s reported incidents of suspected child abuse. Two of the parents of children affected by abuse that we interviewed discussed incidents that were reported to the FAP, but that the FAP did not initially present to the IDC. According to one parent, one incident of child abuse was presented to an IDC at a different installation after the parent contacted the FAP at that installation for advice more than 2 years after the initial report of abuse. According to the other parent, the other incident of child sexual abuse was only presented to the IDC following congressional involvement. FAP personnel at one installation described the process of determining whether a reported incident should be presented to an IDC as a clinical judgement call and noted that they screen out about one-third of reported incidents of child abuse annually. FAP personnel at another installation stated that, as of summer 2019, they had received about 50 reported incidents of child abuse since the start of the calendar year and that they had screened out the majority of them. While installation FAP personnel also described reported incidents of abuse that should be screened out as child abuse per DOD guidance—such as abuse where the alleged offender was not a parent, guardian, or someone in a caregiving role, which is outside of the FAP’s purview—it is unclear how many of the reported incidents that they have screened out should have been presented to the IDC per the guidance. Incidents that are not presented to the IDC are not recorded in the relevant service FAP’s central registry and therefore are not captured in DOD’s consolidated Central Registry, which the department uses to prepare its statutorily required annual reports to Congress on child abuse and domestic abuse. As a result, the actual total number of reported incidents of child abuse across the department—which according to our previously discussed analysis totaled more than 69,000 from fiscal years 2014 through 2018—may be higher. As previously discussed, the Defense Health Board’s August 2019 report noted that it is difficult to establish the true incidence of child abuse across the department due to challenges associated with the underreporting of cases and unreliable capture of data and that as a result, it is difficult to measure and monitor the scope of the problem. When we discussed with DOD FAP officials what the installations we visited told us about how they screen reported incidents of child abuse, officials expressed concerns about these installations not adhering to DOD guidance. However, as previously discussed, the service FAPs are responsible for overseeing installation FAPs. According to service FAP officials, oversight of the screening process is primarily handled by personnel at each installation. Air Force FAP officials stated that the FAP personnel making these screening determinations have to meet certain education requirements. Standards for Internal Control in the Federal Government states that management should establish and operate monitoring activities and evaluate the results. Without each military service developing a process to monitor how reported incidents of child abuse are screened at installations, the services cannot be sure that incidents are being presented to the IDC in a consistent manner. Further, installation FAPs may continue to screen out reported incidents of child abuse, in contradiction of DOD guidance, therefore excluding them from being documented in DOD’s Central Registry. As a result, DOD does not know and cannot accurately report on the total number of reported incidents of child abuse across the department. In addition to other known underreporting, without such initiatives, DOD is further limiting its visibility over incidents and hindering its ability to ensure appropriate responses to incidents. According to our analysis of DODEA data, DOD schools may not be reporting all serious incidents of child-on-child abuse, which hinders DODEA leadership visibility. From school years 2013-2014 through 2017-2018, across its 163 schools, DODEA reported a total of 167 serious incidents involving either an alleged violation of law or an alleged sexual event—on average, one serious incident per school over the 5- year period. The types of reported serious incidents included a student reporting that they were raped by two students in the school parking lot, a student stabbing another student in the finger with a plastic fork and drawing blood, and a wide range of other conduct. There was a slight decrease in the number of serious incidents reported from school years 2013-2014 to 2014-2015, but since school year 2014-2015, the number of serious incidents reported each year increased from a low of 22, to 55 in school year 2017-2018. DODEA officials attribute the increased reporting, in part, to the issuance of additional reporting guidance in August 2016. Figure 4 shows the number of serious incidents involving either an alleged violation of law or an alleged sexual event reported by DODEA from school years 2013-2014 through 2017-2018. According to DODEA officials, all serious incident reports are reviewed by DODEA headquarters to ensure that the schools took the appropriate actions needed to protect students and to ensure that incidents are correctly categorized. These officials stated that the reports also help to increase visibility at the headquarters level about the types of incidents occurring in DODEA schools and where additional resources may be needed. In addition, DODEA officials stated that they retain serious incident reports for 5 years, which allows them to track serious conduct issues when students transfer schools. While the reporting of serious incidents has increased, our analysis of DODEA student misconduct records found that schools’ reporting of these incidents was incomplete. Specifically, our analysis identified 216 student misconduct records for school years 2016-2017 and 2017-2018 that school administrators, following DODEA guidance, could have reasonably classified as serious incidents. The types of incidents described in the student misconduct records included, among other things, the use of physical force by a student on another student that resulted in an injury; a student touching another student’s groin, breasts, or buttocks without consent; and verbal and behavioral sexual harassment. However, for this time period—for which DODEA reported the highest number of serious incidents from school years 2013-2014 through 2017-2018—DODEA only reported 89 serious incidents. In addition, DODEA officials stated that prior to August 2018, up to one-third of schools were not recording student misconduct in the student information system because they were not required to do so and, as a result, we were not able to review any misconduct records for those schools. Challenges related to the reporting of serious incidents were also highlighted in our interviews with parents and DODEA school administrators. Specifically, two of the parents of children affected by child-on-child sexual abuse that we interviewed discussed incidents that occurred within DODEA schools. They both stated that they received information about the incidents as part of Freedom of Information Act requests and that the schools had not reported the abuse as serious incidents. For one of these incidents, we identified a corresponding DODEA child abuse report, but not a serious incident report. Per DODEA guidance, the incident should have been categorized as a serious incident (but not as child abuse) because the offender was a student— child abuse reports are only to be filed if the alleged offender was an individual responsible for the child’s welfare, such as a parent or a teacher. In addition, at one installation in our review, FAP personnel discussed a recent sexual assault within a DODEA school. When we discussed this incident with a senior DODEA official who is to be notified of all serious incidents reported in the region in which the school is located, the official was unaware of the incident because it was not categorized as a sexual assault in the serious incident report and another senior official for the region had handled it directly. Further, administrators at one of the DODEA schools we visited stated that the reporting guidelines are not fully clear and that they often call the superintendent’s office for advice on what to report and how to report it. Standards for Internal Control in the Federal Government states that management should internally communicate the necessary quality information to achieve the entity’s objectives. Specifically, management communicates quality information down and across reporting lines to enable personnel to perform key roles in achieving objectives. However, DODEA’s guidance affords school administrators discretion in what to report because it does not explicitly define what types of serious incidents must be reported. While the guidance identifies and defines a number of incidents that could be reported as serious incidents, and provides detailed examples—like a student intentionally exposing their genitals or a student posting naked or suggestive photos of another student online— the guidance does not mandate that these incidents be reported. Specifically, the guidance states that the lists of events, activities, and paraphernalia described in the guidance as serious incidents are illustrative only and do not identify every incident that may be inappropriate, nor require that each incident result in a serious incident report. While DODEA officials noted that both reporting and their visibility over serious incidents has been improving, they acknowledged that administrators may not be reporting all serious incidents described in the guidance because, in part, it may be easier for them to resolve some incidents—such as students jokingly slapping each other on the buttocks—at the school level instead of filing a serious incident report. These officials stated that they are optimistic a new reporting database for serious incidents that they implemented in August of 2019 will streamline the process for administrators and increase reporting. In addition, in February 2019, DODEA issued guidance related to the reporting of and response to prohibited sexual, sex-based, and other related abusive misconduct, which DODEA officials told us they believe will reduce discretion in how alleged child-on-child sexual abuse is recognized and reported. While the new reporting system and guidance related to child-on-child sexual abuse are positive steps, without additional guidance that clarifies the types of incidents—including non-sexual incidents—that must be reported as “serious incidents,” DODEA may continue to lack full visibility into the extent to which serious incidents are occurring. As a result, systemic issues within a particular school or district may never be reported to DODEA leadership and any additional resources that a school or district needs to prevent future incidents may not be identified. Further, when a student transfers schools, the new school may be unaware of serious conduct issues that were not properly documented, raising safety concerns for the school and installation. DOD and the military services have taken steps to expand child abuse policies and procedures to address child-on-child abuse in response to Congress, but gaps exist in the processes for responding to and resolving incidents of abuse. Specifically, the services may lack pertinent stakeholder perspectives on the IDC after DOD policy changed the permanent voting membership of the committee. In addition, families of child abuse victims may receive inconsistent levels of information following a report of child abuse, which can cause confusion and prevent them from receiving available services. Further, service guidance regarding the extent of commander authority to remove children from unsafe homes overseas is unclear. Finally, the availability of certified pediatric sexual assault forensic examiners is limited, especially overseas. In accordance with provisions in the John S. McCain National Defense Authorization Act for Fiscal Year 2019, DOD and the military services have taken steps to augment existing child abuse policies and procedures to also include child-on-child abuse, specifically the incidence of problematic sexual behavior in children and youth. The statute required, among other things, that the Secretary of Defense establish a policy, applicable across all military installations, to respond to allegations of problematic sexual behavior in children and youth on military installations. The purpose of the policy is to ensure a consistent, standardized response to such allegations across the department. In May 2019, DOD issued a revised FAP instruction that establishes policy, assigns responsibilities, and prescribes procedures for the FAP specific to child abuse, domestic abuse, and problematic sexual behavior in children and youth. In addition, in July 2019, DOD revised the FAP standards to implement policy, assign responsibilities, and provide procedures for addressing problematic sexual behavior in children and youth in military communities. As of October 2019, the military services had not yet issued their updated FAP policies to incorporate the new department- wide policy and standards, but the policies were under development, according to DOD FAP officials. Prior to the issuance of DOD’s updated FAP policy, the Army issued a broader policy on major juvenile misconduct in March 2019. The policy addresses the command response to juvenile misconduct and the referral of juvenile cases to civilian authorities. For Army installations in the United States with areas of exclusive federal jurisdiction, the policy directs such commands to seek to establish concurrent jurisdiction of juvenile criminal offenses. In instances where establishing concurrent jurisdiction is not feasible or recommended, the policy directs commanders to pursue memoranda of agreement with local prosecution authorities that address the referral of juvenile cases to the local juvenile court system for state review and state determination of appropriate disposition. Army officials stated that the Army policy covers more than incidents of problematic sexual behavior in children and youth because the challenges involving children on Army installations are broader than problematic sexual behavior and encompass other types of misconduct, such as fights, vandalism, and shoplifting. Officials from the other services stated that their policies, which are under development, will focus on problematic sexual behavior because that was what was required per statute. In addition, DOD has taken steps to implement a training program for personnel at installations that focuses on problematic sexual behavior in children and youth. Specifically, DOD and DOJ’s Office of Juvenile Justice and Delinquency Prevention entered into an interagency agreement in July 2019 to expand the scope of DOJ’s cooperative agreement with the University of Oklahoma. According to DOD officials, this agreement includes providing training and technical assistance in support of DOD’s response to problematic sexual behavior in children and youth. The 3-year interagency agreement provides $1.5 million in funding, and according to DOD officials, the funding will be used to develop and implement targeted training on problematic sexual behavior in children and youth for FAP personnel at the installations. According to DOJ officials, other efforts include a DOJ and DOD working group on child-on-child sexual abuse—focused on resolving jurisdictional issues, as will be discussed in greater detail later in the report—and the development of a centralized database for tracking incidents of problematic sexual behavior in children and youth, as previously discussed. Further, DODEA has implemented a number of initiatives related to serious student misconduct. These include the issuance of a standalone sexual harassment policy and providing administrators with additional guidance on reporting and responding to sexual activity within DODEA schools, and the development and distribution of standardized language regarding discrimination and sexual harassment for each school’s student handbook. DODEA also created outreach materials for students on how to recognize and respond to sexual harassment. DODEA has conducted training for administrators on these topics. Other training initiatives include training for all counselors, school psychologists, and nurses on problematic sexual behavior in children and youth. As previously discussed, DODEA also introduced a new reporting database for serious incidents in August 2019 that is intended to simplify the serious incident reporting process for administrators. In August 2016, DOD issued guidance to standardize the incident determination process across the military services, which, among other things, reshaped the permanent voting membership of the IDC. However, the new structure may lack stakeholders with the requisite knowledge and expertise to allow the IDC to make fully informed determinations. The standardized process to determine whether an incident meets DOD’s criteria for child abuse was informed by a collaboration between the Air Force and New York University researchers, which yielded a decision- tree algorithm. The process was implemented by the Air Force and then subsequently adopted by the Navy and the Marine Corps. According to Army officials, the Army’s phased implementation of the IDC process was ongoing as of October 2019. As part of the standardization of the process in the 2016 guidance, medical personnel were removed as permanent voting members of the IDC, although they regularly participated in some of the services’ prior incident determination processes, according to Army FAP officials. The external researchers involved in the effort noted that they were primarily involved in the decision-tree algorithm and not the composition of voting members, which was an internal DOD decision. According to DOD, the definitions in the decision-tree algorithm used to determine if an incident meets criteria to be considered child abuse were robust enough that experienced healthcare providers were not needed to determine if an incident met DOD’s criteria for child abuse. In addition, DOD FAP officials stated that participation in the IDC process by medical personnel could take them away from their clinical duties and become burdensome since the IDC at larger installations may meet weekly and for several hours. DOD officials noted that medical personnel, and others, can still be invited to participate in the IDC process as needed to provide information related to specific incidents. While IDC members at four of the installations in our review also noted that medical personnel can still be invited to share relevant case information—in a nonvoting capacity—medical personnel we spoke to at three of these installations noted that they are rarely invited to participate. As a result, medical personnel at one installation we visited stated that they have attempted to write their medical reports in more lay terminology to bridge the gap and to help ensure that critical information is properly relayed during the IDC meeting. Medical personnel with expertise in child abuse stated that they would welcome the opportunity to again participate in IDC meetings about which they have specific knowledge, but that they are contacted to participate once every 2 years at the most. In addition, medical personnel at one of the installations we visited had never heard of the IDC and were unaware of its function. During a number of our interviews and installation visits, medical personnel frequently expressed concerns about the lack of medical expertise in the IDC process. For example, medical personnel at three installations we visited expressed concerns that the absence of medical personnel on the IDC may prevent reported incidents of child abuse from being fully understood. They noted that medical personnel—specifically, pediatricians—have particular utility on the IDC because of the complexity of some of the cases and the need to articulate how medical findings can indicate whether an injury resulted from a nonaccidental use of force. Medical personnel with expertise in child abuse stated that there is a strong medical component to many child abuse cases and that FAP clinicians may not have the requisite medical expertise needed to appropriately interpret that information. Medical personnel also stated that lacking this expertise could result in the IDC incorrectly voting that an incident meets criteria for abuse or does not meet criteria. For example, a pediatrician described one IDC meeting in which they were invited to participate, as a nonvoting member, related to an incident that had medical evidence that the pediatrician referred to as clearly presenting a hallmark finding in child abuse—ear bruising patterns in a very young child. However, the pediatrician stated that the IDC voted that the incident did not meet DOD’s criteria for abuse before allowing medical personnel to present information they had about the incident. According to this pediatrician, after the vote, the IDC allowed the pediatrician to provide information about the incident, but it did not alter the committee’s initial determination. At one of the IDC meetings we observed, IDC members discussed a case that involved bruising. The IDC members noted that they wished that a doctor had been present so that they could determine whether the allegation had any merit. However, no medical personnel were present and the IDC reached a determination without medical input. Members of this IDC also discussed concerns about a downward trend across the service in the number of cases meeting DOD’s criteria for abuse, which they attributed to changes to the voting membership of the committee. In addition, one of the parents that we spoke with described an incident that met DOD’s criteria for child sexual abuse under the military service’s prior incident determination process. However, the parent stated that after the service implemented the new IDC process, the servicemember’s command—which was added as a permanent voting member of the IDC—requested that the determination be reconsidered. The parent stated that the incident was again presented to the IDC and the committee reversed the initial determination, concluding that the incident did not meet DOD’s criteria for child sexual abuse. The parent expressed concerns that the removal of medical personnel from the IDC process played a significant role in the reversal of the determination. Further, at one installation in our review, after the installation implemented the new IDC process, officials set up a separate pre-IDC process to discuss the same cases with medical personnel and others to ensure that they include their perspectives in the determination process. Installation officials stated that they felt the need to establish this redundant process because participation and discussion are more limited under the IDC process and there was an identified gap. In August 2019, the Defense Health Board recommended that DOD reconsider requiring at least one comprehensive pediatric medical health care provider to be a member of all IDCs. However, DOD FAP officials stated that they have no plans to reassess or expand the voting membership based on this recommendation or the concerns expressed by medical personnel across the military services. They stated that there are other meaningful ways in which medical personnel can participate in the IDC process, but that they should not be voting members because their competing clinical responsibilities may lead to a lack of continuity on the IDC and they might not have any direct knowledge of the incidents being discussed. However, as previously discussed, medical personnel are not being regularly invited to participate and, when they are, the information they present may not be considered as part of the voting process. In addition, medical personnel at one installation we visited noted that even if they were regularly invited to participate, since they are not permanent voting members, other clinic responsibilities may take precedence. A 2018 Department of Health and Human Services guide for child protective caseworkers noted that involving teams with a diversity of skillsets, including pediatricians, early in the child abuse determination process can improve accurate and comprehensive assessments, information sharing, and analysis of gathered information to support an accurate substantiation decision. In addition, GAO-developed practices to enhance and sustain collaboration in interagency groups note that it is critical to involve nonfederal partners, key clients, and stakeholders in decision-making. Further, in February 2014, we found that if collaborative efforts do not consider the input of all relevant stakeholders, important opportunities for achieving outcomes may be missed. Without expanding the voting membership of the IDC to include medical personnel, installation officials may not have all of the relevant information to make a fully informed decision about whether an incident meets DOD’s criteria for child abuse. The IDC may make different determinations without the benefit of input from all relevant personnel, thus affecting confidence in the efficacy of the process. Further, without expanding the voting membership to include medical personnel, installations may continue to develop concurrent or redundant processes in order to ensure that all pertinent information about cases is shared. Victims’ families receive inconsistent levels of information related to the response process and available services after an incident of child abuse is reported. The process to respond to and address incidents of child abuse can be lengthy—the average investigation is more than 9 months—and the responding organization and the particular steps it takes depend on variables including the type of abuse, the status of the alleged offender, and the location of the incident. For example, as previously discussed, military criminal investigative organizations primarily only investigate serious felony-level offenses and any type of sexual offense. According to military criminal investigative organization officials, cases that do not meet this threshold may be investigated by other military law enforcement investigators, such as military police or local civilian law enforcement. Additionally, the FAP only reviews incidents of child abuse where the alleged offender was a parent or someone in a caregiving role. As a result, the FAP would not present incidents to the IDC where the alleged offender was another child or an adult who was not in a caregiving role, such as a neighbor who was not babysitting at the time of the incident. Further, as previously discussed, the jurisdiction of the installation where the incident took place determines which entity, such as the state or the federal government, will adjudicate the incident. The process can also differ based on the state and local laws where the incident occurred. For example, according to some state child welfare agencies, they are more likely than the FAP to accept cases of child-on- child abuse, and they review such cases to see if a lack of supervision or other aspect of parental neglect is involved. The legal services that victims are eligible to receive differ depending on the status of the alleged offender and the victim, and the type of abuse alleged. For example, for incidents of child sexual abuse with an alleged servicemember offender, victims and their families are eligible for military- provided legal advice and assistance, even if the abuse occurred off the installation. However, the status of the victim (that is, whether the victim is the dependent of a military member or not) will impact the nature and extent of the legal assistance that can be provided. Of the 20 parents of children affected by abuse that we interviewed, nine stated that they did not understand what to expect during the investigation and resolution process and nine were not aware of all available services and resources offered. Some parents noted that if they had better understood the process and available services, they could have received counseling and other services more quickly. Twelve parents highlighted that a guide summarizing the process and available services would have been helpful. For example, seven parents said that they did not receive and were not offered any services by the military. Multiple respondents also highlighted the lack of sufficient legal assistance. Specifically, five parents stated that they would have liked legal assistance but none was available, and seven parents stated that the legal services offered by the military did not meet their needs. For example, one parent stated that they requested a waiver to receive the services of a Special Victims’ Counsel, but the request was denied for reasons that are unclear. Standards for Internal Control in the Federal Government states that management should externally communicate the necessary quality information to achieve the entity’s objectives. Specifically, management communicates with and obtains quality information from external parties, including the general public, and in this case victims’ families. However, while each organization, such as the FAP, may provide information to families relevant to that organization’s responsibilities and services, the military services have not established efforts to comprehensively inform victims’ families about how child abuse incidents are to be addressed by each responsible organization, for example by consolidating information to help families understand the process and the services available to them. While DOD officials stated that they have plans to develop such a guide for responding to incidents of problematic sexual behavior in children and youth, they stated that they do not have plans to develop a similar guide for responding to incidents of child abuse because information is already available from a number of different sources. However, the parents we spoke with had challenges locating this information in a timely manner following an incident of child abuse and highlighted the need for additional information in a consolidated format to avoid confusion and to more easily receive necessary services. Without each military service establishing efforts to comprehensively inform victims’ families about how reported incidents of child abuse will be addressed, affected families may be confused about the process and where to go for information. In addition, they may not receive the services that they are entitled to and need, such as a Special Victim Counsel or a legal assistance attorney, because they do not know that these resources are available. As a result, DOD may not be providing comprehensive responses to reported incidents of child abuse. The military services’ guidance regarding the extent of commander authority to remove children from their homes on overseas installations is unclear. Within the United States, state and local child welfare agencies have the authority to remove children from unsafe homes. However, officials at an overseas installation stated that there is no law that clearly authorizes commanders to exercise this authority on overseas installations, and there may be no local authorities to provide guidance or services at overseas installations. Rather, service guidance grants installation commanders the authority to remove children from unsafe homes on a temporary basis. Guidance describing this authority is not standardized across the services and installation officials overseas stated that additional guidance would help clarify situations when a child can be removed from an unsafe home. For example, according to Army guidance, an installation commander may authorize emergency placement care when abuse is substantiated and when neither judicial authorization nor parental consent can be obtained, and the removal is necessary to avoid risk of imminent death, serious bodily harm, or serious mental or physical abuse. In addition, commanders may take action in situations when medical protective custody is not appropriate. Per Navy guidance, commanders can only use this authority in situations where there is substantial reason to believe the life or health of the child is in real and present danger. Air Force guidance states that base security and unit leadership are responsible for overseeing the appropriate removal or placement of children with consultation and guidance from the FAP. Per Marine Corps guidance, commanders may implement a child removal order—designed for short- term placement of a child into a place of safety. Individual installation commanders are responsible for issuing a written policy setting forth the procedures and criteria for the removal of child victims of abuse or other children in the household when they are in danger of continued abuse or life-threatening child abuse. Officials at installations overseas stated that the decision to remove a child from an abusive home can vary depending on the commander’s comfort level in doing so. For example, officials at two installations provided an example where a commander removed a child from the home in a situation of suspected abuse, and then a parent requested an Inspector General investigation questioning the commander’s authority to do so. Installation officials stated that the complaint to the Inspector General was not substantiated, but that the ambiguity of the guidance, coupled with the possibility of a commander having his or her actions reviewed by the Inspector General, could affect a commander’s willingness to take action in similar cases. Medical personnel we spoke with highlighted examples where military hospitals overseas have admitted child abuse victims for their safety in situations when installation commanders did not take action to otherwise remove the child from the home. In one example, an infant presenting with physical trauma consistent with abuse was admitted to the hospital for 1 month until the child could be returned to the United States and a state child welfare agency could respond to ensure the child’s safety. Installation officials overseas responsible for addressing incidents of child abuse stated that they believe additional clarity regarding commander authorities would help commanders in making a determination about when to exercise their authority to remove an at-risk child from a home. In comparison to the services’ guidance, some state child welfare agencies have comprehensive checklists and decision matrices to help officials make decisions regarding child removal. One child welfare agency we visited provided a list with 14 specific safety factors, including descriptions of each factor, and a list of 10 protecting interventions. Safety factors include anything that may put a child in danger, for example, questionable caretaker explanations for a child’s injuries, or the family not allowing the child welfare agency access to the child. Protecting interventions include actions such as the family making use of community agencies or services as a safety resource, or the non-offending caretaker moving to a safe environment with the child. There is no comparably detailed guidance for military commanders. Standards for Internal Control in the Federal Government states that management should internally communicate the necessary quality information to achieve objectives. Quality information is reported down and across reporting lines to enable personnel to perform key roles in achieving objectives. However, legal officials and medical personnel at overseas installations stated that existing guidance regarding commander authority to remove children from potentially unsafe homes in overseas environments is unclear. For example, these medical officials stated that terms like “real and present danger” are not well defined, and that there may be no child welfare agency available overseas to provide guidance or services. These officials also stated that this threshold may be too high, and could result in children suffering moderate neglect or abuse because it does not rise to the level of real and present danger. Without clarifying and standardizing across the services, in guidance, the circumstances under which commanders may exercise their authority to remove children from potentially unsafe homes overseas, timely response to incidents may be inhibited and children may be left in unsafe situations. Commanders may also face adverse actions if their authority to remove a child from the home is not well-defined and their decision comes under legal scrutiny. The availability of certified pediatric sexual assault forensic examiners across the military services is limited, especially overseas. Based on our analysis, from fiscal years 2014 through 2018, for all four military services, there were 1,448 incidents that met DOD’s criteria for child sexual abuse and may have therefore necessitated a sexual assault forensic exam. According to our analysis of FAP data over these 5 years, the average age of the victims involved was 10. However, according to Defense Health Agency officials, there are only four child abuse pediatricians who are certified to perform pediatric sexual assault forensic exams: two in the Navy, one in the Army, and one in the Air Force. In addition, according to these officials, the Army has seven sexual assault forensic examiners, initially certified to perform exams on adults, who have completed a 40-hour pediatric course, for a total of 11 certified pediatric examiners across the department. In comparison, according to these officials, there are a total of 466 sexual assault forensic examiners throughout the department who are certified to perform exams on adults—161 are located overseas and 305 are located within the United States. As a result of this disparity between examiners certified to perform exams on adults and those certified for children, children affected by sexual abuse on military installations or as military dependents may lack access to qualified pediatric sexual assault forensic examiners. This lack of access on overseas installations—identified by medical personnel as a significant concern—can prevent them from being examined in a timely manner or may subject them to further trauma if they are first examined by an untrained provider and have to be examined again. When victims of sexual assault receive a forensic exam, the exam may be provided by either a trained sexual assault forensic examiner—that is, a medical provider who has received specialized training in properly collecting and preserving forensic evidence—or a medical provider who has not received such specialized training. Studies have shown that exams performed by trained sexual assault forensic examiners may result in shortened exam time, better quality health care delivered to victims, higher quality forensic evidence collection, as well as better collaboration with the legal system and higher prosecution rates. Navy officials stated that pediatric sexual assault forensic examiners are not a billeted position at any installation and Air Force officials stated that there are no certified pediatric sexual assault forensic examiners billeted to any installation in Japan—which hosts the largest number of active duty U.S. servicemembers outside of the United States—due to inconsistent demand. Medical personnel we spoke with described two options to overcome the lack of certified pediatric examiners: call a certified pediatric examiner in the United States to guide—via telephone—a pediatrician on the overseas installation through the exam; or medically evacuate the victim to the United States. Although DOJ best practices for sexual assault exams note that telemedicine can result in significant positive changes in the methods of examination and evidence collection, medical personnel stated that it is inferior to an in-person exam because the person conducting the exam is not the actual certified examiner, which can open the exam findings up to legal challenges. Medical personnel also stated that a child may need to undergo multiple exams if the initial exam is not performed correctly, which, as noted previously, can add to a victim’s trauma. Additionally, medical personnel stated that there can be technical challenges with getting the right equipment in place and training people who may quickly transition to another installation. If telemedicine processes were to be established at overseas installations, there are still only four child abuse pediatricians across the department who can consult on the exams, and they may not be available to consult on all cases. Further, medical personnel noted that using telemedicine for pediatric exams overseas may result in these exams being physically conducted by someone with little to no experience conducting any type of genital exam. This is because pediatricians in the military typically do not conduct any genital exams on children, even basic or preventative exams. In the event that a girl becomes pregnant, officials stated that she will be sent to a military adult obstetrician, and the military pediatrician would not conduct any of the relevant exams. These personnel also stated that the military does not conduct routine cervical exams on women until they are 21 years of age, so pediatricians likely have no practical experience conducting even standard exams. A 2018 Department of Health and Human Services guide for child protective caseworkers noted that if health care providers do not routinely examine the genitalia of young children, they may mistake normal conditions for abuse or vice versa. One parent that we spoke with about an incident of sexual abuse overseas stated that the child’s pediatrician was not comfortable conducting such an exam, but offered to take a cursory “peek” for anything concerning. The parent declined the offer because they knew the pediatrician was neither trained nor certified to perform such an exam. Although medical personnel stated that a medical evacuation to the United States for an exam is a potential option, medical evacuations are challenging because they can take 5 to 6 days. However, the physical evidence from a sexual assault should be collected as soon as possible and ideally between 1 and 5 days after the assault, according to DOJ best practices. Additionally, installation medical personnel noted that medical evacuations can result in additional stress on the victim from travel, increased complexity of legal and investigation processes, and travel costs that may be greater than training local examiners. DOD medical personnel stated that it can be challenging because in some instances the children can only receive the exam at medical facilities that have a memorandum of understanding in place with the military because the exams are typically funded locally. For example, these officials described an incident of child sexual abuse in Okinawa—a remote location in Japan with no certified examiners. These personnel noted that while a medical evacuation to Hawaii would seem like a good solution—because there is a trained pediatrician there to conduct sexual assault exams—the pediatrician in Hawaii can only examine children who have been referred directly by Hawaii’s child welfare agency. These personnel noted that the next best option is San Diego, where there is a DOD child abuse pediatrician, but by the time the travel is arranged, which can take days, the evidence might no longer be available. These personnel suggested that instead of relying on medical evacuations or telemedicine, better options to ensure that child victims get timely access to care could include certifying pediatricians or adult sexual assault forensic examiners as pediatric examiners during mandatory training or establishing shared regional assets. In the United States, child victims of sexual abuse may have more options to receive pediatric sexual assault forensic exams. Specifically, pediatric exams may be done at a local Children’s Advocacy Center (CAC) or hospital. However, it is still challenging in the United States because CAC coverage is not uniform across the country, and rural patients may have to travel several hours to the closest center. For example, officials at one CAC we visited noted that while they have a certified pediatric examiner, this individual is only available once per week. One parent that we spoke with stated that they had to drive their child 2 hours to the closest CAC to receive an exam when stationed at an installation in the United States. Two parents described delays in receiving an exam in the United States after the incident was reported, which could have prevented quality evidence from being collected. DOJ protocols for sexual assault forensic exams state that these exams should be performed by a healthcare professional specially trained in collecting evidence relating to sexual assault cases, such as a sexual assault nurse examiner or other appropriately trained medical professional. In particular, female children who have not yet reached puberty should only be examined by health care providers specifically trained in pediatric sexual abuse. Further, related DOJ best practices state that evidence should be collected as soon as possible, ideally between 1 and 5 days post assault. However, DOD does not have processes in place to help ensure that children who are sexually abused overseas have timely access to certified pediatric sexual assault forensic examiners. Without processes that help ensure timely access to certified pediatric examiners overseas, child victims of sexual abuse may not receive exams in time for the evidence to be collected for use in prosecution. In addition, the difficulty and time associated with obtaining an exam could potentially increase the stress and trauma of affected victims and their families. Further, because of the variation in resources across military installations, child victims of sexual abuse may have access to different levels of care depending on the geographic location of the installation due to the lack of standardized availability of certified pediatric examiners. DOD collaborates at various levels both inside and outside the department to address reported incidents of child abuse and child-on- child abuse. However, improving communication and establishing comprehensive agreements could enhance the information DOD receives about these incidents as well as the resources available to both the department and victims of abuse. DOD has successfully collaborated with a number of states to help ensure it receives notification from state authorities when servicemembers or military dependents are involved in reported incidents of child abuse off a military installation. DOD is required to address child abuse in military families. However, with approximately 70 percent of active-duty military families living off military installations in the civilian community, service officials do not always have visibility over these incidents since they may first be reported to the relevant civilian authorities instead of to the military. The Defense State Liaison Office has highlighted the importance of state statutes that require the collection and reporting of military affiliation to the appropriate military authorities as part of state child abuse cases, and has identified this as a key issue. According to a senior Defense State Liaison Office official, the office has successfully collaborated with a number of states on child abuse reporting measures to require or allow local jurisdictions to report incidents of child abuse in military families to relevant military service officials. According to DOD, at least half of the states have no such requirements, but at least one is considering passing a law to provide for the requirement. According to this senior Defense State Liaison Office official, the effort will remain a key issue area for the office through at least fiscal year 2020 in order to continue to focus efforts on these remaining states. In August 2019, the Defense Health Board noted that child abuse can be difficult to quantify because of underreporting, and some studies suggest a lower rate of incidents being reported to the FAP if the incidents are first identified at a civilian facility. Therefore, it recommended, in the absence of state legislation, that DOD ensure that all U.S. military installations have memorandums of agreement in place with state child welfare agencies for bilateral information sharing on child abuse cases. A senior Defense State Liaison Office official stated that the office has sought legislation because prior efforts to establish memorandums of agreement were only focused on information sharing and did not specify procedures for state and local child welfare agencies to use in determining whether a family involved in an incident had a military connection. Additionally, the official noted that a statutory basis is important because otherwise state laws that limit who child welfare agencies can share information with about child abuse cases may take precedence. For example, some states have expressed concerns that sharing information about an alleged, but not yet confirmed, incident of child abuse could be detrimental to a servicemember’s career. We found that the extent of collaboration between the military and other state and local authorities (such as child welfare agencies) varied among the installations in our review. For example, child welfare agency officials in Virginia noted that state policies requiring that they notify the FAP about cases with a military affiliation have increased the amount of coordination between the state and the military. However, according to FAP officials at one installation we visited in North Carolina—where approximately 80 percent of dependent children live off the installation—it was rare to receive notification from some counties for child abuse cases with a military affiliation because, at the time of our visit, there was no state policy requiring it. DOD’s continued focus on improving collaboration with the states that have not yet established such a requirement should help to increase the department’s visibility over incidents occurring off the installation. It should also help to ensure that military families obtain the available FAP services for which they are eligible. DOD and DOJ have taken some actions to increase collaboration in addressing the abuse of children on military installations. As previously discussed, the conference report accompanying the John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for the service Secretaries to seek to relinquish jurisdiction over offenses committed on military installations by individuals not subject to the Uniform Code of Military Justice, such as civilians and children. In response, according to DOJ officials, DOD and DOJ have, among other things, established a joint working group to coordinate on issues related to child-on-child sexual assault on military installations, including the relinquishment of exclusive federal jurisdiction to the states. Both DOD and DOJ officials agreed that the federal justice system is not well suited to prosecuting juvenile offenses because it lacks a dedicated juvenile justice system and that state courts, which aim to be rehabilitative in nature, are better suited to adjudicate these cases. Specifically, DOJ’s Justice Manual states that the intent of federal laws concerning juveniles is to help ensure that state and local authorities will deal with juvenile offenders whenever possible. Working group officials stated that they are compiling a list of United States Attorneys’ Offices and the military installations in their respective districts from which they have received referrals, as well as the types of jurisdictions at those installations. These efforts are designed to ultimately result in a comprehensive chart detailing the precise jurisdictional status of each military installation in the United States, which can then be used to inform discussions with each state about the relinquishment of exclusive federal jurisdiction. According to DOJ officials, the working group is also developing templates of coordination documents, such as letters and memoranda of understanding for outreach with the states. Working group officials stated that the group has identified and is attempting to address other issues, such as those regarding privacy concerns related to information to be contained in DOD’s centralized database for problematic sexual behavior in children and youth, which, as previously discussed, is under development. The difficulties of addressing child-on-child sexual assault are exacerbated when the incident occurs overseas, where no U.S. state authorities exist to assume jurisdiction. The Military Extraterritorial Jurisdiction Act can be used to either prosecute child offenders as adults—for certain violent or controlled substance violations—or to initiate federal delinquency proceedings. However, as discussed, while both DOD and DOJ officials stated that they prefer to refer children to state courts, this is currently not possible when the incident occurs overseas. Working group officials stated that this challenge is another issue being actively discussed by the group in an effort to identify potential solutions. For example, they stated that one idea under discussion relates to a specific Virginia state law that asserts concurrent jurisdiction over federal crimes committed by a child, to be assumed only if waived by the federal court or the United States Attorney. The discussion centered on the idea that the Virginia state law could potentially be applied extraterritorially. Therefore, if a sexual assault were to occur on an installation with exclusive federal jurisdiction in Virginia—or theoretically overseas where the United States has jurisdiction—the Virginia courts could assert jurisdiction as long as the relevant United States Attorney’s office has waived jurisdiction. However, whether or not Virginia could use its courts to address matters that occurred overseas and where the juvenile offender is not a resident is not yet clear. Legal officials at one installation who are involved in the working group efforts stated that they were considering whether it was possible to have a single municipal court have sole jurisdiction for any juvenile crimes occurring on overseas installations. However, officials stated that the working group continues to research and discuss these types of issues to improve collaboration between the two departments and identify solutions to these important issues. Service officials stated that while DOD is typically notified by DOJ when it declines to prosecute the abuse of a child on a military installation, the notification does not consistently include detailed reasons for why the case was declined. Officials from the Army Criminal Investigation Command—the military criminal investigative organization with the largest number of cases—stated that they are not informed of the reasons for case declinations because they have been told that the information is considered an attorney work product. Officials from the other military criminal investigative organizations stated that for some cases they do receive reasons why they are declined. However, DOJ officials stated that in cases where a United States Attorney does notify DOD of a declination and the reason, it may be very vague, such as “insufficient evidence,” and may not detail the insufficiencies. DOJ officials stated that while a case may be declined for various reasons, there are three primary reasons for declinations: (1) insufficiency of the evidence (not enough admissible evidence to obtain and sustain a conviction beyond a reasonable doubt); (2) the person is subject to prosecution under another jurisdiction, such as in a state court system; or (3) there is an adequate noncriminal alternative to criminal prosecution. Officials within the Executive Office for United States Attorneys stated that they were not aware of any standard letters used to notify DOD of prosecutorial decisions and that the format and content of the notification are office dependent. According to DOJ officials, the investigating organization is to inform victims of a declination of prosecution. However military law enforcement officials from two services stated that the responsibility for informing victims of a declination of prosecution would be dependent on the circumstances of the individual case, such as whether formal charges had been preferred and any discussion between the military criminal investigative organization and United States Attorney. According to some of the parents we spoke with, this process does not always result in a timely notification of a prosecution declination—including the reasons for the declination—to the victims or their families. For example, one parent we spoke with highlighted the lack of information when they tried repeatedly for nearly one year to contact the military investigators for a case status update—while in the process of filing an information request with DOJ—and were finally told that their child’s case had been declined for prosecution with no additional information on the reasons for the declination. Another parent stated that the Assistant United States Attorney informed them that a child-on-child abuse case would not be prosecuted due to a lack of strong evidence, specifically, a poor child forensic interview conducted by the military criminal investigative organization and the mishandling of electronic evidence. DOJ has committed to assisting DOD in responding to incidents of child- on-child abuse through the working group, as discussed previously. Additionally, DOJ has begun tracking referrals made to United States Attorneys by DOD for child-on-child sexual offenses. Specifically, in September 2018, the Director of the Executive Office for United States Attorneys issued a memorandum that instructed all United States Attorneys to begin tracking referrals of child-on-child sexual offenses from the military. According to these data, between October 1, 2018 and August 5, 2019, the military referred 63 of these cases to United States Attorneys for prosecution. Two of these cases were accepted for prosecution and 19 were declined—the remaining cases were either referred to state or local authorities or were still pending. Per the memorandum, this information is to be provided, on a monthly basis, to the Office of the Deputy Attorney General, the lead DOJ office for the working group. DOJ’s Principles of Federal Prosecution recommends that whenever an attorney declines to prosecute, the prosecutor should ensure the decision and reasons are communicated to the investigating agency involved and to any other interested agency. In addition, Standards for Internal Control in the Federal Government states that management should externally communicate the necessary quality information to achieve objectives. Specifically, management selects appropriate methods of communication, such as a written document—in hard copy or electronic format—or a face-to-face meeting. Management periodically evaluates the entity’s methods of communication so that the organization has the appropriate tools to communicate quality information within and outside of the entity on a timely basis. However, United States Attorneys are not consistently communicating the reasons for declining to prosecute DOD cases involving child victims to the military criminal investigative organizations. Without seeking avenues to improve communication between the military criminal investigative organizations and United States Attorneys for relevant cases involving child victims—to help ensure that investigators are notified when prosecution is declined—investigators may not be informed of the reasons why a case is declined, such as for investigative deficiencies or weaknesses. As a result, DOD may be limited in its ability to improve investigative processes or identify areas where additional investigative training may be needed to improve future incident resolution. Improving this communication through the ongoing DOD and DOJ working group or by other means could also increase the information DOD receives about incident outcomes. Additionally, victims and their families may be better informed of their case disposition and the reasoning behind that disposition. Per the National Children’s Alliance, most military installations in the United States with FAP services are located within 50 miles of a Children’s Advocacy Center (CAC). However, military families may not be able to access CAC services because, according to a 2019 study conducted by the Alliance, only 7 percent of CACs with military installations in their service area reported having a memorandum of understanding, which is needed to authorize services associated with a FAP referral. In addition, according to the Alliance’s 2019 study, while 66 percent of service FAP offices reported having a relationship with their local CAC, 47 percent of those offices reported that contact with the local CAC was infrequent. As shown in figure 5, there are CACs in each state. CACs have considerable experience working with abused children. Specifically, according to the National Children’s Alliance, in 2018 CACs collectively served over 367,000 children, conducted over 260,000 forensic interviews, and completed over 91,000 medical exams and treatments. Further, CACs provide a child-friendly environment to conduct these interviews and exams, which are then reviewed by a multidisciplinary team that includes medical, law enforcement, mental health, and legal personnel, victim advocates, and state child welfare agencies. The purpose of the multidisciplinary team is to determine how to best support the child, such as through therapy, courtroom preparation, and victim advocacy. With regard to child forensic interviews, CACs work to minimize retraumatization of a child by only conducting one comprehensive interview of the child that is typically recorded and involves a team viewing the interview from a separate room. The recorded interview can then be shared with other interested parties with a need to know to include doctors, police, lawyers, therapists, investigators, and judges. This prevents the child from having to talk about the traumatic experience repeatedly in environments where they may be uncomfortable, such as in a police station where they may think they are in trouble. Officials from the Naval Criminal Investigative Service stated that they prefer to use CACs for child forensic interviews when available and where agreements are in place. Both the Army and the Air Force’s military criminal investigative organizations stated that, depending on the circumstances of the case, they may make use of CACs when, for example, agents qualified in child forensic interviews are unavailable. At one U.S. installation we visited, military criminal investigators told us that due to personnel transfers they sometimes do not have investigators available who are qualified to conduct these interviews. Other military criminal investigators with whom we spoke noted that the lack of continuous training for military child forensic interviewers is challenging because regular practice is needed to develop and maintain the skillset. One investigator stated that even though they had not conducted a child forensic interview in 4 years, they were still technically qualified to conduct these interviews. Despite their ability to conduct the interviews, we spoke to military criminal investigators who preferred to rely on child forensic interviewers from the CACs who had more expertise because of the volume of interviews that they conduct. In September 2012, we found that agencies that articulate their agreements in formal documents can strengthen their commitment to working collaboratively. However, according to installation and CAC officials, four of the U.S. installations that we visited either did not have a formal agreement in place with the local CAC or noted that maintaining the agreement was challenging because of the limits that military turnover puts on their ability to build such partnerships. DOD has assigned the responsibility to establish formal agreements with counterparts in the community, such as CACs, within the Family Advocacy Committee at each individual installation. However, given that only 7 percent of CACs with a military installation in their area reported having such an agreement in place according to the National Children’s Alliance’s 2019 study, developing installation-level agreements with CACs has had limited success. In 2015, the Federal Bureau of Investigation established a nationwide memorandum of understanding with the National Children’s Alliance to use CACs to conduct forensic interviews. DOD FAP officials stated that a similar agreement between the military services and the National Children’s Alliance would benefit military families. In August 2019, the National Children’s Alliance recommended the development of a national memorandum of understanding between the National Children’s Alliance, service FAPs, and military criminal investigative organizations within each service. Similarly, in August 2019, a report by the Defense Health Board recommended the development of memorandums of agreement with external entities, such as the National Children’s Alliance and state child welfare agencies. DOD FAP and National Children’s Alliance officials noted that discussions about establishing these types of agreements are not new and believed that agreements would be most effective between the National Children’s Alliance and each respective military service and military criminal investigative organization versus at the installation level. As of February 2019, officials from three of the services indicated that while discussions have been underway, none of these military services have an established agreement, though the status of their efforts varies. For example, as of September 2019, Air Force officials described the effort as being in its infancy, with no established timeframes to achieve goals. Marine Corps FAP officials stated that they were exploring the feasibility of establishing an agreement with the National Children’s Alliance, and Navy FAP officials stated that they were developing a draft agreement for services and support to families impacted by problematic sexual behavior in children and youth. However, given the need for services associated with any type of abuse, such an agreement should not be restricted to problematic sexual behavior. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 stated that the Secretaries of the military departments shall permit, facilitate, and encourage multidisciplinary teams at the installations to collaborate with appropriate civilian agencies for services to support child abuse victims. A national memorandum of understanding could help to break down some of the currently cited barriers to collaboration between CACs and the military, and facilitate such a multidisciplinary approach to addressing incidents of child abuse. For example, DOJ has provided funding for CAC-military partnership pilot projects, which are aimed at improving coordination between CACs and the military to address reported incidents of child abuse. Information from current CAC-military partnership pilot projects indicates that a common barrier to coordination of services is continuity in staffing and leadership for their military counterparts. A base commander’s assignment at a post is time limited, as are some military investigative personnel. These frequent changes in staffing and leadership can result in changes in leadership styles, priorities, and methods of operation and can require a perpetual cycle of building relationships and revising protocols with new counterparts. Without a memorandum of understanding in place between each military service and the National Children’s Alliance, the coordination between the military services and the CACs will continue to be ad hoc and dependent on the relationships of individuals at each installation. Further, without such agreements, the military services may not be fully aware of CAC services and thus may not effectively leverage their facilities or personnel to help address incidents of child abuse involving military dependents. While DOD has taken steps to address recent incidents of child-on-child sexual abuse reported by the media—by establishing policies and beginning to develop a centralized database for problematic sexual behavior in children and youth—the department faces broader challenges, related to visibility, process, and collaboration in addressing the abuse of children. For example, DOD’s visibility over incident outcomes and the extent to which children have been abused—by an adult or another child—is limited by standalone databases, information- sharing challenges, and personnel discretion at the installation level. As DOD develops a centralized database on problematic sexual behavior, it could address some of these challenges by expanding the scope of the database to include any abuse of a child, regardless of offender and type of abuse, and making key decisions related to its development. Further, additional guidance and processes are needed to help reduce information-sharing challenges and installation-level discretion in the tracking and reporting of these incidents. Until DOD resolves these challenges, it will continue to have limited visibility over the extent to which children have been affected by abuse on military installations or as military dependents. Additionally, the department faces gaps in its existing processes for responding to and resolving incidents of child abuse that should be addressed as it continues to develop processes related to problematic sexual behavior in children and youth. For example, given concerns expressed by medical personnel across the military services, DOD should expand the voting membership of the IDC to include medical personnel to ensure that stakeholders with pertinent knowledge and expertise are included. It is critical that IDC determinations are made with all of the relevant information available. Moreover, qualified medical personnel play an essential role in responding to children who have been abused, such as by conducting sexual assault exams. However, according to DOD officials, there are only 11 certified pediatric sexual assault examiners across the department. Without processes to ensure that children who are sexually abused overseas have timely access to a qualified examiner, child victims of sexual abuse may not receive exams in time for the evidence to be collected for use in prosecution and may experience additional stress and trauma. Until DOD addresses these process-related challenges, among others, child victims and their families may not receive the assistance, care, and services that they need. Finally, while DOD has successfully collaborated with a number of states to increase information sharing and with DOJ to address child-on-child sexual offenses occurring on military installations, there are opportunities for DOD to improve its collaboration with external partners to the benefit of military families. For example, there are opportunities to improve communication between the military criminal investigative organizations and United States Attorneys to ensure that DOD is aware of declinations of cases involving the abuse of children and why they were declined. Such avenues could, among other things, help identify needed changes to investigative processes or training. Further, there are opportunities to facilitate awareness and increase the military services’ use of CAC resources, such as through the establishment of a national agreement between the National Children’s Alliance and each military service. Ultimately, improving interagency collaboration could enhance DOD’s visibility over these incidents and increase the resources available to both the department and victims of abuse. We are making a total of 23 recommendations, including 11 to the Secretary of Defense, three to the Secretary of the Army, six to the Secretary of the Navy, and three to the Secretary of the Air Force. The Secretary of Defense, in collaboration with the Secretaries of the military departments, should expand the scope of the department’s centralized database on problematic sexual behavior in children and youth, which is under development, to also track information on all incidents involving the abuse of a child (physical, sexual, emotional, and neglect) reported to the Family Advocacy Program or investigated by a military law enforcement organization, regardless of whether the offender was another child, an adult, or someone in a noncaregiving role at the time of the incident. (Recommendation 1) The Secretary of Defense, in collaboration with the Secretaries of the military departments, should, as part of the ongoing development of the centralized database, identify and define the elements to be tracked by each responsible organization, such as the Family Advocacy Program and military law enforcement. (Recommendation 2) The Secretary of Defense, in collaboration with the Secretaries of the military departments, should develop a plan for how it will use the data it will collect in the centralized database to help ensure data-driven decision-making is used to inform program efforts. (Recommendation 3) The Secretary of Defense, in collaboration with the Secretaries of the military departments, should establish a reliable schedule for the development and implementation of the centralized database on problematic sexual behavior in children and youth that includes key activities, the timeframes and resources needed to execute them, and GAO-identified practices for developing and maintaining a reliable schedule. (Recommendation 4) The Secretary of Defense, in collaboration with the Secretaries of the military departments, should direct the service Family Advocacy Programs and military law enforcement organizations to document in their respective databases the date that they notified the other entity of a reported incident of child abuse. (Recommendation 5) The Secretary of Defense, in collaboration with the Secretaries of the military departments, should issue guidance that describes the process through which the service Family Advocacy Programs are to receive and incorporate information into their central registries regarding child abuse allegations and determinations involving their servicemembers and dependents that were recorded by another service’s installation Family Advocacy Program. Such guidance should include a mechanism to monitor that the process is occurring consistently. (Recommendation 6) The Secretary of the Army should develop a process to monitor how reported incidents of child abuse are screened at installations to help ensure that all reported child abuse incidents that should be presented to an Incident Determination Committee are consistently presented and therefore tracked. (Recommendation 7) The Secretary of the Navy should develop a process to monitor how reported incidents of child abuse are screened at installations to help ensure that all reported child abuse incidents that should be presented to an Incident Determination Committee are consistently presented and therefore tracked. (Recommendation 8) The Secretary of the Navy should ensure that the Commandant of the Marine Corps develops a process to monitor how reported incidents of child abuse are screened at installations to help ensure that all reported child abuse incidents that should be presented to an Incident Determination Committee are consistently presented and therefore tracked. (Recommendation 9) The Secretary of the Air Force should develop a process to monitor how reported incidents of child abuse are screened at installations to help ensure that all reported child abuse incidents that should be presented to an Incident Determination Committee are consistently presented and therefore tracked. (Recommendation 10) The Secretary of Defense should ensure that the Under Secretary of Defense for Personnel and Readiness, in coordination with the Director of the Department of Defense Education Activity, clarifies Department of Defense Education Activity guidance to define what types of incidents must be reported as “serious incidents” to help ensure that all serious incidents of which Department of Defense Education Activity leadership needs to be informed are accurately and consistently reported by school administrators. (Recommendation 11) The Secretary of Defense, in collaboration with the Secretaries of the military departments, should expand the voting membership of the Incident Determination Committee to include medical personnel with the requisite knowledge and experience. (Recommendation 12) The Secretary of the Army should establish efforts to comprehensively inform victims’ families about how reported incidents of child abuse will be addressed following the report, such as a comprehensive guide that explains the process the Family Advocacy Program and military law enforcement organizations will follow, and available victim services. (Recommendation 13) The Secretary of the Navy should establish efforts to comprehensively inform victims’ families about how reported incidents of child abuse will be addressed following the report, such as a comprehensive guide that explains the process the Family Advocacy Program and military law enforcement organizations will follow, and available victim services. (Recommendation 14) The Secretary of the Navy should ensure that the Commandant of the Marine Corps establishes efforts to comprehensively inform victims’ families about how reported incidents of child abuse will be addressed following the report, such as a comprehensive guide that explains the process the Family Advocacy Program and military law enforcement organizations will follow, and available victim services. (Recommendation 15) The Secretary of the Air Force should establish efforts to comprehensively inform victims’ families about how reported incidents of child abuse will be addressed following the report, such as a comprehensive guide that explains the process the Family Advocacy Program and military law enforcement organizations will follow, and available victim services. (Recommendation 16) The Secretary of Defense, in collaboration with the Secretaries of the military departments, should clarify, in guidance, the circumstances under which commanders may exercise their authority to remove a child from a potentially unsafe home on an overseas installation. (Recommendation 17) The Secretary of Defense should ensure that the Under Secretary of Defense for Personnel and Readiness, in coordination with the Director of the Defense Health Agency, establishes processes that help ensure children who are sexually abused overseas have timely access to a certified pediatric sexual assault forensic examiner to conduct the examination. Initiatives could include certifying pediatricians or adult sexual assault forensic examiners as pediatric examiners during mandatory training or establishing shared regional assets. (Recommendation 18) The Secretary of Defense, in collaboration with the Deputy Attorney General, should seek avenues to improve communication between the military criminal investigative organizations and United States Attorneys for relevant cases involving child victims to help ensure that investigators are notified when prosecution is declined, including the reasons for the declination when appropriate, such as details about any investigative deficiencies. (Recommendation 19) The Secretary of the Army should seek to develop a memorandum of understanding with the National Children’s Alliance that makes children’s advocacy center services available to all Army installations and thereby increase awareness of those services across the department. (Recommendation 20) The Secretary of the Navy should continue to develop a memorandum of understanding with the National Children’s Alliance that makes children’s advocacy center services available to all Navy installations and thereby increase awareness of those services across the department. (Recommendation 21) The Secretary of the Navy should ensure that the Commandant of the Marine Corps continues to develop a memorandum of understanding with the National Children’s Alliance that makes children’s advocacy center services available to all Marine Corps installations and thereby increase awareness of those services across the service. (Recommendation 22) The Secretary of the Air Force should seek to develop a memorandum of understanding with the National Children’s Alliance that makes children’s advocacy center services available to all Air Force installations and thereby increase awareness of those services across the department. (Recommendation 23) We provided a draft of this report to DOD for review and comment. In its written comments, DOD concurred with 16 recommendations, partially concurred with six recommendations, and did not concur with one recommendation. DOD also provided technical comments (referred to as enclosure 1 in its written comments), which we incorporated as appropriate. DOD’s written comments are summarized below and reprinted in appendix VI. For the 16 recommendations with which DOD concurred, DOD’s written comments discuss ongoing and planned efforts to implement our recommendations, and in some cases provide target completion dates. DOD did not concur with our first recommendation to expand the scope of its centralized database on problematic sexual behavior in children and youth to track information on all incidents involving the abuse of a child reported to the FAP or investigated by a military law enforcement organization. In its written comments, DOD stated concerns related to privacy and protecting information collected and shared on the alleged conduct of juveniles. DOD also cited a statutory requirement to not disclose directly or indirectly certain juvenile records during the course of juvenile delinquency proceedings and stated that it is the department’s position that it is imperative to protect sensitive juvenile data with any database. We agree that protecting sensitive juvenile data is imperative and acknowledge in the report that privacy and data-safeguarding precautions—such as role-based permissions and other physical, technical, and administrative controls—will need to be taken, as they were in the development of the Defense Sexual Assault Incident Database. In addition, as discussed in the report, the department already maintains databases that include information about both adults and children, such as the service FAPs’ central registries and the databases of the various military criminal investigative organizations, which contain data on both adult and juvenile offenders and victims. DOD does not assert that it would be impossible to establish role-based permissions and the sorts of physical, technical, and administrative controls that would protect the privacy rights of individuals whose information appeared in a central database like the one we recommend. Moreover, the existence of other DOD databases that incorporate such measures supports the notion that it is possible to develop such a database in this situation. Doing so would provide the information needed to track the extent to which children have been affected by abuse and problematic sexual behavior, while safeguarding the personal information of minors. DOD’s written comments also stated that the report conflates three separate and distinct constructs of behavior: juvenile misconduct, problematic sexual behavior in children and youth, and child abuse and neglect committed by adults. As described in our scope and methodology, the scope of our review included child abuse inflicted by both adults and children, which, according to DOD definitions, includes the three categories of behavior noted above. As stated in our report, information is tracked in multiple standalone databases, due, in part, to who inflicted the abuse; as a result, it is difficult to know the extent to which children have been affected by abuse on military installations or as military dependents. In addition, while the response process differs between incidents of adult- inflicted child abuse and incidents of problematic sexual behavior, DOD officials acknowledged that the organizations involved in the response process and the primary data sources are the same. As we also noted, officials stated that a centralized database for all child abuse incidents, tracking the FAP’s determination about whether an incident met DOD’s criteria for abuse, the investigation, and resolution, would be beneficial in determining what happened in a particular case. These officials further stated that such a database would benefit commanders’ oversight of cases for which they are responsible. In addition, without a centralized database that tracks all incidents of abuse involving children, DOD and Congress do not know the extent to which children have been affected by abuse on military installations or as military dependents, or how such incidents have been responded to and resolved. This makes it difficult to identify and address trends that could lead to further prevention efforts. As such, we continue to believe that this recommendation is valid and should be implemented. DOD partially concurred with recommendation 5 to direct the service FAPs and military law enforcement organizations to document in their respective databases the date they notified the other entity of a reported incident of child abuse. In its written comments, DOD stated that it will analyze the efficiency, cost, and feasibility of recording the notification date to law enforcement in FAP databases and that it plans to incorporate a notification field as part of new data standards for DOD’s criminal justice agencies. Similarly, DOD also partially concurred with recommendation 6 to issue guidance that describes the process through which the service FAPs receive and incorporate information into their central registries regarding child abuse allegations and determinations involving their servicemembers and dependents that were recorded by another service’s installation FAP, and that the guidance include a mechanism to monitor that the process is occurring consistently. DOD stated that it will review FAP data reporting policy to explore the potential to reference this process in the scheduled reissuance of DOD policy in 2023. DOD further stated that such information sharing is limited to reported incidents of child abuse that were determined to have met DOD’s criteria for abuse rather than all abuse allegations. We continue to believe that issuing guidance that extends to both allegations and determinations would provide better assurance that the services have complete and up-to-date information about their personnel and their dependents, and increase their visibility over incidents of child abuse. DOD partially concurred with recommendation 12 to expand the voting membership of the IDC to include medical personnel with the requisite knowledge and experience. In its written comments, DOD agreed that the inclusion and consideration of medical information in the determination process is important, and stated that the current process includes medical personnel as nonvoting members. DOD also stated that it will engage the researchers who developed the IDC algorithm and process, as well as other stakeholders—including the Defense Health Agency and the military services—for collaborative input and guidance for a forthcoming revision of the relevant DOD Manual. However, as discussed in the report, medical personnel we spoke with at installations stated that they are not always included in the process, and if they are, their medical expertise is not always included as part of the final determination, contrary to best practices for substantiating child abuse allegations. Further, if medical personnel are not voting members, other clinical duties may take precedence. Therefore, we continue to believe that this recommendation is valid. For recommendations 13, 14, and 16, the Army, the Navy, and the Air Force concurred that they should establish efforts to comprehensively inform victims’ families about how reported incidents of child abuse will be addressed following the report, such as a comprehensive guide that explains the process and available victim services. However, the Marine Corps partially concurred with the related recommendation 15, stating that it is out of scope for the FAP to explain the processes that law enforcement organizations will follow. However, our recommendations state only that the FAP and military law enforcement processes should be effectively communicated to the families, not that the FAP would have to determine or communicate the law enforcement processes to affected families. Further, DOD’s written comments stated that Marine Corps Order 1754.11 addresses the recommendation because it directs victim advocates to be assigned to the non-offending parent of a victim of child abuse who requests services. However, parents we spoke with indicated that they were not aware of all available services and resources offered by the military, and that a comprehensive guide outlining the process would have helped them understand what was going to happen. For these reasons, we continue to believe that the recommendation is valid. For recommendations 20 through 23, DOD’s written comments stated that the services concurred with the overall recommendation to seek to establish memorandums of understanding with the National Children’s Alliance that make children’s advocacy center services available to all military installations and thereby increase awareness of those services across the department. While the Marine Corps and the Air Force concurred (recommendations 22 and 23), DOD noted that individual service differences in organizational structure and process are reflected in the nuances of their responses. For example, the Army partially concurred with recommendation 20. DOD stated that the Army is working with the National Children’s Alliance to develop a broad memorandum of understanding to support partnerships between military installations and local children’s advocacy centers. The agreement is intended to assist in providing support services, education, and prevention services to military families and investigations of child abuse and problematic sexual behavior with a goal to finalize the agreement in fiscal year 2021. The Army also plans to pursue local agreements with children’s advocacy centers who may not participate in the broader service-wide agreement. We believe that such local agreements, in addition to a broader memorandum of understanding with the National Children’s Alliance, would be beneficial and that these actions would meet the intent of our recommendation. Likewise, the Navy partially concurred with recommendation 21. DOD’s written comments stated that the Navy seeks to develop memorandums of understanding both broadly with the National Children’s Alliance, as well as with local children’s advocacy centers who may not be accredited through the National Children’s Alliance. Similar to the Army, we believe that such local agreements would be beneficial in addition to a broader agreement with the National Children’s Alliance and, that together, they would meet the intent of our recommendation. DOD’s comments also stated that the Navy’s planned agreement with the National Children’s Alliance will outline services and support to families affected by problematic sexual behavior. However, as previously discussed in this report, we believe that such an agreement should not be restricted to problematic sexual behavior given the need for services associated with any type of abuse. As such, we continue to believe that the recommendation is valid. We are sending copies of this report to the appropriate congressional committees, the Attorney General of the United States, the Secretary of Defense, the Secretary of the Army, the Secretary of the Navy, the Secretary of the Air Force, and the Commandant of the Marine Corps. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or members of your staff have any questions regarding this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in appendix VII. Department of Defense (DOD) policy defines child abuse as the physical, sexual, or emotional abuse, or neglect of a child by a parent, guardian, foster parent, or caregiver. Our review included any abuse of a child (emotional, physical, or sexual abuse, or neglect) by an adult regardless of their caregiving status and child-on-child abuse—any physical or sexual abuse of a child (under the age of 18) by another child. To assess the extent to which DOD has visibility over reported incidents of child abuse, including child-on-child abuse, occurring on military installations or involving military dependents, we analyzed data from the three primary organizations that DOD officials identified as having responsibility for tracking these incidents: (1) the military services’ Family Advocacy Programs (FAP), (2) the military criminal investigative organizations, and (3) the DOD Education Activity (DODEA). First, we analyzed FAP data from the Army, the Navy, the Marine Corps, and the Air Force on all reported incidents of child abuse for fiscal years 2014 through 2018. We selected this timeframe to evaluate trends over 5 years, and fiscal year 2018 was the most recent year for which complete data were available at the time of our review. Specifically, we analyzed the data to determine the number of reported incidents of child abuse by service and the percent of those that met DOD’s criteria for child abuse. Because the services are required to track more detailed information about incidents of child abuse that they determined met DOD’s criteria for child abuse, we conducted a more detailed analysis of these incidents to describe their characteristics, such as the status of the offender, the relationship between the offender and the victim, the age of the victim, and the type of abuse (emotional, physical, sexual, or neglect). To assess the reliability of the service FAPs’ child abuse data, we reviewed related documentation; assessed the data for errors, omissions, and inconsistencies; and interviewed officials. We determined that the data were sufficiently reliable to describe trends in reported incidents of child abuse across the services and characteristics of such incidents from fiscal years 2014 through 2018. Second, we analyzed data from the military criminal investigative organizations—the Army Criminal Investigation Command, the Naval Criminal Investigative Service, and the Air Force Office of Special Investigations—for the same time period for all investigations with a child victim. We also analyzed child victim investigation data from the U.S. Marine Corps Criminal Investigation Division, a federal law enforcement agency that also investigates some offenses involving child victims. Specifically, we analyzed the data to identify trends in the number of investigations over the past 5 fiscal years. We also analyzed the investigation data to identify key characteristics of the investigations, such as the status of the offender, relationship between the victim and offender, and primary investigative agency. To assess the reliability of the military criminal investigative organizations’ child victim investigation data, as well as that of the U.S. Marine Corps Criminal Investigation Division, we assessed the data for errors, omissions, and inconsistencies, and interviewed officials. We determined that the data were sufficiently reliable to describe trends in child victim investigations across the services and the characteristics of such investigations from fiscal years 2014 through 2018. Third, we analyzed three sources of DODEA data: (1) child abuse reports from school years 2014-2015 through 2017-2018, (2) serious incident reports from school years 2013-2014 through 2017-2018, and (3) student misconduct records from school years 2016-2017 through 2017-2018. We selected these timeframes to evaluate serious incident report trends over 5 years and to align with the FAP and investigation data; school year 2017-2018 was the most recent year for which complete data were available at the time of our review. All DODEA records were redacted by DODEA personnel to ensure the privacy of students and DODEA personnel. DODEA child abuse reports track information about incidents of suspected or alleged child abuse or neglect. We analyzed DODEA’s child abuse reports over 4 school years to identify trends in the number and type of child abuse reports as well as to describe characteristics of the reports. Specifically, we analyzed characteristics such as the relationship between the victim and the offender, the location of the reported abuse, and notifications by DODEA to external organizations, such as the FAP. To assess the reliability of DODEA’s child abuse reports, we reviewed related documentation; assessed the data for errors, omissions, and inconsistencies; and interviewed officials. We determined that the data were sufficiently reliable to describe trends in and characteristics of child abuse reports from school years 2014-2015 through 2017-2018. DODEA serious incident reports track information about alleged or suspected serious incidents resulting in consequences greater than those normally addressed through routine administrative actions. We analyzed DODEA’s serious incident reports relating to child-on-child abuse— involving a violation of law or a sexual event—over the past 5 school years to identify trends in the number and type of serious incident reports as well as to describe characteristics of the reports. Specifically, we analyzed the type of serious incident (assault/battery, child pornography, nonconsensual sexual contact, etc.), whether police were notified, whether the police investigated, and the type of school filing the report. To assess the reliability of DODEA’s serious incident reports, we reviewed related documentation; assessed the data for errors, omissions, and inconsistencies; and interviewed officials. We determined that the data were sufficiently reliable to describe trends in and characteristics of serious incident reports from school years 2013-2014 through 2017-2018, and to compare serious incident reports to DODEA student misconduct records from school years 2016-2017 through 2017-2018. DODEA’s student misconduct records are separate from child abuse reports and serious incident reports—but may also be filed in relation to a serious incident—and track information regarding disciplinary actions and the triggering incident, such as an abusive or indecent act. We requested and received all redacted DODEA student misconduct records over the past 5 school years that involved at least one of 26 incident types that we determined, through conversations with DODEA officials familiar with the records, could relate to a child-on-child serious incident. We received over 26,000 records, some of which related to the same incident, for example, according to DODEA officials, when more than one student was involved. For school years 2016-2017 and 2017-2018, we conducted a content analysis of the student misconduct records, using DODEA’s Serious Incident Reporting Procedures, to determine the number of student misconduct records that school administrators, using that guidance, could have reasonably categorized as a violation of law or sexual event and filed a serious incident report. We selected these 2 school years for the analysis because DODEA’s updated serious incident reporting guidance was issued in August 2016 and was in place for both school years. Because of the large number of DODEA student misconduct records, we conducted our content analysis in two stages. We first conducted an electronic search to identify potentially-relevant records and then conducted a manual review of all potentially-relevant records. One of our data analysts electronically searched the student misconduct record descriptions for key terms that could potentially signify that the incident was of a nature serious enough to warrant the filing of a serious incident report, per DODEA guidance. We selected the search terms using the DODEA guidance (e.g., assault, battery, and rape); additional terms that may signify a medical or police response (e.g., nurse, ambulance, blood, and police) because incidents resulting in an injury may be considered to be serious incidents per the guidance; and terms for common social media outlets (e.g., Facebook and Snapchat) because taking or sharing nude photos of another student without their knowledge is an example of a noncontact sexual act that should result in the filing of a serious incident report. This search resulted in 2,619 student misconduct records—after removing duplicate records—that we then manually reviewed. It is possible that we did not identify some student misconduct records that could have been categorized as serious incidents because we did not include some search terms that would have identified more. Two analysts independently reviewed each of the 2,619 student misconduct records, using the DODEA guidance, and recorded their determination that a record (a) could have been classified as a serious incident report per DODEA’s guidance, (b) was unclear whether it could be classified as a serious incident report, or (c) should not have been classified as a serious incident report per DODEA’s guidance. For records where the two analysts did not initially agree on a determination, they met and discussed the records and reached a final determination. We then compared the number of student misconduct records which we determined school administrators, using the guidance, could have reasonably categorized as a violation of law or sexual event and filed a serious incident report with the number of serious incidents recorded by DODEA for the same time period to determine the extent of DODEA’s visibility into serious incidents. We discussed the student misconduct records, the content analysis, and the comparison to serious incident reports with DODEA officials. Further, we interviewed relevant DOD and service officials at the headquarters level and at a nongeneralizable sample of seven military installations to identify how DOD tracks reported incidents of child abuse from the time of a report to an ultimate adjudication, including how information is communicated within and across the services. We selected at least one installation per service as well as two joint installations, and selected locations based on the number of reported child abuse incidents and the number of investigated child-on-child abuse incidents over the past 5 fiscal years, as well as other factors. Specifically, we selected installations that over the past 5 fiscal years had a high number of reported incidents of child abuse or a high number of child-on-child abuse investigations—or both—in order to maximize the possibility we would interview officials, responders, and care providers who had responded to reported incidents of child abuse. Other selection factors included a mix of types of legislative jurisdiction (such as exclusive and concurrent jurisdiction), at least some installations with DODEA schools, a high number of DODEA serious incident reports, and a mix of geographic locations in the United States and overseas. Because we did not select locations using a statistically representative sampling method, the comments provided during our interviews with installation officials are nongeneralizable and therefore cannot be projected across DOD or a service, or any other installations. While the information obtained was not generalizable, it provided perspectives from installation officials that have assisted with the response to reported incidents of child abuse. We compared information from our data analyses and interviews to DOD guidance; GAO-identified practices for developing and maintaining a reliable schedule; GAO-identified leading practices for results-oriented management; and Standards for Internal Control in the Federal Government related to quality information, control activities, and monitoring activities. To assess the extent to which DOD has developed and implemented policies and procedures to respond to and resolve incidents of child abuse, including child-on-child abuse, occurring on military installations or involving military dependents, we reviewed relevant DOD and service policies, guidance, reports, and memoranda on child abuse, juvenile misconduct, and problematic sexual behavior in children and youth. We also conducted work at a nongeneralizable sample of seven military installations in the United States and overseas. At the installations, we interviewed FAP personnel, medical and mental health personnel, military law enforcement officials, legal personnel, Special Assistant United States Attorneys, military criminal investigators, chaplains, child development center personnel, school liaison officers, military family life counselors, DODEA personnel, and commanders about how they prevent, track, respond to, and resolve these incidents. To obtain the perspectives of parents and guardians of children who have been affected by abuse on military installations or while they were military dependents (either by an adult or another child), we interviewed 20 parents and guardians by phone that volunteered to speak with us about their perspectives on available resources and assistance, case communication, and the investigative and adjudicative processes. To develop the interview protocol for parents and guardians, we reviewed DOD and service policies, interviewed DOD officials, and reviewed our prior work related to sexual assault in the military. We also consulted with a GAO mental health professional on the appropriateness of the instrument as well as guidance on resources to offer participants if relevant. A survey specialist helped to design the interview protocol, another survey specialist reviewed it for methodological issues, and an attorney reviewed it for legal terminology and any other issues. Prior to interviewing parents and guardians, we pretested the interview protocol with three analysts who had children and had experience as a military servicemember or military dependent. We used the pretests to determine whether: (1) the questions were clear, (2) the terms used were precise, (3) respondents were able to provide information that we were seeking, and (4) the questions were unbiased. We made changes to the content and format of the interview protocol based on the results of our pretesting. Further, each team member was trained on the interview protocol to assure its consistent implementation across interviewers and participants. Due to the sensitivity of the information being discussed, we took several steps to help ensure a confidential and safe environment during the phone interviews. All information provided was handled confidentially— callers’ names and contact information were not recorded in our notes and we did not audio record the interviews. We conducted interviews from June to September 2019. We took interview notes on paper and later entered them into a Microsoft Word form. Data entry was verified by the same analyst. The data were electronically extracted from the Word forms into a comma-delimited file that was then imported into Excel for analysis. We summarized the answers to questions about the characteristics of the incidents discussed, such as whether the offender was a child or an adult, the location of the incident, the military dependent status of the victim, and the servicemember status of the offender. Quantitative data analyses were conducted by one analyst and verified by a second analyst. We also conducted a content analysis of the narrative information to identify common themes related to items such as parents’ awareness of available victim services, the clarity of the response process, and areas for improvement. Two analysts reviewed the data collected from the interviews and agreed on the themes into which callers’ comments would be categorized. Standardized coding instructions were developed and tested. One analyst reviewed all the callers’ narrative comments and indicated in the spreadsheet if a theme was present or absent. A different analyst reviewed the first analyst’s coding to see if they reached the same determination. For records where the two analysts did not initially agree on a determination, they met and discussed the records and reached a final determination. The codes were then counted to assess how many callers mentioned a given theme. Because we did not select participants using a statistically representative sampling method, the perspectives obtained are nongeneralizable and therefore cannot be projected across DOD, a military service, or installation. While the information obtained was not generalizable, it provided perspectives from parents and guardians who were willing to discuss their experiences with the reporting, response, and resolution processes. Additionally, we observed each service’s Incident Determination Committee (IDC) process—through which installations determine whether an incident meets DOD’s criteria for child abuse—at a total of four installations. We also attended a symposium hosted by the National Center on Sexual Exploitation on problematic sexual behavior in children and youth. We compared the information from the selected installations, observations, and interviews to GAO-developed practices to enhance and sustain collaboration in interagency groups, Department of Justice (DOJ) best practices for sexual assault forensic examination kits, and Standards for Internal Control in the Federal Government related to quality information. To assess the extent to which DOD collaborates with other governmental and nongovernmental organizations to address incidents of child abuse, including child-on-child abuse, occurring on military installations or involving military dependents, we reviewed written agreements in place with civilian organizations at the nongeneralizable sample of U.S. installations in our review, such as agreements with local civilian law enforcement and state and local child welfare agencies about how incidents of child abuse on the installation are to be addressed. We also interviewed relevant officials from civilian organizations near the five U.S. installations in our review, such as state child welfare agencies, law enforcement organizations, prosecuting attorneys offices, and children’s advocacy centers (CAC) to determine the extent of their collaboration with the military and any related challenges. In addition, we interviewed a senior official from the Defense State Liaison Office regarding their outreach to states to increase information sharing with state child welfare agencies. Further, we interviewed DOJ officials regarding the prosecution of juvenile crimes committed on overseas installations and on some U.S. installations and its coordination with DOD to address these incidents. Finally, we contacted officials from the National Children’s Alliance, which accredits CACs, about its efforts with DOD to improve collaboration between the military and CACs. We compared the agreements and information obtained through interviews with DOJ Principles of Federal Prosecution, GAO-developed key considerations for interagency collaborative mechanisms, and Standards for Internal Control in the Federal Government related to quality information. Tables 2 and 3 present the DOD and non-DOD organizations we visited or contacted during our review to address our three objectives. We conducted this performance audit from January 2019 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. To obtain the perspectives of parents and guardians of children affected by abuse on military installations or while they were military dependents, we interviewed parents and guardians who volunteered to speak with us about their perspectives on available resources and assistance, case communication, and the investigative and adjudicative processes. We announced our interest in anonymously interviewing parents and guardians of children affected by abuse on military installations or while they were military dependents and provided a toll-free telephone number and email address for volunteers to contact us. Department of Defense Military Community and Family Policy officials, who are responsible for Military OneSource—a 24/7 connection for military families to information, answers and support—agreed to post our announcement on the Military OneSource website. We also posted our announcement on our agency social media platforms and disseminated it through officials at some of the installations we visited. It was also featured in an article by a military- focused news outlet. Further details about our methodology for these interviews can be found in appendix I. The interview questionnaire follows. 1. In what state are you currently located, or if you’re overseas, in what 2. Are you currently, or were you previously, associated with a particular military service, including as a military dependent?  Which service? ______________ 3. Are you calling about abuse that your child experienced, or a child for whom you are a guardian experienced, or are you calling about someone else’s child’s experience? My child/a child for whom I am a guardian Someone else’s experience  Go to Question 4  End discussion 4. In what year did the abuse occur? (If multiple years, write the range.) Year provided 5. Was the abuse reported to any military or civilian government office?  Continue to “a” a. In what year was the abuse first reported? 6. Did the abuse occur on the property of a military installation, including in military housing?  What installation was it? _______________ a. Was the child who was affected by abuse a military dependent at the time of the incident?  If “no” here and Q6 above is no, don’t know, or prefer not to answer, End discussion Don’t Know know, or prefer not to answer, End discussion  If “don’t know” here and Q6 above is no, don’t  If “prefer not to answer” here and Q6 above is no, don’t know, or prefer not to answer, 7. Did the abuse occur in a child care facility, a home, a DOD school, or somewhere else? Interviewer: Check all they mention a. (Skip if the abuse occurred on an installation and the installation was provided in Q6) In what state or country did the abuse occur? 8. Was the individual who perpetrated the abuse a servicemember at the time of the incident?  Continue to “a”  Continue to “a”  Continue to “a” a. Was the individual who perpetrated the abuse another child under the age of 18 at the time of the incident? Prefer not to answer 9. Was the individual who perpetrated the abuse a parent, guardian, foster parent, or someone in a caregiving role at the time of the incident, including an older sibling babysitting, or a teacher, etc.? 10. What organization was the abuse first reported to? For example, was it first reported to the Family Advocacy Program, military law enforcement, military criminal investigators, civilian law enforcement, the chain of command, Child Protective Services, or some other organization? Family Advocacy Program (FAP)  Continue to “a” Military law enforcement (Security Forces, Military Police, Provost’s Office, Master-at-Arms, etc.)  Continue to “a” Military criminal investigators (CID, OSI, NCIS, Marine Corps CID)  Continue to “a”  Continue to “a” Chain of Command (to include Commander, Unit, Wing, etc.)  Continue to “a” Child Protective Services (CPS)  Skip to “b” Chaplain military or civilian government organization?  Did you or the chaplain report the abuse to any other  What office?  What office? _______ then continue to “a” Was the abuse ever reported to a civilian Child Protective Services agency?  Continue to “b”  Continue to “b”  Continue to “b”  Continue to “b” Was the abuse ever reported to the Family Advocacy Program?  Continue to “c”  Continue to “c”  Continue to “c” c. Had you ever heard of the Family Advocacy Program before this interview? Prefer not to answer 11. Are you aware that the Family Advocacy Program is responsible for assessing and providing support services to military families affected by child abuse? 12. (If child was abused by a parent/guardian/foster parent/someone in a caregiving role—If Q9 = Yes) Were you notified by the Family Advocacy Program about whether the incident was considered to be child abuse, according to DOD criteria and policy?  Continue to “a”  Skip to “Response to Abuse” section  Skip to “Response to Abuse” section  Skip to “Response to Abuse” section a. Was the Family Advocacy Program’s process for assessing the report of abuse and determining whether it met criteria to be considered child abuse clear to you? b. Is there anything that the Family Advocacy Program could do to clarify the process or make the process clearer? III. Response to Abuse 13. Did the child or family receive any services from the military related to the abuse, for example, psychological or legal counseling or medical care?  Continue to “a” a. What services did the child or family receive from the military? b. What, if any, services provided by the military were particularly helpful? c. What, if any, services were provided by the military but did not meet your family’s needs? i. Why didn’t those services meet your family’s needs? _________________________________________ d. What, if anything, could be improved about the services you received from the military, such as the services themselves, or the ease of access or timeliness of the services provided? 14. Were there services that your child or family were offered by the military, but that you did not receive, either because you did not need them or because of some other factor?  Continue to “a” a. What type of services were offered but not received? b. Why did you not receive these services—for example, was it by choice or was there some factor that prevented you from receiving them? _________________________________________ 15. Did your child or family receive any services from civilian organizations or providers related to the abuse, for example, psychological or legal counseling or medical care?  Continue to “a” a. What services did your child or family receive from civilian organizations or providers? 16. Were there any services—either through the military or a civilian agency—that you think would have been helpful, but were not available?  Continue to “a” a. What services? _________________________________________ IV. Investigation/Resolution of Abuse 17. Was the incident of abuse investigated by any law enforcement organization, including military or civilian law enforcement? For example, was the incident of abuse investigated by the military police, a military investigative organization, civilian state or local law enforcement, the Federal Bureau of Investigation, or some other law enforcement organization?  Continue to “a” Skip to “Miscellaneous Questions” section Skip to “Miscellaneous Questions” section Skip to “Miscellaneous Questions” section a. What law enforcement organization or organizations conducted an investigation? If more than one law enforcement organization conducted an investigation, please tell me all the organizations. Military police (Security Forces, Military Police, Provost’s Office, Marshal-at-Arms, etc) Military investigative organization (CID, OSI, NCIS, Marine Corps CID) 18. (If military conducted an investigation, see response to Q17a) What type of information did you receive from the investigating military organization during the course of the investigation, if any, such as status updates by phone, e-mail, or letter? _________________________________________ a. Did you have a point of contact that you could reach out to at the investigating military organization with any questions or for status updates? 19. After the investigation ended, were you informed about the outcome or informed of any next steps regarding any potential criminal or administrative action against the individual that perpetrated the abuse?  Continue to “a”  Continue to “a” a. Did you have a point of contact that you could reach out to with any questions about the outcome of the investigation or next steps? Prefer not to answer 20. What, if anything, would you recommend that DOD or the military services do to be more responsive to families of children who have been affected by abuse on military installations or as military dependents? 21. What, if anything, would you recommend DOD or the military services do to help prevent child abuse or child-on-child abuse? 22. Is there anything related to child abuse on military installations or of military dependents that we did not discuss but you think we should be aware of? 23. One last question: Was there anyone else present with you during any part of our conversation? Continue to “a” Each military service’s Family Advocacy Program (FAP) has a database—referred to as the “central registry”—where it tracks (1) reports of abuse that did not meet the Department of Defense’s (DOD) criteria for child abuse, about which no identifiable individual information is tracked; and (2) information on reports of abuse that met DOD’s criteria for abuse, which is linked to identifiable servicemembers, their family members, and the alleged offenders. Per DOD guidance, the services are to track 46 data elements on all reported incidents of child abuse that met DOD’s criteria for abuse. The service FAPs only track information in their central registries related to child abuse where the offender was a parent, guardian, foster parent, or someone in a caregiving role. The following describes key characteristics of incidents of child abuse that met DOD’s criteria for abuse as reported to the Army, the Navy, the Marine Corps, and the Air Force FAPs from fiscal years 2014 through 2018. Army FAP. Over the past 5 fiscal years, the Army FAP recorded 32,386 reported incidents of child abuse, of which 50 percent met DOD’s criteria for child abuse. Of the incidents that met DOD’s criteria for abuse, 66 percent involved neglect, 20 percent involved physical abuse, 17 percent involved emotional abuse, and 5 percent involved sexual abuse. The majority of incidents (97 percent) were intrafamilial—meaning that the victim and the offender were from the same family, such as a parent or sibling—and 2 percent of the incidents were extrafamilial or external to the family, such as a babysitter or a childcare provider. Half of the victims and 52 percent of the offenders were male. In addition, a quarter of offenders had prior FAP cases related to child abuse or domestic abuse that met DOD’s criteria for abuse. Figure 6 depicts characteristics of incidents reported to the Army FAP that met DOD’s criteria for child abuse over the past 5 fiscal years. Navy FAP. From fiscal years 2014 through 2018, the Navy FAP recorded 10,744 reported incidents of child abuse, of which 51 percent met DOD’s criteria for child abuse. Of the incidents that met DOD’s criteria for abuse, 59 percent involved neglect, 33 percent involved physical abuse, 14 percent involved emotional abuse, and 6 percent involved sexual abuse. The majority of incidents (96 percent) were intrafamilial and 4 percent of the incidents were extrafamilial. Slightly over half of the victims and offenders were male (52 percent). Additionally, since fiscal year 2017, when the Navy began tracking whether offenders had prior FAP cases related to child abuse or domestic abuse that met DOD’s criteria for abuse, 1 percent of offenders had prior cases. Figure 7 depicts characteristics of incidents reported to the Navy FAP that met DOD’s criteria for child abuse over the past 5 fiscal years. Marine Corps FAP. Over the past 5 fiscal years, the Marine Corps FAP recorded 8,356 reported incidents of child abuse, of which 54 percent met DOD’s criteria for child abuse. Of the incidents that met DOD’s criteria for abuse, 62 percent involved neglect, 20 percent involved emotional abuse, 15 percent involved physical abuse, and 2 percent involved sexual abuse. The majority of incidents (96 percent) were intrafamilial and 4 percent of the incidents were extrafamilial. Slightly over half of the victims and offenders were male (52 percent) and 7 percent of offenders had prior FAP cases related to child abuse or domestic abuse that met DOD’s criteria for abuse. Figure 8 depicts characteristics of incidents reported to the Marine Corps FAP that met DOD’s criteria for child abuse over the past 5 fiscal years. Air Force FAP. From fiscal years 2014 through 2018, the Air Force FAP recorded 17,836 reported incidents of child abuse, of which 41 percent met DOD’s criteria for child abuse. Of the incidents that met DOD’s criteria for abuse, 55 percent involved neglect, 25 percent involved physical abuse, 22 percent involved emotional abuse, and 4 percent involved sexual abuse. The majority of incidents (95 percent) were intrafamilial and 4 percent of the incidents were extrafamilial. Slightly over half of the victims and offenders were male (51 percent and 53 percent, respectively). In addition, 0 percent of offenders had prior FAP cases related to child abuse or domestic abuse that met DOD’s criteria for abuse. Figure 9 depicts characteristics of incidents reported to the Air Force FAP that met DOD’s criteria for child abuse over the past 5 fiscal years. Each military criminal investigative organization—the Army Criminal Investigation Command, the Naval Criminal Investigative Service, and the Air Force Office of Special Investigations—maintains an investigative case management system where it tracks information about the investigation, such as the offense(s), victim(s), and alleged offender(s), among other things. According to military criminal investigative organization officials, they primarily investigate felony level crimes as well as any type of sexual offense. The following are key characteristics of investigations involving child victims investigated by each of the three military criminal investigative organizations from fiscal years 2014 through 2018. Army Criminal Investigation Command. Over the past 5 fiscal years, the Army Criminal Investigation Command conducted or monitored 5,565 investigations involving a child victim. Some of those investigations involved multiple victims, offenders, and offenses. Specifically, those 5,565 investigations included 6,535 victims, 5,965 alleged offenders, and 8,483 offenses. The Army Criminal Investigation Command was the primary investigative organization for almost three-quarters of the investigations (74 percent). For the rest of the investigations, the primary investigative organization was another federal, state, or local civilian law enforcement organization, such as the Federal Bureau of Investigation, which conducted 4 percent of the investigations. Additionally, 42 percent of the investigations involved an intrafamilial relationship—meaning that the victim and the alleged offender were from the same family, such as a parent or sibling—between at least one of the alleged offenders and victims. Figure 10 depicts characteristics of the Army Criminal Investigation Command’s investigations involving a child victim over the past 5 fiscal years. Naval Criminal Investigative Service. From fiscal years 2014 through 2018, the Naval Criminal Investigative Service conducted or monitored 1,513 investigations involving a child victim. Some of those investigations involved multiple victims, offenders, and offenses. Specifically, those 1,513 investigations included 1,731 victims, 1,618 alleged offenders, and 1,812 offenses. The Naval Criminal Investigative Service was the primary investigative organization for about half of the investigations (54 percent). The remainder of the investigations were either joint with another law enforcement organization or the Naval Criminal Investigative Service was only monitoring the investigation. Additionally, 40 percent of the investigations involved an intrafamilial relationship between at least one of the alleged offenders and victims. Figure 11 depicts characteristics of the Naval Criminal Investigative Service’s investigations involving a child victim over the past 5 fiscal years. Air Force Office of Special Investigations. Over the past 5 fiscal years, the Air Force Office of Special Investigations conducted or monitored 1,304 investigations involving a child victim. Some of those investigations involved multiple victims, offenders, and offenses. Specifically those 1,304 investigations included 1,549 victims, 1,384 alleged offenders, and 1,649 offenses—12 percent of investigations involved more than one victim. In addition, 42 percent of investigations involved an intrafamilial relationship between at least one of the alleged offenders and victims. Figure 12 depicts characteristics of the Air Force Office of Special Investigations’ investigations involving a child victim over the past 5 fiscal years. The Department of Defense Education Activity (DODEA) tracks suspected or alleged abuse of students through (1) child abuse reports, and (2) serious incident reports. Child abuse reports. DODEA guidance defines child abuse as the physical injury, sexual maltreatment, emotional maltreatment, deprivation of necessities, or combinations for a child by an individual responsible for the child’s welfare under circumstances indicating that the child’s welfare is harmed or threatened. The term encompasses both acts and omissions on the part of the responsible person. Child abuse reports are to be submitted on any incidents of suspected or alleged child abuse to DODEA headquarters within 24 hours of the occurrence or notification of the incident. From school years 2014-2015 through 2017-2018, DODEA reported 254 suspected or alleged incidents of child abuse. Of DODEA’s 163 schools, 115 reported an incident of child abuse over these 4 school years. Reported child abuse included a range of incidents, such as parents leaving their children unattended, parents physically abusing their children, teachers using physical force on students, and teachers inappropriately touching students. The most common types of abuse were physical abuse (51 percent of reported incidents), multiple types of abuse (11 percent), and sexual abuse (9 percent). The majority of the reported incidents involved the abuse of a child by a parent or guardian (55 percent) or abuse by DODEA personnel (31 percent). Figure 13 depicts characteristics of incidents of child abuse reported by DODEA over 4 school years. Serious incident reports. DODEA guidance defines a serious incident as an event or allegation that impacts school readiness, or the health, safety, and security of DODEA-affiliated personnel, facilities, and property resulting in consequences greater than those normally addressed through routine administrative or preventive maintenance actions. Serious incident reports are normally submitted by the school principal, assistant principal, or designated administrative officer within 2 business days after the event is brought to the attention of the first-line supervisor. DODEA has different categories of serious incidents, such as drug and alcohol events, violation of law events, sexual events, and security incidents. Serious child-on- child abuse incidents are reported as either violation of law events, such as assault and battery or sexual events. From school years 2013-2014 through 2017-2018, DODEA reported 167 serious incidents involving either an alleged violation of law or an alleged sexual event. Only 74 of DODEA’s 163 schools reported such an incident over the past 5 school years. Reported serious incidents included a range of incidents, such as students posting nude photos and videos of other students on social media, inappropriate touching on the school bus, physical assaults, and rape. Of the serious incident reports we received, the most common types were nonconsensual sexual contact (35 percent of reported incidents), assault and battery (25 percent), rape (16 percent), and child pornography (15 percent). The majority of the reported serious incidents involved a single victim (68 percent), but 13 percent of the incidents involved more than one victim and 20 percent did not specify a victim. Figure 14 depicts characteristics of serious incidents involving an alleged violation of law or an alleged sexual event reported by DODEA over the past 5 school years. According to DODEA officials, DODEA implemented a new database for reporting serious incidents in August 2019. These officials noted that one of the goals of the system is to make reporting more straightforward for school administrators and to standardize serious incident reports across schools. DODEA officials anticipate adding child abuse reports to the new database in late calendar year 2019 or early 2020. In addition to the contact named above, Kimberly Mayo (Assistant Director), Molly Callaghan (Analyst in Charge), Vincent M. Buquicchio, Christopher Gezon, Grant Mallie, Joseph Neumeier, Kya Palomaki, Paul Seely, Mike Silver, and Lillian M. Yob made significant contributions to this report. Military Justice: DOD and the Coast Guard Need to Improve Their Capabilities to Assess Racial and Gender Disparities. GAO-19-344. Washington, D.C.: May 30, 2019. Children Affected by Trauma: Selected States Report Various Approaches and Challenges to Supporting Children. GAO-19-388. Washington, D.C.: April 24, 2019. Sexual Violence: Actions Needed to Improve DOD’s Efforts to Address the Continuum of Unwanted Sexual Behaviors. GAO-18-33. Washington, D.C.: December 18, 2017. Child Well-Being: Key Considerations for Policymakers, Including the Need for a Federal Cross-Agency Priority Goal. GAO-18-41SP. Washington, D.C.: November 9, 2017. Military Personnel: DOD Has Processes for Operating and Managing Its Sexual Assault Incident Database. GAO-17-99. Washington, D.C.: January 10, 2017. Sexual Violence Data: Actions Needed to Improve Clarity and Address Differences Across Federal Data Collection Efforts. GAO-16-546. Washington, D.C.: July 19, 2016. Sexual Assault: Actions Needed to Improve DOD’s Prevention Strategy and to Help Ensure It Is Effectively Implemented. GAO-16-61. Washington, D.C.: November 4, 2015. Youth Athletes: Sports Programs’ Guidance, Practices, and Policies to Help Prevent and Respond to Sexual Abuse. GAO-15-418. Washington, D.C.: May 29, 2015. Military Personnel: Actions Needed to Address Sexual Assaults of Male Servicemembers. GAO-15-284. Washington, D.C.: March 19, 2015. Child Welfare: Federal Agencies Can Better Support State Efforts to Prevent and Respond to Sexual Abuse by School Personnel. GAO-14-42. Washington, D.C.: January 27, 2014. Child Maltreatment: Strengthening National Data on Child Fatalities Could Aid in Prevention. GAO-11-599. Washington, D.C.: July 7, 2011. Military Justice: Oversight and Better Collaboration Needed for Sexual Assault Investigations and Adjudications. GAO-11-579. Washington, D.C.: June 22, 2011.", "summary": "With more than 1.2 million school-age military dependents worldwide, per DOD, the department's organizations work to prevent, respond to, and resolve incidents of child abuse. Incidents of child abuse, including child-on-child abuse, can cause a range of emotional and physical trauma for military families, ultimately affecting servicemember performance. GAO was asked to review how DOD addresses incidents of child abuse and child-on-child abuse occurring on a military installation or involving military dependents. This report examines, among other things, the extent to which DOD has (1) visibility over such reported incidents, and (2) developed and implemented policies and procedures to respond to and resolve these incidents. GAO reviewed relevant policies and guidance; interviewed officials at a nongeneralizable sample of seven military installations; analyzed program data; interviewed parents of children affected by abuse; and interviewed DOD, service, and civilian officials, including at children's advocacy centers. The Department of Defense (DOD) has limited visibility over reported incidents of child abuse—physical, sexual, or emotional abuse, or neglect by a caregiver—and child-on-child abuse due to standalone databases, information sharing challenges, and installation discretion. From fiscal years 2014 through 2018, the military services recorded more than 69,000 reported incidents of child abuse (see figure). However, personnel at all seven installations in GAO's review stated that they use discretion to determine which incidents to present to the Incident Determination Committee (IDC)—the installation-based committee responsible for reviewing reports and determining whether they meet DOD's criteria for abuse (an act of abuse and an actual or potential impact, e.g., spanking that left a welt). Per DOD guidance, every reported incident must be presented to the IDC unless there is no possibility that it could meet any of the criteria for abuse. However, personnel described incidents they had screened out that, per DOD guidance, should have been presented to the IDC. Without the services developing a process to monitor how incidents are screened at installations, DOD does not know the total number of reported child abuse incidents across the department. While DOD has expanded its child abuse policies and procedures to include child-on-child sexual abuse, gaps exist. For example, DOD standardized the IDC process in 2016, but the new structure does not include medical personnel with expertise, contrary to best practices for substantiating child abuse allegations. Without expanding the IDC membership to include medical personnel, members may not have all of the relevant information needed to make fully informed decisions, potentially affecting confidence in the efficacy of the committee's decisions. GAO also found that the availability of certified pediatric sexual assault forensic examiners across DOD is limited—according to DOD officials, there are only 11 in comparison to 1,448 incidents of child sexual abuse that met DOD's criteria for abuse from fiscal years 2014 through 2018. Without processes that help ensure timely access to certified pediatric examiners, child victims of sexual abuse overseas may not receive exams in time for evidence to be collected for use in prosecution, increasing the stress and trauma of affected victims. GAO is making 23 recommendations, including that the military services develop a process to monitor how reported incidents are screened at installations, that DOD expand the membership of the IDC to include medical personnel, and that DOD establish processes that help ensure timely access to certified pediatric examiners overseas. DOD concurred with 16, partially concurred with six, and did not concur with one of GAO's recommendations, which GAO continues to believe are valid, as discussed in the report.", "document_type": "gao"}
{"report": "SBA oversees a number of programs designed to provide small businesses with resources and tools, including access to capital, help with federal contracting opportunities, and entrepreneurial counseling and training. Federal procurement regulations generally define a small business as one that is independently owned and operated and not dominant in its field and that meets the size and criteria or standards established by SBA. The Small Business Act of 1953 authorized SBA to establish size standards for determining eligibility for all procurement programs in which small business status is required or advantageous. SBA uses the North American Industry Classification System (NAICS) as the basis for its size standards. The standards vary by industry and are generally expressed as the average number of employees over a 12-month period or average annual receipts in the previous 3 years. As of August 2019, employee- based size standards for federal procurement purposes ranged from 100 to 1,500, and revenue-based size standards ranged from $1.0 million to $41.5 million. The number of employees or average annual receipts indicates the maximum size allowed for a business and its affiliates to be considered small. The Small Business Jobs Act of 2010 required SBA to review at least one-third of all size standards during every 18-month period from the date of its enactment and all size standards at least once every 5 years thereafter. SBA completed the first 5-year review in 2016. According to SBA’s size standard methodology, the agency assesses industry structure and the overall degree of competitiveness of an industry and of firms in the industry when establishing size standards. To assess industry structure, SBA analyzes four primary factors: average firm size, degree of competition in an industry, start-up costs and entry barriers, and distribution of firms by size. SBA also considers the ability of small businesses to compete for contracting opportunities under the current size standards. According to SBA officials, for industries with $20 million or more in federal contracting annually, SBA also examines the small business share of federal contract dollars relative to the small business share of total industry receipts. Each year, SBA negotiates small business prime contracting goals with federal agencies that have procurement authority so that, in the aggregate, the federal government meets its goal of awarding 23 percent of prime contract dollars to small businesses. In September 2018, we reported that SBA considers prior-year achievement and other factors in setting annual agency goals. SBA’s procurement center representatives and OSDBUs assist agencies in meeting small business goals. Agency contracting officers have the authority to enter into, administer, or terminate contracts and are responsible for helping agencies meet small businesses goals, including by setting aside contracts for small businesses. One of the first steps in the federal acquisition process is assignment of the NAICS code that best describes the principal purpose of the acquisition and corresponding size standard (see fig. 1). Generally, the FAR states that if the contract is valued under the simplified acquisition threshold, the contracting officer must set it aside for small businesses. If valued above the simplified acquisition threshold, the contracting officer conducts market research to determine whether a contract should be set aside for small businesses. For contracts not set aside for small businesses, contracting officers generally must include specific small business subcontracting goals for the prime contractor to meet. The FAR requires agencies to evaluate price or cost to the government in every source selection and evaluate the quality of the product or service in the acquisition by considering one or more noncost factors, such as past performance. Contracting officers generally have discretion under the FAR to choose evaluation factors and their relative weights, rating systems, and the past performance they will consider. For example, a contracting officer could consider technical excellence, past performance (including relevance), and price. To select a firm for contract award, agency officials evaluate offers against the criteria specified in the solicitation. According to the FAR, if an offeror has no record of relevant past performance, the offeror cannot be evaluated favorably or unfavorably on past performance. Contracting officers use the Contractor Performance Assessment Reporting System to enter and review evaluations of past performance. The FAR generally requires agencies to document contractor performance on contracts or orders that exceed certain dollar thresholds. Once a project is complete, the assessing official rates the contractor on elements such as quality of the product or service, schedule, cost control, management, and small business utilization. This information then becomes available to other agencies for making source selection decisions. Indefinite delivery/indefinite quantity (ID/IQ) contracts provide flexibility when an agency cannot specify the quantities or timing of a product or service. Contracting officers may issue ID/IQ contracts as single-award or multiple-award contracts. Single-award refers to a situation in which one contract is awarded under a solicitation. The FAR establishes a preference for awarding multiple-award ID/IQ contracts—instances in which more than one prime contractor is awarded a task-order contract (for services) or delivery-order contract (for supplies) under a single solicitation. Once agencies determine their specific needs, such contracts allow agencies to establish a pool of qualified contractors to compete for future orders under streamlined procedures. Contractors compete to be in the pool and generally compete again for task or delivery orders. Multiple-award ID/IQ contracts can be unrestricted (open competition for businesses of all sizes) or restricted to small businesses. They also can have one pool of contractors with separate “tracks” for small and nonsmall businesses to ensure contract opportunities for small businesses (that is, some orders are set aside for small businesses). After a multiple-award ID/IQ contract has been awarded, an agency places delivery or task orders, generally using the fair opportunities process. An order, which is placed when a specific need arises, obligates funds and authorizes work. Orders must be within the scope, issued within the period of performance, and be within maximum value or quantities agreed to in the contract. For multiple-award ID/IQ contracts, the FAR requires that each awardee be given a fair opportunity to compete for subsequent orders. In April 2017, we found that in fiscal years 2011–2015, federal agencies obligated more than $130 billion annually through ID/IQ contracts. We also found that contracting officers said it was easier and faster to place an order under an existing ID/IQ contract than to award a separate contract when a specific need arose. A mentor-protégé program is an arrangement in which mentors—typically experienced prime contractors—provide technical, managerial, and other business development assistance to eligible small businesses, or protégés. SBA established the 8(a) Mentor-Protégé Program in 1998 for mentors to partner with 8(a) socially and economically disadvantaged businesses to improve the ability of 8(a) businesses to compete for prime contracts and subcontracts. The Small Business Jobs Act of 2010 and National Defense Authorization Act for fiscal year 2013 authorized SBA to establish a government-wide mentor-protégé program for all small businesses, which SBA named the All Small Mentor-Protégé Program. Small businesses that have a mentor-protégé relationship through either program can form a joint venture with a mentor (which can be a mid-sized or large business) and compete for set-aside contracts as long as the protégé is a small business with at least a 51 percent interest in the joint venture. Contractor team arrangements take two forms: two or more companies form a partnership or joint venture to act as a prime contractor or a prime contractor agrees with one or more companies to have them act as its subcontractors under a specified federal contract or acquisition program. Companies generally form a contractor team arrangement before submitting an offer. Businesses of all sizes can form joint ventures to compete for contracts. Joint ventures generally have to consist only of small businesses to compete for small business set-aside contracts—the exception being small and nonsmall businesses entered in a mentor- protégé agreement under one of SBA’s programs. Most small businesses awarded set-aside contracts in fiscal year 2017 did not appear poised to outgrow their size standard. According to FPDS-NG data, about 86 percent of the 121,604 set-aside contracts awarded in that year were to small businesses with revenue or employees at or below 25 percent of the size standard for their industry (see fig. 2). These businesses received about 64 percent of the dollar obligations for set-aside contracts in fiscal year 2017. The small businesses closest to their SBA standards (above 75 percent of the size standard) were awarded about 2 percent of the set-aside contracts and about 7 percent of the contract dollar obligations in fiscal year 2017. We performed the same analysis for fiscal years 2013–2016, and the results across the four quartiles were generally the same throughout the time period. Based on our review, a very small percentage of the small businesses that were awarded set-aside contracts in fiscal year 2008 grew to mid- sized in subsequent years and continued to receive any type of contract. As shown in table 1, more than 93 percent of the businesses that were awarded only set-aside contracts in fiscal year 2008 and received any federal contract (including a set-aside or competed contract) in fiscal year 2017 remained small. About 2.5 percent of such businesses had become mid-sized by fiscal year 2017. In addition, we analyzed the extent to which small businesses that grew to be mid-sized in 2013 continued to receive any type of contract in fiscal years 2014–2017. Of the 5,339 small businesses awarded only set- aside contracts in fiscal year 2008 and awarded any sort of federal contract in fiscal year 2013, 104 grew to mid-sized by fiscal year 2013. Of those 104 mid-sized businesses, 23 remained mid-sized in subsequent years and were awarded 75 contracts, and three grew to large and were awarded six contracts (see table 2). Seventeen of the 104 mid-sized businesses became small again. Thirty-seven of the 104 mid- sized businesses were awarded 306 contracts and were categorized as small, mid-sized, or large depending on the NAICS code listed in the contract. That is, businesses can be awarded contracts under several NAICS codes, each with a different size standard. Of the 24 mid-sized businesses not awarded any contracts in 2014–2017, nine were no longer registered in the System for Award Management, a central registration system for federal contractors. Federal regulations generally allow a small business with a contract to continue performing under its contract if it outgrows the size standard that it met in its initial offer. If a business qualified as small and was awarded a single-award contract under a small business set-aside, it generally would be considered small for contracting purposes for the life of that contract. The business can continue providing the service or product. Additionally, the agency can continue counting the contract towards its small business goals unless the business is required to recertify, whether through a regulatory or contractual requirement, and in doing so is deemed other than small. Once the contract ends, the follow-on or renewal contract is a new contract; size is determined as of the date the business bids on the new contract. The regulations are applied similarly to multiple-award contracts. Some multiple-award contracts are set aside for small businesses only. If a business qualified as small at the time of its initial offer, it is generally small for each order issued against the contract for the life of the contract even if it outgrows the size standard. Multiple-award contract orders awarded to businesses that have grown to be other than small during the course of the set-aside contract generally still may be counted toward agency small business goals. They would not be counted if the contractor were required to recertify, whether by a regulatory or contractual requirement, and in doing so was deemed other than small. There are a few instances in which a business must recertify its size status after its initial offer. In the case of an awarded multiple-award, set- aside contract, this would make the concern ineligible for the placement of orders or exercise of options. For example: SBA has stated that mergers and acquisitions create an exception to the general rule that a firm’s size and status is determined at the time of the initial award. Generally, if a business becomes other than small pursuant to a merger or acquisition after its initial offer, the business must recertify its size. Certain requirements for recertification become effective just before the end of the fifth year on a multiple-award contract. A multiple-award contract that runs for more than 5 years, including options, requires each business to recertify size within 120 days before the end of the fifth year and 120 days before exercising options thereafter. The determination of small or other than small is based on the size standard at the time of the recertification. Size determinations are not permanent; a business can recertify later as small if it meets the size standard. A contracting officer may require a business to recertify its size status in response to a solicitation for an order. An SBA recertification determination is based on the size as of the date the business submits its response to the order. The rules are different for agreements, including blanket purchase agreements. A blanket purchase agreement is a simplified method of filling anticipated repetitive needs for supplies or services that functions as a “charge account” with qualified sources of supply. Where the agreement is a set-aside or a reserve award to any type of small business, a business must qualify as small both at the time of the offer and at the time of the order to be considered for the order. The agency may count the business toward its small business goal if the business is small at the time of the order. Both SBA’s 8(a) and All Small Mentor-Protégé programs allow the mentor (including those that are mid-sized businesses) and the protégé to form a joint venture and bid on set-aside contracts based on the protégé’s status as a small business. Once a protégé no longer qualifies as small, the mentor-protégé joint venture will no longer be eligible to bid for new small business set-asides. But, a change in protégé size generally does not affect contracts previously awarded to a joint venture between the protégé and the mentor. The mentor-protégé joint venture may seek any small business contract for which the protégé would qualify. Therefore, the size of the mentor generally does not affect whether a mentor-protégé joint venture can bid for a small business contract. According to SBA officials, the agency does not track the size of mentors. As of September 2018, there were 106 joint ventures formed under the All Small Mentor-Protégé program and 171 joint ventures under the 8(a) Mentor-Protégé program (see table 3). We analyzed FPDS-NG and SBA data to determine the size of the mentors participating in joint ventures under SBA’s All Small Program that were awarded set-aside contracts in fiscal years 2016–2018. Of the 29 joint ventures awarded set-aside contracts during these years, 13 of the 26 mentors were mid-sized businesses. We reviewed options proposed in literature to enhance contracting opportunities for mid-sized businesses and asked stakeholders for their perspectives on potential benefits and drawbacks. Some options for increasing federal contracting opportunities for mid- sized businesses identified in our literature review would help mid-sized businesses more than others, according to stakeholders. They noted that establishing a set-aside for mid-sized businesses—the option designed to help mid-sized businesses most directly—also would pose challenges for small businesses and agencies. In contrast, some options primarily would help small businesses that were growing (revenue or employees approaching the size standards). This, in turn, could offset any of the advantages that mid-sized businesses would derive. For instance, benefiting small businesses could increase competition and result in fewer awards to mid-sized businesses. As shown in table 4, we grouped the options into four categories: (1) establishing a set-aside for mid-sized businesses, (2) modifying the rules for multiple-award contracts, (3) changing how past performance is considered when evaluating bid proposals, and (4) modifying SBA’s size standards. Several stakeholders told us that establishing a separate set-aside category for mid-sized businesses would increase contracting opportunities for mid-sized businesses, but others expressed concerns that the potential threat to small businesses and administrative burden on agencies might outweigh this benefit. Some literature suggests that when businesses outgrow their size standards they struggle to compete against much larger, established businesses for contracts. Also, literature we reviewed suggested small business goals motivate agencies to set aside more and larger contracts for small businesses, resulting in a scarcity of smaller contract solicitations for mid-sized businesses. Members of Congress have proposed establishing pilot programs that would help mid- sized businesses, either defined by business or contract size. Several stakeholders commented directly on separate set-asides and contracting opportunities for mid-sized businesses. An OSDBU director noted former small businesses (those that outgrew their size standard) would benefit from an opportunity to compete with firms of similar size for prime contracts. However, some stakeholders believed the set-aside would not increase opportunities for mid-sized businesses. Specifically, one trade association executive noted that this option continues to shelter small businesses that become mid-sized businesses from competition with larger businesses. The stakeholder added that a set-aside would not address the ability of mid-sized businesses to compete against large businesses on an unrestricted basis. Another trade association executive said there still would be a need to help mid-sized firms develop expertise and encourage competition. Some stakeholders believed the option would have a limited impact or was not necessary. Specifically, one trade association executive said that its members want a good path to growth for small businesses, not a set- aside. Another trade association executive similarly believed mid-sized businesses want to open up contract opportunities, not restrict them by creating more set-asides. One stakeholder also argued that the option could create incentives for large businesses to split their companies to fit new set-aside size standards. Most stakeholders believed the set-aside for mid-sized businesses would take away opportunities from small businesses, with several noting that contracts that normally would be set aside for small businesses might be set aside for mid-sized businesses instead. SBA officials stated that a set- aside would have a negative effect on all small business programs and support the use of larger contracts, resulting in fewer contract awards to small businesses. However, one trade association representative said the set-aside could be structured so that small businesses still could compete for the mid-sized set-aside contracts. An OSDBU director told us contracting officers could limit the effect on small businesses by considering small businesses first, mid-sized businesses second, and large businesses last. In this scenario, it would be large businesses that would be most affected by a set-aside for mid-sized businesses. Stakeholders cited more limitations than benefits for agencies if this option were implemented. Most stakeholders told us a mid-sized business set-aside would increase agency burden, including additional time and cost to define and implement the new set-aside and additional tracking and reporting costs. SBA officials noted that it would create an additional burden for contracting officers and that further study would be needed before implementing a mid-sized set-aside. Some stakeholders also noted the potential burden on agencies of complying with additional contracting goals, with one OSDBU director saying that agencies do not have the resources to meet current small business contracting goals, let alone meet mid-sized contracting goals. Another OSDBU director believed that mid-sized business set-asides likely would violate the World Trade Organization’s Government Procurement Agreement because the United States negotiated exclusions for small businesses in the agreement, but not for mid-sized businesses. Some stakeholders believed it would be very difficult for agencies to define a mid-sized business. One OSDBU director told us that contracting officers would have to perform new market research for mid-sized set-asides and abide by a new layer of requirements. Several stakeholders questioned which agency actually would administer the new set-aside program. Stakeholders identified a few benefits for agencies. Two stakeholders told us agencies could benefit from having a larger supplier base and more choices for services. One OSDBU director said agencies might benefit from retaining former small business contractors for a longer time, and a researcher said agencies might gain access to talent and value they might not get from large businesses. Stakeholders told us that allowing small businesses that grow beyond their size standards to move to the unrestricted version of multiple-award contracts could help mid-sized businesses. As discussed previously, multiple-award contracts can be unrestricted or restricted to small businesses or have separate tracks for small and nonsmall businesses (such as by using set-aside orders). This option proposes that small businesses on the restricted track of a multiple-award contract that outgrow the contract’s small business size standard be moved to the unrestricted track. This practice can be referred to as “on-ramping.” According to the stakeholder proposing this option, if a small business contractor grew to mid-sized, but could not transition to the unrestricted track, all the effort the business put into winning the contract would be wasted simply because it grew. Some multiple-award contracts allow small businesses that outgrow the size standard to move to the unrestricted track of the multiple-award contract, but this is not always the case. Agencies have discretion when making this determination. If a business is allowed to move to the unrestricted track of such a contract, it would be able to place bids on additional orders resulting from the contract. In cases in which a business cannot move to the unrestricted track, it has to leave the contract after completing any ongoing orders. For example, the General Services Administration’s One Acquisition Solution for Integrated Services allows businesses that have outgrown their size standards to move to the unrestricted track, while EAGLE II does not. Most stakeholders we interviewed said this option could increase contracting opportunities for growing small or mid-sized businesses. An OSDBU director said the option would let businesses that grew to be mid- sized move to the unrestricted pool of the multiple-award contract so they could keep their existing contract. A researcher said the option gives more time for small and mid-sized businesses to prepare for full and open competition. But two stakeholders noted that mid-sized businesses already in the unrestricted pool may be negatively affected by increased competition from additional contractors placed in the pool for task orders. Stakeholders offered differing opinions on how this option would affect agencies. Several stakeholders said that agencies would benefit from being able to retain contractors even if the contractors outgrew their size standard. For example, a trade association executive said it would be less disruptive for the agency if the business could continue its contract. Two stakeholders thought that moving a business to the unrestricted pool would reduce agency time and paperwork (compared to re-competing the contract and performing additional evaluations). However, several stakeholders told us that allowing small businesses that grew beyond the size standards to “on-ramp” might increase administrative burden on agencies. For example, it might take longer for an agency to evaluate proposals for unrestricted task order competitions if the pool of competitors grew. SBA officials expressed concern that if task order competitions grew too large, businesses in the unrestricted pool that objected to the increased competition from new contractors might pursue litigation. Stakeholders told us that changing past performance requirements could increase prime contracting opportunities for mid-sized and small businesses, but might increase risk for agencies. Some of the literature we reviewed considered requirements based on the size or number of past contracts (“quantitative past performance requirements”) as a barrier to entry for mid-sized businesses. Options have been proposed that would regulate what types of past performance contracting officers consider and how they establish solicitation requirements. This option proposes lowering (for example, limiting their use or making their terms more flexible) or eliminating quantitative requirements for past performance. Nearly all the stakeholders we interviewed said that lowering quantitative requirements would increase contracting opportunities for mid-sized businesses, small businesses, or both. For example, one researcher said that mid-sized and small businesses would benefit because the barriers to entry on some large contracts would be lowered. Similarly, an OSDBU director said that smaller mid-sized businesses and small businesses get shut out of contract competitions because they cannot meet the past performance requirements, and lower past performance requirements would give them a chance to compete. One trade association executive also pointed out that this option could help mid-sized and small businesses develop a performance record for future solicitations. More than half of the stakeholders told us that eliminating quantitative past performance requirements entirely also would increase contracting opportunities for mid-sized businesses, small businesses, or both. Specifically, two stakeholders said eliminating these quantitative requirements would enable mid-sized and small businesses without records of past performance to substantiate their qualifications in other non-quantitative ways. Stakeholders noted trade-offs for agencies. Some stakeholders believed lowering past performance requirements would benefit agencies because more contractors would be eligible to bid. For example, one OSDBU director said agencies might receive proposals from businesses that could not have met quantitative requirements but have enough expertise to submit a high-quality proposal. However, some stakeholders said lowering quantitative requirements may increase agency burden, citing a longer evaluation period due to a larger pool of bidders. More than half of the stakeholders said eliminating requirements entirely would increase the burden on agencies, for various reasons. One researcher said agencies might not have staff with the technical expertise to assess bids based on a strictly qualitative evaluation. Additionally, several stakeholders noted challenges for agencies in obtaining qualitative performance information using the Contractor Performance Assessment Rating System, citing rating subjectivity and verification difficulties. Several stakeholders said lowering or eliminating quantitative requirements for past performance would increase the risk to the agency of awarding contracts to firms that cannot successfully complete the project. For example, one OSDBU director pointed out that “conventional wisdom” for contracting officers is that a project’s success rate is higher when a company can meet higher past performance requirements. Another OSDBU director said that qualitative forms of evaluation, such as testimony from another agency, are not sufficient and could put the agency’s project at risk for lack of an objective measure of a contractor’s capabilities. In contrast, the third OSDBU director said that quantitative requirements do not lower the risk to the agency because completing a certain number of contracts is not a guarantee of satisfactory future performance. This option proposes that contracting agencies be required to consider the past performance of individual companies in team arrangements as opposed to evaluating only the aggregate past performance of contractors in team arrangements. The literature suggested that requiring agencies to consider each team member’s past performance would provide incentives to mid-sized businesses to work together to compete for contracts with past performance requirements that each would not be able to meet individually. According to SBA officials, the Small Business Act already requires agencies to consider the past performance of each participant in a joint venture or team for bundled contracts and multiple-award contracts above a certain dollar threshold. Also, agencies generally consider the relevant past performance information of individual members of a team arrangement in certain situations if they will perform major or critical aspects of the requirement. However, there are other situations—such as when the contract is not specifically for a small business but instead is bid on by a joint venture that includes a small business—in which agencies are not required to consider each team member’s past performance. Stakeholders identified some benefits to a more flexible consideration of past performance for mid-sized businesses. More than half of the stakeholders believed this option would increase contracting opportunities for mid-sized businesses. For example, a trade association executive said mid-sized businesses currently struggle to fulfill past performance requirements, and this would allow them to combine their past performance with another business to qualify for new and larger contract opportunities. However, some stakeholders noted that mid-sized companies probably have won prime contracts. Therefore, they already might have the requisite past performance to bid on a contract. Nearly all the stakeholders we interviewed thought this option would increase contracting opportunities for growing small businesses because they would be able to team with a small or nonsmall business to bid on contracts for which they otherwise would not have the past performance to qualify. One researcher described a dilemma for small businesses: they cannot compete for contracts without past performance, but they cannot get past performance without winning contracts. SBA officials said that businesses prefer that the past performance of each member be considered instead of the past performance of the joint venture, which could be minimal, especially if it was a new joint venture. Stakeholders identified trade-offs for agencies. More than half of stakeholders said considering past performance of both members in a team arrangement would benefit agencies because more contractors could meet requirements to bid. One OSDBU director said that this option also might allow agencies to benefit from the enhanced capacity and innovative solutions offered by mid-sized businesses. However, several stakeholders cautioned that this option could increase risk for an agency. For example, one trade association representative believed that because team arrangements are the companies’ creation and the government has no involvement in administering them, there is more risk to the agency that the contracting team might not be able to complete the contract. This option proposes that agencies should consider subcontracting past performance when evaluating bid proposals. It has been suggested that this could be done in two ways. First, agencies could be required to consider a business’s past performance as a subcontractor—a route for many small businesses to gain access to federal contracts—when competing for prime contracts. Second, agencies could be required to count the past experience of both the prime contractor and its significant subcontractors towards a solicitation’s past performance requirements. Stakeholders had differing opinions on whether this option would help mid-sized businesses. Several stakeholders believed that allowing mid- sized businesses to leverage their subcontracting experience to meet requirements would increase contracting opportunities for these firms. Furthermore, one researcher thought mid-sized businesses could secure more subcontracting opportunities because large firms might be more willing to team with them. However, some stakeholders believed this option would not increase contracting opportunities for mid-sized businesses, with two stating that this option is less important for them because they likely outgrew their size standard by winning set-aside prime contracts. An OSDBU director thought it also might increase competition from smaller firms. Nearly all of the stakeholders we interviewed said this option could increase prime contracting opportunities for growing small businesses. For example, one OSDBU director said that considering subcontracting as past performance would help small businesses compete for prime contracts, grow, and move forward. A researcher noted this could help small businesses transition to mid-sized. SBA officials similarly stated that small businesses want agencies to consider their subcontracting past performance so they can access contracts for which they would not normally qualify. In May 2019, SBA officials said they were working on implementing legislation that requires SBA to create a pilot program to provide past performance ratings for small business subcontractors. Similar to previous options, stakeholders contrasted the benefits and drawbacks of increased competition for agencies. Several stakeholders thought this option would expand the pool of bidders, making contracts more competitive and bringing more value to agencies. However, four stakeholders noted that verifying prime and subcontracting experience could create more work for agencies. Specifically, two of the four noted that it could be difficult for agencies to determine subcontracting past performance because the ratings in the Contractor Performance Assessment Rating System are tied to the prime contractors. Some stakeholders also noted this option could increase the risk to agencies that projects would not be completed successfully. Specifically, because subcontracting agreements are between the subcontractor and prime contractor, and therefore are not enforceable by agencies, a prime contractor might not use a subcontractor whose past performance was considered during the evaluation process. For example, a researcher pointed out that a prime contractor could use the subcontractor’s experience to win a contract, but then not use the subcontractor for any of the work. Several stakeholders noted that modifying SBA’s size standards would not help mid-sized businesses as such; rather, the modifications could allow a few mid-sized businesses to become eligible for small business set-aside contracts again and help growing small businesses prepare for the transition to mid-sized. It has been noted that agencies increasingly use large, multiple-award contracts that can cause small businesses to outgrow their size standard before they build the capacity (financial resources, business infrastructure, or past performance records) they need to successfully compete for contracts. To address such issues, options have been proposed to modify SBA’s size standards. This option would change the number of years of revenue considered when applying revenue-based size standards. SBA would allow businesses to consider their past 5 years of revenue, pick the lowest 3 years in that period, and average them to determine if they met revenue- based size standards. In a December 2018 amendment to the Small Business Act, Congress extended the number of years of revenue that service businesses use to calculate their size from 3 to 5 years but included no provision related to selecting lowest-revenue years. Stakeholders expressed reservations regarding this option for mid-sized businesses. Some stakeholders said that this option may not increase opportunities for mid-sized businesses because they already had outgrown their size standard. Several stakeholders also said the option offered only a temporary solution. One noted that this option would delay “graduation” from the size standard, but would not address the issue that mid-sized businesses need to continue to grow to secure additional federal contracts. An OSDBU director told us it is critical that small businesses develop and execute a marketing and business plan to transition from small to successful mid-size. Nearly all the stakeholders we interviewed noted that allowing businesses to choose their lowest 3 years of revenue in a 5-year period could prevent an outlier revenue year from causing a small business to prematurely outgrow its size standard. For example, an OSDBU director said that a large, 1-year award is not indicative of a business’s revenue over the long term. Nearly all of stakeholders also said that enabling businesses to choose the lowest 3 years of revenue would help ease the transition to mid-sized. For instance, an OSDBU director said businesses could stay below the size standards for longer and establish a performance record to help secure future contracts. However, several stakeholders expressed concern that very small businesses might lose contracting opportunities due to increased competition (that is, more and larger-sized firms would remain under the size standards). Finally, SBA officials pointed out that this option could be perceived as unfair because it would not benefit businesses in industries with employee-based size standards. This option proposes that businesses be able to subtract research and development expenses from their total revenue when calculating their eligibility for small business status. The stakeholder proposing this option said that businesses close to the size standard have to focus their revenue on pursuing contracts that will support their company as they transition to full and open competition, and so cannot spare money to further invest in researching and developing new products or processes that might improve their business. In addition to encouraging more investment in research and development, subtracting these expenses would lower revenue and allow some mid-sized businesses to be classified as small again. One researcher told us this was a strong option for mid-sized businesses, particularly information technology businesses, because research and development investment is such a large part of their expenses. However, some stakeholders said this option would not increase contracting opportunities for mid-sized businesses. For example, one OSBDU director believed this option would not benefit mid-sized businesses because it did not help these businesses to compete with larger businesses. More than half of the stakeholders we interviewed said that this option might encourage small businesses to invest in research and development. However, several stakeholders noted that this option only would help the small percentage of small businesses that perform research and development. SBA officials pointed out that this option could be perceived as unfair because not all businesses have research and development expenses. They also pointed out that modifying revenue calculations would not benefit manufacturing businesses, which invest more in research and development than other sectors but primarily use employee-based size standards. Stakeholders also noted potential trade-offs for agencies. Several stakeholders told us this option would benefit the government by encouraging investment in research and development with one stakeholder stating that it might result in higher-quality bids. However, several stakeholders and SBA officials also told us that allowing small businesses to subtract research and development expenses would increase the administrative burden on agencies or add too much complexity. For example, two OSDBU directors said it would be difficult for an agency to verify that research and development expenses were correctly claimed and subtracted from revenue. SBA officials noted that there were no industry-by-industry data on research and development expenses. Some stakeholders observed this option could lead to an increase in fraud or manipulation, with one trade association executive saying the option would not increase innovation, just claimed expenses. This option proposes increasing SBA’s revenue-based size standards. The trade association representative who proposed the option believed that small business size standards should be raised so that high revenue- generating small businesses that still are not dominant in their field would not be shut out of set-asides. Increasing revenue-based size standards would benefit some mid-sized businesses by making them eligible again for small business set-asides. However, more than half of the stakeholders told us the option would have a limited impact—it would apply only to the mid-sized businesses small enough to fall under the newly raised standard—or no impact at all (for most other mid-sized businesses). To illustrate the limited impact, one OSDBU director used the example of management consulting services (NAICS 541611), which has a size standard of $15 million (revenue). If the standard were increased to $17 million, it might not affect many businesses. Rather, it would help only the $16 million company to compete for set-asides again, the director said. Furthermore, one researcher said that the increase would not address the systemic disadvantage that mid-sized businesses face in competing with large businesses. Stakeholders identified tradeoffs for small businesses related to this option. Nearly all stakeholders said that raising revenue-based size standards could help growing small businesses better prepare to transition to mid-sized while remaining eligible for set-asides. One OSDBU director said small businesses could add to their performance record and have more time to become competitive with larger businesses. One researcher said that small businesses could get additional time to diversify contract portfolios and fund professional certifications. However, some stakeholders cautioned the option could harm very small businesses because as one stakeholder explained, there would be more competitors for small business set-asides. As noted previously, we found that most small businesses awarded set-aside contracts in 2017 were well below the size standards. Stakeholders also identified trade-offs for federal agencies. Several stakeholders said agencies would benefit from the increased competition. For example, one OSDBU director said agencies might have more bidders, which could lower pricing. Some stakeholders said the option could help agencies reach small business goals more easily because more businesses would be considered small. However, a stakeholder advised that agencies also might need more time and resources to evaluate an increased number of bids. SBA officials explained to us that they comprehensively review all the size standards every 5 years, looking at factors, such as industry trends and small business market share. They contended that if contracts became larger to the detriment of small businesses, small businesses then would have a decreased market share. If small businesses were losing market share, that would be captured by the SBA size standard methodology and the size standards would be adjusted accordingly. They also noted that revenue-based size standards were getting higher and higher as a result of adjustments during SBA’s reviews and adjustments for inflation and that further increases might allow firms that were dominant in their industry to be small, which is contrary to statute. Finally, they stated that just raising size standards without taking into account industry structure and market conditions would enable more experienced businesses to qualify as small and hurt small businesses that need federal assistance the most, especially in competing for set-aside contracts. We provided a draft of this report to the Department of Homeland Security, Department of Defense, General Services Administration, and SBA for their review and comment. The Department of Homeland Security provided technical comments, which we incorporated where appropriate. In emails, the OSBDU director at the Department of Defense and an audit liaison at the General Services Administration stated that the agencies did not have any comments. SBA provided technical comments in an email from the GAO Liaison, which we incorporated as appropriate. We considered a number of these comments to be more than technical in nature and therefore, summarize them here: SBA offered new views on three specific options for increasing federal contracting opportunities for mid-sized businesses presented in the report, which we incorporated where appropriate. SBA also made the larger point that they believe any option to help mid-sized businesses would hurt small businesses. In discussing the various options in the report, we present the views of various stakeholders and SBA on how the options would affect small businesses. SBA stated that we created our own methodology for determining a mid-sized business by multiplying the current size standards and that a formal study should be performed to establish a baseline definition of a mid-sized business. Our goal was not to establish a baseline definition of a mid-sized business. As we note in the report, there is no statutory or regulatory definition of a mid-sized or large business. We applied multipliers to SBA’s size standards only for the purposes of our analysis—specifically, to identify businesses that had outgrown small business size standards and continued to receive federal contracts. SBA stated that we multiplied size standards by a factor of five to define mid-sized businesses in all industries and cited analysis that it had done that indicated that more than 95 percent of businesses are at or below SBA’s size standards. The agency concluded that this means that in some industries, almost all firms would be considered mid-sized under our definition of mid-sized. However, only those firms with revenue or employees up to five times above the SBA small size standard would be considered mid-sized in our analysis. We counted any businesses with revenue or employees at or below the small size standard as small. SBA stated that considering a factor of two or three times the SBA size standards to identify mid-sized businesses would improve our results. We considered a number of different factors when developing our methodology. As noted in the report, we used five times the small size standard to distinguish between mid-sized and large businesses based on the distribution of contracts and obligations among businesses in these two groups. SBA stated that (1) the report should explain the basis and method for selecting the sample of 5,339 businesses awarded set-aside contracts in 2008 and (2) a sample of 104 out of 5,339 firms over that period of time was too small to be generalizeable. The 5,339 businesses awarded set-aside contracts in 2008 and awarded any sort of federal contract in 2013 were not a sample; rather, they were all the businesses that met these criteria. Therefore, we did not generalize to the population based on a sample. Our analysis showed that only 104 of these 5,339 businesses grew to mid-sized by 2013. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Defense, the Acting Secretary of Homeland Security, the Administrator of the General Services Administration, and the Acting Administrator of SBA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact William B. Shear at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix III. This report analyzes (1) the extent to which small businesses grew to be mid-sized and continued to receive federal contracts; (2) instances in which mid-sized businesses can perform work on contracts set aside for small businesses; and (3) options for increasing federal contracting opportunities for mid-sized businesses and views on the strengths and limitations of the options. We present information on contracts awarded to small, mid-sized, and large businesses in fiscal year 2017 in appendix II. For the first objective, we analyzed data from the Federal Procurement Data System-Next Generation (FPDS-NG) for fiscal years 2008 through 2017 (the most recent complete data available when we began our review). For a consistent set of data across the 10-year period, we selected all contracts awarded above the micro-purchase threshold. To determine the relevant contract size standard, we matched the size standards data from the Small Business Administration (SBA) to the contract awards data from FPDS-NG based on the year of award and the contract’s North American Industry Classification System (NAICS) code. We then compared the businesses’ annual revenue or number of employees to SBA size standards. We assessed the reliability of the FPDS-NG data we used by performing electronic testing of selected data elements and reviewing existing information about FPDS-NG and the data the system produces. We determined that these data were sufficiently reliable for the purposes of determining the extent to which small businesses that grew into mid-sized businesses continued to receive federal contracts and the size of businesses awarded contracts during a specific time period. To determine the extent to which contracts were set aside for small businesses, we calculated the percentage of new contracts awarded in fiscal year 2017 that were small business set-asides. To determine the size of the small businesses awarded these set-aside contracts, we divided SBA’s size standards into four segments for each NAICS code— below or at one-fourth of the size standard, above one-fourth to one-half of the size standard, above one-half to three-fourths of the size standard, and above three fourths of the size standard—and determined the number and obligations of set-aside contracts awarded to small businesses in each quartile for fiscal year 2017. We completed this same analysis for fiscal years 2013 through 2016 to see if the results were similar. To determine the extent to which small businesses grew to be mid-sized and continued to receive federal contracts, we used FPDS-NG data from fiscal years 2008 through 2017. Because there is no statutory or regulatory definition of a mid-sized or large business, we applied a number of multipliers to determine size. Businesses with revenues or employees at or below the SBA small size standards were small. We considered businesses with revenue or employees up to five times above the SBA size standard as mid-sized businesses. We considered businesses with revenue or employees more than five times the size standard as large businesses. We used five times the small size standard to distinguish between mid-sized and large businesses based on the distribution of contracts and obligations among businesses in these two groups. We discussed this approach and methodology with SBA officials and officials at three federal agencies that had large obligations for small business contracts in fiscal year 2017. These officials did not raise any questions about our approach, and some reiterated that there was no legal definition of mid-sized businesses. Using these definitions, we selected businesses awarded only small business set-aside contracts in fiscal year 2008 and determined whether these businesses also were awarded any type of federal contract in fiscal year 2017 and if they were in the same or different size category in fiscal year 2017. We then determined the number of businesses awarded set- aside contracts in fiscal year 2008 and awarded any sort of federal contract in 2013 that had become mid-sized in fiscal year 2013 and the extent to which those businesses were awarded any contracts in subsequent years. We also determined the percentage of competed contracts awarded to small, mid-sized, and large businesses in fiscal year 2017. For purposes of this report, competed contracts are those competed using (1) full and open competition, (2) full and open competition after exclusion of sources, and (3) simplified acquisition procedures. To determine the industry sectors with the largest number of set-aside and competed contracts in fiscal year 2017, we collected and analyzed FPDS data for each of the two-digit NAICS industry sectors. See appendix II for more information. For our second objective, we reviewed the Federal Acquisition Regulation and small business laws and regulations to identify provisions that allow small businesses that grow into mid-sized businesses to continue providing services and goods on contracts set aside for small businesses. We reviewed SBA documentation related to its 8(a) and All Small Mentor- Protégé programs because forming joint ventures with small businesses under these programs is one way that mid-sized businesses can provide services and goods under set-aside contracts. We analyzed lists from SBA of the businesses that entered into mentor-protégé agreements as of July 2018 and the mentor-protégé agreements that had formed joint ventures as of September 2018. Using FPDS-NG data, we determined the number of joint ventures formed under the All Small Mentor-Protégé program that had been awarded set-aside contracts from fiscal years 2016 through 2018. We began with 2016 to allow time after the program was created in 2013 for businesses to enter into agreements and form joint ventures. We ended with 2018 because it was the most recent complete year of data available when we conducted this analysis. Using the same multiplier methodology designed for our first objective, we determined the number of mentors awarded set-aside contracts as part of a joint venture that were mid-sized businesses. We assessed the reliability of the SBA and FPDS-NG data we used by interviewing SBA officials about their data and performing electronic testing. We determined that these data were sufficiently reliable for determining the number of mentors awarded set-aside contracts as part of a joint venture that also were mid-sized businesses. We were not able to perform a similar analysis for joint ventures formed under SBA’s 8(a) Mentor-Protégé program because SBA does not maintain a Data Universal Numbering System number for mentors participating in that program. This number is needed to determine the size of the mentor. To identify stakeholder views on options for increasing federal contracting opportunities for mid-sized businesses, we identified a number of proposed options by reviewing literature, including sources identified during our background research and initial interviews. We also conducted a literature search. We used ProQuest to search 13 databases—including Business Premium Collection, EconLit, Global Newsstream, Policy File Index, and ProQuest Dissertations and Theses Global Research Library. We also conducted searches using Lexis Advanced, EBSCO Business Source Corporate Plus, Dialog, DTIC, Scopus, and HeinOnline. The search was limited to 11 years (2008–2018) and to scholarly, trade, think- tank, and government publications. For the searches, we used keywords such as “advanced small businesses,” “federal contracting,” “mid-tier/mid-sized small businesses,” “middle market,” “IDIQ,” “challenges,” and “opportunities.” Our searches yielded 199 sources. To select relevant sources, an analyst reviewed the titles and abstracts and selected 21 as likely to propose options for increasing federal contracting opportunities for mid-sized businesses. A second analyst reviewed the first analyst’s selection for concurrence. The 21 sources we selected included trade association reports, congressional testimonies, and research reports. One analyst read the 21 sources and identified any specific options discussed. We eliminated suggestions or recommendations that were unclear, duplicative, or unconnected to mid-sized businesses. A second analyst read the same sources and verified that the first analyst had correctly identified all the options pertaining to increased contracting opportunities for mid-sized businesses. From this analysis, we compiled a final list of 14 options for which we would obtain stakeholder views. The options selected were grouped into four categories. The list of options included in the report is not exhaustive; the options are intended only to be illustrative of potential approaches to enhancing contracting opportunities for mid-sized businesses. To obtain stakeholders’ views regarding the strengths and limitations of these options, we selected three categories—trade associations, researchers, and federal agencies—from which to develop a nongeneralizable sample of individuals to interview. To identify trade associations, we compiled a list of 20 trade associations that represented small and mid-sized businesses from the literature search and previous GAO work on small business contracting. We searched each organization’s website for any publications the organization may have published on small and mid-sized businesses and federal contracting. We used search terms such as “federal contracting,” “mid-size,” and “size standards.” We identified six trade associations using this process. Because one did not respond to our request, our sample included representatives of the remaining five associations. We also selected three researchers who published on mid-sized businesses and federal contracting. In addition, we selected the directors of the Offices of Small and Disadvantaged Business Utilization (OSDBU) at three federal agencies—Department of Defense, Department of Homeland Security, and General Services Administration—that were among the top five agencies in terms of total dollar obligations for small business contracts in fiscal year 2017. We then interviewed the 11 stakeholders. For each interview, we asked them to provide their views on the strengths and limitations of each option in relation to small, mid-sized, and large businesses and for federal agencies. We performed a content analysis to analyze the responses. First, we created preliminary codes that represented key themes across the interviews of the strengths and limitations of the 14 options, such as “would increase administrative burden on agencies” or “would increase contracting opportunities for mid-sized businesses.” A methodologist reviewed the coding system to ensure it was logical. We pre-tested the coding of responses from three interviews to ensure the appropriateness of the codes. One analyst coded each response to a particular strength or limitation of an option and a second analyst reviewed the coding. If a response did not align with a strength and limitation, the response was coded as “unclassified.” The team discussed the results of the initial coding analysis and made some adjustments to the codes. Once the coding scheme was finalized and the responses from the remaining eight interviews were coded by an analyst, a second analyst reviewed the coding. If the second analyst disagreed with the coding of a particular response, the two analysts spoke and achieved concurrence. After response coding was completed, we tabulated the responses based on the codes. A second person verified the calculation of the stakeholders’ response totals. Because we selected a nongeneralizable sample of stakeholders to interview, their views are not generalizable to other stakeholders who have knowledge about options for increasing contracting opportunities for mid-sized businesses, but their views offered important perspectives. To characterize the number of stakeholders who offered the same opinion, we used “nearly all” for nine or 10 stakeholders, “most” for seven or eight stakeholders, “more than half” for six stakeholders, “several” for four or five stakeholders, and “some” for three stakeholders. We also interviewed SBA officials to obtain their views on how the options might affect small businesses, as well as to gather information related to our other two objectives. We conducted this performance audit from April 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In this appendix, we provide information on contracts awarded to small, mid-sized, and large businesses in fiscal year 2017. Because there is no statutory or regulatory definition of a mid-sized or large business, we applied multipliers to the Small Business Administration’s (SBA) size standards. Small businesses were those with revenue or employees at or below the size standard for their industry. We considered mid-sized businesses as those with employees or revenue up to five times above the size standard and large businesses as those with employees or revenue more than five times the size standard. Our analysis of Federal Procurement Data System-Next Generation (FPDS-NG) data showed that mid-sized businesses received the smallest share—9 percent—of competed contracts (compared with small and large businesses) in fiscal year 2017 (see fig. 3). For purposes of this report, competed contracts are those competed using (1) full and open competition, (2) full and open competition after exclusion of sources, and (3) simplified acquisition procedures. We analyzed FPDS-NG data to determine the number of set-aside and competed contracts awarded in fiscal year 2017 by industry sector. In fiscal year 2017, the largest number of set-aside contracts were awarded in the following sectors: manufacturing; professional, scientific, and technical services; and construction (see fig. 4). In that same year, the largest number of competed contracts were awarded in the following sectors: manufacturing; professional, scientific, and technical services; and wholesale trade. Similarly, in fiscal year 2017 the largest contract obligations (set-aside and competed) were awarded in the sectors of construction; manufacturing; and professional, scientific, and technical services (see fig. 5). For competed contracts, we analyzed FPDS-NG data to determine if the size of businesses awarded contracts varied by industry sector. In fiscal year 2017, small and large businesses were generally awarded more competed contracts than mid-sized businesses, regardless of sector (see table 5). The industry sectors in which small and large businesses were awarded the most competed contracts in fiscal year 2017 were manufacturing; professional, scientific, and technical services; and wholesale trade. Similarly, in fiscal year 2017, mid-sized businesses were awarded the most competed contracts in the manufacturing and professional, scientific, and technical services sectors. The third dominant sector for mid-sized businesses was information. In addition to the contact name above, Paige Smith (Assistant Director), Nancy Eibeck (Analyst in Charge), Edward Chiu, Sarah Garcia, Julia Kennon, Jill Lacey, Barbara Roesmann, Jessica Sandler, and Jena Sinkfield made significant contributions to this report.", "summary": "Small businesses that receive federal contracts set aside for them may outgrow the size standards the Small Business Administration (SBA) uses to define small businesses. (Size standards vary by industry and generally are based on employees or revenue.) Questions have been raised about the extent to which mid-sized businesses can compete with large businesses for federal contracts. GAO was asked to provide information on federal contracting opportunities for mid-sized businesses. This report analyzes, among other objectives, (1) the extent to which small businesses grew to mid-sized and continued to receive federal contracts and (2) options for increasing contracting opportunities for mid-sized businesses. GAO analyzed federal contracting data for fiscal years 2008–2017 (most recent and complete). In the absence of legal definitions of “mid-sized” and “large,” GAO multiplied relevant size standards for small businesses to arrive at parameters for mid-sized and large businesses for its analysis. GAO reviewed literature to identify options for increasing contracting opportunities and interviewed SBA officials and a nongeneralizable selection of 11 stakeholders—trade association representatives, researchers, and small business directors at three agencies with large obligations for small business contracts in fiscal year 2017—to obtain views on the options. SBA provided comments, which we addressed as appropriate. From fiscal year 2008 through 2017, very few small businesses that were awarded limited competition (set-aside) contracts grew to be mid-sized and continued to receive contracts. (GAO defined mid-sized businesses as having revenue or employees up to five times above the small business size standard.) Of the 5,339 small businesses awarded set-aside contracts in fiscal year 2008 and awarded any sort of federal contract (including set-aside or competed) in 2013, 104 became mid-sized by fiscal year 2013. Of those 104 businesses, 23 remained mid-sized through 2017 and won 75 contracts. Another three businesses became large and won six contracts. Options for increasing federal contracting opportunities for mid-sized businesses that GAO identified in its review include establishing a separate set-aside category, changing consideration of past contracting performance, and modifying size standards. Stakeholders told GAO some options would help mid-sized businesses more than others. While a set-aside category for mid-sized businesses would increase opportunities for mid-sized businesses, stakeholders generally believed it could decrease opportunities for small businesses and increase agency burden (time and costs to implement the set-aside). Requiring agencies to consider businesses' past performance as subcontractors or as part of a team would help both mid-sized and growing small businesses by making them more competitive for contracts. Stakeholders said raising size standards based on revenue would allow a limited number of mid-sized businesses to be eligible for set-asides again, but not help the vast majority of mid-sized businesses.", "document_type": "gao"}
{"report": "To receive FEMA assistance under FEMA’s Individuals and Households Program, through which disaster survivors can receive help with housing and other needs, individuals must register by answering a standard series of intake questions. In our May 2019 report, we found that some individuals with disabilities may have faced long wait times and unclear registration questions, and that FEMA’s internal communication across its programs about survivors’ disability-related needs was ineffective. Long wait times: Individuals who tried to apply for assistance using the helpline confronted long wait times, which may have posed greater challenges for those with disabilities. In the days after Hurricane Maria affected Puerto Rico and the U.S. Virgin Islands— when survivors from Harvey and Irma were concurrently contacting the helpline—up to 69 percent of calls went unanswered and the daily average wait time for answered calls peaked at almost an hour and a half, according to our analysis of FEMA data. While long wait times could be burdensome for all individuals, state officials and disability advocates we interviewed said long wait times were especially burdensome for people with certain disabilities, such as those with attention disorders or whose assistive technology prevents multi- tasking when waiting on hold. Unclear registration questions: FEMA’s registration process did not give individuals a clear opportunity to state they have a disability or request an accommodation because the registration did not directly ask registrants to provide this information. According to FEMA officials at the time, information about disability-related needs can help FEMA staff match individuals with disabilities with appropriate resources in a timely and efficient manner and target additional assistance, such as help with the application process. However, individuals with disabilities may not have requested accommodations or reported their disability and related needs during FEMA’s registration-intake due to the unclear questions. As a result, the registration process may have under-identified people with disabilities. For example, in Puerto Rico, an estimated 21.6 percent of people have disabilities, according to 2017 Census data. However, less than 3 percent of all registrants in the territory answered “yes” to the disability-related question in response to Hurricanes Irma and Maria. Ineffective communication across FEMA programs: Individuals may have faced challenges receiving necessary assistance because FEMA did not effectively track and communicate information about individuals’ disability-related needs across its assistance programs after such needs were identified. FEMA officials we interviewed for the May 2019 report explained that accommodation requests and disability-related information identified after registration-intake are recorded in a general “notes” section of a registrant’s case file, which can be easily overlooked as a case file is passed along to subsequent FEMA officials. In our May 2019 report we recommended that FEMA implement new registration-intake questions to improve FEMA’s ability to identify and address survivors’ disability-related needs. FEMA concurred with this recommendation, and officials reported that in May 2019 the agency updated the questions to directly ask individuals if they have a disability. According to FEMA’s analysis of applications for assistance following recent disasters, which used the updated questions, the percentage of registrants who reported having a disability increased. FEMA officials stated this increase gives them confidence the change has improved FEMA’s ability to identify and address disability-related needs of individuals affected by disasters. We also recommended that FEMA improve its communication of registrants’ disability-related information across FEMA programs, such as by developing an alert within survivor files that indicates an accommodation request. FEMA did not concur with this recommendation, explaining that the agency lacks specific funding to augment the legacy data systems that capture and communicate registration information. In its comments on our May 2019 report, FEMA stated that it began a long- term initiative in April 2017 to improve data management and exchange, and improve overall data quality and standardization. After FEMA completes this initiative, which officials said will be in 2024, FEMA expects that efforts to share and flag specific disability-related data will be much easier. We believe that in the interim, FEMA could explore other cost-effective ways to improve communication, such as through agency guidance that encourages program officials to review registrants’ case file notes. As FEMA moves ahead with its initiatives to improve data, we encourage it to consider and ultimately implement technology changes, such as developing an alert within files that indicates an accommodation request, to help improve communication across FEMA programs. State, territorial, and local governments are primarily responsible for response and recovery activities in their jurisdictions, including those involving health and safety. In our May 2019 report, we found that the substantial damage caused by the 2017 hurricanes prevented or slowed some individuals with disabilities from obtaining food and water. According to territorial and nonprofit officials in Puerto Rico and the U.S. Virgin Islands, as well as survivors we interviewed in the U.S. Virgin Islands, this was due to centralized distribution models, in which the majority of food and water was distributed to centralized locations around the islands. Officials from one governmental agency in Puerto Rico said this posed a major barrier to people with mobility challenges or without caregivers receiving food and water because they had to rely on home delivery, which took time and in some cases, did not happen. We also found that Hurricane Maria survivors faced challenges obtaining needed medication and oxygen in Puerto Rico and the U.S. Virgin Islands, according to territorial and nonprofit officials. State, territorial, and local agencies are also primarily responsible for administering shelters, when necessary, for those affected by a disaster. We found in our May 2019 report that individuals with disabilities affected by the 2017 hurricanes may have faced challenges accessing basic services from local shelters, including restrooms and food, according to state, territorial, local, and nonprofit officials in Florida, Puerto Rico, Texas, and the U.S. Virgin Islands. For example, nonprofit officials in Florida and Puerto Rico described instances of shelter residents with impairments that prevented them from accessing shelter restrooms. We also found that transportation was especially challenging for those who relied on public transportation or were unable to walk long distances, such as people with disabilities, according to state, territorial, local, and nonprofit officials we interviewed. For example, Florida state officials reported that few public transportation services, including paratransit, were functional following Hurricane Irma. This may have prevented some people with disabilities from maintaining their health and wellness—such as by shopping for groceries or going to medical appointments—after the storm, according to state officials. Officials we interviewed from Texas, Florida, and Puerto Rico for our May 2019 report said they had difficulty obtaining FEMA data that could help them deliver assistance to individuals, including those with disabilities. The officials explained that data—including names and addresses— showing who has registered for and received assistance from FEMA can help local governments and nonprofits identify who in their community needs assistance. To better facilitate authorized nonfederal partners obtaining these needed data, we recommended that FEMA develop and publicize guidance for partners who assist individuals with disabilities on how to request and work with FEMA staff to obtain the data, as appropriate. FEMA concurred with this recommendation and officials told us in July 2019 that the agency plans to publish data-sharing guidelines on its website, among other actions. Before initiating its new approach to disability integration, ODIC distributed an explanatory memorandum and other documentation to FEMA staff. For example, an April 2018 memorandum to FEMA Regional Administrators outlined a proposal to add new disability integration staff in each FEMA region to foster day-to-day relationships with state, territorial, and local emergency managers and disability partners. Also, ODIC distributed a document that described FEMA’s new approach to deployments. Under the new approach, fewer disability integration staff are to be deployed to disasters and all deployable staff and staff in programmatic offices are to receive training on disability issues during response and recovery deployments. However, in our May 2019 report, we found that these documents did not articulate objectives that could help the agency define success for the new approach. We concluded that without a set of common objectives for FEMA’s new disability integration approach, FEMA risks inconsistent application across its regions. In our report, we recommended that FEMA establish and disseminate a set of objectives for the new approach. FEMA concurred with this recommendation, and in July 2019 officials provided us with the draft of ODIC’s strategic plan for 2019-2022, which includes strategic goals and objectives that the new disability integration approach can help achieve. ODIC officials told us they will be working throughout 2019 with FEMA’s Office of External Affairs to disseminate the plan agency-wide and to nonfederal partners. We will continue to monitor FEMA’s progress toward sharing the objectives of its new approach to disability integration with critical stakeholders. To implement FEMA’s new deployment model, which will shift the responsibility of directly assisting individuals with disabilities from disability integration staff to all FEMA staff, FEMA planned to train all deployable staff and staff in programmatic offices on disability issues. We reported in May 2019 that FEMA officials emphasized the need to integrate disability competencies throughout FEMA’s programmatic offices and deployable staff. However, we found that the agency did not have written plans—including milestones, performance measures, or a plan for monitoring performance—for developing new comprehensive training for all staff. Starting in the 2018 hurricane season, FEMA had taken initial steps toward training some deployed staff on disability issues. For example, FEMA required all staff to complete a 30-minute training on basic disability integration principles and offered targeted “just-in-time” training to deployed staff. We concluded that developing a training plan would better position FEMA to provide training to all staff to help achieve FEMA’s intended goals. In our May 2019 report, we recommended that FEMA develop a plan for delivering training to FEMA staff that promotes competency in disability awareness. In its letter commenting on our May 2019 report, FEMA stated that ODIC is developing a plan to introduce the disability competency in FEMA’s position task books for all deployable staff. The letter explained further that ODIC’s plan will describe how FEMA will communicate the disability integration competency throughout the agency, establish milestones for measuring how effectively the competency is integrated across the agency, and outline how ODIC will monitor and measure integration of the competency across the deployable workforce. In July 2019, FEMA officials told us ODIC plans to hire new staff to focus on integrating the disability competency FEMA-wide. According to the officials, after the position task books are updated, ODIC will work with FEMA’s training components to ensure that disability-related training is consistent with the content of the position task books. FEMA officials also noted that the Field Operations Division, and not ODIC, is responsible for measuring how effectively the disability competency is integrated across FEMA. We will continue to monitor FEMA’s progress toward developing a plan for delivering training to promote competency in disability awareness among its staff. As noted in our May 2019 report, the plan for delivering such training should include milestones, performance measures, and how performance will be monitored. In our May 2019 report, we found that deploying a smaller number of disability integration staff and shifting them away from providing direct assistance to individuals with disabilities may result in nonfederal partners (such as state, territorial, and local emergency managers) providing more direct assistance to individuals with disabilities than they did previously. In February 2017, we reported that the comprehensive introductory training course on disability integration that FEMA offered to its nonfederal partners included substantial information on how to incorporate the needs of people with disabilities in emergency planning. However, according to officials, FEMA stopped offering this 2-day course in September 2017. ODIC officials told us during our 2019 review they had determined that the course, as designed, did not provide actionable training to emergency management partners to meet the needs of individuals with disabilities and planned to replace it. However, we found in May 2019 that although officials had plans to replace the course with new training, they had not provided a timeline, which would help ensure that partners are provided with timely information on inclusive emergency management practices. We recommended that FEMA develop a timeline for completing the replacement course and, in June 2019, FEMA officials said they had begun procuring external consulting services to redevelop it. According to the officials, ODIC had evaluated alternatives to the suspended course and determined that an in-person, exercise-based course with remote participation capabilities would be an appropriate replacement. FEMA officials said the course will take about 1 year to develop and will be ready to field by August 2020. In conclusion, FEMA has taken a number of steps toward addressing our recommendations related to how it supports individuals with disabilities in obtaining disaster assistance. ODIC’s draft strategic plan for 2019-2022, which articulates objectives for the new approach to disability integration, is likely to help facilitate consistent implementation agency-wide. In addition, we are hopeful that FEMA’s revised registration-intake questions, as well as data sharing guidance for nonfederal partners, will help FEMA and its partners better identify and assist registrants with disabilities. However, we continue to believe that implementing changes to disability integration before staff have been fully trained may leave FEMA staff ill-prepared to identify and address the challenges that individuals with disabilities face while recovering from disasters. We will continue to monitor FEMA’s actions as it makes additional progress toward addressing our recommendations. Chairman Payne, Ranking Member King, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Elizabeth Curda, Director, Education, Workforce, and Income Security Issues at (202) 512-7215 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Sara Schibanoff Kelly (Assistant Director), Sara Pelton (Analyst-in-Charge), and David Reed. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Three sequential hurricanes—Harvey, Irma, and Maria—affected more than 28 million people in 2017, according to FEMA. Hurricane survivors aged 65 and older and those with disabilities faced particular challenges evacuating to safe shelter, accessing medicine, and obtaining recovery assistance. In June 2018, FEMA began implementing a new approach to assist individuals with disabilities. This statement describes (1) reported challenges faced by these individuals in accessing disaster assistance from FEMA and its nonfederal partners following the 2017 hurricanes; and (2) the extent to which FEMA has implemented changes in how it supports these individuals. This statement is based on a May 2019 GAO report and selected updates. For the report, GAO analyzed FEMA documents and data from FEMA call centers and also visited 2017 hurricane locations to interview state, territorial, and local officials. GAO also interviewed FEMA officials from headquarters and deployed to each disaster location. To update FEMA's progress toward addressing its recommendations, GAO interviewed FEMA officials and analyzed agency documents. GAO's May 2019 report found that some individuals who are older or have disabilities may have faced challenges registering for and receiving assistance from the Federal Emergency Management Agency (FEMA) and its nonfederal partners (such as state, territorial, and local emergency managers). FEMA's registration did not include an initial question that directly asks individuals if they have a disability or if they would like to request an accommodation. GAO recommended that FEMA use new registration-intake questions to improve the agency's ability to identify and address individuals' disability-related needs. FEMA concurred and, in May 2019, updated the questions to directly ask individuals if they have a disability. GAO found that the substantial damage caused by the 2017 hurricanes prevented or slowed some individuals with disabilities from obtaining food, water, and other critical goods and services from states, territories, and localities. Officials from one state reported that few public transportation services, including paratransit, were functional following the 2017 hurricane affecting the state. The officials said this may have prevented people with disabilities from maintaining their health and wellness—such as by shopping for groceries or going to medical appointments—after the storm. GAO's May 2019 report also found that FEMA had taken limited steps to implement the agency's new approach to assist individuals with disabilities. GAO recommended the agency establish and disseminate objectives for implementing its new approach. FEMA concurred, and developed a draft strategic plan that includes strategic goals and objectives for the new approach, which the agency plans to finalize and disseminate in 2019. GAO recommended that FEMA, as part of its new approach, develop a plan for delivering training to all FEMA staff deployed during disasters that promotes competency in disability awareness. In concurring with this recommendation, FEMA described its plan to incorporate a disability awareness competency into the job requirements for all deployable staff, but has not yet developed a plan for training. GAO's May 2019 report also recommended that FEMA develop a timeline for completing the development of training on incorporating the needs of individuals with disabilities into emergency planning, which it planned to offer to its nonfederal partners. FEMA concurred with GAO's recommendation and, in June 2019, officials began procuring external consulting services to develop a replacement course. According to officials, the course will take about 1 year to develop and will be ready to field by August 2020. In the May 2019 report, GAO made seven recommendations to FEMA; FEMA concurred with six. FEMA has established new registration questions and a timeline to offer training to its partners. GAO continues to believe its recommendations to develop a plan to train its staff on disability awareness, among other actions, are valid.", "document_type": "gao"}
{"report": "In the United States, the roles and responsibilities related to preparing for, assessing, and responding to communicable disease threats in the civil aviation system require immense coordination among a number of federal agencies and aviation stakeholders. Each federal agency has a different mission, which affects its responsibilities for protecting against communicable disease threats. The DHS and HHS are the lead agencies for responding to a communicable disease threat. They focus on protecting our borders at ports of entry, including airports, from threats from abroad and protecting the nation from domestic and foreign health, safety, and security threats, respectively. FAA is responsible for civil aviation and commercial space transportation flight safety in the United States and the safe and efficient movement of air traffic in the national airspace system, as well as for the safety of U.S. airlines, other U.S. operators, and FAA-certificated aircrews worldwide. As part of this responsibility, FAA regulates and certificates airports, airlines, and airmen and provides guidance. In the case of a communicable disease threat, numerous federal, state, and local entities may be called upon to respond, depending on their legal authority and whether the threat is identified before, during, or after the flight. For example, before boarding, HHS and DHS may identify travelers who are not allowed travel, based on public health threats. The CDC can prohibit the introduction of nonresident foreign nationals into the United States from designated countries or places, but only for such time as the CDC deems necessary for public health. During a flight, CDC regulations require pilots to immediately report to CDC any deaths or the occurrence of any travelers with signs or symptoms that may indicate a communicable disease infection during international flights coming to the United States. And, once an aircraft with a suspected ill passenger approaches an airport, federal or local public health officials, first responders (e.g., fire or emergency medical technicians), airport authorities, air traffic control personnel, or a combination of these stakeholders may make decisions about and lead certain aspects of the response based on the situation and available response protocols or preparedness plans. In addition, some response-related roles and responsibilities are established in law or by interagency agreements, and others may be defined in FAA-required airport-emergency plans, although those plans are not required to address communicable disease threats. In addition, FAA supports and coordinates a range of R&D activities for the civil aviation system. The inventory of FAA’s R&D activities is expressed in the National Aviation Research Plan (NARP) and in FAA’s Fiscal Year R&D Annual Review. FAA is required to submit both of these documents annually to Congress. According to FAA’s most recent NARP, FAA’s research budget from all accounts in FY 2017 was $422.3 million. FAA’s research budget supports activities conducted by FAA as well as a range of partners, including other government agencies, universities, and private sector organizations. FAA’s process for developing its commercial aviation research portfolio spans the agency. To develop the NARP and its R&D portfolio, FAA’s program planning teams, which focus on specific research program areas, identify R&D projects to meet one of DOT’s three strategic goals and FAA’s five R&D goals. Further, an executive board in FAA provides guidance and oversight over the agency’s portfolio development process, and a statutorily created advisory committee—consisting of individuals that represent corporations, universities, associations, and others— conducts external reviews of FAA’s R&D programs for relevance, quality, and performance. This advisory committee also makes recommendations to FAA on the proposed R&D portfolios and budgets. In 2015, we found that the United States lacked a comprehensive national aviation-preparedness plan to limit the spread of communicable diseases through air travel, though some individual airport and airline preparedness plans did exist. Accordingly, we recommended that DOT work with relevant stakeholders, such as HHS, to develop a national aviation- preparedness plan for communicable disease outbreaks. We emphasized that a comprehensive national plan would provide a coordination mechanism for the public-health and aviation sectors to more effectively prevent and control a communicable disease threat while also minimizing unnecessary disruptions to the national aviation system. Additionally, U.S. airports and airlines are not required to have individual preparedness plans for communicable disease threats and no federal agency tracks which airports and airlines have them. As such, the extent to which U.S. airports and airlines have such plans is unknown. However, all 14 airports and 3 airlines we reviewed in 2015 had independently developed preparedness plans for responding to communicable disease threats from abroad. These plans generally addressed the high-level components that we identified as common among applicable federal and international guidance for emergency preparedness, such as establishment of an incident command center and activation triggers for a response. While the 14 airports and 3 airlines had plans that address communicable diseases, representatives from these airports and airlines reported facing multiple challenges in responding to threats. Identified challenges that included obtaining guidance; communication and coordination among responders; and assuring employees have appropriate training, equipment, and sanitary workplaces. As we stated in our 2015 report, a national aviation preparedness plan to respond to communicable disease outbreaks could help address these challenges. As of June 2020, DOT, DHS, and HHS stated that the federal government still has not developed a national aviation-preparedness plan to respond to communicable disease outbreaks. In making our recommendation in 2015, we pointed to Annex 9 to the Chicago Convention—an international aviation treaty to which the United States is a signatory—which contains a standard that obligates International Civil Aviation Organization (ICAO) member states to develop a national aviation-preparedness plan for communicable disease outbreaks. DOT and CDC officials in 2015 stated that some elements of a national aviation-preparedness plan already exist, including plans at individual airports. However, as we discussed in our 2015 report, individual airport plans are often contained in multiple documents, and FAA reported that the plans are intended to handle communicable disease threats posed by passengers on one or two flights, rather than an epidemic—which may require involvement from multiple airports on a national level. Most importantly, a national aviation- preparedness plan would provide airports and airlines with an adaptable and scalable framework with which to align their individual plans, to help ensure that individual airport and airline plans work in concert with one another. DOT and CDC officials agreed in 2015 and continue to agree today that a national aviation-preparedness plan could add value. DOT, however, maintains that those agencies that have both legal authority and expertise for emergency response and public health—namely DHS and HHS—are best positioned to take the lead role in developing such a plan within the existing interagency framework for national-level all-hazards emergency preparedness planning. We continue to believe that DOT would be in the best position to lead the effort because FAA and DOT have stronger and deeper ties to, as well as oversight responsibility for, the relevant stakeholders that would be most involved in such a broad effort, namely airlines, airports, and other aviation stakeholders. In addition, DOT’s Office of the Secretary is the liaison to ICAO for Annex 9 to the Chicago Convention, in which the relevant ICAO standard is contained. In response to the current COVID-19 pandemic and in the absence of a national aviation-preparedness plan, DOT officials pointed to ongoing efforts to engage with interagency partners at DHS and HHS, as well as industry stakeholders, to better collaborate on the aviation sector’s communicable disease response and preparedness. For example, DOT told us that it has facilitated conference calls between federal and private sector stakeholders and has collaborated with CDC to update interim guidance for airline crews related to communicable diseases, specifically COVID-19. While these actions are helpful, some aviation stakeholders have publicly highlighted piecemeal response efforts that may have led to some of the confusion among stakeholders and chaos at certain airports that occurred earlier this year following the COVID-19 travel bans and increased screening efforts. For example, stakeholders described actions taken by individual airlines in the absence of FAA guidance, such as to cease operations to certain countries, and a piecemeal approach to establishing standards for safely continuing or expanding service, such as various airline and airport policies regarding facemasks. This piecemeal approach points to the continued need for DOT to implement our 2015 recommendation to develop a coordinated effort to plan for and respond to communicable disease threats. We have included this open recommendation as one of 16 high priority recommendations to DOT. While a national aviation-preparedness plan can help better manage the response to the next aviation pandemic, other efforts such as research and development are also key. In 2017, we found that FAA’s actions related to the management of its R&D portfolio were not fully consistent with statutory requirements, agency guidance, and leading practices. As part of that work, we assessed FAA’s actions to manage its R&D portfolio in three key areas: (1) developing its portfolio of R&D projects, (2) tracking and evaluating those projects, and (3) reporting on its portfolio. We found that FAA could be more strategic in how it develops its R&D portfolio, chiefly in identifying long-term research needs and in improving disclosure of how projects are selected. As a result of these deficiencies, we found that FAA management could not be assured that the highest priority R&D was being conducted. We also found that while FAA tracks and evaluates its research projects consistent with leading practices, it did not fully address all statutory reporting requirements, such as identifying long-term research resources in the National Aviation Research Plan (NARP) or preparing the R&D Annual Review in accordance with government performance-reporting requirements. These reporting deficiencies can limit the usefulness of the reports to internal and outside stakeholders. Accordingly, in 2017, we recommended that DOT direct FAA to (1) take a more strategic approach to identifying long- term R&D research priorities across the agency, (2) disclose how research projects are prioritized and selected, and (3) ensure that the NARP and R&D Annual Reviews meet statutory requirements for content. DOT agreed with all three recommendations. As of June 2020, FAA has fully addressed one of our recommendations and taken partial action on two other recommendations. Specifically, FAA fully responded to our recommendation that FAA disclose the process it uses for prioritizing and selecting research projects by updating in 2018 its internal guidance documents to allow better transparency over project selection. In partially responding to our recommendation to take a more strategic approach to identifying research priorities across the agency, in June 2019, FAA issued a redesigned National Aviation Research Plan (NARP) for 2017-2018. The redesigned plan is a good first step. Also as part of an effort to be more strategic, FAA is beginning to take actions to understand emerging aviation issues requiring FAA’s research attention. This recommendation has not been fully addressed as, according to FAA officials, the agency is still developing guidance to ensure that future NARPs take a strategic approach and incorporate emerging issues into future plans. FAA officials told us they plan to finalize the guidance by the end of 2020. Similarly, with respect to our recommendation aimed at achieving compliance with statutory reporting requirements, the redesigned 2017-2018 NARP included a list of agreements with federal and nonfederal entities on research activities, resource allocation decisions, and a description of technology transfer to government, industry, and academia, among other items. Officials told us that they are finalizing the 2019 R&D Annual Review, which has been redesigned to address other statutory reporting requirements, and will develop guidance to ensure that future documents meet those requirements. FAA has sponsored limited federal research into disease transmission onboard aircraft and in airports. FAA’s research goals focus on areas like improving airport operations and air space management, and developing new technologies, which FAA has aligned to DOT’s strategic goals related to safety, infrastructure, and innovation. Based on our prior work and interviews with FAA officials, we found that FAA’s research in cabin safety for crew and passengers does not focus on disease transmission. For example, according to FAA officials, as of June 2020, ongoing research that most closely relates to disease contamination is research related to monitoring the quality of “bleed air,” which is outside air that is drawn through jet engines into an aircraft cabin. FAA officials said that its Civil Aerospace Medical Institute is participating in this research. Even so, FAA has funded some programs that are relevant to mitigating communicable disease transmission at airports and on aircraft. For example, in 2015 the Transportation Research Board’s Airports Cooperative Research Program (ACRP), which is funded by FAA’s Airport Improvement Program (AIP), decided to hold a series of workshops on topics that are of significance to airports and that are not being addressed by other federal research programs. The decision to hold the first ACRP workshop on communicable disease occurred toward the end of the Ebola virus outbreak. ACRP has also issued reports on reducing communicable disease transmission at airports and aircraft. These reports have provided information and guidance to airports and airlines on infectious disease mitigation onboard aircraft and ways to respond to a communicable disease in airports. For example, a 2013 ACRP report recommends reducing the amount of time aircraft ventilation systems are shutdown at the gate, so that an aircraft’s high efficiency particulate air (HEPA) systems, which can capture more than 99 percent of the airborne microbes, continue to operate. ACRP also has a research project currently under way for publication early next year on effective collaboration to prevent, respond to, and mitigate disease threats. Prior to 2014, FAA also funded some research on disease transmission on aircraft through its Centers of Excellence research consortium. Specifically, in 2004, FAA established the Airliner Cabin Environment Research (ACER) Center of Excellence, which conducts research on, among other things, the safety and health of passengers and crew inside the cabin. In 2010 and 2012, ACER published research on air quality in airline cabins and disease transmission in aircraft. A researcher we interviewed who is affiliated with ACER said that the Center established a laboratory in 2006, called ACERL, which is currently conducting research on the dispersion of airborne particles (including viruses) in the aircraft cabin for CDC’s National Institute of Occupational Safety and Health. As of 2014, ACER began operating independently as a consortium academia, government, and others and is no longer being funded solely by FAA. FAA and DOT principally look to HHS and the CDC for guidance on passenger health issues. HHS has statutory responsibility for preventing the introduction, transmission, and spread of communicable diseases into the United States and among the states. Within HHS, CDC has defined its mission as protecting America from health, safety and security threats, both foreign and domestic. CDC alerts travelers about disease outbreaks and steps they can take to protect themselves. CDC also has the authority to quarantine passengers traveling from foreign countries, if necessary, to prevent the introduction, transmission, or spread of communicable disease. CDC’s National Institute for Occupational Safety and Health has conducted research and issued guidance in the past on disease transmission in aircraft and cabin crew health and, as previously noted, is funding current research through the ACER Center. CDC has also issued COVID-19 guidance for cabin crew safety. There are a variety of technologies that could help address infectious disease transmission associated with air travel, but these technologies are at various stages of maturity. For example, the initial screening of passengers for fevers is typically done with handheld infrared thermometers and has been reportedly discussed for use by Transportation Security Agents. Reports also state that the mass screening of crowds using thermal cameras has been used in some airports in Asia, but such scanners are still being tested for standalone use in the United States, with some concerns reported about the accuracy of the results. Aircraft disinfection has traditionally been done by cleaning crews, but a number of methods are being developed using heat, chemicals, and UV light, and are under examination by researchers. Chairwoman Horn, Ranking Member Babin, and Members of the Subcommittee, this completes my prepared remarks. I would be pleased to respond to any questions that you or other Members of the Subcommittee may have at this time. If you or your staff have any questions about this statement, please contact me at (202) 512-2834 or krauseh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Jonathan Carver, Assistant Director; Paul Aussendorf; Roshni Davé; Hayden Huang; Delwen Jones; Molly Laster; Cheryl Peterson; Gretchen Snoey; and Elizabeth Wood. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The transmission of COVID-19 has been greatly aided by air travel. In light of the pandemic and warnings about the risks of air travel, U.S. passenger airline traffic fell by 96 percent in April 2020 as compared to April 2019. COVID-19 is only the latest communicable disease threat to raise public health concerns regarding the spread of contagion through air travel. Ensuring that the United States is prepared to respond to disease threats from air travel, as well as conducting the necessary research to reduce the risks of contagion, are two vital responsibilities of the federal government. This statement provides information on (1) the U.S. aviation system's preparedness to respond to communicable disease threats and (2) FAA's management of its R&D portfolio, including the extent to which disease transmission on aircraft and at airports has been the focus of FAA research. This statement is based on GAO-16-127 issued in December 2015 and GAO-17-372 issued in April 2017. GAO conducted updates to obtain information on the actions agencies have taken to address these reports' recommendations. The United States still lacks a comprehensive plan for national aviation preparedness to limit the spread of communicable diseases through air travel. In December 2015 during the Ebola epidemic, GAO recommended that the Department of Transportation (DOT) work with relevant stakeholders, such as the Department of Health and Human Services (HHS), to develop a national aviation-preparedness plan for communicable disease outbreaks. GAO concluded that the absence of a national plan undermined the ability of the public-health and aviation sectors to coordinate on a response or to provide consistent guidance to airlines and airports. Moreover, Annex 9 to an international aviation treaty to which the United States is a signatory contains a standard that obligates member states to develop such a plan. DOT is now confronting an even more widespread public health crisis—the Coronavirus Disease (COVID-19) global pandemic—without having taken steps to implement this recommendation. Not only could such a plan provide a mechanism for the public-health and aviation sectors to coordinate to more effectively prevent and control a communicable disease threat, it could also help minimize unnecessary disruptions to the national aviation system, disruptions that to date have been significant. Some aviation stakeholders have publicly highlighted the resulting piecemeal approach to adopting standards during the response to COVID-19, such as various airline and airport policies regarding facemasks, as demonstrating the need for a more coordinated response. The existence of a national plan might have reduced some of the confusion among aviation stakeholders and passengers. While DOT agrees that a national aviation preparedness plan is needed, the agency continues to suggest that HHS and the Department of Homeland Security have responsibility for communicable disease response and preparedness planning. GAO continues to believe that DOT is in the best position to lead this effort given its oversight responsibilities and ties with relevant aviation stakeholders. The Federal Aviation Administration (FAA) has sponsored limited federal research into disease transmission onboard aircraft and in airports. FAA's research goals focus on areas like improving airport operations and air space management, and developing new technologies, which FAA has aligned to DOT's strategic goals related to safety, infrastructure, and innovation. Based on prior work and interviews with FAA officials, GAO found that FAA's research in cabin safety for crew and passengers does not focus on disease transmission. For example, according to FAA officials, ongoing research that most closely relates to disease contamination is research related to monitoring the quality of “bleed air,” which is outside air that is drawn through jet engines into an aircraft cabin. In 2017, GAO found that FAA could be more strategic in how it develops its research and development (R&D) portfolio, chiefly in identifying long-term research needs and explaining how FAA selects projects. Of the three recommendations GAO made in that report to improve FAA's management of its R&D portfolio, FAA fully addressed one, issuing guidance in 2018 on prioritizing and selecting R&D projects. While FAA has made some progress addressing GAO's recommendations on research portfolio development and reporting, further attention to these recommendations could help ensure that FAA strategically identifies research priorities across the agency. GAO made several recommendations in its prior work, including that DOT develop a comprehensive national aviation-preparedness plan, and that FAA identify long-term R&D priorities, among other things. Progress has been made in addressing some of the recommendations. Continued attention is needed to ensure that the remainder of these recommendations are addressed.", "document_type": "gao"}
{"report": "Our analysis of FHA data found that 272,155 HECMs terminated from fiscal years 2014 through 2018. The number of terminations rose from about 24,000 in fiscal year 2014 to a peak of roughly 82,000 in fiscal year 2016, before declining to about 60,000 in fiscal year 2018. In recent years, a growing percentage of HECMs have terminated because borrowers defaulted on their loans. While death of the borrower is the most commonly reported reason why HECMs terminated, the percentage of terminations due to defaults increased from 2 percent in fiscal year 2014 to 18 percent in fiscal year 2018 (see fig. 2). Most defaults were due to borrowers not meeting occupancy requirements or failing to pay property charges. For about 30 percent of terminations, we were unable to readily determine a termination reason from FHA’s data. We also found that servicers’ use of foreclosure prevention options for HECM borrowers was limited or FHA did not have readily available data to assess the extent of use. For example, since 2015, FHA has allowed HECM servicers to offer borrowers who are behind on property charges repayment plans to help prevent foreclosures, but as of the end of fiscal year 2018, only about 22 percent of these borrowers had received this option. Also, while FHA created a low-balance extension in 2016—which allows HECM servicers to delay calling a HECM due and payable if the borrower owes less than $2,000 in unpaid property taxes or hazard insurance—FHA officials told us they do not track how often servicers use this option. Our analysis of FHA data found that approximately 8,800 HECMs that terminated in fiscal years 2014 through 2018 had unpaid property charges of less than $2,000 at the time of termination. Some of these HECMs may have been eligible for a low-balance extension when they terminated. Additionally, we found that it is difficult to estimate the universe of HECMs potentially eligible for mortgagee optional election assignments—an option to help nonborrowing spouses stay in their homes after a borrowing spouse dies. Under this option, if required conditions and time frames are met, the servicer can assign the HECM to FHA. The assignment defers repayment of the HECM as long as the nonborrowing spouse fulfills certain conditions. According to information generated by FHA, HECM servicers submitted 1,445 requests for mortgagee optional election assignments from June 2015 (when FHA made this option available) through September 2018. In total, FHA approved roughly 70 percent of the requests and denied the remaining 30 percent. However, nonborrowing spouses were not listed on loan documentation for HECMs originated prior to August 4, 2014. As a result, FHA does not know how many eligible nonborrowing spouses could have, but did not, apply for the mortgagee optional election assignment, or how many are potentially eligible to apply for it in the future. FHA has begun reaching out to HECM borrowers to inform them of the mortgagee optional election process and ask them to self-identify whether there is a nonborrowing spouse associated with their loan. FHA’s monitoring, performance assessment, and reporting for the HECM program have weaknesses. Since fiscal year 2013, FHA has used the Home Equity Reverse Mortgage Information Technology (HERMIT) system to collect data on the servicing of HECMs, but the system does not contain comprehensive and accurate data about the reasons why HECMs terminate, a key servicing event. According to the HERMIT User Guide, servicers should provide a reason in HERMIT when they terminate a HECM. However, as noted previously, for about 30 percent of the HECMs that terminated in fiscal years 2014 through 2018, we were unable to determine the reason for termination. FHA officials told us termination reasons are available on an individual loan basis in HERMIT but not in an extractable form. FHA does not regularly track and report on HECM termination reasons, due partly to this system limitation. In the report being released today, we are recommending that FHA take steps to improve the quality and accuracy of HECM termination data. These steps may include updating the termination reasons in the HERMIT system for recording these data or updating the HERMIT User Guide to more clearly instruct servicers how to record termination reasons. FHA agreed with this recommendation. Comprehensive and accurate data on HECM terminations would provide FHA with a better understanding of loan outcomes—information FHA and Congress need in order to know how well the program is helping seniors age in place. FHA also has not established comprehensive performance indicators for the HECM portfolio and has not regularly tracked key performance metrics, such as the percentage of HECM terminations due to borrower defaults, the proportion of active HECMs with delinquent property charges, or the percentage of distressed borrowers who have received foreclosure prevention options. For example, HUD’s most recent strategic plan and corresponding performance report do not include HECM-specific performance indicators, and the last comprehensive evaluation of the HECM program was done in 2000. FHA officials told us they were in the planning phase for a new evaluation of the program but had not set a start date and did not expect the evaluation to include an analysis of the reasons for HECM terminations or the use of foreclosure prevention options for borrowers in default. We are recommending that FHA establish, periodically review, and report on performance indicators for the HECM program and examine the impact of foreclosure prevention options in the forthcoming HECM program evaluation. FHA agreed with this recommendation. Better performance assessment could provide FHA important information about how well the HECM program is working. Additionally, we found shortcomings in FHA’s internal reporting and analysis for the HECM program. For example, FHA has not developed internal reports to comprehensively monitor patterns and trends in loan outcomes, such as the percentage of HECM terminations due to borrower defaults. FHA has generated some reports from HERMIT to help oversee the HECM portfolio, but it has been slow to develop regular and comprehensive reporting mechanisms. FHA officials told us that while data on defaults and foreclosure prevention options have generally been available in HERMIT since 2015, FHA was unable to obtain reports on these topics until 2018 because of funding limitations with their HERMIT system contractor. Our review of the regular and ad hoc reports FHA has received from its HERMIT system contractor found that many are lists of loans that meet criteria and do not provide summary statistics that could be used to readily identify patterns or trends in metrics. Further, we found the reports required additional analysis to generate meaningful management information. In the report being released today, we recommend that FHA develop analytic tools, such as dashboards or watch lists, to better monitor outcomes for the HECM portfolio, such as reasons for terminations, defaults, use of foreclosure prevention options, or advances paid by servicers on behalf of HECM borrowers. FHA agreed with this recommendation. With more robust program analysis and internal reporting, FHA would be better positioned to detect and respond to emerging issues and trends in the HECM portfolio. Finally, we found that FHA has not fully analyzed the implications of how it prioritizes foreclosures for HECMs that servicers have assigned to FHA. FHA officials told us the agency generally does not foreclose on borrowers whose HECMs have been assigned to FHA and who are in default due to unpaid property charges. As a result, defaulted borrowers whose loans have not been assigned to FHA face a greater risk of foreclosure than defaulted borrowers with FHA-assigned loans. In addition, FHA’s process may create a financial incentive for HECM borrowers with assigned loans to not pay their property charges. Therefore, we are recommending that FHA analyze the implications of its prioritization process. FHA agreed with our recommendation. Such analysis would help FHA to better understand how its process for prioritizing foreclosures for assigned loans affects the HECM portfolio, HECM borrowers, neighborhoods, and FHA’s insurance fund. FHA’s oversight of HECM servicers has been limited in recent years. FHA has not performed comprehensive on-site reviews of HECM servicers’ compliance with program requirements since fiscal year 2013 and does not have current procedures for conducting these reviews. FHA officials said they planned to resume the HECM servicer reviews in fiscal year 2020, starting with three servicers that account for most of the market. However, as of August 2019, FHA had not developed updated review procedures (they were last updated in 2009) and did not have a risk- based method for prioritizing reviews. In the report being released today, we recommend that FHA develop and implement procedures for conducting on-site reviews of HECM servicers, including a risk-rating system for prioritizing and determining the frequency of reviews. FHA agreed with this recommendation. By resuming HECM servicer on-site reviews and adopting a risk-rating system, FHA would be better positioned to ensure that servicers are following program requirements, including those designed to help protect borrowers. Additionally, we found that while CFPB has examined reverse mortgage servicers and plans to continue doing so, according to CFPB officials the bureau does not share results with FHA because the agencies do not have an agreement in place to share supervisory information. CFPB officials said CFPB and FHA had taken initial steps in 2017 toward developing an information-sharing agreement. However, as of August 2019, an information-sharing agreement had not been completed. Accordingly, we are recommending that FHA and CFPB work together to complete an agreement for sharing the results of CFPB’s examinations of HECM servicers with FHA. CFPB generally agreed with this recommendation, and FHA neither agreed nor disagreed. Sharing these results could aid FHA’s oversight of HECM servicers by providing additional information about the servicers’ performance and operations. CFPB began collecting reverse mortgage consumer complaints in December 2011 and has collected about 3,600 complaints since then. CFPB officials told us they use consumer complaints as part of their criteria for selecting entities to examine, including reverse mortgage servicers, and to inform CFPB’s educational publications. We conducted a detailed analysis of a random, generalizable sample of 100 consumer complaint narratives drawn from all the reverse mortgage complaints CFPB received in calendar years 2015 through 2018. Based on our review of complaint narratives, we found that some of the issues consumers cited most commonly were foreclosures, poor communication from lenders or servicers, problems at loan origination, estate management, and unfair interest rates, fees, or costs. FHA collects and records inquiries and complaints about HECMs, and it has access to CFPB data on reverse mortgage complaints. However, FHA does not use its inquiry and complaint data to help inform HECM program policies and oversight, and the way data are collected does not produce quality information for these purposes. Additionally, we found that FHA has not leveraged CFPB complaint data for HECM program oversight. According to FHA officials, FHA’s two main methods for collecting customer inquiries and complaints are hotlines operated by the agency’s National Servicing Center and the FHA Resource Center. From calendar years 2015 through 2018, the National Servicing Center received about 105,000 HECM-related calls. During this same period, the FHA Resource Center received 147 HECM-related calls. In April 2019, the FHA Resource Center became the primary entity for collecting, recording, and responding to all HECM-related calls. FHA officials told us they transferred these responsibilities from the National Servicing Center to the FHA Resource Center to help improve call management. While this change could help improve customer service, it does not fully resolve limitations we found in FHA’s approach to collecting and recording HECM inquiries and complaints that diminish the usefulness of the information for program oversight. For example, both the National Servicing Center and the FHA Resource Center do not collect call information in a way that would allow FHA to readily analyze the data for themes. Specifically, both centers do not reliably differentiate between inquiries and complaints—a potentially important distinction for determining appropriate agency-level responses. Additionally, while both the centers collect data on the reason for calls, neither does so in a systematic way that would allow FHA to readily determine how frequently issues are being raised. For example, neither center’s data systems contain standardized categories or menus with options for recording reasons for calls. FHA officials said the agency uses complaint and inquiry data to improve customer service. However, FHA does not analyze data for other purposes that could enhance program oversight, such as determining which HECM servicers and lenders receive the most complaints, targeting entities for on-site reviews, or identifying topics that may need additional borrower education. In the report being released today, we recommend that FHA collect and record consumer inquiries and complaints in a manner that facilitates analysis of the type and frequency of the issues raised. FHA neither agreed nor disagreed with our recommendation. We also recommend that FHA periodically analyze available internal and external consumer complaint data about reverse mortgages to help inform management and oversight of the HECM program. FHA agreed with this recommendation. By improving the collection and use of consumer complaint data and better monitoring its own and CFPB’s complaint data, FHA could improve its ability to detect and respond to emerging consumer protection issues regarding HECMs. Chairman Clay, Ranking Member Duffy, and Members of the Subcommittee, this completes my statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Alicia Puente Cackley, Director, Financial Markets and Community Investment at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Beth Faraguna and Steve Westley (Assistant Directors), Holly Hobbs (Analyst in Charge), Steven Campbell, William Chatlos, John Karikari, Matthew Levie, Marc Molino, Jennifer Schwartz, and Tyler Spunaugle. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "This testimony summarizes the information contained in GAO's September 2019 report, entitled Reverse Mortgages: FHA Needs to Improve Monitoring and Oversight of Loan Outcomes and Servicing ( GAO-19-702 ). The vast majority of reverse mortgages are made under the Federal Housing Administration's (FHA) Home Equity Conversion Mortgage (HECM) program. In recent years, a growing percentage of HECMs insured by FHA have ended because borrowers defaulted on their loans. While death of the borrower is the most commonly reported reason why HECMs terminate, the percentage of terminations due to borrower defaults increased from 2 percent in fiscal year 2014 to 18 percent in fiscal year 2018 (see figure). Most HECM defaults are due to borrowers not meeting occupancy requirements or failing to pay property charges, such as property taxes or homeowners insurance. Since 2015, FHA has allowed HECM servicers to put borrowers who are behind on property charges onto repayment plans to help prevent foreclosures, but as of fiscal year-end 2018, only about 22 percent of these borrowers had received this option. FHA's monitoring, performance assessment, and reporting for the HECM program have weaknesses. FHA loan data do not currently capture the reason for about 30 percent of HECM terminations (see figure). FHA also has not established comprehensive performance indicators for the HECM portfolio and has not regularly tracked key performance metrics, such as reasons for HECM terminations and the number of distressed borrowers who have received foreclosure prevention options. Additionally, FHA has not developed internal reports to comprehensively monitor patterns and trends in loan outcomes. As a result, FHA does not know how well the HECM program is serving its purpose of helping meet the financial needs of elderly homeowners. FHA has not conducted on-site reviews of HECM servicers since fiscal year 2013 and has not benefited from oversight efforts by the Consumer Financial Protection Bureau (CFPB). FHA officials said they planned to resume the reviews in fiscal year 2020, starting with three servicers that account for most of the market. However, as of August 2019, FHA had not developed updated review procedures and did not have a risk-based method for prioritizing reviews. CFPB conducts examinations of reverse mortgage servicers but does not provide the results to FHA because the agencies do not have an agreement for sharing confidential supervisory information. Without better oversight and information sharing, FHA lacks assurance that servicers are following requirements, including those designed to help protect borrowers.", "document_type": "gao"}
{"report": "DHS’s efforts to strengthen and integrate its acquisition, IT, financial, and human capital management functions have resulted in the department meeting 3 out of 5 criteria for removal from the High-Risk List—leadership commitment, action planning, and monitoring progress. DHS has partially met the remaining two criteria—capacity and demonstrated, sustained progress, as shown in figure 1. With regard to leadership commitment, DHS’s top leadership, including the Secretary and Deputy Secretary of Homeland Security, has continued to demonstrate commitment and support for addressing the department’s management challenges. They have also taken actions to institutionalize this commitment to help ensure the long-term success of the department’s efforts. One such effort is the Under Secretary for Management’s Integrated Priorities initiative to strengthen the integration of DHS’s business operations across the department. During monthly leadership meetings with the Under Secretary for Management, the department’s Chief Executive Officers have been providing status updates on their respective actions to address this high-risk designation. Furthermore, top DHS leaders, such as the Under Secretary for Management and the department’s Chief Executive Officers, routinely meet with GAO management to discuss progress on high-risk areas. With regard to having an action plan and monitoring effectiveness, in January 2011, DHS produced its first Integrated Strategy for High-Risk Management and has issued 14 updated versions, most recently in September 2018. The September 2018 strategy describes DHS’s progress to-date, planned corrective actions to further strengthen its management functions, and includes performance measures to monitor key management initiatives. DHS’s Management Directorate leads this ongoing effort and DHS’s strategy and approach, if effectively implemented and sustained, provides a path for DHS to be removed from our High-Risk List. DHS has partially met the criteria for capacity but needs to make additional progress identifying and allocating resources in certain areas— namely acquisition, IT, and financial management—to fully demonstrate its capacity. DHS has analyzed components’ acquisition program staffing assessments but has yet to conduct an in-depth analysis across components or develop a plan to address any gaps. With regard to IT staffing, DHS has not fully identified or reported to Congress or the Office of Personnel Management (OPM) on its department-wide cybersecurity specialty areas of critical needs, such as cybersecurity management or incident response, as required by law. Additionally, DHS’s financial statement auditor has identified several capacity-related issues, including resource limitations and inadequate management and staff training, as causes for the material weaknesses reported. The final criterion is demonstrated progress, which remains partially met. In 2010, we identified, and DHS agreed, that achieving 30 specific outcomes in the areas of acquisition management, IT management, financial management, human capital management, and management integration would be critical to addressing the department’s management challenges. As such, these 30 outcomes became the key criteria by which we gauge DHS’s demonstrated progress. We reported in March 2019 that DHS has fully addressed 17 of the 30 needed outcomes, mostly addressed four, partially addressed six, and initiated actions to address the remaining three, as shown in table 1. In the last 2 years, DHS has made particular progress in the areas of human capital and IT management. Specifically, since 2017 DHS has taken steps to fully address 4 outcomes. The department fully addressed two key human capital outcomes by (1) demonstrating that components are basing hiring decisions and promotions on human capital competencies and (2) strengthening employee engagement efforts. In addition, in the last 2 years DHS has fully addressed two IT outcomes by (1) providing ongoing oversight and support to troubled IT investments to help improve their cost, schedule, and performance; and (2) demonstrating significant progress in implementing its IT strategic workforce planning initiative. Important progress and remaining work in all of the five key areas include: Acquisition management. DHS continues to face challenges in funding its acquisition portfolio. In May 2018, we found that recent enhancements to DHS’s acquisition management, resource allocation, and requirements policies largely reflect key portfolio management practices. However, we also found that of the 24 major acquisition programs we assessed with approved schedule and cost goals, 10 were on track to meet those goals during 2017—a decrease from 2016. In addition, we found that DHS’s portfolio of major acquisition programs was not affordable from fiscal years 2018 to 2022 because the planned costs exceeded the planned budget. DHS has taken steps to strengthen acquisition requirements development across the department, such as reestablishing the Joint Requirements Council in June 2014 to review and validate DHS acquisition requirements. However, opportunities remain to further strengthen DHS’s acquisition process by, for example, using the Joint Requirements Council to (1) identify overlapping or common requirements and (2) make recommendations to senior leadership to help ensure that DHS uses its finite investment resources wisely and maintains a balanced portfolio of investments that combine near-term operational improvements with long-term strategic planning. IT management. DHS has updated its approach for managing its portfolios of IT investments across all components. As part of the revised approach, the department is using its capital planning and investment control process and the Joint Requirements Council to assess IT investments across the department on an ongoing basis. For example, as part of its capital planning process for the fiscal year 2020 budget, the Office of the Chief Information Officer worked with the components to assess each major IT investment to ensure alignment with DHS’s functional portfolios, and to identify opportunities to share capabilities across components. This updated approach should enable DHS to identify potentially duplicative investments and opportunities for consolidating investments, as well as reduce component-specific investments. Additionally, DHS has continued to take steps to enhance its information security program. In November 2018, the department’s financial statement auditor reported that DHS had made progress in correcting its prior year IT security weaknesses. However, for the 15th consecutive year, the auditor designated deficiencies in IT systems controls as a material weakness for financial reporting purposes. Work also remains in implementing our six open recommendations concerning DHS’s cybersecurity workforce assessment requirements. DHS also faces challenges in fulfilling its pivotal role in government- wide cybersecurity efforts, as identified in our Ensuring the Cybersecurity of the Nation high-risk area. DHS has established the National Cybersecurity and Communications Integration Center, which functions as the 24/7 cyber monitoring, incident response, and management center for the federal civilian government. However, DHS has continued to be challenged in measuring how the center is performing its functions in accordance with mandated implementing principles. Financial management. DHS received a clean audit opinion on its financial statements for 6 consecutive years—fiscal years 2013 to 2018. However, in fiscal year 2018, its auditor reported two material weaknesses in the areas of financial reporting and information technology controls and financial systems, as well as instances of non-compliance with laws and regulations. These deficiencies hamper DHS’s ability to provide reasonable assurance that its financial reporting is reliable and the department is in compliance with applicable laws and regulations. Further, DHS components’ financial management systems and business processes need to be modernized; the current systems affect the department’s ability to have ready access to reliable information for informed decision-making. As we reported in 2017, DHS officials have faced various challenges in their efforts to address this—lack of sufficient resources, aggressive schedule, complex requirements, and increased costs. Effectively modernizing financial management systems for the Coast Guard, Federal Emergency Management Agency, and Immigration and Customs Enforcement would help address DHS’s risk in this area. Human capital management. DHS has continued to strengthen its employee engagement efforts by implementing our 2012 recommendation to establish metrics of success within components’ action plans for addressing its employee satisfaction problems. Further, DHS has conducted audits to better ensure components are basing hiring decisions and promotions on human capital competencies. OPM’s 2018 Federal Employee Viewpoint Survey data showed that in the past 2 years, DHS’s score on the Employee Engagement Index increased by 4 points—from 56 in 2016 to 60 in 2018—which was 1 point more than the government wide increase over the same period. While this improvement is notable, DHS’s 2018 score ranked 20th among 20 large and very large federal agencies. Increasing employee engagement and morale is critical to strengthening DHS’s mission and management functions. Management integration. Since 2015, DHS has focused its efforts to address crosscutting management challenges through the establishment and monitoring of its Integrated Priorities initiative. The department updated these priorities in September 2017. Each priority includes goals, objectives, and measurable action plans that are discussed at monthly leadership meetings led by senior DHS officials, including the Under Secretary for Management. DHS needs to continue to demonstrate sustainable progress integrating its management functions within and across the department. In closing, it is clear that significant effort is required to build and integrate a department as large and complex as DHS, which has grown to more than 240,000 employees and approximately $74 billion in budget authority. Continued progress for this high-risk area depends primarily on addressing the remaining outcomes. In the coming years, DHS needs to continue implementing its Integrated Strategy for High-Risk Management to show measurable, sustainable progress in implementing corrective actions and achieving outcomes. In doing so, it remains important for DHS to maintain its current level of top leadership support and sustained commitment to ensure continued progress in executing its corrective actions through completion; continue to identify the people and resources necessary to make progress towards achieving outcomes, work to mitigate shortfalls and prioritize initiatives as needed, and communicate to senior leadership critical resource gaps; continue to implement its plan for addressing this high-risk area and periodically provide assessments of its progress to us and Congress; closely track and independently validate the effectiveness and sustainability of its corrective actions, and make midcourse adjustments as needed; and make continued progress in achieving the 13 outcomes it has not fully addressed and demonstrate that systems, personnel, and policies are in place to ensure that progress can be sustained over time. We will continue to monitor DHS’s efforts in this high-risk area to determine if the outcomes are achieved and sustained over the long term. Madam Chairwoman Torres Small, Ranking Member Crenshaw, and Members of the Subcommittee, this completes my prepared statement. I would be happy to respond to any questions you may have at this time. If you or your staff members have any questions about this testimony, please contact me at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals making key contributions to this work include Claudia Becker, Assistant Director; Imoni Hampton, Analyst-in-Charge; Michele Fejfar, Melissa Greenaway, James Lawson, and Tom Lombardi. Key contributors for the previous work that this is based on are listed in each product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "GAO has regularly reported on government operations identified as high-risk because of their increased vulnerability to fraud, waste, abuse, and mismanagement, or the need for transformation to address economic, efficiency, or effectiveness challenges. In 2003, shortly after the department was formed, we designated Implementing and Transforming DHS as a high risk area to the federal government because DHS had to transform 22 agencies into one department, and failure to address associated risks could have serious consequences for U.S. national security. In 2013, we reported that although challenges remained for DHS, the department had made considerable progress. As a result, we narrowed the scope of the high-risk area to focus on strengthening DHS management functions (human capital, acquisition, financial management, and information technology). discusses DHS's progress and remaining actions needed to strengthen and integrate its management functions. This statement discusses DHS’s progress and remaining actions needed to strengthen and integrate its management functions. This statement is based on our 2019 high-risk update and other reports issued from February 2017 through March 2019. Among other things, GAO analyzed DHS strategies and other documents related to the department's efforts to address its high-risk areas. As GAO reported in its 2019 high-risk update, the Department of Homeland Security (DHS) has continued its efforts to strengthen and integrate its acquisition, information technology, financial, and human capital management functions. As a result, the department has continued to meet three out of five criteria for removal from GAO's High-Risk List (leadership commitment, action plan, and monitoring) and partially meet the remaining two criteria (capacity and demonstrated progress). With regard to leadership commitment, DHS's top leadership has continued to demonstrate support for addressing the department's management challenges through, for example, its Integrated Priorities initiative to strengthen the integration of DHS's business operations across the department. Additionally, DHS has established an action plan for addressing the high-risk area and has issued 14 updated versions since 2011.This action plan also demonstrates DHS's ongoing monitoring of these efforts as it describes DHS's progress to-date and planned corrective actions. The two key areas where additional work is needed are DHS's capacity and demonstrated progress. With regard to capacity, DHS needs to make additional progress identifying and allocating resources in the areas of acquisition, information technology, and financial management. With regard to demonstrated progress, DHS should show the ability to achieve sustained improvement across 30 outcomes that GAO identified and DHS agreed were needed to address the high-risk area. GAO found in its 2019 high-risk update that DHS fully addressed 17 of these outcomes, while work remains to fully address the remaining 13. DHS has made some progress in recent years regarding human capital and information technology outcomes, but needs to continue implementing its action plan to show measurable, sustainable progress in achieving the 13 outcomes not yet fully addressed. Since 2003, GAO has made about 2,800 recommendations to DHS to strengthen its management efforts, among other things. DHS has implemented more than 75 percent of these recommendations which have strengthened program management and performance measurement.", "document_type": "gao"}
{"report": "We have previously reported on how federal agencies were planning to protect their federal workers during a pandemic, as well as lessons learned from the H1N1 pandemic, the most recent pandemic experienced by our nation prior to COVID-19. Based on these lessons, and further informed by more recent events, we have identified key issues for federal agencies to consider as their employees reenter the workplace. Agencies should maintain continuous communication with employees, and their representatives, during a pandemic. In particular, agencies should identify employee concerns and communicate human capital guidance such as pay, leave, staffing, and other human capital flexibilities to employees to help to ensure the continuity of agencies’ operations and mission essential functions. It is important that employees understand the policies and requirements of their agencies, and the alternatives, such as telework, that may be available to them. Continuous communication will also help agencies to provide real-time information to employees as conditions evolve. Employees who must work onsite during a pandemic will face varying levels of exposure risk. The level of risk depends, in part, on whether or not they will be in close proximity to people potentially infected with the virus. As a first step, it is important that agencies identify mission essential functions that cannot be performed remotely, as well as the related number of employees who will perform those functions and their risk of exposure. Agencies should consider how they will continue to update their determinations and monitor the associated risks, as these factors could affect decisions on reentry as conditions evolve. It is important for federal agencies to factor-in local conditions of the pandemic at the component and facility level in their determinations regarding workforce reentry rather than applying across-the-board decisions based on agencies’ headquarters locations. Agencies should consider making decisions about reentry, including precautions and safeguards agencies take, based on the local prevalence of the pandemic at each site. As agencies consider local conditions for reentry, they should share information and cooperate with other agencies located in the same area. These reentry decisions could change over time as the pandemic progresses, such as if there is a second or third wave of outbreaks. It is important that agencies’ plans to protect their workforce for a pandemic are operational at all levels of the organization; particularly for those workers who have to perform mission-essential functions onsite. To protect employees as they reenter their workplaces, agencies should have appropriate protection measures in place, by exposure risk level. For example, an agency could make changes to the work environment to reduce workplace hazards, such as by installing sneeze guards as a barrier between employees who must have frequent contact with other employees or the public. Additionally, an agency could provide personal protective equipment (PPE), such as surgical masks, gloves, and N-95 respirators, to employees which, if used correctly, can help prevent some exposures. Agencies will want to ensure that they have an adequate supply of hygiene supplies, such as hand sanitizers, and a plan for distributing those supplies within the agency. Some basic hygiene precautions, such as encouraging employees to wash their hands or use a hand sanitizer after they cough, sneeze, or blow their noses, can be implemented in every workplace. Agencies will also want to provide supplemental cleaning programs for common areas. Avoiding crowded settings through social distancing strategies is one of the best ways to prevent infection during an influenza pandemic. Agencies can implement various social distancing strategies to avoid situations that increase workers’ risk of exposure to a pandemic virus. For those functions that can be performed remotely, agencies may consider maximizing the use of telework, which is discussed in greater detail later in this statement. Other strategies agencies should consider include avoiding unnecessary travel, restricting in-person meetings and gatherings, and allowing flexible schedules to reduce the number of employees in the building at the same time. Agencies should also consider workplace reconfiguration (such as building walls or partitions between workstations), office-specific protocols (such as limiting personal contacts among staff), and making decisions about reopening office fitness and childcare centers as part of separate risk-based decision processes. When medical countermeasures—such as antivirals and vaccines—are developed, it will be important for agencies to decide the extent to which these countermeasures will be provided to employees. In cases where countermeasures are going to be provided to employees, agencies should consider actions necessary to procure them, and establish clearly- defined, well-documented, and consistently-applied protocols to prioritize and allocate their distribution. The Bureau presents an illustrative example on continuity of operations and decision making for resumption of operations. The Bureau has both permanent staff in headquarters and a large local field infrastructure of 248 Area Census Offices (ACO) with short-term staff to implement the decennial census. On March 11, 2020, the World Health Organization declared COVID-19 a pandemic, just a day before invitations to respond to the 2020 Census were scheduled to arrive in mailboxes across the country, and as peak census operations were set to begin. In March and April 2020, the Bureau suspended operations until June 1 and extended key operations. Since that time, the Bureau announced it would resume operations at additional ACOs each week, and as of June 11 all ACOs had resumed operations. Key aspects of reopening Bureau ACOs included (1) a phased approach to resuming operations, (2) operational changes in response to challenges related to COVID-19, (3) addressing worker safety concerns, (4) communicating pandemic plans to ensure continued operations, and (5) contingency planning for risks. The Bureau took a phased approach to resuming operations at its ACOs. To make these decisions, the Bureau considered multiple factors including whether the state in which each ACO was located had begun phased reopening, whether data on federally established health criteria supported the decision to restart, and whether the Bureau could meet the safety needs of ACO employees and the public. In our late May survey of ACO managers, responses on the Bureau’s efforts to reopen offices varied. For example, 66 percent of ACO managers responding to our late May survey reported satisfaction with the process of recalling office staff, 68 percent with readiness to conduct field operations, and 75 percent with readiness to conduct office operations. Operations resumed in a phased manner not only by office, but also by function. As the Bureau resumed operations, it was able to resume operations that required less physical interaction, such as Update Leave, in which field staff deliver questionnaires to homes that might not receive mail delivered to their doors. Operations that require interviewing residents, such as Non-Response Follow Up, were delayed until August. The Bureau has also made a number of changes to its 2020 Census operations to minimize face-to-face interactions. The Bureau modified its Update Leave operation, directing field staff to update the addresses by observation when delivering a questionnaire, instead of knocking on doors to speak with residents. To reduce in-person contact for the Group Quarters operation, which enumerates facilities such as prisons, nursing facilities, and college dormitories, Bureau officials told us they were contacting facilities to encourage them to shift from in-person enumeration to electronic responses. In late April, the Bureau also authorized its ACOs to call those facilities that had previously opted to respond by providing a paper listing of residents. Census staff asked the facilities to mail the listing back to the ACO rather than having the ACO send staff to pick up the paper listing. The Bureau stated that it will coordinate with federal, state, and local health officials to put appropriate protocols and procedures in place and ensure adequate PPE and cleaning supplies. In early May, the Bureau announced that it had ordered this equipment for all field staff and that these materials would be secured and provided before resumption of operations. Bureau officials told us they are distributing PPE and cleaning supplies to its 248 ACOs on a rolling basis, prioritizing delivery to those ACOs that were resuming major field operations, such as Update Leave. In our late May survey, ACO manager satisfaction was relatively high regarding PPE for staff conducting Update Leave (66 percent), the largest field operation being conducted at the time. In contrast, managers at that time reported some of their lowest satisfaction rates when asked about PPE adequacy for their office and field workers more generally (34 and 43 percent, respectively). ACO managers reported higher satisfaction in late May than in early April with their ACO’s ability to safely manage employees and operations during the pandemic (increasing from 55 to 65 percent on average across three questions on this topic). Despite this increase in confidence, managers expressed concerns regarding worker safety in open-ended comments. For example, in late May managers expressed concerns regarding how fingerprinting of large numbers of staff—necessary to fulfill the census mission—could be conducted safely under conditions of social distancing. In addition, more than 15 comments in early April and 11 in late May expressed concerns about the ability of the ACO management teams to telework. These included concerns about the inadequate number of laptops and who was expected to report to their local office. The Bureau created a COVID-19 Internal Task Force to create a communications plan and appropriate workforce flexibilities. The Bureau sent emails to regional staff with updated information on delaying field operations and prepared documents to answer questions about the delays, office operating status, payroll, hiring, and training. Responses to our surveys of ACO managers highlighted the need for the Bureau to ensure open lines of communications. Between early April and late May, respondent satisfaction increased regarding the timeliness and clarity of Bureau communication about its pandemic plan; however, satisfaction in these areas remained relatively low. Specifically, reported satisfaction increased for communication timeliness (from 35 to 45 percent) and clarity (from 42 to 51 percent). More than 50 ACO managers commented about communication challenges such as conflicting direction from different sources and guidance received shortly before the implementation date. In May 2019, we reported that the Bureau did not have contingency plans for many identified risks including for major disasters—such as an epidemic—and recommended that it develop contingency plans for all risks that did not have one. The Bureau updated its risk register for major disasters to include a contingency plan. According to the Bureau’s March 2020 risk register, the contingency plan for any major disaster—including an epidemic—is rapid response, meaning the Bureau would develop a plan to address the risk once it was realized. Bureau officials told us that, depending on the type of major disaster, response would vary widely and even if they had a more detailed contingency plan for a pandemic it would have never addressed the magnitude of the current national emergency that is taking place across the country. We have identified key practices in telework-related literature and guidelines that federal agencies should implement in developing telework programs. Also, in 2011, we reported that the Office of Personnel Management, the General Services Administration, and the Federal Emergency Management Agency had suggested several practices to federal agencies, in various telework or emergency-related guidance documents, for how to ensure telework is part of continuity of operations planning. These practices generally align with those we previously identified. Based on this prior work, we have identified several practices that may be especially useful for agencies to help ensure telework programs contribute to continuity of operations during COVID-19 and other major emergencies. These practices may be especially important if substantial numbers of employees remain out of their workplaces for an extended period or if agencies need to reverse their reentry decisions based on changing public health circumstances. In addition, agencies’ experiences with telework during the current pandemic may suggest opportunities to increase the availability of telework in the future. These practices can be grouped into four general categories: (1) policies and guidance related to telework; (2) technology; (3) performance management; and (4) program evaluation. Major emergencies, such as a pandemic, underscore the importance of establishing and updating clear policies and guidance related to telework as agencies’ continuity of operations may depend on employees working remotely for extended periods. Agencies should assess whether their policies and guidance were sufficient to ensure that their workforces were telework ready and understood the agency’s expectation of employees regarding teleworking during this emergency. The current crisis presents an opportunity for agencies to assess their established policies or requirements to ensure that they (1) balance employees’ personal circumstances and work responsibilities, and (2) effectively facilitated communication and engagement among teleworkers, managers, and coworkers. Agencies should institute processes for communicating human-capital guidance for emergencies (e.g., pay, leave, benefits) to ensure they worked effectively. For example, agencies should consider whether emergency employees (including COOP employees) knew in advance about their mission critical status. If not, agencies should ensure that, in case of future emergencies including a potential resurgence of COVID-19, employees are notified about requirements to report for work, remain at work, work at home, or report to an alternative work site when government operations are disrupted. Similarly, agencies should consider whether their guidance on workplace health and safety issues was adequate to ensure that teleworkers had safe and adequate places to work off-site, and whether information- security training was provided to all individuals, or managers of individuals, who teleworked during the current pandemic. It is important for agencies to correct any identified deficiencies in the guidance and training to improve the use of telework going forward, including for future emergency situations that may again require telework. We have reported that technology concerns are frequently cited barriers to telework. To effectively use telework as a tool to continue operations during major emergencies, agencies must have an appropriate information technology infrastructure in place that allows large numbers of employees to telework simultaneously. As such, it is important for agencies to assess the extent to which their telework infrastructure was adequate to support increased telework, especially during peak periods, including whether technical support was sufficient, and address any access and security issues they identify. Ensuring established organizational performance standards are met is important to maintaining agency operations whether employees are physically present in the office or working remotely. During extended periods of remote work, this could include setting expectations and preferences for how employees communicate with supervisors before telework arrangements begin. Agencies should consider whether their existing procedures and standards meet the needs of employees who teleworked and whether they ensured that telework did not diminish employee and organizational performance. Evaluation of telework may help agencies better understand the impact their increased use of telework had on their ability to achieve goals and accomplish missions, and could allow them to make adjustments to telework moving forward as employees are expected to return to their duty stations. As part of such an evaluation, agencies should assess whether their processes, procedures, and tracking systems to collect data provided the information needed to evaluate telework. In conclusion, federal agencies have a responsibility to provide safe workplaces for employees to perform their jobs. The evolving and growing challenges from the COVID-19 pandemic present critical workforce safety issues for federal agencies to assess and address as they seek to continue their operations. As I have discussed today, agencies should consider a number of factors when making decisions about employees reentering workplaces. Lessons learned from previous pandemic emergencies, as well as from telework use to ensure continuity of operations, can be helpful as agencies navigate ongoing workforce safety and productivity challenges. Consideration of these factors and lessons learned from agencies’ current experiences may better prepare agencies to address and respond to challenges from ongoing and future emergencies. Chairman Connolly, Ranking Member Hice, and Members of the Subcommittee, this concludes my prepared statement. I would be happy to respond to any questions you may have. If you or your staff have any questions about this testimony, please contact J. Christopher Mihm at (202) 512-6806 or MihmJ@gao.gov, or Michelle B. Rosenberg at (202) 512-6806 or RosenbergM@gao.gov. Individuals making key contributions to this testimony include Clifton G. Douglas, Jr., Alexandra Edwards, Sarah E. Veale (Assistant Directors), Keith O’Brien (Analyst-in-Charge), Ulyana Panchishin, Maya Chakko, Karin Fangman, Steven Putansu, and Jacqueline Chapin. Key contributors for the earlier work that supports this testimony are listed in each product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Federal employees perform critical functions across multiple mission areas, from those vital to the long-term well-being of the country to those directly charged with aspects of public safety. Major emergencies, such as the COVID-19 pandemic, can pose threats to employees' safety and conditions may ebb and flow over an extended period. During these situations, federal agencies have a responsibility to provide an environment for employees to perform their jobs safely and effectively. This statement provides (1) key considerations based on GAO's prior work for federal agencies as federal workers reenter the workplace; (2) an illustrative example of how the Census Bureau was forced to suspend major Decennial Census field operations and the process it used to resume operations; and (3) key practices for ensuring telework contributes to continuity of operations. This statement is based on a large body of GAO work on pandemic preparedness, reviews of the Decennial Census, and federal human capital management issued from July 2003 through June 2020. The rapidly escalating challenges from the Coronavirus Disease 2019 (COVID-19) global outbreak present critical workforce issues for federal agencies to assess and address. GAO's prior work on pandemics and human capital issues has shown that agencies should consider a range of factors to carry out their missions while protecting their workforce and the members of the public with whom they interact. Key considerations for federal workers' reentry to workplaces . As federal agencies manage operations during the COVID-19 pandemic and plan for their employees to safely return to workplaces, GAO's prior work has shown that it is important for agencies to identify mission essential functions that cannot be performed remotely when deciding who needs to return to the office. Agencies should also consider the exposure risk level and local conditions when deciding whether to reopen offices across the country. To protect employees as they reenter the workforce, it will be important for agencies to have appropriate protection measures in place. For example, agencies should consider how they can ensure adequate distribution of hygiene supplies. They should also consider changes to the work environment to reduce workplace hazards, and implement social distancing strategies. How the Census Bureau decided to resume Decennial Census operations. The U.S. Census Bureau offers an example of how an agency suspended and resumed operations under the current pandemic. In March 2020, the U.S. Census Bureau suspended field operations of the Decennial Census and took a phased approach to resuming operations at its area census offices. As of June 11, all area census offices had resumed operations. Key aspects of resuming operations at area census offices included: (1) taking a phased approach to restarting operations, such as resuming operations that required less physical interaction first; (2) making operational changes to minimize face-to-face interactions; (3) addressing worker safety concerns; and (4) communicating pandemic plans to ensure continued operations. Key practices for ensuring telework contributes to continuity of operations. Several key practices GAO previously identified are useful for agencies to help ensure telework contributes to continuity of operations during the current pandemic and in the future. Specifically, agencies should consider based on their current experiences whether: (1) their policies and guidance related to telework are sufficient to ensure that their workforces are telework ready and balances are struck between employees' personal circumstances and work responsibilities; (2) the extent to which their telework infrastructure, including technical support and security, is adequate to support increased telework; (3) procedures and standards are in place that ensure telework does not diminish organizational and employee performance; and (4) the processes, procedures, and tracking systems to collect data provide the information needed to evaluate the use of telework. These assessments will assist agencies in considering broader changes to their policies and procedures related to telework as employees are called back to their duty stations.", "document_type": "gao"}
{"report": "The EDA program is one of several programs designed to build partner capacity through the provision of excess defense equipment and services to foreign governments or international organizations such as the North Atlantic Treaty Organization (NATO). These excess items are provided as part of U.S. security assistance efforts and help to support U.S. foreign policy and national security objectives. The Foreign Assistance Act permits the transfer of excess defense articles provided that such transfers will not adversely affect the industrial base. In particular, under the Act, transfers must not reduce the opportunity for U.S. contractors to sell new or used defense equipment to countries requesting the transfer. Excess defense items can include aircraft, ammunition, clothing, radios, trucks, and spare parts. According to DOD officials, the vast majority of EDA items are low- to medium-level technologies that, if not transferred, would either be stored at cost to DOD or destroyed. Excess defense items can be transferred as grants—as permitted by the Foreign Assistance Act—or sold to eligible foreign governments at a reduced cost in “as is, where is” condition pursuant to the Arms Export Control Act. This means that the requesting foreign government is generally required to pay all repair or refurbishment costs, as well as all costs associated with transporting the EDA item—which can be located in the United States or outside the continental United States. As previously mentioned, for purposes of this report, transfers refer to grants of EDA items unless otherwise indicated. DSCA has overall responsibility for administering the EDA program. The Director of DSCA has been delegated authority to make the determination on whether a proposed transfer could adversely affect the industrial base. The military departments determine when defense items are no longer needed and can designate them as excess and, upon approval, can offer them as EDAs. Multiple federal entities play a role in the EDA program, as illustrated in figure 1. Following the interagency coordination, if DSCA determines the proposed transfer will not adversely affect industry and thus can proceed, DSCA notifies Congress about proposed transfers that are valued at over $7 million or that contain significant military equipment. As part of the congressional notification, DSCA provides information on (1) the purpose for which the item is being provided to the country, (2) whether the item has been previously provided to the country, (3) the current value and original acquisition value of the item, and (4) its findings regarding how industry will be affected by the proposed transfer. After a 30-day congressional notification period, DSCA authorizes the proposed transfer in consultation with State, provided that Congress does not object and all agencies concur with the transfer. DSCA follows the same process to review and approve all proposed EDA transfers—including for excess Humvees. One unique difference for Humvee transfers is a 2018 legislative requirement that Humvees be modernized with an armored or armor-capable crew compartment and a new modernized powertrain prior to a transfer, unless a waiver is granted. Humvees, which are four-wheel drive military light trucks, have been part of DOD’s light tactical wheeled vehicle fleet since the 1980s. While the Army is the program office for Humvees, the vehicles have been used by other military departments in support of their own combat operations. Humvees were initially fielded to serve as a light, highly mobile and unarmored vehicle and are commonly used for combat operations; however, the Army National Guard also procures these vehicles for use in homeland defense and natural disaster relief operations. In efforts to adapt the Humvee to modern requirements for combat operations, the Army has increased the performance and protection of the vehicle over time. Over the past 30 years, AM General has produced three models—the M900, M1000, and M1100 series. The company no longer produces the M900 and M1000 series for combat operations and certain parts and components that are unique to these vehicles are obsolete or otherwise not readily available. The M1100 series, which is still in production and supports combat operations and many non-combat related operational and support missions, offers newer capabilities such as increased weight capacity. With the additional weight capacity, the 1100 series is the only model that can support the added armor requirements under the new legislative requirement without a substantial overhaul. Figure 2 highlights some of the capabilities of the different Humvee models. DOD’s light tactical wheeled vehicle strategy has changed since 2010, following lessons learned from military operations in Iraq and Afghanistan. DOD plans to shift from procuring new Humvees to sustaining existing vehicles in its fleet. In its 2014 Tactical Wheeled Vehicle Strategy, the Army stated plans to buy fewer new Humvees because the vehicle no longer fully meets its evolving mobility or protection requirements. While DOD decreased its procurement of Humvees for military operations, it has plans to upgrade and refurbish existing vehicles. There are nearly 300,000 Humvees or vehicles with the Humvee chassis operating globally by the U.S. military and other foreign governments. These vehicles are expected to require ongoing maintenance and upgrades for the next 20 to 30 years. DOD routinely conducts industrial base risk assessments to gain insight on the viability of current suppliers to meet its current and future requirements. The assessment takes into account a range of considerations including (1) factors that could cause a current supplier to go out of business or exit the market and (2) the extent to which an existing supplier relies on DOD, foreign military sales, or commercial sales. While these assessments are not routinely conducted as part of the excess defense article (EDA) transfer process, they may be undertaken to provide input on EDA transfers, as needed. The Army has efforts underway to acquire a new vehicle—the Joint Light Tactical Vehicle (JLTV)—to meet its future requirements. Although a different manufacturer was awarded the JLTV contract, in its industrial base risk assessment for this requirement, the Army stated it intends to maintain two manufacturers—including AM General—to meet its ongoing needs for light tactical wheeled vehicles. In a 2018 congressional briefing, the Army’s Acquisition, Logistics, and Technology Command estimated maintaining a relatively even mix of both vehicles—54,810 Humvees from existing inventory and 49,099 new JLTVs—to sustain operations for the foreseeable future. However, the Army is conducting a more comprehensive review of its light tactical vehicle requirements and plans to release its findings in an updated acquisition strategy expected in 2022. DOD approved nearly half of the total Humvees requested by foreign governments for fiscal years 2012 through 2018. The requests were in support of foreign governments’ security efforts, such as counterterrorism. However, the number that was actually delivered was less than those approved because DOD decreased the number or foreign governments canceled their requests for various reasons. DSCA halted approvals of EDA Humvee requests since the start of fiscal year 2017 and raised concerns about the new statutory requirement to modernize Humvees prior to transfer. DOD approved nearly half of the total Humvees requested by foreign governments for fiscal years 2012 through 2018—7,612 vehicles of the 16,005 excess Humvees requested—but has not approved Humvee requests made since the start of fiscal year 2017. Figure 3 shows the number of Humvees requested and approved for transfer each fiscal year. In our analysis of data provided by the Army and DSCA, we found from fiscal years 2012 through 2018, that 23 countries submitted requests for Humvees, including some requests in fiscal year 2018. The delivery of EDA items under the Foreign Assistance Act to certain countries is given priority to the maximum extent feasible. These countries include certain NATO countries, major non-NATO allies in the Middle East and Africa regions, and the Philippines. We found that the Middle East and Africa regions accounted for 75 percent of the vehicles requested over this period. Figure 4 shows the regional distribution of requests. The majority of requests for Humvees from countries in the Middle East and Africa regions were primarily to support various security-related missions. For example, one country requested excess Humvees for border security, counter smuggling, and counterterrorism operations. Such security-related efforts by foreign countries align with the U.S. 2018 National Defense Strategy, which states DOD’s objective to prevent terrorism globally and aid U.S. foreign partners in their counter-terrorism efforts. Additionally, the strategy aims to strengthen alliances and attract new partners by increasing interoperability to work together and effectively achieve military objectives. DSCA is required to state the comparative foreign policy benefits that the United States would gain from a grant transfer rather than a sale when it notifies Congress about a proposed transfer. In the documents we reviewed, DSCA cited foreign policy benefits such as increasing the capability of countries to take on a greater share of military operations, supporting joint operations with NATO, or counterterrorism and counter- narcotics operations. For example, for one request, DSCA determined that a requested transfer was in the U.S. national interest, as equipping the foreign country’s armed forces with Humvees would allow them to have an increased role in military operations in the Africa region. In turn, this would reduce the country’s reliance on U.S. forces for NATO operations. In addition to requesting vehicles for security-related operations, some countries planned to use vehicles for spare parts or had plans to refurbish the vehicles on their own. We found that about two-thirds of the Humvees delivered through the EDA program from fiscal years 2012 through 2018 were older models—either M900 or M1000 series—rather than the newer M1100 series. Most countries receiving deliveries of older models were seeking to replace existing vehicles in their fleet or to use EDA Humvees for spare parts. As previously mentioned in this report, DSCA has not approved any EDA Humvee requests since the start of fiscal year 2017. One reason, according to our analysis of DSCA data, is the manufacturer’s objections to proposed transfers. Another is because of the legislative provision in the Fiscal Year 2018 NDAA that requires Humvees to be modernized with an armored or armor-capable crew compartment and new, modernized powertrain prior to transferring. The corresponding conference report stated the conferees’ expectation that any modernization and refurbishment work must generally be done at no cost to DOD. According to DOD, the cost to modernize would be incurred by the requesting foreign government. Since the provision’s enactment, DOD has not exercised the authority to waive this legislative requirement for any Humvee request. Foreign governments have not been willing to pay for the modernization, so approvals have halted. Since the enactment of the modernization requirement in December 2017, DSCA has received requests for 4,103 Humvees. According to DSCA officials, when a foreign government submits a letter of request for EDA Humvees, DSCA notifies the country of the modernization requirement and its responsibility to pay for the cost to refurbish the vehicles in accordance with the law. In DOD documents we reviewed, foreign governments cited having limited budgets and being financially unable to purchase defense equipment such as Humvees. As such, they rely on the EDA program to acquire defense items. DSCA officials told us that the modernization work is to be done at no cost to the U.S. government; however, they added that paying the cost to modernize Humvees can be cost-prohibitive for foreign governments. Foreign governments can request, through DSCA, that the modernization requirement be waived. Since December 2017, according to DSCA officials, DSCA has received waiver requests from three foreign governments but has not exercised the waiver authority. According to DSCA officials, these requests likely will remain unapproved for the foreseeable future; however, the provision requiring the refurbishment of excess Humvees prior to transfer is set to expire in December 2020. According to DSCA officials, DSCA plans to resume its normal EDA approval process thereafter. Currently, according to DSCA officials, they are encouraging foreign governments to look at other options to meet their fleet requirements, including purchasing new Humvees. However, DSCA officials acknowledge that, if a foreign government cannot afford to buy new vehicles, DOD does not have any low-cost vehicles to offer as an alternative solution. However, DOD officials and Army documents we reviewed noted that even if foreign governments were able to independently fund the modernization costs, there are not sufficient quantities of the newer model Humvees—M1100 series—in inventory that can support the additional weight of the added armored capabilities for the modernized crew compartment. According to DSCA documentation, the EDA program has a little over a hundred vehicles that could be refurbished to the modernization requirements. Additionally, most of the Humvees in DOD’s inventory are older models that would first require a new expanded vehicle chassis to withstand the weight of adding armor. The officials likened the modernization process for the older model Humvees to essentially building a whole new vehicle. DSCA’s determinations of whether there is an adverse industrial base effect to approve Humvee transfers are largely based on objections from the manufacturer about the proposed transfers. Since 2015, the Humvee manufacturer has objected more frequently to the transfer of vehicles to foreign governments. In all but one instance when the manufacturer objected to a transfer, we found that DSCA and BIS took steps to address concerns of the Humvee manufacturer and reach a resolution, such as providing the manufacturer Humvee refurbishment work. DSCA’s decision on whether there is an adverse industrial base effect to approve a transfer of Humvees is largely based on the manufacturer’s perspective on a proposed transfer. DSCA has considerable latitude for such decisions as the Foreign Assistance Act, as delegated, does not specify how determinations should be made on whether proposed transfers could adversely affect U.S. industries. Historically, DSCA has sought input from BIS to aid its determination about potential industrial base effects of proposed transfers. According to DSCA officials, all proposed EDA Humvee transfers have undergone an assessment of adverse industrial base effect by BIS. We found that BIS actively engages the Humvee manufacturer on proposed transfer requests and supported all but one objection from fiscal years 2012 through 2018. BIS’s standard practice is to collect information from the prime contractor and other suppliers to inform its recommendation to DSCA about possible industrial base effects. As part of its efforts regarding proposed Humvee transfers, BIS notified AM General and provided information on all the transfer requests including the requesting country; number of vehicles requested; the vehicle model; and the country’s plans, if known, to repair or upgrade EDA vehicles, including who the country intends to select for such work. BIS officials told us that they request a response within 7 calendar days on whether the manufacturer supports or objects to the proposed transfer. In instances where the Humvee manufacturer objected to a transfer, BIS required that the manufacturer provide an explanation of its objection. In documents we reviewed, the manufacturer objected for various reasons, including that a transfer would: (1) directly interfere with ongoing marketing or planned sales to the requesting country, or (2) adversely affect its business and that of its suppliers. BIS’s standard procedure is to request proof of ongoing sales efforts if a company states that a proposed transfer will interfere with potential sales. In the cases where the Humvee manufacturer cited ongoing or planned business development with a requesting country, BIS required that the manufacturer provide information of its ongoing efforts to sell its vehicles to the requesting country, including: documentation of recent or planned meetings with foreign government officials and a timeline of the meetings; export licenses; and business plans. If a manufacturer submits an objection, BIS will also check if they have registered business activity with Commerce’s Advocacy Center, which provides assistance to defense companies pursuing contracts with overseas governments and government agencies. If BIS concludes the Humvee manufacturer has a basis for its objection due to ongoing business with the requesting country, it will recommend that DSCA not authorize the transfer. According to DSCA officials, this is largely because it considers the possibility that the transfer could dissuade requesting foreign governments from purchasing new or used vehicles. Thus, providing vehicles through the EDA program at no cost or a discounted price to a foreign government could siphon potential business from the manufacturer or could compete with the manufacturer’s sales efforts. Under the Foreign Assistance Act, a transfer request cannot be fulfilled if doing so will interfere with the manufacturer’s ability to sell equipment to the requesting country. During fiscal years 2012 through 2018, we found only one instance where DSCA, based on BIS’s recommendation, did not support AM General’s objection. In that case—a request for Humvees from Albania—DSCA moved forward and approved a Humvee transfer because the manufacturer could not demonstrate ongoing business with the requesting country. AM General has objected more frequently to the transfer of vehicles to foreign governments since March 2015. In 2015, the JLTV production contract was awarded to another contractor and the Humvee manufacturer sold its commercial automotive plant, both of which occurred in the wake of decreasing or nonrecurring DOD Humvee procurements in comparison to past years. In total, the Humvee manufacturer has challenged 11 transfer requests for over 4,000 vehicles between fiscal years 2015 and 2018. The manufacturer told us that the increasing number of proposed transfers is concerning because the transfers amount to nearly 3 years’ worth of new vehicles it could produce to sustain its production lines. AM General representatives told us they will continue to object to the transfer of older Humvee vehicles (M900 and M1000 models). For these models, the representatives citied concerns that parts for these vehicles are no longer in production, and thus the manufacturer cannot ensure qualified parts are available for maintenance and repairs. They are also concerned that older vehicles have the propensity to break down, which could damage the Humvee brand internationally—particularly, if counterfeit parts are used. In our review of documents describing requesting countries’ use of vehicles, we found that older model vehicles are, at times, accepted by foreign governments to use as spare parts to maintain an existing fleet and to develop their workforce’s capability to repair vehicles. However, we found that since 2015, the majority of vehicles to which the manufacturer objected were the newer M1100 models—stemming largely from a single 2016 request for Afghanistan. To support its objections to this transfer, AM General has stated that its own international sales are an important source of revenue, particularly because DOD has reduced its procurement of Humvees. AM General representatives explained that proposed transfers through the EDA program can threaten their company’s potential future sales to foreign governments that may be less likely to purchase new Humvees if DSCA approves transfers of used vehicles. According to the manufacturer, each transfer is a potential one-for-one reduction of a possible sale of a new vehicle to the requesting country, which can affect its bottom line as well as the suppliers that provide parts and materials to produce the Humvees. In our review of Army procurement data, we found that many countries that requested excess Humvees have not purchased them through the FMS program from fiscal years 2012 through 2018. DSCA officials told us that most of the countries requesting Humvees through the EDA program find it cost-prohibitive to purchase new Humvee vehicles directly from the manufacturer. A new Humvee can cost between $115,000 and $190,000 depending on the model and capabilities included. As a result, these countries rely on EDA Humvees provided through grants to sustain their military fleets. Figure 5 shows the number of Humvees procured by DOD relative to the number of vehicles foreign governments bought through the FMS program and those they were granted via the EDA program. We found that from fiscal years 2012 to 2018 AM General’s objections to proposed EDA Humvee transfers have increased the time that it takes for DSCA and BIS to review and make their determinations. If the manufacturer did not object to a transfer, which was largely the case prior to March 2015, BIS provided its recommendation to DSCA, on average, within 21 days. However, our analysis showed that an objection to a Humvee transfer on average added approximately 152 days to address industry objections. DSCA officials acknowledged that the approval process can be prolonged when the manufacturer objects to a proposed transfer, potentially contributing to longer waiting periods for requesting countries to receive the Humvees. In addition, the longer that vehicles remain in storage, the more likely it is that they will require more repairs to make them operational, resulting in increased costs to the requesting foreign governments to refurbish them, according to a DSCA official. Manufacturer representatives also told us they want to be involved earlier in the process to provide input on the potential effects of proposed transfers. We found that, on average, DSCA notifies BIS about 4 months after a country submits its Humvee request and BIS reaches out to the manufacturer a day or two later. A DSCA official explained that it can be a challenge to involve the manufacturer earlier because the request is not fully stable and could be revised for a number of reasons, including countries canceling the request or changing requirements to obtain different capabilities, and DOD internal policy considerations need to be vetted before reaching out to the manufacturer. In recent years, DSCA and BIS have taken steps to address AM General’s increasing objections to proposed transfers. In 2018, BIS modified its approach to assess adverse effects of Humvee transfers to consider an additional factor. Now, BIS considers the cumulative effect and totality of previous EDA Humvee requests, in addition to assessing each request on a case-by-case basis. According to BIS officials, this was in response to the pattern of consistent objections that they were receiving from the Humvee manufacturer. AM General acknowledged that communication with DSCA and BIS about Humvee EDA transfers has improved. For example, DSCA notified AM General about its decision to sustain the company’s objection and, thus, not move forward on a transfer request made in July 2019 for 2,000 vehicles. AM General told us that in the past, DSCA did not notify AM General about whether it had sustained or overruled the company’s objection to a proposed transfer. AM General’s objections to EDA Humvee transfers have at times led to additional business channels for the Humvee manufacturer. For example, we found that the manufacturer received business opportunities from EDA Humvee transfers to Afghanistan, Iraq, Jordan, and Thailand that included, providing long-term sustainment and refurbishment of Humvees, among other things. In response, the contractor withdrew over a third of its objections between fiscal years 2012 and 2018 based on receiving this type of work or reaching agreements with foreign governments to provide fully operational Humvees. The remaining transfers were cancelled; put on hold pending resolution with the Humvee manufacturer; or in one case, moved forward with an objection in place. The agreements to provide additional support can be financially beneficial to the manufacturer and help sustain its production capabilities. For example, we found that in 2012, the Humvee manufacturer objected to a country’s transfer request of 250 vehicles, but withdrew their objection after reaching an agreement with the foreign government to perform much of the refurbishment work for those vehicles. In another case, we found that for the 2016 proposed transfer of 2,461 vehicles to support the Afghanistan National Security Force, the Humvee manufacturer objected, citing concerns about the large number of vehicles requested, among other concerns (see sidebar). The proposed transfer of EDA Humvees to Afghanistan was requested by DOD after a 2016 Senate report expressed concerns about a lack of insight into the cost-benefit analysis of procuring new equipment instead of refurbishing excess equipment. In response to the proposed transfer, the manufacturer sent a letter to BIS outlining their anticipated role in the Afghanistan transfer, including obtaining Army contracts to add armor kits to EDA vehicles, providing new powered chassis, and if required, new Humvees. The letter also noted the Humvee manufacturer’s withdrawal of its objection to the transfer. DSCA subsequently notified the manufacturer that it did not agree with the terms AM General outlined in the letter to BIS and specified that the proposed transfer would create business opportunities for U.S. industry, including AM General, to refurbish EDA Humvees. DSCA also added that it would continue to ensure that industry is notified of all proposed Humvee EDA transfer requests so that industry can provide input or express concerns. According to DOD officials, the number of Humvees available for transfer to Afghanistan was reduced as DOD decided to split the number of available EDA Humvees in inventory at the time to meet requirements in Afghanistan and Iraq. In total, 1,644 vehicles were identified for transfer to Afghanistan. As part of this effort, according to information we received from the Army, AM General, and the Office of the Undersecretary of Defense for Policy, the Humvee manufacturer was awarded a contract to provide armor kits for the 1,644 EDA Humvees being refurbished by the Army’s Red River Depot. The manufacturer also provided other vehicle parts as part of the EDA transfer request for Afghanistan. According to DOD officials, it currently does not have plans to transfer additional vehicles to Afghanistan to fulfill the remaining EDA vehicles requested as part of the 2016 transfer request and will reevaluate future Afghanistan requirements, as needed. We provided a draft of this report to the Departments of Commerce and Defense for review and comment. Both agencies provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretaries of the Departments of Commerce and Defense. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. This report provides information about (1) DOD’s approval of grant transfers of excess High Mobility Multipurpose Wheeled Vehicles (HMMWV)—commonly pronounced Humvees— requested by foreign governments from fiscal years 2012 through 2018 and (2) how the Humvee manufacturer’s perspectives on the proposed transfers have been addressed by DOD as part of the determination of any adverse industrial base effects. To provide information about DOD’s approval of transfers of excess Humvees, we analyzed data for fiscal years 2012 through 2018 (the most recent available fiscal year at the time of our review) from the U.S. Army Security Assistance Command, Defense Security Cooperation Agency (DSCA), and the Defense Logistics Agency (DLA). These data provided insight about the countries and geographic regions that have requested excess defense article (EDA) Humvees as well as the condition and types of vehicles delivered to foreign governments. We also reviewed documentation provided by requesting countries to identify the intended purpose of the request. We interviewed agency officials responsible for the data to identify the quality controls in place to help ensure the data are accurate and reliable. To assess the reliability of each data source, we compared the data in each DOD component’s data sets to ensure that the information was complete and consistent. We did this by identifying common identifiers used for the Humvee EDA transfers that occurred within the designated 7-year period. According to DSCA officials, the DSCA EDA database is a consolidation of data provided annually by the military departments, and DLA, and is manually entered into the database by DSCA officials. Furthermore, we reviewed the data for issues such as missing data elements and duplicates, among other steps. Based on these steps taken, we determined the data were sufficiently reliable for the purposes of reporting information about EDA Humvee transfer requests. See table 1 of DOD data sources used to track information on excess defense articles. To provide information about how the Humvee manufacturer’s perspectives on the proposed transfers have been addressed by DOD as part of the determination of any adverse industrial base effects, we reviewed documents, data, and interviewed officials from DSCA and the Bureau of Industry and Security (BIS) within the Commerce Department, that advises DSCA on industry effects of proposed EDA transfers. For purposes of this report, unless otherwise indicated, transfers refers to grants of EDA under the Foreign Assistance Act. We reviewed BIS policies and procedures related to the EDA program to identify the factors BIS considers in making adverse effect determinations. We also reviewed data generated by BIS to identify the extent to which the Humvee manufacturer objected to proposed transfers for the 7-year period included in our review. We also reviewed data provided by the Army on the number of Humvees procured for the Army’s use and for vehicles sold to foreign governments through the Foreign Military Sales program from fiscal years 2012 through 2018. To gain insight about DSCA and BIS’s approach to assess industrial base effects of proposed transfers, we selected two transfer requests as illustrative case studies: a 2016 transfer for Afghanistan which was the single largest proposed transfer and a 2016 transfer for Albania as it was the only proposed transfer that BIS did not sustain the manufacturer’s objection. We also spoke with representatives from AM General to obtain their perspectives on the EDA program and gain insight about the effect of EDA transfers on their business. We conducted this performance audit from February 2019 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Marie A. Mak, (202)-512-4841 or makm@gao.gov In addition to the contact name above, Candice Wright (Assistant Director) and Sameena Ismailjee (Analyst-in-Charge) managed this review. James McCully, Lorraine Ettaro, Phillip Farah, Stephanie Gustafson, Miranda Riemer, and Roxanna Sun made significant contributions to this report.", "summary": "DOD can declare defense equipment as excess to U.S. military needs and make it available for transfer as a grant or sale to foreign governments. The Foreign Assistance Act of 1961 authorizes these transfers as grants provided that they do not adversely affect the U.S. national technology and industrial base, among other things. In this regard, transfers pursuant to the Act must not limit U.S. companies' ability to sell new or used defense equipment to countries requesting the transfer. The 2018 NDAA generally requires that Humvee transfers be modernized with a new powertrain and armor prior to being transferred. The Act also generally requires GAO to report on proposed and completed Humvee transfers and the process to determine if transfers will adversely affect the industrial base. This report provides information on (1) excess Humvees requested and approved during fiscal years 2012 through 2018 and (2) how the Humvee manufacturer's perspectives on the proposed transfers have been addressed by DOD as part of the determination of any adverse industrial base effects. GAO analyzed the latest DOD data on EDA Humvee transfers from fiscal years 2012 through 2018; reviewed DOD policies, guidance, and documents to gain insight into the process for determining industrial base effects of proposed transfers; and interviewed agency officials and Humvee manufacturer representatives. Excess High Mobility Multipurpose Wheeled Vehicles (HMMWV)—commonly pronounced Humvees—are among thousands of items that the Department of Defense (DOD) can transfer to foreign governments at their request through the Excess Defense Articles (EDA) program. Twenty-three countries, primarily from the Middle East and Africa, requested 16,005 Humvees for the 7-year period GAO reviewed. DOD approves such requests if it determines: excess U.S. inventory is available at the time of the request, the request aligns with U.S. foreign policy objectives, such as using the vehicles to help combat terrorism, and the U.S. industrial base will not be adversely affected by the transfer. For example, DOD approved a country's request for excess Humvees for border security, counter-smuggling, and counter-terrorism efforts. DOD approved nearly half of the total Humvees requested for fiscal years 2012 through 2018 (see figure). However, DOD has halted further approvals since the start of fiscal year 2017 due to concerns expressed by the Humvee manufacturer and language in the FY 2018 National Defense Authorization Act (2018 NDAA) and conference report that generally says Humvees must be modernized at no cost to DOD. GAO found that DOD considered the Humvee manufacturer's perspectives on proposed transfers and generally took steps to mitigate concerns about transfers that could siphon potential business from the manufacturer or compete with its sales efforts. Further, GAO found that generally, when the manufacturer objected to a transfer, the manufacturer withdrew its objection after receiving business opportunities to repair or upgrade vehicles for DOD or a requesting government's fleet. DOD officials also noted that most of the countries requesting Humvees through the EDA program find it cost-prohibitive to purchase new Humvees directly from the manufacturer. As a result, these countries rely on EDA Humvees to sustain their military fleets.", "document_type": "gao"}
{"report": "We previously reported that DOL is one of more than a dozen federal agencies—known as National Drug Control Program agencies—that have responsibilities for drug prevention, treatment, and law enforcement activities. The Office of National Drug Control Policy (ONDCP) was established in 1988 to, among other things, enhance national drug control planning and coordination. As federal agencies engage in drug control efforts, ONDCP is responsible for, among other things, overseeing and coordinating the implementation of national drug control policy across the federal government. These responsibilities include promulgating a National Drug Control Strategy. In 2017 and 2018, ONDCP lacked a statutorily-required National Drug Control Strategy, and we recently reported that the 2019 National Drug Control Strategy did not fully comply with the law. In December 2019, we recommended that ONDCP develop and document key planning elements to help ONDCP structure its ongoing efforts and to better position the agency to meet these requirements for future iterations of the National Drug Control Strategy. We also found that the 2019 strategy did not contain several pieces of required information, such as quantifiable and measurable objectives, and specific targets for long-term goals, or a description of a performance measurement system. ONDCP subsequently issued the 2020 National Drug Control Strategy on February 3, 2020. We reviewed this Strategy and found that it made progress in addressing several statutory requirements but fell short in meeting others. Furthermore, in our March 2019 High-Risk report, we named drug misuse as an emerging issue requiring close attention. Based on our findings from a body of work related to drug misuse—including 25 new GAO products issued since our 2019 High-Risk report—we have determined that this issue should be on our High-Risk List. DOL’s Phase 1 and Phase 2 grants, targeted to support efforts for addressing the opioid crisis, are authorized by WIOA, which was enacted in 2014 and emphasizes the alignment and integration of workforce programs. ETA is responsible for some WIOA programs, which provide education and other services to help job seekers obtain employment and advance in the labor market, including job search assistance, career counseling, and a variety of occupational skills such as classroom and on-the-job training. In addition, WIOA emphasizes that employers are also customers of the workforce system, and includes provisions that involve them in helping the system provide the skilled workers they need. WIOA requires states to submit plans to DOL every 4 years, and updates to these plans every 2 years, that outline the state’s workforce strategies for core WIOA programs. The next state plans are due in 2020. WIOA gives state and local officials the flexibility to develop and fund services that meet the specific needs of their local communities and meet WIOA goals of increasing employment, retention, and earnings to promote economic self-sufficiency. To that end, WIOA core program performance measures and targets include those related to job attainment and retention; median earnings; and skill and credential attainment. DOL officials told us that states generally use the same WIOA performance measures for the Phase 1 and 2 grants as well. The WIOA-funded workforce development system provides services through a national network of approximately 2,400 American Job Centers (AJCs). State and local entities deliver WIOA-funded employment and training activities and coordinate with partner programs via the AJCs. ETA’s Phase 1 and 2 grants are intended, in part, to serve dislocated workers—adults whose jobs have been terminated, who have been laid- off, or who were self-employed. These grant funds are awarded to states, tribal governments, or outlying areas that, in turn, may work with local workforce boards to administer the grants. Grant recipients generally have 2 years to expend their funds. See table 2 for more information about these grants. Both grants require that recipients partner with community organizations, such as those in health care and justice systems, and with at least one local workforce development board or AJC. While grants cannot be used to pay the costs of in-patient drug treatment and in-patient rehabilitation programs, grantees may use some funding to provide supportive services to participants, such as assistance with child care. States may be using other federal funds to address the workforce impacts of the opioid crisis, including other WIOA-related funding. For example: Ohio received $8 million in September 2018 from DOL’s Trade and Economic Transition National Dislocated Worker Grant, which provides training and career services to dislocated workers affected by layoffs at one or more companies and are seeking reentry into the workforce. The state targeted 16 counties in the state that officials said had been hardest hit by the opioid crisis. State officials said they plan to use this grant to provide services to anyone who meets the criteria of a dislocated worker, and they felt the opioid crisis had a strong enough economic effect for the state to use the grant for those whose employment has been affected by the crisis. DOL’s Women’s Bureau granted Maryland $650,000 in September 2018 to fund two projects providing job-seeking supports to women affected by opioid use disorder. Pennsylvania plans to use HHS funding to expand treatment capacity for underserved populations through targeted workforce development, according to its grant application. Additionally, ETA has recently provided more funding opportunities to support state and local workforce efforts to address the opioid epidemic. In September 2019, ETA, in partnership with the Appalachian Regional Commission and the Delta Regional Authority, announced the 23 grantees on the first round of funding under the Workforce Opportunity for Rural Communities Initiative, which included a focus on serving individuals impacted by the opioid epidemic. Five of the ten awards in the Appalachian region committed to addressing opioid and other SUD impacts as part of their projects. Also, in October 2019, ETA announced another funding opportunity for $20 million in grants under the Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment for Patients and Communities (SUPPORT) Act. The SUPPORT Act directs DOL to conduct a pilot grant program to address the economic and workforce effects associated with SUDs. Beyond those recently funded, workforce efforts to address the opioid crisis may need to continue for many years given the nature of SUD. Research suggests that incentives for avoiding drug misuse, such as obtaining and maintaining employment, can be highly effective in promoting recovery from SUD. However, an estimated 40 to 60 percent of people with SUD experience relapse, according to the National Institute on Drug Abuse. As a result, people with SUD often need ongoing support to reduce this risk. Officials in the four selected states that received Phase 1 and Phase 2 DOL grants told us that the required partnerships with community organizations were essential in their efforts to serve those affected by SUD. These relationships fostered both knowledge sharing and coordination, elements especially important to state officials with limited experience serving this population. For example, in Ohio, state officials said that input from community partners, such as substance use disorder and mental health boards, helped them identify who could best provide supportive services for job seekers in recovery. We found workforce agencies in all four states receiving targeted DOL grants worked to serve job seekers with the following partners: Health care organizations. Workforce officials said they partner with health care organizations to identify people in recovery from SUD who are ready to look for employment. For example, New Hampshire state officials described the state’s “hub-and-spoke” services system, where health care entities such as hospitals refer people affected by SUD to various services. The health care staff coordinate with local workforce agency staff and notify them when an individual in recovery is ready for employment and training services. Other states described similar coordination of services. For example, a local workforce agency in Washington is partnering with a nonprofit health care organization to coordinate workforce development efforts with health and social services. Justice organizations. Workforce agencies partnered with drug courts, detention centers, and other facilities to address the employment readiness and support needs of those in the juvenile and adult justice systems who may have SUD. For example, in Washington, local workforce agency officials told us that they provide training and education services—including reentry workshops and work readiness services—for their area’s juvenile justice facilities, where over 70 percent of the population has a substance use disorder. In New Hampshire, state workforce officials described a partner organization whose officials have relationships with all of the drug courts in the state, and also sit on the board of the drug court in one of the state’s largest counties. They said that drug courts provide people an option to seek recovery services instead of criminal charges, and the local workforce agency provides employment services for people participating in drug courts. Educational institutions. Partnerships with community colleges and universities helped workforce agencies to provide employment training for job seekers interested in participating in recovery services. In two of our selected states, officials reported using funds to support the development of peer recovery specialists. Such peer recovery specialists, according to HHS, can bring the lived experience of recovery to assist others in initiating and maintaining recovery. For example, in Ohio, the state workforce agency partnered with a community college to help people to become peer recovery specialists and licensed chemical dependency counselors. Maryland provided Phase 1 Grant funds to a research-based organization, housed on the campus of state university, which is preparing peer recovery specialists. Furthermore, local workforce agency officials in Ohio also told us that they worked with a university to put together a master’s degree in social work for those with Licensed Social Worker credentials or a bachelor’s degree. Other organizations. Partnerships with community organizations and housing commissions helped states address transportation and housing needs through referrals and coordinated services. For example, local workforce officials in Washington told us they work with partners through subcontracts or memoranda of understanding to help job seekers with childcare and housing so they can attain and retain employment. Also, officials in New Hampshire told us that one state partner works with sober living houses, which are group homes in which people in recovery can live during and after treatment. Several state workforce officials we interviewed noted that a key benefit to the WIOA targeted-assistance grants was forging partnerships which will have lasting impacts on how they conduct services in the workforce system. For example, officials in New Hampshire noted that the state plans to continue to leverage relationships with their partners after the grant expires. Also, officials in Ohio said these partnerships put new processes in place, including referral systems that will facilitate getting people in recovery into the workforce system over the long term. States not receiving targeted grants. Workforce officials in Alabama and Arizona, the states we selected that did not receive targeted DOL grants but are still experiencing high levels of opioid misuse in their communities, stated that they were engaged in some newly formed partnerships to address the workforce aspects of SUD. Alabama workforce officials said they recently began participating in a statewide opioid task force, including serving on the workforce subcommittee with other state departments, such as the state Department of Commerce. Arizona officials said that the state workforce agency partners with the state Department of Corrections and has implemented second chance centers, which offer services such as job training and onsite job fairs, within three prisons. They noted that in one of these prisons, the majority of women are incarcerated for drug-related offenses. Officials in the four selected states that received targeted DOL grants said they used this funding to assist those in recovery from SUD to obtain employment. While many of the services are also offered to other job seekers, officials said grant-funded efforts involved intensive work with SUD-recovering individuals, who may have inconsistent work histories or long periods of unemployment. New Hampshire state workforce officials reported providing individuals in recovery with services, including job training, direct placement in a job, or on-the-job training. As of January 2020, officials said the state had enrolled 177 individuals into its program, including some who are participating in on-the-job training (employment that is partially subsidized by grant funds). Similarly, officials at a local workforce agency in Washington told us that the agency aims to place 125 people affected by SUD into transitional jobs as part of its grant-funded activities. These subsidized jobs allow individuals to add experience to their resumes, as well as gain an employment reference. In Maryland, the state distributed part of its Phase 1 grant funds to local workforce agencies in eight counties directly or indirectly affected by the opioid crisis. These funds provide job seekers with employment, training, and support services that help them prepare for, secure, and retain employment, and advance along career pathways in high-demand industries and occupations—including those related to SUD recovery, such as counseling. Similarly, Ohio workforce officials told us they were reintegrating individuals who are affected by opioid use into the workforce by using some of their Trade and Economic Transition National Dislocated Worker Grant funds to provide career services, guidance, and counseling, along with support services. Several officials noted that, while their agencies may use the same process for those with SUD as those without to get individuals ready for jobs, it is often a longer process when someone is in recovery or otherwise affected by SUD. For example, officials from a local workforce agency in Ohio told us that those in recovery from SUD often need more services and support to work through barriers prior to job placement than other clients without the disorder. Agency workforce staff are to follow up with people in recovery to make sure they are still supported, even after they have found employment or have enrolled in training—sometimes on a weekly basis. Officials said that those in recovery may not have previously had a job or attended post-secondary school before, and must balance their recovery with these new responsibilities. Similarly, state workforce officials in New Hampshire said that many in recovery have not had the opportunity to build skills and confidence. The New Hampshire Work Ready program is a 60-hour program offered through the state’s community colleges that provides help in areas such as how to dress for an interview and the workplace. This program, which is available to all job seekers, also helps people decide what to disclose regarding their personal history and helps them emphasize their strengths. Officials characterized this program as especially helpful for people with criminal backgrounds. In response to the needs of those in recovery, they said the state has created a new “bridge” program to prepare individuals to participate in the Work Ready program, which will be implemented in recovery centers using targeted grant funds. States not receiving targeted grants. Workforce officials in Alabama and Arizona, states that did not receive targeted grants, said that state efforts to address SUD, and more specifically opioid use disorder, were largely focused on the health aspects of the issue. Alabama officials told us that the state workforce agency was not originally part of the Governor’s task force on opioid use disorder. The task force’s recommendations were mostly health care related and addressed issues such as provider practices. However, the task force has recently added a workforce subcommittee with the goal of identifying strategies and resources to provide in-demand career pathways for those affected by SUD, and officials reported that they plan to apply for Phase 2 funding in the future. Arizona state officials said that its workforce development system provided support in communities, but noted that there is not a coordinated strategy statewide. Arizona officials also emphasized that they consider SUD primarily a public health issue, not a workforce issue; they said that while employment is part of a spectrum of services, SUD is an issue that is best addressed on the health side. To assist those affected by SUD in finding employment, local workforce agencies used their targeted grant funding to secure specialists. For example, officials at two local workforce agencies in Ohio told us they had hired or planned to hire new staff to work with the population affected by SUD. One agency plans to hire case managers specializing in mental health, who will team with AJC staff to help ensure clients in recovery get the support they need to be successful. The other agency plans to hire peer recovery specialists and job coaches to help those in recovery develop soft skills. One local workforce agency in Washington also hired peer recovery specialists, and is using them as case managers at an AJC. Another agency in Washington is using Phase 1 grant funds to employ four “navigators” to coordinate services to address the needs of those in recovery. In addition, officials said they are in the process of hiring a job developer to liaise between job seekers, navigators, and employers, and help recruit employers who are willing to hire those in recovery from SUD. Additionally, communities are exploring different workplace programs to support those in recovery. Officials in New Hampshire and Ohio reported using their Phase 1 and Trade and Economic Transition National Dislocated Worker Grant funds, respectively, to pilot recovery-friendly workplace initiatives, which provide training and supports to employers to help them better understand and work with individuals with SUD. Ohio state officials told us that, in three pilot counties, the state will train supervisors and managers and provide second-chance policies and employee assistance programs. According to these officials, recovery- friendly workplaces encourage an environment where employers, employees, and communities can collaborate to create positive change and eliminate barriers for those affected by SUD. In New Hampshire, employers may request that the state designate them as a recovery- friendly workplace. The New Hampshire workplace program will provide an advisor who conducts an orientation with management and staff and helps the employer publicize their participation in this effort so that their employees will know of their commitment, and will know their workplace is a safe place to disclose SUD. Employers in the program also agree to complete certain activities, such as conducting training and making connections with local recovery organizations. New Hampshire officials said they had 220 employers participating in the program as of January 2020. State and local workforce officials said that their efforts to meet the needs of job seekers and employers in communities affected by SUD are relatively new. For example, officials in Ohio said that state efforts are still very much in the preliminary planning stages of their broader implementation goals. They said that, at this point, they are looking at how to educate workforce agencies and staff about how to best address the needs of this population. State and local officials in our four selected states receiving targeted Phase 1 and Phase 2 grants were not yet able to report outcomes. Officials told us that it took time to organize and implement plans, causing delays in beginning activities. Specifically, workforce officials stated that: In Washington, officials said they received the notification for Phase 2 grant funding in March 2019. The state workforce agency finalized the contract with the local workforce agency at the end of May 2019, and began enrolling eligible job seekers in the late summer and early fall of 2019. In New Hampshire, it took the state six months to begin implementing grant activities after receiving funding in July 2018, and officials confirmed in January 2020 that they were still too early in addressing the opioid crisis to have any outcomes. In Maryland, officials originally planned to use funding to train peer recovery specialists to work in the state’s AJCs. However, the state Department of Health secured funding to train peer recovery specialists, and they did not want to duplicate efforts. As a result, they revised their plan to instead create an Opioid Workforce Innovation Fund, which delayed grant activities by six months or more. As of August 2019, Ohio officials said they were just starting to get the local workforce areas on board and acclimated. They reported that they had just completed training for the local workforce agencies on the grant rules and activities, and launched a toolkit to help agencies serve individuals with SUD. Workforce agency officials in all six of our selected states told us they face challenges addressing the needs of job seekers affected by SUD, in part due to their limited experience in serving this population. For example, Health issues. Officials in all six states said they continue to struggle with ensuring job seekers receive necessary services due to lack of medical treatment, mental health services, and recovery services and personnel, especially in rural areas. For example, officials at a local workforce agency in a rural area of Maryland said their area has no addiction specialists, and many people in the area have to travel nearly 2 hours to receive recovery treatment and counseling. Involvement with the justice system. Individuals in recovery may be more likely to have criminal records that complicate obtaining and maintaining employment. Officials in New Hampshire told us that employers might not hire people with a criminal history, and that employers are allowed to ask about criminal history on a job application, even if the individual is in long-term recovery. Appalachian Regional Commission officials said that job seekers with a criminal record also have especially limited employment options in their region because the federal government and its contractors are large employers there, but may not be able to hire someone with a felony conviction, which is an issue for many individuals with SUD. Transportation difficulties. Lack of reliable, affordable transportation presents difficulties for many in recovery. For example, New Hampshire officials told us many people with SUD have lost their license or have no car, and few public transportation options are available in the state outside of urban areas. Local workforce officials in a rural area of Ohio said no reliable public transportation exists near them, and the limited taxi service that exists is very expensive. Housing difficulties. Individuals in recovery may not have access to stable housing, making it difficult to focus on job training or employment. Specifically, officials in Maryland, Ohio, and Washington cited homelessness as an issue among those in recovery. Further, New Hampshire officials said individuals who have a drug conviction may not be eligible for government-subsidized housing. While homelessness can be a result of a substance use-related history, local officials in New Hampshire, Ohio, and Washington told us that there is also a lack of affordable housing in their respective areas. Workforce officials in all six selected states told us that they have had difficulty finding employers who are willing to hire those in recovery. As a result, workforce agencies risk not meeting WIOA performance targets related to (1) job seekers’ obtaining and maintaining employment and (2) effectiveness in serving employers. Workforce officials in all six states cited employer concerns around relapses, safety and reliability, suitability, and stigma. Relapses. Officials from the Appalachian Regional Commission said this was the most challenging aspect of SUD with respect to the workplace. Officials from another organization that works with employees with SUD also told us that employers may be reluctant to hire SUD-affected individuals because state laws or claims related to lack of reasonable accommodations under the Americans with Disabilities Act of 1990 can make it difficult to terminate individuals with a known substance disorder when they relapse. To address this, some employers put in place a zero- tolerance policy, automatically terminating an employee who tests positive for drugs. Safety/reliability. Workforce officials in Maryland said employers are concerned that SUD-affected employees may bring drugs into their workplaces or quit unexpectedly. New Hampshire officials told us that employer liability is an issue as employers are worried about accidents. They also told us employers are concerned about productivity loss due to SUD and, in particular, an employee’s inability to work a regular schedule because they or a family member is dealing with SUD. Ohio officials in one local area told us that employers in white-collar jobs are less willing to hire individuals in recovery because they are concerned about possible theft, and that workforce officials have been working with businesses to secure liability insurance. Suitability. Some employers will not hire a person who is unable to pass a drug test. This may present issues for individuals who take medication as part of their recovery treatments. For example, Alabama officials told us that a major reason that employers in their state did not hire job applicants for vacant positions was because they could not pass initial drug screenings. In addition, under U.S. Department of Transportation regulations on workplace drug and alcohol testing, when an employee performing safety-sensitive functions tests positive for drug use, they must be removed from performing such functions and evaluated for treatment options before returning to work. This includes those in aviation, trucking and locomotive transit. Certain entities regulated by the Nuclear Regulatory Commission are also required to administer drug and alcohol testing. Workforce officials in Washington said that it is also difficult for people with SUD to obtain the available jobs in their state in the health care field and with federal agencies because these jobs required drug testing. Stigma. Employers may also be reluctant to hire those affected by SUD because of its associated stigma. New Hampshire officials said that employers are concerned about people’s perceptions and believe it would hurt business if they declare themselves a recovery-friendly workplace. For example, they told us about an employer who runs a high-end restaurant in the state who expressed concern that customers may not want an individual with SUD preparing their food. Washington officials expressed similar concerns, saying that while some employers embrace being a recovery-friendly employer, others do not publicize this because they are unsure how it will be received by the public. Officials in Alabama also noted the need for honesty and transparency about the stigma of SUD and for employers who are willing to invest in their workers. According to DOL officials, they have begun working with ONDCP and other federal agencies to address the drug crisis. DOL officials noted that, although the National Drug Control Strategy does not include explicit goals and performance targets for DOL or employment and training- related efforts, DOL is using the strategy to guide its efforts in addressing the opioid crisis. DOL officials said they have regular conversations with ONDCP about how ETA can support the ONDCP strategy within its current authority. For example, one DOL official told us she communicates with ONDCP nearly every week. DOL officials also said they attend meetings hosted by ONDCP which occur roughly every 6 weeks and include representatives from all of the agencies involved in the National Drug Control Strategy. According to DOL officials, through these meetings, they have learned about government-wide efforts to support those affected by SUD, and have shared information about DOL’s own efforts to address the opioid crisis. DOL officials told us they communicate with other federal agencies regarding the opioid crisis. For example, DOL officials said that HHS provided a list of available grant funding to address the opioid crisis, and DOL has sent this list to its regional offices to distribute to states. In addition, ETA officials told us that two out of the six regional offices have staff serving on regional opioid task forces, for example, with HHS. DOL has also conducted several webinars with HHS on addressing training and employment needs of individuals and communities affected by SUD. Specifically, DOL officials described: a webinar in October 2018 discussing topics such as the rise in opioid use and a screening and intervention technique; a webinar in May 2019 for program staff working directly with participants in the workforce development programs located in states in the mid-Atlantic region, which are among those with the highest opioid-related deaths; and, a webinar with HHS, ONDCP, and other organizations in August 2019 on peer support recovery, including discussing how DOL grant funds have been used to train SUD-affected individuals to become peer recovery specialists. Internally, DOL officials told us they began a DOL-wide opioid workgroup in April 2019 to improve communication among units and strengthen connections across the agency. According to DOL officials and meeting agendas we reviewed, the workgroup meets about once a month, and discusses what DOL is doing to address the opioid crisis and identify any potential gaps in their efforts. They also invite speakers from external organizations, such as ONDCP, the Centers for Disease Control and Prevention, and the National Institute for Occupational Safety and Health. ETA officials have provided technical assistance to states during the Phase 1 and 2 grant application processes, such as by clarifying allowable grant-funded activities and defining grant eligibility, and during grant implementation. According to officials, ETA assigned Federal Project Officers from one of its six DOL regional offices to work with each state. Officials have also encouraged information sharing among grantees. For example, officials said they hosted quarterly calls among grantee states, where they discussed performance reporting, evaluation, and use of the Federal Bonding Program, and have allowed time for peer- to-peer sharing of grant accomplishments and challenges. To encourage peer-to-peer sharing and engagement, ETA also provided grantees with a list of grantee contacts in all states that received Phase 1 or 2 grants. However, this technical assistance has been limited to those receiving the targeted grants, and is not offered to all states, tribes, and outlying areas that may be interested in conducting related work. DOL officials are working to improve available information on addressing the employment and training needs of those affected by SUD. According to DOL officials, interested entities can access a DOL website called WorkforceGPS with resources and materials on substance abuse, including its effect on the workforce system, and case management resources. DOL also contracted with a research organization to review literature that examines what is known about workforce programs for individuals with SUD. The research is meant to identify key themes and findings related to successfully implementing the Phase 1 grants, such as the role of mental health services in the lives of grant participants and different employment-related interventions. DOL officials said that, as a complementary piece to the literature review, the contractor was tasked with developing a resource guide that identifies promising practices across the public and private sectors, with a goal of providing up to date information on tools, programs, websites from across the country to serve as a resource for grantees who are planning and implementing their own initiatives. Officials said that the contractor shared preliminary results from its research activities with targeted grantees in October 2019. Based on these results, DOL officials reported that there was a lack of evidence about the relationship between opioid use disorder and employment. Therefore, they said, the literature review covers a broader range of information related to SUD in an effort to provide useful information. DOL released the full results of the literature review and resource guide on its website in March 2020. Regarding oversight of grant activities, DOL plans to review grantee performance through required state quarterly reports, which have only recently begun to be submitted. DOL requires that these reports include financial data and program performance information (such as characteristics of, and services received by, participants, as well as participant outcomes). These quarterly reports also contain a narrative section where grantees can share information on project success stories, upcoming grant activities, and promising approaches. The final quarterly report for the grant must summarize the successes and/or challenges in delivering services, as well as address the topics of sustainability, replicability, and lessons learned. DOL officials said they do not have plans to share information from the summaries in the quarterly reports with other states. In addition, DOL officials told us that states generally are to use the same performance measures for these grants as they do for WIOA core programs. However, officials said they realize the SUD population could have different challenges than the rest of the WIOA population and, as a result, they are looking into developing new performance measures to address these differences. Regarding evaluation of grant activities, DOL has contracted with a research organization to conduct a 3-year evaluation of Phase 1 activities. The evaluation is expected to end in September 2021, with a final report to follow. DOL officials confirmed that there will be no interim reports. Although some state and local workforce officials we interviewed were aware of available technical assistance from DOL, they identified a need for more information to help them address challenges in serving communities affected by SUD, as discussed below. Furthermore, our review of DOL documents and guidance such as the ETA announcements to states of the targeted grants and the WIOA state plan guidance, found that these documents did not fully address the questions and concerns of state and local workforce officials. Federal internal control standards regarding risk assessment state that management should identify, analyze, and respond to risks related to achieving its objectives such as WIOA’s goals of increasing employment and retention. These standards also state that management should communicate with its partners to help achieve its objectives. Better communicating information could enhance DOL’s ability to respond to these risks. Specifically, state and local workforce officials and our review identified three areas in which additional DOL actions could help officials address the needs of job seekers in recovery and potential employers: Clarity about expectations and use of funds. Officials in Arizona, Ohio, and Washington said they would like clarification from DOL about its expectations regarding the role of state and local workforce systems in preparing individuals in SUD recovery for employment, or in determining the appropriate use of WIOA grant funds. Clarity around DOL’s expectations for state workforce agencies could be helpful, as Arizona officials emphasized that they consider SUD a public health issue, not a workforce issue, and have viewed SUD as an issue that is best addressed on the health side. Also, information on expectations and the use of non-targeted WIOA grant funds is especially important as states draft their 2020 WIOA state plans, which will set priorities for state workforce agencies for the next 4 years. For example, officials from one local area in Washington told us that they hoped to continue the grant activities and partnerships past the end of the current targeted grant, but they were unsure whether they could do this with non-targeted, WIOA formula grant funding. Our review of the targeted grant announcements found they did not contain information on whether this was an allowable use of funds. ETA issued guidance regarding the 2020 WIOA state plans in February 2020. However, our review of this guidance found that it does not provide specific information about states’ roles in meeting the needs of job seekers in recovery from SUD or their potential employers, or how non- targeted WIOA funding can be used to address those needs. DOL officials acknowledged that the guidance does not include such information, stating that the purpose of the guidance was to focus on the procedures and instructions for states in submitting their state plans, and not to provide specific suggestions on uses of WIOA funds or what particular strategies states should pursue. Clarity on the role of states and the use of WIOA funding would better position state workforce systems to meet the training and employment needs of those affected by SUD and their potential employers. Better information sharing with all states. Officials from four of the six selected states identified areas in which it would be useful for DOL to enhance its information sharing. Specifically, officials in these states told us that it would be useful for DOL to share information about lessons learned and successful strategies in addressing the needs of job seekers in recovery and potential employers with all states—whether or not they received targeted grants. They said such information would be particularly helpful given that many states are in the early stages of developing their programs. Officials stated that information based on the experiences of their peers would assist states in ensuring those in recovery are job ready and in hiring and retaining these workers. For example, officials from Arizona—a state without a targeted DOL grant—told us their communities could benefit from learning about experiences of states or local areas that are addressing the crisis within the workforce system, especially those using an approach that offers wraparound services such as transportation assistance. Additionally, officials from Ohio—a state with a targeted DOL grant—said they would like to learn from more experienced state officials who have been working for 6 months or a year within the workforce system to address the opioid crisis. Workforce officials stated that even if job seekers in recovery are trained and job ready, workforce agencies face challenges in addressing employers’ concerns about hiring these individuals. Workforce officials in five of the six selected states said that information about incentives for employers to hire individuals affected by SUD, and/or education for employers about this population, would be helpful given that perceived risks have led to difficulties with finding employers who are willing to hire this population. In particular, given limitations of federally supported incentive programs and the stigma associated with SUD, a dual approach—education and incentives—may be needed. However, at this point, most information on strategies to address employer concerns, including leveraging pre-existing federal programs, is not widely disseminated. DOL officials stated that they recognize the challenges state and local workforce agencies face in engaging employers in this area and are exploring use of existing programs to incentivize the hiring of job seekers with SUD. However, they acknowledged that to date, limited information has been shared with the large network of state and local workforce agencies. Thus far, DOL has been piloting and promoting one available incentive, the Federal Bonding Program, which is designed to help reduce employers’ risk by offering reimbursement for loss from illegal acts, such as theft or embezzlement, for individuals with criminal records. DOL officials have recognized that other existing incentive programs—targeted to employers of other populations, such as low-income, and other disadvantaged job seekers—may be helpful. They said that because the populations eligible for these programs share similar characteristics as those in recovery, they are exploring how to connect them to employers who are willing to hire those in recovery. For example, the Work Opportunity Tax Credit encourages employers to hire individuals from certain targeted groups who have consistently high unemployment rates, such as individuals with a felony record, by providing employers with a tax credit as an incentive to hire and retain these workers. However, state officials said, and our review confirmed, that these current federal programs may not fully address employer concerns. Specifically, bonds might not protect against other liabilities which may be of concern to employers, such as accidents caused by an employee under the influence of opioids. Furthermore, despite promoting awareness of these programs, DOL officials recognized that these efforts alone may not increase employer participation, particularly given the need to move beyond the stigma associated with that condition. Officials in two states told us that education is an important response in addressing employers’ concerns about the potential stigma associated with hiring individuals with SUD. For example, New Hampshire has a pilot program on recovery- friendly workplaces to educate employers about reducing stigma associated with SUD, as well as related human resource policies and employee assistance programs. Also, Arizona officials stated that workforce agencies need to understand the employer perspective and engage, educate, and involve employers. To date, DOL has been primarily communicating information about emerging, workforce system-based strategies to serve job seekers and employers affected by SUD with Phase 1 and 2 grantee states. As previously noted, DOL has an existing mechanism—its WorkforceGPS website—that could be used to share information more widely. Access to information on promising practices and lessons learned can help workforce agencies in all states learn about possible ways to address the needs of job seekers affected by SUD and their potential employers. More time to use grant funds. Officials in New Hampshire, Ohio, and Washington said that a longer time window in which to use the DOL grant funding would be helpful. For example, New Hampshire officials said the length of time needed for intake and enrollment for clients with SUD is longer than usual for a typical WIOA job seeker; therefore, more time to use the Phase 1 grant funds could help them with the more intense interventions. In addition, state workforce officials in Ohio told us it is complicated and takes time to develop new partnerships and trust at the local level, and to determine what the state and other partners can provide. Similarly, Washington state officials said the limit on the time allowed to use the Phase 1 and 2 grant funds has limited their ability to enroll job seekers in recovery and implement their partnerships. Specifically, state officials said that the delay in receiving funds means they will not have the full 2 years for grant activities. To meet DOL’s reporting deadlines, they will need to complete their activities earlier than anticipated. ETA officials told us that they are considering extending the Phase 1 and 2 grant periods for some states. In commenting on a draft of this report, they also said that If these limitations prevent a state from continuing its grant beyond a certain period of years, states can apply for a new grant should it still meet the conditions for eligibility, such as if the public health emergency declaration for the opioid crisis remains active. In light of the persistent nature of the drug crisis and the complex set of issues facing individuals on the path to recovery, workforce agencies are likely to continue facing challenges in meeting the needs of this population and their potential employers. As the agency responsible for the nation’s workforce system, DOL can play an important role in serving communities and individuals affected by SUD who are seeking employment. However, state officials we interviewed expressed uncertainty about what is expected of them or the specific allowable uses of their non-targeted WIOA funds to address a crisis that has long been considered primarily a health and law enforcement issue. Our work raises concerns about how the workforce system continues to seek clearer direction on the role of states and the use of non-targeted WIOA grant funding in helping ensure the economic well-being of communities affected by this public health emergency. DOL’s current efforts are still in the early stages, and it will take time for the agency to fully identify and disseminate effective, evidence-based strategies. In the meantime, states are seeking the best information currently available to help their workforce systems support job seekers affected by SUD and their potential employers. DOL’s targeted grants provide an opportunity for grantees and non-grantees alike to learn states’ experiences in addressing the effects of the opioid crisis through the workforce system, but information on the current approaches states receiving targeted grants are using is not being shared beyond the targeted grantee community. Sharing this information with all states could better position workforce agencies to address the needs of job seekers affected by SUD and help employers understand and address the perceived risks of hiring job seekers in recovery. While the workforce system may take time to fully build its capacity to work with these job seekers and employers, opportunities exist to learn and make interim progress towards this end. We are making the following two recommendations to DOL: The Assistant Secretary for Employment and Training should clarify DOL’s expectations of the role of state workforce agencies in addressing the employment and training needs of those affected by SUD and how non-targeted WIOA funding can be used to assist job seekers and employers. (Recommendation 1) The Assistant Secretary for Employment and Training should share information from targeted grantees with all state workforce agencies, tribal governments, and outlying areas regarding lessons learned and promising practices in addressing the needs of job seekers affected by SUD and potential employers. (Recommendation 2) We provided a draft of this report to DOL and HHS for review and comment. In its formal comments, which are reproduced in appendix I, DOL agreed with our recommendations. DOL also provided technical comments, which we incorporated as appropriate. HHS did not have comments. In its response, DOL noted that throughout our report, we refer to SUD, but that its targeted grants are limited to addressing SUD caused by opioids. While our report focuses on SUD more broadly, many of the efforts states and federal agencies are involved in focus on opioid use disorder, as a result of HHS’s emergency declaration. DOL also stated that it was in the process of announcing another round of grants in partnership with the Delta Regional Authority and the Appalachian Regional Commission, part of which will be available to address opioid or other SUD. DOL also noted that grant-funding limitations, including the availability of appropriated funds, make it difficult to address states’ concerns about not having enough time to spend their grant funds, and suggested that states may consider applying for a new grant. We have reflected this point of view in the final report. In response to our first recommendation, DOL officials said they anticipate providing information and technical assistance to help workforce system grantees understand how they can address the impacts of SUD on the workforce. ETA plans to issue guidance by the end of 2020 to share promising practices and describe how WIOA funds can be used to support job seekers in recovery and employers. In response to our second recommendation, DOL officials said ETA has created resources that are available to all states based on its experience administering some of the targeted grants. ETA officials cited the recently published literature review and companion resource guide, and said they also plan to share the evaluation of the Phase 1 grants widely when it is available, including any resources or tools developed by states that were awarded Phase 1 grants. In addition, ETA plans to host at least one webinar to share additional promising practices from the targeted grants that could be useful to local workforce boards around the country. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Labor, the Secretary of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Danielle Giese (Assistant Director), Amy Sweet (Analyst-in-Charge), Linda Lootens Siegel, and Anna Kallschmidt made key contributions to this report. Also contributing to this report were Deborah Bland, Alex Galuten, Natalie Herzog, Tom James, Bill Keller, Sheila R. McCoy, Corinna Nicolaou, Monica Savoy, Almeta Spencer, Tonnye Connor-White, and Greg Whitney.", "summary": "The Department of Health and Human Services declared the opioid crisis a public health emergency in October 2017. DOL has awarded grants to help address this crisis. GAO was asked to examine how WIOA-funded programs are addressing the employment and training needs of those affected by SUD. This report examines (1) how workforce agencies in selected states are using WIOA funding to address employment and training needs, (2) challenges agencies face in addressing employment and training needs, and (3) how DOL is supporting communities affected by SUD. GAO interviewed officials in four of the 10 states that received DOL grants in the early award rounds (as of March 2019)—Maryland, New Hampshire, Ohio, and Washington—and two that did not—Alabama and Arizona; reviewed related documentation and relevant federal laws and regulations; and interviewed DOL officials and researchers, selected for their knowledge about these issues. Workforce officials GAO interviewed in four of the 10 states receiving targeted Department of Labor (DOL) grants as of March 2019 said they were using Workforce Innovation and Opportunity Act (WIOA) funding to help meet the unique needs of those affected by substance use disorder (SUD). These officials, who said they had limited experience serving those affected by SUD, worked with required organizational partners and hired specialists to assist job seekers and to provide intensive job readiness services. However, these efforts are relatively new and outcomes are not yet known. Workforce officials GAO interviewed in two selected states without targeted grants said they had viewed SUD primarily as a public health issue, but had recently taken some steps to address it. For example, one state added a workforce subcommittee to an existing opioid task force. State and local workforce officials in all six states identified a range of challenges they face in addressing the needs of SUD-affected job seekers. For example, criminal history or a lack of transportation may make it difficult for these job seekers to obtain and maintain employment. Officials said another challenge is finding employers who are willing to hire those in recovery. They stated that employers are concerned about the risks to their businesses, such as potential employee relapse and possible negative reaction from customers. Officials were seeking more information and assistance to help address such concerns. DOL officials said they support SUD-affected communities mainly by providing information to states that apply for and receive targeted grants. However, officials in two selected states expressed uncertainty about DOL's expectations of states in serving the needs of SUD-affected job seekers and potential employers. Officials in another state said they were unclear on whether they could use non-targeted funds to continue targeted grant activities. GAO's review of related DOL guidance found that it does not provide specific information on expectations of states or the use of WIOA funds outside of targeted grants to address this issue. Further, while DOL has disseminated some information on serving job seekers with SUD (such as in quarterly calls with grant recipients), it does not plan to share information that grantees submit to the agency, such as lessons learned and successes, with all states. Doing so could help states meet the training and employment needs of those in recovery, and the needs of potential employers. GAO recommends that DOL clarify (1) its expectations of state workforce agencies and (2) how WIOA funding can be used in addressing the needs of those affected by SUD and potential employers, and share information with all states on lessons learned and promising practices. DOL agreed with our recommendations.", "document_type": "gao"}
{"report": "Victims of sexual assault may receive a sexual assault forensic examination by a medical provider who may or may not be a trained sexual assault forensic examiner. Medical providers assess victims’ clinical conditions; provide appropriate treatment and medical referrals; and, given consent by the victim, collect forensic evidence through a sexual assault forensic examination that may follow steps and use supplies from a sexual assault evidence collection kit. Under its protocol for sexual assault forensic examinations, the Department of Justice (DOJ) recommends that medical providers collect a range of physical evidence. In addition, sexual assault forensic exams typically include documenting biological and physical findings such as cuts or bruises and a victim’s medical forensic history, such as the time and nature of the assault. Once the exam is complete, medical providers preserve the collected evidence, which may include packaging, labeling, and sealing evidence collection kits and storing kits in a secure location. Medical providers typically perform such exams only for acute cases of sexual assault, such as in cases where the assault occurred within the previous 72 to 96 hours, when the physical and biological evidence on a person’s body or clothes is considered most viable. DOJ, IAFN, and the American College of Emergency Physicians (ACEP) recommend that sexual assault forensic exams be performed by specially trained medical providers—known as sexual assault forensic examiners (examiners). These examiners include physicians, physician assistants, nurse practitioners, and other registered nurses who have been specially educated and have completed clinical requirements to perform sexual assault forensic exams. Sexual assault nurse examiners (SANE) —a particular type of sexual assault forensic examiner—are registered nurses, including nurse midwives and other advanced practice nurses, who have received specialized education and have fulfilled clinical requirements to perform sexual assault forensic exams. Examiner programs have been created in hospital or non-hospital settings whereby specially trained examiners are available to provide first-response care and exams to sexual assault victims. DOJ, IAFN, and some states have issued guidelines pertaining to the minimum level of training examiners should receive in order to properly collect and preserve evidence, identify victims’ medical and emotional health care needs, and provide counseling and referrals for victims. These guidelines include recommendations of objectives and topics that training programs should cover. DOJ administers several grant programs that aim to, among other things, improve response to and recovery from four broad categories of victimization—domestic violence, sexual assault, dating violence, and stalking. In our March 2016 report we describe three key grant programs administered by DOJ’s Office on Violence Against Women that could be used by grant recipients—including states or other eligible entities—to fund or train sexual assault forensic examiners. In our March 2016 report examining the availability of sexual assault forensic examiners, we found that only limited nationwide data exist on the availability of sexual assault forensic examiners—that is, both the number of practicing examiners and health care facilities that have examiner programs. While IAFN reported that, as of September 2015, there were 1,182 nurses with an active IAFN SANE certification in the United States, such data do not represent all practicing examiners nationwide. For example, the data do not account for examiners who completed training through an IAFN or a state training program but never became certified or were certified through another entity, such as a state board of nursing. IAFN also collects data on examiner programs nationwide—that is, data on hospitals, clinics, and other sites where examiners practice. Such data provide an indication of the availability of examiners, but the data are also limited. While 703 examiner programs nationwide voluntarily reported to IAFN’s examiner program database, as of September 2015, IAFN officials noted that the database is often not up to date; and some health care settings where sexual assault forensic exams are conducted, such as child advocacy centers, are not represented. In addition, data collected on staffing characteristics of examiner programs are often unavailable in the IAFN examiner program database. For example, only about one-third of the examiner programs reported on the number of examiners practicing in their program, and about one-third reported on whether examiners were available on-site versus on-call. In three of the six selected states we reviewed in our March 2016 report, grant administrators or officials from sexual assault coalitions were able to provide estimates of the number of practicing examiners, and, in all six states, they were able to provide information on the estimated number of examiner program locations in their state. Of states that reported, the number of practicing examiners and examiner programs varied by state. (See table 1.) However, such data may also present an incomplete picture of the availability of examiners. For example, only one of the six selected states has a system in place to formally track the number and location of examiners. Instead, officials generally reported on the estimated number of examiners or examiner locations that were part of a statewide examiner program or were identified through an ad hoc data collection effort. Although data are limited, grant administrators and sexual assault coalition officials in all six selected states nevertheless told us that the number of examiners available does not meet the need for exams within their states. For example, coalition officials in Wisconsin told us that nearly half of all counties in the state do not have any examiner programs available, and coalition officials in Nebraska told us that most counties in the state do not have examiner programs available. In addition, in four of the six selected states—Colorado, Florida, Nebraska, and Wisconsin— state grant administrators and coalition officials told us that few or some health care facilities in their state have examiners available. As a consequence, officials said victims may need to travel long distances to be examined by a trained examiner or be examined by a medical professional without specialized training. While in the other two selected states—Massachusetts and Oregon—state grant administrators and coalition officials stated that some or most facilities have examiners available, they noted that there is still a need for additional capacity to reduce the burden on those examiners who are available, or to make examiners available in a number of areas where examiners are currently unavailable. In health care facilities where examiners are available, they are typically available through hospitals on an on-call basis, according to literature we reviewed as well as all grant administrators and coalition officials we interviewed for our report. In addition, among facilities that have examiners available, the number of examiners available varies and may not provide enough capacity for facilities to offer examiner coverage 24 hours, 7 days a week, according to state grant administrators and coalition officials we interviewed. Nebraska coalition officials, for example, told us that while one hospital in Omaha has a team of 26 examiners available, other facilities in the state may have as few as three examiners available. Further, officials from Florida and Colorado told us that there are few facilities in their states able to offer full coverage with examiners available 24 hours, 7 days a week. In our March 2016 report, we found that maintaining a supply of trained examiners that meets communities’ needs for exams is challenging for multiple reasons, and that state officials have employed a variety of strategies to address these challenges, as described below. Limited availability of training. Officials in five of the six selected states told us that the limited availability of classroom, clinical, or continuing education training is a barrier to maintaining a supply of trained examiners. Regarding classroom training, some officials told us that training may only be offered once per year in their states. Additionally, officials from both Florida and IAFN told us that there is a need for qualified instructors to run training sessions. Experts and officials from Colorado, Nebraska, and Oregon also told us that medical professionals in rural areas may have difficulty completing the clinical training necessary to become an examiner. Obtaining clinical experience, such as performing exams under the supervision of a trained examiner, is a particular challenge in rural areas where hospitals may treat only a few sexual assault cases per year. One official in Nebraska told us that trained examiners in rural areas might not feel competent to perform exams due to the low number of cases they treat. A lack of continuing education opportunities may also pose a challenge for examiners in maintaining the skills necessary to perform exams. For example, the National Sexual Violence Resource Center (NSVRC) reported that— based on common challenges identified through a survey of, and group discussions among, examiner program coordinators—maintaining competency may be difficult for nurses in rural areas due to a low volume of patients presenting in need of exams and limited access to ongoing and advanced training. Officials told us they have been able to increase the availability of examiner training through alternative training methods such as web- based training courses and simulated clinical training. For example, officials in Colorado told us their state’s web-based examiner training program has made training less expensive and has increased examiner recruitment. Officials in Wisconsin told us they developed a clinical training lab that allows examiners to gain hands-on experience by performing elements of exams on experienced teaching assistants hired for the purpose of training new examiners. Further, in 2014, a DOJ- funded evaluation of examiner training programs found that a web-based training course may help increase the availability of trained examiners; the study also found that implementing web-based training had benefits such as decreasing the costs associated with attending in-person training, expanding training opportunities to remote areas, and allowing examiners to be trained by national experts. Lack of technical assistance and other supportive resources. Officials in four of the six selected states told us that the limited availability of technical assistance and other supportive resources for examiners poses a challenge to maintaining a supply of trained examiners. For example, officials in Florida, Nebraska, Oregon, and Wisconsin explained that, in general, there is a lack of mentorship opportunities and leadership within the examiner community. Officials also noted that the sustainability of examiner programs may be threatened by a lack of internal capacity, such as not having a full-time, paid examiner program coordinator available. Further, in its survey of and group discussions with examiner program coordinators, NSVRC found that examiners and examiner programs needed technical assistance and support in the following areas: aspects of performing exams, training, leadership development and policy issues, and examiner program sustainability. Officials we spoke to told us about strategies that can be used to increase support for examiners and examiner programs, such as offering web- based technical assistance. For example, officials in Massachusetts told us that, through their National Sexual Assault TeleNursing Center, trained SANEs provide remote clinical guidance to two hospitals in the state that do not have trained examiners available. In addition, officials from Colorado told us an examiner program coordinator in an urban hospital in the state provides volunteer on-call technical assistance and clinical guidance to examiners in rural parts of the state, where those resources are not otherwise available. Further, one study we reviewed found several states were engaged in promising practices to increase support for examiners, such as implementing state-wide mentorship programs, developing regional examiner list-serves and online discussion boards, creating formal leadership positions within the examiner community, and requiring examiner program evaluations. Weak stakeholder support for examiners. Officials in five of the six selected states told us that limited stakeholder support for examiners and examiner programs, such as from hospitals and law enforcement, is a challenge to maintaining a supply of trained examiners. Some officials told us that hospitals may be reluctant to support examiners and examiner programs due to a low number of sexual assault cases treated each year. One official told us that hospitals may be reluctant to send nurses to examiner training, as it takes away from their regular shift availability. Additionally, some hospitals do not pay examiners to be on call. Officials in three states told us that hospitals typically either do not pay examiners to be on call or pay on-call examiners significantly less than other on-call medical professionals. Apart from hospital support, officials in Colorado and Oregon explained there is a need for more multidisciplinary support for examiners, such as increased law enforcement, prosecutor, and first-responder understanding of examiners’ role. The literature we reviewed also shows that ambiguity around the role of the examiner in responding to sexual assault may be a source of conflict between examiners and other professionals. For example, examiners were found to have experienced instances where victim advocates or law enforcement questioned examiners’ medical decisions, speed of evidence collection, or asked examiners to comment on the credibility of a victim’s case. One nationally representative survey of examiner programs found that examiner program coordinators felt ongoing education of community stakeholders on sexual assault and examiner programs was needed due to the high turnover in staff at relevant community institutions and agencies, such as law enforcement officers, victim advocates, and prosecutors. Through our interviews with officials, we learned of strategies selected states have used to increase or mitigate limited stakeholder support for examiners and examiner programs. For example, officials in Colorado, Florida, Nebraska, Oregon, and Wisconsin told us that sexual assault response teams have been developed in their states to help community stakeholders to understand examiners’ role and better coordinate to meet the medical and legal needs of sexual assault victims. Low examiner retention rates. Officials in four of the six selected states told us that low examiner retention rates can be an impediment to maintaining a supply of trained examiners. In addition to the challenges of limited training opportunities, technical assistance and other supportive resources, and stakeholder support for examiners, the physically and emotionally demanding nature of examiner work contributes to low examiner retention rates. Further, studies have indicated that dissatisfaction with compensation, long work hours, and lack of support, among other things, may contribute to examiner burnout. Examiners typically work on call in addition to their full time jobs as, for example, emergency department nurses. Officials in Florida told us that examiners may be on call for 6-hour, 12-hour, or even 24-hour shifts. Further, one survey of examiner programs in Maryland found that examiners were required to be on call for an average of 159 hours per month. Wisconsin officials estimated that, although 540 SANEs were trained over a 2-year period, only 42 (less than 8 percent) were still practicing in the state at the end of those 2 years. Chairman Harper, Ranking Member DeGette, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. For further information about this statement, please contact A. Nicole Clowers at (202) 512-7114 or clowersa@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony. In addition to the contact named above, key contributors to this statement were Kristi Peterson (Assistant Director), Patricia Roy, Katherine Mack, Laurie Pachter, and Emily Wilson. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In 2016, about 323,000 individuals age 12 or older were reported victims of sexual assault, according to the Bureau of Justice Statistics. Studies have shown that exams performed by sexual assault forensic examiners—medical providers trained in collecting and preserving forensic evidence—may result in better physical and mental health care for victims, better evidence collection, and higher prosecution rates. Yet, concerns have been raised about the availability of such examiners. The Department of Justice administers grant programs that can be used by states and other eligible entities to train and fund examiners. This statement summarizes GAO's findings from its March 2016 report ( GAO-16-334 ) describing (1) what was known in 2016 about the availability of sexual assault forensic examiners nationally and in selected states and (2) the challenges selected states faced in maintaining a supply of sexual assault forensic examiners. For that report, GAO reviewed literature on the availability of examiners and challenges training and retaining them. GAO also interviewed knowledgeable officials, including recipients of federal sexual assault examiner related grants and officials from sexual assault coalitions in six states (Colorado, Florida, Massachusetts, Nebraska, Oregon, and Wisconsin) selected to achieve variation in factors such as population and geographic location. GAO's March 2016 report examining the availability of sexual assault forensic examiners found that only limited nationwide data existed on the availability of sexual assault forensic examiners—both the number of practicing examiners and health care facilities that had examiner programs. At the state level, GAO found that, in three of the six states it selected to review, grant administrators or officials from sexual assault coalitions were able to provide estimates of the number of practicing examiners and, in all six states, they were able to provide information on the estimated number of examiner program locations in their state. However, officials in all six selected states told GAO that the number of examiners available in their state did not meet the need for exams, especially in rural areas. For example, officials in Wisconsin explained that nearly half of all counties in the state did not have any sexual assault examiner programs available and officials in Nebraska told GAO that most counties in the state did not have examiner programs available. As a consequence, officials said victims may need to travel long distances to be examined by a trained examiner. In health care facilities where examiners were available, they were typically available in hospitals on an on-call basis, though the number available varied by facility and may not provide enough capacity to offer examiner coverage 24 hours, 7 days a week. GAO's March 2016 report also found there were multiple challenges to maintaining a supply of examiners, according to its review of the literature and interviews with officials in the six selected states. These challenges include: Limited availability of training . Officials in five of the six selected states reported that the limited availability of classroom, clinical, and continuing education training opportunities is a challenge to maintaining a supply of trained examiners. For example, officials told us that there is a need for qualified instructors to run training sessions. Weak stakeholder support for examiners. Officials in five of the six selected states reported that obtaining support from stakeholders, such as hospitals, was a challenge. For example, hospitals may be reluctant to cover the costs of training examiners or pay for examiners to be on call. Low examiner retention rates. The above-mentioned and other challenges, including the emotional and physical demands on examiners, contribute to low examiner retention rates. Officials in one of the selected states estimated that while the state trained 540 examiners over a two-year period, only 42 of those examiners were still practicing in the state at the end of those 2 years. Officials described a variety of strategies they have employed that can help address these challenges, such as implementing web-based training courses, clinical practice labs, mentorship programs, and multidisciplinary teams that respond to cases of sexual assault.", "document_type": "gao"}
{"report": "Airlines commonly overbook their flights to avoid revenue losses associated with passenger no-shows as part of their revenue management strategies. Successfully overbooking requires that airlines accurately predict the number of passengers who will not show up for a given flight. In deciding how much to overbook flights, airlines use historical data to identify factors that make passengers more or less likely to show up for their flights; these factors can be passenger or flight specific. For example, according to representatives from an airline industry association, leisure passengers are less likely than business passengers to change their flights at the last minute, because their tickets typically have more restrictions and higher change fees. As a result, according to these representatives, airlines generally oversell fewer seats on flights heavily traveled by leisure passengers, such as flights during the holiday season or flights to common vacation destinations (e.g., Disney World). Similarly, these same representatives said that airlines are less likely to overbook the last flight of the day on a given route because passengers are more likely to show up for these flights. A number of other factors, in addition to overbookings, can lead to airlines denying boarding to passengers. These factors can be driven by safety concerns, operational necessity, or personnel needs. For example, a passenger may be denied boarding for safety or security reasons if they are too intoxicated to fly or if they are unruly (e.g., they get into a fight). Passengers may also be denied boarding to accommodate flight crews that need to get to a different location or U.S. air marshals, who tend to book flights near planned departure times. DOT does not regulate airlines’ overbooking practices, aside from requiring airlines to inform passengers that a flight may be overbooked. Instead, DOT’s regulations primarily focus on oversales, which can be the result of an overbooking and occur when some passengers with confirmed space on a flight cannot be accommodated (i.e., “denied boardings”). Passengers are voluntarily denied boarding if they willingly accept the airline’s offer of compensation, in any amount, in exchange for relinquishing their confirmed seat. Any other passenger denied boarding is considered to have been denied boarding involuntarily. Because of these regulations, airlines generally have a standard process for denying boarding to passengers, both voluntarily and involuntarily, and communicating denied boarding information to passengers. When a flight is oversold, airlines are required to solicit passengers to voluntarily give up their seats, before denying boarding to passengers involuntarily. To encourage passengers to volunteer to relinquish their seat, airlines may offer incentives, such as money or vouchers for future flights. There is no minimum or maximum amount of money or vouchers that the airline is required to offer, and passengers can negotiate compensation amounts. Federal regulation requires that airlines inform each passenger solicited to volunteer for denied boarding whether they are in danger of being involuntarily denied boarding and, if so, the compensation the airline is obligated to pay. In cases where a flight is oversold and airlines do not get enough volunteers who are willing to relinquish their seat, they will select passengers to give up their seats involuntarily—sometimes referred to as being “bumped.” Airlines are required by regulation to establish boarding priority rules detailing the factors they consider when selecting passengers to be denied boarding involuntarily. These factors may include when the passenger checks in, the fare paid, and the passenger’s frequent flyer status. However, according to DOT’s website, the criteria cannot subject a passenger to any unjust or unreasonable prejudice or disadvantage. For example, an airline cannot use a passenger’s race when making decisions about denied boardings. Further, some airlines make exceptions to their boarding priority rules for passengers with disabilities, including generally not denying them boarding. Airlines are required to compensate certain passengers who are denied boarding involuntarily. Minimum compensation amounts are set in regulation and, as shown in table 1 below, vary based on the price of the ticket, the length of time the passenger is delayed reaching their destination, and whether the flight’s arrival airport is domestic or international. Airlines generally must provide compensation by cash or check when the passenger is denied boarding involuntarily, in addition to a written statement explaining the terms, conditions, and limitations of the compensation, and describing the airlines’ boarding priority rules and criteria. The total number of passengers denied boarding—voluntarily or involuntarily— generally decreased from 2012 to 2018. Moreover, denied boardings represented a small percentage of the total number of passengers who boarded flights. On an annual basis, denied boardings accounted for between about 44 (in 2018) and about 100 (in 2012) passengers per 100,000 actual boardings—a rate of less than 0.1 percent of actual boardings. As illustrated in figure 1, of these passengers denied boarding, most are voluntary. For example, in 2018, for every 100,000 actual boardings, about 43 passengers were voluntarily denied boarding and about one passenger was involuntarily denied boarding. Passenger complaints submitted directly to DOT about denied boardings also generally decreased from 2012 to 2018, relative to total complaints and passenger boardings. As shown in figure 2, the number of passenger complaints to DOT about denied boardings represented a small percentage of total passenger complaints from 2012 to 2018, annually accounting for less than 4 percent of all complaints. On an annual basis, from 2012 through 2018, the number of complaints about denied boardings reported to DOT ranged from about 410 (in 2018) to about 650 (in 2015). We have previously reported, however, that DOT’s complaint data provide an incomplete picture of all passenger complaints because passengers may not be aware that they can report complaints to DOT, and DOT’s complaint data do not include complaints from passengers submitted directly to airlines. Specifically, in 2018, we reported that across all complaint categories, DOT estimated it received one complaint for every 50 complaints the airline receives. In an effort to avoid denied boardings, airlines can, in some cases, accommodate passengers in a different section of the aircraft, either by upgrading or downgrading passengers. A revenue management specialist and representatives from an airline industry association we interviewed said that, with limited exceptions, airlines generally do not overbook their premium cabins. Our review of DOT data found that in recent years, until 2018, airlines have generally upgraded fewer passengers to avoid denied boardings. According to representatives from an airline industry association, the decrease in the number of passengers upgraded is likely because airlines have fewer empty premium seats in their first-class cabins than in past years because they are selling more of these seats. For example, a stakeholder said that airlines are now selling upgrades on the day of departure and allowing more customers to use miles to upgrade their seat, leaving fewer available empty premium seats when flights are oversold. DOT permits airlines to downgrade passengers, as long as the airline refunds the passenger the difference in fares. In practice, representatives from an airline industry association said that when a passenger in a premium cabin is to be denied boarding, airlines generally offer the passenger the option of a premium cabin seat on another flight or to downgrade to the economy cabin along with compensation for the fare differential. In our review of seven airline’s contracts of carriage, five explicitly stated that if passengers are downgraded, they will be entitled to an appropriate refund, and the other two airlines do not include information about downgrades in their documents because they do not have different cabins of service. According to representatives from an airline industry association, the refund amount is calculated based on the average difference of fare paid between the two cabins, and it is dependent on the flight’s origin and destination. While the average amount of compensation for passengers involuntarily denied boarding has increased in recent years, a smaller percentage of such passengers received compensation. As previously mentioned, in certain situations, passengers who are denied boarding involuntarily may not be eligible for compensation. For example, airlines are not required to compensate passengers if an airline uses a smaller aircraft than originally planned for operational or safety reasons and thus cannot accommodate all confirmed passengers. Our review found that the percentage of passengers that were involuntarily denied boarding who qualified for compensation decreased from 76 percent in 2012 to 64 percent in 2018. Aircraft substitution may be contributing to fewer passengers being eligible for compensation, according to DOT data. For example, one airline that does not overbook experienced a number of operational issues in 2016 and 2017 that forced it to operate many of its flights with smaller aircraft. As a result, the airline had to deny passengers boarding involuntarily, and these passengers were not eligible for compensation. As figure 3 shows, from 2015 to 2018, most of the passengers who were denied boarding involuntarily and were not eligible for compensation were ineligible due to airlines using smaller aircraft on some flights. Although the total number of involuntary denied boardings decreased from 2012 to 2018, any passenger involuntarily denied boarding could face varying levels of disruptions to their travel plans. Passengers who are rebooked on the next scheduled flight may encounter minimal inconveniences or expenses. However, other passengers may face more significant travel disruptions, according to representatives from consumer advocate organizations we interviewed. Our review of a non- generalizable sample of passenger complaints submitted to DOT in May and June 2019 also identified instances where passengers reported incurring significant costs in terms of time and money as a result of being denied boarding involuntarily. For example, one passenger reported missing a wedding and paid about $450 in additional hotel costs. In another instance, a passenger missed their cruise after being denied boarding involuntarily. Consumer advocates also told us that passengers may incur costs such as lodging, meals, and transportation, or might miss work as a result of being denied boarding involuntarily. Airlines’ ability to rebook passengers who are involuntarily denied boarding on the next available flight can be limited. Over the past several years, airlines have increasingly flown with fewer empty seats— particularly on certain routes—than was typical in the past, according to DOT data. With fewer open seats, airlines have limited options to rebook passengers who are denied boarding. For example, across all departing flights at Hartsfield-Jackson Atlanta International Airport in 2018, on average, 86 percent of seats were filled. These data represent averages across all flights and stakeholders said that factors such as time of day, day of the week, season, and flight origin or destination can affect the number of empty seats on a particular flight. For example, flights on Sunday evening tend to be fuller than flights on Tuesday. One airline revenue management specialist estimated that about 25 to 30 percent of all flights have no empty seats. Representatives from consumer advocate organizations that we interviewed said that planes are operating at record-high levels of capacity, and one advocate stated that no transportation system is designed to operate at or near capacity all of the time, which they believe some airlines are doing on certain routes. In addition, we have previously reported that service to smaller communities is generally less frequent, providing airlines with fewer opportunities to rebook passengers than for more traveled routes. Airlines may also not be able to rebook passengers who are denied boarding on a different airline that has seat availability if they lack commercial agreements to do so. Further, according to representatives from an industry association representing airlines, while most airlines have agreements in place that allow passengers to be rebooked on a different airline, these agreements are primarily used to accommodate passengers on delayed and canceled flights. According to these representatives, passengers who are denied boarding are almost always re-accommodated on the same airline, given that the customer typically volunteers to take a later flight on the same day. Our review of seven airlines’ contracts of carriage found that four of them have documented policies in place to rebook passengers who are denied boarding on a different airline. Our review of DOT data found that fewer passengers are being rebooked on flights that arrive within an hour of their original flight. Specifically, in 2012, 11.5 percent of rebooked passengers were accommodated on such a flight, compared to 0.11 percent in 2018. While DOT collects data on passengers who are delayed less than an hour, no other information is available to measure the amount of time a passenger is delayed when they are denied boarding. However, based on our review of passenger complaints, we found instances where passengers reported having to wait until the following day to board a flight with available seats. Decreases in involuntary denied boardings are due in part to recent airline actions. As mentioned previously, involuntary denied boardings can be costly for both passengers whose travel plans are disrupted, and airlines that have to compensate passengers for such disruptions and then face criticism for denying boarding to passengers with confirmed seats. As a result, airlines have taken a range of actions, primarily intended to reduce such incidents. Some of these actions also provide additional incentives for passengers to volunteer to be denied boarding. Moreover, stakeholders, including consumer advocates and an association representing airlines, agreed that voluntary denied boardings are preferred to involuntary denied boardings, given that airlines and passengers willingly accept the outcome. Reducing the rate or eliminating overbookings. Some airlines have reduced their rate of overbooking or eliminated them altogether in an effort to reduce voluntary and involuntary denied boardings, according to stakeholders and our prior work. In our 2018 report, representatives from three airlines told us their airline had reduced or stopped overbooking flights. Our review of seven airlines’ customer service documents found that two airlines explicitly stated that they do not overbook their flights. Improving the ability to predict no-shows or rebook passengers. According to representatives from an industry association representing airlines, airlines have made investments to improve their software for predicting the number of passenger no-shows in an effort to reduce voluntary and involuntary denied boardings. These representatives also told us that airlines have hired additional personnel dedicated to more precisely forecasting no-show rates and proactively identifying rebooking options for passengers who are denied boarding. Improving communication with passengers. Some airlines have taken steps to notify passengers about potential denied boardings earlier in the travel process—in some cases before travelers have left for the airport—in an effort to encourage volunteers, according to stakeholders we interviewed. These stakeholders said that providing advance notice likely further reduces any burden on passengers associated with changing their travel plans. In 2018, five of the nine airlines we interviewed told us they had begun soliciting volunteers to give up their seat earlier in the process. More specifically, according to representatives from an industry association that represents airlines, some airlines call passengers prior to their arrival at the airport to gauge their willingness to give up their seat. Other airlines solicit volunteers at the check-in kiosk, which limits the need for airlines to identify passengers during the boarding process at the gate. None of the stakeholders we interviewed described any communication methods that were specific to passengers with disabilities. Nevertheless, as previously mentioned, four airlines (out of seven) explicitly state in their contracts of carriage that they generally do not deny boarding to passengers with disabilities. Increasing and diversifying compensation for passengers. Some airlines have offered additional incentives or increased compensation amounts to encourage passengers to voluntarily give up their seat. While airlines have historically provided passengers with travel vouchers to solicit volunteers, some have started offering alternative forms of compensation, such as gift cards for Amazon and other retailers, iPads, or travel vouchers with fewer restrictions or that also cover ancillary fees. Our review of DOT data indicates that relative to the number of passengers denied boarding involuntarily, more passengers have volunteered to give up their seat, lessening the need to deny passengers boarding involuntarily. For example, in 2012, for every one passenger denied boarding involuntarily, about nine volunteered to be denied boarding. In contrast, in 2018, for every one passenger denied boarding involuntarily, about 33 volunteered to be denied boarding. Providing passengers with the opportunity to propose acceptable voluntary denied boarding compensation. Some airlines solicit passengers with flexible travel plans to identify compensation amounts they would willingly accept in exchange for voluntarily giving up their seats and taking another flight. Once passengers submit their required compensation amount to the airline, the airline can then use that information to select passengers with the lowest amount of required compensation to accept a denied boarding. This process allows airlines to, among other things, potentially avoid involuntary denied boardings, and identify which passengers require the least compensation in exchange for their travel flexibility. Airlines conduct this process on their website, via their mobile app, or at the check-in kiosk. In some cases, passengers who would consider changing their plans in exchange for compensation provide the airline with a specific dollar amount that they would be willing to accept to give up their seat. In other cases, airlines require each passenger to select a predetermined amount of compensation that they would accept to give up their seat, as illustrated in figure 4. For example, based on the figure below, an airline that oversold its flight would select a passenger who volunteered to give up their seat in exchange for $250, assuming at least one passenger selected that amount. If no passengers selected that amount, the airline would identify a passenger with the next lowest amount—in this case, $350. Our review identified at least three airlines that use this type of process to solicit volunteers to give up their seats. Providing additional tools to employees. According to stakeholders we interviewed, airlines have given their employees more discretion regarding the offers they can make to encourage passengers to volunteer to be denied boarding on an oversold flight, or provide training on handling such incidents. While representatives from both consumer advocate organizations we interviewed generally supported some of the airlines’ actions to manage oversold flights, they also identified additional actions that airlines or DOT could take. Both consumer advocates we spoke to would like to see airlines increase transparency and passenger education related to denied boarding compensation. For example, these advocates believe that prior to agreeing to be voluntarily denied boarding, airlines should be required to inform passengers: (1) of the current compensation amounts for involuntary denied boardings, and (2) that compensation can be provided by cash or check (as opposed to a voucher). Having such information would allow passengers to make more informed decisions about the compensation they would willingly accept to be voluntarily denied boarding. Additionally, one consumer advocate said explicitly that they would also like airlines to inform passengers who are involuntary denied boarding that compensation amounts set by DOT are minimum amounts. Regarding potential additional actions, the FAA Reauthorization Act of 2018 required that DOT issue a rulemaking clarifying, among other things, that the compensation amounts set by DOT for involuntary denied boardings are the minimum compensation amounts that passengers can receive. In October 2019, DOT officials indicated that DOT intends to issue its final rule in July 2020. Moreover, in November 2018, we made three recommendations to DOT to improve its passenger education efforts by, among other things, capturing feedback from passengers directly, and identifying available short- and long-term budgetary resources for these efforts. DOT agreed with our recommendations and is in the process of implementing them. More broadly, both consumer advocates we interviewed called for an end to overbookings. This could be achieved either voluntarily by airlines or in regulation by DOT. These advocates said that overbooking is an outdated practice that protected airlines from high no-show rates during a time when passengers could make multiple reservations and did not incur change fees. Given that this is no longer the case, it is not necessary for airlines to overbook their flights, according to these consumer advocates. They also pointed out that airlines have significant flexibility in their business operations, including, denying boarding when a flight is overbooked, or changing flight schedules. In contrast, passengers have little, if any, recourse if they need to change their travel plans. Most tickets have restrictions that prevent passengers from making changes to their flights without incurring high change fees. Consumer advocates believe that eliminating overbooking would have limited effects on airlines, given the restrictions on passengers’ tickets. According to three airline revenue management specialists, if airlines were prohibited from overbooking flights, they would likely end up operating aircraft with more empty seats, compared to current trends. Moreover, they also noted that if flights were less full, there could be certain negative implications for airlines and passengers. For example, when fewer seats on a flight are filled with paying passengers, airlines’ average costs per passenger are higher because many aspects of airlines’ operational costs—such as salaries for crew, mechanic services, and airport landing fees—are generally the same, regardless of the number of passengers onboard. These same revenue management specialists also noted that a greater number of empty seats will generally decrease airline’s revenue. One of them estimated that the reduced revenues could amount to tens of millions of dollars. Some airlines would also likely change their revenue management practices, according to airline revenue management specialists. Those changes would largely focus on how airlines price their tickets. While two airlines have made a business decision not to overbook and have accepted the financial trade-off, revenue management specialists said that eliminating overbooking would be difficult for other airlines. In particular, all three revenue management specialists agreed that if airlines were prohibited from overbooking, some airlines may offer fewer discounted fare tickets. Two revenue management specialists also said that it is likely that airlines would increase the average fare across all tickets slightly to account for the increased costs and potential lost revenue. Finally, one revenue management specialist also said that airlines might add additional restrictions on tickets, such as by increasing penalties associated with a passenger not showing up for their flight or cancelling their ticket at the last minute. Moreover, even if airlines stopped overbooking, some passengers would still be denied boarding because factors other than overbooking—including some that are beyond the airline’s control—can lead to denied boardings. We provided a draft of this report to DOT for review and comment. DOT provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have questions concerning this report, please contact me at (202) 512-2834 or VonahA@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. Andrew Von Ah, (202) 512-2834 or VonahA@gao.gov. In addition to the individual named above, other key contributors to this report were: Ed Laughlin (Assistant Director); Amy Suntoke (Analyst-in- Charge); Amy Abramowitz; Sarah Arnett; Melissa Bodeau; Colson Campbell; Lori Fields; Dave Hooper; Mary-Catherine Overcash; Malika Rice; Pam Snedden; Melissa Swearingen; and Elizabeth Wood.", "summary": "Some airlines overbook their scheduled flights (intentionally sell more seats than are available) to compensate for passenger no-shows. It is not illegal for airlines to overbook their flights. However, it can result in an “oversale” where airlines cannot accommodate all passengers on a particular flight. In response, airlines may have to deny boarding to some passengers. DOT is responsible for ensuring airlines adhere to their denied boarding practices as part of its consumer protection enforcement responsibilities. The FAA Reauthorization Act of 2018 included a provision that GAO examine airlines' oversales practices. This report focuses on denied boardings—the result of an oversale—and describes (1) trends in denied boardings and (2) airlines' actions related to denied boardings and mitigating the effects on passengers. GAO analyzed data on denied boardings and related passenger complaints submitted to DOT from 2012 through 2018, and reviewed seven airlines' publicly available documents describing their overbooking and denied boarding policies. Airlines were selected to generally include the largest airlines that GAO previously reported had varying practices on overbookings and denied boardings. GAO also reviewed relevant statutes and DOT regulations, summarized GAO work published in 2018 describing airlines actions to reduce denied boardings, and interviewed DOT officials, one airline industry association, two consumer advocate organizations, and three airline revenue management specialists. The selection of stakeholders was non-generalizable and based on inclusion in prior GAO work and their relevance regarding denied boarding practices. The number of passengers denied boarding (not allowed to board flights they have tickets on) generally decreased in recent years, according to Department of Transportation (DOT) data. Combined, on an annual basis, voluntary and involuntary denied boardings account for less than 1 percent of actual passenger boardings. Voluntary denied boardings. As shown below, most denied boardings are passengers who “voluntarily” gave up their seat for compensation of the airline's choosing, such as airline vouchers. Passengers can negotiate compensation amounts. For every 100,000 actual boardings in 2018, about 43 passengers were voluntarily denied boarding. Involuntary denied boardings. All other denied boardings occur “involuntarily.” These passengers may be eligible for compensation in an amount set by DOT. For every 100,000 actual boardings in 2018, about one passenger was involuntarily denied boarding. While few denied boardings are involuntary, these passengers may encounter significant costs and travel disruptions. GAO's review of passenger complaints submitted to DOT showed instances where passengers involuntarily denied boarding reported missing significant events—e.g., a wedding or a cruise—and incurring additional costs. Airlines can face challenges rebooking passengers, such as those flying to smaller communities, exacerbating these disruptions. Passengers Denied Boarding Voluntarily and Involuntarily per 100,000 Actual Boardings, 2012-2018 Airlines have taken a range of actions, aimed at reducing involuntary denied boardings. Actions include reducing overbookings; requesting volunteers earlier (e.g., at check-in); and increasing compensation for volunteers. While consumer advocates GAO interviewed generally supported these actions, they advocated for an end to overbooking. Three airline revenue management specialists said if airlines were prohibited from overbooking, some airlines may offer fewer discounted fare tickets. Two of these specialists also said airlines might also slightly increase average fares across all tickets.", "document_type": "gao"}
{"report": "DOD requires each Military Service to establish its own EFMP for active duty servicemembers. According to DOD guidance, EFMPs are to have three components—identification and enrollment, assignment coordination, and family support. Identification and enrollment: DOD requires servicemembers to enroll in their Military Service’s EFMP once eligible family members are identified by medical and educational personnel at each installation. Assignment coordination: Before finalizing a servicemember’s assignment to a new location, DOD requires each Military Service to consider any family member’s special needs, including the availability of required medical and special educational services at a new location. Family support: DOD requires each Military Service’s EFMP to help families with special needs identify and gain access to programs and services at their current, as well as proposed locations. As required by the NDAA for Fiscal Year 2010, DOD established the Office of Community Support for Military Families with Special Needs (Office of Special Needs or OSN) to develop, implement, and oversee a policy to support these families. Among other things, this policy must (1) address assignment coordination and family support services for families with special needs; (2) incorporate requirements for resources and staffing to ensure appropriate numbers of case managers are available to develop and maintain services plans that support these families; and (3) include requirements regarding the development and continuous updating of a services plan for each military family with special needs. OSN is also responsible for monitoring the Military Services’ EFMPs and collaborating with the Military Services to standardize EFMP components as appropriate. For example, as part of its guidance for monitoring the Military Services’ EFMPs, DOD requires each Military Service to certify or accredit its family support services provided through the EFMP. In addition, DOD states that each Military Service must balance the need for overarching consistency across EFMPs with the need for each Military Service to provide family support that is consistent with their specific mission. Table 1 provides an overview of the procedures each Military Service must establish for the assignment coordination and family support components of the EFMP that we identified in our May 2018 report. In May 2018, DOD reported that each Military Service provides family support services in accordance with DOD guidance, as well as Military Service-specific guidance. However, we found that, the type, amount, and frequency of assistance families with special needs receive varied by Military Service, which could lead to gaps in assistance (see table 2). For example, in our May 2018 report, we found that the Marine Corps is the only Military Service that specifies a minimum frequency (quarterly) with which families with special needs should be contacted by their family support providers. The other Military Services either do not have requirements for regular contact with these families (Air Force and Army) or require contact only for selected families (Navy). In addition, we reported that unlike the Marine Corps, the Air Force, Army, and Navy choose not to employ special education attorneys. For example, Marine Corps attorneys may represent families with special needs who fail to receive special education services from local school districts, as specified in their children’s individualized education programs (IEP). Officials from the Air Force, Army, and Navy told us that they find other ways to help families with special needs resolve special education issues. For example, Army officials said EFMP managers could refer families with special needs to other organizations that provide legal support. As we reported in May 2018, services plans are an important part of providing family support during the relocation process because they describe the necessary services and support for a family with special needs and provide a record for the gaining installation. However, we found that every Military Service had created relatively few services plans compared to the number of servicemembers or the number of family members enrolled in the EFMP (see table 3). The Military Services and OSN provided a number of reasons as to why they do not develop and maintain a services plan for each family with special needs. For example, Air Force officials said they first consider whether a services plan will help each family receive the required services. In addition, Army and Marine Corps officials said they may not develop a services plan if a family does not request it. According to a Navy official, some families also lack the required services plans because installations may not have the staff needed to develop them. Finally, OSN officials said the Military Services may not have developed many services plans during fiscal year 2016 because DOD had not yet approved a standardized form that all of the Military Services could use, and because some families’ circumstances did not require a services plan. In our May 2018 report, we recommended that DOD assess the extent to which each Military Service is developing a services plan for each family with special needs. DOD concurred with our recommendation, but as of January 2020, we determined that DOD has not fully implemented the recommendation because it has not yet assessed the extent to which each Military Service is developing services plans for each family with special needs. In its annual report to the congressional defense committees in April 2019, DOD stated that it was exploring legislative changes to the law that would require a services plan to be developed and updated only for those families who request services. A senior official from DOD stated that although this proposal received Office of Management and Budget approval, it was not included in the NDAA for fiscal year 2020. Also, in April 2019, in response to our recommendation, DOD reported to us that the Military Services had begun using a standardized form to develop services plans. In January 2020, a senior DOD official said its standardized form provides an option for a family to decline a services plan, and that the Department began collecting data related to services plans in the last quarter of 2019. To meet requirements of the NDAA for Fiscal Year 2010, in April 2017, DOD stated that it issued to the Military Services guidance that directed them to “rogram, budget, and allocate sufficient funds and other resources, including staffing,” to meet DOD’s policy objectives for the EFMP. We reported in May 2018 that DOD relies on each Military Service to determine what level of funds and resources are sufficient and what constitutes an appropriate number of family support personnel. To determine the appropriate number of family support providers and staffing levels, the Military Service officials with whom we spoke said they consider a number of factors, including the number of families with special needs enrolled in the EFMP at any given installation. See Table 4 for a summary of EFMP family support providers and other key personnel at CONUS installations. In May 2018, based on our analysis of EFMP family support providers and other key personnel at CONUS installations, we found that DOD had not developed a standard for determining the sufficiency of funding and resources each Military Service allocates for family support. As a result, the Military Services may not know the extent to which their funding and resources for family support comply with DOD’s policy. Federal internal control standards require that agencies establish control activities, such as developing clear policies, in order to accomplish agency objectives, such as those of the Military Services’ EFMPs. Because DOD had not identified and addressed potential gaps in family support across the Military Services’ EFMPs, such as those we identified in types of assistance, services plans, and resources, we concluded that some families with special needs may not get the assistance they require, particularly when they relocate. We recommended in our May 2018 report that DOD assess the extent to which each Military Service is providing sufficient resources to staff an appropriate number of family support providers. DOD concurred with our recommendation. In April 2019, the most recent update DOD provided on this recommendation, DOD officials said they were planning to pilot a staffing tool to help the Military Services determine the number of family support providers needed at each installation; the pilot is expected to last 2 years before it can be implemented across the Military Services. We reported in May 2018 that OSN had several efforts underway to improve its oversight of the EFMP. For example, to help provide a more consistent EFMP screening process across the Military Services and improve the collection of comparable assignment coordination data, OSN had planned for each Military Service to use standard screening forms for family members with special medical or educational needs prior to making new assignments. In January 2020, DOD told us that the forms were approved, but related guidance had not yet been developed for implementation across all of the Military Services. In addition, OSN planned to centralize the management of EFMP data across the Military Services. In April 2019, DOD reported that 82 percent of the EFMP related data terms were collectable across the Military Services which can improve OSN’s monitoring and reporting capabilities of the EFMP. Despite OSN’s initial efforts, we found that DOD lacked common performance measures for assignment coordination and family support, and therefore is unable to fully assess EFMP performance across the Military Services. In our May 2018 report, we recommended that DOD direct OSN to develop common performance metrics for assignment coordination and family support, in accordance with leading practices for performance measurement. DOD concurred with our recommendation. In April 2019, the most recent update DOD provided on this recommendation, DOD officials told us that each Military Service submits data on assignment coordination and family support to the EFMP data repository on a quarterly basis, and that OSN was currently developing additional performance metrics for assignment coordination and family support. Until these metrics are fully developed and implemented, DOD will remain unable to fully assess the effectiveness of its efforts related to assignment coordination and family support at each of its installations. We also found in May 2018 that OSN did not have a process to systematically evaluate the results of the Military Services’ monitoring activities. Instead, DOD requires each Military Service to monitor its own assignment coordination and family support provided through the EFMP and requires each Military Service to assess performance at least once every 4 years using standards developed by a national accrediting body. In addition, DOD requires personnel from each of the Military Service’s headquarters to periodically visit installations as part of their monitoring activities. We also reported that the Military Services’ family support programs were not accredited by a national accrediting body because, according to Military Service officials, they were unable to obtain funding for engaging in that process. Instead, each Military Service has a self-certification process based on standards that meet those of a national accrediting body, Military Service-specific standards, and best practices. We also reported in May 2018 that OSN officials did not systematically review the results of monitoring activities, such as the certification process, because they rely on each Military Service to self- monitor. In addition, officials said efforts to standardize certification of EFMPs have been unsuccessful because the Military Services cannot agree on a set of standards that can be used across installations. We recommended in our May 2018 report that DOD implement a systematic process for evaluating the results of the Military Services’ monitoring activities. DOD concurred with our recommendation but has not yet fully implemented it. DOD last commented on this recommendation in April 2019 and said the family support component is monitored and evaluated through each Military Service’s certification process, which includes specific standards for the EFMP. In addition, OSN participated in a monitoring site visit to Marine Corps Base Quantico in December 2018 and plans to participate in additional site visits that are coordinated by each Military Service’s certification team. We will consider this recommendation implemented only when DOD provides evidence that it has implemented a systematic process to evaluate the results of each Military Service’s monitoring activities. In conclusion, DOD relies on each Military Service to implement its policy on support for families with special needs. In doing so, they also rely on each Military Service to determine the extent to which its assistance to families with special needs complies with this policy. As it plans for the future, DOD will need to balance the flexibility it provides each Military Service to implement its policy with the need to assess the adequacy of the Military Services’ EFMPs in serving families with special needs, including any gaps in services these families receive. Chairwoman Speier, Ranking Member Kelly, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. For further information regarding this testimony, please contact Jacqueline M. Nowicki, Director of Education, Workforce, and Income Security Issues at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this statement include Bill MacBlane (Assistant Director), Brian Egger (Analyst-in-Charge), Holly Dye, Robin Marion, James Rebbe, Shelia Thorpe, Walter Vance, Kelsey Kreider, and Mimi Nguyen. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Military families with special needs face unique challenges because of their frequent moves. To assist these families, each Military Service implements its own program, known as EFMP. The National Defense Authorization Act (NDAA) for Fiscal Year 2017 included a provision for GAO to review the Military Services' EFMPs, including DOD's role in providing guidance for these programs. This statement focuses on the extent to which (1) each Military Service provides family support in the continental United States and (2) the Military Services monitor and DOD evaluates assignment coordination and family support. This statement is based on a May 2018 GAO report and updates its three recommendations as of January 2020. For the report, GAO analyzed EFMP guidance and documents; reviewed federal laws; analyzed fiscal year 2016 EFMP data; visited military installations, selected for their large numbers of military-connected students; and interviewed officials responsible for implementing, monitoring, and evaluating the EFMPs. In May 2018, GAO found that variation in support provided to military family members with special medical and educational needs through the Department of Defense's (DOD) Exceptional Family Member Program (EFMP) could lead to potential gaps in assistance. GAO recommended that DOD assess the extent to which each Military Service is developing services plans for each family with special needs and is providing sufficient resources to staff an appropriate number of family support providers, as required. DOD concurred. Services plans are important because they describe the necessary services and support for a family with special needs enrolled in the EFMP as well as during the relocation process, such as when a servicemember is assigned to a new location. In April 2019, DOD reported that the Military Services had adopted a standardized form to use when developing services plans; however, DOD has not yet assessed the extent to which each Military Service is developing these plans. In January 2020, a senior DOD official said that the Department began collecting data related to services plans in the last quarter of 2019. In April 2019 (the most recent update), DOD officials said they were planning to pilot a staffing tool to help the Military Services determine the number of family support providers needed at each installation. However, the pilot is expected to last 2 years before it can be implemented across the Military Services. GAO also found that DOD lacked common performance measures for the EFMP and was unable to compare the program's performance across the Military Services. GAO recommended that DOD develop common performance metrics for the program. DOD concurred, and in April 2019 said that it was still in the process of developing performance metrics for assignment coordination and family support. In January 2020, DOD noted that it had not yet developed guidance regarding use of forms that would help improve its ability to collect common performance measures across the Military Services. Further, GAO found that DOD does not have a process to systemically evaluate the results of each Military Service's monitoring activities. GAO also reported that DOD did not systematically review the results of monitoring activities because it relies on each Military Service to self-monitor. DOD officials said efforts to standardize certification of EFMPs have been unsuccessful because the Military Services cannot agree on a set of standards that can be used across installations. GAO recommended that DOD implement a systematic process for evaluating the results of the Military Services' monitoring activities. DOD concurred with the recommendation, but has not yet fully implemented it. In the May 2018 report, GAO made three recommendations to DOD. DOD concurred, but has made limited progress toward addressing them.", "document_type": "gao"}
{"report": "The Veterans Access, Choice, and Accountability Act of 2014 provided up to $10 billion in funding for veterans to obtain health care services from community providers through the Choice Program when veterans faced long wait times, lengthy travel distances, or other challenges accessing care at VA medical facilities. The temporary authority and funding for the Choice Program was separate from other previously existing programs through which VA has the option to purchase care from community providers. Legislation enacted in April, August, and December of 2017 and June 2018 extended the Choice Program and provided an additional $9.4 billion for the Veterans Choice Fund. Authority for the Choice Program will sunset on June 6, 2019. In October 2014, VA modified its existing contracts with two contractors— referred to as third party administrators (TPA)—that were administering another VA community care program to add certain administrative responsibilities associated with the Choice Program. For the Choice Program, each of the two TPAs—Health Net and TriWest—was responsible for managing networks of community providers who deliver care in a specific multi-state region. Specifically, the TPAs were responsible for establishing networks of community providers, scheduling appointments with community providers for eligible veterans, and paying community providers for their services. Health Net’s contract for administering the Choice Program ended on September 30, 2018, with TriWest continuing to administer the Choice Program in its region and the region previously administered by HealthNet until the program ends. Through policies and standard operating procedures for VA medical facilities and contracts with the TPAs, VA established processes for referring and scheduling appointments through the Choice Program: one process for time-eligible veterans and another for distance-eligible veterans. Table 1 provides an overview of the appointment scheduling process that applies when a veteran is referred to the Choice Program because the veteran is time-eligible—that is, the next available medical appointment with a VA clinician is more than 30 days from the veteran’s preferred date or, in the absence of such a date, the date the veteran’s physician determines he or she should be seen. When veterans reside more than 40 miles from a VA medical facility or meet other travel-related criteria, VA uses the appointment scheduling process it developed for distance-eligible veterans. The process for distance-eligible veterans differs from that for time-eligible veterans in that VA medical facilities do not prepare a referral and send it to the TPA. Instead, distance-eligible veterans contact the TPA directly to request Choice Program care. VA’s Choice Program TPA processes claims it receives from community providers for the care they deliver to veterans and pays providers for approved claims. Figure 1 provides an overview of the steps the TPA follows for processing claims and paying community providers. To be reimbursed for its payments to providers, the TPA in turn submits electronic invoices—or requests for payment—to VA. The TPA generates an invoice for every claim it receives from community providers and pays. VA reviews the TPA’s invoices and either approves or rejects them. Invoices may be rejected, for example, if care provided was not authorized. Approved invoices are paid, whereas rejected invoices are returned to the TPA. Under the Prompt Payment Act, VA is required to pay its TPAs within 30 days of receipt of a clean Choice Program invoice. The VA MISSION Act of 2018, among other things, requires VA to establish a permanent community care program no later than 1 year after passage of the Act (June 6, 2019) and authorizes VA to utilize a TPA for claims processing. VA refers to the consolidated program as the VCCP. In December 2016, prior to enactment of the VA MISSION Act of 2018, VA issued a request for proposals for contractors to help administer the VCCP. The VCCP will be similar to the current Choice Program in certain respects. For example, under the VCCP, TPAs will be responsible for establishing regional networks of community providers and processing and paying those providers’ claims. However, unlike the Choice Program, under the VCCP, VA is planning to have medical facilities—not the TPAs—generally be responsible for scheduling veterans’ appointments with community providers. VA awarded contracts for administering the VCCP in three of six regions on December 28, 2018. As of April 3, 2019, VA had not yet awarded contracts for the remaining three regions. Generally, all veterans enrolled in the VA health care system would be able to qualify for care through the VCCP when (1) VA does not offer the care or service required by the veteran; (2) the veteran resides in a state without a full-service VA medical facility; (3) the veteran would have been eligible under the 40-mile criterion of the Choice Program before June 6, 2018; (4) VA cannot provide the veteran with care and services that comply with its designated access standards; or (5) the veteran and the veteran’s referring clinician agree that it is in the best interest of the veteran to receive care in the community. In January 2019, VA proposed new access standards for the VCCP based on average drive times and wait times: For primary care, mental health, and non-institutional extended care services, VA is proposing a 30-minute average drive time standard. For specialty care, VA is proposing a 60-minute average drive time standard. VA is proposing appointment wait-time standards of 20 days for primary care, mental health care, and non-institutional extended care services, and 28 days for specialty care from the date of request with certain exceptions. Eligible veterans who cannot access care within those standards would be able to choose between eligible community providers and care at a VA medical facility. VA expects to issue the final regulation establishing access standards for the VCCP by June 2019. In June 2018, we reported that numerous factors adversely affected veterans’ timely access to care through the Choice Program and could affect implementation of the VCCP. These factors included the following: (1) administrative burden caused by complexities of VA’s referral and appointment scheduling processes; (2) poor communication between VA and its medical facilities; and (3) inadequacies in the networks of community providers established by the TPAs, including an insufficient number, mix, or geographic distribution of community providers. VA has taken steps to help address these factors; however, not all access factors have been fully addressed. For example, to help address administrative burden and improve the process of coordinating veterans’ Choice Program care, VA established a secure e-mail system and a mechanism for TPAs and community providers to remotely access veterans’ VA electronic health records. However, these mechanisms only facilitate a one-way transfer of necessary information. They do not provide a means by which VA medical facilities or veterans can view the TPAs’ step-by-step progress in scheduling appointments or electronically receive medical documentation associated with Choice Program appointments. We made five recommendations to VA to address the factors that adversely affected veterans’ access to Choice Program care. VA agreed or agreed in principle with all five recommendations. Our recommendations and the steps, if any, VA has taken in response to these recommendations are described in table 2. In June 2018, we reported that VA cannot systematically monitor the timeliness of veterans’ access to Choice Program care because it lacks complete, reliable data to do so. VA will need to address these data limitations in order to effectively monitor the care delivered to veterans through the VCCP. The data limitations we identified included the following: A lack of data on the timeliness of accepting referrals and opting veterans in to the program. We found that the data VA uses to monitor the timeliness of Choice Program appointments do not capture the time it takes VA medical facilities to prepare veterans’ referrals and send them to the TPAs, nor do they capture the time spent by the TPAs in accepting VA medical facilities’ referrals and opting veterans in to the Choice Program. VA had implemented an interim solution to monitor overall wait times that relies on VA medical facility staff consistently and accurately entering unique identification numbers on VA clinicians’ requests for care and on Choice Program referrals, a process that is prone to error. Inaccuracy of clinically indicated dates. We found that clinically indicated dates (used to measure the timeliness of care) are sometimes changed by VA medical facility staff before they send Choice Program referrals to the TPAs, which could mask veterans’ true wait times. We found that VA medical facility staff entered later clinically indicated dates on referrals for about 23 percent of the 196 authorizations we reviewed. It is unclear if VA medical facility staff mistakenly entered incorrect dates manually, or if they inappropriately entered later dates when the VA medical facility was delayed in contacting the veteran, compiling relevant clinical information, and sending the referral to the TPA. Unreliable data on the timeliness of urgent care. We found that VA medical facilities and TPAs do not always categorize Choice Program referrals and authorizations in accordance with the contractual definition for urgent care. According to the contracts, a referral is to be marked as “urgent,” and an appointment is to take place within 2 business days of the TPA accepting it, when a VA clinician has determined that the needed care is (1) essential to evaluate and stabilize the veteran’s condition, and (2) if delayed would likely result in unacceptable morbidity or pain. We reviewed a sample of 53 urgent care authorizations and determined that about 28 percent of the authorizations were originally marked as routine care authorizations but were changed to urgent by VA medical facility or TPA staff, in an effort to administratively expedite appointment scheduling. We made five recommendations to VA on improving the completeness and accuracy of data on veterans’ wait times for care. VA agreed with four of the five recommendations. Our recommendations and the steps VA has taken in response to these recommendations are described in table 3. In September 2018, we reported that three key factors affected timeliness of payments to community providers under the Choice Program and that if unaddressed could affect provider payment timeliness for the VCCP. These factors included the following: (1) VA’s untimely payments to TPAs, which in turn extended the length of time TPAs took to pay community providers’ claims; (2) Choice Program reimbursement requirements, which led to claim denials; and (3) inadequate provider education on filing claims. We reported that VA has taken some actions to address these factors. For example, VA updated its payment system and related processes to pay TPAs more quickly. According to VA data, as of July 2018, VA was paying at least 90 percent of the TPAs’ invoices within 7 days, a significant increase from the 50 percent timely payments VA made to TPAs between November 2014 and September 2016. In addition, VA and the TPAs had taken steps to amend certain reimbursement requirements and improve provider education to help providers resolve claims processing issues. However, we found that VA has not fully addressed two of these factors. First, with respect to reimbursement requirements, VA does not have complete data allowing it to effectively monitor adherence with its policy for VA medical facilities to perform timely reviews and approvals of secondary authorization requests. Community providers request secondary authorization requests when veterans need health care services that exceed the period or scope of the original authorization. Incomplete data impacted VA’s ability to meet the requirement. When VA medical facilities delay these reviews and approvals, community providers may have to delay care or deliver care that is not authorized, which in turn increases the likelihood that the providers’ claims will be denied and the providers will not be paid. Second, with respect to provider education on filing claims, VA requires the TPAs to establish a customer call center to respond to calls from veterans and non-VA providers. However, VA does not enforce the contractual requirement for responding to calls from community providers and allows the TPAs to prioritize calls from veterans over calls from community providers. Consequently, VA is not collecting data, monitoring, or enforcing compliance with its contractual requirements for the TPAs to provide timely customer service to providers. As a result, VA does not have information on the extent to which community providers face challenges when contacting the TPAs about claims payment issues, which could contribute to the amount of time it takes to receive reimbursement for services. To address remaining factors that affect provider payment timeliness, we made two recommendations to VA. VA agreed with both recommendations. Our recommendations and the steps VA has taken in response to these recommendations are described in table 4. In summary, consolidating its existing community care programs into the VCCP and launching this new program in June 2019 is a large and complex undertaking, which comes with many risks and challenges for VA. Heeding the lessons learned from its implementation and management of the Choice Program will better position VA to ensure veterans receive timely access to care under the VCCP and avoid past challenges such as delays in scheduling appointments and untimely payments to community providers. Continued oversight of VA’s implementation of the VCCP will be critical given the scale of change and the associated risks. We stand ready to assist this Committee with this continued oversight. Chairman Isakson, Ranking Member Tester, and Members of the Committee, this concludes my prepared statement. I would be pleased to respond to any questions you may have. If you or your staff members have any questions concerning this testimony, please contact me at (202) 512-7114 or silass@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals who made key contributions to this testimony include Marcia Mann (Assistant Director), Michael Zose (Analyst-in-Charge), Jacquelyn Hamilton, Christina Ritchie, Kate Tussey, and Emilie Weisser. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In June 2018, Congress passed the VA MISSION Act of 2018, which requires VA to establish a permanent community care program. VA plans to consolidate the Choice Program and its other VA community care programs into one community care program—the VCCP. This legislation helps address some of the challenges faced by the Choice Program and VA's other community care programs. VA's implementation of the VCCP can benefit from the lessons learned under the Choice Program. Ignoring these lessons learned increases VA's risk for not being able to ensure that all veterans receive timely access to care in the community and that community providers are reimbursed in a timely manner. This testimony focuses on lessons learned from the Choice Program, including recommendations GAO has made to VA to help ensure (1) veterans' timely access to care under the VCCP (2) effective monitoring of veterans' access to care under the VCCP, and (3) timely payments to community providers under the VCCP. This testimony is based on GAO reports on the Choice Program that were issued in June 2018 and September 2018. The Department of Veterans Affairs' (VA) Veterans Choice Program (Choice Program) allows eligible veterans to obtain health care services from providers not directly employed by VA (community providers). The program is largely managed by third party administrators (TPA), who are responsible for establishing provider networks, scheduling veterans' appointments, and paying providers. GAO has identified the following challenges to the Choice Program that VA needs to address as it implements its new Veterans Community Care Program (VCCP). Factors that adversely affected veterans' timely access to care. GAO found that numerous factors adversely affected veterans' timely access to care through the Choice Program. These factors included (1) administrative burden caused by complexities of referral and appointment scheduling processes; (2) poor communication between VA and its medical facilities; and (3) inadequacies in the networks of community providers established by the TPAs, including an insufficient number, mix, or geographic distribution of community providers. VA has taken steps intended to help address these factors, however, some have not been fully addressed. In June 2018, GAO made five recommendations to VA, including that VA establish a system that will facilitate care coordination and exchanges of information among VA medical facilities, VA clinicians, TPAs, community providers, and veterans. VA agreed or agreed in principle with all five recommendations, but has not yet implemented them. Unavailable and unreliable data. GAO found that VA cannot systematically monitor the timeliness of veterans' access to Choice Program care because it lacks complete, reliable data to do so. The data limitations GAO identified included a lack of data on the timeliness of accepting referrals and opting veterans in to the program, inaccurate data on clinically indicated dates (which are used to measure the timeliness of care), and unreliable data on the timeliness of urgent care. In June 2018, GAO made five recommendations to VA, including that VA implement mechanisms to allow VA to systematically monitor the amount of time taken to prepare referrals, schedule appointments, and complete appointments. VA agreed with four of the five recommendations, but has not yet implemented them. Untimely payments to community providers. GAO identified three key factors that affected timeliness of payments to community providers under the Choice Program. These factors included (1) VA's untimely payments to TPAs, which in turn extended the length of time TPAs took to pay providers' claims; (2) Choice Program reimbursement requirements, which led to claim denials; and (3) inadequate provider education on filing claims. GAO found that VA has taken actions to address the factors, such as amending certain reimbursement requirements. However, two of these factors have not been fully addressed. In September 2018, GAO made two recommendations to VA, including that VA collect data on and monitor compliance with its requirements pertaining to customer service for community providers. VA agreed with the recommendations, but has not yet implemented them. GAO has made 12 recommendations to VA to improve its management and oversight of the Choice Program and the VCCP. VA generally agreed with all but one of GAO's recommendations. GAO continues to believe that all of the recommendations are warranted. As of April 2019, these recommendations have not been implemented.", "document_type": "gao"}
{"report": "In general, date labels on packaged foods are not required by federal regulations, except in the case of infant formula. However, manufacturers may choose to provide date labels to help consumers and retailers decide when the food is of best quality. These date labels may carry different introductory phrases, such as ‘“best by,” “sell by,” “use by,” or “best if used by,” prior to the date, according to the preference of the manufacturer. In most cases, these labels indicate quality or freshness (i.e., the last date by which the manufacturer believes the food will be fresh or taste best), according to USDA. However, according to representatives from the Food Marketing Institute, because date labels are not federally regulated, manufacturers use a number of different date labels across industry, and this variation in date labels may result in consumer confusion about the meaning of the labels. Consumer confusion about date labels on packaged foods contributes to food waste, according to various studies. Due to this confusion about what date labels mean, consumers may throw out food that is safe to eat, even if the food does not have visible signs of spoilage, according to these studies. For example, in a 2017 study, participants were asked to state what they thought a particular date label on a certain packaged food meant. The study found that participants did not have a consistent understanding of date labels—some thought the labels indicated safety, and some thought they indicated quality, and others were unsure what the labels meant. Furthermore, three studies estimated that from 34 percent to 70 percent of consumers think that their risk of foodborne illness increases if they consume a packaged food product past the date label. In addition to confusion about the meaning of date labels, one study found that consumers are confused about who is responsible for date labels. According to the study, more than one-third of consumers surveyed believed the federal government regulates date labels, and 26 percent were unsure. USDA and FDA share oversight of nearly all the nation’s food supply but do not regulate most date labels and are not required to do so by federal law. USDA is responsible for the safety and proper labeling of meat, poultry, and egg products, and FDA is responsible for the safety and proper labeling of virtually all other foods. Within USDA, FSIS is the public-health agency responsible for ensuring that meat, poultry, and processed egg products are safe, wholesome, and accurately labeled. According to FSIS, its goals include lowering the incidence of pathogens that cause foodborne illnesses and limiting the occurrence of outbreaks, and ensuring that regulated products are properly packaged and labeled so consumers have access to important information about the product. Also within USDA, AMS provides voluntary quality-grading programs for producers of products such as milk, eggs, and meat. These quality- grading programs are paid for by the producers of these commodities and can require, among other things, that producers participating in these programs use date labels. In addition, USDA has research components that could address issues relating to date labels. For example, USDA’s National Institute of Food and Agriculture supports research through grants to individuals, institutions, and organizations, and the Economic Research Service conducts economic research to inform and enhance decision-making. FDA has statutory authority to regulate the safety of foods and nutrition labels on packaged foods not regulated by FSIS. However, the agency is not required by statute to regulate the use of date labels. FDA also exercises its general authority to assist state and local governments with food safety efforts through its State Cooperative Programs specifically for Grade A milk, molluscan shellfish, and retail and food-service establishments. As part of these programs, FDA provides technical support, guidance, and training to help its regulatory partners with reducing foodborne illnesses associated with these commodities. Also, in coordination with its regulatory partners and industry, FDA develops guidance, including guidance on date labels for certain products and in certain circumstances. Such guidance represents FDA’s best advice for a uniform system of provisions that address the safety and protection of food offered at retail and in food service. As we reported in May 2016, the federal food safety oversight system is supplemented by states, localities, tribes, and territories, which may have their own laws and agencies to address the safety and quality of food. Generally, state, local and tribal governments may choose whether to regulate date labels on packaged foods. For example, the majority of states and the District of Columbia have some date labeling requirements for, most commonly, shellfish, dairy, and eggs, as figure 1 shows. Additionally, some states and the District of Columbia prohibit retailers from selling some packaged foods to consumers if the date on the label has passed. Furthermore, some municipalities choose to regulate date labels in addition to, or in the absence of, state regulations. For example, while Maryland prohibits the sale of grade “A” milk or milk products past the “Sell by” date marked on its cap or container, the city of Baltimore generally prohibits the sale of any perishable food past its expiration date. Tribal governments may also have regulations that address labels on packaged food. For example, the Navajo Nation Code has requirements related to labeling of some packaged foods, such as shellfish. Where no state or local regulations are in place regarding date labeling, manufacturers may decide which of their packaged food products display a date label and what wording to use on the date label. Estimates on the number of different date labels currently in use across industry vary. Figure 2 below provides examples of introductory phrases for date labels currently used by packaged food manufacturers. In response to consumer confusion about date labels and resulting food waste, two industry associations, the Food Marketing Institute and the Grocery Manufacturers Association, in 2017 announced a voluntary industry initiative to encourage manufacturers and retailers to standardize date labels on packaged foods. This initiative calls for manufacturers to use either of two introductory phrases for date labels on packaged foods: (1) a “best if used by” label as an indication of product quality and (2) “use by” for certain perishable products that may be more susceptible to degradation of quality or potential food safety concerns. In December 2018, the Food Marketing Institute and the Grocery Manufacturers Association reported that in a consumer survey on these date labels, 88 percent of those surveyed said these two date label phrases were clear to them and 85 percent said they were helpful. Some advocacy groups work to reduce consumer confusion over date labels as part of their overall food waste reduction efforts. For example, Rethink Food Waste Through Economics and Data (ReFED), an advocacy group committed to reducing food waste in the United States, in 2016 issued a report that outlined key steps to reduce food waste. The report listed standardizing date labeling as one of the top three solutions to reducing food waste with the greatest economic value and net environmental benefit. According to representatives from ReFED, the organization works alongside industry to promote the Food Marketing Institute and the Grocery Manufacturers Association joint voluntary initiative by providing manufacturers with a tool they can use to determine which wording to use on date labels for different products. In addition, according to these representatives, ReFED is developing methods to disseminate information to consumers about the meaning of date labels. Representatives from the Natural Resources Defense Council (NRDC), an international nonprofit environmental advocacy organization, said that NRDC is coordinating with the National Ad Council, a nonprofit organization that provides public-service communications, to develop a multiyear outreach and education campaign aimed at reducing household food waste. According to NRDC representatives, this joint effort may include information about date labels. Furthermore, in 2017, NRDC and the Harvard Food Law and Policy Clinic issued a report that found that the lack of standard date labels leads to a mistaken belief that past-date food is unsafe to consume, which causes unnecessary waste. The report called for Congress to pass legislation or for FDA and USDA to work together to create uniform regulations that standardize date labels throughout the nation. International entities have also taken steps to address date labeling practices and make date labels clearer to consumers. For example, according to its website, the Consumer Goods Forum—an association of 400 manufacturers and retailers across 100 countries that sell globally— has teamed up to help meet the United Nations sustainable development goal that calls for cutting in half per capita global food waste at the retail and consumer levels and reducing food losses along production and supply chains by 2030. As part of this effort to reduce food waste, the Consumer Goods Forum has called for standardized date labels. Other countries have also taken steps to address date labels. For example, in the United Kingdom, in 2008 a nonprofit group conducted research that found that consumers threw out about 22 percent of food that they could have eaten, because they were confused about what the date labels meant. In 2015, the United Kingdom government issued guidance on date labels, specifically that packaged foods must display either a “best before” or “use by” date on the packaging or label of prepacked food products. According to this United Kingdom guidance, a “use by” date label communicates that there may be a safety issue with consuming the product after the date. Furthermore, selling food that is past its “use by” date is prohibited in the United Kingdom. Additionally, the Canadian government has standardized date labels, requiring that prepackaged products with a durable life of 90 days or less be labeled with date markings and storage instructions, where applicable. Such foods must display “best before” and its corresponding French, “meilleur avant.” According to a 2014 USDA report, food loss and waste represents significant amounts of money and other resources invested in food production, including land, fresh water, labor, energy, agricultural chemicals (e.g., fertilizer, pesticides), and other inputs to produce food that does not ultimately meet its intended purpose of feeding people. Furthermore, according to the 2014 report, reducing food waste will become an increasingly important strategy in the future to help feed a growing human population both here and abroad. In the United States, USDA and EPA are leading the federal government’s efforts to reduce food loss and waste, according to officials from the Office of Management and Budget and the Council on Environmental Quality. For example, in 2013, USDA and EPA launched the U.S. Food Waste Challenge for participants across the food supply chain to share best practices on reducing, recovering, and recycling waste. Furthermore, in September 2015, USDA and EPA announced a national goal to reduce food loss and waste in the United States by 50 percent by 2030, which aligns with the United Nations sustainable development goal that calls for cutting in half per capita global food waste. According to FDA officials, the agency was not involved with establishing the national goal because the agency has a limited mission related to food loss and waste; it is primarily responsible for protecting public health by ensuring the safety of the nation’s food supply, among other things. USDA and FDA have taken steps to address consumer confusion about date labels on packaged foods. For example, USDA has issued guidance to consumers and industry, promulgated regulations and implemented policies, and funded research on issues related to date labeling. In addition, FDA has issued information to consumers and supported industry efforts to standardize date labels. In December 2016, USDA’s FSIS announced the availability of a fact sheet that provides guidance related to date labels for industry and consumers. The fact sheet, among other things, explains the meaning of commonly used phrases on date labels and recommends that grocery manufacturers and retailers that use date labels on their products use the language “best if used by” to reduce consumer confusion and resulting food waste. According to an FSIS announcement at the time, the agency chose this phrase because research showed that consumers easily understand the phrase as an indicator of food quality rather than food safety. FSIS also solicited comments on the fact sheet and, in April 2019, after receiving and reviewing comments, updated the fact sheet by, among other things, adding “freeze by’“ to the list of phrases commonly used on labels to describe food quality dates. Figure 3 shows excerpts from the 2019 fact sheet. In addition, FSIS offers a smartphone application (app), called FoodKeeper, that provides information for consumers on the shelf life of products, how to use food when it is at peak quality, and how to store food properly. FSIS developed the FoodKeeper app in 2015 in partnership with Cornell University and the Food Marketing Institute. The app offers users advice on how to store more than 650 food and beverage items, with specific storage timelines for the refrigerator, freezer, and pantry, depending on the nature of the product. In addition, the app allows consumers to note in their devices’ calendars when they purchased the products and to receive notifications when these products are near the end of their recommended storage date, among other things. According to FSIS, the agency is working on an updated version of the app. USDA has promulgated regulations and implemented policies related to required or voluntary date labels on certain products, such as poultry and eggs, and on foods used for its nutrition assistance programs. For example, FSIS has promulgated regulations that require that either the immediate container or the shipping container of all poultry food products be marked with a code or with the date the product was packed. According to FSIS, while USDA does not require date labeling for quality or food safety for products under its purview, the agency requires this “pack date’’ for poultry products to help the agency identify product lots and facilitate trace-back activities in the event of an outbreak of foodborne illness. Additionally, FSIS has promulgated regulations regarding voluntary date labeling. While there are no regulations requiring meat products to have a calendar date, a meat manufacturer may voluntarily place a date on the package. For both poultry and meat products, FSIS’s regulations regarding language on these labels require that this date contain the day and month and be accompanied by a phrase explaining the meaning of the date, specifically “packing” date, “sell by,” or “use before.” The regulations also give manufacturers the option of adding a further qualifying phrase such as “for maximum freshness” or “for best quality.” In the case of meat and poultry products that are hermetically sealed, dried, or frozen, the year must be included as well, to prevent misleading consumers. According to FSIS, a retailer cannot remove or change the date while the product remains in its original packaging if a meat or poultry manufacturer voluntarily places a date on the package. In addition, AMS has promulgated regulations regarding a voluntary egg grademark program, in which egg producers may obtain a USDA grademark, or shield, on their eggs that indicates they meet applicable quality and size standards. The regulations contain requirements related to date labels. Among other requirements to obtain a grademark, the cartons or consumer packaging containing these eggs must show the day of the year on which the eggs were packed. Specific introductory phrasing for the date label—such as sell by, best by, or use before—is not required, but if these terms are used, AMS policies restrict the number of days from the pack date that can be used on a date label. An egg producer may choose to not use an expiration date and still receive AMS certification, but the lot number must be present on each carton. Cartons not identified with a USDA grademark are not subject to federal regulation; however, regardless of whether the eggs bear a USDA grademark, they are subject to state and local date labeling requirements. Furthermore, USDA’s Food and Nutrition Service, which administers 15 federal nutrition assistance programs, policy, last updated in 2017, clarifies that date labels indicate quality, not safety. This policy references the agency’s regulation that prohibits distributors of food assistance from providing food with expired date labels or food that is “out-of-condition,” regardless of the date on the label, to recipients of any Food and Nutrition Service programs. The policy states that, to give program recipients the opportunity to eat all donated foods before their expiration dates, distributors and recipient agencies should use an inventory-management system that distributes products marked with the earliest end date first, even if they were received after other similar products. From 2008 through 2018, USDA provided funding for two grants related to date labels, resulting in three studies: one grant resulted in a 2008 study, and the other grant resulted in studies in 2017 and 2018. The 2008 study assessed participant understanding of date labels on ready-to-eat meat and poultry products to minimize the risk of listeriosis in vulnerable populations. The study found that participants paid attention to the date labels but varied highly in their interpretation of the statements. However, they generally interpreted “sell by” date labels as primarily intended for the retailer’s use on when to pull stock and “best if used by” labels as pertaining more to quality than safety considerations. The researchers reported that participants considered “use by” statements clearer and more helpful than “sell by” or “best if used by” labels and that they believed there was a need for a standardized approach to labeling. The study recommended that if a “sell by” date is used on a product solely for the store to know when to pull a product off the shelf, then a “consume or use by” date should also be implemented on behalf of the consumer. The 2017 study examined, among other things, consumer understanding of phrases on date labels—specifically, “best by,” “fresh by,” “sell by,” and “use by”—on specific products and how these labels affected the participants’ willingness to waste the food. The study found that participants had different levels of willingness to waste food depending on the phrase on the label. In the study, willingness to waste was highest for “use by” and lowest for “sell by,” and this difference held regardless of the product. The researchers suggested that this could be because “use by” may be the least ambiguous and suggestive of food safety, while, conversely, “sell by” may the most ambiguous and least suggestive of food safety. The researchers suggested that if manufacturers move exclusively to the “sell by” date label, this could lead to less waste in the food system. However, while the study identified phrases least likely to result in food waste, it focused on only three products and did not address or make recommendations about steps federal agencies could take to address consumer confusion about date labeling. The 2018 study examined consumer perception of date labels— specifically, “use by” and “best by”—on deli meat and spaghetti sauce. The study found that participants had differing perceptions of date labels by product and what each introductory phrase on the labels meant—that is, whether they reflected safety, quality, taste, or nutrition. Generally, the study found that consumers tended to view “use by” as reflective of safety and nutrition, and “best by” as indicative of quality and taste. The three studies looked at consumer confusion on date labels on certain packaged foods but did not determine which introductory phrase for a date label would be most effective for reducing such confusion across a wide range of products that consumers may purchase, quantify the impact of such confusion on food waste, or determine steps USDA or other federal agencies could take to reduce waste resulting from such confusion. USDA’s 2014 report on food waste noted that food loss (particularly the food waste component) was becoming an increasingly important topic both domestically and internationally. Moreover, according to that report, better estimates of the amount and value of food loss, including food waste, could help serve as quantitative baselines for policymakers and the food industry to set targets and develop initiatives, legislation, or policies to minimize food waste, conserve resources, and improve human nutrition. Previously, we have reported that the nation’s increasingly tight budget environment underscores the need for federal research agencies to set priorities carefully and make effective use of limited research funding. USDA officials told us that their awareness of the role of consumer confusion about date labels and its effect on food waste had increased over time. Furthermore, these officials told us they planned to consider funding additional research if their process for determining research priorities indicates additional research is needed. FDA has taken some actions related to date labeling, such as promulgating regulations regarding date labels on infant formula. For example, since 1985, FDA has required that infant formula display a specific “use by” date on each container of infant formula, which specifies the date after which the formula should not be fed to infants. According to FDA, this label indicates that the manufacturer guarantees the nutrient content and the general acceptability of the quality of the formula up to that date. In addition, since 1993, FDA has published the Food Code, a model for safeguarding public health and ensuring food is unadulterated and honestly presented when offered to the consumer. According to FDA, it represents the agency’s best advice for a uniform system of regulation that address the safety and protection of food offered at retail and in food service, and, while it is not a federal requirement, it is designed to be consistent with federal food laws and regulations. The 2017 FDA Food Code, which is the most recent, contains limited provisions related to date labels applied by manufacturers of packaged foods sold in retail food stores and food-service establishments. For example, the Food Code contains a provision regarding shellfish. It specifies that retailers should only obtain shucked shellfish in packages that identify the “sell by” or “best if used by” date for packages of less than a half-gallon or the date shucked for those of a half-gallon or more. According to Food and Drug Administration (FDA) documents, in response to infant- formula products that were causing illnesses among children because the products lacked sufficient nutrients and industry had too much discretion to decide the appropriate nutritional content of these products, Congress passed the Infant Formula Act of 1980. The act mandates that FDA set uniform standards for the nutritional content of infant formula. Under this act, FDA established a range of regulations affecting infant formula, including a requirement that its labels include “use by” dates. The regulations mandate that manufacturers determine dates on infant formula based on tests that prove the concentration of nutrients is adequate for the health of children up to the marked date. In addition to displaying a “use by” label, manufacturers are required to regularly test for the harmful pathogens (disease-causing bacteria) Salmonella and Cronobacter and demonstrate that the infant formula they produce supports normal physical growth. Additionally, the Food Code’s provisions regarding the labeling of packaged foods state that “food establishment or manufacturers’ dating information on foods may not be concealed or altered.” However, the Food Code is voluntary and does not have provisions for the use of open- code date labels. The Food Code establishes limits for the time that a refrigerated, ready-to-eat food that has been opened or prepared in a food establishment may be held prior to sale or service. The date the food shall be consumed, sold, or discarded must be clearly marked; however, the date is not required to be visible to consumers and is handled separately from the disposition of packaged foods on which a manufacturer has voluntarily placed a date label. The Food Code also specifies how foods prepared in-house using specialized processing methods, such as reduced-oxygen packaging, are to be labeled to ensure they are stored, displayed for sale, or consumed within time limits considered adequate to reduce the risk of foodborne illness. In October 2017, FDA issued information on its website for consumers with food safety tips for foods purchased or received from a charity or bargain store. For example, according to the information, an expired “sell by” date does not necessarily mean that a food is spoiled or unsafe. However, in some cases, if food has not been handled safely, illness- causing bacteria may grow. In addition, the information states that consumers should avoid purchasing packaged foods that require refrigeration and that are past the “use by” or “sell by” dates because these foods may be perishable and may have begun to spoil. Moreover, in May 2019, FDA published an educational fact sheet for consumers on reducing food waste while maintaining food safety. This fact sheet includes information about the meaning of language on date labels, as consumers may waste food if they misunderstand what date labels actually mean. For example, it explains that a “best if used by” date indicates that a product will be at its best flavor and quality. On the fact sheet, FDA recommends consumers download USDA’s FoodKeeper app to know how long various food products will keep in the pantry, in the refrigerator, and in the freezer. Furthermore, FDA has promoted a voluntary industry initiative to standardize approaches to date labeling of packaged foods and improve consumer understanding of the meaning of date labels. In May 2019, FDA issued a letter to industry that described FDA’s position on the voluntary industry standard proposed by the Food Marketing Institute and the Grocery Manufacturers Association in January 2017. This voluntary industry standard called for using the “best if used by” introductory phrase in quality-based date labels on packaged foods. FDA said the agency strongly supports industry’s voluntary efforts to use the “best if used by” introductory phrase when grocery manufacturers choose to include a quality-based date label to indicate when a product will be at its best flavor and quality. While the Food Marketing Institute and the Grocery Manufacturers Association recommended the use of the introductory phrase “use by” to indicate the date by which products should be consumed or discarded for safety reasons, FDA did not address the “use by” product date label for safety reasons in the agency’s letter to industry. USDA and FDA have coordinated on some initiatives focused specifically on date labels on packaged foods. For example, USDA and FDA officials told us the agencies are working together to develop information for food banks, food donors, and recipients of donated food regarding how to interpret date labels on packaged foods donated to food banks to ensure that food that is past the date on the label—but otherwise edible—is not wasted. USDA officials told us the agencies plan to finalize this information in 2019. USDA and FDA are also collaborating with EPA on an initiative to reduce food loss and waste. In October 2018, USDA, FDA, and EPA signed a formal interagency agreement, referred to by the agencies as the Winning on Reducing Food Waste Initiative. The formal agreement states the agencies are committed to increasing collaboration and coordination in existing federal programs in areas of mutual interest relating to the reduction of food loss and waste, and to developing an interagency strategy to address this issue. According to USDA’s website, as part of this collaborative effort, the agencies agreed to coordinate food loss and waste actions, such as education and outreach, research, community investments, voluntary programs, public-private partnerships, tool development, technical assistance, event participation, and policy discussions on the impacts and importance of reducing food loss and waste. According to the agreement, the agencies will seek to work together at the federal level with actors throughout the entire food supply chain to leverage the private and nongovernmental sectors. Specifically, the agreement states the agencies will seek to educate these actors on best practices to reduce food loss and waste in the growing, manufacturing, transporting, selling and disposing of food, handling, preparation and storage of food, as well as creating new uses for excess food. After announcing the formal agreement in October 2018, the three agencies in April 2019 announced a federal interagency strategy to prioritize and coordinate their efforts as they implement the formal agreement. This strategy identifies date labeling as a priority action area and states that “establishing and communicating clearer, coordinated voluntary guidance on food date labels and liability protection around food donation could help increase food recovery and lead to reductions in food waste and food insecurity.” In the strategy, the agencies state that they built on information from several sources, including EPA and USDA’s Call to Action by Stakeholders: United States Food Loss & Waste 2030 Reduction Goal and two reports from nonprofit organizations, all of which cited clarifying or standardizing date labels as a key element for food loss and waste reduction efforts. Establishing a formal agreement is a positive step and aligns with leading practices for interagency collaboration. We have previously found that interagency collaborations benefit from collaborative mechanisms, such as written agreements, in that agencies can strengthen their commitment to working collaboratively, which USDA and FDA have done. In addition to steps they have taken toward interagency collaboration, USDA and FDA have taken steps to work with some nonfederal stakeholders—nonprofit organizations and an international organization— on date labeling. For example, in February 2019, USDA, EPA, and FDA met with NRDC and the Ad Council to discuss campaigns to inform the public about ways to reduce food loss and waste, which includes consumer education on date labeling. According to FDA officials, the agencies will meet with the two groups later in 2019 to continue this discussion. In addition, in April 2019, in keeping with the goal in the agencies’ Winning on Reducing Food Waste Initiative to expand collaboration with nonfederal stakeholders, EPA, FDA, and USDA signed a formal agreement with ReFED to collaborate on efforts to reduce food loss and waste. The agreement outlines actions the agencies and ReFED agree to take, including that the agencies will consult with ReFED to develop approaches for measuring the success of various strategies and techniques being deployed nationwide to reduce food waste. Furthermore, USDA and FDA provide senior staff and executive delegates to represent the United States at the Codex Alimentarius Committee on Food Labelling, among other committees under the purview of the Codex Alimentarius Commission. The commission, an international intergovernmental body, produces the Codex Alimentarius, or “Food Code,” a collection of standards, guidelines, and codes of practice related to food, food production, and food safety. In 2018, the commission revised its voluntary guidance on date labeling to clarify the distinction between dates based on food quality and those based on food safety. Because the Codex Alimentarius is a voluntary reference standard, its guidance, including on date labels, is not binding on member countries, including the United States. However, USDA and FDA have not consulted with all relevant stakeholders. For example, state and local officials we spoke with told us that USDA and FDA had not collaborated with them or consulted them on approaches to date labels. In our prior work, we identified a leading practice for interagency collaboration that calls for ensuring that the relevant participants are included in interagency collaborative efforts. Such efforts can include other federal agencies; state, local, and tribal governments; industry; and nonprofit advocacy organizations. Generally, state, local and tribal governments may choose whether to regulate date labels on packaged foods. For example, the majority of states and the District of Columbia have date labeling requirements. Advocacy organizations and state officials told us that efforts at reducing consumer confusion about date labels could be hindered without federal leadership, as states may continue to have varying approaches. USDA and FDA officials told us that they did not have a specific mechanism to coordinate or consult with state, local, or tribal officials on creating a common approach to date labels. By developing a mechanism to facilitate coordination with nonfederal stakeholders—including state, local, and tribal governments—on actions related to date labels as part of their efforts to reduce food loss and waste, USDA and FDA could better assure that approaches they take to address consumer understanding of date labels are effective in helping reduce consumer confusion and resulting effects such as wasted food. USDA and FDA have taken important steps toward reducing consumer confusion about date labels by, among other things, providing information to consumers and, in USDA’s case, by conducting research on food waste. In addition, in October 2018, USDA, FDA, and EPA signed a formal agreement aimed at improving coordination and communication across federal agencies to educate Americans about the benefits of reducing food loss and waste. However, although USDA and FDA have taken steps to work with some nonfederal stakeholders, such as ReFED, on date labels, they have not worked with state, local, and tribal governments. We have identified that ensuring relevant stakeholders have been included in the collaborative effort as a leading collaboration practice. By developing a mechanism to facilitate coordination with relevant nonfederal stakeholders on actions related to date labels as part of their efforts to reduce food loss and waste, USDA and FDA could better assure that approaches they take to address consumer understanding of date labels are effective in helping reduce consumer confusion and resulting effects such as wasted food. We are making two recommendations to the agencies in our review: The Secretary of Agriculture should work with the Commissioner of FDA to develop a mechanism to facilitate coordination with relevant nonfederal stakeholders, including state, local, and tribal governments, on actions related to date labels as part of their efforts to reduce food loss and waste. (Recommendation 1) The Commissioner of FDA should work with the Secretary of Agriculture to develop a mechanism to facilitate coordination with relevant nonfederal stakeholders, including state, local, and tribal governments, on actions related to date labels as part of their efforts to reduce food loss and waste. (Recommendation 2) We provided a draft of this report to the Departments of Agriculture and Health and Human Services for review and comment. In its comments, reproduced in appendix I, USDA agreed with our recommendation to the agency and described current and future actions to implement the recommendation. Similarly, in its comments, reproduced in appendix II, the Department of Health and Human Services agreed with our recommendation to it and described current and future actions to implement the recommendation. USDA and the Department of Health and Human Services provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees; the Secretaries of Agriculture and Health and Human Services; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or morriss@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made contributions to this report are listed in appendix III. In addition to the contact named above, Anne K. Johnson (Assistant Director), David Bennett (Analyst-in-Charge), Tara Congdon, and Jordan Mettica made key contributions to this report. Carol Bray, Kevin Bray, Serena Lo, Oliver Richard, Danny Royer, Kiki Theodoropoulos, and Sarah Veale also contributed to this report.", "summary": "USDA has reported that almost one-third of the U.S. food supply is lost or wasted at the retail and consumer levels. Studies indicate that some of this waste may occur because of consumer confusion about the meaning of date labels displayed on packaged food. Such labels are not federally regulated, and food manufacturers use different phrases on date labels. USDA and FDA have roles in ensuring the U.S. food supply is safe and properly labeled, but neither agency been directed—or given express authority—to regulate date labels. GAO was asked to examine consumer confusion about date labels. This report (1) describes the steps USDA and FDA have taken to address consumer confusion about date labels and (2) examines the extent to which USDA and FDA have coordinated with each other and with nonfederal stakeholders on date labels. GAO reviewed studies on date labels and FDA and USDA documents; interviewed agency officials and representatives of nonfederal stakeholders, such as industry, advocacy organizations, and state governments; and compared the agencies' efforts to leading practices identified by GAO. The U.S. Department of Agriculture (USDA) and Food and Drug Administration (FDA) have taken steps to address consumer confusion about date labels on packaged foods. For example, to reduce confusion about introductory phrases on date labels, such as whether the dates indicate food is safe to eat (see figure), and resulting food waste, USDA in December 2016 issued a fact sheet on date labels for consumers. In addition, USDA has funded research on issues related to date labels (e.g., how labels affected participants' willingness to waste food) and developed a smartphone application that provides consumers with information on the shelf life of products. FDA has issued educational materials to consumers about the meaning of phrases on date labels and in May 2019 issued a statement that it supports industry efforts to standardize date labels. USDA and FDA have coordinated on some initiatives focused on date labels on packaged foods. For example, agency officials said they were working together to develop information for food banks, food donors, and recipients of donated food on how to interpret date labels so food past the date on the label—but otherwise wholesome—is not wasted. In October 2018, the agencies, with the Environmental Protection Agency, signed a formal agreement to educate consumers about food loss and waste. In addition, USDA and FDA have taken steps to work with some nonfederal stakeholders—such as nonprofit organizations and an international organization—on date labeling. However, USDA and FDA officials told GAO that they do not have a specific mechanism to coordinate with state, local, and tribal officials on creating a common approach to date labels. State, local, and tribal governments may choose to regulate date labels, and the majority of states have date label requirements for certain foods. According to prior GAO work, ensuring that relevant participants are included in interagency collaborative efforts is a leading practice for interagency collaboration. By developing a mechanism to facilitate coordination with nonfederal stakeholders, such as state, local, and tribal officials, on actions related to date labels, USDA and FDA could better assure that approaches they take to address consumer understanding of date labels are effective in helping reduce consumer confusion. GAO is recommending that USDA and FDA develop a mechanism to facilitate coordination with relevant nonfederal stakeholders on actions related to date labels. USDA and FDA agreed with our recommendation and are planning actions to implement the recommendation.", "document_type": "gao"}
{"report": "According to ATF and USMS policy, the Directors of ATF and USMS have the authority to develop various policies, procedures, and guidance that specify the steps the components must or should take while investigating and adjudicating employee misconduct. ATF and USMS can receive allegations of employee misconduct from a variety of sources, including agency staff, the general public, and the DOJ OIG. Allegations of employee misconduct can include, for example, not following procedures associated with managing government-issued property or not reporting time and attendance accurately. Employee misconduct can occur outside of the workplace as well, such as local arrests of employees for domestic violence or driving under the influence of alcohol. ATF and USMS each have an intake or hotline function that is to initially assess the reported information and seriousness of each allegation to determine the appropriate next step in terms of which group or office within their respective component will conduct an investigation, if warranted. The investigation process involves engaging in fact-finding to the extent necessary to make an informed decision on the merit of an allegation. In accordance with ATF and USMS policy, for each misconduct allegation received, the components’ investigative office (Internal Affairs) must provide the DOJ OIG with “right of first refusal.” This review allows the DOJ OIG to either open an investigation or send the allegation back to the component for action. If the DOJ OIG declines the opportunity to investigate, the components assign the case to Internal Affairs. Specifically: For ATF, cases that involve matters related to integrity are investigated by ATF Internal Affairs, while other cases are generally referred to ATF divisions to conduct inquiries (known as management referrals). USMS typically assigns higher-level (i.e., more egregious) misconduct cases to Internal Affairs. For cases typically considered to involve lower-level offenses, USMS managers in divisions or districts conduct inquiries or fact finding locally. Each component has policies, procedures, and guidance for its Internal Affairs and local management for investigating cases of employee misconduct. Based on the investigative findings, the responsible office for each component can make a preliminary determination of whether there is sufficient evidence to support an allegation. After investigations are completed, each component has an adjudication process whereby delegated officials propose discipline. For ATF, a headquarters entity—referred to as the Professional Review Board— proposes discipline for all cases investigated by ATF Internal Affairs. For cases involving misconduct by ATF employees outside of Internal Affairs jurisdiction, division management proposes and decides discipline. USMS utilizes various, delegated agency officials to propose and decide discipline depending on the type of investigation. Discipline for both ATF and USMS employees can range in severity, depending on the unique findings and circumstances of each investigation. For misconduct within USMS warranting a suspension of 14 days or less, local management proposes and decides discipline. For both ATF and USMS, during adjudication, proposing and deciding officials determine whether an allegation is substantiated or unsubstantiated when considering if an action is warranted. For substantiated cases that are determined to warrant action, components use their respective Table of Offenses and Penalties as a guide for disciplinary actions, which provides guidance for determining appropriate penalties. Each component is to provide employees with a letter of proposed discipline and an opportunity to respond before it makes a final decision on the discipline. After discipline is proposed and the employee’s response is considered, final discipline is determined by a delegated official (deciding official), distinct from the proposing official. In addition, delegated officials are to consider particular mitigating and aggravating factors on a case-by-case basis when determining the appropriate penalty for an act of employee misconduct. The relevant factors that are considered, as appropriate, in determining the severity of the discipline include, but are not limited to, the nature and seriousness of the offense and its relation to the employee’s duties, position, and responsibilities. This includes considering whether the offense was intentional or inadvertent, or was committed maliciously or for gain; the employee’s past disciplinary record; and whether the offense was frequently repeated. For both ATF and USMS, there are three categories of employee misconduct outcomes: Corrective/Non-disciplinary action. This is an administrative or non- disciplinary action, such as a letter of counseling or a letter of guidance and direction, that informs an employee about unacceptable performance or conduct that should be corrected or improved. Disciplinary action. This includes actions resulting in a letter of reprimand up to a suspension of 14 days or less. A letter of reprimand describes the unacceptable conduct that is the basis for a disciplinary action, and represents the least severe form of disciplinary action. Suspensions in this category involve the placement of an employee in a nonduty, non-pay status for up to and including 14 days. Adverse action. This involves a suspension of more than 14 days (including an indefinite suspension), demotion to a lower pay band or rate of pay, or removal (an involuntary separation from employment). According to the U.S. Merit Systems Protection Board, an indefinite suspension is appropriate when evidence exists to demonstrate misconduct of a serious nature, such as an employee has committed a crime for which a sentence of imprisonment can be imposed, when the agency has concerns that an employee’s medical condition makes the person’s presence in the workplace dangerous or inappropriate, or when an employee’s access to classified information has been suspended. Also, according to the board, a demotion is a reduction in grade or a reduction in pay, while a removal terminates the employment of an individual. Figure 1 provides an overview of ATF and USMS employee misconduct processes. ATF and USMS have case management systems that are designed to maintain employee misconduct data––such as the date of the alleged incident, source of the allegation, description of the alleged misconduct, and the status of the investigation. ATF’s Professional Review Board uses another system to manage outcome data associated with Internal Affairs investigations. After adjudication of ATF Internal Affairs investigations, the board is to provide this outcome data to ATF Internal Affairs for inclusion in its system. Similarly, after the adjudication of management referrals for action, ATF managers are to provide outcome data to ATF Internal Affairs to include in its system. In addition to the system USMS uses to manage Internal Affairs investigations, the agency has a separate system to record outcome data. Our analysis of ATF employee misconduct data found that ATF opened 1,581 employee misconduct investigations during fiscal years 2014 through 2018. As shown in table 1, the majority of ATF misconduct cases during this period were management referrals to divisions for informational purposes or for action. Table 2 shows that the most common allegation category of misconduct that ATF received from fiscal year 2014 through 2018 was job performance failure, representing 8 percent of all allegations, which includes not attending meetings, submitting reports of inspection late, or becoming agitated during performance feedback, among other things. After investigations are completed, results are forwarded to the Professional Review Board for adjudication, and adjudication results are to be entered into ATF’s Human Resources system. For investigations that were adjudicated during the period we reviewed, six types of offense categories made up about 60 percent of those substantiated and captured in the ATF Human Resources system, as shown in figure 2. The exercise of poor judgment (14 percent) and the failure to adequately secure government property (13 percent) were the most common offenses. The employee misconduct outcomes for offenses ranged from corrective actions (e.g., letters of counseling or caution) to adverse actions such as suspensions and removals. Specifically, of the 503 investigations that had final actions reported in ATF case management system, disciplinary action—suspensions of 15 days or less and letters of reprimand— accounted for 176 (about 35 percent) of the final outcomes. Also, 135 (about 27 percent) of investigations adjudicated resulted in corrective actions (cautions such as a verbal or written warning). Further, 87 (about 17 percent) of these 503 investigations and management referrals were closed for various reasons, such as insufficient evidence of an employee’s inappropriate behavior or clearance of the charges after investigation, while adverse actions represented 47 (about 9 percent) of these outcomes, as shown in figure 3. Our analysis of USMS employee misconduct data show that USMS opened 2,347 employee misconduct investigations during fiscal years 2014 through 2018 that were also closed at the time USMS responded to our request for information. As shown in table 3, USMS Internal Affairs investigated the majority of the component’s employee misconduct cases. As shown in table 4, the most common misconduct allegations for USMS were violations of the code of professional responsibility (21 percent), conduct unbecoming or discourteous behavior (13 percent), and failure to follow procedures (12 percent). As shown in figure 4, general misconduct while on duty and failure of staff to follow instructions were the most frequent offenses from fiscal years 2014 through 2018, representing 383 (about 25 percent) and 266 (about 18 percent) of offenses respectively. Additionally, according to USMS adjudication data, of the 2,347 investigations that were opened in fiscal years 2014 through 2018, USMS had adjudicated 1,729 misconduct cases at the time USMS responded to our request for information (March 2019 for investigations opened in fiscal years 2014 through 2017 and April 2019 for investigations opened in fiscal year 2018). As shown in figure 5, the most common disciplinary outcomes for USMS were non-adverse actions (corrective and disciplinary actions), which accounted for 988 (about 58 percent) of final outcomes. USMS did not take disciplinary action on 533 (about 31 percent) of completed investigations forwarded for adjudication. The deciding official will not determine an action against an employee if he or she does not believe the allegations warrant action. Adverse actions were less common, with removals, suspensions of 15 days or more, and demotions accounting for 83 (about 5 percent) of all employee actions. The remaining 120 (about 7 percent) of completed investigations forwarded for adjudication resulted in retirements, resignations, transfers and other outcomes such as settlement agreements. According to the U.S. Merit Systems Protection Board, to prove a claim of management retaliation, the investigation must show that the employee engaged in a protected activity (e.g., filing an EEO claim); the agency official with knowledge of the employee’s protected activity took, failed to take, or threatened to take a personnel action against the employee; and there is a causal connection between the protected activity and the personnel action. From fiscal years 2014 through 2018, ATF and USMS employees submitted 70 claims of management retaliation directly to their Internal Affairs division or the DOJ OIG, and about 240 to their EEO Office. OSC does not record data in its case management system related to DOJ employee disclosures (claims) by component. From fiscal years 2014 through 2018, ATF, USMS, and the DOJ OIG completed 70 investigations of employee misconduct that alleged management retaliation. ATF Internal Affairs retaliation investigations. According to ATF investigations data, from fiscal years 2014 through 2018, ATF Internal Affairs investigated 23 cases alleging management retaliation. Of these 23 cases, Internal Affairs referred 20 to division management for informational purposes. Of the three cases that were investigated by ATF, two cases were investigated by division management and resulted in the employees being counseled by their supervisors. The third case was investigated by Internal Affairs and resulted in one employee receiving a clearance letter and another receiving a letter of caution, with another two employees retiring. USMS Internal Affairs retaliation investigations. According to USMS investigations data, from fiscal years 2014 through 2018, USMS Internal Affairs investigated 26 cases alleging management retaliation. Of these 26 cases, 12 were closed after the investigation was completed due to insufficient evidence. Of the remaining 14 cases, four resulted in employees retiring during or after adjudication, four had no employee action, three closed due to ongoing related cases, and there was one oral admonishment, one letter of counseling, and one suspension of 14 days. DOJ OIG retaliation investigations. According to our analysis of DOJ OIG data, from fiscal years 2014 through 2018, the DOJ OIG investigated 21 ATF or USMS cases alleging management retaliation (four ATF cases and 17 USMS cases). The DOJ OIG filed all four ATF cases in its management system for informational purposes only (no action), and also sent one of the four cases to ATF for informational purposes. Of the 17 USMS cases, the DOJ OIG filed 12 cases in its management system for informational purposes (no action), found that three cases lacked sufficient evidence, closed one case due to one of the involved employees being reassigned and the other resigning, and in one case made a procedural recommendation to the Director of USMS. Figure 6 shows the number of ATF, USMS, and DOJ OIG management retaliation investigations from fiscal years 2014 through 2018. ATF and USMS employees may file claims of management retaliation through their agency’s EEO office. We analyzed ATF and USMS employee misconduct and EEO data to determine (1) the number of employees who had filed an EEO claim of management retaliation and (2) whether these employees were also subject to a misconduct investigation. ATF EEO management retaliation investigations. From fiscal years 2014 through 2018, the ATF EEO Office received 128 claims from 104 employees that included management retaliation as the basis, but none of these claims have been found to support a finding of retaliation. ATF EEO and employee misconduct data show that employees in 54 of the 128 EEO cases (36 total individuals) were also subject to misconduct investigations that were adjudicated during this time period. Of the 36 employees, 24 submitted their EEO claim subsequent to their misconduct investigation. The remaining 12 employees submitted their EEO claim prior to their first employee misconduct investigation. Figure 7 shows the number of ATF employees who filed EEO claims of management retaliation and were also the subject of an employee misconduct investigation. USMS EEO retaliation claims. From fiscal years 2014 through 2018, the USMS EEO Office received 110 claims from 69 individuals with management retaliation as the basis, of which one resulted in a final agency decision supporting the claim. USMS EEO and employee misconduct data show that individuals in 75 of the 110 EEO cases (49 total individuals) were also subject to a total of 134 employee misconduct investigations that were adjudicated from fiscal years 2014 through 2018. Of these 49 individuals, 32 submitted their EEO complaint subsequent to their misconduct investigation. The remaining 17 employees submitted their EEO claim prior to their first employee misconduct investigation, of which three claims resulted in a settlement agreement. Figure 8 shows the number of USMS employees who filed EEO claims of management retaliation and were also the subject of an employee misconduct investigation. From fiscal years 2014 through 2018, OSC did not report any instances of management retaliation for ATF or USMS. OSC reported one investigation related to one USMS employee who improperly secured personally identifiable information, for which USMS took corrective actions. According to data maintained in an ATF Office of Chief Counsel case management system, ATF recorded eight instances where ATF counsel rendered assistance to OSC on retaliation-related matters. USMS Office of General Counsel does not maintain OSC-related data in any USMS case management system. ATF and USMS have incorporated some key internal controls for processing employee misconduct allegations into their policies and procedures, but have not consistently documented the implementation of these controls. ATF and USMS have also established policy requirements related to timeliness in completing employee misconduct investigations, but have not established performance measures to monitor all of these requirements. Further, both ATF and USMS have established mechanisms to monitor various aspects of the components’ operations, but do not use these mechanisms to fully monitor key internal controls related to their employee misconduct investigation and adjudication processes. ATF and USMS documented the implementation of some key control activities that are important for ensuring the quality and independence in processing allegations of employee misconduct. However, they did not document other key control activities. Supervisory review of investigations. According to Federal Quality Standards for Investigations, supervisory or management review of misconduct investigations helps ensure that investigations are comprehensive and performed correctly. ATF and USMS both require this review in policy for misconduct investigations and have incorporated it in their respective procedures. Both ATF and USMS also have a policy or procedure for documenting this control activity in either their case management system or case file records. We found that ATF consistently documented supervisory review of its employee misconduct investigations. Overall, based on our case file reviews, we estimate that 98 percent of the population of ATF investigations or management referrals for action from fiscal year 2014 through fiscal year 2018 documented supervisory review. For our sample, we found documentation of supervisory review in all 36 Internal Affairs investigations and all 26 management referrals for action. We also found supervisory review for all 12 investigations or referrals in our sample with proposed adverse actions and all nine investigations or referrals in our sample that involved an individual who had filed an EEO claim of management retaliation. For USMS, we found that the agency consistently documented supervisory review of its Internal Affairs investigations, but did not consistently document this review for its district and division investigations. Overall, based on our case file reviews, we estimate that 60 percent of the population of USMS investigations (2,347) from fiscal year 2014 through fiscal year 2018 documented supervisory review. For our samples, we found documentation of supervisory review in 29 out of 30 of Internal Affairs investigations. However, for USMS district and division investigations, we found that 23 of 59 investigations had documentation of supervisory review through the required use of a field incident report. We also found that all 20 investigations in our sample with proposed adverse actions had documentation of supervisory review. Further, we found that six of the 12 USMS investigations in our sample that involved an individual who had filed an EEO claim of management retaliation had documentation of supervisory review. The remaining six cases without documentation of supervisory review were district or division investigations, which are typically considered to involve lower-level offenses. Although USMS policy on Field Operational Reports requires the use of a standard form to document supervisory review for district and division misconduct investigations, USMS officials stated that district and division management periodically document a completed investigation with an electronic email confirmation for various reasons, including that the investigation may involve non-adverse actions. However, according to USMS policy, a memorandum does not serve as a substitution for the required field report. Taking steps to ensure that supervisory review of division and district investigations is documented in accordance with USMS policy would provide greater management assurance that investigations are performed comprehensively and consistently, and that this control is operating as intended. Legal sufficiency review. ATF policy on Integrity and Other Investigations states that managers will review the investigative findings with the Office of Chief Counsel’s management division to propose and decide discipline or other actions. ATF also has procedures for documenting these activities in its case management systems. We found that ATF consistently documented legal sufficiency review during the adjudication phase for its Internal Affairs investigations. Specifically, we found that 32 of 36 cases investigated by Internal Affairs documented legal counsel review during the adjudication phase. One case of these 32 had review for the proposal, but was ultimately cleared. For the four cases without documentation of legal counsel review, this review was not applicable. Specifically, one case involved an employee who received a clearance letter; one case was still pending a final decision; one case involved an employee who was on military leave; and one case involved an employee who had retired. We also found that legal counsel review was documented in 11 of the 12 cases in our sample where adverse action was proposed—all of which were investigated by Internal Affairs—and the remaining case was still pending adjudication as of August 2019. Further, we found documentation of legal counsel review for six of the nine employee misconduct investigations that involved an EEO claim of management retaliation. Of the three investigations that did not have documentation, one was an Internal Affairs case where the final action was pending, and the other two cases were management referrals for action. Regarding ATF Internal Affairs investigations referred to division management for action, we found that legal counsel review was documented for nine of 26 cases during the adjudication phase for the proposed discipline, the final disciplinary action, or both. Documenting legal counsel review for cases referred to division management for action would provide ATF management greater assurance that all proposed discipline or other actions are legally sufficient. Although ATF policy requires managers to review investigative findings with the Office of Chief Counsel when handling management referrals, ATF officials stated that supervisors may handle the matters within the division without informing or consulting with legal counsel if there is no proposed discipline. According to ATF officials, the agency plans to revise its policy on Integrity and Other Investigations in August 2020, the next scheduled recertification of the order, to allow managers discretion in determining whether legal review is needed in instances where discipline is not imposed. USMS policy on Discipline Management Business Rules requires legal review for Internal Affairs investigations that involve a proposed adverse action, but does not require legal reviews for investigations that involve non-adverse actions. USMS also has procedures for documenting this activity in its case management system and physical case files. We found that USMS consistently documented the legal sufficiency internal control. Specifically, we found that all of the 20 proposed adverse actions in our sample documented legal counsel review. Of the 12 cases in our sample that involved an individual who had also filed an EEO claim, three had proposed adverse actions, all of which had documentation of USMS legal review. DOJ OIG right of first refusal. According to ATF and USMS policies on misconduct investigations and management referrals, for each misconduct allegation received, the components must provide the DOJ OIG the opportunity to review the case for right of first refusal. This review allows the DOJ OIG to either open an investigation or defer the case back to the component for investigation. This review is designed to maintain independence by determining which cases warrant investigation outside of ATF and USMS. We found that ATF and USMS consistently forwarded allegations of employee misconduct to the DOJ OIG for right of first refusal. Specifically, our analysis of ATF and DOJ OIG data found that the DOJ OIG did not have a record of receiving five out of 1,581 ATF investigations or management referrals for right of first refusal. There were also 41 instances for which ATF did not have a DOJ OIG case number, which prevented the DOJ OIG from checking its records for evidence that ATF had forwarded the case for right of first refusal. We found that 37 of the 41 cases occurred in fiscal years 2014 or 2015, with only four cases occurring in fiscal years 2016 through 2018. Our analysis of USMS and DOJ OIG data found that the DOJ OIG did not have a record of receiving 10 out of 2,347 investigations for right of first refusal. Verification of accuracy of case management system data. ATF and USMS do not have a policy requirement for the use of a method or tool to verify system data associated with both investigation and disciplinary processes. However, according to Standards for Internal Control in the Federal Government, management is to use quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks. The standards also state that data maintains value to management in controlling operations and making decisions, and management is to design control activities so that all records are complete and accurate. Regular reviews of case management data can identify outliers or abnormalities, such as missing information. ATF officials stated that agency managers verify that the initial information related to the allegation is accurate in the case management system. However, additional reviewers in the misconduct process do not verify investigation and adjudication information subsequent to the allegation in the case management system. The officials added that after Internal Affairs investigations and management referrals for action are completed, the record of investigation and supporting materials are reviewed by management to assess the quality of the investigation before uploading to the case management system. However, we found that information related to the investigation and adjudication of these allegations was sometimes not captured in automated data fields. Since uploaded documents cannot be analyzed easily, the Office of Professional Responsibility manually reviews these documents to compile an annual report on employee misconduct activities, such as the number of investigations and outcomes. According to ATF and USMS officials, employee misconduct procedures include supervisor review in several areas. For example, ATF and USMS officials stated that managers review reports of investigation and other documents to ensure certain information is recorded in case files or case management systems. ATF officials provided evidence that they verify certain data when a case is initiated, such as the identity of the subject and allegation. ATF officials also provided evidence that managers review the report of investigation for quality. USMS officials stated that they confirm that the employee under investigation is the correct employee in the system record and that the case was referred to the DOJ OIG for right of first refusal. ATF officials also stated that reviewers involved in employee misconduct processes compare case file documentation against case management system records. However, we found that hundreds of case management system records were missing key information, such as the final outcomes of employee misconduct investigations and DOJ OIG case numbers for ATF, and dates related to district or division investigations for USMS. We also found that ATF and USMS lack policy for verifying the accuracy and completeness of data recorded in their respective employee misconduct case management systems. This policy could be implemented, for example, through the use of a method or tool, such as a data entry checklist, that would guide agency officials when entering information into systems. Establishing policy could help ensure that case management system data are accurate and complete and would allow ATF and USMS to effectively monitor and report on their employee misconduct processes. ATF and USMS have established requirements in their policies regarding timeliness in completing employee misconduct investigations. However, ATF has not developed performance measures to monitor its timeliness requirements. USMS has developed a measure to monitor its Internal Affairs investigations, but not for its district and division investigations. Standards for Internal Control in the Federal Government state that management should define objectives in measurable terms so that responsible personnel and management are held accountable, and their performance toward achieving those objectives can be assessed. ATF policy on Integrity and Other Investigations requires completing Internal Affairs investigations generally within 120 days, and management referrals for action within 60 days. ATF officials acknowledged the importance of addressing employee misconduct allegations in a timely manner. For example, ATF may withhold a positive human resource action or personnel assignment pending completion of a misconduct investigation, such as a promotion or becoming a member of a task force. ATF employees under investigation for misconduct may also be placed on restricted duty, which depending on the case may prevent the employee from accessing information systems and require the employee to surrender his or her government-issued firearms, vehicle, other property, and credentials. ATF officials stated that ATF management tracks ongoing investigations—for both Internal Affairs investigations and management referrals for action—and the amount of time they are open. ATF Internal Affairs officials stated that managers track the duration of all investigations on a weekly basis, and will inquire about the status of investigations and reasons why any exceed the duration standards. However, ATF has not developed a performance measure to monitor performance against timeliness requirements—for example, whether a certain percent of Internal Affairs investigations during a definite time period were completed within the required 120 days. Based on our analysis of ATF data, Internal Affairs met its policy requirement of completing its investigations within 120 days about 36 percent of the time (86 of 240 investigations). ATF data also show that the agency met its policy requirement of 60 days for about 49 percent (205 of 419) of its management referrals for action (see fig. 9). According to ATF officials, ATF does not use measures to monitor performance related to the duration of Internal Affairs investigations and management referrals for action due to numerous factors, such as investigators handling multiple cases at the same time and the involvement of the DOJ OIG. We have previously reported that other federal agencies have established such performance measures, which have taken these challenges into account when developing their methodology for measuring timeliness. Establishing a performance measure to monitor the timeliness of Internal Affairs and management referrals for action could provide ATF management more complete information in overseeing investigations and help improve the efficiency of employee misconduct processes. USMS policy requires completing Internal Affairs investigations within 90 days, and within 30 days for investigations referred to its districts and divisions. USMS officials noted the importance of addressing employee misconduct allegations in a timely manner, with regards to effecting positive human resource actions such as promotions. USMS Internal Affairs has developed a performance measure to monitor whether it is completing its investigations within the required 90-day time frame. According to USMS officials, the agency plans to change the required time frame for completing Internal Affairs investigations from 90 days to 180 days, which according to the officials is a time standard used by most other law enforcement agencies. USMS does not have a performance measure to monitor the duration of investigations conducted by its districts and divisions. According to USMS officials, these investigations do not involve high-level offenses that would pose a significant risk to the agency. Based on our analysis of USMS data, Internal Affairs met its policy requirement of completing its investigations within 90 days 35 percent of the time (468 of 1,320 investigations for which data were recorded in USMS systems), as shown in figure 10. Our analysis also shows that USMS met its policy requirement of completing its district and division investigations within 30 days over 99 percent of the time (489 of 490 investigations for which data were recorded in USMS systems). Although we found that USMS met its timeliness requirement related to district and division investigations over 99 percent of the time, management responsible for oversight have not developed a performance measure to monitor whether the agency meets its policy requirement. Therefore, the agency will not be able to identify any potential future performance issues. Monitoring these investigations is also important since data on the duration for about 25 percent (165 of 655) of district and division investigations that were opened from fiscal years 2014 through 2018 were not recorded in USMS systems at the time the agency provided the data. Developing a measure for the duration of district and division investigations would provide USMS leadership with greater assurance that the agency is complying with policy requirements. ATF and USMS do not use their existing oversight mechanisms to fully monitor key internal controls related to employee misconduct processes. Standards for Internal Control in the Federal Government call for management to establish and implement activities to monitor the internal control system and evaluate the results, as well as remediate identified internal control deficiencies. ATF has two oversight mechanisms that it uses to monitor internal controls related to financial reporting, compliance activities, and operations—annual self-assessments and internal management reviews. However, according to ATF officials, the component does not use these mechanisms to monitor any internal controls related to its employee misconduct processes. Specifically, according to an ATF official, as part of ATF’s annual self- assessment program, all component divisions, including Internal Affairs, are to test financial processes, such as government credit card payments. The ATF Inspection Division also conducts internal management reviews to test compliance with the same activities that are covered by the self- assessment program. ATF officials stated that the scope of the self- assessment program does not include key internal control activities related to employee misconduct processes due to competing priorities. According to an Inspection Division official, the division also has not conducted an internal management review of the offices responsible for employee misconduct processes (e.g., the Internal Affairs division, the Professional Review Board, Bureau Deciding Official activities) in about 10 years due to competing priorities. ATF officials stated that the agency plans to review these divisions and offices in the future, but did not have any specific plans for how internal management reviews would be used for divisions and offices in the misconduct process or when these reviews would begin. While the scope of these reviews has not been determined, the officials stated that internal management reviews could include testing internal control activities related to allegations of employee misconduct, such as investigative review and approval, legal sufficiency review; and case management information system data reliability and completeness. Monitoring key internal controls related to employee misconduct processes through existing oversight mechanisms would help ATF management ensure that controls are being implemented as required by policy. USMS has two oversight mechanisms that it uses to monitor internal controls related to financial reporting, compliance activities, and operations. Specifically, USMS’s Compliance Review Office, within the Office of Professional Responsibility, conducts on-site management reviews at USMS districts and divisions. USMS also has an annual self- assessment program that requires divisions and districts to self-assess their compliance with certain requirements by testing for and remediating any internal control deficiencies. However, because of competing priorities, USMS does not use these mechanisms to fully monitor key internal controls over employee misconduct processes. According to Office of Professional Responsibility Compliance Review officials, the scope of on-site management reviews conducted at selected USMS districts and divisions during fiscal years 2014 through 2018 did not include employee misconduct processes. The officials also stated that on-site reviews during this period did not include the Internal Affairs and Discipline Management divisions. According to USMS officials, the agency plans to conduct an on-site management review at the Internal Affairs division in fiscal year 2021. The officials added that the compliance review cycle for each district and division currently occurs once every 9 years, but that this review cycle will increase to once every 4 years. Our analysis of USMS annual self-assessment guides showed that from fiscal years 2014 through 2018, the guides included testing for most key controls related to employee misconduct processes. For example, Internal Affairs and Discipline Management self-assessment guides included questions on whether Internal Affairs forwards cases to the DOJ OIG for right of first refusal, the Chief of Internal Affairs reviews investigative reports, investigations are completed within 90 days, and data on allegations is entered into the case management system. The self-assessment guide for USMS districts and divisions included questions to assess compliance with the timeliness of investigations (within 30 days); use of the Table of Offenses and Penalties, consideration of Douglas Factors (certain factors that USMS is to consider about an employee when deciding discipline, such as the employee’s need for training); Delegations of Authority for proposing and deciding officials, and other Human Resource policy areas, such as administrative leave and eligibility for promotion. However, although legal sufficiency review of proposed adverse actions is required by policy and a key internal control, USMS did not design its self- assessment guides for the Internal Affairs and Discipline Management divisions to include testing for such reviews. Revising the scope of on-site management reviews to include employee misconduct processes and revising self-assessment guides to include testing for legal sufficiency of proposed adverse actions would help USMS gain greater assurance that these controls are implemented as required by policy. ATF and USMS have established internal controls related to some employee misconduct investigation and disciplinary processes, but additional actions could strengthen their controls. Specifically, USMS does not ensure that supervisory review of division and district investigations is documented in accordance with agency policy. ATF and USMS also have not developed policy for verifying the accuracy and completeness of information in employee misconduct systems. Ensuring supervisory review is documented as required and establishing policy for verifying information in misconduct systems would provide greater consistency in processes, assurance that controls are operating as intended, and corrective actions are implemented as needed. ATF and USMS policy also have required timelines for completing investigations. However, ATF does not have a performance measure to monitor whether it is meeting its timeliness requirement, such as the percentage of Internal Affairs investigations completed within 120 days. USMS does not have a performance measure to monitor and assess its performance in meeting the required time to complete its district and division investigations within 30 days. Developing performance measures to monitor the timeliness of all investigations could provide more complete information for ATF and USMS management responsible for oversight and allow them to address any related performance issues in a timely manner. Further, ATF and USMS have established oversight mechanisms, such as internal management reviews, to monitor select aspects of the components’ operations, such as financial operations. However, ATF and USMS generally have not used these mechanisms to monitor internal controls related to employee misconduct processes, which would help ATF and USMS management ensure that controls are implemented as required by policy. We are making a total of seven recommendations, including three to ATF and four to USMS. Specifically: The Director of the U.S. Marshals Service should take steps to ensure that supervisory review of division and district investigations is documented in accordance with USMS policy. (Recommendation 1) The Director of ATF should develop policy for verifying the accuracy and completeness of information in ATF employee misconduct systems. (Recommendation 2) The Director of the U.S. Marshals Service should develop policy for verifying the accuracy and completeness of information in USMS employee misconduct systems. (Recommendation 3) The Director of ATF should develop a performance measure to monitor the timeliness of misconduct investigations, according to policy requirements. (Recommendation 4) The Director of the U.S. Marshals Service should develop a performance measure to monitor the timeliness of district and division misconduct investigations, according to policy requirements. (Recommendation 5) The Director of ATF should modify existing oversight mechanisms to include the monitoring of key internal controls related to employee misconduct investigations. (Recommendation 6) The Director of the U.S. Marshals Service should modify existing oversight mechanisms to fully monitor key internal controls related to employee misconduct investigations. (Recommendation 7) We provided a draft of this product to DOJ for review and comment. DOJ concurred with all of our recommendations and did not provide written comments. ATF and USMS provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Attorney General, the ATF Acting Director, the USMS Director, appropriate congressional committees, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or McNeilT@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. To assess the extent to which the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) and United States Marshals Service (USMS) components implemented key internal controls, we selected a stratified random sample of case files within the population of employee misconduct investigations that were opened by each component from fiscal years 2014 through 2018, and that were considered closed as by USMS as of March 13, 2019, for fiscal years 2014 through 2017 and April 26, 2019, for fiscal year 2018, with corresponding data on the outcomes of the investigations (resulting employee actions) as of March 27, 2019 for fiscal years 2014 through 2017 and May 3, 2019, for fiscal year 2018. ATF data are as of April 9, 2019, for internal investigations and as of August 2, 2019, for management referrals. We also stratified our samples based on whether the case files included adverse actions (a suspension of at least 15 days, demotion or removal) and whether an employee under a misconduct investigation had also filed an Equal Employment Opportunity (EEO) claim of management retaliation to assure that representation from both subgroups were included in our sample. We used fiscal year 2014 through 2018 data from the components’ information systems from which to randomly select a generalizable sample of 65 employee misconduct cases for ATF out of a population of 150 and 100 cases for USMS out of a population of 1,281. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval. This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. The sample was designed to produce 95 percent confidence intervals for percentage estimates that are within no more than plus or minus 10 percentage points within component. The precision is not high enough to generalize to the strata level and results should only be generalized to the component level (i.e. ATF and USMS). As part of these samples, we included investigation that resulted in proposed adverse actions and that involved employees who also submitted an EEO claim of management retaliation. Specifically: For ATF, our sample included 12 cases with proposed adverse actions and nine cases that involved individuals who had also submitted an EEO claim of management retaliation. For USMS, our sample included 12 cases with proposed adverse actions and 12 cases that involved individuals who had also submitted an EEO claim of management retaliation. Because some items we assessed applied only to a subset of cases, resulting in a smaller sample size, we report some findings as the range from the lower to upper bound of the 95 percent confidence interval. In cases with particularly small sample sizes, we describe results for the sample only, rather than attempting to generalize to the population of cases within the component. Triana McNeil at (202) 512-8777 or McNeilT@gao.gov In addition to the contact named above Eric Erdman (Assistant Director), Willie (Billy) Commons III, Dominick Dale, Anthony DeFrank, Justin Fisher, Eric Hauswirth, Ying Long, Amanda Miller, and Mike Tropauer made key contributions to this report.", "summary": "Within the Department of Justice, ATF and USMS employ more than 10,000 staff responsible for protecting communities from violent criminals, investigating the illegal use of firearms, and apprehending wanted persons, among other things. Our recent studies of employee misconduct processes have highlighted the importance of internal controls to help ensure the quality and independence of these processes. We have also reported on employee misconduct investigations being used to retaliate against individuals who report wrongdoing. GAO was asked to review ATF and USMS employee misconduct investigation and disciplinary processes. This report (1) summarizes data on the number, characteristics, and outcomes of ATF and USMS misconduct investigations that were opened from fiscal years 2014 through 2018 and were closed by the time of GAO's review, and (2) examines the extent to which ATF and USMS have developed, implemented, and monitored internal controls for their employee misconduct processes. For each component, GAO reviewed policies, guidance, and performance reports; analyzed case management system data; analyzed random samples of misconduct cases; and interviewed officials involved in investigation and discipline processes. From fiscal years 2014 through 2018, the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) and U.S. Marshals Service (USMS) collectively investigated about 3,900 allegations of employee misconduct, as shown in the table below. About one-half of these investigations were closed with no disciplinary action because the components found that the allegations were unsubstantiated. For allegations that were substantiated by an investigation, the most common ATF offenses were poor judgment and failure to adequately secure property, while the most common USMS offenses were general violations of policy or procedure and failure to follow instruction. The most common outcomes for both ATF and USMS substantiated investigations were discipline including suspensions of up to 14 days and lesser penalties such as verbal or written warnings. During this period, ATF and USMS investigated over 300 allegations of management retaliation, with few resulting in discipline. ATF and USMS have developed some internal controls for managing their employee misconduct investigation and disciplinary processes, but have not consistently documented or monitored key control activities. For example: USMS policy requires supervisory review of district and division investigations, but the agency has not consistently documented this control in accordance with policy. ATF and USMS also lack policy for verifing the accuracy and completeness of information in employee misconduct systems. Ensuring supervisory review is documented as required and developing policy for verifying information in misconduct systems would provide greater assurance that controls are operating as intended. ATF and USMS have established policies and goals related to timeliness in completing various types of employee misconduct investigations (e.g., within 120 days). However, ATF has not established performance measures to monitor progress toward meeting the goals. USMS has measures to monitor timeliness for some types of investigations, but not for others. Establishing measures to monitor timeliness of investigations would provide more complete information to ATF and USMS managers responsible for oversight. ATF and USMS have established oversight mechanisms, such as internal management reviews, to monitor certain aspects of the components' operations, such as financial operations. However, ATF and USMS have not fully used these mechanisms to monitor internal controls related to employee misconduct processes, which would help ATF and USMS management ensure that controls are implemented as required by policy. GAO is making seven recommendations, including that USMS ensure supervisory review is documented; and that ATF and USMS develop policy for verifying system information, establish measures to monitor the timeliness of investigations, and improve monitoring of employee misconduct processes. DOJ concurred with our recommendations.", "document_type": "gao"}
{"report": "TSA is responsible for implementing and overseeing security operations at roughly 440 commercial airports as part of its mission to protect the nation’s civil aviation system. TSA is responsible for ensuring that all passengers, their carry-on bags, and their checked baggage are screened to detect and deter the smuggling of prohibited items, such as explosives, into the sterile areas of airports and onto aircraft. Agency procedures generally provide that passengers pass through security checkpoints where their person, identification documents, and carry-on bags are screened by transportation security officers (TSO). TSA uses a variety of screening technologies—screening systems, as well as software and hardware for those systems—to carry out its mission. Figure 1 depicts the various screening technologies a passenger may encounter in primary and secondary security screening. TSA develops detection standards that identify and describe the prohibited items—such as guns, knives, military explosives, and homemade explosives—that each technology is to detect during the screening process. The standards, which are classified, also identify how often the technology should detect prohibited items (referred to as the required probability of detection) and the maximum rate at which the technology incorrectly identifies prohibited items (the probability of false alarm). For explosive materials, the standards also identify what the screening technology is to be able to detect in terms of (1) the minimum amount or weight of the material (the minimum detection mass) and (2) the chemical and physical makeup of the material (density range of the explosive material). S&T supports TSA in the development of standards by, among other things, analyzing the characteristics (threat mass, or the amount of material that constitutes a threat, and density) of explosive materials. The agency uses the resulting data to develop detection standards that are specific to each screening technology. After a detection standard is approved, TSA decides whether to operationalize—put into effect—detection standards by acquiring and deploying technologies to update detection capabilities to meet the standard. That is, it decides whether to take steps to develop new technology capable of meeting the standard and put the new technology in place at commercial airports. Technology can mean new software to upgrade existing screening systems as well as entirely new screening systems. TSA does not always or immediately operationalize detection standards, for reasons which are explained later in this report. To operationalize a detection standard, TSA must acquire technology capable of meeting the standard. TSA officials told us they follow DHS acquisition policies and procedures when acquiring new screening technologies. Officials said they adapt detection standards as detection requirements to guide the acquisition process, meaning the specifications described in the standards are incorporated into the requirements manufacturers must meet when developing new technology. Once manufacturers have developed new technologies that meet detection requirements, the technologies undergo a test and evaluation process, known as the qualification process. The following are key steps in that process: 1. Certification – Certification is a preliminary step in TSA’s qualification process. For TSA to certify that a screening technology meets its detection requirements, S&T’s Transportation Security Laboratory conducts certification testing on a manufacturer’s initial submission of its proposed screening technology to determine whether it meets TSA’s detection requirements (i.e., the rate at which it must accurately detect each category of explosive it is designed to detect, among other things). 2. Integration/Implementation Testing – TSA Systems Integration Facility administers qualification testing to test system performance against additional requirements, such as reliability, availability, and maintainability. TSA also conducts field testing to ensure readiness for operational test and evaluation. 3. Operational Test and Evaluation - TSA deploys units to selected airports to conduct operational testing. Operational testing allows TSA to evaluate the operational effectiveness, suitability, and cyber resiliency of the technology in a realistic environment. After new technologies have been tested and approved, TSA can purchase and deploy them to commercial airports. When a deployed screening system can no longer be updated to meet new detection standards, TSA considers it obsolete and generally designates it for replacement with a newer version of the technology. Figure 2 shows TSA’s process for acquiring and deploying new screening technologies to meet detection standards. DHS guidance provides that its components, including TSA, use risk information about security threats and analysis to inform decision-making. Risk management helps decision makers identify and evaluate potential risks so that actions can be taken to mitigate them. DHS defines a risk assessment as a function of threat, vulnerability, and consequence. DHS guidance also says that risk assessments and transparency are key elements of effective homeland security risk management. TSA has a process to develop new explosives detection standard in response to emerging, credible threats involving a homemade explosive (see sidebar for more information on homemade explosives). According to TSA officials, the first step in the process is to determine whether a new detection standard is needed, which they do by working with S&T and other federal partners to ”characterize” the threat material—that is, identify the chemical and physical properties of the material, such as the threat mass and density. Below is the process (steps) TSA and S&T officials told us they use to characterize a threat material and determine whether a new detection standard is needed. Homemade Explosives Homemade explosives are designed to cause destruction when used in improvised explosive devices. The picture below shows damage to an aircraft panel from a homemade explosive. Beginning in the early 2000s, homemade explosives replaced military and conventional explosives as the preferred tool of terrorists, and challenged the capabilities of existing screening technologies. Unlike conventional threats, homemade explosives are often made of common commercial items and it can be challenging to distinguish them from innocuous gels and liquids stored in personal baggage or cargo. They also have different detonation patterns from conventional explosives in that they often release energy much more slowly, which may lead to incomplete or delayed detonation. This pattern is not well understood, which makes it much more difficult to predict the resulting damage. of the explosive—the minimum amount of the material that constitutes a threat to civil aviation. Material down selection (selection of possible mixtures for testing). Because the exact formulation of the explosive can vary, S&T must test and model various formulations in different proportions to gain an understanding of the homemade explosive. In this step, TSA determines the representative formulations and preparations that are to be prepared and tested, based on data provided by S&T. Synthesis, formulation, and preparation of materials. S&T establishes how the threat material could be made, including its chemical synthesis (as applicable), possible formulations or mixtures of the material with other components, and the preparation of those mixtures. S&T uses this information to develop samples of the threat material for testing. Data acquisition and analysis. S&T examines the samples using micro- computed tomography and explosives detection system, and the resulting data are sent to S&T’s Transportation Security Laboratory for verification. The verified data are then sent to the U.S. Department of Energy’s Lawrence Livermore National Laboratory for analysis. The Transportation Security Administration and the Science and Technology Directorate have ranked 300 conventional and homemade explosives that pose the most likely threat to aviation security based on factors such as availability, stability, performance, and method of initiation. Region of responsibility. Lawrence Livermore National Laboratory generates preliminary results in the form of the “region of responsibility,” which is a map or explosive detection “window” outlining the characteristics of the threat material in terms of density and effective atomic number. These preliminary results are discussed among TSA and S&T stakeholders, with TSA determining the final region of responsibility. The region of responsibility data are used to develop software algorithms that will allow screening technologies to recognize explosive materials whose characteristics fall within the region of responsibility. Detection standard. TSA and S&T also use the region of responsibility data to determine whether the explosive material can already be detected by deployed screening technologies. If screening technologies can already detect the material, TSA will not contract with technology manufacturers to develop a new software algorithm or screening technology. But regardless of whether a new software algorithm or new technology is needed, TSA will draft a new detection standard for the material that, generally, will specify the minimum threat mass and density range to be detected, the acceptable probability of detection, and probability of false alarm. The draft standard is reviewed by TSA senior management before being approved. We found that the work S&T and other stakeholders performed to characterize explosive threat materials was consistent across the threat materials. Specifically, we found that S&T consistently followed the process described to us (as outlined above) for characterizing a threat material in the seven material threat assessments we reviewed. We also reviewed documentation regarding additional testing and analysis S&T performed on select threat materials, and found the additional testing and analyses were performed consistently. TSA has not updated its 2015 guidance for developing new detection standards to reflect key changes in their procedures. In December 2015, TSA issued the Detection Requirements Update Standard Operating Procedure, which a senior official told us served as the agency’s approved guidance for developing detection standards. Our review of the document found that, as of August 2019, it did not accurately reflect (1) designated procedures for developing detection standards, (2) the roles and responsibilities of key stakeholders such as S&T, and (3) TSA’s organizational structure. For example, one way in which the 2015 guidance has not been updated is in the designated procedures it describes for reviewing available intelligence information. Specifically, the guidance calls for an annual assessment of emerging threats, which a senior TSA official told us TSA no longer conducts because relevant emerging threats are now occurring more frequently and intelligence information is processed on an ongoing basis. In another example, the guidance specifies that TSA will form working groups composed of agency officials and stakeholders to assess potential threat materials and develop an analysis plan, and that each working group will define the roles and responsibilities of its members. According to a senior TSA official, the agency does not convene working groups to assess intelligence or develop an analysis plan, although officials regularly meet with stakeholders to discuss the steps needed to characterize new threat materials and document the minutes from these meetings. Finally, while the guidance discusses in detail which TSA offices and management positions are responsible for implementing and overseeing the process, the agency has since reorganized and these offices and positions no longer exist. Therefore, the 2015 guidance is no longer relevant in terms of which offices and positions are responsible for implementing and overseeing the approval of detection standards. Officials told us that, as of August 2019, they had begun revising the guidance to reflect existing standard operating procedures for developing detection standards, but had yet to finalize a draft of the new guidance or document plans or timeframes for completing and approving it. Further, it is not clear to what extent the revised guidance will address designated procedures for developing detection standards, the key roles and responsibilities of stakeholders, and TSA’s new organizational structure. Officials said they had not updated the guidance earlier because both TSA and S&T had been undergoing agency reorganizations. Standards for Internal Control in the Federal Government provides that agencies should identify, on a timely basis, significant changes to internal conditions that have already occurred, such as changes in programs or activities, oversight structure, and organizational structure. Additionally, agencies are to develop and maintain documentation of internal controls, such as policies and procedures necessary to achieve objectives and address related risks. By documenting the processes and procedures TSA uses to develop detection standards, clarifying the roles and responsibilities of stakeholders, and documenting organizational changes, TSA could have better assurance that detection standards are developed in accordance with established policies and practices. Our review of TSA’s steps to develop detection standards from fiscal years 2014 through 2018 found that TSA and S&T did not document all key decisions—those that could potentially affect outcomes—regarding the testing and analyses (characterization) of explosive threat materials and the development of explosives detection standards. We found that TSA and S&T produced a series of detailed material threat assessments to document the characterization of threat materials and consistently developed action memos to justify proposed detection standards. However, we also found that in five of the seven material threat assessments we reviewed TSA and S&T did not consistently document key steps in the testing and analyses of materials, such as how selected samples were prepared for testing. For example, one S&T material threat assessment we reviewed did not document the method used to synthesize (chemically produce) material samples used for testing. Not documenting the method could prevent officials from fully understanding the results of the analysis. Specifically, the assessment noted that there are multiple methods of synthesis, and that the chosen method could affect the makeup of the resulting material and therefore the ability of the screening technologies to detect it. Additionally, while two of the seven material threat assessments cited standard operating procedures for sample preparation for all participating laboratories, three did not cite standard operating procedures for at least one laboratory and two stated that sample preparation information had not been provided by one or more of the participating laboratories. Without documentation, TSA might not have all the necessary information to address future issues involving detection of these materials. We also found four instances in which TSA did not clearly document why select materials were sent for additional testing or did not document key decisions regarding the development and consideration of detection standards. For example, S&T performed additional testing and analysis on select threat materials after the material threat assessment was finalized. However, the documentation of this additional testing left out key elements regarding how and why the additional testing was needed and conducted. The action memo documenting new standards based on the results of the additional testing did not include a justification for why specific threat materials were selected for additional data collection. While a test plan for equivalency testing of one material stated that the additional testing was conducted because data reported in the literature were not considered representative of current threat configurations, similar justification was not included in the action memo justifying changes to the new standard based on the additional testing. Finally, a senior TSA official told us he requested the additional equivalency testing because the values in the previous detection standards appeared to be more conservative than expected and there was no documentation explaining how TSA had arrived at those numbers. According to the official, the previous detection standard was approved before his tenure and the determining officials were no longer with TSA. He also stated that he did not know whether TSA required documentation of testing and analysis when the previous detection standard was being developed. We found that TSA did not document key decisions regarding the development and consideration of detection standards. For example, officials could not provide documentation of conclusions reached on specific key decisions, such as the consideration and decision not to approve a proposed explosives trace detection standard. A senior TSA official said he did not know why the decision had not been documented because the officials involved were no longer with the agency. According to Standards for Internal Control in the Federal Government, documentation is required for the effective design, implementation, and operating effectiveness of an agency. Documentation also provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel, as well as a means to communicate that knowledge as needed to external parties. By documenting key decisions regarding the development of detection standards, including instances in which draft standards are not approved, TSA could better ensure that effective decisions are made and that organizational knowledge is retained regardless of changes in personnel. TSA officials said one way to operationalize detection standards—acquire and deploy technologies to update detection capabilities and meet the detection standard—is to update existing screening systems with new technology, such as software or firmware. When possible, the agency installs software as part of routine maintenance. TSA can also deploy new hardware or replace screening systems to update detection capabilities. According to officials, the agency applies an incremental approach to updating existing screening technologies—it updates technologies when manufacturers are able to develop the requisite capabilities and as resources allow—which can take years to complete. According to officials, all fully deployed TSA screening technologies had detection capabilities that met detection standards approved from 2006 through 2012. That is, as of August 2019, TSA’s fleet of screening technologies met detection standards that were approved in 2012 or earlier. For example: Bottled liquid scanner units met a detection standard that was Advanced technology x-ray units met two detection standards, depending on their manufacturer, that were both approved in 2010; and Explosives trace detection units met a detection standard that was approved in 2012. Further, for each screening technology, the agency has approved two to three new detection standards that have not been operationalized, as of August 2019. For example, in addition to the 2006 detection standard for bottled liquid scanner, TSA approved standards for bottled liquid scanner in 2012 and in 2017 that have not been operationalized. TSA officials said they were working to operationalize some of the detection standards approved since 2012. Officials said they were working with manufacturers to develop new technologies to operationalize some of these standards. In other cases they were in the process of deploying new technologies that meet these standards. For example, as of August 2019, TSA was in the process of updating and replacing explosives detection systems to meet a detection standard that was approved in 2014. Officials said they expected to have the entire fleet updated by September 2023. TSA officials said they were also in the process of updating deployed advanced technology x-ray units for one of its two manufacturers to meet a standard that was approved in 2014. For more information about the detection standards TSA had approved for each technology as of August 2019, and the status of TSA’s progress in operationalizing them, see appendix I. TSA shares information about the capabilities it needs with manufacturers through requests for proposal, requests for information, and broad agency announcements. The agency places approved technologies on a qualified products list—a list of technologies that have been tested and certified as meeting requirements by TSA and DHS—and the agency can then award a contract to one of the manufacturers to purchase and deploy the technology. Before deploying technologies to airports, TSA conducts testing to ensure consistency in the manufacturing process, system configuration, and functionality following production, and then again after the technology is installed at airports. Our analysis of the acquisition information TSA provided found it took from 2 to 7 years to fully develop, certify, test, and deploy screening technologies to airports. For example, when operationalizing explosives trace detection standard 5.0, it took one manufacturer 4 years and a second manufacturer 7 years to develop, and for TSA to deploy, the software needed to update the capability of existing explosives trace detection units to meet the new standard. Figure 3 provides our analysis of TSA’s timeline for operationalizing advanced imaging technology detection standards approved from 2010 through 2016. TSA officials said they approved detection standard 3.3 for advanced imaging technology in October 2010 and began deploying technology that met that standard to airports in August 2011. Officials said they approved a subsequent standard, 4.1, in January 2012, began deploying technology to meet it in October 2014, and completed the deployment in September 2017. Officials said it took 3 years to complete deployment because the demand for advanced imaging technology increased over time as airports experienced an increase in passenger volumes, among other reasons. Since 2012, TSA approved two additional detection standards for advanced imaging technology—4.3 in February 2016 and 4.3.1 in August 2016. TSA officials said they have not operationalized these two standards because the manufacturer has not been able to develop the requisite technology. As such, deployed advanced imaging technology units meet standards approved in 2010 and 2012. TSA officials stated that they do not always, or immediately, operationalize detection standards after they are approved. They said they make these decisions on a case-by-case basis, depending on many factors. These include whether: (1) manufacturers have the technological ability, (2) a new technology is in development, and (3) screening technologies already have the capability. Manufacturers do not have the technological ability. TSA officials said manufacturers do not always have the technical ability to meet detection standards. According to officials, it can be challenging for manufacturers to develop the technology necessary to detect new threats as presented in a detection standard, and in some cases impossible without further research and development. For example, TSA officials said that manufacturers have been unable to develop the requisite technology to meet the most recent detection standards (4.3 and 4.3.1) for advanced imaging technology. However, TSA officials said they have expanded their research and development efforts to try to develop the technology. TSA officials told us they plan to continue developing detection standards irrespective of the capabilities of currently deployed technologies so that they can focus on identifying emerging threats. The new detection standards then serve to set expectations for manufacturers about the capability to which they should aspire and justify research and development necessary to realize that capability. To better manage the difference between the capabilities of deployed technologies and the capabilities described in detection standards, TSA officials said they are in the process of developing a new position of Capability Manager, who would be responsible for managing the development of mission-essential capabilities—such as carry-on baggage screening—from start to finish. Officials said they expect this position will help bridge the gap between approved detection standards and the detection capabilities of deployed screening technologies over time, because the managers will have cross- cutting oversight of the process. A new technology is in development. Officials said that they may not operationalize a detection standard if they expect a new type of screening technology will replace an existing one. For example, officials said that TSA is exploring new alarm resolution technologies—that is, screening technologies that are used to determine whether alarms are false positives. Officials said new alarm resolution technologies may replace the bottled liquid scanner in the future, and therefore they have not pursued operationalizing detection standard 2.3. Screening technologies already have the capability. According to TSA officials, new detection standards do not always add significant detection capabilities. For example, officials decided not to operationalize bottled liquid scanner detection standard 3.0 when it was approved in 2017 because the deployed units already had most of the capabilities called for in the detection standard; TSA developed the new standard to better align with standards for other technologies. Our review of TSA acquisition documents found that TSA considers risk at the beginning of the screening technologies acquisition process.. Specifically, the agency considers risk in two phases—(1) a risk assessment developed from intelligence information and modeling tools, and (2) an annual capability analysis that analyzes and prioritizes capability gaps and determines mitigation options. Figure 4 provides an overview of TSA’s acquisition process for new screening technologies. Risk assessment. TSA uses intelligence information and modeling tools, such as the Risk and Trade Space Portfolio Analysis, to assess risk to the aviation system. The Risk and Trade Space Portfolio Analysis was developed in 2014 to analyze the security effectiveness of alternate combinations of some aviation security countermeasures. Officials said a recent example of a risk-informed deployment decision influenced by the Risk and Trade Space Portfolio Analysis was TSA’s 2017 deployment of 141 advanced imaging technology units to category III and IV airports. Officials said that around 2014, TSA received intelligence about a potential terrorist threat to airports, as well as the results of covert testing at airports that identified screening vulnerabilities. Officials said a 2014 Risk and Trade Space Portfolio Analysis also identified disparities in screening capabilities at smaller airports. In part because of the vulnerability identified by these three factors, as well as ongoing conversations between TSA senior leadership, the DHS Inspector General, and members of Congress, officials said TSA procured and deployed additional advanced imaging technology units to some category III and IV airports that did not have them. Capability analysis. TSA uses the Transportation Security Capability Analysis Process, a structured decision-making tool, to identify and prioritize capability gaps and help direct agency resources towards closing critical gaps to an acceptable level. When existing screening capabilities do not fully meet TSA’s mission needs, the associated capability gap presents a security risk. As part of the Transportation Security Capability Analysis Process, TSA produces Capability Analysis Reports that identify and recommend solutions to closing capability gaps. Recommendations have included procedural changes, such as new training for TSOs, and investments in new technology. TSA’s investment in computed tomography technology for checkpoint screening of carry-on baggage is an example of TSA’s implementation of the Transportation Security Capability Analysis Process to validate capability gaps and identify recommended courses of action. Officials said that in some cases the agency may identify a capability gap that cannot be resolved to an acceptable level with commercially available screening technology, in which case it will pursue additional research and development. TSA officials told us that they operate under the assumption that every airport is a possible entry point into the aviation system for a terrorist, and they do not consider there to be a significant difference in vulnerability among airports when deploying screening technologies. However, officials did not provide analysis or documentation that supported this conclusion. Officials noted the exception to this assumption is a handful of airports that are consistently considered to be the highest risk because of known threats and a high volume of international travelers. Further, officials said that if they had information about a threat to a specific airport that would be mitigated by deploying a screening technology, they would modify their plans for deployment accordingly. However, TSA’s process for how it would change its deployment plans to specific airports based on risk lacks transparency. For example, officials said that as part of the acquisition process they have ongoing discussions with stakeholders about their deployment strategies, including security and intelligence officials who would inform them of any relevant risk information. Officials said these discussions are generally informal and not documented—it was unclear how these discussions have incorporated information about risk in the past, and officials could not provide an example of when risk information at specific airports had directly influenced deployment of technologies to airports in the recent past. In 2018, the agency released its Transportation Security Administration Systems Acquisition Manual, which called for deployment plans to be written documents, and officials said they began documenting their plans for deploying screening technologies in the last two years. TSA officials provided us with one deployment plan—for their 2018 deployment of explosives trace detection units—but we found that it was not transparent about how risk was a factor in officials’ methodology for determining the order of airports to receive the technology. The explosives trace detection plan documented TSA’s schedule of deployment and the roles and responsibilities of relevant stakeholders, among other things. However, while the plan indicated that officials would coordinate with relevant offices within the agency for information about risks that might impact their deployment strategy, we found that the plan did not document how risk had informed their decisions about where and how to deploy the technology, including the assumptions, methodology, and uncertainties considered. Additionally, TSA officials did not document, and could not fully explain, how risk analyses contributed to and factored into the following specific deployment decisions. Deployment of advanced imaging technology to smaller airports. Officials said many factors influenced their decision to deploy advanced imaging technology units to category III and IV airports, including information about threats and a related 2014 risk analysis. However, officials did not document their decisions and could not fully explain their risk analysis, including their process for analyzing and weighing relevant factors. According to officials, the decision was made during discussions with senior leadership, which were risk-informed and supported by whiteboard analyses and classified documents. Additionally, officials told us that, for practical reasons, they deployed units to those category III and IV airports that had the space to accommodate them, but did not further assess the priority of deployment among the smaller airports because they had determined that the risk was uniform and because they planned to deploy the units within a short timeframe. Officials did not document the risk assessment that led to this determination, and could not explain how the three elements of risk—threat, vulnerability, and consequence—were used or assessed. Deployment of targeted threat algorithm. In 2016, TSA deployed a targeted threat algorithm—software to improve detection capabilities—to a limited number of advanced imaging technology units in response to a specific threat. After testing the operational impacts of the software algorithm, the agency decided to stop further deployment. The documentation TSA provided did not explain how officials had analyzed the risk-mitigation benefits of the algorithm, including the underlying assumptions and uncertainty, or how they had weighed those benefits against the operational impacts and costs when they made their decision not to fully deploy the algorithm. TSA officials said they follow the DHS acquisition process to acquire and deploy technologies and their deployment decisions are based on, and informed by, their initial assessments of capability gaps, as well as their understanding that every airport offers equal entry into the aviation system. However, officials had not documented the rationale for these decisions and could not fully explain how risk had informed their decisions about where and in what order to deploy screening technologies. DHS’s Risk Management Fundamentals states that components should consistently and comprehensively incorporate risk management into all aspects of the planning and execution of their organizational missions. Additionally, it says transparency is vitally important in homeland security risk management, and documentation should include transparent disclosure of the rationale behind decisions, including the assumptions, methodology, and the uncertainty considered. By fully disclosing what risk factors are weighed and how decisions are made, TSA officials can better ensure that their deployment of screening technologies matches potential risks (threats, vulnerabilities, and consequences). This is of particular importance given the agency’s limited resources and the fact that screening technologies are not easily relocated. TSA officials said that absent a specific risk to an airport or category of airports that would be mitigated by deploying a screening technology, they consider a number of logistical factors that are aimed at maximizing the operational efficiency of the screening process. These factors influence the number of units of a technology the agency deploys to airports, the order in which they deploy them, and where they are deployed. Officials said they use modeling software to determine the most efficient number of units to allocate to an airport for each type of screening system. This analysis takes into account variables such as the number of flights at an airport, airport passenger volumes, items per passenger, and secondary search rates. Additionally, agency officials said the layout of an airport is a significant determining factor for the number of units it receives. For example, an airport that has centralized checked baggage screening areas will need fewer explosives detection systems than an airport that has checked baggage screening areas dispersed in different locations. Additionally, TSA officials said that logistical and funding factors can influence the order of deployment, including the manufacturer’s ability and resources to develop and deliver technologies. For example, as of June 2019, officials said the agency was in the process of updating the detection capabilities of 62 percent of its advanced technology x-ray fleet because one of its two manufacturers had completed testing and certification of the new technology, but the second manufacturer’s technology had yet to be certified. Officials said they also try to plan their deployment schedule around minimizing disruptions to airport operations, so if an airport could not absorb a full deployment of a technology because it would affect too many passengers, TSA would schedule the deployment in phases to minimize disruptions. Further, TSA officials said that, as a result of these many logistical considerations, they generally fully deploy new screening technologies to category X airports first—generally, airports with the highest passenger volumes—and then proceed in order down to the airport with the lowest passenger volume. Officials said larger airports generally have the infrastructure in place to incorporate new technology without extensive disruption to operations, and they will screen the most passengers by deploying screening technologies to the largest airports first. TSA practices do not ensure that screening technologies continue to meet detection requirements after they have been deployed to airports. According to agency officials, the agency uses certification to confirm that technologies meet detection requirements before they are deployed to airports, and calibration to confirm that technologies are at least minimally operational while in use at airports. Officials stated these processes are sufficient to assure TSA that screening technologies are operating as intended. However, while certification and calibration serve important purposes in the acquisition and operation of screening technologies, they have not ensured that TSA screening technologies continue to meet detection requirements after they have been deployed. Certification occurs prior to deployment. TSA’s certification process is designed to ensure screening technologies meet detection requirements during the acquisition process, prior to the procurement and deployment of the technologies, but it does not ensure screening technologies continue to meet detection requirements after deployment. As previously described, manufacturers provide an initial submission of the screening technology to TSA for certification testing as part of the acquisition process. During the certification process, S&T’s Transportation Security Laboratory tests the technology under controlled conditions to determine whether it meets TSA’s detection requirements. After TSA certifies that a screening technology meets detection requirements and it undergoes additional testing to determine whether it meets other TSA requirements in controlled testing facilities, TSA may deploy it to select airports for operational testing and evaluation to determine how it performs in an airport setting. Certification testing demonstrates that a manufacturer’s screening technology meets detection requirements during the acquisition process, which allows TSA to determine whether it should continue to consider the technology for acquisition. Certification does not ensure that deployed technologies continue to meet detection requirements because it does not account for the possibility that performance of technologies can degrade over time throughout the technologies’ lifecycles after deployment. For example, in 2015 and 2016, DHS removed a sample of deployed explosives trace detection and bottled liquid scanner units from airports for testing in the Transportation Security Laboratory. The laboratory concluded that some deployed units for each technology it tested no longer met detection requirements— either the required probability of detection for certain explosives or the required rate for false alarm, or both. One explosives trace detection unit that was tested was found to have a probability of detection much lower than required. According to TSA officials, the units did not meet detection requirements because they were not adequately maintained, which affected their performance. In light of this, officials stated that they introduced better controls to ensure that routine preventative maintenance is performed as required. However, because TSA does not test the units after they are deployed to airports, it cannot determine the extent to which these controls ensure technologies continue to meet detection requirements. Officials noted that TSA uses a layered security approach at airports, so if one layer should fail—such as a deployed technology—the agency can still rely on other security measures among the various layers of security to detect threats. We have previously reported on the importance that TSA ensure each measure is effective to make the best use of its limited resources, in order to serve its aviation security mission. Calibration does not test whether technologies meet detection requirements. TSA officials stated that daily calibration also helps ensure that screening technologies continue to meet detection requirements after deployment. However, while calibration demonstrates that the screening technology is at least minimally operational, it is not designed to test whether the screening technology meets detection requirements. For example, each explosives detection system is calibrated with an operational test kit that contains items of various densities. To calibrate explosives detection systems, a TSO must run the operational test kit through the unit and verify that the item is correctly displayed on the monitor (see figure 5 below). This process demonstrates whether the system can identify the known items’ densities, but it does not ensure that the system meets detection requirements. As a result, calibration could indicate that the unit is functioning even when its detection capabilities have degraded—that is, calibration determines that the technology is functional, but it does not ensure that the technology is meeting detection requirements. TSA officials stated that they plan to develop a process to review screening technologies on an annual basis to analyze their performance, including detection over time. TSA officials stated that, as of August 2019, they were actively working on developing a review process for the explosives detection system but did not have a date for when they planned to complete it. TSA officials for the passenger and carry-on screening technologies stated that they had not yet started developing a review process for those technologies and the timeline for developing a review process will depend on funding. TSA officials also noted that there are challenges in designing a process to ensure that screening technologies continue to meet detection requirements after deployment. For example, TSA and S&T officials stated that it is not feasible to conduct live explosives testing in airports. Further, according to TSA officials, while it is relatively easy to temporarily transfer smaller screening technologies, such as explosives trace detection and bottled liquid scanner units, to a controlled setting for live explosives testing, it would not be feasible to transfer larger installed units, such as advanced imaging technology. Although testing with live explosives in an airport poses undue risks and transferring larger machines for testing may be costly, TSA could develop other measures. TSA officials stated that there is no requirement to ensure that its screening technologies continue to meet detection requirements after deployment to airports. However, Standards for Internal Control in the Federal Government calls for agencies to establish and operate a system to continuously monitor the quality of performance over time. Without taking additional steps to ensure screening technologies are meeting detection requirements, TSA may run the risk that its deployed screening technologies are not detecting explosives and other prohibited items. Developing and implementing a process to monitor screening technologies’ detection performance over time would help provide TSA assurance that screening technologies continue to meet detection requirements, as appropriate, after deployment. In doing so, TSA would also be better positioned to take any necessary corrective actions if or when screening technologies no longer operate as required. We estimate that TSA spent $3.1 billion to purchase, deploy, install, and maintain its inventory of screening technologies as of the end of fiscal year 2018, based on agency estimates of costs. Of this $3.1 billion, we estimate that TSA spent 71 percent to purchase screening technologies, 9 percent to deploy, about 12 percent to install, and, for fiscal year 2018, about 9 percent to maintain them for 1 year. The highest estimated total expenditures on a per-technology basis were for explosives detection systems ($2.1 billion, or 68 percent), advanced technology x-ray ($443 million, or 14 percent), explosives trace detection ($227 million, or 7 percent), and advanced imaging technology ($197 million, or 6 percent). Table 1 provides information on estimated expenditures for TSA’s September 2018 inventory of screening technologies, by screening technology and life-cycle phase (i.e., purchase, deploy, install, and maintain). Appendix III provides additional information on estimated TSA expenditures, such as prices per unit of technology and estimated expenditures by airport category. TSA has also incurred costs, or has plans to incur costs, for additional actions related to screening technologies. Specifically, it has also incurred costs for modifications to commercial airport facilities to accommodate screening technologies. Further, TSA estimates additional life-cycle costs of $804 million to acquire, deploy, and maintain computed tomography systems through fiscal year 2026. The following provides more information on these estimated expenditures. Airport modifications. TSA incurs costs related to modifying commercial airports to accommodate certain screening technologies, such as checked baggage screening systems (e.g., explosives detection systems). In December 2017, we reported that TSA had obligated at least $783 million from fiscal years 2012 through 2016 to reimburse airports for the allowable design and construction costs associated with installing, updating, or replacing screening technology. For example, TSA may enter into agreements to reimburse airport operators for a percentage of the allowable design and construction costs associated with facility modifications needed for installing, updating, or replacing in-line explosives detection systems. In-line screening systems use conveyor belts to route checked luggage through explosives detection systems, which capture images of the checked baggage to determine if a bag contains threat items not permitted for transport, including explosives. From fiscal years 2012 through 2016, agreements for TSA reimbursements to airports for checked baggage screening systems generally ranged in value from $50,000 to $150 million. As we reported in December 2017, in general, depending on the airport’s size, TSA will reimburse 90 or 95 percent of the allowable, allocable, and reasonable cost of certain projects. For other projects, TSA may provide 100 percent reimbursement—for example, where existing systems require the correction of security or safety deficiencies. Computed tomography. In addition to its fiscal year 2018 inventory, TSA is currently in the process of deploying computed tomography to commercial airports to replace advanced technology x-ray systems. Computed tomography technology applies sophisticated algorithms to detect explosives and other prohibited items and creates a 3D image of carry-on baggage that a TSO can view and rotate 360 degrees. In fiscal year 2018, TSA determined that computed tomography is the best technology available to address rapidly evolving threats in the transportation sector, and plans to eventually deploy it to all checkpoints and replace advanced technology x-ray technology. As recorded in TSA’s Deployed Locations Report, TSA had deployed 11 computed tomography systems to category X and I airports as of September 24, 2018. According to TSA’s September 2018 life-cycle cost estimates, the agency plans to field 883 units by fiscal year 2026. As shown in table 2, TSA also planned to spend $805 million to purchase, deploy, and maintain this new technology through fiscal year 2026. However, in August 2019, TSA officials told us that they expect this estimated total procurement cost of $805 million to likely decrease as the per unit cost had decreased from $400,000 to $233,000 in the initial fiscal year 2019 contract for computed tomography. TSA has invested billions of dollars in screening technologies as it responds to terrorists’ attempts to use homemade explosives to disrupt and damage civil aviation. Forecasted increases in passenger volumes and ongoing terrorist threats make it imperative that TSA employ recommended management and internal control practices. TSA could help ensure that critical detection standards are developed in accordance with approved practices, and that agency goals are effectively met by updating its guidance for developing standards. Additionally, by documenting key decisions in the development of detection standards, TSA could better assure the effectiveness of decision-making and the retention of organizational knowledge in the face of inevitable changes in personnel. Similarly, when making technology deployment decisions, incorporating DHS-recommended practices for risk management would improve TSA’s ability to effectively fulfill its mission to secure the nation’s civil aviation system. While TSA assesses risk when deciding whether to invest in a new technology to address an identified capability gap, it is unclear the extent to which it considers risk when determining where and in what order to deploy approved screening technologies to airports. DHS guidance for homeland security risk management calls for risk to be considered consistently and comprehensively in all aspects of an agency’s work. Additionally, risk management includes transparent disclosure of the rationale behind decision-making so that stakeholders can understand how key factors were weighed. Incorporating these risk management principles into its decision-making for deploying screening technologies to airports would allow TSA to align its deployment strategies with potential threats, vulnerabilities, and consequences. Lastly, TSA cannot ensure that its screening technologies continue to meet detection requirements after they have been deployed to airports. Developing and implementing a policy to ensure that TSA’s screening technologies continue to meet their respective detection requirements after deployment may assure the agency that its deployed screening technologies are effectively detecting explosives and other prohibited items that they are designed to identify, which is a critical part of TSA’s mission. We are making the following five recommendations to TSA: The TSA Administrator should update TSA guidance for developing and approving screening technology explosives detection standards to reflect designated procedures, the roles and responsibilities of stakeholders, and changes in the agency’s organizational structure. (Recommendation 1) The TSA Administrator should require and ensure that TSA officials document key decisions, including testing and analysis decisions, used to support the development and consideration of new screening technology explosives detection standards. (Recommendation 2) The TSA Administrator should require and ensure that TSA officials document their assessments of risk and the rationale—including the assumptions, methodology, and uncertainty considered—behind decisions to deploy screening technologies. (Recommendation 3) The TSA Administrator should develop a process to ensure that screening technologies continue to meet detection requirements after deployment to commercial airports. (Recommendation 4) The TSA Administrator should implement the process it develops to ensure that screening technologies continue to meet detection requirements after deployment to commercial airports. (Recommendation 5) We provided a draft of this product to DHS for comment. We provided a draft of this report to DHS for review and comment. DHS provided written comments, which are reproduced in full in appendix IV. DHS concurred with our five recommendations and described actions undertaken or planned to address them. TSA also provided technical comments, which we incorporated as appropriate. With regard to our first recommendation that TSA update guidance for developing and approving screening technology explosives detection standards, DHS concurred and stated that TSA has included updated guidance in its Requirements Engineering Integrated Process Manual, which was completed in September 2019. According to DHS, the update provides TSA’s process for developing and approving explosives detection standards, including designated procedures and roles and responsibilities of stakeholders, and reflects organizational changes to TSA. TSA provided us with the Requirements Engineering Integrated Process Manual in November 2019, concurrent with DHS comments. We will review the update and the extent to which it addresses the recommendation. This action, if fully implemented, should address the intent of the recommendation. DHS concurred with our second recommendation that TSA ensure that officials document key decisions supporting the development of screening technology explosives detection standards. DHS stated that the updated Requirements Engineering Integrated Process Manual describes the process for documenting key decisions, including testing and analysis decisions, in the development of new detection standards. We will review the update and the extent to which it addresses the recommendation. This action, if fully implemented, should address the intent of the recommendation. DHS also concurred with our third recommendation that TSA document its assessments of risk and the rationale behind its decisions to deploy screening technologies. According to DHS, TSA has instituted an improved process for documenting elements that contribute to deployment decisions—TSA’s August 2019 deployment plan for computed tomography is an example of the process. DHS stated that TSA will continue to include a comparable level of documentation in future deployment plans for screening technologies. We agree the computed tomography deployment site selection strategy is an example of how TSA can document the rationale governing the deployment of a screening technology. Future plans can further benefit by explaining the risk analysis itself along with the role that risk considerations played in the selection of airports for deployment. Formalizing guidance that directs TSA officials to document risk assessments and the rationale behind deployment decisions would help TSA ensure that its deployment of screening technologies matches potential risks. DHS concurred with our fourth and fifth recommendations that TSA, respectively, develop and implement a process to ensure that screening technologies continue to meet all detection requirements after deployment to commercial airports. DHS stated that TSA will develop recurring individual post implementation reviews (PIR) for all screening technologies in accordance with DHS Directive 102-01, to assess multiple aspects of system performance, including detection over time. DHS also stated that TSA intends to examine the component performance of the detection chain rather than a direct measure of detection requirements, due to the limitations of using live explosives and simulants. DHS stated that because the detection chain for each technology is unique and will require individual reviews, TSA is developing a policy on the PIR development process, which it estimates will be completed by March 31, 2020. We appreciate the limitations live explosives and simulants present in testing and the need for reviews that are tailored to meet the unique characteristics of each screening technology. TSA plans to implement the review process on the first screening technology by December 31, 2020. These actions, if implemented across all applicable screening technologies, should address the intent of the recommendations. We are sending copies of this report to the appropriate congressional committees and to the Acting Secretary of Homeland Security. In addition, this report is available at no charge on the GAO website at http://gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in Appendix V. This appendix presents additional details on the TSA screening technologies we reviewed, including their function and the number of units deployed. This report addresses Transportation Security Administration’s (TSA) processes for developing and deploying screening technologies to airports regulated by TSA (i.e., “commercial” airports). Specifically, we examined 1. the extent to which TSA has a process for developing explosives detection standards for screening technologies in response to identified emerging threats; 2. how TSA operationalizes detection standards to update detection capabilities; 3. the extent to which TSA has considered risk when deploying screening technologies to commercial airports; 4. the extent to which TSA ensures screening technologies meet the requirements for detection standards after deployment; and 5. TSA estimated expenditures to purchase, deploy, install, and maintain its inventory of screening technologies as of the end of fiscal year 2018. To address all of our objectives, we identified 11 screening technologies TSA used to screen passengers’ identification documents, person, carry- on bags, and checked baggage at commercial airports as of September 24, 2018, as recorded in TSA’s Government Property Management database. The seven screening technologies in use at commercial airport passenger checkpoints were advanced imaging technology, advanced technology x-ray machine, bottled liquid scanner, boarding pass scanner, chemical analysis device, threat image projection x-ray, and walk-through metal detector. The credential authentication technology and computed tomography, also used at checkpoint screening, were deployed and in use at select airports as TSA pilot projects. The two TSA screening technologies in use at commercial airports for checked baggage were explosives detection systems and explosives trace detection (TSA also uses explosives trace detection for checkpoint screening). We assessed the reliability of TSA’s inventory data by interviewing agency officials and reviewing related documentation, such as the database user manual, among other things. We determined the data were sufficiently reliable to determine the type and number of TSA screening technologies deployed as of September 2018. To better understand how TSA screening technologies have been used, we reviewed reports from the U.S. Department of Homeland Security (DHS) Office of the Inspector General, the Congressional Research Service, past GAO reports, and relevant DHS and TSA documentation, such as DHS and TSA strategic documents and acquisition plans. To observe TSA screening procedures and the operation of screening technologies in the airport setting, we conducted site visits to seven commercial airports. During these visits we discussed screening technology issues with TSA federal security directors or their representatives. We selected these airports to reflect a range of airport categories, technologies, and geographic diversity. The results of these site visits and interviews cannot be generalized to all commercial airports, but they provided us with important context about the installation, use, and maintenance of TSA screening technologies across the different types of airports that TSA secures. We also conducted a site visit to the TSA Systems Integration Facility to better understand how screening technologies are tested and evaluated prior to deployment. Further, we interviewed officials from two industry associations and one screening technology manufacturers association based on input from TSA and DHS Science and Technology Directorate (S&T) officials. To determine the extent to which TSA has a process for developing explosives detection standards, we examined TSA documents such as approved detection standards, action memos summarizing support for proposed detection standards, the Detection Requirements Update Standard Operating Procedure, and briefing slides describing TSA’s process, as of August 2019, for assessing threat materials and developing detection standards. We also evaluated Material Threat Assessment reports that summarized the testing and analyses performed by S&T’s Homemade Explosives Characterization Program, in coordination with S&T laboratories, to characterize (identify the physical density and mass of) explosive materials for detection standards developed from fiscal years 2014 through 2018. We evaluated S&T’s testing and analyses in accordance with TSA and S&T guidance to determine the extent to which these steps were consistent across materials; we did not analyze the sufficiency of the testing and analyses. We also assessed TSA and S&T processes and the extent to which they were documented in accordance with Standards for Internal Control in the Federal Government, and discussed the details of steps taken to develop standards with relevant TSA and S&T officials. In addition, we conducted a site visit to S&T’s Commercial Aircraft Vulnerability and Mitigation Program testing site at the U.S. Army Aberdeen Test Center, Maryland, to better understand how S&T tests the vulnerability of commercial aircraft to explosive materials. To understand TSA’s process and timelines for operationalizing—putting into effect—detection standards, we requested information from TSA about screening technologies subject to explosives detection standards, deployed as of September 24, 2018: advanced imaging technology, advanced technology x-ray, bottled liquid scanner, explosives detection systems, and explosives trace detection. We requested information about the detection standards that deployed screening technologies met, as of August 2019, as well as subsequently approved detection standards, including the date the standards were approved, the dates when TSA achieved certain acquisition milestones when developing and deploying the associated technologies, and the status of ongoing and upcoming efforts to update detection capabilities to meet new standards. We identified the acquisition milestones by reviewing a past GAO report on TSA’s acquisition process and in consultation with GAO acquisition experts. We also reviewed a classified TSA report that evaluated the performance of a particular algorithm in order to understand TSA’s process for developing new screening technologies to meet detection standards. In addition, we reviewed relevant acquisition documents, such as DHS’s Acquisition Management Instruction 102, the 2018 Transportation Security Administration Systems Acquisition Manual, acquisition decision memos, acquisition plans, and Operations Requirements Documents. To understand TSA’s process for deciding whether to operationalize detection standards, we requested and reviewed available documentation for the standards that TSA had not operationalized, such as an operational status transition memo for bottled liquid scanner, and interviewed TSA officials about those decisions. To understand how TSA had considered risk in its approach to deploying screening technologies at airports, we reviewed available documentation related to TSA’s deployment decisions. These included decision memos from acquisition review board meetings and action memos to TSA leadership; risk registers for checked baggage and checkpoint acquisition programs; available deployment plans, such as the agency’s Action Plan for deploying explosives trace detection units to airports in 2018; and acquisition guidance. To understand how TSA assesses capability needs and gaps, we interviewed agency officials about TSA’s Transportation Security Capability Analysis Process and reviewed capability analysis reports from 2018 and 2019, as well as TSA’s prioritized list of capability gaps and needs. We also interviewed acquisition officials, including TSA’s Component Acquisition Executive, about the role of risk in deployment decisions and requested written responses to specific questions. We assessed TSA’s decision-making process for deploying and updating screening technologies, generally, against DHS risk management criteria, such as DHS’s Risk Management Fundamentals. We also reviewed related areas of risk management and decision-making to understand the context in which TSA makes deployment decisions. Specifically, we reviewed the 2017 Transportation Sector Security Risk Assessment and the Cities and Airports Threat Assessment reports to understand the risks facing the nation’s aviation system. We also reviewed TSA’s enterprise risk management framework, such as the Enterprise Risk Management Policy Manual, to understand the role it played in TSA’s deployment decisions. We also interviewed an official from TSA’s Enterprise Performance and Risk office and the Executive Risk Steering Committee. To understand how TSA categorizes airports, we reviewed a 2017 Nationwide Airport Categorization Review memo from TSA’s Security Operations office and interviewed Security Operations officials. To understand how TSA deploys screening technologies across airports and categories of airports, we analyzed TSA’s Deployed Locations Report, which reported on technologies that were in use or available for use at commercial airports from September 24 through September 30, 2018. We also reviewed TSA’s standardized methodology for determining the most efficient number of screening technologies at an airport. Additionally, we reviewed TSA’s Strategic Five-Year Technology Investment Plan from 2015 and the 2017 Biennial Refresh to understand TSA’s plans for ongoing investment in screening technologies. We reviewed various throughput data, such as annual passenger throughput for all commercial airports for fiscal year 2018 and enplanements data for calendar year 2017, to understand and compare TSA’s allocation of screening technologies with throughput data across airports and airport categories. We used this analysis to identify airports that had an unusually large or small number of screening technologies within a category, and interviewed TSA officials to understand the decisions that led to the allocation of screening technologies across airports and airport categories. In addition, we reviewed the status of TSA’s limited deployment of computed tomography units to checkpoints. Specifically, we reviewed TSA’s 2018 Deployment Site Selection Strategy, which described the airports to which TSA would deploy computed tomography units and the methodology it used to select them, slides from recent conferences TSA held with industry representatives where it shared its plans for transitioning to computed tomography, and relevant Operational Requirements Documents. We also interviewed agency officials about their plans for the limited deployment and TSA’s transition from advanced technology x-ray to computed tomography for checkpoint screening. To determine the extent to which TSA ensures its screening technologies continue meeting detection requirements after deploying them to airports, we reviewed TSA acquisition detection requirements for each screening technology as well as TSA guidance related to the testing and evaluation of screening technologies identified by TSA officials in interviews. We also interviewed TSA and S&T Transportation Security Laboratory officials about TSA requirements to test screening technologies, both prior to and after deployment, to determine the extent to which they meet detection requirements. We also observed transportation security officers and a transportation security specialist for explosives conduct verification and calibration procedures on screening technologies at the airports we visited. We reviewed TSA guidance to determine the extent to which its procedures ensure that screening technologies continued to meet detection requirements in airports. We then evaluated the procedures against Standards for Internal Control in the Federal Government for monitoring. To identify TSA’s estimated expenditures to purchase, deploy, install, and maintain its inventory of screening technologies as of the end of fiscal year 2018, we reviewed TSA programs’ life-cycle cost estimates, which, for the purposes of acquisition planning, provide per unit estimates of the cost to purchase, deploy, install, and maintain passenger and checked baggage screening technologies. We chose this methodology in consultation with TSA officials and after determining that historical records of obligations and expenditures do not provided consistent and sufficient detail for the purposes of our analysis. The life-cycle cost estimates include relevant phases for each screening technology (i.e., purchase, deploy, install, and maintain), although not all technologies have cost estimates for each phase of the life cycle. For example, some screening technologies may not specify deployment costs because such costs are included in the initial purchase price of the unit. In other cases, the technology does not have a deployment cost because the unit is small and portable, and placement of the unit is therefore handled by TSA airport staff at no charge. Estimated expenditures for installation also include costs associated with site acceptance testing, which is performed when a system is installed at its operational location. Unlike the purchase, deploy, and install unit prices, the maintenance unit price is the yearly cost of maintenance for one unit, and therefore recurs every year. We assessed the reliability of the life-cycle cost estimates by reviewing documentation on the development of the estimates and interviewing TSA officials, among other things, and determined the estimates were sufficiently reliable for the purpose of estimating the amount of funds spent on acquiring, deploying, installing, and maintaining TSA’s inventory of screening technologies as of the end of fiscal year 2018. Because the life-cycle cost estimates were developed in different years, we used TSA guidelines to adjust costs for inflation and convert our estimates to 2018 dollars. We multiplied these estimates against the number of screening technologies deployed to commercial airports as of September 24, 2018, using data from TSA’s Government Property Management database. For computed tomography, we also obtained information on price and quantity from the technology’s life-cycle cost estimate and TSA officials. We also reviewed prior GAO work on TSA cost sharing programs for airport facility modification related to installation of some of the technologies in our review. We conducted this performance audit from April 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We estimate that TSA spent $3.1 billion to purchase, deploy, install, and maintain its inventory of screening technologies, as of the end of fiscal year 2018, based on agency estimates of costs. Tables 3 through 5 provide information on estimated TSA expenditures by screening technology, life-cycle phase, and airport category. To analyze TSA’s estimated spending to purchase, deploy, install, and maintain its inventory of screening technologies as of the end of fiscal year 2018, we reviewed TSA life-cycle cost estimates, which, for the purposes of acquisition planning, provide per-unit estimates of the cost to purchase, deploy, install, and maintain passenger and checked baggage screening technologies at TSA-regulated airports (i.e., “commercial” airports). Because the life-cycle cost estimates were developed in different years, we used the same guidelines used by TSA to adjust costs for inflation to convert our estimates to 2018 dollars. We multiplied these estimates against the number of screening technologies deployed to commercial airports as of September 24, 2018. In addition to the contact named above, Kevin Heinz (Assistant Director), Barbara Guffy (Analyst in Charge), Kelsey Burdick, Jonathan Felbinger, Tyler Kent, Thomas Lombardi, Erin O’Brien, Kya Palomaki, Rebecca Parkhurst, and Dina Shorafa made key contributions to this report. In addition, key support was provided by Chuck Bausell, Richard Cederholm, Dominick Dale, Aryn Ehlow, Michele Fejfar, Eric Hauswirth, Richard Hung, and Alexis Olson.", "summary": "TSA is responsible for overseeing security operations at roughly 440 TSA-regulated airports as part of its mission to protect the nation's civil aviation system. TSA uses technologies to screen passengers and their bags for prohibited items. The TSA Modernization Act includes a provision for GAO to review TSA's deployment of screening technologies, and GAO was asked to review the detection standards of these screening technologies. This report addresses, among other things, (1) how TSA operationalizes detection standards, (2) the extent to which TSA considered risk when making deployment decisions, and (3) the extent to which TSA ensures technologies continue to meet detection requirements after deployment. GAO reviewed DHS and TSA procedures and documents, including detection standards; visited DHS and TSA testing facilities; observed the use of screening technologies at seven airports, selected for varying geographic locations and other factors; and interviewed DHS and TSA headquarters and field officials. The Department of Homeland Security's (DHS) Transportation Security Administration (TSA) operationalizes, or puts into effect, detection standards for its screening technologies by acquiring and deploying new technologies, which can take years. Detection standards specify the prohibited items (e.g., guns, explosives) that technologies are to detect, the minimum rate of detection, and the maximum rate at which technologies incorrectly flag an item. TSA operationalizes standards by adapting them as detection requirements, working with manufacturers to develop and test new technologies (software or hardware), and acquiring and deploying technologies to airports. For the standards GAO reviewed, this process took 2 to 7 years, based on manufacturers' technical abilities and other factors. TSA's deployment decisions are generally based on logistical factors and it is unclear how risk is considered when determining where and in what order technologies are deployed because TSA did not document its decisions. TSA considers risks across the civil aviation system when making acquisition decisions. However, TSA did not document the extent risk played a role in deployment, and could not fully explain how risk analyses contributed to those decisions. Moving forward, increased transparency about TSA's decisions would better ensure that deployment of technologies matches potential risks. Technology performance can degrade over time; however, TSA does not ensure that technologies continue to meet detection requirements after deployment to airports. TSA certifies technologies to ensure they meet requirements before deployment, and screeners are to regularly calibrate deployed technologies to demonstrate they are minimally operational. However, neither process ensures that technologies continue to meet requirements after deployment. In 2015 and 2016, DHS tested a sample of deployed explosives trace detection and bottled liquid scanner units and found that some no longer met detection requirements. Developing and implementing a process to ensure technologies continue to meet detection requirements after deployment would help ensure that TSA screening procedures are effective and enable TSA to take corrective action if needed. GAO is making five recommendations, including that TSA document analysis of risk in deploying technologies, and implement a process to ensure technologies continue to meet detection requirements after deployment. DHS agreed with all five recommendations and said TSA either has taken or will take actions to address them.", "document_type": "gao"}
{"report": "The Coast Guard owns or leases more than 20,000 shore facilities consisting of various types of buildings and structures. According to Coast Guard guidance, a building is generally defined as a fully enclosed structure that is affixed to the ground, in which personnel work or live, or where equipment is stored. A structure is generally defined as any other construction affixed to the ground that does not meet the definition of a building. The Coast Guard’s shore infrastructure is organized into 13 asset types, known as asset lines. Table 1 provides information on 11 of these asset lines, including examples, numbers of assets, and their replacement value as of 2018. We reported in February 2019 that the Coast Guard faced recapitalization, new construction, and deferred maintenance backlogs for its shore infrastructure totaling at least $2.6 billion as of 2018 and that its backlogs increased by $300 million since fiscal year 2012. Moreover, according to the Coast Guard Civil Engineering program’s 2018 annual report, about 46 percent of the Coast Guard’s shore infrastructure was beyond its overall service life. In 2018, the Coast Guard rated its overall shore infrastructure condition as a C- based on criteria it derived from standards developed by the American Society of Civil Engineers. In addition, some asset lines such as the industrial asset line, whose assets are generally mission-critical, were rated lower. Table 2 shows information about Coast Guard asset lines, including the rate at which the Coast Guard reported that these assets were functioning past their service life, and the condition grades assigned by the Coast Guard for fiscal year 2018. According to Coast Guard guidance, the Office of Civil Engineering and the Shore Infrastructure Logistics Center each play a role in managing the Coast Guard’s infrastructure by assessing risks and helping to mitigate damage from natural disasters or other events. The Office of Civil Engineering is responsible for setting Coast Guard-wide civil engineering policy, which includes facility planning, design, construction, maintenance, and disposal. The Shore Infrastructure Logistics Center is to establish project priorities for the acquisition, programmed depot maintenance, major repair, and modification of shore facilities. This center is also responsible for implementing the Coast Guard’s shore infrastructure policies. According to its guidance, the Coast Guard makes procurement, construction, and improvements funding decisions for its shore infrastructure through enterprise-level planning boards that meet twice a year. These planning boards are to prioritize Coast Guard shore infrastructure needs based on expected appropriations and other prioritization factors or considerations, such as damage caused by natural disasters or the Coast Guard’s need to construct new shore infrastructure or recapitalize existing facilities. The boards are responsible for evaluating potential shore infrastructure projects that have been assessed, ranked, and recommended by Coast Guard managers of various asset lines. For example, aviation asset line managers may recommend to the planning boards aviation-related shore infrastructure projects, such as the recapitalization of runways, landing areas, and hangars. According to the National Academies, climate change poses serious risks to many of the physical and ecological systems on which society depends, although the exact details cannot be predicted with certainty. Moreover, the effects and costs of extreme weather events, such as floods and droughts, are expected to increase in significance as they become more common and intense because of climate change. For example, the National Oceanic and Atmospheric Administration (NOAA) has reported that eight of the 10 costliest tropical cyclones in U.S. history occurred in recent years—from 2005 to 2017. DOD documented seven effects commonly associated with climate change and their potential effects on its infrastructure and operations (see table 3). Although the Coast Guard operates on a smaller scale, it maintains many of the same types of infrastructure as DOD, and these infrastructure are also situated in coastal and riverine locations, and thus subject to the same potential effects from extreme weather events. For example, Coast Guard facilities along the East and Gulf coasts of the United States are vulnerable to hurricanes—which NOAA projects will increase in frequency and severity because of climate change—and may cause flooding or wind damage to Coast Guard infrastructure. Coast Guard infrastructure is also vulnerable to natural disasters that are not associated with climate change. For example, Coast Guard facilities situated on the West Coast, Hawaii and Alaska, are located on or near historic earthquake fault lines. As a result, this infrastructure is more likely to be damaged by earthquakes than infrastructure located elsewhere in the country, according to the Coast Guard. According to Coast Guard officials, it can take months and sometimes years to repair or replace Coast Guard facilities damaged by severe natural disasters. For example, as shown in Figure 1, Coast Guard facilities at Station Port Aransas in Texas suffered significant damage during Hurricane Harvey in 2017. As of April 2019, the Coast Guard was working to demolish these damaged facilities so they could be replaced by one facility that is resilient to hurricanes. DHS established its Critical Infrastructure Risk Management Framework to guide critical infrastructure owners and operators, from both the public and private sector, in investing limited resources to protect critical infrastructure. As shown in Figure 2, the framework consists of five steps that involve (1) setting goals and objectives, (2) identifying infrastructure, (3) assessing and analyzing risk, (4) implementing risk management activities, and (5) measuring the effectiveness of actions taken to address identified risks. According to DHS, agency decision makers can use this framework to prioritize investments, develop plans, and allocate resources for critical infrastructure in a risk-informed way. The framework is based on risk management activities, which call for cost-effective use of resources by taking protective actions that offer the greatest mitigation of risk for any given expenditure. According to the NIPP, a risk management approach that aligns with the five key steps can help guide organizational decision making and prioritize actions to more effectively achieve desired outcomes. Since 2005, the Coast Guard has taken actions to improve the resilience of at least 15 storm-damaged shore facilities and has rebuilt them to new standards largely by using supplemental appropriations provided for this purpose. The Coast Guard has also developed new guidance to increase the likelihood that new or recapitalized buildings will withstand natural disasters and follows updated state and local building codes, which a senior Coast Guard official told us led to more resilient buildings, thus limiting risks to Coast Guard personnel and operations. In 2015, the Coast Guard’s Civil Engineering program initiated a formal assessment of owned and occupied Coast Guard buildings to determine which were vulnerable to 10 natural disasters, which, according to agency officials, it aims to complete in 2025. Since 2005, the Coast Guard has taken actions to improve the resilience of its shore infrastructure, largely by using supplemental appropriations for rebuilding facilities damaged by major storms. Specifically, from December 2005 through June 2019, the Coast Guard was appropriated about $2 billion in supplemental funds to, among other things, rebuild or relocate 15 facilities damaged by hurricanes. During this time, the Coast Guard has relocated facilities further inland or to higher ground, upgraded facilities to be more resilient, and designed new facilities with features to protect them from natural disasters. The 2016 and 2017 hurricane seasons were particularly destructive, and the Coast Guard received $719 million in supplemental funding to restore facilities damaged by Hurricanes Matthew, Harvey, Irma, and Maria. Figure 3 below shows Coast Guard shore infrastructure, and associated replacement values, located along the East and Southeast coasts of the United States and the general paths of selected hurricanes in those regions since 2005. The Coast Guard has used supplemental funding to rebuild or relocate at least 15 damaged facilities to enhance their resilience. To improve the resilience of its facilities when rebuilding after hurricanes, Coast Guard officials reported that they generally either relocated the facility inland for better protection from extreme weather or modified the facility to be more resilient by elevating it to protect it from storm surge and flooding. For example: Station Houston, Texas. After this station was damaged by Hurricane Ike in 2008, the Coast Guard determined that this station’s boathouse could not be built above the local floodplain and still meet mission requirements. As a result, the Coast Guard took steps to protect the boathouse from future water damage by using water resistant materials in its construction, elevating its electrical and telecommunications systems above the flood plain, and placing the heating, ventilation, and air conditioning systems on the roof of the building. Sector Houston-Galveston, Texas. After being damaged by Hurricane Ike in 2008, this regional command facility was relocated further inland to provide the new facility with greater protection from extreme weather. It was also designed to withstand wind speeds of up to 115 miles per hour. Station Sandy Hook, New Jersey. After this station was damaged by Hurricane Sandy in 2012, the old building was demolished and replaced on the same site with a facility that was designed to be more resilient. The station’s first floor was constructed with openings to allow flood waters to pass beneath the station. Station Sabine Pass, Texas. Following damage by Hurricane Ike in 2008, the Coast Guard rebuilt this station in 2013 to better withstand floods and high winds (see fig. 4). The new station’s first floor was elevated to a height that exceeds the projected depth of a 100-year flood to protect station equipment. The station was also designed to resist wind speeds up to 130 miles per hour—sufficient to withstand a Category III hurricane. The Coast Guard has also developed new guidance reflecting higher building standards, and follows updated state and local building codes which a senior Coast Guard official told us led to more resilient buildings. In February 2017, the Coast Guard’s Civil Engineering program issued engineering planning guidance intended to increase the likelihood that new or recapitalized buildings would withstand natural disasters and that the design of these buildings would minimize risks to Coast Guard operations and personnel, among other things. This new guidance contains the following requirements: All new permanent, regularly occupied buildings will either be located at least 2 feet above the Federal Emergency Management Agency’s (FEMA) 100-year base flood elevation or meet the level of the 500- year base flood elevation for the proposed site location. To account for storm surge, sea level rise, or periodic flooding, buildings may also be constructed above this elevation as necessary. To ensure operational continuity and safety after a flood event, critical building systems—such as utility and communications systems—are to be located at least 3 feet above the 100-year base flood elevation. Each site will be evaluated for vulnerability to natural hazards, such as earthquakes, tornadoes, and wildfires. This evaluation will identify risk to Coast Guard operations and personnel. A senior Coast Guard official testified to Congress in November 2017 that Coast Guard buildings rebuilt after being damaged by Hurricane Ike in 2008 suffered minimal damage from Hurricanes Harvey and Irma. The official also said that the resilience of these buildings resulted from the recapitalization efforts that made them more storm-resilient and allowed them to align the buildings with modern building codes and standards. Further, according to Coast Guard civil engineering officials, units impacted by Hurricanes Harvey, Irma and Maria—which had been recapitalized to resilient standards—returned to full mission capability within 2 to 3 days and, in some instances, avoided damage or a loss of mission capability as a result of more resilient construction. For example, operations at Sector Houston-Galveston, which supports a wide range of Coast Guard missions, were not interrupted during Hurricane Harvey, allowing it to serve as the primary federal response hub during this disaster. A senior official from the Coast Guard Facilities Design and Construction Center told us that state and local building codes, which have been updated as a result of lessons learned from natural disasters, have also led to more resilient Coast Guard buildings because the Coast Guard is required to align its facilities standards with these codes. For example, according to this official, Florida updated its building codes after Hurricane Andrew in 1992, which resulted in more resilient buildings in this state. In December 2018, the Coast Guard Civil Engineering program issued updated planning guidance for reconstructing facilities damaged by Hurricanes Matthew, Harvey, Irma, and Maria in 2016 and 2017. According to this guidance, new and renovated facilities are to incorporate resilient construction techniques including, but not limited to, hurricane resistant construction and design, and infrastructure resiliency. These facilities are to have the ability to return to full operations after an event, minimizing any major reconstruction and long-term mission impact. In addition, when the Coast Guard builds a new facility or renovates an existing one that directly supports Coast Guard natural disaster response efforts, that facility is to be built to a higher resiliency level to increase the likelihood that it will remain operational during a natural disaster. In 2015, the Coast Guard’s Civil Engineering program initiated a formal vulnerability assessment of owned and occupied Coast Guard buildings, and according to Coast Guard officials they aim to complete this assessment in 2025. The Coast Guard calls this assessment the Shore Infrastructure Vulnerability Assessment. According to Coast Guard documentation, its focus was to determine the vulnerability of these buildings and Coast Guard personnel to natural disasters. Further, the assessment results are intended to assist with contingency planning by identifying which Coast Guard facilities are likely to remain operational after a natural disaster. According to its documentation, this vulnerability assessment is to be completed in two phases. During Phase I, completed in 2018, the Coast Guard analyzed 3,214 buildings, or approximately 16 percent of its infrastructure, for vulnerabilities to disasters such as floods, earthquakes, and hurricanes. To conduct its analysis, Coast Guard officials analyzed the vulnerability of these buildings to 10 natural disasters using information from other government agencies and professional organizations. For example, the Coast Guard assessed its vulnerability to flooding using FEMA, National Weather Service information, state sources and websites. This analysis identified Coast Guard-wide infrastructure vulnerabilities to coastal risks such as shoreline loss, coastal erosion and earthquakes, as well as tsunami risks on the West Coast of the United States, Alaska, Guam, and Hawaii, and immediate and serious flood risks in Puerto Rico and the Gulf and East Coasts. The Phase I report recommended that Coast Guard units and contingency planners consider these vulnerabilities when preparing contingency plans or making capital investments in Coast Guard facilities. Although the Shore Infrastructure Vulnerability Assessment Phase I report identified multiple vulnerabilities to sixty-eight percent of the assessed infrastructure, Coast Guard Civil Engineering program officials told us they were unable to conclusively determine whether approximately 1,500 assessed buildings were vulnerable to hurricane winds, earthquakes, or tornadoes—leading officials to conclude that they needed to conduct further structural analysis. Accordingly, Coast Guard Civil Engineering program officials initiated plans for Phase II of the assessment, which involves more detailed structural analyses of 1,500 buildings to determine whether they can withstand either earthquakes or tornado and hurricane winds, depending on the building. Since earthquakes strike with essentially no warning, unlike hurricanes and tornadoes, Coast Guard Civil Engineering program officials told us that the Coast Guard considered them to be a greater threat to its personnel and infrastructure. Accordingly, the Coast Guard decided that Phase II of the assessment would prioritize structural analyses for buildings it believes to be more susceptible to damage from earthquakes. Further, it would prioritize the order in which it assesses these buildings based on how critical the building is to Coast Guard operations, building occupant density, and the overall age and condition of the building. The Shore Infrastructure Vulnerability Assessment Phase II analysis began in September 2018 with a contract for about $700,000 to determine if 15 buildings at multiple Coast Guard sites are vulnerable to earthquakes. According to the contract, these assessments are to be completed in October 2021. While the Coast Guard has taken steps to improve the resilience of its shore infrastructure by rebuilding storm damaged facilities and initiating a vulnerability assessment, its overarching processes to improve shore infrastructure resilience are not fully aligned with the five steps of the DHS Critical Infrastructure Risk Management Framework. As previously mentioned, DHS established this framework to guide both public and private resource investment decisions for protecting critical infrastructure. Its five steps include (1) setting goals and objectives, (2) identifying infrastructure, (3) assessing and analyzing risk, (4) implementing risk management activities, and (5) measuring the effectiveness of actions taken to address identified risks. According to the first step of the DHS Critical Infrastructure Risk Management Framework, organizations should define specific goals for what they intend to accomplish and establish objectives to help them achieve the goals (see text box). Organizations that establish broad strategic goals for risk management can also benefit from translating these goals into specific, measurable objectives to assess the extent to which its actions actually reduce risk (see text box). DHS Critical Infrastructure Risk Management Framework—Step 1 Organizations should define specific outcomes, conditions, end points, or performance targets that collectively describe an effective and desired risk management posture. By defining risk management goals and expressing them in terms of the objectives and outcomes the organization intends to accomplish, stakeholders, including those at all levels of government and the private sector, would be better able to tailor their risk management programs and activities to address infrastructure resilience needs. Our review of four key Coast Guard documents related to managing its shore infrastructure showed that some of these documents refer to resilience and identify it as an important factor to its operational success. However, none of the documents we reviewed identified a measurable goal or objective for improving shore infrastructure resilience. Instead, the documents either include goals related to management of the shore infrastructure program, or include no goals at all. Specifically: The Coast Guard Shore Infrastructure Strategic Plan for 2017-2021 includes what it describes as performance and foundational goals, including a foundational goal for improving resilience, contingency preparedness, and response to natural hazards. However, the plan does not link this foundational goal to a specific objective and performance target that could guide Coast Guard actions to improve shore infrastructure resilience. For example, an objective could be to increase the percentage of mission critical buildings that are within or above base flood elevations by a certain date, and annual targets could be established to assess progress toward this goal. The Coast Guard issued its agency-wide strategic plan in November 2018 which states that resilient shore infrastructure is directly connected to Coast Guard operational readiness and successful mission execution. The plan further stated that to meet its operational needs, the Coast Guard will prioritize the repair or replacement of degraded shore infrastructure that negatively affects operations or hinders workforce readiness. However, this plan does not identify the shore infrastructure resilience goals the Coast Guard hopes to achieve or any objectives to measure progress toward these goals. Moreover, this plan does not include goals or measures to guide such prioritization. In February 2019, we reported that Coast Guard Engineering program officials were not able to provide documents showing how they had directed field units to prioritize the repair or replacement of degraded shore infrastructure. In July 2019, the Coast Guard was able to provide one planning document that was specifically created to help manage its response to Hurricanes Harvey, Irma, Maria, and Matthew that included guidance on improving infrastructure resilience. Based on our interviews with Coast Guard engineering program and Shore Infrastructure Logistics Center officials, the Coast Guard is still in the initial stages of incorporating resilience plans and objectives into the shore infrastructure program. In July 2019, Civil Engineering program officials told us that the Coast Guard had updated its Civil Engineering Strategic Plan to direct its personnel to develop a communication plan and resource strategy for infrastructure resiliency projects based on the Shore Infrastructure Vulnerability Assessment’s Phase II results. The Coast Guard provided us with a copy of this plan in August 2019, and while this document includes two measures that can be useful to account for actions taken, it did not include goals or performance targets to guide the prioritization of resiliency projects, and Civil Engineering program officials were not able to provide documents showing how they had made decisions to incorporate resilience into the repair and replacement of degraded shore infrastructure. Coast Guard officials also reported that they had initiated a separate resilience effort in 2018 at the direction of DHS, which required all operational components to participate in the development of the 2018 DHS Resilience Framework, and to develop individual component resilience plans to guide its approach to resilience planning. According to the Coast Guard, their plan was submitted to DHS in August 2019. When we discussed this effort with Coast Guard officials, they were able to provide few details about their efforts and no documentation about their progress to date. We also discussed this effort with DHS officials managing the process, but they were not able to tell us whether this new endeavor will align with or compete for resources with ongoing Coast Guard assessment processes. According to the second step of the DHS Critical Infrastructure Risk Management Framework, organizations should identify infrastructure assets that are critical for security and national preparedness (see text box). DHS Critical Infrastructure Risk Management Framework—Step 2 Organizations should identify assets, systems, and networks that contribute to critical functionality, and collect information pertinent to risk management, including analysis of dependencies and interdependencies. Through this step, it is important to identify assets that are both nationally significant and those that may not be significant on a national level but are, nonetheless, important to state, local, or regional critical infrastructure security and resilience and national preparedness efforts. We found that the Coast Guard identified many occupied buildings that may be important to operations and assessed their vulnerability through its Shore Infrastructure Vulnerability Assessment process, but this process did not identify all shore infrastructure assets that are critical to its missions or screen them for all vulnerabilities. Specifically, through the Shore Infrastructure Vulnerability Assessment Phase I, the Coast Guard identified and screened all occupied Coast Guard buildings over 1,000 gross square feet—about 16 percent of all Coast Guard infrastructure— for vulnerabilities to 10 natural disasters. The analysis found that approximately 68 percent (2,200) of the 3,214 buildings it assessed are vulnerable to certain natural disasters. However, the initial screening did not include other mission critical infrastructure, as the framework recommends, even though the loss of such structures could also impact its ability to carry out its missions. For example, the Coast Guard did not include structures in Phase I of the Shore Infrastructure Vulnerability Assessment, such as aircraft runways, and therefore has not determined whether such structures are vulnerable to flooding following a severe storm, or which ones are at greatest risk for such flooding. Phase II is also not expected to include these assets, as Civil Engineering program officials stated it is not intended to identify any additional infrastructure. Rather, in Phase II for example, Civil Engineering program officials will determine whether roughly 45 percent of the buildings on the West Coast that were screened in Phase I, are vulnerable to earthquakes, as the results of Phase I were inconclusive for these buildings. This DHS framework step recommends that stakeholders identify assets and networks that contribute to critical functionality and analyze their dependencies and interdependencies. The Coast Guard has two such measures to help identify the criticality of its shore infrastructure for conducting its missions. The Mission Essentiality Index measure classifies shore infrastructure assets into one of four tiers based on the degree to which they are mission critical. Similarly, the Mission Dependency Index scores building criticality based on how quickly the loss of utilities would impact operations, and how difficult it would be to relocate operations in advance of a natural disaster. Coast Guard officials told us they used Mission Dependency Index scores to help identify which buildings to include first during Phase II of the Shore Infrastructure Vulnerability Assessment. However, they did not consider either of these measures when they conducted the initial screening for Phase I, which prevented operational risks from being fully considered. Using this information at the beginning of its Shore Infrastructure Vulnerability Assessment process could have provided the Coast Guard with useful information to help it assess its critical infrastructure, as the DHS framework recommends. Coast Guard officials stated in July 2019 that they believe that the mission critical assets collocated with the assessed buildings would have the same vulnerabilities given their geographic proximity. While this may be the case for structures that are collocated with assessed buildings, unoccupied structures (such as piers and runways) may be built with different requirements and building codes; consequently, they may differ in the extent of their vulnerabilities to the same natural hazard threats. Furthermore, the Shore Infrastructure Vulnerability Assessment Phase I report did not demonstrate the extent to which Coast Guard structure are collocated with the occupied buildings the Coast Guard analyzed. They also told us that the Coast Guard has not tracked the performance of its infrastructure, particularly piers and runways, because it has always been able to find alternative means to continue operations. However, by identifying all of its mission critical infrastructure that may be vulnerable to natural disasters, the Coast Guard would be more fully informed of the possible scenarios that could affect their capabilities in the event of a natural disaster, and which infrastructure facilities are most likely to be affected. Such information could also better position the Coast Guard to plan for and execute mission operations from alternative locations if needed. According to the third step of the DHS Critical Infrastructure Risk Management Framework, organizations should assess and analyze risks to understand infrastructure vulnerabilities and threats, as well as the potential consequences of an incident or known vulnerabilities (see text box). DHS Critical Infrastructure Risk Management Framework—Step 3 Organizations should assess and analyze risks, taking into consideration the potential direct and indirect consequences of an incident, known vulnerabilities to various potential threats or hazards, and general or specific threat information. Risks can be assessed in terms of their likelihood and potential consequences. This step supports an assessment strategy that results in sound, scenario-based consequence and vulnerability estimates, as well as an assessment of the likelihood that the given threat or hazard will occur. Organizations should consider potential harm to operations and impacts on mission in executing a critical infrastructure risk management approach. The Shore Infrastructure Vulnerability Assessment process is the Coast Guard’s main action to formally assess and analyze its shore infrastructure, according to Civil Engineering program officials. This process was intended to help contingency planners anticipate which infrastructure is likely to remain operational following a natural disaster, and assist with operational and future capital investment decisions, according to a senior Coast Guard official. We found that through this process, the Coast Guard assessed and analyzed certain elements of risk for its shore infrastructure, such as potential vulnerabilities of certain infrastructure to multiple natural disasters—information which could help inform its processes to improve resilience. However, the Coast Guard has not identified the potential direct and indirect consequences posed by natural disasters on its infrastructure, or the consequences associated with its operational risks—that is, risks affecting its ability to carry out its missions if shore infrastructure is damaged. Specifically: Through Phase I of the Shore Infrastructure Vulnerability Assessment process, the Coast Guard determined that its personnel and operations are generally more vulnerable to certain threats. For example, Phase I determined that about 880 assessed buildings may be vulnerable to earthquakes, which according to the Coast Guard, represent approximately 45 percent of its assessed buildings on the West Coast. Similarly, it also identified about 800 buildings that may be vulnerable to tornadoes and approximately 1,000 buildings vulnerable to hurricanes. However, the Coast Guard has not analyzed the potential consequences of damage to the infrastructure that it identified as vulnerable. For example, it has not assessed the economic losses associated with potential catastrophic disasters, such as costs for rebuilding assets or taking other actions to respond to and recover from natural disasters. Additionally, the Coast Guard has not assessed long-term costs that could result from environmental damage to its property caused by these events. Without also determining consequence information, the Coast Guard is not positioned to provide decision makers with the type of information the DHS Critical Infrastructure Risk Management Framework recommends for making cost effective risk management decisions. As the Coast Guard begins to conduct Phase II, it is unclear whether it will include information on potential consequences in its assessment. The Coast Guard initiated Phase II in September 2018 and intends to assess about 1,500 buildings for vulnerabilities to natural disasters by 2025. Coast Guard officials stated that Phase II would entail following civil engineering standards for conducting the assessments. These assessments are expected to entail on-site contractor assessments of about 1,500 buildings. In 2018, the first year of Phase II, the Coast Guard contracted for an assessment of 15 buildings, and Shore Infrastructure Logistics Center officials said they expect this assessment to be completed in 2021. According to Civil Engineering program officials, the purpose of Phase II is to understand whether 1,500 buildings identified in Phase I as inconclusive are indeed vulnerable to certain natural hazards. This information can help Coast Guard officials better understand the likelihood that vulnerabilities exist, but the plan for Phase II does not support an assessment strategy that results in sound, scenario-based consequence and vulnerability estimates, as well as an assessment of the likelihood that the given threat or hazard will occur or the operational risks that may be affected, as this step recommends. According to the fourth step of the DHS Critical Infrastructure Risk Management Framework, organizations should implement risk management activities by evaluating risk reduction methods that consider countermeasures that result in controlling, accepting, transferring, or avoiding risks (see text box). DHS Critical Infrastructure Risk Management Framework—Step 4 Organizations should evaluate risk reduction methods by considering countermeasures that result in controlling, accepting, transferring, or avoiding risks. Approaches can include prevention, protection, mitigation, response, and recovery activities. Ideally, the selection and implementation of appropriate risk management activities helps to focus planning, increase coordination, and support effective resource allocation and incident management decisions. We found that the Coast Guard identified thousands of infrastructure vulnerabilities to natural disasters through its Shore Infrastructure Vulnerability Assessment process, and has contracted for more detailed structural analyses of the buildings with vulnerabilities that were deemed inconclusive with respect to seismic and windstorm threats. However, the Coast Guard has not taken action to mitigate risks for those buildings with confirmed vulnerabilities. Our analysis of Phase I results showed that of the 3,214 buildings the Coast Guard analyzed, 32 percent had two or more identified vulnerabilities and an average Mission Dependency Index of 34, and 10 percent had three or more identified vulnerabilities with an average Mission Dependency Index of 38. The average Mission Dependency Index score for all 3,214 buildings was 30. These results indicate that the Coast Guard has data on buildings that may be more vulnerable than others and have relatively greater mission value. Despite the availability of this information, the Coast Guard has not taken steps to develop a mitigation strategy for these buildings, as the DHS Critical Infrastructure Risk Management Framework recommends. Coast Guard officials stated that they had sufficient information from Phase I about how their facilities would perform against eight of the ten disasters, so they elected to further study those buildings with inconclusive results on earthquakes and wind. According to the DHS Critical Infrastructure Risk Management Framework, risk assessments are to inform the selection and implementation of mitigation activities and the establishment of risk management priorities for organizations. Effective risk management activities are comprehensive, coordinated, and cost-effective. The framework further states that risk management decisions should be made based on an analysis of the costs and other impacts, as well as the projected benefits of identified courses of action—including the no-action alternative if a risk is considered to be effectively managed already. However, it is unclear whether and to what extent the civil engineering staff and other decision makers consider the Shore Infrastructure Vulnerability Assessment results as part of the planning board processes where decisions are made about which infrastructure projects will be prioritized for funding. Civil Engineering program officials told us that hazard mitigation strategies will be employed for buildings determined to be vulnerable, as the Coast Guard plans and executes major construction and recapitalization projects through its existing planning board processes. They also provided us with updated planning board guidance, issued in March 2019, which directs Coast Guard officials to consider improving shore infrastructure resilience as a significant factor in the decision-making process. They also noted that the Coast Guard’s updated policy described earlier requires compliance with higher building standards, which helps ensure that newly constructed facilities will be more resilient than the ones they replace. Shore Infrastructure Logistics Center officials, however, were unable to provide us with documentation showing whether and to what extent risk reduction methods were considered during past planning board processes. Furthermore, since they are not required to incorporate Shore Infrastructure Vulnerability Assessment results into future planning board decisions, it is unclear whether future Coast Guard planning boards will be focused on addressing the most critical risks, or will consider resilience as a factor when choosing projects to fund. This is of particular concern since in at least 5 cases, the Coast Guard’s backlog list for Procurement, Construction and Improvement projects includes boat stations that the Coast Guard had previously identified as suitable for closure. According to step five of the DHS Critical Infrastructure Risk Management Framework, organizations should use metrics and other evaluation procedures to measure progress and assess the effectiveness of efforts to secure and strengthen the resilience of critical infrastructure (see text box). DHS Critical Infrastructure Risk Management Framework—Step 5 Organizations should use metrics and other evaluation procedures to measure progress and assess the effectiveness of efforts to secure and strengthen the resilience of critical infrastructure. They are an important step in the critical infrastructure risk management process to enable assessment of improvements in critical infrastructure security and resilience. They provide a basis for accountability, document actual performance, promote effective management and provide a feedback mechanism for informed decision making. We found that the Coast Guard has identified some specific measures, but they are too narrow to measure the agency’s progress or assess the effectiveness of its efforts to improve its shore infrastructure resilience. For example, the Coast Guard established metrics to count the number and dollar value of certain projects to improve resilience, such as seismic improvement or floodplain adaptation projects, that the Civil Engineering program plans and accomplishes each year. While these measures can be useful to account for actions taken and funds invested in these particular actions, they do not indicate whether the resilience of Coast Guard shore infrastructure has improved or is improving as a result of the actions being measured. Coast Guard officials told us that they have not used the DHS Critical Infrastructure Risk Management Framework to guide actions to improve the resilience of its critical infrastructure because they have instead focused on implementing the Shore Infrastructure Vulnerability Assessment to provide them information they intend to use to influence resource investment decisions in the future. However, without a complete understanding of the vulnerabilities of its infrastructure and the consequences to Coast Guard operations if it is damaged, the Coast Guard risks questionable recapitalization investments for its resilience when selecting projects to fund from its $2.6 billion maintenance backlogs. Given that the five steps of the DHS Critical Infrastructure Risk Management Framework are intended to guide decision making and prioritize actions to more effectively achieve desired outcomes, having processes that fully align with the five key steps of the framework would provide greater assurance that the Coast Guard is investing its shore infrastructure resources to manage potential damage and expenses from extreme weather events in the future. The Coast Guard’s shore infrastructure program includes a range of facilities and structures that are vital to the agency’s ability to fulfill its missions, and it constitutes a significant fiscal commitment that requires ongoing investment to maintain. By nature of their mission and location, many facilities and structures are vulnerable to potentially catastrophic natural disasters that are projected to occur more frequently and have required over $2 billion in supplemental funding over recent years to replace or repair. The Coast Guard faces the difficult decision of determining how best to invest its limited resources in improving the resilience of its shore infrastructure to better manage the costs of repairing or replacing such infrastructure after natural hazards occur. DHS’s Critical Infrastructure Risk Management Framework provides a decision making approach that can help ensure risk-informed resource investments, but the Coast Guard has not fully aligned its processes for improving shore infrastructure resilience with any of the five steps outlined in this framework. The Coast Guard’s Shore Infrastructure Vulnerability Assessment process is the agency’s main approach to understanding shore infrastructure vulnerabilities, but this process is limited in scope and not expected to be completed until at least 2025. For the Coast Guard’s planning board processes, officials were unable to verify that they have consistently considered resilience as a significant factor when selecting projects or that they plan to do so in the future. This is of particular concern given the current condition of Coast Guard shore infrastructure and the existing $2.6 billion backlogs of infrastructure maintenance and recapitalization projects that compete for finite funding. By fully aligning its processes with DHS’s recommended risk management framework for critical infrastructure, the Coast Guard would be better positioned to reduce its future fiscal exposure to the effects of catastrophic natural disasters. The Commandant of the Coast Guard should ensure that the Deputy Commandant for Mission Support implements risk management processes that more fully align with the five key steps outlined in DHS’s Critical Infrastructure Risk Management Framework to better guide agency shore infrastructure investment decisions. This should include (1) setting goals and objectives, (2) identifying critical infrastructure, (3) assessing and analyzing risks and costs, (4) implementing risk management activities, and (5) measuring the effectiveness of actions taken. (Recommendation 1) We provided a draft of this report to DHS for review and comment. In its comments, reproduced in appendix I, DHS concurred with our recommendation. DHS, through the Coast Guard, also provided technical comments, which we incorporated as appropriate. DHS concurred with the intent of our recommendation to formalize its shore infrastructure risk management processes, and stated that it plans to make progress towards implementing GAO’s recommendation concurrently with the development and implementation of its Component Resilience Plan, in accordance with the recently mandated DHS Resilience Framework. It intends to complete these efforts by the end of 2021. The Coast Guard also intends to develop, by July 2020, goals and objectives for measuring the effectiveness of actions taken to identify resilience readiness gaps and resource needs. We will continue to monitor these efforts. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or AndersonN@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Nathan J. Anderson, (202) 512-3841 or andersonn@gao.gov. In addition to the contact above, Dawn Hoff (Assistant Director), Landis Lindsey (Analyst-in-Charge), Michael Armes, John Bauckman, Jason Berman, Chuck Bausell, Rick Cederholm, Kendall Childers, John Crawford, Billy Commons, Andrew Curry, Dominick Dale, Elizabeth Dretsch, Shannon Finnegan, Michele Fejfar, Peter Haderlein, Eric Hauswirth, Susan Hsu, Michael Pinkham, John Mingus, and Jan Montgomery, made key contributions to this report.", "summary": "The Coast Guard, within DHS, owns or leases more than 20,000 shore facilities such as piers, boat and air stations, and housing units at over 2,700 locations. This infrastructure is often positioned on coastlines where it is vulnerable to damage from extreme weather. Noting the importance of protecting critical infrastructure from such risks, in 2013 DHS updated its risk management guidance for enhancing infrastructure resilience—which is the ability to prepare and plan for, absorb and recover from, or successfully adapt to adverse events. GAO was asked to review Coast Guard efforts to improve the resilience of its shore infrastructure. This report (1) describes Coast Guard actions to improve shore infrastructure resilience since 2005, and (2) examines the extent to which its processes to improve shore infrastructure resilience follow DHS's key steps for critical infrastructure risk management. GAO reviewed and analyzed Coast Guard guidance and data on assessed infrastructure and interviewed Coast Guard officials. GAO also compared Coast Guard policies, procedures, and actions to manage shore infrastructure against DHS's framework for managing risks to critical infrastructure. Since 2005, the U.S. Coast Guard's main actions to improve resilience have been to repair or rebuild shore infrastructure to higher building standards after it has been damaged by extreme weather events. The Coast Guard has received more than $2 billion in supplemental appropriations since 2005 to improve resilience after severe storms (see figure). The Coast Guard has also developed new guidance requiring that repairs and new construction meet higher building standards to make it more resilient. Further, in 2015, the Coast Guard began an assessment of certain occupied buildings to identify their vulnerabilities to ten natural hazards, such as hurricanes and earthquakes. As of 2018, this assessment covered approximately 16 percent of the Coast Guard's shore infrastructure. The Coast Guard aims to complete the assessment in 2025. Coast Guard processes to improve shore infrastructure resilience do not fully align with the Department of Homeland Security's (DHS) key steps for critical infrastructure risk management. These steps are described in DHS's Critical Infrastructure Risk Management Framework, which recommends that DHS components, among other things, identify critical infrastructure, assess risks, and implement risk management activities. While the Coast Guard has identified some vulnerable shore infrastructure through its ongoing assessment, it has not identified all shore assets that may be vulnerable, such as piers and runways; or assessed operational risks affecting its ability to complete missions with these assets. In addition, the Coast Guard has not taken steps to develop mitigation strategies for buildings already identified as vulnerable. Moreover, Coast Guard data show a growing backlog of at least $2.6 billion in recapitalization, new construction, and deferred maintenance projects that compete for finite funding. However, Coast Guard officials were unable to verify that they have consistently selected projects to also enhance resilience. Coast Guard officials stated that they have not used the DHS framework and have instead focused on implementing their ongoing vulnerability assessment. Fully aligning its processes with the DHS framework would better position the Coast Guard to reduce its future fiscal exposure to the effects of extreme weather events. GAO recommends that the Coast Guard revise its processes for improving shore infrastructure resilience to more fully align with key steps of the DHS critical infrastructure risk management framework. This should include, for example, identifying critical infrastructure, assessing risks, and implementing risk management activities. DHS concurred with our recommendation.", "document_type": "gao"}
{"report": "According to OMB, federal agencies reported that they operated 432 data centers in 1998, 2,094 in July 2010, 5,607 in August 2016, and 5,916 in August 2018. As previously mentioned, operating such a large number of centers has been, and continues to be, a significant cost to federal agencies. For example, in 2007, the Environmental Protection Agency (EPA) estimated that the annual cost for electricity to operate federal servers and data centers across the government was about $450 million. Further, according to the Department of Energy (Energy), a typical government data center has 100 to 200 times the energy use intensity of a commercial building. However, in 2009, OMB reported server utilization rates as low as 5 percent across the federal government’s estimated 150,000 servers. These factors contributed to OMB recognizing the need to establish a coordinated, government-wide effort to improve the efficiency, performance, and environmental footprint of federal data center activities. Subsequently, OMB launched the Federal Data Center Consolidation Initiative in 2010 to reduce the growing number of federal data centers and we have reported extensively on federal agencies’ efforts to implement the initiative’s requirements. Among other things, OMB required agencies to consolidate inefficient infrastructure, optimize existing facilities, improve their security posture, and achieve cost savings. For example, each agency was required to maintain a complete inventory of all data center facilities owned, operated, or maintained by or on its behalf, and measure progress toward defined optimization performance metrics on a quarterly basis as part of its data center inventory submission. Recognizing the importance of reforming the government-wide management of IT, Congress enacted FITARA in December 2014. Among other things, the law required agencies to: Submit to OMB a comprehensive inventory of the data centers owned, operated, or maintained by or on behalf of the agency. Submit, by the end of fiscal year 2016, a multi-year strategy to achieve the consolidation and optimization of the agency’s data centers. The strategy was to include performance metrics that were consistent with the government-wide data center consolidation and optimization metrics. Report progress toward meeting government-wide data center consolidation and optimization metrics on a quarterly basis to OMB’s Administrator of the Office of Electronic Government. In addition, according to FITARA, the Office of Electronic Government at OMB was to: Establish metrics applicable to the consolidation and optimization of data centers (including server efficiency), ensure that information related to agencies’ progress toward meeting government-wide data center consolidation and optimization metrics was made available to the public in a timely manner, review agencies’ inventories and strategies to determine whether they were comprehensive and complete, and monitor the implementation of each agency’s strategy. Develop and make publicly available not later than December 19, 2015, a goal broken down by year for the amount of planned cost savings and optimization improvements that were to be achieved through the FDCCI; and, for each year thereafter until October 1, 2020, compare reported cost savings and optimization improvements against those goals. In August 2016, OMB issued Memorandum M-16-19, which established DCOI and included guidance on how to implement the data center consolidation and optimization provisions of FITARA. The memorandum directed each agency to develop a DCOI strategic plan that defined its data center strategy. Among other things, this strategy was to include a timeline for agency consolidation and optimization activities, with an emphasis on cost savings and optimization performance benchmarks that the agency could achieve between fiscal years 2016 and 2018. For example, each agency was required to develop cost savings targets due to consolidation and optimization actions and report any realized cost savings. OMB required each agency to publicly post its DCOI strategic plan to its agency-owned digital strategy website. In addition, OMB’s memorandum included a series of performance metrics in the areas of data center closures, cost savings, and optimization progress. The guidance further noted that agency progress was to be measured by OMB on a quarterly basis, using agencies’ data center inventory submissions and OMB-defined closures, cost savings, and optimization targets. Further, the memorandum stated that OMB was to maintain a public dashboard (the IT Dashboard) to display government-wide and agency- specific data center consolidation and optimization progress. In this regard, OMB began including such progress information on the IT Dashboard in August 2016. Since the enactment of FITARA in December 2014, we have reviewed and verified the quality and completeness of each covered agency’s inventory and DCOI strategy annually. We have also published reports documenting the findings from each of these reviews. In addition, we have examined and reported on agencies’ efforts to optimize their data centers, as well as the challenges encountered and successes achieved. As of December 2019, 75 of the 117 recommendations from these reports had not been fully addressed. The results and recommendations of our previous reviews are detailed in appendix II. In June 2019, OMB issued a memorandum, M-19-19, that updated DCOI and redefined a data center as a purpose-built, physically separate, dedicated space that meets certain criteria. The memorandum also revised the priorities for consolidating and optimizing the federal data centers. Specifically, OMB directed agencies to focus their efforts on their tiered data centers and to stop reporting on spaces not designed to be data centers (i.e., non-tiered data centers) as part of their inventory. The guidance outlined a process by which agencies could request, and OMB would approve, that these facilities be dropped from reporting. The guidance also noted that OMB would set agency-specific data center closure and cost savings targets in collaboration with each agency and in alignment with that agency’s mission and budget. In addition, OMB described criteria for designating certain data centers as mission critical facilities, which would be exempt from new agency-specific closure targets. Those mission critical designations are to be assumed to be granted unless OMB specifically overturns them. OMB’s revised June 2019 DCOI guidance also directed agencies to stop reporting on spaces not designed to be a data center as part of their inventory, and to focus their efforts on their remaining purpose-built data centers. This is a change from the previous DCOI guidance, which required agencies to report on a much wider range of facilities. OMB’s new memorandum also replaced the previous optimization metrics with revised measures that focused on (1) reporting the number of agencies’ virtualized hosts, underutilized servers, and data centers with advanced energy metering; and (2) the percentage of time that data centers were expected to be available to provide services. In contrast to the previous DCOI guidance, the new memorandum did not specify government-wide performance targets for the optimization metrics, such as setting a target for server utilization of 65 percent for all agencies. Instead, OMB worked with agencies to establish agency-specific targets that were also identified in agency DCOI strategic plans and on the IT Dashboard. In addition, the guidance described how agencies could apply for an optimization performance exemption for data centers where typical optimization activities (consolidation of data collection, storage, and processing to a central location) were technically possible but increased the response time for systems beyond a reasonable limit. As in previous years, the 24 agencies participating in DCOI continued to report progress in closing unneeded data centers and achieving related additional cost savings. The agencies reported closing a total of 102 data centers in fiscal year 2019, as of August 2019, and reported plans to close an additional 184 data centers by the end of fiscal year 2019. According to agencies’ data center inventories, almost all of the 24 agencies met or planned to meet their fiscal year 2019 closure targets. In addition, agencies reported that their DCOI-related activities had either achieved, or planned to achieve, the $241.5 million in total planned savings for fiscal year 2019. However, recent OMB DCOI policy changes will reduce the number of data centers covered by the policy and both OMB and agencies may lose important visibility over the security risks posed by these facilities. For fiscal year 2019, 23 of the 24 agencies reported that they met or planned to meet their fiscal year data center closure targets, as established under OMB’s June 2019 guidance. Of those 23 agencies: three agencies reported that they did not have any agency-owned data centers and had a target of zero closures; these agencies were listed on the IT Dashboard as having completed their closure efforts; five agencies were not expected to close any of their operating data centers during the fiscal year, and their target was zero; 13 agencies reported meeting or exceeding their target closures by two agencies—the Departments of Defense (Defense) and Veterans Affairs (VA)—reported closing a number of data centers and had additional closures planned that were expected to meet their respective fiscal year targets. In addition, one agency—the Office of Personnel Management (OPM)— did not submit a DCOI strategic plan and, consequently, did not report a data center closure target. Table 1 details, for each of the 24 agencies, the number of data centers open at the start of fiscal year 2019, the agency’s fiscal year 2019 closure target, the number of data centers closed, and the number planned for closure during the remainder of the fiscal year, as of August 31, 2019. Agencies reported a total of 102 fiscal year 2019 data center closures through August 31, 2019, with an additional 184 planned closures by the end of that fiscal year. Figure 1 aggregates this information to show agencies’ overall fiscal year 2019 progress against the reported total number of federal data centers. In regard to the remaining data centers, as of August 2019, 12 of the 24 agencies reported plans to close 37 data centers in fiscal year 2020 and beyond. Specifically, 10 agencies reported plans to close 31 additional data centers in fiscal year 2020. Further, two agencies—Energy and the Social Security Administration (SSA)—reported plans to close a total of five data centers in 2021, and one agency—the Department of Homeland Security (DHS)—reported plans to close one data center in 2022. Based on our past work reviewing agencies’ DCOI strategic plans, this total number of planned closures is likely to increase when agencies submit their annual DCOI strategic plans in the spring of 2020. However, the ability to track agencies’ progress against their goals is hampered because the agencies are not reporting their planned and achieved closures on a fiscal year basis, and in one case, the agency had not submitted a plan. As of September 2019, neither the agencies’ strategic plans nor the IT Dashboard provided a specific breakdown of the planned and achieved closures for each fiscal year. OMB’s guidance on DCOI strategic plans only requires reporting cumulative numbers, and staff in OMB’s Office of the Federal CIO confirmed that the IT Dashboard is now intended to report agencies’ cumulative numbers of actual and planned data center closures, rather than numbers broken out by fiscal year. This lack of visibility into exactly how many closures the agencies expect to achieve every fiscal year jeopardizes OMB’s and Congress’ ability to effectively oversee agencies’ data center consolidation efforts. In August 2016, OMB expanded its definition of a data center to include many smaller facilities that OMB cited as consuming significant amounts of resources. Specifically, OMB included rooms with at least one server, providing IT-related services, and categorized data centers into two groups: tiered (which had to meet specific characteristics defined by OMB) and non-tiered. We previously reported that, based on this definition, as of August 2018, the 24 agencies planned to have a total of 4,907 operating data centers at the beginning of fiscal year 2019. However, OMB’s June 2019 revised DCOI reporting requirements further changed the definition of a data center, including no longer requiring agencies to report most of the facilities previously categorized as non- tiered data centers. As noted previously, OMB directed agencies to stop reporting on spaces not designed to be data centers as part of their inventory. As a result, agencies are no longer required to report on about 2,000 facilities, some of which are considerable in size and will continue to operate. Based on OMB’s revised definition of a data center, agencies revised their data center inventory counts and now reported 2,727 operating data centers at the beginning of fiscal year 2019. Specifically, our analysis identified 20 data centers of more than 1,000 square feet that agencies had previously reported as planned for closure, but will not be reported under the current definition. In addition, our analysis found 260 data centers over 1,000 square feet, previously categorized as non-tiered, that agencies plan to continue operating, but which will no longer be reported as part of DCOI. This includes SSA, which plans to no longer report on, but to continue operating, five data centers that are each over 8,000 square feet. Similarly, the Department of State (State) plans to no longer report on, but to keep operating, two facilities that are each at least 10,000 square feet in size. Further, many of the smaller facilities that are now exempt from DCOI reporting represent what OMB has said in the past are the types of data centers that should be included in DCOI because of the risks they posed. Specifically, in its 2016 guidance memorandum, OMB stated that these smaller facilities posed a cybersecurity risk, and consequently, identified them as data centers that needed to be included in consolidation efforts under DCOI. In particular, OMB called out server rooms and closets as security risks that should be targeted for closure. However, while OMB’s 2019 guidance noted the need to address security at these locations and encouraged agencies to continue working to consolidate and optimize them, there is no requirement for agencies to continue to track and report on their progress in closing these smaller facilities. In July 2019, we found that IT systems supporting federal agencies, such as those found in the government’s data centers, are inherently at risk. Specifically, we reported that because these systems can be highly complex and dynamic, technologically diverse, and often geographically dispersed, these factors increase the difficulty of protecting their security. Since each physical location represents a potential access point to an agency’s interconnection with other internal and external systems and networks, each location also poses a risk as a point of potential attack. We also noted that IT systems are often riddled with security vulnerabilities—both known and unknown. Cybersecurity vulnerabilities, such as unsecured access points, can facilitate security incidents and cyberattacks that disrupt critical operations; lead to inappropriate access to and disclosure, modification, or destruction of sensitive information; and threaten national security, economic well-being, and public health and safety. Because of OMB’s decision to remove these types of data centers from DCOI reporting, agencies may lose track of the security vulnerabilities that these facilities present due to the consequent reduction in overall visibility and oversight into all data centers. In its June 2019 guidance, OMB also outlined a process by which agencies could request, and OMB approve, that specific facilities be removed from reporting. As part of this process, agencies were allowed to identify data centers to be removed in one reporting period and then actually remove them in the next, unless OMB provided a written denial within 30 days of the original request. Similarly, agencies could request an exemption for mission critical facilities from their closure target; that request also allows 30 days for OMB to object to the request before an agency should consider the request approved. However, there is currently no documentation of OMB’s decisions on requests to remove specific data centers from reporting, or to exempt the data centers from closure targets because the facility is mission critical. Although an agency’s data center inventory included fields for documenting OMB’s decisions with regard to potential exemptions to optimization, there is no requirement or mechanism to document OMB’s approval that a data center could be dropped from reporting or exempt from closure. There is also no mechanism that would allow a third party to determine whether OMB is providing any denials within the 30 days specified in the DCOI guidance. Staff in OMB’s Office of the Federal CIO acknowledged that someone without access to OMB’s repository of agencies’ data center inventories could not determine whether OMB completed its review within the required time period. We recognize that OMB’s data center definition and reporting revisions are an effort to focus agency closure and optimization efforts on certain types of facilities. However, OMB’s own past guidance has acknowledged the security risks posed by the types of facilities that agencies can now exclude from DCOI. While agencies are best positioned to determine whether these locations should be closed or optimized, it is important that these facilities, previously covered by DCOI, continue to be reported on quarterly, regardless of whether they are subject to closure or optimization. Further, the lack of transparency into OMB’s approval process for removing certain facilities from reporting due to a lack of documentation hinders its ability to understand how and why those decisions are made. This, in turn, jeopardizes OMB’s and Congress’ ability to effectively oversee agencies’ data center consolidation and optimization efforts. Since 2013, federal agencies have been required to report on data center cost savings. In this regard, OMB provided guidance regarding how agencies were to report cost savings and avoidances. Specifically, it required agencies to report both data center consolidation cost savings and avoidances, among other areas, as part of a quarterly data collection process known as the integrated data collection. FITARA also called for each agency to submit a multi-year strategy for achieving the consolidation and optimization of data centers that includes year-by-year calculations of investment and cost savings through fiscal year 2018, which has now been extended to 2020. In addition, OMB’s June 2019 memorandum, M-19-19, noted that agency-specific targets would be set in collaboration with each agency and aligned to that agency’s mission and budget. In their fiscal year 2019 DCOI strategic plans, agencies identified a collective goal of achieving $241.5 million in savings. As of August 2019, the 24 DCOI participating agencies had collectively identified in their quarterly reports to OMB a total of $202.36 million in data center-related cost savings for fiscal year 2019, with an additional $39.14 million expected to be realized in the remaining month of the fiscal year. Specifically, 18 agencies reported that they had met or exceeded their cost savings targets, including seven agencies that did not have a cost savings target and did not report achieving any cost savings. Further, 12 agencies reported plans to achieve about $264 million in data center- related cost savings for fiscal year 2020. Five agencies that had cost savings targets—the Departments of Agriculture (Agriculture), Commerce (Commerce), DHS, and State; and the National Aeronautics and Space Administration (NASA)—reported that they had not yet met their targets, but planned to do so. Additionally, as noted previously, OPM had not submitted its DCOI strategic plan as of August 2019 and, therefore, did not identify cost savings targets for fiscal year 2019 and beyond. Table 2 provides a breakdown of each agency’s planned and achieved cost savings for fiscal year 2019, as of August 2019, and planned savings for fiscal year 2020, according to their DCOI strategic plans and quarterly reporting. Agencies that did not report achieving any cost savings provided a variety of reasons for why they had not done so. For example, officials in the Department of Veterans Affairs’ (VA) Office of the CIO reported 12 data center closures, but said they did not report any achieved cost savings because the majority of those data centers were within multi-use facilities that were still owned and maintained by the agency. However, according to VA’s DCOI strategic plan, the agency plans to achieve cost savings in fiscal year 2020 because it expects to stop leasing two data centers, which is expected to reduce data center spending. In addition, officials from three agencies—the Department of Housing and Urban Development (HUD), the General Services Administration (GSA), and the United States Agency for International Development (USAID)— reported that they did not have any agency-owned data centers and had limited opportunity to achieve cost savings related to closing and optimizing their data centers. According to OPM officials, the agency did not have a savings target due to the lack of a fiscal year 2019 DCOI strategic plan, which was attributed by the officials to an oversight that resulted from changes in OPM CIO leadership at the time the plan was due. The officials reported that the agency continued to execute on a plan that was already in place and they did not anticipate any meaningful changes in the agency’s DCOI strategy for 2020. The officials said they expect OPM to submit its fiscal year 2020 strategic plan on time in April 2020. Overall, the 24 participating DCOI agencies have reported a total of $4.7 billion in cost savings and avoidances from fiscal years 2012 through 2019. We have previously stressed that identifying and reporting the savings resulting from agencies’ data center consolidations was an important indicator for monitoring the progress of DCOI. Until OPM submits a plan that identifies its cost savings targets to OMB, the agency’s ability to plan how to achieve DCOI’s expected benefits will be limited. In addition, until the five agencies that still expect to achieve savings establish and meet their cost savings targets, DCOI may not deliver the expected financial benefits. FITARA required OMB to establish metrics to measure the optimization of data centers, including server efficiency, and to ensure that agencies’ progress toward meeting those metrics is made available to the public. Pursuant to this requirement, OMB has used several different sets of performance measures that have changed over time. Most recently, and as previously noted, OMB issued revised DCOI guidance in June 2019 that defined a set of three revised and one new data center optimization metrics to replace the five previous metrics. According to the OMB memorandum that published these changes, the current metrics were intended to focus optimization efforts in key areas where agencies can make meaningful improvements and achieve further cost savings through optimization. Table 3 provides a description of the four data center optimization metrics and how each metric is to be calculated. According to the June 2019 revised DCOI guidance, agencies are to focus their optimization efforts on their remaining open, agency-owned, tiered data centers. OMB also included in the guidance its plans to work with the agencies to set agency-specific optimization performance targets for each fiscal year. According to staff in OMB’s Office of the Federal CIO, these targets are to be established by fiscal year and progress toward meeting them is expected to be provided via the IT Dashboard. For three of OMB’s June 2019 optimization metrics, 19 of the 24 DCOI agencies reported progress in meeting OMB’s fiscal year 2019 data center optimization targets identified on the IT Dashboard. Specifically, as of September 2019: 11 reported that they had met their target for virtualization, 11 reported that they had met their advanced metering target, and 18 reported that they had met their server utilization target. Of the remaining five agencies, OPM had not submitted a DCOI strategic plan as of September 2019 and consequently, did not have established optimization targets or a basis to measure and report optimization progress. The remaining four agencies—the Department of Education (Education), HUD, GSA, and USAID—reported that they did not have any agency-owned data centers in their inventory and, therefore, the optimization metrics were not applicable. In addition, Justice had not established a target for the server utilization metric and, therefore, did not have a basis to measure and report progress. Figure 2 summarizes the DCOI agencies’ progress in meeting each optimization target, as of September 2019. Of the 19 agencies with a basis to report against OMB’s optimization targets, eight agencies—Energy, DHS, the Department of the Interior, State, NASA, NSF, NRC, and SSA—reported meeting three targets as of September 2019. Also, five agencies reported that they had met two targets, and six agencies reported meeting one target. Table 4 lists the DCOI agencies and their status on meeting their OMB optimization performance targets. Of the current DCOI metrics, as shown in table 4, agencies reported greater success in meeting their agency-specific optimization targets than we had reported in our previous reviews, as detailed in appendix II. As of September 2019, the IT Dashboard reported that four agencies had fully completed their overall DCOI optimization efforts for all of their data centers and had no further work to do. The IT Dashboard further reported that another four agencies had met their optimization targets for fiscal year 2019. However, eight agencies had not met their fiscal year 2019 virtualization target. The reasons agencies provided for not meeting the target varied. For example, officials in the Department of Agriculture’s Office of the CIO reported that the department did not meet the virtualization target because the closure date for one of its data centers was moved to fiscal year 2020, which resulted in fewer virtualized hosts for 2019 under OMB’s new definition. Additionally, although EPA did not meet its virtualization target, its DCOI strategic plan described the agency’s intention to meet its goals by expanding its virtualization strategy agency-wide, which would increase the agency’s virtualization performance. In addition, OMB required agencies to report the number of agency- owned data centers with advanced energy metering. As of September 2019, of the 19 agencies with the basis to report, eight reported that they did not reach their target for having such metering in their data centers. For example, officials at the Department of Veteran Affairs reported that they did not meet their advanced energy metering target due to difficulties in getting a contract in place to install the metering. Further, for the new availability metric, there were unexpected variances in how agencies reported information—thus rendering the data for this metric unreliable. Specifically, according to OMB’s quarterly reporting instructions, agencies were to report the number of hours, in the 3-month reporting period, that each data center was expected to be available to provide services. However, several agencies reported information based on annual, instead of quarterly, calculations. In addition, Department of Agriculture officials stated that, for one data center, they reported the total number of availability hours for multiple instances where they provided data center services to other agencies. Based on the various instances of erroneous agency reporting that we identified, we determined that the data for this metric was not sufficiently reliable for us to use. When the problems with these data were brought to agencies’ attention, many agreed that their reporting needed to be updated; in some cases, the agencies updated their information, but not in time for it to be analyzed and addressed in this report. Based on our discussions with agencies, we will continue to monitor their progress in improving the accuracy of their reporting for this metric through our follow-up efforts for this report, as well as our future mandated reviews of DCOI progress. Additionally, and as mentioned previously, Justice had not established a target for server utilization. Officials in the department’s Justice Management Division stated that this was due to OMB’s issuing the revised DCOI guidance and metrics in June 2019. Once they can track server utilization for a few reporting periods, the officials stated that the agency will finalize its definition for underutilized severs and establish an appropriate target for the metric. Overall, while agencies reported more success in meeting the current optimization metrics, most agencies did not meet all of their metric targets for fiscal year 2019. Until these agencies take the steps necessary to meet their optimization targets, it is unlikely that these agencies will achieve the expected benefits of optimization and the resulting cost savings. Given that our April 2019 report included recommendations for all of the agencies except Commerce that missed an optimization target to take action to meet the data center optimization metric targets established under DCOI by OMB, we are not making new optimization- related recommendations to those agencies. GAO’s Green Book provides the standards for internal control in the federal government and an overall framework for establishing and maintaining an effective internal control system. Such a control system addresses, in part, the attainment of a federal entity’s objectives, which is accomplished through monitoring specific performance measures. Such monitoring is also expected to assess the quality of performance over time. In addition, the Green Book discusses the importance of clearly defining an entity’s objectives in order to determine what is trying to be achieved and to establish related performance measures. According to the Green Book, the controls represented by an agency’s performance metrics should include these key characteristics. The controls should be: Clearly defined in measurable terms that are easily understood. Objective and free of bias, rather than subjective. Defined by appropriate parameters that allow for evaluating performance. Understood by all levels of the organization, including what is being achieved with the metric, who is primarily responsible for achieving the metric, how the metric will be achieved, and when the metric will be achieved. Aligned with internal and external requirements, including applicable legislation, regulations, and standards. We found that all four of OMB’s current optimization performance metrics met three of these five characteristics—that is, each was clearly defined, objectively measurable, and aligned with internal and external requirements. However, the performance metrics did not fully meet the two other characteristics—namely they did not include appropriate performance parameters and did not fully include all the information that would allow them to be understood at all levels of the organization. Table 5 provides our assessment of the extent to which the OMB metrics aligned with the characteristics of an effective metric. In addition, appendix III provides additional detail of our assessment of the characteristics of each metric. While all four of OMB’s metrics met three of the five characteristics of an effective metric, none of the metrics addressed the fourth characteristic of providing appropriate performance parameters. Specifically, none of the metrics included statistical universe parameters that would enable a determination of progress against goals. For example, the virtualization metric requires an agency to report the number of its virtual hosts, but does not relate that to the overall number of servers and mainframes at the agency. As a result, the metric does not indicate whether an agency’s reported number of virtual hosts is almost all of that agency’s servers and mainframes, or very few. Similarly, the server utilization metric identifies how many underutilized servers an agency has, but does not give the context of how that relates to the agency’s total population of servers. In both these cases, percentages cannot be calculated to determine progress. For instance, while the number of an agency’s virtualized servers may increase, if the universe of servers were to increase at a higher rate, then progress would actually be negative. In the June 2019 DCOI revised guidance, OMB acknowledged removing targeted averages for its metric targets. However, by doing so, OMB also removed important information that provided a relative sense of the progress indicated by the data. Further, the lack of performance parameters in defining the metrics had an impact on OMB’s public reporting of agencies’ progress. The IT Dashboard displays agencies’ consolidation and progress information through a DCOI Optimization Summary that displays data about the number of agency data center facilities, achieved and planned closures, achieved and planned IT cost savings, and progress of the current performance metrics against the related targets. However, the IT Dashboard does not provide important information, such as in which fiscal year the targets are to be achieved and how the metric information being reported relates to an agency’s operations. For example, the IT Dashboard reports the number of servers and mainframes serving as virtual hosts in agency-managed data centers, but does not provide the total number of servers and mainframes to give the context of how well agencies are managing the number of their virtual hosts. Staff in OMB’s Office of the Federal CIO stated that the lack of performance parameters for the metrics is due to OMB and the agencies needing time to collect baseline data before making changes to the metrics. However, until OMB addresses missing information from the optimization metric definitions, the metrics will lack important and meaningful information about agencies’ DCOI performance that would assist OMB and Congress in their oversight roles. In addition, unless OMB takes action to update the metrics’ definitions to include missing key metric characteristics, agencies’ reporting may not provide an accurate view of their data center optimization progress. Further, without this information on the IT Dashboard, Congress lacks the information needed to inform its decision making and oversight responsibilities. Federal data center consolidation efforts have been underway since 2010, and agencies continue to report progress towards meeting their goals for data center closures and achieving related savings. Specifically, almost all of the 24 DCOI agencies met, or planned to meet, their goals for data center closures in fiscal year 2019. Additionally, in fiscal year 2019, almost all of the agencies met or planned to meet their $249 million total savings target. Agencies’ efforts in both respects have made an important contribution to achieving the overall goals of DCOI. However, agencies’ annual closure goals are not currently reported in their DCOI strategic plans or tracked on the IT Dashboard, requiring us to manually calculate those targets. Unless agencies’ annual closure goals are fully reported and tracked, oversight of DCOI will be hampered. Further, the six agencies without plans to meet their fiscal year data center closure or cost savings targets will continue to be challenged to realize the full benefits of DCOI. As part of the 2019 changes to DCOI, OMB significantly reduced the scope of what is considered a data center, and, in doing so, excluded about 2,000 smaller facilities that were previously reported by agencies in 2018. While OMB previously acknowledged that these types of facilities inefficiently consume resources and pose security risks, agencies are no longer required to report these locations in their inventories. Further, there is currently no documentation of OMB’s decisions on agency requests to remove data centers from reporting, or to exempt mission critical data centers from closure targets. By no longer reporting key facilities as part of DCOI and by not documenting decisions on which facilities are exempt from DCOI, oversight of agencies’ consolidation and optimization efforts may be impaired, and agencies may remain exposed to the related vulnerabilities. Agencies’ progress against OMB’s three revised metrics was mixed, and, for one new metric, agencies reported data that varied so widely, we concluded the data for this metric were not sufficiently reliable for us to report on. However, in comparing OMB’s four metrics against the characteristics of an effective metric, we most notably found that none of the metrics included appropriate performance parameters for evaluating agencies’ progress against goals. Metrics that include more robust and informative agency performance data can play an important role in both achieving the optimization goals and mission of DCOI and allowing for stronger oversight of those efforts. In addition to reiterating our prior open recommendations to the agencies in our review regarding their need to meet DCOI’s closure and savings goals and optimization metrics, we are making a total of eight new recommendations—four to OMB and four to three of the 24 agencies. Specifically: The Director of the Office of Management and Budget should (1) require that agencies explicitly document annual data center closure goals in their DCOI strategic plans and (2) track those goals on the IT Dashboard. (Recommendation 1) The Director of the Office of Management and Budget should require agencies to report in their quarterly inventory submissions those facilities previously reported as data centers, even if those facilities are not subject to the closure and optimization requirements of DCOI. (Recommendation 2) The Director of the Office of Management and Budget should document OMB’s decisions on whether to approve individual data centers when designated by agencies as either a mission critical facility or as a facility not subject to DCOI. (Recommendation 3) The Director of the Office of Management and Budget should take action to address the key performance measurement characteristics missing from the DCOI optimization metrics, as identified in this report. (Recommendation 4) The Secretary of Agriculture should take action to achieve its data center- related cost savings target established under DCOI by OMB. (Recommendation 5) The Secretary of Commerce should take action to achieve its data center- related cost savings target established under DCOI by OMB. (Recommendation 6) The Secretary of Commerce should take action to meet its data center optimization metric targets established under DCOI by OMB. (Recommendation 7) The Administrator of the National Aeronautics and Space Administration should take action to achieve its data center-related cost savings target established under DCOI by OMB. (Recommendation 8) We provided a draft of this report to OMB and the 24 agencies for their review and comment. In response, of the seven agencies to which we made recommendations, five agencies stated that they agreed with the recommendations and two agencies did not state whether they agreed or disagreed with the recommendations. In addition, of the 18 agencies to which we did not make recommendations, three agencies stated that they concurred with the information presented in the report, three other agencies did not state whether they agreed or disagreed with the report, and 12 agencies stated that they had no comments on the report. Further, four agencies provided technical comments on the report, which we incorporated as appropriate. Of the agencies to which we made recommendations, five agreed with the recommendations. In an email, a Director for Strategic Planning, Egovernment, and Audits in the Office of the CIO at Agriculture stated that the department agreed with our recommendation to achieve its data center-related cost savings target established under DCOI and that it planned to meet the cost savings target in 2020. Agriculture also included technical comments, which we have incorporated as appropriate. In written comments, Commerce agreed with our recommendations to achieve its data center-related cost savings target established under DCOI and to meet its data center optimization metric targets established under DCOI by OMB. The department also described actions that they planned to take in order to address the recommendations. Commerce’s comments are reprinted in appendix IV. In written comments, DHS agreed with our recommendation to achieve its data center-related cost savings target established under DCOI. Further, the department stated that, in its November 2019 DCOI data submission, it reported $354.97 million in cumulative DCOI cost savings through fiscal year 2019. Subsequent to reviewing our draft report, the department provided documentation of the savings claimed in their response. In reviewing this data, we confirmed that these cumulative savings included the $33.8 million savings the department had planned for fiscal year 2019. As a result, we consider our recommendation to have been addressed and therefore removed it from the final report. DHS also provided technical comments, which we have incorporated as appropriate. DHS's comments are reprinted in appendix V. In written comments, NASA agreed with our recommendation to achieve its data center-related cost savings target established under DCOI and described actions that the agency planned to take to address the recommendation. NASA stated that it expects to complete these actions by March 31, 2020. NASA's comments are reprinted in appendix VI. In written comments, OPM agreed with our recommendation to develop and submit to OMB a complete DCOI strategic plan. Subsequent to reviewing our draft report, OPM informed us that the agency had published its fiscal year 2019 plan, and that the agency was on track to meet the OMB reporting deadline for fiscal year 2020. We confirmed that OPM's fiscal year 2019 strategic plan was published and publicly available through the agency's website. As a result, we consider our recommendation to have been addressed and therefore removed it from the final report. OPM's comments are reprinted in appendix VII. In addition, two agencies did not state whether they agreed or disagreed with their recommendations. In an email, a GAO liaison on OMB’s Ethics Team provided an annotated copy of our draft report. In OMB’s comments in that copy of the draft, OMB did not agree or disagree with our recommendations. However, OMB took issue with the report’s findings that the removal of facilities from DCOI oversight posed cybersecurity-related risks represented by those facilities. OMB’s comments further recommended that we remove references to cybersecurity from our report’s title and from the body of the report. In raising these objections, OMB’s comments stated that DCOI is focused on consolidating and optimizing the federal data center portfolio and that cybersecurity is not a primary driver of the initiative. OMB added that DCOI was never designed to track or directly address cybersecurity risks. Specifically, OMB’s comments took issue with our finding that data centers not tracked within DCOI are at a greater risk for a cybersecurity incident. These comments noted that many other laws, policies, and procedures directly deal with the cybersecurity posture of all federal IT systems, and that OMB’s DCOI guidance does not affect the applicability of those requirements. The comments also acknowledged that, while past DCOI guidance has stated that the reduction of data centers may improve the cybersecurity posture of federal agencies, this was because agency CIOs could better allocate constrained resources across a smaller portfolio of devices. We agree that agencies are subject to numerous cybersecurity requirements external to DCOI. We also agree that a reduced portfolio of data centers may improve the cybersecurity of an agency. However, our report focuses on OMB’s recent DCOI policy changes that allow agencies to stop tracking and reporting on over 2,000 data centers. In this discussion, we cite our July 2019 report which found that, facilities such as these, represent a potential access point to an agency’s systems and networks and pose a risk as points of potential attack. OMB’s policy changes do not require agencies to continue to close these points of access, nor do they yield the smaller portfolio of devices that OMB referenced in its comments on our draft report. Our report notes that OMB’s policy change to remove those data centers from DCOI reporting may contribute to agencies losing track of the security vulnerabilities that those facilities present because DCOI has provided a mechanism for ongoing visibility and oversight of these facilities separate from the federal government’s cybersecurity framework. As such, we maintain our report accurately characterizes the increased potential for cybersecurity risk that could be posed by these now-unreported physical locations. We also affirm that our related recommendation to OMB to require agencies to report in their quarterly inventory submissions, those facilities previously reported as data centers, even if those facilities are not subject to the closure and optimization requirements of DCOI, is still appropriate. In written comments, State did not say whether it agreed or disagreed with our recommendation to achieve its data center-related cost savings target established under DCOI by OMB. Subsequent to reviewing our draft report, the department informed us of $61.1 million in fiscal year 2019 optimization and consolidation cost savings and avoidances, an amount in excess of its $58.9 million fiscal year 2019 target, and provided documentation to support this claim. The department also stated that this information would be reported in the department's annual DCOI strategic plan update in the second quarter of fiscal year 2020. In reviewing the documentation provided by the department, we confirmed State’s reported $61.1 million in fiscal year 2019 savings. As a result, we consider our recommendation to have been addressed and therefore removed it from the final report. State's comments are reprinted in appendix VIII. Further, of the 18 agencies to which we did not make recommendations, three agencies agreed with the information presented in the report. Via emails, audit liaisons in the Office of the CIO at Justice, the Office of the Assistant Secretary for Policy at Labor, and the Office of Congressional and Legislative Affairs at VA agreed with the findings in the draft report. In addition, three agencies did not state whether they agreed or disagreed with the report. In written responses, Defense and USAID did not state whether they agreed or disagreed with the draft report. The agencies' responses are reprinted in appendices IX and X respectively. In an email, an audit liaison in the OIG-GAO Audit Liaison Office at Interior did not state whether the department agreed or disagreed with the draft report. The department also provided technical comments, which we have incorporated as appropriate. Finally, 12 agencies stated that they had no comment on the report. In written responses, HUD and SSA stated that they had no comments on the draft report. The agencies' responses are reprinted in appendices XI and XII respectively. We also received emails from officials of Education, Energy, HHS, Transportation, Treasury, EPA, GSA, NSF, NRC, and SBA, which stated that the agencies had no comment on the report. EPA also provided technical comments, which we have incorporated as appropriate. We are sending copies of this report to interested congressional committees, the Director of OMB, the secretaries and heads of the departments and agencies addressed in this report, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XIII. This report addresses (1) agencies’ progress on data center closures and the related savings that have been achieved, and agencies’ plans for future closures and savings and (2) agencies’ progress against the Office of Management and Budget’s (OMB) data center optimization targets. To address the first objective, for data center closures, we obtained and analyzed August 2019 data center inventory documentation from the 24 departments and agencies (agencies) that participate in OMB’s Data Center Optimization Initiative (DCOI). To determine data center closures to date, we totaled their reported closures for fiscal year 2019 through August 31, 2019, and, to identify future closures, we totaled their reported planned closures for fiscal years 2019 through 2022. We also compared agencies’ completed and planned closures to the planned fiscal year 2019 consolidation goals, as documented in their DCOI strategic plans. OMB’s guidance for developing agencies’ DCOI strategic plans required agencies to report cumulative numbers for their planned and achieved data center closures; as a result, we calculated agencies’ fiscal year 2019 targets from the data reported in DCOI plans. To verify the quality, completeness, and reliability of each agency’s data center inventory, we compared information on completed and planned data center closures to similar information reported on OMB’s IT Dashboard—a public website that provides information on federal agencies’ major IT investments. We also checked for missing data and other errors, such as missing closure status information. In some cases identified, we followed up with agency officials to obtain further information. We determined that the data were sufficiently complete and reliable to report on their consolidation progress and planned closures. For cost savings and avoidances, we obtained and analyzed documentation from the 24 DCOI agencies. This documentation is required by OMB’s March 2013, August 2016, and June 2019 memorandums and included the agencies’ quarterly reports of cost savings and avoidances posted to their digital services websites and their DCOI strategic plans. To determine cost savings achieved, we totaled agencies’ reported savings and avoidances from the start of fiscal year 2012 through August 2019, as found in the August 2019 quarterly reports posted to the agencies’ digital services websites. To identify future planned savings, we totaled the agencies’ projected savings and avoidances from fiscal years 2019 through 2020, as reported in their DCOI strategic plans. To assess the quality, completeness, and reliability of each agency’s data center consolidation cost savings information, we used the latest version of each agency’s quarterly cost savings report and DCOI strategic plan as of August 31, 2019. We also reviewed the quarterly reports and DCOI strategic plans for errors and missing data, such as missing cost-savings information. In addition, we compared agencies’ cost savings and avoidances with data from our most recent data center consolidation report. Further, we obtained written responses from agency officials regarding the steps they took to ensure the accuracy and reliability of their cost savings data. As a result, we determined that the data were sufficiently complete and reliable to report on agencies’ data center consolidation cost-savings information. For our second objective, we analyzed the September 2019 data center optimization progress information of the 20 DCOI agencies. This progress information was obtained from the IT Dashboard. We then compared the agencies’ current optimization progress information to agencies’ fiscal year 2019 optimization targets, as documented on the IT Dashboard. In addition, to assess the reliability of agencies’ optimization progress information on OMB’s IT Dashboard, we reviewed the information for errors or missing data, such as progress information that was not available for certain metrics. We also compared agencies’ optimization progress information across two reporting quarters to identify any inconsistencies in agencies’ reported progress. We also followed up with the agencies to understand the steps they took to insure that what they reported to OMB was accurate and reliable. We determined that the data were sufficiently complete and reliable to report on agencies’ progress information for virtualization, advanced energy metering, and server utilization. However, for the fourth metric—data center availability—our analysis identified variances in how agencies reported their data. According to OMB’s quarterly reporting instructions, agencies were to report the number of hours, in the 3-month reporting period, that each data center was expected to be available to provide services. Instead, several agencies reported information based on annual, instead of quarterly, calculations. In addition, Department of Agriculture officials stated that, for one data center, they reported the total number of availability hours for multiple instances where they provided data center services to other agencies. Because of these variances and the impact they had on the reported information, we determined that the availability metric data were insufficiently reliable to report on agencies’ progress. To assess whether OMB’s new performance metrics met key characteristics of an effective performance measure, we adapted principles from the Green Book that described characteristics of effective performance measures. The Green Book provides an overall framework for establishing and maintaining an effective internal control system that includes monitoring through performance measures. We then compared each OMB optimization performance metric, as defined in the revised DCOI guidance and reported on OMB’s IT Dashboard, to the criteria we identified from the Green Book to determine the extent to which each metric met each characteristic. We conducted this performance audit from April 2019 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Since the enactment of FITARA in December 2014, we have reviewed and verified the quality and completeness of each covered agency’s inventory and Data Center Optimization Initiative (DCOI) strategy annually. Accordingly, we have published reports documenting the findings and recommendations from each of these reviews. In addition, we have examined and reported on agencies’ efforts to optimize their data centers, as well as the challenges encountered and successes achieved. As of December 2019, 75 of the 117 recommendations from these reports had not been fully implemented. In a report that we issued in March 2016, we noted that agencies had reported significant data center closures—totaling more than 3,100 through fiscal year 2015—but fell short of the Office of Management and Budget’s (OMB) fiscal year 2015 consolidation goal. Agencies also reported significant consolidation cost savings and avoidances—totaling about $2.8 billion through fiscal year 2015. However, we pointed out that many agencies lacked complete cost savings goals for the next several years despite having closures planned. In addition, we reported that 22 agencies had made limited progress against OMB’s fiscal year 2015 data center optimization performance metrics, such as the utilization of data center facilities. Accordingly, we recommended that the agencies take actions to complete their cost savings targets and improve optimization progress. As of December 2019, 17 of the 32 recommendations from this report had yet to be fully addressed. In May 2017, we reported that the agencies continued to report significant data center closures—totaling more than 4,300 through August 2016— with more than 1,200 additional centers planned for closure through fiscal year 2019. The agencies also reported achieving about $2.3 billion in cost savings through August 2016. However, agencies’ total planned cost savings for fiscal years 2016 through 2018 were more than $2 billion less than OMB’s fiscal year 2018 cost savings goal of $2.7 billion. In addition, our May 2017 report identified weaknesses in agencies’ DCOI strategic plans. Of the 23 agencies that had submitted their strategic plans at the time of our review, seven had addressed all of the five required elements of a strategic plan, as identified by OMB (such as providing information related to data center closures and cost savings metrics). The remaining 16 agencies that submitted their plans either partially met or did not meet the requirements. Given these findings, we recommended that OMB improve its oversight of agencies’ DCOI strategic plans and their reporting of cost savings and avoidances. We also recommended that 17 agencies complete the missing elements in their strategic plans, and that 11 agencies ensure the reporting of consistent cost savings and avoidance information to OMB. As of December 2019, five of the 30 recommendations had not been fully addressed. In a subsequent report that we issued in August 2017, we noted that 22 of the 24 agencies required to participate in the OMB DCOI had collectively reported limited progress against OMB’s fiscal year 2018 performance targets for the five optimization metrics. Specifically, for each of the five targets, no more than five agencies reported that they had met or exceeded that specific target. In addition, we noted in the report that most agencies had not implemented automated monitoring tools to measure server utilization, as required by the end of fiscal year 2018. Specifically, four agencies reported that they had fully implemented such tools and 18 reported that they had not done so. Two agencies did not have a basis to report on progress because they did not have any agency-owned data centers. Accordingly, we recommended that OMB formally document a requirement for agencies to include plans, as part of existing OMB reporting mechanisms, to implement automated monitoring tools at their agency-owned data centers. We also recommended that the 18 agencies without fully documented plans take action within existing OMB reporting mechanisms to complete plans describing how they intended to achieve OMB’s requirement to implement automated monitoring tools at all agency-owned data centers by the end of fiscal year 2018. As of December 2019, two of the 19 recommendations had been fully addressed. In May 2018, we noted that the 24 agencies participating in DCOI reported mixed progress toward achieving OMB’s goals for closing data centers by September 2018. Over half of the agencies reported that they had either already met, or planned to meet, all of their OMB-assigned closure goals by the deadline. However, four agencies reported that they did not have plans to meet all of their assigned goals and two agencies were working with OMB to establish revised targets. With regard to agencies’ progress in achieving cost savings, 20 agencies reported planned and achieved savings that totaled $1.62 billion for fiscal years 2016 through 2018. However, this total was approximately $1.12 billion less than OMB’s DCOI savings goal of $2.7 billion. In addition, the 24 agencies continued to report limited progress against OMB’s five data center optimization targets, with one agency meeting four targets, one meeting three targets, six meeting either one or two targets, and 14 meeting none of their targets. Further, as of August 2017, most agencies were not planning to meet OMB’s fiscal year 2018 optimization targets. Because we had previously made a number of recommendations to OMB and the 24 DCOI agencies to help improve the reporting of data center- related cost savings and to achieve optimization targets, we did not make new recommendations in our May 2018 report, but indicated that we would continue to monitor the agencies’ progress toward meeting OMB’s DCOI goals. Most recently, in April 2019, we reported that the 24 DCOI agencies continued to report mixed progress toward achieving OMB’s goals for closing data centers and realizing the associated savings by September 2018. Thirteen agencies reported that they had met, or had plans to meet, all of their OMB-assigned closure goals by the deadline. However, 11 agencies reported that they did not have plans to meet their goals. In addition, 16 agencies reported that they had met, or planned to meet, their cost savings targets, for a total of $2.36 billion in cost savings for fiscal years 2016 through 2018. This is about $0.38 billion less than OMB’s DCOI savings goal of $2.7 billion. This shortfall is the result of five agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans as compared to the savings targets established for them by OMB. Three agencies did not have a cost savings target and did not report any achieved savings. Regarding data center optimization, the 24 agencies reported limited progress in fiscal year 2018 against OMB’s five optimization targets. In this regard, 12 agencies reported that they had met at least one target, while 10 reported that they had not met any of the targets. Two agencies stated that they did not have a basis to report on progress as they did not own any data centers. Further, 20 agencies did not plan to meet all of OMB’s fiscal year 2018 optimization goals. Specifically, only two agencies reported plans to meet all applicable targets, while six reported that they did not plan to meet any of the targets. As a result of these findings, we recommended that 22 agencies take actions to meet the data center closure, cost savings, and optimization performance metrics targets, as appropriate. As of December 2019, none of the 36 recommendations had been fully addressed. As noted previously in this report, the Office of Management and Budget (OMB) issued revised Data Center Optimization Initiative (DCOI) performance metrics in June 2019 as part of its revised DCOI guidance. According to OMB, the four current data center optimization metrics were intended to focus targeted improvements in key areas where agencies can make meaningful improvements and achieve further cost savings through optimization. OMB’s intent was to avoid using averages for metrics and instead identify metrics where agencies could demonstrate continuous improvement beyond the performance period of the June 2019 memorandum. OMB stated this would provide a more accurate measure of the agencies’ data center performance. GAO published the Green Book, which provides the standards for internal control in the federal government and an overall framework for establishing and maintaining an effective internal control system. Such a control system addresses, in part, the attainment of a federal entity’s objectives, which is accomplished through monitoring specific performance measures. Such monitoring is also expected to assess the quality of performance over time. In addition, the Green Book discusses the importance of clearly defining an entity’s objectives in order to determine what is to be achieved and to establish related performance measures. According to the Green Book, the controls represented by an agency’s performance metrics should include several key characteristics. Clearly defined in measurable terms that are easily understood. Objective and free of bias, rather than subjective. Defined by appropriate parameters that allow for evaluating performance. Understood by all levels of the organization, including what is being achieved with the metric, who is primarily responsible for achieving the metric, how the metric will be achieved, and when the metric will be achieved. Aligned with internal and external requirements, including applicable legislation, regulations, and standards. We compared each OMB optimization performance metric, as defined in the revised DCOI guidance and reported on OMB’s IT Dashboard, to the key effective metric characteristics identified in the Green Book. In assessing each of the OMB metrics against the key characteristics, we assigned one of three categories: Met. The metric definition aligned with the characteristics of an effective metric. Partially met. The metric definition aligned with some, but not all, the characteristics of an effective metric. Not met. The metric definition did not align with the effective metric characteristics. OMB’s virtualization metric counted the number of servers and mainframes serving as a virtual host in an agency-managed data center. We found that the virtualization metric met three characteristics, met two of four parts of one characteristic, and didn’t meet one. Table 6 provides our evaluation of the extent to which this OMB metric aligns with key characteristics of an effective metric. OMB’s advanced energy metering metric counted the data centers with advanced energy metering covering the majority of their floor space. We found that the advanced energy metering metric met two characteristics, met three of four parts of one characteristic, and did not meet two. Table 7 provides our evaluation of the extent to which this OMB metric aligned with key characteristics of an effective metric. OMB’s server utilization metric counts the number of underutilized production servers in federal data centers. We found that the underutilized servers metric met three characteristics, met two of four parts of one characteristic, and did not meet one. Table 8 provides our evaluation of the extent to which this OMB metric aligned with key characteristics of an effective metric. OMB’s data center availability metric calculated the ratio of uptime (when the data center services were available) to unexpected downtime (unplanned service outages) in data centers. We found that the data center availability metric met two characteristics, met two of four parts of one characteristic, and did not meet two. Table 9 provides our evaluation of the extent to which the OMB metric aligned with key characteristics of an effective metric. In addition to the contact named above, individuals making contributions to this report included Dave Hinchman (Assistant Director), Justin Booth (Analyst-in-Charge), Lamis Alabed, Chris Businsky, Nancy Glover, Gina Hoover, and Jonathan Wall.", "summary": "In December 2014, Congress enacted federal IT acquisition reform legislation that included provisions related to ongoing federal data center consolidation efforts. OMB's Federal Chief Information Officer launched DCOI to build on prior data center consolidation efforts; improve federal data centers' performance; and establish goals for inventory closures, cost savings and avoidances, and optimization performance. The 2014 legislation included a provision for GAO to annually review agencies' data center inventories and strategies. This report addresses (1) agencies' progress and plans for data center closures and savings; and (2) agencies' progress against OMB's June 2019 revised data center optimization metrics. To do so, GAO assessed the 24 DCOI agencies' data center inventories as of August 2019, reviewed their reported cost savings documentation, evaluated their data center optimization strategic plans, and assessed their progress against OMB's established optimization targets. GAO also compared OMB's revised metrics to key characteristics of an effective performance measure. The 24 agencies participating in the Office of Management and Budget's (OMB) Data Center Optimization Initiative (DCOI) reported progress toward achieving OMB's fiscal year 2019 goals for closing unneeded data centers. As of August 2019, 23 of the 24 reported that they had met, or planned to meet, their fiscal year closure goals, and would close 286 facilities in doing so (see figure). Agencies also reported plans to close at least 37 of the remaining data centers. OMB issued revised guidance in June 2019 that narrowed the scope of the type of facilities that would be defined as a data center. This revision eliminated the reporting of over 2,000 facilities government-wide. OMB had previously cited cybersecurity risks for these types of facilities. Without a requirement to report on these, important visibility is diminished, including oversight of security risks. The 24 DCOI agencies have reported a total of $4.7 billion in cost savings from fiscal years 2012 through 2019. Of the 24 agencies, 23 reported in August 2019 they had met, or planned to meet, OMB's fiscal year 2019 savings goal of $241.5 million. One agency did not complete a plan, but planned to do so in the future. Agencies also reported plans to save about $264 million in fiscal year 2020. The 24 agencies reported progress against OMB's three revised data center optimization metrics for virtualization, advanced energy monitoring, and server utilization. For a new fourth metric (availability), the data were not sufficiently reliable to report on because of unexpected variances in the information reported by the agencies. As of August 2019, eight agencies reported that they met all three targets for the metrics GAO reviewed, five met two targets, and six met one target. In addition, one agency had not established any targets, and four agencies reported that they no longer owned any data centers. While the three revised metrics' definitions included the key characteristics of being clearly defined and objective, none included statistical universe parameters that enable determinations of progress. Specifically, these metrics call for counts of the actual numbers of (1) virtualized servers, (2) data centers with advanced energy metering, and (3) underutilized servers; but the metrics did not include a count of the universe of all servers and all data centers. Accordingly, percentages cannot be calculated to determine progress–for example, the number of virtualized servers may increase, but if the universe of servers increases at a higher rate, then progress would actually be negative. To improve DCOI reporting and performance, GAO is making four recommendations to OMB, and four to three selected agencies. The three agencies agreed with the recommendations while OMB did not state whether it agreed or disagreed. GAO continues to maintain that the four recommendations to OMB are warranted.", "document_type": "gao"}
{"report": "Private health insurance is the leading source of health coverage in the United States. Small and large employers may offer fully insured group plans (by purchasing coverage from an issuer) or self-funded group plans (by setting aside funds to pay for employee health care). Most small employers purchase fully insured plans, while most large employers self- fund at least some of their employee health benefits. While the majority of health insurance coverage is provided through the small or large group market, Americans without access to group health coverage, such as those with employers that do not offer health coverage, may choose to purchase it directly from an issuer through the individual market. (See fig. 1 for total covered life-years reported by issuers to CMS in the individual and fully insured small and large group markets.) We previously identified several factors that can affect concentration in health insurance markets. High concentration levels have often been the result of consolidation—mergers and acquisitions—among existing issuers. In addition, concentration can persist because of the difficulty for new issuers to enter the market. For example, new issuers that do not yet have large numbers of enrollees may have greater challenges negotiating discounts with health care providers. PPACA contains provisions that may affect market concentration and competition among health issuers, such as the establishment of health insurance exchanges within each state’s overall individual and small group markets. One goal of the exchanges is for issuers to have an incentive to compete with one another on price and value because consumers can visit a website to compare and select among health plans participating in the exchanges. Issuer participation in the exchanges is a key factor in assuring that consumers have a choice of health plans. While PPACA does not require issuers offering coverage in an overall market to participate in the exchanges, issuers have an incentive to do so in order to access additional consumers. For example, certain consumers earning from 100 to 400 percent of the federal poverty level are eligible to receive premium tax credits that can reduce premium costs, but only for plans purchased through an exchange. The federal government and some states have also instituted other provisions to encourage issuers to participate in the exchanges. For example, PPACA required the establishment of the Consumer Oriented and Operated Plan (CO-OP) program, which provided loans to new consumer-governed, nonprofit issuers that are required to offer health plans in the individual and small group exchanges. In addition, in Maryland, certain issuers that offered plans outside of the exchange are also required to offer plans through the exchange. PPACA also established other key market reforms that apply both within and outside of the exchanges, such as requiring that issuers offer coverage to all individuals regardless of health status and limiting the ability of issuers to deny coverage or charge higher premiums to individuals and small groups based on health risks or certain other factors. Since the enactment of PPACA in 2010, there have been additional federal policy changes that may influence an issuer’s decision about whether to participate in health insurance markets. For example, in 2014, HHS announced that a program that made payments to issuers whose losses exceeded a certain threshold—known as risk corridor payments—would be budget neutral, which resulted in reduced payments for some issuers. One issuer told us that this lower than expected funding was one of multiple factors that contributed to its decision to reduce the number of insurance markets in which it participated. States’ overall individual health insurance markets were generally concentrated in 2015 and 2016, similar to what we reported for previous years. Individual market exchanges—representing 57 percent of the overall individual market nationally in 2016—were also concentrated in most states and in many cases became more concentrated in recent years. States’ overall individual health insurance markets were generally concentrated among a small number of issuers in 2015 and 2016. On average, there were 16 issuers participating in each state in 2016. However, that same year, the 3 largest issuers cumulatively held 80 percent or more of the market—an indicator of high concentration—in 37 of 51 states, generally consistent with what we previously reported for years 2011 through 2014 (see fig. 2). The remaining issuers in each state often had significantly smaller market shares—on average, 12 of the 16 issuers in each state held less than 5 percent market share. We also found that in over half of states in 2016, a single issuer held at least 50 percent of the market, consistent with prior years. Specifically, a single issuer held at least 50 percent market share in 28 states in 2016. Of these states, a single issuer held between 80 and 90 percent market share in 5 states, and more than 90 percent market share in 2 states. For example, although West Virginia had 15 issuers in 2016, a single issuer, Highmark, held 91 percent market share. This largest issuer position was held by the same company in both 2015 and 2016 in 45 states; in 35 of these states, the largest issuer had been the same since 2011. While states’ overall individual markets generally remained concentrated, they experienced fluctuations in the extent of concentration in recent years. Specifically, from 2014—the last year of data on which we previously reported—to 2016, the market share of the three largest issuers increased in 30 states (with a median increase of 4 percentage points) and decreased in 21 states (with a median decrease of 6 percentage points). (See fig. 3.) However, despite these changes, states that were highly concentrated in 2014—that is, where the market share of the three largest issuers was at least 80 percent—generally remained highly concentrated in 2016. Our analyses found that states’ individual market exchanges—collectively representing 57 percent of enrollment in the overall individual market nationally in 2016—were generally concentrated among a small number of issuers from 2015 to 2017. Each year during this time period, for the 49 states for which we had complete data, on average, between 3 and 5 issuers participated in the individual market exchanges across the states’ rating areas. Further, each year, the three largest issuers held 80 percent or more of the exchange market, on average, across the states’ rating areas, in at least 46 states. For example, in Wisconsin in 2017, the market share of the three largest issuers ranged from 75 percent in 2 of the state’s 16 rating areas to 100 percent in 6 rating areas; on average, the three largest issuers held 92 percent market share across the 16 rating areas. While the number of states meeting this 80 percent average threshold for high concentration remained relatively constant from 2015 through 2017, market share was increasingly concentrated among a smaller number of issuers in many states, as fewer issuers participated in the exchanges by 2017. The number of states with three or fewer issuers, on average, in their rating areas—and where the issuers therefore held, on average, 100 percent or nearly 100 percent market share—increased from 16 states in 2015 to 32 states in 2017. (See fig. 4.) Further, we found that in at least 35 states each year from 2015 through 2017, the average market share of the largest individual market exchange issuer across the states’ rating areas was at least 50 percent. For example, although Kansas had up to three participating exchange issuers in each of its rating areas in 2017, the largest issuer in each rating area— generally Blue Cross and Blue Shield of Kansas—had at least 88 percent market share. We also found that many states’ individual market exchanges became more concentrated from 2015 to 2017. In 32 of the 49 states, the average market share of the largest exchange issuer across the states’ rating areas increased between 2015 and 2017, with a median increase of 13 percentage points. (See fig. 5.) For example: In Arizona, the average market share of the largest exchange issuer across the state’s rating areas increased by about 60 percentage points, from 39 percent in 2015 to 98 percent in 2017. This increase corresponded with a decrease in issuer participation in the exchange; the state’s seven rating areas had between 7 and 12 issuers in 2015, but by 2017 had only 1 or 2 issuers. In 2015, a CO-OP, Compass Cooperative Health Plan, Inc., had 29 percent of the total exchange market share statewide in Arizona and was among the largest issuers in three rating areas, but it exited the exchange after that year. Other, smaller issuers also exited the exchange after 2015 and 2016, and, in 2017, Blue Cross Blue Shield of Arizona was left as the only issuer in five of the state’s rating areas. In South Carolina, the average market share of the largest exchange issuer increased by 41 percentage points, from 59 percent in 2015 to 100 percent in 2017. As in Arizona, the increase corresponded with a decrease in issuer participation in the exchange, from 2 to 4 issuers in the state’s 46 rating areas in 2015 to only 1 issuer—BlueCross BlueShield of South Carolina—in each rating area in 2017. In addition, a CO-OP, Consumers’ Choice Health Insurance Company, had 43 percent of the total exchange market share statewide and was the largest issuer in nearly half of the state’s rating areas in 2015, but it exited the exchange after that year. In contrast, BlueCross BlueShield of South Carolina had 42 percent of the total exchange market share statewide and was the largest issuer in about half of the rating areas in 2015, and by 2017 was the only remaining issuer in the state. In the remaining 17 states, the average market share of the largest exchange issuer across the states’ rating areas decreased between 2015 and 2017, with a median decrease of 12 percentage points. For example: In Maine, the average market share of the largest exchange issuer decreased by 39 percentage points, from 81 percent in 2015 to 42 percent in 2017. Maine Community Health Options, a CO-OP, remained the largest issuer in each of the state’s four rating areas during this period. However, the other two issuers in these rating areas captured more market share. For instance, Harvard Pilgrim Health Care Group had 1 percent or less market share in each rating area in 2015, but as much as 32 percent market share in one of the state’s rating areas in 2017. In Delaware—which only had one rating area—the market share of the largest exchange issuer decreased by 37 percentage points, from 92 percent in 2015 to 55 percent in 2017. Although the state had the same two issuers, Aetna Group and Highmark Group, throughout this time period—and Highmark Group was the largest issuer each year— Aetna Group’s market share increased from 8 percent in 2015 to 45 percent in 2017. Our analyses found that the overall small group health insurance market remained concentrated in recent years, similar to our prior report. Small group exchanges often had low enrollment—typically less than 1 percent of the overall small group market—and also remained concentrated in recent years. State small group health insurance markets were concentrated among a small number of issuers in 2015 and 2016. On average, there were 8 issuers participating in each state in 2016. However, in that same year the 3 largest issuers cumulatively held 80 percent or more of the market—an indicator of high concentration—in about three-quarters of states, generally consistent with what we previously reported for years 2011 through 2014 (see fig. 6). The remaining issuers in each state often had significantly smaller market shares—on average, 5 of the 8 issuers in each state held less than 5 percent market share. Further, we found that the largest issuers held 50 percent or more of the market in 30 states in 2016. For example, though Louisiana had 6 issuers in 2016, the largest issuer held 76 percent market share. Overall, a single issuer held between 80 and 90 percent market share in 10 states, and more than 90 percent market share in 1 state. This largest issuer position was held by the same company in both 2015 and 2016 in 46 states; in 40 of these states, the largest issuer had been the same since 2011. While states’ overall small group markets remained concentrated, they experienced fluctuations in concentration in recent years. From 2014 through 2016, the market share of the 3 largest issuers increased in 35 states (with a median increase of 3 percentage points), remained the same in 1 state, and decreased in 15 states (with a median decrease of 1 percentage point). (See fig. 7.) However, despite these changes, states that were highly concentrated in 2014—that is, where the market share of the three largest issuers was at least 80 percent—generally remained highly concentrated in 2016. Our analyses found that states’ SHOP exchanges remained concentrated from 2015 to 2017, with only slight overall changes in issuer participation or market share. Further, as a proportion of the overall small group market, SHOP exchanges in most states had little enrollment. (See sidebar.) Small Group Health Options Program (SHOP) Enrollment as a Proportion of the Overall Small Group Market As a proportion of the overall small group market, SHOP exchanges in most states had little enrollment—that is, typically less than 1 percent of the overall small group market. For example, in 2016, Alaska’s small group market had 17,257 covered life-years, while its SHOP exchange had 96 covered life-years (0.6 percent). The District of Columbia, Rhode Island, and Vermont were the only states where the SHOP exchange was more than 3 percent of the overall small group market. The District of Columbia and Vermont require all small group plans to be purchased through the state’s SHOP exchange. (See app. III.) In each year, more than 31 of the 46 states for which we had data had three or fewer issuers in the SHOP exchange; therefore between one and three issuers held 100 percent of the market share for the state. Among states with four or more issuers, the market share of the three largest issuers was typically at least 80 percent. For example, California had 6 participating issuers from 2015 through 2017 in the SHOP exchange, and the market share for the three largest issuers in that state ranged from 92 to 93 percent across the 3 years. (See fig. 8.) On average, the number of participating issuers in the SHOP exchange decreased slightly from 2015 through 2017. However, in a few states, there were larger changes in issuer participation and concentration. For example, Ohio’s SHOP exchange had 7 participating issuers in 2015, decreasing to 4 issuers in 2017. Across this time period, the market share of the three largest issuers in Ohio’s SHOP exchange increased from 59 percent to 98 percent. Conversely, New York’s SHOP exchange had 10 issuers in 2015, decreasing to 8 issuers in 2017; but the market share of the three largest issuers decreased by almost 7 percentage points within that time. Further, we found that in at least 38 of 46 states each year from 2015 through 2017, the largest issuer held at least 50 percent of the SHOP exchange market share. In 23 states, the market share of the largest issuer increased during this period, with a median increase of 11 percentage points, and in 6 additional states the largest issuer was the only issuer in the SHOP exchange and thus held 100 percent market share for all 3 years. (See fig. 9.) For example: In Kentucky, the market share of the largest issuer in the SHOP exchange increased by 56 percentage points, from 42 percent in 2015 to 98 percent in 2017. In 2015, the largest issuer was Kentucky Health Cooperative, a CO-OP that exited the SHOP exchange after 2016. The second largest issuer in 2015, Wellpoint Inc. Group, increased market share over this time period, from 33 percent in 2015 to 98 percent in 2017, becoming the largest issuer. The market share of the other remaining issuer in Kentucky’s SHOP exchange, Baptist Health Plan, Inc., decreased from 19 percent in 2015 to 1 percent in 2017. In Ohio, the market share of the largest issuer in the SHOP exchange increased by 54 percentage points, from 29 percent in 2015 to 83 percent in 2017. Across this time period, the largest issuer changed from Medical Mutual of Ohio (which had 29 percent market share in 2015 and 8 percent in 2017) to Wellpoint Inc. Group (which had 12 percent market share in 2015 and 83 percent in 2017). This increase in the largest issuer’s market share corresponded with a decrease in issuer participation. The state had 7 participating issuers in 2015, decreasing to 4 in 2017. The market share of the 3 issuers that left ranged from 9 to 16 percent. In the remaining 17 states, the market share of the largest issuer decreased between 2015 and 2017, with a median decrease of 7 percentage points. In some states, these decreases were significant. For example, in Maine, the market share of the largest issuer, Maine Community Health Options—a CO-OP—decreased by 40 percentage points, from 89 percent in 2015 to 49 percent in 2017. During this time period, while Maine Community Health Options remained the largest issuer, the other two participating issuers gained additional market share. For example, Harvard Pilgrim Health Care Group increased market share from 6 percent in 2015 to 38 percent in 2017. In 2015 and 2016, states’ overall large group health insurance markets remained concentrated, as in prior years. On average, there were 10 participating issuers in each state in 2016. However, in that same year the 3 largest issuers held at least 80 percent market share in 43 of 51 states, which is generally consistent with prior years. (See fig. 10.) In 2016, 3 issuers held 99 or 100 percent of the large group market in 7 states—Alabama, Alaska, Mississippi, Nebraska, North Dakota, South Carolina, and Vermont. The remaining issuers in each state often had significantly smaller market shares—on average, 6 of the 10 participating issuers in each state held less than 5 percent market share. In more than 30 states in 2015 and 2016, market share was not only concentrated among a small number of issuers, but a single issuer held at least 50 percent of the overall large group market, as in prior years. A single issuer held at least 50 percent market share in 33 states in 2016, with significantly higher levels of concentration by the largest issuer in some states. For example, in 2016, a single issuer held at least 90 percent of the market in Alabama and at least 80 percent of the market in 5 other states (Alaska, Mississippi, Montana, South Carolina, and Vermont). Further, this largest issuer position was held by the same company in 2015 and 2016 in 49 states; and, in 47 of those states, the largest issuer position has been held by the same company since 2011. The extent of concentration in the overall large group market remained relatively constant when comparing 2014—the last year of data on which we previously reported—to 2016. The market share of the 3 largest issuers increased in 24 states and decreased in 24. (See fig. 11.) The largest increase was in Wisconsin, where the market share of the 3 largest issuers increased from 38 percent in 2014 to 44 percent in 2016, and the largest decrease was in New York, where the market share of the 3 largest issuers decreased from 55 percent in 2014 to 47 percent in 2016. We provided a draft of this report to HHS for review and comment. The department provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate Congressional committees, the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IX. This table presents covered life-years in each state’s individual market health insurance exchange as a proportion of total covered life-years in each state’s overall individual market in 2016. The four tables below present information on the number of participating issuers and market share of the largest issuers in each state’s individual market exchange from 2015 through 2017. Specifically, Table 2 presents the total number of exchange issuers in each state. Table 3 presents the average number of exchange issuers across each state’s rating areas. Table 4 presents the names and market shares of the single largest exchange issuer, and market share of the largest three issuers, for each state. Table 5 presents the average market share of the largest issuer across each state’s rating areas. This table presents covered life-years in each state’s Small Business Health Options Program (SHOP) exchange as a proportion of total covered life-years in each state’s overall small group market in 2016. The two tables below present information on the participation of issuers in each state’s small group health insurance exchange from 2015 to 2017 and the market share of the largest and three largest issuers from 2015 to 2017. The two tables below present information on the participation of issuers in each state’s overall individual health insurance market from 2011 to 2016, and the market share of the largest and three largest issuers from 2014 to 2016. Table 11 provides market share for the 23 consumer operated and oriented plans (CO-OPs) participating in state individual market and Small Business Health Options Program exchanges for 2015 through 2017. CO- OPs are new consumer-governed, nonprofit issuers created under the Patient Protection and Affordable Care Act. Out of the 23 CO-OPs originally operating in 2014, all but four, operating in five states, had ceased operations by the end of 2017. The two tables below present information on the participation of issuers in each state’s overall small group health insurance market from 2011 to 2016 and the market share of the largest and three largest issuers from 2014 to 2016. The two tables below present information on the participation of issuers in each state’s overall large group health insurance market from 2011 to 2016 and the market share of the single largest and three largest issuers from 2014 to 2016. In addition to the contact named above, William D. Hadley, Assistant Director; Katherine L. Amoroso, Analyst-in-Charge; Priyanka Sethi Bansal; Giselle Hicks; John Lalomio; and Sarah-Lynn McGrath made key contributions to this report. Also contributing were Yesook Merrill; Laurie Pachter; Vikki Porter; Oliver Richard; and Emily Wilson.", "summary": "A highly concentrated health insurance market may indicate less competition and could affect consumers' choice of issuers and the premiums they pay. In 2014, PPACA required the establishment of health insurance exchanges—a new type of marketplace where individuals and small groups can compare and select among insurance plans sold by participating issuers—and the introduction of other reforms that could affect market concentration and competition among issuers. GAO previously reported that enrollment through these newly established exchanges was also generally concentrated. PPACA included a provision for GAO to study market concentration. This report describes changes in the concentration of enrollment among issuers in (1) overall individual, small group, and large group markets, and (2) individual and small group exchanges. GAO determined market share in the overall markets using enrollment data from 2015 and 2016 that issuers are required to report annually to the Centers for Medicare & Medicaid Services (CMS) and compared that data to 2011 through 2014 enrollment data GAO analyzed in previous reports. GAO determined market share in the exchanges from 2015 through 2017 using other sources of enrollment data from CMS and states. For all data sets, GAO used the most recent data available. Enrollment in private health insurance plans continued to be concentrated among a small number of issuers in 2015 and 2016. In the overall large group market (coverage offered by large employers), small group market (coverage offered by small employers), and individual market (coverage sold directly to individuals), the three largest issuers held 80 percent of the market or more in at least 37 of 51 states. This is similar to what GAO previously reported for 2011 through 2014. GAO also found that within the overall individual and small group markets in each state, the health insurance exchanges established by the Patient Protection and Affordable Care Act (PPACA) were also concentrated from 2015 to 2017. For the individual market exchanges, in each year, three or fewer issuers held 80 percent or more of the market, on average, in at least 46 of the 49 state exchanges for which GAO had data. Further, the largest issuers increased their market share in about two-thirds of exchanges. The number of issuers participating in a market and their market shares can affect concentration, and many individual exchanges generally had a decreasing number of participating issuers over time. For the small group market exchanges, in each year, three or fewer issuers held 80 percent or more of the market in at least 42 of the 46 state exchanges for which GAO had data. The small group exchanges also had slight changes in issuer participation and market share over this time period. GAO received technical comments on a draft of this report from the Department of Health and Human Services and incorporated them as appropriate.", "document_type": "gao"}
{"report": "The Mexico City Policy, which the U.S. government announced at the United Nations Conference on Population in Mexico City in 1984, required foreign NGOs to agree they would not, as a condition for receiving U.S. assistance for family planning, perform or actively promote abortion as a method of family planning. As shown in figure 1, subsequent administrations have rescinded or reinstated the policy through executive branch action, typically through presidential memoranda. In a January 2017 Presidential Memorandum, the Trump Administration reinstated and expanded the Mexico City Policy, directing the Secretary of State in coordination with the Secretary of Health and Human Services to implement a plan to extend the requirements of the reinstated policy to all global health assistance furnished by all departments or agencies to the extent allowable by law. Consequently, the policy, later renamed PLGHA, applies to billions of dollars in annual U.S. global health assistance—such as HIV/AIDS, maternal and child health, and malaria— rather than only family planning and reproductive health assistance, which received about $560 million in GHP account funding in fiscal year 2018. State reported that USAID, State, and DOD began applying the PLGHA policy as of May 15, 2017, and HHS applied the policy as of May 31, 2017. The affected departments and agencies applied the policy to: (1) All existing grants and cooperative agreements that provide global health assistance that received new funding after May 2017. Agencies established a PLGHA standard provision for inclusion in relevant grants and cooperative agreements for global health assistance requiring foreign NGOs to agree that, during the term of the award, they would not perform or actively promote abortion as a method of family planning in foreign countries, or provide financial support to any foreign NGO that does. Agency officials stated that after the policy was implemented, when additional funds were to be obligated to relevant awards with foreign NGOs, these organizations would be required to accept the PLGHA terms and conditions to receive these additional funds, or decline the award. (2) All new grants and cooperative agreements that provide global health assistance awarded after May 2017, according to a State report. The PLGHA terms and conditions apply to foreign NGOs that receive global health assistance prime awards or sub-awards. Prime awardees, including U.S. NGOs, may not provide assistance under the awards to any foreign NGOs that perform or actively promote abortion as a method of family planning, are required to include the PLGHA standard provision in sub-awards to foreign NGOs, and may be held liable for the sub- awardee’s failure to comply with the conditions of the policy. According to UN reporting, the legality of abortion varies among countries receiving U.S. global health assistance. This may result in some countries legally permitting abortion services that are not permitted under the PLGHA policy, according to NGO representatives we met with. The representatives noted that under these circumstances, foreign NGOs would be prohibited under the policy from providing such services, even with non-U.S. funds, as a condition of receiving U.S. global health assistance. Additionally, in March 2019, the Secretary of State clarified that foreign NGOs that accept U.S. global health assistance may not provide financial support, “with any source of funds and for any purpose, to another foreign NGO that performs, or actively promotes, abortion as a method of family planning.” According to agency officials, the PLGHA terms and conditions do not apply under the following circumstances: Global health contracts. State reported that the executive branch is taking steps to develop a PLGHA contract clause through a formal rule-making process required to revise the Federal Acquisition Regulation. Awards funded out of the Food for Peace program. Water Supply and Sanitation assistance funded from the Development Assistance account. Assistance provided directly by U.S.-based organizations. The PLGHA policy does apply, however, to sub-awards made by U.S.- based organizations to foreign NGOs. Assistance provided directly to national governments, such as ministries of health. Assistance to multilateral organizations. This includes but is not limited to U.S. global health funds provided to the Global Fund to Fight AIDS, Tuberculosis, and Malaria (the Global Fund) and the Joint United Nations Program on HIV/AIDS (UNAIDS). In a May 2017 briefing on the PLGHA policy, State noted that humanitarian assistance, including State Department migration and refugee assistance activities, USAID disaster and humanitarian relief activities, and U.S. Department of Defense disaster and humanitarian relief were also all excluded from the policy. State also noted that the Secretary of State, in consultation with the Secretary of HHS, may authorize additional case-by-case exemptions to the policy. Congress provided about $8.7 billion for the Global Health Programs account (GHP) in fiscal year 2018, most of which supported HIV/AIDS assistance managed by State and implemented through transfers of funds to several agencies and contributions to multilateral organizations (see table 1). Because of the various exclusions described above, not all global health funds are subject to the PLGHA policy. In particular, State’s fiscal year 2018 contribution of $1.35 billion to the Global Fund is not subject to the policy because it is a multilateral institution. USAID and CDC had the most global health assistance awards subject to the PLGHA policy, representing more planned funding than other agencies (see table 2). In total, U.S. agencies reported that they applied the PLGHA policy to 1,309 prime awards active in May 2017 or made through September 2018. There were 761 active awards when agencies implemented the policy in May 2017, and 548 new awards that began after they implemented the policy. Most awards started in fiscal year 2016 or later, although some started earlier. Average award duration varied among agencies. The estimated total value of these 1,309 awards was almost $29 billion across multiple fiscal years, of which about $12 billion was planned funding that had not yet been obligated as of September 30, 2018, and is subject to the PLGHA policy upon acceptance of the PLGHA terms and conditions. USAID awards represented 50 percent of planned funds that were not yet obligated for awards subject to the PLGHA policy, while CDC awards represented 46 percent of such funds. Other HHS component agencies’ awards subject to the policy combined represented almost 4 percent of planned funds that were not yet obligated. DOD and State awards represented less than 1 percent of these funds. State’s awards were relatively numerous but shorter-term and of smaller dollar value than other agencies’ awards. Agencies reported that, as of September 30, 2018, over $8 billion of the more than $12 billion in estimated planned funding (over 66 percent) for awards subject to PLGHA that were active between May 2017 and September 2018 was for HIV/AIDS assistance (see table 3). All DOD and State planned funding, and almost all HHS planned funding, supported HIV/AIDS assistance. USAID reported that its planned funding was distributed across several global health areas including HIV/AIDS, family planning and reproductive health, maternal and child health, and tuberculosis. Agencies reported that over $8 billion of the more than $12 billion (over 66 percent) of the estimated planned funding for awards subject to PLGHA that were active between May 2017 and September 2018 was for awards in Africa (see table 4). Awards in Asia accounted for the second highest level of planned funding for an individual region at almost $600 million (5 percent). Global awards implemented in more than one region represented about $3 billion in planned funding (26 percent). By global health assistance area and region, HIV/AIDS assistance in Africa accounted for the most planned funding that had not yet been obligated for awards subject to PLGHA: over $6 billion of about $12 billion (52 percent) (see table 5). The next largest category was global HIV/AIDS assistance awards, which accounted for over $1 billion (13 percent). The top 10 countries receiving the most estimated planned funding that had not yet been obligated under awards subject to PLGHA accounted for over $6 billion of more than $12 billion (54 percent) (see table 6). All 10 countries are in sub-Saharan Africa. Of these countries, South Africa had the most planned funding remaining (over $2.4 billion) that was subject to the policy. See appendix II for more details on the locations of awards subject to PLGHA. USAID identified 53 awards—six prime awards and 47 sub-awards in which NGOs declined to accept PLGHA terms and conditions. CDC identified one prime award in which an NGO declined to accept the policy’s terms and conditions. These prime and sub-awards had about $153 million in estimated planned funding remaining that was not obligated at the end of fiscal year 2018 (see table 7). DOD and State did not identify any declinations. The remaining planned funding that was not obligated as of September 30, 2018, represents an estimate of the amount that had been planned for the awards but which was not obligated under these awards because awardees declined to accept the terms and conditions of the PLGHA policy, according to the agencies. USAID identified six prime awards in which NGOs declined to accept PLGHA terms and conditions resulting in an estimated $94 million in planned funding that was not obligated as of September 30, 2018. These six prime awards, presented in table 8, supported different global health assistance areas. Three of the awards were global in scope, two provided assistance to India, and one provided assistance to Zimbabwe. The two largest of the six prime awards declined were global awards to Marie Stopes International (MSI) and International Planned Parenthood Federation (IPPF), both of which publicly stated that they could not meet the conditions of PLGHA because abortion services or referrals are part of reproductive health care services they provide and a right to which their patients are entitled. Together, these two awards had about $79 million remaining in planned funding that was not obligated as of September 30, 2018. The primary objective of these two awards was to increase access to and use of family planning products and services, although the award to MSI also supported maternal and child health and HIV/AIDS and the IPPF award supported HIV/AIDS in addition to family planning and reproductive health, according to information provided by USAID. According to MSI and IPPF representatives, these two awards both included, among other activities, mobile family planning and reproductive health outreach activities that reached underserved rural populations in multiple countries. While MSI and IPPF were able to obtain some funding from other donors when the USAID awards were suspended, the additional funds fell far short of the funds provided by USAID, according to the organizations’ representatives, resulting in reductions in family planning services they provided to recipient countries. CDC identified one prime award in which an NGO declined to accept the PLGHA terms and conditions. According to CDC, this award had about $8.4 million remaining of a 5-year, $10.5 million award ceiling for delivery of HIV services in sexual and reproductive health clinics and in confidential clinics for commercial sex workers in Ethiopia. USAID identified 47 global health sub-awards in which foreign NGOs declined to accept the PLGHA policy’s terms and conditions and thus ceased receiving U.S. funding under those awards following implementation of the PLGHA policy (see table 9). The planned funding that was not obligated for these sub-awards amounted to about $51 million, as of September 30, 2018. As shown in table 9, sub-awards with NGOs that declined to accept the PLGHA terms and conditions involved multiple global health assistance areas. Family planning and reproductive health represented the largest share of planned sub-award value involving declinations, followed by awards supporting multiple global health areas and HIV/AIDS. Sub-awards involving declinations also addressed maternal and child health, tuberculosis, and nutrition assistance. According to data provided by USAID, sub-awards in which NGOs declined the PLGHA terms and conditions occurred in multiple regions, but primarily in countries in Africa. USAID identified 32 sub-awards implemented in African countries involving NGOs that declined the PLGHA terms and conditions following implementation of the policy. The estimated total value of these sub-awards was about $56 million, of which more than half (about $32 million) remained as planned funding that was not obligated as of September 30, 2018 (see table 10). Of the 47 sub-awards for which the PLGHA terms and conditions were declined, 26 were declined by affiliates of either IPPF or MSI. The estimated total award value of these 26 sub-awards amounted to over half of the value of the 47 sub-awards (see figure 2). Four countries had the largest estimated amount of sub-award funds declined by NGOs, with at least $8 million in planned funding that was not obligated as of September 30, 2018 (see table 11). For example, two declined sub-awards implemented in Senegal had a combined $9.7 million in planned funding that was not obligated as of September 30, 2018. These two sub-awards were implemented by an MSI affiliate that, among other services, used the USAID funds to operate mobile family planning clinics for beneficiaries in rural, underserved areas. According to MSI representatives, these sub-awards did not involve abortion services, which MSI indicated are illegal in Senegal. However, the NGO declined the sub-award because of its affiliation with MSI, according to the representatives. Bangladesh had the most sub-awards in which NGOs declined the PLGHA terms and conditions with five. Total planned funding that was not obligated for these five sub-awards amounted to about $9 million as of September 30, 2018. These awards supported multiple areas of global health assistance including family planning and reproductive health, tuberculosis, nutrition, and maternal and child health. We provided a draft of this report to DOD, HHS, State, and USAID, and for review and comment. In their written comments, reproduced in appendix III, USAID stated that it found our estimates of the number and value of awards subject to PLGHA and those in which NGOs declined to accept PLGHA the terms and conditions to be reasonable given the data available. USAID also elaborated on limitations with available data, which we believe are consistent with the data limitations we describe in this report. DOD, HHS, and State did not provide written comments. In addition, HHS, State, and USAID provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretaries of Defense, Health and Human Services, and State, and the Administrator of the U.S. Agency for International Development. In addition, the report is available at no charge on the GAO website at http://www.gao.gov . If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our objectives were to identify (1) global health assistance awards that U.S. agencies determined to be subject to the terms and conditions of the U.S. government’s Protecting Life in Global Health Assistance (PLGHA) policy requiring foreign non-governmental organizations (NGOs) to agree that they would not perform or actively promote abortions as a method of family planning, and (2) planned funding for awards involving NGOs that declined to accept the terms and conditions of this policy. To identify the global health assistance awards subject to the terms and conditions of the PLGHA policy, we obtained data from the Departments of State (State), Health and Human Services (HHS), and Defense (DOD), and the U.S. Agency for International Development (USAID) on all relevant awards active when the policy was first implemented in May 2017 or awarded through September 30, 2018. We identified the relevant agencies based on a February 2018 State report on the initial implementation of PLGHA and discussions with each agency to identify affected component agencies. Component agencies within HHS that identified awards subject to the PLGHA included the Centers for Disease Control and Prevention (CDC), the National Institutes of Health, the Health Resources and Services Administration, and Substance Abuse and Mental Health Services. Within DOD, the Department of the Army and the DOD HIV/AIDS Prevention Program identified awards subject to the policy. To obtain information that was as complete and consistent as possible from each relevant agency on all awards subject to the PLGHA terms and conditions, we created a data collection instrument. This instrument asked the agencies to identify all awards that were subject to the PLGHA, that were either active in May 2017 when the PLGHA policy was first implemented or that were new awards through the end of fiscal year 2018 (September 30, 2018). We analyzed the responses to our data collection instrument to describe the number and estimated total value of the awards, the amount obligated as of September 30, 2018 and the estimated amount of planned funding that was not yet obligated for these awards, the implementing agency, the type of global health assistance, and the recipient countries. Agencies defined estimated total award value as either award ceilings or total award amounts for the life of the award including both funding that recipient organizations may have obligated prior to the PLGHA policy as well as funding that organizations have not yet received but may receive in future years. We asked the agencies to categorize the type of global health assistance based on the Foreign Assistance Standardized Program Structure and Definitions, which State updated in 2016. During the development of this data collection instrument, we discussed drafts with each of the agencies and made modifications as appropriate. We provided definitions for each data element requested that allowed for variations in the ways these agencies collect and record data on awards. To estimate the value of planned funds not yet obligated and therefore subject to the PLGHA policy, we subtracted the obligated amount from the estimated total award value of each award. While this calculation provides an estimate of the funds subject to the PLGHA, it is limited by two factors. First, while planned award funding that was not already obligated before May 2017 when PLGHA was first implemented was made subject to the PLGHA policy, agencies did not have obligations data as of May 2017 readily available but were able to readily identify obligations as of September 30, 2018. Therefore, information provided on planned funding that was not yet obligated as of September 30, 2018, may not capture all of the funding made subject to the PLGHA policy because it does not include obligations between May 2017 and September 30, 2018, for NGOs that accepted PLGHA terms and conditions. Second, estimates of total award value can change over time, according to agency officials. For example, awards could have extensions with additional funding not yet reflected in the estimated total award values agencies provided us. In addition, the estimated total award values the agencies provided could be based on a maximum or ceiling for some awards, which may overstate actual amounts. To identify the prime and sub-awards active in May 2017 that involved NGOs that declined the PLGHA terms and conditions, we developed additional data collection instruments—one for prime awards between agencies and NGOs and one for sub-awards between prime awardees and NGOs—to request information on these awards from the relevant agencies. We followed the same process described above to develop these two additional instruments to identify estimated total value of the awards, obligated amounts as of September 30, 2018, the implementing agency, the type of global health assistance, and the recipient countries. USAID identified 53 declined prime or sub-awards and CDC identified one. For these agencies, identifying these awards involved contacting staff based in overseas posts. The other agencies reported to us that they had no awards in which NGOs declined the PLGHA terms and conditions. A USAID official also noted that the sub-award amounts they provided to us could vary from year to year, which would affect the amounts of remaining planned funding that was not obligated as of September 30, 2018. Nevertheless, we relied on these amounts to estimate the amount of planned funding that was not obligated under these awards as of the end of fiscal year 2018 because the NGOs declined to accept the PLGHA terms and conditions. Efforts taken by prime awardees to replace declined sub-awards were not part of our review. In addition to meeting and corresponding with USAID and CDC officials to discuss awards involving declinations, we interviewed representatives of Marie Stopes International (MSI) and International Planned Parenthood Federation (IPPF)—two prime awardees that publicly declined to accept the terms and conditions of the PLGHA policy. These two NGOs declined the two largest of the six prime awards declined and their local affiliates were implementers of many of the sub-awards that were declined. We discussed with MSI and IPPF the characteristics of these two awards and the accuracy of USAID’s data provided to us on them. We examined the reliability of the data on awards identified by the agencies through testing for logical assumptions such as whether award start dates preceded their end dates, and whether an award’s estimated total value met or exceeded the total amount of funding that had been obligated to it. In addition, we met with agency officials to discuss and correct any discrepancies in the award data they provided. However, we did not independently verify the awards identified or the funds associated with each award. Overall, we found the data on awards subject to the PLGHA policy and in which NGOs declined the terms and conditions of the policy to be sufficiently reliable for the purposes of delineating the agencies, assistance areas, countries, estimated total value of awards, and obligations. As noted earlier, we also calculated the amounts of planned funding that were not obligated as of September 30, 2018, to estimate the amount of funding subject to the policy. We conducted this performance audit from April 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Global health awards that agencies identified as subject to the PLGHA terms and conditions amounted to almost $29 billion in estimated total award value. This amount includes funding that agencies had obligated before implementing the PLGHA policy in May 2017 as well as funding across multiple fiscal years and for potential award extensions. Agencies reported that about $12 billion in funding was not yet obligated as of September 30, 2018. Award funding included assistance to specific countries, as well as awards that were regional or global in scope (see table 12). David Gootnick (202) 512-3149 or gootnickd@gao.gov In addition to the individual named above, Leslie Holen (Assistant Director), Howard Cott, Martin de Alteriis, Kelsey Griffiths, Christopher Keblitis, Andrew Kurtzman, Michael McAtee, Aldo Salerno, Fatima Sharif, and Alexander Welsh made significant contributions to this report.", "summary": "The United States is the world's largest donor of global health assistance. Congress provided about $8.7 billion for the Global Health Programs (GHP) account in fiscal year 2018. In 2017, the President reinstated and expanded a policy, which now requires foreign NGOs to agree that, as a condition of receiving global health assistance, they will not perform or actively promote abortion as a method of family planning or provide financial support during the award term to other foreign NGOs that conduct such activities. The Reagan administration first implemented this policy, known as the Mexico City Policy, in 1984, and subsequent administrations have rescinded and reinstated it. The Mexico City Policy initially applied only to family planning and reproductive health assistance, which received about $560 million of GHP funds in fiscal year 2018. Upon reinstating the policy, the Trump Administration renamed it PLGHA and applied it to all global health assistance to the extent allowable by law. GAO was asked to review the implementation of the PLGHA policy. This report identifies (1) global health assistance awards that U.S. agencies determined to be subject to the U.S. government's PLGHA policy requiring foreign NGOs to agree that they would not perform or actively promote abortion as a method of family planning, and (2) planned funding for awards involving NGOs that declined to accept the terms and conditions of this policy. GAO analyzed data provided by U.S. agencies of awards subject to the PLGHA policy and awards in which NGOs declined to accept the terms and conditions of this policy. U.S. agencies reported to GAO that from May 2017 through fiscal year 2018, they applied the Protecting Life in Global Health Assistance (PLGHA) policy to over 1,300 global health awards. The policy's restrictions on performing or actively promoting abortion as a method of family planning applied to active awards that received new funding after the policy was implemented, and all funding for new awards made after May 2017. As of September 30, 2018, about $12 billion in estimated planned award funding was subject to the policy. The U.S. Agency for International Development (USAID), with over $6 billion, and the Centers for Disease Control and Prevention (CDC), with over $5 billion, awarded about 96 percent of this amount. Agencies implemented these awards across multiple geographic regions and global health assistance areas. About two-thirds of estimated planned funding subject to the policy supported HIV/AIDS assistance, while the remaining third supported other global health areas, such as maternal and child health, and family planning and reproductive health. Over two-thirds of planned funding subject to the policy was for awards in Africa. U.S. agencies identified seven prime awards and 47 sub-awards in which non-governmental organizations (NGOs) declined to accept the terms and conditions of the PLGHA policy, and these awards had about $153 million remaining in estimated planned funding not obligated as of September 30, 2018. The seven prime awards that were declined included six USAID awards and one CDC award and amounted to about $102 million of the $153 million in estimated planned funding that was not obligated. Marie Stopes International and the International Planned Parenthood Foundation declined the two largest of these awards, resulting in about $79 million in planned funding that was not obligated. These two awards included, among other activities, mobile family planning and reproductive health outreach activities to underserved, rural populations in multiple countries. USAID identified all of the 47 sub-awards that were declined, which had a total of about $51 million in planned funds that was not obligated. Thirty-two of the 47 subawards were intended for Africa. by Global Health Assistance Area Source: GAO analysis of agency reported data | GAO-20-347", "document_type": "gao"}
{"report": "Three DOE offices manage 57.2 MT of plutonium declared surplus to defense needs. These offices—NNSA, EM, and DOE’s Office of Nuclear Energy (NE)—and their sites manage a variety of surplus plutonium in the form of pits, metal, oxide, spent nuclear fuel, and other reactor fuels, and they follow specific procedures to manage the plutonium safely and securely. NNSA manages over half of this surplus plutonium. According to NNSA, all three offices share the responsibility for final disposition of surplus plutonium. Figure 1 shows the amounts of surplus plutonium managed by the offices. Figure 2 shows the various forms of this surplus plutonium, including pits, non-pit metal, non-pit oxide, and spent nuclear fuel or other reactor fuels in the inventory, by DOE office. Since 1997, DOE’s surplus plutonium disposition strategies have changed in terms of the method of disposal and the location for disposal, according to DOE documents and officials. These disposition strategies have included immobilization, irradiation as MOX fuel, and dilution. In 1997, NNSA planned to immobilize surplus plutonium by encapsulating it in glass or ceramic materials but terminated its plans in 2002 due to budget constraints. In the mid-2000s, EM briefly considered vitrification, which is a form of immobilization using glass, but never developed a plan to implement it. NNSA planned to irradiate surplus plutonium as part of the MOX fuel strategy but terminated its plans in 2018 because of high costs. NNSA’s plans for irradiation of MOX fuel would also have required disposal of the spent nuclear fuel in a high-level waste repository. EM began implementing a dilute and dispose strategy for a separate portion of surplus plutonium in 2012, but suspended its efforts until it resumed them in 2016. NNSA’s 2018 conceptual plan for the dilute and dispose strategy would replace the MOX fuel strategy with final disposal of the diluted plutonium at WIPP. Figure 3 shows a timeline of the changes in DOE’s strategies since 1997, as well as some key events that have affected the strategies. See appendix II for a timeline of DOE’s disposition strategies and appendix III for a timeline of key events concerning DOE’s Surplus Plutonium Disposition Program. Even if NNSA and EM had successfully implemented strategies for immobilization, vitrification, or irradiation of MOX fuel, DOE would have had no place to dispose of the surplus plutonium that was prepared for disposal because it planned to dispose of this material in a high-level waste repository, and no high-level waste repository has yet been constructed. WIPP would not have been able to take surplus plutonium from these disposition strategies because federal law authorizing disposal of radioactive waste at WIPP specifically bans the disposal of high-level waste and spent nuclear fuel, and the final forms of the surplus plutonium from these disposition strategies would have included both. DOE’s plans for a high-level waste repository have also changed over time. No progress toward licensing and building a high-level waste repository has been made since DOE terminated its licensing efforts in 2010. A high- level waste repository is likely still decades away from becoming operational. Appendix IV contains more information on the progress DOE has made toward licensing and building a high-level waste repository. NNSA’s current dilute and dispose strategy requires that surplus pits, as well as other surplus plutonium in metal form, be converted to plutonium oxide. NNSA’s now-terminated strategy to use surplus plutonium to make MOX fuel also required that surplus plutonium be converted to plutonium oxide. In the early 2000s, NNSA had planned to build a facility—the Pit Disassembly and Conversion Facility at SRS—that was to be dedicated to disassembling pits and converting them to plutonium oxide to meet the high plutonium oxide production requirements for manufacturing MOX fuel. Because of its high costs, however, NNSA canceled the Pit Disassembly and Conversion Facility in January 2012 after having spent $730.1 million on its design, as we reported. In August 2012, DOE provided a report to Congress that described a mix of plutonium oxide production capabilities to replace the canceled Pit Disassembly and Conversion Facility. According to the 2012 report, DOE planned to convert at least 2 MT of surplus plutonium pits to plutonium oxide by 2018 in PF-4 at LANL and an additional 3.7 MT of plutonium oxide at SRS by 2017. According to its 2012 report, NNSA planned for this plutonium oxide to be a reserve of advance feedstock for the MFFF. NNSA anticipated it would begin operations in 2019. According to NNSA, SRS turned out not to be cost-effective at producing plutonium oxide. Specifically, SRS produced 35 kilograms (0.035 MT) of plutonium oxide at SRS’s H Canyon facility over a 2.5-year period ending in 2018. NNSA discontinued plutonium oxide production at H Canyon and focused its plans on expanding ARIES operations at PF-4. According to NNSA, ARIES operations at PF-4 currently host the nation’s only cost-effective plutonium oxide production capability. In 1998, DOE established ARIES at PF-4 at LANL in New Mexico as a technology demonstration project to dismantle pits and convert plutonium metal into an oxide, incorporating automation to reduce liquid waste and workers’ exposure to radiation. ARIES’s technology for converting plutonium to plutonium oxide was designed to generate very little chemical waste and to permit the application of automation, which significantly reduces the risk of workers’ exposure to radiation. Pits have historically been disassembled by a cutting machine. Before ARIES’s technology, recovery of plutonium from cut pits was by an aqueous process—that is, by using liquid chemical processing—which generated significant volumes of both liquid and solid waste. In 2008, NNSA shifted the ARIES mission from a technology demonstration project to a small plutonium oxide production capability. According to NNSA officials, ARIES has produced approximately 1 MT of plutonium oxide from pits since it was established in 1998, with peak production of 242 kilograms (0.242 MT) in 2011 during a partial year of operations. NNSA officials explained that ARIES did not produce larger amounts of plutonium oxide because the agency was still evaluating alternatives for expanding plutonium oxide, but they estimated that ARIES could produce 300 kilograms to 400 kilograms in a full year of operations. In addition, LANL shut down the PF-4 facility, including ARIES, from June 2013 through September 2016 to correct safety and operational issues. During this time, plutonium oxide production using ARIES in PF-4 was suspended. Plutonium oxide is the preferred form for long-term storage of plutonium because it is relatively stable compared to other forms. Plutonium oxide is also the form of plutonium that is most suited for dilution. ARIES consists of glove boxes, furnaces, and other equipment to dismantle a pit and extract the plutonium; convert the plutonium into an oxide form; mill and blend the plutonium oxide; conduct physical and chemical analyses of the plutonium oxide; and package and store the plutonium oxide for eventual disposition. NNSA’s 2018 conceptual plan to dilute and dispose of surplus plutonium calls for plutonium metal to be converted to plutonium oxide using ARIES at PF-4 and then for the plutonium oxide to be diluted at SRS for eventual disposal at WIPP. Figure 4 shows the dilute and dispose strategy as described in NNSA’s 2018 conceptual plan. DOE could convert 43.8 MT, or about 77 percent, of surplus plutonium in its inventory of 57.2 MT to plutonium oxide for dilution and disposal because this plutonium is in a metal form suitable to oxidation, based on our review of DOE’s inventory of surplus plutonium. Most of this surplus plutonium metal—33.3 MT—is in the form of pits and is managed by NNSA. EM manages 6.5 MT of surplus plutonium metal and NE manages the remaining 4 MT of surplus plutonium metal reactor fuel at Idaho National Laboratory. Separately, EM also manages 6.4 MT of surplus plutonium that is already in oxide form. Figure 5 shows the forms of surplus plutonium in DOE’s inventory of 57.2 MT of surplus plutonium requiring disposition. As noted above, EM manages 6.4 MT, or 11 percent, of surplus plutonium that already exists as plutonium oxide. According to NNSA officials, SRS is currently diluting this oxide at a modest rate of about 20 kilograms (0.02 MT) annually. According to NNSA documents, the agency plans to add additional throughput capacity within a decade. The remaining 7 MT of surplus plutonium, or about 12 percent of DOE’s surplus plutonium inventory, is contained in spent nuclear fuel and is not suitable for conversion to plutonium oxide. This material would require additional chemical processing steps to make it suitable for conversion to plutonium oxide. DOE officials said that they planned to dispose of the 7 MT of spent nuclear fuel in a deep geologic repository, which would avoid necessitating development of facilities and processes for conversion to plutonium oxide. DOE officials said that this fuel could also be disposed of through other to-be-determined disposition paths. Currently, EM manages the spent nuclear fuel that contains 7 MT of this surplus plutonium at various locations throughout the country. NNSA’s 2018 conceptual plan calls for converting 26.2 MT of surplus plutonium into oxide by 2045. In September 2019, NNSA approved the production of about 1.2 MT of plutonium oxide through 2025 at LANL. However, plans for converting additional surplus plutonium into plutonium oxide are uncertain primarily because of two issues. These issues are (1) NNSA’s plans for new pit production, which are still in development and which will also take place at LANL; and (2) issues surrounding the agency’s ability to ship newly produced plutonium oxide for dilution to DOE’s Savannah River Site (SRS) in South Carolina. According to agency officials, NNSA and DOE are taking several actions that, if successfully implemented, are designed to allow NNSA to meet its long- term plutonium oxide production goals. These actions include continuing to review plutonium oxide and pit production plans, increasing plutonium storage at LANL, reducing the amount of SRS’s surplus plutonium, and shipping the diluted plutonium from SRS to WIPP. NNSA’s 2018 conceptual plan called for expanding plutonium oxide production capacity in PF-4 for the dilute and dispose strategy to achieve production of 1.5 MT per year by 2033. NNSA planned to sustain this rate of production at LANL for 12 years to convert a total of 26.2 MT of pits to plutonium oxide before ramping down operations in 2045. The agency’s 2018 conceptual plan estimated that this increased production would cost approximately $5 billion over the life of the program. To achieve the 1.5 MT annual production rate, NNSA planned to expand the physical space of ARIES’s operations in PF-4 by about 50 percent, install new equipment such as glove boxes, purchase additional equipment, such as spare parts and new shipping containers, and hire over 200 new staff. To accommodate the larger workforce, NNSA also planned to construct a new employee entrance in PF-4. In September 2019, NNSA approved a short-term plan to produce a total of nearly 1.2 MT of plutonium oxide at PF-4 from 2019 through 2025. This short-term plan closely matches the total plutonium oxide production outlined in NNSA’s 2018 conceptual plan for the same time frame. In February 2019, NNSA officials said that they were reevaluating the agency’s long-term plutonium oxide production goals in the 2018 conceptual plan because of two key issues. These issues are space constraints relating to (1) the agency’s mission to produce new pits in PF- 4 and (2) requirements to remove plutonium from SRS. According to agency officials, NNSA and DOE are taking several actions designed to allow NNSA to meet the long-term plutonium oxide production goals described in its 2018 conceptual plan. As we reported in November 2018, NNSA officials said that a planned nuclear weapons refurbishment and future warhead programs will require the production of new pits. Almost all of the pits in the current U.S. nuclear weapons stockpile were produced before 1990, according to a May 2015 Congressional report. In May 2018, NNSA announced that it intended to build 30 pits annually in PF-4 at LANL by 2026 and 50 pits annually at the MFFF at SRS by 2030, under a plan to repurpose the MFFF for pit production. According to an August 2019 LANL presentation to potential subcontractors, this effort will include the installation of more than 140 new gloveboxes or other enclosures in PF-4 and the construction of more than 700,000 square feet of supporting infrastructure (such as offices, a parking garage, and a cafeteria). The President’s budget for fiscal year 2020 includes over $3 billion for this effort through 2024. In April 2019, the NNSA Administrator said meeting pit production requirements was the agency’s highest infrastructure priority. NNSA also may have to increase pit production at LANL beyond 30 pits per year. For example, in May 2018 the Nuclear Weapons Council stated that it was essential that NNSA provide resources for surge pit production capacity in PF-4 at LANL until pit production is fully established at SRS. In addition, the National Defense Authorization Act for fiscal year 2019 requires the Department of Defense and NNSA to contract with a federally funded research and development center to conduct an assessment of, among other things, a strategy for producing 80 pits per year at LANL. NNSA officials told us in February 2019 that as a result of pit production requirements, the agency might need to use a portion of the processing areas in PF-4 for pit production that the agency had planned to use for plutonium oxide production. Pit production requirements also may use more space in the high-security vault in PF-4 where plutonium must be temporarily stored. Also in February 2019, NNSA officials said that PF-4’s high-security storage space is already near full capacity and that pit production may demand storage space that NNSA had planned to use for plutonium oxide production. NNSA officials said that the agency is taking some actions that are designed to address increasing both pit and plutonium oxide production in PF-4. If successfully implemented, these actions are designed to allow the program to meet the milestones described in the 2018 conceptual plan, according to NNSA officials. These actions include: Reviewing use of operational space in PF-4. LANL reported in March 2019 that the requirement to produce 30 pits per year would have no significant negative impact on plutonium oxide production. However, LANL reported that a number of programs, including pit production, were planning to increase operations in PF-4, placing demands on the aging facility that could lead to more frequent maintenance outages. In August 2019, NNSA officials responsible for plutonium oxide production and pit production said they continue to believe that increased oxide production and pit production can be simultaneously accomplished in PF-4 but that they are continuing to review the issue as the agency’s pit production plans evolve. In NNSA’s comments on our report, the NNSA Administrator said the agency was working to balance the needs of both missions. The Administrator also noted that NNSA’s Office for Cost Estimating and Program Evaluation will assess the effect of plutonium oxide production on pit production as required by section 3120 of the National Defense Authorization Act for fiscal year 2019. The conference report accompanying the act also requires that we review this assessment, which we will initiate in late 2019. Increasing plutonium storage capacity. LANL also reported in March 2019 that it planned to implement several mitigation measures that would allow the storage of more plutonium oxide and other materials in the PF-4 vault. In addition, DOE and NNSA have “swapped” 1 MT of the declared surplus plutonium at SRS with 1 MT of plutonium residues and other primarily non-pit plutonium already stored in LANL’s PF-4 vault. NNSA officials said that the plutonium residues and other primarily non-pit plutonium at LANL would be considered surplus plutonium and would be converted to plutonium oxide, requiring less storage space. Without these mitigation measures, the PF-4 vault would fill up years earlier, according to NNSA officials. NNSA officials said they believe the swap will increase storage space through 2028, at which point LANL would need to ship plutonium oxide to SRS or face a suspension of plutonium oxide production. Storing quantities of plutonium oxide in PF-4’s high-security storage vault is critical because, according to NNSA officials, it is not likely that NNSA will ship plutonium oxide or other forms of plutonium to SRS until a dispute with the state of South Carolina is resolved. Specifically, the National Defense Authorization Act for fiscal year 2003 required DOE to prepare a plan for the construction and operation of the MFFF at SRS so that it could produce MOX fuel at an average rate of at least 1 MT per year. As subsequently amended, the law provides that if DOE did not meet this 1 MT production objective by January 1, 2014, then it was required to remove 1 MT of defense plutonium from South Carolina by January 1, 2016. If DOE missed that deadline, it was required to make substantial payments to South Carolina until the removal was completed. As NNSA faced delays and cost increases in constructing the MFFF and began to reevaluate its surplus disposition strategy, South Carolina sued DOE in February 2016 to begin removing plutonium from the state and to begin to make payments to the state of up to $100 million per year until the surplus plutonium is removed. In December 2017, the court ordered DOE to remove 1 MT of plutonium from South Carolina by 2020. In response, according to court filings, NNSA moved 0.5 MT of plutonium from SRS to its Nevada National Security Site prior to November 2018 and moved another 0.5 MT of plutonium off-site in August 2019. DOE is still required by statue to remove an amount of defense plutonium or defense plutonium material equal to that which was transferred to SRS after April 15, 2002, but not processed by the MOX facility by January 2022. The officials told us that because of this continuing requirement and the threat of further lawsuits by South Carolina, it was unlikely that NNSA could ship plutonium oxide to SRS until the surplus plutonium at SRS is removed. NNSA officials said that the agency is taking some actions designed to address these issues. These actions include: Increasing plutonium oxide production rates with a priority on oxidizing plutonium material from SRS. NNSA officials said in August 2019 that they are in discussions with LANL to increase the short-term production of plutonium oxide to speed the removal of surplus plutonium from South Carolina. According to NNSA officials, NNSA and LANL are considering increasing plutonium oxide production through 2025 beyond what is called for in their short-term plan that the agency approved in September 2019. This would involve shipping additional surplus plutonium metal from SRS to LANL and prioritize converting this material to plutonium oxide. According to agency officials, LANL would produce additional plutonium oxide production by using new ARIES equipment installed in PF-4 in 2019. To achieve this increased production, NNSA officials said that LANL would need to hire 70 personnel through 2025 to operate ARIES. Agency officials said that these steps would increase total plutonium oxide production to approximately 2.1 MT through 2025, an increase of nearly 1 MT over the short-term plan NNSA approved in September 2019. Increasing dilution and disposal rates of the inventory of plutonium oxide already at SRS. DOE and NNSA officials said that they would also increase dilution of existing plutonium oxide at SRS beyond what is called for in the 2018 conceptual plan to help reduce the inventory of plutonium metal already there. In April 2019, NNSA officials said their current dilution rate at SRS was about 20 kilograms (0.02 MT) annually, but that they plan to increase that rate to 1.5 MT by the late 2020s. Under its 2018 conceptual plan, NNSA had planned to achieve that dilution rate by 2031, but the budget request for NNSA for fiscal year 2020 shows that NNSA plans to complete installation of the capability necessary to achieve that dilution rate by as early as fiscal year 2028. The effort—known as the Surplus Plutonium Disposition project—has an estimated cost range from $200 million to $589 million. It includes removing unnecessary equipment from SRS, accelerating the project’s construction schedule, installing long-lead procurement items early in construction, and hiring and certifying additional personnel. According to NNSA officials, this increase in dilution capacity by 2028 would enable NNSA to begin shipping plutonium oxide to SRS for dilution and disposal without suspending plutonium oxide production at PF-4. While NNSA is taking actions to address pit production and shipment issues, the agency continues to work on refining the long-term plutonium oxide production goals in its 2018 conceptual plan. However, NNSA officials said that establishing firm long-term plutonium oxide production plans now would be premature and that the agency would use the next several years to balance plutonium oxide production, pit production, and shipment issues as they refine long-term production plans. We provided a draft of this report to NNSA and DOE for review and comment. In its response to our draft report, reproduced in appendix V, NNSA said that it and DOE are working to balance the needs of its dilute and dispose program, which includes oxide production, and pit production, as well as the need to remove plutonium from the state of South Carolina. NNSA said, as noted in our report, that its Office for Cost Estimating and Program Evaluation would assess the effects of increased plutonium oxide production on pit production. NNSA also said that even with delays in production of plutonium oxide, the dilution and disposition of surplus plutonium will still be substantially less expensive than if the agency had maintained its MOX fuel approach. As stated in our report, we have a large body of work that has examined the MOX fuel approach, NNSA’s management of the MOX project, and DOE’s $17 billion cost estimate to complete the project, which we assessed as being reliable. In addition, NNSA provided us with technical comments and additional documentation, which we incorporated into our report as appropriate. Some of the information that NNSA provided helped clarify near-term plutonium oxide production plans as well as the agency’s progress in balancing the plutonium oxide production plans, pit production, and the need to move plutonium out of the state of South Carolina. This information is incorporated in our report and is reflected in the report’s revised title. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Our report (1) determines the amount of surplus plutonium in the Department of Energy’s (DOE) inventory that could be converted to plutonium oxide for dilution and disposal and (2) examines DOE’s capacity to produce plutonium oxide. To determine the amount of surplus plutonium in DOE’s inventory that could be converted to plutonium oxide for dilution and disposal, we reviewed relevant DOE documents and interviewed officials from DOE, including from DOE’s National Nuclear Security Administration (NNSA) and DOE’s Office of Environmental Management (EM), on the amounts and forms of surplus plutonium in DOE’s inventory that would require conversion to an oxide prior to final disposition. Our review included DOE’s plans for converting surplus plutonium to plutonium oxide beginning in 1997, when DOE first decided to convert surplus plutonium to plutonium oxide for disposition. We also visited the Los Alamos National Laboratory (LANL) in New Mexico to review documentation and interview officials in the Surplus Plutonium Disposition Program for information on past and current inventories of surplus plutonium. NNSA’s Advanced Recovery and Integrated Extraction System (ARIES), the program that currently converts surplus plutonium to plutonium oxide, resides in Plutonium Facility-4 (PF-4) at LANL. To examine DOE’s capacity to produce plutonium oxide, we reviewed relevant DOE documents and interviewed officials from DOE, including from NNSA and EM, on the status of plutonium oxide production in PF-4 and at DOE’s Savannah River Site, where surplus plutonium was converted to plutonium oxide over a 2 1/2-year period. We reviewed relevant DOE documents and interviewed officials from DOE, including from NNSA and EM, on their plans. For example, we reviewed records of decision and environmental impact statements that DOE issued during its management of the Surplus Plutonium Disposition Program. We reviewed planning documents related to the dilute and dispose strategy, including DOE’s life-cycle cost estimate and supporting documents covering issues such as time frames and conversion rates. We visited the ARIES program in PF-4 in January 2018 to review documentation and conduct interviews with officials responsible for plutonium oxide production and the planned expansion of plutonium oxide production. The site visit included a tour of PF-4, ARIES and its operations, and potential spaces in PF-4 for expansion of ARIES operations for converting surplus plutonium metal to oxide. We conducted this performance audit from October 2017 to October 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. DOE first established the Surplus Plutonium Disposition Program in 1997 to dispose of surplus, weapons-usable plutonium at the end of the Cold War. As of April 2019, the United States has declared a total of 61.5 metric tons (MT) of plutonium as surplus to defense needs. DOE has disposed of 3.2 MT of surplus plutonium at the Waste Isolation Pilot Plant (WIPP), an underground repository for transuranic waste located near Carlsbad, New Mexico, and is in the process of disposing of an additional 1.1 MT of surplus plutonium. This leaves 57.2 MT of surplus plutonium in its inventory, as of May 2019. The table below shows the timeline of changes to DOE strategies for managing surplus plutonium for final disposition. 1997 - DOE announces the Surplus Plutonium Disposition Plan, including the Mixed Oxide Fuel Fabrication Facility (MFFF). 2000 - The United States and Russia enter into the Plutonium Management and Disposition Agreement (PMDA), each agreeing to dispose of at least 34 metric tons (MT) of plutonium at a rate of at least 2 MT per year. 2000 - DOE announced it will construct the MFFF. 2002 - The National Defense Authorization Act for fiscal year 2003 requires DOE to prepare a plan for the construction and operation of the MFFF at the Savannah River Site in South Carolina and requires, among other things, that DOE remove 1 MT of plutonium from South Carolina by January 1, 2011, if mixed oxide (MOX) production objectives of an average rate of at least 1 MT per year were not achieved by January 1, 2009. Failure to meet these deadlines would require DOE to make substantial annual payments to South Carolina. 2005 – The Energy and Water Development Appropriations Act for Fiscal Year 2006 extends the original plutonium production and removal deadlines by 3 years (thus making the 1 MT plutonium production deadline January 1, 2012, and removal deadline January 1, 2014). 2014 – The National Defense Authorization Act for fiscal year 2015 requires DOE to issue a report that would study the plan for the MFFF as well as possible alternatives to the MFFF. 2015 - The National Defense Authorization Act for Fiscal Year 2016 requires DOE to carry out an analysis of alternatives for the Surplus Plutonium Disposition Program. 2015 and 2017 - Explanatory statements accompanying fiscal years 2016 and 2017 appropriations legislation contained specific direction to explore design issues associated with the dilute and dispose alternative. 2016 - South Carolina sues DOE in federal district court, contending that DOE failed to meet the MOX-related statutory deadlines. South Carolina sought monetary relief and an injunction compelling the federal government to remove 1 MT of plutonium from the state. 2016 - DOE issues a Record of Decision stating that it would remove plutonium from South Carolina using the dilute and dispose strategy. 2017 - Federal district court issues an injunction ordering DOE to remove 1 MT of plutonium from South Carolina and ordering the parties to negotiate a new deadline. 2017 - The National Defense Authorization Act for Fiscal Year 2018 allowed DOE to terminate construction of MFFF if, among other things, DOE identified an alternative that would cost less than half of the MOX fuel strategy. 2017 - South Carolina and DOE fail to agree on a deadline for removing 1 MT of plutonium from the state, so in December the court imposes a deadline of January 1, 2020. 2018 - Federal appellate court rejects DOE’s appeal of the district court’s order to remove 1 MT of plutonium from South Carolina by January 1, 2020. 2018 - DOE terminates the MOX contract for the government’s convenience. 2019 – DOE acknowledges that it had shipped 0.5 MT of plutonium from South Carolina to Nevada sometime before November 2018 and shipped an additional 0.5 MT out of South Carolina to another state sometime before August 2019. The Nuclear Waste Policy Act of 1982 directed, among other things, that DOE study sites for a repository and that the President evaluate the capacity for the disposal of high-level waste resulting from atomic energy defense activities at one or more repositories developed for the disposal of commercial used (spent) nuclear fuel. In 1985, President Reagan found that there was no basis to conclude that a separate defense high- level waste repository was required. Table 2 shows the changes in plans for developing a high-level waste repository from 2002 through 2018. In addition to the individual named above, the following individuals made contributions to this report: Jonathan Gill (Assistant Director); Robert Sánchez (Analyst in Charge); Antoinette Capaccio; Robert (Scott) Fletcher; Cindy Gilbert; Richard Johnson; Sheryl Stein; Sara Sullivan; and Curtis (Wade) Tanner.", "summary": "The United States has 57.2 MT of weapons-usable plutonium that it has declared surplus and that still requires disposition. This plutonium exists in various metal and non-metal forms, including pits—the central core of a nuclear weapon. To prevent insidious use of this plutonium, DOE plans to disassemble pits into metal; convert the plutonium metal to plutonium oxide, a powder-like substance; dilute it with inert material; and dispose of it at WIPP. In May 2018, NNSA issued a plan conceptualizing the dilution and disposal of 34 MT of surplus plutonium at an estimated cost of $19 billion over the next 3 decades. Under this conceptual plan, pit disassembly and production of plutonium oxide would take place at one facility and dilution would be performed in another, with both operations expanding over the next decade. GAO was asked to review DOE's plans for plutonium oxide production to dispose of surplus plutonium. This report (1) examines the amount of surplus plutonium in DOE's inventory that could be converted to plutonium oxide for dilution and disposal and (2) examines DOE's capacity to produce plutonium oxide. GAO reviewed the inventory of surplus plutonium, plutonium oxide production requirements and production capacity, and DOE planning documents, and interviewed DOE officials. Of the Department of Energy‘s (DOE) inventory of surplus plutonium, about 43.8 metric tons (MT), or 77 percent, is plutonium metal that could be converted to plutonium oxide for dilution and disposal. Of this amount, the National Nuclear Security Administration (NNSA) manages 33.3 MT in the form of pits, DOE's Office of Environmental Management (EM) manages 6.5 MT, and DOE's Office of Nuclear Energy manages 4 MT in the form of reactor fuel. EM manages another 11 percent, or 6.4 MT, of DOE's surplus plutonium that is already in oxide form. Most of this is suitable for dilution and disposal at the Waste Isolation Pilot Plant (WIPP), a repository in New Mexico. An additional 12 percent, or 7 MT, of DOE's surplus plutonium is contained in spent nuclear fuel that is planned for disposal in a geologic repository. See figure. NNSA's 2018 conceptual plan calls for converting 26.2 MT of this surplus plutonium into oxide by 2045. In September 2019, NNSA approved the production of about 1.2 MT of plutonium oxide through 2025 at its Los Alamos National Laboratory (LANL) located in New Mexico. However, plans for converting additional surplus plutonium into plutonium oxide are uncertain because of two issues. These issues include NNSA's still-developing plans for new pit production, which will also take place at LANL, and issues surrounding the agency's ability to ship newly produced plutonium oxide for dilution to DOE's Savannah River Site (SRS) in South Carolina. According to agency officials, NNSA and DOE are taking several actions that, if successfully implemented, are designed to allow NNSA to meet its long-term plutonium oxide production goals. These actions include continuing to review plutonium oxide and pit production plans, increasing plutonium storage at LANL, reducing the amount of SRS's surplus plutonium, and accelerating the shipment of diluted plutonium from SRS to WIPP.", "document_type": "gao"}
{"report": "The federal disaster workforce is designed to scale up or down depending on the timing and magnitude of disasters. Specifically, FEMA has the authority to augment its permanent full-time staff with temporary personnel and deploy non-FEMA staff members when needed. FEMA has historically relied on both permanent and temporary staff members to respond to presidentially declared disasters. FEMA’s disaster workforce is organized according to position categories, employee types, functions, and job titles. Every FEMA employee is assigned to one or more of four position categories. Staff assigned to incident management positions deploy to disaster sites to administer federal emergency response and recovery programs. Staff assigned to the other three position categories—incident support, ancillary support, and mission essential—provide support services to deployed incident management staff, as well as to FEMA more generally. For example, incident support staff assist with disaster operations at the regional or national level, while mission essential staff maintain basic agency operations, such as payroll and information technology. FEMA has several different employee types that operate out of the agency’s national headquarters, regional offices, and joint field offices at specific disaster locations. Each of the different employee types hold one or more of the four position categories described above. Permanent full-time employees are steady-state federal employees that support FEMA’s mission areas and operations on a daily basis. Cadre of On-Call Response/Recovery Employees (CORE) are a type of temporary full-time employee hired to directly support response and recovery efforts related to disasters for a 2- to 4-year term. These positions may be renewed if there is ongoing disaster work and funding is available. Incident Management COREs are a type of CORE employee that maintain a regular state of readiness to provide emergency-state support and can be deployed up to 300 days a year in mission areas. Incident Management Assistance Teams are rapid-response teams of CORE employees that deploy to disaster sites with little to no notice and remain at disaster sites for unspecified amounts of time, depending on mission needs. Members generally receive 4-year appointments. Reservists are on-call employees who work intermittently as required during incident management operations. Reservists must be available to deploy as needed on 24 hours’ notice at all times during their 24 month appointment. FEMA also has the authority to augment its disaster workforce with temporary employees. This includes local hires, Surge Capacity Force volunteers, and FEMA Corps members. FEMA further augments its workforce with technical assistance contractors, who are specialized contractors hired to perform specific responsibilities. See figure 2 for more information on FEMA’s employee types. As shown in figure 3, reservists made up the largest portion of FEMA’s deployed workforce during peak deployments for the 2017 and 2018 disaster seasons. In 2017, reservists made up about 32 percent of FEMA’s deployed workforce, followed by COREs, permanent full-time staff, and local hires. In 2018, reservists made up about 44 percent of FEMA’s deployed workforce, followed by local hires, COREs, and permanent full-time staff. FEMA’s incident management workforce is organized into 23 cadres. Cadres are groups organized by operational or programmatic functions. They are composed of full-time equivalent and intermittent staff members who perform incident-related duties during disaster response. For example, the Public Assistance cadre administers financial assistance to state, local, tribal, and territorial communities for debris removal, implementation of emergency protective measures, and permanent restoration of infrastructure. FEMA’s incident management workforce performs functions to support its response, recovery, and mitigation missions. Each cadre supports at least one of these missions, and some cadres perform functions across more than one. Cadres also generally deploy to an incident at varying points in the response and recovery phases, depending on their functions. For example, FEMA officials said that the Logistics cadre, which coordinates and monitors all aspects of resource planning, movement, and order tracking, typically deploys staff to an incident before the Hazard Mitigation cadre, which supports risk reduction activities later during the recovery phase. See figure 4 for an example of peak deployment by cadre during Hurricane Florence and appendix II for a description of each cadre and their primary duties. FEMA manages the staffing, training, and deployment of its cadres at the national level. FEMA employees whose primary responsibilities are incident management and disaster response, such as Incident Management CORE and reservists, are generally considered national assets and may be deployed to a disaster anywhere in the country, regardless of permanent duty station. FEMA organizes incident management positions into four tiers denoted by increasing levels of leadership managerial responsibilities and further categorizes senior leaders and officers by level of disaster complexity experience. See figure 5 for more information on FEMA’s position tiers and titles. All FEMA incident management employees have a primary title, which specifies their principal roles and responsibilities, and may also hold subordinate titles for additional roles and responsibilities that the employee can perform. Incident management staff members have one primary incident management title and may have multiple subordinate titles. FEMA may also assign specialties—categories used to identify a specific measured (documented or credentialed) skill, task, experience, or certification that may enhance performance of an associated position—to certain staff. For example, a full-time equivalent staff member who works day-to-day in FEMA’s Office of Policy and Program Analysis could hold a primary incident management title as a Facilities Manager in FEMA’s Logistics cadre and a subordinate title of Logistics Specialist in the same cadre, and may be certified to operate certain types of forklifts. FEMA designed and implemented the FEMA Qualification System in 2012 to standardize and manage the agency’s incident workforce capabilities through prerequisite experience, training, and demonstrated performance. FEMA uses the system to track requirements for incident management positions and the proficiency level of staff members in those positions. According to the 2019 FEMA Qualification System Guide, training and demonstrated performance are foundational elements of the system. Required qualification system training consists of courses designed to build competency in specific position responsibilities and is offered in a variety of settings, such as the Department of Homeland Security (DHS) Center for Domestic Preparedness or at a joint field office, and through a variety of mediums, such as in a classroom, online, or on the job. Demonstrated performance involves validation of the ability to successfully and independently perform specific tasks. According to FEMA, employees conduct required training concurrently with demonstrated performance so that training builds on previous experience and coursework. After FEMA assigns an incident management position to staff, they are issued an electronic position task book, which lists and tracks the training and demonstrated performance requirements for that position. Tasks in the position task book need to be signed off by a coach-and- evaluator—an individual that is trained and designated as a subject matter expert by their cadre to evaluate one or more FEMA Qualification System positions. After staff members have worked with a coach-and- evaluator to complete the tasks and trainings included in their task book, they may submit it to cadre management as part of their qualification application package. Submitted qualification packages go through a number of rounds of review before a final decision is made. First, FEMA’s Qualification System Branch conducts an initial review to validate qualification package completion and requirement waivers, among other things. The branch then forwards the qualification package to cadre management for review. Cadre management reviews employees’ applications for all positions, including specialists and first-line supervisors, and a Qualification Review Board also reviews employees’ applications for leadership, upper management, and middle management positions. See figure 6 for an overview of FEMA’s qualification system process. A regional or national Incident Management Assistance Team is generally among the first FEMA units to arrive on the ground at a disaster site to, among other things, set up federal facilities, establish a joint field office, and coordinate with officials from the impacted region and other relevant federal, state, tribal, territorial, or local partners. If there are staffing shortages among regional full-time equivalent staff members, FEMA leadership in the region where the disaster occurs may submit a deployment request for additional incident management staff members through the Deployment Tracking System. After the Incident Management Assistance Team stands up a joint field office, the Federal Coordinating Officer assumes authority over all emergency response and recovery efforts for the disaster, which includes filling staffing needs. To determine the number and type of incident management personnel needed in each position to meet FEMA requirements for any given disaster, the Federal Coordinating Officer coordinates with regional leadership, the joint field office’s Chief of Staff, and cadre management. The basis of this determination is a variety of factors related to the nature and scope of the disaster. For example, Individual Assistance and Public Assistance needs are based in part on preliminary damage assessments to determine the level of program assistance that may be required. To fill identified staffing needs, field leadership uses a standard process to request specific FEMA Qualification System titles and proficiency levels. Once a standard deployment request is approved, the Deployment Tracking System identifies staff members that match the requested positions, skill sets, and qualification status using a preprogrammed, automated process. The Deployment Tracking System then notifies staff members selected in a rotational order based on length of time since their last deployment, among other things. If an employee declines a deployment request, the Deployment Tracking System automatically sends a request to the next staff member with that incident management position title on the deployment order list. Standard deployment requests are filled by deploying employee types in the following order: 1. Incident Management COREs 2. Reservists 3. Full-time equivalent employees other than Incident Management At the incident, the Federal Coordinating Officer and other field leadership staff are responsible for overseeing coordinating the responders working for FEMA. Generally, after response operations and programs are initiated, staffing needs may change. At this point, field leadership may decide to demobilize some personnel deployed in certain cadres. The decision to do so is based on workload, complexity of operations, and disaster needs. According to FEMA’s 2017 Hurricane Season After-Action Report, FEMA did not meet its annual staffing target for disaster personnel during the 2017 hurricane season. FEMA uses force structure and force strength targets to estimate staffing requirements for incidents and analyze the number of staff the agency has available against these targets. FEMA establishes a longer-term target for the number of incident management staff needed to meet mission needs, called force structure, and tracks the actual number of incident management staff who can deploy at a point in time, which it calls force strength. FEMA uses its force strength measure to track progress towards meeting staffing goals set out in the force structure target and also sets interim targets each fiscal year for reaching the longer-term force structure target. In 2015, FEMA’s Workforce Management Division conducted a review of FEMA’s workforce in coordination with the 23 cadres and adopted a force structure target of 16,305. According to FEMA, this target was established based on a number of considerations, including potential incident levels and historical staffing data for incident management staff deployed to different level events. The agency’s force strength at the end of fiscal year 2017 was 11,656. On average, reservists had the largest gap between force strength and established annual targets. For example, at the end of fiscal year 2017, FEMA’s force strength for reservists was 6,793, which was 15 percent short of its target of 7,982 for that year. In 2019, FEMA’s Workforce Management Division completed a similar review of its incident management workforce and updated its force structure target to 17,670 incident management personnel, which it aims to achieve by 2025. This new target was established using an updated methodology based on information on historical disasters and deployed incident management staff, along with input from each cadre’s management and other officials with expertise on staffing patterns throughout disasters. According to FEMA’s 2017 Hurricane Season After-Action Report, FEMA faced shortages across over half of its cadres when disasters made landfall or began during the 2017 season, and we found that FEMA faced similar shortages during the 2018 disaster season. For instance, according to FEMA’s deployment data, 18 of 23 cadres operated with 25 percent or fewer staff available to deploy when Hurricane Maria made landfall shortly after Hurricane Irma hit Florida, including the Individual Assistance, Logistics, and Hazard Mitigation cadres. See figure 7 for more information on cadre staffing levels at the start of major disasters during the 2017 and 2018 disaster seasons. In addition, many staff members who showed availability to deploy declined when requested to do so. For example, according to FEMA officials, the austere conditions in Puerto Rico and fatigue from previous deployments to hurricanes Harvey and Irma contributed to the high declination rate for Hurricane Maria deployment requests. In addition, FEMA officials stated that permanent full-time employees may not consistently update their availability or may be unavailable to deploy because of their steady-state responsibilities. Further, reservists may decline deployment requests because of the lack of employment protections, which can create difficulties with leaving their jobs to work intermittently in disasters. See table 1 for the declination rates for eight major disasters during the 2017 and 2018 disaster seasons. According to FEMA officials, the agency’s staffing shortages may have originated in part from policy changes in 2012. Specifically, officials said that a large number of incident management staff left the agency after changes were made to the agency’s reservist program and qualification system for disaster personnel in 2012. For instance, officials told us many reservists with years of experience and technical skills left FEMA when the reservist program increased the number of days they were required to deploy or when many were assigned to positions in the qualification system with lower levels of responsibility than what they previously held in order to meet force structure targets. FEMA took various actions to address the staffing shortages during the 2017 and 2018 disaster seasons and used new approaches to augment its workforce. For example, in 2017, FEMA reported that it coordinated the deployment of 2,740 Surge Capacity Force volunteers from eight DHS components. DHS also expanded the Surge Capacity Force to other federal agencies outside DHS for the first time in 2017, including 34 federal departments and agencies in the program, increasing the Surge Capacity Force by 1,323 employees. Additionally, FEMA used local hires to augment its workforce and expedited the local hiring process in response to hurricanes Harvey, Irma, and Maria, resulting in the onboarding of 4,095 local hires from August to November 2017. The Federal Coordinating Officer who initially managed the Puerto Rico joint field office instituted a goal of having a staff composed of 90 percent local hires for recovery efforts. According to the official, investing heavily in local hires and converting them to COREs will help build FEMA’s disaster workforce for long-term Puerto Rico recovery efforts and any future disasters that may occur in the region. As mentioned previously, FEMA also conducted a review of its incident management workforce in 2018 to determine the force structure needed to effectively respond to disasters moving forward. FEMA officials we spoke with said the agency has taken several steps to meet its new force structure, such as establishing a program management office that is dedicated to achieving the agency’s staffing targets. Cadre management officials we spoke with said that FEMA has hiring initiatives in place or planned to help meet the staffing needs established from the review and noted that it will take time for new staff to develop the skills and experience to meet mission needs in the field. FEMA’s qualification and deployment processes did not provide reliable and complete information on staff skills and abilities to ensure its workforce was effectively deployed and used to meet field needs during the 2017 and 2018 disaster seasons. In addition, FEMA lacks mechanisms to assess deployment outcomes or the extent to which it deployed the right mix of staff at the right time to meet mission needs. FEMA field officials in our focus groups and interviews said they experienced a number of challenges with the reliability of information from FEMA’s qualification and deployment processes and systems during the 2017 and 2018 disaster seasons. Specifically, these officials reported that qualification status was not an accurate indicator of ability to perform, which affected disaster assistance delivery and created difficulties with ensuring the right mix of staff were deployed and effectively assigning responsibilities at disaster sites. Officials also reported other challenges with identifying and leveraging staff skills, including lack of information about specialized abilities and expertise. In response to its experience with recent disaster seasons, FEMA has taken or planned some actions to improve its qualification and deployment processes. However, these actions have not been fully implemented and do not fully address the information shortcomings that field officials experienced, as discussed below. FEMA’s qualification and deployment processes and systems do not provide accurate and complete information about staff members’ abilities to ensure field leadership and managers get staff with the right skills at the right time or to most effectively employ and leverage the staff that are deployed to support FEMA’s missions. As discussed earlier in this report, field leadership use the Deployment Tracking System to request staff based on mission needs. The system uses an automated process to select who to deploy from a list of available staff by position and qualification status, and relies on the FEMA Qualification System to identify staff members who are qualified in their positions and those who are trainees. Qualified staff members are expected to be able to successfully and independently perform the duties of their position. However, as shown in table 2, our focus groups with incident management staff and interviews with field and regional officials indicate that disaster personnel experienced significant limitations with qualification status in the FEMA Qualification System matching performance in the field. Very few found that it was a good indicator of staff abilities. For example, participants in two of 14 focus groups described positive experiences with qualification status as an indicator of staff abilities; while, in all 14 groups, participants stated that staff members who were designated as qualified in the system did not have the skills or experience to perform effectively in their positions. Officials in 15 of our 29 field and regional office interviews had similar experiences. For example, Individual Assistance managers in one of the joint field offices we visited said that they had 20 staff members who were designated as qualified but not capable of performing basic tasks, such as knowing how to access the program’s information system. Hazard Mitigation managers from the same joint field office said that about half of their staff who were designated as qualified could not proficiently perform their job duties. Participants in our focus groups and field leadership and managers we interviewed cited numerous operational challenges that resulted from qualification status not being an accurate indicator of staff abilities. Specifically, they stated that (1) staff designated as qualified who lacked the skills and knowledge to perform their positions negatively affected disaster assistance delivery, staff workload, and morale and (2) the unreliability of qualification designations hindered their cadre’s ability to staff disasters with the right mix of staff at the right time and effectively assign responsibilities. Table 3 provides examples of the challenges they experienced. Participants in our focus groups also cited a range of challenges with position task books and the qualification process that in their view contributed to qualification status not being an accurate indicator of staff proficiency. For example: Position task book tasks. In 12 of our 14 focus groups with FEMA incident management staff, participants said the tasks in the position task books did not fully reflect the skills or competencies needed to perform positions. For example, a participant in one focus group said that the tasks in her book did not incorporate sufficient soft skills, such as the ability to communicate with sensitivity and empathy and other interpersonal skills, which are important because staff in her cadre often interact with disaster survivors who have suffered great losses. Coach-and-evaluator process. Participants in 12 of our 14 focus groups also raised concerns with how coach-and-evaluators endorsed tasks, such as lack of consistency and objectivity with signing off on tasks. These issues included coach-and-evaluators signing off on large numbers of tasks too quickly or easily, which participants in 12 focus groups said occurred. Some participants who functioned as coach-and-evaluators said they felt pressure from staff to endorse tasks because reservists receive salary increases when they get qualified. Participants also told us that cadre management may push for staff to be qualified to meet qualification rate targets. A participant in one of our supervisory-level focus groups said he felt pressure from both these sources and admitted to signing off on tasks even though he did not feel the staff member could proficiently perform them. He said that the staff member was qualified in the FEMA Qualification System and later deployed to a smaller disaster, where she was the sole person responsible for her functional area and unable to do the job. Difficulties completing position task books. Participants in all 14 of our focus groups also raised various challenges with completing their task books. These challenges include a lack of available coach-and-evaluators to sign-off on tasks; lack of opportunities to deploy or perform certain tasks; and being unable to complete all the training courses in their task books because classes were unavailable, full, or conflicted with mission needs; among others. As a result, staff members who are able to perform their positions may not be designated as qualified in FEMA’s qualification system. Participants in our focus groups and leadership and managers in our field and regional office interviews reported other challenges with being able to fully identify and use staff skills and experience during disasters. For example: Position titles not fully reflecting staff abilities. FEMA allows staff to have one primary position title in which they are qualified or have an open task book (trainee or candidate status). Officials in 14 of our 29 field and regional interviews and participants in eight of our 14 focus groups with incident management staff raised concerns with this policy. Specifically, officials noted that many employees have experience and expertise in multiple cadres or programs within a cadre, but their full abilities are not reflected in FEMA’s qualification and deployment systems. As a result, field leadership and managers may not be able to fully identify and use the available skills and experience of FEMA’s workforce during disasters, which can limit FEMA’s capacity and flexibility to meet disaster needs, especially when demand is high. For example, one regional official said the Deployment Tracking System has Operations Section Chief as her position title but does not capture her ability to deploy as an Individual Assistance Branch Director, another position in which she has considerable experience. Overly broad position titles and lack of information on specialized skills. In addition, participants in our focus groups told us that some cadre position titles are overly broad (five of 14 groups) and that FEMA’s qualification and deployment systems do not include information on specialized skillsets and experience that would be useful for making deployment and staffing decisions (10 of 14 groups). Officials in 14 of our 29 field and regional interviews raised one or more of these same issues. For example, Logistics managers in one of the joint field offices we visited said that the Logistics Specialist title is too general and captures the majority of warehouse personnel without specifying the actual responsibilities they are able to perform. They noted that, as a result, management needs to query staff members when they arrive to help determine their skills and, in many cases, assign responsibilities by trial and error. According to officials, this can create a safety hazard because some responsibilities require specific skills, such as the ability to operate a certain type of forklift. They also noted that while the Deployment Tracking System allows cadres to input specific skillsets, such as forklift certification, this field has not been consistently filled in. Limitations with fully capturing permanent full-time employee and CORE qualifications. In seven of our eight focus groups with permanent full-time employees and COREs, participants stated that it is not a priority for them to complete their task books because they have little or no incentives to be designated as qualified in the FEMA Qualification System. For example, some participants noted that unlike reservists, their pay and professional development is not directly tied to their qualification status or position. Another participant said that he has been deployed to many disasters and does not have any tasks in his task book endorsed because he is focused on meeting mission needs and does not care enough about being qualified in the system to take the time to complete his task book. Some regional and field officials also raised the same issues. For example, Response Division managers in one of the regions we selected for interviews said that some of the best talent at FEMA, such as staff members with years of experience who work full-time in positions similar to their incident management titles, have never opened or completed a task book because there is no incentive for them to do so. As a result, FEMA may not be fully capturing the qualifications and skills of permanent full-time employees and COREs. FEMA has taken a number of actions intended to help address some of the challenges with its qualification and deployment processes that hindered its ability to provide accurate and complete staffing information to field officials. FEMA headquarters officials acknowledged the challenges we identified with the FEMA Qualification System and noted that the system is still evolving. Key efforts to improve the reliability of qualification designations include: Qualifying staff members who could proficiently perform their positions. During the 2017 hurricane season, FEMA took steps to qualify staff members who were not designated as qualified in the FEMA Qualification System but could proficiently perform the duties of their position. For example, according to the agency’s after-action report for the hurricane season, FEMA temporarily changed qualification procedures during the season to more rapidly qualify employees who had demonstrated their skills outside the traditional process. FEMA headquarters officials stated that this helped qualification designations better reflect the skills and abilities of these staff members. Other actions that FEMA has taken to help qualify staff include allowing cadre management to waive certain tasks or training, allowing specified tasks to be signed-off on during training exercises, and, as discussed later in this report, conducting a pilot on deploying staff to specifically serve as coach- and-evaluators during disasters. Revising position task books. FEMA headquarters officials said they began reviewing task books in spring 2017 to help ensure that tasks are measurable and better align with the knowledge, skills, and abilities needed to perform positions. Officials said this project was completed in November 2018 and the revised task books have been implemented. They noted that this will help streamline the qualification process, allow for more objective evaluation, and help ensure tasks better reflect the skills needed on the job. According to FEMA officials, they plan to continue to work with the cadres to ensure task books align with the skills and competencies required to complete disaster missions. Enhanced coach-and-evaluator training. FEMA revised its training for coach-and-evaluators to provide more guidance and tools for how to accurately evaluate staff and improve the integrity of the evaluation process. Specifically, in October 2017, FEMA updated the coach-and- evaluator training class and added material on, for example, techniques for evaluating with integrity, types of observation, and documenting task performance by including comments in the task books. All staff members must pass the class by performing a capstone exercise and taking a written exam before being able to serve as a coach-and-evaluator. Additional controls in the qualification process. Since 2017, FEMA has established additional controls to provide more oversight on the qualification process. For example, headquarters officials said that as part of the qualification review process, officials may review the qualification packages to check how many tasks were endorsed during a given period of time. If the number is unusually large, they may note it for cadre management to consider when making qualification decisions. This step was incorporated in the new FEMA Qualification System Guide that was issued in August 2019. The guide also includes changes to the Qualification Review Board process, such as requiring candidates for leadership and upper-level management positions to attend the review in person and answer questions about their deployments, training history, and task book completion. FEMA has also taken some initial actions and considered options to better identify and use staff skills and experience in the field. For example, FEMA headquarters officials said they are aware that limiting staff to one primary position or one open task book may not fully capture their experience and abilities and are exploring ways to address it. However, they stated that this is a complex issue and allowing staff to hold multiple primary positions could affect the time it takes for staff to complete task books and, on a broader level, pay scales, career progression paths, and training budgets. They also noted that this could create complications with how to deploy staff if multiple cadres need positions filled during times of scarce resources. FEMA headquarters officials told us that staff can be deployed in positions other than their FEMA Qualification System positions but acknowledged that because these other positions are not systematically recorded in the Deployment Tracking System, leadership would need to be aware of staff skills and abilities through informal means. Further, FEMA headquarters officials said that as part of its review of the incident management workforce, they have revised the position titles for certain cadres, which they noted could potentially result in the titles better reflecting staff roles and responsibilities. Officials added that they need to balance the enhanced staffing information that more specific titles provide with the flexibility of broader titles, particularly for entry-level positions. FEMA has also included information on assigning specialized skills to staff in the Deployment Tracking System in its new FEMA Qualification System and deployment guides. While FEMA has taken some steps to improve its qualification and deployment systems, its efforts primarily affect the qualification process moving forward and do not fully address field officials’ experiences with not having reliable information on staff qualifications and skills to effectively use the available workforce to meet mission needs. For example, the changes to the position task books, coach-and-evaluator program, and FEMA Qualification System guide do not affect staff members who have already been qualified in the system but cannot perform their duties, and as discussed later in this report, FEMA currently does not have an effective performance evaluation system in place to identify and address skill deficiencies. In addition, the agency has not taken actions to address the challenges with identifying staff who can serve multiple incident management positions and fully capturing the qualifications of permanent full-time employees and COREs. Also, headquarters officials stated that FEMA has revised some of its position titles, but not all the cadres that reported challenges with overly broad titles had adjustments made to their positions. Further, this initiative is in the early stages of implementation and it is too soon to assess whether the revised positions will provide better information to field officials on staff members’ specific responsibilities. Further, the lack of reliability of qualification status as an accurate indicator of staff skills and abilities has been a persistent issue with the FEMA Qualification System. For example, we stated in our 2015 report on FEMA workforce management that in five of 11 focus groups with permanent full-time employees and COREs, participants cited concerns about the implementation of the FEMA Qualification System, and some observed employees whose training and experience did not reflect the position and qualification level to which they were assigned. Also, in a 2016 report on the reservist workforce, the DHS Office of the Inspector General stated that in five of the seven disaster deployments included in their review, joint field office staff encountered problems obtaining capable reservists with position titles under the FEMA Qualification System. These officials said that reservists sometimes lacked the experience and training to perform their duties, and as a result, were reassigned to positions outside their system titles. One of the purposes of the FEMA Qualification System is to ensure consistency in skill identification and deployable assets for positions across the agency. In addition, FEMA’s 2018-2022 Strategic Plan states that the agency should guarantee that the FEMA Qualification System maximizes the ability of FEMA staff to use their skills and talents to deliver the best outcomes possible. However, as discussed above, FEMA experienced challenges with achieving these objectives. In addition, Standards for Internal Control in the Federal Government directs management to use quality information to achieve the agency’s objectives. It states that, as part of designing control activities for human capital management, management should continually assess the knowledge, skills, and ability needs of the agency to help achieve organizational goals. According to the standards, only when the right personnel for the job are on board and are provided the right responsibilities, among other things, is operational success possible. In addition, according to The Standard for Program Management, program monitoring, reporting, and controls include the development of plans to respond to identified issues. It also states that program management should include timeframes and milestones for achieving program benefits and obtaining feedback from stakeholders to better understand the concerns related to the program and impact of the program. Given the complexity of FEMA’s workforce and the persistent issues with the reliability of qualification designations and other challenges with identifying the knowledge, skills, and abilities of its staff, FEMA would benefit from developing a comprehensive plan—with timeframes and milestones—to address issues with the quality of information its qualification and deployment processes and systems provide to field officials. Such a plan would also benefit from the inclusion of perspectives from field leadership who depend on the information. FEMA officials acknowledged the staffing information challenges we identified and noted that they have not developed a plan to address them because the issues are multifaceted—changes in policy can potentially affect numerous areas of the workforce—and they had been focused on other initiatives, such as revising force structure targets and streamlining the qualification process. However, they said that such a plan would be useful. Developing a plan to address the challenges that hindered FEMA’s ability to provide reliable and complete information about staff skills to field leaders and managers would better enable the agency to use its disaster workforce as flexibly and effectively as possible to meet mission needs in the field. FEMA does not have mechanisms to assess the extent to which its deployment strategies met mission needs in the field during disasters. FEMA’s Deployment Guide states that for the agency to fulfill its preparedness, response, recovery, and mitigation missions, it must be able to effectively and efficiently deploy its responders through a process that sends the right people to the right place at the right time with the right qualifications. FEMA has measures and collects data related to staffing levels and availability, such as comparing cadre force strength to annual targets, comparing staff qualification rates to targets, determining the percent of staff in each cadre that show availability in the Deployment Tracking System, and tracking the number of staff deployed to disasters. However, none of these measures or data directly demonstrate deployment outcomes or how effectively FEMA deployed available staff to meet mission needs. Headquarters officials said that, among other things, they generally have looked at the number of staff members that were deployed to disasters, as well as declinations, to assess the extent to which they were able to meet staffing needs. They noted that this assumed the number, type, and timing of staff deployments matched field needs. However, our focus groups and interviews with field officials indicate that this was not generally the case. For example, in all 17 of our focus groups, participants experienced challenges with the staffing, skill, or experience levels of the deployed workforce, such as having too few staff members with the right technical skills to perform their missions efficiently and effectively. Further, in 12 of the 17 focus groups we conducted, participants said that there were challenges with the timing of deployments, such as staff from certain cadres being deployed too early or redeploying staff from key positions when the mission need was still high. In most of our interviews with field leadership and managers, officials described similar challenges with the number, skill level, or timing of staff deployments. Participants in our focus groups and field officials we interviewed said they make every effort to meet mission needs despite challenges with staff deployment, but noted that these challenges with deployment outcomes not meeting field needs can increase staff workload and delay disaster assistance, among other impacts and inefficiencies. Our work on strategic human capital management states that effective geographic and organizational deployment strategies can enable an organization to have the right people, with the right skills, doing the right jobs, in the right place, at the right time by making flexible use of its internal workforce. Additionally, Standards for Internal Controls in the Federal Government states that management should establish and operate monitoring activities to continually monitor the internal control system, evaluate results, and remediate any deficiencies identified on a timely basis. As part of remediating deficiencies, the standards advise management to report and evaluate issues that were identified as a result of the monitoring and take corrective actions to address them. As discussed earlier in this report, field leadership request staff based on cadres’ anticipated needs using estimates of the severity of damage and the nature and scope of the disaster, among other factors. However, FEMA headquarters officials told us their data systems cannot determine the extent to which field deployment requests were met during disasters. In addition, these officials noted that they have not established other mechanisms to assess deployment outcomes because this is extremely complex and they are considering how best to do so. They noted that they have been working with in-house data science experts to consider what kinds of measures and metrics they could design to assess deployment outcomes, but they did not have any concrete proposals or time frames for when this might be completed. Without mechanisms to assess deployment outcomes, FEMA officials in headquarters lack critical information to monitor and evaluate the extent to which its deployment policies and strategies effectively placed staff with the right skills in the right place at the right time to meet mission needs in the field. As a result, FEMA may miss opportunities to identify when corrective actions are required to better deploy its workforce to meet field needs, such as adjusting the timing and staging of deployments, and the amount of staff deployed. We found significant shortcomings in FEMA’s ability to ensure staff development—which consists of courses, on-the-job-learning, and coaching and mentoring—for the skills and abilities needed in the field. Specifically, although the current approach to developing staff includes efforts to provide training courses, opportunities for on-the-job training and mentoring, and a performance evaluation system, each of these elements has limitations as implemented, and they are not effectively coordinated to help ensure systematic and comprehensive staff development. Staff and managers cited certain recurrent challenges with staff development in focus groups and interviews, such as (1) limitations on the ability to take useful classroom training, (2) challenges providing or receiving on-the-job training and mentoring, (3) inconsistent use of performance evaluations, and (4) difficulty with ongoing development when not deployed to a disaster. One way staff members develop skills and competencies is through completing required courses in their position task books. However, in 10 of our 17 focus groups, participants discussed barriers to taking courses through FEMA’s qualification system that in their view would help them better perform their jobs, such as being unable to take courses that are not in their position task books or if they are already qualified in their positions. Officials in 11 of the 29 field and regional interviews we conducted raised the same issue. FEMA headquarters officials stated that staff are generally required to obtain cadre management approval before they can register for incident management-related courses that are not specifically listed in their position task books, but staff told us it can be difficult to receive approval because of funding limitations. For example, a Hazard Mitigation official at one joint field office we visited described a situation where a staff member wanted to take a course on mitigation and engineering techniques for coastal construction that would have benefitted the work the person was doing, but was not able to get approval. Participants in our focus groups also told us that staff deployed to a position other than their FEMA Qualification System title had been unable to take courses related to the work they were doing. Moreover, staff members said the FEMA Qualification System limits training opportunities for those already qualified in their positions. For example, some staff members said that once they had completed their position task book, they were sometimes unable to get training that included new information on updated policies or procedures specific to their work. An official in one of the FEMA regions we selected for interviews said that some staff members in the region who were qualified would have preferred to be designated as trainees in the FEMA Qualification System because it would allow them to take relevant courses. In March 2020, FEMA officials told us the agency has recently taken actions to make it easier for cadres to send staff to courses that are not required in their position task book or for positions where the person is qualified. Finally, participants in our focus groups with permanent full-time staff members reported challenges with being able to take courses to develop their incident management competencies. These participants told us it is challenging for them to take disaster-related courses while performing their steady-state work. They said this is because there is no budget for localized disaster-related courses in their offices and it can be difficult to get approval and take time from their duties to travel for this type of training. Focus group participants frequently said developing skills on the job was the most useful type of training they receive. Specifically, participants in 12 of our 17 focus groups said on-the-job training was the most useful kind of training and participants in 13 of the 17 focus groups said this is how they received most of their training. In addition, headquarters officials in the Individual Assistance cadre said one of the benefits of on-the-job training during deployments is that it provides an opportunity for staff to learn and practice their craft in a setting that is difficult to simulate during training. The FEMA Qualification System Guide states that FEMA uses coach- and-evaluators as the primary mechanism for staff to learn the specific skills needed for each position. However, staff members we spoke with said they have difficulties developing their skills through the qualification process. Specifically, in seven of the 17 focus groups, participants told us they did not get feedback or coaching on the job. According to staff in our focus groups, the coach-and-evaluator aspect of the qualification system is not the ideal mechanism to support on-the-job training and development because it often emphasizes the evaluation role over the coaching role. In nine of 14 focus groups, participants told us the position task book process focuses more on completing tasks than on performance, development, or building competencies. Officials in eight of our 29 field and regional interviews reported similar experiences. Some staff who did receive coaching said it was often based on the interest level and time that an individual who was willing to invest and was not done in a systematic or consistent way. Moreover, a commonly cited challenge—in 11 of our 14 focus groups— was the lack of coach-and-evaluators to sign off on position task books. Officials in 16 of our 29 field and regional interviews raised the same issue. Participants in our focus groups said they had difficulties finding available coach-and-evaluators at disaster sites. For example, our analysis of FEMA data found that 36 percent of FEMA’s incident management workforce did not have a coach-and-evaluator at the start of their deployment during the 2017 and 2018 disaster seasons. In addition, according to staff in our focus groups and interviews, coach-and- evaluators at the disaster often do not have time to coach staff. For example, officials at one of the joint field offices we visited said mission needs always come first and coaching and evaluating responsibilities are often not anyone’s priority. In addition to on-the-job training challenges related to the FEMA Qualification System, focus group participants also reported more general challenges with on-the-job training. For instance, multiple supervisors in the Logistics cadre at one joint field office said that in addition to doing their own work, experienced staff members need to spend significant time training others, which competes with performing their mission. Furthermore, participants in seven of the 17 focus groups said providing on-the-job training was particularly challenging at the beginning of a disaster, when the disaster is often hectic and at its busiest. Recovery Division officials in a FEMA regional office said a challenge at the start of the disaster is finding staff members who know what to do and have the time to train those who do not. Staff members also described difficulties with providing and receiving on-the-job training in later phases of a disaster. In one focus group with supervisors, a participant said that once the disaster has reached a pace where they have time to train, staff members are often redeployed. Finally, in 16 of our 29 field and regional interviews, officials said there was a lack of mentoring and sustained staff development across disasters. For example, officials at one joint field office told us that once staff members complete their position task book, they generally do not receive any additional coaching or mentoring in that position. This official stated that reservists have a more difficult time identifying mentors than other employee types because they deploy intermittently and likely have different supervisors and coach-and-evaluators each time they deploy. In addition, FEMA officials said coach-and-evaluators are not meant to serve as mentors. FEMA human capital officials said that different offices can develop their own mentoring programs but these may not be available to all employee types. As a result, not all staff members know to ask for, or expect to receive, mentoring. FEMA headquarters officials acknowledged some of these staff development challenges and described actions they have planned, or are underway, to help address some of them. Specifically, FEMA revised the coach-and-evaluator course in 2017 to place a greater emphasis on the coaching responsibilities of the coach-and-evaluator role. For example, the revised course teaches effective coaching strategies, including how to give effective, actionable feedback. Also, in summer 2019, FEMA conducted a pilot with the National Disaster Recovery Support cadre to deploy a single coach-and-evaluator solely in that position and communicated to cadre management that this individual was not to be used for other disaster-related responsibilities. FEMA officials said this pilot was a success. In evaluating the pilot, FEMA said the coach-and- evaluator was able to devote time to proper training and answering any questions presented. Finally, the agency revised the FEMA Qualification System Guide in August 2019, which included clarifying differences between coaching and evaluating. The revised guide states that, as part of the position task book process, a coach explains, demonstrates, trains, assesses, and documents an individual’s task performance while an evaluator observes, assesses, documents, and endorses an employee’s independent performance of specific tasks. Headquarters officials told us that during the 2017 and 2018 disaster seasons, disaster workforce employees inconsistently received performance evaluations when deployed. Performance evaluations at disasters are to be completed on a paper form by a temporary duty supervisor. If the staff member has a coach-and-evaluator, the temporary supervisor may request input regarding progress toward mastering the skills covered by the position task book. The temporary supervisor is supposed to provide that evaluation to cadre management if an evaluation was completed. However, FEMA officials told us there are no mechanisms in place to ensure these steps occur or that the evaluations will be used to help develop staff competencies, and it is not something FEMA officials monitor. Further, FEMA headquarters officials stated there are no controls in place to ensure supervisors rate staff consistently from supervisor to supervisor. These officials told us they are aware of some problems with how the agency conducts performance evaluations for the disaster workforce and are developing changes to address them. For example, in the months prior to the 2017 disasters, the agency began revising its performance evaluation system, but suspended its efforts when that year’s disasters occurred. In 2019, FEMA resumed this initiative and agency officials told us they expect it will be implemented by June 2020. They said the new system will include replacing the paper form with an electronic program that will be integrated into FEMA’s other personnel systems, such as the Deployment Tracking System. Further, in March 2020, FEMA officials told us they are finalizing a directive intended to provide guidance to supervisors at disasters on how they are to provide deployment performance evaluations. In addition, in April 2020, FEMA issued guidance for the administration, implementation, and oversight of a performance management process that will provide reservists with annual performance appraisals. FEMA officials told us this will help ensure that reservist performance appraisals accurately reflect their job performance and assist them in maintaining and improving performance in the future. The agency’s reservist performance management initiative is expected to be completed by January 2021, but officials have not provided specific interim milestones or target dates. Many disaster workforce staff members are not likely to get ongoing development directly from their cadre management when they are not deployed. According to data from FEMA, there was one cadre supervisor of record for every 128 reservists and Incident Management CORE staff as of June 1, 2019. During the 2017 and 2018 disaster seasons, this ratio was higher in certain cadres. For example, there was one supervisor of record for every 807 reservists and Incident Management CORE staff as of June 1, 2017 in the Individual Assistance cadre. FEMA headquarters officials told us they are assessing what the right mix of supervisors to reservists should be across the cadres. Further, staff members told us they have difficulties getting ongoing development through hands-on training outside of a disaster. While FEMA headquarters officials told us that cadres periodically conduct mission rehearsal trainings each year to prepare their staff for disasters, they also said not all staff can attend them because cadre management determines which staff to invite. These trainings are designed for staff members to simulate a potential disaster scenario while in a training environment. Finally, FEMA headquarters officials stated that receiving ongoing development for staff who do not deploy frequently, such as reservists, can be a challenge. The only instances when reservists are paid while not deployed occur when they complete 40 hours a year of mandatory training or 32 hours a year coordinating with their cadre. In addition, an individual in one of our focus groups with permanent full-time employees said reservists had difficulties accessing online mandatory training because they did not have a FEMA laptop. A recovery manager in a FEMA regional office told us that it can be challenging to provide staff development for reservists because they are generally sent to the field to do a discrete job and have limited opportunities to develop their skills and competencies when not deployed. As discussed above, FEMA’s disaster workforce reported challenges receiving staff development through the agency’s existing methods, which consists primarily of classroom training, on-the-job training and mentoring, and performance evaluations. While FEMA has taken actions to address some of the challenges staff experienced, opportunities remain to ensure more effective and consistent staff development. Specifically, FEMA does not have a staff development program in place to provide assurance of effective and comprehensive staff development of the skills and abilities needed during deployments. Further, FEMA headquarters officials said it is primarily the responsibility of staff members to find available coach-and-evaluators at disaster sites and the agency has not developed a mechanism to help ensure deployed staff are consistently paired with coach-and-evaluators. In addition, FEMA headquarters has not taken actions to address the challenges we identified with the lack of mentoring for staff deployed to disasters. Further, given that FEMA’s performance evaluation initiatives are not yet implemented, it is too early to assess how effective they will be in enhancing staff development, including whether they will have mechanisms in place to ensure employees receive useful evaluations or the extent to which they will be coordinated with other development activities, such as coaching through on-the-job training. Standards for Internal Control in the Federal Government states that management recruits, develops, and retains competent personnel to achieve the entity’s objectives. This includes enabling individuals to develop competencies appropriate for key roles, reinforcing standards of conduct, and tailoring training based on the needs of the role. It also includes mentoring to develop individual performance based on standards of conduct and expectations of competence that align the individual’s skills and expertise with the entity’s objectives and help personnel adapt to an evolving environment. In addition, we have previously reported that identifying where an agency’s development process is lacking can help address barriers that hinder its ability to achieve meaningful results. We also reported that it is important for agencies to treat continuous learning as an investment in success as it can address employees’ career development issues, skill-specific training needs, and provide managers with opportunities to identify where training and development is appropriate. Effective and consistent staff development is particularly important because FEMA has hired a large number of reservists over the past few years. Our analysis of FEMA data shows that from June 1, 2017 to May 31, 2019, the agency hired over 3,200 reservists, which was 40 percent of the agency’s entire reservist workforce as of June 1, 2019. Creating a staff development program that systematically and comprehensively addresses staff development through courses, on-the-job training and mentoring, performance evaluation, and ongoing developmental opportunities would provide better assurance that staff develop the skills and competencies needed to meet mission needs during field operations and help ensure the best results for disaster survivors. The large-scale and concurrent disasters during the 2017 and 2018 disaster seasons highlighted the complex challenges facing FEMA’s workforce. The agency deployed 14,684 and 10,328 personnel, respectively, at the peak of each of these disaster seasons, and the increased demand for its workforce is expected to continue. Without accurate and complete information on the knowledge, skills, and abilities of these staff members, field officials face challenges with efficiently providing disaster assistance, managing staff workload, and assigning responsibilities. FEMA has taken some initial actions to improve the information provided by its qualification and deployment systems, such as establishing additional controls in its qualification process. However, developing a plan to address the information challenges experienced during the 2017 and 2018 disaster seasons would be beneficial to enhance field leadership’s ability to identify and leverage staff skills and, given the persistence of some of these challenges, help ensure they do not continue to affect FEMA’s ability to support mission needs in future disasters. Further, in light of the staffing constraints that FEMA faces, it is important that the agency be able to assess how effectively it deploys available staff to disasters to meet field needs. Developing a mechanism to assess FEMA’s deployment outcomes would provide officials in headquarters with critical information to monitor and evaluate the extent to which its deployment policies and strategies effectively place staff with the right skills in the right place at the right time to meet mission needs and take corrective actions if needed. Finally, creating a staff development program for its disaster workforce that addresses access to training, delivery of on-the-job training and mentoring, use of performance evaluations, and developmental opportunities when not deployed would help FEMA ensure more consistent and comprehensive development of the skills and abilities needed during deployments. Consistent and effective staff development is particularly important to help build the skills of staff who are qualified in the FEMA Qualification System but unable to proficiently perform their duties and develop the large number of staff that FEMA has recently hired to meet its new disaster workforce targets. We are making the following three recommendations to FEMA: The FEMA Administrator should develop a plan—with time frames and milestones and input from field leadership—to address identified challenges that have hindered FEMA’s ability to provide reliable and complete information to field leaders and managers about staff knowledge, skills, and abilities. (Recommendation 1) The FEMA Administrator should develop mechanisms, including collecting relevant data, to assess how effectively FEMA’s disaster workforce was deployed to meet mission needs in the field. (Recommendation 2) The FEMA Administrator should create a staff development program for FEMA’s disaster workforce that, at a minimum, addresses access to training, delivery of on-the-job training and mentoring, use of performance evaluations, and consistent developmental opportunities regardless of deployment status. (Recommendation 3) We provided a draft of this report to DHS for review and comment. DHS provided written comments, which are reprinted in appendix III and summarized below. In its comments, DHS concurred with our three recommendations and provided a number of ongoing and planned actions that it intends to leverage in addressing them. DHS also provided technical comments, which we incorporated as appropriate. With regard to our first recommendation for FEMA to develop a plan to address identified challenges with providing reliable and complete staffing information to the field, DHS reiterated some of the steps described in this report that FEMA has taken to improve the coach-and-evaluator program. DHS noted that FEMA plans to engage field leaders on these initiatives to develop a plan to address identified challenges. DHS also reported that FEMA plans to increase training offerings and align its curriculum so that FEMA Qualification System status matches workforce capability. DHS anticipates these efforts will be completed by March 31, 2021. While these are positive initial steps, they focus solely on the coach-and- evaluator program and staff training. Our report identified a number of complex and interrelated challenges with the agency’s qualification and deployment processes that hindered FEMA’s ability to provide reliable information to field officials about staff members’ skills and abilities, including their qualifications, specialized skillsets, and experience within and across program areas. As such, in developing the plan we recommended, it will be important for FEMA to take a comprehensive approach and consider solutions that may cut across multiple systems and processes. We will monitor DHS’s and FEMA’s efforts in this area to assess the extent to which they fully implement our recommendation. With regard to our second recommendation for FEMA to develop mechanisms to assess how effectively FEMA’s disaster workforce was deployed to meet mission needs in the field, DHS reiterated the actions described in this report that FEMA took to establish new force structure targets for its incident management workforce. DHS also reported that FEMA plans to convene subject matter experts to develop mechanisms that demonstrate how effectively FEMA’s disaster workforce deploys to meet mission needs in the field, which are expected to be completed by March 31, 2021. When they are complete, we will assess the mechanisms to determine the extent to which they address our recommendation. Regarding our third recommendation for FEMA to create a staff development program, DHS reiterated some of the actions FEMA has taken to develop its disaster workforce that were described in this report. Our report identified recurrent challenges with FEMA’s efforts to develop staff through training courses, on-the-job training and mentoring, and performance evaluations and noted that the agency’s current and planned efforts do not fully address these challenges. In creating the staff development program we recommended, it is important for FEMA to consider how its overall control environment and the initiatives it puts in place are coordinated to ensure staff receive comprehensive and consistent development to build the skills needed during disaster field operations. DHS anticipates that FEMA’s efforts to implement our recommendation will be completed by March 31, 2021. At that time, we will assess the agency’s actions to determine the extent to which they address the intent of our recommendation. We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the FEMA Administrator, and other interested parties. If you or your staff have any questions about this report, please contact me at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report addresses (1) how the Federal Emergency Management Agency’s (FEMA) disaster workforce is qualified and deployed, and workforce staffing levels during the 2017 and 2018 disaster seasons; (2) how effective FEMA’s qualification and deployment processes were during the 2017 and 2018 disaster seasons in helping ensure workforce needs were met in the field; and (3) the extent to which FEMA’s disaster workforce receives staff development to enhance skills and competencies to support the agency’s disaster missions. Our review focused on FEMA’s incident management workforce, which is composed of FEMA staff who deploy to disaster sites. We defined the 2017 and 2018 disaster seasons as the time periods from August 23, 2017 through January 31, 2018, and September 7, 2018 through November 25, 2018. The 2017 dates represent the start of the FEMA incident period for Hurricane Harvey through the end of the incident period for the California wildfire season. The 2018 dates represent the start of the FEMA incident period for Hurricane Florence through the end of the incident period for the California wildfires. To address all three objectives, we (1) analyzed documentation and data on incident management workforce qualification, deployment, staffing levels, and development; (2) conducted focus groups with members of FEMA’s incident workforce across a range of employee types— permanent full-time employees, Cadre of On-Call Response/Recovery Employees (CORE), Incident Management CORE, reservists and local hires; and (3) interviewed FEMA officials in headquarters and field and regional offices. We compared the results of our analysis and the information we gathered with Standards for Internal Control in the Federal Government, The Standard for Program Management, FEMA strategic documents and guidance, and our prior reports on strategic human capital management and strategic training and development. We analyzed documentation on how FEMA’s incident management workforce is qualified, deployed, and developed. Documentation included the agency’s 2017 Incident Management Handbook, 2015 CORE Program Manual, 2017 Reservist Program Directive, 2015 and 2019 FEMA Qualification System guides, 2019 Coach-and-Evaluator Program Directive, coach-and-evaluator training materials, 2014 Incident Workforce Deployment Directive, and 2019 Deployment Guide. In addition, we analyzed FEMA’s 2018-2022 Strategic Plan, 2017 Hurricane Season After-Action Report, and documentation on FEMA’s staffing targets for its incident management workforce. We analyzed data from FEMA’s Deployment Tracking System to determine incident management staffing levels, the number of staff deployed to a disaster, the number of incident management staff that had a coach-and-evaluator assigned, and the ratio of managers to incident management staff. We also analyzed data FEMA provides to the National Finance Center to determine the number of new staff the agency hired. To assess the reliability of the data, we interviewed officials at FEMA headquarters about their data quality control procedures and reviewed documentation about these data systems. For the Deployment Tracking System, we also conducted electronic testing and reviewed the data for obvious errors and omissions. We found the data to be sufficiently reliable for the purposes of this report. As shown in table 4, to obtain perspectives on how effectively FEMA qualifies, deploys, and develops its disaster workforce, we conducted 17 focus groups with a total of 129 participants who were deployed in incident management positions during the 2017 disaster season, and in some cases, the 2018 disaster season. The focus group locations were selected based on where staff members who were deployed during the 2017 disaster season were located at the time of our review. We also selected these locations to reflect where the 2017 disasters occurred and to obtain variation in geographic location to the extent possible. Participants were selected using a stratified random sample from a universe of incident management staff members who were deployed to a federally declared disaster during the 2017 hurricane and wildfire season. For each employee type, we conducted separate focus groups with participants in supervisory and nonsupervisory positions so they could speak more freely. We also selected participants to obtain a mix of staff from different cadres and a mix of staff that were qualified and not qualified in the FEMA Qualification System. If selected staff members indicated they could not attend, we replaced them with the next individual on our randomized list who had similar attributes. There were between three to 11 participants in each focus group, with an average of eight in each. These focus group discussions were guided by a moderator who used a structured list of discussion topics. The topics focused on staff members’ perspectives on, and experiences with, the level of staffing and skill sets their team had, how they were trained and developed, and the FEMA Qualification System and its qualification determinations. Supervisors were also asked about their staff’s skill sets, training, and qualification status. Focus group sessions were audio recorded and transcribed. We evaluated the transcripts using systematic content analysis to identify key themes on how effective FEMA’s qualification and deployment processes were in helping to meet field needs and the extent to which staff members received staff development to enhance their skills and competencies. An analyst coded the transcripts and a second analyst validated the coding. Any discrepancies were resolved by both analysts agreeing on the coding of the associated statement by a participant. If needed, a third analyst adjudicated any continued disagreement between coders. The results of our focus group analysis are not generalizable to all incident management staff members. However, they provided valuable first-hand experiences with staffing levels and skill sets during disasters, FEMA’s deployment processes, the FEMA Qualification System and the reliability of its qualification designations, and how well staff were trained and developed. We conducted site visits to FEMA’s joint field offices in Columbia, South Carolina; Durham, North Carolina; Guaynabo, Puerto Rico; and Tallahassee, Florida, to obtain officials’ perspectives on staffing levels and skill sets, the effectiveness of FEMA’s qualification and deployment processes and systems in meeting field needs, and the extent to which FEMA’s deployed staff receive coaching and development to enhance their skills and competencies. Officials we interviewed at the joint field offices included federal coordinating officers; chiefs of staff; training managers; and managers in the Individual Assistance, Public Assistance, Hazard Mitigation, and Logistics cadres, among others. We also interviewed an official who was previously a federal coordinating officer at a federally-declared wildfire in California. In addition, we interviewed leadership and managers for FEMA regions VI, VIII, and X to obtain the perspectives of regional officials on the topics above. In each of the regions, we interviewed the regional administrator and managers in both the response and recovery divisions, among others. We selected the joint field offices and regions to conduct interviews based on our focus group locations and to obtain variation in geographic location and disaster activity. We conducted systematic content analysis of this work using the same approach we used to analyze the focus groups. The results from this analysis are not generalizable to all field and regional officials, but provide important perspectives from leadership and managers on FEMA’s mechanisms to qualify, deploy, and develop incident management staff. In addition, we conducted interviews with multiple senior officials in FEMA headquarters. For example, we interviewed officials in the Field Operations Directorate and management in the Individual Assistance, Public Assistance, and Hazard Mitigation cadres to obtain information about how FEMA’s incident management workforce and staff in their cadres are qualified, deployed, and developed, and how the Deployment Tracking System and the FEMA Qualification System are used for these purposes. We also interviewed officials in the Office of the Chief Component Human Capital Officer to learn how FEMA trains and develops this workforce. We obtained information from these officials on the actions FEMA has taken to address the challenges we identified through our focus groups, interviews with field and regional officials, and data analysis. We conducted this performance audit from June 2018 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Chris Currie, (404) 679-1875 or curriec@gao.gov In addition to the contact named above, Kathryn Godfrey (Assistant Director), Johanna Wong (Analyst-in-Charge), James Cook, Lawrence Crockett, Elizabeth Dretsch, Ricki Gaber, Eric Hauswirth, Tracey King, Ronald La Due Lake, Rebecca Mendelsohn, Amanda Miller, and Adam Vogt made key contributions to this report.", "summary": "During the 2017 and 2018 disaster seasons, several large-scale disasters created an unprecedented demand for FEMA's workforce. FEMA deployed 14,684 and 10,328 personnel at the peak of each of these seasons and reported staffing shortages during the disasters. GAO was asked to review issues related to the federal response to the 2017 disaster season. This report addresses (1) how FEMA's disaster workforce is qualified and deployed, (2) how effective FEMA's qualification and deployment processes were during the 2017 and 2018 disaster seasons in ensuring workforce needs were met in the field, and (3) the extent to which FEMA's disaster workforce receives staff development to enhance skills and competencies. GAO analyzed documentation and data on incident workforce qualification and deployment; conducted 17 focus groups with 129 staff members; and interviewed FEMA officials in headquarters, field, and regional offices. The Federal Emergency Management Agency (FEMA) has established mechanisms to qualify and deploy staff to disasters. For example, the FEMA Qualification System tracks training and task performance requirements for disaster workforce positions and has a process to designate staff as qualified in their positions once they have completed these requirements. FEMA's deployment process uses an automated system to deploy staff members to disasters that match field requests for positions and proficiency levels. The process depends on the agency's qualification and deployment systems to identify staff qualification status and skillsets to meet field needs. However, FEMA's qualification and deployment processes did not provide reliable and complete staffing information to field officials to ensure its workforce was effectively deployed and used during the 2017 and 2018 disaster seasons. Specifically, GAO's focus groups with over 100 incident staff members and interviews with field and regional officials indicate that disaster personnel experienced significant limitations with qualification status matching performance in the field, due in part to challenges with how staff are evaluated through the qualification process. In all focus groups with applicable incident personnel, participants cited issues with staff members who were qualified in the FEMA Qualification System not having the skills or experience to effectively perform their positions. For example, one participant described supervising staff members who were qualified in the system but did not know the eligibility requirements for applicants to receive housing assistance, or what information needed to be included in the applicant's file. In addition, participants in the majority of the focus groups reported challenges with using FEMA's deployment processes to fully identify staff responsibilities, specialized skillsets, and experience. FEMA headquarters officials acknowledged the identified information challenges but said they have not developed a plan to address them in part because of competing priorities. Developing a plan to address identified challenges with providing reliable staffing information to field officials would enhance FEMA's ability to use staff as flexibly and effectively as possible to meet disaster needs. Further, FEMA's disaster workforce experienced challenges with receiving staff development through the agency's existing methods to enhance the skills and competencies needed during disaster deployments—challenges FEMA headquarters officials acknowledged. Specifically, GAO's focus groups and interviews indicate that disaster personnel encountered challenges related to the availability of courses, providing and receiving on-the-job training and mentoring, and consistently receiving performance evaluations. For example, in 10 of 17 focus groups, participants cited barriers to taking courses that in their view would help them better perform their jobs. In addition, participants in seven focus groups stated that they did not receive coaching or feedback on the job. Relatedly, FEMA data show that at the start of deployments during the 2017 and 2018 disaster seasons, 36 percent of staff did not have an official assigned to coach and evaluate task performance—the primary mechanism the agency depends on for coaching. Creating a staff development program would help better ensure FEMA's disaster workforce develops the skills and competencies needed to meet mission needs in the field. GAO is making three recommendations, including that FEMA develop (1) a plan to address identified challenges that have hindered its ability to provide reliable information to field officials about staff skills and abilities and (2) a staff development program for its disaster workforce that addresses training access, delivery of on-the-job training, and other development methods. The Department of Homeland Security concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Resellers maintain large, sophisticated databases with consumer information that can include credit histories, insurance claims, criminal records, employment histories, incomes, ethnicities, purchase histories, and interests. As shown in figure 1, resellers largely obtain their information from public records, publicly available information (such as directories and newspapers), and nonpublic information (such as from retail loyalty cards, warranty registrations, contests, and web browsing). Consumer information can be derived from mobile networks, devices (including smartphones and tablets), operating systems, and applications. Resellers also may obtain personal information from the profile or public information areas of websites, including social media sites, or from information on blogs or discussion forums. Depending on the context, information from these sources may be publicly available or nonpublic. In 1973, a U.S. government advisory committee first proposed the Fair Information Practice Principles for protecting the privacy and security of personal information. While these principles are not legal requirements, they provide a framework for balancing privacy with other interests. In 2013, the Organisation for Economic Co-operation and Development (OECD) developed a revised version of the principles (see table 1). The Fair Information Practice Principles served as the basis for the Privacy Act of 1974—which governs the collection, maintenance, use, and dissemination of personal information by federal agencies. The principles also were the basis for many Federal Trade Commission (FTC) and Department of Commerce privacy recommendations and for a framework for consumer data privacy the White House issued in 2012. As we reported in 2013 and as continues to be the case, no overarching federal privacy law governs the collection, use, and sale of personal information among private-sector companies, including information resellers. There are also no federal laws designed specifically to address all the products sold and information maintained by information resellers. Federal laws addressing privacy issues in the private sector are generally narrowly tailored to specific purposes, situations, types of information, or sectors or entities—such as data related to financial transactions, personal health, and eligibility for credit. These laws include provisions that limit the disclosure of certain types of information to a third party without an individual’s consent, or prohibit certain types of data collection. The primary laws include the following: Fair Credit Reporting Act (FCRA). FCRA protects the security and confidentiality of personal information collected or used to help make decisions about individuals’ eligibility for credit, insurance, or employment. It applies to consumer reporting agencies that provide consumer reports. Accordingly, FCRA applies to the three nationwide consumer reporting agencies (commonly called credit bureaus) and to any other information resellers that resell consumer reports for use by others. FCRA limits resellers’ use and distribution of personal data—for example, by allowing consumers to opt out of allowing consumer reporting agencies to share their personal information with third parties for prescreened marketing offers. Gramm-Leach-Bliley Act (GLBA). GLBA protects nonpublic personal information that individuals provide to financial institutions or that such institutions maintain. GLBA sharing and disclosure restrictions apply to financial institutions or entities that receive nonpublic personal information from such institutions. For example, a third party that receives nonpublic personal information from a financial institution to process consumers’ account transactions may not use the information or resell it for marketing purposes. Health Insurance Portability and Accountability Act of 1996 (HIPAA). HIPAA establishes a set of national standards to protect certain health information. The HIPAA privacy rule governs the use and disclosure of an individual’s health information for purposes including marketing. With some exceptions, the rule requires an individual’s written authorization before a covered entity—a health care provider that transmits health information electronically in connection with covered transactions, health care clearinghouse, or health plan—may use or disclose the information for marketing. The rule does not directly restrict the use, disclosure, or resale of protected health information by resellers or others not considered covered entities under the rule. Children’s Online Privacy Protection Act of 1998 (COPPA). COPPA and its implementing regulations apply to the collection of information— such as name, email, or location—that would allow someone to identify or contact a child under 13. Covered website and online service operators must obtain verifiable parental consent before collecting such information. COPPA may not directly affect information resellers, but the covered entities are potential sources of information for resellers. Electronic Communications Privacy Act of 1986 (ECPA). ECPA prohibits the interception and disclosure of electronic communications by third parties unless an exception applies (such as one party to the communication consenting to disclosure). For example, the act would prevent an internet service provider from selling the content of its customers’ emails to a reseller for marketing purposes, unless the customers had consented to disclosure. However, ECPA provides more limited protection for information considered to be “non-content,” such as a customer’s name and address. Federal Trade Commission Act (FTC Act), Section 5. The FTC Act prohibits unfair or deceptive acts or practices in or affecting commerce. Although the act does not explicitly grant FTC the specific authority to protect privacy, FTC has interpreted it to apply to deceptions or violations of written privacy policies. For example, if a retailer’s written privacy policy stated customers’ personal information would not be shared with resellers and the retailer later sold information to such parties, FTC could bring an enforcement action against the retailer for unfair and deceptive practices. Some states also have enacted laws designed to regulate resellers’ sharing of personal information about consumers. For example, in 2018, Vermont passed a law that contains, among other requirements, consumer protection provisions related to data brokers. Among other things, the law requires data brokers to register annually and prohibits the acquisition and use of brokered personal information through certain means and for certain uses. The scope of consumer privacy protections provided under federal law has remained narrow in relation to (1) individuals’ ability to access, control, and correct their personal data; (2) collection methods and sources and types of consumer information collected; (3) new technologies; and (4) some regulatory authorities. The examples in the following sections are drawn from our earlier reports and remain pertinent today. In our 2013 report, we found that no federal statute that we examined generally requires resellers to allow individuals to review personal information (intended for marketing purposes), control its use, or correct it. The Fair Information Practice Principles state that individuals should be able to know about and consent to the collection of their information and have the right to access the information, request correction, and challenge the denial of those rights. We also reported in 2013 that no federal statute provides consumers the right to learn what information is held about them and who holds it for marketing or look-up purposes. FCRA provides individuals with certain access rights, but only when information is used for credit eligibility purposes. And GLBA’s provisions allowing consumers to opt out of having their personal information shared with third parties apply only in specific circumstances. Otherwise, under federal law, individuals generally cannot require that their personal information not be collected, used, and shared. Also, no federal law we examined provides correction rights (the ability to have resellers and others correct or delete inaccurate, incomplete, or unverifiable information) for marketing or look-up purposes. Our 2013 report also found that federal privacy laws are limited in addressing the methods by which, or the sources from which, resellers collect and aggregate personal information, or the types of information collected for marketing or look-up purposes. The Fair Information Practice Principles state that personal information should be relevant, limited to the purpose for which it was collected, and collected with the individual’s knowledge or consent. Federal laws generally do not govern the methods resellers may use to collect personal information. For instance, resellers, advertisers, and others use software to search the web for information about individuals and extract and download bulk information from websites with consumer information. Resellers or retailers also may collect information indirectly (by combining information from transactions). Current federal law generally allows resellers to collect personal information from sources such as warranty registration cards and surveys and from online sources such as discussion boards, social media sites, blogs, and web browsing histories and searches. Current federal law generally does not require disclosure to consumers when their information is collected from these sources. The federal laws that address the types of consumer information that can be collected and shared are not comprehensive. Under most circumstances, information that many people may consider very personal or sensitive can be collected, shared, and used for marketing. This can include information about physical and mental health, income and assets, political affiliations, and sexual habits and orientation. For health information, HIPAA rule provisions generally apply only to covered entities, such as health care providers. The current privacy framework does not fully address new technologies such as facial recognition technology, privacy issues raised by online tracking and mobile devices, and activities by financial technology firms. The original enactment of several federal privacy laws predates these trends and technologies. But in some instances existing laws have been interpreted to apply to new technologies. For example, FTC has taken enforcement actions under COPPA and revised the statute’s implementing regulations to account for smartphones and mobile applications. One example of how privacy law has not kept pace with changes in technology is the use of facial recognition technology, which involves the collection of facial images and may be employed in a wide range of commercial applications. In our 2015 report we concluded that the future trajectory of this technology raised questions about consumer privacy. We found that federal law does not expressly address the circumstances under which commercial entities can use facial recognition technology to identify or track individuals, or when consumer knowledge or consent should be required for the technology’s use. Furthermore, in most contexts federal law does not address how personal data derived from the technology may be used or shared. The privacy issues stakeholders raised about facial recognition technology and other biometric technologies in use at the time of our 2015 report served as yet another example of the need to adapt federal privacy law to reflect new technologies. As such, we reiterated our 2013 recommendation that Congress strengthen the current consumer privacy framework to reflect the effects of changes in technology and the marketplace. The rise of financial services provided by nonfinancial firms—often referred to as fintech—is another example of how new technology may create privacy concerns. For example, fintech lenders offer a variety of loans such as consumer and small business loans and operate almost exclusively online. In our 2018 report, we noted that while these lenders may still assess borrowers’ creditworthiness with credit scores, they also may analyze large amounts of additional or alternative sources of data to determine creditworthiness. We also found that some fintech firms may collect more consumer data than traditional lenders. For example, fintech lenders may have sensitive information such as consumers’ educational background or utility payment information, and according to certain stakeholders, these data may contain errors that cannot be disputed by consumers under FCRA. Furthermore, some data aggregators may hold consumer data without disclosing what rights consumers have to delete the data or prevent the data from being shared with other parties. A leak of these or other data held by fintech firms may expose characteristics that people view as sensitive. GLBA generally requires fintech firms and traditional financial institutions to safeguard nonpublic personal information about customers. Our 2018 report discussed that some fintech firms use new technologies or mobile device features to mitigate data privacy risks and that some regulators have issued guidance to consumers publicizing practices that help maintain privacy when using online products and services, including those provided by fintech firms. Regulators also have issued GLBA guidance to businesses, including fintech firms, recommending that they adopt policies and procedures to prevent, detect, and address privacy threats. Online tracking. In our 2013 report, we found that no federal privacy law explicitly addresses the full range of practices to track or collect data from consumers’ online activity. Cookies allow website operators to recall information such as user name and address, credit card number, and purchases in a shopping cart. Resellers can match information in cookies and their databases to augment consumer profiles. Third parties also can synchronize their cookie files with resellers’ files. Advertisers can use third-party cookies—placed on a computer by a domain other than the site being visited—to track visits to the websites on which they advertise. While current federal law does not, with some exceptions, explicitly address web tracking, FTC has taken enforcement actions related to web tracking under its authority to enforce the prohibition on unfair or deceptive acts. For example, in 2011, FTC settled charges with Google for $22.5 million after alleging that Google violated an earlier privacy settlement with FTC when it misrepresented to users of Apple’s Safari web browser that it would not track and serve targeted advertisements to Safari users. Google agreed to disable its advertising tracking cookies. Mobile devices. In 2013, we also explained that no federal law comprehensively governs applications software for mobile devices. Application developers, mobile carriers, advertisers, and others may collect an individual’s information through services provided on a mobile device. However, FTC has taken enforcement action against companies for use of mobile applications that violate COPPA and FCRA. The agency also has taken action under the FTC Act. We and others have reported that the capability of mobile devices to provide consumer’s location engenders privacy risks, particularly if companies use or share location data without consumers’ knowledge. ECPA might not apply if location data were not deemed content and would not govern entities that are not covered by ECPA. But FTC could pursue enforcement action if a company’s collection or use of the information violated COPPA. More recently, in January of this year, we issued a report on internet privacy that reinforces what we reported in 2013. To varying extents, internet content providers and internet service providers collect, use, and share information from their customers to enable their services, support advertising, and for other purposes. Consumers access such services through mobile phones and tablets, computers, and other internet- connected devices. However, there is no comprehensive federal privacy statute with specific standards. FTC has been addressing internet privacy through its unfair and deceptive practices authority, among other statutes, and other agencies have been addressing this issue using industry- specific statutes. We concluded that recent developments regarding internet privacy suggest that this is an appropriate time for Congress to consider comprehensive internet privacy legislation. To address such privacy concerns, states and other countries have adopted privacy rules. For example, the European Union’s General Data Protection Regulation, which came into force in May 2018, is a set of privacy rules that give consumers control over the collection, use, and sharing of their personal information, and California passed its own privacy law in June 2018 that becomes effective in 2020. In February of this year, we reported that FTC does not have civil penalty authority for initial violations of GLBA’s privacy and safeguarding requirements, which, unlike FCRA, includes a provision directing federal regulators and FTC to establish standards for financial institutions to protect against any anticipated threats or hazards to the security of customer records. To obtain monetary redress for these violations, FTC must identify affected consumers and any monetary harm they may have experienced. However, harm resulting from privacy and security violations (such as a data breach) can be difficult to measure and can occur years in the future, making it difficult to trace a particular harm to a specific breach. As a result, FTC lacks a practical enforcement tool for imposing civil money penalties that could help to deter companies from violating data security provisions of GLBA and its implementing regulations. We recommended that Congress consider giving FTC civil penalty authority to enforce GLBA’s safeguarding provisions. Additionally, in our January 2019 report, we found that FTC had not yet issued regulations for internet privacy other than those protecting financial privacy and the internet privacy of children, which were required by law. FTC uses its statutory authority under the FTC Act to protect consumers from unfair and deceptive trade practices. For FTC Act violations, FTC may promulgate regulations but is required to use procedures that differ from traditional notice-and-comment processes and that FTC staff said add time and complexity. In addition, under this authority, FTC can generally only levy civil money penalties after a company has violated an FTC final consent order. In our recommendation that Congress consider developing comprehensive internet privacy legislation, we also suggested that such legislation consider providing rulemaking and civil money penalty authorities to the proper agency or agencies. In summary, new technologies have vastly changed the amount of personal information private companies collect and how they use it. But our current privacy framework does not fully address these changes. Laws protecting privacy interests are tailored to specific sectors and uses. And, consumers have little control over how their information is collected, used, and shared with third parties for marketing purposes. As a result, current privacy law is not always aligned with the Fair Information Practice Principles, which the Department of Commerce and others have said should serve as the foundation for commercial data privacy. Thus, the privacy framework warrants reconsideration by Congress in relation to consumer interests, new technologies, and other issues. Chairman Crapo, Ranking Member Brown, and Members of the Committee, this concludes my statement. I would be pleased to respond to any questions you may have. For further information on this statement, please contact Alicia Puente Cackley at 202-512-8678 or cackleya@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contact above, Jason Bromberg (Assistant Director), William R. Chatlos, Rachel DeMarcus, Kay Kuhlman (Assistant Director), Christine McGinty (Analyst in Charge), Barbara Roesmann, and Tyler Spunaugle contributed to this statement. Other staff who made key contributions to the reports cited in the testimony are identified in the source products. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Information resellers—companies that collect and resell information on individuals—have dramatically increased the collection and sharing of personal data in recent years, raising privacy concerns. Increasing use of social media, mobile applications, and other technologies have intensified these concerns. This statement is primarily based on findings from GAO's 2013 report on information resellers ( GAO-13-663 ). It also discusses a 2015 report on facial recognition technology ( GAO-15-621 ), a 2018 report on financial technology ( GAO-18-254 ), and two 2019 reports on internet privacy and consumer data protection ( GAO-19-52 and GAO-19-196, respectively). GAO discusses (1) existing federal laws related to the privacy of consumer information held by information resellers and (2) any gaps in this legal framework. For the prior work, GAO analyzed relevant laws, regulations, and enforcement actions and interviewed representatives of federal agencies, trade associations, consumer and privacy groups, and resellers. In recent years, GAO issued reports that relate to information resellers and consumer privacy issues. Two central findings from a 2013 GAO report remain current: No overarching federal privacy law governs the collection and sale of personal information among private-sector companies , including information resellers (data brokers). Instead, a variety of laws are tailored to specific purposes, situations, or entities. For example, the Fair Credit Reporting Act limits use and distribution of personal information collected or used to help determine eligibility for such things as credit or employment. Other laws apply to health care providers, financial institutions, or to online collection of information about children. Gaps exist in the federal privacy framework . With regard to data that private-sector entities use for marketing, no federal statute provides consumers the right to learn what information is held about them and who holds it. In many cases, consumers also do not have the legal right to control the collection or sharing with third parties of sensitive personal information (such as their shopping habits and health interests) for marketing purposes. In 2013 and in 2015, GAO also reported that the statutory framework for consumer privacy did not fully address new technologies—such as online tracking and facial recognition—and the vastly increased marketplace for personal information, including the proliferation of information sharing among third parties. In two 2019 reports, GAO found additional gaps in the federal privacy framework and potential limitations in regulatory authority under current privacy law. Internet content providers and internet service providers collect, use, and share information from customers to enable their services, support advertising, and for other purposes. Although the Federal Trade Commission (FTC) generally has addressed internet privacy through its unfair and deceptive practices authority, and other agencies have used industry-specific statutes, there is no comprehensive federal privacy statute with specific internet privacy standards for the private sector. GAO also reported that the Gramm-Leach-Bliley Act, a key law governing the security of consumer information, does not provide FTC with civil penalty authority for violations of the privacy and data security provisions of the act. New and more advanced technologies and changes in the marketplace for consumer information have vastly increased the amount and nature of personal information collected and the number of parties using or sharing it. Such changes warrant reconsideration of how well the current privacy framework protects personal information. In 2013, GAO recommended that Congress consider strengthening the consumer privacy framework to reflect the effects of changing technologies and markets. In 2019, GAO recommended that Congress consider comprehensive internet privacy legislation. Legislation on these issues has not been enacted to date.", "document_type": "gao"}
{"report": "VA, HUD, and DOL are the federal agencies that provide programs specifically aimed at assisting homeless veterans. They are among the 19 federal member agencies of USICH—an independent establishment within the Executive Branch charged with coordinating the federal response to homelessness and creating a national partnership at every level of government and with the private sector to reduce and end homelessness nationally. USICH, VA, and HUD have jointly established criteria and benchmarks to guide communities that are taking action toward being certified as having ended veteran homelessness. USICH stated that an end to homelessness does not mean that no one will ever experience a housing crisis again, but that every community will have a systematic response in place to prevent homelessness whenever possible and ensure that homelessness is a rare, brief, and non-recurring experience. VA, HUD, and USICH coordinate their efforts towards this goal through a working group, Solving Veterans Homelessness as One, which was established in 2012. We identified 16 federal programs that target services specifically to veterans who are homeless or are at risk of becoming homeless. As shown in table 1, the programs provide permanent and transitional housing, health care, employment assistance, and supportive services, such as assistance with rent, utilities, and moving costs. Twelve of the programs are administered solely by VA, two are jointly administered by HUD and VA, and two are administered by DOL. VA’s and HUD’s homelessness programs follow the principles of “Housing First,” which is intended to provide housing without any preconditions and barriers to entry such as sobriety, treatment, or service participation requirements. The largest of these programs were HUD-VASH, GPD, and SSVF. In fiscal year 2019, HUD-VASH reported over $1 billion in obligations; GPD reported obligations of over $234 million; and SSVF reported $380 million in obligations. Homeless veterans can access program services in several ways, including through: VA Medical Centers (VAMCs). Program services can be provided to homeless and at-risk veterans at their local VAMCs. Service providers. Veterans may also obtain services through local, state, or nonprofit organizations in the community, some of which receive grants from federal agencies to fund program services. Public housing agencies (PHAs). Housing vouchers are administered to homeless veterans by PHAs, which are HUD-funded city, county, and state agencies. VAMCs, service providers, and PHAs may coordinate through Continuums of Care (CoC), which are composed of stakeholders in a geographical area that, among other things, coordinate to provide homeless services, apply for grants, set local priorities, and collect homelessness data for all homeless populations. Each year, HUD awards CoC program funding competitively to nonprofit organizations, states, and other local recipients. The CoC is responsible for its operation and developing and implementing its plan and strategies to prevent and end homelessness. Additionally, the CoC must choose an entity to operate the local information system used to collect client-level data and data on the provision of housing and services to homeless individuals and families and those at risk of homelessness (referred to as the Homeless Management Information System). The CoC also designates an entity that prepares and submits the CoC program application for HUD funding (referred to as the collaborative applicant). HUD requires each CoC to establish and operate a centralized or coordinated assessment system (referred to as Coordinated Entry). This system may include implementing a “no wrong door” approach in which a homeless family or individual can show up at any homeless housing and service provider in a geographic area and get screened for services using the same assessment tool (see figure 1). The goal of Coordinated Entry is to ensure that people experiencing a housing crisis within a CoC are quickly and consistently assessed and referred for services. HUD officials stated that Coordinated Entry is a process that was first developed by some CoCs based on best practices. In 2017, HUD adopted and codified requirements for all CoCs to participate in Coordinated Entry. That same year, VA published requirements for VAMCs to participate in Coordinated Entry. According to VAMC staff and service providers we interviewed, they faced challenges serving homeless veterans and those at risk of becoming homeless due, in part, to the additional level of service and support that some veterans need. For example: Substance use and mental illness. Substance use disorders and mental health issues such as post-traumatic stress disorder (PTSD) and depression, are among the most complex issues many homeless veterans face, according to USICH. In 2018, USICH reported that 28 percent of homeless veterans that receive VA-provided health care have been diagnosed with depression. Thirteen percent have been diagnosed with PTSD. Further, 19 percent struggle with alcohol abuse and 20 percent with drug abuse. VAMC staff and service providers told us that addressing the complex nature of these conditions is often a challenge for them. For example, one SSVF provider told us that it is challenging to find housing for veterans with mental health or substance use disorders; further, HVCES staff at one VAMC told us that employment programs for the general population may not be suitable for clients with these disorders. HUD-VASH staff at one VAMC told us that there are not enough mental health providers in the VA system. Overall, staff from three VAMCs from the GPD and HVCES programs, five GPD service providers, and three SSVF service providers cited challenges related to substance use and mental illness. Aging homeless veterans. In 2018, USICH reported that the number of homeless veterans who were 62 and older increased by 54.3 percent between 2009 and 2016. VA officials told us that this trend is expected to continue and that this population has increased, in part, because the services that VA offers are not targeted to aging veterans. According to VA officials, there is a similar aging trend in the general veteran population. HUD-VASH staff at three VAMCs, HVCES staff at one VAMC, and GPD staff at two VAMCs told us that aging veterans require a higher level of care than what existing programs may be able to fully address. Some of these veterans may suffer from ambulatory and cognitive issues and have difficulties living alone but cannot afford an assisted living arrangement. HUD-VASH staff at one VAMC we visited told us the VAMC was able to hire five occupational therapists to assist aging clients with their specialized needs. Further, VA officials told us that HUD-VASH is collaborating with VA’s Geriatrics and Extended Care programs office to explore how aging homeless veterans can be served through other programs, such as the Medical Foster Home and Community Residential Care programs. This collaboration would allow VA to provide funding for services while the HUD-VASH voucher would pay for housing costs. VA officials also told us that they are working to market HUD-VASH to developers and funders to increase the development of project-based HUD-VASH housing. This would give the program a dedicated housing stock and better serve subpopulations of veterans, such as veterans who are elderly or suffer from mental illness. Resistance to program participation. According to HCHV staff at two VAMCs, HVCES staff at three VAMCs, two PHAs that administer the HUD-VASH voucher program, and five service providers (two GPD, two HVRP, and one SSVF), a key challenge in addressing the needs of homeless veterans is their resistance to participating in a program, particularly if it places restrictions or requirements on them. This issue makes it challenging for outreach and treatment teams to deliver services. In addition to the challenges cited above, veterans must meet certain eligibility requirements to participate in homeless assistance programs which if not met can present challenges to VAMC staff and service providers when providing veterans with services. For example, veterans must meet certain criminal history requirements to be eligible for HUD- VASH. HVCES staff from one VAMC, DCHV staff from two VAMCs, HUD-VASH staff from another VAMC, and three service providers (one GPD and two SSVF) also told us that it is challenging to find housing and employment for homeless veterans with legal or criminal problems and landlords may be resistant to working with them. One PHA that works with HUD-VASH and one SSVF service provider told us that a veteran’s ineligibility for VA health care services also presents a challenge to them. This is because a number of VA homeless programs require a veteran to be eligible for VA health care benefits as a condition to enrollment. Generally, veterans are eligible to receive VA health care benefits if they served in the active military, naval, or air service and were discharged under conditions other than dishonorable. Therefore, according to VA officials, veterans with dishonorable discharges cannot access VA homeless assistance programs and veterans with other-than- honorable discharges may have limited access to them. In addition to meeting the discharge status requirement, a person must also meet the definition of “veteran” to be eligible for VA health services. However, the definition of “veteran” depends on factors including length of service and if the individual served on active duty or was part of the National Guard or the Reserves. Therefore, even if an individual has the appropriate discharge status to be eligible, they may not meet other eligibility requirements for VA health benefits. One service provider told us that they have to work on alternative solutions to help veterans that do not meet eligibility requirements. In 2017, VA in partnership with HUD implemented a flexibility within the HUD-VASH program—the HUD-VASH Continuum—which according to VA officials, will permit PHAs to make up to 15 percent of their total HUD- VASH allocation available to veterans who are ineligible for VA health care services, with some exceptions. According to VA officials, this expands the availability of permanent supportive housing to service members who are not eligible for VA health care. In addition, the House and Senate are considering bills to expand HUD-VASH eligibility. DOL has also implemented statutory changes to HVRP eligibility requirements to provide veterans with better access to job training programs. The program’s eligibility requirements have been broadened to include veterans participating in the HUD-VASH, Tribal HUD-VASH, and SSVF programs, and other veterans that were not previously eligible. VAMC staff and services providers cited broader challenges—not specific to veterans or the assistance programs themselves—as impacting their ability to provide assistance to homeless veterans. Those challenges include VA staffing issues and external factors, such as the lack of affordable housing and limited transportation options. VA staffing shortages. VA officials, HVCES staff at three VAMCs, and HUD-VASH and DCHV staff at two VAMCs told us they faced difficulties with recruitment and retention, which have led to persistent understaffing. For example, staff at four VAMCs for the HUD-VASH, DCHV, and HVCES programs told us that the hiring and onboarding process can often take a long time, and by the time an offer is finalized, qualified applicants have moved on to other jobs. DCHV staff at two VAMCs and HUD-VASH staff at three VAMCs cited understaffed human resources offices and a taxing approval process as contributing factors. HUD-VASH staff from one VAMC told us that it is difficult to fill some positions because the outreach work requires extensive travel within large geographic areas. Further, they told us that in high-cost areas, the VA’s local pay scale is not high enough to attract new recruits for case manager positions. HUD-VASH staff at one VAMC indicated that they have not been fully staffed for several years. Limited staffing may limit the number of veterans who can be served, according to VA officials. For example, DCHV program staff at one VAMC told us that they had to close 83 beds because there is not enough staff to keep them operational. One PHA working with the HUD-VASH program told us that VA’s staffing challenges create a bottleneck of services to clients while staff at one VAMC working with SSVF told us that high turnover of program staff is disruptive for clients. Overall, staffing shortages were cited as a challenge by VAMC staff for several programs: HUD-VASH, DCHV, HCHV, GPD, and HVCES. VAMC staff we interviewed have taken some steps to limit the impact of staffing issues. For example, at one VAMC, staff from the HCHV and GPD programs have conducted cross-training so they can back each other up when staffing shortages occur. Two other VAMCs have brought in staff from other locations to help with the workload or have developed an action plan to address employee burnout. Our past work has highlighted VA’s staffing challenges, including recruiting and retaining clinical staff. For example, we previously reported that difficulties in recruiting and retaining skilled health care providers and human resource staff at VAMCs make it difficult to meet the health care needs of more than 9 million veterans. We have also previously reported that, in addition to high attrition and increased workload, human capital shortfalls can lead to burnout among the staff whose job it is to implement these programs. Housing cost and availability. Limited and high cost housing exacerbate the other challenges VAMC staff and service providers identified. For example, HUD-VASH staff at one VAMC told us that even with subsidies, it is difficult for veterans to obtain housing because HUD-VASH vouchers may not be sufficient to cover rent. HUD-VASH staff from one VAMC and one PHA we interviewed told us that because housing costs are rising, and housing in metro areas remains limited, expensive, and competitive, veterans have fewer housing options available to them. Limited housing was cited as a challenge by VAMC staff and service providers for HUD-VASH and SSVF programs in all types of locations—urban, suburban, and rural areas. Finding and recruiting landlords is a significant challenge in getting veterans housed, according to HUD officials, HCHV staff at one VAMC, HUD-VASH staff from two VAMCs, two PHAs, and five SSVF service providers. According to HUD-VASH staff from one VAMC and SSVF staff from another VAMC, the demand for housing exceeds supply and landlords have few incentives to accept homeless veterans. HUD-VASH staff from one VAMC, one PHA, and one SSVF service provider also told us that some landlords perceive veterans to be risky because some have criminal records or substance use disorders, and may be reticent to work with them out of fear of incurring damages to their property. Some service providers have taken steps to create incentives for landlords to participate in VA’s programs. For example, HUD-VASH staff at one VAMC told us that local providers may partner to cover moving fees for veterans and encourage landlords to accept veterans’ housing vouchers. One PHA has put together landlord forums and is working to build relationships with landlords in their communities. Another PHA has held housing tours and fairs to bring landlords and clients together. VA has also implemented program changes to help address the lack of affordable housing. For example, the new Shallow Subsidies initiative that became effective in September 2019, allows SSVF service providers to provide very low-income veteran families a rental subsidy for a two-year period without requiring recertification. The two-year period ensures no reduction in subsidy even if a recipient’s income situation improves within that time frame and the family is no longer considered very low income. This provides a strong incentive for employment gains because the assistance is not dependent on income during this two-year period. Launched in 2018, VA’s Rapid Resolution initiative is another solution that is designed to prevent or resolve homelessness by providing immediate assistance when a veteran enters an emergency shelter system—such as by offering landlord mediation and conflict resolution, or connecting the veterans to support networks in other places. According to VA officials, Rapid Resolution is being implemented through VA’s SSVF program, in coordination with HUD and USICH. Limited resources (other than staffing). HUD-VASH staff at one VAMC told us they are short on equipment like laptops, office supplies, and office space. Additionally, HUD-VASH staff at another VAMC told us they do not have access to government cars for work- related travel. One HVRP service provider told us that case managers do not have enough vehicles to travel long distances or to remote locations to meet clients. To make up for shortages in resources, one SSVF service provider told us that it develops partnerships with local programs to meet the needs of the client. HUD-VASH staff at one VAMC indicated that they have communicated issues of insufficient resources to their leadership, but the issues have not been addressed to date. Resource limitations were cited as a challenge by HUD-VASH staff from two VAMCs, GPD staff from three VAMCs, DCHV staff at two VAMCs, HCHV staff at two VAMCs, and four service providers (two GPD and two HVRP). Transportation limitations. According to VAMC staff and service providers, the lack of transportation for veterans is a significant challenge for some programs. For example, according to DCHV and HVCES staff at one VAMC, HUD-VASH staff at another VAMC, and one GPD and two HVRP service providers, some veterans may not have vehicles or may live in areas with limited public transportation systems. This makes it difficult for the veterans to access resources, go to job interviews, or secure transportation for jobs. Some service providers told us they make alternative arrangements for their clients to help address these issues. For example, one HVRP service provider told us they might drive veterans to interviews or arrange for public transportation. DCHV staff at one VAMC and HVCES staff at another VAMC told us that they work with community partners to provide alternatives like shuttle services and bus passes. Additional challenges related to specific programs we reviewed are discussed in appendix II. We reviewed the services provided, eligibility requirements, and population served by the 16 programs that exclusively target homeless veterans to identify duplication and overlap. We determined that there is no duplication among the programs, but identified overlap across some program services. Specifically, we identified 18 main services that are commonly offered across the 16 programs and found that at least six of those services overlap across two or more programs (see figure 2). However, we also found these programs differed in meaningful ways, for example in terms of the different types of homeless veterans served or specialized services or focus. As we have previously reported, fragmentation, overlap, and duplication exists across the government, which can present benefits and challenges. Duplication occurs when two or more programs provide the same services to the same beneficiaries. Overlap occurs when two or more programs offer similar services to similar beneficiaries. As shown in figure 2, 15 of the 16 programs overlap in two or more of the following services that they offer. Case management is a process for managing a client’s care that includes assessing the needs of the veteran and evaluating health care options and services to ensure those needs are met while maintaining a primary focus on resolving the veteran’s homelessness through permanent housing. Eleven programs provide case management services: HUD-VASH, Tribal HUD-VASH, HCHV, HCRV, H-PACT, SSVF, GPD, DCHV, CWT-TR, HVRP and VJO. Supportive services might include providing meals, counseling, child care, housing assistance, transportation, and other services essential for achieving and maintaining independent living. Six programs provide supportive services: HUD-VASH, Tribal HUD-VASH, GPD, SSVF, HVRP, and Stand Down. Outreach involves directly contacting veterans in need of homeless services and connecting them with housing, health care, and supportive services. Six programs conduct outreach: HCHV, HVRP, SSVF, CRRCs, VJO, and HCRV. Referrals are the most common way for homeless veterans to find out about program services available to them. Referral services include conducting an assessment of the clients’ needs, connecting them to the appropriate programs, and following up with the clients as well as documenting all referral activities. Six programs provide referral services: HVRP, SSVF, CRRCs, NCCHV, HCRV, and Stand Down. Employment services include help with creating job opportunities for veterans, job searches, interviewing, and other employment assistance. Three programs provide employment-related services: HVCES, CWT-TR, and HVRP. Rental subsidies are offered to veterans through vouchers and grants, which help subsidize rental costs. Three programs offer rental subsidies: HUD-VASH, Tribal HUD-VASH, and SSVF. Although we identified overlap in these services across 15 of the 16 programs, the programs differ in meaningful ways. Specifically, some of these programs serve specific subpopulations of veterans and some provide a specialized service that other programs do not offer. For example, of the 11 programs that offer case management services, one program provides medical care (H-PACT), while others provide services in different areas such as transitional housing (GPD), housing subsidies (HUD-VASH), supportive services (SSVF), preparing veterans for employment (HVRP) and outreach (HCHV). Other programs that offer case management services serve unique subpopulations of homeless veterans such as those with mental health or substance use issues (CWT-TR and DCHV), American Indians and Alaskan Natives living in or near reservations or other Indian areas (Tribal HUD-VASH), or justice- involved veterans in local jails (VJO) and state and federal prisons (HCRV). According to a VA directive, more than one case manager may be involved in care planning and service delivery for veterans with complex needs. In addition, staff at the six VAMC locations we visited told us that clients may be co-enrolled in more than one program and can receive case management services from each of those programs. Figure 3 illustrates how case management may overlap across programs, but each program provides distinct services to the veteran. Similarly, of the three programs that provide employment services, HVCES focuses on establishing partnerships with employers to develop job opportunities for veterans and connect them with community services, while HVRP helps the veteran prepare to pursue and gain meaningful employment. The CWT-TR program, on the other hand, focuses on veterans with more complex issues such as substance use, mental health issues, and challenges in obtaining or sustaining employment that may accompany these conditions. Similar meaningful distinctions in subpopulations of beneficiaries and services exist across the programs that provide other types of services to homeless veterans that overlap— supportive services, outreach, referrals, and rental subsidies. Additional information on program differences, including information on program beneficiaries and services, can be found in appendix II. As we previously reported, in some cases it may be appropriate or beneficial for multiple agencies or entities to be involved in the same programmatic or policy area due to the complex nature or magnitude of the effort. Overlapping programs may also facilitate access to services because persons experiencing homelessness are not steered toward one specific point of entry and, in contrast, can access services through several entry points. However, when multiple programs overlap, there is also a risk of program administrators making inefficient use of available resources if they do not coordinate their efforts. For example, according to VA officials, overlap may result in operational costs if the overlapping services are not coordinated well. Table 2 describes some of the potential benefits and challenges of overlap in services for homeless veterans, as identified by agency officials, VAMC staff, and others we interviewed. Effective collaboration among agencies and service providers can help address some of these potential challenges and may help avoid the potential inefficiency that overlapping services may create. VAMC staff and service providers told us that they have taken steps to limit duplication where appropriate. Additionally, they told us that they collaborate and communicate with each other to avoid or mitigate overlap. VA has also issued guidance directed at enhancing coordination between its homeless programs and eliminating or reducing duplication of services, including the following: Veterans Health Administration (VHA) Directive 1110.04, Integrated Case Management Standards of Practice. This guidance states that case management services should be coordinated, collaborative, and veteran-centered throughout the VHA. It also directs case management teams to develop procedures and processes to support cost effective, high quality case management across the VAMC to eliminate duplication of services where appropriate. VHA Handbook 1162.09, Health Care for Homeless Veterans Program. Under the HCHV program, program coordinators are responsible for ensuring coordination of HCHV services with other homeless programs at the VAMC such as GPD, HUD-VASH, DCHV, VJO, HCRV, SSVF, and CRRCs. GPD’s Case Management Services Grant Program, Final Rule. This final rule stipulates that the case management grant may not be used for veterans receiving case management from certain other programs to ensure that there is no duplication of case management services. VHA Handbook 1162.01 (1), GPD. This guidance states that GPD liaisons are to ensure the coordination of care for homeless veterans in GPD-funded programs by following a plan that clearly delineates the roles of those responsible for the service provision to reduce duplication of services. VHA Handbook 1101.10 (1), Patient Aligned Care Team Handbook. This guidance directs staff to coordinate care in a manner that avoids unnecessary duplication. The following section of this report discusses how federal agencies collaborate more broadly on implementing federal homelessness assistance programs for veterans. We identified two key collaborative mechanisms that federal agencies use to help address veteran homelessness: (1) the Solving Veterans Homelessness as One (SVHO) working group, which coordinates VA, HUD, and USICH’s efforts at the national level, and (2) VA’s integration into Coordinated Entry, which seeks to ensure that homelessness services are coordinated at the local level. As shown in table 3 and as discussed in more detail below, both mechanisms follow leading practices for effective interagency collaboration we have identified in prior work, with some exceptions. According to USICH officials, in 2012, USICH convened the SVHO workgroup to coordinate with HUD and VA on key priorities and maximize efforts to end veteran homelessness. SVHO serves as an interagency decision-making body that plans and executes strategic actions through goal setting, policy gap identification, communication, and action. The SVHO working group fully followed all seven leading practices for effective interagency collaboration that we identified in prior work. A discussion of our assessment follows: Defining Outcomes and Monitoring Accountability. Ending veteran homelessness is one of the national goals listed in USICH’s Federal Strategic Plan to Prevent and End Homelessness. SVHO’s work is organized to support this goal. USICH reports SVHO’s efforts in its annual Performance and Accountability Reports. For example, USICH reported that in fiscal year 2019, SVHO’s efforts led to supplemental guidance and coaching to help sustain the efforts of communities that had been certified as having ended veteran homelessness. Bridging Organizational Cultures. To operate across agency boundaries, SVHO members hold regular meetings. During these meetings, SVHO members have updated one another on each agency’s efforts, discussed strategic objectives, shared program data, and coordinated on technical assistance for service providers. SVHO also held a strategic planning retreat to discuss SVHO’s priorities. Clarifying Leadership. SVHO has a Strategic Decision and Coordination Team that serves as the decision-making body and includes leadership from VA, HUD, and USICH. The team’s decisions are made by consensus, and the role for facilitating the team rotates every four months among the three agencies. The Strategic Decision and Coordination Team’s responsibilities, which include providing strategic guidance on cross-agency issues, providing joint oversight and decision-making, and facilitating the approval of decisions from the individual agencies are outlined in SVHO’s charter. Clarifying Roles and Responsibilities. The SVHO charter outlines the roles and responsibilities of the Strategic Decision and Coordination Team and the Support Team, whose responsibilities include responding to priority projects and elevating issues requiring decision and coordination to the Strategic Decision and Coordination Team. Including Relevant Participants. SVHO members (USICH, VA, and HUD) are the relevant participants because they are the agencies centrally involved in implementing veteran homelessness programs. Identifying Resources. USICH, VA, and HUD contribute staff resources to the working group. Representatives from each of the agencies attend regular SVHO meetings to ensure continuity, provide the necessary subject matter expertise, and make decisions. SVHO has also developed resources to facilitate the group’s meetings, such as agendas to guide discussions. Updating and Monitoring Written Guidance and Agreements. In March 2020, SVHO revised its charter to remove outdated information and to reflect the group’s current structure and operations. The revised charter describes the purpose of establishing SVHO as a formal structure for coordination and decision-making (to enable member agencies to execute joint activities necessary for the goal of preventing and ending veteran homelessness), SVHO’s structure (the group is comprised of a leadership team and support team with various responsibilities), and operating procedures (which involve holding regular meetings). USICH officials told us it was important to have an updated charter that solidified the commitments of the member agencies to the group. VA officials added that updating the charter would help serve as a reminder of the group’s purpose. Coordinated Entry is a process designed to help communities prioritize people who are most in need of assistance by standardizing the assessment process, defining community-wide prioritization policies, and coordinating referrals, among other things. HUD established minimal requirements for Coordinated Entry in a 2012 Continuum of Care Program Interim Rule. HUD officials said they established additional requirements in 2017 in coordination with other federal agencies, including VA. VA also issued a memo in 2017 stating that VAMCs must be actively engaged in their local Coordinated Entry. Efforts to integrate VAMCs into Coordinated Entry fully followed five of the seven leading practices on effective interagency collaboration and partially followed the other two (Bridging Organizational Cultures and Updating and Monitoring Written Guidance and Agreements). A discussion of our assessment follows: Defining Outcomes and Monitoring Accountability. VA established requirements for the VAMCs as they integrate into Coordinated Entry, which include active engagement with the CoC, involvement with case conferencing, and aligning standardized assessments. VA has a checklist that VAMCs use to assess their compliance with Coordinated Entry requirements. According to VA officials, they monitor VA integration into Coordinated Entry through self- assessment checklists that VAMCs are required to submit monthly through an internal VA system. VAMCs are also required to submit monthly operation plans to track their progress. Bridging Organizational Cultures. As we previously reported, collaborating agencies should establish ways to operate across agency boundaries and address their different cultures. VA requires VAMCs to actively engage with all coordinated entry systems within their catchment area. VA has provided some guidance to help VAMCs operate across organizational boundaries as they integrate into Coordinated Entry, but this guidance is broad in some areas. For example, it instructs VAMCs to collaborate with local CoC leadership to establish a clear process for making and receiving referrals and to share aggregate program data with each of their communities as needed. But the guidance does not describe steps that VAMCs can take to do so. In addition, two service providers and staff from two VAMCs told us that it can be challenging to work with multiple CoCs because each has their own processes. Additionally, staff from three VAMCs and one CoC entity told us that staff turnover creates challenges in their coordinated entry systems, including impeding relationship-building among partners. VA’s guidance acknowledges that VAMCs may be working with multiple CoCs, but the guidance does not provide any best practices to help address this issue, nor does it expressly address relationship-building in light of staff turnover. Clarifying Leadership. As previously discussed, VA oversees the integration of the VAMCs into Coordinated Entry. Additionally, USICH and HUD officials told us there was an interagency working group on Coordinated Entry, where several agencies, including USICH, VA, and HUD, convened to discuss, among other things, what was happening in the field and barriers to Coordinated Entry implementation across all homeless programs, including those for veterans. HUD officials told us they also worked closely with VA to fully integrate VAMCs into Coordinated Entry. Clarifying Roles and Responsibilities. VA issued guidance that defined VAMCs roles in Coordinated Entry. For example, one or more representatives must be involved in the community planning process and in case conferencing, with sufficient knowledge and decision-making power to actively engage in each activity. Including Relevant Participants. All homeless assistance organizations should be involved in Coordinated Entry, according to HUD guidance. Coordinated Entry includes CoCs, VAMCs, service providers, and public housing agencies, among others. Staff from one VAMC, one service provider, and one CoC entity that we spoke with described their coordinated entry systems as being inclusive of all relevant stakeholders, including veteran homeless service providers. Identifying Resources. VA funded 86 Coordinated Entry Specialist positions through the HCHV program, of which 81 had been filled, as of January 2020, according to VA officials. Staff from two VAMCs and two CoCs told us that these new positions play an important role in VAMCs’ integration into Coordinated Entry because they serve as a liaison between the CoCs and the VAMCs. Additionally, VA requires that VAMCs dedicate a portion of VA resources (such as HUD-VASH vouchers or VA Homeless Program Residential Treatment beds) for their inclusion into the greater pool of homeless service resources that are accessed by veterans through Coordinated Entry. Updating and Monitoring Written Guidance and Agreements. We previously reported that agencies can strengthen their commitment to working collaboratively by formally documenting their agreements, and that those written agreements are most effective when regularly monitored and updated. As discussed earlier, VA has issued some guidance to help VAMCs integrate into Coordinated Entry. VA has also held webinar trainings and issued some program-specific documents, such as an SSVF Coordinated Entry fact sheet and a “frequently asked questions” document for HUD-VASH. VA has also provided technical assistance by request, according to agency officials. However, as noted earlier, VA’s guidance is broad in some areas and neither provides best practices to help VAMCs working with multiple CoCs, nor expressly addresses relationship-building in light of staff turnover. VA officials told us they do not have plans to issue additional guidance on Coordinated Entry because they believe their current guidance provides sufficient direction. However, several interviewees (staff from three VAMCs, one service provider, and one PHA) told us they need additional guidance on Coordinated Entry, specifically about how to better collaborate among partners. For example, staff from one of the VAMC’s said that while they understood that implementing Coordinated Entry required some flexibility, it would be beneficial if VA provided common parameters that communities could follow. Further, some VA guidance (such as the “frequently asked questions” document for HUD-VASH) may not be accessible by all service providers for VA’s homeless programs because it is stored on the agency’s internal system (the Homeless Programs Hub) or provided via technical assistance only upon request. Staff from two VAMCs stated that VA could better disseminate guidance. Additionally, one service provider and one PHA told us it would be helpful for VA to share best practices on collaboration used in other parts of the country. By providing additional information on how VAMC staff and service providers can collaborate with local partners, such as best practices, and making available guidance readily accessible, VA can help ensure that VAMCs and service providers are able to more effectively collaborate with other local providers to serve homeless veterans. According to VA officials, since 2011, VA has focused on three primary outcome measures for the homelessness assistance programs we selected for review: 1) placement into permanent housing, 2) employment, and 3) negative exits from programs. DOL developed four critical measures for HVRP, including the placement rate for total enrollment, which tracks the total number of program participants employed in one or more jobs. VA and DOL officials told us they review their performance measures annually and adjust them as needed. National level performance data for fiscal years 2015 to 2019 show that five of the seven selected programs we reviewed have generally met their performance targets (see table 4). However, two programs, HUD-VASH and DCHV, have not met some of their targets. Specifically, in four of the last five years, HUD-VASH did not meet its targets for “percent housed in HUD-VASH housing” and “percent housed within 90 days.” In the last two years (2018 and 2019), HUD-VASH did not meet its targets for “negative exits”; however, VA had decreased the target for those years (making it more difficult to meet). DCHV did not meet its targets for “exits to permanent housing” for the last three fiscal years, and “negative exits” for two of the last five fiscal years. According to VA officials, factors that have affected VAMCs abilities to meet HUD-VASH performance targets—some of which are challenges identified by local VAMC staff and providers that we have discussed previously—include an insufficient number of case management staff, which has led to fewer veteran admissions into HUD-VASH and a lack of safe and affordable housing for veterans. VA officials told us that DCHV program outcomes have been affected by factors including discharges to other transitional housing programs (which would not be included under an exit to permanent housing) and limited affordable housing resources. To help improve program outcomes for HUD-VASH, VA officials told us they are focusing on increasing HUD-VASH voucher utilization, such as by using vouchers for non-Veteran Housing Administration eligible homeless veterans through the HUD-VASH Continuum program and expanding project-based HUD-VASH efforts (discussed previously). To improve DCHV program outcomes, VA officials said they are holding in- depth discussions with DCHV staff to highlight lessons learned from those VAMCs that are meeting performance targets. The performance measures used for the selected programs we reviewed reflected most of the attributes of successful performance measures that we identified in prior work (see table 5). VA’s measures fully reflected all six of these attributes. DOL’s measures fully reflected five attributes and partially reflected one, the reliability attribute. Performance measures that include these attributes are effective in monitoring progress and determining how well programs are achieving their goals. A discussion of our assessment of VA’s and DOL’s performance measures follows: Clarity. VA’s and DOL’s policies clearly state the names and descriptions of the performance measures we reviewed. The names and descriptions are also consistent with the methodologies that were used to calculate them. Measureable Target. VA and DOL have established quantifiable, numerical targets for their performance measures, which allows them to compare expected and actual results. VA officials told us they developed the targets for their measures by first obtaining baseline data and then looking at historical and projected performance. HVRP service providers identify their own targets during the annual grant competition process, according to DOL officials. DOL officials told us they provide some parameters, such as the national targets, to help providers develop their individual targets. Objectivity. VA’s and DOL’s performance measurement policies describe what is expected to be measured (for example, the percent housed and percent employed). They also indicate which specific population (veterans) and under what timeframes (the relevant reporting period). Baseline and Trend Data. Nearly all the measures have baseline and trend data for the last five fiscal years. The exceptions are measures that have been recently discontinued. Having baseline and trend data allows VA and DOL to monitor changes in program performance. Linkage. DOL’s performance measures for HVRP align with one of DOL’s agency-wide strategic objectives to provide veterans with resources and tools to gain and maintain employment. DOL officials told us that information about the measures is communicated to grantees through local officials, who review a data dashboard created by DOL officials at headquarters. VA’s performance measures are aligned with VA’s agency-wide goal to end veteran homelessness, as outlined in VA’s most recent strategic plan. VA officials told us they communicate information about the performance measures to the VAMCs and service providers through scorecards. Reliability. Measures reflect this attribute when they produce the same result under similar conditions. Reliability is increased when verification and validation procedures exist, such as checking performance data for significant errors by formal evaluation or audit. VA’s performance measures fully reflected the reliability attribute; DOL’s measures partially reflected it. VA officials told us they ensure data quality through the use of validation processes, error messages, and notifications that appear in real-time as data are entered. Additionally, there are dedicated program offices that work with the VAMC’s and service providers to monitor and reconcile data. Finally, VA’s policies describe steps that should be taken to review and verify the quality of the data. DOL officials told us they review HVRP performance data quality at different levels in the agency (regional and national) and use a data validation tool to identify potential errors. However, DOL officials acknowledge limitations with data quality, namely the lack of an electronic system to compile the data and the potential for human error when entering data into spreadsheets. Further, HVRP service providers may be unclear about the data quality steps to take because DOL’s performance measurement policies provide limited information on data reliability procedures. DOL officials stated that they have conducted webinar training on the data validation tool, but acknowledge that no written policy exits for the data validation process. Without guidance from DOL on the quality control processes that should be applied to performance data, service providers may not understand how to improve data quality and DOL may not have reasonable assurance that these performance data are the most accurate and reliable available. While VA’s measures reflected all the selected attributes of successful performance measures, including communicating linkage, we identified other areas where communication about these measures is not clear. For example, staff from three of the VAMCs we interviewed and two service providers described communication issues related to performance measures for four programs (HUD-VASH, GPD, HVCES, and DCHV). These issues included concerns that VA does not understand the realities on the ground that prevent VAMC staff and service providers from meeting the measures (such as limited housing availability) and VAMC staff being unaware they could use performance scorecards to drill down and learn more about why their performance targets were not met. Additionally, some VAMC staff and service providers we interviewed do not fully understand the measures. For example, DCHV and HCHV staff we interviewed from four VAMCs and three GPD service providers told us they have felt penalized for transitioning veterans from a VA homeless assistance program to another program or to substance abuse or mental health treatment because VA’s performance measures count these transitions as “negative exits.” According to VA officials, however, there are only three instances where participant program exits are counted as negative: 1) when participants are asked to leave for failure to follow rules; 2) when participants leave for failure to comply with program requirements; and 3) when participants leave without telling program staff. VA officials told us they have implemented processes to obtain quarterly feedback from VAMCs and service providers—through operation or actions plans—about the measures, including feedback about not meeting performance targets. However, HUD-VASH staff from one VAMC said that they have reported their concerns about not having information on how to improve performance to VA leadership and GPD staff from another VAMC and two GPD service providers said they have reported their concerns about how negative exits are measured, but the concerns have not been addressed. Additionally, staff from another VAMC were unaware that VA had a way for them to provide formal feedback about the performance measures, suggesting that VA’s feedback process and avenues of communication may lack clarity. We previously reported that improving the communication of performance information among staff and stakeholders can enhance or facilitate the use of performance information by agency managers. Performance information can be used to identify gaps in performance, improve organizational processes, and improve performance. Clearer communication by VA’s Homeless Programs Office about performance measurement—what performance measures capture and how to obtain and provide feedback—would help VAMCs and service providers better understand how their program data are used to measure performance and therefore how to improve performance, which could also help VA better assess program outcomes. VA, HUD, and DOL published some annual reports during the last five fiscal years that monitored the performance of some of the selected homelessness assistance programs for veterans we reviewed. In addition, they conducted a limited number of evaluations to assess their overall effectiveness or impact and conducted other studies that examined other aspects of the programs, such as characteristics of program participants. Program evaluations are systematic studies that use research methods to address specific questions about program performance. We identified two program evaluations conducted by or on behalf of HUD and VA that assessed the impact of HUD-VASH. Published in 2016, the Family Options study examined how the effects of three types of programs—permanent housing subsidies (such as HUD-VASH vouchers), community-based rapid rehousing, and project-based transitional housing—compared with one another and with the usual care available to homeless families. Findings from the Family Options study indicated that giving people experiencing homelessness priority access to deep permanent housing subsidies, such as housing choice vouchers, benefitted program participants by improving housing stability. However, as discussed in the study, heads of households that received permanent housing subsidies experienced a reduction in employment in comparison to participants in other programs. The permanent housing subsidy also cost more than the other programs. The second study was the HUD-VASH Exit study. Published in 2017, the study was part of an effort to improve program effectiveness. It assessed how and why veterans exit the HUD-VASH program, identified obstacles to their obtaining and maintaining housing with a HUD-VASH voucher, described the value of services, and identified barriers to successful collaboration between VA and HUD in administration of the program. Among other things, the study found that the program was successful, as demonstrated by high rates of retention in housing, and that relationships with community partners and the ability to connect veterans to community resources contributed to successful outcomes. While the several other studies and reports we identified did not assess the impact of programs, some did analyze program performance or outcomes (for example, the agencies’ annual performance plans and reports), and others assessed specific aspects of the programs (for example, factors associated with exiting homelessness programs and characteristics of program participants). VA officials noted that resource limitations constrain their ability to conduct impact evaluations. However, they stated that in the future, they plan to evaluate new programs and models, such as the SSVF’s rapid resolution program (discussed previously). DOL officials told us they have commissioned an impact evaluation for HVRP, which is scheduled to be completed in 2022. The study is assessing the effectiveness of the HVRP program on improving homeless veterans’ employment outcomes and will build knowledge about program models including variations. We found that HUD and DOL have developed plans outlining the evaluations they plan to conduct and the steps they used to create their plans, but VA did not. VA’s National Center on Homelessness Among Veterans, which conducts research and assesses the effectiveness of VA’s homelessness programs, has an evaluation agenda listed on its website that describes the Center’s planned studies, but not the steps taken to develop the agenda and prioritize what studies to conduct. HUD and DOL have also developed policies describing the steps the agencies take to ensure evaluation quality and rigor. VA’s National Center on Homelessness Among Veterans, on the other hand, does not have written policies on evaluation quality. VA officials told us they ensure the quality and rigor of the Center’s work by submitting study results for publication through a peer-reviewed standard scientific protocol, consistent with other VA research, but had not yet developed formal written policies as their processes are well known in the Center. However, the Foundations for Evidence-Based Policymaking Act of 2018—enacted in January 2019—will now require VA and other agencies to, among other things, designate an evaluation officer who is to establish and implement an agency evaluation policy and assess the coverage, quality, methods, consistency, effectiveness, independence, and balance of the portfolio of evaluations, policy research, and ongoing evaluation activities of the agency. The Act requires agencies to develop written annual evaluation plans—that discuss steps taken to develop the plan such as how studies were prioritized—to be submitted with their annual performance plan. In June and July 2019, the Office of Management and Budget released its initial guidance on implementing the Act, and additional guidance is forthcoming. The Act also includes provisions for GAO to conduct studies to review agency implementation efforts. VA, HUD, and USICH have taken significant steps to ensure effective collaboration between the agencies and among local service providers when addressing veteran homelessness. However, VA can help local agency staff and service providers better collaborate by fully incorporating leading practices for interagency collaboration. More specific and accessible information on how to collaborate with partners through Coordinated Entry, including on key activities such as making referrals and sharing data, could position local VA staff and service providers to better aid homeless veterans with services at the local level. Opportunities also exist for the agencies to improve some performance measurement procedures. Documenting its data quality processes can help give DOL reasonable assurance that these performance data are the most accurate and reliable available. Additionally, providing clearer communication about performance measurement—what the performance measures capture and how to obtain and provide feedback—can help VA ensure that VAMCs and service providers have a better understanding of how their program data are used in measuring performance (and how to improve performance), which may also help VA better assess program outcomes. We are making a total of three recommendations, two to VA and one to DOL. Specifically: VA’s Under Secretary for Health should provide additional information, such as best practices, about how VA medical centers and service providers participating in Coordinated Entry can collaborate with local partners on key activities (for example, making referrals and sharing data) and ensure that this information and other resources are accessible to VA medical center staff and service providers. (Recommendation 1) The Assistant Secretary for DOL’s Veterans’ Employment and Training Service should document its data quality validation processes for performance data for the Homeless Veterans’ Reintegration Program and disseminate these processes to service providers. (Recommendation 2) VA’s Under Secretary for Health should clearly communicate with local VA staff and service providers about how it measures performance and how to obtain and provide feedback about performance measures. (Recommendation 3) We provided a draft of this report to DOL, HUD, USICH and VA for review and comment. DOL and VA provided written comments, which are reproduced in appendixes III and IV, respectively. HUD and VA provided technical comments, which we incorporated as appropriate. A USICH official stated that USICH did not have concerns with the proposed recommendations and had no additional comments on the draft. In its comments, DOL neither agreed nor disagreed with our recommendation that it document and disseminate its data quality validation processes for performance data for HVRP (Recommendation 2). DOL stated that it agreed with the importance of data quality validation processes and noted that it uses a data validation tool (discussed earlier in our report). In addition, DOL provided new information in its comments on the draft report, stating that the agency released a user manual and training video for field staff and grantees on the validation tool and provided a hyperlink to additional information, including the user manual. While the user manual outlines the steps for downloading the validation tool and how to run validation tests, it does not describe what validation tests are run or the data quality reviews that DOL officials told us occurred at the regional and national level, as discussed earlier in our report. Therefore, we maintain our recommendation that DOL document all of its data quality validation processes for HVRP performance data and disseminate them to service providers to give the agency reasonable assurance that its performance data are the most accurate and reliable available. VA agreed with our recommendations in its written comments (Recommendations 1 and 3) and outlined actions it plans to take to address them, including: Providing additional information, such as successful strategies, about how VAMCs and service providers participating in Coordinated Entry can collaborate with local partners on key activities and enhancing communication through monthly calls on Coordinated Entry collaboration, including case conferencing, streamlined referral processes, and data sharing that will be recorded and accessible any time by staff. Clearly communicating with local VA staff and service providers about how it measures performance and how to obtain and provide feedback about performance measures. VA’s target completion date for these actions is May 2021. In addition, the draft report we originally sent the agencies included recommendations to VA, HUD, and USICH to revise their SVHO working group charter. However, the agencies informed GAO that they had issued a revised charter in late March and VA and HUD provided a copy of the final charter. Based on our review of the charter, we revised our discussion of the charter in the report and removed the recommendations. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Veterans Affairs, the Secretary of the Department of Housing and Urban Development, the Secretary of the Department of Labor, the Executive Director of the U.S. Interagency Council on Homelessness, and other interested parties. In addition, this report will be available at no charge on GAO’s website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report focuses on federal programs that provide services to veterans that are experiencing homelessness or are at risk of being homeless and their dependents. Our report (1) describes the challenges agencies and service providers reported experiencing in implementing selected programs that assist homeless veterans; (2) assesses the extent, if any, of overlap and duplication among programs; (3) evaluates how well federal agencies collaborate to address veteran homelessness; and (4) reviews what is known about the performance of selected programs. We identified a total of 16 programs that specifically target homeless veterans by reviewing agency reports, guidance, and other documentation and past GAO and Congressional Research Service reports. From these 16 programs, we selected 7 that we focused on for our first objective on program challenges and our fourth objective on program performance: Housing and Urban Development-Veterans Affairs Supportive Housing (HUD-VASH); Grant and Per Diem (GPD); Supportive Services for Veteran Families (SSVF); Health Care for Homeless Veterans (HCHV); Domiciliary Care for Homeless Veterans (DCHV); Homeless Veteran Community Employment Services (HVCES); and the Homeless Veterans’ Reintegration Program (HVRP). We selected these programs based on size (largest programs based on funding and the number of veterans served) and services offered (a mix of programs addressing a variety of needs). The results of our review of these programs are not generalizable. For all objectives, we selected and interviewed representatives from the following national advocacy organizations for homeless veterans and other knowledgeable groups to obtain subject matter context: the National Alliance to End Homelessness; the National Coalition for the Homeless; the National Coalition for Homeless Veterans; and American Legion. We judgmentally selected these groups based on their knowledge about homeless veteran policy issues, their ability to share perspectives on a variety of homeless veterans’ subpopulations, and their knowledge about federal homelessness grants. We also interviewed officials from the Department of Veterans Affairs (VA), Department of Housing and Urban Development (HUD), Department of Labor (DOL), and the U.S. Interagency Council on Homelessness (USICH). Additionally, we conducted semi-structured interviews with staff from local VA medical centers (VAMCs) and service providers implementing the selected programs we reviewed; public housing agencies (PHAs) that administer HUD-VASH vouchers; and Continuum of Care (CoC) entities across different locations. Specifically, we interviewed staff from six VAMCs (staff for the HUD-VASH, GPD, SSVF, HCHV, HVCES, and DCHV programs); six CoC entities; six PHAs; and 23 service providers (eight GPD providers, seven SSVF providers, two HVRP providers, two providers that were HVRP, SSVF, and GPD grantees, two providers that were HVRP and GPD grantees, and two providers that were HVRP and SSVF grantees). The results of these interviews are not generalizable. The locations where we conducted these interviews were: Atlanta, Georgia; Kansas City, Missouri; Long Island, New York; Los Angeles, California; Helena, Bozeman, Fort Harrison, and Box Elder, Montana; and Seattle, Washington. We judgmentally selected this sample of sites based on several factors. To select those locations, we started with the 67 communities that were designated as Priority 1 communities by VA in 2015. We then judgmentally selected six of those communities based on the following factors: (1) to reflect a mix of communities with high concentrations of homeless veterans and communities certified as having ended veteran homelessness; (2) to reflect geographic diversity (a mix of urban, suburban, and rural locations); (3) proximity of CoCs and VAMCs (to ensure we could interview both local VAMC staff and service providers); and (4) the presence of our selected programs (to cover as many programs as possible). To identify challenges agencies and service providers reported experiencing in implementing selected programs, we interviewed agency officials, VAMCs, service providers, and PHAs. Specifically, we first asked them a general question about what challenges they face. We then analyzed their responses to develop a list of challenges. A second analyst then verified the steps taken to develop the list of challenges. We also reviewed agency reports, program documentation, and available information on trends on homeless veterans and the general homeless population. To determine the extent of duplication or overlap across programs, we reviewed agency guidance, program descriptions, and other documentation to obtain information on program services and beneficiaries for the 16 veteran homelessness programs we identified, using the process we described above. We then applied GAO guidance on duplication and overlap by comparing the programs using the following definitions: duplication occurs when two or more programs provide the same services to the same beneficiaries; overlap occurs when two or more programs offer similar services to similar beneficiaries. To identify potential benefits and challenges of overlap, we reviewed past GAO reports, and conducted interviews, as outlined above. To assess how federal agencies collaborate to address veteran homelessness, we first identified two collaborative mechanisms—the Solving Veterans Homeless as One (SVHO) working group and VA’s integration into Coordinated Entry—by reviewing agency reports, guidance, and other documentation and interviewing agency officials. We then assessed the collaborative efforts against leading interagency collaboration practices identified in prior GAO work. Specifically, we assessed the extent to which the SVHO working group and VA integration into Coordinated Entry used each leading practice using three categories. “Fully follows” indicates that actions related to a practice reflected most or all of the issues to consider related to the practice; “partially follows” indicates that actions related to a practice reflect some, but not all, the issues to consider related to the practice; and “does not follow” indicates that there have been no actions taken related to the issues to consider for the practice. One analyst reviewed the reports, guidance, and other agency documentation related to the collaborative efforts and made the initial assessment. A second analyst then reviewed this information to make their own determination about the assessment and reach consensus with the first analyst. To determine what is known about the performance of the selected programs we reviewed, we analyzed national performance data for fiscal years 2015 to 2019 from VA and DOL. To assess the reliability of those data, we reviewed the data for obvious errors or inaccuracies by comparing the data to publicly available data from VA’s and DOL’s annual performance reports (to the extent the data were published). We also interviewed VA and DOL officials with knowledge of the systems and methods used to produce these data. We determined that the data we included in the report were sufficiently reliable for purposes of describing program performance for the selected programs we reviewed. To assess if the performance measures the agencies used are effective in monitoring progress, we reviewed VA’s and DOL’s performance measurement guidance. We then compared the measures against selected leading practices we identified in past GAO work. Specifically, our prior work identified ten key attributes for successful performance measures. Measures that include these attributes are effective in monitoring progress and determining how well programs are achieving their goals. We selected six attributes relevant to our analysis. We excluded the remaining four attributes because they are used to assess agency-wide performance and therefore were not applicable to our program-specific analysis. We assessed the performance measures as “fully reflects” if all the performance measures for the selected programs reflected most or all of the definition of the relevant key attribute; “partially reflects” if the measures reflected some, but not all, of the definition of the relevant key attribute; and “does not reflect” if the measures did not reflect the definition of the relevant key attribute. One analyst reviewed the performance measures and guidance and made the initial assessment. A second analyst then reviewed this information to make their own determination about the assessment and reach consensus with the first analyst. To determine the extent to which VA, HUD, and DOL had evaluated selected programs, we conducted a literature search for studies conducted during the last five fiscal years. We also obtained program evaluations from VA, HUD, and DOL. Additionally, we reviewed the agencies’ evaluation policies and interviewed agency officials to obtain additional information about the agencies’ program evaluation efforts. We conducted this performance audit from January 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We identified 16 federal programs that target their services specifically to veterans who are homeless or are at risk of becoming homeless. These programs are funded through the Departments of Veterans Affairs (VA), Housing and Urban Development (HUD) and Labor (DOL). As shown in table 6, the programs provide permanent and transitional housing, health care, rehabilitation, employment assistance, and supportive services, such as assistance with rent, utility, or moving costs. Eligibility requirements vary by program. VA’s Grant and Per Diem (GPD) program awards grants to community- based agencies for transitional housing and case management for homeless veterans. In 2017, VA implemented changes to the program and, as seen in table 7, the program now has six housing models. Each model targets a different population of homeless veterans or focuses on different areas of service. Some VA medical centers (VAMCs), service providers, and public housing agencies (PHAs) we interviewed told us the homelessness programs for veterans we reviewed are working well. Others identified additional challenges that were specific to individual selected programs we reviewed, in particular the GPD program that underwent recent changes. For example, with respect to GPD’s new models, four service providers and staff from three VAMCs told us that the housing models and program guidelines are too restrictive and complex, which hinder the delivery of services. Staff from another VAMC told us that the new housing models are based on best practices but the implementation is challenging. For example, one of these models, Bridge Housing, generally limits the length of stay to 90 days which GPD staff from one VAMC and one provider told us is not enough time to meet the needs of some clients. However, VA officials said that veterans are not asked to leave Bridge Housing after 90 days if the housing plan has not been executed by this time. According to VA officials, GPD grantees can provide transitional housing and services to family members of a veteran, however, the program can only pay per diem for veterans, not their families. In addition, two GPD service providers told us that the bed reimbursement rate is inadequate to cover the cost of providing services to veterans, and GPD staff at one VAMC told us that the existing funding does not cover the full cost of the program. Despite these cited challenges, our review of national performance data shows that VA is generally meeting the performance targets for these six models. Finally, GPD staff at one VAMC told us that there is a shortage of shelters and beds in some areas, and as a result, they cannot accommodate all the homeless veterans that are referred to them. In addition to the contact named above, Allison Abrams (Assistant Director), Erika Navarro (Analyst in Charge), Kimberly Bohnet, Emily Bond, Evelyn Calderon, Lilia Chaidez, Jill Lacey, and Jessica Sandler made key contributions to this report. Also contributing to this report were Ryan Cirillo, Ben Licht, Marc Molino, Sarah Veale, James Whitcomb, and Michael Zose.", "summary": "Despite a large decline over the past decade, an estimated 37,000 veterans in the United States experienced homelessness in 2019. GAO was asked to review federal assistance programs for homeless veterans. Among other objectives, this report (1) discusses challenges agencies and service providers cited in implementing selected programs; (2) evaluates how USICH, VA, and HUD collaborate; and (3) reviews selected programs' performance. GAO analyzed federal guidance and performance data; interviewed VA, DOL, HUD and USICH officials; and met with local VA staff and service providers from selected programs at six sites. Programs were selected based on size (the largest based on funding and veterans served) and the kinds of services they offer; sites were selected for geographic diversity, among other factors. The results of these interviews are not generalizable. The Departments of Veterans Affairs (VA), Housing and Urban Development (HUD), and Labor (DOL) provide programs aimed at assisting homeless veterans. Local VA staff and service providers—who receive grants from federal agencies—provide services to homeless veterans within their communities. In interviews with GAO, they cited challenges in implementing selected programs: Staffing shortages. Shortages in VA case managers may limit the number of veterans they are able to serve. Housing cost and availability. High housing costs and limited stock make it difficult to find affordable housing for homeless veterans. Transportation limitations . Service providers may cover large geographic areas and limited public transportation strains their ability to provide services. Steps that VA and other agencies are taking to address these challenges include contracting out for services to address limited staffing, offering rental subsidies for very low-income veterans, and working with community partners to assist with transportation. Two key federal collaboration mechanisms to address veteran homelessness are a U.S. Interagency Council on Homelessness (USICH) working group to coordinate agencies at the national level and a HUD initiative that coordinates stakeholders at the local level. Both efforts incorporate many leading practices for effective interagency collaboration identified by GAO in prior work. However, local VA staff and service providers stated that they would like additional information—such as on best practices—from VA on how to collaborate more effectively at the local level. While VA has issued some broad guidance, more specific information on effective collaboration on issues such as making referrals and data sharing could better position local VA staff and service providers to aid homeless veterans. VA and DOL have multiple measures in place to assess the performance of the programs GAO selected for review, and most of the measures met their national targets from 2015 to 2019. The measures incorporated most leading practices for performance measurement—such as having measureable targets. However, DOL does not have a written policy on its process for validating its performance data, and as a result may not have reasonable assurance that these are the most accurate and reliable performance data available. Further, some local VA staff and service providers misunderstood how program data were used in assessing performance while others were unaware of VA's feedback processes on performance measures. Additional clarity and communication about VA's performance measures would help local VA staff and service providers better understand how program data are used to measure—and can be used to improve—performance. GAO is making three recommendations: VA should provide additional information on how local providers can collaborate; DOL should document data quality validation processes for its homeless veterans program; and VA should clearly communicate with local VA staff and providers about how it measures performance and how to obtain and provide feedback. VA agreed with the recommendations. DOL neither agreed nor disagreed. GAO maintains that DOL should document its data quality processes, as discussed in the report.", "document_type": "gao"}
{"report": "In the early 2000s, the Navy conceived of a new small surface combatant concept known as LCS. This ship was intended to offer the Navy an affordable, flexible platform that would be able to swap out surface warfare, anti-submarine warfare, or mine countermeasure mission packages to provide for one of those mission needs. As we found in multiple reports, the Navy’s vision for LCS evolved significantly over time in response to diminished capability expectations and significant cost and schedule growth. In 2014, the Secretary of Defense directed the Navy to evaluate alternatives to LCS, citing survivability and lethality concerns. This represented the beginning of the Navy’s pursuit of a solution to address LCS shortcomings and the evolving threat environment acknowledged by the department. The Navy initially envisioned quickly fielding a frigate—referred to as the FF program—based on a minor modified LCS design. The ship was expected to provide a more lethal and survivable multi-mission ship capable of simultaneous surface and anti-submarine warfare, with a planned contract award for the lead ship in 2018. In 2016, we found that the Navy’s planned upgrades for FF did not significantly improve certain survivability areas and lacked capabilities that were prioritized by fleet operators, such as the ship’s range of travel without refueling. Then, in April 2017 we found the Navy’s aggressive FF acquisition schedule increased risk to the government because it included a commitment to buy ships in advance of adequate knowledge. In May 2017, the Navy announced it was revising its frigate plans and began pursuing FFG(X). In 2009, we identified commercial shipbuilding best practices that could be adapted for use by the Navy. We found that successful shipbuilding programs have sound business cases built on attaining critical levels of knowledge at key points in the shipbuilding process before significant investments are made, as shown in figure 1. Regardless of the differences between Navy and commercial shipbuilding, knowledge attainment is crucial to success. Executable business cases use realistic cost and schedule targets to meet performance and quality expectations by balancing inherent uncertainties in acquisition programs. A solid business case provides for the resources necessary to mitigate challenges, such as immature technologies and design requirements. The greater the potential for challenges to occur, the more time and money should be factored into the business case to address them. The Navy has previously agreed, in principle, that knowledge should be attained prior to key milestones to better ensure ships are built to established cost, schedule, quality, and performance standards. In general, the Department of Defense (DOD) acquires new weapon systems, such as Navy surface combatants, through a management process known as the Defense Acquisition System. Under this system, programs typically complete a series of milestone reviews and other key decision points that authorize entry into a new acquisition phase. To execute shipbuilding acquisition programs, the Navy uses the acquisition processes included in the DOD Instruction 5000 series, as well as acquisition instructions established by the Secretary of the Navy. The Navy’s guidance supports a seven-gate review process intended to ensure that requirements align with acquisition plans, and to improve collaboration among stakeholders. Figure 2 provides an overview of the notional framework for Navy shipbuilding acquisition programs described by the DOD and Navy guidance. This acquisition framework includes decision reviews and milestones at key junctures in the acquisition cycle. The Milestone Decision Authority is the individual responsible for determining what events and documentation requirements will apply to an acquisition program, as well as providing approval for a program to proceed to the next acquisition phase. The acquisition framework and Milestone Decision Authority’s purpose is to support careful assessment of a program’s readiness to proceed to the next stage of acquisition activities. The gates and milestones that will be included in an acquisition program’s schedule can be customized based on its circumstances and needs. We have previously found that shipbuilding programs typically have different decision points than other DOD weapon systems. For example, Milestone B for ship programs usually occurs after development of ship specifications and system diagrams is well under way and is typically aligned with the decision to authorize the start of detail design. While Milestone C generally represents the decision to start production for weapon systems, several of the Navy’s more recent shipbuilding programs either do not include a Milestone C review or changed the sequencing of the review to occur after delivery of the lead ship. Programs can receive approval to tailor the requirements for information that must be developed to support this process and to have the decision- making authority delegated to other individuals for acquisition decisions and approvals. The Navy expects that its current plans for FFG(X) will result in a small surface combatant with considerable capability improvements compared to LCS. To achieve this increased capability, the Navy is committing to construct a larger, more expensive ship than LCS. To help refine FFG(X) requirements and identify opportunities for cost savings, the Navy used a conceptual design phase, in which it awarded $75 million in contracts to industry. The Navy’s FFG(X) requirements represent the department’s recognition of its need for a more capable small surface combatant and the limitations of LCS. For LCS and its mission packages, the Navy has devoted nearly $28 billion (constant fiscal year 2018 dollars) to develop and buy a ship that has fallen far short of demonstrating it can meet the minimum level of capability defined at the beginning of the program. Specifically, LCS was designed with reduced survivability requirements as compared to other surface combatants. Over time the Navy lowered several survivability and lethality requirements further and removed some design features— making the ships less survivable in their expected threat environments and less lethal than initially planned. As shown in figure 3, the Navy arrived at its FFG(X) plans after spending several years developing and evaluating a variety of inputs to address problems with LCS and emerging capability needs. The Small Surface Combatant Task Force study report maintained the Navy’s need for 52 small surface combatants, which was revalidated in the Navy’s 2016 Force Structure Assessment. In recognition of LCS’s shortcomings, the Navy significantly reduced the total number of LCS, and began planning for the new frigate based on minor modifications to an LCS design—referred to as FF—to fulfill the 52-ship need. While the FF program was developing its acquisition plans and moving toward a contract award for the lead ship scheduled for 2018, the maritime operating environments continued to rapidly evolve, becoming increasingly complex and contested. In recognition of this, the Chief of Naval Operations directed the Navy to conduct another study, increasing air defense and survivability beyond the FF baseline. In response, the Navy convened a Frigate Requirements Evaluation Team from January to June 2017. The purpose of this team was to build upon FF requirements by analyzing options for air defense and vulnerability upgrades to help determine top-level mission requirements that would yield a more capable frigate. The results of this review led the Navy to cancel its FF acquisition plans and focus on meeting increased requirements through a new FFG(X) Guided Missile Frigate program. Both the FF and FFG(X) requirements reflect the 2015 Small Surface Combatant Task Force report findings that identified a need for increased capabilities for small surface combatants to address evolving threats. As we reported in June 2016, an FF based on a minor modified LCS only partially fulfilled the small surface combatant capabilities that the task force identified as most valued by the fleet. In particular, FF requirements supported a multi-mission ship with some of the fleet’s highest priority mission capabilities, such as surface and anti-submarine warfare, but did not provide air warfare capability. For FFG(X), the Navy maintained the FF requirements and added local air defense as a capability. Table 1 outlines the requirements evolution that the Navy undertook to support a more lethal and capable small surface combatant. To achieve the increased capability expectations for FFG(X), the Navy committed to acquiring a larger, more expensive ship than LCS or the previously planned FF. Figure 4 provides average shipbuilding cost estimates for the three different ships, with costs shown in same-year dollars for comparison. Although the FFG(X) requirements have been finalized, the Navy plans to make final cost and capability tradeoffs through the process of evaluating proposed designs before selecting which one will be built. In an effort to focus on the relationship between requirements and cost, the Navy undertook a conceptual design phase for FFG(X), which enabled industry to inform requirements and identify opportunities for cost savings. In February 2018, the Navy competitively-awarded FFG(X) conceptual design contracts valued at nearly $15 million each to five industry teams. These 16-month contracts were intended to enable industry to mature parent ship designs—designs for FFG(X) that are based on ships have been built and demonstrated at sea—and help refine technical and operational program requirements. The purpose of the conceptual design phase has parallels with the purpose of pre-contractual negotiations in commercial shipbuilding. As we previously have reported, these pre-contractual practices minimize ship buyer risk prior to awarding construction contracts by developing the ship concept and specifications based on negotiations between the ship buyer and the shipyard. The practices include specifying the expected performance and the major equipment on the ship. As part of these activities, commercial shipbuilders and ship buyers analyze one or more ship concepts to identify areas of potential risk and either mitigate these risks or remove the risky elements from the ship before signing a contract. Figure 5 provides an overview of the industry teams and shipyards participating in the FFG(X) conceptual design. Each industry team performed ship development, ship design, workforce planning, and shipyard improvement planning, among other activities, in support of FFG(X) requirements refinement and cost reduction efforts. Industry teams updated the Navy regularly on their design progress and technical approach to fulfill requirements through monthly technical exchange meetings and two design review meetings. Navy officials stated that these meetings were intended to provide information to support the program’s Preliminary Design Review in May 2019 and mitigate risk prior to the Navy’s release of its request for proposal in June 2019 for the FFG(X) detail design and construction competition. Our prior work on shipbuilding best practices emphasizes the importance of having a full understanding of the effort needed to design and construct a ship before awarding a contract for ship construction in order to reduce cost and schedule risk. Navy and industry officials stated that the conceptual design phase facilitated dialogue and information sharing that helped ensure FFG(X) requirements were more fully understood by industry and the government. Specifically, industry officials noted that communication and activities during conceptual design improved their understanding of the impetus for specific Navy requirements, allowing industry the opportunity to get clarification on the intent of some requirements, propose less costly alternatives, and get government feedback on the proposed alternatives. It also improved their understanding of the linkages between FFG(X)’s approved capability requirements and system specifications. In particular, industry officials told us that one-on-one opportunities with the Navy aided knowledge sharing and provided them with a means to ask questions without concern that disclosing such information could jeopardize their competitive position. They emphasized that in other cases where the request for proposals process is their primary means for communicating with the Navy (as opposed to having a conceptual design phase), submitting questions about requirements or system specifications can be challenging because those inquiries are available to the public. As a consequence, contractors may opt to infer more about the intent of requirements to avoid compromising their competitive interests. The conceptual design phase included a formal cost savings effort, with the Navy seeking proposals internally and from industry participants to reduce cost through requirement and system specification refinement. To support this effort, Navy officials stated they established a Frigate Affordability Board to review potential cost reduction measures submitted by both contractors and government that responded directly to Navy requirements and specifications. Navy officials said the Board—co- chaired by the Program Executive Office for Unmanned and Small Combatants and the Naval Sea System Command’s Naval Systems Engineering Directorate, as well as the Chief of Naval Operations’ Surface Warfare Directorate—assessed the potential cost and capability trade-offs of these proposed changes to requirements, and accepted or declined them. Before going to the Board, relevant Navy subject matter experts reviewed the technical and requirements implications of cost reduction measures. The program office subsequently worked with Navy engineering and requirements officials to balance cost with capabilities. If the program office, Navy engineers, and requirements officials could not reach agreement on the appropriate cost and capability mix, then their different positions were presented to the Board. For cost reduction initiatives submitted by industry, the Navy provided feedback on the Board’s decision, and incorporated fully or partially accepted cost reduction initiatives into the FFG(X) system specifications. Navy officials said they informed all industry teams of any changes to the specifications on a monthly basis. Navy officials also stated that industry submitted about 350 cost reduction ideas, with roughly 60 percent partially or fully accepted by the Navy. They estimated $86 million in savings per ship (constant year 2018 dollars) based on changes made in response to the cost reduction measures submitted by industry or government-initiated cost savings measures influenced by engagement with industry. In an effort to accelerate the time between FFG(X) acquisition planning and the fielding of ships, the Navy streamlined the program’s acquisition approach and leveraged knowledge obtained from industry during the conceptual design phase. While the program may benefit from the streamlining efforts, the acquisition approach for FFG(X) required the Navy to submit its budget request for lead ship construction before the program had a comprehensive understanding of the potential ship designs and cost. Recent Navy policy changes have created some uncertainty for Navy cost estimation activities by altering roles and responsibilities within the Navy for completing component cost positions and independent cost assessments. As permitted by DOD and Navy policy, the Navy has streamlined the FFG(X) acquisition approach to move from planning to ship delivery and fielding quicker than in a more traditional acquisition program. The accelerated schedule reflects the Navy’s desire to field a minimum of 52 small surface combatants, which the Navy’s long-range shipbuilding plan states will be achieved by fiscal year 2034. Navy officials stated that the significant amount of knowledge that already existed to inform the program’s early activities and the use of parent designs helped enable the streamlined approach for FFG(X). For example, Navy officials cited previous efforts by the Small Surface Combatant Task Force and the Frigate Requirements Evaluation Team to determine appropriate ship requirements, as well as activities performed in support of the FF frigate acquisition plan that immediately preceded the shift to FFG(X). The Navy also leveraged industry input received from a request for information in 2017 to understand cost drivers and the potential shipbuilders’ abilities to meet top level FFG(X) requirements and incorporate Navy-defined equipment into ship designs. Figure 6 provides a high-level schedule of key activities for the program. To support its decision to pursue an accelerated acquisition schedule, the Navy used the previously discussed conceptual design phase as well as its decisions to limit FFG(X) to parent ship designs and minimize technology development. Navy officials noted the use of parent designs is allowing the program to proceed at a much faster pace from early assessment of capability options to detail design and construction contract award. They added that the parent designs provided a higher- fidelity design baseline from which the conceptual design industry teams incorporated Navy systems and other requirements. Use of parent designs is consistent with our best practices work in shipbuilding, which has found that commercial shipbuilders use previous ship designs to the extent possible. Doing so can reduce technical, schedule, and cost risk in building a ship as compared to a “clean sheet” new ship design. FFG(X) program officials noted the latter approach can take up to 9 years to complete an analysis of alternatives and move through the acquisition process to construction contract award. Navy officials said the program also used opportunities available as an Acquisition Category (ACAT) 1B program to shorten the approval timeline for specific acquisition requirements. For an ACAT 1B program, the head of the DOD component is generally the Milestone Decision Authority but, as appropriate, may delegate approval authorities to lower level offices under its jurisdiction. In the case of FFG(X), the Assistant Secretary for the Navy for Research, Development, and Acquisition serving as the Milestone Decision Authority delegated specific approval authorities to the Program Executive Office for Unmanned and Small Combatants. These approval authorities applied to the program’s life cycle sustainment plan, independent logistics assessment, program protection plan, and a compliance schedule addressing environmental considerations. The Navy also took advantage of opportunities to alter or waive some significant early acquisition activities. For example, the Milestone Decision Authority waived the formal Analysis of Alternatives and Affordability Analysis, decided not to conduct a Milestone A review, and deferred the full “Should-Cost” Analysis to later in the acquisition process. Table 2 defines the purpose of these DOD acquisition program elements and provides an overview of the Navy’s actions related to them. As the first major milestone for many major acquisition programs, Milestone A is a review by the Milestone Decision Authority of key program documents that support the materiel solution and risk reduction. We have previously found that DOD officials place a high value on the information developed for some of these documents, including the Analysis of Alternatives, Affordability Analysis, and Should-Cost Analysis. The Navy’s decision to not conduct a Milestone A review also eliminated a formal opportunity to bring the broad set of FFG(X) stakeholders within the Navy and the Office of the Secretary of Defense together at a relatively early stage to assess the program’s acquisition strategy and affordability and feasibility, as well as technical, cost, and schedule risks. Further, it reduced the FFG(X) acquisition approach to a single milestone decision point—Milestone B—for the broader group of DOD stakeholders to evaluate program progress and readiness to proceed to the detail design and construction contract award planned in July 2020. In the absence of Milestone A, the Navy’s Gate 3 review for FFG(X) provided an opportunity to communicate the program’s progress toward developing requirements and acquisition expectations, albeit to a more limited audience than typically would participate in a Milestone A. In particular, the Navy used Gate 3 to discuss top-level requirements changes and receive capability development document approval from the Chief of Naval Operations. It also included cost discussion related to FFG(X) affordability within the overall Navy shipbuilding portfolio. The gate’s participants included officials from the Navy and the Office of Cost Assessment and Program Evaluation (CAPE) within the Office of the Secretary of Defense. The Navy’s Gate 4 conducted in February 2019 focused on a review of the FFG(X) system specification before the draft detail design and construction request for proposal release. Gate 4 documentation for FFG(X) indicates that participants were limited to stakeholders from the office of the Deputy Assistant Secretary of the Navy for Ships; Naval Sea Systems Command Cost and Design Directorates; Program Executive Office for Unmanned and Small Combatants; the FFG(X) program office; and the Chief of Naval Operations Surface Warfare Directorate. This excludes a number of key stakeholders that Navy guidance calls on to attend and certify gate reviews, such as the Assistant Secretary of the Navy (Financial Management and Comptroller) and the testing community. As a result, the Navy would not have received insight from several key stakeholders during the Gate 4 review for acquisition activities, such as the program life cycle cost estimate development and release of the draft request for proposal. These activities are generally relevant to this gate review, as Navy guidance notes program affordability as a focus and the Navy’s streamlining documentation indicates that the gate was focused on reviewing the FFG(X) system specification before releasing the draft request for proposal. Navy officials noted that stakeholders have regularly received insight into FFG(X) activities through other prior program reviews and will have additional opportunities to review program costs and sustainment plans leading up to Milestone B. We also found that some key stakeholders did not provide formal approval for the initial FFG(X) life cycle sustainment plan that was approved in March 2019. Specifically, only FFG(X) program officials and the Program Executive Officer for Unmanned and Small Combatants— the delegated approval authority—signed the plan. However, as stated in DOD guidance, representatives from the relevant sustainment command and the Program Executive Office for Integrated Warfare Systems are key stakeholders that should provide their signed concurrence when approving the life cycle sustainment plan. The FFG(X) life cycle sustainment plan is a key document outlining the Navy’s plans to address the program’s sustainment needs and costs, as typically around 70 percent of a weapon system program’s total cost is in the sustainment phase after procurement. Navy officials stated that the plan has been reviewed by the independent logistics assessment team members that are evaluating the FFG(X) program’s integrated product support activities, and noted that the Program Executive Office for Integrated Warfare Systems has separate life cycle sustainment plans for government furnished equipment systems included in the FFG(X) design. Navy officials also said that FFG(X) sustainment plans would be reviewed by stakeholders as part of Gate 5 and the Milestone B independent logistics assessment. The FFG(X) acquisition approach required the Navy to submit its nearly $1.3 billion budget request for lead ship construction before the program had established a comprehensive understanding of the potential ship designs and estimated cost for the program. Our shipbuilding and acquisition best practices call for resource decisions to be timed to align with the availability of requisite cost, schedule, and technical knowledge in order to inform key program decisions. Navy officials stated that they had sufficient knowledge to inform key program decisions based on cost estimation and conceptual design efforts that had previously been completed. Navy officials said this included development of an FFG(X) cost estimate by November 2018 to support a realistic budget request for the lead ship. However, at the time of the Navy’s fiscal year 2020 budget request to fund detail design and the lead ship, the Navy had not completed its component cost position, which will formalize the life cycle cost expectations for FFG(X). Further, CAPE had not completed the independent cost estimate for the program. The GAO Cost Estimating and Assessment Guide says that comparing the component cost position with an independent cost estimate to validate methodologies produce similar results reinforces the credibility of a cost estimate. In addition to key cost estimating best practices that had not been completed, the Navy had not received final design review information from the industry teams participating in the conceptual design phase before requesting lead ship funds from Congress. Figure 7 reflects the budget request timeline for the FFG(X) detail design and lead ship contract award, as well as notable cost and design-related program activities that were planned to be completed after the request. The considerable cost growth that we have previously reported is common to many shipbuilding programs, as well as challenges in deviating from shipbuilding plans once a program has begun procuring ships, emphasize the importance of having a strong understanding of program expectations to back the initial procurement decision for FFG(X). Given the timing of the Navy’s budget request for lead ship funding, Congress faces a decision on whether to authorize funding for FFG(X) detail design and lead ship based on a budget request that was not informed by key cost and design information. If Congress authorizes and appropriates FFG(X) funding as the Navy requested in March 2019, it will be critical that the Navy demonstrate the program’s acquisition program baseline reflects the results of the component cost position and independent cost estimate before awarding the detail design and construction contract. Doing so before the contract award will help ensure a more reliable acquisition program baseline upon which future costs and variances are measured and funding decisions are made. Further, it will help mitigate remaining risk that stems from the Navy not being able to account for the actual FFG(X) design and associated estimated cost for ship construction until after the planned July 2020 contract award. Specifically, as currently planned, the Navy’s budget requests for fiscal years 2020 and 2021—which are intended to fund the first 3 ships—will be made before the Navy has agreed to contract pricing for FFG(X). Navy officials stated that they have completed a robust program life cycle cost estimate. They noted that the estimate was informed by Navy modeling of a notional ship design that leveraged data received from industry during conceptual design and reflected ship design elements needed to meet program requirements Navy officials also said that, as of May 2019, some additional work remains for the cost estimate to account for training and military construction considerations, as well as address any needed changes related to the final industry design reviews for the conceptual design phase. They also said that the program life cycle cost estimate informed the Gate 4 review in February 2019, and an updated version of the estimate will provide a basis for the Navy’s efforts to establish the component cost position in October 2019. As of the issuance of this report, we have requested the program life cycle cost estimate from the Navy, including the estimate’s criteria and underlying assumptions, but have not yet received this information. Recent policy changes by the Navy related to cost analysis and estimation have created some uncertainty for Navy cost estimation activities going forward. Specifically, a March 2019 Secretary of the Navy instruction for acquisition program cost analysis shifts the Naval Center for Cost Analysis’s role and responsibilities for Navy cost estimation to the Navy’s systems commands. Previously, the Naval Center for Cost Analysis—organizationally residing completely outside of the systems command structure—would provide an independent cost assessment of the program life cycle cost estimate. The Naval Center for Cost Analysis and the acquisition program, in coordination with the relevant systems command, would discuss and adjudicate any differences between the program life cycle cost estimate and the independent cost assessment to produce the Navy’s component cost position. This independent cost assessment by the Naval Center for Cost Analysis was an important verification of the program office estimates, which were often found to be too optimistic, prior to the Navy finalizing its component cost position. The Navy’s recent changes for cost estimation and analysis may pose a risk of overly optimistic estimates carrying forward in programs. Navy officials stated that they believe Naval Sea Systems Command cost estimators can provide an independent cost estimate, as they are intended to provide technical support to acquisition programs independent of programmatic authority and report to a separate chain of command. However, as stated by the Naval Sea Systems Command, the collective mission of its organizations is to build, buy, and maintain the Navy’s ships. Based on this, we believe, as do CAPE officials with whom we spoke, that shifting independent cost assessment activities to the systems commands diminishes the Navy’s ability to independently verify a program life cycle cost estimate. As a result, the program life cycle cost estimate essentially will become the component cost position based on the lack of additional cost estimation input, such as what the Naval Center for Cost Analysis previously provided. Furthermore, CAPE officials stated that having a systems command execute cost analysis responsibilities for an acquisition program within the same system command effectively eliminates the Navy’s capacity to perform independent cost estimates for its programs based on their shared overarching mission. This position is consistent with the GAO Cost Estimating and Assessment Guide, which states that an independent cost estimate should be conducted by an organization independent of the acquisition chain of command. The Director of CAPE is required to conduct or approve independent cost estimates and cost analyses for all major defense acquisition programs. As noted by CAPE officials, CAPE has previously delegated certain cost estimation responsibilities to the Naval Center for Cost Analysis. With the recent Navy policy changes, CAPE may no longer choose to delegate independent cost estimation activities to Navy cost estimators. For FFG(X), CAPE intends to complete an independent cost estimate to verify the Navy’s component cost position. These plans include site visits and data collection from the shipyards participating in the conceptual design contracts. CAPE confirmed that the final independent cost estimate will reflect the content of the winning proposal, indicating that any FFG(X) proposals that the Navy receives from contractors not involved in the conceptual design phase will be evaluated to ensure the independent cost estimate accounts for those cost and design plans. CAPE officials also stated that their timeline for finalizing the independent cost estimate for FFG(X) is tied to when the Navy decides on the winning proposal for detail design and construction and communicates this information to CAPE. Specifically, CAPE’s final independent cost estimate will reflect only the winning FFG(X) design, so completion of the estimate will occur after the Navy informs CAPE about the FFG(X) design for which it intends to pursue a contract award. CAPE officials said that because the Navy’s decision may not be made in advance of the planned February 2020 Milestone B review for FFG(X), CAPE would likely just provide input to support the milestone and complete the independent cost estimate after that review. The Navy’s decision to pursue a parent ship design for FFG(X) was intended to reduce design uncertainty for the program. The Navy’s planned use of existing technologies for the ship’s mission and combat systems also supports reduced technical risk, though further maturation of some key systems and successful integration and testing will be critical to demonstrate the ship provides required capability within cost and schedule expectations. Adopting a parent design requirement for FFG(X) provided the conceptual design industry teams with a proven baseline ship design. This enabled them to focus on incorporating modifications to meet the Navy’s specific FFG(X) requirements rather than designing a new ship. The Navy did not set any limitations on the extent contractors could modify or deviate from the parent design. However, Navy officials stated they actively reviewed parent design modifications through contract deliverables, technical exchange meetings, and design reviews with industry teams. The design reviews included an interim report in October 2018 and a final report in May 2019 from each industry team on their design progress. FFG(X) program officials noted that the design maturity reviews provided sufficient information to support the Navy’s decision that the designs were mature enough to release the request for proposals for the detail design and construction contract award. In addition, some industry officials told us that the conceptual design work on parent designs enabled them to develop more mature and refined designs than typical for this stage of the shipbuilding acquisition process. They also noted that continuing work in response to the pending competition should move at least some design elements closer to a detail design-level of maturity, and may provide the Navy with greater confidence in the contract proposals it receives from industry. The FFG(X) program’s design concept requires the use of many existing, more mature combat and mission systems to reduce technical risk. As stated in the approved acquisition strategy for FFG(X), the program has a requirement for all integrated systems to have achieved maturity of a technology readiness level (TRL) 6 or higher. TRL 6 is defined by GAO as the capability to produce a prototype system in a production-relevant environment. Program officials confirmed that, as of May 2019, many but not all FFG(X) integrated systems were at TRL 6 or higher. For selected key systems planned for FFG(X), Navy officials stated they will have achieved TRL 7 or higher by the planned July 2020 detail design and construction contract award. Doing so would be consistent with our acquisition best practices, which include maturing new key ship technologies into actual system prototypes and demonstrating them in a realistic environment—achieving a TRL 7—before the award of the contract for lead ship design and construction. This practice helps reduce the likelihood of costly design changes later. Many of the systems planned for FFG(X) have been demonstrated and are in use on other Navy ship classes, which helps the program fulfill capability needs while avoiding developmental risks. Table 3 provides an overview of some of the key existing systems planned for the ship. In addition to the systems that have been utilized by other Navy ships, the FFG(X) program plans to incorporate some systems that are still in development, such as the Enterprise Air Surveillance Radar (EASR) and a new version of the Aegis Weapon System. Navy officials stated that EASR—a complex radar system expected to provide long-range detection and engagement of advanced threats— is critical to FFG(X)’s air and surface warfare missions. It is a scaled down version of the Navy’s Air and Missile Defense Radar that is in production and scheduled for initial integration with the Aegis combat system on a DDG 51 Flight III destroyer in fiscal year 2020. In early 2019, the Navy began testing a full-scale, single-face EASR array engineering developmental model—the full system planned for FFG(X) will have three array faces—at a land-based test site to further demonstrate its functionality. The Navy expects to complete land-based testing of the EASR engineering development model by February 2020. The Navy also plans to integrate a rotating version of EASR and a fixed-face version on other ship classes prior to integrating the radar on the lead FFG(X). The Navy’s results from planned EASR developmental testing at the land-based site will be integral to achieving a TRL 7 and reducing risk prior to the start of FFG(X) detail design. The Navy is developing a new version of the Aegis Weapon System— FFG(X)’s combat management system—to coordinate radar and weapons system interactions from threat detection to target strike. For example, the system will support the ship’s ability to employ the Naval Strike Missile for over-the-horizon offensive capability as well as a 32- cell vertical launch system to employ missiles for air defense. The Aegis Weapon System for FFG(X) will leverage the Aegis common source software that supports the combat systems found on the Navy’s DDG 51-class destroyers and CG 47-class cruisers. Navy officials noted that they anticipate at least 70 percent of the Aegis Weapon System software for FFG(X) will be common to the Aegis software used for DDG 51 Flight III ships. Rigorous testing of the Aegis Weapon System with EASR will be critical for FFG(X), as the radar and combat management system must work in concert for the ship to detect, track, and assess possible targets. Given the radar and software commonalities, the risk level for both of these FFG(X) systems should be reduced once the DDG 51 Flight III radar and Aegis system baseline, upon which the FFG(X) integrated system is based, have been demonstrated through testing on a ship beginning in 2022. Specific to the Aegis Weapon System for FFG(X), software development is expected to run from fiscal year 2022 to late fiscal year 2024. The system’s integration and testing with EASR is scheduled to occur through fiscal year 2024. While the Navy is planning to use many already mature systems on FFG(X), integration and testing of those systems will be critical to demonstrate systems fit and work together as intended on the ship. The Navy completed a technology readiness assessment in spring 2019 to identify potential technical risks, and concluded that FFG(X) does not have any critical technology elements. DOD generally defines a critical technology element as one that may pose major technological risk during development. Navy officials who completed the assessment stated that they reviewed about 150 systems as part of their activities and found none composed of new or novel technologies for which the Navy has insufficient knowledge to demonstrate maturity. The assessment noted one technology—the New Advanced Integrated Line-of-Sight Equipment System (nAILES) multi-coupler for antennas—as a watch item. The Navy would like to utilize nAILES for FFG(X), but according to Navy officials, it is not considered a critical technology because the Navy has identified alternative, proven technologies that will be used to meet the ship’s needs if nAILES is not available for use. The findings of the technology readiness assessment are consistent with the FFG(X) program’s decision to use existing systems that do not require technological innovation to deliver desired capability. However, the findings do not necessarily equate to the program having no technology risk for planned systems. For example, the Aegis Weapon System for FFG(X) did not qualify under the parameters of the technology readiness assessment as a critical technology element. Still, as already discussed, the Aegis Weapon System will carry technical risk for several years until the Navy completes development and demonstrates the system works as intended for FFG(X). The Next Generation Surface Search Radar is another system that is relatively mature—FFG(X) program officials confirmed in May 2019 it is nearing a TRL 6—but requires further development to reduce risk. The FFG(X) test and evaluation master plan and independent technical risk assessment are significant documents yet to be completed that will help to further define risks and plans to address them. The test and evaluation master plan serves to outline the program’s integrated test program and master schedule of major test events or phases. Navy officials expect the test plan to be approved in December 2019 to support the Milestone B decision. They noted that the plan may need to be updated once the FFG(X) design is selected based on the additional information that will be available to inform test planning. The independent technical risk assessment is intended to categorize risks that cover a broad range of factors, including technology maturity, integration needs, and testing. If these factors are not sufficiently accounted for, a program is likely to have difficulty meeting cost, schedule, and performance objectives. An official from the Office of the Under Secretary of Defense for Research and Engineering who is participating in the technical assessment for FFG(X) stated they plan to complete their work to identify any risks in March or April 2020. The official added that at this early stage of their activities, the potential for integration risks associated with the FFG(X) combat system is an area of interest because of the extensive number of existing systems that will need to be integrated into the new ship design. Navy test officials as well as DOD systems engineering and test officials noted potential advantages and risks related to FFG(X) program’s plans for using existing technologies. Similar to what we previously discussed about the use of a parent design, the officials stated that the use of existing systems can increase understanding of the ship and its systems, which may help the FFG(X) program achieve its planned accelerated timeline between development and delivery. However, systems engineering and test officials also indicated that, regardless of maturity, challenges typically arise when DOD takes systems from other platforms and attempts to integrate and use them in new ways on a new platform. They cautioned that programs like FFG(X) that plan to use a lot of government-furnished equipment or non-developmental systems often underestimate the amount of integration challenges they will face. The officials told us this may occur because of overconfidence that the maturity of systems demonstrated through use on other platforms eliminates most technical risk, whereas experience confirms that it is always challenging to get systems to fit and work together as intended on a new platform. Officials from the office of the Director, Operational Test and Evaluation said that the parent design approach for FFG(X) may enable the Navy to reduce some developmental testing activities; however, operational testing expectations would largely be unaffected because there will still be substantial integration to be completed and tested in order to demonstrate mission capabilities. The draft FFG(X) request for proposal indicates that the Navy plans to use a fixed-price incentive contract to help control ship costs and special performance incentive fees. In addition, the Navy plans to use guarantees with limited liability for the shipbuilder to correct defects after ship deliveries. Our prior work has found that using comprehensive ship warrantees instead of guarantees could reduce the Navy’s financial responsibility for correcting defects. After completion of a full and open competition for FFG(X) detail design and construction, the Navy plans to use a fixed-price incentive contract in combination with additional special performance incentive fees to procure the lead and follow-on ships. As we have previously reported, full and open competition allows all responsible sources—or prospective contractors that meet certain criteria—to submit proposals for a contract. The use of competition in contracting is a critical tool for achieving the best possible return on investment for taxpayers. Competitively awarded contracts can save the taxpayer money, improve contractor performance, and promote accountability for results. The fixed-price incentive contracting approach for FFG(X) is intended to incentivize the contractor to control costs and meet performance requirements. This contracting strategy represents a significant departure from previous surface combatant programs in which the Navy negotiated cost-reimbursement contracts for construction of the lead ship. Under cost-reimbursement contracts, the Navy assumes the cost risk because the shipbuilder is reimbursed for its allowable incurred costs to the extent prescribed in the contract, regardless of whether the work is performed to the exact level desired by the Navy. For example, our prior work found that the Navy’s decisions to accept the first two LCS in incomplete, deficient conditions complied with federal acceptance provisions, largely due to the cost- reimbursement type contracts in place to construct these ships. Fixed-price incentive contracts provide an incentive for the shipbuilder to control costs in order to maximize profit. Fixed-price incentive contracts generally include a profit adjustment formula referred to as a shareline, as well as a price ceiling, target cost, and target profit. The structure of the shareline establishes how cost overruns or underruns in relation to a target cost are shared between the government and shipbuilder. For example, the 70/30 shareline that the Navy is planning for FFG(X) lead ship overruns means that the government pays 70 percent of cost and the shipbuilder pays 30 percent when the cost exceeds the target cost up to the price ceiling. Generally, the shareline functions to decrease the shipbuilder’s profit as actual costs exceed the target cost. The price ceiling is generally the maximum the government will pay under the contract and is typically negotiated as a percentage of the target cost. The target cost generally informs the shareline and price ceiling. Given the unknowns associated with design and construction, the Navy plans to account for these unresolved risks by assuming responsibility for cost growth above DOD recommended guidance. As we reported in March 2017, when the Navy assumes a greater share of cost overruns above the target cost, accepts a higher price ceiling, or both, the fixed- price incentive elements may not provide sufficient motivation for the shipbuilders to control costs. Figure 8 depicts the how risk changes as the Navy departs from a 50/50 shareline for cost overruns and a ceiling price of 120 percent. As we previously noted, for the FFG(X) lead ship the Navy is planning to have a shareline of 70/30 for target cost overruns. The Navy also plans to have a 60/40 target cost overrun shareline for the second ship, and a 50/50 overrun shareline for the remaining seven ships included in the detail design and construction contract award. Based on this plan, the first two FFG(X) ships will depart from DOD’s guidance recommending a 50/50 point of departure for negotiations between the government and shipbuilder for cost overruns up to the price ceiling. This results in more cost risk to the government for two ships in the detail design and construction contract. The Navy’s planned price ceiling for the 10 ships included in the contract award may deviate from DOD’s guidance recommending a ceiling price set at 120 percent of target cost as a point of departure for fixed price incentive contracts. Specifically, Navy officials stated that the maximum ceiling price could be as high as 125 percent for all of the ships. However, Navy officials stated that the request for proposal will provide incentive for industry to propose the minimum price ceiling that sufficiently accounts for the proposal’s level of risk, meaning that industry may propose price ceilings below 125 percent. The Navy also plans to include options for a special performance incentive fee for each of the FFG(X) ships, which will be established for the final request for proposal. These incentives have the potential to increase shipbuilder profitability. As outlined in the FFG(X) draft detail design and construction request for proposal and confirmed by program officials, each frigate will have a guaranty period that commences at ship delivery and is expected to end 18 months after delivery. Navy officials stated the guaranty is intended to formalize a period of responsibility during which the shipbuilder must correct defects, with the cost to the government and the contractor based on the contract terms (cost shareline and price ceiling) associated with the ship. During the guaranty period, the shipbuilder would be required to correct all defects for which it is responsible, with proposals required to include a minimum limitation of liability of $5 million per ship. Once the total cost to correct identified defects reaches $5 million, the government would pay the full cost to correct any additional guaranty period defects. The $5 million minimum limitation of liability planned for FFG(X) has a higher dollar value and covers a longer period of time than other recent shipbuilding programs. For example, we previously found that for the Navy’s LPD 25 amphibious transport dock construction, the contract initially included a $1 million limitation of liability. Navy officials stated that the final request for proposal also will include a provision allowing industry to propose a higher liability limit, up to and including no limitation of liability. Navy officials said that any additional liability amount proposed beyond the $5 million guaranty will be assessed as part of the technical evaluation criteria used to select the winning FFG(X) design. We found in March 2016 that the use of a guaranty did not help improve cost or quality outcomes for the Navy and Coast Guard ships we reviewed. We also found that commercial ship buyers and Coast Guard officials stated that warranties foster quality performance because the shipbuilder’s profit erodes as it spends money to correct deficiencies after delivery, during the warranty period. We further reported that the Coast Guard has improved cost and quality by requiring the shipbuilder to pay to repair defects by following Federal Acquisition Regulation warranty provisions. For example, the Coast Guard paid up front for the Fast Response Cutter warranty. The cost of the warranty amounted to 41 percent of the total defect correction costs. Although this ship does not have the size and advanced systems planned for FFG(X), it serves to demonstrate the potential value to the government presented by the use of warranties. The Coast Guard also used a fixed-price incentive contract with a warranty on its Offshore Patrol Cutter—a ship of comparable size to FFG(X). The first Offshore Patrol Cutter has a 2-year warranty, and follow-on ships will have 1-year warranties. The Coast Guard pays a set amount for these warranties, and in return, the shipbuilder must fix all applicable defects identified within the agreed-upon time period regardless of cost. Rather than using guarantees for the FFG(X) contract to provide for the correction of defects, the Navy could help control costs to the government through the use of warranties. Under warranties, the government generally receives a contractual right for the correction of all defects for which the shipbuilder is responsible at the shipbuilder’s expense. The use of warranties is typically not mandatory, but federal and defense acquisition regulations instruct contracting officers to consider various factors when deciding whether a warranty is appropriate for an acquisition. The regulations also instruct contracting officers to use a warranty when it is practicable and cost-effective to do so. We previously found that, unlike a warranty, the Navy almost exclusively paid for defects that were the shipbuilder’s responsibility under a guaranty because of the contract type and terms in contracts that we reviewed. Such conditions limit the incentive to discover every deficiency during the guaranty period, and may negatively affect quality improvements over time. The Navy’s FFG(X) plans suggest that the Navy may be prematurely discounting warrantees as a mechanism to improve ship quality and decrease cost to the government. Navy officials told us that mandating that industry propose a warranty could result in additional costs to the government because the initial cost of the ship could be raised substantially to include the cost of the warranty. Additionally, Navy officials said a requirement for warranty pricing could serve to limit industry participation in the FFG(X) competition if offerors are unwilling to accept the risk associated with a warranty and unable to provide reasonable pricing. The Navy provided no analysis to support these claims and confirm a clear understanding of whether a warranty could provide greater value than the $5 million guaranty the Navy is proposing for FFG(X). As part of the competitive proposal process for FFG(X) detail design and construction, the Navy could maintain its plans to require a guaranty but also seek ship warranty pricing. The full and open competition for the FFG(X) contract award may increase the potential for receiving warranty pricing that provides a cost-effective alternative to the Navy’s guaranty plans. By limiting the request for proposal to guarantees, the Navy misses an opportunity to obtain information on what comprehensive warranty coverage against defects would cost, and use it to evaluate whether warranties could further reduce risk for the FFG(X) program. As the Navy approaches the Milestone B review for FFG(X), it is critical that funding and other major programmatic decisions are fully informed by the knowledge necessary to support them. This is especially important to help ensure that the FFG(X) program does not face some of the same cost, schedule, and performance shortfalls that have been faced by the LCS program. The Navy’s fiscal year 2020 budget request to authorize and appropriate funding for the lead frigate was developed and submitted without the benefit of key cost and design information, such as the independent cost estimate and the final results from conceptual design. As a result, it is necessary that the Navy provide Congress with a clear understanding of FFG(X) cost expectations, including CAPE’s independent cost estimate, prior to awarding the detail design and construction contract. This will help ensure that the FFG(X) program is grounded in cost and design expectations that reflect the specific aspects of the ship that the Navy selects for construction. With the start of the planned $20 billion FFG(X) procurement approaching, the Navy has limited time left to position the government to obtain the best deal possible to fix any deficiencies discovered upon delivery of the first 10 ships. The Navy’s guaranty plan for FFG(X) offers some improvements compared to recent shipbuilding programs, but does not offer the degree of coverage that could potentially be provided by a warranty. The competitive qualities of the FFG(X) acquisition approach present an opportunity for the Navy to, at a minimum, obtain warranty pricing from industry so that the program may use that input to evaluate whether a warranty would be a cost-effective means of reducing the government’s cost risk. We are making two recommendations to the Secretary of the Navy: Ensure that the Assistant Secretary of the Navy for Research, Development, and Acquisition provides to Congress the finalized independent cost estimate prior to award of the detail design and construction contract and demonstrates that the estimate is consistent with the fiscal year 2020 budget request for the lead ship. (Recommendation 1) Ensure that the Assistant Secretary of the Navy for Research, Development, and Acquisition directs the FFG(X) program office to request pricing for warranties for the lead ship and the nine follow-on ship options planned for FFG(X) as part of the detail design and construction request for proposals. (Recommendation 2) We provided a draft of this report to DOD for comment. DOD provided written comments, which have been reproduced in appendix I. In responding to the draft report, DOD concurred and described the actions it planned to take to address our two recommendations. In response to the second recommendation to request pricing for warranties for the lead ship and the nine follow-on ship options planned for FFG(X) as part of the detail design and construction request for proposals, DOD acknowledged that the Navy will receive guaranty rather than warranty pricing, but stated that the solicitation allows industry to propose a higher limitation of liability amount, up to an unlimited limitation of liability, in its guaranty pricing for FFG(X). While this could allow for a better value to the government than has been typical for recent shipbuilding programs, permitting higher limitation of liability guaranty pricing but not requesting warranty pricing from offerors means the Navy will not have complete information on whether a warranty could be more cost-effective than a guaranty. Our prior work found that the use of Federal Acquisition Regulation warranty provisions improved shipbuilding program cost and quality outcomes. As a result, we maintain our belief that the FFG(X) program office should implement this recommendation by seeking warranty pricing as part of the detail design and construction request for proposals. The full and open competition for the FFG(X) contract award may increase the potential for receiving warranty pricing that provides a cost-effective alternative to the Navy’s guaranty plans. DOD stated that modifying the solicitation to incorporate a warranty pricing component would cause an unacceptable delay to the FFG(X) program, but did not provide an analysis to support this assertion or specify the extent of delay associated with adding a warranty pricing request. The current FFG(X) schedule has roughly 10 months between the request for proposals deadline and the contract award, and the program originally had been planning for the solicitation period to end in December 2019 before moving the deadline to September 2019 shortly before its release. We recognize the substantial effort the proposal development and review process requires, but we continue to believe that the government would benefit from adding a request for warranty pricing to the detail design and construction solicitation. While DOD stated that the Navy will support the recommendation after award by requesting pricing for an unlimited warranty before exercising the first ship option, doing so would eliminate any potential warranty pricing advantages that would occur as a result of the competitive conditions that currently exist for the current detail design and construction contract. In addition to DOD’s written response to the report, DOD officials and industry representatives associated with the FFG(X) conceptual design activities provided separate technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Secretary of the Navy. This report will also be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff members have any questions regarding this report, please contact me at (202) 512-4841 or oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to the report are listed in appendix II. Shelby S. Oakley, (202) 512-4841 or oakleys@gao.gov. In addition to the contact named above, the following staff members made key contributions to this report: Diana Moldafsky (Assistant Director), Lori Fields, Kurt Gurka, Stephanie Gustafson, Chad Johnson, Jennifer Leotta, Sean Merrill, Miranda Riemer, Jillena Roberts, Hai Tran, and Alyssa Weir.", "summary": "In response to the shortcomings of the Navy's Littoral Combat Ship program and evolving threats, the Navy began the FFG(X) program. With FFG(X), the Navy intends to deliver a multi-mission ship that will provide anti-surface, anti-submarine, and air warfare capabilities. DOD approved FFG(X) requirements in February 2019.The Navy plans for a competitive contract award to support final FFG(X) design and construction. The program is expected to cost over $20 billion for 20 ships. The House report accompanying the National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to review the FFG(X) program. This report addresses, among other things, the FFG(X) acquisition approach and contracting plans. GAO reviewed requirements, acquisition, design, and cost-related documentation. GAO interviewed Navy and other defense officials, and conducted industry site visits to each shipyard participating in FFG(X) conceptual design activities. GAO also leveraged prior GAO reports and best practices guides. The Navy undertook a conceptual design phase for the FFG(X) Guided Missile Frigate program that enabled industry to inform FFG(X) requirements, identify opportunities for cost savings, and mature different ship designs. The Navy also streamlined the FFG(X) acquisition approach in an effort to accelerate the timeline for delivering the ships to the fleet. As shown in the figure, however, the Navy has requested funding for the FFG(X) lead ship even though it has yet to complete key cost estimation activites, such as an independent cost estimate, to validate the credibility of cost expectations. Department of Defense (DOD) cost estimators told GAO the timeline for completing the independent cost estimate is uncertain. Specifically, they stated that this estimate will not be finalized until the Navy communicates to them which FFG(X) design is expected to receive the contract award. GAO-identified best practices call for requisite cost knowledge to be available to inform resource decisions and contract awards. The Navy plans to use a fixed-price incentive contract for FFG(X) detail design and construction. This is a notable departure from prior Navy surface combatant programs that used higher-risk cost-reimbursement contracts for lead ship construction. The Navy also plans to require that each ship has a minimum guaranty of $5 million to correct shipbuilder-responsible defects identified in the 18 months following ship delivery. However, Navy officials discounted the potential use of a warranty—another mechanism to address the correction of shipbuilder defects—stating that their use could negatively affect shipbuilding cost and reduce competition for the contract award. The Navy provided no analysis to support these claims and has not demonstrated why the use of warranties is not a viable option. The Navy's planned use of guarantees helps ensure the FFG(X) shipbuilder is responsible for correcting defects up to a point, but guarantees generally do not provide the same level of coverage as warranties. GAO found in March 2016 that the use of a guaranty did not help improve cost or quality outcomes for the ships reviewed. GAO also found the use of a warranty in commercial shipbuilding and certain Coast Guard ships improves cost and quality outcomes by requiring the shipbuilders to pay to repair defects. The FFG(X) request for proposal offers the Navy an opportunity to solicit pricing for a warranty to assess the cost-effectiveness of the different mechanisms to address ship defects. GAO recommends that the Navy provide Congress with the independent cost estimate for FFG(X) prior to the detail design and construction contract award and seek ship warranty cost information from industry as part of the request for proposal process. While DOD generally concurred with GAO's recommendations, it did not agree to update its request for proposal to solicit ship warranty pricing. GAO continues to believe this is an essential element of the recommendation, as discussed in the report.", "document_type": "gao"}
{"report": "The military departments have been privatizing utility systems at military installations since 1988. In 1997, Congress provided the military departments permanent statutory authority, codified at 10 U.S.C. § 2688, as amended, to convey, or privatize, utility systems under military jurisdiction, such as those on military installations. The authority defines a utility system as a system for the generation and supply of electric power; the treatment or supply of water; the collection or treatment of wastewater; and the supply of natural gas, among others. When privatizing a utility, the Secretary of a military department makes a decision to convey a system to a private or public entity, and then a utility services contract is awarded. Figure 1 shows examples of common utility systems found on military installations. A utility system includes the associated equipment, fixtures, and structures, as well as easements and rights-of-way. 10 U.S.C. § 2688 states that the Secretary of a military department may convey a utility system, or part of a system, to a municipal, private, regional, district, or cooperative utility company or other entity. DOD’s policy states that the military departments may maintain ownership of utility systems and decide not to privatize them for security reasons, or when privatization is determined to be uneconomical. According to officials, once DOD conveys a utility system and awards a contract for utility services, the contractor is responsible for replacing, repairing, and maintaining the associated equipment and structures as needed. Figure 2 provides photos of the before and after condition of a privatized utility system component at Fort Riley, Kansas where the electrical system was modernized to replace analog monitoring equipment with digital equipment. The Office of the ASD(Sustainment), within the Office of the Secretary of Defense, develops policies for and oversees DOD’s utilities privatization program. There are two main sources of DOD policy for utilities privatization—a DOD instruction on energy management at the installation level and a supplemental guidance specific to utilities privatization. During the period covered by our review, the instruction directed the military departments to attempt to privatize all utility systems, unless the Secretary of the military department determines that the system is exempt from privatization for security or economic reasons. In February 2019, DOD released supplemental guidance, which, among other things, superseded the relevant portions of the instruction (and cancelled prior supplemental guidance), and did not include a preference for privatization or the direction to complete privatization decisions on all covered utility systems. Instead, utilities privatization may now be performed at the discretion of the military departments. The military departments have the responsibility for program implementation, as the statutory authority to privatize utility systems is granted to the Secretaries of the military departments. As such, the military departments determine which systems will be privatized and which systems may be exempted from privatization. Once a military department begins to consider an installation for privatization, the installation command assists and facilitates in carrying out the privatization effort. According to officials, DLA Energy is the contracting agent for the majority of privatized utility services contracts awarded on behalf of the Army since 2004 and for the Air Force since 2008. Navy officials noted that NAVFAC is the contracting agent and administrator for the Navy and Marine Corps privatized utility services contracts. As of December 2019, the military departments have privatized roughly a quarter of the utility systems on military installations (614 of the 2,590 systems); roughly a third of the systems were already owned by entities other than the federal government (733 of 2,590) (see table 2). As reflected in the table, the military departments have identified 580 utility systems that are available for future utilities privatization. As of September 2018, DLA Energy reported that it had 18 ongoing utilities privatization efforts—12 for Army and six for Air Force. Also, the Navy noted that it has three ongoing utilities privatization efforts. According to Air Force and Navy officials, their departments took a “strategic pause” on new utilities privatization efforts in 2015 to determine if privatization is the best option for recapitalizing their deteriorating utility systems. The Navy and Air Force resumed new utilities privatization efforts in fiscal year 2017 and fiscal year 2019, respectively. DLA Energy will act as contracting agent for the Navy on a pilot basis, as well as continuing to do so for the Army and Air Force for future contract awards. The process for privatizing a utility system culminates in two actions: the award of a utility services contract and conveyance of the physical assets of the utility from the military department to the awardee. Once the military department has decided to consider privatizing a utility system at an installation, the department initiates efforts to award one or more utility services contracts. This contracting process is governed by federal statutes, the Federal Acquisition Regulation (FAR), the DOD and military department supplements to the FAR, and military department and agency guidance. For example, DOD is generally required to award utility services contracts using competitive procedures, but can award contracts through other than competitive procedures when authorized by an exception, which we refer to as non-competitive. Figure 3 depicts the five phases of the pre-award contracting process identified by GAO. Acquisition Planning: Acquisition planning includes developing requirements, preparing cost estimates, and conducting market research to determine market interest, among other activities. For utilities privatization efforts, requirements also include the inventory of equipment—such as pipes, valves, and wires—and structures associated with the utility system. For privatized utility services contracts this phase begins with the decision to consider the privatization of utilities at a given installation and generally ends with the approval of an acquisition strategy. Solicitation: Military departments may solicit offers from prospective contractors by issuing a request for proposals. The request for proposals informs the prospective contractors of the government’s requirements, the anticipated terms and conditions that will apply to the contract, the information required in a proposal, and the factors used to evaluate proposals and their relative importance. Those who wish to respond must submit their proposal to the contracting office in the time and manner stated in the request for proposals. We consider the solicitation phase to begin with solicitation issuance and end at the deadline to submit the initial proposals, although the solicitation can be amended later and proposals revised. Initial Evaluation: Proposal evaluation is an assessment of the proposals based on stated evaluation factors and the offerors’ ability to perform the prospective work successfully. For example, proposals undergo technical evaluation to determine offerors’ ability to meet the technical requirements and cost or price evaluation to determine whether the price is fair and reasonable. We consider the initial evaluation phase to begin when potential offerors submit initial proposals and end once government contracting personnel receive approval to enter into negotiations or discussions. Discussion/Negotiation: Negotiations are exchanges, in either a competitive or non-competitive environment, between the government and offerors that are undertaken with the intent of allowing the offerors to revise proposals and obtaining the best value for the government. Negotiations allow, among other things, the offerors to address any government concerns with the proposals. We consider this phase to begin when the contracting officer receives approval to enter into negotiation and end when contracting personnel receive approval to award the contract. Contract Award: We consider the contract award phase to begin when the approval to award the contract is given and to end when the contracting officer signs the contract. In utilities privatization, as a part of the contract award phase, the Secretary of the military department makes a decision to convey the utility systems after the awardee has been selected. While the utilities privatization process must comply with relevant statutes and regulations, it has certain unique attributes. According to DLA Energy and military department officials, installations must conduct a thorough inventory of the physical assets associated with the utility system (e.g., linear feet of water pipes and location, number and location of gas valves, and the number and location of lift station pumps) as well as the system’s workload data to inform the requirements document. This is due to the fact that ownership of these physical assets will convey— i.e., be legally transferred—to the contractor after contract award. Conveyance from the military installation to a regulated public sector utility, such as a municipal water and wastewater authority, requires additional approval from the state’s utility regulatory commission. Finally, privatized utility services contracts are generally long-term, up to 50 years in some cases. According to DLA Energy and military department officials, these factors affect the consideration of requirements and structure of the utilities privatization process in a way not normally found in standard contracts and can affect the time required for discussions and negotiations. The use of lessons learned is a principal component of an organizational culture committed to continuous improvement. Through lessons learned, DOD can continuously look for ways to make improvements to the utilities privatization program to shorten the time to award and enhance effectiveness and efficiency. Collecting and sharing lessons learned serve to communicate knowledge more effectively and to ensure that beneficial information is factored into planning, work processes, and activities. This process also provide a powerful method of sharing ideas for improving work processes, facility or equipment design and operation, quality, and cost-effectiveness. Leading practices of a lessons learned process identified by GAO and others include collecting, analyzing, validating, saving or archiving, and disseminating and sharing information and knowledge gained on positive and negative experiences. Figure 4 shows this process. Since 2005, we have issued four reports that assessed various aspects of DOD’s utilities privatization efforts: In May 2005, we identified several management weaknesses in DOD’s implementation of the utilities privatization program. For example, we identified a number of concerns, such as the reliability of the economic analyses associated with privatization decisions and the adequacy of contract oversight. We made eight recommendations to help ensure the reliability of economic analyses and improve the utilities privatization guidance and procedures, among other things. DOD non-concurred with seven recommendations and partially concurred with one recommendation in its response to the report; however, DOD has since implemented all but one recommendation. In September 2006, we reported that DOD’s progress in implementing the utilities privatization program had been slower than expected and management concerns remained. For example, the targeted time frame for program implementation was delayed by 6 years and concerns remained about the reliability of economic analyses used to support privatization decisions. We made seven recommendations to improve DOD’s management of utilities privatization. DOD generally concurred with and implemented six of these recommendations. In July 2015, we identified that DOD faces challenges in implementing utility resilience efforts, such as collecting and reporting comprehensive utility disruption data, and developing cybersecurity policies for its industrial control systems. We made four recommendations to clarify utility disruption reporting guidance, improve data validation steps, and address challenges to cybersecurity industrial control systems. DOD concurred or partially concurred with all but one recommendation and implemented three recommendations. In September 2018, we reported that DOD lacked guidance to develop performance metrics and implement cybersecurity requirements for privatized utility services contracts. We made two recommendations to provide guidance for development of metrics to track utilities privatization contract performance, and what constitutes covered defense information as it related to utility services contracts. DOD concurred with and implemented both recommendations. Concerns about the length of time to award contracts are not limited to utilities privatization. For example, in July 2018, we reported that although DOD proposed reducing the time it takes to award weapon systems contracts, the department has limited understanding of how long it currently took and therefore lacked a baseline to measure success. We also found that, according to contracting officials, factors such as the quality of proposals, prospective offeror responsiveness to agency request for additional information, and complexity of the technical requirements can add or reduce the time required for evaluation of proposals. We recommended that, to assess time frames for awarding contracts, DOD should develop a strategy to determine what information it should collect and how to use that information. DOD concurred and implemented the recommendation. The time to complete the utilities privatization pre-award process generally took an average of 4 years from issuing the solicitation to awarding a contract for utility services for the contracts we assessed. Utilities privatization officials acknowledged that the process is lengthy, but DOD does not maintain complete data on key steps in the process, including when the process to consider privatization of a utility system began and the time needed to conduct acquisition planning. Consequently, it is not possible to determine the entire time to complete privatization of a utility system. In addition, the time to complete a specific utilities privatization effort may be affected by a number of factors. These factors can include changes to internal or external requirements, the technical complexity of the individual effort, the continuity of personnel involved in the effort, and command support for privatization. The 21 new contracts for privatized utility services awarded from fiscal years 2016 through 2018 generally took an average of 4 years from the time the DOD component issued a solicitation to when the contract was awarded. Utilities privatization officials acknowledged that the process is lengthy. They stated that it is due, in part, to the long-term nature of the contracts—that can be up to 50 years—and the complexity of the contracts. The entire pre-award contracting process could be longer, as we found that, with the exception of the one Navy-awarded contract we reviewed, DOD does not maintain complete data for every phase of the process. The data DOD does not maintain includes key events in the acquisition planning phase, specifically, when the military departments began considering privatizing a specific utility and when the requirements packages—a complete inventory of the associated infrastructure, such as pipes, wires, and valves—were available to use in the solicitation. Table 3 presents the available information on the average time to complete the five phases of the pre-award contracting process identified by GAO for the contracts we assessed. As indicated in table 3, even after excluding the time needed to conduct acquisition planning, there is wide variation in the average time taken from when contracting officials issued the solicitation to when they awarded the privatized utility services contracts. For example, NAVFAC took more than 92 months—or more than 7 years—to award its contract to privatize the Naval Air Station Key West wastewater system. The total time required to award the contract included a 30-month period during which the privatization effort was paused to evaluate alternative paths to meet new Florida wastewater regulations. Navy officials stated that our timeline should not include the 30-month period because the pause did allow any additional work to be accomplished to prepare for contract award. After determining that privatizing the utility system remained the most effective approach, however, the Navy resumed evaluating revised proposals that had been received in response to the amended original solicitation. DLA Energy took, on average, about 45 months—or nearly 4 years—to privatize utility systems and make awards for the 19 contracts it was responsible for. In contrast, the Air National Guard awarded a non- competitive contract to privatize the wastewater system at Truax Field in Wisconsin to a local utility provider in about 6 months. However, according to Air National Guard officials, the local utility provider already maintained the infrastructure for the installation and had previously conducted an assessment of the installation wastewater system used to finalize the privatization requirements. While Air National Guard officials could not provide a date as to when they began to consider utilities privatization, they stated that they spent more than 70 months in acquisition planning before issuing the solicitation due, in part, to unfamiliarity with the utilities privatization process. While no provision of the regulations or policies governing utilities privatization that we reviewed require DOD contracting activities to collect data on the time to complete each phase of the pre-award process, since 2014, DLA Energy officials have attempted to maintain such data for all the contracts for which they were the contracting agent. However, DLA Energy did not maintain data for the completion of milestones within the acquisition planning phase carried out by the military departments. In 2014, DLA Energy officials, with input from Army and Air Force utilities privatization officials, established milestones to plan and monitor key pre- award contracting activities, including a target time to complete each milestone. DLA Energy and military department officials noted that these data help provide insight into, and accountability for, the progress made or challenges encountered during the pre-award process. However, the usefulness of these data are limited because the military departments must provide time frames for the front end of the process and have not done so. We found that a number of factors can affect the time to complete pre-award contracting activities, but for the purpose of establishing milestones to monitor these activities, DLA Energy varied the number of milestones and time frames to complete specific activities depending on whether the contract was competitively awarded and the number of proposals received. While table 3 shows the average time it took to complete pre-award phases by contracting agent for the contracts in our audit scope, figure 5 shows the milestones used by DLA Energy for competitive and non-competitive awards and the target time frames DLA Energy established for each milestone. DLA Energy officials noted that the time frames for the first three steps in the process—determining that one or more utility systems on an installation should be considered for privatization through when the military department provides DLA Energy a complete requirements package—are determined by the military departments. The military departments have not established target time frames for these activities, but have taken steps to understand factors that affect the time frames, which we discuss later in this report. DLA Energy and military department officials noted that despite the limitations in the data, this information has helped them provide better management oversight of the process. A DLA Energy official stressed that the target time frames are meant to improve contract award time frames, incentivize performance, and provide accountability, and that they do not expect every contract to meet targets due to the complex nature of utilities privatization. Nevertheless, DLA Energy and military department officials stated that implementation of the milestones helped reduce the amount of time needed to award privatized utility services contracts. Our analysis of the 18 competitive utility services contracts awarded by DLA Energy from fiscal years 2016 through 2018 indicates that the average time from receipt of requirements to contract award has decreased. For example, our analysis indicated that DLA Energy took an average of nearly 61 months from receipt of requirements to competitively award eight contracts related to solicitations issued prior to 2014. Once the milestone tracking process was initiated in 2014, our analysis indicates that DLA Energy took an average of about 35 months from receipt of requirements to competitively award 10 privatized utility services contracts (see fig. 6). Of the 10 privatized utility services contracts DLA Energy awarded since the process was initiated in 2014, two met or exceeded DLA Energy’s target time frame; six were awarded within 6 months of the target time frame; and two were awarded over a year longer than the target time frame. DLA Energy and military department officials identified several factors that, individually or collectively, could affect the time to award a privatized utility services contract. These factors include the extent to which internal or external requirements remain stable, the technical complexity of the privatization efforts, the continuity of personnel involved in the effort, and command support for privatization. Changes to Internal or External Requirements. According to a utilities privatization official we interviewed, unexpected changes to requirements may affect the time to award a utility services contract. For example, Navy officials stated that Naval Air Station Key West initiated efforts to privatize its wastewater treatment facilities in August 2007 to meet new Florida wastewater regulations by the compliance deadline of July 2010. A Navy installation official stated that they determined that they would be challenged to meet the new regulations with existing facilities and could not upgrade those facilities to meet the new standards due to inadequate personnel and funding. After the Navy issued the original wastewater solicitation in March 2008, the state extended the required compliance date to December 2015. As a result, installation and Navy officials reconsidered utilities privatization and assessed whether the extended deadline would allow them to reach compliance without privatization. After evaluating alternative paths of action to ensure compliance with the new Florida wastewater regulations, installation and Navy officials determined it remained in the best interest of the installation to proceed with the solicitation and contract award. Wright-Patterson Air Force Base officials stated that after they issued their solicitation for privatization of water systems, installation officials discovered that two of the wells were contaminated by firefighting foam. This foam, used to extinguish aircraft fires, contained chemicals which washed off runways and seeped into the groundwater. According to installation officials, concentrations of these chemicals exceeded non-regulatory lifetime health advisory levels, prompting the installation to remove the chemicals before distributing the water for use on base. To address the problem immediately, installation officials reported that they engaged with the Air Force Civil Engineer Center to fund a project to modify the existing water treatment plant to remove the chemicals before distributing the water for use; they could not wait for the award of the utility services contract. Installation officials stated the modification to the water treatment plant included the construction of a building to house a large filtration system to remove the contaminant. These officials also stated the modified water treatment plant was then included in the requirements package for the utilities privatization effort. DLA Energy officials stated that a change to the requirements included in utilities privatization efforts was a frequent occurrence. As this information is central to determining the technical requirements and the cost estimate, changes to an inventory can affect the length of time spent in acquisition planning and in discussions and negotiations. Army officials stated that completion of the list of inventory to be privatized is a time-consuming process as records of the amount of pipes, valves, wires, facilities, or other items is often incomplete. Army and Air Force officials indicated that often multiple surveys of inventory are conducted by both the military departments and the contractor selected to maintain the utility system to finalize requirements. Officials reported that this is because, in part, a final and complete inventory is required so that after award the government can finalize the bill of sale and convey those systems to the utility services contract(s) provider(s). Technical Complexity. According to utilities privatization officials, the technical complexity of the utilities privatization effort can also affect how long it takes to award a utility services contract. At Fort Riley, DLA Energy officials, installation officials, and contractor representatives shared with us that the complexity of the regulated environment of some utilities had an effect on the time to award. For example, a contractor representative stated his regulated utility company was one of the potential contractors vying for a utility services contract at Fort Riley, and this required additional approval from the state’s utility regulatory commission. DLA Energy officials stated and the contractor representative stated that this additional complexity led to a prolonged negotiation and discussion effort as his company sought additional information about the asset inventory to create what it perceived to be a quality proposal for both the utility services contract and its utility regulatory commission. Our analysis of DLA Energy data found that for the majority of the contracts, the discussion/negotiation phase required the longest amount of time during the pre-award contracting process. The time to award for this utility system took one year longer than other utility systems privatized on the same installation. Continuity of Personnel. Utilities privatization officials we interviewed stated that the continuity of personnel involved in the process is critical to awarding a contract in a timely manner. For example, at Naval Air Station Key West, officials told us that staff turnover was prevalent at all levels multiple times during the utilities privatization process. These officials noted this turnover was due, in part, to the isolated location of the installation, which made it difficult to recruit and retain both civilian and military staff. They also noted that in turn, this turnover of staff led to loss of knowledge and dispersion of data. During our visit, we observed that installation staff had difficulty locating documentation and had limited knowledge of what occurred during the pre-award contracting process at the installation. Officials explained that this was due, in part, to the loss of some documentation due to flooding and the management of the process by other commands. In contrast, installation and contracting officials at Fort Riley stated that there was no turnover in the installation staff during the pre-award contracting process and no turnover in the DLA Energy contracting staff once they took responsibility for administering the utility services contracts. Officials at Fort Riley stated that with continuity of staff, knowledge and working relationships were built and maintained. DLA Energy awarded utility services contracts for three utility systems in a comparatively quick time frame compared to the other contracts in our analysis. Command Support for Privatization. Utilities privatization officials stated that the support of the installation’s command leadership can facilitate award of a utility services contract. For example, officials at Fort Riley said the installation commander and director of public works department fully supported utilities privatization as the solution to the installation’s failing utility systems. These officials noted that due to this desire to privatize utility systems, the senior installation leadership openly communicated its goals and support of privatization throughout the pre-award contracting process. For example, Fort Riley officials stated that public works department staff assigned to work on the utilities privatization effort were sequestered or removed from all other assigned responsibilities. Installation officials stated this allowed the employees to focus on the utilities privatization tasks. According to Fort Riley officials, this leadership support was a factor in reducing the time to contract award. According to our analysis, the utility services contracts for Fort Riley were awarded more quickly than the majority of the utility services contracts we assessed at other installations. In contrast, officials at Wright-Patterson Air Force Base said their leadership was reluctant to fully support utilities privatization. While senior military department leadership directed the installation to privatize its utility systems, installation leadership was reluctant to do so due to concerns of job loss for public works department employees and perceived loss of flexibility in installation operations and maintenance funding. Installation officials stated that this reluctance was one factor in the amount of time it took to make the contract awards. According to our analysis, the contracting award process for the three utility systems at that location took longer than the majority of the utility services contracts we assessed at other installations. DOD is generally applying leading practices in its efforts to improve the timeliness of the utilities privatization pre-award contracting process, but is missing opportunities to analyze the effects of its changes and to better share the information with stakeholders. ASD(Sustainment), the military departments, and DLA Energy have taken, or plan to take, actions that demonstrate or partially demonstrate four of the five leading practices identified by GAO and others. However, despite the breadth of activities performed and planned by DOD, the department lacks key data it needs for further analysis and validation of the pre-award contracting process as well as a repository for archiving lessons learned for future stakeholders to access. ASD(Sustainment), the military departments, and DLA Energy have taken actions to implement lessons learned, in part, to reduce the time it takes to award contracts. We assessed whether these actions demonstrated, partially demonstrated, or do not demonstrate each of the five leading practices for implementation of lessons learned identified by GAO and others. Demonstration of these leading practices is critical to ensuring that lessons learned endure and that processes are improved. In reviewing ASD(Sustainment), military department, and DLA Energy pre- award documentation and interviewing knowledgeable officials, we found that all the DOD entities fully demonstrated the third leading practice— which is to validate the applicability of lessons—and demonstrated three other leading practices to varying degrees. None of these entities, however, demonstrated the store and archive leading practice (see table 4). Collect information. The collect information leading practice involves capturing information about events in the area of interest, which can be achieved through various methods. ASD(Sustainment), the military departments, and DLA Energy officials told us that they collect information for utilities privatization lessons learned through activities such as data calls, working groups, workshops, studies, conferences, and meetings. Specific examples of DOD demonstrating this leading practice are as follows: According to an ASD(Sustainment) official, since about 1997, their office has sponsored a monthly Utilities Privatization Working Group attended by representatives of the military departments and DLA Energy officials. The purpose of the working group is to provide a collaborative forum to adjudicate issues and share lessons learned from utilities privatization activities. For example, topics of discussion on the April 2019 agenda included issues or challenges associated with developing an execution framework, methodologies to implement utilities privatization guidance, and updates on current utilities privatization activities from the military departments and DLA Energy. In 2019, ASD(Sustainment) added a requirement to its guidance for an annual utilities privatization program review with each military department to address portfolio lessons learned. According to ASD(Sustainment) officials, their office, the military departments, and DLA Energy officials plan to work together to develop a strategy for complying with the guidance. According to an Army official, the Army established a tri-military department annual Utilities Privatization Post-award Workshop in 2014 that discusses post-award issues among the military departments, DLA Energy, and contractors. Each military department has hosted a workshop. For example, the Navy hosted the November 2018 post-award workshop, which included updates by the military departments and DLA Energy on their utilities privatization activities, including some lessons learned. The Navy commissioned a study in 2016 to help reestablish its utilities privatization program and reduce life-cycle expenditures on infrastructure, including utility systems. The study examined the costs, benefits, and existing policies for private versus government facilities ownership and recommended changes to the Navy’s processes for utilities privatization. The study led to, among other things, the creation of Navy-specific utilities privatization guidance. According to a Navy official, they will establish a community of practice in partnership with DLA Energy to provide a quarterly forum for NAVFAC officials to share lessons learned and discuss utilities privatization problems and solutions. DLA Energy hosts the biannual DLA Energy Worldwide Energy Conference to provide personnel of the military departments, DLA Energy, and contractors the opportunity to learn from each other and top industry experts on the latest trends and initiatives in energy, including utilities privatization. DLA Energy also participates in the annual Department of Energy Annual Energy Exchange Conference. For example, in 2019 it participated in a discussion panel on utilities privatization. These efforts to collect information and lessons learned are positive; however, as discussed earlier in this report, DOD lacks complete and consistent information on the time to award utility services contracts. Reducing the amount of time to award these contracts is a stated goal of DOD. ASD(Sustainment) issues annual data calls to the military departments to collect information such as the number of utility systems privatized by military department, the authority under which a privatization took place, and award dates for the utility services contracts. DLA Energy and military department officials indicated that collecting this information has contributed to efforts to reduce the amount of time needed to award utility services contracts. They acknowledge, however, that the military departments do not collect information on the formal decision to consider privatization of utility systems and the length of time to conduct key acquisition planning activities, such as developing a complete inventory of physical assets to document its requirements. The requirements package is a key component in the pre-award contracting process and includes an inventory of the utility system. This inventory includes items such as the pipes, valves, and wires that make up the utility system. Consequently, neither DLA Energy nor the military departments, with the exception of the Navy, had reliable information on the entire time it took to complete the pre-award contracting process. Without data on the key tasks that need to be completed during the pre-award contracting phase, DOD is missing an opportunity to assess the extent to which updated guidance, training, and other ongoing efforts are having an effect on the time to award utility services contracts. In recognition of this, an Air Force official stated that the Air Force Civil Engineer Center recently implemented a schedule-tracking mechanism to capture these dates, which will be used with all new utilities privatization efforts. Collecting this information consistently across all military departments would allow for a more thorough analysis of contracting process information and could support future process improvement efforts. Analyze information. The next leading practice is to analyze the information collected to determine root causes and identify appropriate actions. Examples of DOD demonstrating the information analysis leading practice include: According to DLA Energy officials, in 2014, they reviewed and analyzed historical data from utility services contracts to revise the utilities privatization procurement time frame. As mentioned previously, this analysis led to the development of milestones and associated time-based targets to achieve each milestone, based on the number of proposals received, to reduce the pre-award contracting process. According to a DLA Energy official, the agency coordinated with its contractor support, and Army and Air Force program management offices to establish the time-based targets. In May 2014, the Air Force conducted a utilities privatization process improvement review with DLA Energy, among others, to streamline the utilities privatization process. The review allowed the Air Force to reduce the planned timelines for the utilities privatization pre-award contracting process, which DLA Energy administers on behalf of the Air Force, between the issuance of competitive solicitations to award by 14 months to 33 months. Similarly, in October 2014, the Army conducted a utilities privatization process improvement review with DLA Energy with a goal to reduce the time needed from issuance of a competitive solicitation to award of utility services contracts to less than 36 months. Army and DLA Energy officials identified opportunities for process or program improvement during the review. Overall, adopted changes reduced the planned timelines for the utilities privatization pre-award contracting process by approximately 5 months to 31 months. An Air Force official stated that lessons learned are collaboratively shared annually and have revealed lessons learned to improve the contracting process. This has led to updates of Air Force request for proposals template. The 2016 Navy study not only collected data on utilities privatization but also provided analysis to understand the opportunities, costs, and benefits associated with privatization. The Navy used the study to enable decisions about whether privatization is the appropriate strategy to reduce life-cycle expenditures on utility infrastructure. The analysis performed for the study resulted in multiple products and findings. For example, the Navy created an Excel-based tool to consolidate utility data, organize data, and prioritize installations for evaluation of the potential to privatize. ASD(Sustainment) revises its utilities privatization guidance and procedures based on lessons learned and changes in laws and regulations. For example, we found that DOD responded to industry feedback by standardizing and clarifying request for proposal templates used in utilities privatization. Based on our analysis, the military departments and DLA Energy fully demonstrated the leading practice for analyzing information, and ASD(Sustainment) partially demonstrated the leading practice. According to DOD, ASD(Sustainment) is responsible for overseeing progress tracking and goal setting for utilities privatization across the department. Therefore, analysis for performance across the department on the time it takes to award utility services contracts is its responsibility. As mentioned previously, while ASD(Sustainment) collects data on the number of utility systems privatized by military departments and award dates for the utility services contracts, it is missing information about key pre-award contracting activities. In the absence of this information, ASD(Sustainment) cannot fully analyze the department’s utilities privatization activities for further lessons learned to help reduce time frames for awarding contracts. Validate applicability of lessons. Once collection and analysis have identified the lessons learned, the next leading practice is to validate that the right lessons have been identified and determine the scope of their applicability. Subject matter experts or other stakeholders may be involved in this step of the process. Examples of DOD’s demonstration of the validation leading practice include: ASD(Sustainment) officials noted that they assess the applicability of lessons by periodically revisiting and revising utilities privatization guidance. These officials said they revised such guidance, for example, in 2002, 2005, 2010, and 2019 to incorporate lessons learned from stakeholders across the process. According to an Army official, the Army periodically assesses the applicability of lessons learned by revising its utilities privatization acquisition process based on the Army’s strategic direction, military department meetings, and utilities privatization policy changes. This included the utilities privatization acquisition process improvement review with DLA Energy to reduce the time needed to award utility services contracts. According to the 2016 Navy study, contractors conducted interviews to validate data, obtain supplementary data, and ascertain qualitative information. In addition, contractors interviewed Air Force and Army utilities privatization representatives to garner lessons learned and understand other DOD components’ approaches to utilities privatization. One result of the study was development of a repeatable methodology and framework, based on specific lessons learned, that can be used to evaluate candidate sites for utilities privatization. In addition, according to an official, the Navy is using the lessons learned from its study to develop its new utilities privatization handbook, a draft of which emphasizes the need throughout the process for the collection, documentation, and sharing of lessons learned to help future installations and refine the utilities privatization program. According to the Navy, it plans to update the handbook on an ongoing basis to reflect lessons learned from its pilot program with DLA Energy, and with the Army and Air Force utilities privatization programs. An Air Force official stated that over an 18-month period they assessed their utilities privatization process and developed a new, comprehensive utilities privatization process for pre-award contracting activities. The Air Force determined the scope of the applicability of lessons learned when it revised its draft utilities privatization playbook to incorporate this new process. DLA Energy revised how it monitors the utilities privatization process based on its analysis of historical utilities privatization data. DLA Energy officials said they validated these changes by testing the milestones and associated targets on a 2014 Army utility services contract and found them to be reasonable. Army and Air Force officials agreed with the assessment. DLA Energy officials stated that they also determined the scope of the applicability of lessons learned by determining to whom and what the lessons learned applied, and by taking actions to continually revisit and revise its templates and procedures. For example, we found that the fiscal years 2012 and 2016 versions of the request for proposals template reflected changes for both the Army and Air Force but we could not determine if they were the result of lessons learned. We also identified revisions DLA Energy made to incorporate lessons learned into operating manuals it uses for the utilities privatization process. For example, a DLA Energy official noted that the agency revised its risk evaluation manual to improve the quality of the risk evaluations the source selection evaluation board performs. Store and archive lessons. The archiving of lessons learned involves the use of a repository, used to disseminate and share information. As appropriate, these repositories should have the capability to store and share data and to secure classified, sensitive, or proprietary data. Archiving lessons learned should remain an ongoing process; otherwise, it risks becoming cumbersome and irrelevant. Our observations on DOD’s efforts to store and archive information on utilities privatization include: According to Air Force officials, the Air Force does not currently store or archive lessons learned for pre-award contracting activities. The Air Force Portfolio and Asset Control and Evaluation System stores and archives lessons learned for post-award contracting activities. The system is available to Air Force, DLA Energy, and other stakeholder agencies like the General Services Administration, but not to other military departments. To populate the database, the Air Force Civil Engineer Center portfolio management division uploads utilities privatization documents into the system, including weekly status reports, briefings, and meeting notes for post-award contracting activities. The system also records lessons learned and provides a social media discussion platform, known as the Contracting Officer’s Representative Toolbox. Our review of the system determined that it is not widely populated. Specifically, as of December 2019, the system contained seven lessons learned, three discussion postings, and five documents in the Toolbox. According to Air Force officials, however, this system was not intended to be a repository for storing and archiving lessons learned for pre-award contracting activities and acknowledged that the Air Force does not currently have another means to do so. According to DLA Energy officials, they do not maintain a specific repository for storing and archiving lessons learned for utilities privatization pre-award contracting activities but make their revised templates and procedures—that they believe generally reflect key lessons learned—available to stakeholders for utilities privatization on a website. According to a DLA Energy official, the website is open to anyone that can access DLA.mil, but most of the content is intended to assist contracting officer’s representatives in conducting their post- award contracting responsibilities. The Army has one key official who has managed its utilities privatization program activities for many years and has a substantial amount of experience and institutional knowledge. This official maintains utilities privatization files in hard copy—such as guidance, memorandums, and relevant studies—therefore this information is not readily available to all relevant stakeholders, such as the other military departments. According to the Army official, the Army does not maintain a repository for storing and archiving lessons learned for utilities privatization pre-award contracting activities. According to Navy officials, NAVFAC has a business management system that provides for the management of business processes, common practices, and process and quality improvement for NAVFAC products and services. The system’s documentation is available for use by all NAVFAC commands and links to applicable policies, guidance, forms, and information so that work will be conducted in a consistent manner. According to officials, this system is updated annually or at significant process changes. However, the Navy is currently developing a module for the business management system for utilities privatization with an estimated completion date of March 2020. Navy officials stated that the module is expected to include pre- award contracting lessons learned when it becomes operational. According to ASD(Sustainment) officials, they do not maintain a repository for storing and archiving lessons learned for utilities privatization pre-award contracting activities. While ASD(Sustainment), the military departments, and DLA Energy officials stated they incorporate lessons learned in various ways, including when they revise policies and/or operating manuals, these officials acknowledge that they do not maintain a repository for storing and archiving lessons learned on specific utilities privatization pre-award contracting efforts. DLA Energy officials, who support both the Army and Air Force utilities privatization efforts, stated that revisions to templates and guidance were sufficient to implement lessons learned. The leading practices for lessons learned indicate that the use of a repository to store lessons learned allows agencies to disseminate and share the lessons learned. Without such a capability, ASD(Sustainment), the military departments, and DLA Energy may be missing opportunities to capture and share lessons learned that could benefit future utilities privatization efforts, including helping DOD achieve its goal of reducing the length of time to contract award. Disseminate and share lessons. A critical step in any lessons learned process is the sharing and disseminating of the knowledge gained. Agencies can disseminate lessons in many ways, such as briefings, bulletins, reports, emails, websites, database entries, the revision of work processes or procedures, and training. Lessons can be “pushed,” or automatically delivered to a user, or “pulled,” where a user searches for them in an archive of lessons learned information. Examples of DOD demonstrating the disseminating and sharing leading practice include: As previously noted when discussing the collection criteria, the DOD officials we spoke with told us that they distribute lessons learned during annual reviews, industry conferences, regular meetings, workshops, training sessions, and working groups. The lack of documentation and archiving of the lessons learned, however, limits the ability of future users to search for and retrieve them. DLA Energy officials stated that they share templates created for the utilities privatization program with the military departments and industry. According to an ASD(Sustainment) official, these users find the information helpful and efficient as the templates can be customized where necessary depending on the type of potential contractor and solicitation and updated for lessons learned. Additionally, DLA Energy revises its standard operating procedures for the pre-award contracting process to incorporate lessons learned and disseminate changes. DLA Energy also provides training for the utilities privatization process, for example on the procedures contracting officials should use to conduct negotiations and past performance evaluations for utility services contracts. As previously discussed, the Air Force Portfolio and Asset Control and Evaluation System is a system used to store and disseminate lessons learned for the post-award utilities privatization process. While used in a limited fashion, the Contracting Officer’s Representative Toolbox consists of a newsfeed and a documents file. The documents section allows users to save and share helpful utilities privatization documents with others. The system also contains a resource center to maintain updated training tools and resources for project oversight including frequently asked questions, best practices, and resolutions to project issues. While the system has the ability to both “push” information to users and allows for users to “pull” data by acting as an archive for documentation, it is not available to Army and Navy utilities privatization staff and does not currently contain lessons learned on the pre-award contracting process. We assessed DLA Energy as fully demonstrating, and ASD(Sustainment) and the military departments as partially demonstrating, this lessons learned criteria. While ASD(Sustainment) and military department officials do disseminate and share lessons learned, the inability of future users to search for and retrieve lessons learned limits their utility. For example, Air National Guard officials stated that they were unfamiliar with the utilities privatization process and encountered delays prior to releasing the solicitation, in part, due to the need to obtain information about how to execute the process. Having the capability for others to retrieve archived lessons learned could potentially assist future stakeholders in the process and help further shorten contracting award time frames. DOD is taking steps to improve the effectiveness and efficiency of the utilities privatization pre-award contracting process and these efforts have contributed to decreasing the time needed to award utility services contracts. In particular, Army and Air Force officials consistently noted that DLA Energy’s establishment of a milestone-based system to track the time to complete key steps in the pre-award contracting process in 2014 has helped provide better management oversight and improve accountability. DOD, however, does not collect consistent information on the time to complete key phases needed to award utility services contracts. Specifically, DOD does not have information on when the military departments identify that one or more utility systems on an installation should be considered for privatization and when the installation delivers a completed requirements package as part of the acquisition planning phase. The lack of consistent data on these two key events may hinder DOD’s efforts to identify additional opportunities to reduce the length of time needed to award utility services contracts. Similarly, DOD recognizes the importance of collecting and disseminating lessons learned for the utilities privatization program, but currently lacks a mechanism to archive lessons learned during the pre-award contracting phase. As DOD has identified 580 utility systems that still may be privatized, having such a capability for others to retrieve archived lessons learned could potentially assist future stakeholders in the process and help further shorten contracting time frames. We are making two recommendations to the Secretary of Defense: The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment collaborates with the military departments and the Defense Logistics Agency to collect consistent information on the time to complete key steps in the pre-award contracting process for privatizing utility services. (Recommendation 1) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment collaborates with the military departments and the Defense Logistics Agency to develop a mechanism to store and archive lessons learned regarding the pre-award contracting process for privatizing utility services. (Recommendation 2) We provided a draft of this report to DOD for review and comment. DOD’s written comments are reproduced in appendix II. DOD partially concurred with both recommendations and provided technical comments, which we incorporated as appropriate. DOD partially concurred with our first recommendation to collect consistent information on the time to complete key steps in the pre-award contracting process for privatizing utility services. DOD suggested that we modify our recommendation to include other DOD contracting activities that may support privatization efforts. Our recommendation, based on the scope of our audit work, was intended to cover recent privatized utility services contracting activities within the military departments, such as Naval Facilities Engineering Command. But we agree that DOD should include any activity that provides support for utilities privatization in its efforts to collect better data. Similarly, DOD partially concurred with our second recommendation to develop a mechanism to store and archive lessons learned regarding the pre-award contracting process for privatization of utility services. DOD suggested that we modify our recommendation to include other DOD contracting activities besides DLA and to recommend that DOD add the lessons learned from the post-award contract process, as post-award contract actions play a critical role in informing pre-award contracting processes. As noted above, we agree that DOD should include any contracting activities that support pre-award utilities privatization efforts. Similarly, while our work did not specifically assess how post-award activities could be incorporated into the lessons learned efforts, we agree that doing so may provide additional insights that would benefit future utilities privatization efforts. In its technical comments, DOD disagreed with our presentation of the time required by the Navy to privatize utilities at Naval Air Station Key West. Specifically, DOD officials believed that we should exclude from our calculations a 30-month period that occurred during the solicitation phase in which it evaluated alternative paths to comply with new Florida wastewater regulations. DOD noted that this pause did not allow any additional work to be accomplished towards contract award. We had identified this pause and the rationale underlying the Navy’s decision to do so in the draft report. We continue to believe it is appropriate to reflect this period in our calculations as the Navy did not cancel the original solicitation and, after deciding to continue to pursue the privatization efforts, evaluated the offerors’ proposals that had been previously received prior to the pause and subsequently awarded the utility services contract based on that solicitation. In that regard, we consider the change in the date by which the Navy had to comply with Florida’s wastewater regulations to be a relevant example of one of the many external factors that can affect the time needed to privatize utilities at a military installation. We did, however, reflect DOD’s disagreement with our characterization where appropriate in the report. We are sending copies of this report to the Secretary of Defense; the Under Secretary of Defense for Acquisition and Sustainment; the Assistant Secretary of Defense for Sustainment; the Secretaries of the Army, Navy, and Air Force; the Director, Defense Logistics Agency; appropriate congressional committees; and other interested parties. This report will also be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by e-mail at dinapolit@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. A House Armed Services Committee report accompanying the National Defense Authorization Act for Fiscal Year 2019 included a provision that we review the Department of Defense’s (DOD) utilities privatization pre- award contracting process (which includes awarding a contract), including lessons learned to improve the process. This report examines (1) the length of time to award contracts for utility services and factors that affect it, and (2) the extent to which DOD is demonstrating leading practices to collect and disseminate lessons learned for utilities privatization. To determine the length of time to complete the pre-award contracting process to award a privatized utility services contract, we focused on utility services contracts awarded from fiscal years 2016 through 2018, the latest full year of available data when we began our audit. The time frame captured at least one privatized utility services contract for each military department—Army, Air Force, and Navy. To identify these contracts, we used information maintained by the Office of the Deputy Assistant Secretary of Defense for Energy, which is part of the Office of the Assistant Secretary of Defense for Sustainment (ASD(Sustainment)) on its utilities privatization master list, which includes such information as installation name and location, and when the contract was awarded. For fiscal years 2016 through 2018, a comparison of the ASD(Sustainment) information and solicitation details provided by the awarding contracting agents identified 28 utility systems at 15 military installations that were privatized during our time frame through 21 contracts. Nineteen of the 21 contracts were awarded using competitive procedures and the remaining two were awarded without providing for full and open competition, which we refer to as non-competitive. Of the 19 competitive awards, 18 were awarded by the Defense Logistics Agency Energy (DLA Energy), which served as the contracting agent for the majority of the Army and Air Force utility services contracts during our review period; and one was awarded by the Naval Facilities Engineering Command, the contracting agent for the Navy and Marine Corps. The two non-competitive contracts were awarded by DLA Energy and the Air National Guard, respectively. For the purposes of our review, we define the pre-award contracting process as the date from when a military department begins to consider privatizing an installation’s utility system(s) to the contract award date. For all 21 privatized utility services contracts awarded between fiscal years 2016 through 2018, we obtained copies of the award documents. In addition, for the 19 competitively awarded utility services contracts, we conducted contract file reviews to record completion dates of pre-award contracting phases and number of proposals received by utility. For the 18 DLA Energy competitive awards, we conducted two on-site reviews of the contract files at a DLA Energy facility to verify the dates and proposals. At the final DLA Energy contract file review, one analyst located and recorded each relevant document and date confirming completion of the pre-award contracting phase, as well as the offer information. A second analyst verified the accuracy of the information. After correcting certain errors, such as incorrectly recorded dates, we determined that this information was sufficiently reliable for purposes of reporting on the length of time to conduct pre-award contracting activities. The Navy provided electronic documents for us to review for its one competitive award. A similar verification process was conducted for the Navy information. For the Air National Guard contract, we obtained the contract and additional information from contracting officials on the time to complete the pre-award contracting process. To supplement this data, we interviewed Air National Guard contracting officials involved in the contract. To compare information on the factors that affected the length of time to award utility services contracts, we: Analyzed dates of comparable events throughout the pre-award contracting process found in the utilities privatization award contract files; and Conducted site visits to speak with DLA Energy, installation, and military department officials, and contractor representatives about their experiences with the utilities privatization pre-award contracting process. We combined a record of all utilities privatization pre-award contracting information into one file. We used this file to compare time to award for all pre-award contracting activities. This data was further compared by competitive and non-competitive status, contracting agent, military department, type of utilities in the proposals, number of utilities in the proposals, and the size of the installation by acreage. The size of each installation was found in the DOD Base Structure Report – Fiscal Year 2017 Baseline, A Summary of the Real Property Inventory. However, due to the small number of contracts we assessed, we were unable to determine if there were any patterns to corroborate whether the factors such as the type of utilities in the proposals, number of utilities in the proposals, and the size of the installation did or did not affect time to contract award. To determine if there was a change in time to award after the implementation of DLA Energy’s 2014 time frames for pre-award contracting activities, we compared the time elapsed between receipt of requirements and award for competitively awarded privatized utility services contracts. The analysis does not reflect seven solicitations issued between fiscal years 2013 and 2018 as they were awarded, or were expected to be awarded after September 30, 2018, and are outside our audit scope. Using data obtained from DLA Energy, however, we determined that the awards or projected awards of these seven contracts generally follow the trend shown in our analysis. To gather information on the factors that affected the time from the decision to enter the utilities privatization process to contract award, we selected a non-generalizable sample of three installations to visit. The installations were selected from a list supplied by the Office of the Deputy Assistant Secretary of Defense for Energy of installations privatized from fiscal years 2016 through 2018. To obtain a variety of utilities privatization characteristics, we selected the installation visits based on the following criteria: (1) representation of each military department; (2) types of utility system privatized (electric, natural gas, water, and wastewater); (3) geographic location of installation; and (4) fiscal year of award. The contract awards for all installations visited were awarded using competitive procedures. Prior to our visit to the three installations, we analyzed contract file documentation and spoke with utilities privatization individuals at military department and DLA Energy headquarters. At the three installations, we conducted interviews with officials at the installation command, public works department, and contracting officials, and contractor representatives to obtain perspectives on their utilities privatization in general and specifically on the factors that affected the time to contract award. We conducted our non-generalizable site visits from May 2019 to July 2019 at (1) Naval Air Station Key West, Florida, (2) Wright-Patterson Air Force Base, Ohio, and (3) Fort Riley, Kansas. No Marine Corps installations were privatized from fiscal years 2016 through 2018. In addition, we spoke with contracting officials from the Air National Guard at Truax Field in Wisconsin. The results of this selection are not generalizable to all utility services contracts or military installations, but provide insights and illustrative examples regarding factors that affect timing in the contract award process used to privatize utility systems. To determine the extent to which DOD demonstrated leading practices identified by GAO and others for collecting and disseminating lessons learned, we compared DOD’s activities related to lessons learned against the five leading practices identified in our prior work to determine whether DOD demonstrated actions consistent with these practices. We then had a second analyst check the same documents and activities to verify our initial results. These leading practices for lessons learned are discussed in a September 2012 GAO report, Federal Real Property Security: Interagency Security Committee Should Implement a Lessons-Learned Process; and a December 2018 GAO report, Project Management: DOE and NNSA Should Improve Their Lessons-Learned Process for Capital Asset Projects. We compared DOD’s lessons learned documentation, including the Air Force’s lessons learned database, DLA Energy’s utilities privatization website and operating manuals, the military departments and DLA Energy contracting process policies and procedures, and Air Force and Navy utilities privatization handbooks against these practices. Based on our analysis, we assessed whether DOD fully (met all of the criteria), partially (met part of the criteria), or did not (met none of the criteria) demonstrate the leading practices. To determine the actions that DOD has taken to learn lessons from the utilities privatization pre-award contracting process and demonstrate leading practices, we interviewed officials from ASD(Sustainment), the military departments, and DLA Energy; obtained and analyzed documents; and attended the 2019 DLA Energy Worldwide Energy Conference to gain a greater understanding of utilities privatization. We conducted this performance audit from March 2019 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Timothy J. DiNapoli, (202) 512-4841 or dinapolit@gao.gov In addition to the contact named above, J. Kristopher Keener (Assistant Director), Joe E. Hunter (Analyst-in-Charge), Stephanie Gustafson, Gina Hoover, Tamera Lockley, Roxanna T. Sun, Anne Louise Taylor, Kari Terrio, and Kristy Williams made key contributions to this report.", "summary": "Since 1988, military departments have privatized utility systems—such as electricity, water, natural gas, and wastewater—on military installations. DOD awards privatized utility services contracts to companies who upgrade, maintain, and operate the systems. Members of Congress and stakeholders have expressed concerns over the length of time it takes to award these contracts. DOD has a goal of reducing the time frames. A House committee asked GAO to review DOD's utilities privatization. This report examines (1) the length of time to award contracts for privatized utility services, and (2) the extent to which DOD is demonstrating leading practices to collect and disseminate lessons learned. GAO reviewed data on all 21 new utility services contracts awarded from fiscal years 2016 through 2018; compared DOD's lessons learned activities with GAO's leading practices; and interviewed DOD and utility company officials. From fiscal years 2016 through 2018, Department of Defense (DOD) components awarded 21 new contracts for privatized utility services on military installations. The contracting process generally took an average of 4 years from solicitation to contract award. However, the entire pre-award contracting process could be longer, as GAO found that DOD does not maintain complete data on the time to conduct key steps in the acquisition planning phase (see table). GAO found that DOD does not maintain data on when military departments begin to consider privatization and when a complete inventory of the associated infrastructure, such as pipes and valves, is available to use in the solicitation. While no DOD regulation or policy that GAO reviewed requires the collection of data on the time to complete all pre-award activities, in 2014, Defense Logistics Agency Energy officials established milestones to plan and monitor key pre-award activities. GAO found that the length of time from receipt of requirements to contract award was reduced from an average of 61 months pre-2014 to an average of 35 months post-2014. The lessons learned efforts of DOD to shorten the time to award contracts have fully or partially demonstrated four of five leading practices. DOD's efforts include: collecting information through working groups and conferences; analyzing past privatization efforts to focus management oversight; validating changes by demonstrating new processes; storing lessons learned through revised guidance; and sharing lessons learned through working groups and training. However, as DOD does not collect consistent information on the total time to award utility services contracts, DOD is missing opportunities to use lessons learned to reduce the time. Further, DOD does not have a repository for archiving specific lessons learned from utilities privatization efforts. Rather, DOD officials note they consider lessons learned as they develop updated guidance, templates, and handbooks. Without a repository of specific lessons learned, such as conducting the privatization process, DOD is missing opportunities to collect and share lessons learned to assist stakeholders on the remaining 580 utility systems it considers available for privatization. GAO recommends that (1) DOD and the military departments collect information on the time to complete key steps when awarding these contracts, and (2) DOD develop a mechanism to store and archive lessons learned from across the department. DOD partially concurred with both recommendations, noting that it would be beneficial to expand the actions GAO had recommended. GAO agrees that such an expansion would be helpful in efforts to collect more data.", "document_type": "gao"}
{"report": "Commercial airports in the United States (i.e., those regulated by TSA) can be generally described as having security-restricted areas and unsecured areas. Security-restricted areas include the area from which screened passengers may board aircraft, as well as areas where access is generally limited to appropriately vetted and credentialed personnel, such as airport and air carrier employees who require access to aircraft or to load and sort baggage. Unsecured areas, or airport public areas, are the areas of the airport that may be accessed by the general public without restrictions (i.e., without passing through security or some other controlled access point), such as ticketing areas, restaurants and shops, baggage claim areas, and areas extending outward from the airport facility to include pathways leading to an airport’s terminal and public parking areas. Airport public areas described by TSA officials, airport operators, airport law enforcement and aviation related associations we met with included airport access roads, curbside drop-off/pick-up areas, parking structures, rental car facilities, bus/transit lines leading to the airport, main entrances to and lobbies of terminal areas, and the security queue leading to security screening checkpoints. See figure 1 for a general illustration of the public areas of a commercial airport in the United States. According to aviation security stakeholders, such as airport operators, law enforcement officers, and industry representatives from trade associations, securing airport public areas presents inherent challenges for numerous reasons. They also stated that, in general, airports are designed to support the movement of large numbers of people through the airport’s public spaces to the security checkpoint and into the airport sterile area or to aircraft for boarding. TSA officials stated that given the large number of people that pass through airport public areas during peak hours, it can be difficult to monitor these areas for security threats. Additionally, the use of some enhanced security measures and reconfiguring terminals (e.g., metal detectors at entrance doors or the movement of security checkpoints closer to airport entrances or ticket counters) may create additional challenges because such measures could result in long lines and smaller congested spaces that would disrupt service. As a result, airport officials stated that any airport modernization projects, law enforcement actions or technologies introduced to potentially enhance security in the public areas of airports should not disrupt the efficiency of the airport’s business operations. In addition, airport officials and industry representatives stated that each airport is unique in its combination of layout and operations, which may determine the type of security approach and method the airport operator utilizes to enhance the security of the airport’s public areas. For example, an airport may have a number of separate terminals, each comprising separate entrances and public areas, creating a security challenge due to the vast area of detached public space. While another airport may have a single main terminal building that includes a hotel, restaurants and shops with two sections (A and B), each maintaining ticket counters and security checkpoints that may require a different security approach due to its unique set of challenges to securing their public areas, such as having large numbers of people congregating in one central location. TSA, the federal agency with primary responsibility for civil aviation security, implements security measures and imposes security requirements to ensure that access to those areas of the airport that could otherwise permit an individual with ill-intent access to cleared passengers and aircraft is controlled. Although TSA’s statutory authorities and responsibilities for the civil aviation system are not limited to this purpose, resource and other practical constraints, as well as TSA’s mission to secure civil aviation without unduly impeding the flow of commerce, have resulted in a regulatory structure largely focused on implementing measures that ensure the security of the aircraft and the traveling public. As a result, airport operators determine the boundaries for the security-restricted areas of their own airport based on the physical layout of the airport and in accordance with TSA requirements—generally documented through TSA-approved security programs. Securing airport public areas requires a collaborative approach involving airport operators, law enforcement, and TSA officials, among others. A number of aviation stakeholders play an important role in recommending enhancements that impact the security of airport public areas. The roles and responsibilities for each of these aviation stakeholders vary, but together provide a collective approach to securing airport public spaces. Airport operators. Airport operators are the owners, administrators and managers of an airport with responsibilities to plan, organize, supervise and direct airport operations, and have direct responsibility for implementing security requirements in accordance with their TSA- approved airport security programs. In accordance with its security program, airport operators must, in general, provide for the availability of law enforcement personnel in the number and manner adequate to support its security program for public areas and TSA screening operations at the airport. Although TSA’s primary responsibility is to implement and oversee aviation security, incident response at commercial airports is essentially the responsibility of the airport operator in conjunction with state or local law enforcement agencies and TSA collaboratively working to respond to any security incidents. Law enforcement. Responding to security incidents such as an active shooter situation or any other criminal matter—whether in an airport public area or within a security-restricted area—is generally the responsibility of law enforcement personnel present at or available for response to the airport, in accordance with an airport’s security program. For example, officials providing the requisite law enforcement support may be federal, state or local officers, or special airport-authority officers. While some airport law enforcement officers are stationed at dedicated posts at or near passenger screening checkpoints, officers also routinely patrol areas around the checkpoints, such as ticketing areas, restaurants and shops, and baggage claim, among others. TSA. TSA assumed primary responsibility for implementing and overseeing the security of the nation’s civil aviation system following the terrorist attacks of September 11, 2001. As previously stated, TSA primarily fulfills its mission through a regulatory structure largely focused on implementing and enforcing measures that ensure the security of the aircraft and traveling public—such as by controlling access to the security-restricted areas of the airport through the screening of passengers, accessible property, checked baggage, air cargo and mail, or ensuring that controlled access points for use by credentialed aviation workers are in place. However, TSA is also responsible for ensuring that airport operators and other aviation stakeholders remain compliant with their TSA-approved airport security programs and other applicable requirements, which it accomplishes by conducting inspections of, for example, an airport operator’s perimeter, access control, and other security measures. As circumstances warrant, TSA also issues information circulars to notify regulated entities of security concerns and security directives to augment or supplement requirements implemented through security programs. Security directives and guidance issued by TSA related to airport public areas have covered such topics as law enforcement requirements to patrol public areas, law enforcement response times and improved communications systems, among others. Key TSA roles at airports include: Federal Security Directors (FSD). The ranking TSA authority at airports, the FSD, provides leadership and coordination of TSAs day- to-day security activities, including ensuring airport operator’s compliance with their airport security program. Assistant Federal Security Directors for Inspections (AFSD-I). Each AFSD-I manages a Compliance Hub staffed by Transportation Security Inspectors who ensure regulatory compliance, respond to incidents, and reduce vulnerabilities in collaboration with regulated and non-regulated entities. The area of responsibility of the Compliance Hub may cover one or more FSD areas. Visible Intermodal Prevention and Response (VIPR) Teams. TSA Law Enforcement Federal Air Marshal Service (LE/FAMS) conducts protection, response, detection and assessment activities in airports and other transportation systems. VIPR teams comprised of TSA and law enforcement, security inspectors, and screening personnel perform various functions that include randomly screening workers, property, and vehicles, as well as patrolling the public areas of airports. According to TSA officials, there are approximately 31 VIPR teams nation-wide providing enhancement to security in airports. Decisions on deployments of VIPR teams are determined by risk associated with the venue, which is either surface transportation venues like passenger rail or bus stations among others, or airports. TSOs. Although TSOs are uniformed security personnel that resemble law enforcement, they are not law enforcement officers. Therefore, TSA relies on the presence of law enforcement at the passenger screening checkpoints to mitigate actual or perceived threats they face and stated that they appreciate the prompt response provided during the LAX shooting incident. Other aviation stakeholders. Additional aviation stakeholders share responsibility in coordinating input and providing recommendations to strengthen security in airport public areas. Such stakeholders include federal, state and local government officials, airline industry partners, aviation associations, and government agencies such as CISA, Federal Bureau of Investigation (FBI), state and local law enforcement partners, emergency management and fire and rescue officials, airline officials, and association members from Airports Council International-North America, Airport Law Enforcement Agencies Network, and American Association of Airport Executives, among others. On November 1, 2013, Gerardo I. Hernandez, a TSO, was shot and killed at a podium located at the base of an escalator which led to the upstairs security checkpoint at LAX, which TSA deemed a part of the checkpoint area, as he checked passengers’ identification and travel documents. According to TSA officials, as passengers and TSOs located upstairs heard the sound of gunshots, they realized there was an active shooter situation and began to run and hide in shops and restaurants. The shooter proceeded upstairs into the security- restricted area and fired additional shots injuring two TSOs and a passenger. Airport police officers responded within 90 seconds and apprehended the shooter within 4 minutes, who, according to law enforcement officials, was specifically targeting TSA employees. As a result, TSA issued an after action report on March 26, 2014, and identified numerous recommendations to enhance the safety and security of TSOs and the screening checkpoint area, such as improving the visibility of law enforcement officers, active-shooter training, and communications systems. On September 24, 2015, the Gerardo Hernandez Act was enacted into law and directed TSA to, as appropriate, conduct outreach to all commercial airports in the United States to ensure they have plans in place to respond to, among other things, active shooter attacks and incidents targeting passenger screening checkpoints, and to report to Congress on the findings from its outreach. On March 22, 2016, suicide bombers using explosives in suitcases killed 16 people and injured more than 200 inside the main terminal area of the Brussels Zaventem International Airport in Belgium. The attack was followed by the June 28, 2016 Istanbul Ataturk International Airport attack in Turkey, where suicide attackers used guns and bombs to kill 46 and injured more than 230 people inside public areas of the airport, including the security checkpoint and parking areas. On January 6, 2017, a passenger obtained a handgun from his checked baggage upon landing at the airport and shot and killed five people and injured six others in the baggage claim area at FLL. Travelers rushed out of the terminal and also ran into security- restricted areas while law enforcement officers responded to the scene. The shooting event ended in less than 80 seconds when the shooter surrendered to law enforcement officers. Approximately 90 minutes after the shooting, speculation of additional gunshots in the airport caused panic and led to an uncontrolled self-evacuation of passengers, and others throughout the airport. Law enforcement and emergency personnel from surrounding areas quickly responded, causing traffic congestion and blocking all airport roadways. As a result, the Broward County Aviation Department (BCAD) issued an after action report on August 15, 2017, to identify coordination challenges airport officials and law enforcement personnel experienced in response to the active shooter incident, and recommended preparedness and response training and exercises among other things. Enacted on October 5, 2018, as part of the Federal Aviation Administration (FAA) Reauthorization Act of 2018, the TSA Modernization Act required, among other things, that TSA and CISA establish a public area security working group and identify and share best practices to secure aviation and other public areas of transportation facilities. On September 27, 2019 a man fired a gun outside of the baggage claim area of the Portland International Airport in Oregon and was injured in a struggle with police officers. See figure 2 for a timeline of these events and response efforts to enhance the security of public areas. In response to the November 2013 shooting at LAX, TSA took various actions to improve security in airport public areas. In March 2014, TSA issued an after-action report on the shooting. TSA officials at LAX stated that confusion about where to run, hide and respond; delayed and inaccurate communications; and the lack of law enforcement visibility were safety concerns that needed to be addressed. As a result, TSA identified short term actions and proposals for increasing airport public area security and enhancing the safety of TSA employees at airports. These include (1) strengthening and mandating active shooter training for TSA employees, (2) installing duress alarms at screening checkpoints, and (3) adopting recommended standards for law enforcement presence at checkpoints, as described below. Active shooter training. In its after-action report, TSA stated that although it provided optional active shooter training courses available online to employees prior to the 2013 shooting, employees were not required to complete the training and could have been unaware of steps to take during a shooting event. According to TSA officials, adequate training and preparation for how to best respond to security incidents, such as an active shooter situation, are important in order to minimize casualties. After its review of the 2013 LAX shooting, on December 19, 2013, TSA mandated that TSA employees complete the training. In addition, TSA later revised the active shooter training to include various training exercises and threat scenarios, according to LAX officials. TSA noted in its after-action report that the active shooter training scenarios and exercises are intended to allow law enforcement officers to practice reacting to a specific incident and immediately assess the appropriateness of their reactions. All of the six airport operators we interviewed stated that active shooter training and frequent drills are important because they instill instinctive reactions and standard communications and procedures in employees during a crisis situation. In addition, airport officials at all six of the airports we visited agreed with the importance of active shooter training to familiarize employees with the steps to take or escape routes to use during an attack. In its after-action report, TSA also noted actions taken to ensure active shooter tactical response plans to reinforce emergency procedures, and conducting emergency evacuation drills twice a year. Airport officials at all of the six airports we visited noted the importance of these drills and stated that they have incorporated the drills into their emergency plans and procedures. Duress alarms. TSA reported that installing duress alarms at screening checkpoint areas would improve communications from TSOs to law enforcement through use of a silent alarm. To enhance emergency response equipment and technology, TSA mandated regular testing of duress alarms, recommended linking closed circuit television (CCTV) cameras to duress alarms, and recommended enhanced use of local airport emergency alert notification systems. TSA also identified the need for FAMS notifications in the event of a security emergency, because previously FAMS did not receive automatic notification of an active shooter incident occurring. Duress alarms are installed at each checkpoint, and when pushed, provide TSOs with a method to notify law enforcement of dangerous situations at or around the checkpoint area. TSOs we met with at five of the six airports we visited discussed a number of situations where travelers have been unruly, threatening, and sometimes physical prior to undergoing or during security screening at the checkpoint. In instances of hostility or threats of attack, TSOs highlighted the importance of having operational duress alarms to help improve the safety and security of the public area and the checkpoint (see figure 3). After the LAX incident, TSA conducted an assessment of all existing alarms and found that some airport checkpoints lacked alarms and some alarms were not fully functional. As a corrective action, TSA issued an operational directive to mandate weekly testing of duress alarms at commercial airports nationwide. In addition, not all airports had duress alarms as a notification capability prior to the LAX incident. As such, TSA subsequently planned to take action to install duress alarms at all airports nationwide. Officials at all of the six airports we visited confirmed the use and weekly testing of duress alarms. Representatives of all four industry associations we contacted stated that the installation of duress alarms in all airports was a useful practice. TSA’s review also recommended linking duress alarms to CCTV cameras to focus camera footage on the area where the duress alarm is activated. All of the six airports we visited have completed or plan to complete linking the alarms to the cameras. Law enforcement presence. Following the LAX shooting, TSA officials, and TSO screeners, wanted to ensure adequate law enforcement presence at the checkpoints. In response to the review of the shooting incident, TSA recommended enhancing law enforcement presence by providing a visible deterrence and establishing quicker incident response times at security checkpoints. In an effort to address the concerns of visibility and responsiveness, TSA recommended standards for law enforcement presence at checkpoints and ticket counters during peak travel times and incorporation of a maximum allowable response time for law enforcement to arrive at an airport checkpoint when notified of a security incident. Prior to the LAX shooting, airport operators were required to comply with existing airport security program requirements to provide adequate law enforcement presence to ensure passenger safety, including responding to threats at security checkpoints. However, when TSA conducted a review of all airport security programs, they found that although most airports specified maximum response times to checkpoints, 71 airports that maintained flexible law enforcement coverage did not list a required response time in their security programs. As a result, TSA required all airports to clearly include a maximum allowable law enforcement response time in all security programs. Officials for the five airports we visited with the highest passenger volumes stated they comply with the response time requirement listed in their security program while officials from a smaller airport we met with told us they include a longer maximum response time as a requirement in their security program. While recommended standards for law enforcement presence and maximum response times are required in airport security programs, nearly all 50 TSOs and TSO supervisors at the six airports we visited expressed concerns for safety and security while conducting screening operations in the passenger checkpoint areas. Many of these TSOs and supervisors said they feel vulnerable to both physical and verbal attacks, and public misperceptions of their overall roles and responsibilities. The majority of TSOs also noted concerns about adequate law enforcement presence and attentiveness in the checkpoint areas and because they are sometimes harassed for conducting their screening duties. While many of the TSOs stated they sometimes feel their concerns are not always met with action, such as supervisors intervening or calling upon law enforcement for assistance, they said they appreciate law enforcement’s response to the LAX shootings and value building relationships with law enforcement present at the checkpoint. All TSOs we interviewed expressed interest in continuing to provide feedback to TSA headquarters and offering suggestions for improving their safety. In addition to actions taken in response to the after-action report for the LAX shooting, TSA took other actions consistent with the Gerardo Hernandez Act. As previously mentioned, the act directed TSA to, as appropriate, conduct outreach to all commercial airports in the United States to ensure they have plans in place to respond to, among other things, active shooter attacks and incidents targeting passenger screening checkpoints, and to report to Congress on the findings from its outreach. In response to the Gerardo Hernandez Act, TSA conducted outreach to all commercial airports and analysis of each airport’s preparedness to respond to security events. TSA determined that all of the airports had plans in place to respond to security incidents in the public areas of airports including active shooters, acts of terrorism, and incidents that target passenger screening checkpoints. TSA also determined that all commercial airports had met TSA regulatory requirements related to security incident response planning. After the LAX shooting and subsequent review, TSA issued an Operations Directive in August 2014, about one year before enactment of the Gerardo Hernandez Act, detailing specific guidance and TSA employee procedures for responding to an active shooter incident. Upon reviewing the act, TSA concluded that the procedures outlined in its directive were consistent with requirements and that no further action was required. For example, TSA had provided guidance for its personnel to mentally prepare themselves in advance for an active shooter incident by predetermining an escape route that offers concealment or cover. TSA guidance had also encouraged employees to use the mantra of “Run, Hide, Fight” during active shooter incidents. Furthermore, over the next few years, TSA released a new training video, issued revised operational guidance, and nation-wide update concerning security measures. In January 2015, TSA released a new active shooter training video, “Active Shooter Incident Response Training” for active shooter incidents specifically depicting an airport environment. The interactive training video was filmed at Indianapolis airport with support and participation from local airport officials, law enforcement officers, and TSA personnel. TSA released the training video with a required completion date of March 31, 2015, and mandated that this be completed as an annual training requirement for all TSA personnel. In July 2016, TSA also issued revised operational guidance for reporting aviation security incidents to the Transportation Security Operations Center, including security breaches and suspicious activities, among others. In November 2017, TSA issued a national update to airport security programs for law enforcement coverage of certain airport public areas under the National Terrorism Advisory System. These documents include guidance and procedures that align with requirements of the Gerardo Hernandez Act for verifying that plans exist, and for identifying and sharing best practices, across airports to respond to security incidents inside the airport perimeter, including active shooters, acts of terrorism, and incidents that target passenger screening checkpoints. More recent actions have also been taken that correspond with requirements of the Gerardo Hernandez Act to have plans to respond to active shooter attacks and incidents targeting passenger screening checkpoints. Specifically, in March 2018, TSA issued a revised security directive to enhance security of airport public areas by identifying responsibilities for local law enforcement coverage of airport public areas, including the passenger screening checkpoints and nearby public areas. Also, in August 2018, TSA issued an information circular describing best practices identified by airport operators to mitigate against insider threats, including practices related to conducting vulnerability assessments, and escort procedures, among others. In response to other security incidents in airport public areas, TSA has taken additional actions to enhance security. Specifically, in 2017, TSA issued the Public Area Security National Framework (Public Area Framework). TSA developed the framework following a series of security summits that gathered stakeholders together to identify ways to mitigate threats against aviation and surface transportation public areas. Between September 2016 and April 2017, TSA and the DHS Office of Infrastructure Protection (now within CISA) co-hosted four public area security summits (see fig. 4). According to TSA officials, the summits leveraged an entire network of transportation and security officials— including industry, government, academic, international, and public stakeholders—to develop a set of best practices and recommendations that could help deter nefarious actors in the transportation environment. As a result of these summits, in May 2017, TSA published the Public Area Framework, which established best practices and recommendations for protecting public areas from harmful attacks. TSA officials described the summits as opportunities for stakeholders to generate meaningful dialogue and exchange ideas as opposed to developing a formal strategy or prescriptive action plan with an implementation time frame. Moreover, this official added that the framework was “intended to be a toolkit for stakeholders, designed by stakeholders.” Industry stakeholders and airport officials we interviewed reported that the security summits were beneficial for gathering key stakeholders together to determine a variety of measures to enhance security of airport public areas, some of which had already been implemented at certain airports. The Public Area Framework categorized 11 best practices across three key areas: sharing information, preventing attacks, and securing public infrastructure (see fig. 5). According to TSA officials, the report was intended to be a framework, which consisted of non-binding best practices developed by and used for aviation and surface transportation security stakeholders to implement public area security improvements in their respective operating environments. During our interviews with 10 sets of airport stakeholders—consisting of airport operators and law enforcement officials from six airports and industry representatives from four aviation trade associations—we found that all 10 stakeholder groups reported that the resulting best practices were useful in increasing their awareness of the various ways in which airports can enhance the security of their public areas. For example, airport operators at Los Angeles International Airport (LAX) and Hartsfield-Jackson International Airport (ATL) and law enforcement officials at Fort Lauderdale-Hollywood International Airport (FLL) stated that establishing an airport operations center, one of the best practices recommended in the framework, provides real-time monitoring capabilities of security-related events throughout the airport and the ability to communicate more effectively in the event of an emergency. Similarly, representatives of industry trade associations, such as Airports Council International-North America and American Association of Airport Executives, stated that enhanced law enforcement patrols throughout public areas provide a visible deterrent against potential attacks during peak travel times while also ensuring adequate resources are available to respond quickly to potential threats. As recommended in the framework, strategies to deploy law enforcement patrols are one of the most basic forms of deterring, detecting and defeating potential attacks and a part of coordinating response planning. Several stakeholders groups also added that while the framework was useful in formally documenting industry agreed upon best practices and recommendations, many of the practices, including the use of airport emergency operations centers and enhanced law enforcement patrols were already being implemented locally by various airport operators nationwide. In addition to issuing the Public Area Framework, TSA took additional actions in recent years in response to other security incidents in airport public areas led by TSA’s Policy, Plans, and Engagement (PPE) office, which was also responsible for engaging airport and surface transportation stakeholders in developing the 2017 Public Area Framework recommendations. Specifically, the TSA Modernization Act, enacted in October 2018, required TSA, in coordination with CISA, to establish a public area security working group to promote collaboration between TSA and public and private stakeholders to develop non-binding recommendations for enhancing security in public areas of transportation facilities. The act also requires TSA to periodically share best practices developed by TSA and transportation stakeholders related to protecting public spaces of transportation infrastructure from emerging threats with transportation security stakeholders. According to TSA officials we interviewed, PPE established the public area security working group in March 2019 to engage with stakeholders and update the original best practices that were developed in the 2017 Public Area Framework. TSA conducted two conference calls—March 2019 and June 2019—with the working group members to update, discuss, and validate the existing best practices. The working group consists of security stakeholders from both aviation and surface transportation modes and includes several of the same stakeholders who participated in the 2017 public area security summits to develop the 2017 Public Area Framework and associated recommended best practices. For the working group, TSA PPE officials reported that TSA selected a subset of stakeholders who were highly engaged and participatory during their prior security summits and who provided the most input during focus group discussions. According to TSA officials, many of the stakeholders previously involved in the development of the Public Area Framework—including several industry associations representing aviation and surface transportation stakeholders—are aware of ongoing issues and emerging threats. For example, while engaging with TSA during the March 2019 and June 2019 conference calls, industry stakeholders identified ways for enhancing the security of airport public areas by 1) utilizing various technologies, such as public address notification systems throughout airports, to better communicate instructions during and after security incidents occur in the public area, and 2) establishing clearer guidance and protocols for resuming business operations after a security incident, such as rescreening passengers and positively identifying lost baggage in the terminal area. Moreover, stakeholders cited the growing concerns about the emergence of unmanned aircraft systems, such as drones, which pose risks to securing airport public areas. In late October 2019, in accordance with the TSA Modernization Act, TSA issued a report listing best practices and recommendations to secure transportation public areas. This report summarizes the working group’s effort to review and update prior best practices cited in the 2017 Public Area Framework as well as identify current challenges. For example, the updated report provides specific tools and resources for enhancing situational awareness, such as resource guides providing informational materials, fact sheets, research reports, and online training videos. Specifically, one of the resource guides highlights security of soft targets and crowded places—sports venues, shopping areas, schools, and transportation systems—as locations that are easily accessible to large numbers of people that have limited security measures in place making them vulnerable to attack. For example, the guide describes how TSA focuses its effort on securing aviation and high-risk locations by deploying law enforcement and canine teams to serve as a visible deterrent. Other resources include a fact sheet regarding challenges posed by unmanned aircraft systems and a research report regarding mass attacks in public spaces, among others. Moreover, the updated report highlights the benefits of an airport operations center, including enhanced communication capabilities and situational awareness. Although TSA and stakeholders have taken actions in response to other security incidents in airport public areas, TSA has not fully developed a plan for future engagement with stakeholders to update security best practices and ensure they are current, relevant, and reflective of any new transportation security advancements or new and emerging threats. According to the October 2019 updated report, TSA intends to engage with stakeholders on a periodic basis to affirm partnerships. However, TSA has not yet clearly defined roles and responsibilities for stakeholders or how frequently to engage with them, such as on a quarterly, semi- annual, or annual basis. The TSA Modernization Act requires TSA to periodically share best practices for protecting transportation public areas. Additionally, both the 2017 Public Area Framework and updated report from the working group emphasize the importance of continuing partnerships efforts and identifying solutions to improve public area security. For example, the working group’s updated report issued in October 2019, in accordance with the TSA Modernization Act, highlighted TSA’s role to build upon the work already accomplished in developing the Public Area Framework’s best practices and recommendations. Additionally, our prior work has identified leading practices that can help sustain collaboration, such as developing a plan identifying roles and responsibilities for parties included in the collaborative effort. Further, standards for project management call for developing a plan with specific actions and time frames. TSA officials stated that they expect to better determine future plans for stakeholder engagement sometime after TSA issues its mandated report on the public area security working group to Congress in March 2020. However, TSA officials told us they currently have no specific plans outlined regarding the process or frequency with which they will engage stakeholders in the future on public area security best practices. By developing a plan that outlines the roles and responsibilities of the working group members, the mechanisms through which the working group will collaborate, and the frequency of when the working group will meet, TSA would be better positioned to ensure the best practices cited by aviation and surface transportation stakeholders remain relevant and emerging threats are proactively identified. Aviation stakeholders—consisting of airport operators, law enforcement officials, airline representatives, among others—have taken a series of actions in response to security incidents that followed the 2013 LAX shooting, consistent with the best practices outlined in TSA’s Public Area Framework. On the basis of our observations and interviews at six airports, we found that aviation stakeholders have taken actions consistent with the best practices identified in the 2017 Public Area Framework—including those related to attack prevention and information sharing—and engaged with industry association representatives to better understand key efforts involved in securing airport public areas. Collectively, these efforts represent a series of actions taken by stakeholders in response to the 2013 LAX shooting, the 2017 FLL active shooter incident, or a combination of TSA requirements to enhance their security posture in securing airport public areas. Attack Prevention: Establish Airport Operations Center. One of the practices cited in the framework under preventing attacks in airport public areas is establishing an airport operations center, which calls for a unified command center to respond to airport security incidents. As cited in the Fort Lauderdale after action report, airport operators and law enforcement personnel experienced coordination challenges in responding to the active shooter incident because of inadequate communication capabilities, including interoperable communications and lack of a dedicated space to coordinate and deploy resources, among others things. Three of the airports we visited—Los Angeles, Atlanta, and Fort Lauderdale—had a dedicated airport operations center in place. During our site visit to Los Angeles, we toured the Airport Response Coordination Center which provides 24/7 response coordination capabilities between LAX airport operators and federal, state, and local law enforcement personnel (see fig. 6). LAX security officials told us that the coordination center provides real-time situational awareness of security-related incidents across the entire airport through monitoring closed-circuit television, as well as direct communications with federal and local law enforcement partners. Moreover, LAX security officials reported that the coordination center also houses the Incident Management Center which is activated as an emergency command center designed to integrate resources for all airport divisions and law enforcement agencies in response to a major security incident. Similarly, Atlanta and Fort Lauderdale had dedicated airport operations centers, including a comparable Incident Management Center, equipped with dedicated work stations, designated color-coded vests, secure video teleconference capabilities, and a mobile command center. Attack Prevention: Develop, Conduct, and Practice Exercises and Response Drills. The Public Area Framework also contained a recommendation that transportation stakeholders develop and conduct exercises and response drills to prepare for real-world incidents and identify potential obstacles to responding effectively. These collaborative engagements are intended to help develop strategies for incident management, such as resuming airport business operations—including evacuating civilians during a law enforcement response, securing and returning abandoned luggage, and rescreening passengers, among other tasks—and identify areas requiring additional coordination. For example, in August 2017, as a follow-up to the framework, the summit commissioned a new working group to specifically address incident response, recovery, and reconstitution. Specifically, the Intermodal Security Training and Exercise Program (I-STEP) developed and conducted a tabletop exercise focused on “Resumption of Trade” following an incident. The exercise was designed for aviation stakeholders—airport operators, law enforcement officials, and airline representatives, among others—to discuss, identify, and improve collective capabilities in responding to physical security incidents at airports and facilitating the orderly re-establishment of airport operations. The exercise assessed stakeholders’ ability to quickly and accurately 1) communicate critical information during and after a security incident in an airport public area, 2) evacuate passengers, employees, and vendors after a security incident, and 3) restore airport operations following a security incident. These training goals stemmed from the lessons learned from the 2017 FLL shooting as well as stakeholders’ discussions during the summits around developing best practices. According to airport operators at Ronald Reagan Washington National Airport (DCA), I-STEP was well received by airport security stakeholders because it was an event that incorporated several key stakeholders’ perspectives at one time and presented an opportunity for constructive viewpoints to be shared. As a result of the I-STEP training exercise, TSA issued an after action report identifying strengths and lessons learned in resuming airport operations, and highlighted areas for improvement including 1) better coordination to ensure all airport vendors and stakeholders receive standardized active shooter training and 2) consideration of the impacts and risks associated with plainclothes law enforcement and lawfully-armed individuals responding to an active shooter incident, among others. Airport stakeholders we met with at DCA also shared examples of locally sponsored training exercises. For example, airport officials stated that table top exercises—discussions amongst emergency response personnel regarding the various roles and responsibilities during an airport emergency response situation—are frequently held at DCA and most focus on the need for better communication. DCA officials added that quarterly training drills are hosted for nearby county law enforcement and fire department officials to better familiarize themselves with DCA airport operations, layout, and prestaging areas. Information Sharing: Enhance Situational Awareness. Airport operators we met with also shared their experiences with implementing best practices and recommendations cited in the Public Area Framework to enhance situational awareness and expand threat awareness education. For example, during our site visit to FLL, one senior airport security official shared examples of actions undertaken in response to the FLL after-action report to enhance workforce employee training and threat awareness education. This included the development of an active shooter training program required for all airport workers, such as airport concessionaires and rental car operators; and enhancing the airport’s credentialing program to better distinguish certain workers requiring access to the secure area, and validate that active shooter training has been completed. Similar to FLL, according to the Public Area Framework, Boston’s Logan International Airport and ATL have implemented vetting programs that include the issuance of identification cards for airport workers on the public side of airports. According to aviation stakeholders, issuing public side credentials allow the airport to have better awareness of who is working within their environment, thereby enhancing overall situational awareness within airport public areas, a practice recommended in the Public Area Framework. Airports we met with have also taken actions that are generally consistent with its three main categories, including infrastructure and public protection. For example, in an effort to enhance the security of airport public areas, several airport operators and law enforcement officers we met told us that they regularly deployed enhanced law enforcement teams to patrol public spaces, including ticketing counters and baggage claim areas, among others. These specialized law enforcement teams— equipped with assault rifles, body armor, and canine teams—patrol airport public areas to provide a visible deterrent against criminal or terrorist activities and provide immediate law enforcement response capabilities. While these enhanced law enforcement teams were initially deployed in response to the LAX shooting, LAX security officials we met with stated that their continued presence in the public areas provides a strong deterrent. During a site visit to LAX, we observed two sets of tactical law enforcement teams patrol the Bradley International Terminal and American Airlines ticketing counter areas. One of the patrol teams included an explosives-detection canine and his handler (see fig. 7). These enhanced law enforcement teams also conduct training exercises to detect explosives that may be hidden throughout airport public areas, such as large atriums and baggage claim areas. These efforts are consistent with TSA airport security requirements and guidance provided in TSA’s Law Enforcement Reimbursement Program. For example, during the site visit to ATL, we observed Atlanta Police Department tactical law enforcement response teams patrol the large atrium meeting area—consisting of restaurants and shops—from an elevated position (see fig. 8). Similarly, during a site visit to FLL, we observed the Broward County Sherriff’s Office deploy an enhanced law enforcement team with an explosives-detection canine succeed in identifying hidden explosive materials inside a handbag during a training exercise in the baggage claim area, the exact location of the January 2017 active shooter attack (see fig. 9). In addition, Charleston International (CHS) airport deployed an active shooter detection system to enhance the security of airport public areas. The Shooter Detection System is a network of sensors placed throughout airport public areas that identify acoustic gunshot signatures and track the movements of a potential active shooter in real time through security camera footage (see fig. 10). According to senior CHS airport officials, the 2015 Emanuel African Methodist Episcopal Church shooting in Charleston that claimed the lives of nine area residents, significantly influenced the airport’s decision to search for active shooter technologies. Charleston County Aviation Authority and senior CHS airport officials reported that the detection system became operational in December 2018. Aviation industry stakeholders, such as those from Airports Council International-North America and American Association of Airport Executives, plan to further research technologies to enhance the security of airport public areas. While CHS is the first U.S. commercial airport to install an active shooter detection system in ticketing counter and baggage claim areas, aviation industry officials stated that several airports nationwide are considering installing similar systems. Attacks in the public areas of both domestic and foreign airports— including Los Angeles, Fort Lauderdale, Brussels, and Istanbul—have prompted TSA and aviation security stakeholders’ efforts to enhance the security of airport public areas. In accordance with the October 2018 TSA Modernization Act, TSA established a public area security working group to build upon the best practices and recommendations previously cited by stakeholders in the 2017 Public Area Framework. The actions taken by TSA and aviation security stakeholders represent a collective effort to enhance the security of airport public areas. Similarly, airport security stakeholders we interviewed took actions consistent with the best practices identified in the 2017 Public Area Framework, such as establishing a unified airport operations center, deploying enhanced law enforcement teams, and using technologies to identify the whereabouts of an active shooter, among others. However, TSA has not fully developed a plan that outlines the roles and responsibilities of the working group members, collaboration mechanisms amongst the working group, and frequency in which the working group will meet. By developing such a plan, TSA would be better positioned to ensure that the working group is proactively meeting to identify and share emerging threats and best practices, instead of reconvening in the aftermath of another security incident involving an airport public area. The Administrator of TSA should develop a plan outlining roles and responsibilities for members of the Public Area Security Working Group, the mechanisms for collaborating, and the frequency of the working group meetings. We provided a draft of this report to DHS for review and comment. DHS provided written comments, which are reproduced in appendix I. DHS concurred with our recommendation and described actions to address it, such as developing Public Area Security Working Group guidelines to include roles and responsibilities, mechanisms of collaboration, and frequency of working group meetings by June 30, 2020. These efforts, if fully implemented, should address the intent of the recommendation. DHS also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of the Department of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. William Russell (202) 512-8777 or RussellW@gao.gov. In addition to the contact named above, Christopher Ferencik (Assistant Director), Katrina Taylor (Analyst-in-Charge), Pamela Davidson, Josh Diosomito, Eric Hauswirth, Thomas Lombardi, Herbert Tinsley, and Adam Vogt made significant contributions to this report.", "summary": "Threats to public areas of airports have increased in recent years. TSA is responsible for civil aviation security, which includes ensuring the security and safety of aircraft and the traveling public. In November 2013, an armed individual entered the Los Angeles International Airport (LAX) and killed a Transportation Security Officer. Subsequent domestic and international attacks in airport public areas further emphasized the need to better secure these areas. In response, Congress has taken action—including passage of the 2015 Gerardo Hernandez Act and the 2018 TSA Modernization Act—to address incident planning and response at airports and the security of public areas of transportation facilities, including airports. GAO was asked to assess actions taken to secure the public areas of TSA- regulated airports. This report (1) describes actions TSA has taken in response to the LAX shooting and the Gerardo Hernandez Act, and (2) examines additional actions taken in response to subsequent security incidents and the TSA Modernization Act. GAO reviewed TSA reports issued after airport attacks; the Gerardo Hernandez Act and TSA Modernization Act; and other TSA documents related to securing public areas. GAO also conducted interviews with and obtained information from TSA and officials from a nongeneralizable sample of 6 TSA-regulated airports, selected based on factors such as size and location. The Transportation Security Administration (TSA) took several actions in response to the 2013 Los Angeles International Airport (LAX) shooting and the Gerardo Hernandez Airport Security Act of 2015. Specifically, TSA took several actions to better address airport security in public areas, including strengthening and mandating active shooter training drills and installing duress alarms at screening checkpoints, among other things. In response to the Act, TSA updated guidance for reporting suspicious behavior and revised directives identifying responsibilities for local law enforcement coverage of passenger screening checkpoints and nearby public areas, among other actions. In response to subsequent airport public area security incidents, such as those in Fort Lauderdale in 2017 and Brussels and Istanbul in 2016, TSA has taken additional actions. Specifically, TSA issued the Public Area Security National Framework in 2017, in coordination with various aviation security stakeholders. The framework categorized 11 best practices and non-binding recommendations for improving security of public areas, including sharing information and preventing attacks. Aviation security stakeholders have also implemented various actions consistent with these best practices, including establishing airport operations centers and deploying enhanced law enforcement teams to serve as a visible deterrent in airport public areas (see figure). In response to the TSA Modernization Act, TSA established a public area security working group in March 2019 to engage with stakeholders such as airport operators and industry associations and update and validate the best practices cited in the 2017 framework. This group met twice in 2019, but TSA has not outlined specific plans for engaging this group in the future. Developing a plan outlining the roles and responsibilities of the working group members, the mechanisms through which the working group will collaborate, and the frequency of when the working group will meet, would better position TSA to ensure the best practices cited by stakeholders remain relevant and emerging threats are proactively identified. GAO recommends that TSA develop a plan for future stakeholder engagement on the security of airport public areas. DHS concurred with the recommendation.", "document_type": "gao"}
{"report": "The activities and resources required to suppress wildfires generally belong to the states and federal agencies with land management missions, such as the U.S. Forest Service and four bureaus (Bureau of Land Management, Bureau of Indian Affairs, National Park Service, and U.S. Fish and Wildlife Service) within the U.S. Department of the Interior. FEMA can provide reimbursement to help support wildfire suppression (e.g., labor costs for overtime or seasonal personnel involved in fire suppression activities). When a wildfire burns on nonfederal lands and threatens to become a major disaster, a state governor or governor’s representative may request federal assistance via a Fire Management Assistance Grant (FMAG) administered by FEMA. While the fire is burning, a governor’s office can submit a verbal request for an FMAG to the designated FEMA regional office, followed within 14 days by a formal written request. The regional administrator then either approves or denies the request after consulting with relevant officials from the U.S. Forest Service or bureaus within the U.S. Department of the Interior about technical aspects of the fire. Eligible FMAG costs include, among other things, equipment and supplies, labor costs, travel and per diem, temporary repairs of damage caused by firefighting activities, mobilization and demobilization of resources, and limited costs of pre-positioning fire prevention or suppression resources. From fiscal years 2009 through 2018, FEMA awarded 374 FMAGs totaling $952,318,049. The average FMAG during this timeframe was about $2.5 million. The state of California received the majority of those grant funds—over $543 million. Figure 1 below illustrates the states that received FMAGs during this 10 year period, figure 2 provides annual FMAG totals, and figure 3 provides a breakout of the dollars distributed by state for this same 10 year period. If a wildfire increases in size and intensity in a manner that overwhelms the ability of state, tribal, territorial or local governments to respond and recover effectively, a state or tribal government can request and the President can approve a major disaster declaration, as with other types of disasters (e.g., a hurricane or flood). A disaster declaration is the primary mechanism by which the federal government gets involved in funding and coordinating response and recovery activities. Under the National Response Framework, the Department of Homeland Security (DHS) is the federal department with primary responsibility for coordinating disaster response, and within DHS, FEMA has lead responsibility. From fiscal years 2009 through 2018, a total of 19 major disasters were declared as a result of wildfires. Figure 4 shows the number and locations of these major disaster declarations. Once a major disaster is declared, FEMA can provide funds for response and recovery efforts through the Disaster Relief Fund and coordinate other federal support through the National Response Framework’s 14 Emergency Support Functions. Federal assistance following a major disaster declaration includes the following: Individual Assistance: FEMA’s Individual Assistance programs provide assistance directly to individuals and households, as well as state, local, tribal, and territorial governments to support individual survivors. This assistance covers necessary expenditures and serious needs that cannot be met through insurance or low-interest loans, such as temporary housing assistance, counseling, unemployment compensation, or medical expenses. See appendix I for a further description of FEMA’s Individual Assistance program. Public Assistance: FEMA’s Public Assistance program provides supplemental federal disaster grant assistance to state, local, tribal, and territorial governments and certain types of private nonprofit organizations for debris removal, emergency protective measures, and the restoration of disaster-damaged, publicly-owned facilities and the facilities of certain private nonprofit organizations. The eligibility rules outline the types of damage that can be reimbursed by the federal government and steps that federal, state, and local governments must take in order to document eligibility. If the debris on private property is determined to be so widespread that it threatens the health, safety, or economic recovery of the community, FEMA may determine that debris removal from private property, including contaminated soil, is eligible for reimbursement under the Public Assistance program. An applicant (a state, territorial, or tribal government) may contract for debris removal. Alternatively, if an applicant lacks the capability to perform or contract for debris removal, the applicant may request that the federal government perform the work. In such cases, FEMA may task another federal agency, typically the U.S. Army Corps of Engineers (USACE), to perform or contract the work by issuing a mission assignment (see description below). See appendix I for a further description of FEMA’s Public Assistance program. Mission Assignment to Other Agencies: FEMA can fulfill disaster response needs through mission assignments—work orders it issues to another federal agency to provide a service or other response need. For example, FEMA may request medical teams from the Department of Health and Human Services and logistical support from the Department of Defense. Hazard Mitigation Grant Program: This program is designed to improve disaster resilience—the ability to prepare and plan for, absorb, recover from, and more successfully adapt to disasters—during recovery. The program funds a wide range of projects, such as use of non-combustible materials on new and existing homes to mitigate risk from future wildfires, adding shutters to windows to prevent future damage from hurricane winds and rains, and rebuilding culverts in drainage ditches to prevent future flooding damage. Table 2 below shows money obligated for Individual Assistance, Public Assistance, mitigation efforts, operations (including mission assignments), and administrative costs for the 19 major disaster declarations resulting from wildfires from fiscal years 2009 through 2018. The U.S. Forest Service within the Department of Agriculture and the Bureau of Indian Affairs, Bureau of Land Management, U.S. Fish and Wildlife Service, and National Park Service within the Department of the Interior, are responsible for managing wildfires on federal lands. Wildfire management consists of three primary components: 1. Preparedness involves acquiring and positioning firefighting assets. 2. Suppression involves selecting among strategies to extinguish or contain a fire, with the aim of protecting firefighters and public safety and using the minimum resources necessary. 3. Fuels Reduction involves acting in advance of wildfires to manage vegetation with the aim of reducing the intensity, severity, or negative effects of a wildfire. We are currently reviewing federal fuel reduction efforts, and how those efforts consider community protection, and plan to issue a report on the subject later this year. State forestry agencies and other nonfederal entities—including tribal, county, city, and rural fire departments—have primary responsibility for managing wildfires on nonfederal lands, and share responsibility for protecting homes and other private structures. When a wildfire occurs on nonfederal lands and begins to exceed the state or local entity’s ability to effectively respond to the wildfire, the state or local entity may seek assistance from neighboring jurisdictions, typically through prescribed mutual aid agreements. For example, during wildfires in California in October and December of 2017, the California Governor’s Office of Emergency Services used the California fire and rescue and law enforcement mutual aid systems, along with the national Emergency Management Assistance Compact to mobilize and organize a large number of emergency services. In total, according to California Governor’s Office of Emergency Services, over 400 state and local government and 200 out-of-state fire departments sent engines, crews, and other assets to assist the local firefighting efforts. When a state or local jurisdiction needs further firefighting assistance, it may request additional support through Geographic Area Coordination Centers overseen by the National Interagency Fire Center. Once a Geographic Area Coordination Center has exhausted the resources it can provide, it can turn to the National Interagency Coordination Center within the National Interagency Fire Center for further assistance. For wildfire disaster declarations from 2015 to 2018, FEMA provided a variety of assistance to state and local emergency management officials consistent with roles and responsibilities in the National Response Framework and National Disaster Recovery Framework. Specifically, FEMA helped these jurisdictions by reimbursing some fire suppression costs, supporting state-led efforts to coordinate the response and provide for the immediate needs of displaced survivors, and helping localities plan and execute recovery. FEMA has obligated over $2.4 billion to assist in response to and recovery from these disasters to date. As previously discussed, although states and other federal agencies have primary responsibility for fire suppression, some state and local fire suppression costs are eligible for reimbursement through FMAGs. Most wildfire-affected states and localities in our scope received this kind of fire suppression support from FEMA initially in the form of the FMAGs. As the fires ultimately led to major disaster declarations, any funding that FEMA would have provided through the FMAGs were ultimately provided under Public Assistance as part of the declaration. To support state-led response and provide for the immediate needs of displaced survivors, FEMA deployed staff to assist in state Emergency Operations Centers and secured needed resources for mass care—such as cots to help with temporary sheltering, according to state officials. In addition, FEMA assigned federal agencies to perform various missions to help with disaster response. For example, the Environmental Protection Agency provided hazardous material cleanup of damaged properties, and USACE provided public works services, such as contracting for debris removal. As response activities continued and recovery began, FEMA and the state emergency management agencies established Joint Field Offices, which are temporary field offices established to coordinate federal and state efforts in disaster response and recovery, and provided resources to help individual disaster survivors with community services and housing needs. For example, following wildfires in November 2018—including the Camp Fire in Butte County—FEMA provided over $55 million to survivors to reimburse them for the cost of temporary lodging and rentals after their homes were destroyed. In addition, FEMA provided funding and support to local jurisdictions to help address community infrastructure needs. For example, FEMA obligated money to pay for wildfire debris removal from public property as well as from private property, given the widespread effect on the community of toxic fire debris. Also to support recovery, in coordination with state and local entities, FEMA established and staffed Disaster Recovery Centers, which are facilities or mobile offices where survivors can go for information about FEMA programs or other disaster assistance programs. Representatives from the relevant state agencies, FEMA, U.S. Small Business Administration, volunteer agencies, and other agencies were at the centers to answer questions about and help survivors apply for disaster assistance and low-interest disaster loans for homeowners, renters, and businesses. Finally, to assist local jurisdictions with longer-term recovery, FEMA provided assistance to some locally-led long-term recovery activities designed to address housing and other survivor needs in the community. Table 3 shows the amount of assistance FEMA provided for each of the six major disasters that we reviewed, and Appendix II provides a more detailed breakdown of each major disaster, including a map of each disaster, the number of structures that were destroyed, and mission assignment data. State and local officials we spoke with reported practices that aided in wildfire response and recovery and also experienced challenges that arose in multiple jurisdictions across different disasters. When asked what worked well, officials from three out of the six California counties told us that FEMA and the California Governor’s Office of Emergency Services collaborated effectively during response and recovery efforts. For example, one of the three counties reported that when posing questions or concerns to the California Governor’s Office of Emergency Services, they were able to quickly obtain answers or further information and get help navigating complex issues. As we reported in 2018, according to officials in the California Governor’s Office of Emergency Services and FEMA, they have developed a strong relationship with each other over time, which helps both agencies deliver consistent, unified information to stakeholders and disaster survivors. Local officials also praised FEMA’s role in helping to set up and operate Disaster Recovery Centers. Officials in four of the six California counties that we interviewed noted that FEMA was quick to send staff to assist local jurisdiction staff and disaster survivors at the facilities established to provide assistance, such as Local Assistance Centers (generally activated by the county in the immediate wake of a disaster to provide government services to survivors) and Disaster Recovery Centers established by FEMA. For example, one of these counties noted that FEMA had staff available at their Local Assistance Center to support requests for Individual Assistance and other items shortly after the disaster was declared, and the county received positive feedback from the public about the varied types of support provided by experienced staff at their Local Assistance Center. Officials in one of the counties mentioned above, as well as FEMA officials, cited as good practices efforts to bring together local and state providers of governmental services to provide a variety of assistance in one place. For example, FEMA credited one county for their efforts in partnering with a local mental health service provider to offer mental health counseling on site at a Disaster Recovery Center, as opposed to referring individuals to such services off site. Similarly, one Disaster Recovery Center we visited in California included representation from a number of different state agencies, such as the state’s contractors’ licensing board, insurance regulators, department of employment opportunities, and franchise tax board. Officials explained that being able to access a variety of state services in a Disaster Recovery Center can be particularly helpful for fire survivors, as they may have evacuated their homes with very little notice and lost all their identifying documentation to the fire. State and county officials described challenges that were present in several of the wildfire disaster declarations that we reviewed. Some of these challenges—such as a complex Public Assistance application process or FEMA staff turnover—are not specific to wildfires and could also affect recovery efforts after a hurricane, flood, or other natural disaster. Some challenges were more specific to and further complicated by the nature of wildfire disasters. These challenges include the complexity and scale of fire debris removal, shortage of temporary housing for wildfire survivors, and lack of local experience dealing with the magnitude of the wildfires encountered in 2017 and 2018. Officials in three of the seven counties we met with said that the onerous and confusing documentation required when applying for Public Assistance grants was a challenge. For example, an official from one county told us that the Public Assistance guidance in effect at the time his county was recovering from disaster contained conflicting information, though he believed this issue has since been resolved. Officials in two counties also described difficulty meeting the deadlines for application submission, especially while managing the other demands of disaster response and recovery. We have previously reported on challenges with FEMA’s administration of the Public Assistance program, including effectively overseeing and staffing the program, among other things. Officials from FEMA’s Public Assistance Division acknowledged that the complexity of the program has been a challenge for local officials in recent years. The officials pointed to the development of a new Public Assistance delivery model as the key initiative to address these challenges. This new delivery model, which includes a new information portal designed to improve local officials’ ability to upload and submit information, was intended to clarify program requirements, improve operations, and respond to previously-identified challenges, according to FEMA officials. FEMA introduced the new model in California during the recovery phase of the 2017 wildfires. Officials from two of the selected counties stated that the new information portal eased the process of submitting documentation for FEMA review. In 2017, we reported on the historical challenges with FEMA’s Public Assistance program and identified additional challenges with the roll-out of the new delivery model, including the need to determine its staffing needs for supporting rollout of the system and strengthen controls over the information system being used. California officials we spoke with also noted that in order for the new delivery model to be used efficiently, it would be helpful for FEMA to provide additional training to stakeholders who use the system. According to FEMA officials, FEMA provided a number of training sessions on the new model to California stakeholders between August 2017 and April 2019. Officials in three of the seven selected counties told us that frequent rotations of FEMA staff during disaster response and recovery was disruptive. For example, after working with state and local officials following a disaster, the rotations of FEMA staff resulted in having to re- share information that was already provided to FEMA, as well as inconsistent advice or interpretation of FEMA guidelines. FEMA officials acknowledged that ensuring continuity following staff turnover has long been an issue in multiple complex disaster environments. They noted a number of reasons why a staff member in a position might turn over. For example, according to FEMA officials, the disasters in 2017, including Hurricanes Harvey, Irma, and Maria, as well as the California wildfires, required FEMA management to redeploy response personnel from one disaster to the next. We have reported on multiple FEMA workforce challenges in prior work and continue to observe workforce challenges in our ongoing work. We are currently reviewing how FEMA deploys and trains staff to meet disaster mission needs and plan to report early in 2020. Debris removal is an important first step in the disaster recovery process, allowing communities to expedite the recovery process by restoring accessibility to public services and space, while ensuring public health and safety in the aftermath of a disaster. Debris removal posed several challenges for state and local jurisdictions affected by the wildfires. Wildfires typically leave no remaining structure, and the resulting ash contains contaminants that must be carefully removed, wrapped, and disposed of before survivors can move back to their properties. This can make the wildfire debris removal process costlier and more complicated than for other types of disaster debris. California’s Department of Resources, Recycling, and Recovery typically handles debris removal after local disasters in the state, but it did not have the capacity to handle the high volume of debris caused by the 2017 Northern California wildfires. As a result, the state asked FEMA to assign USACE with the debris removal mission. According to local officials, there was some confusion over how much contaminated soil should be removed from some properties. Specifically, in some cases, USACE removed more soil than necessary at home sites in an attempt to “scrape” the soil deeply enough to remove all possible contaminants at the site; however, this did not take into account that some contaminants, such as arsenic, occur naturally in the soil. As a result, some property owners were left with large over-excavated pits on their property that needed to be filled in before rebuilding could occur. Figure 5 shows a property site that, according to local officials, had been excavated below the foundation of the home and thus needed to be refilled with soil, complicating the rebuilding effort. In addition, officials from one county stated that USACE staff rotations made it difficult for state and local officials to communicate debris removal options clearly both internally and to the public, leading to confusion among some survivors about their best options for debris removal. In 2018 and 2019, we reported on issues with contracting for wildfire debris removal. We found that USACE’s debris removal contracts, while broad enough to cover any type of debris, had been used primarily to manage hurricane debris removal and did not address issues posed by wildfire debris removal. We also found that miscommunication at the federal level resulted in differing expectations between USACE and state and local officials about debris removal work to be performed, such as the types of structures to be removed from private property and acceptable soil contamination levels. According to USACE officials, they relied on FEMA to manage communication with states and localities and to identify and manage expectations about the scope of work to be performed. We recommended, among other things, that FEMA take the lead to work together with USACE to revise the mission assignment policy and related guidance to better incorporate consideration of contracting needs and to ensure clarity of contracting-related coordination responsibilities. DHS concurred with this recommendation and reported that it will take steps— such as development of mission assignment project management tools and training for mission assignment work—to implement it. According to DHS’s 2017 National Preparedness Report, providing effective and affordable temporary housing for disaster survivors has been a longstanding and continuing challenge. Wildfires pose an additional challenge because in contrast to disasters such as hurricanes or floods where there may be a substantial portion of a home left standing, and property may be habitable after the most dangerous debris is removed, wildfires generally destroy entire structures and leave a pile of contaminated debris and soil. This kind of damage requires a lengthier clean-up and necessarily precludes survivors from occupying the property until state and local officials declare the lot safe for habitation. In the meantime, one of FEMA’s responsibilities under the Mass Care, Emergency Assistance, Temporary Housing and Human Services Emergency Support Function is to help displaced disaster survivors with access to temporary housing. This has posed challenges for some of the counties we spoke with, most notably in select Northern California communities. In particular, officials in two California counties noted that vacancy rates are very low in these areas, and there were few places to house survivors who were either waiting to rebuild on their property or had been living in rental properties that were destroyed. In addition, in one California county there have been a limited number of potential sites available (such as commercial parks or group sites) to place transportable temporary housing units. According to FEMA, several factors limited the number of commercial or group sites available for such housing units, including limited space for the housing units, contaminated utilities, and challenges with local jurisdictions responsible for deciding whether and where to place group sites. According to FEMA officials, the nature of fire debris affects the array of post-disaster housing options that FEMA can offer through its Individual Assistance program. For example, although FEMA can provide replacement assistance for destroyed homes and repair assistance for homes with damage that can be repaired, the complete destruction of homes due to fires significantly lengthens the recovery processes. Rental assistance and lodging reimbursement are limited by lack of access to rental properties, and the use of manufactured housing units is limited by lack of group sites that meet requirements, including adequate space for such units and access to utilities (e.g., potable water not contaminated by fire damage). See Appendix I for more information on this program. FEMA officials acknowledged that providing housing for survivors has long been a challenge for the agency. They also acknowledged that several of FEMA’s housing tools are less relevant to wildfires versus other disasters (as discussed above). According to FEMA, the agency is currently reviewing various aspects of its housing mission to better identify ways to address some of these challenges. Officials from two of the counties we spoke with said that their lack of experience in response to and recovery from wildfires of the magnitudes encountered was very challenging. Officials from one of those counties stated that they did not have the knowledge or skill-set needed at the local level to best identify response and recovery needs and relied heavily on FEMA and California’s Governor’s Office of Emergency Services for resources and training in these areas. Officials from another county stated that neither they nor FEMA were accustomed to the level of destruction in a rural area, which created challenges identifying resources and processes to remove damaged trees from private property, storing the volume of downed trees, and maintaining the few roads available for hauling debris. Officials from another county in California described being unprepared when they were tasked with collecting duplicate payments for private property debris removal after survivors received their insurance benefits. Residents who participated in the private property debris removal program who were paid out of FEMA’s Public Assistance program, and subsequently also received an insurance benefit for debris removal, were required to repay the federal government for the duplicate benefit. According to these county officials, they were not aware that collection would be their responsibility until about 2 years after the initial debris removal took place. The officials noted that the administrative burden for identifying the affected homeowners and the amount owed and then collecting the payments was significant, and taxed their administrative capacity. They said they wished they had been aware sooner that they would have to absorb this duty, so they could put systems in place. According to FEMA and state officials, however, these requirements were included in FEMA’s Public Assistance Program and Policy Guide, which states that local governments are responsible for implementing private property debris removal, including the requirement to collect and reimburse FEMA for any duplicate benefits. Nevertheless, the confusion described by the county government illustrates the difficulty jurisdictional officials with little previous wildfire experience can have navigating complex program rules while simultaneously confronting the disaster aftermath. In June 2019, FEMA Region IX—which provides disaster assistance in California—finalized the after-action report for the October and December 2017 wildfire disasters in Northern and Southern California. FEMA’s 2017 wildfire after-action report offered response and recovery lessons learned from both the challenges identified and successful practices. Some, but not all, of these were mirrored in our interviews with California jurisdictions that were affected by recent wildfires. Among its findings, the 2017 wildfire after-action report identified several areas for improvement. For example, FEMA’s immediate activation of the Transitional Sheltering Assistance program and lack of a unified information system to track applicants’ eligibility for all Individual Assistance programs at the time of the wildfires resulted, in some instances, in applicants receiving sheltering benefits inappropriately (i.e., receiving Transitional Sheltering Assistance benefits despite their residence being undamaged). One potential action to address this challenge identified in the report was to add information on Transitional Sheltering Assistance program applicants into the database that FEMA uses to track disaster information to ensure those applicants have access to all benefits and reduce the potential for duplication. FEMA officials have stated that since the 2017 wildfires, policy changes have been made to address this issue, including adding Transitional Sheltering Assistance program applicant data to the information system used to track eligibility for all Individual Assistance programs. In addition, FEMA reported that the typical contracts USACE had in place for debris removal were not designed to address the nature (i.e., fire- related debris) and scope of work required, particularly with respect to private property debris removal. The agencies worked together to rapidly scope the statements of work for the debris removal contracts to provide services to survivors, but FEMA ultimately found that the contract requirements lacked detail and clarity, resulting in additional costs. USACE prepared its own after-action report after the 2017 wildfires, which also identified challenges with the scope of its debris removal contracts and the mission assignment task orders, and planned to incorporate lessons learned in future debris removal contracts. According to FEMA Region IX officials, many of the issues regarding debris removal stemmed from not having documented processes in place to govern wildfire debris removal specifically. In its after-action report, FEMA identified potential actions to address these challenges—such as developing standard operating procedures in coordination with USACE for fire debris removal—to correct these and other identified areas for improvement. According to USACE officials, FEMA subsequently provided funds through a 2018 wildfire disaster declaration to USACE to develop such standard operating procedures. USACE officials told us they had shared these procedures with FEMA and stated that the procedures will help guide future wildfire private property debris removal operations. The 2017 after-action report also identified a number of strengths and best practices during 2017 wildfire response and recovery efforts in California. For example, the report noted that collaboration and pre- existing relationships between federal and state personnel helped to overcome knowledge gaps about certain programs and improved survivor outcomes (such as the placement of temporary housing units based on work done by an interagency task force). In addition, Facebook provided FEMA with pre- and post-disaster survivor locations (provided voluntarily by the survivor) that helped identify where survivors were located after the wildfires. Using this information, FEMA then worked with the state and private sector in order to help plan for short- and long-term housing solutions. Standards for Internal Control in the Federal Government state that management should identify, analyze, and respond to significant changes that could impact its internal control system, which would include actions established through policies and procedures. Agency management, therefore, should analyze the effect of identified change on policies and procedures and revise such policies and procedures—and other elements of its internal control system—on a timely basis to maintain effectiveness. The combination of back-to-back devastating wildfire seasons in California, overall upward trends in wildfire disaster declarations, and several factors that point to increased likelihood of severe wildfire activity in the future suggest a change that may have significant impacts on FEMA’s operating environment. As shown in figure 6, from 1953 to the present, the number of major disaster declarations from wildfires has increased in nearly every decade since 1950 and most dramatically in the last two decades. During Congressional testimony from March 2018, FEMA’s Region IX Administrator stated that fire season has changed from covering spring through early fall to a now year-round event, and that the unprecedented impacts from the 2017 wildfire season would linger for years to come. Land use practices and climate trends increase the likelihood that severe and intense wildfires will affect people and communities. As we have described in previous reports, land use practices over the past century have reduced forest and rangeland ecosystems’ resilience to fire. Land use practices like fire suppression and timber harvesting have contributed to abnormally dense accumulations of vegetation. These accumulations can fuel uncharacteristically large or severe wildfires. At the same time, development occurring in and around wildlands—an area often called the wildland-urban interface—has increased, placing more people, businesses, and infrastructure at risk. The wildland-urban interface contains 46 million single-family homes, representing about 40 percent of single-family homes in the United States. According to the 2014 Quadrennial Fire Review, 60 percent of new homes built in the United States since 1990 were built in the wildland-urban interface. As the footprint of human activity and settlement into the wildland-urban interface expands, the risk of fire exposure to people and property is expected to increase further. In addition, changing climate conditions, including drier conditions in certain parts of the country, have increased the length and severity of wildfire seasons, according to many scientists and researchers. For example, in the western United States, the average number of days in the fire season increased from approximately 200 in 1980 to approximately 300 in 2013, according to the 2014 Quadrennial Fire Review. According to the U.S. Global Change Research Program’s 2018 National Climate Assessment, warmer and drier conditions have led to a greater incidence of large forest fires (fires with an area greater than 386 square miles) in the western United States and Alaska, a trend expected to continue as climate warms and the fire season gets longer. Despite these trends and projections, FEMA does not plan to comprehensively assess operations to determine whether and how policies and procedures might need to change to better respond to changing operational conditions. According to FEMA officials, they had not considered conducting this kind of review, because they believe their existing mechanisms—specifically after-action reporting, the continuous improvement process, and program specific mechanisms such as the Public Assistance Change Control Tool—will allow them to incorporate relevant lessons into policies and procedures. According to FEMA officials, after a major disaster, FEMA’s standard practice is to identify areas for improvement and develop lessons learned that can improve FEMA planning and policy and support national preparedness by preparing an after-action report which is required by FEMA policy. FEMA has a continuous improvement program which serves as the overarching process by which it identifies and responds to operational lessons learned identified in after-action reporting. According to FEMA officials, FEMA headquarters reviews all completed after-action reports to identify any areas for improvement that may need to be addressed through changes in policies and procedures. Although the continuous improvement process and its reliance on after- action reporting offers the opportunity to incorporate discrete lessons learned into select policies and procedures, there are some limitations in its ability to offer a comprehensive assessment of its internal controls in light of the strong potential that wildfire disasters will continue to increase. By its nature, after-action reporting captures select issues at a specific time and in a specific place, but it is not a dedicated effort to assess how various policies and procedures may need to be changed to better respond to changing operational conditions. For example, in our discussions with fire-affected jurisdictions, we noted that some programmatic or policy challenges were specific to or made more difficult by the nature of wildfires, such as the complexities of debris removal and difficult housing missions. A comprehensive review of internal controls, such as policies, procedures, and training, may shed light on aspects of FEMA’s operations—well tested over the years in hurricane and flood situations—that could be adapted for greater responsiveness to the wildfire environment, helping to ensure attention to a broad range of issues in addition to those that might be noticed in a specific time and place through after-action reporting. In light of the potential for high-impact wildfires to become more frequent, a dedicated effort to comprehensively assess operations could help FEMA better ensure that its management controls—such as policies, procedures, and training—are as well designed as possible to respond to the unique challenges. Devastating wildfires have exacted a large human and financial toll in recent years, with 159 lives lost and $2 billion obligated by FEMA in response during the major disasters of 2017 and 2018. FEMA has provided support personnel and resources to affected state and local jurisdictions to aid in wildfire response and recovery efforts. Given some reports of projected increase in risk from wildfires—as well as the challenges we have noted in providing housing, conducting debris removal operations, and other areas—comprehensively assessing agency operations in response to and recovery from wildfires to determine if any actions or changes to agency policies and procedures are needed could provide guidance or insight for communities that may be affected in the future. Comprehensively identifying, analyzing, and responding to the significant operating changes posed by wildfires, as recommended in internal control standards, could provide FEMA with an opportunity to better ensure the nation is ready to address the unique challenges posed by increased large-scale wildfires. We recommend that the FEMA Administrator comprehensively assess operations to identify any additional updates to its management controls—such as policies, procedures, or training—that could enhance future response and recovery from large-scale and severe wildfires. (Recommendation 1) In August 2019, we requested comments on a draft of this report from the Departments of Agriculture, Defense, Interior, and Homeland Security. The Departments of Agriculture and Defense had no formal or technical comments. In September 2019, FEMA and the Department of the Interior provided technical comments, which we have incorporated as appropriate. In addition, DHS provided an official letter for inclusion in the report, which can be seen in appendix III, stating that it concurred with our recommendation. DHS’s letter describes a number of ongoing and planned actions that it plans to leverage in addressing our recommendation. These actions include, among other things, the use of sheltering and housing field teams to support states’ efforts to house disaster survivors; continued updates to direct housing guidance; developing guidance for the use of FEMA-issued, state-administered direct housing grants authorized by the Disaster Recovery Reform Act of 2018; and development of a project to analyze and improve capabilities and identify areas of innovation in response to wildfire disasters. DHS anticipates that these efforts will be put into effect by December 2020. We will continue to monitor DHS and FEMA’s efforts in addressing our recommendation. We will send copies of the final report to the Secretaries of the departments mentioned above, the FEMA Administrator, and appropriate congressional committees. If you or your staff have any questions about this report, please contact me at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other key contributors to this report are listed in appendix IV. FEMA’s Individual Assistance programs provide assistance directly to individuals and households, as well as state, local, tribal, and territorial governments to support individual survivors. This assistance covers necessary expenditures and serious needs that cannot be met through insurance or low-interest loans, such as temporary housing assistance, counseling, unemployment compensation, or medical expenses. FEMA provides this assistance through seven different programmatic areas, with a substantial amount of the assistance coming from the Individuals and Households Program. The Individuals and Households Program provides financial assistance and direct services to eligible individuals and households who have uninsured or underinsured necessary expenses and serious needs. Individuals and Households Program assistance is intended to meet basic needs and supplement recovery efforts and is not a substitute for insurance. The Individuals and Households Program consists of two forms of assistance: Housing Assistance and Other Needs Assistance. Housing Assistance: Housing assistance may be provided in the form of financial assistance, direct assistance, or a combination of the two. Financial assistance may include lodging expense reimbursement for time spent at hotels or other temporary lodging, rental assistance, and home repair or replacement assistance. Direct housing assistance may be provided when applicants are unable to use rental assistance due to a lack of available housing resources. This type of assistance may include the repair and lease of multi- family housing units—such as apartments—for temporary use by applicants, direct lease assistance, or the provision of transportable temporary housing units, such as recreation vehicles or manufactured housing units. Transportable temporary housing units can be placed on private sites, commercial sites or on group sites. Commercial sites are existing manufactured home sites with available pads that FEMA may lease. Group sites require additional approval when housing needs cannot be met by other direct temporary housing options. They may include publicly-owned land with adequate available utilities. Other Needs Assistance: This consists of financial assistance for other expenses and serious needs caused by the disaster. Some Other Needs Assistance is only provided if an applicant does not qualify for a Small Business Administration disaster loan; this assistance would include personal property, moving and storage, and transportation assistance. Other types of Other Needs Assistance can be provided regardless of SBA loan qualification, including funeral, medical, dental, and child care assistance, and other miscellaneous items. Mass Care and Emergency Assistance This program provides life-sustaining services to disaster survivors immediately before a potential incident, during the response phase, and during the beginning of post-disaster recovery. Services provided include sheltering, feeding, distribution of emergency supplies, support for individuals with disabilities and others with access and functional needs, reunification services for adults and children, support for household pets and service/assistance animals, and mass evacuee support. This program provides supplemental federal financial assistance to states, territories, tribal governments, or private nonprofit entities in order to provide the services of a case manager to a disaster survivor. Through this service, a case manager assists a survivor with developing a disaster recovery plan for meeting his or her unmet needs. Crisis Counseling Assistance and Training Program This program provides supplemental funding to eligible state, territorial, tribal, or local governments, and non-governmental organizations to assist disaster-impacted individuals and communities in recovering from the major disasters through the provision of community-based outreach and psycho-educational services. This program provides legal aid to survivors who qualify as low-income through an agreement with the American Bar Association. The service is limited to cases that would not normally incur legal fees, such as assistance with insurance claims or recovery or reproduction of legal documents lost in the disaster. This program provides unemployment benefits and re-employment assistance services to survivors under the responsibility of the U.S. Department of Labor. This assistance is only available to survivors who are not eligible for regular state unemployment insurance. FEMA employs Voluntary Agency Liaisons who establish and maintain relationships with voluntary agencies active in response and recovery, coordinate with the National Voluntary Organizations Active in Disaster, provide guidance on donations, and act as subject matter experts in development of long term recovery groups with local community organizations, faith-based groups, and other voluntary organizations. FEMA’s Public Assistance program provides supplemental federal disaster grant assistance to state, local, tribal, and territorial governments, and certain types of private nonprofit organizations for debris removal, emergency protective measures, and the restoration of disaster- damaged, publicly-owned facilities and the facilities of certain private nonprofit organizations. The Public Assistance program also encourages protection of these damaged facilities from future events by providing assistance for hazard mitigation measures. The program—which represents the largest share of federal aid from the Disaster Relief Fund—is administered through a partnership between FEMA and the state, tribal or territorial grantee, which provides funding to local or tribal entities who are the subrecipients of a Public Assistance grant award. The Public Assistance program funds both emergency work and permanent work. Public Assistance for Emergency Work FEMA provides funding for emergency work such as emergency protective measures and debris removal that must be conducted immediately to save lives, protect public health and safety, protect improved property, or eliminate or lessen a threat of immediate additional damage. This assistance is divided into two categories, described below. Debris Removal (Category A): Debris removal activities, such as clearance, removal, and disposal, are eligible if the removal is in the public interest based on whether the work eliminates immediate threats to lives, public health, and safety or of significant damage to improved public or private property; ensures economic recovery of the affected community to the benefit of the community at large; or mitigates risk to life and property by removing substantially damaged structures and associated structures. In limited circumstances, based on the severity of the impact of an incident, FEMA may determine that debris removal from private property is eligible under the Public Assistance Program. If debris on private property is so widespread that it threatens public health and safety or the economic recovery of the community, FEMA may provide Public Assistance funding for debris removal from private property. Emergency Protective Measures (Category B): Emergency protective measures conducted before, during, and after an incident are eligible if the measures: eliminate or lessen immediate threats to lives, public health, or safety; or eliminate or lessen immediate threats of significant additional damage to improved public or private property in a cost-effective manner. Examples of such measures include transporting and pre-positioning equipment, flood fighting, supplies and commodities, evacuation and sheltering, child care, security, or searches to locate and recover human remains. Public Assistance for Permanent Work Permanent Work is work required to restore a facility to its pre-disaster design (size and capacity) and function in accordance with applicable codes and standards. This assistance is divided into the five categories listed below: Roads and Bridges (Category C) Water Control Facilities (Category D) Buildings and Equipment (Category E) Utilities (Category F) Parks, Recreational, Other (Category G) Below are details on the six wildfire disasters selected for our review and the support the Federal Emergency Management Agency (FEMA) provided under the major disaster declarations. On September 9, 2015, the Butte Fire began burning across Calaveras County, and on September 12, 2015, the Valley Fire began burning across Lake County. FEMA subsequently approved a Fire Management Assistance Grant (FMAG) for the Butte Fire on September 10, 2015, and an FMAG for the Valley Fire on September 12, 2015. On September 22, 2015, the President issued a major disaster declaration at the request of the state for Lake County, which was ultimately expanded to include Calaveras County. On September 28, 2015, FEMA—in collaboration with the state and counties—opened two Disaster Recovery Centers in Calaveras and Lake Counties, and on October 2, 2015, FEMA opened a third Disaster Recovery Center in Lake County. In total, the Valley and Butte Fires burned 146,935 acres, destroyed 2,876 structures, and resulted in 6 deaths. See figure 7 for a map of the fire locations, and tables 4 and 5 for data on FEMA’s mission assignments, Individual Assistance, and Public Assistance support. On November 28 2016, strong winds pushed a wildfire—named the Chimney Tops 2 fire—beyond the boundaries of the Great Smoky Mountains National Park and into the surrounding wildland urban interface. The fire primarily spread into Sevier County, Tennessee, which includes the cities of Gatlinburg and Pigeon Forge. That same day, FEMA approved an FMAG for Tennessee to support fire suppression activities. On December 15, following a request by the governor of Tennessee on December 9, the President issued a major disaster declaration for Sevier County. On December 23 and December 28, FEMA—in collaboration with the state and counties—opened Disaster Recovery Centers in Gatlinburg and Pigeon Forge, respectively. The Tennessee wildfires ultimately burned approximately 17,000 acres, destroyed 2,545 structures, and led to 14 fatalities. See figure 8 for a map of the fires’ location, tables 6 and 7 for data on FEMA’s mission assignments, Individual Assistance, and Public Assistance support. On October 8, 2017, multiple fires began burning in northern California, spreading rapidly due to high winds and dry conditions. Among these fires was the Tubbs Fire in Sonoma and Napa Counties, which was, at the time, the most destructive fire in California’s history. On October 9, 2017, FEMA approved FMAGs for ten separate fires. On October 10, 2017, the President issued a major disaster declaration at the request of the state for seven counties—Butte, Lake, Mendocino, Napa, Nevada, Sonoma, and Yuba. On October 13, 2017, Solano County and Orange County (in southern California) were added to the declaration. In total, the fires included in this disaster declaration burned 240,138 acres, destroyed 8,924 structures, and resulted in 44 deaths. From October 17 through November 28, FEMA—in in collaboration with the state and counties—established five Disaster Recovery Centers to assist disaster survivors. See figure 9 for a map of the fires’ locations, and tables 8 and 9 for data on FEMA’s mission assignments (including FEMA’s assignment of debris removal responsibilities to the U.S. Army Corps of Engineers), Individual Assistance, and Public Assistance support. Figure 10 provides an aerial snapshot of the destruction in one area of the city of Santa Rosa in Sonoma County. On December 4, 2017, the Thomas Fire started burning in Ventura County. Over the next three days, the Thomas Fire and other wildfires spread rapidly through Ventura and neighboring counties—due in part to the Santa Ana Winds—and FEMA approved a number of FMAGs for these wildfires. On December 20, the governor of California requested a major disaster declaration for Los Angeles, San Diego, Santa Barbara, and Ventura Counties. The request was approved on January 2, 2018 for Santa Barbara and Ventura Counties for Public Assistance. In the week that followed, heavy rains exacerbated the damages caused by the fires, leading to mudflows and debris flows. On January 10, FEMA expanded the disaster declaration to include the flooding, mudflows, and debris flows related to the wildfires. Five days later, FEMA added Los Angeles and San Diego Counties to the disaster declaration, and granted all four counties eligibility for Individual Assistance, in addition to the Public Assistance eligibility previously approved. From January 19 through February 5, 2018, FEMA—in collaboration with the state and counties—established five Disaster Recovery Centers to assist disaster survivors. The Southern California wildfires, debris flows, and mudflows ultimately burned 308,083 acres, destroyed 1,378 structures, and caused 23 fatalities. See figure 11 for a map of the fires’ locations, and tables 10 and 11 for data on FEMA’s mission assignments, Individual Assistance, and Public Assistance support. On July 23, 2018, the Carr Fire began burning in Shasta County. On July 27, 2018, the Mendocino Complex Fire, a combination of the River and Ranch Fires, began burning in Lake County. FEMA soon approved FMAGs for these fires. On August 4, 2018, the President issued a major disaster declaration for Shasta County, which was ultimately expanded to include Lake County. On August 9, 2018, FEMA—in collaboration with the state and counties—established a Disaster Recovery Center in Shasta County, with a second Disaster Recovery Center established in Lake County on August 21, 2018. One of the wildfires—the Mendocino Complex Fire—was the largest fire in California’s history, burning 459,123 acres. In total, the Carr and Mendocino Complex Fires burned 688,774 acres, destroyed 1,894 structures, and resulted in 4 deaths. See figure 12 for a map of the fires’ locations, and tables 12 and 13 for data on FEMA’s mission assignments, Individual Assistance, and Public Assistance support. Figure 13 shows the aftermath of the Carr Fire in one residential neighborhood. On November 8, 2018, the Camp Fire struck the city of Paradise in Butte County. According to California’s Department of Forestry and Fire Protection, the Camp Fire grew into the deadliest and most destructive fire in California history, resulting in 18,793 structures destroyed, 153,336 acres burned, and 85 deaths. On the same day two other major fires—the Woolsey Fire in Los Angeles County and the Hill Fire in Ventura County— began. On November 8-9, FEMA approved FMAGs for these fires, and the President issued a major disaster declaration for these counties on November 12, 2018. FEMA—in collaboration with the state and counties—opened a Disaster Recovery Center in Butte County on November 16 and four other Disaster Recovery Centers in Butte, Ventura, and Los Angeles counties over the next month. In total, the three fires resulted in 20,295 structures destroyed, 254,816 acres burned, and 88 deaths. See figure 14 for a map of the fires’ locations, and tables 14 and 15 for data on FEMA’s mission assignments, Individual Assistance, and Public Assistance support. In addition to the contact above, the following staff members made significant contributions to this report: Kathryn Godfrey (Assistant Director), Adam Couvillion (Analyst-in- Charge), Elizabeth Dretsch, Ricki Gaber, Eric Hauswirth, Hannah Hubbard, Tracey King, John Mingus, Ben Nelson, and Kevin Reeves.", "summary": "In 2017 and 2018, deadly wildfires struck the state of California, tragically resulting in 159 deaths and over 32,000 structures destroyed. FEMA, as the lead federal agency for responding to and recovering from disasters, has obligated about $2 billion in housing, debris removal, and other assistance following these disasters. According to recent environmental assessments, fire seasons are increasing in length, putting more people and infrastructure at risk. GAO was asked to assess a range of response and recovery issues related to the 2017 disasters. Specifically, this report addresses (1) the assistance FEMA provided to jurisdictions in response to major disaster declarations stemming from wildfires from 2015 through 2018, (2) selected jurisdictions' perspectives on FEMA wildfire response and recovery efforts, and (3) the extent to which FEMA has identified and addressed key lessons learned. GAO obtained data on FEMA wildfire disaster assistance and statistics on fire damages and fatalities; reviewed key documentation, such as incident action plans and after action reports; and interviewed officials from FEMA headquarters and regional offices, states, and a nonprobability sample of affected local jurisdictions (e.g., counties). For wildfire-related major disaster declarations from 2015 through 2018, the Federal Emergency Management Agency (FEMA)—consistent with its authorities and responsibilities—helped state and local officials obtain and coordinate federal resources to provide for the needs of wildfire survivors and execute recovery efforts. This support totalled over $2.4 billion and included providing staff to assist at Emergency Operations Centers and establishing Disaster Recovery Centers to coordinate disaster assistance services for survivors. In addition, FEMA provided Public Assistance grant funds to local jurisdictions to help address community infrastructure needs, such as debris removal. FEMA also assigned federal agencies to perform various missions to help with response and recovery—for example, the U.S. Army Corps of Engineers was assigned with contracting for debris removal services in some instances. Officials from jurisdictions that GAO spoke with described practices that aided in wildfire response and recovery, but also reported experiencing challenges. Specifically, officials in affected areas noted that collaboration between FEMA and California's Office of Emergency Services allowed for timely information sharing, and FEMA's assistance at Disaster Recovery Centers greatly assisted survivors in obtaining necessary services. Among the challenges cited were onerous documentation requirements for FEMA's Public Assistance grant program and locating temporary housing for survivors whose homes were completely destroyed. In addition, the unique challenge of removing wildfire debris led to confusion over soil excavation standards and led to overexcavation on some homeowners' lots, lengthening the rebuilding process. FEMA has developed an after-action report identifying lessons learned from the October and December 2017 wildfires, but could benefit from a more comprehensive assessment of its operations to determine if additional actions are needed to ensure that policies and procedures are best suited to prepare for future wildfires. The combination of recent devastating wildfires and projections for increased wildfire activity suggest a potential change in FEMA's operating environment. According to Standards for Internal Control in the Federal Government , such changes should be analyzed and addressed to help ensure that agencies maintain their effectiveness. GAO recommends that FEMA comprehensively assess operations to identify additional updates to policies and procedures that could enhance future wildfire response and recovery efforts. The Department of Homeland Security agreed with our recommendation.", "document_type": "gao"}
{"report": "BIE’s Indian education programs derive from the federal government’s trust responsibility to Indian tribes, a responsibility established in federal statutes, treaties, court decisions, and executive actions. In 2016, the Indian Trust Asset Reform Act included congressional findings stating “through treaties, statutes, and historical relations with Indian tribes, the United States has undertaken a unique trust responsibility to protect and support Indian tribes and Indians...” In addition, “the fiduciary responsibilities of the United States to Indians also are founded in part on specific commitments made in treaties and agreements securing peace, in exchange for which Indians surrendered claims to vast tracts of land…” It is the federal government’s policy to fulfill its trust relationship with and responsibility to the Indian people for the education of Indian children by working with tribes toward the goal of ensuring that Interior- funded schools are of the highest quality and provide for the basic elementary and secondary educational needs of Indian children, including meeting the unique educational and cultural needs of these children. Similar to students in elementary and secondary schools nationwide, some students in BIE schools have documented disabilities that require special educational or supplemental support. More than 6,000 students with disabilities, representing about 15 percent of total enrollment, attend BIE schools. Specific learning disabilities, such as perceptual disabilities, dyslexia, or impairments from brain injury, formed the most prevalent disability category among BIE students with disabilities in school year 2017-2018 (see table 1), affecting more than half of the students with disabilities at BIE schools. An IEP is a written statement for each child with a disability designed to meet the child’s individual needs under IDEA. IDEA requires that every child who receives special education services have an IEP. Before an IEP is developed, a child with a disability must be identified, located, and evaluated through a process known as Child Find. Generally, an adult familiar with the student’s abilities makes an official referral for a special education services evaluation. With parental consent, the student is then evaluated using a variety of assessment tools and strategies designed to help determine the student’s unique needs. Once a child is evaluated and determined to be eligible for special education and related services under IDEA, an IEP is developed describing the school’s delivery of required services to the child. IDEA regulations require that the services specified in a child’s IEP be provided to the child as soon as possible following the development of the IEP. Moreover, IDEA requires that a student’s IEP include, among other things, a projected date for the beginning of services and the anticipated frequency, location, and duration of those services. However, IDEA does not specifically address the steps that schools must take in cases where services are not provided in accordance with the anticipated service duration and frequency in the student’s IEP, such as cases where services were not provided at all or the duration was less than the amount of time specified in a student’s IEP. Educators are required to track the child’s academic progress over the school year and then annually review and update the IEP as needed at least once a year. IDEA requires schools to reevaluate children with IEPs at least once every 3 years to determine whether their needs have changed and if they still qualify for special education and related services under IDEA (see fig. 1). Under IDEA, Interior receives funding to assist in the education of children with disabilities in BIE schools. BIE is responsible for meeting all IDEA requirements for these children, including that an IEP is developed and implemented for each eligible student and that the requirements of any identified education and related services are defined in the IEP. BIE policy requires that IEPs identify services for eligible students under two main categories: education services and related services. Education services include math, reading, and written expression, among others, while related services include occupational therapy, physical therapy, and speech-language pathology, among others, according to BIE’s policy. BIE also requires that IEPs include the type of provider for these services, such as a special education teacher for an education service, or a physical therapist for a related service, as well as information about the duration and frequency of the services to be provided (see fig. 2). BIE schools are required to develop and update students’ IEPs in the Native American Student Information System (NASIS), an online data management system the agency created in 2006 for all BIE schools to record and store a variety of student-related information, including special education data. BIE requires that schools document the special education and related services that their teachers or contracted providers deliver to students with IEPs, and Interior regulations require that schools maintain these and all other special education records for at least 4 years. Multiple offices under the BIE Director are responsible for overseeing and supporting schools’ special education programs to help ensure that they comply with IDEA and other federal requirements for special education (see fig. 3). The School Operations Division was established under the bureau’s recent reorganization to provide direction and assistance to BIE schools in education technology; human resources; communications; educational facilities; safety and facilities; and acquisition and grants. The division is also responsible for providing oversight over BIE school spending, including spending on special education. Sixteen agency field offices called Education Resource Centers are located across the BIE school system and are administered by three separate BIE divisions under the Chief Academic Officer: the Associate Deputy Director-Tribally Controlled Schools, the Associate Deputy Director-Bureau Operated Schools, and the Associate Deputy Director-Navajo Schools. The Centers are primarily responsible for providing oversight and technical assistance to schools in a variety of areas, including their academic programs, fiscal management, and compliance with IDEA. In particular, Interior regulations and BIE procedures require that BIE annually verify that all students with an IEP in the BIE system are provided with special education services in accordance with their IEPs. BIE’s Division of Performance and Accountability (DPA) is primarily responsible for overseeing Education-funded programs at BIE schools, including IDEA and Title I, Part A of the Elementary and Secondary Education Act of 1965, as amended. DPA’s primary oversight responsibilities involve monitoring schools’ implementation of these federal education programs. DPA also provides schools and other BIE offices with technical assistance and training on IDEA requirements, among other program areas. Since 2018, DPA and other BIE divisions have been responsible for working together in monitoring schools the agency considers high risk in administering federal education programs. Specifically, in May 2018, BIE established a new policy and guidance for conducting annual targeted, risk-based monitoring of BIE school programs, which is separate from the requirements for the agency to verify the provision of special education and related services annually. According to this policy, BIE is required to select a sample of 15 schools for this monitoring based on a variety of special education and other risk factors, including special education enrollment and unobligated IDEA funds. BIE’s policy requires that staff from five of its divisions—DPA, School Operations, and the three school divisions responsible for directly supporting BIE schools—coordinate and conduct joint monitoring activities as teams, including a review of schools’ special education programs. These teams are required to issue in depth monitoring reports and technical assistance plans to schools within 30 days of an on-site monitoring visit. Education’s Office of Special Education Programs (OSEP) awards funds to states and BIE, and provides assistance and oversight in their implementation of IDEA. BIE, as with states, is required to report certain compliance information to OSEP. OSEP, in turn, determines BIE’s performance and level of compliance with IDEA and provides assistance to BIE to improve in specific areas. Over the past 8 years, OSEP has found significant problems with BIE’s implementation of IDEA, which in 2019 prompted OSEP to withhold a portion of BIE’s IDEA funds. OSEP issued a determination letter in July 2019 that stated BIE needed intervention in implementing the requirements of IDEA because of its long-standing noncompliance and repeated failure to follow through on OSEP’s required corrective actions, among other issues. BIE had been in “needs intervention” status for each of the last 8 years. As a result of BIE’s continued noncompliance, OSEP in July 2019 withheld 20 percent, or about $780,000, of BIE’s fiscal year 2019 IDEA Part B funds reserved for administrative costs, an action OSEP has taken very infrequently. OSEP provided BIE notice and an opportunity for a hearing, but BIE did not appeal the withheld funds. OSEP’s activities in overseeing BIE’s implementation of IDEA included investigating special education services at one BIE school in 2018. As a result of the investigation, in early August 2018, OSEP sent a letter to the BIE Director about its findings, including that some students at one BIE- operated school had not received services required in their IEPs, including speech language therapy and physical therapy, for almost a year because service contracts with providers had expired. The letter notified BIE that failure to provide services in a student’s IEP violated the IDEA requirement that a free appropriate public education be made available to all eligible students with disabilities. OSEP’s investigation also determined that six other BIE-operated schools were under the same contracts and may not have delivered IEP-required services to students. OSEP’s August 2018 letter required BIE to take several corrective actions within 30 days, including determining whether other schools had IEP service disruptions. In addition, the letter required that BIE develop a plan by the end of October 2018 to prevent contractual problems that could result in a similar disruption of services in the future. As of February 2020, BIE had not notified OSEP that it had completed those corrective actions. OSEP’s oversight of BIE also included visiting BIE schools and agency offices in spring 2019 to examine BIE’s accountability system for IDEA. OSEP presented its findings and corrective actions to BIE in a letter and monitoring report in October 2019. OSEP found that BIE did not have policies and procedures for implementation of IDEA Part B at its schools, and that school officials wanted guidance on IDEA requirements from BIE. OSEP also found evidence of a systemic problem with service providers. For example, officials that OSEP interviewed at one school OSEP visited said they had not had a physical therapist during the entire 2018-2019 school year and did not have a school counselor the previous year. Such staff were required in order to provide services in accordance with students’ IEPs. The corrective actions detailed by OSEP in its October 2019 letter to BIE were to be completed within 90 days, including that BIE develop a plan and timeline for adopting policies and procedures for implementing IDEA. The bureau, however, requested a 60-day extension, which OSEP granted, moving the required date of completion for BIE’s actions to early spring 2020. Our prior work on Indian education found numerous weaknesses in BIE’s management and oversight of BIE schools, including problems with monitoring school spending and conducting annual safety and health inspections of school facilities. As a result of these and other systemic problems with BIE’s administration of Indian education programs, we added Indian education to our High Risk List in February 2017. In our 2019 High Risk update, we found that BIE had made progress in addressing some of these key weaknesses in Indian education, such as demonstrating leadership commitment to change. We reported, however, that the agency needed to show progress in other key areas, such as increasing its capacity to support functions related to administering and overseeing BIE schools. BIE schools did not provide an estimated 20 percent of special education service time to their students during a 4-month period between October 2017 and February 2018, and they did not provide documentation for another 18 percent of service time. Schools frequently did not include reasons for missing services in their service logs, and their practices for whether to make up these services varied. Further, some schools provided no documentation for one or more services, while many schools provided documentation that lacked key information. Difficulties in identifying special education and related service providers, especially in remote areas, limited some schools’ ability to provide services to students. We estimate that BIE schools either did not provide or did not account for 38 percent of the time for the special education and related services required by students’ IEPs, according to our analysis of school documentation. Specifically, we found that schools provided an estimated 62 percent of the service time specified in their students’ IEPs (see fig. 4). Of the service time remaining, we found that schools did not provide an estimated 20 percent of service time to students, and they did not provide any documentation for an additional 18 percent of such service time. When schools did not provide documentation, we were unable to determine whether services were delivered to students. Our analysis was based on a review of service logs at 30 BIE schools during a 4-month period between October 2017 and February 2018 for a nationally representative sample of students with IEPs. Of the students who clearly did not receive service time, according to school service logs, three students at one school received no service time at all during the period of our 4-month review. Officials at the school told us that the special education teacher responsible for providing these services did not fulfill her responsibility to provide services to these students and eventually left the school. They added that the school did not have other qualified staff to provide the services during the period of our review. Our analysis of school service logs found that an estimated one-quarter of the services that were missed did not have a reason listed in the logs, and as a result, we could not determine why the service was not delivered. Of the remaining estimated three-quarters of services that were missed, the top three reasons for missed services were student absences, school-sponsored activities (such as field trips), and provider absences (see fig. 5). BIE requirements do not specify that school service logs must include reasons for missed services. We also found that the schools in our sample did not follow consistent practices for whether to make up regularly scheduled services that are missed. Based on our outreach to officials at the schools in our sample, 23 of the 30 schools that responded varied in their practices for whether to make up services that were missed for reasons including school- sponsored activities or unplanned school closures, such as snow days (see fig. 6). In addition to information about their practices for whether missed special education services are expected to be made up, school officials also provided us with written responses about other factors that may influence this decision. For example, an official at one school responded that while providers of related services are expected to make up missed services when providers are absent, education service providers are not. Alternatively, an official at another school responded that all of the school’s service providers are responsible for finding a way to provide the IEP-required services regardless of the reason for missed service. Additionally, we found that for schools that expect providers to make up missed services, timeframes for doing so varied considerably, based on written responses we received from schools. Specifically, while some school providers typically make up services within a week of the missed service, others aimed to provide them within a month or longer. One school official responded that related services—such as occupational therapy, physical therapy, and speech and language—may not be made up until the following summer, which could potentially result in a delay of up to 9 months if services are missed at the beginning of the school year. BIE does not have official requirements on whether and when schools should make up missed services, and BIE officials provided schools with inconsistent information on this issue. For example, information provided to us by BIE’s Division of Performance and Accountability (DPA) shows that officials advised schools on one occasion that making up missed services is required only when they occur because the provider is not available, but on another occasion advised schools that all missed services should be made up. Further, one official in another BIE office that oversees and supports tribally controlled schools advised schools that making up services is not expected when they are missed due to school-sponsored activities or testing. In contrast, another official with the same division advised schools that services should always be made up regardless of the reason they were missed. While IDEA does not specifically address the steps that schools must take in cases where services are not provided in accordance with the service duration and frequency in the student’s IEP, Education officials said that IDEA does not preclude state educational agencies—including BIE—from establishing their own requirements in this area, as long as they are consistent with IDEA requirements. We found that at least four state educational agencies, including Maryland, New York, North Dakota, and the District of Columbia, have done so. IDEA requires that schools provide special education and related services to eligible students as outlined in their IEPs. However, because BIE schools follow inconsistent practices for whether to make up services when missed, and BIE has not established consistent requirements in this area, there is a risk that some schools may not be providing services in accordance with students’ IEPs. As noted previously, we found that schools did not provide or did not document almost an estimated 40 percent of students’ service time, based on our review of service logs. Missed services may delay students’ progress and increase the risk that they are not receiving a free appropriate public education as required under IDEA. In our generalizable analysis of service logs, we found that for an estimated 18 percent of service time, schools were not able to show whether education and related services were provided to students with IEPs because school service logs were either missing or incomplete. No service logs were provided by schools for 12 of the 138 students in our sample, and incomplete logs were provided for another 51 of the students. By school, 6 of the 30 schools in our sample did not provide any logs for at least one student, and 18 of the remaining 24 schools were missing a portion of the logs. The lack of service logs prevented us from determining whether some students were provided their required IEP services. In addition, we found that many schools’ service logs lacked key information. In particular, service logs frequently omitted or did not clearly indicate service duration and frequency. This information is important for determining whether a school has provided services in accordance with a student’s IEP. Key areas in which service logs varied included: Service duration and frequency: IEPs are required by BIE to specify the weekly frequency and duration of the services throughout the year. However, the service logs we reviewed often did not include both types of information. About one-quarter of the service log entries did not indicate the number of minutes provided, according to our statistical analysis. We estimate that about one-fifth included total minutes, but did not clarify how many times the service was provided. Just over half of the service log entries included both the duration and frequency of each service. Individual vs. combined service entries: Eleven of the 30 schools in our sample provided us with service logs that grouped multiple services together without indicating the specific amount of time or the number of sessions for each service per week. As a result, when these schools recorded that less time was provided, we were unable to identify which of the services were missed. For example, one student was to receive five 60-minute sessions each of reading, written expression, and math per week, according to the student’s IEP. The student’s service log recorded the total number of minutes provided in a day but did not specify which services were provided (e.g., 540 minutes were provided, of a total 900 minutes per week). Based on this information, we could infer from the shortage of total minutes provided that some services were missed, but we were unable to determine whether the student missed reading, written expression, math, or a combination of all three services. School officials responsible for completing service log: Service logs were completed by different types of staff across schools, including paraprofessionals, service providers, or special education coordinators. School practices in documenting special education services varied widely because BIE has not established a standardized process for doing so. BIE officials told us the agency is currently developing a system to standardize how schools document services using a new online module within the Native American Student Information System. Officials provided documentation showing that they were developing this system to allow schools to consistently document both education and related services. BIE’s system, once fully implemented, may allow the agency to monitor and verify service provision more effectively and improve the consistency of schools’ documentation of services. BIE plans to fully implement the system and provide schools with the requisite training to use it by late 2020, according to agency documentation. Officials at 22 of the 30 schools in our sample provided us with information in addition to their service logs, and all 22 schools reported difficulties in recruiting, hiring, or retaining staff or contractors with the required qualifications to provide special education and related services, which some said limited their ability to provide students with high quality required services. In written responses and interviews we conducted, school officials cited school size and remote location as constraints to recruiting, hiring, or retaining qualified service providers. In particular, while schools often rely on contractors to provide related services—such as occupational and physical therapy—officials at 10 of the 30 schools in our sample reported that the availability of qualified contractors was limited. Education services, which are typically provided by school special education staff, were required for nearly all students with IEPs in our sample. Some school officials said in interviews and written responses that in some cases students did not receive education services because their schools either did not have any or enough qualified staff to provide them. For example, according to a BIE official, one BIE school reassigned its only special education teacher to fill a vacant science teacher position and did not provide required services to 18 students during the 2018-19 school year. In another example, one school reported that it did not have qualified staff to provide services to two students with IEPs for 12 consecutive weeks during the 2017-2018 school year. Officials said the school was unable to find a substitute special education teacher, and as a result, each student missed about 5 hours of service time per week during this period. An official at another school said that after advertising for a special education teacher for three years, the position is still vacant. These examples illustrate challenges with hiring and retaining special education staff that may exist more broadly across the country. For example, according to recent Education data, 43 states reported shortages in special education providers in the 2018-2019 school year. However, promising practices may be found within the BIE system as well as across the states that could provide BIE schools direction in addressing shortages of special education providers. For example, two BIE schools recruited and hired special education staff through international work exchange programs meant to facilitate the employment of qualified teachers from other countries. Some schools also reported using outreach to other local BIE or public schools to find and share contractors. Further, OSEP has developed resources for addressing special education teacher shortages that it has made available to states and school districts. In particular, in 2019 OSEP hosted a series of online symposia on general strategies and best practices for schools to attract and retain effective special education personnel. These sessions featured experts and practitioners who discussed strategies for attracting and retaining personnel. Such strategies and other relevant state and tribal resources for addressing special education teacher shortages could provide BIE with additional support to address its own challenges in this area. BIE has not taken steps, however, to establish a mechanism, such as a community of practice, to identify and communicate promising practices for schools, especially those in remote locations, to address their special education staffing and contracting challenges. BIE’s advisory committee on special education stated in its 2018 annual report that BIE needed to better support the recruitment of special education and related service providers at BIE schools. Further, BIE’s 2018-2023 strategic plan has a goal of supporting schools by identifying and sharing best practices and collaborating with schools to recruit, hire, and retain highly effective staff. In addition, federal standards for internal control state that agencies should select an appropriate mechanism for communicating externally. Without greater support from BIE, some schools will continue to struggle to find the special education staff and contractors they need, and students at these schools may not receive the special education services they need to thrive academically. Limited monitoring and technical assistance have hampered BIE’s oversight and support for special education at BIE schools. BIE did not verify the provision of special education and related services at about 30 percent of its schools in school year 2018-2019 due to limited oversight by its largest division. Additionally, BIE has not provided high-risk schools with timely reports after monitoring visits so schools can address their noncompliance with IDEA requirements. Further, staff in BIE’s Education Resource Centers often lack expertise in special education, and school personnel did not always know which agency staff to contact for special education support. BIE did not verify the provision of special education and related services at about 30 percent of its schools in school year 2018-2019, according to available agency documentation. Interior regulations, however, require that BIE annually review all schools’ documentation to verify the provision of special education and related services for every eligible student, among other things. BIE’s guidance for conducting these reviews specifically directs reviewing personnel to verify that students with active IEPs are receiving timely services as indicated on their IEPs. However, the BIE division that oversees about half of all BIE schools, which is led by the Associate Deputy Director-Tribally Controlled Schools, established a policy for its staff to verify provision of services at only a third of its assigned schools each year. The two other divisions, which oversee BIE-operated and Navajo schools, respectively, reported that they conducted reviews at 100 percent of their schools in school year 2018-2019. The Associate Deputy Director-Tribally Controlled Schools who authorized this policy, told us that she believed the policy complied with Interior regulations. However, Interior’s Office of the Solicitor told us that this policy does not comply with Interior’s regulations. BIE officials said the Office of the Associate Deputy Director-Tribally Controlled Schools established this policy to reduce the number of schools the division annually verifies because of the division’s limited staff capacity. Six of 13 staff positions in this division with roles in overseeing or supporting special education were vacant as of February 2020, according to BIE documentation and a senior official. Although BIE developed a strategic workforce plan in 2019 with a goal of addressing staffing shortages across the bureau, the plan does not include information on a strategy or timeframe to address vacancies in positions with responsibilities to oversee and support special education at its schools. BIE’s verification of special education and related services at schools has identified noncompliance with federal requirements. For example, according to BIE, a recent verification visit at one school identified numerous irregularities in its special education documentation, which prompted the school’s superintendent to request that BIE conduct a formal investigation. BIE investigators reported that school staff had falsified service records showing that services were provided when a teacher was not present, and that services were provided in multiple and inappropriate settings (e.g., math services recorded at the same time and date during reading, physical education, and science periods), among other things. As a result, BIE required several corrective actions from the school. As this example illustrates, the verification process provides BIE with an important oversight mechanism. This mechanism, however, is not being fully utilized by BIE’s largest school division. Without BIE annually reviewing documentation to verify the provision of special education for every student at all schools, the agency cannot ensure that students are receiving the services required by their IEPs. BIE monitored 14 schools for high-risk monitoring in school year 2018- 2019, but did not provide the schools with timely monitoring reports and technical assistance plans for their compliance with IDEA and other federal education program requirements. In addition to its annual process of verifying that students with IEPs are receiving required special education and related services, BIE also conducts targeted oversight of schools it deems high risk. BIE’s high-risk monitoring policy, established in May 2018, requires that it select a sample of schools based on risk indicators related to IDEA and other federal education programs, and provide schools with in-depth monitoring of their special education and other education programs. Nine of the 15 schools selected for BIE’s 2018-2019 high-risk monitoring were selected because BIE considered them to be at a higher risk in administering special education. The factors that BIE considered included a large enrollment of students with IEPs and a significant amount of unobligated IDEA funds, among other factors. One school, for example, had not obligated about 50 percent of its IDEA funds within the timeframe required by IDEA. BIE’s monitoring policy requires that it provide both monitoring reports and technical assistance plans to schools within 30 days of a visit. However, BIE sent schools visited in the 2018-2019 school year their monitoring reports in late August 2019—well after its required 30-day timeframe and several weeks after we requested the reports as part of this review. For example, BIE sent two school reports more than 8 months after its monitoring visits, and another two school reports more than 6 months after visits (see fig. 7). Further, the reports sent to schools were not accompanied by technical assistance plans, as required by BIE policy, which are required to outline how BIE will assist schools in addressing findings of noncompliance. BIE officials said that a timeframe for when the plans would be developed and issued to schools had not been established. BIE officials told us the late monitoring reports and the lack of technical assistance plans for schools resulted from BIE not fully implementing its 2018 high-risk monitoring policy. Officials said the monitoring policy requires monitoring teams to be comprised of staff from five BIE divisions: DPA, School Operations, and the three divisions responsible for directly supporting BIE schools. These staff work together to monitor special education and other school programs and develop reports and technical assistance plans for schools. However, BIE officials said that four of these divisions did not contribute staff to lead the monitoring teams, leaving the task of developing monitoring reports to a single division—DPA. DPA officials told us that developing such plans requires the knowledge, expertise, and coordination of staff across all five BIE divisions. They said that without participation from the other divisions, it is unlikely the plans will be developed and sent to schools because DPA itself does not have the staff capacity to do so. BIE officials told us they were aware of problems with coordination on high-risk monitoring across the five divisions and were considering how to make improvements, but did not provide a timeframe for doing so. BIE’s monitoring reports and technical assistance plans are intended to provide high-risk schools with important information about their compliance with IDEA and other federal education funding programs, according to agency documentation. Each of BIE’s monitoring reports for the 14 schools in 2018-2019 included multiple findings of school noncompliance with special education requirements under IDEA or Interior regulations. Specifically, monitoring reports for several schools included findings related to their provision of special education services. For example, one report found that a school maintained no service logs and was not able to demonstrate it had provided any services to students. Without timely monitoring reports, schools lack vital information to address areas of noncompliance, including ensuring that staff and contractors provide and document special education services as required. Further, without the technical assistance plans that BIE policy states are to accompany monitoring reports, schools may not know what BIE resources are available to them for addressing specific special education compliance issues. Staff in BIE’s Education Resource Centers often do not have sufficient expertise on special education to provide appropriate oversight and technical assistance to schools, according to BIE officials. Staff in Education Resource Centers have special education-related responsibilities that include annually verifying that schools are providing special education services and assisting schools when compliance issues with federal special education requirements are identified or when schools request help. Several BIE officials, however, told us these staff often do not have the requisite knowledge about special education to effectively carry out these responsibilities. For example, two senior BIE officials said these staff do not consistently have the expertise required to review documentation on service provision. A staff member at one Education Resource Center said she and her colleagues often do not know what questions to ask school officials during site visits to verify their provision of special education services. Additionally, several officials told us that these staff often do not have the expertise to provide technical assistance to schools on special education. One official said these staff often provide incorrect information to schools because of their lack of expertise. Officials from two schools also told us that some Education Resource Center staff with special education responsibilities do not have sufficient expertise to oversee and assist them with their special education programs. Several BIE officials said Education Resource Center staff need additional training in special education to more effectively carry out their responsibilities. Federal standards for internal control state that agencies should develop staff competencies—including knowledge, skills, and abilities—to achieve agency objectives. However, BIE has not ensured that Education Resource Center staff have the requisite competencies to oversee and support schools’ special education programs because it has not established special education training requirements. Without establishing such requirements and ensuring they are met, staff may not be effective in overseeing and assisting schools with their special education programs, including ensuring that students with IEPs receive required services. School officials said they did not always know which BIE staff to contact for support with their special education programs. Staff in BIE’s Education Resource Centers are responsible for regular outreach to schools about these programs, according to two senior BIE officials. However, officials we interviewed from some schools expressed confusion about the roles and responsibilities of various BIE offices and staff responsible for special education or said there has been a lack of outreach from Education Resource Center staff. For example, the special education coordinator at one tribally controlled school said she had received no information about which Education Resource Center was responsible for supporting her school. Several BIE officials acknowledged that schools do not always know which Education Resource Centers are responsible for supporting them. One senior BIE official also said that some schools are not aware that they can reach out to BIE for assistance with their special education programs. BIE’s 2015 Communications Plan prioritizes regular communication with schools to provide key information and important developments affecting their schools. However, BIE officials said Education Resource Center staff do not consistently reach out to inform schools about how they can support schools’ special education programs. Additionally, as part of its recent reorganization, BIE shifted the roles and responsibilities of many offices and staff, including those responsible for supporting special education at schools. Without BIE taking steps to ensure its Education Resource Center staff communicate with all schools regarding their roles and responsibilities on special education, these staff may not consistently do so. As a result, schools may not know whom to contact for answers to questions, which could hinder their ability to provide effective special education services to students. The purpose of IDEA is to fulfill the promise that all children with disabilities have available to them special education and related services designed to meet their unique educational needs. In exchange for the funds it receives from Education to implement IDEA, BIE must ensure that such an education is available to all of its students with disabilities. The potential for students with disabilities at BIE schools to advance academically depends, in part, on the ability of BIE to oversee and support schools in providing these students with the special education and related services required by their IEPs under IDEA. It is unclear, however, whether all BIE schools are meeting these students’ needs and ensuring that required services are consistently delivered because schools follow different practices for determining whether to make up services for students when they are missed. Further, the challenges that schools face in obtaining qualified special education staff and specialists to provide services—which may also exist for public schools nationwide— also present BIE with an important opportunity to partner with knowledgeable stakeholders and provide direction in this area. BIE also needs to address persistent administrative capacity issues in special education—such as vacancies and a need for training in key agency offices. In addition, BIE should ensure that relevant offices are reaching out to schools to inform them of their roles in overseeing and supporting schools’ special education programs. Finally, BIE must take steps to make sure its offices annually review school documentation to verify that students are receiving special education and related services and provide high-risk schools selected for targeted monitoring with timely reports and technical assistance plans. In addition to IDEA’s requirement that special education services be provided to all eligible students with disabilities, BIE also has a responsibility to work towards the goal of ensuring that BIE schools are of the highest quality and provide for their students’ unique educational needs. Without taking steps to address weaknesses in key areas of special education, BIE cannot ensure that the schools it funds are meeting their responsibilities under IDEA or addressing the unique needs of more than 6,000 BIE students with disabilities. We are making the following seven recommendations to BIE: The Director of BIE should establish consistent requirements for schools on making up missed special education and related services and monitor schools to ensure that they follow these requirements. (Recommendation 1) The Director of BIE should work with knowledgeable stakeholders in Indian education to establish a community of practice or other formal mechanism to identify and disseminate promising practices for schools— especially those in remote locations—on recruiting, hiring, and retaining special education teachers and contracting with providers. The Director of BIE could consider conferring with BIE’s special education advisory committee, OSEP, and relevant tribal and state education officials in addressing this recommendation. (Recommendation 2) The Director of BIE should rescind the policy of its division overseeing tribally controlled schools that does not meet Interior’s requirement to annually review all schools’ documentation to verify the provision of services for every special education student, and ensure that all divisions comply with this requirement. (Recommendation 3) The Director of BIE should update the agency’s workforce plan to include a strategy and timeframe for filling vacant staff positions responsible for overseeing and supporting schools’ special education programs. (Recommendation 4) The Director of BIE should fully implement the agency’s high-risk monitoring policy for IDEA and other federal education programs, including requirements for agency-wide coordination, and ensure that schools selected for such monitoring receive reports and technical assistance plans within 30 days of agency on-site visits, as required by BIE policy. (Recommendation 5) The Director of BIE should establish special education training requirements for staff in the agency’s Education Resource Centers who are responsible for supporting and overseeing schools’ special education programs, and ensure that staff complete those training requirements. (Recommendation 6) The Director of BIE should take steps to ensure that all of the agency’s Education Resource Centers conduct outreach with schools to inform them of their new roles in overseeing and supporting schools’ special education programs under BIE’s reorganization. (Recommendation 7) We provided a draft of this report to the Departments of the Interior (Interior) and Education (Education) for review and comment. Interior provided formal comments, which are reproduced in appendix II, agreeing with all seven recommendations and describing actions BIE plans to take to address them. Education provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretaries of the Interior and Education and interested congressional committees. The report will also be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0534 or emreyarrasm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Our report examines (1) the extent to which eligible Bureau of Indian Education (BIE) students with disabilities are provided the special education and related services required by their individualized education programs (IEP); and (2) the extent to which BIE oversees and supports the provision of these services at its schools. To obtain a generalizable sample of students, we defined our target population as all students at BIE schools with an active IEP covering a full 5-month period between September 2017 and February 2018 and obtained an electronic listing of IEPs for the 2017-2018 school year—the most recent complete school year at the time of our analysis—extracted from the Native American Student Information System (NASIS). We used these data as a basis to define a sample frame and identified 2,904 unique students with an active IEP for the full period from 169 BIE schools. We assessed the reliability of these data by interviewing knowledgeable agency officials and reviewing technical documentation describing the methodology, assumptions, and inputs used to produce the IEP-related data we received from BIE, upon which we created our generalizable sample. We determined these data to be sufficiently reliable for the purposes of our report. We selected a random two-stage cluster sample of 30 BIE schools and 150 students (about 5 per school) who had at least one active IEP covering the full period from the sample frame of 169 schools and 2,904 students. We chose to use a two-stage sampling approach to control (limit) the number of schools that we would need to coordinate with to collect the school-level data required. Because the number of unique in- scope students ranged between 2 and 88 per school, we chose to select schools with probability proportional to size. We computed the target sample sizes of 30 schools and 150 students (about 5 per school) using estimated standard errors of student age that accounted for the additional variance resulting from the complex sampling approach (two-stage cluster sample) for various sample sizes. We then compared the change in standard errors for various sample sizes of schools and students to those from a simple random sample of size 150. Based on these results, we observed that the decrease in standard errors began to level out at a sample size of 30 schools (n=30) and that selecting more than 5 students (m=5) per school would not significantly decrease the standard errors. To estimate the likely margin of error we expected to achieve from this sample, we conducted a simulation of 10,000 samples of 30 schools and 150 students and examined the distribution of outcomes from these results for 3 proportion estimates. The proportion estimates were designed to provide a range of variance outcomes. Based on this simulation of possible results, we expected this sample design to generate percentage estimates to the sample frame (full population) of students with an overall precision of about plus or minus 12 percentage points or fewer. During our review we learned that one school selected in our sample was under a BIE internal investigation into irregularities in the school’s special education documentation. As a result, we removed the five students at this school from the sample. We added an additional randomly selected school as a replacement. As a result, we completed our analysis for 30 of the 31 schools that we sampled. Additionally, we found that a number of students selected within schools were out of the scope of our defined target population, such as when a student transferred to another school during our review period. When possible, we selected additional cases to account for the out-of-scope students. The final sample included 138 students at 30 schools. Based on the final sample of students, we completed our analysis for 96.5 percent of the students that we sampled that were within the scope of our defined target population. We defined the primary unit of analysis as the student and generated estimates at the student level summarized across 17 of the 18 weeks in the time period of our analysis (between October 2, 2017, and February 2, 2018). We chose not to include data collected for the school week from December 25, 2017, through December 29, 2017 because most schools either did not provide services during this week or were closed. We collected and analyzed the data for students’ scheduled services on a weekly basis. The data collection at this level resulted in multiple, repeated observations for each student. For the purposes of generating weighted, generalizable estimates, these data were summarized at the student level for each service type. The sampling weights were computed at the student level so that estimates from this sample will be made to the population of students. The student weight, which is the inverse of the probability of selection, was computed by combining a stage 1 (school) weight and stage 2 (student within selected schools) weight that each accounted for the probability of selection at each stage. The final student weights varied slightly from school to school based on the number of students selected within each school. The final student weights ranged from 16.13 to 24.20, and most were 19.36. We conducted a test run of our document collection and analysis methodology at one BIE-funded school to determine the feasibility of collecting and analyzing school service logs in electronic form. Based on the successful results of the test run, we concluded that this methodology would allow for the collection and analysis of service logs from our sample of schools. We then requested electronic copies of IEPs and any applicable IEP amendments from BIE for the students in our sample. We followed up with BIE on any issues of unclear or missing IEP documentation. After compiling IEPs for the students in our sample, we requested service logs from our sample schools and requested confirmation of key information in students’ IEPs (e.g., the type, duration, and frequency of services for our review period). To generate a data set based on schools’ service logs, we coded, by week, information contained in all service logs using a coding scheme that specified type of service (i.e., education vs. related), frequency of services received, duration of services received, and reasons for missed services. To determine the baseline of minutes and frequency for each service, we calculated the duration and frequency of services required in student IEPs and removed service duration and frequency on days that schools were not in session according to school calendars. In cases in which schools did not provide us with service logs for part or all of our review period, we were not able to determine whether the services were received. In such cases, we recorded these minutes in a separate category, labeled “service time not accounted for.” In a small number of instances, schools recorded service log entries, but unclear notation prevented us from being able to determine whether the service was provided. This accounted for less than half of a percent of service time. Because the information contained in school service logs is self-reported by school personnel or service contractors, we were not able to assess the overall accuracy of this information, such as whether services were actually provided—a limitation that generally applies to research relying on self-reported information. We conducted extensive follow-up with schools, however, to ensure the most complete data collection possible and contacted them when further information or clarification was needed to understand service log entries. Additionally, we obtained student attendance data from BIE to compare with entries in service logs from four schools. As the result of this comparison, we removed one student from our sample whose attendance data showed significantly higher absences than were reflected in school service logs. In many cases, we received service logs that did not convey complete information about some aspects of service provision. For example, some logs used non-numerical notation to show that services were provided, such as checkmarks. In these cases, we assumed that a checkmark indicated that one full service was provided and recorded the number of minutes in a typical service. Additionally, some service logs combined multiple services (e.g., 60 minutes of math, 30 minutes of reading, and 30 minutes of writing) into one log and recorded the total number of minutes that services were provided within a week. As we could not determine which services were expected on which days within a week, we adjusted minutes and frequency for combined services when schools were not in session by prorating the weekly totals accordingly. To collect information on reasons for missed services, we categorized recorded reasons into the following groups: (1) student absence; (2) student disciplinary action; (3) provider absence; (4) provider administrative duties; (5) unplanned school closure; (6) school-sponsored activities; (7) testing; and (8) reason not provided. We recorded missing service logs as a separate category (“service time not accounted for”) and did not include them in our analysis of reasons for missed services. Estimates from this sample are generalizable to the estimated in-scope population of about 2,600 (+/- 130) students with at least one active IEP covering the period from September 1, 2017, through February 1, 2018. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval. This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. All estimates in this report have a confidence interval with a margin of error of plus or minus 12 percentage points or fewer, unless otherwise noted. In addition to the generalizable data we collected on schools’ special education service provision, we asked school officials to respond to an optional set of questions on the challenges schools face, if any, in providing services. Eighteen of the 30 schools in our sample provided responses. Of the schools who did not respond, we obtained information on challenges with service provision from four additional schools during our site visits, which are described below. Together, we obtained perspectives about the challenges schools face in special education service provision from a total of 22 of the schools in our sample. We also requested information from schools about the circumstances under which providers are expected to make up missed special education services, and the timeframe in which these make-up services are expected. Twenty-three of the 30 schools in our sample provided a response. To help inform both of our research objectives, gather additional information about schools’ special education programs, and explore issues related to their provision of special education and related services, we conducted site visits to seven schools in our sample located in New Mexico (4 sites) and Arizona (3 sites), selected for their large numbers of BIE-funded schools. Our criteria for selecting schools included special education student enrollment size, whether a school was operated by BIE or a tribe, and tribal affiliation. At each site, we gathered information from participants—including school administrators and teachers—using semi-structured interview questions. We collected information on school staff’s roles and responsibilities in administering and overseeing special education; policies, practices, and any challenges to providing and documenting special education and related services; and perspectives on guidance and support, if any, from relevant BIE offices. Our site visits also included meetings with BIE officials in Albuquerque, New Mexico, and Window Rock, Arizona. Our interviews with officials focused on their roles and responsibilities in overseeing and supporting schools’ special education programs; staff capacity; intra-agency coordination on special education; policies and procedures related to special education monitoring; and their views on factors, if any, that may affect schools’ ability to provide special education and related services to students with IEPs. To inform both research objectives, we also interviewed officials in several BIE offices with responsibilities for overseeing and supporting schools’ special education programs, including: the Office of the Director; the Division of Performance and Accountability; the Office of the Associate Deputy Director-Tribally Controlled Schools; the Office of the Associate Deputy Director-Bureau Operated Schools; and the Office of the Associate Deputy Director-Navajo Schools. Our interviews with agency officials focused on their roles and responsibilities in overseeing and supporting schools’ special education programs; staff capacity; intra- agency coordination on special education; policies and procedures related to special education monitoring; and their views on factors, if any, that may affect schools’ ability to provide special education and related services to students with IEPs. We compared BIE’s oversight and technical assistance activities against requirements under IDEA and Department of the Interior (Interior) regulations, BIE policies and procedures, and federal standards for internal control to evaluate the sufficiency of their efforts in monitoring and supporting BIE schools’ special education programs. We also conferred with Interior’s Office of the Solicitor regarding their position on whether one BIE division’s policy for reviewing special education documentation at schools conformed to Interior’s regulations. Additionally, we interviewed current and former members of BIE’s advisory committee on special education to obtain their views on the extent to which BIE schools provide required services to students with IEPs and challenges, if any, that schools may face in delivering services. We also interviewed national groups with expertise on Indian education and BIE schools—including the National Congress of American Indians, the National Indian Education Association, and the Tribal Education Departments National Assembly—to obtain their views on special education and related services at BIE schools. Our review of relevant documentation included BIE’s monitoring and technical assistance policies and procedures as well as relevant federal laws and regulations, including requirements under IDEA Part B. This included BIE’s May 2018 policy and procedures on conducting high-risk monitoring of the implementation of federal education programs at BIE schools. In addition, we reviewed the Department of Education’s determination letters and October 2019 monitoring report to BIE assessing the agency’s compliance with IDEA requirements. We conducted this performance audit from July 2018 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Melissa Emrey-Arras, (617) 788-0534 or emreyarrasm@gao.gov In addition to the contact named above, Elizabeth Sirois (Assistant Director), Edward Bodine (Analyst-in-Charge), Liam O’Laughlin, and Angeline Bickner made key contributions to this report. James Ashley, Susan Aschoff, Serena Lo, John Yee, James Rebbe, Sam Portnow, Aaron Karty, James Bennett, Avani Locke, and Olivia Lopez also contributed to this report. Tribal Programs: Resource Constraints and Management Weaknesses Can Limit Federal Program Delivery to Tribes. GAO-20-270T. Washington, D.C.: Nov 19, 2019. High Risk: Progress Made but Continued Attention Needed to Address Management Weaknesses at Federal Agencies Serving Indian Tribes. GAO-19-445T. Washington, D.C.: March 12, 2019. High Risk: Agencies Need to Continue Efforts to Address Management Weaknesses of Federal Programs Serving Indian Tribes. GAO-18-616T. Washington, D.C.: June 13, 2018. Indian Affairs: Further Actions Needed to Improve Oversight and Accountability for School Safety Inspections. GAO-17-421. Washington, D.C.: May 24, 2017. Indian Affairs: Actions Needed to Better Manage Indian School Construction Projects. GAO-17-447. Washington, D.C.: May 24, 2017. Tribal Transportation: Better Data Could Improve Road Management and Inform Indian Student Attendance Strategies. GAO-17-423. Washington, D.C.: May 22, 2017. Indian Affairs: Key Actions Needed to Ensure Safety and Health at Indian School Facilities. GAO-16-313. Washington, D.C.: March 10, 2016. Indian Affairs: Preliminary Results Show Continued Challenges to the Oversight and Support of Education Facilities. GAO-15-389T. Washington, D.C.: February 27, 2015. Indian Affairs: Bureau of Indian Education Needs to Improve Oversight of School Spending. GAO-15-121. Washington, D.C.: November 13, 2014. Indian Affairs: Better Management and Accountability Needed to Improve Indian Education. GAO-13-774. Washington, D.C.: September 24, 2013.", "summary": "BIE funds 185 elementary and secondary schools that serve more than 6,000 Native American students with special needs. The Department of Education has raised concerns about BIE's implementation of IDEA in recent years, including its long-standing noncompliance with IDEA requirements. GAO was asked to examine the provision of special education and related services to eligible BIE students. This report examines the extent to which (1) BIE students with disabilities are provided the special education and related services required by their IEPs, and (2) BIE oversees and supports the provision of special education at its schools. GAO analyzed data on special education and related services for a generalizable sample of 138 BIE students with IEPs at 30 schools over a 4-month period in school year 2017-2018 (the most recent complete school year at the time of our analysis); compared BIE special education practices with its policies and Interior and IDEA requirements; visited schools in two states selected for their large numbers of BIE schools; and interviewed school and agency officials. Schools funded by the Bureau of Indian Education (BIE) are required under the Individuals with Disabilities Education Act (IDEA) to provide services for eligible students with disabilities, such as learning disabilities or health impairments. Services for these students are listed in individualized education programs (IEP). GAO found that BIE schools did not provide or did not account for 38 percent of special education and related service time for students with disabilities, according to analysis of school documentation for a 4-month review period (see fig.). This included one school that did not provide any services to three students. While BIE has plans to improve documentation of such services, it has not established whether and when missed services should be made up, which has led to inconsistent practices among schools. Establishing consistent requirements for making up missed services could help students receive the special education and related services they need to make academic progress. BIE's limited monitoring and technical assistance have hindered its oversight and support for special education at schools. For example: A division of BIE responsible for overseeing about half of all BIE schools decided to verify the provision of special education services at only one-third of its schools per year, although the Department of the Interior (Interior) requires BIE to annually verify the provision of services at all schools. BIE provided required monitoring reports late and did not provide required technical assistance plans to 14 schools that BIE determined were at high risk of not complying with IDEA and other federal education programs in school year 2018-2019. BIE officials said that the field office staff responsible for working with schools on special education often do not have the requisite expertise, which has hampered their oversight and support to schools. Without verifying special education services at every school annually, following high-risk monitoring and technical assistance requirements, and providing training to its staff, BIE cannot ensure that the schools it funds are meeting their responsibilities under IDEA. Strengthening such oversight and support activities can help BIE as it works to address the unique needs of students with disabilities to help prepare them for future education, employment, and independent living. GAO is making seven recommendations, including that BIE establish consistent requirements for schools on making up missed services, annually verify special education services at all schools, comply with high-risk monitoring and technical assistance requirements, and ensure that BIE staff receive needed training. Interior agreed with the recommendations.", "document_type": "gao"}
{"report": "According to FDA documents and officials, import alerts serve several purposes, including the following: Prevent products that appear to violate FFDCA from being distributed in the United States. Free up agency resources to examine other shipments by automatically detaining shipments on import alerts on a case-by-case basis without examining them. Place the responsibility on the importer to ensure that the products being imported into the United States comply with federal laws and FDA regulations. Import alerts may apply to (1) one or more products produced by all firms in a specific geographic area, (2) one or more products produced or shipped by a specific firm, or (3) a specific product because of concerns about the product regardless of what firm produces it or where. Import alerts covering a specific geographical area may apply to an area within a country, to one or more entire countries, or worldwide. For example, FDA established an import alert covering all firms processing shrimp in India because of the presence of filth, decomposition, and Salmonella. For import alerts that apply to geographical areas, all firms in the area that produce the products specified in an import alert are initially placed on that alert, and the specified products are subject to detention without physical examination. If a firm presents evidence establishing that the conditions that gave rise to the appearance of the violation associated with the alert have been resolved and the agency has confidence that future entries will comply with FFDCA, FDA indicates that the firm may be removed from the alert by placing it on a “green list” that FDA creates for the alert. For import alerts that apply to products that a specific firm produces, FDA individually determines—for example, through testing or examination—whether a firm and its products are potentially violative and may be identified for potential detention without physical examination. If so, FDA places them on a “red list” that it creates for the alert. Import alerts that apply to a specific product of concern generally have neither a red list nor a green list because such products cannot be removed from the alerts. Products detained via import alerts may be (1) refused entry, in which case they must be exported to another country or destroyed, or (2) allowed to enter U.S. commerce if they can be shown to not violate FFDCA or can be reconditioned to be brought into compliance with the act. DHS, through CBP, is charged with facilitating international trade at the ports-of-entry for seafood and other imports, while FDA examines or inspects certain seafood imports. CBP is responsible for, among other things, collecting the duties, taxes, and fees assessed on products, including seafood, and managing the import process. CBP collects import entry data through its Automated Commercial Environment/International Trade Data System. These entry data are submitted by a filer (typically, the product importer or a broker) and include a description of the product, manufacturer information, and the country of origin. Generally, FDA electronically receives notification from CBP of all entries of products under FDA jurisdiction at ports of entry through the CBP system described above, which links to FDA’s OASIS. Once entry information is received in OASIS, FDA uses its Predictive Risk-based Evaluation for Dynamic Import Compliance Targeting (PREDICT) screening tool to evaluate each entry line. PREDICT is a computerized tool designed to estimate the risk of imports using information such as the history of the importer or processing facility, inspection history, and country of origin. FDA staff use these risk estimates to target for examination shipments with high levels of risk. FDA cannot physically examine every shipment of such products, owing in part to the volume of imported products; we previously reported that the agency examines about 1 percent of entry lines annually. FDA uses PREDICT to electronically screen all imported food shipment information filed electronically to determine which imports to physically examine at the border. PREDICT uses a variety of data and analyzes data by applying rules—conditional statements that tell PREDICT how to react when encountering particular information—to generate risk scores for imported food. The electronic screening process consists of two phases: Prior notice screening is intended to protect against potential terrorist acts and other public health emergencies. Prior notice screening requires that an importer, broker, or other entity submit information to FDA on food being imported or offered for import into the United States before that food arrives at the port of entry. FDA targets, screens, and reviews the information to ensure that the information meets the prior notice requirements and to determine whether the food potentially poses a terrorism threat or other significant health risk. Admissibility screening is intended to ensure that the food is admissible under FFDCA. As part of admissibility screening, FDA electronically screens entry lines using PREDICT to determine, among other things, whether the product on the entry line is on an import alert. If the product on an entry line is on an import alert, then the entry line may be detained without physical examination. If the product is not on an import alert, then the entry line goes through the typical admissibility screening process through which FDA uses PREDICT to calculate a risk score and determine whether the entry line is identified for potential examination or sampling. Our review of FDA’s Regulatory Procedures Manual found that FDA’s import alert process for seafood products includes three key components: (1) establishing new import alerts to respond to human health risks, (2) placing firms and products on new or existing import alerts (placement decisions), and (3) removing firms and products from existing import alerts when violations are resolved (removal decisions). According to FDA’s Regulatory Procedures Manual, FDA establishes new seafood import alerts to respond to human health hazards. FDA officials may recommend new import alerts for a variety of reasons, including the following: FDA officials detain one or more products for a violation of FFDCA that poses a significant health hazard (e.g., the presence of Salmonella); FDA officials notice a large number of violations affecting firms or products from a specific country or area (e.g., the presence of filth in canned crabmeat from Thailand); FDA enforces regulatory requirements affecting importers that the agency decides could be implemented, in part, through the use of an import alert (e.g., HACCP requirements); or FDA addresses concerns about the safety of specific products, including puffer fish, which contain a deadly neurotoxin, or products produced in geographic areas with known contamination, such as those from areas surrounding Fukushima, Japan, which are at risk of radionuclide contamination. FDA officials in the field or at headquarters may recommend new import alerts. FDA’s Division of Import Operations reviews the recommendations and decides whether to approve them (called the clearance process). After approval, according to FDA officials, FDA revises its screening process at the ports of entry via PREDICT to screen for products, firms, or countries on the new alert. According to FDA’s import alert data, as of July 3, 2018, FDA had 52 active import alerts affecting imported seafood that addressed a wide range of seafood products and violations of FFDCA. The range of violations that these alerts address included the presence of foodborne pathogens, such as Salmonella and E. coli; the presence of unapproved animal drug residues, such as chloramphenicol and nitrofurans; the presence of pesticide chemical residues that are not allowed or do not meet tolerance levels, such as diuron; the presence of decomposition or insect, rodent, or other filth; the presence of illegal or undeclared colors, undeclared food additives, such as high fructose corn syrup, or undeclared food allergens, such as milk; the failure of the firm to meet HACCP requirements; and the failure of the firm to operate in conformity with current good manufacturing practices. According to FDA’s import alert data, overall, from October 1, 2011, through July 3, 2018, the 52 import alerts for imported seafood affected a total of 3,765 unique firms in 111 countries. (See app. I for information describing these 52 alerts.) According to FDA’s Regulatory Procedures Manual, after an import alert has been established, FDA places certain seafood firms or products on the alert and may detain affected products at the port of entry to prevent them from entering U.S. commerce pending the importer of record’s response. The manual specifies that FDA may place firms or products on a new or existing import alert for the following violations of FFDCA: (1) products are manufactured, processed, or packed under insanitary conditions; (2) products are forbidden or restricted for sale in the country in which they were produced or from which they were exported; or (3) products appear to be adulterated or misbranded based on information such as the product’s history of violations, among other things. Examples of adulteration may include pathogens, such as Salmonella, and residues of drugs or pesticides above accepted levels. FDA’s Regulatory Procedures Manual also specifies the following types of evidence that FDA generally may rely on to show that violative conditions exist: one violative sample from FDA’s examination of the product, if the product may have adverse health consequences; information and historical data, such as a firm showing a pattern of exporting violative products, if evidence indicates the product could pose a health hazard; multiple violative samples, for violations (such as decomposition, filth, or labeling) that do not pose a significant public health hazard; and violations identified during inspections of importers or foreign processing facilities. According to FDA officials, about 90 percent of the recommendations to place firms or products on an import alert result from FDA analysis of imported seafood samples that identified product violations, such as drug residues above acceptable levels. Officials stated that the remaining 10 percent of the recommendations arise from FDA inspections of importers or processing facilities that identify firm violations, such as violations of FFDCA related to HACCP requirements. According to FDA’s Regulatory Procedures Manual, once a firm or product has been placed on an import alert, future shipments may be detained without physical examination, and the importer of record must decide how to respond. The importer of record receives a notice stating that the associated entry line is being detained and subject to refusal. The importer of record may request that FDA immediately refuse entry of the product, in which case the product must either be exported or destroyed. Alternatively, the importer of record may (1) submit evidence showing that the product does not appear to be violative or (2) request to “recondition” the product—for example, relabel the product or convert the product into a type of product FDA does not regulate. According to FDA’s Regulatory Procedures Manual, FDA will hold a hearing to determine whether the detained product should be released. If FDA determines that the importer of record has provided sufficient information to overcome the appearance of a violation, the importer of record receives a notice stating that the product is released. If FDA determines that the importer of record’s actions did not bring the product into compliance, the product would be refused and must be exported elsewhere or destroyed. FDA may decide to remove a firm or product from an import alert if there is evidence that the conditions that led to placement on the alert have been resolved, according to FDA’s Regulatory Procedures Manual. Our review of the manual and interviews with FDA officials indicate that FDA sampling and inspections are key activities that support the agency’s removal decisions. Generally, firms petition FDA to remove one or more products or the firms themselves from seafood import alerts, and FDA’s Division of Import Operations reviews the petitions. FDA’s procedures specify the evidence that firms are to submit, which varies depending on the nature of the import alert and the violation of FFDCA. FDA may require one or a combination of the following: a minimum of five consecutive nonviolative commercial shipments as determined by a private laboratory hired by the firm, an on-site inspection of the importer or foreign processing facility, or documentation showing that the cause of the violation has been fully corrected. For example, according to FDA’s procedures, firms or products placed on an import alert based on a violative facility inspection may generally be removed from the alert following a reinspection that shows that corrective actions to resolve the violation have been taken. Private laboratories usually collect and analyze the samples used as evidence to indicate that a commercial shipment does not violate FFDCA and provide support for FDA’s decisions to remove firms and products from import alerts. The procedures also call for the agency to have confidence that future shipments will comply with FFDCA, but they do not specify how FDA should ensure continued compliance. According to FDA officials, when the agency relies on documentation to support a removal decision, FDA generally relies on subsequent inspections of the importers or foreign processing facilities and sampling of their products to have confidence that the firms and their products continue to comply. FDA’s Regulatory Procedures Manual, as supplemented by the ORA Laboratory Manual, specifies that the agency should conduct checks to review whether the work performed by such laboratories can be used as an appropriate basis for FDA’s removal decisions. These checks include the following: Audit samples. FDA’s manuals specify the following two audit goals to ensure that the private laboratories’ analyses that FDA uses to support its removal decisions are valid: (1) to audit samples from at least one of the five nonviolative entries, as determined by a private laboratory that the firm hired, to support a removal decision to ensure the validity of the laboratory’s analysis and (2) to audit at least 10 percent of the work that a private laboratory performed to ensure that the laboratory submits scientifically sound data. In the course of its audits, FDA is to collect samples, called audit samples, to verify analytical results from a private laboratory that demonstrates a product complies with FFDCA. According to FDA, private laboratory analyses are a critical element in public health protection because they support FDA decisions to release detained goods. FDA’s collection of audit samples is intended to provide confidence in the laboratories’ analytical results. On-site assessments. FDA’s ORA Laboratory Manual states that, at times, FDA visits a private laboratory to ascertain that it has the capability or capacity to perform analyses that FDA often relies on to support removal decisions. The manual also states that on-site assessments provide the opportunity to observe that equipment and standards, among other things, needed to conduct the proposed analyses are present and in good order; to review the adequacy of the laboratory’s quality assurance and record-keeping programs; and to observe the techniques and practices of the analysts. Furthermore, the manual states that the on-site assessments are voluntary and that a private laboratory may decline to participate. FDA has established audit goals, requirements, and expectations related to sampling and inspections—key activities to support import alert removal decisions—but does not monitor the extent to which it is meeting them. In our review of FDA’s CMS data for 274 removal decisions from a nongeneralizeable selection of seven import alerts from October 1, 2011, through July 3, 2018, we found that FDA conducted audit sampling and inspections to support removal decisions and subsequent sampling and inspections to ensure continued compliance for a small percentage of the decisions. Specifically: Audit samples prior to removal decisions. For almost all of the 274 removal decisions we reviewed, FDA did not meet its first audit goal— to audit samples from at least one of the nonviolative shipments used to support a removal decision to ensure the validity of the analysis of the private laboratory hired by the firm. All seven of the import alerts we reviewed were established for violations of FFDCA for which FDA’s Regulatory Procedures Manual specifies that firms should enter into U.S. commerce at least five consecutive nonviolative commercial shipments, as determined by a private laboratory hired by the firm, before FDA may consider a removal. Therefore, FDA should have audited samples from at least one nonviolative shipment for all 274 removal decisions related to these seven import alerts. As described earlier, FDA collects audit samples from shipments of imported seafood to conduct such audits. However, we found that FDA did not conduct any sampling, including audit sampling, within 1 year prior to removal for 260 (or 95 percent) of the 274 removal decisions we reviewed. FDA officials told us that they do not monitor the extent to which the agency is meeting its audit goal, such as through analyzing CMS sampling data across all firms and products affected by the alerts and therefore were not aware that the agency had not met the audit goal. Conversely, FDA officials told us that they were aware that the agency historically had not met its second audit goal specified in its procedures—to audit at least 10 percent of each private laboratory’s work to support removal decisions to ensure that each laboratory submits scientifically sound data. While FDA does not regularly monitor whether it is meeting its 10 percent audit goal, in 2014, the agency analyzed data on the audit samples it collected during its audits of shipments covering fiscal years 2003 through 2013. FDA conducted this analysis in response to concerns that district staff raised about the quality of the analyses performed by private laboratories for one of its districts. These concerns included the following: Failure to obtain representative samples from throughout a shipment. Failure to obtain samples randomly from throughout the shipment. Failure to ensure an unbroken chain of custody from the site of collection of a sample to the private laboratory as necessary to ensure the integrity of the sample. Use of untrained temporary employees to collect samples and representing these individuals as employees of the private laboratory. FDA’s 2014 analysis showed that the agency did not achieve its 10 percent audit goal during the 11-year period. According to the analysis, FDA audited about 1 to 2 percent of work performed by private laboratories to support removal decisions. In response to our request, FDA updated its analysis through fiscal year 2018. The updated analysis shows that this percentage has improved in recent years, with FDA auditing about 3 percent of the work that private laboratories performed for fiscal year 2018. However, this level of auditing remains far below the goal of at least 10 percent, as shown in figure 1. According to FDA officials, the agency has not met this audit goal largely because it has limited resources. Inspections prior to removal decisions. For the 274 removal decisions we reviewed, FDA conducted inspections of importers or foreign processing facilities for 28 (about 10 percent) of the removal decisions in the 6 months prior. According to FDA’s procedures, firms or products placed on an import alert based on a violative facility inspection may generally be removed from the alert following a reinspection of the importer or foreign processing facility. In some instances, a firm may present information or documentation sufficient to demonstrate that appropriate corrections are in place to overcome the appearance of a violation and, with appropriate concurrence, may be removed from the import alert. FDA officials added that, regardless of the basis for placement on an import alert, FDA could require an on-site inspection prior to removal, depending on the hazard the violation posed. For example, certain violations may result in a finding of “official action indicated” (OAI), which indicates that an establishment failed to meet regulatory or administrative requirements and may pose a hazard to public health. FDA’s Field Management Directive 86 establishes a goal for FDA staff to conduct a follow-up inspection within 6 months after an OAI finding to verify that the facility has corrected violations. In our review of the 274 removal decisions, we found that for 32 firms that received an OAI inspection finding after FDA issued the directive in December 2011, FDA did not conduct a follow-up inspection for 31 of these firms before removing them from an import alert. According to FDA officials, the agency did not monitor whether its staff decided that inspections would be expected for the 274 removal decisions or whether the facilities that received an OAI inspection finding were reinspected. FDA officials told us that the agency relied on reviewing data on removal decisions individually to ensure that expected inspections had been conducted. Consequently, FDA was not aware of the extent to which the facilities associated with the removal decisions were actually inspected. Sampling or inspections following removal decisions. As shown in figure 2, for the 274 removal decisions we reviewed, FDA subsequently conducted sampling for 6 percent of the products at ports of entry and inspections for 13 percent of the importers or foreign processing facilities within 1 year after removal. FDA does not have a goal for the amount of sampling or inspections that should be conducted following removal decisions; however, as described above, FDA’s procedures call for the agency to base removal decisions on evidence establishing that the conditions that gave rise to the appearance of a violation have been resolved and that the agency has confidence that future shipments will comply with FFDCA. FDA officials said that when the agency does not inspect a facility and relies on documentation describing the actions the firm has taken to address the appearance of a violation to support a removal decision, the agency relies on subsequent sampling and inspections to have confidence in continued compliance. According to FDA, the past violative history of a firm is reflected in the PREDICT screening rules for the examination of future shipments and in the process of prioritizing inspections of foreign facilities. It was unclear from the CMS data that FDA provided the extent to which the agency relied on documentation to support the remainder of its removal decisions. However, based on FDA officials’ statements about subsequent sampling or inspections, we would expect to see a larger percentage of products sampled and firms inspected after their removal from import alerts for FDA to have confidence in continued compliance given the low percentage of inspections we found before removal decisions. FDA officials said they were not monitoring whether staff decided that subsequent sampling and inspections would be expected for these removals, and staff do not continuously monitor post- removal activities. Consequently, FDA officials were not aware of the extent to which the products and foreign processing facilities associated with removal decisions were subsequently sampled and inspected. FDA officials told us that they were generally aware that FDA had conducted limited sampling and inspections to support removal decisions and have confidence in continued compliance. They attributed this limited sampling and inspections to their belief that many import alert removal decisions can be supported by reviewing documentary evidence that FDA requested and the firms provided that describes the actions the firms have taken to address the appearance of a violation. According to FDA officials, such reliance on firm-provided documentation to support removal decisions is, in part, how FDA prioritizes its use of limited laboratory and inspection resources. FDA officials stated that the agency can check the basis of its removal decisions by looking up individual import alert cases in CMS and the agency’s sampling and inspection data in FACTS and OASIS to determine whether the agency would conclude that sampling and inspections to support these decisions would be appropriate, and if so, whether they were done. These officials said that they believed that checking the data on the basis of removal decisions individually and when questions arise from sources internal or external to FDA, instead of regularly analyzing sampling and inspections data, was sufficient to ensure the appropriate level of oversight. However, as discussed above, this approach has not informed them of the extent to which the agency is meeting its audit goals and expectations. Standards for internal control in the federal government state that management should design control activities to achieve objectives and respond to risks. An example of such control activities includes management comparing actual performance with planned or expected results. Such a comparison could include FDA comparing audits conducted with its audit goal (e.g., auditing at least 10 percent of a private laboratory’s work) to ensure that its goal was met. Monitoring the extent to which the agency is meeting its audit goals and expectations for conducting sampling and inspections to support its import alert decisions would enhance its oversight of these activities to better protect U.S. consumers from imported seafood that is not safe and wholesome. FDA and DHS have established a mechanism for coordinating the use of certain resources, but they generally have not coordinated to help ensure that firms comply with seafood import alerts by identifying potential instances of evasion of alerts, according to agency officials. FDA officials stated that the agency can coordinate with CBP in situations that could involve evasion of import alerts, but the agency does not have a formal mechanism for regularly and proactively coordinating to identify evasion. FDA officials said that such coordination could include CBP sharing information that could help FDA identify instances of evasion. As previously noted, CBP is responsible for collecting customs duties on imports, including seafood, and seeks to prevent the evasion of customs duties. As we reported in 2012, CBP personnel are to analyze trends in import data, among other things, to look for anomalies that may indicate evasion and also follow up on allegations from external sources. Once CBP identifies a potential instance of evasion, it can use a variety of techniques at different points in the import process to determine whether evasion is actually occurring. These techniques include collecting samples from shipments of products at U.S. ports of entry and conducting laboratory analyses of these samples to identify their true country of origin. Through its efforts, CBP has identified illegal transshipments—a scheme to conceal the country of origin and thereby evade applicable duties or FDA’s import alerts. For example, CBP reported that in 2016, customs officers seized about 42 tons of Chinese honey that had been transshipped through Taiwan to evade U.S. duties applicable to Chinese honey. According to FDA documents, at the same time, FDA had an import alert for honey because of unsafe drug residues. This alert included Chinese firms, but did not include any firms from Taiwan. In February 2009, we reported on CBP’s expertise in detecting illegal transshipment that could enhance FDA’s ability to detect import alert evasion. We stated that FDA and CBP could work together to help ensure that importers were not attempting to evade duties or import alerts. However, we found that the agencies had not identified ways to maximize and leverage their resources or established processes and policies for working together systematically across agency lines. We recommended, among other things, that FDA and CBP develop mechanisms to share information related to the evasion of import alerts. FDA and CBP agreed with our recommendation, but as of July 2019, the agencies had not fully implemented it. Specifically, FDA and CBP signed a memorandum of understanding (MOU), effective May 2013, to set forth terms for CBP to coordinate with FDA on staffing, space, and equipment requirements for the National Targeting Center. However, the MOU does not address CBP sharing information on potential evasion of import alerts with FDA regularly or the agencies working proactively to identify such evasion. According to CBP officials, FDA and CBP do not coordinate specifically on targeting to detect evasion, but CBP would be willing to coordinate with FDA and provide any applicable expertise in this area. While a collaborative mechanism such as an MOU is not needed to share information, we continue to believe that FDA and CBP should develop a mechanism to help the agencies formally coordinate to identify potential evasion of seafood import alerts. Until these agencies develop such a mechanism, they may be missing opportunities to share information regularly that could benefit each agency’s efforts to detect illegal transshipment and help FDA proactively identify and prevent evasion of seafood import alerts. FDA has not assessed the effectiveness of its seafood import alerts in helping to achieve its food safety mission. Specifically, FDA has not established performance goals and measures for seafood import alerts— key elements of assessing the effectiveness of programs. Performance goals explain the purpose of agency programs and the results—including outcomes—that they intend to achieve. Performance measures provide organizations with the ability to track the progress they are making toward their mission and goals and provide managers with information on which to base their organizational and management decisions. Under GPRAMA, agencies are required to develop long-term strategic plans and establish results-oriented goals in alignment with their missions and identify objectives and strategies needed to achieve those goals. GPRAMA also requires agencies to use performance information to assess their progress toward achieving their goals. According to FDA officials, the agency is implementing a program, which it refers to as an import alert effectiveness program, to review its import alerts. FDA documents note that the focus of this program includes (1) determining if FDA identified the firms on import alerts during its admissibility screening and took the appropriate action, (2) ensuring the accuracy of data FDA maintains in CMS on firms on import alerts, and (3) determining whether the reasons for the alerts are still relevant, and ensuring that the import alerts are accurately posted for clear communication to industry and FDA field staff. We commend FDA for these efforts. However, according to our review of FDA documents describing the activities planned for this program, the program does not include performance goals and measures for import alerts. FDA officials stated that this is because the program is new. Additionally, in February 2019, FDA published a broad plan for the safety of imported food that includes a goal, objective, and strategy related to import alerts. Under its goal to detect and refuse entry of unsafe foods at the border, FDA has an objective to strategically use import alerts and import certifications by using data and information from oversight activities, regulatory cooperation, and other reliable sources to enhance the effectiveness and efficiency of import alerts. However, FDA’s strategy for achieving this objective does not include performance goals or measures that would allow the agency to assess the effectiveness of its seafood import alerts in helping to achieve FDA’s food safety mission. In its 2019 plan for the safety of imported food, FDA states that it intends to develop performance goals and measures for imported food safety. However, FDA has not established a time frame for doing so. Once FDA has developed goals and measures for imported food safety, FDA would be able to establish corresponding performance goals and measures specific to seafood import alerts. By developing such goals and measures, FDA would be better positioned to assess how well its seafood import alert activities are supporting the agency in achieving its food safety mission. Import alerts play an important role in keeping the U.S. food supply—as well as other FDA-regulated products—safe, and FDA has numerous active import alerts affecting imported seafood that address a wide range of seafood products and violations of FFDCA. However, FDA does not have a process to monitor the extent to which it is conducting key activities to support its removal decisions—sampling and inspections. Establishing such a process would provide greater assurance that FDA is conducting its expected level of sampling and inspections to support its removal decisions and have confidence in continued compliance. Additionally, FDA and CBP have yet to develop mechanisms to share information regularly and proactively that can help detect noncompliance with import alerts through evasion. We continue to believe that doing so, as we previously recommended, would enhance the agencies’ efforts to identify potential evasion of seafood import alerts. Further, by establishing a time frame for developing goals and measures for assessing the effectiveness of its imported food safety efforts and also developing such goals and measures specific to seafood import alerts, FDA would be better positioned to assess how well its import alert activities are supporting the agency in achieving its food safety mission. We are making the following three recommendations to FDA: The Commissioner of FDA should establish a process to monitor whether the agency is meeting its audit goals and expectations for sampling and inspections to support its removal decisions for seafood import alerts. This could be done through regularly analyzing data that FDA collects, such as those in CMS, FACTS, and OASIS. (Recommendation 1) The Commissioner of FDA should establish a time frame for developing performance goals and measures for its imported food safety program. (Recommendation 2) The Commissioner of FDA should, as the agency develops goals and measures for its imported food safety program, develop performance goals and corresponding performance measures specific to seafood import alerts. (Recommendation 3) We provided a draft of this report to HHS and DHS for comment. In its comments, reproduced in appendix II, HHS’s FDA agreed with all three of our recommendations. FDA also provided technical comments, which we incorporated as appropriate. DHS provided technical comments, which we incorporated as appropriate. More specifically, FDA agreed with our recommendation that it establish a process to monitor whether the agency is meeting its audit goals and expectations for sampling and inspections to support its removal decisions for seafood import alerts. FDA stated that it agrees that developing metrics and monitoring the import alert removal process is necessary and that these efforts should be guided by the analysis of available data. FDA also stated that it plans to develop goals for its auditing process to ensure audit sampling targets products of higher public health concern and provides the agency support to guide decisions to release individual shipments that have been detained as a result of an import alert. FDA further stated that it intends to enhance its case management system to include checklists for FDA reviewers who process petitions for removal from import alerts to better document that all necessary information is present and has been evaluated to support the removal decision. FDA agreed with our recommendation that it should establish a time frame for developing performance goals and measures for its imported food safety program. FDA stated that the agency is developing performance measures and outcome indicators for imported food safety to help support the agency’s overall goal of reducing the incidence of illness and death attributable to preventable contamination of FDA- regulated foods. Finally, FDA agreed with our recommendation that it should, as it develops goals and measures for its imported food safety program, develop performance goals and corresponding performance measures specific to seafood import alerts. FDA stated that the agency will use the results of its import alert effectiveness program to develop metrics to demonstrate the effectiveness of the program and its use of import alerts. The extent to which FDA’s planned actions will satisfy our recommendations will depend on how FDA implements those actions. As agreed with your offices, unless you publicly announce the contents earlier, we plan no further distribution of this report until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Health and Human Services, the Secretary of Homeland Security, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions regarding this report, please contact me at (202) 512-3841 or morriss@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Table 1 includes information posted on the Food and Drug Administration’s website describing the 52 import alerts affecting seafood that were active as of July 3, 2018. In addition to the contact named above, Anne K. Johnson (Assistant Director), David Moreno (Analyst in Charge), Kevin Bray, Steven Campbell, Stephen Cleary, Michele Fejfar, Ellen Fried, Juan Garay, Caitlyn Leiter-Mason, Ying Long, Cynthia Norris, Dan Royer, and Kiki Theodoropoulos made key contributions to this report. Food Safety and Nutrition: FDA Can Build on Existing Efforts to Measure Progress and Implement Key Activities. GAO-18-174. Washington, D.C.: January 31, 2018. Imported Seafood Safety: FDA and USDA Could Strengthen Efforts to Prevent Unsafe Drug Residues. GAO-17-443. Washington, D.C.: September 15, 2017. Seafood Safety: Status of Issues Related to Catfish Inspection. GAO-17- 289T. Washington, D.C.: December 7, 2016. Imported Food Safety: FDA’s Targeting Tool Has Enhanced Screening, but Further Improvements Are Possible. GAO-16-399. Washington, D.C.: May 26, 2016. 2015 Annual Report: Additional Opportunities to Reduce Fragmentation, Overlap, and Duplication and Achieve Other Financial Benefits. GAO-15- 404SP. Washington, D.C.: April 14, 2015. Food Safety: FDA Can Better Oversee Food Imports by Assessing and Leveraging Other Countries’ Oversight Resources. GAO-12-933. Washington, D.C.: September 28, 2012. Managing for Results: Key Considerations for Implementing Interagency Collaborative Mechanisms. GAO-12-1022. Washington, D.C.: September 27, 2012. Seafood Safety: Responsibility for Inspecting Catfish Should Not Be Assigned to USDA. GAO-12-411. Washington, D.C.: May 10, 2012. Seafood Safety: FDA Needs to Improve Oversight of Imported Seafood and Better Leverage Limited Resources. GAO-11-286. Washington, D.C.: April 14, 2011. Seafood Fraud: FDA Program Changes and Better Collaboration among Key Federal Agencies Could Improve Detection and Prevention. GAO-09- 258. Washington, D.C.: February 19, 2009. Results-Oriented Government: Practices That Can Help Enhance and Sustain Collaboration among Federal Agencies. GAO-06-15. Washington, D.C.: October 21, 2005.", "summary": "Imports account for over 90 percent of U.S. seafood consumption. FDA and the Department of Homeland Security (DHS) both play a role in overseeing imported seafood. FDA is responsible for ensuring the safety of most imported seafood. DHS provides FDA with import data on FDA-regulated products, including seafood. If FDA finds that imported seafood products appear to violate U.S. laws, FDA may place the products, firms, or countries on an import alert. GAO was asked to review FDA's efforts to use import alerts to ensure the safety of imported seafood. This report, among other things, (1) describes FDA's import alert process for seafood products, (2) examines FDA oversight of key activities to support import alert removal decisions, and (3) examines the extent to which FDA has assessed the effectiveness of its seafood import alerts. GAO reviewed FDA procedures and data, including data on 274 removal decisions, for a non-generalizable sample of seven import alerts selected for a range of violations of federal law. GAO also interviewed FDA officials. The Food and Drug Administration's (FDA) import alert process for seafood products includes three key components: (1) establishing new import alerts, which inform FDA field staff and the public that the agency has enough evidence that products appear to violate a federal food safety law to detain those products at U.S. ports of entry without physically examining them; (2) placing firms and products on existing import alerts; and (3) removing firms and products from those import alerts when violations are resolved. As of July 3, 2018—the most recent data at the time of GAO's analysis—FDA had 52 active import alerts affecting imported seafood that addressed a wide range of violations of federal law, including the presence of foodborne pathogens, such as Salmonella , or unapproved animal drug residues. FDA has established audit goals, requirements, and expectations related to sampling and inspections—key activities to support import alert removal decisions—but does not monitor the extent to which it is meeting them. GAO's review of 274 removal decisions from October 1, 2011, through July 3, 2018, found that FDA had supported only a small percentage of its removal decisions by conducting sampling and inspections. For example, FDA has a goal to audit samples from at least one of the shipments used to support each removal decision to ensure the validity of the analysis that a private laboratory performed. However, GAO found that within a year prior to the 274 removal decisions, FDA did not conduct any audits for 260 (95 percent) of the 274 removal decisions. FDA officials said they conducted limited sampling because many import alert removal decisions can be supported by documentary evidence provided by firms. Additionally, for certain violations that indicate a firm failed to meet regulatory or administrative requirements and may pose a public health hazard, an FDA directive establishes a goal for FDA staff to conduct a follow-up inspection within 6 months. However, GAO's review of removal decisions found that for 31of the 32 firms that received such a finding, FDA did not conduct a follow-up inspection before removing them from an import alert. FDA officials said they did not know whether they were meeting their audit goals because the agency does not have a process to monitor the extent to which it is conducting its sampling and inspections. Establishing such a process would provide greater assurance that FDA is conducting its expected level of sampling and inspections to support its removal decisions and has confidence in continued compliance. FDA has not established performance goals and measures for seafood import alerts—key elements for assessing the effectiveness of programs. Goals explain the outcomes a program seeks to achieve, and measures track progress towards those goals. In February 2019, FDA published a broad plan for the safety of imported food. The plan states that FDA intends to develop performance goals and measures related to imported food safety, but FDA has not established a time frame for doing so. By establishing a time frame and developing such goals and measures, FDA would be better positioned to assess how well its seafood import alert activities are supporting the agency in achieving its food safety mission. GAO recommends that FDA (1) establish a process to monitor whether the agency is meeting its audit goals and expectations for sampling and inspections, (2) establish a time frame for developing goals and measures for its imported food safety program, and (3) develop goals and measures for seafood import alerts. FDA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "The Defense Acquisition Regulations Council is responsible for developing fully coordinated recommendations for revisions to the DFARS, which supplements the Federal Acquisition Regulation. The Federal Acquisition Regulation provides executive agencies with uniform acquisition policies and procedures for acquiring products and services, and is prepared and issued through the coordination of the Defense Acquisition Regulations Council and Civilian Agency Acquisition Council. The DFARS contains additional requirements of law, DOD-wide policies, delegations of Federal Acquisition Regulation authorities, deviations from Federal Acquisition Regulation requirements, and policies or procedures that have a significant effect beyond the internal operating procedures of DOD, or a significant cost or administrative impact on contractors or offerors. The DFARS is designed to be read in conjunction with the primary set of rules in the Federal Acquisition Regulation. Stakeholders in the acquisition process include executive agencies’ program and contracting officials, members of Congress and congressional staff, industry and contractors, and members of the public. Specifically, the Defense Acquisition Regulations Council generally makes implementation recommendations to DOD, such as when publication of rules to amend the DFARS is appropriate. DARS staff then implements the Council’s recommendations. The Defense Acquisition Regulations Council is composed of the Chair who is also the DARS Director, Deputy Chair who is also the DARS Deputy Director, and one policy and one legal representative from each of the following DOD components: Defense Contract Management Agency, and Defense Logistics Agency. DFARS changes can originate from different sources, including legislation, recommendations from DOD’s Office of the Inspector General, our recommendations, court decisions, executive orders, or policy changes within DOD. DFARS changes that originate from legislation, including NDAAs, are given the highest priority, according to DARS officials. DARS staff has other related responsibilities, including working with civilian agencies in activities connected with promulgating the Federal Acquisition Regulation. DOD has a rulemaking process to change the DFARS that includes implementing acquisition-related NDAA provisions through regulatory changes or other methods. The DARS staff is responsible for facilitating the process of making these changes in the DFARS. The staff first reviews draft legislation that may affect acquisition regulations before Congress enacts the NDAA. After the NDAA is enacted, DARS staff then identifies which provisions require action. The DARS staff coordinates across the department and provides for public notice of implementation actions when required. However, there is no publicly-available summary reporting of the status of the regulatory changes or other implementation methods linked to specific NDAA provisions. Congress and industry representatives therefore cannot clearly see the status of pending regulatory changes pertaining to acquisition issues addressed in the NDAA. DOD’s acquisition rulemaking procedures are governed by statute, which generally requires agencies to issue a proposed rule for each rulemaking and provide not less than a 30-day public comment period following publication of the proposed rule in the Federal Register. These requirements only apply to those DFARS rules that are related to the expenditure of appropriated funds and have either a significant effect beyond the agency’s internal operating procedures or a significant cost or administrative impact on contractors or offerors. However, the requirements may be waived if “urgent and compelling” circumstances make compliance with the requirements impracticable. In those instances, DOD issues an interim rule rather than a proposed rule. The interim rule is effective on a temporary basis if DOD provides at least a 30-day public comment period after publishing the interim rule in the Federal Register. DOD then may issue a final rule after considering any comments received. As a part of the rulemaking process, the Office of Information and Regulatory Affairs reviews proposed and final regulations. The time period for its review is generally limited to 90 days. See figure 1 for an overview of the DARS’s process to change DFARS rules. DARS staff can implement the provisions by one or more methods, including the rulemaking process described above and other actions, such as: issuing DFARS class deviations, and changing DFARS Procedures, Guidance, and Information (PGI), a non-regulatory document that supplements the DFARS. Before annual NDAAs are enacted, DARS staff told us that they review proposed legislation and committee report language to stay abreast of provisions they may have to implement after NDAAs are enacted. DARS staff solicits input on which provisions may require implementation from DOD components and offices, such as the Defense Contract Management Agency, that have a stakeholder interest in many acquisition-related provisions. DARS staff tracks each of these potential changes in case files, which are referred to in this report as cases. DARS staff also can work with other federal agency offices to implement an acquisition-related NDAA provision through a Federal Acquisition Regulation rule change, interim rule change, or class deviation. In some instances, a provision may specify that DOD take other actions, such as holding a public meeting to obtain interested parties’ opinions on an acquisition topic. Upon review of the enacted NDAA, the Defense Acquisition Regulations Council or DARS staff sometimes decides that a provision should be implemented by another DOD office or in other defense acquisition guidance. For example, the DARS staff could determine that a provision only applies to one DOD component and does not require a DFARS change. In another example, DARS staff could determine that the initially identified provision should be implemented in acquisition guidance, such as DOD Instruction 5000.02. Further, sometimes DARS staff will change implementation methods after having selected one. For example, DARS staff may initially decide to implement a provision with a DFARS change, but upon conducting research to draft the rule change, it may find that the provision would be better implemented with a Federal Acquisition Regulation change. Based on our review of NDAAs from fiscal years 2010-2018, we identified 37 explicitly directive provisions—36 that directed DOD to either make or consider making an acquisition-related regulatory change, and one that directed DOD to issue acquisition-related guidance. DARS officials told us that when a provision directs a change or consideration of an acquisition-related regulatory change, the Defense Acquisition Regulations Council and DARS staff give it the highest priority. We confirmed that, in the Defense Acquisition Regulations Management Information System, this priority is reflected by identifying the NDAA as the source of the change in the synopsis field. We confirmed that the 36 provisions we identified had NDAA as the source of the change. DARS staff has different ways of communicating changes to the regulations and other implementation methods to the public. “Significant revisions” to the DFARS must be published in the Federal Register. DARS staff also publishes the progress of DFARS changes in case reports that are available on its website. Case reports provide a synopsis of each case, which can include the NDAA provision or other source of the case; describe cases combined to address more than one provision; or show multiple cases for a single provision. DARS staff also posts notices of DFARS class deviations and revisions to DFARS PGI on its website. DARS staff provides input for regulatory priorities through DOD’s publicly-available Unified Agenda. This includes all expected rule changes DOD-wide and a Regulatory Plan that identifies the most significant regulatory actions DOD expects to issue within the next 12 months. It is difficult, however, for interested parties, such as Congress and industry groups, to determine if a provision has been implemented using only this publicly-available information. This is due, in part, to the fact that provisions can be implemented through one or multiple methods, and DARS actions can be reflected in more than one case. For example, if an interested party, such as a federal contractor, expects to see a change to the DFARS based on how an NDAA provision is worded, but the DARS staff implements the provision with a class deviation, the interested party may not realize that the provision has been implemented by another method. In addition, DARS staff may consider a provision as implemented with an action such as a class deviation even if a subsequent case to change the DFARS is opened later. We, too, found it difficult to determine the implementation status of acquisition-related NDAA provisions using only publicly-available reports and information. DARS staff was able to create a report for us that showed implementation status by provision. But we were able to determine and verify the implementation status of these provisions only after using a combination of the DARS internal reports, publicly-available reports and information, and data we had requested from the Defense Acquisition Regulations Management Information System database. DFARS and Federal Acquisition Regulation open and closed case reports provide general information on a case, such as the topic and case number. The reports also provide the status of the case. For example, a report may say: “Defense Acquisition Regulations Council director tasked team to draft proposed DFARS rule.” However, the case reports do not provide information on when a regulatory change may be expected. This information can help companies plan for future business opportunities and devise the means to ensure compliance with regulations. See figure 2 for an overview of NDAA provision implementation methods and the mechanisms DOD uses to report status information. Standards for Internal Control in the Federal Government states that management should externally communicate quality information to achieve the entity’s objectives. Specifically, available information should address the expectations of both internal and external users. DARS staff regularly publishes public status updates on cases, rule changes, and PGI changes. However, there is no readily available mechanism for external stakeholders, such as Congress and industry representatives, to determine the implementation status of any particular legislative provision. This is because the status updates published by the DARS staff do not provide the complete implementation status listed by specific legislative provisions. Without communicating the implementation status of legislative provisions, Congress lacks information for oversight of acquisition reforms, and federal contractors lack visibility into how and when changes will occur. For example, the House Armed Services Committee expressed its oversight interest in a provision passed in 2013 that was not implemented in the DFARS until 2018. Additional information on the status of the DFARS change may have been helpful to the committee’s oversight activities. In another example, industry expressed concern about the status of a regulation implementing a fiscal year 2017 NDAA provision related to the lowest price technically acceptable (LPTA) source selection process in order to plan for responding to solicitations following implementation of the rule. DARS staff identified 180 NDAA provisions from fiscal years 2010-2018 that potentially required an acquisition-related regulatory change or another action. DARS staff and other DOD entities have taken some type of action to address all these provisions. Our analysis showed that 112 of the provisions had been implemented. The timeframe for implementation was, on average, just under 1 year. Some implementation efforts took longer than a year for a variety of reasons, such as reconciling multiple years of NDAA requirements or dealing with highly complex topics. The remaining legislative provisions are either in the process of being implemented or DARS staff determined that a regulatory change was not needed. DARS staff prioritized those provisions that expressly directed DOD to change or consider an acquisition-related regulatory change. DARS documentation showed that some of the implementation deadlines in statute were shorter than the time periods that DARS generally allows for the rulemaking process, including public comment and outside agency review. Following its process, DARS staff identified 180 NDAA provisions from fiscal years 2010-2018 that potentially required an acquisition-related regulatory change or another implementation action. We found that DARS staff and, in a few instances, other DOD entities have taken action to address all of those provisions. See figure 3 for the implementation status of all 180 provisions distributed by NDAA fiscal year. We found that DARS officials opened cases within 30 days of NDAA enactment, on average, for the acquisition-related NDAA provisions from fiscal years 2010-2018. For the 112 of 180 provisions that have been implemented, DOD completed the first implementation actions on average within 1 year. DARS staff frequently used a combination of methods to implement provisions, such as using an interim DFARS rule followed by a final rule. When two or more implementation actions are taken, DARS officials generally consider the first action as the action that implements the provision. If a class deviation, interim DFARS rule, or PGI is issued to address an NDAA provision, the DARS staff considers it implemented even if additional actions—such as issuing a final DFARS rule—are still being pursued. We used the same approach for our analyses for determining the implementation status of provisions and time taken to complete implementation. See table 1 below for the average time to complete the first action to implement the 112 NDAA provisions. Figure 4 shows the distribution of time taken to implement all 112 NDAA provisions. Some implementation efforts took longer than a year for a variety of reasons. Publishing an interim DFARS rule generally took less than a year, while publishing a final DFARS rule change took closer to 2 years on average. In the selected DFARS cases studied, we found examples where DOD had to reconcile multiple years of NDAA requirements or manage complex topics, which we have similarly reported on as reasons that influence the time needed to issue regulations in past work. Reconciling Multiple Years of NDAA Requirements: Congress directed DOD to revise the DFARS to reflect updated requirements related to procuring commercial items in section 851 of the fiscal year 2016 NDAA. Congress included a deadline of 180 days from the NDAA enactment, but the DFARS update was not completed until nearly 800 days after enactment. Our review of DARS case files showed that the DARS staff prioritized implementing the provision, but decided to address a related NDAA provision from 2013 through a single DFARS rule change. In this instance, multiple NDAAs included provisions the DARS staff viewed as closely related. As a result, developing language that reconciled the requirements for all of these provisions took additional time and effort. DARS officials told us that they came close to publishing a commercial items rule earlier, but started over because subsequent NDAA provisions included requirements related to commercial items. Managing Complex Topics: Congress directed DOD to revise the DFARS regarding the use of the LPTA source selection process in section 813 of the NDAA for fiscal year 2017. Congress included a deadline of 120 days from enactment in the provision, which DARS staff was unable to meet due to the complexity of the issue and additional requirements added by a subsequent NDAA. Following enactment of the 2017 NDAA, DARS staff developed a proposed rule that would have implemented relevant NDAA sections in under a year. However, prior to publishing that rule, the NDAA for fiscal year 2018 was enacted and contained added LPTA requirements. After the 2018 NDAA was enacted, DARS staff combined all of its related LPTA cases into a new DFARS case and made adjustments to the proposed rule it had been developing. The DARS staff responsible for updating the previous proposed rule requested five extensions from DARS leadership between January and March 2018 to update documentation to address the fiscal year 2018 provisions and prepare additional analyses. After months of coordination and reviews, DARS staff published a proposed rule in December 2018 with a 60-day comment period. Sixteen formal submissions were received by the February 2019 deadline. The DARS staff is currently reviewing those comments and drafting a final rule, which must still go through multiple reviews before it can be published in the Federal Register. Congress directed DOD to consider revising the DFARS regarding an extension of contractor conflict of interest limitations in section 829 of the NDAA for fiscal year 2013. This provision has been in the process of implementation due to a determination that this rule should be informed by a pending Federal Acquisition Regulation change. In this instance, Federal Acquisition Regulation principals opened a case to implement the provision in the Federal Acquisition Regulation 7 months after NDAA enactment, and DARS officials agreed to draft the rule change that would implement the provision. DARS staff published a proposed rule in the Federal Register for public comment approximately 8 months later. However, DARS staff informed us that a few weeks after the public comment period, Federal Acquisition Regulation officials directed them to suspend its activities until a separate, related Federal Acquisition Regulation rule on “closely associated with inherently governmental functions” was finalized. However in August of 2018, section 829 of the NDAA for fiscal year 2013 was repealed by section 812(b)(4) of the NDAA for fiscal year 2019. We identified 36 provisions, a subset of the 180, that expressly directed DOD to make or consider making an acquisition-related regulatory change, as well as one provision that directed DOD to issue guidance. DARS staff implemented 22 of the 37 provisions in about 13 months on average. Of the 37 provisions, 32 had statutory deadlines, ranging from 30 to 365 days after enactment. The DARS documentation showed that the DARS staff prioritized these NDAA provisions by noting the deadlines, but generally did not implement them by the deadline. We found that: DARS staff met the deadlines in eight of 32 instances. In those eight instances, the actions completed were relatively simple, and DARS staff determined that a public comment period was not required. For example, DARS staff changed the DFARS to implement section 801 of the fiscal year 2018 NDAA—which required DOD to revise the DFARS to include three specific statements about DOD acquisitions— in 143 days, ahead of Congress’s 180-day deadline. Four provisions had deadlines for implementation of 60 days or less. For example, sections 841 and 842 in the fiscal year 2012 NDAA called for changes to be made to the DFARS within 30 days. The short deadlines allowed for fewer days than DARS staff allocate for public comment (minimum of 30 days, by law) and outside agency review (no more than 90 days, by executive order). Deadlines that did not allow for these activities as well as time to draft language were typically not met. DARS is responsible for developing and maintaining DOD acquisition regulations, which may include implementing acquisition-related NDAA provisions. The DARS staff has internal tools to track, manage, and communicate the status of DFARS changes, including implementation of NDAA provisions. However, DOD’s DFARS change process does not have a reporting mechanism to clearly communicate to Congress, industry, and other interested parties the status of regulatory or other changes linked to specific NDAA provisions. Without a mechanism to better communicate DOD’s actions to implement NDAA provisions, stakeholders potentially affected by reforms may be unaware of what and when changes may be implemented. Given the actions and length of time that it may take to implement provisions and see a change reflected in the DFARS or elsewhere, stakeholders would benefit from knowing the status of DOD’s actions before implementation has been completed in order to, for example, prepare for compliance. We are making the following recommendation to the Secretary of Defense to ensure that the Director of the Defense Acquisition Regulations System: Develop a mechanism to better communicate to all stakeholders the implementation status of acquisition-related NDAA provisions, particularly those provisions that direct a change or consideration of a change to the DFARS. (Recommendation 1) We provided a draft of this report to DOD for comment. DOD concurred with our recommendation to develop a mechanism to better communicate to all stakeholders the implementation status of acquisition-related NDAA provisions. The department said it will develop a matrix reflecting the implementation status of acquisition-related NDAA provisions and post the matrix on the Defense Pricing and Contracting public website. DOD’s written comments on the report are reprinted in appendix II. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Acting Secretary of Defense; the Under Secretary of Defense for Acquisition and Sustainment; the Secretaries of the Air Force, Army, and Navy; the Director, Defense Acquisition Regulations System; appropriate congressional committees; and other interested parties. This report will also be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by e-mail at woodsw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to this report are listed in appendix III. A House Armed Services Committee report related to the National Defense Authorization Act (NDAA) for Fiscal Year 2019 included a provision for us to review the Department of Defense’s (DOD) process for revising the Defense Federal Acquisition Regulation Supplement (DFARS), among other things. This report (1) determines how DOD implements acquisition-related NDAA provisions and communicates implementation status, and (2) identifies the status of DOD’s efforts to implement acquisition-related NDAA provisions from fiscal years 2010- 2018. To determine how DOD implements acquisition-related NDAA provisions, we reviewed DOD documents and supplemented our work with interviews with relevant DOD officials. Specifically, we reviewed the DFARS Operating Guide, January 2015; presentation on the Defense Acquisition Regulations System Rulemaking Process, DFARS open and closed cases reports, Federal Acquisition Regulation open and closed cases reports; decision matrices from the Defense Acquisition Regulations System (DARS), which document decisions on implementing NDAA provisions from fiscal years 2010-2018; and other applicable reports and information on provisions and cases from the DARS staff and the Defense Acquisition Regulations Council. We also referenced our past reports on DFARS rulemaking; U.S. Code on Publication of proposed regulations; the Federal Acquisition Regulation Operating Guide, July 2015; Federal Register notices related to DOD rulemaking; and the news listing on the DARS website. We adopted the DARS use of the term “implementation,” which includes both regulatory action as well as other actions, such as public meetings or a report. We interviewed DOD officials that are involved in the DFARS rulemaking process. Specifically, we interviewed members of the Defense Acquisition Regulations Council and DARS staff, including the Chair and Deputy Chair, the Regulatory Control Officer that prepares rules for submission to the Office of Information and Regulatory Affairs within the Office of Management and Budget, and DFARS case managers. We also interviewed officials from the DOD components—Air Force, Army, Navy, Defense Contract Management Agency, and Defense Logistics Agency. We interviewed industry representatives from the Aerospace Industries Association, National Defense Industrial Association, and the Professional Services Council. We compared the DARS process with the Standards for Internal Control in the Federal Government. Specifically, we reviewed DOD’s public reports of its implementation actions with internal control principle 15: “management should externally communicate the necessary quality information to achieve the entity’s objectives.” Stakeholders in the acquisition process include executive agencies’ program and contracting officials, members of Congress, congressional staff, industry, contractors, and members of the public. The DARS staff provided a complete data extract of Defense Acquisition Regulations Management Information System as of October 31, 2018, to document the acquisition-related NDAA provisions that DARS staff identified as potentially requiring implementation. The Defense Acquisition Regulations Management Information System is the DARS database to track the status of individual cases that are associated with DARS rulemaking actions. We analyzed the data extract to identify which Title VIII provisions that the DARS identified for implementation from NDAAs from fiscal years 2010-2018, and to identify the cases related to those provisions. We focused on Title VIII—Acquisition Policy, Acquisition Management, and Related Matters—of the NDAAs, which contain acquisition-related provisions. We queried the data extract to identify cases with notes indicating NDAA provisions from fiscal years 2010-2018 as the source of change in the database synopsis field. We found 180 acquisition-related provisions from Title VIII of the NDAAs from fiscal years 2010-2018 that the DARS staff had identified for implementation. For these 180 provisions, we determined the number and types of cases by year, duration of cases, and duration of select steps for cases. We verified the validity of provisions and cases that were not in both the DARS reports that DARS staff manually produced and the Defense Acquisition Regulations Management Information System data with DARS officials as of April 19, 2019. To identify the implementation status of acquisition-related NDAA provisions from fiscal years 2010-2018, we further analyzed data from the Defense Acquisition Regulations Management Information System and DARS reports. For the actions associated with the 180 provisions, we analyzed the status history of each case, associated status dates for cases, and closed status indicators. We also reviewed DARS reports, such as the internal stats charts with case duration and closure metrics that DARS officials told us they manually verify. We reviewed a report that the DARS staff manually produced for us that showed actions and cases by provision for the NDAAs from fiscal years 2010-2018. We independently analyzed the NDAAs from fiscal years 2010-2018 and determined 36 provisions in Title VIII that expressly directed DOD to make or consider making an acquisition-related regulatory change, as well as one provision that directed DOD to issue guidance. We identified these provisions using a keyword search of individual and combined terms and criteria, such as “regulation, defense, and acquisition regulation.” To better understand the Defense Acquisition Regulations Council’s recommendations and DARS implementation process, we selected 12 provisions that directed DOD to make or consider an acquisition-related regulatory change for case studies. The case study selection criteria included the year of the NDAA from which the provision originated for a mix of older and newer provisions and time duration for a mix of shorter and longer cases related to implement the provisions. We used DARS reports and our analysis of the Defense Acquisition Regulations Management Information System data to determine the year and time duration. Since the DFARS Case Standard Timeline is 52 weeks, we selected provisions with cases that were both more and less than 52 weeks. We also selected provisions with cases that were open and closed. We created a data collection instrument for the case studies that captured information, such as which provisions were associated with the case, to standardize our data collection process. For the 12 provisions, we reviewed the associated case files that are generally a record of the implementation process and the Defense Acquisition Regulations Council’s recommendations, and the decisions made by the DARS staff. We also reviewed available publication folders associated with the cases that generally document input and decisions from other agencies, such as the Office of Management Budget’s Office of Information and Regulatory Affairs. Finally, we used the information in the files to verify the information in Defense Acquisition Regulations Management Information System for those specific cases. We found the Defense Acquisition Regulations Management Information System data and information in the files that we reviewed to be sufficiently reliable for purposes of reporting on how the DARS staff implemented NDAA provisions and the time duration to do so. We conducted this performance audit from August 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Penny Berrier, Assistant Director; James Kim; Holly Williams; Beth Reed Fritts; Gail-Lynn Michel; Emily Bond; Lori Fields; Matthew T. Crosby; Lorraine Ettaro; and Tim Bober made key contributions to this report.", "summary": "Congress has pursued acquisition reforms to make DOD's acquisition process more efficient and timely. Some statutes have directed DOD to revise or consider revising its acquisition regulations. The House Armed Services Committee's report accompanying the NDAA for Fiscal Year 2019 included a provision for GAO to review DOD's regulatory implementation of acquisition-related provisions in the NDAAs from fiscal years 2010 through 2018. This report (1) determines how DOD implements acquisition-related NDAA provisions in the DFARS and communicates with stakeholders throughout that process, and (2) identifies the status of implementation of provisions enacted in the specified NDAAs. To conduct this work, GAO reviewed DOD documents and interviewed DOD officials regarding the process for implementing acquisition-related NDAA provisions. GAO also analyzed DOD's data and reports on the implementation status of provisions enacted in NDAAs for fiscal years 2010 through 2018. GAO selected 12 of these provisions as case studies based on factors such as year enacted and time taken for implementation to obtain a mix of older and newer provisions, and shorter and longer implementation timeframes. The staff of the Defense Acquisition Regulations System are responsible for making changes in the Defense Federal Acquisition Regulation Supplement (DFARS)—the Department of Defense's (DOD) regulation augmenting the Federal Acquisition Regulation, which guides government purchases of products and services. They begin their process by first tracking legislation that may affect acquisition regulations before Congress enacts the National Defense Authorization Act (NDAA). After enactment, they identify which provisions to implement through regulatory changes and which to implement through other means. In certain circumstances, rather than change the DFARS, DOD can issue a class deviation, which allows its buying organizations to temporarily diverge from the acquisition regulations. The figure below shows the primary means DOD uses to implement NDAA provisions, and the mechanisms DOD uses to make information on the status of any changes available to the public and others. Department of Defense's (DOD) Methods to Implement and Report on Actions Taken on National Defense Authorization Act (NDAA) Provisions DOD does not have a mechanism to clearly communicate to Congress, industry, and other interested parties the status of regulatory or other changes based on NDAA provisions. Using only publicly-available reports and information, it is difficult for an interested party to find the implementation status of any given acquisition-related NDAA provision. This is because no single DOD source communicates the status of regulatory or other changes in a manner that links the changes to specific NDAA provisions. As a result, interested parties are not always aware of what provisions have been implemented and when. This information is important for congressional oversight and to industry for planning and compliance purposes. Federal internal control standards state that management should address the communication expectations of external users. GAO found that DOD has taken action to address 180 acquisition-related provisions since 2010. On average, implementation was completed within 1 year from enactment. Some complicated provisions took more than 2 years to implement. For example, a fiscal year 2016 NDAA provision, directing a regulatory change for commercial item procurements, took more than 2 years to implement because DOD was reconciling a prior year's related but different NDAA commercial item provision into one DFARS change. DOD should develop a mechanism to better communicate the implementation status of acquisition-related NDAA provisions, particularly those that direct a change or consideration of a change to the DFARS. DOD concurred with the recommendation.", "document_type": "gao"}
{"report": "The Immigration and Nationality Act (INA) of 1952, as amended by the Immigration Reform and Control Act of 1986, authorizes the establishment of the H-2B visa category which allows U.S. employers to bring non-immigrant workers into the United States to perform temporary non-agricultural work. Generally, U.S. employers may apply for H-2B visas when they can establish that (1) their need for an H-2B worker’s labor is temporary, meaning a one-time occurrence, a seasonal need, a peak load need, or an intermittent need; (2) qualified U.S. workers are unavailable to perform the work; and (3) the employment of an H-2B worker will not adversely affect the wages or working conditions of similarly-employed U.S. workers. Generally, an H-2B worker’s authorized stay per the TLC will be no more than 10 months. However, DHS may authorize an extension of up to one year to H-2B workers already in the United States, based on a subsequent TLC, with a maximum stay of up to three years. Pursuant to the INA, as amended by the Immigration Act of 1990, the H- 2B visa program is subject to an annual cap of 66,000 visas. These visas are divided into two semiannual allocations: up to 33,000 workers may be issued H-2B visas or provided H-2B nonimmigrant status in the first half of the fiscal year(October 1 – March 31), and the remaining annual allocation will be available in the second half of the fiscal year (April 1 – September 30). In fiscal years 2005, 2006, 2007, and 2016, Congress amended the INA to include a provision that established a returning worker exemption. This exemption enabled H-2B workers who were counted against the visa cap during one of the three preceding fiscal years to not be counted against the visa cap for the relevant fiscal year. Federal agencies use a multi-step process to screen employers to ensure eligibility to hire H-2B workers and later screen nonimmigrant workers on eligibility to work under the H-2B visa category (see fig. 1). DOL’s Office of Foreign Labor Certification (OFLC) screens and processes TLC applications from employers. OFLC is to review these applications to ensure that no qualified U.S. workers are available for the job in question and that the wages and working conditions offered to H-2B workers will not adversely affect similarly employed U.S. workers. In 2015, DOL and DHS jointly issued regulations that set forth a number of specific requirements that employers must meet in order to obtain a TLC, including taking specific steps to recruit U.S. workers before hiring H-2B workers; paying a wage equal to or exceeding the highest of the prevailing wage or the federal, state, or local minimum wage; paying for H-2B workers’ transportation costs; and guaranteeing a minimum number of work hours to H-2B workers. Although employers may submit a TLC application requesting a specific number of H-2B workers, DOL may approve all the workers requested, approve a smaller number of workers, or deny the application. Employers can petition DHS’s U.S. Citizenship and Immigration Services (USCIS) for a number of workers up to the number approved by DOL, then USCIS screens and processes employer’s petitions. DHS is to send the approved petitions to the Department of State, which screens workers that apply for H-2B visas at U.S. embassies and consulates overseas. The Department of State is responsible for interviewing H-2B applicants and reviewing their visa applications and supporting documentation as part of their adjudication process. DOL is the primary agency that enforces H-2B employer requirements and relevant labor laws. This enforcement authority is delegated within the DOL to the Administrator of the Wage and Hour Division (WHD). WHD conducts investigations, inspections, and law enforcement functions that carry out the provisions of 8 U.S.C. § 1184(c), INA section 214(c), and the regulations pertaining to the employment of H–2B workers, any worker in corresponding employment, or any U.S. worker improperly rejected for employment or improperly laid off or displaced, according to DOL. WHD investigates complaints filed by both foreign and U.S. workers affected by the H–2B program, as well as concerns raised by other federal agencies, such as DHS or the Department of State, regarding particular employers and agents. WHD also conducts targeted or directed (i.e., not complaint-based) investigations of H–2B employers to evaluate program compliance. Through OFLC, DOL may audit adjudicated applications to ensure employers’ compliance with the terms and conditions of their H-2B Registration, Application for Prevailing Wage Determination, Application for Temporary Employment Certification, or H-2B Petition and to fulfill the Secretary’s statutory mandate to certify applications only where unemployed U.S. workers capable of performing such services cannot be found. For non-compliant applications, OFLC may request more information from employers prior to possible debarment. Audits can also be used to establish a record of employer compliance or non-compliance with program requirements and because the information they contain assists DOL in determining whether it needs to further investigate an employer or its agent or attorney. In such instances, OFLC refers its audit findings and underlying documentation to DHS, WHD, or other appropriate enforcement agencies, who in their turn might conduct a targeted investigation. Moreover, DOL’s Office of Inspector General may conduct investigations of applications suspected of potential fraud. DHS can also conduct certain enforcement activities exercised through USCIS. USCIS has the authority to adjudicate the H-2B petition and conduct inquiries on the employer’s H- 2B petition, which includes the approved TLC and any supporting documentation, to prevent fraud and ensure compliance with H-2B requirements. Generally, according to DHS, it processes and approves employers’ petitions in order of receipt until the cap is reached. However, for the second half of fiscal year 2018, USCIS announced that employers had petitioned for more visas during the first five business days of the filing period than were available under the semiannual allocation. As a result, per its regulations, DHS used a computer-generated process to randomly select petitions to consider for approval. Additionally, during fiscal years 2017, 2018 and 2019, Congress enacted provisions that authorized DHS, after consultation with DOL, to make more visas available beyond the statutory cap of 66,000 if the agencies determined that the needs of U.S. businesses could not be satisfied with willing, qualified and able U.S. workers. Under these provisions, the total number of additional visas that DHS could make available could be up to the highest number of returning workers approved in any fiscal year that the returning worker exemption was in place, which was about 65,000 visas in fiscal year 2007, according to DHS and DOL (therefore up to about 131,000 visas could be made available in each of these fiscal years). The Secretary of Homeland Security, after consultation with DOL, decided to make 15,000 additional visas available for each year in fiscal years 2017 and 2018 (81,000 visas total in each year) and 30,000 additional visas for returning workers in fiscal year 2019 (96,000 visas total). The federal agencies announced the availability of these additional visas during different months, based on the date they received statutory authorization, which were all in the second half of the respective fiscal years (July 2017, May 2018, and May 2019). Employer demand for H-2B visas increased from 2010 through 2018 as the U.S. economy strengthened. The number of employer-submitted TLC applications that were certified by DOL increased in each year since 2012, and more than doubled from fiscal year 2010 (about 3,700) to 2018 (about 9,500). Additionally, the number of H-2B workers on DOL- certified applications has increased each year since 2012. In fiscal year 2018, DOL certified applications representing about 147,600 H-2B workers, about a 70 percent increase from fiscal year 2010. As the number of certified TLC applications and workers has generally increased since 2010, national unemployment has declined each year since 2010 (see fig. 2). After DOL certifies the TLC, employers petition DHS to obtain H-2B visas for the workers they plan to employ. Employers that filed petitions for H- 2B workers varied in the number of workers requested and most were concentrated in several industries. According to our analysis of DHS data, in fiscal year 2018, DHS approved petitions from about 3,700 H-2B employers. The number of H-2B visa workers that employers were approved for ranged from one to 1,169, with a median of 12 approved H- 2B workers (see fig. 3 for full distribution). Of the about 3,700 employers, 127 were approved for more than 100 visas. The employers were generally concentrated in administrative and support services (including landscaping); hospitality, amusement and recreation; forestry, fishing, and hunting; construction; and manufacturing industries (see table 1). In our analysis, we found that in 2018, H-2B employers were concentrated in 737 counties in the United States that have, on average, larger labor forces and stronger labor markets than counties without H-2B employers. For each fiscal year from 2015 through 2018, there were about 700 counties with H-2B employers and about 2,400 counties without any H-2B employers, according to our analysis of DHS CLAIMS3 data. Our analysis showed counties with H-2B employers have, on average, larger labor forces than those without H-2B employers and are located mostly along the coasts, but can be found throughout the United States (see fig. 4). Our analysis of DHS and DOL data found that counties with H-2B employers generally had lower unemployment rates and higher average weekly wages than counties that do not have any H-2B employers. Specifically, the approximate 700 counties with H-2B employers had, on average, unemployment rates that were about 0.4 of a percentage point lower than those in counties without H-2B employers. Moreover, lower unemployment was consistent in every month from fiscal years 2015 through 2018, regardless of seasonality (see fig. 5). Further, average weekly wages in counties with H-2B employers were higher by about $113 per week r than in counties without H-2B employers (average weekly wage for counties with H-2B employers is $866 and for counties without H-2B employers is $754). This relationship held for every quarter from fiscal years 2015 through 2018 (see fig. 6). The connection between strong labor markets and employers’ use of H- 2B workers may stem from multiple factors. Counties with strong labor markets may have a smaller pool of unemployed workers to fill seasonal positions leading employers in these counties to use H-2B visas as a way to fill these positions. Alternatively, counties with larger, more urban populations may have stronger labor markets. These larger population counties have more employers than smaller counties; therefore, they are more likely to have at least one employer with H-2B workers. Most selected H-2B employers we interviewed said uncertainty in getting H-2B visas is a challenge to their business planning. We interviewed and gave questionnaires to 35 H-2B employers—19 of which operated small businesses. In our interviews, 21 H-2B employers said the uncertainty of receiving H-2B visas affected their ability to plan for possible business growth and investment. Some employers explained that their operations depended on getting H- 2B workers annually and that any decrease in the number of expected H- 2B workers would substantially impact their business decisions. For example, one Texas-based landscaping employer we interviewed cited uncertainty as a reason to stop accepting new contracts and to reduce investments in new equipment, such as trucks and lawn mowers, and other landscaping supplies. In Maryland, one seafood processing employer said that because of the uncertainty related to receiving H-2B visas they could not implement planned investments, such as expanding their facilities or purchasing trucks for transporting goods, and shut down their business for a time. Similarly, one hospitality employer in Michigan told us that due to the uncertainty of getting visas, they opted not to invest in expanding their hotel amenities or make renovations. In addition, of the 35 H-2B employers we interviewed, seven said the lottery system used by DHS exacerbated the uncertainty of getting H-2B visas. Some of these seven employers described the lottery as seemingly unfair to employers who might have been long-time participants of the program and would not be able to predict if they will be getting visas. Some employers stated that they would prefer that DHS use a more equitable method to award and distribute visas, such as giving every employer a proportion of the visas they petition for. Beyond the uncertainty associated with the H-2B program, employers we spoke with reported varying business experiences during fiscal years 2017 and 2018. Specifically, the 29 H-2B employers who completed our questionnaire—15 of whom did not receive all requested H-2B visas under the standard cap in 2018—reported varied experiences in terms of revenue, purchases of goods and services for their businesses, and the employment of U.S. workers. Revenues. Employers who did not receive all requested H-2B visas under the standard cap more frequently reported revenue declines than employers who received visas, according to our analysis of the questionnaire responses (see fig. 7). Some employers reported that the loss of customers or contracts may have also contributed to these revenue declines. According to the questionnaire responses, 12 of the 14 employers who did not receive all requested H-2B visas under the standard cap reported losing customers and contracts in fiscal year 2018. However, employers’ experiences varied across industries, and other factors besides obtaining H-2B visas may have also affected revenues. For example, seafood processing employers that did not receive all requested H-2B visas under the standard cap more frequently experienced revenue declines than construction employers that did not receive all requested H-2B visas under the standard cap, as the latter may have been better positioned to mitigate the loss of H-2B workers. (Industry and location-specific factors from our case studies are discussed later in this report.) Purchases of goods and services. Based on responses to our questionnaire, employers that did not receive all requested H-2B visas under the standard cap more frequently reported declines in purchases of goods and services than employers who received visas in 2018 (see fig. 8). Employers’ decisions to delay investments on their businesses may have contributed to declines in the purchases of goods and services. Based on questionnaire responses, 11 of the 15 employers who did not receive all requested H-2B visas under the standard cap reported delayed investments in equipment or maintenance repairs. Additionally, some also reported delayed investments in business expansion. Corroborating what H-2B employers reported, nine of the 12 supply companies we interviewed in our case studies said they experienced decreased demand for their services when H-2B employers did not get visas or got them late. Similar to their experiences with revenues, employers’ reported experiences with purchases of goods and services varied across industries as other factors apart from obtaining H-2B visas may have affected employers’ purchases of goods and services. For example, more construction employers who did not receive all requested H-2B visas under the standard cap reported on their questionnaires that they could maintain their levels of purchasing goods and services than hospitality employers who did not receive all requested H-2B visas under the standard cap, possibly due to construction employers’ ability to mitigate the impacts of not receiving H-2B workers. Employment of U.S. workers. Based on our questionnaire responses, no clear pattern emerged among employers with regard to changes in the employment of U.S. workers (see fig. 9). Mainly there is no evidence of a notable number of layoffs of U.S. workers among employers that did not receive all requested H-2B visas under the standard cap. According to our questionnaire responses, three of the 15 employers who did not receive all requested H-2B visas under the standard cap in fiscal year 2018 reported having to lay off or reduce hours of U.S. workers. However, responses regarding increases in U.S. employment are difficult to interpret because our questionnaire did not ask how long newly hired employees actually stayed with employers. Local and industry-specific characteristics affected how selected employers mitigated impacts from the H-2B visa cap and may help explain the varied outcomes reported in revenue, supply purchases, and employment of U.S. workers. For example, 18 of the 35 employers we interviewed said that the characteristics of their own businesses, such as seasonality, affected how they tried to mitigate impacts from the H-2B visa cap. Employers told us that they used several methods to mitigate the effects of not having H-2B workers; however, their success in mitigating impacts varied (see table 2). Seafood processing employers on Maryland’s eastern shore—which includes Dorchester County—hire H-2B workers for picking meat out of crabs, according to a local trade association (see fig. 10). Typically, crabbing season begins on April 1st and ends in late November. These employers are heavily reliant on H-2B workers, and, on average, 54 percent of their workforce is comprised of H-2B workers for fiscal year 2018, according to questionnaire responses. Seafood processing employers we interviewed were also long-time users of the H-2B program. Of the six seafood processing employers we interviewed, five said they had participated in the H-2B visa program for more than 20 years, while the remaining employer had participated for about two years. Seafood processing employers that did not receive all requested H-2B visas under the standard cap in 2018 reported notable impacts to their businesses. Of the five seafood employers that responded to our questionnaire, three did not receive the H-2B visas they requested under the standard cap, and these employers reported that their revenue declined by more than 10 percent. All three employers attributed their revenue declines to not getting the requested H-2B workers in time for the season. Two of the employers who did not receive H-2B workers in time for the season, told us that they shut down their operations for part of the season. Moreover, seafood processing employers told us that not getting H-2B workers, or getting them late in the season, led to a reduction in U.S. employment. For example, one employer we interviewed said the use of truck drivers and administrative staff declined without H-2B workers to perform the crab picking work. In addition, all of the seafood processing employers who did not get their H-2B workers reported declines in supplies purchased (e.g., crabs, boxes, pots, and packaging). Of the five seafood supply companies we interviewed, all of them confirmed that when H-2B employers did not receive all requested H-2B visas under the standard cap, demand for their services and products declined. Employers told us that impacts of the H-2B visa cap were aggravated by several industry-specific factors. For example, one employer said the strict seasonality of crab picking made delays in receiving H-2B workers problematic. In addition, seafood employers said their efforts to recruit U.S. workers faced challenges. Different employers mentioned challenges including this strict seasonality; the nature of the work, which generally does not appeal to U.S. workers including high school and college students; and the employer’s remote location. Finally, some employers emphasized that there was not a good substitute for manual labor when they did not get H-2B workers. One seafood processing employer said the industry had tried to automate crab picking, but was unsuccessful. Selected landscaping employers we interviewed in Dallas-Ft. Worth, Texas said they typically hire H-2B workers to perform residential and commercial landscaping, such as mowing lawns, planting trees, building outdoor living spaces, and performing other lawn care maintenance (see fig. 11). Landscaping employers told us that their season can begin as early as February and can last until mid-December. On average, among the landscaping employers that responded to our questionnaire, 35 percent of their workforce was comprised of H-2B workers. Of the 11 landscaping employers we interviewed, eight said they have participated for about 10 years or more, while three said they have participated in the H-2B visa program for about three years or less. Of the 11 landscaping employers who responded to our questionnaire, three did not get all visas requested under the standard cap. All three employers who did not receive all requested H-2B visas reported revenue declines and said during our interviews that revenue declines were due to not getting H-2B workers or getting them late in the season. Moreover, of the 11 landscaping employers that responded to our questionnaire, six— including employers that did and did not receive all requested visas under the standard cap—reported declines in supply purchases. Landscaping employers told us that low local unemployment and the intensive manual labor in the heat were challenges to recruiting more U.S. workers. Of the 11 landscaping employers we interviewed, three said that when they did not get their H-2B workers, they tried to partially mitigate the situation by having existing staff work additional overtime hours. Other efforts to mitigate the impacts of having fewer H-2B workers included spreading their work across the year and helping returning H-2B workers apply for permanent residency using EB-3 visas—immigrant visas available to certain categories of skilled and unskilled workers. Some landscaping employers said that using EB-3 visas would enable them to have more workers who are permanent residents, which would help promote a more stable workforce, according to our interviews. Selected construction employers we interviewed in Maricopa County, Arizona, said they generally hire H-2B workers to perform manual labor, such as building housing panels or drywalling (see fig. 12). Construction employers said their season generally begins as early as March and lasts until November. On average, among the construction employers that responded to our questionnaire, 8.5 percent of their workforce was comprised of H-2B workers. Of the six construction companies we interviewed, all of them said they have participated in the H-2B visa program for about five years or less. The three construction employers who did not receive all requested H-2B visas under the standard cap in 2018 and responded to our questionnaire reported that they did not experience significant revenue declines. Of these three employers, two reported increased revenues between 2017 and 2018, while one did not report revenue. One employer said during interviews that had they received H-2B workers in 2018 they might have experienced a significant revenue increase compared to 2017 because of the expansion of the construction industry overall in Maricopa County. In addition, among the three construction employers who responded to our questionnaire and did not receive all requested H-2B visas under the standard cap, two reported increased supply purchases during fiscal year 2018. Although construction employers told us that recruiting more U.S. workers was challenging due to low unemployment and the manual nature of the work, several factors may have helped construction employers mitigate the impacts of the visa cap. Of the six construction employers we interviewed, two told us they attempted to mitigate impacts from the visa cap by spreading their work across the year and prebuilding housing frames during the offseason—a practice referred to as even-flowing. Moreover, some construction employers said they either subcontracted work during times they could not hire new U.S. workers, or had their existing U.S. workers work additional overtime hours. Selected hospitality employers in Mackinac Island, Michigan, and Barnstable County, Massachusetts, said they commonly hire H-2B workers to perform work such as housekeeping and working in kitchens (see fig. 13). Generally, some employers said their season begins in April and lasts through the end of October or early November. Of the 12 hospitality employers we interviewed, five said they have participated in the H-2B visa program for between five to 20 years, four said they have participated in the visa program for more than 20 years, and three did not say when they started participating in the visa program. Moreover, H-2B workers comprised an average of 35 percent of the hospitality employers’ workforce, based on questionnaire responses. Of the nine hospitality employers who responded to our questionnaire, six did not receive all requested H-2B visas they petitioned for under the standard cap in 2018. Of those six employers, three reported revenue declines in 2018, while the other three reported increased revenues. However, some hospitality employers said that the lack of H-2B workers did affect the quality of their services or led them to reduce their operations. For example, one resort we interviewed said they had to close down a signature restaurant because they did not receive the H-2B workers necessary for the season. Of the nine hospitality employers that responded to our questionnaire, five reported a decline in supply purchases for 2018. A variety of factors may help explain the outcomes for hospitality employers. On one hand, hospitality employers told us they were challenged to recruit more U.S. workers due to the seasonality of the work and sparse local population, and the fact that students are not available for the whole season. On the other hand, one hospitality employer that did not receive H-2B visas in 2018 said during interviews that they did not experience a revenue decline because guests had booked their reservations in advance. Also, hospitality employers reported using various strategies to mitigate the impact of the cap. For example, of the six hospitality employers who did not receive all requested H-2B visas in 2018, three employers hired more foreign students under the J-1 exchange program for certain students and other visitors. Moreover, four hospitality employers said they applied for H-2B visa extensions, which according to one employer are for H-2B workers already in the United States. In addition, one employer also mentioned that they contracted their housekeeping services to outside cleaning crews, which negatively affected the establishment’s quality of service. In response to the increase in demand for H-2B visas and the uncertainty employers expressed regarding whether they would be approved for workers under the H-2B visa cap, stakeholders and others have suggested changes to the H-2B program. Based on interviews with knowledgeable stakeholders and a review of their publications, we identified six proposals for changing the H-2B visa cap. In our discussion groups and interviews, 12 knowledgeable stakeholders— henceforth referred to as stakeholders—identified potential effects for each of the six proposals. As the stakeholders discussed the various policy proposals, they identified two recurring policy goals: policy proposals should (1) minimize uncertainty and (2) maintain or increase protections for U.S. and H-2B workers. We did not independently assess the individual merits or accuracy of the views expressed by these stakeholders, nor did we assess the feasibility or administrative costs of the proposals discussed. Additionally, we did not assess which options would require Congressional action or which options could be implemented through agency action. Below, we present summaries of the six proposals and some of their potential effects as identified by these stakeholders. The first two proposals listed would eliminate or adjust the cap and the remaining four would keep the current cap in place and address alternative ways to allocate visas. Shortage list. This proposal would eliminate the statutory cap and allow employers to recruit foreign workers for occupations with worker shortages. An expert commission would compile the shortage list annually, based on relevant factors, such as wage growth or job vacancies. Potential effects identified by stakeholders: It would provide more evidence-based and data-driven justifications for the number of visas and the industries/occupations that receive them. It would foster public credibility for the H-2B visa program because it demonstrates a bona fide need for H-2B workers. It would accelerate the H-2B visa approval process for certain industries. Because wage growth would be an indicator of occupational shortages, it may incentivize employers in major H-2B industries to offer higher wages, if economically beneficial. Some employers approved under the current system would not be approved for H-2B visa workers because their occupations are not on the shortage list. It may lack accuracy because national level occupational shortages may not reflect shortages in certain industries and occupations within specific locations or identify local labor market trends. BLS data may not accurately capture such trends. Annual adjustment. This proposal would adjust the cap annually (either up or down) based on economic indicators such as unemployment rate or number of TLC applications approved by DOL. Potential effects identified by stakeholders: It would allow employers to use H-2B workers when U.S. workers are not available due to low unemployment and revert to U.S. workers in times of higher unemployment. Having a flexible cap could be more predictable than the current system. It would be a more accurate reflection of need than using an arbitrary cap. While not discussed in the proposal language, if wage growth is also considered as an economic indicator in the annual adjustment, it might incentivize employers to improve wages, if economically beneficial. Using a national indicator would not fully reflect localized needs for H-2B workers. It would put DOL in a position where it would be determining employers’ needs. Using approved TLC applications is not a good measure of demand because they may not reflect demand for labor. Any delays in processing TLC applications could lead to difficulties in determining the annual adjustment in a timely manner. Returning workers exemption. This proposal would retain the current H-2B visa cap of 66,000 and make the returning worker exemption permanent. Potential effects identified by stakeholders: It could lead to increased predictability. Employers would have more certainty on whether they will be approved for H-2B visas, and H-2B workers would know whether they would have the option to return to their jobs in the United States. There is familiarity—among employers, H-2B workers, and administrators—with returning worker exemption as it has been implemented before. It may be more efficient for employers as returning workers already have training. There could be potential cost savings for program as returning workers have already been vetted. It rewards both workers and employers who are compliant with the H-2B program. A permanent returning worker exemption, like any proposed reform that involves eliminating or increasing the cap, requires better enforcement of worker protections. It could increase the possibility that H-2B workers return to poor working conditions because they have no other economic options. One stakeholder said this could be mitigated by allowing returning workers the flexibility to work for different employers than they worked for in prior years if so desired. Priority list. This proposal would retain the current H-2B visa cap of 66,000 and give priority to applications from employers that offer the highest wages or better working conditions. Potential effects identified by stakeholders: It creates incentives for employers to improve working conditions. It may be easy to implement under current law, and may not require new legislation. It alleviates problems associated with calculating the prevailing wage. It does not account for the wage variation among small and large employers, geographical locations, or industries. Using the highest wages to allocate the visas skews the program to certain occupations and higher-paying geographical locations (even within the same industries and among similarly sized employers). If based solely on wages, a priority list could penalize employers that also have to provide workers with additional benefits such as housing at no cost. It would need to be combined with stronger enforcement, such as employer audits, to ensure that workers are getting paid the promised higher wage or better conditions. Quarterly allocation. This proposal would retain the current H-2B visa cap of 66,000 and allocate visas quarterly rather than twice a year. Potential effects identified by stakeholders: It might improve fairness for employers whose season starts late in the semiannual allocations. It helps ease the burden on DOL’s computer system. It reduces the number of employers applying for visas before their period of need and spreads demand more evenly across the year. It does not seem to mitigate the issue of having demand exceed the cap. In practical terms, quarterly allocation would result in shifting visas away from certain employers and toward others. Demand for H- 2Bs is especially high in April to June, the third quarter of the fiscal year. This option would reduce the number of visas for the third quarter and shift more visas to the fourth quarter. Auction. This proposal would retain the current H-2B visa cap of 66,000 and the visas would be auctioned to the highest employer bidders. Potential effects identified by stakeholders: It uses market forces; employers evaluate how much an H-2B worker is worth. It demonstrates the economic cost of keeping the cap low and determines whether employers are strictly looking for cheap labor. Auction revenues could be used to ensure the H-2B program has less adverse effects on U.S. and H-2B workers, raises wages, or leads to more audits by DOL. Depending on the design of the auction, it may create a system where larger, better funded employers unfairly benefit. It does not address issues of uncertainty faced by employers of H- 2B workers. It increases labor costs which could reduce the profitability using H-2B workers. DHS, in consultation with DOL, has identified some alternatives to the current approach for allocating H-2B visas. In the Joint Explanatory Statement accompanying the fiscal year 2018 DHS Appropriations Act, Congress directed DHS—in consultation with DOL—to review and report on options for addressing the problem of unavailability of H-2B visas for employers that need foreign workers late in each semiannual period of visa availability. In response, DHS issued a report to Congress in June 2019 that laid out six approaches for revising how H-2B visas are allocated among employers—some of which were similar to the proposals identified above. The DHS options include (1) a merit-based system for eligibility that prioritizes employers that have made a significant contribution to the U.S. economy, (2) designation of eligible occupations or industries based on factors such as industry unemployment rates, and (3) distributing visas on a quarterly basis. DHS has not assessed which of the options outlined in the June 2019 report could be implemented by agency action alone and which would require Congressional action, nor has it identified which options have the greatest potential benefit for employers. DHS officials have told us that they currently lack the resources to assess or implement the proposals from their June 2019 report or any other alternatives and, while an assessment may be possible in the future, it would have to be balanced against other administration priorities. Standards for internal control in the federal government call on agencies to identify, analyze, and respond to significant change, including change in the economic environment. Moving forward with assessing available reform options would position DHS and DOL to better inform their own and Congress’s decision- making. In determining the number of additional H-2B visas to make available beyond the standard cap in fiscal years 2017 to 2019, DHS—in consultation with DOL—relied on data from prior years. In each of the three years, federal law authorized DHS after consultation with DOL to provide additional H-2B visas beyond the standard cap if the needs of U.S. businesses could not be met with U.S. workers, up to the maximum number of H-2B returning workers in any prior year when the returning worker exemption was in effect (about 65,000 in 2007, according to the agencies). DHS made up to 15,000 additional visas available in fiscal years 2017 and 2018 and up to 30,000 in 2019. In each year, DHS in consultation with DOL determined the appropriate number of additional visas by looking at demand for visas in prior years. Specifically, in 2017 it determined that 15,000 visas would be sufficient to at least meet the same level of demand as in fiscal year 2016. In 2018, DHS used the same rationale to determine that up to 15,000 additional visas would again be sufficient, based on experience with the additional visas in 2017. Most recently, in 2019, DHS in consultation with DOL raised the number of additional visas to 30,000 in recognition partly of the higher demand in 2018—when employers filed petitions for about 29,000 visas during the first five days of the filing period for additional visas. The demand for returning H-2B workers in prior years and the amount of time remaining in the fiscal year were also factors in the agencies’ decision about how many additional visas to provide. However, using demand in prior years as the primary basis for setting the number of additional visas in the current year is not consistent with standards for internal control in the federal government, which call for agencies to identify, analyze, and respond to significant change, including change in the economic environment. Indeed, the outcome in 2018, when DHS made 15,000 additional visas available but employers applied for almost 30,000 visas, demonstrates the potential limitations of relying solely on past demand as a predictor of future demand. Examples of other types of data that may be relevant to gauging trends in employer demand include unemployment rate, employment, and earnings, which we have previously identified as potential indicators of labor market shortages. Some stakeholders have also suggested that the number of H-2B workers on approved TLC applications is a good measure of visa demand. The agencies said in the 2018 and 2019 temporary rules making additional visas available that they did not have enough time remaining in those fiscal years to conduct a more formal analysis of the adverse effects on U.S. workers that may result from a broader cap increase.Assessing the advantages and disadvantages of considering current economic trends in addition to past demand would help the agencies decide if such an approach would be a better way to estimate employer need in any future years when Congress authorizes visas beyond the H-2B standard cap. According to DHS and DOL, the agencies have also sought to balance employers’ hiring needs and the interests of U.S. workers by setting a higher standard that employers must meet to qualify for additional H-2B visas. To qualify for visas under the standard cap, employers must have an approved TLC, demonstrating, among other things, that they have a temporary need for labor and have taken steps to recruit workers in the United States. From 2017 to 2019, employers applying for the additional visas were also required to attest that without the visas, they were likely to suffer irreparable harm, i.e., suffer a severe and permanent financial loss. According to the 2017 temporary rule announcing the availability of additional H-2B visas above the statutory cap, DHS decided to focus on businesses likely to suffer a severe and permanent financial loss, in part, to be responsive to some stakeholders that U.S. workers could potentially be adversely affected by a general cap increase applicable to all potential employers. To support their attestation of severe and permanent financial loss, employers were required to retain documentation, such as contracts, reservations, or orders that would have to be cancelled absent the requested H-2B workers. DOL officials told us the agency’s Wage and Hour Division evaluates the sufficiency of this documentation in the course of its investigations of H-2B employers, when applicable. Officials said they examine documentation related to loss of contracts and dependence on H-2B workers, among other things, in order to detect significant and voluntary violations of program requirements. DOL has sought to address rising demand for TLCs and H-2B visas through changes to how it assigns TLC applications to analysts for review and processing. Prior to 2018, DOL processed applications sequentially according to the day they were received, and released certifications on a rolling basis as all requirements for certification were met. DOL reported that on January 1, 2018, the first day of the filing period for employers seeking workers to start on April 1, 2018, it received approximately 4,498 applications covering 81,008 worker positions, exceeding the annual visa allotment by nearly 250 percent. According to the agency, this was the first time in recent years that this had happened. On January 17, 208, agency officials announced that beginning February 20, 2018, they would begin to release certified applications sequentially according to the day and time of receipt. This in turn led to a large number of employers with approved TLCs submitting their H-2B visa petitions within a small window. DHS officials explained that receiving a large volume of petitions in a short time frame required USCIS to approve petitions following random selection. In June 2018, anticipating further increases in applications, DOL announced that it would sequentially assign applications to analysts in order of day and—in an adjustment from the earlier procedures—time of receipt to the millisecond. Once applications were assigned, analysts would initiate review of applications in the order of receipt date and time, issue first actions on a rolling basis, and issue certifications as all regulatory requirements were met. DOL reported that in January 2019, it received approximately 5,276 applications covering more than 96,400 worker positions for start dates of work on April 1, exceeding the semiannual visa allocation by nearly 300 percent. Furthermore, DOL reported that on January 1, 2019, within the first five minutes of the filing period for April 1 start dates of employment, the agency’s network infrastructure supporting OFLC’s electronic filing system experienced almost 23,000 log-in attempts, in contrast with 721 attempts in the same time period in 2018. This volume of simultaneous system users caused the electronic filing system to become unresponsive, preventing nearly all employers from submitting applications until the system reopened on January 7, 2019.. DOL’s Office of Foreign Labor Certification conducts recordkeeping audits of adjudicated TLCs to assess employers’ compliance with the terms and conditions attested to in their applications and to fulfill the Secretary’s statutory mandate to certify applications only where unemployed U.S. workers capable of performing the needed work cannot be found. DOL officials told us the agency reviews the original TLC application and requests additional documentation of the employer’s activities when conducting audits to determine whether the employer is in compliance with program requirements. Specifically, employers with minor violations receive a warning; violations described in 20 C.F.R. § 655.71 could lead to increased DOL monitoring and assistance with the employer’s recruitment efforts; and employers with violations described in 20 C.F.R. § 655.73 could be debarred from the H-2B program. DOL is defending a challenge to its implementation of the randomization process for assigning applications filed by employers seeking H-2B visas, in Padilla Construction Co. v. Scalia, No. 2:18-cv-01214-GW-AGR (C.D. Cal.). While DOL has changed its TLC procedures so they call for a randomization process on an on-going basis, DHS generally processes employers’ petitions on a first-come, first- served basis except when a large number of petitions are received in the first five days of the filing period. officials reported that during fiscal year 2018 they initiated 493 audits of H-2B employers, representing seven percent of all employers with approved TLCs issued during the year. They also reported that of the 503 audits completed during fiscal year 2018, which includes audits initiated during 2017, more than half resulted in a warning letter being sent to the employer, with only a small number finding more serious violations (see fig. 14). In our review of a non-generalizable sample of letters sent to H-2B employers with audit results, we found several examples of the types of issues identified by DOL. Several warning letters noted violations related to the period of employment of H-2B workers, such as failing to notify OFLC when H-2B workers left their jobs earlier than planned. In letters of assisted recruitment that we reviewed, employer violations included failure to accurately advertise rates of pay and failure to meet requirements for posting job advertisements in newspapers. Finally, the debarment letters we reviewed cited the employer’s failure to provide the documentation that DOL requested as part of the audit. DOL has not taken a risk-based approach to selecting employers to audit. OFLC’s Certifying Officer has the sole discretion to choose the applications selected for audit, including selecting applications using a random assignment method. DOL officials said the agency has for the most part randomly selected H-2B employers for audits, although they also select some employers because of a prior violation. Officials said that the system currently used to track audits captures data on audit workloads and final audit outcomes, but the agency has a plan to develop a new system that would also track the individual violations found in audits and the industry and job classification associated with the employer. With this capacity, officials said they could take a more risk- based approach to selecting employers for audits, based on trends in violations by industry or job classification. However, officials said that the further development and implementation of this tracking system is currently on hold due to resource constraints with no firm date for moving forward. Standards for internal control in the federal government call on agencies to identify, analyze, and respond to risks to meeting their objectives. Until it implements a risk-based approach to selecting H-2B employers for audits, DOL may miss opportunities to allocate its limited audit resources more efficiently and to detect violations that could adversely affect U.S. and H-2B workers. Taking a more targeted approach is especially important in light of a 2019 Office of Inspector General (OIG) report that stated over the past decade, the OIG and other federal agencies have conducted over 70 criminal investigations in the H- 2B program related to potential fraud involving employers, attorneys, and others. DOL also works to protect U.S. workers through setting the prevailing wage that employers must pay and has taken steps to enhance the accuracy of its prevailing wage determination by limiting the use of employer-provided wage surveys. DOL is responsible for determining the prevailing wage applicable to an H-2B application. An employer must pay a wage at least equal to the prevailing wage obtained from the National Prevailing Wage center within OFLC, or the federal, state, or local minimum wage, whichever is the highest. The prevailing wage that H-2B employers must pay their H-2B and U.S. workers is set by BLS’s Occupational Employment Statistics (OES) survey in all cases except when a wage is set by a valid and controlling collective bargaining agreement or the employer submits an employer-provided survey that meets DOL’s requirements. When they promulgated a final rule in 2015 on the methodology for determining the prevailing wages to be paid H-2B workers, DHS and DOL decided that it would limit the circumstances under which employers may use employer-provided wage surveys to set the prevailing wage. The preamble to the rule described a court decision that found that DOL had arbitrarily allowed wealthy employers to pay for expensive private surveys when other employers in the same occupation who could not afford to conduct such surveys paid the higher OES mean wage. In light of this decision, as well as DOL’s own experience that employer-provided surveys are not any more consistent or reliable, and concerns raised by worker advocates, the agencies determined that the options for accepting employer-provided surveys are more limited. The 2015 regulations require, among other things, that employer-provided surveys be conducted independently by a state agency or university, and meet certain methodological standards. Since 2014, the proportion of H- 2B employers using employer-provided wage surveys to set the prevailing wage has declined from almost 20 percent to less than one percent according to our analysis of DOL data (see fig. 15). DOL officials told us the most significant contributor to the decline in employer-provided wage surveys was the requirement to have a state agency or university independently conduct employer-provided wage surveys—prohibiting employers from directly paying for these surveys. Officials also said that the seafood industry in locations such as Maryland and Louisiana continues to use employer-provided wage surveys, as state agencies have long histories of conducting wage surveys for seafood employers in these areas. Employers we interviewed who depend on temporary foreign labor said the statutory cap on H-2B visas presents challenges for them, and these challenges can be driven at least partly by demand that fluctuates with the economy. Some employers—for example, those with fewer local workers available for hire—may face greater financial risks than others when they are denied H-2B workers due to the cap. More broadly, H-2B employers are challenged by uncertainty regarding whether they will receive H-2B workers in any given year, complicating their efforts to plan future operations, such as expansion or investment. DHS and DOL have taken an important first step towards addressing these challenges by identifying options for allocating visas. However, until the agencies assess such options, they cannot determine which, if any, to implement under their current authority or what legislative changes may be needed to improve the program. In the meantime, as long as DHS and DOL continue to rely primarily on prior year demand to determine the appropriate number of additional visas to make available beyond the standard cap—when granted this authority by Congress—the agencies may miss an opportunity to leverage data on current economic trends and other factors. Assessing the advantages and disadvantages of using current economic data would help the agencies determine the feasibility of more accurate projections, which would help mitigate uncertainty and related challenges for H-2B employers. The steps DOL has taken in recent years to enforce worker protection requirements and promote accurate wage levels so as not to undermine U.S. workers show promise. However, until DOL moves ahead with taking a more targeted approach to selecting employers for audits, it may miss opportunities to efficiently leverage the scarce resources available to identify and prevent worker protection violations. The Director of United States Citizenship and Immigration Services should work with the Assistant Secretary for the Employment and Training Administration to assess options for changing the H-2B visa program and, as warranted, implement changes or submit proposed legislative changes to Congress. DHS and DOL could consider options included in their June 2019 report to Congress and identify those that may be implemented cost effectively and without adversely affecting U.S. workers. (Recommendation 1) The Assistant Secretary for the Employment and Training Administration should work with the Director of United States Citizenship and Immigration Services to assess options for changing the H-2B visa program and, as warranted, implement changes or submit proposed legislative changes to Congress. DOL and DHS could consider options included in their June 2019 report to Congress and identify those that may be implemented cost effectively and without adversely affecting U.S. workers. (Recommendation 2) The Director of United States Citizenship and Immigration Services should work with the Assistant Secretary for the Employment and Training Administration to assess the advantages and disadvantages of considering current economic trends in determining the appropriate number of additional H-2B visas to provide when given this authority by Congress and, as warranted, implement an approach that considers such trends. (Recommendation 3) The Assistant Secretary for the Employment and Training Administration should work with the Director of United States Citizenship and Immigration Services to assess the advantages and disadvantages of considering current economic trends in determining the appropriate number of additional H-2B visas to provide when given this authority by Congress and, as warranted, implement an approach that considers such trends. (Recommendation 4) The Assistant Secretary for the Employment and Training Administration should take steps to target its audits of H-2B employers to employers with the highest likelihood of violating program requirements; such steps could include moving ahead with developing a system for identifying trends in H-2B employer audit outcomes. (Recommendation 5) We provided a draft of this report to DHS and DOL for their review and comment. Both agencies provided written comments, which are reproduced in appendices III and IV, respectively. Both agencies also provided technical comments, which we incorporated as appropriate. In its comments, DHS agreed with our first recommendation to assess options for changing the H-2B visa program, and noted that it plans to work further with DOL to explore options for improving the H-2B visa program and possibly develop proposals for legislative changes. DHS did not agree with our third recommendation to assess the advantages and disadvantages of considering current economic trends—which was the other recommendation we directed to the agency. Specifically, DHS said it would continue to work with DOL—as it has done in prior years--if and when Congress delegates the authority to make additional H-2B visas available beyond the statutory cap to DHS. The agency also expressed its view that Congress is better positioned to determine whether and how many additional visas should be made available to meet the needs of U.S. businesses. In fiscal years 2017 through 2020, DHS was authorized to increase the number of H-2B visas beyond the statutory cap, after consulting with DOL to determine that “the needs of American businesses be satisfied…with United States workers...” In exercising this authority in prior years, DHS stated that “he scope of the assessment called for by the statute is quite broad, and accordingly delegates the Secretary of Homeland Security broad discretion to identify the business needs he finds most relevant.” In light of DHS’s broad view of its authority, we continue to believe that it would be appropriate for DHS, in consultation with DOL, to assess the advantages and disadvantages of considering current economic trends in determining the appropriate number of additional H-2B visas to provide. If they determine that using such data would be warranted, the agencies would then be well positioned to implement such an approach if DHS is granted such authority in the future. Moreover, if—as DHS stated in its response to our recommendation—the agency believes that Congress is best suited to determine what increases in visa numbers may be needed to meet the needs of U.S. businesses, consistent with protecting American workers, it may wish to work with Congress to draft a legislative proposal reflecting this view. DOL agreed with the three recommendations addressed to it. Regarding our second recommendation to work with DHS to assess options for changing the H-2B visa program, DOL said it is prepared to work with DHS to consider options for changing the H-2B program and to provide any technical assistance that Congress may need on this issue. Regarding our fourth recommendation, DOL said it is prepared to draw on its data on labor market and economic trends to provide technical assistance to DHS on the determination of how many additional H-2B visas to make available. Regarding our fifth recommendation, DOL noted that while further development of a system for tracking industry and occupational trends in H-2B employer violations is currently on hold due to budgetary constraints, when this system is available it will provide the capacity to take a risk-based approach to selecting employers for audits. We are sending copies of this report to applicable Congressional committees, the Secretary of Homeland Security, the Secretary of Labor, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix V. Our review: (1) describes trends in the demand for H-2B workers, (2) describes selected employers’ reports of how the visa cap has influenced their economic performance and employment of U.S. workers, (3) summarizes proposals for adjusting the H-2B statutory cap or how visas are allocated, and (4) assesses how the federal agencies that administer H-2B visas sought to meet employers’ H-2B hiring needs and protect U.S. workers. To address our first objective, we analyzed administrative data sets from the Department of Homeland Security (DHS), the Department or Labor (DOL) Employment and Training Administration, and the Bureau of Labor Statistics (BLS). To address our second objective, we conducted case studies of four industries in specific locations. To address our third objective, we held discussion groups and conducted interviews with knowledgeable stakeholders regarding proposals to change the H-2B visa cap we had identified through background research. To address our fourth objective, we reviewed relevant federal laws, regulations, and other documents; reviewed agency data; and interviewed federal officials. We used DOL temporary labor certification (TLC) data and national unemployment rate statistics for fiscal years 2010 through 2018 to provide trends in number of applications DOL has received and national unemployment rate. The TLC data are administrative data on applications from employers for H-2B visas, which we found sufficiently reliable for our purposes after reviewing technical documentation and interviewing knowledgeable agency officials. DOL releases public disclosure files that contain administrative data from employers’ H-2B applications for TLC. Our analysis took the public disclosure files and reported the number of certified applications and workers for each fiscal year from 2010 through 2018. In order to report the national unemployment rate for the United States, we used BLS’ report on historical national unemployment rates. To address how counties with H-2B employers compare to counties without H-2B employers, we utilized several administrative data sets. We used DHS Computer Linked Application Information Management System (CLAIMS3) data, which we found sufficiently reliable for our purposes by reviewing technical documentation, interviewing knowledgeable agency officials, and electronic testing of data, to identify the counties with H-2B employers for each fiscal year from 2015 through 2018. The CLAIMS3 data track all petitions for H-2B visas (as well as other visas). These data include employer address and number of H-2B visas approved. Using the employer address information, we identified the county in which H-2B visa employer is located. After the county is identified, we then aggregated all of the approved H-2B visa petitions within each county. After identifying the counties with H-2B employers, we then combined this with BLS data sets—Local Area Unemployment Statistics (LAUS) and Quarterly Census of Employment and Wages (QCEW)—which we found sufficiently reliable after reviewing technical documentation to get county- level data on unemployment rate, labor force, and average weekly wages to make county-level comparisons. The LAUS is a federal-state cooperative effort in which monthly estimates of total employment and unemployment are prepared for counties and county-equivalents. From this data set, we used the unemployment rate and the labor force statistics by county. The QCEW program publishes a quarterly count of employment and wages reported by employers. From this data set, we used the average weekly wages data across counties for fiscal years 2015 through 2018. After we had combined the CLAIMS3 data with the LAUS and QCEW data sets, we compared summary statistics on unemployment rates and average weekly wages for counties with H-2B employers to counties without H-2B employers. The average weekly wages were inflation adjusted at the state level to constant 2018 dollars. To check whether our results of the comparison were being driven by a few outlying counties, we performed several additional analyses. To see if the results were being driven by counties that relied more heavily on H-2B visas, we created quartiles using the number of H-2B petitions approved within a county and also created quartiles using the percentage of H-2B visas as a percent of the total labor force. Next, in order to determine if the results were because of the population sizes of the counties, we spilt the counties in quartiles based on the size of labor force to compare counties with and without H-2B employers by similar sized counties by population. Finally, we incorporated TLC data on industries to provide comparisons between our selected industries noted above (see appendix II). To examine the experiences of H-2B employers and their suppliers with the H-2B program in recent years we conducted case studies of four industries in specific locations: seafood processing in Dorchester County, Maryland; landscaping in Dallas County, Texas; construction in Maricopa County, Arizona; and hospitality in Mackinac County, Michigan (hotels), and Barnstable County, Massachusetts (restaurants). (See fig. 16). We selected these industries because they were among the heaviest users of the H-2B program in fiscal year 2018. Using DOL data on fiscal year 2018 TLCs, we determined the total number of H-2B workers approved across all TLCs associated with each NAICS code, and then identified the NAICS codes with the greatest number of approved workers. The four selected industries were all among the ten leading industries in terms of number of approved workers (see table 3). Amusement, gambling, and recreation industries and support activities for forestry were also among the top ten. However, representatives of these industries told us that employers typically move from location to location during their seasons, making it difficult to conduct a case study of employers in a particular location. For each industry, we selected one or two counties in which to conduct our case study. We selected these counties to achieve diversity in several factors: the total number of H-2B workers approved for employers in the county in fiscal year 2018; the number of H-2B workers approved under TLCs associated with that particular industry in the county in fiscal year 2018 (e.g., the number of H-2B landscaping or hospitality workers); the proportion of all workers in the county who are H-2B workers in 2018; the proportion of workers in that particular industry that are H-2B workers in the county in 2018 (e.g., the proportion of all landscaping workers in the county that are H-2B workers); county unemployment rate in January 2018; and geographic location (see table 4). As part of each case study, we interviewed H-2B employers who received visas during fiscal year 2018, H-2B employers who did not receive visas during fiscal year 2018, and businesses who supply goods or services to H-2B employers. Across the case studies, we interviewed 15 H-2B employers who received visas, 20 H-2B employers who did not receive visas, and 12 supplier businesses. We conducted a mix of individual and group interviews with employers, and generally used the same questions for each category of employers across industries. For all of our case studies, we worked with industry groups to recruit employers to participate in our interviews. These industry groups reached out to local employers to identify H-2B employers and in some cases also supplier businesses who would be willing to speak with us. In a few cases, we also identified supplier businesses for interviews through our case study interviews with employers. In our interviews with employers, we asked about topics including their efforts to recruit U.S. workers, their experiences with the H-2B program in recent years, any impacts on their businesses of being denied H-2B visas, actions taken to adapt to not receiving visas, and any impacts on supplier businesses of being denied H-2B visas. Besides interviewing employers, we also interviewed a state workforce agency as part of each case study, asking questions about topics including the agency’s role in helping H-2B employers recruit U.S. workers, the outcomes of H-2B employers’ recruitment efforts, and any challenges with such recruiting efforts. In addition, as part of our case studies, we asked the H-2B employers we interviewed to complete a questionnaire. This questionnaire covered topics including the employer’s gross sales in fiscal years 2017 and 2018; the employer’s number of employees in fiscal years 2017 and 2018, both U.S. and H-2B employees; the employer’s purchases of goods and services in fiscal years 2017 and 2018; and any challenges created by not receiving H-2B visas in fiscal year 2018. We received responses from 30 employers, including from five seafood processing employers, 11 landscaping employers, four construction employers, and 10 hospitality employers. Some respondents did not answer every question in the questionnaire. We dropped one of the 30 questionnaire responses from our analyses because the employer reported not receiving H-2B visas in 2017 which, if included, could distort our findings. In our analysis of changes to revenues, supply purchases, and employment based on questionnaire responses, we did not control for factors beyond the H-2B visa cap that may have affected the results. So, any results reported from the questionnaire may be due in part to these unobserved factors. Additionally, we did not independently verify the information provided in the questionnaire responses, which could lead to our analysis not completely representing the full effect of the H-2B visa cap. Finally, the questionnaire responses we received are representative of only the firms that responded and may not be more widely generalizable to the industry level or larger geographic regions. As we performed background research on H-2B visas and the cap, we interviewed several knowledgeable stakeholders. We then identified the proposals for changing the H-2B visa cap in the background interviews and publications of these stakeholders (see table 5). To address what options have been proposed for adjusting the H-2B statutory cap or how visas are allocated, we interviewed 12 knowledgeable stakeholders across two discussion groups and two interviews. The discussion groups were held on July 25, 2019 and July 29, 2019, and the interviews were held on August 1, 2019 and August 12, 2019. As part of our discussion with the experts and knowledgeable stakeholders, we asked for additional proposals that were not included in the six identified in the above table. This discussion led to additional proposals. These additional proposals are presented in table 6. We used several approaches to begin identifying potential stakeholders on the H-2B visa program. First, we reviewed our background interviews with stakeholders for this engagement to craft a preliminary list of potential individuals to contact. Then, we identified additional researchers that have published works on the H-2B visa program. Afterward, we conducted several searches on the Congressional Quarterly website to collect a list of witnesses who testified before Congress on H-2B visa issues. Finally, obtained an additional list of authors who published work on the H-2B visa program and names of individuals that have testified before Congress on issues related to the H-2B visa program. Through this process, we identified 22 stakeholders to be included in our discussion groups and interviews. We selected 12 knowledgeable stakeholders based on several criteria: published work on the H-2B visa program and number of times publications have been cited by other scholars, testified before Congress on H-2B visa issues, advocated for relevant stakeholder groups interested in the H-2B visa program, and identified by peers as being a knowledgeable stakeholder on the H-2B visa program. We also sought to achieve a balance of perspectives by the selected knowledgeable stakeholders (see table 7). To assess DHS’s and DOL’s efforts to meet employers’ hiring needs and protect U.S. workers, we reviewed relevant federal laws, regulations, and documents such as agency procedures and visa application forms. We interviewed DHS and DOL officials. We reviewed DOL data on the number and outcomes of audits conducted of H-2B employers during fiscal year 2018. We assessed the reliability of these data by interviewing DOL officials, and found them to be sufficiently reliable for our reporting purpose, which was to present a summary of the agency’s H-2B audit program in fiscal year 2018. We reviewed 25 letters that DOL sent to H- 2B employers as part of audits completed from September 14, 2017, through April 5, 2019, including eight requests for supplemental information, six warning letters, six assisted recruitment letters, and five debarment letters. The samples of requests for supplemental information, warning letters, and assisted recruitment letters were non-generalizable samples of all letters in these categories. They were judgmentally selected from a randomly generated sample of all letters in the universe to achieve diversity in terms of employer industry and location, among other things. The debarment letters we reviewed represented the full universe of such letters. In our review of the letters, one analyst identified issues discussed in each letter and placed them in broader categories, another analyst verified the issues and categories, and any differences in interpretation were resolved. We analyzed DOL data on how prevailing wage levels were determined for H-2B employers for fiscal years 2014 to 2018. We assessed the reliability of these data through review of related documentation and interviews with DOL officials, and found the data to be sufficiently reliable for our reporting purpose, which was to present the trends in how prevailing wage was set among H-2B employers over a 5- year period. After identifying DHS’s and DOL’s actions through methods such as reviewing documents and interviewing agency officials, we assessed them according to standards for internal control in the federal government related to identifying and responding to change and risk. In our analysis of Department of Homeland Security and Department of Labor data, we found that counties with H-2B employers have lower unemployment rates and higher average weekly wages than counties without H-2B employers. We extended this analysis to determine whether the results are robust to changes in labor force, employers’ usage of H-2B workers, and industries. To see if our results were being driven by larger population counties, we separated counties into quartiles by labor force and compared similar- sized counties. Looking at the top quartile, we found that, similar to our main results, counties with H-2B employers had about 0.3 percentage point lower unemployment rate and about $120 higher average weekly wage than counties without H-2B employers. For the bottom quartile, counties with H-2B employers had about a 0.1 percentage point lower unemployment rate than counties without H-2B employers, but about $34 lower average weekly wages, which we discuss further in the following paragraph. We next split the counties with H-2B employers by their usage of H-2B employees to analyze the connection between intensity of employer usage and strong labor markets. The first way we measure usage of H-2B employees is by the number of approved H-2B petitions within the county. When we compare the top quartile of counties by number of approved H- 2B petitions to all counties without H-2B employers, we found that they have about 0.5 percentage point lower unemployment rates and about $187 higher average weekly wages. We also used the ratio of approved H-2B visas to the county’s labor force population to capture the counties’ reliance on H-2B visas. When we compare the top quartile of counties by proportion of approved H-2B visas to labor force, we find that their unemployment rate is about 0.1 percentage point lower and average weekly wages about $4 higher than counties without H-2B employers. The small difference in wages for counties with a high ratio of H-2B workers to labor force, and the previous finding that counties with H-2B employers in the bottom quartile by labor force have lower average weekly wages, suggests that the difference in wages in our main finding may be partially driven by the counties with larger labor forces. In our final extension of our analysis, we isolated four selected industries to compare whether the counties with H-2B employers within the specified industry have higher average weekly wages in that industry than counties without. In this analysis of fiscal year 2018, we found that for each industry (construction, seafood processing, hospitality, and landscaping) the counties with H-2B employers within the industry have higher average weekly wages than counties without H-2B employers in the industry. These higher average weekly wages for counties with H-2B employers in the industry ranged from about $96 higher for seafood processing to about $238 higher for landscaping. Cindy Brown Barnes (202) 512-7215 or brownbarnesc@gao.gov In addition to the individual named above, Nagla’a El-Hodiri, Assistant Director; Lorin Obler, Analyst in Charge; Genesis Galo, Michael Naretta, Alejandro Oliva, and Sonya Zhu made key contributions to this report. Also contributing to this report were Amy Anderson, Susan Aschoff, James Bennett, Kathryn Bernet, Colleen Candrl, Sherwin Chapman, Pin- En Annie Chou, Pamela Davidson, Rebecca Gambler, Joel Green, Kristy Kennedy, Grant Mallie, Sheila R. McCoy, John Mingus, James Rebbe, Oliver Richard, Margie Shields, Ardith Spence, Almeta Spencer, Kathleen van Gelder, and Jessica Yutzy. Nonimmigrant Visas: Outcomes of Applications and Changes in Response to 2017 Executive Actions. GAO-18-608. Washington, D.C.: August 7, 2018. H-2A and H-2B Visa Programs: Increased Protections Needed for Foreign Workers. GAO-15-154. Washington, D.C.: March 6, 2015. H-2A Visa Program: Modernization and Improved Guidance Could Reduce Employer Application Burden. GAO-12-706. Washington, D.C.: September 12, 2012. H-1B Visa Program: Reforms Are Needed to Minimize the Risks and Costs of Current Program. GAO-11-26. Washington, D.C.: January 14, 2011. H-2B Visa Program: Closed Civil and Criminal Cases Illustrate Instances of H-2B Workers Being Targets of Fraud and Abuse. GAO-10-1053. Washington, D.C.: September 30, 2010. H-1B Visa Program: Labor Could Improve Its Oversight and Increase Information Sharing with Homeland Security. GAO-06-720. Washington, D.C.: June 22, 2006.", "summary": "Since 1990, there has been an annual statutory cap of 66,000 on the number of H-2B visa holders who can work for U.S. employers. DHS administers the program with support from other federal agencies including DOL. In recent years, demand for H-2B visas has exceeded the cap. To meet the needs of U.S. businesses, Congress authorized additional visas in fiscal years 2017-2019. GAO was asked to examine the effects of the annual cap on employers and U.S. workers. This report examines, among other objectives: (1) trends in the demand for H-2B visa workers, (2) selected employers' reports of the visa cap's influence on their performance and employment of U.S. workers, and (3) how federal agencies have sought to meet employers' H-2B hiring needs and protect U.S. workers. GAO analyzed nationwide data on H-2B visas and county labor market indicators. GAO interviewed 35 H-2B employers in four industries that are among the largest users of H-2B visas. The employers were in five counties selected for variation in factors including the share of H-2B workers in the workforce and the unemployment rate. GAO also reviewed relevant federal laws, regulations, and documents and interviewed federal officials and stakeholders. Employer demand for H-2B visa workers has increased as the national unemployment rate has declined. H-2B visas are intended to help employers fill temporary, non-agricultural positions when no U.S. workers are available and are subject to an annual statutory cap of 66,000. From 2010 to 2018, the number of H-2B workers requested on employer applications increased from about 86,600 to 147,600. Regarding local economic conditions, GAO found that counties with H-2B employers generally had lower unemployment rates and higher weekly wages than those without H-2B employers. Most of the 35 H-2B employers GAO interviewed said that business planning was affected by uncertainty about whether they would be able to hire the number of H-2B visa workers they requested given the statutory cap. Employers who did not receive all H-2B visas requested under the statutory cap in 2018 were somewhat more likely than those who did to report declines in revenue (see figure) and purchases of goods and services. However, GAO found no clear pattern in changes to the number of U.S. workers hired by these employers. Employers interviewed by GAO varied in how they adjusted to having fewer H-2B workers. For example, two seafood employers reported shutting down operations in the absence of H-2B workers, and employers said that barriers to finding U.S. workers included remote location and seasonality of the work. Federal agencies have identified program changes that consider employers' hiring needs and protect U.S. workers, but gaps remain in implementation. The Department of Homeland Security (DHS), in consultation with the Department of Labor (DOL), has identified options for changing the H-2B visa allocation process to address employers' uncertainty aboutreceiving visas. However, DHS and DOL have not assessed any of these options or determined which would not require Congressional action, and employers continue to struggle with uncertainty. To help ensure H-2B employers comply with U.S. worker recruitment and other requirements, DOL has audited employers' compliance with these requirements. However, in general, DOL randomly selected employers for these audits, rather than taking a risk-based approach using factors such as violation trends by industry. As a result, the agency may not be using its limited audit resources efficiently or effectively. GAO is making five recommendations. These include that DHS and DOL assess options to adjust the H-2B visa program and DOL take a risk-based approach to selecting H-2B employers for audits. The agencies agreed with these recommendations as well as one other. DHS disagreed with one, which GAO continues to believe is warranted.", "document_type": "gao"}
{"report": "NNSA’s nuclear stockpile missions are largely executed at eight sites that are managed by seven M&O contractors and that comprise the nuclear security enterprise. These eight sites include: three national security laboratories—Lawrence Livermore National Laboratory in California, Los Alamos National Laboratory in New Mexico, and Sandia National Laboratories in New Mexico and other locations; four nuclear weapons production plants—the Pantex Plant in Texas, the Y-12 National Security Complex in Tennessee, the Kansas City National Security Campus in Missouri, and tritium operations at DOE’s Savannah River Site in South Carolina; and the Nevada National Security Site, formerly known as the Nevada Test Site. As shown in figure 1, each of NNSA’s eight sites has specific responsibilities within the nuclear security enterprise. NNSA’s sites are owned by the federal government but managed and operated by M&O contractors. According to DOE, the use of M&O contracts is supported by an underlying principle: the federal government employs highly capable companies and educational institutions to manage and operate government-owned or government-controlled scientific, engineering, and production facilities because these companies and educational institutions have greater flexibility in bringing scientific and technical skills to bear than the government. As we previously found, an M&O contract is characterized by, among other things, a close relationship between the government and the contractor for conducting work of a long-term and continuing nature. To support its missions, NNSA is organized into program offices that oversee the agency’s numerous programs, such as the B61-12 Life Extension Program—overseen by the Office of Defense Programs—and the Nuclear Smuggling Detection and Deterrence Program—overseen by the Office of Defense Nuclear Nonproliferation. Mission-related activities are primarily overseen by these program offices, which are responsible for integrating the activities across the multiple sites performing work. NNSA’s program offices are: Counterterrorism and Counterproliferation; Defense Nuclear Nonproliferation; Defense Nuclear Security; Naval Reactors; and Safety, Infrastructure, and Operations. NNSA receives four different appropriations, which it is responsible for allocating to programs that are managed by the program offices. The program offices obligate these funds to the M&O contracts to execute specific program functions. Obligated funds that are not “costed,” or expended, by the contractor at the end of the fiscal year can carry over for expenditure in a subsequent fiscal year, or the program offices can deobligate the funds and obligate them to a different contract for work in that same program area. In order for funds to be reallocated to a different program, NNSA may need to reprogram funds; such reprogramming may be subject to congressional notice and approval requirements. NNSA headquarters offices generally are to provide leadership, develop policy and budgets, or provide other functional support across NNSA. NNSA headquarters offices include the offices of: Acquisition and Project Management, Cost Estimating and Program Evaluation, Information Management and Chief Information Officer, Management and Budget, and Policy. NNSA has seven field offices across the country. Field office managers report directly to the NNSA Administrator. NNSA field offices, such as NPO, are collocated at the laboratory, plant, and testing sites and are responsible for overseeing NNSA’s M&O contractors, including ensuring compliance with federal contracts. To provide oversight of the M&O contractors, each field office employs subject matter experts in areas such as emergency management, physical security, cybersecurity, safety, nuclear facility operations, environmental protection and stewardship, radioactive waste management, quality assurance, business and contract administration, public affairs, and project management. NNSA’s field offices are: Kansas City Field Office in Missouri, Livermore Field Office in California, Los Alamos Field Office in New Mexico, Nevada Field Office, NPO in Tennessee and Texas, Sandia Field Office in New Mexico, and Savannah River Field Office in South Carolina. Before awarding the consolidated contract at Y-12 and Pantex, NNSA took steps to consolidate its field offices that oversee the contractor at these two sites. Specifically, NNSA combined the former Y-12 Site Office and former Pantex Site Office into the NPO Field Office in 2012. One NPO manager oversees both the Y-12 and Pantex sites, and each site has a deputy manager. The deputy managers oversee their respective sites as well as certain programs at both sites. The NPO Cost Savings Program Manager provides overall administration of the Cost Savings Program. As of fiscal year 2018, NPO had about 130 federal full-time equivalent employees at both sites, according to an NPO official. According to CNS officials, the contractor employs over 9,000 employees at Y-12 and Pantex. According to an NPO official, NPO acts as a single office because the two sites are closely integrated. In December 2011, NNSA issued a request for proposals for a consolidated M&O contract for the Y-12 and Pantex sites. NNSA awarded the M&O contract to CNS in January 2013. However, the award was the subject of three protests to GAO under our bid protest authority. NNSA ultimately reaffirmed its award of the contract to CNS, and CNS began contract performance in July 2014. The consolidated contract includes a total of 10 years, including the base period and all option terms. The contract requires CNS to meet certain performance requirements, and NNSA is to evaluate CNS’s accomplishment of these performance requirements before exercising each option term. During the first 2 full fiscal years of the contract, CNS focused on merger and consolidation activities—that is, merging the two sites under one contractor—and on achieving savings from those activities, according to CNS’s Merger Transformation Plan. Merger savings are associated with efficiencies and reductions in the workforce resulting from the consolidation of the contract. During the third and fourth fiscal years of the contract, CNS focused on transformation savings—or savings based on changing underlying processes to increase standardization, and improve quality and efficiency within and across the organization. From the third full fiscal year of the contract onward, CNS focused on continuous improvement, which constitutes incremental efficiency within established processes. The original contract required CNS to achieve at least 80 percent of its proposed savings and score 80 percent or higher on its performance evaluations in order to have additional option terms exercised. In September 2017, however, NNSA and CNS modified the contract so that delivery of cost savings is only taken into consideration in conjunction with CNS’s performance, as documented in NNSA’s annual Performance Evaluation Reports, when deciding whether to extend CNS additional option terms, also known as gateway decision points. NNSA officials told us they made this modification prior to the first gateway decision in September 2017 because CNS was very close to achieving 80 percent of its proposed cost savings, but it was unclear if CNS would achieve 80 percent. In addition, the initial contract requirements placed equal emphasis on cost savings and the contractor’s performance in meeting the mission, but NNSA officials said they do not view those two goals as equal. Cost savings in and of themselves are only helpful—and only creditable under the contract—if they do not negatively affect the mission, and therefore NNSA officials do not view achieving cost savings as equal to the contractor’s performance in meeting the mission. Following the contract modification in September 2017, NNSA exercised the first 2-year option term, ensuring the contractor will manage and operate Y-12 and Pantex through fiscal year 2021. The gateway decision for the second 2- year option term will occur by the end of June 2020, according to NNSA officials (see fig. 2). Implementation and oversight of the Cost Savings Program involves contractor representatives and NNSA officials at several levels. CNS manages the Cost Savings Program using a matrixed organization that includes several executives such as vice presidents of the Business Management and Transformation and Program Integration departments, according to CNS officials. Throughout each fiscal year, these officials lead various efforts associated with developing and implementing cost reduction initiatives as well as other key aspects of the Cost Savings Program. One CNS Cost Savings Director is responsible for overseeing much of the company’s cost savings efforts, including coordinating between different program offices. Within NNSA, NPO conducts much of the oversight of the Cost Savings Program while NNSA’s Offices of Management and Budget, and Acquisition and Project Management also have some oversight functions. Within NPO, the Cost Savings Program Manager coordinates among different NPO program offices that help review and conduct oversight of the cost reduction initiatives throughout the year as well as with NNSA headquarters offices. NNSA’s Office of Management and Budget provides NNSA with administrative, human resources, and financial support. NNSA’s Office of Acquisition and Project Management is responsible for acquisition support and contracting oversight for the agency throughout the acquisition lifecycle. NNSA established an Executive Steering Committee, comprised of high- ranking officials from different NNSA program areas, as well as the NNSA Associate Principal Deputy Administrator, the NPO Manager, and the NPO Cost Savings Program Manager (as a non-voting member), to provide leadership and guidance for the governance of the cost savings element of the CNS contract. The steering committee members are to set cost savings policy; resolve disputes; and recommend and approve the cost savings amounts to be shared between the government, the contractor (through a cost-savings incentive fee), and site reinvestment projects. The Cost Savings Program is divided into six processes or phases that CNS and NNSA implement and oversee (see fig. 3): the Annual Controlled Baseline phase, the Cost Reduction Proposal phase, the Change Control phase, the Verification phase, and the Disposition phase. Annual Controlled Baseline phase. CNS develops and maintains the Annual Controlled Baseline, which is a document that describes the current scope of work and its cost and schedule. Among other things, NNSA uses the Annual Controlled Baseline to evaluate whether CNS achieved savings from implementation of prior years’ cost reduction initiatives. CNS is expected to submit the Annual Controlled Baseline no later than August 15 prior to the upcoming fiscal year, and NNSA reviews and approves the document. Cost Reduction Proposal phase. CNS develops cost reduction initiatives and updates the Cost Reduction Proposal, which describes CNS’s proposed cost reduction initiatives for the upcoming fiscal year and the expected cost savings to be validated from activities within the current fiscal year. The Cost Reduction Proposal is to be updated annually, no later than September 1 prior to the upcoming fiscal year. Each cost reduction initiative has a defined lifecycle, from identification and development to validation and sustainment. NNSA reviews and approves the document; approval authorizes CNS to begin implementing the initiatives. Change control phase. The change control phase is continuous throughout the fiscal year and allows CNS and NNSA to document and trace changes to the scope, schedule, and cost that affect the Annual Controlled Baseline and the Cost Reduction Proposal. Changes made during this phase to the Annual Controlled Baseline and the Cost Reduction Proposal are generally limited to changes outside of the control of the contractor, including congressional direction or reprogramming, changes to the programmatic mission, additional contractual requirements, and any NNSA-directed or approved changes. Performance phase. During the performance phase, which is also continuous throughout the year, the contractor is to report interim performance against the approved cost reduction initiatives for NNSA to evaluate accordingly, according to NNSA Cost Savings Program procedures. This interim reporting allows NNSA to monitor potential effects on the mission and offer feedback and course correction as needed. NNSA and CNS officials responsible for the Cost Savings Program collaborate regularly via biweekly meetings and tri-annual reviews to monitor CNS’s progress on cost reduction initiatives throughout the fiscal year. CNS generates a year-end Validation Report, which is the final of three tri-annual reports provided throughout the fiscal year. These reports detail the performance of the M&O contractor and progress made against proposed cost savings targets, and list the amount of savings CNS is claiming to have achieved in that fiscal year, to include both annual new savings and savings sustained from prior years. CNS is to submit the Validation Report for each previous fiscal year no later than November 15. Verification phase. After the end of the fiscal year, between November and January, NNSA uses verification checklists to review and verify CNS’s claimed savings for each cost reduction initiative. NNSA can use these verification checklists to record, among other evidence, any observations, interviews, document reviews, analyses, and measurements that NNSA has undertaken to confirm the savings claimed by CNS in the Validation Report. For each cost reduction initiative, NNSA is to verify, among other things, that CNS implemented the initiative, that the initiative resulted in efficiencies that produced cost savings, and that the initiative did not negatively affect the mission. NNSA is also to verify that CNS set aside the claimed savings. Additionally, NNSA is to verify that CNS sustained savings claimed in prior years. NNSA documents its determination of verified annual new and sustained savings in a Verification Report. Disposition phase. Upon completion of the verification phase, in January and February, the distribution, or disposition, of net savings occurs in accordance with the contract. Net savings are verified savings after accounting for execution costs. The contract allows those verified net savings to be shared among the government, the contractor, and site reinvestment projects to improve Y-12 and Pantex. Under the contract provisions, NNSA is to verify and distribute only those savings that remain after deducting the execution costs required to administer, develop, or implement the cost reduction initiatives. For example, the cost of purchasing a machine to automate a process that will, in turn, save labor hours from the previous non-automated process would be an execution cost. Therefore, NNSA-verified savings for each cost reduction initiative should reflect net savings from having implemented the initiative—that is, the gross savings minus the execution costs associated with the initiative. Verified net savings are to be distributed to the contractor, the government, and for site reinvestment projects. Contractor. The contractor is generally to receive a cost-savings incentive fee of about 35 percent of the verified net savings. For new savings related to employee benefits, however, the contractor is not to receive a share, and the savings are to be split between the government (50 percent) and site reinvestment projects (50 percent). The contractor’s cost-savings incentive fee is to be paid out of cost savings that NNSA has verified. The contract requires CNS to reimburse the government for the cost-savings incentive fee in the event that CNS does not sustain the savings for the remainder of the contract performance period. According to CNS’s proposed savings estimates, CNS planned to earn approximately $222 million in cost- savings incentive fees over the potential 10-year contract. Per the contract, the contractor may also receive award fees annually based on NNSA’s evaluation of its performance. The available award fee for each potential year of the contract ranges from approximately $20 million to approximately $40 million. Government. The government generally is to receive 35 percent of the verified net savings. For new savings related to employee benefits, however, the government is to receive 50 percent of the verified net savings. The portion of verified savings that is available for the government allows NNSA to return those savings to the programs for which funds were originally obligated, and the funds can be spent within the same program at Y-12, Pantex, or another site within the nuclear security enterprise. Site reinvestment. The remaining approximately 30 percent of the verified net savings is for site reinvestment projects. As noted above, however, the site reinvestment share for savings related to employee benefits is 50 percent. Site reinvestment projects may include: projects (such as a parking structure, an office building or a cafeteria) that serve the M&O site as a whole rather than a discrete program or implementation costs for future cost savings initiatives, among other things. Types of potential savings associated with the Cost Savings Program include, for example: Annual new savings. In each fiscal year, CNS validates and NNSA verifies annual new savings for the cost reduction initiatives implemented in that year. Examples of annual new savings include positions that were reduced in a certain program area, in a given fiscal year. As discussed previously, cost savings are only creditable under the contract if they do not negatively affect the mission. Sustained savings. In each fiscal year, CNS validates and NNSA verifies sustained savings resulting from cost reduction initiatives implemented in prior years. For example, CNS can claim sustained savings for each year it does not hire back employees into positions that were reduced in a prior year and for which CNS claimed savings. Cumulative contract savings. Cumulative contract savings is the sum of all contract savings that have accumulated from annual new savings and the sustainment of savings produced in prior years. For example, annual new savings verified in fiscal year 2015 would be multiplied by 10 if they are sustained through the life of the potential 10-year contract. Likewise, annual new savings verified in fiscal year 2016 would be multiplied by 9 if they are sustained through the life of the potential 10-year contract, and so forth. These cumulative contract savings are also known as “gateway savings” because NNSA considers the verified cumulative contract savings when making gateway decisions on whether or not to extend the contract for possible option terms. Table 1 shows how CNS proposed it could achieve approximately $2.9 billion over the life of the 10-year contract using this method of calculating cumulative contract savings. Hard savings—savings that directly reduce the overall cost of operations—are the only creditable type of savings under the contract. NNSA is only to verify savings if they do not negatively affect the mission. Examples of hard savings include a reduced number of personnel working to conduct the same scope of work or fewer labor hours required to complete a process due to operational efficiencies achieved, as well as savings in benefits packages (e.g., by requiring employees contribute more to their benefits). NNSA and CNS classify hard savings into four categories: (1) labor, (2) benefits, (3) supply chain, and (4) non-labor (see sidebar). generated by leveraging collective buying power agreements, utilizing competitive sourcing tools, and taking other actions to reduce the price of goods purchased. For example, in fiscal year 2016, CNS noted in its Validation Report that it used strategic sourcing to realize procurement savings. known as demand management savings—are savings generated through reductions in purchased materials quantities, subcontract costs, or licenses. For example, in fiscal year 2016, CNS assumed responsibility for some information technology work—including, among others, help desk support and network administration—that had been previously handled by subcontractors. Doing so reduced contract costs because CNS was able to perform the work at a lower cost than the subcontractor. products or services such as, for example, slowing the rate of a cost increase. NNSA officials said another example of a cost avoidance would be if the contractor has the option to buy more expensive airplane tickets for travel between the two sites but chooses to buy less expensive airplane tickets; the difference between the most expensive option and the actual tickets purchased is a cost avoidance and not considered hard savings that would be creditable under the contract. NNSA verified approximately $170 million in annual new savings and approximately $515 million in cumulative contract savings from fiscal year 2014 through fiscal year 2018. The $515 million in cumulative contract savings that NNSA verified from fiscal year 2014 through fiscal year 2018 is about 80 percent of the approximately $640 million CNS proposed it would save through that fiscal year. NNSA’s oversight of the Cost Savings Program has improved and methods for calculating and verifying cost savings have evolved to address some problems encountered in the early years of the contract that may affect actual contract savings. NNSA verified between approximately $8 million and $63 million in annual new savings each year from fiscal year 2014 through fiscal year 2018, totaling approximately $170 million in annual new savings over this period. Of the $170 million in NNSA-verified annual new savings for fiscal years 2014 through 2018, roughly 10 percent (approximately $17 million) is attributed to the merging of the Y-12 and Pantex sites into a consolidated management structure, according to CNS and NNSA documentation. The remaining roughly 90 percent (approximately $153 million) is attributed to transforming site operations to create a more efficient and sustainable enterprise. Under the contract, savings from the previous year that have been sustained, and for which sustainment has been verified by NNSA, are added to the current year’s verified annual new savings amount, resulting in cumulative contract savings. As of the end of fiscal year 2018, NNSA verified approximately $515 million in cumulative contract savings (see table 2). We found that this $515 million in cumulative contract savings represents a reasonable estimate of the cumulative savings achieved. As part of our review, we traced information from 22 of about 90 cost reduction initiatives for which CNS claimed savings to source documents and reconciled discrepancies with NNSA and CNS officials to understand how NNSA verified the cost savings. Further, we reviewed NNSA’s documented procedures for verifying CNS’s reported data and interviewed officials about that process. Additionally, other reviews provide support that NNSA’s reported $515 million in cumulative contract savings is a reasonable estimate of savings achieved. Specifically, as part of the savings verification process, NNSA’s federal cost accountants ensured that CNS had set aside the money associated with the cost savings and confirmed that the funds were available for distribution under the cost-savings sharing arrangement. DCAA also reviewed CNS’s claimed cost savings for fiscal years 2016 through 2018 and NNSA and DCAA officials said the two entities used similar methods and came to similar conclusions. Labor savings, which include reductions in positions, comprised the largest portion of savings, at nearly two-thirds of the cumulative contract savings achieved from fiscal year 2014 through fiscal year 2018. Savings through changes to employee benefits comprised nearly a quarter of total cumulative contract savings over the period (see fig. 4). NNSA documents we examined showed that CNS, the government, and site reinvestment projects received a certain share of the $515 million in cumulative contract savings that NNSA verified from fiscal year 2014 through fiscal year 2018 in accordance with the terms of the contract. According to NNSA, approximately $262 million of the $515 million was available for the three parties to share during this period. The amount available to the three parties is determined by sharing periods of no more than 2 years negotiated for different categories of savings under the contract. According to NNSA documents, CNS earned about $78 million in cost-savings incentive fees, the government received about $97 million in savings, and site reinvestment projects received about $88 million of the available savings from fiscal year 2014 through fiscal year 2018 (see fig. 5). According to NNSA, the remaining approximately $253 million in cumulative savings was not available for sharing between the three parties because it accumulated outside of the savings sharing period. The $515 million in cumulative contract savings that NNSA verified from fiscal year 2014 through fiscal year 2018 is about 80 percent of the approximately $640 million in cumulative contract savings CNS proposed it would save through that fiscal year. CNS achieved more in cumulative contract savings than it proposed through fiscal year 2015. Specifically, CNS proposed approximately $67 million in cumulative contract savings through fiscal year 2015 and NNSA verified approximately $78 million. From fiscal years 2016 through 2018, however, CNS achieved less in cumulative contract savings than it proposed (see fig. 6). As described above, achieving approximately $2.9 billion in savings over the life of the contract assumed meeting all proposed annual new savings targets and fully sustaining those savings in each year of the contract. According to the terms of the contract, NNSA considers achievement of cost savings when evaluating overall contract performance, and therefore, achievement of proposed cost savings may factor into NNSA’s decision of whether to exercise further contract option terms. Two key issues—benefits savings and fiscal year 2016 labor savings— contributed to CNS not meeting its proposed cost savings targets through the end of fiscal year 2018 and may affect CNS’s ability to achieve its proposed cumulative contract savings of approximately $2.9 billion over the life of the contract. Benefits savings. CNS proposed it could save $594 million over the life of the contract through adjustments to employee benefits, but as of March 2020, CNS officials told us that CNS’s projected benefits savings would total $399 million over the entire 10-year contract, a decrease of almost $200 million from its proposal. According to these officials, several factors have contributed to CNS’s decreased benefits savings estimate, including delays in bargaining unit transition to benefit plans and rates and a decrease in employee contributions to pensions, among other reasons. Fiscal year 2016 labor savings. In fiscal year 2016, CNS claimed approximately $30 million in new labor savings based on a claimed reduction of 283 full-time equivalent employees, but NNSA rejected all of those savings. According to the fiscal year 2016 NNSA Verification Report, CNS failed to realize efficiencies that resulted in full-time equivalent growth in other areas, which offset CNS’s claim of new labor savings. Rejection of these fiscal year 2016 labor savings could result in a loss of approximately $270 million in cumulative savings through the end of the potential 10-year contract period when factoring in potential sustained savings. NNSA officials emphasized that any amount of cost savings is beneficial to the government and that NNSA’s priority for CNS is safe and secure performance of its mission. NNSA officials noted that if CNS does not implement any additional cost reduction initiatives and sustains the savings from all previously-implemented cost reduction initiatives, CNS will still save about $1.7 billion through fiscal year 2024. CNS officials told us that CNS will continue to work toward its cumulative proposed savings of approximately $2.9 billion and hopes to meet or exceed that estimate. According to these officials, doing so will allow CNS to realize its proposed savings and provide the maximum benefit to the government and taxpayers. To achieve its proposed savings, CNS would need to sustain all previously implemented savings, achieve verified annual new savings of approximately $57 million per year every year, and sustain those additional savings through 2024. However, CNS’s proposed annual new savings are substantially lower for fiscal year 2019 through the end of the contract (averaging about $30 million per year) than they were from fiscal year 2014 through fiscal year 2018. This decrease is, in part, because many cost reduction initiatives with high savings potential—such as labor streamlining and changes to employee benefits—have been implemented. For example, CNS eliminated 270 positions and provided voluntary separation severance packages to another 182 employees in fiscal year 2014. This accounted for more than 40 percent ($221 million) of the cumulative contract savings because CNS sustained those savings in fiscal years 2015 through 2018. CNS has already implemented many cost reduction initiatives with high savings potential, so it may be difficult for CNS to meet its proposed cumulative contract savings. CNS and NNSA initially encountered problems with calculating and verifying cost savings—problems that may affect actual contract savings—but methods for calculating and verifying savings have evolved, and NNSA’s oversight of the Cost Savings Program has improved. Specifically, CNS and NNSA initially encountered problems—which have largely been addressed—with: (1) calculating and verifying execution costs; (2) calculating and verifying labor savings; and (3) communicating and collaborating about the Cost Savings Program throughout the year. Calculating and verifying execution costs. NNSA encountered early problems with verifying execution costs for CNS’s cost savings initiatives, but CNS changed its methodology for calculating execution costs each year that ultimately addressed those problems. Since the contract’s inception, CNS has relied on a subcontractor to operate much of the Cost Savings Program. In fiscal year 2014, costs for this subcontractor totaled approximately $7 million. CNS believed that approximately $546,000 of the $7 million should be considered execution costs and counted against the cost savings for that year, but NNSA believed the entire $7 million should be considered execution costs. NNSA and CNS reached agreement that a proportional factor—19.3 percent—of the subcontractor’s time was spent on activities that would qualify as execution activities under the contract for fiscal years 2014 and 2015. NNSA instructed CNS to capture and report the subcontractor’s actual execution costs beginning in fiscal year 2016. CNS began using the subcontract’s actual execution costs in fiscal year 2016, according to NNSA officials. However, NNSA officials said CNS used a proportional factor of the subcontract’s execution costs from previous years to estimate the execution costs of CNS employees for fiscal year 2016. NNSA noted in its fiscal year 2016 Verification Report that using the proportional factor approach for estimating execution costs may not reflect the actual execution costs. CNS officials said they believe this estimation was conservative because it resulted in higher CNS administrative and development costs than subsequent years. Additionally, in fiscal years 2015 and 2016, CNS reported estimates for its total execution costs rather than tracking the actual execution costs for each individual cost reduction initiative, which NNSA officials said made it difficult to verify net savings. In fiscal year 2017, CNS developed a methodology for allocating execution costs—administrative costs, implementation costs, and development costs—to individual cost reduction initiatives and began reporting execution costs at this level in the fiscal year 2017 Validation Report. According to NNSA officials, CNS also began reporting execution costs by individual cost reduction initiative for its subcontractor beginning in fiscal year 2017. In fiscal year 2018, CNS developed execution cost charge codes that allowed CNS to report actual hours spent on cost reduction initiative execution activities— including amounts for its subcontractor—for the first time since the contract began. NNSA officials told us that they are generally satisfied with the way CNS is now capturing execution costs and that the use of charge codes has improved their confidence in CNS’s reporting of certain execution costs. However, CNS’s use of the proportional factor of 19.3 percent of the subcontractor’s execution costs, lack of detail on execution costs for individual cost reduction initiatives, and use of estimated—rather than actual—execution costs could mean that the actual execution costs for fiscal years 2014 through 2017 are not fully captured in reported cumulative savings and actual contract savings could be higher or lower than the reported amount. Even if the actual contract savings are higher or lower than the reported amount, we believe $515 million is a reasonable estimate of the savings achieved to date. Calculating and verifying labor savings. In fiscal years 2014 and 2015, CNS used a headcount methodology to calculate labor savings and demonstrate sustainment of those savings. Using a headcount methodology, CNS could claim labor savings if it could demonstrate and maintain a reduced number of employees to conduct the same scope of work. According to NNSA and CNS officials, one potential problem with using a headcount approach is that CNS could maintain a reduced number of staff but have those staff work overtime. If this occurred, it would result in overall increased contract costs, thereby reducing the net savings from the cost reduction initiative. In fiscal year 2016, CNS modified its methodology for calculating labor savings to use labor hours rather than employee headcounts. Under this modified approach, CNS could claim labor savings if it could demonstrate and maintain reduced labor hours regardless of the number of employees, a method that NNSA and CNS officials said is a better measure of labor savings. However, under this methodology, CNS calculated labor savings based on planned, rather than actual, reductions in labor hours. In fiscal year 2017, CNS modified its methodology again to begin using actual reduced labor hours rather than planned reduced labor hours. However, CNS’s use of headcounts and planned, rather than actual, reduction in labor hours could mean that the labor savings for fiscal years 2014 through 2016 are not accurately reflected in the verified cumulative contract savings, and actual contract savings could be higher or lower than the reported amount. As noted above, even if the actual contract savings are higher or lower than the reported amount, we believe $515 million is a reasonable estimate of the savings achieved to date. Communicating and collaborating about the Cost Savings Program. According to NNSA officials, early years of the contract were marked by limited oversight and poor communication between NNSA and CNS. CNS delegated responsibility for the Cost Savings Program to a subcontractor, and according to NNSA and CNS officials, CNS had limited involvement in the Cost Savings Program and did not communicate with NNSA about cost savings matters. Similarly, NNSA officials told us that one or two individuals at NNSA managed the cost savings component of the contract for the federal government and that communication was poor between those individuals and the technical personnel responsible for evaluating the implementation of CNS’s cost reduction initiatives. As a result of this limited oversight and communication, NNSA officials said CNS did not understand NNSA’s expectations for cost savings data and had to submit five iterations of its first Validation Report. In fiscal year 2017, NNSA established a collaborative working team— known as the Integrated Project Team and consisting of personnel from NNSA and CNS—which meets biweekly to discuss issues related to the Cost Savings Program. Also in fiscal year 2017, NNSA began conducting tri-annual reviews of active cost reduction initiatives. For these reviews, CNS submits performance reports and briefs knowledgeable NNSA officials on the status of individual cost reduction initiatives. NNSA uses this information to identify potential gaps in cost-savings reporting data and, among other things, informs CNS of any concerns with its methodology or NNSA’s ability to verify the cost savings. NNSA officials stated that the increased collaboration and more frequent communication has resulted in improved Validation Reports and fewer revisions. For example, NNSA stated in its fiscal year 2017 Verification Report that the quality and completeness of CNS’s fiscal year 2017 Validation Report “demonstrated substantial improvement” over the fiscal year 2016 report. While CNS’s and NNSA’s methods for calculating and verifying savings and conducting oversight evolved in the early years of the contract to improve the accuracy of cost savings calculations, we believe the $515 million in reported cumulative savings represents a reasonable estimate of the contract savings achieved to date for reasons we described earlier. NNSA officials said three key benefits of the Cost Savings Program are (1) achieving savings; (2) increasing financial transparency; and (3) funding site reinvestment projects. Achieving savings. As discussed previously, the Cost Savings Program resulted in total new annual savings of approximately $170 million and $515 million in cumulative contract savings, from fiscal year 2014 through fiscal year 2018. According to NNSA officials, these cost savings would not have materialized without the Cost Savings Program. We have previously found that DOE could better assess M&O contractors’ cost performance—i.e., their performance on spending, budgeting, strategic sourcing, and cost-effectiveness—to help strengthen contractor oversight and better inform acquisitions decisions. Demonstrating contractors’ efforts to achieve cost savings and NNSA’s associated efforts to evaluate contractors’ cost effectiveness provides evidence that for the CNS contract, NNSA is placing importance on cost performance while overall resource needs are increasing. For example, NNSA has identified an increasing weapons program workload and a need to recapitalize or replace aging facilities and equipment to meet nuclear weapons modernization programs over the next decades. To help achieve these goals, NNSA’s fiscal year 2021 budget request included a 25 percent increase for NNSA’s weapons activities appropriation, which funds programs at NNSA sites including Pantex and Y-12. Identifying cost savings could help NNSA minimize budget increases in an era of increasing workload and assure congressional decision-makers that NNSA is working to effectively steward federal resources. Increasing financial transparency. Because of the Cost Savings Program, which required the establishment of the Annual Controlled Baseline in order to measure potential savings, NNSA has better and more thorough information on the costs of running the two sites, NPO officials said. The Annual Controlled Baseline provides more information because in order to demonstrate savings CNS had to first establish a cost baseline, which required complete information on funding streams as well as how certain rate structures are established, according to NPO officials. Officials from NNSA’s Office of Acquisition and Project Management also said this was the first time that NNSA has been able to gain insight into the actual costs of certain activities at Y-12 and Pantex, as a result of the Annual Controlled Baseline being established. None of the other M&O sites have an established site-wide baseline against which to measure costs or cost savings, according to NNSA and M&O officials we interviewed. Officials from the Office of Acquisition and Project Management said having an Annual Controlled Baseline at other sites would give them additional insight into the cost of certain activities, as opposed to the traditional budget-based view they have into M&O activities. At other M&O sites, NNSA uses a budget-based model, which consists of the government obligating a certain amount of money and getting as much product or service for that amount of money as the sites will provide, NNSA officials said. Instead, NNSA is employing a cost- based model at Y-12 and Pantex, which involves determining the cost to produce a certain amount of product, NNSA officials explained. Funding site reinvestment projects. As part of the Cost Savings Program, a certain percentage of the achieved savings is reinvested back into the sites. According to NNSA officials, this process has allowed NNSA to allocate funds to site reinvestment projects to improve the Y-12 and Pantex sites’ aging infrastructure. As of fiscal year 2019, NNSA reported about a $4.8 billion deferred maintenance backlog throughout the nuclear security enterprise. We previously found that facilities considered not mission dependent—such as cafeterias, parking structures and excess facilities—comprised about 40 percent of the deferred maintenance backlog. NNSA officials said addressing deferred maintenance at these types of facilities is low priority, beyond keeping facilities in a safe condition, because the agency targets scarce budgetary resources to mission critical facilities. According to NNSA officials, NNSA would not likely have allocated funds for these site reinvestment projects at Y-12 and Pantex without the Cost Savings Program because they are often considered lower priority projects. As a result, the nuclear security enterprise as a whole potentially benefits from these site reinvestment projects at Y-12 and Pantex since those reinvestment projects serve to reduce overall deferred maintenance and potentially make funds available for projects to address aging infrastructure at other sites. Site reinvestment projects may lead to additional cost savings as well, NNSA officials said, if, for example, NNSA uses site reinvestment funds to purchase a machine that automates a process and saves labor hours as a result. For example, NNSA invested in a machine to replace three different machines that were previously required to produce a screw. This improved throughput and turnaround time and saved labor hours, according to NNSA documentation. NNSA approved a total of 80 site reinvestment projects at Y-12 and Pantex as of April 2020, for a total of approximately $75 million that was available for reinvestment into the sites. For example, CNS used about $1.2 million in site reinvestment funds to replace analog cameras along Y-12’s perimeter fencing with digital cameras (see fig. 7). This site reinvestment project improved physical security and reduced camera maintenance costs, as well as the security team’s ability to assess alarms and manage alarm response, according to NPO documentation. Because the analog cameras were still functioning, they may have otherwise been a lower priority to replace without the site reinvestment funding, NPO officials said. In addition, the John C. Drummond Center, a new administrative support complex at Pantex, was partially built with savings from the Cost Savings Program (see fig. 8). According to NNSA documentation, the new facility helps eliminate approximately $20 million in deferred maintenance costs of the older administrative buildings it replaced. Although NNSA identified site reinvestment projects as one of the key benefits of the Cost Savings Program, NNSA and CNS had not committed approximately $13 million of site reinvestment funds available at Y-12 and Pantex as of April 2020. NNSA and CNS had not yet committed the site reinvestment funds to specific project efforts, in part because they have not evaluated how best to use the remaining available site reinvestment funds or developed a plan for doing so. The $13 million is currently distributed across several different layers of accounts, in some cases in amounts too small to execute a site reinvestment project. To aggregate the funds in amounts large enough for certain projects, NNSA may need to move funding from one account to another. The funds for site reinvestment projects are distributed in accordance with the terms of the contract and are spread across different programs, projects, or activities (PPA). Beneath the PPA is the DOE budget and reporting code level, which DOE also tracks in its official accounting system (see fig. 9). According to NNSA officials, there were 68 PPAs with 97 budget and reporting codes underneath them that, as of April 2020, had funds available for site reinvestment. According to NNSA officials and CNS representatives, this distribution makes it difficult to use all of the site reinvestment funds. This difficulty is because a given site reinvestment project may require funds to be aggregated across budget and reporting codes in order to have enough funds for executing the project, and while NNSA can move funds between budget and reporting codes that are within the same PPA, movement of funds among PPAs (reprogramming) could require congressional approval. As of April 2020, of the 68 PPAs with available funds for site reinvestment, 17 (or about 25 percent) had multiple budget and reporting codes underneath them, according to NNSA officials. Those 17 PPAs had between 2 to 6 budget and reporting codes underneath them, according to those officials (see fig. 10). We have previously found that comprehensive plans can help organizations identify potential problems before they occur and target limited resources. A comprehensive plan can also detail milestones and key goals, which provide meaningful guidance for planning and measuring progress. Such plans can establish deadlines for achieving objectives and assign responsibility for any implementation. Most of NNSA’s appropriations are “no-year funds” and are, therefore, available for obligation until expended. Without evaluating and developing a plan for how best to use funds for site reinvestment projects—to include determining whether to reprogram funds—NNSA and CNS are not fully utilizing available site reinvestment funds, and the funds could be rescinded from NNSA’s appropriations in later years if the unspent balances persist. NNSA has not sought congressional approval to combine site reinvestment money across different PPAs in order to aggregate these funds to execute larger site reinvestment projects, officials said. Also, while NNSA moves funds weekly between budget and reporting codes that are within the same PPA to execute its work, officials said NNSA has not moved any site reinvestment funds from different budget and reporting codes within the same PPA to fund site reinvestment projects. Once NNSA develops a plan on how best to aggregate or use the remaining and potential future site reinvestment funds, it would be better positioned to: move some funds between budget and reporting codes within the reprogram funds between PPAs, including seeking congressional approval where it may be required. NNSA officials identified the achievement of cost savings as a benefit of the Cost Savings Program that could be useful at other sites and to the nuclear security enterprise generally; however, the officials said they are not planning to implement the Cost Savings Program as part of other future or existing M&O contracts. Most existing NNSA M&O contracts include a “Cost Reduction” clause, under which sites could implement a Cost Savings Program with some attributes of the program at Y-12 and Pantex. According to GAO’s Framework for Assessing the Acquisition Function at Federal Agencies, leading organizations gather and analyze data to identify opportunities to reduce costs, among other reasons. Further, the framework states that incomplete data can prevent an agency from maximizing information tools for strategic acquisition planning and analysis. According to officials from the Office of Acquisition and Project Management, they do not plan to implement the Cost Savings Program or anything similar to it as part of future M&O contracts because of uncertainties regarding (1) the opportunities for similar savings at other sites and (2) the federal costs involved in implementing and overseeing the Cost Savings Program—including the time and effort needed to verify cost savings—and how these costs affect the overall net savings. NNSA site officials and contractor representatives we interviewed also raised questions about these issues. For example, according to NNSA officials and representatives at two sites, the Cost Savings Program may not be exportable to other sites, in part because other sites may not be able to identify cost savings initiatives that would yield the same level of savings as at Y-12 and Pantex. The officials believed that much of the savings identified at those sites resulted from merger savings—savings stemming from consolidating the two sites—that would not be possible without combining two sites under one contract. However, as mentioned previously, our analysis found that the majority—about 90 percent—of annual savings at Y-12 and Pantex resulted from transformation initiatives, or savings based on improving standardization, quality, and efficiency. Merger savings contributed only about 10 percent of the total new annual savings identified from fiscal year 2014 through fiscal year 2018. NNSA officials and contractor representatives at other NNSA sites also raised questions about whether the cost of implementing and maintaining a formal cost savings program might outweigh the benefits at a site. According to NNSA officials, a large number of government employees are involved in implementing and overseeing the Cost Savings Program. According to an official from the Office of Acquisition and Project Management, NNSA has not analyzed the total costs of implementing the Cost Savings Program, including the costs associated with the government effort to oversee the program. For the Cost Savings Program, NNSA verifies net savings after accounting for CNS’s execution costs. However, the verified savings do not take into consideration federal costs for implementing, maintaining, and overseeing the Cost Savings Program. To provide a sense of the scope of the oversight effort, NPO officials said about 100 of the approximately 130 employees at NPO at the end of fiscal year 2018 had some role in the Cost Savings Program, although only one full-time position is dedicated to the Cost Savings Program. Further, NNSA is likely to start its acquisition planning for some M&O contracts in 2022 and 2023. However, NNSA officials, as well as site officials, were uncertain about whether the Cost Savings Program could be exported to other existing or future contracts, including the cost effectiveness of the program, because NNSA has not gathered information on and documented analysis of the costs and potential benefits of the Cost Savings Program. By gathering information on and documenting the analysis of data on the costs and benefits of the Cost Savings Program, NNSA officials and contractor representatives could make better-informed decisions about whether to implement aspects of the Cost Savings Program at other sites. CNS achieved cost savings at Y-12 and Pantex by implementing a variety of cost savings initiatives. Even without a formal Cost Savings Program in place, some efficiencies may be applicable at other sites as a way to save money across the enterprise, according to officials we interviewed from NPO. For example, at Pantex, the contractor discovered it could conduct fewer recurring injections of treatment wells but still achieve the same technical results and comply with standards, according to NNSA officials. This initiative saved over $500,000, according to NNSA’s Verification Report. If other sites experience similar recurring costs, then sharing this initiative might lead to cost savings at those sites. According to DOE’s Order 210.2A on the DOE Corporate Operating Experience Program, each DOE organization is required to submit lessons learned to the DOE Corporate Lessons Learned Database when the operating experience has relevance to other DOE sites and the information has the potential for cost savings. Although NPO did not enter information about lessons learned from the Cost Savings Program into the database, NPO officials said they shared lessons learned with the Executive Steering Committee and that they presumed the Committee had passed information along to other sites. Contractor representatives and NNSA officials from all five of the other NNSA sites we interviewed noted that NNSA has not shared any information about specific successful cost savings initiatives from Y-12 and Pantex that could be applicable to them. Almost half of the NNSA officials and contractor representatives from other sites we interviewed said they were not very familiar with the Cost Savings Program. However, officials at Y-12 and Pantex told us they believe there are certain initiatives that could be useful at other sites and that other sites have asked for information about certain initiatives. Officials from the Office of Acquisition and Project Management said they believe there will be a request for a lessons learned evaluation from NNSA headquarters once the current Y-12 and Pantex contract expires; however, such an effort would begin in several years—as late as 2024 if all option terms are exercised and NNSA began this evaluation immediately. According to NNSA officials, the Cost Savings Program was a new concept and required maturity and proven concepts before sharing any lessons learned. However, by sharing information on potentially beneficial efficiencies and lessons learned from the Cost Savings Program at Y-12 and Pantex throughout the enterprise, NNSA could help achieve cost savings enterprise-wide even without implementing formal cost savings programs at other sites. The Annual Controlled Baseline is another specific aspect of the Cost Savings Program that could be beneficial to implement at other sites, or programs at a site, NNSA officials said. Currently, none of the other NNSA sites have an established site-wide baseline that would allow NNSA to understand the costs involved in running those sites or implementing their programs, according to officials from NNSA’s Office of Acquisition and Project Management. According to NPO officials, the Annual Controlled Baseline provides NNSA with better and more thorough information on the costs of running the two sites. As discussed previously, employing a cost-based model at Y-12 and Pantex—as opposed to the budget-based model at other sites—allows NNSA to understand the contractor’s cost to produce a certain amount of product. Although officials from NNSA’s Office of Acquisition and Project Management said it would be beneficial to have the Annual Controlled Baseline at other sites in order to gain additional insight into the cost of certain activities, they believed a drawback to requiring other sites to institute such a baseline would be deploying the considerable effort and resources to establish the baseline similar to those that were required at Y-12 and Pantex. NNSA has not evaluated whether to require the other sites to have an Annual Controlled Baseline, either for the entire site or for certain programs at different sites. The 2019 DOE Acquisition Guide states that in the context of acquisition planning, good technical, schedule, and cost baselines are essential for developing realistic and measureable targets. By evaluating whether to require all sites to implement an Annual Controlled Baseline, either for the entire site or for certain programs at the different sites, NNSA may be in a better position to achieve greater financial transparency at sites across the nuclear security enterprise. This action, in turn, could potentially identify opportunities for cost savings, help NNSA better understand their contractors’ cost performance, and help the agency administer its sites more efficiently. In recent years, the Cost Savings Program at Y-12 and Pantex has realized hundreds of millions in savings to the nuclear security enterprise, dozens of site reinvestment projects, and increased financial transparency. Although NNSA has identified site reinvestment projects as one of the key benefits of the Cost Savings Program, NNSA and CNS have not committed approximately $13 million of site reinvestment funds available at Y-12 and Pantex, in part because they have not evaluated and developed a plan on how best to aggregate and use the funds. If NNSA develops a plan on how best to use the remaining and potential future available site reinvestment funds, it would be better positioned to aggregate funds for site reinvestment projects. Further, if funds for site reinvestment projects persist in PPAs for too long, NNSA risks their rescission in future years’ appropriations. NNSA officials were uncertain about whether the Cost Savings Program could be exported to other existing or future contracts, including the cost effectiveness of the program, because NNSA has not gathered information on and documented its analysis of the costs and potential benefits of the Cost Savings Program. By gathering information on and documenting its analysis of the results of the Cost Savings Program, NNSA officials and contractor representatives could make a better- informed decision about whether to implement aspects of the Cost Savings Program under existing contracts or as part of future M&O contracts. NNSA has not shared information on specific efficiencies that could be applicable to other sites because NNSA officials have not submitted such lessons learned to DOE’s Corporate Lessons Learned Database. By sharing information on potentially beneficial efficiencies and lessons learned from the Cost Savings Program at Y-12 and Pantex throughout the enterprise, NNSA could help achieve cost savings enterprise-wide even without implementing formal cost savings programs at other sites. Additionally, none of the other NNSA sites have an established site-wide baseline. NNSA has not evaluated whether it should require the other sites to have such a baseline. By evaluating whether to require other sites to institute a baseline—either in whole or in part for certain programs at the different sites—NNSA could increase financial transparency agency- wide. We are making the following four recommendations to NNSA: The NPO Cost Savings Program Manager should work with CNS to evaluate the remaining site reinvestment funds and develop and implement a plan for how best to aggregate and use them. (Recommendation 1) The Associate Administrator for Acquisition and Project Management should gather data on and document an analysis of the Cost Savings Program, including its cost effectiveness, to determine whether it is exportable to existing or future contracts. (Recommendation 2) The NPO Cost Savings Program Manager should share relevant lessons learned with other NNSA sites so that those sites can determine if efficiencies CNS has achieved can be implemented at other sites. (Recommendation 3) The Associate Administrator for Acquisition and Project Management should evaluate whether to require all other sites to institute an Annual Controlled Baseline. (Recommendation 4) We provided a draft of this report to NNSA for review and comment. The agency provided written comments, which are reproduced in appendix I; the agency also provided technical comments that we incorporated in the report as appropriate. NNSA agreed with three of the recommendations and agreed in principle with the fourth. Regarding our second recommendation that NNSA gather data on and document an analysis of the Cost Savings Program, including its cost effectiveness, to determine its exportability to existing or future contracts, NNSA agreed that the potential benefits of a Cost Savings Program should be considered for future contracts, as applicable. However, in its written comments, NNSA stated that the Cost Savings Program was uniquely intertwined with the consolidation of the two sites, Y-12 and Pantex, under one contract. As we discussed in the report, roughly 90 percent of the savings from the Cost Savings Program were attributed to transforming site operations to create a more efficient and sustainable enterprise, and not associated with merging the two sites. We continue to believe that by gathering data and documenting an analysis of the Cost Savings Program for its exportability, NNSA will be able to make better- informed decisions about whether to implement the program at other existing or future contracts. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, the Administrator of NNSA, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or bawdena@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in appendix II. Allison B. Bawden, (202) 512-3841, or bawdena@gao.gov. In addition to the individual named above, key contributors to this report included Hilary Benedict (Assistant Director), Jessica Lewis (Analyst in Charge), Antoinette Capaccio, Cindy Gilbert, Dan Royer, Holly Sasso, Sheryl Stein, Breanna Trexler, and Monique Williams.", "summary": "NNSA relies on M&O contracts to manage and operate its eight laboratory and production sites. In 2013, NNSA awarded a consolidated M&O contract to CNS for the Y-12 and Pantex sites to reduce costs. In the contract, NNSA required that CNS create a Cost Savings Program. CNS proposed it would save about $2.9 billion over the contract's potential 10-year term. The Senate committee report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2019 includes a provision for GAO to review the cost savings achieved from the competition and award of the CNS contract. GAO's report examines the extent to which (1) CNS achieved proposed cost savings from fiscal year 2014 through fiscal year 2018 and (2) NNSA identified benefits associated with the Cost Savings Program and used that information to improve its M&O contracts. GAO reviewed documentation and data on the Cost Savings Program from NNSA and CNS, interviewed NNSA headquarters and field office officials as well as representatives from M&O contractors, and toured the Y-12 site to understand examples of cost savings initiatives. The National Nuclear Security Administration (NNSA) verified about $515 million in cumulative cost savings claimed by Consolidated Nuclear Security, LLC, (CNS) from fiscal year 2014 through fiscal year 2018 (see figure). CNS was awarded the management and operating (M&O) contract for both the Y-12 National Security Complex (Y-12) in Tennessee and the Pantex Plant (Pantex) in Texas. Those savings represented about 80 percent of the approximately $640 million CNS proposed it would save through the end of fiscal year 2018. CNS achieved most of the savings through labor savings—for example, by reducing positions. While CNS's and NNSA's methods for calculating and verifying savings evolved in the early years of the contract, GAO concluded the $515 million in reported cumulative savings represents a reasonable estimate. However, due to differences between proposed and achieved savings through fiscal year 2018, and annual savings projections that are lower for the remaining years of the contract, it may be difficult for the contractor to achieve its total proposed $2.9 billion in savings over the potential 10-year contract that would end in 2024. NNSA officials identified three key benefits of the Cost Savings Program—achieving savings, reinvesting in site infrastructure, and increasing financial transparency—but has not determined whether the program could be implemented at other sites to improve its M&O contracts. For example, NNSA officials said achieving cost savings at other sites could be useful, and most M&O contracts include a clause under which sites could implement a Cost Savings Program with some attributes of the program at Y-12 and Pantex. However, NNSA is not planning to implement the Cost Savings Program—or a variation of it—at other sites. NNSA officials and contractor representatives were uncertain about whether the Cost Savings Program could be exported to other existing or future contracts because NNSA has not gathered information on nor documented its analysis of the Cost Savings Program. GAO has previously found that leading organizations gather and analyze data to identify opportunities to reduce costs, among other reasons. By performing such an analysis, NNSA officials and contractors' representatives could make better-informed decisions about whether to implement aspects of the Cost Savings Program under existing contracts or as part of future M&O contracts to achieve additional savings in the future. GAO is making four recommendations, including that NNSA document its analysis of the Cost Savings Program to determine whether it is exportable to other contracts. NNSA generally agreed with the four recommendations.", "document_type": "gao"}
{"report": "DOD’s guidance states that its FFRDCs are created to (1) provide strategic value through independent, intellectually rigorous, relevant, and timely products and services; and (2) support the department’s goals of long-term improvement in operations and enhanced national security. They are managed by various military departments or divisions within the department, called primary sponsors. FFRDCs are operated by universities, other not-for-profit or nonprofit organizations, or private firms—called parent organizations—under long-term contracts. They provide special research and development services that generally cannot be readily satisfied by government personnel or private contractors. For example, the Lincoln Laboratory develops key radar and electronic warfare technologies for integrated air and missile defense systems. The Software Engineering Institute provides cybersecurity solutions for defense entities. DOD’s FFRDCs are grouped into three categories: research and development laboratories, study and analysis centers, and systems engineering and integration centers. DOD oversees 10 FFRDCs (see table 1). According to the Director of Laboratories and Personnel within the OUSD for Research and Engineering, he took over responsibility for managing FFRDCs in July 2018, following a reorganization of OUSD for Acquisition, Technology and Logistics. DOD and each FFRDC have a sponsoring agreement, which is a stand- alone, bilateral, written agreement between the primary sponsor and the parent organization. It must be approved by the Under Secretary of Defense for Research and Engineering prior to award of an FFRDC contract and is incorporated into the contract. According to DOD’s guidance, the sponsoring agreement defines the FFRDC’s purpose and mission, establishes the conditions under which DOD may award an FFRDC contract, and describes the overarching requirements for operation of the FFRDC. For example, the primary sponsor must include provisions in the sponsoring agreement to prevent real or perceived organizational and personal conflicts of interest. As part of that, sponsors are to require FFRDC parent organizations to establish and maintain policies and procedures to protect information, such as sensitive data, from disclosure and provide training that covers ethics and conflicts of interest. We reported in December 2019 that representatives from the five study and analysis center FFRDCs said they provide annual training covering ethics and conflicts of interest for all personnel. DOD may use FFRDCs to perform work that is closely associated with the performance of inherently governmental functions or that is critical to maintaining control of the department’s missions and operations. Work could include activities such as support for financial analyses, policy development, acquisition planning, source selection, and contract management. In the course of performing work, FFRDCs may need access to acquisition data collected from DOD’s prime contractors and program offices. FFRDCs may obtain these data through DOD personnel, government databases, or directly from prime contractors. Government- held data may be stored and managed in department-wide databases or by individual program offices. For example, the Cost Assessment Data Enterprise is a web application that allows users access to various reports that include information such as major defense acquisition programs’ cost, software, and technical data. On December 21, 2017, DOD issued implementing guidance that marked the launch of its 3-year pilot program. According to DOD officials, prior to the start of the pilot program, FFRDC researchers needed to obtain permission from each data owner (e.g., DOD prime contractor or supplier), typically by signing a nondisclosure agreement. According to DOD officials that requested the authority to allow FFRDCs to have increased access, one of the purposes of the pilot was to allow for a streamlined nondisclosure agreement process. Under the pilot program, FFRDC researchers no longer have to obtain nondisclosure agreements with each data owner. To participate in the pilot, the FFRDC and DOD sponsor must first take steps to ensure certain protections are in place to protect against unauthorized disclosure or use of the data being accessed. For example, according to the statute, in order to be eligible, participating FFRDCs and its personnel (FFRDC researchers) had to agree to be subject to and comply with appropriate ethics standards and requirements applicable to government personnel, including the Ethics in Government Act of 1978, the Trade Secrets Act, and the Procurement Integrity Act. After the protections are in place, the FFRDC and DOD sponsor can enroll individual projects in the pilot program. Per the implementing guidance, the FFRDCs and DOD sponsors agree to collect and provide information about the enrolled projects. For example, DOD sponsors must provide the Laboratories and Personnel Office quarterly updates on a project’s progress obtaining data and, once the project is complete, information on the results of its access to sensitive data under the pilot program. Six of 10 DOD FFRDCs elected to participate in the pilot program during its first 21 months (figure 1). According to FFRDC representatives, the decision regarding whether an FFRDC would participate in the pilot program primarily depended on two factors: (1) the data needs of the FFRDC’s projects and (2) the ability of FFRDCs to access necessary data without the pilot program. Representatives from the six participating FFRDCs told us they elected to participate because they required access to sensitive data and, in some cases, lacked viable options for obtaining that data. For five of these FFRDCs, representatives said their researchers had identified specific projects for which they were interested in using the pilot to gain access to data sources with sensitive data from numerous contractors. Representatives from the four nonparticipating FFRDCs said that the existing processes the FFRDCs have in place provide the access they need for their projects. For example, Lincoln Laboratory representatives said their researchers are often working with an individual program or working to advance a specific technology; therefore, their work is generally with a limited number of contractors. In cases where they have needed access to sensitive data to do this work, they have executed a blanket nondisclosure agreement with their primary sponsor and, in some cases, have executed more tailored nondisclosure agreements with companies when obtaining information directly from a defense contractor. Aerospace has a blanket nondisclosure agreement with the Air Force Space and Missile Systems Center and the center included a provision in its contracts that requires its prime contractors to directly share information with the FFRDC. A Center for Communications and Computing representative said existing processes already provide the access they need for their projects. According to the FFRDC’s sponsor, its work focuses more on technological development rather than acquisitions analysis. Participating FFRDCs reported 33 projects enrolled in the pilot program from January 2018 through September 2019. Pilot projects represented about 1.5 percent of these FFRDCs’ total number of projects as of June 2019. Of the projects enrolled in the pilot program, 11 were complete and 10 were ongoing as of the end of September 2019. In addition, 11 projects enrolled in the pilot initially, only to realize they did not require access to the requested data and thus were removed. One project was put on hiatus pending a decision about whether it will continue. Table 2 summarizes the status of the projects in the pilot program as of September 2019. According to DOD officials, the fiscal year 2018 reorganization of OUSD for Acquisitions, Technology and Logistics into two offices, coupled with changes in leadership, shifted attention away from the pilot program design and implementation. For example, an official from the Acquisition and Sustainment office also told us it missed an opportunity to conduct outreach with its FFRDC because the office did not hold its biannual meetings in 2017 or 2018 due to the reorganization. In these biannual meetings, he explained, they would have discussed the department’s future research priorities and how the pilot program may have helped. This official—who was involved in the pilot’s implementation—also noted that the shift in attention meant they did not engage with the offices that maintain the data repositories as fully as they would have liked. We found it took sponsors and FFRDCs from a few weeks to 7 months to resolve questions about pilot program requirements and update the FFRDCs’ sponsoring agreements to incorporate the pilot protections. During that time, FFRDCs were unable to move forward with certain analyses for their proposed projects. Of the six projects we selected for further review (shown in table 3), four have been completed and FFRDCs reported benefits from their pilot program participation. The two remaining projects are on hiatus or removed. DOD officials and representatives from the four completed projects shared with us the following benefits: Systems and Analyses Center assessment of the U.S. munitions defense industrial base capacity. Portions of the research required access to sensitive data about the availability and production levels of manufacturing parts for a large number of contractors and suppliers. The FFRDC researchers used these data in their microlevel assessments of the manufacturing capacity and supply chain resiliency of the U.S. defense munitions industrial base. They said they were able to provide DOD’s Industrial Policy office with a more complete picture by combining these microlevel analyses with broader analyses of employment trends and economic outputs. A DOD industrial policy official who requested the work also said that the analysis enabled her office to meet an executive branch reporting requirement, which DOD did not have the manpower to conduct. National Defense Research Institute support for analysis of munitions industrial base. FFRDC researchers worked in collaboration with government officials to perform analyses on the adequacy of the munitions and missiles industrial base using government-held data from prime contractors and subcontractors. For example, the researchers supported working groups examining propulsion and chemicals in munitions and provided analysis for a report to Congress on solid rocket motors. The DOD official that requested the work and National Defense Research Institute representatives said that, without the pilot, the FFRDC would not have been able to access the data used to support DOD in these efforts. The official also noted that in this case the FFRDC helped fill a gap in DOD’s workforce to meet a congressional reporting requirement. National Security Engineering Center and Software Engineering Institute analysis of software acquisitions practices. FFRDCs supported a Defense Innovation Board study that aimed to identify correlations between software complexity, cost, and schedule evolution. FFRDC researchers’ access to and use of the data provided important insights about the quality and reliability of the department’s data. Specifically, DOD gained further insight into the kinds of software data the department holds and the significant gaps that would need to be addressed to improve overall DOD-held data quality. The Defense Innovation Board’s report included findings related to the quality of the software data accessed and analyzed by these two FFRDCs. Project AIR FORCE assessment of contractor risk. According to FFRDC representatives and an Air Force official involved in the work, the pilot program facilitated the FFRDC’s access to sensitive data held by the Defense Contract Management Agency that researchers used to identify early indicators of contractor performance risks. In response to the results of this work, the Air Force has funded a follow- on project to further research the potential of data analytics to provide early indicators of challenges in contract execution, according to an Air Force official involved with project. In addition to the benefits at the project level, several DOD sponsor officials and FFRDC representatives also noted the benefits of using the streamlined nondisclosure agreement process to accomplish their work. According to several DOD sponsor officials and FFRDC researchers we spoke to, completing the requested analysis without the pilot program would have required individual nondisclosure agreements with hundreds of individual contractors and suppliers. Systems and Analyses Center representatives said this would have been essentially impossible, and would have prevented researchers from completing important parts of the analyses. In another case, a Software Engineering Institute representative told us that, before the pilot, their team could not access software data when attempting to complete a 2017 project involving DOD software costs and production time frames. For that project, the DOD organization responsible for the data repository had recommended researchers send out a data request letter to each of the contractors with data in the system. Researchers sent out roughly one hundred requests to contractors for permission, but received no responses. They pointed out this was in part because contractors have no incentive to respond to an FFRDC’s request for access to their data. As a result, Software Engineering Institute was unable to use updated data for the 2017 report. While several sponsoring agency officials noted benefits of using the data for analyses to inform key program decisions, they also noted that a causal relationship between the pilot program activities and acquisition process improvements would be hard to establish, in part due to the length of time needed for projects to effect change. DOD officials responsible for two completed projects examining the munitions industrial base said they expect the analyses performed will lead to improved acquisition processes but that it would take many years to see the benefits. Specifically, they said the FFRDCs’ work helped identify areas for improvement in the department’s budget and acquisition strategy to better signal future demand to its lower tier munitions industrial base suppliers. In addition to noting these expected improvements, several DOD officials also acknowledged that expanding access of sensitive data to more people increases the potential for unauthorized use or disclosure but said that the pilot program put in place important protections to help mitigate these risks. DOD’s guidance to implement the pilot program outlined protections the FFRDCs must agree to, in order to guard against unauthorized disclosure or use of sensitive data, and required that these protections be incorporated into the sponsoring agreements between the FFRDCs and the DOD sponsor. However, we found some instances where details of the required protections were not incorporated into the agreements. We also found that the Laboratories and Personnel Office, which is responsible for managing the pilot program, does not have a procedure to verify whether protections were implemented, in part because it has not developed a process for doing so. Table 4 explains the protections. Some of these protections were already part of the FFRDCs’ business operations, while others are new. For example, the prohibition on their use of sensitive data to compete against a third party was already a fundamental aspect of FFRDCs’ role in supporting DOD. Similarly, participating FFRDC representatives told us that certain protections, such as implementing nondisclosure agreements and training, required only small adjustments to their existing procedures. However, the pilot’s financial disclosure program, annual certifications by parent organizations, and instructions for researchers to notify contracting officers of employment offers when supporting source selection decisions were new and specific to the pilot, according to a DOD official involved in the pilot’s implementation. We found that not all the details of protections were incorporated into the sponsoring agreements we reviewed. According to DOD’s implementing guidance, to participate in the pilot program, FFRDCs must agree to and follow these protections, which are to be incorporated into FFRDCs’ sponsoring agreements. We found that all six participating FFRDCs’ sponsoring agreements were updated and that most of the protections were incorporated. However, none included the instructions for FFRDC personnel involved in source selections to notify contracting officers if they are contacted about employment by an entity whose proposal is being evaluated and recuse themselves. We also found that the sponsoring agreements omitted one of the three officials that should be notified in the event of a Trade Secrets Act violation. These details were also not included in the templates DOD provided sponsors to use when updating FFRDC sponsoring agreements. When we raised these gaps to the attention of the DOD office responsible for managing the pilot program, the officials we spoke with were unaware of these omissions. In addition, the Laboratories and Personnel Office has not taken steps to ensure that another protection—the certification of the annual review of financial disclosure forms—has occurred, even though it was incorporated into the sponsoring agreements. The implementing guidance states that FFRDCs’ parent organizations must certify the annual review of financial disclosure forms and archive these forms for 6 years. However, only two FFRDC parent organizations provided us with this certification. According to representatives from parent organizations of the other four FFRDCs, the review of financial disclosures is generally performed as part of their conflict of interest programs. They review the disclosures on an annual or rolling basis when researchers are assigned to new projects but had not certified, as the sponsoring agreements require, that they have taken this step for the pilot program. We found that the Laboratories and Personnel Office had not taken steps to verify FFRDC parent organization compliance with this protection, such as collecting or reviewing the certification. When we raised this gap to the attention of the DOD office responsible for managing the pilot program, the officials were unaware of the missing annual certifications. By not ensuring the annual review is occurring, DOD has limited information about FFRDCs’ adherence to this pilot program protection. The pilot’s implementing guidance also states that, before government personnel provide access to sensitive data, the FFRDCs and researchers must have addressed these protections. However, the Laboratories and Personnel Office has not taken steps to ensure it is done. In our review of the six specific projects, we found that different people were checking that some of the protections were in place. For example, For two of the six projects, a primary sponsor official had a copy of the FFRDC addendum, and collected and reviewed the nondisclosure agreements and certifications of financial disclosure for individual researchers on each project. For a third project, a DOD official in the office that requested the project told us she confirmed that FFRDC researchers working on the project were part of the pilot program and told the official from the data repository that he could share information with the researchers. For the remaining three projects, representatives for a data repository that provided researchers with data access told us they confirmed that the addendum was incorporated into the sponsoring agreement and that researchers had the individual protections, such as a nondisclosure agreement, in place before providing access to the data. Standards for internal controls in the federal government state that responsibilities for control activities, such as sponsors ensuring the protections are incorporated into the agreements and that FFRDCs are following these protections, should be documented through policy and procedures. Without a process that includes clearly defined roles and responsibilities to ensure the protections are followed, DOD cannot ensure that its goal to safeguard sensitive data is achieved. DOD established what information sponsors must collect about the projects enrolled in the pilot in its implementing guidance to sponsors and notified FFRDCs about these responsibilities. The requirements include: pre-action information to be collected when the project is enrolled in the pilot, which includes basic details about the project, the data required, and planned analysis; quarterly status updates, which include progress obtaining access to sensitive data and any challenges or barriers to access; and post-action information regarding the results of pilot access when the project’s analyses are completed, which includes a summary of how the pilot supported FFRDC research, and any benefits accrued to DOD from pilot participation. The implementing guidance also instructs sponsors to collect information about the project’s results again 6 months after the project is completed. The Laboratories and Personnel Office, which is responsible for managing the pilot program, sends an email quarterly requesting that sponsors submit information. However, we found that the Laboratories and Personnel Office did not receive pre-action information from 11 of the 33 projects in the pilot program. For example: Systems and Analyses Center has not submitted pre-action information for six of its eight projects. An official from the Systems and Analyses Center’s primary sponsor office told us that in his view, pre-action information can be obtained by other means, and he had not requested it. Project AIR FORCE and Arroyo Center have submitted project pre- action information for five projects to the FFRDCs’ primary sponsors: Air Force and Army, respectively. However, the primary sponsors have not provided this information to the Laboratories and Personnel Office. An Air Force official explained that he gets a request from the Laboratories and Personnel Office for the quarterly reports, but not the pre-action information, and thus had not provided it. In addition, the office did not collect a quarterly report for three projects in the pilot, and, as of September 2019, two completed projects had passed the 6-month post-completion time frame and only one had submitted post-action information. These gaps in reporting have occurred because the Laboratories and Personnel Office is not monitoring the project information it receives to ensure sponsors are submitting all required reporting. DOD’s implementing guidance states that primary sponsors will collect and submit this information for each project enrolled in the pilot program. Further, GAO’s leading practices for pilot design state, among other things, that a well-designed pilot program should have a clear approach to gathering information for the purpose of supporting the future evaluation of the pilot and tracking the pilot program’s implementation and performance. Consistency in collecting pre-action, quarterly, and post- action reports is important because each contains different information, which could be useful for the department to track the pilot program’s progress and in an evaluation of the pilot program. For example, without the pre-action information from Systems and Analyses Center, Arroyo Center, and Project AIR FORCE, the Laboratories and Personnel Office will not have general descriptions of their pilot projects or information about the kinds of data these FFRDCs initially planned to access. Without complete information, DOD will not be able to effectively evaluate the pilot program and inform future decisions about the program’s status. We found that DOD followed some but not all of the leading practices for evaluating its pilot program. According to GAO’s leading practices for pilot design, a well-developed and documented pilot program can help ensure that agency assessments produce information needed to make effective program and policy decisions. Such a process enhances the quality, credibility, and usefulness of evaluations, in addition to helping to ensure that time and resources are used effectively. Five leading practices form a framework for effective pilot design and evaluation. (See figure 2.) We found DOD generally addressed the first of the leading practices of pilot design—establish objectives—by establishing goals for the pilot program, summarized below. Make sensitive data previously restricted or unavailable available for analysis. Use sensitive data in accordance with the FFRDC contract. Safeguard sensitive data. Document results of pilot program. Document risks or costs of FFRDC access to sensitive data. Gain analytic value from FFRDC access to sensitive data. Demonstrate benefits to government from sharing sensitive data. Inform future actions for making FFRDC access to sensitive data permanently available. However, DOD has not fully addressed the other leading practices. Specifically, we found that the Laboratories and Personnel Office does not have: a plan that (1) includes an assessment methodology to ensure DOD is collecting the correct information to evaluate whether the pilot has met the department’s goals and (2) defines how DOD will use the information collected to evaluate the implementation and performance of the pilot program, when the evaluation will take place, and by whom; a plan for identifying or documenting lessons learned; and a plan for gathering input from stakeholders, such as DOD sponsors, FFRDCs, and officials from DOD’s data repositories, for the pilot program’s evaluation. According to officials involved in its implementation, DOD did not consider creating such plans when developing the pilot program. The pilot program guidance, however, stated that information collected would be used for the department to assess the ongoing efficacy of the pilot program and GAO’s evaluation. These officials explained that, when the pilot’s guidance was formulated, the department was in the process of reorganizing the former Acquisition, Technology, and Logistics Office and they pointed out that GAO was to do the assessment of the pilot program. They said the reason the program collected information—such as quarterly reports—was to inform our review. Thus, they had no plan to assess the information collected and no plans to talk to stakeholders or to collect and share lessons learned. While our review occurred during pilot implementation, an evaluation of the pilot conducted by DOD after more projects are completed would provide an opportunity to identify lessons learned and gather valuable input from stakeholders—such as the offices that manage the data repositories and the sponsors requesting the projects. We found cases where FFRDC researchers had problems accessing data and where gathering this input from stakeholders involved with pilot projects would have been useful for DOD. For example, some FFRDC researchers described barriers when trying to gain access to certain government- and department-wide databases. In one case, Project AIR FORCE researchers reported not being able to access information in the Electronic Document Access database and various databases containing contractor performance information because researchers lacked military or government email addresses. Further, some of the databases that FFRDC researchers and a DOD official said would be useful are not owned by DOD. We found that guidance for one such database explicitly prohibits disclosure of contractor evaluation data to any contractor or non- government entity. In addition, the researchers were able to gain only partial access to DOD’s Acquisition Information Repository (a database containing acquisition documents for DOD’s major weapons programs) and, as a result, were unable to access individual documents, such as program assessment reports. They told us the repository is set up such that the researchers must request access to individual documents directly from document owners, who set permissions when uploading documents, rather than from a central source that can grant access across the repository. Without further evaluation of the pilot, DOD is missing an opportunity to benefit from gathering input from its stakeholders and identifying lessons learned, such as learning and understanding more about these barriers to accessing certain databases. There is still time for DOD to develop an evaluation plan with elements described in our leading practices. The pilot program ends December 21, 2020. Our review comes at a time when 11 of the overall 33 projects have been completed; therefore, information exists to report on outcomes. Officials and representatives from the Laboratories and Personnel Office and participating sponsors and FFRDCs expressed a continued need for access to the sensitive data. Without an evaluation plan, DOD will have difficulty determining the effectiveness of the pilot to meet its goal of accruing more analytic value for the department while also safeguarding sensitive data. The FFRDC pilot program has already provided DOD with some benefits, as a few FFRDCs have reported success in completing analysis that would not have been possible without it. However, in implementing the pilot, DOD has room for improvement. A key control of this pilot that provides access to sensitive data is ensuring protections are in place to prevent improper disclosure. Another control is to establish a process to ensure these protections are followed, yet the responsible office within DOD has not done so. Further, despite the fact that the pilot is past the midpoint of implementation, this office still has an opportunity to develop a plan on how to evaluate it. But to do this, it must develop a mechanism to ensure it is collecting complete information on the pilot activities. Ensuring comprehensive reporting and implementing a well-developed evaluation plan will help DOD understand and articulate the benefits the department has accrued because of FFRDC’s access to sensitive data. Further, through identifying lessons learned and obtaining stakeholder input, the Laboratories and Personnel Office has an opportunity to better understand the challenges FFRDCs and the department face when attempting to access and use sensitive data included in government- and department-wide databases. Such an evaluation could help inform Congress’ decision whether to extend, make FFRDC access permanent, or end the pilot. We are making the following six recommendations to the Department of Defense: The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to take steps to ensure that the details of the pilot program’s data protections are incorporated into the existing agreements. (Recommendation 1) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to take steps to ensure that the FFRDCs and sponsors are implementing the pilot program’s protections for sensitive data. (Recommendation 2) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to establish a monitoring and oversight mechanism to ensure that primary sponsors submit complete information on pilot projects, as required by DOD’s guidance for the pilot program. (Recommendation 3) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to develop a plan that outlines the methodology by which DOD will assess the pilot and how and when information collected will be analyzed to evaluate the pilot program. (Recommendation 4) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to develop a plan to identify and evaluate lessons learned from the pilot program. (Recommendation 5) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to develop a plan for obtaining input from stakeholders on the pilot program. (Recommendation 6) We provided a draft of this product to DOD for comment. DOD provided a letter response, reproduced in Appendix I. DOD agreed with our recommendations and described actions that it intends to take in response. We also provided excerpts of this product to FFRDCs for comment, of which three provided technical comments that we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in Appendix II. In addition to the contact named above, Tatiana Winger (Assistant Director), Leslie Ashton (Analyst-in-Charge), Evan Nemoff, Tanya Waller, Jenny Chanley, Laura Greifner, Christine Pecora, and Roxanna Sun made key contributions to this report.", "summary": "FFRDCs provide federal agencies with research and development functions, technical systems engineering capabilities, and policy development and decision-making studies, among other services. The Federal Acquisition Regulation states that FFRDCs have a special relationship with DOD, which can give FFRDCs access to sensitive data beyond what would commonly be shared with contractors. The National Defense Authorization Act for Fiscal Year 2017 directed DOD to establish a 3-year pilot program that allows FFRDCs streamlined access to sensitive data maintained by DOD. It also included a provision for GAO to report on the pilot program within 2 years of implementation. This report addresses the extent to which (1) FFRDCs are using the pilot program, (2) DOD put procedures in place to protect data accessed, and (3) DOD is evaluating the pilot program. GAO reviewed DOD guidance and FFRDC processes, pilot reports for January 2018 through September 2019, and DOD's plans and efforts for evaluating the pilot program. GAO also selected a nongeneralizable sample of six projects—at least one from each FFRDC with an enrolled project as of December 2018—for further review. In addition, GAO assessed the pilot program against leading practices for pilot design. The Department of Defense (DOD) launched a 3-year pilot program in December 2017 to enable a streamlined process to share certain sensitive data, such as data collected from its contractors, with its Federally Funded Research and Development Centers (FFRDC). At times, FFRDCs need to access such data to support DOD. The pilot was intended to reduce the burden on FFRDCs to seek permission from hundreds of contractors to access information needed for their research. Six of DOD's 10 FFRDCs have taken part in the pilot, enrolling a combined total of 33 projects, as shown in the table. DOD officials and FFRDC representatives reported that the streamlined process made the use of sensitive data feasible. As a result, FFRDCs with completed projects in GAO's sample indicated they were able to provide more robust analyses or insights to DOD. DOD guidance for the pilot program established procedures to protect sensitive data. But GAO found that DOD did not incorporate all of the details of the required protections into its agreements with FFRDCs. Further, GAO found that not all FFRDCs were performing annual certification of financial disclosure forms, as required by its agreements with DOD. DOD does not have a process to ensure that all the protections pertaining to FFRDCs' streamlined access to sensitive data are being followed. Without a process that defines roles and responsibilities, DOD cannot ensure that FFRDCs adhere to the protections. DOD developed goals for the pilot program and outlined what information was to be obtained for each participating project, actions that are consistent with GAO's leading practices for pilot design. However, DOD has not developed a plan for evaluating the program nor has it consistently collected information on about a third of the pilot projects. Leading practices for pilot design call for an evaluation plan, which should include an assessment methodology and identify responsibilities as to how the evaluation will be conducted. Without an evaluation plan and a mechanism to collect information on pilot projects, DOD will not be positioned to identify the effectiveness of the pilot program and benefit from lessons learned. Such information will be useful as Congress considers the path forward after the pilot ends in December 2020. GAO is making six recommendations, including that DOD take steps to ensure data protections are in the agreements and followed, collect information on projects, and evaluate the pilot. DOD agreed with the recommendations.", "document_type": "gao"}
{"report": "The Constitution gives Congress the power to coin money, and under that authority, Congress has specified the coins that can be produced and the metal composition of circulating coins, including the penny, nickel, dime, quarter, and half-dollar. Congress has also passed legislation prohibiting the use of appropriated funds to redesign the $1 note. Within the Department of the Treasury (Treasury), BEP produces notes and the Mint produces coins. To ensure that notes and coins are available in sufficient quantities to meet public demand, the Federal Reserve orders new notes from BEP and new coins from the Mint. The Federal Reserve pays BEP for the cost of producing the notes; the Mint pays for the cost of producing coins and the Federal Reserve pays the Mint for the face value of the coins. The Federal Reserve distributes the notes and coins to approximately 8,400 depository institutions—banks, savings and loans, and credit unions—in the United States through cash offices operated by its 12 regional Reserve Banks. The Reserve Banks also are responsible for ensuring the quality and integrity of notes in circulation by assessing the condition of each note and destroying any that are unfit. When a depository institution deposits currency with a Reserve Bank, each currency note is verified on high-speed processing equipment using electronic authentication and fitness sensors. During the “piece- verification” process, the deposited currency is counted, suspect counterfeit notes are identified and segregated, and unfit notes are destroyed. The fit currency is packaged and used to fill future orders for currency from depository institutions. The destroyed notes are replaced with new notes from BEP as there is public demand for cash. The federal government spent about $1.3 billion to produce, process, and circulate notes and coins in 2017. These costs are offset by the financial benefit the government realizes when it issues notes or coins because currency usually costs less to produce than its face value. This benefit, which is known as seigniorage, is the difference between the face value of currency and its cost of production; this difference provides a financial benefit to the government when the government issues currency. In calendar year 2017, the Federal Reserve reported transferring about $81 billion to the Treasury, and the Mint reported transferring about $269 million in fiscal year 2017. The seigniorage the Federal Reserve and the Mint pay into the Treasury reduces the need for the government to borrow money, and as a result, the government pays less interest over time. Other countries have taken steps to reduce currency costs by replacing notes with coins of the same value and eliminating the smallest value coin. For example, Canada introduced a $1 coin in 1987 and a $2 coin in 1996 that replaced corresponding-valued notes, and the United Kingdom replaced its £1 note with a £1 coin in 1983. These countries expected a cost reduction because, while coins are generally more expensive to produce than notes, the coins can last substantially longer in circulation. For example, in both countries, the $1 and £1 notes, respectively, lasted 18 months or less while coins, according to experts, can be expected to last more than 30 years. As a result, these countries’ governments expected to save money because over 30 years, the number of coins they would produce was far less than the number of notes they otherwise would have made. These countries may have realized further financial benefits by replacing notes with coins because the public may hold more cash if a note is replaced with a coin and, as a result, the government would achieve a greater benefit from seigniorage. As we reported in 2011, because of differences in how people use coins and notes, the public may hold more than one coin for each note being replaced. Since people often store coins at home and store notes in their wallets, coins, as a result, circulate less frequently than notes and therefore more coins are needed to meet public demand. Thus, for a given denomination of currency, a larger number of coins would need to be maintained in circulation to meet the public’s demand for cash than would be needed if that denomination were provided in notes. For example, we previously reported that when Canada replaced its $1 note and the United Kingdom replaced its £1 note with a coin, both countries anticipated they would need to produce 8 coins to replace 5 notes, or a 1.6-to-1 replacement ratio. In previous work, we reported a positive annual net benefit to the government of replacing the $1 note with a $1 coin. In 2011, we reported a 30-year net benefit of $5.5 billion. Based on these results and the experiences of other countries, we have previously recommended that Congress consider and pass proposals to replace the $1 note with a $1 coin and, to ensure success of the coin, also provide for the elimination of the $1 note. While the production of the $1 coin has been authorized in law, elimination of the $1 note has not, and the U.S. has continued producing it. The U.S. has not eliminated any coins or altered any coin’s metal composition since 1982. Some countries have also eliminated their low- denomination coins to reduce currency costs. In 2013 Canada eliminated its one-cent coin because the cost to make it was more than it was worth and the coin’s usefulness had declined due to inflation. Over time, the costs of making these coins has increased due, in large part, to increases in the costs of metals used in coins—copper, zinc, and nickel. Since fiscal year 2006, both the penny and nickel have cost more to produce than their face value, according to our analysis of Mint data. (See fig. 1.) For example, in 2017, the Mint spent approximately 1.8 cents to produce each penny and approximately 6.6 cents to produce each nickel. Because the Mint sells coins to the Federal Reserve at face value, both coins cost more to produce than the Mint receives for them. As a result, in 2017, the Mint incurred net losses of about $69 million to produce the penny and about $21 million to produce the nickel. The dime and the quarter, however, cost less to produce than their face value. The combined cost to produce all widely circulating coins (the penny, the nickel, the dime, and the quarter) is less than their combined face value, so the government continues to realize positive seigniorage overall from producing circulating coins. The Coin Modernization, Oversight, and Continuity Act of 2010 authorized the Secretary of the Treasury to conduct research on alternative materials that could be used in coins. In response, the Mint conducted research on alternative metals, identified metal alloys that offered the potential for cost savings, and reported its results to Congress in 2012, 2014, and 2017. According to our analysis, the government would likely incur a net loss over 30 years if it replaced the $1 note with a $1 coin. We conducted a number of simulations that used different sets of assumptions to estimate the net benefit to the government of replacing the $1 note with a $1 coin. In almost every simulation, the net benefit to the government from switching to a $1 coin was negative, or an overall net loss (see app. I). For each set of assumptions, we simulated the status quo scenario in which notes are not replaced by coins, as well as two replacement scenarios. Under “gradual replacement,” the Federal Reserve would replace $1 notes with $1 coins as the notes became unfit for circulation. Under “active replacement,” notes would be replaced by coins more quickly because the Federal Reserve would destroy unfit notes as well as some fit notes each year and replace them with $1 coins. In both replacement scenarios, we assumed that the public would increase its holdings of cash when coins are used instead of notes and that the replacement ratio would be 1.5 coins for each note. We found that the present value of the net loss incurred by the government over 30 years would be about $2.6 billion with gradual replacement and about $611 million with active replacement (see fig. 2). Each simulation we conducted accounts for both costs and benefits to the government. The costs include production and processing costs for $1 coins and $1 notes, as appropriate. The coin replacement scenarios each include one-time startup costs that would be incurred upfront, in addition to recurring increased costs of producing higher-denomination notes when the $1 note would no longer made. In each simulation, we calculated benefits to the government as interest savings on debt that would be avoided because of seigniorage, or the difference between the face value of the currency that would be produced and the cost of producing it. These simulations represent the first time we have found that replacing the $1 note with the $1 coin would result in a net loss to the government rather than a net benefit. The simulations are based on current data and projections from CBO and the Federal Reserve, among others, that have changed over time. For example, the increased lifespan of the $1 note relative to that of the $1 coin and the decreased cost to the Federal Reserve for processing currency are key factors in these estimates and substantially reduced the relative costs of the status quo scenario. For our 2011 report, we assumed a median lifespan of 3.3 years for the $1 note based on Federal Reserve data. Since then, the $1 note lifespan has increased, and our current simulations assume a median lifespan of 7.9 years based on the most recent data from the Federal Reserve. Due to this substantially longer note lifespan, fewer $1 notes need to be produced over a 30-year period, which reduces the cost of producing them and diminishes the relative advantage of the long coin life. In our 2011 simulations, a $1 coin was assumed to last about 10 times as long as a $1 note (34 years to 3.3 years); in our current simulations, the lifespan of the coin remains the same but is now only about 4.3 times as long as that of the note (34 years to 7.9 years). Meanwhile, the relative cost of producing coins and notes has remained about the same. According to the Federal Reserve, the increased lifespan of the $1 note is largely attributable to a series of improvements in Federal Reserve currency processing procedures and equipment that has reduced the number of notes destroyed each year. For example, prior to April 2011, depository institutions were required to deposit currency in stacks of like- notes with the portrait side of the note facing up. After discovering it was destroying many notes that were otherwise fit for circulation because they were “misfaced,” the Federal Reserve undertook an effort to increase the percentage of notes that were properly faced by manually checking and correcting notes’ orientation. Subsequently, during 2010 and 2011, the Reserve Banks installed new sensors on their high-speed processing equipment, which enabled the Reserve Banks to authenticate notes regardless of facing. In addition to increased note life, the costs that we anticipate the Federal Reserve would incur for processing notes has decreased since our 2011 analysis because it is processing fewer $1 notes. Although the cost per note for processing has remained the same—$0.003 per note, based on Federal Reserve data—the number of notes processed in 2017 was about 1.6 billion less per year than at the time of our 2011 analysis. According to Federal Reserve officials, the public may be handling and using $1 notes less and holding on to them longer. This could cause notes to circulate less frequently, reducing the number of notes processed. Our simulations show that the losses to the government from replacing the $1 note with a $1 coin would not be incurred evenly over the 30-year period. Much of the cost of producing coins to replace notes would be borne by the government in the earlier years of our simulations, while the benefits to the government would accrue gradually and become relatively more important in later years. For example, in the gradual replacement scenario, more than half of the net loss to the government occurs in the first 10 years of the 30-year period. The large net losses in the early years largely reflect the upfront costs of replacing $1 notes in circulation with $1 coins and meeting increased demand for currency. In our simulations, the interest savings then accrue over a relatively long period of time due to the 34-year median lifespan of the coin. Our simulations reflect uncertainty in the underlying projections and assumptions. In general, however, projections that are closer in time are more certain. For example, an estimate over a 10-year period would be more certain than an estimate over a 30-year period. Consequently, within our results, the estimated net loss in the first 10 years is more certain than the estimated net loss over the 30-year period. Representatives from 7 of the 10 stakeholder industries we met with would be negatively affected by a switch to a $1 coin because they stated they would incur additional costs as a result of such a change. For example, representatives from the armored carrier industry told us that they anticipate increased costs because of the additional weight of transporting $1 coins compared to $1 notes as well as the need to modify or procure additional coin-processing equipment. Representatives of the gaming industry, which includes casinos and companies who make electronic games found in casinos, said a switch to the $1 coin would be costly because the industry has generally moved away from the use of coins in favor of notes and casinos would incur additional costs for transporting and storing coins. Of the 7 stakeholder industries that said they would incur additional costs, 3 provided us with estimates of these costs. All 3, which represent industries with machines that would require modification to accept $1 coins, approximated these costs by multiplying an estimated number of units affected by an estimated per-unit cost of changing the machines. For example, a representative of the gaming industry estimated that about 98 percent of the approximately 1,000,000 electronic gaming machines in the U.S. and Canada were manufactured with no provision for accepting coins. According to this representative, the costs to convert machines to accept $1 coins could range from $130 to $175 per unit because the level of modification needed would vary. Some machines would require, for example, a newly designed faceplate, a coin acceptance mechanism, and a box for collecting coins. Most representatives from stakeholder industries said there would be no benefit to them from a switch to a $1 coin, but 3 of the 10 representatives acknowledged some benefits of doing so. Two representatives said that coins are generally less likely to jam or be rejected by the payment mechanisms than notes. The other representative—from the bulk vending industry, which sells products such as gum balls and small toys through coin-operated equipment—said a $1 coin would help the industry increase sales and offer higher-quality products than it offers now for 25 or 50 cents. According to this representative, virtually all these machines accept quarters but some require two or three quarters for a purchase. A $1 coin would increase the likelihood that consumers would have the necessary change to use these machines thus increasing their sales, according to this representative. Representatives from the remaining 3 stakeholder industries reported that switching to a $1 coin would have little or no impact on their operations. For example, a representative of operators of toll roads and bridges said that all major toll operators have adopted some form of cashless, electronic collection system. The use of cash, including coins, for toll payment has declined to 18 percent of all toll revenue in 2015, down from 29 percent in 2010, and most existing coin collection machinery currently accepts $1 coins. Similarly, a representative from the parking industry noted a trend toward increased use of cashless transactions along with a decrease in the number of coin-operated parking meters. A switch to a $1 coin would have minimal effect on the industry because virtually all parking meters take quarters. The remaining representative said additional information, such as whether a new $1 coin would be issued and whether it would have the same properties as currently circulating $1 coins, would be needed to determine whether it would incur costs from a switch to a $1 coin. A representative of an organization that advocates replacing the $1 note with a $1 coin said that switching to the $1 coin could make it easier for people with visual impairments to identify the denomination. We have previously reported that different denominations of US currency are identical in size, making it difficult for the blind or visually impaired to distinguish among them. Moreover, according to the representative, eliminating the $1 note would reduce the number of note denominations, and the $1 coin may be easier to recognize by its physical difference from other coins. Although anyone who uses currency could be affected by a switch to a $1 coin, the extent of public support for making such a change is unclear, particularly when doing so would not provide a benefit to the government. Our most recent work on public perceptions of $1 coins in 2002 found few survey respondents were using $1 coins and 64 percent opposed replacing the $1 bill with a $1 coin. A majority of survey respondents favored replacing the $1 note with a $1 coin when told that doing so could save about half a billion dollars per year—our then-current estimated net benefit to the government; we did not seek to gauge public perceptions about the same action if it were to cause a loss. Similarly, the organization advocating in support of $1 coins has reported increased public interest in a change from the $1 note when substantial cost savings are factored in. However, according to Federal Reserve officials, the public continues to express its preference for the $1 note because both the $1 coin and $1 note are available and the public overwhelmingly uses $1 notes. Moreover, Reserve Banks currently hold more than 1-billion $1 coins because there is little demand for them from the public, further demonstrating public preference for the $1 note, according to these officials. The Mint estimates that it would save about $27 million annually, or about $252 million in present value over 10 years if Congress directed it to suspend the production of the penny (see table 1). However, the Mint’s estimated savings are based on its penny production data from a single fiscal year—2017. Specifically, since the Mint lost $27.3 million from making 8.4-billion pennies that year, this amount would also represent the savings to the Mint through cost avoidance if it had not produced any pennies. Because the number of pennies produced and the base metal costs vary from year to year, future changes to production volumes and costs could alter the estimated savings. The present value of the estimated savings could also be affected by the choice of discount rate. The Mint has suspended production of some coins in the past due to a lack of demand for those coins. Specifically, the Mint suspended production of the half-dollar coins for circulation in fiscal year 2006 and the Presidential $1 coins for circulation in 2011. The Mint suspended production of these coins because demand for them was low. In contrast, demand for the penny remains strong, as the Mint produced about 8.4 billion pennies in fiscal year 2017 in response to orders from the Federal Reserve. Penny inventories at Federal Reserve Banks can meet demand for about 1 month, according to Federal Reserve officials. According to Mint officials, the Mint has not taken a position on proposed legislation introduced in the 115th Congress that would suspend production of the penny for 10 years, among other things. However, the Mint has developed a preliminary plan to implement a penny suspension if required to do so by law. According to this plan, suspending penny production would take place over a 2.5-year timeframe: the first year would be devoted to planning and preparing for penny suspension, and the next 1.5 years would be devoted to ending the Mint’s contracts with its suppliers, addressing the disposition of affected Mint personnel, and deciding what to do with excess production equipment and physical space. The Mint would also conduct outreach and communication to the public, Congress, and Mint employees during this time. The Mint is also taking steps to reduce the financial loss from producing the penny. According to Mint officials, they and the Federal Reserve are working with industry stakeholders specifically to identify alternative practices that would reduce dependency on the manufacture of additional pennies. For example, the Federal Reserve and Mint met with stakeholders to discuss these practices in January 2019. According to Mint officials, the Mint would not need to produce as many pennies if the pennies currently in circulation were more actively circulated. Mint officials stated that billions of pennies are held by banks, armored carriers, or the public. According to Mint officials, if pennies were to circulate more quickly, the demand for new pennies would be reduced, and production of new pennies could decrease and would reduce the financial losses from penny production. The Mint also estimates it would save between $2.2 million and $9.1 million annually, or between $21 million and $85 million in net present value over 10 years, by changing the metal composition of the nickel (see table 2). The Mint’s estimated savings are based on fiscal year 2017 production of 1.3-billion nickels at a cost of $86 million. The nickel currently consists of about 75 percent copper and 25 percent nickel. Based on research, the Mint reported it would achieve cost savings by changing the metal composition to about 80 percent copper and 20 percent nickel (80/20) or by changing the metal composition to a copper, nickel, manganese, and zinc combination (C99750T-M). Because the number of nickels produced and their cost varies from year to year, future changes to production volumes and costs could alter the estimated savings. Both changes in the composition of the nickel are seamless changes because nickels made of these alloys would have the same weight and electromagnetic signature as the current nickel, according to the Mint. As a result, these nickels would function the same for the public and in vending machines. However, according to Mint data, even if the Mint changes to one of these alternative metal compositions, the unit cost of producing the nickel would likely remain greater than the face value of the coin. In fiscal year 2017, the Mint spent approximately 6.6 cents to produce each nickel, which would have been reduced to about 6.4 cents if the Mint had produced the 80/20 nickel and 5.9 cents for the C99750T-M nickel. Based on authorities granted in the Coin Modernization, Oversight, and Continuity Act of 2010, the Mint has conducted research and identified potential alternative metal compositions for the dime and quarter. This research shows that the same alloys that could reduce the cost of producing the nickel could be used to reduce the costs of producing the dime and quarter. Specifically, this research indicated potential savings of $74 million over 10 years by using the C99750T-M alloy in the dime and quarter, although additional testing of the alloy is required. Changing the metal composition of circulating coins could help the Mint achieve more effective and efficient operations by reducing production costs, resulting in savings to the government and the taxpayer. The Secretary of the Treasury and Mint officials do not have the authority to alter the metal content of coins—except the penny—as metal content is determined in statute. The Mint has sought authority from Congress to change the metal composition of the nickel, dime, and quarter, if those changes meet certain requirements. Specifically, in its fiscal year 2019 budget proposal, the Mint proposed a legislative change to its authorities that would enable the Secretary of the Treasury to alter the metal composition of coins, if those changes did not affect the weight or electromagnetic signature of the coins. This proposed change is consistent with the Treasury’s 2018–2022 strategic plan, which includes a goal to introduce efficiencies to lower the unit costs of coins produced by the Mint. Legislation supporting this proposal has not been introduced. Without the authority to change the metal composition of coins, the Mint cannot fully realize operational efficiencies, even though it has identified methods to reduce the cost of coins without altering their characteristics. Government officials we spoke with raised concerns about the potential effects of a penny suspension, such as regional penny shortages or other unintended consequences. Specifically, Federal Reserve officials noted that suspending production could create a shortage of pennies if demand is greater than the supply of pennies. These officials explained that even if there are enough pennies to meet overall demand, the distribution of pennies across the country may be uneven and not matched to the location of greater demand. In this case, the Federal Reserve could incur additional costs to transport pennies to balance supply and demand across the country. Federal Reserve officials also said a suspension could potentially be successful if there was a reduction in penny demand and steps taken to mitigate potential disruptions to the penny supply. Mint officials expressed concern about potential unintended consequences of a penny suspension and effects on Mint operations. Specifically, according to Mint officials, suspending penny production may cause an increase in the number of coins returned to circulation because the public may react to a suspension by using its pennies in addition to the other coins in its coin jars. The resulting influx of coins into circulation may be sufficient to satisfy some or all of the demand for new coins for a period of time and cause the Mint to decrease or suspend production of coins. Mint officials said that costs the Mint would incur due to a disruption of coin production operations and loss of income from seigniorage could be as high as $3 billion over 7 to 10 years. These officials also raised concerns about the ability to securely store larger-than-usual quantities of all coins because the existing infrastructure, particularly vault storage, may be insufficient. Mint officials noted that, while other countries have stopped producing coins, suspending penny production may have a similar impact as not producing the penny. When Canada stopped producing its penny, it began to actively take the coins out of circulation, and the public knew the penny would eventually no longer be used. While the proposal to suspend penny production does not remove the penny from circulation or use in commerce, Federal Reserve and Mint officials told us that the results of suspending penny production are uncertain, partly because a suspension has not been tried before. Representatives from 9 of the 10 stakeholder industries said they do not anticipate incurring costs if the penny were suspended; most said they were not concerned about this action because the coins are either not used or minimally used in their industry. Three selected stakeholders said they would be affected by a penny suspension—associations representing armored carriers, banks, and retailers—as well as the company that manufactures the penny blanks for the Mint. They expressed uncertainty about how the suspension would be carried out and effects it might cause, such as penny shortages, and provided the following views and information: Armored Carriers – A penny suspension may not have a significant effect on operations since a suspension would not necessarily reduce the number of coins processed or transported, according to armored carrier representatives we spoke to. However, if penny shortages occurred, the carriers may have to move pennies from one geographic region to another to satisfy variations in demand from their customers, incurring additional transportation costs. Alternatively, suspension of the penny may cause the public to turn in pennies, along with coins of other denominations, which could exceed the secure storage capacity of carriers and coin terminals. Bankers – According to an association representing banks, bankers are unclear if the government would issue any guidance about rounding cash transactions to avoid inconsistent approaches. Because banks have received questions from customers about changes to currency in the past, the association emphasized the need for public education before suspending the penny. Retailers – Retailers have not determined the impact of suspending the penny on their industry, according to a retailer association. However, many retailers sell items priced below $1 as an important part of their business and merchandising strategy, according to these representatives, so it is important for retail businesses to be able to continue to make change down to the penny at the end of cash transactions. Vendor – a representative of the company that supplies the Mint with penny coin blanks told us that a penny suspension would force a decision whether to sell or deactivate the penny blank production equipment during the 10-year suspension. If sold, the vendor may then not have the equipment if the government decided to produce the penny again. None of the representatives from stakeholder industries raised concerns about changes to the nickel as long as the changes to the nickel are seamless. Producing money for use in commerce is an important function of the U.S. government. The Federal Reserve, along with the Treasury’s BEP and the Mint, work together to ensure that there is an adequate supply of U.S. coins and notes for use around the world. In addition to ensuring an adequate supply of these coins and notes, it is also important to ensure that the government is producing these items efficiently. Because our current estimate shows the federal government would likely incur a net loss from replacing the $1 note with a $1 coin, we are no longer recommending that Congress consider replacing the $1 note with the $1 coin. The Treasury cannot alter the metal content of coins unless Congress provides that authority to the Treasury. If Congress were to grant the Treasury the authority to change the metal composition of coins, as the Mint has proposed, then it could use the results of its research to lower the costs of coin production while producing coins that look, feel, and function the same as current coins. Further, the Mint could decrease its production costs without affecting the characteristics of the coins. Without this authority, the Mint cannot provide the best value to the taxpayer and produce coins in the most efficient and cost-effective manner possible. Congress should consider amending the law to provide the Secretary of the Treasury with the authority to alter the metal composition of circulating coins if the new metal compositions reduce the cost of coin production and do not affect the size, weight, appearance, or electromagnetic signature of the coins. (Matter for Consideration 1) We provided a draft of this report to Treasury, including the Mint and BEP, and the Federal Reserve for their review and comment. In comments, reprinted in appendix III, the Mint agreed with our matter for congressional consideration and clarified its position on the potential cost impact of a penny suspension. The Mint’s comments stated that if the penny were suspended, consumers may return large amounts of all coins, not just pennies, which would decrease the need for future coin production. Without demand for coin production, the Mint estimated costs from idle production capacity and loss of seigniorage from coins to be up to $3 billion over 7 to 10 years. The Mint also commented that the effect of suspending penny production could be the same as the effect of stopping penny production. We revised our report to reflect the Mint’s perspective. The Department of the Treasury concurred with comments provided by the Mint. BEP did not have any comments. The Federal Reserve provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Treasury, the Director of the U.S. Mint, the Director of the Bureau of Engraving and Printing, the Chair of the Board of Governors of the Federal Reserve System, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This appendix describes the economic simulations discussed in this report, including the assumptions we used and their sources, as well as the alternative simulations we conducted. To estimate the net effect on the government of replacing the $1 note with a $1 coin, we simulated the benefits and costs to the government of issuing currency—including both notes and coins—under different scenarios and assumptions over a 30-year period. For each set of assumptions we considered, we simulated three scenarios—the status quo scenario, in which the $1 note would continue to be produced, and two replacement scenarios, in which the $1 coin would replace the $1 note. In the gradual replacement scenario, $1 notes are replaced as they become unfit for circulation, while under the active replacement scenario, $1 notes are replaced more quickly. We then compared the net benefit to the government in each replacement scenario to the net benefit under the status quo. As part of our analysis, we also ran alternative simulations with different sets of assumptions, to examine how the assumptions underlying our analysis would affect the estimated net benefit to the government. The various assumptions include the extent to which the public would increase its holdings of cash when coins are used instead of notes, the expected rate of growth in the demand for currency over 30 years, the costs of producing and processing both coins and notes, and the life span of both forms of currency (see table 3). In our replacement scenarios, we assumed that the replacement would be implemented starting in 2018, and during that year the U.S. Mint (Mint) would invest in new equipment to establish its production capability for $1 coins. We also assumed that production of the paper note would stop as soon as $1 coins were introduced. A key assumption in our analysis is the extent to which the public may hold more cash when notes are replaced by coins. Because of differences in how people use notes and coins, the public may need more than one coin for each note than would otherwise have been demanded. For example, people may take coins out of their pockets and store them at the end of each day, rather than retain them in their wallets as they do notes. These factors cause coins to circulate more slowly than notes, and more $1 coins would need to be maintained in circulation to meet the public’s demand for $1 notes. Consistent with simulations in our previous reports, we assumed in our economic simulations that the public would hold more $1 coins, requiring that more than one coin would be needed to replace each note. Therefore our replacement scenarios use a replacement ratio of 1.5 – that is 1.5 $1 coins for each $1 note to be replaced. For our alternate simulations we allow the replacement ratio to vary, to include a case in which no additional currency is demanded when coins are used (i.e., the replacement ratio is 1.0). As part of this sensitivity analysis, we found that a key driver of the estimated net benefit is the extent to which the public would hold more cash when $1 coins are used instead of notes. We altered some assumptions to simulate how the change would affect our estimate of the net benefit or loss to the government. See table 4. We present our analysis to show the effect of changes under both gradual and active replacement, and we show the results both with and without gains from seigniorage. To assess the effect of the public’s holding more or less cash as a result of needing fewer or greater numbers of coins to replace each note in circulation, we conducted separate simulations in which we: decreased the replacement ratio from our current estimate of 1.5 coins per note to 1 coin per note, and increased the replacement ratio from our current estimate of 1.5 coins per note to 2 coins per note. To assess the effect of the Board of Governors of the Federal Reserve System (Federal Reserve) not releasing into circulation the $1 coins it currently holds, we: assumed that the approximately 1.2-billion $1 coins held by the Federal Reserve would not enter circulation and would continue to be held by the Federal Reserve. To assess the effect of changing production costs for notes and coins, we conducted separate simulations in which we: increased the costs of producing notes from our current estimate of 3 cents to 4.9 cents without changing the costs of producing coins; increased the costs of producing coins from our current estimate of 14.6 cents to 17.5 cents without changing the costs of producing notes; and increased the costs of producing both notes and coins from our current estimates of 3 cents to 4.9 cents for notes and 14.6 cents to 17.5 cents for coins. To assess the effect of decreased demand for currency if people switched to electronic means of payment, we conducted separate simulations in which we assumed: demand for currency grows at a slower rate—75 percent of the growth in demand in the replacement scenarios—after fiscal year 2028, and demand for currency grows at a slower rate—50 percent of the growth in demand in the replacement scenarios—after fiscal year 2028. This report: (1) determines the estimated net benefit to the government, if any, of replacing the $1 note with a $1 coin and selected stakeholders’ views on this change, and (2) examines what is known about potential cost savings to the government from suspending production of the penny and changing the metal composition of the nickel coin as well as selected stakeholders’ views on these changes. To estimate the net benefit or loss to the government of replacing the $1 note with a $1 coin, we conducted economic simulations under different scenarios and assumptions over a 30-year period. We simulated a “status quo” scenario and two “replacement” scenarios. In the status quo scenario, notes remain the dominant form of $1 currency. In each replacement scenario, notes are replaced by $1 coins under various assumptions. We then compared each replacement scenario to the status quo scenario with respect to net benefits to the government. As part of our analysis, we also ran alternative simulations with different sets of assumptions, to examine how the assumptions underlying our analysis affect the estimated net benefit to the government. The various assumptions underlying our simulations include the extent to which the public holds more cash when coins are used instead of notes, the cost to produce $1 notes and $1 coins, and the lifespan of notes and coins, among others. Our analyses are projected over 30 years because that period roughly coincides with the life expectancy of the $1 coin. We interviewed relevant officials from the Board of Governors of the Federal Reserve System (Federal Reserve), the Bureau of Engraving and Printing (BEP), and the U.S. Mint (Mint). We also obtained data for our assumptions from these agencies and economic projection data from the Congressional Budget Office. More detailed information on the structure, assumptions, and inputs of our economic simulations are found in appendix I. To determine how the Federal Reserve estimates the life-span of the $1 note (a key input to our economic simulations), we reviewed work papers and analyses from prior work. We interviewed knowledgeable Federal Reserve officials about the methodology for calculating a note’s life-span and reviewed data on a note’s estimated life from calendar years 2005 through 2017. We also observed note-processing operations and equipment at the Federal Reserve’s Cash Technology Office (located in the Federal Reserve Bank of Richmond), reviewed Federal Reserve’s and Treasury Department’s cash-processing policy and procedure manuals, and interviewed knowledgeable officials about technological innovations in Federal Reserve note processing since 1998. We took steps to assess the reliability of data used, such as interviewing knowledgeable agency officials, and determined that the data were sufficiently reliable for the purposes of this report. To determine selected stakeholder views on changes to currency, we identified 91 entities that could potentially be affected by reviewing prior GAO, Mint, and Federal Reserve reports and the results of a literature search. We eliminated some of these entities from further consideration because we could not identify a way to contact them or they did not respond to our efforts to contact them. We sought entities with the broadest representation so we generally eliminated individual companies, with the exception of those that are primary suppliers of raw material for the production of notes or coins. Of the remaining 36 entities, we selected and interviewed 10 organizations representing potentially affected industries, primarily based on the entities’ role with respect to currency and the currency change likely to affect it most. We also selected and interviewed a private company involved in the production of materials used in coins and two organizations that advocate for a switch to a $1 coin and for continued use of the penny, respectively. We categorized each entity’s role with respect to currency as a maker (involved in, or represents those involved in, supply of materials for production of coins or notes); a mover (involved in, or represents those involved in, transporting, processing, or facilitating use of coins or notes); or a user (involved in, or represents those involved in, transactions where coins or notes are exchanged). We also categorized each entity as being most affected by, or most interested in, changes to the $1, nickel, or penny. We used information we collected or had used in prior work about these stakeholders and also used professional judgement and logic to determine in which role category they belonged. In some cases, we assigned an entity to more than one category. In addition to categorizing stakeholders, when making our selection, we also considered the extent an entity’s area of representation overlapped with another to avoid duplication. If a selected entity did not respond to our request for an interview, we sought to replace that entity with a similar one, if available. Since our selection is comprised of a non-representative sample, the results are not generalizable to all stakeholders. The stakeholders we selected are: American Bankers Association, aba.com Americans for Common Cents, pennies.org Association of Gaming Equipment Manufacturers, agem.org Coin Laundry Association, coinlaundry.org Dollar Coin Alliance, dollarcoinalliance.org International Bridge, Tunnel and Turnpike Association, ibtta.org International Parking & Mobility Institute, formerly the International Parking Institute, parking-mobility.org Jarden Zinc Products, jardenzinc.com National Armored Car Association, nationalarmoredcar.org National Automatic Merchandising Association, namanow.org National Bulk Vendors Association, nbva.org Retail Industry Leaders Association, rila.org We also reviewed information on public perceptions and opinions about the use of a $1 coin from prior GAO work and publicly available information from an organization that advocates for a transition to a $1 coin. To examine what is known about potential cost savings to the government from suspending production of the penny coin and from changing the metal composition of the nickel coin, we analyzed penny and nickel production cost data from the Mint for fiscal years 2003 through 2017 to include a range of the number of coins produced and cost changes from metal price fluctuations and reviewed Mint studies on potential alternative metals and on coin production cost savings that could result from changing coin metal composition for these coins. We reviewed and analyzed the Mint’s preliminary plan if Congress were to authorize suspending production of the penny. We took steps to assess the reliability of the Mint data we used, such as reviewing relevant documentation, and determined that the data were sufficiently reliable for the purposes of this report. We also interviewed Mint and Federal Reserve officials, and the same set of selected stakeholders noted above. To understand the rationale and steps Canada implemented for eliminating the Canadian penny, we reviewed documents from the Canadian Senate, Department of Finance, and the Royal Canadian Mint. To understand the results of the elimination of the Canadian penny, we interviewed an official from the Royal Canadian Mint. We also conducted a literature search of relevant English language articles published from 2011 to May 2018 to provide information on the rationale and potential benefit to governments of making changes to coins and notes, along with information about the experiences of other English-speaking countries that have made such changes. We conducted this performance audit from December 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, John W. Shumann (Assistant Director); Travis Thomson (Analyst-in-Charge); Amy Abramowitz; Lindsay Bach; Dave Hooper; Delwyn Jones; Malika Rice; Oliver Richard; Ardith Spence; and Elizabeth Wood made key contributions to this report.", "summary": "The U.S. spent about $1.3 billion in 2017 to produce, process, and circulate coins and paper notes for use in the economy. Since 2006, both the penny and nickel have cost more to make than their face value. Other countries have replaced notes with coins of the same value to reduce costs. Since 1990, GAO had estimated replacing the $1 note with a $1 coin would provide a benefit to the federal government. GAO was asked to examine the potential cost savings to the government from making changes to currency. This report (1) estimates the net benefit to the government, if any, of replacing the $1 note with a $1 coin and selected stakeholders' views on this change; and (2) examines what is known about potential cost savings from suspending penny production and changing the metal composition of the nickel, and selected stakeholders' views on these changes. GAO conducted economic simulations of continued use of $1 notes and replacing notes with $1 coins, examined cost data from the U.S. Mint, and interviewed officials from the Federal Reserve, U.S. Mint and Bureau of Engraving and Printing as well as 10 selected stakeholders representing industries that could potentially be affected by currency changes. GAO's analysis found that replacing the $1 note with a $1 coin would likely result in a net loss to the government over 30 years. GAO found the government would incur a loss of about $611 million if notes were actively replaced and about $2.6 billion if $1 notes were replaced gradually (see figure). These simulations represent the first time GAO has found that replacing the $1 note with a $1 coin would result in a net loss to the government rather than a net benefit. GAO's estimates are based on current data and economic projections, which have changed over time. For example, the lifespan of the $1 note has more than doubled since a 2011 GAO analysis, from 3.3 years to 7.9 years, largely due to changes in note processing technology. Stakeholders generally identified few benefits from replacing $1 notes with $1 coins. Seven of 10 stakeholders GAO met with said that replacing the $1 note with a $1 coin would result in additional costs. For example, armored carriers told GAO that their transportation costs would increase because coins weigh more than notes. The U.S. Mint estimates that it could save approximately $250 million over 10 years by suspending penny production and between $2 million and $9 million per year by changing the metal composition of the nickel. It also estimates that it could save about $74 million over 10 years by changing the metal composition of the dime and quarter. However, Federal Reserve officials and some stakeholders expressed concern about temporarily suspending the penny due to the potential for external effects, such as penny shortages. Stakeholders were unconcerned about changes to the nickel as long as the changes would not affect how the coin functioned, for example, in vending machines. Since Congress specifies in law which coins are made and their metal composition, the Mint has proposed legislation to enable the Secretary of the Treasury to change the metal content of coins as long as the weight or machine acceptance of the coins is unaffected. Without such authority, the Mint might not be producing coins as cost-effectively as possible. Congress should consider taking steps to authorize the Secretary of the Treasury to adjust the metal content of circulating coins.", "document_type": "gao"}
{"report": "The Director of National Intelligence serves as head of the IC and acts as the principal adviser to the President and National Security Council on intelligence matters related to national security. The IC is comprised of 17 executive branch agencies and organizations, generally referred to as IC elements. These IC elements include two independent agencies, eight elements within the Department of Defense, and seven elements across five other executive departments. Table 1 provides a list of the 17 IC elements. In its first National Intelligence Strategy, issued in 2005, ODNI highlighted the importance of a diverse talent pool to address the complex challenges the IC faced. In its most recent strategy released in 2019, ODNI reaffirmed and emphasized the IC’s commitment to developing and retaining a diverse workforce to address enduring and emerging mission requirements. The 2019 National Intelligence Strategy defines diversity as a collection of individual attributes that include, but are not limited to national origin, language, race, color, mental or physical disability, ethnicity, sex, age, religion, sexual orientation, gender identity or expression, socioeconomic status, veteran status, and family structure. The Intelligence Authorization Act for Fiscal Year 2004 directed the Director of Central Intelligence to develop a pilot project to test and evaluate alternative innovative methods to promote equality of employment opportunities in the IC for women, minorities, and individuals with diverse ethnic and cultural backgrounds, skills, language proficiency, and expertise. The first pilot was initiated at Trinity Washington University in Washington, D.C., with a 1 year contract totaling $250,000. The college developed and designed curricular components to align with IC mission skills sets and competencies and competitively selected students to participate in the college’s IC CAE Scholars Program. In the first year, the program sponsored nine students who were selected as IC CAE scholars. After the initial pilot year at Trinity Washington University, the pilot program was expanded to three additional colleges—Tennessee State University in Nashville, Tennessee; Florida International University in Miami, Florida; and Clark Atlanta University in Atlanta, Georgia. In 2005 ODNI, on behalf of the IC, established the IC CAE program. ODNI reported that by 2007, 65 scholars participated in the program from these four CAE colleges. By 2008, ODNI had expanded the pilot to six additional colleges. Overall, the 10 participating colleges increased the student population to 338 IC CAE scholars. During the 2008 to 2009 academic year, ODNI established a continuity strategy with the initial 10 IC CAE pilot colleges and the program continued to expand its academic outreach to additional colleges. In 2009, a total of 17 colleges were participating in the program and these colleges had arrangements with academic consortia that increased the total outreach to 31 colleges. During ODNI’s management of the program from 2005 through 2011, ODNI established general goals and oversaw the program’s implementation by defining and collecting performance measures on a range of IC CAE activities and working with a contractor to summarize this information in annual reports. We describe ODNI’s management of the program in more detail in appendix I. The Intelligence Authorization Act for Fiscal Year 2010 codified the IC CAE program to authorize the Director of National Intelligence to carry out grant programs to enhance the recruitment and retention of an ethnically and culturally diverse IC workforce with capabilities critical to the national security interests of the United States. In 2010, ODNI launched an Intelligence Community Efficiency Studies Initiative that included an examination of the size, structure, and functions of the ODNI. One recommendation was to consolidate and streamline the education and training programs in the IC by transferring the functions and responsibilities of the IC CAE program from ODNI to DIA. DIA began managing the program on October 1, 2011. The memorandum of understanding between ODNI and DIA implementing the decision of the transfer established that while DIA would manage the IC CAE program, ODNI would continue to provide periodic strategic guidance and regular budgetary oversight for program. Figure 1 shows various IC CAE program milestones, such as grant announcements and program transition dates, among other details. According to ODNI and DIA officials, program management and oversight of the IC CAE program is currently transitioning from DIA back to ODNI, following a DIA roles and mission review in 2018. According to ODNI and DIA officials, officials are working to complete the transition in fiscal year 2020 to enable ODNI to assume responsibility for the program. According to ODNI officials, as of April 2019, the transition plans were still in progress and ODNI was still in the planning stage of the transition. For example, officials noted they were drafting an implementation plan for the transition as well as a transfer memorandum to document the transfer. According to DIA officials, during this process, ODNI and DIA officials were also holding weekly coordination meetings and sharing program documents, such as college reports collected by DIA, and program guidance. ODNI officials also stated that in February 2019, they hired a contractor to conduct a study of the program prior to the final transition date. According to ODNI officials, the study, along with their interactions with DIA, will help ODNI determine how to manage the program, identify any challenges or successes of the program, and consolidate the data collected on the program to date. Officials expect the study to be completed by October 2019. The IC CAE Senior Advisory Board consists of representatives of the IC elements and key organizations that may include representatives from the National Intelligence University, a U.S. Combatant Command (rotating basis), and the Office of the Undersecretary of Defense for Intelligence. The board, which meets quarterly, was created to provide policy and guidance for the IC CAE program and ensure that participating IC elements are included in discussions of policy matters. As outlined in the board’s official charter and business rules, board members are responsible for attending board meetings, voting on issues before the board, evaluating colleges for grant funding, acting as points of contact for the program, and promoting the program as leverage to affect future IC missions. According to DIA officials, the board advises the IC CAE program manager on standards for the IC CAE program relating to college selection, strategies to foster collaboration, and other issues as needed. The IC CAE program awards grants to colleges on a competitive basis. IC CAE grants help colleges establish new intelligence-related programs and support existing programs at selected colleges. The grants can be issued for up to 5 years. From fiscal years 2004 through 2018, a total of 29 colleges have received 46 IC CAE grants. Of these 29 colleges, 13 have formed a consortium with one or more colleges to enhance collaboration with resources from other colleges in the same geographic area. The IC considers colleges with active grants as active IC CAE colleges, and those colleges that sustain the program after grant funding ends are called legacy colleges. Figure 2 shows the location of IC CAE colleges and which colleges led an academic consortium. See appendix II for additional details on the years that grants were awarded, grant funding amount, and a list of consortium colleges. Since 2011, DIA has issued grants for the IC CAE program through a process initiated by an announcement published online by the DIA grants officer. Grant announcements vary by year, but generally include guidelines for colleges to follow in completing their grant proposal. For example, the 2014 grant announcement listed eight program components a college’s proposal would be evaluated on, to include study abroad opportunities and annual colloquium or speaker series on intelligence and national security issues, along with other requirements such as cost program management and sustainment plans. Following submission, a grants officer reviews colleges’ grant proposals for technical and financial sufficiency. The IC CAE program office then reviews grant proposals for program sufficiency. From there, the IC CAE Senior Advisory Board’s Source Selection Board reviews applications deemed sufficient and makes a recommendation on which should be funded and at what funding level. The DIA CAE program office then forwards the selected proposals to a grant officer who notifies the college of the award. The grant announcements we reviewed may add specific program components as an area of focus for a specific year. For example, the 2019 grant announcement added a program component that required colleges submitting a proposal for a grant to offer courses or programs in three or more listed science, technology, engineering, and mathematics topics of interest to the IC. Examples of some other program components included in grant announcements since 2014 include the following: IC Curriculum. A key objective of the program is to strengthen academic programs in intelligence or national security in minority- serving, historically rural and under-resourced population colleges. Specifically, colleges shall explain how they plan to creatively expand, upgrade, enrich, or integrate undergraduate and graduate course offerings to better prepare students to perform work in intelligence or national security. Foreign Language. Colleges should demonstrate a capability to offer language study programs or courses in one or more specified languages of interest to the IC. Facilitate Student Participation in Academic Programs. IC CAE students shall be involved in the program and aware of the numerous benefits. Colleges are required to facilitate student participation in on- campus programs and activities such as workshops, seminars, and other off-campus activities such as national security or intelligence conferences, seminars, or workshops. Annual Colloquium. IC CAE colleges are required to hold annual colloquium or speaker series on intelligence or national security issues. These events should invite rural and under-resourced regional colleges and universities, government speakers, and industry partners with a primary goal of maximizing relationships and outreach. The colloquium should be at least 1 day in length, or a speaker series may include shorter presentations scheduled over weeks or months, which equates in the number of hours to a daylong colloquium. Program Management and Sustainment Plans. IC CAE colleges are required to have both program management and sustainment plans. The program management plan must detail the responsibilities of personnel to attain explicitly stated, measureable, and achievable program objectives. The sustainment plan must detail what the college will do during the grant period to build sustainability of the IC CAE program at that institution after the funding expires. The IC CAE program is especially interested in colleges with diverse populations of talent and in geographic diversity—specifically, Historically Black Colleges and Universities, Hispanic-Serving Institutions, Tribal Colleges and Universities, Asian American and Pacific Islander-Serving Institutions, and majority serving institutions with significant populations of minorities or women. The IC CAE program is also interested in majority serving institutions with significant populations of minorities and women that possess credentials in disciplines and specializations that meet IC core mission requirements. Figure 3 shows the minority designation of the 29 colleges receiving grants and figure 4 shows the minority designation of the 43 consortium colleges. See appendix II for a list of schools and their minority designations. As part of the IC CAE program, DIA also administers other programs that provide intelligence-related learning experiences to IC CAE students and to increase advanced capabilities in national defense. For example: IC CAE Professional Development Summit. These annual summits allow the IC to interact with the principal investigators—the individuals responsible for the IC CAE program at their respective colleges—to provide them with relevant and up-to-date information to support the creation and teaching of IC-centric curricula. According to DIA, the summit is intended to foster collaboration with the IC and college representatives by providing DIA with a platform to meet the needs of the IC. According to DIA, IC CAE Senior Advisory Board members are an integral part of the summit and provide context and perspective from the agencies they represent. National Security and Analysis Intelligence Summer Seminar. This 2-week seminar is designed to provide IC CAE students with knowledge about the intelligence career field in general, and analytic tradecraft in particular. The seminar is intended to provide students from across the IC CAE colleges an opportunity to engage directly with intelligence professionals in both seminar learning and scenario- based simulation training, focusing on threats to the U.S. homeland by extremist terrorists. According to DIA officials, the seminar is only open to a limited number of IC CAE students from active and legacy colleges. For example, two sessions were held during 2017 and a total of 80 students were competitively selected by their respective colleges to attend. According to ODNI officials, the summer seminar also holds a career fair and provides mentoring opportunities for the participating students so that those interested in an IC career have an opportunity to interact with recruiters. IC CAE Summer Internship. In the summer of 2017, the IC CAE program held its first IC CAE summer internship program. According to DIA officials, rather than establish a new IC CAE internship program, DIA leveraged the IC elements’ existing internship programs and tracked IC CAE student participation in these programs. The IC CAE internship offers IC CAE students additional opportunities, such as an opening and closing ceremony for the internship, an IC career fair at the National Security and Analysis Intelligence Summer Seminar event, and IC mentors upon request. DIA identified a total of 141 IC interns from colleges that had an IC CAE program in 2017 and 2018. However, according to ODNI officials, not all IC interns identified participated in their school’s IC CAE program. The internship opportunities among the IC elements vary. For example, according to FBI officials, their internship program is a primary pipeline for entry-level positions and, in 2017, they had 1,200 interns with 300 hired into entry-level positions. According to DIA data, the FBI identified 31 IC CAE scholars in its 2017 internship program and 21 scholars in 2018. According to Department of State’s Bureau of Intelligence and Research officials, their office has approximately 15 to 20 summer interns each year. According to Department of State officials, two of their interns were IC CAE scholars since the program began in 2017. While DIA has continued to implement the IC CAE program by issuing grants to colleges, DIA has not sufficiently planned or overseen the program since the transition from ODNI in 2011. Specifically, we found that DIA did not fully implement five of the six key practices of sound planning that we have identified in our prior work. While DIA continued the program’s mission to increase the pool of diverse applicants for the IC, it lacked results-oriented goals, an overall strategy for the program, an evaluation of external factors, performance measures, and a plan to assess the program’s performance in order to determine the appropriateness of the goals and effectiveness of implemented strategies. Our assessment of the extent to which DIA incorporated these key practices of sound strategic management planning into the IC CAE program is reflected in table 2. DIA annual reports for the IC CAE program and IC CAE grant announcements emphasize that the overall mission of the program is to increase the pool of diverse applicants for the IC. DIA’s annual reports describe the program’s mission as developing national security and intelligence education programs in order to increase the pool of culturally, geographically, and ethnically diverse, multidisciplinary job applicants who possess highly desired skills and competencies in areas of critical need to the IC. This mission statement is also contained in IC CAE grant funding opportunity announcements for 2014, 2017, 2018, and 2019, which also refer to broader IC human capital and diversity guidance. For example, one goal from the IC’s Equal Employment Opportunity and Diversity Enterprise Strategy (2015-2020) is to recruit from groups with lower than expected participation rates and diverse candidates who will meet the IC’s current and future mission requirements. Since 2011, DIA has not established results-oriented goals for the IC CAE program or an overall strategy that details the agency resources and processes that are required to achieve the program’s mission. First, DIA failed to document specific policy, programmatic, or management goals for the IC CAE program. DIA developed a business plan for the program in 2011; however, this plan describes short-term goals for program management, outreach, and education and most of these goals were intended to be complete by mid-2012. DIA’s documentation does not indicate whether these goals were achieved or whether DIA continued to use the goals to guide the program after 2012. Current DIA internal guidance states that the IC CAE program office carries out the program’s mission by providing grants to colleges to support the establishment of intelligence-centric curricula. However, this guidance fails to provide results-oriented goals that are defined in measurable terms to guide the program. For example: DIA has not described the number of potential IC employees it expects to be able to educate or make aware of IC careers by supporting intelligence programs at IC CAE colleges. This could include specific goals for targeting underrepresented populations within the IC, such as women and minorities. According to several IC element officials, IC elements use the percentage of women and minorities in the U.S. civilian labor force as a target for their own diversity recruitment efforts. However, DIA has not developed any results-oriented goals that include specific targets or milestones for recruiting potential IC employees who have participated in the IC CAE program. In addition, DIA has not developed specific goals for the program that identify how to prioritize among program requirements contained in IC CAE grant announcements . Specifically, it is not clear from IC CAE program documentation how gender and ethnic diversity is prioritized relative to other IC needs, such as the IC’s long-standing need for technical and language skills. For example, IC CAE grant announcements state a general goal of increasing the pool of qualified women and racial and ethnic minorities to the IC. At the same time, IC CAE grants have supported training in science, technology, engineering, and math, and critical languages, but DIA has not established specific targets or milestones that would allow it to track the program’s development of a diverse pool of applicants with the skills that the IC requires. Second, while DIA has developed some plans and continues to award grants for the IC CAE program, we found that DIA has not documented an overall strategy that details the agency resources and processes required to achieve the program’s mission. In 2016, DIA officials stated they began developing a document outlining the general structure of the IC CAE program, but as of May 2019, the document has not been issued. DIA has also documented its standard operating procedures for monitoring colleges’ implementation of grants in part to ensure that all programmatic goals are met, but it is not a strategic document that describes processes for achieving the program’s mission or goals. Further, DIA continues to award IC CAE grants to colleges based on program components or criteria that have changed over time, but these changes are not clearly linked to an overall program strategy. For example, in 2014, DIA added the diversity of a college’s student population as one of the criteria it used to select grant proposals. Colleges with a minority-serving designation or with a student population that is more than 75 percent ethnically and culturally diverse are given an excellent rating, while colleges with a student population that is less than 25 percent diverse are given a poor rating. In 2017, DIA then added criteria requiring colleges to be part of a consortium in a manner that promotes diversity. These two diversity criteria have been given more weight than all other criteria since 2017, while previous announcements gave greater weight to the development of national security curricula. This change in approach may align with the program’s overall mission to increase diversity in the IC, but DIA has not outlined an overall strategy that explains how such changes to the grant selection criteria would achieve a results-oriented goal like increasing the number of minority applicants to the IC. Two interconnected sound planning practices are to establish results- oriented goals and strategies to achieve those goals. These goals should be documented in measurable terms that are focused on results so that the agency can determine how it will achieve its mission. Once goals are established, strategies explain how these goals would be achieved. Since assuming responsibility for the program in 2011, DIA officials stated that their focus for managing the IC CAE program has been tactical, focusing on tasks like awarding, executing, and monitoring grants to IC CAE colleges, rather than strategic planning. In addition, DIA officials highlighted staff turnover as a challenge to managing the program and stated DIA has had five IC CAE program directors in its 8 years of program management. DIA officials stated that DIA has received little guidance about the goals of the IC CAE program from ODNI, and they instead rely on the IC CAE Senior Advisory Board to define goals and strategies that reflect the needs of IC elements. DIA officials stated that their only source of guidance from ODNI for the IC CAE program was the 2011 memorandum of understanding between DIA and ODNI, which DIA officials characterized as being high level and lacking specificity. DIA officials also said that they do not have the authority to create a strategic recruitment plan or set recruiting targets for the IC. The board only meets quarterly to advise the IC CAE program office on standards and strategies and board members occasionally review grant proposals. The IC CAE program managers are responsible for the program, and therefore, defining and documenting its goals and strategies. As the IC CAE program transitions back to ODNI, ODNI will not be able to determine whether the program is meeting the diversity goals of the 2019 National Intelligence Strategy without results-oriented goals for the program and a documented strategy showing how those goals are to be achieved. DIA has identified external factors that could affect the IC CAE program’s success, such as program branding and the ability of colleges to sustain the program after the grant period ends, but has not developed a process to fully evaluate them. One example of an external factor that could affect the IC CAE program’s success is the fact that not all students are aware of their participation in an IC CAE program. Colleges participating in the IC CAE program have not always featured participation in the program prominently, based on our analysis of selected websites, which are often managed by an academic department or institute. This can limit the visibility of the program and the IC’s support of it for both current and potential students. Since at least 2014, DIA has required colleges to demonstrate how they plan to promote their program as an IC CAE program to ensure that students, faculty, and administrators are aware of it. Colleges are also required to feature up-to-date program information on the college’s website. However, in November 2018, a DIA official noted that some colleges continue to use the IC CAE brand without oversight and accountability to provide intelligence-related courses. According to officials from selected IC CAE colleges and IC elements, students graduating from these programs are not always aware that they have participated in an IC CAE program. One college official stated that the certificates or degrees do not necessarily indicate that the student graduated from an IC CAE program. Another college official stated the college needs to directly inform students who apply to the program that they are participating in an IC CAE program. NGA and NSA officials stated that some employees at their agencies first became aware there was an IC CAE program at their college after being informed directly by their respective agency. While DIA requires that colleges develop marketing plans, it does not have a process to evaluate external factors such as the long-term effect of colleges’ efforts to advertise their programs’ connection with the IC CAE program. Without adequately advertising IC CAE programs, IC CAE colleges may not be able to recruit a strong pool of qualified students with the skills that the IC requires. Another example of external factors that could affect program success is the ability of colleges to sustain their IC CAE program. The intent of the IC CAE program has been to enable colleges to continue the program beyond the end of the grant period and maintain a continuous talent pool for the IC. However, DIA has not fully evaluated the challenges colleges may encounter if they are not able to secure continuous funding for the IC CAE program. When DIA awards grants, colleges are awarded a base year of funding and renewable up to 4 additional option years. It may take time for a college to develop intelligence-related courses and have students graduate from the IC CAE program. Colleges then need to apply for another grant in order to continue to receive federal funding following expiration of any additional option years. Since 2011, colleges have been required to demonstrate a plan to sustain their programs after the initial grant period ends. However, according to some IC CAE college officials, it is nonetheless difficult to continue the program and secure external funding once the grant is over. Some college officials have also said that the loss of grant funding can result in colleges discontinuing key aspects of the IC CAE program and can limit consortium college participation in activities. We have also observed that some colleges may have suspended their programs entirely. Specifically: Colleges may be able to sustain some, but not all components of their program once grant funding ends. For example, one college has sustained an IC CAE program since 2005 even though the college did not receive grant funding from 2008 through 2012. According to college officials, loss of grant funding resulted in the college suspending professional development activities. The program received additional IC CAE grants in 2012 and in 2017 and college officials stated they hold professional development workshops and one-on-one mentoring sessions between students and representatives from IC elements. Without grant funding, consortium colleges may not have funding for student travel to IC CAE events. Consortium colleges face a specific challenge since many of the IC CAE events are hosted by the lead IC CAE college. We spoke with faculty at two consortium colleges who said that grant funding from the program helps reduce the cost of their students’ travel to off-campus IC CAE events, such as annual colloquiums at the lead college that are attended by subject matter experts from IC elements. The distance students may need to travel can be especially challenging for colleges that are not located near the lead college, including one community college that is 400 miles away from the lead consortium college according to an IC CAE college official. DIA identifies some programs as legacy colleges, but some colleges have not updated their IC CAE program websites. For example, we reviewed the IC CAE program websites for two colleges that had received a grant from DIA after 2012, but the colleges had stopped updating their websites in 2014 and 2016. DIA has identified sustainment of the IC CAE program following termination of grant funding at colleges as a significant challenge. At a recent meeting with the IC CAE Program’s Senior Advisory Board, the head of DIA’s program office stated that the sustainment of IC CAE programs after federal funding ends tended to be a systematic failure, especially for many smaller colleges that may lack the resources of larger colleges, and that there have been no consequences for failure. While DIA acknowledges this problem, it does not have a process to systematically evaluate this issue or consider alternative approaches for colleges that may need additional support to maintain relevant curricula or professional development activities. For example, DIA has not evaluated whether some colleges’ difficulty with sustaining their IC CAE program may invalidate underlying assumptions about how the program is structured, including whether awarding grants to colleges to develop and maintain an intelligence-focused curriculum is the most effective means of establishing long-term relationships with those colleges and fostering a diverse talent pool for the IC. A key practice of sound planning is to fully evaluate key factors external to the organization that are beyond its control. IC CAE colleges decide how to brand the program as well as how to allocate resources in order to sustain their IC CAE program. These decisions could significantly affect the achievement of the IC CAE program’s mission and goals. Both ODNI and DIA officials are aware of some external factors that could affect the success of the IC CAE program, such as branding and program sustainment. As of March 2019, ODNI officials have stated that they are developing plans to address branding and sustainment as the program transitions to ODNI. DIA drafted a plan for post-grant requirements for colleges in order to maintain their IC CAE designation, though this draft plan does not address the sustainment challenges that may make it difficult for those colleges to follow these additional requirements. However, DIA internal guidance and the most recent Senior Advisory Board charter do not outline a process to identify and continuously evaluate external factors that could affect program performance. As the new program manager, ODNI may be unable to assess whether factors like program branding or sustainment might affect the IC CAE program’s implementation and potential for success without a process in place to evaluate the effect of these and other potential external factors. DIA lacks comprehensive performance measures for the IC CAE program that would allow DIA to measure program success. Specifically, DIA has not (1) clearly and consistently defined performance measures to be reported and collected, (2) collected on or reported complete information on the program, and (3) determined whether data collected may be incomplete or unreliable due to reporting challenges. DIA has not clearly and consistently defined the performance measures that need to be reported by the colleges in order to determine the IC CAE program’s success. DIA required colleges to provide reports on significant accomplishments related to the objectives in their grant proposals. However, we reviewed final grant reports that colleges submitted to DIA from 2014 to 2018 that revealed differences in how colleges reported measures. For example: Two colleges reported that a total of 664 students received an IC CAE certificate, 99 completed an internship, and 128 received a conditional offer of employment between 2012 and 2017. However, the report did not indicate whether these offers of employment were from IC elements or the number actually hired. A legacy college reported that 49 students received a conditional offer of employment or were hired by an IC element, but it did not indicate the total number of program participants. The final report from a legacy college that had an IC CAE program from 2013 to 2015 reported the total number of internships, but it did not report conditional offers of employment or total program participation. In 2017, DIA revised the reporting template for colleges to require progress on the goals and objectives in the approved grant proposal. However, the information colleges provided varies because DIA’s performance measures are not clearly stated so that colleges can report them consistently, and they are not scoped to evaluate specific program outcomes. For example, IC CAE programs are required to report their progress in developing critical language studies, but there is no minimum requirement on the type of information that a college should report in the updated template. Comparing the reporting template for two colleges from 2018, one college’s narrative provided a high level overview of its foreign language options at the college and reported that IC CAE scholars will be encouraged to participate in the language courses, whereas another college’s narrative provided details on the number of students participating in the foreign language program and details on stipends provided to students who studied abroad. DIA’s updated reporting template also required IC CAE colleges to report the aggregated totals of IC CAE participants, conditional offers of employment, internships, and hires into the IC. However, some colleges track different types of information for these metrics. For example, the way colleges count student participants in the IC CAE program varies. Some colleges only track students enrolled in the IC CAE certificate or degree program, while other colleges report much larger totals of participants, including those who are not enrolled in an IC CAE certificate or degree program but may participate in some IC CAE events. In addition, DIA’s updated reporting template did not clearly describe the hiring data that colleges are required to report. For example, colleges are required to report the total number of conditional offers that IC CAE scholars receive, but it does not specify whether this number is for all employers or just IC elements. Furthermore, it is not clear whether students that received a conditional job offer in one semester are being reported again as a hire in the following semester. Without clearly defined performance measures, decision makers may not be able to clearly identify the accomplishments of the program among the various participating colleges. DIA is responsible for reporting on the IC CAE program’s performance to ODNI, but DIA has not collected complete performance measures that cover the entire program and has not reported a complete summary of the performance measures it has collected. Since 2011, the DIA program office has collected some information from IC CAE colleges in order to monitor compliance with the colleges’ grant proposals. This information was reported by IC CAE colleges in their interim and final reports that include narrative descriptions of IC CAE program activities and descriptive data about program participants. However, DIA has not collected complete information that captured relevant performance measures for the IC CAE program. For example, between 2011 and 2016, DIA officials stated colleges provided DIA a spreadsheet of information on IC CAE program activities, including descriptions of IC CAE courses and events, study abroad program participation, IC element interaction, and information about individual IC CAE scholars. However, the data provided by the colleges varied. For example, based on a review of spreadsheets that DIA provided from the fall of 2014, some colleges provided details on IC CAE sponsored events, IC element interaction, and student employment, while other colleges did not provide any information in these areas. We also found that colleges summarized this information in their final grant. DIA’s annual reports to ODNI from 2012 to 2017 reported little of the information that DIA collected over this time period. The annual reports described financial data and provided some description of select college activities, but they did not summarize information related to any of the program’s core requirements such as curriculum development, critical language study, or professional development. For example, DIA has not collected or reported data on the number of IC CAE scholars who have studied a critical language from 2012 to 2017. The reports also did not include the total number of IC internships, conditional job offers, or hires after 2012. Moreover, college officials stated they do not report on performance measures after the grant period ends, which may limit DIA’s ability to provide comprehensive data for both active grant colleges and legacy colleges each year. DIA officials stated that legacy IC CAE colleges that have sustained the program but no longer receive a federal grant are not obligated to provide reports to DIA. According to DIA officials, DIA is currently developing a plan that would require colleges to report information in order to maintain their IC CAE designation after the grant period ends. For example, a college official from a legacy program that first received a grant in 2006 stated that the college no longer shares information with DIA because DIA had not requested it do so after the grant ended. The official noted that the college is no longer receiving support to facilitate IC recruitment of its students. DIA officials stated they have relied on colleges rather than the IC elements themselves to report data on IC CAE scholars. DIA informs colleges through its reporting template that data on internships, conditional job offers, and hires into the IC are definitive evidence of the success and sustainability of a college’s IC CAE program. However, due to challenges with collecting these data, the information being provided to DIA by the colleges may be incomplete and unreliable. While DIA has not reported on the total number of IC CAE scholars that have been hired from 2012 to 2017, it has collected some information from IC CAE colleges. For example, three colleges from our sample reported that a total of 23 IC CAE scholars were hired by the IC between the beginning of the fall semester of 2017 and the end of the fall semester in 2018. However, according to officials at these colleges, it is difficult to provide complete data on students’ employment as they no longer have direct contact with students after they graduate and some IC elements discourage applicants from discussing their employment offers with others. As a result, the information the colleges report to DIA may be incomplete because they are not able to track all the students who have graduated from the IC CAE program. ODNI also reported similar challenges when it managed the program from 2005 through 2011. ODNI reported a total of 61 IC CAE scholars were hired into the IC between 2005 and 2011 based on IC CAE college data, but noted that the hiring data from IC elements was higher than the total reported by colleges. Further, IC elements have noted that there are security risks associated with tracking the number of IC CAE scholars that receive a conditional offer of employment or have been hired into the IC. At the February 2019 IC CAE professional development summit, for example, Senior Advisory Board members from the CIA and the FBI advised IC CAE colleges that storing or sharing information about potential IC applicants on unsecured college systems is a security risk. Some IC element officials have suggested that the best way to track applicants would be to obtain a list of IC CAE scholars from the colleges and match the names against IC element applicants. However, according to officials, the IC elements would need an individual’s full legal name and college, and some IC CAE college officials raised privacy concerns with sharing student information. An IC CAE college official stated that even during the grant period, the college only provided DIA aggregated totals on student data because of privacy concerns. DIA and ODNI have collected some data on the number of applicants from IC CAE colleges and new hires from the IC elements, but they have only recently done so in a systematic manner. Officials from DIA’s IC CAE program office said they cannot force IC elements to report employment information and that the burden is on the IC elements to track and report that data. According to ODNI officials, in response to a provision in the Intelligence Authorization Act for Fiscal Year 2017, ODNI sent out a request to IC elements for data on hiring and demographic information that included questions about the number of IC CAE graduates hired by the IC. As of April 2019, officials stated that they have collected hiring and demographic information from six of the largest IC elements that includes data about the number of IC CAE graduates hired by the IC. The officials said they expect this to be a large enough sample to report in June 2019. However, according to ODNI officials, ODNI has not yet determined how it will define performance measures for the IC CAE program or how it will continue to collect and report these performance measures. A key practice of sound planning requires the development of a set of performance measures that will be applied to gauge progress toward attainment of the plan’s goals. We have also established that key attributes of successful performance measures, which include measures that cover core program activities, are that they are clearly defined and consistent and can be reliably produced. Furthermore, Standards for Internal Control in the Federal Government state that management should use relevant data from reliable sources; process this data into high-quality information that is complete, accurate, and valid; and communicate high-quality information to all levels of the department. Comprehensive performance measures would allow DIA to gauge the success of the IC CAE program in developing a pool of diverse talent with skills needed in the IC, but DIA has not defined performance measures in program guidance and documentation. In its 2012 annual report, DIA stated that it intended to redesign ODNI’s data collection tool in order to simplify reporting. However, DIA did not report data collected with this tool and stopped collecting these data altogether in 2016 after informing IC CAE colleges that the collection effort required a lengthy approval process from the Office of Management and Budget. DIA officials continued to require colleges to report performance measures after 2016 through a reporting template. In April 2019 DIA officials stated that they intended to make improvements to this template given that the way colleges have tracked student participation has varied. However, DIA did not clearly and consistently define performance measures for all aspects of the program, process them via a data system or spreadsheet, or report them to ODNI. As the new IC CAE program manager, ODNI will not be able to gauge the success of the IC CAE program in achieving its mission without defining, collecting, and reporting on comprehensive performance measures. Since 2012, DIA has not conducted a comprehensive assessment of the IC CAE program. According to a 2013 amendment to the memorandum of understanding with ODNI, DIA was responsible for providing ODNI with an annual review of the program’s performance and including possible outcomes, such as specific benefits to the IC. ODNI was responsible for evaluating this information to ensure the appropriate and efficient expenditure of IC resources and performance improvement. However, DIA’s annual reports to ODNI from 2012 to 2017 did not comprehensively assess the program’s performance or the extent to which the program had achieved its mission. These reports only provide a few details about IC CAE program activities and summarize grant expenditures. For example, the 2016 annual report for the IC CAE program provided information on the number of grants awarded, a list of IC CAE colleges participating in the program, funding and execution data, and a sample of IC CAE program events from three colleges. However, the report did not provide complete details on the status of the program at each IC CAE college, such as a summary of the performance metrics it had collected from all of the colleges with an active grant. DIA officials said that they only included the information in annual reports that ODNI requested in the memorandum of understanding and lacked resources to provide a comprehensive assessment. However, the memorandum of understanding requires DIA to provide an annual review of the IC CAE program’s performance to possibly include outcomes such as the number of students who completed IC CAE coursework and specific benefits to the IC. As ODNI officials work with DIA to transition the IC CAE program back to ODNI, ODNI officials began working with the MITRE Corporation in February 2019 to evaluate the IC CAE program. ODNI officials said they will rely on MITRE’s findings and their own interactions with IC CAE colleges to determine how to manage the program. Officials stated they expect the evaluation to be complete by October 2019. However, ODNI has not yet developed a plan to conduct continuous and comprehensive assessments of the IC CAE program. A key practice of sound strategic planning is the use of assessments, through objective measurement and systematic analysis. For example, an evaluation plan can assist an agency in determining the appropriateness of a program’s goals, the effectiveness of implemented strategies, and the potential need for corrective action. The memorandum of understanding between DIA and ODNI in 2011 and amended in 2013, designated performance reporting as a DIA responsibility, but DIA did not identify performance assessment as a responsibility in program guidance. The IC CAE program office’s standard operating procedures provide that the grant officer’s representative monitors an IC CAE college’s compliance with its grant assistance agreement and collects performance and financial data reports. However, there is no mention of a systematic, outcomes-based assessment of these reports or the program as a whole. Without such assessments, the IC will not be able to determine whether the IC CAE program is effectively increasing the pool of diverse applicants. Congress will also be unable to determine the return on investment in this long-standing program. IC elements participate in the IC CAE program in a variety of ways, including by attending IC CAE college workshops and recruitment events and participating in the annual IC CAE program meeting, among other events. Table 3 shows the varying levels of participation in the IC CAE program among the eight selected IC elements, as reported by IC element officials. IC elements’ participation in the IC CAE program varies according to the specific organizational needs of each IC element. Some IC elements do not participate actively in the program because they do not directly hire employees into their intelligence office or because they conduct only limited hiring. For example, according to officials from the Department of Energy’s Office of Intelligence and Counterintelligence, the office is small and hiring is therefore limited. Further, officials stated the office often hires specialized personnel with advanced degrees and would not hire IC CAE scholars from undergraduate programs. Similarly, State Department officials from the Bureau of Intelligence and Research stated that they do not participate in events since they do not have direct hiring authority. Further, these officials stated that the State Department’s participation in IC CAE events is also constrained by limited personnel and financial resources. Other IC elements, such as the CIA and the NSA, have developed separate relationships with colleges and programs to address their specific hiring needs. CIA. The CIA has reduced its involvement with the IC CAE program to better align its needs according to CIA officials. In 2009, CIA selected senior officers to serve as advisors to 16 IC CAE colleges. The CIA advisors were directed to make a minimum of two visits per year and conducted a wide range of activities to include presenting at colleges events, counseling IC CAE scholars, and discussing CIA and IC career opportunities. However, about a third of the advisors were pulled back because, according to CIA officials, the IC CAE colleges were not meeting hiring expectations. Since 2014, CIA has focused its efforts on only six of the IC CAE colleges based on the return on investment from these colleges and alignment with CIA hiring needs. In addition, according to CIA officials, CIA has designated five universities as signature colleges to recruit skilled applicants from a range of cultures and backgrounds. According to CIA officials, the signature college program targets large, diverse colleges where the CIA has received a significant number of applications. Its criteria for selection of signature colleges include high diversity, the size of the college, and potential for developing a deep relationship. Two of the five CIA signature colleges are also in the IC CAE program and are currently receiving or have received grant funding. NSA. According to NSA officials, NSA has been involved in the IC CAE program since its inception, and its involvement includes participating in a variety of events such as colloquium, summer seminars, and recruitment events. In addition, NSA has also sponsored two types of Centers of Academic Excellence, one for cyber defense and one for cyber operations. The goal of these programs is to develop technical skills by promoting higher education and research in cyber defense and producing professionals with cyber defense expertise. In addition, the programs also aim to broaden the pool of skilled workers capable of supporting a cyber-secure nation. The programs involve awarding a designation as a Center of Academic Excellence in Cyber Defense or Cyber Operations to U.S. universities based on criteria. No funding is provided to the U.S. universities. According to NSA officials, these programs are independent of the IC CAE program and have different goals from the IC CAE program. Officials stated NSA’s CAE programs are focused specifically on increasing the pipeline of cyber talent. Further, some IC elements’ recruitment strategies incorporate the IC CAE program as part of their strategy, but it is not the only aspect of the elements’ approach to recruiting. For example, according to NGA’s Campus Recruitment Strategy, the agency targets high-quality colleges that provide access to diverse applicants in high-quality, mission-aligned degree programs across a broader geographic reach. The strategy has 31 designated colleges that were selected based on a variety of criteria, including demographic diversity and academic programs that align with the agency’s mission areas. According to NGA officials, they continue to recruit from at least seven IC CAE colleges; however, being an IC CAE college was not part of the primary selection criteria for colleges in NGA’s campus recruitment strategy. As program manager, DIA has relied on the IC CAE Senior Advisory Board and its charter as a means to engage IC elements in the program. However, not all IC elements participate on the Senior Advisory Board or in the IC CAE program. For example, in the November 2017 board meeting, only 9 of the 17 elements attended the meeting and a quorum was not established. Without a quorum, votes held during a meeting are not valid and actions cannot be approved. Moreover, during board meetings, members have raised concerns about limited attendance, citing concerns that only about half of the members regularly attend. According to some IC element officials, they do not attend IC CAE program events, including the Senior Advisory Board meetings, because the program does not meet their IC element’s organizational needs. For example, as discussed above, some IC elements have developed separate relationships with colleges not in the IC CAE program. Further, as discussed above, some IC elements have developed separate relationships with colleges and programs to address their specific hiring needs. As a result, some IC element officials have stated they have intentionally reduced their recruitment at some IC CAE colleges. Since not all IC elements participate in the IC CAE program or attend the board meetings, DIA has had to conduct other outreach to engage IC elements. According to DIA officials, since 2017 the IC CAE program office has conducted additional ad hoc outreach to engage with IC elements. For example, DIA officials have stated the IC CAE program office has utilized ODNI forums, such as the IC Recruitment Council and IC Chief Human Capital Office Council to engage with IC elements on the IC CAE program. However, DIA officials also stated that not all IC elements attend these ODNI council meetings because different offices within the IC elements are responsible for attending the meetings. Some IC elements are set up differently with regard to which office within the IC element participates in the IC Recruitment Council, so the IC element representatives to the IC CAE Board can differ from those who attend the IC Recruitment Council. While these ad hoc outreach efforts are likely a positive step to improving coordination, there remains a lack of engagement by all IC elements. Standards for Internal Control in the Federal Government state that management should establish and operate monitoring activities, to include a determination of when to revise the program baseline to address program needs. Further, the standards state that management should evaluate and document the results of ongoing monitoring and separate evaluations to identify issues. As program manager, DIA has not established a process for monitoring and assessing IC elements’ participation in the IC CAE program, and the board’s charter does not describe such a process. As result, DIA does not fully understand the reasons for the lack of engagement on the part of IC elements. IC elements that do not attend board meetings are not engaged in the discussions and decisions being made about the program. Similarly, IC elements that do not participate actively in the program have limited contact and interaction with IC CAE colleges, which has hampered the effectiveness of the IC CAE program. Without a process for monitoring and assessing IC elements’ participation in the IC CAE program, ODNI will not be able to tailor the program to meet the needs of the IC and address the overall program goal of creating a diverse pool of applicants for the IC. Assessing and addressing IC elements’ reasons for not participating in the program would increase ODNI’s understanding of the factors that inhibit participation and inform an approach to mitigating these factors and achieving program goals. The IC CAE program is a collaborative effort that allows IC elements to participate at college events, such as colloquia, speaker series, and campus recruitment events. The IC CAE Senior Advisory Board was created to provide policy and guidance for the IC CAE program and ensure that participating IC elements are included in decisions related to policy matters. The board’s charter states the Senior Advisory Board members are responsible for attending board meetings, voting on issues before the board, acting as points of contact for the program, and promoting the program. However, the charter does not define the expected or required level of participation of IC elements at IC CAE colleges. The IC CAE program manager, DIA, has communicated the schedule of IC CAE college events during Senior Advisory Board meetings and also asked for IC elements to participate in various events. Through the IC CAE grant process, IC CAE colleges are required to host a variety of events to educate IC CAE colleges about the IC. Based on the IC CAE grant announcements, these events are predicated on IC element participation. Specifically, recruitment fairs at colleges are facilitated by IC elements and IC element officials are speakers at colloquia events, with a primary goal of maximizing relationships and outreach. However, some colleges have experienced challenges with engaging with IC elements to attend these events. For example: An official from a legacy IC CAE college noted that it has been difficult to get IC elements to attend college events or recruit from the college. The official stated that IC element participation has been ad hoc and based on personal relationships with the IC elements rather than assistance from the IC CAE program office. For example, the official noted that at recent events the college was only able to attract 8 IC elements to a recruiting event compared to the 20 representatives across 12 IC elements who attended the events in the past. An official from an active IC CAE college also noted that some IC elements are not well informed about the IC CAE program. For example, the official noted that the college would like more IC elements to attend IC CAE college events. However, the official stated that the responsibility of developing relationships with IC elements has been placed on the college. According to the official, the IC elements should be more aware of which colleges have IC CAE programs and should be the first stop for IC element recruitment. The official also stated IC CAE colleges would like the IC element to drive the relationships with colleges. Our leading collaboration practices include (1) having participating agencies clarify roles and responsibilities and (2) ensuring that participating agencies document how they are collaborating in a written agreement and develop ways to continuously update and monitor these agreements. Roles and responsibilities can be defined through laws, policies, memorandums of understanding, or other requirements. The IC has defined the mission for the IC CAE program, but the current program manager, DIA, has not clarified IC element roles and responsibilities for program participation and the Senior Advisory Board charter does not clarify what is expected of the IC elements regarding participation at IC CAE events. According to DIA officials currently managing the program, the Senior Advisory Board charter is the key to getting IC element participation in the program and overall program success. An update to the Senior Advisory Board charter could include all relevant participants and define roles and responsibilities. Without clearly defined roles and responsibilities, the IC elements are not taking full advantage of what the IC CAE program has to offer, including participation in events and college engagement. Thus, the IC CAE colleges will not be able to fully execute their IC CAE programs and the program may not be able to meet its goal of creating a pool of diverse applicants for the IC. In 2005, ODNI established the IC CAE program with a goal of creating an increased pool of culturally and ethnically diverse, multi-disciplinary job applicants for the IC. However, the current program manager, DIA, has not sufficiently planned and overseen the program and the IC is unable to determine whether the program has been successful in meeting its goal to create an increased pool of culturally and ethnically diverse job applicants for the IC. Specifically, DIA has not developed results-oriented goals or documented an overall strategy for the program, evaluated external factors that could significantly affect the program’s success, defined and collected comprehensive metrics, or conducted an assessment of the program’s performance. As ODNI takes over the program, it needs to address these sound planning practices in order to determine whether the program is being implemented successfully and to help ensure the IC has a trusted, diverse workforce with the right expertise. Further, without sufficient planning and oversight, decision makers will also be unable to determine the return on investment in this long-standing program. In addition, ODNI also needs to improve IC element participation in the program. The IC CAE program is a collaborative effort that encourages participation among all IC elements. However, DIA has not established a process to monitor and assess IC element participation in the program or clearly defined IC elements roles and responsibilities for the IC CAE program. A process for monitoring and assessing IC element participation and addressing IC elements’ reasons for not participating in the program will increase understanding of the factors that inhibit participation and inform ODNI’s approach to mitigating these factors and achieving its goal for the program. Further, without clearly defined roles for IC element participation in the program, IC CAE colleges may not be most effectively executing their IC CAE programs and the program overall may not be able to meet its goals. We are making the following seven recommendations to the Director of National Intelligence as the IC CAE program transitions to ODNI: The Director of National Intelligence should establish and document results-oriented goals that include specific targets or milestones for the IC CAE program. (Recommendation 1) The Director of National Intelligence should establish and document strategies to achieve the results-oriented goals that are established for the IC CAE program. (Recommendation 2) The Director of National Intelligence should develop and document a process to identify and continuously evaluate external factors that could affect the program’s ability to achieve identified goals. This should include, but not be limited to, a consideration of program branding and post-grant sustainment. (Recommendation 3) The Director of National Intelligence should define and document comprehensive performance measures for the IC CAE program, collect and evaluate the completeness and reliability of information it receives from grant recipients and IC elements, and report this information on a regular basis. (Recommendation 4) The Director of National Intelligence should establish a requirement for and develop a plan to periodically evaluate the IC CAE program’s performance through objective measurement and systematic analysis. (Recommendation 5) The Director of National Intelligence should develop a process for assessing why some IC elements are not participating in the IC CAE program and address these reasons in order to ensure the program is structured to meet the needs of IC elements. (Recommendation 6) The Director of National Intelligence should clearly define IC elements’ roles and responsibilities for participation in the IC CAE program to better facilitate interagency collaboration in support of the program. (Recommendation 7) We provided a draft of this report to ODNI for review and comment. In written comments, ODNI concurred with all seven of our recommendations but did not identify the steps it plans to take to address the recommendations as the IC CAE program transitions to ODNI. ODNI’s comments are reprinted in their entirety in appendix III. ODNI also provided technical comments, which we incorporated as appropriate. We also provided a draft of this report to the CIA, Department of Defense, DIA, FBI, NGA, NRO, NSA, the Department of State’s Bureau of Intelligence and Research, and the Department of Energy’s Office of Intelligence and Counterintelligence for review and comment. These agencies concurred without providing comments on the draft report. NGA provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees. We are also sending copies to the Secretaries of Defense, Energy, and State; the Directors of National Intelligence, DIA, CIA, NGA, NRO, and NSA; and the Attorney General. In addition, this report will be available at no charge on our website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact Brian M. Mazanec at (202) 512-5130 or mazanecb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. The Office of the Director of National Intelligence (ODNI) was the Intelligence Community (IC) Centers for Academic Excellence (CAE) program manager from 2005 through 2011. Internal documents and grant announcements from that period state that the program’s mission was to increase the pool of eligible applicants in core skills areas, specifically targeting women, racial and ethnic minorities and individuals with varied cultural backgrounds, regional expertise, and language proficiency. ODNI outlined four goals in its 2008 guidance for the program, including a focus on developing relationships with colleges, providing resources and grants to competitively selected colleges, providing technical assistance in the design and implementation of colleges’ IC CAE programs, and documenting results to improve the efficacy of the IC CAE program. Each of these goals included supporting objectives. For example, the goal of providing support, resources, and grants to competitively selected colleges included four supporting objectives, such as instituting long-term practices to increase relationships with minority-serving institutions and providing access to IC internships, co-ops, and graduate fellowships. These goals and objectives were aligned with the program’s overall mission, but they were not defined in measurable terms that would allow future assessments of whether they were being achieved. For example, ODNI did not establish targets for the goals or supporting objectives listed above that would have allowed it to determine how successful it had been at supporting long- term programs at minority-serving institutions or providing access to IC employment opportunities. In addition, ODNI defined a strategy to support its program goals, and the strategy included the following four elements: outreach to high schools; operations at colleges, including curriculum development; infrastructure at the colleges to support these operations such as faculty and administrators; and relationships between IC CAE programs and IC elements. These elements of ODNI’s strategy described specific operational requirements for the program. For example, IC CAE grant announcements in 2006, 2009, and 2011 supported a wide range of academic activities that prioritized the development of curricula in national security studies, science and technology programs, study abroad programs, courses in critical languages, and pre-collegiate outreach through activities like summer camps to raise awareness and interest in IC careers. ODNI also defined assessment and evaluation as an overarching part of the program’s strategy, as shown in figure 5. ODNI worked with a contractor to conduct annual performance evaluations through 2012. The contractor developed an evaluation methodology and reviewed colleges’ interim reports, collected and verified performance data, and developed findings and recommendations. For example, the contractor recommended that IC CAE colleges broaden their critical language offerings and increase the number of IC CAE Scholars enrolled in foreign languages courses in each of the annual reports from 2007 to 2010. ODNI defined performance measures and reported data on activities, including the number of IC CAE courses and events, demographic information, and employment outcomes. Specifically, IC CAE colleges were required to report these data quarterly, and the contractor compiled the data annually into its program reviews. Table 4 shows selected performance measures outlined in ODNI’s final report that summarized information collected from 2004 through 2011. Table 5 and table 6 show the 46 grants managed by Office of the Director of National Intelligence (ODNI) and the Defense Intelligence Agency (DIA). The total amount of grant funding projected to be obligated from fiscal year 2005 and fiscal year 2021 is $69,053,618, not including a $250,000 contract in September 2004 to initiate a pilot Intelligence Community (IC) Centers for Academic Excellence (CAE) program at Trinity Washington University. Tables 7 and 8 list the IC CAE colleges by designation of eligibility for Department of Education funding as a minority serving institution under various statutory grant programs including programs authorized by the Higher Education Act of 1965, as amended. Eligibility for grant funding under these statutory programs as determined by the Department of Education in 2018 does not designate or certify any college as a particular type of institution, for example, as a Hispanic Serving Institution. The colleges listed in tables 5 and 6 are listed in the order that they received a grant by fiscal year and some IC CAE colleges received multiple grants. Grants fund a base year and up to 4 additional option years. The consortium colleges below are listed alongside the IC CAE college that received a grant. ODNI and DIA awarded IC CAE grants to colleges following an announcement for proposals in fiscal years 2006, 2009, 2011, 2014, 2017, 2018, and 2019. In addition to the individual named above, Kristy Williams, Assistant Director; Jason Bair; Tracy Barnes; John Bumgarner; Meeta Engle; Gina Hoover; Amie Lesser; Benjamin Licht; Ned Malone; Parke Nicholson; Alice Paszel; Sarah Veale; and Lillian Yob made key contributions to this report.", "summary": "A trusted, diverse workforce with the right expertise is critical to ensuring the IC achieves its mission of delivering distinctive, timely insights with clarity, objectivity, and independence. ODNI established the IC CAE program in 2005 to educate highly qualified students of diverse backgrounds and encourage them to pursue careers in the IC. ODNI and DIA have provided 29 colleges a total of 46 IC CAE grants through fiscal year 2018, totaling approximately $69 million through fiscal year 2021. This report evaluates the extent to which (1) DIA has planned and overseen the IC CAE program since 2011 and (2) selected IC elements are participating in the IC CAE program and have clearly defined roles. GAO reviewed IC CAE documentation related to DIA program planning and oversight from 2011 through 2019 and applied key practices of sound planning to evaluate DIA's management of the program. GAO interviewed selected IC elements and IC CAE college officials and reviewed related documentation to assess program planning and implementation. The Defense Intelligence Agency (DIA) has not sufficiently planned and overseen the Intelligence Community (IC) Centers for Academic Excellence (CAE) program—intended to create an increased pool of culturally and ethnically diverse job applicants for the IC—after the program transitioned from the Office of the Director of National Intelligence (ODNI) to DIA in 2011. Specifically, DIA has not applied most of GAO's key practices of sound planning in overseeing the program (see table), thus challenging decision makers' ability to determine the program's return on investment. Specifically, while DIA has developed some short-term goals and plans for the program, DIA has not established results-oriented program goals or an overall strategy that details the agency resources and processes required to achieve the program's mission. Similarly, DIA collected some data for the program and required colleges to provide reports on significant program accomplishments, but these data are not complete or reliable and have not been used to comprehensively evaluate the program's success. As oversight responsibility for the IC CAE program transitions back to ODNI in fiscal year 2020, ODNI will not be able to determine the extent to which the program has been successful in achieving its mission without establishing and documenting goals with targets and milestones; developing strategies to achieve those goals; and defining, collecting, and reporting comprehensive performance measures. Selected IC elements are participating in the IC CAE program to varying degrees, but DIA has not established a process for monitoring and assessing IC elements' participation or clearly defining IC elements' role in the program. The IC CAE program is a collaborative effort that allows IC elements to participate in college events, such as IC CAE recruitment events. However, not all IC elements participate in the program. As IC CAE program manager, DIA has engaged with IC elements in a variety of ways, but this engagement has not resulted in consistent participation among the IC elements. Moreover, program documentation has not clearly defined IC elements' roles and responsibilities for participation. Without a process for monitoring and assessing IC elements' participation and clearly defining roles and responsibilities, ODNI will neither be able to identify reasons for the lack of IC element engagement nor ensure that IC elements are taking advantage of the IC CAE program and its goal of creating a diverse pool of applicants for the IC. GAO is making seven recommendations to the Director of National Intelligence, including that ODNI establish and document results-oriented goals and strategies for the IC CAE program; define, collect, and report comprehensive performance measures; and clearly define the roles and responsibilities of the IC elements for participation in the program. ODNI concurred with the recommendations but did not identify steps it plans to take to implement them.", "document_type": "gao"}
{"report": "Two GSA offices have roles in managing data related to federal real property. The Public Buildings Service (PBS) acts as a landlord for the federal government by acquiring new space for government agencies and tracking data on the property it acquires. PBS manages and publishes three databases that provide information to public stakeholders and researchers on federally owned and leased properties, and on properties eligible for disposal. Another office, the Office of Government-wide Policy (OGP), collects, manages, and reports on all federal real-property data through the FRPP database. OGP has managed the FRPP since its inception in fiscal year 2005 by collecting data from federal agencies on their real property assets. OGP is also responsible for compiling and managing the public database required by FASTA. FRPP is the most comprehensive database of federal real property holdings, containing details for about 398,000 assets (buildings, structures, and land). It is not public, but it also does not contain any classified national security information. FRPP data show the range of agency assets, including single buildings in a given location or multiple buildings located on installations, like a national park or research center. The FRPP identifies whether buildings are on installations, but does not identify whether buildings are public-facing or secure (and thus inaccessible by the public). We have repeatedly identified reliability issues with the FRPP, and GSA has taken actions to improve the reliability of FRPP data. Specifically, in 2016, GSA established its validation and verification (V&V) process. After agencies submit their data annually to FRPP, GSA identifies questionable entries (called anomalies) from 20 separate categories. Through these categories, GSA flags assets that are very small in size, changed from the previous year, or have unusual financial statistics, among other things. GSA then provides an annual list of anomalies to the agencies that entered the data. Agencies have 10 months to research each anomaly and correct errors or validate that the data are correct. GSA has provided instructions to agencies on how to respond to the V&V process. GSA also requires agencies to certify accuracy of the data and established database rules that require agencies to submit complete information on assets. GSA officials said that it must ultimately rely on agencies to submit correct data. FASTA required GSA to publish a single, comprehensive, descriptive database of all federal real property by December 16, 2017, while allowing it to exclude assets for reasons of national security, such as those that are secure installations. FASTA also required the database to be made public to the extent its release is consistent with national security and procurement laws. GSA officials said that GSA used the FRPP as the basis for developing the database it released to the public at the end of 2017. GSA presents the data in two ways: as a downloadable spreadsheet or in a searchable mapping application. FASTA requires that the public database be machine-readable and permit searching or sorting of data to the extent practicable. Further, GSA guidance also calls for agencies to provide accurate and complete data. Specifically, GSA requires agencies to include either a complete street address or geo-coordinates for all 398,000 assets in the FRPP; for example, GSA’s FRPP data dictionary establishes the format agencies are to use when inputting asset addresses—number, street, city, zip code. This requirement carries over to the 305,000 assets included in the public database. We found that almost 214,000 of the assets in the public database included some street address information, but most of the addresses were incomplete or incorrectly formatted. Specifically, only approximately 70,000 (33 percent) fully met the standards. Since another 91,000 assets did not include a street address, a computer would only be able to locate about 23 percent of the 305,000 civilian federal assets using street addresses in the public database (See fig. 1.) GSA officials who manage the FRPP said that they were aware that many street addresses were not readable and have asked agency officials to review the accuracy of address information and correct it in future submissions. They acknowledged, however, that their efforts were not fully successful. As discussed later, GSA is currently taking steps to ensure that agencies provide more complete geo-coordinates when they submit data to the FRPP. For the remaining 67 percent of the assets (144,000) with some street address information that did not fully meet the standards, we found two types of problems—incomplete addresses and addresses that were not formatted correctly. First, more than 28,000 assets had street addresses that were incomplete. For example, instead of having individual address listings, we found that all 215 buildings at the Goddard Space Flight Center had a single listing of “Greenbelt Road.” This road actually stretches over 6 miles and many other buildings are located along the road. The front gate’s complete address is “8800 Greenbelt Road.” In these instances, GSA officials said that its public-mapping program selects the mid-point of the street, which in this case is over a mile from the public entrance to the installation. (See fig. 2.) As a result, someone using the database would not be able to determine exactly where Goddard is. Second, we found about 115,000 assets had street address information that was incorrectly formatted based on FRPP instructions. While we did not conduct a complete analysis of all these assets, we found examples of some of the address issues, such as: Extra descriptive information about the property in the address field. For example, “N220 AG Science Bldg North U of Kentucky” and “Beltsville AG Research Center, 10300 Baltimore Avenue.” The data in the address field for these two assets—which belong to the Department of Agriculture—could not be directly read by a computer or displayed on a map. Unrecognizable text. For example, “2881 F;B Road” and “1-15, Exit 172, 1 Mile East.” The data for these assets, which belong to the Department of Agriculture, could not be directly read by a computer or displayed on a map. GSA officials said that users may be able to interpret the individual asset addresses in the database but that GSA’s automated computer system could not map unreadable addresses. Similarly, a private-sector user who tried to use the public data to map federal facilities for clients said that he was unable to map many of the assets because addresses were not readable by his computer. As a result, he said that he excluded incomplete or unreadable addresses from the database he created. He noted that incomplete data would reduce clients’ interest. We also found problems with assets for which agencies provided geo- coordinates (latitude and longitude). Specifically, GSA guidance states that geo-coordinates must include a minimum of four decimal places. Of the 305,000 assets included in the public database, almost 220,000 included geo-coordinates but more than half—about 141,000—did not meet FRPP standards because they were not precise enough to map the location of the assets. GSA officials noted agencies are required to enter some type of information in the field for address or geo-coordinates, but an “open data” format did not prevent agencies from reporting information that was not strictly a street and address number. Consequently, some agencies may have entered incorrect values for the geo-coordinates just to complete the field. Our analysis supports this view; few (550 of about 131,000) of the assets with both sufficiently detailed geo-coordinates and street addresses pointed to the same location. In addition to the open data issue described above, officials also explained that GSA did not have a “business validation rule” in place that prevented agencies from inputting coordinates with less than four decimal places. GSA has taken a number of actions to correct the issues with geo- coordinates that they say should help address this problem for the next release of the public data in 2020. For example, GSA added V&V anomaly categories for fiscal year 2018 data that identified GPS coordinates pointing to unlikely locations, such as a location in the water, which identified about 80,000 potential anomalies. Agencies are currently checking these. Additionally, GSA added a feature to the fiscal year 2019 FRPP submission form that will force agencies to provide geo- coordinates that are detailed enough for their data to be accepted. GSA officials said that they would consider taking additional steps once they have analyzed the results of the GPS coordinate anomaly categories. GSA has asked agencies to review addresses for accuracy, and officials indicated that they have discussed plans to improve this data. However, GSA has not taken specific steps to work with agencies to ensure they input correct street addresses in the public database in light of the “open data” format. The lack of correct street addresses can affect users who may be interested in acquiring or leasing assets or who may be interested in installing telecommunications devices on an asset, from knowing exactly where those assets are located. As a result, until the street address information is complete and correctly formatted, the public may unknowingly pursue assets that are not available or suited to their needs. We found that while GSA has identified close to 30,000 potential errors in the FRPP database over the first 2 years of the V&V process, agencies confirmed only 5 percent as errors (1,291 of 28,572). Agencies validated the remaining 27,281 anomalies as correct or left them unresolved. The low number of errors being identified indicates that GSA’s V&V process is not efficiently identifying errors in the data, either in terms of the anomaly categories themselves or the thresholds at which GSA flags data as an anomaly. This situation could ultimately mean that agencies are spending time researching correct information that was flagged as potentially erroneous or not fully actually researching anomalies and allowing mistakes to remain uncorrected. Agencies identified no anomalies as errors for five of GSA’s 16 anomaly categories in 2017, raising questions about the anomaly categories GSA has identified. OMB guidance suggests that agencies only do extra tasks that are justified by their cost. GSA officials who manage the V&V process said that the high number of anomaly categories for which agencies found no errors could reflect that the anomaly categories are flagging correct data as anomalies or that agencies are validating data as correct without actively checking it. We found examples of both. For example, we examined a selected sample of 14 V&V data anomalies at DOE sites in New Mexico. GSA flagged the buildings for being very small—office buildings less than 400 square feet and warehouses less than 64 square feet—and found that the information in the public database was correct. Figure 3 illustrates how such information flagged as being questionable, is actually correct according to GSA’s reporting rules for agencies, which specify data categories, such as the types of buildings GSA considers to be warehouses. Specifically, GSA flagged assets at DOE’s Los Alamos and Sandia National Laboratories because their square footage fell below certain amounts. But, in reality, these assets met GSA’s criteria for offices and warehouses despite being small. We also found instances where an agency verified information as correct that was incorrect. Figure 4 illustrates examples data validated as correct that was actually erroneous. Specifically, an agency erroneously reported water towers and antenna arrays as office buildings. Staff responsible for managing the V&V process for their agency’s assets said that they did not always consult the personnel with the best knowledge of the assets in resolving anomalies. Instead, they relied on their own judgment when determining whether to forward the anomalies to asset managers to ultimately check the data and correct any errors. This resulted in some errors going uncorrected. Thresholds—the points at which GSA flags data as anomalies—lead to a large number of data elements flagged, which can challenge the resources of affected agencies. Officials at two of our selected agencies said that the number of anomalies that the V&V process produces annually overwhelms their ability to validate the data. The large number of unresolved V&V anomalies appears to support this conclusion. GSA’s guidance allows agencies 10 months to validate the anomalous data, but the number of anomalies that remain unresolved after 10 months has risen sharply. Figure 5 shows that while agencies addressed all anomalies in the first year, they have since struggled to keep up. As of October 2019, 106,231 anomalies, or approximately 71 percent, remained unresolved after 10 months. Officials who are responsible for resolving anomalies at two selected agencies said that more realistic anomaly categories or thresholds could reduce the number of anomalies and better target actual errors, an approach that could help agencies better prioritize their resources when researching anomalies. GSA staff who manage the FRPP said that they brainstormed internally and used industry standards and policy initiatives to develop anomaly categories. They also explained that they adjust thresholds within each category. However, GSA officials said they had not reviewed the anomaly categories or their thresholds to see if they consistently capture incorrect data. This approach puts the stated goals of the V&V process—which are to improve data accuracy, promote data consistency among the agencies, and enable OMB to measure data quality improvement—at risk. In the absence of better information about the validity of categories and thresholds, the current process for V&V is taking up limited agency resources without efficiently correcting errors in the data. GSA and reporting agencies decided not to provide certain useful information from the public database in two ways, thereby reducing the data’s completeness and ultimately its utility. First, GSA withheld data from the public database without consulting agencies about their sensitivity. Second, selected agencies withheld information that was already publicly available or withheld similar types of information inconsistently within their agencies. GSA chose to withhold 15 categories of data from the public database for all agencies. FASTA authorized the withholding of information from the public database for national security or procurement-related issues. GSA officials who manage the FRPP said that GSA does not have the security or intelligence expertise to issue guidance on national security issues. As a result, they sought input from the ISC on what information to withhold. ISC reviewed the security risks of FRPP data and provided written recommendations in a memo to GSA in November 2017. Specifically, ISC recommended that certain categories of data on assets be withheld from the public database because of the security risk that they could pose individually or in combination. ISC also recommended that agencies use internal guidance on restricting the public release of real property information and ISC’s mission criticality criteria to determine any individual real property assets to withhold entirely from the public database. GSA implemented ISC’s first recommendation by withholding 15 FRPP data categories for all assets from the public database without consulting the relevant agencies on this decision, considering the specific sensitivity of these categories for all assets, or assessing the effect withholding them would have on the database. ISC officials acknowledged that the memo that they prepared for GSA could have been clearer as to ISC’s intent that departments and agencies should consider the recommendations in making a final determination. According to ISC officials, they believed that implementation would involve GSA communicating these recommendations and leaving decisions on what to withhold to officials within individual departments and agencies who control real property assets. Specifically, the following five categories of data were among the 15 withheld by GSA: property’s/installation’s name, replacement value of an asset, annual operating and maintenance costs for owned assets, annual-operating and maintenance costs for leased assets, and breakdown of annual operating and maintenance costs (e.g., utilities costs, janitorial costs, sewage costs, etc.). Because GSA did not consult with agencies on this decision, the agencies did not have an opportunity to consider whether or not the 15 data categories GSA withheld included information that is sensitive or already publicly available. As a result, the public database is incomplete in ways that adversely affect users and limits agencies’ public accountability for reporting accurate information. For example, identifying assets in the public database is difficult without the property’s name—one of the data categories GSA withheld—especially given the insufficient location data in the database discussed earlier. Returning to the incomplete address example discussed earlier (NASA Goddard Space Flight Center), the public data also do not include the property’s name, “Goddard Space Flight Center,” leaving users with limited information to identify the buildings. As a result, someone using the public database cannot identify assets on NASA’s Goddard Space Flight Center campus without using outside sources for additional information. (See table 1.) As discussed in the next section, we found that some of the information from these 15 excluded data categories, such as property names, is often already in the public sphere. For example, “Goddard Space Flight Center” and its address are clearly disclosed on NASA’s public website, but GSA withheld the name for 215 NASA buildings at this address, including Goddard’s public visitors’ center. Using the public database alone, a member of the public would need to go through numerous steps to determine if assets are part of Goddard Space Flight Center and still have no way of being sure. (See fig. 6.) Moreover, third-party, private sector stakeholders we spoke with such as brokers, lessors, consultants, and a non-profit organization that work in federal real-property markets, noted that some of the data categories GSA withheld would be among the most useful to their work. For example, 10 of 14 stakeholders we spoke to said that financial data, such as operating costs and annual rent, were among the most useful FRPP data categories to their analyses of real property markets and business opportunities. Additionally, four stakeholders cited the property’s name as among the most important data categories for their work in analyzing federal real property. While GSA withheld the 15 categories of data across all agencies, it allowed each agency to determine if any specific assets should be withheld entirely from the public database, in accordance with ISC’s second recommendation. ISC officials told us that this was appropriate because individual departments and agencies that control real property assets should determine what information to withhold. GSA provided agencies with guidance that explained its decision to withhold the 15 data categories and instructed agencies to consult ISC’s mission criticality criteria and any additional internal agency criteria in determining what information to withhold from public release. ISC’s mission criticality criteria provide a page-long list of uses of real property assets that warrant consideration for national security exclusion, but do not provide other instructions for agencies to consult while making decisions on what information to withhold. Further, OMB Circular— Management of Reporting and Data Integrity Risk also instructs agencies to integrate a risk-based approach towards meeting reporting objectives, an approach that requires “management practices that identify, assess, respond, and report on risks.” However, we found that our selected agencies did not consistently identify internal guidance to supplement GSA’s instructions within their agencies. In September 2018, ISC recommended that GSA not withhold from the public database newly added data categories that provide information already in the public sphere. Additionally, the OPEN Government Data Act requires OMB to foster greater sharing, dissemination, and access to public information and issue guidance that, among other things, takes into account the requirement that data must be disclosed if it would otherwise be made available under a Freedom of Information Act request. For purposes of this report, we refer to this requirement as “assuming openness.” However, GSA’s instructions to agencies lacked specifics to help agencies apply a consistent, risk-based approach in determining which, if any, assets or asset-specific information should be withheld from public release. As a result, we found that some of the selected agencies withheld asset-related information from the public database that is available on their own public websites or from other official sources. Withholding information that is already publicly available unnecessarily reduces the completeness and utility of the public database that FASTA indicated should be comprehensive. For example: DHS’s Immigration and Customs Enforcement (ICE) withheld buildings at five of its publicly-accessible service-processing centers that are shown on a detention facility locator mapping system on its own website. ICE officials told us that they did not consider what information is already publicly available when deciding what information to withhold from the public database. FCC withheld all of its real property assets. FCC’s own website and regulations, however, list the locations and functions of FCC offices. The U.S. Coast Guard withheld information on its public-recruiting offices and lighthouses that it advertises on its public website. All buildings and structures that were not specifically used for the purpose of aids to navigation were withheld from the public data set. As a result public users can look up information on the Coast Guard’s aids to navigation, but cannot look up some of its publicly accessible locations, such as recruiting offices and lighthouses. In contrast, DOE decided to withhold none of its 20,378 assets from the public database. According to a DOE official responsible for submitting data to FRPP, DOE does not have a specific process for assessing what properties to make public. However, it is aware that much of the information in the public database is also publicly available through other sources. Table 2 shows how selected agencies took different approaches to withholding information from the public database. Under risk-based criteria assuming openness (as mentioned earlier), agencies may consider whether information made public in one instance should be withheld in another instance. However, neither ISC’s mission criticality criteria nor GSA’s instructions addressed the issue of consistency within specific agencies. Specifically, we found that selected agencies withheld the same assets differently over time, and similar assets inconsistently. Table 3 shows how reporting agencies made different decisions on whether to withhold the same types of assets. At times, some agencies withheld certain asset types that ISC’s mission criticality criteria did not identify as warranting withholding, resulting in almost 7,000 assets such as parking structures and disposed assets being withheld. This led to inconsistencies as to whether these agency assets were included or not in the public database, limited transparency about these assets, and prevented users from fully analyzing federal real property assets in these categories. In other cases, selected agencies withheld similar assets inconsistently, did not always follow written procedures and withheld similar assets. For example: DOI headquarters provided its bureaus with GSA’s instructions on withholding assets, but individual bureaus applied the instructions differently. For example: The Fish and Wildlife Service reports that it has 369 publicly accessible national wildlife refuges, but it withheld selected real property assets at 11 of them. However, the withheld assets are the same types as the assets the Service disclosed at other refuges. For example, it reported all but two of 447 restrooms and 10 of 2,066 recreational structures on its national wildlife refuges. The Fish and Wildlife Service told us it will re-evaluate its withholding for the fiscal year 2019 FRPP database. The National Park Service (NPS) reported that it has 374 publicly accessible national parks, monuments, memorials, historic sites, and recreation areas. NPS withheld some real property assets from 15 of those sites. For example, it reported all but 2 of 1,045 service buildings at its sites. These withheld assets are the same types as those disclosed at other sites. NASA withheld assets at a centralized level, but headquarters officials told us that they have not established instructions or policies for these decisions. NASA officials told us that they withhold real property assets shared with agencies working in defense and/or national- security, which led NASA to withhold 1,517 assets in fiscal year 2017. In fiscal year 2018, however, we found that NASA withheld all assets at certain field centers, causing the number to more than double from 1,517 in fiscal year 2017 to 3,696 in fiscal year 2018. Finally, our comparison of the fiscal year 2018 FRPP and public databases found that seven agencies did not identify whether data on 3,845 assets should be withheld despite GSA guidance to do so for every asset. GSA included these assets in the public database without consulting agencies on the assets’ sensitivity or risks in releasing information on them. GSA officials said that these data should not have been accepted and that they had implemented controls to ensure that agencies identify whether data should be withheld. It is difficult for a user of the public database to determine when assets are located on a secure installation that the public cannot access. For example, returning to the NASA Goddard Space Flight Center illustration from earlier in the report, assets located at the Space Flight Center are listed individually, with no indication that the assets are all located on a secure installation. The public database lists all 215 assets at the same location—Greenbelt Road in Greenbelt, MD, but provides no further indication that the assets are part of a larger, secure facility. (See fig. 7.) Currently, GSA requires civilian agencies to report individual assets, including those on secure installations. Detailed, asset-specific information could be useful for government decision makers, and GSA applied this approach to the public database. However, asset-level information can cause challenges for users when they are located on secure installations because GSA withheld the installation names from the public database. Listing assets individually could prompt fruitless public interest in inaccessible secure facilities. One expected use of the public database is for the private sector to identify possible locations for installing commercial telecommunications infrastructure, such as cell towers and antennas. However, as this infrastructure cannot be installed on secure installations, the public database would be more useful to such companies if they could readily determine whether a potential location was on a secure installation or not. For example, officials on a secure installation we visited told us that reporting individual buildings does not make sense because there are few, if any, legitimate reasons for public interest in the individual assets on a secure installation. FASTA required GSA to develop a comprehensive database and provide the public with database access, but recognized the importance of protecting national security. In that respect, a key organizational issue faced by GSA and agencies is how to present data for reporting assets on campuses that are not accessible to the public. While non-disclosure is permitted, such actions to withhold this information may reduce the usefulness of the public database as a whole. The Department of Defense (DOD) takes a different approach for its secure military bases in the public database. According to GSA officials, DOD submits a separate summary-level report for public release. This summary-level information shields sensitive information and alerts users that those assets are not accessible or of use to private-sector interests. Civilian agencies’ assets located on closed federal installations are similar to those on DOD bases in that the public may have less interest in or reason for knowing about assets that are not available to the public. Officials from NASA and two DHS bureaus said that the installation-level approach to reporting would be more appropriate for their circumstances than the asset-level reporting currently applied to civilian agencies and would likely allow them to release more information to the public. Officials from DHS added that they already release some information to the public on the web site. We found that other selected agencies also release information about secure installations on their public websites, including NASA and its Goddard Space Flight Center. In our interviews with 14 private sector stakeholders, we found varying levels of awareness and understanding of GSA’s publicly available real- property datasets. Of the 14 private sector stakeholders we interviewed, eight told us that they were aware of the public database. Of these, five told us they tried to use it. Several selected stakeholders—regardless of whether or not they had used the database—cited concerns about the usefulness of the data, specifically with its reliability, completeness, formatting, and organization. For example, officials from one brokerage firm told us that, while the information could theoretically be useful for agency consolidation efforts, the database was too cumbersome to analyze for that purpose. Similarly, officials with a federal real-estate- consulting firm told us that they do not refer customers to the public database because they believe that the data are not complete, correct, or intuitive. Moreover, one member of a federal real-property trade association noted serious limitations in the database’s completeness and organization. In addition, one user said that he hoped the public release would allow better access to real property data but that the poor quality, completeness, and organization of the data means access to data is no better than it was before the release. Further, six of the private sector stakeholders we interviewed were not aware of the public database, including a stakeholder who confused it with GSA’s Lease Inventory database. The lack of a single location on GSA’s website that contains information about all of GSA’s real property databases may contribute to the awareness, confusion, and usefulness issues expressed by these stakeholders. Specifically, public access to the FRPP public database, the GSA’s Lease Inventory database and two other publicly available real-property databases is found in different places on GSA’s website: Public FRPP http://publicfrppdata.realpropertyprofile.gov (managed by GSA’s Office of Government-wide Policy) GSA lease inventory https://www.gsa.gov/real-estate/real-estate- services/leasing-policy-procedures/lease-inventory (managed by GSA’s Office of Leasing) GSA inventory of owned & leased properties https://www.gsa.gov/tools/buildings-real-estate-etools/inventory-of- owned-and-leased-properties (managed by GSA’s Public Building Services) GSA disposal inventory https://disposal.gsa.gov/s/ (managed by GSA’s Office of Property Disposal) The Open Government Data Act requires the Administrator of GSA to maintain a single public interface online as a point of entry dedicated to sharing an agency’s data assets with the public. While the databases serve different purposes, some asset-level data are similar, such as location or size. According to a GSA official, these databases are operated by different offices within GSA. This situation poses challenges to listing the database on a consolidated webpage. Nevertheless, GSA officials agreed that there could be clearer links and said that they plan to add them based on our findings. Without a consolidated webpage or clear links showing how the databases relate to each other and how to access each database, users of the various databases may not be aware of what databases do exist to search for assets that could be available to the public. The public database’s presentation issues, combined with stakeholder confusion and lack of awareness, could contribute to low numbers of people who accessed the database compared to another GSA-managed real property database. GSA data indicate that users accessed civilian agency data from the public database 147 times per month on average from December 2017 through July 2019 and some months fewer than 10 times. However, according to a GSA official, the number of times users access the public database through the GSA website doesn’t necessarily reflect the extent to which people use the data. The official explained that, since GSA only issues the data once a year, users only need to access and save it once for use in a given year and that GSA usually sees a peak in users accessing the data when GSA publishes its annual update to the database. As indicated in figure 8, there was a peak in users accessing the database when GSA first issued the 2016 data in December 2017, and again in March and April 2018 when GSA published 2017 data (28 and 162 times, respectively), and in June 2019 when GSA published the 2018 data (170 times). In comparison, users access another real property database, GSA’s Inventory of Owned and Leased Property database—which is updated weekly—more often than they access the public database. Users access the Inventory of Owned and Leased Property database to search for properties controlled by GSA. Specifically, since the public database was released in December 2017, the public has continued to access GSA’s Inventory of Owned and Leased Property almost 10 times more per month than the public database on average (see fig. 8). Federal agencies spend billions of dollars annually to operate and maintain hundreds of thousands of real property assets. GSA’s public database, extracted from FRPP data, is a comprehensive, descriptive, database of federal real property. Through the database, the public should be able to learn about federal assets, whether people are conducting research or interested in potential uses such as leasing or purchasing. Issues with the data, however, undermine these uses. GSA has taken a number of actions to improve the accuracy of the data, such as implementing the V&V process for identifying and correcting possible errors. But until GSA has better processes to ensure accuracy of street address information and identify anomalies, the public data will continue to lack the type of database most useful to the public. Moreover, the absence of a risk-based, consistent approach for withholding assets from the public database or reporting assets to it further erodes its utility. Finally, utilization of the data base is low; GSA’s choices on how the database information is presented and how users find out about and access the public database and other real-property databases may contribute to this lack of use. Unless GSA improves the accuracy, completeness, and usefulness of the public database, its intended benefits—to the public and the federal government—will remain unrealized. We are making the following six recommendations to GSA: The Administrator of GSA should coordinate with agencies to ensure that street address information in the public database is complete and correctly formatted. (Recommendation 1) The Administrator of GSA should coordinate with agencies to review V&V anomaly categories to better target incorrect data. (Recommendation 2) The Administrator of GSA should work in consultation with agencies to determine which, if any, data should be withheld from public release. (Recommendation 3) The Administrator of GSA should instruct each agency to apply a consistent, risk-based approach in determining which, if any, assets or asset-specific information should be withheld from public release. (Recommendation 4) The Administrator of GSA should allow agencies to provide summary data for secure installations. (Recommendation 5) The Administrator of GSA should link all of GSA’s publicly available real- property data sources. (Recommendation 6) We provided a draft of this report to GSA, DHS, DOE, DOI, FCC and NASA for comment. GSA provided written comments, which are reprinted in appendix II and summarized below. We received, via email from DOI, technical comments, which we incorporated as appropriate. DOI, in its email comments, also suggested revisions to two recommendations, which we clarified as appropriate. DHS and NASA provided, in email, technical comments, which we incorporated as appropriate. DOE and FCC told us they had no comments. GSA agreed with five of our six recommendations but disagreed with our third recommendation. GSA wrote that allowing agencies to unilaterally determine which categories of data to withhold from the public would not be useful and would complicate comparisons among agencies. We did not intend that our recommendation allow agencies to decide without consulting with GSA, and we have clarified our recommendation accordingly. We continue to believe this recommendation, as clarified, is valid. As we reported, GSA currently withholds 15 variables—categories of data—for all federal assets, including the name of every federal building and structure. While this approach is consistent for all assets, it reduces the overall usefulness of the data by withholding information that federal agencies already make public. In addition, the ISC told us that the landholding agencies, not GSA, are in the best position to know what data about their assets are sensitive. We amended the recommendation by removing the reference to categories of data and adding that GSA work in consultation with agencies to determine what data to withhold. This change would create a consistent way for agencies to release useful data while withholding sensitive data for individual assets, a step they already take by withholding assets from the public database. GSA plans to work with the ISC and federal agencies to review related guidance and modify it as needed. We support these plans. In addition, DOI suggested in email comments that we revise our second recommendation to include coordinating with agencies to review V&V anomaly categories to better target incorrect data. Our original recommendation did not preclude coordination, and since we agree that such coordination would help improve the V&V process, we clarified the recommendation accordingly. We are sending copies of this report to the appropriate congressional committees, the Administrator of the General Services Administration, the Acting Secretary of Homeland Security, the Secretary of Energy, the Secretary of the Interior, Chair of the Federal Communication Commission, the Administrator of the National Aeronautics and Space Administration, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Costs related to the everyday functions of an asset Code to identify an installation (i.e. buildings, structures, land or any combination of these) Code to identify a part of an installation (i.e. buildings, structures, land or any combination of these) Building name or the name of an entire installation (such as an agency campus) Total number of full and part time federal employees Total number of full and part time contract employees Identifies whether an asset is part of a field office (any location that is not the headquarters location for the agency) In addition to the contact named above, Keith Cunningham (Assistant Director), Lynn Filla-Clark (Analyst-in-Charge), Melissa Bodeau, George Depaoli, James Duke, Rami Khalfani, Terence Lam, John Mingus, Joshua Ormond, Crystal Wesco, and Elizabeth Wood made key contributions to this report.", "summary": "The lack of reliable data on federal assets is one of the main reasons Federal Real Property Management remains on GAO's high risk list. In 2016, legislation required GSA to publish a single, comprehensive, and descriptive database of federal real property that would be available to the public. The database could be used for research and other potential applications. GAO was asked to study the public database. This report assesses (1) GSA's efforts to improve the reliability of FRPP's data and the public database, (2) the public database's completeness, and (3) the presentation of the data in the public database. GAO reviewed federal laws, documents, and data, including GSA's fiscal years 2017 and 2018 FRPP and public databases. GAO interviewed officials at GSA and from six federal agencies selected in locations with enough questionable data in the public database to analyze, among other things, and studied assets in Washington, D.C., Illinois, and New Mexico. GAO also interviewed selected stakeholders involved in federal real property management, such as real estate brokers. The General Services Administration (GSA) has worked in recent years to improve reliability of the Federal Real Property Profile (FRPP), which tracks federal real property assets. However, numerous errors in the database were carried into the public version. GSA extracted data from the FRPP's 398,000 civilian federal assets to create a public database to be used, for example, by researchers and real estate developers. However, GSA's data verification process did not address key errors. GAO found that 67 percent of the street addresses in the public database were incomplete or incorrectly formatted. For example, the database lists “Greenbelt Road” as the address for over 200 buildings at NASA's Goddard Space Flight Center, but the road stretches over 6.3 miles, thereby reducing a user's ability to locate specific buildings. The public database is not complete because GSA and selected agencies decided not to provide certain useful information. Specifically, GSA withheld assets' information without consulting those agencies managing the assets and allowed agencies to withhold information that is already publicly available. For example, GSA withheld the name “Goddard Space Flight Center” from the public database, but NASA's website lists this name and the Center's location. Unnecessarily withholding information limits the database's utility and undermines analysis. The public database's usefulness is further limited by how GSA presents the information. Because the database does not identify if an asset is part of a secure installation, the public does not know if assets, such as the unnamed buildings at Goddard, are accessible to the public. Unless GSA improves the public database's accuracy, completeness, and usefulness, its benefits may not be realized. GAO is making six recommendations to GSA, including improving the accuracy of the database, consulting with agencies on assets' information withheld from the database, and improving the public database's presentation. GSA agreed with five of the recommendations. GAO clarified the recommendation on withholding information on agencies' assets, to address GSA's comments.", "document_type": "gao"}
{"report": "Nursing homes and assisted living facilities provide important long-term care to vulnerable individuals in institutional or residential settings. Specifically, nursing homes provide care to elderly and disabled individuals, many of whom have physical and cognitive limitations requiring skilled nursing care. Assisted living facilities provide a residential alternative to nursing home care for individuals who prefer to live independently but need assistance to maintain their independence. Like nursing homes, they may provide residents with a variety of services to assist with activities of daily living, such as bathing and dressing, but the facilities are generally not licensed to provide 24-hour skilled nursing care and typically offer a more limited range of medical care. As we reported in our January 2018 report on CMS’s oversight of assisted living facilities under the Medicaid program, the demand for assisted living services, which offer the benefit of community living, is expected to increase as a result of the aging of the nation’s population, increased life expectancy, and older adults’ desire to remain in the community. Additionally, the cost of nursing home care for an individual generally exceeds the cost of assisted living facility services, further incentivizing a shift among consumers and payers to assisted living for elderly individuals, including those with increasingly complex health needs who would otherwise need nursing home care. Oversight of nursing homes is a shared federal-state responsibility. Federal law imposes both a comprehensive set of quality standards that nursing homes must meet to participate in the Medicare and Medicaid programs, and federal and state oversight responsibilities to enforce these standards. CMS, which is charged with implementing these standards and conducting federal oversight, contracts with state survey agencies to perform both routine inspections—known as standard surveys—and conduct investigations of elder abuse incidents, including complaints and facility-reported incidents. CMS provides guidance implementing statutory and regulatory requirements to protect residents from elder abuse in its State Operations Manual, which specifies requirements for reporting, investigating, and notifying law enforcement about elder abuse in nursing homes. CMS regional offices monitor state compliance with federal requirements for nursing home oversight. Generally, states establish their own oversight requirements for assisted living facilities. These facilities are largely overseen by state agencies within, for example, the state health or aging departments; however, when assisted living facilities provide services to Medicaid beneficiaries, they are also indirectly subject to CMS oversight through the agency’s oversight of state Medicaid agencies. As we have previously reported, states can provide Medicaid coverage for assisted living services under multiple authorities, but most commonly states use an HCBS waiver under section 1915(c) of the SSA. Under these waivers, CMS requires states to develop a quality assurance system that monitors beneficiary health and welfare—including tracking and responding to incidents that may cause harm to a beneficiary’s health and welfare, such as elder abuse. States must demonstrate to CMS that they are meeting these quality assurance obligations in their waiver renewal reports, typically submitted about 2 years before an HCBS waiver is scheduled to end. States must also report summary information annually to CMS on any health and welfare deficiencies occurring under their HCBS waivers. CMS regional offices oversee state compliance with waiver requirements. In addition to state survey agencies, state Medicaid agencies, and the agencies that license and regulate assisted living facilities, there are other entities charged with protecting nursing home and assisted living facility residents from elder abuse. These agencies’ roles and missions can vary by state. For example, Adult Protective Services (APS) programs in each state are generally responsible for identifying, investigating, resolving, and preventing abuse of older adults, and such programs may investigate complaints of elder abuse in nursing homes and assisted living facilities. Additionally, Medicaid Fraud Control Units and local law enforcement can also play a role in investigating elder abuse. Consequently, incident management may be coordinated among multiple separate agencies. Federal requirements include those for nursing homes and state survey agencies specific to reporting, investigating, and notifying law enforcement of elder abuse in nursing homes. For example, federal requirements specify the time frames within which nursing homes must report alleged elder abuse to state survey agencies and, similarly, specify time frames for state survey agencies to report elder abuse to CMS. In contrast, there are no similar requirements for assisted living facilities and, instead, states must establish their own policies to ensure the reporting and investigation of elder abuse in assisted living facilities covered by Medicaid. (See fig. 1 for federal requirements for reporting, investigating, and notifying law enforcement about elder abuse in nursing homes and assisted living facilities.) As illustrated in figure 1, there are key differences between federal requirements for reporting, investigating, and notifying law enforcement about elder abuse occurring in nursing homes compared to assisted living facilities. CMS officials told us that the difference in requirements between nursing homes and assisted living facilities reflects the different regulatory relationship between the agency and the two facility types. According to CMS officials, CMS has direct regulatory authority over nursing homes, but does not have direct authority over assisted living facilities. As noted, states are largely responsible for establishing their own policies for overseeing the reporting and investigation of abuse in assisted living facilities. (See app. I for profiles of selected states with HCBS waivers regarding elder abuse reporting, investigating, and notification.) Differences in federal requirements include the following: Reporting. Federal law and CMS policy define specific time frames for nursing home staff and state survey agencies to report incidents of abuse that occur in nursing homes, respectively, and CMS requires states to establish their own reporting time frames for assisted living facilities serving HCBS waiver participants. Nursing homes must ensure that allegations of elder abuse are reported to the state survey agency immediately, but no later than 2 hours after the allegation is made if the incident involves serious bodily injuries and within 24 hours if it does not. In addition, state survey agencies must report to CMS all complaints and certain facility-reported incidents of abuse through a computer-based complaint and incident tracking system and immediately alert CMS regional offices when an especially significant or sensitive incident occurs that attracts public or broad media attention. In contrast, CMS requires state Medicaid agencies that pay for care in assisted living facilities through HCBS waivers to establish their own required time frames for reporting incidents. Consequently, reporting time frames and processes for assisted living facilities can vary by state. For example, Connecticut requires incidents to be reported to the state Medicaid agency and Adult Protective Services within 2 business days, while Oklahoma requires that initial incident reports are submitted within 1 business day. Investigation. CMS prescribes investigation time frames and priority categories for incidents occurring in nursing homes and requires states to establish their own time frames and priority categories for incidents in assisted living facilities. CMS requires state survey agencies to assess reports of elder abuse in nursing homes and assign a priority investigation status based on the seriousness of the allegations. The required investigation time frames are tied to the priority status. For example, if the allegation indicates that there continues to be an immediate risk of serious injury, harm, impairment, or death of a resident unless immediate corrective action is taken, the survey agency must initiate an onsite investigation within 2 business days of receiving the report. CMS also requires nursing homes to have written policies and procedures for conducting internal investigations of suspected elder abuse and to submit findings from these investigations to the state survey agency within 5 business days of the incident. In contrast, CMS does not prescribe investigation time frames or define priorities for incidents occurring in assisted living facilities; instead, CMS requires that state Medicaid agencies with HCBS waivers establish their own policies for prioritizing reports of abuse and initiating investigations in assisted living facilities. Consequently, investigation time frames and prioritization can vary by state. For example, Connecticut does not specify a process for prioritizing incident investigations in its HCBS waiver, but officials told us the state requires the Medicaid program to initiate an investigation immediately; whereas South Dakota requires face-to- face contact with a victim within 24 hours if the incident is life or health-threatening. Family or Legal Guardian Notification Although the Centers for Medicare & Medicaid Services (CMS) requires facilities to notify a resident’s representative of a deterioration in the resident’s condition, CMS does not require nursing homes, assisted living facilities, state survey agencies, or state Medicaid agencies to notify a victim’s family or legal guardian of alleged elder abuse. However, CMS’s guidance for nursing homes notes the importance of family or legal guardian notification. Specifically, CMS guidance requires facilities to take actions to prevent further harm from occurring to a victim of alleged elder abuse and cites law enforcement notification, as well as family or legal guardian notification as examples of protective measures facilities may take to comply with that requirement. In addition, CMS requires states to develop a policy for notifying participants, family, or legal guardians of the findings of any critical incident investigations under its home and community-based services waiver program. CMS officials told us family or legal guardian notification is generally a state responsibility, and state officials told us that it is largely a facility responsibility governed by the facility’s policies. Some states include family or guardian notification requirements in state guidance on mandatory reporting of elder abuse. In interviews with stakeholders representing consumers and elder abuse investigators we learned that family notification can both help but also pose some privacy challenges. Law enforcement notification. Although federal law requires nursing homes to establish policies for ensuring that law enforcement is notified of elder abuse that occurs in their facilities, and CMS policy requires state survey agencies to notify law enforcement of substantiated findings of elder abuse that occur in nursing homes, these actions are not required when a similar incident occurs in an assisted living facility. Furthermore, CMS also does not require state Medicaid agencies to establish their own law enforcement notification requirements for assisted living facilities as part of the state’s HCBS waiver agreements. CMS and state officials told us that, generally, state agencies coordinate with law enforcement regardless of where the abuse occurs. Some states also require law enforcement notification as part of their state mandatory reporter laws. (See app. I for descriptions of selected state mandatory reporter laws.) For example, Connecticut requires Medicaid waiver program staff members to inform law enforcement of all suspected crimes, including abuse. Both GAO and HHS-OIG have identified, among other things, gaps in notifying law enforcement about abuse in nursing homes and recommended that CMS make changes to help ensure that nursing homes and state survey agencies notify law enforcement. In the course of our review, we found states may align certain requirements for investigating, reporting, and notifying law enforcement about elder abuse in assisted living facilities with federal requirements for nursing homes. Officials from all three selected states in our review told us they apply certain federal nursing home requirements and time frames to assisted living facilities when overseeing reports and investigations of alleged elder abuse. For example, officials from Oklahoma and South Dakota told us they align or are in the process of aligning time frames within which assisted living facilities are required to report incidents of elder abuse to state authorities with the time frames federally required for nursing homes, and said that alignment reduces confusion, especially among facilities that offer both types of residential care. Given its attenuated role in overseeing the reporting, investigation, and law enforcement notification of elder abuse in assisted living facilities, CMS officials told us the agency is taking steps to gather and disseminate best practices to help states better manage their response to elder abuse incidents in assisted living facilities. Specifically, officials told us that CMS has initiated an effort to more closely examine how states operate their incident management systems, which are used to track reports and investigations of elder abuse in assisted living facilities covered by their HCBS waivers. In May 2018, CMS surveyed states requesting information on how those states operate an incident management system for their HCBS waiver programs to track reports and investigations of elder abuse. CMS officials said they will take information learned through the survey as well as through on-site reviews that the agency has been conducting in five states since June 2019, to develop best practices and technical guidance on collecting and reporting critical incidents. We provided a draft of this report to HHS for review and comment. HHS noted that federal oversight of nursing homes and assisted living facilities is not directly comparable given the differences between HHS’s statutory authority to oversee both facility types. HHS noted that although CMS’s oversight of assisted living facilities is more limited, CMS works in partnership with states—through providing guidance, technical support, training, and oversight of states’ quality reporting—to ensure the safety of Medicaid beneficiaries in assisted living facilities. We recognize that CMS is operating in different statutory frameworks with respect to both nursing homes and assisted living facilities, and we have noted the distinction in our report. HHS further noted that CMS is undertaking efforts to strengthen federal oversight of nursing homes and states with HCBS programs, including through addressing our past recommendations. HHS comments are reproduced in appendix III. HHS also provided technical comments, which were incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Health and Human Services, appropriate congressional committees, and other interested parties. The correspondence is also available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. We reviewed state-level requirements for reporting, investigating, and notifying law enforcement about elder abuse in three selected states that cover services in assisted living facilities under their Home and Community-based Services (HCBS) waivers—Connecticut, Oklahoma, and South Dakota. These states are collectively responsible for safeguarding as many as 16,800 assisted living residents—2,751 of whom are covered by Medicaid—from elder abuse. All three states have mandatory reporting requirements that typically require various identified health care providers and facility staff to report suspected elder abuse to adult protective services or law enforcement, regardless of the setting in which the victim was abused or whether the victim is an HCBS waiver participant who would be protected under the Centers for Medicare & Medicaid Services (CMS) program requirements. Further, the states developed guidance for their HCBS programs that establishes additional reporting, investigation, and notification requirements—beyond their mandatory reporting law requirements—that caregivers, facilities, program staff, and state agencies must follow in response to incidents that occur to residents receiving services under Medicaid waiver programs. Selected information about assisted living facilities and state- level requirements for each of the three states is summarized in figures 2 through 4. GAO has issued reports reviewing the Centers for Medicare & Medicaid Services’ (CMS) oversight of the health and welfare of residents in nursing homes and assisted living facilities. For example, selected GAO reports from approximately the past 5 years included a review of the incidence of abuse in nursing homes and a review of what is known about the incidence of abuse in assisted living facilities. Reports often included key recommendations. (See table 1.) In addition to GAO’s audits of federal oversight of nursing homes and assisted living facilities, the Office of Inspector General within the Department of Health and Human Services (HHS-OIG) routinely audits a broad range of both the Centers for Medicare & Medicaid Services’ (CMS) and states’ oversight activities related to long-term care facilities. We identified three HHS-OIG reports issued between 2014 and 2018 that provide examples of HHS-OIG’s examinations of the reporting, investigation, and notification of law enforcement of elder abuse in nursing homes. (See table 2.) Although the specific scope of these reports varied, common findings included gaps in notifying law enforcement. For example, HHS-OIG examined Medicare claims data to identify cases where hospital staff had identified potential abuse and found that nursing homes failed to report many of these incidents to state survey agencies or notify law enforcement despite federal and state requirements and recommended that CMS provide training, clarify guidance, and track referrals to law enforcement. State auditors may also audit their states’ oversight of nursing homes and assisted living facilities. We identified nine reports issued by state auditors between 2014 and 2018 that examined their states’ oversight of elder abuse reporting and investigation across both settings. (See table 3.) Although the scope of individual reports across the states varied, state auditors identified instances of state entities not complying with state or federal requirements for a variety of reasons—including weaknesses in policies and procedures, resource constraints, and information management challenges—and recommended improvements. For example, in 2014 California state auditors found that thousands of complaint investigations—including over 300 classified as immediate jeopardy—were left open for almost a year, in part because the state did not specify time frames for completing investigations. John E. Dicken, (202) 512-7114 or dickenj@gao.gov. In addition to the contact named above, Karin Wallestad (Assistant Director); Jasleen Modi (Analyst-in-Charge); and Elise Pressma made key contributions to this report. Also contributing were Thomas Garloch, Cathy Hamann, Laurie Pachter, and Jennifer Whitworth.", "summary": "The federal government and states share responsibility for the health and welfare of about 1.5 million individuals—most of them vulnerable older adults—receiving long-term care in nursing homes and assisted living facilities covered by Medicare and Medicaid. For nursing homes, which provide skilled nursing care, federal law defines applicable quality standards and CMS provides guidance for nursing homes and the state survey agencies to help protect residents from elder abuse. For assisted living facilities, which provide assistance with activities of daily living in a residential setting, CMS defines the framework states must establish to oversee these facilities if covered under Medicaid. This includes requiring states to demonstrate to CMS that they are assuring quality including the obligation to protect against elder abuse. GAO was asked to review federal oversight of elder abuse reporting, investigation, and law enforcement notification in both nursing homes and assisted living facilities. In this report, GAO describes federal requirements for reporting, investigating, and notifying law enforcement about elder abuse in both types of facilities. GAO reviewed relevant laws and regulations and agency guidance, and interviewed CMS and state officials from three states selected for variation in HCBS waiver program size and geography. GAO also interviewed representatives from national stakeholder groups representing consumers, facilities, Medicaid directors, and abuse investigators. In comments on this report, HHS highlighted the distinct oversight frameworks for the two settings and noted that CMS is undertaking efforts to strengthen oversight. The Centers for Medicare & Medicaid Services (CMS) oversees the Medicare and Medicaid programs and is responsible for safeguarding the health and welfare of beneficiaries living in nursing homes and assisted living facilities. This includes safeguarding older residents from abuse—referred to as elder abuse. CMS delegates responsibility for overseeing this issue to state survey agencies, which are responsible for overseeing nursing homes. When assisted living facilities provide services to Medicaid beneficiaries, they are indirectly subject to CMS oversight through the agency's oversight of state Medicaid agencies. GAO found that there are specific federal requirements for nursing homes and state survey agencies for reporting, investigating, and notifying law enforcement about elder abuse in nursing homes. (See table below). For example, state survey agencies must prioritize reports of elder abuse in nursing homes based on CMS's specified criteria and investigate within specific time frames. In contrast, there are no similar federal requirements for assisted living facilities—which are licensed and regulated by states. Instead, CMS requires state Medicaid agencies to develop policies to ensure the reporting and investigation of elder abuse in assisted living facilities. For example, CMS requires that state Medicaid agencies establish their own policies and standards for prioritizing reports when investigating incidents in assisted living facilities. Officials from the three selected states in GAO's review said they apply certain federal nursing home requirements and investigation time frames for assisted living facilities when overseeing elder abuse.", "document_type": "gao"}
{"report": "The vast majority of the 42 railroads subject to the statutory mandate to implement PTC—including 30 commuter railroads, Amtrak, seven Class I and four Class II and III freight railroads—are implementing one of three types of PTC systems. These systems include the Interoperable Electronic Train Management System (I-ETMS), the Advanced Civil Speed Enforcement System II (ACSES), and Enhanced Automated Train Control (E-ATC). While these PTC systems are functionally similar, the technologies they use differ. For example, to determine a train’s location, ACSES and E-ATC rely on equipment embedded on the track while I- ETMS uses Global Positioning System information. ACSES and E-ATC both supplement existing train control systems to provide all required PTC functionality, while I-ETMS was designed as a new system to provide this functionality. As noted above, testing is one of the many steps to achieving full implementation. Through multiple stages of testing, which are summarized below, railroads must demonstrate that the PTC system meets functional requirements. Laboratory testing: locomotive and wayside equipment testing in a lab environment to verify that individual components function as designed. Field testing: includes several different tests of individual components and the overall system, such as testing each locomotive type to verify that it meets functional requirements and field integration testing—a key implementation milestone to verify that each PTC component is integrated and functioning safely as designed. Revenue service demonstration (RSD): an advanced form of field testing in which the railroad operates PTC-equipped trains in regular service under specific conditions. RSD is intended to validate the performance of the PTC system as a whole and to test the system under normal, real-world operations. Using results from field and RSD testing, combined with other information, host railroads must then submit a safety plan to FRA for system certification and approval. We previously reported that these safety plans have been up to 5,000 pages in length. Once FRA approves a safety plan, the railroad receives system certification, which is required for full implementation, and is then authorized to operate the PTC system in revenue service. According to FRA officials, the FRA may impose conditions to the PTC safety plan approval as necessary to ensure safety, resulting in a conditional certification. Interoperability is achieved when the locomotives of any host railroad and tenant railroad operating over the same track segment can successfully communicate with and respond to the other railroad’s PTC system, allowing uninterrupted movements over property boundaries. For example, when a locomotive enters another railroad’s territory as a tenant, it immediately needs information about the upcoming track—such as any temporary speed restrictions in place due to track work (see fig. 1). To achieve interoperability, railroads have to complete a series of steps including (1) additional installation work (such as installing equipment on a tenant railroad’s locomotives) and scheduling (such as coordinating with the relevant railroad to reach any needed agreements and identify dates for testing), (2) laboratory testing, (3) field testing, and (4) RSD or revenue service operations. Many railroads will complete much of the implementation for their own PTC systems, such as starting RSD on some or most of their track, before they begin to take steps to achieve interoperability with other railroads. However, a railroad can take steps to achieve interoperability with other railroads while simultaneously completing field testing or other stages of testing on its own PTC system. FRA is responsible for overseeing railroads’ implementation of PTC, and the agency monitors progress and provides direct assistance to railroads implementing PTC. For example, each railroad had to develop an FRA- approved PTC implementation plan that includes project schedules and milestones for certain activities, and a railroad is required to report quarterly and annually to FRA on its PTC implementation status relative to its implementation plan. FRA also provides technical assistance to railroads, addresses questions, and reviews and approves railroads’ documentation, including test and safety plans. FRA has a national PTC project manager, designated PTC specialists in the eight FRA regions, and approximately a dozen engineers, test monitors, and other staff responsible for overseeing technical aspects of implementation. FRA also has oversight tools, which includes authority to impose civil penalties when a railroad fails to meet certain statutory PTC requirements. Since 2017, FRA reports that it has assessed nearly $400,000 in civil penalties against railroads that failed to comply with their implementation plan milestones or reporting requirements. Since the end of 2018, some railroads have reported making progress on testing and implementation of their own PTC systems. Figure 2 shows the 42 railroads’ reported progress by PTC implementation stage. Six railroads—two Class Is and four commuters—reported to FRA that they had implemented PTC on all of their own territories but had not completed interoperability as of March 31, 2019, and almost all these railroads reported being in this stage at the end of 2018. In addition, as of March 31, 2019, no additional railroads beyond the four that were complete at the end of 2018 reported reaching full implementation. Nearly all railroads still implementing PTC plan to reach full implementation in the last quarter of 2020, based on our analysis of railroads’ extension requests. Few railroads reported moving into RSD during the first quarter of 2019, and the extent of RSD testing being conducted by railroads in this stage varied considerably. Of the 19 railroads that reported RSD testing on some portion of their own track as of March 31, about half (9 of 19) reported RSD testing on more than 75 percent of their total route miles, while about a quarter (5 of 19) reported RSD testing on less than 25 percent of their total route miles. RSD testing also varied between Class I railroads and commuter railroads. On average, the 5 Class I railroads in this stage reported RSD on 86 percent of route miles, while commuter railroads reported an average of 39 percent of route miles in RSD. Moreover, based on our analysis, 11 railroads—7 commuters and 4 Class II and III railroads—reported that they remained in field testing as of March 31, 2019. Similar to railroads in RSD testing, the extent of field testing reported by railroads varied. Of the 11 railroads in field testing, most (7) reported field testing on the majority or all of their route miles, whereas 4 railroads—all commuters—reported conducting field testing on less than half of their route miles. Based on railroads’ responses to our questionnaire, railroads’ PTC implementation status did not change significantly as of May 31, 2019; two additional railroads—both commuters—began RSD testing on some portion of their track, and one commuter railroad began field testing. As of March 31, 2019, 11 of the 31 host railroads that must have interoperable PTC systems reported to FRA that they had achieved interoperability with at least 1 of their tenant railroads. Collectively, of the 227 unique host-tenant relationships that require interoperability, FRA reported that railroads had achieved interoperability for 38 (17 percent) of these relationships. The number of tenants each railroad must work to achieve interoperability with ranges from 1 to 31 railroads, based on railroad reports to FRA. For example, Class I railroads, as host railroads, average about 18 tenants, while commuter railroads average about 3 tenants. A railroad does not generally start work to achieve interoperability with all the railroads it interoperates with at once, according to FRA; instead a railroad will prioritize its interoperability work. For example, representatives from one Class I railroad we interviewed said it prioritized achieving interoperability in the following sequence: first with commuter-railroad tenants given the need to ensure passenger safety; second with other Class I railroads given the high total miles of track they share; and finally with smaller Class III railroads. In addition, a railroad may be in multiple interoperability steps (e.g., installing, testing) with different tenants at the same time. FRA counts a relationship as having achieved interoperability if the tenant is operating PTC on all of the host’s track miles. This binary measure for interoperability—that is, achieved or not—does not describe the extent to which railroads have started work on interoperability or, according to representatives from two railroads we interviewed, reflect when interoperability has been achieved along most but not all of its host’s track. Railroads reported to FRA that they had begun work on interoperability for more than 90 percent of the remaining host-tenant relationships that need to achieve interoperability. In particular, based on their quarterly reports, railroads were installing for 82 host-tenant relationships and testing for 89 host-tenant relationships as of March 31, 2019. Overall, the status of interoperability work did not vary much among Class I, commuter, and Class II and III railroads. FRA officials and others we spoke with could not provide an estimate of how long it takes on average for two railroads to complete the individual steps to achieve interoperability. Representatives from industry associations we interviewed said that it can vary. An FRA specialist we interviewed agreed, explaining that interoperability field testing, for example, varies based on track availability. One railroad might complete testing in 4 days while another railroad might need weeks because it can only test at specific times. In its quarterly reports, FRA asks host railroads to provide the scheduled date for completing interoperability testing with each tenant railroad. As of March 31, 2019, seven railroads reported that they did not anticipate completing interoperability testing with at least one tenant until the last quarter of 2020. In responding to our May 2019 questionnaire, most railroads reported that vendor and software issues remain major or moderate challenges for PTC implementation. As part of our ongoing work related to PTC, we have reported that railroads have faced challenges associated with the limited number of vendors that design PTC systems, provide the software and hardware, and conduct testing. However, as representatives of half of the railroads we interviewed emphasized, vendor and software issues are more acute now because as the 2020 deadline nears, less time remains to address these issues and associated delays. Software and vendor issues can be interrelated as a small pool of vendors develop and update the software that supports railroads’ PTC systems. Representatives from several railroads and FRA specialists we interviewed said that software issues routinely arise in lab testing, field testing, and RSD that require vendor revisions before a railroad’s PTC implementation can continue. For example, representatives from one railroad said that existing software defects affecting its PTC system must be addressed and a new version of the software is needed before they can start RSD. They added that they had no control over this process, as they must rely on the vendor to provide reliable software. Representatives from this railroad also noted that resolving software issues is often not entirely within a railroad’s control due to the need for vendor support, in contrast to some earlier challenges leading up to the 2018 deadline, where, for example, the railroad itself had more control as it was installing equipment and could more clearly track progress. Moreover, the limited supply of vendors and high demand for services as railroads work simultaneously to implement PTC by the 2020 deadline continue to pose problems. For example, representatives from one railroad said their vendor has consistently had issues meeting milestones and delivering on its commitments. Representatives from a small railroad said they had limited internal resources to implement PTC, making the railroad’s progress heavily reliant on its vendor. Representatives from two other railroads and FRA officials also highlighted implementation delays caused by recalls for some locomotive equipment, which has caused additional work for railroads as well as the vendor. Specifically, the equipment had to be removed, sent in for repair, and then re-installed. More than half of the railroads implementing PTC also responded to our questionnaire that interoperability was a major or moderate challenge. Railroads said that interoperability can be complicated by software issues and coordinating host and tenant railroad schedules, when asked to describe the biggest challenges to achieving interoperability. Fifteen railroads specifically mentioned software issues, and representatives from several railroads noted that interoperability will require reliable software. For example, one railroad reported that certain software functionality remains to be developed, tested, and implemented to facilitate interoperability and to address software reliability issues that have caused system disruptions. Also, 14 railroads noted that scheduling time with other railroads to begin interoperability testing can be cumbersome and time consuming. For example, several railroads that we interviewed and that responded to our questionnaire said that scheduling can be complicated by whether other railroads have made enough progress on their own PTC implementation to begin work on interoperability. According to FRA officials, interoperability challenges also differ across PTC systems and geographic areas. Below, we use the Northeast Corridor and the Chicago metropolitan area—where most railroads are implementing ACSES and I-ETMS, respectively—to illustrate the challenges faced in working to achieve interoperability. However, railroads in other areas or implementing other PTC systems may face some of these same challenges or face additional different challenges. Over a dozen railroads operating on the Northeast Corridor and in the surrounding area are required to implement PTC. The Northeast Corridor runs from Washington, D.C., to Boston, Massachusetts, and Amtrak predominantly owns track on the corridor. Eight commuter railroads, Amtrak, and most freight railroads are implementing a form of the ACSES system on at least a portion of their equipment and track. In some cases, railroads in the Northeast will be operating two different PTC systems concurrently on the same track, which will add to the complexity of interoperability, according to FRA. Examples of interoperability challenges faced in the Northeast include: Software issues. PTC software presents particular challenges in the Northeast because software is being supplied by multiple vendors and has been developed to accommodate railroads’ existing systems that have different configurations. Therefore, according to FRA officials, ACSES does not have a common set of requirements or specifications. Also, even if two railroads use the same vendor for their locomotive equipment or software, each railroad may use a different version of software. In addition, representatives from two railroads that operate in the Northeast told us they built different software functionality into their PTC systems to accommodate their own operations, so additional work is needed to resolve such differences to achieve interoperability. In light of these software issues, representatives from one industry association and one railroad we interviewed said that Northeast Corridor railroads are discussing creating a software management process to aid interoperability. Boundary issues. A train needs to seamlessly operate PTC when it crosses the boundary between two railroads’ territories, as previously described. According to a rail industry association, as of June 2019, there are about 20 boundaries on the Northeast Corridor where more work is needed to ensure seamless operation. FRA officials and one industry association said boundary issues are complex and time- consuming to resolve but not insurmountable. For example, FRA officials said a railroad could install its own equipment such as transponders and wayside devices across the boundary to create an overlap between their system and that of the other railroad. Securing PTC wireless communication. FRA requires that PTC wireless railroad communications be encrypted. However, a solution that aims to encrypt all PTC wireless communication and data transmittal among railroads operating ACSES in the Northeast is currently in lab development. In August 2016, Amtrak received a grant from FRA to create this solution for ACSES. Amtrak originally planned to implement this solution in December 2018, but Amtrak has experienced delays and currently estimates that it will implement the solution by January 2020. However, Amtrak has reported several risks that it will need to overcome to meet this implementation deadline. Further delays could affect railroads’ ability to fully implement PTC in the Northeast by the December 2020 deadline. FRA noted it will continue to monitor and support the railroads as they implement security measures in the Northeast. Ten I-ETMS railroads that operate in the greater Chicago metropolitan area received extensions to implement PTC. Throughout PTC implementation, FRA, industry associations, and railroads have identified Chicago as a place where interoperability would be challenging due to the dense freight, passenger, and commuter operations in the area. Examples of such challenges include: Software issues. According to FRA and railroads we interviewed, software issues have slowed interoperability work by railroads implementing I-ETMS. The underlying problem is the memory available on the locomotive equipment, which is needed to store its railroad’s track data, according to FRA and railroads we interviewed. To be interoperable, the locomotive equipment also needs to store and exchange multiple railroads’ track data, causing the memory to fill up very quickly. According to railroad representatives, memory limitations for I-ETMS locomotive equipment prohibited railroads with large track data files—mainly the Class I freight railroads—from being able to interoperate. The vendor for this equipment has been working on a software solution for this problem, and according to a few railroads we interviewed, the vendor delivered an interim software solution in March 2019 that allowed the four largest Class I railroads to achieve interoperability. However, this software was delivered 7 months later than initially planned, and an additional software solution is still needed to allow the locomotive equipment’s memory to store the data of all railroads operating I-ETMS, according to representatives from two railroads and an industry association we interviewed. Other technical issues. Railroads in the Chicago area conducted modeling to help ensure that sufficient communications capacity (e.g., spectrum and radio capacity) would be available to support PTC interoperability in the region. According to one industry association, while actual PTC operations in the area are minimal right now relative to full expected operations, railroads must continue to monitor the communications capacity as more railroads progress with their own PTC implementation and start to interoperate. For example, railroads may have to re-engineer their radio networks, such as re-routing certain communications through different radio towers and other network connections, if issues are subsequently identified. Scheduling interoperability work with other railroads. Within the Chicago area, the total number of railroads and the number of railroads that have to be interoperable on a single line complicates interoperability. Chicago is the busiest rail hub in North America and handles one-fourth of the nation’s freight rail traffic. Nearly 500 freight trains and over 700 passenger trains travel through the area on tracks owned by several different railroads every day. For example, one commuter railroad, for one of its lines, operates over track owned by four host railroads that alternates with its own track. Achieving interoperability for this line will involve sequencing and scheduling with multiple railroads to activate PTC along the entire line, including across the numerous boundaries between different railroads’ territories, according to representatives from that railroad. According to one FRA specialist, work to achieve interoperability in the Chicago area will ramp up in late 2019 or early 2020. As a result, many railroads will have to coordinate schedules to sequence interoperability work across the dozens of host-tenant relationships in the area. FRA officials told us that the agency continues to provide assistance to railroads on interoperability and to support railroads through the testing process. In summer 2019, FRA began an effort to meet with all freight, non-Class I tenant railroads that have to be interoperable with host railroads required to implement PTC. FRA officials said they will use meetings with these 72 individual tenant railroads to discuss PTC requirements and review the railroads’ plans for implementing PTC with their host railroads. FRA officials said they have also continued to meet regularly with railroads still in field testing or starting RSD on their own PTC systems. For example, FRA officials said the agency meets weekly or monthly with each railroad that has not yet initiated RSD to provide targeted technical assistance to resolve any issues. FRA and representatives from one railroad also told us that FRA has met with vendors to discuss specific equipment or software issues and to stress the importance of resolving these issues. FRA also participates in meetings held by the railroad industry’s PTC working groups, including those focused on the Northeast Corridor and Chicago area, as needed. In addition, FRA officials told us that they are working with industry to improve the safety plan review process. Specifically, according to a June FRA presentation, FRA is working with two railroads and an industry association to create templates for streamlined, more consistent safety plans for two types of PTC systems—I-ETMS and E-ATC. The goal of the template is to reduce the burden on both railroads and FRA by using a shorter format and, where possible, relying on standardized system documents. FRA officials anticipate that the templates will be ready for other railroads to use in fall 2019. In addition, FRA has contracted for help in reviewing safety plans. However, representatives from four railroads and two industry associations we interviewed noted that they remained concerned about the amount of time it has taken FRA to review safety plans. FRA reported in February 2019 that it took on average 331 days to review a safety plan. While it is too early to determine the effect of FRA’s efforts to improve the safety plan review process, much work remains for FRA in the next 18 months. According to FRA, 23 railroads will be submitting safety plans in the next 12 months. While FRA has conditionally certified 13 PTC systems as of March 31, 2019, these railroads, too, are required to continue to work with FRA to provide additional documents to respond to FRA’s conditions. Some of these railroads also plan to resubmit safety plans for FRA to review, hoping to receive an unconditional certification before the December 2020 deadline. In March 2018, we reported that railroads had expressed a need for additional clarification about applying for an extension and that FRA could provide more consistent information to railroads. We recommended that FRA identify and adopt a method for systematically communicating extension-related information to railroads. In 2018, FRA held three symposiums for railroads to consistently communicate information to help railroads prepare to qualify for an extension and to understand what was required to have a fully implemented PTC system. FRA held two similar sessions in February and June 2019. Representatives from most of the railroads we interviewed (six of eight) said they have been happy with the communication with FRA, via these sessions as well as regular meetings with FRA’s PTC field specialists and other staff. For example, representatives of two railroads said it was helpful to have the FRA Administrator attend the sessions with railroads and talk directly to railroad representatives. In addition, clarity of information from FRA was the lowest rated challenge in response to our questionnaire, with 29 railroads reporting this as a minor challenge or not at all a challenge. While FRA has made improvements, the extended 2020 deadline for full PTC implementation is less than 18 months away, and FRA and railroads have substantial work to complete and challenges to address before that deadline. Moreover, unlike the 2018 deadline, no additional extensions are available beyond December 2020. In March 2018, we recommended that FRA develop an approach to use the information it gathers on railroads’ PTC implementation progress to prioritize the allocation of resources to address the greatest risk. FRA agreed with this recommendation, and while FRA officials have described testing and interoperability as areas of focus in 2018 and 2019, they have not articulated or demonstrated how, within these broad areas, they are monitoring risk and prioritizing resources. For instance, FRA plans to meet with all 72 tenant railroads in over 30 meetings rather than use the data it collects from host railroads to target this outreach. In addition, while FRA will have to review dozens of new and resubmitted safety plans in the coming months, FRA officials have not identified how they will prioritize these reviews relative to other reviews (e.g., other documentation that railroads submit as they continue testing on their own systems and for interoperability). According to FRA, it has communicated to railroads in industry-wide meetings that conditional certification for a PTC system is generally sufficient to meet the statutory requirement for full implementation; FRA noted this would only not be sufficient if a railroad’s PTC system did not otherwise meet the technical requirements in regulations and one or more of the conditions related to such non-compliance. However, representatives from two railroads we interviewed also said it was unclear whether conditional certification would be enough for a railroad to comply with the 2020 deadline, and uncertainty remains about which conditions must be addressed to meet the statutory requirement for full implementation. Related to system certification, representatives from three railroads and one industry association we interviewed also said FRA still needed to clarify how it would handle situations where a host or tenant railroad is not fully implemented by the 2020 deadline. Although the FRA Administrator has publicly said he will enforce the implementation deadline (which is December 31, 2020, for most railroads) and recommend assessing the maximum civil penalty against a railroad that did not meet its deadline, FRA has not clarified if this would apply in situations where a host or tenant relationship affects another railroad’s implementation. We continue to see value in FRA developing a risk-based approach to allocating its limited resources and will continue to monitor FRA’s actions on this recommendation. Going forward, FRA will also need to transition to overseeing PTC as a routine part of railroad operations after the 2020 deadline. Similarly, railroads will need to transition from implementation—largely done by contractors—to operating and maintaining their own PTC systems. Several railroads, in response to our questionnaire, said that they anticipate difficulties funding ongoing operations and maintenance as well as managing software and other updates. Therefore, December 31, 2020, represents not only the deadline for full PTC implementation but also a point after which railroads and FRA will face a new operational and oversight environment. Chairman Wicker, Ranking Member Cantwell, and Members of the Committee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Susan Fleming, Director, Physical Infrastructure at (202) 512- 2834 or FlemingS@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Susan Zimmerman (Assistant Director); Katherine Blair Raymond; Delwen Jones; Emily Larson; Joanie Lofgren; Shannin G. O’Neill; Josh Ormond; Madhav Panwar; Marcus Robinson; Maria Wallace; Crystal Wesco; and Elizabeth Wood. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Forty-two railroads are currently subject to the statutory mandate to implement PTC, a communications-based system designed to automatically slow or stop a train that is not being operated safely. Railroads were required to implement PTC by December 31, 2018, but would receive extensions up to December 31, 2020, if specific statutory requirements were met. GAO was asked to review railroads' PTC implementation progress. This statement discusses (1) railroads' implementation progress and any related implementation challenges and (2) FRA's plans for overseeing railroads' implementation. GAO analyzed railroads' most recent quarterly reports covering activities through March 31, 2019; received responses from all 42 railroads on a brief questionnaire; and interviewed officials from FRA and 8 railroads, selected to include variation in implementation status and type of railroad, among other criteria. Amtrak, commuter railroads, and freight railroads continue to make progress implementing positive train control (PTC), but significant work remains to achieve interoperability among the railroads' individual PTC systems. Since the end of 2018, many railroads reported making progress on testing and implementation of their own PTC systems. Four railroads reported reaching full implementation as of March 31, 2019, the same number in this stage at the end of 2018. However, many railroads remained in earlier stages of implementation, such as the 11 railroads that reported being in field testing. Nearly all railroads plan to complete full PTC implementation in the last quarter of 2020. Full implementation with interoperability is achieved when the PTC system on the locomotive of a “tenant” railroad and the PTC system of a “host” railroad whose track is being used can successfully communicate, allowing uninterrupted movements over property boundaries. As of March 31, 2019, 11 of the 31 host railroads that must have interoperable PTC systems reported that they had achieved interoperability with at least 1 of their tenant railroads. Collectively, 38 of the 227 unique host-tenant relationships that require interoperability have been completed (17 percent), according to the Federal Railroad Administration (FRA). Most railroads reported to GAO that vendor and software issues were currently major or moderate challenges for PTC implementation. Over half of railroads also reported that interoperability was a major or moderate challenge, and can be complicated by software issues and coordinating host and tenant schedules, among other issues. For example, one railroad said that certain software functionality still had to be developed, tested, and implemented to address reliability issues and facilitate interoperability. FRA continues to provide assistance and support to railroads on PTC interoperabilty and the testing process, but workload challenges for the agency persist. FRA will continue to face a substantial workload through 2020 as it oversees railroads' PTC implementation and reviews documents, including lengthy safety plans required for railroads to obtain PTC system certification. While FRA officials have described supporting interoperability and testing as areas of focus, they have not demonstrated how, within these broad areas, they are monitoring risk and prioritizing resources, as GAO recommended in March 2018. GAO continues to see value in FRA developing a risk-based approach to allocate its resources to oversee PTC. In March 2018, GAO recommended FRA take steps to systematically communicate information to railroads and to use a risk-based approach to prioritize agency resources and workload. FRA concurred with these recommendations. FRA has taken actions to systematically communicate information to railroads. GAO will continue to monitor FRA actions with regard to allocating agency resources to oversee PTC.", "document_type": "gao"}
{"report": "In February 2011, Boeing won the competition to develop the Air Force’s next generation aerial refueling tanker aircraft, the KC-46. The KC-46 is to be equipped with subsystems that allow for two types of refueling—(1) a refueling boom that is integrated with a computer-assisted control system, and (2) a permanent hose and drogue refueling system. This dual refueling capability is an enhancement over prior tanker aircraft because it enables the KC-46 to use boom refueling for Air Force aircraft and drogue refueling for Navy or allied aircraft on a single flight. The majority of legacy tankers, such as the KC-135s, were configured for only one of these types of refueling and had to land and be reconfigured to use the other refueling system. During boom refueling, an operator on the KC-46 tanker aircraft extends the boom—a rigid, telescoping tube—and inserts it into a receptacle on the aircraft being refueled. The KC-46 also has a remote vision system, which consists of a display, cameras, and computer processors, in lieu of a window that legacy tankers use. The system allows operators to observe the position of the boom and the receiving aircraft, and to reposition the fuel delivery system to facilitate refueling. In contrast, during drogue refueling, an operator uses the hose and drogue system— comprised of a long, flexible refueling hose and a parachute-like metal basket that provides stability—to provide fuel to receiver aircraft. Drogue refueling is available via the centerline drogue system in the middle of the tanker aircraft or via wing aerial refueling pods located on each wing. While refueling with the drogue or wing aerial refueling pods, the operator uses the remote vision system to identify when to extend or reel in the hoses. The wing aerial refueling pods can be used for simultaneous refueling of two Navy or allied aircraft—an enhanced capability that only 20 of the 414 KC-135 tankers currently have the capability to do. Figure 1 shows the boom and drogue refueling subsystems on the KC-46. The KC-46 tanker is a commercial derivative aircraft that is based on Boeing’s commercial 767 aircraft. To convert a 767 to a KC-46 tanker, Boeing modified the aircraft design in two phases. In the first phase, Boeing changed the design of the 767 to include a cargo door, new fuel tanks, and an advanced flight deck display borrowed from the 787 aircraft. This baseline non-military aircraft is called the 767-2C and is being built on Boeing’s existing 767 production line. In the second phase, Boeing added military systems to the 767-2C and brought it to a KC-46 configuration in a separate Boeing modification facility. The completed KC-46 aircraft are then taken to a test and delivery center for Air Force acceptance. By using a commercial derivative aircraft, the Air Force intended to avoid the long process and costs associated with designing, testing, and evaluating a new aircraft. It also wanted to reap the benefits of decades of reliability upgrades Boeing made to the aircraft for commercial customers, an established commercial infrastructure for spare parts, and maintenance and training data needed for sustainment that have been validated and verified by the commercial industry, among other things. According to an Air Force Policy Directive in place at the time of contract award, programs that are based on commercial derivative aircraft are required to achieve Federal Aviation Administration certification to the maximum extent practical. The Air Force went further and required the contractor to exhaust all possible solutions to obtain Federal Aviation Administration certification on both commercial and military-unique parts—including the boom, centerline drogue system, and wing aerial refueling pods—before seeking military certification. The Federal Aviation Administration previously certified the airworthiness of Boeing’s 767 commercial passenger airplane (referred to as a type certification), and in December 2017, awarded the amended type certificate for the 767-2C aircraft to Boeing. The amended type certificate allowed Boeing to use the 767-2C aircraft as the baseline non-military aircraft for the KC-46. Then, in September 2018, the Federal Aviation Administration certified the design of the KC-46 with a supplemental type certificate. The supplemental type certificate signifies the Federal Aviation Administration’s approval of the KC-46’s airworthiness, including mission systems such as its aerial refueling components. According to program officials, the Air Force granted a limited duration airworthiness certification for the KC-46 in November 2018 to support the initial fielding, which they said is common for new aircraft. The Air Force is continuing testing to obtain a military type certification from the Air Force Engineering Directorate, expected in several years. See figure 2 for a depiction of the conversion of the 767 aircraft into the KC-46 tanker with the boom deployed and the Federal Aviation Administration’s airworthiness certificate needed at each stage. During development, Boeing is expected to prove the aircraft’s design and demonstrate that the aircraft performs as expected. This type of testing is referred to as developmental testing. This testing was originally planned to occur within a 15-month window starting in early 2015 and ending in 2016. Initial operational test and evaluation—expected to occur after developmental testing and referred to in our report as operational testing—is conducted on production aircraft, or production representative articles. During this testing, the Air Force determines whether systems are operationally effective and suitable to support a full-rate production decision. The Air Force obtained a military flight release in November 2018, which allows it to start operational testing. To support operational testing, the Air Force is undertaking testing to certify the KC-46 to refuel various receiver aircraft, such as the F-15 fighter and B-52 bomber. After the first four KC-46 aircraft are delivered and two receiver aircraft are certified for refueling, the Air Force will begin operational testing. The Air Force awarded Boeing a fixed-price incentive (firm target) contract to develop the KC-46, which includes the design, manufacture, and delivery of four test aircraft. Barring any changes, the contract specifies a ceiling price of $4.9 billion for Boeing to develop the first four aircraft. Once that price was reached, Boeing would assume responsibility for all additional costs for developing those aircraft. The Air Force used a fixed-price incentive development contract because KC-46 development was considered to be a relatively low-risk effort to integrate mostly mature military technologies onto an aircraft designed for commercial use. The contract limits the government’s financial liability and provides the contractor incentives to reduce costs to earn more profit. The contract specifies a 60/40 incentive ratio for sharing savings in the event of underruns, or sharing costs in the event of overruns in relation to the target cost. The government’s share is 60 percent, while Boeing’s is 40 percent. Cost sharing ends when the contract price reaches the $4.9 billion ceiling. Thereafter, Boeing is responsible for all additional costs associated with the overruns. The contract also specifies that Boeing must correct any deficiencies and bring development and production aircraft to the final configuration at no additional cost to the government. In addition, the contract includes options for Boeing to manufacture 175 aircraft with firm-fixed-price contract options for the first two production lots, and options with not-to-exceed fixed prices for production lots 3 through 13. For purposes of this report, a production lot refers to a set number of aircraft that must be built and delivered in a given time frame and procured with a specific year of budget funding. The original contract required Boeing to deliver 18 operational aircraft, nine sets of wing aerial refueling pods, and two spare engines by August 2017. The Under Secretary of Defense for Acquisition, Technology and Logistics approved the KC-46 program to enter low-rate initial production in August 2016. Originally, the Air Force planned for the first two production lots to be low-rate production lots. The 19 aircraft associated with these two lots, or 11 percent of the 175 production aircraft, were to be built concurrent with developmental flight testing. The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics approved additional low-rate production lots—lots three through five—in 2016 and 2017 to avoid interrupting the planned production of additional aircraft. As we have reported over the past several years, Boeing had problems developing the aircraft, which resulted in schedule delays and a decision by Boeing and the program office to separate the delivery of the first 18 aircraft from the delivery of the first nine sets of wing aerial refueling pods. As of March 2019, the Air Force has exercised options for the first four low-rate production lots, for 52 aircraft totaling about $7.8 billion. As a result, the number of aircraft being produced concurrent with developmental flight testing has increased to 52 aircraft, or 30 percent of the total number Air Force expects to purchase. Traditionally, the Department of Defense tracks concurrency to determine financial risk to the federal government; however, in this case, due to the terms of the development contract, the government’s liability was limited to sharing in cost overruns only up to the contract’s ceiling price. Figure 3 shows the number of aircraft the Air Force plans to procure in each lot. The KC-46 program’s cost estimates have remained lower than initially estimated, consistent with our past reports. The KC-46 program’s total acquisition cost estimate is currently about $43 billion, or about $9 billion lower than the original 2011 estimate. The Air Force was able to decrease its cost estimate in large part because funds set aside for potential design changes were not needed. After a 3-year delay from the original plan, the Air Force began conditionally accepting the first seven KC-46 aircraft in early 2019. The KC-46 program’s total acquisition cost estimate remains lower than the initial estimate, consistent with our April 2018 report. As of January 2019, the Air Force estimates that the total program acquisition cost for the KC-46, which includes development, procurement, and military construction costs, will be about $43 billion. This is about $9 billion, or 17 percent, less than the original estimate of $51.7 billion made in 2011. Correspondingly, the average acquisition cost of each aircraft has also decreased by 17 percent because aircraft quantities have remained the same. Table 1 provides a comparison of the initial and current quantity and cost estimates. The estimates include, among other things, the expected costs of the development and procurement contracts awarded to Boeing, government test and evaluation costs, program office expenses for advisory and assistance services from support contractors, as well as contingency funding that might be needed to address the potential risk of requirements changes or other unexpected issues. Overall, the Air Force decreased its development and procurement cost estimates by about $1.3 billion and $6 billion, respectively. As we have previously reported, the main reason for the decrease is it has not needed the large amount of contingency money the Air Force included in the initial estimates for possible requirements changes. Military construction cost estimates also decreased by about $1.4 billion as the Air Force decided, for example, to reuse existing facilities at its operating bases rather than build new ones. In contrast, as of February 2019, Boeing representatives estimate that costs to complete development have increased to about $6.2 billion, or about $1.3 billion over the contract ceiling price of $4.9 billion, due to development problems. Specifically, Boeing experienced problems related to wiring the aircraft, design issues with the fuel system components, a fuel contamination event that corroded the fuel tanks of one of the development aircraft, and test delays. According to the fixed- price incentive contract, the government is generally not responsible for these additional costs to the extent they exceeded the ceiling price of the development contract. The Air Force conditionally accepted the first seven KC-46 production aircraft between January and March 2019, about 3 years later than originally planned, with three critical deficiencies related to the refueling subsystems. Although the federal government generally has no obligation to accept work that does not meet contract requirements, program officials told us that the Air Force negotiated minimum specifications under which it would begin conditionally accepting aircraft. Officials told us that among other benefits, conditionally accepting these aircraft provides the Air Force additional military capability and the aircraft can be used to start operational testing. These aircraft are among the 18 aircraft required by the original contract. As of April 2019, Boeing was producing the remaining 45 additional aircraft associated with the first four low-rate initial production lots. Some of the aircraft just started production on Boeing’s 767 production line. Others are further along and being modified to become KC-46 aircraft in a separate facility, or are being tested and taken to the delivery center for Air Force acceptance. Still others are in storage, either waiting to be transferred to the KC-46 modification center to be retrofitted with the latest wiring configuration or transferred to the delivery center to prepare for Air Force acceptance. Figure 4 shows where these 45 aircraft are in Boeing’s production and delivery process, along with the seven aircraft already delivered. Boeing is not expected to meet its most significant delivery requirement so far until mid-2020, 34 months after originally planned and almost 20 months later than we found in April 2018. Specifically, program officials anticipate that the Air Force will accept the first 18 aircraft by August 2019, and nine sets of wing aerial refueling pods by June 2020—which together with two spare engines constitute the contractual delivery requirement contained in the development contract. According to program officials, Boeing continued to have difficulty providing design documentation needed to start Federal Aviation Administration testing for the wing aerial refueling pods over the past year, which caused the additional delays beyond what we reported last year. Figure 5 shows the original and current delivery schedules for completing the development contract requirement. In February 2019, the Air Force stopped accepting KC-46 aircraft from Boeing because it had identified foreign object debris, including tools, in aircraft it had already accepted, as well as in the aircraft that were in the final stages of acceptance. Boeing issued a corrective action plan outlining steps the company needed to take to improve its foreign object debris identification and prevention activities before the Air Force would accept additional aircraft. Some of the steps included conducting daily inspections of each aircraft for foreign object debris, having Boeing production personnel submit lost tool reports to their superiors, and developing strategies for containing the debris issue, such as only taking the exact amount of small parts needed for an individual job in the aircraft build. The Air Force began accepting aircraft again after Boeing took steps to address the problem. However, in March 2019, Boeing found additional foreign object debris as it was conducting its newly implemented daily inspections and the Air Force suspended deliveries again. Boeing implemented additional corrective actions to the Air Force’s satisfaction and, as of April 2019, the Air Force has authorized the resumption of KC-46 deliveries. Program officials stated that Boeing is responsible for the costs to inspect and remove foreign object debris from aircraft that have already been accepted and that are in various stages of the Boeing manufacturing process. Because of the delivery delays to date and other factors in the existing tanker fleet, an Air Mobility Command official said leadership is currently planning to fly and maintain some legacy KC-135 tankers longer than planned until the KC-46 is available to conduct missions. According to the official, the Air Force plans to reallocate $57 million in fiscal year 2020 funds from the KC-46 program to the KC-135 program to support this decision. The funding would cover the cost to fly and sustain some KC- 135 aircraft above what the Command had planned, including the associated personnel costs. Air Mobility Command officials said that decisions about retaining some legacy KC-135 aircraft will be reviewed annually thereafter. If these aircraft are retained, funding would be reallocated from the KC-46 program to support the decision. The program continues to expect that the KC-46 aircraft will ultimately meet its high-level system performance goals, such as those related to aerial refueling and operational availability. However, the Air Force and Boeing expect that the critical deficiencies that could affect the aircraft’s aerial refueling operations will take several years to address at a cost to both the government and Boeing. Program officials reported that, similar to what we reported last year, they expect the KC-46 will ultimately meet all of its 21 performance goals. These goals include nine key performance parameters and five key system attributes set by the Air Force, as well as seven technical performance measures Boeing established to track its own progress toward meeting contract specifications. Appendix I provides a description of each of the performance goals. According to Air Force test officials, the program plans to ascertain if the aircraft meets its 14 key performance parameters and key system attributes during the operational test period, which began in May 2019. For example, the Air Force will test the tanker’s ability to effectively refuel receiver aircraft with boom and drogue refueling on the same mission. The Air Force will also collect data to assess the operational availability of the aircraft. Operational availability is defined as the percentage of time the aircraft is available to complete its mission, which includes refueling aircraft or transporting cargo or people, when needed. The KC-46 needs to be available at least 80 percent of the time. Air Mobility Command officials will continue to monitor operational availability of the aircraft after it has been fielded to inform maintenance and future upgrade decisions. An important key system attribute is reliability and maintainability, which has implications on aircraft availability and life cycle costs. In general, aircraft that are reliable and easy to maintain are typically available more often to perform missions and can experience lower life cycle costs. To help assess this key system attribute, the Air Force set a reliability growth goal that is based on the mean time between unscheduled maintenance events due to equipment failure. This is defined as the total flight hours divided by the total number of incidents requiring unscheduled maintenance. The goal is 2.83 flight hours between unscheduled maintenance events due to equipment failure by the time the aircraft reaches 50,000 flight hours. As of February 2019, the program had completed 3,928 flight hours, achieving 2.51 hours at that time. Program officials believe that the reliability will improve as additional flight hours are completed and as unreliable parts are identified and replaced. According to Boeing representatives, the company met or is projected to meet the seven technical performance measures it tracked during KC-46 development. For example, the aircraft is now below the target weight of 204,000 pounds. In addition, program officials said that the aircraft is within the range of gallons of fuel used per flight hour that is specified in the contract. Boeing also projects that the aircraft will meet other measures, such as Air Force maintainers being able to fix mechanical problems on the aircraft within 12 hours 71 percent of the time once the aircraft has accumulated 50,000 fleet hours of service. Boeing and the Air Force are working to resolve three critical deficiencies related to the performance of the aerial refueling systems that the Air Force discovered during developmental testing. These deficiencies are related to contract specifications, which are at a greater level of specificity than the performance goals. The Air Force determined that the deficiencies in these systems could result in damage to some of the aircraft that are being refueled by the KC-46 and identified them as Category 1 urgent deficiencies that need to be addressed. The Air Force expects that it will take 3 to 4 years for Boeing to develop design solutions for these issues and a few more years to retrofit existing aircraft. A description of the deficiencies and how they are being addressed are discussed below. Remote Vision System Did Not Provide Visual Clarity in All Lighting Conditions: During developmental flight testing, there were instances when the aerial refueling operator was not able to make contact with the receiver aircraft for refueling as intended. This was because the remote vision system camera and processor had difficulty making timely adjustments to some environmental conditions. According to Boeing and program officials, these conditions include certain sun angles, where the glare from the sun can cause the receiver aircraft to washout or blackout on the display screen, making it difficult for the aerial refueling operator to sufficiently see the receptacle of the receiver aircraft to start refueling. The remote vision system also does not provide sufficient depth perception to safely refuel in all lighting conditions. Boeing has already made changes to the remote vision system software to improve visibility for refueling operators. According to program officials, the changes included adjusting the contrast on the display screen and increasing the speed at which operators can switch between different screen viewing options. However, these changes did not address the Air Force’s concerns regarding whether the system could support refueling in all conditions as called for under the contract, which requires sufficient visual clarity in all lighting conditions. Boeing has agreed to redesign the remote vision system to meet the requirement. According to program officials, Boeing has not yet developed a solution, but has reported the redesign will include additional software and hardware changes. Program officials estimate that it may take Boeing 3 to 4 years to develop a solution for the remote vision system and have it certified by the Federal Aviation Administration so that aircraft parts will continue to be certified to the greatest extent possible. It will then take a few more years after that to retrofit all aircraft that are operating without the new system at that time. Boeing did not provide a cost estimate for this solution, but will fix and retrofit all aircraft at no cost to the government. In the meantime, program officials said the Air Force has placed limitations on some boom refueling operations. Lack of Remote Vision System Clarity Also Caused Undetected Contacts with Receiver Aircraft: As we reported in April 2018, during developmental flight testing, there were instances where the boom nozzle contacted a receiver aircraft outside the refueling receptacle. According to program officials, in many of these instances, the aerial refueling operators were unaware that those contacts had occurred. Boom nozzle contact outside the receptacle can damage antennae or other nearby structures. It is especially problematic for low-observable receiver aircraft, such as the F-22 fighter, because boom contact can also damage their special coatings and render them visible to radar. Boeing and program officials now anticipate that any hardware or software changes Boeing makes to the remote vision system, as discussed above, will also address the issue of undetected contacts with receiver aircraft. Efforts to address this issue are expected to be made at no cost to the government. Boom Stiffness Hampered KC-46 Refueling of Lighter Receiver Aircraft: During developmental flight testing, pilots of lighter receiver aircraft, such as the A-10 and F-16, reported the need to use more power to move the boom forward while in contact with the boom to maintain refueling position. According to program officials, the KC-46 boom currently requires more force to compress it sufficiently to maintain refueling position than the boom on the KC-135 or the KC- 10. In addition, program officials said that the additional force exerted by the lighter aircraft can also create an issue when the boom is disconnected. This is because the additional required power can cause the receiver aircraft to lunge forward into the boom and strike it, possibly damaging the receiver aircraft and the boom. The severity of the damage depends on the location of the refueling receptacle, which differs based on the aircraft type. In the case of the A-10, the receptacle is located on the nose of the aircraft and the boom stiffness creates a greater risk to the pilot because a boom strike could damage the windshield. For the F-16, the receptacle is located behind the cockpit and a boom strike could damage the vertical surfaces of its tail. The Air Force is currently allowing F-16s to be refueled by the KC-46 in operational test and training environments, but not the A-10 until the boom stiffness has been fixed. Modifications to address the boom stiffness will add cost for the government. Program officials said the development contract did not specify the amount of force needed to compress the boom. As part of the KC-46 low-rate initial production decision, the Air Force concurred with Boeing’s proposed specifications, which are built into the current boom. Therefore, program officials said the Air Force will be responsible for costs associated with designing a solution for the boom stiffness and retrofitting aircraft. They said the deficiency will require a hardware change. Program officials believe that it will likely take 3 to 4 years to develop a solution and get it certified by the Federal Aviation Administration. It will then take additional time to retrofit about 106 aircraft in lots 1 to 8. The total estimated cost for designing and retrofitting aircraft is more than $300 million. The Air Force has taken steps to keep Boeing incentivized to address the deficiencies in a timely manner. In particular, at the time the Air Force accepted each aircraft, the government had already made progress payments to Boeing comprising 80 percent of the estimated price for each aircraft. Air Force officials stated that the program is currently withholding the remaining 20 percent payment on each aircraft until Boeing meets all contract specifications and corrects critical deficiencies. Over the next year, Boeing is to conduct developmental testing on the wing aerial refueling pods and correct deficiencies, and the Air Force is to finish analyzing test data to validate that performance and contract specifications have been met. In addition, the Air Force is to complete operational testing (planned for completion in December 2019) to determine if the KC-46 and its subsystems are fully capable of performing its mission in a realistic operational environment. Since our last report in April 2018, Boeing completed developmental testing and obtained airworthiness certification from the Federal Aviation Administration for the KC-46 aircraft and two of its three aerial refueling systems—the boom and the centerline drogue system. This has allowed the Air Force to start accepting aircraft. Figure 6 shows the status of the KC-46 test activities. Developmental Testing: As of March 2019, Boeing had completed about 92 percent of the overall KC-46 developmental test program. The roughly 8 percent remaining, which consists of 2,303 of the 29,181 total developmental test points, relates to the wing aerial refueling pods. According to program officials, Boeing, in coordination with its supplier for this subsystem, submitted test plans to the Federal Aviation Administration in December 2018 for approval to begin flight testing the wing aerial refueling pods. These officials also told us that developmental testing on the pods began in early June 2019. Boeing projects that the Air Force will verify that the pods meet contract specifications and they will be airworthy by May 2020. The Air Force is also currently reviewing developmental test data to validate that performance and contract specifications have been met and identify aircraft deficiencies. As of March 2019, the Air Force has identified the three critical deficiencies that we discussed earlier in this report. It also identified 160 Category 2 urgent deficiencies that Air Force policy notes can be addressed through workarounds, which can include manual updates or procedural restrictions. For example, the flight control system does not have an indicator that would alert the KC-46 operators that they are overriding the automatic system that keeps the boom aligned with the receiver aircraft. If the boom is not aligned with the receiver aircraft, it can cause damage to the boom and the receiver aircraft. Program officials said that, as a result, the Air Force has currently placed limitations on some boom refueling operations. The number of Category 2 urgent deficiencies went up by about 26 percent between mid- February and the end of March 2019. Program officials attributed this growth to the progress the Air Force is making in analyzing test data and validating whether the aircraft meet contract specifications. The Air Force may identify additional deficiencies as it completes these developmental testing activities and during operational testing. Operational Testing: According to program officials, the Air Force Operational Test and Evaluation Center plans to conduct KC-46 operational testing from mid-May to December 2019. Operational testing is centered on five overarching test objectives. Three test objectives are focused on the ability of the KC-46 to perform operations for refueling, airlift, and aeromedical evacuation, including how quickly the KC-46 can offload fuel to a receiver aircraft. The fourth objective is focused on the ability of the KC-46 to meet its mission tasking, which includes measures such as the KC-46’s availability and ability to complete a mission. The fifth objective addresses whether the KC-46 is logistically supportable through measures including aircrew and maintainer training, and how well the demand can be met with available parts. According to Air Force test officials, operational testing consists of about 500 test conditions, each of which may include multiple test points. The Air Force plans to use four KC-46 aircraft for operational testing. During operational testing for aerial refueling, the Air Force will test whether the KC-46 can deliver fuel through the boom or centerline drogue system to 18 different types of receiver aircraft in operational conditions, including refueling another KC-46. The Air Force needs to certify receiver aircraft for refueling before these aircraft can be used for operational testing with the KC-46. Boeing and the Air Force are in various stages of testing and certifying 18 receiver aircraft. In its 2018 annual report, the Department of Defense’s Office of the Director, Operational Test and Evaluation reported that the duration of the KC-46 operational test period will depend on how long it takes the Air Force to certify all 18 receiver aircraft. As of March 2019, two aircraft have been tested and certified by the Air Force as a receiver to the KC-46. Five have completed testing, but have not yet been certified, and testing for two others has begun. Figure 7 shows the status of testing and certifications for the KC-46 receiver aircraft currently planned for operational testing. The Air Force plans to obtain additional certifications for aircraft that are not being used for operational testing. The Air Force schedule for completing receiver testing continues to shift. According to Department of Defense developmental and operational test officials and program officials, it is taking longer than expected to complete receiver aircraft certification testing in advance of operational testing due in part to receiver aircraft availability. According to these officials, Air Force major commands have been reluctant to allow their receiver aircraft to be tested with the KC-46 over concerns that the lack of visual clarity in the remote vision system and the boom’s stiffness could cause the boom to strike and damage the receiver aircraft. Program officials told us that, as a result, negotiations between the KC-46 program and Air Force major command officials concerning the use of receiver aircraft are taking longer than expected. These difficulties have resulted in delays to certification tests, in some cases for several weeks. The lack of availability of specific aircraft when they are scheduled to be tested may require the Air Force to reschedule other receiver aircraft. These schedule changes can require some resequencing of test planning and approval activities. In addition, because the wing aerial refueling pods have not been certified and delivered, the Air Force will need to conduct operational testing on refueling operations for them later. To conduct this test, major commands with receiver aircraft that require drogue refueling would need to provide receiver aircraft again. According to program test officials, the start of operational testing for the wing aerial refueling pods will depend on whether the Air Force Operational Test and Evaluation Center uses pods that have not been certified for airworthiness by the Federal Aviation Administration or waits until Boeing delivers a certified subsystem. Problems requiring changes could be identified during KC-46 operational testing, developmental and operational testing for the wing aerial refueling pods, or receiver aircraft certification testing. The development contract makes Boeing responsible to correct any deficiencies discovered during these test periods that do not meet contract specifications. Based on our own observations, as well as our discussions with Department of Defense officials who have been involved with the KC-46 program for many years, we identified aspects of its acquisition approach that could provide insights to future programs. Specifically, the insights could apply to programs considering a fixed-price development contract and determining what sustainment approach to use for commercial derivative aircraft. For example, the KC-46 development contract provided some financial protection to the government from increases in development and some life cycle costs. However, other aspects of the contract did not require Boeing to demonstrate high levels of aircraft performance prior to being awarded production contracts or receiving payment for its work. Current and former program officials also provided insights about key aspects of program management that they believe are essential for executing fixed-price development contracts based on their experiences. In addition, the Air Force’s new approach for sustaining the KC-46, relying heavily on the Federal Aviation Administration to certify even military-unique aircraft systems, could be useful in considering future acquisition approaches. We previously recommended in March 2012 that the Under Secretary of Defense for Acquisition, Technology and Logistics closely monitor the cost, schedule, and performance outcomes of the KC-46 program to identify positive or negative lessons learned for future acquisition programs. We noted that, as one of only a few major acquisition programs to award a fixed-price incentive (firm target) development contract in recent years, evaluating performance and identifying lessons learned would be illustrative, important for informing decision makers, and help guide and improve future defense acquisition programs. The Department of Defense agreed with the recommendation and compiled lessons learned during the source selection phase of the program. However, the department has not yet identified and reported on lessons learned during program implementation to evaluate cost, schedule, and performance outcomes as we recommended. Program officials said they are collecting lessons learned, but will not report them until after the development contract is complete in 2021. However, by waiting until 2021, other acquisition programs considering using a similar approach will not be able to take advantage of KC-46 lessons learned, including the ones we identify below that could reduce government risk and save taxpayer money. The Air Force used a fixed-price incentive (firm target) contract type to limit the government’s financial risk for KC-46 development. The KC-46 development contract was designed to provide a profit incentive for Boeing to control or even reduce overall costs. The use of a fixed-price contract did not result in a reduction in development costs below target costs, but did help control the government’s costs. Specifically, the Air Force was able to avoid $1.3 billion in costs exceeding the contract ceiling that Boeing has incurred while developing the aircraft, according to program officials, as of February 2019. Boeing initially declared cost overruns related to wiring while manufacturing the first development aircraft in the spring of 2014. At that time, it discovered wire separation issues, which were caused by an inaccurate wiring design. It took Boeing about 6 months to correct the wiring design and resume wiring work on the developmental aircraft. Boeing declared other cost overruns later in development as it faced challenges in obtaining Federal Aviation Administration certification for the aircraft, which caused significant testing delays. Together, the wiring problems, certification and testing delays, and other setbacks have resulted in a projected 3-year schedule delay. To the extent these costs exceeded the contract ceiling price, Boeing has borne the costs to address these issues, which included retaining more personnel such as design engineers and testers than it originally planned. The KC-46 contract also contains three specific clauses that further benefited the government by limiting its financial risk: Correction of deficiencies clause: This clause requires Boeing to pay for aircraft retrofits when the government determines that the company is not meeting contract specifications. According to the development contract, Boeing is responsible for correcting deficiencies discovered during engineering and manufacturing development, and in production and deployment. Based on the initial schedule, operational testing would have ended in 2017. Up to 19 low-rate initial production aircraft would have been covered by this clause and deficiencies would have been almost exclusively identified through testing activities. Because of delays in the development phase, more aircraft will now be covered by the correction of deficiencies clause. According to the integrated master schedule, Boeing will still be completing development activities in 2020. As a result, the correction of deficiencies clause is expected to now cover the 52 low-rate production aircraft already ordered as well as any other aircraft ordered while development activities are ongoing. Boeing will now be responsible for correcting deficiencies identified during testing as well as in day-to-day operations on all of these aircraft. Fuel usage rate clause: This clause requires Boeing to meet a specified fuel usage rate for each individual aircraft, which will help the Air Force control some of the KC-46’s life cycle costs. According to the contract clause, if an individual aircraft does not meet the fuel usage rate, Boeing would have to propose a corrective action at no cost to the Air Force. The Air Force could also make an equitable price adjustment based on a formula that projects the additional costs the Air Force would incur over the expected 40-year life of the aircraft. Long-term pricing: The KC-46 contract includes long-term pricing terms for 175 production aircraft. In agreeing to these terms, Boeing had to estimate its costs through 2027. The pricing in the contract protects the government from cost increases including inflation and higher supplier costs that were not already embedded in the prices. The contract includes a variety of purchasing options so that the Air Force is not locked into acquiring a set amount of aircraft each year. It identifies the most cost effective approach for procuring the 175 production aircraft, which is typically between 12 and 15 aircraft for each production lot. It also identifies the additional costs the Air Force would incur if it procured fewer or more aircraft in each production lot that would deviate from the most cost effective approach. Program officials stated that including the long-term pricing in the contract has helped it secure adequate funding from Congress to procure the most cost effective number of aircraft in each of the four low-rate production lots it has awarded so far. Several aspects of the fixed-price incentive development contract, however, did not reduce risk to the government and further complicated existing program challenges. First, production lot awards were not linked to Boeing’s performance. Second, progress payments to Boeing were based on costs the contractor incurred rather than on its demonstrated performance. Third, the contract did not identify the timing of when production aircraft would be delivered to the Air Force for acceptance. Production lot awards are not tied to demonstrated performance: The development contract linked the award of production lots to schedule milestones rather than to contractor performance. The contract specified that the first and second low-rate production lots had to be awarded within 30 days and 14 months of the low-rate initial production decision, respectively. According to the initial plan, Boeing would have completed 13 months of developmental testing and 66 percent of the flight test program with the KC-46 by the low-rate initial production decision. As we have previously reported, however, the program experienced delays. At the time of the low-rate initial production decision, the program had only completed about one-third of the planned flight test program. The Air Force decided to award both low-rate production lots within a week of the decision despite the lower amount of testing knowledge. Program officials stated that they awarded the contract because Boeing met the low-rate decision criteria, including demonstrating successful refueling operations. Further, based on the correction of deficiencies clause, they believed at that time that Boeing would be responsible for paying to correct all deficiencies it discovered during subsequent testing on aircraft it produced. Our prior work on best practices, however, emphasizes that awarding production lots before performance is demonstrated introduces risk of cost increases, schedule delays, and performance problems. Progress payments are not based on demonstrated performance: The KC-46 contract included a financing approach that requires the Air Force to make progress payments to Boeing up to 80 percent of its incurred costs. These progress payments incentivized Boeing to make progress on building the aircraft, and the program’s withholding of some payment incentivizes the company to resolve deficiencies more quickly. In general, Department of Defense guidance recognizes that performance-based payments incentivize a contractor to optimize its activities to meet the goals that are important to the government, such as completing a certain amount of engineering or developmental testing by specific milestones. It also notes that they are not practical on all contracts, and contracting officers should consider whether the benefits outweigh the time and effort to establish and administer them. The guidance also notes that progress payments based on costs incurred by a contractor may not reflect the contractor’s progress towards meeting program goals or incentivize a contractor to meet those goals. On the KC-46 for example, the program office had made 80 percent of the allowed progress payments for the four development aircraft by November 2015─9 months before the low-rate initial production decision, despite only completing 15 percent of the flight test points at that time. KC-46 program officials said that once the low-rate production contracts were awarded in August 2016, Boeing prioritized completing the manufacturing of those aircraft because it had previously started manufacturing them with its own funds. It also focused on completing aspects of developmental testing related to the boom and centerline drogue so that it could begin delivering aircraft to the Air Force. In general, once the Air Force accepts an aircraft, Boeing is eligible to receive additional payment for its work on that aircraft. Program officials, however, would have preferred that Boeing placed more emphasis on completing receiver aircraft certifications so that when aircraft were accepted, the Air Force could begin operational testing, which is led and paid for by the government. Contract originally did not identify aircraft delivery time frames: The original development contract did not identify a specific delivery period for production aircraft. Instead, it specified that Boeing was supposed to deliver the first 18 aircraft by August 2017. According to program officials, not identifying a delivery period was an oversight. Program officials stated that the Air Force needed more specific aircraft delivery information to develop detailed plans for establishing operating bases and performing depot maintenance, including training pilots and maintainers. For example, if training is done too early, the Air Force may have to provide refresher training to pilots and maintainers. If it is done too late, then the Air Force may not be able to use the aircraft as soon as it could or to the extent it had planned. The Air Force was eventually able to get specific delivery dates for the aircraft as part of negotiations it had with Boeing to modify the development contract after Boeing did not meet the original August 2017 contract delivery date. According to current and former KC-46 program officials, stable requirements and a skilled acquisition workforce are essential for executing a fixed-price incentive contract. Stable Requirements: The current KC-46 program manager said that there were no major requirements changes on the program between 2011 and 2018. The only requirements change occurred in 2019 to address the critical deficiency identified on the boom which, as we discussed earlier, the Air Force is paying to fix. As we previously found in 2012, controls were put in place to limit requirements changes. These controls were in response to a 2011 memorandum issued by the Office of Cost Assessment and Program Evaluation in the Office of the Secretary of Defense. The memorandum maintained that, on the whole, the Department of Defense had demonstrated limited ability to maintain stable requirements and limit changes to program baselines on previous complex weapon system programs, and that minimizing such change would be essential to the success of the KC-46. For the KC-46 program, any engineering or contract changes affecting system requirements or that have the potential to impact program cost, schedule, and performance baselines must be approved by the Air Force Service Acquisition Executive in consultation with the Secretary and the Chief of Staff of the Air Force. Skilled Acquisition Workforce: Some current and former program managers also noted that having personnel with strong negotiating and cost estimating skills, as well as data rights expertise, is essential for programs with fixed-price incentive development contracts. One former program official explained that in general, contractors such as Boeing on the KC-46 do not know exactly how they are going to build a weapon system until they have completed detailed systems engineering and design drawings, which occurs in the development phase. We previously found in November 2016 that as top-level capability requirements are defined and decomposed into lower-level design requirements, they become more specific and the number of requirements grows. This growth can be exponential, with tens of thousands of detailed design requirements derived from a relatively small number of capability requirements. While the government generally does not specify how a contractor designs a weapon system for fixed-price incentive contracts, officials we spoke with said KC-46 program managers and engineers have been involved in almost daily discussions with Boeing to make design tradeoffs. As such, one former program executive officer said program offices that are using fixed-price incentive development contracts should ensure that program management staff, including contracting officers and engineers, has strong negotiating skills to protect the government’s interest during these daily negotiations where design tradeoffs are made. Further, these program offices need financial management staff with strong cost estimating skills to support the negotiations when necessary. This official indicated that the KC-46 program office had people with these skills. However, several former program officials stated that the KC-46 program office needed personnel with data rights expertise. They said that they had to rely on a data rights expert from outside the KC-46 program to assist in drafting a section of the request for proposal that would allow the Air Force to obtain data it would need to maintain KC- 46 aircraft. The officials indicated that the Air Force has few data rights experts and that it would be beneficial to have contracting officers and attorneys in the program offices with data rights expertise. For example, program officials anticipate that there will be ongoing discussions and negotiations with Boeing about the type of data it will need for the Air Force to perform depot maintenance activities over the life of the program. The Air Force plans to use a sustainment approach on the KC-46 that it has not yet used on other aircraft, that presents added complexity, and which Boeing is having difficulty supporting. Under the new approach, the Federal Aviation Administration will certify nearly all parts of the aircraft, including most of the military-unique parts such as the centerline drogue, boom, and wing aerial refueling pods. By certifying through the Federal Aviation Administration, the Air Force expects to take advantage of commercial aircraft updates that occur regularly and to obtain new or refurbished parts for the aircraft through a global parts pool that commercial users of the 767 aircraft rely on to maintain their aircraft. Further, the Air Force, instead of a contractor, will provide product support for the aircraft. Previous commercial derivative aircraft programs, including the KC-10, did not have the Federal Aviation Administration certify military-unique functions such as aerial refueling, and the Air Force has relied on the KC-10 contractor for product support over the lifetime of that program. According to the KC-46 acquisition strategy, Boeing will initially provide product support for the KC-46 for a period of up to 5 years. During that time, the Air Force will gradually take over the responsibility and then maintain the aircraft for the lifetime of the program, which is expected to be 40 years. The KC-46 program’s experience in obtaining and maintaining Federal Aviation Administration certification, including participation in the parts pool, can offer insights for future acquisition programs to consider. The Air Force also required Federal Aviation Administration certification to a greater extent than the Air Force policy in place at the time the development contract was awarded. Specifically, the contract states that the contractor shall obtain Federal Aviation Administration certification for all the aircraft’s mission equipment. In cases where this is not workable, the contract says that the contractor must exhaust all possible solutions prior to not obtaining full certification. As we mentioned earlier in the report, Boeing is having difficulty getting certification for the military-unique portions of the aircraft related to the aerial refueling systems, which has contributed to significant program delays. Boeing’s commercial business unit already obtained Federal Aviation Administration certification for the commercial parts of the aircraft. However, according to program officials, Boeing’s defense business unit, which is responsible for obtaining certifications for the military-unique parts, was not as well versed on the certification process. We previously reported that, according to Boeing officials, the company and the supplier had underestimated the extent of design drawing details required by the Federal Aviation Administration to certify that the parts conformed to the approved design. The supplier of the wing aerial refueling pods spent several years negotiating agreements with several of its key sub-tier suppliers to obtain the necessary documentation. To reduce the risk of further delays, in 2015, Boeing co-located some of its employees with the supplier to provide technical support to complete the documentation for certification over the past several years. Based on a study completed by Morgan Borszcz Consulting in 2014, the Air Force expected to benefit from saving up to $420 million by maintaining the Federal Aviation Administration certification for the KC-46 over the life of the program. Savings were primarily estimated in three areas: 1. $200 million could be saved by having Boeing maintain responsibility for all design changes on the aircraft, including working with the Federal Aviation Administration to certify design changes and updating instruction manuals based on the changes. 2. $70 million could be saved by having Boeing address any safety issues identified by the Federal Aviation Administration in Airworthiness Directives. 3. Between $57 million and $150 million in costs could be avoided if the Air Force maintains Federal Aviation Administration certifications and does not recertify parts to military standards. The study also stated that the Air Force could save money by participating in the 767 aircraft parts pool, mentioned above, though it did not specify the amount of savings. The parts pool limits the risk of diminishing manufacturing sources over time and the costs the Air Force typically incurs when qualifying new suppliers. Program officials told us that they decided to use a worldwide 767 parts pool because more than 75 percent of KC-46 parts are expected to be available through that parts pool, which reduces the need for the Air Force to procure these parts in advance and place them in its distribution system. Programs that do not have Federal Aviation Administration certified parts have to find and qualify suppliers for needed parts on their own and they must find and qualify new suppliers if one goes out of business over the operational lifetime of the aircraft. In using the 767 parts pool, the Air Force anticipated readily obtaining parts as needed for maintaining the KC-46 aircraft as well as repairing parts and putting them back into the pool. Since the time the study was completed, however, program officials have learned that the Air Force cannot put parts back into the parts pool because commercial members of the pool do not want to use repaired or reconditioned parts that were used on Air Force aircraft. As a result, the Air Force will not achieve all of the savings it anticipated. Program officials explained that commercial companies do not fly their aircraft under the same conditions as the Air Force, and these companies believe it is too risky for them to use parts that were once used on a KC-46. Program officials said the Air Force can still purchase parts from the parts pool though. The Air Force can also refurbish and use its own parts as long as the parts and the processes it uses to refurbish the parts meet Federal Aviation Administration certification standards and mechanics are properly certified. However, it remains to be seen if the Air Force can maintain the certifications because it has not yet had to do this on other aircraft and requires adherence to Federal Aviation Administration procedures. The Air Force’s approach to building the KC-46 has been somewhat unique—deriving a military aircraft from a commercial model using a fixed-price incentive contract, among other things. After experiencing delays of nearly 3 years, the Air Force started accepting aircraft that can now be used for operational testing and support of worldwide missions. While work remains to ensure that critical deficiencies are corrected, the KC-46 program offers lessons that could be shared with other Department of Defense acquisition programs that are considering using a fixed-price-type development contract or a commercial derivative aircraft regarding contracting for and sustaining weapon systems. In particular, the contract provided substantial protections to the government against cost increases that Boeing experienced while developing the aircraft, but it also used a financing approach that did not tie Boeing’s performance to completing important program goals. In addition, the Air Force’s effort to leverage commercially available parts to reduce sustainment costs created challenges. We previously recommended that the Department of Defense develop and share KC-46 lessons learned for future acquisition programs; however, it does not plan to do so until 2021. By sharing identified lessons now with other program leaders considering fixed-price- type contracts or developing commercial derivative aircraft, programs may be able to increase the effectiveness of any new similar development programs. We are making the following recommendation to the Department of Defense: The Secretary of Defense should ensure that the KC-46 program office disseminates insights we identified in this report about the KC-46’s contracting and sustainment planning experiences for consideration by acquisition programs, in particular those considering a fixed-price-type development contract or a commercial derivative aircraft. We provided a draft of this product to the Department of Defense for comment. In its comments, reproduced in appendix II, the department concurred with our recommendation, but did not identify the specific actions it would take to implement the recommendation. It also provided, in technical comments, language clarifying that the Air Mobility Command cost estimates for flying and maintaining KC-135s longer, as a result of KC-46 delivery delays, did not also account for any savings that would be achieved from not flying KC-46 aircraft. We provided additional detail in the report to address this comment. We also incorporated other technical comments as appropriate. We are sending copies of this report to the Acting Secretary of Defense, the Acting Secretary of the Air Force, and appropriate congressional committees. The report is also available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or ludwigsonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. The program office has 21 performance goals that are critical to the KC- 46 aircraft’s military capability and track progress to meeting contract specifications. These performance goals include nine key performance parameters, five key system attributes, and seven technical performance measures. Table 2 provides a description of each key performance parameter and key system attribute. Table 3 provides a description and status of each technical performance measure. In addition to the contact named above, Cheryl Andrew, Assistant Director; Lorraine Ettaro; Kurt Gurka; Stephanie Gustafson; Katheryn Hubbell; Jean Lee; Malika Rice; Jenny Shinn; and Steve Woods made key contributions to this report. GAO, KC-46 Tanker Modernization: Program Cost is Stable, but Schedule May Be Further Delayed, GAO-18-353 (Washington, D.C.: Apr. 18, 2018). GAO, KC-46 Tanker Modernization: Delivery of First Fully Capable Aircraft Has Been Delayed Over One Year and Additional Delays are Possible, GAO-17-370 (Washington, D.C.: Mar. 24, 2017). GAO, KC-46 Tanker Aircraft: Challenging Testing and Delivery Schedules Lie Ahead, GAO-16-346 (Washington, D.C.: Apr. 8, 2016). GAO, KC-46 Tanker Aircraft: Key Aerial Refueling Capabilities Should Be Demonstrated Prior to the Production Decision, GAO-15-308 (Washington, D.C.: Apr. 9, 2015). GAO, KC-46 Tanker Aircraft: Program Generally on Track, but Upcoming Schedule Remains Challenging, GAO-14-190 (Washington, D.C.: Apr. 10, 2014). GAO, KC-46 Tanker Aircraft: Program Generally Stable but Improvements in Managing Schedule Are Needed, GAO-13-258 (Washington, D.C.: Feb. 27, 2013). GAO, KC-46 Tanker Aircraft: Acquisition Plans Have Good Features but Contain Schedule Risk, GAO-12-366 (Washington, D.C.: Mar. 26, 2012).", "summary": "Aerial refueling—the transfer of fuel from airborne tankers to combat and airlift forces—is critical to the U.S. military's ability to effectively operate globally. The Air Force initiated the KC-46 program in 2011 to replace about a third of its aging KC-135 aerial refueling fleet. Boeing was awarded a fixed-price incentive contract to develop the first four aircraft, which are being used for testing. Boeing was also required to deliver the first 18 fully capable aircraft by August 2017. The program plans to eventually field 179 aircraft. This report assesses the program's progress toward meeting cost, schedule, and performance goals. The report also assesses how the program's contracting and sustainment planning approach could inform other acquisition programs. GAO analyzed cost, schedule, performance, test, manufacturing, contracting, and sustainment planning documents; and interviewed officials from the KC-46 program office, other defense offices, such as the Defense Contract Management Agency, the Federal Aviation Administration, and Boeing. Costs for the KC-46 program remain lower than expected, as shown below. The Air Force accepted the first KC-46 in January 2019, but Boeing remains nearly 3 years behind schedule. As shown below, Boeing now plans to deliver the first 18 aircraft with all three aerial refueling subsystems by June 2020. Program officials expect the KC-46 to meet key performance goals over the next few years as it accumulates 50,000 fleet hours. However, the Air Force is accepting aircraft that do not fully meet contract specifications and have critical deficiencies, including ones that affect (1) the operators' ability to guide the fuel delivery boom into position, and (2) the boom itself. The deficiencies could affect operations and cause damage to stealth aircraft being refueled, making them visible to radar. Program officials estimate it will take 3 to 4 years to develop fixes for the deficiencies and a few more years to retrofit up to 106 aircraft. The Air Force and Boeing will incur costs to fix the deficiencies, with the Air Force's portion estimated to be more than $300 million. The Air Force is withholding 20 percent payment on each aircraft until Boeing fixes the deficiencies and non-compliances. Meanwhile, the Air Force has limited some refueling operations. GAO identified a number of insights that could benefit other programs, including the use of a fixed-price-type development contract and a correction of deficiencies clause in the contract that protected the government from some cost increases. The Department of Defense agreed to provide lessons learned about the KC-46 program for future acquisition programs based on a recommendation GAO made in March 2012, but does not plan to do so until development is complete in 2021. GAO believes other programs could benefit from insights identified in this report if they were disseminated sooner. GAO recommends that the Department of Defense disseminate insights in this report about the KC-46's contracting and sustainment planning experiences for consideration by acquisition programs, particularly those that plan to use a fixed-price-type development contract or a commercial derivative aircraft. The Department of Defense concurred with the recommendation.", "document_type": "gao"}
{"report": "Consumers receive broadband service from telephone, cable, mobile, satellite, and utility companies that own and operate the telecommunications infrastructure. Fixed technologies, like cable or fiber, can provide broadband to single locations like customers’ homes or businesses. Mobile technologies provide internet access wherever a customer has access to a signal. Customers connect to a mobile wireless network through a mobile device, such as a smartphone. Internet service that is high speed and provides an “always-on” connection, so users do not have to reestablish a connection each time they access the internet, is commonly referred to as “broadband.” FCC’s benchmark speed for what constitutes “advanced telecommunications capability,” a subset of broadband, has increased over time as consumers use the internet for an expanding range of purposes that requires faster speeds. In 2015, FCC set a benchmark speed for fixed advanced telecommunications capability to 25 megabits per second when downloading and 3 megabits per second when uploading (25 Mbps/3 Mbps). Internet service at various speeds allows for a variety of online activities, such as those shown in figure 1. In addition to fixed providers, satellite providers have begun meeting this benchmark, and FCC has recognized them as a viable source of advanced telecommunications capability. FCC has not set a similar benchmark for mobile services. The federal government has emphasized the importance of ensuring Americans have access to broadband, and a number of agencies provide funding to subsidize broadband deployment in areas, such as rural areas, in which the return on investment has not attracted private investment. As we have previously reported, rural areas may have features that increase costs of deploying and maintaining broadband networks. For instance, low population density, low broadband adoption rates, or mountainous or rugged terrain can make it especially costly for fixed and mobile providers to deploy infrastructure to rural areas with an expectation of getting a return on their investment. The Communications Act of 1934, as amended by the Telecommunications Act of 1996, specifies that consumers in “rural, insular, and high-cost areas” should have access to telecommunications and information services at rates that are “reasonably comparable” to rates charged for similar services in urban areas. Consequently, federal programs exist to support investment in broadband deployment for high-cost areas through federal grants, loans, and other subsidies. The largest share of federal support comes from FCC’s Universal Service Fund, which includes four component programs designed to ensure access to affordable communications for schools, libraries, rural health care providers, low-income consumers, and those in rural and high-cost areas. The largest component of the Universal Service Fund is the high- cost program—which includes the Connect America Fund and the Rural Digital Opportunity Fund—that targets financial support to rural high-cost areas for the deployment and maintenance of voice and broadband- capable networks. Table 1 shows selected federal programs funding the deployment of broadband infrastructure. FCC has other roles and responsibilities in regulating nationwide communications activities in addition to those identified above. FCC collects deployment data twice a year from broadband providers in order to better identify areas where broadband service is available. FCC, RUS, and NTIA have used and continue to use these data to inform their broadband programs. Furthermore, FCC and NTIA jointly determine the amount of spectrum—a finite natural resource that makes a variety of wireless communications possible—allocated for federal, nonfederal, and shared use. FCC also regulates the use of licensed and unlicensed spectrum through its regulatory process. This report’s broad view of a decade of federal efforts to advance broadband access builds on our prior work. FCC Universal Service Fund. In 2014, we examined FCC efforts to increase broadband deployment in unserved areas and identified legal, policy, and economic concerns—for example, low returns on investment—in deploying broadband in unserved and underserved areas. We also examined varying approaches for financing broadband deployment in high-cost areas, including local funding sources and a variety of ownership structures over the infrastructure. FCC deployment data. In 2014, we examined FCC’s efforts to reform its high-cost program and the extent to which FCC was collecting data to determine the effectiveness of these reforms, among other objectives. We identified gaps in FCC’s data analysis and reporting, including a lack of transparency and accountability of spending. We recommended that FCC analyze how it uses its high-cost program funding and make that analysis publicly available at least annually. FCC has taken action to implement our recommendation to address the lack of transparency and accountability of spending. In 2018, we reviewed data that FCC collected from providers to describe the locations of existing broadband infrastructure and help federal programs identify unserved and underserved areas to target for federal funding. We found that these data overstated broadband access, especially in tribal lands, and recommended that FCC take actions to improve these data. FCC concurred with the recommendations and has begun taking action, but the agency has not yet fully implemented any of the report’s three recommendations. Broadband adoption. In 2015, we stated that adopting broadband at home can provide a number of benefits, including access to employment opportunities (searching for and applying to jobs); education (research, web-based learning, and homework); and services for economic and social gain (such as telemedicine and entertainment). We reviewed federal efforts to address broadband adoption barriers that consumers face and recommended that FCC revise its strategic plan to more clearly state if broadband adoption is a priority, and if so, what outcomes FCC intends to achieve, action that the agency took the following year. We also recommended that NTIA include performance metrics for the agency’s broadband adoption efforts in its annual performance plan. Both FCC and NTIA implemented these recommendations. RUS grant and loan programs. In 2017, we assessed whether RUS’s procedures and activities related to its broadband grant and loan programs are consistent with leading management practices. We found that its activities and procedures were consistent with four leading practices and partially consistent with six leading practices. We made five recommendations to RUS to improve management practices for specific programs, like the Community Connect Program. In response, RUS implemented two of the recommendations to develop and document clear goals and performance measures for its broadband loan and grant programs and to establish and implement procedures to conduct a risk assessment of each program. RUS agreed with, but has not yet implemented, the report’s other three recommendations to: (1) conduct periodic evaluations of completed grant projects to determine the outcomes associated with these projects; (2) establish a timeline for implementing a centralized internal data system for staff to use in managing and monitoring loans and grant awards; and (3) develop, update, and maintain complete written policies and procedures for RUS’s programs as a way to retain and communicate organizational knowledge internally among agency staff. Broadband competition. In 2017, we found that infrastructure costs and other factors can affect competition among broadband providers. Such costs can limit competition in urban areas but more significantly limit competition in non-urban and less populated areas. We made two recommendations to FCC to solicit and report on the views of stakeholders regarding: (1) how well FCC’s programs promote broadband competition and (2) how varying levels of broadband deployment affect broadband prices and service quality. In response, FCC implemented these recommendations by soliciting public comments in July 2018 to seek feedback on the effectiveness of its actions addressing competition among broadband providers and on how varying levels of broadband deployment affect prices and service quality. In December 2018, FCC reported comments that it had received from this solicitation in the first version of a biennial report on the broadband market. In May 2019, FCC also reported stakeholder comments related to the agency’s broadband deployment data, including service quality data. Tribal broadband access. In 2018, we examined challenges that tribes face in accessing broadband services, focusing on two particular areas: (1) tribes’ ability to obtain and access spectrum for providing broadband and (2) tribes’ partnerships with private sector companies and others, and the ability to obtain funding to deploy broadband infrastructure on tribal lands. We found that tribes cited a number of barriers to obtaining licenses for spectrum. We also found that although tribes said partnerships with the private sector improved access to broadband, there are few such partnerships, and that tribes face regulatory barriers in applying for funding from RUS grant programs. We made three recommendations to FCC to collect data on tribal access to spectrum, analyze unused spectrum over tribal areas, and make information about available spectrum more accessible. We also made one recommendation to RUS to identify and address any regulatory barriers that may impede efforts by tribes to obtain RUS funding. FCC agreed with the recommendations, and RUS neither agreed nor disagreed with its recommendation. The agencies have not yet implemented these recommendations. According to the U.S. Census Bureau’s Annual Capital Expenditures Survey data, the telecommunications industry that provides various types of broadband services—fixed, mobile, or satellite (and other)—spent an estimated $795 billion (2018 dollars) in total capital expenditures from 2009 through 2017. Selected broadband providers that we contacted stated that they used the majority of their capital expenditures to improve the capability and reliability of their existing broadband infrastructure or expand infrastructure into new areas. For instance, one provider said it expanded wireless broadband service in Iowa and another provider constructed new towers to transmit fixed-wireless broadband signals across parts of Oklahoma. Some providers we contacted said that their capital expenditures may include funds from federal broadband programs or other items not related to broadband, such as the purchase of real or personal property or the acquisition of other broadband companies. Providers that we contacted offered few, if any, details about their investments. Instead, they said that their detailed expenditures were proprietary or they referred us to their annual reports, which contain limited information on capital expenditures. Census data showed that annual total capital expenditures increased from about $78 billion in 2009 to about $97 billion in 2017 (an increase of about 24 percent), with an average annual growth rate of about 2.8 percent. See figure 2. Industry capital expenditures for specific telecommunications sectors varied. For example, estimated expenditures for fixed services consistently exceeded estimated expenditures for mobile services, although mobile services experienced a greater increase—55 percent compared to an 8 percent increase in estimated expenditures for fixed services from 2009 through 2017. In comparison to industry spending, federal investment is much smaller, representing about 6 percent of total industry capital expenditures. However, this investment is critical to supporting deployment of broadband in rural areas where industry might not otherwise invest, due to potentially higher costs and lower investment returns. According to FCC, RUS, and NTIA data, federal program investments totaled about $47.3 billion (2018 dollars) to target broadband infrastructure in unserved or underserved areas from 2009 through 2017. Of these three federal agencies, FCC provided the largest share of support through the Universal Service Fund’s high-cost program—which is an ongoing program. A second agency, RUS, offered loans and grants. And NTIA primarily funded broadband deployment through the Recovery Act, which provided one-time funding for projects that are largely complete and are no longer active. To illustrate: FCC’s high-cost program. The high-cost program disbursed about $41.7 billion (2018 dollars) in support of both deployment and maintenance of voice and broadband-capable networks from 2009 through 2017. RUS’s programs. RUS provided grants or loans, or a combination of both, through a variety of funding programs. The Broadband Initiatives Program—a Recovery Act program—awarded about $2.2 billion (2018 dollars) in grants to industry for infrastructure projects in fiscal year 2010. RUS’s Community Connect Grant Program—a grant program designed to fund broadband deployment in rural areas where such service did not exist—awarded $95 million (2018 dollars) in grants from fiscal year 2009 through fiscal year 2017. In addition to these grants, RUS provided infrastructure loans that recipients must repay to the government with interest. Specifically, RUS provided about $4.0 billion (2018 dollars) in loans to private providers through the Broadband Initiatives Program, the Telecommunications Infrastructure Loan Program, and the Broadband Loan Program. NTIA’s Broadband Technology Opportunities Program, Comprehensive Community Infrastructure projects. This Recovery Act program awarded a one-time $3.3 billion in competitive grants to states, municipalities, and non-profit and commercial organizations in fiscal year 2010. Nearly all of the 116 broadband infrastructure projects have been completed. All three programs have used metrics to show progress in closing deployment gaps. Specifically, FCC has a metric for locations served, whereas RUS and NTIA measure miles of fiber-optic cable deployed, in addition to having metrics that count particular types of locations served or reported number of new subscribers. FCC collects data from providers about new locations to which they deployed broadband using high-cost program support. Deployment data submitted by providers that receive support from FCC’s high-cost program showed that they used those funds to make broadband available to about 2.3-million new residential and small business locations, mostly from 2015 through 2017. In commenting on a draft of this report, FCC stated that this figure has increased to about 4.2- million new locations. This updated figure is based on data through 2019 that are not yet publicly available; they are expected to be released later in 2020. Providers report these data to FCC, which are subject to verification by the Universal Service Administrative Company—the not-for-profit corporation designated by the FCC as the administrator of the Universal Service Fund, including the high- cost program. As of May 2020, FCC officials said that FCC has authorized Connect America Fund Phase II support to deploy broadband at 25 Mbps/3 Mbps or higher to more than 631,000 locations by 2025 or sooner. In December 2016, RUS released the final Broadband Initiatives Program progress report, which noted that the program deployed 66,521 miles of fiber-optic cable, added 5,468 wireless access points, and resulted in 334,830 subscribers receiving new or improved broadband. In December 2016, NTIA reported that the Broadband Technology Opportunities Program resulted in the deployment of 117,072 miles of new or upgraded broadband infrastructure. NTIA also reported that awardees connected nearly 26,000 community anchor institutions— such as schools, libraries, and hospitals—to broadband and provided access to nearly 14,149 homes and businesses. Although the agencies used metrics to show progress, the metrics used were not always the same, making it difficult to draw comparisons among programs. The impact of these federal programs goes beyond the number of miles of fiber or the number of subscribers. Although these programs promoted the availability and use of broadband throughout the country, our prior work found that they also stimulated economic development and created new jobs. For example, we reported in 2012 that NTIA’s and RUS’s Recovery Act programs had created about 9,000 full-time jobs. Recovery Act grantees we interviewed for our prior work gave examples of the types of economic development broadband enabled, such as tourism-oriented businesses being better able to provide web sites and online reservation systems. They also reported that broadband infrastructure improved broadband speed for schools, community colleges, and health care providers. Several studies have attempted to measure the economic benefits of broadband. A 2006 study prepared for the Department of Commerce claimed to be the first attempt to quantify the impact of broadband on economic growth. The study found that, between 1998 and 2002, communities in which broadband was available experienced more rapid growth in employment, the number of businesses, and businesses in information technology sectors, relative to comparable communities without broadband. Subsequently, other studies have attempted to assess the economic impact of broadband. For example, a 2016 study from the Hudson Institute found that rural broadband providers directly and indirectly added $24.1 billion to the U.S. economy and the rural broadband industry supported about 70,000 jobs in 2015, both through its own employment and the employment that its purchases of goods and services generated. About the same time, a 2016 broadband forum sponsored by the National Science Foundation and NTIA concluded that during the past decade, research has deepened the understanding of the potential impacts of broadband on the economy and society. The study made clear the need for more research on the impact of broadband. On December 11, 2018, FCC opened the new Office of Economics and Analytics, consisting of economists, attorneys, and data professionals to, among other things, provide economic analysis, including cost-benefit analysis, for FCC proceedings. FCC’s annual Broadband Deployment Report, which reports on broadband deployment generally and not just deployments made with FCC funding, states that broadband availability has increased both nationally and for specific segments of the population, as shown in figure 3. National: About 94.4 percent of the U.S. population had fixed broadband service available at customer premises, such as residences, with minimum speed of 25 Mbps/3 Mbps in 2018, up from 81.2 percent of the population in 2012. Rural: About 77.7 percent of the rural population had fixed broadband service available with minimum speed of 25 Mbps/3 Mbps in 2018, up from 45.7 percent of the rural population in 2012. Tribal: About 72.3 percent of tribal lands had fixed broadband service available at the same speeds in 2018, up from 32.2 percent of the tribal population in 2012. Although these data show broadband availability increasing in a variety of ways, the data also demonstrate that fixed broadband is still much more readily available to urban consumers than it is available to consumers in rural areas. FCC’s Broadband Deployment Report shows that as of 2018, about 22.3 percent of the rural population and 27.7 percent of tribal population did not have fixed broadband service available with minimum speed of 25 Mbps/3 Mbps; whereas, only about 1.5 percent of the urban population did not have fixed broadband service available at the same speed. As we will discuss later in this report, limitations with how FCC collects and uses deployment data from providers to measure broadband access overstate the extent to which broadband is available, a weakness we have pointed out and that FCC has taken steps to address. As the availability of broadband service has increased over time, some segments of the population continue to lag behind others in adopting broadband, even if it is available, and therefore are unable to benefit from it. Our prior work has shown that several factors have been, and continue to be, barriers to broadband adoption—specifically, affordability, lack of perceived relevance, and lack of computer skills. FCC identified these barriers in its National Broadband Plan of 2010, and our more recent work in 2015 showed that these three barriers persist. We found that: the cost of a subscription for internet service and the purchase of computer equipment was the most frequently identified barrier; the perception that broadband does not provide enough utility relative to its cost acted as another barrier; and the lack of exposure to or knowledge about computers, such as by those aged 65 or older and those with low levels of income and education, was another barrier. Compounding the effect of these adoption barriers is the lack of competition. FCC has reported that competition could result in lower prices and higher quality services from broadband providers. However, our prior work from 2017 found that 51 percent of the U.S. population had only one fixed broadband provider offering minimum speed of 25 Mbps/3 Mbps. According to the FCC’s 2018 Communications Marketplace Report, that percentage has decreased to about 27 percent of the U.S. population who had only one fixed terrestrial broadband provider offering minimum speed of 25 Mbps/3 Mbps. In addition, FCC’s report stated that 68 percent of the population had at least two providers and approximately 95 percent had at least one provider. Competition in rural areas can be particularly challenging as rural areas generally do not have enough demand to support multiple carriers. Over time, the types of technologies that provide access to broadband have evolved. Federal agencies have responded by making changes to their programs that support broadband. Specifically, FCC and RUS have expanded which types of broadband providers are eligible to receive support from their programs, allowing increased participation by satellite and wireless broadband providers. Satellite and mobile broadband may be able to overcome some impediments to access faced by other services, such as high deployment costs and geographical barriers that pose challenges for deploying broadband over fixed networks using fiber or cable. In turn, this expansion of eligibility corresponds with the shrinking gap in broadband deployment discussed previously. In the case of FCC, the agency has taken action since the 1990s to address technological changes related to broadband deployment. For example, changes in communications technology and competition in the communications marketplace led FCC to reform the high-cost program for purposes beyond maintaining telephone service, including supporting broadband deployment. In 2011, FCC adopted new rules that fundamentally changed the high-cost program and expanded the program to support broadband capable networks. Under these rules, FCC established new funding streams within the high-cost program, such as the Connect America Fund, which addresses fixed broadband availability gaps in underserved and unserved areas, and the Mobility Fund, which supports deployment of wireless networks to provide mobile broadband. FCC also updated its regulatory framework to recognize changes in existing technology and potential technologies in delivery of broadband. For example, in 2016, FCC deemed geostationary satellites eligible to participate in the second phase of the Connect America Fund. Additionally, since 2017, FCC also recognized low and medium earth orbiting satellites as broadband-capable technologies that may be eligible to participate in programs after deployment. According to FCC officials, prior to these changes, they did not consider satellite as broadband- capable due to its high signal latency and internet speeds that were below the FCC benchmark speed, issues that recent technological advances have improved. Similar to FCC, RUS funding programs used to focus funding on telephone service but over time, RUS has reformed them to provide funding for broadband infrastructure and deployment. For example, according to RUS officials, since 1995 the RUS Telecommunications program has only funded systems that were capable of providing high-speed internet and now supports broadband services. In addition to past program transformation efforts, both FCC and RUS have proposed actions to further reform or expand their programs that provide funding for broadband deployment. For example, in August 2019, FCC started the rulemaking process for the new Rural Digital Opportunity Fund. In January 2020, FCC adopted a Report and Order establishing a framework for the fund, providing up to $20.4 billion through two funding rounds, each providing support over overlapping 10-year periods. This fund is the next iteration of the high-cost program, and it continues the overarching goals of prior high-cost programs to expand service into rural areas. The Rural Digital Opportunity Fund will focus its first round of funding on census blocks that FCC deployment data have marked as completely unserved, and per the FCC order will incentivize parties participating in the program to serve tribal census blocks. Similarly, in April 2020, FCC initiated a rulemaking to establish the 5G Fund—which would replace the Universal Service Mobility Fund Phase II—and make up to $9 billion available to carriers to aid in deployment of advanced 5G mobile wireless services in rural America. In addition to FCC’s actions, RUS officials said they are planning for future funding rounds for the ReConnect Program after they have awarded the initial phase of funding. As of April 15, 2020, RUS had closed the application phase for a second round of funding under the program. Despite federal efforts to address broadband gaps, there are still limits on participation in some programs. For example, RUS’s ReConnect Program offers a mix of grants and loans to incentivize broadband infrastructure in areas not currently served by existing service providers. However, the ReConnect Program limits eligibility to fixed and satellite broadband providers, with mobile wireless networks ineligible for funding. In order to participate in FCC’s high-cost program, a provider must meet FCC’s definition of an eligible telecommunication carrier (ETC). However, our prior work and stakeholders we interviewed for this review identified barriers to attaining ETC status. As we previously reported, tribal entities cited the statutory requirements applicable to ETC designation as a primary barrier to accessing federal funds. Additionally, cable providers we spoke with for this review also said they viewed the ETC designation as a potential barrier to entry into the high-cost program. At present, FCC and states have complementary authority to make ETC designation decisions. During our 2018 review of barriers that tribes face in supporting broadband investment, FCC officials said that most of the carriers that were eligible for ETC designation at that time were the telephone companies that were in existence when the 1996 Act was enacted into law. Further, FCC officials said they determined that the statute is clear that only ETCs may receive program support, and therefore the agency does not have the authority to allow non-ETCs to receive high-cost support payments. More recently, greater numbers of companies that are not traditional telephone companies have received ETC designation, particularly in connection with the Connect America Fund Phase II. In addition to the ETC designation matter described above, industry stakeholders highlighted several other issues that can affect access to federal support, noting that these issues may require further action by relevant federal agencies or possible legislative action. The issues cited included: Technology Neutral Federal Programs: Most federal broadband programs focused on fixed technology over other technologies, but as described above, there have been reforms to broaden eligibility to other providers. Even with those reforms, industry stakeholders representing satellite and mobile service providers noted that there are program requirements that affect one technology more than another technology. For example, satellite providers that sought funding through FCC’s Connect America Fund said that, after they sought funding in 2018, FCC changed how it planned to measure latency, a change affecting only satellite providers. Spectrum Availability: Availability of adequate spectrum was cited by a range of stakeholder groups we spoke with as an issue that could affect providers’ ability to deploy services. Spectrum availability affects many broadband services. For example, in addition to satellite and mobile providers, fixed-wireless—a point-to-point or point-to- multipoint broadband service delivery option—is a type of fixed broadband service often utilized in rural areas that needs spectrum to deliver service. In its comments, FCC noted that the changes it made to the exact testing conditions were in response to requests by satellite providers and that the agency balanced changing expectations with the benefits of minimizing unnecessary burdens on carriers and their customers imposed by the testing regime. Federal Funding Mechanisms: Some industry stakeholders we spoke with noted that the programs’ funding requirements and the type of federal funding mechanism utilized—such as grants, loans, or hybrids of grants and loans—could affect a provider’s ability to access federal funds. For example, RUS awards its Reconnect Program funding through grants, loans, or both. Providers we spoke with noted that for all of the funding options, the Reconnect Program requires a lien on the funded assets. For some providers, allowing a lien against an asset would violate stockholder agreements or other aspects of their business. Additionally, representatives from an association representing state public service commissions we spoke with also stated that the FCC high-cost fund and its various funding programs should be reviewed periodically to ensure that both the contributions and funding outcomes are in the best interests of consumers and providers. Although agencies have modified their broadband funding programs to keep up with changes in broadband services, other changes that could be beneficial to the public would require statutory changes. The last major overhaul to telecommunications law occurred under the 1996 Act, which established many of the telecommunications programs that now fund broadband deployment and established statutory constructs like ETCs, which, as discussed above, can affect provider eligibility. Given the significant and ongoing changes in how Americans use the internet and the technologies that provide access to it, members of Congress have proposed legislative actions in recent years to sustain progress in closing the broadband deployment gap. Among the proposals were a range of federal funding and incentives aimed at improving funding to rural areas and addressing issues related to deployment. As we previously reported, FCC’s definition of broadband availability can lead to overstatements of fixed broadband availability. For instance, in 2013 FCC began collecting broadband availability data by census blocks. The agency counts an entire census block as served if a provider reports that it does offer—or could offer without an extraordinary commitment of resources—service some, but not necessarily all, of the locations in the census block. FCC has recognized that by measuring availability at the census block level, not every person may have access to broadband in a block that the data show as served. FCC has noted that census blocks in rural areas tend to cover larger geographic areas than in urban areas and providers may only deploy service to a portion of the census block. Deployment reporting in this manner does not allow FCC to answer with certainty questions like how many Americans have broadband available to their homes or where it needs to target its resources. Several selected providers and industry associations we contacted also expressed concerns about how deployment is measured, and said the measurement approach could make it difficult for them to make informed investment and deployment decisions. In 2013, FCC declined to gather fixed broadband data at a level more granular than the census block—such as address-level data—because the agency concluded that the complexity and filing burden on the industry would outweigh the benefit. In 2018, we recommended, among other things, that FCC develop and implement methods—such as a targeted data collection—for collecting and reporting accurate and complete data on broadband access specific to tribal lands. Subsequently in 2019, FCC began to address this recommendation by establishing the Digital Opportunity Data Collection— a more granular, nationwide data broadband deployment collection effort. FCC issued a Report and Order, and Second Further Notice of Proposed Rulemaking, on this new initiative to improve the accuracy of broadband deployment mapping data in August 2019. This new initiative requires fixed broadband providers to identify their service areas using free-form geographic shapes called polygons. The polygons would identify the presence of service with more geographic precision than the current census-block method affords. Officials from FCC and USTelecom stated this new mapping effort was informed in part by a 2019 USTelecom mapping and data-collection pilot project. According to USTelecom officials, this pilot project combined several data sources to determine “serviceable locations,” which refers to specific locations or structures that could need broadband. With these serviceable locations, USTelecom’s pilot project sought to distinguish between buildings not typically needing broadband service—such as a barn or storage shed—from a primary residence or small business. According to USTelecom representatives, a combination of polygons and serviceable locations data would yield a better picture of where to target new investments in deployment to achieve the greatest increase in access to broadband. FCC officials noted that the impact of the polygon approach may vary depending on the service features of each census block. Figure 4 below shows an example of how the new polygon approach may result in improved data compared to the census block approach currently used by FCC. As part of the rulemaking, FCC has requested comments on several issues, including how providers should define their polygons, and the procedures that fixed providers should follow if their polygons are disputed. At the time it issued the report and order, FCC had ongoing investigations into the coverage maps of some wireless providers, and therefore limited the new data collection obligations to fixed broadband providers while seeking comment on how best to incorporate mobile wireless coverage data into the effort in the future. FCC also proposed the use of public input to help verify the accuracy of the polygons and sought comment on whether it should discontinue the collection of census block data if the polygon-based deployment data prove to be gathering better deployment data once it is established. Additionally, on March 23, 2020, the Broadband Deployment Accuracy and Technological Availability Act was enacted. The act directs FCC to issue final rules on data collection for both fixed and wireless deployment within 180 days. In commenting on a draft of this report, FCC noted that it is in the process of implementing the statute, and that the statute largely affirms rules that FCC adopted in August 2019 but differs in some respects. FCC also noted that, while it is working to implement the requirements of the act, it is unable to comply with all of the requirements without a further appropriation. This change has the potential to improve how both FCC and RUS target deployment gaps by providing more accurate and granular information that could better identify truly unserved areas and results in better targeting of federal funds to those areas. As we discussed above, our prior work has found that FCC’s current deployment data lack accuracy in a manner that overstates where consumers have access to broadband, which, in turn, limits federal agencies’ efforts to effectively target their broadband funds. Specifically, our 2018 report noted that overstating access increases the risk that unserved areas remain unserved, since areas that deployment data show as served are not eligible for funding. Although that report specifically looked at this risk for tribal areas, it is potentially a concern for other unserved areas. FCC officials noted that improved data would help inform future funding under the high-cost program by more accurately targeting unserved areas. RUS officials told us that they use FCC deployment data as a source for RUS mapping and program eligibility requirements for the ReConnect Program. RUS officials also believe that their program would benefit from more accurate deployment data. While efforts are under way to improve deployment data and mapping efforts, both FCC and RUS are continuing to move forward with their programs for funding broadband infrastructure in underserved and unserved areas. The lack of accurate data regarding locations that are not served by broadband may affect the ability of these agencies’ programs to target federal funds. In particular, providers and industry associations noted there was a risk of federal programs subsidizing deployment into areas that already have service, at the expense of an unserved area that does not have any service. Given this risk, FCC and RUS each have “eligible area” validation processes that they use to determine if areas are already being served and therefore ineligible for federal support. Additionally, FCC and RUS have engaged in interagency coordination efforts to keep each agency’s program staff apprised of key dates and issues in an effort to avoid overlap between the programs. Officials from both FCC and RUS have stated that their programs are complementary and noted that their eligibility validation processes reduced the likelihood of service overlaps. The agencies use different processes to determine which geographic areas are eligible for funding, and they share information about the results of these processes as they are able. According to FCC officials, the agency has used a process for validating unserved areas in connection with its support of the Universal Service high-cost program. This validation process primarily relies on providers verifying the Form 477 data they self-report, and then using that data to create and publish a list of unserved census blocks prior to awarding funding. Results from this process have informed how FCC establishes eligible areas, such as those eligible to receive funds in the Connect America Fund’s second phase of awards. A similar process is under way in connection with the Rural Digital Opportunity Fund. In an additional step to address concerns regarding program overlap, FCC officials noted that the Rural Digital Opportunity Fund would exclude census blocks that have been awarded funding through the RUS ReConnect Program. RUS’s process to verify eligible areas includes provider participation in the verification, as well as onsite testing and research by field staff, to independently verify eligibility of the geographic areas in each ReConnect grant or loan application. RUS officials said they use the publicly available FCC data on unserved areas as a key factor in eligibility decisions. Specifically, RUS officials said that they focus ReConnect eligibility on areas FCC reported as unserved as of 2015 when the first round of FCC’s Connect America Fund program was started. They also noted that throughout the application and funding process, they seek input from providers through public notices and emails to solicit feedback on whether areas selected for proposed funding are already served. After receiving provider feedback on eligible areas, RUS then deploys field staff to conduct on-location tests and other reviews as necessary to determine if the area was unserved prior to funding. In addition to their eligibility validation steps, FCC and RUS officials told us they share information about where their broadband deployment programs are funding new deployments, as well as other relevant information related to program activity, such as the timing of program applications and awards. FCC and RUS officials told us that they share program information through participation in interagency meetings and working groups that focus on broadband deployment issues and through posting relevant program information—such as funding decisions—online. For example, the Chief of the FCC’s Wireline Competition Bureau and the RUS Administrator, along with other FCC and RUS staff, met in July 2019, January 2020, and February 2020 to discuss issues related to their respective broadband funding programs, including the roll out of the Rural Digital Opportunity Fund and second round of the ReConnect Program. FCC officials told us that at these meetings agency representatives discussed the anticipated timing of elements of their respective programs and ways in which to maximize coordination and avoid overlap. Officials from both agencies also noted there has been ongoing communication between FCC and RUS at the staff level concerning program status and developments through phone calls and meetings. In addition to this ongoing coordination between RUS and FCC, representatives of FCC and RUS also said they participate in relevant working groups through the American Broadband Initiative, such as the Initiative’s Federal Funding Workstream, which meets bi-weekly to discuss broadband funding and deployment. The range of collaboration activities undertaken by RUS and FCC staff is especially important because both agencies have similar goals but different timelines for moving forward with their programs. For example, as of March 2020, FCC had the eligible area validation process under way for the Rural Digital Opportunity Fund. Although some timelines are still to be determined, FCC plans to start bidding on the program’s auction phase in October 2020. At the same time, RUS was in the midst of announcing approved projects for the first funding round of the ReConnect Program, and it had applications open for the second round with applications due by April 15, 2020. FCC officials noted that they plan to maintain close coordination with RUS to reduce the likelihood of overlap with any areas that may be deemed eligible to receive ReConnect funding in the program’s second round. Through prior reforms of their respective broadband programs, FCC and RUS have taken steps to try and effectively target federal dollars to support broadband deployment while avoiding the potential to duplicate funding in an area. Continuing to improve collaboration and information sharing regarding eligibility and program timelines will be critical for both agencies to achieve greater efficiency in their program’s ability to target funds to unserved areas and thus make progress toward closing the deployment gap. We provided a draft of this report to the Federal Communications Commission, the Department of Agriculture’s Rural Utilities Service, and the Department of Commerce’s National Telecommunications and Information Administration for comment. FCC and RUS provided technical comments, which we incorporated as appropriate. NTIA had no comments. We are sending copies of this report to the appropriate congressional committees, the Chairman of the Federal Communications Commission, the Secretary of the Department of Agriculture, the Secretary of the Department of Commerce, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Another estimate of capital expenditures by USTelecom—a broadband industry association—uses a different scope and methodology than the U.S. Census Bureau’s Annual Capital Expenditures Survey. The association reports total capital expenditures for U.S. broadband providers, as shown in table 2. USTelecom’s data include expenditures from fixed (wireline), mobile (wireless), and cable companies. Its primary source of data are publicly traded companies’ financial statements filed with the Securities and Exchange Commission. USTelecom also makes estimates for companies that do not report financial information publicly. Its data exclude some companies, such as satellite providers, telecommunication resellers, and electric utilities. USTelecom publishes estimates annually for advocacy or research purposes. Its capital expenditures data differ from the Census Bureau’s Annual Capital Expenditures Survey estimates for telecommunications industry in scope, methodology, and timing, among other things. For instance, the Census Bureau’s survey is broader in scope. Specifically, in addition to the types of companies USTelecom includes in its data, the Census Bureau’s survey data include satellite providers, resellers, and other telecommunications providers. In addition, the Census Bureau collects data through a survey instrument from both publicly traded and privately held companies across the United States. It makes statistical inferences about the capital expenditures for the entire telecommunications industry, whereas USTelecom collects data mainly from financial reports for a defined set of providers and makes estimates for companies that do not report financial information publicly. Moreover, USTelecom typically releases its capital expenditure data within a year after companies release their financial data while preliminary survey results from the Census Bureau are made public about 2 years after companies report and data reliability assessments occur and the final revised results are made public about 3 years after companies report. In addition to the USTelecom and Census Bureau estimates, investment firms, such as Goldman Sachs and UBS, also estimate or may report industry capital expenditures for selected publicly traded companies providing broadband service. In addition to the contact named above, Andrew Huddleston (Assistant Director); Steve Martinez (Analyst in Charge); Oluwaseun Ajayi; Michelle Bacon; Carl Barden; Melissa Bodeau; Hannah Laufe; Dan Luo; Malika Rice; Sandra Sokol; and Betsey Ward-Jenks made key contributions to this report.", "summary": "Broadband is critical for economic, educational, and personal uses. Industry and federal investments have made broadband available to the vast majority of Americans. For example, FCC's high-cost program provides funding to broadband providers to deploy broadband in rural, insular, and high-cost areas. However, some rural areas continue to lack access due, in part, to challenges with providing service to areas where deployment costs are high and returns on investment are low. GAO was asked to examine the current state of broadband investment and deployment. This report examines (1) industry and federal investments to deploy broadband in the United States since 2009, and (2) efforts federal agencies are making to address deployment challenges. GAO analyzed industry and federal government data from 2009 through 2017 or 2018 (the most recent year of available data) on broadband investments and deployment; reviewed statutes and regulations, rulemaking proceedings, and FCC, RUS, and NTIA program information; interviewed federal officials; and obtained information about deployment challenges from interviews with 32 industry, academic, and consumer stakeholders, including 16 broadband providers selected to represent a range of provider sizes and types of technologies. Telecommunications industry and federal government investments have expanded access to broadband in the United States. From 2009 through 2017, the industry made capital investments of about $795 billion, including investments in broadband infrastructure, according to U.S. Census Bureau survey data. Federal investments totaled about $47.3 billion to target broadband infrastructure for rural areas over the same time, according to data from the Federal Communications Commission (FCC), the Rural Utilities Service (RUS), and the National Telecommunications and Information Administration (NTIA). FCC's Universal Service Fund high-cost program expanded service to about 2.3 million residential and small business locations, mostly from 2015 through 2017, according to data FCC collects from providers. FCC reported that fixed broadband service was available to 94.4 percent of the U.S. population in 2018, up from 81.2 percent in 2012, although affordability and digital literacy remain barriers to adoption and use. While service availability for people in rural areas increased from 45.7 percent in 2012 to 77.7 percent in 2018, service in rural areas continues to lag behind urban areas, according to FCC's broadband availability report (see figure). FCC and RUS have taken actions to address deployment challenges, such as taking steps to improve their ability to pinpoint where gaps in broadband deployment still exist. In August 2019, FCC proposed an initiative to change how it collects broadband deployment data, with the goal of using a new methodology to improve data accuracy and FCC's ability to target funds to locations that lack access. FCC and RUS have also coordinated on broadband deployment issues. For example, to avoid funding areas where broadband service is already deployed, agency officials regularly communicate on information about where their broadband deployment programs are funding new deployments. Continued communication and coordination on topics such as collecting and using improved data will be especially important in assuring that federal dollars are effectively targeted as agencies' efforts to improve mapping and target resources progress. In prior work, GAO recommended that FCC improve its mapping data and RUS better manage its broadband programs. The agencies have addressed some, but not all, of the recommendations. FCC and RUS reviewed a draft of this report and provided technical comments.", "document_type": "gao"}
{"report": "To adjudicate a self-petition filed by a foreign national claiming to have suffered domestic abuse, USCIS adjudicators determine whether the self- petitioner has established the statutory eligibility requirements. A foreign national satisfies the applicable eligibility requirements by demonstrating that he or she (1) has a qualifying relationship with a U.S. citizen or LPR, such as a marriage; (2) was battered or subjected to extreme cruelty by his or her U.S. relative during the qualifying relationship; (3) is residing or has resided with the abusive U.S. citizen or LPR during the qualifying relationship; and (4) is of good moral character. A foreign national filing a VAWA self-petition as an abused spouse is also required to demonstrate that he or she entered into or intended to enter into the marriage in good faith and not in order to evade U.S. immigration law. For a good moral character determination, the petitioner typically should submit a local or state police clearance letter or a state-issued criminal background check from each place where he or she has lived for 6 months or more in the past 3 years immediately prior to filing the VAWA petition. The burden of proof is on the self-petitioner to demonstrate, by a preponderance of the evidence, that he or she has satisfied the statutory eligibility requirements. Considered evidence may include, for example, a criminal background check to establish the good moral character of a self- petitioner or testimony in the form of an affidavit to establish abuse on the part of the U.S. citizen or LPR relative. If the self-petition is approved, the point at which the petitioner will be able to apply for and obtain LPR status will depend on whether he or she is an immediate relative of a U.S. citizen (i.e., U.S. citizen’s unmarried child under age 21, spouse, or, where the citizen is at least 21, their parent), or other relative of a U.S. citizen or LPR, who, unlike immediate relatives, are subject to annual immigration limits. Under U.S. immigration law, there are confidentiality protections for VAWA self-petitioners. Any information about the self-petitioner is considered confidential and, with certain exceptions, officials from DHS are prohibited from releasing any information about the petitioner, including that the petitioner has sought immigration relief. In addition, adjudicators are prohibited from using information provided solely by the alleged abuser to make an adverse determination of admissibility or deportability against self-petitioners, unless such adverse information has been corroborated through independent sourcing consistent with departmental policy. Finally, according to DHS policy, DHS officials typically do not take enforcement actions, such as executing an order of removal, against abuse victims when they are present at certain locations, such as domestic violence shelters, victims’ services programs, and community-based organizations. The self-petition adjudication process begins when a foreign national submits a Form I-360, “Petition for Amerasian, Widow(er), or Special Immigrant,” with supporting evidence, to USCIS. USCIS’s Vermont Service Center then begins the pre-adjudication phase and takes several actions. First, the service center makes a prima facie determination, which is an initial review of self-petition filings, to determine whether the self-petitioner has submitted evidence that, on its face, is responsive to each of the eligibility requirements noted above, in order to allow qualified aliens access to certain public benefits, if needed. If the self-petitioner has not submitted evidence to address each of the eligibility requirements, USCIS policy directs the service center to issue a request for evidence to the self-petitioner to provide additional evidence for the full adjudication of the petition. In addition, the service center conducts a safe address assessment on the self-petition to identify the address to be used for future communications with the self-petitioner to protect the self- petitioner’s confidentiality and safety. Finally, the service center’s Background Check Unit uses the TECS database to determine whether the self-petitioner is connected to any administrative or criminal investigations, is the subject of a national security concern, or is a public safety threat. The Vermont Service Center also checks the TECS database to determine whether any derogatory information exists on the foreign national that may impact the submitted self-petition. Figure 1 provides an overview of the USCIS self-petition process. To begin the adjudication phase, an adjudicator incorporates a self- petition filing into the self-petitioner’s Alien file. Adjudicators stated they review the evidence available in the self-petition filing and the Alien file and generally take one of three actions—approve, deny, or refer the petition for an administrative investigation. Adjudication may also be withheld. Approve. If a USCIS adjudicator determines that the evidence submitted by the self-petitioner satisfies the eligibility requirements noted above, the self-petition is approved. Once USCIS approves a self-petition, DHS will generally defer any removal action against the individual, as he or she goes through the process of applying for LPR status. According to USCIS data, of the 82,357 self-petitions adjudicated from fiscal year 2009 through fiscal year 2018, 72 percent were approved. Self-petitioners who obtain LPR status are not eligible for U.S. citizenship until they have been an LPR in the United States for at least 3 years. Deny. An adjudicator may deny a self-petition if the petitioner has not demonstrated that he or she is more likely than not eligible for petition approval, considering all credible evidence provided by the self- petitioner. In some circumstances, an adjudicator will issue a request for evidence to the petitioner to provide an opportunity for the petitioner to send additional information or documents. In response to this request, the petitioner has an opportunity to provide additional evidence; if that evidence does not sufficiently demonstrate that the petitioner meets the eligibility requirements, or additional evidence is not provided, USCIS may deny the self-petition. In other circumstances, an adjudicator will issue a notice of intent to deny to the self-petitioner in cases where it does not appear likely that the self-petitioner could overcome the deficiencies. This provides the self- petitioner an opportunity to respond. If the self-petitioner’s response does not sufficiently demonstrate that the petitioner meets the eligibility requirements or a response is not provided, the self-petition is subsequently denied. An adjudicator may also deny a self-petition if the petitioner abandons his or her self-petition or withdraws the self- petition by providing notice to USCIS in writing. According to USCIS data, among self-petitions adjudicated from fiscal year 2009 through fiscal year 2018, about 28 percent were denied. Of that, about 3 percent were withdrawn, revoked, or closed administratively. If a self- petition is denied and the self-petitioner has other valid immigration status, he or she may remain in the United States. Otherwise, the self- petitioner may be placed in removal proceedings. Adjudication withheld. An adjudicator may also withhold adjudication of a visa petition or other application if there is an ongoing investigation involving eligibility, in connection with a benefit request, and disclosure of information to the applicant or petitioner concerning the adjudication would prejudice the investigation. If adjudication is withheld from a self-petition, USCIS takes no further adjudicative action at that time, pending completion of the related investigation. Refer a petition for an administrative investigation. In addition to approving or denying a self-petition, an adjudicator may refer a self- petition to CFDO for an administrative investigation in cases when an adjudicator suspects fraudulent activity within the self-petition. In such cases, CFDO completes an administrative investigation and returns a Statement of Findings to the adjudicator. The Statement of Findings indicates whether fraud was found, not found, or whether the administrative investigation was inconclusive in finding fraud. After reviewing the Statement of Findings, immigration officers stated the adjudicator continues the adjudication process for the self-petition and may ultimately approve or deny the self-petition. According to USCIS data, the total number of VAWA self-petitions filed by foreign nationals increased from 7,360 in fiscal year 2014 to 12,801 in fiscal year 2018, an increase of about 74 percent. The number of filings by spouses—a subset of the above petitioners—increased from 7,131 in fiscal year 2014 to 11,213 in fiscal year 2018, an increase of 57 percent. Filings by spouses represented about 93 percent of self-petition filings from fiscal year 2014 to fiscal year 2018. See table 1. Immigration benefit fraud involves the willful or knowing misrepresentation of material facts for the purpose of obtaining an immigration benefit without lawful entitlement. According to USCIS officials, self-petition fraud is a form of immigration benefit fraud which can occur in a number of ways, such as through document fraud, including submission of falsified affidavits, or making false statements material to the adjudication. For example, a self-petitioner may submit a fraudulent marriage certificate with his or her self-petition in an attempt to establish a qualifying relationship with a U.S. citizen or LPR. Or a self-petitioner may submit a fraudulent affidavit falsely attesting that he or she was battered or subjected to extreme cruelty during the qualifying relationship with the U.S. citizen or LPR. For the purposes of this report, self-petition fraud is construed broadly to include any misrepresentation of material fact(s), such as making false statements, submitting forged or falsified documents, or conspiring to do so, in support of a VAWA self-petition. USCIS may deny, or revoke approval of, a self-petition upon determining that the self-petitioner is, or was, not eligible for petition approval by a preponderance of evidence, due to fraud material to the adjudication process. While it is unlawful to fraudulently obtain approval of an immigration benefit, U.S. immigration law does allow VAWA self- petitioners who may have committed such fraud to retain eligibility for LPR status when they or their family would otherwise suffer extreme hardship. GAO’s A Framework for Managing Fraud Risks in Federal Programs (Fraud Risk Framework) is a comprehensive set of leading practices that serves as a guide for program managers to use when developing efforts to combat fraud in a strategic, risk-based manner. The framework describes leading practices for establishing an organizational structure and culture that are conducive to fraud risk management; assessing fraud risks; designing and implementing controls to prevent and detect potential fraud; and monitoring and evaluating to provide assurances to managers that they are effectively preventing, detecting, and responding to potential fraud. Under the Fraud Reduction and Data Analytics Act of 2015, agencies are required to establish financial and administrative controls that are aligned with the Fraud Risk Framework’s leading practices. In addition, guidance from the Office of Management and Budget affirms that managers should adhere to the leading practices identified in the framework. The Fraud Risk Framework includes control activities that help agencies prevent, detect, and respond to fraud risks, as well as structures and environmental factors that influence or help managers achieve their objectives to mitigate fraud risks. The framework consists of four components for effectively managing fraud risks: commit, assess, design and implement, and evaluate and adapt. Leading practices for each of these components include the following: Commit: create an organizational culture to combat fraud at all levels of the agency, and designate an entity within the program office to lead fraud risk management activities. Assess: assess the likelihood and impact of fraud risks and determine risk tolerance and examine the suitability of existing controls and prioritize residual risks. Design and implement: develop, document, and communicate an antifraud strategy, focusing on preventive control activities. Evaluate and adapt: collect and analyze data from reporting mechanisms and instances of detected fraud for real-time monitoring of fraud trends, and use the results of monitoring, evaluations, and investigations to improve fraud prevention, detection, and response. Figure 2 provides an overview of the Fraud Risk Framework and its control activities. USCIS has an antifraud culture and a dedicated entity for managing fraud risks in the self-petition program. The first component of GAO’s Fraud Risk Framework—commit—provides that agencies should commit to combating fraud by creating an organizational culture and structure conducive to fraud risk management. In particular, agencies should create an organizational culture to combat fraud at all levels, by demonstrating a senior-level commitment to integrity and combatting fraud, and by involving all levels of the agency in setting an antifraud tone that permeates the organizational culture. The first component of the Fraud Risk Framework also calls for an agency to create a structure with a dedicated entity to lead fraud risk management activities. Consistent with the Fraud Risk Framework, we found USCIS has promoted an antifraud culture in several ways. It has demonstrated a senior-level commitment to combating fraud and involvement at all levels. Within the Vermont Service Center, senior officials who oversee the VAWA self-petition unit, as well as adjudicators who review petitions, are evaluated on activities related to managing fraud risks in the self-petition process. For example, according to performance appraisal documentation, senior officials are evaluated on their ability to consistently identify immigration fraud. Specifically, experienced adjudicators and supervisors stated that they are evaluated on their ability to review fraud referral sheets submitted by adjudicators to determine whether the adjudicator has appropriately identified and described suspected fraudulent activity in a self-petition. In addition, senior officials told us they independently review a sample of self-petitions adjudicated during each fiscal year for quality assurance purposes, to include identification of suspected fraud. Adjudicators are evaluated by their supervisors on their ability to identify fraud within the self-petition adjudication process, which includes identifying suspected fraudulent activities in self-petitions, submitting fraud referral sheets to their supervisors, and collaborating with CFDO on resolving self-petition adjudications where suspected fraudulent activity has been identified. In addition to being evaluated on their ability to identify fraud, officials have implemented several activities that contribute to an antifraud tone. For example, officials at the Vermont Service Center stated that VAWA self-petition unit adjudicators and CFDO immigration officers collaborate and share information to combat potential fraud through activities that include monthly meetings, regular contact through their co-location, and an electronic bulletin board. Officials stated that during monthly meetings, immigration officers answer questions from adjudicators on fraudulent schemes and activities uncovered in their administrative investigations of self-petitions. In addition, adjudicators we spoke to stated that because they are co-located with CFDO, they have direct access to immigration officers to obtain feedback on identifying suspected fraudulent self- petitions prior to submitting a formal fraud referral sheet. Finally, CFDO maintains an electronic bulletin board for sharing information with adjudicators on new potentially fraudulent activities they have identified through their administrative investigations. Adjudicators we spoke to stated that the bulletin board assists with identifying fraud indicators during adjudication. We also found that USCIS has created a dedicated entity to lead fraud risk management activities for the self-petition program. According to USCIS officials, the CFDO unit at the Vermont Service Center, in conjunction with FDNS headquarters, is that dedicated entity. Within the Vermont Service Center, CFDO officials stated the CFDO unit consists of three immigration officers and a supervisory immigration officer who have defined antifraud responsibilities, such as conducting administrative investigations of self-petition filings that are referred by adjudicators who suspect fraudulent activity. In addition, the immigration officers are responsible for liaising with law enforcement entities, such as ICE HSI, to provide logistical support in law enforcement matters. According to the officials, CFDO and FDNS fulfill other fraud risk management responsibilities described in GAO’s Fraud Risk Framework, including leading or assisting with fraud training for adjudicators. While USCIS has taken some steps to assess fraud risks in the self- petition program, the agency has not conducted a formal assessment of such program risks. The second component of the Fraud Risk Framework—assess—calls for federal managers to plan regular fraud risk assessments, and to assess risks to determine a fraud risk profile. A fraud risk profile is the summation of key findings and conclusions from a fraud risk assessment, including the analysis of the types of internal and external fraud risks, their perceived likelihood and impact, managers’ risk tolerance, and the prioritization of risks. The fraud risk assessment should be tailored to the program, and in identifying and assessing risks to determine the fraud risk profile, the focus should be on likelihood and impact of inherent fraud risks. This means not only fraud risks already known through program experience, but also other fraud risks that may not yet have been experienced but can be identified, based on the nature of the program. Such risks can be either internal or external to the program. USCIS has not assessed fraud risks and determined a fraud risk profile for the self-petition program, as USCIS officials told us that they were unfamiliar with the concept of a comprehensive fraud risk management process, as provided in the Fraud Risk Framework. Instead, USCIS officials said they viewed fraud risk management more practically, from the standpoint of adjudicating self-petitions and referring suspected fraudulent activity to CFDO. As part of those efforts, CFDO staff review fraud referrals to determine potential fraud schemes and trends that may exist in the self-petition program. While these are positive steps, they do not constitute an assessment of program fraud risks that would position USCIS to develop a fraud risk profile for the self-petition program. More specifically, the Fraud Risk Framework calls for agencies to identify inherent fraud risks of a program, examine the suitability of existing fraud controls, and then to prioritize “residual” fraud risks—that is, risks remaining after antifraud controls are adopted. According to USCIS officials we spoke with, the self-petition program is vulnerable to fraud. For example, USCIS officials stated that they have seen cases in which self-petitioners submitted false or forged leases in an attempt to show they resided with the alleged abuser during a period of abuse, as well as foreign marriage or divorce certificates later found to be falsified in an attempt to establish that the self-petitioner entered into a marriage with a U.S. citizen in good faith. While these are examples of individual fraudulent activities, USCIS officials cannot be assured they have identified inherent fraud risks to the program without undertaking a fraud risk assessment as provided in the Fraud Risk Framework. USCIS officials we spoke with acknowledged the benefits of conducting a fraud risk assessment and noted that a formal analysis of self-petition fraud referrals and administrative investigations could help to better understand the extent of fraud risks that exist in the self-petition program. Further, the Fraud Risk Framework highlights the need for fully assessing fraud risks when there are changes to the program or operating environment—conditions that apply in the case of the self-petition program. USCIS data indicate that the number of self-petitions filed has been growing in the past 5 fiscal years, and at the end of fiscal year 2018, USCIS had received 12,801 self-petitions and had over 19,000 self- petitions pending adjudication. In this environment, identification of inherent fraud risks, coupled with assessments of the likelihood and expected impact of those risks, could help USCIS better target its fraud prevention and detection efforts. Planning and conducting regular fraud risk assessments, as provided in the Fraud Risk Framework, would better position USCIS to identify fraud risks in the self-petition program. Regularly assessing fraud risks in the self-petition program to determine a fraud risk profile would also help USCIS better determine the extent to which the agency has designed and implemented adequate fraud prevention controls. USCIS has controls designed to help prevent and detect fraud in the self- petition program, but has not developed a risk-based antifraud strategy for the program consistent with the Fraud Risk Framework. The third component of the Fraud Risk Framework—design and implement—calls for agencies to design and implement a strategy with specific control activities to address risks identified in the fraud risk assessment. In particular, managers should develop and document an antifraud strategy based on the fraud risk profile (developed as part of the fraud risk assessment of the second component of the Framework), and design and implement specific control activities to prevent and detect fraud. The basis for these activities should be the prioritized residual risks identified earlier, meaning that the agency adopts a risk-based antifraud control strategy. This approach is in line with Standards for Internal Control in the Federal Government, which requires managers to design a response to analyzed risks. USCIS has instituted some fraud controls for the self-petition program, particularly controls related to preventing and detecting fraud. USCIS’s specific fraud control activities include, for example, the Vermont Service Center Background Check Unit conducting TECS checks on foreign nationals who submit self-petitions during the pre-adjudication stage to determine whether the self-petitioner is connected to any administrative or criminal investigations, is the subject of a national security concern, or is a public safety threat. In addition, USCIS has a process for adjudicators to refer petitions when they suspect fraudulent activities to CFDO for administrative investigation. Specifically, USCIS official stated that in cases where an adjudicator suspects potential fraud in a self-petition, the adjudicator is to complete and submit a supervisor-approved fraud referral sheet to CFDO. After receiving a referral, the center is to determine whether the referral has sufficient information to warrant an administrative investigation. CFDO also provides fraud training to adjudicators. While these controls help USCIS prevent and detect potential fraud in the self-petition program, USCIS has not developed and implemented a risk- based antifraud strategy based on a fraud risk assessment as provided under the Fraud Risk Framework. This is because, as noted earlier, the agency has not undertaken an assessment of inherent program fraud risks. USCIS officials told us that even with adjudicator and CFDO staff experience with identifying and investigating potential fraud in self- petitions, unknown fraud risks may nevertheless remain. USCIS officials acknowledged the benefits of conducting a fraud risk assessment, such as designing and implementing new control activities, as well as revising existing controls, if they determine that controls are not effectively designed to reduce the likelihood or impact of an inherent fraud risk to a tolerable level. USCIS officials told us that adjudicators and CFDO staff conducting administrative investigations have identified trends in fraudulent activities; however, officials also stated that it is difficult for staff to identify fraud risks that are present but that are not identified through adjudication or investigation. Examining antifraud controls, and adjusting them as necessary based on an antifraud strategy, would help the Vermont Service Center to better ensure that its controls are addressing fraud risks in the self-petition program, including inherent risks. USCIS is developing training on fraud-related issues for the self-petition program. The third component of the Fraud Risk Framework, discussed earlier, identifies training as a leading antifraud practice and as an antifraud control to increase fraud awareness of possible fraud schemes. Training and education intended to increase fraud awareness among managers and employees, among others, can serve as a preventive measure to help create a culture of integrity and compliance within the program. Increasing fraud awareness can also enable managers and employees to better detect potential fraud. To achieve these benefits, the Fraud Risk Framework notes that a leading practice is to require all employees to attend training upon hiring and on an ongoing basis thereafter. Training should convey fraud-specific information that is tailored to the program and its fraud risk profile. Specifically, it should include information on fraud risks, such as examples of specific types of fraud that employees are likely to encounter, and information on how to identify fraud schemes. USCIS has a training program in place for new adjudicators that provides general information on identifying potential fraudulent activities as part of any adjudication and has plans to develop new fraud awareness training tailored specifically to the self-petition program. According to CFDO officials, USCIS provides general training to new adjudicators during a 6- day classroom training program. During this training, new adjudicators are instructed on eligibility and evidence requirements across several application types, including the VAWA self-petition. The training includes information on eligibility requirements, supporting documentation needed, and evidentiary requirements for these applications. Application forms are used to teach adjudicators fraud identification, and adjudicators are given a list of common fraud indicators to assist when reviewing applications, according to adjudicators. This training also includes a 2-hour presentation on the VAWA self-petition program where general fraud concepts, such as document fraud, are discussed. While adjudicators receive general training when hired, USCIS had not provided tailored antifraud training on the self-petition prior to fiscal year 2019. Adjudicators we spoke to noted that fraud schemes continue to evolve, and that fraud schemes and tactics are becoming more sophisticated and thus more difficult to identify during adjudication of VAWA self-petitions. Adjudicators we spoke to also noted that ongoing training that included information on evolving fraud schemes and tactics specific to the self-petition program would help increase their ability to identify potentially fraudulent activities. Further, adjudicators noted that additional training on how to identify potential fraud when a petitioner submits an attested affidavit would help to identify potentially fraudulent self-petitions. In response to our discussions and adjudicator feedback, a senior CFDO official stated that they recognized the need for fraud training, including training tailored to the self-petition program, and planned to hire an additional four immigration officers in fiscal year 2019 to increase fraud training for adjudicators, among other duties. In response to discussions we had during our review, officials at the center also stated they planned to develop and implement tailored fraud training for the self-petition program by the end of fiscal year 2019. CFDO officials stated they also plan to continually update the training based on any new potentially fraudulent activity identified in the self-petition program. USCIS has data analytics capabilities, but has not applied these capabilities as an antifraud tool for the self-petition program. The third component of the Fraud Risk Framework, discussed earlier, cites data analytics as a leading practice in developing specific control activities to prevent and detect fraud—in particular, to mitigate the likelihood and impact of fraud. In addition, Standards for Internal Control in the Federal Government provide for ongoing monitoring of operations and internal controls, and data analytics can aid in this task. According to the Fraud Risk Framework, data analytics can include a variety of techniques, such as data mining (identifying suspicious activity or transactions, including anomalies, outliers, and other red flags, within data) and data matching (comparing information in one source to another to identify inconsistencies), which can enable programs to identify potential fraud. Further, predictive analytics can identify particular types of behavior, including fraud, before transactions are completed. According to USCIS officials, the agency has developed and uses data analytics capabilities as part of its efforts to identify and prevent fraud within immigration benefit programs. These officials said the agency has not had sufficient resources to pursue data analytics separately for each type of immigration benefit program. Thus, they stated that USCIS deploys its data analytics resources strategically across immigration benefit programs, based on factors including, among other things, the volume of filings or applications for specific benefit programs, the amount of data available for electronic analysis, and whether the type of application is one that can lead to a change in immigration status, such as asylum or permanent residency. Under this approach, for example, USCIS officials stated that marriage and employment-based benefit programs are areas where there is a greater amount of electronic data available for analysis. USCIS’s development and use of data analytics in other programs are positive actions in helping the agency in its efforts to prevent and detect fraud risks to immigration benefit programs. However, USCIS has not conducted a comprehensive assessment of fraud risks in the self-petition program to provide an understanding of the likelihood and impact of program risks and to help inform the level of resources USCIS should allocate to addressing those risks. Consistent with the Fraud Risk Framework, using data analytics capabilities in the self-petition program could help position USCIS to better identify and assess fraud risks in the program. Such data analysis does not by itself necessarily confirm the existence of fraud, but the use of data analytics could help USCIS to determine indicators of potential fraud. Further, consistent with the Fraud Risk Framework, this type of analysis can aid in decisions on prioritization of investigative resources. According to the Fraud Risk Framework, specific data analytic tests that are most effective in helping managers prevent or detect potential fraud vary by program because of the different fraud risks programs face. By using information on previously encountered fraud schemes or known fraud risks, managers can identify signs of fraud that may exist within their data. In the absence of an assessment of fraud risks in the self- petition program, we asked USCIS officials about fraud risks or schemes they have identified in the program and analyzed program data to identify examples of ways USCIS could use program data to better prevent or detect potential self-petition program fraud. As examples, we analyzed variables that generally serve to identify individuals, such as address and Social Security number, because multiple instances of the same identifier in program data can indicate misuse of personally identifying information. In addition, we examined other variables associated with self-petition filings and outcomes of self-petition adjudications, as trends in variables associated with denial outcomes, for example, can provide indicators of potential fraud. We offer the following examples not as illustrations of confirmed or even potential fraud, but rather to help illustrate the use of data analytics as a tool for helping to prevent and detect fraud in the self- petition program. For example, one area in which we identified multiple instances of the same variable was with addresses. While not necessarily indicative of fraud, our review of USCIS data showed that from fiscal year 2009 to January 2019, thousands of self-petition filings had addresses that were used in multiple self-petition filings. According to USCIS officials, this is not unexpected and further research would be required to authoritatively explain the multiple address use we identified. The self-petitioner program also allows self-petitioners to use a “safe address” for communications, in an effort to ensure confidentiality in filing of the petition. According to USCIS officials, self-petitioners often use an assisting attorney or representative’s business address as their safe address. In prior work on other immigration benefits, we have highlighted where DHS officials have used multiple instances of the same address in program data to target investigative follow-up. Our analysis of data on the number of times unique addresses were used in filing self-petitions showed, for instance, 37,201 filings had addresses used at least 10 times each from fiscal year 2009 to January 2019. In some cases, an address was used hundreds of times—–in a group of 6,302 self-petitions, there were 31 instances in which addresses were used 100 or more times. Table 2 provides examples of multiple uses of addresses, which we selected for illustrative purposes from among all the multiple uses of addresses we identified. It shows, for example, in the last row, that there was one unique address that was used 845 times in self-petition filings, all of which were separate filings. Thus, the total number of self-petitions involved with this address was 845. Another example of multiple instances of the same variable was identification numbers. In particular, our review of USCIS data showed that from fiscal year 2009 to January 2019, there were thousands of self- petition filings that used duplicative identifying numbers – Social Security numbers and Alien numbers. According to USCIS officials, as with multiple uses of the same address, further research would be required to authoritatively explain the multiple identification number use we identified. For example, according to USCIS officials, a foreign national parent and child may file separate self-petitions, resulting in multiple petitions using the same Social Security number. Also, it is common for people to file more than one self-petition if, for instance, they are able to obtain additional evidence after a decision is made. Our analysis of the number of times unique Social Security numbers were provided in self-petition filings, as well as unique Alien numbers, showed that for each, there were several thousand filings in which the numbers were used in multiple self- petition filings. In prior work, we have highlighted examples where multiple instances of the same Social Security number in program or payment data has indicated Social Security number misuse, such as where multiple individuals may have been using the same Social Security number for employment, and use of Social Security numbers to create synthetic identities, to obtain benefits for ineligible individuals using the Social Security numbers of eligible applicants. Table 3 provides examples of multiple uses of Social Security and Alien Registration numbers, selected for illustrative purposes from among all the multiple uses of identification numbers we identified. It shows, for example, in the last row, that there were 28 instances in which a unique Alien number was used in five different self-petition filings, all of which were separate filings. Thus, the total number of self-petitions involved with these 28 Alien numbers was 140. Another example of multiple instances of the same variables was assistance provided to self-petitioners by attorneys or other organizations. According to USCIS officials, self-petitions filed with assistance are expected, as organizations specialize in providing assistance to petitioners and applicants for immigration benefits, including self-petitions. Thus, USCIS officials noted that the appearance of the same attorneys or other organizations in program data is not necessarily indicative of fraud without further investigation. However, USCIS officials also noted that application “mills,” in which a relatively large number of incomplete or deficient self-petitions are submitted through a single preparer, also exist and could indicate avenues for further investigation. For example, if investigation revealed submission of self-petitioner affidavits or other supporting evidence across multiple self-petitions and using common information, such as duplicate wording, that could be an indicator of potential fraud. In July 2019, the U.S. Attorney for the District of Vermont announced an indictment against a self-petition preparer, charging the man with filing false statements with USCIS, including more than 1,800 fraudulent submissions for more than 1,000 self-petitioners over four years. The preparer is alleged to have falsely claimed that his clients were victims of abuse, without their authorization, according to the U.S. Attorney’s office. Our analysis of USCIS data from fiscal year 2009 to January 2019 showed that a large portion of self-petitions were filed with assistance by either attorneys or by other organizations. In the case of attorneys, according to our analysis, about 80 percent of self-petitions were filed by foreign nationals with assistance from attorneys or accredited representatives from fiscal year fiscal year 2009 through January 2019. However, while USCIS collects attorney identifying information on the paper form that self-petitioners submit, officials told us the agency does not capture this information electronically. Therefore, it is not available for analysis. Such analysis could indicate particular attorneys’ or representatives’ relative shares of self-petitions, and allow USCIS to conduct further analysis, as appropriate. In the case of organizations providing assistance, we found that about 70 percent of self-petitioners from fiscal year 2009 through January 2019 listed various organizations in their filings, and we identified a number of organizations assisting hundreds of self-petitioners each. For example, in one case an organization was listed as providing assistance in over 500 filings and in another case two entities were listed as providing assistance in over 400 filings each. However, according to USCIS officials, one legal organization providing assistance for 500 filings over a 10-year period is not uncommon or necessarily an indicator of fraud, given that, unlike other petitions, most VAWA self-petitions are filed with the assistance of an attorney or legal representative. Consistent with leading practices as described in the Framework, analyses of multiple uses of unique identifiers or instances of certain variables in self-petition program data could help USCIS identify areas for more targeted review, to determine what accounts for the duplicates in the program data and the extent to which they or other variables could be indicators of potential fraud. Moreover, according to the Fraud Risk Framework, data analytics, such as data mining, can identify suspicious activity or transactions, including anomalies, outliers and other red flags in a program’s data. Activity or transactions that deviate from expected patterns can potentially indicate fraudulent activity and program managers who effectively use data analytics to detect potential fraud look for unusual transactions or data entries that do not fit an expected pattern. However, as noted earlier, USCIS has not applied data analytics as an antifraud tool for the self-petition program. For example, as previously discussed, while adjudicating self-petitions, USCIS officers may request additional evidence from petitioners for reasons including incomplete or inconsistent information provided in filings, or suspected fraud, USCIS officials told us. The officials told us the agency does not compile data on the nature of these requests for additional evidence, which number in the thousands annually. Maintaining and analyzing such data—especially when adjudicators are requesting further information because they suspect possible fraud— could provide program-level insights into where self-petitions are incomplete or suspected to be fraudulent. Further, as noted earlier, USCIS does not assess data on the outcomes of self-petition adjudications to determine whether there are any trends or patterns in such data that could be indicative of fraud. In particular, denials or referrals can be based on multiple factors, including potential fraud. Analyzing such outcomes for any patterns or trends that could suggest potential fraud could help USCIS strengthen its efforts to identify and prevent fraud risks in the self-petition program. For example, USCIS officials told us they sometimes observe patterns or trends among self- petitions that may seem suspicious and warrant further review and noted as an example an increase in cases involving potentially false claims of abuse from self-petitioners from one country. While not necessarily indicative of fraud, and to provide some example of trend analysis on data on the outcomes of self-petition adjudications, we analyzed data on the outcomes of adjudications from the 10 countries with the largest number of self-petition filings and found the denial rate by country of birth of the self-petitioner varied by as much as a factor of three. Additional analysis across data on adjudication outcomes could help better identify areas for further investigation or review. In addition, the Fraud Risk Framework notes that one leading practice for using data analytics as an antifraud tool is to verify key information, including self-reported data and information necessary to determine eligibility. To effectively prevent and detect instances of potential fraud, managers are to take steps to verify reported information, particularly self- reported data and other key data necessary to determine eligibility for programs or receiving benefits. For example, according to officials, USCIS does not check the validity of key identification information submitted by self-petitioners, and it does not analyze outcomes across characteristics of self-petitions—practices our prior work indicates could strengthen USCIS’s use of data analytics. More specifically, although USCIS may conduct background checks on self-reported self-petitioner information, officials told us the agency does not have the capability to check the validity of Social Security numbers or passport information that self-petitioners report in their Form I-360 filings. Nevertheless, USCIS officials told us the agency routinely performs overseas verification of self- petitioner documents, such as birth certificates, marriage/divorce certificates, and passports. Based on our analysis of USCIS data, the agency maintains data that could be used for data analytics. For example, the majority of self-petition filings have full name information, addresses, Alien numbers, and, to a lesser extent, Social Security numbers. This relative completeness of data items provides opportunities for data-matching, which, as noted, is a key data analytics technique. USCIS officials told us that generally, they see the value of developing a data analytics capability for the self-petition program, noting that such a capability would be beneficial both in aiding fraud detection and prevention efforts, as well as by allowing timely, accurate reporting on self-petitioner data as part of routine program management and oversight. However, officials also noted that while expanding the range of electronic self-petitioner data maintained would increase analytical capabilities, there would be a cost to implementation, which would need to be balanced against the benefit of the additional antifraud tool, and any data analytics would need to be conducted so as to not target individuals or groups solely based on certain self-petitioner characteristics indicated by data. In other work, we have noted that leading practices in data analytic techniques alone may not be sufficient to prevent fraud in obtaining benefits but can help an agency prioritize and enhance fraud investigations. Developing and implementing a data analytics capability for the self-petition program would provide USCIS with tools to aid in identifying potential fraud in self-petition filings and aid in focusing resources. Further, analysis and insights developed through use of data analytics could inform the self-petition program’s periodic fraud risk assessments, which, as described earlier, are a key aspect of the fraud risk management process. The DHS VOICE Office provides assistance to potential victims of immigration-related crimes. In April 2017, in response to Executive Order 13768, ICE established the VOICE Office to provide professional services and assistance to potential victims and family members of victims of crimes committed by removable aliens. The office’s assistance to potential U.S. citizen and LPR victims includes, among other things, providing ICE community relations officers who serve as local representatives to help potential victims understand the immigration enforcement and removal process; victim assistance specialists who provide potential victims with direct service referrals for matters such as counseling; and information, such as the potential offenders’ immigration and custody status. In addition, the office provides referral information to the ICE HSI tip line and answers questions and concerns regarding immigration enforcement through the VOICE Office’s toll-free hotline. Data collected by the VOICE Office from hotline calls shows that in fiscal year 2018, a total of 1,543 calls were made to the VOICE Office. Of those 1,543 calls, 130 calls, or 8 percent, were from self-identified victims of marriage-related fraud requesting assistance. VOICE officials indicated that they would consider VAWA self-petition fraud as a subset of marriage fraud; however, self-petition fraud is not separately identifiable in their data. Of those 130 calls, the Office referred 78 alleged victims to ICE’s HSI Tip Line. For example, in one case from fiscal year 2018, a caller claimed that his or her spouse married the caller for immigration purposes and attempted to falsely press criminal domestic violence charges against the caller as a means of obtaining immigration status. The Office offered the caller local victim services and referred the caller to both USCIS and the ICE HSI Tip Line. Of the remaining 52 calls from self-identified victims of marriage-related fraud, the office provided the caller with an ICE community relations officer, and the officer recommended actions to victims, such as calling the ICE HSI Tip Line, or providing the victim with a victim assistance specialist to discuss available resources. For example, in another case from fiscal year 2018, a caller claimed his or her spouse married the caller to obtain immigration relief, and falsely accused the caller of domestic violence to obtain legal residency. The VOICE office referred the caller to ICE HSI and explained the victim assistance services available to the caller. See figure 3 for a description of calls made to the VOICE Office and subsequent office action. According to CRCL officials, assessing tips from self-identified victims of immigration fraud poses a challenge, since domestic abusers may use the immigration system against their victims by providing false tips in order to have them removed. Per statutory protections for self-petitioners, DHS treats tips as inherently suspect, and is barred from making adverse determinations of inadmissibility or deportability in adjudications based solely on information provided by certain individuals, such as the alleged abuser or a member of the abuser’s household. However, DHS may consider such information if it can be independently corroborated consistent with DHS policy. As for the alleged abuser’s information, which may have been included in a VAWA self-petition, USCIS officials noted that USCIS never provides such information to anyone including law enforcement even where allegations of criminal conduct are reported with a self-petition. As a result, U.S. citizens and LPRs face no consequences solely from being named in a self-petition regardless of its outcome. DHS has a referral process for suspected fraud in self-petitions, which may result in a referral to ICE for criminal investigation. Within USCIS, FDNS immigration officers review self-petition fraud referrals, conduct administrative investigations when warranted, and in limited circumstances, refer cases to ICE for criminal investigation. Fraud referrals related to self-petitions typically originate from five sources: (1) the TECS checks that the Vermont Service Center Background Check Unit conducts prior to adjudication, which include notifications that indicate potential national security concerns, public safety threats, and fraud leads in the preadjudication screening process; (2) USCIS adjudicators reviewing self-petitions at any time during the adjudication process; (3) other USCIS offices that may encounter potential self-petition fraud in the course of their work on other USCIS applications; (4) other law enforcement sources, including other federal law enforcement entities; and (5) benefit fraud tips received from the general public. After receiving a referral, FDNS immigration officers determine whether the referral has sufficient information to warrant further investigation. According to FDNS’s fraud detection standard operating procedures, FDNS immigration officers either determine that the referral becomes a lead and the lead is accepted, or the referral is declined. After accepting the referral, immigration officers are responsible for conducting an administrative investigation to, among other things, obtain relevant information needed by Vermont Service Center adjudicators to render the appropriate adjudicative decision. If, after conducting research and analyzing the information associated with a lead, the FDNS immigration officer determines that a reasonable suspicion of fraud is articulated and actionable, the lead is elevated to a case. Upon conclusion of the administrative investigation, FDNS immigration officers close the accepted lead and case and record their findings in a Statement of Findings. The Statement of Findings concludes the administrative investigation with one of three types of findings: (1) Fraud Found: the investigation determined fraudulent activities exist in the self- petition; (2) Fraud Not Found: the investigation did not uncover fraudulent activities in the self-petition; or (3) Inconclusive: the investigation could not make a determination of whether fraudulent activity occurred. Once completed, the Statement of Findings is returned to the appropriate referral source. In cases where FDNS immigration officers find self-petition fraud, the case can be referred to ICE’s HSI for criminal investigation. According to a 2008 immigration benefit fraud memorandum of agreement between USCIS and ICE, FDNS is to refer individual petitions involving suspected fraud to HSI where (1) the alien is the subject of a TECS record; (2) USCIS suspects misconduct on the part of the self-petitioner’s attorney, notary, interpreter, or preparer of the application; or (3) evidence of a criminal conviction of an offense that is not grounds for inadmissibility or removability is present, among other things. Typically, referrals are sent to the National Lead Development Center, where they are distributed to ICE Special Agent In-Charge local offices for further investigation, according to FDNS officials. If a referral is the result of a task team, petitions may be referred directly to ICE Special Agent In-Charge local offices. ICE either accepts the referral and conducts a criminal investigation or declines the referral and sends it back to FDNS. If a referral is declined by ICE, FDNS continues its administrative investigation. Figure 4 provides an overview of the self-petition fraud referral process. According to FDNS data, from fiscal year 2014 to March 2019, FDNS created 2,208 fraud referral leads and cases that involved a VAWA self- petition. Total leads and cases increased from 198 in fiscal year 2014 to 801 in fiscal year 2019 (data as of March 2019), an increase of about 305 percent. USCIS officials attributed this increase to an overall increase in the number of self-petitions filed and an increase in fraud leads and cases obtained through USCIS’s fraud tip hotline. FDNS data showed that 2,151 leads and cases were accepted by FDNS between fiscal year 2014 and March 2019, or about 97 percent. Table 5 shows the number of fraud leads and cases that contain a self-petition and the disposition of accepted leads and cases between fiscal years 2014 and March 2019. From fiscal year 2014 to March 2019, FDNS found a disposition for 631 of the closed cases that involved a VAWA self-petition. According to USCIS officials, a fraud lead or case is not typically closed within the same year that it is filed. This accounts for differences between the total number of fraud cases and leads filed and the total number of completed cases and closed leads within the same fiscal year. Of the 631 closed cases with a disposition, FDNS found fraud in 332, or 53 percent. Table 6 shows the disposition of closed self-petition fraud leads and cases between fiscal year 2014 and March 2019. According to FDNS data, from fiscal year 2014 to March 2019, FDNS made 68 fraud referrals to ICE for criminal investigation that involved a VAWA self-petition. We inquired with ICE about the status and disposition of these cases. As previously mentioned, for purposes of accepting a referral for criminal investigation, ICE does not make distinction between self-petition fraud and marriage fraud investigations. As a result, information on the 68 fraud referrals to ICE is encompassed in ICE’s immigration benefit fraud investigation data and could not be separated for analysis. Therefore, we could not provide status and disposition information on these referrals. The VAWA self-petition program is designed to protect foreign nationals who are victims of domestic abuse. The decision to approve or deny a VAWA self-petition is consequential, as the program allows an eligible foreign national victim to remain in the country, obtain work authorization, and apply for LPR status independent of their abuser. According to CRCL, VAWA self-petition relief brings safety, security and stability to legitimate victims who might not otherwise be able to escape domestic abuse. However, approving a fraudulent petition could affect the integrity of the program. USCIS has implemented some aspects of GAO’s Fraud Risk Framework in managing the self-petition program, such as having a dedicated antifraud entity, but could improve efforts to detect and prevent potential fraud in the program. More specifically, conducting regular fraud risk assessments and determining a fraud risk profile for the program could help USCIS identify fraud risks in the self-petition program and better determine the extent to which the agency has designed and implemented adequate fraud prevention controls. Further, basing antifraud controls on inherent risks identified through regular fraud risk assessments could help ensure USCIS’s antifraud controls are addressing fraud risks in the self-petition program. Lastly, developing and implementing a data analytics capability could provide USCIS with tools to aid in identifying potential fraud in self-petition filings. Analysis and insights developed through the use of data analytics could inform the self- petition program’s regular fraud risk assessments. We are making the following three recommendations to USCIS: The Director of USCIS should plan and conduct regular fraud risk assessments of the self-petition program to determine a fraud risk profile, as provided in GAO’s Fraud Risk Framework. (Recommendation 1) The Director of USCIS should develop and implement an antifraud strategy with specific control activities, based upon the results of fraud risk assessments and a corresponding fraud risk profile, as provided in GAO’s Fraud Risk Framework. (Recommendation 2) The Director of USCIS should develop and implement data analytics capabilities for the self-petition program, as a means to prevent and detect fraud, as provided in GAO’s Fraud Risk Framework. (Recommendation 3) We provided a draft of this report to DHS for review and comment. DHS provided comments, which are reproduced in full in appendix I and discussed below. DHS also provided technical comments, which we incorporated as appropriate. In its comments, DHS concurred with our three recommendations and described actions planned to address them. With respect to our first recommendation that USCIS plan and conduct regular fraud risk assessments of the self-petition program to determine a fraud risk profile, DHS stated that the USCIS FDNS plans to capture data digitally for both I-360 and I-751 self-petitions filed on the basis of domestic abuse, and discuss any patterns observed with stakeholders in order to develop a fraud risk profile. Further, DHS stated USCIS will use the results of data analytics to conduct and update regular fraud risk assessments. With regard to our second recommendation that USCIS develop and implement an antifraud strategy with specific control activities based upon the results of fraud risk assessments and a corresponding fraud risk profile, DHS stated USCIS plans to create an antifraud strategy that includes both adjudicators and FDNS officers in order to emphasize fraud detection prior to adjudication of self-petitions. With respect to our third recommendation that USCIS develop and implement data analytics capabilities for the self-petition program as a means to prevent and detect fraud, DHS stated that USCIS will apply their data analytics capabilities, driven by the results of the self-petition fraud risk assessments, to develop analytic tools that verify information provided by self-petitioners and identify indicators of potential fraud. Further, DHS stated that USCIS will use the results of data analytics to inform antifraud training and will distribute the results to USCIS senior leadership when warranted. We are sending copies of this report to the appropriate congressional committees and the Acting Secretary of Homeland Security. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Rebecca Gambler at (202) 512-8777 or GamblerR@gao.gov or Rebecca Shea at (202) 512-6722 or SheaR@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of our report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contacts named above, Jeanette Henriquez (Assistant Director), Kelsey M. Carpenter, Pamela Davidson, April Gamble, Eric Hauswirth, Brandon Jones, Brendan Kretzschmar, Sasan J. “Jon” Najmi, Christopher H. Schmitt, and Eli Stiefel made key contributions to this report.", "summary": "In fiscal year 2018, foreign nationals filed nearly 13,000 VAWA self-petitions alleging domestic abuse by a U.S. citizen or LPR family member. The Immigration and Nationality Act, as amended by VAWA, provides for immigration relief for self-petitioning foreign nationals who are victims of battery or extreme cruelty committed by their U.S. citizen or LPR family member. The self-petition process allows such victims to obtain classification as an immigrant and ultimately apply for LPR status. GAO was asked to review fraud risks in the self-petition process and how, if at all, DHS assists U.S. citizens or LPRs who may have been falsely identified as domestic abusers. This report examines the extent to which (1) USCIS has adopted relevant leading practices in GAO's Fraud Risk Framework for the self-petition program; and (2) DHS provides assistance to U.S. citizens or LPRs who may have been falsely identified as domestic abusers in the self-petition process, and steps DHS takes when suspected fraud is identified. GAO reviewed documents, interviewed officials, analyzed program data, and assessed the agency's approach to managing fraud risks against GAO's Fraud Risk Framework. Within the Department of Homeland Security (DHS), U.S. Citizenship and Immigration Services (USCIS) has responsibility for the Violence Against Women Act (VAWA) self-petition program for foreign national victims of battery or extreme cruelty committed by their U.S. citizen or lawful permanent resident (LPR) spouse or parent, or their adult U.S. citizen son or daughter. According to USCIS officials, the self-petition program is vulnerable to fraud, such as self-petitioners' use of false or forged documents. USCIS has adopted some, but not all, of the leading practices in GAO's Fraud Risk Framework. While USCIS has established a culture and a dedicated entity to manage fraud risks for the program, it has not fully assessed fraud risks and determined a fraud risk profile to document its analysis of the types of fraud risks the program could be vulnerable to. Further, the number of self-petitions filed has grown by more than 70 percent over the past 5 fiscal years. At the end of fiscal year 2018, USCIS had received 12,801 self-petitions and had over 19,000 self-petitions pending adjudication. Planning and conducting regular fraud risk assessments would better position USCIS to identify fraud risks when reviewing self-petitions. USCIS has instituted some fraud controls, such as developing antifraud training for self-petition adjudicators, but has not developed and implemented a risk-based antifraud strategy based on a fraud risk assessment. Developing and implementing an antifraud strategy would help USCIS better ensure its controls are addressing potential fraud risks in the program. DHS provides assistance to victims of immigration-related crimes and refers suspected self-petition fraud for review and investigation. Within DHS, U.S. Immigration and Customs Enforcement provides professional services and assistance to potential victims of immigration-related crimes, including self-petition fraud. As shown in the figure below, USCIS also has a referral process for suspected fraud in self-petitions, which may result in a referral for criminal investigation. According to agency data, from fiscal year 2014 to March 2019, USCIS created 2,208 fraud referral leads and cases that involved a VAWA self-petition. Total leads and cases increased from 198 in fiscal year 2014 to 801 in fiscal year 2019 as of March 2019, an increase of about 305 percent. GAO is making three recommendations, including that USCIS conduct regular fraud risk assessments to determine a fraud risk profile for the program and develop an antifraud strategy with specific control activities. DHS concurred.", "document_type": "gao"}
{"report": "USDA administers its nutrition education programs through multiple agencies in two mission areas—Food, Nutrition, and Consumer Services and Research, Education, and Economics (see fig. 1). Within the Food, Nutrition, and Consumer Services mission area, FNS oversees nutrition assistance programs with nutrition education components, such as SNAP, WIC, and child nutrition programs. For SNAP-Ed and WIC, the FNS national office develops program policies and guidance and works with the FNS regional offices to provide technical assistance to state agencies. The FNS regional offices also review SNAP-Ed and WIC state plans. The Team Nutrition initiative is administered by FNS national officials who also work on child nutrition programs. FNS staff develop Team Nutrition materials, training resources, and guidance and provide assistance to state agencies and local entities overseeing the child nutrition programs. Within the Research, Education and Economics mission area, NIFA national officials oversee EFNEP, in part by providing program guidance, reviewing grant recipient plans, and conducting some monitoring and oversight of local implementing entities. The NIFA national office, together with the FNS national office, administers the FINI program. Although NIFA has primary responsibility over the grant award process, FNS has been overseeing an independent evaluation of program efforts. Interventions for USDA’s nutrition education programs are provided through varied local entities and settings. For example, land-grant universities may provide SNAP-Ed and EFNEP interventions, while local health clinics may provide WIC interventions. USDA’s programs also provide nutrition education in varied settings, ranging from grocery stores to hospitals (see fig. 2). Sometimes multiple nutrition education programs operate in the same setting. For example, SNAP-Ed may provide classes for students while Team Nutrition may distribute teacher training materials and nutrition education curricula to the same school. Most of USDA’s nutrition education programs target interventions to low- income populations with varied characteristics, as shown in table 1, and the programs also differ in how nutrition education fits into their structures. For example, SNAP-Ed and EFNEP are primarily focused on providing nutrition education to participants, while Team Nutrition provides nutrition education to both child nutrition program implementers and participants. WIC provides benefits for food and referrals to health and other social services, as well as nutrition education, including breastfeeding promotion and support, to participants. FINI provides benefits for purchasing healthy foods and may include additional nutrition education programming. Programs also provide nutrition education through various intervention methods, ranging from direct education, such as cooking demonstrations, classes on healthy eating, and one-on-one counseling, to social media campaigns and efforts to change policies, systems, or environments. SNAP-Ed provides direct education through a variety of nutrition educators, although its interventions also may involve social marketing and policy, systems, and environmental changes (PSE). PSE is intended to shape policies, practices, and physical environments to support and improve nutrition education, physical activity habits, and obesity prevention efforts. In fiscal year 2018, approximately 76 percent of SNAP-Ed interventions included direct education, whereas 54 percent included PSE, according to USDA data. EFNEP primarily provides direct education through paraprofessionals, also known as peer educators. Paraprofessionals typically live locally in the community, which allows them to recruit and receive referrals for new participants. University and locally-based professional staff train and supervise the paraprofessionals. In addition, EFNEP has incorporated PSE interventions in recent years. For example, USDA provides PSE training for EFNEP program implementers, as one step toward adopting the PSE approach. WIC programs also provide direct education, such as counseling and group discussions, and, according to federal regulations, are allowed to use other intervention methods as long as they are easily understood by participants and bear a practical relationship to participant nutritional needs, household situations, and cultural preferences. For example, WIC programs may conduct demonstrations or grocery store tours to help consumers understand how to read nutrition labels or shop on a budget. Team Nutrition creates and disseminates web-based and hard-copy educational materials to child nutrition program implementers in part to educate child nutrition program participants. For example, Team Nutrition provides curricula, posters, tools, guides, recipes, and cookbooks for schools and child care sites. Team Nutrition also provides annual grants to enhance nutrition education intervention efforts in schools and child care settings, as well as training for program implementers through its partnership with the Institute of Child Nutrition. FINI supports healthy eating choices by incentivizing the purchase and consumption of fruits and vegetables. For example, some FINI programs provide vouchers redeemable for qualifying fruits and vegetables. Further, according to USDA officials, a FINI program may partner with another USDA nutrition education program, such as SNAP-Ed or EFNEP, to provide nutrition education. USDA agencies also provide nutrition education through other research and guidance directed at the general public: USDA’s Center for Nutrition Policy and Promotion (CNPP), within FNS, works with the U.S. Department of Health and Human Services to develop the Dietary Guidelines for Americans, dietary guidance linking scientific research to the nutrition needs of consumers. CNPP also takes the lead on consumer nutrition education, including MyPlate, which translates the Dietary Guidelines for Americans for consumers. USDA’s Economic Research Service (ERS) conducts research and issues publicly available reports related to promoting the purchase and consumption of healthy, economical foods. ERS also provides data relevant to the nutrition of U.S. households and communities. USDA’s Agricultural Research Service (ARS) serves as a repository for publicly available nutrition education information and data. ARS manages the website Nutrition.gov, the Historical Dietary Guidance Digital Collection, and the FoodData Central data system, which provides food nutrient data for consumers. According to USDA data, 3.8 million and 436,000 people participated in direct education interventions for SNAP-Ed and EFNEP, respectively, in fiscal year 2018. Direct education participation in these two programs, which are focused primarily on nutrition education, has decreased in recent years. Between fiscal years 2010 and 2018, SNAP-Ed direct education participation declined by 33 percent and EFNEP declined by 28 percent. Program officials we spoke with noted some factors that may in part explain these trends. For example, USDA officials said direct education has been less of a focus in SNAP-Ed in recent years, as the department has encouraged programs to use policy, systems, and environmental change interventions and social marketing, in addition to the traditional direct education, following implementation of the Healthy Hunger-Free Kids Act of 2010. USDA officials said that all WIC recipients are offered nutrition education, and therefore they report that 6.9 million people were offered nutrition education through the program in fiscal year 2018. Although officials consider this to be the best proxy for WIC nutrition education participation, more than 5.2 million of these WIC recipients were infants or children ages 5 and under. In addition, WIC recipients do not need to participate in nutrition education to receive the program’s food benefits. As a result, USDA’s proxy overcounts the number of people who participated in WIC nutrition education. For Team Nutrition, USDA tracks the reach of its nutrition education using the volume of materials distributed. Between fiscal years 2012 and 2018, Team Nutrition distributed around 5.1 million of its hard-copy materials, such as curricula, technical assistance and training tools, and other materials, to child nutrition program implementers, including schools and day care providers. Further, from March 2014 through fiscal year 2018, there were about 11 million unique views of Team Nutrition materials hosted on USDA’s Team Nutrition website. Additionally, USDA is collecting participation data for FINI through the FINI National Evaluation. The evaluation is ongoing and FINI participation data will be available after it concludes, according to USDA officials. USDA’s data show that nearly $907 million was expended on nutrition education programs in fiscal year 2017, the most recent year for which complete data are available, with $826 million expended on two programs—WIC and SNAP-Ed (see fig. 3). Specifically, states expended $422 million on WIC nutrition education and nearly $404 million on SNAP-Ed in that year. Further, grantees expended $51 million on EFNEP, $16 million on Team Nutrition, and $13 million on FINI in fiscal year 2017. USDA has total annual expenditure data at a national level for its nutrition education programs, but it does not have detailed information on how the funding is expended that can be routinely analyzed in its two programs with the largest expenditures—WIC and SNAP-Ed. Since both programs allow states to use various types of nutrition education interventions, information on spending by type of intervention may help USDA compare costs, and with additional information, potentially assess the cost effectiveness of various nutrition education interventions. For WIC, USDA collected detailed information on nutrition education spending at the local agency level in 2016 through a survey and analyzed the costs associated with different types of nutrition education interventions. In contrast, USDA collects information on SNAP-Ed local implementing agency expenditures in narrative annual reports that make it difficult to assess spending by type of nutrition education intervention. Through studies and data collection, USDA has gathered some information on the effectiveness of its nutrition education interventions. For example, in 2018, USDA completed the WIC Nutrition Education Study, which assessed WIC nutrition education in both descriptive and evaluative ways (see text box). Additionally, USDA officials said a new study is underway looking at how the WIC nutritional risk assessment tailors the benefit package participants receive, including the nutrition education offered. USDA has also funded various grants and cooperative agreements that have evaluated WIC nutrition education to some extent, according to USDA officials. Findings from WIC Nutrition Education Study: Phase II Report This 2018 study was designed to address research questions about the impact of the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC)’s nutrition education on participant nutrition and physical activity behaviors, among other things, in six pilot sites. Although pilot sites’ nutrition education practices varied, the study did not find significant differences in reported exposure to nutrition education, such as the number of contacts with an educator or receipt of materials to reinforce nutrition messages during visits, or significant differences in participant outcome behaviors, among participants by site. For EFNEP, USDA regularly collects participant data to assess the effectiveness of EFNEP interventions nationwide, and several studies have also assessed the cost effectiveness of EFNEP. Specifically, EFNEP participants take standardized food and physical activity questionnaires and provide information on their dietary consumption in the past 24 hours before and after participating in an intervention, such as a class. EFNEP administrators use this information to measure participant behavior change and also report it to USDA through EFNEP’s data reporting system (see text box). USDA is then able to aggregate these data at the national level and use them to assess the effectiveness of EFNEP interventions nationwide. Further, several studies have evaluated EFNEP cost effectiveness, including one which used national data to estimate EFNEP cost effectiveness by state. Outcomes Reported on Expanded Food and Nutrition Education Program (EFNEP) Participant Questionnaires Questionnaires are administered when adult EFNEP participants enter the program and again when they exit the program to measure behavior change in core areas, including diet quality and physical activity, food safety, food resource management, and food security. For fiscal year 2018, the majority of participants reported improvements in diet quality (92 percent), food resource management (80 percent), food safety (79 percent), and physical activity (78 percent). Further, almost half of participants reported improvement in food security (47 percent). In addition to WIC and EFNEP, USDA is currently collecting information from its grantees on FINI effectiveness as part of its forthcoming FINI National Evaluation. An interim evaluation report found a positive, but modest, impact of FINI on monthly household fruit and vegetable expenditures, but no measurable impact on adults’ daily fruit and vegetable consumption. Previously, USDA assessed the effectiveness of the Healthy Incentives Pilot, which was a predecessor to FINI. The pilot tested the impact of making fruits and vegetables more affordable for SNAP participants and found that participants consumed almost one- quarter of a cup more fruits and vegetables per day than non- participants. USDA has also taken steps to support evaluation of the effectiveness of SNAP-Ed interventions both through its own research and the development of an evaluation framework. In 2012 and 2013, USDA reviewed selected SNAP-Ed interventions to identify potential models of effective SNAP-Ed interventions and impact evaluations. Specifically, USDA evaluated five interventions aimed at increasing fruit and vegetable consumption in preschool or elementary-age children, one intervention aimed at increasing fruit and vegetable consumption in low-income seniors, and one intervention aimed at increasing low-income women’s knowledge of healthy eating choices. Also in 2013, the FNS Western Regional Office began an effort to develop the SNAP-Ed Evaluation Framework (Framework), which was finalized in 2016 and fully adopted for national use. The Framework was developed as a way to evaluate program interventions, and with the intention of encouraging use of policy, systems, and environmental change interventions, according to USDA officials. All states are currently using the Framework to evaluate SNAP-Ed program interventions, according to USDA officials; however, because the Framework allows for myriad ways to measure outcomes, information reported by states on the effectiveness of SNAP-Ed interventions varies widely. Within states, SNAP-Ed implementing entities can select from 51 indicators and various outcome measures in the Framework to evaluate their interventions. Although USDA has identified 7 of the Framework’s 51 indicators as priority indicators, and encouraged states to use these, each indicator has multiple outcome measures and data collection methods associated with it (see text box). Therefore, even if the same indicators are selected to evaluate the effectiveness of different SNAP-Ed interventions, each state may select different outcome measures and data collection methods, and report different information on effectiveness. In our prior work, we found that agencies that seek to manage an excessive number of performance measures may risk creating a confusing excess of data that will obscure rather than clarify performance issues. Elements of a Selected Medium Term Change Indicator: Healthy Eating The medium term change indicator for healthy eating acts as a priority indicator among the 51 indicators included in the Supplemental Nutrition Assistance Program Education (SNAP-Ed) Evaluation Framework. SNAP-Ed programs may use this indicator to measure healthy eating behavioral changes reported by SNAP-Ed participants before and after participation in a series of direct nutrition education classes. Within this indicator, programs may select from various outcome measures and data collection tools: Programs may select from 13 outcome measures to assess the participants on this indicator. Some options include eating more than one kind of vegetable, drinking water, and using MyPlate to make food choices. To assess these outcome measures, programs may select from 11 surveys and other data collection tools compiled by the U.S. Department of Agriculture, such as a food behavior checklist for adults and a beverage and snack questionnaire for older youth. In addition, USDA receives information on states’ evaluations of effectiveness that is not easily analyzed nationwide. Although states report information on SNAP-Ed interventions to USDA in a data system, including information on participation, demographic characteristics of direct education participants, and types of education interventions, the data system is not structured to allow states to report information on intervention effectiveness, including cost effectiveness. Instead, USDA uses SNAP-Ed state plans and annual reports to collect information on state efforts to evaluate program effectiveness, among other things. However, in their plans and reports, states identify the Framework indicators they use and describe their evaluation efforts and outcomes in narrative form, limiting USDA’s ability to aggregate evaluation information across states or interventions, according to USDA officials. One local SNAP-Ed official said her state’s most recent annual report was approximately 60 pages long, highlighting the magnitude of the narrative information some states provide. While USDA officials acknowledged these challenges, they said a narrative report is used to accommodate the differences among SNAP-Ed programs. USDA officials said that because the Framework is still relatively new, they are working to determine both how to assist states’ efforts to use it to evaluate SNAP-Ed effectiveness and to ensure these evaluations provide USDA with useful information for assessing these programs. Further, USDA officials said they are currently in the process of determining future SNAP-Ed reporting protocols to improve program implementation and impact. Federal internal control standards state that agencies should use relevant, quality information from reliable sources to inform decision- making and evaluate performance in achieving key objectives. Without information that can be compared across states or easily aggregated or reviewed nationwide, USDA is unable to assess the effectiveness of interventions used across the country to determine whether SNAP-Ed is achieving program goals. USDA’s national office does not have a formal coordination mechanism for department-wide nutrition education efforts; however, the department has taken some steps to coordinate efforts related to nutrition. For example, USDA convened staff from various program offices in November 2017 for a two-day Intra-Departmental Nutrition Workgroup Meeting. The focus of the meeting was not specifically nutrition education, but included a discussion of current and potential USDA efforts to encourage healthy food choices for certain age groups. The department also has a few committees that address nutrition issues, including the Human Nutrition Coordinating Committee and the Interagency Committee on Human Nutrition Research. Although these committees do not focus on nutrition education, they convene USDA officials and other federal partners on a regular basis. Despite the lack of a focus on nutrition education in these meetings, USDA officials who participated said these opportunities were useful for sharing related information with staff from across the department. USDA has also taken some steps to coordinate efforts across nutrition education programs that have an intersection of target populations, though this has not consistently occurred at the federal level, according to USDA officials. For example, in recent years, WIC officials collaborated with Team Nutrition officials on the development of infant feeding and breastfeeding resources for use in child care settings to ensure consistent messaging. However, USDA officials reported that other programs with similar target populations have not coordinated. For example: USDA officials told us WIC and SNAP-Ed officials have limited interaction, although both programs serve low-income families with young children and coordination could help reinforce key messaging from each program. Several regional SNAP-Ed officials said that they had limited involvement with Team Nutrition, although both programs may serve students in schools and sharing resources could help maximize program impact. Both SNAP-Ed and EFNEP focus on providing nutrition education to similar populations and are delivered by land-grant universities, yet there is limited coordination between the two programs. Regional officials who work on SNAP-Ed reported limited familiarity with EFNEP and said they have learned about EFNEP efforts intermittently through state and local officials, rather than from the national office. Similarly, representatives of the two land-grant universities we spoke with who solely administer EFNEP had limited information regarding SNAP-Ed efforts, though they expressed interest in coordinating efforts to maximize both programs’ reach and avoid duplication of effort. In the absence of formal coordination mechanisms from USDA headquarters, other efforts have developed to help coordinate nutrition education programs nationwide, though USDA national office involvement is limited. Association of SNAP Nutrition Education Administrators: Representatives of SNAP-Ed state implementing agencies formed the Association because they lacked a mechanism to communicate with FNS national office staff or one another on topics related to nutrition education, according to a representative of this group. Officials from FNS’s national and regional offices formally participate in the group’s annual conferences and other activities, but this representative told us that members of the group would appreciate more opportunities to interact directly with these officials. SNAP-Ed Program Development Group: Land-grant universities established this separate SNAP-Ed-focused workgroup to strengthen SNAP-Ed programs and nutrition networks at the state, regional, and national levels, and identify linkages between SNAP-Ed and the land- grant university system’s broader outreach, education, and research mission. SNAP-Ed officials from FNS’s national office do not regularly participate in this group, yet the NIFA administrator of EFNEP sits on the group’s leadership committee. Food, Nutrition, and Consumer Services Nutrition Council: This group convenes national and regional staff in the Food, Nutrition, and Consumer Services mission area on nutrition-related topics and is currently led by regional officials, although the group was previously led by both national and regional officials. FNS officials told us the Nutrition Council has not regained momentum at the national office level since leadership transitioned to the regional office level, and one regional official with leadership responsibilities on the Council told us the group would benefit from more leadership support from FNS national office staff. State Nutrition Action Councils (SNACs): At the state level, SNACs are primarily comprised of state representatives from FNS programs and develop statewide cross-program nutrition education plans. FNS’s national office has supported SNACs as a model for coalescing state programs around nutrition education and obesity prevention efforts but has delegated leadership of the SNACs to the regions, who work directly with state agencies. USDA does not have a dedicated individual or entity with leadership responsibility for nutrition education, and program staff who work on nutrition education are currently focused on their individual programs, according to USDA officials. Although FNS has a senior nutrition advisor who supports national and regional officials who work on FNS programs, the advisor’s role does not encompass department-wide coordination on nutrition education. Further, program staff whose responsibilities include nutrition education serve the needs of their individual programs and lack formal communication channels with one another, according to USDA officials. Previously, from 1998 through 2008, USDA had a centralized Nutrition Services Staff that served as a formal coordinating entity for FNS and held cross-program nutrition education meetings, which were useful for information sharing, according to USDA officials. In 2008, this division, which had been comprised largely of nutritionists, was dissolved, with its staff with nutrition expertise largely dispersed to individual program offices. According to national and regional officials, in recent years, coordinating nutrition education has not been a priority for USDA, and there has been a loss of staff resources dedicated to nutrition education in the department overall. National and regional officials said it is hard to find time to coordinate across nutrition education efforts because they face competing priorities and increased workloads, at times because staff with nutrition education expertise have left employment with USDA and not been replaced. According to regional officials, a voluntary group of FNS national and regional officials who meet to discuss nutrition issues has experienced diminishing participation in recent years, in part due to these reasons. Regional officials and land-grant university officials said that more formal coordination mechanisms to provide leadership and promote cross- department coordination and information sharing on nutrition education could help increase efficiency, maximize the use of federal resources, and avoid potential duplication of effort. One regional official said she regularly reaches out to a colleague to obtain information on other FNS nutrition education programs, but a centralized tool could provide this information quicker and more efficiently. Another regional official said she compiled information on USDA nutrition education grant opportunities for states in her region, but it would be helpful if this information were centrally compiled by the national office. Regional officials and land- grant university officials we spoke with also said formal collaboration mechanisms, such as a document or tool with information on all of USDA’s nutrition education efforts, examples of best practices for coordination, or an annual meeting to encourage information sharing, would be useful. Federal internal control standards state that agencies should communicate quality information across reporting lines to enable personnel to perform key roles in achieving objectives, and management should set the tone at the top and throughout the agency to ensure priorities are understood by all stakeholders. In our prior work, we reported that effective coordination can help reduce overlap and duplication, and we found that sustained leadership is an essential element to developing collaborative working relationships. We also identified leading practices that federal agencies can use to enhance the effectiveness of their collaborative efforts, such as agreeing on roles and responsibilities and establishing policies and procedures to work across organizational boundaries. USDA has acknowledged the importance of nutrition education coordination for maximizing the reach and potential impact of federal nutrition education and nutrition assistance programs in some of its program regulations and guidance, and this emphasis is consistent with new federal requirements. For example, FNS’s SNAP-Ed plan guidance directs states to coordinate SNAP-Ed activities with other national, state, and local nutrition education, obesity prevention, and health promotion initiatives and interventions, such as WIC and EFNEP. In our 2004 review of USDA’s nutrition education efforts, we found that increased coordination, such as sharing curricula, lessons learned, and data collection tools across efforts, could help USDA’s nutrition education programs make more efficient and effective use of resources. Consistent with this focus, the Agriculture Improvement Act of 2018 (Farm Bill) requires USDA to submit an annual report to Congress that includes an evaluation of the level of coordination between SNAP-Ed, EFNEP, and other USDA nutrition education programs. Some USDA nutrition experts are in agencies disconnected from the nutrition education programs (see fig. 4), yet these agencies play a significant role in developing and compiling dietary guidance, research, and other information related to nutrition education (see table 2). Despite their role in developing and compiling research and information related to nutrition education, consultation with these experts by the program offices is limited, according to USDA officials, possibly because they are located in separate agencies. For example, Although CNPP leads a cross-cutting committee that reviews nutrition education materials developed by USDA program staff to ensure materials are consistent with the Dietary Guidelines for Americans,CNPP officials noted they have been infrequently consulted by program officials while materials are under development or activities are being implemented. This may have been in part related to organizational structure, as until recently CNPP and FNS were separate agencies that individually reported to the Office of the Under Secretary for Food, Nutrition, and Consumer Services, according to USDA officials. Some nutrition education program staff also told us they currently use the core nutrition messages on USDA’s website when developing nutrition education materials—messages that CNPP officials noted were developed in 2010 and have not been updated to reflect the latest edition of the Dietary Guidelines for Americans. This approach may lead to inefficiencies in the development of nutrition education materials. Although ERS conducts nutrition research, nutrition education program officials were not always aware of or using ERS resources, possibly because most of the programs reside in a different USDA mission area. A prior working group attempted to bridge the organizational divide between ERS and some of USDA’s other agencies and offices that work on nutrition education by assisting efforts to share information, but the group has since dissolved. Currently, some national and regional officials we spoke with who work on nutrition education programs had limited awareness of ERS’s nutrition education research. For example, some program officials in the national office were unsure whether ERS did work related to nutrition education and learned of ERS research through automated email updates. Further, one regional official learned of ERS data on food insecurity, which can help states meet federal requirements for targeting nutrition education services to local areas based on their level of need, through a meeting with an outside agency. Nutrition education program officials were also generally unaware of ARS’s efforts related to nutrition education. Specifically, USDA nutrition education program officials we spoke to said they had little direct contact with ARS officials and were generally unaware of ARS efforts related to nutrition education. Further, regional officials who work on SNAP-Ed had not used or distributed ARS resources to state officials and also seemed generally unaware of ARS’s nutrition education efforts. USDA lacks a mechanism for systematically integrating its internal nutrition expertise into its nutrition education programs, which may inhibit the effectiveness of the department’s efforts. Federal internal control standards state that agencies should use quality information from reliable internal sources, among others, to inform decision-making. Further, in our prior work, we found that identifying and addressing needs by leveraging resources is a leading practice for collaboration. Nutrition education program officials are missing opportunities to benefit from relevant expertise within USDA but outside their program offices. Failing to leverage its own internal expertise hinders USDA’s development of nutrition education materials that are informed by the latest nutrition guidance and research. Poor nutrition contributes to costly chronic diseases that are among the leading causes of death for Americans, and USDA’s nutrition education programs and related efforts strive to educate Americans on nutrition and improve their dietary choices. Because USDA’s nutrition education programs are primarily targeted to low-income adults and children, who may receive federally-funded nutrition assistance benefits, these programs also have the potential to improve the likelihood that recipients will spend those benefits to obtain foods that have a positive impact on their health. However, in order to reach these goals, USDA needs to ensure that its programs are effectively educating participants to maximize the impact of the federal investment in nutrition education. Although USDA has some information on the effectiveness of its nutrition programs, without improvements to how USDA gathers information on the effectiveness of SNAP-Ed interventions nationwide, USDA will be unable to ensure one of its largest investments in nutrition education is meeting its goals. The 2018 Farm Bill included a requirement for USDA to begin reporting annually on the level of coordination between its nutrition education programs, and USDA has acknowledged the importance of coordination and information sharing to maximize nutrition education programs’ impacts. However, the department currently lacks a formal mechanism to ensure this occurs. As a result, USDA risks missing opportunities to increase efficiency, maximize the use of federal resources, and avoid potential duplication of effort. In addition, without coordination between nutrition education program officials and others with nutrition expertise in the department, programs will develop nutrition education materials that fail to fully leverage the latest nutrition guidance and research, possibly missing opportunities to effectively influence the dietary choices of their target populations in the process. We are making the following three recommendations to USDA: 1. The Administrator of FNS should improve how FNS gathers information on the effectiveness of SNAP-Ed interventions, in order to ensure that these interventions are meeting program goals. (Recommendation 1) 2. The Secretary of Agriculture should direct the Under Secretaries for Food, Nutrition, and Consumer Services and for Research, Education, and Economics to develop a formal mechanism, such as a designated individual or group of individuals, for providing cross-department leadership for USDA’s nutrition education efforts and facilitating cross- program information sharing. (Recommendation 2) 3. The Secretary of Agriculture should direct the Under Secretaries for Food, Nutrition, and Consumer Services and for Research, Education, and Economics to identify and implement mechanisms to fully leverage the department’s nutrition expertise for its nutrition education efforts. (Recommendation 3) We provided a draft of this report to USDA for review and comment. In its comments, reproduced in appendix II, USDA generally agreed with our recommendations. USDA also noted that FNS has efforts underway to comply with the 2018 Farm Bill requirement that the department report annually on the level of coordination between its nutrition education programs. USDA also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Agriculture, congressional committees, and other interested parties. In addition, this report will be available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Our report examines the extent to which the U.S. Department of Agriculture (USDA) (1) has information on participation, expenditures, and effectiveness for its nutrition education programs; and (2) coordinates its nutrition education efforts and leverages internal nutrition expertise for these efforts. The scope of our review includes five federal programs that provide nutrition education: Supplemental Nutrition Assistance Program Education (SNAP-Ed), the Expanded Food and Nutrition Education Program (EFNEP), the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), Team Nutrition, and the Food Insecurity Nutrition Incentive (FINI) Grant Program. Among USDA programs that provide nutrition education, four of these received the greatest amount of federal funding for nutrition education in fiscal year 2018—WIC, SNAP-Ed, EFNEP, and Team Nutrition. In addition, we included FINI because it is a grant program in which nutrition education can be a component, and the program’s goal is to incentivize healthy eating. We also reviewed USDA efforts that provide nutrition education through nutrition-related research and guidance directed at the general public. In addition to the methods discussed below, to address both of our research objectives, we reviewed relevant federal laws, regulations, and guidance, as well as our prior work on USDA nutrition education efforts and leading practices for collaboration. We interviewed officials from relevant USDA agencies, including the Food and Nutrition Service (FNS) and the National Institute of Food and Agriculture (NIFA), which oversee the nutrition education programs described in this report. We also interviewed officials from other USDA agencies overseeing nutrition- related research and guidance, including the Agricultural Research Service, the Center for Nutrition Policy and Promotion, and the Economic Research Service. Additionally, we interviewed officials from the seven FNS regional offices, including officials who work on SNAP-Ed and the Child and Adult Care Food Program. We also interviewed representatives of selected organizations knowledgeable about USDA’s nutrition education efforts. We assessed USDA’s efforts to collect information on its nutrition education programs, coordinate its nutrition education efforts, and leverage internal nutrition expertise against GAO’s standards for internal controls in the federal government. To address the first objective, we analyzed USDA data on nutrition education participation. Two of the nutrition education programs, SNAP- Ed and EFNEP, collect data on direct education participation. We analyzed SNAP-Ed total direct education participation data for fiscal years 2010 through 2018 collected through SNAP-Ed’s data reporting system, the Education and Administrative Reporting System (EARS). We analyzed EFNEP total direct education participation data for fiscal years 2010 through 2018. These data are reported through the Web-based Nutrition Education, Evaluation and Reporting System (WebNEERS), an integrated data collection system, sponsored by NIFA, and used at the county, state, and federal levels. To assess the reliability of the SNAP-Ed and EFNEP participation data, we interviewed FNS and NIFA officials and reviewed relevant documentation. We determined that these data were sufficiently reliable for the purpose of reporting the number of direct education participants in SNAP-Ed and EFNEP. We also reviewed available USDA data on the number of people reached by nutrition education efforts in SNAP-Ed and EFNEP other than through direct participation. SNAP-Ed collects information on the number of people reached by nutrition education efforts that are not direct education, such as policy, systems, and environmental change interventions and social marketing. However, states face challenges with tracking individuals reached by these education interventions, and these data are likely to include duplicate records of individuals, according to USDA officials. Therefore, we concluded that these data were not sufficiently reliable for the purpose of reporting the number of people indirectly reached by SNAP-Ed. EFNEP also collects information on indirect education reach. This information tracks other family members of adults who participated in direct education who therefore may also benefit from the information shared, according to USDA officials. To assess the reliability of these data, we interviewed NIFA officials and reviewed relevant documentation. We determined that these data were sufficiently reliable for the purpose of describing the number of people indirectly reached by EFNEP. Because USDA officials consider the total number of WIC participants to be the best proxy for WIC nutrition education participation, as all WIC participants are offered nutrition education, we analyzed WIC total participation, and participation by women, infants, and children, for fiscal years 2010 through 2018. These data are reported on the FNS- 798/798A Financial Management and Participation Report form, which contains programmatic and financial data reported by state agencies, Indian Tribal Organizations, and U.S. territories through the Food Programs Reporting System (FPRS). To assess the reliability of these data, we interviewed FNS officials and reviewed relevant documentation. We determined that these data were sufficiently reliable for reporting the number of WIC participants offered WIC nutrition education. Data were unavailable on participation for Team Nutrition—a program which provides training and technical assistance to child nutrition program operators, and creates and disseminates materials for child nutrition program participants. As a proxy measure for program reach, we analyzed data on nutrition education materials disseminated to participants and the online views and downloads of nutrition education materials. To assess the reliability of these data, we interviewed FNS officials and reviewed relevant documentation. We determined that these data were sufficiently reliable for the purpose of reporting the number of Team Nutrition materials disseminated. To address the first objective, we also analyzed WIC, SNAP-Ed, EFNEP, Team Nutrition, and FINI total nutrition education expenditure data for fiscal year 2017, the most complete data available as of April 2019. Like WIC participation data, WIC expenditure data are reported on the FNS- 798/798A Financial Management and Participation Report form through FPRS. Federal SNAP-Ed and Team Nutrition expenditure data are reported on the SF-425 form, which state agencies submit quarterly, also through FPRS. USDA tracks nationwide expenditures for EFNEP and FINI through NIFA’s payment system, Automated Standard Application for Payments; grants management system, Cooperative Research, Education, and Extension Management; and financial management system, Financial Management Modernization Initiative. To assess the reliability of these data, we interviewed officials from FNS and NIFA and reviewed relevant documentation. We determined that these data were sufficiently reliable for the purpose of reporting nationwide expenditures for these five programs. To determine what information USDA has on the effectiveness of its nutrition education programs, we reviewed relevant program evaluations from USDA issued within the last 10 years. We selected these evaluations based on information we obtained from USDA and other knowledgeable officials through interviews and relevant documents. To provide additional context on program operations for SNAP-Ed and EFNEP, we reviewed various program data. We analyzed the EARS data on the total number of SNAP-Ed implementing agencies, including the number of land-grant universities that were implementing agencies, and the types of education provided by SNAP-Ed programs in fiscal year 2018. For EFNEP, we reviewed data on participant outcomes, reported through WebNEERS, for fiscal year 2018. To assess the reliability of these data, we interviewed officials from FNS and NIFA and reviewed relevant documentation. We determined that these data were sufficiently reliable for the purpose of our reporting objectives. To gain the perspective of officials involved in the implementation of nutrition education efforts, we interviewed representatives of four land- grant universities. Land-grant universities are the sole provider of EFNEP and one of the main providers of SNAP-Ed. We judgmentally selected a non-generalizable sample of four land-grant universities based on various criteria, including the recommendations of knowledgeable officials, geographic dispersion, and other factors, such as the percentage of the university’s state population in poverty. Two of the universities we selected solely administer EFNEP and two administer both EFNEP and SNAP-Ed. We gathered information from these land-grant university representatives on how they provide nutrition education through their programs and the extent to which they coordinate with other SNAP-Ed programs in their county and state, as well as with other USDA nutrition education programs. We also gathered information on support they receive from the USDA national office for coordination, if any; their perspectives on challenges USDA faces to coordinating nutrition education across its programs, if any; and their views on opportunities for USDA to improve coordination across nutrition education programs. Information collected from the land-grant university representatives cannot be generalized to all land-grant universities nationwide. We conducted this performance audit from December 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Rachel Frisk (Assistant Director), Kristen Jones (Analyst-in-Charge), and Sara Rizik made key contributions to this report. Also contributing to this report were Monika Gomez, Stacy Ouellette, Almeta Spencer, Rachel Stoiko, Curtia Taylor, Walter Vance, Sarah Veale, and Adam Wendel.", "summary": "The Centers for Disease Control and Prevention reports that many Americans' diets lack adequate sources of good nutrition and that this contributes to costly chronic health conditions. USDA funds and administers a variety of nutrition education efforts, which aim to help educate Americans on nutrition and improve their dietary choices. GAO was asked to review these efforts. This report examines the extent to which USDA (1) has information on participation, expenditures, and effectiveness for its nutrition education programs; and (2) coordinates its nutrition education efforts and leverages internal nutrition expertise for these efforts. GAO reviewed relevant federal laws, regulations, guidance, and GAO's prior work on nutrition education and leading practices for collaboration; analyzed USDA data on nutrition education participation in fiscal year 2018 and expenditures in fiscal year 2017, the most recent year with complete data available; and reviewed program evaluations and available outcome data for fiscal year 2018. GAO also interviewed USDA officials and representatives of relevant organizations. The U.S. Department of Agriculture (USDA) administers five key programs that provide nutrition education and has information on participation, expenditures, and effectiveness for most of these programs. USDA tracks the number of participants in direct education, such as classes and counseling, as well as other measures of program reach. For example, Supplemental Nutrition Assistance Program Education (SNAP-Ed), one of USDA's largest nutrition education programs, served 3.8 million participants through direct education in fiscal year 2018. USDA also collects nationwide expenditure data for all of its nutrition education programs, which totaled nearly $907 million in fiscal year 2017—the most recent year with complete data available. In addition, USDA collects some information on the effectiveness of most of its nutrition education programs; yet information USDA collects from states on SNAP-Ed effectiveness cannot be easily aggregated or reviewed. States provide this information in narrative reports, which hinders USDA's ability to assess the effectiveness of interventions used across the country and determine whether SNAP-Ed is achieving its goals. USDA does not have a formal coordination mechanism for its nutrition education efforts and does not fully leverage the department's nutrition expertise. According to USDA officials, coordinating nutrition education efforts has not been a priority in recent years, and the department does not have a dedicated individual or entity with leadership responsibility for nutrition education. This has resulted in limited coordination across USDA's nutrition education programs, including programs with similar target populations. GAO previously reported that effective coordination can help reduce overlap and duplication. In its absence, USDA's nutrition education programs are missing opportunities to share information and avoid duplicating efforts. Further, some USDA nutrition experts are not located in agencies or offices overseeing the nutrition education programs, and possibly because of this, program staff consult these experts on a limited basis, if at all. Failing to leverage its internal expertise hinders USDA's development of nutrition education materials that are informed by the latest nutrition guidance and research and may reduce the effectiveness of these efforts. GAO is making three recommendations to USDA, including that USDA improve how it gathers information on SNAP-Ed effectiveness, develop a formal mechanism for coordinating nutrition education across the department, and take steps to fully leverage the department's nutrition expertise for its nutrition education efforts. USDA generally agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "USPS is required to provide postal services “to bind the nation together through the personal, educational, literary, and business correspondence of the people prompt, reliable, and efficient services to patrons in all areas and postal services to all communities.” To help fulfill this mission, USPS has developed a network of facilities that provides access to retail services and supports postal collection, processing, transportation, and “last mile” delivery of mail—functions that are generally co-located. The retail portion of these facilities includes public areas where USPS provides customers with retail services. The public areas can include full service counters where employees assist customers, self-service areas, and post office boxes. The non-public portions include features such as workrooms, where mail processing occurs, and employee support areas, such as lunch tables and lockers. Although USPS data show that customer visits and transactions have declined over the past 5 years, the size of USPS’s retail network has remained largely unchanged during that period (see fig. 1). We have previously reported that USPS has made some efforts to reduce the number of its retail facilities to align with the significant decline in mail volume and address rising costs. Several factors, however, have limited USPS’s ability to make further reductions. For example, in 2011 USPS moved to evaluate the potential closure of almost 3,700 retail facilities but abandoned their effort due to stakeholder concerns. Instead, USPS explored options to adjust its retail service without closing offices. Legal restrictions also limit USPS’s ability to close retail facilities. For example, USPS cannot close a small post office solely for operating at a deficit. If USPS wishes to close a retail facility, among other steps, it must take into consideration not only the economic savings but also the effects on the communities served, its employees, and the services provided, and it must provide customers with at least 60 days’ notice before the proposed closure date. Federal statute defines the types of services that USPS may and may not provide. As previously noted, PAEA placed limitations on the nonpostal products and services USPS could provide. In particular, it allowed USPS to continue to provide nonpostal products and services that were both offered as of January 1, 2006, and were permitted by PRC to continue. While PAEA generally prohibits USPS from initiating new nonpostal services, USPS uses a separate statutory authority to provide such services to federal executive agencies. If a nonpostal service is to be provided to a federal executive agency, generally, USPS and the parties must specify the terms and conditions of their collaboration, including the activities to be performed by USPS and the terms of reimbursement, if applicable. USPS is currently not authorized to provide nonpostal services to state or local entities. USPS may also lease excess space at retail facilities—including parking, office space, and roof areas—to other entities at both the facilities that USPS owns and those it leases from other entities. USPS officials said some leases give them the right to sublease space at leased facilities, but the officials did not know how many facilities where USPS had such rights. USPS collaborates with the GSA, which is the nation’s largest public real-estate organization, to lease space to other federal government entities. USPS works with a real estate firm to lease space to private or other entities, such as to state and local government entities. According to USPS’s analysis, retail facilities accounted for a relatively small portion—about $5.17 billion, or 7.1 percent—of USPS’s modified operating costs in fiscal year 2018. (See fig. 2.) Personnel costs accounted for the majority of retail facility operating costs ($4.87 billion), including nearly $4.4 billion in employee compensation. Non-personnel costs amounted to about $0.30 billion, which included rent ($0.09 billion), utilities ($0.06 billion), and depreciation and amortization ($0.15 billion). In terms of revenue, according to USPS’s analysis, retail facilities accounted for a relatively small portion—about 15 percent ($10.5 billion)—of USPS’s total revenue ($70.6 billion) in fiscal year 2018. As shown in figure 3, postage meters and validation and walk-in stamp sales generated most of that revenue. In fiscal year 2018, the majority of retail facilities were profitable (see fig. 4), with some considerations. USPS’s analysis showed that the total retail revenue generated at retail facilities was more than double retail facility costs in fiscal year 2018. However, over a third of retail facilities did not generate enough revenue to cover USPS’s retail costs. In particular, over half of the rural facilities were unprofitable, while the overwhelming majority of suburban and urban facilities were profitable. Overall, we found that of the unprofitable facilities, 89 percent were located in rural areas. Among the 10 most unprofitable facilities, though, 5 were in urban areas and 3 were in suburban areas. USPS officials told us some of these unprofitable urban and suburban retail facilities were in areas where rent, utilities, and maintenance were very costly, and the revenue generated was not enough at those facilities to make them profitable. Finally, while the number of customer visits to retail facilities has declined by 25 percent since fiscal year 2009, customers are still using them. In fiscal year 2018, among the retail facilities for which USPS had data, the average number of customer visits was 46,624 annually, or an average of 154 visits per day. The most unprofitable facilities averaged 31,731 customer visits in fiscal year 2018, or about 105 customer visits per day. According to studies we reviewed and USPS officials and two consumer groups we interviewed, USPS’s retail facility network produces economic, social, environmental, and civic benefits (see table 1). Some postmasters who responded to our survey told us that their retail facilities generated economic and social benefits. For example, 90 percent (134 of 149) of postmasters managed a retail facility within walking distance of other businesses or community buildings. Of those, about 31 percent (41 of 134) of postmasters indicated that the retail facility increased patronage of nearby businesses and community buildings to a great or very great extent. Almost half (71 of 149) of the postmasters stated that their retail facility served as a place for residents to interact in person to a great or very great extent. The economic and social benefits may benefit rural communities more than urban and suburban areas, according to our survey of postmasters, as well as the reports we reviewed and stakeholder interviews. One survey respondent stated: “Offices in rural communities are extremely important to the area in which they serve. Postmasters are often town leaders and hold various positions on councils, boards, and non-profit organizations. The USPS is usually the only government office in the community, and it is recognized by many as their only connection to the outside world.” According to USPS OIG and representatives from the two consumer groups we interviewed, retail facilities serve as a gathering place and help build social identity and connectivity, especially in rural areas. According to USPS, small business owners depend on access to USPS retail facilities across the country and in rural areas since USPS facilities are often the only retail shipping provider. In addition, according to USPS, retail facilities play important roles in connecting rural communities. USPS has offered a variety of nonpostal products and services since 2008 (see table 2). As previously described, PAEA permitted USPS to continue offering certain nonpostal products and services that were approved by PRC, and did not alter USPS’s statutory authority to provide nonpostal services to federal executive entities. The nonpostal products and services we identified were either permitted to continue by PRC pursuant to PAEA or are services USPS provided through partnerships with other federal government entities, such as those made through interagency agreements. USPS chooses where to offer its nonpostal products and services based on several factors, and as noted in table 2 above, USPS does not offer each nonpostal product and service at all retail facilities. USPS officials told us they determine where to offer nonpostal products and services based on several factors, such as customer demand and analysis of the potential to generate revenue. In some cases, USPS consults with other entities to make these determinations. For example, USPS officials told us that the company that supplies the greeting cards conducts market research to determine locations where there is demand. For passport services, USPS and U.S. Department of State officials determine locations based on demand and whether retail facilities have adequate staff to perform related functions, among other factors. USPS collects a fee for some, but not all, of the identified nonpostal products and services offered at retail facilities. According to data from USPS’s revenue and cost analysis, nonpostal products and services generated a small amount of the total revenue collected at retail facilities—about $431 million in fiscal year 2018, which accounted for 4.1 percent of total retail facility revenue, and 0.6 percent of USPS’s total revenue. As figure 5 shows, passport applications and photo services, as well as money orders, accounted for the greatest percentage of revenue. Nonpostal products and services may also generate revenue from additional transactions that are made by customers during their visits to obtain these offerings. For example, in 2016, USPS OIG estimated that USPS generated almost $6.6 million in fiscal year 2015 from individuals purchasing money orders to pay for passport-related services. In addition, USPS OIG reported that passport services increased foot traffic at retail facilities. A representative from a postal employee union also told us that nonpostal products and services offered at retail facilities can drive increased foot traffic in post office lobbies. USPS officials said that USPS incurs various costs for nonpostal products and services related to (1) the time it takes mail clerks to perform transactions; (2) equipment and materials, such as passport photo equipment for passport photo services; and (3) any needed physical changes to the facility. USPS is required to analyze whether revenues cover costs for some of the identified nonpostal products and services, but not all. USPS officials said they are not required to report on whether revenues cover costs for services provided through federal interagency agreements, such as passport application processing. In addition, they told us they do not track the costs for offerings that are considered to be “non-commercial, non-revenue generating services,” such as community bulletin boards. USPS officials also said some of the nonpostal products and services we identified do not incur any costs; for example, greeting and gift card displays are provided at the vendor’s expense. While USPS does not track costs for all offerings and make all costs publicly available, USPS reported that most of the nonpostal products and services we identified (as shown in table 2), and for those that USPS tracks costs, generated more revenue than costs in fiscal year 2018. Specifically, USPS reported that money orders earned almost $12 million and philatelic sales earned about $1.1 million in revenue above their costs in fiscal year 2018. USPS reported the only nonpostal product or service that did not cover its costs in 2018 was in-bound international money transfers. Based on our discussions with USPS officials, we identified three pilots USPS has conducted since 2008 to provide nonpostal services on behalf of other federal government entities at retail facilities; two of these pilots involved USPS mail clerks performing in-person identity proofing and biometric capture and one of the pilots involved sharing retail facilities’ lobby space. USPS and the U.S. Census Bureau (Census Bureau) conducted a pilot to evaluate the feasibility of having USPS assist the Census Bureau with nationwide hiring for the 2020 Census. For this pilot, which lasted from March 2015 to July 2015, USPS mail clerks at 12 retail facilities used Census Bureau equipment to conduct in-person identify proofing and other administrative processes to help hire temporary Census Bureau employees. USPS and the Federal Bureau of Investigation (FBI) began a pilot in September 2018 in which USPS mail clerks began scanning and sending the fingerprints of individuals participating in the FBI’s Identity History Summary Checks—a program that enables individuals to request their arrest and conviction records (see fig. 6). USPS and the Census Bureau conducted a pilot from April 2018 to July 2018 as part of the Census Bureau’s 2018 testing activities to determine whether interactive kiosks could be used at retail facilities to allow customers to fill out their Census questionnaire. USPS installed kiosks at 30 retail facilities in Providence County, Rhode Island, that offered Internet access limited to the Census Bureau’s online questionnaire. According to officials from the Census Bureau and FBI, the agencies benefited from the three pilots, to varying degrees, and USPS generated revenue from the two in-person proofing and biometric capture pilots. USPS officials told us that they received $125,000 from the 2015 in- person proofing pilot with the Census Bureau, and Census Bureau officials said this pilot provided their staff with convenient locations to meet with prospective applicants. Regarding the 2018 kiosk pilot, Census Bureau officials said this resulted in 111 completed questionnaires. However, after the pilots ended, the agencies did not enter into subsequent partnerships. Census Bureau officials told us they did not wish to extend the in-person proofing pilot due to limited funding, and they did not wish to extend the Census kiosk pilot because the number of completed tests did not justify the cost and effort. For the FBI pilot, USPS had generated almost $425,000 in revenue, as of December 2019, from the fees paid by participating customers since September 2018, according to USPS documentation. FBI officials told us this pilot has improved their customer experience and enabled them to reduce their response time for providing information to customers. In March 2019, USPS and the FBI expanded the pilot from two to 28 retail facilities, and this pilot was still ongoing at the time we published our report. USPS officials told us they also created the Digital Business Services Team in June 2019 in part to pursue additional revenue-generating partnerships with federal executive agencies. The officials said a major focus of the team was to expand USPS’s in-person identity proofing and fingerprinting services, and they estimated—with certain assumptions regarding acquiring partnerships—fingerprinting services could generate about $87 million in annual revenue after a 5-year rollout. USPS is currently discussing potential new partnerships for in-person proofing pilots with federal government entities. USPS officials, federal government entities, and stakeholders we interviewed and postmasters we surveyed told us that the identified nonpostal products and services (as shown in table 2) currently offered at retail facilities provided the following non-revenue benefits. Enhanced consumer benefits. Access to certain nonpostal products and services at retail facilities enhanced consumers’ convenience, according to USPS officials, and representatives from consumer groups and a postal employee union. For example, in September 2019, USPS analysis found that 32 percent of the FBI pilot customers selected USPS’s fingerprinting services over others who offer similar services due to USPS’s location and rated their satisfaction with USPS’s service highly. A representative from the postal employee union said that many customers are happy that they can purchase a greeting card and gift card when visiting a retail facility. In addition, a representative from one of the consumer groups we interviewed said that some nonpostal products and services, such as international money transfers and money orders, may be otherwise unavailable to certain populations, such as those who do not have access to a bank. This representative also told us that some low-income consumers only have internet access through their phone, which makes it difficult to fill out forms online; these consumers therefore could benefit from having certain forms, such as voter registration and selective service, available at a retail facility. Enhanced government benefits. Officials from five of the six federal government entities that had partnerships with USPS said their partnerships supported their ability to fulfill their missions, such as by efficiently using resources and increasing customer convenience. For example, officials from all six of these federal government entities said USPS’s extensive network of retail facilities helped them reach customers or users. Also, officials from three of these federal entities told us that USPS’s services cut the processing time for certain applications or services. Enhanced community benefits. Representatives from the two consumer groups told us that community services offered at retail facilities—such as food drives, school tours, and community bulletin boards—may help sustain communities and increase social connectedness. Postmasters we surveyed also reported ways their communities benefited from nonpostal services provided at their retail facilities. For example, one postmaster we surveyed reported that his or her retail facility collected eyeglasses for a local community organization. Another postmaster we surveyed reported that during the holiday season, his or her retail facility offers decorative rubber stamps, which have become a community tradition. According to the postmasters we surveyed, some of the nonpostal products provided significant nonrevenue value, although the degree to which these provided value depended on whether the retail facility was located in a rural, suburban, or urban area. We asked postmasters to identify whether certain nonpostal products and services were offered at the selected facility, and if so, how much value the product or service provided to the community. Overall, passport services were the most highly valued nonpostal product or service. About 95 percent (36 of 38) of postmasters at retail facilities that offered passport services said passports provided great or very great value to their communities. Money orders were the next most highly valued nonpostal product or service. These were offered at more of the retail facilities selected for our survey than passport services. For the retail facilities that offered money orders, about 78 percent (115 of 147) of postmasters said this product provided great or very great value to their communities. Burial flags were the third highly valued nonpostal product or service, for some types of locations. About 66 percent (21 of 32) of postmasters managing rural retail facilities and about 70 percent (35 of 50) of postmasters managing suburban retail facilities said burial flags provided great or very great value compared to about 43 percent (9 of 21) of postmasters overseeing urban retail facilities. Among retail facilities that offered international money transfers (SureMoney), selective service forms, philatelic products, and gift cards, around one-third or more of postmasters reported these as providing some value or little to no value in their communities. Representatives of one postal employee union, postmasters we surveyed, and officials from the six federal government entities that had partnered with USPS reported minimal challenges related to providing the identified nonpostal products and services at retail facilities. For example, the representative from the postal employee union told us that the only challenge for postal workers was when locations did not have adequate staff to handle passport services. Very few of the postmasters selected for our survey identified challenges related to offering the thirteen nonpostal products and services we asked about. Officials from only two of the six federal government entities mentioned challenges, and none of them were significant in nature. For example, officials from the Department of Veterans Affairs mentioned that the only challenge was ensuring adequate supplies of burial flags at retail facilities. Although the stakeholders we interviewed and postmasters we surveyed cited few challenges associated with nonpostal products and services, USPS OIG has reported that USPS could take actions to further increase the use of some of these offerings. In 2015, USPS OIG suggested that USPS conduct better-targeted marketing for its money orders or consider pricing changes to the fees charged for money transfers. In 2016, USPS OIG identified several areas in which USPS could improve customer experience for passport services, such as improving the clarity of information provided to customers and improving the accuracy of offerings on USPS’s website. USPS generally agreed with the findings but reported it had already implemented or had begun to implement changes to improve customer service issues raised in report. USPS currently leases some of its owned excess space—including space at its retail facilities, such as parking, office space, and roof areas—to other entities, generating additional revenue and other benefits. According to USPS as of January 2020, USPS was leasing space in 232 facilities (about 3 percent) of its 8,362 owned facilities to federal and local government and private entities. USPS generated about $29 million from its leases in fiscal year 2018. USPS officials told us they are currently researching the feasibility and benefits of leasing space to entities to place automated teller machines in retail facility lobbies and parking lots as a way to generate revenue. Stakeholders we interviewed and USPS OIG have said leasing space may also result in non-revenue benefits for USPS and consumers. For example, a postmaster who managed a retail facility said that leasing office space to two local government entities likely increased foot traffic in the facility and increased community access to government services provided by the tenants. USPS, however, has little additional vacant rentable space. As of September 2018, USPS reported that it had vacant rentable space available in 307 (about 4 percent) of its 8,362 owned facilities. In 2018, USPS OIG reported that USPS faced unique challenges in leasing such excess space, including poor condition and limited size, lack of handicap accessibility, limited parking, lack of accessibility without interfering with USPS operations, and lack of a separate restroom. According to GSA officials, some available space is small—USPS reported that about 9 percent of available space is less than 500 square feet—and may require significant investment from GSA or the potential tenant agency to be suitable for occupancy (see fig. 7). GSA officials said it has been difficult to find federal government entities willing to lease retail facility space from USPS. For example, officials from the Census Bureau told us that they leased some space at two USPS facilities to support 2010 Census activities, such as to support hiring personnel and a location for training. However, they were not able to lease as much as they would have liked because there was very little available space that met Census requirements, and the space that was available would have required costly modifications prior to use. In addition, USPS officials told us there can be costs related to leasing space, such as USPS’s making needed renovations. USPS does not track the extent to which space at retail facilities, such as lobbies and parking lots, is shared with other entities without any payment. However, USPS officials told us that while community groups have asked to use retail facility space and parking lots, these requests do not happen frequently. Only two of the 149 postmasters we surveyed (about 1 percent) indicated that the selected retail facility they oversaw shared space with other entities at no charge. Studies we reviewed and postal experts and stakeholders we interviewed have suggested that USPS may be well positioned to offer additional nonpostal products and services due to its trusted brand, vast retail facility network, and experience with other nonpostal efforts. Examples of such additional offerings are set out in the following table (see table 3). USPS officials, stakeholders, and studies we reviewed indicated these identified additional nonpostal activities—if USPS were authorized to offer them—could offer a variety of non-revenue benefits to consumers, government entities, and communities. Examples of some of the suggested types of non-revenue benefits are shown in table 4. However, most of these additional nonpostal products and services would not increase revenues or greatly benefit their communities, according to the postmasters we surveyed. Specifically, a majority of postmasters did not think any of the nine additional nonpostal products and services we asked about would increase revenues or benefit the community to a great or very great extent. However, postmasters indicated that some potential services were more promising than others. In particular, about 40 percent (59 of 149) of postmasters indicated that notary services would increase revenues or benefit the community to a great or very great extent, while 36 percent (53 of 149) of postmasters indicated that driver’s license and other state license services would increase revenues or benefit the community to a great or very great extent. Few postmasters in our survey indicated that a benefit of offering any of the nine additional nonpostal products and services would be that they would be providing a product or service that is not offered elsewhere in the community. USPS officials, officials from other federal government entities, and stakeholders told us that most of the nonpostal products and services identified above would likely have limited revenue potential. They, as well as studies we reviewed, indicated a variety of reasons why USPS might not generate significant net revenue from the additional nonpostal products and services we identified. Low potential for a significant market share. USPS officials and stakeholders told us USPS could face challenges gaining enough of a market share for some of the additional nonpostal products and services to make a profit. For example, USPS officials said if they offered notary services, they would likely gain only a small share of the market because other retailers, such as banks, already offer these services for free. Also, representatives from four financial associations said they believed consumer demand for financial products and services was already being met or would best be met by existing financial entities, and that many consumers may not likely obtain these services from USPS. In addition, representatives from two financial associations said serving underbanked populations would likely result in limited profits because these tend to be riskier customers who may default more often, and the services they use result in slim profits for current providers. High operational costs. USPS OIG has reported that USPS incurs low customer foot traffic and high labor costs compared to other retail facilities. According to USPS officials, these factors make it difficult to compete on a cost-per-transaction basis for nonpostal products and services and make leasing space in owned facilities and subleasing space at USPS’s leased retail facilities attractive, because leasing and subleasing would not incur personnel costs. Additionally, USPS officials told us that offering some of the additional nonpostal products and services could require significant investment costs—such as major technology investments and additional training for mail clerks—further reducing USPS’s ability to make a profit. For example, USPS OIG estimated in 2015 that USPS could generate $1.1 billion annually after a 5-year ramp up from expanding the financial products it already offered. USPS officials, however, said that expanding such offerings at retail facilities would likely require extensive investments in physical and information technology security and incur ongoing costs. Accounting for these sorts of costs would mean USPS would likely generate about $100 million to $200 million in net revenue as opposed to $1.1 billion. In addition, representatives from the American Association of Motor Vehicle Administrators and the Maryland Motor Vehicle Association told us that if USPS were to offer state driver’s license services, it would need to invest in equipment and training. Specifically, USPS would need to purchase secure computer systems that require multiple electronic interfaces and train mail clerks to handle complex document verification for issuing state driver’s licenses in order to meet requirements set by the REAL ID Act of 2005, among other concerns. Limitations on amounts charged. USPS officials and other stakeholders indicated that the fees charged for some of the additional nonpostal products and services would be too low to result in high revenues. For example, USPS officials found there would be little potential revenue from providing photocopying services or placing vending machines in retail facilities. Also, NPS officials and representatives from AFWA told us that current providers of National Parks and Federal Recreational Lands Passes and state hunting and fishing licenses generally do not generate much revenue due to the fee structure. In addition, USPS may or may not charge other entities a fee for sharing information at retail facilities. For example, USPS’s partnerships to share information on other government entities’ behalf, such as providing selective service registration forms and displaying information for DOJ’s National Crime Victims’ Rights week, do not generate revenue. Last, any effort to expand community services would not be intended to generate revenue. We have previously reported that foreign posts began offering nonpostal products and services to increase revenues, such as offering banking or financial services and making additional government services available in their retail facilities, but these efforts have had mixed results. Some foreign postal operators have expanded their financial offerings at retail facilities and have generated significant revenue from these efforts. For example: The United Kingdom’s postal retail operator—The Post Office—has an agreement with virtually all the retail banks in the United Kingdom that enables customers to use retail facilities to access their banking services. According to The Post Office, the financial services it offers generated €205 million in 2017 and €215 million in 2018. (Cost data were not available.) Officials from The Post Office told us this agreement has not only generated income, but also increased foot traffic to their retail locations. France’s postal operator, La Poste, established a bank in 2005 to provide a full range of banking products and services through its retail facilities. According to La Poste, its banking services generated net revenues of €5.5 billion in 2018, which was down from €5.6 billion in net revenue in 2017. However, USPS officials and representatives from banking associations cautioned that the financial and regulatory infrastructures of other countries are too different from those of the United States to suggest that USPS could achieve similar results. In addition, foreign postal operators are starting to sell their banks or have a franchise model with relatively lower-paid staff. There were mixed views on whether there would be demand for any or all of these nonpostal products and services. On the one hand, USPS officials, officials from other federal government entities, postmasters we surveyed, and stakeholders generally said that there was little demand for many of the additional nonpostal products and services USPS could offer at its facilities. In particular, postmasters did not indicate very high demand for any of the nine additional nonpostal products or services we asked about in our survey. The highest response for products and services in demand was only about 36 percent (53 of 149) of postmasters who said there was demand for notary services, and the next highest was about 33 percent (49 of 149) of postmasters who said there was demand for printing and photocopying services. Also, officials from the 12 federal government entities noted that while they were open to new partnerships in which mail clerks perform transactions on their behalf or in which the entities would have access to retail facility space, officials did not identify many specific examples of a need for such services. None of the postmasters we surveyed reported that they had been approached by community members in the last 10 years to share or lease space at their retail facility. On the other hand, based on studies we reviewed and interviews with representatives from consumer groups, there may be demand for certain offerings at retail facilities, such as check cashing and payday loan services, particularly if offered at a lower price than competitors. For example, in 2014, the Pew Charitable Trusts conducted a nationally representative survey of 1,626 adults and estimated that only around a quarter of American adults would be very likely or likely to use certain financial products, including check cashing, prepaid cards, bill pay, and small-dollar loans, if offered at USPS retail facilities. However, for those surveyed who were already using such services, respondents indicated they would likely obtain these at USPS retail facilities if offered at a lower price. Regarding feasibility, stakeholders and postmasters identified various issues related to what types of nonpostal products and services made sense to them. Postmasters indicated that displaying information about government programs and printing and photocopying services were the most feasible potential service, about 64 percent (95 of 149) and about 63 percent (96 of 149), respectively. In contrast, postmasters thought grocery pick-up and public wifi services were the least feasible, with about 78 percent (116 of 149) and 64 percent (96 of 149) of postmasters, respectively, indicating that they were not feasible. A representative from one of the employee unions told us that the success of additional nonpostal efforts would depend on the retail locations’ having adequate staffing levels. The representative also said that any nonpostal efforts should be designed to align with community needs and the work that retail facility employees already conduct. Officials from the Federal Communications Commission and representatives from a telecommunications association told us telecommunications companies may consider leasing space at USPS’s retail facilities that are suitable for wireless antennas to help build out 5G networks, subject to network design and business needs. In addition, representatives from one state motor vehicle administrator told us state motor vehicle administrations may consider partnering with USPS to provide state vehicle tag and title services on their behalf, such services, would not require substantial investment for USPS to undertake. Finally, USPS officials, federal government entities, and postmasters we surveyed, and stakeholders identified a variety of other limitations that would affect the viability of nonpostal products and services. We found that these limitations included limited interest for partnerships from other federal government entities, the size and unfavorable characteristics of retail facility space, complexities related to existing regulatory structures and entering into new markets, and personnel concerns. Examples of these limitations are described in table 5. Moreover, we also asked postmasters what challenges would prevent their selected retail facility from offering these additional nonpostal products and services. The most commonly cited challenges were insufficient staff or the need for additional staff training, particularly for notary services, driver’s license or other state license services, financial services, and banking services. As discussed above, given that these potential nonpostal products and services may have benefits but face concerns about their viability, USPS and policy makers need to consider the benefits, costs, and limitations of potential nonpostal efforts before introducing new efforts. In particular, though some efforts could create benefits like enhanced access for consumers, a variety of challenges may limit revenue generation in such a way that the potential offerings are unlikely to significantly improve USPS’s financial condition. Moreover, there are a number of limitations to be considered, including a potential lack of demand and factors affecting USPS’s ability to implement such offerings. We provided a draft of this report to the Census Bureau; Federal Communications Commission; GSA; U.S. Department of Veterans Affairs; U.S. Department of Homeland Security’s Federal Emergency Management Agency and Transportation Security Administration; U.S. Department of the Interior’s Fish and Wildlife Service and National Park Service; U.S. Internal Revenue Service; U.S. Department of Justice’s FBI and Office for Victims of Crime; U.S. Department of State; and USPS. The Census Bureau and GSA sent us technical comments, which we incorporated as appropriate. In its response, USPS reiterated its legal constraints but noted that there were other limitations affecting its ability to expand its offering of nonpostal products and services. These limitations include low net revenue potential, low potential for significant market share, high operational costs, and limits on amounts that could be charged. USPS noted, however, that it continues to explore partnerships with other federal agencies. We are sending copies of this report to the appropriate congressional committees, the Postmaster General, the Chairman of PRC, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. the costs, revenues, and non-revenue benefits associated with U.S. Postal Service’s (USPS) retail facilities; the nonpostal efforts USPS has conducted since 2008 to increase revenues and non-revenue benefits from its retail facilities, and the costs, revenues, non-revenue benefits, and challenges of such efforts; and the key considerations of additional nonpostal efforts that USPS could take to increase revenues and non-revenue benefits from its retail facilities. For this report, we focused on USPS-managed retail facilities, including USPS-operated post offices, postal stations, branches, and carrier annexes, as defined in USPS’s Annual Reports to Congress. We also used the term “nonpostal” to refer to activities that are not directly related to mail delivery. We selected 2008 to begin our analysis because that was when new restrictions on nonpostal services took effect. To describe the costs, revenues, and non-revenue benefits of USPS’s retail facilities, we reviewed USPS’s financial analyses of its retail network from fiscal years 2017 and 2018, the only years USPS conducted such analyses at the time we published this report. For these analyses, USPS identified the sources and amounts of retail facility costs (e.g., personnel, rent, and utilities) and revenues (e.g., stamp sales and post office box rental fees) for most of USPS’s retail facilities. Because some retail facility costs USPS identified (e.g., rent and personnel) also support non- retail functions, USPS used models to distribute such costs across each retail facility. USPS’s analyses also calculated the net revenues (revenues minus costs) for those retail facilities included in its analysis. Further, USPS examined characteristics of retail facilities, such as the facilities’ surrounding population densities, to describe factors that may be related to retail facilities that did or did not achieve positive net revenues. Based on interviews with USPS officials and USPS documents we reviewed, we determined the reliability of these analyses were sufficient to describe costs and revenues of retail facilities. We compared retail facility costs and revenues with USPS’s total costs and revenues to determine how much retail facilities contribute to total costs and revenues. In addition, we identified the number of retail facilities located in urban, suburban, or rural areas where the revenues did and did not cover costs. We defined these geographic categories using USPS’s definitions in its retail facility cost and revenue analysis. We also interviewed USPS officials to describe why revenue at retail facilities may not cover costs. To describe non-revenue benefits of USPS’s retail facility network, we reviewed relevant publications and studies, such as those conducted by USPS, USPS’s Office of Inspector General (OIG), and USPS’s oversight body, the Postal Regulatory Commission (PRC). We also examined 2018 data from USPS on the number of customer visits at the retail facilities for which data were available. Additionally, we interviewed representatives from two consumer groups, which we selected based on their ability to provide us with consumer viewpoints on retail facilities and their offerings, and two organizations representing USPS workers at retail facilities: American Postal Workers Union (APWU) and United Postmasters and Managers of America (UPMA). To examine the nonpostal efforts USPS has conducted to increase revenues and non-revenue benefits from its facilities since 2008 and their results, we first identified nonpostal products and services offered at USPS’s retail facilities through interviews with USPS officials and reviews of relevant publications, such as PRC’s report on USPS’s product offerings and USPS OIG reports. See table 6 for a complete list of USPS OIG reports we reviewed. We also identified the number of retail facilities that offered each identified nonpostal product and service, where data was available, using USPS’s facility data from November 2019, as well as information from USPS’s Inspection Service on facilities that participated in the Department of Justice’s Office of Victim’s National Crime Victims’ Rights week. Because USPS’s facility data included facilities without retail functions, we merged these with data from USPS’s fiscal year 2018 financial analysis of its retail network to ensure we included only facilities with retail functions. To describe the revenues of the nonpostal products and services we identified, we reviewed data from USPS’s fiscal year 2018 financial analysis of its retail network. We also reviewed USPS OIG reports to identify indirect ways that nonpostal products and services can contribute to revenue at retail facilities. To identify the costs of these activities, for which information was available, we reviewed USPS’s fiscal year 2018 Annual Compliance Report, PRC’s fiscal year 2018 Annual Compliance Determination Report, and non-public data provided to us by USPS. From these reports, we also determined whether the revenue USPS generated from these nonpostal efforts did or did not exceed costs. We reviewed only fiscal year 2018’s revenue and costs because we did not have revenue data prior to fiscal year 2017 and USPS was unable to provide information on trends. As one of the nonpostal efforts USPS can currently take at its retail facilities includes leasing space for revenue, we reviewed USPS’s data on the amount of vacant rentable space for fiscal years 2017 and 2018 (the only available years), on tenants at its facilities as of January 2020, and on the amount of revenue USPS collected from its leased space from 2018. Based on interviews with USPS officials and USPS documents we reviewed, we determined the reliability of these data were sufficient to describe USPS’s efforts and results of leasing excess space at retail facilities. To obtain stakeholder views on USPS’s nonpostal efforts, including the costs, revenues, non-revenue benefits, and challenges of these efforts, we interviewed officials from six federal government entities that partnered with USPS on initiatives as well as representatives from APWU, UPMA, and two consumer groups. See table 7 for a complete listing of the entities we interviewed. We also interviewed officials from the U.S. General Services Administration (GSA), which leases space on behalf of USPS to other federal government entities, and the Association of United States Postal Lessors, which represents entities that lease space to USPS. We also interviewed postal stakeholders at three retail facilities that had vacant leasable space, leased space to other entities, or offered nonpostal products and services. We selected these locations to obtain information on a variety of USPS’s current nonpostal efforts. To examine the benefits and key considerations of offering additional nonpostal efforts at USPS retail facilities, we interviewed USPS officials, postal stakeholders, and postal experts, selected based on prior work; reviewed prior GAO reports and relevant USPS OIG studies; and attended a forum exploring community use of USPS’s delivery infrastructure, including retail facilities. From the studies we reviewed and stakeholder suggestions, we selected and categorized examples of nonpostal efforts that were mentioned at least twice. To obtain stakeholder views on the potential benefits and limitations of such offerings, we interviewed representatives from consumer, industry, and state licensing groups. We selected these entities because of their potential to be affected by USPS offering additional nonpostal products and services. We also interviewed officials from the six federal entities that have partnered with USPS on initiatives and an additional six federal government entities that could potentially establish expanded or new partnerships with USPS. See table 7 above for a complete listing of the entities we interviewed. We also interviewed two foreign postal operators—France’s La Poste and the United Kingdom’s Post Office— that have experience with nonpostal products and services similar to those we reviewed and reviewed relevant documentation, such as their annual fiscal year 2018 financial reports. We selected these postal operators based on prior work and other studies and to provide us with insight into their experiences. Finally, we reviewed statutes, including the Postal Accountability and Enhancement Act (PAEA); regulations; and legal rulings, to evaluate USPS’s current legal authority to provide these services. The views presented in our report are not generalizable to those of all stakeholders. Further, we surveyed USPS postmasters to obtain additional perspectives on the benefits of USPS’s retail facilities, the nonpostal efforts offered at those facilities, and the key considerations of offering additional nonpostal products and services. Specifically, we conducted a non-generalizable, web-based survey of postmasters who managed retail facilities located in urban, suburban, and rural areas from August to September 2019. We defined these geographic categories using USPS’s definitions, as described above. Using the dataset of facilities from the USPS revenue study, we removed all facilities that were located in an overseas American territory, any facility missing a geographic category code, and certain kinds of facilities that were not relevant to the survey. Using these filters we identified a sample frame of 26,600 retail facilities. We randomly sorted the facilities within each of the three geographic categories and took the first 150 from each random sort. We then matched the selected USPS facilities with the postmaster responsible for them, using USPS’s postmaster data provided to us in May of 2019. Because postmasters may manage more than one retail facility, we capped the number of surveys an individual postmaster could receive at one. If a postmaster already selected in the random sort occurred again the selected facility was omitted from the sample. We restricted the total sample size to no more than 100 unique postmasters within each stratum. The sample is comprised of 83 postmasters who oversee an urban retail facility, 100 postmasters who oversee a suburban retail facility, and 100 postmasters who oversee a rural retail facility. Approximately 52 percent of our sample—or 146 postmasters—completed the survey. The survey questionnaire can be viewed in appendix II. In developing, administering, and analyzing the survey, we took steps to minimize non-sampling error that may result from differences in how a question is interpreted and the sources of information available to respondents. To help reduce measurement error, we consulted an experienced former postmaster for input on the development of the survey instrument, and also conducted pretests of the draft questionnaire with four postmasters drawn from the intended survey population. The questionnaire was modified throughout development and pretesting to improve clarity of the questions, and we removed questions when our modifications were unable to remedy observed difficulties in interpretation. To maximize survey response, we sent pre-notification letters by postal mail to the selected respondents prior to launching the web survey. After launching the survey, we sent multiple email reminders and extended the submission deadline, and also conducted follow-up phone calls. Lori Rectanus, (202) 512-2834 or rectanusl@gao.gov. In addition to the contact name above, Kyle Browning (Assistant Director); Anne Doré (Analyst in Charge); Isabelle Aboaf; Carl Barden; Karen Chen; Barbara El Osta; Geoff Hamilton; Serena Lo; Tina Paek; Samuel Portnow; Malika Rice; Matthew Valenta; and Lauren Voloder made key contributions to this report.", "summary": "USPS manages over 31,000 retail facilities, which help it provide postal services throughout the country. However, USPS faces financial challenges. In general, USPS is prohibited by statute from providing nonpostal products and services (i.e., services not directly related to mail delivery) unless approved by the Postal Regulatory Commission. But given the ubiquity of the retail network, some stakeholders have suggested that offering additional nonpostal products and services could help USPS generate revenue and provide benefits for consumers and communities. GAO was asked to review opportunities to enhance the value of USPS's retail facilities. This report examines: (1) the costs, revenues, and other benefits associated with USPS's retail facilities; (2) USPS's nonpostal efforts since 2008 at retail facilities and the outcomes; and (3) considerations of new nonpostal efforts at retail facilities. GAO analyzed USPS retail facility costs and revenue data from fiscal years 2017 and 2018 (the only years available); reviewed relevant documents and reports from USPS and others; conducted a non-generalizable survey of USPS postmasters who managed rural, suburban, and urban retail facilities; and interviewed USPS officials, and stakeholders, including postal employee unions, industry and consumer groups, and federal agencies that partner with USPS to obtain views on current and potential nonpostal efforts. GAO is making no recommendations. USPS, in its comments, reiterated that it faces various constraints to new offerings at retail facilities. In 2018, U.S. Postal Service's (USPS) retail facilities, such as post offices, generated about $10.5 billion in revenue and cost approximately $5 billion to operate, making them profitable overall. While such facilities accounted for about 15 percent of USPS's total fiscal year 2018 revenues, and about 7 percent of its total costs, stakeholders identified other benefits that retail facilities provide for communities—particularly in rural areas—such as local access to government information and services. Since 2008, USPS has offered a variety of nonpostal products and services at its retail facilities that have generated some revenue and other benefits. USPS data show that the nonpostal products and services for which USPS captures revenue data, such as money orders, generated about $431 million in total revenue in fiscal year 2018 and were profitable overall. Stakeholders said many of these nonpostal products and services also provided other benefits, such as enhanced convenience for customers, and postmasters GAO surveyed said some offerings, such as passport services, were highly valued in their communities. Offering additional nonpostal products and services at USPS retail facilities could provide consumer, government, or community benefits, but viability may be limited. Stakeholders said new offerings, such as expanded financial products or government services could, for example, enhance consumers' access and government efficiencies. In particular, some noted that USPS could provide a viable banking alternative for those lacking banking services. However, USPS officials, postmasters GAO surveyed, and stakeholders GAO interviewed said that additional offerings may generate minimal revenue and that USPS may face factors limiting the viability of these offerings. For example, groups representing states' licensing agencies said offering state hunting and fishing licenses could be problematic given different state requirements. Also, stakeholders said USPS may not have the expertise nor the required capital to enter the market of some of these new offerings. Given such concerns, USPS and policy makers need to carefully weigh costs, benefits, and limitations of any new offerings.", "document_type": "gao"}
{"report": "OSS is software distributed under a license that provides broad rights to use, modify, and redistribute the original source code. Open source licenses impose certain obligations on users who exercise these rights. Specific obligations vary among the many different open source licenses. Common obligations include making the source code available, publishing a copyright notice, or giving any recipient of the program a copy of the license. Certain restrictive open source licenses allow users to copy, modify and distribute software provided that modified versions (i.e., derivatives) are subject to the same license terms and conditions as the original code. This is intended to prevent software that is derived from or contains code issued under such a license from becoming a closed- source product that can be marketed and sold exclusively. The reuse of OSS is viewed as a promising means to reduce development costs while improving software quality. According to software experts, software reuse has the potential to: increase reliability because systems will be developed with thoroughly tested and proven components, increase productivity by reducing the time and effort needed to develop software, reduce costs by enabling the sharing of knowledge and practices needed to develop and maintain software, and establish a more standard and consistent approach to software development and maintenance by using common components and procedures. In August 2016, OMB issued a memorandum to the heads of departments and agencies to ensure that new custom-developed source code be made available for reuse across the federal government. The memorandum was intended to improve the way federal agencies buy, build, and deliver information technology and software, and required that all agencies establish a pilot program under which at least 20 percent of new custom-developed code would be released as OSS for 3 years. OMB also required that agencies develop a metric to calculate the percentage of code released as OSS to gauge its progress on implementing the pilot program. OMB’s memorandum also identified four supporting requirements, among others, needed to implement an OSS pilot program: Issue an OSS policy that ensures code is available for government- wide reuse. Conduct an OSS three-step software solutions analysis that includes: (1) a strategic analysis and analysis of alternatives, (2) consideration of existing commercial solutions, and (3) consideration of custom development. In addition, agencies must consider several factors throughout each of the three-steps of the analysis such as cloud computing and open standards. Secure data rights to government-wide reuse and inventory new custom code, in accordance with the guidance provided by the code.gov website. Facilitate the OSS community by developing and releasing the code in a manner that (1) fosters communities around shared challenges; (2) improves the ability of the OSS community to provide feedback on, and make contributions to, the source code; and (3) encourages federal employees and contractors to contribute back to the broader OSS community by adding value to existing projects. In doing so, agencies should comply with the following principles: (1) leverage existing communities, (2) engage in open development, (3) adopt a regular release schedule, (4) consider code contributions, and (5) document source code to facilitate use and adoption. DOD’s CIO is responsible for implementing OMB’s requirements for the department’s OSS pilot program. The CIO reports directly to the Secretary of Defense, and is responsible for the department’s information technology (including national security systems and defense business systems), information resources management, and efficiencies. The CIO is also responsible for developing strategies and policy on the operation and protection of all of the department’s information technology and information systems. Other responsibilities include maintaining a consolidated inventory of mission critical and mission essential information systems, evaluating and monitoring performance measurements, and other duties to manage the information environment throughout the department. In addition, the Defense Digital Service is responsible for assisting the CIO in implementing the OSS pilot program, among other initiatives. The Defense Digital Service is composed of commercially experienced software developers, software designers, product managers, and problem solvers within DOD. The organization works on specific projects or programs in support of the DOD in a hands-on way to materially improve digital services across the department. The Defense Digital Service also works with the CIO to monitor the identified programs, facilitate the process to open source the code, and populate the source code inventory located on Code.mil. In June 2018, DOD’s CIO issued a report to Congress as directed by section 875(b) of National Defense Authorization Act for Fiscal Year 2018 (Public Law 115-91). The report provided Congress with the department’s plan to implement the OSS pilot program established by OMB’s memorandum. In the report, the CIO committed to sharing its unclassified, custom-developed source code as widely as possible in four ways: (1) review and select software programs that have self-identified to the Defense Digital Service as ready to open source its code; (2) query its contracts database to identify contracts that contain appropriate data rights language; (3) determine if source contained within existing source code repositories can be made available; and (4) issue a department- wide data call to identify and select programs where new, custom code is being developed. The CIO also reported that the department would prioritize and assess identified software programs and work with components to develop mechanisms to report progress. The report included selection criteria for identifying candidate software programs: (1) programs with contractually secured government data rights; (2) programs with contractual rights to enable, support, and enforce DOD and government-wide sharing and reuse of custom-developed code; (3) programs that have appropriate administration to ensure that government’s rights are maintained; and (4) programs that utilize best practices to ensure custom-developed code, documentation, and other associated materials are delivered in a reusable manner. The CIO also reported that the Defense Digital Service would assist programs. Specifically, the Defense Digital Service is to develop guidelines, processes, and answers to frequently asked questions regarding the use of OSS. In October 2018, the CIO issued a memorandum to the Chief Management Officer, secretaries of the military departments, Chairman of the Joint Chiefs of Staff, under secretaries of Defense, chiefs of the military services, general counsel, Director of Cost Assessment and Program Evaluation, Director of Operational Test and Evaluation, and the Assistant Secretary of Defense for Legislative Affairs notifying them of the need to implement an OSS pilot program in accordance with OMB’s 2016 memorandum. The CIO required these organizations to take four actions within 30 days of issuing the memorandum: Identify all unclassified custom-developed source code created or paid for by the department—regardless of data rights and open source status—created on or after August 2016 and provide the CIO information required by the guidelines on the code.gov website; Identify and provide a point of contact that can participate in open source efforts; Direct authorizing officials to rapidly review and approve unclassified code for public, open source release after appropriate security, code, and policy review; and Direct contracting officers to secure the least restrictive data rights to custom-developed source code in all future contracts in accordance with DOD federal acquisition regulations. DOD was mandated by law to initiate the OSS pilot program established by OMB memorandum M-16-21 which required (1) releasing at least 20 percent of newly custom-developed code each year for the term of the pilot program, and (2) collecting additional data concerning new custom software to inform measures to gauge the performance of the pilot program. Further, the OMB memorandum also required DOD to (1) issue an OSS policy, (2) conduct an OSS analysis, (3) secure data rights and inventory custom code, and (4) facilitate the OSS community. As of late April 2019, DOD had not fully implemented the OSS pilot program mandated by law. DOD had partially implemented the requirement of releasing at least 20 percent of newly custom-developed code as OSS. Specifically, as of July 2019, the Code.gov website reported that the department had released less than 10 percent of its custom developed code. The department was in the early stages of its pilot program and had not determined when the pilot would be fully implemented. The CIO reported that the size of the department makes it nearly impossible to inventory all of its source code custom developed since August 2016. As such, the CIO stated that it would be difficult to meet the OMB memorandum’s goal of releasing at least 20 percent of its new custom code as OSS. In addition, DOD had not implemented the requirement to develop a consistent measure to gauge the performance of the department’s pilot program. DOD had not developed such a measure due to a lack of consensus in the department about what data should be collected. According to the CIO, if the measure is “lines of code,” then it unfairly discounts projects that invest a significant amount on research, but are small otherwise. If the measure is “project hours,” then it discounts those projects that came about from sparks of innovation that took little time to develop. If the measure is “project count,” then it ignores the other two possible measures. The CIO noted that since the components will be expected to collect and report the required data, the CIO office plans to reach out to them to facilitate consensus around data needs and what measure should be used to calculate and monitor performance. Regarding the four OMB memorandum requirements supporting the implementation of the pilot program, the department partially implemented three and did not implement the remaining one. Table 1 describes the extent to which DOD implemented the OMB supporting requirements, and is followed by a description of DOD’s efforts on each requirement. Issue OSS policy. DOD had not implemented the requirement to issue an OSS policy. According to DOD’s CIO, the department has existing acquisition policies applicable to OSS, such as the 5000 series and a memorandum issued in October 2009 that clarifies OSS. However, according to DOD officials, these policies are outdated and do not comply with OMB’s memorandum. For example, while the department’s policies indicated that programs must conduct an analysis of alternatives, trade studies, or a business case analysis prior to initiating any technology acquisition or custom code development, they did not require programs to consider the value of publishing custom code as OSS and negotiate data rights reflective of its value. The department acknowledged that it does not have a policy that addresses the OMB memorandum’s requirement. According to the CIO, the department had been slow to develop a policy because these types of changes require significant resources, coordination, and buy-in across the department that will take additional time to address. The CIO also stated that the department plans to update its existing OSS memorandum by the end of the 2019 calendar year and issue it as policy. In particular, DOD intends to work with acquisition and program management officials to define approaches, processes, and best practices to expand software reuse. If the department effectively implements this intended step consistent with the OMB memorandum, DOD should be able to fully address this requirement. Conduct OSS analyses. DOD had partially implemented the requirement to conduct analyses to consider alternative software solutions. According to the DOD CIO, of the three elements required by OMB for a three-step analysis, the department’s current 5000 series policy addresses some of these elements. For example, as previously mentioned, the policy required programs to conduct an analysis of alternatives, trade studies, or a business case analysis prior to initiating any technology acquisition or custom code development. However, according to the CIO, DOD’s policy did not require programs to consider the value of publishing custom code as OSS and negotiate data rights reflective of its value-consideration. According to the CIO, the department has plans to address gaps in its existing policy by the end of the 2019 calendar year. If the department effectively implements this intended step consistent with the OMB memorandum, DOD should be able to fully address this requirement. Secure data rights and inventory new custom code. DOD had partially implemented this requirement by initiating the process to secure data rights for government-wide reuse and inventory its new custom code. Specifically, the October 2018 memorandum called for defense organizations to direct contracting officers to secure the least restrictive data rights for custom-developed source code in all future contracts, and identify all unclassified custom-developed source code created after August 2016. However, when we discussed this matter with DOD information technology and software officials responsible for the management of software in December 2018—2 months after the October 2018 memorandum had been issued— seven of 11 officials were unaware of the statutory mandate to initiate the pilot program in compliance with the OMB memorandum. Further, DOD has not provided a milestone for when it expects its inventory to be completed. Facilitate OSS community. DOD had partially implemented the requirement to establish an OSS community. According to the DOD CIO’s Custom Developed Source Code Data Call memorandum, dated October 2018, the DOD CIO is working with Defense Digital Service to develop guidelines and processes on when and how to open source code. In February 2017, DOD announced the launch of Code.mil, an open source initiative led by Defense Digital Service, that allows software developers around the world to collaborate on unclassified code written by federal employees in support of DOD projects. Finally, the Defense Information Systems Agency established a website (Forge.mil) where community members can collaborate on open source and DOD community source software. The Forge.mil website also enables collaborative development through services such as software version control, requirements management, discussion forums and document repositories. However, DOD had not yet fully engaged in open source code development, established a regular release schedule for its software code, or fully documented its source code to facilitate use and adoption department-wide. According to the CIO, the department is in the early stages of implementing the OSS pilot program and had not yet published a revision to the existing OSS policy memorandum. The CIO stated that the office plans to request collaboration and input from organizations throughout DOD for improvement initiatives and identifying specific processes and expectations for improving custom-developed software within the Department. However, DOD has not provided a milestone for when the requirements will be fully implemented and stated that achieving 100 percent compliance is not a realistic expectation. Until DOD fully implements the OSS pilot program mandated in the National Defense Authorization Act for Fiscal Year 2018 including the requirements of OMB memorandum M-16-21, the department will likely miss opportunities to achieve related cost savings and efficiencies. Further, the department will not be effectively positioned to ensure management oversight and implementation of the pilot program. DOD officials representing 11 components reported that OSS can potentially yield financial benefits and increase efficiency. Officials provided the following examples of financial benefits: Officials in the office of the Navy Chief Information Officer, the Army Communications-Electronics Command, the Office of the Under Secretary of Defense for Acquisition and Sustainment, and the Office of the Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics stated that OSS is generally less expensive than commercial off-the-shelf (COTS) leading to cost savings. An Air Force official we spoke with stated that the increased use of OSS may potentially result in cost savings. Further, the official noted general criteria used by the Air Force to identify software projects that may potentially be appropriate for the use of OSS. Specifically, OSS should be considered if, among other things, the required maintenance would not result in a reduction of cost savings or efficiency if the maintenance is performed in-house. An official we spoke with at the Defense Advanced Research Projects Agency stated that OSS has positive benefits in terms of reducing costs by reducing duplicative efforts. In addition, this official also stated that OSS allows institutions of any size and budget to partake in shared investments providing access to software capabilities at a much lower cost. A program manager from the Defense Information Systems Agency reported that the agency had identified an OSS solution that provided more functionality at less cost than the commercial solution provided through a vendor. The program manager explained that when the agency implemented the new OSS solution, it realized $20 million in annual savings over the commercial solution that had been maintained by a vendor. Officials also shared examples of how OSS can increase efficiency in software development. For example, Officials from the offices of the Navy CIO and the US Marine Corps CIO stated that OSS solutions may increase efficiency by providing a rapid resolution to the needs and requirements of users. In contrast, rapid development efforts are not conducive for COTS solutions because of the long process required to obtain solutions that are needed quickly. Similarly, an Air Force official noted that the increased use of OSS may result in increased efficiency by providing rapid responses to user requirements. A program manager from the Defense Information Systems Agency reported that the selection of an OSS solution rather than a COTS solution contracted through a vendor had resulted in increased efficiency. The official explained that the use of the OSS solution allowed the agency to develop and maintain in-house skills that would not have been available had they opted to contract with a vendor providing a skilled workforce. Officials from the 11 components expressed mixed views on managing cybersecurity risks that could be posed by using OSS. Specifically, officials from three components expressed their views that security concerns and the lack of a cybersecurity governance process could result in the sporadic use of OSS. For example: A Navy CIO official viewed insider threats, such as a disgruntled employee embedding malicious code, as a factor that could significantly limit the use and sharing of OSS. According to Navy officials, without a process to verify that the software is free of malicious code, the Navy would risk the assurance it requires to increase the use of OSS. The official said that, in contrast, such concerns are not an issue when it comes to COTS software because of the test and verification process to ensure it is free from malicious code. An official in the office of the Marine Corps CIO stated that OSS is used sporadically in their software development efforts because some cybersecurity officials within the Marine Corps discourage its use due to security concerns. An official from the Army’s Communications-Electronics Command noted that DOD lacks a governance process once the originating entity releases the source code as open source. The originating entity no longer retains control over redistributed versions of the source code. According to Communications-Electronics Command officials, Army project managers may be hesitant to utilize OSS because of this perceived security risk. On the other hand, DOD officials from eight components stated that the potential cybersecurity risks posed by the use of OSS were manageable and that the use of OSS should not be limited. For example: The policy advisor from the Office of the Under Secretary of Defense for Acquisition and Sustainment noted that scanning tools to analyze and identify safe and reliable open source code are not being used. Employing available scanning tool options could result in discovering available OSS. The policy advisor also noted that building security into software operations, rather than through the development of software, would enable users to know if code has been subverted and to react appropriately. A program management official from the Office of the Under Secretary of Defense for Acquisition and Sustainment suggested that security concerns may be mitigated by establishing a secure repository for trusted code. An official in the Office of the Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics reported that, as long as OSS is properly vetted to ensure it is secure and free from malware, it offers an opportunity for the department to achieve cost savings and efficiencies. Pilot testing the use of OSS is an important way to ascertain and improve the way DOD buys, builds, and delivers information technology and software solutions. However, the department is in the early stages of implementing its pilot program and had not determined when the pilot would be fully implemented. Specifically, DOD has not made 20 percent of its new code available for reuse nor has it identified a measure to gauge the performance of its pilot program. Moreover, DOD has not yet established milestones for securing data rights and conducting an inventory or facilitating community. Until DOD fully implements its pilot program and establishes milestones for completing the OMB requirements, the department will not be positioned to take advantage of significant cost savings and efficiencies. We are making the following four recommendations to DOD: The Secretary of Defense should ensure the department implements the pilot program by releasing at least 20 percent of newly custom- developed code as OSS. (Recommendation 1) The Secretary of Defense should ensure the department identifies a measure to calculate the percentage of code released to gauge its progress on implementing the pilot program. (Recommendation 2) The Secretary of Defense should ensure the department establishes milestones for completing the requirements of OMB memorandum M- 16-21 of securing data rights and conducting an inventory. (Recommendation 3) The Secretary of Defense should ensure the department establishes a milestone for completing the OMB memorandum’s requirement of facilitating an OSS community. (Recommendation 4) DOD provided written comments on a draft of this report, which are reproduced in appendix II. In its comments, the department did not concur with our first recommendation, partially concurred with our second recommendation, and concurred with the third and fourth. DOD did not concur with the first recommendation on ensuring that the department implements the pilot program by releasing at least 20 percent of newly custom-developed code as OSS. The department stated that it does not believe that the pilot program as described in the OMB memorandum is implementable as proposed. For example, DOD asserts that most of DOD’s custom developed software is created for weapons systems and releasing the associated code is sensitive for national security reasons. In addition, the size and complexity of DOD presents unique challenges for the department compared to other federal agencies such as inventorying all software development projects to establish a baseline. DOD added, however, that the OMB memorandum explicitly states that national security exceptions do not apply to the pilot program. DOD also stated that it recognizes the value of collaborative software development and has plans to release additional guidance on releasing OSS and procedures for maintaining its inventory. Once DOD establishes a baseline inventory of custom-developed software and the procedures for maintaining it, the department states it will be able to determine if the 20 percent is an appropriate goal. We understand the potential constraints DOD faces and that national security considerations are to be factored into decisions DOD will need to make about which custom developed software to include in the pilot. However, DOD is mandated by law to implement the OSS pilot program established by OMB memorandum M-16-21. Further, the OMB memorandum instructs agencies to refrain from selecting code for release that would fall under exceptions such as national security risk. As such, DOD has flexibility on making decisions about which custom-developed code to include in the pilot. While we agree that a baseline inventory is needed, DOD must include at least 20 percent of new custom-developed code each year for the term of the program to satisfy the mandate. DOD partially concurred with the second recommendation on ensuring that the department identifies a measure to calculate the percentage of code released to gauge its progress on implementing the pilot program. Specifically, the department stated that the additional guidance it plans to release before the end of 2019 on OSS will include measures to gauge how much code has been developed and how much has been released. In addition, DOD noted that these measures will support good management of the overall portfolio of information technology, even in the absence of the mandated pilot program established by the OMB memorandum. We believe that the measure to calculate the percentage of code should be used to assist the department in meeting the OMB memorandum’s requirements. We also agree with the benefits of developing a measure to manage the portfolio of information technology. DOD concurred with the third and fourth recommendations related to establishing milestones for completing the OMB memorandum’s requirements of securing data rights, conducting an inventory, and facilitating an OSS community. According to the department, it has issued a memorandum that directed contracting officers to secure the least restrictive data rights to custom-developed source code, in furtherance of the OMB requirements, and also included a data call that forms an initial basis of an inventory of custom-developed software. Regarding facilitating an OSS community, DOD stated that it has formed a community of practice called DevSecOps that is open to all software development organizations in the department and plans to use this forum to facilitate collaboration on the use of OSS. We are sending copies of this report to the appropriate congressional requesters and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-4456 or at harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Our objectives were to: (1) assess the extent to which the Department of Defense (DOD) has implemented the open source software (OSS) pilot program and other related requirements established by the Office of Management and Budget (OMB), and (2) describe the views of responsible DOD officials on the use of OSS to achieve efficiency at the department. To address the first objective, we selected six requirements from the OMB memorandum titled the Federal Source Code Policy: Achieving Efficiency, Transparency, and Innovation through Reusable and Open Source Software (M-16-21, Aug. 8, 2016) as criteria to assess the extent to which DOD has implemented the OSS pilot program. Two requirements establish the OSS pilot program: (1) releasing at least 20 percent of newly custom-developed code each year for the term of the pilot program, and (2) developing a metric to gauge the performance of the pilot program. The other four requirements support the implementation of the pilot program: (1) issuing an OSS policy, (2) conducting an OSS analysis, (3) securing data rights and inventorying custom code, and (4) facilitating the OSS community. We met with officials from OMB to collect background information on the selection of requirements for the pilot program established in memorandum M-16-21. We also met with officials from the Office of the DOD Chief Information Officer (CIO) and the Defense Digital Service to discuss the status of the department’s implementation of the OSS pilot program. We reviewed DOD’s June 8, 2018 report to Congress and its October 2018 memorandum that details the department’s plans to implement the pilot program and compared them to the six requirements. To determine the extent to which the pilot program had been implemented, we evaluated DOD’s efforts to address each of the requirements using a 3- stage gradient scale (implemented, partially implemented, and not implemented). The requirement was assessed to be fully implemented if DOD provided us with sufficient evidence that the requirement had been fully met. We assessed a requirement to be partially implemented if DOD provided us with documentation of initial plans or had initiated action towards implementing the requirement. We determined that a requirement was not implemented when DOD did not provide us with documentation of planned or initiated actions to implement the requirement. To address the second objective on views of various responsible DOD officials, using professional judgement, we selected components across the department responsible for the management and development of OSS. The scope of stakeholders selected represent department-wide nongeneralizable views including military components, defense agencies, and other offices. At least one representative was selected from the following components: (1) Office of the Under Secretary of Defense for Acquisition and Sustainment, (2) Office of the DOD CIO, (3) office of a military service CIO, (4) Military Service Software Center or Command Center, (5) the Defense Information Systems Agency, and (6) the Defense Advanced Research Projects Agency. We conducted interviews with DOD officials from the following entities: Office of the Under Secretary of Defense for Acquisition and Sustainment; Office of the DOD CIO; Offices of the Navy and Marine Corps CIOs; Office of the Air Force Chief Technology Officer; Army Communications Electronics Command; the Defense Information Systems Agency; the Defense Advanced Research Projects Agency; and the Office of the Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics. In order to summarize and report the views of the responsible DOD officials, we conducted structured interviews with representatives from the selected components. Each interview consisted of the same discussion topics based on the pilot program requirements established in OMB’s memorandum. The scope of this objective represents individual thoughts, views, and opinions and is not intended to convey an official or department response. Prior to each interview, participants were provided with OSS discussion topics, and the OMB memorandum, titled the Federal Source Code Policy: Achieving Efficiency, Transparency, and Innovation through Reusable and Open Source Software (M-16-21, Aug. 8, 2016). The contents of each interview were reviewed and summarized to identify the general views of OSS and on the anticipated implementation of the pilot program requirements established in OMB’s memorandum. We noted similarities and differences in the responses provided by the officials in the use of OSS including, but not limited to, potential benefits of using OSS, managing associated risk, and opinions on implementing the pilot program in compliance with the OMB memorandum. Discussions were split into two topic areas: practices on the use of OSS, and OMB’s memorandum to establish an OSS pilot program. Specifically, the discussion topics were presented during each meeting as follows: Part I: Practices on the use of OSS Your experience or your organization’s history with the practice of using OSS as a means to achieve cost reduction or efficiencies when buying, building, or delivering information technology and software solutions. The processes or practices that you or your organization perform to leverage open source code for projects that require the acquisition or development of custom source code. The extent to which you or your organization shares or uses open source software. For example, do you share or use open source software: (1) within your organization only, (2) across the DOD with other military services or defense agencies, (3) with other federal agencies, or (4) outside the federal government with the public. Also, how is the source code shared, leveraged, catalogued, stored, and accessed. Policies or guidance currently in use for OSS. General views and opinions on the use of open source code. Part II: OMB’s Memorandum to Establish an OSS Pilot Program When and how you or your organization became aware of OMB’s memorandum on Federal Source Code Policy. Your opinions and views about the OMB memorandum. Any specific concerns or reservations about the requirements contained in the OMB memorandum. The extent to which you or your organization may already be performing the steps in OMB’s proposed Three-Step Software Solutions Analysis. Discuss the feasibility of the pilot program requirement to release at least 20 percent of new custom-developed code as OSS. We conducted this performance audit from August 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Eric Winter (Assistant Director), John Ortiz (Analyst-in-Charge), Rebecca Eyler, Franklin Jackson, and Kate Nielsen made key contributions to this report.", "summary": "Open source software is code that is released under a license which grants users the right to modify, share, and reuse the software. Making code available for reuse as open source can have major benefits such as decreasing costs and improving efficiencies. The National Defense Authorization Act for Fiscal Year 2018 required DOD to submit a plan to Congress for initiating the open source software pilot program established by OMB memorandum M-16-21. DOD submitted its plan to Congress in June 2018. The act includes a provision for GAO to report on DOD's implementation of the open source software pilot program. GAO's objectives were to (1) assess the extent to which DOD has implemented the open source software pilot program and other related requirements established by OMB; and (2) describe the views of responsible DOD officials on the use of open source software to achieve efficiency, transparency, and innovation at the department. To address these objectives, GAO compared DOD's plan for implementing the program to OMB's memo. GAO also interviewed defense officials at 11 DOD components including military departments, and defense agencies on their views about the benefits and risks of making code available as open source software. The Department of Defense (DOD) has not fully implemented an open source software pilot program and related Office of Management and Budget (OMB) requirements as mandated by the National Defense Authorization Act for Fiscal Year 2018. OMB memorandum M-16-21 calls for agencies to implement a pilot program, which it defines as (1) releasing at least 20 percent of new custom developed code as open source, and (2) establishing a metric for calculating program performance. However, DOD has not fully implemented the program and has not established the metric. The OMB memorandum also requires agencies to implement other supporting activities. These include issuing policy on government-wide use of code, conducting analyses of software solutions, securing data rights and inventory code, and facilitating the open source community. DOD has not implemented the policy requirement and has partially implemented the remaining three requirements. Regarding the policy and analysis requirements, DOD plans to issue a policy and conduct analyses by the end of the 2019 calendar year. If the department effectively implements these intended steps consistent with OMB direction, DOD should be able to fully address these requirements. For the requirement of securing data rights and inventorying code, DOD issued a memorandum that directs contracting officers to secure data rights and to identify all source code created after August 2016. However, DOD's components have not executed these activities nor has DOD identified a milestone for when they will be completed. For the facilitating community requirement, DOD issued a memorandum that encourages conversations to foster communities and allow others to contribute knowledge, among other initiatives. However, DOD has not fully engaged in open development, established a release schedule, or fully documented its source code to facilitate use and adoption. To address these areas, DOD's Chief Information Officer plans to issue guidance but has not established a milestone for doing so. Until DOD fully implements the pilot program and develops milestones for two of the four OMB requirements (secure data rights and inventory code, and facilitate community), it will not be positioned to satisfy the mandate established in the law. DOD officials from 11 components expressed their opinions that an open source pilot program would potentially result in financial benefits and increased efficiency. However, there were disparate views on how to manage the cybersecurity risk of using open source software. Specifically, officials from three components noted that security concerns could result in the sporadic use of OSS, whereas eight officials stated that the potential cybersecurity risks were managable. GAO is making four recommendations to ensure DOD implements the program and develops milestones for completing requirements in the OMB memo. DOD agreed with two but did not agree with one and partially agreed with another. As discussed in this report, GAO maintains that all recommendations are needed to satisfy the act.", "document_type": "gao"}
{"report": "TSA is the primary federal agency responsible for implementing and overseeing the security of the nation’s civil aviation system and, in general, is responsible for ensuring that all passengers and belongings transported by passenger aircraft to, from, within, or overflying the United States are adequately screened. Over 43,000 TSOs stationed across the nation’s approximately 440 commercial airports are responsible for inspecting individuals and belongings to deter and prevent passengers from bringing prohibited items on board an aircraft or into the airport sterile area. Within TSA, two offices—T&D and Security Operations—are to work together to manage TSOs and ensure their training is current and relevant. T&D is responsible for developing initial and ongoing training curricula for TSOs based in part on TSA’s standard operating procedures that govern how TSOs screen passengers and baggage. Security Operations is responsible for allocating TSO staff to airports, and scheduling TSO work hours and training availability. Within Security Operations, FSDs are responsible for overseeing security operations at the nation’s commercial airports, many overseeing multiple airports within a specific geographic area. FSDs report to one of three executive directors, who in turn are responsible for annually assessing FSD performance, including oversight of TSO training. TSA’s screener training is comprised of a compendium of courses that includes basic training for initial hires, recurrent training, remedial training, and return-to-duty training. The National Training Plan specifies annual training requirements and contains the core curriculum for TSOs, including the classes and hours required for TSOs to complete. In accordance with the Aviation and Transportation Security Act, screeners must complete a minimum of 40 hours of classroom instruction and 60 hours of on-the-job training, and must successfully complete an on-the- job training examination. Until 2016, new TSOs completed these training requirements at or near their home airports through the New Hire Training Program. In January 2016, TSA centralized this training under the TSO Basic Training Program at the TSA Academy in Glynco, Georgia. Further, in August 2018, TSA launched the first phases of TSO Career Progression, in which new hire screeners receive local training and gain experience in a limited number of screening functions before advancing to the next stage of training at the TSA Academy, roughly around the four-month mark. In 2015, in response to the DHS Office of Inspector General covert test findings that highlighted areas of concern in the passenger screening process, TSA implemented a TSO re-training effort, beginning with a nationwide training called “Mission Essentials—Threat Mitigation.” According to TSA, this training provided the opportunity for the TSO workforce to become familiar with the threat information that underlies TSA’s use of checkpoint technologies and operational procedures to mitigate risks. In 2009, OPM developed and published regulations that require agencies to evaluate training programs annually. According to the regulations, these training evaluations are to help agencies determine how well such plans and programs contribute to mission accomplishment and meet organizational performance goals. One commonly accepted training evaluation model, endorsed by OPM and commonly used in the federal government to evaluate training, is known as the Kirkpatrick model. The Kirkpatrick model consists of a four-level approach for soliciting feedback from training course participants and evaluating the impact the training had on individual development, among other things. The following describes what each level within the Kirkpatrick model is to accomplish: Level 1: The first level measures the training participants’ reaction to, and satisfaction with, the training program. A level 1 evaluation could take the form of a course survey that a participant fills out immediately after completing the training. Level 2: The second level measures the extent to which learning has occurred because of the training effort. A level 2 evaluation could take the form of a written exam that a participant takes during the course. Level 3: The third level measures how training affects changes in behavior on the job. Such an evaluation could take the form of a survey sent to participants several months after they have completed the training to follow up on the impact of the training on the job. Level 4: The fourth level measures the impact of the training program on the agency’s mission or organizational results. Such an evaluation could take the form of comparing operational data before, and after, a training modification. Since 2015, TSA’s T&D has developed and updated TSO training programs in response to findings from covert tests and reporting on emerging threats that identified risks to aviation security. T&D uses an online database to track results from covert tests and reporting on emerging threats, and any changes to training that T&D makes as a result. According to T&D data from May 2015 through June 2019, T&D officials reviewed 62 risks that warranted a review for a potential change to training, and 56 of the risks led officials to make training changes across its TSO curriculum. Overall, T&D made changes affecting 40 different training courses. Based on our review of TSO training curriculum from May 2015 through June 2019, we found that changes T&D made to its TSO training took many forms. In some cases, T&D changed training to place additional emphasis on a certain aspect of a current standard operating procedure or provide context on the importance of following it. For example, in 2019, T&D updated its instructor-led course—”Mission Essentials: Resolution Tools and Procedures”—to address covert tests where TSOs failed to detect simulated explosive devices hidden in bags or concealed on individuals at checkpoints. The training included a review of methods terrorists may use to plan and carry out attacks in order to emphasize the importance of following the standard operating procedure. The updated training also included leading practices for searching belongings and a discussion of issues that may affect a TSO’s ability to detect threat items hidden in belongings or on individuals. In fiscal year 2019, T&D also updated instructor-led courses on its explosives detection system for checked baggage to respond to covert test findings that TSOs failed to detect certain simulated explosive devices. The updated training included images of simulated explosives hidden in checked bags that replicated scenarios similar to the failed covert tests. In other cases, T&D developed TSO training in response to new or updated standard operating procedures for using technologies. T&D officials said that for this type of TSO training, they wait until TSA’s Requirements, Capabilities, and Analysis office updates or establishes new standard operating procedures for using new technologies and then develops training based on the revisions. For example, T&D developed TSO training to cover the differences between a prior and updated version of the standard operating procedure for screening passengers and belongings at security checkpoints. T&D included curriculum to cover the major changes in the standard operating procedure and incorporated additional training to address a covert test in which TSOs failed to detect a simulated explosive device at a screening checkpoint. T&D also developed training for TSOs who check passenger IDs and travel documents. The training focused on updates to the standard operating procedure and included procedures specific to the 2005 REAL ID Act, which TSA will fully implement in 2020. Additionally, T&D incorporated this new training to address covert tests that had found issues with identifying false or fraudulent travel documents. In addition to updating or developing new training involving instructor-led courses, TSA responded to identified risks by developing or updating job aids or briefings for TSOs. For example, TSA developed the “It’s Not the Container” briefing in 2017 to address risks highlighted by an attempted attack in Australia and included tactics used to conceal explosives in benign items. The briefing provided best practices for using screening technologies to identify concealed explosives, which aligns with current standard operating procedures. T&D also developed the “Electronics vs. Electrical Devices Job Aid”—covering how TSOs should handle the devices at checkpoints—which instructors circulated during classroom training and provided to TSOs at the screening checkpoints. TSA uses established models and processes for updating and evaluating TSO training, and these processes follow leading practices for training and evaluation development. TSA updates its trainings using a training development process that can be segmented into five broad, interrelated elements, and is typically referred to as the ADDIE model. The elements include (1) analysis, (2) design, (3) development, (4) implementation, and (5) evaluation. In our prior work, we have found that these five elements of the ADDIE model help to produce a strategic approach to federal agencies’ training and development efforts. See figure 1 for how T&D aligns its training development process with the ADDIE model. T&D’s guidance and our prior work on federal agency training development identify various leading practice attributes for developing training. Such attributes include that the training development process: (1) is formal and based on industry recognized standards; (2) provides the ability to update training based on changing conditions and, if necessary, quickly; (3) includes mechanisms to ensure programs provide training that addresses identified needs; (4) ensures measures of effectiveness are included in training program; (5) prevents duplication of effort and allows for consistent message; (6) allows for stakeholder feedback; (7) provides for continuous evaluation of effort; and (8) includes mechanisms to ensure training programs are evaluated. We found that T&D’s training development process incorporates all of the identified leading practice attributes, as shown in table 1. Two examples of TSA’s implementation of selected leading practice attributes are that T&D (1) has methods for updating training quickly, if needed, and (2) has mechanisms to ensure TSO training is evaluated. Specifically: Methods to quickly update training. In alignment with the leading practice that agencies should have a process to enable quick updates to training to respond to changing conditions, T&D has alternative processes to develop and deliver training to TSOs faster than the approximately 6 months its standard process takes to develop or revise training. For example, in 2018, T&D formalized a set of alternative processes to rapidly develop and deliver training to TSOs. One such alternative is for T&D to use its Rapid Response process, which allows for a response time to the field of 72 hours. Additional options include the Rapid Update/Revision or Rapid Development (Priority Training) processes to allow for a new training to be issued in approximately 30 days. T&D officials said that the rapid development processes are used when an issue, such as an emerging threat, requires a response in days or weeks. T&D’s guidance outlines situations when these processes are appropriate for use and provides checklists to help T&D personnel follow key steps. Mechanisms to help ensure evaluations of training effectiveness. T&D has mechanisms for ensuring it evaluates the effectiveness of its TSO training programs. In particular, T&D uses the Kirkpatrick model to evaluate its training and, according to its policy, all of its courses are to be evaluated at Level 1 of the model, which measures training participants’ reaction to, and satisfaction with, the training. T&D is also to plan course evaluations for each training during the curriculum development process, determine the formal review cycle, and include it in the curriculum development paperwork. According to its policy, T&D must complete a curriculum review at least once every 5 years, but may do so at shorter intervals. During the curriculum review, T&D examines the training to confirm the content is valid with respect to the applicable listing of tasks and competencies, current law, policy, procedures, and equipment. As a part of this process, T&D assesses participant evaluations to determine whether changes to TSO training are needed. As of October 2019, T&D’s efforts to evaluate new or updated TSO training made from May 2015 through June 2019 are in line with its policy. For example, T&D officials said they updated participant evaluations for TSO training they changed during this time period to address risks identified by covert testing and reports on emerging threats. These officials told us that they had not yet formally analyzed the results of the evaluations. This progress is in line with T&D policy, which requires a review of each training every 5 years. We verified this by obtaining evaluations T&D collected for the six selected sample courses we reviewed. T&D provided us level 1 survey responses it had collected that measure training participants’ reaction to, and satisfaction with, the training programs for four of the courses. T&D implemented the four courses from calendar years 2015 to 2019. Based on those dates and T&D policy, T&D should complete curriculum reviews for the courses between 2020 and 2024. TSA relies on a database that both field and headquarters staff use to monitor TSO training compliance. According to TSA policy, TSA documents and maintains the training status of all TSOs across approximately 440 commercial airports through its Online Learning Center database. Within the database, TSA records training completion in three ways: 1. TSOs self-certify they completed the training activity, such as reading 2. A training staff member at a commercial airport will record training completion on behalf of a TSO for instructor-led courses and on-the- job training; 3. The database automatically records completion for training actions, such as online training. After recording training completion, the database calculates the percentage of TSOs at a given airport who are on pace for completing their required annual training. According to TSA guidance, the agency has set its annual TSO target compliance rate at 90 percent per commercial airport. While TSA has guidance outlining roles and responsibilities for training oversight at a high level, TSA headquarters and field officials told us their processes for monitoring training compliance—including analyzing training compliance data, reporting their results, and taking action to address the results—were not documented. Below are descriptions of these roles and responsibilities at the field and headquarters levels, based on what officials from each level told us. TSA personnel in the field have various responsibilities for overseeing training compliance: FSDs. FSDs, who oversee operations at one or more airports, have the primary responsibility for ensuring that TSOs within the airports they oversee have fulfilled their training requirements. FSDs are assessed on training compliance among TSOs at their respective airports during their annual performance reviews. All seven FSDs we interviewed said they use the Online Learning Center database to verify that TSOs are on track for meeting their training requirements. Further, these FSDs said they meet regularly with their on-site training staff to discuss how training is going and whether TSOs are at risk of not meeting their training requirements. Executive Directors. Executive Directors oversee the FSDs who work within their respective portfolios and discuss training compliance with the FSDs during their annual performance review. To monitor FSDs’ efforts, Executive Directors also review data from TSA’s Online Learning Center database on TSO training compliance for airports within their area of responsibility. According to an Executive Director we spoke with, if an Executive Director notices that TSO training compliance rates for an airport whose FSD they oversee are lower than the 90 percent compliance target, he or she may reach out to the FSD to obtain information on the causes and discuss an action plan to improve training compliance. TSA personnel at headquarters also have various responsibilities for overseeing training compliance: T&D. T&D officials said that on a monthly basis they analyze TSO training compliance data from TSA’s Online Learning Center database to identify how TSOs nationwide are meeting requirements and whether there may be trends that indicate a need for changes to training during the fiscal year. For example, officials told us that in fiscal year 2019 they noticed that airports were generally behind in meeting annual training requirements and determined this was due to the effects of the federal government shutdown. In response, they stated they adjusted the duration of some training courses to shorten the amount of time it would take TSOs to complete the training within the remainder of the fiscal year. Security Operations. Security Operations tracks individual airport progress toward meeting TSA’s annual 90 percent compliance target. Security Operations officials said they receive and review monthly training compliance reports from T&D. They are responsible for analyzing the data to monitor whether airports are on pace toward meeting the annual TSO training compliance target. For example, TSA has set the required training completion pace goal at 8.3 percent per month for each commercial airport—-so that by maintaining the pace, by the end of the fiscal year, TSOs at each airport will have completed their required annual training. Officials told us that if they identify instances where an airport’s overall TSO training compliance rate for a given month is below this goal during the course of a fiscal year, they will reach out to the FSD responsible. They will provide the FSD a point of contact at a comparable airport with higher compliance rates to share best practices for addressing the issue. While TSA headquarters officials from Security Operations and T&D are responsible for analyzing and addressing TSO training compliance, they focus on monthly airport progress toward the 90 percent TSO training target, rather than annual changes in compliance rates. In particular, they do not look back at prior year airport compliance data to assess whether airports did not meet the compliance target across fiscal years, and whether they require corrective action at the headquarters level. However, we reviewed annual TSO training compliance data across fiscal years for each of the 435 commercial airports that reported data from fiscal years 2016 through 2018. We found that while all airports met TSA’s 90 percent training compliance target in fiscal years 2016 and 2017, the compliance rates for five airports dropped well below 90 percent in 2018. These five airports’ TSO compliance rates dropped 15 to 26 percentage points from their reported compliance rate in 2017. T&D and Security Operations headquarters officials said they were not aware that five airports had not met TSA’s TSO training compliance target in fiscal year 2018, nor the causes for it. Headquarters officials said that they did not identify this development because their focus is on monthly nationwide trends, rather than instances of noncompliance at individual airports across fiscal years, which field officials would be responsible for addressing. However, unlike headquarters officials, field officials do not have the visibility to identify if or when such noncompliance may be occurring across other commercial airports; and whether it may indicate a broader issue. For example, the five airports whose TSO compliance rates dropped significantly between fiscal years 2017 and 2018 varied by size and location. As a result, FSDs and Executive Directors would generally not have been aware that other airports experienced noncompliance or been in a position to determine whether the noncompliance was due to related reasons. Based on TSA’s process, TSA headquarters officials from T&D and Security Operations are best positioned to identify training compliance trends and their causes when they occur, as they have visibility into training compliance data across the agency in a way that field officials do not. Headquarters officials from T&D and Security Operations told us the field- level processes for overseeing training compliance are not documented because TSA has intentionally given field officials the flexibility to manage TSO workload and training to meet the individual needs of their airports. They said they did not document their processes at the headquarters level because they already understood what to do and were not required to document the analysis results. However, the headquarters officials said there may be a benefit to documenting the headquarters process to ensure consistency in how they carry out the process in the event of attrition. Standards for Internal Control in the Federal Government calls for agencies to develop and maintain documentation of their internal control system. This documentation allows management to retain organizational knowledge and communicate that knowledge to external parties. This documentation of controls is also evidence that controls are identified, can be communicated to those responsible for their performance, and can be monitored and evaluated by the entity. Moreover, internal control standards state that internal control monitoring should generally be designed to ensure that ongoing monitoring occurs in the course of normal operations to ensure that known weaknesses are resolved. By documenting its headquarters process for monitoring TSO training compliance—including its process for analyzing monthly training compliance data, the results of its analyses, and actions taken in response—TSA could better ensure its headquarters staff are aware of their responsibilities for overseeing TSO training compliance and consistently carry these responsibilities out as staff change over time. Additionally, by monitoring for instances of TSO noncompliance at individual airports across fiscal years in its analysis of training compliance data, TSA headquarters would be better positioned to determine whether they constitute a trend warranting corrective action at the headquarters level. TSOs’ ability to perform their duties effectively in screening passengers and their belongings is crucial to the security of the nation’s aviation system. While TSA has made updates to its TSO training programs to address risks identified in covert testing, additional actions could improve its processes for monitoring TSO training compliance so that the agency can identify and address any potential training issues. In particular, by documenting its process for monitoring TSO training compliance— including those for analyzing monthly training compliance data, reporting the results of its monitoring efforts, and taking action to address potential issues—TSA could help ensure that all of the various officials responsible for monitoring training compliance, including new staff over time, understand the process and can consistently implement it. Further, by monitoring for instances of airport TSO non-compliance across fiscal years in its analysis of training compliance data, TSA would be better positioned to ensure that it is aware of potential trends so it may determine whether corrective action at the headquarters level is warranted. We are making the following two recommendations to TSA: The TSA Administrator should direct T&D and Security Operations to document their processes for monitoring TSO training compliance— including those for analyzing training compliance data, reporting the results from their analysis, and actions taken to address the results. (Recommendation 1) The TSA Administrator should direct T&D and Security Operations to monitor for instances of TSO non-compliance by individual commercial airports across fiscal years that could potentially warrant corrective action at the headquarters level. (Recommendation 2) We provided a draft of our report to DHS for review and comment. In its comments, reproduced in appendix I, DHS concurred with both of our recommendations. DHS also provided technical comments, which we incorporated as appropriate. With respect to our first recommendation that TSA document its process for monitoring TSO training compliance, DHS stated that, among other things, Security Operations will collaborate with T&D to develop and maintain an internal control mechanism that will document responsibilities at the field and headquarters level for monitoring TSO training completion compliance, and actions taken to address the results. With respect to our second recommendation that TSA monitor for instances of TSO noncompliance by individual commercial airports across fiscal years, DHS stated that T&D and Security Operations will begin monitoring trends in non-compliance at individual airports and for specific courses. Further, T&D has developed an internal website to share its findings with Security Operations through monthly compliance reports. We are sending this report to the appropriate congressional committees and to the acting Secretary of Homeland Security. In addition, this report is available at no charge on the GAO website at http://gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Jason Berman (Assistant Director), Julia Vieweg (Analyst-in-Charge), Benjamin Crossley, Elizabeth Dretsch, Michael Dworman, Eric Hauswirth, Susan Hsu, Tom Lombardi, and Heidi Nielson made key contributions to this report.", "summary": "TSA is responsible for screening millions of airline passengers and their baggage each day at the nation's commercial airports for items that could threaten aircraft and passengers. In carrying out its mission, TSA requires its screener workforce to complete various trainings on screening procedures and technologies. TSA updated its security screening procedures and technologies in recent years to address risks identified through covert tests in 2015 and reports of emerging threats. The TSA Modernization Act of 2018 included a provision for GAO to examine the effectiveness of TSA's updated screener training. This report addresses: (1) changes TSA made to screener training since 2015; (2) how TSA updates and evaluates screener training; and (3) how TSA ensures screener compliance with training requirements. GAO analyzed TSA documentation on training development, compliance monitoring, and a non-generalizable sample of six recently updated training courses—selected to reflect a range of training types and topics. GAO also reviewed TSA data on airport screener training compliance rates from fiscal years 2016 through 2018, and interviewed TSA officials. Since 2015, the Department of Homeland Security's (DHS) Transportation Security Administration (TSA) developed and updated screener training to address potential risks to commercial airports identified through covert testing and reports on emerging threats. From May 2015 through June 2019, TSA identified 62 potential risks that warranted review for a potential change in training. TSA made training changes in response to 56 of the identified risks—affecting 40 different training courses. TSA also responded to risks by developing or updating job aids or briefings for screeners. TSA uses established models for developing, updating, and evaluating its screener training. The figure below shows TSA's process for updating and evaluating its screener training, in accordance with a training development model that is widely accepted and used across the federal government. TSA relies on an online database to monitor screener compliance in completing required training at the nation's commercial airports. However, TSA has not documented its process for monitoring screener training compliance, including for analyzing compliance data and reporting and addressing instances of noncompliance at airports. Moreover, while TSA monitors airport compliance rates in a given year, it does not analyze the data across fiscal years for potential trends in noncompliance by individual airports that may warrant corrective action at the headquarters level. GAO found that in fiscal years 2016 and 2017, screeners at 435 commercial airports met TSA's 90 percent target compliance rate, while in 2018, five airports had compliance rates well below this target, dropping 15 to 26 percentage points from the prior year. TSA officials stated they were unaware of this development. By documenting its screener training compliance monitoring process and monitoring screener training compliance data across fiscal years, TSA would be better positioned to ensure it is aware of potential noncompliance trends warranting corrective action at the headquarters level. GAO is making two recommendations, including that TSA (1) document its process for monitoring screener training compliance and (2) monitor screener compliance data across fiscal years. DHS concurred with the recommendations.", "document_type": "gao"}
{"report": "Federal agencies have varying roles in planning, approving, and implementing infrastructure projects, depending on their missions and authorities. Some federal agencies help fund or construct infrastructure projects, and others grant permits or licenses for activities on private or federal lands. Agencies that manage federal lands, such as the Bureau of Land Management, may construct infrastructure on lands they manage and must also approve projects on those lands. The circumstances under which federal agencies may need to consult with tribes will vary based on the agencies’ responsibilities for infrastructure projects as well as an infrastructure project’s potential effects on tribes’ land, treaty rights, or other resources or interests. Federal agencies are generally responsible for identifying relevant tribes that may be affected by proposed projects, notifying the tribes about the opportunity to consult, and then initiating consultation, as needed. One or more tribes located near or far from the proposed project site may have treaty rights within lands ceded in treaties or interests in lands with cultural or religious significance outside of lands ceded in treaties. Additionally, the Federal Permitting Improvement Steering Council— which was created to make the process for federal approval for certain (large) infrastructure projects more efficient—has issued two annual reports that identified best practices for, among other things, consulting with tribes. These best practices include: training staff on trust and treaty rights; providing clear information on proposals in a consistent and timely manner; holding consultations on lands convenient to tribes when possible; compensating tribes for consultant-like advice; and working to build strong, ongoing dialogue between tribal authorities and agency decision makers, among others. In 2017, Executive Order 13807 directed agencies to implement the techniques and strategies identified by the steering council as best practices, as appropriate. For purposes of this testimony, Native American cultural resources means Native American cultural items as defined by NAGPRA, archaeological resources that are remains of past activities by Native Americans, and historic properties to which Indian tribes attach cultural or religious significance. ARPA, NAGPRA, and section 106 of the NHPA are examples of federal laws that apply to Native American cultural resources. These laws and their implementing regulations contain many different provisions applicable to Native American cultural resources, including requirements for federal agencies to consult with Indian tribes in certain circumstances. ARPA and NAGPRA, among other things, prohibit trafficking of certain archaeological resources and Native American cultural items, respectively. In August 2018, we reported on federal laws that address the export, theft, and trafficking of Native American cultural items and any challenges in proving violations of these laws. That report included a discussion of ARPA and NAGPRA. In addition, we reported in August 2018 that ARPA and NAGPRA contain provisions prohibiting the removal of archaeological resources and Native American cultural items from certain lands unless certain conditions are met, including consultation with Indian tribes. Specifically, ARPA prohibits, among other things, the excavation or removal of archaeological resources from public or Indian lands without a permit from the federal agency with management authority over the land. If the federal agency determines that issuance of such a permit may result in harm to, or destruction of, any religious or cultural site, the agency must notify any Indian tribe which may consider the site as having religious or cultural importance and meet, upon request, with tribal officials to discuss their interests. NAGPRA prohibits the intentional removal from, or excavation of, Native American cultural items from federal or tribal lands unless an ARPA permit has been issued and other requirements are met. Specifically, regulations implementing NAGPRA require federal agency officials to take reasonable steps to determine whether a planned activity on federal lands may result in the excavation of human remains or other cultural items. Officials are also required to consult with certain tribes, including any tribe on whose aboriginal lands the planned activity will occur, about the planned activity. After consultation, the federal agency official must complete and follow a written plan of action that includes, among other things, the planned treatment, care, and disposition of human remains and other cultural items recovered. NAGPRA and its implementing regulations also include provisions regarding inadvertent discovery of Native American cultural items on federal and tribal lands. Specifically, the person making the discovery must notify the responsible federal agency or tribal official, stop any activity occurring in the area of the discovery, and make a reasonable effort to protect the human remains or other cultural item discovered. The NAGPRA regulations specify procedures for the agency and tribal officials to take after receiving a notification and when the activity that resulted in the inadvertent discovery can resume. In March 2019, we reported that under section 106 of the NHPA and its implementing regulations, federal agencies are required to consult with Indian tribes when agency “undertakings” may affect historic properties— including those to which tribes attach religious or cultural significance— prior to the approval of the expenditure of federal funds or issuance of any licenses. The implementing regulations require agencies to consult with Indian tribes for undertakings that occur on or affect historic properties on tribal lands or may affect historic properties to which Indian tribes attach religious or cultural significance, regardless of where the historic properties are located. In addition, these regulations establish the following four-step review process for federal agencies, with tribal consultation required for each step: (1) initiating the section 106 process, (2) identifying historic properties, (3) assessing adverse effects, and (4) resolving adverse effects. As we found in March 2019, tribes and selected federal agencies identified a number of factors that hinder effective consultation on infrastructure projects, based on our review of the comments submitted by 100 tribes to federal agencies in 2016 on tribal consultation and our interviews with officials from 57 tribes and 21 federal agencies. Tribes identified a variety of factors that hinder effective consultation. For the purposes of this testimony, we are highlighting those factors that more than 60 percent of the 100 tribes identified as concerns. For example: Agencies’ timing of consultation. Sixty-seven percent of tribes that provided comments to federal agencies in 2016 identified concerns with agencies initiating consultation late in project development stages; according to one tribal official we interviewed, late initiation of consultation limits opportunities for tribes to identify tribal resources near proposed project sites and influence project design. Agency consideration of tribal input. Agencies often do not adequately consider the tribal input they collect during tribal consultation when making decisions about proposed infrastructure projects, according to 62 percent of tribes that provided comments to federal agencies in 2016. Tribes’ comments included perceptions that agencies consult to “check a box” for procedural requirements rather than to inform agency decisions. Agency respect for tribal sovereignty or the government-to- government relationship. Other concerns were related to agencies’ level of respect for (1) tribal sovereignty or (2) the government-to- government relationship between the United States and federally recognized tribes, according to 73 percent of tribes that provided comments to federal agencies in 2016. Comments included concerns that some agency practices are inconsistent with this relationship. For example, tribes cited agencies limiting consultation to tribal participation in general public meetings and sending staff without decision-making authority to represent the U.S. government in consultation meetings. Agency accountability. Sixty-one percent of tribes that provided comments to federal agencies in 2016 raised concerns related to the extent of agencies’ accountability for tribal consultation, stating that some agencies or officials are not held accountable for consulting ineffectively or for not consulting with relevant tribes. For example, comments included concerns that tribes may not have appeal options short of litigation when they believe that federal officials did not adhere to consultation requirements. In addition, officials from 21 federal agencies included in our March 2019 report identified factors that they had experienced that limit effective consultation for infrastructure projects. For the purposes of this testimony, we are highlighting those factors that more than 60 percent of the 21 agencies identified as concerns. For example: Maintaining tribal contact information. Officials from 14 of 21 agencies (67 percent) cited difficulties obtaining and maintaining accurate contact information for tribes, which is needed to notify tribes of consultation opportunities. For example, ongoing changes or turnover in tribal leadership make it difficult to maintain updated tribal information, according to some agency officials we interviewed. Agency resources to support consultation. Officials from 13 of 21 agencies (62 percent) cited constraints on agency staff, financial resources, or both to support consultation. Officials from these agencies said that they have limited funding to support consultation activities, such as funding for their staff to travel to in-person consultation meetings for infrastructure projects. Agency workload. Officials from 13 of 21 agencies (62 percent) identified a demanding workload for consultation as a constraint, because of large numbers of tribes involved in consultation for a single project, high volumes of consultations, or lengthy consultations, among other reasons. Officials from some of these agencies said that it may be difficult to stay on project schedules when there are multiple tribes to consult with or multiple agencies involved. In March 2019, we also found that the 21 agencies in our review had taken some steps to facilitate tribal consultation, but the extent to which these steps had been taken varied by agency. For example: Developing information systems to help contact affected tribes. Eighteen agencies developed systems to help notify tribes of consultation opportunities, which generally include contact information for tribal leaders or other tribal officials. Three of these agencies also included information on tribes’ geographic areas of interest. For example, the Department of Housing and Urban Development developed a system that aims to identify over 500 tribes’ geographic areas of interest and includes their contact information. The Federal Permitting Improvement Steering Council identified developing a central federal database for tribal points of contact as a best practice. We recommended that the council should develop a plan to implement such a database and consider how it will involve tribes to help maintain the information, among other actions. Developing policies to communicate how they considered tribal input. Five agencies’ tribal consultation policies specify that agencies are to communicate with tribes on how tribal input was considered. For example, the Environmental Protection Agency’s policy directs the most senior agency official involved in a consultation to send a formal, written communication to the tribe to explain how the agency considered tribal input in its final decision. However, 16 agencies did not call for such communications in their policies. We recommended that these agencies update their tribal consultation policies to better communicate how tribal input was considered in agency decision- making. Addressing capacity gaps through training. Most of the 21 selected federal agencies have taken steps to facilitate tribal consultation for infrastructure projects by providing a range of training opportunities for staff involved in tribal consultation to help build agency officials’ knowledge of tribal consultation topics. For example, the U.S. Army Corps of Engineers coordinates an immersive, 4-day training, hosted by a tribe on the tribe’s land or reservation for agency staff and other participating agency officials, which focuses on cultural competency important for tribal consultation. Utilizing various approaches to address resource constraints. Some of the selected federal agencies used various approaches to help address resource constraints agencies and tribes may face when consulting on infrastructure projects, according to agency officials. For example, the Bureau of Land Management’s policies state that the agency may use its appropriated funds and designated accounts to reimburse tribal members’ travel expenses to attend meetings in connection with some consultations. The Nuclear Regulatory Commission collects fees from project applicants to cover agency costs related to consultation. In conclusion, effective consultation is a key tenet of the government-to- government relationship the United States has with Indian tribes, which is based on tribal sovereignty. Failure to consult, or to consult effectively, sows mistrust; risks exposing the United States to costly litigation; and may result in irrevocable damage to Native American cultural resources. In our March 2019 report, we made recommendations to 17 agencies to take steps to improve their tribal consultation practices, which agencies generally agreed with and in one case, have implemented. However, sustained congressional attention to these issues and the relevant factors impacting the effectiveness of agencies’ consultation efforts may help to minimize the negative impacts on tribes’ cultural resources, when relevant federal laws and regulations apply. Chairman Gallego, Ranking Member Cook, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. For further information regarding this testimony, please contact Anna Maria Ortiz at (202) 512-3841 or ortiza@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement include Lisa Van Arsdale (Assistant Director), Brad Dobbins, Leslie Kaas Pollock, and Jeanette Soares. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Federal agencies are required in certain circumstances to consult with tribes on infrastructure projects and other activities—such as permitting natural gas pipelines—that may affect tribal natural and cultural resources. According to the National Congress of American Indians, federal consultation with tribes can help to minimize potential negative impacts of federal activities on tribes' cultural resources. The Secretary of Homeland Security has waived federal cultural resource laws that generally require federal agencies to consult with federally recognized tribes to ensure expeditious construction of barriers along the southern U.S. border. This testimony discusses examples of (1) federal laws and regulations that apply to Native American cultural resources and (2) factors that impact the effectiveness of federal agencies' tribal consultation efforts. It is based on reports GAO issued from July 2018 through November 2019 related to federal laws that apply to Native American cultural resources, tribal consultation for infrastructure projects, and border security. It also includes additional information about the consultation requirements in these cultural resource laws and regulations. Examples of federal laws and regulations that apply to Native American cultural resources include: The Native American Graves Protection and Repatriation Act (NAGPRA). In August 2018, GAO reported that NAGPRA prohibits the intentional removal from, or excavation of, Native American cultural items from federal or tribal lands unless a permit has been issued and other requirements are met. NAGPRA and its implementing regulations contain provisions to address both the intentional excavation and removal of Native American cultural items as well as their inadvertent discovery on federal and tribal lands. Section 106 of the National Historic Preservation Act (NHPA). In March 2019, GAO reported that section 106 of the NHPA and its implementing regulations require federal agencies to consult with Indian tribes when agency “undertakings” may affect historic properties—including those to which tribes attach religious or cultural significance—prior to the approval of the expenditure of federal funds or issuance of any licenses. In March 2019, GAO reported that tribes and selected federal agencies identified a number of factors that impact the effectiveness of consultation on infrastructure projects, based on GAO's review of the comments on consultation submitted by 100 tribes to federal agencies in 2016 and GAO's interviews with officials from 57 tribes and 21 federal agencies. Examples of these factors include: Agency consideration of tribal input . Sixty-two percent of the 100 tribes that provided comments to federal agencies in 2016 identified concerns that agencies often do not adequately consider the tribal input they collect during consultation when making decisions about proposed infrastructure projects. Maintaining tribal contact information . Officials from 67 percent of the 21 federal agencies in GAO's review cited difficulties obtaining and maintaining accurate contact information for tribes, which is needed to notify tribes of consultation opportunities. GAO also found that the 21 agencies in GAO's review had taken some steps to facilitate tribal consultation. For example: Eighteen agencies had developed systems to help notify tribes of consultation opportunities, including contact information for tribal leaders or other tribal officials. Five agencies' tribal consultation policies specify that agencies are to communicate with tribes on how tribal input was considered. GAO recommended in March 2019 that 17 federal agencies take steps to improve their tribal consultation practices. The agencies generally agreed and one agency has implemented the recommendation.", "document_type": "gao"}
{"report": "The military services preposition stocks ashore and afloat so that DOD is able to respond to multiple scenarios during the initial phases of an operation until the supply chain has been established. The military services maintain their own configurations and types of equipment and stocks to support their respective prepositioned stock programs: The Army stores sets of combat brigade equipment, supporting supplies, and other stocks at land sites in several countries and aboard ships. The Marine Corps stores equipment and supplies for its forces aboard ships stationed around the world and at land sites in Norway (see fig. 1). The Navy’s prepositioned stock program provides construction support, equipment for off-loading and transferring cargo from ships to shore, and expeditionary medical facilities to support the Marine Corps. The Air Force’s prepositioned stock programs include assets such as direct mission support equipment for fighter and strategic aircraft as well as base operating support equipment to provide force, infrastructure, and aircraft support during wartime and contingency operations. Prepositioned stocks are employed by the geographic combatant commanders, who have the authority to, among other things, organize and employ forces assigned to them as they deem necessary to accomplish assigned missions. DOD apportions the services’ prepositioned stocks among the geographic combatant commands according to joint guidance, and the afloat prepositioned stocks may be apportioned to more than one geographic combatant command. Requirements for prepositioned stocks are developed based on an approved operation plan. The services determine how best to meet the needs of the geographic combatant commanders, which may include the use of prepositioned stocks. Geographic combatant commanders periodically review their plans, assess the risk to those plans, and report the results to the Chairman of the Joint Chiefs of Staff. The approval of the Secretary of Defense is generally required to use the prepositioned stocks. DOD’s implementation plan for managing prepositioned stocks includes information that addresses three of the seven required elements enumerated in section 321 of the NDAA for Fiscal Year 2014. However, the plan, which is 5 pages in length, lacks the detail needed to fully address the remaining four required elements (see table 1). The Assistant Secretary of Defense for Logistics and Materiel Readiness approved the implementation plan on August 29, 2017, but an official from the Office of the Secretary of Defense told us that DOD did not formally issue the plan. As such, it does not bear a DOD seal, signature, or issuance number and most prepositioning service officials we spoke with were not aware of the plan’s existence. As shown in the table, DOD fully addressed three elements in section 321 of the NDAA for Fiscal Year 2014 by describing how the department will achieve its vision, desired end state, and goals, assigning roles and responsibilities, and including a schedule for the implementation of the plan. However, we assessed the remaining elements as partially addressed or not addressed because DOD did not provide the required information in its implementation plan. Specifically: Element two (comprehensive list of DOD’s prepositioned materiel and equipment programs, partially addressed). DOD’s implementation plan contains a list of the department’s prepositioned stock programs but that list omits one Army and eight Air Force prepositioned stock programs. In table 2, we compare the list of prepositioned stock programs that service officials provided to us with the list in DOD’s implementation plan. An official from the Office of the Secretary of Defense told us in April 2017 as part of a previous review that the department would not address this required element in the implementation plan because the department lists its prepositioned stock programs in its annual report to Congress. The implementation plan notes that DOD submits a comprehensive list of materiel to Congress each year per 10 U.S.C. §2229a. However, the annual report to Congress does not include a comprehensive list of the department’s prepositioned materiel and equipment programs. Rather, the annual report describes most of the department’s prepositioning programs but it omits one Army and six Air Force programs not listed in the implementation plan. Apart from the statutory requirement, Standards for Internal Control in the Federal Government state that management should communicate quality information externally so that external parties can help the entity achieve its goals and address risks. Without a comprehensive list of prepositioned materiel and equipment programs, DOD decision makers do not have all of the information they need to conduct effective oversight to assist the department in achieving its vision and goals. Element three (detailed description of how the plan will be implemented, partially addressed). The plan identifies policy, governance, and assessment initiatives through which the department aims to achieve its goals. However, the plan does not provide a detailed description of how the department will implement these three initiatives. Specifically, the plan states that DOD will identify policy gaps and revise or develop policy at all levels to better oversee prepositioned stocks; assigns the Under Secretary of Defense for Acquisition, Technology, and Logistics, the Chairman of the Joint Chiefs of Staff, and the services to review and revise the current prepositioning policies as appropriate; and tasks the geographic combatant commanders to ensure that theater campaign plans provide clear guidance for service prepositioned stock planning. However, the plan does not provide details on when geographic combatant commanders should finalize clear guidance for service prepositioned stock planning or describe what the guidance should include. The plan also states that DOD will use a governance body composed of the Under Secretary of Defense for Acquisition, Technology, and Logistics; the Chairman of the Joint Chiefs of Staff; the geographic combatant commanders; and the services to provide joint oversight of the prepositioned stock programs. However, the plan is unclear as to whether the Global Prepositioned Materiel Capabilities Working Group is the governance body. For example, the plan states that DOD’s joint oversight framework will include the Global Prepositioned Materiel Capabilities Working Group but also assigns the group to present capability shortfalls and gaps to a governance body and implement governance body decisions. Further, the plan states that DOD will use current systems of record and established metrics to evaluate performance and measure prepositioned stock status and capability. However, these are existing mechanisms to monitor the services’ programs and do not provide details on how the department will assess implementation of the plan itself. In 2017, a Joint Staff official told us that the implementation plan would be broad and high-level but would be more detailed than the DOD’s strategic policy. However, the plan’s descriptions of the implementation initiatives lack sufficient detail on what the department will do to implement the plan. Apart from the statutory requirement, Standards for Internal Control in the Federal Government establish that objectives should be defined in specific and measureable terms that clearly define what is to be achieved. Without sufficient detail, DOD risks being unable to fully support the emphasis and high priority that the 2018 National Defense Strategy gives to prepositioned stocks. Element six (description of the resources required to implement the plan, not addressed). DOD’s implementation plan does not describe the resources required to implement the plan. Rather, the plan states that prepositioning programs are resourced and managed by the services in support of combatant command operational and training requirements. In describing the joint oversight framework, the plan states that DOD will leverage the processes that already exist to resource prepositioning stock requirements including a focused effort on prepositioning as part of the annual planning, programming, budget and execution process, and the Joint Capabilities Integration Development System. Officials from the Office of the Under Secretary of Defense for Policy told us when they were developing the implementation plan that they understood this element as requiring information about the resources such as funding, personnel, and technology that would be needed to implement the plan. However, the plan does not include a description of the funding, personnel, or technology resources required to implement the plan. DOD officials reported that the services received $1.2 billion for prepositioned stocks in fiscal year 2018 and that the annual report to Congress also contains further information on the funding. However, this information does not describe the resources needed to implement DOD’s plan for prepositioned stocks as required by the NDAA for Fiscal Year 2014. Apart from the statutory requirement, Standards for Internal Control in the Federal Government establish that organizations should gather relevant operational and financial information for effective monitoring. Without a description of the resources required for implementation, decision makers do not have enough information to understand whether the department has sufficient resources to implement the plan. Element seven (description of how the plan will be reviewed and assessed to monitor progress, partially addressed). DOD’s implementation plan describes how the department will monitor the services’ prepositioned stock capabilities and readiness but does not describe how the department will review and assess the plan itself. The plan states that the department will use standard metrics contained in the readiness reporting systems of record to monitor prepositioning capability and readiness of the services’ programs. The plan assigns the services and combatant commands to assess prepositioned stock programs and posture annually and notes that all of the services are to begin reporting through the Defense Readiness Reporting System in the first quarter of fiscal year 2018. However, similar to element three, the plan does not fully address the mandated element in that it does not describe how the department will review or assess the plan as a tool toward achieving the stated vision and desired end state. The plan directs the Global Prepositioned Materiel Capabilities Working Group—which is responsible for providing oversight of prepositioned stock programs and resolving joint issues concerning prepositioned stocks—to assess actions to ensure desired results are achieved but does not describe how it is to do this. Apart from the statutory requirement, Standards for Internal Control in the Federal Government state that management should monitor its internal controls to determine their effectiveness and make modifications as necessary. Without reviewing and assessing the implementation plan, DOD will be unable to determine whether the current plan is helping the department progress toward its identified vision and desired end state for its prepositioned stock programs. DOD did not fully address the required elements in the implementation plan because, according to officials from the Office of the Secretary of Defense for Policy and the Joint Staff, implementation of the plan for managing prepositioned stock programs is the role of the services. According to these officials, DOD developed the implementation plan without details to allow the services to determine how to implement their respective prepositioning stock programs. Further, an official from the Under Secretary of Defense for Policy noted that DOD’s annual report to Congress on prepositioned stock programs contains some of the required information. However, as discussed earlier, we found that the annual report to Congress does not include all of the information to satisfy the required elements, such as a comprehensive list of the department’s prepositioned stock programs; and most service officials we spoke with were unaware of the plan. Moreover, section 321 of the NDAA for Fiscal Year 2014 required DOD to develop an implementation plan that contained all seven elements. Absent an implementation plan that fully addresses all of the elements required in the NDAA for Fiscal Year 2014 and aligns with internal control standards, DOD continues to provide incomplete information to Congress and stakeholders within the department on its prepositioned stock programs. In 2011, Congress began mandating DOD take steps to develop a joint strategy. Beginning in 2005 and subsequently in 2011, we reported that DOD lacked a joint oversight framework of the services’ programs. However, as shown in figure 2, DOD has made limited progress in addressing congressional requirements and our reporting recommendations related to joint oversight of prepositioned stock programs. DOD’s recent approach to joint oversight has been to update guidance and implement other related efforts. For example, over the past 2 years, the Office of the Secretary of Defense and the Joint Staff have updated existing documents and issued new policy documents, which each contain broad statements about the need for joint oversight of the services’ prepositioned stock programs: In December 2016, the Chairman of the Joint Chiefs of Staff updated its Logistics Planning Guidance for Prepositioned War Reserve Materiel. The document states that all service prepositioned stock programs require joint alignment with national priorities and global combatant command requirements across the full range of military operations. The instruction specifically directs the Joint Staff to develop a framework for joint oversight processes for synchronizing the services’ prepositioning strategies to minimize duplicative efforts and to maximize efficiencies and return on investment for prepositioned stocks. However, this document does not detail how the Joint Staff is to develop this framework and does not describe the elements that are to be included as a part of an effective approach for joint oversight. In March 2017, the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics issued its Pre-Positioned War Reserve Materiel Strategic Policy. One of the purposes of the document is to establish joint oversight of the military services’ pre- positioning efforts to maximize efficiencies across DOD. The directive assigns the Chairman of the Joint Chiefs of Staff with the responsibility to develop a framework for synchronizing the services’ prepositioning strategies to minimize duplicative efforts and to maximize efficiency and return on investment across DOD. However, similar to the instruction above, this document does not detail how the Joint Staff is to develop this framework or describe the elements that are to be included as a part of an effective approach for joint oversight. In August 2017, the Assistant Secretary of Defense for Logistics and Materiel Readiness finalized DOD’s implementation plan for managing prepositioned stock programs, which we discuss earlier in this report. The plan calls for improved DOD guidance that builds a framework and establishes joint oversight to synchronize service prepositioned stock programs with DOD’s strategic guidance and priorities. The plan also calls for balancing service prepositioned stock programs to maximize effectiveness and efficiency while minimizing potential duplication across the department. However, in addition to the shortcomings of the plan that we discuss earlier in this report, the plan also does not provide a detailed discussion of what is needed to implement a department-wide framework for joint oversight. Further, although the plan states that clear policy is the foundation for joint oversight, the plan itself was not issued as formal guidance, and, as noted earlier, most prepositioning service officials we spoke with were not aware of the plan’s existence. DOD officials stated that they are continuing to update existing guidance as needed and that the services are responsible for implementing and managing their own prepositioned stock programs. DOD also provides Congress annual reports on the status of the services’ prepositioned stock programs. However, in June 2015, we reported that the annual report provided inconsistent information among the services’ programs using a nonstandardized definition of “prepositioned stocks” and that the annual report is not an effective tool for joint oversight. We recommended that DOD develop a standardized definition of “prepositioning” for its annual report that is consistent with the definition used in the department’s joint service guidance and apply this definition consistently to identify prepositioning materiel and equipment across DOD. DOD concurred with our recommendations. However, as of October 2018, DOD continued to use varying definitions of prepositioned stocks. A broad definition exists at the strategic level, but service-level definitions vary depending on what each service’s prepositioned stock needs are. For example, the Army’s definition of prepositioned stocks is based on the equipment and stocks required to meet the unique mission requirements of brigade combat team configurations. Within this definition, the Army includes equipment sets used for training units, but the other services do not. DOD officials stated that although there is a broad definition of prepositioned stocks, the services are responsible for managing their individual programs to include what equipment and stocks are a part of their respective programs based on their mission and needs. Further, in 2008, DOD directed the establishment of the Global Prepositioned Materiel Capabilities Working Group and assigned it responsibility for addressing joint issues concerning war reserve materiel requirements and positioning. According to DOD’s prepositioned stock implementation plan, the working group is DOD’s focused joint oversight framework effort to execute the following for prepositioned stock programs: analyze service and combatant commander input in the annual report identify potential opportunities to enhance efficiency and reduce operational risk, present capability shortfalls/gaps to a governance body for implement governance body decisions in coordination with the services and combatant commands, and assess actions to ensure desired results are achieved. According to DOD guidance, the Assistant Secretary of Defense for Sustainment and the Chairman of the Joint Chiefs of Staff appoint co- chairs for the working group, which will include members from the military services, the Defense Logistics Agency and the combatant commands and meet annually or more often, as needed. However, since 2011, our work has shown that DOD has been unable to ensure that the working group’s activities include the full range of the tasks the group was established to perform because the working group lacks clear oversight and reporting relationships to authoritative bodies within DOD. We recommended that DOD assess the continued relevance of the Global Prepositioned Materiel Capabilities Working Group’s assigned tasks, and DOD concurred. In September 2012, we reported that, according to DOD officials, the main responsibility of the working group had been to consolidate the services’ individual submissions on their prepositioned stock programs into DOD’s annual report for Congress, and that the working group had met only sporadically and had not yet addressed many of the duties specified in its charter. This continues to be the case. We found that, according to DOD officials, quarterly working group meetings were frequently postponed, attendance was not fully representative of all stakeholders, and the discussions during a September 2018 meeting we observed were primarily focused on gathering information from the services for preparations for the upcoming annual report to Congress and receiving service updates on the current status of their respective prepositioned stock programs. DOD has not fully implemented joint oversight of the services’ prepositioned stock programs because the department’s guidance lacks detail and the department has not fully implemented requirements within other intended joint oversight efforts, such as the working group. Instead, DOD’s approach has been for the services to manage their own respective programs with limited oversight at the department level. Standards for Internal Control in the Federal Government state that objectives should be defined in specific and measureable terms that clearly define what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement. These standards also state that management should evaluate performance and hold individuals accountable for their internal control responsibilities. In addition, the NDAA for Fiscal Year 2014 mandates a framework for joint departmental oversight that reviews and synchronizes the military services’ prepositioned stock strategies to minimize potentially duplicative efforts and maximize efficiencies in prepositioned stocks across the DOD. Further, our prior work in the area of fragmentation, overlap, and duplication in the federal government has found that Congress and executive branch agencies have opportunities to contribute toward fiscal sustainability and act as stewards of federal resources. These include taking actions to reduce, eliminate, or better manage duplication, overlap, or fragmentation among federal agencies and programs; achieve cost savings; or enhance revenues. “Fragmentation” refers to those circumstances in which more than one organization within an agency is involved in the same broad area of national need and opportunities exist to improve service delivery. Without strengthening joint oversight across the department, DOD continues to have a fragmented approach to its management of prepositioning programs, which has led to inefficiencies. For example, according to Joint Staff officials, there is no uniform process by which the services are reporting the readiness of prepositioned stock assets. Joint Staff officials also said that having a joint oversight approach would help them have a more complete picture on the readiness of prepositioned stocks across the services and help the services in developing more consistent reporting methods. Service officials we interviewed have also noted that there may be duplication among DOD’s prepositioned stock programs resulting from limited joint oversight. For example, Navy officials stated that because each service utilizes medical assets as a part of its prepositioned stock programs, there is potential duplicative medical equipment across the services, which may result in inefficiencies. Finally, our ongoing classified work is finding a lack of joint oversight related to DOD’s management of prepositioned stocks in Europe. Although DOD’s current approach relies on the services managing their own prepositioned stock programs and Title 10 requires the services to train, man, and equip their forces, without fully implementing joint oversight—including providing more detailed information on how to implement such an approach in its guidance and reviewing its other efforts, such as the working group—DOD will continue to experience fragmented management of its prepositioned stock programs. Further, given the lack of progress DOD has made in the past several years, providing information to Congress on its efforts in this area could help hold the department to greater accountability. Prepositioned stocks play a pivotal role during the initial phases of an operation. We have reported for over a decade on the importance of DOD having a department-wide strategic policy and joint oversight of the services’ prepositioned stock programs, and Congress has required that DOD take action in this area. DOD issued guidance to include an implementation plan for managing prepositioned stock programs. However, the plan does not address all of the required elements enumerated in section 321 of the National Defense Authorization Act for Fiscal Year 2014, and DOD’s various guidance documents include broad direction for joint oversight. Without revising the implementation plan to have more complete information—including a full list of programs, a detailed description of how DOD will implement key initiatives, a description of the resources required, and an approach for monitoring and assessing the plan itself—the services will continue to operate their prepositioned stock programs with limited direction from DOD. Further, without fully implementing joint oversight, including providing more details in guidance and reviewing related efforts, and providing accountability to Congress on how the department will implement such oversight, DOD’s current fragmented management approach will continue to exist, which creates the potential for duplication and inefficiencies among the services’ prepositioned stock programs. We are making the following six recommendations to DOD: The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in coordination with the Chairman of the Joint Chiefs of Staff, issue a more detailed implementation plan or include implementation plan details in identified formal department-wide guidance to include an updated list to provide quality information, including all of DOD’s prepositioned materiel and equipment programs. (Recommendation 1) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in coordination with the Chairman of the Joint Chiefs of Staff, issue a more detailed implementation plan or include implementation plan details in identified formal department-wide guidance to include a detailed description of how DOD will implement the three key initiatives in the plan—policy, governance, and assessment—including clearly identifying what is to be achieved in these areas. (Recommendation 2) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in coordination with the Chairman of the Joint Chiefs of Staff, issue a more detailed implementation plan or include implementation plan details in identified formal department-wide guidance to include a description of the resources (i.e., relevant operational and financial information) required to implement the plan including dollar and personnel amounts. (Recommendation 3) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in coordination with the Chairman of the Joint Chiefs of Staff, issue a more detailed implementation plan or include implementation plan details in identified formal department-wide guidance to include a description of how the department will review and assess the implementation plan for effectiveness. (Recommendation 4) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in coordination with the Chairman of the Joint Chiefs of Staff, take steps to fully implement joint oversight of DOD’s prepositioned stock programs, including providing detailed information on how to implement such an oversight approach in department guidance and reviewing other joint oversight efforts, in order to synchronize the military services’ preposition stock strategies to avoid fragmentation. (Recommendation 5) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in coordination with the Chairman of the Joint Chiefs of Staff, update Congress on the department’s progress in joint oversight management in the prepositioned stock annual report or in a separate report. (Recommendation 6) We provided a draft of this report to DOD for review and comment. In its comments, reproduced in appendix II, DOD concurred with each of the six recommendations and described planned actions it will take to implement them. We are providing copies of this report to the appropriate congressional committees; the Secretary of Defense; the Assistant Secretary of Defense for Sustainment; and the Chairman of the Joint Chiefs of Staff. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In addition to the contact named above, individuals who made key contributions to this report include Alissa H. Czyz, Assistant Director; Vincent M. Buquicchio; Pamela Davidson; Mae Jones; Cody Knudsen; and Yong Song.", "summary": "The military services preposition stocks worth billions of dollars at strategic locations around the world to provide U.S. forces with critical assets before supply chains have been established. In the 2018 National Defense Strategy, DOD emphasized that prepositioned stocks provide key logistical support for the department's missions. For many years, GAO has identified the potential for duplication among the military services' prepositioned stock programs due to a fragmented management approach and limited joint oversight within DOD. In the NDAA for Fiscal Year 2014, Congress required DOD to develop an implementation plan to manage prepositioned stock programs. DOD finalized its plan in August 2017. The act included a provision for GAO to review the plan and report on related issues. GAO assessed the extent to which (1) DOD's implementation plan addresses mandated reporting elements and (2) DOD has made progress in implementing a joint oversight approach for managing the services' prepositioned stock programs. GAO compared the implementation plan and DOD's joint oversight approach with congressional requirements and federal standards for internal control and interviewed DOD officials. The Department of Defense's (DOD) implementation plan for managing the military services' prepositioned stock programs does not fully address four of the seven elements required by the National Defense Authorization Act (NDAA) for Fiscal Year 2014. For example, DOD's plan did not include all information required by the NDAA, such as a complete list of the services' programs, information on how DOD would pursue key initiatives, or the resources required to implement the plan. DOD officials told GAO that they developed a plan without detail to allow the services to determine for themselves how to implement their programs. However, absent an implementation plan that fully addresses NDAA requirements, DOD continues to provide incomplete information to Congress on the department's prepositioned stock programs. Since 2011 when Congress required DOD to take action and since 2005 when GAO first reported on the issue, DOD has not fully implemented a joint oversight approach for managing prepositioned stock programs (see figure). DOD's recent approach for implementing joint oversight has been to update guidance documents and develop other efforts, such as a working group, but the services continue to manage their programs with little joint oversight. Without taking steps to fully implement joint oversight, including providing detailed information on how to achieve this in guidance and reviewing other efforts, DOD's management will continue to be fragmented and it risks duplication and inefficiencies among the services' programs. Moreover, updating Congress on DOD's progress would help assure decision makers that DOD intends to follow their direction in establishing joint oversight of prepositioned stock programs. GAO is making six recommendations, including that DOD provide information required by the NDAA, fully implement joint oversight of prepositioned stock programs, and update Congress on progress made. DOD concurred with all of the recommendations.", "document_type": "gao"}
{"report": "In 2004, FEMA initiated the IPAWS program to integrate EAS and other public-alerting systems into a larger, more comprehensive public-alerting system. As shown in figure 1, IPAWS serves as a centralized gateway to deliver alerts to the public. After an alerting authority creates and sends an alert to IPAWS, the system then routes the alert to the public using one or more of the following pathways: Emergency Alert System. Allows authorized federal, state, territorial, tribal, and local government agencies to use EAS media platforms— including radio and television—to send alerts. IPAWS also allows the U.S. President to activate EAS to communicate to the public through all EAS media platforms during a national emergency. Wireless Emergency Alerts. Allows authorized federal, state, territorial, tribal, and local government agencies to send text-like messages to mobile devices in geographically selected areas as one-way cellular broadcasts. Various factors affect whether a WEA message will be received on a mobile device, such as whether the device is WEA-capable and within range of a cell tower where a participating wireless carrier provides WEA services to its customers. According to CTIA, a wireless industry association, more than 100 nationwide and regional wireless carriers participate and have the capability to provide WEA messages to 99 percent of American wireless subscribers. IPAWS alert feed for internet services. Allows internet companies authorized by FEMA—such as Google, Facebook, and The Weather Channel—to retrieve IPAWS alerts and distribute them to social media, websites, applications, and subscription services. Government agencies and industry organizations play different roles in providing, protecting, and leveraging the nation’s emergency alerting capability. FEMA. FEMA is responsible for operating, maintaining, and administering access to IPAWS, including managing the application process. As discussed earlier, public safety agencies that wish to use IPAWS must apply to FEMA to become approved alerting authorities. FEMA, in consultation and coordination with FCC, must carry out various actions to modernize and implement IPAWS. For example, FEMA must ensure IPAWS can send alerts to a specific geographic location and to multiple communications systems and technologies, educate government users of IPAWS and provide training opportunities to them, and conduct nationwide tests of IPAWS, among other things. Legislation was enacted that expands FEMA’s responsibilities for IPAWS. FCC. FCC creates the rules for EAS and WEA, the two primary alerting pathways authorities use to send public alerts through IPAWS. FCC establishes the technical standards, procedures, and testing protocols for EAS participants. FCC also manages an online system used to collect and analyze results of nationwide EAS tests. FCC establishes technical requirements participating wireless carriers must follow for delivering WEA messages to WEA-capable mobile devices. Federal alerting authorities. Authorized federal alerting authorities may create alerts and use IPAWS to send alerts to the public. For example, the National Weather Service (NWS), within NOAA, uses software NWS developed to issue WEAs for severe weather risks such as flash floods and tornadoes. USGS intends to send earthquake-related alerts through IPAWS but as of September 2019, had yet to send such an alert through IPAWS. USGS has partnered with Washington, Oregon, and California to test and implement a West Coast earthquake early warning system called “ShakeAlert” that is intended to send WEA messages to mobile devices several seconds after the initiation of an earthquake. State, territorial, tribal, and local alerting authorities. According to FEMA policy, state, territorial, tribal, and local government agencies first complete FEMA’s application process to gain access to IPAWS and obtain the proper authorization to issue alerts for specific geographic jurisdictions. As discussed earlier, government agencies that issue alerts through IPAWS can include emergency management or law enforcement agencies at the state, county, or city government level. Non-governmental organizations such as a local emergency management association may be granted an authority to issue alerts through IPAWS with approval from FEMA or an alerting authority. For information on FEMA’s IPAWS application process, see figure 2 below. Industry. Industry develops and owns the infrastructure that enables alerts to be created, authenticated, and delivered to the public. Alerting software companies provide software tools that allow alerting authorities to create and send alert messages via the internet to IPAWS. Alerting software companies also provide “opt-in” or subscription-based alerting services to public safety agencies that allow the public to sign up to receive alerts. EAS participants that transmit EAS messages include radio and television broadcasters, cable operators, wireline video service providers, satellite radio providers, and direct broadcast satellite providers. Wireless carriers operate wireless networks that allow alerting authorities to send one- way geographically targeted WEA messages to WEA-capable mobile devices. Manufacturers develop, test, and provide WEA-capable mobile devices, in coordination with participating wireless carriers, to consumers. Internet and web services companies may also distribute alert information from an IPAWS alert feed to internet applications, websites, or social media. We have previously reviewed FEMA’s progress in implementing IPAWS. In 2013, we found that FEMA had taken actions to improve the capabilities for IPAWS and to increase federal, state, and local capabilities to alert the public, but barriers remained to fully implement an integrated system. We made six recommendations, including that FEMA work with FCC to establish guidance for states to fully implement and test IPAWS components and implement a strategy for regular nationwide testing. The agencies implemented all of the report’s recommendations. Our analysis of FEMA data found 1,401 alerting authorities at the federal, state, local, territorial, and tribal levels had access to IPAWS to send alerts as of September 2019, a substantial increase from 2013 (soon after it became operational) when fewer than 100 authorities had access. According to FEMA officials, nearly 70 percent of the nation’s population is covered by a local alerting authority that can use IPAWS to send alerts. Further, according to FEMA documentation, from a state authority perspective, all 50 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands have at least one state-level authority that can use IPAWS to send alerts to any locality within that state or territory. Local authority access to IPAWS to send alerts varies, however, as FEMA officials stated that about two-thirds of the nation’s 3,000 counties do not have access to IPAWS to send alerts. Although access to IPAWS at the state level enables alerts to be sent, for example, to jurisdictions that may have lost their capability during an emergency, gaps in access to IPAWS for local officials could limit the timeliness of alerts as emergencies occur. For example, officials from an alerting authority told us that with the exception of alerts issued by NWS, all emergencies start locally. If a locality does not have access to issue an alert through IPAWS, information must be communicated from the locality to an authorized state official to issue the alert, which could result in delays in getting critical information to the public. Reasons for this gap at the local level could be related to a variety of factors. For example, some counties may still be in the process of applying for access. Other counties may not be able to gain access to IPAWS due to state or local laws, or a state’s EAS communications plan may specify that only certain types of agencies can issue alerts. For example, state EAS communications plans may authorize the governor of the state, an emergency management office, state law enforcement agency, or a non-governmental organization as the authorized agencies for sending alerts. In addition, an academic who specializes in rural emergency management told us that unfunded staff positions in emergency management are commonplace in rural areas and the areas may lack funding to apply for IPAWS access. Figure 3 highlights areas of the country that were covered by a local or tribal alerting authority as of September 2019. Alerting authorities at the state, territorial, and local levels have increasingly used WEA messages since 2012 (see fig. 4). In addition, these authorities used more WEA messages than EAS alerts each year, with a large difference occurring between 2017 to 2018, when WEA messages increased by 89 percent while EAS alerts increased by 35 percent. While usage of WEA and EAS by state, territorial, and local authorities has generally increased since 2012, our analysis of FEMA data found that this increase was driven by a small group of alerting authorities in certain parts of the country. Some locations may be more prone to experience certain types of emergencies, particularly weather related emergencies such as hurricanes. However, the potential exists in any location for an alert to be sent to the public if an alerting authority determines an imminent threat to public safety exists. Specifically, our analysis of WEA alert data from April 1, 2012 to October 1, 2019 found: A total of 236 of the 1,372 state, territorial, and local alerting authorities sent a WEA message. A total of 69 of the 1,372 state, territorial, and local alerting authorities accounted for nearly 80 percent of WEA messages sent at those levels. Most of the country has received a low number (fewer than 10) or no alerts sent by state, territorial, and local authorities, while limited parts of the country have received higher numbers of alerts (see fig. 5). At the federal level, our analysis of FEMA data found that NWS sends the vast majority of WEA messages sent through IPAWS, a number that from April 1, 2012 to October 1, 2019 totaled more than 46,000. The most common WEA messages sent by NWS were related to flash flooding (28,640), tornadoes (15,985), hurricanes (571), and dust storms (386). An academic we interviewed said it is important to note that local alerting authorities use the NWS warnings to issue alerts instructing the public to take specific protective actions, for example, to evacuate using certain roads. For more information on when a person may receive a WEA message on a WEA-capable mobile device through IPAWS, see appendix II. NWS uses multiple alerting mechanisms to send alert messages to people around the country. As one of its mechanisms, NWS uses the Integrated Public Alert and Warning System to send Wireless Emergency Alert messages to mobile devices in areas facing w eather risks, such as this geographically targeted message to a cell phone in Washington D.C. in July 2019. To test the capability and effectiveness of IPAWS, FEMA, FCC, NWS, and state and local public safety agencies have carried out nationwide and localized alert tests since 2016. Nationwide EAS Tests: FEMA, in coordination with FCC, conducted four annual nationwide EAS tests from 2016 to 2019. The tests assessed how well EAS alerts were received and retransmitted using the two ways an EAS alert can be delivered: (1) over the internet via IPAWS and (2) through the legacy “over the air” radio and television broadcast stations. According to FCC’s analysis, about 76 percent of an estimated 26,000 EAS participants took part in the 2018 test, with about 96 percent of participants reporting they received the test alert. While the vast majority of EAS participants reported no complications, FCC’s analysis identified some problems with the 2018 test, including EAS participants reporting audio quality issues (less than 2 percent), EAS equipment issues, out-of- date software, user error, and complications accessing IPAWS (less than 1 percent each). To help address such issues, FCC provided advisories in advance of the next nationwide EAS test. In addition, FEMA has publicly identified how FCC could improve future nationwide tests, including improving the accuracy of reporting and other audio and visual technical issues. FEMA officials said they are working with FCC to resolve technical issues found in recent tests. Nationwide WEA Tests: FEMA, in coordination with FCC, carried out the first nationwide WEA test in October 2018. FEMA sent a test alert through IPAWS to participating wireless carriers, which then transmitted the alert to their subscribers’ WEA-capable devices across the country. FEMA officials viewed the first nationwide WEA test as a success with regard to the technical execution of delivering a nationwide WEA message via IPAWS. However, officials acknowledged a main lesson from the test was a need to collect data on how effectively WEA messages are being received. While FCC collects EAS test data to assess how well the EAS test was received and retransmitted, a similar mechanism does not exist for the WEA pathway. According to wireless industry representatives we interviewed, the WEA system was designed to use a one-way broadcast cellular technology that prevents the wireless network from collecting data from mobile devices. Instead, FCC has used voluntary public responses, media reports, and informal surveys conducted by state and local public safety agencies to assess results. For example, FCC’s report on the 2018 WEA test found that media sources reported inconsistent WEA delivery in different parts of the country and that informal surveys conducted by state and local agencies showed variability in WEA delivery. FCC also reported that issues were found during the WEA test related to duplicate messages and audio and vibration cadence that could have affected individuals with disabilities. At the time of our review, FEMA officials said they were preparing to conduct the next nationwide WEA test in late 2020 and developing a survey to accompany the test to collect data on WEA message delivery. The District of Columbia Homeland Security and Emergency Management Agency used the Integrated Public Alert and Warning System to send a geographically targeted WEA test to mobile devices in Washington, D.C. in June 2019. owned cell phones received the test alert within a range of 6 seconds and several minutes. In May 2019, FCC rules initially went into effect that will allow alerting authorities to send WEA tests to the public without FCC approval—called State/Local WEA Test. Participating wireless carriers are required to provide the capability, but subscribers must manually opt-in to receive these alerts on their mobile devices. In November 2019, a major wireless carrier obtained a waiver from FCC to conduct two WEA tests under these rules to assess the carriers’ ability to perform enhanced geo- targeting for WEA messages. Officials from alerting authorities we contacted for seven case studies on the use of IPAWS cited benefits and limitations of using the system during recent disasters such as wildfires, a hurricane, a flood, an earthquake, a chemical fire, a power shortage, and a law enforcement event. Benefits. Officials from authorities we interviewed said that IPAWS has a wide reach because most people have mobile devices to receive WEA messages, and WEA can also reach visitors to their area. Compared with opt-in alerting systems that can have a low percentage of subscribers, officials from alerting authorities we interviewed said that IPAWS provides an opportunity to reach more people during emergencies. In addition, they said that states can act as back-ups for local authorities that have lost their alerting capabilities to help ensure that alerts can be sent. Our analysis found that state and local alerting authorities used IPAWS to send alerts regarding a variety of emergencies, examples of which are shown in table 1. Alerting authority officials also said they plan to use IPAWS in a variety of ways in the future, including for mudslides, rip currents, hazardous materials incidents, and law enforcement emergencies such as terrorism or active shootings. Limitations. Officials from alerting authorities we contacted cited three main limitations. First, they said it was difficult to write effective WEAs within the 90 character limit. For example, officials from an authority said that within the character limit it is difficult to explain the risk, who the alert is from, and what the public should do. As we discuss later, FCC has expanded the character limit. Second, officials expressed concerns about the ability to target WEAs to specific geographic areas, which caused some to lack confidence in the system or not use it at all. Third, officials from alerting authorities said that because WEA is a one-way communication system, they do not know if the alerts reached the intended public. For example, officials from one authority described sending an evacuation order but not knowing whether people in the intended area received it. In another example, while an alert was helpful in alerting the public about a suspicious package, officials from one authority said the alert was received 4 miles beyond its intended target, which led them to speculate about the number of people who received the alert. More information about the use of IPAWS during events we selected as case studies is provided in appendix III. FEMA has taken recent steps to modernize IPAWS by implementing various improvements and exploring new technologies. For example, FEMA is moving IPAWS to a cloud-based data center to enhance the system’s availability and is modernizing the stations that serve as the main broadcast source for national emergency alerts, according to FEMA’s 2018 performance report for IPAWS. In addition, officials described how FEMA has assisted with developing technical standards for new IPAWS capabilities and engaged the private sector to explore possibilities for integrating alerts into technologies such as digital billboards, Braille reader devices, and internet-connected devices in homes and vehicles. FCC has published rules that require participating carriers to implement new or improved capabilities for wireless alerts sent through IPAWS. Improved alert message content and capabilities. FCC required wireless carriers to support several capabilities to help alerting authorities communicate clearly and effectively, including the ability to send longer messages (expanding the limit from 90 to 360 characters) and the ability to send alerts in Spanish. Initially, FCC set a May 1, 2019, deadline for carriers to support all of these capabilities but later extended it to December 19, 2019, to allow time for carriers to complete testing with IPAWS. FEMA completed the necessary updates to support formal testing with the IPAWS gateway in mid- November 2019. Two academics we interviewed who have researched emergency alerting told us that alerts with expanded character length are more effective in prompting people to take protective actions, compared with shorter ones. Other new capabilities required include “alert prioritization,” meaning that alerts must be displayed as soon as they are received and a new “public safety message” category for advisories that prescribe one or more actions likely to save lives or safeguard property during an emergency (e.g., boil water notices, emergency shelter locations). As discussed earlier, a state/local WEA test option was also required to allow alerting authorities to send test messages to a subset of the public without prior approval from FCC. More precise geographic targeting. FCC required carriers to deliver alerts to areas that match the targeted geographic area, to within one- tenth of a mile, a capability that FCC calls enhanced geo-targeting. FCC initially required carriers to implement enhanced geo-targeting by November 30, 2019, but later extended it to December 19, 2019, to allow time for carriers to complete testing with IPAWS, as with the capabilities discussed above. FEMA completed the necessary updates to support formal testing with the IPAWS gateway in mid- November 2019. Previously carriers have been required to transmit alerts to the geographic area that best approximates the emergency area identified by the alerting authority. As FCC’s chairman has explained, these less precise geographic targeting capabilities can result in overbroad alerting, where people may receive the alert even though they are located well outside of the target area. Several local WEA tests in 2018 found overbroad alerting when targeting specific geographic locations. Officials from many alerting authorities we interviewed told us they are concerned about the inability to geographically target alerts with accuracy, which can make some reluctant to send WEA messages. According to several wireless and device industry representatives we interviewed and letters that wireless carriers have sent FCC, enhanced geo-targeting is a particularly challenging capability to implement because changes must also be made by different sectors of industry—such as manufacturers of cell phone handsets and chipsets. Some industry representatives also told us that only some, mostly newer model cell phones will be able to receive the more precise geo-targeted alerts and that many older devices currently in the population will not support this new capability because it requires a new chipset. Other recent improvements. FCC has also required implementation of new alert content and categories, such as: “Clickable” links—Embedded links in alerts so people receiving them can click on the link to see a photo of a suspect, for example. This capability has been implemented. Blue Alert—A new type of alert to notify the public of threats to law enforcement and to request help apprehending dangerous suspects. This capability has been implemented. 24-hour alert retrieval—Alerts must remain available on devices for 24 hours after receipt, or until the consumer chooses to delete them. FCC required carriers to implement this capability by November 30, 2019, and FEMA officials told us this capability did not require technical changes to the IPAWS gateway. Although FEMA and FCC are taking actions to improve alerting capabilities, developments in technology are changing the alerting landscape. Our analysis of agency documents and interviews with public- safety stakeholders indicated two emerging and unresolved areas. Multimedia. In 2018, an FCC advisory committee recommended that alerting systems should carry graphics and other multimedia. For example, four public-safety stakeholders told us it would be helpful to include multimedia (e.g., photos and maps) directly within WEA messages. Doing so would allow the public to see the information without clicking an embedded link. In 2015 and 2016, FCC sought comment on the technical feasibility of including multimedia and in 2018 issued another public notice on the topic to refresh the record. The proceeding remains open and FCC has not taken additional action. Internet streaming. The public may not receive broadcast EAS alerts when watching television that is streamed through an internet connection. A 2017 Pew Research Center survey found that 28 percent of American adults and 61 percent of adults age 18 to 29 said that streaming is their primary way of watching television. Representatives from two internet service providers told us they have developed solutions that enable customers to receive EAS alerts when the customers are using their applications to stream content. However, EAS alerts may not override other streaming services such as video and gaming because of technical limitations and the limited information that content service providers maintain about a user’s location, according to industry representatives. For example, representatives from an association representing internet companies told us that providing emergency alerts through internet streaming services presents technological challenges and that its members would have concerns about collecting locational information about their customers. The effect of potentially not receiving an EAS alert while streaming is unclear. While more Americans are streaming their television and multimedia, many use a second screen such as a cell phone while watching television and could receive any relevant alert as a WEA message. A 2018 Nielsen survey found that 45 percent of respondents very often or always use a second screen such as a smartphone while watching television. FCC has sought comment about this issue in general. FCC officials told us that extending EAS to new technologies for viewing video content raises legal and technical considerations and that they continue to evaluate the efficacy, costs, and benefits of doing so. Pursuant to statute, FCC is responsible for establishing technical standards and requirements for WEA, as discussed earlier. Further, FCC’s 2018–2022 strategic plan identified a performance goal to facilitate the effectiveness and reliability of EAS and WEA, and following a nationwide test in 2018 FCC’s Public Safety and Homeland Security Bureau recommended that additional measures be taken to improve the reliability and effectiveness of WEA. Developing goals and performance measures is consistent with leading practices for performance management. GPRA, as amended and expanded by GPRAMA, creates a framework for articulating goals and measures that can provide federal agencies with a clear direction for successful implementation of activities and improve the efficiency and accountability of agencies’ efforts. Goals explain the purpose and intended results that a program seeks to achieve in its work. Performance measures that are linked to goals allow a program to track the progress it is making toward achieving its goals. While GPRA and GPRAMA apply to the department or agency level, we have previously reported that their provisions can serve as leading practices at other organizational levels, such as component agencies, offices, programs, and projects. Additionally, federal internal control standards discuss the importance of goals, stating that management should define objectives clearly. This involves defining objectives in specific and measurable terms so that they can be easily understood and performance toward achieving those objectives can be assessed. Federal internal control standards also state that measurable objectives should be specific and stated in quantitative or qualitative form. FCC has required carriers to implement new WEA capabilities and taken steps to understand more about WEA performance, but FCC has not developed goals and performance measures to help monitor how well the new capabilities perform during emergencies. Instead, we found FCC has taken an ad-hoc approach to monitoring WEA performance. In particular, when we asked whether FCC planned to develop standards or benchmarks to measure WEA performance, FCC officials said they intend to use certain test results, as discussed below, to understand more about WEA performance. Partnered geo-targeting tests. FCC intends to partner with localities to test the accuracy of participating wireless providers’ enhanced geo- targeting capabilities starting in early 2020. Four localities have applied to participate as of November 2019, according to FCC officials. To perform each test, FCC and its partner at each given location intend to use online surveys to collect information on which individuals receive the test alert and under what circumstances. However, at the time of our review we found that while FCC has broadly identified the purpose of the tests as testing the accuracy of enhanced geo-targeting, it has not defined specific, measurable goals that are specific to this testing effort. For example, FCC has not stated what would be an appropriate success rate for enhanced geo- targeting accuracy. We also found that FCC has not connected its survey questions to specific performance measures that could be compared across test locations. According to FCC officials, FCC has not announced whether it will compare results across localities or use specific performance measures to assess geo-targeting performance. FCC officials said they have no plans to test other new WEA capabilities, including the expanded message length, and that at the time of our review it was too early to say how results from the partnered tests would be analyzed and shared more broadly with public-safety stakeholders. State and local tests. As discussed earlier, FCC officials told us that 39 alerting authorities at the state and local level received approval from FCC to conduct their own WEA tests as of November 2019. FCC officials also told us that that they encourage alerting authorities that seek approval for WEA tests to share performance data with FCC. According to FCC officials, FCC has received data from nine localities as of November 2019 and will use the test results internally to develop a broader understanding of WEA performance. When we asked what FCC has learned from the data, FCC officials said they have received some results but are still in the process of analyzing them. By developing goals and performance measures for its efforts to monitor the new WEA capabilities, FCC would have clearer direction for what it plans to achieve and more specific means to assess the performance of the capabilities. For example, performance measures related to FCC’s planned survey questions for geo-targeting could include the percentage of participants who received the alert and the percentage who received the alert within the target geographic area. Another performance measure for the new capabilities could include the extent to which messages of up to 360 characters are fully or partially displayed on a mobile device, or not at all, for example. Without specific goals and performance measures, FCC will have difficulty knowing if it is making progress toward its stated strategic goal of ensuring the effectiveness of WEA. The results of data collected on performance measures could provide assurance that new WEA capabilities are working as intended during emergencies, or could point to areas where performance is lacking and where FCC might need to take other actions such as working with industry to resolve issues, updating WEA requirements, or conducting additional analysis. Monitoring performance is all the more important because of uncertainty about the extent to which all cell phones will be able to receive WEA messages with the new capabilities. In addition, new capabilities have the potential to make WEA a more powerful tool and possibly further increase its use. Our analysis shows that WEA has outpaced the use of EAS as an alerting method, and according to the Pew Research Center, Americans are increasingly connected to digital devices, with 96 percent of American adults owning cell phones in 2019 and 81 percent owning smartphones. However, as discussed earlier, officials from many alerting authorities we interviewed had concerns with WEA performance. Many officials from alerting authorities told us that they were looking forward to the new capabilities— including enhanced geo-targeting and expanded message length—which will improve their ability to alert the public. Having specific performance information about the effectiveness of these capabilities could increase alerting authorities’ confidence in the system and help make these authorities more informed users of IPAWS. The IPAWS Modernization Act requires FEMA to instruct and educate federal, state, tribal, and local government officials in the use of IPAWS. FEMA has multiple efforts underway to educate and train alerting authorities about IPAWS. Training. Through FEMA’s Emergency Management Institute, FEMA offers training courses on IPAWS, including a mandatory course that IPAWS applicants must take before they can become authorized users of the system. FEMA is revising its training, according to FEMA officials, and they estimated that the new courses will be available midway through 2020. Online resources. On a regular basis, FEMA emails tips and conducts webinars, which are recorded and made available online. FEMA has developed a library of IPAWS resources, including a toolkit and fact sheets. FEMA also created an online collaboration group for IPAWS users to share information and best practices and plans to expand the capabilities of this group, according to FEMA. Testing environment. FEMA created a controlled testing environment called the IPAWS lab that alerting authorities can use to send test alerts and receive hands-on or remote assistance from FEMA staff. According to FEMA, demand for IPAWS lab support has increased, and FEMA hosted more than 200 sessions with IPAWS users in calendar year 2018. FEMA implemented a new requirement in October 2019 for all alerting authorities to send a monthly test alert using the IPAWS lab and upgraded the IPAWS lab environment to support the increased testing. In-person presentations. FEMA officials regularly present at public safety conferences and other events and use these opportunities to share information about IPAWS and encourage potential new users. FEMA has also assessed alerting authorities’ educational needs, but it has not fully addressed the recommendations it identified to support these needs or developed plans for ongoing assessments. In 2017 FEMA conducted an analysis—interviewing a sample of alerting authorities and assessing their responses to identify common challenges in using IPAWS. FEMA found that alerting authorities need more training and practice in using IPAWS and experience challenges with using their alerting software, among other things. Our interviews with selected alerting authorities and software providers revealed similar concerns, including that for some a lack of confidence is a potential barrier in using IPAWS. For example, representatives from two of the three alerting software providers we interviewed told us they have issued alerts through IPAWS at the request of their customers. According to these representatives, alerting authorities turn to their software providers as experienced users of the system because authorities have limited local staff, or if they cannot send an alert because of a technical reason. Four academics we interviewed said that FEMA should provide additional training for alerting authorities that is focused on drafting effective messages. Less than 20 percent of state, territorial, and local alerting authorities have sent WEA messages as of October 1, 2019. The limited use of IPAWS could lead to decreased proficiency or confidence. For example, an official from one alerting authority told us the jurisdiction did not use IPAWS at first because officials were not confident about using it. Our analysis of available information found that FEMA has addressed 4 of the 31 recommendations in its 2017 analysis. For example, FEMA revised its IPAWS training and added software requirements to its memorandum of agreement with alerting authorities. However, the extent to which FEMA has addressed other potentially useful recommendations is not clear because FEMA has not developed a plan to address the recommended actions. For example, one priority recommendation was to create skills checklists that provide a complete inventory of the types of skills alerting authorities need to use IPAWS. FEMA officials told us they had addressed many of the challenges identified in the 2017 analysis, including developing some timelines. However, FEMA did not provide documentation about how all the recommendations would be addressed. FEMA officials also told us they intend to periodically obtain information from alerting authorities about their needs and have begun another round of interviews with alerting authorities. However, these plans have not been documented. FEMA officials said they also use other methods to keep abreast of educational needs and challenges, such as attending conferences and reaching out to their contacts at emergency management associations that represent alerting authorities. In addition, alerting authorities send comments and feedback via email, according to FEMA officials. However, FEMA did not provide documentation about how it uses information obtained from these methods. As discussed earlier, FEMA is required by statute to educate federal, state, tribal, and local government officials. FEMA’s IPAWS strategic plan also includes a goal to make emergency alerting more effective, which as the plan explains, requires FEMA to engage non-federal alerting authorities to build competence and promote hands-on familiarity with IPAWS. The FEMA National Advisory Council has emphasized these points, recommending that FEMA improve alerting authorities’ ability to transmit effective alerts by developing and providing education, guidance, and best practices on how to use IPAWS as effective emergency messaging. Federal standards for internal control state that management should externally communicate necessary quality information. Open two-way external reporting lines allow for this communication. For example, management obtains quality information from external stakeholders—which in FEMA’s case would be information from alerting authorities—using established reporting lines. Additionally, federal internal control standards state that documentation provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel. Documenting how FEMA plans to address key recommendations from its 2017 analysis could help guide its efforts to educate alerting authorities and hold it accountable for addressing identified needs. Without a documented plan, FEMA may not systematically implement each recommendation, which could result in alerting authorities continuing to struggle with known challenges. In addition, by continuing its analytical efforts and implementing a mechanism to regularly obtain and analyze alerting authorities’ needs, FEMA could learn if these needs are changing and develop educational efforts to address them. Taking such actions will help FEMA enhance alerting authorities’ proficiency with, and confidence in, using IPAWS. FEMA has identified increasing adoption of the system and assisting authorities in gaining access to IPAWS as strategic goals. In addition, in June 2019 the FEMA Administrator issued a “call to action” policy memorandum to FEMA’s regional offices to help improve IPAWS adoption at the local level. As described earlier, FEMA has taken various steps in recent years to increase the adoption of IPAWS, for example, by informing local public safety agencies about IPAWS at conferences and encouraging them to apply for access to the system. In addition, FEMA has developed resources that are available on the IPAWS website that describe the expectations and steps for how a public safety agency may apply to become an IPAWS alerting authority. The number of authorities completing an initial step in the application process to obtain access to IPAWS has increased in recent years from 52 applicants in 2017 to 104 applicants in 2018 and to 122 applicants from January 2019 to September 2019. However, while more agencies are starting the application process, our analysis of FEMA data found that 430 IPAWS applications were pending as of September 2019, some of which dated back to 2012. Our analysis found that 152 applicants, or about one-third of the 430 applications, began the process (initiated the memorandum of agreement process) from 2012 to 2016. In addition, some applicants had yet to complete the key initial steps in the process. For example, after completing the required IPAWS web-based training and procuring IPAWS compatible software, public safety agencies must return a signed memorandum of agreement to FEMA before the application can move forward. We found that FEMA sent a draft memorandum of agreement to 108 applicants between 2014 and 2017 that had not yet returned the agreement to the agency as of September 2019. This could indicate that several applicants may be stalled in the early stages of the process and may benefit from FEMA’s assistance in completing the application or answering questions. FEMA officials said that once a completed application is received, approving it should take about 30 days but that factors outside FEMA’s control can contribute to processing delays and thereby increase the number of pending applications. For example, FEMA officials said it is out of their control when applicants do not return signed memorandums of agreement because that step of the process is handled at the state and local level. Representatives from an IPAWS applicant we interviewed said the amount of time it took to receive approval from the state authority was one of the reasons that their application was delayed. Although delays involving certain applications may be out of FEMA’s control, FEMA may be able to help other applicants. However, FEMA provided no evidence that it had followed up with applicants, when it had last contacted them, or how follow up should be prioritized. FEMA officials said one employee serves as a primary lead for managing the entire application process, which would require a labor-intensive process of following up with hundreds of applicants. FEMA officials said that managing pending applications is a challenge for the IPAWS office due to resource constraints. To help address these constraints, in 2019, FEMA awarded a contract to begin developing a new tool with the goal of streamlining FEMA’s management of applications. Officials said they anticipate the tool, estimated to be available in early 2020, will help them better manage the pending applications and conduct outreach as well as move new applications through the process. In 2016 FEMA conducted a study of the IPAWS application process and highlighted certain factors that contributed to an increasing backlog and response time, including FEMA officials not knowing that a follow up task for an applicant was waiting to be addressed. The study further indicated that determining the next step was manual and often reactive. Officials also said that staff will be able to run an aging report on applications to help them prioritize follow-up efforts. However, the agency has not established procedures to prioritize and follow up with applicants. FEMA officials acknowledged that establishing procedures to prioritize and follow up on the in-process applications would be beneficial. While these applications are pending, people in areas that are not covered by IPAWS authorities may not receive critical alerts and warnings from local authorities through IPAWS. Effective emergency alerting is vital to helping save lives and property during natural disasters and other threats to public safety, highlighting the importance of IPAWS as a way to disseminate critical information. However, FCC lacks specific goals and performance measures and FEMA lacks plans and processes, which may contribute to decreased confidence in and use of IPAWS by alerting authorities. In particular, because FCC does not have specific goals and performance measures to monitor WEA improvements, FCC will have difficulty assuring that these improvements are working as intended during emergencies and identifying areas where performance is lacking, which could undermine authorities’ confidence in using IPAWS. In addition, because FEMA has not documented next steps or plans for educating alerting authorities and establishing a process to regularly assess their educational needs, some authorities may continue to lack proficiency and confidence in using IPAWS. Furthermore, absent a strategy to address the substantial number of pending IPAWS applications, FEMA’s efforts to increase IPAWS adoption and expand alerting coverage are hindered. We are making a total of three recommendations, including one to FCC and two to FEMA. Specifically: The Chairman of FCC should develop specific, measurable goals and performance measures for its efforts to monitor the performance of new WEA capabilities, such as enhanced geo-targeting and expanded alert message length. (Recommendation 1) The Director of the IPAWS program should document how it plans to address key actions needed to educate alerting authorities in their use of IPAWS and implement a mechanism that will allow FEMA to regularly and systematically obtain and analyze feedback on alerting authorities’ educational needs. (Recommendation 2) The Director of the IPAWS program should establish procedures to prioritize pending IPAWS applications and to follow up with applicants to address these applications. (Recommendation 3) We provided a draft of this report to FCC, the Departments of Homeland Security (FEMA), Commerce (NOAA), and the Interior (USGS) for review and comment. FCC and the Department of Homeland Security provided written comments, reprinted in appendixes IV and V respectively. FCC, FEMA, and NOAA provided technical comments, which we incorporated as appropriate. In its written comments, FCC stated that it agreed with us on the importance of gathering and assessing specific performance information about the effectiveness of WEA capabilities. Separately, FCC officials noted that FCC was taking steps to gather this data, which will help inform the development of metrics, as we recommended. In its written comments, DHS concurred with our two recommendations to FEMA and provided information about activities that FEMA would undertake to implement them. We are sending copies of this report to the appropriate congressional committees, Chairman of FCC, Secretaries of Homeland Security, Commerce, and the Interior, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or Vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. This report examines (1) the trends in how alerting authorities use and test IPAWS and their experiences using IPAWS, and (2) actions that FEMA and FCC have taken to modernize IPAWS and increase its adoption, and the challenges they face. For background information on emergency alerting, we identified key issues and federal roles and responsibilities by reviewing applicable laws and regulations, our prior work, and reports prepared by FEMA, FCC, the Department of Homeland Security’s (DHS) Office of Inspector General, the Congressional Research Service, and academics. We also identified recent trends regarding natural disasters and the use of digital devices and the internet that could affect the use and frequency of emergency alerting. To identify natural disaster trends, we reviewed our prior work, a 2018 report prepared by the U.S. Global Change Research Program, and information on wildfires prepared by the California Department of Forestry and Fire Protection. We identified trends about the use of digital devices and the internet by reviewing surveys conducted from 2017 to 2019 by the Pew Research Center and The Nielsen Company, which regularly conduct national surveys on those topics. We also reviewed proposed federal legislation on emergency alerting. To examine the use of IPAWS and selected alerting authorities’ experiences using IPAWS, we analyzed IPAWS access and usage throughout the country from 2012 to 2019. We focused on identifying the authorities that used IPAWS from 2017 to 2019 following the passage of the IPAWS Modernization Act of 2015 (enacted in 2016). We analyzed IPAWS testing by judgmentally selecting samples of authorities conducting tests. In our calculations of the number of alerts issued by state, territorial, tribal, and local authorities, we focused on alerts for disasters and threats to public safety and excluded test alerts and alerts for missing persons and child abductions. We reviewed FEMA’s processes for ensuring the completeness and reliability of these alerting data and determined that they were sufficiently reliable for the purposes of examining trends in the use of emergency alerts. We also reviewed reports by FCC and local authorities on EAS and WEA test results. To obtain information on alerting authorities’ experiences using IPAWS, we conducted seven case studies of emergency events. To select them, we analyzed alerts that local authorities issued through IPAWS, FEMA’s list of federally declared disasters from 2017 to 2019, NOAA’s list of billion dollar disasters from 2017 to 2019, and our prior work on natural disaster preparedness and recovery from 2017 to 2019; considered recommendations from stakeholders; conducted internet searches; and reviewed news reports. We selected these case studies to include various areas of the country that experienced different types of disasters and threats to public safety during this time. These included natural disasters (wildfires and an earthquake), weather events (a hurricane and a flood), manmade disasters (a chemical fire and a power shortage), and a law enforcement event (a suspicious package). We then interviewed local alerting authorities in those areas. As a test case study, we interviewed District of Columbia emergency management officials. We conducted site visits with state and local emergency management officials in Los Angeles and Ventura, California; Bristol, Panama City, and Tallahassee, Florida; and Washington, D.C. To examine the actions that FEMA and FCC have taken to modernize IPAWS and increase its adoption, and the challenges they face, we reviewed FEMA documents such as IPAWS strategic plans and a performance report; FCC regulations, notices, and comments on FCC proposed rulemakings regarding EAS and WEA; and assessed the information against statutory requirements contained in the IPAWS Modernization Act and federal internal control standards. We focused on four areas of the Act that were key in the implementation of the program. These areas required FEMA, in consultation and coordination with FCC, to: ensure that IPAWS is capable of distributing alerts on the basis of geographic location, risks, and technologies; educate state, tribal, and local governments to understand how IPAWS works, and how and when to use IPAWS; establish training opportunities for alerting authorities; and conduct nationwide tests of IPAWS alerts. We compared FCC’s actions to leading practices based on the Government Performance and Results Act of 1993 (GPRA) as enhanced by the GPRA Modernization Act of 2010 (GPRAMA), which create a framework of goal setting and performance management for federal agencies. While GPRA and GPRAMA apply to the department or agency level, we have previously reported that their provisions can serve as leading practices at other organizational levels, such as component agencies, offices, programs, and projects. We also reviewed recommendations in reports prepared by the FEMA National Advisory Council IPAWS Subcommittee and FCC’s Communications Security, Reliability, and Interoperability Council, and disaster after-action reports prepared by FEMA and state and local governments. As an additional step in assessing the challenges that FEMA faces in increasing IPAWS adoption, we analyzed FEMA’s pending IPAWS applications as of September 2019 to determine which steps in the application and approval process had been completed and how long the applications were in process. We also interviewed four selected IPAWS applicants to obtain their views on the application process. To obtain a variety of perspectives, we selected applicants that were different types of organizations (an airport, a university, a local government, and a federal agency) in different areas of the country. In addition, for both objectives, we interviewed officials from FEMA, FCC, NOAA, USGS, and 18 state, local and territorial alerting authorities; representatives from 4 industry associations, 2 advocacy groups, and 15 companies, including wireless carriers, internet service providers, internet content providers, IPAWS software providers, and mobile device manufacturers; and 7 academics. To obtain a variety of perspectives, we selected industry associations and companies that represented different telecommunications industry sectors and have different roles in emergency alerting (broadcasting, cable, wireless, internet service, and application developers) and academics with different areas of expertise (public health, engineering, natural hazards, disaster preparedness, rural emergency management, and communication). We also interviewed staff from a county board that oversees emergency management activities in that jurisdiction and officials from a city that is planning to apply for IPAWS access. The results of these interviews are not generalizable to all stakeholders, but provide insight on the use of IPAWS and related emergency alerting issues. Our interviewees are listed in table 2 below. Selected Alert Sent by the Florida Division of Emergency Management for Bay County during Hurricane Michael: October 10, 2018: GOVERNOR EVAC ALERT 6 to 13 FT STORM SURGE EXPECTED IN BAY COUNTY Zones A,B,C EVACUATE NOW Selected alerts sent by Bay County Emergency Services during Hurricane Michael: October 10, 2018: Dangerous w inds are beginning to occur in Bay County Shelter in place now October 15, 2018: Bay County remains under a boil w ater notice. Please boil or use bottled w ater for consumption. October 15, 2018: FOOD AND OR WATER ARE AVAILABLE NORTH OF 15th AND CR 386 AT 1011 CR 386 SOUTH Bay County officials said the county lost its ability to issue alerts at this point. Hurricane Michael, October 2018: Hurricane Michael was a category 5 storm that NWS reported made a catastrophic landfall near Mexico Beach and Tyndall Air Force Base, Florida, producing devastating winds and storm surge near the coast, and rain and wind inland (see fig. 6). According to a State of Florida report, Hurricane Michael was the most powerful storm to hit the Panhandle region and the third most intense storm to make landfall in the mainland United States in recorded history. During the storm, several counties could not send alerts because of power outages and inoperable cellular towers. Officials from an alerting authority we interviewed in Florida commended the state’s ability to send IPAWS alerts on behalf of the county, which had lost its communications capabilities during the storm. Authorities also said IPAWS provides an ability to warn the public about approaching hurricanes and share critical lifesaving information such as the location of food, water, and shelter. However, authorities expressed frustration about the inability to accurately geo-target WEA messages to evacuation zones and about how the WEA text character limit forced them to issue multiple WEA messages regarding the same alert. Some officials said they were frustrated when certain local EAS alerts were not delivered by broadcasters, which could prevent some people from receiving them. Alert issued by the California Governor’s Office of Emergency Services: Dec. 6, 2017: Strong w inds over night creating extreme fire danger. Stay alert. Listen to authorities. Alert issued by the Ventura County Sheriff’s Office – Office of Emergency Services: Dec. 7, 2017: VENTURA COUNTY-FAST MOVING BRUSH FIRE NORTH OF OJAI.GO TO READYVENTURACOUNTY.ORG FOR INFO Selected alerts issued by the City of Los Angeles: Dec. 6, 2017: For information regarding the Skirball Fire in Los Angeles please go to Tw itter.com/LAFD Nov. 9, 2018: MANDATORY EVACUATION in West Hills: W of Valley Circle, N to Roscoe Blvd, S to Vanowen. Selected alert issued by Santa Barbara County: Dec. 16, 2017: EVAC ORDER: SB City: east of Hw y 154 to Mission Canyon Rd and north of 192. Leave now . Southern California Wildfires, December 2017 and November 2018: The southern California area experienced large wildfires in recent years, including the Thomas fire in December 2017 and the Woolsey fire in November 2018 (see fig. 7). The California Department of Forestry and Fire Protection reported in August 2019 that the Thomas fire, which affected Santa Barbara and Ventura Counties, was the second-largest wildfire in the state’s history and destroyed more than 1,000 structures. The Woolsey fire, which affected Los Angeles and Ventura Counties, had a footprint over 150 square miles and resulted in the evacuation of about a quarter-million people. According to Los Angeles County, the Woolsey fire was the most destructive fire in the county’s history. California officials we interviewed said IPAWS is an effective tool for wildfire evacuations and that because most people have cell phones, they do not have to subscribe to receive WEA messages. Officials also praised the capability of IPAWS to allow a state alerting authority to send alerts to at-risk counties ahead of potential wildfires. However, officials said it is a challenge to know when and where other alerting authorities in the area are sending alerts and that there may be little time. For example, an official told us that the Thomas fire moved at 60 miles per hour. Officials also said that even though WEA messages were targeted to an area during the fires, they did not know whether people received them because geo-targeting was not precise and because cell towers may have been damaged. Alerts issued by the New York City Emergency Management Department on Oct. 24, 2018: Police Activity: Residents on W 58th St btw Columbus & 8th Av shelter in place immediately The suspicious device on W 58 St & 8th Ave w as safely removed by NYPD Bomb Squad. Suspicious Package in New York City, October 24, 2018: On October 24, 2018, the New York City Emergency Management Department issued a WEA shelter-in-place order regarding a suspicious package at the Time Warner Center in Manhattan that was found to contain an improvised explosive device (see fig. 8). According to officials, police removed the device and determined it was no longer a threat. About an hour after the initial alert was issued, the city issued another WEA canceling the shelter- in-place alert. New York City officials said IPAWS is the city’s most effective alert and warning tool, compared with its own alerting system to which about 9 percent of the population has subscribed. Regarding the suspicious package, authorities were able to draw an alerting area covering a 3 to 4 block radius. The officials also said that WEA messages were instrumental in helping to capture a suspect in a bombing incident in the city’s Chelsea neighborhood in 2016. However, officials said the October 2018 alert was received as far as 4 miles from the targeted area, which led them to speculate about the number of people who received the alert. Officials also said they would like IPAWS to incorporate more languages for use in alerts and provide them with the ability to use photographs or maps in future alerts. Alerts issued by the Douglas County Emergency Management Agency: March 14, 2019: From Douglas County Sheriffs Office. Record flooding on Elkhorn River. Evacuate Now . March 15, 2019: From Douglas County Sheriff: Evacuate City of Valley NOW. Use Q Street. Hw y 275 closed. Alert Issued by Platte County Emergency Management: March 14, 2019: Travel is not advised in and around Columbus and Platte County due to extensive flooding. Flood in Nebraska, March 2019: In March 2019, Nebraska experienced one of the most devastating floods in recent history, according to the state government (see fig. 9). We interviewed officials in Douglas County and Platte County, areas that experienced torrential rain and flooding. One county sent a geo-targeted WEA evacuation alert to people living near a river while the other county sent a WEA advising the public to not travel within the county. A county official in Nebraska said that if the planned future enhancements to WEA take place and are found to be successful, WEA will ultimately be of greater value than other means of notification such as the county’s previous subscription system, which had a low participation rate. The official stressed the difficulty in explaining the threat, the source of the alert, and a protective action within the 90 character WEA limit. The official also noted that some local broadcasters were not equipped to recognize an EAS law enforcement alert for further transmission. An official in another county said that some people did not receive the WEA messages. Power shortage in Michigan, January 2019: On January 30, 2019, Consumers Energy, a primary energy supplier in Michigan, experienced a fire at a natural gas storage facility at a time when there was high energy demand because of extreme cold temperatures (see fig. 10). According to NWS, Michigan’s Lower Peninsula experienced the lowest temperatures in decades—down to minus 20 degrees with wind chills of down to minus 40 degrees. As a result, the state’s Emergency Operations Center asked the Michigan State Police, an IPAWS alerting authority, to issue WEA and EAS alerts asking people to lower their thermostats to conserve natural gas. Michigan State Police officials said that IPAWS allowed the alerting authority to send a WEA message to 68 counties, which was an effective and quick way to reach many people. However, officials said they attempted to send an EAS alert to all 68 counties in Michigan’s Lower Peninsula but were limited to a total of 31 counties per alert, per FCC regulations. They said that after the EAS alert was sent, the actual EAS broadcast message was not displayed on television because the entire list of the 31 county names, which must be read first according to FCC regulations, took up the allowable 2-minute time span for an EAS broadcast. Alerts issued by the City of Houston on Sept. 20, 2017: Shelter in Place in northw est Spring Branch due to hazardous fire. Check local media. Shelter in place is CANCELLED for Northw est Spring Branch after hazardous fire. Chemical Fire in Houston, Texas, September 20, 2017: The Houston Fire Department requested that the Houston Office of Emergency Management issue a WEA shelter-in-place order following a chemical fire at a bearing supply company that resulted in the release of potentially hazardous smoke (see fig. 11). Houston officials said they believe that IPAWS allowed the alerting authority to reach a broad area at risk using the WEA message. However, officials said it is possible that a lack of training on behalf of the alerting authority, among other things, limited their ability to issue the alert in a timely fashion. They said it took the alerting authority 43 minutes and multiple attempts to properly prepare and send the message using its IPAWS-compatible software before the message was successfully sent to the public. Earthquake in Alaska, November 30, 2018: A magnitude 7.0 earthquake struck north of Anchorage, Alaska, on November 30, 2018 (see fig. 12). We interviewed officials from three local governments that were affected by the earthquake. Officials at one borough said they did not issue an IPAWS alert because the earthquake had a short intensity and they did not receive reports of fatalities or widespread damage. However, the officials said that if the earthquake’s intensity had been greater, they would have issued used IPAWS to alert people about shelter locations. NWS used IPAWS to issue a tsunami warning but local officials did not issue any alerts through IPAWS. Officials in Alaska said that it is helpful that another government agency can be a backup alerting authority and provide alerts through IPAWS on behalf of the local government. However, an official said the inability to precisely geo-target alerts about tsunami risks to coastal areas prevented the official from sending out an alert due to concerns that people who were not affected by the earthquake would receive the alert. Another official said the cost of procuring alerting software that is compatible with IPAWS may be a challenge for some local governments. In addition to the individual named above, Sally Moino (Assistant Director); Michael Sweet (Analyst in Charge); David Aja; Melissa Bodeau; Mark Goldstein; Bob Homan; Kate Perl; Cheryl Peterson; Sam Portnow; Malika Rice; and Andrew Stavisky made key contributions to this report.", "summary": "Public alerts and warnings are critical to protect lives and provide information during emergencies, such as wildfires and floods. The IPAWS Modernization Act, enacted in 2016, required FEMA, in consultation and coordination with FCC, to enhance and test the capabilities of IPAWS and increase its adoption among state and local public safety agencies. GAO was asked to review the federal response to recent natural disasters. This report examines, among other things: (1) trends in the use of IPAWS and (2) actions that FEMA and FCC have taken to modernize IPAWS and increase its adoption. GAO analyzed relevant data and documentation and assessed FCC's efforts against leading government performance management practices and FEMA and FCC's efforts against internal control standards. GAO interviewed federal officials involved in emergency alerting. GAO also interviewed a non-generalizable selection of IPAWS alerting authorities and applicants, local governments, public safety and industry associations, and communications companies. GAO selected alerting authorities that experienced different types of disasters and threats to public safety from 2017 to 2019. Use of the Integrated Public Alert and Warning System (IPAWS) has increased since its launch in 2012. IPAWS enables authorized federal, state, territorial, tribal, and local alerting authorities to send a Wireless Emergency Alert (WEA) to mobile devices, such as cell phones and an Emergency Alert System (EAS) alert to media platforms, such as radios and television. The Federal Emergency Management Agency (FEMA) operates IPAWS and the Federal Communications Commission (FCC) establishes rules for telecommunications providers to deliver WEA and EAS alerts. A public safety agency must submit an application and receive approval from FEMA to become an IPAWS alerting authority. In September 2019, more than 1,400 alerting authorities had access to IPAWS, up from fewer than 100 authorities in 2013. All states have at least one state alerting authority, but gaps in local authority access remain (see figure) that could limit the timeliness of alerts as emergencies occur at the local level. GAO found 430 pending IPAWS applications as of September 2019, some of which dated back to 2012. FEMA has not established procedures to prioritize and follow up with applicants and FEMA officials acknowledged that doing so would be beneficial. FEMA and FCC have taken steps to modernize IPAWS and improve alerting. For example, FEMA has made system upgrades and FCC has made various WEA improvements, such as requiring wireless phone carriers to provide more precise geographic targeting of alerts. Prior to these improvements, officials from many alerting authorities said the inability to geographically target alerts with accuracy made the officials reluctant to send WEA messages. FCC intends to partner with certain localities to test geographic targeting and, according to FCC officials, plans to use other tests to learn about how the improvements perform during emergencies. However, FCC has not developed goals and performance measures for these efforts. Doing so would help FCC more clearly assess whether the WEA improvements are working as intended. Furthermore, having specific performance information could increase alerting authorities' confidence in and use of IPAWS. GAO is making three recommendations, including that FEMA establish procedures to prioritize and address pending IPAWS applications and that FCC develop goals and performance measures to monitor the WEA improvements. FEMA concurred with GAO's recommendations. FCC stated it was taking steps to gather data to inform the development of metrics as GAO recommended.", "document_type": "gao"}
{"report": "The chemical industry relies on the use of natural resources as inputs to make chemical products, and the industry’s outputs, in turn, can have an impact on the environment. The International Trade Administration of the Department of Commerce identifies the chemical industry as one of the largest manufacturing industries in the United States, with more than 10,000 companies producing more than 70,000 products. The term ‘sustainability’ can have many interpretations depending on the context in which it is used. Sustainability may refer to economic, environmental, or social sustainability. Achieving all three—a concept known as the “triple bottom line”—has become a goal of some businesses, including many in the chemical industry. Mitigating the potential negative health and environmental consequences of chemical production requires thoughtful design and evaluation throughout the life cycle of chemical processes and products —that is, a thorough assessment of effects resulting from stages of the life cycle such as sourcing the raw materials, processing raw materials into products, handling and disposal of by-products and industrial waste, product use, and end-of-life disposal or recycling (see fig. 1). Attempting to improve one stage of the life cycle without considering the others runs the risk of moving sustainability problems around rather than solving them. Analyzing the full life cycle of a process or product can reveal benefits as well as trade-offs or unintended consequences of different choices along the way. Consistent with the goals of sustainable chemistry, which include making chemicals in a purposefully more environmentally benign way, several federal requirements and directives address chemical and other risks to public health and the environment. For example, EPA’s ability to effectively implement its mission of protecting public health and the environment is critically dependent on credible and timely assessments of the risks posed by chemicals. Such assessments are the cornerstone of scientifically sound environmental decisions, policies, and regulations under a variety of statutes, such as the Toxic Substances Control Act (TSCA) (as amended), which provides EPA with authority to obtain information on chemicals and to regulate those that it determines pose unreasonable risks; the Safe Drinking Water Act (SDWA) (as amended), which authorizes EPA to regulate contaminants in public drinking water systems; and the Federal Food, Drug, and Cosmetic Act (as amended), which authorizes the Food and Drug Administration to oversee the safety of food, drugs, medical devices, and cosmetics. The Federal Acquisition Regulation generally requires that federal agencies advance sustainable acquisition by ensuring that 95 percent of new contract actions for the supply of products and for the acquisition of services meet certain sustainability goals. Various economic factors influence the development of sustainable products. Consumers are increasingly seeking products that help them reduce their own environmental footprints, and companies are responding by developing products made with safer chemicals and by increasing the use of recycled, biobased, and renewable materials. The supply of such products can be influenced by the costs of production, competitive advantage, and reputational effects. For example, if a more sustainable product or process helps a firm differentiate from another firm and creates a competitive advantage that consumers recognize and value, it will enable firms to create more sustainable products. There are a number of inherent challenges in the market for sustainable products in the industry. For example, substantial upfront costs coupled with uncertainty about consumer demand may be a barrier to entering the market. If the benefits of taking a more sustainable approach are valued by consumers, companies may be able to recoup the higher costs by charging higher prices without reducing demand. However, if the benefits are not easily understood and measureable (e.g., long-term health benefits), or are external to consumers (e.g., broad environmental impacts), then consumers may not be willing to pay higher prices for more sustainable products. In addition to market incentives that encourage firms to produce more sustainable products, government entities can, when appropriate, take actions such as subsidies, award programs, or tax credits, or limits, bans, and taxes. Governments may also provide environmental and health- related information to help guide the choices of consumers, workers, downstream users, and investors. For new markets and investments to be realized, sufficient information is needed on the environmental damage and health hazards that can be associated with some chemicals and the possibilities that exist to develop alternatives that overcome these challenges. In February 2018, we reported that stakeholders vary in (1) how they define sustainable chemistry, (2) how they assess sustainability, and (3) which environmental and health factors they considered most important. Most companies that responded to our survey agreed that a standardized set of factors for assessing sustainability would be useful. Stakeholders do not agree on a single definition of sustainable chemistry. In total, we asked 71 representatives of stakeholder organizations how they or their organization defines sustainable chemistry. The most common response we received was that sustainable chemistry includes minimizing the use of non-renewable resources. Other concepts that stakeholders commonly associated with sustainable chemistry included minimizing the use of toxic or hazardous chemicals, considering trade- offs between various factors during each phase of the life cycle, minimizing energy and water use, and increasing biodegradability or recyclability. Based on a review of the literature and stakeholder interviews, we identified several common themes underlying what sustainable chemistry strives to achieve, including: improve the efficiency with which natural resources—including energy, water, and materials—are used to meet human needs for chemical products while avoiding environmental harm; reduce or eliminate the use or generation of hazardous substances in the design, manufacture, and use of chemical products; protect and benefit the economy, people, and the environment using consider all life cycle stages including manufacture, use, and disposal (see fig. 1) when evaluating the environmental impact of a product; and minimize the use of non-renewable resources. Stakeholders such as chemical companies, federal agencies, and others use many different approaches for assessing the sustainability of chemical processes and products. While the varying approaches provide flexibility to meet the priorities of the user, the lack of a standardized approach makes it very difficult for customers, decision makers, and others to compare the sustainability of various products to make informed decisions. Some companies and organizations design their own approaches for assessing chemical sustainability and use those approaches to make internal decisions on product design and processing, while others use metrics, chemical selection guides, or third-party certifications and assessment tools that are common to their industry. For example, chemical companies use several established metrics to measure their efficiency in using materials to generate products. The variety of metrics used—and variation in the underlying factors included in their calculation—hinders the ability of companies and others to compare the sustainability of chemical processes or products. In addition to common metrics, some sectors have developed guides that companies and others can use to compare the sustainability of materials used in chemical processes, including solvent selection guides and reagent guides. Solvent selection guides assess solvents based on a variety of sustainability criteria, such as environmental, health, and safety impacts; recyclability; and regulatory concerns. One pharmaceutical company reported a 50 percent decrease in the use of certain hazardous solvents after the introduction of a solvent selection guide. NGOs, federal agencies, and professional associations are also developing product certification programs and assessment tools. Certification programs set minimum criteria that products must meet to be certified, such as biodegradability, toxicity, performance, or water usage. Certifying bodies make databases of certified products publicly available and allow manufacturers to affix certification labels or logos to their products. Companies prioritize various environmental and health factors differently when assessing sustainability, according to our survey of 27 companies. We asked respondents to indicate the relative importance their company gives to each of 13 environmental and health factors by comparing a pair of factors and selecting the factor they considered more important to optimize, even if that benefit came at the expense of the other factor. For example, a company might compare “energy use” with “water use” and determine that it was more important to their company to maximize the sustainability benefit relative to the “energy use” of a process even if it resulted in less sustainable use of water. We found that, overall, “toxicity of the product” was the most important factor for the companies surveyed and “percentage of renewable or biobased content” was the least important factor when making trade-offs (see fig. 2). However, there were sizable differences between companies and sectors regarding which factors they considered most important to optimize. For a more detailed description of our analysis, see our report Chemical Innovation: Technologies to Make Processes and Products More Sustainable. The literature and the results of our interviews and survey indicate that the lack of a standard definition for sustainable chemistry, combined with the lack of standard ways of measuring or assessing sustainability, hinder the development and adoption of more sustainable chemistry technologies. It is difficult for consumers, purchasers, policymakers, and even manufacturers to compare the sustainability of one process or product with another when such processes and products are assessed using different metrics that incorporate different factors. In addition, while there were sizable differences between the companies that responded to our survey with regard to which environmental and health factors they considered most important to prioritize, most agreed that it would be useful to have a standardized set of factors for assessing sustainability across their industry sector and (to a lesser degree) across the entire industry. There are many technologies available and in development that can improve chemical sustainability at each stage of the chemical life cycle. Our February 2018 report focused on three categories: catalysts, solvents, and continuous processing. Because each chemical process or product has unique requirements, there is no one-size-fits-all solution to sustainability concerns. Catalysts are used to make chemical processes run faster or use less material. One common application is the catalytic converter in an automobile, where the catalyst converts pollutant gases in the exhaust into harmless chemicals. Without catalysts, many everyday items such as medicines, fibers, fuels, and paints could not be produced in sufficient quantities to meet demand. Unfortunately, the most common catalysts— including those used in automobile catalytic converters—are rare, nonrenewable metals such as platinum and palladium. Researchers are working to replace such metals with alternatives, including abundant metals (e.g., iron and nickel) and metal-free catalysts (such as biocatalysts) where possible. For example, in 2016, Newlight Technologies won a Presidential Green Chemistry Challenge Award for developing and commercializing a biocatalyst technology that captures methane (a potent greenhouse gas) and combines it with air to create a material that matches the performance of petroleum-based plastics at a lower cost. Several companies are now using this material to make a range of products, including packaging, cell phone cases, and furniture. Solvents are key components in chemical reactions. They are used to dissolve other substances so reactions can occur, to separate and purify chemicals, and to clean the equipment used in chemical processes, among other uses. Solvents constitute a large portion of the total volume of chemicals used in industrial chemical processes. However, many conventional solvents are considered hazardous, both to the environment and to human health. There are a variety of alternatives that can be used in some situations, including biobased solvents, less hazardous solvents such as water or ethanol, and solvent-free or reduced-solvent technologies. For example, biobased solvents called citrus terpenes, which are extracted from citrus peel waste, can be used as flavoring agents or fragrances in cleaning products. According to a representative from Florida Chemical, citrus terpenes may be a low-toxicity alternative compared to traditionally used petroleum-based products for the hydraulic fracturing industry’s concerns about contamination of source and groundwater. However, the regionality and seasonality of the citrus supply can present a challenge to production. Historically, industrial chemicals have been produced mainly using an approach known as batch processing, where the starting materials are combined in a closed vessel or vat and allowed to react, then transferred to the next vat for the next stage of processing while the first vat is cleaned, and the process is repeated with the next batch. This approach can use significant amounts of solvents for cleaning the vats between batches, consume considerable energy, result in potentially long wait times, and create safety risks. An alternative to batch processing is continuous processing, which allows chemical reactions to occur as the reaction mixture is pumped through a series of pipes or tubes where reactions take place continuously. This approach can improve product yield, product quality, reaction time, and process safety while reducing waste and costs. For example, researchers developed a process for manufacturing the active ingredient in medications including Benadryl® and Tylenol® PM using microreactors that minimized waste, reduced the number of purification steps, and reduced production times compared to traditional batch processing. The federal government and other stakeholders play a number of roles, sometimes in collaboration, to advance the development and use of more sustainable chemical processes and products. Federal programs support research on the impacts of chemicals on human and environmental health, support the development of more sustainable chemical processes and their commercialization, and aid the expansion of markets for products manufactured with more sustainable chemicals and processes. Other stakeholders play similar roles and some additional roles that contribute to the development and use of more sustainable chemical processes and products. Federal programs conduct and fund basic research on the characteristics and biological effects of chemicals, which underpins the development and use of more sustainable chemistry products and processes. Decision makers must have a scientific understanding of the potential harmful impacts of exposure to chemicals in order to effectively minimize the harmful effects of chemicals through regulations and other means, and to assess the regulated community’s compliance with them. Industry needs this information to make informed decisions about the selection, design, and use of more sustainable chemicals in their products and processes, including their impact on workers. Federal programs fund and study the impacts of chemicals on human health and the environment, develop new methodologies for testing and predicting these effects, award grants for research on chemicals and new methodologies, identify more sustainable chemical alternatives, and evaluate the risks of chemicals. (See table 1.) Federal programs also seek to support the development and facilitate the commercialization of new, more sustainable chemistry processes by conducting and funding basic and applied research to develop more sustainable processes and products; providing loan guarantees, grants, and technical assistance to researchers and companies; and recognizing innovative technologies through an award program, among other programs. (See table 2.) Federal programs also aid market growth for products made with sustainable chemicals and processes by informing consumers about these products and by facilitating their purchase by federal offices. It can be challenging for consumers seeking out more sustainably manufactured products to identify them or verify company claims. Federal programs can help companies seeking to manufacture more sustainable products strive to ensure that their products are differentiated from less sustainable products in order to reach these consumers. For example, federal programs conduct evaluations of the chemical content of products, manage product certification and labeling programs, provide information to consumers and federal purchasers on the chemical content of products, and develop purchasing and sustainability plans to support agency purchase and use of more sustainable products. EPA’s Safer Choice voluntary certification and labeling program helps consumers make informed purchasing decisions and incentivizes manufacturers to select more sustainable chemical alternatives so they can differentiate their products in the market. Other stakeholders—such as the chemical manufacturing industry, companies and retailers, state governments, academic institutions, and NGOs—also seek to influence the development and use of more sustainable chemistry processes and products through activities such as supporting workforce development and developing tools and resources for industry. These stakeholders may work on collaborative efforts, such as sustainability initiatives and developing industry-specific standards. The chemical industry conducts and supports research into more sustainable chemistry technologies and other activities. Companies and retailers, such as Kaiser Permanente and Target, create demand for more sustainable products from their suppliers by setting sustainability criteria for purchases. Academic institutions conduct research on the impacts of chemicals and sustainable chemistry technologies and train the next generation of chemists and engineers. States seek to protect public health by regulating chemicals in products. NGOs also play a diverse range of roles such as supporting workforce development, facilitating collaboration between other stakeholders, and developing tools and resources for industry. Sustainable chemistry is an emerging field within the chemical sciences that has the potential to inspire new products and processes, create jobs, and enhance benefits to human health and the environment. Stakeholders offered a range of potential options to realize the full potential of these technologies. However, there are a number of challenges to implementing more sustainable chemistry technologies, including technological, business, and industry-wide and sector-specific challenges. The field of sustainable chemistry has the potential to inspire new products and processes, create jobs, and enhance benefits to human health and the environment. Stakeholders noted that much more work is needed to realize its full promise and offered a range of potential options to realize the full potential of these technologies, including the following: Breakthrough technologies in sustainable chemistry and a new conceptual framework could transform how the industry thinks about performance, function, and synthesis. An industry consortium, working in partnership with a key supporter at the federal level, could help make sustainable chemistry a priority and lead to an effective national initiative or strategy. Integrating sustainable chemistry principles into educational programs could bolster a new generation of chemists, encourage innovation, and advance achievement in the field. A national initiative that considers sustainable chemistry in a systematic manner could encourage collaborations among industry, academia, and the government, similar to the National Nanotechnology Initiative. There are opportunities for the federal government to address industry-wide challenges such as developing standard tools for assessment and a robust definition of sustainable chemistry. Federal agencies can also play a role in demonstrating, piloting, and de- risking some technology development efforts. Stakeholders noted that there are a number of challenges to implementing more sustainable chemistry technologies, including (1) technological and business challenges, (2) industry-wide and sector- specific challenges, and (3) challenges with coordination between stakeholders. One example of a technological challenge is the fact that alternatives to current solvent use can sometimes pose the same inherent toxicity and volatility risks as their conventional counterparts. Alternatives can also vary in supply and quality and can be expensive. Less toxic solvents, such as water, may require specialized equipment, greater energy input, or elevated pressure, and they can be difficult to scale up for industrial use. Companies told us they face many business challenges in implementing sustainable chemistry technologies, including the need to prioritize product performance; weigh sustainability trade-offs between various technologies; risk disruptions to the supply chain when switching to a more sustainable option; and consider regulatory challenges, among others. Stakeholders also noted the challenge of overturning proven conventional practices and acknowledged that existing capital investments in current technologies can create barriers for new companies to enter a field full of well-established players. Our survey and interviews also found that there are several industry-wide and sector-specific challenges to implementing more sustainable chemistry technologies, such as the lack of a standard definition for sustainable chemistry and lack of agreement on standard ways of measuring or assessing it. Without a standard definition that captures the full range of activities within sustainable chemistry, it is difficult to define the universe of relevant players. Without agreement on how to measure the sustainability of chemical processes and products, companies may be hesitant to invest in innovation they cannot effectively quantify, and end users are unable to make meaningful comparisons that allow them to select appropriate chemical products and processes. There is no mechanism for coordinating a standardized set of sustainability factors across the diverse range of stakeholders at present, despite the motivation of some specific sectors to do so. Moreover, although the federal government has worked with stakeholders through its research support, technical assistance, certification programs, and other efforts, there are still gaps in understanding. Many stakeholders told us that without such basic information as a standardized approach for assessing the sustainability of chemical processes and products, better information on product content throughout the supply chain, and more complete data on the health and environmental impacts of chemicals throughout their life cycle, they cannot make informed decisions that compare the sustainability of various products. Sector-specific challenges exist as well. For example, pharmaceutical sector representatives told us that changing the manufacturing process for an already marketed drug triggers a new FDA review, which can result in delays and additional costs—thus discouraging innovation that could make their chemical processes more sustainable. In conclusion, according to stakeholders, transitioning toward the use of more sustainable chemistry technologies requires that industry, government, and other stakeholders work together. As they and others noted, there is a need for new processes that make more efficient use of the resources that are available, reuse products or their components during manufacturing, and account for impacts across the entire life cycle of chemical processes and products. Furthermore, they highlight the importance of disseminating environmental and health-related information to help guide the choices of consumers, chemists, workers, downstream users, and investors to facilitate further progress. They also indicated that momentum in this field will require national leadership in order to realize the full potential of sustainable chemistry technologies. Chairwoman Stevens, Ranking Member Baird, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact me at 202-512-6412 or personst@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony include Karen Howard (Assistant Director), Diane Raynes (Assistant Director), Katrina Pekar-Carpenter (Analyst-in-Charge), Patrick Harner, Summer Lingard-Smith, Krista Mantsch, Anika McMillon, Rebecca Parkhurst, and Ben Shouse. Other staff who made key contributions to the report cited in the testimony are identified in that report. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Chemistry contributes to virtually every aspect of modern life, and the chemical industry supports nearly 26 percent of the gross domestic product of the United States. While these are positive contributions, chemical processes and production can have negative health and environmental consequences. Mitigating these potential consequences requires thoughtful design and evaluation of the life cycle effects of chemical processes and products. This testimony—based on a 2018 technology assessment, GAO-18-307 —discusses (1) how stakeholders define and assess the sustainability of chemical processes and products, (2) available or developing technologies to make chemical processes and products more sustainable, (3) the roles of the federal government and others in supporting the development and use of more sustainable chemical processes and products, and (4) opportunities and challenges in the field of sustainable chemistry. For the 2018 report, GAO selected for assessment three technology categories—catalysts, solvents, and continuous processing; interviewed stakeholders from various fields, such as government, industry, and academia; convened a meeting of experts on sustainable chemistry technologies and approaches; and surveyed a non-generalizable sample of chemical companies. Stakeholders vary in how they define and assess the sustainability of chemical processes and products; these differences hinder the development and adoption of more sustainable chemistry technologies. However, based on a review of the literature and stakeholder interviews, GAO identified several common themes underlying what sustainable chemistry strives to achieve, including: improve the efficiency with which natural resources are used to meet human needs for chemical products while avoiding environmental harm; reduce or eliminate the use or generation of hazardous substances, minimize the use of non-renewable resources; and consider all life cycle stages when evaluating a product (see figure). There are many technologies available and in development that can improve chemical sustainability at each stage of the chemical life cycle. GAO identified three categories of more sustainable chemistry technologies—catalysts, solvents, and continuous processing. Catalysts are used to make chemical processes run faster or use less material. Without catalysts, many everyday items such as medicines, fibers, fuels, and paints could not be produced in sufficient quantities to meet demand. However, the most common catalysts—including those used in automobile catalytic converters—are rare, nonrenewable metals such as platinum and palladium. Researchers are working to replace such metals with alternatives, including abundant metals (e.g., iron and nickel) where possible. Solvents are used to dissolve other substances so reactions can occur, to separate and purify chemicals, and to clean the equipment used in chemical processes, among other uses. Solvents constitute a large portion of the total volume of chemicals used in industrial chemical processes. However, many conventional solvents are considered hazardous. There are a variety of alternatives that can be used in some situations, including biobased solvents. An alternative to traditional batch processing is continuous processing, which allows chemical reactions to occur as the reaction mixture is pumped through a series of pipes or tubes where reactions take place continuously. Compared to batch processing, this approach can improve product yield, product quality, and process safety while reducing waste and costs. The federal government and other stakeholders play several roles, sometimes in collaboration, to advance the development and use of more sustainable chemistry technologies. The federal government supports research, provides technical assistance, and offers certification programs, while other stakeholders conduct research, develop industry-specific standards, support workforce development development, and address chemicals of concern in consumer products, among other roles. Strategic Implications While using more sustainable options entails challenges--including technological, business, and industry-wide and sector-specific challenges, the field of sustainable chemistry has the potential to inspire new products and processes, create jobs, and enhance benefits to human health and the environment. Stakeholders identified strategic implications of sustainable chemistry and offered a range of potential options and realize the full potential of these technologies, including the following: Breakthrough technologies in sustainable chemistry and a new conceptual framework could transform how the industry thinks about performance, function, and synthesis. An industry consortium, working in partnership with a key supporter at the federal level, could help make sustainable chemistry a priority and lead to an effective national initiative or strategy. Integrating sustainable chemistry principles into educational programs could bolster a new generation of chemists, encourage innovation, and advance achievement in the field. A national initiative that considers sustainable chemistry in a systematic manner could encourage collaborations among industry, academia and the government, similar to the National Nanotechnology Initiative. There are opportunities for the federal government to address industry-wide challenges such as developing standard tools for assessment and a robust definition of sustainable chemistry. Federal agencies can also play a role in demonstrating, piloting, and de-risking some technology development efforts. According to stakeholders, transitioning toward the use of more sustainable chemistry technologies will require national leadership and industry, government, and other stakeholders to work together.", "document_type": "gao"}
{"report": "CMS and private payers use a variety of quality measures to assess different aspects of health care quality. Process measures assess the extent to which providers effectively implement clinical practices (or treatments) that have been shown to result in high-quality or efficient care, such as the percentage of patients with a myocardial infarction who receive an aspirin prescription on discharge. Others are outcome measures, which track the results of health care, such as mortality, infections, and patients’ experiences of that care. To calculate providers’ performance on quality measures, CMS and private payers ask providers to report a variety of clinical data. Historically, providers have collected data for quality measures through a detailed, manual review of paper medical records. Other quality measures use data from billing records and patient surveys. More recently, a limited number of electronic quality measures have been developed to allow providers to report data electronically using electronic health records. Since the early 2000s, CMS has created a number of distinct quality reporting programs within Medicare. These programs generally focus on different sites of care, such as hospitals, physician offices, and nursing homes. Beginning in the early 2000s, CMS launched a number of related programs that offer financial incentives to providers receiving Medicare payments to report their performance on specified quality measures. Some of these programs, such as the Hospital Inpatient Quality Reporting program, are pay-for-reporting programs, in which providers may receive higher payments if they report their performance on the quality measures used in the programs. Others, such as the Hospital Value-based Purchasing program, are pay-for-performance programs, in which the level of providers’ performance on the quality measures affects the amount of the payment they receive. CMS also incorporates pay-for- performance in various alternative payment models, such as accountable care organizations—where CMS pays groups of providers based in part on the collective performance of those providers, rather than the fee-for- service traditionally paid in Medicare. At any given point in time, CMS has a set of quality measures it is currently using in its various Medicare quality programs as well as efforts underway to identify different quality measures to better meet program needs. These quality measures may either already have been developed or potentially could be developed. A variety of different entities may develop new health care quality measures, such as the Joint Commission, the National Committee for Quality Assurance, and various medical specialty societies. In some cases CMS itself contracts with entities for the development of measures for use in its Medicare quality programs. CMS has developed a set of guidelines for developing new quality measures that are described in its Blueprint for the CMS Measures Management System. The Blueprint lays out the steps measure developers should follow to first identify health care topics or conditions where new measures are needed, and then develop and test specific new measures to fill those identified gaps. According to CMS estimates, it can take 2 years or more to complete all of these steps. As part of this process, CMS encourages entities that develop measures to submit them to the National Quality Forum (NQF), a nonprofit organization that evaluates and endorses measures—that is, determines which measures should be recognized as the best available for a given aspect of care. NQF has endorsed over 700 quality measures. In addition, NQF plays a major role in CMS’s process for determining which measures to use in its Medicare quality programs. Since 2009, NQF has been the sole organization to function under contract to CMS as the consensus-based entity as described by the provisions of sections1890 and 1890A of the Social Security Act (SSA). The consensus-based entity manages the Measure Applications Partnership, which is a formal process for obtaining stakeholder input on proposed new measures for Medicare quality programs, along with other measure endorsement and maintenance activities. CMS also relies on other contractors to conduct analyses or disseminate information related to the development and use of quality measures in its Medicare quality programs. CMS has established strategic objectives for the measures CMS develops or uses in its Medicare quality programs. CMS’s quality measurement strategic objectives have evolved over the last decade as CMS has expanded Medicare quality programs and has collaborated with other organizations that use or develop quality measures, such as private insurance companies. In 2017 CMS announced a revised version of these objectives in its Meaningful Measures Initiative. These eight quality measurement strategic objectives are for CMS to adopt measures that are patient-centered and meaningful to patients, clinicians, and address high-impact measure areas that safeguard public health, are outcome-based where possible, fulfill each program’s statutory requirements, minimize burden for providers, create significant opportunity for improvement, address measure needs for population-based payment through alternative payment models, and align across programs and/or with other payers. In addition, to provide greater specificity for its objective to address high- impact measure areas that safeguard public health, CMS has designated 19 specific meaningful measure areas. See appendix I for the list of these meaningful measure areas and the six broad quality priority areas that they address. CMS’s quality measurement activities are funded through the federal budget and appropriations process. Each appropriation includes language that describes an authorized purpose or purposes for which the funds may be used. Such language may specifically reference certain activities such as quality measurement or could refer to a broad purpose under which activities such as quality measurement may have been authorized. Available funds are first obligated—that is, committed to a specific purpose—and then expended when an actual payment is made. Expenditures can occur one or more fiscal years after the obligation was incurred. Funds that are available in a given fiscal year but not obligated during that year are known as unobligated balances. Unobligated balances can be carried over to the next fiscal year, unless their availability expires under the terms of their appropriation. Most CMS funding that is explicitly appropriated for quality measurement activities is available indefinitely, until obligated and expended. CMS maintains information in its core budget database on the amount of funding for its quality measurement activities, such as when funding for that purpose is specifically authorized by appropriations. However, CMS’s database does not capture all of the funding the agency has obligated that pays for quality measurement activities or the extent to which this funding has supported CMS’s quality measurement strategic objectives. Our review of CMS’s quality measurement funding information also shows that CMS maintains a substantial amount of unobligated balances—funding that CMS has not yet used and remains available—for quality measurement activities. CMS officials report that the agency records funding information for its quality measurement activities in its core budget database, HIGLAS. CMS has information on quality measurement funding primarily when the appropriation is specifically authorized for that purpose. CMS officials identified eight appropriations that specifically designate funding for Medicare quality measurement activities over the 10-year period we reviewed (fiscal years 2009-2018). These include five appropriations that have funded the consensus-based entity established under sections 1890 and 1890A of the SSA, to carry out various activities under contract with CMS in accordance with those provisions. CMS officials identified another three appropriations that focused on more discrete aspects of quality measurement, such as developing new quality measures for clinicians. From fiscal years 2009 through 2018, a total of $429.9 million was authorized for these eight appropriations (see table 1). In addition, CMS officials identified some funding used—that is, obligated—for quality measurement activities, from appropriations authorized for more general purposes. They obtained information on such usage from HIGLAS based on the presence of labels, such as “quality measure development,” in the project code and project description data fields in HIGLAS. According to CMS officials, these data fields provide the most detailed categorization of activities in HIGLAS. Table 2 shows the specific project codes and project descriptions used in HIGLAS to characterize use of quality measurement funding in fiscal year 2018. These obligations are from both appropriations that specifically authorize quality measurement activities and also from general appropriations whose authorized purposes do not specifically mention quality measurement activities. As shown in table 2, the project codes and their descriptions used in HIGLAS provide high-level information that largely matches the information known from the appropriation authorizing such use. Our review of the funding information in HIGLAS found that the data do not capture the total amount of funding CMS has obligated that pays for quality measurement activities. As we have noted, CMS officials identified funding obligated for quality measurement activities in HIGLAS either because 1) the funding came from appropriations specifically designated for quality measurement purposes, or 2) the funding came from appropriations for more general purposes but had specific HIGLAS project codes to identify its use for quality measurement activities. However, CMS officials told us that they thought there were additional quality measurement activities funded from appropriations for general purposes that could not be identified by project codes in HIGLAS. As a result, they could not determine from HIGLAS what amount of these funds paid for quality measurement activities as opposed to other activities. CMS officials stated that while they do not have information on the amount of this unidentified quality measurement funding, they estimated that it was less than the amount of quality measurement funding identified in HIGLAS. Furthermore, CMS’s funding information in HIGLAS also is not sufficiently detailed to show the extent to which the funding was used for activities that support CMS’s eight quality measurement strategic objectives. While some HIGLAS project descriptions—like “Hospital Outcome Measures”— correspond with one of these objectives, as shown in table 2 most do not. In addition, the documents that CMS uses to plan and monitor spending for quality measurement activities generally do not include information showing how much funding CMS has obligated for activities related to CMS’s quality measurement strategic objectives. CMS officials stated that they considered it unduly burdensome to attempt to use HIGLAS to track quality measurement funding according to their strategic objectives. First, they said that quality measurement activities overall constitute a small portion of the funding recorded in HIGLAS. In addition, officials noted that CMS’s strategic objectives change over time. Finally, CMS officials stated their belief that all of CMS’s quality measure activities help to address the agency’s objectives. As a result, CMS cannot determine how its specific funding for quality measurement activities addresses each of its quality measurement strategic objectives and how possible changes in its funding allocations among those activities could help to promote its objectives more effectively. Federal standards for internal control call for agencies to use complete and accurate information and to identify types or categories of information that enable the agency to achieve its objectives. Without more complete information on the total amount of funding obligated to quality measurement activities, CMS officials cannot accurately assess the magnitude of resources they have provided for quality measurement. In addition, even if CMS quality measure activities generally address one or another of its strategic objectives, having information on the extent of funding for each quality measurement strategic objective could help CMS officials assess the amount of funding each of the agency’s priorities is receiving. Doing so would enable CMS officials to make adjustments in accordance with their objectives. While collecting more complete and detailed information on funding for quality measurement activities in HIGLAS—or using some other method that CMS determines is feasible—would require additional effort, CMS could realize corresponding benefits. CMS officials told us that at present, when they need to obtain a higher level of detail about funding for quality measurement activities, they do not use HIGLAS and instead typically conduct a manual review of any available underlying documentation, such as documents related to individual contracts. For example, in order to respond to a statutory requirement to report on its spending to develop certain quality measures for physicians, CMS officials told us they needed to review a set of individual contracts associated with those measures. CMS officials noted that this process is often laborious and that the content of available documents may not enable them to obtain all the desired funding information for the specific quality measurement activities in question. Collecting more information routinely about funding for quality measurement activities has the potential to make such manual reviews of documents less necessary and burdensome. The limitations in CMS’s information on funding for quality measurement activities have implications for CMS’s ability to communicate information outside the agency. As required by the Congress, CMS issued its first annual report on quality measurement funding in March 2019. In this report, CMS itemized information on such funding into four broad categories: “Duties of the consensus-based entity,” “Dissemination of quality measures,” “Program assessment and review,” and “Program oversight and design.” CMS’s report listed a number of more specific activities within these categories without providing the amount of funding it allocated for each of the described activities. More detailed funding information could help the Congress to better understand how CMS is using appropriations for quality measurement, and could assist with effective oversight of these activities. Internal control standards call for agencies to consider the needs and expectations of external users, such as Congress, when collecting and communicating information. Our review of the funding information CMS provided determined that the agency has maintained substantial unobligated balances related to its quality measurement activities from fiscal years 2010 through 2018. Unobligated balances represent funding that CMS did not use in the year it was appropriated, and that remains available for use in future years. All but one of the eight appropriations that specifically authorize spending for quality measurement activities are available indefinitely. Five of these appropriations funded quality measurement activities under sections 1890 and 1890A of the SSA. In the case of these five appropriations, with the exception of fiscal year 2009, CMS had unobligated balances each year that were larger than or similar to the total amount the agency had obligated from those appropriations that year (see figure 1). Figure 1 also shows three other appropriations more narrowly focused on developing new measures for clinicians and post-acute care providers under Medicare (appropriated by MACRA section 102 and the IMPACT Act sections 2a and 2d). Since 2015, unobligated balances for these appropriations also generally exceeded annual obligations. See appendix II for more detailed information. CMS officials stated that unobligated balances reflect broader spending decisions—for quality measurement as well as other activities—the agency makes to meet its strategic objectives and any related legislative requirements. CMS officials said that in general, they chose to use the available quality measurement funds conservatively to ensure there were no gaps in funding to carry out their statutory responsibilities, in view of uncertainty about the availability and timing of funding in future years. They also said that they took into account the total amount of appropriated funds—including unobligated balances—in developing the scope and duration of quality measurement activities. The officials noted that it often takes more than one year to implement these activities, in order to gather information, select contractors, or solicit and award grant applications. Regarding the level of unobligated balances to be carried over from one fiscal year to the next, CMS officials told us that they work to obligate all appropriations in accordance with statutory requirements, and do not have thresholds for maximum unobligated balances. Maintaining large unobligated balances means that CMS is retaining funds for future quality measurement activities rather than using them for current quality measurement activities. One example of how such choices can affect the scope and timing of CMS’s quality measurement activities was the outcome of a CMS competition for cooperative agreements, announced in March 2018, to develop new clinician quality measures to address identified measurement gaps. Drawing on funds from the appropriation dedicated to developing, improving, updating, or expanding new clinician quality measures (MACRA 102) that were available for use until 2022, CMS set a maximum amount for the awards of $30 million over three years. CMS officials determined that the $30 million ceiling meant that there was adequate funding for seven awardees, while CMS indicated that additional applicants scored well on CMS’s selection criteria and addressed areas of need. For fiscal year 2018, MACRA 102 had an unobligated balance of $42 million, with an additional $15 million appropriation in place for fiscal year 2019. As of May 23, 2019, CMS officials told us that they had not announced new competitions to develop clinician quality measures. CMS takes different approaches in deciding which Medicare quality measures to use in its programs, which to remove, and which new measures to develop. However, CMS lacks procedures to ensure that these decisions are consistent with its quality measurement strategic objectives, and CMS has not yet developed or implemented performance indicators to evaluate its overall progress toward achieving these objectives. For selecting measures to be used in its Medicare quality programs, CMS has an annual process, as defined by the Patient Protection and Affordable Care Act. CMS makes a number of decisions that influence measure selection throughout the process. Each year CMS asks measure developers to submit candidate quality measures to CMS for potential selection. CMS makes preliminary decisions on which of these measures to use in its quality programs, and it publishes this selection of measures in its annual Measures under Consideration list (MUC). The MUC list then undergoes public review by multiple stakeholders. After this review, CMS chooses which measures to include in the formal rulemaking processes that ultimately determine which measures are added to its quality programs. See table 3. To make decisions on which measures to include in the MUC list, CMS officials review the submissions. According to CMS, officials from each Medicare quality program, referred to as quality program leads, separately review each measure submitted for use in that program. CMS officials told us that as necessary, they consult with technical experts and with other CMS or Department of Health and Human Services (HHS) officials. According to CMS officials, the program leads make recommendations to higher level officials, such as division directors, on whether CMS should accept or reject each measure. CMS internal guidance outlines factors that, among other things, officials should consider. Some of these factors reflect the strategic objectives laid out in the Meaningful Measures Initiative, and the guidance also indicates that officials may consider additional factors in their decision-making. CMS officials told us that, when making measure selection decisions, program teams are given the flexibility to develop criteria that best suits their programs’ needs, noting that some programs are intended to address a broad range of areas, such as the Inpatient Quality Reporting Program, while others have a more limited focus, such as the Hospital Readmissions Reduction Program. CMS officials told us that the director of CMS’s Center for Clinical Standards and Quality, which is responsible for quality measurement, makes the final measure selection decisions and, in doing so, generally accepts the recommendations of the program teams. Our analysis of CMS’s quality measures indicates that the number of candidate quality measures submitted to CMS for the MUC list has decreased from 335 measures in 2014 to 67 in 2018. CMS officials told us the decline in the number of candidate measures submitted reflected CMS efforts to more clearly define a targeted set of quality measurement priorities for measure developers and to reduce provider reporting burden. Minimizing provider burden is one of CMS’s strategic objectives, and, according to CMS officials, it represents a priority communicated by the CMS administrator. For more information about CMS’s measure selection decisions for its annual MUC list in 2014 through 2018, see appendix III. CMS officials also make decisions annually about which existing measures CMS will remove from its Medicare quality programs. According to CMS officials, the process for deciding which measures to remove is an ongoing, iterative process, and discussions on which measures to remove generally occur in parallel with discussions for selecting measures, with discussions on both measure selection and removal coming to a conclusion in the drafting of the annual proposed and final rules for each program. For measures that are being used in its quality programs, CMS relies on measure developers to monitor the performance of their measures based on principles defined in CMS’s Blueprint. According to the Blueprint, information from developers’ monitoring efforts, including recommendations from technical experts, should be conveyed to and evaluated by CMS officials. CMS officials told us that their decisions to remove measures often take into account the recommendations made by technical experts. In addition, CMS has promulgated through federal rulemaking eight factors for determining whether to remove existing measures from its Medicare quality programs, some of which reflect its quality measurement strategic objectives. CMS officials also said that in deciding to remove measures from CMS quality programs in 2018 they, in part, considered an assessment of the costs of reporting measures relative to the benefit of continued use of the measures. CMS decisions to remove measures have been included in notices of proposed rulemaking in the Federal Register, which allows for public comment and further consideration before issuance of final rules to that effect. In addition to making decisions on the selection and removal of measures, CMS officials also make decisions regarding which new measures to develop. Our review of CMS contract documents, including task orders, indicates that CMS typically awards multiple year contracts to conduct ongoing assessments of quality measures and to develop measures for specific Medicare quality programs, such as inpatient psychiatric facilities or post-acute care providers. Those task orders often call on contractors to convene technical expert panels and conduct additional analyses to assess what measures are currently available for use and what gaps exist in available measures. CMS officials told us they review these reports and provide informal feedback to the contractors. CMS also establishes parameters that guide these efforts. For example, in its 2016 Measure Development Plan for Medicare’s new physician payment system, after soliciting public input, CMS designated six medical specialty areas in which to focus its measure development efforts, and subsequently added five more specialties on which to focus the work of its contractors. For more information about outside entities that perform quality measurement activities under contract with CMS and the efforts CMS has taken to coordinate these activities across its contractors, see appendix IV. CMS has taken some steps that provide opportunities for CMS officials to consider how quality measures may help address the agency’s quality measurement strategic objectives. CMS officials said that in 2018 they began using the Measure Review Template, a spreadsheet used to consolidate information on quality measures submitted to CMS by measure developers. CMS officials told us that they use the spreadsheet to inform their discussions, such as by considering how measures are distributed across the 19 meaningful measure high-impact areas. CMS is also developing another tool, the Quality Measure Index, that is intended to provide a standard methodology to score measures on dimensions that include several of CMS’s eight quality measurement strategic objectives. In addition, CMS officials told us that on occasion they have made limited assessments across measures concerning specific strategic objectives. CMS officials told us that these limited assessments across measures are generally performed when a measure submitted for use in its Medicare quality programs is closely related to another measure, which affects the CMS objective to increase measure alignment. In addition, they said they have identified a few indicators that they use to continuously assess their decision-making process, such as the percentage of outcome measures. CMS also documents some information about its quality measurement decisions. For example, the agency announces its final selection of quality measures to be added to and removed from its Medicare quality programs in the annual federal proposed and final rules for each of those programs. The rationale for selecting each measure is provided as a summary of the peer-reviewed evidence of the impact that use of the measure will have on clinical care. In addition, CMS maintains an internal tracking system, which assembles the information that measure developers provide about the measures they submit to CMS. This system includes some information related to CMS’s quality measurement strategic objectives, such as the meaningful measures high-impact area the measure is intended to address. While these steps provide some information about the linkages between certain quality measures and some of CMS’s quality measurement strategic objectives, CMS lacks procedures to ensure systematic assessment of each quality measure against each of its eight quality measurement strategic objectives. For example, while CMS has implemented the Measure Review Template to consolidate some information on measures, the template does not provide procedures for systematically assessing how each measure will help CMS achieve all eight of its quality measurement strategic objectives. The Quality Measure Index—currently under development—has the potential to be used in a systematic assessment of each measure, but according to CMS officials, as of March 2019 the agency had not yet determined how it planned to use this tool once its testing was complete. Furthermore, CMS lacks procedures to ensure a systematic assessment of whether the collective set of measures it decides to develop or use will help CMS achieve each of the objectives, which could help determine the extent to which each of the objectives is being effectively addressed. The limited assessments across measures that CMS officials said they perform do not consider whether each of CMS’s objectives is being addressed. For example, one of CMS’s eight quality measurement strategic objectives directs CMS to address 19 high-impact measure areas. CMS officials told us that, for each quality program, they look at whether measures generally address the high-impact measure areas, but gaps in these areas remain to be filled. In 2018, there were no measures used in CMS quality programs that addressed the high-impact area “equity of care” and 13 of 17 Medicare quality programs had no measures that addressed the “community engagement” area. Measure developers did not submit measures to CMS that addressed these areas, and CMS did not identify specific initiatives to address them. CMS officials told us, however, that CMS supports discussions of key methodological considerations for collecting and analyzing measure data that could help enable future development of these measures. Last, CMS lacks procedures for documenting the consistent application of those systematic assessments. Federal internal control standards indicate the importance of documenting decisions to support achieving agency objectives. Specifically, CMS does not document, either in its public reporting or internal tracking system, how each measure it decides to use is expected to promote each of its eight quality measurement strategic objectives. For decisions on developing new measures, the agency records less information. For example, CMS does not maintain a consolidated list of decisions to initiate the development of new quality measures across the various Medicare quality programs. CMS officials also told us that they generally do not maintain documentation of discussions on how or why they selected one measure for development over another. If CMS develops procedures to consider the effect of each of its quality measurement decisions on each of its quality measurement strategic objectives, then documentation of these procedures would help to show that they are implemented consistently. Federal standards for internal control state that management should design and implement internal control activities, such as tools and documentation of decisions, to support the agency in achieving its objectives. Without procedures that ensure that its quality measures fully address its strategic objectives, CMS increases the risk that the measures it decides to develop and use will not help the agency achieve its quality measurement strategic objectives as effectively as possible. CMS has not developed and implemented performance indicators that would be needed to determine if it is making progress in meeting its quality measurement strategic objectives. Establishing these indicators and using them to evaluate its progress towards meeting each of its quality measurement strategic objectives would enable CMS to determine whether its quality measurement efforts are sufficient or whether changes in these efforts are needed. According to federal internal control standards, after agencies establish objectives, they should establish a set of performance indicators and use them to assess their effectiveness in achieving their objectives and identify improvements in their work, as needed. However, CMS has not established performance indicators for its strategic objectives that would provide a basis for determining its progress towards achieving these objectives. Such performance indicators would relate to each of CMS’s quality measurement strategic objectives and provide information on interim progress toward achieving these objectives. For example, CMS could establish one or more indicators of its progress toward addressing the 19 high-impact measure areas that safeguard public health, and an indicator of providers’ reporting burden for quality measurement to see if it showed an overall reduction. CMS officials told us that they assess the impact of the agency’s quality measurement activities by reviewing changes over time in health care providers’ reported performance on selected quality measures. However, these measures are for providers’ quality of care, and are not indicators designed to determine the agency’s progress in achieving its eight strategic objectives for quality measurement. Specifically, CMS has completed the National Impact Assessment of Quality Measures report every 3 years since 2012. These reports focus on trends in the performance of health care providers on a number of specific quality measures. Such analyses do not evaluate CMS’s performance in developing and choosing to use measures that promote its quality measurement strategic objectives. CMS has convened the Meaningful Measurement and Improvement Affinity Group, a workgroup of CMS officials involved in quality measurement. This workgroup’s stated mission is to champion the Meaningful Measures Initiative and facilitate its implementation across the agency. CMS officials told us that the workgroup has begun to discuss potential ways to evaluate the agency’s progress in achieving the eight strategic objectives laid out in the Meaningful Measures Initiative. However, the information CMS officials provided on the workgroup’s activities, as of March 2019, indicated that the group had not yet determined how to gauge such progress, such as by establishing performance indicators. CMS plays a leading role in the process of developing new quality measures and selecting measures for use in its various quality programs in Medicare. These programs in turn affect the quality of care the program’s beneficiaries receive. However, CMS lacks complete information on the amount of resources it has obligated for its quality measurement activities and how its allocation of those resources relates to its quality measurement strategic objectives. The agency also lacks procedures to ensure that the decisions it makes to develop and use measures for its quality programs are consistent with those objectives. Finally, CMS has not developed and implemented performance indicators to evaluate its progress towards achieving these objectives. Taken together, these issues limit CMS’s ability to determine whether its allocation of resources and quality measurement decisions are optimal or whether changes are needed in its approach. We are making the following three recommendations to CMS: The Administrator of CMS should, to the extent feasible, maintain more complete information on both the total amount of funding allocated for quality measurement activities and the extent to which this funding supports each of its quality measurement strategic objectives. (Recommendation 1) The Administrator of CMS should develop and implement procedures to systematically assess the measures it is considering developing, using, or removing in terms of their impact on achieving CMS’s strategic objectives and document its compliance with those procedures. (Recommendation 2) The Administrator of CMS should develop and use a set of performance indicators to evaluate the agency’s progress towards achieving its quality measurement strategic objectives. (Recommendation 3) We provided a draft of this report to HHS for review and comment. In its written comments, which are reproduced in appendix V, HHS concurred with our recommendations. Regarding our first recommendation, HHS stated that it has undertaken a review of its fiscal accountability processes for its quality improvement activities and is implementing more granular tracking of funding specific to quality measurement to the extent it is feasible. Regarding our second recommendation, HHS stated that it will determine what steps may be needed to further document how its measure decisions impact the achievement of CMS’s quality measurement strategic objectives. HHS’s comments did not address the need to develop and implement procedures for systematically assessing measures against the strategic objectives, as we recommended. Regarding our third recommendation, HHS stated it would consider how best to evaluate its progress in meeting its quality measurement strategic objectives. In addition, HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, the Administrator of the Centers for Medicare & Medicaid Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or farbj@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. As part of its Meaningful Measures Initiative, the Centers for Medicare & Medicaid Services (CMS) identified 19 meaningful measure areas to specify its priorities under its quality measurement strategic objective to address high-impact measure areas that safeguard public health. The 19 areas are linked to six broader health care quality priorities previously identified in the 2011 National Strategy for Quality Improvement in Health Care. See table 4. The Centers for Medicare & Medicaid Services (CMS) has identified five separate appropriations that for various fiscal years have funded the activities assigned to the consensus-based entity (currently the National Quality Forum), along with certain other quality measurement activities, as described in sections 1890 and 1890A of the Social Security Act. See table 5. Three additional appropriations focus on specific Medicare quality measurement activities, such as post-acute care measures. See table 6. Tables 7 to 12 below present descriptive information that the Centers for Medicare & Medicaid Services (CMS) collects through its issue tracking system on the measures submitted to CMS by measures developers for potential use in CMS’s Medicare quality programs. The Centers for Medicare & Medicaid Services (CMS) has used the majority of its Medicare quality measurement funding for activities conducted by outside organizations under contract with CMS. Between fiscal years 2009 through 2018, the amount of obligations to contracted organizations increased from $10 million to nearly $55 million. See table 13. The total amount of funds obligated to each contractor in fiscal years 2009 through 2018 to perform Medicare quality measurement activities varied, ranging from $1,000 to $139,397,410. For fiscal years 2009 through 2018, 91 percent of funds obligated to contracted organizations for Medicare quality measurement activities went to 12 of 59 contracted organizations. See table 14. CMS has undertaken efforts to coordinate the Medicare quality measurement activities performed by its contractors. For example, CMS works with a CMS contractor, Battelle, to facilitate monthly webinars with its Measure & Instrument Development and Support (MIDS) contractors. The purpose of the webinars is to provide contractors with a forum to discuss each other’s quality measurement activities and to exchange ideas. For more information about CMS’s formal efforts to coordinate the quality measurement activities of its contractors, see table 15. In addition to the contact named above, Will Simerl, Assistant Director; Eric Peterson, Analyst-in-Charge, Jonathan Adams, George Bogart, Krister Friday, Cathy Hamann, Katie Mack, and Dan Ries made key contributions to this report. Also contributing were Vikki Porter and Ethiene Salgado-Rodriguez. Health Care Quality: HHS Should Set Priorities and Comprehensively Plan Its Efforts to Better Align Health Quality Measures. GAO-17-5 (Washington, D.C.: October 13, 2016). Patient Protection and Affordable Care Act: Procedures for Reporting Certain Financial Management Information Should Be Improved. GAO-14-697 (Washington, D.C.: September 22, 2014). Budget Issues: Key Questions to Consider When Evaluating Balances in Federal Accounts. GAO-13-798 (Washington, D.C.: September 30, 2013). Health Care Quality Measurement: HHS Should Address Contractor Performance and Plan for Needed Measures. GAO-12-136 (Washington, D.C.: January 13, 2012). Program Evaluation: Improving the Flow of Information to the Congress. GAO/PEMD-95-1 (Washington, D.C.: January 30, 1995).", "summary": "To encourage greater value in health care, CMS adjusts its Medicare payments to many health care providers based on measures of the quality of care. Therefore, the decisions CMS makes to choose certain quality measures have significant consequences. These decisions may involve selecting specific existing measures for CMS to use, stopping the use of some measures, or identifying new measures to be developed. The Bipartisan Budget Act of 2018 contains a provision for GAO to review CMS's quality measurement activities. For this report, GAO (1) assessed the information CMS maintains on funding of health care quality measurement activities, and (2) described and assessed how CMS makes decisions to develop and to use quality measures. GAO analyzed CMS funding data for 2009 through 2018 and data on CMS quality measurement selections for 2014 through 2018. GAO reviewed CMS documentation related to its decisions on quality measurement and interviewed program and contractor officials. The Centers for Medicare & Medicaid Services (CMS), within the Department of Health and Human Services (HHS), maintains information on the amount of funding for activities to measure the quality of health care provided under Medicare. CMS's information shows it has carried over from each year to the next large amounts of available funding—known as unobligated balances—for quality measurement activities from fiscal years 2010 through 2018 (see figure). CMS officials said they maintained such available funding to ensure there were no gaps in funding for future years. However, CMS officials also told GAO that the information it maintains does not identify all of the funding the agency has obligated for quality measurement activities. Further, it does not identify the extent to which this funding has supported CMS's quality measurement strategic objectives, such as reducing the reporting burden placed on providers by CMS's quality measures. With more complete and detailed information, CMS could better assess how well its funding supports its quality measurement objectives. CMS takes different approaches for deciding which quality measures to develop and to use. However, CMS lacks assurance that the quality measures it chooses address its quality measurement strategic objectives. This is because CMS does not have procedures to ensure systematic assessments of quality measures under consideration against each of its quality measurement strategic objectives, which increases the risk that the quality measures it selects will not help the agency achieve those objectives as effectively as possible. These procedures, such as using a tool or standard methodology to systematically assess each measure under consideration, could help CMS better achieve its objectives. In addition, CMS has not developed or implemented performance indicators for each of its quality measurement strategic objectives. Establishing these indicators and using them to evaluate its progress towards achieving its objectives would enable CMS to determine whether its quality measurement efforts are sufficient or changes are warranted. GAO recommends that CMS (1) maintain more complete and detailed information on its funding for quality measurement activities, (2) establish procedures to systematically assess measures under consideration based on CMS's quality measurement strategic objectives, and (3) develop and use performance indicators to evaluate progress in achieving its objectives. HHS concurred with all three recommendations.", "document_type": "gao"}
{"report": "The JOM program provides supplementary financial assistance, through contracts, to meet the unique and specialized educational needs of eligible American Indian and Alaska Native students. Eligible students, under Interior’s regulations, are generally Indian students age 3 through grade 12 who are either a member of an Indian tribe or at least one- quarter degree Indian blood descendant of a member of an Indian tribe. BIE contracts with tribal organizations, Indian corporations, school districts, and states—which we collectively refer to as JOM contractors as that is the term used by Interior—that administer local JOM programs and disburse funds to schools or other programs providing JOM services. Most JOM funds are distributed through tribal contractors, according to BIE. BIE generally relies on BIA officials to disburse JOM funds, as noted previously (see fig. 1). BIE’s director is generally responsible for directing and managing JOM functions, including establishing policies and procedures, coordinating technical assistance, and approving the disbursement of JOM funds. In 2014, BIE established one centralized position dedicated solely to administering JOM as part of a broader re-structuring initiative and the position has been consistently staffed since 2018. The current JOM program specialist is responsible for planning, developing, administering, and coordinating the JOM program. It is the federal government’s policy to fulfill its trust responsibility for educating Indian children by working with tribes to ensure that education programs are of the highest quality. In 2016, Congress found in the Indian Trust Asset Reform Act that “through treaties, statutes, and historical relationship with Indian tribes, the United States has undertaken a unique trust responsibility to protect and support Indian tribes and Indians.” As further stated in the Act, the fiduciary responsibilities of the United States to Indians are also founded in part on specific commitments made in treaties and agreements, in exchange for which Indians surrendered claims to vast tracts of land. The JOM program is the only federally-funded Indian educational program that allows for student, parent, and community involvement in identifying and meeting the educational needs of American Indian and Alaska Native students, according to the National Johnson-O’Malley Association—a tribally led organization which advocates for JOM programs. The JOM regulations require prospective contractors to formulate an education plan in consultation with an Indian Education Committee, generally made up of parents of American Indian and Alaska Native students, and to submit the plan to BIE. Indian Education Committees have the authority to, among other things, participate fully in planning, developing, implementing, and evaluating their local JOM programs. According to BIE officials, JOM funds can be used to support a wide variety of supplemental education programs. For example, these funds support programs providing Native cultural and language enrichment; academic support; dropout prevention; and the purchase of school supplies, according to BIE (see fig. 2). JOM programs, particularly for students who are not living near tribal land, may be the only way students can access tribal language and cultural programs. According to BIE officials, JOM funding is primarily disbursed to contractors through three different funding mechanisms: self- determination contracts, self-governance compacts, and 477 plans. Most JOM contractors—over 200—are funded through self-determination contracts, according to data provided by BIE. These three funding mechanisms result in different oversight authority for Interior. However, the Johnson-O’Malley Supplemental Indian Education Program Modernization Act (Modernization Act)—enacted on December 31, 2018—requires all JOM contractors to submit annual reports to the Secretary of the Interior with the number of eligible Indian students during the previous fiscal year, an accounting of the amounts expended, and the purposes for which those amounts were expended. BIE officials said some contractors can also be subject to site visits to oversee the program. Under regulations, JOM funds are to be distributed to contractors by a formula that factors in the number of eligible students to be served and average per-student operating costs. Interior conducted its most recent official JOM student count in 1995. As a result, subsequent JOM distributions have been based on the number of students served by contractors in 1995—271,884 students. BIE officials said that the total number of eligible students has increased since 1995, although no official count has been completed. As a result, the funding contractors receive may not reflect changes in the number of students served by contractors. The size of JOM contracts currently ranges from less than $1,000 to nearly $4 million, according to data provided by BIE. The Modernization Act requires BIE to determine the number of eligible students served or potentially served and to complete a rulemaking process to, among other things, modernize program rules. BIE published a preliminary report on its initial determination of eligible students in October 2019 and is continuing to work on finalizing its count of eligible students. Additionally, in response to the Modernization Act, Interior promulgated new final JOM program regulations that became effective March 26, 2020. BIE does not maintain a complete and accurate list of all JOM contractors. BIE officials said JOM funds are disbursed by awarding officials in various BIA offices in different locations, and there is no systematic process to identify and collect information on all the awarded contracts. BIE began efforts to identify all the contractors and the amount of their awards in May 2019 after we asked for this information. As of December 2019, BIE said they identified more than 340 contractors. BIE officials said they have not verified the accuracy and completeness of their current list of contractors. According to federal internal control standards, an agency should have relevant, reliable information to run and control its operations. BIE officials said their current list of JOM contractors is incomplete because some Interior officials responsible for administering and disbursing JOM funds did not respond to their requests for information. In addition, BIE officials said they may not have contacted all the relevant officials within Interior when they developed the list. BIE officials also said they do not know how many contractors may be missing from their list. Further, they said they did not validate the accuracy of the information they received on JOM contractors. Our analysis of BIE’s list of JOM contractors identified data reliability concerns. For example, we found 19 contractors that were listed twice, meaning the total number of contractors provided by BIE contained duplicates and was not an accurate count. BIE officials said that maintaining a complete list of contractors would be very helpful in their efforts to oversee and administer the JOM program, including allowing them to share program information more effectively with all contractors. For example, BIE did not inform all contractors about four consultation sessions it was holding in July 2019 on a proposed rule to change JOM regulations because BIE did not have contact information for all contractors, according to a BIE official. As a result, some contractors may have missed the opportunity to participate in the consultation sessions. Two JOM school contractors we interviewed told us they were not informed by BIE about the consultation sessions that took place in their state. These contractors said they had to create their own networks of contractors to inform each other about JOM-related developments and events because they cannot rely on communication from BIE. In addition, BIE officials said that a complete and accurate list of contractors would help them determine the number of eligible JOM students, as mandated by the Modernization Act. In the two previous efforts to update the count, BIE relied on contractors to submit the number of eligible students they serve. However, BIE officials acknowledged that the last effort to complete a count in 2014 failed, in part, because some contractors never received any communication that BIE was conducting a count. As a result, these contractors never submitted a count of students. Without a systematic process for maintaining a complete and accurate list of contractors, BIE may continue to face barriers administering the program. BIE does not have a process for tracking and monitoring the timeliness of JOM disbursements to contractors. According to BIE officials, the bureau does not establish a target date for disbursing funds to JOM contractors. JOM contractors and BIA and BIE officials we interviewed said the disbursements of JOM funds to some contractors are routinely provided later than expected based on contractors’ past experience. For example, 27 school contractors did not receive a portion of their calendar year 2018 funding until September 2019, according to the BIA official primarily responsible for disbursing the contractors their funds. Further, some of these contractors did not receive any disbursement in the 2019 calendar year until August, months after funds are typically disbursed. Delays in disbursing funds can hinder contractors’ ability to effectively manage their JOM programs and serve students. For example, the three JOM school contractors we interviewed told us that delays in disbursements have negatively affected their ability to plan their JOM activities because they do not know when they will receive their funding. The contractors also said their JOM programs are not as robust as they could be because they regularly delay spending and retain prior disbursements to use in the following year in anticipation of future delays in disbursements. Even with these carry-over funds, contractors said they have had to delay JOM programs for students due to late disbursements of funds, which negatively affect students who depend on JOM for educational support. We were unable to determine the full extent to which Interior disburses JOM funds in a timely manner because BIE and other Interior offices do not track and monitor the timeliness of JOM disbursements to contractors. Federal internal control standards state that agency management should design control activities to achieve objectives and respond to risks, such as by comparing actual performance to planned or expected results and analyzing significant differences. BIE, however, has not established target disbursement dates for contracts and therefore has no standard against which to measure the timeliness of disbursements. Furthermore, BIE does not systematically track the time between receiving its appropriation and the disbursement of contractor funds. BIE officials acknowledged that establishing a target date for disbursing funds to contractors and tracking progress in meeting that date could help ensure funds are provided in a timely manner. In an effort to monitor the disbursement of contractor funds, BIE officials said they have recently started to track the balance of JOM funds at each Education Resource Center. However, they acknowledged that tracking the balance of funds has limited usefulness in tracking the timeliness of disbursements because the information about fund balances does not include whether or not individual contractors have received their funds. BIE officials said having more detailed information on the disbursement of JOM funds would be helpful to ensure funds are provided in a timely manner. In addition, we recently reported that funds associated with self- determination contracts and self-governance compacts for tribes, which include JOM funds, are not always disbursed in a timely manner. We recommended that the Assistant Secretary of Indian Affairs should establish a process to track and monitor the disbursement of funds to tribes that are associated with self-determination contracts and self- governance compacts. However, this recommendation does not address all JOM contractors because non-tribal contractors are not eligible for self-determination contracts or self-governance compacts, and not all tribal contractors receive JOM funds through these mechanisms. Without also establishing a process for tracking and monitoring the disbursement of JOM funds through multiple funding mechanisms, BIE does not have reasonable assurance that funds will be disbursed in a timely manner. BIE has not formally assessed the usefulness of the information it has collected from JOM contractors for over 25 years. One contractor questioned whether the information was useful for the agency’s administration of the program because they never received any feedback or comments from BIE about the information they submitted. The contractor said they spent a considerable amount of time completing their annual report, which totaled over 60 pages and included information and signatures from over 40 different Indian Education Committees that oversee local JOM programs funded by the contract. In addition, all four contractors we interviewed that submitted annual reports said the information requested in the forms could be streamlined. For example, BIE’s annual report form asks each school or project site to report both the “number of eligible students actually served” and “the number of students actually served.” No instructions are provided to distinguish between the two populations, and the contractors said the reported number is identical since students must be eligible to be served by JOM. All four contractors we met with that said they submitted an annual report and renewal application also told us the information collection forms were burdensome to complete. For example, they said the forms were difficult to fill out, in part because they are not compatible with computer word processing programs, and as a result, responses have to be handwritten or completed with a typewriter. All of the forms BIE uses to collect information from contractors subject to JOM reporting requirements are also out of date. For example, the JOM renewal application form expired in 1993, meaning the Office of Management and Budget’s (OMB) approval to collect the information has lapsed. Agencies are required to submit all proposed information collections to OMB for approval. OMB reviews the proposal to assess the need for collecting the information and whether its collection minimizes burden on the public, among other things. Federal internal control standards also state that management should have a process to continually identify information requirements. In a 2015 presentation, BIE officials recognized the need to update the outdated forms to reflect technological developments and reduce the paperwork burden for contractors, but no revisions to the forms have been made. BIE officials said they plan to update the JOM application and reporting documents through the formal OMB review and approval process, but they do not have a timeline for doing so. We have previously reported that outdated forms may not be necessary or useful and may be an unnecessary burden on the public. Until BIE updates the forms, some contractors will continue to struggle to complete them. Further, by assessing the usefulness of the information they are collecting from JOM contractors, BIE may identify opportunities to both collect information that could improve program management and streamline information requests. BIE has not provided or developed training for JOM contractors, according to BIE officials. National Johnson-O’Malley Association officials told us that BIE and BIA used to provide training that was helpful to JOM contractors on topics such as filling out annual reports and applications for JOM contracts, particularly to new staff managing these programs, but they no longer do so. A nonprofit organization for Indian education we interviewed also said JOM contractors need training on a range of issues, including how to complete JOM annual reports and other documentation, and on how to operate following implementation of the Modernization Act. According to the nonprofit organization, regular training on JOM is particularly important because certain aspects of the program, such as conducting annual assessments to determine the learning needs of Indian children served by the program, can be technically challenging. Officials from one tribal contractor we interviewed said the tribe provides its own training to school staff that implement local JOM programs on such topics as how to conduct Indian Education Committee meetings, how to fill out reimbursement claims, and how to organize and maintain financial records for program administrators and parents on Indian Education Committees. The contractor said that BIE training on topics, including how to conduct and how often to hold Indian Education Committee meetings, would be particularly helpful. Another tribal contractor we interviewed, which BIE data identified as receiving among the largest amount of JOM funds of all contractors, said that other contractors they interact with do not have sufficient program knowledge or resources to provide training and could benefit from BIE training. According to a BIE official, a former JOM Program Specialist, the need for training for JOM contractors is particularly important as there is frequent turnover among contractor staff responsible for administering programs. Officials from the nonprofit organization for Indian education also told us that high turnover rates among administrators of local JOM programs necessitates regular training for new staff. They added that more senior staff working on local JOM programs would also benefit from regular training because they may be implementing their programs inefficiently or ineffectively. BIE officials told us they have provided program updates and answered questions at conferences hosted by organizations representing JOM contractors. Not all contractors, however, are able to attend these conferences given their limited resources, according to three contractors we interviewed. Internal controls standards state that management should develop training based on the needs of individuals’ roles. BIE officials acknowledged that developing and providing training is needed, but they told us they are currently focused on other aspects of managing the JOM program and have not prioritized training. For example, the agency has set a goal in its strategic plan to develop a JOM program handbook by July 1, 2020. By providing training, BIE can ensure that contractors have the information they need to better serve their students. BIE has not clearly defined roles and responsibilities or identified the staff necessary for conducting critical JOM functions. According to federal internal control standards, management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. BIE’s lack of defining or identifying roles and responsibilities related to administering contracts, reviewing the appropriateness of contract types, and conducting program oversight is described in the following bullets. Administering contracts. BIE did not identify staff to administer some contracts, which has contributed to some JOM programs affected by these contracts going unfunded. According to BIE and BIA officials, BIE did not assign any staff to administer at least 20 contracts in California, including helping contractors renew their contracts when they expired, typically after 3 years. As a result, these contracts—totaling over $300,000—expired and were not renewed, disrupting JOM services. A BIE official informed us there were lapses in administering these contracts because BIE closed the office responsible for administering them as a result of its reorganization which began in 2014, and never assigned anyone to assume responsibility for the contracts associated with that office. BIE has not assessed whether similar lapses in coverage may have occurred in other states or regions. BIE officials identified the unallocated funds from California in September 2019. In October 2019, BIE officials began efforts to identify and contact officials responsible for all the JOM programs whose contracts lapsed in California due to gaps in BIE’s administration of the program and began the process to start new JOM programs in the future. However, without identifying staff to administer all JOM contracts, problems with renewing and awarding contracts may persist. Reviewing the appropriateness of contract types. Interior’s Office of the Solicitor does not have a role in reviewing the issuance of new JOM contracts, according to a senior attorney in that office. The Office of the Solicitor’s lack of a role in reviewing JOM contracts increases the risk that contracts are not used appropriately. For example, we found that BIE has been using self-determination contracts to disburse JOM funds to non-tribal contractors, which is not authorized by the Indian Self Determination and Education Assistance Act. Under the Act, only Indian tribes and tribal organizations are eligible to enter into self-determination contracts; these contracts may not be used for non-tribal entities, such as school districts and states. The use of self- determination contracts for contractors that are not eligible to receive them can result in costs to the government. Self-determination contracts include provisions that would not otherwise be included in non-tribal JOM contracts, according to a senior attorney in the Office of the Solicitor. For example, self-determination contracts may include contract support costs and extend the Federal Tort Claims Act to tribal government employees administering the federal program(s) under these contracts. Therefore, school contractors that were disbursed JOM funds through self-determination contracts may have received contract support costs and legal protections they would not have been eligible to receive, according to the senior attorney. BIE officials told us that they have not determined how long self- determination contracts have been used to disburse JOM funds to non-tribal entities, how many non-tribal contractors were awarded these contracts, or whether the government has incurred costs as a result of using the wrong types of contracts. They said this information will be difficult to obtain because it is not systematically collected. After we found that BIE was using self-determination contracts to disburse JOM funds to school contractors, a senior attorney in the Office of the Solicitor said that her office would provide assistance as requested to BIE in transitioning these contracts to appropriate contracts. By systematically including the Office of the Solicitor in the process for reviewing JOM contracts, BIE can ensure that its contracts are the appropriate type and can minimize the risk of future inappropriate costs to the federal government. Conducting JOM oversight activities. BIE has not defined the roles and responsibilities related to overseeing JOM programs or identified staff dedicated to this function. For example, BIE has not identified staff at Education Resource Centers or other BIE offices with the capacity to conduct site visits and review JOM annual reports submitted by contractors. As a result, the bureau’s oversight of JOM contractors is done on an ad-hoc basis and sometimes not done at all, according to BIE officials. For example, in an internal memo addressed to BIE’s Director, a senior BIE official said that because the bureau has not identified staff with the capacity to conduct site visits, most Education Resource Centers have not conducted any site visits in at least 5 years. Officials from one tribal JOM contractor that said it is subject to BIE oversight told us that BIE has not conducted a site visit of their program in 10 years. They noted that BIE’s past site visits resulted in recommendations that improved their program activities and procedures and changed how they defined student eligibility. In addition, the head of an Education Resource Center said that JOM oversight activities are collateral duties that his staff do not have time to fulfill. Further, the responsibilities of officials who are charged with overseeing JOM programs have not been clearly defined. For example, BIE has not defined the responsibilities related to conducting site visits, such as what aspects of the program should be reviewed and which contractors should be selected for site visits. This lack of clearly defined responsibilities has resulted in inconsistencies in how officials are conducting oversight activities and potential gaps in coverage of contractors that are subject to oversight. BIE’s lack of oversight may also increase the risk of misuse and abuse of JOM funds. According to Interior’s Office of Inspector General, there have been three identified cases of theft related to the JOM program that occurred between 2004 and 2010. For example, a program coordinator of a JOM contract stole program funds as part of an embezzlement fraud scheme and was ordered to pay nearly $36,000 in restitution. By identifying staff who have the capacity to carry out oversight activities and clearly defining related responsibilities such as conducting site visits and reviewing JOM annual reports, BIE could provide support to contractors in improving their program activities and procedures and reduce the risk of potential fraud and abuse of JOM funds. Senior BIE officials acknowledged that they have not identified the staff necessary for conducting these critical JOM functions and, in November 2019, the Director of BIE approved hiring three additional JOM specialists. The core responsibilities of the new specialist positions, according to a knowledgeable BIE official, will be to support the administration of contracts, oversee contractors, and provide technical assistance. However, the exact roles and responsibilities for the new employees and the extent to which BIE staff in the Education Resource Centers will continue their role in providing programmatic support have not yet been determined. An official knowledgeable about the new JOM specialist positions added that defining the specific roles and responsibilities for these positions will be an iterative process in which BIE will assess the new staffs’ capacity to assume all the JOM responsibilities that are currently assigned to other staff. Until all the roles and responsibilities related to JOM program management have been identified and clearly defined, challenges in administering contracts, reviewing the appropriateness of contract types, and overseeing the program may persist. American Indian and Alaska Native students have unique educational and cultural needs, which can include learning Native languages, cultures, and histories, and obtaining additional academic support. The JOM program is intended to address these needs that may not otherwise be provided through the public school system. BIE plays a critical role in administering the JOM program, which is central to the bureau’s mission of providing Indian students quality education opportunities starting in early childhood in accordance with a tribe’s needs for cultural and economic well-being. However, BIE lacks key JOM program information necessary for effective oversight, including complete information on which contractors are participating in JOM. BIE also has not assessed the usefulness of the information it collects from contractors, and relies on outdated forms to collect data. Without improved program data, BIE cannot effectively oversee the program. In addition, BIE does not provide training for JOM contractors. This lack of training may result in contractors misinterpreting JOM regulations and managing their programs inconsistently. Further, BIE has not clearly defined the roles and responsibilities of staff involved in administering the JOM program, which has resulted in gaps in program management and oversight. Until staff roles and responsibilities are clearly defined and identified, gaps in managing and overseeing the program may persist, resulting in an increased risk of potential misuse or abuse of JOM funds. Without taking steps to improve the management and oversight of the JOM program in these key areas, BIE cannot ensure that the program is truly serving the educational needs of eligible American Indian and Alaska Native students. We are making the following five recommendations to Interior: The Director of the Bureau of Indian Education should develop a systematic process for identifying JOM contractors and maintaining an accurate and complete list of contractors and other relevant information about contractors, such as the amount of JOM funds they receive and their current points of contact. (Recommendation 1) The Director of the Bureau of Indian Education, in coordination with the Bureau of Indian Affairs as needed, should establish a process to track and monitor the timeliness of JOM disbursements to non-tribal contractors, including identifying a target date for disbursing funds to these contractors. (Recommendation 2) The Director of the Bureau of Indian Education should develop a timeline to assess the usefulness of the information they are collecting from JOM contractors and update JOM information collection forms, including converting them to an electronic format to reduce the burden on contractors to complete them. (Recommendation 3) The Director of the Bureau of Indian Education should develop and provide training to contractors on administering the JOM program. (Recommendation 4) The Director of the Bureau of Indian Education should clearly define the roles and responsibilities and identify the staff necessary for conducting critical JOM functions, including administering contracts, reviewing the appropriateness of contract types, and overseeing those contractors that are subject to BIE oversight. (Recommendation 5) We provided a draft of this report to Interior for review and comment. We also provided relevant report sections to and requested technical comments from the National Indian Education Association and the National Johnson-O’Malley Association. In its comments reproduced in appendix I, Interior concurred with our five recommendations and described actions BIE and BIA plan to take to address them. In our draft report, we recommended that BIE needs to clearly define the roles and responsibilities and identify the staff necessary for conducting technical assistance, among other critical JOM functions. We removed reference to technical assistance from our report because, after we provided our draft report, Interior promulgated new, final JOM program regulations that include a new process for requesting and providing technical assistance. We did not receive any comments from the nonprofit organizations. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Interior, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0534 or emreyarrasm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Beth Sirois (Assistant Director), Brian Schwartz (Analyst-in-Charge), Ben DeYoung, and Alex Galuten made key contributions to this report. Additional assistance was provided by Edward Bodine, Gina M. Hoover, Thomas M. James, Grant M. Mallie, Sheila R. McCoy, Anna Maria Ortiz, Jeanette M. Soares, Joy K. Solmonson, Curtia O. Taylor, and William T. Woods.", "summary": "American Indian and Alaska Native students enrolled in public schools have performed consistently below other students on national assessments from 2005-2019. The JOM program provides academic and cultural supports, through contracts, to meet the specialized and unique educational needs of American Indian and Alaska Native students enrolled in public schools and select private schools. In fiscal year 2019, Interior allocated about $23 million for the JOM program, according to Interior's budget documentation. GAO was asked to review issues related to Interior's JOM program, administered by BIE. This report examines the extent to which BIE (1) has key program information, (2) provides training to JOM contractors, and (3) clearly defines and identifies JOM roles and responsibilities. GAO reviewed relevant federal laws, regulations, and both BIE and JOM contractor documents; analyzed existing data and information on JOM; and interviewed agency officials, five JOM contractors of different types, and two nonprofit organizations selected for their knowledge of the JOM program. The Department of the Interior's (Interior) Bureau of Indian Education (BIE) does not have key information to manage the Johnson-O'Malley (JOM) program which provides supplemental education services to meet the specialized and unique needs of American Indian and Alaska Native students. For example, BIE does not maintain a complete and accurate list of all its JOM contractors, who provide services including targeted academic supports, Native language classes, and cultural activities. In May 2019, BIE began to identify all the contractors, but officials acknowledged that their list is still incomplete, and GAO found problems with the list, such as duplicate entries. Federal internal control standards state that an agency should have relevant, reliable information to run its operations. Maintaining a complete list of contractors would improve BIE's administration of the JOM program. BIE does not provide any training for JOM contractors. For example, BIE does not provide training to contractors on how to effectively manage their JOM programs or meet program requirements. By providing training for contractors, BIE could ensure that contractors understand the program and are equipped to provide services to meet the educational needs of their students. In addition, BIE has not clearly defined the roles and responsibilities or identified the staff needed to effectively administer the JOM program (see figure). For example, when BIE closed a field office in California, staff were not identified to administer the office's contracts, including helping contractors renew their contracts when they expired. Also, BIE has not identified a role for Interior's attorneys in reviewing the contracts and some contractors have types of contracts for which they are not eligible. Further, BIE has not identified staff to conduct consistent program oversight, which is important to mitigating the risk of misuse and abuse of JOM funds. Until all JOM roles and responsibilities have been defined and identified, challenges may persist. GAO is making five recommendations, including that the Director of BIE should maintain an accurate and complete list of JOM contractors, develop JOM training, and clearly define roles and responsibilities and identify staff for carrying out JOM functions. Interior agreed with the recommendations.", "document_type": "gao"}
{"report": "DOD officials identified three key department-wide initiatives that include a number of cybersecurity practices aimed at improving cyber hygiene: the DC3I, the CDIP, and the Cyber Awareness Challenge training. These efforts recognize the importance of command leadership, best practices for DOD network users, and technical countermeasures against cybersecurity threats. DC3I. In September 2015, the Secretary of Defense and the Chairman of the Joint Chiefs of Staff signed the DC3I in an effort to transform DOD cybersecurity culture by enabling and reshaping leaders, cyber providers, personnel who perform cyberspace operations, and general users to improve individual human performance and accountability on DOD’s network. The DC3I memorandum identifies 11 tasks assigned to various DOD components to respond to and implement across the department— such as the development of cybersecurity training briefs for DOD leadership, integration of cybersecurity into operational training and exercises, and the development of a resourcing plan to support scheduled inspections of units conducting cyberspace operations. From September 2015 to December 2016, U.S. Cyber Command was initially responsible for ensuring that relevant components implemented the DC3I. In December 2016, the Deputy Secretary of Defense assigned the DOD CIO as the official responsible for ensuring that components implemented the initiative because, in part, the DOD CIO has DOD-wide oversight authority. CDIP. The CDIP is one of seven actions identified in DOD’s Cybersecurity Campaign to prompt commanders and senior leaders to enforce full cybersecurity compliance and accountability across the department. In October 2015, the Deputy Secretary of Defense signed the CDIP to reinforce basic cybersecurity technical requirements identified in policies, directives, and orders as a means to defend DOD information networks, secure DOD data, and mitigate risks to DOD missions. The CDIP memorandum identifies 17 tasks for all commanders and supervisors to implement across the department. These tasks include removing operating system software that no longer receives security updates from vendors, configuring servers consistent with DOD guidance on secure configurations, and addressing vulnerabilities for servers and network infrastructure in a timely manner. Cyber Awareness Challenge Training. This training is intended to help the DOD workforce (including service members, civilians, and contractors) to maintain awareness of known and emerging cybersecurity threats, reinforce best practices to keep information and information systems secure, and ensure that network users stay abreast of changes in DOD cybersecurity policies. DISA develops the training content and periodically updates the training. In addition, the Cyber Workforce Advisory Group that includes officials from the DOD CIO, DISA, and DOD components, solicit input about ways to improve the training and meets annually to approve updates to the Cyber Awareness Challenge. Federal law and a DOD initiative and strategy highlight the important role of leadership in improving cybersecurity culture and performance across the department. For example, the Federal Information Security Modernization Act of 2014 (FISMA) requires agency heads—including the Secretary of Defense—to ensure that senior agency officials provide security for the information and information systems that support the operations and assets under their control. Additionally, the DC3I states that leaders will be held accountable by the chain of command for the cybersecurity performance of their organization and the individuals who comprise it, and for the role cybersecurity performance plays in accomplishing assigned missions. It also states that leaders will set an example and help individuals master appropriate cyber behavior, will take action against those who commit gross negligence or errors of commission, and may use all available means, both legal and administrative, as they deem appropriate. Further, the 2018 DOD Cyber Strategy states that reducing the department’s network attack surface (i.e., the different points in a network where attackers can try to enter or extract information) requires an increase in cybersecurity awareness and accountability across the department. The strategy also states that the department would hold DOD personnel accountable for their cybersecurity practices and choices. The 2019 Cybersecurity Readiness Review, directed by the Secretary of the Navy, describes best practices for effective cybersecurity leadership. These best practices, according to the readiness review, require Navy leaders to be informed on cybersecurity issues facing their organization, engaged in ensuring cybersecurity issues are addressed, and hold their organization accountable for cybersecurity performance. A number of DOD officials and components have key roles and responsibilities for cybersecurity, including the three key cyber hygiene initiatives. For example: Secretary and Deputy Secretary of Defense. FISMA makes the Secretary of Defense responsible for providing information security protections commensurate with the risk and magnitude of harm facing the department. In addition, Executive Order 13800, issued in May 2017, aligns with FISMA by holding agency heads accountable for implementing risk management measures commensurate with the risk and magnitude of the harm that would result from unauthorized access, use, disclosure, disruption, modification, or destruction of IT and data. DOD Chief Information Officer. FISMA requires DOD to develop, document, and implement a program to provide security for information and information systems (commonly referred to as a cybersecurity program) and directs the Secretary of Defense to delegate to the DOD CIO (and military department CIOs) authority to ensure compliance with the law. In addition, the DOD CIO is responsible for overseeing implementation of the three key cyber hygiene initiatives. DOD Component heads. DOD component heads are responsible for ensuring that IT under their purview complies with DOD Instruction 8500.01. In addition, component heads are responsible for ensuring that their network users complete annual security awareness training. DOD Component CIOs. DOD component CIOs are responsible for developing, implementing, maintaining, and enforcing a component cybersecurity program on behalf of their respective component heads. In doing so, component CIOs are responsible for ensuring that their components implement the CDIP tasks. Chairman of the Joint Chiefs of Staff. The Chairman of the Joint Chiefs of Staff is responsible for advising the President and Secretary of Defense on operational policies, responsibilities, and programs. The Chairman also assists the Secretary of Defense in implementing operational responses to cyber threats and ensures cyberspace plans and operations are compatible with other military plans and operations. The staff members who support the Chairman of the Joint Chiefs of Staff are referred to as the Joint Staff, which is comprised of members from all of the military services. U.S. Cyber Command. The Commander of U.S. Cyber Command has the mission to direct, synchronize, and coordinate cyberspace planning and operations to defend and advance national interests in collaboration with domestic and international partners. In addition, the Commander is responsible for, among other things, issuing orders and directives to all DOD components for the execution of global operations aimed at securing and defending the department’s networks. Defense Information Systems Agency. The Director of DISA is responsible for developing, implementing, and managing cybersecurity for the department’s network and works with other components to secure DOD systems. For example, the Director is responsible for developing cybersecurity awareness training for all users on DOD’s network. JFHQ-DODIN. The Commander of JFHQ-DODIN is responsible for, among other things, commanding, controlling, planning, directing, coordinating, integrating, and synchronizing DOD defensive cybersecurity operations. JFHQ-DODIN also performs two types of cyber readiness inspections to ensure DOD units comply with requirements related to network security and to evaluate the ability of units to accurately detect and mitigate vulnerabilities and anomalous activity on DOD’s network. The security of federal cyber assets has been on our High-Risk List since 1997. In September 2018, we issued an update to this high-risk area that identified actions needed to address cybersecurity challenges facing the nation—including improving implementation of government-wide cybersecurity initiatives aimed at securing federal systems and information. We also have identified ensuring the cybersecurity of the nation as one of nine high-risk areas that need especially focused executive and congressional attention. In August 2017, we reported on DOD’s progress in implementing the department’s cyber strategies. We found that DOD had implemented the cybersecurity elements of the DOD Cloud Computing Strategy and had made progress in implementing the 2015 DOD Cyber Strategy and DOD Cybersecurity Campaign, which was comprised of multiple initiatives including the CDIP. However, DOD’s process for monitoring implementation of the DOD Cyber Strategy resulted in the closure of tasks before they were fully implemented. We also found that DOD lacked a timeframe and process for monitoring implementation of the DOD Cybersecurity Campaign objective to transition to commander-driven operational risk assessments for cybersecurity readiness. We recommended that DOD (1) modify criteria for closing tasks as implemented and reevaluate tasks previously determined to be completed to ensure they meet modified criteria and (2) establish a timeframe and monitor implementation of the DOD Cybersecurity Campaign objective to develop cybersecurity readiness assessments to help ensure accountability. DOD partially concurred with both recommendations. As of January 2020, neither recommendation had been implemented. DOD has not fully implemented its three cyber hygiene initiatives. Specifically, (1) the DOD CIO and DOD components have not implemented seven of the 11 DC3I tasks due in fiscal year 2016; (2) DOD has implemented six of 10 CDIP tasks that the DOD CIO oversees and does not know the extent that seven other CDIP tasks are implemented; and (3) DOD did not know the extent to which users for selected components completed the Cyber Awareness Challenge training in 2018 and one component did not use the required training. In addition, the department does not know the extent that cyber hygiene practices to protect its networks from key cyberattack techniques have been implemented. DOD has not implemented seven of the 11 DC3I tasks despite fiscal year 2016 deadlines for each of the tasks being established by the department. In particular, DOD components have implemented four DC3I tasks and have not implemented the seven remaining tasks, as shown in figure 1. As shown above, DOD has implemented four DC3I tasks. For example, DOD CIO implemented a task that requires that office to assess the effect of cyber workforce shortfalls on DOD’s mission and provide recommendations to address these shortfalls (task 10 in figure 1 above). Specifically, in April 2019, DOD CIO provided a plan to the Office of Personnel Management to address cyber workforce shortages by filling vacant positions, enhancing outreach and recruitment, and expanding on hiring authorities. However, DOD has not implemented the remaining seven DC3I tasks. For example: DOD has not fully implemented leadership cybersecurity training briefs (task 1). In April 2016, U.S. Cyber Command developed two training briefs to be used in leadership training. However, as of October 2019, DOD components have not received either training brief, according to DOD officials. In September 2016, U.S. Cyber Command provided the Deputy Secretary of Defense a DC3I status report and informed him that two products were developed to address this task and that they would be disseminated to DOD components. However, as of October 2019, neither U.S. Cyber Command nor the Office of the DOD CIO had disseminated these leadership training briefs across the department, according to DOD officials. In reviewing the training briefs, we found that, if they had been incorporated into DOD leadership training, leaders would have been better positioned to address cybersecurity risks. For example, they may have learned, among other things, how to understand, assess, and interpret cyber-reportable events and incidents and how they affect military operations. DOD has not developed cyber-provider training (task 2). In February 2019, the office of the DOD CIO completed a review of all military and civilian IT positions to identify the work roles of all cyber providers in the department. However, the office has not developed educational and training requirements for cyber providers. DOD CIO officials told us that, consistent with task 2, they are drafting a DOD Manual, Cyber Workforce Qualification and Management Program, which would document educational and training requirements for the work roles for each cyber provider. DOD CIO officials expect to complete the manual around April 2020. DOD has not fully implemented criteria for assessing cybersecurity in operational training and exercises (task 5). In March 2016, the Joint Staff developed criteria for assessing military service and combatant command efforts to integrate cybersecurity into operational training and exercises. For example, the Joint Staff developed a checklist of cybersecurity elements that should be included in cyberspace-related training objectives and assessed during training events. In May 2016, the Vice Chairman of the Joint Chiefs of Staff required that the criteria be used to assess military service and combatant command efforts to integrate cybersecurity into operational training and exercises. In May 2019, Joint Chiefs of Staff officials told us the criteria was not incorporated into the Chairman’s annual training guidance, citing personnel turnover, and that they do not have plans to incorporate the criteria. According to the DC3I, operational and tactical commanders and leaders need to interpret the effect that cyber insecurity may have on the mission and integrate cyber effects into mission planning. If Joint Staff had updated the Chairman of the Joint Chiefs of Staff guidance for operational training, DOD commanders would have had criteria they could use to assess the effect that cyber insecurity may have on military missions. The lack of progress in implementing the tasks occurred, in part, because the DOD CIO did not take steps to ensure that the DC3I tasks were implemented. DOD CIO officials told us they were not aware of their responsibility to oversee implementation of the DC3I. Initially, U.S. Cyber Command was assigned as the entity responsible for overseeing implementation of the DC3I; however, in December 2016, the Deputy Secretary of Defense approved the transition of the DC3I mission lead to the department’s CIO. According to this transition memorandum, the CIO was to leverage existing authorities and departmental efforts to lead and provide oversight of cybersecurity culture and compliance transformation. Additionally, DOD CIO officials told us that the office is focusing its resources on other CIO initiatives, such as implementing the cyber landscape initiative. However, the DC3I included a task (task 11 in figure 1 above) that required an assessment of the resources needed to ensure that DOD implemented the DC3I and this task had not been completed at the time of our review. If DOD CIO does not take appropriate steps to ensure that the DC3I tasks are implemented, the department risks compromising the confidentiality, integrity, and availability of mission-critical information as a result of human error by users on the department’s networks. Since 2015, DOD has implemented six of 10 CDIP tasks that the DOD CIO is to oversee, but has not achieved desired performance targets for the remaining four tasks even though there is a requirement to implement all 10 by the end of fiscal year 2018. In the 2015 CDIP memorandum, the Deputy Secretary of Defense directed DOD components to implement all 17 CDIP tasks for all system users, IT hardware, and IT software to remove preventable vulnerabilities from DOD’s network that could allow adversaries to compromise information and information systems. According to a March 2019 memorandum, the Deputy Secretary of Defense challenged the department to achieve 90 percent implementation of the 10 CDIP tasks overseen by DOD CIO by the end of fiscal year 2018. In table 1, we list the 17 tasks and indicate the 10 tasks that the department’s CIO oversees. The department has achieved its performance targets for six of the 10 CDIP tasks that the DOD CIO oversees. For example, in October 2018 DOD achieved its performance target for one task that requires the department to move all of DOD’s web servers into a DOD “demilitarized zone,” or DMZ, according to DOD’s fiscal year 2018 Federal Information Security Modernization Act report to the director of the Office of Management and Budget. Placing these web servers in a DMZ directs web traffic intended for those servers—including malicious traffic—to systems within perimeter firewalls that screen the traffic before allowing access to organizations networks. By implementing the task and moving 11,000 web servers into the DMZ, DOD has reduced the risk that malicious traffic can reach its web servers. However, the department has not achieved the department-wide goal for the four remaining CDIP tasks overseen by DOD CIO. For example, DOD did not achieve its performance target for a task that required components to ensure they were compliant with endpoint security guidance. DOD CIO officials told us that the remaining four CDIP tasks are challenging for the department to achieve the 90 percent performance target because some DOD components use aging information technology systems and these older systems may not be equipped to implement all CDIP tasks. We have previously reported that legacy systems have operated with known cybersecurity vulnerabilities that are either technically difficult or prohibitively expensive to address. In light of the security risks posed by DOD component legacy systems, we stated that it is imperative that agencies carefully plan for their successful modernization. DOD did not achieve the 90 percent goal for four of the 10 CDIP tasks by the end of fiscal year 2018 due in part to DOD components not developing plans with scheduled completion dates to implement these four tasks, according to DOD officials. DOD CIO officials told us that they had not required DOD components to develop plans with scheduled completion dates for the remaining four CDIP tasks. CIO officials believed that the DOD components would implement the CDIP memorandum since it was signed by the Deputy Secretary of Defense and it required them to report on their progress in implementing the CDIP tasks. While the Deputy Secretary of Defense did require DOD components to implement these four tasks and report on their progress, components have not achieved performance targets. If DOD components do not develop plans with scheduled completion dates to implement the remaining four CDIP tasks, the department may fail to remove preventable, well-known vulnerabilities from its network and may allow adversaries to compromise the confidentiality, integrity, or availability of sensitive information and information systems. DOD does not know the extent to which components have implemented the seven CDIP tasks that the CIO does not oversee because the responsible components have not reported on their progress, according to DOD officials. For example, DOD has not reported on the extent to which components have disabled hyperlinks to websites that users receive in email messages. Disabling hyperlinks in email messages can help to prevent phishing attacks. DISA officials told us that the agency implemented a security protocol that disables these hyperlinks in DISA’s email server. Consequently, DOD components that use DISA’s email service are compliant with this task’s requirement; however, not all DOD components use DISA’s email service and the extent to which other email services comply with this task is unknown. The CDIP memorandum signed by the Deputy Secretary of Defense stated that the department’s progress in implementing all CDIP tasks would be reported. However, the department has not reported on the progress it has made implementing the seven CDIP tasks that the CIO does not oversee in part because the Deputy Secretary of Defense did not identify, in the CDIP memorandum, a component to oversee the implementation of these tasks and report on their progress. According to DOD CIO officials, some of these seven tasks are more tactical and may be more appropriately tracked at echelons below the office of the DOD CIO. For example, one of these seven tasks requires that commanders ensure the physical security of their network infrastructure devices. We agree that lower echelons may more effectively track the progress of some tasks; however, information about the progress that components make implementing these tasks is not reported to the CIO or any other DOD component, according to DOD officials. In addition, DOD CIO officials told us that JFHQ-DODIN collects some information from inspections it performs to verify the extent that inspected units implement technical guidance documents, some of which relate to these seven CDIP tasks. However, according to DOD officials, JFHQ-DODIN does not report this information to the CIO or any other DOD component. In addition, JFHQ-DODIN inspects a sample of DOD units and therefore does not have information about the status of these tasks across the department. For those units that are inspected, no DOD component is aggregating data from these inspections to identify the extent to which these seven tasks are implemented. If the Deputy Secretary of Defense does not identify a DOD component to oversee the implementation of the seven CDIP tasks that DOD CIO does not oversee and report on progress implementing them, the department will have less assurance that cybersecurity vulnerabilities are being addressed in a timely manner and systems are being securely configured. The 16 selected components we included in our sample did not always collect information on the number of users (1) that completed the fiscal year 2018 Cyber Awareness Challenge training, (2) that did not complete the training, and (3) whose network access was revoked for not completing the cyber awareness training. Specifically: Unknown number of users that completed the cyber awareness training. Two of the 16 did not collect information on the number of users that completed the fiscal year 2018 Cyber Awareness Challenge training. In particular, the Army and the Defense Finance and Accounting Service could not provide data on the extent that users had taken the required training in fiscal year 2018. Unknown number of users that did not complete the cyber awareness training. Six of the 16 components did not collect information on the number of users that did not complete the cyber awareness training. In particular, the Navy, Air Force, Marine Corps, U.S. European Command, and the Defense Media Activity did not collect information on the users who did not complete the training in fiscal year 2018. In addition, the Army’s training compliance system did not have records for all Army users in 2018, which limited the Army’s ability to determine if all of its users completed the fiscal year 2018 Cyber Awareness Challenge training. Unknown number of users whose network access had been revoked for not completing the required training. Eight of the 16 components that we contacted did not collect data on the number of users whose network access had been revoked for not completing the required training, as implied by DOD policy. Selected DOD components did not know the extent to which their network users implemented the 2018 Cyber Awareness Challenge training by completing it because the DOD component heads did not ensure that their respective components were accurately monitoring and reporting the necessary information. Navy officials told us that they believed it was not DOD or the military service’s policy for the service headquarters to track whether their network users had completed the training. According to Navy officials, there is also no value for large organizations like the Navy, with over 600,000 users, to track and report these data at the headquarters level. However, DOD policy requires all network users to take the Cyber Awareness Challenge training annually. In addition, DOD policy states that all individuals with network access must complete this training to retain access. NIST also advises that agencies capture training compliance data at an agency level, so data can be used to conduct agency-wide analysis and reporting. Multiple DOD policy and guidance documents—including DOD Manual 8570.01-M, and Chairman of the Joint Chiefs of Staff Instruction 6510.01F—state that the DOD component heads are responsible for ensuring that users complete the Cyber Awareness Challenge training and two of these documents require recording training compliance. For example, according to DOD Manual 8570.01-M, Information Assurance (IA) Workforce Improvement Program, components must document and maintain the status of awareness compliance for each user. Further, service policy and guidance places the responsibility on the DOD component heads or senior-level leaders at the headquarters’ level for ensuring that cybersecurity training is completed and documented. For example, Secretary of Navy Instruction 5239.3C, Department of Navy Cybersecurity Policy (May 2, 2016), states that the Chief of Naval Operations and the Commandant of the U.S. Marine Corps shall ensure all authorized users of Department of Navy information systems and networks receive initial cybersecurity awareness orientation as a condition of access and, thereafter, complete annual refresher training, monitor and report workforce cybersecurity training and maintain supporting records. Similarly, Army Regulation 25-2, Army Cybersecurity (Apr. 4, 2019), states that the Deputy Chief of Staff, G3/5/7 is responsible for ensuring that cybersecurity training is integrated and conducted throughout the Army. If the DOD component heads do not ensure that their respective components accurately monitor and report information on the extent that users have completed the Cyber Awareness Challenge training—as well as have access revoked for not completing the training—the components may be unable to ensure that DOD users are trained in the steps needed to address cybersecurity threats to the department. One of the 16 selected components in our review—DARPA—did not require its users to take DOD’s Cyber Awareness Challenge training, according to DARPA officials, even though it is required by policy. Instead, DARPA has required its users to take cybersecurity training that it developed. While DARPA developed its own training program, we found that this training program did not address all of the requirements identified in a DOD staff manual or the cybersecurity training topics identified by the Cyber Workforce Advisory Group. DARPA officials recognized that its cybersecurity training was not equivalent to the DOD’s Cyber Awareness Challenge training program, which according to DOD CIO officials, addressed the training topics identified by the DOD Cyber Workforce Advisory Group. They explained that DARPA designs its courses to be concise to allow their personnel to focus on accomplishing the agency’s mission and that users can obtain additional information from references cited in the course materials. In addition, these officials told us that they were unaware their users were required to take the Cyber Awareness Challenge training that DISA developed. The DOD CIO is responsible for overseeing the implementation of the Cyber Awareness Challenge training, according to DOD CIO officials. However, DOD CIO officials told us they were not aware that DARPA has not required its users to take the Cyber Awareness Challenge training that DISA developed and they did not assess the extent that components complied with the requirement for components to use the DISA- developed training. If the DOD CIO does not ensure that DARPA and any other DOD components take the Cyber Awareness Challenge training developed by DISA, users in these components may take actions that lead to or enable exploitations of DOD information systems. DOD identified key techniques that adversaries use most frequently and that pose significant risk to the department’s networks and identified cyber hygiene practices to protect the department’s networks from these techniques. Specifically, JFHQ-DODIN has identified the cyberattack techniques that the agency observes adversaries using most frequently to attack the department’s networks. In addition, the National Security Agency, the Defense Information Systems Agency, and the DOD CIO identified 177 cyberattack techniques and prioritized the techniques according to the level of risk each posed to the department’s networks. The agencies prioritized the techniques using various criteria including the prevalence of the technique and whether the department could detect the use of the technique. Further, the department has established cyber hygiene practices to mitigate most of the frequently occurring techniques and those that the department identified as the highest priority, according to DISA and JFHQ-DODIN officials. However, the department does not know the extent that these cyber hygiene practices have been implemented across the department to protect its networks from these key cyberattack techniques. Components have visibility of the extent that they have implemented practices within their component, according to DOD officials. For example, DISA officials told us that they require their component to implement cyber hygiene practices to protect DOD networks from key cyberattack techniques and are able to determine the extent that those practices are implemented within DISA. However, no component or office within the department has complete visibility of the department’s efforts to implement these protective practices across the department, according to DOD officials. FISMA states that agency heads shall be responsible for, among other things, providing information security protections commensurate with the risk and magnitude of harm that could result from unauthorized access, use, disclosure, disruption, modification or destruction of such information systems. Executive Order 13800 states that agency heads will be held accountable for managing cybersecurity risk to their enterprises. The order requires agency heads to use the NIST’s Framework for Improving Critical Infrastructure Cybersecurity (commonly referred to as the NIST Cybersecurity Framework) to manage their agency’s cybersecurity risk. The Cybersecurity Framework calls for senior executives to monitor cybersecurity risk in the same context as financial risk and other organizational risks. In doing so, the Cybersecurity Framework calls for agencies to, among other things, assess cybersecurity risks (including threats), prioritize cybersecurity outcomes and requirements based on that risk, and establish processes to assess and monitor the implementation of the cybersecurity outcomes and requirements. The department does not know the extent that practices to protect DOD networks from key cyberattack techniques have been implemented across the department in part because no DOD component monitors the extent to which such practices are implemented, according to DOD officials. Officials from JFHQ-DODIN told us that they are able to detect when adversaries are using techniques to attack the department’s networks. However, detecting an attack after it has commenced may still enable an adversary to inflict harm on the department’s networks and the information therein. If the Secretary of Defense does not direct a component to monitor the extent to which practices to protect its network are implemented, gaps in protection could go undetected. These gaps can jeopardize military operations, performance of critical functions, and protection of information within DOD systems and networks. DOD requirements and best practices recognize that senior DOD leaders need key information to make risk-based decisions. Specifically, the DC3I memorandum requires the commander of U.S. Cyber Command, in coordination with the DOD CIO, to provide quarterly updates to the Deputy Secretary of Defense and the Vice Chairman of the Joint Chiefs of Staff on the progress in implementing the DC3I. Further, the CDIP memorandum requires the department to report progress implementing the CDIP tasks. In addition, NIST Special Publication 800-50, Building an Information Technology Security Awareness and Training Program, states that the CIO should ensure that agency heads and senior managers are informed of the progress of the security awareness and training program’s implementation. Senior DOD leaders receive two recurring reports on the department’s cybersecurity posture that include information on one cyber hygiene initiative. Specifically, the Cyber Hygiene Scorecard (Scorecard) is a report measuring compliance with DOD cybersecurity policies, procedures, standards and guidelines. The Scorecard provides information to the Secretary of Defense, the Deputy Secretary of Defense, and DOD component heads about the extent that the 10 CDIP tasks overseen by the DOD CIO are implemented. In addition, the Cyber Landscape Report is a quarterly report that includes information highlighting cybersecurity risks to DOD networks, U.S. critical infrastructure, DOD weapon systems, the cloud, and DOD’s cyber workforce. Based on our analysis, the Cyber Landscape Report also includes some information from the CDIP initiative. However, senior DOD leaders have not received information on the other two cyber hygiene initiatives or cyber hygiene practices to protect DOD networks from key cyberattack techniques in these recurring reports. Specifically, neither the Scorecard nor the Cyber Landscape Report includes information on the extent that the DC3I and the Cyber Awareness Challenge training have been implemented. In addition, neither of these recurring reports identifies key cyberattack techniques the department faces nor do they include information on the extent that the department has implemented cyber hygiene practices to protect DOD networks from these techniques, according to DOD officials. Senior DOD leaders are not receiving complete information in part because the DOD CIO has not assessed the extent that the missing information could improve senior leaders’ ability to make risk-based decisions. According to DOD officials, DOD CIO has not revised the recurring reports or developed a new report in response to such an assessment. DOD CIO officials told us that they do not believe that senior DOD leaders need to be made aware of all cyber hygiene topics we describe here—and in some cases that information could be managed at lower echelons within the organization. While some cyber hygiene information could be managed by lower-echelon DOD leaders, the DC3I memorandum requires information about its progress to be reported to senior leaders. The NIST guidance calls for similar reporting. Additionally, a DOD official told us that the department uses the Cyber Hygiene Scorecard to respond to the department’s requirement to annually report progress on implementing its information security program to the Office of Management and Budget under FISMA. Further, these officials told us that the Scorecard was not originally designed to include the information from our analysis such as information about the DC3I. They told us that this Scorecard was designed to provide an oversight tool to monitor the progress components made implementing the CDIP tasks overseen by DOD CIO. However, while DOD uses the Scorecard with the intention to meet the FISMA annual reporting requirement, the Scorecard does not provide information about 53 of the 69 risk-management FISMA indicators that are called for by the Office of Management and Budget. In addition, DOD CIO is not precluded from revising the Scorecard to include additional information. As one of two recurring reports sent to senior DOD leaders, the Cyber Hygiene Scorecard may be well positioned to provide additional information reflecting progress made implementing cyber hygiene initiatives and associated cybersecurity practices, including the DC3I and efforts to protect DOD networks from the key cyberattack techniques used by adversaries. Further, a DOD CIO official told us that its officials did not include information about the DC3I in the Cyber Hygiene Scorecard because they believed it would be challenging to measure the culture-related objectives in the DC3I. While the DC3I’s culture-related objectives may be difficult to measure, the extent to which assigned DOD components have taken actions to implement the DC3I tasks is measurable. If the DOD CIO does not assess the extent that the missing information could improve senior leaders’ ability to make risk-based decisions—and does not follow up to revise the recurring reports or develop a new report—senior DOD leaders will not be positioned well to make effective and risk-based decisions and manage cybersecurity risks. As DOD has become increasingly reliant on IT systems and networks to conduct military operations and perform critical functions, risks to these systems and networks have also increased because IT systems are often riddled with cybersecurity vulnerabilities—both known and unknown. These vulnerabilities and human error can facilitate security incidents and cyberattacks that disrupt critical operations; lead to inappropriate access to and disclosure, modification, or destruction of sensitive information; and threaten national security. DOD has taken actions to address cyber vulnerabilities in the department through establishing the DC3I, the CDIP, the Cyber Awareness Challenge training, and cyber hygiene practices to protect its networks from cyberattack techniques that adversaries may use. However, the department faces challenges implementing the DC3I and CDIP because the DOD CIO has not taken appropriate steps to ensure that the DC3I tasks are implemented, DOD components have not developed plans with scheduled completion dates to implement the remaining four CDIP tasks overseen by DOD CIO, and the Deputy Secretary of Defense has not identified a DOD component to oversee the implementation of the seven other CDIP tasks and report on progress implementing them. By improving oversight through implementing the DC3I tasks, DOD components developing plans with scheduled completion dates to implement the remaining four CDIP tasks that the DOD CIO oversees, and identifying a DOD component to oversee implementation of the seven other CDIP tasks and report on progress implementing them, the department can be better positioned to safeguard DOD’s network by removing preventable, well-known vulnerabilities. If the components address gaps we identified in the extent that they account for whether their users completed the 2018 Cyber Awareness Challenge training will help the department gain assurance that its workforce is prepared to identify and appropriately respond to cybersecurity risks. Additionally, by ensuring that DARPA, and any other similar DOD components, requires its users to take the required DISA- developed training, DOD users may be more aware of threats and vulnerabilities to the department’s networks and may be better equipped to prevent exploitations of DOD information systems. The department does not know the extent that cyber hygiene practices have been implemented to protect DOD networks from key cyberattack techniques. By directing a component to monitor the extent to which practices to protect DOD’s networks are implemented, DOD would be better positioned to ensure that its networks are secure and decrease potential risks to military operations, critical functions, and information assurance. Finally, the lack of information on two cyber hygiene initiatives and cyber hygiene practices in recurring reports provided to senior DOD leaders is concerning because of the need for those leaders to have a complete picture of the state of the department’s cybersecurity posture. By directing DOD CIO to assess the extent that the missing information could improve senior leaders’ ability to make risk-based decisions and revise the recurring reports or develop a new report, DOD leaders would then be better positioned to make effective decisions and manage cybersecurity risks. We are making seven recommendations to the Department of Defense. The Secretary of Defense should ensure that the DOD CIO takes appropriate steps to ensure implementation of the DC3I tasks. (Recommendation 1) The Secretary of Defense should ensure that DOD components develop plans with scheduled completion dates to implement the four remaining CDIP tasks overseen by DOD CIO. (Recommendation 2) The Secretary of Defense should ensure that the Deputy Secretary of Defense identifies a DOD component to oversee the implementation of the seven CDIP tasks not overseen by DOD CIO and report on progress implementing them. (Recommendation 3) The Secretary of Defense should ensure that DOD components accurately monitor and report information on the extent that users have completed the Cyber Awareness Challenge training as well as the number of users whose access to the network was revoked because they have not completed the training. (Recommendation 4) The Secretary of Defense should ensure that the DOD CIO ensures all DOD components, including DARPA, require their users to take the Cyber Awareness Challenge training developed by DISA. (Recommendation 5) The Secretary of Defense should direct a component to monitor the extent to which practices are implemented to protect the department’s network from key cyberattack techniques. (Recommendation 6) The Secretary of Defense should ensure that the DOD CIO assesses the extent to which senior leaders’ have more complete information to make risk-based decisions—and revise the recurring reports (or develop a new report) accordingly. Such information could include DOD’s progress on implementing (a) cybersecurity practices identified in cyber hygiene initiatives and (b) cyber hygiene practices to protect DOD networks from key cyberattack techniques. (Recommendation 7) We provided a draft of this report to the department for review and comment. In written comments, reprinted in appendix III, DOD concurred with one of our seven recommendations, partially concurred with four, and did not concur with the remaining two. DOD separately provided technical comments, which we incorporated as appropriate. The department concurred with our recommendation (Recommendation 5) that the DOD CIO ensure all components, including DARPA, require their users to take the Cyber Awareness Challenge training developed by DISA. The department partially concurred with four of our recommendations. The department partially concurred with our recommendation that the DOD CIO take steps to ensure that DC3I tasks are implemented. The department concurred that tasks two and six in the DC3I should be implemented and stated that these two tasks are the only two still actively being pursued. The department stated that the remaining five tasks were either implemented or have been overcome by events. However, the department did not provide evidence that the other five tasks were implemented or demonstrate how these tasks were overcome by events during the audit or in its comments on a draft or our report. In addition, JFHQ-DODIN officials stated that the principles outlined in the DC3I are important for the department to achieve its cybersecurity goals. For example, several of these five tasks were focused on improving cybersecurity awareness and training at all levels within the department. Therefore, it is unclear why DOD believes that these cyber hygiene tasks have been overcome by events; DOD did not elaborate. Implementing all seven DC3I tasks that have not been implemented can better position the department to achieve the goals of the DC3I to (1) mitigate the risks of compromising the confidentiality, integrity, and availability of mission- critical information as a result of human error by users on the department’s networks; and (2) transform DOD cybersecurity culture by enabling and reshaping leaders, cyber providers, personnel who perform cyberspace operations, and general users to improve individual human performance and accountability on DOD’s network. The department partially concurred with our recommendation that DOD components develop plans with scheduled completion dates to implement the four remaining CDIP tasks overseen by DOD CIO. DOD provided classified comments on this recommendation. Thus, we cannot respond in detail to their comments. We plan to respond to DOD’s comments in a classified version of this report, which we plan to issue later in 2020. Developing plans that would facilitate implementation of these four CDIP tasks would better position DOD to meet the Deputy Secretary of Defense’s goal of removing preventable vulnerabilities from DOD’s network that could allow adversaries to compromise information and information systems. The department partially concurred with our recommendation that components accurately monitor and report information on the extent that users have completed the Cyber Awareness Challenge training and information on the number who have been denied access to the network for not completing the training. The department concurred that it should ensure components accurately report the number of users who have completed the training. However, it did not concur that components should report the number of users who have been denied access to the network because they have not completed the training. The department stated that a statistic showing this information would not be meaningful and would be burdensome to collect. We disagree that such a measure would not be meaningful because it would help leaders hold network users accountable and better position DOD components to comply with DOD policy. Recognizing that trained and aware users are the first and most vital line of defense, DOD components should document and maintain the status of awareness compliance for each user. In its current approach, DOD is unable to confirm whether all of its network users have completed the cybersecurity training, as required. For example, as stated above, 8 of the 16 (50 percent) of the DOD components we requested training information from told us they did not monitor whether users who did not complete the annual training were blocked from DOD networks and systems. If the Secretary of Defense does not ensure that DOD components accurately monitor and report information on the number of users whose access to the network was revoked because they have not completed the training, the components will jeopardize the department’s ability to ensure that DOD users are trained in steps needed to address cybersecurity threats to the department. In responding to this recommendation, DOD also stated that the Navy indicated that it provided us data on the number of its users who completed the training and the total number of its users. The department stated that we could compute the number of Navy users who had not completed the training by computing the difference between the total number of users and the number of users who completed the training. We updated our assessment of the Navy in our report. We now indicate that the Navy was able to identify the number of users who had completed the training in fiscal year 2018. However, we disagree that the difference between the total number of users and the number of users who completed the training equates to the number of users who did not take the training. DOD CIO officials told us during our audit that computing the number of users using this method is not reliable because there are multiple explanations for the difference between the total number of users and the number of users who took the training. For example, officials told us that some military users leave the service before they complete the annually required training and are included in the service’s total number of users but are not included in the number of users who took the training. The department partially concurred with our recommendation that the CIO assess the extent to which senior leaders have information to make risk-based decisions and then revise accordingly the recurring reports. The department stated that it will revise the recurring reports by merging the Cyber Hygiene Scorecard and a scorecard related to the Cyber Landscape to assist senior leaders’ decision-making. However, the department stated that it did not fully agree with the recommendation because, as written in the draft report, the department believed the recommendation was stating that DOD should have “complete” information. Based on DOD’s comment, we clarified the recommendation to state that senior DOD leaders should have more complete information to make risk-based decisions. We believe this is critical because the cyber hygiene tasks and practices highlighted in the report were identified by the most senior leaders in the department—including the Secretary of Defense, Deputy Secretary of Defense, and Chairman of the Joint Chiefs of Staff—as being the tasks and practices that were essential to protecting DOD information, systems, and networks from the most common and pervasive cybersecurity risks faced by the department. The department also stated that risk is a function of multiple variables, that are continually evolving. We agree that risk is a function of multiple variables—including threats and vulnerabilities—that are continually evolving. As such, we think that information, such as the extent to which cyber hygiene practices have been implemented across the department to protect its networks from evolving key cyberattack techniques, will position senior leaders to make more effective and risk-based decisions and manage cybersecurity risks. The department did not concur with two recommendations. In particular: DOD did not concur with our recommendation that the Deputy Secretary of Defense identify a component to oversee the implementation of the seven CDIP tasks that the CIO does not oversee and report on progress implementing those tasks. The department stated that, since the CDIP’s approval in 2015, the department has issued new or updated versions of a number of cyber- related strategies, including the DOD Cyber Strategy. The department also stated that the Deputy Secretary of Defense directed DOD to develop a classified top 10 list of cybersecurity critical-risk areas and an associated scorecard that provides the Deputy Secretary a quarterly assessment of the department’s progress in reducing the risk for each of these areas. The department also stated that the cyber landscape is constantly evolving with changes in technology, threats, and vulnerabilities, and that this requires DOD to reassess its cybersecurity priorities. The department stated that implementing our recommendation would override these recent efforts and focus the department’s efforts on monitoring areas with lower levels of risk. We disagree that implementing our recommendation would override the department’s recent efforts. In fact, implementing the seven tasks would align with one of the 2018 DOD Cyber Strategy’s objectives to “secure DOD information and systems against malicious cyber activity.” We agree with DOD that the department should reassess cybersecurity priorities in light of changes in technologies, threats, and vulnerabilities. However, DOD did not provide evidence during the audit or in responding to the draft report that the department had assessed the CDIP tasks required by the Deputy Secretary of Defense in 2015. Specifically, the department has not determined whether they remain valid or aligned with the current cybersecurity threat environment, that the vulnerabilities associated with these seven tasks were mitigated or addressed, and that a senior-level DOD official provided written direction canceling the Deputy Secretary of Defense’ CDIP taskings. More importantly, our analysis of the seven tasks that DOD is not currently tracking progress on are consistent with basic cybersecurity standards established by DOD guidance and NIST— and which DOD is planning to apply to certain defense contractors in future contract awards to protect DOD information that is stored or transits through their networks as a part of the Cybersecurity Maturity Model Certification framework. For example, Task 14 requires commanders and supervisors to ensure physical security of their network infrastructure devices. This task aligns with general NIST guidance regarding physical access protections. NIST guidance states that organizations should manage and protect physical access to assets and facilities where information systems reside. Task 15 requires commanders and supervisors to report all commercially provided internet connections to DOD’s unclassified network. This task aligns with general NIST guidance regarding the use of external networks. NIST guidance states that organizations should catalogue all external information systems. Task 16 requires commanders and supervisors to ensure alignment to a Computer Network Defense Service Provider. This task is consistent with DOD requirements on cybersecurity activities to protect the DOD Information Network. The requirements state that DOD IT must be aligned to DOD network operations and security centers, which provide any required cybersecurity services. Task 17 requires commanders and supervisors with Computer Network Defense Service Provider responsibility to ensure the cyber incident response plan(s) are properly exercised and documented. This task aligns with general NIST guidance regarding incident response. NIST guidance states that organizations should provide incident response handling training and implement incident handling capabilities, as well as a process to ensure that response processes and procedures are executed, and maintained ensuring response to detected cybersecurity incidents. If the Deputy Secretary of Defense does not implement this recommendation, the department will have less assurance that cybersecurity vulnerabilities are being addressed in a timely manner and systems are being securely configured. The department did not concur with our recommendation that a component monitor the extent of implementation of practices to protect the department's network from key cyberattack techniques. The department determined that the information in its response to this recommendation included sensitive information. Therefore, we are redacting the department’s response to this recommendation from DOD’s written comments that we are reprinting in Appendix III. However, we still believe the recommendation is valid. As stated in our report, no component or office within the department has complete visibility of the department’s efforts to implement these protective practices across the department, according to DOD officials. Taking action to implement the intent of this recommendation would help address that gap. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; DOD’s Chief Information Officer; the Secretaries of the Army, Navy, and Air Force; the Commandant of the Marine Corps; the Chairman of the Joint Chiefs of Staff; the Commanding Generals of U.S. Strategic Command, U.S. European Command, U.S. Southern Command, and U.S. Cyber Command; and the Directors of DISA, the National Security Agency, DARPA, the Defense Commissary Agency, the Defense Contract Management Agency, the Defense Finance and Accounting Service, the Defense Media Activity, and the Defense Technology Security Administration. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact us: Joseph Kirschbaum at (202) 512-9971 or kirschbaumj@gao.gov, or Nick Marinos at (202) 512-9342 or marinosn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. For the purposes of this review, we adapted a definition of cyber hygiene developed by Carnegie Mellon University’s Software Engineering Institute. The institute defines cyber hygiene as a set of practices for managing the most common and pervasive cybersecurity risks faced by organizations today. We discussed the definition of cyber hygiene with Department of Defense (DOD) officials to identify DOD initiatives aimed at improving cyber hygiene. DOD officials identified the Cyber Discipline Implementation Plan (CDIP) as DOD’s main cyber hygiene initiative aimed at implementing technical improvements to DOD networks. In addition, DOD officials identified the DOD Cybersecurity Culture and Compliance Initiative (DC3I) and DOD’s Cyber Awareness Challenge training as two initiatives designed to establish best practices for DOD network users including military personnel, civilians, and contractors. To determine the extent to which DOD has implemented its three cyber hygiene initiatives and practices to protect its networks from cyberattack techniques that adversaries may use, we conducted analyses for each initiative. To determine the extent to which DOD implemented the DC3I, we reviewed the 11 tasks that require components to take actions that are specified in the DC3I memorandum that the Secretary of Defense and the Chairman of the Joint Chiefs of Staff issued in September 2015. We analyzed documentation we collected from U.S. Cyber Command, the office of the DOD Chief Information Officer (CIO), and the Joint Staff that demonstrate actions these components took in response to each of the 11 DC3I tasks and determined the extent to which each task was implemented. To determine the extent to which DOD implemented the CDIP, we reviewed the 17 tasks that require components to take actions specified in a memorandum that the Deputy Secretary of Defense issued in October 2015. We interviewed officials from the office of the DOD CIO about the extent to which DOD components implemented the CDIP tasks, the reasons the components had not fully implemented all of the tasks, and to determine the extent that the DOD CIO knew if DOD components had implemented the remaining seven CDIP tasks. We also reviewed documentation on the extent that DOD components implemented the tasks overseen by DOD CIO by analyzing data included in the Cyber Hygiene Scorecard. We also assessed the reliability of the data in the Scorecard by reviewing the methods the DOD CIO uses to ensure the data reported to the Scorecard are accurate and interviewing cognizant officials. We determined the data are sufficiently reliable for our purposes. To determine the extent that DOD implemented the Cyber Awareness Challenge training, we analyzed the extent that the DOD CIO and the DOD component CIOs ensured that personnel they oversee completed the fiscal year 2018 Cyber Awareness Challenge training. To carry out this analysis, we collected and analyzed information from the DOD CIO and a sample of 16 DOD components. We selected this sample of components by identifying important groupings of components and selecting from these groups to ensure that our sample represented a significant number of DOD personnel as well as a variety of types of components. These groups were: the military services and the Joint Staff, combatant commands, agencies and field activities, and the Office of the Secretary of Defense. Military services and Joint Staff. We selected the four military services because they are the components within DOD with the most personnel. We also included the Joint Staff because this component reflects the strategic perspective for the department as a whole. Combatant commands. We randomly selected three combatant commands from the group of 11 combatant commands—including geographic (e.g., U.S. Central Command) and functional (e.g., U.S. Transportation Command). We selected three of the 11 combatant commands to include the perspectives of multiple combatant commands in our sample. We selected these combatant commands: U.S. European Command, U.S. Southern Command, and U.S. Strategic Command. Agencies and Field Activities. We assembled a list of non-service and non-combatant command components organized by the types of functions that each component performs. We then organized these components by functional groupings. Specifically, we created functional groupings for the components that fall under each of the six Under Secretaries of Defense because these officials oversee components with similar functions. We also included a seventh functional group of miscellaneous components that are not overseen by any of the Under Secretaries of Defense. We then accounted for the size of the components on this list by identifying the larger agencies and the smaller field activities. From this list, we randomly selected one component from each of the seven groups. In doing so, we selected five of the 20 agencies and two of the eight field activities. We chose this ratio of agencies to field activities to reflect the ratio of agencies to field activities in DOD. That is, DOD agencies are about 71 percent of DOD’s non-service and non-combatant command components and about 71 percent of our sample. We selected these five agencies: Defense Advanced Research Projects Agency, Defense Commissary Agency, Defense Contract Management Agency, Defense Finance and Accounting Service and the National Security Agency. We selected these two field activities: Defense Media Activity and Defense Technology Security Administration. The Office of the Secretary of Defense. We also randomly selected one of 16 components from the Office of the Secretary of Defense. This group included the offices that support the six Under Secretaries we discussed above such as the Under Secretary of Defense for Policy as well as other offices including the Office of Cost Assessment and Program Evaluation and the Office of the DOD Chief Management Officer. We selected one component from this group to ensure we reflected the perspective of components at the DOD headquarters level. We selected the Office of the DOD Chief Information Officer. To collect information from this sample of 16 components, we developed a standard set of questions we provided to each component on topics related to both objectives. In particular, we asked DOD components to provide the number of network users that completed the fiscal year 2018 Cyber Awareness Challenge training, the number of network users that did not complete the training, and the number of network users who had their access to the network removed as a result of not taking the training. We also asked other questions including a question about the information that senior leaders are provided regarding cyber hygiene practices. Each component provided written responses to our questions and in some cases provided documentation corroborating their responses. We conducted a content analysis of the components’ responses and the documentation they provided. To complete this content analysis, two analysts assessed the components’ responses, compared and discussed their separate analyses, and reached agreement on their conclusions about their analysis. We compared the information we collected from these components to a provision in NIST Special Publication 800-50, Building an Information Technology Security Awareness and Training Program, which advises agencies to capture training compliance data at an agency level. Further, we interviewed officials from Defense Information Systems Agency and JFHQ-DODIN to determine the extent to which DOD had implemented cyber hygiene practices that the department has implemented to protect its networks from key cyberattack techniques that adversaries may use. To determine the extent to which senior DOD leaders receive information on the department’s efforts to address cyber hygiene initiatives and practices, we first defined senior DOD leaders as the Secretary of Defense, the Deputy Secretary of Defense, and DOD component heads. To identify the information that could be included in reports that senior DOD leaders receive about DOD efforts to mitigate cyberattack techniques, we identified techniques that are most likely to be used by adversaries against DOD’s networks or that could cause severe adverse effects on DOD’s operations. In particular, we identified 22 key cyberattack techniques from two sources: Joint Force Headquarters DOD Information Network (JFHQ-DODIN) provided a list of eight cyberattack techniques that the agency observed adversaries using most frequently in January 2019. JFHQ- DODIN officials also determined that these data are representative of the cyberattack techniques that they have recently observed. We identified 14 cyberattack techniques by analyzing a review conducted in 2016 by the National Security Agency, the Defense Information Systems Agency, and the DOD CIO. In the review, the agencies identified 177 cyberattack techniques and ranked the techniques according to the level of risk the techniques posed to DOD’s unclassified and Secret-level networks. The agencies used a number of different criteria to rank these techniques, including the prevalence of the technique, visibility of the technique, and whether other, closely associated alternative techniques exist. We selected the 14 cyberattack techniques that the agencies identified as the highest priority. Next, we analyzed the contents of two recurring reports that senior leaders receive on the department’s cybersecurity posture: the Cyber Hygiene Scorecard and the Cyber Landscape Report. In particular, we analyzed these reports to determine if they included information about DOD’s implementation of key cyber hygiene initiatives that we describe in the first objective. We also analyzed the reports to determine if they included the lists of key cyberattack techniques and information about the extent that the department had implemented cyber hygiene practices to protect DOD networks from these cyberattack techniques. We conducted this performance audit from January 2019 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Department of Defense (DOD) Chief Information Officer (CIO) and other relevant DOD components implemented four of the 11 tasks required in the Cybersecurity Culture and Compliance Initiative (DC3I) and the remaining seven tasks were not fully implemented as of October 2019. Table 2 provides additional information of actions taken to address and implement all 11 DC3I tasks. In addition to the individuals named above, Tommy Baril (Assistant Director), Kaelin Kuhn (Assistant Director), James P. Klein (Analyst-in- Charge), Tracy Barnes, Amy Bush, Peter Casey, Amie Lesser, Carlo Mozo, Richard Powelson, Michael Silver, Andrew Stavisky, and Walter Vance made significant contributions to this report. Kiana Beshir, Chris Businsky, Shaun Byrnes, and Richard Sayoc also contributed to the report.", "summary": "DOD has become increasingly reliant on information technology (IT) and risks have increased as cybersecurity threats evolve. Cybersecurity experts estimate that 90 percent of cyberattacks could be defeated by implementing basic cyber hygiene and sharing best practices, according to DOD's Principal Cyber Advisor. Senate Report 115-262 includes a provision that GAO review DOD cyber hygiene. This report evaluates the extent to which 1) DOD has implemented key cyber hygiene initiatives and practices to protect DOD networks from key cyberattack techniques and 2) senior DOD leaders received information on the department's efforts to address these initiatives and cyber hygiene practices. GAO reviewed documentation of DOD actions taken to implement three cyber hygiene initiatives and reviewed recurring reports provided to senior DOD leaders. The Department of Defense (DOD) has not fully implemented three of its key initiatives and practices aimed at improving cyber hygiene. Carnegie-Mellon University defines cyber hygiene as a set of practices for managing the most common and pervasive cybersecurity risks. In discussions with GAO, DOD officials identified three department-wide cyber hygiene initiatives: the 2015 DOD Cybersecurity Culture and Compliance Initiative, the 2015 DOD Cyber Discipline Implementation Plan, and DOD's Cyber Awareness Challenge training. The Culture and Compliance Initiative set forth 11 overall tasks expected to be completed in fiscal year 2016. It includes cyber education and training, integration of cyber into operational exercises, and needed recommendations on changes to cyber capabilities and authorities. However, seven of these tasks have not been fully implemented. The Cyber Discipline plan has 17 tasks focused on removing preventable vulnerabilities from DOD's networks that could otherwise enable adversaries to compromise information and systems. Of these 17, the DOD Chief Information Officer is responsible for overseeing implementation of 10 tasks. While the Deputy Secretary set a goal of achieving 90 percent implementation of the 10 CIO tasks by the end of fiscal year 2018, four of the tasks have not been implemented. Further, the completion of the other seven tasks was unknown because no DOD entity has been designated to report on the progress. The Cyber Awareness training is intended to help the DOD workforce maintain awareness of known and emerging cyber threats, and reinforce best practices to keep information and systems secure. However, selected components in the department do not know the extent to which users of its systems have completed this required training. GAO's review of 16 selected components identified six without information on system users that had not completed the required training, and eight without information on users whose network access had been revoked for not completing training. Beyond the initiatives above, DOD has (1) developed lists of the techniques that adversaries use most frequently and pose significant risk to the department, and (2) identified practices to protect DOD networks and systems against these techniques. However, the department does not know the extent to which these practices have been implemented. The absence of this knowledge is due in part to no DOD component monitoring implementation, according to DOD officials. Overall, until DOD completes its cyber hygiene initiatives and ensures that cyber practices are implemented, the department will face an enhanced risk of successful attack. While two recurring reports have provided updates to senior DOD leaders on cyber information on the Cyber Discipline plan implementation, department leadership has not regularly received information on the other two initiatives and on the extent to which cyber hygiene practices are being implemented. Such information would better position leaders to be aware of the cyber risks facing DOD and make more effective decisions to manage such risks. GAO is making seven recommendations to DOD, including that cyber hygiene initiatives be fully implemented, entities are designated to monitor component completion of tasks and cyber hygiene practices, and senior DOD leaders receive information on cyber hygiene initiatives and practices. Of the seven recommendations, DOD concurred with one, partially concurred with four, and did not concur with two. GAO continues to believe that all recommendations are warranted.", "document_type": "gao"}
{"report": "BMI is used as a screening tool for obesity. An individual with a BMI of 30 or higher is considered to have obesity. Over the past two decades, both the prevalence of obesity and estimates of the medical spending associated with individuals with obesity have increased. For example, a 2018 study estimated that the percentage of national medical expenditures used to treat obesity-related illnesses in adults increased from 6.13 percent in 2001 to 7.91 percent in 2015, a 29 percent increase. This study also found that the high medical costs of obesity are due to extremely high medical costs among a small percentage of the population who have severe obesity (those with a BMI of 40 or higher). In addition, a 2017 study found that medical expenditures rise most rapidly for individuals with a BMI of 40 or higher. One option for the treatment of obesity is the use of prescription obesity drugs. As of June 2019, there were nine prescription drugs approved by FDA to treat obesity. Four obesity drugs—benzphetamine, diethylpropion, phendimetrazine, and phentermine—were approved by FDA in 1961 or earlier for short-term use, which is generally about 12 weeks, and are available as generic drugs. The remaining five obesity drugs were approved by FDA in 1999 or later for long-term use and are available as brand-name drugs— bupropion/naltrexone (Contrave), liraglutide (Saxenda), lorcaserin (Belviq), orlistat (Xenical), and phentermine/topiramate (Qsymia). Each of these five brand-name obesity drugs underwent one or more randomized, controlled clinical trials for safety and efficacy prior to FDA approval of the drug—a total of 15 clinical trials across the five drugs. Obesity drugs work in different ways; some may help an individual feel full sooner or less hungry, while others may reduce fat absorption in the body. Results vary by medication and by person, but, according to the National Institutes of Health, on average, people who take obesity drugs as part of a lifestyle program lose between 3 and 9 percent more of their starting body weight than people in a lifestyle program who do not take obesity drugs. As with other prescription drugs, obesity drugs may have side effects such as headache, dizziness, dry mouth, nausea, and diarrhea. And, as with other prescription drugs, health care providers may prescribe an obesity drug for off-label use— that is, for a different medical condition, in a different dosage, or for a different duration than for which the drug is FDA approved. Obesity drugs should be used as an adjunct to lifestyle therapy (e.g., diet, physical activity, and behavioral counseling), according to guidelines from several medical associations. According to these guidelines, the use of obesity drugs is indicated for individuals with a BMI of 27 or higher with one or more obesity comorbidities (such as type 2 diabetes), or individuals with a BMI of 30 or higher who have a history of failure to achieve clinically meaningful weight loss (that is, weight loss of 5 percent or more) or who are unable to sustain weight loss. In addition, the guidelines recommend evaluating the patient’s weight loss after about 12 to 16 weeks of treatment with an obesity drug and discontinuing the drug if the patient has not lost a certain amount (e.g., at least 5 percent) of their initial body weight. Although obesity is classified as a disease, some health care providers, including those who specialize in the care of patients with obesity, continue to stigmatize patients with obesity. For example, a 2018 study reported that health care providers may perceive patients with obesity as being less compliant and having less self-discipline than other patients. Additionally, health care providers may not initiate discussions about weight loss with patients because of lack of time, other important issues or concerns, a belief that a patient is not motivated or interested in losing weight, or concern over a patient’s emotional state, according to another 2018 study. The prevalence of obesity was about 38 percent among all U.S. adults (about four of every 10 adults) from 2013 through 2016, according to nationally representative estimates from CDC. The estimate of prevalence among adults covered by Medicare was about 40 percent, and among those with Medicaid or other public health insurance (excluding Medicare) it was about 42 percent. In addition, the prevalence of obesity among adults with private health insurance coverage and among the uninsured was similar, at about 37 percent and 38 percent, respectively. These national estimates also showed that about 24 percent of Medicare beneficiaries had Class 1 obesity, about 10 percent had Class 2 obesity, and about 6 percent had Class 3, or severe, obesity. (See fig. 1.) According to CDC estimates, adults age 18 to 64 and adults age 65 and older had a similar prevalence of obesity, about 39 percent and 38 percent, respectively. However, a higher percentage of adults age 18 to 64 than adults age 65 and older had Class 3 obesity. (See table 1.) Appendix III provides additional information on the prevalence of obesity among adults, as well as on the prevalence of adults who were overweight, which is defined as a BMI of 25 to <30, including 95 percent confidence intervals. Relatively few U.S. adults, including adults with obesity and adults who reported trying to lose weight, used obesity drugs from 2012 through 2016, according to nationally representative estimates. Guidelines suggest prescribing obesity drugs as an adjunct to other diet and lifestyle changes, or when other approaches have not resulted in clinically significant weight loss. Those health care providers who prescribe obesity drugs consider several factors, such as whether there are any contraindications of the obesity drug for their patients and the cost of the drug. Some limited data are available on individuals who have used obesity drugs, including data on whether these individuals adhered to taking the prescribed obesity drug or maintained their weight loss over time. Available data indicate that relatively few U.S. adults, including those with obesity, used obesity drugs. Specifically, of the estimated 233 million U.S. adults, fewer than a million used any of the nine obesity drugs, according to AHRQ’s nationally representative estimates from MEPS data for 2012 through 2016. Of the estimated 71.6 million U.S. adults with obesity, an estimated 660,000 per year, on average, used an obesity drug, according to these data. Similarly, among those who reported trying to lose weight, relatively few of them (about 3 percent) reported that they took prescription medication for weight loss, according to CDC’s nationally representative estimates from NHANES for 2013 through 2016. Additionally, six of the studies we reviewed examined this topic and found that few U.S. adults have used obesity drugs. For example, one study reported that in 2011, 2,554 obesity drug prescriptions were filled per 100,000 people, with about 87 percent of those prescriptions for phentermine, a generic obesity drug. Three other studies assessed the use of obesity drugs among veterans receiving care from the Veterans Health Administration and similarly found that few patients were prescribed obesity drugs. One of these studies found that about 1 percent of the 153,939 veterans who enrolled in the MOVE! Weight Management Program from 2013 through 2016 were prescribed an obesity drug (orlistat, phentermine, phentermine/topiramate, liraglutide, or bupropion/naltrexone) within 1 year of MOVE! initiation. According to officials from groups representing physicians and advocacy groups we interviewed, and seven studies we reviewed, some physicians and other health care providers may not be open to or comfortable with prescribing obesity drugs. For example, providers may not perceive obesity drugs to be safe or effective. According to officials from one advocacy and research group, concerns about the safety of obesity drugs may be related to the adverse consequences associated with past obesity drugs. In addition, one medical association we contacted indicated physicians consider clinical preventive service recommendations from the U.S. Preventive Services Task Force on the use of obesity drugs. The task force recommends that clinicians offer or refer adults with a BMI of 30 or higher to intensive, multicomponent behavioral interventions. Further, a systematic review of evidence of the benefits and harms of behavioral therapy and use of obesity drugs conducted for the task force found that obesity drugs, but not behavior-based interventions, were associated with higher rates of harm. The potential for harm (i.e., adverse events) may discourage physicians and other health care providers from prescribing these drugs. In addition, officials we interviewed and the studies we reviewed noted that a lack of insurance coverage, high out-of-pocket costs, and the patient’s means to afford obesity drugs may also discourage physicians from prescribing obesity drugs. The officials and studies also noted that physicians might have gaps in knowledge about obesity drugs. For example, officials from one medical association noted that lack of education is a barrier to physicians in prescribing obesity drugs for patients who would be candidates for them, and officials from another medical association said that many clinicians are not aware that there are FDA-approved drugs for obesity, and therefore they do not think about prescribing them. One study we reviewed found that, of the 111 primary care providers responding to a survey, most reported limited experience with obesity drugs as a barrier to prescribing them. While guidelines on the use of obesity drugs suggest prescribing obesity drugs as an adjunct to other diet and lifestyle changes, or when other approaches have not resulted in clinically significant weight loss, physicians and other health care providers may not understand the recommendations outlined in the guidelines. For example, one study found that many of the health care providers responding to a survey reported responses inconsistent with the guideline-recommended thresholds to initiate and continue use of obesity drugs. Physicians and health care providers who do prescribe obesity drugs take several factors into consideration. Specifically, before prescribing an obesity drug, these providers consider the likely benefits of weight loss, the drug’s possible side effects, the patient’s current health issues and other medications, family medical history, and the cost of the drug, according to the National Institutes of Health. According to officials from an advocacy group, specific considerations include (1) the patient’s other health conditions that may increase the risk from using a particular obesity drug (contraindications); (2) the ability of an obesity drug to treat both the patient’s obesity and other health conditions; (3) the patient’s ability to afford a particular obesity drug, given their insurance coverage and other financial resources; (4) patient preference regarding the dosage and form of the drug; and (5) the average efficacy (weight loss) of an obesity drug. Further, when treating obesity, providers use the least invasive treatments, such as lifestyle-based therapies first, then escalate to obesity drugs if noninvasive treatments prove ineffective, according to officials from the same advocacy group. Some limited data are available on individuals who have used obesity drugs, including data on the distribution of BMI, the use of obesity drugs in conjunction with other items or services, whether these individuals adhered to using the prescribed obesity drug or maintained their weight loss over time, and the impact that using obesity drugs has on other medical services directly related to obesity. The following is a summary of available information on specific aspects of individuals who have used obesity drugs. Distribution of BMI across individuals who have used obesity drugs. CDC’s nationally representative estimates for 2013 through 2016 found that the BMI of adults who reported that they used obesity drugs ranged from 21 to 64, with a median BMI of 34. However, these data are limited because they do not indicate how long the individual used the drugs before their BMI was measured. Use of obesity drugs in conjunction with other items or services. Two studies we reviewed examined the use of obesity drugs in conjunction with other items or services. These studies found that participants who used an obesity drug in conjunction with other services, such as behavioral counseling, lost more weight than those who did not take the drug with the other services. For example, in one 2019 study, participants who received intensive behavioral therapy combined with an obesity drug, liraglutide, had nearly double the weight loss (an average of about 12 percent of their body weight) compared to the participants who received only intensive behavioral therapy (an average of about 6 percent of their body weight). In addition, the 15 clinical trials for the brand-name obesity drugs that we reviewed generally found that a significantly higher percentage of participants who used the obesity drug combined with other items or services (such as a low-calorie diet or increased physical activity) achieved 5 percent or more weight loss compared to participants who used a placebo with the other items or services. One clinical trial that used an intensive behavior modification program (28 group sessions) found higher average weight loss (9 percent loss of initial body weight) for participants who used the obesity drug (bupropion/naltrexone) than for the placebo group. This clinical trial also found that the placebo group with the intensive behavior modification had higher weight loss than placebo groups in the other clinical trials, none of which used intensive behavioral therapy. Adherence to using the prescribed obesity drug. FDA’s analysis of Sentinel System data of obesity drugs dispensed in 2008 through 2017 found that in the majority of patients using obesity drugs, cumulative treatment duration was 90 days or less. FDA analyzed data for 267,836 new users of obesity drugs and found that about 58 percent of patients who used any of the obesity drugs did so for 90 days or less; about 31 percent used any of the obesity drugs for 30 or fewer days. The average duration for the first use of any of the nine obesity drugs was 69 days. (See appendix V for more data from FDA’s analysis.) FDA’s findings are consistent with the findings of two of the three studies that we reviewed that measured adherence to using the prescribed obesity drug. These studies reported that use of obesity drugs dropped significantly after 30 days. For example, one 2018 study that reviewed 1 year of data on 26,522 patients who had new prescription drug claims for one of four obesity drugs (liraglutide, lorcaserin, bupropion/naltrexone, and phentermine/topiramate) found that adherence to using any of the four obesity drugs dropped markedly during the first month following the initial claim for the drug. In addition, while the 15 clinical trials we reviewed were not designed to measure adherence to taking obesity drugs, they provide some information on whether or not study participants adhered to using these drugs during the trials. Participant dropout rates for these clinical trials ranged from 14 percent to 66 percent for the obesity drug treatment and the placebo groups, which could indicate difficulty in adherence to the study regimen; however, participants using the placebo generally had higher dropout rates than those using the obesity drug. The reasons for discontinuation among study participants in the clinical trials included side effects, such as headaches and nausea; being unavailable for follow up; and withdrawal of consent. Maintaining weight loss over time by individuals who have used obesity drugs. The recent systematic review conducted for the U.S. Preventive Services Task Force noted that data on long-term weight loss with obesity drugs are limited. The review found that individuals using obesity drugs were more likely to maintain their weight loss over 12 to 36 months compared with placebo, but noted that the evidence was limited by the small number of trials for each medication, poor follow up with participants, and limited applicability (given that participants had to meet narrowly defined inclusion criteria), among other limitations. We also identified six studies—each of which reviewed one of the FDA-approved obesity drugs—that examined weight loss maintenance, generally after about 1 year. For example, a 2018 study for one obesity drug (lorcaserin) found that while the obesity drug initially improved upon weight loss achieved with weight loss maintenance counseling, this advantage was not maintained at 1 year. That is, after 1 year, there was no significant difference in weight loss maintenance between the participants treated with the obesity drug along with counseling, compared to those treated with placebo along with counseling. Another study that examined clinical trial data for one obesity drug (bupropion/naltrexone) concluded that participants who lost at least 5 percent of their body weight after 16 weeks were likely to maintain clinically significant weight loss (of at least 5 percent) after 1 year of treatment with the drug. The impact of using obesity drugs on medical services directly related to obesity. We did not identify any studies on the impact that the use of obesity drugs had on the utilization of medical services directly related to obesity. In terms of studies on the impact on health outcomes, the systematic review conducted for the U.S. Preventive Services Task Force concluded that health outcomes data for individuals receiving treatment with obesity drugs were limited. The review reported that clinical trials of obesity drugs for weight loss examined few outcomes beyond quality of life measures, and that none of the drug-based maintenance trials reported the effects of the obesity drug interventions on health outcomes. The review noted that the trials included in the review were of highly selected populations with multiple exclusions relevant to health outcomes (e.g., history of serious medical conditions). The review further noted that while it appears that weight loss interventions, including obesity drugs, can reduce diabetes incidence, larger studies with longer-term follow up are required to understand the full benefits of these interventions on health outcomes and whether those effects are long lasting. Health insurance coverage for obesity drugs is limited—that is, not all public and private health insurance provided coverage for obesity drugs or may have additional requirements to determine these drugs are medically necessary. Medicare Part D plans may opt to cover obesity drugs, and state Medicaid programs or Medicaid managed care plans within states may choose either to cover or exclude obesity drugs from coverage. We found that both Medicare Part D and Medicaid reimbursed for a relatively small number of prescriptions for obesity drugs in 2016 and 2017. For private health insurance—which includes employer- sponsored health insurance, individually purchased health plans, and FEHBP plans—we found that coverage varied and, when obesity drugs were covered, the coverage could have additional requirements such as prior authorization or determination that a drug is medically necessary for the patient. Medicare. Under Medicare’s prescription drug benefit, Medicare Part D plans may choose to cover obesity drugs—in these cases, obesity drugs are considered supplemental drugs under an enhanced alternative coverage plan. Medicare beneficiaries who select a Part D plan that offers supplemental benefits, which may include coverage of excluded drugs such as obesity drugs, must pay the full premium cost for those additional benefits (i.e., Medicare does not subsidize them). Medicare Part D plans can choose whether or not to offer enhanced alternative coverage, and not all Medicare Part D plans that provide enhanced alternative coverage cover obesity drugs as supplemental drugs. For example: Roughly half of the Medicare beneficiaries covered by one large insurer’s Medicare Part D plans in one state have coverage for obesity drugs as a supplemental drug under enhanced alternative coverage, according to officials from that insurer. Officials at another large insurer told us that their Medicare Part D plans have historically covered supplemental drugs based on consumer demand, and obesity drugs do not typically meet their threshold for offering supplemental coverage. The officials noted that their plans have limited funds to cover supplemental drugs and that consumer demand is typically highest for other types of drugs, such as drugs to treat erectile dysfunction. Enhanced Alternative Coverage and Supplemental Drugs under Medicare Enhanced alternative coverage is alternative prescription drug coverage under Medicare Part D with value exceeding that of Medicare Part D’s defined standard coverage. Enhanced alternative coverage may include basic prescription coverage and supplemental benefits such as supplemental drugs. Supplemental drugs are drugs—including drugs for weight loss—that would be covered Part D drugs but for the fact that they are specifically excluded as Part D drugs under Medicare Part D’s basic prescription drug coverage. Medicare Part D plans may offer these excluded drugs, such as obesity drugs, as a supplemental drug under enhanced alternative coverage. A Medicare Part D plan can choose which drugs it covers as a supplemental drug under enhanced alternative coverage—that is, not all plans cover the same supplemental drugs as part of enhanced alternative coverage. Data from CMS on Medicare Part D reimbursement for obesity drugs provide some insight on coverage. For example, our analysis found that in 2017, 27 Medicare Part D plans reimbursed for obesity drugs under enhanced alternative coverage for 209 Medicare beneficiaries. (See table 2 for 2016 and 2017 data.) See appendix VI for more information. Medicaid. State Medicaid programs or Medicaid managed care plans within states may choose either to cover or exclude obesity drugs from coverage. Our analysis found that in 2017, Medicaid programs or Medicaid managed care plans in 41 states reimbursed pharmacies and other providers for at least one claim for an obesity drug, for a total of 30,800 prescriptions. (See table 3 for 2016 and 2017 data.) Medicaid managed care organizations may provide coverage of obesity drugs not covered by the state plan, according to CMS. See appendix VII for more information. Employer-sponsored and individually purchased health plans. Coverage of the nine obesity drugs varied in employer-sponsored and individually purchased health plans, according to the insurers and pharmacy benefit managers we interviewed. For example: Officials from one large insurer told us that coverage of obesity drugs is included in plans for about 90 percent of their members; only a small percentage of members do not have plans with this coverage. Officials from another large insurer surveyed its health plans in different geographic locations and found that, of those that responded, four of the six employer-sponsored and three of the six individually purchased health plans covered the nine obesity drugs. They said that many of the plans that covered obesity drugs in their employer- sponsored markets also covered these drugs in their individual market. Officials at a large pharmacy benefit manager said employers that provide employer-sponsored health insurance can choose to customize their formulary and decide whether to include obesity drugs. They said their select and premium prescription drug formularies include obesity drugs, so companies that decide to offer those formularies would cover obesity drugs, but many companies choose to customize their formularies and may not include obesity drugs. Even if employer-sponsored and individually purchased health plans offer coverage of obesity drugs, these plans often put requirements in place to determine a beneficiary’s eligibility for coverage of obesity drugs, according to officials from insurers and pharmacy benefit managers we interviewed. For example, plans may require beneficiaries to obtain prior authorization, require a determination of medical necessity of the drug for the patient, and review the drug’s effectiveness prior to making a coverage decision. For example, an official from one large insurer told us their drug formulary does not include obesity drugs because the clinical evidence indicates that other therapies are more effective for weight loss. However, this official also said that some of its plans would cover obesity drugs as a nonformulary option if a physician or other health care provider indicates that the obesity drug is medically necessary (e.g., after a patient has tried other treatment options, such as behavioral therapy). Further, if a patient is offered coverage of an obesity drug but fails to receive a clinical benefit within a specified time frame, insurers and pharmacy benefit managers told us the following: A patient and his or her physician may decide together whether the patient should continue or discontinue the obesity drug, and plans often defer to physicians to determine whether an obesity drug is medically necessary for a patient. Some plans may require additional information from a patient’s physician every 6 to 12 months for reapproval of coverage of an obesity drug, such as reporting outcomes (e.g., weight loss) while using the drug. Plans could require prior authorization to continue using an obesity drug. An individual may be able to try a different obesity drug covered by the formulary. For the largest employer-sponsored health care program in the United States—FEHBP, managed by the Office of Personnel Management—we found that some FEHBP plans offered by large insurers excluded obesity drugs from coverage. We examined the formularies for 12 plans offered by three large FEHBP insurers and found that the formularies for two plans from one insurer indicated some type of coverage of obesity drugs in 2018. One plan offered coverage for 50 percent of the plan’s allowed amount for weight management drugs, and the other plan offered coverage of two obesity drugs as tier 2 drugs, which have higher copayments than tier 1 drugs. For individually purchased health plans offered on health care exchanges, nine of the 34 states with federally facilitated exchanges had at least one plan in the silver tier of coverage that included some type of coverage for obesity drugs in 2018, according to a 2018 study. The study found that covered obesity drugs were generally the older drugs and that the newer drugs tended to be covered with higher copayments or more likely to require prior authorizations than other medications. Out-of-pocket payments from the patient or patient’s family made up two- thirds of the amounts paid for obesity drugs, according to nationally representative estimates for 2012 through 2016. These amounts could include insurance copayments and deductible amounts, and payments for obesity drugs not covered by insurance. Private health insurance paid about one quarter of the amount paid for obesity drugs, and Medicare and other public health insurance paid the remainder. Average annual medical spending and prescription drug spending were higher for adults who used any of the nine obesity drugs than for those who did not, according to these estimates. However, the differences in these estimates do not establish any causal relationship between using obesity drugs and having higher average annual medical or prescription drug spending. Out-of-pocket payments made up about two-thirds of total amounts paid for obesity drugs for U.S. adults and private health insurance paid a quarter, according to AHRQ’s nationally representative estimates from MEPS data for 2012 through 2016. Medicare, Medicaid, and other public health insurance paid the remainder; however, estimates for each of these sources of payment are imprecise. (See fig. 2.) Similar to studies on the use of obesity drugs, AHRQ’s estimates also found that 80 percent of amounts paid for any of the nine obesity drugs was for one obesity drug, phentermine, which is available as a generic drug. We also examined available spending data from CMS on payments for obesity drugs and found the following: Medicare Part D prescription drug plans spent $19,714 for obesity drugs in 2016 and $140,296 in 2017, according to our analysis of CMS’s Prescription Drug Event data. These amounts include Medicare Part D plan reimbursements for any of the nine obesity drugs under enhanced alternative coverage. CMS’s data also showed that total beneficiary spending—that is, the total amount Medicare beneficiaries paid out of pocket as copayments or deductibles—for any of these prescriptions totaled $4,048 in 2016 and $5,376 in 2017. See appendix VI for more information. Total Medicaid state and federal spending—that is, reimbursement amounts for the nine obesity drugs—was at least $5,017,424 in 2016 and $7,453,442 in 2017, according to our analysis of available data from CMS’s Medicaid State Drug Utilization data. These amounts do not include all Medicaid spending for obesity drugs under Medicaid managed care. For example, if a Medicaid program pays a managed care organization for drugs as part of their capitated payment for all Medicaid services, they are not reimbursed on a per-drug basis, and obesity drugs covered by Medicaid in that state would show up as a $0 reimbursement amount in CMS’s Medicaid State Drug Utilization data. According to CMS data, Medicaid spending for obesity drugs was the greatest in California in 2016 and 2017. See appendix VII for more information. In addition, when the number of prescriptions dispensed are counted, FDA’s estimates from 2017 IQVIATM data—which are projected nationally from prescriptions dispensed in about 59,900 outpatient retail pharmacies—found that most prescriptions dispensed for obesity drugs were paid for by private insurance. FDA’s analysis found that almost 64 percent of prescriptions dispensed for any of the nine obesity drugs was paid for by private health insurance, and 35 percent of prescriptions dispensed was paid for by cash (i.e., out-of-pocket) payments paid for by the patient or their family in 2017. The remaining 1 percent of prescriptions dispensed for obesity drugs was paid for by Medicare Part D and Medicaid at an estimated 0.9 percent and 0.1 percent, respectively. For all U.S. adults age 18 to 64, the estimated average annual medical and prescription drug spending per adult was higher for those who used an obesity drug than for those who did not use an obesity drug. Specifically, the estimated average annual medical expenditures were $7,575 per adult who used an obesity drug and $4,302 for those who did not, according to AHRQ’s nationally representative estimates from MEPS data for 2012 through 2016. Further, the estimated average annual prescription drug expenditures per adult were $2,198 for those who used an obesity drug and $1,111 for those who did not. However, these data do not necessarily indicate that use of obesity drugs leads to higher average annual medical and prescription drug spending. For U.S. adults with obesity, there was not a significant difference between the estimated average annual medical and prescription drug expenditures per adult for those who used an obesity drug and those who did not use an obesity drug. This may be due to the small sample size of 279 adults with obesity who used an obesity drug in the MEPS data. Appendix VIII provides more information on AHRQ’s estimated expenditures for obesity drugs and other medical and prescription drug spending. We did not identify any studies other than AHRQ’s estimates from MEPS data that specifically addressed the medical spending for adults who used obesity drugs compared to those who did not. We provided a draft of this report to HHS for review and comment. HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Health and Human Services, appropriate congressional committees, and other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IX. The Bipartisan Budget Act of 2018 included a provision for GAO to review the prevalence of obesity and the use of obesity drugs in the Medicare and non-Medicare populations, including spending for and coverage of these drugs. We examined (1) the prevalence of obesity among adults in the United States; (2) what is known about the use of obesity drugs and the individuals who use them; (3) what is known about health insurance coverage of obesity drugs; and (4) what is known about spending on obesity drugs and about medical spending for adults who used obesity drugs compared to those who did not. To address our reporting objectives, we examined estimates from federal agencies within the Department of Health and Human Services (HHS), including the Centers for Disease Control and Prevention’s (CDC) estimates from the National Health and Nutrition Examination Survey (NHANES), the Agency for Health Care Research and Quality’s (AHRQ) estimates from the Medical Expenditure Panel Survey (MEPS), and the Food and Drug Administration’s (FDA) estimates from IQVIA and the Sentinel System. We also analyzed Medicare Part D Prescription Drug Event data and Medicaid State Drug Utilization data from the Centers for Medicare & Medicaid Services (CMS). For each data source, we examined the latest available data at the time of our review. In addition, we conducted a literature review; interviewed officials and reviewed documents from stakeholder organizations, federal agencies, insurers, and others; and examined relevant laws and regulations. We examined CDC’s nationally representative estimates from NHANES of the prevalence of obesity among U.S. adults and use of obesity drugs. NHANES is a cross-sectional survey designed to monitor the health and nutritional status of the civilian, noninstitutionalized U.S. population. The survey consists of interviews conducted in participants’ homes and standardized physical examinations, including measured height and weight, conducted in mobile examination centers. CDC analyzed data from two 2-year cycles of NHANES (2013 through 2014 and 2015 through 2016) for the prevalence of obesity [defined as a body mass index (BMI) of 30 or higher] for all adults by age (18 and older, 18 through 64, and 65 and older), health insurance coverage, and class of obesity. The insurance categories were mutually exclusive: (1) Medicare, which includes all adults who reported having Medicare, regardless of whether they reporting having another type of health insurance (e.g., private health insurance) in addition to Medicare; (2) private health insurance (excluding individuals with Medicare); (3) Medicaid/public health insurance (excluding Medicare); and (4) uninsured. We also examined CDC’s estimates from NHANES on the prevalence of overweight (defined as a BMI of 25 to <30) among U.S. adults. In addition, we examined CDC’s estimates from NHANES for 2013 through 2016 on adults who took prescription medications for weight loss. NHANES asks participants if they tried to lose weight, and, for those who did, if they took diet pills prescribed by a doctor. CDC’s estimates included the lower and upper bounds of the 95 percent confidence intervals (the interval that would contain the actual population value for 95 percent of the samples NHANES could have drawn). We examined AHRQ’s nationally representative estimates from MEPS data on the use of and payment sources for obesity drugs. MEPS collects nationally representative data on health care use, expenditures, sources of payment, and insurance coverage for the U.S. civilian, noninstitutionalized population. For this analysis, AHRQ estimated the distribution of payments for obesity drugs using MEPS pooled data for years 2012 through 2016. We also examined AHRQ’s estimates from MEPS of annual expenditures for medical care and all prescription drugs—for those individuals who used obesity drugs and those who did not—and annual expenditures for obesity drugs. AHRQ’s estimates included the lower and upper bounds of the 95 percent confidence intervals. We examined FDA’s nationally projected data on the prescriptions dispensed for obesity drugs from outpatient retail pharmacies using 2017 IQVIA™ National Prescription Audit Extended Insights and IQVIA™ Total Patient Tracker. IQVIA™ is proprietary data that includes data for prescriptions dispensed at approximately 59,900 U.S. outpatient retail pharmacies. FDA analyzed IQVIA data and provided aggregated results for the nationally estimated number of prescriptions dispensed for the nine obesity drugs from U.S. outpatient retail pharmacies, by payment method. These patterns may not apply to other settings of care (e.g., mail-order or specialty pharmacies or clinics). In addition, the analysis captures data when a prescription was dispensed; it does not indicate that the patient took the obesity drug, and it does not indicate if the drug was prescribed off label for something other than weight loss. We examined FDA’s national estimates of prescriptions for obesity drugs dispensed by outpatient pharmacies for new users of obesity drugs (by number of days supplied and by age and gender of patient) from the agency’s Sentinel System. FDA’s Sentinel System uses prescription drug dispensing data from populations with federal or commercial insurance to characterize drug utilization of a large U.S. population with private and public health insurance. FDA examined drug dispensing data from January 1, 2008, through December 31, 2017, from 17 of 18 Sentinel data partners, including Medicare, which contributed fee-for- service enrollee data. FDA analyzed dispensings for 267,836 new users of the nine prescription obesity drugs. FDA estimated the duration of the first treatment episode (in days) for patients’ prescription dispensings for any of the nine obesity drugs using a 14-day episode gap—that is, if there were more than 14 days between exhausting the previous dispensing’s days supplied for that prescription and refilling the prescription, then FDA counted it as a new treatment episode. FDA estimated cumulative treatment duration by summing days’ supply of all dispensings of an obesity drug during a patient’s presence in the database, without regard to time between dispensings. For information on the number of claims for obesity drugs that were reimbursed, the number of plans that provided reimbursement, and the amount reimbursed for obesity drugs under the Medicare prescription drug program known as Medicare Part D, we analyzed Medicare Prescription Drug Event data from CMS for 2016 and 2017. We analyzed Medicare Part D plan reimbursements (payments to pharmacies) and beneficiary spending (the total amount Medicare beneficiaries paid out of pocket as copayments or deductibles) for the nine obesity drugs for claims that CMS’s data coded as reimbursed as a supplemental drug under enhanced alternative coverage. We excluded 1,787 claims in 2016 and 1,775 claims in 2017 for one obesity drug, orlistat (Xenical), that were listed in CMS’s data as covered under Medicare Part D (and were not coded as a supplemental drug under enhanced alternative coverage). According to CMS officials, orlistat has off-label indications including diabetes and hyperlipidemia, and when orlistat is used for these indications the drug would be covered under Medicare Part D, and the Medicare Part D plan is responsible for ensuring it is dispensed appropriately per Medicare Part D policy. We also excluded 25 claims in 2016 and 26 claims in 2017 for prescription obesity drugs listed as over-the-counter in the prescription drug event data because, according to CMS, these appear to be outliers. Because our analysis was limited to those instances in which a Medicare Part D plan reimbursed for an obesity drug as a supplemental drug under enhanced alternative coverage, the number of Medicare Part D plans that provided coverage for obesity drugs could be higher. For example, some plans may have covered obesity drugs, but none of the beneficiaries enrolled in these plans filled a prescription for such a drug. For information on obesity drugs reimbursed by state Medicaid programs or Medicaid managed care programs within those states, we analyzed CMS’s Medicaid State Drug Utilization data for 2016 and 2017. We analyzed the data to estimate the number of prescriptions reimbursed and total Medicaid state and federal spending—that is, the Medicaid amount reimbursed (state and federal reimbursement, including dispensing fees)—for the nine obesity drugs. These amounts do not include all Medicaid spending for obesity drugs because managed care organizations can be paid for the drugs as part of their capitated payment for all Medicaid services, they are not reimbursed on a per-drug basis, and their payments are not recorded in CMS’s Medicaid State Drug Utilization data. Because our analysis was limited to those instances in which Medicaid reimbursed for an obesity drug, the number of states in which state Medicaid programs or Medicaid managed care plans provided coverage for obesity drugs could be higher. For example, a state could have provided coverage for obesity drugs, but no beneficiaries in that state filled a prescription for an obesity drug. We obtained information and reviewed studies from officials from eight stakeholder organizations (representing medical associations and advocacy groups for obesity research and treatment) on the use of obesity drugs and guidelines for using obesity drugs and to obtain their perspectives on what physicians and other health care providers take into consideration when prescribing these drugs, among other things. These stakeholders were selected because of their medical or scientific expertise, relevant publications, or familiarity with the treatment of obesity and obesity drugs. We also reviewed data and documents and interviewed officials from HHS agencies: CDC, FDA, AHRQ, CMS, and the National Institutes of Health. In addition, we reviewed guidance documents and obtained information from the Office of Personnel Management, which administers the Federal Employees Health Benefits program (FEHBP). FEHBP is the largest employer-sponsored health insurance program in the United States, providing health insurance coverage to about 8 million federal employees, retirees, and their dependents in 2016 through contracts with private health insurance plans. We obtained information about the health insurance coverage of obesity drugs from officials from the three largest pharmacy benefit managers, four large insurers, and two organizations knowledgeable about prescription drug benefits for employer-sponsored health plans. We also reviewed drug formularies for selected private health insurance plans, including FEHBP plans, to determine if any of the nine obesity drugs were included. We conducted a literature review of relevant peer-reviewed studies published from January 2012 through January 2019. We identified studies through a search of bibliographic databases, including ProQuest, Scopus, MEDLINE, and International Pharmaceutical Abstracts , using terms such as “obesity,” “weight loss,” and “prescriptions.” Of the 765 citations we identified, we reviewed 220 full studies, which we examined for information related to the use of obesity drugs and individuals who use them, coverage of obesity drugs, and spending for obesity drugs for individuals who used them compared to those who did not. We determined 19 studies were relevant to the use of obesity drugs and 1 study was relevant to coverage of obesity drugs. Our literature review focused on studies with a U.S.-based, adult population (age 18 and older); we excluded studies related to childhood obesity and studies on animals. We also examined available information on the clinical trials conducted prior to FDA approval of the prescription obesity drugs for the U.S. market, including 64 studies from our literature review that summarized one or more of the clinical trials. We also identified 17 additional studies in our literature review that provided relevant background information. Additionally, we reviewed five studies provided by stakeholder organizations (in addition to the studies we had identified in our literature review) that we determined were relevant to our research objectives, as well as guidelines for the use of obesity drugs in obesity treatment. To determine the reliability of the data we used for all four objectives— CDC’s estimates from NHANES, AHRQ’s estimates from MEPS, FDA’s data from IQVIA and the Sentinel System, and CMS’s Medicare Part D Prescription Drug Event data and Medicaid State Drug Utilization data— we reviewed documentation on data collection processes and discussed limitations of the data with the relevant federal agency officials. In addition, we conducted data reliability checks on the data, when appropriate. We determined the data used in this report were sufficiently reliable for our purposes. We conducted this performance audit from April 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 4 provides additional information on the nine prescription drugs approved by the Food and Drug Administration (FDA) to treat obesity that we included in our review. This appendix presents national estimates of the prevalence of obesity among U.S. adults age 18 and older, based on the Centers for Disease Control and Prevention’s (CDC) estimates from the National Health and Nutrition Examination Survey (NHANES) for 2013 through 2016. It presents the estimates and the ranges for the 95 percent confidence intervals for prevalence of obesity by age and class of obesity (see table 5), and by insurance coverage and class of obesity (see table 6). It also presents national estimates of the prevalence of overweight (defined as a body mass index of 25 to <30) among U.S. adults, by age and insurance coverage (see table 7). Table 8 is a list of selected studies, categorized by specific topic area, that we reviewed that pertain to our research objectives, including information related to the use of obesity drugs and individuals who use them, physician considerations in prescribing obesity drugs, and health insurance coverage of obesity drugs. We identified these studies either through our literature review of peer-reviewed studies published from January 2012 through January 2019 or from one of the stakeholder organizations we contacted. This appendix presents estimates of prescriptions dispensed for new adult users of obesity drugs by duration of use and by age and gender, using data from the Food and Drug Administration’s (FDA) Sentinel System from 2008 through 2017. Of the 267,836 new users of obesity drugs included in this analysis, the first treatment episode did not exceed 30 days in about 54 percent of patients and exceeded 90 days in about 22 percent of patients. Cumulatively, about 42 percent of patients who used any of the obesity drugs did so for more than 90 days across treatment episodes. (See table 9.) Overall, most new users of obesity drugs were female (82.2 percent) and under age 65 (91.7 percent). (See table 10.) Phentermine and bupropion/naltrexone (Contrave) were the most commonly used obesity drugs in FDA’s Sentinel System analysis. This appendix presents information on Medicare Part D plan reimbursement for obesity drugs under enhanced alternative coverage from our analysis of Centers for Medicare & Medicaid Services’ (CMS) Prescription Drug Event data. Medicare Part D plans can choose whether or not to offer enhanced alternative coverage, and not all Medicare Part D plans that provide enhanced alternative coverage cover obesity drugs as supplemental drugs. As of February 2017, 1,949 Medicare Part D plans provided enhanced alternative coverage to 18.9 million Medicare beneficiaries, according to the Medicare Payment Advisory Commission. Additionally, in 2015, total Medicare Part D spending for prescription drugs was about $137 billion—this represents payments from all payers including beneficiaries (cost sharing), and excluding rebates and discounts from pharmacies and manufacturers that are not reflected in prices at the pharmacies. Tables 11 and 12 show the number of claims reimbursed, the number of plans that provided reimbursement, and the amount reimbursed for obesity drugs under Medicare Part D enhanced alternative coverage for 2016 and 2017, respectively. This appendix presents information on Medicaid reimbursements for obesity drugs under state Medicaid programs or Medicaid managed care programs within those states from our analysis of Centers for Medicare & Medicaid Services’ (CMS) Medicaid State Drug Utilization data. State Medicaid programs or Medicaid managed care programs reimbursed for at least one obesity drug prescription in 42 states in 2016 and 41 states in 2017. The amount that Medicaid reimbursed and the total number of prescriptions for obesity drugs reimbursed by Medicaid in 2016 and 2017 are shown by state (tables 13 and 14), and by obesity drug (tables 15 and 16). Over half of the prescriptions for obesity drugs reimbursed under Medicaid in 2016 and 2017 were for the generic obesity drug, phentermine. This appendix presents nationally representative estimates of U.S. adults’ average annual expenditures (spending) for medical care, all prescription drugs, and for obesity drugs from the Agency for Healthcare Research and Quality (AHRQ) based on data from the Medical Expenditure Panel Survey (MEPS) for 2012 through 2016. Table 17 shows the estimated average annual expenditures for all prescription drugs and table 18 shows the estimated average annual medical expenditures, including prescription drugs, per adult who used and per adult who did not use any obesity drugs. For adults age 18 to 64, the differences in the estimated average annual expenditures for all medical care and for all prescriptions drugs per adult who used and who did not use any of the nine obesity drugs in our review were statistically significant. However, the differences in these estimates do not indicate that there was a causal relationship between using obesity drugs and having higher average annual medical or prescription drug expenditures. Table 19 shows the estimated average annual expenditures per adult for obesity drugs. In addition to the contact above, Kim Yamane (Assistant Director), Lisa A. Lusk (Analyst-in-Charge), George Bogart, Zhi Boon, Kaitlin Dunn, Laurie Pachter, and Merrile Sing made key contributions to this report. Also contributing to this report were Alexander Cattran, Leia Dickerson, Diona Martyn, Christina Ritchie, and Ethiene Salgado-Rodriguez.", "summary": "Obesity has been associated with an increased risk of developing conditions such as heart disease, stroke, diabetes, and certain types of cancer. Treatment options for individuals with obesity include lifestyle therapy, such as diet, exercise, and behavioral counseling; obesity drugs; surgery; or a combination of these efforts. The Bipartisan Budget Act of 2018 (P.L. 115-123) included a provision for GAO to review the prevalence of obesity and the use and insurance coverage of obesity drugs. This report examines the prevalence of obesity in the United States, and what is known about the use and health insurance coverage of obesity drugs, among other objectives. GAO examined data from agencies within the Department of Health and Human Services (HHS) on the prevalence of obesity (using estimates for 2013 through 2016) and the use, spending, and coverage of obesity drugs; conducted a literature review of relevant studies published from January 2012 through January 2019 in peer-reviewed and other publications; reviewed drug formularies for selected health plans; and reviewed documents and interviewed officials from federal agencies and stakeholder organizations (including medical associations, advocacy groups, pharmacy benefit managers, and insurers). HHS provided technical comments on a draft of this report, which were incorporated as appropriate. The prevalence of obesity—that is, body weight higher than what is considered a healthy weight for a given height—was about 38 percent among all U.S. adults, according to the latest available national estimates at the time of GAO's analysis. This prevalence was similar for adults with different types of health insurance. Treatment for adults with obesity may include one or more of nine prescription drugs that the Food and Drug Administration has approved for weight management (i.e., obesity drugs), though relatively few adults have used these drugs. Of an estimated 71.6 million U.S. adults with obesity, an estimated 660,000 per year, on average, used an obesity drug from 2012 through 2016, according to national estimates. Among adults who reported trying to lose weight, about 3 percent reported that they took prescription medication for weight loss from 2013 through 2016, according to national estimates. Coverage of obesity drugs varied across different types of health insurance, including Medicare and Medicaid. Plans cited factors such as low consumer demand and strong evidence supporting other treatments in their coverage decisions. GAO's analysis of Centers for Medicare & Medicaid Services' data indicates that some Medicare prescription drug plans and state Medicaid programs reimbursed for some obesity drugs in 2016 and 2017. Coverage for private health insurance plans also varied, and plans may require the patient to obtain prior authorization for the drugs to be covered, according to officials from insurers and pharmacy benefit managers GAO interviewed. For example, officials from one insurer said that some of their plans only cover obesity drugs after a patient has tried other treatment options such as behavioral counseling.", "document_type": "gao"}
{"report": "In September 2017, two Category 5 hurricanes struck the USVI, causing catastrophic damage across the entire territory and neighboring Caribbean islands. On September 6, 2017, Hurricane Irma struck St. Thomas and St. John and on September 19, 2017, Hurricane Maria struck St. Croix (see fig. 1). The storms severely damaged the territory’s critical infrastructure, devastating more than 90 percent of aboveground power lines and shutting down electricity and telecommunications for months. Further, 52 percent of the territory’s housing units were damaged, ports and airports were closed for weeks, and hundreds of thousands of tons of debris were generated, often blocking roads and making transportation hazardous. In addition, according to a September 2018 report from the USVI Hurricane Recovery and Resilience Task Force, the territory’s economic activity— especially tourism—was severely reduced in the months following the storms, leading to job losses and a total estimated economic impact of $1.54 billion. In response to the request of the Governor of the USVI, the President declared a major disaster the day after each hurricane struck the territory. Major disaster declarations can trigger a variety of federal response and recovery programs for government and nongovernmental entities and households and individuals, including assistance through the Public Assistance program. Under the National Response Framework and National Disaster Recovery Framework, DHS is the federal department with primary responsibility for coordinating disaster response and recovery, and within DHS, FEMA has lead responsibility. The Administrator of FEMA serves as the principal adviser to the President and the Secretary of Homeland Security regarding emergency management. FEMA’s Public Assistance program provides funding to state, territorial, local, and tribal governments as well as certain types of private nonprofit organizations to assist with responding to and recovering from major disasters or emergencies. As shown in figure 2, Public Assistance program funds are categorized broadly as “emergency work” or “permanent work.” Within these broad categories are separate subcategories. In addition to the emergency work and permanent work categories, the program includes category Z, which represents indirect costs, administrative expenses, and other expenses a recipient or subrecipient incurs in administering and managing the Public Assistance program that are not directly chargeable to a specific project. FEMA’s Public Assistance program also provides funding for cost- effective hazard mitigation measures to reduce or eliminate the long-term risk to people and property from future natural and man-made disasters and their effects. Specifically, FEMA provides funding for hazard mitigation measures in conjunction with the repair of disaster-damaged facilities to enhance their resilience during future disasters. For example, a community that had a fire station damaged by a disaster could use Public Assistance funding to repair the facility and incorporate additional measures such as installing hurricane shutters over the windows to mitigate the potential for future damage. Once the President has declared a disaster, FEMA, the state or territorial government (the recipient), and local or territorial entities (the subrecipient) work together to develop damage assessments and formulate project worksheets for eligible projects. Project worksheets detail the scope of work and estimated cost for repairing or replacing disaster-damaged infrastructure as well as any hazard mitigation measures that may help to increase the resilience of this infrastructure during future disasters. After a project has completed FEMA’s review process and is approved, FEMA obligates funding for the project by placing money into an account where the recipient has the authority to draw down—or withdraw—funding to pay the subrecipient for eligible work upon completion (see fig. 3). In addition, a state or territorial governor may designate a governor’s authorized representative (GAR) to oversee all aspects of disaster assistance, including Public Assistance funding. Specifically, the GAR is responsible for ensuring compliance with program requirements by providing oversight into how goods and services are procured for projects, such as construction materials or modular school units. The GAR also confirms that subrecipients submit complete documentation demonstrating that all work completed is in accordance with a project’s approved scope of work and Public Assistance program requirements. The GAR then approves the paperwork and the recipient can draw down funding from the account holding the obligations to reimburse subrecipients for completed work. When a project has been completed, FEMA conducts a close-out process to certify that all work has been completed and reconciles the actual cost incurred. If the actual cost of the completed work is greater than the amount of money FEMA obligated for the project, FEMA will reimburse the subrecipient for these additional costs. The Sandy Recovery Improvement Act of 2013 authorized the use of alternative procedures in administering the Public Assistance program, thereby providing new flexibilities to FEMA, states, territories, and local governments for debris removal, infrastructure repair, and rebuilding projects using funds from this program. Unlike in the standard Public Assistance program where FEMA will fund the actual cost of a project, the Public Assistance alternative procedures allow awards for permanent work projects to be made on the basis of fixed-cost estimates to provide financial incentives for the timely and cost-effective completion of work. Under these procedures, if the actual cost of the project exceeds the fixed-cost estimate agreed upon by FEMA and the recipient, the recipient or subrecipient is responsible for the additional costs at the time of the close-out process. However, if the actual cost of completing eligible work for a project is below the estimate, the recipient may use the remaining funds for additional cost-effective hazard mitigation measures to increase the resilience of public infrastructure. In addition, these funds may also be used for activities that improve the recipient’s or subrecipient’s future Public Assistance operations or planning. As of October 1, 2018, FEMA had obligated more than $1.4 billion through the standard Public Assistance program for 475 projects across the USVI. As shown in figure 4, FEMA obligated funding for both emergency work and permanent work projects. As of October 1, 2018, of the more than $1.4 billion FEMA obligated, the USVI had expended approximately $586.9 million—about 41 percent of total Public Assistance program obligations to the USVI—to reimburse subrecipients for completed work. Of this $586.9 million, the USVI had expended about $532.8 million (91 percent) for emergency work projects in categories A and B and $49.1 million (8 percent) for permanent work projects in categories C through G. The majority of FEMA’s obligations and the funding the USVI expended as of October 1, 2018, are for emergency work because these projects began soon after the disasters struck and focused on debris removal and providing assistance to address immediate threats to life and property. In contrast, permanent work projects take time to identify, develop, and ultimately complete as they represent the longer-term repair and restoration of public infrastructure. Emergency work. Of the more than $1.4 billion FEMA had obligated as of October 1, 2018, about $873.8 million (60 percent) was obligated for 322 emergency work projects in Public Assistance categories A and B. Category A: Debris Removal. FEMA obligated about $94.0 million for 71 projects focused on debris removal activities across the territory. For example, FEMA obligated $45.0 million to the USVI Department of Public Works for territorywide debris removal efforts and $39.1 million to the USVI Water and Power Authority for these activities in St. Croix (see fig. 5). Of the $94.0 million FEMA obligated for debris removal, the USVI had expended about $54.6 million (58 percent) as of October 1, 2018. Category B: Emergency Protective Measures. FEMA obligated about $780 million for 251 projects focused on emergency measures. For example, FEMA obligated about $187 million for the Sheltering and Temporary Essential Power program, which is intended to provide essential repairs or restore power to private residences to allow affected individuals to return or remain in their homes, thereby reducing the demand for other shelter options. In addition, FEMA obligated approximately $101 million for the purchase and installation of modular units to be used as temporary classrooms and other facilities while permanent school buildings are repaired or replaced (see fig. 6). Of the $780 million FEMA obligated for emergency protective measures, the USVI had expended about $478 million (61 percent) as of October 1, 2018. Permanent work. Of the more than $1.4 billion in Public Assistance funding FEMA had obligated as of October 1, 2018, about $516.3 million (36 percent) was obligated for 153 permanent work projects across categories C through G. These permanent work projects include about $349.4 million for cost-effective hazard mitigation measures aimed at reducing the future risk of disaster-damaged facilities in conjunction with their repair. Further, of the $516.3 million FEMA obligated for permanent work in the USVI, approximately $500.4 million—or 97 percent of all permanent work obligations—was obligated to the USVI Water and Power Authority for the permanent repair of electrical distribution systems and other utilities across the territory. Category C: Roads and Bridges. FEMA obligated about $5.2 million for 35 projects focused on repairing roads and bridges in the territory, 18 of which included hazard mitigation measures totaling about $1.5 million. For example, FEMA obligated about $410,000 for one project to repair a road on St. Thomas damaged by floodwaters. This project included approximately $227,000 for hazard mitigation measures, such as replacing the damaged road surface with reinforced concrete and building a retaining wall. As of October 1, 2018, the USVI had not expended funding in this category. Category D: Water Control Facilities. As of October 1, 2018, FEMA did not have any projects in this category. According to FEMA officials, the USVI does not have water control infrastructure that would fall under category D, such as dams, levees, or berms. Category E: Buildings and Equipment. FEMA obligated $6.0 million for 77 projects focused on repairing and rebuilding damaged buildings and equipment, 16 of which included hazard mitigation measures totaling about $1.8 million. For example, FEMA obligated about $1.5 million to repair damage to the airport terminal building in St. Thomas—a project where hazard mitigation measures comprised 87 percent of the project’s total cost (see fig. 7). These measures include replacing the terminal’s roof with materials designed to withstand higher wind speeds to increase the building’s resilience during future storms. Of the $6.0 million FEMA obligated for category E, the USVI had expended about $148,000 (2.5 percent) as of October 1, 2018. Category F: Utilities. Of the $516.3 million FEMA obligated for permanent work projects, $502.2 million (97 percent) was obligated for 15 projects focused on repairing utilities, 7 of which included hazard mitigation measures totaling about $346.0 million. For example, FEMA obligated $286.1 million and $50.2 million for permanent electrical distribution system repairs in St. Croix and St. John, respectively. This includes replacing damaged wooden utility poles with more resilient composite fiberglass poles that can withstand 200 mile per hour winds as well as power transmission lines and transformers (see fig. 8). Of the $502.2 million FEMA obligated for category F, the USVI had expended about $49.0 million (10 percent) as of October 1, 2018. Category G: Parks, Recreational, and Other Facilities. FEMA obligated about $2.9 million for 26 projects focused on repairing parks, playgrounds, and other recreational facilities, 1 of which included hazard mitigation measures. Specifically, FEMA obligated about $453,000 to repair the Lindbergh Park and Water Playground in St. Thomas—a project that included about $18,000 for hazard mitigation measures. As of October 1, 2018, the USVI had not expended funding in this category. Future projects. In addition to the more than $1.4 billion in Public Assistance funding FEMA had obligated as of October 1, 2018, FEMA expected to review an additional 900 future projects for eligibility representing an estimated $779.4 million in potential funding. Of this estimated total amount, FEMA anticipates $128.5 million (16 percent) in costs for future emergency work projects and $650.9 million (84 percent) in costs for future permanent work projects. In July 2018, FEMA approved a June 2018 request from the Governor of the USVI to transition to using the Public Assistance alternative procedures program for permanent work in the territory. The alternative procedures provide new flexibilities to FEMA and the USVI that are not available through the standard Public Assistance program. In September 2018, FEMA issued the Public Assistance Alternative Procedures Permanent Work Guide for the USVI to provide guidance on the implementation of the program in the territory. FEMA and USVI officials stated that a section of the Bipartisan Budget Act of 2018 and the flexibilities provided by the program itself influenced the USVI’s decision to transition to using the alternative procedures. First, Section 20601 of the Bipartisan Budget Act of 2018 authorized FEMA, when using the Public Assistance alternative procedures, to provide assistance to fund the replacement or restoration of disaster- damaged infrastructure that provide critical services to industry standards without regard to pre-disaster condition. FEMA and USVI officials told us that the territory therefore has a valuable opportunity to use the alternative procedures to repair and rebuild its critical services infrastructure—including the USVI’s education system, electrical grid, and emergency medical care system, among others—so it is in a better condition than it was prior to the 2017 hurricanes. Second, USVI officials stated that under the standard Public Assistance program currently being used in the USVI, the territorial government is responsible for providing the initial funding to reimburse subrecipients for completed work prior to drawing down funds from the account holding the FEMA- obligated amounts of money for each project. They explained that because of the financial liquidity challenges facing the territory, this process was problematic and required USVI officials to prioritize projects based on the availability of the territory’s funding. USVI officials stated that the Public Assistance alternative procedures will help to address this challenge by providing the territory with more flexibility regarding when and how to fund projects. For example, in certain cases, the USVI is able to consolidate permanent work projects approved under the alternative procedures and share obligated funding across these projects. In addition, the USVI is able to use any excess funds for cost-effective hazard mitigation measures or for activities that improve the recipient’s or subrecipient’s future Public Assistance operations or planning. As of November 2018, FEMA and USVI officials stated they were working to identify and develop permanent work projects using the Public Assistance alternative procedures and discussing the process for developing the fixed-cost estimate for each project. Specifically, unlike in the standard Public Assistance program where FEMA will fund the actual cost of a project, the Public Assistance alternative procedures use a fixed-cost estimate which is agreed to prior to obligation and the USVI will be financially responsible for any actual costs that exceed this amount. Given the USVI’s difficult fiscal situation, FEMA and USVI officials stated that ensuring these fixed-cost estimates are as accurate as possible will be critical. However, FEMA officials also noted that if FEMA and the territory cannot come to an agreement on a fixed-cost estimate for any given project, the USVI does have the option to move forward through the standard Public Assistance program. According to FEMA’s Public Assistance Alternative Procedures Permanent Work Guide for the USVI, all cost estimates for projects using these procedures must be finalized by March 2020. We will continue to monitor the USVI’s plans for using the alternative procedures as part of our broader work assessing disaster recovery efforts in the USVI and will issue a follow-on report later this year. We provided a draft of this report to DHS and the USVI government. We requested comments from DHS and the USVI government, but none were provided. DHS did provide technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the Administrator of FEMA, the USVI government, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you and your staff have any questions, please contact me at (202) 512- 8777 or curriec@gao.gov. GAO staff who made key contributions to this report are listed in appendix II. Chris Currie, 202-512-8777 or curriec@gao.gov. In addition to the contact named above, Joel Aldape (Assistant Director), Bryan Bourgault, Leanna Diggs, Aaron Gluck, Eric Hauswirth, Brian Lipman, Amanda Miller, Heidi Nielson, and Kevin Reeves made key contributions to this report.", "summary": "In September 2017, two major hurricanes—Irma and Maria—struck the USVI, causing billions of dollars in damage to its infrastructure, housing, and economy. FEMA—a component of the Department of Homeland Security—is the lead federal agency responsible for assisting the USVI as it recovers from these natural disasters. Among other responsibilities, FEMA administers the Public Assistance program in partnership with the USVI territorial government, providing the USVI grant funding for response and recovery activities, including debris removal efforts, life-saving emergency protective measures, and the repair, replacement, or restoration of public infrastructure. GAO was asked to review the federal government's response and recovery efforts related to the 2017 hurricanes. This report describes (1) the status of FEMA's Public Assistance program funding provided to the USVI in response to the 2017 hurricanes as of October 1, 2018, and (2) the USVI's transition to implementing the Public Assistance alternative procedures in the territory. GAO reviewed program documents and data on obligations and expenditures as of October 1, 2018, and interviewed officials from FEMA and the USVI regarding the Public Assistance program specifically and disaster recovery efforts more generally. GAO also conducted site visits to the USVI islands of St. Croix, St. Thomas, and St. John. GAO is not making any recommendations in this report, but will continue to monitor the progress of the USVI's recovery as part of its ongoing work. The Federal Emergency Management Agency (FEMA) obligated more than $1.4 billion in grant funding for Public Assistance projects in the U.S. Virgin Islands (USVI) as of October 1, 2018, in response to the 2017 hurricanes. FEMA obligated about $873.8 million for emergency work—debris removal activities and emergency measures to lessen the immediate threat to life, public health, and safety—and about $516.3 million for permanent work—including the repair or replacement of public infrastructure such as roads, electrical utilities, and schools. For example, FEMA obligated about $101 million for the purchase and installation of modular units to be used as temporary classrooms and other facilities while permanent school buildings are repaired or replaced. FEMA's obligations for permanent work also included funding for hazard mitigation measures to reduce the risk of damage during future storms—for example, by replacing wooden utility poles with composite fiberglass poles (see figure). FEMA and the USVI are transitioning from using the standard Public Assistance program in the territory to using the Public Assistance alternative procedures program. Unlike in the standard Public Assistance program where FEMA will fund the actual cost of a project, the alternative procedures allow awards to be made on the basis of fixed-cost estimates to provide financial incentives for the timely and cost-effective completion of permanent work projects. FEMA and USVI officials stated that the alternative procedures will give the USVI more flexibility in determining when and how to fund projects and provide an opportunity to repair and rebuild the USVI's critical services infrastructure—such as its education system and electrical grid—so it meets industry standards without regard to pre-disaster condition. As of November 2018, FEMA and USVI officials were discussing the process for developing projects under the Public Assistance alternative procedures. GAO will continue to monitor the USVI's plans for using the alternative procedures as part of its broader review assessing the USVI's disaster recovery efforts and will issue a follow-on report later this year.", "document_type": "gao"}
{"report": "Individuals engage in countless online transactions every day—from checking their bank accounts and making retail purchases to signing up for federal benefits and services. However, securing such transactions is a complex endeavor. A key part of this process is verifying that the person who is attempting to interact for the first time with an organization, such as a federal agency or a business, is the individual he or she claims to be. This process, known as identity proofing, is essential to prevent fraud, which could cause harm to both individuals and organizations. Identity proofing may occur in person or through a remote, online process. In the case of in-person identity proofing, a trained professional verifies an individual’s identity by making a direct physical comparison of the individual’s physical features and other evidence (such as a driver’s license or other credential) with official records to verify the individual’s identity. Verification of these credentials can be performed by checking electronic records in tandem with physical inspection. In-person identity proofing is considered a strong method of identity proofing. However, it may not always be feasible to require that all applicants appear in person. In such cases, remote identity proofing is performed. Remote identity proofing is the process of conducting identity proofing entirely through an online exchange of information. When remote identity proofing is used, there is no way to confirm an individual’s identity through their physical presence. Instead, the individual provides the information electronically, or performs other electronically verifiable actions that demonstrate his or her identity. Because many federal benefits and services are offered broadly to large numbers of geographically dispersed applicants, agencies often rely on remote identity proofing to verify the identities of applicants. Remote identity proofing involves two major steps: (1) resolution and (2) validation and verification. During the resolution step, an organization determines which specific identity an applicant is claiming when they first attempt to initiate a transaction, such as enrolling for federal benefits or services, remotely. The most common form of remote interaction is through an organization’s website. The organization starts the identity resolution process by having the applicant provide identifying information, typically through a web-based application form. Examples of information that an organization may collect for identity resolution include name, address, date of birth, and Social Security number. The organization then electronically compares the applicant’s identifying information with electronic records that it already has in its databases or with records maintained by another entity, such as a CRA, to determine (or “resolve”) which identity is being claimed. For example, if an individual named John Smith were applying, the organization would obtain enough identifying information about him to determine which “John Smith” he is from among the thousands of John Smiths that it may have in its records or that may be documented in the records of the CRA that it is using for this process. Once the resolution process is complete, the process of validation and verification occurs. In this process, steps are taken to verify whether the applicant is really who they claim to be. For example, in the case of John Smith, it is not enough simply to determine which John Smith is being claimed, because the claimant may not really be John Smith at all. Organizations need to obtain electronic evidence from the remote applicant to verify their identity. Organizations can use a variety of techniques to accomplish this goal. Knowledge-based verification is a technique that commonly has been used for this purpose. With knowledge-based verification, organizations ask applicants detailed and personal questions, under the presumption that only the real person will know the answers to these questions. To do this, the organization poses a series of multiple choice questions through an online web form, and the applicant selects the appropriate responses and submits the answers through the web form. If the applicant has chosen the correct responses, through the remotely accessed web form, their identity is considered to be verified, and the validation and verification step is complete. Figure 1 depicts the typical process that organizations use for remote identity proofing (including the use of knowledge-based verification). As previously mentioned, to perform knowledge-based verification for remote identity-proofing, federal agencies and other organizations often use services provided by CRAs. The CRAs assemble and evaluate consumer credit and other information from a wide variety of sources. Equifax, Experian, and TransUnion—the three nationwide CRAs—use the personal information they obtain about individuals from organizations, such as financial institutions, utilities, cell phone service providers, public records, and government sources, to compile credit files containing detailed records about individuals. They then use the information in these files to offer a variety of services to federal agencies and other entities. These services can include identity verification, as well as verification of income and employment of a candidate for a job or an applicant for benefits or services. To support organizations that rely on knowledge-based verification, CRAs generate multiple choice questions that organizations can use to test applicants’ knowledge of information in their credit files. The organizations using the CRA services do not generate the questions themselves, because they do not have access to the credit history information maintained by the CRAs. Rather, the CRAs’ remote identity proofing systems transmit the questions and multiple choice answers to the organization through an automated electronic connection with the organization’s website. The organization’s website then displays the questions and multiple choice answers to the applicant through the web application that the applicant is using to apply for access to benefits or services. Typically, the questions generated by CRA identity proofing systems ask about lenders, mortgage details, current and past home addresses, or credit card accounts. Once the applicant has selected answers to the questions and enters them in the online application, the organization’s automated system electronically relays the applicant’s responses to the CRA’s remote identity proofing system; this system then compares the responses with information in the applicant’s credit file. If this comparison determines that the applicant correctly responded to the questions, then the applicant’s identity is considered to be verified. The CRA’s identity proofing system electronically transmits the results of its comparison to the organization’s website to allow the applicant, whose identity is now considered verified, to proceed with applying for benefits or services. The Federal Information Security Modernization Act of 2014 (FISMA) is intended to provide a comprehensive framework for ensuring the effectiveness of security controls over information resources that support federal operations and assets, as well as the effective oversight of information security risks. FISMA assigns responsibility to the head of each agency to provide information security protections commensurate with the risk and magnitude of the harm resulting from unauthorized access, use, disclosure, disruption, modification, or destruction of information systems used or operated by an agency or on behalf of an agency. FISMA assigns responsibility to NIST for developing comprehensive information security standards and guidelines for federal agencies. These include standards for categorizing information and information systems according to ranges of risk levels and guidelines for establishing minimum security requirements for federal information systems. To fulfill its FISMA responsibilities, NIST has issued technical guidance on many different aspects of information security, including identity proofing. NIST issued its first guidance related to identity proofing in 2011. In 2017, NIST released an updated version of its guidance, which included guidance on identity proofing that outlines technical requirements for resolving, validating, and verifying an identity based on evidence obtained from a remote applicant. OMB requires agencies to implement NIST’s technical guidance on information security subjects within one year of issuance. In the case of NIST’s updated guidelines for remote identity proofing, agencies would have needed to implement the guidance by June 2018 to meet OMB’s time frames. FISMA assigns responsibility to OMB for overseeing agencies’ information security policies and practices. OMB, in turn, has established requirements for federal information security programs and has assigned agency responsibilities to fulfill the requirements of statutes such as FISMA. OMB policies and guidance require agencies to employ a risk- based approach and decision making to ensure that security and privacy capabilities are sufficient to protect agency assets, operations, and individuals. OMB has not issued guidance to agencies specifically on identity proofing. However, OMB developed a draft policy document in April 2018 that is intended to provide guidance to agencies on strengthening the security of information and information systems to ensure safe and secure access to federal benefits and services. While it has not yet been issued, the draft policy indicates that OMB intends to provide policy- level guidance for agencies to identify, credential, monitor, and manage user access to information and information systems and adopt sound processes for authentication and access control. The six agencies that we reviewed rely on a variety of remote identity proofing techniques, including knowledge-based verification, to ensure that the individuals who enroll for federal benefits and services are who they claim to be. These agencies typically use knowledge-based verification services offered by CRAs, which generate questions for the individuals applying for benefits or services and check the applicants’ answers to verify their identity. However, to the extent that they use knowledge-based verification, these agencies face risks because an attacker could obtain and use an individual’s personal information to answer knowledge-based verification questions and successfully impersonate that individual. Although commonly used by federal agencies for remote identity proofing, knowledge-based verification techniques pose security risks because an attacker could obtain and use an individual’s personal information to answer knowledge-based verification questions and successfully impersonate that individual. As such, NIST’s 2017 guidance on remote identity proofing effectively prohibits the use of knowledge-based verification for sensitive applications. The guidance states that the ease with which an attacker can discover the answers to many knowledge- based questions and the relatively small number of possible responses cause the method to have an unacceptably high risk of being successfully compromised by an attacker. In its guidance, NIST states that the agency no longer recommends using knowledge-based verification because it tends to be error-prone and can be frustrating for users, given the limitations of human memory. According to NIST officials, private-sector providers of remote identity proofing solutions and officials at the agencies we reviewed, alternative methods for verifying an individual’s identity are available that are not knowledge-based and can provide stronger security assurance than knowledge-based verification. Specific examples of such techniques include: Remote assessment of physical credentials. Recently developed technology allows an agency to remotely examine a physical credential, such as a driver’s license or a passport, to verify an individual’s identity. For example, an agency may have the individual use their mobile device, such as a cell phone, to capture and submit an image of their driver’s license to an agency or commercial provider of identity proofing services. The agency or commercial provider can then compare the image to documentation on file to confirm the authenticity of the credential. Technological advances in how images are captured and processed by mobile devices, such as cell phones, can provide improved assurance that the photos transmitted by these devices are genuine and that the credentials are authentic. Verification of mobile device possession. Many individuals use their cell phones on a near-continuous basis and keep their phones with them. These actions create a record of the owner’s connection with these mobile devices that is difficult for an imposter to falsify. Accordingly, an organization can query records maintained by cell phone carriers to verify the identity of an individual who is in possession of a specific mobile device and phone number. By doing this, the organization can determine how long the individual has had that particular device, compare unique identifiers, and determine if the location matches the individual’s billing information. The organization can be confident that the individual legitimately possesses the device if the device has been in use for some time and its current location corresponds to one where the device has been known to be used by its owner. Since an individual’s location information is obtained directly from the device and compared with cell phone carrier records, data entry errors by the individual, such as mistyping a phone number, are minimized and the risk of impersonation is reduced. Verification through mobile device confirmation codes. An additional method that organizations use to help verify an individual’s identity is to verify that an individual possesses a telephone number that they have supplied as part of the remote identity proofing process. Organizations perform verification of an individual’s possession of a phone number by sending a code to that phone number through the short message service (SMS) or another protocol, and ask the individual to enter the code into the online identity proofing application. This process can provide additional assurance about the individual’s identity because the verification code is transmitted through a separate electronic channel (specifically, the telephone system) from the online application where the remote identity proofing process was initiated. However, unlike the process for verifying the possession of a mobile device, the use of these codes may not prevent an imposter from using a stolen phone or stolen phone number. An imposter may be able to successfully complete the identity verification process if the applicant’s possession of the physical device has not been independently verified. In its remote identity proofing guidance, NIST requires federal agencies to use confirmation codes as a supplement to other identity proofing measures. Verification through postal mail confirmation codes. Another method that organizations use to help verify an individual’s identity is to send a confirmation code, such as a personal identification number (PIN), through the mail system to the individual’s address of record. The individual then enters the PIN in the organization’s online application to confirm that they received the code in the mail. Like the use of mobile device confirmation codes, the use of postal mail codes can provide additional assurance about the individual’s identity because the code is sent through a separate medium from the online application where the remote identity proofing process was initiated. Even with these alternatives to knowledge-based verification, however, there are limitations to the security assurances that can be provided. One way to overcome these security limitations is for a trained professional to conduct identity proofing in person. This is generally considered to be a strong approach because it allows for direct physical comparison of an individual’s documentation, including photographic evidence, to the individual attempting to enroll. Verification of the credentials being submitted can be performed by checking electronic records in tandem with physical inspection. Figure 2 provides examples of alternative identity verification and validation methods that federal agencies have reported using. Each of the alternatives to knowledge-based verification has other limitations, including implementation challenges. For example, in-person identity proofing is expensive to implement because it requires organizations to staff and maintain offices or other physical access points in multiple locations, and it can be inconvenient for applicants because it requires travel to one of these locations. Mobile device verification may not always be viable because not all applicants possess a mobile device that can be used to verify their identity. In addition, fraudsters can manipulate or “spoof” phone numbers that redirect phone calls and SMS confirmation codes to an attacker. Sending confirmation codes by postal mail can result in a delay in an individual being able to gain access to the services or benefits he or she is seeking. As previously discussed, in 2017, NIST released an updated version of its technical guidance on remote identity proofing. NIST’s 2017 guidance effectively prohibits the use of knowledge-based verification for sensitive applications because of the security risks associated with this technique. For applications where identity verification is important, the guidance prohibits agencies from providing access to online applications based solely on correct responses to knowledge-based questions. Rather, the guidance provides detailed specifications regarding the required features of the identity evidence (such as driver’s licenses and birth certificates) that an individual is to provide and how agencies are to verify that evidence. Agencies are restricted to using knowledge-based verification only for the very limited role of linking a single piece of identity evidence to an individual and only for applications where the identity verification process is not of critical importance. As a result, agencies are effectively prohibited from using traditional knowledge-based questions—the type of questions typically used in identity verification services provided by CRAs—as part of their processes. Thus, in order for agencies to ensure the effectiveness of their remote identity proofing processes, they are required to find ways to eliminate the use of knowledge-based verification. Three of the six agencies we reviewed—GSA, IRS, and VA—have taken steps to enhance the effectiveness of their remote identity proofing processes. GSA and IRS recently eliminated knowledge-based verification from their Login.gov and Get Transcript services, respectively. VA has implemented alternative methods, but only as a supplement to the continued use of knowledge-based verification. Among the other three agencies, two of them–SSA and USPS–are investigating alternative methods and have stated that they intend to reduce or eliminate their use of knowledge-based verification sometime in the future; however, these agencies do not yet have specific plans for doing so. One other agency, CMS, has no plans to reduce or eliminate knowledge-based verification for remote identity proofing. GSA has implemented alternative methods to knowledge-based verification for Login.gov. While GSA used knowledge-based verification on its Login.gov service in the past, the agency has recently implemented alternative verification techniques that do not rely on knowledge-based verification. Specifically, GSA conducts independent verification of an applicant’s possession of a mobile device, an alternative technique we previously discussed. GSA contracts with a third-party vendor to compare status information about the phone number provided by an individual with telephone company records to confirm the individual’s identity. Further, GSA officials responsible for Login.gov stated that they plan to include additional alternative verification methods to Login.gov in the near future. Specifically, by the end of May 2019, the agency plans to implement software capable of analyzing and validating photos of documentation, such as driver’s licenses, provided by applicants to further enhance the verification of their identities. In 2018, the agency tested this technology through a pilot program. GSA officials responsible for Login.gov stated that they are pursuing several other initiatives to further enhance the verification techniques they use for Login.gov. For example, they are researching new software methods for confirming the authenticity of face images and other biometric information that could be transmitted by applicants to confirm their identity. According to the officials, additional work is needed to ensure that a fraudulent image, such as a photo of a mask, is not being provided in lieu of a live image—a threat known as a “presentation attack.” The GSA officials also said they would like to work with other federal agencies to leverage data that have already been verified, such as USPS-validated mailing addresses, passport and visa information maintained by the Department of State, and IRS tax data. However, the officials cited legal and regulatory restrictions to sharing agency data as a challenge to being able to make use of resources such as these. GSA’s recent elimination of knowledge-based verification from its Login.gov identity proofing process is consistent with NIST’s 2017 guidance on remote identity proofing and reduces the risk of fraud associated with using Login.gov. The additional enhancements and coordination that the agency is working on, if successful, will likely further improve the effectiveness of its remote identity proofing processes. IRS has implemented alternative methods to knowledge-based verification for Get Transcript. While IRS used knowledge-based verification on its Get Transcript service in the past, the agency has recently implemented alternative verification techniques that do not rely on knowledge-based verification. Specifically, IRS conducts independent verification of an applicant’s possession of a mobile device and uses mobile device confirmation codes, alternative techniques we previously discussed. IRS contracts with a CRA to compare status information about the phone number provided by an individual with telephone company records to confirm the individual’s identity. Further, IRS officials responsible for Get Transcript’s identity proofing and authentication services stated that they plan to continue to add alternative verification methods to Get Transcript in the future. They stated that, in June 2017, in response to the release of NIST’s updated digital identity proofing requirements, the agency started a task force to examine the updated requirements and make recommendations on possible changes to IRS’s processes to meet the updated guidance. According to the officials, the task force developed a digital identity risk assessment process that the agency started using to assess external facing online transactions in October 2018. IRS’s recent elimination of knowledge-based verification from its Get Transcript identity proofing process and the additional enhancements that the agency is working on, if successful, will likely further improve the effectiveness of its remote identity proofing processes. VA has taken steps to better ensure the effectiveness of its remote identity proofing processes, but it continues to rely on knowledge-based verification for certain categories of individuals. As previously mentioned, VA relies on two different providers, a commercial identity verification service (called ID.me) and DOD’s DS Logon, to conduct identity proofing for its benefits systems. These providers use a mix of knowledge-based verification and alternative techniques. DOD’s DS Logon verifies applicants using knowledge-based verification, while the commercial provider uses both knowledge-based verification processes as well as stronger alternative techniques. For example, the commercial provider uses cellular phone data to verify an applicant’s identity based on the device subscriber’s relationship to a claimed identity and the subscriber’s tenure with the carrier. VA’s commercial provider can also remotely authenticate identity documents. In this regard, applicants can scan the front and back of driver’s licenses, state identification, and passports, and upload the images to the commercial provider, which then analyzes the images to ensure that the documents meet standards and contain valid information. Further, the provider verifies applicants by having them take photos of themselves and then using facial recognition technology to match the applicants’ images with their identity documents. VA officials in the agency’s information technology and benefits program offices believe that the alternative forms of identity proofing used by its commercial provider as a supplement to knowledge-based verification provide an acceptable level of assurance. Nevertheless, the officials acknowledged that it is important to eventually eliminate knowledge- based verification from the agency’s identity-proofing processes. However, the agency does not have specific plans with time frames and milestones to eliminate the use of knowledge-based verification. VA officials stated that it has not yet established plans for doing so because of its reliance on DOD’s DS Logon service, which still uses knowledge- based verification. Until it develops a specific plan with time frames and milestones to eliminate its reliance on knowledge-based verification, VA and the individuals it serves will continue to face a degree of identity fraud risk that could be reduced. SSA continues to rely on knowledge-based verification for its My Social Security service, but SSA officials stated that the agency intends to eliminate knowledge-based verification in the future. According to the SSA Chief Information Security Officer, in fiscal year 2019, the agency intends to pilot alternative verification methods, such as using the commercial ID.me service. In addition, the official said SSA plans to research other alternatives that could be used to replace knowledge- based verification, including modernizing its legacy systems so that they can use Login.gov or another shared identity management platform. The agency has set a goal of eliminating the use of knowledge-based verification in fiscal year 2020. As an interim measure to reduce the risks associated with knowledge- based verification, SSA officials stated that they limit the period of time and the number of attempts that an individual has to answer the knowledge-based verification questions. These limitations are designed to prevent a potential fraudster from researching the answers to the questions. In addition, SSA also sends a confirmation code via email or SMS, which individuals must enter online before being given access to their account. SSA does not yet have specific plans and milestones to achieve its goal of implementing enhanced remote identity proofing processes by fiscal year 2020. SSA officials stated that they cannot develop specific plans until they are able to identify an alternative method or methods that can be used successfully by all members of the public with which the agency interacts. Until SSA develops specific solutions for eliminating knowledge- based verification, the agency and the individuals that rely on its services will remain at an increased risk of identity fraud. USPS has not yet fully implemented alternative methods to better ensure the effectiveness of its remote identity proofing processes. According to officials responsible for the agency’s identity proofing program, USPS mitigates the risk of using knowledge-based verification by sending a written confirmation to the physical address associated with each identity- proofing transaction and provides instructions for what to do if the transaction is unauthorized or fraudulent. In addition to this mitigation measure, the officials reported that they regularly evaluate new capabilities to further increase confidence in their identity-proofing processes and are planning several additional measures to supplement the use of knowledge-based verification. Specifically, in September 2018, USPS began allowing customers to request a confirmation code via the mail to allow them to enroll in Informed Delivery. In addition, the agency is planning on implementing verification of mobile device possession and SMS enrollment code verification in 2019 and other techniques at a subsequent time. According to USPS officials, these alternative techniques are expected to reduce the agency’s use of knowledge-based verification. The officials said that USPS has not completely eliminated the use of knowledge-based verification because available alternatives to the agency’s current processes do not satisfactorily address critical factors that they consider when deciding whether to implement alternative processes. These factors include cost, projected ability to reduce fraud and protect consumers, projected extent of the population that could be covered, and the burden on customers to complete the process. The officials stated that the agency intends to implement alternative methods in the future for its Informed Delivery service but does not yet have specific plans with time frames and milestones. The officials noted that part of the reason for the slow implementation of alternative methods is that NIST technical guidance does not provide direction on how alternative methods are to be implemented and that additional guidance from NIST would be helpful to the agency for developing and implementing a plan to eliminate knowledge-based verification for the Informed Delivery service. While the supplemental processes implemented by USPS to date may help to reduce the risks associated with using knowledge-based verification, they do not eliminate such risks. Until it completes a plan with time frames and milestones to eliminate its reliance on knowledge-based verification for Informed Delivery, USPS and its customers will remain at increased risk of identity fraud. CMS has not implemented alternative methods to better ensure the effectiveness of the remote identity proofing processes used for its Healthcare.gov service. CMS officials in the Office of Information Technology and the Office of Consumer Information and Insurance Oversight stated that the agency uses a two-step email verification process to reduce the risks associated with knowledge-based verification. Specifically, individuals applying for an account on Healthcare.gov provide basic information (e.g., name, email address, password) and then are asked to acknowledge an email confirmation they receive from CMS. The email confirmation is intended to prove that the individual applying for a Healthcare.gov account is in possession of the email address that same individual provided to CMS. However, this process confirms only the email address that was used to create the account; it does not confirm the identity of the individual who is applying for the account. CMS stated that it uses this process because other mitigating measures are not cost effective. However, NIST’s guidance does not permit agencies to use knowledge-based verification on the basis of cost effectiveness. Further, the agency does not have specific plans with time frames or milestones to eliminate its use of knowledge-based verification for Healthcare.gov. CMS officials acknowledge that they do not have a plan to reduce or eliminate the use of knowledge-based verification because they have not yet identified any effective alternatives to knowledge-based verification for Healthcare.gov. According to these officials, based on a user study they conducted, individuals who use the agency’s services prefer knowledge- based verification over any available alternatives. In addition, the officials stated that certain alternatives, such as mobile device verification, may not always be suitable for the population they serve. As one example, not all applicants have a mobile device that could be used to remotely verify the individual’s identity. The CMS officials noted that NIST technical guidance does not provide direction on how alternative methods are to be implemented, given that they may not always be suitable for the population served by Healthcare.gov. However, until CMS takes steps to develop a plan with time frames and milestones to eliminate the use of knowledge-based verification, CMS and Healthcare.gov applicants will remain at an increased risk of identity fraud. While NIST has issued guidance to agencies related to identity proofing and OMB is drafting identity management guidance, these efforts are not sufficient to ensure that agencies adopt secure methods for remote identity proofing. As previously discussed, NIST’s guidance effectively prohibits the use of knowledge-based verification during the validation and verification phases of the remote identity proofing process, but does not provide direction to agencies on how to successfully implement alternative methods for remote identity proofing for large and diverse segments of the population. Further, OMB has not issued guidance requiring agencies to report on their implementation of remote identity proofing processes, which is essential for monitoring agencies’ progress. Best practices in IT management state that organizations should provide clear direction in order to achieve objectives. Specifically, the Control Objectives for Information and Related Technologies (COBIT), a framework of best practices for IT governance, states that organizations should provide clear direction for IT projects, including relevant and usable guidance, and ensure that those implementing the technology have a clear understanding of what needs to be delivered and how. However, NIST has not issued any supplemental implementation guidance to its 2017 technical guidance to ensure that agencies have a clear understanding of what needs to be done to implement alternative methods of remote identity proofing, as called for in the technical guidance. For example, NIST’s technical guidance provides abstract descriptions of identity evidence that individuals must provide, such as a credential containing a photograph or other biometric identifier as well as anti-counterfeiting security features. The guidance states that such credentials can be provided in person or remotely but does not detail the processes needed for providing credentials remotely. For example, the guidance does not discuss the advantages and limitations of currently available technologies that agencies could successfully use to remotely verify credentials provided by individuals or to make recommendations to agencies on which technologies should be adopted. As previously discussed, several potential limitations could make choosing an alternative method difficult. Technologies such as secure, remote verification of a physical credential may not be commercially available. Also, some alternative technologies require that individuals use cell phones and maintain a verifiable record of having them in their possession. The NIST guidance does not discuss how agencies should accommodate segments of the public who do not possess advanced technological devices, such as cell phones, that may be needed for successful remote verification. Because the guidance does not include specific advice or direction on implementing alternative technologies, agencies may be unable to determine what alternative methods are viable for the populations they serve. As previously discussed, several of the agencies we reviewed send confirmation codes to applicants via cell phone, email, or postal mail, as ways that they believe compensate for risks associated with using knowledge-based verification. However, NIST officials do not consider such methods for remote verification to be effective in compensating for the risks associated with knowledge-based verification. Instead, the NIST technical guidance requires agencies to send confirmation codes by mail when they use any remote identity proofing method, including more advanced, alternative verification methods, such as verification of mobile device possession. Officials from CMS, SSA, and USPS stated that they have not eliminated their use of knowledge-based verification in part because the existing NIST technical guidance does not provide direction on how alternative methods are to be implemented, given the various limitations of those alternative methods that agencies have identified. The officials stated that federal agencies could benefit from additional guidance on implementing the alternative verification techniques called for in the NIST technical guidance. In response to these agencies’ comments about being unable to fully implement the remote identity proofing guidance, NIST officials stated that agencies were expected to use their own judgment to determine how to meet the remote identity proofing requirements. The officials added that it was NIST’s position that the updated guidance was comprehensive enough for agencies to follow. Thus, at the time of our review, NIST did not have plans to assist agencies by issuing implementation guidance to supplement its existing technical guidance. NIST officials stated that they are available to provide assistance on an individual basis to agencies that seek their advice. Without additional guidance from NIST on how agencies are to implement the alternative identity proofing methods specified in an agency’s existing technical guidance, agencies may not be using the most effective and secure identity-proofing methods, thus exposing their systems to risk of fraud. FISMA requires the Director of OMB to oversee agency information security policies and practices. However, OMB has not provided agencies with guidance establishing reporting requirements for OMB to use in monitoring agencies’ progress in implementing secure remote identity proofing processes. For example, OMB has not proposed including reporting requirements for remote identity proofing in its draft policy on identity, credential, and access management, nor has it included reporting requirements in its FISMA reporting guidance to agencies for fiscal year 2019. According to OMB staff, OMB plans to issue guidance to agencies on the implementation of identity, credential, and access management. OMB issued a draft of this guidance for public comment in April 2018. However, the draft guidance does not include a requirement for agencies to report on progress in implementing secure remote identity proofing processes. Because it does not require agency reporting on progress in implementing secure remote identity proofing processes, OMB does not have visibility into the extent that agencies rely on insecure methods, particularly knowledge-based verification. Without establishing effective oversight measures, OMB cannot adequately monitor agency progress in implementing the secure identity proofing methods called for in NIST’s 2017 technical guidance. As a result, agencies may be at risk of implementing weak methods of remote identity-proofing for individuals who seek access to services and benefits from the federal government, which may put both the federal government and individuals at risk for fraud. The six agencies that we reviewed rely on a variety of remote identity proofing techniques to help ensure that the individuals who enroll for federal benefits and services are who they claim to be. Several of the selected agencies use knowledge-based verification processes that rely on CRAs to pose questions to individuals and check their answers as a way of verifying their identities before granting them enrollment in a federal benefit or service. However, given recent breaches of sensitive personal information, these agencies face risks because fraudsters may be able to obtain and use an individual’s personal information to answer knowledge-based verification questions and successfully impersonate that individual to fraudulently obtain federal benefits and services. Two agencies we reviewed, GSA and IRS, recently implemented remote identity proofing processes for Login.gov and Get Transcript that allow individuals to enroll online without relying on knowledge-based verification. However, four agencies (CMS, SSA, USPS, and VA) were still using knowledge-based verification to conduct remote identity proofing. Moreover, none of the four agencies have developed specific plans to eliminate knowledge-based methods from their processes. Without such plans, these federal agencies and the individuals that rely on such processes will remain at risk for identity fraud. NIST has issued technical guidance regarding remote identity proofing, but it may not be sufficient to help ensure that federal agencies adopt more secure methods. NIST’s guidance does not provide direction on how agencies can adopt more secure alternatives to knowledge-based verification while also addressing issues of technical feasibility and usability for all members of the public. In addition, OMB has not issued guidance setting agency reporting requirements that OMB could use to track implementation of more secure processes across the federal government. Without additional guidance, federal agencies are likely to continue to rely on risky knowledge-based verification that could be used to fraudulently gain access to federal benefit programs and services. We are making a total of 6 recommendations to CMS, NIST, OMB, SSA, USPS, and VA. Specifically: The Administrator of the Centers for Medicare and Medicaid Services should develop a plan with time frames and milestones to discontinue knowledge-based verification, such as by using Login.gov or other alternative verification techniques. (Recommendation 1) The Director of the National Institute of Standards and Technology should supplement the agency’s 2017 technical guidance with additional guidance to assist federal agencies in determining and implementing alternatives to knowledge-based verification that are most suitable for their applications. (Recommendation 2) The Director of the Office of Management and Budget should issue guidance requiring federal agencies to report on their progress in adopting secure identity proofing processes. (Recommendation 3) The Commissioner of Social Security should develop a plan with specific milestones to discontinue knowledge-based verification, such as by using Login.gov or other alternative verification techniques. (Recommendation 4) The Postmaster General of the United States should complete a plan with time frames and milestones to discontinue knowledge-based verification, such as by using Login.gov or other alternative verification techniques. (Recommendation 5) The Secretary of the Department of Veterans Affairs should develop a plan with time frames and milestones to discontinue knowledge-based verification, such as by using Login.gov or other alternative verification techniques. (Recommendation 6) We requested comments on a draft of this report from the eight agencies included in our review. In response, we received written comments from six agencies—Commerce (on behalf of NIST), HHS (on behalf of CMS), IRS, SSA, USPS, and VA. Their comments are reprinted in appendices II through VII, respectively. Of the six agencies to which we made recommendations, four of them (Commerce, SSA, USPS, and VA) agreed with our recommendations, and one agency (HHS) did not concur with our recommendation. One agency (OMB) did not state whether it agreed or disagreed with our recommendation. In addition, multiple agencies (GSA, IRS, OMB, USPS, and VA) provided technical comments on the draft report, which we have incorporated, as appropriate. The following four agencies agreed with the recommendations that we directed to them: Commerce agreed with our recommendation. The department stated that it will develop additional guidance to assist federal agencies with alternatives to knowledge-based verification and expects to do so within one year from issuance of this report. Comments from Commerce are reprinted in appendix II. SSA agreed with our recommendation. The agency stated that it will continue to seek improvements in its existing remote identity proofing process. SSA also stated that, in addition to a roadmap it developed in fiscal year 2019 to update its knowledge-based verification process to a more secure multi-factor authentication technology, it will take steps to ensure compliance with NIST standards for remote identity proofing. SSA’s comments are reprinted in appendix V. USPS agreed with our recommendation. The agency stated that it will be developing a roadmap to implement additional identity-proofing tools and techniques through 2020. Comments from USPS are reprinted in appendix VI. VA agreed with our recommendation. The department stated that it will develop a specific plan with time frames and milestones to eliminate knowledge-based verification from the aspects of the remote identity proofing process that it controls. Further, in its response, VA requested that GAO direct a recommendation to the Department of Defense (DOD) to discontinue DS Logon and consider using Login.gov instead. However, we are not issuing any recommendations to DOD because our scope of work did not include auditing DOD’s remote identity proofing processes. Nevertheless, we have adjusted our recommendations to CMS, SSA, USPS, and VA to clarify that Login.gov is one option for identity proofing that they should consider when developing their plans to discontinue the use of knowledge-based verification. VA’s comments are reprinted in appendix VII. One agency did not concur with our recommendation. Specifically, HHS raised several issues related to our findings. The agency stated that it uses a risk-based approach to designing systems controls and that a unilateral prohibition on the use of knowledge-based verification without alternatives is not a feasible solution. We agree with this comment and strongly support a risk-based approach to designing security controls, as required by FISMA. However, we believe that alternatives to knowledge- based verification exist that should be assessed and incorporated as appropriate. Similarly, HHS noted that for other applications across the department, it has considered factors such as consumer user experience, cost, and operational feasibility in addition to NIST guidelines. We agree that many factors need to be considered in assessing what method or methods of identity proofing are most appropriate for any given application but believe it is important for agencies to develop plans for addressing those factors that also eliminate the use of risky techniques, such as knowledge-based verification, that could have a negative impact on consumers and agencies. In response to our specific recommendation to CMS, HHS stated that it does not believe that suitable alternative methods exist that would work for CMS’ population of users, such as those in the rural community, due to distance or individuals without cell phones. However, we continue to believe that CMS should develop a plan to discontinue the use of knowledge-based verification. We recognize that there are members of the population that may not be reached with certain identity proofing techniques; however, a variety of alternative methods to knowledge- based verification are available that CMS can consider to address the population it serves. Comments from HHS are reprinted in appendix III. In addition, OMB did not state whether it agreed or disagreed with our recommendation. Further, in an email response, OMB staff from the office of the Federal Chief Information Officer provided a technical comment, which we incorporated. However, OMB did not otherwise comment on the report findings or our recommendation made to the agency. The IRS also provided written comments on the draft report. In its comments, the agency described the status of its efforts to strengthen identity verification processes, including the fact that it has eliminated the use of knowledge-based verification. Comments from IRS are reprinted in appendix IV. Finally, GSA provided only technical comments on the draft report, as previously mentioned. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, to the Administrators of the Centers for Medicare and Medicaid Services and General Services Administration; the Commissioners of Internal Revenue and Social Security; Director of the Office of Management and Budget; the Postmaster General of the United States; and the Secretaries of the Departments of Commerce and Veterans Affairs. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Nick Marinos at (202) 512-9342 or marinosn@gao.gov, or Michael Clements at (202) 512-8678 or clementsm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VIII. Our objectives were to (1) describe selected federal agency practices for remote identity proofing of individuals seeking access to major web-based applications using services provided by consumer reporting agencies and the risks associated with those practices, (2) assess selected federal agencies’ actions to ensure the effectiveness of agencies’ remote identity proofing processes, and (3) assess the sufficiency of federal identity proofing guidance developed by OMB and NIST in assuring the security of federal systems. To address the first objective, we made an initial, non-probability selection of federal agencies that (1) maintained major public-facing web applications to provide access to federal benefits or services and (2) relied on identity proofing solutions provided by the three nationwide consumer reporting agencies (CRAs)—Equifax, Experian, and TransUnion—to verify the identities of individuals applying for such benefits or services. We considered a “major” application to be one that could involve interaction with millions of individuals across the entire country. To select six agencies from this group, we reviewed prior GAO reports to identify potential agencies for review. We then interviewed officials at these agencies and at CRAs to confirm that these agencies use CRAs as part of their identity proofing processes and to obtain information about additional federal agencies that also employ CRAs for identity proofing for major applications. We included GSA in these interviews because its mission is to support federal agencies and it was likely to be aware of additional federal agencies that fit our criteria. From the information we gained from our interviews and research, we selected these six agencies: the Centers for Medicare and Medicaid Services (CMS), General Services Administration (GSA), Internal Revenue Service (IRS), Social Security Administration (SSA), United States Postal Service (USPS), and the Department of Veterans Affairs (VA). At each of these agencies, we reviewed documentation that described the current remote identity proofing processes the agencies are using for their major public-facing web applications. In addition, we interviewed agency officials responsible for identity proofing to obtain details of the techniques used to verify remote users of these applications. To the extent that these entities used CRAs to conduct knowledge-based verification as part of their remote identity-proofing processes, we discussed the risks associated with using knowledge-based methods as well as the potential advantages and limitations of using alternative methods that are not knowledge-based. We also obtained information from officials at NIST about the risks of knowledge-based methods and the availability of alternative methods. To address the second objective, we assessed remote identity proofing processes used by the selected agencies to determine the extent that they rely on knowledge-based verification to enroll online applicants for federal benefits and services. We also identified alternative methods used by these agencies, either in place of or in addition to knowledge-based verification, and assessed the extent to which agencies had implemented these methods to mitigate the risk of using knowledge-based methods. We compared the remote identity proofing processes at these agencies with the requirements as specified in NIST Special Publication 800-63, Digital Identity Guidelines, to determine whether the processes met the requirements of the NIST guidance. We also interviewed officials responsible for these identity proofing programs to obtain information about agencies’ plans, if any, to eliminate the use of knowledge-based verification from their remote identity proofing processes in the future and obtained relevant documentation of such plans. To address the third objective, we reviewed NIST Special Publication 800-63, Digital Identity Guidelines, to identify federal requirements for remote identity proofing. We compared the guidance to the Control Objectives for Information and Related Technologies (COBIT), a framework of best practices for IT governance, to determine whether the NIST guidance contained clear direction, including relevant and usable guidance, to ensure that those implementing the technology have a clear understanding of what needs to be delivered and how. To assess the sufficiency of this guidance, we consulted with subject matter experts at NIST, ID.me, a private-sector provider of remote verification technologies, and relevant officials at the selected federal entities. Based on information we had obtained about available alternative methods, we determined the extent to which gaps existed in the NIST guidance with regard to implementation of alternative technologies. We also obtained the views of federal agency officials on the extent to which NIST guidance provided sufficient direction to assist them in implementing appropriate remote identity proofing methods. Further, we reviewed OMB’s draft Identity, Credential, and Access Management policy and compared it to the requirements in FISMA and identified shortfalls. We also interviewed OMB staff to discuss the sufficiency of the office’s current guidance and to determine whether the office planned to issue additional guidance establishing reporting requirements for federal entities or conduct other forms of oversight of federal entities’ implementation of the NIST identity proofing guidance. We conducted this performance audit from November 2017 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the individuals named above, John de Ferrari and John Forrester (assistant directors); Tina Torabi (analyst-in-charge); Bethany Benitez, Christina Bixby, Chris Businsky, Kavita Daitnarayan, Nancy Glover, Andrea Harvey, Thomas Johnson, David Plocher, Rachel Siegel, and Winnie Tsen made key contributions to this report.", "summary": "Many federal agencies rely on CRAs, such as Equifax, to help conduct remote identity proofing. The 2017 breach of data at Equifax raised concerns about federal agencies' remote identity proofing processes. GAO was asked to review federal agencies' remote identity proofing practices in light of the recent Equifax breach and the potential for fraud. The objectives of this review were to (1) describe federal practices for remote identity proofing and the risks associated with those practices, (2) assess federal agencies' actions to ensure the effectiveness of agencies' remote identity proofing processes, and (3) assess the sufficiency of federal identity proofing guidance. To do so, GAO identified remote identity proofing practices used by six agencies (CMS, GSA, IRS, SSA, USPS, and VA) with major, public-facing web applications providing public access to benefits or services. GAO compared the agencies' practices to NIST's remote identity proofing guidance to assess their effectiveness, and compared NIST's and OMB's guidance to requirements in federal law and best practices in IT management to assess the sufficiency of the guidance. Remote identity proofing is the process federal agencies and other entities use to verify that the individuals who apply online for benefits and services are who they claim to be. To perform remote identity proofing, agencies that GAO reviewed rely on consumer reporting agencies (CRAs) to conduct a procedure known as knowledge-based verification. This type of verification involves asking applicants seeking federal benefits or services personal questions derived from information found in their credit files, with the assumption that only the true owner of the identity would know the answers. If the applicant responds correctly, their identity is considered to be verified. For example, the Social Security Administration (SSA) uses this technique to verify the identities of individuals seeking access to the “My Social Security” service, which allows them to check the status of benefit applications, request a replacement Social Security or Medicare card, and request other services. However, data stolen in recent breaches, such as the 2017 Equifax breach, could be used fraudulently to respond to knowledge-based verification questions. The risk that an attacker could obtain and use an individual's personal information to answer knowledge-based verification questions and impersonate that individual led the National Institute of Standards and Technology (NIST) to issue guidance in 2017 that effectively prohibits agencies from using knowledge-based verification for sensitive applications. Alternative methods are available that provide stronger security, as shown in Figure 1. However, these methods may have limitations in cost, convenience, and technological maturity, and they may not be viable for all segments of the public. Two of the six agencies that GAO reviewed have eliminated knowledge-based verification. Specifically, the General Services Administration (GSA) and the Internal Revenue Service (IRS) recently developed and began using alternative methods for remote identity proofing for their Login.gov and Get Transcript services that do not rely on knowledge-based verification. One agency—the Department of Veterans Affairs (VA)—has implemented alternative methods for part of its identity proofing process but still relies on knowledge-based verification for some individuals. SSA and the United States Postal Service (USPS) intend to reduce or eliminate their use of knowledge-based verification sometime in the future but do not yet have specific plans for doing so. The Centers for Medicare and Medicaid Services (CMS) has no plans to reduce or eliminate knowledge-based verification for remote identity proofing. Several officials cited reasons for not adopting alternative methods, including high costs and implementation challenges for certain segments of the public. For example, mobile device verification may not always be viable because not all applicants possess mobile devices that can be used to verify their identities. Nevertheless, until these agencies take steps to eliminate their use of knowledge-based verification, the individuals they serve will remain at increased risk of identity fraud. NIST has issued guidance to agencies related to identity proofing and OMB has drafted identity management guidance, but their guidance is not sufficient to ensure agencies are adopting such methods. Sound practices in information technology (IT) management state that organizations should provide clear direction on how to implement IT objectives. However, NIST's guidance does not provide direction to agencies on how to successfully implement alternative identity-proofing methods with currently available technologies for all segments of the public. For example, the guidance does not discuss the advantages and limitations of currently available technologies or make recommendations to agencies on which technologies should be adopted. Further, most of the agencies that GAO reviewed reported that they were not able to implement the guidance because of limitations in available technologies for implementing alternative identify proofing methods. NIST officials stated that they believe their guidance is comprehensive, and at the time of our review they did not plan to issue supplemental implementation guidance to assist agencies. The Federal Information Security Modernization Act of 2014 ( FISMA) requires that OMB oversee federal agencies' information security practices. Although OMB has the authority under this statute to issue guidance, OMB has not issued guidance requiring agencies to report on their progress in implementing NIST's identity proofing guidance. OMB staff plan to issue guidance on identity management at federal agencies, but their proposed guidance does not require agencies to report on their progress in implementing NIST guidance. Until NIST provides additional guidance to help agencies move away from knowledge-based verification methods and OMB requires agencies to report on their progress, federal agencies will likely continue to struggle to strengthen their identify proofing processes. GAO is making recommendations to six agencies to strengthen online identify verification processes: GAO recommends that CMS, SSA, USPS, and VA develop plans to strengthen their remote identity proofing processes by discontinuing knowledge-based verification. GAO recommends that NIST supplement its technical guidance with implementation guidance to assist agencies in adopting more secure remote identity proofing processes. GAO recommends that OMB issue guidance requiring federal agencies to report on their progress in adopting secure identity proofing practices. Four agencies—Commerce (on behalf of NIST), SSA, USPS, and VA—agreed with GAO's recommendations. These agencies outlined the additional steps they plan to take to improve the security of their remote identity proofing processes. One agency, HHS (on behalf of CMS), disagreed with GAO's recommendation because it did not believe that the available alternatives to knowledge-based verification were feasible for the individuals it serves. However, a variety of alternative methods exist, and GAO continues to believe CMS should develop a plan for discontinuing the use of knowledge-based verification. OMB provided a technical comment, which GAO incorporated, but OMB did not provide any comments on GAO's recommendation.", "document_type": "gao"}
{"report": "Social Security has been the foundation of retirement security in the United States. Enacted in 1935, Social Security provides for the general welfare of older Americans by, among other things, establishing a system of federal old-age benefits, including a retirement program. Officially titled Old-Age and Survivors Insurance (OASI), the Social Security retirement program provides benefits to retired workers, their families, and survivors of deceased workers. About 51 million retirees and their families received $798.7 billion in Social Security retirement benefits in 2017, according to Social Security Administration (SSA), which is responsible for administering the program. About 40 years after the creation of Social Security, landmark legislation was enacted in 1974 that has played a major role in establishing the structure for private sector employers’ involvement in sponsoring retirement plans for their workers: the Employee Retirement Income Security Act of 1974 (ERISA). ERISA is a complex law administered by multiple federal agencies including the Department of Labor (DOL), the Internal Revenue Service (IRS) within the Department of the Treasury (Treasury), along with the Pension Benefit Guaranty Corporation (PBGC), and has evolved with many significant amendments over the years (see app. II). ERISA was enacted, in part, to address public concerns about the security of pension benefits, including the prominent failure of a couple of large, private sector pension plans. The act, as amended, does not require any employer to establish a retirement plan, but those who do must meet certain requirements and minimum standards. For example, ERISA establishes certain requirements for all employer-sponsored plans, including responsibilities for plan fiduciaries (those who manage and control plan assets, among others), as well as minimum funding standards for defined benefit (DB) plans, which traditionally promise to provide a monthly payment to retirees for life. ERISA also established the PBGC, the government corporation responsible for insuring the pension benefits of nearly 37 million American workers and retirees who participate in nearly 24,800 private sector defined benefit plans. Under ERISA, tax-qualified DB plans (or the employers who sponsor them) may have to pay insurance premiums to the PBGC, based on the funding level of their plans. The IRS also administers the Internal Revenue Code (IRC), which has provisions that affect pensions and retirement savings. While SSA administers the Social Security program, and the DOL, PBGC, and IRS each are generally responsible for administering aspects of ERISA, several other agencies also have important roles in various parts of the retirement system. For example, the Department of Health and Human Services oversees the Centers for Medicare and Medicaid Services (CMS), which administers the major health care programs that provide coverage for retirees, as well as the Administration on Aging, which encourages and assists state grantees that provide services for older adults. In addition, agencies such as the U.S. Department of Agriculture and the Department of Housing and Urban Development oversee food and housing programs for older adults. Other agencies also play a role in providing various services and supports for older adults. For example, the Department of Transportation administers a program that improves access and alternatives to public transportation for seniors and individuals with disabilities. The Consumer Financial Protection Bureau, as part of its mandate to provide financial literacy education, helps consumers navigate financial choices related to retirement. The Federal Trade Commission can have consumer protection and investor oversight roles and responsibilities related to individuals borrowing against their pensions. In addition, these federal agencies and others work together to help combat elder financial exploitation, which experts have described as an epidemic with society-wide repercussions. Citing our prior work on this topic, in October 2017, Congress enacted the Elder Abuse Prevention and Prosecution Act, calling on the Department of Justice to work with other federal, state, and local law enforcement agencies to improve data collection and provide technical assistance focused on combatting elder abuse. The need for government services and support for older adults in retirement will continue to grow as the proportion of older adults in the United States continues to rise significantly in the future. In 1970, those age 65 and over accounted for about 10 percent of the population, but by 2060, they are expected to account for about 23 percent (see fig. 1). This reflects long-term decreases in birth rates and increases in life expectancy. The U.S. retirement system is supported by three main pillars—Social Security, employer-sponsored plans, and individuals’ savings—that serve as important sources of retirement income for Americans. Currently, each of these pillars faces various risks and other challenges. If left unchanged, these risks present the federal government with significant potential fiscal exposures, which may legally commit or create expectations for future federal spending. The first pillar, Social Security (specifically, Social Security’s retirement program), is facing financial difficulties, as are other federal programs that provide essential supports to many older Americans, such as Medicare and the PBGC’s insurance programs (see fig. 2). In addition, multiple federal agencies help fund a broad array of home and community-based services for older adults. As the number of older adults needing assistance continues to grow, the pressure to increase federal funding for these services is likely to increase. As the foundation of retirement security in the United States, Social Security’s retirement program, financed primarily by payroll taxes, helps reduce poverty among beneficiaries, many of whom rely on Social Security for the majority of their income once they retire. Our analysis of data from the Federal Reserve Board’s most recent Survey of Consumer Finances (SCF) showed that in 2016, among households age 65 and over, the bottom 20 percent, ranked by income, relied on Social Security retirement benefits for 81 percent of their income, on average. But Social Security is facing financial difficulties that, if not addressed, will affect its long-term stability. During the many years that the revenue for Social Security’s retirement program exceeded costs, the program built up reserves in the trust fund. However, since 2010, Social Security has been paying out more in benefits than it received and has relied on interest income to help cover expenses. For 2018, the cost of the program was expected to exceed total income by $2 billion and, as a result, asset reserves were expected to decline. If no changes are made, current projections indicate that by 2034, the retirement program trust fund will only be sufficient to pay 77 percent of scheduled benefits. The underlying cause of Social Security’s financial difficulties is the aging population, driven by lower fertility rates and increased life expectancy, and accelerated by the ongoing retirement of the baby boom generation. The first baby boomers began receiving Social Security retirement benefits in 2008, and growing numbers will become eligible for Social Security benefits in coming years. Our analysis indicates that the number of baby boomers turning 65 is projected to increase from an average of about 10,200 per day in 2018 to more than 11,000 per day in 2029 (see fig. 3). As with the Social Security retirement program, reserves had also built up over time in the trust fund for Social Security’s disability program, but in 2005, the program began paying out more than it was taking in. To avoid benefit reductions, which were expected to begin in 2016, Congress passed a law in late 2015 that temporarily reallocated some payroll tax revenue from the retirement trust fund to the disability trust fund. Even with this added boost, if no further changes are made, reductions in disability benefits are projected to be needed beginning in 2032, according to SSA’s most recent report. For both the Social Security retirement and disability programs combined, the number of workers contributing to Social Security for each aged, disabled, dependent, or surviving beneficiary is declining, due to the aging population and other factors. While there are currently 2.8 workers contributing to Social Security per beneficiary, this ratio is expected to decline to 2.2 by 2035, and to 2.0 by 2095 (see fig. 4). It is difficult to predict exactly what would occur if either Social Security’s retirement or disability programs were to become insolvent because the Social Security Act does not provide for any procedure for paying less than full benefits. According to SSA, benefits could be reduced across the board by a set percentage, certain benefits could be prioritized, or benefits could be delayed. The major health care programs that include coverage for retirees, Medicare and Medicaid, also face increasing financial challenges due to program and demographic changes. For example, over the years, Congress has made changes to Medicare so that more people have become eligible, even if under age 65. Also, Congress has added two more parts to Medicare: one part allowing insurance under private plans approved by Medicare (Medicare Advantage), and another part providing prescription drug coverage. As of 2017, over 58 million people were enrolled in one or more parts under Medicare. Projections indicate that in the coming decade, as more members of the baby-boom generation become eligible for benefits, the number of Medicare beneficiaries will rise to 75 million in 2027. Similar to the challenges facing Social Security, spending for Medicare Part A (Hospital Insurance) is projected to outpace revenue over time, and the trust fund for Medicare Part A is projected to be unable to pay full benefits beginning in 2026. At that time, the Hospital Insurance trust fund will only be sufficient to pay 91 percent of hospital-related Medicare spending. Medicaid, which provides health care coverage and financing for millions of low-income individuals, including those age 65 or older, also faces financial challenges. Medicaid is the nation’s primary payer for long-term services and supports, and the elderly—along with those with disabilities—are among the highest cost Medicaid beneficiaries. The federal government and states share in the financing of the Medicaid program, with the federal government matching most state expenditures for Medicaid services using a statutory formula. Estimated Medicaid outlays for fiscal year 2017 were $592.2 billion, of which $370.6 billion was financed by the federal government and $221.6 billion by the states. Over the next 7 years, Medicaid expenditures are expected to increase significantly, reaching just over $1 trillion in 2026. The PBGC insures the pension benefits of most private sector DB plans through one of its two programs: the Single-Employer Insurance Program and the Multiemployer Insurance Program. The single-employer program is the larger of the two programs. As of the end of fiscal year 2018, the single-employer program insured about 26 million workers and retirees participating in about 23,400 private sector single-employer DB plans. As of the end of fiscal year 2018, the multiemployer program insured about 11 million workers and retirees in about 1,400 private sector DB plans created through a collective bargaining agreement between two or more employers and a union. Although PBGC is one of the largest of any federal government corporations, with over $110 billion in assets, its pension benefit guarantees are increasingly at risk due to its substantial liabilities. At the end of fiscal year 2018, PBGC’s net accumulated financial deficit was over $51 billion, and its exposure to potential future losses for underfunded retirement plans was estimated to be nearly $185 billion. We designated the single-employer program as high risk in July 2003 and added the multi-employer program to our high-risk list in January 2009. Concerns about PBGC’s financial future have kept both programs on GAO’s high-risk list. As long as PBGC’s long-term financial stability remains uncertain, the retirement benefits of millions of U.S. workers and retirees are at risk of greater reductions should their benefit plans be terminated below PBGC’s current guaranteed benefit levels. In contrast to Social Security, PBGC is not funded by tax revenues, but by the premiums paid by plans or their sponsors, the assets acquired from terminated plans, and investment returns on these funds. The primary drivers of the government’s fiscal exposure related to PBGC’s deficit are the collective financial risk of the many underfunded pension plans insured by PBGC and the long-term decline in the number of participants covered by traditional DB plans. Since 1985, there has been a 78 percent decline in the number of plans insured by PBGC and more than 13 million fewer workers actively participating in PBGC-insured plans. There has also been a recent trend of single-employer plan sponsors transferring the liability for some of their participants to insurance companies via group annuity “buy-outs,” further reducing the number of participants in PBGC-covered plans. As a result of these trends, even though PBGC premium rates have increased significantly in recent years, PBGC’s premium base has been eroding over time as fewer sponsors are paying premiums for fewer participants. In addition, more recently, PBGC’s net accumulated financial deficit has escalated dramatically due to the critical and declining status of a number of large multiemployer pension plans. As we previously reported, PBGC’s multiemployer plan is projected to become insolvent in approximately 6 years, and if that happens, participants in the insolvent multiemployer plans who rely on PBGC guarantees will receive only a small fraction of current statutory guarantees. According to PBGC, most participants would receive less than $2,000 a year, and in many cases less. Our prior work has found that federally-funded services for older Americans were not reaching many older adults who may need them, and that the funding for these programs had decreased while the number of older adults had increased. The federal government helps provide state and local governments with funding for a broad array of home and community-based services for older adults through multiple federal agencies and programs. In addition to long-term care services funded by Medicaid, these programs also include services funded under the Older Americans Act of 1965, as amended, which provides grants to states for such services as home-delivered and congregate meals, home- based care, transportation, and housing. In our 2015 report, we recommended that the Department of Health and Human Services (HHS) should facilitate development of a cross-agency federal strategy to help ensure that federal resources are used effectively and efficiently to support a comprehensive system of home and community-based services and related supports for older adults. While HHS agreed with our recommendation, the agency has yet to develop a cross-agency strategy involving all five agencies that fund these services. As the number of older adults needing assistance continues to grow, the gap in services can only be expected to widen. Absent any changes, state and local governments are facing—and will continue to face—a gap between receipts and expenditures in the coming years, putting greater pressure on the federal government to increase funding. The second pillar of the U.S. retirement system, employer-sponsored retirement plans, is also an important source of income relied upon by many Americans in their retirement. However, not everyone has access to employer-sponsored plans, and among those who do, certain provisions and requirements of the plans can make it difficult for individuals to accumulate savings over time. Bureau of Labor Statistics data indicate that about one-third of private sector workers in the United States did not have access to an employer- sponsored retirement plan in 2016, and about two-thirds did. Of those with access, the vast majority (about 76 percent) participated in the plan, either because they were automatically enrolled by the plan sponsor or they chose to participate. Although individuals without access to an employer-sponsored plan can save for retirement on their own, having access to an employer- sponsored retirement plan makes it easier to save, and more likely that an individual will have another source of income in retirement beyond Social Security. Our prior work found that employees working for smaller firms and in certain industries, such as leisure and hospitality, are significantly less likely to have access to an employer-sponsored plan compared with those working in larger firms and in certain other industries, such as information services. Also, we found that low-income workers are much less likely than high-income workers to have access to an employer-sponsored plan. Among those individuals who have access to employer-sponsored plans in the private sector, the structure of plans has changed over time, with a shift from traditional DB pension plans to defined contribution (DC) plans, such as 401(k)s, as the primary type of retirement plan (see fig. 5). DB plans are traditional retirement plans that generally promise to provide a benefit for the life of the participant, based on a formula specified in the plan that typically takes into account factors such as an employee’s salary, years of service, and age at retirement. DC plans are employer- sponsored account-based retirement plans, such as a 401(k) plan, that allow individuals to accumulate tax-advantaged retirement savings in an individual account based on employee and/or employer contributions, and the investment returns (gains and losses) earned on the account. The amount of assets held in individual retirement accounts (IRA) also has increased significantly. Most of the assets in IRAs are funded by assets rolled over from DC plans, and sometimes DB plans, when individuals change jobs or retire. With DB plans, participants can accumulate retirement savings simply by continuing to work for the employer offering the plan, and the employer is responsible for ensuring that the amount in the plan is sufficient to pay promised benefits at retirement. However, even when DB plans were more prevalent, many workers did not have access, and those with access to DB plans could still face challenges under certain circumstances. For example, when DB plan participants change employers, their accrued benefits are less portable than accrued savings in a DC plan. If the change in employers takes place before they have met vesting requirements, DB plan participants can lose all the benefits accumulated from employer contributions to that point, which in the private sector, generally means everything. Also, for DB plans that base benefits on final average salary, benefit accruals are significantly “backloaded.” As a result, if a DB plan participant changes employers mid-career, it could result in missing out on the time when the biggest benefit accruals would have occurred. In addition, when entering retirement, although those with DB plans can generally rely on receiving a set monthly benefit for life, they may still face challenges. For example, participants in certain financially troubled plans—such as those in the multiemployer plans discussed earlier—could see their benefits being suspended or cut. In addition, if a DB plan participant is offered and accepts a lump-sum payment in place of a lifetime annuity, the participant may face challenges similar to those with DC accounts in terms of managing the spend down of their retirement savings. With DC plans, responsibility for planning and managing retirement savings is shifted from employers to employees. Participants in DC plans are often required to make complex financial decisions—decisions that generally require financial literacy and that could have significant consequences for their financial security throughout retirement. For example, workers with DC plans have to decide whether to participate in the plan, how much to contribute to their accounts and how to manage their investments to strike the right balance between risk and returns. One way DC plan enrollment and contribution levels can be encouraged is by putting automatic mechanisms in place. For example, DC plan sponsors can encourage participation in the plan by adopting auto- enrollment, whereby eligible workers are enrolled into a plan automatically, unless they choose to opt out. DC plan sponsors can also encourage increases in contribution rates by adopting auto-escalation, whereby the employee’s contributions are automatically increased to a predetermined level on a set schedule, unless they choose to opt out. Participants in DC plans also have to decide whether to borrow from their accounts if other needs arise, or cash out their accounts when they change jobs. When leaving an employer, those with DC accounts may be allowed to transfer their accumulated balances into a new employer plan or an individual retirement account (IRA), but they may also be tempted to cash out their accounts, even though they may face associated tax consequences. Similarly, when entering retirement, those with DC accounts may decide to transfer the account balance into an IRA, or they may decide to receive the funds in a lump-sum payment. While some DC plans also offer monthly payments through an annuity, most do not provide lifetime income options or other options that can help participants draw down their retirement funds in a systematic way. Findings from the most recent SCF indicate that an individual’s ability to accumulate retirement savings depends on the individual’s income level. In addition, the disparities in average account balances by income level have increased markedly over time (see fig. 6). For example, according to SCF data, households in the top 10 percent of income level appeared to be substantially better prepared for retirement than most others, with an average account balance of more than $720,000 in 2016. In contrast, households with below average income, in the second quintile, had an average account balance of about $47,000. Among lower-income households, our prior work suggests that cashing out accounts when changing jobs may be a significant drain on retirement savings, along with unexpected events that may also cause them to withdraw funds from their accounts prior to retirement. Retirement experts have posited a variety of reasons for employers’ shift to DC plans. One oft-cited reason is that the structure of DC plans gives employers better control over how much they spend on wages and benefits packages. With DC plans, employers may choose whether to make contributions to participants’ individual accounts; in contrast, DB plans promise a certain future monthly benefit to employees in retirement, and the employer must bear the risk of making adequate contributions to the plan to make good on that promise. Another reason retirement experts cite for the shift to DC plans was the introduction of 401(k) accounts in the Internal Revenue Code in 1978, which they credit with fostering the adoption of account-based plans by sanctioning the use of salary deferrals as a source of contributions. Some retirement experts have also suggested that employees’ preferences and demands have changed over time, making DC plans more feasible and, in some respects, more appealing. For example, some analysts have noted that the portability of an account-based plan can be better suited to meet the needs of a more mobile workforce. The third pillar of the retirement savings system—individuals’ personal savings—is the remaining important source of retirement income, and it also faces certain risks and challenges. Personal savings can include a variety of assets, such as amounts saved from income or wages; contributions to accounts outside of a retirement plan; non-retirement financial wealth that is inherited or accumulated over time; and equity from tangible assets such as a home. These savings are expected to augment any income from the first two pillars: Social Security and employer-sponsored retirement plans. Over the past several decades, however, the personal saving rate—which is calculated as the proportion of disposable income that households save—has trended steeply downward, from a high of 14.2 percent in 1975, to a low of 3.1 percent in 2005, before recovering somewhat to 6.8 percent in 2018 (see fig. 7). While the specific implications of a historically low national saving rate on any current or future retiree are less clear, the decline in the U.S. personal savings rate over time is concerning and could have implications for retirement security, particularly when coupled with the recent trend of low wage growth. After accounting for inflation, average wages remain near the levels they were in the 1970s for most individuals (see fig. 8), adding to the difficulty of increasing their level of saving. In addition, many households have accumulated little wealth. SCF data show that among households in which the head of the household was working, the average value of all financial assets, excluding savings in retirement accounts, was $70,700 in 2016. For households in which the head was retired, this average was $89,700. For those who become home owners and build up equity in a home, this equity can serve as an important asset, providing a potential income source in retirement either by selling the home or obtaining a reverse mortgage. However, increased household debt levels may affect the amount of income available from this source, as well as from other assets. Data on the make-up of debt indicate that home ownership has been declining, while education debt has been rising, especially since 2013. Another challenge with implications for individuals’ ability to accumulate personal savings is that economy-wide, aggregate health care expenditures are projected to continue to grow as a percentage of the overall economy, and individuals have to contend with rising health care costs as they strive to save for retirement. CMS projections estimate that the annual growth rate of out-of-pocket health care spending for the U.S. population, per capita, will increase from 3.0 percent in 2018 to about 3.8 percent by 2026. While these costs are projected to rise for the population as a whole, individuals age 65 and over face the highest out-of-pocket health-related expenses. Further, health care expenses can be larger relative to other expenses for many retirees and hard to predict, making the amount of income retirees need to plan to spend on health care difficult to determine. Simultaneously, trends in longer life expectancy have the potential to increase economic vulnerability for retirees. Specifically, life expectancy for those age 65 or older has increased significantly over the past century and is projected to continue to increase. For example, a man turning 65 in 2030 is expected to live to age 85.0, on average, an additional 5.3 years compared to a man who turned 65 in 1980, who was only expected to live to age 79.7, on average. A woman turning 65 in 2030 is expected to live to age 87.2, on average, an additional 3.5 years compared to a woman who turned 65 in 1980, who was only expected to live to age 83.8, on average. Moreover, these life expectancies are averages, with some individuals living well beyond their life expectancy. As a result, people must now prepare for this greater longevity risk—that is, the risk that they will spend more years in retirement and potentially outlive their savings. For those who lack sufficient personal savings or other assets to augment their Social Security benefit or income from any employer-sponsored plan, the only option to maintain a desired standard of living may be to continue working past age 65. Our prior work has found that labor force participation among older workers has increased during the last decade and that, compared to current retirees, workers age 55 or older were more likely to expect to retire later and to work during retirement. Our prior work has also identified challenges maintaining retirement savings should older workers become unemployed. Over the past 40 years, the nation has taken an incremental approach to addressing the U.S. retirement system; however, such an approach may not be able to effectively address the interrelated foundational nature of the challenges facing the system today. Without a more comprehensive re-evaluation of the myriad challenges across all three pillars of the retirement system, identifying effective, enduring solutions may be difficult, and the consequences could be significant. Unless timely action is taken, many older Americans risk not having sufficient means for a secure and dignified retirement in the future. Congress has generally sought to address retirement-related issues and concerns one issue at a time. As highlighted in appendix II, at least 25 laws pertaining to retirement have been enacted since ERISA. Some laws—such as the Social Security Amendments of 1983 and the Pension Protection Act of 2006—made large changes to the retirement system. Other laws were more targeted. For example in 1984, Congress amended ERISA to address concerns that women were not receiving their share of private pension benefits by, among other things, permitting certain breaks in service without loss of pension credits, and changing treatment of pension benefits for widowed and divorced spouses. Similarly, in 1996, Congress created a simplified retirement savings vehicle for employers with 100 or fewer employees to help address concerns that smaller employers were not sponsoring plans. The number of agencies that play roles in the current retirement system has also contributed to the incremental approach to addressing concerns, with no single federal agency being responsible for taking a broad view of the system as a whole. As described earlier, there are at least 10 agencies that have a role in overseeing some part of the system, or that are involved in providing supports and services to older Americans. In addition to DOL, IRS, and PBGC, which are the agencies generally responsible for administering ERISA, SSA administers the Social Security program; and the Department of Health and Human Services oversees CMS, which administers the health care programs for retirees. In addition, various other agencies play a role in providing a range of services and supports to assist older adults through retirement. Having multiple agencies involved in the system has also contributed to a complex web of programs and requirements. For example, our prior work identified more than 130 reports and disclosures stemming from provisions of ERISA and the Internal Revenue Code. Although each plan sponsor is required to submit only certain of these reports and disclosures, determining which ones can be challenging, and we found that the agencies’ online resources to aid plan sponsors with this task were neither comprehensive nor up to date. We made several recommendations to address these issues that have not been fully implemented. While three federal commissions have focused on various retirement issues (see app. III), it has been nearly 40 years since the last comprehensive evaluation of the nation’s approach to financing retirement by a federal commission. The 1979 President’s Commission on Pension Policy conducted a broad study of retirement-related issues and made a series of over-arching recommendations, such as creation of a minimum universal pension system that would provide a portable benefit for all workers that would be a supplement to Social Security. Other recommendations included federal protections for participants in state and local government plans, more consistent tax treatment of pension plans and retirement savings vehicles, provisions to strengthen Social Security, as well as proposals regarding employment of older workers and disability programs. However, many of the commission’s recommendations were not implemented. The issues identified nearly 40 years ago by the 1979 commission’s comprehensive re-evaluation of the U.S. retirement system continue to be issues facing the nation today. In fact, these issues have only become more complex and more urgent due to fundamental changes that have occurred since 1979—especially the growing fiscal exposure to the federal government and the shift from DB to DC plans, with its associated increase in risks and responsibilities for individual workers. Taken together, these changes may make it harder for retirees to achieve financial security in retirement, especially for those without access to employer-sponsored plans and at the lower end of the income scale. A panel of 15 retirement experts convened by GAO in November 2016 agreed that there is a need for a new comprehensive evaluation of the U.S. retirement system. They noted weaknesses in the current system’s ability to help ensure that all individuals can provide for a secure retirement. They also discussed the burden that the current system’s complexity places on individuals, employers, and federal government. Although there was agreement among many panelists that a more comprehensive approach would be needed to provide a secure retirement for future retirees, opinions varied on the types of solutions needed. For example, some panelists suggested that a new government-sponsored savings vehicle should be created, while others supported modifying the existing employer-sponsored system to make any needed changes. In addition, several panelists commented on how the current system can be overly complex and confusing for employers, especially small employers. They discussed how the current private sector system poses financial and litigation risk for employers, especially with respect to investment decisions, fiduciary duty, and fees. For example, one panelist suggested that DC plan sponsors may welcome the federal government providing more guidance on the types of investments that would be regarded as prudent and safe as a way to reduce their litigation risk. Panelists also noted that the experiences of other countries can provide useful insights for ways to improve U.S. retirement programs and policies. For example, some panelists described the approach being taken by the United Kingdom (UK) as a potential model for expanding access to retirement savings plans. In the UK model, universal access for workers was implemented by mandating that all employers automatically enroll employees in either their own or the government-sponsored retirement savings plan, the National Employment Savings Trust. In our 2017 report, we suggested five policy goals for a reformed U.S. retirement system as a starting point for discussion: (1) promoting universal access to a retirement savings vehicle, (2) ensuring greater retirement income adequacy, (3) improving options for the spend down phase of retirement, (4) reducing complexity and risk for both participants and plan sponsors, and (5) stabilizing fiscal exposure to the federal government (see table 1 for more detail on these goals). Reforming the nation’s retirement system to create a system that meets all of these goals, or others identified by the Congress, will require a careful and deliberative approach. For example, some type of consensus about the goals would need to be established as a first step. Broad questions are likely to be raised about how each of the goals should be achieved. The examination of relevant issues by past federal commissions, the discussions at our November 2016 panel, as well as what we can learn from the experiences of other countries, further illustrate how complex any reform effort is likely to be. Also, we recognize that some of these goals may compete with each other—in particular, ensuring greater retirement security and minimizing fiscal exposure to the federal government. Therefore, a balanced approach will be required, which can only result from a more holistic examination of the issues by those representing a broad range of perspectives. As a result, we recommended that Congress consider establishing an independent commission to comprehensively examine the U.S. retirement system and make recommendations to clarify key policy goals for the system and improve the nation’s approach to promoting more stable retirement security. We suggested that such a commission include representatives from government agencies, employers, the financial services industry, unions, participant advocates, and researchers, among others, to help inform policymakers on changes needed to improve the current U.S. retirement system. Chairman Collins, Ranking Member Casey, and Members of the Committee, this concludes my prepared remarks. I would be happy to answer any questions that you may have. For further information regarding this testimony, please contact Charles A. Jeszeck at (202) 512-7215 or jeszeckc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contact above, Margie K. Shields, Assistant Director; Jennifer Gregory, Analyst-in-Charge; Justine Augeri; and Gustavo O. Fernandez made key contributions to this publication. Also contributing to this report were Barbara D. Bovbjerg, Managing Director, Education, Workforce, and Income Security Issues; Oliver Richard, Chief Economist; Frank Todisco, Chief Actuary; James Bennett, Deborah Bland, Corinna Nicolaou, and Adam Wendel, with assistance from others who worked on our 2017 report. We convened a panel of retirement experts in November 2016 to obtain their insights on the condition of retirement in the United States and various options for a new approach to help ensure that all individuals can provide for a secure retirement. This appendix provides a description of our methodology for selecting the panel. (See text box for final list of 15 experts participating in our panel.) To identify the experts to invite to this meeting, we compiled an initial list based on interviews with experts conducted during recent GAO retirement income security work and the organizations invited to participate in a 2005 GAO forum on the future of the defined benefit system and the Pension Benefit Guaranty Corporation. Potential experts were identified based on the following criteria: Organizational type: To ensure that we considered the unique roles or situations of various entities involved in retirement income policy, we selected panelists from the federal government, state or local government, research institutes or universities, advocacy or membership organizations, and financial services firms. Organizational reputation: To ensure that our panelists span political perspectives, we selected panelists from organizations known to be conservative, moderate, and liberal (to the extent the reputation for the organization could be easily identified). Subject matter expertise: To ensure that the discussion considered as many aspects of retirement income security as possible, we selected panelists with expertise across a range of areas, including defined benefit (DB) plans, defined contribution (DC) plans, individual retirement accounts (IRA), demographic trends, vulnerable populations, actuarial science, income in retirement, financial literacy, and behavioral finance. Range of views: To ensure that our discussion was inclusive of different philosophies regarding the role of government with regard to the population and the economy, we selected panelists to represent the viewpoints of individuals and business. Representation of diverse groups: To ensure that the discussion benefited from different viewpoints, we selected panelists to reflect gender, racial, and ethnic diversity. An initial list of 41 potential experts was shared with GAO management officials with expertise in retirement issues, actuarial science, and strategic planning, as well as GAO methodologists, for their comments and suggestions. From this, we developed a shorter list eventually arriving at our final group of 15, listed above. These final 15 panelists were also evaluated for conflicts of interest. A conflict of interest was considered to be any current financial or other interest that might conflict with the service of an individual because it (1) could impair objectivity and (2) could create an unfair competitive advantage for any person or organization. All potential conflicts were discussed by GAO staff. The 15 experts were determined to be free of conflicts of interest, and the group as a whole was judged to have no inappropriate biases. Panelists engaged in a day-long discussion about our nation’s approach to retirement policy (see text box). The discussion was guided by a list of questions developed in advance, and the meeting was conducted by a GAO moderator to ensure that all panelists had an opportunity to participate and provide responses. State of Retirement Expert Panel Agenda Session 1: How Well Is Our Current National Approach to Retirement Security Working? Preamble: Retirement income sources in the United States have often been referred to as a three-legged stool – Social Security, employer-sponsored retirement plans, and personal savings. 1. Can the U.S, retirement system today still be accurately described by these three retirement income sources? Why/why not? 2. Are there aspects of our nation’s approach to retirement income security that are working well? If so, are these aspects functioning well for all, or only for particular populations? 3. Are there aspects of our nation’s approach to retirement income security that are concerning? If so, what are your biggest concerns? 4. Are there any specific populations you are particularly concerned about? If so, which ones and why? Session 2: Reevaluating the Roles of the Federal Government, Employers, and Individuals Preamble: Key actors in assuring a secure retirement have traditionally included the federal government, employers, and individuals, but their roles have evolved over time. Are there ways roles could or should be adapted or modified to address the strengths and weaknesses that have been identified for: o Federal government? o Employers? o Individuals? Session 3: Reevaluating Our Nation’s Approach to Retirement Policy Preamble: Various proposals for a broader, more cohesive approach to retirement policy have been made over time. 1. Do you believe there is a need for some type of national retirement policy? 2. If such a policy were to be proposed-- 2a. What could or should be the primary goals of such a policy? 2b. What could or should be the roles of key actors in achieving those goals? 3. What do you believe could be the greatest benefits of a national retirement policy? 4. What do you believe could be the greatest risks or potential downsides of a national retirement policy? What barriers exist to creating a national retirement policy and how could the federal government best address these barriers? The chronology highlights below selected federal legislation related to retirement security in the United States since 1960. It is based on a larger chronology included in our prior special product on the nation’s retirement system (GAO-18-111SP). The chronology is intended to illustrate the incremental approach that the nation has taken to improving the U.S. retirement system and to convey the changes that the legislation enacted at the time. It is not intended to provide an exhaustive list of legislation that has impacted retirement in the United States, to make statements about current provisions of the law, or to provide comprehensive descriptions of each law. Chronology of Selected Federal Legislation Shaping Retirement in the United States (1960–Present) 1961 Social Security Amendments of 1961 Selected provision: Enacted a provision for men, comparable to the provision enacted for women in 1956, concerning early retirement at age 62. Self-Employed Individuals Tax Retirement Act of 1962 Selected provision: Imposed minimum distribution requirements for self-employed participants in a qualified plan generally beginning at age 70 ½. Social Security Amendments of 1965 Selected provisions: Enacted new titles to the Social Security Act for Medicare and Medicaid. Medicare provided hospital, post-hospital extended care, and home health coverage to almost all Americans age 65 or older; Medicaid provided states with the option of receiving federal funding for providing health care services to certain low-income and medically needy individuals. Age Discrimination in Employment Act of 1967 Selected provisions: Made it unlawful for an employer to discriminate against any individual with respect to compensation, terms, conditions, or privileges of employment because of age; and required the Secretary of Labor to carry on a continuing program of education and information, which could include research with a view to reducing barriers to the employment of older persons. Employee Retirement Income Security Act of 1974 (ERISA) Selected provisions: Regulated private sector employers who offer pension or welfare benefit plans for their employees. Title I: Imposed reporting and disclosure requirements on plans; imposed certain responsibilities on plan fiduciaries. Title II: Strengthened participation requirements for employees age 25 and over; established vesting rules; required that a joint and survivor annuity be provided; and established minimum funding standards. In addition, provided individual retirement accounts (IRAs) for persons not covered by pensions. Title IV: Required certain employers and plan administrators to fund an insurance system to protect certain kinds of retirement benefits (i.e., to pay premiums to the federal government’s Pension Benefit Guaranty Corporation (PBGC)). Revenue Act of 1978 Selected provisions: Established qualified deferred compensation plans called 401(k) plans after 26 U.S.C. § 401(k), which allowed for pre-tax employee contributions to such plans (known as elective deferrals). Multiemployer Pension Plan Amendments Act of 1980 Selected provisions: Strengthened the funding requirements for multiemployer pension plans; authorized plan preservation measures for financially troubled multiemployer plans; and revised the manner in which insurance provisions applied to multiemployer plans. Tax Equity and Fiscal Responsibility Act of 1982 Selected provisions: Reduced the maximum annual addition (employer contributions, employee contributions, and forfeitures) for each participant in a defined contribution (DC) plan; reduced the maximum annual retirement benefit for each participant in a defined benefit (DB) plan; introduced special rules for “top heavy” plans (i.e., plans in which more than 60 percent of the present value of the cumulative accrued benefits under the plan for all employees accrue to key employees, including certain owners and officers); and expanded minimum distribution requirements to all qualified plans. Social Security Amendments of 1983 Selected provisions: Gradually raised the normal retirement age from 65 to 67, depending on an individuals’ year of birth; expanded coverage; increased the self-employment tax for self-employed persons; subjected a portion of Social Security benefits to federal income tax for the first time; and changed how cost-of-living adjustments are calculated when trust funds are low. Deficit Reduction Act of 1984 Selected provisions: Amended nondiscrimination testing requirements for 401(k) plans and required minimum distribution rules, and restricted prefunding of certain employee post-retirement welfare benefits (such as disability and medical benefits). Retirement Equity Act of 1984 Selected provisions: Changed participation rules by lowering the minimum age that a plan may require for enrollment (from age 25 to 21), and permitted certain breaks in service without loss of pension credits. Also, strengthened treatment of pension benefits for widowed and divorced spouses. Single-Employer Pension Plan Amendments Act of 1986 Selected provisions: Raised the per-participant PBGC premium from $2.60 to $8.50; established certain distress criteria that a contributing sponsor or substantial member of a contributing sponsor’s controlled group must meet in order to terminate a single-employer plan under a distress termination; established certain criteria for PBGC to terminate a plan that does not have sufficient assets to pay benefits that are currently due (referred to as “involuntary terminations”); and created a new liability to plan participants for certain non-guaranteed benefits. Federal Employees’ Retirement System Act of 1986 Selected provisions: Established the Federal Employees’ Retirement System (FERS). Unlike the existing Civil Service Retirement System (CSRS), retirement and disability benefits under FERS were structured to be fully funded by employee and employer contributions and interest earned by the bonds in which the contributions were invested. The DB under FERS was lower than under CSRS, but FERS also included a DC plan component: the Thrift Savings Plan. Omnibus Budget Reconciliation Act of 1986 Selected provisions: Required employers that sponsor pension (DB plans) and retirement savings plans (DC plans such as a 401(k)) to provide benefit accruals or allocations for employees who work beyond their normal retirement age. Tax Reform Act of 1986 Selected provisions: Established faster minimum vesting schedules; adjusted limitations on contributions and benefits for qualified plans; limited the exclusion for employee elective deferrals to $7,000; and amended nondiscrimination coverage rules. Also, restricted the allowable tax-deductible contributions to IRAs for individuals with incomes above a certain level and who participate in employer-sponsored pension plans, and imposed an additional 10 percent tax on early distributions (before age 59 ½) from a qualified retirement plan. Omnibus Budget Reconciliation Act of 1987 Selected provisions: Strengthened funding rules for pension plans and the level and structure of PBGC premiums. Omnibus Budget Reconciliation Act of 1993 Selected provisions: Reduced compensation taken into account in determining contributions and benefits under qualified retirement plans, and expanded taxation of Social Security benefits. Retirement Protection Act of 1994 Selected provisions: Strengthened funding rules for pension plans. Small Business Job Protection Act of 1996 Selected provisions: Created a type of simplified retirement savings vehicle for small employers; added a nondiscrimination safe harbor for 401(k) plans; amended the definition highly compensated employee; and modified certain participation rules for DC plans. Taxpayer Relief Act of 1997 Selected provision: Established Roth IRAs, under which contributions are after-tax, but distributions after age 59½ are tax-free. Senior Citizens’ Freedom to Work Act of 2000 Selected provision: Amended the Social Security Act to eliminate the earnings limit for individuals who have reached their normal retirement age. Economic Growth and Tax Relief Reconciliation Act of 2001 (EGTRRA) Selected provisions: Increased the individual elective deferrals that may be made to a 401(k) plan; added “catch-up contributions” that allow individuals age 50 or older to make additional contributions; increased the maximum annual contributions to DC plans and individual retirement accounts; increased the maximum annual benefits under a DB plan; increased the compensation limit for qualified trusts; reduced the minimum vesting requirements for matching contributions; and changed the rules that permit plans to cash-out, without consent. Sarbanes-Oxley Act of 2002 Selected provision: Added a new requirement that individual account pension plans provide notice to participants and beneficiaries in advance of periods during which the ability of participants or beneficiaries to take certain actions with respect to their accounts will be temporarily suspended, limited, or restricted (referred to as “blackout periods”). Deficit Reduction Act of 2005 Selected provisions: For plan years that begin after December 31, 2005, set the PBGC flat-rate premium for multiemployer plans at $8.00; and, for each plan year that begins after 2006, indexed future premium levels to the national average wage index. Pension Protection Act of 2006 (PPA) Selected provisions: Strengthened the minimum funding requirements for DB plans; set certain benefit limitations for underfunded DB plans; enhanced the protections for spouses; amended plan asset diversification requirements; changed provisions concerning the portability of pension plans; allowed the adoption of automatic enrollment and target date funds for DC plans; and increased reporting and disclosure requirements for plan sponsors. Worker, Retiree, and Employer Recovery Act of 2008 Selected provision: Modified PPA’s funding requirements to grant relief for single-employer DB plans. Moving Ahead for Progress in the 21st Century Act (MAP-21) Selected provisions: Provided funding relief for single-employer DB plans by changing the interest rates used to reflect a 25-year historical average; increased premium rates for sponsors of single-employer and multiemployer DB plans; and included other provisions intended to improve the governance of PBGC. American Taxpayer Relief Act of 2012 Selected provisions: Extended the tax-free treatment of distributions from IRAs made for charitable purposes; allowed for certain in-plan transfers to a Roth account. Multiemployer Pension Reform Act of 2014 (MPRA) Selected provisions: Allowed severely underfunded multiemployer plans, under certain conditions and with the approval of federal regulators, the option to reduce the retirement benefits of current retirees to avoid plan insolvency; and expanded PBGC’s ability to intervene when plans are in financial distress. Bipartisan Budget Act of 2018 Selected provisions: Established a temporary Joint Select Committee on Solvency of Multiemployer Pension Plans. The goal of the Joint Select Committee was to improve the solvency of multiemployer pension plans and PBGC. Since the enactment of ERISA, there have been three federal commissions on retirement issues: The President’s Commission on Pension Policy, the National Commission on Social Security Reform, and the President’s Commission to Strengthen Social Security (see table 2). We examined these commissions to gain insights on possible structures for federal commissions, the scope of work these commissions can take on, and the types of recommendations they can make. In 1978, President Carter signed an executive order authorizing the Carter Commission, which was established when committee members were appointed in 1979. The commission was to conduct a 2-year sturdy of the nation’s pension systems and the future course of national retirement income policies. President Carter appointed all 11 commission members. The commission also had an executive director and 37 staffers. Its final report, Coming of Age: Toward a National Retirement Income Policy, was released in February 1981. The commission was ordered to: Conduct a comprehensive review of retirement, survivor, and disability programs existing in the United States, including private, federal, state, and local programs. Develop national policies for retirement, survivor, and disability programs that can be used as a guide by public and private programs. The policies were to be designed to ensure that the nation had effective and equitable retirement, survivor, and disability programs that took into account available resources and demographic changes expected into the middle of the next century. Submit to the President a series of reports including the commission’s findings and recommendations on short-term and long-term issues with respect to retirement, survivor, and disability programs. The commission was charged with covering the following issues in its findings and recommendations: overlaps and gaps among the private, state, and local sectors in providing income to retired, surviving, and disabled persons; the financial ability of private, federal, state, and local retirement, survivor, and disability systems to meet their future obligations; appropriate retirement ages, the relationship of annuity levels to past earnings and contributions, and the role of retirement, survivor, and disability programs in private capital formation and economic growth; the implications of the recommended national policies for the financing and benefit structures of the retirement, survivor, and disability programs in the public and private sectors; and specific reforms and organizational changes in the present systems that may be required to meet the goals of the national policies. In its final report, the Carter Commission prescribed a goal for retirement income policy and made numerous recommendations. According to the report, a desirable retirement income goal is the replacement of pre- retirement income from all sources. Recommendations focused on strengthening four areas: employer pensions, Social Security, “individual efforts” (personal savings, employment of older workers, and disability), and public assistance. Recommendations were also made regarding the administration of the U.S. retirement system. Examples of ways to strengthen each area follow: Strengthening Employer Pensions. The commission recommended establishing a Minimum Universal Pension System (MUPS) for all workers. MUPS was intended to provide a portable benefit that was supplemental to Social Security. It would have built upon existing employer plans and existing plans that did not meet the requirements would have needed to be amended. Another recommendation was to establish a Public Employee Retirement Income Security Act (i.e. a public sector version of ERISA) so that public and private sector employees would receive similar protections. Strengthening Social Security. The commission recommended mandatory universal coverage, raising the retirement age for workers who were not approaching retirement, re-examining or making adjustments to the special minimum benefit as well as the spousal benefit and other miscellaneous benefits. Strengthening Individual Efforts. The commission recommended that contribution and benefit limitations for all individuals should be treated more consistently for all types of retirement savings. The commission also recommended a refundable tax credit for low- and moderate-income individuals to encourage saving for retirement. For older workers, recommendations included improving unemployment benefits to provide short-term income maintenance and keep them in the labor force. The commission also recommended further in-depth study of the Disability Insurance program. Strengthening Public Assistance. The commission made recommendations to address inflation protection for retirement income and setting Social Security’s Supplemental Security Income at the poverty line level and eliminating its assets test. Administration. The commission recommended consolidating the administration of all federal retirement systems as well as consolidating ERISA administrative functions under one entity. It also recommended an interdepartmental task force to coordinate executive branch agencies dealing with retirement income. In 1981, President Reagan signed an executive order establishing the Greenspan Commission. The President asked the commission to conduct a 1-year study and propose realistic, long-term reforms to put Social Security on sound financial footing and to reach bipartisan consensus so these reforms could be passed into law. The President, the Senate Majority Leader, and the Speaker of the House of Representatives each made five appointments, with no more than three of the five appointments coming from one political party to ensure a bipartisan commission. The President was responsible for appointing the commission’s chair. The commission had a staff of 23. The final report, Report of the National Commission on Social Security Reform, was issued on January 20, 1983. The commission was ordered to Review relevant analyses of the current and long-term financial condition of the Social Security Trust Funds Identify problems that could threaten the long-term solvency of such funds Analyze potential solutions to such problems that would both assure the financial integrity of the Social Security system and appropriate benefits Provide appropriate recommendations to the Secretary of Health and Human Services, the President, and Congress. In its final report, the Greenspan Commission found both short and long- term financing problems and recommended that action should be taken to strengthen the financial status of the Social Security program. Twelve commission members voted in favor of a consensus package with 13 recommendations to address Social Security’s short-term deficit, including, for example: Expand Social Security to include coverage for nonprofit and civilian federal employees hired after January 1, 1984, as well as prohibiting the withdrawal of state and local employees. Shift cost-of-living adjustments to an annual basis. Make the Social Security Administration its own separate, independent agency. Make adjustments to spousal and survivor benefits. Revise the schedule for Social Security payroll taxes. Establish the taxation of benefits for higher-income persons. In addition, these 12 commission members agreed that the long-range deficit should be reduced to approximately zero, and their recommendations were projected to meet about two-thirds of the long- range financial deficit. Seven of the 12 members agreed that the remaining one-third of the long-range financial deficit should be met by a deferred, gradual increase in the normal retirement age, while the other 5 members agreed that it should be met by an increase in future contribution rates starting in 2010. After the Greenspan Commission’s final report was issued, Congress enacted the Social Security Amendments of 1983. The amendments incorporated many of the Greenspan Commission’s recommendations and made comprehensive changes to Social Security coverage, financing, and benefit structure. These changes included addressing Social Security’s long-term financing problems by gradually increasing the retirement age from 65 to 67, among other things. In 2001, President Bush signed an executive order establishing the President’s Commission to Strengthen Social Security. The President asked the Commission to produce an interim report describing the challenges facing the Social Security system and the criteria by which the Commission would evaluate reform proposals, as well as a final report to set forth the Commission’s recommendations regarding how to strengthen Social Security with personal accounts. The commission had a staff of sixteen members appointed by the President, of which no more than eight members were of the same political party. The final report, Strengthening Social Security and Creating Personal Wealth for All Americans, was issued in December 2001. The commission was asked to submit to the President bipartisan recommendations to modernize and restore fiscal soundness to the Social Security system according to the following principles: Modernization must not change Social Security benefits for retirees or The entire Social Security surplus must be dedicated to Social Social Security payroll taxes must not be increased; Government must not invest Social Security funds in the stock market; Modernization must preserve Social Security’s disability and survivors Modernization must include individually controlled, voluntary personal retirement accounts, which will augment the Social Security safety net. In its final report, the Commission offered three models for Social Security reform. All three models shared a common framework whereby voluntary individual accounts were established in exchange for a reduction in the Social Security defined portion of benefit. According to the report: Reform Model 1 would have established a voluntary personal account option, but did not specify other changes in Social Security’s benefit and revenue structure and was intended to achieve full long-term sustainability. Reform Model 2 would have enabled future retirees to receive Social Security benefits that would be at least as great as then current retirees and increased Social Security benefits paid to low-income workers. Model 2 would have established a voluntary personal account without raising taxes or requiring additional worker contributions. It was intended to achieve solvency and balanced Social Security revenues and costs. Reform Model 3 would have established a voluntary personal account option that generally enabled workers to reach or exceed then-current scheduled benefits and wage replacement ratios. It was intended to achieve solvency by adding revenues and by slowing benefit growth less than price indexing. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Strengthening the U.S. retirement system to be more accessible and financially sound is important to ensuring that all Americans can retire with dignity and security, and to managing the fiscal exposures to the federal government from various retirement-related programs. Currently, the U.S. retirement system, and many of the workers and retirees it was designed to help, face major challenges. This testimony discusses (1) the fiscal risks and other challenges facing the U.S. retirement system, and (2) the need to re-evaluate our nation's approach to financing retirement. It is based on a 2017 report, GAO-18-111SP , on the nation's retirement system, with updated statistics when more recent estimates from publicly available sources were available. Fundamental changes over the past 40 years have led to various risks and challenges for the three main pillars supporting the U.S. retirement system. For example, current projections indicate that by 2034, the Old-Age and Survivors trust fund for Social Security's retirement program—the first pillar—will only be sufficient to pay 77 percent of scheduled benefits, due in part to the aging of the population (see figure). Other federal government retirement-related programs also face financial uncertainty. For example, the Pension Benefit Guaranty Corporation, which insures the pension benefits of most private sector defined benefit plans, estimates a greater than 90 percent chance the multiemployer program will be insolvent by 2025. Meanwhile, employer-sponsored plans—the second pillar—have experienced a shift from traditional defined benefit (DB) plans that generally provide set monthly payments for life, to defined contribution (DC) account-based plans, like 401(k)s. DC plans provide greater portability of savings that can be better suited to the needs of a more mobile workforce, but also require individuals to assume more responsibility for planning and managing their savings. While DC plans can provide meaningful retirement security for many, especially higher earners, lower earners appear more prone to having little or no savings in their DC accounts. Further, individuals' savings—the third pillar—may be constrained by economic trends such as low real wage growth and growing out-of-pocket health care costs. Combined with increased longevity, these challenges can put individuals at greater risk of outliving their savings and fiscal pressures on government programs will likely grow. Congress generally has sought to address retirement-related issues in an incremental fashion. Also, no one agency is responsible for overseeing the U.S. retirement system in its entirety, so there is no obvious federal agency to lead a comprehensive reform effort. It has been nearly 40 years since a federal commission has conducted a comprehensive evaluation of the nation's approach to financing retirement. Without a more comprehensive re-evaluation of the challenges across all three pillars of the system, it may be difficult to identify effective, enduring solutions. Unless timely action is taken, many older Americans risk not having sufficient means for a secure and dignified retirement. In the 2017 report, GAO recommended that Congress should consider establishing an independent commission to comprehensively examine the US retirement system and make recommendations to clarify key policy goals for the system and improve how the nation promotes retirement security.", "document_type": "gao"}
{"report": "Established by the National Housing Act, FHA’s single-family mortgage insurance program helps home buyers obtain financing by providing insurance on single-family mortgage loans. The mortgage insurance allows FHA-approved private lenders to provide qualified borrowers with mortgages on properties with one to four housing units and generally compensates lenders for nearly all the losses incurred on such loans. To support the program, FHA imposes up-front and annual mortgage insurance premiums on FHA borrowers. The agency has played a particularly large role among first-time, minority, and low-income home buyers. For example, in fiscal year 2017, about 82 percent of FHA- insured home purchase loans went to first-time home buyers and more than 33 percent went to minority home buyers. FHA requires servicers to undertake certain home retention and foreclosure mitigation actions to help delinquent homeowners catch up on late mortgage payments. Before initiating foreclosure actions, FHA requires servicers to contact the borrower, collect information on the borrower’s finances, and attempt informal methods of resolving the delinquency. If informal steps are not appropriate for a borrower’s circumstances, the servicer evaluates the borrower for a series of home retention actions, which include a formal forbearance and repayment plan and a loan modification. Under certain circumstances, the servicer may consider a foreclosure mitigation option, such as a preforeclosure (short) sale or a deed-in-lieu of foreclosure. If the home retention and foreclosure mitigation actions are unsuccessful, the servicer or mortgage note holder is generally entitled to pursue foreclosure to obtain title to the property. The foreclosure process is governed by state laws, but foreclosed properties are typically auctioned at a foreclosure sale. Most foreclosed properties are disposed of in one of two ways. Claims without Conveyance of Title (CWCOT). Through FHA’s CWCOT program, the servicer attempts to secure a third-party purchase of an eligible property for an adjusted fair market value that is less than the amount of the servicer’s projected claim. Conveyance. If the foreclosure process is completed and no third party purchases the home at the foreclosure sale, the home usually becomes the property of the servicer. Servicers convey these properties to FHA, which sells them out of its REO inventory. During the default and foreclosure process, servicers must meet two FHA time requirements. The first requires servicers to initiate a foreclosure (first legal action) or utilize a loss mitigation option within 6 months of borrower default. The second requirement, for the “reasonable diligence” period, requires servicers to obtain good and marketable title and possession of a property within a specified time frame that varies by state. The servicer secures the property and obtains possession once the property is vacant. Servicers are subject to financial penalties for missing these deadlines. In both cases, servicers must curtail the debenture interest that they otherwise would be entitled to collect from the date of the missed time frame. Servicers are responsible for maintaining vacant foreclosed properties in accordance with FHA requirements, which specify allowable reimbursable amounts to preserve and protect the property. A servicer needing additional funds to complete the required maintenance must submit an “overallowable” request to FHA. When a servicer forecloses on a property with an FHA-insured mortgage and the property is not sold to a third party through CWCOT, the property is held in the servicer’s name until the servicer conveys the title to FHA. As seen in figure 1, these properties span a range of home types and ages. FHA requires servicers to preserve and protect the property and ensure it meets FHA’s conveyance condition standards before conveying title. FHA’s preservation and protection requirements include a number of specific steps for securing, maintaining, and repairing properties and documenting property conditions. FHA reimburses the servicer for up to $5,000 per property for required work, and the servicer may request overallowable funds if needed. FHA’s conveyance condition standards are broader requirements, including that a property be undamaged by natural disaster and in “broom swept” condition, have all damage covered by hazard insurance repaired, and be undamaged by the servicer’s failure to properly secure or maintain the property. HUD regulations state that the servicer must obtain good and marketable title and convey the property to HUD within 30 days of the date on which the servicer filed the foreclosure deed for record or certain other key dates, whichever is later. If a servicer does not believe it will be able to convey the property by this time, it may request an extension from FHA. The servicer files an insurance claim with FHA when it conveys the title. If a servicer does not convey the property within the required time frame and has not received an approved extension, the servicer must curtail the debenture interest and property preservation and protection expenses it claims as of the date of the missed deadline. Shortly after conveyance, FHA pays the Part A claim to the servicer, which includes the unpaid principal balance and debenture interest on the insured mortgage. At this time, FHA becomes responsible for maintaining the property until it is sold. FHA pays the part of the claim that covers eligible property preservation and protection expenses incurred by the servicer once the servicer submits title evidence and documentation of expenses (in Part B of the claim form). FHA inspects the property and reviews title evidence before selling the property out of its REO inventory. In some cases, FHA reconveys the title to the servicer if it finds the servicer did not comply with requirements related to property condition or title. When a property is reconveyed, FHA reassigns the title to the servicer and requests repayment of the claim amount. The servicer then must correct any title or property condition issues before it may convey the property and submit a claim to FHA again. Throughout this process, FHA and servicers use the P260 Asset Disposition and Management System (asset disposition system) to communicate and upload documentation about the properties. FHA articulates its property preservation and protection requirements and conveyance condition standards in a mortgagee letter and policy handbook that we refer to collectively as FHA’s conveyance condition policies and procedures. From 2010 through 2017, servicers conveyed about 610,000 properties to FHA. The number of properties conveyed annually peaked in 2012 at about 111,000 (see fig. 2). In 2017, servicers conveyed fewer than 32,000 properties to FHA. The decline in recent years is consistent with improvements in the housing market since the 2007–2011 housing crisis. In this report, we define FHA’s property conveyance process as beginning when the servicer both obtains good and marketable title and takes possession of a property and ending when FHA assigns a marketing contractor to sell the property out of its REO inventory (see fig. 3). Several FHA contractors and offices play key roles in the conveyance process. Compliance contractor. A nationwide compliance contractor called the mortgagee compliance manager is responsible for protecting FHA’s interests in properties conveyed to FHA and communicates directly with servicers about the properties. The compliance contractor reviews property inspections to ensure properties meet conveyance condition standards, reviews requests from servicers for extensions of conveyance times or for overallowable expenses, reviews servicer claims for compliance with requirements, and responds to servicer inquiries about pre- and postconveyance responsibilities. The compliance contractor is located in Oklahoma City, Oklahoma, and is overseen by FHA’s National Servicing Center. Maintenance contractor. Maintenance contractors, called field service managers, are responsible for inspecting properties recently conveyed to FHA and preserving properties in FHA’s REO inventory. FHA has multiple maintenance contractors; they are responsible for properties in different regions. Upon conveyance, the maintenance contractor conducts a comprehensive property inspection to determine if the property meets conveyance condition standards and completes the HUD Property Inspection Report. The maintenance contractor also conducts other inspections at a property before conveyance, as warranted, including a preconveyance inspection at the request of the servicer and an overallowable inspection if requested by the compliance contractor. While these contractors conduct general maintenance on the property, they typically do not make major repairs, because FHA generally sells conveyed properties in as-is condition. Marketing contractor. Marketing contractors, called asset managers, are responsible for marketing and selling the homes in FHA’s REO inventory. FHA homeownership centers. FHA carries out its mortgage insurance and REO disposition programs through four regional offices called homeownership centers (HOC). The centers are located in Atlanta, Georgia; Denver, Colorado; Philadelphia, Pennsylvania; and Santa Ana, California. Officials in each HOC are responsible for overseeing the maintenance and marketing contractors for their region and reviewing HUD Property Inspection Reports to determine if conveyed properties should be reconveyed to the servicer due to condition issues. This determination is then forwarded to the compliance contractor for an additional review. HUD’s Office of Finance and Budget. Staff from this office are responsible for reviewing servicer mortgage insurance claims for compliance with FHA requirements. The office selects a sample of claims from the past 3 years to review whether the property preservation and protection expenses were within allowable limits and whether the servicer curtailed debenture interest and property preservation and protection expenses accurately, among other things. A number of other federal and federally sponsored entities participate in the mortgage market. Along with FHA, the Department of Veterans Affairs and the Department of Agriculture operate programs that guarantee single-family mortgages made by private lenders. Additionally, two government-sponsored enterprises—Fannie Mae and Freddie Mac (enterprises)—purchase and securitize single-family mortgages. However, the property disposition programs for these entities are not directly analogous to FHA’s. In contrast to FHA, the Department of Veterans Affairs and the enterprises take custody of and are responsible for properties closer to the time of the foreclosure sale. The enterprises require servicers to convey properties to them within 24 hours of foreclosure sale or deed-in-lieu of foreclosure, while the Department of Veterans Affairs requires servicers to provide notice of their intent to convey properties within 15 days of foreclosure sale. Also in contrast to FHA, the Department of Agriculture does not take possession of foreclosed properties with guaranteed loans, but rather oversees their disposition by lenders. In FHA’s case, properties are often held in the lender’s or servicer’s name for an extended period after the foreclosure sale. Following the foreclosure sale, FHA requires servicers to oversee properties during redemption periods, to evict residents if properties not in redemption periods are occupied, and to continue property preservation and protection activities. In addition, before conveyance, servicers must identify and pay any homeowners association (HOA) fees and utility bills that are due. As described in figure 3, servicers also must make any required repairs, meet other conveyance requirements, and pass an FHA property inspection, or face the prospect of having the property reconveyed. FHA officials said this approach reduces FHA’s holding time and costs and that the agency does not have the infrastructure to manage and fund property repairs itself. From July 2010 through December 2017, the property conveyance process took a median of 70 days, but this figure varied widely by year. Our analysis of FHA data found that, from 2011 through 2015, the median number of days to complete the conveyance process increased four-fold (from 41 to 161 days) and varied more widely around the median each successive year (see fig. 4). Conveyance time frames declined substantially in 2016 and 2017 (to a median of 137 days and 112 days, respectively) while continuing to vary considerably around the median. In comparison, FHA officials said the conveyance process generally should take about 37 days to complete—30 days for servicers to make necessary repairs and convey title to FHA and 7 days for FHA to inspect the property, communicate any condition issues identified during the inspection, and assign a marketing contractor to promote and sell it. We also found that the time it took properties to complete the conveyance process varied by HOC region. For the entire July 2010–December 2017 period, the Philadelphia HOC had the highest median time frame (91 days) and the Atlanta HOC the lowest (56 days). The Santa Ana and Denver HOCs had medians of 78 and 67 days, respectively. A number of factors may have contributed to differences among the HOCs, such as the number of properties conveyed in each region (which can affect servicer and HOC capacity) and the age of the housing stock (which can affect the time needed to make repairs). The time to complete the conveyance process includes, when applicable, the time needed for FHA to reconvey a property—that is, transfer ownership to the servicer due to condition or title issues—and for the servicer to convey it to FHA a second time. FHA officials said they try to avoid reconveyances because they prolong the conveyance process and result in FHA incurring additional preservation and protection costs. Figure 5 shows examples of condition issues at properties we visited in the Baltimore, Maryland, and Atlanta, Georgia, metropolitan areas that were in the reconveyance process. Our analysis of FHA data found that reconveyances were not common enough to significantly affect median conveyance time frames, but substantially lengthened the conveyance process when they did occur. As shown in table 1, servicers conveyed 406,863 properties to FHA from 2012 through 2017—the period within our scope for which FHA had reliable reconveyance data. In comparison, FHA reconveyed 8,874 properties to servicers during that time frame. The annual number of reconveyances rose from 1,019 in 2012 to 1,935 in 2015, before declining to 1,099 in 2017. We also found that the median time to complete FHA’s conveyance process in 2012–2017 was more than 614 days longer for reconveyed properties than the median for properties not reconveyed. However, the difference between the medians declined over time, dropping from 777 days in 2012 to 267 days in 2017 (see fig. 6). Servicers and FHA must take several steps to complete the conveyance process for reconveyed properties, which may account for some of the length of the time frames. Once the compliance contractor has notified the servicer that a property has condition issues that must be resolved to avoid reconveyance, the servicer may appeal. FHA officials said appeals can add up to 120 days to the conveyance process. If the servicer is unable to resolve the issues and the appeals are denied, FHA reconveys the property and the servicer must reimburse FHA for the original claim amount. The servicer then must complete any required repairs, resolve any title issues, prepare a new evidence package for FHA showing that condition and title issues were addressed, and submit a request to FHA’s compliance contractor to convey the property again. FHA’s compliance contractor then has 10 business days to review the evidence package and notify the servicer of its decision. Once conveyance is approved, the servicer may resubmit a new mortgage insurance claim form and evidence that the property deed has been filed in FHA’s name. Two factors that likely contributed to the increase in the time to complete FHA’s conveyance process are increased use of other disposition methods and property damage stemming from extended default and foreclosure periods. Increased use of third-party sales. FHA data indicate that from 2010 through 2017 servicers increasingly disposed of properties through third- party sales using the CWCOT program. As previously noted, in 2014 FHA began requiring servicers to offer all eligible properties for sale through CWCOT before using the conveyance process. According to our analysis of FHA property disposition data, in fiscal years 2010–2017, the share of properties disposed of through CWCOT rose from about 1.4 percent to almost 44 percent, while the share of conveyance and REO sales dropped from about 84 percent to 42 percent (see fig. 7). The remaining properties were disposed of through notes sales or preforeclosure sales. Increased use of CWCOT may have extended property conveyance time frames for two reasons. First, servicers must attempt to sell all eligible properties through CWCOT while simultaneously preparing them for conveyance, which may add additional time to the conveyance process according to FHA officials. Second, properties conveyed to FHA because they are not eligible for or sold through the CWCOT program are generally in poorer condition and require more repairs, according to servicer representatives. This may contribute to extended conveyance time frames. For example, a representative from one mortgage industry group told us that properties ineligible for CWCOT and conveyed to FHA generally require more than the $5,000 in preservation and protection costs that FHA allows. In these cases, servicers may request additional funds from the compliance contractor, but processing the requests may prolong the conveyance process, as discussed later in this report. Representatives from one servicer and two mortgage industry groups stated they prefer the CWCOT program because it reduces the need to convey properties. They said the conveyance process is costly and comes with the risk of reconveyance. FHA data show that REO sales generally had higher loss severity rates (the financial loss on a defaulted loan as a percentage of the unpaid principal balance) than properties disposed of through alternative methods, including the CWCOT program. For example, for the last quarter of fiscal year 2017, FHA reported that the loss severity rate for properties sold through REO was 54.8 percent, while the combined loss severity rate for properties disposed of through alternative methods was 43.8 percent. However, some of this difference may be attributable to the poorer condition of conveyed properties, as discussed previously. National Mortgage Settlement In February 2012, the Department of Justice and 49 states settled with the five largest mortgage servicers— Ally Financial, Inc. (formerly GMAC), Bank of America Corporation, Citigroup Inc., J.P. Morgan Chase & Co., and Wells Fargo & Company — to address mortgage servicing, foreclosure, and bankruptcy abuses. The agreement settled state and federal investigations finding that these servicers routinely signed foreclosure-related documents without verifying their validity and without the presence of a notary public—a practice known as “robosigning.” Extended default and foreclosure periods. According to FHA officials, properties with long default and foreclosure periods may be in poor condition because they deteriorate if servicers delay property maintenance and repairs. FHA officials said this was common for properties conveyed to FHA after the 2012 National Mortgage Settlement because some servicers delayed foreclosure proceedings to limit their exposure to litigation in 2010 and 2011 (see sidebar). FHA officials said that after the Department of Justice issued the National Mortgage Settlement in February 2012, servicers who had been delaying default and foreclosure started conveying large numbers of properties. According to FHA and servicer representatives, damaged properties can take longer to convey because they require extensive repairs to meet FHA’s conveyance condition standards. The results of our analysis of FHA data are broadly consistent with these observations. The number of properties conveyed to FHA increased by 31 percent (from 84,363 to 110,567) between 2011 and 2012, the year of the settlement. Additionally, the default and foreclosure period for conveyed properties (the time between the borrower defaulting on the mortgage and the servicer obtaining title to and possession of the property) increased over most of the July 2010–December 2017 time frame. As shown in figure 8, the median default and foreclosure period was 416 days for properties conveyed in July–December 2010, peaked at 664 days (about 60 percent higher) for properties conveyed in 2015, and fell to 612 days for 2017 conveyances. The overall upward trend was even more pronounced for properties with default and foreclosure periods at the 75th percentile. The 75th percentile was 555 days for properties conveyed from July through December 2010, peaked at 1,152 days (about 108 percent higher) for properties conveyed in 2016, and declined to 1,068 days for 2017 conveyances. Certain regulatory and policy changes also may have increased the default and foreclosure periods since 2013. HUD issued a mortgagee letter in 2013 that increased the reasonable diligence time frames and allowed servicers additional time to complete foreclosures in certain states. For example, the reasonable diligence time frame for properties in New York increased from 13 to19 months. Also, in 2014 mortgage servicing rules issued by the Consumer Financial Protection Bureau went into effect that restricted servicers’ ability to initiate a foreclosure and gave borrowers additional time to pursue loss mitigation options. Specifically, servicers may not initiate foreclosure proceedings if a borrowers’ application is pending for a loan modification or other alternatives to foreclosure. In addition, we found that properties with longer default and foreclosure periods generally took longer to complete the conveyance process than properties with shorter default and foreclosure periods (see fig. 9). Specifically, from July 2010 through December 2017 properties with the longest default and foreclosure periods—those in the highest quartile— took 93 days at the median to complete the conveyance process and 238 days at the 75th percentile. In comparison, properties with the shortest default and foreclosure periods—those in the lowest quartile—took 57 days at the median to complete the conveyance process and 136 days at the 75th percentile for that same period. As previously stated, FHA officials told us that properties with long default and foreclosure periods may have deteriorated if servicers were not maintaining them. These properties may have required additional repairs to bring them into conveyance condition. As previously noted, overall conveyance time frames declined in 2016 and 2017 from their peak in 2015. FHA officials attributed this improvement largely to the decreasing number of conveyances affected by the National Mortgage Settlement. As discussed earlier, the settlement contributed to a wave of properties that took a long time to convey, potentially due to damage sustained during extended default and foreclosure periods. FHA officials also indicated that the improved housing market in recent years has resulted in fewer foreclosures and, therefore, fewer property conveyances to FHA. Consequently, servicers and contractors may be better able to manage the workload associated with property conveyances and complete the process more quickly. From July 2010 through December 2017, servicers generally did not convey properties to FHA within the regulatory 30-day time frame (preconveyance period). During the preconveyance period, servicers must ensure the property has good and marketable title, conduct routine inspections and maintenance on the property, and ensure the property meets conveyance condition standards. If servicers do not believe they will be able to convey a property within 30 days, they may request an extension. The median number of days servicers took to complete the preconveyance period increased from 31 in July–December 2010 to 140 in 2015 (see fig. 10). This figure declined after 2015, dropping to 101 days in 2017. Variation around the median was considerable, especially in more recent years. For example, in 2017 the time to complete the preconveyance period was 43 days at the 25th percentile, compared with 268 days at the 75th percentile. The percentage of properties for which servicers did not convey in 30 days plus any approved extension grew from about 31 percent in July– December 2010 to about 72 percent in 2017. For the entire period from July 2010 through December 2017, the corresponding percentage was 55 percent. Representatives of 13 of the 20 servicers we interviewed said that meeting the 30-day timeline was one of their top challenges with the conveyance process. Representatives of servicers and mortgage industry groups cited several reasons for servicers needing additional time to convey. For example, representatives of 11 servicers cited the heavily damaged condition of the properties they acquired as one of the primary reasons for not conveying properties within 30 days. Servicer representatives also noted other reasons, including four who cited waiting for responses on hazard insurance claims and five who cited difficulty in obtaining HOA bills to pay. In addition, representatives of two mortgage industry groups and three servicers told us that meeting all the conveyance and title requirements simultaneously is a major challenge. For example, representatives of one mortgage industry group said a servicer may have completed required property repairs and paid HOA fees and utility bills, but if the property were subsequently vandalized, the servicer would have to delay conveyance to complete repairs. By that point, the servicer might no longer be current on HOA and utility payments. Servicers have the option to request an extension to the preconveyance time frame if they think they will be unable to convey a property in 30 days. Servicers requested a conveyance extension for about 40 percent of the properties conveyed from July 2010 through December 2017. FHA approved the extensions in about 40 percent of these cases. In addition, representatives from six of the 20 servicers we interviewed said FHA’s process for reviewing servicers’ overallowable requests (additional funds needed to complete work) negatively affected their ability to convey properties in 30 days. Once a servicer makes an overallowable request, FHA’s compliance contractor has 5 business days to review it and either reject the request or approve all or some of the requested amount. (We discuss the compliance contractor’s ability to meet this and other time requirements in the following section.) Servicers may appeal any rejections, in which case the compliance contractor has 3 business days to make a final determination. Six servicer representatives said that the time it takes the compliance contractor to make overallowable decisions may cause them to exceed the 30-day time frame, especially when they submit multiple requests for the same property. For context, our analysis of FHA data found that in 2017, the median number of servicer overallowable requests per property was 13, and the median number of appeals per property was six. In contrast to the preconveyance requirement, servicers usually met the time requirement for giving title evidence to FHA. Title evidence includes documentation that FHA is the legal owner of the property, including a copy of the mortgage documentation, a legal description of the property, and a copy of the recorded deed in FHA’s name. Servicers may provide title evidence to FHA at any point during the conveyance process up to 45 days after filing the deed. If servicers believe they will be unable to provide title evidence within 45 days, they may submit an extension request to the compliance contractor. According to FHA data, servicers were able to provide title evidence within 45 days plus any approved extension for 84 percent of properties conveyed from July 2010 through December 2017. FHA’s compliance and maintenance contractors generally met the required time frames for key conveyance tasks for properties conveyed from 2011 through 2017. However, when the contractors did not meet their required time frames, the delays may have lengthened the time to complete the conveyance process for some properties. Compliance contractor. FHA established a time frame of 5 business days for the compliance contractor to conduct various reviews in the pre- and postconveyance periods. In the preconveyance period, the compliance contractor reviews requests for overallowables, conveyance extensions, and conveyance of a property with surchargeable damage. The contractor also reviews title evidence and extension requests for title evidence, which are generally submitted after conveyance. Table 2 shows the percentage of properties conveyed from 2011 through 2017 for which the compliance contractor met the 5 business day requirement, according to our analysis of FHA data. Although the contractor mostly met the required time frames, when it did not, the delay may have lengthened the time to complete the conveyance process. Our analysis of FHA data indicates that when the compliance contractor missed the deadlines, it missed them by a median of 4–10 days, depending on the requirement. The compliance contractor’s review of overallowable requests, conveyance extension requests, and surchargeable damage requests generally occurs during the preconveyance period when servicers have 30 days to convey the property to FHA. As noted earlier, some servicer representatives we interviewed said that waiting for the compliance contractor to approve or deny overallowable requests hindered their ability to convey the property in 30 days. The compliance contractor must complete at least 95 percent of the reviews within the 5-day time frame to meet FHA’s standard for minimum acceptable performance. FHA uses monthly scorecards when reviewing the contractor’s performance against this standard. FHA officials told us they had not issued any deficiency notices to the current compliance contractor, but that discussions with the contractor can occur when it does not meet the 95 percent standard in particular months. FHA officials also noted that some of the contractor’s reviews may take longer than 5 days if resolving them requires obtaining additional documentation or substantial back-and-forth communications with the servicer. Maintenance contractors. After conveyance, FHA’s maintenance contractors have 2 calendar days from the date they are assigned a property to conduct the comprehensive property inspection and upload the results into a HUD Property Inspection Report in FHA’s asset disposition system. They then have 5 calendar days to complete a Property Condition Report, which details the functionality of the property’s systems, the existence of any transferable warranties, and any legal actions, such as code violations or pending demolition orders. FHA starts measuring compliance with these time requirements 24 hours after the properties are assigned to the compliance contractor (to account for holidays and late afternoon assignments). According to our analysis of FHA data, the maintenance contractors completed property inspections and uploaded the results within 3 days (the 2-day requirement plus 24 hours) for about 90 percent of properties conveyed from 2011 through 2017. The contractors met the 5-day requirement to complete the Property Condition Report about 77 percent of the time. When the maintenance contractors missed these time frames, they missed them by a median of 1 and 2 days, respectively. The longer a property remains uninspected after the servicer has conveyed it, the greater the chance that it will be damaged or vandalized before inspection. If a property is damaged during this period, disputes may arise between FHA and the servicer about which entity is responsible for the damage. FHA is responsible for maintaining the property once the servicer complies with all HUD regulatory requirements leading to conveyance, including filing the deed (in FHA’s name) for record and filing the conveyance claim. However, FHA may hold the servicer responsible for the damage if the claim was suspended due to the need for review or correction resulting from certain types of noncompliance with HUD requirements or if the servicer could not prove the damage occurred after FHA became responsible for maintaining the property. Disagreement over this issue can add time to the conveyance process. FHA measures each maintenance contractor’s performance monthly using a formula that considers both the contractor’s timeliness in completing property inspections and uploading the results (2-day requirement plus 24 hours) and in completing the Property Condition Report (7-day requirement plus 24 hours) for each property. If the contractor misses either deadline, it is not considered timely for that property. FHA considers timeliness for 95 percent of properties each month as satisfactory. According to FHA officials, FHA has taken actions when the performance of maintenance contractors was not satisfactory. A HOC official said that the actions may include issuing a defective performance letter, which requires the contractor to provide a remedy plan, and issuing a cure notice in coordination with HUD’s contracting office. FHA updated aspects of the conveyance process in recent years to help address some of the challenges experienced by servicers and the agency. For example, FHA increased property preservation and protection allowances in 2016 to help address servicer feedback and to better align allowances with other mortgage industry participants, according to FHA officials. In February 2016, FHA issued Mortgagee Letter 2016-02, which increased allowance amounts that servicers may claim for specific types of property preservation and protection work. It also increased the total maximum amount servicers may claim for a property without submitting an overallowable request from $2,500 to $5,000. However, the mortgagee letter eliminated all exclusions from the maximum amount, which previously included one-time major repairs, such as a roof replacement. FHA officials said the agency increased the allowance amounts to account for the standard increases in property preservation costs over time, and to align allowances with those of the enterprises and the Department of Veterans Affairs. Seventeen of the 20 servicers we interviewed said that FHA’s current property preservation and protection allowances are not sufficient to complete the work needed to convey properties. While representatives of eight of the 20 servicers told us the changes FHA made to allowances in 2016 helped them complete work within allowance amounts, representatives of the remaining 12 servicers said the changes did not help or helped in some ways but presented more challenges in other ways. Representatives of an association of mortgage lenders and servicers said that they preferred the previous system, because some work was excluded from the maximum allowance. For example, representatives of one servicer said that due to the 2016 changes, they now must submit an overallowable request for standard maintenance items, such as grass cuts, once they have exceeded the maximum allowance amount. Our analysis of FHA data found that the percentage of properties with at least one overallowable request increased steadily from 2011 through 2017—from about 53 percent to about 90 percent—despite the 2016 changes (see fig. 11). For properties with at least one overallowable request, the median number of requests before the 2016 changes (from July 1, 2010, through February 4, 2016) was seven, compared with eight after the changes (from February 5, 2016, through the end of 2017). In 2017 alone, the median number of overallowable requests per conveyed property was 13. However, it may be too early to tell what effect the 2016 mortgagee letter will have on servicer’s ability to conduct work within the allowances. FHA officials said that although the change in the allowance amounts was partly intended to reduce overallowable requests, the poor condition of many properties with extended default and foreclosure periods may have increased such requests. The officials stated that some of these properties were still being conveyed to FHA in 2017. FHA enhanced the information system servicers and contractors use to manage conveyed properties, but officials noted the need to update another system FHA uses to process and pay claims. In March 2018, FHA incorporated its preconveyance inspection pilot, discussed in more detail later in this report, into the asset disposition system, the information system servicers use to convey properties to FHA. FHA officials and contractors also use the system to track properties from conveyance through REO sale. With the update, servicers may request a preconveyance inspection and see the results of the inspection in the system, according to FHA officials. Before FHA added the pilot to the asset disposition system, servicers and FHA used email to communicate about properties in the pilot. In October 2016, FHA added a feature to the asset disposition system that enables FHA officials, contractors, and servicers to electronically monitor the status of reconveyed properties. According to FHA officials, all communication between FHA and servicers on reconveyed properties previously was by email, including the servicer’s notification to FHA that it was ready to convey a property again, and the photographs required to document property condition. However, according to FHA officials, FHA’s claims system is not equipped to process more than one claim per property, so claims for properties FHA reconveys and which the servicer then conveys to FHA a second time must be processed manually. Officials from FHA’s Office of Financial Services said that manual processing delays claim payments to servicers—sometimes by more than a year. Seven of the 20 servicers we interviewed identified delayed claim payments for reacquired properties as a challenge. FHA officials said that they have made an internal business case for funding to modernize the system, but have not succeeded in securing the funding in prior years. FHA updated its written direction to servicers on conveyance condition in 2016, but limitations in the contents and methods of communicating these policies and procedures have contributed to compliance challenges for some servicers. In its February 2016 mortgagee letter, FHA re- emphasized its existing directions to servicers about property conveyance, provided additional details on how to calculate claim amounts and document property preservation and protection work, and clarified descriptions of some preservation and protection requirements. Additionally, in December 2016, FHA issued an updated single-family housing policy handbook that consolidated all policies and procedures for servicers into one document, including those on maintaining and conveying foreclosed properties. However, servicers and other industry stakeholders with whom we spoke and our review of FHA’s policies and procedures on conveyance condition identified several limitations, as follows. Lack of clarity or specificity. Representatives from 15 of the 20 servicers we interviewed said they found FHA’s policies and procedures on conveyance condition to be unclear or subjective, and 13 cited specific parts of the conveyance condition standards they found to be unclear or missing. For example, one servicer was unsure about the extent of repairs required when a property had water seepage in the basement. We found that FHA’s policies and procedures include information on how to treat a basement that is flooded or a property with moisture damage, but does not address basement leaks, cracks, or seepage. Representatives of four servicers said that FHA’s policies and procedures do not sufficiently address how servicers should handle properties with potential structural or foundation damage. Consistent with this viewpoint, we found that FHA’s handbook and mortgagee letter do not explain what a servicer should do if it believes a property has damage affecting its structural integrity. In addition, representatives of three servicers said FHA’s expectations of them are unclear when a roof is damaged but does not currently have a leak. According to FHA’s policies and procedures, servicers must ensure all roofs “are free of active leaks or other sources of water intrusion.” However, FHA does not specify what servicers should do if there is roof damage but no active leak. Two of the servicers said they were uncertain whether they should replace the damaged roof that is not leaking, or convey the property and risk reconveyance if it rains before FHA inspects the property and the roof leaks. Perceived inconsistency in interpretation. Representatives from 10 of the 20 servicers we interviewed said FHA is somewhat or not at all consistent in determining whether properties meet FHA’s conveyance requirements. Among the remaining 10, one stated that FHA is completely consistent and nine said that FHA is mostly consistent. In addition, two of the 20 servicers said the answers they receive from FHA to the same question differ depending on whom they ask. HOC officials also noted cases in which their interpretation of policies and procedures differed from the compliance contractor’s. For example, officials from three HOCs told us that the compliance contractor sometimes disagrees with their determination that a property is not in conveyance condition when the contractor reviews the HOC’s reconveyance decision. Limited communication methods. In addition to formal written policies and procedures on conveyance condition, FHA fields servicer questions, primarily through its compliance contractor, by phone. The compliance contractor also issues an annual newsletter on topics such as common reconveyance triggers and best practices for submitting successful overallowable and extension requests. However, some servicers we interviewed suggested other possible ways to communicate policies and procedures that they said they would find helpful, including the following: Representatives of five servicers said they would like FHA to publish an authoritative set of frequently asked questions (FAQ) on conveyance condition. FHA has an FAQ web page that includes information on conveyance condition, but, as of April 2019, did not include FAQs about the specific property preservation and protection issues discussed above (water seepage, structural integrity, and roof damage with no active leaks). In addition, a link in the web page labeled “foreclosure/conveyance” led to a few FAQs on conveyance condition and property preservation requirements, but the answers consisted solely of language from FHA’s existing policies and procedures. One servicer’s representatives suggested that FHA could issue policies and procedures in a format similar to Fannie Mae’s Property Preservation Matrix and Reference Guide. This guide has features that FHA’s policies and procedures do not have, as discussed below, including photographic examples, detailed requirements for photographic documentation, and “if-then” statements detailing what servicers should do if they encounter certain challenges at a property. Representatives of two servicers suggested that FHA host regular industry calls. While the compliance contractor told us that it takes ad hoc calls and holds regular teleconference calls with a number of individual servicers, an FHA official told us the contractor is only authorized to respond to servicer questions by providing relevant parts of FHA’s written policies and procedures and is not supposed to respond with interpretations (clarifications, or explanations) of existing policies and procedures. Representatives from one servicer said industrywide calls with FHA staff would give servicers a way to obtain fuller explanations of FHA’s expectations. One servicer suggested that FHA provide training to servicers about the conveyance process. The servicer noted that while FHA provides training on other aspects of its program, including loss mitigation, it does not do so for the conveyance process or submitting claims. Limited direction on photographic evidence. FHA’s policies and procedures provide instructions for servicers and contractors on how to document property conditions, but contain limited direction on photographic evidence. Servicers must thoroughly document the condition of the property when they first obtain possession so that FHA does not hold them responsible for damage caused by the borrower. Servicers also must take before and after pictures of any work they do on the property. FHA’s policies and procedures on photographic documentation say only that the servicer must use digital photography, ensure a date-stamp is printed within each photograph, and ensure that each photograph is labeled to describe the contents of the photograph. FHA has not communicated in writing any requirements for photograph dimensions, color, distance, framing, or content or suggestions for documenting conditions that may be difficult to see. Servicers and FHA officials stated that they face challenges in documenting property conditions in a way that most accurately informs the compliance contractor about the property. The compliance contractor reviews documentation, including photographs, uploaded into the asset disposition system by servicers to make decisions on overallowable and extension requests. The compliance contractor also reviews documentation from maintenance contractors on inspection results and reconveyance recommendations by HOC officials. An FHA maintenance contractor told us that the compliance contractor sometimes responds that the condition described is not apparent from the photographs in the asset disposition system. According to members of an industry group representing servicers, in some cases this may result in FHA requiring servicers to repair damage caused by the borrower, because the servicers’ photographs did not prove the damage was present when they first gained possession of the property. To illustrate how photographs can effectively or ineffectively capture property condition problems, figure 12 provides two examples of flooring issues at properties conveyed to FHA. In one photograph, the buckling of the floor is apparent, but in the other, the waterlogged and warped condition of floor is harder to discern. An experienced FHA maintenance contractor told us there are creative ways to document some conditions that are difficult to photograph. For example, to document a damp floor, one can photograph a piece of paper (which darkens when wet) before and after placing it on the floor. This method is not included in FHA’s handbook or mortgagee letter. Limitations in the content and delivery of FHA’s policies and procedures on conveyance condition suggest room for improvement and are inconsistent with the federal internal control standard for communicating externally. This standard calls for management to externally communicate the necessary quality information to achieve an entity’s objectives. Federal agencies can help ensure compliance by communicating with and obtaining information from external parties and by periodically evaluating and selecting appropriate methods of communication, taking into account factors such as the audience, the purpose and type of information being given, and legal or regulatory requirements. However, FHA has not identified where the conveyance condition policies and procedures could be improved because it has not assessed information from servicers—for example, the frequency or content of their questions to the compliance contractor. FHA also has not thoroughly evaluated its methods for communicating its policies and procedures. As a result, FHA has limited assurance that servicers understand FHA’s expectations for conveyed properties and that contractor decisions are made consistently. Weaknesses in these areas can contribute to inefficiencies such as delays in executing conveyances and reconveyance of properties to servicers. FHA has not provided written direction to HOC officials on choosing among alternatives to address conveyed properties that do not meet FHA’s condition standards. According to officials from FHA headquarters and the National Servicing Center, HOC officials can (1) reconvey the property’s title to the servicer, (2) issue a demand letter establishing a debt to FHA for the cost of the work needed, or (3) enter into a reconveyance bypass agreement with the servicer that requires the servicer to complete repairs within a certain number of days. The latter two options avoid reconveyance and therefore may expedite resale of the property. These three options are mentioned in different parts of FHA’s policies and procedures, but the agency has not outlined the circumstances that would warrant use of each method. FHA has not provided direction to the HOCs, partly because HOC officials have the authority to choose a method based on the expected financial return on the property. However, HOC officials with whom we spoke differed in the factors that they considered when deciding how to address properties that do not meet FHA’s conveyance condition standards. Officials from three HOCs cited criteria that any property with more than $5,000 in damage due to servicer neglect should be considered for reconveyance, while a bypass agreement or demand letter may be issued if the amount of servicer neglect is less than $5,000. However, FHA officials were not able to tell us where this criterion is written. An official from the fourth HOC said the decision to reconvey partly depends on the strength of the housing market. If the HOC believes it can sell the property in its current condition—even if the condition does not meet FHA’s conveyance standards—the HOC will be more likely to issue the servicer a demand letter than reconvey the property. In contrast, an official from one of the other HOCs told us the state of the housing market did not factor into decisions on reconveyance. Furthermore, according to FHA officials, HOCs may also reconvey a property with only small amounts of damage if the servicer frequently conveys properties not in conveyance condition, in order to impress on the servicer the importance of complying with FHA requirements. The HOC officials generally agreed that bypass agreements offer a way for small repairs to be fixed quickly. However, an official from one HOC said the HOC did not issue bypass agreements often because servicers’ property preservation and protection vendors may take longer than the time specified in the agreement to complete repairs and, since the title is in FHA’s name, FHA has no recourse with the servicer. An official from another HOC also said that he did not like issuing bypass agreements because servicers do not always complete repairs quickly. FHA does not produce reports on the HOCs’ use of reconveyance, demand letters, and bypass agreements, so the frequency with which the HOCs employ these methods is unknown. Some servicer representatives with whom we spoke noted apparent inconsistency among the HOCs. For example, representatives of three servicers said that some HOCs do not issue bypass agreements at all. Similarly, representatives of one servicer told us they have infrequently, if ever, received a demand letter for small condition issues at properties; rather, FHA reconveys the properties for minor condition issues. FHA’s lack of written direction on alternatives to reconveyance is inconsistent with federal internal control standards, which call for designing control activities, including policies, to achieve objectives. Granting HOC officials discretion in dealing with properties that do not meet condition standards gives them flexibility to respond to specific circumstances. However, without written direction on factors to consider when determining whether they should reconvey a property, issue a demand letter, or enter into a bypass agreement with the servicer, FHA lacks reasonable assurance that HOCs make determinations consistently and in line with the agency’s regulatory goals for the REO program—to dispose of properties in a manner that expands home ownership, strengthens neighborhoods and communities, and ensures a maximum return to the mortgage insurance fund. Balancing these goals may require using different methods to address properties that do not meet conveyance standards. For example, in some cases issuing a demand letter or a bypass agreement for certain properties may result in FHA marketing and selling the property more quickly than it would by reconveying the property. A quicker sale, in turn, may help avoid the negative effects of a vacant property on the surrounding neighborhood. However, if FHA accepts a property in poor condition, it may receive less in proceeds when selling the property, which negatively affects FHA’s mortgage insurance fund. FHA began a pilot program in 2017 to inspect properties that meet certain criteria before conveyance, but has not developed a plan to assess the results of the pilot program. FHA selected three large servicers to participate in this preconveyance inspection pilot. These servicers may request preconveyance inspections for properties with characteristics that increase their chances of being reconveyed, according to FHA officials. For example, eligible properties include those that experienced recurring vandalism, received overallowable repairs of greater than $5,000, or have potential structural defects, foundation issues, or damp or wet basements. Based on the inspection results, the properties are approved to convey, approved to convey subject to repair with no additional inspection, or denied conveyance through the pilot (see table 3). After conveyance, FHA inspectors conduct a thorough inspection to confirm that the property meets conveyance condition standards. Properties that do not meet the standards may be reconveyed. As of November 2018, FHA had not developed plans for evaluating the effectiveness of the pilot in achieving the goals of reducing the number of properties reconveyed due to property condition and minimizing the time it takes to convey properties. FHA officials told us that they will develop a plan to assess pilot outcomes when sufficient data are available. However, without an evaluation plan, FHA may not collect the right information during the pilot to rigorously assess results. GAO’s guide for designing evaluations states that key components of an evaluation design include the evaluation questions or objectives; information sources and measures; data collection methods; an analysis plan, including evaluative criteria or comparisons; and an assessment of study limitations. Certain characteristics of FHA’s pilot underscore the importance of incorporating these components into evaluation design. For example, because the pilot is intended to expedite the conveyance process through preconveyance inspections, it will be important to isolate the impact of the inspections, potentially by making comparisons to a control group. A properly selected control group can rule out competing explanations for observed outcomes. Additionally, because the pilot may affect participating servicers in ways that extend beyond the speed of the conveyance process or the probability of reconveyance, it will be important for FHA to thoroughly consider the information sources and measures it uses, including participant feedback. For example, representatives of the three participating servicers told us they had concerns about FHA holding properties in the pilot to higher conveyance condition standards than nonpilot properties and the time it takes to complete the preconveyance inspection process. According to the representatives, this process, which includes 7 calendar days for the inspection and 5 business days for the HOCs to review the inspection report, has resulted in longer holding times and increased vandalism risks. Without a well-designed evaluation, FHA risks making decisions about preconveyance inspections based on incorrect or incomplete information on the pilot’s benefits and drawbacks. While FHA increased the use of other property disposition methods in recent years, servicers still convey thousands of foreclosed properties to FHA annually. If the process of transferring ownership from the servicer to FHA is not efficient, these properties may sit vacant for prolonged periods, deteriorate, and contribute to neighborhood decline. As a result, it is critical for FHA to have effective and efficient policies and procedures for the conveyance process. While FHA has made recent updates to its handbook, mortgagee letters, and information systems, additional improvements would better align its processes and procedures with federal internal control standards and GAO guidance on designing evaluations: By addressing limitations in the content (including its detail) and communication of its policies and procedures on conveyance condition, FHA could help reduce uncertainty and inconsistency in the conveyance process that may contribute to inefficiencies, such as reconveyance of properties to servicers. Second, by providing direction to HOC officials on factors to consider when deciding whether to use alternatives to reconveyance for properties that do not meet conveyance condition standards, FHA could increase the likelihood that alternatives will be used consistently and in line with FHA’s goals for the REO program. Third, by developing a plan for how it will evaluate the outcomes of the pilot to inspect certain properties prior to conveyance, FHA could help ensure the pilot generates the performance information needed to make effective management decisions about future policies. By addressing these issues, FHA could make the conveyance process more efficient and therefore help reduce negative impacts on neighborhoods. We are making the following three recommendations to FHA: The Commissioner of FHA should enhance the content and communication of FHA’s policies and procedures on conveyance condition, including by considering the program stakeholder views discussed in this report and other stakeholder input. (Recommendation 1) The Commissioner of FHA should provide written direction to HOC REO directors on factors to consider when determining whether to reconvey a property with condition issues, issue a demand letter, or enter into a bypass agreement with the servicer. (Recommendation 2) The Commissioner of FHA should develop a formal plan for evaluating the outcomes of the preconveyance inspection pilot that includes key elements of evaluation design—such as evaluation objectives and measures—and utilizes participant feedback and control groups, as appropriate. (Recommendation 3) We provided a draft of this report to FHA, the Department of Veterans Affairs, and the Federal Housing Finance Agency (the conservator and regulator of Fannie Mae and Freddie Mac) for their review and comment. The Department of Veterans Affairs and the Federal Housing Finance Agency did not provide comments. FHA provided written comments reproduced in appendix II. FHA neither agreed nor disagreed with our first recommendation to enhance the content and communication of its policies and procedures on conveyance condition. FHA cited the 2016 updates to its policy handbook and mortgagee letter and said it recognized the importance of external communication, training, and in- person meetings to ensure servicers have the information they need to operate in compliance with FHA programs. Our report discusses these updates, but also identifies areas for additional improvements to address limitations in the clarity and comprehensiveness of FHA’s policies and procedures and methods for communicating them. FHA agreed with our second and third recommendations to provide written direction on considering alternatives to reconveyance and to develop a plan for evaluating the preconveyance inspection pilot. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Housing and Urban Development, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or garciadiazd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Our objectives were to examine (1) timelines for Federal Housing Administration (FHA) foreclosed property conveyances in June 2010– December 2017 and the extent to which servicers and FHA met time requirements and (2) changes FHA has made to the conveyance process in recent years and any ongoing process challenges. To address the first objective, we obtained data from FHA’s Single Family Insurance System–Claims Subsystem and from the P260 Asset Disposition and Management System (asset disposition system) on the 610,802 foreclosed properties mortgage servicers conveyed to FHA from January 1, 2010, through December 31, 2017. For purposes of our analysis, we generally excluded properties conveyed to FHA from January 2010, through June 2010 because they were managed using different data systems and contractors than FHA currently uses. After excluding these properties, we analyzed data for 544,421 properties conveyed to FHA from July 2010 through December 2017. (We use calendar years in this report unless otherwise noted.) We calculated the number of days it took each property to complete the conveyance process. We defined the start of the conveyance process as the date the servicer obtained possession and acquired marketable title for a property and the end of the process as the date on which FHA assigned a marketing contractor to sell the property. For each annual cohort of conveyed properties, we calculated the 25th, 50th (median), and 75th percentile time frames and compared these statistics across years. To analyze the effect that reconveyances had on the length of the conveyance process, we compared length of time for conveyance in 2012–2017 for properties that were reconveyed to those that were not. According to FHA staff, data on reconveyances were unreliable prior to 2012, so we excluded those properties from this comparative analysis. We interviewed FHA officials about factors that may have affected conveyance time frames from 2010 through 2017, including increased use of other disposition methods and servicers delaying foreclosure actions and the resulting impact on property conditions, since the asset disposition system does not disclose the reasons for any delays. To analyze changes in the use of different property disposition methods and to examine the loss severity rates for these methods, we reviewed FHA data for fiscal years 2010–2017. To understand the relationship between properties with long default and foreclosure periods and conveyance time frames, we measured the time between the borrower defaulting on the mortgage and the servicer obtaining title to and possession of the property (effectively, the end of the foreclosure process) for properties conveyed to FHA from July 2010 through December 2017. We divided the range of default and foreclosure periods into four quartiles. For each quartile, we calculated the length of the conveyance process at the median and at the 25th and 75th percentiles. We then compared these statistics across quartiles. To determine the extent to which mortgage servicers and FHA contractors met their respective time requirements for the conveyance process, we identified relevant time requirements in Department of Housing and Urban Development (HUD) regulations and policies. We also reviewed the performance work statements for FHA’s mortgagee compliance manager (compliance contractor) and field service managers (maintenance contractors) to identify the contractors’ time requirements for the conveyance process. For servicers and contractors, we selected key time requirements for which electronic data were available, including the following: Thirty calendar days from acquiring title and possession of a property, plus the length of any approved time extension, to convey property to FHA. Forty-five days from conveying a property to FHA, plus the length of any approved time extension, to provide FHA with title evidence. Five business days to review each overallowable request submitted by a servicer. Five business days to review the sufficiency of title documentation submitted by the servicer. Five business days to determine whether a servicer can convey a property with surchargeable damage. Five business days to approve or deny a servicer’s conveyance or title extension request. Two calendar days, plus an additional 24 hours, to complete and upload the HUD Property Inspection Report from the date the property was assigned. Five calendar days to complete a Property Condition Report from the date the Property Inspection Report was completed. For each property, we calculated the number of days it took servicers and contractors to complete these required steps in the conveyance process and compared that number to the maximum number of days FHA allows for each step. For each annual cohort of properties conveyed in 2010– 2017, we calculated the 25th, 50th (median), and 75th percentile time frames for completing the steps. We also calculated the percentage of properties for which servicers or FHA contractors met their time requirements for each step. We reviewed FHA’s procedures for monitoring the performance of compliance and maintenance contractors for conveyed properties. We also reviewed examples of contractor quality control plans and FHA quality control reports and scorecards used to assess the contractors’ compliance with minimum time frames and other requirements. Additionally, we interviewed FHA officials about the contractors’ compliance with their respective time requirements and what steps FHA took, if any, to address any noncompliance. We assessed the reliability of data from the Single Family Insurance System–Claims Subsystem and the asset disposition system by reviewing FHA documentation about the data systems and data elements. We interviewed FHA staff and contractors knowledgeable about the data to discuss interpretations of data fields and trends we observed in our analysis. We also conducted electronic testing, including checks for outliers, missing data fields, and erroneous values. We excluded from each analysis properties with missing or erroneous information in the applicable data fields. We also excluded from each analysis properties for which the applicable data fields were five absolute deviations from the median (which we consider to be outliers). In addition, we excluded certain properties conveyed in calendar years 2010–2017 that had conveyance dates that were out of sequence. For example, we excluded properties for which the date a servicer obtained possession and good and marketable title occurred after the date the servicer conveyed the property to FHA. The number of properties we excluded in any analysis using these methods represents no more than 3.2 percent of properties conveyed from July 2010 through December 2017, which we consider to be insignificant when compared to the remaining properties included in the analysis. After taking these steps, we believe that the data were sufficiently reliable for purposes of characterizing the overall length of FHA property conveyances and compliance with key time requirements. To determine what changes FHA made to the conveyance process in recent years, we reviewed relevant FHA regulations, policies, and procedures issued in 2010 or later, including FHA’s February 2016 mortgagee letter (a written instruction to FHA-approved lenders) on conveyances. We compared the requirements and property preservation and protection allowance amounts in the mortgagee letter to those in the prior mortgagee letter. We also reviewed FHA documentation on changes to the asset disposition system, FHA’s data system for conveyed properties, and on FHA’s preconveyance inspection pilot program that began in 2017. We interviewed FHA officials on the reasons for the recent changes and on the extent to which they reviewed any analogous requirements and property preservation and protection allowances of other mortgage entities (including Fannie Mae, Freddie Mac, and the Department of Veterans Affairs) when making the updates. To supplement our review of FHA’s recent changes to property preservation and protection allowances, we used the asset disposition system data to analyze changes in the frequency and number of servicer overallowable requests since the 2016 mortgagee letter went into effect. To examine what, if any, challenges exist with the conveyance process, we randomly selected a nongeneralizable sample of 20 large- and medium-sized, bank and nonbank servicers of FHA-insured mortgages. We defined large-sized servicers as those with 100,000 or more active FHA-insured mortgages as of December 31, 2017, and medium-sized servicers as those with 10,000–99,999 active FHA-insured mortgages as of that date. These servicers accounted for more than one-third of active FHA-insured mortgages as of December 31, 2017. We conducted semistructured interviews with the servicers about their experience with FHA property conveyances, including the sufficiency of FHA’s policies and procedures, time lines, and allowance amounts and any challenges they experienced with the process. We also discussed the extent to which the 2016 mortgagee letter assisted or hindered their conveyance efforts. In addition, we spoke with two national industry groups representing mortgage servicers about recent changes and any challenges their members experienced with the conveyance process. We reviewed FHA’s requirements for servicers and contractors on conveyed properties. In cases in which servicers stated that FHA’s policies and procedures on particular conveyance requirements was insufficient or unclear, we examined the 2016 mortgagee letter, HUD’s single-family housing policy handbook, and frequently asked questions on HUD’s website—to determine whether it addressed the topics and was sufficiently thorough to be applied to properties with different circumstances. We assessed whether the policies and procedures were consistent with federal internal control standards for external communication. In particular, we examined whether the policies and procedures communicated necessary quality information to achieve program objectives and whether FHA had evaluated appropriate methods to communicate them. Where applicable, we also compared FHA’s policies and procedures to features of Fannie Mae’s guide for servicers on how to preserve and protect vacant properties. We also assessed FHA’s policies and procedures on reconveyances and alternatives to reconveyance against federal internal control standards for designing control activities. To review the preconveyance inspection pilot that FHA began in 2017 and any challenges with the pilot, we interviewed the three participating servicers about FHA’s implementation of the pilot and the extent to which preconveyance inspections reduced the likelihood of reconveyance or addressed other challenges. We spoke with FHA National Servicing Center officials about their monitoring of pilot outcomes and their plans for assessing results. We assessed FHA’s planning and evaluation efforts against key components of evaluation design from GAO’s guide for designing evaluations. Furthermore, we interviewed a number of individuals and entities about challenges they experienced in implementing their property conveyance responsibilities, the sufficiency of FHA’s policies and procedures, and methods for assessing contractor performance. These included FHA headquarters and National Servicing Center officials with responsibilities for aspects of the conveyance process; FHA’s compliance contractor; and Real Estate-Owned Division officials, the largest maintenance contractor, and staff responsible for overseeing the maintenance contractors at each of FHA’s four homeownership centers. Finally, we visited eight recently conveyed or reconveyed properties in the Baltimore, Maryland, and Atlanta, Georgia, areas to observe property conditions, learn about the maintenance contractors’ property inspection processes, and understand challenges in documenting and addressing condition issues. We chose these locations to provide some geographic dispersion and coverage of different FHA homeownership centers. The properties were selected by Philadelphia and Atlanta homeownership center staff based on our request to visit a mix of recently conveyed and reconveyed properties in metropolitan areas and time periods that we chose. As a result, the conditions we observed are illustrative rather than representative of all conveyed properties. We conducted this performance audit from September 2017 to June 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Steve Westley (Assistant Director); Melissa Kornblau (Analyst in Charge); Rachel Batkins; William Chatlos; Emily Flores; John McGrail; Samuel Portnow; Barbara Roesmann; Tovah Rom; and Jena Sinkfield made key contributions to this report.", "summary": "FHA insures hundreds of thousands of single-family home mortgages annually. When an FHA borrower defaults, the mortgage servicer in many cases forecloses, obtains title to the property, and conveys ownership to FHA. FHA inspects the property, acquires it if it complies with condition standards and title requirements, and lists the property for sale. FHA may reconvey noncompliant properties to servicers. During conveyance, homes may sit vacant for months and can deteriorate, contributing to neighborhood blight. Senate Report 114-243 included a provision for GAO to review FHA's effectiveness and efficiency in reaching determinations of conveyable condition. This report discusses (1) timelines for FHA property conveyances in 2010–2017 and whether servicers and FHA met time requirements, and (2) changes FHA has made to the conveyance process in recent years and any ongoing process challenges. GAO analyzed FHA data on properties conveyed in 2010–2017, reviewed FHA's policies and procedures, and interviewed 20 randomly selected mortgage servicers accounting for more than one-third of active FHA mortgages. From July 2010 through December 2017, the process for conveying foreclosed properties to the Federal Housing Administration (FHA) took a median of 70 days. The conveyance process—which GAO measured from a mortgage servicer's obtaining title to and possession of the property to FHA's marketing of the property—involves servicers making repairs, transferring ownership, and filing a mortgage insurance claim, and FHA inspecting the property. FHA attributes the length of time to complete the process partly to foreclosure processing delays that left properties vulnerable to damage and vandalism, which can increase the time servicers need to bring properties into conveyance condition. Property damage also may increase the likelihood that FHA will reconvey a property (transfer it to the servicer) for not complying with condition standards, further extending the conveyance process. For about 55 percent of properties conveyed in July 2010–December 2017, servicers exceeded the required time to obtain title and possession of a foreclosed property and convey it to FHA. For 2017 alone, the corresponding figure was 72 percent. As a result, servicers were not eligible to be reimbursed for all repairs and interest expenses for those properties when filing insurance claims with FHA. In recent years, FHA changed aspects of its conveyance process to help address some of the execution challenges the agency and servicers have faced. For example, in 2016, FHA enhanced its data system for conveyed properties to reduce manual administrative processing. FHA also began a pilot program in 2017 to decrease the number of properties FHA reconveys by inspecting properties before conveyance. However, GAO found shortcomings in FHA policies, procedures, and assessment efforts that are inconsistent with federal evaluation criteria and internal control standards, as follows: FHA's policies and procedures lack detail that could help servicers and contractors determine if a property is in compliance, and the agency has not examined alternative methods of communicating this information. Fifteen of the 20 servicers GAO interviewed said existing policies, procedures, and communications often were not clear or specific enough to address property conditions or repair decisions they encountered. FHA also relies on brief written policies to explain standards and makes limited or no use of other methods, such as photographs or industry-wide calls. FHA has not provided written direction on when to use alternatives to reconveyance—such as agreements under which servicers make repairs or repay FHA for any repair costs after conveyance—for properties not meeting condition standards. In the absence of such direction, FHA may not be addressing these properties in the most consistent or effective manner. FHA has not developed a plan to assess the outcome of its inspection pilot. Without rigorous assessment, FHA risks making decisions about the future of the pilot based on inaccurate or incomplete information. Addressing these shortcomings could help improve the efficiency and effectiveness of FHA's property conveyance process. GAO recommends that FHA (1) enhance the content and communication of policies and procedures on conveyance condition, (2) provide written direction on alternatives to reconveyance, and (3) develop a plan to assess a pilot program. FHA agreed with the second and third recommendations and did not agree or disagree with the first.", "document_type": "gao"}
{"report": "Congress provided DOD the authority to use other transactions in the late 1980s and has expanded the authority over time. In 1989, Congress provided the Defense Advanced Research Projects Agency the authority to temporarily use other transactions for research projects. These transactions were intended to spur research and development that would benefit both commercial companies and the government. In 1991, Congress allowed the military departments to use the authority as well and made the authority permanent. In 1993, Congress provided the Defense Advanced Research Projects Agency the authority to award other transactions for prototype projects. Congress expanded the authority to the military departments and other defense agencies in 1996. In 2001, Congress allowed DOD to provide for follow-on production in prototype other transactions. Further, DOD could award follow-on production other transactions, without using competitive procedures, to the participants of a successfully completed, competitively awarded prototype project, provided several conditions were met. Congress codified DOD’s other transaction authority for prototype and follow-on production other transactions at section 2371b of title 10, U.S. Code, in 2015. Congress did not define a prototype project in statute. DOD’s November 2018 Other Transactions Guide, however, defined a prototype project as addressing a proof of concept, model, novel application of commercial technologies for defense purposes, or a process including a business process, among other types. Under section 2371b, DOD can use other transactions for prototype projects that are directly relevant to enhancing the mission effectiveness of military personnel and the supporting platforms, systems, components, or materials proposed to be acquired or developed by DOD, or to improve those in use by the armed forces. In addition, Congress has required DOD to meet at least one of the following four conditions to use a prototype other transaction: 1. There is at least one non-traditional defense contractor or non-profit research institution participating to a significant extent in the prototype project. 2. All significant non-government participants in the transaction are small businesses or non-traditional defense contractors. 3. At least one-third of the total cost of the prototype project is to be paid out of funds provided by sources other than the federal government. 4. The senior procurement executive determines in writing that exceptional circumstances justify the use of a transaction that provides for innovative business arrangements or structures that would not be feasible or appropriate under a contract, or would provide an opportunity to expand the defense supply base in a manner that would not be practical or feasible under a contract. Section 2371b of title 10, U.S. Code, does not limit DOD to awarding prototype other transactions to non-traditional companies. DOD could award traditional defense contractors a prototype other transaction under the first, third, and fourth conditions listed above. It could also award prototype other transactions to consortiums, which may be comprised of non-traditional companies, traditional defense contractors, and others such as non-profit research institutions. These consortiums may be administratively managed by a single firm. Consortium management firms in general provide administrative support to consortium members, such as distributing requests for proposals, holding proposal writing workshops, negotiating the general terms and conditions of prototype projects with consortium members, and making payments to consortium members. For example, Advanced Technology International, a consortium management firm, reported that it represented 298 members in the Countering Weapons of Mass Destruction consortium as of September 2019, according to its website. The website also states that 87 percent of the consortium’s members were non-traditional companies. Contracting offices generally designate a subset of their contracting officers to award other transactions, including prototype other transactions. In this capacity, these individuals are referred to as agreements officers. According to senior contracting officials at offices we included in this review, agreements officers are typically more experienced contracting officers that have demonstrated the ability to exercise business acumen and judgement in a less structured contracting environment and have a strong working knowledge of intellectual property. All of the contracting offices we included in this review required agreements officers to complete training courses offered by their office or the Defense Acquisition University related to the award of other transactions. The Director, Defense Procurement and Acquisition Policy, issued an Other Transaction Guide for Prototype Projects in January 2017 that included general information about planning, evaluating, and awarding prototype other transactions. The Office of the Under Secretary of Defense for Acquisition and Sustainment issued updated guidance in November 2018 that covered all types of other transactions, including case studies and lessons learned to help agreements officers when awarding other transactions. For example, the November 2018 guide states the following: An agreements officer should consider whether a company is supplying a new key technology, providing a material increase in the performance of a product, or making some other contribution when determining if a non-traditional company or non-profit research institution will be participating to a significant extent. DOD components should not establish predetermined percentages of total costs or labor hours to determine significant participation. As such, agreements officers can use their own discretion when using cost and labor hour information to determine if a non-traditional company is playing a significant role on a prototype other transaction. The Competition in Contracting Act does not apply to other transactions, but competition and fairness are still important considerations and agencies may determine how competition will be structured. Other transactions may take longer to award than FAR-based contracts due to factors such as drafting and negotiating all the terms and conditions in an other transaction. Fiscal law requirements are applicable to other transactions and the decision to use an other transaction does not expand or restrict available appropriations. Therefore, multiple funding types, including research, development, test, and evaluation; procurement; and operations and maintenance appropriations may be appropriate depending on the intent and stage of the prototype. Modifications of ongoing transaction projects are fairly common and other transactions should address how changes will be handled. DOD significantly increased its use of prototype other transactions from fiscal years 2016 through 2018, both in terms of the number of prototype other transactions awarded and the amount obligated on prototype other transactions. Most prototype other transactions involved at least one non- traditional company that was participating to a significant extent. About 71 percent of the obligations were awarded to three consortiums and two traditional defense contractors. DOD is currently preparing a report to Congress on its use of the other transaction authority and working to address certain limitations in its data collection efforts, including improving data related to consortiums. FPDS-NG data showed that DOD obligated a total of $7.2 billion on prototype other transactions from fiscal years 2016 through 2018. The total number of new prototype other transactions increased five-fold from 34 to 173 during this time frame. According to a Defense Pricing and Contracting official, DOD is encouraging the use of these transactions as a way to acquire innovative technology from non-traditional companies that it could not typically access. There were also modifications and orders related to these prototype other transactions and those awarded in prior years that resulted in a change in obligations, such as providing funding to members of consortiums to carry out new projects. As discussed in more detail later in the report, FPDS-NG did not identify the number of projects carried out by consortiums. Overall, obligations made on prototype other transactions nearly tripled from $1.4 billion to $3.7 billion (see fig. 1). The Army, Defense Advanced Research Projects Agency, and the Air Force accounted for 97 percent of all new awards and actions that resulted in a change in obligations from fiscal years 2016 through 2018. They also accounted for 97 percent of the total amount obligated on these new awards and actions (see table 1). Appendix I shows more detailed information. The Army was responsible for over two-thirds of the new awards and actions made from fiscal years 2016 through 2018—valued at nearly $5.3 billion—but some of these were awarded on behalf of other DOD components, such as the Air Force, Navy, and Defense Innovation Unit. Officials from the Air Force Research Laboratory and Navy’s Office of Naval Research told us that they relied on the Army to award prototype other transactions on their behalf because, in some cases, the Army had previously awarded a transaction, such as to a consortium, which they could leverage to meet their own components’ needs. The Army awarded prototype other transactions on behalf of the Defense Innovation Unit, as it did not have the authority to award prototype other transactions, until November 2018. DOD reported that at least one non-traditional company or non-profit research institution participated to a significant extent—one of four statutory conditions that Congress established for the appropriate use of a prototype other transaction—in 88 percent of the 1,250 new awards and actions made from fiscal years 2016 through 2018 (see fig. 2). We found that from fiscal years 2016 through 2018, the top five recipients by obligations were either consortiums or traditional defense contractors. Awards to these five recipients accounted for $5.1 billion or 71 percent of the obligations on new awards and actions during this time frame (see table 2). Three of the top five recipients were consortium management firms— Advanced Technology International, Consortium Management Group, and National Center for Manufacturing Sciences. In general, a consortium management firm does not complete the prototype, but rather helps manage consortium members. The other two companies among the top five were traditional defense contractors—Lockheed Martin and Aerojet Rocketdyne. As stated earlier, according to statute, traditional defense contractors can be awarded prototype other transactions under three possible conditions: by partnering with at least one non-traditional defense contractor or non- profit research institution participating to a significant extent, paying at least one-third of the total project cost, or having the government agency’s senior procurement executive determine in writing that exceptional circumstances justify the use of a prototype other transaction. Paying one-third of the project’s costs is an example of cost-sharing. For the eight new prototype other transactions these two companies were awarded from fiscal years 2016 through 2018, four involved non- traditional companies or non-profit research institutions that participated to a significant extent and the remaining four involved cost-sharing arrangements. In the one prototype other transaction awarded to Lockheed Martin that we reviewed, the Army entered into the transaction, currently valued at $17.5 million, to prototype two removable sensors for unmanned aircraft. According to the Army agreements officer, Lockheed Martin was awarded an other transaction instead of a FAR-based contract because Lockheed Martin needed to collaborate with four other companies that were awarded prototype other transactions. The Army agreements officer told us he concluded that it would have been difficult for all the contractors to collaborate if some were operating under prototype other transactions and Lockheed Martin was subject to the requirements of a FAR-based contract. Since Lockheed Martin did not have a non-traditional company participating to a significant extent on the prototype other transaction it was awarded, the company was required to pay at least one-third of the cost of the project to comply with statutory requirements. Lockheed Martin used a combination of in-kind contributions, such as test articles, and independent research and development funds for its share of total project costs. In response to congressional direction, DOD expects to submit a report in November 2019 on its use of the prototype other transaction authority in fiscal year 2018. This report will include, among other elements, data on new prototype other transactions awarded in fiscal year 2018; actions made in fiscal year 2018 on these prototype other transactions and ones awarded in prior fiscal years; detailed information on the DOD organizations using the authority; the purpose and status of projects; and those prototype other transactions that led to a follow-on production other transaction. This report, which was originally due to be delivered in December 2018 was delayed, according to DOD, as FPDS-NG was not configured to capture all the data needed to prepare the report. DOD’s Defense Pricing and Contracting is collecting the required data directly from DOD components. DOD and military component officials whom we interviewed acknowledged limitations in the FPDS-NG data on prototype other transactions. DOD officials stated they have addressed some of these limitations and officials are discussing how to improve the information collected in the future. For example, as noted above, we found four other transaction awards for the production of products and four procurements for experimental purposes identified as prototype other transactions in FPDS-NG. According to Defense Pricing and Contracting officials, until June 2019, DOD did not have the ability to differentiate between prototype and production other transactions in FPDS-NG; therefore, both prototype and production other transactions were reported as prototype other transactions. The General Services Administration—the organization that is responsible for managing and updating FPDS-NG— added an option in FPDS-NG that would allow users to identify other transactions as either for a prototype or production, as appropriate, beginning in June 2019. DOD officials stated that they are discussing the best approach for consistently identifying procurements for experimental purposes in FPDS-NG. This could include adding an option to FPDS-NG for users to identify these procurements or including unique letters in the award number. DOD officials are also working to address FPDS-NG data limitations related to consortiums that reduce DOD’s management insight on the use and award of prototype other transactions. Army contracting officials noted that FPDS-NG tracks information about the base prototype other transaction that is awarded to a consortium, but does not track data about each project conducted through the consortium, such as whether a non- traditional company is participating on each project. The Army Deputy Assistant Secretary for Procurement issued a policy, effective October 1, 2019, that changes how the Army reports other transactions into FPDS- NG to improve data on projects conducted by consortium members. The policy, however, does not discuss how it will track non-traditional company participation. Further, FPDS-NG does not track the extent of competition among consortium members. DOD officials stated that, while FPDS-NG data shows DOD competitively awarded 48 percent of all prototype other transaction obligations for fiscal years 2016 through 2018, they believed this figure understates the degree of competition actually achieved. These issues are illustrated in the following examples. FPDS-NG shows that Advanced Technology International was awarded a prototype other transaction in fiscal year 2018 with a ceiling of $10 billion for the Countering Weapons of Mass Destruction Consortium. FPDS-NG also shows that, as of March 2019, the Army had obligated $116 million as modifications under the base transaction, and, according to the Army Contracting Command-New Jersey agreements officer, were for consortium members to carry out various prototype projects. FPDS-NG did not identify the number of projects that are being carried out by the consortium or which consortium members were participating on the projects. The Army Contracting Command-New Jersey agreements officer that awarded this prototype other transaction, however, maintained her own records to help her manage and oversee the consortium’s efforts. According to this agreements officer, as of March 2019, all 44 projects carried out by consortium members involved non-traditional companies—37 prototype projects were carried out by non-traditional companies that served as prime contractors and seven prototype projects were carried out by traditional defense contractors with subcontractors that were non-traditional companies that participated to a significant extent. FPDS-NG shows that the base contract, as well as all the modifications, for a different prototype other transaction the Army Contracting Command-New Jersey awarded to Advanced Technology International was non-competitively awarded. These modifications accounted for 69 percent or $2.6 billion of the non-competitive obligations DOD made through new awards and actions from fiscal years 2016 through 2018. However, according to command contracting officials, during this time frame, all the obligations on modifications were associated with projects that were competitively awarded among consortium members. DOD senior contracting officials stated that FPDS-NG tracks only whether the base transaction was competitively awarded and that modifications made to transactions awarded to consortiums automatically retain the same competitive or non-competitive designation as the base contract. Agreements officers used multiple methods to determine whether and to what extent non-traditional companies were participating on the prototype other transactions we reviewed. We found that agreements officers first determined whether a company was traditional or non-traditional by reviewing government databases and consulting with subject matter experts, among other approaches; and then determined the extent to which a non-traditional company was expected to participate on a prototype other transaction. In accordance with DOD’s November 2018 Other Transactions Guide, agreements officers determined whether non-traditional companies participated on nine of the 11 transactions in our non-generalizable sample that met this statutory condition. For the other two transactions, one involved a cost sharing arrangement between the Army and a traditional defense company; therefore, the agreements officer did not have to make this determination. The other instance involved the award of an other transaction to a consortium. In this instance, the consortium and agreements officer set out to negotiate general terms and conditions that would flow down to subsequent prototype projects carried out by consortium members. The agreements officer plans to make the determination about whether a non-traditional company is participating or meeting another statutory condition on a case-by-case basis for each subsequent prototype project that is funded. Agreements officers typically used more than one method to determine if a company was a non-traditional company for the nine transactions in our sample. For example, agreements officers considered, in varying combinations, a contractor’s assertion, data from government information systems, subject matter expert input, or market research when making the determination (see table 3). As reflected in table 3, in none of the cases we reviewed did an agreements officer rely solely on the contractor’s assertion that a company was a non-traditional company. The following two examples illustrate the type of actions agreements officers took to determine that a contractor was a non-traditional company: In a $19.3 million prototype other transaction awarded by Washington Headquarters Services for a large autonomous ship, the prime contractor stated that it and a subcontractor were non-traditional companies. This was because neither had performed work on a DOD contract or subcontract that was subject to full cost accounting standards in the preceding year, which is one of the statutory criteria to be considered a non-traditional company. To confirm that the prime contractor and the subcontractor were non-traditional companies, the agreements officer checked the System for Award Management, leveraged market research, and relied on input from technical officials from the Navy’s Surface Warfare Directorate and Unmanned Maritime Systems Program Office with industry knowledge about contractors. In a $10 million Army prototype other transaction for an artificial intelligence war-gaming capability, the contractor that was to perform all the work stated that it met the statutory definition of a non- traditional company. To verify its status, the agreements officer determined that the contractor did not have a record in the System for Award Management, which would ordinarily be required if the company had previously done business with the federal government. The agreements officer also conducted market research to verify that the company was not a DOD subcontractor that was subject to cost accounting standards in the preceding year. After determining whether a company was a non-traditional company, agreements officers then used various methods to determine whether one or more non-traditional companies would play a significant role on the nine prototype other transactions before they were awarded. These methods included assessing whether the contractor was performing all the work on the prototype, evaluating whether the services or technologies provided by the non-traditional companies were critical, using input from subject matter experts, or considering the percentage of total costs or labor hours performed by the contractor (see table 4). As shown in table 4, the proportion of award values received by non- traditional companies on the prototype other transactions, which ranged from 16 to 100 percent, did not always indicate the significance of the non-traditional companies’ contributions. Consistent with DOD guidance, agreements officers took various factors into account when determining whether a non-traditional company is participating to a significant extent. In the transaction in which the non-traditional company was expected to receive about 16 percent of the total award value, the agreements officer considered the engineering work performed by a non-traditional company—which was a subcontractor on the effort—to be critical to developing the data port for a robotic satellite servicing vehicle. In another example, the agreements officer and Navy subject matter experts determined a non-traditional company that would receive less than 25 percent of a transaction’s overall award value was participating to a significant extent since it was providing the vessel and crew necessary to execute testing of the large autonomous ship that was being prototyped. Agreements officers followed their commands’ established review processes, which involved higher level reviews by senior officials and legal reviews, in nine of the 11 transactions in our sample. Agreements officers did not obtain higher level reviews in the two remaining transactions, but senior contracting officials plan to take action to address the issues we identified. Award times for these transactions ranged from 45 to 370 days. Each of the DOD contracting offices we assessed established policies for reviewing prototype other transactions before award, though the processes differed. For example, the contracting offices we evaluated generally required other transactions to be reviewed by an official at least one level above the agreements officer and to be subject to a legal review. Some contracting offices required additional reviews at higher dollar thresholds. In addition, the officials responsible for reviewing transactions within a contracting office sometimes differed based on the expected dollar value of the transaction. For example, at the Army Contracting Command-New Jersey, the Branch Chief can review only transactions valued at less than $10 million. Transactions exceeding that amount would be reviewed by a higher ranking official, such as the Center Director. According to senior contracting officials, the review process is intended to ensure that prototype other transactions meet the statutory requirements for use of the authority before award. The review process also facilitates component efforts to obtain the “best deal” for the government based on decisions by the agreements officers, senior contracting and program officials, and legal advisors about factors such as whether to compete the transaction and whether to obtain technical data rights for the prototype. Table 5 provides more specific information on the review process required by the Air Force Research Laboratory, Army Contracting Command-New Jersey, Defense Advanced Research Projects Agency, and Washington Headquarters Services. Agreements officers in our review documented aspects of their decision making prior to awarding a transaction. For example, agreements officers generally documented the condition under section 2371b of title 10, U.S. Code, that was met to use a prototype other transaction, and some documented the negotiation process with the commercial companies regarding terms and conditions of the transactions. We also found that the Air Force Research Laboratory, Defense Advanced Research Projects Agency, and Washington Headquarters Services required agreements officers to document acquisition planning for prototype projects to some extent. Senior contracting officials from these organizations stated that acquisition planning helps programs manage risks and provides direction to agreements officers who are new to awarding other transactions. For an Air Force other transaction we reviewed, documentation included the purpose and objectives for the prototyping project, the anticipated cost and type of funding needed for the project, and the transaction award schedule. We found the Army agreements officer who awarded the prototype other transaction to the Countering Weapons of Mass Destruction Consortium developed an acquisition planning document, even though policy did not require her to do so. The agreements officer said she did this because the transaction had a $10 billion ceiling and she considered this a best practice. Legal counsel, several senior level contracting officials, and program officials reviewed the acquisition planning document before the agreements officer awarded the transaction. Defense Advanced Research Projects Agency also required agreements officers to document the reason for non-competitive awards in a memorandum. We previously reported that competition promotes the efficient use of taxpayer resources and establishes accountability for results by helping to drive down prices and motivate better contractor performance. We found that an Air Force Research Laboratory agreements officer also documented the reasons why a $1.2 million prototype other transaction to develop a manufacturing process to reduce a missile engine’s production costs was awarded non-competitively, even though policy did not require such documentation. The agreements officer said the reason for the non-competitive award was that a non-traditional company is the manufacturer of the engine and has the experience to create cost-saving innovations to its manufacturing process. The agreements officer stated legal counsel reviewed this documentation prior to awarding the other transaction and that management was aware of the non-competitive status of this transaction during acquisition planning. Contracting officials from the Army Contracting Command-New Jersey stated that, consistent with statute, they did not require acquisition planning or non-competitive award documentation because they wanted to maintain few requirements to award prototype other transactions. While senior contracting officials told us that they were able to streamline the review process for other transactions compared to the actions typically required before awarding other procurement contracts, they cautioned that the time needed to award a prototype other transaction can vary significantly. We found that, for the 11 prototype other transactions we reviewed, award times—which we defined as the time a contracting office released a solicitation until the time the government awarded the other transaction—ranged from 45 to 370 days. By way of reference, we recently reported that the time from solicitation issuance until the time the government awarded 129 weapon systems-related procurement contracts ranged from less than a month to over 4 years, with a median of about 9 months. For the 11 prototype other transactions we reviewed, contracting officials noted that the times varied due to factors such as prior knowledge about the contractor and the complexity of the prototype project. An Air Force prototype other transaction for improving missile engine manufacturing processes was awarded in 45 days because the agreements officer said she had extensive knowledge about the capabilities of the contractor prior to awarding this transaction and was, therefore, able to plan for and develop other transaction documentation early. The agreements officer told us that she had knowledge of this company because it had previously been a subcontractor on a technology demonstration. Conversely, the Army took 370 days to award a prototype other transaction because the government needed time to assess what contracting instrument to use to ensure multiple contractors collaborated to build an autonomous airborne network of sensors. According to the agreements officer, the government needed time to research the effects of several possible teaming arrangements, including creating a new consortium, using an existing consortium, awarding a FAR-based contract, placing all of the contractors on a single other transaction, or awarding individual other transactions to each contractor. In nine of the 11 prototype other transactions we reviewed, agreements officers followed their contracting offices’ policies to have prototype other transactions reviewed before awarding the other transactions. In a $4.6 million Army prototype other transaction to develop a capability to modernize legacy hardware systems, the appropriate senior-level contracting official reviewed the transaction, such as by checking terms and conditions and ensuring that the contract file was complete. Legal counsel also reviewed the transaction prior to award, as required. In a $6.1 million Defense Advanced Research Projects Agency prototype other transaction, the agreements officer consulted the Deputy Director about this transaction before negotiations and a contracting official one level above the agreements officer reviewed the transaction, as the agency’s policy required. The agreements officer also consulted legal counsel, an optional policy action, to draft and negotiate a clause that would waive specific topics of the commercial rights license that did not apply to the government. The agreements officer stated that by working with legal counsel to develop a clause, he was able to meet the program’s objective to prototype a capability to emulate and validate microchip designs and accommodate the contractor’s desire to use its commercial license. For the remaining two prototype other transactions, we found, and contracting officials agreed, that agreements officers did not meet policy requirements for obtaining higher level review before award. In the first case, a Defense Advanced Research Projects Agency agreements officer did not have a higher level official review a $7.8 million prototype other transaction before it was awarded. Agency policy required the agreements officer to consult with the Deputy Director of the Contract Management Office about the negotiation strategy and discuss any issues that arose during negotiations. In addition, policy required the agreements officer to have a contracting official one level above the agreements officer review the prototype other transaction before award, regardless of dollar value. The agreements officer told us that he did not consult with the Deputy Director or obtain the required review because he thought the terms and conditions were straightforward and the dollar value was too low to require a review by an official above him. Senior-level Defense Advanced Research Projects Agency contracting officials told us they were not aware that this prototype other transaction was awarded without the required consultation and review until we brought this to their attention. According to these officials, internal controls were in place that should have prevented the award of prototype other transactions without the required consultation and review—such as making current policy documents readily accessible to agreements officers, communicating policy changes to agreements officers, and using a data system to track the development of other transactions prior to award. These officials stated that they plan to check compliance with the required pre-award consultation and review during their next internal file review of awarded other transactions that will be completed in fiscal year 2020. If they find instances of noncompliance, officials stated that the agency will take corrective actions, such as providing additional training for agreements officers. They also stated that they subsequently reviewed the $7.8 million prototype other transaction and determined that no changes needed to be made to the other transaction in this instance. In the second case, the Army Contracting Command-New Jersey agreements officer—who was also the Center Director—had his Branch Chief review a $10 million prototype other transaction prior to award. Command policy, however, generally requires the Center Director to review prototype other transactions valued at or greater than $10 million. The Center Director stated that he did not serve as the reviewer on this transaction because he would have been reviewing his own work. Army Contracting Command-New Jersey officials stated that the management review should have been conducted by another Center Director or a higher contracting official, but noted that this was an atypical situation not addressed in the command’s policies. As such, Army Contracting Command-New Jersey officials plan to revise their management review policy by fall 2019 to address who should be responsible for conducting a higher level management review when someone who is designated to conduct the management review serves as the agreements officer for the transaction. The officials stated that they reexamined the prototype other transaction and found that there were no issues with the terms and conditions and, therefore, no changes needed to be made to the transaction. Based on the stated intent of Defense Advanced Research Projects Agency and Army Contracting Command-New Jersey contracting officials to address issues we identified in our review, we are not making recommendations at this time but will monitor their actions to address the issues. We provided a draft of this product to DOD for comment. DOD provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense; the Acting Principal Director of Defense Pricing and Contracting; the Secretaries of the Air Force and Army; and the Directors of the Defense Advanced Research Projects Agency and the Washington Headquarters Services. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or dinapolit@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Cheryl Andrew, Assistant Director; Carmen Yeung, Analyst-in-Charge; Pete Anderson; Lorraine Ettaro; Kurt Gurka; Daniel Glickstein; Julia Kennon; Roxanna Sun; and Leanne Violette made key contributions to this report.", "summary": "In 2015, Congress granted DOD permanent authority to use agreements known as other transactions to acquire prototype projects that, among other things, demonstrate whether technologies and products can be adapted for DOD's use. This contracting approach can help DOD attract companies that do not typically do business with DOD—such as commercial science and technology firms. This is because other transactions are not subject to certain federal contract laws and requirements. GAO was asked to review DOD's use of other transactions for prototype projects. For the purposes of this report, GAO refers to these instruments as prototype other transactions. This report examines, among other issues, (1) DOD's use of prototype other transactions for fiscal years 2016 through 2018 and (2) the extent to which agreements officers followed established review processes before awarding selected transactions. GAO analyzed Federal Procurement Data System-Next Generation data and examined relevant documents from a non-generalizable sample of 11 prototype other transactions. These transactions represented various dollar values from the four DOD components that had the highest obligations through prototype other transactions in fiscal year 2018. GAO also examined DOD and component policies and interviewed DOD officials. The Department of Defense (DOD) significantly increased its use of agreements known as other transactions for prototype projects from fiscal years 2016 through 2018 (see figure). DOD data shows that companies that typically did not do business with DOD participated to a significant extent on 88 percent of the transactions awarded during this time. The Army awarded the most transactions; some of which were on the behalf of other DOD components that wanted to leverage transactions the Army previously awarded to meet their own components' needs. In nine of the 11 prototype other transactions GAO reviewed, DOD contracting officials, known as agreements officers, followed their components' established review policies before awarding the transactions. Agreements officers did not obtain higher level reviews on the two remaining transactions. In both cases, agency officials reviewed the transactions after GAO brought these situations to their attention and found no issues with the awarded transactions. A Defense Advanced Research Projects Agency agreements officer did not have a higher level review of a $7.8 million transaction before it was awarded, as required. An Army Contracting Command-New Jersey Center Director served as the agreements officer on a $10 million transaction. The Director, who would typically review transactions of this value, had his Branch Chief review this transaction prior to award. The Defense Advanced Research Projects Agency also plans to complete an internal file review of awarded transactions to check compliance with its review policy in fiscal year 2020 and take corrective actions, if necessary. The Army Contracting Command-New Jersey plans to clarify who should review transactions in such situations. GAO is not making recommendations based on the stated intent of senior contracting officials to address these issues.", "document_type": "gao"}
{"report": "In response to the national public health and economic threats caused by COVID-19, four relief laws were enacted as of June 2020, including the CARES Act in March 2020. These laws have appropriated $2.6 trillion across the government. Six areas—the Paycheck Protection Program (PPP), Economic Stabilization and Assistance to Distressed Sectors, unemployment insurance, economic impact payments, the Public Health and Social Services Emergency Fund, and the Coronavirus Relief Fund— account for 86 percent of the appropriations (see fig. 1). Total federal spending data are not planned to be readily available until July 2020. It is unfortunate that the public will have waited more than 4 months since the enactment of the CARES Act for access to comprehensive obligation and expenditure information published by federal agencies about the programs funded through these relief laws. In the absence of comprehensive data, we collected obligation (government financial commitments) and expenditure data from agencies, to the extent practicable, as of May 31, 2020. For the six largest spending areas, we found that obligations totaled $1.3 trillion and expenditures totaled $643 billion. The majority of the difference was due to PPP, for which the Small Business Administration (SBA) obligated $521 billion. The amounts for loan guarantees will not be considered expenditures until the loans are forgiven, and, for those that are not forgiven, whether they are timely repaid. We also collected spending data on other programs affected by the federal response. For example, we found that the Department of Health and Human Services (HHS) has provided $7 billion in COVID-19 Medicaid funding related to a temporary increase in the Federal Medical Assistance Percentage (FMAP), the statutory formula the federal government uses to match states’ Medicaid spending. Based on the information we collected, government-wide spending totaled at least $677 billion, as of May 31, 2020. Given the sweeping and evolving public health and economic crisis, agencies from across the federal government were called on for immediate assistance, requiring an unprecedented level of dedication and agility among the federal workforce, including those serving on the front lines, to quickly establish services for those infected with the virus. Consistent with the urgency of responding to serious and widespread health issues and economic disruptions, agencies have given priority to moving swiftly where possible to distribute funds and implement new programs. In moving quickly, however, agencies made trade-offs, and they have made only limited progress so far in achieving transparency and accountability goals. In particular, we identified several challenges related to the federal response to the crisis, as well as recommendations to help address these challenges, including the following: Viral testing. The Centers for Disease Control and Prevention (CDC) reported incomplete and inconsistent data from state and jurisdictional health departments on the amount of viral testing occurring nationwide, making it more difficult to track and know the number of infections, mitigate their effects, and inform decisions on reopening communities. However, HHS issued guidance on June 4, 2020, to laboratories that identifies required data elements to collect and how to report them to CDC. Distribution of supplies. The nationwide need for critical supplies to respond to COVID-19 quickly exceeded the quantity of supplies contained in the Strategic National Stockpile, which is designed to supplement state and local supplies during public health emergencies. HHS has worked with the Federal Emergency Management Agency and the Department of Defense to increase the availability of supplies. However, concerns remain about the distribution, acquisition, and adequacy of supplies. Paycheck Protection Program. As of June 12, 2020, the Small Business Administration (SBA) had rapidly processed over $512 billion in 4.6 million guaranteed loans through private lenders to small businesses and other organizations adversely affected by COVID-19. As of May 31, 2020, SBA had expended about $2 billion in lender fees. SBA moved quickly to establish a new nationwide program, but the pace contributed to confusion and questions about the program and raised program integrity concerns. First, borrowers and lenders raised a number of questions about the program and eligibility criteria. To address these concerns, SBA and the Department of the Treasury (Treasury) issued a number of interim final rules and several versions of responses to frequently asked questions. However, questions and confusion remained. In June 2020, Congress passed, and the President signed into law, the Paycheck Protection Program Flexibility Act of 2020, which modified key program components. Second, to help quickly disburse funds, SBA allowed lenders to rely on borrower certifications to determine borrowers’ eligibility, raising the potential for fraud. We recommend that SBA develop and implement plans to identify and respond to risks in PPP to ensure program integrity, achieve program effectiveness, and address potential fraud. SBA neither agreed nor disagreed, but we believe implementing this recommendation is essential. Economic impact payments. The Internal Revenue Service (IRS) and Treasury moved quickly to disburse 160.4 million payments worth $269 billion. The agencies faced difficulties delivering payments to some individuals, and they face additional risks related to making improper payments to ineligible individuals, such as decedents, and fraud. For example, according to the Treasury Inspector General for Tax Administration, as of April 30, almost 1.1 million payments totaling nearly $1.4 billion had gone to decedents. We recommend that IRS consider cost-effective options for notifying ineligible recipients how to return payments. IRS agreed. Unemployment insurance (UI). States are implementing three new, federally funded UI programs created by the CARES Act and, as of May 2020, states had received 42 million UI claims. The Department of Labor (DOL) has taken steps to help states manage demand, but DOL is developing its approach to overseeing the new UI programs. We will be evaluating DOL’s monitoring efforts in future reports. Further, the UI program is generally intended to provide benefits to individuals who have lost their jobs; under PPP, employers are generally required to retain or rehire employees. According to DOL, no mechanism currently exists that could capture information in real time about UI claimants who may receive wages paid from PPP loan proceeds. We recommend that DOL, in consultation with SBA and Treasury, immediately provide help to state unemployment agencies that specifically addresses PPP loans and the risk of improper payments associated with these loans. DOL neither agreed nor disagreed with the recommendation, but noted it was planning forthcoming guidance. Contract obligations. Government-wide contract obligations in response to the COVID-19 pandemic totaled about $17 billion as of May 31, 2020. Goods procured include ventilators, and services contracted for include vaccine development. In addition, the CARES Act provided $1 billion for Defense Production Act purchases—$76 million of which was awarded to increase production of N95 respirators. The nation has made some progress in fighting COVID-19. However, the virus continues to pose risks to all Americans, and there is a concern of another wave of infection this fall. This could coincide with the seasonal influenza and hurricane season—further straining federal agencies responsible for responding to these events, as well as the health care system. Additionally, the nation’s initial response to COVID-19 highlights the challenges presented by an inherent fragmentation across responsibilities and capabilities in the federal biodefense response and health care system, which includes private, public (local, state, and federal governments), and nonprofit entities. Lessons from the initial response, as well as experience from past economic crises, disasters, and emergencies, highlight areas where continued attention and oversight are needed—with the focus on improving ongoing response efforts and preparing for potential additional waves of infection. These lessons include the following: Establishing clear goals and defining roles and responsibilities for the wide range of federal departments and other key players are critically important actions when preparing for pandemics and addressing an unforeseen emergency with a whole-of-government response. Providing clear, consistent communication in the midst of a national emergency—among all levels of government, with health care providers, and to the public—is key. Collecting and analyzing adequate and reliable data can inform decision-making and future preparedness—and allow for midcourse changes in response to early findings. Establishing transparency and accountability mechanisms early on provides greater safeguards and reasonable assurance that federal funds reach the intended people and are used for the intended purposes. Such mechanisms also help ensure program integrity and address fraud risks. While Congress has taken a number of actions to help address the pandemic, it continues to consider additional actions—both to improve ongoing efforts and implement new ones—and develop plans for congressional oversight of the nation’s response to and recovery from COVID-19. Congressional oversight plays a vital role in spurring agency progress on matters of national importance. On the basis of our work on past large-scale government responses to economic downturns and other crises, we recommend Congress consider taking legislative action in the following areas: Aviation-preparedness plan. In 2015, we recommended that the Department of Transportation (DOT) work with federal partners to develop a national aviation-preparedness plan for communicable disease outbreaks. DOT agreed, but as of May 2020, maintained that HHS and DHS should lead the effort. Thus far, no plan exists. We recommend that Congress take legislative action to require DOT to work with relevant agencies and stakeholders to develop a national aviation-preparedness plan to ensure safeguards are in place to limit the spread of communicable disease threats from abroad while at the same time minimizing any unnecessary interference with travel and trade. Full access to death data. The number of economic impact payments made to decedents highlights the importance of consistently using key safeguards in providing government assistance to individuals. IRS has access to the Social Security Administration’s full set of death records, but Treasury and its Bureau of the Fiscal Service, which distribute payments, do not. We recommend that Congress provide Treasury with access to the Social Security Administration’s full set of death records and require that Treasury consistently use it to help reduce similar types of improper payments. Medicaid. We previously found that during economic downturns—when Medicaid enrollment can rise and state economies weaken—the FMAP formula does not reflect current state economic conditions. We previously developed a formula that offers an option for providing temporary automatic, timely, and targeted assistance. We recommend that Congress use this formula for any future changes to the FMAP during the current or any future economic downturn to help ensure that the federal funding is targeted and timely. In the report we issued yesterday, we also describe potential indicators that could be used to monitor public health and economic recovery. The report also contains 41 enclosures that contain information about a wide range of federal programs or initiatives that were created, expanded, or funded in the COVID-19 relief laws. In conclusion, both Congress and the administration have acted to respond to public health and economic threats posed by COVID-19. Federal agencies and personnel acted quickly to stand up new programs or expand existing programs to, among other things, aid individuals, states, and businesses. But much work remains in protecting the health and well-being of Americans, both today and in the coming months, as the nation may be forced to simultaneously confront new waves of COVID-19 infections and seasonal influenza. In our initial report we make recommendations to help improve the effectiveness of the federal government’s ongoing response. Our ongoing oversight will continue to focus on improving the government’s response and recovery efforts as well as the nation’s preparedness for future outbreaks. Chairman Clyburn, Ranking Member Scalise, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. For further information about this testimony, please contact A. Nicole Clowers, Managing Director, Health Care, at (202) 512-7114 or clowersa@gao.gov; Katherine Siggerud, Chief Operating Officer, at (202) 512-5600 or siggerudk@gao.gov; or Orice Williams Brown, Managing Director, Congressional Relations, at (202) 512-4400 or williamso@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The outbreak of COVID-19 quickly spread around the globe. As of June 17, 2020, the United States had over 2 million reported cases of COVID-19, and over 100,000 reported deaths, according to federal agencies. Parts of the nation have seen severely strained health care systems. The country has also experienced a significant and rapid downturn in the economy. Four relief laws, including the CARES Act, were enacted as of June 2020 to provide appropriations to address the public health and economic threats posed by COVID-19. In addition, the administration created the White House Coronavirus Task Force. The CARES Act includes a provision for GAO to report regularly on its ongoing monitoring and oversight efforts related to the COVID-19 pandemic. Yesterday, GAO issued its first report ( GAO-20-625 ). Like the report, this testimony focuses on key actions the federal government has taken to address the COVID-19 pandemic, GAO recommendations for improvement, and evolving lessons learned relevant to the nation’s response to pandemics, among other things. GAO reviewed data and documents from federal agencies about their activities and interviewed federal and state officials as well as industry representatives. GAO also reviewed available economic, health, and budgetary data. In response to the national public health and economic threats caused by COVID-19, four relief laws were enacted as of June 2020 that appropriated $2.6 trillion. This funding provided support to individuals, health care providers, businesses, and state and local government. While complete government-wide data will not be available until July, GAO determined that as of May 31, 2020, a total of about $1.2 trillion of assistance has been provided—close to $700 billion in expenditures and over $500 billion in loan guarantees. Consistent with the urgency of responding to widespread health issues and economic disruptions, agencies have worked hard to give priority to moving swiftly. In moving quickly, however, agencies made trade-offs; thus, only limited progress has been made so far in achieving transparency and accountability goals. GAO also identified challenges with the federal response to the crisis, including: Paycheck Protection Program (PPP). The Small Business Administration (SBA) moved quickly to establish a new nationwide program, but the pace contributed to confusion and questions and raised program integrity concerns. GAO recommends that SBA develop and implement plans to identify and respond to risks in PPP to better ensure program integrity. SBA neither agreed nor disagreed. Implementing GAO’s recommendation is essential. Economic impact payments. The Internal Revenue Service (IRS) and the Department of the Treasury (Treasury) faced difficulties delivering payments to some individuals, and made some payments to ineligible individuals, such as decedents. GAO recommends that IRS should consider cost-effective options for notifying ineligible recipients how to return payments. IRS agreed. Unemployment Insurance (UI). The program could have an unintentional overlap with benefits provided under PPP. GAO recommends that the Department of Labor (DOL) immediately provide help to state unemployment agencies that specifically addresses PPP loans, and the risk of improper payments associated with these loans. DOL is planning additional guidance. Aviation-preparedness plan. In 2015, GAO recommended that the Department of Transportation (DOT) work with federal partners to develop a national aviation-preparedness plan for communicable disease outbreaks. Thus far, no plan exists. GAO recommends Congress require DOT to produce a plan. Full access to death data. It is important to consistently use safeguards when providing assistance to individuals. The Treasury and Bureau of Fiscal Service do not have access to the Social Security Administration’s full set of death records. GAO recommends that the Congress give Treasury that access and require that Treasury consistently use it. Medicaid. GAO previously found that during economic downturns, the Federal Medical Assistance Percentage (FMAP) formula does not reflect current state economic conditions. GAO recommends that, during an economic downturn, Congress use a formula to provide timely and targeted assistance during economic downturns. In the report, GAO makes three new recommendations for agencies and three matters for consideration for Congress that address these issues.", "document_type": "gao"}
{"report": "A complete and accurate address list is the cornerstone of a successful census because it identifies all living quarters that are to receive a census questionnaire and serves as the control mechanism for following up with households that do not respond. If the address list is inaccurate, the Bureau may miss people, count them more than once, or include them in the wrong locations. As figure 1 shows, the Bureau’s approach to building complete and accurate address lists consists of a series of operations and are conducted throughout the decade. These operations include partnerships with the United States Postal Service (USPS) as well as tribal, state, and local governments. Other federal agencies, local planning organizations, the private sector, and nongovernmental entities may also contribute to these operations by providing the Bureau with updated address information as part of the Bureau’s continuous maintenance of the MAF. Like other information collected for the census, data collected through the LUCA program are subject to protections under title 13 of the U.S. Code. This means that data collected from the census cannot be used for non- statistical purposes or shared with unauthorized parties. The fundamental structure of LUCA has not changed since the Bureau first implemented it during the 2000 decennial cycle. The Bureau implements LUCA once every 10 years, near the end of the decennial census cycle. The Bureau invites governments to review the MAF for their respective areas. These governments must abide by Title 13 by protecting the address data from disclosure. Participating governments can then submit address updates for inclusion in the address list before enumeration. The Bureau can accept or reject these address updates, which participants then have the opportunity to appeal through an appeals office that OMB administers and that the Bureau funds (see figure 2). While the structure of the program is largely the same as in previous enumerations, the Bureau has made some changes to promote participation and reduce perceived participation barriers. For example, in 2010, the Bureau extended review timelines from 90 to 120 calendar days in response to LUCA participants’ feedback that they did not have enough resources to complete a sufficient review within the Bureau’s original time frame. Additionally, in the 2010 and 2020 cycles, the Bureau permitted state governments to participate in LUCA. State participation can provide coverage for local governments that may not have the resources to participate in the operation. Moreover, following the 2010 Census and in response to our prior recommendations, the Bureau assessed LUCA’s contribution to the final census population counts. Doing so improved the Bureau’s ability to determine how helpful LUCA was in gathering address information from participants across the nation. In September 2014, the Bureau decided that it would only need to verify addresses door to door in those areas it could not resolve with the aid of computer imagery and third-party data sources—what the Bureau calls in- office address canvassing. The Bureau used this method of address canvassing to reduce the costs of the labor-intensive “in-field address canvassing”, which cost about $450 million during the 2010 Census. As part of this effort, the Bureau planned to rely on in-office address canvassing as the primary method for validating address updates submitted during LUCA 2020. After the Bureau builds its address list, it must enumerate residents and follow up with them as necessary. Historically one of the most cost- intensive operations of the decennial census, the Bureau implements Non-response Follow-up after the self-response period so that it can (1) determine the occupancy status of individual nonresponsive housing units and (2) enumerate them. The Bureau allows up to six enumeration attempts for each nonresponsive housing unit or case. Any addresses added from LUCA submissions become eligible to be enumerated. Other sources of address data complement the Bureau’s data-collection efforts. For instance, according to experts, systematic collection of address data is now common at the state and local level, which allows many governments to readily provide address information to the Bureau. Since 2013, the Bureau has also received address updates throughout the decade from the USPS as well as from tribal, state, and local governments through its Geographic Support System (GSS) Program, increasing the frequency of address updates. Outside of the auspices of Title 13-protected census data, states and federal agencies have worked toward making a national address database publicly available. For example, the National Address Database, managed by the U.S. Department of Transportation as part of its work with the Bureau on federal address data issues, is an open source database which enables governments to view and submit their address information, including geospatial coordinates, for use across governmental agencies. In 2015, we reported on the National Address Database and Title 13, suggesting that Congress consider assessing statutory limitations within Title 13 on address data to foster progress toward such a national address database. However, there has been no legislative action at the time of this report. We found the Bureau’s implementation of LUCA 2020 largely followed its operational plan, including key milestones, as well as outreach and training objectives. Milestones. Through July 2019, the Bureau had met its milestones laid out in the LUCA 2020 Operational Plan as summarized in table 1, with two minor changes that provided participating governments additional time. First, in starting up the program, the Bureau was able to mail out advance notice packages a month earlier than specified in the 2020 Operational Plan to give potential participants additional time to assess the resources they would need to participate before receiving the formal invitation. Secondly, the Bureau extended the deadline for participating governments to submit address updates because natural disasters affected large regions of the country. Outreach and training. The Bureau performed outreach and training according to its LUCA 2020 Operational Plan. For example, the Bureau provided technical training workshops for government representatives, including training on address privacy laws. The Bureau implemented a streamlined participation process and received address updates from participating governments covering 96 percent of the estimated population of the country. Based on the Bureau’s post-2010 recommendations to improve LUCA for the 2020 Census, the Bureau did not ask participants to provide their full address lists (an option in 2010), but invited governments to review only the Bureau’s address list and offer updates. As shown in table 2, the Bureau saw little change in the number of governments invited to participate, registering to participate, and responding from the 2010 Census. The changes in participation options prevent precise analysis of participation beyond counting the number of governments that responded in some fashion. Moreover, in 2000, the Bureau implemented LUCA with two phases of data collection—one for rural addresses and one for urban, with some governments eligible to provide address updates during both phases. This differs from later decennials which condensed LUCA into a single phase. However, the Bureau’s measure for government participation excludes important information about the degree of that participation. For instance, only 8,389—or 21 percent of the nearly 40,000 tribal, state, and local governments—participated in LUCA 2020. According to Bureau officials and subject matter specialists we interviewed, address data are generally improved when both a state and another level of government participate in LUCA, even if the respective address updates cover some of the same addresses. According to the Bureau, such redundancies can help address the possibility of coverage gaps in any one government’s address updates. Governments at the more local level can apply their targeted, on-the-ground intelligence in cases where a state government may lack the resources and data to cover the entire population as part of its review of the MAF. As figure 3 shows, the degree of local participation in LUCA varied greatly across the country. For example, while state governments in New Mexico and Oklahoma participated, many counties and local governments (e.g., towns and cities) within those states did not. Moreover, states like Texas and South Dakota lacked any form of coverage in LUCA for many of their counties. In contrast, large parts of the west coast and the southeast benefitted from participation in LUCA by governments at multiple levels. The Bureau maintains participation data on government type and shows information similar to figure 3 on its external website. However, the percentage of the population covered by at least one form of government submission—identified by the Bureau as a primary performance measure—does not identify participation in this way, nor does it distinguish between governments representing a mix of urban and rural geographic areas that have participated. Bureau officials told us that state-centric participation was a focus for LUCA 2020 and that they encouraged local governments to coordinate with state governments on their address lists. The purpose of the legislation that prompted LUCA was to help ensure accuracy of the census by permitting various levels of government to review the Bureau’s address data. We have previously reported that a program’s measures should be consistent with the program’s initial (or updated) statutory mission. The Census Address List Improvement Act of 1994 called for the Bureau to solicit input on the address list from tribal and local governments as well as state governments. The Bureau may be able to find opportunities to obtain more complete coverage by tracking metrics related to the types of governments participating in LUCA and the degree to which tribal, state, and local governments are complementing each other’s address updates. In doing so, the Bureau could ensure that the LUCA program is contributing to accurate enumeration. Tracking these metrics would also give the Bureau valuable feedback on the success of its nationwide outreach and could increase the accuracy of the MAF. Fieldwork in other 2020 Census operations increased as a result of (1) LUCA’s original operational design, and (2) subsequent implementation decisions the Bureau made in response to receiving a larger number of address updates than it expected from participants. By design, the Bureau had planned not to review suggested changes occurring in geographic areas previously determined to be high growth, since the Bureau had already planned to canvass such areas for addresses door- to-door later. When the Bureau received more than two million more address updates than it had expected, it decided to review a sample of updates in areas not slated automatically for in-field review, passing even more work directly on to Non-Response Follow-Up (NRFU) at a potential cost of more than $25 million (in constant 2020 dollars). The Bureau received 11 million address updates proposed by participating governments, but about 5.1 million of these did not match addresses in the MAF— approximately two million more than expected. Bureau officials had not formalized any specific estimates but initially expected that participants would propose about 5 million address updates to the MAF, of which about 2.8 million would not match to the MAF and would need to be reviewed. As figure 4 shows, 2.5 million of the 5.1 million new address updates that LUCA participants submitted were in high-growth areas and passed directly on to in-field address canvassing. While the Bureau’s reengineered approach to address canvassing for 2020 substantially reduced fieldwork, this pass-through of additional workload represents a missed opportunity for the Bureau to further reduce costs for in-field address canvassing. With a planned cost of $185 million (in fiscal year 2019 costs) for 2020, in-field address canvassing is one of the most expensive census operations, according to the Bureau’s July 2019 lifecycle cost estimate. Another decision also led to increased workload. The Bureau streamlined its address validation process in response to the higher-than-anticipated number of address updates received. To manage this workload, the Bureau reviewed only a sample of address updates suggested by governments with 200 or more addresses otherwise eligible for review (861,000 total updates out of 2.5 million) that were in areas not already flagged for in-field address canvassing. As a result, the Bureau added more than 1.6 million address updates to the MAF without review as shown above, even though they were eligible for in-office address canvassing. The Bureau will attempt to enumerate households during the census through self-response methods, such as online or paper questionnaires. If the Bureau does not initially receive responses, these addresses will become part of the NRFU workload. Had these addresses been canvassed in office, it is likely that many of them would have been rejected, based on the rejection rate for other addresses. Specifically, the Bureau rejected 39 percent (334,000 out of 861,000) of the address updates it reviewed in its sample. If a similar rate of rejection were to have occurred in both groups, roughly 624,000 additional address updates would have been rejected instead of being included in the enumeration universe with possible unnecessary NRFU follow up. Assuming the same average cost of NRFU per case as in 2010, these additional cases receiving census questionnaires could result in an unnecessary $25 million in costs (in constant 2020 dollars). Standards for Internal Control in the Federal Government indicates that agencies should use quality information to achieve their objectives. The Bureau’s decisions to limit the reviews conducted on submitted LUCA updates mean that the Bureau will have some addresses in the MAF for address canvassing and NRFU of unknown quality that will result in potentially unnecessary fieldwork. Creating the conditions whereby the Bureau can expand the scope of in-office review of tribal, state, and local additions to the MAF will better position the Bureau to reduce its fieldwork and related costs. The Census Address List Improvement Act of 1994 required that OMB establish a process to adjudicate differences between the Bureau and LUCA participants over proposed address updates to the MAF. The Bureau and the LUCA appeals office that OMB established will conduct the feedback and appeals phases of LUCA, respectively, from July 2019 through January 2020. Feedback to participants began in July 2019, and the subsequent appeals process is expected to run through January 2020. The Bureau and OMB expect fewer LUCA appeals for 2020 than in 2010 due in part to the Bureau’s decision to review only a portion of submitted address updates and provisionally accept the rest. In 2010, participants could appeal 13.3 million addresses, while according to the Bureau only about 1.7 million addresses will be eligible in 2020. According to OMB, as of mid-October 2019, the LUCA appeals office had begun processing files containing appealed addresses from 1,122 participants. Officials indicated the appeals office will not determine the total number and dispositions of addresses processed until after the end of the operation. As in 2010, OMB is giving participants 45 calendar days to appeal the Bureau’s individual address reviews. Since 2000, the LUCA appeals process has resulted in approval of more than 90 percent of LUCA appeals that participating governments have submitted, including more than 1.6 million appealed addresses (91 percent) in 2010. OMB officials noted that the practice for the appeals process is to side with the participants if the weight of evidence on either side of an appealed address is equal, which may account for the high percentage of approved appeals. OMB is replicating this practice for 2020, according to the final regulation establishing the LUCA appeals process in July 2019. Yet the Bureau’s post-2010 evaluation showed that, among all forms of late additions to the MAF, addresses that were reinstated to the MAF because of a LUCA appeal were the least likely to be found valid as either residential or commercial addresses. Ultimately, the Bureau enumerated individuals at 55 percent of such addresses for the 2010 Census (compared to 83 percent of addresses added late to the MAF through other operations). The 2010 LUCA appeals process resulted in the Bureau contacting and enumerating over 700,000 households that otherwise would be less likely to be enumerated, yet the high rate of erroneous addresses added to the MAF through appeals reinstatement will be an additional source of NRFU workload, making that operation more costly than necessary. Given that LUCA is one of several operations used to build the MAF, it is important for the Bureau to assess and determine how the high rate of LUCA address updates that are reinstated through the appeals process affect other operations and, thus, LUCA’s cost-effectiveness. Standards for Internal Control in the Federal Government states that management should use quality information to achieve the entity’s objectives. In its post-2010 evaluation, the Bureau acknowledged that it needed to research the reason for this seemingly low enumeration rate and to form a plan to resolve the cause. However, it has yet to do so. Evaluating the enumeration outcomes of appealed addresses and identifying factors that led to these results could help to reduce the cost of unnecessary enumeration attempts, as well as costs associated with the administration of the appeals process. The Bureau provided us with estimates for what LUCA would cost for the 2020 Census, but it was unable to provide sums for other address- building operations. The Bureau estimates that LUCA 2020 operations will cost $29.6 million. Among other expenses, this includes certain information technology costs, printed materials for outreach, and salaries for Bureau staff and contractors throughout the decade. Beyond the LUCA operation, the Bureau has several other initiatives that provide information for the MAF, such as the USPS’s Delivery Sequence File and the GSS Program. According to Bureau cost documentation, these operations are funded through the Bureau’s Geographic Support Program at a level of $59 million annually since 2016. However, the Bureau does not isolate the costs of operations within the Geographic Support Program that may provide information on the relative cost- effectiveness of LUCA and related operations in updating the MAF. Bureau officials and stakeholders that we spoke with have cited the GSS initiative—which processes tribal, state, and local modifications to the MAF throughout the decade—as an alternative design for LUCA. Officials told us that costs for GSS are not tracked separately from other initiatives that update the MAF and the Bureau’s geocoding database. Standards for Internal Control in the Federal Government states that agencies should establish and operate monitoring activities, such as tracking program costs. Additionally, GAO’s 21st Century Challenges: Reexamining the Base of the Federal Government indicates that, to meet current and future challenges, it is important to evaluate whether programs are using the most cost-effective or net-beneficial approaches when compared to other tools and operation designs. Since the Bureau does not isolate costs specific to various design components it uses to build and update its address list, it is not possible to evaluate the relative cost-effectiveness of LUCA’s current design in the context of other address-list building the Bureau has undertaken for the 2020 Census. Identifying and tracking these costs would help the Bureau to determine the cost-effectiveness of its address-building activities and identify improvements. While the Bureau largely implemented its approach for LUCA 2020 as planned, the Bureau missed several opportunities to maximize the benefits of LUCA toward improving the quality and reducing the cost of the census. Specifically, increased fieldwork, time for participants to review their address lists, and use of data on hard-to-count populations all emerged as challenges for the Bureau to address in any future implementation of LUCA or a similar program. Data from LUCA reviews could have helped administrative records modeling. In 2020, the Bureau is planning to use administrative records to reduce the amount of follow-up it does seeking responses from vacant or nonexistent addresses. Bureau officials noted that the Bureau learns information from its review of the quality of LUCA updates that could benefit its modeling with administrative records, perhaps resulting in more cases where administrative records are deemed good enough to reduce NRFU further. Standards for Internal Control in the Federal Government states that agencies should use quality information to achieve their objectives, in part by obtaining relevant data from reliable sources. The Bureau did not, however, plan to use information about addresses gathered during LUCA—such as during its reviews of address updates during LUCA validation—to help with its use of administrative records for the 2020 Census, nor determine how best, and when, to transfer data between the respective Bureau teams to make this happen. However, having information on the likelihood of addresses existing can help the Bureau tailor its strategy for following up with addresses that do not produce census responses. In addition, incorporating information learned about addresses added through the appeals process may also improve the results of the Bureau’s modeling with administrative records, which could in turn reduce workload during NRFU. Time constraints continue to limit participation. Officials of multiple participating governments and other subject matter specialists told us that the constrained timing of LUCA continues to be a barrier for governments to fully participate. For 2020 and in prior iterations of LUCA, insufficient time was one of the leading factors behind governments’ decisions not to participate. Our prior work on re-examining the base of the federal government highlights the importance of ensuring that a program is meeting its original purpose. Since its inception, LUCA has been intended to ensure that tribal, state, and local governments have the opportunity to review the Bureau’s decennial address list. In the 2010 Census, the Bureau increased the length of time governments had for reviewing the MAF from 90 days to 120 days, and kept this length for 2020. Yet, if governments lack the resources needed to review address lists, and if governments run out of time, they either may not participate, or their address updates may not reflect a comprehensive review of the MAF for their jurisdictions. Bureau officials agreed that more time for governments to participate would be better. Facilitating increased participation, along with expanding the scope of in-office reviews of LUCA submissions, however, may require the Bureau to realign its schedule for other phases of tribal, state, and local outreach. Figure 5 shows one potential opportunity for the Bureau to do this. The Bureau scheduled a 5-month gap between the end of its in-office address canvassing (and thus LUCA address validation) and the beginning of in-field address canvassing. Bureau officials said this period is needed to determine the right number of listers to hire and train, as well as to prepare official address materials needed for later operations. However, the 2020 schedule gave participants less time to submit updates than they could have had if the Bureau’s address validation phase had taken place later. Moreover, as previously noted, participants had from July 2017 to February 2018 to register for LUCA; officials noted that it could be possible to provide the review materials on a rolling basis so that participants who registered early could have more time to review their address lists. Finding opportunities like this to give participants more time for their review could improve the Bureau’s coverage. The Bureau did not use its data on hard-to-count areas to help guide LUCA. During LUCA 2020, the Bureau missed an opportunity to target efforts in order to improve address listing in areas considered by the Bureau to be hard-to-count. We have previously reported on the importance of targeting a program’s benefits to those with the greatest needs and the least capacity to meet those needs. The Bureau maintains publicly available data at the census tract level on the extent to which a geographic tract (roughly the population size of an urban neighborhood) is considered hard-to-count. Bureau officials told us, however, that they had not previously considered reviewing these data regularly when monitoring LUCA participation or prioritizing in-office review workloads. When an address is missing, the people at that address are more likely to be missed by the census. Bureau officials managing LUCA told us that using the Bureau’s data on hard-to-count areas could have given them insights into whether they were receiving LUCA participation for areas most in need of improvements in census coverage and whether they needed to better target their LUCA outreach. Moreover, Bureau officials told us that they would prefer to have more opportunity to provide feedback to participants regarding their submitted updates and their address lists. Given the time constraints discussed elsewhere in this report, data showing which participants are in hard-to-count areas could help the Bureau prioritize governments with which to invest time giving feedback. According to Bureau officials, this information could also help the Bureau prioritize its resources in other address list-building efforts, such as which areas the Bureau should conduct additional rounds of in-office address canvassing to ensure that recent address updates are not missed. Conditions surrounding LUCA have changed since LUCA was first implemented in the 2000 Census. For example, the dissemination of publicly available address data has increased, and the Bureau has developed other mechanisms for governments to provide input to its address list. However, LUCA’s designed role in the census has not fundamentally changed or been reexamined since its authorizing legislation. Moreover, the Bureau will soon begin its process for planning geographic programs for 2030. This presents an opportunity to reexamine LUCA’s contributions to building a complete and accurate address list. In 2005, we identified criteria for reexamining federal programs in order to address fiscal instability while updating federal programs and priorities to meet current and future challenges. These criteria are based on a need to inform Congress of our insights in order to help its budget and programmatic deliberations and oversight activities. These criteria include whether the program is using the most cost-effective approach when compared to other tools and program designs; whether a program is targeted to those with the greatest need; and what would be the likely consequences of eliminating an operation. Our review of Bureau documents and evaluations—along with interviews of Bureau officials, subject matter specialists, and state-level LUCA participant stakeholders—identified several issues for the Bureau to resolve with stakeholders, Congress, and other federal agencies as part of the planning process for the 2030 Census: Assessing whether LUCA should continue to have a role in building the address list. The first issue for the Bureau, Congress, and other stakeholders to resolve is whether LUCA should continue to be a vehicle for tribal, state, and local additions to the MAF. The Bureau receives intergovernmental inputs into the MAF through multiple sources, such as GSS and surveys of local governments to determine jurisdictional boundaries. The Bureau’s decisions on the scope of LUCA address validation for 2020 also mean that the effects of LUCA on address list quality are unclear. Yet, a committee of state- level stakeholders and subject matter specialists emphasized the value of having a forum for governments to review the Bureau’s address list—a feature that is currently unique to LUCA. By registering for LUCA under the authority of Title 13 nondisclosure requirements, governments can also receive feedback from the Bureau on their individual address updates, which the chair of a nationwide group of state-level population data officials told us was valuable. Moreover, stakeholders told us that having a program like LUCA late in the decennial cycle may help promote awareness of the census at the state and local level. Determining how frequently to have governments review the MAF. The method and frequency with which governments can review the MAF is another issue for the Bureau to resolve. A committee of state-level stakeholders and subject matter specialists told us that having more opportunities for tribal, state, and local review of the MAF during the decade would increase participation and thus quality of the MAF by relaxing the time constraints that have historically deterred participation in LUCA. Bureau officials also told us that a continuous program would provide more opportunities for governments to refine their address lists based on feedback from the Bureau. However, increasing the frequency of address updates, reviews, and appeals during the decade would increase program administration costs, and such a program’s design would need to account for the fact that smaller governments and LUCA nonparticipants already cite the lack of human and financial resources as a barrier to participation. Considering whether to make it easier for governments to access and share address data. Given the prevalence of modern address sources and services, the question of how closely to protect data on census addresses is another issue for the Bureau to resolve in conjunction with Congress and stakeholders. We have previously recommended that Congress consider revising Title 13 nondisclosure protections for address data. Bureau officials and subject matter specialists we interviewed said if federal agencies and tribal, state, and local governments could more easily share address lists, there could be benefits to address list quality. Bureau officials have also described scenarios in which it may be possible to enact targeted modifications to Title 13 so that only address data are affected. However, subject matter specialists we interviewed also noted that Title 13 protections can give reassurances to local residents and facilitate participation in building local address lists. Allowing widespread disclosure and use of the Bureau’s address list could also raise questions about which address lists are considered authoritative. Determining the role that a National Address Database should play in contributing to the Bureau’s address list. Deciding whether or how to leverage an existing publicly accessible address list as part of the Bureau’s decennial efforts is another issue to resolve. We have previously recommended that agencies responsible for interagency address and geospatial policy take actions to facilitate collection of national geospatial address data. First piloted in 2015 and now managed by the U.S. Department of Transportation (DOT), the National Address Database (NAD) provides publicly available address and geographic coordinates to government and non-government users. State-level stakeholders and DOT officials said a centralized, open-source form of address data would benefit public services, such as emergency response. Going forward, however, it will be important to address resource constraints that limit the NAD’s reach. DOT’s lead official for the NAD said that there are two permanent staff who oversee nationwide outreach and data collection, and at the time of this report, the NAD only has data from partners in 23 states. These issues have been prompted by developments that have taken place this decennial cycle, such as the development of the NAD and the advent of additional inputs into the MAF such as GSS; therefore, the Bureau has not yet had an opportunity to evaluate them in its decennial planning. Standards for Internal Control in the Federal Government underscores the need to identify, analyze, and respond to significant changes, as well as use quality information and communicate externally with stakeholders. With strategic planning for 2030 geographic programs in mind, the Bureau has an opportunity to engage with stakeholders, other federal agencies as appropriate, and Congress to resolve these issues and evaluate how various alternatives could impact the cost, quality, and public perception of the census. The above issues do not exist in isolation, however, and need to be resolved jointly. For instance, decisions to make address data more accessible would increase inter-agency data sharing and thus incentives for governments to participate in open-source address initiatives like the NAD. Decisions on whether to continue LUCA in its current form will affect the tools, such as GSS, available to tribal, state, and local governments to provide updates to the MAF. As the Bureau engages with affected partners on these issues, it will be important to consider various scenarios that could flow from resolving these issues in concert with each other. The Bureau’s implementation of LUCA for 2020 is on track in terms of milestones thus far, and the process for governments to appeal rejected LUCA address updates is ongoing and will continue through January 2020. The Bureau also implemented planned changes to participation options for governments and tracked participation by government. However, the Bureau’s primary metric for representing the coverage of the nation by the LUCA operation does not leverage other information the Bureau already has on the degree of useful overlap in coverage across different levels of participating governments. Identifying and reporting metrics on the extent to which governments participating in LUCA overlap in their coverage of residents, as well as the characteristics of participants such as type of government and the nature of their geographic area, could provide more complete and useful feedback on the success of LUCA and assurance of getting desired coverage while avoiding gaps. We also found that opportunities exist for the Bureau to further reduce fieldwork and make its address list-building efforts more cost effective. In the future, the Bureau could more fully use its in-office address validation process for LUCA to reduce costs and improve decennial accuracy. Further, identifying the factors that lead to enumeration outcomes of the LUCA appeals process may also produce lessons learned that could help lower the amount of fieldwork and thus costs. Moreover, maintaining more detailed cost data for the Bureau’s other related address list development efforts will help position the Bureau to evaluate the relative cost-effectiveness of LUCA in building the address list. Likewise, the Bureau could also leverage the results of its in-office review of LUCA updates, as well as its evaluation of the appeals process, to inform its administrative records modeling and potentially reduce the number of required in-field NRFU visits. The Bureau can similarly take additional steps through programs like LUCA to promote greater coverage in the census. By realigning the schedule of LUCA where appropriate, the Bureau could give tribal, state, and local governments more time to review the address list in their areas and thus more time to provide quality updates to the Bureau. Moreover, using data on participation in LUCA and related programs, in concert with existing data on hard-to-count areas, would help the Bureau target its resources for building the address list and conducting decennial outreach to those areas most in need. We have also identified fundamental issues related to the Bureau’s address list activity that will require a forward-looking, stakeholder- inclusive approach for the Bureau to resolve. Re-examining LUCA and the related issues will not be easy, and could take time. The Bureau is uniquely positioned to lead the identification and assessment of what the alternatives are, and particularly how they might affect the cost and quality of the decennial census. Reporting out on the alternatives and their justifications, and developing legislative proposals, as may be appropriate, will help the Bureau, Congress, and the users of census data benefit from cost and quality improvements in decennials to come. We are making the following eight recommendations to the Department of Commerce and the Census Bureau: The Secretary of Commerce should ensure that the Director of the Census Bureau identifies metrics on the extent to which governments participating in LUCA overlap in their coverage of residents, as well as the characteristics of participants such as type of government and geographic area, and reports on such metrics. (Recommendation 1) The Secretary of Commerce should ensure that the Director of the Census Bureau takes steps to conduct in-office reviews of a greater share of addresses submitted by governments before the addresses are added to the Bureau’s address list for potential field work. (Recommendation 2) The Secretary of Commerce should ensure that the Director of the Census Bureau, as part of the Bureau’s assessment of LUCA for 2020, consults with OMB to report on the factors that led to enumeration outcomes of addresses reinstated to the Bureau’s master address list by the LUCA appeals process. (Recommendation 3) The Secretary of Commerce should ensure that the Director of the Census Bureau identifies and tracks specific costs for related address list development efforts. (Recommendation 4) The Secretary of Commerce should ensure that the Director of the Census Bureau improves the use of LUCA results to inform procedures of other decennial operations, such as sharing information on address update quality to inform NRFU planning or administrative records modeling. (Recommendation 5) The Secretary of Commerce should ensure that the Director of the Census Bureau realigns the schedule of LUCA-related programs to provide participants with more time to review addresses. (Recommendation 6) The Secretary of Commerce should ensure that the Director of the Census Bureau uses the Bureau’s data on hard-to-count areas to inform geographic activities such as: targeting LUCA outreach to tribal, state, and local governments; planning additional rounds of in-office address canvassing; and providing feedback to tribal, state, and local governments on gaps in their respective address data. (Recommendation 7) The Secretary of Commerce should ensure that the Director of the Census Bureau, as part of the Bureau’s strategic planning process for geographic programs, reexamines LUCA in conjunction with stakeholders, other federal agencies as appropriate, and Congress to address the issues we have identified, including but not limited to: Identifying and assessing alternatives and describing corresponding effects on the decennial census. Reporting out on the assessment of alternatives, including justifications. Developing legislative proposals, as appropriate, for any changes needed to LUCA and address data in order to implement preferred alternatives. (Recommendation 8) We provided a draft of this report to the Secretary of Commerce, the Acting Director of the Office of Management and Budget, and the Secretary of Transportation. In its written comments, reproduced in appendix I, the Department of Commerce agreed with our findings and recommendations and said it would develop an action plan to address them. The Department’s response also describes several claims of cost savings and efficiency gains attributable to various address list-building activities. While we have previously reported on the Census Bureau’s 2020 address list-building efforts, we have not audited claims made in the Department’s response or elsewhere regarding potential cost savings from innovations for the 2020 Census. The Census Bureau, Office of Management and Budget, and U.S. Department of Transportation each also provided us with technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Commerce, the Undersecretary of Economic Affairs, the Director of the U.S. Census Bureau, the Acting Director of the Office of Management and Budget, the Secretary of Transportation, and the appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Ty Mitchell (Assistant Director), Devin Braun, Charles Culverwell, Rob Gebhart, Allison Gunn, Lisa Pearson, Kayla Robinson, Robert Robinson, Cynthia Saunders, and Peter Verchinski made significant contributions to this report.", "summary": "A complete address list is a cornerstone of the Bureau's effort to conduct an accurate census. LUCA is one of several operations the Bureau uses to produce its address list. It gives tribal, state, and local governments the opportunity to review the address list for their areas and provide the Bureau with any updates before the census. GAO was asked to review the status of LUCA, including its effect on other operations, as well as LUCA's overall effectiveness and necessity. This report examines (1) LUCA's status and its likely effects on 2020 field operations, and (2) what considerations the Bureau and other stakeholders could use to reexamine LUCA for 2030. GAO reviewed Bureau plans, analyzed data from LUCA participation and the Bureau's review of submissions, and held 9 discussions on a possible reexamination of LUCA with relevant Bureau officials, a council representing participating governments, and census data subject matter specialists. The Census Bureau generally followed the operational design for its Local Update of Census Addresses (LUCA) program, which is intended to give tribal, state, and local governments the ability to review and offer modifications to the Bureau's Master Address File (MAF). The Bureau met milestones, apart from extending the participation window for natural disaster-stricken areas, and generally followed plans for outreach, training, and participation options. However, some decisions created additional fieldwork. The Bureau received more updates from participants than it expected, so it only reviewed roughly 860,000 of the 5.1 million updates that did not match to the MAF (see figure below). The rest will be added to potential fieldwork. Had more addresses been reviewed in-office, many may have been rejected, based on the rejection rate for reviewed addresses. Avoiding this unnecessary fieldwork could have saved the Bureau millions of dollars when following up with non-responding households. The Bureau has not reexamined LUCA with respect to the cost, quality, and public perception of the census since the program was authorized in 1994. Yet much has changed since then, from the tools the Bureau uses in building its address list to the provision of publicly accessible address data. As the Bureau turns to its strategic planning process for 2030, it will have several issues to address regarding the future of LUCA, including: whether LUCA should continue to have a role in building the address list given the advent of other address-building initiatives; how often to have governments review the MAF for the census, in light of the costs and benefits of administering such a program more frequently; whether statutory nondisclosure protection of census address data is still needed given that address data sources and services are more prevalent. GAO is making eight recommendations to the Department of Commerce, including that the Bureau ensure more LUCA submissions are reviewed and reexamine LUCA to address the related issues GAO identified as part of the Bureau's strategic planning process for the 2030 Census. The Department of Commerce agreed with our findings and recommendations and described several cost savings and efficiency gains—which we have not audited—from their related address list-building efforts. The Census Bureau, Office of Management and Budget, and U.S. Department of Transportation each also provided us with technical comments, which we incorporated as appropriate.", "document_type": "gao"}
{"report": "Within VHA, the Caregiver Support Program Office, VISNs, and VAMCs all have a role in administering and overseeing the Family Caregiver Program. Caregiver Support Program Office. The Caregiver Support Program Office administers the Caregiver Support Program, which has two main components— 1. the Family Caregiver Program, which is available to eligible post-9/11 veterans, and their qualified caregivers, and 2. the Program of General Caregiver Support Services, which is available to covered veterans from any service era and their qualified caregivers. The Caregiver Support Program Office develops policy and procedures and provides guidance, oversight, and support for both components of the Caregiver Support Program. As of April 2019, this office had 11 full-time staff, with authorization to hire eight additional staff. VISNs. Each of VHA’s 18 VISNs has a lead official for the Family Caregiver Program—either a VAMC CSC who serves in the VISN lead role for at least 25 percent of the CSC’s time or a VISN employee who is responsible for the Family Caregiver Program as one of their VISN duties. The VISN lead official’s role is to provide guidance to CSCs within the VISN and to help address their questions or concerns. VISN lead officials are also responsible for disseminating information, collecting data when needed, conducting quality assurance audits, assisting and coordinating responses to inquiries from the Caregiver Support Program Office, and monitoring the Family Caregiver Program workload across the VISN. VAMCs. The program is administered at the local level at 140 VAMCs. Each VAMC has staff that are assigned to the program on either a full- time or part-time basis, as well as other VAMC staff that may assist with specific Family Caregiver Program-related activities as a collateral duty. VAMC staff assigned to the Family Caregiver Program may include the following: CSCs. CSCs are the primary program staff administering the program at VAMCs. They are generally licensed clinical social workers or registered nurses. CSCs have clinical responsibilities that may include identifying and coordinating appropriate interventions for caregivers or referrals to other VA or non-VA programs, such as mental health treatment, respite care, or additional training and education. CSCs also have administrative responsibilities that may include responding to inquiries about the program, overseeing the application process, and entering information about applications and approved caregivers into IT systems. During the first quarter of fiscal year 2019, there were approximately 436 CSCs assigned to 140 VAMCs. Administrative staff. Administrative staff are typically responsible for activities such as mailing communications to program applicants and participants, scheduling appointments, entering data into CAT, and otherwise supporting the administrative needs of the program. During the first quarter of fiscal year 2019, 24 of the 140 VAMCs or health care systems had administrative staff members assigned to the program. Clinical staff. Some VAMCs have clinical staff assigned to the program, which can include registered nurses, doctors, nurse practitioners, occupational therapists, or psychologists. These staff typically conduct in-home monitoring and may help with clinical eligibility determinations of veterans during the application process. During the first quarter of fiscal year 2019, 17 VAMCs had approximately 12 full-time-equivalent doctors, nurse practitioners, occupational therapists, and psychologists assigned to the program. Other VAMC clinical staff who are not assigned to the Family Caregiver Program may assist the program as a collateral duty. For example, they may serve as members of the clinical eligibility team or assist with program monitoring—including quarterly contacts and annual home visits—or program appeals (see fig. 1). The Caregiver Support Program Office directly funds the salaries for staff assigned to the Family Caregiver Program at VAMCs. Specifically, it funds the salaries of the CSCs, as well as some other staff who are assigned to the program, such as administrative staff or clinical staff. However, some VAMCs may also choose to fund additional staff for the program, if they identify a need. Additionally, the portion of time spent by VAMC staff assisting the program as a collateral duty may be reimbursed by the Caregiver Support Program Office. To participate in the program, caregivers and veterans must submit applications to their local VAMC or to VHA’s Health Eligibility Center. CSCs manage the multi-step application process, which includes administrative and clinical eligibility determinations, among other requirements (see fig. 2). According to VHA policy, VAMCs should review applications for the program within 45 days. However, this review can be extended up to 90 days if the veteran’s caregiver has not completed required training, or the veteran is hospitalized during the application process. Once caregivers and veterans are enrolled in the Family Caregiver Program, VHA policy requires CSCs or other VAMC clinical staff to periodically monitor the veteran’s overall health and well-being and the adequacy of the care and supervision being provided by the caregiver. This monitoring is to be documented in CAT as well as in the Computerized Patient Record System because it is a clinical encounter. The monitoring includes quarterly contacts. These contacts are supposed to occur every 90 calendar days, unless otherwise clinically indicated. They may be conducted as home visits, or if approved by the veteran’s primary care team, the contacts can be completed via telephone, a face-to-face visit at a VHA medical facility, or using clinical video telehealth. annual home visits. Caregivers and veterans must receive at least one home visit each year. According to Caregiver Support Program Office officials, the annual home visit counts as one of the quarterly contacts. If a veteran demonstrates an improvement or decline in their functioning while in the program, VAMC staff are supposed to reassess the veteran to determine whether they remain clinically eligible for the program or whether a change in the stipend tier level (increase or decrease) may be appropriate. A reassessment may result in a discharge from the program, a tier level change, or no change. However, the VA Secretary announced a moratorium on discharges and tier level decreases on December 21, 2018 due to continued concerns from veterans, caregivers, and others about VAMCs’ inconsistent application of eligibility requirements. According to Caregiver Support Program Office officials, there is no current timeline for when the moratorium will be lifted. In August 2018, the VA Office of Inspector General (VA OIG) issued a report on its review of the Family Caregiver Program, which focused on whether the program effectively provided services and support to qualified veterans and their caregivers. The VA OIG found that program applications were not reviewed in a timely manner, eligibility criteria were not consistently applied, caregivers and veterans were not routinely monitored, and that VHA had failed to effectively establish a governance structure that promoted program management accountability. The VA OIG made six recommendations to improve the program, including recommendations to establish a governance structure and to assess the adequacy of the program’s staffing levels at VAMCs. In May 2019, the VA OIG reported that the program had implemented the two recommendations related to establishing a governance environment and designating VISN lead officials for the program. Specifically, the Family Caregiver Program issued an updated directive for the program and additional standard operating procedures in October 2018 to address the governance environment recommendation and issued a memorandum regarding VISN lead officials in January 2019 to address the VISN lead official recommendation. According to the VA OIG, the remaining recommendations have not yet been implemented. The VA MISSION Act, which was enacted in June 2018, included provisions directing VA to implement an IT system to support the Family Caregiver Program and the incremental expansion of program eligibility. Specifically, the Act required VA to implement an IT system by October 1, 2018. According to the Act, the IT system is to allow for data assessment and comprehensive monitoring of the program. The VA MISSION Act also required VA to submit an initial report to Congress regarding the status of the planning, development, and deployment of this system within 90 days of enactment of the Act and a final report by October 1, 2019. The final report is to include a certification by the VA Secretary that the system has been implemented, along with a description of how the Secretary is using the system to monitor the workload of the program. In addition, the VA MISSION Act requires an incremental expansion of eligibility for the Family Caregiver Program. Specifically, within 2 years of the VA Secretary certifying the IT system for the Family Caregiver Program, VHA is to expand program eligibility to caregivers of veterans with a serious injury incurred or aggravated in the line of duty on or before May 7, 1975 or on or after September 11, 2001. Two years after this initial expansion of eligibility, VHA is to further expand program eligibility to include any veteran with a serious injury incurred or aggravated in the line of duty and in need of personal care services as specified in the statute. The Caregiver Support Program Office policy requires every VAMC to have at least one full-time CSC to administer the program. The policy also requires VAMCs to have an eligibility determination process, but does not specify staffing requirements for that process beyond stating that “appropriate” providers should be involved. This broad guidance provides VAMCs with flexibility in determining which providers to include in the eligibility determination process. We found that each of the four VAMCs we visited staff their Family Caregiver Program differently, including both the staff assigned to the program as well as other VAMC staff assisting the program as a collateral duty. While all four VAMCs had at least one CSC on staff, as required, other staff assigned to the program varied and included administrative staff, a non-CSC social worker, and non-CSC registered nurses. Furthermore, the differences we identified with VAMC staff assisting the program as a collateral duty included staff that assist with clinical eligibility determinations as well as staff that assist with other program requirements. Specifically, each of the four VAMCs had assembled their own clinical eligibility teams, which varied in composition and could include physicians, therapists, or mental health professionals. Other variations with staff assisting the program included three VAMCs that utilized members of the Home Based Primary Care team to assist with initial home visits, quarterly contacts, and annual home visits, and a VAMC that used physicians to assist the program with assigning stipend tier levels (see table 1). We found that VHA’s Caregiver Support Program Office does not have complete and accurate staffing information for the Family Caregiver Program. First, the Caregiver Support Program Office does not have complete information on all staff supporting the program. The office only tracks staff funded by the Caregiver Support Program Office, but does not track program staff that are VAMC-funded or other VAMC staff that assist the program as a collateral duty. For example, one site we visited had a VAMC-funded nurse that conducted quarterly contacts and home visits, but this nurse was not being tracked by the program office. Similarly, the program office was not tracking the time and resources related to VAMCs’ clinical eligibility team members. At each of the four VAMCs we visited, members of the clinical eligibility teams dedicated between 3 and 12 hours a month preparing for and attending the eligibility meetings. Furthermore, although the program’s VISN lead officials collect data on the Caregiver Support Program Office funded positions at each VAMC annually at a minimum and submit these data to the Caregiver Support Program Office, there is no documented process to validate the data’s accuracy. VHA employs a process that relies on VISN lead officials collecting data from facilities, and as a result, the overall accuracy of the data depends on the accuracy of the data VAMCs report. Based on our review of the staffing data, we identified discrepancies between the Caregiver Support Program Office’s staffing data for the first quarter of fiscal year 2019 and the number of staff we observed at all four VAMCs we visited. At two VAMCs, the number of CSCs that the Caregiver Support Program Office reported was higher than what we found. Caregiver Support Program Office officials said that these staffing discrepancies could be due to vacant positions. However, officials at the two VAMCs did not indicate that they had vacant positions at the time of our site visits. The third VAMC had a part-time registered nurse staffed to the program that was not included in the Caregiver Support Program Office’s staffing data even though this position was funded by VHA. Caregiver Support Program Office officials could not provide a reason for this discrepancy. The fourth VAMC had an administrative staff member funded by the Caregiver Support Program Office that was not included in the staffing data. Additionally, the Caregiver Support Program Office does not know the exact number of CSCs assigned to the program. The program office funds CSC positions, which can be filled by registered nurses or social workers. The program office also funds registered nurses who are not CSCs. However, the program office’s staffing data does not distinguish between the two types of registered nurse positions because they do not currently have the capability to collect such staffing details. As a result, Caregiver Support Program Office officials told us they could not identify registered nurses who are CSCs from other registered nurses assigned to the program. Officials reported that they are working on finding a way to collect details on the types of registered nurse positions. Although Caregiver Support Program Office officials said that they are taking steps to collect more information about the staff involved in supporting the Family Caregiver Program to prepare for the MISSION Act expansion, these efforts do not fully address the problems with data completeness and accuracy that we identified. Officials said that they are starting to collect data on the Family Caregiver Program staff more frequently. Specifically, the program plans to collect information on Caregiver Support Program Office funded staff at each VAMC from the VISN lead officials quarterly instead of annually, to align with how other national programs collect such data. During the course of this review officials said they had begun working on updating the method they use to collect staffing data. Caregiver Support Program Office officials said that this revised data collection instrument will include mandatory fields and data entry rules to ensure that the data reported are more consistent. However, officials did not provide any timelines for when they will begin using the updated method. Program officials have begun to develop a staffing model in anticipation of future program growth when eligibility expands to include pre-9/11 veterans. To create the staffing model, officials are identifying current program staff at the VHA, VISN, and VAMC levels and the tasks these staff perform. However, officials indicated that the model will use Caregiver Support Program Office staffing data because those are the only staffing data available for the program. As a result, VAMC-funded staff and collateral staff will not be included. Consequently, the completeness of the staffing model will be compromised and the current and future staffing resources identified by the model may not accurately estimate the program’s needs. The lack of complete and accurate staffing data for the Family Caregiver Program is inconsistent with federal internal control standards that require management to use quality information to achieve its objectives. Without complete and accurate information about the total number and types of staff that support the program, VHA does not know whether the program’s staffing approach and available resources are sufficient to meet the program’s requirements as well as the needs of participating caregivers and veterans. Furthermore, without complete and accurate staffing data, it is unclear how the Caregiver Support Program Office will develop projections of the staff that will be needed to enroll and support additional caregivers and veterans when the Family Caregiver Program’s eligibility is expanded as required by the MISSION Act. Within VHA, the Caregiver Support Program Office monitors the timeliness of VAMCs’ processing of applications for the Family Caregiver Program. Specifically, Caregiver Support Program Office officials told us that they review a monthly report from CAT. These reports show the number of applications in process at each VAMC and how long they have been in process. Officials also said that they share this information with VISN lead officials each month. However, since the inception of the program, VAMCs have had difficulty meeting VHA’s requirement to review applications within 90 days. Our analysis of CAT data found that about 68 percent of the 17,576 applications submitted from October 2017 through September 2018 were reviewed within 90 days. In January 2019, a memorandum was issued that required all VAMCs to develop action plans to address application processing delays beyond 90 days. Further, any VAMCs with more than 10 applications beyond 120 days or any exceeding 365 days were required to submit their action plans to the Caregiver Support Program Office. As a result of this memo, 11 VAMCs have submitted action plans. Caregiver Support Program Office officials said that they have assigned staff to monitor the action plans and have discussed the plans with the leadership of the VISNs that oversee these VAMCs. Additionally, in February 2019, VHA established a national level performance metric to measure application processing timeliness for the program that will be updated on a monthly basis, according to Caregiver Support Program Office officials. VHA’s goal is for 90 percent of Family Caregiver Program applications submitted in fiscal year 2019 to be processed within 90 days. The May 2019 report from CAT shows that 94 percent of the 1,246 current applications have been in process 90 days or less. VHA’s Caregiver Support Program Office lacks system-wide data from CAT or other sources on the completion of VAMCs’ required quarterly contacts and annual home visits conducted with caregivers and veterans in the Family Caregiver Program. Although these contacts and visits are supposed to be documented in CAT, the system has limited reporting capabilities. As a result, Caregiver Support Program Office officials are unable to obtain system-wide data that would allow them to monitor VAMCs’ completion of these requirements. Furthermore, officials could not readily provide these data for the four VAMCs we visited because doing so would have required them to manually review each veteran’s record (921 records across the four VAMCs). Given CAT’s reporting limitations, some VAMC and VISN lead officials we spoke with indicated that they have developed their own methods for tracking contacts and visits at the facility or regional levels. For example, officials at one VAMC told us they had developed a spreadsheet for the purpose of tracking quarterly contacts and annual home visits. Further, the VISN lead officials from one VISN told us that their VAMCs report information on their ability to schedule and complete contacts and visits on a monthly basis. The program office does not collect these data from the VAMCs or VISNs. The Caregiver Support Program Office has been able to collect limited information on the extent to which quarterly contacts and annual home visits are completed through 1) bi-annual audits of a sample of Family Caregiver Program participant records that are rolled into in an annual report and 2) site visits to select VAMCs. Caregiver Support Program Office officials told us that the audits of program participants’ records serve as their main source of information on the completion of required contacts and visits. However, the focus of the audits vary each year, which means that officials cannot monitor trends in performance over time because the information is not comparable year-to-year. For example, in fiscal years 2017 and 2018, the focus was on the records of caregivers and veterans who had been discharged from the program, and in fiscal year 2016, the focus was on newly approved caregivers and veterans. In addition, because the audits are focused on a random sample of individual participants’ records, they do not provide the Caregiver Support Program Office with information to determine whether individual VAMCs are meeting these requirements. Program office officials also told us that their site visits to VAMCs include a review of the processes for required quarterly contacts and annual home visits. As of January 2019, program office officials had conducted 16 site visits since fiscal year 2016— representing about 11 percent of VAMCs. Caregiver Support Program Office officials also report that they intend to develop a site visit plan as part of MISSION Act implementation planning. Without system-wide data on VAMCs’ monitoring efforts, the program office does not know whether contacts and visits are being completed as required or whether VAMCs may need more staff to conduct them. The VAMC officials we spoke with acknowledged that their ability to complete quarterly contacts and annual home visits was dependent upon having enough staff. For example, one VAMC official reported that its facility did not complete an entire quarter of contacts and visits to caregivers and veterans because they did not have sufficient staffing resources. Similarly, a VISN lead official said that the VAMCs in its network also have had trouble meeting monitoring requirements due to insufficient staff. The lack of system-wide data on VAMCs’ completion of required contacts and visits is inconsistent with federal internal control standards that require management to use quality information to achieve their objectives. Furthermore, without these data, the Caregiver Support Program Office is also limited in its ability to estimate the additional staff that will be needed to conduct these contacts and visits once the program’s eligibility expands. VHA and OIT have worked jointly over the last four years to both fix and replace the existing Family Caregiver Program IT system, CAT, but these efforts have not led to the implementation of an IT system that fully supports the needs of the program. The VA MISSION Act included provisions that directed the department to implement an IT system for the Family Caregiver Program by October 1, 2018 and required certification of the system from the VA Secretary by October 1, 2019. However, the department reported to congressional committees in October 2018 that meeting the system implementation deadline of the VA MISSION Act was not feasible. Consequently, that deadline has not yet been met. Specifically, VHA and OIT undertook two related efforts beginning in 2015: CAT Rescue was initiated in July 2015 as a short-term project intended to improve both the quality of CAT’s data and the system’s reliability and security. However, schedule delays and significant defects identified during system testing contributed to CAT Rescue’s termination in April 2018. According to OIT officials, the department spent about $2.86 million on CAT Rescue. However, the project did not deliver viable software improvements. Caregivers Tool (CareT), a companion project to CAT Rescue, was initiated in September 2015 and was intended to produce a replacement for CAT. The project was to develop and deliver a replacement system with expanded capabilities, such as easier caregiver application submission and enhanced caregiver program analysis capabilities. However, the CareT acquisition depended on CAT Rescue, which did not deliver the needed data improvements. When CAT Rescue was terminated, data improvement and migration activities that were previously part of CAT Rescue were moved to the CareT project and contract extensions were necessary to allow more time for system development and testing in relation to these expanded requirements. Subsequently, OIT and VHA Caregiver Support Program Office officials acknowledged that development delays and the number and critical nature of system defects identified during user acceptance testing had led to the VHA Caregiver Support Program Office’s loss of confidence in CareT as a viable replacement for CAT. As a result, VA suspended the CareT acquisition in January 2019 to assess the way forward. Ultimately, work on CareT ended in late February 2019. According to OIT officials, the department spent about $8.11 million on CareT between 2015 and 2019. However, no fully functioning system replacement was delivered as intended. VA commissioned two independent assessments that examined issues impacting the CAT Rescue and CareT projects. These assessments, completed by Digital Service at VA and the MITRE Corporation in early 2019, cited a number of deficiencies that likely contributed to the termination of CAT Rescue and impacted the ability of CareT to successfully deliver new system capabilities. For example, the assessments identified deficiencies in the following areas: Requirements management: The department did not effectively implement a process for requirements development and prioritization. As a result, OIT, program office staff, and the development contractors did not have a shared understanding of how the system was to perform. In addition, the requirements identified may have been overly complex and insufficient to facilitate IT development. Efforts to elaborate on the requirements over the course of the projects were not consistent and led to delays. Further, significant defects identified during testing were not effectively prioritized and requirements remained unmet. Leadership: CAT Rescue and CareT did not have stable leadership and experienced staff throughout the department’s efforts to address issues with the program. Specifically, there was a lack of sustained leadership or a product owner needed to create and enforce a technical vision across contractors and the department. Without such leadership, there was a lack of effective governance and shared accountability across VHA, OIT, and the development contractor. According to the assessments, these deficiencies, among others, have resulted in VA’s inability to successfully deliver IT improvements as planned. We have previously reported that successfully overcoming challenges in areas such as those identified in the independent assessments of CAT Rescue and CareT is critical to increasing an agency’s odds for delivering an IT system acquisition. With the loss of confidence in CareT as a viable solution and the subsequent results of the independent assessments, VA has redirected its efforts for a third time and initiated a new project, referred to as the Caregiver Record Management Application (CARMA), in March 2019. Specifically, CARMA is focused on acquiring a solution to CAT using a commercial product that is to be configured to fit the needs of the Family Caregiver Program and support the program’s expansion. The first CARMA release, planned for late October 2019, is intended to replace CAT and improve program reporting. According to program officials, this release is expected to include expanded capabilities needed to develop system-wide reports on the completion of the required quarterly contacts and annual home visits. The second release, planned for January 2020, is intended to refine initial functionality and improve stipend processing capabilities. Additional product releases are expected at least through the summer of 2020 to incorporate new capabilities, such as online application submissions for veterans and the ability to connect to existing VA systems that manage veteran and caregiver identity and relationship management. However, it is unclear what additional work may be necessary to accommodate the expansion of the Family Caregiver Program given that the department is only in the early stages of planning. Further, the department has not yet established a target date for certifying CARMA. According to OIT officials, the cost for CARMA is estimated to be between $5.7 million and $6.3 million, but additional costs for licensing and modifications to legacy systems are also expected. As of June 2019, OIT and VHA with assistance from the Digital Service at VA had taken steps to identify key project stakeholders, estimate costs, establish a timeline, and compile the initial set of requirements for implementing the first release. The department had also identified a Product Manager, who is to be responsible for coordinating efforts between OIT and VHA. According to officials from the Digital Service at VA, the CARMA project plans to use a better, more agile approach for managing requirements. In addition, the staff asserted that the department has established and filled the new Product Manager position, which is intended to improve project leadership. Despite these actions, VA has not yet demonstrated results to show whether these changes will be sufficiently effective to overcome the issues that contributed to the failure of both CAT Rescue and CareT. It is also not yet certain when VA will successfully implement and certify its IT system as required by the VA MISSION Act. Further, because the expansion of the program is contingent on the certification by the VA Secretary that the IT system fully supports the program, continued delays with the IT system will postpone needed assistance for caregivers and veterans who may qualify for these benefits when eligibility requirements are expanded. Thus, it will be important that VA ensure that the actions taken to improve requirements management and leadership of the CARMA project are effectively implemented in order to improve the likelihood that the project will deliver an IT system that fully supports the Family Caregiver Program. As VA prepares for the expansion of the Family Caregiver Program to include caregivers of veterans who served prior to September 11, 2001, it will be important that VA have an informed understanding of the staffing resources needed to support the program. However, we found that VA continues to struggle to have the information and tools needed to effectively monitor the Family Caregiver Program. Since the Family Caregiver Program was implemented in 2011, it has experienced difficulties in meeting program requirements, such as for monitoring program enrollees, potentially impacting the caregivers and veterans it is intended to support. As both we and the VA OIG have reported, determining and ensuring there are sufficient program staff to support the program is one of VHA’s greatest obstacles in meeting program requirements. In particular, VHA’s Caregiver Support Program Office does not have complete and accurate staffing data with which to assess current and future staffing levels because it is not collecting data on all VAMC staff who support the program, and the data that are collected are not validated. The Caregiver Support Program Office is further impeded in its ability to assess whether VAMCs’ staffing levels for the program are adequate because it lacks system-wide data on the completion of periodic contacts and visits with caregivers and veterans. While the new IT system should address this issue, the program office would benefit from having an interim method to collect this information as VA’s previous efforts to fix and replace CAT have not been successful. Until the program office has reliable data for oversight and planning, the difficulties VA has experienced since the Family Caregiver Program was implemented could be further exacerbated when the program’s eligibility expands to include the caregivers of veterans of all eras. We recommend that the Secretary of the VA direct the Under Secretary for Health to take the following actions: Collect complete staffing data for the Family Caregiver Program that includes Caregiver Support Program Office funded staff, VAMC funded staff, and staff that assist the program as a collateral duty at each VAMC. (Recommendation 1) Establish a process to ensure that the Family Caregiver Program staffing data that are collected and reported to the Caregiver Support Program Office are accurate. (Recommendation 2) Identify and use an interim method to collect data from VAMCs on their completion of required quarterly contacts and annual home visits with caregivers and veterans that can be used until a new IT system is implemented. (Recommendation 3) VA provided written comments on a draft of this report, which are reprinted in appendix I. In its written comments, VA concurred with all three recommendations. VA also provided technical comments, which we incorporated as appropriate. With respect to our recommendation on collecting complete staffing data, VA concurred and stated it is in the process of developing a data collection mechanism that will allow for the capture of more specific data about staffing roles and disciplines of Family Caregiver Program staff. VA also concurred with our recommendation to establish a process that ensures that the staffing data collected and reported to the Caregiver Support Program Office are accurate. VA stated that the VISN lead officials will be responsible for reviewing and validating the staffing data submitted and that the data collection mechanism under development for this purpose will have data validation processes in place for its data fields to ensure that the data entered by VISN leads are accurate. VA also concurred with our recommendation that it needs to identify and use an interim method to collect data on the completion of required quarterly contacts and annual home visits that can be used until a new IT system is implemented. In its technical comments, VA noted that data on quarterly contacts and annual home visits with caregivers and veterans are also captured in the Computerized Patient Record System because these visits are considered clinical encounters, which we note in our report. VA further stated that staff should be able to track the workload entered into the Computerized Patient Record System if VAMCs have appropriately set up their IT systems with the designated code for this program. VA asserted that in response to our recommendation it is exploring the feasibility of using data from the Computerized Patient Record System as an interim solution for monitoring the completion of quarterly contacts and annual home visits system-wide. Additionally, VA reiterated that the first release of CARMA, which it plans to release in October 2019, should include the capabilities necessary to develop system-wide reports on the completion of required quarterly contacts and annual home visits. We are sending copies of this report to the Secretary of Veterans Affairs, the appropriate congressional committees, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Sharon M. Silas at (202) 512-7114 or silass@gao.gov or Carol C. Harris at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. In addition to the contacts named above, Bonnie Anderson (Assistant Director), Mark Bird (Assistant Director), Alison Goetsch (Analyst-in- Charge), Emily Loriso, and Jennifer Stavros-Turner made key contributions to this report. Also contributing were Jennie F. Apter, Chris Businsky, Krister Friday, Monica Perez-Nelson, and Ethiene Salgado- Rodriguez. VA Health IT: Use of Acquisition Best Practices Can Improve Efforts to Implement a System to Support the Family Caregiver Program. GAO-19- 581T. Washington, D.C.: May 22, 2019. Veterans Affairs: Addressing IT Management Challenges Is Essential to Effectively Supporting the Department’s Mission. GAO-19-476T. Washington, D.C.: April 2, 2019. VA Health Care: Improvements Needed to Manage Higher-Than- Expected Demand for the Family Caregiver Program. GAO-15-245T. Washington, D.C.: December 3, 2014. VA Health Care: Actions Needed to Address Higher-Than-Expected Demand for the Family Caregiver Program. GAO-14-675. Washington, D.C.: September 18, 2014. Information Technology: Critical Factors Underlying Successful Major Acquisitions. GAO-12-7. Washington, D.C.: October 21, 2011.", "summary": "Since 2011, the VA Family Caregiver Program has provided assistance to caregivers of seriously injured post-9/11 veterans at VAMCs nationwide. However, GAO previously reported that some VAMCs have struggled to manage the program's workload. The VA MISSION Act of 2018 requires the expansion of program eligibility to veterans of all eras contingent upon implementation and certification of a new IT system. The VA MISSION Act included a provision for GAO to review VA's efforts to implement a new IT system. GAO was also asked to examine staffing for the program. This report examines the extent to which VA 1) has established staffing requirements and has data to track program staffing; 2) monitors whether VAMCs are meeting departmental requirements for application review timeliness and required contacts; and 3) has implemented an IT system that fully supports the program. GAO reviewed program documentation and data. GAO also interviewed VHA officials and officials from four VAMCs and their VISNs that varied in their numbers of applications and approved caregivers. GAO also interviewed OIT officials and reviewed documentation related to their efforts to acquire and develop an IT system for the program. Within the Department of Veterans Affairs (VA), the Veterans Health Administration (VHA) has established staffing requirements for its Program of Comprehensive Assistance for Family Caregivers (Family Caregiver Program) that allow for variation, but its staffing data are not complete or accurate. VHA requires its local VA medical centers (VAMC) to have at least one Caregiver Support Coordinator to manage the program. Otherwise, VAMCs have flexibility in determining the additional staff needed. VHA's Caregiver Support Program Office funds most Family Caregiver Program staff at VAMCs. VAMCs also may fund additional program staff or have other VAMC staff assist the program as a collateral duty, but GAO found that the program office only tracks the staff it has funded. GAO also identified discrepancies between the number of staff it observed at selected VAMCs and the program office's staffing data. Without complete and accurate staffing data, the program office does not have reliable information about the program's current staffing levels, which could hamper its efforts to project needed staff when the program's eligibility is expanded. The program office routinely monitors VAMCs' performance in meeting departmental timeliness requirements for reviewing enrollment applications for the Family Caregiver Program. However, it is not able to monitor whether VAMCs are completing required quarterly contacts and annual home visits to enrolled caregivers and veterans. The Family Caregiver Program's current information technology (IT) system—the Caregiver Application Tracker (CAT)—has limited reporting capabilities and cannot provide system-wide data on the completion of these contacts and visits even though this information is documented in CAT. GAO found that some VAMCs and the regional Veterans Integrated Service Networks (VISNs) that oversee them use spreadsheets to track the completion of these requirements, but the program office does not collect these data. Without system-wide data on contacts and visits, the program office is limited in its ability to monitor and identify when VAMCs may need additional staff to meet these requirements, including once the program's eligibility is expanded. VA has yet to implement a new IT system that fully supports the Family Caregiver Program as required by the VA MISSION Act. VHA and the Office of Information and Technology (OIT) have been working jointly on projects since 2015 to improve and replace CAT. However, two of these projects were terminated without delivering viable software improvements or a replacement system. According to two independent assessments, these prior efforts lacked both effective leadership and implementation of the processes needed for requirements management. VA has asserted that its third project, in which OIT and VHA have begun to acquire and implement a commercial product to replace CAT, will take steps to avoid the issues that have impacted its past efforts. However, the initial replacement for CAT is not expected until late October 2019. Further, despite this initial deployment and additional releases expected through the summer of 2020, the department has not yet fully committed to a date by which it will certify that the new IT system fully supports the program. Until the system is implemented and certified, the expansion of eligibility for the Family Caregiver Program will be delayed. GAO is making three recommendations to VA to collect complete staffing data, establish a process to ensure the data are accurate, and establish an interim method for collecting system-wide data on required contacts and visits. VA concurred with all three recommendations.", "document_type": "gao"}
{"report": "IAEA’s policy-making bodies include the Board of Governors, which consists of 35 member states, including the United States as a de-facto permanent member; and the General Conference, which consists of all 171 member states of IAEA. The agency’s staff, led by the Director General, is referred to as the Secretariat and is organized into six departments that implement programs approved by the Board of Governors and the General Conference. The Division of Nuclear Security, within the Department of Nuclear Safety and Security, implements the nuclear security program. Figure 1 shows the position of DNS within the agency. The agency’s other departments include the Department of Safeguards, which carries out technical measures and activities by which IAEA seeks to verify that nuclear material subject to safeguards is not diverted to nuclear weapons or other proscribed purposes; and the Department of Technical Cooperation, which provides nuclear technologies and expertise to member states. In addition to the departments, the agency has offices that report to the Director General, such as the Office of Legal Affairs. IAEA’s statute is the foundation of the agency’s dual mission of promoting the peaceful uses of nuclear energy and verifying through safeguards that nuclear technologies and materials are used for peaceful purposes and not diverted to nuclear weapons. Nuclear security is not an explicit part of this broader mission, but the agency has identified several of its statutory authorities as underpinning its nuclear security role. For example, the statute authorizes the agency to exchange scientific and technical information on peaceful uses of atomic energy, which IAEA does under its nuclear security program. In addition, a number of international treaties establish a nuclear security role for the agency, including: The Convention on the Physical Protection of Nuclear Material (CPPNM) and its 2005 amendment. This convention originally addressed the security of nuclear materials in international transport. A 2005 amendment, which entered into force in 2016, requires parties to establish, implement, and maintain a physical protection regime for nuclear materials and facilities in domestic use, storage, and transport. The amendment encourages states to consult with IAEA to obtain guidance on the design, maintenance, and improvement of their national systems of physical protection of nuclear material. International Convention for the Suppression of Acts of Nuclear Terrorism. This convention refers to IAEA as a source of guidance to States parties on measures for security of nuclear materials and charges IAEA with transmitting information to States parties, following an offense under the convention, on the disposition or retention of radioactive material, devices, or facilities taken control of during the response. In addition, United Nations Security Council Resolution 1540 calls upon states to refrain from supporting by any means non-state actors that attempt to, among other activities, acquire, use, or transfer nuclear, chemical, or biological weapons and their delivery systems. The resolution also calls upon states to engage in activities similar to those described in IAEA’s Nuclear Security Plan. For example, the resolution calls on states to take and enforce effective measures to establish domestic controls to prevent the proliferation of nuclear weapons, including physical protection measures; IAEA provides guidance and other support for applying such measures to civilian nuclear materials. The resolution also calls for measures to prevent illicit trafficking, to establish effective export controls, and to renew and fulfill commitments to multilateral cooperation in particular within the framework of the IAEA. IAEA funds its programs primarily through (1) its regular budget, for which all member countries are assessed an annual contribution, and (2) extra-budgetary cash contributions, which are voluntary. In addition, IAEA has a Technical Cooperation Fund—generally supported through voluntary annual contributions of member states—to be used for technical cooperation projects. The State Department coordinates the United States’ policy with and financial contributions to IAEA and is the lead U.S. agency for interacting with IAEA. In 2018, IAEA’s total regular budget was $437.9 million, and approximately $103.5 million was unfunded (to be funded through extra- budgetary contributions). In 2018, the Division of Nuclear Security’s regular budget was approximately $6.9 million, and $25.2 million was unfunded (to be funded through extra-budgetary contributions). The Nuclear Security Fund, established after the September 2001 terrorist attacks, holds the extra-budgetary funding for most of IAEA’s nuclear security activities. Figure 2 shows the levels of regular and extra- budgetary funding for DNS over the last three biennial budget cycles, from 2014 to 2019. According to IAEA officials, the agency operates under substantial budget constraints as a number of member states advocate for zero-nominal- growth budgets. This has generally caused IAEA’s programs to operate under minimal growth in their regular budgets from year to year and to seek efficiencies on an ongoing basis. Extra-budgetary contributions are not subject to these constraints. As part of an initiative to secure all vulnerable nuclear material around the world, the United States hosted 47 world leaders in Washington, D.C., for a Nuclear Security Summit in 2010. The summit organizers invited a range of participants, taking into account the scale of their nuclear energy programs and countries’ access to weapons-usable materials. Additional summits were held in Seoul, South Korea, in 2012; the Hague, the Netherlands, in 2014; and again in Washington, D.C. in 2016. The Nuclear Security Summits brought heads of state together to discuss and bring high-level international attention to nuclear security issues. These summits led to, among other things, the removal or elimination of nuclear material from civilian facilities across the globe, ratification and implementation of treaties, conversion of reactors to operate on low- enriched uranium, and the strengthening of regulations. Summit participants issued an Action Plan in Support of the IAEA during the final summit in 2016 to document their commitments to IAEA’s nuclear security mission. Commitments in the Action Plan in Support of the IAEA included recognizing the leading role of the agency for coordinating multilateral nuclear security activities as well as committing high-level support for the IAEA’s nuclear security activities and advocacy for IAEA’s coordination role and provision of guidance. IAEA’s nuclear security activities are conducted primarily under its nuclear security program, which consists of four subprograms. Under these subprograms, IAEA carries out a wide range of nuclear security activities, including developing and promoting the use of nuclear security guidance documents, providing assistance to member states, and developing training programs. IAEA also coordinates international nuclear security efforts. IAEA’s nuclear security activities are conducted primarily under the agency’s nuclear security program, which consists of four subprograms: Nuclear Security of Materials and Facilities. This subprogram covers the security of nuclear and other radioactive material and associated facilities and activities including transport. Nuclear Security of Materials Outside of Regulatory Control. This subprogram covers detection of criminal or intentional unauthorized acts involving nuclear or radioactive material and responding to nuclear events. Information Management. This subprogram is responsible for establishing and maintaining systems to collect and analyze nuclear security information. Program Development and International Cooperation. This subprogram covers international nuclear security coordination and provides education and training programs. It also manages donor relations and the Nuclear Security Fund. The Nuclear Security Program is implemented by IAEA’s Division of Nuclear Security (DNS), which is structured into four sections that correspond to the four subprograms. Figure 3 shows the projects carried out by each section. Other IAEA offices coordinate with DNS to carry out the agency’s nuclear security activities. For example, IAEA’s Department of Nuclear Energy collaborates with DNS to convert reactors to run on low-enriched uranium, return nuclear materials resulting from the conversion to the country of origin, and assist with the disposition of disused radioactive sources. The agency’s Office of Legal Affairs supports DNS by promoting universal adoption of the Convention on the Physical Protection of Nuclear Material and its 2005 amendment and helping member states with legal and regulatory understanding of the convention and drafting review. Under the four subprograms, IAEA conducts a broad range of nuclear security activities including (1) developing nuclear security guidance; (2) providing assistance to member states in areas such as establishing legal, regulatory, and technical nuclear security infrastructure, and converting reactors to operate on non-weapons usable materials; (3) providing training and education; and (4) coordinating international nuclear security efforts. IAEA develops nuclear security guidance documents and encourages member states to adopt and implement the guidance to improve their nuclear security regimes. IAEA’s Nuclear Security Guidance Committee, established by the Director General in 2012, makes recommendations to IAEA on what nuclear security guidance to develop and approves guidance publications. The Nuclear Security Guidance Committee is open to all member states. DNS’s four sections contribute to the development of guidance. For example, the Information Management section develops guidance relating to computer security at nuclear facilities, and the Nuclear Security of Materials and Associated Facilities section develops guidance in the area of physical protection of nuclear materials and facilities. DNS develops two main sets of guidance documents: the Nuclear Security Series and Codes of Conduct. The Nuclear Security Series, launched in 2006, is continuously updated by IAEA in cooperation with experts from member states. The series comprises four broad categories of publications: Nuclear Security Fundamentals, which establish the fundamental objectives and essential elements of states’ national nuclear security regimes. Recommendations, which set out measures that states should take to achieve and maintain effective regimes. Implementing Guides, which provide guidance on implementing security measures. Technical Guidance, which provides detailed guidance on specific methodologies and techniques for implementing security measures. Within each category, there are specific guidance documents, such as “Establishing the Nuclear Security Infrastructure for a Nuclear Power Programme” and “Nuclear Security Systems and Measures for Major Public Events.” The publications’ principal users are regulatory bodies for nuclear and radiation security and other relevant member-state authorities, such as those involved in law enforcement and forensics, border control and customs, and intelligence gathering. Other users include international organizations with responsibilities relevant to nuclear security; organizations that design, manufacture, and operate nuclear facilities; and organizations involved in the use of radiation related technologies. Another set of publications, the Codes of Conduct, are meant to serve as guidance to states for the development and harmonization of policies, laws and regulations. They include a Code of Conduct on the Safety and Security of Radioactive Sources. IAEA provides a variety of nuclear security assistance, which member states may request through the Integrated Nuclear Security Support Plan (INSSP) process, in which DNS works with member states to jointly conduct a comprehensive and systematic review of their nuclear security regimes and identify potential areas for improvement. DNS works with member states that request an INSSP to develop implementation strategies, based on the nuclear security needs identified, for IAEA or potential donors to provide assistance to the state. The INSSPs serve as input for the work plans of each DNS section. Member states may also request ad hoc assistance outside this process. IAEA’s nuclear security assistance includes helping member states establish legal, regulatory, and technical infrastructure to secure nuclear materials and facilities, and helping states detect and respond to “materials out of regulatory control”—material present in sufficient quantity that it should be under regulatory control but is not. IAEA may help to identify the need for assistance through advisory missions and peer reviews, such as International Physical Protection Advisory Service missions. These missions assist countries in strengthening their national civilian nuclear security regimes by providing (1) guidance on the protection of nuclear material and facilities, as well as of sealed radioactive sources and other radioactive material; (2) best practices in nuclear security; and (3) peer advice on implementing international agreements related to physical protection of nuclear material and facilities. Since 1996, IAEA has conducted 84 International Physical Protection Advisory Service missions in 50 countries. In addition, IAEA conducts International Nuclear Security Advisory Service missions to help member states establish effective nuclear security regimes that address nuclear and other radioactive “material out of regulatory control.” According to IAEA officials, in 2016 the agency suspended International Nuclear Security Advisory Service missions while DNS updated the supporting guidance, but it intends to restart such missions in 2019. IAEA also assists member states hosting major public events in strengthening nuclear security measures before and during the events. Assistance provided for major public events includes coordination meetings, workshops, and training on the use of detection equipment. The agency reported that, from July 2017 through June 2018, it assisted states with preparing for at least seven major public events, such as the 29th Southeast Asian Games in Malaysia in August 2017 and the G20 Buenos Aires Summit in Argentina in November 2018. In addition, IAEA assists with converting reactors to operate on low- enriched uranium rather than highly enriched uranium and contributes to the design of reactor cores that operate on low-enriched uranium. IAEA also assists with the repatriation of fissile and radioactive material from countries that no longer require or cannot adequately secure those materials to more secure storage in other countries. As previously noted, IAEA’s Department of Nuclear Energy assists DNS with converting reactors to run on low-enriched uranium. The Department of Nuclear Energy also works with DNS on management strategies for disused radioactive sources. With regard to radioactive material, IAEA reported that, from July 2017 through June 2018, it helped repatriate three highly radioactive materials from Lebanon to Canada and 27 such materials from South America to Germany and the United States. IAEA conducts several types of nuclear security training and education activities to support member state capacity building, including workshops and exercises. The agency reported that, from July 2017 through June 2018, it provided in-person training for more than 2,400 participants from 149 member states on subjects including physical protection of nuclear material and computer security. IAEA has also developed e-learning courses to make training more accessible. In addition, IAEA supports member states in developing Nuclear Security Support Centers. The purpose of these centers is to effectively develop nuclear security knowledge and associated technical skills in states to promote the long term sustainability and effectiveness of nuclear security in those states. The agency also supports the International Nuclear Security Education Network, a partnership through which IAEA, educational and research institutions, and other stakeholders cooperate to promote nuclear security education. This network connects 170 institutions from 62 member states to assist them in establishing and enhancing nuclear security education. Network members collaborate in areas such as the development of peer- reviewed textbooks, instructional material, computer-based teaching tools, and exercises and materials for laboratory work; faculty development in different areas of nuclear security; joint research and development activities to share scientific knowledge and infrastructure; and quality assurance. IAEA coordinates international nuclear security efforts through activities such as hosting information exchange meetings, organizing events and conferences, and promoting universal adoption of international legal instruments. Twice a year, IAEA hosts information exchange meetings to coordinate nuclear security activities with other organizations, such as the Global Initiative to Combat Nuclear Terrorism. IAEA reported hosting information exchange meetings in November 2017 and April 2018. The agency organizes a range of events and conferences, including the International Conference on Nuclear Security, which brings together ministerial-level representation to discuss important issues related to nuclear security. IAEA’s activities to promote the universal adoption of international agreements relevant to nuclear security—such as the Convention on the Physical Protection of Nuclear Material and its 2005 amendment—include working with states directly, speaking at conferences, and offering model legislation for states to follow. In addition, IAEA manages the Incident and Trafficking Database, which catalogues reports by participating states about details of thefts, losses, and other unauthorized activities and events involving nuclear and other radioactive material out of regulatory control. The details of such incidents are accessible to participating states, with limited information accessible to other UN-affiliated organizations. IAEA plans its nuclear security work through a range of documents, including a biennial Programme and Budget (P&B). However, IAEA does not prioritize its nuclear security activities. In addition, IAEA’s performance measures have limitations, and agency reports on nuclear security do not consistently include performance information. IAEA has two primary planning documents for nuclear security: Nuclear Security Plan. This 4-year planning document describes the nuclear security program’s tasks and outputs by project. The Nuclear Security Plan, which is approved by the Board of Governors, identifies broad priority areas, such as physical protection and nuclear security detection architecture and response. Programme and Budget (P&B). This biennial document, which is approved by the General Conference, identifies current IAEA program funding levels and future funding needs. The P&B also lays out objectives and associated outcomes and performance measures for the entire agency, including the nuclear security program and its subprograms. Figure 4 shows the objectives for the nuclear security program. In addition, the P&B identifies planned outputs for each project under the nuclear security subprograms. However, these documents contain only broad statements on prioritizing activities, providing limited guidance to DNS. Specifically, the Nuclear Security Plan calls for the agency to carry out its nuclear security activities in a prioritized manner with available resources, without further guidance about how to prioritize activities. Similarly, the P&B establishes two broad criteria for prioritization: 1) completion and maintenance of the universally applicable Nuclear Security Series recommendations and guidance, and provision of assessment and evaluation services at the request of member states, and; 2) the provision, upon request, of assistance based on an analysis of needs, including those identified through INSSPs. These criteria for prioritization are broad, effectively including almost all of the DNS’s activities. When we compared these criteria to DNS’s projects described in the 2018-2019 P&B, 12 of 13 projects aligned with at least one criterion. For example, one project under the Information Management section is to develop and implement INSSPs and a voluntary self-assessment tool for member states to use. This project aligns with the second criterion—the provision of assistance, including assistance identified through the INSSPs—because developing INSSPs helps the agency provide assistance to member states. DNS officials said that they use the criteria in the P&B as broad expectations set by member states for the nuclear security program, noting that they do not prioritize among activities because member states do not agree on priorities. Instead of actively prioritizing activities, DNS officials said they respond to requests from member states as those requests come in and to the extent that resources are available, taking into account conditions on funding. According to leading practices identified in the Project Management Institute’s The Standard for Program Management, organizations’ resource management plans should describe the guidelines for making decisions about priorities for using program resources and resolving resource conflicts. However, DNS does not have guidelines for prioritizing activities; there is no guidance in the Nuclear Security Plan, and the criteria for prioritization in the P&B are too broad for division officials to distinguish among competing needs. Such detailed guidelines would help DNS ensure it is appropriately targeting its limited program resources. IAEA has established several performance measures for its nuclear security program and subprograms, but these measures do not fully align with leading practices. IAEA issues several reports on the results of the nuclear security program, but these reports contain only some of the agency’s performance measures. IAEA has established four high-level performance measures in the P&B that it uses to determine progress toward the nuclear security program’s goals: (1) the number of member states requesting and receiving assistance through INSSPs, (2) the number of member states establishing or improving nuclear security measures based on advice from the agency, (3) the number of activities duplicated by other initiatives, and (4) the number of activities carried out in conjunction with the agency. In addition, the P&B identifies from four to six performance measures for each nuclear security subprogram. For example, the number of states requesting assistance or participating in IAEA activities to improve computer and information security capabilities is a performance measure for the Information Management subprogram. According to IAEA’s P&B, the agency follows a “results-based management” approach, which is driven by articulating desired results and measuring actual performance against those results. The P&B states that key elements of this approach include establishing program baselines and targets and measuring actual performance against these baselines and targets to determine whether the program is achieving its planned outcomes. We reviewed IAEA’s nuclear security program performance measures against four leading practices for performance management we have previously reported on: (1) linking performance measures to the offices responsible for implementing the programs, (2) limiting measures to the vital few, (3) determining whether performance measures for the defined objectives are appropriate for evaluating the agency’s performance in achieving those objectives, meaning that measures and processes for measuring performance align with the objective, and (4) measuring performance against baselines. The practice of measuring performance against baselines is also consistent with IAEA’s results-based management approach. Table 1 shows the extent to which DNS’s performance measures meet leading practices. We found that DNS’s performance measures fully met two of the four leading practices. First, IAEA’s nuclear security program performance measures linked to the offices responsible for implementing them, as DNS’s four sections are responsible for implementing the four subprograms of the corresponding name. For instance, a performance measure linked to the Information Management subprogram within DNS is the number of states requesting assistance or participating in IAEA activities to improve computer and information security capabilities, and the Information Management section implements the associated subprogram, whose projects include information and computer security. Second, IAEA’s nuclear security program performance measures are limited to the vital few; as discussed above, there are four high-level measures for the program and between four and six measures for each subprogram. We found that IAEA’s performance measures partially met the third of the four leading practices. Specifically, they were generally appropriate for evaluating their corresponding outcomes and objectives. The program objective of playing a central role and enhancing international cooperation in nuclear security fully aligned with its associated outcome of improved global coordination and cooperation in supporting national efforts to improve nuclear security. Also, the associated measures by which IAEA assesses progress toward this outcome—the number of activities duplicated by others and the number of activities carried out in conjunction with IAEA—fully aligned with the outcome and objective. However, for the other two program objectives, outcomes and measures partially aligned with the objectives. For example, one of the nuclear security program’s objectives is contributing to global nuclear security efforts by establishing guidance and providing for its use through advisory services and capacity building; there is a performance measure related to advisory services, but no measure related to guidance. We found that IAEA’s nuclear security program performance measures did not meet the fourth leading practice, in that they did not include baselines or targets. For example, the performance measure regarding the number of states that have established or improved national nuclear security measures and systems on the basis of advice from the agency does not include a baseline of the number of states that already have established effective nuclear security measures. The measure also does not include a target for the number of states that should establish or improve nuclear security measures. Without established baselines or targets for each performance measure, IAEA’s ability to demonstrate results for its nuclear security program is limited. DNS officials acknowledged that the performance measures for the nuclear security program and subprograms do not have targets or baselines. They said that this is deliberate, based on nuclear security being a national responsibility and the limitations of IAEA’s nuclear security mandate. However, many of the performance measures for the nuclear security program and subprograms are focused on activities the agency carries out, for which DNS can develop targets and baselines; they are not focused on activities of member states. DNS officials also said that the division struggles to develop measures because the nuclear security environment—for example, threats to computer security—is continually evolving. However, many of these measures—such as adherence to the Convention on the Physical Protection of Nuclear Material—are independent of the security environment, and uncertainty should not prevent programs from developing measures to track their performance. By developing baselines and measurable targets to demonstrate results, DNS can more effectively monitor and assess the performance of its Nuclear Security Program. IAEA issues four sets of reports that provide information on its nuclear security program to member states, key stakeholders, and the public, including: Nuclear Security Report. This annual report, developed by DNS, describes the nuclear security program’s major achievements and expenditures of the prior year, as well as goals for the following year. Program Performance Report. This internal, agency-wide report describes progress in implementing all of the agency’s programs and identifies the resources used for each program in a given year. IAEA Annual Report. This report provides a high-level overview of the agency’s accomplishments and includes a section on the nuclear security program. Individual reports for each donor. These reports detail how DNS uses extra-budgetary contributions from each donor country (or government agency) in a given year; these reports are not shared with other countries or agencies. We have previously reported that program managers should communicate necessary quality information so that both internal and external parties can help the program achieve its objectives. Communicating necessary quality information through reporting is consistent with IAEA’s results-based management approach, according to which results-based reports help the organization, stakeholders, and funders to better understand the impact of a given program or project. We have also found that completeness is an element of quality reporting; completeness entails reporting on every performance goal and measure. In May 2013, we recommended that State work with IAEA and its member states to systematically report on the results of the agency’s performance measures. IAEA has subsequently taken steps to improve reporting, such as aligning the Nuclear Security Report with the P&B. In 2018, DNS restructured the format of the Nuclear Security Report so that each section of the report more clearly aligns with the nuclear security program and its subprograms. According to IAEA officials, DNS devotes substantial resources—including two full-time staff—to meeting all of its reporting requirements. Our analysis of three IAEA reports for 2016—the Nuclear Security Report, the Annual Report, and the Program Performance Report—found that DNS reports on some performance measures for its nuclear security program, but not all. Specifically, in the Nuclear Security Report, DNS reports on one measure fully and one partially and does not report on two measures. Specifically, DNS reports fully on the number of activities carried out in conjunction IAEA reports partially on the number of states that request and receive assistance, as identified in INSSPs. The agency reports on the number of states that completed INSSPs and provides examples of assistance but does not report whether that assistance was requested through INSSPs. For example, in the 2016 Nuclear Security Report, IAEA reported that five member states formally approved INSSPs. The agency also reported several examples of assistance to member states, such as training workshops on radiological crime scene management for Colombia in February 2015, Lithuania in February 2015, and the Philippines in June 2015. However, the report did not specify whether the need for that assistance was identified through INSSPs. IAEA does not report on the number of member states that have established or improved national nuclear security measures based on advice from IAEA or the number of activities duplicated by other initiatives. None of the three IAEA reports we reviewed consistently includes performance measures for the nuclear security subprograms. Table 2 shows the extent to which at least one of the three 2016 reports we reviewed includes measures for program and subprogram performance. Member states have expressed concerns with the effectiveness of IAEA’s reporting on the nuclear security program. In 2018, IAEA member states included language in the Nuclear Security Resolution to encourage the agency to improve communication with the public and member states about its nuclear security activities and their global impact. U.S. officials we interviewed said that they are dissatisfied with the reports, including with the quality of information on nuclear security activities, and would like to see, among other things, better reporting on how those activities support the agency’s mission, rather than reports that merely describe activities completed. IAEA officials provided two reasons why IAEA is limited in communicating more comprehensive information on nuclear security program performance in its reports. First, IAEA officials said that there are sensitivities around the data IAEA collects about member states, and member states are hesitant to share information on their security weaknesses. However, IAEA can report on its program performance without reporting sensitive information about individual states. Many of its measures pertain to numbers of states, and in cases where there are sensitivities, IAEA could aggregate data to a regional level to conceal state-specific information. Second, IAEA officials said that member states may not consistently make available to the agency the information it would need to measure the impact of its work. For example, to measure the number of states that established or improved national nuclear security measures based on advice from the agency, IAEA would need to know whether states implemented the agency’s recommendations. However, as previously mentioned, most of IAEA’s performance measures are focused on activities the agency carries out and not activities of member states. For example, one of the nuclear security subprogram’s measures is the number of states that participate in the Nuclear Security Guidance Committee. IAEA should have the data it requires to report on measures focused on activities carried out or facilitated by the agency. The lack of completeness in DNS’s reporting limits the effectiveness of the agency’s communication on the nuclear security program’s performance. By consistently including the results of its performance measures in at least one of its reports, IAEA could better communicate internally and with external stakeholders on the nuclear security program’s performance. IAEA member states disagree over the agency’s role in nuclear security. These disagreements have frequently contributed to DNS’s challenges over resources and the agency’s central coordinating role in nuclear security. According to U.S. and member-state officials and experts, IAEA member states disagree over the agency’s role in nuclear security. According to U.S. officials, member states supportive of the agency’s nuclear security role—such as the United States—see nuclear security as an issue with trans-border implications and believe the agency is well suited to supporting and facilitating cooperation on international, regional, and national nuclear security efforts. U.S. officials said that some member states do not see nuclear security as an international responsibility, but rather only as a national one, and disagree with IAEA’s nuclear security role to various extents. The disagreements over the agency’s role are rooted in a number of issues: Questions regarding the statutory basis for IAEA’s nuclear security work. Some U.S. officials and experts told us that some member states question IAEA’s nuclear security work because it is not established in the agency’s statute. IAEA officials told us that disputes over the statutory basis for IAEA’s nuclear security work are no longer an issue, and officials representing member states that had raised questions about the statutory basis for the work conceded that the matter was settled. However, these member-state officials said they felt strongly that because of the weak statutory basis, IAEA’s nuclear security work should be limited to core areas such as physical protection of nuclear facilities, rather than emerging areas such as cybersecurity. According to U.S. officials, other member states acknowledge the limited statutory basis for IAEA’s nuclear security work but still recognize the IAEA’s nuclear security role, which includes cybersecurity and newer areas of work. Perception of nuclear security as a barrier to or competition with IAEA support of civilian nuclear programs. According to IAEA, U.S. and several member-state officials, some states are concerned that IAEA’s nuclear security work could create barriers to their civilian nuclear programs—for example, by requiring recipients of IAEA technical cooperation to adhere to nuclear security guidance. In addition, according to U.S. and some member-state officials, some member states view IAEA’s nuclear security work as competing for resources with the agency’s other programs, such as the Technical Cooperation program, which assists member states with developing civilian nuclear programs. U.S. officials said that the Group of 77 generally advocates for more of the agency’s funds to be allocated to such programs. Resistance to nuclear security as a proxy for disagreement on other issues. U.S. officials, many mission officials, and many experts said that political disagreements among member states on unrelated or tangentially related international nuclear issues undermine IAEA’s nuclear security work. For example, U.S. officials and many member- state officials and experts told us that disagreement between nuclear weapons states and nonnuclear weapons states about nuclear disarmament manifests itself as political resistance in various IAEA forums to the agency’s nuclear security activities. Resistance to the Nuclear Security Summits. Some U.S. and member-state officials and experts said that some IAEA member states resented the perceived exclusive nature of the Nuclear Security Summits. As previously mentioned, the final summit in 2016 resulted in an Action Plan in Support of the IAEA in which signatories made commitments to support IAEA’s nuclear security mission. According to several mission officials and experts we interviewed, some excluded member states do not believe that the agency should carry forward the summits’ work, which in their view represents the priorities of the approximately 50 summit participants rather than all 171 IAEA member states. One expert said that within IAEA, there is resistance to anything associated with the summits among the member states that did not participate and that those states do not want IAEA involved in regulating or implementing anything resulting from the summits. IAEA officials and others we interviewed said that the disagreements over the agency’s nuclear security role create tangible challenges for the agency concerning funding, as member states that do not support the agency’s nuclear security role resist efforts to substantially raise DNS’s regular budget. As a result, according to IAEA, U.S., and several member-state officials, DNS continues to rely heavily on extra-budgetary contributions and has a smaller proportion of regular budget funding than other IAEA divisions, including other parts of the Department of Nuclear Safety and Security. DNS’s regular budget funding represents less than a quarter of total nuclear security program funding, with 78 percent of the funding coming from extra-budgetary contributions (see fig. 5). As we have previously reported, the extra-budgetary contributions on which DNS relies are voluntary, unpredictable from year to year, and inflexible, as they are often directed to specific purposes and often carry additional conditions. As a consequence, the nuclear security program’s large reliance on extra-budgetary support affects program management and human resources in ways that may undermine effective management of the program. IAEA officials identified several ways in which the nuclear security program’s heavy reliance on extra-budgetary funding affects program management. Planning and prioritization. According to IAEA officials, because extra-budgetary contributions are predominantly directed to specific purposes and can only be used for direct assistance to states, rather than support costs, they may not align with DNS’s most critical needs. IAEA officials also said that reliance on extra-budgetary contributions leads DNS to plan its activities around conditions stipulated for the contributions rather than planning around overall program needs. Donor states may also use the contributions to create cost-free expert positions for their own personnel that may not meet DNS needs. U.S. officials said, however, that even within the constraints of extra- budgetary contributions, DNS could take steps to work with donors to conduct work on a broader range of projects and initiatives, such as providing donors with plans to address longer-term, strategic needs. Program sustainability. IAEA officials, several member-state officials, and some experts we interviewed raised concerns about the effect of extra-budgetary contributions on the sustainability of IAEA’s nuclear security efforts. For example, several experts suggested that the DNS’s planning of work around individually-funded projects means that IAEA’s focus tends to be on short-term activities rather than long- term sustainability, including through follow-up on prior work. IAEA officials did not agree with the concern about follow-up work, but did acknowledge that long-term reliance on extra-budgetary contributions was unsustainable. Human resource management. IAEA officials also identified ways in which the reliance on extra-budgetary funding affects DNS’s human resource management. First, extra-budgetary funding generally supports positions that are initially designed to last for only 2 or 3 years, leading to few long-term positions in the division and making it difficult to sustain continuity of knowledge and experience over time. Second, staff hired for positions supported by extra-budgetary funding tend to look for regular-budget-funded positions elsewhere in the agency, which hurts recruitment as well as retention within DNS. Furthermore, the division must dedicate several staff to reporting on the use of extra-budgetary funding provided by each donor. U.S. officials acknowledged the detrimental impact of DNS’s high reliance on extra-budgetary contributions on staffing, but said that they are open to working with DNS to mitigate this impact. Member states have emphasized through the 2017 and 2018 Nuclear Security Resolutions, which are approved by the General Conference, the need to continue providing appropriate resources for the agency to implement its nuclear security activities. Furthermore, signatories of the Action Plan in Support of the IAEA, including the United States, committed to “contribute effectively to the implementation of the IAEA Nuclear Security Plan, including through reliable and sufficient resources.” The United States and other member states supportive of IAEA’s nuclear security role have advocated for increasing the agency’s regular budget for nuclear security. IAEA officials stated that, because of the politics around the agency’s nuclear security work, as well as the zero-growth policy, it is unlikely that the regular budget for nuclear security will increase substantially in the short term. As a result, IAEA officials have undertaken short-term solutions to minimize the impact of its reliance on extra-budgetary funding, such as reaching out to major donors and cultivating new sources of funding. However, such new sources of voluntary funding also would not be guaranteed or predictable and therefore would not improve the stability of the division’s funding stream. According to IAEA officials, the agency has not identified options to stabilize DNS’s budget within the existing constraints. IAEA officials and experts suggested other options for making the nuclear security budget more stable and flexible. One option could involve making structural changes to the Nuclear Security Fund, such as assessing a percentage of each extra-budgetary contribution and allocating those assessed funds for general expenditures without conditions. This could give the program more flexibility in using the funds and to support longer- term needs or projects. Another option could involve shifting funding within the Department of Nuclear Safety and Security to balance the proportion of regular and extra-budgetary funding between the Nuclear Safety and Nuclear Security divisions. U.S. and IAEA officials identified drawbacks to some of these options but IAEA has not comprehensively identified and analyzed options to stabilize DNS’s budget within the existing constraints. By working with the United States and other member states to analyze options to stabilize funding for the agency’s nuclear security program, IAEA could ensure that it has sufficient, reliable resources to implement the Nuclear Security Plan. The member-state disagreements discussed above—together with IAEA’s not following key practices for collaboration—limit IAEA’s ability to fulfill its central coordinating role in nuclear security. As noted in the Nuclear Security Plan, an objective of IAEA’s nuclear security program is “to play the central role and enhance international cooperation in nuclear security.” Numerous U.S., IAEA, and member-state officials and experts we interviewed said that there is a need for coordination of international nuclear security efforts and that IAEA is the appropriate entity to take on that role. These officials and experts cited IAEA’s perceived international legitimacy, technical expertise, and broad range of nuclear security efforts as key attributes that would allow the agency to play that coordinating role. DNS officials told us that they fulfill the agency’s central coordinating role in nuclear security in two key ways: (1) by providing nuclear security guidance that establishes the terms of reference for any nation working to improve its nuclear security and that is used by all member states and (2) by hosting and participating in key meetings. They said they further fulfill the role by using the agency’s international legitimacy and neutrality to work with countries that may be wary of international assistance from western countries. In addition, according to the agency’s Nuclear Security Plan, managing international nuclear security education through the Nuclear Security Support Centre and International Nuclear Security Education Networks is part of the central coordinating role. However, we found that IAEA is not fully implementing its central coordinating role in nuclear security, based on feedback from member states and experts and our evaluation of the extent to which IAEA has followed key practices that can sustain effective collaboration. Some experts told us that IAEA’s limited approach to its central coordinating role is a response to the resistance among some member states to the agency’s nuclear security role. According to many officials and experts we interviewed, IAEA’s approach to its central coordinating role is limited: Minimal outreach to key nuclear security stakeholders. Many experts expressed concern about the level of coordination with nongovernmental organizations and industry and said that IAEA would benefit from conducting more outreach to key nuclear stakeholders, including states. According to one expert, although IAEA may only conduct nuclear security activities at member-state request, IAEA could conduct more outreach to states about the assistance the agency could provide. Furthermore, some member- state officials and experts said the staff the agency sends to nuclear security meetings are not of the appropriate level of seniority. One expert said that IAEA does not engage actively with the Nuclear Security Contact Group, which, as previously mentioned, was established at the last Nuclear Security Summit to continue the work of the summit process after it ended. Specifically, according to this expert, the agency downgraded the level of representation it sent to Nuclear Security Contact Group proceedings to an official unauthorized to speak for DNS. However, U.S. officials said that senior DNS officials represented IAEA in more recent NSCG meetings. Logistical rather than substantive management of events. Several member-state officials and experts told us that IAEA limits its role at the events it organizes to logistical coordination rather than substantive management. According to one expert, to coordinate some of its support centers, IAEA convenes periodic meetings where participants share what they are doing, but it does not actively manage the support centers to reduce duplication. Some experts told us that multiple support centers in the same region teach the same content to the same students, raising concerns about duplicative activities. Another expert said that IAEA could more actively manage the support centers by starting discussions about best practices and, for example, the value of certification. To further examine IAEA’s fulfillment of its central coordinating role, we reviewed certain key practices that we have previously found can enhance and sustain collaborative efforts, such as: defining and articulating a common outcome, establishing joint strategies and compatible policies and procedures to operate across boundaries, identifying and addressing needs by leveraging resources, and agreeing on roles and responsibilities. IAEA’s planning documents—the Nuclear Security Plan and the P&B— define and articulate a common outcome. However, DNS has not established joint strategies or compatible policies and procedures with other nuclear security stakeholders, identified and addressed needs by leveraging resources, or agreed on roles and responsibilities. DNS officials said that they discuss these issues—such as resources and roles—at information exchange meetings with other organizations with a role in international security and said that these meetings have not resulted in agreed-upon or documented roles and responsibilities. In addition, the meetings have not resulted in the documentation of needs or resources, joint strategies, or compatible policies or procedures. As a result of IAEA’s approach to its central coordinating role in nuclear security, the agency may be missing opportunities to fully leverage its international legitimacy, technical expertise, and broad range of nuclear security efforts. By following key practices for collaboration, DNS could more formally define IAEA’s central coordinating role in nuclear security and strengthen the role even within the context of member-state disagreements. IAEA’s DNS plays a crucial role in preventing dangerous releases of radiation by assisting nations in securing their nuclear materials and protecting their nuclear facilities against sabotage. IAEA plans its nuclear security activities through a range of documents, but does not prioritize those activities. The agency’s P&B contains criteria for prioritization, but the criteria are too broad to help DNS make resource decisions. Guidelines for prioritizing activities would help DNS ensure that it is applying its resources toward the areas of greatest program needs. In addition, IAEA’s performance measures do not have baselines and targets. By developing baselines and targets to demonstrate results, DNS can more effectively monitor progress toward achieving the program’s objectives. Furthermore, none of the three IAEA reports on the nuclear security program fully addresses performance measure results. Improved reporting could help IAEA more effectively communicate internally and with external stakeholders on program performance. The nuclear security program relies heavily on extra-budgetary contributions, which adversely affects program management. Options exist to address this issue but IAEA has not analyzed these options. IAEA and its member states acknowledge the agency’s central coordinating role in nuclear security, but the agency has not followed key practices for collaboration. This has left IAEA’s approach to the central coordinating role vulnerable to member-state disagreements, and IAEA’s implementation of the role has not met the expectations of various member states. We are making the following five recommendations to the Department of State: The Secretary of State should work with IAEA and its member states through the Board of Governors to develop detailed guidelines for prioritizing nuclear security activities. (Recommendation 1) The Secretary of State should work with IAEA and its member states through the Board of Governors to improve the nuclear security program’s performance measures by developing baselines and measurable targets. (Recommendation 2) The Secretary of State should work with IAEA and its member states through the Board of Governors to improve how DNS reports to member states by consistently including the results of performance measures in at least one of the reports. (Recommendation 3) The Secretary of State should work with IAEA and its member states through the Board of Governors to analyze options to stabilize DNS’s funding within current fiscal and political constraints to enhance the sustainability of IAEA’s nuclear security program. (Recommendation 4) The Secretary of State should work with IAEA and its member states through the Board of Governors to strengthen the agency’s central coordinating role by following key practices for collaboration. (Recommendation 5) We provided a draft of this report to the Departments of State and Energy and to the International Atomic Energy Agency for review and comment. In its written comments, reproduced in appendix III, State concurred with all five of our recommendations. We are sending copies of this report to the appropriate congressional committees, the Secretary of State, the Secretary of Energy, and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao,gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines (1) the structure and range of nuclear security work that the International Atomic Energy Agency (IAEA) conducts; (2) ) how IAEA plans and prioritizes its nuclear security work, and how it measures and reports on its performance; and (3) the challenges that IAEA’s nuclear security program faces. We focused our review on IAEA’s nuclear security program, specifically on activities carried out by the Division of Nuclear Security (DNS), within the Department of Nuclear Safety and Security. To address all three objectives, we interviewed U.S. officials, IAEA officials, officials representing IAEA member states, and other nuclear security experts. We selected the U.S. agencies most involved in nuclear security policy, including interacting with IAEA. The Department of State is the lead agency for interacting with IAEA and has represented the United States in the Nuclear Security Contact Group (NSCG) since September 2018; the Department of Energy’s National Nuclear Security Administration provides technical expertise and loans staff to IAEA; Nuclear Regulatory Commission, as the regulator for the U.S. civilian nuclear industry, provides perspectives on how IAEA’s guidance may impact states’ regulations, among other things; the Department of Defense collaborates with IAEA to develop IAEA training (for example, for border monitoring); and the National Security Council leads interagency coordination to develop U.S. priorities for nuclear security and initially represented the United States in the NSCG through August 2018. To gain the perspectives of IAEA member states, we selected member states based on their involvement in IAEA’s nuclear security work and suggestions from State and nuclear security experts; the selected member states represent a range of informed opinions, but cannot be generalized to the universe of IAEA member states. While we reached out to various member states, we predominantly received responses from member states who have voiced support regarding IAEA’s nuclear security work. Our statements about member states we spoke to should be interpreted with the understanding that few member states that have voiced opposition to IAEAs nuclear security work responded to our requests. Throughout this report, we use the phrase “member states we spoke to” or “member states who responded” to refer to all those who provided us information. In light of political sensitivities surrounding IAEA’s nuclear security work, we agreed not to identify the member states whose officials we interviewed. We selected nuclear security experts based on a literature search and a snowball sampling technique. Specifically, from our initial literature search, we selected seven authors who had published at least two articles since 2010 that were relevant to our review. However, two authors declined or did not respond to our interview request. During our interviews with the authors identified in the literature search, as well as with U.S. government officials, we asked for suggestions of individuals who were knowledgeable on IAEA’s nuclear security work or nuclear security more broadly. We added to our sample individuals named at least twice by other interviewees. Not all experts in the sample were available to participate in interviews. We summarized the information gathered from experts and other interviewees in the report by using “some” to refer to three members of a group, “several” to refer to four or five members of a group, and “many” to refer to more than five members of a group. We interviewed officials representing 12 member states, and 20 experts. To determine the structure and range of IAEA’s nuclear security work, we reviewed pertinent legal instruments, such as the Statute of the IAEA, the Convention on the Physical Protection of Nuclear Material and its 2005 amendment, and the International Convention for the Suppression of Acts of Nuclear Terrorism. We also reviewed IAEA’s planning documents, including the 2018-2019 Programme & Budget (P&B); 2017 and 2018 Nuclear Security Resolutions; and the 2018-2021 Nuclear Security Plan. To review how IAEA plans and prioritizes its nuclear security work, we reviewed these planning documents against the Project Management Institute’s The Standard for Program Management and interviewed IAEA officials responsible for planning and prioritizing the agency’s nuclear security work. To examine how IAEA measures and reports on performance, we reviewed the previously mentioned IAEA documents, as well as IAEA’s Nuclear Security Reports from 2016-2018. We also reviewed the 2015-2016 P&B, the 2016 Nuclear Security Report, 2018 Annual Report, and the 2016 mid-term program performance report to understand IAEA’s use of objectives, outcomes, and performance indicators. We chose the 2016 reports because at the time of our review, the 2016 program performance report was the most recent completed. We compared the agency’s planning documents and reports with leading practices for performance management and reporting, including leading practices derived from our prior work, and IAEA’s results-based management approach. We derived some of these leading practices from standards and practices developed for federal agencies, such as those established in Standards for Internal Control in the Federal Government. Although federal standards are not required to be used by international organizations such as IAEA, the leading practices based on these standards can be instructive for assessing IAEA performance measurement and reporting practices. To examine the challenges the agency’s nuclear security role faces, we reviewed the IAEA documents listed above as well as proceedings from meetings and conferences and budgetary contributions data from the United States and other member states. We analyzed statements from IAEA, U.S., and member-state officials and from experts about IAEA’s nuclear security challenges. We also assessed actions IAEA and member states have taken to potentially mitigate challenges by comparing those actions with written commitments made in support of the agency’s nuclear security work. We also reviewed IAEA’s central coordinating role in nuclear security against certain key practices that we have previously found to enhance and sustain collaborative efforts. We selected five practices as relevant to our analysis, and combined two practices—those on establishing joint strategies and establishing compatible policies and procedures. In our analysis we considered IAEA’s role as a coequal entity among many rather than one that has authority over other entities. We conducted this performance audit from March 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Nuclear Security Contact Group (NSCG) was established at the 2016 Nuclear Security Summit to continue the work of the summit process, including maintaining high-level political attention and momentum on nuclear security, assessing and following up on commitments made at the summits, and developing and maintaining connections to nongovernmental experts and the nuclear industry. NSCG has 48 members as of March 2019, and its membership is open to all International Atomic Energy Agency (IAEA) member states. According to State officials, the group actively focuses on recruiting new members. NSCG advertises itself through IAEA Board of Governors statements and has issued joint statements to encourage other member states to join. Canada was the first country to chair NSCG, followed by Jordan and Hungary, which is the current chair. NSCG formally meets on the margins of the IAEA General Conference. According to U.S. officials, NSCG has convened two to three times per year since its inception after the 2016 summit. IAEA is an observer, and an IAEA representative may comment on how NSCG proposals would impact IAEA. Representatives to the NSCG are government agencies. The National Security Council was the lead agency to represent the United States in the NSCG through August 2018, and State has been the lead agency since September 2018. According to officials and experts we interviewed, NSCG serves as a forum for proposing and developing ideas rather than as a formal decision-making body. Member states described the benefits NSCG has provided. For example, some member-state officials said NSCG helps maintain contact among summit participants and between nuclear security officials in their respective capitals—where nuclear security policy would be implemented—and those at IAEA. In addition, a member-state official said that the group is a very important instrument for developing key messages as part of a communication strategy. As a result of this strategy, some ideas developed in NSCG have been introduced into IAEA proceedings by NSCG member states, or into national policymaking discussions. NSCG has also prepared unofficial position papers. Furthermore, according to a member-state official we interviewed, NSCG has discussed or developed internal papers on a number of topics related to IAEA, including: ways to improve IAEA’s coordinating role in nuclear security whether more regulation is needed in nuclear security IAEA’s role in dealing with emerging nuclear security challenges promoting a more resource stable and empowered Division of Nuclear communication and outreach within IAEA the agency’s networks of nuclear security training centers. According to State officials, U.S. priorities for NSCG include ensuring that it is productive and action-oriented, with representatives ready to share views, brainstorm on ways forward, and lead change both at home and internationally. State is also focused on preparing the NSCG’s input for its representatives to significant conferences, such as IAEA’s ministerial- level and technical conferences and the 2021 Review Conference on the Amendment to the Convention on the Physical Protection of Nuclear Material. According to a member-state official, the NSCG has discussed how to engage in preparation for the review conference, the framework of the review, what to ask of member states, and whether to revise the Convention. According to U.S. officials and some member-state officials, NSCG has also promoted implementation of summit commitments, in which individual members are responsible for tracking and following up on commitments made by countries in certain areas. For example, the United States is the lead for following up on commitments related to insider–threat mitigation, and the Department of Energy led a meeting in Belgium in February 2019 on that topic. State officials said that NSCG also follows up on commitments made during the 2016 International Conference on Nuclear Security. Many experts we interviewed said that the NSCG process lacks transparency. Specifically, it does not publish its proceedings, which these experts said made it difficult to discern its accomplishments. U.S. and several member-state officials and experts said that a quiet approach was necessary to protect the group from IAEA member-state politics. Several representatives said that NSCG is mindful of the political sensitivities around its association with the Nuclear Security Summits, and is committed to supporting IAEA’s nuclear security role without becoming a distraction. In addition, one expert said that more openness would weaken the group as a discussion forum. For example, publishing proceedings would require getting consensus among members, which would shift the focus of the group from discussion to decision-making. U.S. officials said that NSCG planned to revamp and update its website, and to use it to highlight nuclear security successes and events, such as nuclear security support for major public events or regional training events, but did not plan to promote its own work. In addition to the contact named above, the following staff members made key contributions to this report: William Hoehn (Assistant Director); Alisa Beyninson; Antoinette Capaccio; R. Scott Fletcher; Ellen Fried; Drew Lindsey; Steven Putansu; Liz Spurgeon; and Sara Sullivan.", "summary": "Nuclear terrorism remains a significant threat to the security of the United States and its allies and partners. U.S. efforts to prevent nuclear terrorism include working with IAEA, an autonomous international agency affiliated with the United Nations. The Department of State coordinates the United States' policy with and financial contributions to IAEA. IAEA's nuclear security program aims to assist countries in enhancing the physical protection, control, and accounting of their nuclear and radiological material and nuclear facilities. GAO was asked to review IAEA's nuclear security program. This report examines (1) the structure and range of nuclear security work that IAEA conducts, (2) how IAEA plans and prioritizes its nuclear security work and measures performance, and (3) the challenges that IAEA's nuclear security program faces. GAO analyzed key IAEA documents and interviewed IAEA officials, U.S. and foreign government officials, and nuclear security experts. The International Atomic Energy Agency (IAEA) carries out its nuclear security program under its Division of Nuclear Security through four subprograms. IAEA activities under these subprograms include developing guidance, providing training, and assisting countries in enhancing nuclear and radiological material security. IAEA plans its nuclear security work through several key documents, including a Nuclear Security Plan, which calls for activities to be prioritized. However, IAEA's planning documents do not include guidelines for prioritization. Instead, IAEA officials said they respond to member states' requests as they arrive and to the extent resources are available. By developing guidelines for prioritizing its nuclear security activities, IAEA could help ensure that it is allocating its resources to the areas of greatest need. IAEA has developed performance measures for its nuclear security program, but these measures do not have baselines or targets. This limits IAEA's ability to demonstrate the results of its nuclear security program. IAEA member states disagree over the agency's role in nuclear security, and according to U.S. and other member-state officials and experts GAO interviewed, these disagreements create challenges for the agency, such as funding its nuclear security efforts. Officials added that states that do not support the agency's nuclear security role resist efforts to substantially raise the agency's regular budget for nuclear security, contributing to the program's heavy reliance on voluntary, or extra-budgetary, contributions from member states. GAO previously reported that extra-budgetary funding is unreliable. Reliance on such funding affects nuclear security program planning, human resources, and sustainability. Experts and U.S. agency officials have suggested options to stabilize nuclear security program funding, but IAEA has not analyzed such options. By working with the United States and other member states to analyze options to stabilize nuclear security program funding, IAEA could ensure that it has sufficient, reliable resources to implement the Nuclear Security Plan. GAO is making five recommendations to the Department of State, including that it work with IAEA to develop guidelines for prioritizing IAEA's nuclear security activities, develop program baselines and targets, and work with the United States and other member states to analyze options to stabilize nuclear security funding. State concurred with all five recommendations.", "document_type": "gao"}
{"report": "MDA is responsible for developing a number of systems, known as elements, with the purpose of defending against ballistic missile attacks. MDA’s mission is to combine these elements into an integrated system- of-systems, known as the Ballistic Missile Defense System. The goal of the BMDS is to combine the abilities of two or more elements to achieve objectives that would not have been possible for any individual element. These emergent abilities are known as integrated capabilities or BMDS- level capabilities. Table 1 provides a list and description of elements included in our review. When MDA was established in 2002, it was granted exceptional flexibilities to set requirements and manage the acquisition of the BMDS—developed as a single program—that allow MDA to expedite the fielding of assets and integrated ballistic missile defense capabilities. These flexibilities allow MDA to diverge from DOD’s traditional acquisition life cycle and defer the application of certain acquisition policies and laws designed to facilitate oversight and accountability until a mature capability is ready to be handed over to a military service for production and operation. Some of the laws and policies include such things as: obtaining the approval of a higher-level acquisition executive before making changes to an approved baseline, reporting certain increases in unit cost measured from the original or current baseline, obtaining an independent life-cycle cost estimate prior to beginning system development and/or production and deployment, and regularly providing detailed program status information to Congress, including specific costs, in Selected Acquisition Reports. In response to concerns related to oversight, Congress and DOD have taken a number of actions. For example, Congress enacted legislation in 2008 requiring MDA to establish cost, schedule, and performance baselines—starting points against which to measure progress—for each element that has entered the equivalent of system development or is being produced or acquired for operational fielding. MDA reported its newly established baselines to Congress for the first time in its June 2010 BMDS Accountability Report. Since that time, Congress has required more details for the content of these baselines. Additionally, to enhance oversight of the information provided in the BMDS Accountability Report, MDA continues to incorporate suggestions and recommendations from GAO. However, not all of our recommendations have been fully implemented. For example, in April 2013, we recommended that MDA stabilize its acquisition baselines so that meaningful comparisons can be made over time to support oversight. MDA stated that the information presented in the BAR is sufficient; however, we continue to find that the lack of stable baselines makes comparison difficult and in some instances, impossible. MDA develops capabilities and then delivers them to the military services. Using this process, MDA declares an asset or capability ready for delivery for potential operational use while communicating the capabilities and limitations of the asset. Representatives from the receiving military service or combatant command then have the ability to assess this evidence and decide whether to accept the new capability. Because the military services conduct minimal missile defense testing of their own, this process is one of the only ways to convey vital performance information. The accuracy of this information is especially important as it informs training materials, doctrine, and deployment decisions and provides evidence supporting these assertions. MDA supports its assertions of capabilities with evidence from three sources: models and simulations, ground testing, and flight testing. Ground tests and models and simulations permit more flexibility in scheduling and design, but both are dependent on logistically more difficult flight tests to provide real-world performance data. As a result, MDA’s ability to organize, conduct, and evaluate flight tests is one of the most important factors in whether MDA is able to adhere to its schedule and declare an asset or capability ready for delivery. Though MDA has flexibilities in managing the acquisition process, it must follow the same contracting regulations that apply to DOD, including the Federal Acquisition Regulation and the Defense Federal Acquisition Regulation Supplement (DFARS). For this report, we reviewed MDA’s use of a particular type of contract action that authorizes a contractor to begin work before contract terms, specifications, or price have been agreed upon. These “undefinitized contract actions” are permitted by the DFARS, with certain limitations. Undefinitized contract actions are generally used when negotiation of a definitive contract action is not possible in sufficient time to meet the government’s requirements and the government’s interest demands that the contractor be given a binding commitment so that contract performance can begin immediately. Under the DFARS, undefinitized contract actions must include a specific “not-to- exceed” price. Once the action’s terms, specifications, and price have been agreed upon or determined, a process known as definitization, the contract action converts to a “definitive” contract. Under the DFARS, undefinitized contract actions must contain definitization schedules that provide for definitization by the earlier of (1) 180 days after issuance or (2) the date on which the amount of funds obligated under the action is more than 50 percent of the not-to-exceed price. Once the government has received a qualifying proposal from the contractor, however, the government can extend the undefinitized period another 180 days. Similarly, the government may obligate up to 75 percent of the not-to-exceed price, if the contractor submits the qualifying proposal before 50 percent of the not-to-exceed price has been obligated. DOD’s regional Ballistic Missile Defense (BMD) effort consists of a number of specific weapon systems or elements that compose the BMD system as a whole. According to DOD, various versions of these weapon systems are being deployed in Europe, Korea and other regions. The European effort known as the European Phased Adaptive Approach (EPAA) integrates the upgrades to Aegis BMD Weapon System, Aegis BMD interceptors, C2BMC and sensors, and was originally planned for delivery in four phases. Additionally, each phase is designed to rely on increasingly capable missiles, sensors, command and control, and integration to defend Europe against increasingly longer range ballistic missiles. DOD delivered the first phase, for short- and medium-range defense of Europe, in December 2011, and delivered the second phase for medium- range missiles in December 2015. Its efforts for both of these phases were also characterized by schedule delays, technical challenges that led to reductions in the scope of capability delivered, as well as testing reductions, which reduced confidence in capabilities that had been delivered. According to its capability plans, the purpose of EPAA Phase 3 is to provide a “robust Intermediate-Range Ballistic Missile (IRBM) defense.” Figure 1 depicts the weapon systems that DOD deployed in support of the European Phased Adaptive Approach capability. As we have previously reported, MDA encountered numerous challenges in an effort to meet its original EPAA goals and we have made several recommendations to improve MDA’s management of its integrated capability efforts, including EPAA, to reduce risk for individual elements and to improve testing practices overall. For instance: In January 2011, we recommended that DOD develop life-cycle cost estimates and establish an integrated schedule for EPAA. DOD partially concurred and concurred, respectively, to the recommendations. An independent life cycle cost estimate was prepared, however an integrated schedule that produced sufficient detail was never completed. In April 2012, we recommended that DOD assess the extent to which the dates announced by the President in 2009 are contributing to concurrency and recommend schedule adjustments where significant benefits can be obtained. DOD did concur with this recommendation, however never included a specific assessment of the extent to which capability delivery dates for the European Phased Adaptive Approach announced by the president in 2009 were contributing to concurrency; instead, it asserts that BMDS technology development is fundamentally driven by completion of technical milestones, not schedule declarations. In May 2017, we recommended that MDA address deficiencies in its testing scheduling policy to better align it with best practices for scheduling. DOD did not concur with this recommendation. Consequently, the department continues to allow MDA to schedule and plan its test program without risk analyses, or assigning resources to each test. Unless the department takes action to address these challenges, the department should continue to expect MDA to fall further behind in its test program. In fiscal year 2018, MDA focused additional regional capability efforts on the Korean Peninsula. This new effort was requested by the United States Forces Korea in December 2017 to counter North Korean ballistic missiles. Capabilities for the Korean effort are currently planned for delivery between February 2018 and April 2021, and are based on element-level upgrades as well as integration enhancements between THAAD and Patriot. In December 2017, MDA achieved a significant asset delivery milestone, completing the deployment of 44 operational ground-based interceptors (GBI). In deploying these interceptors, MDA also fulfilled a goal set by the Secretary of Defense in March 2013 to increase the inventory of GMD interceptors from 30 to 44 by the end of December 2017. Although MDA achieved this goal, it did not deliver two of the four GBIs planned for fiscal year 2018. One of the GBIs is intended for use in an upcoming flight test that was delayed to fiscal year 2019. The other delayed GBI delivery was the result of the boost vehicle contractor mishandling the booster avionics module—a critical component that houses the flight computer and navigation systems. The contractor is working on replacing the component but the rework has delayed delivery of the final GBI to fiscal year 2020. Other on-time capability deliveries included the release of new software versions for several major BMDS elements, including C2BMC (Spiral 8.2- 3), BOA 6.1, THAAD (THAAD 3.0), AN/TPY-2 (CX 3.0), and GMD (GS 7A). Another expected software release was Aegis Weapon System (BL 9.2), but that was delayed to at least March 2019 to accommodate verification and validation of models and simulations and to accompany the delivery of the Aegis BMD SM-3 Block IIA. In terms of asset deliveries, specifically interceptors used to counter enemy missiles, MDA successfully delivered all 53 THAAD interceptors specified in the baseline for fiscal year 2018, as well as an additional five interceptors the delivery of which had been delayed from the previous year. For a summary of MDA’s asset delivery status for fiscal year 2018, see table 2. Although MDA made a number of deliveries, including all planned THAAD interceptors, it did not meet its fiscal year 2018 asset delivery goals due to a variety of factors. The Aegis BMD SM-3 Block IB program, which received full production authority early in fiscal year 2018 after years of delays, delivered 12 of 36 planned interceptors in fiscal year 2018. This shortfall was due to the discovery of a parts quality issue that necessitated suspending deliveries until MDA could complete an investigation of the issue’s impact on the interceptor’s performance. In addition, the Aegis BMD SM-3 Block IIA program delivered one of four planned test interceptors due to a flight test failure early in the year suspending further deliveries pending completion of a failure review board. Moreover, according to MDA officials, construction contractor performance issues will result in the Aegis Ashore Missile Defense System Complex—Poland not being delivered until at least 18 months after the planned December 2018 date. As discussed later in this report, this facility is central to MDA’s plans for the EPAA Phase 3, such that a delay in the completion of this facility resulted in a delay in the planned EPAA Phase 3 delivery to the warfighter. MDA conducted seven fiscal year 2018 flight tests as planned, and during one of those seven the interceptor failed. According to MDA’s Integrated Master Test Plan, MDA scheduled eleven flight tests of the systems included in our review. MDA’s ability to adhere to its flight test schedule for fiscal year 2018 was hampered by several issues, including technical challenges, test failures requiring new tests to be inserted into the schedule, and range and target availability. Of the four tests not conducted, MDA delayed two to future fiscal years, and deleted two, with their objectives planned to be mostly fulfilled by separate events. Table 3 highlights MDA’s fiscal year 2018 flight tests. MDA also added several test events to its schedule over the course of fiscal year 2018. They are listed below in table 4. The two most significant flight tests scheduled for fiscal year 2018 were delayed into fiscal year 2019. Specifically, FTG-11, GMD’s first salvo test (launching multiple interceptors at a single target), was delayed until the second quarter of fiscal year 2019 to accommodate other BMDS testing priorities while GMD fixed software issues uncovered during pre-test planning. In addition, FTO-03 Event 1, a test designed to assess the Aegis BMD SM-3 Block IIA capability against an IRBM was to be the first (and only) operational test of the EPAA Phase 3 architecture before MDA delivered the capability. This test was delayed to accommodate the demand for range and test assets following the insertion of a new test into the schedule. Fiscal year 2018 legislation expanded and accelerated several MDA programs. In December 2017, Congress passed and the President signed into law the Department of Defense Missile Defeat and Defense Enhancements Appropriations Act, 2018 (MDDE), which increased missile defense appropriations. The MDDE provided approximately $2 billion in appropriations for missile defense. MDDE provided funds in support of plans that would expand and accelerate several missile defense programs beyond the agency’s previous baselines. According to MDA, the administration directed the Secretary of Defense to develop options for accelerating missile defense capabilities in response to North Korea flight testing a new intercontinental ballistic missile in July 2017. According to MDA, it collaborated with Office of the Secretary of Defense and the Joint Chiefs of Staff to identify programs and capabilities that could be accelerated and delivered within the current Future Years Defense Plan and directly address the North Korean missile threat. DOD then took those options back to the administration to finalize the MDDE plan, which was subsequently presented to Congress. These plans most significantly affected the GMD program and the Aegis BMD SM-3 Block IIA. Under the plans and with the funds provided by MDDE, the GMD program will increase its inventory from 44 GBIs to 64 GBIs by 2023. Each of these new interceptors will be equipped with the Redesigned Kill Vehicle (RKV), accelerating the latter program’s schedule by approximately one year. MDA also intends to use $451 million from MDDE to procure 16 additional Aegis BMD SM-3 Block IIA interceptors. The Aegis BMD SM-3 Block IIA program was still in development at the time, and these funds represented the first time Congress appropriated procurement funds, and not research and development, for the program. The RKV program, in part to support the accelerated schedule, adopted a new program schedule that required concurrency in some areas. As we previously reported, the original RKV strategy avoided concurrency by aligning production decisions with flight testing. However, to accommodate the newly accelerated schedule, the program began procuring some components before completing qualification testing. Under this new plan, qualification testing would only be completed around the same time as the planned first flight test. MDA’s contracting plans for the RKV have been closely aligned to the test schedule, to the point that MDA will have more than half of its planned RKV buy under contract before conducting a successful intercept test. The program planned to award a production contract for Lot 1 and the long-lead materials contract for Lot 2 following a major design review, but before the first flight test. Following the first flight test (CTV-03+) in first quarter fiscal year 2020, the program planned to award a production contract for Lot 2 and long-lead materials for Lot 3. Upon completion of the first intercept test (FTG-17) in the first quarter of fiscal year 2021, the program planned to award the production contract for the final planned lot, Lot 3. Through the course of fiscal year 2018, the RKV program has been unable to meet its cost and schedule milestones. Specifically, the prime contractor has reported accumulating negative cost and schedule variances with no signs of arresting these trends. The contractor also reported inefficiencies stemming from bringing large numbers of new staff onto the project, as well as requiring more personnel for the project than they originally anticipated. According to MDA, as fiscal year 2018 progressed, the program discovered that some components would not meet performance requirements. MDA therefore postponed the critical design review from fiscal year 2018 to fiscal year 2021. Moreover, MDA no longer plans to achieve its goal of fielding 64 interceptors by 2023. In addition, MDA anticipates RKV’s total cost has increased by nearly $600 million as a result of the design issues. See appendix VI for information on RKV and the GMD program. The Aegis BMD SM-3 Block IIA schedule planned for an initial production decision in fiscal year 2018, but one month after the MDDE’s enactment, the program experienced its second consecutive failure in a significant flight test—FTM-29—that introduced significant uncertainty into the Aegis BMD SM-3 Block IIA’s schedule. In an effort to maintain the program’s schedule, the Undersecretary of Defense for Acquisition and Sustainment in an Acquisition Decision Memorandum provided selective authorization to use procurement funds. The memorandum placed a cap on how much the program could spend, and had a list of approved “pacing items” (which excluded parts still under investigation for the test failure) on which the funds could be spent. Under the terms of the memorandum, MDA would have to meet a series of requirements to lift these limitations, such as completion of the failure review board and implementation and demonstration of corrective actions. MDA operated under these limitations for the remainder of the fiscal year. MDA used undefinitized contract actions (UCA) in fiscal year 2018, particularly in programs receiving MDDE appropriations. In May 2018, we found that MDA’s use of UCAs in recent years had increased in both total not-to-exceed value and in the length of the undefinitized period. While MDA improved its performance in timely definitization of these contract actions in fiscal year 2018, the total not-to-exceed value of the undefinitized contract actions MDA initiated in 2018 far exceeded previous years we reviewed. UCAs allow work to begin on a program before the government and contractor have agreed to all contract terms, such as price or scope. MDA states that undefinitized contract actions are necessary, particularly in the case of programs accelerated by the MDDE appropriation, because they allow work to begin immediately. Coming to agreement on all terms before beginning work would have added months to program schedules that, MDA stated, could not accommodate such a delay. Undefinitized contract actions are permitted under the Defense Federal Acquisition Regulation Supplement, but we have found in the past that the use of these contracts can pose particular risks for the government. Examples of recent UCAs follow: In October 2017, MDA issued a sole source undefinitized contract action for $60 million (according to DOD and MDA, the value was later increased to $88 million) for the purposes of transitioning the Aegis BMD SM-3 Block IIA program from development to production. This work will improve the manufacturing readiness of the contractor’s production facilities, with the goal of eventually supporting a production rate of two interceptors per month. According to MDA officials, definitizing this contract action proved difficult. The contractor’s initial cost and fee position were substantially higher than MDA’s and independent government estimates, even after those estimates were revised upwards when they were found not to include costs specific to the Aegis BMD SM-3 Block IIA. MDA initially planned for a definitization in April 2018. By that time, all terms had been agreed to except for the contractor’s fee. According to MDA officials, the parties deadlocked until August 2018, when, with the authorization of the Director, MDA, contracting officials “unilaterally definitized” the contract. MDA officials told us that when a unilateral definitization occurs, the government essentially imposes its terms on a “take-it-or-leave-it” basis, effectively halting negotiations. According to MDA officials, in this case, the contractor acceded to the government’s terms and continued work on the project. When asked about possible consequences to this action, MDA officials stated that it is possible for contractors in this situation to seek administrative relief, but in this case, they stated such an appeal would be unlikely to succeed, and believed the contractor would be unlikely to pursue it. It is also possible, officials said, that the contractor would either be reluctant or refuse to accept an undefinitized contract action from MDA in the future. In fiscal year 2017, MDA issued a sole source undefinitized contract action for the design and initial production of the RKV. This contract had a not-to-exceed value of $1.1 billion. MDA issued the contract with an estimated definitization date of May 14, 2018. Despite the issues encountered by the RKV program described above, MDA reported that it definitized this contract action on schedule in May 2018, for the same price as the original not-to-exceed value, $1.1 billion. MDA issued several undefinitized contract actions in 2018. For example, in April 2018, MDA issued a sole source undefinitized contract action for the production of Aegis BMD SM-3 Block IIA “pacing items”, with a not-to-exceed value of $387 million. The Undersecretary of Defense for Acquisition and Sustainment issued a memorandum stating the circumstances under which MDA could obligate additional procurement, defense wide funds. MDA officials stated that “pacing items” were those items whose lead times were not long enough to qualify for long-lead procurement, but which were still substantial enough (more than 2 years) to cause delays if their production waited until the successful completion of operational testing. These officials also explained that the pacing items excluded any components which were still under investigation for the failure of FTM-29. Before that test’s failure and the ensuing involvement of the Undersecretary, MDA planned for a not-to-exceed value of $672 million. MDA initially planned for a definitization date of December 2018, but it has since been delayed. MDA issued its largest undefinitized contract action for the fiscal year (as measured by its not-to-exceed value of $6.56 billion) in January 2018. For the past several years, the GMD program planned to transition away from its all-inclusive contract to a structure involving three new contracts: one for systems engineering, integration, and testing; one for ground systems readiness, operations, and support; and one for all-up round interceptors. This Development, Operations and Sustainment, and Production approach would have been a significant undertaking. It would have required that MDA take control of the technical baseline for the entire program. MDA also believed that this strategy would provide for enhanced competition and reduced organizational conflicts of interest. With the MDDE appropriation and associated program acceleration, the Director, MDA decided that managing the transition to this new contracting strategy, in addition to fielding 20 new ground-based interceptors was too risky. Thus, MDA issued an undefinitized contract action that provided a six-year extension to the main development and sustainment contract for GMD. The contract action has a not-to-exceed value of $6.56 billion, a value higher than that for all undefinitized contract actions issued by MDA in the previous 5 years combined. MDA was able to definitize most elements of this contract in March 2019. Figure 2 illustrates MDA’s increasing use of undefinitized contracts as measured by the sum of their not-to-exceed values. In fiscal year 2018, MDA delivered regional capabilities to counter threats from North Korea, but did not meet all of its 2018 goals for its effort in Europe to counter intermediate-range ballistic missile (IRBM) threats from Iran, known as the European Phased Adaptive Approach (EPAA) Phase 3. Specifically, the agency delivered planned upgrades and additional assets for the Korean Peninsula—an effort it began in 2017. However, the delivery of the third and final phase of the EPAA has been delayed by 18 months. Despite this delay, testing intended to demonstrate EPAA Phase 3 capability has been significantly reduced and de-scoped or deferred past the new delivery date, which reduces the warfighter’s insight on the system’s capabilities and limitations. MDA delivered upgrades on time to the Korean Peninsula in February and September 2018. Notably, the upgrades provided initial integration between THAAD and Patriot—key elements of the effort in Korea— improving THAAD and Patriot’s ability to coordinate during engagements. MDA also delivered element-level upgrades for THAAD, including additional interceptors, as well as a new software release that expanded THAAD’s ability to counter new threats and improved its performance in the presence of debris. These upgrades were assessed in an April 2018 flight test that demonstrated interoperability between THAAD and Patriot by exchanging Link-16 messages over tactical data links while tracking a missile target, and an April 2018 BMDS-level ground tests that provided further performance data for these upgrades in a simulated environment. MDA plans to deliver additional capabilities for the Korean Peninsula in the future. We currently have ongoing work related to these areas. Details will be included in a future report. MDA’s effort to deliver the third and last phase of the EPAA has been delayed from December 2018 to May 2020. MDA planned to deliver the EPAA Phase 3, for defense against IRBM threats, at the end of calendar year 2018, but construction delays for Aegis Ashore, the linchpin of Phase 3, delayed its completion by 18 months. In fiscal year 2018, the delay for EPAA Phase 3 was caused by challenges at the construction site for Aegis Ashore in Poland. According to MDA officials, delays to the Aegis Ashore were primarily driven by military construction contractor performance issues. As these delays continued to accumulate, MDA initially planned to make up for them by increasing concurrency between the construction phase and the installation and checkout phases of the project, and concurrently working at the sites in Romania and in Poland. As we previously reported, these increasing levels of concurrency posed a growing risk for the program and its ability to achieve its target delivery date. In March 2018, MDA officials recognized that plans for Aegis Ashore had become untenable, and the project’s schedule would have to be extended. This plan required the development of a new delivery schedule for EPAA Phase 3 resulting in delivery in May 2020. MDA experienced testing disruptions throughout the EPAA Phase 3 development, including delays and failures, but overcame some of them in fiscal year 2018. The consequence of the testing disruptions is that EPAA Phase 3 will be delivered to the warfighter with less data than planned about performance against planned threats. According to DOD’s acquisition guidance and the BMDS Warfighter Capability Acceptance document, testing is fundamental to ensuring that DOD acquire a system that works, and to provide data necessary to characterize the system’s effectiveness in operational settings. Thus, the warfighter relies on testing to understand the system’s capabilities and limitations and therefore how to fight with what MDA has built. As we previously found, EPAA Phase 3 testing disruptions started in 2016, when MDA delayed the first and second intercept flight tests of the Aegis BMD SM-3 Block IIA, the interceptor planned for fielding in EPAA Phase 3. Although this test was successfully conducted in February 2017, testing difficulties continued when it failed the second intercept flight test. MDA continued to experience challenges with testing necessary to demonstrate the EPAA Phase 3 capability in fiscal year 2018, which resulted in less robust testing. Specifically, as we discussed earlier in this report, the interceptor failed its first intercept test, FTM-29, against an intermediate range target, EPAA Phase 3’s intended threat. Following a failure investigation, and developmental work, MDA rectified the Aegis BMD SM-3 Block IIA design flaws and successfully demonstrated them against a medium-range ballistic missile target in October 2018, during FTM-45. MDA decided to use a medium range target in this test and concluded that it was sufficient to assess Aegis BMD SM-3 Block IIA fixes. However, according to MDA documentation, the test against a medium range target does not provide the same challenges as an intermediate range target. In December 2018, it successfully demonstrated for the first time an intercept of an IRBM during a test called FTI-03, previously called FTO-03 Event 1. While this test was successful, its scope was reduced from an attempt against a raid of two targets to instead a single intercept, in part, due to a test range safety asset malfunction. With these flight tests, according to MDA officials, it completed its flight testing requirements for EPAA Phase 3 delivery and that adding additional tests would be disruptive to their overall test plan. Our analysis indicates that flight testing to demonstrate EPAA Phase 3 performance against IRBMs—the goal of Phase 3—has been reduced by 80 percent and even with the added 18-month delay, MDA no longer plans to conduct a flight test against a raid prior to delivery in fiscal year 2020. Figure 3 shows both the original and current plans for demonstrating EPAA Phase 3 performance through flight testing. Figure 3 above shows that the original plan included five IRBM intercepts across three tests, including tests to assess capability against small raids requiring simultaneous intercepts of multiple missiles—a likely tactic in a real-world attack— prior to delivery of EPAA Phase 3. However, as figure 3 also depicts, the current plan reduces the number of intercept tests against an IRBM and does not include a flight test against a raid until after EPAA Phase 3 capability is declared. Although the delivery has been delayed 18 months, in part due to the delay in construction at the Aegis Ashore site in Poland, the current plan significantly reduces the amount of data needed to support the EPAA Phase 3 capability and limitation assertions. As we previously reported, test and evaluation activities are an integral part of developing and producing weapon systems, as they provide knowledge of a system’s capabilities and limitations as it matures and is eventually delivered for use by the warfighter. Consequently, the 18-month delay provides an opportunity to add in additional tests and an ability to provide further data to the warfighter or to make any design changes discovered during testing. As we previously reported, delivering capability before testing is complete has led to performance unknowns and increases the likelihood of cost increases if future testing discovers any design flaws. MDA made further progress in fiscal year 2018 in its mission to defend the United States and its allies from enemy ballistic missiles, including achieving a significant integrated capability milestone for defending the United States. However, MDA did not meet all of its goals for the fiscal year. Specifically, not all programs delivered all planned assets in fiscal year 2018 and shortfalls were attributed to developmental delays and testing challenges. The acceleration of several programs following a budget increase in December 2017 introduced concurrency, which indicates a familiar risk: accounting for insufficient margin in an effort to meet schedule-driven milestones, rather than pursuing a knowledge- based approach. Construction delays related to another integrated capability, EPAA Phase 3, may, in fact, present an opportunity to build more knowledge in that area. EPAA Phase 3 intends to provide a robust defense against IRBM and raids of multiple targets, but tests to demonstrate that capability have been reduced from five to one with the test against the raid scenario not occurring before the capability is delivered. Our prior work has shown that proceeding with limited test data can result in late, and costly, discovery of performance problems. More thorough assessment of the capabilities and limitations of the system could mitigate that risk by building a more solid base of knowledge. We are making one recommendation to MDA: The Director, MDA, should utilize additional schedule margin afforded by the EPAA Phase 3 delay to conduct additional testing necessary to thoroughly assess the capabilities and limitations of Phase 3 against IRBMs and a raid scenario prior to delivery. (Recommendation 1) We provided a draft of this report to DOD for comment. DOD’s comments are reproduced in appendix IX. DOD and MDA also provided technical comments, which were incorporated as appropriate. In its comments, DOD partially concurred with our recommendation to utilize additional schedule margin afforded by the 18-month delay to the EPAA Phase 3 delivery to conduct additional testing necessary to thoroughly assess the capabilities and limitations against IRBMs and a raid scenario prior to delivery. DOD stated that all EPAA Phase 3 BMDS functions requiring a flight test environment were already successfully demonstrated and MDA has addressed the intent of our recommendation by adding ground tests to further assess EPAA Phase 3 capabilities. However, in order for the agency to meet the full intent of our recommendation, additional flight testing to demonstrate capability against EPAA Phase 3 threats is necessary. Flight testing against IRBM threats and raid scenarios could provide additional confidence in modeled performance, even for aspects of the model that have the achieved accreditation threshold. Our finding is supported by MDA’s own assessment of testing needed for EPAA Phase 3, which originally included five IRBM intercepts and two raid flight tests. These testing requirements were reduced even after EPAA Phase 3 flight test failures and delays. Specifically, our analysis indicates that flight testing to demonstrate EPAA Phase 3 performance against an IRBM has been reduced 80 percent. Moreover, MDA will not conduct a flight test against a raid—a likely tactic in a real-world attack—prior to delivery. As we identified in this report, MDA experienced testing disruptions throughout the EPAA Phase 3 development, which resulted in significant data collection reductions, especially regarding performance against planned threats. According to the Director, Operational Test and Evaluation (DOT&E), these testing challenges, in large part, precluded MDA from testing Aegis BMD against some expected threat types, ranges, and raid sizes. Consequently, the use of models and simulations- based ground tests to supplement such significant reduction in real-world data collections could be problematic. Specifically, we have previously reported that some of MDA’s models and simulations used in its ground tests do not provide realistic representation of the BMDS, the environments it encounters, or the modeled threats. This year, we found that as a result of testing perturbations, certain aspects of Aegis BMD 5.1 will not be validated until after EPAA Phase 3 delivery. Relying on unaccredited models increases chances for modeling errors, and a single undetected modeling error can distort the results for the entire assessment. Lastly, DOD stated that the demands on the test program due to the evolutionary nature of the BMDS acquisition leave no margin (cost or schedule) for adding additional flight tests. While we agree that adding a flight test requires additional costs and coordination, the reductions to EPAA Phase 3 testing constitute a significant reduction in performance data and decreases warfighter’s knowledge base about how best to deploy a system under operationally realistic conditions, such as raids. We continue to believe the 18-month delay affords the schedule to conduct additional flight testing. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Defense, the Undersecretary of Defense for Research and Engineering, and to the Director, MDA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix X. Key findings for Fiscal Year 2018 Aegis Ballistic Missile Defense (BMD) demonstrated integration with allies. Aegis BMD 5.1 demonstrated increased capability, but testing disruptions delayed its delivery to March 2019 and deferred raid assessment to 2020. MDA re-planned schedules for some future Aegis capabilities due to funding challenges. Aegis Ballistic Missile Defense is the naval component of the Missile Defense Agency’s (MDA) Ballistic Missile Defense System. It consists of the Aegis combat system, including a radar, and Standard Missile-3 (SM- 3) interceptors. MDA is developing the Aegis BMD in versions called spirals that expand on preceding capabilities. Since 2015, MDA has been delivering Aegis BMD spirals that are integrated with capabilities developed by the Navy. These jointly developed Aegis Weapons System Baselines (AWS BL) allow for Integrated Air and Missile Defense (IAMD) where ballistic missiles and air threats (i.e., cruise missiles) can be engaged at the same time. Table 5 identifies Aegis BMD spirals, associated integrated Aegis Weapons System Baselines and key capabilities, and their delivery date. The first suite of integrated ballistic missile defense and anti-air warfare (AAW) capabilities was delivered with AWS Baseline 9.C1/B1, which included an overhaul of Aegis computing architecture. However, in order to expand the number ships with IAMD, MDA also began a program to integrate Aegis BMD 5.0 CU capabilities with the legacy AWS architecture. While initially scheduled for delivery in 2015, Aegis BMD 4.1 was delayed multiple times, and finally in 2017 the delivery was split into two phases. The first interim phase was completed in 2017, but did not provide integration between BMD and AAW capabilities. The second phase will integrate BMD and AAW, and is currently planned for delivery in 2020. Additional upgrades capitalizing on Navy’s improvements to the AWS Baseline 5.4 computing architecture are planned for delivery in 2023. The program is also developing Aegis BMD 5.1 with capabilities to support the final phase of European Phased Adaptive Approach. This spiral is designed to control the new Standard Missile-3 Block IIA and to intercept intermediate-range ballistic missiles. It also includes the Engage on Remote (EOR) capability, where Aegis BMD intercepts a threat before it is visible to its own radar, based entirely on tracks from a forward-based sensor. Aegis BMD 5.1 is integrated with AWS Baseline 9.C2/B2. Additionally, MDA and the Navy are developing AWS Baseline 10.0, which will capitalize on the Navy’s effort to replace the Aegis SPY-1 radar with a more capable SPY-6, and to overhaul the entire Aegis combat system. AWS Baseline 10.0 includes Aegis BMD 6.0 capabilities, which is planned to counter more threat types, larger raids, better discrimination, and improved communication with its interceptors. AWS Baseline 10.0 is planned for delivery in 2023. For specifics on Aegis Ashore and the Aegis SM-3 interceptors, see appendixes II, III and IV, respectively. Table 6 provides key fiscal year 2018 AWS program facts. In fiscal year 2018, MDA demonstrated the ability of Aegis BMD to engage some simple and complex threats as well as integration with European and Asia-Pacific allies for new and legacy spirals. As table 5 above shows, Aegis BMD participated in a number of flight tests and exercises, which provided additional information about its capabilities and interoperability with allies in two regions, where MDA is currently focusing its regional integrated capability efforts. For example: Formidable Shield-17 demonstrated the ability of Aegis BMD 4.0.3, which was delivered in fiscal year 2015, to interoperate with North Atlantic Treaty Organization partners using communication architectures during cruise missile and ballistic missile engagements, and to use remote data provided by NATO partners to conduct remote engagements. Pacific Dragon demonstrated interoperability between U.S. Aegis BMD assets, Japanese destroyers, and Republic of Korea naval assets. JFTM-05 Event 2 demonstrated coordination between U.S. and Japanese destroyers using communications architecture to conduct ballistic missile engagements. MDA demonstrated some aspects of Aegis BMD EOR, as well as the ability of Aegis BMD 5.1 to engage a medium range and an intermediate range ballistic threat, but testing disruptions delayed data available to inform capabilities and limitations of the Aegis BMD 5.1, contributing to a 3-month delivery delay. MDA encountered challenges during tests for Aegis BMD 5.1, which resulted in a reduction of flight tests and delays in collecting data needed to accredit models for a system-level assessment. Specifically, during the conduct of FTM-29, Aegis BMD partially demonstrated EOR capability, lacking full demonstration because the weapon system did not exercise all aspects of communication in the later stages of the engagement due to an Aegis BMD SM-3 Block IIA malfunction. MDA decided not to retest FTM-29 and adjusted its test plan to only demonstrate the fixes to the SM-3 Block IIA in a new test called FTM-45, deferring a full EOR assessment by about a year to the subsequent test named FTI-03. This reduction in flight tests affected MDA’s ability to collect data for model verification which in turn, delayed the delivery of Aegis BMD 5.1. A model is a representation of an actual system that involves computer simulations and is used to predict how the system might perform or survive under various conditions. MDA, as well as independent DOD testing organizations, and the warfighter rely heavily on models to test operational performance that cannot be completely assessed using intercept flight tests because of the system’s scope and complexity and safety constraints. Flight tests, however, provide important information about real-world performance that is used to verify models. In order to ensure that key aspects of Aegis BMD 5.1 performance are well understood at delivery, MDA delayed the spiral from December 2018 to March 2019. This was done in part to allow for analysis from FTM-45 (conducted in October 2018) and FTI-03 (conducted in December 2018). According to the BMDS Operational Test Agency, data from these tests provided key information about Aegis BMD EOR performance— a key capability for Aegis BMD 5.1—that was used to verify its models, which were used to more thoroughly assess the extent of that capability. While EOR data will support Aegis BMD 5.1 delivery, another key aspect of its performance will not be verified until late in fiscal year 2020. Specifically, MDA planned to assess Aegis BMD 5.1 raid performance for the first time in December 2018, but the test was de-scoped to a single intercept due, in part, to a test range safety asset malfunction. The next planned raid assessment is scheduled for the fourth quarter of fiscal year 2020, well after Aegis BMD 5.1 delivery. According to the Director, Operational Test and Evaluation (DOT&E) these testing challenges, in large part, precluded MDA from testing Aegis BMD against some expected threat types, ranges and raid sizes. While some of them were outside of MDA’s control, others stem from decisions about its test plan. For instance, MDA’s inability to assess Aegis BMD 5.1 against an IRBM raid resulted from the malfunction of test range safety assets; however, according to DOT&E, FTM-29 failure is an example of insufficient development testing that should have discovered the SM-3 Block IIA issue prior to the flight test. DOT&E officials told us that they are currently working with MDA to ensure sufficient developmental testing is scheduled and conducted prior to undertaking operational tests. In fiscal year 2018, funding challenges contributed to the delay of MDA and the Navy’s effort to develop integrated AWS Baseline 5.4 and AWS Baseline 10.0. According to MDA program documentation, the delays resulted from funding reductions in fiscal year 2018. However, while AWS Baseline 5.4—which includes BMD 4.1—was delayed entirely from 2019 to 2020, AWS Baseline 10.0 – which includes BMD 6.0—delayed completion of some technical content, but its delivery timeframe did not change. Specifically: Integrated AWS Baseline 5.4 was originally planned to be completed in September 2019, but MDA and the Navy delayed its certification to March 2020. While MDA delivered Aegis BMD 4.1 capabilities in fiscal year 2017, subsequent efforts focused on integrating the ballistic missile defense with the remaining suite of AWS Baseline 5.4 capabilities. According to MDA, the delay to this effort was driven by a $14 million funding reduction to the Navy’s Program Executive Office Integrated Warfare System, which is jointly funding this baseline. As a result of the reduction, MDA received $16 million from the Navy, rather than $32 million it was expecting, to continue work on Baseline 5.4. According to Aegis BMD program officials, to mitigate the nine month delay, MDA renegotiated the associated contract, but it is anticipating approximately $1.5 million increase in fiscal year 2019 and approximately $4 million to fiscal year 2020 costs. MDA and the Navy re-planned AWS Baseline 10.0, after a funding reduction of $31.45 million against BMD 6.0. According to Aegis BMD program documentation, the BMD 6.0 development efforts stopped between January 2018 and May 2018. Program officials indicated that MDA renegotiated the associated contract to reflect the reduced funding, but the stop work and consequent restart incurred additional costs. Specifically, the program estimated that the disruption resulted in cost growth of approximately $51 million across the development timeline between fiscal year 2019 and 2024. Key findings for Fiscal Year 2018 According to Missile Defense Agency officials, deficiencies in the performance of the military construction contractor resulted in a significant delay and increased cost for the Aegis Ashore facility in Poland. The program continues to make progress despite challenges at both the Poland and Romania sites. Aegis Ashore is a land-based, or ashore, version of the ship-based Aegis Ballistic Missile Defense (BMD). Aegis Ashore is designed to track and intercept ballistic missiles in the middle of their flight using Aegis BMD Standard Missile-3 (SM-3) interceptors. Key components include a vertical launching system, interceptors, and an enclosure, called a deckhouse, that contains the SPY-1 radar and command and control system. Aegis Ashore will share many components with the sea-based Aegis BMD and will use future versions of the Aegis weapon system currently in development, including the SM-3 Block IIA interceptor. The Missile Defense Agency (MDA) plans to equip Aegis Ashore with a modified version of the Aegis weapon system software that will share many components with the sea-based Aegis BMD. DOD constructed an Aegis Ashore test facility in Hawaii in April 2014. The test facility has been used to flight test Aegis Ashore, and in some cases, Aegis BMD SM-3 interceptors. MDA deployed its first operational site in Romania in fiscal year 2016 as part of the European Phased Adaptive Approach (EPAA) Phase II. A second site in Poland was scheduled for delivery in 2018 as part of EPAA Phase III. Both operational sites are intended to provide additional coverage for the defense of Europe. The Poland site experienced construction delays over several years until March 2018, when MDA determined with stakeholders that the site would not be complete in time for the EPAA Phase III deadline. MDA has since established a new schedule baseline which delays the delivery of the site by 18 months, to May 2020. For further details on the Aegis Weapon System and Aegis BMD interceptors, see appendixes I, III and IV. Table 7 provides key fiscal year 2018 Aegis Ashore program facts. According to MDA officials, construction of the Aegis Ashore site in Poland has failed to meet schedule milestones from the start of the contract. According to officials, prior to this year, MDA and the Army Corps of Engineers, which manages military construction at the site, have undertaken a number of measures to mitigate or reverse these delays, including modifying contracts to permit joint occupancy of the site, modifying the main contract to provide more granular project data to the Army Corps of Engineers, moving key personnel on site, and adding a second shift. Program officials stated that they also withheld some award fees from the contractor as a result of these delays. Despite these efforts, MDA has found the contractor’s performance is still particularly poor in the areas of construction management, identification, procurement, timely delivery of important materials, and timely hiring of staff with appropriate skills. To make up for these delays, MDA introduced increasing levels of concurrency into its schedule, and shortened key phases of the delivery process. Activities such as Installation and Checkout were shortened from 16.5 months to 6.5 months, and would occur concurrently with the final phases of construction at the site. As recently as last year, GAO reported that additional delays or concurrency at the site would threaten the scheduled delivery date. Through the first quarter of fiscal year 2018, the contractor’s performance did not improve. According to program officials, in December 2017, MDA participated in a meeting with the Army Corps of Engineers, the Navy, and other government stakeholders, and concluded that the schedule for delivery had become untenable and schedule recovery was not possible. MDA later concluded that the site would not be ready for delivery until May 2020, a delay of 18 months. The costs of this delay will be significant. Following the determination of the new delivery date, MDA developed a new project schedule that, officials stated, incorporated historical data from the Romania site, independent outside analysis, trends in the contractor’s performance over time, and the resources that would be required at each stage of the schedule. MDA estimated that the additional efforts by MDA, the Army Corps of Engineers, and the Navy to mitigate the delay and provide assistance through the completion of the project totaled at least $90 million. According to program officials, the construction contract provides for significant liquidated damages, with the current daily assessment in excess of $125,000. MDA continues to oversee work at the Aegis Ashore site in Romania, despite the Navy’s acceptance of the site for operational use. MDA continues work on a variety of remaining items such as seismic hardening, shielding electrical infrastructure against high-energy electro- magnetic pulses, and cooling systems. In the case of cooling systems, the work is the result of the system failing to perform to specifications. MDA has yet to assess the full cost, schedule, and performance impacts of the necessary repairs and modifications, but MDA reported that none of the above issues had any impact on the Romania sites operational availability or performance. In the case of the Poland site, MDA sought to secure the permission of the Polish government to operate the facility’s SPY-1 radar in the 3.1 to 3.5 GHz radio frequency spectrum. This section of the spectrum is important to the full functioning of the Aegis Ashore system, but portions of it have been allocated for commercial use in Poland. MDA was able to de-conflict the operations of its radar with other systems on these frequencies, and in March 2018 secured the approval of the Polish government to operate the SPY-1 radar across the full range of frequencies. Key findings for Fiscal Year 2018 The Aegis Ballistic Missile Defense (BMD) Standard Missile-3 Block IB program received authorization for full production this year and performed successful intercepts in flight tests. Discovery of a parts quality issue partway through the year forced the program to suspend deliveries and thus miss most of its delivery target for fiscal year 2018. The Aegis Standard Missile-3 (SM-3) Block IB is a ship- and shore-based missile defense interceptor designed to intercept short- to intermediate- range ballistic missiles during the middle stage of their flight. The SM-3 interceptor has multiple versions in development or production: the SM-3 Blocks IA, IB, and IIA. Compared to the SM-3 Block IA, the Block IB features an enhanced seeker for improved target discrimination, better engagement coordination capabilities, an improved throttleable divert and attitude control system for adjusting its course, and increased range. The SM-3 Block IB interceptor is linked with Aegis Ballistic Missile Defense (BMD) Weapons System, and Aegis Ashore. For additional information about the Aegis Weapon Systems, see Appendix I and for Aegis Ashore, see Appendix II. Since fiscal year 2015, Aegis BMD SM-3 Block IB production has been delayed by several technical issues. Program officials, in 2015, delayed the decision to enter full-rate production until they could implement further testing and design changes, a decision consistent with a GAO recommendation at the time. In fiscal year 2016, two failures during testing forced a suspension of interceptor deliveries, though the program made up for this backlog in fiscal year 2017. Table 8 provides key fiscal year 2018 Aegis BMD SM-3 Block IB program facts. In February 2017, the Undersecretary of Defense for Acquisition, Technology, and Logistics issued an Acquisition Decision Memorandum requesting an additional flight test for the Aegis BMD SM-3 Block IB before authorizing a full production decision, as well as several independent supporting analyses. The memorandum issued these requirements in support of a planned full production decision in the first quarter of fiscal year 2018. As we previously reported, MDA has delayed full production multiple times over the life of the Aegis BMD SM-3 Block IB which was initially scheduled for fourth quarter, fiscal year 2012. MDA completed the requested intercept test, known as FS-17-4 in October 2017. The test was undertaken as part of NATO’s Formidable Shield naval exercises. In this test, an Arleigh Burke-class destroyer in the northern Atlantic fired an Aegis BMD SM-3 Block IB Threat Upgrade at an MRBM target and successfully intercepted it. With this result, the interceptor was approved for full production. In September 2018, MDA participated in JFTM-05 Event 2, a joint flight test with the Japanese navy, in which a Japanese ship successfully fired an Aegis BMD SM-3 Block IB Threat Upgrade interceptor at a simple separating short-range ballistic missile. MDA participated in and supported the engagement. Upon full production authorization, MDA sought to pursue a multi-year procurement with the prime contractor for 204 interceptors through 2023. While MDA requested and the 2019 National Defense Authorization Act and the Defense Appropriations Act, 2019 authorized this procurement, the program did not receive the funding to support the request. Program officials state that they are still evaluating the impacts on their plan. MDA estimates the procurement will have a projected price of $2.021 billion. During routine component testing, MDA discovered an issue with the Aegis BMD SM-3 Block IB’s throttleable divert and attitude control system (TDACS) resulting in delays of interceptors in fiscal year 2018. According to program officials, MDA employs a “manufacturing surveillance unit” whose purpose is to pro-actively assess component performance and quality at various stages of unit production. Program officials stated that the unit discovered, in January 2018, that one of several thrusters on the TDACS did not perform to specification. In response to this finding, MDA suspended deliveries of the interceptor until it could determine the impact of the deficiency on the interceptor’s performance. According to program officials, MDA contracted with the Applied Physics Laboratory to act as an independent technical authority for the investigation, which took approximately six months. Once concluded, the investigation found that the performance of the component, while below the defined specification, did not endanger the overall operation of the system. The component’s performance was accommodated within the margin the government and contractor built into the overall design, and was acceptable as built as a result. The investigation reached this conclusion in August 2018. MDA closely monitored the function of the component in JFTM-05, during which the system performed nominally. Program officials reported that the prime contractor has experienced similar issues defining and communicating important specifications to subcontractors at various levels of its supply chain. Similarly, the contractor has also had difficulty ensuring that all subcontracted components meet defined specifications. Program officials stated that they continue to take measures to mitigate these issues, including using the manufacturing surveillance team. Key findings for Fiscal Year 2018 A mid-year funding increase accelerated the program's schedule and increased the number of interceptors. The Aegis Ballistic Missile Defense (BMD) Standard Missile - 3 (SM-3) Block IIA experienced a test failure, leading to significant changes to the test plan. The latest development in the Aegis BMD Standard Missile – 3 (SM-3) family, the Aegis BMD SM-3 Block IIA interceptor provides increased speed, more sensitive seeker technology, and a more advanced kinetic warhead as compared to previous versions of the Aegis BMD interceptors. It is expected to defend against short-, medium-, and intermediate-range ballistic missiles, and will have significantly increased range compared to earlier Aegis BMD SM-3 models. Additionally, most of the Aegis BMD SM-3 Block IIA components will differ from other standard missile versions and therefore require new technology being developed specifically for them. For additional information on the Aegis BMD SM-3 Block IB interceptor, see appendix III. Initiated in 2006 as a cooperative development program with Japan, the Aegis BMD SM-3 Block IIA program is an essential component of the European Phased Adaptive Approach (EPAA) Phase 3 architecture, particularly its ability to defend against longer-range threats. According to program officials, the Aegis BMD SM-3 Block IIA interceptor’s range exceeds that of its native radar, thus, the only way to make full use of its extended range is by relying on remote sensor data. For additional information on Aegis Weapon Systems, see Appendix I. Table 9 provides key fiscal year 2018 Aegis BMD SM-3 Block IIA program facts. In December 2017, Congress passed and the President signed the “Department of Defense Missile Defeat and Defense Enhancements Appropriations Act, 2018”, as part of a larger continuing resolution which significantly increased missile defense appropriations. According to program officials, the impetus for seeking these additional appropriations was increased levels of missile development and testing activity from North Korea. MDA intends to use $451 million in procurement funds for the purchase of 16 additional Aegis BMD SM-3 Block IIA interceptors. These were the first procurement funds the program had received. The program had yet to receive an initial production authorization, so all previous manufacturing activity occurred using research and development funds. To this point, however, the Aegis BMD SM-3 Block IIA interceptor had succeeded in only one of its two intercept flight tests, and its ability to engage a longer-range target using remote sensor data, known as “engage on remote”, had yet to be tested. The following month, in January 2018, the interceptor failed an important intercept test, causing significant disruption to the program’s schedule which is discussed below. The Undersecretary of Defense for Acquisition and Sustainment subsequently released an acquisition decision memorandum which laid out near-term limitations on the use of procurement funds for the Aegis BMD SM-3 Block IIA, as well as providing for a series of steps MDA needed to take in order to obligate the remaining funds. These measures included the completion of an independent cost estimate, independent technical risk assessment, the successful completion of a replacement flight test, and the successful completion of the planned operational flight test scheduled for the first quarter of fiscal year 2019. Until MDA could meet these requirements, the Undersecretary authorized MDA to obligate only $162 million for the purchase of a limited subset of “pacing items.” According to program officials, “pacing items” are those with longer lead times for production, but which fall short of the threshold for long-lead procurement. Program officials also stated that the list of pacing items was restricted to components not implicated in the recent test failure. Program officials stated that they expected the Undersecretary to certify that these requirements had been met in the third quarter of fiscal year 2019. In January 2018, MDA conducted flight test FTM-29. In this test, the Aegis Ashore facility in Hawaii fired an Aegis BMD SM-3 Block IIA interceptor at an intermediate-range ballistic missile (IRBM), using remote sensor data, for the first time. After the interceptor launched, its third- stage rocket motor (TSRM) failed to ignite. As a result, the interceptor had inadequate thrust to complete the engagement and failed its objective to intercept the target. As a result of this test failure, MDA faced two challenges: first, identifying and remedying the source of the failure through a failure review board, and second, adjusting the program’s schedule to provide opportunities to confirm these mitigations. MDA and the government of Japan convened a failure review board (FRB) to investigate the causes of the test failure. The board’s conclusions found that the TSRM failed to ignite due to a combination of a faulty arm-fire device (AFD), which initiates the TSRM’s firing, and incorrect programming of the TSRM ignition sequence. In the case of the Aegis BMD SM-3 Block IIA, the AFD contains two linear “chains” of explosive pellets, which then ignite the rocket motor. MDA documents state that the AFD’s manufacturer expects a missile to ignite both chains simultaneously to ensure the highest degree of reliability. The FRB found that the Aegis BMD SM-3 Block IIA’s programming did not fire the AFD’s two chains simultaneously, but one after the other, or “sequentially”. When fired in this manner, quality issues with the AFD that would not have any material impact in a simultaneous firing can cause the AFD to malfunction when firing one after the other. The FRB concluded that the most likely cause of the AFD’s failure was a missing explosive charge in the first explosive chain. When this chain ignited, it fizzled and failed to ignite the TSRM. The fizzle was powerful enough to disrupt the functioning of the second explosive chain, however, which subsequently failed to ignite the TSRM as well. To correct for this error, MDA has changed the programming of the Aegis BMD SM-3 Block IIA to fire the AFD simultaneously. MDA has also instituted new quality measures at the assembly line for the AFD. These measures include additional quality assurance checks to ensure that all explosive pellets are present in both chains, as well as the use of X-ray- like scanners which can look inside a completed AFD to confirm the presence of all of the explosive pellets. Having identified the source of the failure, MDA had to choose what form any new test would take, and how it would impact the remaining schedule, in particular the first operational test of the Aegis BMD SM-3 Block IIA, which also happened to be the first operational test of the European Phased Adaptive Approach (EPAA) Phase III, and the only such test scheduled before MDA declared it ready for delivery. This test, then known as FTO-03 Event 1 (and subsequently re-named FTI-03) was scheduled for the first quarter of fiscal year 2019. One option was for MDA to schedule a scaled-back test, known as FTM- 45, of an Aegis BMD SM-3 Block IIA against a medium-range target. MDA stated that though FTM-29 failed, analysis of sensor data and missile telemetry indicated that the Engage on Remote capability would have succeeded had the interceptor reached the target. Therefore, FTM- 45 could be an “organic” engagement, using only the radar co-located with the interceptor. FTM-45 would need only to test that the mitigations identified by the FRB worked, as well as testing the final phases of the interceptor’s operations which had been interrupted in FTM-29. MDA had a medium-range ballistic missile (MRBM) target it could repurpose for this test, which would limit testing disruptions by not further delaying FTO-03 E1/FTI-03. FTM-45 was MDA’s preferred course of action FTM-45 lacked the support of several external, Department of Defense stakeholders, such as the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation, Joint Functional Component Command Integrated Missile Defense, and Office of the Director, Operational Test and Evaluation. These offices asserted that a complete re-test of FTM-29, known as FTM-29a, provided the most risk reduction in advance of FTO-03 / FTI-03. . MDA opted not to pursue FTM-29a, and cited several reasons. MDA acknowledged the differences between intermediate-range and medium- range engagements, but determined that the actual differences between FTM-45 and FTM-29a were within acceptable margins. FTM-29a would also prove more expensive and more logistically difficult. MDA concluded that FTM-45 met the requirements for risk reduction at the least disruption to the program’s schedule. MDA conducted FTM-45 in October 2018 and FTI-03 in December 2018. Initial reports indicate both were successful. Appendix V: Command, Control, Battle Management, and Communications (C2BMC) Key findings for Fiscal Year 2018 MDA re-planned schedules for some future Aegis capabilities due to funding challenges. MDA delivered Spiral 8.2-1 providing significant performance and cyber improvements, but some fixes were required after fielding. MDA mitigated prior challenges with Spiral 8.2-3 and demonstrated capability upgrades. Uncertainty in Ballistic Missile Defense System-level requirements could disrupt Spiral 8.2-5 schedule. C2BMC is a global system of hardware—workstations, servers, and network equipment—and software that integrates all missile defense elements of the Ballistic Missile Defense System (BMDS). Specifically, it allows users to plan operations, see the battle develop, and manage BMDS sensors. As the integrator, C2BMC enables the defense of a larger area than the individual BMDS elements operating independently and against more missiles simultaneously, thereby conserving interceptor inventory. C2BMC is fielded at U.S. Strategic Command, U.S. Northern Command, U.S. European Command, U.S. Indo-Pacific Command and U.S. Central Command. MDA is developing C2BMC in spirals, or software and hardware upgrades that build upon prior capabilities to improve various aspects of the integrated BMDS performance. The spiral delivered in fiscal year 2018 includes BMDS Overhead Persistent Infrared Architecture (BOA) — a system within the C2BMC enterprise. BOA receives spaced-based sensor information on boosting and midcourse ballistic objects and feeds that data to C2BMC for use in cueing BMDS sensors and weapon systems, and for situational awareness. The agency completed fielding and transition to operations of Spiral 8.2-1 with BOA 5.1 to U.S. Northern Command and U.S. Indo-Pacific Command in January 2018, and Spiral 8.2-3 with BOA 6.1 to U.S. European Command and U.S. Central Command in December 2018. Spiral 8.2-3 will replace Spiral 8.2-1 at the U.S. Northern Command and U.S. Indo-Pacific Command in the third quarter of Fiscal Year 2019. Table 10 provides an overview of C2BMC Spiral upgrades, planned fielding timeframes and associated capabilities, and Table 11 provides key fiscal year 2018 C2BMC program facts. In January 2018, C2BMC completed fielding and transition to operations of Spiral 8.2-1 providing a significant overhaul of the BMDS command and control hardware infrastructure. Spiral 8.2-1, replaced the legacy Spiral 6.4, at the U.S. Northern Command and U.S Indo-Pacific Command. Spiral 8.2-1 improves sensor coverage, ballistic missile track management, and cyber security, optimizing raid size tracking capability and capability for processing new threats to support the defense of United States. Further details on these capabilities follow: Spiral 8.2-1 delivery includes the BOA 5.1, which provides improvements in early missile launch detection, allowing more time for all subsequent BMDS actions. It cues land-based sensors allowing them to acquire threats sooner, allowing them longer time to track and thus improving engagement probability. Spiral 8.2-1 expands the capability for processing of threat tracks, called System Track, from a single sensor—the Army/Navy Transportable Radar Surveillance-2 (A/N TPY-2) —to include additional sensors for homeland defense and BOA. This allows for additional data sources about threat characteristics that C2BMC subsequently provides to other BMDS elements. The delivery of Spiral 8.2-1 also improves cybersecurity. Spiral 8.2-1 replaced Spiral 6.4, which, as we found in May 2018, had cyber vulnerabilities that, if exploited, could have degraded mission capabilities like BMD planning, radar control, track reporting, and situational awareness. Lastly, the program also delivered additional upgrades, to specifically augment BMDS capabilities for the Korean Peninsula. These upgrades were delivered in December 2017 and June 2018, to provide improvements in communication between THAAD and Patriot, and improved cybersecurity in that region. MDA demonstrated Spiral 8.2-1 upgrades in Ground Test-07a and Ground Test-18 Sprint 1. Table 11 above provides an unclassified overview of C2BMC testing completed in support of fiscal year 2018 deliveries. While MDA delivered these upgrades and overcame development challenges, some fixes had to be implemented after deployment. Specifically, as we found in May 2018, MDA identified performance risks for Spiral 8.2-1 that could have affected interoperability with other elements and threat tracking and delayed the delivery to address these challenges. According to MDA’s fiscal year 2018 program management documentation, the program implemented the necessary mitigations to address these challenges; however fixes were also needed to be implemented after the Spiral was delivered. Moreover, the post- deployment fixes required diversion of resources from the subsequent Spiral 8.2-3, delaying demonstration of a certain aspect of that effort. In fiscal year 2018, MDA completed most of its development effort for its next spiral named Spiral 8.2-3. In addition, MDA completed a test, demonstrating new capabilities and mitigations to earlier development challenges. As we found in May 2018, in fiscal year 2017, the program was tracking two element level risks to C2BMC capability needed for EPAA Phase 3 called Engage on Remote. Specifically, program documentation indicated that processing of data about threat missile flight paths, known as threat tracks, had issues that could reduce the likelihood of the successful engagements utilizing Aegis BMD in Engage on Remote scenarios. C2BMC has faced similar challenges with threat tracking capabilities for prior spirals, which required delaying certain aspects of integration with Aegis BMD until fixes were implemented. While the program was addressing the aforementioned performance risks in fiscal year 2018, it encountered additional challenges. First, it needed to divert some resources from Spiral 8.2-3 to implement fixes to Spiral 8.2-1 that were needed after it was deployed. Second, the program needed to divert additional resources to meet a new Warfighter request for geographic redundancy. Specifically, while the original concept was to have 8.2-3 for Central and European Command at the same location, MDA met the Warfighter request by installing the spiral at different locations so that losing one location would not result in the loss of all capability for the Warfighter. Finally, once a key mitigation was completed, the program encountered delays in availability of laboratories needed to assess it. As result, MDA decided to test the mitigation during the GT-07b campaign, along with other Spiral 8.2-3 capabilities. While assessing mitigations for the first time in a large scale campaign is risky – should the mitigation be insufficient or have underseen downstream effects – initial results from GT-07b campaign indicate they were successful. The test demonstrated successful collaboration between Spiral 8.2-3 and Aegis BMD in support the Engage on Remote, as well as other capabilities. Table 11 provides additional information on capabilities demonstrated during GT-07b. While C2BMC program has identified element level requirements for Spiral 8.2-5, requirements for BMDS-level capabilities associated with this spiral are still under development. This Spiral is intended to integrate the Long Range Discriminating Radar (LRDR) and provide additional BMDS- level planning, track processing, and battle management capabilities, in the fiscal year 2021 timeframe, and its acquisition baselines are expected to be included for the first time in the upcoming BMDS Accountability Report. However, according to the November 2018 program execution review, emerging BMDS-level requirements may delay efforts to complete the development of the spiral in time to support LRDR functionality in 2021. Program documentation also indicates that some BMDS capabilities as well as future C2BMC spirals could be at risk of deferral, including the subsequent Spiral 8.2-7. Appendix VI: Ground-based Midcourse Defense (GMD) Key findings for Fiscal Year 2018 MDA continues to increase GMD capacity and reliability. GMD issues uncovered during salvo test planning demonstrate the value of rigorous and frequent testing. MDA recently uncovered major design concerns with the Redesigned Kill Vehicle. GMD is a missile defense interceptor system designed to defend the United States against a limited intermediate and intercontinental ballistic missile attack from rogue states, such as North Korea and Iran. To counter such threats to the homeland, GMD, in conjunction with a network of ground-, sea-, and space-based sensors, launches interceptors from missile fields based in Fort Greely, Alaska and Vandenberg Air Force Base, California. After launching from in-ground silos, the interceptor boosts towards the incoming enemy missile and releases an Exoatmospheric Kill Vehicle to find and destroy the threat. GMD also has ground support and fire control capabilities that the warfighter uses to operate the system. Table 12 provides key fiscal year 2018 GMD program facts. MDA fielded three new upgraded interceptors in early fiscal year 2018, meeting its directive from the Secretary of Defense to increase the total number of fielded interceptors to 44 by the end of 2017. The new interceptors are equipped with an upgraded version of the kill vehicle, called the Capability Enhancement (CE)-II Block I, and boost vehicle, called the Configuration 2. MDA completed production and fielded eight of these new interceptors after successfully conducting its first intercept flight test of the upgraded interceptor in May 2017. Although the program encountered some production challenges with the C2 boost vehicle, such as multiple components initially failing qualification testing, the issues were not significant enough to prevent the program from meeting its December 2017 fielding goal. The upgraded interceptors were designed to be more reliable than their predecessors and their addition to the fleet is intended to improve overall system reliability, as the older interceptors have a greater risk of experiencing in-flight reliability failures. Table 13 below describes the current fleet of 44 fielded interceptors and plans to field an additional 20 interceptors equipped with the Redesigned Kill Vehicle (RKV) and modified Configuration 2 boost vehicle. MDA also successfully completed two ground tests in fiscal year 2018 to provide performance assessment data; develop interceptor shot doctrine and tactics, techniques, and procedures; and assess recent performance upgrades to GMD’s fire control software. In addition to adding more CE-II Block I interceptors, in fiscal year 2018, MDA accelerated RKV development and initiated plans to increase the total number of fielded interceptors to 64 by the end of 2023 in response to a North Korean missile threat escalation in 2017. In November 2017, DOD requested $2 billion for what it called the Missile Defeat and Defense Enhancements, $774 million of which was designated for GMD to: (a) build a new 20-silo missile field at Fort Greely, Alaska; (b) procure long-lead components for four additional interceptors; (c) continue booster development; (d) accelerate RKV development; and (e) add a target to an initial non-intercept RKV flight test. MDA subsequently issued an undefinitized contract action in the form of a sole-source contract modification to Boeing in January 2018 to extend the current GMD development and sustainment contract. The contract modification was awarded with a total maximum value not to exceed $6.565 billion for efforts pertaining to the Missile Defeat and Defense Enhancements and extended the current contract’s period of performance 2023. In March 2019, MDA definitized $4.141 billion of the contract to build the new missile field, among other items, but deferred the production of 20 additional interceptors. According to MDA, this contract modification brings the total cumulative value of the GMD development and sustainment contract, including options, to $10.8 billion. MDA conducted its first salvo flight test of the GMD system, called Flight Test Ground-based Interceptor (FTG)-11 on March 25, 2019 after nearly three decades of GMD development. GMD demonstrated a salvo intercept by firing a CE-II Block I-equipped interceptor followed by a CE- II-equipped interceptor. The leading interceptor destroyed the target representing an intercontinental ballistic missile equipped with countermeasures designed to complicate missile defense operations. With the target reentry vehicle destroyed, the trailing interceptor struck one of the remaining objects, as it was designed to do. Demonstrating a salvo capability is particularly important because, during a ballistic missile attack, the warfighter intends to launch a number of interceptors to increase the probability of successfully intercepting the incoming missile(s). FTG-11 was further delayed from the end of fiscal year 2018 to mid-fiscal year 2019 to accommodate other BMDS testing priorities while GMD fixed software issues uncovered during pre-test planning. MDA initially planned to conduct the salvo test in fiscal year 2006 but subsequent test failures, developmental challenges, and fielding priorities delayed the salvo test to fiscal year 2018. Figure 4 below provides an overview of the multiple times MDA has delayed the salvo test over the years. By mid-2017, GMD began experiencing delays developing a software upgrade that is intended to provide the kill vehicle with the functionality needed for FTG- 11. Around that same time, MDA also realized that its BMDS-level integrated test schedule could not be executed as planned due to a lack of test range and asset availability. According to a May 2018 report MDA submitted to Congress, the agency delayed FTG-11 from the fourth quarter of fiscal year 2018 to the second quarter of fiscal year 2019 to de- conflict the integrated test schedule. Around the time MDA submitted the report to Congress, the GMD program also uncovered performance concerns with the kill vehicle software upgrade that further delayed the software’s completion. As such, the delay to FTG-11 to accommodate other BMDS testing priorities also afforded MDA the time necessary to complete the software improvements and pre-test planning. The performance issues MDA uncovered in pre-test planning for FTG-11 demonstrate the value of rigorous and frequent GMD testing. Congress and DOD have recognized the need for rigorous, operationally realistic GMD testing, including conducting a salvo test. Congress also passed legislation and the president signed into law a requirement for an annual GMD flight test, subject to several exceptions. However, GMD has historically averaged less than 1 test per year whereas Aegis Ballistic Missile Defense (BMD) Standard Missile (SM)-3 averaged over 2.5 tests per year (see figure 5 below). Moreover, GMD’s prior tests achieved less than 50 percent operational realism whereas Aegis BMD SM-3 averaged over 70 percent, according to Director for Operational Test and Evaluation assessments. The warfighter relies on testing to understand GMD’s capabilities and limitations. Without this knowledge, the warfighter lacks the information to operate GMD effectively and efficiently. Although MDA attempted to accelerate RKV development as part of the Missile Defeat and Defense Enhancements, the program accepted too much risk and has since experienced development challenges that set the program back likely by over two years and increased the program’s cost by nearly $600 million, according to the agency. In response to advancements in the North Korean missile threat, MDA accelerated RKV development by concurrently performing development and production and reducing the number of necessary flight tests to produce and field new RKV-equipped interceptors. Moreover, the RKV had already experienced development delays prior to the acceleration and was operating with no schedule margin for any further delays as it approached a critical design review in October 2018. The program subsequently encountered design, systems engineering, quality assurance, and manufacturing issues, which resulted in the program postponing the critical design review. The most significant development issue that emerged in 2018 pertained to RKV’s performance and its planned use of commercial off-the-shelf hardware and re-use of Aegis SM-3 Block IIA components. In multiple previous reports, we raised concerns regarding MDA’s use of these components as well as RKV’s aggressive development schedule. In our May 2017 report, we also recommended that DOD perform a comprehensive review of the RKV. Although such a review could have potentially provided DOD with a better understanding of RKV’s technical and schedule risks, DOD indicated in its response that the comprehensive review we recommended was unnecessary and therefore did not perform the review. Even though some of these risks have since manifested, we continue to believe an independent, thorough vetting of RKV’s acquisition risks is necessary, as we previously recommended. Although RKV continued to carry significant acquisition risks, MDA implemented a recovery plan that attempted to minimize the addition of further risks by opting to prioritize controlling technical risks over preserving the 2023 fielding goal via an aggressive schedule. At the time of our review, the program projected that it would conduct a critical design review for RKV in early fiscal year 2021 followed by a non-intercept flight test in fiscal year 2022, an intercept test in fiscal year 2023, and deployment starting a few months later. The extended design period provided the program additional time to source or design new components before moving forward with testing and production. Production decision gates also remained aligned to the critical design review and subsequent flight tests. The recovery plan also placed greater emphasis on addressing technical risks rather than fielding deadlines to determine RKV’s path forward. Our prior work has shown that stabilizing system design before making major production commitments and relying on knowledge rather than deadlines to make acquisition decisions at key milestones are best practices of successful product developers. MDA’S Deputy Director stated during a March 2019 press briefing that “the best thing to do was to go back and assess that design and take the time to do it right.” The Deputy Director also acknowledged that it would have been the wrong step to do “what the Missile Defense Agency did years ago, which is to go ahead and produce what we’ve got and then deal with reliability issues in the fleet and erode the confidence of the warfighter.” On May 24, 2019, MDA directed the GMD prime contractor, Boeing, to stop all work for the RKV. This action occurred a few days before the issuance of our report and, as such, we were not able to assess the effects and incorporate this information into our report. Key findings for Fiscal Year 2018 Targets program met some of its fiscal year 2018 goals. Target availability will be a risk for the Missile Defense Agency's aggressive test schedule through 2021. Medium Range Ballistic Missile T1/T2 target's continued cost growth and schedule delays have led to limited testing. The Missile Defense Agency’s (MDA) Targets and Countermeasures program (hereafter referred to as Targets program) procures missiles to serve as targets during the developmental and operational testing of independent or integrated ballistic missile defense system (BMDS) elements. Specifically, this program supplies MDA with short-, medium-, intermediate-, and intercontinental-range targets to test, verify, and validate the BMDS elements’ performance in threat relevant environments. As targets are solely test assets, they are not operationally fielded. The number of targets that the program supplies vary based on each element’s requirements and testing schedule. While some targets have been used for years, others have been recently added or are now being developed to more closely represent current and future threats. The quality and availability of these targets is instrumental to the execution of MDA’s flight test schedule. Table 14 provides information on the Targets program’s performance in fiscal year 2018. The Targets program delivered four of eight targets as planned for fiscal year 2018, and delayed the remaining targets based on test schedule requirements and developmental complexities. One target, the intercontinental-range ballistic missile, was delayed 9 months, from the third quarter of fiscal year 2018 to the first quarter of fiscal year 2019, to align with changes to the test schedule for the Ground-based Midcourse Defense (GMD) program. The GMD program discovered some software issues with its system during pre-test planning that had to be resolved prior to moving forward with flight test FTG-11, which will use the intercontinental-range ballistic missile. According to Targets program officials, the Targets program requested that the contractor delay the delivery of the intercontinental-range ballistic missile to avoid dealing with sensitive aspects of the target, such as fueling, that would necessitate special storage of the target. The two intermediate-range ballistic missiles for the BMDS-level operational test FTO-03 E1 were delayed from the second quarter of fiscal year 2018 to the first quarter of fiscal year 2019 to accommodate a new test for the Aegis Ballistic Missile Defense (BMD) Standard Missile-3 Block IIA program following the failure of one of its interceptors during flight test FTM-29. MDA’s decision to conduct a new test—FTM 45—to ensure the cause of failure had been resolved created test range and asset availability issues that necessitated delaying the BMDS-level operational test FTO-03 E1, and the targets for the test, to a later point in time. The one medium-range ballistic missile for flight test FTM-31 was delayed due to developmental complexities and test range availability. The Targets program flew a total of six targets in fiscal year 2018 to support MDA’s flight test schedule, including four short-range, one medium-range, and one intermediate-range, all of which performed nominally. The risk of a target malfunction or failure was lower in fiscal year 2018 than it has been in previous years, because all of the targets had flown in flight tests previously (i.e., none of the targets were new). However, the Targets program is currently planning to fly two new medium-range targets in fiscal year 2019, and the flight tests with these targets either precede or are adjacent to other important tests in MDA’ test plan. We have previously reported that, new, untested targets introduce higher risk for malfunction or failure that can mean costly and time-consuming retests. Accordingly, we recommended that MDA add a non-intercept flight test for each new target type to verify its performance and reduce risks for future flight tests. MDA has not implemented this recommendation and has continued to use new targets during flight tests. The Targets program conducted one of two critical design reviews in fiscal year 2018. A critical design review assesses the final design of a target to ensure that it can proceed into production and testing and can meet its stated performance requirements within cost, schedule, and risk. The Targets program conducted a critical design review for the medium- range ballistic missile type 3 configuration two (MRBM T3c2) target in the third quarter of fiscal year 2018. The MRBM T3c2 is a new target that Targets program officials said involves minimal design because it leverages flight-proven hardware and a significant amount of heritage software from the intermediate- and intercontinental-range targets currently in production. However, the Targets program plans to conduct another critical design review for the MRBM T3c2 target in the first quarter of fiscal year 2019 due to the addition of hit detection software which will enable real-time feedback on the target’s impact points. The Targets program did not complete the critical design review for the short- range ballistic missile type four G (SRBM T4-G) in the third quarter of fiscal year 2018, after it had been delayed a year, from the third quarter of fiscal year 2017. The Targets program subsequently delayed the critical design review for the SRBM T4-G target another year, to the third quarter of fiscal year 2019. According to the Targets program, the delay in the critical design review for the SRBM T4-G is due to some technical challenges associated with developing the target and the contractor’s limited staffing and workload. The Targets program may face challenges providing some targets to support MDA’s test schedule due the aggressiveness and volatility of the test schedule. We have previously found that MDA’s test schedule is aggressive, in that it includes too many tests and little to no margin between tests to ensure executability. Thus, when setbacks occur, such as target or system malfunctions, the margin between tests erodes. MDA relieves pressure in its test schedule by delaying and canceling tests instead of including sufficient schedule margin to ensure executability, as we previously recommended. When the schedule slips for one test, there are often reverberating impacts to other tests. Consequently, MDA’s test plan has continued to be volatile, with frequent delays, cancellations and other changes, which make it challenging for the Targets program to manage all of the resources and schedules for its various targets to ensure successful, on-time availability and execution. When targets are not available for testing as planned, the tests either receive substitute targets which can mean trade-offs in the performance aspects demonstrated during the test or the test is delayed, which prolongs the demonstration of systems for the warfighter. One way that the Targets program has tried to ensure the availability of targets for MDA’s aggressive test schedule is through the use of concurrency—overlap between development, testing, and production—for some targets. We have previously reported that some concurrency is understandable, but committing to production before development and testing is complete is a high-risk strategy that often results in performance shortfalls, unexpected cost increases, schedule delays, and test problems. The Targets program is using concurrency for the MRBM T3c2 target. According to the Targets program, it is using concurrency for the MRBM T3c2 target due to the urgent need to support essential testing within MDA’s test schedule. The first flight test with the MRBM T3c2 target is FTM-31, which is scheduled for the fourth quarter of fiscal year 2019. Qualification testing and production are ongoing and scheduled to be completed in April 2019 (third quarter of fiscal year 2019). The target must be delivered in advance of the planned test date to complete final preparations for transport to the test site. Thus, the Targets program has very little to no time to resolve any issues prior to delivering it for FTM-31, as shown in figure 6. According to the Targets program, late completion of qualification testing or failures that result in major redesigns may delay FTM-31, as well as significantly impact the cost and schedule for this target. Another way that the Targets program tries to ensure availability of targets for MDA’s aggressive test schedule is to maintain aggressive delivery schedules for some targets. For example, the Targets program has an aggressive delivery schedule for its intermediate- and intercontinental-range targets through fiscal year 2021. According to the contractor for the intermediate- and intercontinental-range targets, there are specific time-spacing requirements that the contractor needs in order to produce and configure targets for a test in relation to the production and configuration of targets for other tests. The contractor said that these specific time-spacing requirements are needed due to limitations with the testing, storage, movement, and transport of these targets. Specifically, we observed that the facility where these targets go through final assembly prior to use in a flight test can currently hold two fully assembled intermediate-range targets and the component for one intercontinental-range target which is assembled at the launch site due to its size. As shown in figure 7, almost all of the tests through fiscal year 2021 are at risk of the target not being available as planned. One of the most severe risks to target availability is in fiscal year 2020 when an intermediate-range target is scheduled for a test in the third quarter, followed by a test using dual (i.e., two) intermediate-range targets in the following quarter. According the contractor’s specific time-spacing requirements, it needs five months, but the approximate amount of time between these tests is three months. According to the Defense Contract Management Agency (DCMA), if MDA includes multiple intermediate- and intercontinental-range missions in the test plan within close proximity without accounting for the contractor’s specific time-spacing requirements, it will be, at best, very challenging for the contractor, and at worst, unachievable. The Targets program has a target—the medium-range ballistic missile type one/type two (MRBM T1/T2)—that continues to have cost growth and schedule delays, which we have previously reported. However, this target’s costs have continued to be unstable, and despite changes and rebaselines, the contractor has been unable to meet projections. Figure 8 below shows the cost growth from 2014 through 2018. In 2017, the Targets program conducted a review of the MRBM T1/T2 target to address significant cost growth and set new projections. Again, in 2018, the Targets program and the contractor planned to conduct another review to address additional cost growth since the prior year’s rebaseline. Despite relatively steady periods of performance following a rebaseline, DCMA officials believe that this contractor will continue to have cost growth. The DCMA established that some of the root causes for the cost growth are incomplete contract requirements and program requirements changes. Additionally, MDA and DCMA officials have acknowledged that the contractor did not adequately account for the costs associated with this target at the outset. How much cost growth there will be moving forward is unknown. In addition to cost growth, the MRBM T1/T2 target has continued to have schedule delays due to technical failures, which has led to the decision to forego some testing as a cost-cutting and time-saving measure. For example, the contractor’s first flight of this target has been delayed approximately 5 years beyond the original plan, from third quarter fiscal year 2014 to fourth quarter fiscal year 2019. The primary reason for this delay has been an unusually high number of failures during pre-test qualification testing, according to the DCMA. The DCMA believes that the test failures are due to the elimination of sub-section testing, which it understands the program and contractor initiated as a cost-cutting and time-saving measure. According to DCMA, sub-section testing involves piecing together different components of the target and then testing that sub-section before the target is fully assembled. This type of testing can help the contractor isolate any integration issues between components in a specific area of the target. However, DCMA said that the contractor is testing the components and then fully assembling the target. Once fully assembled, they are conducting testing and experiencing the unusually high number of failures. When these types of failures occur, according to DCMA, the contractor conducts root cause analysis to make corrections and resolve the issue; however, DCMA officials noted that there is no commonality in the root causes. Thus, the contractor may not understand what steps to take to resolve the issue and ensure that the target performs as expected during a flight test. It is currently unclear how the MRBM T1/T2 target will perform during upcoming tests, because of the Targets program’s decision to forego some qualification testing and not confirming the target’s performance through a non-intercept test, as we have previously recommended. However, the Targets program stated it considers the MRBM T1/T2 performance a minimal risk because the MRBM T1/T2 is largely based on a prior target’s design which, according to the program, was successfully flown twice. The MRBM T1/T2 is currently scheduled to fly in two critical tests in fiscal year 2019 and 2020. The first is an intercept flight test for the Terminal High Altitude Area Defense (THAAD) program in the fourth quarter of fiscal year 2019, which supports the delivery of an urgent capability to the warfighter. After this first flight test with this target, the next test with this target is MDA’s third and largest operational flight test of the BMDS to-date—FTO-03 E2—with five targets flying simultaneously and, three interacting weapon systems—THAAD, Patriot, and Aegis BMD. This test is currently scheduled for the fourth quarter of fiscal year 2020. Both of these tests are important and the use of this new target in these tests increases the risk that the tests will not go as planned and that retests may be necessary; however, a retest for FTO-03 E2 would be extremely costly and very difficult to replan. Appendix VIII: Terminal High Altitude Area Defense (THAAD) Key findings for Fiscal Year 2018 THAAD met most of its fiscal year 2018 delivery and testing goals. THAAD is rebaselining to address Joint Emergent Operational Needs for Korea. THAAD may face challenges meeting its aggressive flight test schedule through 2021. MDA and Army closer to resolving the impasse regarding the transfer of THAAD. THAAD is a rapidly-deployable, globally-transportable, ground-based system able to defend against short-, medium-, and limited intermediate- range ballistic missile attacks through a threat missile’s middle to end stages of flight. A THAAD battery is comprised of five major components: (1) launchers, (2) a fire control unit, (3) communications system, (4) a radar, and (5) interceptors. The current program of record includes a total of seven batteries and 660 interceptors. THAAD has delivered all seven batteries to the Army for operational use and plans to continue production through fiscal year 2029 for remaining items, such as interceptors and software upgrades. The Army has THAAD batteries deployed in Guam and South Korea. Table 15 provides key fiscal year 2018 THAAD program facts. THAAD met its fiscal year 2018 goals for deliveries and flight testing. THAAD exceeded the number of interceptors it had originally planned to deliver in fiscal year 2018 because it is recovering from a parts quality issue. The parts quality issue was with a connector in the interceptor, and although THAAD stopped interceptor deliveries in order to resolve the issue, it did not stop interceptor production. Consequently, there was a stockpile of interceptors just awaiting a redesigned connector in order to be delivered. We previously reported on this parts quality issue and noted that interceptor deliveries, with the redesigned connector, resumed in April 2017 and interceptor production and deliveries have been steady since. In addition to delivering the interceptors, THAAD delivered the seventh, and final, battery of equipment. The delivery was later than previously planned to accommodate the Army’s operational timelines and a new software upgrade to improve THAAD’s performance against certain threats and in the presence of debris during the intercept of a threat missile. Although THAAD was successful in delivering its planned assets for fiscal year 2018, it only conducted one of two planned non-intercept tests. Specifically, FTX-36 was canceled due to target availability from an external vendor and its objectives were reassigned to FTX-35, which was successfully conducted in April 2018. FTX-35 supported the material release of the THAAD 3.0 software (i.e., it is available for use by the warfighter) and the requirement for interoperability testing. THAAD is in the process of rebaselining from two separate acquisition efforts, known as THAAD I and II, to a single acquisition effort, known as THAAD III, to incorporate changes to address the United States Forces Korea (USFK) Joint Emergent Operational Needs (JEON). The purpose of a rebaseline is to update a program’s established plans (i.e., baseline) due to a change in requirements, costs, or schedule. USFK JEON is a rapid acquisition effort to field ballistic missile solutions within the next 3 years to improve the defensive posture of Korea. Specifically, the USFK JEON’s ballistic missile solutions are focused on improving integration between THAAD and Patriot as shown in figure 9, which could enable the defense of larger areas and more assets and provide the warfighter greater flexibility in planning and executing defensive actions. In fiscal year 2018, THAAD delivered software upgrades that provided the initial integration between THAAD and Patriot to improve their ability to coordinate when engaging a threat missile, in support of USFK JEON. These upgrades were assessed in an April 2018 flight test—FTX-35—that demonstrated interoperability between THAAD and Patriot by exchanging messages over tactical data links while tracking a missile target, and an April 2018 BMDS-level ground test which provided further performance data in a simulation environment. THAAD currently plans to deliver USFK JEON upgrades through fiscal year 2021. We currently have ongoing work related to this and details will be included in future reports. MDA has nearly tripled THAAD’s flight tests—from three to eight— between fiscal years 2019 and 2021 to support both USFK JEON, an urgent operational need for the Army, and interoperability testing. Consequently, the schedule margin between each test has decreased from more than a year to three to six months. According to our best practices for scheduling, a practical amount of schedule margin is needed to account for risks and uncertainties. In addition, schedule margin can provide time to analyze the results from the preceding test and correct any identified issues before moving forward with further testing which may be reliant on the results of the preceding test. We have previously reported that MDA leaves little to no schedule margin in its flight test schedule to ensure executability and the test schedule is success- oriented, in that it does not plan for failures which makes it difficult to absorb test failures when they occur. In addition to the reduced schedule margin between THAAD’s tests, some of its tests in this timeframe are higher risk. For example, one test will be flying a new, untested target which increases the risks for that test, and another test will be the largest and most complex operational test to- date, flying five targets simultaneously. Therefore, the test schedule is aggressive, complex, and is at risk of not being completed as planned. However, THAAD has not identified its flight test schedule as a risk. Also, THAAD officials and an official from DOD’s Director of Operational Test and Evaluation have asserted that the flight test schedule is doable, if everything goes according to plan, and that the biggest risk is fatigue among the personnel supporting the tests. While THAAD has a generally successful record for conducting flight tests, its current flight test schedule includes almost as many flight tests in 3 fiscal years as it did for the prior 9 fiscal years. Figure 10 below details the changes in THAAD’s flight testing from its previous plan to its current plan. In addition to the increase in testing and lack of margin between tests, another risk to THAAD’s flight test schedule is that some tests have not yet been funded, as shown in figure 10 above. Funding is essential to enable the planning and execution of each flight test. While THAAD is tracking the lack of funding for these tests as a risk, there is no mitigation strategy if all testing to support USFK JEON remains unfunded. If a single test is not funded or executed, the Army will perform a risk-based assessment using the available data to decide whether or not to deploy the capability for use by the warfighter. If THAAD does not conduct the testing as planned, it will forego the demonstration and confirmation of capability performance which leaves the warfighter with the decision to either not use the capability or use it with an increased risk that it may not perform as intended. THAAD officials noted, however, that the Army’s decision to deploy a capability is based on multiple sources of data such as laboratory and ground testing, not just flight testing. MDA and the Army are nearing a resolution regarding the transfer of the THAAD and AN/TPY-2 programs to the Army; however, the resolution will likely resemble the current arrangement wherein MDA maintains primary responsibility through production and the Army operates and sustains them. We previously reported that MDA and the Army were at an impasse over the transfer of the THAAD and AN/TPY-2 programs because MDA was willing to transfer them as-is, but the program cannot meet the Army’s mission requirements and it would take an estimated $10.1 billion to do so. Table 16 lists the differences between the programs of record and the Army’s requirements. When MDA was established in 2002, it was tasked with using existing and new technologies to rapidly develop weapon systems for the warfighter, and once mature, the weapon systems were to be handed over to a military service for production, operation, and sustainment. At this point, MDA has some weapon systems where production is either nearing completion or is complete. Consequently, Congress set forth a requirement in the National Defense Authorization Act for 2018 that MDA transfer all programs in production to the military services by 2021, which includes THAAD and AN/TPY-2. As part of this requirement, Congress requested a status report on MDA’s transfer of programs in production to military services not later than December 12, 2018. MDA prepared a report for the Under Secretary of Defense Acquisition and Sustainment who then requested the deadline be extended to June 2019 to enable further analysis and development of a viable option. However, according to program officials, at a March 2018 meeting between MDA and the Army, the Army stated that it prefers that THAAD and AN/TPY-2 remain with MDA. According to officials, they discussed transferring the sustainment only because MDA is best suited to maintain primary responsibility through production in order to integrate the BMDS and keep pace with the threat, as well as protect resources through the budgetary process. In addition to the contact named above, LaTonya Miller, Assistant Director; Matthew Ambrose; Pete Anderson; James Bennett; Jon Felbinger; Kurt Gurka; Helena Johnson; Joe Kirschbaum; Wiktor Niewiadomski; Steven Stern; Brian Tittle; Hai V. Tran; and Alyssa Weir made key contributions to this report. Missile Defense: Some Progress Delivering Capabilities, but Challenges with Testing Transparency and Requirements Development Need to Be Addressed. GAO-17-381. Washington, D.C.: May 2017. Missile Defense: Opportunities Exist to Reduce Acquisition Risk and Improve Reporting on System Capabilities. GAO-15-345. Washington, D.C.: May 2015. Missile Defense: Mixed Progress in Achieving Acquisition Goals and Improving Accountability. GAO-14-351. Washington, D.C.: Apr. 2014. Missile Defense: Opportunity to Refocus on Strengthening Acquisition Management. GAO-13-432. Washington, D.C.: Apr. 2013. Missile Defense: Opportunity Exists to Strengthen Acquisitions by Reducing Concurrency. GAO-12-486. Washington, D.C.: Apr. 2012. Missile Defense: Actions Needed to Improve Transparency and Accountability. GAO-11-372. Washington, D.C.: Mar. 2011. Defense Acquisitions: Missile Defense Transition Provides Opportunity to Strengthen Acquisition Approach. GAO-10-311. Washington, D.C.: Feb. 2010. Defense Acquisitions: Production and Fielding of Missile Defense Components Continue with Less Testing and Validation Than Planned. GAO-09-338. Washington, D.C.: Mar. 2009. Defense Acquisitions: Progress Made in Fielding Missile Defense, but Program is Short of Meeting Goals. GAO-08-448. Washington, D.C.: Mar. 2008. Defense Acquisitions: Missile Defense Acquisition Strategy Generates Results but Delivers Less at a Higher Cost. GAO-07-387. Washington, D.C.: Mar. 2007. Defense Acquisitions: Missile Defense Agency Fields Initial Capability but Falls Short of Original Goals. GAO-06-327. Washington, D.C.: Mar. 2006. Defense Acquisitions: Status of Ballistic Missile Defense Program in 2004. GAO-05-243. Washington, D.C.: Mar. 2005. Missile Defense: Actions Are Needed to Enhance Testing and Accountability. GAO-04-409. Washington, D.C.: Apr. 2004.", "summary": "For over half a century, the Department of Defense (DOD) has funded efforts to defend the United States from ballistic missile attacks. From 2002 to 2017, MDA has received about $142 billion and has requested 46.7 billion through fiscal year 2023 to develop the BMDS. The BMDS consists of diverse and highly complex land-, sea-, and space-based systems and assets located across the globe, including planned sites in Romania and Poland to protect United States forces and allies in Europe. The National Defense Authorization Act for Fiscal Year 2012, as amended, included a provision that GAO annually assess and report on MDA's progress. Among other objectives, this report addresses for fiscal year 2018 (1) the progress MDA made in achieving delivery and testing goals and (2) the extent to which MDA made progress in developing and delivering integrated regional BMDS capabilities. GAO reviewed the planned fiscal year 2018 baselines and other program documentation and assessed them against program and baseline reviews and GAO's acquisition best practices guides, and interviewed officials from relevant agencies. In fiscal year 2018, the Missile Defense Agency (MDA) made progress toward achieving its delivery and testing goals for some of the individual systems—known as elements—that combine and integrate to create the Ballistic Missile Defense System (BMDS). MDA is also making progress testing for integrated capabilities, which are achieved by combining BMDS elements. However, MDA did not meet its planned goals. The figure below shows MDA's progress delivering assets and conducting tests against its fiscal year 2018 plans. MDA delivered a significant integrated capability for defending the United States, meeting a goal set by the Secretary of Defense in March 2013 to increase the inventory of ground-based interceptors by December 2017. Other on-time deliveries included software upgrades and additional assets. However, developmental challenges and testing failures contributed to MDA being unable to deliver all assets as planned. MDA completed four of eight flight tests. MDA successfully conducted testing to support a production decision; however, it was unable to complete its annual test plan due to failures, cancellations, and delays. MDA has delayed the delivery of the BMDS's European Phased Adaptive Approach (EPAA) Phase 3—which is intended to protect allies from Iranian threats—until 2020. Construction contractor issues at the planned Aegis Ashore site in Poland drove the delay. At the same time, testing for EPAA Phase 3 against planned threats has been substantially reduced and other vital testing has been deferred until after delivery. MDA officials consider EPAA testing for Phase 3 delivery complete. However, DOD guidance and acquisition best practices stress the importance of testing to understand the extent of capabilities and how to deploy them. The 18-month delay to EPAA Phase 3 provides MDA an opportunity to conduct additional testing and collect more performance data. This testing could provide the warfighter with more information and confidence in the system's ability to protect our allies against expected ballistic missile threats. GAO is recommending that MDA use the schedule margin afforded by the European Phased Adaptive Approach Phase 3 delay to conduct testing necessary to more thoroughly assess, prior to delivery, the capabilities and limitations of Phase 3 against the expected missile threat. DOD partially concurred with our recommendation. GAO continues to believe the recommendation is valid.", "document_type": "gao"}
{"report": "Federal agencies depend on computerized information systems and electronic data to process, maintain, and report essential information, and to operate and control physical processes. Virtually all federal operations are supported by computer systems and electronic data, and agencies would find it difficult, if not impossible, to carry out their missions and account for their resources without these cyber assets. Hence, the security of these systems and data is vital to public confidence and the nation’s safety, prosperity, and well-being. However, computer networks and systems used by federal agencies can be riddled with security vulnerabilities—both known and unknown. These systems are often interconnected with other internal and external systems and networks, including the internet, thereby increasing the number of avenues of attack. Cybersecurity incidents continue to impact federal entities and the information they maintain. According to DHS’s U.S. Computer Emergency Readiness Team (US-CERT), agencies reported 31,107 information security incidents in fiscal year 2018. These incidents involved several threat vectors, such as web-based attacks, phishing attacks, and the loss or theft of computer equipment, among others. These incidents and others like them can pose a serious challenge to economic, national, and personal privacy and security. Safeguarding federal computer systems has been a long-standing concern, with 2020 marking the 23rd anniversary since GAO first designated information security as a government-wide high-risk area. We expanded this high-risk area to include safeguarding the systems supporting our nation’s critical infrastructure in 2003, protecting the privacy of personally identifiable information in 2015, and establishing a comprehensive cybersecurity strategy and performing effective oversight in 2018. Most recently, we continued to identify federal information security as a government-wide high-risk area in our March 2019 high-risk update. Beginning in fiscal year 2015 and continuing through fiscal year 2019, we made approximately 1,700 information security related recommendations. These recommendations identified actions for agencies to take to strengthen their information security programs and technical controls over their computer networks and systems. Nevertheless, many agencies continue to be challenged in safeguarding their information systems and information, in part, because they have not implemented many of these recommendations. As of the end of September 2019, approximately 650 of our prior information security related recommendations had not been implemented. DHS plays a key role in the cybersecurity posture of the federal government and in the cybersecurity of systems that support the nation’s critical infrastructures. Specifically, FISMA gave DHS responsibilities for administering the implementation of agency information security policies and practices for non-national security information systems, in consultation with OMB. One of DHS’s responsibilities is to issue binding operational directives to federal civilian agencies that align with OMB’s policies, principles, standards, and guidelines. These directives apply to the federal civilian agencies that fall under DHS’s FISMA authorities, but do not apply to national security systems or certain systems operated by the Department of Defense or the intelligence community. See appendix II for a list of agencies to which the directives apply. In introducing the authority to issue binding operational directives, the Senate report accompanying FISMA 2014 noted that OMB would continue to have federal information security enforcement responsibilities through its budget powers and its discretion in setting overarching information security policies. Accordingly, OMB has issued several memorandums regarding cybersecurity, including: OMB M-15-01, Fiscal Year 2014-2015 Guidance on Improving Federal Information Security and Privacy Management Practices, required DHS to perform regular scans of public facing segments of federal civilian agency networks for vulnerabilities on an ongoing basis, as well as in response to newly discovered vulnerabilities. OMB has since rescinded this memorandum and replaced it with guidance for fiscal year 2018-2019 (M-19-02). OMB M-15-13, Policy to Require Secure Connections Across Federal Websites and Web Services, requires that all publicly accessible federal websites and web services only provide services through a secure connection using hypertext transfer protocol secure (HTTPS). OMB M-19-02, Fiscal Year 2018-2019 Guidance on Federal Information Security and Privacy Management Requirements, provides agencies with guidance and deadlines to comply with FISMA and reaffirms the value of agencies identifying and prioritizing their high value assets (HVA) as directed by DHS and OMB. OMB M-19-03, Strengthening the Cybersecurity of Federal Agencies by Enhancing the High Value Asset Program, expands the HVA program to support and provide guidance to both Chief Financial Officers Act (CFO Act) and non-CFO Act agencies in HVA identification, assessment, remediation, and incident response. Under M-19-03, an agency may designate federal information or a federal information system as a HVA when it falls under one or more of the following categories: Informational Value. The information, or the system that processes, stores, or transmits the information, is of high value to the federal government or its adversaries. Mission Essential. The agency that owns the information or information system cannot accomplish its primary mission essential functions, as approved in accordance with the National Continuity Policy, found in Presidential Policy Directive 40 (PPD-40), within expected timelines without the information or information system. Federal Civilian Enterprise Essential. The information or information system serves a critical function in maintaining the security and resilience of the federal civilian enterprise. Several entities within DHS have responsibilities for the binding operational directives. The department’s Cybersecurity and Infrastructure Security Agency’s (CISA) Cybersecurity Division is the lead entity for initiating, developing, issuing and overseeing the implementation of the directives. CISA oversees the Federal Network Resilience (FNR) division and the National Cybersecurity and Communications Integration Center (NCCIC) in carrying out specific roles related to the directives. Federal Network Resilience. FNR manages the coordination process for the directives, and oversees implementation of required actions at federal civilian agencies. To do so, FNR collects initial recommendations for new directives, drafts the directives, conducts agency outreach, and tracks agencies’ implementation of the directives. FNR is to collaborate with OMB, NIST, the National Security Council, federal chief information officers (CIOs), and chief information security officers (CISOs) on cybersecurity risk management and operational governance and training; conduct operational assessments for agencies; and assist agencies in identifying areas to improve cybersecurity. National Cybersecurity and Communications Integration Center. NCCIC is the federal civilian coordinator for information sharing concerning cybersecurity risks, incidents, analysis, and warnings with federal and nonfederal entities. The National Cybersecurity Assessments and Technical Services (NCATS), a group within NCCIC, conducts automated network and vulnerability scans of federal civilian agencies’ internet-accessible systems to identify vulnerabilities and configuration errors. Based on these scans, NCATS produces weekly cyber hygiene reports for each agency. The weekly reports describe vulnerabilities detected, affected systems, and mitigation guidance. In addition to the weekly reports, since early June 2019, NCATS has provided agencies with daily notification of any newly detected critical and high severity vulnerabilities. NCATS also conducts reviews of agencies’ high value assets, including security architecture and risk and vulnerability assessments on an ongoing basis. In addition to the DHS components described previously, several other entities assist in coordinating the binding operational directive process. Specifically, DHS’s FNR division coordinates with: Chief information officers and the Federal CIO Council: Federal agencies’ CIOs and the council serve as a source of input for new directives. The council is the principal forum for improving agency practices related to the design, acquisition, development, modernization, use, sharing, and performance of federal information resources. Chief information security officers and the Chief Information Security Officer Council: Federal agencies’ CISOs and the council discuss pending directives. The CISO Council, which is a subcommittee of the Federal CIO Council, collaborates to share information, transfer knowledge, and develop a unified approach to address federal IT security challenges. Small Agency Council: Members discuss pending directives and the potential impacts on small agencies. The council is a voluntary management association representing about 80 small agencies. National Institute of Standards and Technology (NIST): NIST experts are to ensure that binding operational directives do not conflict with NIST standards and guidelines. NIST is responsible for developing standards and guidelines that include minimum information security requirements for federal agencies. To this end, NIST has issued guidance to agencies in implementing an information security program. For example, Security and Privacy Controls for Federal Information Systems and Organizations, NIST Special Publication 800-53, provides guidance to agencies on the selection and implementation of information security and privacy controls for systems. General Services Administration (GSA): GSA coordinates with DHS and OMB, on an as-needed basis, to align cybersecurity services offered in its commercial IT contracts with DHS requirements for assessments, penetration testing, and additional cybersecurity services available to agencies, particularly related to HVAs. DHS developed and issued eight binding operational directives from May 2015 through April 2019 to address known cyber threats, risks, and vulnerabilities. These directives instruct agencies to, among other things: mitigate critical vulnerabilities discovered by DHS’s NCCIC through its scanning of agencies’ internet-accessible systems; better secure their HVAs by participating in risk and vulnerability assessments (RVA) and security architecture reviews (SAR) conducted on their assets; and address several urgent vulnerabilities in network infrastructure devices identified in a NCCIC analysis report. Table 1 provides a list of the directives and their issuance dates. DHS designed a process to develop and oversee the binding operational directives, but it has not followed key components of the process. Specifically, DHS has not involved stakeholders early in directive development and has not consistently overseen agencies’ implementation of some directives through validation of reported results. FISMA requires that DHS develop and oversee the implementation of binding operational directives to safeguard federal information and information systems from a known or reasonably suspected information security threat, vulnerability, or risk and to implement the policies, principles, standards, and guidelines developed by the director of OMB, such as OMB memoranda M-19-03 and M-19-02. Pursuant to FISMA, DHS designed and is using a draft process for developing and overseeing the implementation of cybersecurity binding operational directives. According to CISA officials, the department was to follow this process since issuance of the second directive on securing high value assets (BOD 16-01) in June 2016. In October 2017, DHS documented the process, which it has since updated. According to CISA officials, as of January 2020, this document was still in draft and was undergoing internal agency review. According to the draft process, DHS is to engage in five steps to develop and implement binding operational directives (as discussed below and in more detail in appendix III): 1. Identify a potential directive topic and determine the extent to which it needs to be addressed. DHS’s FNR is to identify topics for new directives from a wide variety of sources, including technical assessments, operational findings of cybersecurity issues, and discussions with external partners such as the Federal CIO Council, NIST, or OMB. FNR is to consider, among other things, whether or not a potential directive topic could be best addressed using the directive process, as well as considering its potential value and impact. Once a topic is identified, FNR officials are to conduct research on the topic and solicit feedback from stakeholders, such as DHS CISA representatives, federal agency chief information officers and chief information security officers, and relevant OMB, NIST, and GSA officials. Once the research is completed, FNR is to make a determination on whether to proceed in developing a directive. 2. Develop a draft directive, send it to relevant stakeholders for review, and obtain approval to issue it. After FNR officials develop the draft directive, they are to send it to relevant stakeholders (e.g. CISA, OMB, NIST, and the DHS Office of General Counsel) for a review of the scope and contents of the directives. FNR staff are to incorporate any feedback from stakeholders into the draft directive and then send it to the CISA director for approval and issuance. 3. Distribute the approved directive to all relevant agencies. FNR officials are to notify agencies of the directive’s issuance via an email and a telephone call within 24 hours of the signing of the directive. In addition, FNR may choose to publicly post the directive to the DHS website. After FNR distributes the directive, agencies are to begin to address the directive’s requirements. 4. Implement and report on agencies’ efforts and progress in addressing the directive requirements. A CISA team is to review agency compliance with the directive through directive-related scans and compliance checks. The team is to distribute scorecards that indicate agency compliance with the directive requirements. 5. Close out the directive. DHS is to close a directive after it has validated that all of the requirements listed in the directive have been completed by all federal executive branch departments and agencies; the directive is no longer necessary because it has been revoked, suspended, or codified into law; or the directive needs to be amended. FISMA requires DHS to consult with NIST, consider NIST’s standards and guidelines, and ensure that the directives it plans to implement do not conflict with NIST’s established standards and guidelines. Consistent with this requirement, DHS’s draft process calls for CISA to coordinate with stakeholders, such as NIST and GSA, early in the directive identification process to incorporate their input as a necessary part of executing the directive process. CISA has not coordinated with key stakeholders early in the development process. According to NIST officials in the Information Technology Laboratory/Computer Security Division, which is responsible for working on directive issues, CISA coordinates with them to ensure that a new directive does not conflict with NIST guidance, but does not do so early in the process. Specifically, the NIST officials stated that often DHS did not reach out to NIST on the most recent directives until 1 to 2 weeks before they were to be issued, and then did not incorporate the NIST technical comments that were provided. As a result of the lack of timeliness in DHS’s outreach to NIST, the directives may not include all key technical considerations. In addition, CISA also has not coordinated with GSA on the directives early in the development process. For example, officials in GSA’s Office of the Chief Information Officer told us that CISA did not coordinate with them on vendor issues before the directive on email and web security was issued. CISA officials acknowledged that, in the past, the agency mainly relied on an ad hoc approach to coordination and did not always coordinate early in the planning process with stakeholders, including NIST and GSA, even though early coordination is called for in the current DHS process. CISA officials also explained that, in certain circumstances, they may need to accelerate the development process when a directive needs to be issued quickly due to elevated risk, such as the directive on addressing threats to network devices in response to a specific hacking threat. CISA officials told us that they have begun to have a more formalized coordination process with key stakeholders, including NIST and GSA. NIST officials also noted that DHS and NIST have started regular coordination meetings to discuss directive-related issues earlier in the process. Nevertheless, CISA has yet to determine when in the directives’ development—for example, during early development and at directive approval—coordination with specific entities should occur. Until CISA addresses this, a lack of effective coordination with stakeholders in the early stages of directives’ development process and later in implementation is likely. This could result in directives that do not fully address key technical considerations, leaving agency systems at risk of being exposed to threats or vulnerabilities. FISMA requires DHS to oversee agencies’ implementation of its binding operational directives. To do this, DHS has outlined a process for validating agencies’ reported results as part of the Close Out step of its directives process. As part of this process, CISA is supposed to validate that agencies have addressed all requirements before a directive is considered to be fully implemented. Guidance from OMB and executive orders also emphasize using a risk-based approach to information security. Specifically, to protect against cyber threats, agencies must make decisions about how to most effectively secure their systems and data, based on an assessment of the risks they face. CISA has not validated agencies’ actions on all five selected directives. Specifically, the agency validated the implementation of two directives by using cyber hygiene scanning and provided weekly reports to the 99 executive branch civilian agencies. However, for the three other directives, CISA relied on agencies to self-report implementation and did not independently validate that the requirements had been met. According to CISA officials, the agency had to rely on agency submissions for these three directives because many of the potentially impacted devices were inside the agencies’ networks and were not visible to CISA’s scans, or were weaknesses identified in specific information security processes that CISA could not assess via scanning. For example, one directive required agencies to address vulnerabilities in specified network infrastructure devices internal to the network and then report to CISA either (1) completion of the actions, or (2) a plan of actions and milestones to complete the actions. The officials added that it is the agency’s responsibility to manage its own plan of actions and milestones, including verifications, and that they are not able to independently validate all of the actions because of a lack of an automated mechanism to detect findings inside agency networks and the lack of resources to do manual assessments. While we recognize that CISA does not have the automated tools or capacity to independently validate every self-reported action taken by agencies to meet binding operational directive requirements, CISA can take a risk-based approach to validation. Guidance from OMB and executive orders emphasize risk-based approaches to information security. However, CISA did not take a risk-based approach, and it also did not have a strategy in place to check selected agency-reported actions to validate their completion. Without taking such an approach or having a strategy in place, the likelihood for requirements to not be completely or correctly addressed is increased. This could leave computer networks and systems used by federal agencies riddled with security vulnerabilities—both known and unknown. Agencies’ implementation of the directives has resulted in improvements that better safeguard federal information systems from a known or reasonably suspected information security threat, vulnerability, or risk. For example, according to DHS and agency data, in response to the directive on Critical Vulnerability Mitigation (BOD 15-01), agencies were able to mitigate about 2,500 out of about 3,600 critical vulnerabilities within 30 days of detection. However, not all agencies had been able to address all the directives’ requirements within the required timelines established in four out of the five directives we reviewed. Moreover, DHS faced constraints in implementing the HVA program. Agencies and DHS cited a number of reasons for not fulfilling the requirements, including a lack of resources and technical expertise, as well as vendor constraints and operational issues. The five directives are discussed below and in more detail in appendix IV. The civilian executive branch agencies to which the five selected binding operational directives apply are implementing and reporting on the requirements as called for in the directives. These five directives identify specific requirements to address known cyber threats, risks, and vulnerabilities and time frames for agency compliance, as well as requirements regarding how agencies are to report their progress on implementation of each directive to DHS. However, not all agencies are doing so within the directives’ established timelines (see directive details that follow). Issued on May 21, 2015, BOD 15-01, Critical Vulnerability Mitigation directed agencies to mitigate critical vulnerabilities discovered by DHS’s NCCIC through cyber hygiene scans of agencies’ internet-accessible systems. Agencies were to mitigate critical vulnerabilities within 30 days of NCCIC’s notification. If agencies were unable to mitigate critical vulnerabilities within 30 days, they were to provide plans and status updates to DHS on a monthly basis until each vulnerability was fully addressed. According to DHS and agency data, since the directive issuance in 2015, the federal civilian agencies were able to mitigate about 2,500 out of about 3,600 critical vulnerabilities within 30 days of detection. Specifically, according to NCATS data, as of May 2018, the median number of days agencies were taking to mitigate critical vulnerabilities from the point of initial detection had been reduced from approximately 16 days (May 2015 to May 2016) to 6 days (from May 2017 to May 2018). In addition, the agencies increased the percentage of critical vulnerabilities closed within 30 days of initial detection, from about 58 percent (May 2015 to May 2016) to 85 percent (from May 2017 to May 2018). See table 2 for more information on the critical vulnerability mitigation timeframes. In its fiscal year 2017 report to Congress on federal cybersecurity directives, DHS reported that the agencies were able to address vulnerabilities more quickly due, in part, to DHS setting clear expectations and timelines regarding mitigating critical vulnerabilities through its directive. Prior to the directive, there was no requirement for patching critical vulnerabilities within a certain time frame. As a result of the faster vulnerability mitigation, agencies are reducing the time their systems and networks are exposed to the cybersecurity risks associated with critical vulnerabilities. In addition to the federal civilian agencies’ improvements in critical vulnerability mitigation, the 12 selected agencies showed improvement in the average time needed to mitigate critical vulnerabilities. Specifically, in the third year after the directive issuance, according to NCATS data, four of the 12 selected agencies reported no critical vulnerabilities and five agencies reported a reduction in the average time needed to mitigate them. For example, one agency reduced the time it took to mitigate critical vulnerabilities from about 60 days to about 17 days on average. Further, all of the 12 selected agencies increased the percentage of critical vulnerabilities closed within 30 days of initial detection, from about 61 percent (from May 2015 to May 2016) to about 90 percent (from May 2016 to May 2017). While all covered agencies did not always meet the 30-day requirement, their mitigations were validated by DHS and reached 87 percent compliance by 2017. Officials attributed the recent decline in percentage mitigated to a 35-day partial government shutdown. Figure 1 provides information on the percent of critical vulnerabilities agencies (federal civilian agencies and the 12 we reviewed) were able to mitigate within 30 days, as required under the directive. In April 2019, DHS rescinded BOD 15-01 and replaced it with BOD 19-02, Vulnerability Remediation Requirements for Internet-Accessible Systems. This directive expands the requirements for agencies from addressing only critical vulnerabilities to addressing both critical and high vulnerabilities. Agencies are now required to mitigate critical vulnerabilities within 15 days of the vulnerabilities being identified through NCATS scanning (rather than within 30 days, as previously required), and to mitigate high vulnerabilities within 30 days of identification. According to the directive, if agencies are not able to mitigate the identified vulnerabilities in the required timeframes, they are to submit a remediation plan to DHS outlining constraints, interim mitigation actions, and estimated completion dates. Issued on September 27, 2016, BOD 16-02, Threat to Network Infrastructure Devices, addressed several urgent vulnerabilities in network infrastructure devices identified in an August 2016 NCCIC report. The report identified a known threat across federal networks and provided technical mitigation solutions. Specifically, it addressed hacking tools targeting firewalls, Cisco Adaptive Security Appliance devices, and devices running Cisco Internetwork Operating System (specifically the integrity of its ROM Monitor program). This directive required agencies to perform all mitigation actions identified in the NCCIC analysis report within 45 days, and to report either full mitigation or provide a detailed plan explaining constraints preventing mitigation. Agencies that were unable to achieve full mitigation within 45 days were instructed to provide monthly status updates until full mitigation was completed across their networks. According to DHS’s March 2019 report to OMB, within 6 months of issuance, the federal civilian agencies were able to remediate approximately 50 percent of impacted devices through patching and through upgrading outdated software. CISA reported that agencies completed all requested actions by October 2018, which was 2 years past the deadline. According to CISA officials, agencies were not able to meet the timeline due to remediation challenges, such as replacing large amounts of end-of-life devices, replacing mission critical devices, and adjusting default configurations on impacted devices. While CISA did not independently validate agencies’ actions in addressing the vulnerabilities as the devices were internal to the network, CISA reported that agencies secured over 11,000 network infrastructure devices across the federal civilian government (see figure 2). In addition to the federal civilian agencies’ status, five of the 12 selected agencies reported full mitigation of the risks outlined in the directive requirements within the 45-day deadline (November 14, 2016). An additional five agencies did not report full mitigation within 45 days, but provided detailed plans of action and milestones to DHS every 30 days thereafter until full mitigation, as required. These five agencies had completion dates ranging from April 2017 to October 2018. The remaining two agencies were unable to demonstrate that they had completed the directive requirements. However, DHS reported that the covered federal civilian agencies were able to complete all actions associated with this directive by October 2018. Issued on September 13, 2017, BOD 17-01, Removal of Kaspersky- branded Products, required federal civilian agencies to (1) determine whether the agency had Kaspersky-branded products on its information systems within 30 days (October 13, 2017); (2) develop a plan to remove such products from its information systems within 60 days (November 13, 2017); and (3) begin implementing its plan for removal within 90 days (December 13, 2017) and provide DHS with updates every 30 days until the products were fully removed from agency information systems. According to DHS’s fiscal year 2017 report to Congress, by April 2018, officials from federal civilian agencies had either attested that Kaspersky- branded products were not present on their information systems or removed such products, as required by the directive. Similarly, officials at the 12 selected agencies stated and reported that they performed the required analysis to identify the use or presence of Kaspersky-branded products and reported to DHS by the 30-day deadline (October 13, 2017). Of these, 10 agencies reported that they did not find the use or presence of Kaspersky-branded products in its information systems. One agency found Kaspersky-branded products in its systems but removed the product before the 60-day planning deadline. The remaining agency identified the use or presence of Kaspersky-branded products in its information systems and developed a detailed plan of action and provided status reports to DHS every 30 days until completion on December 6, 2017. Subsequently, these requirements were enacted into law in the National Defense Authorization Act for Fiscal Year 2018, which further prohibited federal agencies from using products and services developed or provided by Kaspersky Labs. Issued on October 16, 2017, BOD 18-01, Enhance Email and Web Security, required agencies to implement specific security standards that have been widely adopted in industry to ensure the integrity and confidentiality of internet-delivered data, minimize spam, and better protect users who might otherwise fall victim to a phishing email that appears to come from a government-owned system. As such, this directive required several actions related to email and web security with three different due dates: within 90 days (by January 2018), within 120 days (by February 2018), and within 1 year (by October 2018). Tables 3 and 4 outline the email and web security requirements and appendix V provides more detailed information on these requirements. The federal civilian agencies had made significant progress in addressing individual email and web security requirements of the directive. However, few agencies had fully addressed all of the directive’s email and web security requirements for all domains. A domain is a unique identifying address assigned to an internet-accessible system such as .gov or dhs.gov, and an individual agency may have multiple domains. NCATS scans each agency domain and measures it against the individual email and web requirements. According to our analysis of NCATS’ May 2019 scanning data, the agencies were between about 83 to 99 percent complete in addressing each individual email and web requirement across all domains (see figure 3). Similarly, three of the 12 selected agencies, were 100 percent complete in addressing each individual email and web requirement for all domains. In addition, the remaining nine agencies’ domains were from about 82 to almost 100 percent complete in addressing the individual email and web requirements. However, according to NCATS’ March 2018 agency scanning data, only three of 83 agencies (4 percent) had fully addressed all of the directive’s email and web security requirements due within the 120 day deadline across all of their domains. Within 1 year of issuance, according to NCATS’ October 2018 scanning data, six of 83 agencies (7 percent) had fully addressed all directive requirements. According to NCATS’ May 2019 scanning data, three additional agencies fully addressed the requirements. However, three agencies had fallen out of compliance (leaving the total compliance rate at 7 percent). Compliance with the email and web security requirements was slightly better for the 12 selected agencies. According to NCATS’ March 2018 scanning data, one of the 12 selected agencies fully addressed the directive’s requirements due at the 120 day deadline (8 percent). Within 1 year of issuance, according to NCATS’ October 2018 scanning data, one additional agency fully addressed the requirements (17 percent). According to NCATS’ May 2019 scanning data, three of the 12 agencies fully addressed the requirements (25 percent). See figure 4 for details. One of the key challenges that agencies have experienced in implementing the directive’s email requirements is related to strengthening email security by disabling the 3DES weak email cipher. Specifically, according to CISA’s March 2019 report to OMB, more than 50 agencies are dependent on email vendors that do not allow agencies to disable the 3DES cipher. FNR officials stated that after several agencies informed them of having vendor constraints, DHS started to work with vendors on behalf of the agencies. As a result, DHS issued a temporary exception in September 2018, 7 months after the initial deadline, for those agencies encountering this vendor constraint. According to CISA’s March 2019 report to OMB, in February 2019, one of the vendors began retiring the weak email cipher 3DES, but has not set a firm timeline on when it will be fully retired. In a June report to OMB, DHS stated that another email vendor had released a tool that agencies could implement to address the requirement to remove the weak email cipher 3DES. As of the end of April 2019, seven of the 12 selected agencies were affected by this vendor issue. CISA officials noted that they are working with industry officials, including at a leadership level, to ensure they understand when 3DES will be fully disabled. Once that happens, CISA reported that they will provide agencies with any additional support needed to address vendor management issues and the associated email and web requirements. Additionally, FNR officials stated that many agencies struggled to implement a DMARC-related requirement on their systems due to its complexity. FNR officials noted that they have provided agencies with training through a non-profit organization and hosted a variety of outreach events, including presentations, to help agencies work through the complexity of implementing DMARC. Issued on May 7, 2018, the purpose of BOD 18-02, Securing High Value Assets, is to enhance DHS’s approach to secure the federal government’s high value assets (HVAs) from cybersecurity threats. It replaces an earlier directive and requires agencies to: 1. Identify and submit coordination points of contact for HVA assessments within 7 days of issuance of the directive. 2. Submit a current and prioritized HVA list inclusive of all agency components within 30 days of issuance of the directive and review the agency HVA list and provide quarterly updates to DHS. 3. Participate in DHS-led assessments of HVAs, if selected. 4. Ensure identified major or critical weaknesses are mitigated within 30 days of receipt of the risk and vulnerability assessment (RVA) reports and/or security architecture review (SAR); notify DHS that each identified weakness was addressed; and report the status of any remaining major or critical weaknesses to DHS every 30 days until full remediation. As stated earlier, in an RVA, the assessor uses a number of techniques to identify weaknesses in the security posture of a given HVA; for a SAR, the assessor analyzes the architecture of the HVA and develops recommendations for improving HVA security related to system design and interconnections. Techniques for RVA assessments can include network mapping, vulnerability scanning, phishing tests, wireless assessments, web application assessments, and database assessments. A SAR provides a holistic analysis of how an HVA’s individual security components integrate and operate, including how data is protected during operations. According to a DHS report to OMB, assessments can identify HVA weaknesses that require significant network design changes and extended timelines to resolve. In December 2018, OMB issued a memorandum that expanded the definitions of HVAs, instructed agencies to prioritize their HVAs, and instructed agencies to conduct assessments of HVAs as directed by DHS. Subsequently, CISA issued supplemental guidance for BOD 18- 02 that divided HVAs into three tiers based on criticality and impact. The guidance defined Tier 1 systems as systems of critical impact to both the agency and the nation; Tier 2 systems as ones that have a significant impact on both the agency and the nation; and Tier 3 systems as those with a high impact on the agency. In addition, the supplemental guidance outlined the following required reviews: Tier 1 HVAs require one RVA and one SAR to be led by DHS every 3 Tier 2 HVAs require one RVA and one SAR to be conducted by an independent assessor or third party every 3 years, and Tier 3 HVAs require one RVA and one SAR agency self-assessment every 3 years. In response to the directive and supplemental guidance, most of the federal civilian agencies have taken several steps to address the requirements, including identifying points of contact; submitting current and prioritized HVA lists, if appropriate; participating in DHS-led assessments if selected; and beginning to address identified weaknesses. Specifically, CISA’s October 2019 data showed that federal civilian agencies have reported a total of 851 HVAs (212 Tier 1 and 639 Tier 2 and Tier 3 systems). In addition, CISA’s October 2019 data showed that at the beginning of October 2019, DHS had conducted 61 assessments in fiscal year 2018 and 73 in fiscal year 2019. This includes a mix of both RVAs and SARs. DHS has also taken steps to identify major or critical weaknesses from the HVA assessments. Specifically, CISA’s October 2019 data showed that, as of the end of September 2019, the 134 assessments identified 196 major or critical weaknesses. DHS and the agencies have not completed the required assessments and mitigations consistent with OMB guidance and DHS policy. To address the review requirement for Tier 1 HVAs in accordance with the OMB and DHS-defined frequency of assessments, DHS should complete at least a total of 142 assessments a year. However, DHS completed only about half of the required annual assessments this year (with 73 assessments completed in fiscal year 2019). In addition, DHS has yet to issue the guidance, standards, and methodologies for Tier 2 or Tier 3 HVA assessments, which are to be conducted by third parties and agencies, respectively. As a result, agencies cannot begin conducting assessments for the remaining 639 HVA systems. Further, agencies have not been able to mitigate the identified weaknesses within the required timeframes. Specifically, CISA’s October 2019 data showed that of the 196 major or critical weaknesses identified government-wide, agencies were not able to mitigate 160 within the required initial 30-day time frame; 75 major or critical weaknesses were still not mitigated as of early October 2019. Similarly, for the 12 selected agencies we reviewed, CISA’s October 2019 data showed that as of early October, the department performed a total of 58 assessments, which resulted in the discovery of 86 major or critical weaknesses. However, 64 of these major or critical weaknesses were not mitigated within the required initial 30-day time frame, and 32 major or critical weaknesses were still not mitigated as of early October 2019. In addition to the above requirements, DHS established a government- wide performance metric for agencies to address 45 percent of critical/high severity weaknesses discovered through HVA assessments within 30 days of them being reported, as required by the directive. However, DHS reported that agencies were only addressing these weaknesses within 30 days about 30 percent of the time. According to DHS, this shortcoming is largely due to the variety and difficulty of weaknesses identified by affected agencies in each calendar quarter, as well as the different maturity levels of agencies in addressing these weaknesses. Further, the performance metric for addressing the HVA weaknesses is not fully aligned with the directive’s requirements. Specifically, while the directive states that agencies should address weaknesses within 30 days, the directive also states that if the senior accountable officer for risk management at the agency determines that a risk cannot be adequately addressed within 30 days, the agency must develop and submit a remediation plan to DHS for its review. However, DHS’s metric does not provide for such an option. In implementing this directive, DHS recognized the need to measure the extent to which agencies are addressing the requirements and, therefore, improving government-wide cybersecurity. However, without a performance metric that is aligned with the binding operational directive process DHS has established, it will be challenged in demonstrating the overall efficacy of a binding operational directive in achieving cybersecurity goals. Agency and DHS officials reported that agencies faced technical and resource challenges in addressing the various directive requirements within established timelines. This is consistent with challenges reported by officials at the 12 selected agencies. DHS has recognized these challenges and taken actions on them. However, DHS faces a variety of challenges in implementing the HVA program that remain outstanding. Agencies reported various challenges in addressing the directive requirements within the established timelines. The challenges included (1) outdated systems that require costly updates or replacements before they can be brought into full compliance; (2) the lack of specialized expertise to address technical requirements; (3) the complexity of achieving full DMARC compliance; and (4) general issues associated with addressing weaknesses in agency HVAs. To address the first and second challenges (outdated systems and specialized expertise), in its March 2019 report to OMB, DHS provided the following considerations for OMB: (1) examine agency budgets to ensure agencies are deploying all available resources and capabilities against threats to government networks and data; (2) provide supplemental funds to agencies to support implementation of current and future binding operational directives; and (3) examine agency budgets to ensure agencies are deploying all available resources to obtain specialized training for staff or to hire specialized skill sets. According to CISA officials, OMB has contacted agencies that listed budget as a constraint in their plan of action and milestones and is currently discussing how OMB can provide assistance. DHS has also provided support to agencies in addressing the third challenge on DMARC. For example, CISA officials stated that they offer webinars focused on DMARC implementation to those agencies that do not have necessary technical expertise. With regard to the fourth challenge on HVAs, DHS reported that agencies government-wide faced a variety of challenges in addressing the weaknesses in their HVA programs, including issues with network segmentation and vulnerability to phishing attacks. In general, according to DHS, these types of weaknesses may not be easy to address within the required 30 days because they require long term planning and training, system or device procurement, and system integration and testing. The 12 selected agencies concurred with DHS’s view of the challenges they faced in addressing outstanding weaknesses associated with their HVAs. For example, one agency reported an enterprise-level deficiency related to an HVA that requires significant changes to its network design, with a projected remediation timeline of over a year in its plan of action and milestones. Another agency stated that it was unable to fully address a critical weakness within the DHS 30-day timeline, but did develop a remediation plan for the weakness and reported its progress to DHS as appropriate. In addition, another agency reported that it did not fully address a weakness within 30 days and also did not submit the required monthly reports. DHS reported that it has established an HVA Community of Interest with federal civilian agencies to identify and promote best practices within agencies and improve the security and privacy posture of HVA systems. Continued support from OMB and DHS in addressing the technical and resource constraints facing the agencies in addressing the requirements set in the directives will allow agencies to react quickly, efficiently, and effectively to the requirements of the directives. While OMB guidance and DHS policy are clear on DHS’s responsibilities and time frames for the directive on the HVA program (BOD 18-02), DHS has yet to complete its HVA activities in a timely manner. Specifically, the HVA program manager within CISA stated that the department did not have sufficient resources to do all of the required assessments. As noted earlier, thus far, DHS has only conducted about half of the annual assessments required in DHS’s own supplemental guidance. The official stated that the department was now reassessing the prioritization and planning process of the HVA program. Further, CISA officials reported that they do not expect to issue the guidance, standards, and methodologies on Tier 2 and 3 HVAs until at least the end of fiscal year 2020. However, agencies cannot begin conducting Tier 2 third-party or Tier 3 agency self-assessments on HVA systems until DHS develops and issues the guidance, standards, and methodologies for these reviews, potentially leaving these critical systems at risk. Moreover, a CISA official stated that DHS will need to work with GSA to add qualified contractors for Tier 2 assessments to the appropriate GSA contract vehicle. The official stated that there is an ongoing effort with GSA to get contractors for third-party assessments certified by DHS added to the GSA schedule. According to DHS officials from the HVA office, the department is now reassessing key aspects of the program. However, it does not have a schedule or plan for completing this reassessment, or to address outstanding issues on completing required assessments, identifying needed resources, and finalizing guidance to agencies and third parties. Without such a schedule and plan, agencies may continue to face prolonged cybersecurity threats. Although DHS has designed a process to develop and oversee the implementation of binding operational directives, it is not following all the steps in the draft process. Specifically, the department has not involved key stakeholders, such as NIST and GSA, early in the process. Additionally, although guidance from OMB and executive orders emphasize risk-based approaches to information security, CISA did not take such an approach in validating selected agency-reported actions. Until DHS addresses the coordination and validation issues, the likelihood is increased that directives will not fully address key technical considerations and requirements are not fully addressed. Federal civilian agencies have made many significant improvements in cybersecurity by implementing the directives’ requirements. However, an important performance metric for addressing vulnerabilities identified by HVA assessments does not align with the process DHS has established. Further, DHS has only completed about half of the required assessments for fiscal year 2019. In addition, DHS does not plan to issue the guidance, standards, and methodologies on Tier 2 and 3 systems until at least the end of fiscal year 2020. Given these shortcomings, DHS has been reassessing key aspects of the HVA program. However, there was no schedule or plan for completing the HVA reassessment and for addressing the outstanding issues on completing the required assessments, identifying needed resources, and finalizing guidance for Tier 2 and 3 systems. Without such a schedule and plan, agencies may continue to face increased and prolonged cybersecurity threats. We are making four recommendations to the Department of Homeland Security: The Secretary of Homeland Security should determine when in the directive development process—for example, during early development and at directive approval—coordination with relevant stakeholders, including NIST and GSA, should occur. (Recommendation 1) The Secretary of Homeland Security should develop a strategy to independently validate selected agencies’ self-reported actions on meeting binding operational directive requirements, where feasible, using a risk-based approach. (Recommendation 2) The Secretary of Homeland Security should ensure that the binding operational directive performance metric for addressing vulnerabilities identified by high value asset assessments aligns with the process DHS has established. (Recommendation 3) The Secretary of Homeland Security should develop a schedule and plan for completing the high value asset program reassessment and addressing the outstanding issues on completing the required high value asset assessments, identifying needed resources, and finalizing guidance for Tier 2 and 3 HVA systems. (Recommendation 4) We provided a draft of this report to DHS for review and comment. We also provided informational copies of the report to the other agencies involved in the review: OMB; NIST; the Departments of Education, the Interior, Justice, and the Treasury; the Federal Deposit Insurance Corporation; the Federal Retirement Thrift Investment Board; the General Services Administration; the National Aeronautics and Space Administration; the Securities and Exchange Commission; the Social Security Administration; and the Tennessee Valley Authority. In written comments (reproduced as appendix VI), DHS agreed with our recommendations and described steps planned or under way to address them. For example, in its written response, DHS noted that the department is working to formalize a risk-based strategy to validate agency results with an estimated completion date of September 30, 2020. It also added that the department is working with OMB to address the need for independent validation. DHS and NIST also provided technical comments on the draft report, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Acting Secretary of Homeland Security. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6240 or at dsouzav@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. Our objectives were to evaluate (1) the Department of Homeland Security’s (DHS) process for developing and overseeing the implementation of binding operational directives (directives) and (2) the effectiveness of the directives, including agencies’ implementation of directive requirements. To address our first objective, we reviewed DHS documentation, including its policies and process information related to departmental development, approval, and coordination of the directives. We also reviewed DHS written requirements and process for overseeing how agencies are implementing the directives. In addition, we reviewed requirements from law and guidance including the Federal Information Security Management Act of 2014 (FISMA), and memoranda from the Office of Management and Budget (OMB). We evaluated DHS’s process against these requirements. Further, we interviewed officials from DHS, OMB, and National Institute of Standards and Technology (NIST) to obtain their views and verify the information provided. To address our second objective we selected five binding operational directives that had active requirements at the time we were designing our review and analysis in December 2018. These were: BOD 15-01, Critical Vulnerability Mitigation Requirement for Federal Civilian Executive Branch Departments and Agencies’ Internet- Accessible System, issued May 21, 2015. (This directive was revoked and replaced by BOD 19-02, Vulnerability Remediation Requirements for Internet-Accessible Systems in April 2019.) BOD 16-02, Threat to Network Infrastructure Devices (designated as closed by DHS, March 2019), issued September 27, 2016 BOD 17-01, Removal of Kaspersky-branded Products, issued BOD 18-01, Enhance Email and Web Security, issued October 16, BOD 18-02, Securing High Value Assets, issued May 7, 2018 We then randomly selected a sample of 12 agencies from the civilian executive branch agencies, to which DHS directives apply, to determine the extent to which these agencies have taken steps to address the directives’ requirements. Specifically, we randomly selected agencies from among those that had reported actual cybersecurity expenditures of over $30 million in fiscal year 2017 (the most recent data available at the time we began our review). The 12 selected agencies were (1) Department of Education; (2) Department of Homeland Security; (3) Department of the Interior; (4) Department of Justice; (5) Department of the Treasury; (6) Federal Deposit Insurance Corporation; (7) Federal Retirement Thrift Investment Board; (8) General Services Administration; (9) National Aeronautics and Space Administration; (10) Securities and Exchange Commission; (11) Social Security Administration; and (12) Tennessee Valley Authority. We developed a data collection instrument based on the directives’ requirements. We administered the data collection instrument to the selected agencies and collected supporting documentation, such as compliance reports, corrective plans of action/plans of actions and milestones, and remediation plans and responses to the requirements outlined in five directives (15-01, 16-02, 17-01, 18-01, and 18-02). In addition, we reviewed the directives and other relevant requirements as well as DHS’s process for evaluating agency actions to address the requirements and to develop binding operational directive-related performance metrics. We also reviewed DHS’s fiscal years 2018 and 2019 annual performance reports and quarterly performance report updates, fiscal year 2019 reports to OMB, and fiscal years 2016 and 2017 reports to Congress on agencies’ (government-wide) implementation status of binding operational directives. We assessed steps DHS was taking to measure agencies’ performance against DHS’s established metrics. Specifically, we reviewed the 99 civilian executive branch agencies’ and 12 selected agencies’ performance against the specific directives requirements. We analyzed agency documentation, including status reports and plans of action and milestones, as well as scanning data from the National Cybersecurity and Communications Integration Center for both selected agencies and government-wide. We also reviewed DHS performance reports regarding the extent to which DHS’s government-wide performance metrics for mitigation of vulnerabilities on internet-facing systems and for closure of certain vulnerabilities on high value assets align with agencies’ existing requirements from OMB and DHS, such as closure timelines of selected types of vulnerabilities and weaknesses. We compared these performance reports and metrics with existing requirements found in DHS’s directives to assess whether they were aligned. In addition, we reviewed detailed scanning data and output from a data analysis tool from DHS’s database to determine the extent to which the 99 civilian executive branch agencies and our selected 12 agencies are mitigating vulnerabilities on internet-accessible systems and whether or not they are being mitigated within given timeframes. In addition, to analyze the implementation of email and web security requirements, we reviewed detailed scanning data on the status of the 99 civilian executive branch agencies and our selected 12 agencies. To assess the reliability of the scanning data and related DHS analysis that we used to support the findings in this report, we interviewed agency officials to determine the steps taken to ensure the integrity and reliability of the data and reviewed relevant documentation to substantiate the evidence obtained through interviews with agency officials. We determined that the data used in this report were sufficiently reliable for the purposes of our reporting objectives. We supplemented our analyses with interviews of DHS and selected agency officials to obtain their views on the steps they have taken to address the directives’ requirements. We conducted this performance audit from October 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Within the Department of Homeland Security (DHS) Cybersecurity and Infrastructure Security Agency’s Cybersecurity Division, the Federal Network Resilience (FNR) is responsible for managing the process for developing and overseeing the binding operational directives, including coordination and implementation. The process is documented in the department’s draft Cybersecurity Division Binding Operational Directives Process and outlines five steps and their substeps: Step 1: Identify and Determine. This step includes three substeps—1.1 triggers, 1.2 business case development, and 1.3 socialization. The identification of a directive begins with a trigger that identifies a particular topic. The trigger may be from an administrative priority, technical assessment, operational finding, or discussions with external entities such as the Federal Chief Information Officer Council, National Institute of Standards and Technology (NIST), Office of Management and Budget (OMB) or a private sector organization. Once a topic is identified, FNR officials conduct research on the topic and solicit feedback from stakeholders. FNR then directs topics to the Binding Operational Directives Discussion Group. According to the draft process, recommended members of this group include representatives from Cybersecurity and Infrastructure Security Agency (CISA) and ad hoc and external partners, such as OMB officials, federal CIOs and chief information security officers (CISO), NIST officials, and General Services Administration officials. During substep 1.1, the group should decide whether to proceed to substep 1.2, business case development for a directives’ topic. The group maintains an online repository for proposed topics, active directives, and topics that have been previously considered, but archived for future use or historical documentation purposes. During business case development, a lead within the discussion group researches risks, threat actors, and mitigation strategies. The group incorporates information from subject matter experts and programs that provide information on current threats facing agencies and mitigation actions (e.g., Continuous Diagnostics and Mitigation and EINSTEIN). Once drafted, the business case is sent to FNR leadership, such as the Director and Deputy Director, for review. In the socialization substep 1.3, the discussion group may obtain additional feedback through various outreach efforts or through CIO, or CISO Council meetings. Step 2: Develop and Approve. This step includes two substeps—2.1 table top and 2.2 BOD material finalization. In step 2, FNR staff draft the directive. A table top exercise is an optional step that FNR staff may take to test required actions at selected agencies. As part of drafting the directive, the FNR staff coordinates with stakeholders, such as National Cybersecurity Assessments and Technical Services (NCATS), OMB and selected other agencies to develop an action plan template. This template instructs agencies on how to track and submit their progress on a particular directive. In addition, the team drafts a communications plan to disseminate directive-related information to agencies and the public. During substep 2.2, BOD material finalization, the action plan template and communications plan are sent along with the draft directive to all associated stakeholders (e.g. FNR, OMB, NIST, and Department of Homeland Security (DHS) Office of General Counsel) for review. After FNR staff incorporate any additional feedback, the draft directive package is then sent to the CISA Director for signature and then release. Step 3: Distribute. This step includes three substeps—3.1 notification, 3.2 baseline evaluation delivery, and 3.3 begin mandatory actions. According to FNR officials, the approval of a directive is the start of several processes in this step. During substep 3.1, all affected federal civilian agencies receive notification through an email and a directive issuance call within 24 hours of the signing of the directive. In addition, the DHS website (cyber.dhs.gov) and the OMB MAX portal may post the directive depending on the content of the directive. The notification of the directive is followed with agency baseline evaluation delivery, substep 3.2. As part of this substep, the validation team, including representatives from NCATS, may deliver baseline evaluations to provide agencies a better understanding of where they stand in addressing the directive prior to issuance, depending on the nature of the directive. In the last substep 3.3, agencies begin mandatory actions as noted in the directive. Step 4: Implement and Report. This step includes three substeps—4.1 action plan submission, 4.2 continuous coordination, and 4.3 implementation and reporting. The step begins with FNR’s establishment of a Binding Operational Directives Implementation Team to manage the requirements of a specific directive. This team includes a technical lead who reviews and tracks agency plan submissions as part of substep 4.1; a validation team whose members validate agency compliance with the directive; and a data analyst, who is to compile all agency-submitted action plans and draft a monthly status report. According to the draft process document, the validation team conducts directive-related scans and compliance checks, and develops and distributes scorecards that indicate agency compliance with directive requirements. For some directives, such as BODs 16-02, 17-01, and 18- 02, DHS relied on agency self-reporting to confirm that an agency had addressed the requirements, and the validation team did not verify compliance. During substep 4.2 FNR staff and the affected agency maintain continuous coordination through email and phone conversations to address any challenges involved with implementing the directive. Substep 4.3 implementation and reporting consists of processes agencies may need to establish internally to address and report on directive requirements until completion, such as points of contact and methods of communication with FNR. The implementation team produces monthly status reports for FNR leadership, such as the Director and Deputy Director, showing which agencies have complied or not complied with directive requirements. Based upon this information, FNR officials decide whether to escalate instances of agency noncompliance. In addition, FNR officials stated that they have a monthly check-in with OMB, during which they provide status reports as well as conduct less formal weekly discussions. For Congress, CISA produces an annual binding operational directives’ implementation report, in addition to responding to more frequent congressional information requests. To date, DHS has submitted two congressional reports for fiscal year 2016 and 2017. According to FNR officials, as of September 2019, the fiscal year 2018 report is undergoing OMB review. Step 5: Close Out. This step includes two substeps—5.1 results validation and 5.2 setting a higher bar. The draft process document describes the following scenarios that may lead to results validation; if a directive: (1) has been completed by all agencies; (2) is no longer necessary because it has been revoked, suspended, or codified into law; or (3) needs to be amended. In the first scenario, once the validation team affirms that the requirements have been met, FNR officials are to notify affected federal agency officials that their agencies have fulfilled all requirements. FNR officials then draft a binding operational directive completion letter that the Secretary of DHS or the Secretary’s designee signs. According to FNR officials, a directive does not fully close out after the Secretary signs a completion letter, because the directive is still in effect even after agencies have fulfilled all of the particular directive’s requirements. If a directive is revoked or amended, FNR officials draft a letter noting the reasons for such actions which the Secretary of DHS then signs. Agencies are expected to adhere to the newly implemented requirement, which is how DHS describes substep 5.2, setting a higher bar. Figure 1 provides the life cycle of a binding operational directive. The Department of Homeland (DHS) had issued eight binding operational directives (BOD) as of October 2019. A full list of DHS’s directives’ numbers and titles with a summary of their corresponding DHS and agency requirements follows. Agencies or departments are to: Review and mitigate the critical vulnerabilities on their internet facing systems identified by DHS’s National Cybersecurity and Communications Integration Center within 30 days of issuance of agencies’ weekly cyber hygiene reports. Within 30 days will provide a detailed justification to DHS outlining any barriers, planned steps for resolution, and a time frame for mitigation, if unable to mitigate vulnerability. DHS’s Federal Network Resilience Division will work directly with the department or agency to attempt to assist or address any constraints limiting expedited resolution of the vulnerability. DHS’s NCCIC will leverage weekly agency scans to track each department or agency’s progress in mitigating its critical vulnerabilities. DHS will provide quarterly cyber hygiene report updates to the OMB to ensure department and agency results are synchronized with OMB cybersecurity oversight initiatives. Agencies or departments are to: Identify and submit the name of a lead point of contact to DHS’s FNR branch within 7 days of this directive’s issuance. The point of contact will be responsible for coordinating the agency’s high value asset assessments with DHS. (Submission of the same information for at least one backup point of contact is encouraged.) Participate in assessments, mitigation, and remediation activities by: Signing a DHS-provided rules of engagement document authorizing DHS to conduct risk and vulnerability assessments on agency high value assets. Beginning to implement DHS-issued mitigation measures listed in this directive’s appendix for agency high value assets Participating in the high value asset assessments authorized by the rules of engagement. Participating in a security architecture assessment for select high value assets, if requested to do so by DHS. Mitigating the high-priority vulnerabilities identified by DHS in the high value asset final assessment report within 30 days of DHS’s receipt of the report or determine that mitigation is not feasible within that time frame. Providing additional status updates every 30 days until all high- priority vulnerabilities have been addressed. DHS will identify agency high value assets for assessment and report their findings to agencies. DHS will validate whether any relevant protections have been appropriately implemented during each high value asset assessment and will provide the agency with a report on the extent of sufficient implementation. If an agency does not comply with the requirements of this binding operational directive, DHS will follow up with each deputy secretary or equivalent, as appropriate. Agencies or departments are to: Perform all actions in the Solution sections of the technical annexes to the NCCIC Analysis Report AR-16-20173 no later than 45 days after issuance of this directive. Report to DHS, through the OMB MAX Connect Portal, either full mitigation or provide a detailed plan of action and milestones explaining the constraints preventing mitigation and the associated compensating controls established no later than 45 days after issuance of this directive. Provide additional reports or plans of action and milestones every 30 days thereafter until full mitigation is achieved. DHS’s NCCIC will continue to analyze information for additional mitigation steps to protect federal networks and will develop technical annexes in the future under this directive as necessary. If an agency does not comply with the requirements of this directive, DHS will follow up with each deputy secretary or equivalent, as appropriate. Perform all actions in the Solution sections of the technical annexes to the NCCIC Analysis Report AR-16-20173 no later than 45 days after issuance of this directive. Report to DHS, through the OMB MAX Connect Portal, either full mitigation or provide a detailed plan of action and milestones explaining the constraints preventing mitigation and the associated compensating controls established no later than 45 days after issuance of this directive. Provide additional reports or plans of action and milestones every 30 days thereafter until full mitigation is achieved. Agencies or departments are to: Report security incidents to the DHS United States Computer Emergency Readiness Team in accordance with the guidelines found at https://www.us-cert.gov/incident-notification-guidelines, which are updated as necessary. Include metric information from the chief information officer, inspector general, and senior agency official for privacy, detailed in the annual FISMA metrics, in the Fiscal Year 2016 Annual Federal Information Security Management Act Reports, found at https://www.dhs.gov/publication/fy16-fisma-documents. Submit CIO, IG, and privacy metrics by November 10, 2016, to OMB and DHS via CyberScope. View the Fiscal Year 2017 Annual FISMA CIO metrics available at https://www.dhs.gov/publication/fy17-fisma-documents and plan accordingly so they can include these metrics in their Fiscal Year 2017 FISMA Reports. DHS will track submission of Fiscal Year 2016 Annual Federal Information Security Management Act Reports and privacy metrics, and follow up with OMB or the relevant agency to address non-compliance as appropriate. Agencies or departments are to: Within 30 calendar days after issuance of this directive, identify the use or presence of Kaspersky-branded products on all federal information systems and provide a report to DHS that includes: A list of Kaspersky-branded products found on agency information systems. If agencies do not find the use or presence of Kaspersky-branded products on their federal information systems, they should inform DHS that no Kaspersky- branded products were found. The number of endpoints impacted by each product. The methodologies employed to identify the use or presence of the products. Within 60 calendar days after issuance of this directive, develop and provide to DHS a detailed plan of action to remove and discontinue present and future use of all Kaspersky-branded products beginning 90 calendar days after issuance of this directive. Agency plans must address the following elements: Agency name. Point of contact information, including name, telephone number, and email address. List of identified products. Number of endpoints impacted. Methodologies employed to identify the use or presence of the products. List of agencies (components) impacted within department. Mission function of impacted endpoints and/or systems. All contracts, service-level agreements, or other agreements the agency has entered into with Kaspersky. Timeline to remove identified products. If applicable, FISMA performance requirements or security controls that product removal would impact, including, but not limited to data loss/ leakage prevention, network access control, mobile device management, sandboxing/detonation chamber, web site reputation filtering/web content filtering, hardware and software whitelisting, vulnerability and patch management, anti- malware, anti-exploit, spam filtering, data encryption, or other capabilities. If applicable, chosen or proposed replacement products/capabilities. If applicable, timeline for implementing replacement products/ capabilities. Foreseeable challenges not otherwise addressed in this plan. Associated costs related to licenses, maintenance, and replacement (coordinate with agency chief financial officers). At 90 calendar days after issuance of this directive, and unless directed otherwise by DHS based on new information, departments or agencies will begin to implement the agency plan of action and provide a status report to DHS on the progress of that implementation every 30 calendar days thereafter until full removal and discontinuance of use is achieved. DHS will rely on agency self-reporting and independent validation measures for tracking and verifying progress. DHS will provide additional guidance through the federal cybersecurity coordination, assessment, and response protocol following the issuance of this directive. Agencies or departments are to: Within 30 calendar days after issuance of this directive, develop and provide to DHS an agency plan of action for BOD 18-01 to: Enhance email security by configuring within 90 days after issuance of this directive: All internet-facing mail servers to offer STARTTLS, and All second-level agency domains to have valid sender policy framework (SPF)/domain-based message authentication, reporting and conformance (DMARC) records, with at minimum a DMARC policy of “p=none” and at least one address defined as a recipient of aggregate and/or failure reports. Within 120 days after issuance of this directive, ensuring: Secure sockets layer (SSL)v2 and SSLv3 are disabled on mail Triple data encryption standard (3DES) and Rivest cipher 4 (RC4) ciphers are disabled on mail servers (see temporary policy exception for 3DES). Within 15 days of the establishment of centralized NCCIC reporting location, adding the NCCIC as a recipient of DMARC aggregate reports. Within 1 year after issuance of this directive, setting a DMARC policy of “reject” for all second-level domains and mail-sending hosts. Enhance web security by: Within 120 days after issuance of this directive, ensuring: All publicly accessible federal websites and web services provide service through a secure connection (hypertext transfer protocol secure (HTTPS)-only, with HTTP strict transport security (HSTS)), SSLv2 and SSLv3 are disabled on web servers, and 3DES and RC4 ciphers are disabled on web servers. Identifying and providing a list to DHS of agency second-level domains that can be HSTS preloaded, for which HTTPS will be enforced for all subdomains. Upon delivery of its plans of action for BOD 18-01, within 30 days of this directive, departments or agencies will begin implementing their plans. At 60 calendar days after issuance of this directive, departments or agencies will provide a report to DHS on the status of that implementation. They will continue to report every 30 calendar days thereafter until implementation of the agency’s BOD 18-01 plan is complete. DHS will review each agency plan of action for BOD 18-01 after receipt and may contact agencies with concerns. DHS will coordinate the agency-provided lists of domains for HSTS preloading with DotGov. DHS will rely on scanning by its National Cybersecurity Assessments and Technical Services team for tracking and verifying progress with agency compliance with this directive. DHS will notify agencies when the NCCIC establishes a central location for the collection of agency DMARC aggregate reports DHS will provide additional guidance through a DHS coordination call and other engagements and products following the issuance of this directive. Agencies or departments are to: Identify and submit coordination points of contact (POC) for high value asset assessments. If selected to participate in DHS-led HVA assessment, departments or agencies will complete and submit to DHS a single rules of engagement (ROE), and for each HVA and related system(s) to be assessed, one ROE Appendix A titled “Risk and Vulnerability Assessment (RVA) Services for High Value Assets and Related Systems,” authorizing DHS to conduct HVA RVAs on that agency HVA and related systems. Participate in the HVA assessments authorized by the ROE and one or more Appendix A submissions for “RVA Services for High Value Assets and Related Systems.” Participate in a security architecture review (SAR) of each HVA to be assessed. Impose no restrictions on the timing and/or frequency of the assessments, the services to be provided by DHS, or the scope of systems that are part of or related to the HVA being assessed. Ensure timely remediation of identified vulnerabilities and report Within 30 days of receipt of the RVA and/or SAR reports identifying major or critical weakness to an assessed HVA, remediate all major or critical weaknesses and provide notification to DHS that each identified weakness was addressed. If it is determined by the designated senior accountable official for risk management that full remediation cannot be completed within the initial 30-day time frame, develop and submit to a designated DHS email address, a remediation plan for each HVA with remaining major or critical weaknesses within 30 days of the receipt of the RVA and/or SAR reports. This remediation plan shall include justification for the extended timeline, the proposed timeline and associated milestones to remediation (not to exceed 1 year), interim mitigation actions planned to address immediate vulnerabilities, and, if relevant, the identification of constraints related to policy, budget, workforce, and operations. This remediation plan must be signed by the designated senior accountable official for risk management prior to submission to DHS. Report the status of each remaining major or critical weakness to a designated DHS email address every 30 days until full remediation is achieved for all assessed HVAs. Status reports must address RVA and SAR results through combined reporting and must be submitted every 30 days starting 30 days after the submission of the remediation plan described above. Notify DHS at a designated email address and through the monthly status reports of any modifications to remediation plan timelines and when full remediation has been achieved. The notifications for modifications and full remediation must be certified under signature of the designated senior accountable official for risk management. DHS will centrally manage agency progress and report submissions, and will engage each agency head in all cases where the agency has not met the deadlines outlined in the agency/department required actions list. DHS collects, maintains, and prioritizes agency-submitted HVAs, and will notify enterprise chief information officers, chief information security officers, and HVA points of contact of specific HVAs selected for DHS-led assessments based on OMB-led determinations. DHS maintains all agency HVA submissions on HSIN. DHS provisions HSIN accounts for designated agency HVA POCs and provides instruction on HSIN use, as needed. DHS provides standard templates for identifying and submitting agency HVAs and for remediation plans and progress reports. DHS plans and conducts RVAs and SARs for OMB-selected agency HVAs, and provides formal reports containing assessment findings and recommendations to the designated agency HVA POCs. Agencies or departments are to: Ensure access and verify scope. Ensure cyber hygiene scanning access by removing cyber hygiene source internet protocol (IP) addresses from block lists. Within 5 working days of the change, notify the Cybersecurity and Infrastructure Security Agency (CISA) at a designated email address of any modifications to the agency’s internet-accessible IP addresses. This includes newly acquired internet-accessible IP addresses or re-assigned internet-accessible IP addresses that are no longer part of the agency’s asset inventory. Upon request from CISA, departments or agencies will submit updated cyber hygiene agreements to a designated DHS email address. Review and remediate critical and high vulnerabilities. Review cyber hygiene reports issued by CISA and remediate the critical and high vulnerabilities detected on the agency’s internet- accessible systems as follows: Critical vulnerabilities must be remediated within 15 calendar days of initial detection. High vulnerabilities must be remediated within 30 calendar days of initial detection. CISA will monitor federal agency progress and will engage agency senior leadership, such as the chief information security officer, the chief information officer, and the senior accountable officer for risk management, as necessary and appropriate, when the agency has not met the required agency action deadlines specified. CISA also will track the remediation of critical and high vulnerabilities through persistent cyber hygiene scanning and will validate compliance with the directive requirements through these reports. CISA will provide regular reports to federal civilian agencies on cyber hygiene scanning results and current status, and a federal enterprise scorecard report to agency leadership. CISA will provide standard remediation plan templates for federal civilian agencies to populate if remediation efforts exceed required time frames. CISA will engage agency POCs to discuss agency status and provide technical expertise and guidance for the remediation of specific vulnerabilities, as requested and appropriate. CISA will engage agency chief information security officer, the chief information officer, and the senior accountable officer for risk management, throughout the escalation process, if necessary. CISA will provide monthly cyber hygiene reports to OMB to identify cross-agency trends, persistent challenges, and facilitate potential policy and/or budget-related actions and remedies. The report will also ensure alignment with other OMB-led cybersecurity oversight initiative. The scope of Binding Operational Directive (BOD) 18-01, Enhance Email and Web Security, includes complex technical concepts that require background knowledge on various topics for both email and web security. The following information provides more detail on the directive’s technical requirements. When enabled by a receiving mail server, STARTTLS signals to a sending mail server that the capability to encrypt an email in transit is present. While it does not force the use of encryption, enabling STARTTLS makes passive man-in-the-middle attacks more difficult. SPF (Sender Policy Framework) and DKIM (Domain Keys Identified Mail) allow a sending domain to effectively “watermark” its emails, making unauthorized emails (e.g., spam, phishing email) easy to detect. When an email is received that does not pass an agency’s posted SPF/DKIM rules, DMARC (Domain-based Message Authentication, Reporting & Conformance) tells a recipient what the domain owner would like done with the message. Setting a DMARC policy of “reject” provides the strongest protection against spoofed email, ensuring that unauthenticated messages are rejected at the mail server, even before delivery. Additionally, DMARC reports provide a mechanism for an agency to be made aware of the source of an apparent forgery, information that they would not normally receive otherwise. Multiple recipients can be defined for the receipt of DMARC reports. Hypertext Transfer Protocol (HTTP) connections can be easily monitored, modified, and impersonated; Hypertext Transfer Protocol Secure (HTTPS) remedies these vulnerabilities. HTTP Strict Transport Security (HSTS) ensures that browsers always use an https:// connection, and removes the ability for users to click through certificate- related warnings. In 2015, OMB M-15-13, Policy to Require Secure Connections Across Federal Websites and Web Services, required all existing federal websites and web services to be accessible through a secure connection (HTTPS-only, with HSTS). In 2017, the .gov registry began automatically preloading new federal .gov domains as HSTS-only in modern browsers. SSL (secure sockets layer) is a computing protocol that ensures the security of data sent via the internet by using encryption. SSLv2 was released in 1995. Most modern clients do not support SSLv2, but a cross- protocol security bug (DROWN) demonstrated that merely serving SSLv2 enables the inspection of traffic encrypted with the more modern and secure protocol, transport layer security. SSLv3 was released in 1996 and considered to be insecure after a man- in-the-middle exploit (POODLE) was published in 2014. RC4 (Rivest Cipher 4) is a stream cipher algorithm that is used in popular protocols such as SSL (to protect internet traffic) and wired equivalent privacy (WEP) to secure wireless networks. In 2014, NIST marked RC4 as “not approved” for use in federal information systems. 3DES (3 key triple data encryption standard) is an implementation of the data encryption standard (DES) algorithm that uses three passes of the DES algorithm instead of one as used in ordinary DES applications. Triple DES provides much stronger encryption than ordinary DES, but it is less secure than advanced encryption standard. In 2017, NIST urged all users of 3DES to migrate as soon as possible. In addition to the contact named above, Neelaxi Lakhmani (assistant director), Kathleen S. Epperson (analyst-in-charge), Season Burris, Christopher Businsky, Noah Levesque, David Matcham, T. Bruce Rackliff, Karl Seifert, and Priscilla Smith made key contributions to the report.", "summary": "DHS plays a key role in federal cybersecurity. FISMA authorized DHS, in consultation with the Office of Management and Budget, to develop and oversee the implementation of compulsory directives—referred to as binding operational directives—covering executive branch civilian agencies. These directives require agencies to safeguard federal information and information systems from a known or reasonably suspected information security threat, vulnerability, or risk. Since 2015, DHS has issued eight directives that instructed agencies to, among other things, (1) mitigate critical vulnerabilities discovered by DHS through its scanning of agencies' internet-accessible systems; (2) address urgent vulnerabilities in network infrastructure devices identified by DHS; and (3) better secure the government's highest value and most critical information and system assets. GAO was requested to evaluate DHS's binding operational directives. This report addresses (1) DHS's process for developing and overseeing the implementation of binding operational directives and (2) the effectiveness of the directives, including agencies' implementation of the directive requirements. GAO selected for review the five directives that were in effect as of December 2018, and randomly selected for further in-depth review a sample of 12 agencies from the executive branch civilian agencies to which the directives apply. The Department of Homeland Security (DHS) has established a five-step process for developing and overseeing the implementation of binding operational directives, as authorized by the Federal Information Security Modernization Act of 2014 (FISMA). The process includes DHS coordinating with stakeholders early in the directives' development process and validating agencies' actions on the directives. However, in implementing the process, DHS did not coordinate with stakeholders early in the process and did not consistently validate agencies' self-reported actions. In addition to being a required step in the directives process, FISMA requires DHS to coordinate with the National Institute of Standards and Technology (NIST) to ensure that the directives do not conflict with existing NIST guidance for federal agencies. However, NIST officials told GAO that DHS often did not reach out to NIST on directives until 1 to 2 weeks before the directives were to be issued, and then did not always incorporate the NIST technical comments. More recently, DHS and NIST have started regular coordination meetings to discuss directive-related issues earlier in the process. Regarding validation of agency actions, DHS has done so for selected directives, but not for others. DHS is not well-positioned to validate all directives because it lacks a risk-based approach as well as a strategy to check selected agency-reported actions to validate their completion. Directives' implementation often has been effective in strengthening federal cybersecurity. For example, a 2015 directive on critical vulnerability mitigation required agencies to address critical vulnerabilities discovered by DHS cyber scans of agencies' internet-accessible systems within 30 days. This was a new requirement for federal agencies. While agencies did not always meet the 30-day requirement, their mitigations were validated by DHS and reached 87 percent compliance by 2017 (see fig. 1). DHS officials attributed the recent decline in percentage completion to a 35-day partial government shutdown in late 2018/early 2019. Nevertheless, for the 4-year period shown in the figure below, agencies mitigated within 30 days about 2,500 of the 3,600 vulnerabilities identified. Agencies also made reported improvements in securing or replacing vulnerable network infrastructure devices. Specifically, a 2016 directive on the Threat to Network Infrastructure Devices addressed, among other things, several urgent vulnerabilities in the targeting of firewalls across federal networks and provided technical mitigation solutions. As shown in figure 2, in response to the directive, agencies reported progress in mitigating risks to more than 11,000 devices as of October 2018. In addition, GAO reviewed DHS policies and processes related to the directives and assessed them against FISMA and Office of Management and Budget requirements; administered a data collection instrument to selected federal agencies; compared the agencies' responses and supporting documentation to the requirements outlined in the five directives; and collected and analyzed DHS's government-wide scanning data on government-wide implementation of the directives. GAO also interviewed DHS and selected agency officials. GAO is making four recommendations to DHS: (1) determine when in the directive development process—for example, during early development and at directive approval—coordination with relevant stakeholders, including NIST, should occur; (2) develop a strategy for when and how to independently validate selected agencies' self-reported actions on meeting directive requirements, where feasible, using a risk-based approach; (3) ensure that the directive performance metric for addressing vulnerabilities identified in high value asset assessments aligns with the process DHS has established; and (4) develop a schedule and plan for completing the high value asset program reassessment and addressing the outstanding issues on completing the required assessments, identifying needed resources, and finalizing guidance to agencies and third parties. DHS concurred with GAO's recommendations and outlined steps and associated timelines that it planned to take to address the recommendations. Another key DHS directive is Securing High Value Assets, an initiative to protect the government's most critical information and system assets. According to this directive, DHS is to lead in-depth assessments of federal agencies' most essential identified high value assets. However, an important performance metric for addressing vulnerabilities identified by these assessments does not account for agencies submitting remediation plans in cases where weaknesses cannot be fully addressed within 30 days. Further, DHS only completed about half of the required assessments for the most recent 2 years (61 of 142 for fiscal year 2018, and 73 of 142 required assessments for fiscal year 2019 (see fig. 3)). In addition, DHS does not plan to finalize guidance to agencies and third parties, such as contractors or agency independent assessors, for conducting reviews of additional high value assets that are considered significant, but are not included in DHS's current review, until the end of fiscal year 2020. Given these shortcomings, DHS is now reassessing key aspects of the program. However, it does not have a schedule or plan for completing this reassessment, or to address outstanding issues on completing required assessments, identifying needed resources, and finalizing guidance to agencies and third parties.", "document_type": "gao"}
{"report": "An underride crash can occur during a collision between a passenger vehicle and a large truck—a tractor-trailer or a single-unit truck, such as a delivery or dump truck—if the height difference between the vehicles is sufficient to allow the smaller vehicle to slide under the body of the truck. The front and rear of passenger vehicles are designed to crumple in a crash and absorb the main force of an impact, while sensors detect the impact and activate safety features within the passenger compartment, such as air bags and seatbelt pretensioners. However, the point of impact in an underride crash could be the hood of the passenger vehicle or—more severely—the windshield. Such impacts can result in “passenger compartment intrusion” by the large truck into the passenger area of the smaller vehicle. This intrusion can kill passengers or leave them with severe head and neck injuries. Underride guards on large trucks essentially lower the profile of the truck’s body to be more compatible with that of a passenger vehicle. An underride guard designed to withstand the force of a crash can prevent the car from sliding under the truck and provide an effective point of impact that will activate the car’s safety features to protect the car’s occupants. Figure 1 shows images from a video depicting the difference in underride crashes with and without passenger compartment intrusion on the rear of a tractor- trailer. Rear and side underride guards limit a passenger vehicle’s ability to go under those areas of a trailer in a crash (see fig. 2). Front guards— currently used on tractors in some other countries, such as European Union countries—can reduce the likelihood that a truck would ride over a passenger vehicle in a crash, a situation sometimes referred to as “override”. In addition to saving lives and reducing serious injuries, improving traffic safety—including reducing underride crashes—may provide other benefits to society. Specifically, NHTSA has reported that preventing such crashes may result in savings in police and crash investigation resources and reduced property damage, among other things. Federal requirements, in regulations issued by NHTSA and FMCSA, exist for the installation of rear guards on most large trucks, but there are no federal requirements for side or front guards. NHTSA’s mission is to “save lives, prevent injuries and reduce economic costs due to road traffic crashes through education, research, safety standards and enforcement activity.” As part of this mission, NHTSA requires that rear guards be installed on most trailers. Federal regulations requiring rear guards of specific dimensions date back to 1952, but the most current regulations—which set force and energy absorption standards, in addition to dimensional requirements—became effective in 1998. These crashworthy rear guards must be designed and tested to protect occupants in a crash of up to 30 miles per hour. In December 2015, NHTSA published a notice of proposed rulemaking (NPRM) that proposed to align U.S. regulations with stronger Canadian rear guard standards. The Canadian standard includes a stronger energy absorption requirement: 20,000 joules—a measurement of energy—as compared to 5,650 joules in the U.S. NHTSA has not taken action on this NPRM since it was proposed in December 2015. Single- unit trucks that are more than 30 inches above the ground are required to meet the dimensional specifications for rear guards set in 1952 but are not required to meet any force or energy absorption standards. NHTSA introduced an advance notice of proposed rulemaking (ANPRM) in July 2015 that considered requiring rear guards with strength and energy absorption criteria for all newly built single-unit trucks. However, NHTSA has since withdrawn the ANPRM, stating that—based on the comments received as well as analysis of the petitions—the changes being considered were not justified. Although there are no federal requirements for crashworthy side underride guards, some crashworthy side guards are being developed. For example, one aftermarket manufacturer has developed a side underride guard that was crash-tested by IIHS and successfully prevented underride crashes in tests at 35 and 40 miles per hour. Similar looking technologies—including aerodynamic side skirts and pedestrian/cyclist side guards—are installed on some trailers and single- unit trucks, but they are not meant to mitigate underride crashes (see fig. 3). FMCSA’s primary mission is “to reduce crashes, injuries, and fatalities involving large trucks and buses,” and it does this, in part, through developing safety regulations. These regulations include requirements for rear guards for trailers consistent with Federal Motor Vehicle Safety Standards and for single-unit trucks that are more than 30 inches above the ground, as well as for multiple types of commercial vehicle inspections that are performed by, for example, motor carriers and drivers to ensure that commercial vehicles are safely operating. Table 1 describes the types of commercial vehicle inspections. For fatal crashes, including fatal underride crashes, data are collected by law enforcement officials at the location of the crash, aggregated at the state level, and then transferred to NHTSA’s Fatality Analysis Reporting System (FARS). FARS is a census of all fatal traffic crashes in the U.S. When a fatal crash occurs, a state or local police officer typically completes a crash report form unique to each state. These forms can include a variety of data fields, such as the time of the crash, weather conditions, and the number of killed or injured persons. In the case of an underride crash, officers may indicate an underride crash occurred in a specific field for recording this crash type or in a narrative field. FARS analysts—state employees who are trained by NHTSA’s data validation and training contractor to code state crash data for input into FARS—in each state receive and analyze the data in the crash report forms in order to compile a record of the fatal crash. FARS analysts rely on the information within the crash report form in order to enter accurate data. To encourage greater uniformity of crash data, NHTSA, FMCSA, and other agencies and associations cooperatively developed the Model Minimum Uniform Crash Criteria (MMUCC) in 1998. The MMUCC guideline, currently in the fifth edition, identifies a minimum set of motor vehicle crash data elements and their definitions that states should consider collecting, but are not required to collect. The MMUCC is updated about every 4 to 5 years. Prior to publication of each edition, an expert panel from the relevant agencies and associations convenes to review all proposed changes suggested by traffic safety stakeholders to determine what will be included in the MMUCC. According to NHTSA officials, the next updated version of the MMUCC is expected to be issued in 2022. From 2008 through 2017, the annual number of fatalities resulting from underride crashes involving one or more trucks reported in FARS ranged between 189 and 253, resulting in an annual average of approximately 219 fatalities (see table 2). Comparatively, the FARS data show an annual average of about 34,700 total traffic fatalities and approximately 4,000 fatalities involving large trucks over the same period. Therefore, reported underride crash fatalities on average accounted for less than 1 percent of total traffic fatalities and 5.5 percent of all fatalities related to large truck crashes during this time frame. Although reported underride crash fatalities make up a small proportion of total traffic fatalities, NHTSA officials told us that severe underride crashes—involving passenger compartment intrusion—are more likely to result in a fatality or serious injury than crashes in which the passenger vehicle’s safety features engage and are able to protect the occupants. Officials from four state DOTs we spoke to also stated that while underride crashes are not common, the consequences—fatalities or serious injuries, including head or neck injuries—are more likely to be severe. An official from one state DOT noted that their agency did not consider underride crashes to be a high priority issue. However, upon further review of the state’s underride crash data, this official stated that while underride crashes may occur infrequently, they present a higher risk of fatality than the official had previously realized. An official in another state told us they do not regularly review underride crash data but, upon analysis of the data, found that underride crashes constituted a larger percentage than they anticipated—16 percent—of all fatal large truck crashes in the state in 2017. NHTSA’s FARS data show that most of the reported underride crash fatalities occurred when the crash impact was located at the rear or sides of a trailer. From 2008 through 2017, approximately 45 percent (825 of 1836) of reported fatalities in underride crashes with a recorded point of impact on the large truck occurred when the initial impact of the crash was the rear of the trailer. About 32 percent (590 of 1836) of reported underride crash fatalities were in crashes where the side of the trailer was the point of initial impact. Approximately 21 percent (392 of 1836) of reported underride crash fatalities were in crashes with the initial impact at the front of the tractor. These 392 fatalities from crashes involving the front of a tractor could be crashes in which the tractor impacted the rear of a passenger vehicle but might also have occurred in a head-on collision between the car and the tractor. The point of impact for underride crash fatalities with passenger compartment intrusion—the most severe form of underride—had similar distributions, with most reported fatalities occurring when the initial point of impact was the rear or side of the trailer. State and local police officials we interviewed said that the underride crash fatality cases they are familiar with occurred in high speed scenarios, often exceeding 55 miles per hour. For example, officials representing a state police department described scenarios in which passenger vehicles traveling at high speeds rear-ended tractor-trailers stopped on the highway’s shoulder or slowed for highway construction; similar scenarios occurred when tractor trailers failed to slow for stopped traffic and crashed into the rear of passenger vehicles. However, on average, 62 percent of fatalities from underride crashes with passenger compartment intrusion reported in 2008 through 2017 did not include a reported speed. For example, for these fatalities in 2017, 72 percent had speed coded in FARS as missing or not reported. A state and a local police official told us that determining the speed of an underride crash can be challenging due to the often severely damaged condition of the passenger vehicle following an underride crash. Officials representing state police said that they are better able to document whether or not speeding was a factor in an underride crash, rather than an exact speed. IIHS representatives also acknowledged the difficulty in documenting the speed involved in an underride crash, and further stated that this difficulty brings into question the accuracy of the speed data that are recorded in FARS for underride crashes. Stakeholders we interviewed told us that underride crash fatalities are likely underreported in FARS due to several factors, such as variability across states in defining underride crashes, inconsistencies in state crash reporting forms and documentation methods, and limited information provided to state and local police on how to consistently identify and record underride crash data. These factors could contribute to police officers incorrectly and inconsistently documenting underride crash data on the crash report form. As a result, FARS analysts may not have sufficient information to properly categorize the crash as an underride, ultimately affecting the number of underride crash fatalities identified in FARS. Standards for Internal Control in the Federal Government notes that management should use quality information to achieve the entity’s objectives. Underreporting of underride crashes would affect the quality of NHTSA’s data, thereby affecting the agency’s ability to accurately identify the magnitude of underride-related crashes and limiting its ability to make informed decisions on rulemaking or other efforts that would help the agency meet its mission to improve traffic safety. Other researchers and organizations have also commented on the quality of NHTSA’s underride crash data. For example, IIHS representatives told us that they compared underride crash cases in FARS and in NHTSA’s and FMCSA’s Large Truck Crash Causation Study—a study of large truck crashes from 2001 through 2003—and identified some cases that involved underride crashes but that were not categorized as such in FARS. Consequently, IIHS representatives stated that they have used more general rear impact crash data as a proxy for underride crashes due to their finding that underreporting of underride crashes occurs in FARS. Additionally, the University of Michigan’s Transportation Research Institute reported that it can be difficult or impossible to identify underride in available computerized crash data files, such as FARS. State and local police officers do not use a standard definition of an underride crash when collecting data at the scene of a crash. NHTSA officials told us that the agency’s definition for an underride crash—”a vehicle sliding under another vehicle during a crash”—is found in the FARS coding and validation manual, a document primarily used by FARS analysts and researchers. The FARS coding and validation manual further distinguishes underride crashes as those with and without passenger compartment intrusion. The MMUCC, which includes definitions of various crash-related elements, does not include a definition of an underride crash. Among officials from the five state police departments we interviewed, underride crash definitions varied, even within states. For example, in one state, an official from one local police department said that a passenger vehicle would need to have over 50 percent of its hood underneath the trailer to constitute an underride crash, while other officials within the state police used a broader definition consistent with NHTSA’s definition, i.e., a vehicle going underneath another vehicle by any amount. A state police official and a local police official we interviewed indicated that they would like a clearer definition of the conditions that constitute an underride crash to help them better identify these crashes. Further, representatives from NHTSA’s data validation and training contractor told us that when they have identified anomalous patterns in underride crash data in FARS, the main reason for these anomalies has been varying definitions of this crash type, as reporting officers have many interpretations of what constitutes an underride crash. A standard definition of an underride crash, for example in the MMUCC, would provide greater assurance that underride crashes are accurately recorded. While all states have a crash report form to gather data following a crash, these state forms vary in whether and how underride crash-related information is collected. Specifically, for the most recent crash report forms we examined from the 50 states and the District of Columbia, as of October 2018: 17 state forms have a specific field for “Underride.” Eleven of these forms also have data fields for passenger compartment intrusion. 32 state forms have a point of impact or area damaged field for “undercarriage.” The point of impact field is generally intended to be used to indicate the locations of initial impact or area that was damaged for all vehicles involved in the crash. Some state police and transportation officials we spoke with noted that this field could be used to indicate that an underride crash occurred, as the initial point of impact on a large truck could be the undercarriage in such a crash. Two states, California and Hawaii, do not have a data element related to underride crashes or undercarriage on their state crash report forms. The presence of an underride field in state crash report forms may affect the extent to which underride crash fatalities are captured in FARS. For example, we observed that after a state revised its form to remove the underride field, the number of reported underride crash fatalities significantly decreased, potentially indicating that underride crashes were being underreported after the change. Conversely, in another state, we observed that the number of reported underride crash fatalities significantly increased following the addition of an underride field to the crash report form, potentially indicating that underride crashes were being reported more accurately following the change. States have their own discretion to develop crash report forms based on several factors that may be particular to each state. For example, states include or exclude certain data elements on their crash report forms based on the traffic safety priorities within that state. Officials we interviewed from two state police departments told us that they do not have an underride field on their crash report forms because underride crashes are not a traffic safety priority for them. In another state, state DOT officials told us that they chose to include an underride field on the crash report form to better align with the FARS data fields, including those fields related to underride. States may include certain data elements on their crash report form based on the recommended data elements in the MMUCC. However, while the MMUCC was developed to encourage greater uniformity of crash data, its guidelines are voluntary, and it does not currently include references to underride or override crash data elements. In its June 15, 2017, report, the Post-Accident Report Advisory Committee—a group appointed by the FMCSA Administrator to provide input on additional data elements to be included in police accident reports involving commercial motor vehicles—suggested that MMUCC data elements be updated to include a collection of information about whether underride and override are involved in a crash. However, according to the MMUCC’s standard development process and NHTSA officials, to adopt new data elements, the entire MMUCC expert panel—which is comprised of stakeholders representing NHTSA, FMCSA, the Governors Highway Safety Association, states, data collectors, data managers, data users, and safety stakeholders—must reach at least 70 percent agreement for approval of new changes to the MMUCC. Under the MMUCC’s standard development process, the MMUCC expert panel will consider recommendations and proposed changes to the MMUCC guidelines, including those proposed by NHTSA in the months preceding the next MMUCC update in 2022. In states that do not include a specific underride crash field in the state crash report form, state and local police officers we interviewed told us that officers responding to a crash may describe underride crashes in the diagram or narrative fields of the form. However, these officers said that a police officer may inappropriately document an underride crash as a rear impact crash. Similarly, officers may categorize the crash as both an underride and an override crash, which NHTSA’s FARS coding and validation manual indicates would be incorrect. Selected state officials told us that unless the officer documenting the crash specifically describes an underride crash in the narrative field, FARS analysts at the state level who review the crash report forms will not have the information to know if a crash involved underride. Police officers we interviewed in states that include “undercarriage” rather than a specific underride crash field in the crash report form told us that they may use the option as a proxy for an underride crash; however, this field may be used inconsistently. For example, in one state, state police officers said they would select “undercarriage” on the crash report form to reflect an underride crash, whereas a local police officer in the same state said that local officers would not use that field to identify an underride crash occurred and, instead, would document the underride crash in the narrative. NHTSA’s data validation and training contractor told us that it is not a recommended practice for officers to select “undercarriage” as a proxy for underride crashes, noting that this inconsistency could lead to inaccuracies in the resulting FARS data. Including underride as a recommended data field in the MMUCC would provide greater assurance that underride crashes are accurately recorded. State and local police officials we interviewed said that they receive limited or no training on how to identify and record information for underride crashes. Officials from all five state police departments we spoke with said that they develop their own crash reporting training for police. This training emphasizes overall crash reporting with a limited focus, if any, on underride crashes. An official representing one state police office said that the state police provide training on how to complete crash reports and general traffic safety, whereas FARS analysts—often within the state DOT—are concerned with the quality of data collection for data analysis purposes, which is not a primary focus of law enforcement training. State and local police officials we interviewed said they generally have limited to no follow-up or continuous training on crash reporting beyond initial police academy training. Local police we interviewed also told us that while they develop and implement their own crash report training, they may also receive training from the state police. Some state police officers that we spoke with said that they conduct training for local police departments when requested. One local police official we spoke with said that officers have limited exposure to underride crashes in these training sessions and that the average officer would likely not know how to appropriately identify an underride crash. Officials we spoke with from three state and two local police departments stated that additional information to police departments on underride crashes could help improve data collection and overall traffic safety. NHTSA provides training to FARS analysts on reviewing crash report forms and appropriately inputting data in FARS, but does not provide information on crash data collection to state and local police who initially collect the data. According to NHTSA’s data validation and training contractor, the contractor trains FARS analysts on identifying underride crashes. Specifically, the contractor trains FARS analysts to review the crash report forms for sufficient detail to meet the definition of an underride crash and determine if a crash involved underride for entry in FARS. NHTSA officials told us that it is the responsibility of state police academies to train law enforcement officers to conduct on-site investigations and complete crash report forms. NHTSA officials said that they do not currently provide underride identification information directly to state and local police who initially collect the crash data. However, NHTSA does provide information to state and local police on other topics, such as improving traffic safety and driver behavior, for example through DOT’s Enforcement and Justice Services Division. NHTSA officials acknowledged that it would be feasible to also provide information on identifying and recording underride crashes. Standards for Internal Control in the Federal Government notes that management communicates quality information externally through reporting lines so that external parties can help the entity achieve its objectives and address related risks. By providing information to state and local police departments—such as materials or instruction on the definition of an underride crash and how to appropriately document these crashes— NHTSA could improve the quality and completeness of underride crash data that police collect. Underride guards for the rear, side, and front of tractor-trailers and single- unit trucks are in varying stages of development. NHTSA has issued an NPRM proposing to strengthen rear guard requirements for trailers, and estimates that about 95 percent of all newly manufactured trailers already meet the stronger requirements. While FMCSA requires commercial vehicles to be inspected to ensure they are safe, rear guards may not be regularly inspected. Side underride guards are being developed, but stakeholders identified challenges to their use, such as the stress on trailer frames due to the additional weight. NHTSA has not performed research on the overall effectiveness and cost of these guards, and manufacturers we interviewed told us that they are hesitant to invest in developing side underride guards without such research. In response to a 2009 crash investigation, the National Transportation Safety Board (NTSB) recommended that NHTSA require front guards on tractors. NHTSA officials stated that the agency plans to complete research to respond to this recommendation in 2019. However, stakeholders generally stated that the bumper and lower frame of tractors typically used in the U.S. may mitigate the need for front guards for underride purposes. NTSB has further recommended that NHTSA develop standards for crashworthy underride guards for single-unit trucks—such as dump trucks—but NHTSA recently concluded that these standards would not be cost effective. All seven of the eight largest trailer manufacturers—which are responsible for about 80 percent of the trailers on the road in the U.S.—we spoke with told us that they have been building to the stronger Canadian rear guard standard since those requirements became effective in 2007. Some manufacturers said that since trucking company operations may span the border between Canada and the U.S., it was easier to build to a single standard rather than manufacture trailers that comply with either the Canadian requirements or the U.S. requirements. NHTSA is considering strengthening the U.S. requirements for rear guards to align with the Canadian rear guard standards. As part of the 2015 NPRM on strengthening the U.S. requirements to the level of the Canadian standards, NHTSA estimated that 93 percent of all newly manufactured trailers in the U.S. are already equipped with a rear guard that meets the Canadian standard. In July 2018, NHTSA officials told us that figure had increased to 95 percent of all newly manufactured trailers, with the remaining 5 percent from smaller manufacturers who may not wish to incur the additional cost or weight of a Canadian-style rear guard. Trucking industry stakeholders told us that the average lifecycle of a trailer varies: one said the lifespan is 10 to 15 years and another stated a 12-year lifespan. NHTSA performed a cost-benefit analysis as part of the 2015 NPRM in which it preliminarily estimated that requiring newly manufactured trailers to include rear guards built to the new standard would be cost-beneficial. Specifically, NHTSA’s analysis found that the cost of a rear guard that meets the Canadian standard was approximately $500 per trailer, which was $229 more than a guard that complies with the existing U.S. requirement. NHTSA’s analysis also found that a Canadian-style rear guard was heavier than its U.S. counterpart. The rear guard NHTSA studied that complies with current U.S. regulations weighed 172 pounds, whereas those meeting the Canadian standard weighed between 191 and 307 pounds. Regarding benefits, NHTSA estimated in 2015 that— accounting for the trailers that already meet the stronger standard— adopting the Canadian standard would prevent about one fatality and three serious injuries per year. According to DOT, these estimates may have since changed, as a higher percentage of trailers are now manufactured to meet the Canadian standards. Comments on this NPRM varied. Some comments were in support of the measure, citing the safety benefits. Other comments noted that automated driver assistance technology may offer better outcomes. Further, some comments called for NHTSA to take additional steps to improve the safety capabilities of rear guards, such as allowing fewer exemptions from compliance. NHTSA has not taken action on this NPRM since it was proposed in December 2015. NHTSA officials we interviewed could not provide information on when the NPRM would move forward. The largest trailer manufacturers have also taken steps to further improve the design of rear guards to prevent underride crashes in a range of scenarios. Because IIHS found that the weakest points for rear guards are generally the outer edges furthest from the center of the guard, it created a procedure to test the ability of rear guards to withstand crashes at different overlap points, starting at the center of the guard and moving closer to the endpoints. Specifically, this procedure involves three crash tests using full width, 50-percent, and 30-percent overlap of the front of the car with the rear guard, as depicted in figure 4. According to IIHS, as of September 2018, all of the top eight trailer manufacturers operating in the U.S. have successfully passed these tests. Some of these manufacturers provide the improved rear guards as a standard feature on all new trailers, while others offer them as an option for purchase. In addition to strengthening rear guards on trailers, advancements in automatic braking systems in passenger vehicles may help reduce the frequency of underride crashes. These systems, though not federally- required, have been available and installed in some passenger vehicles and tractors and are designed to detect objects or other vehicles in front of the vehicle and automatically apply the brakes to avoid or lessen the severity of an impact. According to NHTSA, twenty automakers representing more than 99 percent of the U.S. automobile market have agreed to make automatic braking systems a standard feature on newly- built passenger vehicles starting in 2022. These braking systems may help reduce the number of passenger vehicles striking the rear of tractor- trailers, potentially reducing the frequency of underride-related crashes, fatalities, and injuries. FMCSA regulations require commercial vehicles operating in interstate commerce to be inspected to ensure they are safe. However, the rules do not specifically include an inspection of the rear guard. After a rear guard has been installed on a new trailer, stakeholders told us that the guard may be damaged during normal use (see fig. 5), for example by backing into loading docks. However, only certain roadside inspections—which are performed at random or if an officer suspects a problem—specifically require the rear guard to be inspected. Specifically, of the eight types of roadside inspections, representatives of the Commercial Vehicle Safety Alliance (CVSA)—which helps develop roadside inspection standards— told us that four require the rear guard to be inspected. Stakeholders we interviewed told us that a trailer could go its entire lifecycle—estimated as typically 10 to 15 years—without ever being selected for a roadside inspection. FMCSA data show that although rear guard violations may be identified during roadside inspections, they constitute a small percentage of all violations. For example, out of about 5.8 million violations identified during roadside inspections in 2017, approximately 2,400, or 0.042 percent, were rear guard violations. In an effort to learn more about rear guard violations, CVSA encouraged commercial vehicle inspectors to specifically focus on rear guards during their roadside inspections performed from August 27 through 31, 2018. According to these data, for the more than 10,000 trailers inspected during that 5-day time frame, about 900 violations (about 28 percent of all violations identified) for rear guard dimensional or structural requirements were identified, including almost 500 instances where the rear guard was cracked or broken, or missing altogether. A CVSA representative stated there was a greater percentage of violations identified because inspectors were asked to specifically focus on the rear guard during this effort. Inspectors performing annual inspections—which can include employees of the motor carrier—rely on a checklist established in FMCSA regulations, known as “Appendix G.” This appendix specifies what equipment must be inspected, such as the brake system, lighting, and wheels. Appendix G does not list the rear guard as an item to be inspected. In August 2018, CVSA petitioned FMCSA to amend Appendix G to include rear guards as an item to be inspected. According to CVSA, in September 2018, FMCSA provided acknowledgment of its intent to review CVSA’s petition. FMCSA’s regulations, including those regarding commercial vehicle inspections, help the agency achieve its safety mission of reducing crashes, injuries, and fatalities. Further, Standards for Internal Control in the Federal Government notes that management should use quality information to achieve the entity’s objectives. Prior to receiving CVSA’s petition to amend Appendix G, FMCSA officials told us that not including rear guards in Appendix G does not affect commercial vehicle safety, as FMCSA regulations require all parts and accessories specified within the regulations—which includes the rear guard—to be in safe and proper operating condition at all times. According to DOT, the agency does not believe that motor carriers are ignoring the application of these regulations to rear guards. However, without explicitly including the inspection of the rear guard in Appendix G, there is no assurance that rear guards in operation will be inspected at least annually to ensure they perform as designed to prevent or mitigate an underride crash. This omission potentially affects FMCSA’s safety mission to help ensure the safe operation of tractor-trailers on the nation’s highways. While not currently required in the U.S., crashworthy side underride guards are being developed which could entail both costs and benefits to society. For example, there is currently one IIHS-crash-tested aftermarket manufacturer of side underride guards in North America, which has sold about 100 sets of side underride guards. According to the manufacturer, the cost of the guards starts at about $2,500 per trailer, though the price could decrease in the future as the manufacturing process becomes more efficient and greater quantities are built and sold. These side underride guards have been crash-tested by IIHS and successfully prevented underride crashes in tests at 35 and 40 miles per hour. As a result, the benefits of such guards might include a reduction in the number of fatalities in underride crashes. The manufacturer estimated that more widespread use of side underride guards would occur over the next 3 to 5 years. However, the manufacturer also said that more information on how side underride guards might affect everyday operations is needed before more widespread adoption by the industry. Additionally, some trailer manufacturers told us that they are in the process of developing side underride guards, but none are currently available for purchase. For example, a representative from one trailer manufacturer developing its own side underride guards estimated that it would be feasible to have these guards designed, tested, and available for sale within the next 2 years. However, the representative said that the manufacturer is hesitant to invest additional resources because of uncertainty about potential future regulatory requirements. Specifically, the manufacturer does not want to invest additional resources to develop a side underride guard that might later have to be redesigned to meet federal requirements, if such requirements were to be established and to differ from the manufacturer’s design specifications. Representatives from several trailer manufacturers, trucking industry organizations, and police departments we spoke with cited challenges with the use of side underride guards that would need to be addressed prior to widespread adoption by the industry. Officials from Canada and the European Union—which also do not require the use of side underride guards that can withstand the force of a vehicle crash—noted similar challenges. Weight: According to the aftermarket side underride guard manufacturer, the side underride guards currently available for sale weigh between 575 to 800 pounds in total. Representatives from two trucking industry organizations we spoke with stated that the additional weight from side underride guards may require carriers to put more trailers on the roads to ship goods in order to stay under federal maximum weight restrictions (generally 80,000 pounds). Federal regulations allow for certain exemptions in the federal weight limits, such as for auxiliary batteries. Some stakeholders also stated that the additional weight from side underride guards would increase fuel costs (assuming all else remains the same) and could put stress on the trailer’s frame, reducing its lifespan and potentially increasing maintenance costs. Road clearance: Some stakeholders we interviewed—including two trucking industry organizations, a tractor-trailer fleet operator, and a trailer manufacturer—stated that side underride guards limit a trailer’s clearance from the ground, which could limit the geographic locations that could be serviced by a trailer or—if the guards drag along the ground—result in damage to the guards or even the trailer. Conditions involving limited clearance could include traveling over raised railroad crossings or navigating sloped loading docks. While aerodynamic side skirts may also drag along the ground in similar conditions, they are more flexible than side underride guards and less likely to damage the trailer. Effects on under-trailer equipment and access: Installation of a side underride guard may limit access to or displace equipment currently underneath a trailer, including spare tires, fuel tanks, and aerodynamic side skirts. Additionally, the rear axles of some trailers can be adjusted to evenly distribute the weight of the trailer’s cargo. For example, trailer manufacturers told us that when the axle is moved to the furthest rear position of the trailer, a fixed-length side underride guard could leave a gap large enough for a car to still have an underride crash. Further, some police officers we interviewed told us that it could be challenging to perform roadside inspections of trailers equipped with side underride guards because the guards could limit access to the underside of the trailer. Representatives from three trucking industry organizations we spoke with indicated that crash avoidance technologies may be more effective than underride guards at minimizing underride crashes, including side underride crashes. However, while these technologies have the potential to mitigate crashes, it is unlikely that they will be available on a more widespread scale in a time frame soon enough to render underride guards unnecessary. While automatic braking systems for passenger vehicles are to become a standard feature on newly built vehicles starting in 2022, IIHS representatives told us that these systems are less effective at detecting and mitigating side crashes than rear or frontal crashes. Specifically, the representatives stated that automatic braking systems would not be effective in situations where the passenger vehicle impacts the side of a trailer at an oblique angle rather than at a perpendicular angle. According to stakeholders we interviewed, it will take a considerable amount of time for the passenger fleet to adopt automated vehicle technologies, with some stating that there will be a mix of automated and non-automated technologies on the nation’s highways for decades—longer than the 3 to 5 years estimated by the side underride guard manufacturer for more widespread use of these guards. NHTSA recently issued a study on the safety performance of certain materials used for side underride guards. However, NHTSA has not performed research on the overall effectiveness and costs associated with or the design of side underride guards. NHTSA’s mission is to “save lives, prevent injuries and reduce economic costs due to road traffic crashes, through education, research, safety standards and enforcement activity.” Additionally, a statement of federal principles on regulatory planning and review indicates that in deciding whether and how to regulate, agencies should assess all costs and benefits of available alternatives, including the alternative of not regulating, and that the agency should base its decisions on the best reasonably obtainable scientific, technical, economic, and other information. Additional research on the effectiveness and cost associated with side underride guards could better position NHTSA to determine whether these guards should be required and, if so, appropriate standards for their implementation. Such research may also help provide information to address the challenges stakeholders cited with side underride guards. In general, there are two types of tractors used in tractor-trailer combinations: conventional tractors, wherein the tractor is lower to the ground and the engine is in front of the cab where the driver sits, and “cab-over” tractors, which are designed so the driver sits atop the engine (see fig. 6). Conventional tractors are generally used in North America, whereas cab-over tractors are used more frequently in the European Union. Since 2000, the European Union has required tractors to include front guards to improve the protection of passengers in cars involved in head- on collisions with tractors. These guards are designed to lower the front profile of a cab-over tractor to be more compatible with that of a passenger vehicle to reduce the potential for underride or override, and to help absorb the force of a collision. Some conceptual designs for front guards on conventional tractors have been proposed by researchers in the U.S., but there are no designs available for purchase or installation as there are for side underride guards. Some research organizations have developed computer models of front guards, but these guards have not been produced for U.S. tractor configurations. Representatives from three trucking associations we spoke with stated that their members were not researching, producing, or installing front guards. A government official from Canada—where the conventional tractor design is also commonly used—said that they did not know of any tractor manufacturers or truck fleets that use front guards. Representatives from a tractor manufacturer that operates in both the U.S. and the European Union told us that front guard designs currently used in the European Union would not be compatible with conventional tractors used in the U.S., stating that these guards would need to be installed in the same space that the bumper, frame, and some equipment—including crash avoidance technologies—already occupy. The design of conventional tractors may mitigate the need for front guards for underride or override purposes, as the lower bumpers and frame make the height of conventional tractors more compatible with passenger cars. A 2013 NHTSA study found that tractors with lower bumper heights were less likely to be involved in an override crash than those with higher bumper heights. Government officials from the European Union told us that they did not see the need for conventional tractors to have front guards, since the lower bumpers essentially function as guards in frontal crashes. Officials from a state DOT, a state police department, and a local police department all stated that they do not see the need for front guards because the tractor is already so low to the ground. Further, state and local officials we spoke with noted that the front underride crashes they have seen often occurred at higher speeds, such as when a truck fails to stop for congested traffic or in a head-on collision at higher speeds. In these cases, the speed combined with the much greater weight of the truck could cause the truck to override the car (in the first scenario) or the car to underride the tractor (in a head-on collision). According to these officials, the force of the crash at those speeds— regardless of whether there was underride or override—would very likely be unsurvivable. Additionally, automatic braking systems in tractors and passenger vehicles may further mitigate the need for front guards for underride or override purposes. These technologies—which, according to a tractor manufacturer we interviewed, have been available and installed in some tractors—can potentially stop a tractor from, for example, overriding a passenger vehicle by automatically applying brakes in situations where a potential rear-end collision is detected. Representatives from a tractor manufacturer told us that about 70 to 80 percent of all newly manufactured tractors it produced are equipped with these braking systems and estimated that more than 50 percent of newly built tractors sold by all manufacturers in the U.S. include these systems. Additionally, front guard researchers we spoke with told us that some front underride guard systems would be optimally effective when paired with automated technologies, such as automatic braking systems. While stakeholders generally agreed that North American tractor designs may mitigate the need for front guards for underride or override purposes, NTSB has called for greater use of front guards. Specifically, in 2010, NTSB recommended that NHTSA, among other things, develop performance standards for front guards and, after doing so, require all newly manufactured trucks weighing more than 10,000 pounds to install these front guards. NTSB issued these recommendations based on its investigation of a June 2009 multi-car crash on an Oklahoma interstate, in which the driver of a tractor trailer failed to slow down for traffic stopped on the roadway. NTSB reported that the tractor-trailer’s high impact speed and structural incompatibility with the passenger vehicles contributed to the severity of the crash. As of December 2018, NHTSA had not implemented NTSB’s recommendations. NHTSA reported to NTSB in 2014 that it was in the process of conducting further examination of crash data, but that efforts in developing standards for front guards are a secondary priority to upgrading rear guard standards. NTSB stated that NHTSA’s response was disappointing and that it continues to believe that NHTSA actions are needed to implement this recommendation. Additionally, NTSB recommended in 2015 that NHTSA develop performance standards and protocols for assessing forward collision avoidance systems in commercial vehicles, which could also help to stop a tractor from overriding a passenger vehicle. According to NTSB, although NHTSA has performed some research on this technology, NTSB has deemed NHTSA’s responses as unacceptable. NHTSA officials told us that the agency anticipates completing relevant research and testing in 2019 that would give the agency the information it needs to make appropriate decisions on next steps related to these NTSB recommendations. FMCSA regulations require rear guards for certain single-unit trucks, such as delivery or dump trucks, that are more than 30 inches above the ground. However, according to representatives of the trucking industry we interviewed as well as NTSB, the wide variety of single-unit trucks makes it challenging to develop a one-size-fits-all requirement for underride guards. Single-unit trucks can vary widely with respect to weight, dimensions, and purpose and can include large pick-up trucks, fire trucks, and dump trucks. The FMCSA regulations exempt certain single-unit trucks—such as those already low to the ground—from the requirement to have a rear guard if the vehicle is constructed and maintained such that the body or other parts of the vehicle provide rear end protection comparable to rear guards required for other single-unit trucks. A trucking industry representative we spoke with said that his association was not aware of any manufacturers currently designing or planning to design crashworthy rear, side, or front underride guards for single-unit trucks due to the variability of single-unit truck design. Some U.S. cities, such as Boston, require pedestrian/cyclist side guards be installed on municipally owned single-unit trucks, but these guards are not designed to mitigate a passenger vehicle underride crash. Research shows that crashes involving single-unit trucks occur less often and are less likely to cause serious injuries and fatalities than those involving tractor-trailers. For example, a 2013 NTSB study of crash data from 2005 through 2009 found that single-unit truck crashes occurred less often, resulted in fewer fatalities, and were less likely to cause serious injuries than tractor-trailer crashes. NHTSA has also acknowledged that single-unit trucks represent the majority of the registered heavy vehicle fleet, but account for a lower percentage—27 percent—of rear end fatalities. To help address fatalities associated with underride crash fatalities involving single-unit trucks, as part of its 2013 study, NTSB recommended that NHTSA develop standards for crashworthy rear, side, and front guards for single-unit trucks, as well as devote efforts to crash avoidance technologies and include more variables in FARS to improve data collection. NTSB also noted that, because of the variability in vehicle design and cargo body styles, safety countermeasures for single-unit trucks would need to be adapted for different truck types to address technical challenges to their implementation. NHTSA published an ANPRM in 2015 that considered requiring rear guards with strength and energy absorption criteria for all newly built single-unit trucks. However, NHTSA subsequently found that the costs of this requirement outweighed the benefits. Comments on this ANPRM varied. For example, the American Trucking Associations stated that it believed NHTSA underestimated the costs associated with installing crashworthy rear guards for single-unit trucks. In contrast, IIHS, in its comments on the ANPRM, questioned NHTSA’s assumptions and stated that the agency was undervaluing the benefits and overestimating the costs. Specifically, IIHS noted that NHTSA overestimated the additional weight of the rear guards, thereby overestimating the cost by about 35 to 40 percent. IIHS also stated that due to concerns with the underlying data, NHTSA underestimated the number of crashes into the rear of single-unit trucks with passenger compartment intrusion. NHTSA officials told us that they disagreed with IIHS’s assessment and stated that the data NHTSA used in the ANPRM were valid and appropriate. The ANPRM also considered requiring single-unit trucks to install red and white retroreflective tape meant to increase the visibility of these trucks, especially in the dark. NHTSA found that this requirement would be cost- effective at preventing or mitigating crashes involving single-unit trucks. However, NHTSA has since withdrawn the ANPRM, stating that—based on the comments received as well as analysis of the petitions—the changes being considered were not justified. The likely underreporting of underride crashes and fatalities due to variability in the data collection process limits NHTSA’s ability to accurately determine the frequency of such crashes. An underride field in MMUCC and additional information from NHTSA on how to identify and record these crashes would provide greater assurance that state and local police officers are accurately reporting data on underride crashes. Such reporting would, in turn, enable NHTSA to better identify and support measures—such as rulemakings and research efforts—to help address this issue. While the stronger rear guards being voluntarily implemented by the largest trailer manufacturers show promise in mitigating the potentially devastating effects of rear underride crashes, rear guards will only be effective if they are properly maintained and replaced when damaged. The lack of specific requirements that rear guards be inspected annually for defects or damage potentially affects the safety of the traveling public and FMCSA’s ability to achieve its safety mission. Finally, designs of crashworthy side underride guards show promise at mitigating underride crashes, but manufacturers may be reluctant to move forward with further development of these types of guards without information from NHTSA on the effectiveness, cost, and implementation standards for these devices. With additional research on resolving the challenges associated with side underride guards, these guards may be closer to being a feasible solution than automated driver assistance technologies designed to prevent or mitigate side impacts that could lead to an underride crash. We are making the following four recommendations to DOT: The Administrator of the National Highway Traffic Safety Administration should recommend to the expert panel of the Model Minimum Uniform Crash Criteria to update the Criteria to provide a standardized definition of underride crashes and to include underride as a recommended data field. (Recommendation 1) The Administrator of the National Highway Traffic Safety Administration should provide information to state and local police departments on how to identify and record underride crashes. (Recommendation 2) The Administrator of the Federal Motor Carrier Safety Administration should revise Appendix G of the agency’s regulations to require that rear guards are inspected during commercial vehicle annual inspections. (Recommendation 3) The Administrator of the National Highway Traffic Safety Administration should conduct additional research on side underride guards to better understand the overall effectiveness and cost associated with these guards and, if warranted, develop standards for their implementation. (Recommendation 4) We provided a draft of this report to DOT for comment. In its written comments, reproduced in appendix II, DOT stated that it concurred with our recommendations. DOT also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Transportation, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Our work for this report focused on truck underride crashes, and the U.S. Department of Transportation’s (DOT) efforts related to this issue. In particular, this report examines (1) the data DOT reports on underride crashes, and (2) the development and use of underride guard technologies in the U.S. For both objectives, we conducted a literature review to identify studies regarding truck safety, in general, and underride guards, in particular, published from 1970 through 2018. We conducted a search for relevant peer-reviewed articles, government reports, trade and industry articles, and think tank publications. Key terms included various combinations of “underride,” “crash,” “collision,” and “guard.” We included those studies that were methodologically sound and covered underride crash data, guard technologies, and benefits and costs relevant to our scope. Additionally, we interviewed and analyzed the perspectives of government officials from DOT, the National Highway Traffic Safety Administration (NHTSA), the Federal Motor Carrier Safety Administration (FMCSA), and the National Transportation Safety Board. We interviewed officials from foreign transportation agencies—Canada and the European Union—that were selected based on our review of literature identified above and recommendations from preliminary interviewees. We also interviewed a variety of relevant non-governmental organizations to gain their perspectives on topics related to underride crashes and guards. These organizations represent a variety of key players in their respective fields on underride crash-related topics. We grouped these entities into the following categories: (1) trailer manufacturers, (2) trucking industry organizations, (3) tractor-trailer fleets and related organizations, (4) traffic safety organizations, and (5) research organizations. We interviewed seven of the top eight trailer manufacturers in the United States, as identified by the Insurance Institute for Highway Safety. We requested an interview with Stoughton Trailers, but they declined to participate. The organizations we contacted as part of this work are listed at the end of this section. We also interviewed NHTSA officials and conducted semi- structured interviews with officials in five selected states, including officials in five state departments of transportation and five state and two local police departments to understand and identify limitations, if any, in how underride crash-related data are collected and analyzed. The results of these interviews are not generalizable to all states and localities; however, they offer examples of the types of experiences state DOTs and police have with underride crashes and inspections. We selected states based on several factors to identify states that were similar in highway traffic trends and large truck-related fatality rates, but collected underride crash data differently. Selection factors included highway vehicle miles traveled per state, total underride crash fatalities by state in 2016 as reported by NHTSA, and the presence of an underride crash data field on each state’s crash report form. Based on these factors, we selected and conducted interviews with state DOT and state police officials in California, Illinois, Indiana, Pennsylvania, and Tennessee. We also corresponded with officials from the Ohio DOT for clarification questions. We interviewed local police departments in Chicago, Illinois and Terre Haute, Indiana. To identify the data DOT reports on truck underride crashes, we analyzed existing DOT data on underride crashes and fatalities from 2008 through 2017, the 10 most recent years for which these data are available. We reviewed DOT documentation for policies and procedures on data collection and data reliability assessments for underride crash-related data. NHTSA fatality data came from the Fatality Analysis Reporting System (FARS). FARS is a census of all fatal traffic crashes in the United States that provides uniformly coded, national data on police-reported fatalities. We analyzed these data to determine the reported number of fatalities involving underride crashes. To assess the reliability of the FARS data, we reviewed relevant documentation and spoke with agency officials about the data’s quality control procedures. We determined that the data were sufficiently reliable for the purposes of this report, specifically to provide a high-level overview of underride crash fatalities within recent years. However, we did identify potential underreporting of underride crashes and fatalities, as discussed in this report. We also reviewed NHTSA’s annual Traffic Safety Facts reports—which use FARS data—to determine the annual number of traffic and large truck crash fatalities from 2008 to 2017, the 10 most recent years for which these data are available. We reviewed state crash report forms from all 50 states and the District of Columbia to understand the variability of underride crash-related data elements and how such variability could affect DOT’s data collection and analysis efforts. We compared NHTSA’s data collection efforts to federal internal control standards related to use of quality information. To describe the development and use of truck underride guard technologies in the United States, we reviewed research and documentation on underride guards. Primarily, we reviewed documents relating to underride guards from NHTSA and FMCSA, as well as information from traffic safety groups, trucking industry organizations, research organizations, and selected foreign transportation agencies. We reviewed NHTSA’s regulations requiring rear guards, FMCSA’s regulations requiring commercial vehicle inspections, DOT’s documentation on underride guard technologies, and DOT data on commercial vehicle inspections. To assess the reliability of DOT’s commercial vehicle inspection data, we reviewed relevant documentation and spoke with agency officials about the data’s quality control procedures. We determined that the data were sufficiently reliable for the purposes of this report, specifically to provide a high-level overview of commercial vehicle inspections within recent years. We compared DOT’s efforts to pertinent agency regulations on commercial vehicle inspections, federal internal control standards related to use of quality information, and a statement of federal principles on regulatory planning and review. We spoke with relevant non-governmental organizations to obtain their perspectives on the perceived benefits and costs of rear, side, and front underride guards, and the potential factors that may influence the benefits and costs. We conducted this performance audit from January 2018 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Susan Fleming, (202) 512-2834 or flemings@gao.gov. In addition to the contact named above, Sara Vermillion (Assistant Director); Daniel Paepke (Analyst in Charge); Carl Barden; Jessica Du; Mary Edgerton; Timothy Guinane; David Hooper; Gina Hoover; Madhav Panwar; Joshua Parr; Malika Rice; Oliver Richard; Matthew Rosenberg; Pamela Snedden; and Michelle Weathers made key contributions to this report.", "summary": "Truck underride crashes are collisions in which a car slides under the body of a truck—such as a tractor-trailer or single-unit truck—due to the height difference between the vehicles. During these crashes, the trailer or truck may intrude into the passenger compartment, leading to severe injuries or fatalities. Current federal regulations require trailers to have rear guards that can withstand the force of a crash, whereas the rear guards required for single-unit trucks do not have to be designed to withstand a crash. There are no federal side or front underride guard requirements. GAO was asked to review data on truck underride crashes and information on underride guards. This report examines (1) the data DOT reports on underride crashes and (2) the development and use of underride guard technologies in the U.S. GAO analyzed DOT's underride crash data for 2008 through 2017; reviewed NHTSA's proposed regulations and research on new guard technologies; and interviewed stakeholders, including DOT officials, industry and safety groups, and state officials selected based on reported underride crash fatalities and other factors. According to crash data collected by police and reported by the Department of Transportation's (DOT) National Highway Traffic Safety Administration (NHTSA), fatalities from “underride” crashes, such as those pictured below, represent a small percentage of all traffic fatalities. From 2008 through 2017, an average of about 219 fatalities from underride crashes involving large trucks were reported annually, representing less than 1 percent of total traffic fatalities over that time frame. However, these fatalities are likely underreported due to variability in state and local data collection. For example, police officers responding to a crash do not use a standard definition of an underride crash and states' crash report forms vary, with some not including a field for collecting underride data. Further, police officers receive limited information on how to identify and record underride crashes. As a result, NHTSA may not have accurate data to support efforts to reduce traffic fatalities. Underride guards are in varying stages of development, and gaps exist in inspection of rear guards in current use and in research efforts for side guards. NHTSA has proposed strengthening rear guard requirements for trailers (the rear unit of a tractor-trailer) and estimates about 95 percent of all newly manufactured trailers already meet the stronger requirements. Although tractor-trailers are inspected, Federal Motor Carrier Safety Administration annual inspection regulations do not require the rear guard to be inspected, so damaged guards that could fail in a crash may be on the roadways. Side underride guards are being developed, but stakeholders GAO interviewed identified challenges to their use, such as the stress on trailer frames due to the additional weight. NHTSA has not determined the effectiveness and cost of these guards, but manufacturers told GAO they are unlikely to move forward with development without such research. Based on a 2009 crash investigation, the National Transportation Safety Board (NTSB) recommended that NHTSA require front guards on tractors. NHTSA officials stated that the agency plans to complete research to respond to this recommendation in 2019. However, stakeholders generally stated that the bumper and lower frame of tractors typically used in the U.S. may mitigate the need for front guards for underride purposes. Regarding single-unit trucks , such as dump trucks, NTSB has recommended that NHTSA develop standards for underride guards for these trucks, but the agency has concluded these standards would not be cost-effective. GAO recommends that DOT take steps to provide a standardized definition of underride crashes and data fields, share information with police departments on identifying underride crashes, establish annual inspection requirements for rear guards, and conduct additional research on side underride guards. DOT concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Space transportation is the movement of objects, such as satellites and vehicles carrying cargo, scientific payloads, or passengers, to or from space. In the United States, commercial space transportation is carried out using orbital and suborbital launch vehicles owned and operated by private companies. Key parties involved in commercial space transportation activities include: The commercial launch provider—the entity that conducts the launch of a vehicle and the payload it carries. The launch customer—the entity that pays the launch provider to carry a payload into space. Customers include the U.S. government— which has not operated its own launch vehicles since the retirement of the Space Shuttle in 2011 and primarily relies on commercial launch providers to, among other things, resupply the International Space Station, launch satellites, and carry out national security and defense missions. Customers also include private companies, such as satellite owners, and researchers. The launch site operator—the entity that hosts the launch (or reentry, or both) of the launch vehicle from its launch site. Almost all launch site operators are either commercial launch providers or state or municipal government entities. The U.S. share of the global commercial space transportation market has grown in recent years. For example, according to FAA, 64 percent of the 33 worldwide commercial orbital launches in 2017 occurred at U.S. launch sites, up from about 48 percent in 2014. Commercial launch providers currently use, and are developing for future use, a variety of vehicles to launch payloads. Historically, launch providers have carried payloads into orbit using vertically launched expendable launch vehicles—those vehicles that launch only once. In more recent years, a launch provider, SpaceX, has introduced launch vehicles that can be reused for multiple launches, such as Falcon 9 and Falcon Heavy, where one part or all of the launch vehicle returns to a landing pad, either on land or on a converted barge offshore, after the payload is launched into orbit. Commercial launch providers are also moving toward reusable suborbital launch vehicles, some intended for human space tourism. These vehicles include horizontal hybrid suborbital launch vehicles, such as Virgin Galactic’s SpaceShipTwo, and vertical reusable suborbital launch vehicles, such as Blue Origin’s New Shepard. Figure 1 depicts examples of expendable and reusable vertical launch vehicles. Launch site infrastructure, and those who own and operate it, also varies across individual launch sites. The type of infrastructure and its design depends on the type of operations that the launch site supports. For example, some launch sites may have a launch pad for vertical launches but not a runway for horizontal launches; others may have infrastructure specifically to support launch vehicle reentry operations. While many different types and designs exist, figure 2 below shows a few examples of major pieces of launch site infrastructure. Within FAA, AST is responsible for regulatory oversight of the commercial space transportation industry. AST’s primary means of oversight is licensing or permitting commercial launch and reentry vehicle operations and non-federal launch sites, as well as conducting safety inspections of licensed launch providers and site operators. AST is organized into three management and support offices, including the Office of the Associate Administrator, and five operational divisions—responsible for the majority of AST’s primary mission areas, such as licensing and overseeing launches. In addition, the FAA Reauthorization Act of 2018, signed into law in October 2018, requires that AST develop an Office of Spaceports. According to FAA officials, as of May 2019, the size and design of this office have not yet been finalized. AST’s workforce size is expected to increase to help accommodate anticipated growth in the industry and AST’s workload (see table 1). As of February 2019, AST had 104 full-time equivalent positions and an operations budget of about $25 million—an increase of 25 full-time equivalent positions and about $8 million since fiscal year 2015. FAA requires launch providers conducting a launch or reentry within U.S. borders to obtain a license or permit, as well as those conducting a launch or reentry abroad, if the launch provider is a U.S. entity. FAA considers a commercial launch to be one in which the contract for the main payload’s launch was open to international competition or the launch was privately financed without government support. FAA also requires, with some exceptions, a site operator’s license, which authorizes an entity, such as a state or local government, to host commercial space launch operations from a specific launch site. FAA is to conduct safety inspections of licensed commercial space transportation launch operations, which involves monitoring of pre-operational, operational, and post operational activities. In February 2018, the National Space Council recommended that DOT update the regulations on launch and re-entry licensing to better accommodate changes that have occurred in the industry. The White House subsequently directed DOT to publish a proposed rule by February 1, 2019, with a revised framework that allows more flexibility in how companies can meet the regulatory requirements. DOT published a notice of proposed rulemaking for the revisions to its licensing regulations in April 2019. Around the mid-20th century, the federal government began constructing the infrastructure that supports the majority of commercial orbital space launches today. The Department of Defense (DOD) constructed launch sites to support ballistic missile testing and satellite launches, including sites that are now home to Cape Canaveral Air Force Station in Florida and Vandenberg Air Force Base in California. Those sites conducted their first test launches in 1950 and 1958, respectively. The National Aeronautics and Space Administration (NASA) was created in 1958, and began acquiring land adjacent to Cape Canaveral Air Force Station in 1962 to support its human spaceflight lunar program; this land is now home to the Kennedy Space Center. In recent years, nearly all FAA-licensed launches in the United States occurred at three federal ranges, which were originally built by the federal government (see fig. 3). All 61 of the FAA-licensed commercial orbital launches from 2015 through 2018 occurred at launch sites that are on or co-located with federal ranges, with 44 of the 61 launches taking place at Cape Canaveral Air Force Station and Kennedy Space Center (collectively referred to as “Cape Canaveral”). In addition, one of the 11 licensed commercial suborbital launches occurred at a launch site co- located with a federal range. While the federal government made the initial infrastructure investment at federal ranges, the launch complexes used for commercial launch operations at these sites are now operated under use agreements by non-federal entities, such as state governments or commercial launch providers. For example, four of the launch complexes at Cape Canaveral are operated by commercial launch providers, while two others are operated by the State of Florida. Two other federal ranges have launch pads that are also operated by non-federal entities—Vandenberg Air Force Base in California and the Mid-Atlantic Regional Spaceport, which is co-located with NASA’s Wallops Flight Facility in Virginia. The Air Force and NASA generally still have responsibility for maintaining common-use infrastructure—that is, infrastructure that may be shared by multiple users, such as access roads and fuel pipelines. As part of the operators’ use agreements (the details of which vary depending on the launch site and launch site operator), however, funding for improvements to infrastructure used solely by that site operator is generally left to the site operator. This arrangement is in part because the infrastructure improvements are necessary to support the unique needs of specific commercial launch vehicles using those sites. At another launch site, the federal government followed a different infrastructure investment model. In the 1990s, the Air Force partnered with the state of Alaska to help fund the construction of a state-owned site to support federal government launches and missile testing rather than constructing a new federal range. This site, known as the Pacific Spaceport Complex – Alaska, conducted its first government launch in 1998. Major infrastructure includes two launch pads with shared vehicle integration and transfer facilities. According to spaceport officials at this site, in addition to government launches, Alaska Aerospace, a state entity that operates the site, has contracts with three commercial launch providers, which anticipate conducting commercial orbital launches there in the future. Appendix II provides additional information on launch sites co-located with federal ranges, as well as funding sources and characteristics for other U.S. commercial launch sites. While the federal government has not directly funded the construction of infrastructure at launch sites in recent years, state and local governments have done so. According to interviews we conducted and our review of publicly available documents of state-government entities that were formed to promote space-related development, state and local governments are investing in infrastructure to obtain the economic benefits of attracting space-related businesses to their areas. In two cases, state governments became operators of launch sites co-located with federal ranges and invested in infrastructure improvements at those sites to support commercial orbital launch vehicles. The Commonwealth of Virginia—through Virginia Commercial Space Flight Authority, an independent state entity created in part to develop and promote Virginia’s commercial space transportation industry— invested $90 million in improvements to a launch pad at the Mid- Atlantic Regional Spaceport. This represented a share of the total costs, which were shared by Northrop Grumman Innovation Systems, a commercial launch provider that has an agreement to use the pad for commercial launches, including cargo resupply missions to the International Space Station. The State of Florida—through Space Florida, an independent special district that serves the state’s space-related needs—has provided over $140 million in infrastructure investments. Those investments upgraded launch pads and the supporting infrastructure at Cape Canaveral, as well as provided grants matched by commercial launch providers for improvements to infrastructure used by those providers. In other cases, state and local governments have invested in wholly new commercial launch sites or are adapting existing airport infrastructure to use as launch sites. According to these launch site operators, these sites are currently used for suborbital launches but could support orbital launches in the future. The state of New Mexico funded the construction of the commercial launch site known as Spaceport America through $225 million in state appropriations and local taxes in two counties. The state also has a 20-year lease agreement with Virgin Galactic, which plans to conduct commercial suborbital space tourism launches from the site. This launch site, with its 12,000-foot-by-200-foot runway, hosted one FAA- licensed suborbital test launch in 2018. In California, the Mojave Air and Space Port (Mojave) is a general aviation airport that obtained an FAA license to conduct commercial suborbital launches in 2004. In addition to continuing its general aviation operations, Mojave currently provides a runway and mission preparation area to commercial launch providers testing vehicles designed for orbital and suborbital launches. This site hosted three FAA-licensed suborbital test flights in 2018. According to a representative from Mojave, the site generally funds infrastructure maintenance with rents and user fees, while launch providers build their own facilities. In July 2018, Mojave also received a $1.4 million grant through FAA’s Airport Improvement Program for the purpose of extending an airport taxiway. According to a Mojave representative, the location of the taxiway extension will be available for hangar development by both aviation and commercial space users on a first- come, first-serve basis. The project was completed in April 2019. Commercial launch providers fund infrastructure improvements at existing launch sites—both co-located with federal ranges and elsewhere—to ensure the sites are tailored to their unique launch vehicles. For example, under its agreements to use launch pads at the federal ranges at Cape Canaveral and Vandenberg Air Force Base, SpaceX representatives told us they invested “hundreds of millions” of dollars in new infrastructure and infrastructure improvements, such as constructing new liquid fuel lines and improving launch pad cooling systems. According to SpaceX representatives, the company made these investments to support the specific needs of its launch vehicles and the rapid pace at which it is currently launching. Virgin Galactic and Stratolaunch—two other commercial launch providers developing suborbital and orbital launch vehicles, respectively—funded the construction of hangars and testing facilities for their launch vehicles at Mojave Air and Space Port. Three of the seven commercial launch providers that we spoke with constructed or are currently constructing new launch sites for their exclusive use. Representatives from two of them said doing so allows them to schedule launches without having to compete with other launch providers at existing launch sites. Two of these commercial launch providers also told us they had not received any government funding for these sites, while the third told us it had received some support from the state government where the site is located. As the commercial space transportation industry continues to evolve, it may lead to more investments in launch sites that are not currently supporting commercial orbital launches. For example, some commercial launch providers are developing launch vehicles consisting of a rocket launched from an airplane in flight, enabling launches from runways rather than launch pads. This could change how and which entities fund launch site infrastructure. Commercial space transportation is a global industry. We identified seven countries, including the United States, that have launch providers with the capability to support an orbital launch of a commercial payload (see fig. 4). In 2017, 7 of the 22 FAA-licensed launches conducted in the United States contained a payload from a non-U.S. launch customer, including several communications satellite operators and one civilian space agency, according to FAA. Similarly, some U.S. launch customers we interviewed said they have used non-U.S. launch providers. According to representatives of the seven domestic and non-U.S. companies we interviewed that use launch services for placing their products into Earth orbit or other trajectories, several factors influence their selection of a launch provider. Many of these representatives acknowledged that as part of their business decision, a prerequisite is that the launch provider’s vehicle and launch site must have the capabilities to meet the customer’s mission requirements, such as having the capability to bring the payload to the desired orbit at the desired time. That capability, in turn, depends on factors such as the lift capacity of a provider’s launch vehicle—which dictates the maximum weight the vehicle can carry—and the geographic locations of its launch sites. For example, launch vehicles operating from sites closer to the equator can place payloads into certain orbits using less fuel due to Earth’s rotational velocity. The direction a launch vehicle can travel from a launch site also affects the orbits into which the vehicle can most efficiently place a payload. For example, Vandenberg Air Force Base in California—which allows launch vehicles to travel west over the Pacific Ocean—is more efficient for certain orbits, while Cape Canaveral—which allows vehicles to travel east over the Atlantic Ocean—is more efficient for others. Beyond selecting a launch provider that has capabilities to meet a launch customer’s mission requirements, six of the seven launch customers we spoke with said the price of a launch is a key deciding factor. For example, a representative from an international satellite operations company told us that the company achieved significant savings by procuring a series of launches from its selected provider. According to the representative, using a different launch provider would have cost almost twice as much—a price that would have forced the company to delay its launch plans. According to data published in FAA’s Annual Compendium of Commercial Space Transportation: 2018, there is wide variation in the commercial price of launches worldwide, ranging from an estimated $62 million to $178 million per launch. The exact price paid for many launches is considered proprietary by both launch customers and commercial launch companies, and is therefore not reported publicly. Moreover, price can be affected by the size and weight of the payload, the intended orbit being reached, and other mission-related factors. As a result, direct comparison of launch prices is difficult. In addition to price, a launch provider’s availability and reliability are also key factors, according to launch customers we spoke with. Six of the launch customers we spoke with mentioned availability as a key factor, which is the launch customer’s ability to reserve a place on the launch provider’s launch schedule. For example, a representative from a domestic small satellite operations company said it can be difficult to find available launches in the United States because the company relies on sharing launches with larger payloads, and few U.S. launches travel to the company’s desired orbit. As a result, the company has procured launch services from Indian and Russian launch providers. Five launch customers mentioned reliability—generally a launch vehicle’s history of successful launches—as a key factor, in part due to the financial impact of a failed launch. For example, a representative from a non-U.S.-based satellite operations company said that in the event of a failed launch, insurance would generally cover the cost of the lost payload, but not lost revenue that would have been generated by the payload in orbit. Some launch customers noted that choosing a launch provider is a complex decision, and that the key factors they consider can be interdependent. For example, the representative from the non-U.S.-based satellite operations company said that while a launch provider may offer a lower price on a less reliable vehicle, the lack of reliability could increase the customer’s payload insurance costs, effectively increasing the launch price. A representative from a company seeking to launch into deep space told us they would only consider a provider that is not only reliable but also has years of successful operations and a proven business plan. According to FAA officials, FAA has been considering changes in its licensing regulations since 2015 and recently has accelerated these efforts. Dating back to 2015, according to FAA officials, FAA had been taking an iterative approach by first making “quick wins”—that is, making administrative changes or straightforward regulatory revisions—with a long-term goal of fully consolidating and streamlining the regulations over a period of several years. FAA’s approach changed, however, when in May 2018, a Presidential Directive was issued that addressed both the timing and content of FAA’s regulatory updates. The directive contained a deadline to publish a proposed regulation for public comment by February 1, 2019. It also directed the Secretary of Transportation to replace the current prescriptive regulations for commercial space launch licensing—in which a certain technology or action is required—with a regulatory framework that is performance-based—in which applicants have flexibility in how they achieve required outcomes, such as a specific level of safety. In response to this directive, DOT published a notice of proposed rulemaking (NPRM) in April 2019 to solicit comments on a proposed rule that will incorporate performance-based requirements. According to FAA officials, they had planned for the NPRM to be published by February 1st, 2019, consistent with the deadline in the directive, but the publication was delayed due to the lapse in DOT’s appropriations that took place in early 2019. A timeline of key actions related to launch licensing regulation is shown in figure 5 below. The preamble of the NPRM states that the proposed rule intends to satisfy the requirements of the Presidential Directive, including consolidating and revising multiple regulatory parts to apply a single set of licensing and safety regulations across several types of operations and vehicles, and replacing prescriptive regulations with performance-based rules. The preamble further states that these changes will give industry greater flexibility to develop means of compliance that maximize their business objectives while maintaining public safety. The proposed rule also seeks to address recommendations made by an Aviation Rulemaking Committee (ARC) that was created in March 2018 as a forum for industry to discuss procedures and requirements for launch and reentry licensing. For example: The ARC recommended that FAA propose rules to eliminate potentially duplicative requirements for launches at federal ranges. Currently, launch providers at federal ranges are subject to FAA’s requirements in addition to those of the range operator (NASA or the Air Force), which may be duplicative of each other. The preamble to the NPRM states that, while FAA has not included language to eliminate duplicative approvals, FAA would continue to work with the appropriate agencies to streamline launch and reentry requirements at ranges and federal facilities. The ARC also recommended more flexibility in licensing such that a single license structure could accommodate a variety of vehicle types and launch or re-entry sites. The preamble states that the proposed rule would, among other actions, eliminate the current limitation specifying a launch license covers only one launch site. As part of the rulemaking process, FAA must comply with a number of requirements before the final rule can be issued. FAA is statutorily required to provide a period of time to solicit public comments on the proposed regulation. FAA must then reasonably respond to public comments submitted on the NPRM and determine whether any changes to the proposed rule may be required as a result of the comments. Some changes made in response to comments would allow AST to proceed with publication of the final regulation. However, major changes not contemplated in the NPRM could necessitate a supplemental NPRM, which could affect the timing of the final regulation’s publication. FAA provided 60 days after publication in the Federal Register for the public comment period. And, while officials told us that they plan to work toward publishing the final rule by the end of 2019, the schedule was affected by DOT’s lapse in appropriations. They also noted that the quantity and content of the public comments and the time and resources required to respond to them will influence that date. Officials estimate that the public comments could number in the thousands. Further, there is a lack of industry consensus in some areas. For example, according to the cover letter accompanying the final ARC report, the report did not include specific recommendations that were agreed upon by all participants. Almost half of the industry stakeholders that participated in the ARC and provided comments on the ARC final report (8 of 19) did not fully concur with the report. Industry stakeholders disagreed on issues such as the requirements for testing flight safety systems, which would be considered as part of the licensing process. The lack of consensus among ARC participants suggests that the NPRM may also generate significantly different perspectives. Furthermore, FAA officials emphasized that the NPRM addresses a highly complex and technical issue, using a wholly revised performance- based regulatory framework, an approach that could affect implementation timelines. We found in the past that the complexity of the issues addressed by rulemakings is a major factor influencing the time needed to issue a regulation. FAA officials told us they intend to complete other related activities that support the rule, such as finalizing guidance documents to provide transparency and help ensure that licensing applicants understand the new requirements. Such guidance may, for example, provide examples of how to comply with the new performance-based requirements. FAA also intends to implement new administrative tools to help AST review licensing applications more quickly. Specifically: Guidance: FAA released a number of draft guidance documents in the form of Advisory Circulars with the NPRM. These Advisory Circulars cover a range of topics, such as providing ways for applicants to comply with requirements for flight safety analysis and lightning hazard mitigation, and provide at least one way an applicant could demonstrate compliance with each performance-based requirement in the proposed rule. FAA officials told us that they plan to publish these Advisory Circulars in final form simultaneous with publishing the final regulation. Through the ARC process, FAA sought input from industry on the standards that should be used to demonstrate compliance with the performance-based regulations. In the long term, however, FAA told us that they are encouraging the industry to develop voluntary consensus standards that the FAA could then accept as an acceptable way of demonstrating compliance. Administrative Tools: FAA officials said they are in the early stages of looking at ways to reduce the administrative burden on FAA and licensing applicants during the licensing process. For example, FAA officials told us that in 2019 they will be examining ways to automate and streamline the licensing process. FAA officials told us that they would like to implement a system whereby applicants, for the first time, would submit applications electronically to an FAA-sponsored system rather than by hard copy or attachments to an email. According to the preamble of the NPRM, FAA’s proposal would allow an applicant to submit its application by email as a link to a secure server, and would remove the requirement that an application be in a format that cannot be altered. In addition to easing the burden of developing paper applications, FAA officials told us they envision that an electronic system would enable both FAA and industry to view the application during the application process and more easily communicate about its progress. In recent years, AST has improved some aspects of how it determines its workforce needs. Our work on strategic workforce planning underscores the importance of determining both current and future workforce needs and identifying potential gaps in employee skills. The improvements made to date provide AST with greater insight into the optimal number of people currently needed in certain positions. However, these improvements do not improve AST’s ability to systematically assess the workforce needs of its management and support offices, nor does AST project its future workforce needs. Moreover, AST has yet to collect information on staff skills and competencies that would enable it to identify potential gaps in those skills, gaps that further limit AST’s ability to effectively and efficiently align its available staff resources with current and future workloads. To assist FAA decision makers in understanding and meeting AST’s staffing needs, AST developed and annually updates a 5-year workforce plan for its office. The current plan—covering the period from 2018 through 2022—indicates that AST’s approach for workforce planning has a 5-year time frame. However, the plan discusses immediate workforce and resource needs in general terms. One of the key principles we identified in our prior work on effective strategic workforce planning is the importance of determining the workforce needs that are critical to achieving an organization’s current and future programmatic goals. Such a determination of workforce needs should include both the optimal number of staff needed in specific positions and the required skillsets and levels of expertise for staff. Since 2016, AST has taken several steps to better understand how it uses its staff resources in carrying out its mission to license and oversee space launch operations. The majority of AST’s operations budget— about 75 percent in fiscal year 2018—was used to fund salaries and related expenses. AST now comprehensively monitors and measures staff time spent on specific activities and measures and tracks the volume of its work—information it can use to better understand workforce needs. AST officials told us that these steps facilitate more informed decision-making about the number of staff needed in specific positions for the next budget cycle. However, these steps do not provide the information AST needs to determine the optimal size and composition of its entire workforce or enable it to project workforce needs sufficiently into the future. AST launched a revised timecard system in June 2016 to more comprehensively account for staff time spent on specific activities. According to AST officials and our review of relevant documentation, including a list of revised time codes, the revised system allows staff to record hours worked on individual tasks, such as launch observations or consultations with launch companies prior to application submission (i.e., pre-application consultation), training, and leave. Time codes were revised for all AST staff—that is, staff in its five operational divisions, management office, and two support offices (see fig. 6)—to account for all major tasks they perform. AST officials told us that the new timecard data, in combination with workload metrics, can help inform its current workforce needs. For its five operational divisions, AST officials have developed and continue to refine a set of workload metrics, which, along with other data, enable AST to identify the resources that are used to carry out key AST activities, such as licensing and overseeing launches. These metrics track the number of work activities (e.g., regulatory waivers issued or safety inspections conducted) that are ongoing or were completed over a certain time period. For example, in fiscal year 2018, AST was engaged in pre- application consultations with about 23 commercial launch providers and was evaluating more than 16 license applications on average per month. Officials analyze these metrics in combination with timecard data to determine the number of staff hours and average number of days spent completing specific activities. For example, between March and August 2017, FAA officials reported that for each ongoing project, staff spent an average of about 60 hours per month on pre-application consultations. Officials plan to use the results of this analysis in the fiscal years 2021– 2022 budget cycle to help estimate the number of staff currently needed in specific positions within its five operational divisions. However, with regard to its management and two support offices— which represent about one-third of AST’s total staff—AST has not yet developed workload metrics. Staff in AST’s management and support offices are responsible for overseeing research and development; advising and assisting other offices on technical matters; coordinating and liaising with international entities and other federal agencies; as well as performing other support operations, such as budget and financial planning. Officials told us that although they would like to develop these metrics, they put the effort on hold because of competing priorities within AST, such as updating its licensing regulations. Officials said that they had first focused on better understanding the workforce needs of the operational divisions, which have responsibility for the majority of AST’s primary mission areas, such as licensing and overseeing launches. In discussing this approach AST officials stated that recent budget constraints have limited their ability to address all of their current identified workforce needs, which, according to their most recent workforce plan, are in nearly all areas of their office. As a result, officials said that they use their limited number of authorized positions to fill their most immediate workforce needs, typically in the operational divisions. However, without workload metrics that would allow AST to determine the number of staff needed for its workload regardless of what office or division, it is difficult for AST to determine the appropriate number and composition of staff to most effectively carry out its statutory priorities and help ensure that it uses its limited resources in the most efficient way. In addition, AST officials told us that they recognize that past hiring decisions and balance of workload among staff may not have been fully aligned with AST’s statutory priorities and that the composition and ratio of staff may no longer be appropriate given the evolution of the industry and the revised regulatory structure under way. As a result, officials stated that in the coming months they intend to take a fresh look at the organization of the Office of Commercial Space Transportation as a whole to better balance the needs of the industry with the organizational requirements. In addition to developing an Office of Spaceports, as required by the FAA Reauthorization Act of 2018, officials told us that they will consider re-organizing the offices and divisions, as well as the workload and staff currently within them. AST also has taken steps to improve its ability to estimate its workload for a 2-year budget cycle, which, according to AST officials, will help them determine and justify near-term workforce needs. Specifically, from the new workload metrics discussed above, AST officials told us they had identified five key activities that best reflect historical workload trends and that officials then plan to combine with their assumptions about how the industry will evolve over the next 2 years. Officials told us that they plan to use this approach for the first time in the fiscal years 2021–2022 budget cycle. In past budget cycles, AST relied primarily on the projected number of launches to estimate its workload; this number, officials noted, is the most important factor but resulted in an incomplete reflection of the five operational divisions’ workload. For example, officials told us that the workload of its operational divisions encompasses a range of activities leading up to a launch that would not be captured in its workload estimates if AST only looked at the number of launches. Now, under their planned approach, AST officials said that they will better account for the full range of regulatory activities and the timeline of its licensing process. While planned improvements to AST’s workload estimates better account for the full range of AST’s regulatory activities, limiting these estimates to the 2-year budget cycle reduces AST’s ability to anticipate and respond to emerging workforce needs. AST recognizes the importance of longer- term workforce planning by developing and annually updating a 5-year workforce plan. Also, as noted above, key principles for effective strategic workforce planning emphasize the importance of forward-thinking planning to help organizations align their workforce to meet future programmatic goals. According to AST officials, they estimate the workload for 2 years in part because it is intended to help them identify and justify workforce needs during the 2-year budget process, as well as prioritize addressing immediate workforce needs. Officials also said that substantial uncertainty surrounds longer-term industry forecasts, and consequently, any assessment of longer-term workforce needs. For example, they pointed to a number of factors that lead to the unpredictability of how the industry will evolve, including the variable pace at which new launch companies progress and the future of the commercial suborbital launch sector, particularly the nascent space launch tourism industry. They also noted that a launch vehicle accident or other risks could affect the industry’s rate of growth. In our prior work, we have discussed some approaches used by other agencies to help assess future workforce needs when faced with uncertainties. One approach involves scenario planning, in which a federal agency operating in a changing environment used a range of scenarios, each of which represented different future environments that the agency may face, to help predict how the scope and volume of its activities might change in each scenario. For AST, such an approach could entail developing a range of workload projections based on different industry and regulatory environments that it thinks it may face, along with associated workforce management strategies to address those environments. AST officials said that they were considering projecting their workload estimates further into the future and intend to work with FAA’s Office of Aviation Policy and Plans—the office that helps develop FAA’s 20-year aerospace industry forecasts—to leverage that office’s forecasting expertise. However, AST has not established a timeline with milestones or formally committed to conducting longer-term workload projections. Longer-term workload projections may be particularly beneficial to AST to help make well-timed decisions about hiring and training staff and to help ensure AST has qualified staff available when they are needed. For example, according to officials, it can take a few years for systems safety engineers to be trained and have the sufficient experience to lead projects. Further, AST officials told us that hiring technically qualified personnel, including positions that require considerable training and experience to be a fully functioning employee, is challenging. Without an understanding of its projected workload beyond a budget cycle, AST will be limited in its ability to effectively and strategically plan for its longer- term workforce needs and take action when the opportunity arises. As such, AST remains at risk of not having the right number of staff in the right positions to keep pace with and respond to changes in the commercial space transportation industry. Our prior work on strategic workforce planning underscores the importance for organizations to determine the skills and competencies that are critical to successfully achieving their current and future missions and goals. Once the necessary skills and competencies have been identified, key principles for effective strategic workforce planning call for an organization to identify—and subsequently develop strategies to address—gaps between the skills and competencies needed and those that its workforce has. Those gaps should include both current skills gaps (i.e., skills that its workforce currently needs but does not possess) and emerging skills gaps (i.e., skills that its workforce may need in the future but does not possess). Further, according to federal Standards for Internal Control, an organization’s management should ensure that the workforce skills necessary to achieve programmatic goals are continually assessed. This step is especially important as changes in national security, technology, budget constraints, long-term fiscal challenges, and other factors may occur in the environment within which federal agencies operate. AST, however, does not currently collect the information needed for it to conduct a skills gap analysis. Rather, AST has a basic understanding of the skills and competencies of its workforce. For example, its current workforce plan includes the following information on AST’s workforce: Level of education—the percentage and number of employees having attained bachelor’s, master’s, and doctorate degrees. Occupation—the percentage and number of employees in mission- critical occupations (e.g., aerospace engineers). Age—the percentage and number of employees by age range. Tenure—the average number of years employees have been in their current position and employed by FAA. Retirement eligibility—the number of employees who will be eligible to retire each year during the 5-year period of the staffing plan. AST officials acknowledged that the workforce information it currently collects is insufficient to allow them to systematically identify gaps in specific staff skills or competencies—such as expertise in flight safety analysis or launch vehicle propulsion—needed for evaluating certain launch license applications. Officials told us that they do prioritize filling positions, through hiring or contracting, that address the organization’s most immediate needs. However, this strategy focuses on positions, as opposed to identifying specific skills or competencies within those positions. AST officials told us that they are planning to develop and annually administer to staff and managers a skills assessment survey that would collect information about the specific skills and competencies that individual staff currently possess. Officials told us that the results of the survey would allow them to assess the current skills of AST’s workforce and in combination with other information, such as expected attrition and retirement rates, help identify current and emerging skills gaps. In July 2018, officials told us that they plan to complete the survey and administer it in time for inclusion in their workforce plan for fiscal years 2019–2023, estimated to be issued in April or May 2019. However, officials subsequently stated that their survey plans have been delayed for multiple reasons, including DOT’s lapse in appropriations. Accordingly, as of May 2019, AST had neither developed a draft of the skills assessment survey, nor established a formal timeline for finalizing it or a plan for periodically administering the survey. Furthermore, officials told us that they are currently negotiating with the union’s bargaining unit to gain approval to administer a survey that does not maintain anonymity to non-management staff. They said that if they cannot obtain the bargaining unit’s approval, they will need to develop an alternative plan because they do not believe that collecting anonymous data on staff skills would allow them to identify skills gaps for these staff. Officials told us that they also intend to include in the survey skills and competencies that may be needed in the future. They stated that they did not know for certain if or how they would identify what those new skills might be, but that they are considering soliciting feedback from industry stakeholders, such as through FAA’s Commercial Space Transportation Advisory Committee, to help identify any future competencies that may be needed as a result of the evolution in the industry. Without systematic information on specific skills and competencies of its entire workforce, AST lacks reasonable assurance that its current workforce possesses the requisite skills and competencies and may not be able to efficiently identify opportunities to move staff within AST to help address identified skills gaps. And, ultimately, AST may not be prepared to make strategic decisions on how to address emerging skills gaps and align its staff to achieve future programmatic goals, such as identifying and acquiring potential new skills and competencies needed under a revised regulatory structure. FAA officials and representatives from the commercial space and aviation industries we met with agree that FAA’s current approach to accommodating commercial space launch and reentry operations into the National Airspace System (NAS) is inefficient. FAA has the responsibility for ensuring the safe and efficient use of the NAS, a limited national resource, for and by all users, including commercial and business airlines and commercial launch providers, among others. To this end, according to FAA officials and documents describing operational procedures and risk evaluation, FAA takes measures during a commercial space operation aimed at preventing fatalities, injuries, and property damage, and ensuring that nothing interferes with the launch vehicle’s operations. FAA’s current approach, as described in documents that explain how FAA mitigates risk to people and property during a space launch, is to close the airspace around a commercial launch operation—in some cases hundreds of square miles for several hours—to other airspace users, such as commercial airlines. Prior to launch, FAA establishes the size and duration of the airspace closure, also known as an aircraft hazard area, and, days ahead, notifies potentially affected airspace users about the upcoming closure. FAA calculates the size and boundaries of the aircraft hazard area generally based on the risk to life and property posed by a launch vehicle’s expected trajectory, as well as potential trajectories in the case of a vehicle’s failure and the subsequent paths of falling debris. The duration of the closure is generally dependent on the period of time in which the launch or reentry is expected to occur—known as a launch window—which varies by the type of launch or reentry vehicle, among other things. The aircraft hazard area extends from sea level up to unlimited height, and generally does not change in size or shape during the entirety of the launch window (see fig. 7). According to FAA officials, the designated aircraft hazard areas are larger and remain in effect longer than may actually be needed to ensure public safety. For example, according to FAA officials and launch documentation, to protect public safety, the duration of an airspace closure is always longer than the launch window. In fact, in some cases, the airspace closure may be scheduled for more than 3 hours, which is substantially longer than the time typically required for space launch and reentry operations from Cape Canaveral (about 30 minutes). FAA officials explained that they are not able to monitor or respond to dynamic circumstances associated with space launch vehicles in the NAS in real- time. As a result, FAA closes the airspace for when and where it is potentially—rather than actually—hazardous. FAA officials told us that the agency’s approach to date for accommodating space launch operations into the NAS has helped ensure public safety during launches. For instance, during fiscal years 1989 through 2018, FAA reported that it licensed 357 launches or reentries, and in this time there were no fatalities, serious injuries, or significant property damage to the uninvolved public. However, according to FAA officials and research, FAA’s approach creates inefficiencies in how the airspace around launch operations is used—such as causing flight delays for commercial airlines. FAA officials and commercial space industry representatives said it also makes scheduling these operations more challenging for launch providers, and affects FAA’s operational efficiency. The effects on each of these groups are described below. Commercial airlines. FAA has estimated that, in fiscal year 2017, about 1,200 commercial airline flights were directly affected—that is, rerouted or delayed—around 22 space launch operations, resulting in an estimated 39,000 additional miles flown. The majority of these miles were flown in proximity to Cape Canaveral in Florida, which hosted the majority of domestic launches that year. FAA further estimated that, of the 15 space launches from January to October 2018 around Florida where airspace tends to be busy due to the high volume of commercial airline traffic along the East Coast, an average of 60 aircraft per launch were directly affected. For all commercial launch sites, FAA estimates that the number of directly affected aircraft ranged up to 153 for an individual launch with an average of fewer than 10 aircraft per launch outside of the Florida area. According to FAA officials, these estimates are based on historical data on the number of aircraft that typically fly through that area at the time of the airspace closure. Because launches can be delayed by hours or days for reasons such as unforeseen weather conditions or technological issues, airlines and other affected airspace users may face challenges when attempting to plan around a launch to avoid flight reroutings and delays. Representatives of a major airline trade association told us that the spread of launch activity beyond Cape Canaveral, as well as the development of new launch vehicles, has heightened their concerns about inefficiencies in how airspace around launch operations is used. Launch providers. The size and duration of aircraft hazard areas can make it difficult for FAA to find time slots to accommodate commercial space launches because of its responsibility to ensure the efficient use of the national airspace, a limited resource. All the launch providers we spoke with that had conducted launches at U.S. commercial launch sites said they have been able to find suitable launch windows that met with FAA approval. However, one launch provider told us of an occasion when FAA had denied the originally requested launch date and time because it fell within a time of unusually congested airspace. In addition, more than half of the launch providers told us that they anticipate challenges obtaining approval for a requested launch date or time in the future. FAA. In addition to effects on NAS users, FAA officials told us that FAA itself also experiences operational inefficiencies in managing air traffic during launches. This inefficiency is, in part, because FAA’s current policies and procedures were developed for aircraft operations and either have not yet been fully adapted for commercial space operations, or a relevant policy or process is missing altogether. For example, FAA’s current procedures for launch providers and FAA to follow when they request, schedule, and conduct launches require different FAA facilities to negotiate unique agreements for each separately licensed operation or activity. This process can be time- consuming. For example, one launch provider told us that it took 1½ years to finalize minor changes to a letter of agreement. As we discuss later, FAA is taking steps to standardize these letters. According to FAA documentation and officials we spoke to, FAA aims in the long term to increase utilization of the NAS by integrating launch vehicle operations into the NAS with other users, rather than its current approach of segregating launch and reentry operations through airspace closures. Specifically in 2011, FAA began identifying actions it could take and developing plans to address challenges associated with closing portions of the airspace during launch operations. It did so in light of the increasing frequency of commercial space launch and reentry operations and the spread of operations to new locations. According to FAA officials, the actions and plans continue to evolve as FAA learns more and reacts to anticipated changes in the commercial space transportation industry. Further, officials told us that FAA’s vision for full integration of commercial space launch operations cannot be defined by a single solution or an end goal because the demands of these operations on the NAS are constantly changing. Consequently, FAA officials said that full integration of commercial space operations into the NAS will reflect a collection of visions or approaches that improve predictability and efficiency while maintaining safety. For example, according to FAA documents and officials we spoke to, FAA’s approach for experimental launches will always be to close the airspace around the launch to other users. In contrast, FAA may develop standards for some launch vehicles, such as hybrid launch vehicles with repeated successful operations, which specify a safe distance and duration of separation in the airspace. FAA has two key internal documents to help guide the development and implementation of its actions as it seeks to better integrate commercial space launches and reentry operations into the NAS and reduce FAA’s operational inefficiencies. A concept of operations: FAA officials expect to finalize a concept of operations in 2019, which will provide a long-term, high-level vision for FAA’s efforts to efficiently integrate commercial space operations. According to FAA officials, it will describe, among other things, FAA’s existing approach to and associated shortfalls in accommodating commercial space operations, as well as proposed tools, policies, and procedures to address those shortfalls. According to FAA officials, it also will inform FAA’s current and future efforts to identify needs for new or modified technologies, tools, procedures, and policies. Roadmap for the Integration of Space Operations in the National Airspace System (Roadmap): This document serves as a planning and tracking tool for FAA’s operational arm—the Air Traffic Organization—to use as it seeks to more efficiently manage the airspace during commercial space launch and reentry operations while maintaining safety. It identifies, prioritizes, and tracks the specific changes needed to begin addressing the related shortfalls that FAA officials told us will be discussed in the concept of operations. According to the Roadmap, some of the activities are exploratory, and FAA expects that new activities will be identified and added to the development schedule as FAA continues to work with stakeholders to determine how best to manage the airspace, and conceptualizes and develops key technologies. The first Roadmap was released in November 2016, and, according to FAA officials, FAA plans to update it annually. FAA officials told us they expect to release the third and most recent version in 2019. The activities it identifies are divided into: short-range (to have been completed in calendar year 2018); mid-range (through 2022); and long-range (through 2023 and beyond) time frames, during which FAA plans to develop and incorporate new technologies, policies, processes, and regulations. In completing the actions needed to implement the approaches outlined in the Roadmap, FAA officials told us that they are actively working with FAA’s Performance Analysis Directorate to develop a set of metrics to measure the progress and effectiveness of its actions. Officials also highlighted that because the demands of commercial space operations on the NAS are constantly changing, as noted above, there is no defined end goal. To this end, the purpose of any metrics officials develop will be to help determine if their actions are helping increase efficiency while maintaining safety, not measure their progress toward a goal of full airspace integration. FAA officials told us they plan to have a set of metrics completed by early 2019. Some of these metrics will likely use currently available data, such as the number of aircraft rerouted and how many additional miles rerouted aircraft fly, while others are still being identified. Further, FAA officials told us that FAA coordinates actions related to commercial space integration through an interagency working group established in 2015. The group meets monthly and members include officials from across FAA lines of business, as well as other federal agencies, including the Department of Defense. The Roadmap shows that FAA’s actions to better integrate commercial space launch and reentry operations into the NAS include, but are not limited to: developing new technologies; updating and assessing needed changes to policies, procedures, and coordinating with aviation- and space-industry stakeholders. FAA’s technology efforts are related primarily to collecting real-time data on a launch vehicle’s position and path, automatically generating the required aircraft hazard area, and integrating those data into the existing structure of the air traffic control systems. As a result, FAA officials said that FAA may ultimately be able to dynamically change the size and duration of the aircraft hazard area in some types of launches, thereby reducing the amount and duration of airspace closed to other users. In the short term, FAA is assessing how existing air-traffic control technologies and procedures could be used to help reduce the effects of launches on other NAS users. According to an FAA official, for example, four initiatives currently used to manage air traffic during other airspace constraints could potentially be used during space launch operations. One initiative would enable air traffic controllers to strategically control the number of flights approaching the aircraft hazard area so that if these flights were in the hazard area at the time of a launch vehicle failure, controllers could still clear the area quickly enough to protect public safety. This FAA official told us that if they decide to pursue these initiatives, they hope to complete some of the necessary steps to do so by summer 2019. For potential use in the longer-term, FAA is piloting prototypes of two key technologies by running them alongside existing air-traffic control systems during selected launches, thereby testing their capabilities without their being fully operational. The Space Data Integrator (SDI) is designed to receive real-time data on launch vehicle position and movement and display real-time aircraft hazard areas to enable improved situational awareness. FAA officials told us that, as FAA is assessing approaches to shift from static to more dynamic hazard area calculation capability, initial SDI capabilities will likely be deployed in advance of more integrated and improved real-time hazard area generation capabilities. In addition, FAA officials told us that they are exploring alternative acquisition strategies that could enable partial system implementation for the technology by 2022. Because FAA has not made a final investment decision, the date of system-wide implementation of SDI is unknown. According to FAA officials, the Hazard Risk Assessment and Management (HRAM) tool, if pursued, is intended to help automatically communicate SDI data to air traffic control systems and, in the future, to present air traffic controllers with information that would allow them to decide how to best manage the airspace. Officials also said that HRAM involves modifying an existing air traffic management tool, currently has very limited capabilities, and is still only under consideration as a possible approach. Over the next year these officials plan to work on some of the tool’s components, assess what types of data are valuable to air traffic controllers, and determine whether to continue developing this technology or consider alternative technologies. According to the Roadmap, FAA has identified policies and standard operating procedures that need to be created or updated to enable it to better manage the operating environment during space launches. Actions taken to date include, for example: developing training materials to inform air traffic personnel about commercial space operations in the NAS; developing a high-level strategy for integrated space vehicle operations going forward; and standardizing the terms of reference for commercial space operations for use by FAA, NASA, and DOD. In addition, according to the Roadmap, FAA plans to standardize some letters of agreement—the document specifying procedures that a launch provider and FAA use to request, schedule, and conduct launches. Officials said they hope to issue documentation of these changes by September 2019. FAA officials told us that these changes will result in letter of agreement templates for use by FAA. FAA officials said FAA also plans to continue reviewing its regulations, policies, and procedures to identify other areas that need updating or entirely new language. FAA is taking steps to foster coordination between commercial space and aviation industries to help develop and increase buy-in for new and revised approaches to improve the efficiency of the national airspace for all users. Most notably, in November 2017, FAA chartered an aviation rulemaking committee to examine the issue of equitable airspace access among various users. Committee members include a mix of commercial space transportation and aviation industry representatives. Topics being addressed include identifying potential criteria that FAA may use when considering competing user priorities for airspace, as well as potential tools that could help mitigate the effects on other airspace users during launch operations. FAA officials told us that the committee anticipates issuing a report and recommendations to FAA in April 2019, and some members of the committee highlighted that the meetings benefited their understanding of other users’ unique needs; economic benefits; and experiences with regard to integrating space operations. Also, an FAA official said the agency has sponsored four “Industry Days” events since 2014 for the commercial space industry. At each event, multiple FAA offices discussed their roles and responsibilities associated with space launches and answered questions from industry. For the first time, at its 2018 event, FAA invited aviation industry representatives to encourage continued dialogue between the commercial space and aviation industries. FAA officials also noted that they solicited ideas on priority actions from participants and are currently reviewing those ideas to help inform their next steps. Separately, FAA expanded the membership of its Commercial Space Transportation Advisory Committee to include representatives of the aviation industry in addition to the commercial space transportation industry to foster further dialogue between these groups. The commercial space transportation industry provides a service that has become essential to many aspects of government, business, and society. The capability to launch payloads into space enables national security missions, mobile communications, and scientific research, among many other applications. AST’s role as a regulator of commercial space launch providers is fundamental to the continued safe growth of the industry. With the anticipated growth and potential organizational restructuring of AST, as well as the evolution of the commercial space transportation industry, it is vital that AST ensure that the size, composition, and skills of its workforce are aligned with its projected workload, based on anticipated future mission and programmatic goals. AST’s workforce plan states that AST needs additional staff in nearly all areas. However, current budget and long-term fiscal pressures heighten the need for agencies to strategically manage their workforce, a process that includes making strategic decisions about how and where to prioritize limited resources. AST does not have a complete understanding of its current and projected workload, nor does it know the number of staff and types of staff skills and competencies necessary to meet those workload needs. Without this information, AST risks managing its workforce reactively to a rapidly changing environment instead of strategically planning for the future. We are making the following four recommendations to FAA: 1. The Associate Administrator of AST should develop workload metrics that encompass the whole office and that would allow AST to determine an appropriate workforce size and composition. (Recommendation 1) 2. The Associate Administrator of AST should establish a timeline for finalizing workload projections that extend beyond the 2-year budget cycle and that include an approach for addressing uncertainty. (Recommendation 2) 3. The Associate Administrator of AST should ensure that its skills assessment survey collects information from staff on skills and competencies in those areas that are both currently needed and may be needed in the future. (Recommendation 3) 4. The Associate Administrator of AST should develop and document a plan for periodically assessing whether staff possess the necessary skills and competencies to achieve programmatic goals, such as annually administering a skills assessment survey. (Recommendation 4) We provided a draft of this product to DOT and NASA for review and comment. In its written comments reproduced in appendix III, DOT concurred with our recommendations. DOT and NASA also provided technical comments that we incorporated, as appropriate. We are sending copies of this report to the appropriate congressional committees, DOT, NASA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at 202-512-2834 or KrauseH@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. Our objectives for this report were to: (1) describe how the construction of infrastructure at selected U.S. commercial launch sites has been funded; (2) describe key factors that influence where orbital launches occur; (3) summarize actions the Federal Aviation Administration (FAA) has taken to streamline its commercial space launch regulations; (4) examine how well-positioned FAA’s Office of Commercial Space Transportation (AST) is to determine its current and future workforce needs; and (5) identify actions FAA is taking to better integrate commercial space launch operations into the National Airspace System (NAS). The scope of this report focuses on topics related to FAA’s oversight of the U.S. commercial space transportation industry. Therefore, the report does not discuss launch indemnification and the safety of human spaceflight, or examine international outer space treaty obligations. For all objectives, we reviewed relevant statutes, regulations, and directives governing FAA’s oversight of the U.S. commercial space transportation industry. In addition, we interviewed AST officials and conducted semi-structured interviews with all seven commercial space launch providers that had conducted an FAA-licensed launch operation as of January 2018. To describe how infrastructure at selected commercial launch sites has been funded, we first identified, through review of FAA information on launch site operator licenses and launch licenses, all U.S. commercial launch sites—those that have an FAA site operator license to conduct commercial launch operations and those that may not have a site operator license but have hosted FAA-licensed launch operations. From these 15 identified U.S. commercial launch sites, we selected 9 for review because the launch site has hosted FAA-licensed launch operations between January 1, 2015, and December 31, 2018. We reviewed relevant publicly-available documents, such as launch sites’ business plans, user guides, and other planning documents related to U.S. commercial launch sites. We interviewed the eight launch site operators of the nine selected launch sites. The perspectives of the selected launch site operators are not generalizable to those of all launch site operators; however, the information obtained provides a balanced and informed perspective on the topics discussed. In addition, we interviewed members of the Commercial Spaceflight Federation’s working group on commercial launch sites. See table 2 for a full list of entities interviewed. To describe key factors influencing where orbital launches occur, we reviewed data from FAA’s 2018 Annual Compendium of Commercial Space Transportation as well as FAA data on recent launches within the United States. We interviewed representatives from seven launch customers, selected based on the following criteria: The company is not a government entity. The company’s payload was commercial, as documented in FAA’s commercial space launch compendiums. The customer had multiple launches in 2016 and 2017, with at least one of those launches occurring in 2017. The customer has had at least one launch in the United States that was licensed by FAA. Among the companies that met these criteria, we chose our final selections to have a mix of the following characteristics: domestic and non-U.S. companies, those that had launched exclusively at one launch site versus multiple launch sites, and those that are involved in traditional space activities, such as satellite communications companies and remote-sensing companies and those that are pursuing non-traditional space activities, such as asteroid mining and satellite servicing. The perspectives of the selected launch customers are not generalizable to those of all launch customers; however, the information obtained provides a balanced and informed perspective on the topics discussed. To summarize actions FAA is taking to streamline its commercial space launch regulations, we reviewed relevant statutes, regulations, and FAA guidance. We also reviewed FAA’s documents related to the rulemaking, including its schedule of rulemaking activities and the Streamlined Launch and Reentry Licensing Requirements notice of proposed rulemaking issued in April 2019, and reviewed and analyzed the Streamlined Launch and Reentry Licensing Requirements Aviation Rulemaking Committee final report. We interviewed FAA officials and representatives of the Commercial Spaceflight Federation about FAA’s ongoing and planned actions related to the rulemaking. Finally, we reviewed the minutes from the June 2018 meeting and attended the October 2018 meeting of the Commercial Space Transportation Advisory Committee, in which FAA officials and industry representatives discussed FAA’s actions on the rulemaking. To examine how well-positioned AST is to make strategic decisions about its current and future workforce needs, we reviewed FAA documents, including its budget justification and workforce plans from the past 3 years. We also reviewed FAA’s year-end reports on its workload metrics from fiscal years 2017 and 2018, and portions of FAA’s preliminary labor analyses using its revised timecard data and workload metrics. We identified key principles on effective strategic workforce planning from our previous work to use as criteria to assess FAA’s actions. We interviewed AST officials about their plans and actions to improve its workforce planning and assessed those actions against the identified key principles for effective strategic workforce planning. We focused our analysis on those principles that are related to determining current and future workforce needs. To identify actions FAA is taking to better integrate commercial space launch operations into the National Airspace System, we reviewed and analyzed relevant FAA documents, including a document that discusses FAA’s vision for integrating commercial space transportation operations into the NAS and the Roadmap for the Integration of Space Operations in the National Airspace System. In addition, we interviewed FAA officials within AST, Air Traffic Organization, and the Office of NextGen regarding their ongoing and planned actions for improving the integration of commercial space transportation operations into the NAS. We also interviewed industry stakeholders to obtain perspectives on this topic. These stakeholders included representatives from Airlines for America, a trade association for the U.S. airline industry, and from launch providers. Finally, we attended an FAA-sponsored industry conference in October 2018 on FAA’s airspace integration efforts. We conducted this performance audit from July 2017 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 3 shows selected characteristics and capabilities of U.S. commercial launch sites included in our review of infrastructure funding. Table 4 includes other U.S. commercial launch sites that did not have FAA-licensed activity from 2015 to 2018 and were not included in our review of infrastructure funding. In addition to the individual named above, Heather Halliwell (Assistant Director); Gretchen Snoey (Analyst-in-Charge); Namita Bhatia Sabharwal; Giny Cheong; Gerald L. Dillingham; Camilo Flores; Joshua Garties; Richard Hung; Delwen Jones; Elke Kolodinski; Maureen Luna Long; Malika Rice; Travis Schwartz; and Andrew Stavisky made key contributions to this report.", "summary": "The commercial space transportation industry provides launch services that enable national-security and commercial satellites, among other things, to be sent into orbit for government and private customers. Continued growth and evolution in the industry is expected as reliance on space-based applications increases. AST is charged with overseeing the industry, including licensing and monitoring launch vehicle operations. GAO was asked to review developments in this industry. This report (1) describes FAA's actions to integrate commercial space launches into the national airspace and (2) examines how well-positioned AST is to determine its current and future workforce needs, among other objectives. GAO reviewed relevant statutes, regulations, and FAA guidance; compared FAA's workforce management efforts to key principles for effective workforce planning; and interviewed FAA officials and U.S. commercial launch providers that had conducted an FAA-licensed launch as of January 2018, among other industry stakeholders. The Office of Commercial Space Transportation (AST) within the Federal Aviation Administration (FAA), in collaboration with other FAA offices, is taking a range of actions, such as testing new technologies, to improve how efficiently FAA integrates space vehicle launch operations into the national airspace. According to FAA officials, the amount of airspace that FAA closes to other airspace users is larger and remains closed longer than may be needed to ensure public safety. To help remedy this situation, FAA is piloting prototype technologies that would collect launch vehicles' location data in real-time and transmit them to air traffic controllers. Officials said the earliest these technologies could be implemented would be 2022. In March 2019, FAA published an announcement seeking interest from industry on partnering with FAA to further develop the technologies. Meanwhile, FAA is assessing how existing air traffic control technologies could be used to help reduce the effects of launches on other airspace users. Since 2016, AST has taken steps to improve how it determines its current workforce needs to carry out its mission including licensing commercial launch vehicle operations. These steps include more comprehensively monitoring staff time spent on specific activities and measuring the volume of the staff's work. While AST officials told us that AST is planning to continue to improve its workforce-planning efforts, GAO found that some aspects of AST's efforts fall short of key principles of strategic workforce planning. Such principles underscore the importance of determining both current and future workforce needs and identifying potential gaps in employee skills. For example: AST does not project its workload beyond a 2-year budget cycle, limiting its ability to effectively and strategically plan for its longer-term workforce needs. According to officials, it can take a few years for engineers with certain skills to be trained and have sufficient experience to lead projects. Further, AST officials told GAO that hiring technically qualified personnel, including positions that require considerable training and experience to be a fully functioning employee, is challenging. AST officials said that they are considering projecting their workload estimates further into the future, but they have neither formally committed to doing so nor established a timeline with milestones. AST officials acknowledged that the information AST currently collects on the skills of its staff is not sufficient to allow them to identify gaps between the skills and competencies needed and those that its workforce currently possesses or may need in the future, such as expertise in flight safety analysis. AST officials told GAO that they plan to develop a tool that could collect information annually from staff and managers about the specific skills and competencies that individual staff currently possess. As of May 2019, however, AST had neither developed a draft of the tool nor established a timeline for finalizing it. Without this information, AST lacks reasonable assurance that its current workforce possesses the requisite skills and competencies, and AST may not be best positioned to proactively determine how to align its staff to carry out its mission. GAO is making four recommendations on workforce planning to AST, including that AST establish a timeline for finalizing longer-term workload projections and that AST ensure that it collects information from staff on skills and competencies in those areas that are currently needed and may be needed in the future. AST concurred with the recommendations.", "document_type": "gao"}
{"report": "The BSA established reporting, recordkeeping, and other AML requirements for financial institutions. As the delegated administrator of the BSA, FinCEN has issued implementing regulations. In complying with BSA/AML requirements, U.S. financial institutions assist government agencies in detecting and preventing money laundering and terrorist financing by, among other things, establishing and maintaining compliance programs, conducting ongoing monitoring of customers and transactions, and reporting suspicious activity. Oversight and enforcement of compliance with the BSA involve several federal agencies, including FinCEN and the Internal Revenue Service (IRS). FinCEN has overall authority for administering and enforcing compliance under the BSA and may seek civil penalties and injunctions to compel compliance. In addition, each of the federal banking regulators has independent authority to initiate enforcement actions against supervised institutions for violations of law and to seek civil money penalties for BSA violations, among other things. FinCEN has delegated authority to IRS to investigate most criminal violations of the BSA. The Department of Justice prosecutes violations of federal criminal money-laundering statutes, including violations of the BSA, and several law enforcement agencies conduct BSA-related criminal investigations. The federal banking regulators have also issued BSA/AML regulations that require banks to establish and maintain a BSA/AML compliance program that includes, among other things, policies, procedures, and processes to identify and report suspicious activity. The banking regulators are required to review banks’ compliance with BSA/AML requirements and regulations, which they generally do every 1 to 2 years as a part of their routine safety and soundness examinations. FinCEN has also delegated examination authority for BSA/AML compliance for certain entities, including money transmitters, to IRS. In general, money transmitters must register with FinCEN and provide certain information on their structure and ownership. According to Treasury, in all but one state, money transmitters are required to obtain licenses from states in which they are incorporated or conduct business. State supervisory agencies also may conduct BSA/AML examinations of licensed money transmitters. To ensure consistency in the application of BSA/AML requirements, in 2005 the federal banking regulators collaborated with FinCEN on developing an examination manual that was issued by FFIEC for federal bank examiners conducting BSA/AML examinations of banks. The examination manual has been revised several times since its release, and the most recent comprehensive revision was released in 2014. According to the examination manual, a key function of the federal banking regulators’ BSA/AML examinations is to assess whether banks have established the appropriate policies, procedures, and processes based on their BSA/AML risk to identify and report suspicious activity. The supervisory process also assesses whether banks provide sufficient detail in reports to law enforcement agencies to make the reports useful for investigating suspicious transactions that are reported. Moreover, federal banking regulators conduct risk-focused BSA/AML examinations of banks—that is, they review key BSA/AML risks or specific risk areas identified by the bank and tailor examination procedures based on each bank’s risk profile. Among other things, examiners review whether banks have an adequate system of internal controls to ensure ongoing compliance with BSA/AML regulations. Similarly, in 2008 FinCEN issued a BSA examination manual to guide reviews of money transmitters and other types of MSBs, including reviews by IRS and state regulators. Both the FFIEC and FinCEN examination manuals are publicly available. Money transmitters and banks are subject to requirements under the BSA. They are generally required to design and implement a written AML compliance program, report certain transactions to Treasury, and meet recordkeeping (including identity documentation) requirements for transfers of $3,000 or more. At a minimum, each AML compliance program must establish a system of AML compliance policies, procedures, and internal controls to ensure ongoing compliance; designate an individual to coordinate and monitor day-to-day provide training for appropriate personnel; and provide for an independent audit function to test for compliance. Additionally, banks must include appropriate risk-based procedures for conducting ongoing customer due diligence as part of their AML compliance program. BSA/AML regulations require that each bank or money transmitter tailor a compliance program that is specific to its own risks based on factors such as the products and services offered and the customers and locations served. BSA/AML compliance programs for banks—including those that service money transmitters—are expected to include the following: Customer identification program. Banks must have written procedures for opening accounts and must specify what identifying information they will obtain from each customer. At a minimum, the bank must obtain the following identifying information from each customer before opening the account: name, date of birth, address, and identification number, such as a Social Security number or a passport number. Banks’ customer identification programs must also include risk-based procedures for verifying the identity of each customer to the extent reasonable and practicable. Additionally, a bank’s customer identification program should contain procedures for circumstances when a bank cannot verify the customer’s identity, including procedures for when the bank should not open an account and when the bank should close an account. Customer due diligence procedures. These procedures assist banks in determining when transactions are potentially suspicious. Procedures must be designed to achieve two minimum regulatory requirements: (1) understanding the nature and purpose of customer relationships so customer risk profiles can be developed and (2) conducting ongoing monitoring, based on the level of risk associated with the customer, to identify and report suspicious activity and to maintain and update customer information on a risk-basis. Additional due diligence procedures. Due diligence procedures also should define when and what additional customer information will be collected for customers who banks determine may pose a higher risk for money laundering or terrorist financing. Procedures should be based on each customer’s risk profile and specific risks posed. Banks review higher-risk customers and their transactions more closely at account opening and more frequently throughout the term of their relationship with the bank. In addition, banks and money transmitters must also have policies and procedures to monitor transactions and identify suspicious activity. Monitoring generally includes (1) manual review of transaction summary reports to identify suspicious transactions or (2) automated monitoring systems that use computer algorithms to identify patterns of unusual activity. As we previously reported, banks with large transaction volumes typically use automated monitoring systems. Banks and money transmitters also must comply with certain reporting requirements: Currency Transaction Reports. Banks and money transmitters must electronically file this type of report for each transaction or a combination of transactions in a single day—such as a deposit, withdrawal, exchange, or other payment or transfer—in currency of more than $10,000. Suspicious Activity Reports (SAR). Under FinCEN regulation, banks and money transmitters are required to file this type of report when (1) a transaction involves or aggregates at least $5,000 in funds or other assets for banks or at least $2,000 in funds or other assets for money transmitters and (2) the institution knows, suspects, or has reason to suspect that the transaction is suspicious. In addition, banks’ compliance programs generally include policies and procedures that describe criteria for deciding to close or not to open an account. For example, although there is no requirement for a bank to close an account that is the subject of a SAR filing, a bank should develop criteria in policies and procedures that indicate when it will escalate issues identified through repeat SAR filings on accounts, including criteria on when to close an account. The federal banking regulators generally do not direct banks to open, close, or maintain individual accounts. The money transfer industry is diverse, ranging from Fortune 500 companies with numerous outlets worldwide to small, independent money transmitters. Some money transmitters are in communities with population concentrations that do not necessarily have access to traditional banking services. Money transmitters may send and receive funds domestically—intrastate or interstate—or internationally. Money transmitters typically work through agents—separate business entities generally authorized to send and receive money transfers. Most money transfers are initiated in person at retail outlets. Money transmitters generally operate through their own retail storefronts or through grocery stores, financial service outlets, convenience stores, and other retailers that serve as agents. In one common type of money transmitter transaction—known as a cash- to-cash transfer—a sender enters a money transmitter agent location and provides cash to cover the transfer amount and fees (see fig. 1). For transfers at or above $3,000, senders must generally provide basic information about themselves (including name and address) at the time of the transfer request. The agent processes the transaction, and the money transmitter’s headquarters screens it to validate BSA/AML compliance. The money is then transferred to a recipient via a distributing agent or bank. In an international money transfer, the money may be distributed through an agent in the destination country, wired through the money transmitter’s bank to the distributor agent’s bank, or transferred by other means to a specified agent in the recipient’s country. The distributor agent pays out cash to the recipient in either U.S. dollars or local currency. Money transfers can pose money-laundering and terrorist-financing risks, as funds related to illicit activity may go undetected due to the large volume of transactions or to money transmitters’ inadequate oversight of the various entities involved. We and others have identified money- laundering and terrorist-financing risks associated with money transmitters, including risks related to agents, customers, geographic location, and products. Agents. Money transmitters often work with multiple agents, and maintaining adequate oversight can be challenging, given the decentralized nature of the agent system. According to data collected by the Conference of State Bank Supervisors, as of December 31, 2018, 204 money transmitters reported that they had more than 440,000 agents—with nine of these money transmitters reporting that they had at least 10,000 agents. These agents present money- laundering risks if they knowingly or unknowingly fail to follow BSA/AML requirements or the policies and programs established by the money transmitter. For example, an agent may not follow the recordkeeping requirements for transfers above the regulatory funds transfer threshold or above lower thresholds that a money transmitter has self-imposed. MSB principals are required to conduct risk-based monitoring of their agents. Customers. Certain customers may pose heightened risk because of the nature of their business, occupation, or anticipated transaction activity. Additionally, in certain instances, they may be able to launder money while remaining anonymous. For example, customers may use false identities or straw men (individuals hired to conduct transfers on behalf of others) to keep from being identified as the original source of the funds. Examples of suspicious customer activity that may indicate money laundering include identification documents that cannot be easily verified; the use of different taxpayer identification numbers with variations of the same name; frequent or large transactions with no record of past or present employment; and reluctance to provide identification for transactions subject to identification requirements. Geographic location. Certain geographic locations may be more vulnerable to money laundering or terrorist financing via money transfers. High-risk geographic locations can be either international or domestic. According to FinCEN’s MSB examination manual, examples of international high-risk geographic locations include countries subject to sanctions by the Office of Foreign Assets Control or countries and territories identified as being noncooperative. Domestic high-risk geographic locations include High Intensity Drug Trafficking Areas (HIDTA) and High Intensity Financial Crime Areas (HIFCA). Products. According to the FFIEC and FinCEN MSB examination manuals, certain products and services, such as money transfers, may pose a higher risk of money laundering because of the degree of anonymity they can offer. For example, the Financial Action Task Force identified money-laundering and terrorist-financing risks associated with mobile payments because these services can sometimes allow for anonymous transactions, depending on the level of AML measures the mobile payments provider has in place. The task force also reported that virtual currency—digital representations of value such as Bitcoin that are not government-issued legal tender—could facilitate international remittances as virtual-currency- based products and services are developed. Federal agencies and international organizations have identified instances where money transfers have been used to launder proceeds from illicit activities such as human smuggling and trafficking, drug trafficking, and consumer fraud, including the following examples: In 2017, a large money transmitter entered into a $586 million settlement with the Department of Justice, the Federal Trade Commission, and the U.S. Attorney’s offices for several states after it was accused of, among other things, processing money transfers that were suspected of being used to pay human smugglers in China. In 2012, the Department of Justice found that a large money transmitter’s agents knowingly participated in a scheme in which victims wired funds to the transmitter’s agents and outlets in response to fraudulent claims such as promising victims they would receive large cash prizes or lottery winnings, falsely offering various high- ticket items for deeply discounted prices, falsely promising employment opportunities, or posing as a relative of the victim and claiming to be in trouble and in urgent need of money. In a 2011 case, seven people were sentenced for money laundering and drug trafficking involving the transfer of funds from the U.S. Virgin Islands to Alaska. Hundreds of thousands of dollars in payment for the drugs were sent using a large money transmitter in amounts averaging less than $2,000 per wire transfer, a money-laundering method known as structuring. See figure 2 for an illustrated example of structuring. In April 2005, FinCEN and the federal banking regulators issued interpretative guidance to further clarify BSA/AML requirements to banks that provide banking services to MSBs (including money transmitters) operating in the United States. According to the interagency guidance, a bank’s level and extent of due diligence beyond the minimum expectations should be based on an assessment of the individual customer’s BSA/AML risks. If a particular MSB relationship indicates a low risk of money laundering or other illicit activity, the bank may not be routinely expected to perform further due diligence beyond minimum expectations. Minimum expectations include applying the bank’s customer identification program and confirming FinCEN registration (if required), agent status (if applicable), and state and local licensing requirements (if applicable). Banks are also to conduct a basic BSA/AML risk assessment to determine the level of risk associated with the account and whether further due diligence is necessary. In order to properly assess risks, the interpretive guidance clarifies that banks should consider the purpose of the account, the types of products and services offered by the MSB, the locations and markets it serves, and the anticipated account activity (see text box). Examples of Basic Information Banks Should Consider When Assessing a Money Transmitter’s Money-Laundering Risk, According to the Interagency Guidance Purpose of account: Whether the money transmitter needs the bank account to transfer funds to its principal U.S. account or to foreign-based agents in other countries. Products and services offered: Whether the money transmitter is a principal with a fleet of agents, or is it an agent itself, and whether money transmission the customer’s primary or ancillary business (such as a grocery store that derives a small fraction of its overall revenue from providing money transmission services). Locations served: Whether the money transmitter’s market domestic or international and whether it targets local residents or broad markets. Anticipated account activity: Relevant considerations include the expected transaction amounts and whether the money transmitter is operating out of one location and using one bank branch, or whether it has several agents making deposits at multiple branches throughout the bank’s network. If a bank concludes from its risk assessment that the MSB customer presents a higher level of money-laundering or terrorist-financing risk, it will be expected to conduct additional due diligence in a manner commensurate with the heightened risk. According to the interagency guidance, the appropriate amount of due diligence depends in part on the level of perceived risk and the size and sophistication of the particular MSB. Appropriate due diligence can include reviewing the MSB’s BSA/AML compliance program, the results of the MSB’s independent testing of its program, and written agent management and termination practices for the MSB, as well as conducting on-site visits to the MSB. The interagency guidance also provides examples of “risk indicators” to assist banks with their risk assessments. Examples of potentially lower- risk indicators include a money transmitter that primarily markets to customers that conduct routine transactions with moderate frequency in low dollar amounts; is an established business with an operating history; or only remits funds to domestic entities. Examples of potentially higher- risk indicators include a money transmitter that allows customers to conduct transactions in higher dollar amounts with moderate to high frequency; is a new business without an established operating history; offers only, or specializes in, cross-border transactions, particularly to countries posing heightened risk for money laundering or terrorism financing; or is located in an area designated as a HIFCA or HIDTA. The guidance notes that in determining the level of risk, a bank should not focus on any single indicator. Rather, an effective risk assessment should be a composite of multiple factors, and depending on the circumstances, certain factors may be weighed more heavily than others. Banks’ customer risk assessments also determine the level of ongoing monitoring for suspicious activity they must perform on each customer. The interagency guidance states that, based on the bank’s assessment of the risks of its MSB customers (including money transmitters), monitoring should include periodic confirmation that initial projections of account activity have remained reasonably consistent over time. Examples of potentially suspicious activity include a money transmitter transferring funds to a different jurisdiction than expected or depositing currency significantly in excess of expected amounts without any justifiable explanation, such as an expansion of business activity or new locations. Officials from several banks we spoke with described their additional due diligence procedures for implementing BSA/AML requirements when accepting new money transmitter customers or monitoring existing ones. These include obtaining and reviewing the money transmitter’s BSA/AML policies, using questionnaires and interviews to collect detailed information from the money transmitter on its business operations—such as services offered, transaction volume, and cash activity—and site visits to verify the information collected. Officials from one bank told us that additional due diligence includes a review of the money transmitter’s business location, longevity, principal owners, transaction volume, and cash activity. Bank staff collect this information via a questionnaire administered through an in-person interview at a branch. After reviewing the information, the bank’s BSA/AML compliance department may choose to speak one-on-one with the potential money transmitter customer or conduct a site visit. When monitoring a new money transmitter customer for suspicious activity, compliance staff compare answers from the due diligence questionnaire against the customer’s cash log and wire activity to determine if the activity is outside normal parameters. The compliance department investigates any suspicious leads and reports them to the bank’s SAR committee to decide whether to file a SAR. Federal banking examiners determine whether a BSA/AML examination should include a review of a bank’s money transmitter accounts based on the overall risk profile of the bank. The FFIEC examination manual directs examiners to tailor the BSA/AML examination scope and procedures to the specific risk profile of the bank. Examiners begin a BSA/AML examination by reviewing and assessing the adequacy of the bank’s BSA/AML risk assessment. This review includes determining whether bank management has developed an accurate risk assessment that identifies significant risks to the bank (see text box). This determination is based on factors such as whether management has adequately considered all products, services, customers, transaction number and volume, and geographic locations, and whether management’s assessment methodology within these specific risk categories was adequate. Bank Secrecy Act/Anti-Money Laundering (BSA/AML) Examination Procedures for Banks In order to effectively apply resources and ensure compliance with BSA requirements, the Federal Financial Institutions Examination Council (FFIEC) examination manual is structured to allow examiners to tailor the BSA/AML examination scope and procedures to the specific risk profile of the bank. At a minimum, examiners are expected to follow core examination procedures to ensure that the bank has an adequate BSA/AML compliance program commensurate with its risk profile. The core procedures encompass four areas: Scoping and planning: Identifying the bank’s BSA/AML risks, developing the examination scope, and documenting the plan. BSA/AML risk assessment: Assessing the BSA/AML risk profile of the bank and evaluating the adequacy of the bank’s BSA/AML risk assessment process. BSA/AML compliance program: Determining whether the bank has developed, administered, and maintained an effective program for compliance with the BSA and all of its implementing regulations. Developing conclusions and finalizing the examination: Formulating conclusions, communicating findings to management, preparing report comments, developing an appropriate supervisory response, and closing the examination. In addition to the core examination procedures, the examination manual also contains sections of expanded examination procedures that address specific lines of business, products, customers, or entities that may present unique BSA/AML compliance challenges and exposures for which banks should institute appropriate policies, procedures, and processes. As examples, the examination manual contains expanded examination procedures with respect to nonbank financial institutions, electronic banking, and funds transfers. The examination manual indicates that not all of the core and expanded examination procedures are likely to be applicable to every bank. The specific examination procedures that need to be performed depend on the BSA/AML risk profile of the bank, the bank’s history of BSA/AML compliance, and other relevant factors. Examiners also review the bank’s written BSA/AML compliance program and determine whether the bank has adequately incorporated the risk it identified through its risk assessment into its BSA/AML compliance program. This review and determination include completing relevant core examination procedures for assessing key elements of the bank’s compliance program, such as the customer identification program and policies, procedures, and processes related to customer due diligence, suspicious activity reporting, and currency transaction reporting. As part of these core examination procedures, examiners conduct risk-based transaction testing, which OCC staff noted allows examiners to evaluate the adequacy of the bank’s compliance with regulatory requirements; determine the effectiveness of its policies, procedures, and processes; and evaluate suspicious activity monitoring systems. For example, examiners might determine to select and review a sample of customer accounts in testing the bank’s compliance with its policies, procedures, and processes or for possible suspicious activity. The FFIEC examination manual contains an expanded examination section for banks with significant relationships with nonbank financial institutions, which include MSBs. This expanded section references and incorporates the April 2005 interagency guidance for providing banking services to MSBs and includes related examination procedures. Consistent with this guidance, these procedures direct examiners to assess whether the bank has minimum due diligence policies, procedures, and processes in place for new or existing MSB accounts. Examiners are then to determine whether the bank’s policies, procedures, and processes to assess MSB risks effectively identify higher-risk accounts and the amount of further due diligence necessary. To assist in this effort, the manual directs examiners to perform risk- focused transaction testing on a sample of higher-risk MSB accounts. In discussion groups held with federal bank examiners, examiners from all discussion groups noted that their review of the transaction activity of money transmitter accounts is essential to determining whether the bank understands the money transmitter’s business and has appropriately assessed the risk. For example, one examiner said that customer due diligence procedures at account opening should include the appropriate qualitative and quantitative questions so that the bank can make a reasonable determination of the types and volumes of transactions that will be flowing in and out of the account. Examiners from all discussion groups said that when assessing the bank’s risk assessment of a money transmitter, they focus on whether the bank has considered the risk factors discussed in the examination manual, including geography, customer type, products, services, and transactional volume. In some discussion groups, examiners noted that they may review money transmitter accounts if these accounts are included in the sampling of bank customer accounts as part of the core examination procedures. One examiner said that because banks in her region do not tend to specialize in money transmitters or have a significant degree of risk from them, the only time she reviews money transmitter accounts is if they are included in her sample for transaction testing. Examiners from one discussion group said that they may review money transmitters as part of expanded examination review procedures for nonbank financial institutions if the bank has a large portfolio of money transmitter accounts. For example, one examiner said he generally does not set out to look for and review money transmitter accounts when conducting a BSA/AML examination, but in one case his examination team learned that during the course of a merger, a bank acquired a number of nonbank financial institutions, including MSBs. As this bank did not have prior experience with these kinds of customers, the examination team decided to include them in the scope of their review. Examiners in all discussion groups said that they neither instruct nor recommend that banks close accounts with money transmitters or other types of MSBs. Although IRS and state agencies also examine money transmitters and other MSBs, examiners from all discussion groups said that BSA/AML requirements and guidance do not allow banks to rely on IRS or state oversight. These examiners said these reports could provide banks with a useful additional source of information when conducting their due diligence on MSB customers. However, these examiners added that the reports would not substitute for or reduce the due diligence expected of banks in complying with BSA/AML compliance program requirements.. Examiners from most discussion groups observed that they know very little about the quality of state or IRS examinations of MSBs and their frequency. Examiners in our discussion groups said the challenges that some banks face in ensuring BSA/AML compliance for their MSB customers include those related to customer due diligence, risk assessments, customer identification, and BSA/AML compliance staff and resources. Customer due diligence. Examiners from most discussion groups said that some banks do not fully understand the customer due diligence requirements for banking MSBs. Examiners in some discussion groups said that banks do not always fully review or understand the documents and information obtained from their MSB customers in conducting due diligence. One examiner described an instance where bank staff could not understand documentation collected from MSB customers in a foreign language. Examiners in some discussion groups said banks do not understand the need to conduct ongoing monitoring of MSB accounts, including of the flow and volume of customers’ transactions. For example, one examiner in a different discussion group described an instance of a community bank that was unaware that an MSB account had $2 billion flowing through annually even though the bank had only $1 billion in assets. Examiners in some discussion groups said that banks also may not fully understand their automated software for monitoring suspicious activity or how to set the proper software parameters for capturing potentially suspicious transactions. One examiner in a different discussion group said that without proper monitoring, a bank would not know when sudden changes in MSB customers’ transaction types or volumes would be considered suspicious and should be reported. Risk assessment. Examiners in many discussion groups said some banks do not appropriately assess their MSB customers’ risk, either because they do not consider relevant risk factors or they rate all MSB customers at the same risk level. One examiner in a discussion group said he examined a bank with many money transmitter customers that transmitted funds to several countries and found that the bank did not assess the risk levels of the countries to which the money transmitters sent funds. An examiner in a different discussion group said that banks often assess all MSBs at the same level of risk because they do not understand the difference between the various risk levels. Another examiner in the same discussion group added that banks often do not understand the guidance clarifying that banks should assess each customer’s risk individually. This statement was corroborated by our review of several banks’ BSA policies that stipulated that all money transmitters and other MSBs should be considered high risk, contrary to the 2005 guidance. Customer identification. Examiners from many discussion groups said banks do not always identify their MSB customers—for example, when a bank acquires another bank without being aware that the acquired bank has MSB customers. Examiners in some discussion groups said that failure to properly identify MSB customers stems partly from inadequate due diligence or risk assessment. BSA/AML compliance staff and resources. Examiners in many discussion groups said that some banks do not have sufficient BSA/AML compliance staff or resources to manage their BSA/AML compliance programs. For example, an examiner in one discussion group described a bank with nearly 70 money transmitters and more than 200 check cashers but only four staff in its BSA/AML compliance department, which the examiner considered inadequate. Examiners in many discussion groups said that BSA/AML deficiencies generally stem from overall weakness in a bank’s BSA/AML compliance program or internal controls, and not from providing services to money transmitters or any particular customer type. An examiner from one discussion group noted that a bank with weak internal controls around money transmitters likely has weak internal controls across its BSA/AML compliance program. Examples of deficiencies provided by examiners across discussion groups include banks failing to follow written policies and procedures, rating entire categories of customers as high-risk rather than assessing individual customer risk, not conducting on-site customer reviews, failing to conduct other due diligence, and not properly monitoring and reporting suspicious activities. Moreover, our review of bank examination documents found that BSA/AML-related deficiencies mostly stemmed from weakness in banks’ BSA/AML compliance programs and internal controls overall—for example, in customer identification programs, customer due diligence procedures and practices, and risk assessments—and not from a bank providing services to MSBs or any other customer type. According to examiner discussion groups and examination documents we reviewed, not all banks with MSB customers experience BSA/AML compliance challenges. Examiners in some discussion groups noted that banks that successfully provide accounts to MSBs, including money transmitters, tend to have a strong BSA/AML compliance program. For example, examiners in some discussion groups said that such banks have internal controls commensurate with the BSA/AML risks of the MSB customers, including conducting appropriate monitoring and due diligence of customers, and understand the full scope of MSB customers’ activities. The examiners stated that these banks also have sufficient BSA/AML compliance staff who received training. Similarly, our review of bank examination documents included examples of banks with MSB customers that complied with BSA/AML compliance program requirements, such as a community bank with 80 money transmitters. In the examination documents we reviewed, examiners noted that although the bank engaged in higher-risk business, it was managing the risk appropriately. While views among examiners in our discussion groups varied, examiners in some discussion groups identified challenges in assessing banks’ customer due diligence for money transmitters and other MSB customers. As discussed earlier, the FFIEC examination manual includes an expanded examination section for nonbank financial institutions that provides procedures and guidance for examiners when assessing banks’ compliance controls for MSB customers, including money transmitters. The procedures direct examiners to determine whether the banks’ policies, procedures, and processes to assess risks posed by MSB customers allow the banks to effectively identify higher-risk accounts and the amount of further due diligence that is necessary. The expanded examination guidance provides examples of actions banks can take to meet the additional due diligence requirement for customers they deem to be higher risk. Examiners from many discussion groups said they believe these procedures and guidance are sufficient. One examiner noted that assessing controls is the same for a bank’s MSB customers as for any other type of customer. However, examiners from some discussion groups said it was unclear how much due diligence is reasonable to expect banks to conduct for their money transmitters and other MSB customers. An examiner in one discussion group said it was not clear from the examination procedures and guidance how much banks were expected to question and request information from their MSB customers or monitor their MSB customers’ due diligence efforts without expecting banks to act as the de facto regulator for MSBs. Other examiners noted that although banks are responsible for understanding the kinds of transactions that flow through an MSB, to some extent banks do not have visibility into these individual transactions, as they are aggregated before flowing into the account at the bank. Similarly, another examiner said there was uncertainty about how critical an examiner should be of a bank’s due diligence efforts in cases where a bank’s documentation on an MSB customer’s BSA/AML compliance program is lacking. One examiner noted that while the examination guidance provides examples of due diligence actions banks can consider performing, those actions are not requirements. The examiner said it was therefore not clear to what extent examiners should apply these examples as criteria and expect banks to have implemented them. Further, examiners in some discussion groups said that it can be difficult to evaluate banks’ risk assessments, including processes for identifying higher-risk customers that require additional due diligence. One examiner said that it is unclear from the examination procedures how to determine whether banks’ risk assessment processes for identifying higher-risk customers are adequate. An examiner in a different discussion group said that in evaluating banks’ risk assessment of new money transmitter customers, he looks for whether banks ask why new customers switched banks. However, other examiners in the same discussion group noted that this is not a standard question. Our review of the expanded examination section found a lack of examples of specific steps or processes that examiners can take in assessing banks’ compliance for additional due diligence. For example, this section’s procedures contain only a general reference that examiners should determine whether the banks’ policies, procedures, and processes effectively allow the banks to identify and conduct risk-based due diligence for higher-risk customers and lack specific examples to assist examiners in evaluating additional due diligence activities. The section’s guidance states that examiners could take actions, including reviewing an MSB’s BSA/AML compliance program or conducting on-site visits to help evaluate a bank’s compliance. But neither the guidance nor the procedures clarify what these reviews or visits might entail. In comparison, the expanded section’s guidance and procedures include examples of specific steps that examiners can take when assessing banks’ compliance with minimum due diligence requirements for MSB accounts, such as applying the bank’s customer identification program and confirming FinCEN registration status and state licensing, if applicable. Officials from the Federal Reserve and OCC said that the examination manual is not intended to provide explicit criteria for examiners when they are assessing the adequacy of a bank’s program. They said that establishing explicit criteria would result in a “check the box” approach to BSA/AML compliance, such that banks are given a uniform set of requirements to follow, irrespective of the money-laundering or terrorism- financing risks associated with their banking activities. They said that if banks only needed to meet specific requirements, such an approach would encourage banks to do the minimum to establish a BSA/AML compliance program and would not effectively detect and deter money laundering and terrorism financing. As discussed earlier, the examination manual is instead structured to allow examiners to tailor the BSA/AML examination scope and procedures to the specific risk profile of the bank. Staff from the federal banking regulators said that as a result, examiners are expected to apply their judgment in evaluating banks’ BSA/AML compliance programs. However, while regulators want compliance programs to be tailored to the unique risks a bank’s operations present, examiners need sufficient guidance to determine whether a given bank’s BSA/AML-related policies, processes, and procedures are adequate. Regulators and FinCEN issued the 2005 interagency guidance to clarify BSA/AML requirements and supervisory expectations for banks when providing banking services to money transmitters and other MSBs. Since then, examiners have relied on this guidance when reviewing banks’ MSB customer accounts. However, the examination procedures and related guidance may not provide all of the information examiners need to conduct their assessments, as indicated by the examiners in some of our discussion groups who reported that it is not clear to them how to determine whether banks’ due diligence efforts are adequate. Providing clarifying information would not compromise examiners’ ability to exercise judgement during an examination. Rather, it would provide them with greater certainty that they are evaluating banks’ compliance with BSA/AML requirements appropriately. Federal internal control standards state that agencies should identify, analyze, and respond to risks related to achieving the defined objectives. Unless federal banking regulators take steps to improve examiners’ ability to evaluate banks’ compliance controls with respect to money transmitter accounts, examiners may not be fully achieving the BSA/AML examination objectives of identifying and assessing risks and banks’ ability to manage risks, as set out in the examination manual in assessing banks’ compliance with BSA/AML requirements. Internal control standards also state that agencies should internally communicate the necessary quality information to achieve their objectives. With respect to examiners, such communication could include providing updates to examination procedures, examiner training, or a combination of methods. We estimate that 32 percent of banks nationwide provided accounts to money transmitters from 2014 through 2016, based on the results of a survey we conducted jointly with other GAO work on derisking. For calendar year 2016, of the 91 banks that reported having money transmitters as customers, 71 banks of varying asset sizes reported having 41,089 money transmitter accounts (see table 1). Overall, of the 91 banks that reported having money transmitters as customers, close to half of them (40 banks) terminated at least one of their money transmitter accounts and almost one-third of them (29 banks) limited the number of accounts with money transmitters, both for reasons related to BSA/AML risk, from 2014 through 2016 (see table 2). Because extra-large banks reported having a much greater number of accounts with money transmitters, these banks also reported a greater proportion of account terminations, compared with small and medium banks. Specifically, 18 banks of all sizes that responded to the survey reported that they terminated 1,098 accounts in 2016—with 89 percent of these account closures (976 out of 1,098) reported by six extra-large banks. In particular, one extra-large bank accounted for more than half (601 out of 1,098) of the account terminations in that year. See table 3 for more information on account terminations in 2016. See appendix II for more information on account terminations and limitations. Some terminations and limitations of money transmitters’ bank accounts appear to be associated with managing BSA/AML risk. However, some terminations and limitations raise derisking concerns. Some reasons that banks reported for terminating accounts were associated with managing BSA/AML-related risk, including the filing of SARs associated with the account and customers failing to provide information necessary for the bank to conduct adequate BSA/AML due diligence. Some banks also reported terminating accounts to reduce the risk that a customer’s activity could harm a bank’s reputation, known as reputational risk (see table 4). These survey results are consistent with the results of our prior work on banks in the Southwest border region. The most commonly cited reason in our survey for terminating accounts was the filing of SARs. Officials we interviewed from one bank told us that they investigate customers that have triggered multiple SAR filings and considered setting up controls to limit account activities. Officials of another bank told us that a federal bank examiner suggested that the bank consider closing an account with a money transmitter customer because of SAR filings associated with it. The second most commonly cited reason for terminating accounts was that a customer failed to provide information requested by a bank for conducting BSA/AML due diligence. Officials we interviewed from two banks told us that customers may not be able to provide information and documentation or may not disclose that they are an MSB when opening new accounts. Officials of a bank that maintained accounts with money transmitters told us they terminated accounts in instances where a money transmitter did not submit required documentation. Another commonly cited reason for terminating accounts was reputational risk—the potential that negative publicity regarding an institution’s business practices, whether true or not, will cause a decline in the customer base, costly litigation, or revenue reductions. One bank’s officials said in an interview that when examiners inquired as to whether bank officials factor reputational risk into their decision-making about money transmitters, they viewed such inquiries as implicit suggestions that the bank had an issue with reputational risk that needed to be addressed. Examiners in our discussion groups also shared similar comments on suspicious activity monitoring and banks’ requests for information. Specific to suspicious activities, one examiner noted that banks generally have an internal policy stating that if a specific number of SARs are filed on the customer, the bank will automatically terminate the account. Regarding banks’ information requests, examiners in some discussion groups said they observed that banks may terminate an MSB’s account if the MSB does not comply with the bank’s request for due-diligence- related documentation. Three of the most common reasons banks reported for limiting accounts with money transmitters were that (1) the cost of BSA/AML compliance made the customer type unprofitable, (2) the banks were unable to manage the BSA/AML risk associated with the customer type, and (3) the customer type fell outside of a bank’s risk tolerance (see table 5). One of the most commonly cited reasons for limiting the number of accounts with money transmitters was compliance costs associated with managing BSA/AML risk. Officials of about two-thirds of the banks we interviewed said their BSA/AML compliance costs had increased over time, with eight institutions specifically citing past or planned upgrades to their monitoring software systems as one source of increasing costs. Moreover, officials of one bank said their compliance costs had increased in recent years as a result of regulatory scrutiny, which they said had increased as MSBs came to comprise a larger portion of their customer base. In response to this heightened scrutiny, officials said the bank had installed a new transaction-monitoring platform, which incurred a one-time migration cost and would incur higher monthly fees, and was considering expanding its compliance department. Officials of three banks told us in interviews that 50 percent of their compliance costs stem from BSA/AML compliance. As we have reported previously, money transmitters are generally low-profit customers for banks, in that the revenue from their accounts may not be sufficient for some banks to offset the associated costs of BSA/AML compliance. For example, officials of one bank said the bank spent about $250,000 annually to maintain its BSA-related monitoring software and training, which they believed was a significant portion of the bank’s $25 million annual income. These officials told us that unlike the bank’s other customers, which use the bank’s other products and refer business, money transmitters are not the bank’s core customers and do not use other products or services, so the bank would rather focus its time and resources on its core customers. Similarly, officials of another bank said they decided not to bank MSBs because any revenue generated would not cover the additional resource and compliance costs. Banks’ inability to manage BSA/AML risks associated with money transmitter customers was another commonly cited reason for limiting the number of accounts. For example, officials of one bank we interviewed said they did not accept any MSB customers, including money transmitters, because they were not willing or able to take on the required risk and level of BSA/AML monitoring. Another commonly cited reason for limiting accounts was that a customer type fell outside of a bank’s risk tolerance. In interviews, banks expressed concerns about their MSB customers’ ability to maintain an adequate BSA/AML compliance program. One bank’s officials told us that owners of gas stations may offer check cashing or money transmission services to generate additional revenue, but they may not be aware that offering such services would subject their business to BSA/AML compliance requirements. Another bank’s officials also said that many business owners do not know that they have to register with FinCEN to operate as an MSB. Officials from a third bank said that some MSBs may not understand the BSA/AML regulations and, at their customers’ request, may inadvertently commit a violation such as structuring that may generate a SAR (for example, by breaking up a money transfer in excess of $10,000 into multiple transfers to avoid generating a Currency Transaction Report). Similarly, examiners in many discussion groups said that the staffing and resource costs required for adequate monitoring and due diligence on MSB customers, including money transmitters, are reasons why some banks may choose not to bank MSBs. Moreover, examiners in many discussion groups also said some banks offer MSBs accounts and then find out that they do not have the necessary BSA/AML expertise or that the business is not profitable for them. For example, one examiner said that when a larger bank in his area terminated all of its money transmitter accounts, a number of smaller banks looking for profit offered accounts to these money transmitters. However, the examiner added that the smaller banks did not understand the level of customer due diligence and monitoring that was required for these accounts and the associated costs, and they terminated the accounts. In contrast, examiners in some discussion groups said that some community banks have accepted money transmitter customers as a way to generate potentially substantial fee income. According to survey responses from banks, the most commonly cited reason for limiting the number of money transmitter accounts was that the customer type drew heightened BSA/AML regulatory oversight—behavior that would indicate derisking. Banks also commonly cited this reason for terminating money transmitter accounts. For example, officials from one bank told us that the bank no longer offered services to MSBs because it wanted to be viewed favorably by regulators. Officials of another bank said that money transmitter account closures were generally the result of onerous regulatory requirements and increased regulatory scrutiny. Officials from two banks we spoke with said that they received greater regulatory scrutiny after increasing their number of MSB customers, which affected their willingness to open additional accounts with MSBs. According to officials of one of the two banks, when the bank increased its MSB customers from one to two, the institution was assessed as high risk by examiners. Related to heightened regulatory oversight, some banks’ officials we interviewed also expressed concerns that some examiners’ expectations go beyond what is described in the examination manual. For example, they said examiners expected banks to know their customers’ customers—although BSA/AML regulations do not require banks to obtain information on their customers’ customers. Bank officials said ascertaining such information was difficult because money transmitters’ customers are one step removed from the bank. Some banks’ officials also told us that they felt obligated to follow examiners’ verbal suggestions, even when the suggestions did not appear in the final examination report as recommendations. Other banks’ officials we interviewed stated that although examiners did not explicitly recommend that banks exit certain lines of business, officials felt pressure from the examiners to do so. For example, officials from one bank said examiners suggested that if the bank exited certain lines of business, the bank would not have deficiencies in its BSA/AML compliance program. We reported similar concerns in our March 2018 report. About half of the banks we interviewed for that report said that the fear of regulatory scrutiny served as a disincentive for banks to maintain accounts with money transmitters. Some banks’ officials expressed uncertainty about the amount of due diligence required for regulatory purposes because regulations included ambiguous language or because examiner practices exceeded regulations. These bank officials suggested that regulators could provide more specific guidance for banks on risk management, such as by including example scenarios and answers to frequently asked questions. Conversely, some banks we interviewed had a different experience. For example, officials of one bank told us that examiners’ interpretation of BSA/AML principles did not differ from the bank’s understanding of those principles. Officials added that when they initially began preparing risk assessments, they sought feedback and advice from their examiners and that examiners now use the bank’s risk assessment as an example for other banks. Moreover, these officials said that if they need clarification on BSA/AML compliance requirements, they contact FinCEN, which has been responsive to their questions. Officials of another bank told us they have a good relationship with their federal regulator and said that examiners follow BSA guidance and have been consistent in conducting their examinations. Officials of two other banks told us that their BSA/AML examinations have been consistent with guidance and requirements and that examiners have not told officials what types of customers to avoid. We also reported in February 2018 that recent BSA/AML law enforcement and regulatory enforcement actions have caused some banks to become more conservative in the types of businesses to which they offer accounts. In our interviews for the February 2018 report, officials of three banks and an industry group expressed concerns about potential enforcement actions, including civil penalties, if banks’ employees make mistakes in BSA/AML monitoring. In 2012, federal regulators assessed civil money penalties—including a $500 million penalty assessed by OCC and a $165 million penalty by the Federal Reserve—against HSBC Bank for, among other things, failing to maintain an effective BSA/AML compliance program and failing to conduct appropriate due diligence on foreign correspondent bank account holders. As another example, in March 2018, OCC issued consent orders for civil penalties against three senior executives of the Merchants Bank of California for violations of consent orders related to monitoring BSA/AML compliance. In our interviews, officials of an industry association told us that fines associated with BSA violations are especially difficult for community banks to absorb and could result in the bank going out of business. Similarly, examiners from a discussion group said some banks may decide not to offer accounts to MSBs to avoid heightened regulatory scrutiny. For example, examiners said some banks likely want to avoid BSA/AML risk entirely when they decide not to offer MSBs accounts. One examiner thought that some banks lack understanding regarding the business models of MSBs and that it is easier for them not to provide them accounts. In some cases, banks offer MSBs bank accounts but on a limited basis. For example, examiners from one discussion group said that in some cases, banks manage their BSA/AML risks by maintaining existing MSB accounts but not offering accounts to new MSB customers. In a 2015 speech, a senior Treasury official noted banks’ concerns about the cost of complying with BSA/AML requirements, uncertainty about supervisors’ expectations regarding appropriate due diligence, and the nature of the enforcement and supervisory response if they make a mistake. Moreover, the official stated that the banks held the perception that supervisory and enforcement expectations lack transparency, predictability, and consistency. The official also said that this perception feeds into higher anticipated compliance costs and may eclipse any potential economic gains of taking on new MSB customers. To address these concerns, the senior official stated that policymakers needed to continue to improve their understanding of the scope, nature, and drivers of the problem through better data collection and continue to explore ways to improve the effectiveness of their communication. According to money transmitters we spoke with, effects of account terminations due to derisking include ceasing of operations, loss of revenue, higher costs for services provided, and failure of the business. For example, officials from one large money transmitter that operates in the United States and internationally said that in recent years, about 100 of the money transmitter’s agents have lost accounts with their local and regional banks each month. The officials added that when banks terminate accounts with the money transmitter or its agents, the money transmitter cannot conduct the necessary transactions with its agents to facilitate the cash transfer. As a result, officials told us, account terminations can cause the money transmitter to cease operations in a particular country or cause the agents to go out of business. These officials also told us that some banks have terminated accounts with their institution while maintaining accounts with other money transmitters. These officials said they obtained legal injunctions for unfair competitor treatment in some of these cases. Officials of a smaller, regional money transmitter said that they have experienced 10 account terminations since 2006. Moreover, the officials said that they have to switch banks every 2 to 3 years because of account terminations and that it is getting more difficult to find a bank willing to take on money transmitters as customers. For example, the officials said that they called about 300 banks in a state and only two banks were willing to open accounts with them. The money transmitter’s officials said it has had to cease operations in three states due to account terminations. The officials said that the money transmitter now focuses on opening accounts with community banks and credit unions, but these institutions may be too small to handle the money transmitter’s volume of deposits. Another money transmitter told us that it takes about 3 months to open an account with a bank. Moreover, as a result of account terminations and limitations by banks, the money transmitter has had to reduce its number of employees from 220 to 180 and has not been able to open new locations. Another money transmitter said that account terminations have affected its ability to obtain accounts with other banks. In our March 2018 report, we found that some money transmitters—those that may be considered higher risk based on the 2005 interagency guidance—may utilize nonbank channels for transferring money as a response to account terminations. Specifically, we reported that as a result of banks’ account terminations and limitations, some money transmitters serving fragile nations have relied on nonbank channels, such as cash couriers and armored trucks, to transfer money domestically and abroad. We further reported that using cash couriers or armored trucks to move money increases costs and risks of theft and safety. Account terminations and limitations by banks also affect money transmitters that do not serve customers abroad—money transmitters that could be considered lower risk based on the 2005 interagency BSA guidance. For example, a company that acquired another business offering money transmission services to customers within the United States also experienced account terminations. When the company acquired the new business and thus the business’s money transmission license, its bank refused to service the company because of its newly acquired status as a money transmitter. In another example, officials of a money transmitter that serves only U.S. customers told us they have difficulty opening accounts and have experienced account terminations often. Officials said that their business has stopped at times because they did not have any bank accounts to facilitate money transmission. Additionally, account closures also may affect money transmitters’ customers. For example, some money transmitters we interviewed said they passed on increased costs resulting from account closures to their customers. Specifically, officials of one large money transmitter said that because of derisking, banks that still do business with them are charging higher fees. The officials added that they try to absorb the higher fees but have passed on the increased costs to their customers in some markets. In contrast, some money transmitters told us in interviews that although their costs have increased, they have not increased customer fees. Several money transmitters told us that banks did not always provide reasons for terminating their accounts. Some said they believe that banks terminate accounts due to regulatory pressure, compliance costs, or changes in a bank’s policy or risk appetite. One money transmitter stated that the problem of account terminations due to derisking stems from banks being too afraid to bank MSBs, including money transmitters. In response to banks’ account terminations and limitations, some money transmitters—including those with characteristics considered to be higher and lower risk according to the 2005 interagency guidance—now maintain accounts with multiple banks to help ensure they can continue operating should a bank close their account. For example, officials of the company that acquired another business offering domestic money transmission services told us they maintain accounts with more than one bank, but they said it is difficult and costly to do so. Officials of another money transmitter said that to help prevent disruptions to their ability to transfer funds when they experience an account closure, they try to have back-up accounts at other banks. Some money transmitters also engage with their banks’ management to better understand what banks expect from them in meeting compliance requirements. For example, an official from one money transmitter said the money transmitter tries to meet with its banks’ financial crimes teams to better understand how it can help minimize the risk of facilitating money transfers for terrorist-financing and money-laundering purposes. Officials of another money transmitter told us that as a result of meeting with bank management, the money transmitter added additional employees to its compliance department and bought new monitoring software to fulfill its bank’s requirement for monthly monitoring of transactions. FinCEN and the federal banking regulators have responded to concerns about the derisking of money transmitters and other MSBs on a national level by issuing guidance to banks to clarify expectations for providing banking services to these customer types. In March 2005, the federal banking regulators and FinCEN issued a joint statement noting that MSBs were losing access to banking services as a result of concerns about regulatory scrutiny, the risks presented by MSB accounts, and the costs and burdens associated with maintaining such accounts. According to the joint statement, these concerns might have stemmed, in part, from banks’ misperception of the requirements of the BSA and the erroneous view that MSBs present a uniform and unacceptably high risk of money laundering or other illicit activity. The joint statement recognized that the MSB industry provides valuable financial services, especially to individuals who may not have ready access to the formal banking sector. It further noted that it is important that MSBs comply with the requirements of the BSA and applicable state laws and remain within the formal financial sector and be subject to appropriate AML controls. The joint statement announced the intent of the regulators and FinCEN to issue the interagency guidance for banks on providing services to MSBs, which, as previously discussed, was intended to clarify BSA requirements and supervisory expectations as applied to accounts opened or maintained for MSBs. More recently, in November 2014, FinCEN issued a statement reiterating that banks can serve the MSB industry while meeting their BSA obligations and referring to the interagency guidance to banks on providing services to MSBs. The statement noted concerns that banks were indiscriminately terminating the accounts of all MSBs, or refusing to open accounts for any MSBs, thereby eliminating them as a category of customers. It noted, similar to the March 2005 joint statement, that regulatory scrutiny, the perceived risks presented by MSB accounts, and the costs and burdens associated with maintaining such accounts appeared to play a part in these decisions. In the 2014 statement, FinCEN cautioned that a wholesale approach to MSB customers runs counter to the expectation that financial institutions can and should assess the risks of customers on a case-by-case basis. Similarly, it noted that a blanket direction by U.S. banks to their foreign correspondents not to process fund transfers of any foreign MSBs, simply because they are MSBs, runs counter to the risk-based approach. FinCEN stated that refusing financial services to an entire segment of the industry can lead to an overall reduction in financial sector transparency, and that such transparency is critical to making the sector resistant to the efforts of illicit actors. Federal banking regulators also issued separate statements addressing BSA/AML risk posed by MSBs and foreign banks. See table 6 for a summary of key statements and guidance related to MSBs issued in recent years by FinCEN and the federal banking regulators. In 2018, we reported that regulators had taken only limited steps to understand how banks’ regulatory concerns and BSA/AML compliance efforts may be influencing banks to derisk. We reported that regulators had taken some actions in response to derisking, including issuing the guidance previously discussed, and that some agencies took steps aimed at trying to determine why banks may be terminating accounts. We also reported that regulators had conducted retrospective reviews on some BSA/AML requirements. We noted that actions regulators had taken to address concerns raised in BSA/AML retrospective reviews had focused primarily on the burden resulting from the filing of Currency Transaction Reports and SARs. However, we noted that these actions had not been aimed at addressing—and, if possible, ameliorating—the full range of factors that influence banks to engage in derisking, particularly how banks’ regulatory concerns and BSA/AML compliance efforts may be influencing their willingness to provide services. We concluded that without a broader assessment of the full range of BSA/AML factors that may be influencing banks to derisk, FinCEN, the federal banking regulators, and Congress do not have the information needed to determine if BSA/AML regulations and their implementation are achieving their regulatory objectives in the most effective and least burdensome way. Therefore, we recommended that FinCEN and the federal banking regulators conduct a retrospective review of BSA regulations and their implementation for banks, with a focus on how banks’ regulatory concerns may be influencing their willingness to provide services. According to the federal banking regulators and FinCEN, they and Treasury established an interagency working group in early 2018 that they believe will address our recommendation. The interagency working group is intended to identify ways to improve the efficiency and effectiveness of BSA/AML regulations, supervision, and examinations while continuing to meet the requirements of the BSA and its implementing regulations, supporting law enforcement, and reducing BSA/AML compliance burden. Staff from FinCEN and the federal banking regulators identified several interagency statements that the working group has completed. Interagency Statement on Sharing BSA Resources (issued on October 3, 2018): This statement clarified how banks may reduce the costs of meeting BSA requirements effectively by sharing employees or other resources in a collaborative arrangement with one or more banks. The statement highlighted potential benefits to sharing resources and provided examples of resources that may be appropriate to share, such as certain internal controls, independent testing, and BSA/AML training functions. The statement also highlighted potential risks of sharing resources and cautioned that any collaborative arrangements should be designed and implemented according to each bank’s risk profile. Joint Statement on Innovative Efforts to Combat Money Laundering and Terrorist Financing (issued on December 3, 2018): This statement clarified the working group’s position with respect to innovative approaches in BSA/AML compliance and encouraged banks to consider such approaches. For example, some banks are experimenting with artificial intelligence and digital identity technologies applicable to their BSA/AML compliance programs. The statement notes that these innovations and technologies can strengthen BSA/AML compliance approaches and that the regulators welcome these types of innovative approaches to further efforts to protect the financial system against illicit financial activity. According to the statement, pilot programs undertaken by banks to test and validate the effectiveness of innovative approaches should not subject banks to supervisory criticism even if the pilot programs ultimately prove unsuccessful. Joint Statement on Risk-Focused Bank Secrecy Act/Anti-Money Laundering Supervision (issued on July 22, 2019): This statement was intended to improve the transparency of the risk-focused approach used for planning and performing BSA/AML examinations. In this statement, FinCEN and the banking regulators emphasized that they scope their examinations in response to the unique risk profile for each bank because banks vary in focus and complexity. The regulators also clarified common practices for assessing a bank’s risk profile, including leveraging available information such as the bank’s own risk assessment, contacting the banks between examinations, and considering the bank’s ability to identify, measure, monitor, and control risks. Federal banking regulators and FinCEN staff said the working group’s focus on regulatory reform and on reducing the burden associated with BSA/AML compliance may indirectly address derisking concerns, including those related to money transmitters. In particular, they said these efforts may help agencies as they clarify their supervisory expectations for banks with respect to managing BSA/AML risk. For example, the staff said that the joint statement on the risk-focused approach to supervision clarifies that the role of the examiner is not to determine what level of risk a bank should assume. Instead, the examiners should review risk management practices to evaluate whether a bank has effective processes to identify, measure, monitor, and control risks and to assess the effectiveness of a bank’s processes. They said that reminding examiners and institutions of the risk-focused approach will help dispel the perception that banks will be criticized for taking certain higher-risk customers when the bank is properly managing that risk. Similarly, they said that the joint statement on innovation could help address derisking concerns because it allows banks to leverage new technologies and innovative approaches to help reduce costs of implementing the strong risk management practices that may be necessary to provide banking services to some higher-risk customers. The actions taken to date by the interagency working group are important steps toward improving the efficiency and effectiveness of BSA/AML regulations and supervision. As previously discussed, one reason some banks reported terminating or limiting money transmitter accounts was because of the cost associated with BSA/AML compliance. The interagency statements on sharing BSA resources and innovative efforts to combat money laundering and terrorist financing could help reduce banks’ implementation costs associated with providing banking services to potentially higher-risk customers. However, consistent with our prior work, our evidence demonstrates that banks terminate or limit customer accounts not only as a way to address legitimate money-laundering and terrorist-financing threats, but also as a way to manage regulatory concerns, which may indicate derisking. Reminding examiners and banks of the risk-focused examination approach may help to dispel the perception that banks will be criticized for taking certain higher-risk customers when the bank is properly managing that risk and may indirectly address some factors that influence banks to derisk. Nevertheless, the working group has not yet considered whether there are other supervisory concerns that factor into banks’ decisions to derisk. As we stated in our prior work, it is important to evaluate and address the full range of factors that may be influencing banks to derisk. Therefore, we maintain that FinCEN and the banking regulators should continue to work toward implementing our prior recommendation to conduct a retrospective review of BSA/AML regulations focusing on how banks’ regulatory concerns may be influencing their willingness to provide services. Regulators and FinCEN issued the 2005 interagency guidance to clarify BSA/AML requirements and supervisory expectations with regard to accounts banks open or maintain for money transmitters and other MSBs. However, some examiners in our discussion groups said they were unclear about how much due diligence is reasonable to expect banks to conduct for their money transmitters. Improving examiners’ ability to evaluate banks’ BSA/AML compliance controls with respect to money transmitter accounts would help ensure that such evaluations are done in accordance with BSA/AML examination objectives of identifying and assessing risks and banks’ ability to manage risks, as set out in the examination manual. Options for making such improvements could include providing examiners with more detailed examination procedures, enhanced information, additional training, or a combination of methods. We are making a total of four recommendations to the Federal Reserve, OCC, FDIC, and NCUA: The Board of Governors of the Federal Reserve System should, in coordination with the other federal banking regulators, and with input from BSA/AML examiners and other relevant stakeholders, take steps to improve examiners’ ability to evaluate the effectiveness of banks’ BSA/AML compliance controls with respect to money transmitter accounts. Steps may include providing updates to examination procedures, examiner training, or a combination of methods. (Recommendation 1) The Comptroller of the Currency should, in coordination with the other federal banking regulators, and with input from BSA/AML examiners and other relevant stakeholders, take steps to improve examiners’ ability to evaluate the effectiveness of banks’ BSA/AML compliance controls with respect to money transmitter accounts. Steps may include providing updates to examination procedures, examiner training, or a combination of methods. (Recommendation 2) The Chairman of the Federal Deposit Insurance Corporation should, in coordination with the other federal banking regulators, and with input from BSA/AML examiners and other relevant stakeholders, take steps to improve examiners’ ability to evaluate the effectiveness of banks’ BSA/AML compliance controls with respect to money transmitter accounts. Steps may include providing updates to examination procedures, examiner training, or a combination of methods. (Recommendation 3) The Chairman of the National Credit Union Administration should, in coordination with the other federal banking regulators, and with input from BSA/AML examiners and other relevant stakeholders, take steps to improve examiners’ ability to evaluate the effectiveness of banks’ BSA/AML compliance controls with respect to money transmitter accounts. Steps may include providing updates to examination procedures, examiner training, or a combination of methods. (Recommendation 4) We provided a draft of this report to the Federal Reserve, FDIC, NCUA, OCC, and Treasury’s FinCEN for review and comment. The federal regulators provided technical comments on the draft report, which we have incorporated as appropriate. The Federal Reserve, FDIC, NCUA, and OCC also provided written comments (reproduced in appendixes III through VI). They agreed with GAO’s recommendations and expressed a commitment to implement them. We are sending copies of this report to the appropriate congressional committees, the Director of the Financial Crimes Enforcement Network, the Chairman of the Board of Governors of the Federal Reserve System, the Chairman of the Federal Deposit Insurance Corporation, the Comptroller of the Currency, and the Chairman of the National Credit Union Administration. The report will also be available at no charge on our website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-8678 or clementsm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are listed on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII. This report (1) describes regulators’ Bank Secrecy Act (BSA)/anti-money laundering (AML) supervisory expectations for banks that provide services to money transmitters and other money services businesses (MSB) and examiner views on bank challenges in complying with these requirements; (2) examines challenges reported by examiners in conducting BSA/AML assessments; (3) examines the extent to which banks are terminating or limiting money transmitters’ access to banking services and the effects on money transmitters; and (4) evaluates how the Department of the Treasury’s Financial Crimes Enforcement Network (FinCEN) and the federal banking regulators have assessed and responded to concerns about the derisking of money transmitters. The federal banking regulators included in our review are the Board of Governors of the Federal Reserve System (Federal Reserve), the Federal Deposit Insurance Corporation (FDIC), the Office of the Comptroller of the Currency (OCC), and the National Credit Union Administration (NCUA). We define “derisking” as the practice of banks limiting certain services or ending their relationships with customers to, among other things, avoid perceived regulatory concerns about facilitating money laundering. We developed this definition in our prior work addressing account terminations and branch closures in the U.S. Southwest border region. To describe regulators’ BSA/AML supervisory expectations for banks that provide services to money transmitters and other MSBs and federal bank examiners’ views on banks’ challenges in complying with these requirements, we reviewed joint guidance issued by FinCEN and the federal banking regulators in April 2005 on banking MSBs and the Federal Financial Institutions Examination Council’s (FFIEC) BSA/AML examination manual, which federal banking regulators use to examine banks for BSA/AML compliance. We also interviewed the federal regulators named above. Further, we interviewed representatives of 16 banks, six credit unions, and relevant industry groups and trade associations. Because of our judgmental sampling, the views expressed by these groups may not be representative. To identify the universe of banks for interviews, we used data from FDIC’s Statistics on Depository Institutions database as of December 31, 2016. Next, we excluded banks that did not offer the product types relevant to our study, including credit card banks and banks that offer nontraditional accounts; multiple subsidiaries of large holding companies; and federal branches of foreign banks. We also excluded banks with insufficient information to determine the types of accounts offered. In addition, we excluded banks selected to participate in a web- based survey (we describe our survey methodology below). After these exclusions, our initial list consisted of 5,922 banks. Because the primary regulators (Federal Reserve, OCC, and FDIC) do not track which banks have money transmitter customers, we used a judgmental sample to randomly select banks to interview from each of the primary regulators based on asset size (small, medium, and large). For small and medium banks, we interviewed one bank of each size from each of the three regulators. For large banks, all were regulated by OCC, and we interviewed two of these banks. We defined banks’ asset-size categories as follows: (1) “small” consisted of banks with assets of less than $1 billion, (2) “medium” consisted of banks with assets of $1 billion to less than $10 billion, and (3) “large” consisted of banks with assets of $10 billion to less than $50 billion. Once we selected our sample, we contacted each bank to confirm that it had money transmitter or other types of MSB customers. If a bank did not have money transmitter or other MSB customers or declined to speak with us, we selected another bank in the same asset-size category. We initially selected nine banks to interview—three in each asset-size category—but one large bank declined to speak with us. Because there were no other large banks in our sample, we interviewed two large banks, for a total of eight small, medium, or large banks. We also jointly interviewed eight extra-large banks (with assets of $50 billion or more) in coordination with our other work on derisking. Because NCUA tracks which credit unions have money transmitter customers, we obtained data from NCUA on credit unions that served money transmitters as of April 2017 and stratified them according to small, medium, and large asset-size categories. We defined credit unions’ asset-size categories as follows: (1) “small” consisted of credit unions with assets of less than $100 million, (2) “medium” consisted of credit unions with assets of $100 million to $500 million, and (3) “large” consisted of credit unions with assets of more than $500 million. We chose three credit unions with the largest numbers of money transmitter customers and randomly selected one credit union from each asset-size category, for a total of six credit unions. From our initial selection, we emailed or called each of the six credit unions to ascertain if it had a money transmitter customer. If a credit union did not have a money transmitter customer or declined to speak with us, we selected another credit union in the same asset-size category. We then conducted two discussion groups per regulator with bank examiners from the Federal Reserve, OCC, FDIC, and NCUA to understand how they applied the FFIEC manual in assessing BSA/AML compliance controls of banks with money transmitter customers. To determine the composition of the discussion groups, we identified BSA/AML specialists or subject-matter experts from the district and regional offices of each federal banking regulator located in geographic areas with relatively large numbers of money transmitters. To do this, we first identified the states with the largest numbers of registered money transmitters by analyzing FinCEN money transmitter registration data from January 2015 through May 2017. We then requested rosters of staff designated as BSA/AML subject-matter experts and specialists from each regulator for each district or regional office in those states. We administered a questionnaire to the individuals on each roster asking about their experience with examining banks with money transmitter customers and other questions, such as years of experience in conducting bank examinations. We excluded from consideration BSA/AML subject-matter experts and specialists who either self-identified as supervisors or who had not examined a bank with a money transmitter customer in the past 3 years. We then randomized and selected BSA/AML subject-matter experts and specialists for participation in our discussion groups. Depending on scheduling and availability, the number of participants for each discussion group ranged from six to 14. Each session was digitally recorded and transcribed by an outside vendor, and we used the transcripts to summarize participants’ responses. An initial coder assigned a code that best summarized the statements from discussion group participants and provided an explanation of the types of statements that should be assigned to a particular code. A separate individual reviewed and verified the accuracy of the initial coding. The initial coder and reviewer discussed orally and in writing any disagreements about code assignments and documented consensus on the final analysis results. Discussion groups were intended to generate in-depth information about the reasons for the participants’ views on specific topics. The opinions expressed by the participants represent their points of view and may not represent the views of all BSA/AML subject-matter experts and specialists at the federal banking regulators. For purposes of this report, we used the following terms to describe the number of discussion groups in which an issue is mentioned: “some” to describe two to three groups out of the eight discussion groups, “many” to describe four to five discussion groups, and “most” to describe six to seven discussion groups. To examine challenges reported by federal bank examiners in assessing banks’ BSA/AML compliance controls around money transmitters, we asked examiners in our discussion groups to identify any challenges they encountered when assessing these compliance controls. We also reviewed examination guidance and procedures for assessing BSA/AML compliance controls around money transmitters. We assessed this information against federal internal control standards related to identifying risks and communicating information. We also reviewed bank examination and related documentation from the federal BSA/AML examinations of 56 selected banks and credit unions to gain additional context about BSA/AML examinations, including BSA/AML compliance violations—10 from FDIC, 12 from the Federal Reserve, 22 from OCC, and 12 from NCUA. For the documentation review, we selected a nongeneralizable sample of banks and credit unions based on asset-size categories and geographic location (based on each regulator’s field, district, or regional offices) from each federal banking regulator. For banks, we used the same asset-size categories described earlier for our interview selection process. We also included six banks that were issued final BSA/AML enforcement actions—two each from OCC, FDIC, and the Federal Reserve—for calendar years 2014 through 2016. For credit unions, we selected randomly from the same asset-size categories we used for selecting credit unions for interviews—along with geographic locations—and randomly selected four credit unions from each asset-size category, for a total of 12 credit unions. To obtain geographic representation, we ensured that each bank and credit union selected within each asset-size category also represented multiple geographic locations. For each of the 56 banks and credit unions, we requested and reviewed bank examination reports and related workpaper documentation for 2014, 2015, and 2016, including scoping and planning memorandums, bank- or examiner-prepared BSA/AML risk assessments, and conclusion memorandums or documents that summarized BSA examiner findings. For some banks, we also received banks’ BSA policies as part of the examination report and supplemental documentation package. To examine the extent to which banks are terminating or limiting money transmitters’ access to banking services and their reasons why, we administered a web-based survey to a nationally representative sample of banks in the United States for a total survey sample of 406 banks. We did not include credit unions in our sample. In the survey, we asked banks about terminations of money transmitter accounts and limitations on account offerings related to BSA/AML risk and the reasons for these decisions for the 3-year period from January 1, 2014, to December 31, 2016. We obtained a weighted survey response rate of 46.5 percent. While we designed the survey to be nationally representative of all banks in the United States, some results are statistically nongeneralizable because of the relatively low number of banks that reported having money transmitters as customers. For survey questions that are statistically nongeneralizable, we present only the number of responses to each survey question, and these results are not generalizable to the population of banks. Moreover, not all banks responded to every survey question or provided information for every year covered by our survey; therefore, we are not able to provide trend information from 2014 through 2016. We administered the survey from July 2017 to September 2017. To obtain information on the effects of bank account terminations on and limitations in the number of accounts with money transmitters, we interviewed a nongeneralizable sample of representatives from 11 money transmitters. To select the money transmitters, we obtained money transmitter licensure data from the Conference of State Banking Supervisors’ Nationwide Multistate Licensing System. Using the number of state licenses as a proxy for the size of the money transmitter, we developed five size categories and selected the top four money transmitters in the first stratum (40 or more licenses) along with one money transmitter in the second, third, and fourth strata (20–39, 10–19, and 2–9 licenses, respectively) and four money transmitters in the fifth stratum (one license). To evaluate how FinCEN and the federal banking regulators have assessed and responded to concerns about derisking of money transmitters, we reviewed agency documentation and guidance the agencies issued to banks related to derisking and MSBs, and we interviewed agency management. We also reviewed a prior GAO report that evaluated regulators’ response to derisking along the Southwest border and assessed actions regulators have taken to respond to a recommendation we made in that report. We utilized multiple data sources throughout our review. We assessed the reliability of FDIC’s Statistics on Depository Institutions database by reviewing related documentation and conducting electronic testing for missing data, outliers, or any obvious errors. Furthermore, we used NCUA data that track which credit unions bank money transmitters, the Nationwide Multistate Licensing System, and FinCEN’s MSB registration database to help select our nongeneralizable samples of credit unions and money transmitters to interview. We did not assess the data reliability of these sources because we used these data purely to inform our sampling population, and once we selected our samples, we took additional steps to confirm that the institutions we selected had MSB or money transmitter customers and were willing to speak to us. For FinCEN’s MSB registration database, as previously discussed, we used the data to help identify which states had the most money transmitters registered. In analyzing the data, we found a clear difference in the number of MSB registrations between the top five states (California, Texas, Michigan, Florida, and Illinois) with the most MSBs (ranging from close to 800 to almost 4,000 MSBs) and the remaining states (all with fewer than 500 MSBs). Because we used these data to help facilitate the identification of BSA/AML subject-matter experts and specialists who had experience examining banks with money transmitter customers, we did not need to confirm the exact number of MSBs registered. As a result, we did not assess the reliability of FinCEN’s registration database. We concluded that all applicable data were sufficiently reliable for the purposes of describing BSA/AML risks and compliance challenges and identifying banks to survey on account terminations and limitations. We conducted this performance audit from August 2016 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. From July 2017 through September 2017, we administered a web-based survey to a nationally representative sample of banks. In the survey, we asked banks about account terminations and restrictions (also referred to as limitations) for reasons associated with managing Bank Secrecy Act/anti-money laundering (BSA/AML) risk; whether banks are terminating or limiting accounts with money transmitters; and the reasons for these decisions. We collected information for the 3-year period from January 1, 2014, to December 31, 2016. Responses to selected questions from our survey that are directly applicable to the research objectives in this report are shown in tables 7–19 below. While we designed the survey to be nationally representative of all banks in the United States, results specific to money transmitters are statistically nongeneralizable because of the relatively low number of banks that reported having money transmitters as customers. Because these survey questions are statistically nongeneralizable, we present only the number of responses to each survey question, and the results are not generalizable to the population of banks. Moreover, not all banks responded to every survey question or provided information for every year covered by our survey; therefore, we are not able to provide trend information from 2014 through 2016. Our survey included multiple-choice and open-ended questions. For a more detailed discussion of our survey methodology, see appendix I. In addition to the contact named above, Stefanie Jonkman (Assistant Director), Kun-Fang Lee (Analyst-in-Charge), Carl Barden, Lilia Chaidez, Giselle Cubillos-Moraga, Joshua Garties, Toni Gillich, Shamiah Kerney, Jill Lacey, Patricia Moye, Aku Shika Pappoe, Jennifer Schwartz, Jena Y. Sinkfield, Tyler Spunaugle, Verginie Tarpinian, and Deme Yoo made key contributions to this report.", "summary": "The World Bank and others have reported that some money transmitters have been losing access to banking services. Money transmitters play an important role in the financial system, in part because they provide financial services to people less likely to use traditional banking services. GAO was asked to review the causes and potential effects of derisking by banks. This report examines, among other issues, (1) the extent to which banks are terminating or limiting services for money transmitters, (2) challenges in assessing banks' BSA/AML compliance related to money transmitters, and (3) regulators' actions to address derisking concerns. GAO reviewed bank examination reports and documents, held eight discussion groups with federal bank examiners, surveyed a nationally representative sample of 406 banks (excluding credit unions), and interviewed federal and bank officials, money transmitters, industry associations, and other stakeholders. From 2014 through 2016, 40 of 86 banks with money transmitter customers that responded to GAO's survey indicated they terminated at least one money transmitter account for money-laundering-related reasons. Money transmitters transfer money for their customers to recipients domestically or internationally. Common reasons given for terminating accounts included the customer not providing information needed to satisfy the banks' due diligence requirements under Bank Secrecy Act (BSA)/anti-money laundering (AML) regulations and that the cost of BSA/AML compliance made these customers unprofitable. However, banks also cited concerns that these customers drew heightened regulatory oversight; this may indicate “derisking,” the practice of banks limiting services or closing accounts with customers to avoid any perceived regulatory concerns about facilitating money laundering. Federal bank examiners in some of GAO's discussion groups identified challenges in assessing banks' compliance with due diligence requirements. In 2005, the Department of the Treasury's (Treasury) Financial Crimes Enforcement Network (FinCEN) and the federal banking regulators issued interagency interpretive guidance to clarify BSA/AML requirements and supervisory expectations for banks providing banking services to money transmitters. The guidance was incorporated in the Federal Financial Institutions Examination Council BSA/AML examination manual. However, examiners from some discussion groups said it was unclear how much due diligence is reasonable to expect banks to conduct for their money transmitter customers. For example, while the manual's examination guidance pertaining to money transmitters states that due diligence on higher-risk accounts can include reviewing the money transmitter's BSA/AML compliance program or conducting on-site visits, the related examination procedures do not clarify what these reviews or visits might entail. Unless federal banking regulators take steps to improve examiners' ability to evaluate banks' compliance with BSA/AML requirements as applied to money transmitter accounts, examiners may not be fully achieving examination objectives. In response to derisking concerns associated with money transmitters, FinCEN and the federal banking regulators have issued general guidance that discourages banks from terminating accounts with any particular customer type without evaluating individual customers' risks. In prior work, GAO noted that regulators had not fully evaluated how banks' regulatory concerns may be influencing decisions to derisk. GAO recommended that FinCEN and the federal banking regulators conduct a retrospective review of BSA regulations and their implementation, with a focus on how banks' regulatory concerns may affect their decisions to provide services. According to federal banking regulators and FinCEN, they and Treasury established an interagency working group in early 2018 that they believe will address the recommendation. The working group has taken important steps toward improving the efficiency and effectiveness of BSA/AML supervision, including issuing an interagency statement intended to improve the transparency of the risk-focused approach examiners use to plan and conduct BSA examinations. However, the working group has not yet evaluated the full range of factors that may influence banks to derisk. GAO is making a total of four recommendations to the federal banking regulators that each regulator improve examiners' ability to evaluate banks' BSA/AML compliance as applied to money transmitter accounts. The federal banking regulators agreed with GAO's recommendations. GAO also reiterates its recommendation in GAO-18-263 that FinCEN and the federal banking regulators conduct a retrospective review of BSA regulations and implementation.", "document_type": "gao"}
{"report": "Created as part of the Employee Retirement Income Security Act of 1974, as amended (ERISA), traditional IRAs provide tax advantages to help individuals—including small business owners, independent contractors, and other workers not covered by employer-sponsored retirement plans— save for retirement. Employees who have employer-sponsored retirement plans, such as a 401(k), can also roll over these assets into an IRA when they retire or change jobs. Since the enactment of ERISA, different types of IRAs with different features for individuals and small businesses have been created. The two IRA types with federal income tax benefits for individuals are traditional IRAs (which allow eligible individuals to make tax-deductible contributions and accumulate tax-deferred investment earnings) and Roth IRAs (which allow eligible individuals to make after-tax contributions and accumulate investment earnings tax free). IRA owners are able to invest their IRA savings in a wide variety of asset types. IRA owners generally make tax-favored contributions to their accounts to purchase assets from investment options offered through banks or other IRS-qualified firms acting as custodians of the IRA assets. Most IRA custodians limit holdings in IRA accounts to firm- approved stocks, bonds, mutual funds, and CDs. Some custodians offer so-called “self-directed IRAs” that allow investments in a broader set of unconventional assets—such as real estate, certain precious metals, private equity, and virtual currency—than is permitted by most IRA custodians. As we previously reported, custodial agreements for these accounts often require IRA account owners to be responsible for directing their investments, and to oversee the selection, management, monitoring, and retention of all investments in the account. The account owners bear the consequences of any mistakes made in managing their accounts, such as being noncompliant with IRA rules. Through our prior work, we identified the following four areas where complex rules are likely to apply to IRA owners investing in unconventional assets: Barred investments. Investments in life insurance contracts and collectibles, such as artwork and antiques, are prohibited. Although precious metals are generally prohibited collectibles, certain types of coins and bullion are permitted provided that they meet specific purity and custody requirements. Prohibited transactions. IRA owners are not permitted to engage in prohibited transactions that personally benefit the owner or other disqualified persons in a way other than as a vehicle to save for retirement. Examples of such prohibited transactions include IRA owners selling their own property to an IRA, or taking a salary from an IRA-funded business. IRA owners who believe that an otherwise prohibited transaction should be permitted, may apply to the Department of Labor (DOL) to request an exemption for a specific transaction. Unrelated business income. Earnings and profits made in tax- deferred savings vehicles like IRAs generally are reinvested in the account without generating current federal tax liability, but investments in certain unconventional assets can generate ongoing tax liability for IRA owners. Any IRA that earns $1,000 or more of gross income from an unrelated business must file Form 990-T Exempt Organization Business Income Tax Return with IRS and pay related taxes. Fair market value (FMV). When IRA owners invest in less conventional and nonpublicly traded assets, custodians may find it challenging to properly report the FMV of those assets. Starting with tax year 2015, IRS began requiring IRA custodians to report selected information on unconventional assets in their clients’ accounts. For some hard-to-value unconventional assets, IRA owners may need to supply custodians with independent appraisals or other evidence to substantiate an asset’s current FMV. Failure to abide by the rules governing IRAs with unconventional assets can have significant consequences for IRA owners. For example, if an IRA owner engages in a prohibited transaction that has not been exempted by DOL, the IRA will lose its tax-favored status, and the account is treated as distributing all of its assets to the owner at the FMV on the first day of the year in which the prohibited transaction occurred. Noncompliance with IRA rules—if not detected—can also lead to millions of dollars in uncollected tax revenue for the government. Individuals who invest in certain unconventional assets using Roth IRAs can avoid taxation on investment gains. For example, founders of companies (or key initial investors) who use IRAs to invest in nonpublicly traded shares of their newly formed companies can realize many millions of dollars in tax-favored gains on their investment if the company is successful. IRS is responsible for enforcing IRA tax laws, including rules that apply when IRA owners invest in unconventional assets. Within IRS, four business operating divisions have responsibilities for enforcing compliance with IRA rules. Table 1 provides an overview of each division’s IRA enforcement activities. Third-party reporting by IRA custodians provides information that taxpayers can use in preparing their tax returns and that IRS can use to identify noncompliant taxpayers and help close the tax gap. In 2015, IRS began requiring custodians to report new information to help identify IRAs with hard-to-value unconventional assets. IRS Form 5498 IRA Contribution Information has a new box 15a for custodians to report the portion of the IRA FMV attributable to nonmarket assets as well as a box 15b with codes describing the type of nonmarket assets. Custodians are to report similar information on IRS Form 1099-R identifying distributions of IRA assets that do not have a readily available FMV. The first article in the Taxpayer Bill of Rights is the right to be informed which means that taxpayers have the right to know what they need to do to comply with tax laws. IRS’s Publication 1, Your Rights as a Taxpayer, further states that taxpayers are entitled to clear explanations of the laws and IRS procedures in all forms, instructions, publications, notices, and correspondence. To help taxpayers and their advisors better understand tax rules, such as those governing IRAs with unconventional assets, IRS produces several types of resources. Taxpayers (or their advisers and paid tax preparers) with complicated returns or transactions may require detailed and technical resources, such as guidance published in a weekly IRS publication called the Internal Revenue Bulletin (IRB). Tax regulations— issued by the Department of the Treasury (Treasury)—are published in the IRB together with technical IRS guidance such as revenue rulings and revenue procedures. IRS has stated that only guidance published in the IRB contains IRS’s authoritative interpretation of the law. IRS also produces resources that are less technical and intended to be more easily understood by most taxpayers. IRS issues hundreds of publications on a variety of tax topics, and many are updated annually. IRS also produces a variety of information on its website (IRS.gov) such as online tools, instructions, and FAQs. IRS’s Publication 590-A, Contributions to Individual Retirement Arrangements (IRAs), and Publication 590-B, Distributions from Individual Retirement Arrangements (IRAs), serve as a general IRA handbook for IRA owners and a logical starting point for all IRA owners with tax questions, including those with unconventional assets. At more than 120 pages combined, Publications 590-A and 590-B comprise one of IRS’s longest publications on retirement related topics. Publications 590-A and 590-B provide some limited information on the four compliance topics that we identified through prior work as likely to affect IRA owners with unconventional assets. However, the two-part publication lacks additional information that IRA owners with unconventional assets need to comply. Publications 590-A and 590-B recommend that taxpayers research IRS’s website (IRS.gov) for additional information. We found some additional information on IRS’s website about three of the four compliance topics. This information was typically in the form of FAQs in a section of IRS’s website about retirement plans (https://www.irs.gov/retirement-plans). Table 2 summarizes: what information for IRA owners with unconventional assets can be found in Publications 590-A and 590-B; what other IRS sources provide relevant information; and what information was not readily available on the IRS website for the four compliance areas likely to affect IRA owners with unconventional assets. Appendix II describes in more detail the information available and the information lacking in Publications 590-A and 590-B and other IRS sources. Given the complexity of the four compliance topics we identified as well as the relatively few numbers of taxpayers affected and the already large publication size, it may not be feasible to provide complete information on these topics within Publications 590-A and 590-B. IRS publications (like 590-A and 590-B) are intended to explain the law in plain language for taxpayers and their advisors. They generally summarize and translate into layperson’s terms more complex and technical information from authoritative sources like the Internal Revenue Code and more authoritative guidance like tax regulations, revenue procedures, and revenue rulings. IRS analysis indicates that perhaps only about 2 percent of IRAs have invested in hard-to-value unconventional assets. However, even small numbers of taxpayers with particular circumstances have the right to know what they need to do to comply with tax laws. IRA owners with unconventional assets who turn to Publications 590-A and 590-B are unlikely to fully understand how certain IRA investment decisions can increase their risks for noncompliance. Misunderstanding the rules governing IRAs could result in increased tax liability for taxpayers making unintentional errors and jeopardize their retirement savings. Given the serious consequences that could result for a taxpayer found to be noncompliant, IRS’s current publications are not clearly providing information for IRA owners with unconventional assets. Adding information to Publications 590-A and 590-B would be one solution that IRS could explore, but we recognize that it may not be practical for IRS to add substantially more information to Publications 590-A and 590-B for a relatively small percentage of IRA owners. Alternatives to adding more pages to Publications 590-A and 590-B could include directing readers with questions about rules affecting unconventional IRA assets to other IRS resources, such as IRS web pages or tax regulations that contain more technical and specialized information. As shown in table 2 above, we found some additional information on IRS web pages that would be helpful to IRA owners with unconventional assets. Adding language in Publication 590-A or 590-B directing taxpayers to specific web page URL addresses for additional information could help taxpayers more easily locate this information. For more technical or specialized information, IRS could direct readers of Publications 590-A and 590-B to the relevant sections of the Internal Revenue Code and related tax regulations. This additional information could help IRA owners better understand and navigate the potential compliance challenges associated with certain types of unconventional assets. In October 2017, the Deputy Commissioner for Service and Enforcement commissioned a cross-divisional team comprised of representatives from all four IRS operating divisions to identify, assess, and mitigate risks of IRA noncompliance. In its February 2018 interim presentation, the IRS cross-divisional team categorized potential noncompliance risks over an IRA life cycle into two mitigation strategies, which are summarized below. 1. Noncompliance risks for most contribution and distribution IRA rules can be mitigated systemically through automated enforcement. For example, IRS can detect excess IRA contribution deductions and unreported IRA distributions by matching information from taxpayer returns with information reported by custodians. For the large population of IRA owners investing in conventional assets held by custodians, IRS relies on automated enforcement. 2. Noncompliance risks associated with the small population of IRAs with hard-to-value unconventional assets or under direct control of the IRA owner are generally mitigated through case-by-case audits. For example, noncompliance with the complex rules governing prohibited transactions and unrelated business income is generally not reflected on individual tax returns. Some custodians rely on IRA owners to provide asset value information and may not have complete and accurate data to report to IRS. Undervaluing IRA assets hampers automated enforcement, for example, to detect excess contributions and taxable distributions. Noncompliance involving IRAs with unconventional assets is generally detected through labor-intensive audits of individual taxpayers. IRS’s SB/SE division uses field audits to pursue complex individual tax return cases, including those that could involve IRAs with unconventional assets. In February 2018, an IRS cross-divisional team that studied the risks of IRA noncompliance reported that, from fiscal years 2012 to 2016, IRS audited about 26,000 tax returns with IRA issues. IRS officials provided us examples of SB/SE job aides and training materials designed to help examiners recognize different types of noncompliance associated with IRAs invested in unconventional assets. For example, the job aides provide instructions on prohibited transactions, barred collectibles, and FMV issues involving IRAs. When interviewing taxpayers, examiners are instructed to ask a series of questions covering subjects such as: what kind of advice the taxpayer received from promoters or custodians of self-directed IRAs, whether the taxpayer had direct involvement in purchasing unconventional assets through a control feature known as “checkbook access,” whether the taxpayer has a limited liability company (LLC) tied to the how the taxpayer determined the FMV of unconventional assets. IRS officials told us that enforcing rules associated with IRAs investing in unconventional assets can be particularly challenging for investments involving LLCs or special partnership arrangements. An IRA owner may establish an LLC that is owned by the IRA. Once the LLC is set up, a business checking account is linked to the IRA funds and the account owner is named the manager of the LLC with control over the checkbook. This allows IRA owners to purchase assets directly from investment sponsors without having to wait for custodians to execute a purchase or sale. The LLC may be used to invest in businesses that could generate unrelated business income. According to IRS officials, prohibited transactions may also be more likely to occur when custodians allow “checkbook” access to IRAs, in part because the marketing of this IRA structure is appealing to individuals who want less oversight of their IRA transactions and are more likely to intentionally engage in self-dealing transactions. IRS examination officials told us that the 3-year statute of limitations for assessing taxes owed remains an obstacle in pursuing noncompliance that may span the many years of an IRA investment. For example, abuses involving prohibited transactions frequently are not reflected on any filed tax return and may be difficult to detect within the general 3-year statute of limitations period. IRS agreed with our October 2014 recommendation for the Commissioner to work in consultation with the Department of the Treasury (Treasury) on a legislative proposal to expand the statute of limitations on IRA noncompliance. IRS said Treasury is aware of IRS’ support for changing the limitation period for IRA noncompliance. Treasury reviews and presents the administration’s tax proposals and has not released a legislative proposal as of October 2019. With electronically compiled data for tax year 2016 filed in 2017, IRS was positioned for the first time to quantify the number of IRAs with specified types of hard-to-value assets. IRS officials said that even with the new custodian reporting, the broad IRA asset type data alone may be inadequate for improving audit selection criteria and identifying potentially abusive IRAs in a timely manner. In February 2018, using the newly available data, an IRS cross-divisional team identified that about 2 million IRAs included one or more types of hard-to-value assets for tax year 2016. However, custodians reported an FMV dollar amount for hard-to- value assets for only 1.6 million of those IRAs, as shown in table 3. The combined FMV was approximately $137 billion. As shown in table 3, about 400,000 (about 20 percent) of the Form 5498s reporting that the IRA held investments in one or more of the specified unconventional categories were missing the 2016 FMV dollar amount for those assets. The cross-divisional team identified that undervaluation risk affects custodian reporting. IRS officials said that the team did not review the custodian reporting patterns as part of its initial analysis of the 2016 Form 5498 data. Forthcoming tax regulations on IRAs may help to improve custodian reporting of FMVs on Form 5498. IRS officials told us that the new IRA regulations would address FMV for certain categories of hard-to-value unconventional assets. IRS officials also told us that it would be premature to publish new guidance for IRA owners and custodians on the FMV of unconventional assets until the new regulations are issued. The tax year 2016 Form 5498 information indicated about 141,000 IRAs invested in LLCs—an asset type which IRS has determined presents greater noncompliance risk. Prior to the newly available asset type data, SB/SE conducted an interim Compliance Initiative Project (CIP) using external state government information to identify businesses, including LLCs and partnerships, owned by IRAs as a way to select IRA owners for audit. Completed in October 2019, the interim compliance research revealed that audits detecting prohibited transactions can result in substantial tax adjustments. In September 2018, SB/SE approved a new CIP using the asset type data from Form 5498s for tax year 2017 to select a sample of traditional and Roth IRAs that had an ownership in an LLC or real estate. The latest compliance research field work began in February 2019 and is to be completed in January 2021. IRS officials told us they plan to use this research in combination with the interim research results to improve criteria for selecting tax returns with IRAs at greater risk of noncompliance for audit. To detect abusive transactions, IRS can require taxpayers to self-report certain transactions that have been used by other taxpayers to avoid taxes. Transactions become “reportable” (meaning a taxpayer must report it to IRS) when IRS designates them as a “listed transaction” or “transactions of interest.” Listed Transaction. A listed transaction is reportable when it is the same or substantially similar to one of the types of transactions that IRS has determined to be an avoidance transaction. In 2004, IRS determined that Roth IRA “stuffing” is an abusive tax avoidance transaction that taxpayers must report to IRS as a listed transaction. “Stuffing” involves shifting value through transactions that disguise Roth IRA contributions exceeding annual IRA limits, such as selling receivables at less than FMV to a Roth IRA, or other transactions between a closely-held business in which the Roth IRA invests and another closely-held business of the Roth IRA owner. Transaction of Interest. A transaction of interest is one that IRS and Treasury believe to have the potential for tax avoidance or evasion, but which lacks enough information for IRS and Treasury to determine whether the transaction should be identified as a tax avoidance transaction. As of December 2019, IRS has not identified or classified any IRA asset types or investment transactions as reportable transactions of interest. Taxpayers are required to disclose all types of reportable transactions on Form 8886, Reportable Transaction Disclosure Statement. Similarly, advisers helping taxpayers conduct reportable transactions are required to file Form 8918, Material Advisor Disclosure Statement. Results from the ongoing IRS compliance research may yield insights about existing and emerging abusive schemes involving IRAs. This information could be useful for evaluating the feasibility of requiring greater disclosure by IRA owners and their custodians and advisors. For example, IRS could consider requiring reporting of known abusive IRA arrangements and prohibited transactions as listed transactions. Also, IRS could explore disclosure of high-risk IRA asset types susceptible to gross valuation misstatements, such as LLCs, as transactions of interest. We recently found that IRS’s Research, Analysis and Statistics office had developed the capability to analyze the narrative fields of tax forms. Additional disclosure of potentially abusive IRA transactions coupled with greater use of tax form’s narrative fields may help IRS to select IRA owner tax returns for more detailed review. The cases identified by such detailed review would help IRS better allocate limited audit resources. Responsibility for addressing IRA noncompliance detected through case- by-case audits is fragmented among multiple IRS organizational units. This fragmentation creates challenges for IRS examiners from different units that may need to share expertise and collaborate on enforcement of complex rules applicable to IRAs that invest in unconventional assets. In February 2018, the IRS cross-divisional team concluded that no one IRS operating division alone can effectively identify and penalize IRA noncompliance regarding unrelated business income and undervaluation of unconventional assets. Unrelated business income. SB/SE and TE/GE officials told us that detecting unrelated business income unreported by an IRA can also require the involvement of multiple IRS divisions. IRS responsibility and expertise in detecting noncompliance with the rules for unrelated business income resides in IRS’s TE/GE division. TE/GE is responsible for enforcing the unrelated business income taxation rules across tax-exempt organizations. Its Exempt Organizations group audits Form 990-T filed by tax- exempt charities and its examiners are required to check if tax exempt charities have reported unrelated business income. Examiners in TE/GE’s Employee Plans group have been trained on how to determine if a tax-exempt employee retirement plan has engaged in activities that constitute unrelated trade or business. SB/SE has primary responsibility for auditing individuals owning IRAs, and its examiners are to verify that all returns within the taxpayer’s sphere of influence are filed. IRS officials told us that when SB/SE examiners discover potential unrelated business income issues when reviewing an individual taxpayer’s IRAs, those examiners can seek assistance from TE/GE examiners via an internal Specialist Referral System used to refer cases to other divisions. Although IRS officials described to us how SB/SE examiners, at their own initiative, can seek out expertise on unrelated business income, the topic is not addressed in SB/SE examiner training materials and job aids on auditing IRAs with unconventional assets. SB/SE officials provided us training slides used to teach examiners how to recognize excess contributions, prohibited transactions, barred collectibles, and valuation issues involving IRAs. While the slides instruct examiners to contact a Senior Program Analyst or Counsel for assistance with complicated issues or cases, there is no information educating SB/SE examiners about unrelated business income or informing examiners that specialized knowledge about this topic resides in the TE/GE division. Without resources, such as training materials or job aides, that provide such information, SB/SE examiners carrying out the ongoing compliance initiative project are not positioned to surface unrelated business income tax issues for referral to TE/GE. Given that IRS plans to use those research results to refine its audit selection criteria, IRS is missing an opportunity to learn more about IRA noncompliance with unrelated business income taxation. Undervaluation of unconventional assets. In February 2018, the cross-divisional IRA team cited undervaluation of unconventional assets as another compliance risk that involves the expertise and enforcement responses from multiple IRS units. If SB/SE examiners determine in auditing an IRA owner that the IRA custodian had inaccurately reported IRA asset values, other IRS divisions can take action against the custodian. LB&I can penalize a large financial institution custodian, although the cross-divisional IRA team reported the $50 penalty for filing an incorrect Form 5498 poses little deterrent effect. For the approximately 75 non-bank IRA trustees approved by IRS, TE/GE can revoke a non-bank’s trustee status for violating any fiduciary, accounting, or financial requirements. The cross-divisional IRA team explored an approach for joint examination to more effectively identify and penalize noncompliance associated with prohibited transactions, unreported unrelated business income, and undervaluation of IRA assets. Based on knowledge from prior examinations, the team identified a small subset of non-bank trustees publicly marketing alternative investments that held IRAs more than $5 million in reported FMV as of tax year 2016. As of February 2018, the team reported that it had been premature for the separate divisions to commit examination resources. As of October 2019, IRS officials said they plan to reconvene the cross-divisional IRA team after the ongoing SB/SE compliance initiative project is complete in 2021. IRS officials said the plan is for the team to use the compliance research results to refine audit selection. Also, the team could continue work on establishing a joint examination approach for IRA noncompliance associated with hard-to-value unconventional assets. IRA owners that invest in unconventional assets—such as real estate, certain precious metals, virtual currency, or private equity—assume greater responsibility for navigating complex rules that govern tax-favored retirement investments. To understand these rules, taxpayers are likely to consult IRS Publications 590-A and 590-B. While this two-part publication provides some information on compliance issues likely to affect IRA owners with unconventional assets, the information in the publication as well as on IRS web pages is limited. By assessing options for making such information clearer, IRS could better inform taxpayers and help them comply. This is particularly important because misunderstanding the rules governing IRAs can result in increased tax liability for these taxpayers and jeopardize their retirement savings. Noncompliance associated with nonpublicly traded IRA assets has been difficult for IRS to detect and time consuming to pursue. In contrast to automated enforcement for IRAs with conventional investments, noncompliance involving IRAs with unconventional assets is generally detected on a case-by-case basis through labor-intensive audits of individual taxpayers. In recent years, IRS has begun collect information from IRA custodians that IRS can use to quantify the dollar amounts of specified types of hard-to-value assets held by IRAs. However, the broad IRA asset type data alone may not be sufficient for audit selection and identifying potentially abusive IRAs in a timely manner. When IRS lacks sufficient data to detect abusive transactions, IRS can require taxpayers to self-report certain transactions that have been used by other taxpayers to avoid taxes. Additional disclosure of certain IRA transactions coupled with mining the narrative fields of tax forms could help IRS to efficiently identify potentially abusive IRA activity and better allocate limited audit resources. Fragmented responsibility among IRS operating divisions creates additional challenges for IRA enforcement. The division responsible for tax-exempt entities trains its examiners on how to determine if an employee retirement plan has unrelated business activities subject to taxation. Yet, examiners in the division that audits complex individual tax returns, including those involving IRAs, do not receive similar training. Training for those examiners on unrelated business income tax issues, and how examiners can refer those cases to other divisions for assistance, could help improve collaboration on IRA enforcement. We are making the following three recommendations to IRS: The Commissioner of Internal Revenue should assess options for updating Publications 590-A and 590-B to either include more information or direct taxpayers to other resources for IRA owners with investments in unconventional assets. Such information could include: storage requirements for IRA investments in certain precious metals; valuation methods for hard-to-value IRA assets; the Department of Labor’s process for granting exemptions to IRA prohibited transactions rules; and IRA investments with the potential to create unrelated business income tax liabilities. (Recommendation 1) The Commissioner of Internal Revenue, building on forthcoming compliance research using new IRA asset data, should evaluate the feasibility of requiring disclosure for high-risk IRA asset types associated with abusive schemes as transactions of interest. (Recommendation 2) The Commissioner of Internal Revenue should develop resources (such as training materials or job aids) for Small Business/Self-Employed examiners conducting IRA owner audits that explain how IRAs with unconventional assets can generate unrelated business income tax liability, and how examiners can refer cases to unrelated business income experts in IRS for assistance. (Recommendation 3) We provided a draft of this report to the Treasury and IRS for review and comment. In its comments, reproduced in appendix III, IRS generally agreed with our recommendations. For recommendation 1, IRS said it will review its educational publications and web pages for appropriate updates within the scope of the tax code. For recommendation 2, IRS said that it will determine whether there are abusive schemes associated with certain IRA asset types, and if the data indicate such a correlation, it will evaluate the feasibility of requiring disclosure of such arrangements as transactions of interest. For recommendation 3, IRS said it will review and update resources for examiners conducting IRA owner audits, including guidance on how to address unrelated business income tax (UBIT). It will incorporate guidance for agents on how to refer such cases to UBIT experts when assistance is needed. IRS also said that it will renew its efforts at ensuring collaboration with relevant subject matter experts. IRS in consultation with Treasury also provided technical comments which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of the Treasury, the Commissioner of Internal Revenue, the Secretary of Labor, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact James R. McTigue, Jr. at (202) 512-9110 or Charles A. Jeszeck at (202) 512-7215. You may also reach us by email at jeszeckc@gao.gov or mctiguej@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff making key contributions to this report are listed in appendix IV. This appendix describes: what information for individual retirement account (IRA) owners with unconventional assets can be found in Publications 590-A and 590-B, what other Internal Revenue Service (IRS) sources provide relevant information, and what information was not readily available on the IRS website for the four compliance areas we identified through prior work as likely to affect IRA owners with unconventional assets. The information below does not include the Internal Revenue Code or detailed and technical resources published in the weekly Internal Revenue Bulletin, such as tax regulations, revenue rulings, and revenue procedures. Publications 590-A and 590-B explain what types of IRA investments are barred, such as collectibles, but the publication does not have additional information that could be useful to IRA owners with allowable investments in coins and bullion. The publications define collectibles as including artworks, rugs, antiques, metals, gems, stamps, coins, alcoholic beverages, and certain other tangible personal property. The publications explain that if a traditional IRA invests in collectibles, the amount invested is considered distributed, and that the IRA owner could be subject to an additional tax on early distributions. Publications 590-A and 590-B further explain that an exception exists for IRA investments in certain types of coins and bullion. However, the two-part publication does not indicate that certain types of bullion must be stored by a bank or an IRS-approved non-bank trustee. The two-part publication also does not mention that IRA investments in life insurance contracts are not permitted. Two IRS web pages listing frequently asked questions (FAQs) about retirement plans contain additional information about bullion storage requirements and IRA investments in life insurance contracts. Both web pages state that investing IRA funds in life insurance contracts and collectibles are prohibited, and they also note the exception for certain precious metals. One of the web pages further explains that allowable bullion must be stored with a bank or an IRS-approved non-bank trustee. Publications 590-A and 590-B define prohibited transactions in general terms, list examples, and explain the consequences of engaging in a prohibited transaction. The two-part publication also cautions that the risk of engaging in a prohibited transaction in connection with an IRA account may be increased when an IRA owner invests in nonpublicly traded assets or assets that an IRA owner directly controls. However, the publication does not provide any information about applying for an exception to the prohibited transaction rules. We found some limited information about exemptions to the prohibited transaction rules on an IRS web page entitled “Retirement Plan Investments FAQs.” The web page explains that the Department of Labor (DOL) has granted class exemptions for certain types of investments under specific conditions, and that a plan sponsor may apply to DOL to obtain an administrative exemption for a particular proposed transaction that would otherwise be prohibited. However, the web page does not provide any links to DOL information such as a DOL publication that explains the prohibited transactions exemption process. In February 2018, IRS updated Publications 590-A and 590-B to include information about IRAs with unrelated business income. Publications 590- A and 590-B now explain that an IRA is subject to tax on unrelated business income if the IRA carries on an unrelated trade or business. Publications 590-A and 590-B state that the IRA trustee is required to file a Form 990-T if an IRA has $1,000 or more of unrelated trade or business gross income. For more information, Publications 590-A and 590-B direct taxpayers to consult Publication 598, Tax on Unrelated Business Income of Exempt Organizations. Publication 598 lists IRAs as one of many exempt entities subject to taxes on unrelated business income, and the requirement to file Form 990-T for gross income of $1,000 or more. Publication 598 describes dozens of activities by tax-exempt organizations that would be considered an unrelated business; but the publication does not include any examples specific to IRA investments that could also be considered unrelated business activities and subject to taxes. Our search did not find additional information on IRS.gov relating to IRAs and unrelated business income taxes. Publications 590-A and 590-B do not provide guidance about how to accurately determine the FMV of hard-to-value unconventional assets. IRS requires custodians to report (on Form 5498) an IRA’s FMV at year’s end as well as some additional information for IRAs with unconventional assets. The instructions for completing Form 5498 explain that IRA custodians are responsible for ensuring that “all IRA assets (including those not traded on established markets or not having a readily determinable market value) are valued annually at their FMV.” However, neither the form’s instructions nor Publications 590-A and 590-B provide guidance or tips on how to determine the FMV of non-publicly traded or other hard-to-value assets. As we previously reported, some unconventional assets may require a third-party appraisal to determine their FMV. One IRS web page titled, “Valuation of Plan Assets at Fair Market Value,” provides some additional FMV information but it is intended more for the valuing assets in employer-provided retirement benefits like traditional pensions and 401(k) plans. The web page states that an accurate assessment of the FMV of retirement plan assets is essential for complying with Internal Revenue Code requirements and avoiding prohibited transactions. The web page also states that for defined contribution plans like a 401(k) plan, investments must be valued at least once per year in accordance with a consistent and uniform method. For traditional pensions (defined benefit plans), tax regulations define FMV for purposes of valuing plan assets as “the price at which the property would change hands between a willing buyer and a willing seller, neither being under any compulsion to buy or sell and both having reasonable knowledge of relevant facts.” The Department of the Treasury (Treasury) plans to issue regulations on IRAs. IRS officials told us that these new regulations would address FMV for certain categories of hard-to-value unconventional assets. IRS officials also told us that it would be premature to publish new guidance for IRA owners and custodians on the FMV of unconventional assets until the new regulations are issued. In their October 2019 update of planned guidance projects, Treasury’s Office of Tax Policy and IRS listed planned IRA regulations. In addition to the contacts named above, MaryLynn Sergent and David Lehrer (Assistant Directors), Ted Burik, Susan Chin, Steven Flint, Emily Gruenwald, Mark Kehoe, Jungjin Park, and David Reed made key contributions to this report. James Bennett, Amy Bowser, Jacqueline Chapin, Edward J. Nannenhorn, Andrew J. Stephens, Walter Vance, and Adam Wendel also provided support.", "summary": "Unconventional IRA investments—such as real estate, certain precious metals, private equity, and virtual currency—can introduce risks to account owners who assume greater responsibility for navigating the complex rules that govern tax-favored retirement savings. IRS enforces tax rules relating to IRAs and can assess additional taxes. GAO was asked to examine the challenges associated with enforcing rules governing IRAs invested in unconventional assets. This report examines (1) the extent to which IRS offers guidance to help taxpayers understand the rules governing unconventional IRA assets; and (2) the challenges IRS faces in enforcing those rules. GAO identified and analyzed IRS information to help taxpayers understand four compliance areas. GAO reviewed IRS analysis of nonmarket IRA assets reported by IRA custodians, and IRS audit procedures and training materials; and interviewed relevant IRS officials to identify enforcement challenges. The Internal Revenue Service's (IRS) Publications 590-A and 590-B serve as a general handbook for millions of taxpayers with individual retirement accounts (IRA). However, the two-part publication provides limited information for IRA owners with unconventional assets surrounding complex tax rules in four compliance areas: (1) barred investments, (2) prohibited transactions, (3) unrelated business income, and (4) fair market value. GAO found other limited information about these topics on IRS's website. With only about 2 percent of IRAs invested in unconventional assets, adding more pages to Publications 590-A and 590-B may not be practical. By assessing options for informing IRA owners investing in unconventional assets, such as directing them to web pages with specialized information and technical regulations, IRS could better help them comply. Noncompliance involving unconventional IRA assets is difficult to detect and time consuming for IRS to pursue. Whereas IRS relies on automated enforcement for IRAs invested in conventional assets held by custodians and trustees, enforcement for IRAs invested in unconventional assets or under IRA owner control requires labor-intensive audits of individual taxpayers. Using newly compiled information, IRS identified about 2 million IRAs that held certain types of hard-to-value assets as of 2016; however, about 20 percent of the forms were missing fair market value amounts for these assets (see fig.). IRS officials said this type of reporting alone may be inadequate for audit selection and identifying potentially abusive IRAs. When IRS lacks sufficient data to detect abusive transactions, IRS can require taxpayers to self-report certain transactions that have been used by other taxpayers to avoid taxes. Additional taxpayer or custodian disclosure of potentially abusive IRA transactions coupled with IRS analysis of reported details may help IRS to select IRA owner tax returns to audit. Fragmented responsibility among IRS divisions creates challenges for examiners who need to share expertise and collaborate on IRA enforcement. The division responsible for tax-exempt entities trains its examiners on how to determine if an employee retirement plan has engaged in business activities subject to taxation. However, examiners in the division that audits complex individual tax returns, including those involving IRAs, do not receive such training. Training for those examiners could help improve collaboration on IRA enforcement. GAO is recommending that IRS (1) assess options for updating its IRA publications to provide more information for taxpayers with unconventional assets, (2) evaluate the feasibility of requiring disclosure for high-risk IRA asset types associated with abusive tax schemes, and (3) develop auditor resources (such as training materials or job aids) that explain how IRAs with unconventional assets can generate unrelated business income tax. IRS generally agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "The Medicaid Drug Rebate Program was established through the Omnibus Budget Reconciliation Act of 1990 and requires drug manufacturers to pay rebates to states on outpatient drugs as a condition of having their drugs covered by Medicaid. The 340B Program, named for the statutory provision authorizing it in the Public Health Service Act, was created in 1992 following the enactment of the Medicaid Drug Rebate Program and allows covered entities to purchase outpatient drugs at discounted prices. HRSA and CMS both have roles in overseeing compliance with the prohibition on duplicate discounts. The Medicaid Drug Rebate Program helps to offset the federal and state costs of most outpatient prescription drugs dispensed to Medicaid beneficiaries. Under the rebate program, drug manufacturers pay rebates to states as a condition for the federal contribution to Medicaid spending for the manufacturers’ outpatient drugs. State Medicaid programs generally must cover all of the drugs of manufacturers that participate in the rebate program. Originally, rebates were available only for drugs paid for by the state on a FFS basis, but the Patient Protection and Affordable Care Act extended the program to outpatient drugs paid for under Medicaid managed care; there are more Medicaid enrollees, prescriptions, and spending for drugs under managed care than FFS. The rebates received for both FFS and managed care are shared by the federal government and states. The amount of Medicaid rebates for a drug is based on a statutory formula. Using that formula CMS calculates a unit rebate amount for each drug and provides that amount to states so they can determine the amount of rebates to request. Every quarter, each state multiplies the number of units of each drug it either paid for on a FFS basis or provided through its managed care plans by the CMS-provided unit rebate amount. For drugs provided under FFS, the state calculates the number of units based on drug claims it reimbursed, while states use drug utilization data provided by managed care plans to determine the number of units of each drug that were provided by the plans to Medicaid beneficiaries. Each state then sends rebate requests to each manufacturer reflecting the total quarterly amount of rebates owed for each of the manufacturer’s drugs. States are to exclude claims for 340B drugs from their rebate requests. Participation in the 340B Program is voluntary for both covered entities and drug manufacturers, but there are strong incentives for both to do so. Covered entities can realize substantial savings through the program’s price discounts. In addition, covered entities can generate revenue to the extent that they can purchase 340B drugs for eligible patients whose insurance reimbursement exceeds the price paid. Incentives for participation by drug manufacturers are strong because they must participate in the 340B Program to receive Medicaid reimbursement for their drugs. Covered entities generally become eligible for the 340B Program by qualifying as certain federal grantees or as one of six specified types of hospitals. Eligible federal grantees include federally qualified health centers, which provide comprehensive community-based primary and preventive care services to medically underserved populations, as well as certain other federal grantees, such as family planning clinics and Ryan White HIV/AIDS program grantees, among others. Eligible hospitals include critical access hospitals—small, rural hospitals with no more than 25 inpatient beds; disproportionate share hospitals—general acute care hospitals that serve a disproportionate number of low-income patients; and four other types of hospitals. To participate in the 340B Program, covered entities must register with HRSA and annually recertify their continuing eligibility. Once their eligibility is approved by HRSA, covered entities can begin purchasing drugs from manufacturers at the 340B discounted prices. Covered entities may provide drugs, including 340B drugs, to patients through one or more dispensing methods. Specifically, covered entities may dispense these drugs through pharmacies—either through in-house pharmacies they own; through the use of contract pharmacy arrangements, in which they contract with outside pharmacies and pay them to dispense drugs on their behalf; or both. In addition, providers who work at covered entities, such as doctors and nurses, may administer 340B drugs to patients directly, such as during office visits. These are known as provider-administered drugs. As a condition of participating in the 340B Program, covered entities must follow certain requirements. For example, they are prohibited from diverting a 340B drug to an individual who is not a patient of the covered entity. Covered entities are also prohibited from subjecting manufacturers to duplicate discounts. Both states and covered entities play key roles in preventing duplicate discounts and forgone rebates. States must know whether covered entities provided 340B drugs to Medicaid beneficiaries in order to exclude those drugs from the rebate requests they submit to manufacturers. When covered entities provide 340B drugs to Medicaid beneficiaries, it is known as “carving in;” if covered entities do not dispense these drugs to Medicaid beneficiaries, it is known as “carving out.” As shown in figure 1, if a state is not aware that a covered entity provided 340B drugs to Medicaid beneficiaries, it would not know to exclude those drugs from its rebate requests, which could lead to duplicate discounts. In contrast, if a state mistakenly believes the entity used 340B drugs when it did not, it might exclude those drugs from its rebate requests and would forgo eligible rebates. To help prevent duplicate discounts, in 1993, HRSA and CMS collaborated to establish the Medicaid Exclusion File (MEF) as a mechanism to assist in the identification of 340B drugs provided to Medicaid FFS beneficiaries. The MEF lists the covered entities that reported to HRSA that they choose to use or “carve in” 340B drugs for their Medicaid FFS patients. Specifically, HRSA requires that covered entities that decide to carve in these drugs for Medicaid provide the agency with the provider number or numbers that the entities use to bill the state for those drugs. The entity and the provider number or numbers it specifies are then listed on the MEF. HRSA guidance specifies that all drugs billed with the provider numbers listed on the MEF should be 340B drugs so a state that choses to use the MEF knows the drugs should be excluded from rebate requests; there is no requirement for states to use the MEF to identify 340B drugs. If a covered entity wants its contract pharmacy to dispense 340B drugs to patients covered under Medicaid FFS, HRSA guidance requires the covered entity, the contract pharmacy, and the state Medicaid program to have an arrangement to prevent duplicate discounts; any such arrangement must be reported to HRSA. When the MEF was created, Medicaid drug rebates were only required for drugs provided under FFS. As such, in a 2014 policy release, HRSA clarified that the MEF is only intended for use for Medicaid FFS, that is, only covered entities that elect to carve in 340B drugs for Medicaid FFS are required to provide the provider numbers used for billing Medicaid FFS for inclusion on the MEF. The MEF is not intended to capture whether covered entities have decided to carve in 340B drugs for Medicaid managed care and, if so, what provider numbers they use for billing for those drugs. HRSA has not created a mechanism for covered entities to use to identify 340B drugs provided to Medicaid managed care beneficiaries, but encourages covered entities to work with states to develop strategies to prevent duplicate discounts for drugs reimbursed through managed care. While HRSA requires covered entities to use the MEF, there is no similar requirement for state Medicaid programs. CMS provides states the flexibility to determine procedures for identifying and excluding 340B drugs from their Medicaid rebate requests. Under a May 2016 final rule, states’ contracts with Medicaid managed care plans that provide coverage of outpatient drugs must require the plans to provide the states with drug utilization data that is necessary for the states to claim Medicaid rebates. In addition, the contracts must require the plans to establish procedures for excluding 340B drugs from the drug utilization data provided to states for purposes of rebate collection. To oversee covered entities’ compliance with 340B Program requirements, in fiscal year 2012, HRSA implemented a systematic approach to conducting audits of a small sample of covered entities, and began conducting audits of 200 entities per year in fiscal year 2015. HRSA audits include covered entities that are randomly selected based on risk-based criteria (approximately 90 percent of all audits conducted each year), or targeted based on information from stakeholders such as drug manufacturers about potential noncompliance (10 percent of the audits conducted). HRSA’s criteria for risk-based audits include a covered entity’s volume of 340B drug purchases, number of contract pharmacies, time in the program, and complexity of its program. Among other things, HRSA’s audits include reviews of each covered entity’s policies and procedures, an assessment of the entity’s compliance with respect to 340B Program requirements, including the prevention of duplicate discounts in Medicaid FFS, and reviews of a sample of prescriptions filled during a 6-month period to identify any instances of noncompliance. Under HRSA’s audit procedures, a covered entity with audit findings is required to 1) submit a corrective action plan to HRSA that indicates it will determine the full scope of any noncompliance (beyond the sample of prescriptions reviewed during an audit) and 2) outline the steps it plans to take to correct findings of noncompliance, including any necessary repayments to manufacturers, among other things. If the HRSA audit shows that duplicate discounts may have occurred, the covered entity must, as part of its corrective action plan, contact the state Medicaid program to determine whether duplicate discounts actually occurred—namely, whether the state requested a rebate on the claims in question, and if so, contact the drug manufacturer to offer repayment. HRSA closes the audit when a covered entity submits a letter attesting that its corrective action plan, including its assessment of the full scope of noncompliance, has been implemented and any necessary repayments to manufacturers have been resolved. In addition, HRSA may re-audit a covered entity (i.e. subject it to a targeted audit) to determine whether it has implemented its corrective action plan. To oversee the Medicaid Drug Rebate Program, CMS receives copies of states’ Medicaid rebate requests each quarter. States are required to submit this data to manufacturers for FFS and managed care drugs, which should not include drugs purchased through the 340B Program, within 60 days of the end of the quarterly rebate period. Specifically, states provide drug utilization data that includes the drug name, national drug code (a unique identifier for each drug), the unit rebate amount, the number of units reimbursed, the rebate amount claimed, and the number of prescriptions, among other things. CMS has a system that reviews this information for errors, such as the inclusion of drugs from manufacturers that no longer participate in the Medicaid Drug Rebate Program, and generates a discrepancy report for the state. CMS also has a system in place to identify, for state review, cases in which the utilization data reflect a substantial increase or decrease in the number of FFS records submitted compared to prior quarters; such a review is not currently performed for managed care. In addition, CMS reviews state Medicaid programs’ contracts with managed care plans using a checklist to ensure that the contracts include elements required by statute or regulation. State Medicaid programs’ policies varied in whether they allowed covered entities to use 340B Program drugs for Medicaid beneficiaries. Most states allowed covered entities to decide whether to use, or “carve in,” 340B drugs for Medicaid beneficiaries at their in-house pharmacies and for provider-administered drugs. Fewer states allowed covered entities to dispense these drugs to Medicaid beneficiaries at contract pharmacies, particularly beneficiaries whose drugs were covered under FFS. Table 1 below summarizes states’ policies on covered entities’ use of 340B drugs for Medicaid beneficiaries for both FFS and managed care by dispensing method. In addition to varying by state, policies on the use of 340B drugs sometimes varied within a state; that is, some states had different policies depending on whether the drugs were provided to Medicaid FFS or managed care beneficiaries, the dispensing method used, or both. For example, Oregon allowed covered entities to decide whether to dispense 340B drugs at contract pharmacies to Medicaid managed care beneficiaries, but required covered entities to carve out (not use) these drugs at contract pharmacies under Medicaid FFS. Illinois required covered entities to carve in 340B provider-administered drugs and those dispensed at in-house pharmacies for Medicaid beneficiaries in both FFS and managed care, but prohibited their use for Medicaid beneficiaries at contract pharmacies. See appendix II for information on each state Medicaid program’s policies regarding covered entities’ use of 340B drugs. The states that allowed or required covered entities to carve in 340B drugs for Medicaid beneficiaries used several different procedures to identify and exclude those drugs from Medicaid rebate requests. These procedures included relying on the MEF, requiring covered entities to use a 340B claim identifier—a code on the claim that indicates that the drug used was purchased at the 340B discounted price, or using other state- developed procedures to identify and exclude 340B drugs from rebate requests. The procedures states used varied between Medicaid FFS and managed care, and among dispensing methods. For example, states were more likely to use HRSA’s MEF to identify and exclude provider- administered drugs in both Medicaid FFS and Medicaid managed care and to use a 340B claim identifier to identify and exclude drugs dispensed at in-house pharmacies. Some states used a combination of procedures or created their own state-specific procedures. For example, 11 states required that covered entities inform them of their decisions to carve in 340B drugs for Medicaid beneficiaries. The states then maintained a list of these covered entities or their providers, which they used to exclude 340B drugs from rebate requests. Oregon required covered entities to provide the state with a list of each 340B drug dispensed to a Medicaid managed care beneficiary at a contract pharmacy so that the state could exclude those drugs from its rebate requests. Vermont required covered entities, on a monthly basis, to send the state a file listing each 340B drug provided to a Medicaid beneficiary; the state used this information to exclude those drugs from rebate requests. See table 2 for a summary of the procedures used by states to identify 340B drugs provided to Medicaid beneficiaries, and appendix III for a listing of the procedures by state. State Medicaid programs’ policies related to 340B drugs were not always documented and some states’ policies may not prevent duplicate discounts. Some states had written policies for the use of 340B drugs, and procedures to identify them, for some dispensing methods, but not for others, such as states that had documented policies for in-house pharmacies but not contract pharmacies. Without written policies, covered entities in those states may not be aware of requirements for dispensing and identifying 340B drugs, increasing the risk of duplicate discounts. Specifically, we found that nine states did not have written policies or procedures on the use or identification of 340B drugs for all dispensing methods. Seven of the nine states had policies or procedures regarding the use and identification of 340B drugs that were used in practice, but these policies and procedures were not always documented. For example: Connecticut did not have documented policies on the use and identification of 340B drugs, but officials from the state reported that it allowed covered entities to provide these drugs to Medicaid beneficiaries and relied on the MEF to identify and exclude them from rebate requests. While Pennsylvania and Ohio had written policies regarding the use of 340B drugs in Medicaid FFS and for some dispensing methods under managed care, the states’ policies requiring covered entities to carve out these drugs for Medicaid managed care beneficiaries at contract pharmacies were not documented. The remaining two states did not have policies or procedures, documented or otherwise, for all dispensing methods: Officials from Washington, D.C. reported that D.C. did not have a policy regarding the use of provider-administered 340B drugs nor did it have procedures to identify and exclude those drugs from its Medicaid drug rebate requests. A Rhode Island Medicaid official told us that the state did not have written policies regarding the identification of 340B drugs dispensed to Medicaid FFS beneficiaries at in-house pharmacies, and that the state did not have procedures, written or otherwise, by which to exclude such drugs from rebate requests. Additionally, while the state had a written policy for identifying and excluding 340B drugs administered by providers at hospitals, officials told us that they had no policy or exclusion procedures for drugs administered by providers at other types of covered entities. In addition, we found that states’ policies may not prevent duplicate discounts. For example, some states used the MEF to identify and exclude 340B drugs from their rebate requests in a manner contrary to the MEF’s purpose as set forth by HRSA. As noted previously, HRSA guidance specifies that the MEF is not intended to be used to identify and exclude 340B drugs provided to Medicaid managed care beneficiaries from Medicaid drug rebate requests. Covered entities are only required to be listed on the MEF if they carve in 340B drugs for Medicaid FFS. Since the MEF may not accurately reflect covered entities’ use of 340B drugs for Medicaid managed care, states’ use of the MEF in this instance may increase the risk of duplicate discounts or forgone rebates unless states require covered entities to make the same decisions on the use of 340B drugs for FFS and managed care. For example, as shown in figure 2, a state’s use of the MEF for managed care would likely result in a duplicate discount if covered entities carve out 340B drugs for Medicaid FFS, but carve in these drugs for managed care, as those entities would not be listed on the MEF. Consequently, the state would not know to exclude drugs provided by those entities from the managed care plans’ utilization data that are used for requesting rebates. If covered entities did the opposite—carved in for FFS and carved out for Medicaid managed care—then the state would likely forgo Medicaid rebates as it would exclude drugs from its rebate request that were not purchased through the 340B Program. Seven of the 13 states that used the MEF exclusively to identify and exclude Medicaid managed care drugs from rebate requests for at least one dispensing method did not require covered entities to make the same carve-in decisions for both FFS and managed care. Additionally, while the six remaining states required covered entities to make the same decision regarding use of 340B drugs in FFS and managed care, that requirement was not always clearly explained in the states’ policies. For example, an official from Arkansas, which used the MEF for identifying and excluding 340B drugs from rebate requests, told us that covered entities are required to make the same carve-in decisions for both Medicaid FFS and managed care. However, it is unclear how covered entities would be aware of that requirement, as it was not documented in the state’s policy manuals at the time of our information request. Finally, states that rely on the MEF or state-developed lists of providers carving in 340B drugs for Medicaid beneficiaries may not be able to identify instances where covered entities are unable to purchase drugs at the 340B Program discounted price, and instead need to purchase drugs outside of the 340B Program. For example, orphan drugs are excluded from the discounted 340B Program price for some covered entities. In these situations, states that rely on the MEF or other state-developed lists of providers may be forgoing rebates. For example, if covered entities do not have a separate provider number for billing Medicaid for these non- 340B drugs, the states would be excluding both 340B and non-340B drugs from their rebate requests. State Medicaid officials in Oregon and Pennsylvania acknowledged that their states were likely forgoing rebates when covered entities listed on the MEF were unable to purchase drugs at the 340B Program price. While these state officials indicated that they did not consider the lost rebates financially significant, the loss of these rebates would also increase federal Medicaid expenditures, since rebates are shared between the state and the federal government. CMS oversight of state Medicaid programs’ efforts to prevent duplicate discounts is limited. States have the flexibility to select the procedures used for identifying and excluding 340B drugs from rebate requests. Although CMS collaborated with HRSA to establish the MEF as a tool for identifying 340B drugs in Medicaid FFS, CMS does not require states to use the MEF in their duplicate discount prevention efforts. Instead, CMS has provided states with options of procedures they could consider for identifying and excluding 340B drugs from rebate requests. For example, CMS’s February 2016 final rule on covered outpatient drugs, which detailed requirements for Medicaid reimbursement of covered outpatient drugs, included in its preamble examples of procedures that states could use to identify and exclude 340B drugs in FFS without prescribing any specific required procedure. Additionally, as noted earlier, the final rule CMS issued in May 2016 on Medicaid managed care included a provision relating to duplicate discounts for Medicaid managed care drugs. Specifically, it mandated that state Medicaid programs’ contracts with managed care plans that provide outpatient drugs require the plans to establish procedures for excluding 340B drugs from utilization data provided to states for use in seeking rebates, but did not specify what procedures plans should use. Most recently, in January 2020, CMS released a bulletin to state Medicaid programs on best practices for preventing duplicate discounts. CMS has some visibility into state Medicaid programs’ 340B-related policies and procedures through its oversight activities, but these activities are not intended to, and do not enable CMS to, assess compliance with the duplicate discount prohibition. For example, CMS has a system in place that reviews copies of states’ quarterly Medicaid drug rebate requests; however, CMS officials told us that these requests do not contain detailed, claim-level information that could be used to determine if specific drugs purchased through the 340B Program were incorrectly included. Additionally, CMS reviews states’ contracts with Medicaid managed care plans to ensure that they include language requiring the plans to have procedures to exclude 340B drugs from Medicaid rebate data provided to states, but CMS officials told us that the contract language does not have to specify or describe those mechanisms, limiting the information available regarding duplicate discount prevention efforts. CMS also required states to submit their plans for reimbursing covered entities for 340B drugs provided under Medicaid FFS to ensure that the states’ payment methodologies complied with federal requirements, but these reviews were not focused on ensuring that such drugs were excluded from rebate requests. CMS officials told us that they do not track which procedures states use to prevent duplicate discounts; review states’ policies or procedures for identifying and excluding 340B drugs from rebate requests for deficiencies or to ensure effectiveness; or audit states’ compliance with the prohibition on duplicate discounts. This is problematic because, as noted previously, we found that not all state Medicaid programs have written policies and procedures that specify the extent to which covered entities can use 340B drugs for Medicaid beneficiaries, or how they are to identify these drugs so the state can exclude them from Medicaid rebate requests. If states do not have written policies, covered entities may not be aware of whether, or under what circumstances, they are permitted to provide 340B drugs to Medicaid beneficiaries or how to properly inform the state of their use, which could result in errors that lead to duplicate discounts and forgone rebates. We found some evidence of confusion from covered entities about state policies. For example, officials from Apexus, which manages HRSA’s 340B Prime Vendor Program, told us that Apexus’s call center, which fields questions from covered entities and other stakeholders about the 340B Program, most frequently receives questions related to clarifying states’ duplicate discount-related policies. These inquiries about state requirements indicate that there is currently confusion among covered entities. CMS’s limited oversight of state Medicaid programs’ efforts to prevent duplicate discounts is also problematic because we found that states’ policies and procedures were not always effective at preventing duplicate discounts, or in line with federal guidance. For example, the MEF is only intended to be used for Medicaid FFS. CMS officials told us that, while the agency was not aware of any states using the MEF for Medicaid managed care, such use would be concerning because it is not an accurate tool for that purpose. However, as previously shown in table 2, we found that eight states relied on the MEF to identify and exclude Medicaid managed care drugs dispensed at in-house pharmacies from rebate requests and 13 states used the MEF to identify and exclude managed care drugs administered by providers. The lack of CMS oversight of state Medicaid programs’ policies and procedures related to duplicate discount prevention is inconsistent with federal standards for internal control for information and communication, which state that management should obtain relevant data from reliable internal and external sources in a timely manner based on the identified information requirements so that data can be used for effective monitoring. Without reviewing states’ policies and procedures, CMS does not have the information needed to effectively oversee states’ compliance with the Medicaid drug rebate statute, which exempts 340B drugs from Medicaid rebate requirements, and ensure that states have effective policies and procedures for preventing duplicate discounts. The lack of oversight of states’ policies and procedures also results in CMS not having reasonable assurance that states are seeking rebates for all eligible drugs, and since Medicaid rebates are shared by the states and the federal government, forgoing rebates increases Medicaid costs for both states and the federal government. We identified several areas of weaknesses in HRSA’s oversight processes that impede its ability to ensure that duplicate discounts are prevented or remedied: Covered entities’ compliance with state policies and procedures is not assessed. HRSA’s auditors are instructed to look for the potential for duplicate discounts in Medicaid FFS by assessing whether the covered entity’s information on the MEF is correct; whether the entity is following its policies and procedures to prevent duplicate discounts; and whether a sample of claims reveals any noncompliance. Auditors are also instructed to use information provided by the covered entity to determine if the covered entity is following state policies. However, HRSA officials told us that its auditors are not expected to independently identify or verify state Medicaid programs’ policies to determine whether the covered entity is actually following what the state requires. Instead, HRSA officials stated that it is a best practice for covered entities to include a description of state Medicaid programs’ policies related to the 340B Program, such as how relevant drugs are to be identified, in their policy and procedure manuals. In addition, HRSA told us that its auditors interview covered entity staff about the controls in place to prevent duplicate discounts, and may discuss state requirements during these interviews. The auditor is then required to use this information to determine whether the covered entity is following state policy. For example, if the covered entity says that the state requires a 340B claim identifier, the auditor is to look to see if the covered entity used that identifier in the sample of claims that are reviewed. However, the auditor is not expected to determine if the state actually requires a claim identifier, or allows covered entities to use 340B drugs. The fact that HRSA does not assess whether covered entities are actually following state policies and procedures regarding the use and identification of 340B drugs for Medicaid beneficiaries is inconsistent with federal standards for internal control related to information and communication. Those standards state that management should obtain relevant data from reliable internal and external sources in a timely manner based on the identified information requirements and evaluate both internal and external sources of data for reliability so that it can be used for effective monitoring. This lack of HRSA oversight is especially concerning because we found that the covered entities we interviewed did not always have a correct understanding of their states’ policies. For example, officials from two of the four Pennsylvania covered entities we spoke with told us they were dispensing 340B drugs to Medicaid managed care beneficiaries at contract pharmacies, despite state officials telling us the state does not allow that practice. As a result of this confusion, duplicate discounts may have occurred as the state was not excluding drugs dispensed by contract pharmacies from its Medicaid rebate requests. Additionally, of the 13 covered entity policy and procedure manuals we reviewed, only four had descriptions of their states’ policies and two of those descriptions were incorrect. If HRSA were to audit the majority of those 13 covered entities, its auditors would likely be unable to appropriately assess the entities’ compliance with state requirements. Without fully assessing compliance with state policy, HRSA’s audits do not provide the agency with reasonable assurance that covered entities are taking the necessary steps to prevent duplicate discounts. As a result, drug manufacturers are at risk of being required to erroneously provide duplicate discounts for Medicaid drugs. Not all identified duplicate discounts are repaid. HRSA officials told us that covered entities’ obligations for preventing duplicate discounts are the same for Medicaid FFS and managed care. However, as we reported in 2018, HRSA audits do not assess for the potential for duplicate discounts in Medicaid managed care despite the fact that the potential for duplicate discounts related to Medicaid managed care has existed since 2010, when manufacturers were required to begin paying Medicaid rebates under managed care in addition to FFS. As we noted in 2018, HRSA indicated that it does not audit for duplicate discounts in managed care because the agency has not issued guidance on how covered entities should prevent this. As a result, we recommended that HRSA issue guidance to covered entities on the prevention of duplicate discounts under Medicaid managed care and incorporate into its audit process an assessment of covered entities’ compliance with the prohibition on duplicate discounts as it relates to Medicaid managed care claims. HHS concurred with these recommendations and, as of October 2019, HRSA reported that it was working to determine next steps related to these recommendations. However, HRSA has noted that the agency lacks explicit general regulatory authority to issue regulations on most aspects of the 340B Program, and also told us, in October 2019, that guidance does not provide the agency with appropriate enforcement capability. As a result, HRSA requested authority in the President’s budget request for fiscal year 2020 to issue regulations on all aspects of the 340B Program, as the agency believes that binding and enforceable regulations would provide it with the ability to more clearly define and enforce policy. In addition, the agency is not pursuing additional guidance under the 340B Program at this time. We note, however, that the law prohibits the payment of duplicate discounts and requires HRSA to issue guidance to covered entities describing methodologies and options for avoiding duplicate discounts. In the absence of federal guidance, HRSA instructs covered entities to work with their states on duplicate discount prevention. HRSA requires covered entities to work with affected drug manufacturers regarding the repayment of duplicate discounts in FFS that are identified through HRSA or manufacturer audits. However, HRSA officials told us that the agency does not require covered entities to take the same actions to address duplicate discounts for managed care claims that HRSA learns about through its audits or other means. For example, HRSA officials told us that they did not follow up on a letter from a state that confirmed a duplicate discount occurred on a Medicaid managed care claim, because the agency did not yet have guidance for covered entities related to Medicaid managed care claims. Additionally, HRSA officials told us they would not require a covered entity to develop a corrective action plan or make offers of repayment to a manufacturer if a drug manufacturer’s audit of that covered entity identified a duplicate discount in managed care. Although HRSA officials told us that they expect covered entities to work in good faith with all parties involved to resolve potential duplicate discounts in managed care, HRSA does not require these actions if a duplicate discount is identified in managed care, as it does in FFS. This is particularly problematic as the majority of Medicaid enrollees, prescriptions, and spending for drugs are in managed care, and the drug manufacturers we contacted believe that duplicate discounts are more prevalent in Medicaid managed care than FFS. HRSA expecting but not requiring covered entities to address identified duplicate discounts related to Medicaid managed care is contrary to federal law, which provides that covered entities are liable to drug manufacturers for duplicate discounts that are identified through HRSA or manufacturer audits. It is also inconsistent with federal internal control standards related to monitoring, which state that management should oversee the prompt remediation of deficiencies and the audit resolution process, which begins when the results of an audit or other review are reported to management, and is completed only after action has been taken that corrects identified deficiencies. Without HRSA requiring covered entities to address identified duplicate discounts in Medicaid managed care as they would duplicate discounts in FFS, drug manufacturers may erroneously provide both 340B discounts and Medicaid rebates on the same drug claim. The prevention of duplicate discounts in the 340B and Medicaid Drug Rebate Programs requires extensive coordination between state Medicaid programs and covered entities, and among agencies within HHS. Similar levels of coordination are required to ensure that states are not forgoing rebates on drugs not purchased at the 340B price, which would result in increased costs for both state and federal governments. Limitations in federal oversight impede CMS’s and HRSA’s ability to ensure compliance with the prohibition on duplicate discounts. CMS does not assess whether states have 340B policies and procedures and, if so, whether they are documented, effective, and accessible to stakeholders. As a result, it is unable to proactively identify and correct problematic policies and procedures, and prevent duplicate discounts and forgone rebates. Additionally, without knowing state Medicaid programs’ 340B policies, HRSA is unable to perform a comprehensive review of whether covered entities are taking the necessary actions to prevent duplicate discounts. In addition, HRSA’s audits are not assessing compliance with the prohibition against duplicate discounts in managed care because the agency has yet to put forth guidance on this issue. While HRSA is not currently pursuing 340B-related guidance, the agency continues to work on determining next steps to respond to our 2018 recommendations on the issue. In the meantime, however, HRSA still must ensure that covered entities are complying with 340B Program requirements, including the prohibition on duplicate discounts in managed care. Failure to do so not only puts drug manufacturers at risk of providing duplicate discounts, but also compromises the integrity of the 340B Program. We are making a total of three recommendations, including one to CMS and two to HRSA. Specifically: The Administrator of CMS should ensure that state Medicaid programs have written policies and procedures that specify the extent to which covered entities can use 340B drugs for Medicaid beneficiaries, are designed to effectively identify if 340B drugs were used, and if so, how they should be excluded from Medicaid rebate requests. The policies and procedures should be made publically available and cover FFS, managed care, and all of the dispensing methods for outpatient drugs. (Recommendation 1) The Administrator of HRSA should incorporate assessments of covered entities’ compliance with state Medicaid programs’ policies and procedures regarding the use and identification of 340B drugs into its audit process, working with CMS as needed to obtain states’ policies and procedures. (Recommendation 2) The Administrator of HRSA should require covered entities to work with affected drug manufacturers regarding repayment of identified duplicate discounts in Medicaid managed care. (Recommendation 3) HHS provided written comments, which are reproduced in app. IV, and technical comments, which we have incorporated as appropriate. In its written comments, HHS concurred with one of our three recommendations and did not concur with the remaining two recommendations. HHS concurred with our recommendation that CMS ensure that state Medicaid programs have written policies and procedures for identifying 340B drugs and excluding them from Medicaid rebate requests and stated that it will work with states to strengthen policies and procedures related to 340B drugs for Medicaid beneficiaries. HHS did not concur with our recommendation that HRSA incorporate assessments of covered entities’ compliance with state Medicaid programs’ policies and procedures into its audit process. HHS stated that HRSA does not have authority to determine whether state Medicaid policies and procedures are “accurate and appropriate.” We agree that HRSA is not the appropriate party for reviewing and assessing state Medicaid programs’ policies and procedures, which is why we recommended that CMS, not HRSA, strengthen its oversight of states’ 340B-related policies and procedures, a recommendation with which HHS concurred. We recommended that HRSA update its 340B Program audits to include assessments of whether covered entities are following state Medicaid programs’ policies and procedures regarding the use and identification of 340B drugs. HHS stated that HRSA does not have authority to enforce covered entities’ compliance with state Medicaid programs’ policies and procedures and that doing so would be “beyond the scope of the 340B Program” and would require additional training for HRSA auditors, who currently “do not have this level of expertise.” While we understand that HRSA does not have authority to enforce compliance with state Medicaid programs’ policies and procedures, covered entities’ compliance with state Medicaid programs’ policies and procedures is fundamental to preventing duplicate discounts and assessing compliance with state policies and procedures is essential to ensuring covered entities’ compliance with the 340B Program’s prohibition on duplicate discounts. Further, HRSA already audits for compliance with certain aspects of states’ 340B-related Medicaid policies for preventing duplicate discounts. Specifically, HHS states that covered entities are expected to include a description of state policy in their policy and procedure manuals. If such descriptions exist, HRSA auditors are required to review those descriptions and determine if covered entities are following them. Thus, HRSA auditors already interpret state Medicaid policies and procedures when performing audits and the agency already enforces compliance with state policies by issuing audit findings when covered entities are not following them. However, as noted in our report, HRSA does not require its auditors to review state Medicaid programs’ actual policies and procedures. Instead, the auditors currently rely on covered entities’ descriptions of those policies and procedures, which we found were not always accurate. Additionally, knowledge of state policies would allow HRSA to incorporate an assessment of compliance into all audits as opposed to only those of covered entities that have such descriptions in their manuals. Finally, without considering states’ actual policies and procedures and ensuring that covered entities are following them, HRSA’s audits cannot effectively identify the potential for duplicate discounts. For example, simply checking covered entities’ actions against information on the MEF does not provide useful information if the covered entities are in one of the many states that do not use the MEF and instead direct entities to identify 340B drugs dispensed to Medicaid beneficiaries via a different mechanism, such as 340B identifiers. HHS states that implementing this recommendation would be burdensome and difficult to operationalize because HRSA would need to be notified of any changes to states’ policies and procedures. We understand that the lack of knowledge of state Medicaid programs’ policies related to duplicate discount prevention at the federal level complicates the ability of HRSA and its auditors to determine what state- level requirements exist and to apply them to audits. This is, in part, why we recommended that CMS ensure that state Medicaid programs’ policies are publicly available—a recommendation that, as noted above, HHS concurred with—and that HRSA work with CMS to obtain these policies as needed. Though we understand that this creates an additional step in HRSA’s audit process, we continue to believe that including an assessment of covered entities’ compliance with state Medicaid programs’ policies and procedures related to 340B drugs is necessary to identify potential duplicate discounts and to ensure covered entities’ compliance with 340B Program requirements. HHS also did not concur with our recommendation that HRSA should require covered entities to work with affected drug manufacturers regarding repayment of identified duplicate discounts in Medicaid managed care. In its response, HHS noted that because HRSA does not have guidance related to preventing duplicate discounts in Medicaid managed care, “it is difficult to assess compliance in this area.” However, our recommendation is not asking HRSA to assess compliance related to duplicate discounts in Medicaid managed care; instead, we are recommending that, when actual duplicate discounts have been identified, HRSA require covered entities to remedy those duplicate discounts. As noted in the report, actual duplicate discounts may be identified and confirmed by state Medicaid agencies through audits or other means. Given that HRSA officials told us that covered entities’ obligations for preventing duplicate discounts are the same for Medicaid FFS and managed care, the steps for addressing identified noncompliance should be similar, and thus, the agency should require and not just “encourage” covered entities to work with manufacturers to remedy any duplicate discounts related to managed care as they do for those related to FFS. Additionally, the potential for duplicate discounts related to Medicaid managed care has existed since 2010, when manufacturers were required to begin paying Medicaid rebates under managed care in addition to FFS. Ten years later, HRSA still has not issued guidance on how covered entities should prevent duplicate discounts in Medicaid managed care and has indicated that it is not pursuing new guidance at this time. This inaction continues to leave the 340B Program vulnerable to noncompliance with federal law. HHS concurred with our 2018 recommendations that HRSA issue guidance to covered entities on the prevention of duplicate discounts under Medicaid managed care and incorporate into its audit process an assessment of covered entities' compliance with the prohibition on duplicate discounts as it relates to Medicaid managed care claims. Until these recommendations are implemented, HRSA must, at a minimum, ensure that covered entities work with manufacturers regarding any identified duplicate discounts in managed care to help ensure compliance with 340B Program requirements. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services, the Administrator of HRSA, the Administrator of CMS, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at DraperD@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix V. Officials from all three drug manufacturers and the organizations that work on their behalf that we contacted reported challenges preventing and detecting duplicate discounts due to a lack of information. For example, officials from drug manufacturers told us that state Medicaid programs do not always provide data on the individual claims for which they were requesting rebates. Specifically, to obtain rebates, states submit requests to participating manufacturers for all drug purchases made that quarter; these requests contain the total quarterly amount owed for each of the manufacturers’ drugs, but not information detailing each claim for which rebates are being sought. Although the Centers for Medicare & Medicaid Services (CMS) encourages states to respond to reasonable manufacturer requests for claim-level data, the provision of such data is not required. Without this claim-level data, manufacturers reported that it is difficult to determine if rebate requests include claims for drugs purchased at the 340B discounted price. Additionally, manufacturers lack complete information on the extent to which covered entities use 340B drugs for Medicaid beneficiaries. This is because the Medicaid Exclusion File (MEF), a list maintained by the Health Resources and Services Administration (HRSA) to assist in the prevention of duplicate discounts, is only required to reflect the provider numbers used by covered entities that choose to use (carve in) 340B drugs provided directly by the covered entity to Medicaid fee-for-service (FFS) beneficiaries. The MEF does not include information on whether covered entities are using 340B drugs for Medicaid managed care beneficiaries and may not include information on contract pharmacies that are dispensing these drugs to Medicaid beneficiaries on covered entities’ behalf. Despite these limitations, the drug manufacturers we contacted reported that when claim-level data is available they review that data to detect potential duplicate discounts before they issue rebate payments. For example, officials from one drug manufacturer told us that they compare the provider numbers on the claim-level data obtained from states with the information on the MEF and dispute rebate requests for any claims from a provider number listed on the MEF. However, officials from some drug manufacturers told us that this approach is ineffective for preventing duplicate discounts for drugs dispensed at contract pharmacies because, as noted above, the MEF may not include information on contract pharmacies, and the claim-level data may only list the provider number for the dispensing pharmacy, not the prescribing covered entity. The drug manufacturers we contacted also reported trying to identify duplicate discounts after rebates have been paid by looking at 340B purchasing patterns. For example, officials from one drug manufacturer told us they look at covered entities’ purchases and assess whether the proportion of 340B purchases is consistent with their carve-in status. Specifically, these officials explained that if a covered entity is not listed on the MEF, then the entity should not be using 340B drugs for Medicaid FFS patients. Therefore, if all or nearly all of the purchases made by that covered entity were at the discounted price, it could indicate the presence of duplicate discounts. While the MEF is only intended to indicate covered entities that are using 340B drugs for Medicaid FFS beneficiaries, officials reported that drug manufacturers also rely on the MEF as a proxy for covered entities’ carve-in practices for Medicaid managed care since there is no equivalent data source. If there are concerns that duplicate discounts occurred, officials from the drug manufacturers we contacted indicated that they may conduct what is referred to as a “good faith inquiry,” in which the manufacturer, or a consultant working on the manufacturer’s behalf, requests data from covered entities on a specific set of drug claims for which they have paid rebates to determine if those claims involved 340B drugs. If drug manufacturers confirm that a duplicate discount did occur, officials reported that they may work to negotiate a repayment from the state or covered entity, depending on which party was responsible for the error. Additionally, one official who works on behalf of manufacturers told us that manufacturers also will work with covered entities to remedy the cause of the duplicate discount to prevent future occurrences. Drug manufacturers told us that it is not always clear whether states or covered entities are responsible for duplicate discounts, and thus, which party should be contacted regarding repayment. Additionally, drug manufacturers reported that some states refer them directly to covered entities to resolve all inquiries. Medicaid program officials in Michigan and Texas, for example, said that their states refer manufacturers to the covered entities because they believe that the covered entities would most likely be responsible for any duplicate discounts that occurred due to a failure to correctly apply the required claim identifiers. If drug manufacturers need assistance resolving their concerns or obtaining repayment for duplicate discounts, they can access options made available by HRSA and CMS. Specifically, drug manufacturers can request approval from HRSA to audit a covered entity to investigate suspicions of duplicate discounts in both Medicaid FFS and managed care. To receive approval from HRSA to conduct an audit, a drug manufacturer must document reasonable cause and provide an audit plan. In addition, HRSA requires the drug manufacturer to use an independent auditor who follows government auditing standards. According to HRSA, from October 2011 through August 2019, 45 audits were requested by drug manufacturers and 26 requests were approved. Of the 26 audits approved by HRSA, the agency received 13 final audit reports, six of which had duplicate discount-related findings. However, while audits can be a tool for identifying duplicate discounts and obtaining repayment, some drug manufacturers we spoke with indicated that the cost of audits may outweigh the benefits received in the form of repayments. Additionally, as noted previously, HRSA does not require covered entities to repay manufacturers for duplicate discounts that occur in managed care. Drug manufacturers also may use the state hearing process or pursue a dispute resolution in conjunction with states through CMS if their issues with state Medicaid programs cannot be resolved through inquires. According to CMS officials, through the dispute resolution process, the agency provides drug manufacturers and states with guidance to assist in determining responsibilities and identifying next steps to work through conflicts. CMS officials said that, in general, they have received five to 10 Medicaid drug rebate disputes per year, about half of which are related to 340B duplicate discount issues. Appendix II: State Medicaid Programs’ Policies on Covered Entities’ Use of 340B Drugs, by Dispensing Method California allows covered entities to dispense 340B drugs at contract pharmacies if there is an approved arrangement between the state, the covered entity, and the contract pharmacy. At the time of our information request, California officials indicated that they only had approved arrangements for certain hemophilia centers and had no approved arrangements with other types of covered entities. New Hampshire allows covered entities to provide 340B drugs to Medicaid beneficiaries, but generally does not allow them to bill Medicaid for these drugs. The one exception is that the state does allow covered entities that are approved family planning clinics to bill Medicaid for 340B drugs administered by providers to Medicaid beneficiaries. The term 340B drugs refers to drugs purchased by covered entities at a discounted price through the 340B Program. Carve out means that the state did not allow covered entities to provide 340B drugs to Medicaid beneficiaries. Carve in means that the state required covered entities to provide 340B drugs to eligible Medicaid beneficiaries. New Hampshire allows covered entities to provide 340B drugs to Medicaid beneficiaries, but generally does not allow them to bill Medicaid for these drugs. The one exception is that the state does allow covered entities that are approved family planning clinics to bill Medicaid for 340B drugs administered by providers to Medicaid beneficiaries. Managed care plans in this state do not cover outpatient drugs dispensed at pharmacies; they only cover provider-administered drugs. Appendix III: State Medicaid Programs’ Procedures for Identifying 340B Drugs, by Dispensing Method State does not allow covered entities to use 340B drugs for Medicaid fee-for-service beneficiaries for this dispensing method, and thus does not need a procedure to identify these drugs. Massachusetts requires contract pharmacies to include the covered entities’ National Provider Identifier on claims using 340B drugs, which the state then uses to exclude those claims from its rebate request. New Hampshire allows covered entities to provide 340B drugs through this dispensing method, but does not allow them to bill Medicaid for these drugs. Rhode Island uses a 340B claim identifier to identify and exclude associated drugs administered by providers at hospitals, but does not have any procedures to identify these drugs administered by providers at other types of covered entities. State does not allow covered entities to use 340B drugs for Medicaid managed care beneficiaries for this dispensing method and thus does not need a procedure to identify these drugs. New Hampshire allows covered entities to provide 340B drugs through this dispensing method, but does not allow them to bill Medicaid for these drugs. Managed care plans in this state do not cover outpatient drugs dispensed at pharmacies; they only cover provider-administered drugs. In addition to the contact named above, Michelle Rosenberg (Assistant Director), David Lichtenfeld (Analyst-in-Charge), Amanda Cherrin, and Sarah Tempel made key contributions to this report. Also contributing were Jennie Apter, Ethiene Salgado-Rodriguez, and Jennifer Whitworth.", "summary": "Covered entities can receive substantial discounts on outpatient drugs through the 340B Program, an estimated 25 to 50 percent of the cost of the drugs, according to HRSA. Additionally, Medicaid drug rebates are an important source of savings for states and the federal government, saving more than $36 billion in fiscal year 2018. However, ensuring that manufacturers are not subject to both discounts requires coordination within HHS, and between covered entities and states. GAO was asked to provide information on the prevention of duplicate discounts. Among other things, this report examines HHS's efforts to ensure compliance with the prohibition on duplicate discounts. GAO reviewed documentation, including federal policies and those from all 50 states and Washington, D.C. on preventing duplicate discounts. GAO also interviewed officials from CMS, HRSA, and 16 covered entities from four states selected to obtain variation in the types of entities and other factors. The 340B Drug Pricing Program (340B Program) and the Medicaid Drug Rebate Program require manufacturers to provide discounts on outpatient drugs in order to have their drugs covered by Medicaid. These discounts take the form of reduced sales prices for covered entities participating in the 340B Program—eligible hospitals and federal grantees—and rebates on drugs dispensed to Medicaid beneficiaries, shared by states and the federal government. However, federal law prohibits subjecting manufacturers to “duplicate discounts” in which drugs provided to Medicaid beneficiaries are subject to both 340B Program discounted prices (i.e., are 340B drugs) and Medicaid rebates. To prevent duplicate discounts, state Medicaid programs must know when covered entities dispense 340B drugs to Medicaid beneficiaries, so the state programs can exclude those drugs from their Medicaid rebate requests. GAO found that limitations in the Department of Health and Human Services's (HHS) oversight of the 340B and Medicaid Drug Rebate Programs may increase the risk that duplicate discounts occur. HHS's Centers for Medicare & Medicaid Services (CMS) conducts limited oversight of state Medicaid programs' efforts to prevent duplicate discounts. CMS does not track or review states' policies or procedures for preventing duplicate discounts, and GAO found that the procedures states used to exclude 340B drugs are not always documented or effective at identifying these drugs. As a result, CMS does not have the information needed to effectively ensure that states exclude 340B drugs from Medicaid rebate requests. CMS also does not have a reasonable assurance that states are seeking rebates for all eligible drugs, potentially increasing costs to state and federal governments due to forgone rebates. HHS's Health Resources and Services Administration's (HRSA) audits of covered entities do not include reviews of states' policies and procedures for the use and identification of 340B drugs. As a result, the audits are unable to determine whether covered entities are following state requirements, and taking the necessary steps to comply with the prohibition on subjecting manufacturers to duplicate discounts. GAO reported in 2018 that HRSA had not issued guidance on, and did not audit for, duplicate discounts in Medicaid managed care and recommended the agency do so as the majority of Medicaid enrollees, prescriptions, and spending for drugs are in managed care. HRSA is working to determine next steps to address these recommendations. In this report, GAO found that, unlike Medicaid fee-for-service, when duplicate discounts in Medicaid managed care claims are identified, HRSA does not require covered entities to address them or work with manufacturers to repay them. As a result, manufacturers may be subject to duplicate discounts for drugs provided under managed care. Given these limitations in federal oversight, HHS does not have reasonable assurance that states and covered entities are complying with the prohibition on duplicate discounts. GAO is making three recommendations, namely that: 1) CMS ensure that state Medicaid programs have written policies and procedures that are designed to prevent duplicate discounts and forgone rebates; and that HRSA 2) incorporate covered entities' compliance with state policies into its audits, and 3) require covered entities to work with manufacturers regarding repayment of identified duplicate discounts in managed care. HHS agreed with the recommendation to CMS, but disagreed with those to HRSA. GAO continues to believe these are needed to improve oversight and the integrity of the 340B Program, as explained in the report.", "document_type": "gao"}
{"report": "The primary purpose of the CORD Project is to develop and implement strategies for reducing obesity among low-income children. According to CDC, strategies that have been used to prevent and manage obesity include screening patients using body mass index (BMI), so children and their parents understand their risks; supporting healthy behaviors—such as eating vegetables and promoting physical activity—in early care and education centers and schools; and educating parents on how to reinforce healthy living habits at home. BMI is used to determine overweight and obesity (see sidebar). Percentiles are calculated from the Centers for Disease Control and Prevention (CDC) growth charts developed from national survey data collected between 1963 and 1994. Funding for the CORD Project was first made available through the enactment of the Patient Protection and Affordable Care Act, about one year after the CORD Project was authorized. In January 2011, CDC published the funding opportunity announcement—which outlined the goals of the grant as well as the eligibility criteria and other requirements—and, in September 2011, the first demonstration projects began. Congress subsequently appropriated additional funding for the CORD Project in April 2015 and January 2018, bringing the total amount appropriated to $65 million for fiscal years 2010 through 2023. CDC officials told us that during this time period the CORD Project was the primary source of CDC funding for childhood obesity research focused on low-income children. CDC implemented the CORD Project in three separate grant phases, with different design approaches and grantees. (See fig. 1.) Across the three CORD Project phases—only the first of which is complete—CDC has awarded ten grants to entities to implement demonstration projects aimed at reducing obesity in low-income children. (See table 1.) In the first phase of the CORD Project, CDC also awarded a grant to the evaluation center to conduct a cross-site evaluation of the implementing grantees’ demonstration projects. CDC made four key design changes between the three CORD phases. CDC changed the scope of the project (i.e., type of strategies implemented), the type of evaluations (i.e., how it evaluated the strategies), the purpose of the study design, and the extent of participation by state Medicaid or CHIP programs. (See fig. 2.) CDC officials designed the CORD Project based on the language and requirements in CHIPRA and the CORD Project Plan developed by HHS, according to CDC officials. For CORD phases 2 and 3, CDC officials modified elements of the design in response to lessons learned, time frames for implementation, and recommendations related to childhood obesity made by national organizations such as the U.S. Preventive Services Task Force (hereafter referred to as the Task Force). Scope of CORD Project. After CORD phase 1, CDC officials shifted the scope of the CORD Project from prevention to the treatment of children who are overweight or have obesity, according to CDC officials. Specifically, CDC designed CORD phase 1 to require grantees to implement demonstration projects that integrated public health and primary care strategies by promoting children and their families’ use of healthy behaviors and by modifying community environments. CORD phase 1 grantees implemented strategies in two types of settings: (1) community and (2) health care settings. Public health strategies are activities and programs delivered in community settings, such as schools and early care and education centers. Grantees also implemented primary care strategies, which in general are BMI screenings or other activities implemented in health care settings, such as during physician visits in federally qualified health centers. While CORD 1 grantees implemented strategies in both types of settings, the specific strategies that each CORD phase 1 grantee implemented varied. (See text box and app. I for additional information about the strategies CORD 1 grantees implemented.) Examples of Strategies Childhood Obesity Research Demonstration Phase 1 Grantees Implemented California Demonstration Project Public health strategies implemented included training staff at early care and education centers on health behavior change strategies and providing centers with large self-service water containers to promote increased water intake. Primary care strategies implemented included body mass index (BMI) screenings for children participating in early care and education centers. BMI is a measure used to determine overweight and obesity. Massachusetts Demonstration Project Public health strategies implemented included training teachers in participating elementary schools on how to implement evidence-based health education curricula that encouraged learning about nutrition and physical activity. Primary care strategies implemented included establishing a healthy weight clinic located in the participating health centers. Texas Demonstration Project Public health strategies implemented included providing classroom-based nutrition and gardening curricula in the early care and education centers. Primary care strategies included modifying electronic health records systems to increase provider awareness and action related to maintaining healthy weight, such as prompting clinicians to refer children who were overweight or had obesity to additional services. Pediatric Weight Management Interventions The Centers for Disease Control and Prevention (CDC) defines pediatric weight management interventions as intensive behavioral interventions designed to address excess weight through child and parental counseling on diet, physical activity, or behavior change management. The U.S. Preventive Services Task Force refers to these as interventions for weight management interventions. For CORD phases 2 and 3, CDC shifted the scope of the CORD Project to the treatment of children who are overweight or have obesity. Specifically, CDC modified the scope to only focus on implementing pediatric weight management interventions, one type of primary care treatment strategy (see sidebar). CDC officials told us they changed the scope of CORD phase 2 in response to the shorter, 2-year funding period authorized by law. CDC officials stated that unlike CORD phase 1, the shorter time frame for CORD phase 2 did not allow for a planning year to establish and solidify community relationships across multiple community settings while also enabling sufficient time to implement the strategies and analyze outcome and other data. CDC also modified the scope for CORD phase 2 and 3 to focus on pediatric weight management interventions in response to existing national recommendations related to childhood obesity, according to CDC officials. The Task Force recommended that primary care providers screen children 6 years and older for obesity and offer, or refer children with obesity to, pediatric weight management interventions. In making its recommendation, the Task Force found that pediatric weight management interventions should involve at least 26 hours of contact between the provider and the child, family, or both over a period of 2 to 12 months. According to CDC’s funding opportunity announcement for CORD phase 2, a 2007 expert committee convened by the American Medical Association similarly recommended that all health care providers address weight management and lifestyle issues with children at least once a year and provide behavior counseling on key obesity-related behaviors. CDC officials stated they designed CORD phase 2 to meet the guidelines and standards outlined in these recommendations. After Congress extended the CORD Project for a 6-year period beginning in fiscal year 2018 and appropriated additional funding, CDC designed the scope of CORD phase 3 as a 5-year grant to continue efforts to implement pediatric weight management interventions only. CDC officials stated they considered returning to an integration of public health and primary care strategies for CORD phase 3, similar to CORD phase 1, but decided that the best use of resources was to focus on integrating pediatric weight management interventions into communities, which includes linking families with resources already available in the community, such as low-cost physical activity offerings. Common Outcome Measures for Childhood Obesity Research Demonstration (CORD) Phase 1 Frequency of fruit and vegetable consumption Frequency of sugar-sweetened beverage consumption Physical activity Sleep time Screen time (e.g., watching television and playing video games) Body mass index Quality of life (e.g., physical, emotional, and social) Type of evaluations. In CORD phase 1, CDC awarded a grant to another entity—the evaluation center—to conduct a cross-site evaluation to aggregate results of the three implementing grantees’ demonstration projects. In designing CORD phases 2 and 3, CDC did not award grants to independent entities to conduct cross-site evaluations of the implementing grantees’ demonstration projects. In CORD phase 1, the cross-site evaluation was intended to help inform national policy decision- making, including recommendations regarding the applicability of CORD strategies in other communities. To assess the effectiveness of CORD phase 1, CDC designed the cross-site evaluation to examine the demonstration projects using a set of common outcome measures, which the evaluation center developed in collaboration with CDC officials and implementing grantees (see sidebar). CDC officials told us they removed the cross-site evaluation component for CORD phases 2 and 3 in part due to challenges executing the cross- site evaluation in CORD phase 1. Officials explained, for example, that the difficulty in developing common outcome measures that could be analyzed across the three demonstration projects that were both valid and specific enough to the strategies was a challenge given the variation in the strategies implemented by each grantee, data collection time frames, and methodologies. In addition, CDC officials stated the implementing grantees had sufficient capacity to conduct their own evaluations. For these reasons, CDC officials said they concluded that the cross-site evaluation was not an efficient use of resources. The CORD phase 1 grantees also identified the following challenges related to the cross-site evaluation: Grantees told us there was insufficient time to develop the common outcome measures prior to implementing the strategies. One grantee noted this resulted in them needing to collect some data retrospectively instead of collecting it in real time. Grantees also collected data at different time frames from each other, which resulted in limited data for measuring outcomes via the common measures. Evaluation center officials stated that the lack of a common timeline for collecting data resulted in them only being able to analyze changes in common outcomes measures at the two common time points across all three grantees—baseline and 12 months—even though some grantees collected data at later time points (e.g., 24 months after implementation began). Thus, evaluation center officials said they were unable to determine whether changes in outcomes observed were sustained 24 months after implementation. Grantees reported challenges in creating valid common outcome measures applicable across the varying age ranges, locations, and strategies implemented for the three demonstration projects that affected results of the cross-site evaluation. For example, only the Massachusetts demonstration project chose to implement strategies in the Special Supplemental Nutrition Program for Women, Infants and Children offices, making any data collected about that strategy unable to be included in the cross-site evaluation. CDC required the implementing grantees in all CORD phases to conduct their own evaluations and report on outcomes associated with the strategies implemented under their demonstration projects. Specifically, CDC expected the grantee-specific evaluations to measure health outcomes—such as changes to BMI, nutrition, and physical activity—and quality of life, and to report information on the processes, outcomes, and costs of the individual demonstration projects in the evaluations. Purpose of study design. While CDC designed CORD phases 1 and 2 to build knowledge and evidence on strategies for reducing obesity among low-income children, CDC designed CORD phase 3 to focus on translating strategies proven to reduce childhood obesity into routine use for low-income families. More specifically, for CORD phases 1 and 2, CDC required grantees to use or adapt strategies that previously had not been rigorously tested in low-income children. For example, in CORD phase 2, the Arizona demonstration project adapted a preexisting program—which was aimed at preventing child behavior issues through motivational interviewing techniques and parent education—to improve weight-related health behaviors in low-income children. By comparing low-income participants receiving the strategies with those who did not, the Arizona demonstration project aims to develop evidence about whether or not these strategies work to reduce obesity in low-income children. Arizona officials told us that while the CORD phase 2 study design is appropriate for helping to expedite the translation of knowledge into practice, it has nonetheless been challenging to implement the demonstration project in a 2-year period. The officials explained 2 years is a short period of time for this type of demonstration project. For CORD phase 3, CDC is requiring grantees to take an existing evidence-based pediatric weight management intervention and convert it into a user-friendly package of information, containing all materials clinical or community-based entities would need to easily, efficiently, and completely replicate the pediatric weight management intervention. Materials may include implementation manuals, training curricula, technical assistance, and evaluation materials. CORD phase 3 grantees are required to partner with clinical or community entities that will then use the package to implement the set of pediatric weight management interventions in their community. Additionally, CORD phase 3 grantees are required to make edits to the packaged materials based on the results of the implementation and develop sustainability and dissemination plans to implement the pediatric weight management intervention at additional locations. CDC officials and agency documentation outlined multiple reasons why they modified the study design for CORD phase 3. For example, in its funding opportunity announcement for CORD phase 3, CDC noted that there have been challenges in moving research-based, national recommendations, like Task Force recommendations, into practice. According to CDC officials, this challenge is especially great in low- income communities, where there are a limited number of available pediatric weight management interventions that are rigorous enough to meet the standards outlined by the Task Force. Additionally, officials noted that when these interventions are available, families are generally charged for the services. CDC officials told us that, according to the literature, it can take many years for evidence-based clinical interventions to make it into mainstream practice. Thus, by designing CORD phase 3 to package evidence-based pediatric weight management interventions that will be targeted to communities with low-income families, CDC officials told us they hope to reduce the number of years before adoption of such treatment strategies is prevalent. Participation by state Medicaid or CHIP program officials. In the design for CORD phase 1, implementing grantees were not required to develop relationships with officials from their state Medicaid or CHIP offices or with other payers, but these relationships were encouraged, according to CDC officials. At each subsequent CORD phase, CDC modified its expectations of grantees regarding the involvement of state Medicaid and CHIP program officials in the demonstration projects. Specifically, CDC added a requirement that the implementing grantees form a payer advisory board with representatives from state Medicaid or CHIP offices and encouraged grantees to collaborate with other relevant health care stakeholders, such as private payers, to foster discussions about how to obtain reimbursement for CORD strategies. Noting the importance of establishing these types of relationships, CDC officials told us that grant funding and in-kind donations—which CDC encouraged grantees to identify and use to supplement CORD grant funding—are not sustainable sources of funding for continued implementation of childhood obesity programs. As a result, the officials told us reimbursement from insurers, such as Medicaid or CHIP, is necessary to sustain the implemented strategies at the level of intensity required by the Task Force recommendations. For example, in CORD phase 2, officials from the Arizona demonstration project told us they included representatives from United Healthcare’s private and Medicaid health plans and also a representative from Mercy Care, a not-for-profit Medicaid plan, on their payer committee. Arizona demonstration project officials stated they were working with representatives of the state Medicaid program and private health plans to determine what kind of evidence payers would need to reimburse for obesity-related services. Reimbursement is a key focus in CORD phase 3, and CDC officials told us they plan to assist grantees in determining which services provided within the pediatric weight management interventions may be reimbursable. Specifically, CDC officials stated they will coordinate opportunities for information sharing, technical assistance, and networking between CORD phase 3 grantees, states, and CMS in order to explore broader Medicaid and CHIP coverage options for the services delivered through the grants. CMS officials noted that medical services provided under the grant could be reimbursable under states’ Medicaid and CHIP programs, including under the Early and Periodic Screening, Diagnostics and Treatment benefit. While CDC changed some design elements of the CORD Project between the phases, according to CDC officials, the agency used a consistent approach in managing grantees. Specifically, CDC officials told us that in CORD phases 1 and 2 they promoted collaboration between themselves and the grantees, as well as among the grantees, and monitored the grantees through regular interactions with them. CDC officials told us they used a team of personnel with different expertise to oversee the CORD phase 1 and 2 grants. For example, the team included a project officer who specialized in program management to oversee the day-to-day operations, as well as subject matter experts, including one experienced in evaluation design. CDC officials told us they interacted with CORD phase 1 and 2 grantees on regular conference calls and conducted annual site visits to each grantee. Grantees stated that CDC’s site visits aided them in implementing their demonstration projects by keeping them and their community partners accountable. In addition, grantees told us that CDC collaborated with them to provide expertise on, or troubleshoot the design of, the implementation of their demonstration projects. For example, Arizona demonstration project officials told us that CDC officials helped them to figure out how to best achieve their desired sample size for a strategy they were implementing. CDC officials told us they plan to continue a similarly collaborative management approach for CORD phase 3. CDC officials stated they also monitored CORD phase 1 and 2 grantees by requiring grantees to regularly report on their efforts and generally plan to monitor CORD phase 3 grantees the same way. For example, CDC required CORD phase 1 grantees to submit annual progress reports at least 90 days before the end of the budget period that included descriptions of progress made towards the research goals, information on expenditures, and a detailed budget justification for the new budget period. CDC also required CORD phase 1 grantees to submit both annual progress reports and a final progress report. CORD phase 1 grantees told us that CDC officials were helpful in providing administrative support that ensured grant paperwork was completed consistent with requirements. The evaluation center’s cross-site evaluation and the implementing grantees’ evaluation findings reported some improvements in BMI and other outcomes measured among children who received CORD phase 1 strategies. Specifically, the evaluation center reported that positive changes on these outcomes were observed most often among the following groups of children, providing some evidence of the effectiveness of the strategies delivered: Children who received primary care strategies, such as individualized counseling. Children who received public health strategies, such as an evidence- based nutritional program, in addition to the primary care strategies. In evaluating the CORD 1 demonstration projects, the evaluation center did not examine which specific strategies were the most effective. The primary objective of the cross-site evaluation was to determine if there was evidence that an integrated approach had any advantage over implementing either public health only or primary care only strategies. The evaluation center examined the extent to which the three CORD 1 demonstration projects collectively were associated with positive changes over time in behavior or reductions in BMI. Because of the considerable variation in each of the three demonstration projects, the evaluation center grouped the various strategies implemented by the three grantees into two categories for the analysis: public health and primary care plus. Next, the evaluation center categorized children by the types of strategies they received (public health only, primary care plus only, or both public health and primary care plus) and by age (2 to 5 years, 6 to 8 years, and 9 to 12 years). The evaluation center tested whether each of the possible combinations of strategy and age showed improvement over a 12-month period for each common measure. Using this approach, the evaluation center found some improvements for all of the common outcomes measured; however, improvements were not observed for each strategy or age group. Specifically, of the 81 possible combinations of strategy and age, 52 demonstrated some improvement over the 12-month period; however, only 16 of them showed a statistically significant improvement. (See fig. 3.) For example, BMI improved for children over the 12-month period in three of the strategy and age combinations, but the improvement was statistically significant for just one of those combinations. Among the 52 groups that showed improvements at 12 months, most of the differences observed were very small. For example, from the start of the intervention to 12 months after the intervention, there was about a 1 percent increase in the percentage of children who reported they were physically active for 60 minutes at least one day a week. The implementing grantees—each of which conducted their own evaluations—also reported some improvements in the children who received CORD phase 1 strategies. Similar to the evaluation center’s findings, the implementing grantees did not report improvements for all participating age groups or all outcomes they examined. Among their findings, the grantees reported the following: Children at participating early care and education centers in Texas, who were exposed to strategies such as classroom-based nutrition and gardening curricula, demonstrated modest improvements in BMI over a 2-year period when compared with children who did not receive these Texas demonstration project strategies. The Texas demonstration project also reported improvements in BMI for some children who participated in a weight management program administered in YMCAs compared with a different weight management program administered in primary care clinics. Specifically, researchers found that the YMCA program was more effective in reducing BMI for low-income children at 3 months but not at 12 months after implementation of the program. Children who received both public health and primary care strategies under the California demonstration project experienced some improvement on some outcome measures when compared to children who only received one type of strategy. For example, children who are overweight or have obesity who received both public health and primary care strategies reported playing less hours of video games during the week than those who only received the primary care strategies. During CORD phase 1, the Massachusetts demonstration project observed some improvements over time in the children who received CORD strategies. For example, the percentage of seventh grade students with obesity decreased from the start of implementation compared with 24 months after implementation in the two communities where the strategies were implemented. However, these results were modest; the decrease in the percentage of students with obesity was less than 3 percent in both communities. CDC officials, implementing grantees, and the evaluation center noted that modest or no effects were likely in part due to small sample sizes because of recruitment issues. Regarding recruitment, CDC officials told us that two of the three CORD phase 1 demonstration projects had issues with recruitment that caused sample size issues and ultimately statistical power issues. Specifically, when there is a smaller sample size, a study may be underpowered, which means that statistically significant effects are less likely to be detected even when differences exist. CDC officials explained that having limited statistical power affects the ability for more specific modeling or analysis to determine for whom the strategies works best (e.g., those with obesity or severe obesity). Grantees and CDC officials told us that when faced with recruitment issues, grantees made changes to their recruitment strategies. For example, grantees reduced the minimum BMI required for children participating in the demonstration projects in an attempt to increase participation. However, grantees told us they were still not able to reach their anticipated number of participants. Additionally, an official from the evaluation center told us some common outcome measures used in the cross-site evaluation were limited. The official explained that, had the grantees had more time to reach consensus on how to collect the data for the common outcome measures, or had the common outcome measures been identified in advance of the implementing grantees developing their own evaluations using measures specific to their demonstration projects, the evaluation center might have had more precise data to demonstrate improvements among participants. In planning for CORD phase 1, CDC officials acknowledged that the demonstration projects might not result in significant changes for some outcomes. CDC’s funding opportunity announcement noted that changes in health indicators, such as BMI, are long-term objectives, and that the period of funding for the projects might be too short to demonstrate significant improvement in these outcomes. A CDC official stated that although strong results were not found across each of the demonstration projects, the results of the implemented strategies provided evidence that these strategies could be implemented in a real-world setting. Thus, they noted the lack of stronger and larger effects does not mean that the demonstration projects were not successful. CORD phase 1 grantees told us they continue to analyze the data and expect to publish additional findings, even though the grant period has concluded. For example, the evaluation center told us they had enough data from the CORD Project to continue publishing for many years and planned to publish studies examining how existing community policies— such as physical activity policies—affected the outcomes of the implemented strategies. One of the implementing grantees also told us that having more time to fully analyze, use, and publish results from the data was needed. CDC officials and CORD grantees identified several factors that affected grantees’ ability to implement strategies to reduce childhood obesity among low-income children. According to CDC officials, policymakers and researchers should consider these factors when implementing similar strategies in the future. CDC officials or implementing grantees identified the following factors they observed across the CORD grantees: Staff turnover. CDC officials told us that the turnover of principals and other administrative personnel trained to provide the strategies is one factor that negatively affected the implementation of the strategies in schools or clinics. For example, CDC officials noted that in the Massachusetts demonstration project, researchers had to establish a relationship with a new principal of one of the participating schools when the other principal left, which delayed progress in implementation at that location. Similarly, the Arizona demonstration project also experienced staff turnover at the clinics, which led to a need for retraining and challenges in staff flows. CDC officials suggested that future research should consider incorporating staff retraining costs in the design of public health strategies to help mitigate this challenge. Family support. CDC officials told us that grantees had to provide more support than initially anticipated to families to better ensure their participation. CDC officials told us that grantees addressed this challenge by allowing siblings to also attend or participate in the activities or by holding activities on weekends or after school to accommodate parents’ work obligations. Strategies should be designed to be flexible for families, as there are competing demands on the families participating in the demonstration projects, CDC officials explained. Pertinent programs and policies. Implementing grantees noted that the preexistence of programs or policies that promoted healthy behaviors in the public health and primary care sectors positively affected their implementation of CORD strategies. For example, Massachusetts demonstration project officials told us that the strategies they implemented complemented an existing statewide program that promoted opportunities for healthy eating and active living in the communities, schools, childcare centers, and businesses. Grantee officials attributed the organizational commitment and motivation they observed in participating schools to these preexisting activities. Commitment from partner organizations. Implementing grantees found that the commitment of partner organizations, such as schools, was an important factor affecting implementation. According to implementing grantees, determining the willingness and ability of an organization to implement the strategies is important—by identifying, for example, leaders who support the strategies and can help ensure staff commitment to execute them. The Massachusetts demonstration project reported that 90 percent of the stakeholders they worked with noted the presence of leadership and administrative support for the project reduced feelings of conflict between program implementation and other priorities. Alternatively, the California demonstration project identified the lack of a strong supporter in a leadership position as a barrier to implementation. Parental stresses. Implementing grantees found that parental stresses related to social economic status (e.g., food insecurity or accessibility challenges, including transportation to intervention sites) was a major factor negatively affecting family participation and the implementation of the strategies. Grantees explained that understanding the effect of these stresses on a family’s ability to focus on the strategies to reduce childhood obesity is important and grantees should plan for ways to mitigate those stresses. CDC told us that the CORD phase 3 demonstration projects may be able to help mitigate some of the challenges identified from prior CORD experiences, as noted above. For example, CDC officials told us they plan to work with CORD phase 3 grantees to find ways to mitigate challenges associated with staff turnover, which could include taping trainings or allowing for virtual opportunities for retraining. Additionally, the CORD phase 3 grants are implementing pediatric weight management interventions in different settings—some in clinical settings and some in community settings—which CDC officials said may provide parents with additional flexibility to participate in the strategies. CDC has taken steps to share CORD phases 1 and 2 design materials and available results with researchers and others. For example, CDC shared on its website information for CORD phases 1 and 2, including project summaries, background information about the grantees, and published literature describing the project designs and results. In addition, CDC shared lessons learned about the CORD Project and evidence-based childhood weight management programs during a series of webinars. According to CDC officials, the intended audience for the webinars included public health practitioners and researchers; local, state, and federal government agency officials; health care professionals; policy analysts; and community health workers. CDC officials also told us they have presented CORD results and lessons learned at conferences and at meetings organized by other HHS agencies. Specifically, a CDC official and grantees summarized results from the first phase of the CORD Project at the American Academy of Pediatrics’ Annual Conference in 2016. CORD phase 1 results were also presented at the 2018 Annual Meeting for the Association of State Public Health Nutritionists. Additionally, in November 2017, CORD phase 1 grantees met with researchers from the National Institutes of Health’s Childhood Obesity Prevention and Treatment Research program to share lessons learned from their respective research. To further disseminate CORD results, CDC officials highlighted their planned report to Congress, as required by CHIPRA, which was subsequently issued in September 2019. The report describes the findings for CORD phase 1 and provides brief descriptions of the CORD phase 2 grantees and their demonstration projects, since the results of those projects are not yet available. The report identifies CORD phase 1 findings, including information about the costs of implementing the strategies. CDC officials noted that the implemented public health strategies, such as providing classroom-based nutrition and gardening curriculum or programs that promote physical activity, cost less than primary care strategies. Specifically, CDC reported that the costs of public health strategies in early care and education centers ranged from $26 to $96 per child, the costs of some primary care strategies ranged from $164 to $181 per child, and the cost of more intensive family-based weight management programs ranged from $2,107 to $2,220 per child. CDC, in collaboration with other HHS agencies, has also taken some steps to promote the wider adoption of CORD strategies in low-income communities. For example, CDC and CMS have had preliminary discussions about how CMS could help CORD grantees understand how Medicaid and CHIP programs could reimburse for the obesity-related strategies they are implementing as part of the CORD Project, which CDC officials told us could help to sustain and expand these strategies to other low-income communities. CMS officials told us they are considering whether to issue guidance to state Medicaid and CHIP programs that explains how some states have been able to reimburse entities for the provision of overweight- and obesity-related services. CDC officials told us that after discussions with CMS officials, they provided information to CMS in October 2018 that could be used for a possible CMS information bulletin to state Medicaid and CHIP officials on childhood obesity. In addition, CDC and the Office of the Assistant Secretary for Planning and Evaluation within HHS have awarded a cooperative agreement to the National Association of Community Health Centers to increase the implementation of an evidence-based childhood weight management program—Mind, Exercise, Nutrition, Do It!—by federally qualified health centers. According to HHS officials, the National Association of Community Health Centers is assisting 14 federally qualified health centers in five states (Arizona, Florida, Illinois, Mississippi, and North Carolina) to implement this intervention and, based on lessons learned, plans to develop an implementation guide to support the expansion of this strategy to other health centers. CDC officials also told us they are coordinating with the National Cancer Institute within the National Institutes of Health to share knowledge with CORD phase 3 grantees about how to develop business models to support the expansion of successful strategies, which aligns with one of CDC’s goals for CORD phase 3 to determine how to increase the adoption of successful strategies beyond the CORD intervention sites. CDC officials and implementing grantees provided us some examples of CORD strategies and materials that continue to be used in the states where they were implemented or have been implemented in other low- income communities. Officials from the Massachusetts demonstration project told us that some of the primary care strategies they developed during CORD phase 1 are still provided in the healthy weight clinics that participated in the project. Officials from the Arizona demonstration project told us they have received funding from the U.S. Department of Agriculture to develop a new training module for health care providers interested in implementing the project’s pediatric weight management intervention. They explained that the new training module will include information on parenting strategies specific to child health behaviors (e.g., monitoring of physical activity) and examples of stories from the families who participated in the Arizona demonstration project. CDC officials also told us that materials that CORD grantees used as part of their strategies are publically available for use by researchers and other communities. These materials include a primary care resource guide developed in collaboration with the American Academy of Pediatrics, the Coordinated Approach to Child Health early childhood kit, and a healthy weight clinic implementation guide. We provided a draft of this report to HHS for comment. HHS provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Health and Human Services, the appropriate congressional committees, and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix II. The Children’s Health Insurance Program Reauthorization Act of 2009 (CHIPRA) authorized the Department of Health and Human Services (HHS) to establish the Childhood Obesity Research Demonstration (CORD) Project. CHIPRA specified that HHS provide project grants to universities or other eligible entities to implement activities to reduce childhood obesity among low-income children. HHS designated the Centers for Disease Control and Prevention (CDC) as the agency responsible for designing, awarding, and managing the grants. Subsequent laws provided additional funding and extended the CORD Project for two more phases. The first phase of the CORD Project began in September 2011 and was completed in September 2016. The purpose of CORD phase 1 was to determine whether implementing strategies in public health sectors, including early care and education centers, schools and community organizations, and primary care sectors, such as health care clinics, could improve low-income children’s risk factors for obesity. CDC funded three implementing grantees: San Diego State University, the Massachusetts State Department of Public Health, and the University of Texas Health Science. CORD phase 2 started in June 2016. The purpose of this phase was to further test if strategies implemented in the primary care sector would reduce the body mass index (BMI) in children with obesity, or who were overweight with risks including medical and behavioral risks and family history. CDC funded the following grantees: the Massachusetts State Department of Public Health and Arizona State University. As of July 2019, CORD phase 2 was ongoing. Early care and education centers (23 centers) Collected height and weight for children aged 2-5 years. Trained center staff on health behavior change strategies to use at their centers. Provided centers with large self-serve water containers and cooking kits with child-friendly cooking and serving items. Elementary schools (13 schools) Worked with school nurses and trainees to collect BMI measurements from kindergarteners, third graders, and fifth graders in the El Centro Elementary School District; and kindergarteners, second graders and fifth graders in the Brawley Elementary School District. Provided schools physical activity equipment. Installed water jets and other water containers to provide self-serving access by students. Developed lesson plans promoting sleep for grades kindergarten through sixth grade. Community (three community organizations and three independent restaurants) Provided a water dispenser at two community recreations centers in Brawley and El Centro and at one Boys and Girls Club in Brawley. Developed community gardens at the Boys and Girls Club in Brawley and a recreation center in El Centro. Introduced healthy children’s menu items in three restaurants. The grantee conducted a non- randomized study which sought to determine whether strategies implemented in both public health sectors and primary care sectors would be more effective at preventing and controlling childhood obesity when compared with strategies implemented in public health sectors only, primary care sectors only, or when strategies were not implemented. Community health clinics (three clinic sites) Modified the clinics’ electronic health record systems to improve health care provider screening and treatment of childhood obesity including through the use of alerts and prompts. To facilitate the adoption of the system changes, a patient care coordinator was hired to work across the participating clinics. Hired community health workers and a community health worker coordinator to administer the Family Wellness Program, a 12-month program that delivered wellness and physical activity workshops, motivational interviewing, and newsletters. Early care and education centers (nine centers) Trained mentors to provide support to staff to implement evidence-based programs on nutrition and physical activity. Schools and after school programs (six schools and 17 after school programs) Provided evidence-based health education curricula and training to teachers to encourage student learning about nutrition and physical activity. Implemented a nutrition curriculum for after-school program staff to use with children aged 5 to12 years. Community Implemented a communications campaign, including text messaging, small billboards, transit ads, and handouts, to spread the demonstration project’s brand and to change community norms and practices in physical activity and healthy eating. Special Supplemental Nutrition Program for Women, Infants and Children (one program in each community) Collaborated with the Special Supplemental Nutrition Program for Women, Infants and Children to implement intervention activities including training nutritionists and nutrition assistants in best practices on assessment and counseling for childhood obesity prevention and developing an obesity counseling toolkit for providers. Health centers (two centers) Modified existing electronic health records to deploy a computerized, point-of-care decision support alert at the time of a well-child care visit for a child who is overweight or has obesity. The alert prompted clinicians to document weight status, nutrition and physical activity counseling, and place referral to the on-site healthy weight clinic for weight management support. Implemented a healthy weight clinic in each participating health centers. Each healthy weight clinic was staffed with a physician, a nutritionist, and a community health worker who met with each patient and family. Patients participating in the healthy weight clinics engaged in dietary and physical activity assessment, goal setting, and were connected to community resources to support healthy lifestyles. Early care and education centers (28 centers) Provided classroom materials on nutrition and gardening and bilingual parent tips sheets on nutrition, activity, and screen time. Provided physical activity equipment to participating centers. Schools (40 schools) Trained school staff on a nutrition and physical education classroom curricula. Sent text messages in English or Spanish to participants once a week that emphasized program concepts and linked families to resources. Community Provided training sessions to teach community health workers, teachers, parents, physicians, and others stakeholders about advocacy and the implementation of environmental changes for healthy eating and active living. Health care clinics (11 clinics) Provided BMI screening for children who are overweight or have obesity, which included decision supports to integrate guidelines for the appropriate clinical screening, evaluation and treatment into day-to-day practice. Modified electronic health records to identify children who were overweight and had obesity, provide prompts for treatment, and provide clinicians with access to referral information for weight management. The Texas demonstration project implemented and evaluated a primary and secondary obesity prevention program. In the primary prevention intervention, the grantee collected data on risk factors and the utilization of health care services and community programs. This intervention was focused on the entire community, with the goal of preventing the development of obesity. The secondary prevention program consisted of a randomized control trial, targeted to children who were already overweight or had obesity. Children and their families were randomly assigned to either a community centered or a primary care centered weight management program. The Arizona demonstration project is led by Arizona State University. The purpose of the project is to implement an adapted program in three pediatric primary care clinics located in Maricopa County, Arizona. These clinics serve a minority patient demographic of about 60 to 65 percent, of which the largest groups are Mexican American and American Indian. Program adaptation: Adaptation began by assessing the needs and capacity of a primary care organization and the families they serve. The program was then pilot-tested in a general pediatrics clinic and a clinic for children with advanced obesity to determine feasible delivery modifications as well as enhanced content for obesity management and prevention. During and at the end of the pilot trial, feedback was solicited from stakeholders and families. A draft of the adapted version of the program was then developed, additional feedback was sought from experts and stakeholders and a second pilot-testing phase was completed. Feedback was again collected from families who received the intervention and from stakeholders who participated in the pilot. The intervention protocol and content were further refined to implement in the three pediatric primary care clinics. Effectiveness study: Participants were identified during clinic well- and sick-child visits and through queries of electronic health records. After completing a family health assessment, families were randomly assigned to the adapted program or services as usual. Participating families completed routine assessments about family health behaviors, child health behaviors, family well-being and support, and other topics. Following the assessments, feedback sessions were initiated. The first feedback session focused on understanding (a) the caregivers’ perception of their needs; (b) their child’s health, adjustment, and behavior; and (c) the caregivers’ motivation to change parenting and family management practices in support of health behavior change. Additionally, over a 6-month period, families participated in eight to 16 parenting sessions tailored to the specific needs identified in the family health routine assessment and focused on a specific behavior change goal, such as setting limits on snacking between family meals or monitoring children’s sedentary and physical activity time. In the second and third feedback sessions, the coordinator began by checking in with the family about their progress, discussing barriers they experienced, and exploring the ways that the previous feedback and parenting sessions were helpful for them in catalyzing and supporting healthy lifestyle behavior change. Additionally, coordinators provided families with referrals to existing resources in the community. In weeks where a face-to-face session was not scheduled or did not occur, the coordinator conducted a 15- to 30-minute phone-based coaching session. The purpose was to maintain contact with the family and help problem- solve challenges, reinforce positive achievements, and continually address motivation to change and barriers to engagement. Primary care screening and assessment of child BMI: Children were referred to the demonstration project by their primary care provider during a health care visit where a height and weight was obtained and it was determined that the child was overweight or had obesity. After the referral was made, parents were mailed an introductory letter and fact sheet by the study team. A bilingual study coordinator contacted parents by phone and explained that the research study was to examine strategies to improve the care that is provided for children who require weight management. The coordinator obtained verbal informed consent from the parent and administered a 20-minute baseline survey. Child assigned to intervention: After the parent completed the survey, the child was randomly assigned to a healthy weight clinic in one of the two federally qualified health centers or to the weight management program delivered at one of the two YMCAs. Each of the two intervention groups received an intensive 6- month intervention, followed by a 6-month maintenance period that delivered 30 or more hours of contact time over one year. In addition, children in both intervention groups were exposed to quality of care improvements in their federally qualified health centers, which included primary care provider weight management training and text messages to participating families for self-guided behavior change support. Healthy weight clinic: This intervention was clinic-based and used a multidisciplinary team, including, a pediatrician, community health worker, dietician, and access to behavioral/mental health providers, as needed. The team was trained to deliver motivational interviewing and behavioral modification techniques to engage families in setting and following through on healthy eating and activity goals. Visits alternated between group visits with other children and families in the program and individual visits for the first 6 months and individual visits in the second 6 months. During the first 6 months of the intervention, the community health worker or dietitian made bi-weekly phone calls to the family on weeks they did not have an in person visit. During the second 6 months, they provided once-monthly calls. YMCA weight management program: This intervention was a community- based intervention where staff at two local YMCAs were trained to implement the program. Two YMCA group leaders provided support, education and activities during sessions, which included goal setting and action planning, a parent discussion, and 60 minutes of physical activity for the children. The program was delivered over 12 months, which included 16 weekly sessions, followed by four sessions delivered every other week and concluded with five monthly sessions. In addition to the contact named above, Shannon Slawter Legeer (Assistant Director), Deitra H. Lee (Analyst-in-Charge), and Kristen M. Pinnock made key contributions to this report. Also contributing were Krister Friday, Richard Lipinski, Laurie Pachter, Ethiene Salgado- Rodriguez, and Emily Wilson Schwark.", "summary": "Childhood obesity affects nearly 14 million children aged 2 to 19 years in the United States. Children in low-income families are disproportionately affected, with about 1 in 5 having obesity. Studies suggest that children with obesity are likely to become adults who are overweight or have obesity, which can contribute to poorer health and higher health care expenditures. CDC was designated as the agency to design and manage the project and has awarded grants in three separate phases. GAO was asked to examine the CORD Project, including what has been learned regarding strategies to reduce childhood obesity. In this report, GAO describes 1) the extent to which CDC changed the design of the CORD Project between grant phases, 2) the results of the CORD Project and factors that have affected implementation, and 3) efforts by CDC and others to disseminate results and lessons learned. To conduct this work, GAO reviewed planning and grant documentation for the three CORD phases, published articles about the design of CORD phase 1 and 2, and documentation describing the results of CORD phase 1. GAO also interviewed CDC officials, CORD phase 1 and 2 grantees, and officials from other HHS agencies involved in the design of the CORD Project. HHS provided technical comments on a draft of this report, which GAO incorporated as appropriate. The Centers for Disease Control and Prevention (CDC) has made four key changes to the design of the Childhood Obesity Research Demonstration (CORD) Project between each of the three grant phases. Established by law in 2009, the project provides research grants to develop and implement strategies to reduce obesity among low-income children. One of CDC's design changes, for example, was to modify the scope of the project (i.e., type of strategies implemented by grantees). After CORD phase 1, CDC officials shifted the scope from prevention—through the implementation of strategies in community settings, such as schools, and in health care settings—to the treatment of children who were overweight or had obesity. According to CDC officials, the agency made this change due to the shorter time frame for implementing CORD phase 2 and in response to existing national recommendations related to childhood obesity. CDC also changed the purpose of the project's study design prior to phase 3. Whereas CORD phases 1 and 2 were intended to build knowledge and evidence of effective strategies, CDC modified CORD phase 3 to focus on translating effective strategies into routine use by converting them into a package of materials that others could replicate. To evaluate the effectiveness of CORD phase 1—the only phase that is complete—CDC awarded a grant to an independent entity to aggregate results across the three grantees, and each grantee conducted their own evaluation. The evaluation center and the grantees reported some improvements in children who received CORD 1 strategies. For example, the evaluation center reported small but positive changes in outcomes measured, which included body mass index and fruit and vegetable consumption. These improvements were most often observed among children who received primary care strategies, such as individualized counseling, and children who participated in public health strategies, such as an evidence-based nutritional program, in addition to the primary care strategies. CDC and grantees identified several factors during the first two phases that affected the ability to implement strategies to reduce obesity among low-income children. For example, grantees noted that the preexistence of programs and policies that promoted healthy behaviors positively affected their implementation of CORD strategies. CDC officials identified the turnover of principals and other school or clinic staff as negatively affecting the implementation and suggested that future researchers incorporate staff retraining costs into their strategies as a way to help mitigate this challenge. CDC has taken steps to share CORD design materials and results through published literature, websites, and conferences. It has also coordinated with other Department of Health and Human Services (HHS) offices and agencies to promote the wider adoption of CORD strategies in low-income communities. For example, CDC has collaborated with an office in HHS to fund a project to increase the use of a specific weight management program used in CORD phase 1.", "document_type": "gao"}
{"report": "The Secret Service has processes to vet individuals differently depending on the person’s expected proximity to the President, using a combination of physical screening and background checks, as illustrated in figure 1. According to Secret Service officials, physical screening includes the use of equipment such as wands and magnetometers to secure the property. Background checks assess in part whether an individual has a history of criminal activity. In some cases, enhanced background checks identify other types of threats. The Secret Service develops and executes a security plan to ensure that the outer, middle, and inner layers at the travel location are secure. Officials from the Secret Service confirmed that agency policies aim to provide comprehensive planning guidance for their agents’ activities but are not meant to be all inclusive. Outer Layer: The Secret Service uses physical screening measures to establish a layer of security around the Mar-a-Lago property. According to Secret Service officials, state and local law enforcement and the U.S. Coast Guard may monitor entry onto the property or perform visual checks of individuals entering the property and surrounding waterways. Depending on where the President is, guests may be required to pass through a physical security checkpoint that employs magnetometers, wands, and visual checks to assess physical threats. Middle Layer: Officials from the Secret Service said that they use physical screening measures for individuals and any rooms that the President may access during his visit. Officials told us that if they are notified of the President’s planned arrival to a specific room, they will secure that room. Inner Layer: In advance of the President’s arrival, the Secret Service has a process requiring vetting of individuals who are expected to be within close proximity to the President for a planned purpose or in certain secure areas. According to Secret Service officials, individuals who need access to secure areas but who are not expected to interact directly with the President, such as wait staff and other workers, are to undergo a background check in addition to physical screening. Individuals who are expected to meet the President are to undergo a background check and an enhanced background check. Officials from the Secret Service said that they are responsible for collecting the findings from these checks and making recommendations to the Executive Office of the President on whether individuals with derogatory findings should be allowed to access a space. According to officials from the Secret Service, staff at Mar-a-Lago routinely undergo background checks. In order to conduct the background check, the Secret Service is to use personally identifiable information for each individual, and those Individuals’ names may be checked against indexes maintained by the Secret Service and other federal, state, and local law enforcement organizations. The Secret Service’s guidance notes that submission of the requested information to run a background check is based on individuals voluntarily providing the needed information. In order to conduct an enhanced background check, the Secret Service collaborates with the Federal Bureau of Investigation and other federal agency partners. According to officials from the Secret Service, the Executive Office of the President is responsible for identifying individuals who are expected to meet with the President and providing the Secret Service with the names and the personally identifiable information needed to complete these checks. According to Secret Service guidance, White House staff is expected to submit all names to the Secret Service at least 72 hours in advance of the President’s arrival. Advance agents are also responsible for setting deadlines for completing background checks. Officials from the Secret Service said that, based on the information received from these checks, the Secret Service will make a recommendation to the Executive Office of the President on whether an individual should be granted access to the President. According to these officials, the Executive Office of the President ultimately determines whether or not an individual will have access. However, the Secret Service is responsible for ensuring that the area is safe and that the individual is physically screened. DOD’s White House Communications Agency and the Secret Service each have specific responsibilities for establishing secure communications and secure areas for handling classified information when the President travels to domestic locations, such as Mar-a-Lago. DOD’s White House Communications Agency: This organization is an information technology unit within DOD that supports the President and his staff during presidential trips. This organization’s mission is to provide information services to the President, Vice President, National Security Staff, Secret Service, and others when directed. According to agency guidance and officials, the White House Communications Agency is responsible for installing secure communications equipment that enables the exchange of classified information in areas that may be used by these entities. Secret Service: According to officials from the Secret Service, they send an advance team that coordinates with the White House Communications Agency to set up a conference center for the President where classified information may be exchanged, among other things. These officials stated that they provide security at the entrance of this conference center and perform security sweeps to ensure that it is safe and secure. DOD and the Secret Service coordinate to establish and secure several areas that are available for handling classified information when the President travels to locations such as Mar-a-Lago, as shown in figure 2. These areas include a conference center, spaces used by staff of the National Security Council and Executive Office of the President, and presidential transportation vehicles. Details associated with these areas and facilities are sensitive and have been omitted from this report. The Secret Service and DOD are subject to regulations governing reimbursements to employees for official travel. Processes exist to review these travel-related expenses when personnel from these agencies travel. These processes are the same when personnel accompany the President to Mar-a-Lago. Two regulations implement statutory requirements and executive branch policies for travel, allowing agencies to pay for or reimburse their employees’ per diem expenses (lodging, meals and incidental expenses) and other travel-related expenses: The Federal Travel Regulation (FTR), issued by the General Services Administration applies to the Secret Service’s personnel. The Joint Travel Regulations (JTR), issued by the Department of Defense apply to DOD personnel. Both regulations allow agencies to pay for employees’ daily expenses when they are traveling within the continental United States, based on allowances set by the General Services Administration for the applicable location and date (per diem rates) or the actual expense of travel. Under the Federal Travel Regulation, the maximum amount that a civilian employee may be reimbursed is 300 percent of the applicable per diem rate. The Joint Travel Regulations allow uniformed service members to be reimbursed up to 300 percent of the per diem rate when they are traveling in the continental United States, but they can be reimbursed more than 300 percent of the per diem rate for lodging when traveling outside the continental United States. Officials from the Secret Service stated that they apply the same cost oversight processes for all presidential travel. Expenses for lodging and operational space are centrally billed to the agency, and employee meals and incidental expenses are reimbursed to the traveler. In accordance with policy, the Secret Service tries to acquire lodging at the General Services Administration’s per diem lodging rate and must submit a waiver request for any room that exceeds this designated rate by any amount. The Secret Service field office closest to the travel destination is responsible for arranging for these spaces, negotiating rates, and if necessary submitting a waiver request to officials in the Secret Service’s Logistics Resource Center. The Logistics Resource Center is to review the waiver request, determine whether a more cost effective method exists to meet the need, and approve or reject the request. In some cases, the Secret Service may not be able to acquire rooms at the per diem lodging rate, or agents may need rooms for operational purposes that exceed 300 percent of the per diem rate, which is more than is allowed for lodging under the Federal Travel Regulation. For example, the Secret Service may use a room for operational purposes or reserve rooms adjacent to the President to better protect him. In addition, to meet operational demands, the Secret Service may require a certain number of agents to stay at the hotel in which the President is staying, so that they are within a certain proximity of the President at all times. Furthermore, officials from the Secret Service said that members of the Secret Service canine teams must stay at hotels that allow animals, and rooms at these hotels may exceed the General Services Administration lodging rate. The authorities the Secret Service has relied on to pay for hotel rooms needed to meet its operational requirements do not limit how much the agency can pay. Further, Congress passed a law in May of 2017 excepting the Secret Service from regulatory caps on room rentals, regardless of room purpose. Nevertheless, consistent with the Secret Service waiver process, personnel are still required to submit waiver requests for operational spaces to justify the need to book rooms during the President’s trips to Mar-a-Lago at prices higher than the General Service Administration lodging rates. We confirmed that a blanket waiver request was submitted and approved for all rooms at Mar-a-Lago that exceeded the General Services Administration per diem lodging rate during the President’s trips to Mar-a-Lago that are covered by this review. Additionally, we reviewed Secret Service documentation and confirmed that Secret Service personnel did not exceed the 300 percent threshold for lodging. For meals and incidental expenses, the Secret Service’s employees who are on official duty are to submit a claim for reimbursement electronically or by paper and receipts, as applicable, at the conclusion of the trip. Approving officials are to approve (or deny) expenses, and the Secret Service’s Financial Management Division authorizes reimbursement for approved travel. DOD personnel use the same processes for travel to Mar-a-Lago as they do for other Presidential trips to oversee costs for lodging, meals, and incidental expenses. According to officials from DOD’s Defense Travel Management Office, their office establishes travel policy that applies to the four organizations that travel in support of the President’s trips. DOD personnel use the Defense Travel System to submit travel documents, including vouchers and receipts, as applicable. According to officials, lodging may be booked and reimbursed on an individual basis or centrally billed if a block of rooms is needed over the same period. Meals and incidental expenses are reimbursed to the traveler. Like Secret Service’s personnel, DOD personnel must obtain approval from an authorizing official prior to the trip to exceed the General Services Administration per diem lodging rate, consistent with the Federal Travel Regulation and Joint Travel Regulations. According to officials from the Defense Travel Management Office, DOD typically would not reimburse expenses above the approved lodging rate if lodging at the approved rate was available within the region. However, officials from the Defense Finance and Accounting Service indicated that presidential trips may require such a deviation. These approvals are to be tracked in the Defense Travel System. The White House Military Office, which includes the White House Communications Agency, also sends personnel with the President when he travels. White House Communications Agency officials told us that its lodging and operational space for these personnel are typically coordinated by the White House Travel Office and that DOD personnel pay for the associated costs and seek reimbursement from DOD after the trip is complete. According to officials from the White House Communications Agency, some personnel are required to remain at, or near, the Mar-a-Lago property. If they are not required to stay at or near the property, they will try to obtain lodging at hotels in the area at the General Services Administration’s per diem rate for lodging. DOD officials told us that according to the Joint Travel Regulations, DOD is not authorized to pay or reimburse daily expenses above the 300 percent ceiling. In connection with the President’s travel to Mar-a-Lago between February 3, 2017 and March 5, 2017, DOD personnel exceeded the General Services Administration per diem rate but did not exceed the 300 percent threshold. According to officials from the Defense Travel Management Office, operational space used for official business is governed by the Federal Acquisition Regulation. White House Communications Agency officials told us that they have generally used space near the Mar-a-Lago property but leased property, effective September 2017, near Mar-a-Lago to reduce the cost of supporting the President’s trips to the property. Treasury has regular processes for receiving payments designated as gifts to the United States and gifts to reduce the public debt. Treasury officials stated that any payments received from The Trump Organization or the President that are designated as gifts would be handled using these processes. Under federal law, Treasury may receive general gifts to the U.S. Government and may also receive gifts to reduce the public debt. Treasury has developed processes to accept these types of payments, as shown in figure 3. Treasury officials said there are three accounts available to receive payments as gifts—a general gift account, a general fund receipts account, and an account for gifts to reduce the public debt. Any of these accounts could receive payments designated as gifts by the President or The Trump Organization. Treasury officials told us they would deposit such payments into the account for gifts to the U.S. Government unless the payment source specified that the funds should be used to reduce the public debt. Treasury received one payment from The Trump Organization, for $151,470 that was submitted through Treasury’s processes on February 22, 2018. In May 2017, The Trump Organization issued a policy addressing profits generated from foreign government patronage at its businesses. The Trump Organization’s policy states that it will make a single lump-sum payment annually after the end of its fiscal year, which ends on December 31st. We did not identify any other payments that Treasury received from The Trump Organization or the President between January 21, 2017, and August 1, 2018. In September 2018, an attorney for The Trump Organization confirmed that the organization had not made any payments since February 22, 2018. We provided copies of this draft report to DOD, DHS, the Department of Justice, the General Services Administration, the Department of Treasury, and the Executive Office of the President for comment. We also provided a section to the Trump Organization for comment. DHS provided written comments, which are reprinted in their entirety in appendix I. DHS, DOD, the Department of Treasury and the Department of Justice also provided technical comments, which we incorporated into this report as appropriate. The Executive Office of the President and the Trump Organization provided no comments. As agreed with your offices, unless you publicly release this report earlier, we will not issue the report until 30 days from the report date. At that time, we will also provide copies to the Secretary of Defense, the Director of the Secret Service, the Secretary of Homeland Security, the Attorney General, the Director of the Federal Bureau of Investigation, the Administrator of the General Services Administration, and the Secretary of the Treasury. In addition, this report will be available at no charge on the GAO website at www.gao.gov. If you or your staff have any questions about this report, please contact Joseph (Joe) Kirschbaum at (202) 512-9971 or at KirschbaumJ@gao.gov or Diana Maurer at (202) 512-9627 or at MaurerD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in Appendix II. In addition to the contacts named above, Gina R. Hoffman, Assistant Director; Joseph P. Cruz, Assistant Director; Tracy Barnes, Nicholas Benne, Jennifer Kamara, Joanne Landesman, Amie Lesser, Thomas Lombardi, Carol Petersen, Michael Silver, Janet Temko-Blinder, Christopher Turner, Kayli Westling, and Alex Winograd made key contributions to this report.", "summary": "The President has made numerous trips to the Mar-a-Lago property in Palm Beach, Florida, during which he met with foreign leaders and conducted presidential activities. GAO was asked to review the establishment of secure areas for use by the President at Mar-a-Lago. This report provides information on, among other things, (1) vetting of individuals expected to be near the President; (2) efforts to establish secure areas for handling classified information; and (3) regulations and processes for agency expenditures on employees who travel with the President. This is a public version of a sensitive report that GAO issued in October 2018. Information that the Secret Service and DOD deemed sensitive has been omitted. GAO analyzed laws, regulations, policies, and procedures; reviewed agreements between federal agencies and trip after-action reports; and interviewed DOD and Secret Service officials. GAO also reviewed vouchers from the four presidential trips to Mar-a-Lago from February 3, 2017 through March 5, 2017.GAO also reviewed documentation and descriptions of specific security practices with DOD and Secret Service officials. The Executive Office of the President has not responded to requests regarding its role in assisting DOD and the Secret Service in carrying out their responsibilities. The U.S. Secret Service (Secret Service) vets individuals differently depending on the person's expected proximity to the President when he travels, including during his visits to Mar-a-Lago. According to Secret Service officials, vetting may include using physical screening (measures to detect physical threats to the president and secure the property) and background checks intended to identify individuals who have prior criminal activity or present other types of threats. Individuals at Mar-a-Lago who are not expected to meet with the President or enter spaces the President may visit pass through an outer layer of security consisting of physical screening checkpoints surrounding the property. The Secret Service physically screens all individuals who will access areas where the President will be present, such as a dining room. According to Secret Service officials, individuals who have a meeting with the President generally undergo both physical screening and enhanced background checks. The Department of Defense (DOD) and the Secret Service coordinate to establish and secure several areas that are suitable for handling classified information when the President travels to Mar-a-Lago. These areas include a conference center, spaces used by staff of the National Security Council and the Executive Office of the President, and presidential transportation vehicles. Details associated with these areas and facilities are sensitive and have been omitted from this report. The Secret Service and DOD are subject to regulations that govern the reimbursement of employees for official travel expenses. Both organizations have processes to review these travel-related expenses when their personnel travel with the President and try to acquire lodging at the General Services Administration's per diem lodging rate. When the Secret Service is not able to acquire rooms at the per diem lodging rate, including when it needs rooms for operational purposes that exceed 300 percent of the per diem rate (a threshold set by the General Services Administration), employees must submit a waiver request. DOD personnel must also obtain approval when costs exceed the General Services Administration's lodging rate. Our review of DOD vouchers and Secret Service documentation confirmed that personnel did not exceed the 300 percent threshold for lodging during the Mar-a-Lago trips examined in this review. We assessed the costs of Presidential travel in a separate report.", "document_type": "gao"}
{"report": "Prior to January 1, 2018, TRICARE provided benefits through three main plan options for its non-Medicare-eligible beneficiary population— TRICARE Prime, Standard, and Extra. These options varied by enrollment requirements, choices in civilian and military treatment facility providers, and the amount beneficiaries must contribute toward the cost of their care. (See table 1.) The NDAA for Fiscal Year 2017 terminated the TRICARE Standard and Extra plans beginning on January 1, 2018, and introduced TRICARE Select. Beneficiaries who had used the TRICARE Standard and Extra plans as of December 31, 2017 were automatically enrolled in TRICARE Select for the first year of the new plan, but as of January 1, 2019, beneficiaries were required to actively enroll in TRICARE Select. DOD uses two regional managed care support contractors to develop networks of civilian providers to serve all TRICARE beneficiaries. Within the regions, contractors are required to develop these networks of providers in areas called Prime Service Areas (PSA), which are geographic areas usually within an approximate 40-mile radius of a military inpatient treatment facility, as well as in areas outside of PSA locations, or non-PSAs. To develop the networks, the contractors enter into contracts with some providers—referred to as network providers—to treat TRICARE patients at an agreed upon reimbursement rate. Beneficiaries can also receive care from certified nonnetwork providers. However, beneficiaries visiting a nonnetwork provider may have to pay for the care at the time of the visit and later file a claim for reimbursement, whereas beneficiaries visiting a network provider are only responsible for paying a copayment or cost-sharing amount. The NDAA for Fiscal Year 2017 also mandated that DOD ensure at least 85 percent of the TRICARE Select beneficiary population be covered by the TRICARE network of providers by January 1, 2018, and that DOD determine access standards and ensure the program meets or exceeds access standards of “high-performing health care systems in the United States” for health care appointments. DOD has contracted both of these efforts to the two contractors. Non-Prime beneficiaries’ ratings of TRICARE were generally unchanged during the first year following the transition from TRICARE Standard and Extra to TRICARE Select. Specifically, there was no statistically significant change from the 2017/2018 surveys to the 2019 survey in the percent of beneficiaries who positively rated their health care and their health plans—defined as giving responses of 8 or higher (out of 10) on each survey question. In the 2019 survey, 80 percent of beneficiaries rated their health care positively, and 67 percent rated the TRICARE health plan positively. (See fig. 1.) Non-Prime TRICARE beneficiaries also rated three different types of providers and of the three, ratings of primary care providers decreased from the 2017/2018 to 2019 surveys. The percent of beneficiaries who positively rated their primary care providers decreased from 85 to 80 percent in the 2019 survey. We found no statistically significant differences from the 2017/2018 surveys to the 2019 survey in beneficiaries’ positive ratings of specialty care and mental health care providers, with 83 and 73 percent of beneficiaries reporting positive ratings of their specialty care and mental health care providers, respectively in 2019. (See fig. 2.) In the first year of TRICARE Select, a higher percentage of non-Prime beneficiaries reported experiencing problems finding civilian health care providers who accepted TRICARE than before the transition, particularly for specialty care. We found there was a statistically significant increase in the percentage of beneficiaries who reported problems finding a provider that would accept TRICARE from 27 to 32 percent from the 2017/2018 surveys to the 2019 survey. In particular, there was a statistically significant increase in the percentage of beneficiaries who reported problems finding a specialty care provider in the 2019 survey (24 percent) compared to the 2017/2018 surveys (18 percent). The percent of beneficiaries who reported problems accessing primary care or mental health care remained statistically unchanged with 26 and 31 percent reporting problems, respectively, in the 2019 survey. (See fig. 3.) We also found that a higher percentage of beneficiaries located in PSAs reported experiencing problems finding providers, whereas there was no change for beneficiaries located in non-PSA areas. Specifically, from the 2017/2018 surveys to the 2019 survey, there was a statistically significant increase in the percentage of beneficiaries located in PSAs who reported problems finding any type of civilian provider (27 to 34 percent), primary care providers (21 to 29 percent), and specialty care providers (18 to 25 percent). There was no statistically significant change among beneficiaries in PSAs reporting problems finding mental health care providers or among beneficiaries in non-PSAs for any provider types. (See fig. 4.) To help beneficiaries find providers that accept TRICARE patients, managed care support contractors develop networks of providers in PSAs and some non-PSA locations. Each month, these contractors report to DOD the percent of Select beneficiaries who were covered by the TRICARE network of providers, according to contractor-developed measures of adequate access to care. Although nearly one-third of beneficiaries reported experiencing problems finding a civilian provider, DOD officials told us that nearly 100 percent of beneficiaries in the East have had adequate access to a network provider since January 1, 2018, exceeding the 85 percent requirement. For the West, DOD officials said the contractor reported that more than 85 percent of beneficiaries had adequate access to a network provider as of August 2018. According to DOD officials, the two contractors used different methods to ensure adequate access to a network provider: In the East region, the contractor decided to develop networks of civilian providers in the entire region. In the West region, the spread-out geography of the region made it difficult to develop networks of civilian providers throughout the entire region. Therefore, the contractor used mapping software to determine areas within non-PSAs which had large populations of TRICARE Select beneficiaries. As a result, the contractor identified 12 areas in the West region—which it called Select Areas—to develop additional networks of civilian providers in order to meet the requirement. Contractors also provide resources to beneficiaries to help them identify providers that accept TRICARE patients. Contractors maintain lists of network and other TRICARE-certified providers that accept TRICARE patients, and monitor and report on the accuracy of these lists to DOD monthly. However, contractor representatives noted that providers can decide to accept or not accept TRICARE patients at any time, and these changes are not always reflected in the lists. Contractor representatives explained that if a beneficiary cannot find a provider to accept TRICARE for needed care, the beneficiary can submit a complaint to the contractor. Contractor representatives said that they can address beneficiary complaints by attempting to identify and certify new providers, but noted that some subspecialty providers are not available in all areas. Representatives from one contractor told us that they have identified alternative sources of care when providers are not available. For example, when no civilian psychiatrists were accepting new patients in a remote area, the contractor offered beneficiaries telehealth services from a military treatment facility. There was no statistically significant change after the transition to TRICARE Select in the percent of non-Prime beneficiaries who reported being able to get an appointment as soon as they needed. In the 2019 survey, 88 percent of beneficiaries reported that they could usually or always obtain an appointment for primary care as soon as they needed and 86 percent reported being able to do so for specialty care. (See fig. 5.) Similarly, about 65 percent of beneficiaries reported waiting a week or less between scheduling an appointment for non-urgent care and meeting with their doctor in the 2019 survey, and 83 percent of beneficiaries reported waiting 2 weeks or less. There was no change in the percent of civilian providers nationwide who reported accepting new TRICARE patients if they were also accepting other new patients after the transition to TRICARE Select. Across all provider types, 67 percent of providers in the 2019 survey reported accepting new TRICARE patients if they were also accepting other new patients; this percentage was not statistically significantly different from the 2017/2018 surveys. There was also no statistically significant change among specific provider types—in the 2019 survey, 47 percent of mental health care providers and about 90 percent of primary care and specialty care providers reported accepting new TRICARE patients if they were accepting any new patients. When we analyzed provider responses by network status and specialty, the surveys indicated a decrease in the percentage of network mental health providers who were accepting new TRICARE patients if they were also accepting other new patients. The percent of these network mental health providers decreased a statistically significant amount from 91 percent in the 2017/2018 surveys to 84 percent in the 2019 survey. (See fig. 6.) However, there was no change in the overall percentage of all network or all nonnetwork providers that were accepting new TRICARE patients if they were also accepting any new patients—93 percent of network and 58 percent of nonnetwork providers in the 2019 survey. When we analyzed provider responses by location, we found that provider acceptance of new TRICARE patients decreased by a statistically significant amount in non-PSAs. Specifically, a lower percentage of providers located in non-PSAs reported accepting new TRICARE patients if they were accepting other new patients, decreasing from 72 percent in the 2017/2018 surveys to 68 percent in the 2019 survey. This percentage did not significantly change for providers in PSAs. (See fig. 7.) There were few changes in the reasons providers gave for not accepting TRICARE patients in the first year of TRICARE Select. (See Table 2 for a list of reasons providers offered in the 2019 survey.) Of 14 categories of reasons that providers gave, there was a statistically significant change in two categories between the 2017/2018 surveys to the 2019 survey. Specifically, the percentage of providers who listed reimbursement as a reason for not accepting new TRICARE patients declined from 11 percent to 8 percent, and the percentage of providers who listed that the doctor was not available or too busy increased from 4 percent to 8 percent. DOD provided technical comments on a draft of this report, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. James Cosgrove, (202) 512-7114 or cosgrovej@gao.gov. In addition to the contact named above, individuals making key contributions to this report include Tom Conahan (Assistant Director), A. Elizabeth Dobrenz and Jeffrey Mayhew (Analysts-in-Charge), Jennie Apter, Alexander Cattran, Jacquelyn Hamilton, Vikki Porter, and Jeffrey Tamburello. Defense Health Care: TRICARE Surveys Indicate Nonenrolled Beneficiaries’ Access to Care Has Generally Improved, GAO-18-361 (Washington, D.C.: Mar. 29, 2018). Defense Health Care: More-Specific Guidance Needed for Assessing Nonenrolled TRICARE Beneficiaries’ Access to Care, GAO-14-384 (Washington, D.C.: Apr. 28, 2014). Defense Health Care: TRICARE Multiyear Surveys Indicate Problems with Access to Care for Nonenrolled Beneficiaries, GAO-13-364 (Washington, D.C.: Apr. 2, 2013). Defense Health Care: DOD Lacks Assurance That Selected Reserve Members Are Informed About TRICARE Reserve Select, GAO-11-551 (Washington, D.C.: June 3, 2011). Defense Health Care: Access to Civilian Providers under TRICARE Standard and Extra, GAO-11-500 (Washington, D.C.: June 2, 2011). Defense Health Care: 2008 Access to Care Surveys Indicate Some Problems, but Beneficiary Satisfaction Is Similar to Other Health Plans, GAO-10-402 (Washington, D.C.: Mar. 31, 2010).", "summary": "DOD provided health care to more than 9 million eligible beneficiaries through TRICARE in fiscal year 2018. Most of these beneficiaries were enrolled in TRICARE's managed care plan—TRICARE Prime. However, about 2 million beneficiaries received care primarily from civilian providers through TRICARE's non-Prime options: TRICARE Standard and Extra. Effective January 1, 2018, these two options were eliminated and TRICARE Select was implemented. TRICARE Select has similar benefits for provider choice and obtaining care from civilian providers as TRICARE Standard and Extra, but includes access standards to ensure at least 85 percent of enrollees are covered by TRICARE's network of civilian providers, among other things. The National Defense Authorization Act (NDAA) for Fiscal Year 2008 included a provision for GAO to review results of DOD surveys of non-Prime beneficiaries and civilian providers. Additionally, the NDAA for Fiscal Year 2017 included a provision for GAO to review access to care after implementation of TRICARE Select in 2018. This report addresses both provisions. GAO analyzed DOD's survey results to determine changes after implementation of TRICARE Select in (1) non-Prime beneficiaries' ratings of TRICARE, (2) non-Prime beneficiaries' reported ability to find providers and obtain appointments, and (3) civilian providers' reported acceptance of TRICARE. GAO analyzed the results of the 2017-2019 surveys, and interviewed agency officials and DOD contractors. DOD provided technical comments, which GAO incorporated as appropriate. On January 1, 2018, the Department of Defense (DOD) implemented a new health plan option—TRICARE Select—for beneficiaries who primarily obtain care from civilian providers rather than through TRICARE's managed care plan—TRICARE Prime. DOD surveys indicate few changes in these non-Prime beneficiaries' satisfaction and access to care during the first year following the implementation of TRICARE Select, though GAO cannot directly attribute these differences to implementation due, in part, to other changes in the TRICARE program during the same time frame. Specifically, GAO found the following: There was no change in the percent of beneficiaries reporting positive ratings of their TRICARE health care and health plans—80 percent and 68 percent, respectively—in the first year of TRICARE Select. There was an increase in the percent of beneficiaries reporting problems accessing specialty providers from 18 to 24 percent in the first year of TRICARE Select. However, as the figure shows, there was no statistically significant change in the percent of beneficiaries reporting they received care as soon as needed for primary and specialty care appointments. There was no change in the percent of providers that reported accepting new TRICARE patients if they were also accepting any new patients—about 90 percent of primary care and specialty care providers, and 47 percent of mental health care providers—in the first year of TRICARE Select.", "document_type": "gao"}
{"report": "In 1994, Executive Order 12898 directed each federal agency to develop an environmental justice strategy that identifies and addresses disproportionately high and adverse human health or environmental effects of its programs, policies, and activities on minority populations and low-income populations. Together, the 1994 executive order and the 2011 MOU include eight areas that agencies’ environmental justice efforts should address, as appropriate, such as NEPA implementation and public participation. Working group members have documented their environmental justice strategies using environmental justice strategic plans. We have previously reported on the importance of certain leading practices in developing or updating strategic plans and developing periodic progress reports, including in our October 2011 review of EPA’s environmental justice efforts. We reported that a multi-year strategic plan articulates the fundamental mission of an organization and lays out its long-term general goals for implementing that mission, including resources needed to achieve the goals. To that end, during strategic planning, which should occur at least every 4 years, an agency should review its mission statement, review its strategic goals, align strategic goals and strategies, and align strategic and annual performance goals. In addition, a strategic plan should contain a description of how the goals will be achieved, including human capital, information, and other resources needed. Finally, agencies should develop annual performance plans with annual performance goals—linked to the overall strategic goals—and describe how the goals will be measured to assess progress in achieving them. As one method for assessing such progress, we identified key attributes of successful performance measures, such as having measurable targets. The 1994 executive order also created an interagency working group to coordinate federal environmental justice efforts by serving the following seven functions: Provide guidance to federal agencies on criteria for identifying disproportionately high and adverse human health or environmental effects on minority populations and low-income populations. Coordinate with, provide guidance to, and serve as a clearinghouse for each federal agency as it develops an environmental justice strategy, in order to ensure consistent administration, interpretation, and enforcement of programs, activities, and policies. Assist in coordinating research by, and stimulating cooperation among, EPA; the Department of Health and Human Services (HHS); Department of Housing and Urban Development (HUD); and other agencies conducting certain research, data collection, or analysis. Assist in coordinating data collection. Examine existing data and studies on environmental justice. Hold public meetings. Develop interagency model projects on environmental justice that demonstrate cooperation among federal agencies. After a period of relative inactivity, 16 agencies and CEQ recommitted to collaborating on environmental justice efforts through a revitalized interagency working group when they signed the 2011 MOU. We have previously found that federal agencies have used a variety of mechanisms to implement interagency collaborative efforts, including working groups, and that interagency collaboration mechanisms benefit from key features, which raise issues to consider when implementing such mechanisms. These features include defining and articulating a common outcome; reinforcing agency accountability for collaborative efforts through agency plans and reports; developing mechanisms to monitor, evaluate, and report on results; agreeing on or clarifying roles and responsibilities; including all relevant participants and determining their ability to commit resources; identifying and addressing resource needs; and documenting written guidance and agreements. The 1994 executive order did not create new authorities or programs to carry out federal environmental justice efforts. As a result, federal environmental justice efforts seek to use existing federal laws, programs, and funding to address environmental and health problems that disproportionately burden minority and low-income communities, such as exposure to environmental pollutants. Example of Capacity Building Funded by an EPA Environmental Justice Grant in Spartanburg, South Carolina EPA provided a $20,000 environmental justice grant to a community organization in Spartanburg, South Carolina, in 2000 to support three research projects on the health of residents and former employees at a fertilizer plant and landfill sites. The target area, on the south side of Spartanburg, had a 96 percent African-American population according to EPA’s 2002 IWG Status Report. EPA’s initial $20,000 grant paid for research to help confirm health issues related to nearby hazardous waste sites. According to EPA officials, this initial investment has helped Spartanburg secure investments in the community. As a result, Spartanburg now has community health centers, affordable housing, and a recreation center. Several environmental laws regulate pollutants in the air, water, or soil and generally require a regulated facility to obtain permits from EPA or a state. These laws also authorize the issuance of administrative orders, among other things, to require cleanup of contamination. For example: Under the Clean Air Act, EPA, along with state and local government units and other entities, regulates air emissions of various substances that harm human health. The Clean Water Act regulates discharges of pollutants into waters of the United States, including lakes, streams, and other water bodies. The Resource, Conservation, and Recovery Act prohibits the treatment, storage, and disposal of hazardous waste without a permit. In addition, the Comprehensive Environmental Response, Compensation, and Liability Act authorizes EPA to compel the responsible parties to clean up contaminated sites and also allows EPA to conduct cleanups and then seek reimbursement from the responsible parties. Federal enforcement actions include administrative orders issued by EPA and civil or criminal judicial actions brought by the Department of Justice (DOJ). Under NEPA, federal agencies must evaluate the environmental impacts of their proposed major federal actions using an environmental assessment or a more detailed environmental impact statement, with some exceptions. CEQ is responsible for overseeing federal agencies’ implementation of NEPA. In 1997, the council issued guidance stating that agencies should consider environmental justice issues at several stages of the NEPA process, as appropriate. This guidance provides principles for considering whether particular agency actions raise environmental justice issues, such as looking at the demographic composition of the affected area and seeking public participation. HHS has programs and initiatives that address environmental health issues. Such efforts include the Centers for Disease Control and Prevention’s National Environmental Public Health Tracking Network—a data initiative which brings together health and environmental data from national, state, and city sources—and the Centers for Disease Control and Prevention’s National Report on Human Exposure to Environmental Chemicals—a series of reports that uses biomonitoring to assess the U.S. population’s exposure to environmental chemicals. Title VI of the Civil Rights Act of 1964, as amended, prohibits discrimination based on race, color, or national origin in programs or activities that receive federal financial assistance. To carry out and enforce the provisions of the act, federal agencies have developed programs to receive and investigate allegations of discriminatory actions taken by recipients of federal funding. In addition to these laws and programs, EPA also established a National Environmental Justice Advisory Council (NEJAC) in 1993 to provide advice and recommendations to EPA’s Administrator about issues related to environmental justice. NEJAC provides a forum for diverse perspectives, with representatives from various sectors, including academia, community groups, industry and business, non-governmental and environmental organizations, state and local governments, and tribal governments and indigenous groups. In recent years, NEJAC has issued reports on key environmental justice issues, including one on industrial waterfront areas (ports) and another on water and wastewater infrastructure. Most of the agencies that signed the 2011 MOU have developed environmental justice strategic plans that contain strategic goals, but most have not shown clear progress toward these goals. Specifically, 14 of the 16 agencies have developed environmental strategic plans, and 12 also established strategic goals in these plans, but several agencies have not updated their plans in recent years. In addition, most agencies have not issued annual progress reports or established methods to assess progress. Most of the 16 agencies have developed environmental strategic plans, and most of these plans included strategic goals to help direct the agencies’ environmental justice efforts. As shown in table 1, 14 of the 16 agencies issued environmental justice strategic plans after 2011, when they agreed to develop or update such plans under the 2011 MOU. Of the 14 agencies that developed environmental justice strategic plans, 12 also established strategic goals in these plans, as shown in table 1. Many of the agencies had multiple goals with common themes. For example, eight agencies included goals that involved providing assistance, such as grants, technical assistance, or direct services, to environmental justice communities. Eight agencies also included goals that involved promoting public participation; seven agencies included goals that involved identifying and addressing environmental justice issues; four agencies included goals related to training or educating agency staff on environmental justice; four agencies included goals related to promoting enforcement of Title VI; three agencies included goals related to conducting research on environmental justice issues; and three agencies included goals related to incorporating environmental justice considerations into policies or guidance. Two agencies—the Department of Defense (DOD) and Small Business Administration (SBA)—did not issue environmental strategic plans after 2011 even though by signing the MOU they agreed, as appropriate, to develop or update their environmental justice strategies by early 2012. DOD issued such a plan in 1995, shortly after the executive order was signed but has not updated its plan since. We have previously reported that strategic planning serves as the starting point and foundation for defining what the agency seeks to accomplish, identifying the strategies it will use to achieve desired results, and then determining how well it succeeds in achieving goals and objectives. DOD officials said that the agency has not prioritized environmental justice efforts. By updating its environmental justice strategic plan, DOD would have a foundation for such efforts. SBA has never issued an environmental justice strategic plan. SBA officials said that the agency is uncertain whether it has a role in implementing environmental justice and they were in the process of reviewing whether SBA should continue its membership in the working group. By assessing whether to participate in the 2011 MOU, SBA could clarify its role. Of the 14 agencies that developed environmental justice strategic plans after 2011, six agencies have updated those plans and one has updated its priority areas on its website. The 2011 MOU directs agencies to update their strategic plans periodically, and GAO’s leading practices for strategic planning suggest that strategic plans should be updated every 4 years. Five of the six agencies—the U.S. Department of Agriculture (USDA), Department of the Interior (DOI), DOT, EPA, and General Services Administration (GSA)—issued updated strategic plans in 2016 in response to a request from the working group that all agencies update their strategic plans. The sixth agency, the Department of Energy (DOE), issued an updated strategic plan in 2017. HHS posted a list of “priority areas of focus” for environmental justice for 2015 through 2016 on its website. Agency officials noted that this was less resource-intensive than conducting a full review and update of the strategic plan. The remaining seven agencies—Commerce, Education, DHS, HUD, DOJ, Department of Labor (DOL), and Department of Veterans Affairs (VA)—have not updated their plans since issuing them after 2011. Six of these agencies issued their environmental justice strategic plans in 2012, and one of these agencies, DOJ, issued its revised strategic plan and a companion guidance document in 2014. As a result, as of 2019, these plans are more than 4 years old and may not reflect the agencies’ current approach. Some of these agencies have taken preliminary steps to update their plans, but with the exception of DHS, they do not have a time frame for developing an update according to agency officials. DHS officials stated that the agency was developing an updated environmental justice strategic plan, which is scheduled for formal internal review during calendar year 2019 and for release in 2020. DOJ officials stated that they plan to meet in 2019 to review and discuss possible updates to their strategic plan, but the agency does not intend to update it unless any significant changes have taken place since they reissued it in 2014. According to HUD officials, HUD prepared a draft of an updated environmental justice strategic plan for 2016 through 2020 and posted it online for public comment in November 2016, but the agency has not worked on the draft plan since then. According to agency officials, the draft plan has not been finalized because of staff losses and because HUD leadership prioritized other issues, such as long-term disaster recovery, over environmental justice issues. Officials from Commerce stated that the agency has not updated its environmental justice strategic plan because of the time and resources that this would require. Officials from Education, DOJ, DOL, and VA said that they do not believe it is necessary to update their agency plans because they are continuing to implement their existing plans or because their approach to environmental justice work has not changed since their plans were issued. However, in updating their plans, which are no longer current, the agencies could explain that significant changes were not made. By updating their strategic plans or by reaffirming the validity of their current plans, these agencies (Commerce, Education, DHS, HUD, DOJ, DOL, VA) would have a current plan to guide their environmental justice activities as they committed to do in the 2011 MOU. While 12 agencies have developed an environmental justice strategic plan with strategic goals, most of them have not shown clear progress toward achieving their environmental justice goals and the purpose of the executive order. Specifically, the agencies have not comprehensively assessed how environmental justice fits with their overall missions or their progress toward the implementation of their strategic goals by issuing annual progress reports or by establishing methods to gauge their progress, such as performance measures. Furthermore, officials from most agencies said that they are unable to determine how much progress they have made toward achieving the major requirement from the executive order because they do not have a way to assess progress. Of the 14 agencies that developed environmental justice strategic plans after 2011, we found that seven of the agencies—Commerce, DHS, DOE, DOL, EPA, GSA, and HUD—assessed and discussed how their environmental justice efforts aligned with their overall missions. For example, HUD’s environmental justice strategic plan contains a section that describes HUD’s mission to create strong, sustainable, inclusive communities and quality, affordable homes for all. The section then discusses its overall strategic goals and their relationship to environmental justice. For example, HUD’s goal to build inclusive and sustainable communities free from discrimination includes a subgoal to promote energy-efficient buildings and location-efficient communities that are healthy, affordable, and diverse. Similarly, Commerce includes a section in its environmental justice strategic plan entitled “Relationship of Environmental Justice to Agency Mission and Agency Strategic Plan Goals or Objectives.” Among the agency-wide goals that support environmental justice, Commerce describes the National Oceanic and Atmospheric Administration’s (NOAA) efforts to manage fisheries, coastal habitats and species, and protected areas, and to provide information and warnings about weather conditions to the nation, including vulnerable populations. In our review of the 14 agencies’ environmental justice strategic plans, we found that seven of these plans did not clearly show how the agencies assessed alignment between the agencies’ environmental justice plans and overall mission, although the 1994 executive order directed each agency to make achieving environmental justice part of its mission by identifying and addressing, as appropriate, disproportionately high and adverse human health or environmental effects of its programs, policies, and activities on minority populations and low-income populations. In addition, EPA officials questioned how some environmental justice strategic plans from agencies related to their agency’s core missions and stated that to be effective, environmental justice should be considered throughout agencies’ missions. Our previous work found that effective strategic plans include, among other things, agency missions and long-term goals, and that to encourage the use of performance information, agency-wide goals and measures should align. Specifically, we have previously found that an agency’s program goals should flow from its mission statement and that its strategic goals—those that explain what results are expected and when they should be achieved—should also grow out of the mission statement. Although half of the agencies’ environmental justice strategic plans did not clearly show that their agencies assessed their connection to their overall mission, officials from DOI, DOJ, USDA, and VA said that they considered their agencies’ overall strategic plan’s mission and goals when they developed their environmental justice strategic plans. HHS officials commented that although HHS’s overall strategic plan is at a very high level, some elements within its environmental justice strategic plan, such as research, align with its overall strategic plan. The remaining agencies did not explain whether they had considered their agencies’ overall mission and goals when developing their environmental justice strategic plans. The 1994 executive order requires that each federal agency makes achieving environmental justice part of its mission and requires the working group to provide guidance to agencies in developing their environmental justice strategies. However, the working group has not provided guidance to federal agencies on how to develop a strategic plan, including how to demonstrate they have considered their broader agency missions in developing their environmental justice strategic plans. According to the working group’s charter, the working group creates committees to carry out its responsibilities under this executive order, and one of those committees—the Strategy and Implementation Progress Report Committee—is to be available as a resource to federal agencies as they develop and update their environmental justice strategies. However, according to officials from EPA, which chairs the working group, this committee has not provided guidance to agencies on what to include in their strategic plans because each agency determines the direction of their plans. By developing such guidance, the working group could assist agencies in planning more strategically about which parts of their mission are important for achieving environmental justice. Of the 14 agencies that developed environmental justice strategic plans after 2011, all have issued at least one annual progress report on the implementation of these plans, but most have not issued such reports every year, as they agreed to do in the 2011 MOU (see table 2). As shown in table 2, two of the 16 agencies—DHS and DOJ—have issued progress reports every year. In addition, several agencies issued progress reports consistently during the first few years after signing the 2011 MOU but subsequently stopped issuing reports. For example, four agencies—DOE, HHS, DOI, and DOL—issued progress reports through 2016 but have not issued reports for 2017. Four additional agencies issued reports through either 2014 or 2015 but have not issued any reports since then. Only four agencies—DHS, DOJ, EPA, and GSA— have issued progress reports for 2017. The two agencies that did not develop environmental justice strategic plans after 2011—DOD and SBA—have not issued any progress reports. According to the 2011 MOU, each agency should issue an annual report on the progress it has made over the previous year in implementing its environmental justice strategic plan. However, agency officials from most of the agencies said that they had not issued annual progress reports because of competing priorities. In addition, officials from some agencies, including USDA, DOE, and VA, cited the change in administration in January 2017 as a factor in delaying or not issuing their progress reports. Officials from DOE, HHS, and DOT said that they planned to issue overdue progress reports in the near future. The remaining agencies who have not issued a progress report since 2016 or earlier either did not have plans to issue progress reports or did not provide information on the status of their progress reports. However, we have previously found that annual program performance reports can provide essential information needed to assess federal agencies’ performance and hold agencies accountable for achieving results. Further, we have previously found that reporting is part of a broader performance management process that includes identifying mission and desired outcomes, measuring performance, and using this information to report on performance and to identify gaps in performance. By issuing progress reports each year, the agencies—Commerce, DOD, DOE, DOI, DOL, DOT, Education, HUD, HHS, USDA, and VA—can have more reasonable assurance that they have the information needed to assess their performance and to demonstrate results. The agencies’ progress reports generally describe the environmental justice activities that the agencies conducted but do not include any methods to assess progress. In our review of the most recent progress reports issued by each of the 14 agencies, we found that these reports contain information on activities undertaken by the agency over the previous year. Some of the reports are organized by the goals that the agencies identified in their environmental justice strategic plans and include information on the agencies’ future plans for environmental justice efforts. However, most agencies have not established a method that would allow them to evaluate their progress toward their environmental justice goals, such as establishing performance measures. According to Office of Management and Budget (OMB) guidance, performance measures are a means of evaluating efficiency, effectiveness, and results. The guidance also describes different types of these measures, including outcome measures—indicating an agency’s progress toward achieving the intended results of its efforts—and output measures—usually expressed quantitatively and describe the level of activities that will be provided over a period of time (e.g., the number of meetings held or the number of people trained). Agencies may assess their progress using milestones, which are scheduled events signifying the completion of a major deliverable or a phase of work (e.g., a date by which the agency will release a certain product), according to OMB guidance. While not performance measures, milestones can help agencies track the actions they have completed in implementing their environmental justice strategic plans. Of the 16 agencies that signed the 2011 MOU, four agencies—DOI, EPA, HHS, and USDA—have established performance measures or milestones for their environmental justice efforts. Of these four agencies, two agencies—HHS and EPA—have reported on their progress toward achieving the performance measures or milestones they established. Examples of how the four agencies measured the progress of their environmental justice efforts include the following: DOI established performance measures in its 2012 environmental justice strategic plan and reported on progress using these measures in its 2013, 2014, and 2015 annual progress reports. DOI changed from performance measures to milestones in its 2016 strategic plan. For example, in the 2016 plan, DOI has target years for establishing public outreach strategies and creating a best practices report on public outreach activities for environmental justice communities. According to agency officials, DOI made this change because the performance measures from the 2012 plan were difficult and time- consuming to use, were not helpful in tracking progress, and did not result in actionable outcomes. DOI believed that an action plan would be easier to use for identifying actions to meet goals and for measuring progress. DOI has not yet reported on the milestones from its 2016 strategic plan. Its most recent progress report is from fiscal year 2016, the first year that the strategic plan covers. Agency officials stated that DOI plans to report on the milestones in its fiscal year 2017 progress report but did not provide a timeline for when this report would be issued. In its environmental justice strategic plan for 2016 through 2020, EPA established four goals for reducing environmental and health hazards: reducing children’s exposure to lead, reducing contamination of small and tribal drinking water systems, reducing fine particle air pollution, and reducing contamination at hazardous waste sites. EPA established performance measures for tracking progress toward each of these goals at the national level. For example, EPA’s goal is to achieve air quality that meets national standards for fine particle pollution in all areas of the country, with special emphasis on communities with poor air quality and low-income populations. EPA collected data from air monitors to determine its progress toward achieving this goal. In its progress report for fiscal year 2017, EPA reported an increase from 43 percent of low-income populations living in counties that attained the standards in 2006 through 2008 to 92 percent in 2014 through 2016. According to agency officials, EPA plans to continue reporting on the goals in the future. EPA has also established several other performance measures and milestones for its environmental justice activities. For example, in its environmental justice strategic plan for 2016 to 2020, EPA provides the status for 28 environmental justice activities that it had included in its environmental justice 2014 strategic plan. HHS established many performance measures and milestones in its 2012 environmental justice strategic plan and reported on its progress toward these measures and milestones in its annual progress reports. In its most recent progress report, HHS reported that, as of January 2017, 30 of the 37 actions that it committed to undertake in the 2012 strategic plan had a status of “complete or substantial progress,” three had achieved “some progress,” and four could not be carried out and were deemed “inactive.” For example, HHS reported that it has conducted outreach events to educate local communities on the purpose and functions of the HHS Office for Civil Rights. In this report, HHS also stated that it will no longer be reporting on these measures and milestones going forward and that it would be developing a new plan of action to achieve its environmental justice goals. HHS has not yet developed such a plan and therefore does not have any current performance measures or milestones. USDA established several performance measures and milestones for its five strategic goals in its environmental justice strategic plan for 2016 through 2020. For its first environmental justice strategic goal, USDA established performance measures involving increased funding for environmental justice-related programs. USDA established milestones for the rest of its goals. Its five strategic goals are: ensure USDA programs provide opportunities for environmental justice communities; increase capacity-building within environmental justice communities; expand public participation in program operations, planning activities, and decision-making processes to benefit environmental justice communities; ensure USDA’s activities do not have disproportionately high and adverse human health impacts on environmental justice communities and resolve environmental justice issues and complaints; and increase awareness, skills, and abilities of USDA employees regarding environmental justice issues. However, the agency has not issued a progress report since its 2016 strategic plan and has not yet reported on these measures and milestones. Agency officials said that USDA has collected information on these measures and milestones, but has not issued progress reports with this information. In our interviews with agency officials, a few described plans for developing new performance measures. In particular, EPA has proposed to implement a measure that would involve identifying key decisions across the entire agency in which environmental justice was taken into account. According to EPA officials, a significant way to incorporate environmental justice into an agency’s mission, including its programs, policies, and activities, is to include environmental justice considerations in its various decision-making processes. For example, EPA has set a goal of including environmental justice issues in the analyses for regulatory or permitting decisions, such as Clean Air rules or permits; officials stated that they could count the number of such decisions that that have included environmental justice issues in the underlying analyses for the decisions. Under the new performance measure, every EPA office would be responsible for identifying a certain number of decisions it has made and explaining how these decisions were affected by environmental justice considerations. The measure would also allow EPA to share examples of how various offices are taking environmental justice into account, so that other offices could learn from these examples (e.g., integrating environmental justice into permitting decisions). EPA plans to pilot this new measure through September 2019. The remaining 12 agencies have not established any performance measures or milestones. In the absence of annual progress reports that evaluate progress using performance measures or milestones, we interviewed agency officials about the progress they had made toward the primary directive in Executive Order 12898—to identify and address disproportionately high and adverse human health or environmental effects of their programs, policies, and activities on minority or low-income populations. Officials from most of these agencies said that they are unable to determine how much progress they have made toward achieving this directive. Specifically, officials from six of the agencies (Commerce, DOD, Education, DOJ, DOL, and VA) stated that they do not have a method for gauging their progress, although several of these agencies stated that they are able to identify specific accomplishments they have made toward addressing environmental justice issues. A seventh agency, DOT, said that it has made significant progress, but faced challenges in developing quantitative performance measures. Officials from DHS and GSA said that they gauge their progress by tracking the completion of action items or goals from their environmental justice strategic plans, and DOE said that it periodically gauges its progress through conducting qualitative reviews of its environmental justice work. Finally, DOD and SBA reported no efforts to gauge progress toward implementing the executive order. Officials for most of the 12 agencies that have not developed performance measures for their environmental justice efforts said they have not done so because it would be difficult and they are unsure how to do so. For example, DOJ officials commented that it would be difficult to develop meaningful measures that are indicative of true progress toward achieving environmental justice. EPA officials commented that encouraging agencies to adopt performance measures for environmental justice would align with their agency’s efforts and would involve, among several things, providing guidance and training to the agencies. The 2011 MOU states that annual progress reports issued by the agencies should include performance measures as deemed appropriate by each agency. In our previous work, we have found that it is important for agencies to establish a method to assess their progress toward their goals; such methods should ideally include performance measures or milestones. We have also reported that performance measures are important for tracking progress in achieving goals and are a key element of effective strategic planning. Performance measures provide managers with information on which to base their decisions, including how effectively offices are integrating environmental justice in their decisions. Performance measures also create powerful incentives to influence organizational and individual behavior. Leading practices we have identified include clearly relating performance measures to the performance they will be used to evaluate and creating a set of performance goals and measures that addresses important and varied aspects of program performance. The executive order directs the working group to provide guidance to agencies in developing their environmental justice strategies. However, the working group has not provided guidance to its members on methods to assess and report on their environmental justice progress, such as through performance measures, according to officials from EPA, which chairs the working group. According to these officials, EPA is still pursuing its own agency-wide performance measures. By developing such guidance or creating a committee, the working group could assist agencies in tracking and measuring their progress in achieving their environmental justice goals. Most agencies that signed the 2011 MOU reported taking various actions to identify and address environmental justice issues related to their programs, policies, and activities; most also reported having limited resources for these efforts. Examples of actions they reported taking included improving research and data collection by creating data tools, considering environmental justice issues when implementing NEPA and enforcing environmental laws, and revising processes to ensure greater public participation. Most agencies used resources from existing related programs (e.g., civil rights or environmental programs) to support environmental justice efforts, although two agencies provided dedicated resources specifically to environmental justice efforts from fiscal years 2015 through 2018. Most of the 16 agencies reported planning and implementing actions to identify and address environmental justice issues to carry out the 1994 executive order and 2011 MOU. The executive order contains four areas that agencies’ environmental justice strategies should include, as appropriate: Promote enforcement of all health and environmental statutes in areas with minority populations and low-income populations. Ensure greater public participation. Improve research and data collection relating to the health of and environment of minority populations and low-income populations. Identify differential patterns of consumption of natural resources among minority populations and low-income populations (e.g., subsistence fishing or hunting). The 2011 MOU contains four additional areas that the 16 agencies agreed federal environmental justice efforts should include, as appropriate: Implement the National Environmental Policy Act (NEPA). Implement Title VI of the Civil Rights Act of 1964, as amended. Consider impacts from climate change. Consider impacts from commercial transportation and supporting infrastructure (goods movement). Each of the 14 agencies that produced an environmental justice strategic plan discussed in their most recent plan how they would identify and address environmental justice issues related to at least one of these eight areas. Although most agencies did not formally report on progress annually, all of the 14 agencies provided examples—in their strategic plans or progress reports, in other related documents or on their websites, or in interviews with us—of actions they implemented to identify and address environmental justice issues. In addition to the eight areas outlined in the 1994 executive order and 2011 MOU, agencies also provided examples of actions they took to provide internal training and conduct external capacity building. See appendix II for additional examples of agency actions to identify and address environmental justice issues. Improve research and data collection. In their most recent environmental justice strategic plans, 11 agencies discussed planning to improve research and data collection on environmental justice issues. At least eleven agencies provided examples of research or data actions they implemented, including creating data tools. For example, in 2015, EPA publicly released its Environmental Justice Mapping and Screening Tool (EJSCREEN), a web-based mapping tool that includes environmental and demographic data at a local level, allowing users to identify potential exposure to environmental pollutants and related health risks across different communities. Officials from DOJ’s Environmental and Natural Resources Division told us that they regularly use EJSCREEN to help determine if cases involve environmental justice issues. Also, since 2015, EPA and HHS’s National Institute on Minority Health and Health Disparities and National Institute of Environmental Health Sciences have co-funded a collaborative research and data effort called the Centers of Excellence on Environmental Health Disparities Research. This effort facilitates research on diseases that are a burden on populations with environmental justice issues and promotes knowledge sharing among researchers. Example of Addressing Environmental Justice Issues in EPA Rulemaking In January 2017, EPA released a final rule amending its Risk Management Program, a program under the Clean Air Act that requires facilities using extremely hazardous substances to develop a risk management plan to submit to EPA at least once every 5 years. The rule changes were identified by a Chemical Facility Safety and Security Working Group composed of the Administrator of EPA, and the department heads of Labor, Homeland Security, Justice, Agriculture, and Transportation, which was created in 2013 by Executive Order 13650 after chemical facility incidents that resulted in fatalities. The executive order requires that the working group develop ways to improve operational coordination with state, local, tribal, and other partners, including enhancing federal agency information sharing. In a May 2014 report, the working group cited the need to familiarize all agencies with Executive Order 12898 on environmental justice. It identified concerns of communities living adjacent to chemical facilities, many of them low-income and minority, and the need to share information with these communities, including first responders. Under EPA’s 2017 rule, risk management plans must be provided to members of the public upon request. The notice publishing the final rule contained a section on environmental justice comments and its response to address environmental justice concerns. In May 2018, EPA proposed to rescind several amendments to its rule. Industry and some states raised concerns about the cost and burden to carry out the rule. Promote enforcement of health and environmental statutes. In their most recent environmental justice strategic plans, 13 agencies discussed planning to promote enforcement of health or environmental statutes in some form. At least 12 agencies provided examples of actions they implemented to promote enforcement, including ensuring enforcement of environmental laws in communities with environmental justice issues and addressing such issues in the resolution of cases against violators. For example, in its 2017 progress report, EPA reported combining EJSCREEN with enforcement and compliance data to help regional offices and state, local, and tribal authorities focus reviews of compliance with environmental laws in overburdened communities. EPA reported reviewing all enforcement cases to see if communities with environmental justice issues were affected and tracking how agency enforcement actions to resolve these cases benefitted the affected communities. As a result, EPA reported tracking that 45 percent of Supplemental Environmental Projects—a type of beneficial environmental project implemented as part of a civil enforcement action settlement—in fiscal year 2017 were in locations with potential environmental justice issues. Ensure greater public participation. In their most recent environmental justice strategic plans, 14 agencies discussed planning to ensure greater public participation in decision-making processes. All 14 agencies provided examples of public participation actions they implemented, including seeking public input on their environmental justice strategic plans or consulting communities directly during environmental analyses under NEPA, siting decisions, or enforcement cases. For example, in its 2016 progress report, DOI reported formally inviting tribes to participate in environmental analyses and revising policies on tribal-government relations. DOI also continued to have publicly designated environmental justice coordinators for each of its bureaus (e.g., Bureau of Land Management), many of which deal directly with tribes or manage natural resources they rely on, such as land or water. Example of an EPA Environmental Justice Grant to Study Microplastics in Tribal Foods In 2017, the Sitka Tribe of Alaska received an Environmental Protection Agency (EPA) Environmental Justice Small Grant to study microplastics in its traditional food sources, such as mussels and clams. Microplastics are tiny pieces of plastic that are less than 5 millimeters in length and, according to EPA, may contain toxic chemicals that can pose human health and ecosystem risks when ingested by aquatic animals. According to EPA, the tribe planned to collect samples of water and traditional foods from four locations within its traditional territory and test them for the presence of microplastics and associated toxins. The results were to be shared with the tribe and the public to inform decisions about harvesting traditional foods. Local students collected and tested Butter Clam and Blue Mussel samples in 2018, which showed that more than 80 percent of the mussels and 100 percent of the clams contained microfibers and other microplastic particles. Identify differential patterns of consumption of natural resources. Because many Native Americans and other minority communities rely on hunting, foraging, or fishing for food, five agencies planned actions to identify or address risks to these food sources in their most recent environmental justice strategic plans. At least eight agencies provided examples of actions they implemented in this area, including collecting or providing information on human health risks associated with the consumption of polluted fish or wildlife. For example, in its 2015 progress report, USDA reported that the Forest Service’s Alaska Regional Office coordinated with DOT’s Federal Aviation Administration to accelerate cleanup of petroleum-contaminated soil at a mixed-ownership site containing national forest lands. According to USDA, the need for accelerated cleanup arose because increasing sea-levels and tidal surges that were encroaching on the area would have washed the pollutants into nearby waters supporting a local subsistence fishery. Implement NEPA. In their most recent environmental justice strategic plans, 12 agencies discussed planning to consider environmental justice issues in their NEPA analyses. At least 13 agencies provided examples of NEPA actions they had implemented, including providing internal guidance on how to include environmental justice issues in NEPA analyses. For example, at DOI, it is departmental policy for all bureaus to include consideration of environmental justice in the NEPA process and some bureaus have developed their own guidance for doing so. For example, DOI’s 2015 National Park Service NEPA Handbook requires the agency’s environmental analyses to discuss and evaluate the impact of proposals on minority and low-income populations and communities, including the distribution of the benefits and risks among different communities and populations. Implement Title VI of the Civil Rights Act of 1964. In their most recent environmental justice strategic plans, 11 agencies planned to consider environmental justice issues when implementing their Title VI programs. At least 10 agencies provided examples of Title VI environmental justice actions they implemented, some of which focused on providing training and guidance. For instance, in 2016, DOJ, DHS, HUD, HHS, and DOT jointly issued interagency guidance on Title VI to state and local agencies involved in emergency activities. DHS and DOJ reported that DHS’s Office for Civil Rights and Civil Liberties and DOJ’s Civil Rights Division coordinated to distribute this guidance in the aftermath of the 2017 hurricane season to ensure that federal funding recipients (e.g., state and local agencies) were aware of their obligations to provide emergency management services across communities without discrimination. Consider impacts from climate change. In their most recent environmental justice strategic plans, nine agencies discussed planning to address impacts from climate change on communities with environmental justice issues. At least 11 agencies provided examples of actions they implemented in this area, including providing communities with information on how climate change may affect them. For example, in its 2016 progress report, DOI reported that the U.S. Geological Service working with the Swinomish Indian Tribal Community and Skagit River System Cooperative to build a coastal model to evaluate the impacts of sea-level rise, storm surge, and waves, including effects on foods such as salmon and shellfish. DOI reported that the model was used to inform tribal climate adaptation and resilience plans. Consider impacts from goods movement. In their most recent environmental justice strategic plans, three agencies discussed planning to address environmental justice issues arising from goods movement, and at least five agencies provided examples of actions they implemented in this area. For example, DOT’s Federal Highway Administration developed a detailed freight and land use handbook in 2012, which highlights potential negative impacts in communities with minority or low- income residents (e.g., air quality or light pollution) and provides guidance on integrating freight and land-use planning to balance freight’s beneficial economic impacts and harmful environmental impacts for affected communities. For example, the handbook advises using off-peak deliveries or anti-idling technologies to reduce impacts from emissions. Provide internal training. Eleven agencies also provided us with examples of training programs to help their staff identify and address environmental justice issues within their work. For example, EPA developed an introductory training on environmental justice, which was required training for all EPA staff agency-wide when it was first launched in 2015. More recently, EPA reported providing environmental justice training in 2017 to more than 1,000 employees and contractors across the government who were responsible for implementing NEPA. DOI developed a web-based introductory training on environmental justice in 2015 that is available to all DOI employees and became required training for project managers for the Central Hazardous Materials Fund in 2016. Example of an EPA Environmental Justice Grant to Build Community Capacity to Reduce Exposure to Contaminated Soil through Community Education In 2017, the Trumbull Neighborhood Partnership in Warren, Ohio, received an EPA Environmental Justice Small Grant for an educational initiative to reduce residents’ exposure to soil contamination from former industrial activities, such as steel production. According to EPA, with support from the grant, the neighborhood partnership planned to create a curriculum of best practices, repurpose vacant land, and share a range of educational materials with residents to help them learn how to avoid exposure to contaminated soil. As part of the educational campaign on safe soil handling practices for residential and community land use, the partnership created a website to host educational materials and also shared the materials in person with residents and contacted local contractors to help ensure safe demolition practices. Conduct external capacity building. Thirteen agencies also provided examples of actions they implemented to fund and assist communities with environmental justice issues to build their capacity to access available resources and participate in federal decisions that affect them. For example, since its inception in 1994, EPA’s Environmental Justice Small Grants Program has awarded more than $24 million to over 1,400 organizations working with communities with environmental justice issues. EPA provides these grants for up to $30,000 to support projects that help communities build understanding of local environmental and public health issues, develop strategies for addressing these issues, and facilitate discussions about community priorities. From fiscal year 2015 through 2018, most of the 16 agencies reported supporting environmental justice efforts through existing related program funding and staffing resources that were not specifically dedicated to environmental justice. EPA and DOE were the only agencies that dedicated resources specifically for environmental justice efforts in their budgets. In fiscal year 2018, EPA provided about $6.7 million, which, according to EPA officials, supported 31 full-time-equivalents (FTE) for Office of Environmental Justice staff in its headquarters and environmental justice coordinators in regional offices and two environmental justice grant programs. These staff support data tools such as EJSCREEN, provide training sessions, and coordinate federal efforts through the Interagency Working Group on Environmental Justice. The two grant programs provide communities with funding to research and understand potential environmental and health issues in their communities. For fiscal years 2015 through 2018, EPA awarded an average of about $1.2 million annually in environmental justice grants to communities through the Environmental Justice Small Grants Program and Environmental Justice Collaborative Problem-Solving Cooperative Agreement Program. EPA officials also reported using other related resources to support environmental justice efforts, but said the agency does not track these resources separately. In fiscal year 2018, DOE provided about $1.6 million and, according to DOE officials, one FTE for its environmental justice program in its Office of Legacy Management. These resources support activities to manage problems and concerns arising from the materials and chemicals on DOE sites by giving communities and tribes near these sites opportunities and tools to participate in DOE decisions. DOE also uses its funds and staff to sponsor the annual National Environmental Justice Conference and Training Program and to participate in the interagency working group. Eleven of the remaining 14 agencies reported undertaking some examples of environmental justice efforts with support from funding and staff from existing related programs (e.g., civil rights or environmental programs) from fiscal year 2015 through 2018. According to budget documents and agency officials, these 11 agencies did not formally track resources used to support environmental justice activities. Four of these agencies—USDA, DOI, GSA, and HUD—provided us with estimates of staffing or funding resources used to support environmental justice efforts. USDA estimated that a total of about eight FTEs annually were charged by many different staff for fiscal years 2015 through 2018 and that between $10,000 and $22,500 in funding annually supported the National Environmental Justice Conference and Training Program. DOI reported that it has one full-time Environmental Justice Outreach Specialist and that most DOI bureaus have an Environmental Justice Coordinator who handles environmental justice responsibilities as a collateral duty. DOI also reported funding one small research project related to environmental justice. GSA reported that staffing related to environmental justice efforts constituted a portion of the total FTE allocation within its Office of Civil Rights and estimated that this amounted to less than one FTE annually for fiscal years 2015 through 2018. HUD also estimated that less than one FTE was used specifically to support environmental justice efforts annually for the period, with one designated environmental justice lead and other staff serving on the working group as needed. Officials from the other seven agencies did not quantify estimates of resources but told us that staff conduct these activities as collateral duties. For example, DHS told us that its Office of the Chief Readiness Support Officer, Office for Civil Rights and Civil Liberties, the Office of General Counsel support its environmental justice efforts as needed. In another instance, DOJ designated an Environmental Justice Director, created a Senior Litigator for Environmental Justice position, and reported that the department has other staff that spend a portion of their time working on environmental justice efforts. Several agencies also reported establishing internal working groups or other coordinating bodies to help implement their environmental justice efforts, which means using some staffing resources to support these coordinating efforts. Three agencies—DOD, Education, and SBA—reported providing no funding or staffing resources to carry out any environmental justice efforts and also did not report any examples of environmental justice efforts from fiscal year 2015 through 2018. Agency resources for environmental justice were one of the concerns several stakeholders that we interviewed raised (see textbox). Stakeholder Perspectives on Federal Environmental Justice Efforts Several stakeholders expressed concerns about agency resources, agency responsiveness to and awareness of environmental justice issues, legal tools for raising environmental justice concerns, or overall prioritization of environmental justice efforts. Stakeholders expressed concerns about the limited availability of resources for environmental justice efforts, including staff to carry out environmental justice work and funding for related programs. One stakeholder told us that agencies need to prioritize their environmental justice efforts because they have not identified all communities with potential environmental justice issues and lack the resources to address all environmental justice issues. Several stakeholders discussed concerns about variation in agency staff familiarity with environmental justice issues or responsiveness to issues raised. Stakeholders also expressed concerns about the ability of existing legal tools to address environmental justice issues in the absence of a legal framework that specifically addresses them. For example, stakeholders said that risks from cumulative pollutant exposure are not addressed by existing environmental statutes. Several stakeholders also expressed concern about federal prioritization of environmental justice issues overall, including enforcement, changes to existing environmental regulations, and limited consideration of environmental justice in rulemaking processes. Some stakeholders we interviewed, including representatives from local and national nonprofit organizations, university professors, federal officials, and employees of private companies, also said that agencies’ efforts to build community capacity and develop tools that address environmental justice issues have been helpful. Stakeholders told us that EPA’s Environmental Justice Small Grants Program has helped communities, and DOE’s National Environmental Justice Conference and Training Program brings together grassroots leaders, stakeholders, and agencies. Stakeholders said that EJSCREEN is a useful tool for agencies and the public to screen for communities with potential environmental justice issues. Stakeholders also said agencies could use EJSCREEN in additional ways (e.g., in rulemaking and permitting) and discussed some limitations for its use (e.g., data limitations and the need to directly engage communities). The working group has collaborated in issuing guidance and in several other areas regarding environmental justice. The working group has also demonstrated three of the key features of interagency collaboration that we reviewed—leadership, clarity of roles and responsibilities, and written guidance and agreements. However, its use of two features of interagency collaboration—participation and organizational outcomes and accountability—was limited. Collaboration from an Interagency Working Group Committee Assists with Environmental Justice Issues in Lowndes County, Alabama A November 2017 American Journal of Tropical Medicine and Hygiene study of hookworm conducted in Lowndes County, Alabama, highlighted a long-standing situation created by poor wastewater management affecting a largely rural, minority population in the state. The makeshift septic tanks that residents use in the absence of proper wastewater treatment infrastructure do not function properly in the moist, rich soil common in that area. This problem increased residents’ exposure to parasites, such as hookworm, through untreated wastewater. According to agency officials, in 2018, the General Services Administration collaborated with the Rural Communities Committee of the Interagency Working Group on Environmental Justice to help apply for Department of Agriculture rural development grant funding for decentralized sewer systems in Lowndes by using federal surplus personal property as matching funds. As of March 2016, the Equal Justice Initiative and Alabama Center for Rural Enterprise were working to identify and employ alternative decentralized technologies to treat wastewater in the county. The two entities were also attempting to write and implement policies requiring residents to connect to public sewers. In 2017, the Impacts from Commercial Transportation committee released a compendium on publicly available federal resources to assist communities impacted by goods movement activities. In fiscal year 2017, with input and vetting from the Rural Communities committee, USDA compiled and launched a web page with links to community tools, funding opportunities, educational or training assistance, and case studies to support rural communities according to USDA officials. In March 2016, the NEPA committee issued guidance entitled, “Promising Practices for Environmental Justice Methodologies in NEPA Reviews.” According to working group officials, this guidance can assist federal agencies with incorporating environmental justice during their NEPA reviews. In March 2019, the committee also completed guidance for communities entitled, “Community Guide to Environmental Justice and NEPA Methods.” Hookworms can be found in soil contaminated by untreated wastewater. In 2016, the working group’s Rural Communities committee participated in a brownfields redevelopment conference to help local organizations understand and access resources to redevelop brownfields in their communities. In 2016, the Regional Interagency Working Groups committee coordinated technical assistance to communities in EPA’s regions 2 and 4. For example, the group is working in North Birmingham, Alabama, and other communities to evaluate air, water, and waste issues. With respect to the five key features of interagency collaboration that we reviewed, we found that the working group demonstrated leadership, clarity of roles and responsibilities, and written guidance and agreements. However, its use of two other key features—participation and clear goals—was limited. In our September 2012 report on interagency collaborative mechanisms, we identified leadership as a key feature of collaborative groups and stated that identifying a leader and sustaining that role throughout the groups’ efforts are important. For the working group, EPA’s Administrator was identified as the chair of the group in both the 1994 executive order and the 2014 Charter for Interagency Working Group on Environmental Justice. EPA officials we interviewed described the agency’s role as providing guidance to the working group agencies and coordinating their efforts. More specifically, EPA officials we interviewed said that as chair of the working group, EPA’s responsibilities include the following: Convene monthly meetings with the working group. Provide public access to working group agencies’ environmental justice strategic plans and annual implementation progress reports, a list of working group agencies, and other information relevant to the working group. Lead the development and publication of the working group’s plans and reports. Our September 2012 report identified the need for collaborative groups to have clarity about the roles and responsibilities of the participating agencies. We stated that clarity can come from agencies working together to define and agree on their respective roles and responsibilities, as well as steps for decision-making. The working group has done this by assigning roles to its chair and most of its member agencies. In particular, according to working group officials, the topics for the nine working group committees were based on the seven functions that the executive order assigned to the working group and public input. Officials from 13 of the working group members agreed to either chair or become a member of one or more committees. The topics that these committees address, their chair, members, and purpose are identified in table 3: Our September 2012 report on interagency collaborative mechanisms stated that agencies that articulate their agreements in formal documents can strengthen their commitment to working collaboratively. Since 2011, when the 16 agencies and CEQ recommitted to carrying out environmental justice efforts, the working group has developed several such documents including: MOU on Environmental Justice. This document, signed in 2011, is an agreement among member agencies to recommit to addressing environmental justice issues. It also listed the four areas that the agencies agreed to work on: NEPA, Title VI of the Civil Rights Act, impacts from climate change, and impacts from goods movement. Charter for Interagency Working Group on Environmental Justice. This document, which was adopted in 2011 and updated in 2014, outlines the governance structure for the working group. It also lists four committees to help carry out the working group’s responsibilities under the executive order: public participation, regional interagency working group, Title VI, and strategy and implementation progress reports. Framework for Collaboration. This document, which was issued in 2016 and covered a 3-year period through 2018, listed four goals of the working group to advance greater federal agency collaboration. It also listed and described the purpose of the nine working group committees. In our September 2012 report, we found that it is important to ensure that the relevant participants have been included in the collaborative effort. Participation in working group activities has been mixed. In the 2011 MOU, the 16 signing agencies and CEQ agreed to address environmental justice issues and participate as members of the working group. According to agency officials, most working group members attend the monthly meetings. The most active members of the working group, in terms of participation in working group committees, have been EPA and DOJ. EPA, the chair of the working group, also chaired or co-chaired six committees, and DOJ chaired or co-chaired four. Both also participated in all eight of the active committees (see table 4). However, four agencies—DOD, Education, SBA, and VA—did not attend any of the working group’s monthly meetings in fiscal year 2018. These agencies also did not participate as leaders or members in any working group committees in fiscal year 2018. Furthermore, DOD and SBA did not have a designated representative as of March 2019. These four agencies had various reasons for not participating more actively in the working group or its committees. DOD officials said that DOD has not been involved with the working group since August 2017, when its working group representative retired, because it does not have the resources to participate in the working group. Education officials also said that they have had a limited role with the working group because many of the topics discussed have not been relevant to their agency’s missions. For example, according to Education officials, while research has established that schools with poor environmental health conditions often serve disadvantaged students, Education does not have authority to plan, fund, construct, maintain, or operate school facilities and grounds. As discussed earlier, SBA officials we interviewed said that they were unclear on whether environmental justice applied to SBA’s mission and that they were in the process of reviewing whether SBA should continue its membership in the working group. VA officials confirmed that it has also been inactive with the working group, but will call in to a meeting if there are topics of relevance. EPA officials commented that it is difficult to characterize what specific opportunities are missed from the lack of representation by an agency. However, they also commented that nonparticipation limits the working group’s ability to fulfill its mandates in a strategic, methodical way across the entire federal government. EPA officials further stated that the limiting factor for the working group in its efforts to address the executive order on environmental justice has always been the will of leadership across federal government to make clear, measurable commitments of those priorities and to adequately resource the attainment of those commitments. However, the participants signed the 2011 MOU about 8 years ago, and the agreement has become dated and may not reflect the agencies’ current commitments or abilities to participate in the working group or the broader environmental efforts. Our 2012 report on interagency collaborative mechanisms stated that written agreements and documents are most effective when they are regularly updated and monitored. By updating the 2011 MOU and renewing the commitment among participating agencies, EPA and the working group agencies would have more reasonable assurance that those agencies who sign the agreement are committed to participating. Our September 2012 report found that collaborative mechanisms such as the working group benefit from clear goals to establish organizational outcomes and accountability. The report stated that participants might not have the same overall interests or may even have conflicting interests, but by establishing a goal based on common interests, a collaborative group can shape its own vision and define its purpose. The executive order that created the working group assigned the working group seven functions to carry out, as listed in table 5. While the working group has developed documents with agreed-upon goals, which is beneficial to collaboration, none of them address all the seven functions of the executive order. The working group’s organizational documents do not contain strategic goals aligned to address the executive order as suggested by our previous work on establishing clear goals for collaborative mechanisms. Further, the three functions involving environmental justice research, data collection, and studies are not described as part of the goals of the working group, as laid out in its various documents: The 2011 MOU includes four focus areas for the working group members: NEPA, Title VI, impacts from climate change, and impacts from goods movement. These do not include the executive order functions of environmental justice data collection, research, and studies. The 2011 Charter for Interagency Working Group on Environmental Justice states that the committees were created to help carry out the working group’s responsibilities under the executive order. The committees focus on certain working group roles and responsibilities, including NEPA, goods movement, strategic planning, and public participation. However, none of the committees focus on environmental justice research, data collection, or studies. The working group’s fiscal year 2016-2018 Framework for Collaboration’s has four goals for collaboration: (1) enhance communication and coordination to improve the health, quality-of-life, and economic opportunities in overburdened communities; (2) enhance multi-agency support of holistic community-based solutions to provide assistance as needed to address environmental justice issues; (3) advance interagency strategies to identify and address environmental justice issues in agency programs, policies, and activities; and (4) develop partnerships with academic institutions to assist in providing long-term technical assistance to overburdened communities. These goals do not pertain to environmental justice research, data collection, or studies. We found that the organizational documents do not provide strategic goals with clear direction for the committees to carry out the functions of the working group as laid out in the executive order. Our analysis, which compares the functions of the executive order to documented working group roles and responsibilities, shows that coordinated data collection and examination of research and studies on environmental justice are not included in these documents or committee purposes and have not been a focus of the interagency working group since at least 2011. A DOI official acknowledged that the working group has not addressed all of these functions from the executive order; the official attributed the omission to a lack of resources for the working group. EPA officials commented that some individual agencies, such as HHS and EPA, have done work in environmental justice data collection and research. As leaders of the working group, EPA officials told us that the 2011 MOU, committee groups, and framework for collaboration reflect the current priorities of the working group, based on the public’s input. They were unsure whether a coordinated effort in the data collection, research, and studies areas was needed, but they said such an effort could be useful. They said that the most useful role of the working group in research may be as a forum for sharing of information and providing training opportunities. By clearly establishing strategic goals in the working group’s organizational documents to carry out the 1994 executive order, EPA, in consultation with working group members, could enhance its strategic direction for intergovernmental environmental justice efforts. The interagency working group on environmental justice and its 16 member agencies have put in place the building blocks for an environmental justice program across the federal government. They have conducted a number of efforts over the last 25 years to implement the Executive Order on Environmental Justice. Through these efforts, they have developed tools such as EJSCREEN and guidance for incorporating environmental justice under NEPA. Most of the agencies have also developed strategic plans since 2011, although two agencies we reviewed have not, and many others have not kept their plans updated. SBA is in the process of reviewing whether it should continue its membership in the working group, which should clarify its role after SBA completes its review. DOD developed an environmental justice strategic plan in 1995 after the executive order was issued but not since 2011 when the interagency working group members signed the MOU. By updating its environmental justice strategic plan, DOD would have a foundation for its environmental justice efforts. Another seven agencies developed environmental justice strategic plans in 2012 but have not updated them since. By updating their strategic plans, these agencies— Commerce, DHS, DOJ, DOL, Education, HUD, and VA—would have a current plan to guide their environmental justice activities as they committed to do in the 2011 MOU. Moreover, most agencies—Commerce, DOD, DOE, DOI, DOL, DOT, Education, HHS, HUD, USDA, and VA—have not shown clear progress toward achieving their environmental justice goals in the 8 years since they signed the working group’s 2011 MOU because they have not consistently issued progress reports. By issuing progress reports each year, the agencies can provide essential information needed to assess their performance and demonstrate results. The 16 agencies and CEQ signed the 2011 MOU to establish a collaborative initiative across agencies to carry out environmental justice efforts. Under the leadership of EPA, they have also put in place a structure to coordinate with each other on their environmental justice efforts. One area that the group has not coordinated, however, is in developing guidance on what to include in strategic plans, such as demonstrating how environmental justice is part of an agency’s mission, or developing methods to assess and report on progress, which many of the agencies said they needed. Under GAO’s leading practices for strategic planning, agencies’ plans should address their missions, articulate goals, and lay the groundwork for assessing progress. Only half of the agencies that developed environmental justice strategic plans after 2011 clearly assessed how their plans fit into their overall missions. By developing guidance on what agencies should include in their environmental justice strategic plans, the working group could assist agencies in planning more strategically about what parts of their mission are important for achieving the environmental justice directives outlined in Executive Order 12898. Few of the agencies had performance measures or other methods to assess progress. By developing guidance on methods that the agencies could use to assess and report on progress, or creating a committee to do so, the working group could assist agencies in tracking and measuring their progress in achieving their environmental justice goals. In addition, the working group faces challenges of unclear strategic goals and mixed levels of participation. As noted in our earlier work, collaborative mechanisms, such as the working group, benefit from clear goals to establish organizational outcomes and accountability. Although the 1994 executive order created the working group to carry out the functions of the executive order, the working group’s framework focuses on how the agencies will collaborate rather than setting clear strategic goals to carry out the executive order. As a result, several of the executive order’s functions are not being carried out by the working group. By clearly establishing, in its organizational documents, strategic goals for the federal government’s efforts to carry out the 1994 executive order, EPA and the working group members could enhance the strategic direction for intergovernmental environmental justice efforts. Furthermore, by updating the 2011 MOU and having the 16 agencies and CEQ renew their commitment to participating in the interagency collaborative effort and the working group, EPA, as chair of the working group and consulting with other working group members, would have more reasonable assurance that those who sign the agreement are committed to participate. We are making a total of 24 recommendations to 15 agencies of the Interagency Working Group on Environmental Justice—nine to the federal agencies that need to develop or update strategic plans (recommendations 1-9); 11 to the federal agencies that need to develop annual progress reports (recommendations 10-20); and four to the Environmental Protection Agency as chair of the working group (recommendations 21-24). The Secretary of Commerce should update the department’s environmental justice strategic plan. (Recommendation 1) The Assistant Secretary of Defense for Sustainment should update the department’s environmental justice strategic plan. (Recommendation 2) The Secretary of Education should update the department’s environmental justice strategic plan. (Recommendation 3) The Secretary of Homeland Security should update the department’s environmental justice strategic plan. (Recommendation 4) The Secretary of Housing and Urban Development should update the department’s environmental justice strategic plan. (Recommendation 5) The Attorney General of the United States should update the department’s environmental justice strategic plan. (Recommendation 6) The Secretary of Labor should update the department’s environmental justice strategic plan. (Recommendation 7) The Administrator of the Small Business Administration should complete the agency’s assessment of whether to participate in the 1994 Executive Order and the 2011 Memorandum of Understanding, and, if appropriate, develop an environmental justice strategic plan. (Recommendation 8) The Secretary of Veterans Affairs should update the department’s environmental justice strategic plan. (Recommendation 9) The Secretary of Agriculture should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 10) The Secretary of Commerce should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 11) The Assistant Secretary of Defense for Sustainment should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 12) The Secretary of Education should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 13) The Secretary of Health and Human Services should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 14) The Secretary of Energy should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 15) The Secretary of Housing and Urban Development should issue a progress report on its environmental justice efforts each year. (Recommendation 16) The Secretary of the Interior should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 17) The Secretary of Labor should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 18) The Secretary of Transportation should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 19) The Secretary of Veterans Affairs should issue a progress report on the department’s environmental justice efforts each year. (Recommendation 20) The Administrator of EPA, as chair of the working group, should develop guidance for agencies on what they should include in their environmental justice strategic plans. (Recommendation 21) The Administrator of EPA, as chair of the working group, should develop guidance or create a committee of the working group to develop guidance on methods the agencies could use to assess progress toward their environmental justice goals. (Recommendation 22) The Administrator of EPA, as chair of the working group, and in consultation with the working group, should clearly establish, in its organizational documents, strategic goals for the federal government’s efforts to carry out the 1994 Executive Order. (Recommendation 23) The Administrator of EPA, as chair of the working group, and in consultation with the other working group members, should update the 2011 Memorandum of Understanding and renew the agencies’ commitments to participate in the interagency collaborative effort and the working group. (Recommendation 24) We provided a draft of this report to CEQ and 16 federal agencies— Commerce, DHS, DOD, DOE, DOI, DOJ, DOL, DOT, Education, EPA, GSA, HHS, HUD, SBA, USDA, and VA—for review and comment. Fourteen agencies provided comments on our report. The comments of 12 agencies—DHS, DOD, DOE, DOI, DOJ, DOL, DOT, Education, EPA, HHS, USDA, and VA—are reproduced in appendixes III-XIV, respectively. HUD and SBA provided comments by email. Of these 14 agencies, eight agencies—DHS, DOE, DOI, DOJ, HHS, SBA, USDA, and VA—agreed with our recommendations. Of the other six agencies that provided comments, EPA agreed with two recommendations and disagreed with two others; DOD agreed with one recommendation and disagreed with one other; DOT partially agreed with the recommendation; DOL and HUD neither agreed nor disagreed with their recommendations, and Education did not agree with its two recommendations. We also made recommendations to Commerce, but it did not provide comments in time to include them in our report. Although we did not make recommendations to them, CEQ and GSA reviewed our report. CEQ provided technical comments, which we incorporated as appropriate; GSA did not have any comments on our report. In addition to CEQ, we also received technical comments and clarifications from DHS, DOJ, DOT, EPA, HHS, and USDA, which we incorporated as appropriate. We directed four recommendations to EPA as chair of the Interagency Working Group on Environmental Justice; the recommendations are aimed at improving the strategic direction of the working group and the federal government’s efforts. EPA stated that it appreciates our work on this subject area and understands the need for interagency coordination and is working closely and collaborating with its federal partners. EPA agreed with the two recommendations to develop guidance for agencies on what they should include in their environmental justice strategic plans (recommendation 21) and to develop guidance or create a committee of the working group to develop guidance on methods the agencies could use to assess progress toward their environmental justice goals (recommendation 22). However, EPA disagreed with the recommendations to update the 2011 MOU and renew the agencies’ commitments to participate in the interagency collaborative effort and the working group (originally recommendation 23, now recommendation 24) and to clearly establish strategic goals for the federal government’s efforts to carry out the 1994 Executive Order (originally recommendation 24, now recommendation 23). EPA stated that it disagrees with recommendations 23 and 24; instead of updating the MOU, the agency will lead efforts to update the working group’s fiscal year 2016-2018 Framework for Collaboration to include guidance for strategic plans, tracking progress toward goals, and defining alignment with the executive order. The agency also said that it believes that the intent of recommendation 24 could be combined with recommendation 23, making recommendation 24 unnecessary. We believe that EPA misunderstood recommendation 24 and do not agree it should be combined with recommendation 23. We agree with EPA that the working group can benefit from greater guidance on strategic plans, tracking goals, and alignment with the executive order to carry out federal environmental justice efforts. In our report, we list three organizational documents—the 2011 MOU, the 2011 Charter for Interagency Working Group on Environmental Justice, and the Framework for Collaboration. Our recommendation is for EPA to clearly establish strategic goals for federal efforts to carry out the executive order and does not specify which organizational document needs to be updated to address these issues. To help avoid confusion about the intent of this recommendation, we made two changes in the report. First, we clarified in the report that we were referring to the interagency working group’s strategic goals and organizational documents to show that we are not specifically recommending that the MOU be updated to meet this recommendation. Second, we switched the order of recommendations 23 and 24 so that our recommendation to establish strategic goals (previously recommendation 24) would no longer follow our recommendation to update the MOU. We disagree with EPA that it does not need to update the working group’s MOU because it plans to update the working group’s Framework for Collaboration. We believe that the MOU needs to be updated to address the matter of participation by the members who signed it but do not participate. As discussed in our report, the 2011 MOU is an agreement among member agencies to commit to addressing environmental justice issues. We do not have an opinion on when this document needs to be updated, however, and we believe that it can be updated after the working group discusses its strategic goals and updates its other organizational documents. Federal agencies may clarify how they can best participate through discussions of the working group’s goals and how they can meet the purposes of the executive order. DOD agreed with the recommendation that it update its environmental justice strategic plan (recommendation 2), but disagreed with the recommendation that it issue a progress report on its environmental justice efforts each year (recommendation 12). DOD provided two primary reasons why it disagreed with this recommendation. First, DOD stated that it had achieved the intent of Executive Order 12898 by including environmental justice considerations in its decision-making processes, primarily by using the NEPA review process. Second, the department stated that it has limited ability to further the implementation of environmental justice and create new goals and metrics in operating locations and mission. DOD stated that it is bound by its mission with limited opportunities to change where the department operates. According to DOD, for it to create new bases or close existing ones, it must first obtain congressional approval and then perform a NEPA analysis prior to implementation; also, its mission does not include a federal role in regulating or directing off-base activity or land uses; and aside from the U.S. Army Corps of Engineers civil regulatory functions, it does not routinely issue environmental permit decisions like federal regulatory agencies. DOD stated that these reasons make it a significant challenge for the department to meet our recommendation and therefore does not see a tangible benefit to additional reporting. We disagree with DOD that it does not need to issue a progress report on its environmental justice efforts each year. As we state in the report, the purpose of an annual progress report is to provide essential information needed to assess federal agencies’ performance and hold agencies accountable for achieving results. Reporting is part of a broader performance management process that includes identifying mission and desired outcomes, measuring performance, and using this information to report on performance and to identify gaps in performance. DOD would be reporting on goals that it set within its mission and authorities. For this reason, we continue to believe that by issuing progress reports each year, DOD could have more reasonable assurance that it has the necessary information to assess its performance and to demonstrate results. DOT stated that it partially concurs with recommendation 19 that it issue progress reports annually. DOT stated that it commits to issuing progress reports on its environmental justice efforts “when it determines that the circumstances of its activities so warrant.” However, we continue to believe that DOT should issue progress reports each year because doing so would give DOT more reasonable assurance that it has the information needed to assess its performance and to demonstrate results. DOL neither agreed nor disagreed with the two recommendations for it to (1) update its environmental justice strategic plan and (2) issue a progress report on its environmental justice efforts each year (recommendations 7 and 18). DOL stated that it values our review of its work in this area and will review the recommendations and take appropriate actions to improve program performance and delivery of services. HUD also neither agreed nor disagreed with our recommendations for it to update its environmental justice strategic plan and issue a progress report on its environmental justice efforts each year (recommendations 5 and 16). In an email, a HUD audit liaison official stated that the agency had no comments at this time and will continue to work with the current administration and the working group to update its environmental justice strategic plan and issue a progress report on its environmental justice efforts. Education stated that our report did not sufficiently account for the limitations on its legal authority in the subject area of environmental justice and that our report would be more accurate and comprehensive if it included more information about the department’s limited role. Education also stated that it did not agree with the recommendations to update its environmental justice strategic plan (recommendation 3) and issue a progress report on its environmental justice efforts each year (recommendation 13) because it does not believe this is the most appropriate course of action for the department or an efficient use of resources. We disagree with Education’s assessment. In the report, we discuss Education officials’ comments that they have a limited role with the working group because many of the topics discussed have not been relevant to their agency’s missions. We also discuss Education’s legal authority by including Education officials’ comment that the department does not have federal authority to plan, fund, construct, maintain, or operate school facilities and grounds. As discussed in the report, by updating its strategic plan, Education would have a current plan to guide its environmental justice activities, as it committed to do in the 2011 MOU. By issuing progress reports each year, Education could have more reasonable assurance that it has the necessary information to assess its performance and to demonstrate results. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. We are sending copies of this report to the appropriate congressional committees; the Chair of the Council on Environmental Quality; the Attorney General, Department of Justice; the Administrators of the Environmental Protection Agency and General Services Administration; the Acting Administrator of the Small Business Administration; the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Housing and Urban Development, the Interior, Labor, Transportation, and Veterans Affairs; and the Acting Secretary of the Department of Homeland Security. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix XV. This report examines (1) the extent to which the 16 working group agencies have developed environmental justice strategic plans and shown progress toward environmental justice goals since 2011; (2) the actions agencies have taken to identify and address environmental justice issues related to their programs, policies, and activities since the executive order was issued in 1994 and the resources they have used to do so in recent years; and (3) the extent to which the Interagency Working Group on Environmental Justice (working group) has collaborated on environmental justice efforts. Sixteen federal agencies and one agency of the Executive Office of the President are involved in environmental justice efforts: the Council on Environmental Quality (CEQ), Environmental Protection Agency (EPA), General Services Administration (GSA), Small Business Administration (SBA), Department of Agriculture (USDA), Department of Commerce (Commerce), Department of Defense (DOD), Department of Education (Education), Department of Energy (DOE), Department of Health and Human Services (HHS), Department of Homeland Security (DHS), Department of Housing and Urban Development (HUD), Department of the Interior (DOI), Department of Justice (DOJ), Department of Labor (DOL), Department of Transportation (DOT), and Department of Veterans Affairs (VA). To address these objectives, we reviewed Executive Order 12898 (Federal Actions to Address Environmental Justice in Minority Populations and Low-Income Populations), the 2011 Memorandum of Understanding on Environmental Justice (MOU), working group documents, and agency environmental justice strategic plans and progress reports, and interviewed federal agency officials about the documents. We also attended the 2018 National Environmental Justice Conference and Training Program, in which leaders from various sectors share ideas and approaches to achieving environmental justice. At this conference, we observed sessions to gain background and context and interviewed some attendees whom we identified and arranged to interview prior to the conference. We also visited sites in Oakland, California, and Richmond, California, to add context to our review with observations of communities with environmental justice issues. We selected these sites because they had minority and low-income populations with environmental and health concerns. Including interviews we conducted at the conference, we conducted 33 interviews with environmental justice stakeholders about federal environmental justice efforts and related issues. Of these interviews, 10 were with representatives from national nonprofit organizations, seven were with representatives from nonprofit groups who work on local issues, six were with university professors, four were with employees of private companies, two were with current or former government officials, and four were with mixed groups of stakeholders. We identified these stakeholders for interviews from our background interviews and document reviews. The views of the stakeholders we interviewed cannot be generalized to all similar stakeholders, but they represent a range of stakeholder perspectives and provide illustrative examples of views of agency efforts. To examine the extent to which the 16 agencies developed environmental justice strategic plans since 2011, we determined which agencies had completed an environmental justice strategic plan after signing the 2011 MOU and which agencies had also updated their plans at EPA’s request in 2016. We made these determinations by reviewing the website of each agency for its environmental justice documents, reviewing the environmental justice strategic plans, and interviewing agency officials about the origin and status of these environmental justice strategic plans. To examine the extent to which the 16 agencies showed progress toward environmental justice goals since 2011, we determined whether each agency had completed annual environmental justice progress reports for each year for fiscal year 2012 through fiscal year 2017 by reviewing the website of each agency to identify these progress reports, reviewing the progress reports we located, and interviewing agency officials about the status and content of these progress reports. We also reviewed the environmental justice strategic plans and progress reports to assess whether agencies included a method to assess progress in accordance with GAO’s leading practices for strategic planning and reporting, including establishing goals and establishing a method to assess progress toward goals. Specifically, we analyzed whether each agency’s environmental justice strategic plan included goals and performance measures or milestones, and whether each agency assessed progress toward these goals using performance measures or milestones in subsequent progress reports. We also interviewed agency officials about their progress toward the goals of Executive Order 12898. To examine the actions the 16 agencies took to identify and address environmental justice issues related to their programs, policies, and activities since the executive order was issued in 1994, we reviewed agency environmental justice strategic plans, progress reports, and related documents to identify illustrative examples of agency efforts in each of the areas outlined in Executive Order 12898 and the 2011 MOU as well as two additional areas identified by agencies. We also interviewed officials from each agency to confirm or gather additional information on these examples. The analysis included a detailed review of the most recent environmental justice strategic plan and progress report for each agency to identify examples of agency actions and a content analysis of the most recent environmental justice strategic plan for each agency. From this review, we (1) counted how many agencies discussed plans to identify and address environmental justice issues related to the areas outlined in the 1994 executive order and 2011 MOU in their most recent environmental justice strategic plan, (2) developed a list of illustrative examples of agency efforts to identify and address environmental justice issues related to these areas, and (3) counted how many agencies provided examples of actions they implemented related to these areas. The examples are not a generalizable sample of the types or instances of agency actions, but illustrate the various ways that different agencies are implementing plans to identify and address environmental justice issues and different approaches to doing so that may be useful for other agencies, the Interagency Working Group on Environmental Justice, and environmental justice stakeholders. We report a minimum count of agencies that provided examples for each area because most agencies did not formally report on progress annually and the information we reviewed does not provide a complete record of agency environmental justice efforts. To examine what resources working group members used to support their environmental justice efforts for fiscal year 2015 through 2018, we obtained and reviewed agency budget justification documents and agency estimates of resources data to determine which agencies (1) had any funding or staffing resources dedicated specifically for environmental justice in their budgets, (2) supported environmental justice efforts with a mix of existing funding and staff from related programs, or (3) did not report any examples of environmental justice efforts or use any resources specifically for any environmental justice efforts. We assessed the reliability of the agencies’ estimated resources data, including for agencies that estimated no resources were used to support any environmental justice efforts, by corroborating it with agency budget justification documents or internal agency budget documentation, interviewing agency officials about the data, and comparing it with information on any reported examples environmental justice efforts. We found it reliable for our purposes of describing which agencies had any resources dedicated specifically for environmental justice in their budgets and of presenting estimates of other funding and staffing resources used to support environmental justice efforts. To determine the extent to which the working group has collaborated on environmental justice efforts, we reviewed working group documents including the group’s fiscal year 2016-2018 Framework for Collaboration and associated progress reports, its published guidance entitled Promising Practices for Environmental Justice Methodologies in NEPA Reviews, and its resource guide entitled Goods Movement Federal Resources Compendium. We also conducted semi-structured interviews with officials from working group committees. We compared the working group’s organization, documents, and actions with key features of collaborative mechanisms that GAO has identified, including clarifying roles and responsibilities, participation, establishing written guidance and agreements, and establishing outcomes and accountability. We selected these features because they were most relevant to the activities of the working group organization. We conducted this performance audit from November 2017 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Agencies provided examples of actions to identify and address environmental justice issues: Improve research and data collection In 2017, the Department of Housing and Urban Development (HUD) and the Environmental Protection Agency (EPA) entered into a memorandum of understanding (MOU) to improve communication and data sharing about public and HUD-assisted housing located near contaminated Superfund sites to help both agencies prioritize actions protecting against human health and environmental risks. The Department of the Interior (DOI) provided an example in which the National Park Service used EPA’s Environmental Justice Mapping and Screening Tool (EJSCREEN) in 2015 to check for populations with respiratory health risks near a prescribed burn area (i.e., a planned, controlled fire to manage wildfire risks) in Jean Lafitte, Louisiana, as part of an environmental assessment (see fig. 1 for example of EJSCREEN display). Promote enforcement of health and environmental statutes The Department of Justice (DOJ) officials told us that its attorneys consider environmental justice issues when pursuing cases to enforce federal environmental laws, and in 2014 it updated and reissued guidance on how its attorneys should identify and address environmental justice issues in their work. For example, DOJ reported in its 2017 progress report that it sought and incorporated community input on resolutions for a 2017 case involving several petrochemical facilities alleged to be violating the Clean Air Act that were located in Texas and Louisiana communities with environmental justice issues. DOJ reported that some of the injunctive relief and monitoring requirements included in the case settlement reflected suggestions made by the community. According to internal DOI guidance from 2018, the Central Hazardous Materials Fund, which supports cleanup of contaminated sites on federal lands through the Comprehensive Environmental, Response, Compensation, and Liability Act, requires projects to be screened for any potentially affected environmental justice communities and for the requesting bureau to work with any communities that are identified near the proposed project. In its 2014 progress report, Commerce reported that the National Oceanic and Atmospheric Administration (NOAA) developed a handbook on procedures for government-to-government consultation with federally recognized Indian tribes and Alaska Native Corporations as part of an effort to facilitate meaningful and timely input from Tribes into federal decisions that directly affect them. In 2013, DOJ and EPA reported seeking and incorporating input from low-income and minority communities on resolutions for several Clean Water Act violations for sewer overflows in cities in Tennessee, Mississippi, and Washington; these resolutions included requiring the cities to address overflows at specific sites impacting these communities and developing Supplemental Environmental Projects for the cities to fix leaking private sewer pipes. Identify differential patterns of consumption of natural resources In its 2016 progress report, DOJ reported that its Environment and Natural Resources Division negotiated a settlement to help improve the passage of steelhead and salmon—fish that are important to the Muckleshoot and Puyallup tribes—on the White River in Washington. In its 2016 progress report, DOI reported that the U.S. Geological Service worked with the Stillaguamish tribe in Washington, to assess the effects of possible wastewater contamination on fish and wildlife in the Stillaguamish River. The Department of Homeland Security (DHS) issued an agency-wide directive on the National Environmental Policy Act (NEPA) implementation in 2014 and the accompanying 2014 NEPA instruction manual included public involvement requirements for populations with environmental justice issues. For agency staff to implement this guidance, DHS included questions about potential environmental justice issues related to the proposed action in its NEPA assessment system. Since at least 2012, as part of the NEPA process for HUD-assisted projects, HUD requires the environmental review record to document any adverse and disproportionate impacts on low-income or minority populations, and steps to engage the community in meaningful participation about mitigating the adverse impacts or moving the project. The General Services Administration’s (GSA) 1999 Public Building Service NEPA Desk Guide includes a section specifically on environmental justice, which states that each GSA NEPA review should include some level of environmental justice analysis. In its 2015 progress report, GSA reported that it continues to consider environmental justice issues for proposed Public Buildings Service projects. The U.S. Department of Agriculture’s (USDA) 1997 Departmental Regulation on Environmental Justice directs USDA component agencies to incorporate environmental justice into their NEPA processes (e.g., Rural Development’s official guidance includes a section on integrating environmental justice and socioeconomic analyses into environmental reviews as part of the NEPA process). Implement Title VI of the Civil Rights Act of 1964 In its 2017 progress report, EPA reported that its External Civil Rights Compliance Office provided training and technical assistance on federal civil rights obligations to local agencies, tribal governments, and 38 states across the agency’s 10 regions through outreach calls and meetings in 2017. Consider impacts from climate change According to the Department of Commerce, NOAA has developed information, tools, and services to help society understand, plan for, and respond to climate variability and change. As part of this effort, NOAA built a web-based resource called Digital Coast, which can be used to identify the risk of potential sea-level rise and inundation to vulnerable populations (e.g., low-income). According to the Department of Energy’s (DOE) 2015 progress report, the 2015 National Environmental Justice Conference and Training Program focused on climate change and climate justice. DOE also issued a 2015 report on the vulnerabilities that tribal energy systems, such as electric grid infrastructure, have to climate change and extreme weather, and announced a grant opportunity to establish clean energy projects and energy efficiency projects on tribal lands. Consider impacts from goods movement In its 2017 progress report, EPA reported prioritizing funding projects to reduce elevated diesel emissions from equipment moving goods and people near seaports and airports through its Diesel Emissions Reduction Act grants. The Department of Transportation’s (DOT) Federal Highway Administration created an Environmental Justice Tools Peer Network to share transportation practitioners’ experiences using EJSCREEN and other relevant data tools in decisions about transportation planning or project development. DOJ officials told us that new attorneys and staff in its Environment and Natural Resources Division—the primary division responsible for prosecuting environmental cases—received training on environmental justice issues. In its most recent environmental justice strategic plan, DOT reported that it offers environmental justice training throughout the agency to help federal employees and grantees ensure compliance with environmental justice policies. For example, in its 2015 progress report, DOT stated that its Federal Highway Administration and Federal Transit Administration offered courses and webinars on such topics as environmental justice fundamentals, planning, and analysis; Title VI; and freight impacts. USDA officials told us that its National Resources Conservation Service developed a webinar in 2014 to assist conservation planners, partners, and technical service providers understand, analyze, and document environmental justice issues related to planned conservation actions under NEPA, such as data sources and potential mitigation measures. In its 2017 progress report, EPA reported holding training sessions for community organizations on how to use EJSCREEN, how to apply for grants, and other strategies and resources to deal with specific environmental justice issues, such as lead exposure and poisoning. Since 2007, DOE has sponsored an annual conference, the National Environmental Justice Conference and Training Program, with support from other agencies, to bring together community leaders; federal, state, and local government representatives; tribal leaders; environmental justice organizations; and others. The conference provides a forum to share information, tools, and strategies for identifying and dealing with specific environmental justice issues that communities may be facing, and agencies in the working group reported participating. Since at least 2012, HUD has offered online training on environmental justice for HUD grantees to help build their capacity to meet environmental review responsibilities for HUD-assisted projects. In 2017, DOI and EPA entered into an MOU to collaborate on environmental justice and economic development issues by assisting underserved communities through academic partnerships, technical assistance, and training, in collaboration with the communities. In its 2016 progress report, the Department of Labor reported that the Employment and Training Administration’s Job Corps, a job training program for low-income and at-risk youth, offers training in fields such as green building and hazardous waste removal. J. Alfredo Gómez, (202) 512-3841 or gomezj@gao.gov. In addition to the individual named above, Susan Iott (Assistant Director), Allen Chan (Analyst-in-Charge), Peter Beck, Hannah Dodd, Juan Garay, Rich Johnson, Matthew Levie, Ben Licht, Cynthia Norris, Amber Sinclair, Kiki Theodoropoulos, and Elise Vaughan Winfrey made key contributions to this report.", "summary": "Environmental justice seeks to address the disproportionately high distribution of health and environmental risks among low-income and minority communities by seeking their fair treatment and meaningful involvement in environmental policy. In 1994, Executive Order 12898 directed 11 federal agencies to identify and address environmental justice issues related to their activities and tasked an interagency working group to coordinate federal environmental justice efforts. In 2011, 16 agencies, including the 11 original agencies, recommitted to planning and reporting on environmental justice efforts by signing an MOU. GAO was asked to review federal environmental justice efforts. This report examines agencies' environmental justice actions, strategic plans and progress reports, and working group collaboration. GAO reviewed agency environmental justice plans, reports, and funding data; interviewed agency officials; and compared working group collaboration to leading collaborative practices. Most of the 16 agencies that are members of the interagency working group on environmental justice—created by Executive Order 12898 in 1994—reported taking some actions to identify and address environmental justice issues, such as creating data tools, developing policies or guidance, and building community capacity through small grants and training. For example, the Environmental Protection Agency (EPA) created a mapping tool that can help identify low-income and minority communities exposed to health or environmental risks. Several agencies, such as EPA and the Departments of Justice, Homeland Security, and the Interior, also developed policies or guidance to analyze environmental justice issues during environmental reviews or enforcement activities. Most of the agencies supported their efforts with funds and staff from related programs, but EPA and the Department of Energy provided funds ($8.3 million in fiscal year 2018) and staff specifically for environmental justice. Agencies' progress toward environmental justice is difficult to gauge, however, because most do not have updated strategic plans and have not reported annually on their progress or developed methods to assess progress. As they agreed to do in a 2011 Memorandum of Understanding (MOU), most of the agencies developed environmental justice strategic plans, but only six have updated them more recently. Few agencies have measures or methods for assessing progress, and the working group has not provided guidance to help agencies with such assessments. The number of agencies issuing annual progress reports has declined (see fig.). Updated strategic plans and annual progress reports, along with guidance on performance measures and methods, would help agencies provide essential information to assess their progress. The working group, chaired by EPA, has developed committees and written agreements to carry out its responsibilities to coordinate agencies' environmental justice efforts, but it is not carrying out several functions in the 1994 Executive Order. GAO has found that collaborative mechanisms, such as the working group, benefit from clear goals, but the working group's organizational documents do not contain clear strategic goals aligned to address the order. Clear strategic goals to carry out the executive order could enhance the group's strategic direction for intergovernmental environmental justice efforts. GAO is making 24 recommendations, including that agencies update environmental justice strategic plans and report on progress annually, and that EPA consult with other working group members to provide guidance on assessing progress and to set strategic goals. Of the 15 agencies with recommendations, eight agreed. Other agencies' responses included partial agreement, disagreement, and no comment. GAO continues to support its recommendations.", "document_type": "gao"}
{"report": "1993. For the version of this requirement that applied to DOL, HHS, and Education, the Stevens Amendment has appeared in all but one of the full appropriations acts passed in Congress since 1993. In proposing the amendment in 1988, Senator Ted Stevens described the role of states, local governments, and the federal government in forging a partnership to share in the costs of many projects and said that the federal contribution should be identified as a matter of taxpayer concern. Further, Senator Stevens said that taxpayers “ought to be informed how much money comes from Federal sources in any program, project, or grant activity.” More recently, the Consolidated Appropriations Act of 2018 and the appropriations for these agencies in 2019 renewed this requirement for DOL, HHS, and Education (see sidebar). total costs of the project or program that will be financed by non-governmental sources. Consolidated Appropriations Act of 2018, Pub. L. No. 115- 141, 132 Stat. 348, div. H, Title V, Sec. 505 (Mar. 23, 2018). The combined amount of grant funding that went to state and local governments from these three departments in federal fiscal year 2017 amounted to approximately $504 billion, or almost 75 percent of the $675 billion total distributed by all federal grant-making agencies to state and local governments that year. HHS had the largest amount of grant outlays to state and local governments with about $455 billion (67.4 percent of the total), Education distributed about $42 billion (6.2 percent), while Labor distributed about $7 billion (1.1 percent). Generally, agencies or their subdivisions provided grantees with the exact text of the Stevens Amendment, paraphrased its language, or in some cases referred grantees to other guidance containing the Stevens Amendment. Figure 1 summarizes what we found at each of the three agencies we reviewed with regard to the Stevens Amendment guidance they provide to grantees. According to ETA officials, grants from DOL’s ETA comprised more than 95 percent of DOL’s $22.1 billion in active grant awards as of October 1, 2018. ETA’s Office of Grants Management (OGM) developed standard terms and conditions that serve as a template for written agreements for grant awards. The terms and conditions template includes language largely similar to the Stevens Amendment, with one instance of paraphrasing, which is permissible under the relevant regulations. ETA’s paraphrased language states that the Stevens Amendment requirements apply to “all non-federal entities receiving federal funds,” whereas the actual Stevens Amendment wording is that the requirements apply to “all grantees receiving federal funds in this Act, including but not limited to state and local governments and recipients of federal research grants.” The Stevens Amendment has been in the agency’s terms and conditions library for grant awards since fiscal year 2014. ETA officials said they added the Stevens Amendment requirements to the terms and conditions because they wanted to ensure that grantees knew about the Stevens Amendment’s existence. ETA also disseminates its standard terms and conditions template to grantees on behalf of five other DOL grant-making subagencies, including the Veterans Employment and Training Service, the Chief Evaluation Office, the Bureau of International Labor Affairs, the Women’s Bureau, and the Office of Disability Employment Policy. According to ETA officials, OGM administers the front-end application processing for grants awarded by these subagencies, while the subagencies are responsible for any post-award grantee oversight. OGM also administers the final grant closeout for these subagencies. According to DOL officials, together with ETA, these subagencies awarded more than 99.8 percent of DOL’s active grant funds as of October 2018. According to DOL officials, three other DOL grant-making subagencies, the Mine Safety and Health Administration (MSHA), the Occupational Safety and Health Administration (OSHA), and the Bureau of Labor Statistics (BLS), administer their sub-agencies’ grant award processes themselves. Officials said that two of these agencies, MSHA and OSHA, disseminate their own separate grant award terms and conditions, and have their own separate guidance for grantees regarding compliance with the Stevens Amendment. For example, OSHA paraphrased the Stevens Amendment language in its terms and conditions. OSHA officials told us that instead of stating the amendment’s requirements in three parts, OSHA broke them out into four requirements that reflect the full content of the Stevens Amendment’s original language. MSHA also had its own terms and conditions that contain the exact language of the Stevens Amendment’s requirements, according to officials. The third agency, BLS, told us that its grantees only produce narrowly focused press releases and that these documents do not fall within the Stevens Amendment description of documents “describing projects or programs funded in whole or in part with Federal money.” BLS officials said that since none of the other qualifying public statements mentioned in the Stevens Amendment are part of BLS grantee operations, BLS cooperative agreements do not produce public statements that qualify for Stevens Amendment compliance. At the department level, HHS publishes a Grants Policy Statement that contains language equivalent to the Stevens Amendment, but it does not quote the amendment verbatim. Consistent with the Stevens Amendment, the Grants Policy Statement provision directs grantees to disclose information on the percentage and dollar amount of federal contributions to grantees’ programs or projects in addition to the same information for nongovernmental sources, but collapses the three Stevens Amendment requirements into two requirements with slight wording changes. Officials told us that HHS expects its operating divisions to follow the Grants Policy Statement together with the relevant HHS regulations, but does not instruct operating divisions on what to include in their grant award terms and conditions. A number of HHS operating divisions provide grantees with grant award terms and conditions that contain the exact language of the Stevens Amendment. Examples include: Centers for Disease Control and Prevention (CDC) - Provides grantees with general terms and conditions for both research and non-research grants and cooperative agreements that include a requirement for an “Acknowledgement of Federal Support” that is an exact re-statement of the Stevens Amendment. Health Resources and Services Administration (HRSA) - Provides grantees with the Standard Form 424 Application Guide (grants application guide), which includes a section that quotes the exact language of the Stevens Amendment. Office of the National Coordinator for Health Information Technology (ONC) - Added a section in 2018 to the terms and conditions section for every Funding Opportunity Announcement that specifically references the Stevens Amendment verbatim. One operating division, the Centers for Medicare and Medicaid Services (CMS) provides grantees with a section of its terms and conditions titled “Public Reporting” that shows the language of the Stevens Amendment, but with the addition of tribal governments to the list of applicable grant recipients. Another operating division, the National Institutes of Health (NIH), publishes its own grants policy statement separate from the one published by HHS. The NIH grants policy statement contains the standard terms and conditions for all NIH grant awards. It uses the same Stevens Amendment guidance language HHS uses, with the same paraphrasing of the language that collapses the three Stevens Amendment requirements into two requirements. Four relevant HHS operating divisions told us that they relied solely on a reference to the HHS Grants Policy Statement to instruct grantees with regard to the Stevens Amendment requirements. This reference made no specific mention of the Stevens Amendment and did not include either the exact or paraphrased Stevens Amendment language in the agencies’ grant agreement terms and conditions or funding opportunity announcement. HRSA’s “Acknowledgement of Federal Funding” provision in its grants application guide contains the exact language of the Stevens Amendment and its requirements. Further, HRSA’s application guide provides grantees with what HRSA officials stated was a sample acknowledgement and disclaimer paragraph written in “plain language” that the operating division developed to assist HRSA grantees in complying with the Stevens Amendment. HRSA officials said that they consulted with HHS’s Office of General Counsel to simplify the language, while ensuring that it met the requirements of the Stevens Amendment. HRSA’s sample acknowledgement and disclaimer paragraph reads, “This supported by the Health Resources and Services Administration (HRSA) of the U.S. Department of Health and Human Services (HHS) as part of an award totaling $XX with xx percentage financed with nongovernmental sources. The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement, by HRSA, HHS or the U.S. Government.” Later in the section, HRSA further defines the Stevens Amendment’s “other documents describing projects or programs” as including, among other things, HRSA-supported documents such as manuals, toolkits, resource guides, case studies, and issues briefs. In addition to HRSA’s efforts to interpret the Stevens Amendment, HRSA posted a web page in October 2018 that provided grantees with additional written guidance and a list of “frequently asked questions” about communicating and acknowledging federal funding. The HRSA web page provided examples of HRSA disclosure statements to show grantees how disclosure language should be drafted to comply with the Stevens Amendment. The web page also featured frequently asked questions, one of which clarified that the disclosure should reflect the overall amount of the grant rather than the cost of developing the publication where the acknowledgement appears. The other frequently asked question directed grantees to consult with HRSA officials if they intend to use language that differs from the examples provided to ensure that their alternative wording complies with the Stevens Amendment requirements. HRSA officials also instructed their grantees on compliance with grant award terms and conditions, including the Stevens Amendment, through informal discussions during workshops and conference calls. HRSA officials provided examples of Stevens Amendment discussions such as a May 2018 Healthy Grants Workshop presentation to grantees, as well as a July 2018 question and answer period during an HRSA conference call with grantees. The grantee conference call featured several HRSA presenters, including one representing the Division of Grants Policy. Officials said that during these technical assistance calls, HRSA wanted to ensure that grantees were made aware of legislative mandates, but the calls were not tailored to focus on a specific mandate. Education grantees that receive discretionary and formula grants are provided with information on the Stevens Amendment through a Grant Award Notification attachment. The attachment is included with the terms and conditions of the grant award and has the exact language of the Stevens Amendment’s requirements, but paraphrases with regard to the types of entities to which the Stevens Amendment applies. Instead of applying the requirements to “all grantees receiving federal funds included in this act including but not limited to state and local governments and recipients of federal research grants” as noted in the Stevens Amendment, Education’s phrasing applies the requirements specifically to “U.S. Department of Education grantees.” According to Education officials, the grant notification process involves providing guidance to grantees and ensuring that they are made aware of various statutory requirements, including the Stevens Amendment. Education officials told us that another way that their agency communicates information about the Stevens Amendment to grantees is through Education’s required post-award conference call, during which the program offices reinforce grant recipients’ need to be aware of the requirements. The regulations that govern DOL, HHS, and Education’s management of grant awards state that “the Federal awarding agency must manage and administer the federal award in a manner so as to ensure that Federal funding is expended and associated programs are implemented in full accordance with U.S. statutory and public policy requirements.” The Stevens Amendment is a statutory requirement that these federal agencies must ensure is implemented by their grantees. Agencies’ management of the grant award so as to ensure implementation of the Stevens Amendment can be accomplished by various means, including the monitoring of grantees, through processes such as reviews of grantee reports and correspondence, desk audits, and grantee site visits. In our review of Stevens Amendment grants management practices at DOL, HHS, and Education, we found that DOL’s ETA had processes in place that were able to identify instances of grantee noncompliance with the Stevens Amendment and demonstrate that noncompliance was being remedied. ETA’s grants management processes took the form of grantee monitoring. Figure 2 below summarizes the Stevens Amendment monitoring practices of DOL, HHS, and Education. ETA officials told us that their subagency’s active grants represented more than 95 percent of DOL’s total active grant dollars, or approximately $21.1 billion. According to ETA officials, their operating plan for grant oversight targets 26 percent of the active ETA grants universe for monitoring each fiscal year, representing approximately 2,100 grants in fiscal year 2019. The regional office staff in each of ETA’s six regional offices conduct a risk analysis of the grants within their regions and assign a risk rating to each grant indicating its risk level. The grant’s risk level, which includes factors such as the dollar amount of the grant award and whether the grantee is on track to meet the grant’s performance goals, determines which grants ETA selects for monitoring and inclusion in its regional monitoring plans for that fiscal year. Each annual regional monitoring plan consists of a list of grants and schedule of ETA staff monitoring reviews. According to ETA, monitoring reviews are used to measure grantee progress toward achieving project goals, identify areas of grantee compliance, offer opportunities for technical assistance to help resolve compliance issues, and ensure that federal funds are used responsibly. ETA conducts these reviews either through an on-site monitoring visit or an “Enhanced Desk Monitoring Review” that is conducted remotely. ETA officials stated the regional monitoring plans are designed to be flexible management tools, and are updated throughout each fiscal year to ensure that ETA meets its operating plan’s goal to monitor 26 percent of its grants annually. According to ETA’s Grantee Handbook, upon completion of the monitoring review, ETA drafts a monitoring report to each of the grantees reviewed. The monitoring report includes, among other things, compliance findings and the required grantee corrective action for any noncompliance with the findings, along with the due date for the corrective action. In response to our request for examples of grantee noncompliance with the Stevens Amendment requirements, ETA officials from each of the agency’s six regional offices conducted a manual search of monitoring reports from fiscal years 2016 and 2017. ETA officials in four of the six ETA regional offices located monitoring reports with a finding stating that grantees’ public materials did not include the Stevens Amendment’s required language or information to properly identify the project’s federal funding dollar amount and the project’s percentage of federal and nongovernmental funding. Three of the four monitoring reports provided the grantee with the exact language of the Stevens Amendment and instructed the grantee to ensure that statements, such as brochures, promotional materials, and other public announcements, contain a statement that identifies the project’s funding sources in accordance with the three requirements of the Stevens Amendment. In the fourth monitoring report, while finding that the grantee did not include the required funding source statement in its documents, ETA’s comments in the monitoring report did not provide the grantee with the full language of the Stevens Amendment and had omitted the requirement to provide the percentage and dollar amount of costs financed by nongovernmental sources. For each of these examples, ETA officials showed that the grantees subsequently corrected their documents to bring them into compliance with the Stevens Amendment requirements. In the fourth example, the grantee’s subsequent inclusion of the required funding source statement in its documents showed that the program was 100 percent funded by federal dollars. In August 2018, ETA also created a “Core Monitoring Guide” that references the Stevens Amendment requirements as an element to be monitored by ETA officials when speaking with grantees. ETA intended this guide to be used as a tool in the on-site review of a grantee’s activities, and it provides officials with a series of checklists as well as the steps to take when conducting monitoring. For Stevens Amendment compliance, the guide includes a “Question for Review and Discussion” that uses the exact language of the Stevens Amendment. However, the Stevens Amendment is only one issue among many addressed in the guide, and ETA officials said they do not have the resources to audit all of the elements included in the guide. ETA officials said that their grant reviewers select from one to four sections of the guide to use when conducting monitoring, depending on the nature of the grant, and that the choice of which items to monitor is based on a risk analysis of the grantee and the grant projects’ quarterly financial reports. ETA officials acknowledged that the scope of their monitoring overall is limited to 26 percent of their grant universe for a given fiscal year, therefore the extent of noncompliance among ETA grantees cannot be determined. Of the eight DOL subagencies we spoke to other than ETA, two, OSHA and BLS, stated that they did not monitor grantees for compliance with Stevens Amendment requirements. These subagencies’ officials said they did not monitor for Stevens Amendment compliance because monitoring is not explicitly required under the statute and, in the case of BLS, because it believes that the type of press releases generated by their grantees do not fall within the scope of the Stevens Amendment. Six of the eight DOL subagencies told us that they conducted grantee compliance monitoring. However, based on the information and documents provided by officials from these six subagencies, they have not demonstrated that they have processes to manage grantees’ compliance with the Stevens Amendment. For example, the Chief Evaluation Office, the Bureau of International Labor Affairs (ILAB), and the Veterans’ Employment and Training Service stated that they do not track the extent of grantee compliance; and the Mine Safety and Health Administration said that it does not maintain records of grantee compliance with the Stevens Amendment. In addition, both ILAB and the Women’s Bureau, while stating that they conducted monitoring of grantee compliance with the Stevens Amendment, provided examples of grantee disclosures that did not meet all of its requirements. Further, the Office of Disability and Employment Policy (ODEP) said that all of its grantees were fully compliant with the Stevens Amendment, but produced no examples of grantee disclosures. The Uniform Administrative Requirements that govern certain federal agencies, including DOL, state with regard to the management of grants that “the Federal awarding agency must manage and administer the Federal award in a manner so as to ensure that Federal funding is expended and associated programs are implemented in full accordance with U.S. statutory and public policy requirements.” Other than ETA, DOL’s subagencies have not developed the processes needed to manage and administer grantees’ compliance with the Stevens Amendment. Without these processes, these DOL subagencies are not able to ensure that grant programs are being implemented by grantees in full accordance with the statutory requirements of the Stevens Amendment. At the department level, HHS officials said that they have no knowledge about whether their operating divisions conduct monitoring and enforcement of the Stevens Amendment, and they did not collect information from their operating divisions on grantee compliance with the Stevens Amendment’s requirements. According to HHS officials, any efforts to manage grant awards for adherence to Stevens Amendment requirements would be carried out by staff at the agency’s 10 relevant operating divisions. HHS officials said that operating divisions have an obligation to monitor their grantees for compliance with all of the agency’s standard grant award terms and conditions, which includes the Stevens Amendment. Of the 10 relevant HHS operating divisions we spoke with in our review, officials from eight of them—the Administration for Children and Families (ACF), the Agency for Healthcare Research and Quality (AHRQ), the Centers for Disease Control and Prevention (CDC), the Food and Drug Administration, the Health Resources and Services Administration (HRSA), the National Institutes of Health (NIH), the Office of the Assistant Secretary for Health (OASH), and the Substance Abuse and Mental Health Services Administration—told us that they did not monitor grantees’ compliance with Stevens Amendment requirements. Two of these operating divisions, ACF and AHRQ, further stated that the Stevens Amendment did not require them to monitor for grantee compliance. These operating divisions maintained the position that they are not required to monitor for grantee compliance despite HHS policy regarding operating division monitoring of grants that states “…to fulfill their role in regard to the stewardship of Federal funds, OPDIVs monitor their grants to identify potential problems and areas where technical assistance might be necessary. This active monitoring is accomplished through review of reports and correspondence from the recipient, audit reports, site visits, and other information available to the OPDIV.” As mentioned earlier in this report, agencies are required to manage grant awards and have a number of possible means available to do so—including grant monitoring. However, grant monitoring is not explicitly required by the Stevens Amendment. NIH officials stated that they do not specifically monitor for Stevens Amendment compliance and that NIH officials have not received any reports of noncompliance with the Stevens Amendment. They said they would address any non-compliance issues if they were raised. Similar to NIH, HRSA officials told us that they conduct grantee monitoring, but do not specifically review grantee documents for compliance with Stevens Amendment requirements unless there is a cause for concern regarding noncompliance. Similarly, CDC officials said that their grantee monitoring practices do not specifically target Stevens Amendment compliance. CDC officials further explained that while grant program officers may find instances of noncompliance during a grant review, it would be tangential to other issues more central to the focus of the grant review, such as grantee financial performance and goal accomplishment. Officials from OASH stated that while they do not specifically review grantees’ written statements for Stevens Amendment compliance, they provide grantees with guidance regarding how to comply with its requirements. For example, a grantee asked whether a Stevens Amendment acknowledgement statement had to be included on billboards the recipient rented to promote their program’s services. OASH determined that the grantee did not need to include the statement on the billboards. ONC officials told us that all of their grantees were in compliance with the Stevens Amendment. ONC officials stated that while they do not specifically look for Stevens Amendment compliance, it was their belief that ONC monitoring practices would identify instances of noncompliance for their small number of grantees. ONC officials told us their belief is based on interactions with a wide range of grantees’ employees during monitoring visits that seek to ensure that all compliance issues among their grantees are addressed. The remaining HHS operating division in our review, CMS, told us that its monitoring processes include reviews for Stevens Amendment requirements and that CMS had a process for reviewing grantee documents. According to HHS policy, the results and accomplishments of the activities CMS funds should be made public and CMS requires grantees to make the results and accomplishments of their activities available to the research community and to the public at large. The grantee must submit any materials to CMS in advance of publication, including brochures, recruitment materials, informational materials, advertisements, website copy, website pages, videos, and op-ed articles that report results from or describe information obtained through the grant award. CMS officials told us they reviewed for Stevens Amendment compliance, and provided us with examples of materials they said were from grantees that were in compliance. However, in our analysis of the sample grantee materials from CMS, we found that the grantees were not in compliance with the cost requirements of the Stevens Amendment. Despite the claims and efforts of some HHS operating divisions with regard to monitoring for Stevens Amendment compliance, none of HHS’s operating divisions could demonstrate that they had a process to manage and administer grantees’ compliance with the Stevens Amendment requirements. In addition to the previously-mentioned Uniform Administrative Requirements applicable to all grant awards, HHS regulations that govern the agency’s grant making state that, “The Federal awarding agency must manage and administer the Federal award in a manner so as to ensure that Federal funding is expended and associated programs are implemented in full accordance with U.S. statutory and public policy requirements.” Further, these regulations also state, “The Federal awarding agency must communicate to the non- Federal entity all relevant public policy requirements, including those in general appropriations provisions, and incorporate them either directly or by reference in the terms and conditions of the Federal award.” Neither HHS, nor its operating divisions, had developed processes to manage and administer grantees’ Stevens Amendment compliance. Without having processes to manage and administer their grantees’ compliance with the Stevens Amendment, which is included in HHS’s appropriations provisions, there is no way for HHS or its operating divisions to ensure that grantees are in full accordance with the statutory requirements of the Stevens Amendment appropriations provision and the agency-communicated conditions of the federal award. Further, without monitoring grants in accordance with their Grants Policy Statement, HHS and its operating divisions are not able to identify potential problems related to grantees’ Stevens Amendment compliance. Education officials stated that its program offices do not explicitly track individual grantees for Stevens Amendment compliance. Education officials told us that their grant review process does not collect Stevens Amendment documentation nor do they gather information regarding the extent of grantee compliance with the appropriations provision. As a consequence, Education cannot determine the extent of their grantees’ compliance with the requirements of the Stevens Amendment. Representatives from Education’s Office of General Counsel stated that Education has an “obligation to correct” instances of Stevens Amendment noncompliance, but does not have an “obligation to monitor” its grantees to determine whether they are in compliance. Education officials told us that due to limited resources, they use risk assessment results to identify and prioritize which items among their standard terms and conditions they will monitor during the course of a grant review. Officials told us they had not received any complaints related to the Stevens Amendment. The uniform regulations that govern federal agencies, including Education’s management of grants, state that “the Federal awarding agency must manage and administer the federal award in a manner so as to ensure that Federal funding is expended and associated programs are implemented in full accordance with U.S. statutory and public policy requirements.” Education has not developed the processes it needs to manage and administer grantees’ compliance with the Stevens Amendment which is included in Education’s appropriations provisions. Without these processes, Education is not able to ensure that grant programs are being implemented by grantees in full accordance with the statutory requirements of the Stevens Amendment appropriations provision and the agency-communicated conditions of the federal award. With two exceptions, the subagencies and operating divisions we reviewed that stated they conducted monitoring had no information on the methods used by grantees to calculate the federal funding dollar amounts or funding percentage figures required by the Stevens Amendment. As an example, DOL’s ETA officials told us that they do not know how the dollar amounts reported by grantees were calculated, and they have not inquired about the level of detail factored into indirect costs involving the grantee organization’s structure and the percentage of funds spent on salaries. In addition, officials at DOL’s ILAB said that it is not always clear how grantees calculate these costs, and the Stevens Amendment does not provide specific guidance on how costs should be determined. Officials also noted that some grantees expressed confusion regarding the requirements and how to calculate the total federal funds, including in cases where there may be collaboration across federally-funded programs. Similarly, officials from HHS’s NIH operating division noted that calculations can be difficult given that research programs can have multiple funding streams that feed into a grant project and grantees’ research portfolios are now more complex than they have been in the past. Officials at one DOL subagency, ODEP, said that grantees calculate the total funds received in the grant awarding document and that these funds include negotiated indirect cost rates. The remainder of the DOL subagencies and HHS operating divisions that produced examples of either compliance or noncompliance with the amendment did not have information on how grantees made their disclosure calculations. In addition, officials at one HHS operating division, HRSA, said the HRSA Notice of Award lists the total federal and non-federal amounts for the grant project or program. Grantees can use this information to calculate the percentage of federal funding and nongovernmental funding. However, in the Stevens Amendment compliance examples that HRSA provided to us, this calculation was not necessary because these projects were 100 percent funded by the HRSA grant award. In addition, HRSA officials told us that they are not aware of any other methods that grantees would need to use to arrive at the percentage. With regard to indirect costs, HRSA officials said that these costs are already included in the federal award amount and, therefore, any calculation of funding percentage should already account for the inclusion of both direct and indirect costs. Congress has repeatedly taken action to include the Stevens Amendment requirements with agencies’ appropriations. Ensuring grantee compliance with accountability requirements is achieved through investment of federal agency resources that reflect decisions regarding how best to ensure efficient and effective use of grant funds while reinforcing statutory requirements. DOL’s largest grant making subagency, ETA, showed that its grantee review processes were, to some extent, actively monitoring for Stevens Amendment compliance and that when ETA found compliance issues, it was able to provide grantees with the technical assistance needed to correct them. While a couple of HHS operating divisions showed some evidence that they were enhancing their guidance to grantees with regard to the Stevens Amendment, none of the operating divisions could demonstrate that they had a process to manage and administer grantees’ compliance with the Stevens Amendment requirements. Education officials stated that while their agency does not have an “obligation to monitor” its grantees to determine whether they are in compliance with the Stevens Amendment, they do have an “obligation to correct” instances of noncompliance if brought to their attention. While none of the agencies in this review can determine the extent of their grantees’ compliance with the Stevens Amendment, DOL’s ETA has monitored grantee compliance with the provision, and when noncompliance is found, has taken steps to bring their grantees into compliance. However, with no such processes in place, the remaining DOL subagencies, HHS’s operating divisions, and Education are not able to manage or administer grantee compliance with the Stevens Amendment appropriation provision so as to ensure that grant funds are being expended in full accordance with these statutory and regulatory requirements. We are making a total of three recommendations, one to each of the three agencies in our review, to take steps to manage grantees’ compliance with the Stevens Amendment. Specifically: The Secretary of Labor should direct its subagencies, other than ETA, to design and implement a process to manage and administer grantees’ compliance with the Stevens Amendment, including determining to what extent to provide guidance to grantees on calculations. (Recommendation 1) The Secretary of Health and Human Services should direct its operating divisions to design and implement processes to manage and administer grantees’ compliance with the Stevens Amendment, including determining to what extent to provide guidance to grantees on calculations. (Recommendation 2) The Secretary of Education should design and implement a process to manage and administer grantees’ compliance with the Stevens Amendment, including determining to what extent to provide guidance to grantees on calculations. (Recommendation 3) We provided a draft of this report to DOL, HHS, and Education for review and comment. We received written concurrence from DOL, and written comment letters from DOL’s OSHA, HHS, and Education. The comment letters are reprinted in appendixes I, II, and III, respectively and are summarized below. DOL stated that it concurs with our recommendation. OSHA provided written comments and stated that it generally agreed with GAO’s recommendation. OSHA said it will take steps to establish processes to monitor grantee compliance with Stevens Amendment requirements, to include reviewing what assistance the agency can provide to grantees on how to calculate funding percentages. OSHA further stated that it has begun updating its grant and cooperative agreement instructions to include the Stevens Amendment language verbatim, rather than paraphrasing the language, and is adding guidance to grant monitoring guidelines to assist OSHA’s Regional Offices in reviewing compliance with the Stevens Amendment. DOL subagencies ILAB, BLS, ETA, and ODEP also provided technical comments, which we incorporated into the report where appropriate. In its written comments, HHS stated that it concurs with our recommendation and would implement the recommendation to the fullest extent feasible. HHS officials said they would direct all operating divisions to design a process for implementing and monitoring the Stevens Amendment and would update HHS grants policy to reflect this new process. HHS also provided technical comments, which we incorporated into the report where appropriate. Education provided written comments stating that it did not concur with our recommendation, but would consider enhancing its existing approach to compliance with the Stevens Amendment. We reiterate our recommendation that Education should design and implement a process to manage and administer grantees’ compliance with the Stevens Amendment, including determining to what extent to provide guidance to grantees on calculations. Education had three concerns regarding the recommendation. First, Education said that our recommendation is not based on any evidence of noncompliance with the Stevens Amendment by Education grantees. As noted in our report, we found that Education lacks information regarding whether its grantees are, or are not, complying with the requirements of the Stevens Amendment. As indicated in this report, Education officials told us that they do not collect documentation from grantees to monitor their compliance with the Stevens Amendment, nor do they analyze information regarding the extent of grantee compliance with the Stevens Amendment. As a consequence, Education does not know the extent to which its grantees are or are not complying with the statutory requirements of the Stevens Amendment. Without this knowledge, Education does not have assurance that its grant awards are managed and administered in accordance with federal regulations. Second, Education referred to its tiered risk-based approach to grantee monitoring that balances compliance requirements with limited monitoring resources in alignment with the President’s Management Agenda. According to Education, implementation of the recommendation would require them to devote limited resources to managing and administering grantee compliance with the Stevens Amendment when there is no evidence of grantee noncompliance. We acknowledge that the cross- agency priority goal in the President’s Management Agenda refers to maximizing the value of grant funding by applying a risk-based, data- driven framework that balances compliance requirements with demonstrating successful results. However, because Education does not collect information or documentation on this aspect of grantee compliance, it lacks the data needed to make an informed risk-based assessment with regard to monitoring for Stevens Amendment compliance. The recommendation could be implemented within the context of Education’s risk-based approach to grantee monitoring as long as Education gathers the grantee compliance information needed to apply their risk-based, data-driven framework. Third, Education said that it has already taken numerous steps to make its process for awarding and overseeing grant funds transparent to the public. However, these steps do not eliminate the legal requirements that grantees must comply with the Stevens Amendment, and that federal agencies, including Education, must manage and administer the federal award in a manner that is fully in accordance with statutory requirements. Education did state that it would consider enhancing its existing approach to Stevens Amendment compliance with actions that further explain the requirements to grant recipients. While such efforts could enhance grantees’ understanding of the Stevens Amendment, they would not give Education the grantee compliance information it needs to apply to a risk- based, data-driven framework or to manage and administer its grant awards in accordance with federal regulations. For all of these reasons we continue to believe that our recommendation to Education is valid and that Education should fully implement it. We are sending copies of this report to the Secretaries of Labor, Health and Human Services, and Education, as well as interested congressional committees and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or sagerm@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. In addition to the contact named above, Tom James (Assistant Director), Anthony Bova (Analyst-in-Charge), Jacqueline Chapin, Jehan Chase, Robert Robinson, Wesley Sholtes, and Walter Vance made key contributions to this report.", "summary": "Since 1989, an appropriations provision, colloquially known as the “Stevens Amendment,” has reflected Congress's longstanding effort to ensure transparency and accountability in federal grant spending. GAO was asked to review agency guidance and grantee compliance related to the Stevens Amendment. This report (1) describes the guidance DOL, HHS, and Education provide to grantees regarding the Stevens Amendment; (2) examines the extent to which DOL, HHS, and Education are managing grantees' compliance with the Stevens Amendment; and (3) describes what is known about how grantees calculate the dollar amounts and percentages of their federal and nongovernmental funding disclosures. GAO asked for agency guidance documents, reviewed monitoring reports, interviewed officials on agencies' Stevens Amendment oversight efforts, and asked agencies how grantees calculate funding amounts. The Stevens Amendment is an appropriations provision that requires grantees of the Departments of Labor (DOL), Health and Human Services (HHS), and Education (Education) to disclose for a grant program the percent of the costs financed with federal funds, the federal dollar amount, and the percentage and dollar amount financed by nongovernmental funds. The provision requires that recipients of grants funded by DOL, HHS, and Education make certain funding disclosures when issuing statements, press releases, bid solicitations, and other documents describing their grant project or program. DOL, HHS, and Education generally provide written guidance to grantees with the exact text of the Stevens Amendment or a paraphrased equivalent. In addition, a number of operating divisions within HHS referenced the HHS Grants Policy Statement, which includes language equivalent to the Stevens Amendment, as a way to instruct grantees. One HHS operating division, the Health Resources and Services Administration, provided grantees with additional guidance in the form of a web page that contained examples of funding disclosure statements and frequently asked questions intended to clarify the Stevens Amendment's requirements. One DOL subagency, the Employment and Training Administration (ETA), whose active grants represented more than 95 percent of DOL's total grant dollars, had processes for managing grantees' compliance that were able to identify instances of grantee noncompliance with Stevens Amendment requirements. ETA's operating plan for grant oversight targets 26 percent of its active grants for risk-based monitoring each fiscal year, representing approximately 2,100 grants in fiscal year 2019. The other DOL subagencies either stated that they did not monitor grantees for compliance with Stevens Amendment requirements or did not have processes in place for managing grantee compliance with the requirements of the Stevens Amendment. Most HHS operating divisions said they did not review grantees for Stevens Amendment compliance. Education also did not monitor for grantee compliance with the Stevens Amendment's requirements. Regulations governing federal agencies' management of grants require federal agencies to manage and administer the federal award in a manner that ensures that programs are implemented in full accordance with U.S. statutory and public policy requirements. Without processes for managing compliance, some DOL subagencies, HHS operating divisions, and Education are unable to ensure that grant programs are being implemented by grantees in full accordance with the statutory requirements of the Stevens Amendment. Most of the subagencies and operating divisions monitoring compliance did not gather information from grantees about how the grantees calculate the dollar amounts and percentages in their Stevens Amendment funding disclosures. For example, DOL's ETA officials said that they do not know how the dollar amounts reported by grantees were calculated, and have not inquired about the level of detail factored into indirect costs involving the grantee organization's structure and the percentage of funds spent on salaries. Similarly, officials from HHS's National Institutes of Health operating division noted that calculations can be difficult given that a research program can have multiple funding streams that feed into a grant project and grantees' research portfolios are now more complex than they have been in the past. GAO recommends that DOL subagencies (other than ETA), HHS operating divisions, and Education design and implement processes to manage grantees' compliance with the Stevens Amendment. In responding to the report, DOL, one DOL subagency, and HHS agreed with GAO's recommendation. Education disagreed with GAO's recommendation, citing limited monitoring resources and other reasons. GAO believes the recommendation should be fully implemented, as discussed in the report.", "document_type": "gao"}
{"report": "We have previously reported on various aspects of national preparedness, including examining the extent to which FEMA programs encourage disaster resilience and identifying gaps in federal preparedness capabilities. We have found that when federal, state, and local efforts aligned to focus on improving disaster resilience and preparedness, there was a noticeable reduction in the effects of the disaster. However, our prior and ongoing work also highlight opportunities to improve disaster resilience and preparedness nationwide. Hazard mitigation is a key step in building resilience and preparedness against future disasters. In July 2015, we found that states and localities experienced challenges when trying to use federal funds to maximize resilient rebuilding in the wake of a disaster. In particular, they had difficulty navigating multiple federal grant programs and applying federal resources towards their most salient risks because of the fragmented and reactionary nature of the funding. In our 2015 report, we recommended that the Mitigation Framework Leadership group—an interagency body chaired by FEMA—create a National Mitigation Investment Strategy to help federal, state, and local officials plan for and prioritize disaster resilience. As of May 2019, according to FEMA officials, the Mitigation Framework Leadership group is on track to address the recommendation, and they expect the strategy to be published by July 2019. In September 2017, we reported that the methods used to estimate the potential economic effects of climate change in the United States—using linked climate science and economics models—could inform decision makers about significant potential damages in different U.S. sectors or regions, despite the limitations. For example, for 2020 through 2039, one study estimated between $4 billion and $6 billion in annual coastal property damages from sea level rise and more frequent and intense storms. We found that the federal government has not undertaken strategic government-wide planning on the potential economic effects of climate change to identify significant risks and craft appropriate federal responses. As a result, we recommended the Executive Office of the President, among others, should use information on the potential economic effects of climate change to help identify significant climate risks facing the federal government and craft appropriate federal responses, such as establishing a strategy to identify, prioritize, and guide federal investments to enhance resilience against future disasters; however, as of June 2019, officials have not taken action to address this recommendation. In November 2017, we found that FEMA had taken some actions to better promote hazard mitigation as part of its Public Assistance grant program. However, we also reported that more consistent planning for, and more specific performance measures related to, hazard mitigation could help ensure that mitigation is incorporated into recovery efforts. We recommended, among other things, that FEMA (1) standardize planning efforts for hazard mitigation after a disaster and (2) develop performance measures for the Public Assistance grant program to better align with FEMA’s strategic goal for hazard mitigation in the recovery process. FEMA concurred with our recommendations, and as of March 2019, officials have reported taking steps to increase coordination across its Public Assistance, mitigation, and field operations to ensure hazard mitigation efforts are standardized and integrated into the recovery process. Additionally, FEMA officials reported taking actions to begin developing disaster-specific mitigation performance measures. However, FEMA has yet to finalize these actions, such as by proposing performance measures to FEMA senior leadership. As such, we are continuing to monitor FEMA’s efforts to address these recommendations. In March 2011, we reported that FEMA had not completed a comprehensive and measurable national preparedness assessment of capability gaps—for example the amount of resources required to save lives, protect property and the environment, and meet basic human needs after an incident has occurred. Developing such an assessment would help FEMA to identify what capability gaps exist and what level of resources are needed to close such gaps. Accordingly, we suggested that FEMA complete a national preparedness assessment to evaluate capability requirements and gaps at each level of government to enable FEMA to prioritize grant funding. As of December 2018, FEMA had efforts underway to assess urban area, state, territory, and tribal preparedness capabilities to inform the prioritization of grant funding; however, the agency had not yet completed a national preparedness assessment with clear, objective, and quantifiable capability requirements against which to assess preparedness. We are continuing to monitor FEMA’s efforts to complete such an assessment. Furthermore, in March 2015, we reviewed selected states’ approaches to budgeting for disaster costs to help inform congressional consideration of the balance between federal and state roles in funding disaster assistance. Specifically, we reported that none of the 10 states in our review maintained reserves dedicated solely for future disasters, and some state officials reported that they could cover disaster costs without dedicated disaster reserves because they generally relied on the federal government to fund most of the costs associated with disaster response and recovery. In response to the 2017 disasters, we also have ongoing work to review national preparedness capabilities to assist communities in responding to and recovering from disasters. Based on our preliminary observations, some states and localities we interviewed reported that while they are prepared to deal with immediate response issues in the aftermath of a disaster, gaps exist in their capacity to support longer term recovery. One reason for this, according to these state and local officials, is because federal preparedness grant funds are largely dedicated to maintaining response capabilities and sustaining personnel costs for local emergency management officials. While these preparedness grants fund critical elements of the national preparedness system, there are some limitations to using them. Specifically, some state and local officials told us that the preparedness grant activities are generally focused on terrorism issues rather than all-hazards. In addition, they reported that the preparedness grants are generally spent on maintaining response capabilities rather than to enhance their capacity for disaster recovery—such as additional training and exercises. In addition to the state, territory, and urban region assessments that FEMA is conducting, FEMA is currently in the process of developing the first national Threat and Hazard Identification and Risk Assessment. This national assessment may help FEMA and policymakers better understand how to target federal resources in a way that enhances the nation’s capacity to respond and recover from future catastrophic or sequential disasters. We are continuing to evaluate national preparedness efforts and plan to report on FEMA’s Threat and Hazard Identification and Risk Assessment process in January 2020. In September 2018, we reported that the response to the 2017 hurricanes and wildfires in Texas, Florida, and California showed progress made since the 2005 federal response to Hurricane Katrina. We also found that FEMA coordinated closely with Texas, Florida, and California emergency management officials and other federal, local, and volunteer emergency partners to implement various emergency preparedness actions prior to the 2017 disasters in each state, and to respond to these disasters. According to FEMA and state officials, these actions helped officials begin addressing a number of challenges they faced such as meeting the demand for a sufficient and adequately-trained disaster workforce and complex issues related to removing debris in a timely manner after the hurricanes and wildfires. In contrast, we also reported in September 2018, that in Puerto Rico and the USVI a variety of challenges—such as the far distance of the territories from the U.S. mainland, limited local preparedness for a major hurricane, and outdated local infrastructure—complicated response efforts to hurricanes Irma and Maria. Many of the challenges we identified are also described in FEMA’s 2017 Hurricane Season FEMA After-Action Report, including: the sequential and overlapping timing of the three hurricanes—with Maria being the last of the three—caused staffing shortages and required FEMA to shift staff to the territories that were already deployed to other disasters; the far distance of both territories from the U.S. mainland complicated efforts to deploy federal resources and personnel quickly; and the incapacitation of local response functions due to widespread devastation and loss of power and communications, and limited preparedness by Puerto Rico and the USVI for a category 5 hurricane resulted in FEMA having to assume response functions that territories would usually perform themselves. We also reported that FEMA’s 2017 Hurricane Season FEMA After-Action Report noted that FEMA could have better leveraged information from preparedness exercises in the Caribbean, including a 2011 exercise after- action report for Puerto Rico which indicated that the territory would require extensive federal support during a large scale disaster in moving commodities from the mainland to the territory and to distribution points throughout. In our September 2018 report, we also found that FEMA’s efforts in Puerto Rico after Hurricane Maria were the largest and longest single response in the agency’s history. According to FEMA, the agency’s response included, among other things, bringing in approximately $1 billion in food and supplies; and distributing food, commodities, and medicine via approximately 1,400 flights, which constituted the longest sustained air operations in U.S. disaster history. FEMA officials explained that the agency essentially served as the first responder in the early response efforts in Puerto Rico, and many of services FEMA provided—such as power restoration, debris removal, and commodity distribution—were typically provided by territorial or local governments. We also reported in September 2018, that in the USVI, recent disaster training and the pre-positioning of supplies due to the anticipated impact of Hurricane Irma facilitated the response efforts for Hurricane Maria, which made landfall less than two weeks later. According to FEMA’s federal coordinating officer, the lead federal official in charge of response for the USVI, the federal government deployed assets, including urban search and rescue teams and medical assistance teams. In addition, due to the sequence of Hurricane Irma hitting the USVI immediately before Hurricane Maria, the Department of Defense (DOD) already had personnel and resources (i.e., ships) deployed to the area, which enabled DOD to respond to Hurricane Maria faster than it otherwise would have. Additional challenges we have reported on regarding response operations have included providing short-term housing and sheltering for disaster survivors. The Department of Homeland Security’s (DHS) 2017 National Preparedness Report states that providing effective and affordable short- term housing for disaster survivors has been a longstanding and continuing challenge. For example, following the California wildfires, local officials faced challenges identifying shelter for displaced survivors, in part due to a housing shortage that existed before the wildfires. Federal, state, and local officials formed housing task forces which facilitated a joint decision-making approach to address these challenges. While this approach has enabled the state to meet its most pressing short-term housing needs, according to FEMA officials, the state faces other challenges in the long term. For example, FEMA officials in the region covering California told us that because of the nature of damage following a wildfire and because of housing shortages in California, some of FEMA’s forms of housing assistance have been less relevant in the wake of the California wildfires than for other disasters. We will continue to evaluate these and other challenges and plan to report in fall 2019. We also have ongoing work to review efforts to provide mass care— which includes sheltering, feeding and providing emergency supplies— following the 2017 hurricanes. Our preliminary observations indicate that during and immediately following the hurricanes, the number of people seeking public shelters outpaced the capacity. In Texas and Florida, emergency managers we spoke with described having unprecedented numbers of residents needing shelters but not always enough staff initially to operate the shelters. In Texas, Puerto Rico, and the USVI, hurricanes Harvey, Irma, and Maria flooded or destroyed many buildings planned for use as shelters, according to emergency management and local government officials in these areas. As a result, some remaining shelters were at maximum capacity. In the USVI, residents of some public housing units that had sustained significant damages sought help at the territory’s Department of Human Services because there was no more space in the shelters, according to local government officials. While they were turned away from the shelters, these families were able to take refuge in the lobby of the Department of Human Services building. We will continue to evaluate these and other challenges and plan to report in summer 2019. In December 2018 and April 2019, we reported that, in response to hurricanes Harvey, Irma, and Maria, as well as the 2017 California wildfires, FEMA and other federal partners relied heavily on advance contracts—which are established before a disaster to provide for life- sustaining goods and services such as food, water and transportation typically needed immediately after a disaster—and post disaster contracts—which can be used for various goods and services, such as debris removal and installation of power transmission equipment. FEMA is required to coordinate with states and localities and encourage them to establish their own advance contracts with vendors. In December 2018, we reported on inconsistencies we found in that coordination and in the information FEMA used to coordinate with states and localities on advance contracts. As a result of this and other challenges identified, we made nine recommendations to FEMA, including that it update its strategy and guidance to clarify the use of advance contracts, improve the timeliness of its acquisition planning activities, revise its methodology for reporting disaster contracting actions to Congress, and provide more consistent guidance and information for contracting officers in coordinating with states and localities to establish advance contracts. FEMA concurred with all of these recommendations, and we are continuing to monitor its efforts to implement each recommendation. Furthermore, in April 2019, we reported on challenges that we found in the federal government’s use of post-disaster contracts. These challenges included a lack of transparency about contract actions, challenges with requirements development, and with interagency coordination. In our report, we found that FEMA had begun taking some steps to address the consistency of post-disaster contract requirements with contracting officers, but that inaccurate or untimely estimates in the contracts we reviewed sometimes resulted in delays meeting the needs of survivors. As a result of our findings in this report, we made 10 recommendations to FEMA and other federal agencies that use these post-disaster contracts related to improving the management of such contracts. FEMA and other agency officials concurred with nine of the recommendations and have reported taking actions to begin implementing them. We will continue to monitor FEMA’s progress in fully addressing these recommendations. FEMA provides multiple forms of disaster recovery assistance after a major disaster has been declared, including Public Assistance and Individual Assistance. Through these grant programs, FEMA obligates billions of dollars to state, tribal, territorial, and local governments, certain nonprofit organizations, and individuals that have suffered injury or damages from major disaster or emergency incidents, such as hurricanes, tornados, or wildfires. In September 2016, we reported that, from fiscal years 2005 through 2014, FEMA obligated almost $46 billion for the Public Assistance program and over $25 billion for the Individual Assistance program. According to FEMA’s May 2019 Disaster Relief Fund report, total projected obligations through fiscal year 2019 for the Public Assistance and Individual Assistance programs for just the 2017 hurricanes—Harvey, Irma, and Maria—are roughly $16 billion and $7 billion, respectively. Given the high cost of these programs, it is imperative that FEMA continue to make progress on the challenges we have identified in our prior and ongoing work regarding its recovery efforts. FEMA’s Public Assistance program provides grants to state, tribal, territorial, and local governments for debris removal; emergency protective measures; and the repair, replacement, or restoration of disaster-damaged, publicly owned facilities. It is a complex and multistep program administered through a partnership among FEMA, the state, and local officials. Prior to implementing the Public Assistance program, FEMA determines a state, territorial or tribal government’s eligibility for the program using the per capita damage indicator. In our September 2018 report on federal response and recovery efforts for the 2017 hurricanes and wildfires, we reported on FEMA’s implementation of the Public Assistance program, which has recently undergone significant changes as a result of federal legislation and agency initiatives. Specifically, we reported on FEMA’s use of its redesigned delivery model for providing grants under the Public Assistance program, as well as the alternative procedures for administering or receiving such grant funds that FEMA allows states, territories, and local governments to use for their recovery. Our prior and ongoing work highlights both progress and challenges with FEMA’s Public Assistance program, including the agency’s methodology for determining program eligibility, the redesigned delivery model, and the program’s alternative procedures. FEMA’s Public Assistance program provides grants to repair public infrastructure such as water storage systems, roads, and power lines. In September 2012, we found that FEMA primarily relied on a single criterion, the per capita damage indicator, to determine a jurisdiction’s eligibility for Public Assistance funding. However, because FEMA’s current per capita indicator, set at $1 in 1986, does not reflect the rise in (1) per capita personal income since it was created in 1986 or (2) inflation from 1986 to 1999, the indicator is artificially low. Our analysis of actual and projected obligations for 508 disaster declarations in which Public Assistance was awarded during fiscal years 2004 through 2011 showed that fewer disasters would have met either the personal income-adjusted or the inflation-adjusted Public Assistance per capita indicators for the years in which the disaster was declared. Thus, had the indicator been adjusted annually since 1986 for personal income or inflation, fewer jurisdictions would have met the eligibility criteria that FEMA primarily used to determine whether federal assistance should be provided, which would have likely resulted in fewer disaster declarations and lower federal costs. We recommended, among other things, that FEMA develop and implement a methodology that more comprehensively assesses a jurisdiction’s capacity to respond to and recover from a disaster without federal assistance, including fiscal capacity and consideration of response and recovery capabilities. DHS concurred with our recommendation and, in January 2016, FEMA was considering establishing a disaster deductible, which would have required a predetermined level of financial or other commitment before FEMA would have provided assistance under the Public Assistance program. In August 2018, FEMA told us that it was no longer pursuing its proposed disaster deductible due to concerns about the complexity of the proposal. FEMA is considering options that leverage similar approaches, but does not have an estimated completion date for implementation. In addition, the DRRA requires FEMA to initiate rulemaking to (1) update the factors considered when evaluating requests for major disaster declarations, including reviewing how FEMA estimates the cost of major disaster assistance, and (2) consider other impacts on the capacity of a jurisdiction to respond to disasters, by October 2020. Until FEMA implements a new methodology, the agency will not have an accurate assessment of a jurisdiction’s capabilities and runs the risk of recommending that the President award Public Assistance to jurisdictions that have the capacity to respond and recover on their own. Prior to our September 2018 report, we had previously reported on the Public Assistance program in November 2017. Specifically, we reported that FEMA redesigned the delivery model for providing grants under the Public Assistance program. As part of the redesign effort, FEMA developed a new, web-based case management system to address past challenges, such as difficulties in sharing grant documentation among FEMA, state, and local officials and tracking the status of Public Assistance projects. Both FEMA and state officials involved in testing of the redesigned delivery model stated that the new case management system’s capabilities could lead to greater transparency and efficiencies in the program. However, we found that FEMA had not fully addressed two key information technology management controls that are necessary to ensure systems work effectively and meet user needs. We recommended, among other things, that FEMA (1) establish controls for tracking the development of system requirements, and (2) establish system testing criteria, roles and responsibilities, and the sequence and schedule for integration of other relevant systems. FEMA concurred with these recommendations and has fully implemented the first recommendation. Regarding the second recommendation, FEMA has not yet finalized its decision on whether to integrate its new case management system with its current grants management system. As of March 2019, we are awaiting a final decision from officials to determine whether their actions fully address our recommendation. FEMA’s original intention was to implement the redesigned delivery model for all future disasters beginning in January 2018. However, in September 2017, FEMA expedited full implementation of the redesigned model shortly after Hurricane Harvey made landfall. In September 2018, we reported that local officials continued to experience challenges with using the new Public Assistance web-based, case management system following the 2017 disasters, such as not having sufficient guidance on how to use the new system and delays with FEMA’s processing of their projects. In February 2019, we also reported that FEMA and the USVI were transitioning from using the standard Public Assistance program to using Public Assistance alternative procedures. FEMA and USVI officials stated that the alternative procedures will give the USVI more flexibility in determining when and how to fund projects and allow the territory to use any excess funds for cost-effective hazard mitigation measures, among other uses. Further, when using the alternative procedures, the Bipartisan Budget Act of 2018 allows FEMA, the USVI and Puerto Rico to repair and rebuild critical services infrastructure—such as medical and education facilities—so it meets industry standards without regard to pre-disaster condition (see Figure 1). Regarding the implementation of the Public Assistance program in Puerto Rico, in March 2019, we reported that Puerto Rico established a central recovery office to oversee federal recovery funds and was developing an internal controls plan to help ensure better management and accountability of the funds. In the interim, FEMA instituted a manual process for reviewing each reimbursement request before providing Public Assistance funds to mitigate risk and help ensure financial accountability. We also reported that officials we interviewed from FEMA, Puerto Rico’s central recovery office, and municipalities said they experienced initial challenges with the recovery process, including concerns about lack of experience and knowledge of the alternative procedures; concerns about missing, incomplete, or conflicting guidance on the alternative procedures; and concerns that municipalities had not been fully reimbursed for work already completed after the hurricanes, causing financial hardships in some municipalities. FEMA officials stated that the agency is taking actions to address reported recovery challenges, such as additional training for new FEMA employees and drafting supplemental guidance for the alternative procedures process. We continue to monitor FEMA’s efforts in our ongoing work. As part of our ongoing work, we are continuing to examine hurricane recovery efforts in the USVI and Puerto Rico. Our preliminary observations indicate that the USVI plans to take a cautious approach in pursuing permanent work projects using the Public Assistance alternative procedures program, which requires the use of fixed-cost estimates. Specifically, USVI officials we interviewed told us that developing such fixed-cost estimates that accurately incorporate the future impact of inflation and increases in materials and labor costs for certain projects was difficult. Further, these officials stated that since the territory is financially responsible for any costs that exceed these fixed-cost estimates, the USVI plans to pursue projects that do not include high levels of complexity or uncertainty to reduce the risk of cost overruns. From our ongoing work on Puerto Rico’s recovery efforts, we have learned that, in March 2019, Puerto Rico’s central recovery office released the Disaster Recovery Federal Funds Management Guide, including an internal controls plan for the operation of the recovery office. On April 1, 2019, FEMA removed the manual reimbursement process and began a transition to allow the central recovery office to take responsibility for review and reimbursement approval of federal recovery funds. We will review this transition process as a part of our ongoing work. Our preliminary observations also indicate that some of the challenges we reported in our March 2019 report continue. For example, officials from Puerto Rico’s central government agencies told us they did not feel they had sufficient guidance on the FEMA Public Assistance program and where they did, written and verbal FEMA guidance was inconsistent or conflicting. For example, officials from one agency expressed their desire for more FEMA guidance communicated in writing as it frequently happened that different FEMA officials would interpret existing guidance differently. Similarly, officials from two agencies described situations where they had initially been directed to follow one interpretation of a policy, only to be directed to follow a different, conflicting interpretation in the subsequent months. Puerto Rico agency officials also stated that the lack of sufficient instruction led to a “back and forth” with FEMA for clarifications, which led to delays in the phases of project development. FEMA officials in Puerto Rico stated that the agency has developed specific guidance for disaster recovery in Puerto Rico and that there are various ways, such as in-person meetings, where officials from Puerto Rico can obtain clarification. We are continuing to examine this issue as part of our ongoing review of Puerto Rico’s recovery. In addition, our preliminary observations from our ongoing work for both the USVI and Puerto Rico indicate that FEMA, USVI and Puerto Rico officials have reported challenges with the implementation of the flexibilities authorized by section 20601 of the Bipartisan Budget Act. This section of the Act allows for the provision of assistance under the Public Assistance alternative procedures to restore disaster-damaged facilities or systems that provide critical services to an industry standard without regard to pre-disaster condition. Officials from Puerto Rico’s central government stated that they disagreed with FEMA’s interpretation of the types of damages covered by section 20601 of the Bipartisan Budget Act of 2018. In response, FEMA officials in Puerto Rico stated they held several briefings with Puerto Rico’s central recovery office to explain FEMA’s interpretation of the section. Further, FEMA officials in the USVI told us that initially, they had difficulty obtaining clarification from FEMA headquarters regarding how to implement key components of section 20601 of the Act. As of May 2019, FEMA officials in the USVI stated that they continue to move forward with developing alternative procedures projects. USVI officials also told us that FEMA had been responsive and helpful in identifying its options for using the new authorities the Act provides. We will continue to evaluate these identified challenges and any efforts to address them, as well as other aspects of recovery efforts in the USVI and Puerto Rico, and plan to report our findings in late 2019 and early 2020, respectively. FEMA’s Individuals and Households Program provides individuals with financial assistance, such as grants to help repair or replace damaged homes, and temporary direct housing assistance, such as recreational vehicles. The Individual Assistance program provides financial and direct assistance to disaster victims for expenses and needs that cannot be met through other means, such as insurance. In May 2019, we reported on FEMA’s effort to provide disaster assistance under the Individual Assistance program to older adults and people with disabilities following the 2017 hurricanes. We found that aspects of the application process for FEMA assistance were challenging for older individuals and those with disabilities. Further, according to stakeholders and FEMA officials, disability-related questions in the Individual Assistance registration materials were confusing and easily misinterpreted. While FEMA had made some efforts to help registrants interpret the questions, we recommended, among other things, that FEMA (1) implement new registration-intake questions that improve FEMA’s ability to identify and address survivors’ disability-related needs, and (2) improve communication of registrants’ disability-related information across FEMA programs. DHS concurred with the first recommendation and described steps FEMA plans to take, or is in the process of taking, to address it. However, DHS did not concur with the second recommendation, noting that it lacks specific funding to augment its legacy data systems. FEMA officials stated that they began a long-term data management improvement initiative in April 2017, which they expect will ease efforts to share and flag specific disability-related data. While we acknowledge FEMA’s concerns about changing legacy systems when it has existing plans to replace those systems, we continue to believe there are other cost-effective ways that are likely to improve communication of registrants’ disability-related information prior to implementing the system upgrades. For example, FEMA could revise its guidance to remind program officials to review the survivor case file notes to identify whether there is a record of any disability-related needs. We also have work underway to assess FEMA’s Individuals and Households Program, a component program of Individual Assistance. Through this program, as of April 2019, FEMA had awarded roughly $4.7 billion in assistance to almost 1.8 million individuals and households for federally-declared disasters occurring in 2017 and 2018. Specifically, we are analyzing Individuals and Households Program expenditures and registration data for recent years; reviewing FEMA’s processes, policies, and procedures for making eligibility and award determinations; and examining survivors’ reported experiences with this program, including any challenges, for major disaster declarations occurring in recent years. We plan to report our findings in early 2020. FEMA’s experiences during the 2017 disasters highlight the importance of continuing to make progress on addressing the long-standing workforce management challenges we have previously reported on and continue to observe in our ongoing work. In September 2018, we reported that the 2017 disasters—hurricanes Harvey, Irma, and Maria, as well as the California wildfires—resulted in unprecedented FEMA workforce management challenges, including recruiting, maintaining, and deploying a sufficient and adequately-trained FEMA disaster workforce. FEMA’s available workforce was overwhelmed by the response needs caused by the sequential and overlapping timing of the three hurricanes. For example, at the height of FEMA workforce deployments in October 2017, 54 percent of staff were serving in a capacity in which they did not hold the title of “Qualified”—according to FEMA’s qualification system standards—a past challenge we identified. FEMA officials noted that staff shortages, and lack of trained personnel with program expertise led to complications in its response efforts, particularly after Hurricane Maria. In February 2016, we reported on, among other things, FEMA’s efforts to implement, assess, and improve its Incident Management Assistance Team program. We found that while FEMA used some leading practices in managing the program, it lacked a standardized plan to ensure that all national and regional Incident Management Assistance Team members received required training. Further, we found that the program had experienced high attrition since its implementation in fiscal year 2013. We recommended, among other things, that FEMA develop (1) a plan to ensure that Incident Management Assistance Teams receive required training, and (2) a workforce strategy for retaining Incident Management Assistance Team staff. DHS concurred with the recommendations. FEMA fully implemented our first recommendation by developing an Incident Management Assistance Team Training and Readiness Manual and providing a training schedule for fiscal year 2017. In response to the second recommendation, FEMA officials stated in July 2018 that they plan to develop policies that will provide guidance on a new workforce structure, incentives for Incident Management Assistance Team personnel, and pay-for-performance and all other human resource actions. We are continuing to monitor FEMA’s efforts to address this recommendation. In November and December 2017, we reported on staffing challenges in FEMA’s Public Assistance program. In November 2017, we reported on FEMA’s efforts to address past workforce management challenges through its redesigned Public Assistance delivery model. As part of the redesign effort, FEMA created consolidated resource centers to standardize and centralize Public Assistance staff responsible for managing grant applications, and new specialized positions to ensure more consistent guidance to applicants. However, we found that FEMA had not assessed the workforce needed to fully implement the redesigned model, such as the number of staff needed to fill certain new positions, or to achieve staffing goals. Further, in December 2017, we reported on FEMA’s management of its Public Assistance appeals process, including that FEMA increased staffing levels for the appeals process from 2015 to 2017. However, we found that FEMA continued to face a number of workforce challenges, such as staff vacancies, turnover, and delays in training, which contributed to processing delays. Based on our findings from our November and December 2017 reports, we recommended, among other things, that FEMA (1) complete workforce staffing assessments that identify the appropriate number of staff needed to implement the redesigned Public Assistance delivery model, and (2) document steps for hiring, training, and retaining key appeals staff, and address staff transitions resulting from deployments to disasters. FEMA concurred with our recommendations to address workforce management challenges in the Public Assistance program and have reported taking some actions in response. For example, to address the first recommendation, FEMA officials have developed preliminary models and estimates of staffing needs across various programs, including Public Assistance, and plan to reevaluate the appropriate number of staff needed and present recommendations to senior leadership by the end of June 2019. To address the second recommendation, FEMA has collected information on the amount of time regional appeals analysts spend on appeals, and the inventory and timeliness of different types of appeals. FEMA officials stated in September 2018 that they plan to assess this information to prepare a detailed regional workforce plan. As of June 2019, we are evaluating plans and documents provided by FEMA to determine whether they have fully addressed this recommendation. In our March 2019 report on the status of recovery efforts in Puerto Rico, we also reported Puerto Rico officials’ concerns about FEMA staff turnover and lack of knowledge among FEMA staff about how the Public Assistance alternative procedures are to be applied in Puerto Rico. As part of our ongoing work, we are continuing to examine recovery efforts in Puerto Rico. Our preliminary observations indicate that the concerns we reported on in our March 2019 report continue. For example, Puerto Rico agency officials said that the lack of continuity in FEMA personnel has been a challenge for communication and project development. Further, officials from all seven Puerto Rico government agencies we interviewed felt that the FEMA staff they interacted with did not have a complete understanding of FEMA processes and policies. We are continuing to evaluate FEMA’s recovery efforts in Puerto Rico and plan to issue our findings in late 2019. In April 2019, we reported on the federal government’s contracting efforts for preparedness, response, and recovery efforts related to the 2017 hurricanes and California wildfires. We found, among other things, that contracting workforce shortages continue to be a challenge for disaster response and recovery. Further, although FEMA’s 2017 after-action report recommended increasing contract support capacities, it did not provide a specific plan to do so. We also found that while FEMA evaluated its contracting workforce needs in a 2014 workforce analysis, it did not specifically consider contracting workforce needs in the regional offices or address Disaster Acquisition Response Team employees. In our April 2019 report, we recommended, among other things, that FEMA assess its workforce needs—including staffing levels, mission needs, and skill gaps—for contracting staff, to include regional offices and Disaster Acquisition Response Teams, and develop a plan, including timelines, to address any gaps. FEMA concurred with this recommendation and estimates that it will implement it in September 2019. In our May 2019 report on FEMA disaster assistance to older adults and people with disabilities following the 2017 hurricanes, we found that FEMA began implementing a new approach to assist individuals with disabilities in June 2018, which shifted the responsibility for directly assisting individuals with disabilities from Disability Integration Advisors— which are staff FEMA deploys specifically to identify and recommend actions needed to support survivors with disabilities—to all FEMA staff. To implement this new approach, FEMA planned to train all of the agency’s deployable staff and staff in programmatic offices on disability issues during response and recovery deployments. According to FEMA, a number of Disability Integration Advisors would also deploy to advise FEMA leadership in the field during disaster response and recovery. We found that while FEMA has taken some initial steps to provide training on the changes, it has not established a plan for delivering comprehensive disability-related training to all staff who will be directly interacting with individuals with disabilities. We recommended, among other things, that FEMA develop a plan for delivering training to FEMA staff that promotes competency in disability awareness and includes milestones and performance measures, and outlines how performance will be monitored. DHS concurred with this recommendation; however, officials stated that FEMA is developing a plan to include a disability integration competency in the guidance provided for all deployable staff, rather than through training. We will monitor FEMA’s efforts to develop this plan and fully address our recommendation. In addition to our prior work on FEMA’s workforce management challenges related to specific programs and functions, we are continuing to evaluate FEMA’s workforce capacity and training efforts during the 2017 and 2018 disaster seasons. Our preliminary observations indicate that there were challenges in FEMA’s ability to deploy staff with the right kinds of skills and training at the right time to best meet the needs of various disaster events. For example, according to FEMA field leadership we interviewed, for some of the functions FEMA performs in the field, FEMA had too few staff with the right technical skills to perform their missions—such as inspections of damaged properties—efficiently and effectively. For other functions, these managers also reported that they had too many staff in the early stages of the disaster, which created challenges with assigning duties and providing on-the-job training. For example, some managers reported that they were allocated more staff than needed in the initial phases of the disaster, but many lacked experience and were without someone to provide direction and mentoring to ensure they used their time efficiently and gained competence more quickly. Groups of FEMA field managers we interviewed told us that difficulties deploying the right mix of staff with the right skills led to challenges such as making purchases to support FEMA operations, problems with properly registering applicants for FEMA programs, or poor communication with nonfederal partners. Nonetheless, FEMA staff have noted that, despite any suboptimal circumstances during disaster response, they aimed to and have been able to find a way to deliver the mission. As part of this ongoing work, FEMA field leadership and managers also reported challenges using agency systems to ensure the availability of the right staff with the right skills in the right place and time. FEMA uses a system called the Deployment Tracking System to, among other things, help identify staff available to be deployed and activate and track deployments. To help gauge the experience level and training needs of its staff, the agency established the FEMA Qualification System (FQS), which is a set of processes and criteria to monitor staff experience in competently performing tasks and completing training that correspond to their job titles. According to the FQS guidance, staff who have been able to demonstrate proficient performance of all the relevant tasks and complete required training receive the designation “qualified,” and are expected to be ready and able to competently fulfill their responsibilities. Those who have not, receive the designation “trainee,” and can be expected to need additional guidance and on-the-job training. FQS designations feed into the Deployment Tracking System as one key variable in how the tracking system deploys staff. Among other challenges with FEMA’s Deployment Tracking System and Qualification System, FEMA managers and staff in the field told us an employee’s recorded qualification status was not a reliable indicator of the level at which deployed personnel would be capable of performing specific duties and responsibilities or their general proficiency in their positions, making it more difficult for managers to know the specialized skills or experience of staff and effectively build teams. We are continuing to assess these and other reported workforce challenges and plan to report our findings in January 2020. In April 2019, we reported on FEMA’s Grants Management Modernization program, which is intended to replace the agency’s 10 legacy grants management systems and modernize and streamline the grants management environment. We found that, of six important leading practices for effective business process reengineering and information technology requirements management, FEMA fully implemented four and partially implemented two for the Grants Management Modernization program. The two partially implemented leading practices were (1) establishing plans for implementing new business processes and (2) establishing complete traceability of information technology requirements. In addition, we found that the program’s initial May 2017 cost estimate of about $251 million was generally consistent with leading practices for a reliable, high-quality estimate; however, it no longer reflected the current assumptions about the program at the time of our review. Moreover, the program’s schedule–specifically its final delivery date of September 2020—did not reflect leading practices for project schedules, as the date was not informed by a realistic assessment of development activities. Lastly, we found that FEMA fully addressed three and partially addressed two of five key cybersecurity practices. The two partially addressed practices were (1) assessing security controls, and (2) obtaining an authorization to operate the system. We made 8 recommendations to FEMA to implement leading practices related to reengineering processes, managing information technology requirements, scheduling system development activities, and implementing cybersecurity. DHS concurred with all of our recommendations and provided estimated completion dates for implementing each of them through July 2020. Thank you, Chairman Rouda, Ranking Member Comer and Members of the Subcommittee. This concludes my prepared statement. I would be happy to respond to any questions you may have at this time. If you or your staff has any questions concerning this testimony, please contact Christopher P. Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement were Joel Aldape (Assistant Director), Matthew T. Lowney (Analyst-in-Charge), Rebecca Mendelsohn, David (Ben) Nelson, and Amanda R. Parker. In addition, Aditi Archer, Bryan Bourgault, Lorraine Ettaro, Aaron Gluck, Kathryn Godfrey, Taylor Hadfield, Eric Hauswirth, Robert (Denton) Herring, Adam Hoffman, Susan Hsu, Sara Kelly, Amy Moran Lowe, Heidi Nielson, Danielle Pakdaman, Sara Pelton, Amanda Prichard, and Johanna Wong made contributions to this statement. Key contributors for the previous work that this is based on are listed in each product. Opportunities to Reduce Potential Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue. GAO-11-318SP, March 1, 2011. Federal Disaster Assistance: Improved Criteria Needed to Assess a Jurisdiction’s Capability to Respond and Recover on Its Own. GAO-12- 838, September 12, 2012. Fiscal Exposures: Improving Cost Recognition in the Federal Budget. GAO-14-28, October 29, 2013. Emergency Preparedness: Opportunities Exist to Strengthen Interagency Assessments and Accountability for Closing Capability Gaps. GAO-15-20, December 4, 2014. High-Risk Series: An Update. GAO-15-290, February 11, 2015. Budgeting for Disasters: Approaches to Budgeting for Disasters in Selected States. GAO-15-424, March 26, 2015. Hurricane Sandy: An Investment Strategy Could Help the Federal Government Enhance National Resilience for Future Disasters. GAO-15-515, July 30, 2015. Disaster Response: FEMA Has Made Progress Implementing Key Programs, but Opportunities for Improvement Exist. GAO-16-87, February 5, 2016. Disaster Recovery: FEMA Needs to Assess Its Effectiveness in Implementing the National Disaster Recovery Framework. GAO-16-476, May 26, 2016. Federal Disaster Assistance: Federal Departments and Agencies Obligated at Least $277.6 Billion during Fiscal Years 2005 through 2014. GAO-16-797, September 22, 2016. Climate Change: Information on Potential Economic Effects Could Help Guide Federal Efforts to Reduce Fiscal Exposure. GAO-17-720, September 28, 2017. Disaster Assistance: Opportunities to Enhance Implementation of the Redesigned Public Assistance Grant Program. GAO-18-30, November 8, 2017. Disaster Recovery: Additional Actions Would Improve Data Quality and Timeliness of FEMA’s Public Assistance Appeals Processing. GAO-18- 143, December 15, 2017. 2017 Disaster Contracting: Observations on Federal Contracting for Response and Recovery Efforts. GAO-18-335, February 28, 2018. Federal Disaster Assistance: Individual Assistance Requests Often Granted but FEMA Could Better Document Factors Considered. GAO-18- 366, May 31, 2018. 2017 Hurricanes and Wildfires: Initial Observations on the Federal Response and Key Recovery Challenges. GAO-18-472, September 4, 2018. Homeland Security Grant Program: Additional Actions Could Further Enhance FEMA’s Risk-Based Grant Assessment Model. GAO-18-354, September 6, 2018. Continuity of Operations: Actions Needed to Strengthen FEMA’s Oversight and Coordination of Executive Branch Readiness. GAO-19- 18SU, November 26, 2018. 2017 Disaster Contracting: Action Needed to Better Ensure More Effective Use and Management of Advance Contracts. GAO-19-93, December 6, 2018. U.S. Virgin Islands Recovery: Status of FEMA Public Assistance Funding and Implementation. GAO-19-253, February 25, 2019. High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas. GAO-19-157SP, March 6, 2019. Puerto Rico Hurricanes: Status of FEMA Funding, Oversight, and Recovery Challenges. GAO-19-256, March 14, 2019. Huracanes de Puerto Rico: Estado de Financiamiento de FEMA, Supervisión y Desafíos de Recuperación. GAO-19-331, March 14, 2019. Disaster Recovery: Better Monitoring of Block Grant Funds Is Needed. GAO-19-232, March 25, 2019. FEMA Grants Modernization: Improvements Needed to Strengthen Program Management and Cybersecurity. GAO-19-164, April 9, 2019. 2017 Hurricane Season: Federal Support for Electricity Grid Restoration in the U.S. Virgin Islands and Puerto Rico. GAO-19-296, April 18, 2019. Disaster Contracting: Actions Needed to Improve the Use of Post- Disaster Contracts to Support Response and Recovery, GAO-19-281, April 24, 2019. Disaster Assistance: FEMA Action Needed to Better Support Individuals Who Are Older or Have Disabilities. GAO-19-318, May 14, 2019. 1. Review of U.S. Virgin Islands recovery planning and progress; 2. Puerto Rico disaster recovery planning and progress; 3. 2017 wildfire response and recovery; 4. Federal internal control plans for disaster assistance funding; 5. Electricity grid restoration and resilience after the 2017 hurricane 6. Mass care sheltering and feeding challenges during the 2017 7. Department of Transportation highway and transit emergency relief 8. Drinking water and wastewater utility resilience; 9. Review of disaster death count information in selected states and 10. Department of Health and Human Services disaster response efforts; 11. Disaster and climate change impacts on Superfund sites; 12. FEMA Public Assistance program fraud risk management efforts; 13. Wildland fire collaboration on fuel reduction efforts; 14. Preparedness challenges and lessons learned from the 2017 15. FEMA workforce management and challenges; 16. Small Business Administration response to 2017 disasters; 17. Development of the GAO disaster resilience framework; 18. FEMA Individuals and Households Program operations and 19. National Flood Insurance Program post-flood enforcement; 20. Emergency alerting capabilities and progress; 21. National Flood Insurance Program buyouts and property acquisitions; 22. Economic costs of large-scale natural disasters and impacts on 23. Community Development Block Grants – disaster recovery; and 24. Disaster Housing Assistance Program. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Recent hurricanes, wildfires, and flooding have highlighted the challenges the federal government faces in responding effectively to natural disasters. The 2017 and 2018 hurricanes and wildfires affected millions of individuals and caused billions of dollars in damages. In March 2019, the Midwest experienced historic flooding that affected millions of acres of agriculture and damaged significant infrastructure. Since 2005, federal funding for disaster assistance is at least $450 billion. Increasing reliance on federal help to address natural disasters is a key source of federal fiscal exposure, particularly as certain extreme weather events become more frequent and intense due to climate change. This statement discusses, among other things, FEMA's progress and challenges related to disaster resilience, response, recovery, and workforce management. This statement is based on GAO reports issued from March 2011 through May 2019, and also includes preliminary observations from ongoing GAO reviews of FEMA operations. For ongoing work, GAO reviewed federal laws; analyzed documents; interviewed agency officials; and visited disaster damaged areas in California, Florida, South Carolina, North Carolina, Puerto Rico, Texas, and the U.S. Virgin Islands, where GAO also interviewed FEMA and local officials. GAO's issued and ongoing work identified progress and challenges in the Federal Emergency Management Agency's (FEMA) disaster resilience, response, recovery, and workforce management efforts, as discussed below. Disaster Resilience. GAO found that federal and local efforts to improve resilience can reduce the effects and costs of future disasters. FEMA has made progress in this area, but in July 2015, GAO found that states and localities faced challenges using federal funds to maximize resilient rebuilding following a disaster. GAO recommended that the Mitigation Framework Leadership Group—an interagency body chaired by FEMA—create a national strategy to better plan for and invest in disaster resilience. FEMA is working to address this recommendation and plans to publish the strategy by July 2019. Response and Recovery. In September 2018, GAO reported that the response to the 2017 disasters in Texas, Florida, and California showed progress since Hurricane Katrina in 2005. Specifically, FEMA and state officials' pre-existing relationships and exercises aided the response and helped address various challenges. However, GAO and FEMA identified challenges that slowed and complicated FEMA's response to Hurricane Maria, particularly in Puerto Rico. GAO's issued and ongoing work also identified challenges in implementing FEMA Public Assistance grants. For example, FEMA and Puerto Rico officials identified challenges with Public Assistance policies and guidance that have complicated and slowed the recovery. GAO did not make recommendations, but continues to evaluate recovery efforts and will report its findings later this year. FEMA Workforce Management. GAO has previously reported on long-standing workforce management challenges, such as ensuring an adequately-staffed and trained workforce. For example, GAO reported in September 2018 that the 2017 disasters overwhelmed FEMA's workforce and a lack of trained personnel with program expertise led to complications in its response efforts, particularly after Hurricane Maria. While FEMA has taken actions to address several of GAO's workforce management-related recommendations since 2016, a number of recommendations remain open as the 2019 hurricane season begins. Also, GAO is currently reviewing FEMA's workforce management efforts and lessons learned from the 2017 disasters and will report its findings early next year. GAO has made numerous recommendations in its prior reports to FEMA designed to address the challenges discussed in this statement. As of May 2019, FEMA has addressed about half of these recommendations and GAO is monitoring FEMA's ongoing efforts.", "document_type": "gao"}
{"report": "We have previously reported on various aspects of national preparedness, including examining the extent to which FEMA programs encourage disaster resilience and identifying gaps in federal preparedness capabilities. We have found that when federal, state, and local efforts aligned to focus on improving disaster resilience and preparedness, there was a noticeable reduction in the effects of the disaster. However, our prior and ongoing work also highlight opportunities to improve disaster resilience and preparedness nationwide. Hazard mitigation is a key step in building resilience and preparedness against future disasters. In July 2015, we found that states and localities experienced challenges when trying to use federal funds to maximize resilient rebuilding in the wake of a disaster. In particular, they had difficulty navigating multiple federal grant programs and applying federal resources towards their most salient risks because of the fragmented and reactionary nature of the funding. In our 2015 report, we recommended that the Mitigation Framework Leadership group—an interagency body chaired by FEMA—create a National Mitigation Investment Strategy to help federal, state, and local officials plan for and prioritize disaster resilience. As of May 2019, according to FEMA officials, the Mitigation Framework Leadership group is on track to address the recommendation, and they expect the strategy to be published by July 2019. In September 2017, we reported that the methods used to estimate the potential economic effects of climate change in the United States—using linked climate science and economics models—could inform decision makers about significant potential damages in different U.S. sectors or regions, despite the limitations. For example, for 2020 through 2039, one study estimated between $4 billion and $6 billion in annual coastal property damages from sea level rise and more frequent and intense storms. We found that the federal government has not undertaken strategic government-wide planning on the potential economic effects of climate change to identify significant risks and craft appropriate federal responses. As a result, we recommended the Executive Office of the President, among others, should use information on the potential economic effects of climate change to help identify significant climate risks facing the federal government and craft appropriate federal responses, such as establishing a strategy to identify, prioritize, and guide federal investments to enhance resilience against future disasters; however, as of June 2019, officials have not taken action to address this recommendation. In November 2017, we found that FEMA had taken some actions to better promote hazard mitigation as part of its Public Assistance grant program. However, we also reported that more consistent planning for, and more specific performance measures related to, hazard mitigation could help ensure that mitigation is incorporated into recovery efforts. We recommended, among other things, that FEMA (1) standardize planning efforts for hazard mitigation after a disaster and (2) develop performance measures for the Public Assistance grant program to better align with FEMA’s strategic goal for hazard mitigation in the recovery process. FEMA concurred with our recommendations, and as of March 2019, officials have reported taking steps to increase coordination across its Public Assistance, mitigation, and field operations to ensure hazard mitigation efforts are standardized and integrated into the recovery process. Additionally, FEMA officials reported taking actions to begin developing disaster-specific mitigation performance measures. However, FEMA has yet to finalize these actions, such as by proposing performance measures to FEMA senior leadership. As such, we are continuing to monitor FEMA’s efforts to address these recommendations. In March 2011, we reported that FEMA had not completed a comprehensive and measurable national preparedness assessment of capability gaps—for example the amount of resources required to save lives, protect property and the environment, and meet basic human needs after an incident has occurred. Developing such an assessment would help FEMA to identify what capability gaps exist and what level of resources are needed to close such gaps. Accordingly, we suggested that FEMA complete a national preparedness assessment to evaluate capability requirements and gaps at each level of government to enable FEMA to prioritize grant funding. As of December 2018, FEMA had efforts underway to assess urban area, state, territory, and tribal preparedness capabilities to inform the prioritization of grant funding; however, the agency had not yet completed a national preparedness assessment with clear, objective, and quantifiable capability requirements against which to assess preparedness. We are continuing to monitor FEMA’s efforts to complete such an assessment. Furthermore, in March 2015, we reviewed selected states’ approaches to budgeting for disaster costs to help inform congressional consideration of the balance between federal and state roles in funding disaster assistance. Specifically, we reported that none of the 10 states in our review maintained reserves dedicated solely for future disasters, and some state officials reported that they could cover disaster costs without dedicated disaster reserves because they generally relied on the federal government to fund most of the costs associated with disaster response and recovery. In response to the 2017 disasters, we also have ongoing work to review national preparedness capabilities to assist communities in responding to and recovering from disasters. Based on our preliminary observations, some states and localities we interviewed reported that while they are prepared to deal with immediate response issues in the aftermath of a disaster, gaps exist in their capacity to support longer term recovery. One reason for this, according to these state and local officials, is because federal preparedness grant funds are largely dedicated to maintaining response capabilities and sustaining personnel costs for local emergency management officials. While these preparedness grants fund critical elements of the national preparedness system, there are some limitations to using them. Specifically, some state and local officials told us that the preparedness grant activities are generally focused on terrorism issues rather than all-hazards. In addition, they reported that the preparedness grants are generally spent on maintaining response capabilities rather than to enhance their capacity for disaster recovery—such as additional training and exercises. In addition to the state, territory, and urban region assessments that FEMA is conducting, FEMA is currently in the process of developing the first national Threat and Hazard Identification and Risk Assessment. This national assessment may help FEMA and policymakers better understand how to target federal resources in a way that enhances the nation’s capacity to respond and recover from future catastrophic or sequential disasters. We are continuing to evaluate national preparedness efforts and plan to report on FEMA’s Threat and Hazard Identification and Risk Assessment process in January 2020. In September 2018, we reported that the response to the 2017 hurricanes and wildfires in Texas, Florida, and California showed progress made since the 2005 federal response to Hurricane Katrina. We also found that FEMA coordinated closely with Texas, Florida, and California emergency management officials and other federal, local, and volunteer emergency partners to implement various emergency preparedness actions prior to the 2017 disasters in each state, and to respond to these disasters. According to FEMA and state officials, these actions helped officials begin addressing a number of challenges they faced such as meeting the demand for a sufficient and adequately-trained disaster workforce and complex issues related to removing debris in a timely manner after the hurricanes and wildfires. In contrast, we also reported in September 2018, that in Puerto Rico and the USVI a variety of challenges—such as the far distance of the territories from the U.S. mainland, limited local preparedness for a major hurricane, and outdated local infrastructure—complicated response efforts to hurricanes Irma and Maria. Many of the challenges we identified are also described in FEMA’s 2017 Hurricane Season FEMA After-Action Report, including: the sequential and overlapping timing of the three hurricanes—with Maria being the last of the three—caused staffing shortages and required FEMA to shift staff to the territories that were already deployed to other disasters; the far distance of both territories from the U.S. mainland complicated efforts to deploy federal resources and personnel quickly; and the incapacitation of local response functions due to widespread devastation and loss of power and communications, and limited preparedness by Puerto Rico and the USVI for a category 5 hurricane resulted in FEMA having to assume response functions that territories would usually perform themselves. We also reported that FEMA’s 2017 Hurricane Season FEMA After-Action Report noted that FEMA could have better leveraged information from preparedness exercises in the Caribbean, including a 2011 exercise after- action report for Puerto Rico which indicated that the territory would require extensive federal support during a large scale disaster in moving commodities from the mainland to the territory and to distribution points throughout. In our September 2018 report, we also found that FEMA’s efforts in Puerto Rico after Hurricane Maria were the largest and longest single response in the agency’s history. According to FEMA, the agency’s response included, among other things, bringing in approximately $1 billion in food and supplies; and distributing food, commodities, and medicine via approximately 1,400 flights, which constituted the longest sustained air operations in U.S. disaster history. FEMA officials explained that the agency essentially served as the first responder in the early response efforts in Puerto Rico, and many of services FEMA provided—such as power restoration, debris removal, and commodity distribution—were typically provided by territorial or local governments. We also reported in September 2018, that in the USVI, recent disaster training and the pre-positioning of supplies due to the anticipated impact of Hurricane Irma facilitated the response efforts for Hurricane Maria, which made landfall less than two weeks later. According to FEMA’s federal coordinating officer, the lead federal official in charge of response for the USVI, the federal government deployed assets, including urban search and rescue teams and medical assistance teams. In addition, due to the sequence of Hurricane Irma hitting the USVI immediately before Hurricane Maria, the Department of Defense (DOD) already had personnel and resources (i.e., ships) deployed to the area, which enabled DOD to respond to Hurricane Maria faster than it otherwise would have. Additional challenges we have reported on regarding response operations have included providing short-term housing and sheltering for disaster survivors. The Department of Homeland Security’s (DHS) 2017 National Preparedness Report states that providing effective and affordable short- term housing for disaster survivors has been a longstanding and continuing challenge. For example, following the California wildfires, local officials faced challenges identifying shelter for displaced survivors, in part due to a housing shortage that existed before the wildfires. Federal, state, and local officials formed housing task forces which facilitated a joint decision-making approach to address these challenges. While this approach has enabled the state to meet its most pressing short-term housing needs, according to FEMA officials, the state faces other challenges in the long term. For example, FEMA officials in the region covering California told us that because of the nature of damage following a wildfire and because of housing shortages in California, some of FEMA’s forms of housing assistance have been less relevant in the wake of the California wildfires than for other disasters. We will continue to evaluate these and other challenges and plan to report in fall 2019. We also have ongoing work to review efforts to provide mass care— which includes sheltering, feeding and providing emergency supplies— following the 2017 hurricanes. Our preliminary observations indicate that during and immediately following the hurricanes, the number of people seeking public shelters outpaced the capacity. In Texas and Florida, emergency managers we spoke with described having unprecedented numbers of residents needing shelters but not always enough staff initially to operate the shelters. In Texas, Puerto Rico, and the USVI, hurricanes Harvey, Irma, and Maria flooded or destroyed many buildings planned for use as shelters, according to emergency management and local government officials in these areas. As a result, some remaining shelters were at maximum capacity. In the USVI, residents of some public housing units that had sustained significant damages sought help at the territory’s Department of Human Services because there was no more space in the shelters, according to local government officials. While they were turned away from the shelters, these families were able to take refuge in the lobby of the Department of Human Services building. We will continue to evaluate these and other challenges and plan to report in summer 2019. In December 2018 and April 2019, we reported that, in response to hurricanes Harvey, Irma, and Maria, as well as the 2017 California wildfires, FEMA and other federal partners relied heavily on advance contracts—which are established before a disaster to provide for life- sustaining goods and services such as food, water and transportation typically needed immediately after a disaster—and post disaster contracts—which can be used for various goods and services, such as debris removal and installation of power transmission equipment. FEMA is required to coordinate with states and localities and encourage them to establish their own advance contracts with vendors. In December 2018, we reported on inconsistencies we found in that coordination and in the information FEMA used to coordinate with states and localities on advance contracts. As a result of this and other challenges identified, we made nine recommendations to FEMA, including that it update its strategy and guidance to clarify the use of advance contracts, improve the timeliness of its acquisition planning activities, revise its methodology for reporting disaster contracting actions to Congress, and provide more consistent guidance and information for contracting officers in coordinating with states and localities to establish advance contracts. FEMA concurred with all of these recommendations, and we are continuing to monitor its efforts to implement each recommendation. Furthermore, in April 2019, we reported on challenges that we found in the federal government’s use of post-disaster contracts. These challenges included a lack of transparency about contract actions, challenges with requirements development, and with interagency coordination. In our report, we found that FEMA had begun taking some steps to address the consistency of post-disaster contract requirements with contracting officers, but that inaccurate or untimely estimates in the contracts we reviewed sometimes resulted in delays meeting the needs of survivors. As a result of our findings in this report, we made 10 recommendations to FEMA and other federal agencies that use these post-disaster contracts related to improving the management of such contracts. FEMA and other agency officials concurred with nine of the recommendations and have reported taking actions to begin implementing them. We will continue to monitor FEMA’s progress in fully addressing these recommendations. FEMA provides multiple forms of disaster recovery assistance after a major disaster has been declared, including Public Assistance and Individual Assistance. Through these grant programs, FEMA obligates billions of dollars to state, tribal, territorial, and local governments, certain nonprofit organizations, and individuals that have suffered injury or damages from major disaster or emergency incidents, such as hurricanes, tornados, or wildfires. In September 2016, we reported that, from fiscal years 2005 through 2014, FEMA obligated almost $46 billion for the Public Assistance program and over $25 billion for the Individual Assistance program. According to FEMA’s May 2019 Disaster Relief Fund report, total projected obligations through fiscal year 2019 for the Public Assistance and Individual Assistance programs for just the 2017 hurricanes—Harvey, Irma, and Maria—are roughly $16 billion and $7 billion, respectively. Given the high cost of these programs, it is imperative that FEMA continue to make progress on the challenges we have identified in our prior and ongoing work regarding its recovery efforts. FEMA’s Public Assistance program provides grants to state, tribal, territorial, and local governments for debris removal; emergency protective measures; and the repair, replacement, or restoration of disaster-damaged, publicly owned facilities. It is a complex and multistep program administered through a partnership among FEMA, the state, and local officials. Prior to implementing the Public Assistance program, FEMA determines a state, territorial or tribal government’s eligibility for the program using the per capita damage indicator. In our September 2018 report on federal response and recovery efforts for the 2017 hurricanes and wildfires, we reported on FEMA’s implementation of the Public Assistance program, which has recently undergone significant changes as a result of federal legislation and agency initiatives. Specifically, we reported on FEMA’s use of its redesigned delivery model for providing grants under the Public Assistance program, as well as the alternative procedures for administering or receiving such grant funds that FEMA allows states, territories, and local governments to use for their recovery. Our prior and ongoing work highlights both progress and challenges with FEMA’s Public Assistance program, including the agency’s methodology for determining program eligibility, the redesigned delivery model, and the program’s alternative procedures. In September 2012, we found that FEMA primarily relied on a single criterion, the per capita damage indicator, to determine a jurisdiction’s eligibility for Public Assistance funding. However, because FEMA’s current per capita indicator, set at $1 in 1986, does not reflect the rise in (1) per capita personal income since it was created in 1986 or (2) inflation from 1986 to 1999, the indicator is artificially low. Our analysis of actual and projected obligations for 508 disaster declarations in which Public Assistance was awarded during fiscal years 2004 through 2011 showed that fewer disasters would have met either the personal income-adjusted or the inflation-adjusted Public Assistance per capita indicators for the years in which the disaster was declared. Thus, had the indicator been adjusted annually since 1986 for personal income or inflation, fewer jurisdictions would have met the eligibility criteria that FEMA primarily used to determine whether federal assistance should be provided, which would have likely resulted in fewer disaster declarations and lower federal costs. We recommended, among other things, that FEMA develop and implement a methodology that that more comprehensively assesses a jurisdiction’s capacity to respond to and recover from a disaster without federal assistance, including fiscal capacity and consideration of response and recovery capabilities. DHS concurred with our recommendation and, in January 2016, FEMA was considering establishing a disaster deductible, which would have required a predetermined level of financial or other commitment before FEMA would have provided assistance under the Public Assistance program. In August 2018, FEMA told us that it was no longer pursuing its proposed disaster deductible due to concerns about the complexity of the proposal. FEMA is considering options that leverage similar approaches, but does not have an estimated completion date for implementation. In addition, the DRRA requires FEMA to initiate rulemaking to (1) update the factors considered when evaluating requests for major disaster declarations, including reviewing how FEMA estimates the cost of major disaster assistance, and (2) consider other impacts on the capacity of a jurisdiction to respond to disasters, by October 2020. Until FEMA implements a new methodology, the agency will not have an accurate assessment of a jurisdiction’s capabilities and runs the risk of recommending that the President award Public Assistance to jurisdictions that have the capacity to respond and recover on their own. Prior to our September 2018 report, we had previously reported on the Public Assistance program in November 2017. Specifically, we reported that FEMA redesigned the delivery model for providing grants under the Public Assistance program. As part of the redesign effort, FEMA developed a new, web-based case management system to address past challenges, such as difficulties in sharing grant documentation among FEMA, state, and local officials and tracking the status of Public Assistance projects. Both FEMA and state officials involved in testing of the redesigned delivery model stated that the new case management system’s capabilities could lead to greater transparency and efficiencies in the program. However, we found that FEMA had not fully addressed two key information technology management controls that are necessary to ensure systems work effectively and meet user needs. We recommended, among other things, that FEMA (1) establish controls for tracking the development of system requirements, and (2) establish system testing criteria, roles and responsibilities, and the sequence and schedule for integration of other relevant systems. FEMA concurred with these recommendations and has fully implemented the first recommendation. Regarding the second recommendation, FEMA has not yet finalized its decision on whether to integrate its new case management system with its current grants management system. As of March 2019, we are awaiting a final decision from officials to determine whether their actions fully address our recommendation. FEMA’s original intention was to implement the redesigned delivery model for all future disasters beginning in January 2018. However, in September 2017, FEMA expedited full implementation of the redesigned model shortly after Hurricane Harvey made landfall. In September 2018, we reported that local officials continued to experience challenges with using the new Public Assistance web-based, case management system following the 2017 disasters, such as not having sufficient guidance on how to use the new system and delays with FEMA’s processing of their projects. In February 2019, we also reported that FEMA and the USVI were transitioning from using the standard Public Assistance program to using Public Assistance alternative procedures. FEMA and USVI officials stated that the alternative procedures will give the USVI more flexibility in determining when and how to fund projects and allow the territory to use any excess funds for cost-effective hazard mitigation measures, among other uses. Further, when using the alternative procedures, the Bipartisan Budget Act of 2018 allows FEMA, the USVI and Puerto Rico to repair and rebuild critical services infrastructure—such as medical and education facilities—so it meets industry standards without regard to pre-disaster condition (see Figure 1). Regarding the implementation of the Public Assistance program in Puerto Rico, in March 2019, we reported that Puerto Rico established a central recovery office to oversee federal recovery funds and was developing an internal controls plan to help ensure better management and accountability of the funds. In the interim, FEMA instituted a manual process for reviewing each reimbursement request before providing Public Assistance funds to mitigate risk and help ensure financial accountability. We also reported that officials we interviewed from FEMA, Puerto Rico’s central recovery office, and municipalities said they experienced initial challenges with the recovery process, including concerns about lack of experience and knowledge of the alternative procedures; concerns about missing, incomplete, or conflicting guidance on the alternative procedures; and concerns that municipalities had not been fully reimbursed for work already completed after the hurricanes, causing financial hardships in some municipalities. FEMA officials stated that the agency is taking actions to address reported recovery challenges, such as additional training for new FEMA employees and drafting supplemental guidance for the alternative procedures process. We continue to monitor FEMA’s efforts in our ongoing work. As part of our ongoing work, we are continuing to examine hurricane recovery efforts in the USVI and Puerto Rico. Our preliminary observations indicate that the USVI plans to take a cautious approach in pursuing permanent work projects using the Public Assistance alternative procedures program, which requires the use of fixed-cost estimates. Specifically, USVI officials we interviewed told us that developing such fixed-cost estimates that accurately incorporate the future impact of inflation and increases in materials and labor costs for certain projects was difficult. Further, these officials stated that since the territory is financially responsible for any costs that exceed these fixed-cost estimates, the USVI plans to pursue projects that do not include high levels of complexity or uncertainty to reduce the risk of cost overruns. From our ongoing work on Puerto Rico’s recovery efforts, we have learned that, in March 2019, Puerto Rico’s central recovery office released the Disaster Recovery Federal Funds Management Guide, including an internal controls plan for the operation of the recovery office. On April 1, 2019, FEMA removed the manual reimbursement process and began a transition to allow the central recovery office to take responsibility for review and reimbursement approval of federal recovery funds. We will review this transition process as a part of our ongoing work. Our preliminary observations also indicate that some of the challenges we reported in our March 2019 report continue. For example, officials from Puerto Rico’s central government agencies told us they did not feel they had sufficient guidance on the FEMA Public Assistance program and where they did, written and verbal FEMA guidance was inconsistent or conflicting. For example, officials from one agency expressed their desire for more FEMA guidance communicated in writing as it frequently happened that different FEMA officials would interpret existing guidance differently. Similarly, officials from two agencies described situations where they had initially been directed to follow one interpretation of a policy, only to be directed to follow a different, conflicting interpretation in the subsequent months. Puerto Rico agency officials also stated that the lack of sufficient instruction led to a “back and forth” with FEMA for clarifications, which led to delays in the phases of project development. FEMA officials in Puerto Rico stated that the agency has developed specific guidance for disaster recovery in Puerto Rico and that there are various ways, such as in-person meetings, where officials from Puerto Rico can obtain clarification. We are continuing to examine this issue as part of our ongoing review of Puerto Rico’s recovery. In addition, our preliminary observations from our ongoing work for both the USVI and Puerto Rico indicate that FEMA, USVI and Puerto Rico officials have reported challenges with the implementation of the flexibilities authorized by section 20601 of the Bipartisan Budget Act. This section of the Act allows for the provision of assistance under the Public Assistance alternative procedures to restore disaster-damaged facilities or systems that provide critical services to an industry standard without regard to pre-disaster condition. Officials from Puerto Rico’s central government stated that they disagreed with FEMA’s interpretation of the types of damages covered by section 20601 of the Bipartisan Budget Act of 2018. In response, FEMA officials in Puerto Rico stated they held several briefings with Puerto Rico’s central recovery office to explain FEMA’s interpretation of the section. Further, FEMA officials in the USVI told us that initially, they had difficulty obtaining clarification from FEMA headquarters regarding how to implement key components of section 20601 of the Act. As of May 2019, FEMA officials in the USVI stated that they continue to move forward with developing alternative procedures projects. USVI officials also told us that FEMA had been responsive and helpful in identifying its options for using the new authorities the Act provides. We will continue to evaluate these identified challenges and any efforts to address them, as well as other aspects of recovery efforts in the USVI and Puerto Rico, and plan to report our findings in late 2019 and early 2020, respectively. The Individual Assistance program provides financial and direct assistance to disaster victims for expenses and needs that cannot be met through other means, such as insurance. In May 2019, we reported on FEMA’s effort to provide disaster assistance under the Individual Assistance program to older adults and people with disabilities following the 2017 hurricanes. We found that aspects of the application process for FEMA assistance were challenging for older individuals and those with disabilities. Further, according to stakeholders and FEMA officials, disability-related questions in the Individual Assistance registration materials were confusing and easily misinterpreted. While FEMA had made some efforts to help registrants interpret the questions, we recommended, among other things, that FEMA (1) implement new registration-intake questions that improve FEMA’s ability to identify and address survivors’ disability-related needs, and (2) improve communication of registrants’ disability-related information across FEMA programs. DHS concurred with the first recommendation and described steps FEMA plans to take, or is in the process of taking, to address it. However, DHS did not concur with the second recommendation, noting that it lacks specific funding to augment its legacy data systems. FEMA officials stated that they began a long-term data management improvement initiative in April 2017, which they expect will ease efforts to share and flag specific disability-related data. While we acknowledge FEMA’s concerns about changing legacy systems when it has existing plans to replace those systems, we continue to believe there are other cost-effective ways that are likely to improve communication of registrants’ disability-related information prior to implementing the system upgrades. For example, FEMA could revise its guidance to remind program officials to review the survivor case file notes to identify whether there is a record of any disability-related needs. We also have work underway to assess FEMA’s Individuals and Households Program, a component program of Individual Assistance. Through this program, as of April 2019, FEMA had awarded roughly $4.7 billion in assistance to almost 1.8 million individuals and households for federally-declared disasters occurring in 2017 and 2018. Specifically, we are analyzing Individuals and Households Program expenditures and registration data for recent years; reviewing FEMA’s processes, policies, and procedures for making eligibility and award determinations; and examining survivors’ reported experiences with this program, including any challenges, for major disaster declarations occurring in recent years. We plan to report our findings in early 2020. FEMA’s experiences during the 2017 disasters highlight the importance of continuing to make progress on addressing the long-standing workforce management challenges we have previously reported on and continue to observe in our ongoing work. In September 2018, we reported that the 2017 disasters—hurricanes Harvey, Irma, and Maria, as well as the California wildfires—resulted in unprecedented FEMA workforce management challenges, including recruiting, maintaining, and deploying a sufficient and adequately-trained FEMA disaster workforce. FEMA’s available workforce was overwhelmed by the response needs caused by the sequential and overlapping timing of the three hurricanes. For example, at the height of FEMA workforce deployments in October 2017, 54 percent of staff were serving in a capacity in which they did not hold the title of “Qualified”—according to FEMA’s qualification system standards—a past challenge we identified. FEMA officials noted that staff shortages, and lack of trained personnel with program expertise led to complications in its response efforts, particularly after Hurricane Maria. In February 2016, we reported on, among other things, FEMA’s efforts to implement, assess, and improve its Incident Management Assistance Team program. We found that while FEMA used some leading practices in managing the program, it lacked a standardized plan to ensure that all national and regional Incident Management Assistance Team members received required training. Further, we found that the program had experienced high attrition since its implementation in fiscal year 2013. We recommended, among other things, that FEMA develop (1) a plan to ensure that Incident Management Assistance Teams receive required training, and (2) a workforce strategy for retaining Incident Management Assistance Team staff. DHS concurred with the recommendations. FEMA fully implemented our first recommendation by developing an Incident Management Assistance Team Training and Readiness Manual and providing a training schedule for fiscal year 2017. In response to the second recommendation, FEMA officials stated in July 2018 that they plan to develop policies that will provide guidance on a new workforce structure, incentives for Incident Management Assistance Team personnel, and pay-for-performance and all other human resource actions. We are continuing to monitor FEMA’s efforts to address this recommendation. In November and December 2017, we reported on staffing challenges in FEMA’s Public Assistance program. In November 2017, we reported on FEMA’s efforts to address past workforce management challenges through its redesigned Public Assistance delivery model. As part of the redesign effort, FEMA created consolidated resource centers to standardize and centralize Public Assistance staff responsible for managing grant applications, and new specialized positions to ensure more consistent guidance to applicants. However, we found that FEMA had not assessed the workforce needed to fully implement the redesigned model, such as the number of staff needed to fill certain new positions, or to achieve staffing goals. Further, in December 2017, we reported on FEMA’s management of its Public Assistance appeals process, including that FEMA increased staffing levels for the appeals process from 2015 to 2017. However, we found that FEMA continued to face a number of workforce challenges, such as staff vacancies, turnover, and delays in training, which contributed to processing delays. Based on our findings from our November and December 2017 reports, we recommended, among other things, that FEMA (1) complete workforce staffing assessments that identify the appropriate number of staff needed to implement the redesigned Public Assistance delivery model, and (2) document steps for hiring, training, and retaining key appeals staff, and address staff transitions resulting from deployments to disasters. FEMA concurred with our recommendations to address workforce management challenges in the Public Assistance program and have reported taking some actions in response. For example, to address the first recommendation, FEMA officials have developed preliminary models and estimates of staffing needs across various programs, including Public Assistance, and plan to reevaluate the appropriate number of staff needed and present recommendations to senior leadership by the end of June 2019. To address the second recommendation, FEMA has collected information on the amount of time regional appeals analysts spend on appeals, and the inventory and timeliness of different types of appeals. FEMA officials stated in September 2018 that they plan to assess this information to prepare a detailed regional workforce plan. As of June 2019, we are evaluating plans and documents provided by FEMA to determine whether they have fully addressed this recommendation. In our March 2019 report on the status of recovery efforts in Puerto Rico, we also reported Puerto Rico officials’ concerns about FEMA staff turnover and lack of knowledge among FEMA staff about how the Public Assistance alternative procedures are to be applied in Puerto Rico. As part of our ongoing work, we are continuing to examine recovery efforts in Puerto Rico. Our preliminary observations indicate that the concerns we reported on in our March 2019 report continue. For example, Puerto Rico agency officials said that the lack of continuity in FEMA personnel has been a challenge for communication and project development. Further, officials from all seven Puerto Rico government agencies we interviewed felt that the FEMA staff they interacted with did not have a complete understanding of FEMA processes and policies. We are continuing to evaluate FEMA’s recovery efforts in Puerto Rico and plan to issue our findings in late 2019. In April 2019, we reported on the federal government’s contracting efforts for preparedness, response, and recovery efforts related to the 2017 hurricanes and California wildfires. We found, among other things, that contracting workforce shortages continue to be a challenge for disaster response and recovery. Further, although FEMA’s 2017 after-action report recommended increasing contract support capacities, it did not provide a specific plan to do so. We also found that while FEMA evaluated its contracting workforce needs in a 2014 workforce analysis, it did not specifically consider contracting workforce needs in the regional offices or address Disaster Acquisition Response Team employees. In our April 2019 report, we recommended, among other things, that FEMA assess its workforce needs—including staffing levels, mission needs, and skill gaps—for contracting staff, to include regional offices and Disaster Acquisition Response Teams, and develop a plan, including timelines, to address any gaps. FEMA concurred with this recommendation and estimates that it will implement it in September 2019. In our May 2019 report on FEMA disaster assistance to older adults and people with disabilities following the 2017 hurricanes, we found that FEMA began implementing a new approach to assist individuals with disabilities in June 2018, which shifted the responsibility for directly assisting individuals with disabilities from Disability Integration Advisors— which are staff FEMA deploys specifically to identify and recommend actions needed to support survivors with disabilities—to all FEMA staff. To implement this new approach, FEMA planned to train all of the agency’s deployable staff and staff in programmatic offices on disability issues during response and recovery deployments. According to FEMA, a number of Disability Integration Advisors would also deploy to advise FEMA leadership in the field during disaster response and recovery. We found that while FEMA has taken some initial steps to provide training on the changes, it has not established a plan for delivering comprehensive disability-related training to all staff who will be directly interacting with individuals with disabilities. We recommended, among other things, that FEMA develop a plan for delivering training to FEMA staff that promotes competency in disability awareness and includes milestones and performance measures, and outlines how performance will be monitored. DHS concurred with this recommendation; however, officials stated that FEMA is developing a plan to include a disability integration competency in the guidance provided for all deployable staff, rather than through training. We will monitor FEMA’s efforts to develop this plan and fully address our recommendation. In addition to our prior work on FEMA’s workforce management challenges related to specific programs and functions, we are continuing to evaluate FEMA’s workforce capacity and training efforts during the 2017 and 2018 disaster seasons. Our preliminary observations indicate that there were challenges in FEMA’s ability to deploy staff with the right kinds of skills and training at the right time to best meet the needs of various disaster events. For example, according to FEMA field leadership we interviewed, for some of the functions FEMA performs in the field, FEMA had too few staff with the right technical skills to perform their missions—such as inspections of damaged properties—efficiently and effectively. For other functions, these managers also reported that they had too many staff in the early stages of the disaster, which created challenges with assigning duties and providing on-the-job training. For example, some managers reported that they were allocated more staff than needed in the initial phases of the disaster, but many lacked experience and were without someone to provide direction and mentoring to ensure they used their time efficiently and gained competence more quickly. Groups of FEMA field managers we interviewed told us that difficulties deploying the right mix of staff with the right skills led to challenges such as making purchases to support FEMA operations, problems with properly registering applicants for FEMA programs, or poor communication with nonfederal partners. Nonetheless, FEMA staff have noted that, despite any suboptimal circumstances during disaster response, they aimed to and have been able to find a way to deliver the mission. As part of this ongoing work, FEMA field leadership and managers also reported challenges using agency systems to ensure the availability of the right staff with the right skills in the right place and time. FEMA uses a system called the Deployment Tracking System to, among other things, help identify staff available to be deployed and activate and track deployments. To help gauge the experience level and training needs of its staff, the agency established the FEMA Qualification System (FQS), which is a set of processes and criteria to monitor staff experience in competently performing tasks and completing training that correspond to their job titles. According to the FQS guidance, staff who have been able to demonstrate proficient performance of all the relevant tasks and complete required training receive the designation “qualified,” and are expected to be ready and able to competently fulfill their responsibilities. Those who have not, receive the designation “trainee,” and can be expected to need additional guidance and on-the-job training. FQS designations feed into the Deployment Tracking System as one key variable in how the tracking system deploys staff. Among other challenges with FEMA’s Deployment Tracking System and Qualification System, FEMA managers and staff in the field told us an employee’s recorded qualification status was not a reliable indicator of the level at which deployed personnel would be capable of performing specific duties and responsibilities or their general proficiency in their positions, making it more difficult for managers to know the specialized skills or experience of staff and effectively build teams. We are continuing to assess these and other reported workforce challenges and plan to report our findings in January 2020. In April 2019, we reported on FEMA’s Grants Management Modernization program, which is intended to replace the agency’s 10 legacy grants management systems and modernize and streamline the grants management environment. We found that, of six important leading practices for effective business process reengineering and information technology requirements management, FEMA fully implemented four and partially implemented two for the Grants Management Modernization program. The two partially implemented leading practices were (1) establishing plans for implementing new business processes and (2) establishing complete traceability of information technology requirements. In addition, we found that the program’s initial May 2017 cost estimate of about $251 million was generally consistent with leading practices for a reliable, high-quality estimate; however, it no longer reflected the current assumptions about the program at the time of our review. Moreover, the program’s schedule–specifically its final delivery date of September 2020—did not reflect leading practices for project schedules, as the date was not informed by a realistic assessment of development activities. Lastly, we found that FEMA fully addressed three and partially addressed two of five key cybersecurity practices. The two partially addressed practices were (1) assessing security controls, and (2) obtaining an authorization to operate the system. We made 8 recommendations to FEMA to implement leading practices related to reengineering processes, managing information technology requirements, scheduling system development activities, and implementing cybersecurity. DHS concurred with all of our recommendations and provided estimated completion dates for implementing each of them through July 2020. Thank you, Chairman Thompson, Ranking Member Rogers and Members of the Committee. This concludes my prepared statement. I would be happy to respond to any question you may have at this time. If you or your staff has any questions concerning this testimony, please contact Christopher P. Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement were Joel Aldape (Assistant Director), Amanda R. Parker (Analyst-in-Charge), Matthew T. Lowney, Rebecca Mendelsohn, and David (Ben) Nelson. In addition, Aditi Archer, Bryan Bourgault, Lorraine Ettaro, Aaron Gluck, Kathryn Godfrey, Taylor Hadfield, Eric Hauswirth, Robert (Denton) Herring, Adam Hoffman, Susan Hsu, Sara Kelly, Amy Moran Lowe, Heidi Nielson, Danielle Pakdaman, Sara Pelton, Amanda Prichard, and Johanna Wong made contributions to this statement. Key contributors for the previous work that this is based on are listed in each product. Opportunities to Reduce Potential Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue. GAO-11-318SP, March 1, 2011. Federal Disaster Assistance: Improved Criteria Needed to Assess a Jurisdiction’s Capability to Respond and Recover on Its Own. GAO-12- 838, September 12, 2012. Fiscal Exposures: Improving Cost Recognition in the Federal Budget. GAO-14-28, October 29, 2013. Emergency Preparedness: Opportunities Exist to Strengthen Interagency Assessments and Accountability for Closing Capability Gaps. GAO-15-20, December 4, 2014. High-Risk Series: An Update. GAO-15-290, February 11, 2015. Budgeting for Disasters: Approaches to Budgeting for Disasters in Selected States. GAO-15-424, March 26, 2015. Hurricane Sandy: An Investment Strategy Could Help the Federal Government Enhance National Resilience for Future Disasters. GAO-15- 515, July 30, 2015. Disaster Response: FEMA Has Made Progress Implementing Key Programs, but Opportunities for Improvement Exist. GAO-16-87, February 5, 2016. Disaster Recovery: FEMA Needs to Assess Its Effectiveness in Implementing the National Disaster Recovery Framework. GAO-16-476, May 26, 2016. Federal Disaster Assistance: Federal Departments and Agencies Obligated at Least $277.6 Billion during Fiscal Years 2005 through 2014. GAO-16-797, September 22, 2016. Climate Change: Information on Potential Economic Effects Could Help Guide Federal Efforts to Reduce Fiscal Exposure. GAO-17-720, September 28, 2017. Disaster Assistance: Opportunities to Enhance Implementation of the Redesigned Public Assistance Grant Program. GAO-18-30, November 8, 2017. Disaster Recovery: Additional Actions Would Improve Data Quality and Timeliness of FEMA’s Public Assistance Appeals Processing. GAO-18- 143, December 15, 2017. 2017 Disaster Contracting: Observations on Federal Contracting for Response and Recovery Efforts. GAO-18-335, February 28, 2018. Federal Disaster Assistance: Individual Assistance Requests Often Granted but FEMA Could Better Document Factors Considered. GAO-18- 366, May 31, 2018. 2017 Hurricanes and Wildfires: Initial Observations on the Federal Response and Key Recovery Challenges. GAO-18-472, September 4, 2018. Homeland Security Grant Program: Additional Actions Could Further Enhance FEMA’s Risk-Based Grant Assessment Model. GAO-18-354, September 6, 2018. Continuity of Operations: Actions Needed to Strengthen FEMA’s Oversight and Coordination of Executive Branch Readiness. GAO-19- 18SU, November 26, 2018. 2017 Disaster Contracting: Action Needed to Better Ensure More Effective Use and Management of Advance Contracts. GAO-19-93, December 6, 2018. U.S. Virgin Islands Recovery: Status of FEMA Public Assistance Funding and Implementation. GAO-19-253, February 25, 2019. High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas. GAO-19-157SP, March 6, 2019. Puerto Rico Hurricanes: Status of FEMA Funding, Oversight, and Recovery Challenges. GAO-19-256, March 14, 2019. Huracanes de Puerto Rico: Estado de Financiamiento de FEMA, Supervisión y Desafíos de Recuperación. GAO-19-331, March 14, 2019. Disaster Recovery: Better Monitoring of Block Grant Funds Is Needed. GAO-19-232, March 25, 2019. FEMA Grants Modernization: Improvements Needed to Strengthen Program Management and Cybersecurity. GAO-19-164, April 9, 2019. 2017 Hurricane Season: Federal Support for Electricity Grid Restoration in the U.S. Virgin Islands and Puerto Rico. GAO-19-296, April 18, 2019. Disaster Contracting: Actions Needed to Improve the Use of Post- Disaster Contracts to Support Response and Recovery, GAO-19-281, April 24, 2019. Disaster Assistance: FEMA Action Needed to Better Support Individuals Who Are Older or Have Disabilities. GAO-19-318, May 14, 2019. 1. Review of U.S. Virgin Islands recovery planning and progress; 2. Puerto Rico disaster recovery planning and progress; 3. 2017 wildfire response and recovery; 4. Federal internal control plans for disaster assistance funding; 5. Electricity grid restoration and resilience after the 2017 hurricane 6. Mass care sheltering and feeding challenges during the 2017 7. Department of Transportation highway and transit emergency relief 8. Drinking water and wastewater utility resilience; 9. Review of disaster death count information in selected states and 10. Department of Health and Human Services disaster response efforts; 11. Disaster and climate change impacts on Superfund sites; 12. FEMA Public Assistance program fraud risk management efforts; 13. Wildland fire collaboration on fuel reduction efforts; 14. Preparedness challenges and lessons learned from the 2017 15. FEMA workforce management and challenges; 16. Small Business Administration response to 2017 disasters; 17. Development of the GAO disaster resilience framework; 18. FEMA Individuals and Households Program operations and 19. National Flood Insurance Program post-flood enforcement; 20. Emergency alerting capabilities and progress; 21. National Flood Insurance Program buyouts and property acquisitions; 22. Economic costs of large-scale natural disasters and impacts on 23. Community Development Block Grants – disaster recovery; and 24. Disaster Housing Assistance Program. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Recent hurricanes, wildfires, and flooding have highlighted the challenges the federal government faces in responding effectively to natural disasters. The 2017 and 2018 hurricanes and wildfires affected millions of individuals and caused billions of dollars in damages. In March 2019, the Midwest experienced historic flooding that affected millions of acres of agriculture and damaged significant infrastructure. Since 2005, federal funding for disaster assistance is at least $450 billion. Increasing reliance on federal help to address natural disasters is a key source of federal fiscal exposure, particularly as certain extreme weather events become more frequent and intense due to climate change. This statement discusses, among other things, FEMA's progress and challenges related to disaster resilience, response, recovery, and workforce management. This statement is based on GAO reports issued from March 2011 through May 2019, and also includes preliminary observations from ongoing GAO reviews of FEMA operations. For ongoing work, GAO reviewed federal laws; analyzed documents; interviewed agency officials; and visited disaster damaged areas in California, Florida, South Carolina, North Carolina, Puerto Rico, Texas, and the U.S. Virgin Islands, where GAO also interviewed FEMA and local officials. GAO's issued and ongoing work identified progress and challenges in the Federal Emergency Management Agency's (FEMA) disaster resilience, response, recovery, and workforce management efforts, as discussed below. Disaster Resilience. GAO found that federal and local efforts to improve resilience can reduce the effects and costs of future disasters. FEMA has made progress in this area, but in July 2015, GAO found that states and localities faced challenges using federal funds to maximize resilient rebuilding following a disaster. GAO recommended that the Mitigation Framework Leadership Group—an interagency body chaired by FEMA—create a national strategy to better plan for and invest in disaster resilience. FEMA is working to address this recommendation and plans to publish the strategy by July 2019. Response and Recovery. In September 2018, GAO reported that the response to the 2017 disasters in Texas, Florida, and California showed progress since Hurricane Katrina in 2005. Specifically, FEMA and state officials' pre-existing relationships and exercises aided the response and helped address various challenges. However, GAO and FEMA identified challenges that slowed and complicated FEMA's response to Hurricane Maria, particularly in Puerto Rico. GAO's issued and ongoing work also identified challenges in implementing FEMA Public Assistance grants. For example, FEMA and Puerto Rico officials identified challenges with Public Assistance policies and guidance that have complicated and slowed the recovery. GAO did not make recommendations, but continues to evaluate recovery efforts and will report its findings later this year. FEMA Workforce Management. GAO has previously reported on long-standing workforce management challenges, such as ensuring an adequately-staffed and trained workforce. For example, GAO reported in September 2018 that the 2017 disasters overwhelmed FEMA's workforce and a lack of trained personnel with program expertise led to complications in its response efforts, particularly after Hurricane Maria. While FEMA has taken actions to address several of GAO's workforce management-related recommendations since 2016, a number of recommendations remain open as the 2019 hurricane season begins. Also, GAO is currently reviewing FEMA's workforce management efforts and lessons learned from the 2017 disasters and will report its findings early next year. GAO has made numerous recommendations in its prior reports to FEMA designed to address the challenges discussed in this statement. As of May 2019, FEMA has addressed about half of these recommendations and GAO is monitoring FEMA's ongoing efforts.", "document_type": "gao"}
{"report": "DOD space systems support and provide a wide range of capabilities to a large number of users, including the military services, the intelligence community, civil agencies, and others. These capabilities include positioning, navigation, and timing; meteorology; missile warning; and secure communications, among others. Space systems can take a long time to develop and involve multiple segments, including space, ground control stations, terminals, user equipment, and launch, as figure 1 below shows. DOD satellite systems are also expensive to acquire. Unit costs for current DOD satellites can range from $500 million to over $3 billion. The associated ground systems can cost over $6 billion to develop and maintain and the cost to launch a satellite can climb to well over $100 million. Table 1 provides highlights of the current status of DOD’s major space programs. As the table shows, DOD is also in the beginning phases of acquiring several constellations of new satellites and ground processing capabilities—including for missile warning, protected communications, space-based environmental monitoring, and space command and control. We have work underway to assess the Air Force’s space command and control development efforts and examine DOD’s analysis of alternatives for wideband communication services. For a more complete description of these major space programs, see appendix I. In addition, DOD is exploring alternatives for acquiring wideband satellite communications as well as funding development of new launch vehicles as it pursues a new acquisition strategy for procuring launch services. Our prior work has shown that many major DOD space programs have experienced significant cost increases and schedule delays. For instance, the total program cost for the Advanced Extremely High Frequency (AEHF) satellite program, a protected satellite communications system, has grown 117 percent since the program’s original cost estimate and its first satellite was launched more than 3.5 years late. For the Space Based Infrared System (SBIRS), a missile warning satellite program, the program cost grew 265 percent from its original estimate and the launch of the first satellite was delayed roughly 9 years. Both programs moved to the production phase where fewer problems tend to surface, and where there is typically less risk of significant cost and schedule growth. A more recent major satellite program, Global Positioning System (GPS) III, has seen an almost 4-year delay due to technical issues and program cost growth of about 32 percent. Cost and schedule growth has also been a challenge for satellite ground systems and user equipment. Ground system delays have been so lengthy, that satellites sometimes spend years in orbit before key capabilities can be fully exploited. For example, The command and control system for GPS III satellites, known as the Next Generation Operational Control System, or OCX, is approximately 5 years behind schedule. As a result, the Air Force has had to start two separate back-up efforts to modify the current ground system to ensure the continuity of GPS capabilities and to make anti- jamming capabilities available via Military Code, or M-code, until OCX is delivered. Our ongoing review of GPS includes an assessment of OCX schedule risk and potential impacts on OCX delivery, acceptance, and operation. We expect to issue our report on GPS in spring 2019. Development of GPS user equipment that can utilize the M-Code signal has lagged behind the fielding of GPS M-code satellites for more than a decade, due to prolonged development challenges. In December 2017, we found that while DOD had made some progress on initial testing of the receiver cards needed to utilize the M-code signal, additional development was necessary to make M-code work with the over 700 weapon systems that require it. We also found that DOD had begun initial planning to transition some weapon systems to use M-code receivers, but significantly more work remained to understand the cost and schedule of transitioning to M-code receivers across DOD. Further, in December 2017, we found that multiple entities were separately maturing their own receiver cards. We recommended that DOD assign responsibility to a single organization to collect test data, lessons learned, and design solutions so that common design solutions are employed and DOD could avoid duplication of efforts. DOD concurred with the recommendation, but has not yet taken action on it. We have previously reported that over 90 percent of the capabilities to be provided by Mobile User Objective System communications satellites—currently, five satellites are in orbit, the first of which launched in 2012—are being underutilized because of difficulties with integrating the space, ground, and terminal segments and delays in fielding compatible user terminals. Largely because of technical and management challenges, the Joint Space Operations Center Mission System (JMS) Increment 2 program—intended to replace and improve upon an aging space situational awareness and command and control system—was almost 3 years behind schedule and 42 percent over budget before the Air Force stopped development work last year. Earlier this month, we reported that operational testing in 2018 found that JMS Increment 2 was not operationally effective or suitable due, in part, to missing software requirements, urgent deficiencies that affected system performance, and negative user feedback. Cost and schedule growth in DOD’s space programs is sometimes driven by the inherent risks associated with developing complex space technology; however, over the past 10 years we have identified a number of other management and oversight problems that have worsened the situation. These include making overly optimistic cost and schedule estimates, pushing programs forward without sufficient knowledge about technology and design, and experiencing problems in overseeing and managing contractors, among others. We have also noted that some of DOD’s programs with operational satellites, such as SBIRS, were also exceedingly ambitious, which in turn increased technology, design, and engineering risks. While SBIRS and other satellite programs provide users with important and useful capabilities, their cost growth has significantly limited the department’s buying power at a time when more resources may be needed to protect space systems and recapitalize the space portfolio. DOD faces significant challenges as it replenishes its satellite constellations. First, DOD is confronted with growing threats in space, which may require very different satellite architectures and acquisition strategies. Second, DOD is in the midst of planning major changes to its leadership for space. While these changes are designed to streamline decision-making and bring together a dispersed space workforce, they could cause some disruption to space system acquisition programs. Third, in fiscal year 2016, Congress gave DOD authority to speed up acquisition timeframes by streamlining acquisition processes and oversight. GAO is examining DOD’s application of streamlining to its weapons programs. For space, challenges with past streamlining efforts may offer some lessons learned. And fourth, DOD may face resource and capacity challenges in taking on multiple space acquisitions at one time. For example, our work and other reports point to potential gaps in the space acquisition workforce and ongoing difficulties managing software development. According to Air Force Space Command and others, U.S. space systems face intentional and unintentional threats that have increased rapidly over the past 20 years. These include radio frequency interference (including jamming), laser attacks, kinetic intercept vehicles, and ground system attacks. Additionally, the hazards of the already-harsh space environment (e.g., extreme temperature fluctuations and radiation) have increased, including numbers of active and inactive satellites, spent rocket bodies, and other fragments and debris. According to a February 2019 Defense Intelligence Agency report, China and Russia in particular are developing a variety of means to exploit perceived U.S. reliance on space-based systems and challenge the U.S. position in space. The report also states that Iran and North Korea have demonstrated some counterspace capabilities that could pose a threat to militaries using space-based services. In response, recent governmentwide and DOD strategic and policy guidance have stressed the need for U.S. space systems to be survivable or resilient against such threats and DOD has taken steps to be more resilient in some of its new programs. As we found in October 2014, one way to do this is to build more disaggregated systems, including dispersing sensors onto separate satellites; using multiple domains, including space, air, and ground to provide full mission capabilities; hosting payloads on other government or commercial spacecraft; or some combination of these. With capabilities distributed across multiple platforms, rather than centralized onto just a few satellites, it may be more difficult for an adversary to target all assets to attack full system capabilities, and if an attack does take place, the loss of one smaller satellite or payload could result in less capability loss than damage to, or loss of, a large multifunctional satellite. In addition to disaggregation, DOD could make satellites more maneuverable and build in defense capabilities to protect themselves as a means to increase survivability. We also found in October 2014 that some of these options could have beneficial impacts on acquisition. For example, acquiring smaller, less complex satellites may require less time and effort to develop and produce. This may be in part due to improved requirements discipline, as more frequent production rates may allow program managers to delay new requirements to the next production cycle instead of incorporating them into ongoing timelines midstream. Building more, less-complex satellites might also provide DOD the opportunity to use commercial products and systems that have already been tested in the market. At the same time, however, addressing the need to make satellites more resilient could introduce complications. For example, DOD may need to acquire higher quantities of satellites, which may make it more difficult to manage acquisition schedules. In addition, potentially more development and production contracts may result in more complexity for program offices to manage, requiring increased oversight of contractors. Adding more satellites and new technologies may also complicate efforts to synchronize satellite, terminal, and ground system schedules, limiting delivery of capabilities to end users. Our work has also found potential barriers to making satellites more resilient. For example, in October 2014, we found that disaggregation could require DOD to make significant cultural and process changes in how it acquires space systems—for instance, by relying on new contractors, relinquishing control to providers who host government payloads on commercial satellites, using different contracting methods, and executing smaller but more numerous and faster-paced acquisition programs. It will likely require DOD to be more flexible and agile when it comes to satellite acquisitions, especially with regard to coordinating satellite delivery with interdependent systems, such as user equipment. Yet, as we have previously found, DOD’s culture has generally been resistant to changes in space acquisition approaches, and fragmented responsibilities have made it very difficult to coordinate and deliver interdependent systems. Senior leaders have recognized the need to change the space acquisition culture, and as discussed below, changes are being made to space leadership and acquisition approaches. More recently, in July 2018, we found that two factors have contributed to DOD’s limited use of commercially hosted payloads. First, DOD officials identified logistical challenges to matching government payloads with any given commercial host satellite. For example, most of the offices we spoke with cited size, weight, and power constraints, among others, as barriers to using hosted payloads. Second, while individual DOD offices have realized cost and schedule benefits from using hosted payloads, DOD as a whole has limited information on costs and benefits of hosted payloads. Further, the knowledge DOD obtained is fragmented across the agency—with multiple offices collecting piecemeal information on the use of hosted payloads. The limited knowledge and data on hosted payloads that is fragmented across the agency has contributed to resistance among space acquisition officials to adopting this approach. We recommended, and DOD concurred, that the department bolster and centralize collection and analysis of cost, technical, and lessons learned data on its use of hosted payloads. Lastly, in October 2018, we found that DOD faced mounting challenges in protecting its weapon systems—satellites and their ground systems included—from increasingly sophisticated cyber threats. We reported that this was due to the computerized nature of weapon systems, DOD’s late start in prioritizing weapon system cybersecurity, and DOD’s nascent understanding of how to develop more secure weapon systems. In operational testing, DOD routinely found mission-critical cyber vulnerabilities in systems that were under development, yet program officials GAO met with believed their systems were secure and even discounted some test results as unrealistic. Using relatively simple tools and techniques, testers were able to take control of systems and operate largely undetected, due in part to basic issues such as poor password management and unencrypted communications. DOD has recently taken several steps to improve weapon system cybersecurity, including issuing and revising policies and guidance to better incorporate cybersecurity considerations. Further, in response to congressional direction, DOD has also begun initiatives to better understand and address cyber vulnerabilities. We and others have reported for over two decades that fragmentation and overlap in DOD space acquisition management and oversight have contributed to program delays and cancellations, cost increases, and inefficient operations. For example, in February 2012 we found that fragmented leadership contributed to a 10-year gap between the delivery of GPS satellites and associated user equipment. The cancellations of several large programs over the past 2 decades were in part because of disagreements and conflicts among stakeholders. In July 2016, in response to a provision of a Senate Report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2016, we issued a report that reviewed space leadership in more depth and concluded that DOD space leadership was fragmented. We identified approximately 60 stakeholder organizations across DOD, the Executive Office of the President, the Intelligence Community, and civilian agencies. Of these, eight organizations had space acquisition management responsibilities; eleven had oversight responsibilities; and six were involved in setting requirements for defense space programs. At the same time, many experts stated that no one seemed to be in charge of space acquisitions. Our report highlighted the pros and cons of various options to reorganize space functions recommended in prior congressionally-chartered studies. The issue has taken on more importance in recent years, as DOD has realized satellites are highly vulnerable to attacks and needs to make dramatic changes in space system architectures and operations. We have found that leadership has not been focused enough to overcome interagency rivalries and resistance to change, and it has not been able to get concurrence on future architectures. The President’s Administration and DOD have taken significant steps to change space leadership. Most recent is the President’s Space Policy Directive-4, issued on February 19, 2019, and DOD’s subsequent legislative proposal submitted on March 1, 2019, to establish a United States Space Force as a sixth branch of the United States Armed Forces within the Department of the Air Force. The Policy Directive states that this is an important step toward a future military department for space and that the Space Force will (1) consolidate existing forces and authorities for military space activities, as appropriate, to minimize duplication of effort and eliminate bureaucratic inefficiencies; and (2) not include the National Aeronautics and Space Administration, the National Oceanic and Atmospheric Administration, the National Reconnaissance Office, or other non-military space organizations or missions of the United States Government. According to the Policy Directive, the Space Force would include the uniformed and civilian personnel conducting and directly supporting space operations from all DOD Armed Forces, assume responsibilities for all major military space acquisition programs, and create the appropriate career tracks for military and civilian space personnel across all relevant specialties. Pertaining to organization and leadership, the Policy Directive creates a civilian Under Secretary of the Air Force for Space, to be known as the Under Secretary for Space, appointed by the President, and establishes a Chief of Staff of the Space Force, who would serve as a member of the Joint Chiefs of Staff. Furthermore, the Policy Directive states that as the Space Force matures, and as national security requires, it will become necessary to create a separate military department, to be known as the Department of the Space Force. This department would take over some or all responsibilities for the Space Force from the Department of the Air Force. The Policy Directive requires the Secretary of Defense to conduct periodic reviews to determine when to recommend that the President seek legislation to establish such a department. Our past work has identified fragmentation in space leadership, but because implementation has not yet occurred, it remains to be seen whether this policy directive and proposed legislation would resolve these issues. In implementing these changes there are many complexities to consider. For example, because space capabilities are acquired and used across the military services and defense agencies, it will be important to address many details on how to implement a Space Force among these equities. Our past work suggests that without close attention to the consequences of the compromises that will inevitably have to be made to carve out a new force structure from existing space functions, there is risk of exacerbating the fragmentation and ineffective management and oversight the Space Force is intended to address. For instance, earlier this month, DOD established the Space Development Agency to unify and integrate efforts across DOD to define, develop, and field innovative solutions. But it is unclear how this new organization will mesh with the Air Force Space and Missile Systems Center, which acquires satellites, the Defense Advanced Research Projects Agency, which creates breakthrough technologies and capabilities, and similar organizations. Moreover, even if changes are implemented effectively, they are only a first step toward addressing space acquisition problems. As we discuss below, programs will still need to embrace acquisition best practices, such as using demonstrable knowledge to make decisions. Our prior work has found that they will also need to be open to flexible and innovative approaches, and work effectively with a very wide range of stakeholders, including those that will not be part of the Space Force, such as the intelligence agencies, civilian space agencies, the current military services, as well as entities within the Office of the Secretary of Defense who help oversee and manage acquisitions. Senior leaders have acknowledged that additional changes are needed and have taken steps to help bring them about, such as the restructuring of the Air Force’s Space and Missile Systems Center, which is designed to break down stovepipes and streamline acquisition processes. DOD is managing a number of new space acquisition programs using a new authority, established under Section 804 of the National Defense Authorization Act for Fiscal Year 2016, which is to provide a streamlined alternative to the traditional DOD acquisition process. Specifically, the programs—which include follow-on missile warning and protected communications satellites, among others—will be exempted from the acquisition and requirements processes defined by DOD Directive 5000.01 and the Joint Capabilities Integration and Development System. Instead, program managers are encouraged to use a tailored approach to documentation and oversight to enable them to demonstrate new technologies or field new or updated systems within 2 to 5 years. We have ongoing work looking across the military departments at how middle-tier acquisition authority is being implemented, including for the Air Force’s space acquisition programs, and plan to issue a report later this spring. GAO and others have highlighted lessons learned from past efforts to streamline, specifically with an approach adopted for space systems in the 1990s known as Total System Performance Responsibility (TSPR). TSPR was intended to facilitate acquisition reform and enable DOD to streamline its acquisition process and leverage innovation and management expertise from the private sector. Specifically, TSPR gave a contractor total responsibility for the integration of an entire weapon system and for meeting DOD’s requirements. We found in May 2009 that because this reform made the contractor responsible for day-to-day program management, DOD did not require formal deliverable documents—such as earned value management reports—to assess the status and performance of the contractor. As a result, DOD’s capability to lead and manage the space acquisition process diminished, which magnified problems related to unstable requirements and poor contractor performance. Further, the reduction in DOD oversight and involvement led to major reductions in various government capabilities, including cost- estimating and systems-engineering staff. This, in turn, led to a lack of technical data needed to develop sound cost estimates. Best practices that we identified in the aftermath of TSPR include retaining strong oversight and insight into programs; using quantifiable data and demonstrable knowledge to make decisions to proceed, not allowing development to proceed until certain thresholds are met, empowering program managers to make decisions on the direction of the program but also holding them accountable for their choices, and canceling unsuccessful programs. Similarly, in its study of TSPR programs, the Defense Science Board/Air Force Scientific Advisory Board Joint Task Force emphasized the importance of managing requirements, sufficiently funding programs, participating in trade-off studies, and assuring that proven engineering practices characterize program implementation, among other actions. See appendix II for a more complete list of the best practices we have identified for developing complex systems. DOD is simultaneously undertaking new major acquisition efforts to replenish its missile warning, protected communications, GPS, and weather satellites. At the same time, it is boosting efforts to increase space situational awareness and protect space assets. It is also helping to fund the development of new launch vehicles, and it is considering additional significant acquisitions in wideband satellite communications and in support of missile defense activities. While there is increased attention within DOD on funding for space and building the Space Force, such widespread acquisition activities could still pose resource challenges. For example: Funding requests for space system modernization have in the past 10 years represented a small percentage (3.9 to 5 percent) of total weapon system modernization funding DOD requested. Space is competing with ships, aircraft, and the nuclear triad, among other programs for funding. This can be challenging, because over the past 2 years, DOD has begun over 9 new space acquisition programs to recapitalize current space capabilities and enhance system resiliency. In the past, we have found that it has been difficult for DOD to fund multiple new space programs at one time, particularly when it was concurrently struggling with cost overruns and schedule delays from its legacy programs. For example, OCX system development challenges have resulted in a $2.5 billion cost increase and approximate 5-year delay to the system becoming operational— using more resources for a longer time—at a cost to other programs. It is unclear whether DOD has a sufficient workforce to manage multiple new space programs. We issued a report this month that found DOD did not routinely monitor the size, mix, and location of its space acquisition workforce. We collected and aggregated data from multiple DOD space acquisition organizations and found that at least 8,000 personnel in multiple locations nationwide were working on space acquisition activities at the end of 2017. Echoing concerns raised in our prior work, we also found that DOD had difficulty attracting and retaining candidates with the requisite technical expertise. Officials from the Air Force’s Space and Missile Systems Center were concerned that there are not enough experienced mid- level acquisition personnel and also expressed concern that the bulk of military personnel assigned to program management positions were more junior in rank than the Center was authorized to obtain. We recommended that DOD (1) identify the universe of its space acquisition programs and the organizations that support them, and (2) collect and maintain data on the workforce supporting these programs. DOD concurred with our first recommendation but not the second. Software is an increasingly important enabler of DOD space systems. However, DOD has struggled to deliver software-intensive space programs that meet operational requirements within expected time frames. Although user involvement is critical to the success of any software development effort, we found in our report issued earlier this month on DOD software-intensive space programs that key programs that experienced cost or schedule breaches often did not effectively engage users to understand requirements and obtain feedback. Program efforts to involve users and incorporate feedback frequently did not match plans. The lack of user engagement has contributed to systems that were later found to be operationally unsuitable. The programs we reviewed also faced challenges in delivering software in shorter time frames, and in using commercial software, applying outdated tools and metrics, as well as having limited knowledge and training in newer software development techniques. DOD acknowledged these challenges and is taking steps to address them, including identifying useful software development metrics and ways to include them in new contracts. We recommended, and DOD concurred, that the department ensure its guidance addressing software development provides specific, required direction on the timing, frequency, and documentation of user involvement and feedback. Moreover, it should be noted that software development has been a struggle for other non-space weapons programs as well. The Defense Innovation Board recently reported that the department’s current approach to software development is broken and is a leading source of risk to DOD—it takes too long, is too expensive, and exposes warfighters to unacceptable risk by delaying their access to the tools they need to assure mission success. Chairman Fischer, Ranking Member Heinrich, and Members of the Subcommittee, this concludes my statement. I am happy to answer any questions that you have. If you or your staff have any questions about this statement, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contacts for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this statement include Rich Horiuchi, Assistant Director; Burns C. Eckert; Emily Bond; Claire Buck; Maricela Cherveny; Erin Cohen; Susan Ditto; Laura Hook, and Anne Louise Taylor. Key contributors for the previous work on which this statement is based are listed in the products cited. Our previous work on weapons acquisitions in general, and space programs in particular, identified best practices for developing complex systems. We summarize these best practices in table 3, below. DOD Space Acquisitions: Including Users Early and Often in Software Development Could Benefit Programs. GAO-19-136. Washington, D.C.: March 18, 2019. Defense Space Systems: DOD Should Collect and Maintain Data on Its Space Acquisition Workforce. GAO-19-240. Washington, D.C.: March 14, 2019. Weapon Systems Cybersecurity: DOD Just Beginning to Grapple with Scale of Vulnerabilitie., GAO-19-128. Washington, D.C.: October 9, 2018. Military Space Systems: DOD’s Use of Commercial Satellites to Host Defense Payloads Would Benefit from Centralizing Data. GAO-18-493. Washington, D.C.: July 30, 2018. Weapon Systems Annual Assessment: Knowledge Gaps Pose Risks to Sustaining Recent Positive Trends. GAO-18-360SP. Washington, D.C.: April 25, 2018. Global Positioning System: Better Planning and Coordination Needed to Improve Prospects for Fielding Modernized Capability. GAO-18-74. Washington, D.C.: December 12, 2017. Space Launch: Coordination Mechanisms Facilitate Interagency Information Sharing on Acquisitions GAO-17-646R. Washington D.C.: August 9, 2017 Satellite Acquisitions: Agencies May Recover a Limited Portion of Contract Value When Satellites Fail. GAO-17-490. Washington, D.C.: June 9, 2017 Space Acquisitions: DOD Continues to Face Challenges of Delayed Delivery of Critical Space Capabilities and Fragmented Leadership. GAO-17-619T. Washington, D.C.: May 17, 2017. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-17-333SP. Washington, D.C.: March 30, 2017. Global Positioning System: Observations on Quarterly Reports from the Air Force. GAO-17-162R. Washington, D.C.: October 17, 2016. Defense Space Acquisitions: Too Early to Determine if Recent Changes Will Resolve Persistent Fragmentation in Management and Oversight. GAO-16-592R. Washington, D.C.: July 27, 2016. Evolved Expendable Launch Vehicle: DOD Is Assessing Data on Worldwide Launch Market to Inform New Acquisition Strategy. GAO-16-661R. Washington, D.C.: July 22, 2016 Defense Weather Satellites: DOD Faces Acquisition Challenges for Addressing Capability Needs. GAO-16-769T, Washington, D.C.: July 7, 2016. Defense Weather Satellites: Analysis of Alternatives is Useful for Certain Capabilities, but Ineffective Coordination Limited Assessment of Two Critical Capabilities. GAO-16-252R. Washington, D.C.: March 10, 2016. Space Acquisitions: Challenges Facing DOD as it Changes Approaches to Space Acquisitions. GAO-16-471T. Washington, D.C.: March 9, 2016. Space Acquisitions: GAO Assessment of DOD Responsive Launch Report. GAO-16-156R. Washington, D.C.: October 29, 2015. Space Situational Awareness: Status of Efforts and Planned Budgets. GAO-16-6R. Washington, D.C.: October 8, 2015. GPS: Actions Needed to Address Ground System Development Problems and User Equipment Production Readiness. GAO-15-657. Washington, D.C.: September 9, 2015. Evolved Expendable Launch Vehicle: The Air Force Needs to Adopt an Incremental Approach to Future Acquisition Planning to Enable Incorporation of Lessons Learned. GAO-15-623. Washington, D.C.: August 11, 2015. Defense Satellite Communications: DOD Needs Additional Information to Improve Procurements. GAO-15-459. Washington, D.C.: July 17, 2015. Space Acquisitions: Some Programs Have Overcome Past Problems, but Challenges and Uncertainty Remain for the Future. GAO-15-492T. Washington, D.C.: April 29, 2015. Space Acquisitions: Space Based Infrared System Could Benefit from Technology Insertion Planning. GAO-15-366. Washington, D.C.: April 2, 2015. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-15-342SP. Washington, D.C.: March 12, 2015. Defense Major Automated Information Systems: Cost and Schedule Commitments Need to Be Established Earlier. GAO-15-282. Washington, D.C.: February 26, 2015. DOD Space Systems: Additional Knowledge Would Better Support Decisions about Disaggregating Large Satellites. GAO-15-7. Washington, D.C.: October 30, 2014. U.S. Launch Enterprise: Acquisition Best Practices Can Benefit Future Efforts. GAO-14-776T. Washington, D.C.: July 16, 2014. 2014 Annual Report: Additional Opportunities to Reduce Fragmentation, Overlap, and Duplication and Achieve Other Financial Benefits. GAO-14-343SP. Washington, D.C.: April 8, 2014. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-14-340SP. Washington, D.C.: March 31, 2014. Space Acquisitions: Acquisition Management Continues to Improve but Challenges Persist for Current and Future Programs. GAO-14-382T. Washington, D.C.: March 12, 2014. Evolved Expendable Launch Vehicle: Introducing Competition into National Security Space Launch Acquisitions. GAO-14-259T. Washington, D.C.: March 5, 2014. The Air Force’s Evolved Expendable Launch Vehicle Competitive Procurement. GAO-14-377R. Washington, D.C.: March 4, 2014. Space Acquisitions: Assessment of Overhead Persistent Infrared Technology Report. GAO-14-287R. Washington, D.C.: January 13, 2014. Space: Defense and Civilian Agencies Request Significant Funding for Launch-Related Activities. GAO-13-802R. Washington, D.C.: September 9, 2013. Global Positioning System: A Comprehensive Assessment of Potential Options and Related Costs is Needed. GAO-13-729, Washington, D.C.: September 9, 2013. Space Acquisitions: DOD Is Overcoming Long-Standing Problems, but Faces Challenges to Ensuring Its Investments are Optimized. GAO-13-508T. Washington, D.C.: April 24, 2013. Satellite Control: Long-Term Planning and Adoption of Commercial Practices Could Improve DOD’s Operations. GAO-13-315. Washington, D.C.: April 18, 2013. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-13-294SP. Washington, D.C.: March 28, 2013. Launch Services New Entrant Certification Guide. GAO-13-317R. Washington, D.C.: February 7, 2013. Evolved Expendable Launch Vehicle: DOD Is Addressing Knowledge Gaps in Its New Acquisition Strategy. GAO-12-822. Washington, D.C.: July 26, 2012. Space Acquisitions: DOD Faces Challenges in Fully Realizing Benefits of Satellite Acquisition Improvements. GAO-12-563T. Washington, D.C.: March 21, 2012. Space and Missile Defense Acquisitions: Periodic Assessment Needed to Correct Parts Quality Problems in Major Programs. GAO-11-404. Washington, D.C.: June 24, 2011. Space Acquisitions: Development and Oversight Challenges in Delivering Improved Space Situational Awareness Capabilities. GAO-11-545. Washington, D.C.: May 27, 2011. Space Acquisitions: DOD Delivering New Generations of Satellites, but Space System Acquisition Challenges Remain. GAO-11-590T. Washington, D.C.: May 11, 2011. Global Positioning System: Challenges in Sustaining and Upgrading Capabilities Persis., GAO-10-636. Washington, D.C.: September 15, 2010. Defense Acquisitions: Challenges in Aligning Space System Components. GAO-10-55. Washington D.C.: October 29, 2009. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "DOD space systems provide critical capabilities that support military and other government operations. They can also be expensive to acquire and field, costing billions of dollars each year. As DOD seeks to replenish its satellite constellations, it faces a number of challenges to ensuring funds are used effectively. Because space-based capabilities are fundamental to U.S. national security and civilian activities, it is essential that DOD manage its space system acquisitions carefully and avoid repeating past problems. This statement provides an update on DOD's space acquisitions, focusing on challenges facing acquisitions of new space systems. This statement is based on GAO reports issued over the past 10 years on DOD space programs. In addition it draws on recent work performed in support of GAO's 2019 annual reports on the progress of major defense acquisition programs as well as duplication, overlap, and fragmentation across the federal government, among other sources. DOD is simultaneously undertaking new major acquisitions to replenish its missile warning, protected communications, navigation, and weather satellites. At the same time, it is boosting efforts to increase space situational awareness and protect space assets. Such widespread acquisition acitivites could face a wide range of resource and management challenges that GAO has reported on, including: Growing threats to satellites . Threats to satellites from both adversaries— such as jamming and cyber attacks—and space debris are increasing. DOD is making changes to how it designs its space systems to increase the resilience and survivability of space capabilities. But it has been challenged in adopting new approaches, such as using commercial satellites to host payloads, and in prioritizing cybersecurity for all of its weapon systems. For hosted payloads, GAO recommended, and DOD concurred, that the department bolster and centralize collection and analysis of cost, technical, and lessons learned data. Implementing leadership changes . DOD is planning major changes to leadership for space. It recently proposed legislation to establish a United States Space Force—initially to be housed within the Department of the Air Force—that would, according to the President's Space Policy Directive, consolidate existing military space activities and minimize duplicative efforts across DOD. GAO found in July 2016 that changes are needed to reduce fragmentation that has negatively affected space programs for many years. But open questions remain about governance as new programs get underway and whether the changes themselves may result in further fragmentation. For example, it is unclear at this time how the new Space Development Agency will mesh with organizations currently involved in testing and acquiring new space technologies. Having the right resources and know-how . While there is increased attention on funding for space and building the Space Force, new programs can still face resource challenges. DOD has begun over 9 new space programs at a time when it is also seeking increased investments in ships, aircraft, and the nuclear triad, among other programs. Moreover, it is unclear whether DOD has a sufficient workforce to manage its new programs. GAO issued a report earlier this month that found DOD does not routinely monitor the size, mix, and location of its space acquisition workforce. Further, DOD has difficulty attracting and retaining candidates with the requisite technical expertise. GAO recommended that DOD collect and maintain data on its space acquisition workforce. DOD did not concur, but GAO maintains that DOD should have better information on such personnel, especially in light of its proposal for establishing the Space Force. GAO also found in March 2019 that key software-intensive space programs often did not effectively engage users to understand requirements and obtain feedback. GAO recommended, and DOD concurred, that the department ensure its guidance addressing software development provides specific, required direction on the timing, frequency, and documentation of user involvement and feedback. Past GAO reports have recommended that DOD adopt acquisition best practices to help ensure cost and schedule goals are met. DOD has generally agreed and taken some actions to address these recommendations.", "document_type": "gao"}
{"report": "Each year more than a dozen federal agencies and state and local jurisdictions were involved in the Fourth of July events on the National Mall. Some were involved in the overall planning, production, and execution of the events as a whole, while others played specific roles in only one event. Figure 1 shows the overall geographic layout of each event that occurred on the National Mall. For 2016 through 2019, the National Park Service (NPS) was responsible for the overall organization and execution of Fourth of July events on the National Mall, including the National Independence Day Parade; A Capitol Fourth Concert; Independence Day Fireworks Display; and in 2019, A Salute to America. Successful completion of these events depended on NPS, including United States Park Police (Park Police), coordinating with federal agencies and state and local jurisdictions to ensure that attendees could safely and securely attend each event. In addition, during the 2019 events, because of the addition of the Salute to America event and attendance by the President of the United States, additional federal agencies were involved with the planning, production, and execution of the events. Overall event security for the 2016 though 2019 events on the National Mall was coordinated among several federal agencies and state and local jurisdictions. Specifically, the following organizations provided security personnel and assets for the events overall: Park Police provided overall coordination with federal, state, and local law enforcement agencies and assisted with event security. State and local law enforcement agencies assisted with security, traffic, and crowd control. The District of Columbia Government (DC Government) provided a comprehensive command, control, and coordination system, in conjunction with federal partners, to ensure seamless event activities and the safety and security of all attendees. The DC Government tasked multiple offices in its organization to help with ensuring event security, including the Metropolitan Police Department, which deployed uniformed officers in areas surrounding the National Mall and provided traffic control and road closures. The National Guard deployed hundreds of personnel who provided security, movement of supplies, and crowd management at road intersections and metro stations. The Washington Metropolitan Area Transit Authority provided buses to barricade road closures for 2016 through 2018. The Federal Bureau of Investigation; Bureau of Alcohol, Tobacco, Firearms and Explosives; and Department of Energy deployed specialized law enforcement and security support units during all of the Fourth of July events on the National Mall. Given the large crowds and potential for high temperatures in July in Washington, D.C., it was important that organizers ensured that adequate medical resources were available to attendees and participants for the 2016 through 2019 events. This was accomplished by coordination between the following federal agencies and state and local jurisdictions: The Department of Health and Human Services (HHS) provided medical aid stations for attendees and participants and veterinarian services for NPS working animals at various locations on the National Mall. In 2019, HHS provided additional efforts, including a larger medical aid station at the end of the parade route to assist with heat casualties and a command and control team to coordinate and support HHS personnel on the National Mall. The Federal Emergency Management Agency participated in the public safety planning for the Fourth of July events on the National Mall in 2016 through 2018. Because of increased security levels in 2019, the agency coordinated the support of federal agencies and state and local jurisdictions, and deployed an emergency response team. The DC Government deployed numerous personnel from its offices, including Fire and Emergency Medical Services and the Department of Health to respond to health emergencies at the events. The Smithsonian Institution (SI) and the Federal Protective Service (FPS) assisted by providing their facilities as safe havens for citizens to seek shelter in the event of severe weather or other emergency. Each year, SI staffed its facilities near the National Mall with security protection officers, grounds cleanup crews, and emergency medical technician support as part of its assistance. In addition, FPS personnel staffed federal buildings near the National Mall and operated safe haven locations. For 2016 through 2019, the National Independence Day Parade ran along Constitution Avenue NW from 7th Street NW to 17th Street NW. A private entity produced the parade, and obtained a Public Gathering Permit from NPS. The entity managed the parade, its participants, and associated costs, with funding from nonfederal sponsors. NPS participated in the parade by coordinating with an additional private entity, which fully funded the creation and operation of a parade float for NPS. In addition, several military bands regularly participated in the parade. To ensure security for the parade, the Park Police requested assistance annually from the DC Government, National Guard, and FPS. The DC Government assisted by ensuring roads were closed to vehicle traffic on the parade route, the National Guard assisted by providing personnel and assets for road closure, and FPS provided personnel to ensure parade- route safety. For 2016 through 2019, the Capitol Fourth Concert was broadcast live from the West Lawn of the U.S. Capitol by the Public Broadcasting Service. As we have previously reported, a private entity in the District of Columbia has produced the annual concert for many years. The private entity received federal funding from NPS through a cooperative agreement that provided funding from NPS and the Department of the Army and sponsorships from other private entities. The private entity was responsible for producing the concert, including the selection of musical acts and coordination with DOD for military band attendees. Because of its jurisdiction over the Capitol grounds, the United States Capitol Police (Capitol Police) provided perimeter security and security screening of concert attendees, in coordination with multiple federal agencies and local jurisdictions. In addition, the Architect of the Capitol provided security barriers, fencing, ground protection, turf restoration, trash removal, and setup and teardown of the security elements on the Capitol grounds. For 2016 through 2019, annually, NPS entered into a contract with a private entity that was responsible for producing and executing the fireworks display. For 2016 through 2018, the fireworks were launched from the Reflecting Pool between the Lincoln Memorial and the Washington Monument, with that area being restricted to visitors for safety and security. However, in 2019, the launch site was relocated to the West Potomac Park. The Park Police provided security over the fireworks and coordinated with federal law enforcement agencies, which provided security and conducted a sweep of the fireworks launch area. Many attendees of the fireworks display viewed the show from their personal watercraft on the Potomac River. The United States Coast Guard (Coast Guard) provided security on Potomac River waterways to ensure attendee safety and to establish a secured perimeter for the launch site. In 2019, the Salute to America event was held for the first time in front of the Lincoln Memorial. The event included military band performances, a military display, a speech by the President of the United States, military aircraft flyovers, and a fireworks display. The planning of the event began after a meeting at the White House where the Secretaries of the Interior and Defense were tasked with event planning, production, and execution. The Executive Office of the President (EOP) coordinated the content of the event and contracted with a private entity, which was responsible for general event production. NPS and EOP entered into a reimbursable agreement whereby EOP, and the private entity with which it contracted, coordinated and produced the event, paid for with NPS appropriations. The EOP determined the guest list and distributed tickets for the event. According to EOP, it distributed tickets in a manner similar to that for other White House events. The Secretary of the Interior tasked NPS with permitting for the event, coordinating with the United States Secret Service (Secret Service) on security and with the DC Government on movement of DOD assets, relocating the existing contracted fireworks display from the Reflecting Pool to the West Potomac Park and coordinating the acceptance of an additional donated fireworks display. The DC Government had additional responsibilities in 2019 compared to prior years because of the vehicles that the Department of the Army provided for the Salute to America event. For example, DC Government personnel consulted with engineers to verify that affected roads, sewer pipes, and bridges could withstand the weight of bringing in the M2 Bradley Infantry Fighting Vehicles and conducted damage assessments after the event, during which no damage was identified. The DOD Joint Staff received orders from the Secretary of Defense directing United States Northern Command (NORTHCOM) to organize a flyover and provide support to the Salute to America event. NORTHCOM tasked the Coast Guard, Department of the Army, Department of the Navy, United States Marine Corps, and United States Air Force to ensure that various DOD assets were in attendance. The Department of the Army stood up a Joint Operations Center within its Joint Force Headquarters Branch, National Capital Region, to coordinate the various DOD assets involved. Prior to the President’s speech, several military bands performed for the audience in front of the Lincoln Memorial. Displayed on both sides of the performing bands were two M2 Bradley Infantry Fighting Vehicles provided by the Department of the Army. At designated times during the President’s speech, DOD aircraft participated in flyovers, including Air Force B-2 Stealth Bombers, Air Force One, F-22 Raptors, and F- Navy F/A 18 Hornet Blue Angels and F-35 Lightning IIs; Marine Corps MV22 Osprey helicopters and Marine One; Army AH-64 Apache and CH-47 Chinook helicopters; and Coast Guard H-65 Dolphin and H-60 Jayhawk helicopters and HC- 144 Medium Range Surveillance Aircraft. Following the President’s speech and associated flyovers, a fireworks display, donated by two private entities through a donation agreement with NPS, was presented from the Lincoln Memorial. Because the President, Vice President, and other government officials attended the event, the Secret Service had primary responsibility for security of the event and surrounding areas, in coordination with the Park Police. The Secret Service requested the assistance of the Transportation Security Administration, which provided security screening for the event. The Coast Guard provided additional support on the Potomac River during the event, because of the additional firework display, and requested the assistance of the United States Customs and Border Protection, which provided additional waterway security. The Federal Aviation Administration provided an air traffic controller that shut down the airspace around the National Mall and assisted with the Salute to America flyovers. Because of the additional fireworks, the Park Police provided additional security at the storage site of the fireworks and additional road closures. The Park Police coordinated with state and local law enforcement to provide escorts for the M2 Bradley Infantry Fighting Vehicles. According to estimates we obtained, federal agencies and state and local jurisdictions combined spent millions of dollars annually for the Fourth of July events on the National Mall during 2016 through 2019. Not all costs were tracked separately by the organizations for each of the Fourth of July events. Therefore, in order to develop a comprehensive estimate of the costs, we grouped costs into five categories, which include general event costs as well as costs for each of the specific events held on the National Mall. Table 1 summarizes the event costs we obtained, by year and event. In addition to costs that could be directly attributed to Fourth of July events on the National Mall, there were other costs incurred associated with federal personnel and assets that we did not capture as event costs because they would have been incurred regardless of whether the Fourth of July events had occurred. For example, costs such as salaries of federal civilian, military, and law enforcement personnel who worked during the events were not included in cost estimates because those salaried personnel would have been paid even if the Fourth of July events did not occur. We categorized costs attributable to more than one specific event, or to agencies that did not track costs by event, as general event costs. According to documents we reviewed and interviews with agency officials, more than $2 million was spent annually on general event costs. Table 2 contains general event cost by federal agency and state or local jurisdiction, and by year. General event costs consisted primarily of the personnel and supplies costs for HHS medical aid stations, Department of the Interior overtime, holiday pay and supply costs, and costs for DC Government personnel payroll. Specifically, federal and local law enforcement agencies provided security, screening of attendees, traffic control, road blockades, and escorts for participants at all the events. These agencies incurred salaries, overtime, and overtime with differential pay for civilian and law enforcement personnel. Other costs included providing personnel and supplies for fire and emergency medical services, crowd control, information and directions for attendees, cleaning of the grounds, and safe haven areas in case of an emergency. According to documents we reviewed and agency officials we interviewed, no federal agency recorded costs specifically attributable to the National Independence Day Parade for 2016 through 2019. A private entity produced the parade and managed its participants and associated costs, which was funded through nonfederal sponsors. The security during the event, provided by the National Guard and FPS, was not included in cost estimates because those salaried personnel would have been paid regardless of the parade. The majority of federal participants in the parade were local ceremonial military personnel, including military bands, marching platoons, color guards, Army Old Guard fife and drum corps, an Army anthem vocalist, and other ceremonial military participants who would have received their salaries and benefits on the Fourth of July even if the parade did not occur. According to documents we reviewed and agency officials we interviewed, the concert cost the federal government an estimated $4 million annually from 2016 through 2019. The concert takes place on the grounds of the Capitol and the costs are primarily for the contractor that plans and executes the concert. NPS provided minimal operations support to the entity that produced the concert but was responsible for funding the concert from its annual appropriations and with additional funding that the Department of the Army provides each year. Table 3 contains the concert cost by agency for 2016 through 2019. The Capitol Police is the primary law enforcement agency responsible for security and screening the attendees on the Capitol grounds. The Capitol Police estimated that it incurred several hundred thousand dollars annually in overtime and holiday pay costs that would not have been incurred had the concert not taken place. The Architect of the Capitol incurred other concert costs for its involvement. Additionally, DOD had bus rental costs for movement of ceremonial military personnel in 2017. Other costs not considered directly attributable to the concert included salaries and benefits of federal military participants. The salary costs for these personnel would have been incurred regardless of their participation in the Fourth of July events. Independence Day Fireworks Display event costs were estimated from $253,000 to $409,000 annually from 2016 through 2019 (see table 4). Each year, NPS contracted with a private entity, which produced and executed the fireworks display. The cost associated with this contract was the majority of the cost of the event. In addition to the contract costs, other fireworks display event costs included paying overtime for security personnel during the event, conducting security sweeps prior to the event, securing areas for storage of fireworks, closing roads, and performing cleanup after the fireworks. In addition, the Coast Guard had personnel travel costs in 2019. Other event costs not considered directly attributable to the fireworks display were for Coast Guard personnel and boats that patrolled a security perimeter around the event area. The Coast Guard stated that these boats and personnel would have been operating on the Fourth of July regardless of whether the fireworks display event occurred. The 2019 Salute to America Event cost an estimated $4.3 million, primarily related to the EOP contract with a private entity to plan and execute the event (see table 5). The cost of that contract was approximately $2.45 million and was funded with NPS appropriations through a Memorandum of Agreement with EOP. The movement of DOD ground assets to the Washington, D.C., area was also a cost for the event. Specifically, DOD used a contractor to transport vehicles and other military equipment to the event area at a cost of more than $1.12 million. The Secret Service had significant involvement with events on the National Mall, and specifically with the Salute to America event, because the President, Vice President, and other government officials attended. In order to prepare for and execute security, the Secret Service used numerous special agents from its Washington, D.C., Field Office, and incurred overtime pay and cost for materials. In addition, the airspace in the area was shut down for this event, which included a fireworks display. Various federal agencies incurred overtime costs for storing the donated fireworks and for keeping additional roads closed. While DOD and the Coast Guard provided military flyovers during the Salute to America event, most of the costs associated with the flyovers, such as crew salaries, fuel, and asset depreciation, were not attributable to the event. According to DOD and the Coast Guard, flying hours associated with the event were used to satisfy annual training requirements for their pilots. However, some travel costs were incurred for pilots and crew, which we included in the cost estimates in table 5. Finally, the estimate includes salaries for a small number of DOD civilian personnel who were paid holiday or overtime pay. We found that the majority of the agencies funded costs of the Fourth of July events with annual appropriations and did not receive any other funding. NPS used amounts from multiple appropriation accounts to pay for costs of the Fourth of July events for 2016 through 2019. For the Salute to America event in 2019, NPS used the Operation of the National Park System, Centennial Challenge, and Federal Lands Recreation Enhancement Act (FLREA) accounts to cover costs. NPS obligated $2.45 million of the FLREA amounts to pay for the private entity with which the EOP contracted to plan and execute the event. NPS also used the Centennial Challenge appropriation account to pay for certain costs attributable to the Salute to America event. NPS used the Operation of the National Park System account to fund the other Fourth of July events during 2016 through 2019. The Department of the Army transferred funds from its annual appropriations for fiscal years 2016 through 2019 to NPS to support the Capitol Fourth and Memorial Day concerts. The Army entered into an agreement with NPS each fiscal year and transferred a lump sum to NPS. NPS allocated the funding for the two concerts each fiscal year. The DC Government received an appropriation each fiscal year from the federal government for emergency planning and security costs in the District of Columbia that remains available until expended. This appropriation is for the costs of providing public safety at events related to the presence of the National Capital in the District of Columbia. According to DC Government officials, DC Government obligated the entire amount appropriated in fiscal year 2019 for the various events in the District of Columbia, including the Fourth of July events on the National Mall. DC Government officials stated that they did not request additional appropriations from Congress because they used funds from other appropriations to cover the cost of events exceeding the fiscal year 2019 appropriation. Park Police reimbursed local law enforcement outside of the District of Columbia for assistance with security, traffic, and crowd control, costs which were estimated from $85,000 to $132,000 annually from 2016 through 2019. These costs are included as regular operations. Finally, according to the officials at agencies we contacted, none of them delayed, deferred, or canceled any programs or activities as a result of resources being used for the Fourth of July events for 2016 through 2019. We provided a draft of this report to the EOP, DOD, Department of the Interior, Department of Homeland Security, Capitol Police, Architect of the Capitol, DC Government, SI, American Red Cross, Department of Energy, HHS, Washington Metropolitan Area Transit Authority, Department of Transportation, and Department of Justice for review and comment. The EOP, DOD, Department of the Interior, DC Government, Washington Metropolitan Area Transit Authority, and Department of Justice provided technical comments, which we incorporated as appropriate. The Department of Homeland Security, Capitol Police, Architect of the Capitol, SI, Department of Energy, HHS, and Department of Transportation informed us that they had no comments on the draft report and the American Red Cross did not provide comments. As agreed with your offices, unless you publically announce the contents of this report earlier, we plan no further distribution until 8 days from the report date. At that time, we will send copies to the Executive Office of the President, the Secretary of the Interior, the Acting Secretary of Homeland Security, the Secretary of Defense, the Secretary of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2989 or kociolekk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Our objectives were to describe for the Fourth of July events on the National Mall for 2016 through 2019 (1) the total costs that federal agencies and state and local jurisdictions are estimated to have incurred and (2) the appropriations that were used to pay for the estimated federal costs; the extent, if any, to which the federal government has reimbursed costs incurred by state and local jurisdictions; and the extent, if any, to which federal agencies delayed, deferred, or canceled other programs or activities as a result of resources being used for Fourth of July events. To accomplish these objectives, we obtained and reviewed documentation, such as financial data, contracts, and relevant agreements, from federal agencies and state and local jurisdictions that contributed resources to the events. Agencies we contacted were the Executive Office of the President, Department of Defense (DOD), Department of the Interior, Department of Homeland Security, United States Capitol Police, Architect of the Capitol, District of Columbia Government, Smithsonian Institution, American Red Cross, Department of Energy, Department of Health and Human Services, Washington Metropolitan Area Transit Authority, Department of Transportation, and Department of Justice. In addition, we reviewed cost documents and other agency records to gain an understanding of the assets, including financial, physical, and human capital that each agency devoted to the events. In addition, we interviewed officials about estimated costs; any reimbursed costs; and any delayed, deferred, or canceled programs or activities. The scope of our review consisted of estimated costs of the events and associated appropriations incurred by federal agencies and state and local jurisdictions. For the purposes of this engagement, we defined estimated costs as the costs that are directly traceable to the planning, production, and execution of the specific Fourth of July events on the National Mall. For example, we included the transportation costs of moving material, equipment, and supplies to the National Mall for Fourth of July events as well as personnel overtime and holiday pay expenses for federal employees. The costs also included contracts that various federal agencies awarded to private entities that were specifically attributable to the events. We excluded costs that are not directly attributable to the planning, production, and execution of the specific Fourth of July events on the National Mall, such as salary costs for civilian federal employees and military personnel who performed duties during the events that would have been incurred regardless of whether the events took place. Also, we excluded the costs to operate and maintain DOD aircraft that were used in the Salute to America event. According to DOD, the aircraft were existing DOD assets, and the flying hours associated with the event were used by DOD to meet annual pilot training requirements that were required regardless of the events on the Fourth of July. In addition, we excluded the cost associated with private entity parade participation and firework donations. Finally, cost estimates were provided by and attributable to each agency and department, and we did not independently verify the data during this audit. We conducted this performance audit from July 2019 through June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Kristen Kociolek, (202) 512-2989 or kociolekk@gao.gov In addition to the contact named above, Jonathan Meyer (Assistant Director), Kevin Scott (Auditor in Charge), and John Ledford made major contributions to this report. Other key contributors include Carl Barden, Anthony Clark, Marcia Carlsen, Elizabeth Erdmann, Pat Frey, Richard Geiger, Jason Kelly, Jason Kirwan, Quang Nguyen and Shahrzad Nikoo.", "summary": "The Second Continental Congress formally adopted the Declaration of Independence on July 4, 1776. Since that day, Americans have celebrated this holiday through events held in towns and cities across the country. In the nation's capital, Washington, D.C., visitors have celebrated on the National Mall by attending federally sponsored events such as the National Independence Day Parade; A Capitol Fourth Concert; Independence Day Fireworks Display; and in 2019, A Salute to America. GAO was asked to review the impacts and estimated costs associated with the Fourth of July events on the National Mall. Specifically, this report describes the following for the Fourth of July events on the National Mall for 2016 through 2019: (1) the total costs federal agencies and state and local jurisdictions are estimated to have incurred and (2) the appropriations that were used to pay for the estimated federal costs; the extent, if any, to which the federal government reimbursed costs incurred by state and local jurisdictions; and the extent, if any, to which federal agencies delayed, deferred, or canceled other programs or activities as a result of resources being used for Fourth of July events. To perform this work, GAO reviewed documentation and interviewed personnel from federal agencies and state and local jurisdictions about their estimated costs and resources used for the events. From 2016 through 2019, hundreds of personnel from numerous federal agencies, state and local jurisdictions, and private entities planned, produced, and executed events on the National Mall that celebrated Independence Day of the United States. The National Park Service (NPS) was responsible for the overall execution of Fourth of July events on the National Mall. In addition, various federal agencies—including the Department of Homeland Security, United States Capitol Police, United States Coast Guard, and Department of Justice—helped to ensure safety. Beyond the federal effort, the District of Columbia Government (DC Government) and local law enforcement played a role in the overall events. Further, given the crowds and potential for high temperatures in July in Washington, D.C., it was important that organizers—including the Department of Health and Human Services—ensured adequate medical resources were available to attendees and participants. The estimated costs for the events held in 2016, 2017, and 2018 ranged from $6 million to $7 million annually, and included contract costs with private entities tasked with producing and executing the concert and fireworks. They also included the costs for overtime and holiday pay for federal employees working at the events. In 2019, with the addition of the Salute to America event, the Department of Defense (DOD) and Executive Office of the President undertook additional efforts. Estimated costs for the 2019 events on the National Mall increased to more than $13 million. This increase was attributable to the cost for DOD to transport several vehicles to the National Mall, the production and execution of the Salute to America event, and the additional security involved because the President attended the event. In addition, there were costs not directly attributable to the events, including salaries of some federal employees who performed duties during the events, as well as costs for fuel and depreciation on DOD assets. These costs were classified as not directly attributable to the Fourth of July events because they would have been incurred regardless of whether the events occurred. For example, according to DOD, the flight time related to the military flyovers for the Salute to America event were required training hours that pilots must complete annually, and therefore the related expenses, such as pilot salaries and fuel costs, were not included in event cost estimates. Finally, federal agencies and the DC Government primarily used annual federal appropriations to pay for the event costs. The DC Government received an appropriation each year to provide for public safety at certain events within the District of Columbia. According to DC Government officials, DC Government obligated the entire amount appropriated in fiscal year 2019 for the various events in the District of Columbia, including the Fourth of July events on the National Mall. DC Government officials stated that they did not request additional appropriations from Congress because they used funds from other appropriations to cover the cost of events exceeding the fiscal year 2019 appropriation. Agency officials did not identify any federal activities that were delayed, deferred, or canceled because of the resources used for the Fourth of July events on the National Mall in 2016, 2017, 2018, and 2019.", "document_type": "gao"}
{"report": "While the core mission of protecting federal facilities has remained constant as FPS has moved from one agency to another, its responsibilities have changed. In the 1970s, GSA created FPS as part of its Public Buildings Service (PBS). While in GSA’s PBS, FPS was responsible for protecting GSA’s held or leased facilities, providing both physical security and law enforcement services. To protect buildings, FPS officers developed physical security risk assessments, installed security equipment, and oversaw contract guard services. As a part of its law enforcement services, among other duties, FPS officers enforced laws and regulations aimed at protecting federal facilities and the persons in such facilities and conducted criminal investigations. Following the attacks on September 11, 2001, the Homeland Security Act of 2002 was enacted. It created DHS and moved FPS from GSA to the new department, effective in March 2003. Within DHS, FPS was placed in U. S. Immigration and Customs Enforcement (ICE), where its responsibilities grew beyond solely protecting GSA buildings to include homeland security activities such as implementing homeland security directives and providing law-enforcement, security, and emergency- response services during natural disasters and special events. In 2009, DHS proposed transferring FPS from ICE to NPPD. In explaining this transfer in DHS’s fiscal year 2010 budget justification to Congress, DHS stated that having FPS and NPPD’s Office of Infrastructure Protection in the same organization would further solidify NPPD as DHS’s lead for critical infrastructure protection. FPS was placed in NPPD and continued to lead physical security and law enforcement services at GSA- held or GSA-leased facilities and continued its efforts in homeland security activities. In November 2018, legislation was enacted that reorganized NPPD to an organization that had a greater statutory focus on managing cyber risks and authorized the Secretary of Homeland Security to determine the appropriate placement for FPS within DHS and begin transfer of FPS to that entity. Throughout FPS’s organizational placements in DHS, we have reported on persistent challenges it faced in meeting its mission to protect facilities. In 2011, we reported on FPS’s challenges in transferring mission support functions from ICE to NPPD. While FPS was in NPPD, we reported on FPS’s challenges related to managing and overseeing contract guards and collaborating with GSA and the United States Marshals Service (Marshals) on facility security. We made recommendations to help address these challenges and FPS has made progress on some of these recommendations. For example, in September 2018, FPS and GSA established a formal agreement on roles and responsibilities related to facility protection, as we recommended. However, in our January 2019 report, we identified challenges related to other aspects of overseeing contract guards and collaboration with other agencies on physical security that had persisted. As of June of 2019, FPS continues to work on establishing a contract guard-management system. However, FPS is unable to assess its guards’ capabilities across its portfolio because the system is not fully implemented nor does it interact with its training system. As of 2019, federal physical security continues to be part of our federal real-property management’s high-risk area. In 2002, we reported on organizational and accountability criteria for establishing DHS. From this prior work, we identified key criteria that are relevant to assessing potential placement options for FPS, as shown in table 1. For our January 2019 report, we applied these key criteria for evaluating organizational placement to eight agencies that could be potential placement options for FPS. We found that none of the selected agencies met all the organizational placement criteria; thus, any of the organizational placement options could result in both benefits and trade- offs. In instances where placing FPS within DHS met our criteria (that is, instances where DHS was similar to FPS), FPS could experience benefits. In those instances where the criteria were not met, we reported it would be incumbent upon any agency to consider and address any potential trade-offs in order to ensure the decision was successful. We reviewed FPS as a “standalone” entity reporting directly to the Deputy Secretary of DHS and found this placement option met several key criteria. Table 2 below summarizes our analysis. For the first four criteria—(1) mission, goals, and objectives; (2) responsibilities; (3) organizational culture; and (4) information sharing and coordination—we determined that DHS met the criteria if the agency or its subcomponents had any similarities to FPS. For the last criterion— mission support—we determined that DHS met the criterion if the agency or its subcomponents had similarities to FPS or could provide FPS needed mission support. Mission, Goals, and Objectives. In January 2019, we reported that FPS’s mission focused on the protection of federal facilities and the people working in and visiting those facilities. DHS was similar to FPS in that its mission statement and goals as stated in its strategic plan include an explicit focus on the protection of infrastructure or specific facilities. Our prior work found that placing an agency into an organization that has a similar mission might help ensure that the agency’s mission receives adequate funding, attention, visibility, and support. Our January 2019 work reported that one of DHS’s goals—as noted in its strategic plan covering fiscal years 2014 to 2018—was to reduce risk to the nation’s critical infrastructure. DHS and FPS share objectives that focus on mitigating risks and responding to incidents. Responsibilities. In January 2019, we reported that FPS has facility- protection and physical-security responsibilities and law-enforcement, and contract-guard oversight responsibilities. DHS was similar to FPS as it had responsibilities for physical security and performed law enforcement functions. As a part of its physical security activities, FPS conducted facility security assessments, identified countermeasures (e.g., equipment and contract guards) best suited to secure a facility, and oversaw contract guards. As a part of its law enforcement activities, FPS proactively patrolled facilities, responded to incidents, and conducted criminal investigations. FPS also provided additional operational law enforcement support, at the direction of the Secretary of Homeland Security, to address emerging threats and homeland security incidents. One of FPS’s most critical activities was overseeing about 13,500 contract guards who were posted at federal facilities and were responsible for controlling access to facilities, responding to emergency situations involving facility safety and security, and performing other duties. FPS was responsible for ensuring, among other things, that these guards are performing their assigned duties and have the necessary training and certifications. DHS, however, only used a limited number of contract guards and therefore had less responsibility. At the time of our review, DHS officials told us they procured about 130 guards. Organizational Culture. In January 2019, we reported that while there are many areas relevant to organizational culture, law enforcement was a key aspect of FPS’s organizational culture, according to officials we interviewed from an association of security companies and a former, high- ranking official in NPPD. DHS had a similar culture in that it was a law enforcement agency. Information Sharing and Coordination. In January 2019, we reported that Component Intelligence Programs (CIP) were organizations in DHS that collected, gathered, processed, analyzed, produced, or disseminated information related to national homeland security. In 2016, DHS designated a division within FPS as a CIP, a move that allowed FPS more access to information on threats other DHS agencies have identified and actions they plan to take. While DHS, like FPS, had access to and could share information related to national homeland security, DHS did not have joint responsibility for coordinating facility protection with FPS. Rather, FPS shared this responsibility with GSA, and these two agencies and Marshals had joint responsibility for protecting courthouses. FPS has faced challenges with coordinating with these agencies in the past. For example, in September 2011, we reported that FPS, Marshals, and other agencies involved in protecting courthouses (i.e., GSA and the Administrative Office of the U.S. Courts) faced challenges related to coordination, such as in the implementation of roles and responsibilities and the use or participation in existing collaboration mechanisms. Mission Support. In January 2019, we reported that mission support was comprised of financial management, human capital, information technology systems for financial management, and law enforcement training. FPS owned and used many of the key operational and business- related information technology (IT) systems and applications it needs to carry out its mission. However, FPS received some mission support services from other agencies in DHS, such as human capital and some aspects of information technology. We found that if FPS changed its organizational placement it would need mission support in these areas. For example, FPS did not have delegated examining authority to allow it to fill competitive civil service jobs and relied on NPPD to provide this service. DHS had the authority to fill competitive service jobs that could support FPS needs. Further, FPS used a financial management IT system owned by ICE. DHS could provide FPS access to financial management systems that can support FPS. Finally, FPS offered its own training courses and would still need access to DHS’s Federal Law Enforcement Training Centers. In our January 2019 report, we did not assess FPS as a placement within DHS’s Management Directorate. Further, we recommended DHS (1) identify the specific goals of a change in FPS’s placement—that is, what DHS expects to achieve by moving FPS to another agency, and (2) fully evaluate placement options for FPS based on what DHS expects to achieve by changing FPS’s placement, an assessment of FPS’s current placement, and other best practices such as an analysis of alternatives assessing the benefits and trade-offs. DHS agreed with our recommendations. In May 2019, FPS officials told us that the Acting Secretary’s decision to place FPS within the Management Directorate was based upon an assessment of placement options within DHS using criteria and analyzing the trade-offs. GAO has not yet received DHS’s assessment of placement options. We will assess the actions DHS has taken in response to our recommendations when we receive DHS’s assessment. Our prior work offers valuable insights for agencies to consider when evaluating or implementing a reorganization or transformation, and can provide insights for making any transition regarding FPS. These include considering (1) key questions for consolidations and (2) leading practices when implementing an organizational change. Two sets of considerations for organizational transformations provide insights for making any FPS organizational placement. First, in May 2012, we reported on key questions for agency officials to consider when evaluating and implementing an organizational change that involves consolidation. Table 3 provides a summary of these key questions. Answering these questions would help provide FPS with assurance that important aspects of effective organizational change are addressed. Second, we reported in July 2003 on key practices and implementation steps for mergers and organizational transformations. The practices we noted are intended to help agencies transform their cultures so that they can be more results oriented, customer focused, and collaborative in nature (see table 4). In summary, the questions and practices for organizational change that we previously identified could provide insights to DHS and FPS for any transition. . Madam Chairwoman Torres Small, Ranking Member Crenshaw, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff has any questions concerning this testimony, please contact Lori Rectanus at 202-512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contacts named above: Amelia Bates Shachoy (Assistant Director); Roshni Davé; George Depaoli (Analyst-in-Charge); Geoffrey Hamilton; Kelly Rubin; Sarah Veale; and Amelia Michelle Weathers made key contributions to the testimony. Other staff who made contributions to the reports cited in the testimony are identified in the source products. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "FPS conducts physical security and law enforcement activities for about 9,000 federal facilities and the millions of employees or visitors who work in or visit these facilities. Legislation enacted in November 2018 required DHS to determine the appropriate placement for FPS. The legislation also gave the Secretary of DHS authority to move FPS within DHS. In May 2019, DHS announced its decision to place FPS within the DHS Management Directorate as a direct report to the Under Secretary for Management. GAO has reported that FPS faces persistent challenges in meeting its mission to protect facilities, and, as of 2019, physical security continues to be part of GAO's federal real property management high-risk area. For example, FPS has not yet fully implemented its guard management system. Thus, FPS is unable to obtain information to assess its guards' capability to address physical security risks across its portfolio. This statement describes considerations for FPS's placement in DHS's Management Directorate based upon five key organizational placement criteria GAO identified, as well as steps to transition FPS based upon GAO's prior work on organizational change. This testimony is based on reports GAO issued from 2002 through 2019, particularly, GAO's January 2019 report on FPS's organizational placement. Detailed information on the scope and methodology for this work can be found in these published products, cited throughout this testimony. In its January 2019 report, GAO identified five key criteria relevant for evaluating placement options for the Federal Protective Service (FPS) within the Department of Homeland Security's (DHS) or other federal agencies. (See table.) Placing FPS, in the DHS Management Directorate was not an option GAO assessed in its January 2019 report. However, GAO did assess the option of making FPS a “standalone” entity reporting directly to the Deputy Secretary of DHS. GAO found that this placement met the first criteria ( mission, goals, and objectives ) and the third criteria ( organizational culture ) but did not completely meet the other criteria. For example, FPS had joint responsibility for coordinating facility protection with other federal agencies. DHS did not have joint responsibility for coordinating facility protection with FPS. GAO recommended DHS fully evaluate placement options for FPS. DHS concurred, and officials stated they conducted an assessment. GAO has not yet received DHS's assessment of placement options. GAO's prior work on implementing an organizational change provides valuable insights for making any transition regarding FPS. These insights include key questions to consider such as: “What are the goals of the consolidation?” “How have stakeholders been involved in the decision-making?” In addition, GAO has identified key practices for organizational transformation, practices that include ensuring that top leadership drives the transformation and establishing a communication strategy to create shared expectations, among others. These questions and practices could provide insights to DHS and FPS as they implement FPS's new placement.", "document_type": "gao"}
{"report": "In connection with the Strengthening DHS Management Functions high- risk area, we monitor DHS’s progress in the area of employee morale and engagement. In 2010, we identified, and DHS agreed, that achieving 30 specific outcomes would be critical to addressing the challenges within the department’s high-risk management areas. These 30 outcomes are the criteria by which we gauge DHS’s demonstrated progress. We rate each outcome on a scale of not-initiated, initiated, partially addressed, mostly addressed, or fully addressed. Several of these outcome criteria relate to human capital actions needed to improve employee morale. Specifically, we monitor DHS’s progress to: seek employees’ input on a periodic basis and demonstrate measurable progress in implementing strategies to adjust human capital approaches; base hiring decisions, management selections, promotions, and performance evaluations on human capital competencies and individual performance; enhance information technology security through improved workforce planning of the DHS cybersecurity workforce; and improve DHS’s FEVS scores related to employee engagement. Since we began monitoring DHS’s progress on these outcomes, DHS has worked to strengthen employee engagement through several efforts both at DHS headquarters and within its component agencies. In this statement, we discuss nine recommendations related to DHS employee engagement and workforce planning, eight of which have been implemented by the department. Within DHS, the Office of the Chief Human Capital Officer (OCHCO) is responsible for implementing policies and programs to recruit, hire, train, and retain DHS’s workforce. As the department-wide unit responsible for human capital issues within DHS, OCHCO also provides guidance and oversight related to morale issues to the DHS components. Seeking employees’ input and demonstrating progress to adjust human capital approaches. DHS, OCHCO, and the components have taken action to use employees’ input from the FEVS to inform and implement initiatives targeted at improving employee engagement. For example, in 2017 and 2018 DHS implemented our two recommendations for OCHCO and DHS components to establish metrics of success within their action plans for addressing employee satisfaction problems and to better use these plans to examine the root causes of morale challenges. DHS components have continued to develop these employee engagement action plans and several components report implementing initiatives to enhance employee engagement. For example, the U.S. Secret Service’s action plan details a sponsorship program for all newly hired and recently relocated employees. In addition, one division of U.S. Immigration and Customs Enforcement (ICE) used FEVS survey data to identify a need for increased engagement between employees and component leadership. ICE’s employee action plan includes goals with milestones, timelines, and metrics to improve this engagement through efforts such as leadership town halls and leadership site visits. At the headquarters level, DHS and OCHCO have also established employee engagement initiatives across the department. For example, DHS established initiatives for employees and their families that aim to increase awareness and access to support programs, benefits, and resources. Through another initiative—Human Resources (H.R.) Academy—DHS provides education, training, and career development opportunities to human resource professionals within the department. DHS uses an Employee Engagement Steering Committee to guide and monitor implementation of these DHS-wide employee engagement initiatives. As a result of these steps, among other actions, we have considered this human capital outcome area fully addressed since 2018. Basing hiring decisions and promotions on competencies and performance. OCHCO has conducted audits to better ensure components are basing hiring decisions and promotions on human capital competencies and individual performance and we have considered this outcome fully addressed since 2017. Our past work has highlighted the importance of selecting candidates based on qualifications, as doing otherwise can negatively affect morale. Working to ensure that components’ human capital decisions are based on performance and established competencies helps create a connection between individual performance and the agency’s success. Enhancing information technology security through improved workforce planning for cybersecurity positions. In February 2018, we made six recommendations to DHS to take steps to identify its position and critical skill requirements among its cybersecurity workforce. Since then, DHS has implemented all six recommendations. For example, in fiscal year 2019, regarding its cybersecurity position identification and coding efforts, we verified that DHS had identified individuals in each component who are responsible for leading those efforts, developed procedures, established a process to review each component’s procedures, and developed plans for reporting critical needs. However, DHS has not yet implemented a recommendation we made in March 2019 to review and correct its coding of cybersecurity positions and assess the accuracy of position descriptions. Specifically, we stated that DHS had not correctly categorized its information technology/cybersecurity/cyber-related positions. We noted that having inaccurate information about the type of work performed by 28 percent of the department’s information technology/cybersecurity/cyber-related positions is a significant impediment to effectively examining the department’s cybersecurity workforce, identifying work roles of critical need, and improving workforce planning. DHS officials stated that they plan to implement this recommendation by March 2020. As a result, this outcome remains mostly addressed. Until DHS accurately categorizes its positions, its ability to effectively identify critical staffing needs will be impaired. Improving FEVS scores on employee engagement. Since our last High-Risk report in March 2019, DHS has demonstrated additional progress in its employee engagement scores, as measured by the FEVS Employee Engagement Index (EEI). The EEI is one of three indices OPM calculates to synthesize FEVS data. The EEI measures conditions that lead to engaged employees and is comprised of three sub-indices related to employees’ views on leadership, supervisors, and intrinsic work experience. As a result of continued improvement on DHS’s EEI score, we have moved this outcome rating from partially addressed to mostly addressed based on DHS’s 2019 score. As shown in figure 1, DHS increased its EEI score across 4 consecutive years, from a low of 53 percent in 2015 to 62 percent in 2019. In particular, DHS improved its score by two points between 2018 and 2019 while the government average remained constant over the same period. With its 2019 score, DHS also regained the ground that it lost during an 8-point drop between 2010 and 2015. While DHS has made progress in improving its scores including moving toward the government average, it remains below the government average on the EEI and on other measures of employee morale. For example, in 2019 DHS remained six points below the government-wide average for the EEI. In addition to the EEI and other indices OPM calculates, the Partnership for Public Service uses FEVS data to produce an index of the Best Places to Work in the Federal Government®. The Partnership for Public Service’s analysis of FEVS data indicates low levels of employee satisfaction and commitment for DHS employees relative to other large federal agencies. In 2019, the Partnership for Public Service ranked DHS 17th out of 17 large federal agencies for employee satisfaction and commitment. Across the department, employee satisfaction scores vary by component. Some DHS components have EEI scores above the government average and rank highly on the Partnership for Public Service’s index. For example, the U.S. Coast Guard and U.S. Citizenship and Immigration Services have EEI scores of 76 and 74, respectively, and rank 85th and 90th, respectively, out of 420 subcomponent agencies on the Partnership for Public Service’s index. Further, some DHS component agencies have improved their scores in recent years. The U.S. Secret Service raised its EEI score 7 points between 2018 and 2019, and it moved from the last place among all subcomponent agencies on the Partnership for Public Service’s Ranking in 2016 to 360th out of 420 subcomponent agencies in 2019. However, other DHS component agencies continue to rank among the lowest across the federal government in the Partnership for Public Service rankings of employee satisfaction and commitment. For example, in 2019 out of 420 subcomponent agencies across the federal government, the DHS Countering Weapons of Mass Destruction office ranked 420th, the DHS Office of Intelligence and Analysis ranked 406th, and the Transportation Security Administration ranked 398th for employee satisfaction and commitment. As a result, continuing to increase employee engagement and morale remains important to strengthening DHS’s management functions and ability to implement its missions. DHS employee concerns about senior leadership, among other things, is one area that negatively affects DHS’s overall employee morale scores. In 2015, we identified effective management practices agencies can use to improve employee engagement across the government. One of these practices is the direct involvement of top leadership in organizational improvement efforts. When top leadership clearly and personally leads organizational improvement efforts, it provides an identifiable source for employees to rally around and helps processes stay on course. A DHS analysis of its 2012 FEVS scores indicated DHS low morale issues may persist because of employee concerns about senior leadership and supervisors, among other things, such as whether their talents were being well-used. Within the 2019 FEVS results for both DHS and government wide, leadership remains the lowest of the three sub-indices of the EEI. In addition, for several years DHS components have identified several root causes of engagement scores. For example, in 2019, the Transportation Security Administration identified the performance of managers, time constraints and understaffing, and lack of manager and leadership accountability for change as root causes of the component’s engagement scores in recent years. Another component, U.S. Citizenship and Immigration Services, identified in 2019 that the areas of leadership performance, accountability, transparency, and training and development opportunities were 2018 engagement score root causes. We have previously reported that DHS’s top leadership, including the Secretary and Deputy Secretary, have demonstrated commitment and support for addressing the department’s management challenges. Continuing to identify and address the root causes of employee engagement scores and addressing the human capital management challenges we have identified in relation to the DHS management high- risk area could help DHS maintain progress in improving employee morale. Implementing our recommendation to review and correct DHS coding of cybersecurity positions and assess the accuracy of position descriptions will assist the department in identifying critical staffing needs. In addition, as we reported in May 2019, vacancies in top leadership positions could pose a challenge to addressing aspects of DHS’s high- risk area, such as employee morale. There are currently acting officials serving in ten positions requiring Senate confirmation. Filling vacancies—including top DHS leadership positions and the heads of operational components—with confirmed appointees, as applicable, could help ensure continued leadership commitment across DHS’s mission areas. We will continue to monitor DHS’s progress in strengthening management functions, and may identify additional actions DHS leadership could take to improve employee morale and engagement. In conclusion, DHS has made notable progress in the area of human capital management, specifically in improving employee engagement and morale, but still falls behind other federal agencies. It is essential for DHS to continue improving employee morale and engagement given their impact on agency performance and the importance of DHS’s missions. Continued senior leadership commitment to employee engagement efforts and filling critical vacancies could assist DHS in these efforts. Madam Chairwoman Torres Small, Ranking Member Crenshaw, and Members of the Subcommittee, this completes my prepared statement, I would be happy to respond to any questions that you may have at this time. If you or your staff have any questions concerning this statement, please contact Christopher P. Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement were Alana Finley (Assistant Director), Mara McMillen (Analyst-in-Charge), Nina Daoud, Michele Fejfar, Andrew Howard, and Tom Lombardi. In addition, Colette Alexander, Richard Cederholm, Ben Crossley, Eric Essig, Laura Ann Holland, Tammi Kalugdan, Neelaxi Lakhmani, Shannin O’Neill, Kevin Reeves, John Sawyer, and Julia Vieweg made contributions to this statement. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "DHS is the third-largest cabinet-level department in the federal government, employing more than 240,000 staff in a broad range of jobs, including countering terrorism and homeland security threats, providing aviation and border security, emergency response, cybersecurity, and critical infrastructure protection. Since it began operations in 2003, DHS has faced challenges with low employee morale and engagement. Federal surveys have consistently found that DHS employees are less satisfied with their jobs compared to the average federal employee. For example, DHS's scores on the FEVS and the Partnership for Public Service's rankings of the Best Places to Work in the Federal Government® are consistently among the lowest for similarly-sized federal agencies. This statement addresses our past and ongoing work monitoring human capital management and employee morale at DHS and select work on employee engagement across the government. This statement is based on products GAO issued from September 2012 through May 2019 as well as GAO's ongoing efforts to monitor employee morale at DHS as part of GAO's high-risk work. For these products, GAO analyzed DHS strategies and other documents related to DHS's efforts to address its high-risk areas, interviewed DHS officials, conducted analyses of FEVS data, and interviewed officials from other federal agencies that achieved high employee engagement scores, among other things. GAO provided a copy of new information in this statement to DHS for review. DHS confirmed the accuracy of this information. The Department of Homeland Security (DHS) has undertaken initiatives to strengthen employee engagement through efforts at its component agencies and across the department. For example, at the headquarters level, DHS has instituted initiatives to improve awareness and access to support programs, benefits, and resources for DHS employees and their families. In 2019, DHS improved its employee engagement scores, as measured by the Office of Personnel Management's Federal Employee Viewpoint Survey (FEVS)—a tool that measures employees' perceptions of whether and to what extent conditions characterizing successful organizations are present in their agency. As shown below, DHS increased its scores on a measure of employee engagement, the Employee Engagement Index (EEI), across 4 consecutive years, from a low of 53 percent in 2015 to 62 percent in 2019. While DHS has made progress in improving its scores, in 2019 it remained six points below the government-wide average for the EEI. For several years, DHS and its component agencies have identified root causes for their engagement scores including concerns about leadership accountability and understaffing, among others. This statement discusses nine recommendations related to DHS employee engagement and workforce planning. DHS implemented all but one of these recommendations—to review and correct its coding of cybersecurity positions and assess the accuracy of position descriptions. Finally, filling vacancies could help ensure continued leadership commitment across DHS's mission areas.", "document_type": "gao"}
{"report": "Promoting the rule of law abroad has been a U.S. government priority for decades. As early as 1985, rule of law was added to the Foreign Assistance Act of 1961 as a policy priority. Prior to the 1990s, rule of law assistance was primarily focused on activities in Latin America and the Caribbean. With the end of the Cold War and subsequent collapse of the Soviet Union, the U.S. government invested resources to support rule of law and justice sector reform in Central and Eastern Europe. Following the September 11, 2001 terrorist attacks, Afghanistan became a primary recipient country of U.S. rule of law assistance. The United States continues to support rule of law activities around the world. Department of State’s Standardized Definition of Rule of Law Rule of law is a principle of governance under which all persons, institutions and entities, public and private, including the State itself, are accountable to laws that are publicly promulgated, independently adjudicated, equally applied and enforced, and consistent with international treaties and customary law. Rule of law is demonstrated by adherence to the principles of publicly accepted legitimacy of the law, institutions, and process; checks and balances on structures of power; supremacy of the law; equality before the law; accountability to the law; fairness; effective application of the law; equitable access to justice; participation in decision-making; legal certainty; avoidance of arbitrariness; and procedural and legal transparency. Activities include support for strengthening of judicial systems including court administration, management, and operations; judicial proceedings; constitutional and legal reform efforts; judicial independence; access to justice; and legal education and associations. Millennium Challenge Corporation (MCC), also provide assistance that can be related to improving the rule of law. At each agency, several offices participate in rule of law assistance. The Bureau for International Narcotics and Law Enforcement (INL) is the lead office for rule of law within State. According to INL, it has three main objectives related to rule of law assistance: (1) effectiveness, (2) accountability, and (3) respect for fundamental rights and freedoms. One principle that also guides INL’s rule of law assistance is effectively coordinating assistance with other donors, other bureaus and offices within State, and interagency partners, according to INL. According to USAID, USAID designs, oversees and manages rule of law programming primarily through country-level missions, which ensures programming is tailored to local context. These programs are, in turn, supported by Washington-based regional and pillar bureaus. As the home base for USAID’s Democracy, Human Rights and Governance (DRG) programs, the DRG Center (1) leads USAID efforts to achieve self-reliant, citizen-responsive, democratic societies that respect human dignity, rights and the rule of law; (2) provides proactive and responsive technical support to missions and bureaus on core DRG sectors, including rule of law; and (3) conducts assessment, design, and evaluation of related DRG programs around the world to support more effective, systemic, cost- efficient and sustainable development. DOJ does not directly fund rule of law assistance, but its Office of Overseas Prosecutorial Development, Assistance and Training (OPDAT) and the International Criminal Investigative Training Assistance Program (ICITAP) implement activities funded by agencies such as State, USAID, and DOD through interagency agreements. From fiscal years 2014 through 2018, State and USAID allocated $2.7 billion for rule of law assistance, with annual allocations increasing from $496 million in fiscal year 2014 to $551 million in fiscal year 2018, or 11 percent. Within this time period, allocations fluctuated. Specifically, allocations increased by 20 percent from fiscal years 2014 through 2016, and subsequently decreased by 7 percent from fiscal years 2016 through 2018. According to the Congressional Research Service, the fluctuations in rule of law funding mirrored the fluctuations in foreign operations appropriations, which also increased by 11 percent from fiscal years 2014 through 2018. State allocated more than $2 billion from fiscal years 2014 through 2018 in that time period and USAID allocated over $700 million. See figure 1 for annual allocations by State and USAID for rule of law assistance from fiscal years 2014 through 2018. In fiscal year 2018, activities promoting justice systems and institutions received more allocated funding than all other types of rule of law assistance combined. Justice Systems and Institutions funds were allocated toward activities such as improving the systems, capacity, and sustainability of the civil and criminal justice sectors by harmonizing policies, fostering public / private partnerships, providing training programs, and strengthening administrative and operational systems. Recipients of this assistance can include police forces, prosecutors, judges, public defenders, bar associations, and training institutions. See figure 2 for rule of law allocations by program element, and appendix II for more information on how State and USAID track rule of law funding. DOJ’s ICITAP and OPDAT track funding in obligations, not allocations. According to DOJ, State and USAID used DOJ to implement certain rule of law programs, obligating $691 million from fiscal year 2014 through July 2019. Of this amount, $327.6 million went to ICITAP and $363.5 million went to OPDAT. From fiscal years 2014 through 2018, State and USAID allocated funds for rule of law assistance to 20 regional or programmatic operating units in Washington, D.C., and 72 field-based operating units, primarily bilaterally to country missions. The top four recipients of rule of law allocations were State’s Western Hemisphere Region and bilateral programs in Afghanistan, Mexico, and Colombia. These four recipients were allocated $1.7 billion of $2.7 billion, or 63 percent of the total rule of law allocations from fiscal years 2014 through 2018. The top three bilateral recipients, Afghanistan, Mexico, and Colombia, received 40 percent of rule of law assistance during this time period, which exceeded the total allocation to all 69 other bilateral recipients combined. Figure 3 shows worldwide distribution of bilateral rule of law assistance allocations from fiscal years 2014 through 2018. See appendix III for a complete list of countries and regional programs listed by funds received. State and USAID participate in an annual foreign assistance budget process, managed by State’s F bureau, which determines the allocation of foreign assistance funds for a variety of projects for all recipient countries and programs worldwide. According to agency officials, allocations of rule of law assistance are determined during this process. Agencies develop budget requests on an annual basis, usually starting this process 2 years before the start of any particular fiscal year. According to agency officials, the requests begin with the overseas missions providing annual reports and performance plans to State and USAID headquarters. They said that, during this process, each mission determines its need for financial resources related to foreign assistance, including rule of law assistance. Officials also hold interagency roundtable discussions regarding various aspects of foreign assistance. According to State officials, State chairs a roundtable on rule of law assistance that includes other interagency partners such as DOJ, DOD, MCC, and others. According to these officials, this roundtable allows the relevant agencies and bureaus to make decisions related to the amount of rule of law assistance funding that goes to specific regions and countries and align the funding with broader foreign assistance goals. Each agency also compiles and analyzes these annual reports and performance plans and provides initial budget requests to the Office of Management and Budget (OMB) in September. From September to November, OMB reviews each agency’s budget request submission and conducts analysis on how the budget requests align with the overall federal budget. After OMB conducts its review, it communicates to each agency the level of funding it can request from Congress. The President usually submits the overall federal budget request to Congress on the first Monday in February. As part of this request, each agency, including State and USAID, provides a more detailed Congressional Budget Justification that explains the need for specific funding levels to the relevant congressional subcommittees. Once the House of Representatives and the Senate agree on the language of the bills, including the levels of funding, and pass the State and Foreign Operations appropriations bill, the President can then sign it into law. Once the President signs the State and Foreign Operations appropriations bill into law, OMB apportions the amount of funds that State, USAID, and other agencies may use. Agencies then allocate and obligate these funds for certain programs. In the case of rule of law assistance, these obligated funds are often used to engage in partnership with implementing partners overseas through contracts, grants, or cooperative agreements, according to agency officials. Improving the rule of law in partner countries overseas is identified as an important objective in several strategic documents including the 2017 National Security Strategy, the 2018-2022 State-USAID Joint Strategic Plan, the 2018-2022 DOJ Strategic Plan, and bureau-specific plans. Each of these strategic documents is linked to U.S. national security goals and discuss U.S. agencies’ roles in improving the rule of law in partner countries. See figure 4. 2017 National Security Strategy. This strategy identifies the rule of law as a central U.S. governing principle and a part of the foundation of American alliances overseas. It also states that the United States should provide assistance to support democracy and rule of law in partner countries. 2018-2020 State-USAID Joint Strategic Plan. This plan articulates the importance of improving the rule of law in partner countries overseas and identifies this as a strategic objective. It also requires coordination between the two agencies to deliver sustainable assistance that strengthens their democratic institutions. The plan also calls for State and USAID to work together at the country level to develop country-specific strategies that ensure investments are sustainable and that results are valued by partner countries. Foreign Affairs Manual (FAM). The FAM includes specific roles and responsibilities for rule of law assistance and notes that the lead office for such assistance, INL, is responsible for, among other things, the “development of assistance programs directed at U.S. Government objectives abroad on international criminal justice issues.” Bureau-specific plans and documents. INL and several other State bureaus also have their own strategic documents with elements that relate to the provision of rule of law assistance. Specifically: INL’s Functional Bureau Strategy provides a framework for connecting its responsibility for providing rule of law assistance with its specific programs overseas. The strategy also defines how the bureau matches U.S. foreign policy goals with its foreign assistance portfolio, including its allocation to rule of law assistance. State’s other functional bureaus and offices are guided by strategic documents that relate to rule of law assistance. According to State officials, programs provided by these bureaus and offices can touch on rule of law-related efforts such as training on techniques related to investigating and prosecuting trafficking cases. These bureaus include the Bureau of Democracy, Human Rights, and Labor; the Bureau of Counterterrorism; and the Office to Monitor and Combat Trafficking in Persons. State’s regional bureaus are also guided by strategic documents that can relate to rule of law assistance. For example, the Joint Regional Strategy for the Bureau of European and Eurasian Affairs includes a strategic goal related to protecting core U.S. interests by advancing democracy and human rights and strengthening civil society. USAID Strategy on Democracy, Human Rights, and Governance. This strategy identifies the strengthening of institutions that enable the rule of law as part of USAID’s work to foster greater accountability of leaders to citizens and the law. USAID programs are designed to strengthen the institutional and decisional independence of judiciaries, develop judicial self-governance, and introduce best practices in judicial effectiveness. The strategy also states that USAID will continue to offer timely support for institutional development of oversight bodies, including legislatures and auditor general’s offices. 2017 DOJ Strategic Plan. The strategic plan identifies the development of rule of law as a key responsibility for DOJ. According to DOJ officials, DOJ has two main offices that provide rule of law assistance. Both of these offices are within DOJ’s Criminal Division. ICITAP. This office works with foreign governments to develop professional and transparent law enforcement institutions that protect human rights, combat corruption, and reduce the threat of transnational crime and terrorism. ICITAP focuses on law enforcement, correctional institutions, and forensics (whereas OPDAT works primarily with prosecutors and courts). According to DOJ, ICITAP and OPDAT often coordinate their rule of law assistance efforts and pursue a comprehensive approach to criminal justice reform in countries with both a Resident Legal Advisor and an ICITAP advisor. ICITAP programs are implemented by a combination of federal employees and contractors. OPDAT. According to DOJ officials, OPDAT builds foreign partners who can work with the U.S. agencies to enhance cooperation in transnational cases and to fight crime before it reaches the United States. OPDAT has Resident Legal Advisors, Intermittent Legal Advisors, and International Computer Hacking and Intellectual Property Advisors posted at U.S. embassies overseas who provide assistance and case-based mentoring to foreign counterparts to develop justice systems that can combat transnational crime, corruption, and terrorism consistent with the rule of law. According to these officials, OPDAT’s efforts and programming align with, reinforce, and further U.S. law enforcement and national security objectives. The Integrated Country Strategy (ICS) outlines goals and objectives for country-level priorities, such as rule of law assistance. The ICS is developed jointly by State and USAID in the country mission and establishes overall goals and objectives of the U.S. government in the particular country. The ICSs are 4-year strategic plans for whole-of- government priorities in a given country. According to State, the goals and objectives in the ICS are linked to and informed by the National Security Strategy, the State/USAID Joint Strategic Plan, and department regional and functional bureau strategies. ICS documents are organized around higher-level goals to be achieved by meeting objectives and sub- objectives. For example: In Kosovo, the ICS lists two objectives that help achieve the goal of improved rule of law: (1) ensuring that all Kosovo’s citizens have access to reliable, transparent, and accountable governance and justice and that it is responsive to citizens’ needs, and (2) improving delivery of services, implementation of laws and regulations, and committing to countering corruption. In Colombia, the goal to advance Colombia’s capacity to strengthen governance includes the objective of extending the effective presence of democratic institutions and processes, such as the rule of law. To further detail USAID’s in-country efforts, USAID develops a Country Development Cooperation Strategy (CDCS) to plan agency goals and objectives, which are achieved by meeting intermediate and sub- intermediate results for its work in a specific country, such as the provision of rule of law assistance. According to USAID, the CDCS objectives are integrated into the ICS and inform overall rule of law assistance goals and strategy. Some examples include the following: In Liberia, the 2013-2019 CDCS states that the overall goal of “Strengthened Liberian Institutions” should be reached by achieving, among others, the development objective of more effective, accountable, and inclusive governance. This development objective would in turn be achieved by meeting, among others, the intermediate result of increased access to justice, according to the CDCS. In the Philippines, the 2013-2019 CDCS includes the sub-intermediate result of “judicial efficiency improved” as supporting the intermediate result of “economic competitiveness enhanced.” This intermediate result must be reached to achieve the development objective of broad-based and inclusive growth, which in turn contributes to the goal of a more stable, prosperous and well-governed Philippines, according to the CDCS. The mission-wide strategies for the four selected countries varied in how they prioritized rule of law assistance. In Kosovo, Liberia, and the Philippines rule of law was a higher-level priority, such as a goal in the ICS or development objective in the CDCS. In Colombia, the ICS includes improving rule of law as an objective, but not a main goal, and the CDCS lists rule of law as a sub-intermediate result. Depending on the emphasis of rule of law assistance in a particular country, the in-country mission may develop strategies, in addition to the ICS and CDCS, to address a specific priority such as rule of law. In two of the four selected countries, we found that missions had developed additional strategic documents specific to rule of law assistance. In Kosovo, the mission developed a specific rule of law strategy document to guide activities across State, USAID, and DOJ in support of the rule of law goal in the ICS. In Colombia, State and USAID developed a mission rule of law strategy in 2015. In addition, agency officials said they had adapted strategies to fit changing contexts. For example, when a spate of violence targeted human rights defenders and social activists in 2018, the mission in Colombia developed a human rights strategy as a supplement to the rule of law strategy. State, USAID, and DOJ conduct assessments, consult with host governments, and use interagency reviews to identify local rule of law needs. Agency officials noted that local context affects the nature of rule of law programs, and that needs assessments are critical to understanding this context. While each country faces unique and specific rule of law challenges, and agencies have flexibility to conduct foreign assistance as they deem appropriate, some key interventions are consistent across several or all of the selected countries. See appendix IV for more information on key interventions and priority issues in each selected country. Assessments. State and USAID officials said that they can identify needs by conducting assessments of the rule of law in some of the countries we reviewed. They also sometimes contract with other organizations to conduct these assessments as part of the broader contract for a program. DOJ noted that they have used these assessments as an initial baseline against which to evaluate progress, identify critical local assistance needs, inform development of mission strategies such as the ICS and CDCS, and prepare for future activities. According to U.S. officials, program implementing partners can also use assessments to prepare for specific projects and activities according to the terms of grants and contracts with U.S. government agencies. For example, according to officials: In Colombia, State concluded a letter of agreement with the Pan American Development Foundation to conduct an assessment of the function of the local justice sector. Following this assessment, INL officials said they funded a project with the foundation to strengthen the capacity of Colombia’s Attorney General to address issues related to the original assessment. Also in Colombia, USAID’s Justice for Sustainable Peace program conducted a local justice study with civil society organizations and academic experts in 45 municipalities and also conducted six regional political economy analyses during the initial phase of the project, among other analytical tools that shaped the project’s implementation. In the Philippines, a USAID assessment of closed cases and similar studies supported by the World Bank showed that judicial inefficiency was the most serious concern of litigants. Subsequently, USAID officials said they designed and funded a project intended to, among other things, address the two most significant results of inefficiency: docket congestion and court delay. They did this through supporting case inventories and disposition, streamlined litigation procedures, and automated case management. Late in fiscal year 2018, USAID also funded a project to improve access to justice by increasing access to legal information and assistance, and strengthening formal and informal alternative dispute resolution mechanisms. Host government consultation. U.S. officials said they have also involved the host government in identifying rule of law needs. For example: In Liberia, USAID worked closely with the Liberian government while preparing the 5-year CDCS to best capture local views on justice sector needs, according to USAID officials. In the Philippines, DOJ followed up judicial and prosecutor trainings with informal conversations to elicit local official views on rule of law needs and gaps, according to DOJ officials. According to USAID officials, USAID and the government of the Philippines convened interagency meetings consisting of justice system stakeholders to jointly develop the Joint Country Action Plan which includes rule of law priorities and programmatic activities. In Colombia, USAID and the Colombian Ombudsman’s Office jointly identify overlapping areas of interest and develop programs that fit these priorities, according to USAID officials. Interagency review. U.S. officials described collaborative efforts used at missions to identify local rule of law needs. For example: In the Philippines, officials from State, USAID, and DOJ discuss local needs and capacity gaps in the Law Enforcement Working Group and ad-hoc rule of law technical panels. Agency officials noted that, unlike an independent assessment, these groups review proposed and ongoing activities to ensure they meet technical needs identified by all agencies, including potential projects before solicitations for proposals are made public. In Kosovo, U.S. officials who participate in the rule of law working group jointly discuss potential needs and areas of intervention for local rule of law assistance. Also in the Philippines, State and USAID officials jointly serve on technical evaluation committees to ensure that the design matches local needs and U.S. assistance goals. In the selected countries, U.S.-supported rule of law assistance is implemented through country-specific programs, and we identified five examples, among others, of distinct types of rule of law activities. 1. Technical assistance to build human and institutional capacity in the justice system. U.S. agencies provide assistance to improve rule of law capacity in the form of trainings and exchange programs, and through the use of embedded advisors in local institutions. In Liberia, for example, trainings supported by USAID address a variety of issues. According to officials there, trainings are used in programs to increase the number of magistrates, supplement legal education, increase capacity of the Liberian Land Authority, integrate rule of law and property rights concepts into surveyors’ training, and increase the capacity and number of pro bono legal aid providers. (See fig. 5.) Multiple exchange programs provide training to enhance the rule of law, but local government officials from all four selected countries received training at International Law Enforcement Academies, which provide local law enforcement and justice sector officials with rule of law-related training and technical assistance. (See fig. 6.) 2. Embedded advisors. Embedded advisors provide onsite advice to local government officials and may operate in some of the selected countries as either a supplement or the primary agents of training and capacity building, according to agency officials. In several of our selected countries, the U.S. government embeds advisors with local government agencies or courts. According to U.S. officials, these advisors simultaneously provide technical assistance to local officials, but also can report back to the U.S. mission on the opinions and suggestions of local government. In Colombia, DOJ officials said that advisors now focus primarily on counter-narcotics issues but previously worked with host government agencies on human rights and rule of law-related work. They noted that DOJ advisors trained thousands of Colombian judges and attorneys prior to this shift in emphasis. Also in Colombia, USAID supports embedded advisors to provide technical assistance to the Office of Colombia’s Attorney General on human rights defender homicides and gender-based violence and the Inspector General’s Office to support public official disciplinary actions related to human rights protections. In Kosovo, OPDAT and ICITAP embedded advisors provide advice and training to their counterparts in a variety of Kosovo government agencies, including the Ministry of Justice, Ministry of the Interior, and Kosovo Corrections. 3. Legislative and regulatory reform. U.S. agencies and funded implementers work with local governments and programs to reform specific laws and administrative procedures. For example, U.S. programs introduced or expanded the concept of and legal structure for plea bargaining into Colombia, Kosovo, and the Philippines, according to U.S. officials in those countries. In the Philippines, members of the national court system provided data showing how the expanded use of continuous trial methods and plea bargaining, supported by U.S.-funded programs, increased courts’ ability to process cases and begin to reduce the pre-trial detainee population. 4. Resource and equipment provision. Programs provide resources directly to government agencies and civil society groups that are engaged in advocacy around rule of law issues. In the Philippines, for example, USAID provided funds to install e-courts to improve how courts record case information, monitor case flow, and provide public access to the status of cases, according to USAID. They said this productivity tool automates the tasks and functions of the courts, improving overall efficiency, transparency and accountability. (See fig. 7.) 5. Public outreach. Missions conduct interagency public outreach campaigns to promote the rule of law in the host country, including greater awareness of legal rights, responsibilities, access, and resources, according to agency officials. Interagency coordination via the Rule of Law working group allows the Kosovo mission to conduct consistently voiced rule of law-themed public communication, for example. The mission jointly publishes a rule of law tweet to update the public on relevant issues, supports “anti-corruption week,” and provides feedback to host government officials to emphasize U.S. activities and views on specific rule of law issues. (See fig. 8.) In some situations, agency officials have the flexibility to amend a project during the lifespan of the project. For example, in Colombia, officials noted that a sudden rise in violence against social activists and community leaders led State, USAID, and DOJ to adjust their rule of law strategy and programming to focus more on the prevention and prosecution of those crimes. We found that agencies in the four selected countries coordinate rule of law assistance in various ways that do not consistently include relevant agencies, and the sufficiency of these coordination efforts is unknown. Officials in Colombia, Liberia, and the Philippines described their respective approaches to coordinating rule of law assistance as follows, citing their Law Enforcement Working Groups as the usual forum for formal coordination. In Colombia, INL officials said they operated a rule of law project coordination group specifically for INL staff, but the group did not always include other relevant agencies, such as USAID. INL officials said they sometimes also coordinated rule of law assistance amongst agencies through a Law Enforcement Working Group—which also did not always include other relevant agencies—or through the Human Rights Working Group, which did include State, USAID, and DOJ, according to INL. In Liberia, agency officials said that State and USAID sometimes coordinated rule of law assistance through a Law Enforcement Working Group, but the mission did not have a rule of law-specific working group. In the Philippines, agency officials said they coordinated rule of law assistance through a Law Enforcement Working Group, which they said included all relevant members. Although the mission also operated a Counterterrorism Working Group, agency officials noted that rule of law was not a common topic at its meetings. The mission did not have a rule of law-specific working group. By contrast, agencies at the fourth mission we visited—in Kosovo—used a rule of law-specific working group, which included all relevant agencies, to coordinate rule of law activities. Agency officials, including the Chief of Mission, described the working group as a highly effective means of ensuring interagency collaboration and coordination, and as having had a significant positive impact on the effectiveness of rule of law assistance in Kosovo. Agency officials in Kosovo described the working group as a more effective means of coordinating rule of law assistance than other thematic working groups they had utilized in other posts, such as one on Law Enforcement. State and USAID guidance and our prior work have highlighted the importance of coordinating with relevant entities for interagency efforts, such as rule of law assistance, which is provided by multiple U.S. agencies. The 2018-2022 State-USAID Joint Strategic Plan notes that State and USAID will work with their interagency partners to coordinate strategies and programs, including their efforts related to providing rule of law assistance. In addition, we have reported on the importance of interagency collaboration when efforts involve multiple agencies, and have noted that interagency coordination mechanisms or strategies may reduce potentially duplicative, overlapping, or fragmented efforts. The Law Enforcement Working Groups were designed for purposes other than coordinating rule of law activities and are not required to include agencies that play a key role in providing rule of law assistance. The FAM notes that the Law Enforcement Working Group is the primary forum meant to coordinate U.S. law enforcement operations and law enforcement assistance programs under Chief of Mission authority. State and USAID categorize law enforcement assistance differently from rule of law assistance. Specifically, the FAM states that law enforcement assistance coordinated by the Law Enforcement Working Groups includes bilateral or multilateral foreign assistance programs where the host country unit receiving the assistance is authorized to use force. In addition, the FAM permits but does not require the inclusion of development agencies, including those that provide rule of law assistance, such as USAID, in the Law Enforcement Working Groups. The extent to which interagency rule of law assistance coordination mechanisms are sufficient is unknown, because officials, led by the Chief of Mission, at overseas embassies have not assessed the sufficiency of interagency coordination of foreign assistance at overseas posts or ensured that such coordination includes all relevant agencies and bureaus. Given that strategic guidance is largely decentralized, country- level coordination and collaboration efforts are critical to achieving agency and government-wide objectives. Without assessing the sufficiency of a mission’s mechanisms for coordinating rule of law assistance, such mechanisms may not be as effective as they could be, and may also increase the risk of duplicating efforts or fragmenting limited resources. In addition, agencies may also be missing opportunities to leverage interagency resources. Improving the rule of law in partner countries overseas is a key objective of America’s foreign and national security policy. Ensuring that State, USAID, DOJ, and other agencies involved in providing rule of law assistance coordinate their efforts effectively—including involving all relevant entities—is key to providing that assistance in an efficient and accountable way. Overseas missions have the ability to develop whole-of- government strategies that guide their priorities and activities in a given country. As a result, the quality of strategic planning and coordination at the mission level is critical. Agency officials at overseas posts often work in a decentralized manner to design, implement, and coordinate rule of law assistance. While there is a range of coordination mechanisms in place, in selected countries, the extent and nature of interagency coordination varied and the sufficiency of those efforts is unknown. One of the key mechanisms used in-country to coordinate rule of law assistance is designed for other purposes, and, therefore, does not consistently include agencies that play a key role in providing rule of law assistance. Without assessing the sufficiency of their coordination methods, agencies could be missing opportunities to fully leverage limited resources for rule of law assistance, and could also be duplicating efforts and not providing assistance as effectively and efficiently as possible. The Secretary of State should require Chiefs of Mission at overseas missions that receive allocations for rule of law assistance to assess the sufficiency of their coordination methods to verify that this assistance is coordinated with all relevant interagency partners. (Recommendation 1) We provided a draft of this report to State, USAID, DOJ, and DOD for their review and comment. State and USAID provided written comments, which are reproduced in their entirety in appendices V and VI, respectively. State, USAID, DOJ, and DOD provided technical comments, which we incorporated as appropriate. In its written comments, State accepted our recommendation and agreed that an assessment of coordination mechanisms would improve the overall provision of rule of law assistance. State also said that, on behalf of the Secretary of State, INL will provide guidance to require posts to perform an assessment on their coordination of rule of law assistance and come to a determination if coordination sufficiently involves all relevant agency partners. In its written comments, USAID noted that it prioritizes rule of law as a fundamental development outcome, and that it works with State and DOJ in its pursuit of this and other related objectives. USAID also noted a preference for formal rule of law-specific coordination groups to align efforts and reduce duplication. We are sending copies of this report to the appropriate congressional committees and to the Secretaries of State and Defense, the Acting Administrator of USAID, the Attorney General, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2964 or gurkinc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII. This report examines (1) how much funding the Department of State (State) and U.S. Agency for International Development (USAID) allocated for rule of law assistance in fiscal years 2014 through 2018; (2) how agencies strategically plan and coordinate the allocation of rule of law assistance; and (3) what processes agencies have to design, implement, and coordinate rule of law assistance programs in selected countries. This is the first of two reports that will address this issue. To identify which agencies were relevant for a review of global rule of law assistance, we spoke with officials from State, USAID, the Department of Justice (DOJ), the Department of Defense (DOD), and representatives from nongovernmental organizations (NGO) involved in the rule of law sector. On the basis of these interviews and our previous work, we selected State, USAID, and DOJ to review. To address our first objective, we analyzed funding data from State and USAID, and obligation data from DOJ. We primarily relied on allocation data provided by State’s Office of Foreign Assistance Resources (F) for fiscal years 2014 through 2018—the most recent data available at the time of our review. F’s data included allocation data disaggregated by specific recipient country or regional program. Allocation data also was reviewable by the relevant rule of law program area and program elements as listed in State’s and USAID’s Standardized Program Structure and Definitions (SPSD). Rule of law is listed as a program area under the Democracy, Human Rights, and Governance (DR) category within the SPSD as “DR 1” and is composed of five program elements— DR 1.1 through DR 1.5. According to F officials, in fiscal year 2018, F changed its policy to allow operating units to designate activities with other SPSD codes to also count toward rule of law through the “cross- attribution” process. We assessed the reliability of State’s allocation data and determined the data to be sufficiently reliable for the purposes of reporting allocation totals and allocations disaggregated by program element and recipient country. F gathered this information from its FACTSInfo data system, which itself draws from data reported in annual Operational Plans prepared by relevant operating units, according to F officials. We verified the allocation data for the four countries we selected for our review by reviewing the allocated funds listed in the annual Operational Plans for each respective country. The data in the Operational Plans matched the allocation data from FACTSInfo. In addition to the allocation data provided by F, we collected limited obligation data from State’s Bureau for International Narcotics and Law Enforcement (INL) and DOJ. Since DOJ functions primarily as a rule of law assistance program implementer, it reported all of its funding as obligations from State via interagency agreements. DOJ reported obligated funds separately for its two rule of law-focused bodies, the International Criminal Investigative Training Assistance Program (ICITAP) and the Office of Overseas Prosecutorial Development, Assistance and Training (OPDAT). DOJ’s data described all obligated funding for rule of law assistance globally from fiscal years 2014 through July 2019. To evaluate the reliability of DOJ’s data, we asked INL to confirm that DOJ’s obligation totals for the four selected countries matched INL’s. Ultimately, we found the data reported by INL and DOJ to be consistent and sufficiently reliable for the purposes of our reporting objective. To address our second objective, we reviewed documents and interviewed officials in Washington, D.C. We compared strategies and guidance described for the whole of government, specific departments and agencies, and bureaus and offices within those departments. We also reviewed the annual foreign assistance budget process to describe how agencies at headquarters collaborate to determine foreign assistance allocations generally and for rule of law assistance in particular. We reviewed the Integrated Country Strategy documents for each selected country, as well as USAID’s Country Development Cooperation Strategy. We reviewed these documents to identify rule of law thematic priorities and any guidance regarding roles and responsibilities, program implementation, and intra- or interagency coordination. To address our third objective, we selected a non-generalizable sample of four countries: Colombia, Kosovo, Liberia, and the Philippines for site visits or in-depth analysis. We also reviewed one international program— the Regional Training Center, based in Accra, Ghana, part of the International Law Enforcement Academy Program. In selecting these countries, we considered, among other things, (1) countries in which at least two of the three focus agencies had allocated or obligated rule of law assistance funds during fiscal years 2014 through 2018; (2) countries that were among the top half of recipients of rule of law assistance allocations from State and USAID during the same period, as reported in publically available information; (3) geographic dispersal of selected countries, to ensure that no more than one country was selected in each of State’s designated regions; and (4) suggestions from State, USAID, DOJ, and NGO officials with experience in the rule of law sector. We traveled to the Philippines in August 2019 and to Ghana, Liberia, and Kosovo in September 2019. We met with and interviewed officials from State, USAID, and DOJ, and from NGOs that had implemented U.S.- funded rule of law assistance projects, as well as local government officials who had participated in U.S.-funded rule of law assistance activities. We did not travel to Colombia, but conducted interviews with State, USAID, DOJ, NGO, and local government officials in Colombia by phone. We also interviewed officials in Washington, D.C., in person. To examine the processes used by State, USAID, and DOJ to design, implement and coordinate rule of law assistance in selected countries, we reviewed documents and interviewed agency and local government officials and implementing organization staff. We interviewed U.S. and local officials in Washington, D.C.; Colombia; Ghana; Liberia; Kosovo; and the Philippines on methods of identification of local needs, the process of program / activity design, and means of coordinating implementation among multiple agencies, among other topics. We also visited projects in the Philippines, Liberia, and Kosovo, where we were able to observe activities and speak with project implementers, partners, and beneficiaries. We compared the collaboration mechanisms used at these three missions to the collaboration requirements in the 2018-2022 State-USAID Joint Strategic Plan. We conducted this performance audit from December 2018 to June 2020, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Department of State (State) and U.S. Agency for International Development (USAID) categorize and track their foreign assistance according to the Standardized Program Structure and Definitions (SPSD). State and USAID use the SPSD to define overall foreign assistance themes, and to code foreign assistance funds in order to track how U.S. agencies allocate their resources. The SPSD divides foreign assistance into category, program area, and program element. The SPSD comprises seven categories, including Democracy, Human Rights, and Governance, within which rule of law is a specific program area. Rule of law is composed of five program elements: (1) Constitutions, Laws and Legal Systems, (2) Culture of Lawfulness, (3) Checks and Balances with Judicial Independence and Supremacy of Law, (4) Justice Systems and Institutions, and (5) Fairness and Access to Justice. According to State officials, allocated funds are linked to specific SPSD codes in the annual Operational Plans, which are developed by either country-specific or regional operating units. Operating units also determine which program area and program element is the appropriate code for a specific activity. While the SPSD provides definitions of each program element, the definitions may overlap and operating units have some leeway to apply the SPSD codes based on their judgement. Table 1 shows funding associated with each rule of law program element and provides examples of rule of activities that were allocated funds in the selected countries. Beginning in fiscal year 2018, State’s Office of Foreign Assistance Resources (F) began to track allocated funds that were not coded as part of the rule of law program area, but were also planned to be used for rule of law themes, according to F officials. This process is referred to as “cross-attribution.” Cross-attributed funds are designated by operating units in their annual Operational Plan. State officials provided an example from fiscal year 2018, explaining that funding classified under two program elements from the Peace and Security program area were cross- attributed to rule of law. Table 2 shows the cross-attributed allocated funds in fiscal year 2018. Definition: Support the development of constitutions, laws, and legal systems that are procedurally and substantively fair, derived through participatory democratic processes, and consistent with international human rights standards. Both the substance of the law and the process by which it is developed must be legitimate and should be transparent. Includes analysis and dissemination of jurisprudence, innovations, and best practices in constitutional and law-making processes. Includes programs that assist in strengthening systems and processes for developing and enacting laws. Supports efforts to end impunity and enable peaceful transitions to democracy. Customary or religious dispute resolution mechanisms are included as laws, and legal systems do not have to be written or formal to be legitimate. Definition: Foster and maintain a culture that is generally law-abiding, including through legal literacy, public awareness, constituency building, and citizen engagement in legal processes. Ensure that the public is educated about laws and regulations, perceives laws as legitimate and worthy of adherence, and respects the authority of law and legal institutions. Develop citizen demand for an effective and accountable justice system, and develop associations to advocate for all citizens. This includes programs that spur a culture of lawfulness by changing beliefs and attitudes by socializing people into a rule of law culture and changing norms so that people abide by the law. This also includes rule of law programs or civil society programs with a very specific focus on rule of law-related citizen awareness and education—i.e., supporting civil society organizations to participate in public hearings as part of a larger effort to strengthen the parliament or working with a civil society organization to provide legal representation of indigent populations as part of an overall judicial strengthening strategy. Definition: Strengthen judicial independence as a means to maintain separation of powers and check excessive power in any branch or level of government. Strengthening judicial independence includes reducing improper influences on the judiciary through: open and participatory processes for judicial selection and appointment; security of tenure; satisfactory budget allocations to ensure adequate infrastructure, training, and working conditions; judicial self-governance including management of administrative, budgetary, ethics, and disciplinary processes and reform; and transparent court operations and judicial processes. Enhance the judiciary’s ability to check abuses of power by any branch or level of government through creating and strengthening constitutional or judicial review. This element also helps ensure that government is bound by law, and government decision-making is in accordance with the law. Work to create an independent and impartial justice system through institutional and behavioral change, and also to promote public respect for the justice system and judicial decision-making. Definition: Improve the systems, capacity, and sustainability of civil and criminal justice sector and institutions, improve the ability and skills of justice sector actors, and enhance coordination amongst them where appropriate (includes harmonization of policies, procedures, and systems, and public / private partnerships relating to both criminal and civil law). Justice sector actors and institutions include: police, border security, prosecutors, forensics experts, judges, court personnel, public defenders, mediators, arbitrators, conciliators, corrections personnel, private bar, law schools, legal professional associations, and training institutions for each of them. Support educational and training programs for all justice system actors, to include reform of pedagogy and curricula, continuing and in- service training, and support of accreditation and legal professional associations to promote professionalism; and encourage public service. Improve administrative and operational systems, including strategic planning, budget, procurement, and personnel. Definition: Work toward an equitable justice system by ensuring fairness in law and process. Fairness programs include non-discrimination law fair trial standards, effective administrative law systems to guard against arbitrary government action, and observance by all justice system actors and institutions of international treaties and customary law. Support monitoring and advocacy by justice sector NGOs, including strategic lawyering, trial monitoring, and policy dialogue. Improve equitable access to justice through increasing the quality and quantity of state and non- state justice services, with a particular focus on women, youth, the poor, LGBT persons, and other marginalized or vulnerable groups. This includes access to state and non-state dispute-resolution fora; court redistribution; mobile courts; the removal of language, gender, cultural, sexual orientation, gender identity and physical barriers; circulation of laws and legal decisions; alternative dispute resolution systems; and expanding access to legal services (e.g., public defenders’ offices, legal aid and legal services, labor law services, justice or legal resources centers). This also includes programs to educate the citizenry about their rights, how to access services, and how to encourage change. Programs primarily focused on trafficking in persons should be captured under Peace and Security (PS) PS.5 and programs focused on alien smuggling under PS.4. Definition: Develop modern police forces through capacity-building (training and education both in the classroom and in the field) with focus on creating police institutions that can effectively fight crime and serve the public. Activities include, but are not limited to, police academy reform, organizational restructuring, professionalization, developing internal affairs, civil service reform (pay and rank reform), management and leadership, equipment and infrastructure support, aviation support, gender sensitivity, community-oriented policing, and public affairs. Assistance can also support the establishment and sustainment of effective, professional, and accountable law enforcement services (civilian police, stability / formed police units, and specialized units trained and equipped for specific issues such as port and maritime security, border security, gangs, or kidnapping). As the foundation for such a service is fundamentally rooted in the rule of law and respect for human rights, activities conducted in support of this element should be coordinated with programs under the Rule of Law elements in the Democracy, Human Rights, and Governance (DR) category. Definition: Provide consultation on facilities, system, and process design; increase the capabilities and professionalization of corrections personnel at all levels through training, with the goal of developing sustainable operations and infrastructure in compliance with international guidelines, especially with respect to human rights. Implement an objective classification system to separate inmates by risk and status (felony / misdemeanor / pretrial); reduce pretrial detentions and other causes of overcrowding; eliminate factors that lead to violent uprisings and intergang violence; provide specialized equipment and vehicles to ensure secure operations; and develop a path toward independent international accreditation of facilities and operations to ensure effective, transparent, and accountable corrections systems. Activities conducted herein are in support of long-term development of effective, transparent, and accountable penal systems (described under the Democracy, Human Rights, and Governance (DR) Category). For this review, we collected and analyzed foreign rule of law assistance allocation data from the Department of State’s (State) Office of Foreign Assistance Resources (F). F tracks funding allocations by operating unit, which may be either one particular country, such as Afghanistan or Colombia, or a regional or programmatic unit, such as “State Western Hemisphere Regional” or “Near East Regional Democracy.” Allocations to regional and programmatic operating units shown in table 3 below are not inclusive of the allocations to individual countries on this list. For example, the funding allocated to State’s Western Hemisphere Regional operating unit does not include the funding allocated for the Colombia operating unit. While the regional operating units may conduct activities within particular countries, because the funds are managed from the regional perspective, they are considered different streams of funding. Both regional and country-specific operating units include funds for both State and the U.S. Agency for International Development. This appendix provides a review of rule of law-related issues in selected countries in four different geographic regions. We selected a non- generalizable sample of four countries—Colombia, Kosovo, Liberia, the Philippines—to review specific rule of law programs and the ways agencies coordinate their rule of law assistance in-country. The following pages include some key facts and background information about those countries, key challenges to the rule of law, and U.S. rule of law assistance activities. In 2016 the government of Colombia and the Revolutionary Armed Forces of Colombia (FARC) signed a final peace accord calling for demobilization of armed insurgents, the establishment of new transitional justice institutions, and the introduction of the FARC as a non-violent actor in the Colombian political community, according to the Central Intelligence Agency’s World Factbook. The World Factbook also reports that conflict resulted in many lives lost, more than seven million internally displaced persons, and tens of thousands of “disappeared” victims. While the FARC has laid down its arms, challenges posed by remaining insurgent groups and narco traffickers remain. According to U.S. officials, in the absence of a full establishment of rule of law and equal access to justice for all populations, the country risks sliding back into conflict. Key Challenges to the Rule of Law In recent years, according to officials from the Department of State’s (State) Bureau for International Narcotics and Law Enforcement (INL), the presence of illegal armed groups and narcotics trafficking organizations—which have led to an increase in violence against human rights defenders and social activists—has challenged the government of Colombia’s ability to project the rule of law into rural and former conflict zones. In addition, the Integrated Country Strategy for Colombia notes that much of the gold production in Colombia is carried out by organized criminal actors and armed groups, which robs the government of tax revenue, harms human health and the environment, and prevents licit producers from entering the market. $14,400 GDP per capita (2017 est.) Professional / technical capacity Need for enhanced skills for targeting complicated criminal acts (narcotics trafficking, money laundering, and dismantling organized crime) Corruption Corruption related to narcotics trafficking risks overwhelming the government U.S. Rule of Law Assistance Activities Colombia is one of the largest recipients of U.S. rule of law assistance in the world, and programs have sought to address an array of interrelated issues, according to U.S. officials. These officials said that State and the U.S. Agency for International Development (USAID) have collaborated on responding to violence against human rights defenders. INL works with the Department of Justice to improve the capacity of local prosecutors and law enforcement. USAID officials said that they support programs to increase access to justice, including strengthening indigenous justice, instituting alternative dispute resolution mechanisms, and collaborating with the Colombian Public Defender’s Office to expand legal representation for indigent and at-risk communities. They also said that they strengthen the investigation and prosecution of gender-based violence and social leader cases, investigation of public officials failing to protect social leaders, and justice and reparations for victims of armed conflict. Following violent internal conflict from 1998 through 1999, Kosovo remained under the stewardship of the United Nations (UN) until it declared independence in 2008, according to the Central Intelligence Agency’s World Factbook. According to Department of State (State) officials, the 2013 Brussels Agreement resulted in Kosovo and Serbia further partially normalizing relations; however, Kosovo is not universally recognized as a state and is not currently permitted to join the UN, North Atlantic Treaty Organization, or European Union (EU), among others. With U.S. support, the government of Kosovo has sought to reform its legal institutions with the aim of joining the EU. The United States is committed to helping the government of Kosovo reach this goal. 10,887 square kilometers in area (slightly larger than Delaware ) 1,907,592 (July 2018 est.) In 2018, administration of the legal system transferred from foreign oversight to full Kosovo government control, according to State officials. Consequently, local officials said they had to staff courts, translate casefiles kept in other languages, set new rules and regulations, and accomplish a range of other administrative functions in addition to day-to-day court operations. In addition, Kosovo’s legal system had to integrate the previously parallel Serbia-run legal system into Kosovo’s legal and judicial institutions, according to State officials. Albanian 92.9%, Bosniak 1.6%, Serb 1.5%, Turk 1.1%, Ashkali 0.9%, Egyptian 0.7%, Gorani 0.6%, Romani 0.5%, other / unspecified 0.2% (2011 est.). These estimates may exclude northern Kosovo because of census boycotts by Serb and Romani communities. Professional / technical capacity Need for enhanced basic and advanced skills to address complicated criminal acts (money laundering, cybercrimes, trafficking in persons) Corruption Nepotism and cronyism are persistent features of the civil service and political culture $10,900 GDP per capita (2017 est.) U.S. agencies have provided assistance to the government of Kosovo through a variety of means. The Department of Justice embeds advisors in multiple offices of the government of Kosovo, including the Ministry of Justice, Ministry of the Interior, Kosovo Corrections, and police inspectorate, according to agency officials. These officials also said that the advisors provide traditional classroom- based technical training to Kosovo government officials, as well as real-time advice on particular cases and guidance for the development of new regulations. Officials also said that several U.S.-funded small-grant and educational exchange programs have enhanced the capability of local officials and civil society representatives to manage and advocate for an improved justice sector. To ensure an inclusive and transparent judicial process, officials from the U.S. Agency for International Development said they train local government officials in areas such as transparent procurement processes, and local and central government officials on drafting policies and legislation. Agencies at the U.S. Embassy in Kosovo also collaboratively operated a public affairs campaign to engage with Kosovo citizens on rule of law issues, according to U.S. officials. Liberia, which the World Bank categorizes as a low income country, has a history that includes a 14-year civil war as well as the West African Ebola epidemic of 2014 and 2015. When the United Nations peacekeeping mission in Liberia completed its nearly 14-year deployment, the withdrawal of the several thousand peacekeeping personnel caused a significant economic recession, according to U.S. officials. The recession was exacerbated by drops in commodity prices, which left the government of Liberia unable to pay salaries to officials for months at a time, according to U.S. and Liberian officials. Within this context, the U.S. government has identified rule of law assistance as a priority for Liberia. U.S. officials stated that, by improving local rule of law, the United States can simultaneously address weaknesses in multiple sectors of Liberia’s government and social services, including land management, health, and justice. Key Challenges to the Rule of Law According to U.S. officials in Liberia, enhancing Liberia’s land-management system is key to helping establish rule of law throughout Liberia. Land disputes were one underlying cause of the civil war and remain a threat to stability, according to U.S. officials. These officials explained that disputes are complicated by the destruction of the national property registry during the war, a critical shortage of qualified arbiters and surveyors, and some judicial officials’ poor understanding of property laws. Further, the officials said that persistent and slow-to-resolve land disputes highlight gaps in the administrative capacity of courts, local officials’ lack of technical skills necessary to resolve such disputes, and the ease with which corruption may subvert the rules-based order. Corruption Allegations of corruption threaten the government’s authority but present an opportunity for empowering local anti-corruption actors U.S. Rule of Law Assistance Activities Both the Department of State (State) and U.S. Agency for International Development (USAID) have embedded trainers within Liberian government ministries, such as the Ministry of Justice and the Liberia Land Authority. USAID funded an integrated rule of law and property dispute program to address multiple areas of weakness. State adapted a Centers for Disease Control and Prevention- sponsored rapid response program to identify and resolve potentially destabilizing conflicts. USAID also supported wider access to justice by funding a new legal aid network and providing fellowships for law students to work in rural communities. The government of the Philippines’ expansion of the anti-drug campaign has counteracted progress made in reducing congestion in the Philippine courts and trial duration, according to U.S. officials. One local official we interviewed noted that violations of drug laws make up more than 70 percent of the criminal docket and that large numbers of arrests have led to a highly congested court system. A high volume of arrests and slow processing of cases has also resulted in a dramatic increase in the number of pretrial detainees, according to U.S. officials. 105,893,381 (July 2018 est.) Court docket congestion Anti-drug campaign has overwhelmed an already burdened case management system $8,400 GDP per capita (2017 est.) U.S. Rule of Law Assistance Activities Department of State (State) U.S. Agency for International Development (USAID) and Department of Justice (DOJ) programs are designed to respond to the shift in the government of the Philippines’ priorities, according to U.S. officials. State provided training to Philippine law students, judges, and law enforcement officials that emphasized improved collection and interpretation of evidence. State also funded the establishment of legal aid clinics to improve community access to representation. USAID funded the introduction of “e-courts” and other information technologies in the judicial sector to improve the efficiency and transparency of court proceedings. USAID also funded programs to introduce new legal mechanisms, such as plea bargaining and continuous trial, to reduce the pre-trial detainee population and speed the administration of justice. DOJ has programs to increase prosecutor-police cooperation and to build capacity to combat specific threats, including trafficking in persons, cybercrime, terrorism, and financial crime. In addition to the contact named above, Joe Carney (Assistant Director), Brian Hackney (Analyst in Charge), Benjamin Legow, Carolina Morgan, Afsana Oreen, Abena Serwaa, Parul Aggarwal, Debbie Chung, Justin Fisher, Jenny Grover, Chris Keblitis, and Alex Welsh made key contributions to this report.", "summary": "Rule of law strengthens protection of fundamental rights, ensures a robust civil society, and serves as a foundation for democratic governance and economic growth. According to State, countries with a strong rule of law provide a more level playing field for American businesses to engage and compete, and countries with a weak rule of law can potentially export transnational threats and economic insecurity, undermining the interests of the United States. GAO was asked to review U.S. rule of law assistance around the world. This report examines (1) how State and USAID allocated funds for this assistance in fiscal years 2014 through 2018, (2) how agencies strategically plan and allocate this assistance globally, and (3) what processes agencies have to design, implement, and coordinate this assistance in selected countries. GAO reviewed State, USAID, and DOJ documents and data for fiscal years 2014 through 2018 and interviewed officials in Colombia, Kosovo, Liberia, the Philippines, and Washington, D.C. GAO chose these countries on the basis of funding amounts and other factors. The Department of State (State) and the U.S. Agency for International Development (USAID) allocated more than $2.7 billion for rule of law assistance from fiscal years 2014 through 2018—the latest available data as of GAO's review. Of that, State allocated over $2 billion and USAID allocated over $700 million. State and USAID funded some of these programs through the Department of Justice (DOJ). Rule of law assistance funded a variety of activities including improving justice institutions, legal reform, and promoting a culture of lawfulness. The agencies implemented these programs globally but allocated most funds to the Western Hemisphere and Afghanistan. After Congress appropriates funding, agencies determine rule of law allocations through the foreign assistance budget process. State and USAID identify rule of law as a goal in agency-wide strategic documents and hold an annual interagency roundtable regarding rule of law assistance to determine those allocations. Rule of law assistance is guided by national and agency-, bureau-, and mission-specific strategies that are linked to the national security goals of the United States. These strategies discuss the agencies' roles and responsibilities in improving the rule of law. State and USAID guidance highlights the importance of coordination between agencies as they design and implement rule of law assistance, but not all agencies are included in some of the key coordination mechanisms used in four countries GAO selected for review. Agency officials in the selected countries cited the use of some informal and formal coordination practices, such as the use of law enforcement working groups, but State policy does not require all entities that may be involved in rule of law assistance to participate in these working groups. For example, in three of the four selected countries, officials described coordinating rule of law assistance, in part, through these working groups, which may not include critical agencies such as USAID. According to State policy, these working groups are designed to achieve other goals using agencies and offices that are not involved in providing rule of law assistance. Without verifying that interagency coordination includes all relevant entities, missions may not know whether they are fully leveraging interagency resources or ensuring that they do not duplicate or overlap rule of law assistance. GAO recommends that State require overseas missions where rule of law assistance funds have been allocated to assess whether this assistance is coordinated with all relevant interagency partners. State concurred with our recommendation.", "document_type": "gao"}
{"report": "To be eligible for Job Corps, youth must generally be 16 to 24 years old at the time of enrollment; be low-income; and have one or more barriers to education and employment, such as being homeless, a school dropout, or in foster care. The vast majority of students live at Job Corps centers in a residential setting, while the remaining students commute on a daily basis from their homes to their respective centers. This residential structure is unique among federal youth programs and enables Job Corps to provide a comprehensive array of services to students 24 hours a day, 7 days a week. These services include housing, meals, clothing, medical and dental care, academic instruction, and job training. ETA administers the Job Corps program through its Office of Job Corps under the leadership of a national director and a field network of six regional offices located in Atlanta, Boston, Chicago, Dallas, Philadelphia, and San Francisco. Of the 119 centers, 94 are operated under contracts with various businesses, Native American tribes, and nonprofit organizations. Job Corps’ predominantly contractor-operated structure is unique among ETA’s employment and training programs, according to ETA officials, as other programs it administers are generally operated by states through grants. Several Job Corps contractors have operated centers for two or more decades, and some contractors operate multiple centers. For example, by the end of program year 2016, over two-thirds of Job Corps’ contract centers were operated by seven contractors. The remaining 25 centers (called Civilian Conservation Centers) are operated by USDA’s Forest Service through an interagency agreement with DOL. Figure 1 presents a map of ETA’s Job Corps center locations and regions. Multiple offices within DOL at the national and regional levels are involved in Job Corp center contracting (see fig. 2). Three offices within ETA award and monitor Job Corps center contracts. The Office of Job Corps oversees program operations and monitors contractors who operate Job Corps centers. Each regional office has between seven and nine program managers who carry out these functions and assist in the contracting process. The Office of Contracts Management awards and manages Job Corps center and other support contracts, and oversees ETA’s Contract Review Board, which, among other things, generally reviews all competitive Job Corps center contracts over $1 million. Each regional office has one contracting officer who is the designated official with the legal authority to enter into, administer, and terminate Job Corps contracts on behalf of the government. In addition, regions have contract specialists who assist the contracting officer in managing Job Corps center and support contracts. Contracting officers and contract specialists in Job Corps’ regional offices report to the Office of Contracts Management. The Office of Financial Administration monitors Job Corps’ budget and spending, communicates information about the availability of funds for Job Corps center and support contracts, and calculates and pays incentive fees to contractors, among other types of fees. At the national level, budget analysts carry out these functions and are assigned to each Job Corps regional office. In addition, other DOL offices are involved in Job Corp center contracts. Specifically, DOL officials said that the Office of the Solicitor provides legal advice and representation to ETA on legal matters related to Job Corps center contracts, such as protests and contractors’ failure to meet specific contractual requirements. DOL’s department-wide Procurement Review Board within the Office of Procurement Policy reviews and approves all noncompetitive Job Corps center and support contracts. DOL’s Office of Small and Disadvantaged Business Utilization reviews and makes recommendations on all ETA procurements over a certain threshold. DOL officials told us this includes reviewing whether to set aside Job Corps center and support contracts for small businesses. Similar to other federal agencies, ETA is generally required to use full and open competition—meaning all responsible parties are permitted to compete—when awarding contracts. Competition is considered a cornerstone of the federal acquisition system and a critical tool for achieving the best return on investment for taxpayers. In addition, competitively-awarded contracts can help conserve scarce resources, improve contractor performance, curb fraud, and promote accountability. In fiscal year 2017, over 80 percent of obligations at federal civilian agencies (non-defense) were awarded competitively. Despite the preference for competition, federal procurement law recognizes that full and open competition is not feasible in all circumstances and authorizes contracting without full and open competition under certain conditions. For example, contracting officers may award a contract noncompetitively if one of seven exceptions listed in Federal Acquisition Regulation (FAR) subpart 6.3 applies. Examples of allowable exceptions include circumstances when products or services required by the agency are available from only one source, or when the need for products and services is of such an unusual and compelling urgency that the federal government faces the risk of serious financial or other injury. Generally, exceptions to full and open competition must be supported by written justification and approval documents that contain sufficient facts and rationale to justify use of an exception. ETA’s process for awarding and monitoring Job Corps center contracts generally consists of several phases, which we have categorized into six areas that reflect the federal contracting process. As shown in figure 3, Job Corps’ contracting process starts with acquisition planning and concludes with contract administration. Within each phase, regional program and contracting officials conduct various contracting activities, such as evaluating proposals received from prospective contractors. In addition, budget analysts support the acquisition process by communicating information about the availability of funding for Job Corps center contracts, among other duties. Interested parties—including actual or prospective offerors—may make written objections (which are referred to as protests) of an agency’s actions concerning the solicitation and award of contracts. For example, interested parties may object to the award of a Job Corps center contract if they believe the contract was awarded improperly. Parties may file protests in several different venues, including with the agency, GAO, or the U.S. Court of Federal Claims. Parties that disagree with the agency’s protest decisions or GAO’s recommendations can file a protest with the U.S. Court of Federal Claims. The legal procedures and the length of time it can take to resolve a protest varies based on the venue in which the protest was filed. For example, protests filed with the agency should be resolved within 35 days, while GAO generally decides protests within 100 days. In some instances, interested parties may seek to halt the award or suspend performance of the contract until the protest is resolved. This can introduce potential delays in the agency’s acquisition process or interrupt the performance of an existing contract. Protests may be resolved in a variety of ways depending on which venue the protest was filed. For protests that are found to have merit, the agency may take actions such as issuing a new solicitation, re-competing a contract, or terminating a contract. Parties can also withdraw their protest at any time during the process. ETA established a performance management system (commonly referred to as the Job Corps’ Outcome Measurement System) to assess center performance and program effectiveness. In program years 2016 and 2017, center contractors collected and reported to ETA data related to performance measures that generally fall under three areas of services provided to students: (1) direct center services (e.g., helping students attain a high school diploma or high school equivalency); (2) short-term career transition services (e.g., placement of graduates in a job related to their training); and (3) long-term career transition services (e.g., job placements of graduates 6 and 12 months after completing the program). For each measure reported, ETA established a national performance goal and assigned a weight that represents its relative importance for achieving student outcomes. The sum of the ratings on each performance measure was used to develop an overall ranking for each center. According to ETA officials, they revised Job Corps’ outcome measurement system for program year 2018 to align with requirements under the Workforce Innovation and Opportunity Act (WIOA). Under the Act, ETA is required to annually assess the performance of each Job Corps center and to report to Congress on their performance based on specified performance indicators. Officials said they are currently tracking Job Corps data on eight performance measures related to various student outcomes such as measurable skills gain and credential attainment (i.e., earning a high school diploma or its equivalent, or completing career and technical training). ETA reported these new measures for program year 2018. In certain situations, it may become evident that services could lapse before a subsequent contract can be awarded. In these cases, because of time constraints, contracting officers may, for example: (1) extend the existing contract or (2) award a short-term stand-alone contract to the incumbent contractor on a sole-source basis to avoid a lapse in services. Both these extensions and new sole-source contracts are informally referred to as bridge contracts by some in the acquisition community, and we have used this definition in previous work. In our October 2015 report, we found that the three selected agencies included in our review—the Departments of Defense, Health and Human Services, and Justice—had limited or no insight into their use of bridge contracts, as bridge contracts were not defined or addressed in department-level guidance or in the FAR. In response, we recommended that the Administrator of the Office of Federal Procurement Policy (OFPP)—an office within the Office of Management and Budget (OMB) that provides government-wide guidance on federal contracting— take the following actions: (1) develop a standard definition for bridge contracts and incorporate it as appropriate into relevant FAR sections and (2) provide guidance to agencies as an interim measure until the FAR is amended. OFPP agreed with these two recommendations; however, as of May 2019, OMB had not yet implemented them. We acknowledge that in the absence of a government-wide definition, agencies may have differing views of what constitutes a bridge contract. For example, ETA informed us that it does not consider competitive contracts that exercise the “Option to Extend Services” under FAR 52.217-8 to be bridge contracts. However, ETA and DOL could not provide us with a documented definition of bridge contracts for their agency. Contracts and extensions (both competitive and noncompetitive) are included in our definition of a bridge contract because the focus of the definition is on the intent of the contract or extension. Nearly three-quarters of the Job Corps centers (68 of 97) were operated by contractors under bridge contracts at some point during program year 2016. Of the 68 centers that operated under bridge contracts, 58 centers had at least one bridge contract awarded on a sole source basis, or noncompetitively. The other 10 centers had bridge contracts based on use of the “Option to Extend Services” clause. While GAO has found that bridge contracts are generally envisioned as short-term, over two- thirds of the centers (49 of 68) that used bridge contracts in program year 2016 operated under them for at least 12 months, with over a third of these centers operating under bridge contracts for at least 2 years or potentially longer. Figure 4 shows the minimum length of time ETA used bridge contracts to operate Job Corps centers. Our in-depth review of 10 centers highlights how a center may use bridge contracts for longer periods of time. For example, for 1 of the 10 centers we reviewed and that operated under bridge contracts for 30 months, ETA first opted to exercise the option to extend services clause with the same contractor for 6 months, between May and October 2014. By the end of the extension, ETA was unable to award the follow-on contract and instead awarded a 2-year bridge contract to the same contractor. ETA stated that with respect to this center, it needed to use a bridge contract due to several factors, including protests, funding challenges, and internal efforts to strengthen aspects of the procurement process. Subsequently, ETA awarded a competitive follow-on contract in September 2016 to a new contractor. ETA cited several reasons that contributed to its need to use bridge contracts during program year 2016, according to the justification and approval documents we reviewed and our interviews with national and regional officials. For example, acquisition workforce challenges were a primary reason ETA cited for its need to use bridge contracts. ETA also frequently cited protests by Job Corps contractors; at times citing protests that dated back to 2011. Leading up to program year 2016, ETA national officials said they encountered a number of acquisition workforce challenges that affected their ability to competitively award Job Corps center contracts. These challenges included: (1) staff attrition in key contracting positions, (2) the need to hire and train new contracting staff, and (3) the need to divert staff to address new requirements under WIOA and other issues. As discussed earlier, ETA has one contracting officer position for each of its six regions. ETA officials said they faced significant attrition in the Office of Contracts Management around 2013 when all but one of the six regional contracting officers left or retired, leaving them with limited regional resources to award center contracts. San Francisco was the only Job Corps region that did not lose its contracting officer. Officials said that this may help to explain why the region operated under fewer bridge contracts as compared to the other five regions. In addition, ETA officials said the agency decided to centralize contracting positions in the national office in 2013 due to concerns about oversight of regional contracting staff. In 2015, ETA decided to reestablish its regional contracting structure, with one contracting officer in each region. To address the large number of staff departures, ETA hired new contracting officers and all of the contracting officers we spoke with told us that they joined ETA’s Office of Contracts Management in 2015 or 2016. When ETA filled its staff vacancies, it hired contracting officers who had prior experience at other agencies. Nonetheless, some contracting officers said it still took time for them to get up to speed due to the uniqueness and complexity of Job Corps center operations contracts. Program officials said that the additional time needed to explain program requirements to new contracting staff slowed down the contracting process. Also, national officials said that contracting officers were unable to competitively award center contracts because of the time needed to carry out acquisition planning tasks, which as we have previously reported, are important to establishing a strong foundation for the contracting process. Such activities include market research, which is used to collect and analyze information about capabilities within the market available to satisfy agency needs. According to ETA’s Acquisition Handbook, market research should occur at least 16 months prior to the anticipated award of a new center contract and after the requirements have been developed by the Office of Job Corps. Figure 5 provides an example of how acquisition workforce challenges affected one of the centers in our in-depth review. Additionally, in the written justification and approval documents for noncompetitive bridge contracts related to 35 of the 68 Job Corps centers that operated under bridge contracts during program year 2016, ETA officials noted that they had to divert contracting staff to implement contracting changes that resulted from the passage of WIOA. WIOA included provisions that affected the Job Corps contracting process, including requiring that certain criteria be considered when selecting an entity to operate the centers. DOL issued regulations implementing these provisions in August 2016. Additionally, in written justification and approval documents for noncompetitive bridge contracts related to 34 of the 68 Job Corps centers that operated under bridge contracts during program year 2016, ETA officials noted that they diverted staff from awarding Job Corps procurements to address financial issues encountered by the program. GAO and DOL’s Office of Inspector General previously reported on earlier problems with ETA’s financial management oversight of Job Corps. In particular, DOL’s Inspector General reported insufficient management oversight and inadequate documentation led to ETA obligating funds that had yet to be appropriated across multiple years. In response, ETA officials said that the agency had, among other actions taken, provided training to its program and contracting staff in program year 2016. In our review of ETA’s written justification and approval documents for noncompetitive bridge contracts related to 42 of the 68 Job Corps centers that operated under bridge contracts during program year 2016, ETA officials cited protests from Job Corps offerors as a reason for using bridge contracts. Some of these justifications cited specific center protests, while others cited the accumulation of protests beginning in 2011. According to ETA officials, in general, each time a protest is filed, the center contract in question is either not awarded or performance on the contract is suspended until the protest is resolved. Our analysis of DOL’s data of protests filed with GAO, the agency, or the U.S. Court of Federal Claims shows that a total of 11 protests were filed in program year 2016 related to seven centers; however, Job Corps offerors filed 44 protests in the four proceeding program years. Figure 6 presents DOL’s data on the number of Job Corps center protests by decision outcome filed in program years 2012 to 2016 before GAO, the agency, or the U.S. Court of Federal Claims. ETA officials said that the accumulation of protests filed since 2012 contributed to the agency’s heavy reliance on bridge contracts in 2016. ETA officials explained that they temporarily suspended the issuance of solicitations for center contracts prior to program year 2016 to address the issues raised in the protests. This resulted in a backlog of contracts waiting to be competitively awarded. We found that protests were not the only factor contributing to ETA’s need to use bridge contracts. Figure 7 provides an example of how a protest and other factors affected one center in our in-depth review. In one partially sustained protest filed at GAO, GAO found that ETA failed to meaningfully consider whether another contractor was capable of performing the procured services before it awarded a noncompetitive bridge contract to the incumbent contractor. In this instance, ETA published a notice of its intent to award a sole-source contract, inviting companies to submit a statement demonstrating their capabilities within 7 days. However, a day after publishing the notice, DOL’s chief procurement officer signed the justification for the sole-source contract, and DOL entered into the sole-source contract with the incumbent contractor without considering other prospective contractors’ capability to perform the procured services. ETA officials told us that some of the protests were caused in part by the agency’s decision to set aside more Job Corps center contracts for small businesses. Federal regulations require all federal agencies with procurement authority to “provide maximum practicable opportunities” for small businesses to win awards for government contracts, thereby meeting specific government-wide goals. ETA officials said that the agency’s decision to set aside more center contracts for small businesses precluded larger incumbent contractors—some of which had historically operated centers—from competing for some center contracts. In response, ETA officials said some of these contractors filed protests that challenged ETA’s decisions to set aside center contracts for small businesses. ETA identified a number of other contracting issues as reasons for using bridge contracts. For example, in the justification and approval documents we reviewed related to contracts for four centers, ETA officials said procurements for competitive Job Corps center contracts were suspended because the pre-award processes had been compromised due to the unauthorized release of confidential contractor information in 2015. This included sensitive information on the incumbent contractor’s staffing levels and rates of pay, among other information. In response to this unauthorized release, ETA delayed new competitive procurements and used bridge contracts to continue services until the released information was no longer applicable and would not harm the contractor’s ability to compete. ETA officials said they more recently used various strategies to improve the contracting process, which allowed them to award competitive contracts more quickly and reduce their reliance on noncompetitive bridge contracts to operate Job Corps centers. According to our analysis of FPDS-NG data and contracting documentation, most of the centers (48 of 68) that operated under bridge contracts during program year 2016 transitioned to competitively awarded contracts by the end of program year 2017. The strategies ETA identified as contributing to reducing the backlog of centers awaiting contract awards included: Prioritizing staff efforts on competitive awards. Contracting officials said that they awarded competitive contracts for an average of 12 to 14 Job Corps centers in a region at the same time, which they noted is a high volume of contract activity to execute concurrently. They said that Job Corps center contracts typically can take approximately 8 to 12 months from solicitation to award for new 5- year competitive procurements. In regions without a contracting officer, officials said that they had to rely on contracting officers from other regions and the national office to handle the workload. In addition, some program officials said that they were instructed to prioritize competitive procurements over some of their other program responsibilities, such as conducting on-site visits at Job Corps centers. As of January 2019, officials said they were able to clear the entire procurement backlog for center contracts during 2018. Using oral presentations to evaluate prospective contractors. ETA officials said they increased their use of oral presentations, in accordance with FAR 15.102, from prospective contractors during the initial evaluation phase of the contract award process. In a typical initial evaluation, regional program and contracting officials assess prospective contractors’ ability to meet the contract requirements, among other areas. Contracting and program officials told us that reviewing technical proposals can be very time consuming because each proposal can be more than 100 pages long; thus, in ETA’s view, oral presentations can streamline the proposal review process. Awarding indefinite-delivery/indefinite-quantity (IDIQ) contracts. In November 2016, ETA awarded IDIQ contracts that allow ETA to quickly award task orders in the event a center may experience a lapse in services, such as when a center contractor files for bankruptcy and abandons the center. ETA officials also said that such contracts could be used when a center contract is expiring and no follow-on contract has been awarded. According to the solicitation for the IDIQ contracts, selected contractors should be able to quickly take over center operations with limited disruption, provide the upkeep of the facility, and ensure safe living and learning environments for students, among other duties. Twelve contractors were awarded IDIQ contracts and may compete for task orders to operate specific centers. Regional contracting officials said the process for awarding a task order is generally faster than their typical competitive center contracts. They also noted that IDIQ contracts have been a helpful tool in continuing operations at centers during protests. Regional officials told us that incumbent contractors would previously file protests when they were unsuccessful in winning new center contracts because their existing contract was extended while the protest was resolved. In the future, ETA officials said they can quickly award a task order from an IDIQ contract to replace an incumbent contractor during a protest. Also, under the terms of the solicitation for the IDIQ contracts, contractors who received one of the 12 IDIQ contracts would be prohibited from competing for task orders for centers where they are the incumbent contractor. In program year 2017, ETA awarded task orders to continue services at four centers. Despite ETA’s efforts to reduce its use of bridge contracts, we identified ongoing acquisition planning and workforce challenges. These challenges fall into three categories: (1) planning for future procurements; (2) addressing acquisition workforce vacancies; and (3) implementing a new contracting approach. These areas could pose a risk to ETA’s management of Job Corps center contracts, including its ability to minimize the use of bridge contracts in the future, if unresolved. Based on our analysis, we project that more than half (57 of 97) of Job Corps center contracts may need new contracts in program years 2021 and 2022, according to our analysis of FPDS-NG data and contract documentation (see fig. 8). Contracting officials expressed concerns about their capacity to conduct acquisition planning to award future center contracts given that two of six regions are currently without contracting officers, despite efforts to fill all vacant contracting officer positions. For the centers that we projected will need new contracts in program years 2021 and 2022, ETA will need to begin conducting acquisition planning relatively soon. According to contracting officials, acquisition planning and market research can take anywhere from 6 months to several years, depending on the requirement. Once these steps are completed, officials said it can take approximately 8 to 12 months from solicitation to award for new 5-year competitive procurements. Therefore, acquisition planning for a Job Corps center contract set to expire in January 2021 would need to begin before early 2020. We have previously reported that agencies have faced challenges allowing sufficient time to conduct acquisition planning, which can increase the risk that the government may receive services that cost more than anticipated, are delivered late, and are of unacceptable quality. According to the FAR, agencies should generally begin acquisition planning as soon as the agency need is identified, preferably well in advance of the fiscal year when the contract needs to be awarded to obtain timely services. The FAR also notes that the lack of advance planning is not a basis for justifying the use of other than full and open competition. Contracting officials said that finding ways to stagger Job Corps center contracts could help prevent a future procurement backlog. However, they had not received documented guidance from the national office on how to stagger center contracts to help mitigate this problem. In particular, national and regional contracting officials told us that one possibility for staggering center contracts is to decline to exercise option years. Officials in one region said that they are exploring this option, but noted it is still fairly uncommon for them not to exercise option years. GAO’s prior work emphasized the importance of comprehensive planning to ensure agencies effectively execute their missions and are accountable for results. Also, federal internal control standards state that agency leadership should anticipate and plan for significant changes by using a forward-looking process to identify risks that would affect its ability to achieve its objectives. Without a comprehensive strategy that considers when current center contracts will expire and how—or whether—Job Corps staff can effectively plan for and competitively award future center contracts, ETA is at increased risk of again having a backlog of center contracts to award competitively and, in turn, needing to use bridge contracts. Contracting officials said that filling vacant contracting officer positions in 2015 and 2016 was essential to reducing the procurement backlog of competitive contracts to operate Job Corps centers. By the end of program year 2016, ETA officials said contracting officers were in all six regions. However, at the time of our review, ETA was again without contracting officers in two of Job Corps’ six regions. According to officials, staff vacancies can create workload challenges. Each region is assigned one contracting officer who is responsible for awarding contracts for center operations, among other support contracts. Most of these contracting officers oversee 15 or more centers operated by contractors. When one of the six regional contracting officer positions has a vacancy, the contracting workload for that region is redistributed to other regions and the national office, which can have significant implications. For example, at the time of our review, national contracting officials told us that they were assisting the two regions where contracting officers had recently left. They said this increased their workload, as they had to attend to the contracting needs of these two regions while fulfilling their national contracting oversight duties. Similarly, contracting officials we spoke with in five regions noted vacancies in other positions that support the contracting process, such as those for contract specialists who provide support during the contracting process and program officials who provide technical expertise during proposal evaluations. In addition, program officials we spoke with during our site visits told us that some program manager positions have been vacant for at least a year in three regions. As a result, program officials said they have to manage and oversee additional centers to ensure coverage until those positions are filled. Further, past workforce assessments of ETA indicate that staff vacancies have been a longstanding challenge. For example, a 2013 study found that there were an insufficient number of program and contracting officials to efficiently and effectively handle the workload for Job Corps. Similarly, a 2014 assessment found that the Office of Job Corps, the Office of Contracts Management, and the Office of Financial Administration were understaffed to meet their missions. ETA officials said they have not developed a written acquisition workforce strategy to address staff vacancies for Job Corps. We have previously reported on the benefits of federal agencies planning strategically for their acquisition workforces, particularly for those agencies that rely heavily on contracting personnel with the necessary experience and skills to award and oversee complex contracts to accomplish their missions. In addition, our prior work has highlighted key components of agencies’ strategic workforce plans, including identifying gaps between current and needed workforce capabilities and developing strategies to meet these capabilities. Agency officials stated that DOL assesses and prioritizes needs across the agency when authorizing hiring actions, including for the Job Corps program. National and regional contracting officials told us that they have not been included in decisions regarding efforts to fill vacancies in critical contracting positions or to determine the number of contracting positions and the location of those positions (i.e., among the regions). ETA officials said that DOL has a new initiative to reorganize several functions across the agency, including potentially consolidating procurement functions. As previously discussed, ETA has restructured its contracting function twice over the past 6 years, consolidating contracting positions in the national office in 2013 and then moving them back to the regions in 2015. When asked about this new reorganization and how it might affect Job Corps procurements, DOL officials responded that they are in the planning phase, which is expected to conclude in the second half of fiscal year 2019. Officials commented that the goal of the reorganization is “to maximize DOL’s Federal buying power through effective procurement management.” According to officials, they plan to maintain a contracting office focused on supporting the Job Corps program. However, they did not provide additional information on the structure and location of this new Job Corps contracting office, or more specific time frames for when it would be established. It was unclear the extent to which the agency had evaluated how structural changes could affect its current contracting office and procurements, or whether they had consulted key stakeholders. GAO’s principles for effective strategic workforce planning emphasize the need to align an agency’s human capital program with its current and emerging mission and programmatic goals, and develop long-term strategies for acquiring, developing, and retaining staff to achieve those goals. Further, federal internal control standards state that agency leadership needs to demonstrate commitment to various workforce planning activities and determine the critical skills and competencies that will be needed to achieve key results. Without a comprehensive workforce strategy, ETA risks not having a sufficient number of trained acquisition personnel to ensure that it is able to adequately plan for and competitively award future center contracts as current center contracts expire. ETA has begun awarding fixed-price contracts for Job Corps center operations, which is a significant departure from the agency’s longstanding approach of using cost-reimbursement contracts, according to contracting officials. Under cost-reimbursement contracts, ETA pays allowable and reasonable costs incurred by the contractor to the extent prescribed by the contract. As of March 2019, ETA officials told us they had awarded 12 fixed-price contracts for Job Corps center operations. Officials said they did not have a timeline for transitioning other centers to fixed-price contracts for Job Corps center operations, but said that as center contracts expire, they will be reviewed to determine if a fixed-price contract would be appropriate. Regional contracting officials noted two primary advantages of using fixed-price contracts to operate Job Corps centers. First, they said fixed- price contracts reduce the government’s risk because the government pays only for work that meets specifications outlined in the contract. Second, regional officials said fixed-price contracts are easier to manage and administer compared to cost-reimbursement contracts because they are less administratively burdensome and require less oversight of contractor costs. For example, under cost-reimbursement contracts, regional program officials play a role in examining and approving contractor invoices to verify that they are allowable under the contract, and reasonable for the product or service identified. Under fixed-price contracts, contractors will have to demonstrate that they delivered on the contract or otherwise become subject to default, but program officials do not need to verify each expense to the same degree, according to regional and national contracting officials. ETA officials noted that the Office of Contracts Management provided training to program and contracting officials on the overall procurement process and the transition to fixed-price contracts to ensure they understood how to administer future contracts. ETA used various approaches to monitor contractor performance to ensure selected centers were operating appropriately and to encourage contractors to achieve certain program outcomes. These approaches included (1) risk-based center monitoring and (2) contract monitoring to hold contractors accountable. ETA primarily conducts two types of center assessments as part of the agency’s national risk-based monitoring strategy to identify emerging problems at Job Corps centers, including those operated by contractors. Regional office center assessments. ETA officials said they generally conduct unannounced visits to examine all aspects of center operations to ensure contractors comply with program requirements. For centers that operate for the full 5-year period of performance through a competitively awarded contract, these assessments are typically conducted twice over that time period. According to one regional director, these unannounced visits provide the opportunity to hear directly from Job Corps students and observe the conditions at the facilities. Program officials said that these visits are critical because some issues are not always apparent based on the data and reports they receive. For example, one program official said that during a center visit, she found questionable facility conditions at some student dormitories that had not been reported. Another program official said that during a center visit, she was able to observe the dynamics between students and center leadership and staff. Regional office targeted assessments. Regional program officials said they conduct onsite targeted unannounced assessments that typically focus on specific deficiencies that were identified as areas of concern in prior reviews or through other sources of information such as the student satisfaction survey. For example, contractor performance concerns could trigger this type of review. In particular, regional program officials said that center contractors who do not achieve national performance targets for student outcomes could be subject to a review. Following a center assessment, program officials prepare a report to summarize their findings and contractors may be required to submit and implement corrective action plans to address any deficiencies identified, according to Job Corps’ Policy and Requirements Handbook. Contractors who do not meet expected performance levels are placed on a performance improvement plan. According to some regional program officials, bridge contracts may lead to monitoring challenges. In particular, some regional program officials said that it is more difficult to address long-term challenges when centers operate under a bridge contract because the contract may only be in place for a few months while the procurement process for the next contract is underway. In some cases, they said the current contractor may not be operating the center by the time program officials conduct an assessment and issue their report. Program officials also noted that the short-term nature of bridge contracts can make it difficult for center contractors to recruit and retain high-quality staff. Some officials said that some program staff will look for a new job if they are uncertain whether a longer-term contract will be awarded. To monitor contractor performance, ETA used additional tools that generally reflect federal acquisition practices government-wide. Contractor performance assessments. ETA contracting and program officials are required to evaluate contractor performance annually and record the final assessment in the Contractor Performance Assessment Reporting System (CPARS). DOL, similar to other federal agencies, is required to use the system to document contractor performance. This system serves as a key source of information about the performance of Job Corps center contractors and includes ratings on their quality of service, management, and cost control. Based on our review of CPARS, we found that ETA completed annual contractor performance assessments during 2016- 2017 for all 10 Job Corps centers in our in-depth review. According to ETA’s guidance and program officials, these assessments can include information from regional monitoring visits and performance data on student outcomes and safety. Contract option years. Job Corps center contracts may be awarded for an initial term of no more than 2 years, with three 1-year options. For each option year, ETA has an opportunity to assess the contractor’s performance to determine whether to continue with the contract. ETA and regional officials said that they have typically exercised option years for Job Corps center contracts. However, in recent years, officials in one region said they have declined to exercise option years when questions are raised about a contractor’s performance. Officials said they are implementing provisions under WIOA that prohibit ETA from exercising an option year in a Job Corps center contract under certain circumstances. WIOA generally prohibits ETA from exercising an option year if, in the prior 2 program years, the center: (1) has been ranked in the lowest 10 percent of all Job Corps centers; and (2) did not achieve at least an average of 50 percent of its expected level of performance with respect to each primary performance indicator. ETA officials said that to date, every contractor has exceeded these minimum performance standards and, therefore, they have not had to decline an option year on these grounds. Formal notices to contractors. When ETA finds performance challenges, it may issue formal notices to contractors starting with a letter of concern to notify contractors of the deficiencies. If deficiencies are not addressed, a formal letter referred to as a cure notice may be sent to notify contractors that their failure to perform specific contract specifications may endanger the contract. If the contractor does not correct the condition, ETA may issue a notice (referred to as a “show cause”) informing the contractor that it intends to terminate the contract for default. DOL has indicated that it will terminate a contract for default if the contractor fails to satisfactorily address any serious performance challenges identified. None of the center contractors included in our in-depth review received a cure notice or a show cause notice from ETA during program year 2016. However, we found that ETA issued letters of concern to two center contractors in our in- depth review after it identified issues related to safety and student conduct. The letters of concern required the contractor to submit a corrective action plan and explain how it would address the areas of non-compliance identified by ETA, such as the presence of controlled substances at one of the centers. In the cost-reimbursement contracts for the 10 centers we reviewed, ETA generally included various incentive fees to encourage contractors to meet or exceed specific targets or technical goals, such as those for student achievement. Specifically, contracts for seven centers in our in- depth review included the following fees: Technical performance incentive fee. This fee is payable based on the contractor’s performance on specific outcome measures established by ETA, such as the number of students obtaining a high school diploma or high school equivalency. These fees varied but were up to 2.4 percent. One of the 10 contracts we reviewed received slightly over half of the incentive fee they were eligible to earn. Technical performance excellence bonus. This bonus is payable to top performing center contractors that exceed Job Corps’ national performance targets. Contractors can earn this bonus on top of the technical performance fee that they are eligible to earn. These fees varied but were up to 0.6 percent. While all of the contracts we reviewed included this provision for program years 2016 and 2017, we found that only one of the contractors received it. Cost incentive fee. This fee is payable based on the contractor’s efforts to meet the government’s needs within the estimated cost of the contract. Contractors can earn higher fees by completing the work at a lower cost. The fees received varied from 3 percent to 4 percent. For example, in program year 2017, contracts for four of the centers we reviewed included cost incentive fees. Two contractors received the maximum fee of 4 percent, while the other two contractors received a fee of at least 3 percent, according to the fee information provided by ETA. For bridge contracts, ETA officials said that they did not include incentive fees, given the intended short-term nature of these contracts. Instead, they said they included fixed fees, which do not vary based on actual costs or performance. In our in-depth review, we found that seven centers that had noncompetitive bridge contracts in program years 2016 or 2017 included only fixed fees that were paid regardless of contractor performance. While each contract we reviewed included estimates of how much a contractor might earn in technical incentive fees, the final amount paid by ETA was determined by whether the contractor met or exceeded Job Corps’ national center performance targets, which ETA shares with the Job Corps community. ETA officials noted that performance targets can vary from year to year based on the national goals of the program. As a result, they said a contractor with the same performance in two years, as measured by ETA performance targets, may qualify for a technical incentive fee in one year but not in another. Contracting and program officials at the national and regional levels with contract oversight responsibilities reported having limited or no insight into how contractors earn incentive fees to operate Job Corps centers, despite the critical role these fees can play in motivating contractor performance. During our interviews, program and contracting officials said they were unaware of how the final fee amounts were calculated, and noted that ETA’s Office of Financial Administration is currently responsible for making these determinations. In particular, some contracting officials said that they simply execute the contract actions calculated and approved by ETA’s Office of Financial Administration. Because of their limited insight, some program officials said that it is difficult for them to address questions from contractors about how fees are calculated. National officials from ETA’s Office of Financial Administration expressed concern and said they were somewhat surprised that program and contracting officials told us that they were unaware of how contractor fees were determined and calculated. ETA officials said that budget analysts currently perform the fee calculations in a worksheet, which is later reviewed by their supervisor, and that ETA officials expected program and contracting officials to be familiar with the process. Officials from the Office of Financial Administration provided the fee calculations for the centers in our in-depth review, and noted that Job Corps’ Policy and Requirements Handbook includes some publicly-available information about fee calculations. However, at the time of our review, the Office of Financial Administration had not developed an internal documented process to share information about its fee calculations on specific Job Corps center evaluations with program and contracting officials. Further, in one region, program officials monitoring contractors described what they see as a potential disconnect between the incentive fees paid to a Job Corps center contractor and the contractor’s performance assessment. In this case, two contractors were paid an incentive fee for meeting performance targets, and received a “marginal” rating on an annual performance assessment, according to the program official monitoring the contractors. Without a coordinated and documented process, program and contracting officials may continue to have a limited awareness of how incentive fees are earned by contractors. In 2009, OFPP developed guidance that states incentive strategies should be developed through close collaboration among the contracting officer, program officials, and other key staff. Further, federal internal control standards state that agency leadership should document operational processes in policies, and communicate these policies to key personnel so that they can implement their assigned responsibilities. The questions raised by program and contracting officials in our discussions about how Job Corps contracts’ incentive fees are structured and related to certain outcomes increases the risk that ETA, including its contracting and program officials, will miss opportunities to maximize the use of incentives to help monitor and improve the performance of center contractors. Contracts are key means through which ETA secures operators for Job Corps centers across the country and delivers comprehensive services to Job Corps students. ETA has implemented some strategies to address the contracting challenges that led to the widespread use of bridge contracts during program year 2016. While bridge contracts can be a useful tool to ensure that there is no lapse in services provided to Job Corps students, our work has found that when noncompetitive bridge contracts are used frequently or for prolonged periods of time, the government is at risk of paying more than it should for products and services. Further, ongoing acquisition planning and workforce challenges, which our work has found are associated with the use of bridge contracts, could pose risks to its ability to manage and award future Job Corps contracts in a way that avoids a reliance on bridge contracts in the future. Further, ETA’s efforts to reduce its reliance on bridge contracts in program year 2016—a step in the right direction—may result in an unintended consequence later down the road. Specifically, we project that more than half of the recently awarded competitive contracts may expire and services will need to be re-solicited in program years 2021 and 2022. A comprehensive strategy that accounts for Job Corps’ current and future workload could help ETA better anticipate its workforce needs in critical positions, and thereby helping to reduce its risk of relying on bridge contracts in the future. In the absence of such a strategy, ETA is likely to be back in the same position it was 3 years ago, when more than two- thirds of its Job Corps centers were operating under some form of bridge contract. ETA used various monitoring and contracting tools, including incentive fees, to encourage Job Corps center contractors to meet or exceed performance outcomes. However, contracting and program officials we spoke with were not aware of how these incentive fees had been calculated and paid. Additionally, ETA’s Office of Financial Administration had no documented process for sharing information with ETA’s program and contracting officials about the calculation and payment of these fees or how a contractor’s performance impacted these fees. In the absence of a coordinated and documented process, program and contracting officials may lack key information regarding contractor performance. We are making the following two recommendations to ETA: The Assistant Secretary of ETA should develop, document, and implement a comprehensive strategy that (1) accounts for Job Corps’ projected workload requirements and (2) considers its acquisition workforce needs—including the number of staff, skills, and other supports necessary to plan, award, and monitor Job Corps center contracts—to enable it to effectively plan for and competitively award future Job Corps center contracts. (Recommendation 1) The Assistant Secretary of ETA should develop a coordinated and documented internal process to share relevant information on incentive fees paid to contractors with staff in its key offices. (Recommendation 2) We provided a draft of this report to DOL for its review and comment. We received written comments from DOL, which are reprinted in appendix IV. In addition, DOL provided technical comments which we incorporated as appropriate. DOL concurred with our two recommendations. DOL stated that it will develop, document, and implement a comprehensive strategy that accounts for Job Corps’ projected workload requirements and considers its acquisition workforce needs. DOL noted that it has released a new procurement plan which reflects its decision to re-procure 28 Job Corps centers prior to the final option year of their contract. DOL said that this action would result in each region having no more than five procurements each year, which it considers a manageable procurement workload for its current staffing level. DOL also stated that it would develop a written process for determining and awarding incentive fees to Job Corps contractors. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees and the Secretary of Labor. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Cindy S. Brown Barnes at (202) 512-7215 or brownbarnesc@gao.gov, or Timothy J. DiNapoli at (202) 512-4841 or dinapolit@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report examines (1) the extent to which, and why, the Employment and Training Administration (ETA) used bridge contracts to operate Job Corps centers during program year 2016; (2) the strategies ETA used to decrease its use of noncompetitive bridge contracts; and (3) how ETA monitored contractor performance at selected Job Corps centers. To address these three objectives, we used several data collection methods, which are described in greater detail below. These methods include analyzing data from the Federal Procurement Data System-Next Generation (FPDS-NG), conducting a nongeneralizable review of 10 Job Corps centers that operated under bridge or noncompetitive contracts, and conducting interviews with ETA regional and national officials. In addition, we reviewed relevant federal laws and regulations, and agency policies and procedures such as Job Corps’ Policy and Requirements Handbook, the Acquisition Handbook for Job Corps Regional Contracts, and other information ETA provided related to incentive fees and the number of staff vacancies and protests filed in program years 2012 to 2016. We also reviewed ETA’s evaluations of contractor performance, and past GAO reports on the use of bridge and noncompetitive contracts, and the evaluation of contractor performance. To identify the extent to which ETA used bridge contracts to operate Job Corps centers, we analyzed FPDS-NG data for center contracts that were in effect—that is, contracts that were newly awarded or ongoing—in program year 2016. We selected this program year because it reflected the most recent year with complete available data at the time we began our review. We did not review data for centers operated by the U.S. Department of Agriculture (USDA) because they are operated through an interagency agreement between DOL and USDA and are therefore not relevant for the purpose of this review. We also used FPDS-NG data to identify centers that appeared to have operated under bridge contracts at some point during program year 2016. Since there is no government-wide definition for bridge contracts and ETA does not have a documented definition, we used GAO’s definition that has defined them as an extension to an existing contract beyond the period of performance (including base and option years), or a short-term stand-alone contract awarded to an incumbent contractor to avoid a lapse in service. We acknowledge that in the absence of a government-wide definition, agencies may have differing views of what constitutes a bridge contract. Contracts and extensions (both competitive and non-competitive) are included in GAO’s definition for bridge contracts. While ETA does not consider contracts that exercise the “Option to Extend Services” under Federal Acquisition Regulation (FAR) 52.217-8 to be bridge contracts, we include these contracts because our definition is focused on the intent of the contracts or extensions—that is, whether they serve as a mechanism to “bridge services” until the next follow-on contract can be competitively awarded. Based on our definition, we identified 68 centers that operated under bridge contracts in program year 2016. We verified our contract selections with ETA officials to ensure we identified all centers contracts that were in effect—that is, contracts that were newly awarded or ongoing—in program year 2016. We also reviewed relevant contracting documentation, such as justification and approval documents for noncompetitive contracts and contract modifications. To calculate the length of time ETA used bridge contracts to operate Job Corp centers, we included those centers that had a bridge contract at some point during program year 2016. We report the length of time that ETA used bridge contracts to operate Job Corps centers as the minimum amount of time these contracts were in use. We did not review bridge contracts that were completed prior to program year 2016 because it was outside the scope of our review. Therefore, our analysis may underestimate the length of time ETA operated some centers under bridge contracts. Based on our electronic testing, review of contract files and documentation, and discussions with ETA officials, we determined that the data were sufficiently reliable for the purposes of assessing ETA’s use of bridge contracts for Job Corps center operations, and the characteristics of these contracts. To estimate upcoming center procurements from program years 2019 to 2023, we used FPDS-NG data and information from agency officials to determine when the period of performance might end for certain center contracts. In this analysis, we excluded centers that were still operating under noncompetitive bridge contracts, operating under task orders, or were no longer open. Competitively awarded Job Corps center contracts generally have periods of performance that total a maximum of 5 years, which includes a 2-year base and three 1-year options. GAO’s analysis accounts for this complete period of performance; however, if all three option years are not exercised, the center would need a new contract sooner. To identify the strategies that ETA used to decrease its use of noncompetitive bridge contracts, we reviewed FPDS-NG data to identify the number of the bridge contracts ETA used in program year 2016 that transitioned to competitive follow-on contracts by the end of program year 2017. We also reviewed agency guidance and contracting documentation, and followed up with ETA contracting and program officials at the national and regional levels to verify our contract selections. We conducted a nongeneralizable in-depth review of 10 Job Corps centers that operated under bridge or noncompetitive contracts during program year 2016 to provide illustrative examples. The 10 centers we selected were Alaska, Carville, Cassadaga, Keystone, Milwaukee, Northlands, Paul Simon, Pinellas, Turner, and Woodland. We selected these 10 centers because (1) they were operated by contractors with varying levels of success in achieving ETA’s student performance indicators, according to ETA’s performance data, and (2) to ensure we included at least one center from each of Job Corps’ six regions. Specifically, we selected 6 of the 10 Job Corps centers because they were generally the lowest performing contract center in their region based on ETA’s performance data from program year 2015. We reviewed performance data for this program year because it allowed us to identify the actions, if any, ETA took to help improve low performing centers in program years 2016 and 2017. The other four Job Corps centers were randomly selected from the remaining Job Corps centers, which reflected a mix of center performance levels. We excluded from our selection centers that were not operational or were closed in program years 2016 or 2017, operated under a task order, or that had an open protest as of June 30, 2018. In addition, we excluded centers with a competitive, non- bridge contract, and centers operated by the U.S. Department of Agriculture. After selecting the 10 centers, we reviewed the contract file for all bridge contracts, the contract preceding the bridge contracts, and, if awarded by the time of our review, the competitive follow-on contract. We also interviewed contracting and program officials to understand the reasons why ETA used bridge contracts and any challenges related to their use. In addition, we obtained and reviewed ETA’s evaluations of contractor performance from the Contractor Performance Assessment Reporting System (CPARS) for these centers to understand how ETA monitored contractor performance. We also examined other information related to incentive fees paid to contractors for the 10 centers in our in-depth review. The results of our in-depth review provide insight into ETA’s contracting practices for Job Corps center operations contracts but cannot be generalized to all Job Corps centers. We conducted site visits to three of Job Corps’ six regional offices: Atlanta, Boston and Dallas. We selected these offices to capture the regions that awarded a large number of bridge or noncompetitive contracts, and to reflect both geographic diversity and a mix of contractor performance. For the remaining three regions—Chicago, Philadelphia, and San Francisco—we conducted phone interviews. For each regional visit or call, we interviewed program officials in the Office of Job Corps, including the regional director and program managers (who may serve as contracting officer representatives). In addition, we interviewed regional contracting officials in the Office of Contracts Management, including the regional contracting officer and contract specialists who support the contracting officer in carrying out their responsibilities. Additionally, we interviewed national officials in ETA’s Office of Job Corps and Office of Contracts Management to better understand ETA’s process for awarding and monitoring Job Corps center contracts at the national level. We also interviewed budget officials in ETA’s Office of Financial Administration to better understand how incentive fees are calculated and paid to contractors. We conducted this performance audit from February 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In January 2015, ETA national and regional officials implemented a national risk-based monitoring strategy to identify emerging problems at Job Corps centers, including those operated by contractors. We reported on this strategy as part of our prior work. Table 2 provides a summary of ETA’s monitoring strategy. In addition to the contact named above, Mary Crenshaw (Assistant Director), Janet McKelvey (Assistant Director), Ashanta Williams (Analyst- in-Charge), Anna Blasco, LaToya Jeanita King, Matthew Saradjian, Lindsay Taylor, Tomás Wind, and Jocelyn Yin made key contributions to this report. Additional assistance was provided by Sandra Baxter, James Bennett, Sarah Cornetto, Caitlin Croake, Andrea Dawson, David Forgosh, Lauren Gilbertson, Kurt Gurka, Julia Kennon, Sheila R. McCoy, Corinna Nicolaou, Monica Savoy, Ben Sinoff, Kathleen van Gelder, Almeta Spencer, Walter Vance, and Alyssa Weir. Information Technology: Agencies Need Better Information on the Use of Noncompetitive and Bridge Contracts. GAO-19-63. Washington, D.C.: December 11, 2018. Job Corps: DOL Could Enhance Safety and Security at Centers with Consistent Monitoring and Comprehensive Planning. GAO-18-482. Washington, D.C.: June 15, 2018. Defense Contracting: Use by the Department of Defense of Indefinite- Delivery Contracts from Fiscal Years 2015 through 2017. GAO-18-412R. Washington, D.C.: May 10, 2018. New Trauma Care System: DOD Should Fully Incorporate Leading Practices into Its Planning for Effective Implementation. GAO-18-300. Washington, D.C.: March 19, 2018. Defense Contracting: DOD Needs Better Information on Incentive Outcomes. GAO-17-291. Washington, D.C.: July 11, 2017. Job Corps: Preliminary Observations on Student Safety and Security Data. GAO-17-596T. Washington, D.C.: June 22, 2017. Federal Contracts: Agencies Widely Used Indefinite Contracts to Provide Flexibility to Meet Mission Needs. GAO-17-329. Washington, D.C.: April 13, 2017. Elections: DOD Needs More Comprehensive Planning to Address Military and Overseas Absentee Voting Challenges. GAO-16-378. Washington, D.C.: April 20, 2016. Defense Acquisition Workforce: Actions Needed to Guide Planning Efforts and Improve Workforce Capability. GAO-16-80. Washington, D.C.: December 14, 2015. Sole Source Contracting: Defining and Tracking Bridge Contracts Would Help Agencies Manage Their Use. GAO-16-15. Washington, D.C.: October 14, 2015. Federal Construction Subcontracting: Insight into Subcontractor Selection Is Limited, but Agencies Use Oversight Tools to Monitor Performance. GAO-15-230. Washington, D.C.: January 29, 2015. Job Corps: Assessment of Internal Guidance Could Improve Communications with Contractors. GAO-15-93. Washington, D.C.: January 22, 2015. Standards for Internal Control in the Federal Government. GAO-14-704G. Washington, D.C.: September 2014. Market Research: Better Documentation Needed to Inform Future Procurements at Selected Agencies. GAO-15-8: Washington, D.C.: October 9, 2014. Contractor Performance: Actions Taken to Improve Reporting of Past Performance Information. GAO-14-707. Washington, D.C.: August 7, 2014. Federal Contracting: Noncompetitive Contracts Based on Urgency Need Additional Oversight. GAO-14-304: Washington, D.C.: March 26, 2014. Acquisition Workforce: Federal Agencies Obtain Training to Meet Requirements but Have Limited Insight into Costs and Benefits of Training Investment. GAO-13-231. Washington, D.C.: March 28, 2013. Defense Contracting: Competition for Services and Recent Initiatives to Increase Competitive Procurements. GAO-12-384. Washington, D.C.: March 15, 2012. Acquisition Planning: Opportunities to Build Strong Foundations for Better Service Contracts, GAO-11-672. Washington, D.C.: August 9, 2011. Federal Contractors: Better Performance Information Needed to Support Agency Contract Award Decisions. GAO-09-374. Washington, D.C.: April 23, 2009. 2010 Census: Census Bureau generally Follows Selected Leading Acquisition Planning Practices, but Continued Management Attention is Needed to Help Ensure Success. GAO-06-277. Washington, D.C.: May 18, 2006. Defense Acquisitions: DOD Has Paid Billions in Award and Incentive Fees Regardless of Acquisition Outcomes, GAO-06-66. Washington, D.C.: December 19, 2005. Human Capital: Framework for Assessing the Acquisition Function at Federal Agencies. GAO-05-218G. Washington, D.C.: September 2005. Human Capital: Key Principles for Effective Strategic Workforce Planning. GAO-04-39. Washington, D.C.: December 11, 2003.", "summary": "Job Corps' 119 centers, which are operated primarily by contractors, provide an array of services to help low-income youth find a job, go to college, or enter the military. ETA is generally required to award competitive contracts, but can award noncompetitive contracts in certain instances. Some noncompetitive contracts act as bridge contracts—which can be a useful tool to avoid a lapse in service but, when used frequently and for prolonged periods, can increase the risk of the government overpaying for services. This report examines (1) the extent to which ETA used bridge contracts to operate Job Corps centers in program year 2016; (2) strategies ETA used to decrease the use of noncompetitive bridge contracts; and (3) how ETA monitored contractor performance at selected Job Corps centers. GAO analyzed data from program years 2016 and 2017(the most current data available at the time we began our review) from the Federal Procurement Data System-Next Generation, and reviewed contract documents. GAO also conducted an in-depth review of 10 centers that reflected a mix of contractor performances and at least one center from Job Corps' six regions, and interviewed ETA officials. In program year 2016, the Department of Labor's (DOL) Employment and Training Administration (ETA) operated 68 of its 97 Job Corps centers using bridge contracts. GAO has generally defined a bridge contract as an extension to an existing contract or a new noncompetitive contract awarded to the current contractor to avoid a lapse in service. GAO found that ETA operated most of these Job Corps centers (49 of 68) under bridge contracts for at least a year, with over a third operating under bridges for 2 years or potentially longer. ETA cited workforce challenges such as staff vacancies and the need to address issues raised in protests as contributing to its use of bridge contracts. ETA officials said they used various strategies to decrease their use of noncompetitive bridge contracts, including prioritizing efforts to award more contracts competitively. By the end of program year 2017, most of the centers operating under bridge contracts during program year 2016 (48 of 68) had transitioned to competitive contracts. Despite these efforts, ETA continues to face workforce challenges. Contracting officials expressed concern about having sufficient staff to award a large group of contracts that will begin to expire in program years 2021 and 2022 (see figure). ETA officials said it takes about 8 to 12 months from solicitation to contract award for new 5-year competitive procurements. Therefore, acquisition planning for a center contract set to expire in January 2021 would usually need to begin early 2020. However, ETA does not have a comprehensive workforce strategy to address its workforce challenges or support these new contract awards. As a result, ETA risks relying on noncompetitive bridge contracts again in the future. Note: Centers are operated on a program year basis, which runs from July 1 of a given year to June 30 of the following year. ETA used various strategies to monitor and incentivize contractor performance at the 10 centers GAO reviewed, including conducting onsite visits to Job Corps centers and paying incentive fees to contractors. However, contracting and program officials GAO interviewed had limited or no insight into how ETA calculates and pays incentive fees. Without coordinating and documenting the process for calculating incentive fees, ETA's program and contract officials may lack key information regarding contractor performance. GAO is making two recommendations, including that ETA develop (1) a comprehensive strategy to account for workforce needs and future center contracts, and (2) a coordinated and documented process for sharing information on incentive fees paid to contractors. DOL agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Enrollment is generally the first step veterans take to obtain health care services, within VA or through community care. VA’s Health Eligibility Center manages the process of accepting applications, verifying eligibility, and determining enrollment, in collaboration with VA medical centers. VA requires veterans’ enrollment applications be processed within 5 business days of receipt, including pending applications that require additional information from the applicant to process. Once enrolled, veterans can access VA health services by scheduling an appointment. VA’s scheduling policy establishes the procedures for scheduling medical appointments, as well as sets the requirements for staff directly or indirectly involved in the scheduling process (e.g., training). A scheduler at the VA medical facility is responsible for making appointments for new and established patients (i.e., patients who have visited the same VA medical center in the previous 24 months), which are then recorded in VA’s electronic scheduling system. VA scheduling policy requires patients who have requested an appointment and have not had one scheduled within 90 days to be placed on VA’s electronic wait list. VA determines wait times at each facility based on outpatient appointment information from its scheduling system. VA is required to publish information on appointment wait times at each VA medical facility for primary care, specialty care, and hospital care and medical services, which it does through two public websites. In November 2014, VA began posting monthly wait times for scheduling appointments at all VA medical facilities. One public website provides links to spreadsheets containing data for each VA medical facility, such as the average wait times for primary, specialty, and mental health care appointments and the number of patients on the electronic wait list. In April 2017, VA created a second public “Access and Quality in VA Healthcare” website to post both patient access data and information on VA medical facilities’ performance on various quality metrics. This website aims to help veterans find wait times at a specific facility. This information would allow veterans and their family members to use the wait-time data on this website to determine the best option for obtaining timely care. In order to receive needed care in a timely manner, veterans may need to obtain care outside of VA medical facilities through one of VA’s community care programs. VA has purchased health care services from community providers through various community care programs since 1945. Veterans may be eligible for community care when they are faced with long wait times or travel long distances for appointments at VA medical facilities, or when a VA facility is unable to provide certain specialty care services. Since 2014, Congress has taken steps to expand the availability of community care for veterans. The Veterans Access, Choice, and Accountability Act of 2014 provided up to $10 billion in funding for veterans to obtain health care services from community providers. The law established a temporary program—called the Veterans Choice Program (Choice Program)—to offer veterans the option to receive hospital care and medical services from a community provider when a VA medical facility could not provide an appointment within 30 days, or when veterans resided more than 40 miles from the nearest VA facility or faced other travel burdens. VA contracted with two third-party administrators (TPA) to establish networks of community providers, schedule veteran appointments with those providers, and pay those providers for services rendered through the Choice Program. In June 2018, the VA MISSION Act of 2018 was enacted to further address some of the challenges faced by VA in ensuring timely access to care. The Act required VA to implement within 1 year a permanent community care program—the Veterans Community Care Program (VCCP). The act identified criteria that all veterans enrolled in the VA health care system would be able to qualify for care through the VCCP; for example, if VA does not offer the care or service needed by the veteran or VA cannot provide the veteran with care and services that comply with its designated access standards. The access standards include appointment wait times for a specific VA medical facility; for example, veterans may be eligible for care through the VCCP if VA cannot provide care within 20 days for primary and mental health care, and 28 days from the date of request for specialty care, unless veterans agree to a later date in consultation with their VA health care provider. VA has taken a number of actions to address our recommendations regarding deficiencies we found in wait-time measurement and implementation of its scheduling policy. For wait-time measurement, these actions included changes to the wait-time measurement definitions, provision and documentation of scheduler training, and improved oversight through audits, all of which have been in a state of flux for the past 6 years. On July 12, 2019, VA provided us additional updates on efforts to implement our related recommendations. This new information fully addresses one of our recommendations. In December 2012, we found that outpatient medical appointment wait times reported by VA were unreliable, and, therefore, VA was unable to identify areas that needed improvement or mitigate problems for veterans attempting to access care. VA typically has measured wait times as the time elapsed between the ‘start date’—a defined date that indicates the beginning of the measurement—and the ‘end date’, which is the date of the appointment. At the time of our 2012 report, VA measured wait times as the number of days elapsed from the start date identified as the desired date—the date on which the patient or health care provider wants the patient to be seen—to the date of the appointment. We found that the reliability of the reported wait-time measures was dependent on the consistency with which schedulers recorded the desired date in the scheduling system, as required by VA’s scheduling policy. However, VA’s scheduling policy and training documents for recording the desired date were unclear and did not ensure consistency. We observed that not all schedulers at VA medical centers that we visited recorded the desired date correctly. Therefore, we recommended that VA either clarify its scheduling policy to better define the desired date, or identify clearer wait- time measures that are not subject to interpretation and prone to scheduler error. VA concurred with the recommendation, which we have identified as among those recommendations that warrant priority attention. Actions VA has taken or is taking to address this recommendation include: changes to the start date and definitions for wait-time measurement, provision and documentation of scheduler training, and improved oversight through scheduler audits. In addition, we are currently assessing new information VA provided in July 2019, which will include obtaining additional evidence and clarification from VA to see whether it has fully addressed our concerns. VA’s Actions to Change Start Dates for Wait-Time Measurement While the terminology for the start dates of the wait-time measurement has changed several times over the past 6 years, we believe that the current definitions of the start dates are substantively the same as those we reviewed—and found to be deficient—in our 2012 report. VA subsequently introduced new terms with similar definitions—from “desired date” to “preferred date”—without fundamentally addressing the deficiency. See table 1 for the changes to and definitions of the start dates for measuring outpatient appointment wait times and wait-time goals since June 2010. As table 1 shows, for new patients and established patients seeking appointments without a return-to-clinic date specified by their provider, VA changed the terminology of the start date to preferred date in its July 2016 scheduling policy from what it had established in its June 2010 policy. However, the definition of preferred date is substantively the same as the definition of desired date in the previous scheduling policy, the latter of which we found to be subject to interpretation and prone to scheduler error in our 2012 report. We continue to believe that the preferred date is also subject to interpretation and prone to scheduler error, which poses concerns for the reliability of wait times measured using the patient’s preferred date. In its updated July 2016 scheduling policy, VA also changed the terminology of the start date to the “clinically indicated date” for established patients whose provider has documented a clinically appropriate return-to-clinic date in the patient’s electronic health record. The clinically indicated date is substantively the same as the definition of desired date for established patients in the previous scheduling directive. While VA has not clarified the definitions of start dates, VA has taken actions intended to improve the accurate recording of the clinically indicated date in three ways: 1. VA requires clinical leadership (such as the Associate Chief of Staff) at each VA medical facility to ensure that providers enter the clinically indicated date in the electronic health record for future appointments; 2. VA standardized the entry of the clinically indicated date in the electronic health record to improve the accuracy of the date, which was implemented across all VA medical facilities as of July 2018; and 3. VA created a technology enhancement to enable the automatic transfer of the clinically indicated date from the electronic health record to the scheduling system. As a result, the scheduler no longer has to retrieve the date from veterans’ electronic health records and manually enter it into the scheduling system. VA reported that this enhancement was implemented at all but three VA medical facilities as of January 2019. In July 2019, VA reported to us that the error rate for the patient indicated date (either the clinically indicated date, or in the absence of that date, the patient’s preferred date) was 8 percent of about 667,000 appointments audited in the most recent biannual audit cycle, ending March 31, 2019. VA cites an almost 18 percent improvement in reducing the number of errors caused by manual entry of the clinically indicated date due to the use of the technology enhancements. VA’s Actions to Provide and Document Scheduler Training Although VA updated its scheduling policy in 2016, we believe the instructions, which form the basis for wait-time measurement, are still subject to interpretation and prone to scheduler error, making training and oversight vital to the consistent and accurate implementation of the policy. VA reported that 97 percent of all staff who scheduled an appointment within 30 days completed the required scheduling training as of July 2, 2019. VA stated that the department will closely monitor compliance with scheduler training completion for the remaining staff. Given the high turnover among schedulers, it is important that VA remain vigilant about scheduler training, ensuring all who need it receive it. VA’s Actions to Improve Oversight through Scheduler Audits VA has taken a number of actions to improve oversight of the scheduling process through biannual scheduling audits at VA medical centers and second level audits, as well as completion of the first system-wide internal audit of scheduling and wait-time data. Biannual scheduler audits. VA’s July 2016 scheduling policy required biannual audits of the timeliness and appropriateness of schedulers’ actions and accuracy of entry of the clinically indicated date and preferred date, the start dates of wait-time measurement as identified by the revised scheduling policy. In June 2017, VA deployed a standardized scheduling audit process for staff at VA medical centers to use. As part of our recommendation follow-up in July 2019, VA reported 100 percent completion of the required biannual scheduling audits in fiscal year 2018. As noted above, VA reported to us that the error rate for the patient indicated date (either the clinically indicated date, or in the absence of that date, the patient’s preferred date) was 8 percent of about 667,000 appointments audited. While VA asserts that errors in the clinically indicated date have decreased, an error rate of 8 percent still yields errors in more than 53,000 appointments audited. Given these errors, we remain concerned about the reliability of wait times measured using preferred date (one part of the patient indicated date), and have requested additional information from VA about these errors. Second level scheduler audits. In November 2018, VA implemented a second-level scheduling audit (Audit the Auditors program), which is overseen by the VA integrated service networks tasked with oversight of VA medical facilities within their regions. Each medical center within a network region is paired with another medical center and they audit each other’s scheduling audit. Throughout the cycle, medical centers share their findings with each other and the network. The goal is to standardize scheduling audit practices across the network and to ensure reliability of the scheduler audit results. According to VA, the first cycle was completed April 30, 2019, by all VA medical centers. First internal system-wide audit of wait-time data and scheduling. In its first internal audit completed in August 2018, VA was unable to evaluate the accuracy and reliability of scheduling and the wait-time data. Specifically, VA was unable to determine the accuracy and reliability of the scheduling and wait-time data, databases, and data flow from the electronic health record and scheduling system to the VA Access and Quality website because they were not able to obtain the rules for calculating wait times. Given our continued concerns about VA’s ability to ensure the reliability of the wait-time data, we plan to obtain additional information from VA about its methodology and assessment of evidence underlying the audit findings. In December 2012, we also found inconsistent implementation of VA’s scheduling policy that impeded VA medical centers’ scheduling of timely medical appointments. Specifically, we found that not all of the clinics across the medical centers we visited used the electronic wait list to track new patients that needed medical appointments as required by VA’s scheduling policy, putting these patients at risk of being lost for appointment scheduling. Furthermore, VA medical centers’ oversight of compliance with VA’s scheduling policy, such as ensuring the completion of required scheduler training, was inconsistent across facilities. Scheduler training was particularly important given the high volume of staff with access to the scheduling system—as of July 2, 2019, VA reported there were approximately 33,000 staff that had scheduled an appointment within the last 30 days. We also found that VA medical centers identified the outdated and inefficient scheduling system as one of the problems that can impede the timely scheduling of appointments and may impact their compliance with VA’s scheduling policy. We recommended VA ensure that VA medical centers consistently and accurately implement VA’s scheduling policy, including use of the electronic wait list, as well as ensuring that all staff with access to the scheduling system completes the required training. VA concurred with this recommendation, which we also have identified as among those recommendations that warrant priority attention. VA’s actions to improve implementation of the scheduling policy, including updated information VA provided in July 2019, fully addresses this recommendation. VA issued an updated scheduling policy in July 2016 that provided clarification on scheduling roles and responsibilities for implementing the policy and business rules for scheduling appointments, such as using the electronic wait list, and required biannual scheduler audits. VA also ensured almost all schedulers received training on the updated scheduling policy and improved oversight through audits, as previously described. In addition, VA plans to rapidly deploy a single nationwide scheduling system that is intended to simplify the operating environment for schedulers and may mitigate challenges identified in our 2012 report. The new scheduling system will be a resource-based system where each provider’s schedule is visible on one screen, instead of requiring the need to toggle through multiple screens as it currently exists. VA plans to roll out the new scheduling system starting in 2020, which is expected to be implemented in coordination with the planned modernization of the electronic health records system across VA facilities. According to VA, the scheduling system will be available for use in advance of the completion of the electronic health record implementation at some sites. In addition to the recommendations we made to improve VA’s wait-time data and implementation of its scheduling policy, we have also made recommendations to address other factors that affect the timeliness by which veterans obtain appointments. These recommendations have targeted VA’s enrollment processes and its management of veterans’ initial requests for care. While VA has taken some steps to address these recommendations, they have not yet been fully addressed. For example, we have found that VA’s wait-time measures do not yet capture the time it takes the agency to enroll veterans in VA health care benefits, or manage a veterans’ initial request for care. In September 2017, we found that VA did not provide its medical centers, who historically receive 90 percent of enrollment applications, with clear guidance on how to resolve pending applications, which led to delays in veteran’s enrollment. For example, we found instances in which pending applications remained unresolved for more than 3 months. We concluded these delays in resolving pending applications, along with previously documented delays due to errors in enrollment determinations, may result in veterans facing delays when obtaining health care services or incorrectly denied benefits. We made several recommendations to address these deficiencies, two of which we determined to be priority recommendations for VA to clearly define roles and responsibilities for (1) resolving pending applications and (2) overseeing the enrollment process. VA has made progress in addressing these priority recommendations by beginning to update, but not yet finalizing, its policies, procedures, and guidance on enrollment processing. In 2017, VA’s Health Eligibility Center began conducting secondary reviews of enrollment determinations. However, in fiscal year 2018, Health Eligibility Center staff found that 18 percent of rejected enrollment determinations and 8 percent of ineligible enrollment determinations that underwent secondary reviews were incorrect. These recommendations remain unimplemented as of July 2019. Once enrolled, we have found that VA’s management of veterans’ initial request for care have led to delays; and although VA has clarified timeliness requirements, it has yet to fully capture the wait veterans experience in scheduling initial appointments. In a number of reports from 2015 to 2018, we found instances in which newly enrolled veterans were not contacted to schedule initial primary care appointments, and did not complete initial primary care appointments and mental health evaluations according to VA timeliness requirements. These delays may be understated in VA data, because VA’s wait-time measures do not take into account the time it takes VA medical center staff to contact the veteran to determine a preferred date (the starting point for wait-time measurement) from the veteran’s initial request or referral. We found that the total amount of time it took for veterans to be seen by providers was often much longer when measured from the dates veterans initially requested to be contacted to schedule an appointment or were referred for an appointment by another provider than when using the veterans’ preferred dates as the starting point. See figure 1 for an example of how the two wait-time calculations differ for an initial primary care appointment. We made several recommendations to VA, including a priority recommendation to monitor the full amount of time newly enrolled veterans wait to be seen by a provider. VA has taken several steps to address the priority recommendation, including revising an internal report to help identify and document newly enrolled veterans and monitor their appointment request status. The report is intended to help VA and its medical centers oversee the enrollment and appointment process by tracking the total time from application to appointment. However, VA is still in the process of enhancing its electronic enrollment system to capture the application date for all newly enrolled veterans. Until the enhancements are implemented, VA may not consistently capture the start date for newly enrolled veterans, which, in turn, affects the reliability of its wait-time data. The priority recommendation remains unimplemented as of July 2019. VA has not implemented several of our recommendations related to the Choice Program that could impact veterans’ timely access to care under the VCCP. These recommendations address (1) establishing achievable community care wait-time goals and a scheduling process consistent with those goals, (2) collecting accurate and complete data to systematically monitor veteran community care wait times, and (3) other factors that could adversely affect veterans’ access to community care. VA has begun taking steps to address these recommendations as it implements the VCCP. Our review of the Choice Program in June 2018 found that despite having a wait-time goal, VA developed a scheduling process for the Choice Program that was not consistent with achieving that goal. The Veterans Access, Choice, and Accountability Act of 2014 required VA to ensure the provision of care to eligible veterans within 30 days of the clinically indicated date or, if none existed, within 30 days of the veteran’s preferred date. However, we found that those veterans who were referred to the Choice Program for routine care because services were not available at VA in a timely manner could potentially wait up to 70 calendar days for care. Under VA’s scheduling processes, this potential wait time included VA medical centers having at least 18 calendar days to prepare veterans’ Choice Program referrals to TPAs and another 52 calendar days for appointments to occur as scheduled by TPAs. Based on this finding, we recommended that VA establish an achievable wait-time goal for the VCCP that will permit VA to monitor whether veterans are receiving community care within time frames that are comparable to the amount of time they would otherwise wait to receive care at VA medical facilities. We also recommended that VA should design an appointment scheduling process for the VCCP that sets forth time frames within which (1) veterans’ referrals must be processed, (2) veterans’ appointments must be scheduled, and (3) veterans’ appointments must occur that are consistent with the wait-time goal VA has established for the program. VA agreed with both recommendations, which remain unimplemented, and officials stated that they are in the process of finalizing metrics to capture wait-time performance and designing an appointment scheduling process. Without specifying wait- time goals that are achievable, and without designing appointment scheduling processes that are consistent with those goals, VA lacks assurance that veterans are receiving care from community providers in a timely manner. In June 2018, we reported that VA could not systematically monitor wait times for veterans accessing care under the Choice Program due to incomplete and inaccurate data. Without complete and accurate data, VA was not able to determine whether the Choice Program was achieving its goals of (1) alleviating the wait times veterans experienced when seeking care at VA medical facilities, and (2) easing geographic burdens veterans may have faced when accessing care at VA medical facilities. We made three recommendations to address VA’s incomplete and inaccurate data related to the Choice Program, and VA is taking steps to implement two of those recommendations. We found that the data VA used to monitor the timeliness of Choice Program appointments captured only a portion of the total appointment scheduling process. Though VA had a 30-day wait-time goal to provide veterans with care under the Choice Program, VA’s timeliness data did not capture (1) the time VA medical centers took to prepare veterans’ referrals and send them to the TPAs, and (2) the time spent by TPAs in accepting the referrals and opting veterans into the Choice Program. For example, we found that it took VA medical center staff an average of 24 calendar days after the veteran’s need for care was identified to contact the veteran, compile relevant clinical information, and send the veteran’s referral to the TPAs. For those same authorizations, it took the TPAs an average of 14 calendar days to accept referrals and reach veterans to opt them into the Choice Program. In 2016, VA also conducted its own manual review of appointment scheduling times and found that wait times could be longer than the 30 days (see fig. 2). Specifically, out of a sample of about 5,000 Choice Program authorizations, VA analyzed (1) the timeliness with which VA medical centers sent referrals to the TPAs, and (2) veterans’ overall wait times for Choice Program care. VA’s analysis identified average review times when veterans were referred to the Choice Program to be greater- than-30-day wait time for an appointment at a VA medical facility. For example, for overall wait times (i.e., the time veterans’ need for care was identified until they attended initial Choice Program appointments), wait times ranged from 34 to 91 days across the 18 VA integrated service networks. The national average was 51 days. In September 2017, VA began implementing an interim solution to monitor overall wait times, but this solution relied on VA medical center staff consistently and accurately entering data on referrals, a process that is prone to error. In June 2018, we recommended that VA establish a mechanism to monitor the overall wait times under the VCCP. VA agreed with this recommendation, and stated that it is developing a monitoring mechanism that will be incorporated into a new system that will be fully implemented across all VA medical facilities by fiscal year 2021. We also reported that the clinically indicated dates included on referrals that VA medical centers sent to the TPAs, which are used to measure the timeliness of care, may not have been accurate, further limiting VA’s monitoring of veterans’ access to care. Our review of 196 Choice Program authorizations found that clinically indicated dates were sometimes changed by VA medical center staff before they were sent to the TPAs, which could mask veterans’ true wait times. We found that VA medical center staff entered later clinically indicated dates on referrals for about 23 percent of the 196 authorizations reviewed. We made two recommendations to improve the accuracy of the Choice Program data. For example, we recommended that VA establish a mechanism under the VCCP that prevents clinically indicated dates from being modified. VA agreed with our recommendation, and stated that a new system will interface with VA’s existing referral package to allow a VA clinician to enter in a clinically indicated date while restricting schedulers from making alterations to it. In June 2018, we also reported that numerous factors adversely affected veterans’ timely access to care through the Choice Program and could affect access under the VCCP. These factors included the following: (1) administrative burden caused by complexities of VA’s referral and appointment scheduling processes; (2) poor communication between VA and its medical facilities; and (3) inadequacies in the networks of community providers established by the TPAs, including an insufficient number, mix, or geographic distribution of community providers. VA has taken steps to help address these factors; however, none have been fully addressed. For example, to help address administrative burden and improve the process of coordinating veterans’ Choice Program care, VA established a secure e-mail system and a mechanism for TPAs and community providers to remotely access veterans’ VA electronic health records. However, these mechanisms only facilitate a one-way transfer of necessary information. They do not provide a means by which VA medical facilities or veterans can view the TPAs’ step-by-step progress in scheduling appointments or electronically receive medical documentation associated with Choice Program appointments. We made five recommendations to VA to address the factors that adversely affected veterans’ access to Choice Program care. VA agreed or agreed in principle with all five recommendations and has taken some steps in response to these recommendations. However, our recommendations remain unimplemented. On June 6, 2019, VA began implementing the VCCP, which created a consolidated community care program. Under the VCCP, VA began determining veteran eligibility based on designated access standards, such as wait-time goals of 20 days for primary and mental health care and 28 days for specialty care and other criteria identified in the MISSION Act. According to VA officials, the implementation of the VCCP also included the use of the new Decision Support Tool—a system that combines eligibility and other information to help veterans, with assistance from VA staff, decide whether to seek care in the community. VA officials previously identified the Decision Support Tool along with another new system—known as the Health Share Referral Management system—as key efforts in addressing many of our recommendations related to VA’s community care wait-time data and monitoring issues. VA expects the Health Share Referral Management system, which will manage community care referrals and authorizations as well as facilitate the exchange of health information between VA and community providers, to be fully implemented across all VA medical facilities in fiscal year 2021. We began work in May 2019 to review VA’s implementation of the VCCP, including how it will address issues such as appointment scheduling. In addition to the actions described above, VA has taken other steps to improve veterans’ access to care by, for example, offering veterans access to routine care without an appointment. We have ongoing work related to same-day services provided in VA primary care and mental health clinics. In order to improve access, VA implemented the same-day service initiative in 2016, and by 2018 offered same-day services in over 1000 facilities. As part of the initiative, VA medical facility staff are directed to address veterans’ primary care and mental health needs that day through a variety of methods, including face-to-face visits, telehealth, prescription refills, or by scheduling a follow-up appointment. Our ongoing work indicates that the six VA medical facilities we visited were generally providing same-day services prior to the initiative; however, according to VA officials, ongoing staffing and space shortages created challenges implementing the initiative. Our ongoing work also indicates that VA does not have performance goals and measures to determine same-day services’ impact on veterans’ access to care. We plan to issue our report on VA’s same-day services initiative in August 2019. In closing, we have identified various weaknesses in VA’s wait-time measurement and scheduling processes over the years. These weaknesses have affected not only VA’s internal delivery of outpatient care, but also that provided through community providers. As we have highlighted here, we have made a number of recommendations to address these weaknesses. VA has taken actions to address our recommendations, but additional work is needed for some. The implementation of enhanced technology, such as a new scheduling system, is crucial and will provide an important foundation for improvements. However, this is not a panacea for addressing all of the identified problems. Moving forward, VA must also continuously ensure that it has clear and consistent policies and processes, adequate oversight, and effective training. Chairman Takano, Ranking Member Roe, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Debra A. Draper, Director, Health Care at (202) 512-7114 or DraperD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony were Sharon Silas (Acting Director), Ann Tynan (Assistant Director), Cathy Hamann, Aaron Holling, Akbar Husain, Kate Tussey, and E. Jane Whipple. Also contributing were Jacquelyn Hamilton and Vikki Porter.", "summary": "The majority of veterans utilizing VA health care services receive care in VA-operated medical facilities, including 172 VA medical centers and more than 1,000 outpatient facilities. For nearly 20 years, GAO has reported on the challenges VA medical facilities have faced providing health care services in a timely manner. When veterans face wait times at VA medical facilities, they may be able to receive services from VA's community care programs, which VA estimates will be 19 percent of its $86.5 billion in health care obligations in fiscal year 2020. This testimony focuses on GAO's large body of work on veterans' access to care and the status of VA's efforts to address GAO's recommendations, including those from GAO's June 2018 report on VA's community care programs and from GAO's December 2012 report on VA's scheduling of timely medical appointments that VA has provided information on through July 2019. It also includes preliminary observations on related ongoing work. GAO has issued several reports recommending that the Department of Veterans Affairs (VA) take action to help ensure its facilities provide veterans with timely access to medical care. VA has taken a number of steps to address GAO's recommendations to improve wait-time measurement and its appointment scheduling policy. However, additional actions are needed to fully address most of GAO's recommendations. GAO found in 2012 that outpatient appointment wait times reported by VA were unreliable because VA did not ensure consistency in schedulers' definitions of the dates by which wait times were measured. GAO recommended that VA clarify these definitions. VA concurred and has taken a number of actions in response, including improved oversight through scheduling audits. However, VA's first internal audit in August 2018 was unable to evaluate the accuracy and reliability of its wait-time data due to the lack of business rules for calculating them, indicating that additional efforts are needed to address this issue. GAO also found in 2012 that not all facilities GAO visited used the electronic wait list to track new patients that needed medical appointments, as required by VA's scheduling policy. This put patients at risk for being lost for appointment scheduling. GAO recommended VA ensure consistent implementation of its policy, and that all schedulers complete required training. VA concurred, and with the information VA provided in July 2019 GAO considers VA's actions, including updating its scheduling policy and completing scheduler training, sufficient to fully address the recommendation. While improvements to VA's scheduling policy and processes will help ensure veterans receive timely access to care, there are other factors that may also affect access that are not currently reflected in VA's wait-time data. For example, GAO found instances in which the time it took the agency to initially enroll veterans in VA health care benefits was more than 3 months. GAO has also made recommendations to improve appointment scheduling and ensure timely access to care from non-VA providers in VA's community care programs that remain unimplemented. GAO found in June 2018 that the data VA used to monitor the timeliness of the Veterans Choice Program's appointments captured only a portion of the total appointment scheduling process. Although VA had a wait-time goal of 30 days, VA's timeliness data did not capture certain processes, such as the time taken to prepare veterans' referrals and send them to a third-party administrator. GAO found that if these were accounted for, veterans could potentially wait up to 70 calendar days to see a community care provider. VA officials stated that most recommendations will be addressed with new program tools it plans to implement. For example, VA is implementing a system for referral management and appointment scheduling expected to be available in all VA medical facilities by fiscal year 2021. While technology may be an important tool, VA will also need clear and consistent policies and processes, adequate oversight, and effective training to help avoid past challenges. GAO has made a number of recommendations to VA to address timely scheduling and reliable wait-time data for outpatient appointments and through community care. VA generally agreed with GAO's recommendations. As of July 2019, VA has taken actions to fully implement one recommendation discussed in this statement. GAO continues to believe that all of the recommendations are warranted.", "document_type": "gao"}
{"report": "Records are the foundation of open government, supporting the principles of transparency, participation, and collaboration. Effective management of federal agency records is important for efficient government operations: it ensures that sufficient documentation is created; that agencies can efficiently locate and retrieve records needed in the daily performance of their missions; and that records of historical significance are identified, preserved, and made available to the public. Requirements for managing federal records include the following: The FRA establishes requirements for the management of records in federal agencies. Every federal agency is required to preserve records that document the organization, functions, policies, decisions, procedures, and essential transactions of the agency to furnish the information necessary to protect the legal and financial rights of the government and of persons directly affected by the agency’s activities. The act also gives NARA regulatory responsibilities for records management as well as general responsibilities for archiving records. In response to a presidential memorandum to begin an executive branch effort to reform records management policies and develop a framework for the management of electronic government records, the Director of OMB and the Archivist of the United States jointly issued the Managing Government Records Directive to heads of federal departments and agencies. This directive aimed at creating a robust records management framework for electronic records that complied with statutes and regulations to achieve the benefits outlined in the presidential memorandum. It required agencies to eliminate paper and use electronic recordkeeping to the fullest extent possible. Among other things, the directive identified two requirements related to electronic records that agencies were to implement between December 2016 and December 2019. By December 31, 2016, federal agencies were to manage all permanent and temporary email records in an accessible electronic format. Email records were to be retained in an appropriate electronic system that supports records management litigation requirements, including the capability to identify, retrieve, and retain the records for as long as they are needed. By December 31, 2019, federal agencies are to manage all permanent electronic records in an electronic format to the fullest extent possible for eventual transfer and accessioning by NARA. Under the FRA and its implementing regulations, NARA has general oversight responsibilities for records management and the preservation of permanent records documenting the activities of the government in the National Archives of the United States. Thus, NARA is responsible for overseeing agency management of temporary and permanent records used in everyday operations and, ultimately, for taking control of permanent agency records judged to be of historic value. In particular, NARA is responsible for: issuing records management guidance covering topics such as managing electronic records; assigning an appraisal archivist to each agency to answer agency questions about federal records management; providing services to agencies, such as records scheduling and working with agencies to implement effective controls over the creation, maintenance, and use of records in the conduct of agency business; approving the disposition (destruction or preservation) of records; providing storage facilities for agency records; and conducting inspections or surveys of agency records and records management programs. NARA is also responsible for reporting to Congress on the state of federal records management. It accomplishes this responsibility, in part, by requiring all federal agencies to submit an annual report to the Office of the Chief Records Officer for the federal government. As part of these annual reports, agencies are required to include three submissions: The Senior Agency Official Records Management Report includes responses about the agency’s progress toward the targets and requirements in the Managing Government Records Directive. The Federal Email Management Report includes a self-evaluation of their email management. The Records Management Self-Assessment includes a self- evaluation of their compliance with federal records management statutes, regulations, and program functions. In addition to NARA’s responsibilities, the FRA requires each federal agency to make and preserve records that document the organization, functions, policies, decisions, procedures, and essential transactions of the agency. Effective Records Management Must Address Electronic Records, Including Email The FRA covers documentary material, regardless of physical form or media, although, until the advent of computers, records management and archiving mostly focused on handling paper documents. However, as information is increasingly created and stored electronically, records management has had to take into account the creation of records in various electronic formats, including email messages. As such, agencies need to adapt their records management practices to manage those electronic files that may be federal records. NARA’s implementing regulations and guidance, such as periodic NARA bulletins, provide direction to agencies about the management of electronic records. To ensure that the management of agency electronic records is consistent with provisions of the FRA, NARA requires each agency to maintain an inventory of all agency information systems that identifies basic facts about each system, such as technical characteristics and the electronic records it contains. NARA also requires that agencies maintain all federal records, including those in electronic format, in its systems. Further, NARA requires agencies to provide instructions to staff regarding how to maintain the agency’s operational records and what to do when they are no longer needed for current business. Like other records, electronic records must be scheduled either under agency- specific schedules or pursuant to a general records schedule. Further, in order to effectively address NARA regulations, agencies are to establish policies and procedures that provide for appropriate retention and disposition of their electronic records. Disposition involves transferring records of permanent, historical value to NARA for the archiving of records (preservation) and the destruction of all other records that are no longer needed for agency operations. In addition to adherence to general requirements governing electronic records, according to the electronic records management regulation, agencies are to also issue instructions to staff that specifically address retention and management of their email records. The regulation requires agencies’ email records to be managed as are other federal records with regard to the adequacy of documentation, recordkeeping requirements, agency records management responsibilities, and records disposition. The FRA Amendments enacted on November 26, 2014, include, among other things, disclosure requirements for official business conducted using a non-official electronic messaging account. The law states an officer or employee of an executive agency may not create or send a record using a non-official electronic messaging account unless the officer or employee (1) includes a copy to an official electronic messaging account of the officer or employee in the original creation or transmission of the record or (2) forwards a complete copy of the record to an official electronic messaging account of the officer or employee not later than 20 days after the original creation or transmission of the record. In 2015, we reported that the 24 major federal departments and agencies covered by the Chief Financial Officers Act of 1990 had taken action in response to the Managing Government Records Directive, but not all of the agencies met all of the requirements. In that report, we stated that most of the agencies, including the Department of Commerce (Commerce) and the National Aeronautics and Space Administration (NASA), described plans to manage permanent electronic records, reported progress in managing permanent and temporary email records, and identified unscheduled records. We also noted that certain requirements were not fully met by a few agencies, including the National Science Foundation (NSF) and Office of Personnel Management (OPM), because these agencies were either still working on addressing the requirement, or did not view the requirement as being mandatory. Specifically, we reported that NSF did not submit a Senior Agency Official report that would have provided information to NARA on how it intended to manage permanent records electronically. In addition, we reported that NSF did not report to NARA on its possession of permanent 30-year-old records, and had not completed its identification of, or reported on, any portion of its unscheduled records. As a result, we recommended that NSF establish a date by which the agency would complete, and then report to NARA, its plans for managing permanent records electronically and its progress toward managing permanent and temporary email records in an electronic format. We also recommended that the agency complete the identification of unscheduled records stored at agency records storage facilities. NSF concurred with our recommendations and, in response, completed its plans for managing permanent records electronically and managing permanent and temporary email records in an electronic format. We verified in February 2017 that the agency reported these plans to NARA. For OPM, the agency had not designated their Senior Agency Official at the assistant secretary level or its equivalent because they did not view the requirement as mandatory. We recommended that the designated Senior Agency Official be at or equivalent to the level of an assistant secretary. OPM concurred with our recommendations and, in response, designated the Chief Information Officer as the Senior Agency Official with direct responsibility for ensuring that OPM efficiently and appropriately complies with all applicable records management statutes, regulations, and NARA policy. The 17 agencies selected for review varied in the extent to which their records management policies, procedures, and documentation addressed 10 key requirements in the Managing Government Records Directive, the FRA and its amendments, and implementing regulations related to electronic records. Specifically, most of the selected agencies addressed the requirements related to establishing records management programs, submitting records schedules to NARA, incorporating activities for electronic records into their overall records management program, developing plans for managing permanent electronic records in an electronic format, managing email records in an electronic format, and using non-official electronic messaging. However, agencies did not fully address the requirements related to maintaining an inventory of electronic information systems, establishing controls and preservation considerations for their electronic information systems, and issuing retention and management requirements for email. According to the FRA and its amendments, agencies are to establish effective records management programs, which includes developing comprehensive records schedules, in order to achieve adequate and proper documentation of the policies and transactions of the federal government and to aid in the effective and economical management of agency operations. Specifically, each agency is required to: establish and maintain an active, continuing records management program that, among other things, includes effective controls over the creation, maintenance, and use of records and submit lists and schedules of records to the Archivist of the United States that describe, among other things, when eligible temporary records must be disposed of. As shown in figure 1, the majority of the 17 selected agencies addressed these requirements. Establishing a records management program: Fourteen of the 17 selected agencies—Armed Forces Retirement Home (AFRH), Consumer Financial Protection Bureau (CFPB), Commerce, U.S. Election Assistance Commission (EAC), Federal Housing Finance Agency (FHFA), Federal Trade Commission (FTC), NASA, NSF, Office of Management and Budget (OMB), Office of National Drug Control Policy (ONDCP), Overseas Private Investment Corporation (OPIC), OPM, Peace Corps, and Special Inspector General for Afghanistan Reconstruction (SIGAR)—had developed policies and procedures that outlined their records management program. The agencies’ records management documentation discussed, among other things, the requirement for effective controls over the creation, maintenance, and use of records at the agency. However, three agencies—Marine Mammal Commission, Presidio Trust, and the Morris K. Udall and Stewart L. Udall Foundation (Udall Foundation)—did not have an active, continuing agency records management program, including documentation that described effective controls over the creation, maintenance, and use of records at the agency. All three agencies indicated that they have taken or intend to take actions to establish such a program. Marine Mammal Commission officials responsible for records management stated that the agency had engaged a contractor who completed and submitted for agency review and approval a draft policy that would govern its records management program. As of January 2020, the Executive Director stated that the commission has a signed policy and draft handbook to govern its records management program and that it is working towards full implementation and compliance by December of 2022. Presidio Trust officials responsible for records management stated that the agency intends to address the requirements and plans to have records management policies and procedures at the agency in fiscal years 2020 and 2021. Udall Foundation officials responsible for records management stated that the agency had entered into an interagency agreement with NARA for consulting services to assess its current records management environment. According to the same officials, their intent is to review NARA’s recommendations and develop a plan to comply with the FRA, federal regulations, and NARA guidelines as they relate to records management. The agency did not provide an estimated date for completing these activities. Until these agencies establish an active and continuing records management program, they cannot provide assurance that, among other things, effective controls are in place over the creation, maintenance, and use of records in the conduct of current business. Submitting lists and schedules of records to the Archivist: Thirteen of the 17 selected agencies—AFRH, CFPB, EAC, FHFA, FTC, Marine Mammal Commission, NASA, ONDCP, OPIC, Peace Corps, Presidio Trust, SIGAR, and the Udall Foundation—had submitted a comprehensive list of records and disposition schedules to the Archivist. The remaining four agencies—Commerce, NSF, OMB, and OPM—had partially addressed this requirement because they had submitted only partial lists and schedules to the Archivist. Each of these agencies acknowledged they did not provide comprehensive lists of records and disposition schedules and stated they were currently working toward submitting them to the Archivist. OMB officials stated that they plan to complete this task by the end of calendar year 2019, while the other agencies did not provide an estimated date for completion. Without submitting lists of records and disposition schedules to the Archivist, Commerce, NSF, OMB, and OPM are at risk of maintaining records that are no longer relevant or needed. The Managing Government Records Directive was aimed at creating a robust records management framework for electronic records that complies with statutes and regulations. In order to ensure transparency, efficiency, and accountability, the directive instructed agencies to manage all permanent and temporary e-mail records in an accessible electronic format by December 2016 and manage all permanent electronic records in an electronic format to the fullest extent possible by December 2019. The directive also required NARA to develop revised guidance for transferring permanent electronic records and issue new guidance describing methods for managing, disposing of, and transferring e-mail. Accordingly, NARA regulations and guidance outline requirements for agencies to establish a framework for managing electronic records, including requirements pertaining to electronic systems and email. Additionally, the FRA Amendments described the disclosure requirements for official business conducted using non-official electronic messaging accounts. Based on our analysis, these documents identify, among other things, eight key requirements that agencies should include in their policies and procedures to ensure that they can effectively manage electronic records. These requirements are summarized in table 1. The 14 agencies with an established records management program varied greatly in the extent to which they addressed these electronic records requirements, as seen in figure 2. Incorporate activities for electronic records into the agency’s overall records management program: Thirteen of 14 agencies that had established records management programs—AFRH, Commerce, CFPB, EAC, FHFA, FTC, NASA, NSF, OPIC, ONDCP, OPM, Peace Corps, and SIGAR—developed written policies and procedures that incorporated the management of electronic records into their records management program. The remaining agency—OMB—did not address this requirement. Staff from OMB responsible for records management stated that the Executive Office of the President’s (EOP) Office of Administration is responsible for records management for all Executive Office components and has procedures that incorporate the management of electronic records into their records management program. However, the officials did not provide evidence that the existing policies and procedures incorporated the management of electronic records into their records program. Without being able to ensure that records management considerations are incorporated into the design and implementation of electronic systems, OMB risks not being positioned to properly manage records electronically. Maintain an inventory of electronic systems: Three of the 14 agencies that had established records management programs— Commerce, FHFA, and SIGAR—also maintained an inventory of electronic information systems that documented the information and records produced and maintained by each application. Officials responsible for records management at these agencies stated that their inventory was maintained with the agency’s security plans. Additionally, three of the 14 agencies that had established a records management program —FTC, NSF, and Peace Corps—partially addressed the requirement, as their policies and procedures addressed some, but not all, of the necessary elements. More specifically: FTC documented various technical characteristics, such as authorizations, purpose and function of the electronic information systems, and authorized procedures for the disposition of records. However, the agency did not include the characteristics for reading and processing the records contained in the system, inputs and outputs, contents of the files and records, and cycle updates. NSF documented the categories of records in the electronic information systems, record access procedures, purpose of the systems, and retention and disposition of the system’s records. However, the agency did not specify the technical characteristics of the systems, identify inputs and outputs, or describe update cycles. Peace Corps documented update cycles and the purpose of the electronic information systems. However, the documentation did not specify the technical characteristics necessary for reading and processing the records contained in the system, identify system inputs and outputs, define the contents of the files and records, determine restrictions on access to and use of the system, and specify how the agency ensures the timely disposition of records. According to officials responsible for records management at each of these agencies, they intend to address or would consider addressing the requirement. However, none of them provided a time frame for doing so. The remaining eight agencies—AFRH, CFPB, EAC, NASA, OMB, ONDCP, OPM, and OPIC—either did not maintain an inventory of electronic information systems or did not provide documentation that outlined the technical characteristics, such as identifying all inputs and outputs necessary for reading and processing records contained in the system. Records management officials at AFRH, CFPB, EAC, NASA, OPM, and OPIC stated that they intend to address the requirement, but did not provide a time frame for doing so. Staff from OMB and ONDCP responsible for records management stated that EOP’s Office of Administration is responsible for records management for all components and maintains an inventory of electronic information systems. However, the officials did not provide evidence of this inventory. Without maintaining an inventory and documentation of electronic information systems used to store agency records, these agencies are at a heightened risk of records being lost and not identified and scheduled in accordance with agency records schedules. Manage permanent electronic records in an electronic format: The Managing Government Records Directive requires each agency to develop and begin to implement plans to manage all permanent records in an electronic format. In accordance with this requirement, 12 of the 14 agencies that had established records management programs—AFRH, CFPB, Commerce, FHFA, FTC, NASA, NSF, OMB, ONDCP, OPIC, Peace Corps, and SIGAR—described their efforts to address the requirement in their Senior Agency Official reports to NARA. For example, these agencies described, among other things, plans on how permanent electronic records were being captured, retained, searched, and retrieved. However, two agencies—EAC and OPM—did not address how they plan to manage permanent electronic records in their Senior Agency Official reports or other agency documentation. EAC officials stated that they were still deciding on a solution to manage permanent records, and OPM officials stated they were planning to update policies to ensure automated systems incorporate proper records management life cycle controls. Further, neither agency provided a time frame for developing and implementing a plan. By not having a plan to manage their permanent records in an electronic format, these agencies face an increased risk that they may not be positioned to manage permanent electronic records. Incorporate required recordkeeping functionalities: Eight of the 14 selected agencies that established records management programs— Commerce, CFPB, FHFA, FTC, NASA, NSF, ONDCP, and SIGAR— had documented policies, procedures, or other records management documentation that addressed the required functionalities for recordkeeping systems. Additionally, one agency—OPIC—partially addressed this requirement because it included some, but not all, of the required functionality. More specifically, the agency did not identify whether the system could declare records and assign unique identifiers, capture records, maintain security, and preserve records. According to OPIC officials, the agency intends to work toward having better documentation outlining system functionalities in alignment with the requirements; however, the officials did not provide a time frame for completing this documentation. Further, five of the 14 agencies that had established records management programs— AFRH, EAC, OMB, OPM, and Peace Corps—did not address this requirement. Officials responsible for records management at each of these agencies stated that their records management system encompassed all of the aforementioned functionality or that the agency was working toward a full electronic records management system. However, these agencies’ policies and procedures did not include the required functionalities for recordkeeping systems. According to the same officials, each agency intends to have written documentation that outlines the records management functionalities; however, they did not provide a time frame in which the documentation will be completed. Without using electronic recordkeeping systems with appropriate functionalities, these agencies face increased risk of not being able to reliably access and retrieve the records needed to conduct agency business. Establish records management controls and preservation considerations: Seven of the 14 agencies that had established a records management program—CFPB, Commerce, FHFA, FTC, NASA, OPM, and SIGAR—included all records management controls in their electronic information systems policy and included preservation considerations in the design, development, and implementation of electronic information systems. Additionally, six of the 14 agencies that had established records management programs—AFRH, EAC, OMB, ONDCP, OPIC, and the Peace Corps—had policies that partially addressed establishing the records management controls for their electronic information systems. More specifically: AFRH records management documentation included information controls to ensure the reliability, authenticity, and integrity of records. However, the documentation did not define controls for usability, content, context, and structure. EAC’s documentation included controls for reliability, authenticity, integrity, and usability. However, the agency did not define controls for content, context, and structure. OMB and ONDCP’s documentation outlined controls for authenticity, integrity, usability and content. However, the documentation did not define controls for reliability, context, and structure. Staff stated that both offices’ records management was handled by the Office of Administration in the EOP and that the office had acquired an object-based data storage system that was expected to address all of the required controls. However, the offices did not provide any evidence that the new system or the associated policies and procedures would address the required controls. OPIC’s documentation defined controls for authenticity, integrity, and usability. However, the documentation did not define controls for reliability, content, context, and structure. Peace Corps’ documentation included controls for reliability, authenticity, integrity, and content. However, the agency did not define controls for usability, context, and structure. Additionally, each agency’s documentation did not describe how the agency ensures that records in the system are retrievable and useable for as long as needed to conduct agency business. Records management officials at each of the agencies acknowledged that not all of the controls or preservation considerations were included in their systems and that they planned to work toward implementing all of the controls; however, the agencies did not provide a time frame for documenting the controls. The remaining agency—NSF—did not address this requirement because its existing policies and procedures did not demonstrate that the agency had established the required controls. NSF officials stated that they intend to comply with this requirement but did not provide a time frame for doing so. Without ensuring that records management controls and preservation considerations are incorporated into electronic information systems, the agencies cannot ensure these systems can produce retrievable and useable records for as long as needed to conduct agency business. Manage permanent and temporary email records in an electronic format: Thirteen agencies—AFRH, CFPB, EAC, FHFA, FTC, NASA, NSF, OMB, ONDCP, OPIC, OPM, Peace Corps, and SIGAR— addressed this requirement. The remaining agency—Commerce—did not address this requirement. Officials responsible for records management at Commerce stated that they use an email management system for email, email preservation, and litigation holds. However, their policies and procedures did not show how the agency managed both permanent and temporary email records in an accessible electronic format. Until Commerce ensures that its systems are capable of managing permanent and temporary email records and have the capability to identify, retrieve, and retain these records, the agency faces an increased risk that its emails are not able to be preserved or accessed when needed. Issue retention and management requirements: Nine of the 14 agencies that had established records management programs— AFRH, Commerce, CFPB, EAC, FHFA, FTC, NASA, Peace Corps, and SIGAR—issued instructions or had policies on retention and management requirements for electronic mail. Additionally, two of the 14 agencies that had established records management programs— OPIC and OPM—had policies that partially addressed this requirement. More specifically: OPIC’s policies and procedures documented that agency email messages and attachments that meet the statutory definition of a record are to be documented as an official record. However, the agency documentation did not discuss retention requirements for calendars. Officials responsible for records management stated that they intend to update the records and information management handbook to include the calendar requirement, but did not provide a time frame for updating the handbook. OPM’s policies and procedures described how employees were to ensure that email records included most of the requirements, but the policies and procedures did not address retaining calendars and draft documents. Officials responsible for records management stated that they intend to review and update its records management policy, but did not provide a time frame for doing so. The policies of the remaining three agencies—NSF, OMB, and ONDCP—did not address this requirement for various reasons. NSF officials responsible for records management stated that the agency issued instructions regarding record retention and management of email to staff through memos and bulletins. However, these documents did not include instructions to staff that ensured the names and addresses of the sender, date of message, attachments, calendars, and draft documents would be retained. Additionally, staff from OMB and ONDCP responsible for records management stated that the Office of Administration within the EOP captured and managed all email on behalf of all components. According to these staff, email is permanent until the end of the presidential administration, at which time the email is transferred to NARA in accordance with each component’s records schedules. However, the staff did not provide evidence that the existing policies and procedures included these instructions. By not issuing instruction to staff on retention and management requirements for email, agencies are at risk of not being able to retrieve email and its associated metadata when needed to conduct agency business. Use of non-official electronic messaging: Twelve of the 14 agencies that had established records management programs — CFPB, Commerce, FHFA, FTC, NASA, NSF, OMB, ONDCP, OPM, OPIC, Peace Corps, and SIGAR—had policies and procedures outlining the rules that their employees are to follow when creating records using a non-official electronic messaging account. The remaining two of 14 agencies that had established records management programs—AFRH and EAC—did not have written documentation describing the agencies’ disclosure requirements for official business conducted using non-official electronic messaging accounts. The EAC records management officials acknowledged that the agency did not outline this requirement and stated that policies and procedures were being drafted to address this requirement; however, the officials did not provide an estimated completion date. AFRH stated that it had updated its “Network Rules of Behavior” document and its IT information security awareness training to new employees to reflect the requirement, but we were unable to verify the updates. Without establishing rules for employees on the use of non-official electronic messaging accounts, agencies are at risk of not retaining email records sent from personal accounts. The 10 aforementioned requirements are important elements to address while establishing a framework for managing electronic records. While most of the selected agencies had established policies and procedures addressing the requirements, some had not. Until these agencies do so, they will lack assurance that electronic records are being managed in a way that promotes openness and accountability in documenting agency actions and decisions. NARA provided various forms of assistance to the selected agencies, which included issuing guidance regarding electronic records management, training, and professional development. In addition, NARA monitored the selected agencies’ compliance with records management regulations and implementation of policies, guidance, and other records management best practices through its self-assessment program. However, NARA had not ensured that any of the selected small or micro agencies that self-assessed to be at high risk of improper records management in calendar year 2017 were taking appropriate actions to improve their records management program. According to the FRA, NARA is responsible for providing guidance and assistance to federal agencies with respect to ensuring economical and effective records management, adequate and proper documentation of the policies and transactions of the federal government, and proper records disposition. In accordance with its responsibilities, NARA provided guidance and assistance to the selected agencies through various methods. All of the selected agencies stated that NARA guidance and assistance were generally helpful and that they relied on it to some extent for implementing the electronic records management requirements discussed in this report. Specifically, NARA issued guidance particular to electronic records creation, policies and procedures, management, and disposition. Officials from the selected agencies found NARA’s guidance related to managing email and its December 2017 General Records Schedule to be helpful when fulfilling their responsibilities with respect to electronic records. The guidance related to managing email describes federal agencies’ responsibilities for email management and the Capstone approach to email management. The Records Officer at AFRH stated that the agency used guidance that described the minimum set of metadata elements that must accompany transfers of permanent electronic records to NARA. The General Records Schedule provides mandatory disposition instructions for records that are common to several or all federal agencies. An FHFA official responsible for records and information management stated this guidance was useful because it was used at the agency during regular records management activities, such as records disposition. Further, NARA provided assistance to the selected agencies such as professional development training and assigning an archivist to assist each agency. Officials responsible for records management at the selected agencies stated that NARA offers agencies records management training and professional development to federal employees and contractors. For example, NARA provides a certificate program for Federal Agency Records Officers and records management professionals to manage information collected by their agency. In addition, these officials stated that NARA offers a bi-monthly Records Information Discussion Group where individuals involved with federal records management can share their experiences and discuss the latest developments from NARA. Additionally, officials responsible for records management at the selected agencies stated that NARA assigns an archivist to each agency to field questions about federal records management, including services such as records scheduling and appraisal, and technical assistance. For example, Udall Foundation officials responsible for records management stated that the agency worked with its assigned archivist who provided direction on how to manage agency records, connected the agency with other records management subject matter experts, and fielded questions on the scheduling of agency records. In addition to providing assistance to federal agencies, NARA also has the responsibility to monitor compliance with records management regulations and implementation of NARA policies, guidance, and other records management best practices by federal agencies. One way in which NARA accomplishes this is to require federal agencies to conduct an annual self-assessment that evaluates the agency’s reported compliance with federal records management statutes, regulations, and program functions and is also useful to target resources to areas needing improvement. NARA scores each agency’s responses and, based on this score, determines whether an agency is at risk of not complying with statutory and regulatory records management requirements. For the self-assessments that covered calendar year 2017, four of our 17 selected agencies—AFRH, Marine Mammal Commission, Presidio Trust and the Udall Foundation—were assessed as being at high risk of not complying with statutory and regulatory records management requirements. See table 2 for how 16 of the 17 selected agencies scored. While NARA requires agencies to self-assess their records management programs annually, it does not ensure that agencies that scored poorly on their self-assessments develop a plan to improve their programs or monitor their progress in such efforts. According to NARA officials, after reviewing the reports the agency conducted phone interviews with staff from selected small and micro agencies to determine if there were any common factors for why they scored poorly on the self-assessments and what NARA could do to help them improve their records management programs. Given the self-assessment process was designed to measure agency compliance and to target resources to areas needing improvement, it is important for NARA to ensure the small and micro agencies that have assessed their programs as high-risk are taking appropriate actions to improve their records management programs. Until NARA requires high- risk small and micro agencies to develop plans to make necessary improvements to their record management programs and monitor their progress, it cannot be certain that these agencies are managing electronic records in accordance with governing regulations. Similarly, agencies that have not submitted self-assessments may also not be addressing statutory and regulatory records management requirements. While most of the selected agencies addressed the key electronic recordkeeping requirements, others did not. Specifically, many agencies did not address requirements related to electronic system and email implementation, including establishing controls for their electronic information systems, incorporating preservation considerations into systems, and issuing retention and management requirements for email. Until these agencies do so, they will lack assurance that electronic records are being created, managed, retained, preserved, and disposed of in a way that improves performance and promotes openness and accountability by better documenting agency actions and decisions. NARA continues to assist the selected agencies in managing electronic records by providing guidance and training as well as monitoring their compliance with records management regulations. However, while NARA oversees the selected agencies’ compliance through records management self-assessments, it has not ensured that the selected small and micro agencies that were at high risk of improper records management have developed plans to address weaknesses in their records management programs. We are making 42 recommendations to 15 agencies. Specifically, we are making the following recommendations to NARA: The Archivist of the United States should 1. require small and micro agencies that were determined to be at high risk of not complying with statutory and regulatory records management requirements to develop plans and timelines to address their records management weaknesses (Recommendation 1) 2. monitor the agencies’ progress towards these efforts on a regular basis. (Recommendation 2) In addition, we are making 40 recommendations to 14 agencies to fully address the electronic recordkeeping requirements found in the Managing Government Records Directive and the Presidential and Federal Records Act Amendments of 2014 in their policies and procedures. Appendix II contains these recommendations. We requested comments on a draft of this report from NARA and the 17 other agencies included in our review. All of the agencies provided responses, as further discussed. In written comments, NARA concurred with our recommendations and stated that the agency will develop an action plan to require small and micro agencies that consistently score in the high risk category on NARA’s annual records management self-assessment to address their records management weaknesses. In addition, NARA stated that it will continue to gather data to identify where inspections, guidance, and training are needed to ensure that small and micro agencies are improving their records management programs. NARA’s comments are reprinted in appendix III. Of the 17 other agencies in our review, six agencies (CFPB, Commerce, NASA, NSF, OPM, and the Udall Foundation) concurred with our recommendations; five agencies (Marine Mammal Commission, OMB, ONDCP, OPIC, and Presidio Trust) did not state whether they agreed or disagreed with our recommendations; and six agencies (AFRH, EAC, FHFA, FTC, Peace Corps, and SIGAR) stated that they had no comments on the report. Multiple agencies also provided technical comments, which we incorporated as appropriate. Among these agencies, the following six concurred with our recommendations and, in most cases, described steps planned or under way to address them: The Consumer Financial Protection Bureau provided written comments in which the agency stated that it did not object to our recommendation. The agency added that it would establish a time frame to update its current inventory of electronic systems used to store agency records, so that the inventory includes all of the required elements. CFPB’s comments are reprinted in appendix IV. In written comments, the Department of Commerce concurred with our two recommendations and stated that the agency intends to take additional steps to implement them. Specifically, with regard to our recommendation regarding up-to-date records schedules, the agency stated that it will ensure that its records schedules are updated and submitted to NARA no later than December 2020. Commerce also stated that, while it believes its current electronic system that manages email meets our recommendation, the agency intends to take additional steps by updating its policies and ensuring that users are correctly implementing the system to address federal recordkeeping requirements by December 2020. Commerce’s comments are reprinted in appendix V. The National Aeronautics and Space Administration provided written comments in which it concurred with our recommendation. The agency added that it is currently developing a comprehensive inventory to serve as an authoritative source for identifying where the agency’s electronic records reside, which should be completed by June 2021. NASA comments are reprinted in appendix VI. In written comments, the National Science Foundation concurred with our four recommendations. NSF stated that the agency is updating its schedules and intends to ensure that its records management practices and policies address current requirements and best practices for federal records management. NSF’s comments are reprinted in appendix VII. The Office of Personnel Management provided written comments in which it concurred with our five recommendations and noted steps that the agency has begun or is planning to take to address them. OPM stated that, in fiscal year 2020, it intends to issue a strategic plan on the digitization and management of permanent and electronic records, update agency policies and procedures to include the required electronic information system function for recordkeeping systems, and implement the requirements of the agency’s Capstone email policy. The agency also noted that, in fiscal year 2021, it plans to complete the updates needed on all agency disposition schedules and develop an inventory of all electronic information systems that store agency records. OPM’s comments are reprinted in appendix VIII. In written comments, the Udall Foundation concurred with our recommendation and described the steps it plans to take in fiscal years 2020 and 2021 to establish records management policies and procedures. For example, according to the foundation, in September 2020, it plans to complete the initial build-out of required infrastructure to manage electronic records. Further, in March 2021, it plans to finalize a formal records management policy and associated procedures for creating, maintaining, and using records across the agency. The Udall Foundation’s comments are reprinted in appendix IX. Further, the following five agencies did not state whether they agreed or disagreed with the recommendations: In written comments, the Office of Management and Budget did not state whether it agreed or disagreed with our recommendations. However, OMB stated that it is diligently working with NARA to revise and update its records schedule and intends to closely review and close any gaps in documentation that GAO identified. The office also provided technical comments, which we incorporated as appropriate. OMB’s comments are reprinted in appendix X. In an email from the Executive Director, the Marine Mammal Commission did not state whether it agreed or disagreed with our recommendations. However, according to the executive director, the commission now has a signed records management policy that describes staff responsibilities for the management of electronic records and email as well as a draft records management handbook. The official also stated that the commission will continue efforts to fully implement the records management policy and procedures aiming toward full implementation and compliance by December 2022. The Commission also provided technical comments, which we incorporated as appropriate. In an email from the Acting General Counsel, the Office of National Drug Control Policy did not state whether it agreed or disagreed with our recommendations. The office provided technical comments, which we incorporated as appropriate. In an email from its GAO audit liaison, the Overseas Private Investment Corporation did not state whether it agreed or disagreed with our recommendations. However, the liaison stated that the agency intends to implement a new solution for electronic records and information management that includes the recordkeeping functionalities required by NARA. The liaison added that the agency plans to update its records and information management policies and procedures to strengthen the records management controls and preservation guidance in fiscal year 2021. In an email from the Chief Financial and Administrative Officer, Presidio Trust did not state whether it agreed or disagreed with our recommendations. However, the official stated that the trust had recently implemented the Capstone approach for email and would continue to work on records management throughout 2020 and 2021. Lastly, we received emails from the Armed Forces Retirement Home’s Information Technology Manager, the U.S. Election Assistance Commission’s Communication Specialist, the Federal Housing Finance Agency’s Privacy Act Officer, the Federal Trade Commission’s attorney representative in the Office of General Counsel, the Peace Corps Agency Records Officer, and the Special Inspector General for Afghanistan Reconstruction’s Director of Information Technology. All of the emails stated that these agencies had no comments on the draft report. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Commerce; the Administrator of the National Aeronautics and Space Administration; the Archivist of the United States; the Chief Executive Officers of the Armed Forces Retirement Home and Overseas Private Investment Corporation; the Executive Directors of the U.S. Election Assistance Commission and Udall Foundation; the Directors of the Consumer Financial Protection Bureau, Federal Housing Finance Agency, National Science Foundation, Office of Management and Budget, Office of National Drug Control Policy, Office of Personnel Management and Peace Corps; the Chairman of the Federal Trade Commission, Marine Mammal Commission, and the Presidio Trust Board; the Special Inspector General for Afghanistan Reconstruction and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9342 or marinosn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix XI. Our objectives were to determine the extent to which (1) selected agencies’ policies and procedures address electronic recordkeeping requirements in the Managing Government Records Directive and the Presidential and Federal Records Act Amendments of 2014 and (2) NARA assisted selected agencies in managing their electronic records. To determine the agencies for our review, we identified agencies that established a Senior Agency Official for Records Management and submitted an annual report on NARA’s website between fiscal year 2015 and fiscal year 2017. Of the 95 agencies that met these criteria, we removed two from consideration because they were not part of the executive branch: one was a judicial branch agency and the other was a legislative branch agency. The 93 remaining agencies to include in our review represented the following categories: (1) executive departments, (2) Executive Office of the President, and (3) independent agencies. To ensure that a variety of agencies were selected across the designated categories, we chose a selection of 17 agencies and ensured that at least two agencies were selected from the three identified categories. In order to generate this selection, we sorted the list of 93 agencies by assigned random numbers and selected the top 17 agencies in this list, while ensuring that at least two agencies from each category were selected. The selection of 17 agencies cannot be used to make generalizable statements about the full population of agencies. The 17 agencies selected were: 1. Armed Forces Retirement Home 2. Consumer Financial Protection Bureau 3. Department of Commerce 4. U.S. Election Assistance Commission 5. Federal Housing Finance Agency 6. Federal Trade Commission 7. Marine Mammal Commission 8. Morris K. Udall and Stewart L. Udall Foundation 9. National Aeronautics and Space Administration 10. National Science Foundation 11. Office of Management and Budget 12. Office of the National Drug Control Policy 13. Office of Personnel Management 14. Overseas Private Investment Corporation 17. Special Inspector General for Afghanistan Reconstruction To address the first objective, we identified key requirements specified in the Federal Records Act, the Presidential and Federal Records Act Amendments of 2014, and its implementing regulations, and the Office of Management and Budget’s (OMB) and NARA’s Managing Government Records Directive. In selecting the requirements for our assessment, we focused on requirements related to electronic records management, such as managing permanent and temporary records, managing email records, and managing electronic records management programs. To assess whether agencies’ policies and procedures addressed the key requirements, we collected and analyzed policies, procedures, and other documentation that described how agencies are positioned to effectively manage electronic records. In particular, we reviewed agencies’ recordkeeping handbooks, agencies’ bulletins, file plans, records schedules, and electronic system user guides. Further, we collected and reviewed documentation that described agencies’ actions or planned actions to meet the specified deadlines in the Managing Government Records Directive. Specifically, we analyzed agencies’ records schedules, reports from NARA’s Senior Agency Official for Records Management’s web page, agencies’ email management system specifications, and agencies’ Capstone approach to email management. We also verified with NARA records management officials whether selected agencies submitted records schedules by the December 31, 2016, deadline specified in the Managing Government Records Directive. We assessed these documents against each of the key requirements to determine each agency’s status in developing policies and procedures to address federal record keeping requirements. Subsequent to our initial assessment, we conducted interviews with records management officials from the 17 selected agencies to discuss steps taken and obtain additional supporting evidence to determine the agencies’ status for implementing key federal recordkeeping requirements. We followed up with those agencies that did not fully address the key federal recordkeeping requirements to determine reasons for their lack of implementation. For the second objective, we reviewed federal laws and guidance, such as the Federal Records Act, NARA regulations, and OMB’s and NARA’s Managing Government Records Directive, to determine NARA’s role and responsibilities in assisting the 17 agencies in managing their electronic records. Subsequently, we collected and analyzed guidance and other documentation from NARA, such as the agency’s Records Management Oversight and Reporting Handbook, Guidance on Senior Agency Officials for Records Management bulletin, and Frequently Asked Questions about Selecting Sustainable Formats for Electronic Records, to determine whether the documentation addressed all of the requirements needed to assist agencies in managing their electronic records. We also analyzed responses in agencies’ fiscal year 2017 and 2018 Senior Agency Official for Records Management reports stating what assistance the agencies would like NARA to provide. We then conducted interviews with NARA’s Chief Records Officer and other agency officials regarding their interactions with the 17 agencies on the use of electronic recordkeeping and implementation of federal records management policies and practices to determine to what extent NARA assisted selected agencies in managing their electronic records. We also conducted interviews with officials from each of the 17 selected agencies to gain insight into how the agencies use the resources provided by NARA. Lastly, we reviewed NARA’s annual self-assessment program that evaluates agencies’ reported compliance with federal records management statutes, regulations, and program functions to obtain information on how NARA was determining which agencies needed assistance with implementing their records management programs. We supplemented our document reviews and analysis with interviews of selected agency officials responsible for records management and NARA agency officials to gain an understanding of these and other relevant documents aimed at helping agencies implement their records management programs. Additionally, to identify which of our selected agencies were to be categorized as small and micro agencies, we used OMB’s definition of small agencies as agencies with fewer than 6,000 employees and micro agencies as agencies having fewer than 100 employees. We conducted our work from March 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We are making a total of 40 recommendations to 14 of the 17 agencies in our review to fully address the electronic recordkeeping requirements in their policies and procedures. The Chief Executive Officer of the Armed Forces Retirement Home should take the following four actions: Establish a time frame to develop an inventory of electronic information systems used to store agency records that includes all of the required elements. (Recommendation 3) Establish a time frame to update its policies and procedures to include all of the required electronic information system functionalities for recordkeeping systems. (Recommendation 4) Establish a time frame to update the agency’s policies and procedures to include the (1) following records management controls required for electronic information systems: usability, content, context, and structure and (2) required preservation mechanisms to ensure that records in its electronic recordkeeping system will be retrievable and useable. (Recommendation 5) Ensure existing policies and procedures describe the rules for using personal email accounts when conducting official agency business to include instructing the employee to (1) copy an official electronic messaging account of the employee in the original creation or transmission of the records and (2) forward a complete copy of the record to an official electronic messaging account of the employee no later than 20 days after the original creation or transmission of the record. (Recommendation 6) The Secretary of Commerce should take the following two actions: Establish a time frame to ensure all records schedules are up-to-date and submitted to NARA. The schedules should include all required information, including when eligible temporary records must be destroyed or deleted and when permanent records are to be transferred to NARA. (Recommendation 7) Ensure the electronic system that manages email provides the capabilities to manage permanent and temporary email records and to identify, retrieve, and retain records. (Recommendation 8) The Director of the Consumer Financial Protection Bureau should take the following action: Establish a time frame to develop an inventory of electronic information systems used to store agency records that includes all of the required elements. (Recommendation 9) The Executive Director of the Election Assistance Commission should take the following five actions: Establish a time frame to develop an inventory of electronic information systems used to store agency records that includes all of the required elements. (Recommendation 10) Establish a time frame to develop a plan on how the agency intends to manage permanent electronic records. (Recommendation 11) Establish a time frame to update its policies and procedures to include all of the required electronic information system functionalities for recordkeeping systems. (Recommendation 12) Establish a time frame to update the agency’s policies and procedures to include the (1) following records management controls required for electronic information systems: content, context, and structure and (2) required preservation mechanisms to ensure that records in its electronic recordkeeping system will be retrievable and useable. (Recommendation 13) Develop a written policy that describes the rules for using personal email accounts when conducting official agency business to include instructing the employee to (1) copy an official electronic messaging account of the employee in the original creation or transmission of the records and (2) forward a complete copy of the record to an official electronic messaging account of the employee no later than 20 days after the original creation or transmission of the record. (Recommendation 14) The Chairman of the Federal Trade Commission should take the following action: Establish a time frame to update the agency’s electronic information system inventory to include the following characteristics: reading and processing the records contained in the system, inputs and outputs, contents of the files and records, and cycle updates. (Recommendation 15) The Chairman of the Marine Mammal Commission should take the following action: Use recently developed policies and procedures to implement and maintain an active, continuing agency records management program that includes policies and procedures to provide for effective controls over the creation, maintenance, and use of records in the conduct of current business. (Recommendation 16) The Administrator of the National Aeronautics and Space Administration should take the following action: Establish a time frame to develop an inventory of electronic information systems used to store agency records that includes all of the required elements. (Recommendation 17) The Director of the National Science Foundation should take the following four actions: Establish a time frame to ensure all records schedules are up-to-date and submitted to NARA. The schedules should include all required information, including when eligible temporary records must be destroyed or deleted and when permanent records are to be transferred to NARA. (Recommendation 18) Establish a time frame to update the agency’s electronic information system inventory to include the following characteristics: technical characteristics of the systems, identify inputs and outputs, and describe update cycles. (Recommendation 19) Establish a time frame to update the agency’s policies and procedures to include all of the records management controls required for electronic information systems and the required preservation mechanisms to ensure that records in its electronic recordkeeping system will be retrievable and useable. (Recommendation 20) Develop policies and procedures for the required retention and management requirements for email, including instructions to staff to ensure that the names and addresses of the sender, date of message, attachments, calendars, and draft documents will be retained. (Recommendation 21) The Director of the Office of Management and Budget should take the following five actions: Ensure, in conjunction with the Executive Office of the President’s Office of Administration, that existing policies and procedures incorporate the management of electronic records into its overall records management program. (Recommendation 22) Establish a time frame to develop an inventory of electronic information systems used to store agency records that includes all of the required elements. (Recommendation 23) Establish a time frame to update its policies and procedures to include all of the required electronic information system functionalities for recordkeeping systems. (Recommendation 24) Establish a time frame to ensure, in conjunction with the Office of Administration, that policies and procedures include the (1) following records management controls required for electronic information systems: reliability, context, and structure and (2) required preservation mechanisms to ensure that records in its electronic recordkeeping system will be retrievable and useable. (Recommendation 25) Ensure, in conjunction with the Office of Administration, that existing policies and procedures include the required retention and management requirements for email. (Recommendation 26) The Director of the Office of National Drug Control Policy should take the following three actions: Establish a time frame to develop an inventory of electronic information systems used to store agency records that includes all of the required elements. (Recommendation 27) Establish a time frame to ensure, in conjunction with the Office of Administration, that policies and procedures include the (1) following records management controls required for electronic information systems: reliability, context, and structure; and (2) required preservation mechanisms to ensure that records in its electronic recordkeeping system will be retrievable and useable. (Recommendation 28) Ensure, in conjunction with the Office of Administration, that existing policies and procedures include the required retention and management requirements for email. (Recommendation 29) The Director of the Office of Personnel Management should take the following five actions: Establish a time frame to ensure that all records schedules are up-to- date and submitted to NARA. The schedules should include all required information, including when eligible temporary records must be destroyed or deleted and when permanent records are to be transferred to NARA. (Recommendation 30) Establish a time frame to develop an inventory of electronic information systems used to store agency records that includes all of the required elements. (Recommendation 31) Establish a time frame to develop a plan to manage permanent electronic records. (Recommendation 32) Establish a time frame to update its policies and procedures to include all of the required electronic information system functionalities for recordkeeping systems. (Recommendation 33) Establish a time frame to update the agency’s policies and procedures on retention and management for email to include retaining electronic calendars and draft documents. (Recommendation 34) The Chief Executive Officer of the Overseas Private Investment Corporation should take the following four actions: Establish a time frame to develop an inventory of electronic information systems used to store agency records that includes all of the required elements. (Recommendation 35) Establish a time frame to develop policies and procedures that define required electronic information system functionalities for recordkeeping systems including declaring records and assigning unique identifiers, capturing records, maintaining security, and preserving records. (Recommendation 36) Establish a time frame to update the agency’s policies and procedures to include the (1) following records management controls required for electronic information systems: reliability, content, context, and structure; and (2) required preservation mechanisms to ensure that records in its electronic recordkeeping system will be retrievable and useable. (Recommendation 37) Establish a time frame to update the agency’s policies and procedures on retention and management for email to include policies for retaining electronic calendars. (Recommendation 38) The Director of the Peace Corps should take the following three actions: Establish a time frame to update the agency’s electronic information systems inventory to (1) specify technical characteristics necessary for reading and processing the records contained in the system, (2) identify system inputs and outputs, (3) define the contents of the files and records, (4) determine restrictions on access and use, and (5) specify how the agency ensures the timely disposition of records. (Recommendation 39) Establish a time frame to update its policies and procedures to include all of the required electronic information system functionalities for recordkeeping systems. (Recommendation 40) Establish a time frame to update the agency’s policies and procedures to include (1) following records management controls required for electronic information systems: usability, context, and structure and (2) required preservation mechanisms to ensure that records in its electronic recordkeeping system will be retrievable and useable. (Recommendation 41) The Executive Director of Udall Foundation should take the following action: Establish a time frame to develop and maintain an active, continuing agency records management program that includes policies and procedures to provide for effective controls over the creation, maintenance, and use of records in the conduct of current business. (Recommendation 42) Nick Marinos, (202) 512-9342, marinosn@gao.gov In addition to the individual named above, Marisol Cruz Cain (Assistant Director), Anjalique Lawrence (Assistant Director), Elena Epps (Analyst- in-Charge), Roger Bracy, Kami Brown, Christopher Businsky, Alan Daigle, Nancy Glover, Charles Hubbard, Lee McCracken; Brian Palmer, and Monica Perez-Nelson made significant contributions to this report.", "summary": "The Federal Records Act , a subsequent directive, and NARA regulations establish requirements for agencies to ensure the transparency, efficiency, and accountability of federal records, including those in electronic form. In addition, NARA plays an important role in overseeing and assisting agencies' records management efforts. GAO was asked to evaluate federal agencies' implementation of the aforementioned requirements related to electronic records. The objectives were to determine the extent to which (1) selected agencies' policies and procedures address the electronic recordkeeping requirements in the Managing Government Records Directive and the Presidential and FRA Amendments of 2014 and (2) NARA assisted selected agencies in managing their electronic records. To do so, GAO selected 17 agencies and reviewed their records management policies and procedures. GAO also reviewed laws and requirements pertaining to NARA's roles and responsibilities for assisting agencies in managing their electronic records. Further, GAO analyzed NARA guidance and other documents that discussed NARA's efforts in carrying out these responsibilities. Seventeen agencies GAO selected for review varied in the extent to which their policies and procedures addressed the electronic recordkeeping requirements in the Managing Government Records Directive and the Federal Records Act ( FRA ) and its amendments. More specifically, 14 of the 17 agencies established records management programs, while three agencies did not. Of those 14 agencies with established records management programs, almost all addressed requirements related to incorporating electronic records into their existing programs, but many did not have policies and procedures to fully incorporate recordkeeping functionalities into electronic systems, establish controls and preservation considerations for systems, and issue instructions on email requirements (see table). NARA provided guidance and assistance to the selected agencies, including guidance on electronic records management and training. All of the agencies stated that the assistance was generally helpful and that they relied on it to some extent for implementing the key requirements discussed in this report. Further, NARA oversaw the selected agencies' implementation of federal records management regulations through their self-assessment progam. However, NARA had not ensured that the selected small or micro agencies that self-assessed to be at high risk of improper records management in calendar year 2017 were taking appropriate actions to make improvements to their records management programs. NARA officials stated they conduct follow-up with the agencies that report poor scores, but they do not proactively require the agencies to address their weaknesses. Until NARA requires these agencies to develop plans to make necessary improvements, these agencies will likely miss important opportunities to improve their record management practices. GAO is making 40 recommendations to 14 of the 17 selected agencies to improve their management of electronic records. GAO is also recommending that NARA (1) require high-risk smaller agencies to create improvement plans and (2) monitor progress on a regular basis. Six agencies, including NARA, agreed with the recommendations, while 11 did not state whether they agreed or disagreed, or had no comments.", "document_type": "gao"}
{"report": "There are three main pillars of retirement income in the United States: Social Security benefits, employer-sponsored or other retirement savings plans, and individual savings and assets. Social Security is a cash benefit that partially replaces earnings when an individual retires or becomes disabled. The monthly benefit amount depends on a worker’s earnings history and the age at which he or she chooses to begin receiving benefits, as well as other factors. Social Security benefits are paid to workers who meet requirements for the time they have worked in covered employment, that is, jobs through which they have paid Social Security taxes. To qualify for retirement benefits, workers must typically have earned a minimum of 40 quarters of coverage (also referred to as credits) over their lifetime. Social Security benefits are calculated based on the highest 35 years of earnings on which workers paid Social Security taxes. Those who wait until the full retirement age, which has gradually increased from 65 to 67, to claim Social Security receive unreduced benefits. Social Security provides larger benefits, as a percentage of earnings, to lower earners than to higher earners. Social Security makes up a large portion of income for many older Americans, and older Americans face greater risk of poverty without Social Security benefits. We previously reported that data from the Federal Reserve Board’s most recent Survey of Consumer Finances showed that in 2016, among households age 65 and over, the bottom 20 percent, ranked by income, relied on Social Security retirement benefits for 81 percent of their income, on average. According to a 2014 Census report, about 43 percent of people age 65 or older would have incomes below the poverty line if they did not receive Social Security. The most common type of employer-sponsored retirement plan is a defined contribution plan, such as a 401(k) plan. Defined contribution plans generally allow individuals to accumulate tax-advantaged retirement savings in an individual account based on employee and employer contributions, and the investment returns (gains and losses) earned on the account. Individuals or employers may make contributions up to statutory limits. Individuals typically pay fees for account maintenance, such as investment management or record keeping fees. An employee may take funds out of the account prior to age 59 ½, but will owe taxes, possibly including an additional tax, for early withdrawal. Workers can also save for retirement through an individual retirement account (IRA). IRAs allow workers to receive favorable tax treatment for making contributions to an account up to certain statutory limits. Most IRAs are funded by assets rolled over from defined benefit and defined contribution plans when individuals change jobs or retire. Individuals must have taxable earnings to contribute to an IRA, and the amount of their contribution cannot exceed their earned income. IRAs also have account maintenance fees, which are generally higher than those charged to participants in employer-sponsored plans. IRAs are a major source of retirement assets. As we reported in 2017, IRAs held about $7.3 trillion in assets compared to $5.3 trillion held in defined contribution plans. Individuals may augment their retirement income from Social Security and employer-sponsored plans with their own savings, which includes any home equity and other non-retirement savings and investments. Non- retirement savings and investments might include income from interest, dividends, estates or trusts, or royalties. Through our review of literature and interviews with experts, we identified several federal and state efforts that may provide support to caregivers: Medicaid. This federal-state health financing program for low-income and medically needy individuals is the nation’s primary payer of long- term services and supports for disabled and aged individuals. Within broad federal requirements, states have significant flexibility to design and implement their programs based on their unique needs, resulting in 56 distinct state Medicaid programs. Under Medicaid requirements governing the provision of services, states generally must provide institutional care to Medicaid beneficiaries, while home and community based long-term services and supports is generally an optional service. All 50 states and the District of Columbia provide long-term care services to some Medicaid beneficiaries in home and community settings under a variety of programs authorized by statute. Some of these programs include self-directed services under which participants, or their representatives if applicable, have decision- making authority over certain services and take direct responsibility for managing their services with the assistance of a system of available supports. Under one such program, participants can hire certain relatives to provide personal care services. Tax-related provisions. Caregivers may be able to use dependent care accounts, tax credits, or tax deductions for financial assistance with caregiving costs. Dependent care accounts are set up through an employer and allow individuals to set aside pre-tax funds to care for a qualifying individual, such as a spouse who is unable to care for himself or herself. As an example of a tax credit, beginning in 2018, caregivers may be eligible to obtain a $500 non-refundable credit for qualifying dependents other than children, such as a parent or a spouse. As an example of a deduction, taxpayers may deduct the cost of qualifying medical expenses. The Family and Medical Leave Act of 1993 (FMLA). This act generally provides up to 12 weeks of unpaid leave per year for eligible employees to help care for a spouse, child, or parent with a serious health condition or for their own serious health condition, among other things. Employees are generally eligible for FMLA leave if they have worked for their employer at least 12 months, at least 1,250 hours over the past 12 months, and work at a worksite where the employer employs 50 or more employees or if the employer employs 50 or more employees within 75 miles of the worksite. The Older Americans Act of 1965. This act was passed to help older individuals remain in their homes and includes grant funding for services for older individuals. Since its reauthorization in 2000, the Older Americans Act of 1965 has provided supports for caregivers through programs such as the National Family Caregiver Support Program. This program provides grants to states to fund a range of supports to help caregivers. For example, the program provides access to respite care. According to the National Institute on Aging, respite care provides in-home or facility-based care by a trained care provider to give the primary caregiver short-term relief from caregiving. Paid sick leave. This form of leave provides pay protection to workers for short-term health needs, and paid family leave is used by employees for longer-term caregiving. No federal sick or paid family leave policy exists. However, as of March 2019, 10 states (AZ, CA, CT, MA, MD, NJ, OR, RI, VT, WA) and the District of Columbia (DC) have guaranteed paid sick days for specific workers, according to the National Partnership for Women and Families, with eligibility varying by state. As of February 2019, six states (CA, NJ, NY, RI, MA, and WA) and DC have paid family leave laws in effect or soon will be implementing them, according to the National Partnership for Women and Families. The covered family relationships, wage replacement rate, and funding mechanism of these programs vary by state. An estimated 45 million people per year provided unpaid eldercare from 2011 through 2017, according to American Time Use Survey (ATUS) data. About 26 million people—roughly one in 10 adults in the U.S. population—cared for their parent or spouse, and about 22 million people cared for other relatives, such as grandparents, aunts and uncles, or non- related adults (see fig. 1). Among parental and spousal caregivers, 88 percent (about 23.4 million people) provided care to a parent, and 12 percent (3.2 million people) provided care to a spouse. About 7.4 million parental or spousal caregivers (close to 30 percent) provided care for more than one person. We examined several demographic and economic characteristics of parental and spousal caregivers compared to the general population. Women and men were almost evenly divided in the general population, but women were more likely than men to be parental or spousal caregivers, according to ATUS data from 2011 through 2017. Women made up 52 percent of the general population, but represented 56 percent of parental caregivers and 63 percent of spousal caregivers (see fig. 2). Parental caregivers were younger than spousal caregivers, but both groups were older, on average, than the general population. The average age of parental caregivers was 50, and the average age of spousal caregivers was 70, according to ATUS data. While about half of the general population was under 45, most parental caregivers were over 50, and most spousal caregivers were over 65 (see fig. 3). While far fewer in number, spousal caregivers were considerably older than parental caregivers. Almost three-quarters of spousal caregivers were over Social Security claiming age for full retirement benefits compared to less than 10 percent of parental caregivers. The racial/ethnic distribution of parental and spousal caregivers was consistent with the general population in that a significant majority of caregivers were white. When compared to the general population, caregivers were more likely to be white and less likely to be minorities. The distribution in the marital status of parental caregivers was similar to the general population in that most people in the general population were married, followed by single, divorced, widowed, and separated. About two-thirds of parental caregivers were married, and not surprisingly, almost all spousal caregivers were married. Parental caregivers were more educated than spousal caregivers and the general population, according to ATUS data. For example, 38 percent of parental caregivers had completed college compared to 26 percent of spousal caregivers (see fig. 4). These differences may reflect that spousal caregivers are generally older and may come from a generation in which women were less likely to attend college. Parental caregivers were more likely to be employed and to have higher earnings than spousal caregivers and those in the general population. Over 70 percent of parental caregivers worked either full-time or part-time compared to 26 percent of spousal caregivers and 62 percent of the general population (see fig. 5). This may be related to the older age of many spousal caregivers, as the percentage of spousal caregivers out of the labor force was about equal to the percentage over age 65. Further, parental caregivers tended to earn higher wages than spousal caregivers. Among wage and salary workers with a single job, parental caregivers earned $931 per week while spousal caregivers earned $513 per week, and the general population earned $743 per week, according to ATUS data. We found that women who provided parental or spousal care were more likely to be employed part-time and to have lower earnings than men who were parental or spousal caregivers (see fig. 6). Women caregivers were less likely to work than men caregivers, but among those who worked, women caregivers were more likely to work part-time, according to ATUS data. For example, among parental caregivers, 66 percent of women were employed either full-time or part-time compared to 77 percent of men, but 17 percent of women worked part-time compared to 10 percent of men. Similarly, among spousal caregivers, women were less likely to be employed than men. In addition, differences in the employment status of women and men caregivers are similar to differences between women and men in the general population. When we examined the distribution of men and women caregivers in earnings quartiles, we found that men caregivers were more likely to be among the highest earners. For parental caregivers, 43 percent of men compared to 25 percent of women were among the highest earners. For spousal caregivers, 22 percent of men compared to 14 percent of women were among the highest earners. Regression results show that these differences between men and women caregivers were significant for parental and spousal caregivers, and remained significant after controlling for caregiver age and years of education. In terms of education, women parental caregivers were more likely to have completed some college or more (69 percent) while women spousal caregivers were less likely to have done so (50 percent) compared to men parental and spousal caregivers (63 and 56 percent, respectively). Similar to the education levels of the parental and spousal caregiving populations generally, these results may reflect generational differences. Spousal caregivers were more likely to provide care daily compared to parental caregivers, and parental caregivers who lived in the same house as their parents were unsurprisingly more likely to provide care daily than those who did not, according to ATUS data. The vast majority of spousal caregivers (81 percent) provided care on a daily basis compared to 21 percent of parental caregivers. When we examined the frequency of caregiving among those who lived in the same house as their parents, we found that about 63 percent of these parental caregivers provided care daily, suggesting there is a positive relationship between frequency of care and cohabitation (see fig. 7). Experts we spoke with said the frequency of care may depend on whether the care recipient has a disability and the type of disability. For example, someone with a severe disability may be more likely to require care daily compared to someone with a less severe disability. Women and minorities tended to provide care more frequently. Among parental and spousal caregivers, 30 percent of women provided care daily compared to 25 percent of men. While the majority of caregivers were white, as discussed above, black and Hispanic caregivers were more likely to provide daily care than white caregivers—35 percent of black caregivers and 39 percent of Hispanic caregivers provided care daily compared to 26 percent of white caregivers (see fig. 8). While most parental caregivers were married, parental caregivers who were never married were more likely to provide daily care than divorced, widowed, separated, and married caregivers. Daily caregiving may be concentrated among those with the fewest financial resources. Parental or spousal caregivers with lower levels of education and earnings were more likely to provide care daily (see fig. 9). For example, 48 percent of caregivers without a high school degree provided care daily compared to 21 percent who had completed college. Those who worked part-time were also more likely to provide care daily compared to those who worked full-time (27 percent versus 18 percent, respectively). Those who provided care daily were also more likely to be among the lowest earners. In addition to examining frequency of care, we also found that most parental or spousal caregivers provided care that lasted several years. The majority of parental or spousal caregivers (54 percent) provided care for at least 3 years, and 16 percent provided care for 10 years or more. On average, parental or spousal caregivers provided care for about 5 years, regardless of gender. The number of years of care provided increased with the age of the parental or spousal caregivers (see fig. 10). Women caregivers, spousal caregivers, and Hispanic caregivers were more likely to provide long-term daily care. Among parental or spousal caregivers who said they provided care daily and provided care for at least 5 years, 61 percent were women. In comparison, among all parental and spousal caregivers, 56 percent were women. Twenty-nine percent of spousal caregivers provided long-term daily care compared to 8 percent of parental caregivers. In addition, 16 percent of Hispanic caregivers provided long-term daily care compared to 10 percent of whites and 12 percent of blacks. An estimated 68 percent of working parental and spousal caregivers said they experienced at least one of eight job impacts about which they were asked, according to our analysis of data used in the 2015 National Alliance for Caregiving and AARP sponsored study, Caregiving in the U.S. The highest percentage of parental and spousal caregivers—more than half—reported that they went in late, left early, or took time off during the day to provide care (see fig. 11). Spousal caregivers were more likely to experience adverse job impacts than parental caregivers. About 81 percent of spousal caregivers said they experienced at least one of the eight job impacts they were asked about compared to 65 percent of parental caregivers. Spousal caregivers were more likely to reduce their work hours, give up work entirely, or retire early, compared to working parental caregivers. For example, 29 percent of spousal caregivers said they went from working full-time to part-time or cut back their hours due to caregiving, compared to 15 percent of parental caregivers. Our prior work has reported that some older workers felt forced to retire for professional or personal reasons and that individuals approaching retirement often have to retire for reasons they did not anticipate, including caregiving responsibilities. In addition, our prior work has reported that job loss for older workers, in general, can lead to lower retirement income, claiming Social Security early, and exhaustion of retirement savings. We also found that older workers face many challenges in regaining employment. Consistent with these results, we also found that spousal caregiving was negatively associated with the number of hours caregivers worked. Specifically, spousal caregivers who were ages 59 to 66 worked approximately 20 percent fewer annual hours than married individuals of the same age who did not provide spousal care, according to HRS data from 2002 to 2014. We found that spousal caregivers who were at or near the age of full retirement eligibility had lower levels of IRA assets, non-IRA assets, and Social Security income compared to those who did not provide care. We did not detect the same relationship between parental caregiving and retirement income, which may be due, in part, to the older age of the caregivers we examined. Spousal caregivers at or near retirement age had lower levels of retirement assets and income compared to married individuals who did not provide spousal care. Spousal caregivers tended to have lower levels of IRA assets, non-IRA assets—such as real estate or stocks—and Social Security income than non-caregivers (see table 1). After controlling for certain characteristics of caregivers, we found that spousal caregivers still had less retirement assets and income than non- caregivers. For example, spousal caregivers had an estimated 39 percent less in non-IRA assets than non-caregivers, after controlling for characteristics such as level of education and race/ethnicity. When we compared women and men spousal caregivers, we found both had less in IRA and non-IRA assets than non-caregivers, but only women had less in Social Security income. Specifically, we found that women and men caregivers had 37 to 54 percent less in IRA and non-IRA assets than non-caregivers, after controlling for demographic and other characteristics. However, the effect of spousal caregiving on Social Security income was only significant among women. Women caregivers had 15 percent less in Social Security income than married women who did not provide care. Many older Americans rely on Social Security for a significant portion of their retirement income. Therefore, a lower Social Security benefit could have serious consequences for these individuals’ retirement security. One possible explanation experts offered for why spousal caregivers may have less in retirement income and assets than non-caregivers is that the care recipient may be in poor health, resulting in reduced workforce participation of both members of the household, which could then have a large negative impact on household wealth. This scenario could leave spousal caregivers in a precarious financial situation heading into retirement. We did not find that parental caregivers at or near retirement age had lower levels of retirement assets or income than non-caregivers. We compared the retirement assets and income of parental caregivers to the retirement assets and income of individuals who did not provide parental care and did not find a statistically significant effect of parental caregiving on IRA assets, non-IRA assets, defined contribution balances, or Social Security income. See appendix I for more information on this analysis. We may not have seen a significant effect of parental caregiving for a few reasons. First, because of the scope of the HRS data we used, we limited the analysis to individuals who provided care in the 6 years leading up to ages 65 or 66. Therefore, this analysis does not capture the possible effects of parental caregiving prior to age 59, which may be during the middle of a person’s career or during their peak earning years. Second, similar to spousal caregivers, experts said a caregiver may reduce their workforce participation to care for a parent; however, parental caregiving may not affect household income because married caregivers’ spouses may be able to continue working and offset any lost earnings. In addition, unlike spousal care, parental care may be provided by multiple individuals, so the effect on retirement security may be distributed across siblings. Our analysis could not definitively identify the causal effect or lack of effect of caregiving on retirement income due to three main limitations. First, because caregiving is not random but is a function of an individual’s circumstances, it is difficult to isolate its effect. For example, individuals who provide care may do so because they have jobs that are more flexible, or because they have better family support. Second, there may be other ways of providing care beyond an individual giving their time that were not captured in the HRS data and therefore could not be included in our analysis. For example, a child may provide financial assistance to a parent rather than providing time. However, the HRS does not capture whether financial help to parents was specifically used for caregiving expenses. Third, common to analyses of this type, alternate measures of certain variables may produce different estimates. For example, we controlled for a caregiver’s level of education based on data included in the HRS; however, a measure of education that included the type of education, such as whether the person was a trained caregiver, might have changed our estimates. As a result of these limitations, our estimates may not capture the effect of caregiving on retirement income for the broader population. Our analysis of literature and expert interviews found that parental or spousal caregivers could face several retirement security challenges: Caregivers may have high out–of-pocket expenses. Caregivers may face immediate out-of-pocket expenses that could make it difficult to set aside money for retirement or that could require them to prematurely withdraw funds from existing retirement accounts. These financial burdens can include, for example, travel and medical expenses for a care recipient. AARP’s study, Family Caregiving and Out-of-Pocket Costs, estimated that family caregivers spent an average of nearly $7,000 on caregiving costs in 2016. Caregiving costs amounted to about 14 percent of income for white family caregivers and 44 percent and 34 percent for Hispanic and black caregivers, respectively. Caregivers may reduce their workforce participation. In addition to foregone earnings, caregivers who reduce their workforce participation may also lose access to employer-provided retirement benefits, such as participating in an employer-sponsored 401(k) plan or receiving an employer’s matching contributions. About 68 percent of working parental and spousal caregivers reported job impacts due to caregiving responsibilities, which included reducing their workforce participation. For those who leave the workforce, re-entry can be challenging, and wages and retirement savings can be negatively affected long-term. Caregivers may not contribute to retirement accounts. Caregivers may face challenges contributing to retirement accounts due to caregiving, and some working caregivers may not be eligible for employer-sponsored retirement benefits. For example, some part-time employees may not be eligible to participate in employer-sponsored retirement plans, or some employees may lose access if they reduce their workforce participation. Individual and employer-sponsored retirement accounts serve as important supplements to Social Security as income replacements in retirement. Caregivers may have lower Social Security benefits. Caregivers may have less in Social Security benefits if they reduce their workforce participation. Social Security benefits are calculated using the highest 35 years of earnings. If a caregiver retires after working for 33 years, he or she would have 2 years of zero income in their benefit calculation, which would result in lower benefits throughout retirement compared to what their benefit would have been if they had a full 35- year earnings history. Social Security makes up a large portion of retirement income from many older Americans, so a lower Social Security benefit could have significant consequences for financial security. We identified four policy categories that could potentially address retirement security challenges faced by caregivers. To do so, we identified specific actions that could improve caregivers’ retirement security based on a review of literature and interviews with experts. We then grouped these actions into four categories: 1) decrease caregivers’ out–of-pocket expenses, 2) increase caregivers’ workforce attachment and wage preservation, 3) increase caregivers’ access or contributions to retirement accounts, and 4) increase caregivers’ Social Security benefits. See figure 12 for example actions in each category. Experts we interviewed identified potential benefits of each of the four policy categories. They also identified specific groups of parental or spousal caregivers who could benefit, including women, lower-income caregivers, and working caregivers (see table 2). As discussed previously, women were more likely to provide parental and spousal care, to work part-time, and to have lower earnings than men caregivers. In addition, over one-third of parental caregivers and almost two-thirds of spousal caregivers were in the bottom two income quartiles, and caregivers in the bottom earnings quartile were more likely to provide care daily. Experts also said all four categories have potential costs and challenges (see table 3). Experts identified three implementation issues that would need to be addressed regardless of the policy category. Determining responsibility for implementation. It is unclear who would be responsible for implementing and funding certain actions under each approach, according to experts. Some may require legislative changes, steps by employers, or public-private partnerships that integrate both sectors. The RAISE Family Caregivers Act enacted in January 2018 requires the Department of Health and Human Services (HHS) to develop a strategy, including recommendations related to financial security and workforce issues, to support family caregivers and to convene an advisory council to help develop the strategy. The advisory council will include representatives from federal agencies, employers, state and local officials, and other groups. Between October 12, 2018 and December 3, 2018, HHS sought nominations for individuals to serve on the advisory council. Defining caregiving for benefit eligibility. Experts said some actions may require a definition of caregiving to use in determining eligibility for benefits. Current definitions related to federal caregiving policy vary. For example, FMLA defines a caregiver by specific familial relationships. In contrast, the RAISE Family Caregivers Act defines a family caregiver more broadly as an “adult family member or other individual who has a significant relationship with, and who provides a broad range of assistance to, an individual with a chronic or other health condition, disability, or functional limitation.” Identifying and verifying caregivers. Experts said some actions may require a mechanism for identifying and verifying a caregiver’s status. Experts noted that many caregivers do not identify themselves as such, particularly those caring for a spouse, and therefore do not claim existing benefits. In addition, certain actions may require a decision about whether benefits extend to the primary caregiver or to all caregivers, for example, siblings who may jointly provide care to a parent. Several experts we interviewed said caregivers could benefit more from a retirement system that incorporates actions across the policy categories so that actions can work in tandem to address caregivers’ needs. For example, if caregivers have lower out-of-pocket caregiving costs, they might be able to contribute more to their retirement savings. If caregivers can contribute more to their retirement savings because they have better access to accounts, they might have to rely less on Social Security in retirement. Some experts pointed to Hawaii’s Kupuna Caregivers Program as an example of a program with complementary goals—to alleviate out-of-pocket expenses and reduce barriers to staying fully employed while providing care for a family member. Specifically, according to experts, the program provides a financial benefit of $70 per day for up to 365 days to caregivers who work at least 30 hours a week to spend on respite care, home health care workers, meal preparation, and transportation costs for a care recipient age 60 or older. Although the program is in the early stages of implementation, experts said several states already see it as a model for meeting these two goals. Experts also said it would be helpful to implement actions that address the needs of caregivers in the long- and short-term and across their lifespans. In general, experts said each of the policy categories could help longer-term caregivers more than short-term caregivers. However, they said certain actions to decrease caregivers’ out-of-pocket expenses or to increase workforce attachment could also help in addressing immediate needs. For example, experts said actions such as paid time off and flexible work schedules could help those caring for individuals with acute conditions to attend doctor’s appointments. Experts also said policies should address the needs of caregivers with different levels of workforce attachment. For example, one expert said there are disparate policy impacts to consider depending on whether someone is a salaried worker, an hourly worker, or a caregiver who does not work. Similarly, someone who depends on other types of government assistance, such as Social Security Disability Insurance, may also have different needs. Another expert said the age at which caregiving takes place may impact retirement security; people may be caring for older parents or a spouse at a point in their careers when they are supposed to be catching up on retirement contributions or have peak earnings, so they may not be able to make up for lost time in terms of retirement savings. Finally, several experts mentioned public awareness as critical to helping people understand the implications of caregiving on retirement security. They stressed the importance of financial literacy and making caregivers aware of existing and new benefits. Experts said people are not well informed about their Social Security benefits or their options for private retirement savings. In addition, it can be difficult to understand the long- term impacts of becoming a caregiver, and experts pointed to the need for education about how the decision, along with those to leave the workforce or reduce workforce participation, could affect caregivers’ long- term financial security. One expert noted that education and services that help families proactively think about their financial security and plan for caregiving needs could be useful. Educating the public about what supports exist, new supports as they become available, and eligibility and enrollment procedures, is critical to ensuring caregivers take advantage of available supports. We provided a draft of this report to the Department of Labor, the Department of Health and Human Services, the Department of the Treasury, and the Social Security Administration for review and comment. The Departments of Labor, Health and Human Services, and the Treasury provided technical comments, which we incorporated as appropriate. The Social Security Administration told us they had no comments on the draft report. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretaries of Labor, Health and Human Services, and Treasury, the Acting Commissioner of Social Security, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or jeszeckc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made contributions to this report are listed in appendix IV. The objectives of this review were to (1) examine what is known about the size and characteristics of the parental and spousal caregiving population, including differences among women and men; (2) examine the extent to which parental or spousal caregiving affects retirement security; and (3) identify and discuss policy options and initiatives that could improve caregivers’ retirement security. This appendix provides information about the methods we used to answer these questions. Section I describes key information sources we used, and section II describes the empirical methods we used to answer the first and second research questions and the results of supplementary analyses. To answer our research questions, we analyzed data from three nationally representative surveys—the American Time Use Survey (ATUS), the Health and Retirement Study (HRS), and Caregiving in the U.S.—conducted an extensive literature search, and interviewed relevant experts or stakeholders. This section provides a description of our data sources and the steps we took to ensure their reliability for the purposes of our review. To answer the first objective, we analyzed data collected through ATUS’ eldercare module from 2011 through 2017, the most recent year of data available. The ATUS—which is sponsored by the Bureau of Labor Statistics and conducted by the U.S. Census Bureau—provides nationally representative estimates of how, where, and with whom Americans spend their time. Individuals interviewed for the ATUS are randomly selected from a subset of households that have completed their eighth and final month of interviews for the Current Population Survey (CPS). Starting in 2011, the ATUS began asking questions about eldercare. We weighted the data and calculated relative standard errors to reflect CPS guidance on the sample design. A relative standard error is equal to the standard error of a survey estimate divided by the survey estimate. We analyzed data used in the 2015 Caregiving in the U.S. study sponsored by the National Alliance for Caregiving and the AARP Public Policy Institute to estimate job impacts of parental and spousal caregiving for working caregivers. The survey was conducted through online interviews. To identify caregivers, respondents were asked whether they provided unpaid care to a relative or friend 18 years or older to help them take care of themselves. Respondents were also asked to whom they provided care, which allowed us to identify parental and spousal caregivers. We considered someone to be a parental caregiver if they provided care to a parent or a parent-in-law. We considered someone to be a spousal caregiver if they provided care to a spouse or partner. To determine the job impacts of caregiving, respondents were asked whether they were currently employed while providing care or whether they were employed in the last year while providing care and whether they experienced any of the following job impacts as a result of caregiving: Went in late, left early, or took time off during the day to provide care Went from working full-time to part-time, or cut back hours Took a leave of absence Received a warning about performance or attendance at work Gave up working entirely Turned down a promotion Lost any job benefits All estimates derived from random samples are subject to sampling error. All percentage estimates from this survey have margins of error at the 95 percent confidence level of plus or minus 5 percentage points or less, unless otherwise noted. To analyze the effects of caregiving on retirement security, we analyzed data collected through the HRS, a nationally representative survey sponsored by the National Institute on Aging and the Social Security Administration and conducted by the Survey Research Center at the University of Michigan’s Institute for Social Research. This biennial longitudinal survey collects data on individuals over age 50 and contains information on unpaid parental and spousal caregivers. Each biennial period is referred to as a “wave.” The HRS includes both members of a couple as respondents. There are currently 12 waves of core data available from 1992 to 2014 with about 18,000 to 23,000 participants in any given wave. The initial 1992 cohort consisted of respondents who were then ages 51 to 61, and these respondents have been interviewed every 2 years since 1992. New cohorts have been added over time to maintain the representation of the older population from pre-retirement through retirement and beyond. We used data from 2002 to 2014 for our analyses; we did not use data prior to 2002 because data on spousal caregivers were formatted differently. We adjusted asset and income values for inflation. We weighted the data and calculated standard errors to reflect HRS guidance on the sample design. For each of the datasets described above, we conducted a data reliability assessment of variables included in our analyses. We reviewed technical documentation, conducted electronic data tests for completeness and accuracy, and contacted knowledgeable officials with specific questions about the data. We determined that the variables we used from the data we reviewed were sufficiently reliable for the purposes of describing and comparing the caregiving populations to each other or to non-caregivers. We also cited studies conducted by other researchers to supplement our findings; each of these studies was reviewed by two social scientists with expertise in research methodology and was found to be sufficiently methodologically sound for the purposes of supplementing our descriptions or comparisons. To gain an understanding of policy options that could improve caregivers’ retirement security, we reviewed prior GAO work, conducted an extensive literature review of journal articles, working papers, and think-tank studies on caregiving and topics related to retirement security, and conducted preliminary interviews with experts in caregiving or retirement security. Based on this information, we identified specific actions that could affect caregivers’ retirement security, which we categorized into four different categories based on common themes. We then conducted semi- structured interviews with or received written responses from a range of experts and stakeholders—including some of the experts we met with to identify specific policy actions—to obtain their views on the benefits and costs of the specific policy options and approaches we identified, and we also asked them to identify any additional actions. We selected experts and stakeholders who are engaged in research or advocacy around caregiving or retirement issues, or those who might be affected by the actions identified. We also aimed to interview experts or stakeholders who might have different viewpoints regarding the identified actions. See table 4 for a list of the experts or stakeholders we interviewed or received written comments from over the course of our work. This section discusses the quantitative analysis methods we used to describe the characteristics of parental and spousal caregivers and the regression analyses we conducted to estimate the impact of caregiving on retirement security. We used ATUS and HRS data for these analyses. To describe the characteristics of parental and spousal caregivers, we conducted descriptive analyses to examine differences between parental and spousal caregivers and the general population. For all univariate and multivariate statistics calculated using the ATUS data, we constructed variance estimates using replicate weights. The ATUS eldercare module defines caregiving as “assisting or caring for an adult who needed help because of a condition related to aging.” The eldercare module contains one observation per eldercare recipient, and for each recipient, includes information about the duration of care provided to the recipient, the age of the recipient, the relationship of the recipient to the care provider, and whether the care recipient and the care provider share a household. To analyze data on eldercare providers rather than recipients, we restructured the data into a single observation per care provider. While any given care provider could provide care to multiple recipients, we defined care provider types as follows: Spousal caregivers were those who provided care to a spouse or cohabiting domestic partner, regardless of whether they also provided care to another person. Parental caregivers were those who provided care to a parent or parent-in-law, regardless of whether they also provided care to another person. Caregivers of another relative were those who provided care to someone related to them (such as a grandparent or aunt or uncle), regardless of whether they also provided care to another person. Caregivers of a non-relative were those who provided care to an unrelated person, such as a friend or neighbor, regardless of whether they also provided care to another person. Data on frequency of care—how often a respondent provided eldercare— is collected once for each care provider, rather than for each recipient, and therefore did not require restructuring. However, as noted above, data on the duration of care—how long a respondent provided care—is collected for each care recipient. Therefore, we analyzed the duration of care for the relevant care recipient (parent or spouse) using the same caregiver types as described above. For example, if someone provided both parental and spousal care, the duration of care for the relevant recipient would be used. We conducted descriptive analyses to examine parental and spousal caregivers’ characteristics including gender, age, race and ethnicity, marital status, level of education, employment status, and earnings. The following are important considerations of these analyses: Age. We examined caregivers who provided care to an adult recipient of any age, and, except where indicated in the text, we compared the characteristics of adult caregivers to the general adult population of all ages. We used four age categories (15 to 44, 45 to 50, 51 to 64, and 65 and older). We chose these age groups so that we could examine the characteristics of care providers with a similar age profile to those we examine in our analysis of household income and assets. Presence of a living parent. We did not have information in the ATUS to determine whether those who provided parental care had living parents; therefore, our analyses included all parental caregivers who said they provided care to a parent or parent-in-law within the past three to four months, even if the parent was deceased by the time of their interview. Certain analyses, where indicated in the text, control for the presence of a parent in the respondent’s household. Earnings. ATUS provides current information on respondent’s usual weekly earnings at their main job. Because we did not have current information on earnings from all jobs, for this analysis only, we restricted the sample to those respondents who have a single job. Because we did not have current information on self-employment income, we restricted our analysis of earnings to those respondents who are wage and salary workers. In our report, we present data on the unadjusted demographic and economic characteristics of caregivers and the general population. We present the unadjusted characteristics so that readers can view the actual demographic profile of caregivers. However, we also conducted logistic regression analyses that predict the likelihood of caregiving as a function of various demographic and economic characteristics and found that most characteristics are qualitatively similar in the multivariate and univariate context. Our independent variables for this multivariate analysis were age, education, gender, marital status, race, ethnicity, and labor force status—employed, unemployed, or not in the labor force. Where indicated, as mentioned above, we included a categorical variable for whether the respondent’s parent lives in the respondent’s household. Where indicated, we included quartiles of usual weekly earnings; in logistic regressions that included weekly earnings as an independent variable, the analyses were restricted to wage and salary workers with a single job. See appendix III for more detail about these logistic regression analyses. To analyze the impact of caregiving on retirement assets and income, we compared the assets and retirement income of caregivers and non- caregivers. We conducted separate analyses for each type of care, as described below. To determine the effect of spousal caregiving on retirement security, we took two approaches: 1. We conducted descriptive analyses to examine differences between spousal caregivers and non-caregivers in terms of assets at or near retirement and Social Security income during retirement. We also examined differences between spousal caregivers and non-caregivers in terms of work, education, and health status of both the person providing and the person receiving care. 2. We conducted regression analyses to examine whether observed differences in assets and Social Security income were still statistically significant when we controlled for these differences in the spousal caregiving and non-caregiving populations. In order to construct our analysis sample of spousal caregivers, we took the following steps. First, we identified married individuals at ages 65 or 66. We chose these ages because they are at or near the full retirement age at which individuals can receive unreduced Social Security benefits. We then identified the respondents that provided spousal care in the current wave or in the prior two waves of data, a 6-year period of time. To determine whether someone provided spousal care, the HRS asks the respondent whether they received help with activities of daily living (ADLs) or with instrumental activities of daily living (IADLs) and who helped with these activities. If the respondent indicated that their spouse or partner provided help, we then identified that person as a spousal caregiver. This resulted in a sample of about 5,000 observations. We found that about 10 percent of the sample provided spousal care in the 6 years we examined. We also obtained information on the asset levels, hours worked, and other descriptive attributes at ages 65 or 66. To determine the level of Social Security retirement income, we looked ahead to the household’s Social Security income at age 71 using data from future waves of the HRS because some individuals may receive benefits at a later age. We found differences between spousal caregivers and non-spousal caregivers, and differences were often statistically significant (see table 5). As the table shows, spousal caregivers tended to have lower asset levels—IRA assets, non-IRA assets, or defined contribution account balances—as well as lower levels of Social Security income. Although the asset levels of spousal caregivers did not increase as much as for non-caregivers, the differences were not statistically significant. Spousal caregivers also tended to work fewer hours, were less likely to have a college degree, and were more likely to be in self-reported poor or fair health. Spouses receiving care also had different characteristics than spouses not receiving care, indicating that the care recipient also could affect household assets. Spouses receiving care tended to work less and to be in poorer self-reported health. Spouses receiving care also worked fewer hours—1,100 compared to 2,700 for spouses who did not receive care (see table 5). About 66 percent of spouses that received care were in self-reported fair or poor health, as opposed to 15 percent of those who did not receive care. We also compared differences between spousal caregivers and non- caregivers by gender (see table 6). We found some of the same differences between men and women spousal caregivers and non- caregivers as we did among spousal caregivers and non-caregivers more generally. However, there were also additional differences. For example, among women, growth in assets was larger among caregivers, and was statistically significant. However, differences in the cumulative hours worked was not statistically significant. In order to investigate whether observed differences in retirement assets or income might be due to factors other than caregiving, we controlled for additional variables using a multiple regression. Specifically, we generated a binary variable which took the value of one if the respondent had provided spousal care and took the value of zero if not and examined the estimated coefficient on this variable. We ran six different regression models for each of the assets, with six different sets of controls, in addition to the spousal caregiving variable. The different models are as follows, with each building on the model prior. Unless otherwise noted, the findings presented in the report are from model 5. Model 1 estimated the differences, with only controls for the year of the wave. This helps control for the effects that would be experienced by all retirees in that year, like an economic recession. Model 2 included the controls from model 1 and also whether the person has a college degree. This helps control for the effects of education on assets and income. Model 3 included the controls from models 1 and 2 as well as earnings for the respondent in the period before we observed them caregiving. This helps control for caregivers having lower earnings before caregiving, which could affect assets and income. Model 4 included the controls from models 1, 2, and 3 and also demographic characteristics, such as race and ethnicity, which can be associated with assets or income. Model 5 included the controls from models 1, 2, 3, and 4 and also controlled for the self-reported health of the potential caregiver. Model 6 included the controls from models 1, 2, 3, 4, and 5 and also controlled for the self-reported health of the potential care recipient. Having a spouse in poor health might affect assets or income, even if no caregiving was provided. We estimated effects on four different types of assets and income at ages 65 and 66: IRA assets, non-IRA assets, defined contribution balances, and Social Security income (see table 7). We took the logarithm of the value before running the regression to normalize the distribution. We also considered the possibility that caregiving might not only affect the level of assets, but might affect the accumulation or growth of assets. We did that by including models that estimated the effect on the growth of IRA and non-IRA assets. The table below shows the parameter estimates of the effect of spousal caregiving with different levels of controls or dependent variables. In the table, the columns represent the different models (1 through 6). The rows represent different dependent variables—different types of assets or Social Security income for which we estimated the effect of spousal caregiving. In the table, the upper panel shows the effects on women’s assets and income based on caregiving. The middle panel shows the effects on men’s assets and income based on caregiving, and the final panel shows the effect when the men’s and women’s samples were pooled. As the table shows: For women, men, and when the sample was pooled, we found significant negative effects of spousal caregiving on both IRA and non-IRA assets. However, the coefficient decreased in magnitude when we added additional controls. For example, when we controlled for the health of the person receiving the help, the coefficient almost fell by half, from about .5 to about .25 in the case of non-IRA assets. This indicates that it is difficult to differentiate the effect of spousal caregiving from the effect of having a spouse in poor self-reported health. For women, men, and when the sample was pooled, we found significant negative effects of spousal caregiving on Social Security income. But for men, the effect was only significant at the 10 percent level for models with fewer controls. In addition, when we added controls for demographics and health, the effect for men no longer was significant. For the growth of assets, we found negative effects for non-IRA assets for women, but not for men and not for the pooled sample. However, the effects were only significant at the 10 percent level and not significant when we controlled for the health of the care recipient. In addition to the regression coefficients, we also calculated the differences in percent terms, which may be easier to interpret (see table 8). We found results that were strongest when comparing women spousal caregivers to women who did not provide spousal care. The effect for women was resilient to the inclusion of controls. In the model that included the health of the recipient (model 6), the effect ranged from a 40 percent reduction in IRA assets, to an 8 percent reduction in household Social Security income. For men, we found effects for IRA assets, but the effects for Social Security income were not resilient to the inclusion of controls besides the education of the recipient. To determine the effect of parental caregiving on retirement security, we conducted descriptive analyses to examine differences between parental caregivers and non-caregivers in terms of assets at or near retirement age and Social Security income during retirement. In order to construct our analysis sample of parental caregivers, we took the following steps. First, we identified individuals at age 65 or 66 who had living parents or parents-in law. We made this restriction because having living parents at ages 60 to 66 (and the opportunity to provide care) might be associated with higher socio-economic strata. Therefore, we did not want to compare caregivers to those who did not provide care because their parents were deceased. We then identified the respondents that provided parental care in the current wave or in the prior two waves of data. To determine who is a parental caregiver, the HRS asks respondents two separate questions. The first asks whether a respondent spent a total of 100 hours or more since their last interview or in the last 2 years helping a parent or parent-in-law with basic personal activities like dressing, eating, or bathing. The second question asks whether a respondent spent a total of 100 hours or more since their last interview or in the last 2 years helping a parent or parent-in-law with other things, such as household chores, errands, or transportation. We limited the analysis to those with living parents or in-laws. This resulted in a sample of about 2,499 observations. We found that about 57 percent of the sample provided parental care in the 6 years we examined. Unlike our analysis of spousal caregivers, we found that parental caregivers had higher levels of assets at or near retirement than non- caregivers, but differences between parental caregivers and non- caregivers were not statistically significant (see table 9). The following tables provide information about the characteristics of various types of eldercare providers. Table 13 shows the adjusted odds of providing care for people with different economic and demographic characteristics, from multivariate analyses. Models 1, 2, 3 and 4 show the adjusted odds of providing parental care, and models 5 and 6 show the adjusted odds of providing spousal care. Model 1 estimates the probability of providing parental care as a function of gender, age, marital status, race, education, and labor force status. Model 2 estimates the probability of providing parental care as a function of gender, age, marital status, race, education, and income quartiles. This model is restricted to employed workers, and therefore does not include labor force status as a regressor. Model 3 is identical to model 1, except that model 3 includes an indicator for whether the parental caregiver and the parental care recipient live in the same household. Model 4 is identical to model 2, except that model 4 includes an indicator for whether the parental caregiver and the parental care recipient live in the same household. Model 5 estimates the probability of providing spousal care as a function of gender, age, marital status, race, education, and labor force status. Model 6 estimates the probability of providing spousal care as a function of gender, age, marital status, race, education, and income quartiles. Like model 2, this model is restricted to employed workers, and therefore does not include labor force status as a regressor. In addition to the contact named above, Erin M. Godtland (Assistant Director), Nisha R. Hazra (Analyst-in-charge), Benjamin Bolitzer, Jessica Mausner, and Rhiannon C. Patterson made key contributions to this report. Also contributing to this report were Susan Aschoff, Deborah Bland, Justin Fisher, Avani Locke, Michael Naretta, Mimi Nguyen, Rachel Stoiko, Shana Wallace, and Adam Wendel.", "summary": "According to the U.S. Census Bureau, the number of people in the United States over age 65 is expected to almost double by 2050. As Americans age, family caregivers, such as adult children and spouses, play a critical role in supporting the needs of this population. However, those who provide eldercare may risk their own long-term financial security if they reduce their workforce participation or pay for caregiving expenses. GAO was asked to provide information about parental and spousal caregivers and how caregiving might affect their retirement security. This report (1) examines what is known about the size and characteristics of the parental and spousal caregiving population, including differences among women and men; (2) examines the extent to which parental or spousal caregiving affects retirement security; and (3) identifies and discusses policy options and initiatives that could improve caregivers' retirement security. GAO analyzed data from three nationally representative surveys; conducted an extensive literature review; and interviewed experts who are knowledgeable about caregiving or retirement security, engaged in research or advocacy around caregiving, or represent groups that might be affected by the identified policy approaches. An estimated one in 10 Americans per year cared for a parent or spouse for some period of time from 2011 through 2017, and women were more likely than men to provide care, according to Bureau of Labor Statistics survey data. Both parental and spousal caregivers were older than the general population, with spousal caregivers generally being the oldest. In addition, spousal caregivers were less likely to have completed college or to be employed, and they had lower earnings than parental caregivers and the general population. Most parental and spousal caregivers provided care for several years, and certain groups were more likely to provide daily care, including women and minorities. Some caregivers experienced adverse effects on their jobs and had less in retirement assets and income. According to data from a 2015 caregiving-specific study, an estimated 68 percent of working parental and spousal caregivers experienced job impacts, such as going to work late, leaving early, or taking time off during the day to provide care. Spousal caregivers were more likely to experience job impacts than parental caregivers (81 percent compared to 65 percent, respectively). According to 2002 to 2014 data from the Health and Retirement Study, spousal caregivers ages 59 to 66 had lower levels of retirement assets and less income than married non-caregivers of the same ages. Specifically, spousal caregivers had an estimated 50 percent less in individual retirement account (IRA) assets, 39 percent less in non-IRA assets, and 11 percent less in Social Security income. However, caregiving may not be the cause of these results as there are challenges to isolating the effect of caregiving from other factors that could affect retirement assets and income. Expert interviews and a review of relevant literature identified a number of actions that could improve caregivers' retirement security, which GAO grouped into four policy categories. Experts identified various benefits to caregivers and others from the policy categories—as well as pointing out possible significant costs, such as fiscal concerns and employer challenges—and in general said that taking actions across categories would help address caregivers' needs over both the short-term and long-term (see figure). Several experts also said public awareness initiatives are critical to helping people understand the implications of caregiving on their retirement security. For example, they pointed to the need for education about how decisions to provide care, leave the workforce, or reduce hours could affect long-term financial security.", "document_type": "gao"}
{"report": "The Great Lakes Pilotage Act of 1960 established the system of compulsory pilotage on the Great Lakes. Senate committee reports accompanying the legislation indicate pilotage requirements in the Great Lakes were established because they were viewed as essential to helping ensure maritime safety. The committees also recognized that international coordination between the United States and Canada would be required at a federal level and the act specifically precludes any state, municipality, or local authority from regulating any aspect of pilotage in the waters of the Great Lakes-Seaway. All oceangoing commercial vessels are required to use U.S. or Canadian registered pilots during their transit through regulated waters of the Great Lakes-Seaway. Generally, these vessels are assigned a U.S. or Canadian pilot depending on (1) the order in which they transit a particular area of the Great Lakes-Seaway and (2) their destination port(s). Vessels do not choose which pilot they receive. The U.S. waters of the Great Lakes-Seaway are divided into three pilotage districts, each operated by an association of independent pilots certified by the Coast Guard (see figure 1). The registered pilots only operate within their designated district and do not cross district boundaries. If a vessel needs to cross a district boundary to reach the next port, there will be a change of registered pilots at predetermined locations. Each pilotage district is further divided into “designated” and “undesignated” areas. Designated areas of the Great Lakes-Seaway include areas that are generally more challenging to navigate and require pilots to be fully engaged in the navigation of vessels in their charge at all times. In undesignated areas, which are generally open bodies of water, pilots are required to be “on board and available to direct the navigation of the vessel at the discretion of and subject to the customary authority of the master.” Given the size of the Great Lakes-Seaway, and depending on the port calls planned, registered pilots can be onboard vessels for multiple days. This contrasts with marine pilot transits in most U.S. coastal waters that may be just a few miles each way. Commercial vessels transiting the Great Lakes-Seaway are also generally smaller than many of the vessels that operate at coastal ports. As a result, pilotage fees typically represent a greater proportion of the vessel costs than many larger commercial vessels operating in coastal waters. Pursuant to the Great Lakes Pilotage Act of 1960, the Coast Guard regulates the operation of U.S. pilotage services and establishes the rates they may charge. These rates are to be established through a full rulemaking process at least every 5 years, but must be reviewed and adjusted on an annual basis. The rate-setting process currently includes a 10-step methodology generally designed to account for the estimated annual revenues needed by registered U.S. Great Lakes pilots to provide pilotage services and total vessel traffic expected in each of the three U.S. pilotage districts. (See appendix I for further details on the pilotage rate-setting methodology.) Among other regulatory roles, the U.S. Coast Guard is also responsible for developing competency standards for pilot training and issuing pilot registrations, providing oversight of the pilot associations, and determining the total number of authorized pilots operating in the U.S. waters of the Great Lakes-Seaway. For the 2019 shipping season, 54 U.S. pilots were authorized to serve the Great Lakes- Seaway. The Great Lakes Pilotage Advisory Committee (GLPAC) was established in November 1998 to provide advice and make recommendations to the Coast Guard on matters relating to Great Lakes pilotage. The GLPAC, which meets at least once annually, is comprised of seven members that include the presidents of the three U.S. Great Lakes-Seaway pilotage districts; three members that represent the ports, shipping industry, and vessel operators, respectively; and one member with a finance and accounting background that is selected by unanimous vote of the other six members. The number of U.S. pilots in the Great Lakes-Seaway decreased from 44 in 2007 to 36 in 2014, which, according to the Coast Guard, resulted in pilot shortages and contributed to shipping delays. In 2016, the Coast Guard initiated a number of changes to its pilotage rate-setting methodology that were intended, in part, to provide sufficient pilot compensation to attract, hire, and retain appropriate numbers of qualified Great Lakes pilots. As shown in Figure 2, after continuing to increase between 2014 and 2016, hourly rates for U.S. pilotage services in 4 of the 6 pilotage areas of the Great Lakes-Seaway were reduced for the 2017 shipping season. Since 2017, they have increased by about 10 percent annually. According to the Coast Guard, these hourly rates are intended to generate the revenues needed to cover the annual operating expenses of the pilot associations; compensate working pilots; maintain infrastructure, such as pilot boats and dispatch equipment; and train new pilots. In May 2016, shipping industry stakeholders filed a complaint in the U.S. District Court for the District of Columbia contesting specific elements of the Coast Guard’s 2016 rate-setting methodology. In November 2017, the court dismissed 3 of the 5 original claims and found for the industry plaintiffs for the two remaining claims. In March 2018, the court remanded the matter to the Coast Guard to address those two claims while leaving the 2016 rule in place. In November 2018, a coalition of shipping industry stakeholders filed an additional complaint challenging the underlying data and decision-making process used by the Coast Guard for determining the 2018 Great Lakes pilotage rates. This case is still pending before the court. The Coast Guard uses several mechanisms to obtain stakeholder input on the Great Lakes Pilotage Program, which stakeholders have used to raise a number of issues to the Coast Guard’s attention. Some of the mechanisms are more formal and include obtaining stakeholder input on proposed rule changes and at annual meetings, while other mechanisms are informal and are employed on an as-needed basis. Since 2016, shipping industry stakeholders and pilots have identified a number of issues, or suggestions, they would like to see integrated within the Great Lakes Pilotage Program. Issues identified by shipping industry stakeholders relate, in large part, to the financial impacts associated with the Coast Guard’s methodology for calculating pilotage rates, as well as other areas where enhanced transparency or oversight is suggested. Issues identified by pilots and their representatives include updating the list of “designated waters” to include areas like Great Lakes ports and addressing changes that may be needed to respond to the increasing volume and variety of vessels needing Great Lakes pilotage services, such as cruise ships. The Coast Guard uses several mechanisms to obtain stakeholder input on the Great Lakes Pilotage Program. Formal mechanisms include obtaining stakeholder comments during the rulemaking process and soliciting input during annual meetings of the Great Lakes Pilotage Advisory Committee. According to the Coast Guard, additional inputs are also provided more informally during ad-hoc communications and operational coordination efforts. The federal rulemaking process represents a key mechanism by which the Coast Guard obtains stakeholder input regarding proposed changes to annual rates pilots may charge for services. Pursuant to the Administrative Procedure Act, the Coast Guard publishes a notice of proposed rulemaking in the Federal Register and allows a minimum of 30 days for public comment on any applicable changes to the rate-setting methodology and proposed pilotage rates. According to Coast Guard Great Lakes Pilotage Program officials, public participation is essential to the rulemaking process and they consider all comments and information received. In the final rule published to the Federal Register, the Coast Guard summarizes the nature of the public comments received on the notice of proposed rulemaking and characterizes how the comments were incorporated into the final rule, as applicable. For example, the 2018 Final Rule summarizes the comments received in eight different categories, including pilot compensation benchmarks and staffing model calculations. According to Coast Guard officials, they have historically received about five to seven comments each year. However, they received nearly 60 comments regarding the proposed rulemaking in 2016 given the broader scope of revisions and the higher rate of pilot compensation proposed in that year. As previously stated, the GLPAC is to meet at least once annually to provide advice and make recommendations to the Coast Guard on matters relating to Great Lakes pilotage. This committee is governed by the Federal Advisory Committee Act, which calls for a published agenda, public participation, and a written transcript of the proceedings. Our review of 2017 and 2018 GLPAC meeting transcripts indicate the meetings were well-attended and provided a venue for sharing a variety of ideas and perspectives; as well as for providing specific input to the Coast Guard. In addition to the annual GLPAC meetings, Coast Guard officials also noted that GLPAC members participate in scheduled phone calls to discuss pertinent matters—such as a discussion of executive orders or revised regulations—on an as-needed basis. According to the Coast Guard, since 2013 there have been up to three GLPAC meetings per year, ranging in length from 5 hours to 2 days. Coast Guard Great Lakes Pilotage Program officials also stated that GLPAC recommendations from the 2014 meeting were a key input for many of the rate-setting methodology changes implemented in 2016. Although the Coast Guard is not required to implement them, program officials commented that considerable weight is given to GLPAC-issued recommendations. At the September 2018 meeting, the Committee developed three recommendations addressing issues related to the billing dispute process and issuance of temporary registrations to applicant pilots. According to the Coast Guard, these recommendations are still being considered for future action. Coast Guard program officials reported that they have extensive ad-hoc communications with shippers, pilots associations, and their Canadian counterparts to coordinate pilot assignments and help reduce vessel traffic delays on the Great Lakes-Seaway. These stakeholders corroborated their communications with the Coast Guard during our meetings with them. Other venues for information sharing and stakeholder interaction identified by Coast Guard officials include visits to the pilots’ offices to perform oversight functions, meetings with shipping industry representatives and Canadian counterparts (Great Lakes Pilotage Authority) at maritime meetings and conventions; as well as interactions with Coast Guard officials from District 9 (Cleveland, OH), which is responsible for broader Coast Guard activities in the Great Lakes-Seaway. According to these Coast Guard program officials, operational coordination and routine meetings with stakeholders provide ongoing opportunities to obtain input on the Great Lakes Pilotage Program and help inform potential changes that may be needed. Since 2016, when the Coast Guard implemented several significant programmatic changes, shipping industry stakeholders and pilots have identified a number of issues. Collectively, these issues have been the subject of discussion during annual GLPAC meetings, documented in written comments submitted as part of the annual rulemaking process, and included in supplemental correspondence to the Coast Guard and Members of Congress. Issues identified by shipping industry stakeholders relate, in large part, to the financial impacts associated with the Coast Guard’s methodology for calculating pilotage rates, as well as other areas where enhanced oversight is suggested. The key issues cited by shipping industry stakeholders in recent years generally fall into four categories: (1) financial oversight and cost accounting, (2) vessel traffic estimates, (3) pilot compensation and staffing, and (4) billing and dispute resolution. Some of these issues remain the subject of ongoing litigation initiated by a coalition of shipping industry stakeholders against the U.S. Coast Guard. (See appendix II for additional details on selected issues identified by shipping industry stakeholders, including a summary of the specific claims that are in litigation). Financial oversight and cost accounting. Since 2016, shipping industry stakeholders have cited several issues regarding the timeliness and transparency of financial information provided by the U.S. pilot associations that is used during the rulemaking process. These issues include a request for disclosure of individual pilot compensation levels, and additional clarification and transparency regarding the use of the pilot districts’ working capital funds. For example, shipping industry representatives claim that disclosure of individual pilot compensation levels would help ensure that compensation practices remain fair and are not a disincentive to attracting and retaining Great Lakes pilots. At the September 2018 GLPAC meeting, a pilots’ representative noted that this information was previously provided for District 1, but was eliminated due to concerns that the data could be used out of context. For example, this individual stated that although all pilots in his association generally receive the same rate of pay, some may obtain higher annual compensation because of additional days worked. According to Coast Guard officials, they do not collect or retain individual compensation data on pilots; however, they do review such data during visits to the pilot associations’ offices to help ensure fair compensation practices. Vessel traffic estimates. In 2016, the Coast Guard began using a 10- year rolling average of Great Lakes-Seaway vessel traffic volumes to estimate projected vessel traffic for each district in the coming year as part of its annual pilotage rate-setting calculations. According to the Coast Guard, this change was implemented to help reduce rate volatility and remedy traffic overestimates that occurred in the past, largely based on shipping industry projections. However, given the increasing volume of vessel traffic on the Great Lakes-Seaway since the 2008-2009 recession, shipping industry stakeholders contend that the 10-year rolling average represents a significant underestimate of vessel traffic volume. For example, in the 2017 shipping season, vessel traffic in 5 of the 6 pilotage areas of the Great Lakes-Seaway exceeded the estimates (calculated using a 10-year rolling average) by over 25 percent. According to its 2018 Notice of Proposed Rulemaking, the Coast Guard noted that use of the rolling average will result in pilots taking in more revenue than projected in some years, and in other years will result in less revenue. Coast Guard officials believe that, over the long term, this methodology will help ensure infrastructure is maintained and that pilots receive adequate compensation and rest between assignments to enhance pilot retention. Shipping industry organizations challenged the Coast Guard’s use of 10 years of traffic data in the complaint filed with the U.S. District Court for the District of Columbia in November 2018, and that case is ongoing. Pilot compensation and staffing needs. The data sources and methodology used by the Coast Guard to develop a target compensation benchmark for U.S. Great Lakes pilots have been subject to ongoing disagreement among pilots and shipping industry stakeholders for several years. Since 2016, the Coast Guard has used two primary data sources as a basis for comparison—the average compensation of Canadian Great Lakes-Seaway pilots, and compensation data for first mates on domestic Great Lakes vessels (lakers). Shipping industry stakeholders identified concerns with some of the specific adjustments made by the Coast Guard related to both of these data sources and filed complaints in 2016 and 2018 in federal court contesting the Coast Guard’s methodology. A related issue identified by shipping industry stakeholders concerns the number of average pilot working days the Coast Guard uses to determine the number of pilots needed each season. For example, the Coast Guard uses 270 working days as a baseline to calculate pilot compensation figures, but uses 200 working days to calculate staffing requirements so as to account for a 10-day per month rest standard for pilots. The Coast Guard states that this 10-day rest standard is not a requirement and generally does not apply during the busiest times of the season. During the busiest time, pilots generally remain available to work additional days to service the increased vessel traffic on the Great Lakes-Seaway. The 2018 complaint filed by shipping industry stakeholders includes a claim challenging the Coast Guard’s use of a 270-working day assumption, and that case is ongoing. Billing and dispute resolution. Other issues cited by shipping industry stakeholders pertain to billings from pilot associations and the Coast Guard’s dispute resolution process. The primary billing issues cited by shipping industry stakeholders since 2016 include an increase in the number of tug boats requested, as well as cases where double pilotage was employed that shipping industry officials did not believe were necessary. In the case of tug boat usage, pilot representatives acknowledged that there may have been an increase in tug boat usage, but they noted that they do not have any financial incentive to call for the use of tug boats and they only request them, in coordination with the shippers’ agents, when they deem them necessary. According to Great Lakes Pilotage Program officials, the Coast Guard routinely reviews inquiries from shippers on this issue, but noted that decisions to use tug boats remain safety decisions that are made between the vessel operators and the Great Lakes pilots. In contrast, authorizations for double pilotage are provided on a case-by-case basis by the Director of the Great Lakes Pilotage Program. According to the Coast Guard, there were instances in which pilot associations charged for double pilotage without obtaining authorization from the Director and, in such instances, the Coast Guard has ruled in favor of vessel operators with regard to billing disputes. Both of these issues were topics addressed at the September 2018 GLPAC meeting, as well as discussion regarding reasonable time frames for filing billing disputes. According to Great Lakes Pilotage Program officials, some disputes were filed after an extended period of time had elapsed, making it more difficult to adjudicate the issues. For this reason, the Coast Guard reported that it is considering introducing a maximum amount of time allowable for vessel operators to initiate a billing dispute, and corresponding time frames for pilot associations and the Coast Guard to respond and adjudicate, respectively. Issues raised by Great Lakes pilots and their representatives generally include the following categories: (1) recognition of the pilots’ unique qualifications and role, (2) review of “designated waters,” and (3) review of protocols for vessel priorities. Recognition of pilots’ unique qualifications and role: Representatives of the U.S. Great Lakes pilots state that the shipping industry remains overly focused on pilotage costs and may fail to recognize the unique qualifications that registered Great Lakes pilots possess and the fundamental public interest the pilots serve by ensuring the safety of vessel navigation and environmental protection on the Great Lakes- Seaway. The pilots noted that, in addition to the often challenging weather conditions they face, they also serve a security role in that they may be the only U.S. citizen on board to provide situational awareness to U.S. authorities in the event of any suspicious activities given that foreign vessels in the Great Lakes-Seaway can travel close to major infrastructure and U.S. cities. The pilots also stated that it can be easy for the shipping industry to select individual routes and billings to make a case that U.S. pilots charge significantly more than their Canadian counterparts, but they contend that is not an accurate picture of actual system-wide costs. Review of designated waters: Great Lakes pilots commented that “designated water” determinations have not been reviewed for over 50 years and they should be reassessed. In particular, pilots note that increases in the volume and variety of vessels; as well as expanded port infrastructure on the Great Lakes-Seaway since establishment of the Great Lakes Pilotage Program in 1960, warrant the consideration of additional areas as “designated waters,” which are generally more challenging to navigate and require registered pilots to be in full navigational control of the vessels at all times as they transit these designated areas. For example, pilots contend that the Straits of Mackinac and all ports on the Great Lakes-Seaway should be considered designated waters. Coast Guard officials reported that it is their understanding that masters are already relying on pilots to direct navigation in waters such as the Straits of Mackinac. Additionally, the officials stated that the Coast Guard does not have the authority to make these designation changes through regulation; rather, such revisions require a presidential declaration. Review of protocols for vessel priorities: Great Lakes pilots also commented that increases in the volume and variety of vessel traffic on the Great Lakes-Seaway in recent years may necessitate a review of the first-come, first-served standard for assigning pilots to vessels. For example, the pilots note that plans for increasing the volume of cruise ships on the Great Lakes-Seaway may require adjustments to the priority process for assigning pilots given that cruise ships are generally on fixed itineraries and tight timelines. This issue was discussed at the 2018 GLPAC meeting and is the subject of ongoing discussions among the Coast Guard and Great Lakes-Seaway stakeholders. Some shipping industry stakeholders, and a recent report commissioned by the Conference of Great Lakes and St. Lawrence Governors and Premiers, have suggested that it is time to evaluate potential governance alternatives to help ensure the Great Lakes pilotage system is efficient, cost-effective, and better serves the needs of the maritime shipping industry and the public. Some of the proposed alternatives include changes that could be implemented within the existing governance system, such as the consolidation of the three U.S. pilotage districts and a review of some pilotage requirements. Other changes, such as transferring the pilotage rate-setting function from the Coast Guard to another entity, would entail more sweeping reforms and require statutory changes. Finally, some proposals, such as the introduction of competitive pilotage services, would reflect an even more significant change from the existing model of Great Lakes pilotage consisting of federal oversight and economic regulation of independent pilot associations, known as a regulated monopoly. Some shipping industry stakeholders and the report commissioned by the Conference of Great Lakes and St. Lawrence Governors and Premiers suggest that consolidation of the three existing U.S. Great Lakes-Seaway pilotage districts might help reduce administrative costs. According to these sources, such a consolidation could also limit the complexity associated with vessel agents and shippers interacting with multiple pilot associations over the course of a single journey on the Great Lakes- Seaway. Apart from consolidating all three of the existing districts into one, industry stakeholders did not identify any other proposed alternatives for changing the existing district boundaries. According to representatives of the Great Lakes pilots, the expansive area of the Great Lakes-Seaway and natural geographic boundaries lend themselves to maintaining the three pilot associations. The pilot representatives also noted that if the districts were to be consolidated, shippers and agents would lose some degree of localized service currently provided by each district, such as knowledge of local conditions and transit times. It remains unclear to what extent cost savings could be realized through consolidation of the three existing U.S. pilotage districts. According to the pilot association presidents, there are relatively few administrative and support staff employed for such a large geographic area and some perform multiple functions. Specifically, the pilots reported that, collectively, there were 23.5 administrative positions (non-pilots), comprised mostly of 8.5 seasonal dispatchers and 10 pilot boat operators. Assuming that existing pilot boat operations would generally remain consistent following district consolidation, administrative and dispatch services represent the principal source of potential cost savings. Based on our review of the Canadian Great Lakes Pilotage Association (GLPA) model, which operates a single, consolidated administrative office, it is not clear that the number of administrative staff, including dispatchers, would be reduced after consolidation of the three U.S. pilotage districts and associations. For example, during the 2018 shipping season, the Canadian Great Lakes Pilotage Association included 21 administrative positions, of which 10 were designated as dispatchers— which is similar in proportion to the existing U.S. Great Lakes Pilotage dispatcher distribution. It is also important to note that even with a potential consolidation of administrative functions within one U.S. pilotage district; pilots would still be limited to operating within the geographic area where they are licensed. According to pilots and Coast Guard program officials, cross- licensing is generally not feasible for multiple waterways between districts given the extent of local specialized training and knowledge required and is not practiced anywhere else in the United States or the Great Lakes- Seaway. Some shipping industry stakeholders state that a broader review of Great Lakes pilotage requirements may be necessary, particularly the compulsory use of pilots in “undesignated” or open areas of the Great Lakes. According to these stakeholders, such a review is warranted given the significant technology improvements that have occurred since initial passage of the Great Lakes Pilotage Act in 1960. Any proposed changes to the existing pilotage requirements could not be implemented through Coast Guard regulatory changes and would require legislative changes or a presidential declaration. Although a significant portion of a Great Lakes-Seaway vessel transit may occur in “undesignated” open waters, the Coast Guard and pilots’ representatives cited several logistical challenges that would likely occur if pilotage requirements in these areas were revised or eliminated. For example, if a pilot was not on board a vessel in open waters, there likely would be no way to get one on board in the event of severe weather, equipment failure, or other emergency. In addition, the officials noted that if a pilot did not remain on board the vessel for the entire transit, one would still be required to navigate the vessel in and out of each port destination. This would entail additional costs for picking up the disembarking pilot and transporting the pilot to a designated shore location and then later to transport another pilot to the vessel to navigate into port. These additional pilot transfers may require the acquisition of additional pilot boats, which are generally customized and can cost in excess of $1 million. Alternately, each individual port could employ its own registered pilot and make the necessary infrastructure investments, including pilot boats and related dispatch equipment, but the result could be an overall increase in the number of pilots operating in the system, which could also increase pilotage costs. Finally, an increasing number of vessels that otherwise are not compelled to use pilots (e.g., domestic oil tankers) are requesting pilotage services due, in part, to requirements by insurance providers. Because of this increase in the requests for pilotage services, a change in open water pilotage requirements may not result in a reduction in the number of pilots required in some areas of the Great Lakes-Seaway. The report commissioned by the Conference of Great Lakes and St. Lawrence Governors and Premiers cites an opportunity for enhanced input into the governance process through the establishment of an advisory board or other oversight mechanism, such as those used commonly in state pilotage commissions nationwide. According to the report, such a mechanism would provide for increased industry participation in the governance process beyond the consultative inputs currently available through the GLPAC and rulemaking processes, and could include responsibility for the pilotage rate-setting function. The principal advantage cited for this increased level of participation would be to better align pilotage services with user needs. Under this proposal, an advisory board would be formed and the board members would be involved in the full range of pilotage governance functions as generally provided by state pilotage commissions. These responsibilities commonly include safety oversight and related functions, such as selecting individuals for admission into the training program, overseeing the training process, issuing licenses, investigating accidents or pilot complaints, taking disciplinary actions, and establishing pilotage rates. All of these activities are current regulatory functions performed by the Coast Guard and statutory changes would be required to designate a new pilotage regulatory body and delineate these responsibilities. Given that stakeholders we met with generally do not advocate for transferring any of the safety oversight and related regulatory functions from the Coast Guard, for the purposes of this report we will focus on the potential tradeoffs associated with having an advisory board formed that would only take responsibility for the Great Lakes pilotage rate-setting function from the Coast Guard. With regard to the rate-setting function, the introduction of an advisory board to determine pilotage rates may not improve one of the core issues cited by both shipping industry and pilot stakeholders at the most recent GLPAC meeting that was held in September 2018. That is, no matter what entity has responsibility for pilotage rate-setting—a new advisory board or the Coast Guard—such an entity would face similar rate-setting challenges posed by the competing interests of pilots and shipping industry representatives. Further, according to a recent report reviewing the pilotage system in the state of Washington, proposed changes to pilotage rates are often evenly split between shipping industry representatives and pilot representatives and final determinations routinely come down to committee chairpersons or independent board members, sometimes without full transparency regarding how decisions were reached. In contrast, the current GLPAC process provides for considerable input by committee members, stakeholder and public participation, and is documented through publicly available transcripts. Coupled with the rulemaking requirements that incorporate public review and comments, we found that the existing mechanisms represent a fairly transparent system of pilotage rate-setting as compared to the process used by some coastal states. One variation used in some U.S. coastal states to help overcome the challenge of competing stakeholder interests during the pilot rate-setting process is the establishment of an independent rate-setting entity, similar to a public utility commission. In fact, one of the principal recommendations in the Washington report was to transfer the rate- setting function from the state pilotage commission to an independent utility and transportation commission in an effort to establish a more clearly defined, rigorous, and transparent process with enforceable timelines. In many respects, we found that the Coast Guard is currently performing this independent function as its rate-setting process includes many of the characteristics identified as a best practice, such as a defined methodology, clear data submission and review process, and the absence of any direct material interest in the outcome of the rate determinations. While individual stakeholders may not agree with the specific inputs and assumptions used by the Coast Guard, the current process is generally transparent and provides an opportunity for informed stakeholder feedback and identification of any grounds on which they can choose to take legal action. Another option presented by various stakeholders is to transfer pilotage rate-setting authority to another federal entity. Under this scenario, the Coast Guard would retain its jurisdiction over safety and related regulatory functions, but responsibility for pilotage rate-setting would be transferred to another federal entity. One specific entity that has been identified as a potential replacement for the Coast Guard is the Saint Lawrence Seaway Development Corporation (SLSDC). According to some stakeholders we spoke with, the SLSDC would have more of a vested interest in ensuring that pilotage rate changes consider the potential impact of such changes on the viability of commercial shipping in the Great Lakes-Seaway. SLSDC representatives declined to comment specifically on this proposal, but they cited historical precedent to indicate that if SLSDC were statutorily required to assume pilotage rate-setting responsibilities, additional staffing resources would likely be needed. It should be recognized that shipping industry and pilotage stakeholders will continue to have vested interests in each of the rate-setting inputs and assumptions that are used to determine pilotage rates and some degree of contention is likely to remain no matter the entity responsible. In addition, pilots’ representatives previously filed a complaint regarding the transfer of pilotage rate-setting authority from the Coast Guard to the SLSDC in the 1990s, and they told us that they continue to oppose such a move. According to pilot representatives, they are concerned with a potential transfer of the pilotage rate-setting function to SLSDC given its role in trade promotion, which could potentially affect SLSDC’s ability to remain fully independent in this role. Whether the Coast Guard maintains responsibility for pilotage rate-setting or that function is transferred to another federal entity like SLSDC, the continued role of a federal entity in performing the pilotage rate-setting process would ensure that Administrative Procedure Act requirements still apply, thereby retaining transparency and providing stakeholders and the public an opportunity for review and comment. While there may be some potential for redundancy or increased administrative burden on the pilot associations if the safety oversight and pilotage rate-setting functions were split between the Coast Guard and another federal entity, similar division of responsibilities currently exist in the handful of states that use an independent rate-setting entity, such as a public utility commission. It is the Coast Guard’s position that authorizing two federal agencies to oversee different aspects of the Great Lakes Pilotage Program could be challenging. For example, Coast Guard officials noted that a transfer of the rate-setting function may not consider potential impacts to other authorities associated with rate setting, such as limiting the number of pilot pools; prescribing a uniform system of accounts; performing audits; determining the number of pilots to be registered; and establishing conditions for services. The existing model of Great Lakes pilotage consisting of federal oversight and economic regulation of independent pilot associations is referred to as a regulated monopoly. This model of regulating pilotage is employed almost exclusively within U.S. coastal states and is also a common method for delivering marine pilotage services worldwide. However, there is also some precedent for pilots serving as government employees. One reason why this government employee model has been identified as one potential alternative for U.S.-registered pilots in the Great Lakes- Seaway is because a majority of the Canadian pilots that operate in the Great Lakes-Seaway are federal employees. Although making U.S. Great Lakes pilots federal employees could eliminate the need for the Coast Guard to provide administrative and financial oversight of independent pilots, we found that U.S. Great Lakes pilot associations provide many administrative and logistical functions, such as dispatching and pilot transfers, which would need to be assumed by the federal government under this type of alternative model. According to pilots’ representatives, one of the principal impacts of the government employee model would likely be the provision of some financial benefit to the shipping industry, given that taxpayers would potentially be assuming the cost of pilotage salaries, benefits, and retirement-related benefits. Additional costs to the U.S. government would also likely be required to fund initial procurement of existing pilot association infrastructure and assets, such as offices and pilot boats. Another factor to consider in evaluating the pilots as federal employees model involves how the Coast Guard budget process may also affect the future funding and operation of pilotage operations. A significant expansion of the pilotage program staffing and associated resource requirements would likely pose an additional challenge to ensure sufficient annual appropriations are obtained, given the ongoing need to balance funding and resources across the Coast Guard’s 11 statutory missions. According to representatives of the Canadian Great Lakes Pilotage Association, pilotage operations in their jurisdiction are to be financially self-supporting through pilotage tariffs, and the Canadian government does not provide an annual appropriation for this purpose. They noted that government pension benefits are also incorporated into the pilotage rates to help achieve these offsets. Similar mechanisms could also potentially be used to fund the additional costs borne to the U.S. government within a federal employee pilot model. Additional considerations associated with a government employee model include the different compensation and overtime structures, and the potential for reduced flexibility afforded to the government if fewer numbers of pilots are needed due to reduced pilotage demand. For example, according to representatives of the U.S. pilot associations, each pilot presently receives the same compensation for each working day they are available, regardless of seniority. However, the U.S. federal government routinely employs a system of graduated compensation based on years employed and may face difficulties in hiring or terminating pilot employees if necessary due to shifting pilotage demand. Another approach identified within the government employee model is the use of harbor pilots. This option would generally entail pilots working directly for an individual or group of ports as municipal or port employees. According to one pilot representative, the key challenge identified with such an approach is that individual ports would each require its own infrastructure and pilot boats to service incoming vessels, which could represent a substantial investment. In addition, the geography of the Great Lakes and the long transits many times involved present additional hurdles associated with pilot transfers and related logistical support services make the harbor pilot approach less feasible. Shipping industry stakeholders have also proposed that the Coast Guard consider the introduction of some level of competition for pilotage service delivery, which would represent the most significant change to the existing model of pilotage regulation. According to shipping industry stakeholders, the introduction of competition would be intended to provide an additional incentive for pilot associations to contain costs. Some specific mechanisms identified include introducing a competitive bidding process to provide pilotage services under multi-year contracts, or allowing individual pilots or groups of pilots to compete for business from vessel operators. The concept of using some form of competitive bidding to grant multi-year contracts for pilotage service delivery is generally consistent with government cost-containment efforts. However, stakeholders we spoke with were unable to identify any pertinent examples where market competition for pilotage services was currently used within U.S. coastal states to provide a basis for further evaluation of this model. According to the Coast Guard and pilot representatives, several features of the Great Lakes-Seaway pilotage system present challenges for potentially implementing competitive pilotage services in the Great Lakes- Seaway. Most notably, the nature of marine pilotage requires several years of specialized training and local experience that entail significant time and investment to acquire. These requirements generally result in a limited supply of available pilots that could compete for a competitive contract in the same geographical area. This represents a potential barrier to market entry and could lead to a single, entrenched service provider, which may reduce the competitive pressure toward cost containment. Further, if registered pilots did not have the assurance of steady employment in the Great Lakes, there may be increased incentives for them to seek opportunities outside of the region, thereby reducing the overall pool of available pilots. Other mechanisms of pilotage competition, such as allowing individual pilots or pilot associations to compete for business, would represent a fundamental shift from the norms of compulsory pilotage services worldwide. As a representative of the American Pilots Association stated at the September 2018 GLPAC meeting, one of the foundations of the existing regulated monopoly system is that pilots provide services using their independent judgement to ensure marine safety and the public interest and should not be subject to any potential financial incentive or business pressure from a vessel operator. Similar statements can be seen in Florida state statutes, which specify the need for economic regulation of marine pilotage at the state level, rather than competition in the marketplace, to better serve and protect the public health, safety, and welfare. In contrast, shipping industry stakeholders suggest that there are likely comparisons to the deregulation implemented in other industries where public safety is also of paramount concern, such as commercial aviation. However, an evaluation of models of competition used in other industries was outside the scope of our review. An additional challenge noted by pilot representatives is that, in a competitive model, pilots may prefer to pursue customers offering more regular or profitable work rather than operate in a non-discriminate manner as is currently the case under the existing numbered rotation system of pilotage assignment. Along these lines, research conducted by KPMG on international models of marine pilotage, found that although a model “comprised of independent contractor pilots could result in theoretically more competitive rates, the combination of what appears to be relatively the same demand for pilotage services in the market, and the uniqueness of pilot skillsets have resulted in a scenario where competition is limited in reality.” The authors’ findings also suggest that, in the few cases where competitive pilotage was introduced, it was generally unsuccessful; and that absent sufficient oversight, direct competition among pilots could potentially lead to incentives to cut costs through reduced focus on safety and quality of service. In May 2019, we provided a draft of this report to the Department of Homeland Security and the Coast Guard for review and comment. The Coast Guard provided technical comments which we incorporated into the report. We are sending copies of this report to the appropriate congressional committee, the Secretary of Homeland Security, the U.S. Coast Guard, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (206) 287-4804 or AndersonN@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Pursuant to the Great Lakes Pilotage Act of 1960, the Coast Guard regulates pilotage for oceangoing vessels on the Great Lakes—including setting the rates for pilotage services and adjusting them on an annual basis. For the 2018 shipping season, these base pilotage rates ranged from $271 to $653 per pilot hour depending on the specific areas where pilotage service is provided. According to the Coast Guard, the three U.S. pilot associations use this revenue to cover operating expenses, compensate working pilots, maintain infrastructure, such as pilot boats, dispatch equipment, and personal pilotage units; and train new pilots. The Coast Guard uses the following 10-step methodology to calculate revenues needed for each Great Lakes pilotage association based on the estimated volume of foreign vessel traffic for the upcoming shipping season. Step 1 – Recognize previous operating expenses. The Director of the Great Lakes Pilotage Program reviews audited operating expenses from each of the three U.S. Great Lakes pilot associations. This number forms the baseline amount that each association is budgeted. There is a 3-year delay between the year the expenses were incurred and when they are included in the rate-setting calculation. For example, the 2019 pilotage rates are calculated using 2016 operating expenses. Step 2 – Project operating expenses, adjusting for inflation or deflation. The Coast Guard applies 3 years of inflation adjustors to the baseline of operating expenses identified in Step 1. The inflation adjustors routinely used are from the Bureau of Labor Statistics’ Consumer Price Index. Step 3 – Estimate the number of working pilots. The Coast Guard determines the number of working pilots that need to be compensated via collection of pilotage fees. As part of this step, the Coast Guard also uses a “staffing model” to determine how many pilots may be needed for each district to handle expected shipping traffic at the beginning and close of the season. According to the Coast Guard, this number helps inform the Director of the Great Lakes Pilotage Program regarding how many total pilot credentials may be authorized for each district to help meet future demand. Step 4 – Determine target pilot compensation. This step contains two phases to determine the revenue needed for pilot compensation. In the first phase, the Coast Guard determines a target “compensation benchmark” for each of the working pilots. For the 2018 shipping season, this number was derived from 2015 data provided by the American Maritime Officers Union regarding labor contracts, along with annual inflation adjustments deemed applicable by the Director. The second phase entails multiplying this compensation figure by the number of working pilots in each pilotage district and area. Step 5 – Project working capital fund. This value is obtained by adding total operating expenses (step 2) and total pilot compensation figure (step 4) and multiplying that figure by the annual rate of return from the preceding year for new issues of high-grade corporate securities. Step 6 – Project needed revenue. The Director of the Great Lakes Pilotage Program adds the total values produced for operating expenses, total pilot compensation, and the working capital fund. This number, which is calculated separately for each district and area, represents the total projected revenue needed for the upcoming season. Step 7 – Calculate initial base rates. This step consists of first calculating the 10-year vessel traffic average for each district and area. Then, the figure for needed revenue is divided by the 10-year traffic averages. Step 8 – Calculate average weighting factors by area. Since each vessel that requires a U.S. Great Lakes pilot pays a multiple of the “base rate” based on its size (ranging from 1.0 for the smallest vessels to 1.45 for the largest vessels), the Coast Guard calculates the extra revenue that has historically been produced by the weighting factor in each area. Step 9 – Calculate revised base rates. The Coast Guard modifies the base rate to account for the extra revenue generated by the weighting factors. This is done by dividing the initial base rate by the average weighting factor to produce a revised rate. Step 10 – Review and finalize rates. According to the Coast Guard, this step can be referred to informally as “director’s discretion” and is principally intended to help ensure that the rates meet the goals set forth in applicable law and regulation. The Coast Guard reported that no additional adjustments were included as part of this step for the 2018 Final Rule. After the base pilotage rates are set, the Coast Guard also considers whether surcharges are necessary, such as those used to help fund the training of new pilots. This amount is calculated as a percentage of total revenue for each district and that percentage is applied to each bill until the total amount of the surcharge is collected. Financial Oversight and Cost Accounting Shipping industry stakeholders identified a number of issues related to improving the timeliness and transparency of pilotage association financial information used in pilotage rate-setting process. Among these include (1) addressing the 3-year time lag that exists to incorporate pilotage expenses into the rate calculations; (2) presentation of financial information in a uniform format; (3) disclosure of individual pilot compensation data; and (4) clarifying the purpose and authorized uses of the working capital fund. 3-year time lag to incorporate pilotage expenses. Shipping industry stakeholders suggest that the Coast Guard make an effort to reduce the 3-year time lag to incorporate pilotage expenses into the rate-setting calculations. For example, audited financial information for the 2016 shipping season is used in the development of the 2019 rulemaking. At the most recent GLPAC meeting in September 2018, Coast Guard representatives identified several reasons for this time lag, including about 6 months required for an auditor to conduct an independent review of pilotage expenses and multiple stages of federal review that can take an additional 6 months for the Coast Guard to develop and publish the proposed rate in the Notice of Proposed Rulemaking each year. Pilot representatives and Coast Guard officials generally agree that shortening this lag would be preferable, but are unable to identify a method by which this could be achieved given the existing time frames required for the financial auditing and rulemaking processes. Uniform format for financial reporting. Shipping industry stakeholders have requested that audited financial statements for the pilot associations be presented in a uniform format. According to an industry representative, the audited financial statements (prepared individually by each pilotage association each year after the shipping season) differ primarily due to the standard accounting practices of the different organizational structures. Specifically, two pilot associations are partnerships and one is a corporation. Our review indicates that a consistent format is used by the Coast Guard and its designated independent reviewer to present summary information of applicable expenses for all three pilot associations as part of the rulemaking process. Public reporting of individual pilot compensation. Shipping industry stakeholders contend that individual pilotage compensation levels should be disclosed to help ensure revenues are being shared equally among the associations’ workforce. According to one pilot representative, individual compensation data were previously provided for District 1 as part of audited financial statements, but was eliminated because the information was being used out of context. The pilot representative noted that although all pilots in his association generally receive the same rate of pay, some may obtain higher annual compensation due to additional days worked. According to Coast Guard officials, they do not collect or retain individual compensation data on the pilots, but they do review such data during visits to the pilot association offices to help ensure fair compensation practices. Enhanced transparency of the working capital fund. Members of the shipping industry also identified an issue related to the “working capital” component of the rate-setting process. According to these stakeholders, this fund could potentially be used to augment general revenue and compensation levels and there is a lack of transparency regarding how these funds are being applied to fund capital improvements. This position was the basis of one of the claims included in the complaint filed by a coalition of industry stakeholders in November 2018. In that complaint, the plaintiffs claim that the Coast Guard’s failure to eliminate the working capital element as a basis for additional revenue requirements or to bound revenue raised as working capital to particular uses is arbitrary and capricious, among other things. That case is ongoing. According to pilots’ representatives, this fund is important to help fund capital improvements, particularly through the winter months, but they also recognize that additional clarity could be provided about its intended uses and potential limitations. In November 2018, the Coast Guard issued guidance to each of the pilotage association’s presidents regarding the reporting and uses of the working capital fund. Specifically, the Coast Guard directed the associations to segregate revenues generated by this fund and place them into a separate account at least once per quarter, and further clarified that funds from this account could be applied only toward capital projects, infrastructure improvements/maintenance, and non-recurring technology purchases necessary for providing pilotage services. In 2016, the Coast Guard initiated changes to its rate-setting methodology regarding how it estimates projected vessel traffic for each district and the corresponding hours worked for related pilotage services. Citing a recommendation issued by the Great Lakes Pilotage Advisory Committee in 2014, the Coast Guard initially proposed using a rolling average of 5 years of historical shipping data to estimate traffic volume as part of its ratemaking calculations for the 2016 shipping season. However, based on public comments received on the 2016 Notice of Proposed Rulemaking, the Coast Guard increased this number to 10 years of historical data. According to the Coast Guard, this change was implemented to further reduce rate volatility and help remedy traffic overestimates that occurred in the past, largely based on industry projections. Given the increasing volume of actual Great Lakes-Seaway vessel traffic in recent years, shipping industry stakeholders contend that the 10-year rolling average used for rate-setting calculations represents an underestimate of traffic volume. Responding to the 2018 Notice of Proposed Rulemaking, industry commenters asserted that the 10-year average included a period of substantially depressed traffic volume caused by the recession in 2008-2009, which if used to estimate future traffic volume could result in increased pilotage rates. See Table 1 for a summary of the variance between actual traffic volumes during the 2017 Great Lakes-Seaway shipping season compared with the estimates calculated using a 10-year rolling average. In the November 2018 complaint, shipping industry organizations argued that the Coast Guard’s use of 10 years of traffic data, in contrast with the shorter periods used to determine expenses and manning levels, was arbitrary and capricious, among other things, and that case is ongoing. The process and sources used by the Coast Guard to develop a target compensation benchmark for Great Lakes pilots have been subject to ongoing disagreement among stakeholders. Prior to 2016, the Coast Guard used compensation data for first mates on domestic Great Lakes vessels as the basis for comparison. This data was based on labor contracts of the American Maritime Officers Union (AMOU). However, in 2016, when the AMOU determined it would no longer provide this data to the Coast Guard, program officials revised the rate-setting methodology to begin using the average compensation of Canadian vessel pilots as the primary source, along with a 10 percent adjustment that program officials believed was appropriate to reflect the different level of benefits provided to Canadian pilots as government employees. After the court found that the 10 percent adjustment to the Canadian compensation level benchmark was not supported by reasoned decision-making and remanded the matter to the Coast Guard, for the 2018 rulemaking, the Coast Guard reverted to using the pre-2016 compensation data of domestic “laker” first mates. However, the November 2018 complaint included a claim that the Coast Guard improperly applied an adjustment of “guaranteed overtime” to the compensation benchmarks based on additional input provided by the AMOU during the notice and comment period. This case is ongoing. Regardless of the basis used, the benchmark pilot compensation levels have not varied greatly in recent years after accounting for annual inflation adjustments. That is, target compensation in 2016 was $326,114 and has increased to $359,887 in 2019, an average annual increase of approximately 3.3 percent. One related change implemented by the Coast Guard in 2016 that can also affect pilot compensation figures includes the determination to calculate pilotage rates based on the actual number of working Great Lakes pilots rather than the total number authorized. For example, in 2019 there were 54 total authorized U.S.-registered Great Lakes pilots, but only 51 were actually employed and available to provide pilotage services. According to the Coast Guard, this change serves, in part, to remove any financial incentive of pilot associations to operate with fewer pilots than allowable to increase individual compensation levels. The shipping industry has also identified issues regarding the number of working days the Coast Guard uses to calculate compensation figures and its application of a 10-day per month rest standard for pilots. For example, in 2016, the Coast Guard began using 200 working days per season as the basis for staffing calculations—down from 270—to allow for up to 10-days of rest per month. According to the Coast Guard, this change was made, in part, to address recommendations from the National Transportation Safety Board regarding reducing possible “pilot fatigue.” However, shipping industry stakeholders have suggested that if 200 days is the benchmark for working days, it should also be used to determine pilot compensation levels. Instead, the Coast Guard multiplies the weighted daily rate derived from AMOU compensation data by 270 to calculate the target annual compensation. This issue is also the subject of a claim included in the 2018 complaint, which alleges that the Coast Guard’s use of the 270-day multiplier value is arbitrary and capricious, among other things. The shipping industry stakeholders further contend that the 10-day rest standard may need to be revisited to ensure adequate pilot availability and avoid any unnecessary increases in total pilot numbers. The Coast Guard states that this 10-day rest standard is not a requirement and generally does not apply during the busiest times of the season, when pilots would remain available to work additional days to service increased vessel traffic on the Great Lakes-Seaway. Billing Concerns and Dispute Resolution There is ongoing concern among shipping industry stakeholders about certain billings from pilot associations they view as unnecessary and the Coast Guard’s dispute resolution process. The primary billing issues cited by shipping industry stakeholders since 2016 include an increase in the number of tug boats requested, as well as cases where double pilotage was employed that vessel operators did not believe were necessary. In the case of tug usage, pilot representatives generally recognize an increase in tug usage but respond that they do not have any financial incentive to call for the use of tug boats and that pilots only request them, in coordination with the shippers’ agents, when they are deemed necessary. Pilot representatives at the 2018 GLPAC meeting also stated that tug boats represent additional insurance to avoid any potential collisions in an increasingly risk-averse environment. Further, they noted that the newer pilots that have come onboard in recent years may also be a contributing factor for an increase in tug usage. According to the Coast Guard, the program routinely reviews inquiries from shippers and masters on this issue, but decisions to use tug boats remain safety decisions between the master and pilot. In contrast, authorizations for double pilotage are provided on a case-by- case basis by the Director of the Great Lakes Pilotage Program as specified in regulation. In general, the Director may authorize double pilotage when aids-to-navigation have been removed due to ice and weather conditions, dead ship tows, adverse weather and sea conditions, or any abnormal condition that will likely result in extended transits in designated waters. According to the Coast Guard, there were instances in which pilot associations charged for double pilotage without obtaining authorization from the Director of the Great Lakes Pilotage Program. In such cases, the Coast Guard has ruled in favor of vessel operators with regard to billing disputes. According to Great Lakes Pilotage Program officials, if vessel operators believe a billing error was made, they should first engage directly with the respective pilot association to review the charges and rectify any mistakes. If no agreement is reached with the pilot association, then the vessel operator can make an appeal to the Coast Guard to conduct a further review. If the Coast Guard review determines that a chargeback is justified, they can issue an advisory opinion that the pilot association refund any amount not approved by the Coast Guard or reissue the bill. At the September 2018 GLPAC meeting, Coast Guard representatives noted that some billing concerns were presented after more than 2 years and did not include sufficient details to effectively review and make an informed decision. The Coast Guard is currently working on a proposal to establish reporting timelines for presenting and making determinations on billing disputes. Another billing concern cited by industry stakeholders at the 2018 GLPAC meeting includes objections to an absence of limits to charges when pilots are onboard a vessel but it cannot get underway due to inclement weather or for other reasons. Pilot representatives point out that such delays consume pilotage resources and the charges are needed to provide an incentive for shippers and agents to remain efficient when ordering and releasing a pilot. Shipping industry stakeholders note that there are a range of factors that can cause a pilot to be detained onboard and the charges, which can exceed $20,000 per day, are unreasonable and represent a large, unforeseen cost. According to Coast Guard officials, they plan to continue engagement with GLPAC members on this issue, recognizing that pilot resources should be employed efficiently, but also that weather/ice conditions may require pilots to remain onboard a vessel for an extended period of time at significant additional cost. In addition to the contact named above, Christopher Conrad (Assistant Director), Ryan Lambert (Analyst-in-Charge), Chuck Bausell, Dominick Dale, Michele Fejfar, Eric Hauswirth, and Tracey King made key contributions to this report.", "summary": "The Great Lakes-St. Lawrence Seaway maritime transportation system is the longest inland navigation system in the world. In 2016, the Coast Guard implemented a number of changes, including amending its methodology for setting the rates charged to shippers for using U.S. marine pilotage services in these waters. GAO was asked to review the Coast Guard's management of the Great Lakes Pilotage Program. This report (1) describes how the Coast Guard obtains stakeholder input on the Great Lakes Pilotage Program, and identifies key stakeholder issues that exist; and (2) discusses alternatives to the current structure and governance of the Great Lakes pilotage system identified by stakeholders, and the reported tradeoffs they may present. GAO reviewed applicable laws, Coast Guard rulemakings from 2016-2019, Great Lakes Pilotage Advisory Committee meeting minutes for 2017 and 2018, and issues identified by stakeholders. GAO also interviewed a range of stakeholders, including shipping industry and pilot representatives, to obtain perspectives on the Coast Guard's management of the program and any alternative governance options that may exist. The Coast Guard manages the Great Lakes Pilotage Program to implement federal requirements that any oceangoing or foreign commercial vessel entering the Great Lakes-St. Lawrence Seaway use a registered marine pilot to safely navigate the vessel through the system. The Coast Guard employs several mechanisms for communicating with stakeholders and obtaining their input on the program. These include the federal rulemaking process, meetings of the Great Lakes Pilotage Advisory Committee, and ad-hoc communications with local pilotage stakeholders. Since 2016, when the Coast Guard implemented several programmatic changes, shipping industry stakeholders and pilots have identified a number of issues that they would like to have considered for the program. The issues cited by shipping industry stakeholders relate, in large part, to the financial impacts associated with the Coast Guard's methodology for calculating pilotage rates. The issues raised by Great Lakes pilots and their representatives are varied and include changes that may be needed to respond to the increasing volume and variety of vessels needing Great Lakes pilotage services, such as cruise ships. U.S. Pilot Associations in the Great Lakes-St. Lawrence Seaway Shipping industry stakeholders and others have suggested potential alternatives to the structure and governance of Great Lakes pilotage. The proposed alternatives include consolidating the three U.S. pilot associations and districts, revising the existing governance structure and entities responsible for pilotage rate-setting, and introducing some level of competition for providing pilotage services. Each of these options presents various tradeoffs. For example, it is unclear if consolidating the three associations and districts would result in cost savings because there are relatively few administrative positions that could be reduced. According to the Coast Guard and pilot representatives, the specialized training and local experience needed to become registered pilots also presents a challenge to implementing competition because there is generally a limited supply of pilots available to compete in the same geographic area. Further, many of the governance structures and procedures of the existing Great Lakes pilotage system were established by statute and revisions would require legislative changes.", "document_type": "gao"}
{"report": "The federal government faces long-standing challenges in strategically managing its workforce. As shown in table 1, in addition to strategic human capital management, skills gaps played a role in 16 of the 34 other high-risk areas on our 2019 High-Risk List, including information technology management and acquisitions, and veterans’ health care. We have also designated as priority 29 of our prior recommendations to OPM because, upon implementation, they may have an especially significant impact on OPM’s operations. Twenty-one of these priority recommendations are aimed at addressing government-wide human capital challenges, including some of the ones discussed above. OPM agreed or partially agreed with most of these recommendations. OPM has implemented 10 of these priority recommendations to date, but needs to take additional action on the other 11. For example, OPM should continue to streamline hiring authorities to strengthen the government’s ability to compete in the labor market for top talent and improve the federal hiring process. We will continue to monitor OPM’s progress in implementing our recommendations. The government’s system of current employment policies was designed generations ago for a workforce and types of work that largely no longer exist. Much has changed since the Civil Service Reform Act of 1978 and the Classification Act of 1949 laid the foundation of today’s federal personnel system. We have identified several structural challenges within the federal human capital system that impede the ability of agencies to recruit, retain, and develop employees, both today and in the future. For example: Classification system. The General Schedule classification system—which defines and organizes federal positions, primarily to assign rates of pay—has not kept pace with the government’s evolving requirements. Recruiting and hiring. Federal agencies need a hiring process that is applicant friendly and flexible, and meets policy requirements. Pay system. Employees are compensated through an outmoded system that (1) rewards length of service rather than individual performance and contributions, and (2) automatically provides across- the-board annual pay increases, even to poor performers. Performance management. Federal agencies have faced long- standing challenges developing modern, credible, and effective employee performance management systems and dealing with poor performers. Additionally, the changing nature of federal work and high percentage of employees eligible for retirement could produce gaps in leadership and institutional knowledge. It could also threaten to aggravate the problems created from existing skills gaps. For example, 31.6 percent of permanent federal employees who were on board as of September 30, 2017 will be eligible to retire in the next five years, with some agencies, such as the Department of Housing and Urban Development and the Environmental Protection Agency, having particularly high levels of employees eligible to retire. In March 2019, we identified key trends in agency operations and attitudes toward work that are affecting how federal work is done and consequently, the skills and competencies that workers will need to accomplish agency missions (see fig. 1). Agencies will need to apply talent management strategies that are adapted to these trends to recruit, develop, and retain a high-performing workforce and better meet their missions. In light of trends and other challenges facing the government’s human capital management efforts, our prior work has identified actionable strategies that agencies may be able to use to effectively manage the future federal workforce in key talent management areas (see table 2). We noted that while these strategies are not an exhaustive list, collectively they suggest basic steps that agencies can take within existing authorities to position themselves to meet their talent needs. These practices are based on our review of related reports, group interviews with federal Chief Human Capital Officers (CHCO), and interviews with selected private organizations and foreign governments. For each strategy, we highlight examples of the challenges agencies face, actions OPM can take to implement related recommendations from our prior work, and practices that may help agencies implement the strategy. Align human capital strategy with current and future mission requirements. With shifting attitudes toward work, technological advances, and increased reliance on nonfederal partners, agencies need to identify the knowledge and skills necessary to respond to current and future demands. Key practices include identifying and assessing existing skills, competencies, and skills gaps. In May 2014, we reported that agencies should be aware of existing skills and competencies in their workforce to help inform workforce planning. As one example, the Department of the Treasury CHCO told us that, following the Puerto Rico debt crisis—where it needed to be able to identify the necessary skills to manage the crisis—the agency decided to implement an Integrated Talent Management System to facilitate workforce and succession planning as well as learning and performance management. Acquire and assign talent. To ensure agencies have the talent capacity to address evolving mission requirements and negative perceptions by some of federal work (e.g., that it is too bureaucratic), agencies can cultivate a diverse talent pipeline through strategic partnerships with academic and other institutions, highlight their respective missions, recruit early in the school year, support rotations, and assign talent where needed. As one example, consulting firm representatives that we interviewed for our prior work stated that their internship programs are among their most successful practices for cultivating a talent pipeline because the firms can offer full-time positions to rising seniors during the internship. A representative from one consulting firm said that, after experiencing challenges in recruiting on college campuses, the firm built a competitive internship program to promote the firm’s brand and reputation. Participants in the firm’s 10-week program are paid and assigned challenging projects, and successful participants are given job offers upon completion. According to the representative, approximately a quarter of the firm’s workforce is former interns. Similarly, CHCOs and federal employee and management group representatives we interviewed noted that internships are important for establishing a pipeline for recruitment. The federal government’s Pathways Programs, which consist of the Internship Program, the Recent Graduates Program, and the Presidential Management Fellows Program, were designed to promote employment opportunities for students and recent graduates by providing distinct paths to federal internships and potential careers in government. The Internship Program provides paid opportunities for students (high school, vocational, technical, undergraduate, and graduate) to work in agencies and explore federal careers while still in school. Students who successfully complete academic and program requirements may be eligible for non-competitive conversion to a term or permanent position in the civil service. In our prior work, we have also reported on the importance of cultivating a diverse talent pipeline through active campus recruiting which includes developing long-term institutional relationships with faculty, administrators and students, and by building a “brand” on campus. Other strategies to expand a talent pool include developing strategic partnerships with such entities as trade schools, apprentice programs, and affinity organizations from across the country. Another strategy for attracting strong candidates is for agencies to highlight their missions and innovative work, which, according to our expert and CHCO interviews, can help counter negative perceptions of federal employment. For example, the Department of Homeland Security (DHS) provides “Day in the Life” information on its work to promote public awareness of how its everyday tasks tie in with its mission of protecting the United States, according to the DHS CHCO. The DHS CHCO stated that promoting agency mission can be done while cultivating a talent pipeline and assessing applicants’ abilities. The department holds recruitment events where potential candidates can participate in law enforcement-related activities such as fitness testing. The CHCO noted that these events both promote homeland security careers and help prospective candidates determine if a position is a good fit for them. Incentivize and compensate employees. While federal agencies may struggle to offer competitive pay in certain labor markets, they can leverage existing incentives that appeal to workers’ desire to set a schedule and to work in locations that provide work-life balance. However, agencies do not always promote these benefits and incentives as part of a total compensation package, in part because managers are not always aware of the importance of doing so. Some agencies are addressing this issue by advertising and helping employees use available benefits, work-life balance programs, and other resources. For example, the National Science Foundation offers employees many opportunities to learn about existing benefits, according to the foundation’s CHCO. These opportunities include triannual retirement seminars where employees receive personalized retirement estimates, quarterly financial planning seminars where employees receive a free 1-hour consultation, and annual benefit fairs where employees can learn about various health care providers, the work-life programs, and the employee assistance program. Our prior analysis of CHCO and expert interviews also found that employees may value different benefits and incentives depending on their stage in life. By better understanding the desires of the workforce at various life stages, agencies can better tailor benefits packages and incentives to their employees. For example, the Social Security Administration’s CHCO said that the agency’s younger workers value work-life and wellness programs, so the agency implemented a health- tracking program and a fitness discount program for all employees. CHCOs also suggested identifying and incorporating the benefits that would be most useful to various groups of employees, such as sabbaticals for midlevel employees or paid parental leave for employees starting families. One CHCO found that her cybersecurity workforce values subsidies for training and additional certifications more than bonus pay. Further, OPM’s 2018 Federal Work-Life Survey Governmentwide Report found that the number of respondents who anticipate adult dependent care responsibilities in the next 5 years (31 percent) is double the number of respondents with current adult dependent care needs (15 percent). OPM officials stated in light of this change, agencies may need to provide greater workplace flexibilities and other support services to retain talent. Some CHCOs we interviewed for prior work said that they believe that paid parental leave could be a powerful retention tool for federal workers. Representatives from consulting firms that we interviewed said that they have observed positive impacts from these types of benefit programs. For example, representatives from one firm said that providing employees with peace of mind when managing life events helps them feel more committed to the organization. Engage employees. Engaged employees are more productive and less likely to leave, according to OPM. Agencies can better ensure their workforces are engaged by managing employee performance, involving employees in decisions, and developing employees. Experts we interviewed for prior work said that employees desire an environment where they can collaborate with their peers and feel a sense of comradery. In contrast, even a small number of poor performers can negatively affect employee morale and agencies’ capacity to meet their mission, according to CHCOs and our previous work. In the 2017 Federal Employee Viewpoint Survey (FEVS), 64 percent of federal employee respondents agreed that their supervisor provides them with constructive suggestions to improve job performance and 31 percent agreed that steps are taken to deal with poor performers. Without effective performance management, agencies risk not only losing the skills of top talent, they also risk missing the opportunity to effectively address increasingly complex and evolving mission challenges. Agencies can make performance management more effective by improving the selection and training of supervisors and managers, creating a “line of sight” between individual performance and organizational results, and implementing meaningful reward programs. Our prior analysis found that employees seek autonomy in the workplace, meaningful work, and opportunities to achieve results by developing creative and innovative solutions. Also, experts noted that in some cases, connecting federal employees to a sense of inclusion and meaning can compensate for the opportunity to make higher salaries in other sectors. Creating an inclusive work environment is one practice that can help increase employee involvement in decisions. CHCOs and federal employee and management group representatives said that more can be done to prioritize training, even in an era of resource constraints. In 2017, only 55 percent of FEVS respondents were satisfied with training. As an example of an agency prioritizing training efforts, the Social Security Administration has national and regional development programs that offer 12 to 18 months of training and rotations for entry-, mid-, and senior-level employees to strengthen foundational, technical, and leadership knowledge and skills, according to the agency’s CHCO. For example, its Leadership Development Program assigns selected GS-9 through GS-12 employees to developmental assignments in new areas of work, and provides leadership training that broadens their perspective of the agency’s mission. Chairman Connolly, Ranking Member Meadows, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff have any questions about this testimony, please contact Robert Goldenkoff, Director, Strategic Issues, at (202) 512-2757 or GoldenkoffR@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Shirley Hwang (Assistant Director), Shelby Kain (Analyst-In-Charge), Sarah Green, Allison Gunn, and Alexander Ray. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The federal workforce is critical to federal agencies' ability to address the complex social, economic, and security challenges facing the country. However, the federal government faces long-standing challenges in strategically managing its workforce. We first added federal strategic human capital management to our list of high-risk government programs and operations in 2001. Although Congress, OPM, and individual agencies have made improvements since then, federal human capital management remains a high-risk area because mission-critical skills gaps within the federal workforce pose a high risk to the nation. This testimony focuses on (1) key hiring and other human capital management challenges facing federal agencies, and (2) talent management strategies identified from GAO's prior work that agencies can use to be more attractive employers in a tight labor market. This testimony is based on GAO's large body of work on federal human capital management issued primarily between July 2014 and July 2019. To conduct these studies, GAO reviewed government-wide employment data and interviewed officials from OPM and subject matter specialists from think tanks, academia, government employee unions, and other areas. Outmoded approaches to personnel functions such as job classification, pay, and performance management are hampering the ability of agencies to recruit, retain, and develop employees. At the same time, agency operations are being deeply affected by a set of evolving trends in federal work, including how work is done and the skills that employees need to accomplish agency missions. Given these challenges and trends, federal agencies will need to apply talent management strategies such as the following: Align human capital strategy with current and future mission requirements. Agencies need to identify the knowledge and skills necessary to respond to current and future demands. Key practices include identifying and assessing existing skills, competencies, and skills gaps. Acquire and assign talent. To ensure the appropriate capacity exists to address evolving mission requirements, agencies can use internships, cultivate a diverse talent pipeline, highlight their respective missions, and recruit early in the school year. Incentivize and compensate employees. While agencies may struggle to offer competitive pay in certain labor markets, they can leverage existing incentives that appeal to workers' desire to set a schedule and to work in locations that provide work-life balance. Engage employees. Engaged employees are more productive and less likely to leave, according to the Office of Personnel Management (OPM). Agencies can better ensure their employees are engaged by managing their performance, involving them in decisions, and providing staff development. Of the 29 recommendations to OPM that GAO has designated as priorities for implementation, 21 are aimed at improving strategic human capital management efforts government-wide. OPM agreed or partially agreed with most of these recommendations, of which 11 are still open. GAO will continue to monitor OPM's progress in addressing them.", "document_type": "gao"}
{"report": "The boards that oversee the Social Security and Medicare trust funds are technically separate entities under the Social Security Act, but the same set of trustees have served as board members for each trust fund and the boards meet concurrently. For each board, four of the trustees are ex officio, i.e. members by virtue of their office and position: the Secretary of the Treasury, Secretary of Labor, Secretary of Health and Human Services, and the Commissioner of Social Security. The remaining two trustees are members of the public, nominated by the President, confirmed by the Senate, and the public trustees must not be from the same political party as one another. The public trustee positions were established in 1983; they have been vacant since 2015. The boards issue two separate Trustees reports each year, one on the Social Security trust funds and one on the Medicare trust funds. Under the Social Security Act, these reports are due by April 1 of each year. The Trustees reports provide information on the present and projected statuses of the trust funds, including their projected balances over the next 10 years (short-term), the next 75 years (long-term), and the assumptions and methods used to make these projections. The reports provide estimates of the projected costs and incomes of the trust funds, and any dates that the boards project the trust funds’ reserves to become depleted, among other information about the programs. Because projections are inherently uncertain, the reports include three projection scenarios: intermediate, low-cost, and high-cost alternatives, along with other information about uncertainty. The intermediate scenario is based on assumptions that reflect the boards’ best estimate of future experience. The low-cost scenario makes assumptions that are relatively more favorable with respect to the projected statuses of the trust funds, while the high-cost scenario does the opposite. For example, the low-cost scenario assumes more workers will pay into the trust funds and fewer beneficiaries will receive benefits, while the high-cost scenario assumes fewer workers and more beneficiaries. Officials in the Social Security Administration Office of the Chief Actuary (SSA OCACT) and the Centers for Medicare & Medicaid Services Office of the Actuary (CMS OACT) work with other agency officials and trustees to develop assumptions and draft and revise the Trustees reports in an annual cycle, according to agency officials and the board meeting minutes and report development schedules we reviewed (see fig. 1). At the end of each cycle, the boards have established a working group that is largely responsible for overseeing the day-to-day development of the next year’s report. This working group consists of officials in the four agencies that are led by the ex officio trustees (Treasury, DOL, HHS, and SSA), officials from SSA OCACT and CMS OACT, and the public trustees, when confirmed. All of the working group’s discussions and agreements are subject to the approval of the boards. The Secretary of the Treasury serves as the Managing Trustee and Chairperson of the boards. Treasury staff has historically coordinated the report development process, including organizing the development schedule and hosting the working group and boards’ meetings. The reports are drafted by SSA and CMS. SSA OCACT and CMS OACT officials we interviewed said they work to update the assumptions—both long-term and short-term—they propose as the basis for the trust fund projections in the reports. Assumptions are the demographic, economic, and program-specific factors that the actuaries use to model the future financial status of the trust funds (see appendix I). For each assumption, SSA OCACT and CMS OACT go through a process of updating the values for the 75-year projection period as needed and use those updated values as inputs in their models to project future costs and income for the trust funds. As a part of this process, the working group discusses issues that inform the assumptions proposed by SSA OCACT and CMS OACT. The work on assumptions is divided according to the specializations and expertise of the two actuarial offices, and is developed by staff from a range of disciplines, including actuaries, demographers, and economists. SSA OCACT develops the demographic and economic assumptions that are common to both reports, including rates for fertility, mortality, and growth in gross domestic product. SSA OCACT also prepares the programmatic assumptions for Social Security, such as the numbers of retirement and disability beneficiaries and the anticipated income into the trust funds from payroll taxes. CMS OACT prepares the programmatic assumptions that are specific to Medicare, such as the number of Medicare beneficiaries and expected growth in health care costs. SSA OCACT and CMS OACT officials update assumptions and revise their methodologies based on recent data, if a change is warranted. For example, the 2017 Social Security Trustees report projected an increase in the total fertility rate. Information collected in the subsequent year showed that fertility rates had not risen as expected, so officials reduced the fertility rate assumptions for the 2018 report. SSA OCACT officials told us that they look at both the reasonableness of the assumptions individually and in the aggregate, as some assumptions interrelate. According to agency officials we interviewed, the assumptions generally undergo gradual or no changes from year to year, unless there are significant policy changes. SSA OCACT and CMS OACT also update their models by incorporating more recent data into them. For example, in the 2017 Social Security Trustees report, the model for projecting average age benefit levels of retired worker and disabled worker beneficiaries who are newly entitled to benefits used a sample of these beneficiaries from 2013. In the 2018 report, this model was updated to use a sample from 2015. The working group considers and works towards consensus on the assumptions proposed by SSA OCACT and CMS OACT. Members of the working group meet periodically to discuss the assumptions and come to an agreement on the values for them. In these meetings, the working group often hears presentations from internal or external experts on specific topics. For example, in one meeting, SSA staff led a presentation and discussion on Disability Insurance, and DOL staff led a presentation and discussion on how globalization might affect long-term economic trends. To inform their discussions, the working group may also review reports from technical panels or invite panel members to discuss their findings and recommendations at a working group meeting. For example, in September 2012, the working group discussed a Medicare technical panel recommendation that the board continue to present alternative projections in which average Medicare spending per beneficiary rises faster than the current law baseline; the working group and board agreed to implement this recommendation. Throughout the working group’s activities, the members representing the ex officio trustees generally serve as a liaison between the trustee for their agency and the working group. When confirmed, public trustees participate directly on the working group. After consideration, the working group finalizes long-term assumptions at a fall board meeting. The long-term assumptions serve as the basis for the short-term assumptions and the 75-year trust fund projections. For each long-term assumption, the boards set the “ultimate value”, i.e. the constant rate or number that is projected to be met in a particular year (within 10 years in most cases) and then continued through the remainder of the 75-year projection period. For example, for the 2019 Trustees reports, the boards set the ultimate value for the annual change in covered earnings as a percent of total labor compensation for each year beginning in 2028 and continuing through 2093. In most cases, according to agency officials we interviewed, the working group achieves consensus on the assumptions before the fall board meeting. However, when the working group is unable to reach consensus, the boards settle any outstanding issues and tend to either make no changes or incremental changes over time to avoid major swings in year-to-year projections, according to some agency officials we interviewed. Once the long-term ultimate values are set by the boards, the working group then discusses the short-term assumptions that bridge the gap between current data and the ultimate values. The working group first considers and works toward agreement on the short-term economic assumptions and then the health assumptions. Short-term economic assumptions can vary during the early years of the projection period. The projection of Medicare’s HI Trust Fund depletion date is based on detailed short-term growth rate assumptions for individual types of Medicare services, such as inpatient hospital care. Once the assumptions are set, officials at SSA OCACT develop the projections that determine the actuarial status and then draft the Social Security Trustees report, and officials at CMS OACT do the same for the Medicare Trustees report. The reports include information on and values of the assumptions, projected financial statuses of the trust funds and programs, actuarial analyses and estimates, and technical information on the methodologies and projections. In addition, the reports note changes to the assumptions, methodology, and projections from prior reports, and explain the implications for the trust funds. The reports also include statements of opinion by the relevant agency’s Chief Actuary regarding whether the techniques and methodologies used are generally accepted within the actuarial profession and whether the assumptions used and the resulting actuarial estimates are reasonable. When the drafts are completed, SSA OCACT and CMS OACT circulate them to the working group for comments and agreement. According to one former public trustee, these comments are mostly related to the presentation of the information, such as word choices, as members have previously agreed to the assumptions. SSA OCACT or CMS OACT officials respond to these comments, and make revisions to the reports in several rounds, engaging with the working group for comment on each new version of the reports. As with the earlier round when the working group worked toward consensus on the assumptions, the working group members that represent the ex officio trustees can brief the trustee from their agency and bring any input back to the working group to help ensure that the trustees agree with the reports. The final drafts of the Trustees reports are presented and approved at the annual spring meeting of the boards. Under the boards’ bylaws, members of the boards must be present at these meetings to approve the reports. During the meeting, agency officials provide an overview of the reports to the trustees and other attendees, and explain changes in the overall projections from the previous year’s reports. For those trust funds with an estimated depletion date, agency officials explain the estimated dates of depletion and the potential implications for beneficiaries. After any discussion, the trustees sign the reports and the boards formally issue them to Congress. Public trustees, when confirmed, play unique roles as members of the boards and also the working group that develops the Trustees reports. Former public trustees we interviewed said their role was to represent the public in the report development process, independent of the ex officio trustees and other agency officials in the administration. To become members of the board, public trustees must be nominated by the President and confirmed by the Senate, and the public trustee cannot both be from the same political party. Those we interviewed stressed the importance of not allowing personal and political opinions to influence their work on the Trustees reports. As a result, according to both agency officials and former public trustees, having public trustees in place lends credibility to the reports. Former public trustees stated that they worked closely with their counterpart public trustee to coordinate their comments and input to the working group. Historically, public trustees sometimes questioned or encouraged changes to some assumptions used in the reports, according to former public trustees and some agency officials. When in place, public trustees regularly attend working group meetings, whereas ex officio trustees do not. According to the former public trustees we interviewed, they saw part of their role as facilitating conversations as leaders and moving the group towards consensus on assumptions. Additionally, former public trustees and some agency officials said trustees are more hesitant to change the assumptions in the reports when there are no public trustees in place, out of concern that any change could be viewed as politically motivated. For example, in 2017 the boards discussed whether or not to change the long-range real interest rate assumption from the rate used in the previous year’s Trustees reports. The boards decided to keep the assumptions unchanged, in part because there were no public trustees in place. When they are in place, public trustees can also help communicate the message of the Trustees reports to policy makers and the public. As an example, the Trustees reports can be technical and difficult to understand; to address this, the public trustees introduced a summary of the reports in 1991, which presented the reports’ findings in a way that is more accessible to the general public. Former public trustees said they were able to inform policy makers on the contents of the reports through congressional testimony and direct conversations with congressional staff. One former public trustee reported that he was a resource for the media, spending hours on the phone providing his perspective and explaining the reports’ implications for policy decisions. In addition, public trustees published a separate message that allowed them to present what they believe to be the main idea of the reports. The boards issued the Trustees reports to Congress after the April 1 statutory deadline in 17 of the 25 years from 1995 to 2019, including every year from 2009 through 2019 (see fig. 2). Since 2009, the boards have issued the reports at least 2 months late six times; they only issued the reports this late one time in the 14 years from 1995 to 2008. Agency officials and former public trustees provided a number of reasons why the Trustees reports have been late in recent years. Agency officials and former public trustees said they may delay reports in order to include the impact of late-breaking legislation or policy changes on the assumptions or data. SSA OCACT told us that this decision is based on (1) if the policy change results in substantial changes to assumptions and (2) if the policy change affects a policy that is directly governing a trust fund. For example, agency officials and former public trustees stated that the Patient Protection and Affordable Care Act (PPACA), enacted on March 23, 2010, significantly contributed to the 2010 reports being issued August 5, 2010, over 4 months past the deadline. PPACA significantly affected many of the factors that were the basis for the Medicare Trustees report projections, such as reducing projected Medicare expenditures through various policy changes, including a change to the payment formula for the Medicare Advantage program—the private health plan alternative to traditional Medicare. According to one former public trustee, if the boards had issued a report that did not reflect the changes made by PPACA, it would not have been applicable to the current outlook of the Medicare trust funds and therefore not as useful to Congress and the public. Agency officials have also reported that there have been instances of waiting for more complete or recent data sets to become available before calculating the actuaries’ projections. According to CMS officials, a tradeoff exists between updating data and meeting the deadline. For example, Treasury officials told us that because the working group decided that CMS OACT should not wait for January 2019 Medicare Advantage enrollment data, the 2019 Trustees reports were issued earlier (April 22) than they would have been if they had waited for the complete end of year data, as they had in previous years. Agency officials and public trustees also cited difficulties in scheduling the spring board meetings as a factor that contributed to delays in issuing the Trustees reports. The boards’ bylaws require the annual reports to be adopted by a majority of the trustees who are present and voting. However, sometimes Treasury staff experienced difficulty scheduling the meeting. According to Treasury officials responsible for scheduling the meeting, they generally wait until the first drafts of the Trustees reports are completed before they schedule the spring board meeting to avoid having to reschedule the meeting if the draft reports are provided after the working group’s internal deadline. For the last 15 years (2005-2019), report development schedules from SSA OCACT indicated that the draft reports were provided to the working group after the internal deadline 12 times. In the other 3 years, the report development schedules did not show the actual date that the draft reports were provided. As a result of scheduling the meeting later in the process, Treasury staff has sometimes not been able schedule a meeting that all of the Trustees can attend prior to the statutory deadline of April 1. Other challenges that contribute to delays include government shutdowns and staff having conflicting concurrent responsibilities, according to some agency officials or former public trustees. When the government shut down for 11 business days in October 2013, the board meeting minutes show that it affected the timelines for the 2014 Trustees reports. However, according to HHS, government shutdowns have never materially delayed the release dates for the Trustees reports. Some former public trustees and one agency official we spoke to stated that agency officials involved in the report process sometimes had other duties competing for their time, which could result in delaying their work on the Trustees reports, while other agency officials stated this was not a factor. Agency officials’ scheduling process is inconsistent with GAO’s guide on best practices for schedule management. Agency officials and former public trustees said they attempted to meet the statutory deadline each year, but did not believe issuing the report after the deadline created serious negative consequences. As a result, agency officials and former public trustees involved with developing the reports in recent years said they developed a schedule designed to meet the deadline knowing it would most likely not be met. Several agency officials and former public trustees described the schedule as “ambitious” and difficult to achieve. If the schedule is unrealistic from the start of the process, and if involved parties view it as an unlikely goal, rather than the expected outcome, then the schedule does not serve as a useful tool for managing the timely development of the Trustees reports. In addition to designing an unrealistic Trustees reports schedule, agency officials did not always document actual progress in meeting scheduled dates or modify the schedule in a way that would allow them to overcome early setbacks. Treasury officials, who organize the schedule for developing the Trustees reports, stated that the initial proposed schedule is updated only once during the report process, after the first drafts of the reports are completed. According to best practices, the schedule should be updated regularly with actual progress and remaining work. Without doing so, it could be difficult to respond to actual events while still meeting set deadlines. Treasury has not regularly archived the final version of the schedule with the dates that milestones were actually met. According to best practices, the final iteration of the schedule that was actually followed should be archived and used to inform and improve future schedules. Treasury officials were able to provide us with the archived, updated schedules for only 6 of the last 25 years, and these schedules were incomplete. While these updated schedules showed the actual dates that the draft reports were provided and the planned dates for later milestones, they did not include the dates for milestones before the draft reports were provided or the actual dates for the later milestones, including the reports’ issuance dates. Further, although the boards have regularly missed the statutory deadline, the initial report-development schedules have not significantly changed in recent years. Based on the proposed schedules for the Trustees reports that are presented in the spring board meeting minutes we reviewed, the initial schedules for each milestone in the report development process, such as obtaining agreement on assumptions or circulating drafts, has not significantly changed in the last 6 years, although the schedules have consistently proven difficult to meet (see table 1). Without recording the actual report production schedule that was followed, participating officials do not have the historical data that would assist them in making meaningful and effective changes to future schedules. Finally, according to best practices, it is important that stakeholders, including decision makers, have access to information on the progress of the project. Agency officials and former public trustees stated that they do not have a policy or practice of informing Congress of delays or changes to the schedules for the Trustees reports, even in years when the board issues the reports months after the deadline. Given this, Congress, the recipient of the Trustees reports, remains uninformed of the reports’ release date or the factors contributing to a delay in any given year. This uncertainty may hinder Congress from planning legislative sessions in advance that would use the findings of the Trustees reports. For example, congressional committees of jurisdiction may be hindered in scheduling hearings on or around the time of the reports’ release date and having access to the latest data from the reports to inform their oversight. Trustees and agency officials set out at the start of each report cycle with a schedule to meet the statutory deadline to issue the Social Security and Medicare reports each year by April 1. However, over the past 25 years, they have mostly issued the reports after the deadline. Some of the factors contributing to the boards delivering the reports late may seem reasonable to agency officials and trustees. For example, investing time to make the report consistent with new legislation impacting Social Security or Medicare programs, or waiting for end of year data to be available, may make the report more useful than if it contained older information. However, other factors related to the management of the schedule for developing the reports, such as not formally tracking the reports’ progress or adjusting the schedule based on lessons learned in prior years, may have contributed to delays. Taking steps to improve the management of the report-development schedule would better position the trustees and agency officials to anticipate and plan for scheduling the spring boards meeting and to meet the statutory deadline in future years. Additionally, recognizing that there may continue to be instances in which the issuance of the reports will be delayed, establishing a policy to inform Congress of potential delays and factors contributing to those delays would enhance Congress’s ability to conduct oversight and make decisions about these important programs. We are making the following two recommendations to the Secretary of the Treasury: The Secretary of the Treasury, as Chairperson of the Boards of Trustees, should work with the other trustees to take steps—in consultation with the chief actuaries of SSA and CMS—to improve the management of the report development schedule in order to provide the Trustees reports to Congress by the statutory deadline. These steps could include regularly updating the schedule using actual progress and archiving the final iteration of the schedules. (Recommendation 1) The Secretary of the Treasury, as Chairperson of the Boards of Trustees, should work with the other trustees to establish a policy to inform Congressional committees of jurisdiction when the trustees determine that the reports are expected to miss the issuance deadline. This outreach should include 1) the factors that are contributing to delays, and 2) the reports’ expected issuance dates. (Recommendation 2) We provided a draft of this report to the Secretary of the Treasury, the Secretary of Health and Human Services, the Secretary of Labor, and the Commissioner of Social Security for review and comment. Treasury and SSA provided formal written comments, and both agencies agreed with our recommendations. (See appendixes II and III.) Treasury, SSA, and HHS provided technical comments, which we incorporated as appropriate. DOL had no comments. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of the Treasury, the Secretary of Health and Human Services, the Secretary of Labor, the Commissioner of Social Security, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Elizabeth Curda at (202) 512-7215 or curdae@gao.gov or James Cosgrove at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. In addition to the contact named above, Mark Glickman (Assistant Director), Gregory Giusto (Assistant Director), Paul Schearf (Analyst-in- Charge), Christie Enders, and Samuel Gaffigan made key contributions to this report. Additional assistance was provided by Bill Boutboul, Juana Collymore, Robert Dacey, Alex Galuten, Yvette Gutierrez, Janice Latimer, Emei Li, Sheila R. McCoy, Art Merriam, Mimi Nguyen, Stacy Ouellette, Oliver Richard, Joseph Silvestri, Dawn Simpson, Ardith Spence, Almeta Spencer, Frank Todisco, and Walter Vance.", "summary": "The Social Security Act requires boards of trustees to issue reports to Congress by April 1 each year on the financial status of the Social Security and Medicare trust funds. Policymakers and others can use these reports to understand the programs' finances, conduct oversight, and consider legislative proposals for the programs. GAO was asked to review the timeliness of these reports. This report (1) describes how the boards of trustees develop the annual Trustees reports, and (2) examines the extent to which the boards of trustees have provided the reports to Congress by the April 1 deadline since 1995, and what factors account for any delays. GAO reviewed boards of trustees meeting minutes from 1995-2018, working group agendas from 2011-2018, and report development schedules and the annual Trustees reports from 1995-2019; as well as relevant federal law. GAO also interviewed agency working group officials from SSA and CMS; the Departments of Health and Human Services, Labor, and the Treasury; and eight former public trustees who served since 1995. Annual reports on the status of Social Security and Medicare trust funds are developed through a collaboration between agency officials and trustees, which include relevant Cabinet members and public members nominated by the President (if confirmed). Offices of the Chief Actuaries from the Social Security Administration (SSA) and the Centers for Medicare & Medicaid Services (CMS) submit data and draft reports to a working group of agency officials representing trustees and any public trustees. The working group reviews the information and, after gaining consensus, submits it to the boards of trustees for final approval. The boards of trustees send the final reports to Congress. The trustees missed the April 1 statutory deadline for submitting the reports to Congress in 17 of the 25 years from 1995 to 2019, and have issued them more than 2 months late in 6 of the last 10 years (see figure). According to agency officials and former public trustees GAO interviewed, factors that may account for delays include late-breaking changes to assumptions or data, and difficulty scheduling the boards' meetings. Additionally, contrary to GAO's guide on best practices for project schedules, officials have not taken steps to update the report-development schedules to reflect actual progress, maintained a formally documented baseline schedule to incorporate lessons learned from prior years, or notified Congress of their progress. Without taking steps to improve report-development schedule management, these trust fund reports will likely continue to be untimely, missing the April 1 statutory deadline. Also, without improved efforts to keep congressional committees informed, Congress will be unaware of when the reports will be issued, potentially hindering oversight of the trust funds. GAO recommends that Treasury take steps to work with the other trustees to improve schedule management for developing the annual Trustees reports, and to establish a policy to inform congressional committees of jurisdiction about expected delays in issuing the reports. Treasury agreed with the recommendations.", "document_type": "gao"}
{"report": "SAMHSA and other organizations recognize recovery homes—peer-run and peer-managed supportive homes—as an important step in SUD treatment and recovery. Definitions of and terms for recovery homes can vary, and recovery homes may differ in the types of services offered and resident requirements. Alcohol- and drug-free homes for individuals recovering from SUD may be referred to as “recovery residences,” “sober homes,” or other terms. For the purposes of our March 2018 report, we used the term “recovery homes” to refer to peer-run, nonclinical living environments for individuals recovering from SUD in general. Recovery homes generally are not considered to be residential treatment centers, are not eligible to be licensed providers for the purposes of billing private insurance or public programs—such as Medicaid—and residents typically have to pay rent and other home expenses themselves. Recovery home residents may separately undergo outpatient clinical SUD treatment, which is typically covered by health insurance. In addition, recovery homes may encourage residents to participate in mutual aid or self-help groups (e.g., 12-step programs such as Alcoholics Anonymous) and may require residents to submit to drug screening to verify their sobriety. Residents may be referred to recovery homes by treatment providers, the criminal justice system, or may voluntarily seek out such living environments. In our March 2018 report, we found that the prevalence of recovery homes nationwide was unknown, because complete data were not available. We found these data are not collected at the federal level to provide a nationwide picture, in part, because there was no federal agency responsible for overseeing them. However, as we reported in March 2018, two national organizations with missions dedicated to recovery homes collect data on the prevalence and characteristics for a sub-set of recovery homes and the number of homes that were not affiliated with these organizations was unknown. NARR collected data on recovery homes that sought certification by one of its 15 state affiliates that actively certify homes. As we previously reported, as of January 2018, NARR told us that its affiliates had certified almost 2,000 recovery homes, which had the capacity to provide housing to over 25,000 individuals. Oxford House, Inc. collected data on the prevalence and characteristics of its individual recovery homes (known as Oxford Houses). In its 2018 annual report, Oxford House, Inc. reported that there were 2,542 Oxford Houses in 45 states. Officials from four of the five selected states we reviewed for our March 2018 report (Florida, Massachusetts, Ohio, and Utah) told us that since 2007, state agencies had conducted, or were in the process of conducting, law enforcement investigations of unscrupulous behavior and potential insurance fraud related to recovery homes. According to the state officials, the outcomes of some of these investigations included criminal charges and changes to health insurance policies. Across the four states, officials told us that the potential insurance fraud may have relied on unscrupulous relationships between SUD treatment providers (including laboratories that perform tests to check for substance use) and recovery home operators. Officials explained that recovery home operators establish these relationships, because they cannot directly bill health insurance themselves due to the fact that recovery homes are not considered eligible providers for the purposes of billing health insurance. For example, treatment providers may form relationships with recovery home operators who then recruit individuals with SUD in order to refer or require residents to see the specific SUD treatment providers. This practice is known as patient brokering, for which recovery home operators receive kickbacks, such as cash or other remuneration from the treatment provider, in exchange for patient referrals. The extent of potential fraud differed across the four states, as discussed below. Officials from several state agencies and related entities described investigations into fraud related to recovery homes in southeastern Florida as extensive, although the scope of the fraud within the industry is unknown. In 2016, the state attorney for the 15th judicial circuit (Palm Beach County) convened a task force composed of law enforcement officials tasked with investigating and prosecuting individuals engaged in fraud and abuse in the SUD treatment and recovery home industries. The task force found that unscrupulous recovery home operators or associated SUD treatment providers were luring individuals into recovery homes using deceptive marketing practices. These practices included online or other materials that willfully misdirected individuals or their family members to recruiters with the goal of sending these individuals to specific treatment providers so that the recruiters could receive payments from those treatment providers for each referral. According to officials from the Florida state attorney’s office, these individuals—often from out of state—were lured with promises of free airfare, rent, and other amenities to recover in southern Florida’s beach climate. Recruiters brokered these individuals to SUD treatment providers, who then billed their private insurance plans for extensive and medically unnecessary urine drug testing and other services. Officials from the Florida state attorney’s office told us that SUD treatment providers were paying $300 to $500 or more per week to recovery home operators or their staff members for every individual they referred for treatment. In addition, these officials cited one case in which a SUD treatment provider billed an individual’s insurance for close to $700,000 for urine drug testing over a 7-month period. Officials from the state attorney’s office noted that the recovery homes that the task force investigated were not shared homes in the traditional, supportive sense, but rather existed as “warehouses” intended to exploit vulnerable individuals. As a result of these investigations, as of December 2017, law enforcement agencies had charged more than 40 individuals primarily with patient brokering, with at least 13 of those charged being convicted and fined or sentenced to jail time, according to the state attorney’s office. In addition, the state enacted a law that strengthened penalties under Florida’s patient brokering statute and gave the Florida Office of Statewide Prosecution, within the Florida Attorney General’s Office, authority to investigate and prosecute patient brokering. An official from the Massachusetts Medicaid Fraud Control Unit told us that the unit began investigating cases of Medicaid fraud in the state on the part of independent clinical laboratories associated with recovery homes in 2007. The unit found that, in some cases, the laboratories owned recovery homes and were self-referring residents for urine drug testing. In other cases, the laboratories were paying kickbacks to recovery homes for referrals for urine drug testing that was not medically necessary. According to the Medicaid Fraud Control Unit official, as a result of these investigations, the state settled with nine laboratories between 2007 and 2015 for more than $40 million in restitution. In addition, the state enacted a law in 2014 prohibiting clinical laboratory self-referrals and revised its Medicaid regulations in 2013 to prohibit coverage of urine drug testing for the purposes of residential monitoring. At the time of our March 2018 report, Ohio had begun to investigate an instance of potential insurance fraud related to recovery homes, including patient brokering and excessive billing for urine drug testing. Officials from the Ohio Medicaid Fraud Control Unit told us that the unit began investigating a Medicaid SUD treatment provider for paying kickbacks to recovery homes in exchange for patient referrals, excessive billing for urine drug testing, and billing for services not rendered, based on an allegation the unit received in September 2016. Officials from other state agencies and related state entities, such as the state’s substance abuse agency and NARR affiliate, were not aware of any investigations of potential fraud on the part of recovery home operators or associated treatment providers when we interviewed with them. According to these state officials, this type of fraud was not widespread across the state. In our March 2018 report, we reported that officials from the Utah Insurance Department told us that the department was conducting ongoing investigations of private insurance fraud similar to the activities occurring in Florida, as a result of a large influx of complaints and referrals the department had received in 2015. These officials told us that the department had received complaints and allegations that SUD treatment providers were paying recruiters to bring individuals with SUD who were being released from jail to treatment facilities or recovery homes; billing private insurance for therapeutic services, such as group or equine therapy, that were not being provided, in addition to billing frequently for urine drug testing; and encouraging individuals to use drugs prior to admission to qualify them and bill their insurance for more intensive treatment. In addition, insurance department officials told us that they believed providers were enrolling individuals in private insurance plans without telling them and paying their premiums and copays. According to these officials, when doing so, providers may lie about the individuals’ income status in order to qualify them for more generous insurance plans. Officials found that providers were billing individuals’ insurance $15,000 to $20,000 a month for urine drug testing and other services. Officials noted that they suspect that the alleged fraud was primarily being carried out by SUD treatment providers and treatment facilities that also own recovery homes. The officials said the department had not been able to file charges against any treatment providers, because it had been unable to collect the necessary evidence to do so. However, according to the officials, the state enacted legislation in 2016 that gave insurers and state regulatory agencies, such as the state’s insurance department and licensing office, the authority to review patient records and investigate providers that bill insurers. As we noted in our March 2018 report, this authority may help the insurance department and other Utah regulatory agencies better conduct investigations in the future. In addition to actions taken in response to state investigations, our March 2018 report described steps taken by three of the five selected states (Florida, Massachusetts, and Utah) to formally increase oversight of recovery homes by establishing state certification or licensure programs. Florida enacted legislation in 2015 and Massachusetts enacted legislation in 2014 that established voluntary certification programs for recovery homes. Further, Florida established a two-part program for both recovery homes and recovery home administrators (i.e., individuals acting as recovery home managers or operators). According to officials from the Florida state attorney’s office and Massachusetts Medicaid Fraud Control Unit, their states established these programs, in part, as a result of state law enforcement investigations. Utah enacted legislation in 2014 to establish a mandatory licensure program for recovery homes. According to officials from the Utah substance abuse agency and the state licensing office, Utah established its licensure program, in part, to protect residents’ safety and prevent their exploitation and abuse. In our March 2018 report, we found that although state recovery home programs in Florida and Massachusetts are voluntary, there are incentives for homes to become certified under these states’ programs, as well as incentives to become licensed under Utah’s programs. Specifically, all three states require that certain providers refer patients only to recovery homes certified or licensed by their state program; therefore, uncertified and unlicensed homes in the three states are ineligible to receive patient referrals from certain treatment providers. Further, state officials told us that state agencies are taking steps to ensure providers are making appropriate referrals. For example, according to officials from the Florida substance abuse agency, treatment providers may refer individuals to certified recovery homes managed by certified recovery home administrators only and must keep referral records. To become state-certified or licensed, recovery homes in Florida, Massachusetts, and Utah must meet certain program requirements, including training staff, submitting documentation (such as housing policies and a code of ethics), and participating in onsite inspections to demonstrate compliance with program standards. However, specific requirements differ across the three states. For example, while all three state programs require recovery home operators or staff to complete training, the number of hours and training topics differ. In addition, for recovery homes to be considered certified in Florida, they must have a certified recovery home administrator. Similar to Florida’s certification program for the homes, individuals seeking administrator certification must meet certain program requirements, such as receiving training on recovery home operations and administration, as well as training on their legal, professional, and ethical responsibilities. Features of the state- established oversight programs also differ across the three states, including program type, type of home eligible for certification or licensure, certifying or licensing body, and initial fees. As we noted in our March 2018 report, the state-established oversight programs in Florida, Massachusetts, and Utah also include processes to monitor certified or licensed recovery homes, and take action when homes do not comply with program standards. For example, an official from the Florida Association of Recovery Residences—the organization designated by the state to certify recovery homes—told us that the entity conducts random inspections to ensure that recovery homes maintain compliance with program standards. State-established oversight programs in the three states also have processes for investigating grievances filed against certified or licensed recovery homes. Further, officials from certifying or licensing bodies in all three states told us their organizations may take a range of actions when they receive complaints or identify homes that do not comply with program standards, from issuing recommendations for bringing homes into compliance to revoking certificates or licenses. According to officials from Florida’s certifying body, the entity has revoked certificates of recovery homes that have acted egregiously or have been nonresponsive to corrective action plans. Officials from the certifying and licensing bodies in Massachusetts and Utah told us that they had not revoked certificates or licenses, but had possibly assisted homes with coming into compliance with certification standards or licensure requirements. Officials from Ohio and Texas told us that their states had not established state oversight programs like those in Florida, Massachusetts, and Utah, but said their states had provided technical assistance and other resources to recovery homes in an effort to increase consistency, accountability, and quality. Officials from the Ohio substance abuse agency told us that since 2013 the state has revised its regulatory code to define recovery homes and minimum requirements for such homes. Officials also told us that the agency did not have authority to establish a state certification or licensure program for recovery homes. According to these officials, the state legislature wanted to ensure that Ohio’s recovery homes community maintained its grassroots efforts and did not want a certification or licensure program to serve as a roadblock to establishing additional homes. However, officials from the Ohio substance abuse agency told us that the agency encourages recovery homes to seek certification by the state’s NARR affiliate—Ohio Recovery Housing—to demonstrate quality. In addition, these officials told us that the state substance abuse agency also provided start-up funds for Ohio Recovery Housing, as well as continued funding for the affiliate to provide training and technical assistance, and to continue certifying recovery homes. According to officials from Ohio Recovery Housing, the NARR affiliate regularly provides the state’s substance abuse agency with a list of newly certified recovery homes, as well as updates on previously certified homes as part of ongoing efforts to develop a recovery home locator, under its contract with the agency. Officials from the Texas substance abuse agency told us that establishing a voluntary certification program would be beneficial. However, the state legislature had not enacted legislation establishing such a program at the time of our review. At the time of our report, the agency was in the process of developing guidance for providers on where and how to refer their patients to recovery housing, which includes a recommendation to send patients to homes certified by the Texas NARR affiliate. Chairman Grassley, Ranking Member Wyden, and Members of the Committee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff members have any questions concerning this testimony, please contact me at (202) 512-7114 or deniganmacauleym@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals who made key contributions to this testimony include Tom Conahan (Assistant Director), Kristin Ekelund (Analyst-in-Charge), Drew Long, Sarah Resavy, and Emily Wilson. Other staff who made key contributions to the report cited in the testimony are identified in the source product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Substance abuse and illicit drug use, including the use of heroin and the misuse of alcohol and prescription opioids, is a growing problem in the United States. Individuals with a substance use disorder may face challenges in remaining drug- and alcohol-free. Recovery homes can offer safe, supportive, drug- and alcohol-free housing to help these individuals maintain their sobriety and can be an important resource for recovering individuals. However, as GAO reported in March 2018, some states have conducted investigations of potentially fraudulent practices in some recovery homes. This statement describes (1) what is known about the prevalence of recovery homes across the United States; and (2) investigations and actions selected states have undertaken to oversee such homes. It is largely based on GAO's March 2018 report (GAO-18-315). For that report, GAO reviewed national and state data, among other things, and interviewed officials from the Department of Health and Human Services, national associations, and five states—Florida, Massachusetts, Ohio, Texas, and Utah. GAO selected these states based on their rates of opioid overdose deaths, their rates of dependence or abuse of alcohol and other drugs, and other criteria. In March 2018, GAO found that the prevalence of recovery homes (i.e., peer-run or peer-managed drug- and alcohol-free supportive homes for individuals in recovery from substance use disorder) was unknown. Complete data on the prevalence of recovery homes were not available, and there was no federal agency responsible for overseeing recovery homes that would compile such data. However, two national organizations collected data on the prevalence of recovery homes for a subset of these homes. The National Alliance for Recovery Residences (NARR), a national nonprofit and recovery community organization that promotes quality standards for recovery homes, collected data only on recovery homes that sought certification by some of its state affiliates. As of January 2018, NARR told us that its affiliates had certified almost 2,000 recovery homes, which had the capacity to provide housing to over 25,000 individuals. Oxford House, Inc. collected data on the number of individual recovery homes it charters. In its 2018 annual report, Oxford House, Inc. reported that there were 2,542 Oxford Houses in 45 states. The number of recovery homes that were not affiliated with these organizations was unknown. In March 2018, GAO also found that four of the five states in its review—Florida, Massachusetts, Ohio, and Utah—had conducted, or were in the process of conducting, investigations of potentially fraudulent recovery home activities in their states. Activities identified by state investigators included schemes in which recovery home operators recruited individuals with substance use disorder to specific recovery homes and treatment providers, and then billed those individuals' insurance for extensive and unnecessary drug testing for the purposes of profit. For example, officials from the Florida state attorney's office told GAO that, in some instances, substance use disorder treatment providers were paying $300 to $500 or more per week to recovery home operators for every individual the operators referred for treatment. Then, in one of these instances, the provider billed an individual's insurance for hundreds of thousands of dollars in unnecessary drug testing over the course of several months. Further, these officials told GAO that as a result of these investigations at least 13 individuals were convicted and fined or sentenced to jail time. To increase oversight, officials from three of the five states—Florida, Massachusetts, and Utah—said they had established state certification or licensure programs for recovery homes in 2014 and 2015. Officials from the other two states—Ohio and Texas—had not established such programs, but were providing training and technical assistance to recovery homes.", "document_type": "gao"}
{"report": "Depots are government-owned, government-operated industrial installations that maintain, overhaul, and repair a multitude of complex military weapon systems and equipment for the Department of Defense. Depots are essential to maintaining readiness for DOD and play a key role in sustaining weapon systems and equipment in meeting operational, contingency, and training requirements. There are 17 depots operated by the military services that perform depot-level maintenance on a wide range of vehicles and other military assets, including aircraft, engines, helicopters, combat vehicles, ships, and software. Five are Army depots, four are Naval shipyards, three are Navy fleet readiness centers, two are Marine Corps production plants, and three are Air Force air logistics complexes. Figure 1 below shows the location of these 17 depots across the United States. The depots are part of a larger, DOD-wide logistics enterprise that involves a number of different organizations (See fig. 2.). Office of the Under Secretary of Defense for Acquisition and Sustainment. This office is responsible for, among other things, ensuring the defense industrial base, including depots, is robust, secure, resilient and innovative. Office of the Assistant Secretary of Defense for Sustainment. This office serves as the principal assistant and advisor to the Under Secretary of Defense for Acquisition and Sustainment on material readiness. Among other responsibilities, the Assistant Secretary of Defense for Sustainment prescribes policies and procedures on maintenance, materiel readiness, and sustainment support. Office of the Deputy Assistant Secretary of Defense for Materiel Readiness. This office establishes and maintains maintenance policies and programs to maintain the desired levels of weapon systems and military equipment readiness to accomplish the Department's missions. Further, according to DOD officials as well as DOD’s March 2018 report to Congress on sharing best practices, the Office of the Deputy Assistant Secretary of Defense for Materiel Readiness has established a governance framework for materiel maintenance at DOD depots. There are a number of stakeholders involved in this framework, including the Maintenance Executive Steering Committee (Committee) and the Joint Group-Depot Maintenance. Maintenance Executive Steering Committee. This Committee consists of senior maintenance and logistics representatives from the Office of the Secretary of Defense, the Joint Staff, the Defense Logistics Agency, and the military services. According to DOD, this Committee advises the Deputy Assistant Secretary of Defense for Materiel Readiness on initiatives affecting efficiency, effectiveness, and affordability of maintenance management and operations. The Committee also serves as a forum for a coordinated review of maintenance policies, systems, programs and activities and helps optimize and steer DOD enterprise maintenance practices and strategy. Joint Group–Depot Maintenance. As a standing committee of the Maintenance Executive Steering Committee, the mission of the Joint Group–Depot Maintenance is to promote and review depot maintenance functions at the enterprise level to achieve effective and affordable depot maintenance support for weapon systems and to execute responsibilities assigned in DOD maintenance of military materiel policy. Military service organizations. Each military service has its own logistics or materiel command component, which provides day-to-day management and oversight of the military services’ depots. The Chairman of the Joint Chiefs of Staff is responsible for formulating policies for gathering, developing, and disseminating joint lessons learned for the armed forces. Chairman of the Joint Chiefs of Staff (CJCS) Instruction 3150.25G, Joint Lessons Learned Program, defines: best practice as “a validated method or procedure which has consistently shown results superior to those achieved with other means, and appears to be worthy of replication,” and lesson learned as “a resolved issue or best practice that improves operations or activities and results in an internalized change to capability, process, or procedure.” The Joint Staff’s Joint Lessons Learned Program collects, validates, and disseminates lessons learned to support sustainment and improvement of joint force readiness and effectiveness via refinements in doctrine, organization, training, materiel, leadership and education, personnel, facilities, and policy. Specific military service guidance on their respective lessons learned programs share the same purpose. Best practices and lessons learned are captured in the Joint Lessons Learned Information System—DOD’s system of record for lessons learned—and are generally focused on sharing operational information from after-action reports and joint training exercises, rather than maintenance-related lessons learned. The DOD maintenance community, including the military service logistics or materiel command component and depots, do not typically coordinate with the military services’ lessons learned centers or enter lessons learned into the Joint Lessons Learned Information System. Our prior work has identified multiple challenges that can affect depot performance, including having the right facilities and having personnel with the right skills, among other challenges (See fig. 3.). Specifically, in April 2019 we reported on the condition of facilities at DOD depots, such as the condition of these depots are poor and the age of equipment is generally past its useful life, and the military services do not consistently track the effect that these conditions have on depot performance. To address these challenges, we recommended that DOD improve its data collection on the effect of facilities and equipment condition on depot performance, among other things. DOD concurred, and stated, in general, that the Service Chiefs for the Army, Navy, Air Force, and Marine Corps will ensure that their respective material commands take actions to implement the recommendations for their respective service. Also, in December 2018 we reported on depot workforce challenges, such as hiring personnel in a timely manner and providing inexperienced personnel with the training necessary to become proficient in skilled operations. According to DOD officials, these workforce challenges contributed to delays in the maintenance of some weapon systems. To address these workforce challenges, we recommended that the military services assess the effectiveness of the actions they have taken to maintain critical skills in the depot workforce. DOD concurred, and stated that each of the four services will take action to assess the effectiveness of the hiring, training, and retention programs at their respective depots, shipyards, fleet readiness centers, and air logistics complexes. The Related GAO Products page at the end of this report provides a list of our depot-related reports and testimonies. DOD shares best practices and lessons learned among the depots through a variety of venues, including networking, working groups, and benchmarking. Networking. DOD shares best practices and lessons learned through informal networking, such as personal contacts and conferences. All 17 depots reported engaging in networking to share best practices and lessons learned and coordinating with their materiel commands, program managers and/or program offices, and academia. The majority of the depots also coordinated with industry, other depots, and/or a point of contact or group within the Office of the Secretary of Defense (see table 1 below). All 17 depots reported that the DOD Maintenance Symposium (Symposium), an annual department-wide conference addressing the maintenance of weapon systems and equipment, is the most regularly attended and most beneficial venue for networking. All 17 depots reported attending the Symposium regularly or occasionally, with depot officials stating in the survey and interviews that the Symposium provides opportunities to build relationships and network with peers in DOD and external contacts in industry. Depots reported in our survey that the Symposium was valuable because it offered opportunities to make contacts with equipment vendors and other services, as well as break-out sessions and informal discussions to exchange ideas. During the Symposium, a number of maintenance awards, including the Robert T. Mason Award for Depot Maintenance Excellence, are awarded to recognize maintenance excellence (see sidebar). Three depots reported that the recognition of the award-winning depots gives other depots the opportunity to reach out to the award-winning depots for relevant information. This success has been shared with Fleet Readiness Centers East and Southeast, which are both implementing similar systems. Successfully training new artisans is particularly important for depot performance, as our prior work has shown that this workforce is aging and the Department of Defense faces challenges in hiring and retaining workers with key skills. Officials cited examples of maintenance taking months or years longer than expected, in part due to shortages in skilled personnel. Working Groups. DOD depots’ leadership and staff use working groups and communities of practice as venues for the DOD maintenance community to collaborate and to share expertise on specific topics. When surveyed, 13 of 17 depots reported they share best practices and lessons learned in working groups, and they identified more than 60 such working groups. Our analysis of survey responses shows that depots value working groups because they improve depot support to the warfighter by allowing the depot to evaluate best practices, review new technology, exchange data, initiate relationships, and gain stakeholder support. In our interviews, depot officials affirmed the value of working groups to promote collaboration and open discussions among peers focused on specific topics of common interest. We found that the working groups fall into three topic areas: new technologies, specific weapon systems, and depot management. For example: New technologies. The Joint Technology Exchange Group was chartered to improve coordination in the introduction of new or improved technology, new processes, or new equipment into DOD depot maintenance activities. To do this, the Joint Technology Exchange Group facilitates a number of forums and working groups centered on specific technologies, which allow representatives from the depots to learn from other services, academia, and industry (See fig. 4.). One example of this is cold spray, a new technology that sprays high velocity metal particles to repair worn surfaces and damaged parts that are unrepairable by traditional processes. Working groups facilitated by the Joint Technology Exchange Group have shared the usefulness of cold spray technology, and 12 depots from all service branches reported that they have begun adopting the technology. One depot estimates that its annual savings from using cold spray will be $202,000 annually, as well as additional time savings. Weapon systems. According to Navy officials, depot officials and maintainers for the CH-53E/MH-53E heavy lift helicopter participate in the H-53 Fleet Support Team working group. Fleet Readiness Center East reported that its production team was able to implement lessons learned from this group for repairing misalignment in a piece of the helicopter’s tail. As a result, the safety of the helicopter was increased. See figure 5 for details on this heavy lift helicopter. Depot management. Depot commanders participate in the Industrial Base Commanders’ monthly teleconference to share best practices and lessons learned related, in part, to management of depot operations. Twelve of the 17 depots indicated that the Industrial Base Commanders’ monthly teleconference is beneficial. The depots reported that the Industrial Base Commanders’ monthly teleconference allows base commanders time to share and to work on specific depot maintenance problems and is particularly productive in the areas of personnel and policy. Benchmarking. To benchmark, depot officials visit another depot to compare performance and find improvement ideas, particularly best practices and lessons learned related to weapon systems and depot management. Our analysis of site visit and survey data shows 10 of the 17 depots reported benchmarking trips. For example, in 2018 the Marine Corps Albany Production Plant sent a team of managers and technicians from their electronics and fabrications branches on a benchmarking trip to learn best practices from the team at Tobyhanna Army Depot. They visited six areas, where they observed processes and ideas that they could take back to their plant. In its trip report, the Marine Corps Albany Production Plant team highlighted a number of processes that increased efficiency in the electronics shop at Tobyhanna Army Depot, such as steps to eliminate unnecessary travel in sheet metal processes and updated electronics workstations. According to our prior work, benchmarking is useful for reducing internal resistance to change—a barrier to sharing best practices and lessons learned cited by the depots—because knowing what others actually are accomplishing changes perceptions of what can be done and what should be attempted. One depot told us that it intentionally brings maintainers and depot officials together on benchmarking trips so that the maintainers can benefit firsthand from seeing the best practices and lessons learned. DOD has communication challenges, such as the lack of awareness of venues, that may hinder the ability of the 17 depots to share best practices and lessons learned. While many sharing venues exist, such as working groups, the depots’ knowledge of them has gaps. According to our survey, 12 of the 17 depots reported being unaware of the existence of some venues where best practices and lessons learned can be shared. Additionally, 7 of the 17 depots reported not knowing who to contact to participate in some venues for sharing best practices and lessons learned. Moreover, in our interviews officials explained that staff turnover is also a challenge. Specifically, officials from one depot said that when the depot representative to a venue leaves, the institutional knowledge of the venue and its point of contact can be lost. They recounted having to resort to cold-calling other depots for information. Depots also reported that their staff did not attend best practices and lessons learned venues because they believed that those venues were for higher command levels. For example, one depot expressed confusion about the Industrial Base Commanders’ meeting and reported that while the depot officials were aware of the meeting, they believed that it was for officials at a higher level, such as their Materiel Command. Department of Defense Instruction 4151.18 states that DOD materiel maintenance programs should adopt business practices and quality management processes to continuously improve maintenance operations and maintenance production, achieve cost savings and avoidance, and realize process cycle time reduction. Further, GAO’s Standards for Internal Control in the Federal Government states that management should communicate quality information down and across reporting lines to enable personnel to perform key roles in achieving objectives. However, the Office of the Secretary of Defense has not created, shared, or maintained a comprehensive and updated list of all depot-specific DOD sharing venues (i.e., working groups) that includes points of contact. Officials from the Office of the Secretary of Defense stated that the Joint Technology Exchange Group maintains a list on its website. However, the list is incomplete, only containing three of the over 60 working groups we identified in our analysis of our interview and survey data. Moreover, we found that not all depot officials were aware of the Joint Technology Exchange Group and so would not be familiar with the Joint Technology Exchange Group’s website. Without a centralized list of venues and points of contact, it is unclear what groups exist and who to contact to participate, which may impede sharing of best practices and lessons learned. Each military service has initiatives or organizations to encourage the sharing of best practice and lessons learned; however, the Army has not maintained its lessons learned organizations. The depots from the Navy, Marine Corps, and Air Force reported, in our survey and interviews, that their military services have initiatives and organizations that encourage knowledge sharing regarding best practices and lessons learned among the depots. For example: Navy’s Fleet Readiness Center’s Naval Sustainment System. The Naval Sustainment System is an initiative to increase maintenance capacity and readiness among the Navy’s fleet readiness centers by process reviews and benchmarking. The depots reported in our survey that it improves production by encouraging them to identify constraints and to share lessons learned. The Naval Sustainment System is also in the process of being adopted by the shipyards. Navy’s “One Shipyard” Concept. The “One Shipyard” concept is a Navy workforce initiative in which maintainers are exchanged among the shipyards to ensure that the shipyards will have the required number of workers and skill sets to meet current and planned maintenance requirements. A Navy depot stated that as a result of the communication required by this concept, they are better able to share best practices. Marine Corps’ Marine Depot Maintenance Command. Based on responses to our survey, Marine Corps officials stated that the Marine Corps depots have a single command structure. With this structure, all process improvement meetings are held with both depots in attendance, resulting in the sharing of best practices and lessons learned between the two depots. Air Force’s Art of the Possible. The Air Force Sustainment Center created this management program to focus attention on restrictions in workflow in the depots. Depots report that it creates a culture of collaboration and sharing of best practices and lessons learned because it focuses on process improvement and creates a culture in which it is acceptable to discuss problems with other depots. Competition for Workload To determine which depot will receive new workload, the Department of Defense (DOD) Instruction 4151.24, Depot Source of Repair Determination Process (Oct.13, 2017) outlines a process under which workloads necessary to sustain core logistics capabilities are assigned to DOD depots that have the requisite competencies. Two Army depots reported that this process created competition for workload that hinders sharing for them. Depot officials stated that they fear that other depots will take workload from them if they share weapons system maintenance best practices. In one such instance, Marine Corps depot officials stated they visited an Army depot and observed a best practice for repairing 50-caliber machine gun receivers. However, when the Marine Corps depot reached out for technical details, the Army depot was not inclined to share, for a variety of reasons including competition for the same workload. Then, the Marine Corps depot asked Marine Corps Logistics Command to facilitate, and they resolved the issue by finding a Navy depot that had similar technology and was willing to share. In contrast, the Army does not have similar initiatives or organizations. Army regulations direct the establishment and maintenance of two organizations for sharing depot best practices and lessons learned. First, Army Regulation 750-1 directs the Army Materiel Command to establish and maintain the Army Materiel Lessons Learned Analysis Program to identify potential systemic materiel sustainment issues and examine root and contributing causes. Second, Army Regulation 11-33 directs Army Materiel Command to establish and maintain the Center for Army Acquisition and Materiel Lessons Learned to provide support in the collection, analysis, dissemination, and archiving capability of materiel lessons learned, with the objective of creating a knowledge sharing culture within the Army. Moreover, the Standards for Internal Control in the Federal Government states that management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. Senior Army officials concurred that competition between depots for jobs can be a barrier for sharing, particularly when it involves the preservation of specific depot workloads. However, depots in other services did not report competition for workload to be a barrier to sharing. The Army stated it established these organizations for sharing materiel best practices and lessons learned; however, Army Headquarters, Army Materiel Command, and Army depot officials stated that they were not aware of analysis or knowledge sharing of depot best practices and lessons learned that were performed by these organizations. Further, the Army did not maintain these organizations for sharing materiel best practices and lessons learned. First, officials from Army Futures Command confirmed that the Army Materiel Lessons Learned Analysis Program was transferred from Army Materiel Command to Army Futures Command in July 2018 and no longer focuses specifically on materiel lessons learned. Second, the officials confirmed that the Army ceased to maintain the Center for Army Acquisition and Materiel Lessons Learned in early 2017 due to direct funding limitations. In addition, some Army depots reported being unable to identify peers in other depots to share with, and they reported that competition hinders sharing (see sidebar). Senior Army officials concurred that there are cultural challenges, which result in the depots being less open to sharing and implementing best practices and lessons learned. Establishing and maintaining effective organizations dedicated to sharing materiel best practices and lessons learned would encourage knowledge sharing among the Army depots. DOD is implementing some best practices and lessons learned among the 17 depots that have led to benefits, including cost and time savings. In response to our survey, 16 of the 17 depots reported benefits from successfully implementing best practices and lessons learned, such as sharing technology to reduce costs and improving maintenance processes to repair parts and systems. These implemented best practices and lessons learned can be defined as intra-service (within a military service), inter-service (between two or more military services), or DOD and external entities (between a military service and private industry). Intra-service collaboration. Depots within each military service are collaborating to implement best practices and lessons learned to improve depot management processes and repairs related to weapon systems. For example, Red River Army Depot implemented a best practice learned from Anniston Army Depot to improve its depot management process in meeting its production schedule. The production schedule is a plan that identifies, among other things, working hours for maintainers, available storage, and parts supply. To facilitate the implementation of this best practice, Army Tank-Automotive and Armaments Command, which oversees these two depots, hosted a joint event for the purpose of Anniston’s sharing how a small group of individuals at its depot is responsible for maintaining visibility of all end-item (i.e., components and parts ready for their intended use) production schedules. According to Army officials, Red River did not have an organization that performed a similar function, and during the joint event, depot officials from Red River saw this as a lesson learned that they could take back to their depot and implement. Additionally, Anniston shared how it conducts its risk assessments, or program reviews, and weekly execution meetings, among other processes, in meeting its production schedules. As a result, Army officials told us that Red River implemented the best practices they thought would be beneficial in helping them make progress in meeting their production schedules. In another example, two Air Force depots that maintain the Navy’s C-130 aircraft are working together to implement a best practice, which, according to program documentation, has led to cost and time savings (See fig. 6.). Specifically, the Navy’s C-130 aircraft, which, according to Ogden officials, is maintained at Ogden Air Logistics Complex and Warner Robins Air Logistics Complex, contains a shelf bracket, which holds the pieces of the aircraft together. The aircraft becomes structurally vulnerable and unfit for operations and training if the shelf bracket is removed. The process of blasting, inspecting, plating, and reinstalling the shelf bracket takes an average of 63 days. During this time, some maintenance activities cannot occur until the shelf bracket is reinstalled. To address this issue, engineers at Ogden told us they created a series of specially-sized pins to lock the Navy’s C-130 aircraft in place to help maintain the structural integrity of the airframe while other areas of the aircraft are being repaired. As a result of this best practice, maintainers have eliminated 16 days in the maintenance process for the C-130. Also, depot officials told us for a one-time cost of $13,000 for one set of specially-sized pins, eliminating 16 days in the maintenance process in turn generates a cost avoidance of $32,000 per day (the cost to dock the aircraft) or more than $500,000 per aircraft. In implementing this best practice, the total annual benefit to the C-130 fleet at Ogden amounts to 288 days of aircraft availability and about $9 million in cost avoidance. Officials at Ogden told us they have implemented this new process and are discussing this best practice with maintainers at Warner Robins for implementation at their depot as well. Further, Air Force depots are partnering to further implement another best practice, cold spray technology, which allows depots to repair damaged parts instead of replacing them. Replacing these damaged parts can be expensive or difficult if they are low in supply. Also, limited parts and long lead times can cause delays in the supply system, and existing repair processes have a long turnaround time. Cold spray technology has not been fully implemented; however, even with its limited implementation, cold spray technology has yielded cost and time benefits (See fig. 7.). According to Air Force officials, Ogden has been collaborating with the Oklahoma City Air Logistics Complex to cold spray its F-16 gearboxes until Ogden can obtain adequate workload to sustain the cold spray technology. According to Ogden officials and program documentation, cold spraying each gearbox costs about $1,300 whereas replacing each gearbox costs about $38,000; at 13 units per year, this amounts to almost $500,000 in annual cost avoidances. Additionally, it would take 95 weeks to build and receive a new gearbox unit; however, with the cold spray repair the unit is back in service in 4 weeks. Ogden officials are currently working to include cold spraying gearboxes for the F-15, C-5 and E-3 weapon systems to its workload. Inter-service collaboration. Depots from two or more military services are collaborating to implement best practices and lessons learned which has led to benefits. For example, the Navy’s Fleet Readiness Center Southwest implemented a best practice learned from Ogden Air Logistic Complex to improve testing of electrical circuits. Specifically, according to depot officials, a maintainer at Ogden created a method—Intermittent Fault Detection and Isolation System—which tests systems and software to detect, isolate, and repair intermittent problems due to open circuits, short circuits, and poor wiring by replicating the environment of the aircraft in flight (See fig. 8.). According to Ogden officials and program documentation, by implementing this best practice, they have recovered out-of-service assets and generated about $62 million in cost savings. For example, after testing its F-16 chassis, Ogden officials recovered 138 out- of-service assets—amounting to $42 million of flight hardware returning to service. Moreover, officials at Fleet Readiness Center Southwest visited Ogden during a benchmarking trip to discuss the process of implementing the Intermittent Fault Detection and Isolation System to test their systems. According to officials from the Office of the Secretary of Defense, the intermittent faults due to aircraft electrical systems amounted to more than $300 million in operating and support costs in fiscal year 2014. The Fleet Readiness Center Southwest used the Intermittent Fault Detection and Isolation System to test its F/A-18 aircraft generators, which provide electrical power to the aircraft. As a result of testing these generators using the Intermittent Fault Detection and Isolation System, the mean time between failures for the generators has increased, according to officials, from 104 flight hours to over 400 flight hours, and the Navy anticipates a reduction of about 30 to 90 days of repair time. DOD and external entities. Depots are also partnering with private industry to implement best practices and lessons learned, which has led to time-savings benefits (See fig. 9.). For example, according to program officials, the Air Force, Navy, original equipment manufacturer, and contractor collaborated to implement a best practice for the U-2 aircraft. Specifically, in 2018, generators for the Air Force’s U-2 aircraft had decreased their mean time between failures from 1,000 hours to 400 hours. To sustain the fleet, the Air Force was cannibalizing—removing parts from one aircraft to another—generators from aircraft in depot maintenance to those preparing for deployment. The U-2 program office identified the Navy’s F/A-18 A/B generator as similar to the U-2 generator and learned valuable information on the repair and overhaul process, root cause analysis of failure of critical parts, and the Navy’s recommendation for procuring and building overhaul generator kits. In order to implement the Navy’s processes, the Air Force program office, working with the original equipment manufacturer and contactor, incorporated the Navy’s best practices in overhauling its generator kit concept. As a result, the Air Force is no longer cannibalizing these generators and the mean time between failures has returned to about 1,000 hours of flight time. DOD has not been able to implement some best practices and lessons learned among the 17 depots, but DOD is taking steps to mitigate challenges to implementation. In its March 2018 Report to Congress on Sharing of Best Practices for Depot-Level Maintenance Among the Military Services, DOD noted some of the challenges in implementing best practices such as differing military service priorities, strategies, and resourcing of technologies and infrastructure. In responding to our survey, 15 of the 17 depots reported challenges in implementing best practices and lessons learned, including insufficient resources, restrictions related to information technology, approval process, and acquisition and contracting policies, among others (See table 2.). Insufficient resources. Ten of the 17 depots reported insufficient resources as a challenge to implementation for various reasons. First, depots reported not having adequate time, staff, or funding to attend knowledge sharing activities or to analyze data from best practices and lessons learned. According to depot officials, not being able to attend knowledge sharing activities has made networking more difficult because these activities allowed them to discuss best practices and lessons learned with colleagues from other depots and industry. Second, in addition to not having adequate funding, depots also reported identifying sources of funding as a challenge to implementing best practices and lessons learned for specific weapon systems. For example, according to officials from one depot, they have been unable to identify a funding source to implement the laser de-painting system for the F-16, which would allow the aircraft to stay in service longer and would produce less hazardous materials than the current blasting process to remove paint from the aircraft. Third, depots reported insufficient equipment to implement a best practice. For example, one depot reported not having enough hand-held tablets, which contain electronic technical data and best practices from private industry to assist maintainers working on a weapon system. Another depot reported that it has not implemented the tablets and are relying on paper documentation to maintain its weapon systems. According to depot officials, the lack of tablets has had direct effects at the depot, such as delays in standing-up new capability and maintainers waiting on available tablets to perform their work. To mitigate challenges with insufficient resources, DOD, military service, and depot officials have taken a variety of steps. For example, officials from the Office of the Secretary of Defense held an event through the Joint Technology Exchange Group to discuss available funding sources for new and emerging technologies, such as the funding sources for the cold spray technology. According to officials at a Navy depot, depots can petition the Office of Naval Research for federal laboratory designation. With this designation, depots can partner with private industry to evaluate technology in any area that is consistent with the federal laboratory’s mission and may receive funds from private industry for technology research and development. Specific to the tablets, depot officials told us that the materiel command has taken responsibility for managing the funding of these assets and the depots will receive a technical upgrade every 4 years. Moreover, in February 2019 the Office of the Secretary of Defense launched the Enterprise Sustainment Dashboard (Dashboard), a web-based tool that will provide access to an online central repository of sustainment data for the military services and will allow senior leaders to steer resources to needed programs. The Dashboard will allow users to analyze metrics such as materiel availability (condition of a weapon system to perform an assigned mission), operational availability (availability of active inventory to conduct military service operations), and cost per day availability (maintenance cost per day for a population of weapon systems by type, model, and series). The Dashboard will also consolidate inventory, availability, and cost data systems from each of the military services. This Dashboard is in its early phase and the implementation plan includes milestones extending into fiscal year 2020. Restrictions related to information technology. Ten of the 17 depots reported restrictions related to information technology as a challenge to implementation of best practices. Specifically, depots reported having outdated and incompatible software systems and a lack of a consolidated database for departments and product lines, which may hinder their ability to connect computer systems to automate a repair process. Additionally, depots stated that it may take years to obtain authority and approval to operate information technology systems, making data collection, sharing, and implementation of best practices difficult. For example, one depot reported a technology tool was not user friendly and had a rigid infrastructure, making it difficult for maintainers to use to analyze metrics to improve depot maintenance. Specifically, depot officials told us that this technology tool performs its functions as designed but is limited in its scope of meeting depot requirements, such as identifying bottlenecks in the maintenance process. In another example, one depot reported cybersecurity concerns with commercial off-the-shelf products, which may not be compatible with the depot’s information technology system. To mitigate challenges related to information technology, depots reported using information systems, such as SharePoint, as a primary source for collecting, storing, organizing, sharing, and accessing information via a web browser. For example, Navy officials told us that there are SharePoint sites for different departments within their organization, including portals dedicated to training, aircraft, and business processes and procedures, which capture best practices and lessons learned from subject matter experts. In another example, an Air Force depot reported that its SharePoint portal includes a section focused on practical problem solving methods for some of its continuous process improvement projects, such as balancing weight on an aircraft and issues related to the wings of the C-130T. Further, depot officials told us they conducted an analysis to mitigate concerns about a technology tool, mentioned above, that was not user friendly and had a rigid infrastructure. Based on this analysis, depot officials found a modeling and simulation tool that would help resolve challenges in several key areas, including projecting workload and personnel required to perform depot maintenance and determining the depot’s capability for the volume of work that can be inducted into the depot, among other areas. The modeling and simulation tool has not been implemented yet because it was recently funded in September 2019. Moreover, in 2018, we reported on steps that DOD is taking to improve its information technology systems. Specifically, the Secretary of Defense asked the Defense Business Board to provide actionable recommendations that DOD could adopt to transform its six core business processes, including acquisition and procurement, logistics and supply, and real property management, and their supporting information technology systems. We recommended, in part, that DOD identify timeframes and deliverables for identifying and adopting optimal information technology solutions. DOD concurred with this recommendation and is taking steps to improve its information technology systems, such as issuing its initial plan for business operations reform in April 2019, collecting federal and private industry benchmarks, and reviewing information technology costs. Approval process. Eight of the 17 depots reported that the approval process and guidance for implementing best practices is challenging. Specifically, depots reported that the layers of leadership approval prevent timely implementation of best practices and, at times, can cause enthusiasm for a project’s implementation to wane. Depot officials also told us that implementing new ideas for maintaining or repairing weapon systems is challenging because they have to get multiple approvals from their chain of command as well as the program manager for a specific weapon system, thus making implementation more difficult and less timely. For example, depot officials told us that implementing best practices at the depot from one weapon system to another requires retesting of the practice and approval from each program manager. Additionally, in response to the survey, a depot reported that many of the essential, time-sensitive engineering decisions for one of its new weapon system reside at another location, which has caused delays in making timely decisions. In another example, depot officials told us that they had to get approval from individual program managers to implement the cold spray technology and the Intermittent Fault Detection Isolation System. To mitigate challenges in the approval process, such as these, depot officials told us it is beneficial when technological development that affect the DOD-wide logistics enterprise or an entire military service occurred at a higher organizational level, making it easier for new ideas to be implemented at the lower levels. For example, one depot reported on the Navy’s approach of implementing a best practice across its platforms to eliminate corrosive plating on its weapon systems. Navy officials told us that these decisions are made at the headquarters level and implemented across the depots. Moreover, one depot reported allowing decision authority for specific weapon systems to reside within the depot, rather than at another location, to help the depot make timely decisions on implementing new ideas. Finally, the Office of the Assistant Secretary of Defense is providing specific guidance in implementing best practices and lessons learned, such as the memorandum issued in April 2019 on the Intermittent Fault Detection and Isolation System directing the military services to adopt this best practice. Acquisition and contracting policies. Five of the 17 depots reported acquisition and contracting policies as a challenge to implementation. Specifically, depots reported that current acquisition and contracting policies are complex and time consuming, which causes government to lag behind industry in implementing best practices. For example, officials from one depot told us that even when two depots need the same item to repair a weapon system, each depot was encouraged to pursue a separate contract. Depot officials described this as an inefficient and burdensome process, which sometimes resulted in an inferior item. Similarly, officials from another depot told us that they started an initiative to make equipment and software more similar across their service’s depots; however, they were unable to implement this initiative for similar reasons. Further, officials from one depot told us that the procurement of a weapon system does not always include access to all data necessary to maintain the system. According to depot officials, this limits their ability to implement a best practice or lesson learned from a similar weapon system because the contractor retains ownership of the intellectual property needed to repair or optimize the system. To mitigate challenges related to acquisition and contracting policies, depot officials told us that military services are purchasing enough new technology for all their depots rather than have each depot purchase technology individually. For example, according to Navy officials, they purchased the equipment to implement cold spray technology across all four shipyards, which makes implementing the best practice or lesson learned more timely. Additionally, officials from one depot told us that they use public-private partnerships to bridge gaps for systems that lack access to the necessary data rights to conduct maintenance on the systems. Our February 2019 report identified additional steps DOD is taking to mitigate challenges related to intellectual property, especially software sustainment. First, our prior work found that DOD is in the early stages of addressing a statutory provision for DOD to (1) develop policy on the acquisition or licensing of intellectual property; and (2) establish a cadre of intellectual property experts to help support the acquisition workforce on intellectual property matters. Second, in our prior work, we reported that DOD officials we spoke with emphasized that there are situations in which the data rights needed may not be known until years into sustainment and that it would be useful if data rights could have a pre-negotiated price and be an option as part of the initial contract. Such an option would give the government the right, but not the obligation, to purchase the data rights at the pre-negotiated price if needed in the future. The sharing and implementation of best practices and lessons among the 17 depots is crucial to sustaining military readiness by ensuring that the military services can regularly maintain critical weapon systems and return them to the warfighter for use in training and operations. Successful collaboration of maintenance best practices and lessons learned across military services, private industry, and academia is increasingly essential as DOD operates, and thus needs to maintain, weapon systems. DOD shares best practices and lessons learned among the depots through a variety of venues, including networking, working groups, and benchmarking. However, DOD has communication challenges, including a lack of awareness of many sharing venues, which may hinder the ability of the depots to share best practices and lessons learned. The Office of the Secretary of Defense has not created, shared, or maintained a comprehensive and updated list of all depot-specific DOD sharing venues (i.e., working groups) that includes points of contact. Without a centralized list and points of contact, it is unclear what groups exist and who to contact to participate, which may impede sharing of best practices and lessons learned. Further, while the Army stated it established lessons learned organizations for sharing materiel best practices and lessons learned, it did not maintain them due to organizational restructuring and resource constraints. Establishing and maintaining effective organizations dedicated to sharing materiel best practices and lessons learned would encourage knowledge sharing among the Army depots. We are making two recommendations, including one to the Under Secretary of Defense for Acquisition and Sustainment and one to the Secretary of the Army. Specifically, the Secretary of Defense should direct that: The Under Secretary of Defense for Acquisition and Sustainment should ensure that the Deputy Assistant Secretary of Defense for Materiel Readiness create, share, and maintain a comprehensive and up-to-date list of all DOD sharing venues (i.e., working groups), including points of contact, related to depot maintenance. (Recommendation 1) The Secretary of the Army should ensure that Army Materiel Command reestablish and maintain organizations dedicated to sharing materiel best practices and lessons learned, as required by Army regulations. (Recommendation 2) We provided a draft of this report to DOD for review and comment. In written comments on a draft of this report, DOD concurred with the recommendations. DOD’s comments are reprinted in their entirety in appendix III. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretaries of the Army, Navy, and Air Force, and the Commandant of the Marine Corps. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff has any questions about this report, please contact Diana Maurer at (202) 512-9627 or maurerd@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to this report are listed in appendix IV. To conduct the work for our reporting objectives, we reviewed relevant laws and the Department of Defense (DOD) and military service guidance that govern depot maintenance and the sharing of best practices and lessons learned. We included in our scope DOD depots performing major depot-level maintenance. We conducted a survey of DOD’s 17 depots performing depot-level maintenance to gain an understanding of how each depot shares with each other and implements best practices and lessons learned. The response rate for the survey was 100 percent. These depots included: Anniston Army Depot, Anniston, Alabama Corpus Christi Army Depot, Corpus Christi, Texas Letterkenny Army Depot, Letterkenny, Pennsylvania Red River Army Depot, Texarkana, Texas Tobyhanna Army Depot, Tobyhanna, Pennsylvania Norfolk Naval Shipyard, Portsmouth, Virginia Pearl Harbor Naval Shipyard, Honolulu, Hawaii Portsmouth Naval Shipyard, Kittery, Maine Puget Sound Naval Shipyard, Bremerton, Washington Fleet Readiness Center East, Cherry Point, North Carolina Fleet Readiness Center Southeast, Jacksonville, Florida Fleet Readiness Center Southwest, San Diego, California Albany Production Plant, Albany, Georgia Barstow Production Plant, Barstow, California Ogden Air Logistics Complex, Ogden, Utah Oklahoma City Air Logistics Complex, Oklahoma City, Oklahoma Warner Robins Air Logistics Complex, Warner Robins, Georgia We analyzed survey responses to gain an understanding, for example, of which depot officials are coordinating with others to share best practices and lessons learned, which sharing venues are attended, and the extent to which this information sharing is beneficial. To ensure that the survey questions were clear, comprehensible, and technically correct, we conducted expert reviews of our draft survey with four subject matter experts with knowledge and experience in auditing DOD depots. We also conducted two pre-tests of our draft survey with the depot commanders of Anniston Army Depot and Warner Robins Air Logistics Complex, respectively.During each pre-test, conducted by teleconference, we read the instructions and each survey question aloud and asked the depot commanders to tell us how they interpreted the question. We then discussed the instructions and questions with each depot commander to identify any problems and potential solutions by determining whether (1) the instructions and questions were clear and unambiguous, (2) the terms we used were accurate, (3) the survey was unbiased, and (4) the survey did not place an undue burden on the depot officials completing it. We noted any potential problems and modified the survey based on feedback from the subject matter experts and depot commanders, as appropriate. We sent a fillable survey and a cover email to 17 depots on May 29, 2019, and asked them to complete the survey and email it back to us by June 14, 2019. We closed the survey on July 3, 2019. Data were auto- extracted from the Adobe PDF form into an Excel spreadsheet. Our examination of the survey results included both a quantitative data analyses on closed-ended questions and a review of open-ended responses to identify common themes. Additionally, to gather detailed examples of DOD’s efforts to share best practices and lessons learned, we visited a non-generalizable sample of 5 depots (Anniston Army Depot, Anniston, Alabama; Norfolk Naval Shipyard, Portsmouth, Virginia; Fleet Readiness Center Southwest, San Diego, California; Marine Corps Albany Production Plant, Albany, Georgia; and Ogden Air Logistics Complex, Ogden, Utah). To select our sample, we considered variation in geographic location, military service representation, and types of weapon systems maintained. At these sites, we conducted group discussions with individuals across the depot to gain insight into their roles in sharing best practices and lessons learned. Qualitative data analyses were conducted by our staff who have subject matter expertise to identify themes and select examples of best practices or lessons learned shared through collaboration with another depot. We then obtained and analyzed documentation of sharing, such as working group charters and trip reports documenting results from visiting another depot; as well as benefits experienced from implementing a best practice or lessons learned, including time and cost savings. We interviewed officials from the Office of the Under Secretary of Defense (Acquisition and Sustainment) (Deputy Assistant Secretary of Defense for Materiel Readiness), Joint Chiefs of Staff (Joint Lessons Learned Division), and the military service headquarters (Headquarters, Department of Army G4; Deputy Assistant Secretary of the Navy for Expeditionary Programs and Logistics Management; Headquarters Marine Corps, Installations & Logistics; and Air Force Acquisition, Logistics & Product Support. We also interviewed officials from the military service logistics or materiel components (Army Materiel Command; Naval Sea Systems Command; Naval Air Systems Command (Commander, Fleet Readiness Center); Marine Corps Logistics Command; and the Air Force Materiel Command) as well as the military lessons learned centers (Center for Army Lessons Learned, Naval Warfare Development Command, Marine Corps Center for Lessons Learned, and the Air Force LeMay Center for Lessons Learned). Finally, we reviewed our prior reports related to challenges experienced at DOD depots and DOD’s report to Congress on the sharing of best practices for depot-level maintenance among the military services. We assessed the documentary and testimonial evidence we collected against DOD and military service guidance on lessons learned and materiel maintenance and GAO’s Standards for Internal Control in the Federal Government. Specifically, the information and communication component of internal control—the actions management uses to internally communicate the necessary quality information to achieve the entity’s objectives—was significant to this audit. We conducted this performance audit from January 2019 through January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. During the course of our work examining the extent to which the Department of Defense (DOD) experiences benefits and has challenges with (1) sharing and (2) implementing best practices and lessons learned among the depots, we collected information from the depots on the working groups and communities of practice in which they participate. The list below is compiled from analysis of our survey data, in which we surveyed all 17 of DOD’s depots, as well as the interviews we conducted during our site visits to a non-generalizable sample of five depots. Note that this is not a list of all the possible working groups and communities of practice which exist among the depots, simply those which the depots shared with us. 1. 448th Supply Chain Management Wing 2. Air Force Metrology and Calibration Working Group 3. Air Force Sustainment Center Logistics Directorate’s Strategic 4. Aircraft Cyber Threat Working Group 5. Aircraft Maintenance Group Summit 6. Aircraft Storage Strikeboard 8. Army Safety and Occupational Health Information Management 9. Army Safety and Occupational Health Management System Working 10. Carrier Team One 11. Cold Spray Action Team 12. Commander, Fleet Readiness Centers Advanced Technology & 13. Commercial Technologies for Maintenance Activities Working Group – 14. Commodities, Electronics, Missiles, & Propulsion Maintenance 15. Coordinate Measuring Machine Community of Practice 16. Corporate Electrical Community of Practice 17. Corrosion Control Working Groups 18. Cyber Resiliency Office for Weapon Systems Working Groups 19. Depot Maintenance Activation Working Group 20. Depot Maintenance Enterprise Action Group 21. Diminishing Manufacturing Sources and Material Shortages 22. DOD Digital Manufacturing Users Group 23. DOD Unmanned Systems & Robotics Summit 24. DOD Voluntary Protection Programs 26. Enterprise IT Systems Strikeboard 27. F-35 Joint Risk Working Group 28. H-53 Fleet Support Team 29. Heavy Metal Working Group 30. Industrial Base Commander’s Meetings 31. Integrated Quality Teams 32. Investment Working Group 33. Joint Additive Manufacturing Steering Group 34. Joint Additive Manufacturing Working Group and Community of 35. Joint Intermittence Team 36. Joint Requirements Working Group 37. Joint Robotics Working Group 38. Joint Technology Exchange Group 39. Metrics Community of Practice 40. Modernization Working Group 41. National Center for Defense Manufacturing and Machining 42. Naval Surface Warfare Center, Carderock Division Human 43. Naval Undersea Warfare Center Division, Keyport Human 44. Navy Forum for Small Business Innovation Research/Small Business Technology Transfer Transition 45. Non-Destructive Inspection Forum 46. Non-Destructive Testing Working Group 47. Norfolk Naval Shipyard Technology and Innovation Community of 48. Organic Industrial Base Commander’s Summit 49. Project Management Executive Steering Committee 50. Public-Private Partnership Community of Practice 51. Quality Performance System Community of Practice 52. Quality Work Environment Working Group 53. Residential Economic Development Inc. 54. RepTech Working Group 55. Shipyard departmental level Communities of Practice: C200, C1200, C1200N, C600, C400, etc. 56. Shipyard-only Community of Practice 57. Software Engineering Institute Agile Collaboration Group 58. Software Maintenance Group Summit 59. Sub Team One 60. Tri-Air Logistics Complex Summits 61. Weapon-system Specific Enterprise Cross-talks: C-130 Enterprise Crosstalk, A-10 Enterprise Crosstalk, etc. In addition to the contact listed above, Jodie Sandel (Assistant Director), Laura Czohara (Analyst-in-Charge), Clarine Allen, Felicia Lopez, Amie Lesser, Christina Murphy, Clarice Ransom, Andrew Stavisky, and Courtney Tepera made key contributions to this report. Navy Maintenance: Persistent and Substantial Ship and Submarine Maintenance Delays Hinder Efforts to Rebuild Readiness. GAO-20-257T. Washington, D.C.: December 4, 2019. Naval Shipyards: Key Actions Remain to Improve Infrastructure to Better Support Navy Operations. GAO-20-64. Washington, D.C.: November 25, 2019. F-35 Aircraft Sustainment: DOD Faces Challenges in Sustaining a Growing Fleet. GAO-20-234T. Washington, D.C.: November 13, 2019. Depot Maintenance: DOD Should Adopt a Metric That Provides Quality Information on Funded Unfinished Work. GAO-19-452. Washington, D.C.: July 26, 2019. Military Depots: Actions Needed to Improve Poor Conditions of Facilities and Equipment That Affect Maintenance Timeliness and Efficiency. GAO-19-242. Washington, D.C.: April 29, 2019. Weapon System Sustainment: DOD Needs to Better Capture and Report Software Sustainment Costs. GAO-19-173. Washington, D.C.: February 25, 2019. Army Modernization: Steps Needed to Ensure Army Futures Command Fully Applies Leading Practices. GAO-19-132. Washington, D.C.: January 23, 2019. DOD Depot Workforce: Services Need to Assess the Effectiveness of Their Initiatives to Maintain Critical Skills. GAO-19-51. Washington, D.C.: December 14, 2018. Navy and Marine Corps: Rebuilding Ship, Submarine, and Aviation Readiness Will Require Time and Sustained Management Attention. GAO-19-225T. Washington, D.C.: December 12, 2018. Navy Readiness: Actions Needed to Address Costly Maintenance Delays Facing the Attack Submarine Fleet. GAO-19-229. Washington, D.C.: November 19, 2018. Depot Maintenance: DOD Has Improved the Completeness of Its Biennial Core Report. GAO-19-89. Washington, D.C.: November 14, 2018. Air Force Readiness: Actions Needed to Rebuild Readiness and Prepare for the Future. GAO-19-120T. Washington, D.C.: October 10, 2018. Weapon System Sustainment: Selected Air Force and Navy Aircraft Generally Have Not Met Availability Goals, and DOD and Navy Guidance Need to Be Clarified. GAO-18-678. Washington, D.C.: September 10, 2018. Military Readiness: Analysis of Maintenance Delays Needed to Improve Availability of Patriot Equipment for Training. GAO-18-447. Washington, D.C.: June 20, 2018. F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency. GAO-18-75. Washington, D.C.: October 26, 2017. Depot Maintenance: Executed Workload and Maintenance Operations at DOD Depots. GAO-17-82R. Washington, D.C.: February 3, 2017. Depot Maintenance: Improvements to DOD’s Biennial Core Report Could Better Inform Oversight and Funding Decisions. GAO-17-81. Washington, D.C.: November 28, 2016. Naval Shipyards: Actions Needed to Improve Poor Conditions that Affect Operations. GAO-17-548. Washington, D.C.: September 12, 2017. Army Working Capital Fund: Army Industrial Operations Could Improve Budgeting and Management of Carryover. GAO-16-543. Washington, D.C.: June 23, 2016. Military Readiness: Progress and Challenges in Implementing the Navy’s Optimized Fleet Response Plan. GAO-16-466R. Washington, D.C.: May 2, 2016. Defense Inventory, Further Analysis and Enhanced Metrics Could Improve Service Supply and Depot Operations. GAO-16-450. Washington, D.C.: June 9, 2016. Navy Working Capital Fund: Budgeting for Carryover at Fleet Readiness Centers Could Be Improved. GAO-15-462. Washington, D.C.: June 30, 2015. Sequestration: Documenting and Assessing Lessons Learned Would Assist DOD in Planning for Future Budget Uncertainty. GAO-15-470. Washington, D.C.: May 27, 2015. Operational Contract Support: Actions Needed to Enhance the Collection, Integration, and Sharing of Lessons Learned. GAO-15-243. Washington, D.C.: March 16, 2015.", "summary": "DOD operates depots nationwide to maintain complex weapon systems and equipment through overhauls, upgrades, and rebuilding. These depots are crucial to sustaining military readiness by ensuring that the military services can regularly maintain critical weapon systems and return them to the warfighter for use in training and operations. For fiscal year 2018, DOD reported $19 billion in total maintenance expenditures and about 84,000 personnel performing depot-level maintenance. In June 2018, the Senate Armed Services Committee, in a report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2019, included a provision for GAO to review DOD's sharing and implementation of best practices and lessons learned among the depots. GAO evaluated the extent to which DOD experiences benefits and has challenges with (1) sharing and (2) implementing best practices and lessons learned among the depots. GAO reviewed agency guidance; surveyed 17 depots; conducted site visits at five depots; and interviewed DOD, military service, and depot officials. The Department of Defense (DOD) experiences benefits from sharing best practices and lessons learned among its depots, but communication and organization challenges exist. Best practices and lessons learned are shared among the depots through a variety of venues, including networking, working groups, and benchmarking trips to other depots. However, DOD has communication challenges, such as the lack of awareness of venues for sharing information. While Office of the Secretary of Defense officials reported posting a list of working groups, the list only contains three of the more than 60 working groups GAO identified. Without a centralized list of sharing venues and points of contact, it is unclear what groups exist and who to contact to participate, which may impede sharing of best practices and lessons learned. Further, while the Army stated it established lessons learned organizations for sharing maintenance best practices and lessons learned, it did not maintain them due to organizational restructuring and resource constraints. Establishing and maintaining effective organizations dedicated to sharing materiel best practices and lessons learned would encourage knowledge sharing among the Army depots. DOD is experiencing benefits and taking steps to mitigate challenges with implementing best practices and lessons learned among the depots. Depots reported that implementing some best practices and lessons learned has led to benefits, including time and cost savings. For example, Navy Fleet Readiness Center Southwest, California, implemented an intermittent fault detection system from Ogden Air Logistics Complex, Utah, on its F/A-18 aircraft generators. According to officials, the depot reduced repair time from 90 days to 30 days and quadrupled the generators' time between failures. Depots reported a variety of challenges to implementing lessons learned and best practices, including a lack of resources, lengthy approval processes, and acquisition and technology restrictions. DOD is taking steps to mitigate challenges to implementation, such as creating a new technology tool for viewing metrics on weapon systems' cost and availability which will allow senior leaders to steer resources to needed programs. GAO is making two recommendations to improve the depots' ability to share best practices and lessons learned by creating a comprehensive list of sharing venues, including points of contact, and re-establishing and maintaining materiel lessons learned organizations. DOD concurred with the recommendations.", "document_type": "gao"}
{"report": "Wireless broadband connects users to the Internet using spectrum to transmit data between the customer’s location and the service provider’s facility, and can be transmitted using fixed wireless and mobile technologies, as shown in figure 1. Fixed wireless broadband technologies establish an Internet connection between fixed points—such as from a radio or antenna that may be mounted on a tower, to a stationary wireless device located at a home— and generally requires a direct line of sight. Mobile wireless broadband technologies also establish an Internet connection that requires the installation of antennas, but this technology provides connectivity to customers wherever they are covered by service, including while on the move, such as with a cell phone. Spectrum is the resource that makes wireless broadband connections possible. Spectrum frequency bands each have different characteristics that result in different levels of ability to cover distances, penetrate physical objects, and carry large amounts of information. Examples of some of the frequency bands that can be used by commercial and nonfederal entities for broadband services are shown in figure 2. The frequency bands that can be used for broadband services are either licensed or unlicensed. For licensed spectrum, FCC can assign licenses through auctions, in which prospective users bid for the exclusive rights to transmit on a specific frequency band within geographic areas. Having exclusive rights ensures there will be no interference from other spectrum users in that band. License holders may sell or lease their license, in whole or in part, to another provider, a process that is known as a secondary market transaction, with FCC’s approval. FCC has assigned licenses administratively in two frequency bands that can be used for broadband services. FCC also authorizes the use of unlicensed spectrum, where an unlimited number of users can share frequencies without a license, such as wireless microphones, baby monitors, and garage door openers. In contrast to users of licensed spectrum, unlicensed users have no regulatory protection from interference by other licensed or unlicensed users in the bands. In March 2010, FCC issued the National Broadband Plan that included a centralized vision for achieving affordability and maximizing use of high- speed Internet. The plan made recommendations to FCC, including that FCC should take into account the unique spectrum needs of tribal communities when implementing spectrum policies and evaluate its policies and rules to address obstacles to spectrum access by tribal communities. With regard to tribal lands, the plan recommended that FCC increase its commitment to government-to-government consultation with tribal leaders and consider increasing tribal representation in telecommunications planning. FCC established the Office of Native Affairs and Policy in July 2010 to promote the deployment and adoption of communication services and technologies to all native communities, by, among other things, ensuring consultation with tribal governments pursuant to FCC policy. For our November 2018 report, we identified 18 tribal entities from FCC’s license data that held active spectrum licenses in bands that can be used to provide broadband services as of September 2018. Of those 18, 4 obtained the spectrum through a secondary market transaction and 2 from an FCC spectrum auction. We interviewed 16 tribal entities that were using wireless technologies at the time to provide service, and 14 told us that they were accessing unlicensed spectrum to do so. While representatives from most of the 16 tribal entities reported some advantages of unlicensed spectrum, such as the spectrum is available at no cost, they also discussed their experiences with the limitations of unlicensed spectrum, including issues with interference and speed or capacity. Some of the stakeholders we contacted and FCC have highlighted the importance of exclusive-use licensed spectrum for tribal entities. For example, FCC’s Office of Native Affairs and Policy reported in 2012 that unlicensed spectrum is not an option across all tribal lands and that tribal access to robust licensed spectrum is a critical need. In addition, representatives from the stakeholders we interviewed told us that there are non-technological benefits for tribal entities to obtain greater access to licensed spectrum, including: enhanced ability to deliver additional Internet services, enhanced ability to sell or lease spectrum for profit, and additional opportunities to obtain federal funding that requires entities to hold or have access to licensed spectrum. Furthermore, two tribal stakeholders and representatives from several tribal entities told us that having access to licensed spectrum would enable tribes to exercise their rights to sovereignty and self- determination. For example, representatives from four of the tribal entities told us that having access to licensed spectrum would ensure that spectrum is being used in a way that aligns with tribal goals and community needs, further supporting their rights to self-determination. In our November 2018 report, we described barriers tribal entities reported facing in accessing licensed spectrum. First, representatives from tribal entities we contacted said that obtaining a spectrum license through an auction was too expensive for many tribal entities. Indeed, over 60 percent (983 of 1,611) of the winning bids from a 2015 spectrum auction were more than $1 million. Representatives from some tribal entities told us they were unable to obtain financing to participate in auctions because tribal governments cannot use tribal lands as collateral to obtain loans and that participating in spectrum auctions requires auction-specific expertise that tribal entities may not have. Second, tribal entities reported facing barriers obtaining spectrum through secondary market transactions. Most of the spectrum allocated for commercial use has already been assigned through spectrum auctions and other mechanisms to private providers that may not be providing service on tribal lands. As such, there may be tribal areas where providers hold licenses for bands but are not using the spectrum to provide Internet service. All three of the tribal associations we contacted confirmed that there were unused spectrum licenses over tribal lands, and representatives from a nationwide provider indicated that they only deploy services if there is a business case to support doing so. Accordingly, the secondary market is one of few avenues available to tribal entities that would like to access licensed spectrum. However, representatives from tribal entities we contacted told us it could be challenging to participate in the secondary market because there is a lack of willing sellers, license holders are not easily identified, and tribal entities may not be aware of how to pursue secondary market transactions. For example, representatives from a tribal entity that had been successful in obtaining a license through the secondary market told us that an Indian-owned telecommunications consulting company was pivotal in identifying the license holder and facilitating the transaction, and without such assistance, the transaction would not have occurred. At the time of our November 2018 report, we found that FCC had taken some actions to increase tribal access to spectrum. In particular: FCC issued a proposed rulemaking in March 2011 that sought comments on three proposals to create new spectrum access opportunities for tribal entities (see fig. 3). As of July 12, 2019, FCC had not adopted new rules or taken further action on the 2011 rulemaking. FCC issued a proposed rulemaking in May 2018 that sought comment on establishing a priority window for tribal nations located in rural areas to obtain a license in the Educational Broadband Service spectrum band (also known as the 2.5 GHz band). In the proposed rulemaking, FCC had found that significant portions of this band were not being used, primarily in rural areas. FCC had not finalized this rule at the time of our November 2018 report, but published a draft order in June 2019 that would establish a priority filing window so that tribal entities could get access to unassigned spectrum in the 2.5 GHz band on rural tribal lands prior to an FCC auction. FCC adopted this order on July 10, 2019. FCC’s Office of Native Affairs and Policy conducts training, consultation, and outreach to tribal entities on spectrum-related issues, such as communicating with tribal entities prior to FCC auctions or when FCC regulatory actions or policies would affect tribal governments and spectrum over their lands. FCC’s 2010 National Broadband Plan stated that ongoing measurement of spectrum utilization should be developed to better understand how spectrum resources are being used because some studies indicated that spectrum goes unused in many places much of the time. The plan stated that any spectrum utilization studies that FCC conducts should identify tribal lands as distinct entities. The plan also stated that FCC should make data available that would promote a robust secondary market for spectrum licenses, such as information on how and to whom spectrum is allocated on tribal lands. In FCC’s 2018 strategic plan, FCC stated that it will implement ongoing initiatives that will assist in spectrum policy planning and decision making, promote a robust secondary market in spectrum, and improve communications services in all areas of the United States, including tribal areas. Additionally, federal internal control standards state that agencies should use quality information, including information that is complete, to inform the decision-making processes and communicate with external entities. Tribal governments are an example of such external entities. However, in our 2018 report, we found that FCC had not consistently collected data related to tribal access to spectrum or communicated important information to tribes. In particular: FCC did not collect data on whether spectrum license-holders or auction applicants are tribal entities. Without this information, FCC did not have a comprehensive understanding of the extent that tribal entities are attempting to obtain or access licensed spectrum or have been successful at obtaining and accessing it. FCC did not analyze the extent that unused licensed spectrum exists over tribal lands, even though FCC had the information—broadband availability data from providers and information on geographic areas covered by spectrum licenses—needed for such an analysis. Although FCC officials told us evaluating the effectiveness of FCC’s secondary market policies is a way to increase the use of unused spectrum, FCC’s approach did not include an analysis of unused spectrum licenses on tribal lands. As a result, FCC’s evaluations of the secondary market may not have accurately reflected how its policies affect tribal entities. Because the secondary market is one of few ways for tribal entities to access licensed spectrum, such an assessment would enable FCC to better promote a robust secondary market that provides opportunities for tribes to access spectrum. FCC did not communicate information to tribes that could benefit them in their efforts to obtain spectrum in the secondary market. As described earlier, the secondary market is a significant mechanism for tribal entities to obtain spectrum licenses, but representatives from the tribal entities we interviewed reported challenges related to participating in the secondary market, such as not knowing whom to contact should they wish to engage in a secondary market transaction to obtain a spectrum license. We concluded that FCC’s efforts to promote and support tribal entities’ access to spectrum had done little to increase tribal use of spectrum. In particular, FCC lacked information that could help inform its decision- making processes related to spectrum policy planning, which is intended to improve communications services in all areas of the United States, including tribal lands. By collecting data on the extent that tribal entities are obtaining and accessing spectrum, FCC could better understand tribal spectrum issues and use this information as it implements ongoing spectrum initiatives. Furthermore, the ability of tribal governments to make informed spectrum planning decisions and to participate in secondary market transactions is diminished without information from FCC on the spectrum transactions that occur over tribal lands. Providing this information directly to tribal entities could enable them to enter into leasing, partnership, or other arrangements to obtain spectrum. In our November 2018 report, we recommended that FCC (1) collect data on the extent that tribal entities are obtaining and accessing spectrum and use this information as FCC implements ongoing spectrum initiatives; (2) analyze data to better understand the extent that unused spectrum licenses exist over tribal lands, such as by analyzing the data for a sample of tribal lands, and as appropriate use this information to inform its oversight of the secondary market; and (3) make information on spectrum-license holders more accessible and easy to understand for interested parties, including tribal entities, to promote their ability to purchase or lease spectrum licenses from other providers. FCC agreed with these recommendations and described the actions it plans to take to implement them. For example, according to FCC, it will consider ways to collect data on the extent to which tribal entities are obtaining and accessing spectrum; analyze data from a sample of spectrum licenses on tribal lands to inform FCC’s spectrum policies; and transition to a more user-friendly system for its licensing data. Chairman Hoeven, Vice Chairman Udall, and Members of the Committee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staff have any questions about this testimony, please contact Andrew Von Ah, Director, Physical Infrastructure Issues at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Sally Moino and Anne Doré. Other staff who made contributions to the report cited in this testimony are identified in the source product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Broadband service on tribal lands continues to lag behind the rest of the country, especially on rural tribal lands. Broadband service can be delivered through wireless technologies using radio frequency spectrum. According to FCC, increasing tribal access to spectrum would help expand broadband service on tribal lands. This statement is based on GAO's November 2018 report ( GAO-19-75 ) related to spectrum use for broadband services by tribal entities and selected updates. Specifically, it discusses (1) tribal entities' ability to obtain and access spectrum to provide broadband services and the reported barriers that may exist, and (2) the extent to which FCC promotes and supports tribal efforts to obtain and access spectrum. For that report, GAO interviewed 16 tribal entities that were using wireless technologies. Selected entities varied geographically, among other characteristics. GAO analyzed FCC's license and auction data as of September 6, 2018, reviewed FCC's rulemakings on spectrum for broadband services, and interviewed other tribal and industry stakeholders and FCC officials. The information obtained was not generalizable to all tribes or industry participants. As an update, GAO reviewed FCC's June 2019 draft order related to spectrum in the 2.5 GHz band. The tribal entities—tribal governments and tribally owned telecommunications providers—GAO contacted for its November 2018 report cited various barriers to obtaining spectrum licenses in bands that can be used to provide broadband services. Based on data from the Federal Communications Commission (FCC) as of September 2018, GAO identified 18 tribal entities that held active spectrum licenses in such bands. For example, of these 18 tribal entities, 4 obtained licenses through secondary market transactions—that is, they bought or leased the license from another provider, and 2 obtained a license through an FCC spectrum auction. The barriers tribal officials identified to obtaining licensed spectrum include high costs at auctions and, in the case of secondary market transactions, a lack of information on who holds licenses over tribal lands. Because most spectrum allocated for commercial use has already been assigned, the secondary market is one of the few avenues available to tribal entities that would like to access licensed spectrum. At the time of GAO's November 2018 report, FCC had taken some actions to increase tribal access to spectrum. For example, FCC issued proposed rulemakings in 2011 and 2018 that sought comment on tribal-specific proposals, such as establishing tribal-licensing priorities and initiating processes to transfer unused spectrum licenses to tribal entities. FCC had not finalized these rules at the time of GAO's report, but FCC published a draft order in June 2019 that would create a tribal-licensing priorty window, whereby tribal entities would have an opportunity to obtain spectrum in the 2.5 gigahertz (GHz) band prior to the spectrum being auctioned. FCC adopted the order on July 10, 2019. FCC stated that it will implement spectrum initiatives and that it recognizes the importance of promoting a robust secondary market to improve communications throughout the United States, including tribal lands. However, GAO found that FCC had not consistently collected data related to tribal access to spectrum. For example: FCC did not collect data on whether spectrum auction applicants are tribal entities and therefore did not have a comprehensive understanding of the extent that tribal entities are attempting to obtain licensed spectrum. FCC did not analyze the extent that unused licensed spectrum exists over tribal lands. Although FCC officials said evaluating the effectiveness of FCC's secondary market policies is a way to increase the use of unused spectrum, FCC's approach did not include an analysis of unused spectrum licenses on tribal lands. As a result, FCC's evaluations of the secondary market may not accurately reflect how its policies affect tribal entities. By collecting data on the extent that tribal entities are obtaining and accessing spectrum, FCC could better understand tribal spectrum issues and use this information as it implements ongoing spectrum initiatives. Further, given that the secondary market is one of few ways for tribal entities to access licensed spectrum to provide Internet service, FCC could promote a more robust secondary market by analyzing unused licensed spectrum over tribal lands and using that information to inform FCC's oversight responsibilities. In the November 2018 report, GAO made three recommendations to FCC, including that FCC should collect data on tribal access to spectrum and analyze unused licensed spectrum over tribal lands. FCC agreed with the recommendations and described actions to address them.", "document_type": "gao"}
{"report": "In January 2019, we reported on the initial steps the Army has taken to consolidate all its modernization efforts under one authority. Establishing Army Futures Command is reported to be the most significant institutional change to the Army since it reorganized in 1973 after the Vietnam War. According to Army documentation, the intent of the new command is to provide unity of command, accountability, and modernization at the speed and scale required to prevail in future conflicts. The organization is led by a four-star general like its organizational peers: Army Materiel Command, Training and Doctrine Command, and Forces Command. The Army declared the commencement of operations for the command in July 2018, and has begun to define its organizational structures. Army Futures Command is expected to be fully operational by July 2019, meaning it will have sufficient staff with operational facilities, secure funding, and the ability to execute its assigned mission, roles, and responsibilities. Army Futures Command is headquartered in Austin, Texas. According to Army officials and documentation, the Army chose Austin because of its proximity to science, technology, engineering, and mathematics talent, as well as private sector innovators that officials believe will assist the command in achieving its modernization goals. According to senior Army leadership, the new command headquarters will have around 300 staff in place by July 2019, a workforce that may grow to as many as 500 employees—100 military and 400 civilians. Our analysis of Army’s plans for initial staffing at the Army Futures Command headquarters, based on data from July 1, 2018, found that about one-third of headquarters staff would be involved directly in modernization efforts, such as engineers and operations specialists, and the remaining two-thirds would consist of support staff, including legal counsel and contracting professionals. According to Army Futures Command officials and documentation, the new organization will be organized around three major components: Futures and Concepts Center is responsible for identifying and prioritizing capability and development needs and opportunities. This organization subsumed the Army Capabilities Integration Center on December 7, 2018. The center was formerly part of Army Training and Doctrine Command and is located at Fort Eustis, Virginia. Combat Capabilities Development Command is responsible for conceptualizing and developing solutions for identified needs and opportunities. This organization subsumed the Research, Development and Engineering Command on February 3, 2019 and is located at Aberdeen Proving Ground, Maryland. Combat Systems Directorate is responsible for refining, engineering, and producing new capabilities. This directorate will communicate with the program executive offices and program management offices reporting to the Assistant Secretary of the Army for Acquisition, Logistics and Technology. Combat Systems Directorate is in the process of being established and is located in Austin, Texas. Among other things, the reorganization is intended to establish Army Futures Command to oversee development of Army’s six modernization priorities. The Army’s then-Acting Secretary and the Chief of Staff in an October 3, 2017 memorandum identified these priorities to guide Army modernization: next generation combat vehicle, air and missile defense, and soldier lethality. As we reported in January 2019, to pursue the six priority areas, the Army established eight cross-functional teams. These teams were initially created as a pilot effort to increase the efficiency of requirements and technology development for modernization before the announcement of the new command. They were subsequently moved into Army Futures Command in 2018. These cross-functional teams are located throughout the country in areas of relevance to their mission. The eight cross- functional teams and the priority areas they address are outlined in table 1. These cross-functional teams are intended to: take steps toward achieving the six modernization priorities; leverage expertise from industry and academia; identify ways to use experimentation, prototyping, and demonstrations; and identify opportunities to improve the efficiency of requirements development and the overall defense systems acquisition process. The cross-functional team pilots were structured to help achieve these goals. Each cross-functional team consists of core staff and subject matter experts from across the Army. To facilitate the rapid approval of requirements, each cross-functional team is led by a general officer or a senior civilian official who could communicate directly with the highest levels of the Army. The goal of staffing these teams is to ensure that each team had individuals who specialized in acquisition, requirements, science and technology, test and evaluation, resourcing, contracting, cost analysis, sustainment, and military operations. The goal of bringing different experts together is to facilitate collaboration and immediate opportunities for stakeholders to provide input as opposed to the more traditional requirements development process, in which input has typically been provided separately. Officials told us that, while all of these subject matter experts may have provided input on the requirements development process in the past, placing them on a single team offers the promise of streamlining those efforts and could eliminate the need for multiple reviews. Figure 1 below compares the requirements development process under cross-functional teams to how the Army has traditionally developed requirements. In January 2019, we recommended that Army Futures Command incorporate leading practices for effective cross-functional teams. We determined that the documentation that established the cross-functional team pilots fully addressed four of our eight leading practices for effective teams, and at least partially addressed another four. The leading practices and their implementation by the cross-function teams are described in table 2 below. In addition to the practices listed above, the cross-functional team pilots generally applied leading practices for requirements development. One leading practice the teams generally applied was promoting communication between requirements developers, warfighters, and industry representatives. This enables the cross-functional teams to better match developer resources with end-user needs. While applying this practice, the cross-functional team pilots had initial progress in writing requirements documents more efficiently. According to cross-functional team officials, they were able to shorten the requirements development process for several capabilities. However, we found that Army Futures Command does not have a formal plan to identify and share lessons learned from cross-functional team pilots to incorporate or expand application of these leading practices. Doing so would allow Army Futures Command the opportunity to accelerate the progress these teams made and spread the benefits across all of the teams and a wider range of specific military capabilities they are pursuing. We recommended that the Army (1) incorporate cross- functional teams’ experiences in applying leading practices and (2) execute a process for identifying and incorporating lessons learned. The Department of Defense concurred with these recommendations, and stated that Army Futures Command expects to apply leading practices and capture lessons learned by the end of 2019. Our January 2019 report also identified leading practices for mergers and organizational transformations. These leading practices are listed in table 3 below. We found that the Army Futures Command had implemented some of these practices, particularly leadership’s dedication to the new command and the clear statement of its mission. However, we have previously reported that, according to federal internal controls standards, it is important to implement all of these practices in order to establish the organizational structure necessary to enable an entity to plan, execute, control, and assess the organization in achieving its objectives. Establishment of this structure is particularly important for the Army where leadership and its priorities can change frequently. Therefore, we recommended in January 2019 that Army Futures Command fully apply these leading practices. The Department of Defense concurred with the recommendation, and stated that it would start pilot processes in fiscal years 2019 and 2020. In addition to further implementing leading practices, Army Futures Command can reduce risk to meeting its goals by fully assessing the workforce necessary to develop requirements—the testable and measurable characteristics necessary for the design of a proposed system. Historically, the Army has been unable to ensure that requirements for new capabilities are feasible due, in part, to a declining workforce for requirements development. In June 2017, we reported that the Army had prioritized combat readiness over resourcing its requirements development process to meet future readiness needs. We recommended that the Army assess the resources, particularly personnel, necessary for requirements development. The Army concurred with the recommendation, and has stated it would implement this recommendation once Army Futures Command is fully operational. As Army Futures Command centralizes and takes responsibility for requirements development, this recommendation is even more pertinent. Therefore, we recently elevated the status of the recommendation to a priority recommendation for the Secretary of the Army, as we believe it warrants greater attention from the Department of the Army. As Army Futures Command approaches full operating status, it is important to define not only how the command functions, but how it works with other organizations. In our January 2019 report, we found that Army Futures Command had not yet established policies and procedures detailing how it will execute its responsibilities in coordination with other Army organizations that do not directly report to it. One such organization is the Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology—the civilian authority responsible for the overall supervision of Army acquisition matters—and the acquisition offices it oversees. To mitigate concerns about coordination, the Army issued a directive in August 2018, signed by the Secretary of the Army, designating the military deputy to the Assistant Secretary as an advisor to Army Futures Command, and Army Futures Command officials have stated that the Assistant Secretary will retain full acquisition authorities as required by law. The command expects to continue to refine its coordination with the Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology. Since announcing the modernization efforts in 2017, the Army has directed more funding toward closing near-term capability gaps, focused on fiscal years 2019 through 2023. For example, as part of the planning for the fiscal year 2019 budget process, the Army identified 67 high- priority programs, such as the M-1 Abrams tank and the AH-64 Apache helicopter, which require further investment. To support these priorities, the Army identified a need for $16 billion in increased funding in fiscal years 2019 through 2023. The 2018 Army Modernization Strategy report identified the need for additional resources for near-term efforts, including plans to spend billions of dollars for acquisition of maneuverable short range air defense capabilities in fiscal years 2020 through 2024. In addition to the near-term capabilities the Army is pursuing, it has identified a number of long-term needs—those focused after fiscal year 2024—and begun to align research and development efforts with these needs. The Army identified long-term capabilities for all of the modernization priorities, as well as dates that science and technology efforts should transition to programs of record. As part of this overall effort, the Army has evaluated its science and technology portfolio to realign funding toward its six modernization priorities. In an October 2017 Army review, the eight cross-functional teams examined science and technology investments to identify which efforts contributed to the priorities and which did not. The review was performed for the Office of the Deputy Under Secretary of the Army. Based on that work, as of our January 2019 report, the Army had taken steps to realign over $1 billion from previous priorities and toward the new priorities for fiscal years 2019 through 2023. Army officials stated that they expect to undertake similar reviews annually. The Army is executing near-term modernization programs, but could better manage how it evaluates them and estimate their costs. In September 2018, we reported that the Army used its six priority capabilities to identify key mission areas—such as long-range artillery, air and missile defense, brigade combat teams, and cyber and electronic warfare—that require near-term modernization investments. Based on its assessments, the Army prioritized and proposed several near-term solutions to address its critical capability gaps. These solutions included adding personnel—and different types of personnel—to combat forces, updating existing weapon systems, and investments in research and development. However, the Army had not established processes for evaluating whether its modernization efforts allow it to deter or defeat potential adversaries during a major conflict. We also found that the Army had not fully estimated the costs or sources of funding for its near-term modernization efforts. In particular, we found that the Army did not report in its modernization strategy the extent to which it relied on Overseas Contingency Operations appropriations. We recommended that the Army (1) develop a plan to finalize the processes for evaluating how its near-term investments contribute to the Army’s ability to decisively defeat a major adversary, and (2) finalize its cost analysis of near-term investments and report those costs to Congress in its fiscal year 2020 budget request. Army officials told us in April 2019 that the Army has taken steps to implement these recommendations. The most recent efforts to modernize follow several past efforts. Unfortunately, the Army has a history of failed, costly weapon system procurements to replace older weapons systems. These failures are due, in part, to requirements that could not be met and the immaturity of key technologies. Many of these programs failed to provide any capability to the warfighter despite the considerable time and funding expended. Some examples of these cancelled programs are listed in table 4 below. While the Army has dedicated significant funding towards its long-term modernization priorities, other changes may also be needed. Among them, we recommended in our January 2019 report, that Army Futures Command take steps to follow our leading practices to mature technology to a sufficiently high level prior to system development, which can reduce risk. There are indications that, in some cases, the Army plans to mature technology to a sufficiently high level prior to system development. For example, officials from the Future Vertical Lift cross-functional team told us they will complete technology demonstrations on two competitive prototypes before choosing to develop a design for the Future Attack Reconnaissance Aircraft. However, we found that the Army may continue its past practice of proceeding into system development with less mature technologies. In particular, we identified some plans to mature technologies in a relevant environment prior to authorizing the start of a new acquisition program, rather than the higher level of demonstrating them in an operational environment as recommended by our leading practices. This increases risk that new capabilities will require further maturation in system development, which could raise costs and extend timelines for delivery of equipment to the warfighter. We recommended in our January 2019 report that the Army should demonstrate technologies in an operational environment before starting a formal acquisition program. The Department of Defense concurred with the recommendation and stated that the Army Futures Command will execute a new development process that will include operational technology demonstrations. Pilot processes for this are expected to begin in 2019. In summary, we recognize that the Army is early in its modernization efforts but could make changes now that would be helpful. Army Futures Command should implement not only the leading practices we describe as well as the lessons learned by its own cross-functional teams. The Army should also increase the transparency of its efforts by clarifying how it evaluates its progress towards modernization goals and clearly stating the full costs of pursuing those goals. Finally, the Army can reduce the risk to the long-term modernization of its capabilities by ensuring that the technologies it uses in future weapon systems are fully mature. Chairman Norcross, Ranking Member Hartzler, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to answer any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Jon Ludwigson, Acting Director, Contracting and National Security Acquisitions at (202) 512-4841, or ludwigsonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are J. Kristopher Keener (Assistant Director), Joe E. Hunter (Analyst-in-Charge), Emily Bond, Matthew T. Crosby, Cale Jones, Kevin O’Neill, John Pendleton, and Roxanna Sun. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Army is investing in near- and long-term modernization efforts to maintain its technological edge over potential adversaries. It is doing this by upgrading and updating current weapon systems, developing new capabilities, and reshaping its doctrine, force structure, training, and leader development. This testimony is based on prior GAO work conducted 2016 through 2019 and addresses the Army's progress in: (1) establishing Army Futures Command, and (2) developing its near-term and long-term modernization strategies. It also highlights several actions recommended in prior reports related to Army modernization. To conduct this work, GAO assessed the Army's near- and long-term modernization efforts, application of leading practices to those efforts, budget documents, and the effectiveness of the process for developing requirements for major weapon systems. This statement includes updates to this information, as of April 2019. In January 2019, GAO reported on initial steps the Army has taken to consolidate its modernization efforts under one authority—Army Futures Command. Army officials call it their most significant institutional change since 1973, when the Army was reorganized after the Vietnam War. As a precursor to this new command, the Army established eight cross-functional teams as a pilot program to increase the efficiency of requirements and technology development in six key modernization areas. These areas are described in the table below. Since announcing the modernization efforts in 2017, the Army has directed more funding toward closing near-term capability gaps. For example, as part of the planning for the fiscal year 2019 budget process, the Army identified 67 high-priority programs that require a $16 billion investment between now and fiscal year 2023. In addition to the near-term capabilities the Army is pursuing, it has identified a number of long-term needs—those focused after fiscal year 2024—and taken steps to realign research and development efforts and funding with those needs. Over the past 2 years, GAO highlighted several steps Army should take to improve its modernization efforts, including: Apply leading practices to Army Futures Command's cross-functional teams, and capture their lessons learned. Assess the resources, particularly personnel, necessary to support its requirements development process. Increase the transparency of its efforts by clarifying how it evaluates whether its modernization efforts are achieving the Army's goals and clearly stating the full costs of pursuing those goals. Reduce risk by ensuring technologies are fully mature—such as demonstrating technologies in an operational environment before starting a formal acquisition program. By implementing these recommendations, Army Futures Command could better ensure its ability to deliver enhanced capabilities to the warfighter and decrease the risk of cost and schedule growth. Over the past 2 years, GAO has made recommendations related to this body of work. Department of Defense and Army concurred with all the recommendations and are working to implement them.", "document_type": "gao"}
{"report": "DOD is the largest U.S. government department and one of the most complex organizations in the world. In support of its military operations, the department manages many interdependent business functions, including logistics management, procurement, health care management, and financial management. DOD relies extensively on IT to support its business functions. The department’s IT budget organizes investments in four categories, called mission areas—enterprise information environment, business, warfighting, and intelligence. Figure 1 shows the amount of DOD’s total requested fiscal year 2020 IT budget (of $36.1 billion) that the department plans to spend on each of its mission areas (including the approximately $8.9 billion it plans to spend on developing, modernizing, operating, and maintaining its business systems). The department further organizes its IT budget by segments. In this regard, the business mission area segments are logistics/supply chain management, human resource management, health, financial management, acquisition, real property management, training and readiness, other business services, and defense security enterprise. Figure 2 shows the department’s projected fiscal year 2020 spending (of $8.9 billion) for each segment in the business mission area. As amended in 2016, the U.S. Code requires DOD to perform certain activities aimed at ensuring that its business system investments are managed efficiently and effectively. Specifically, the amendments established four sets of requirements for the department related to (1) issuing policy and guidance for managing defense business systems; (2) developing and maintaining a defense business enterprise architecture; (3) establishing a Defense Business Council to provide advice to the Secretary on managing defense business systems; and (4) obtaining approvals before systems proceed into development (or if no development is required, into production or fielding). Further, the amendments to the code established specific designations and thresholds that, among other things, provided additional details about the department’s requirements: Covered defense business systems. The code defines a covered defense business system as a system that is expected to have a total amount of budget authority of over $50 million over a period of 5 years or more. Priority defense business systems. The code establishes a category of system, called a priority defense business system. This refers to a system that is (1) expected to have a total amount of budget authority of over $250 million over the period of the current future-years defense program, or (2) designated by the DOD chief management officer (CMO) as a priority defense business system based on specific program analyses of factors including complexity, scope, and technical risk, and after notification to Congress of such designation. Thresholds and officials responsible for review and certification of defense business systems. The code states that, unless otherwise assigned by the Secretary of Defense, military department CMOs are to have approval authority for their covered defense business system investments of below $250 million over the future- years defense program. The CMO is to have approval authority for defense business systems owned by DOD components other than the military departments, systems that will support the business process of more than one military department or other component, and priority defense business systems. Certification requirements. The code requires that a defense business system program be reviewed and certified at least annually, on the basis of its compliance with the business enterprise architecture and appropriate business process reengineering. In addition, the code requires that the business system program be reviewed and certified on the basis of having valid, achievable requirements and a viable plan for implementing the requirements; having an acquisition strategy designed to eliminate or reduce the need to tailor commercial off-the-shelf systems; and being in compliance with the department’s auditability requirements. DOD Instruction 5000.75: Business System Requirements and Acquisitions assigns roles and responsibilities for managing defense business system investments. Table 1 identifies the key entities and their responsibilities for managing defense business system investments. GAO designated the department’s business systems modernization efforts as high risk in 1995 and has continued to do so in the years since. In addition, since 2005, we have issued 12 reports in which we assessed DOD’s actions to respond to the business system investment management requirements contained in U.S. Code Title 10, Section 2222. These 12 reports, which are listed in appendix I, collectively contained 29 recommendations to help strengthen the department’s management of its business systems. For example, In 2014, we reported that DOD had taken steps to comply with key provisions in the NDAA for Fiscal Year 2005; however, the department continued to face challenges in fully complying with the provisions and modernizing its business systems environment. As a result, we recommended that the department improve its business system certification and approval process. DOD agreed with, and implemented this recommendation. In 2015, we reported that DOD had implemented 5 of the 16 recommendations made by GAO since June 2011 to address each of the overarching provisions for improving business systems management described in the NDAA for Fiscal Year 2005. The department had partially implemented the other 11 recommendations. We also reported that DOD’s business enterprise architecture and process reengineering efforts were not fully achieving the intended outcomes described in statute. Thus, we made a recommendation aimed at ensuring that the department better achieve business process reengineering and enterprise architecture outcomes and benefits. DOD agreed with this recommendation and took steps toward implementing it. In 2018, we reported that DOD had made progress in complying with most legislative provisions for managing its defense business systems, but that additional actions were needed. For example, the NDAA for Fiscal Year 2016 required DOD and the military departments to issue guidance to address five requirements for reviewing and certifying the department’s business systems. We reported that, while DOD had issued guidance addressing all of the requirements, the military departments had shown mixed progress. Accordingly, we recommended that the military departments issue guidance to address certifiying their business systems on the basis of the five requirements. While DOD partially agreed with these recommendations, the military departments implemented them. As of June 2019, the department had implemented 15 of the 29 recommendations contained in the 12 reports. In addition, we closed two of the recommendations as “not implemented” because the actions taken by the department did not sufficiently address the recommendations. The department had not yet taken actions to address the other 12 recommendations. Table 2 identifies the four sets of requirements for strengthening DOD’s management of defense business systems identified in the U.S. Code. In addition, the table identifies a fifth category associated with human capital, which supports the department’s execution of the other four sets of requirements. The table also identifies the 12 GAO recommendations that remained to be implemented as of June 2019. DOD took actions toward addressing GAO’s recommendations related to business system requirements contained in the U.S. Code. In doing so, the department made progress in strengthening the management of its defense business system investments. Specifically, between June 2019 and November 2019, the department demonstrated that it had implemented four of the 12 remaining recommendations aimed at strengthening business systems management. As of November 2019, for example, the department had taken actions to implement a recommendation that helped the department comply with the code’s requirement to establish guidance for effectively managing its defense business system investments. In this regard, the Office of the CMO demonstrated that DOD had implemented our recommendation to improve the department’s policy to require full consideration of sustainability and technological refreshment requirements for its defense business systems investments. Specifically, the department demonstrated that its Instruction 5000.75, DOD Directive 5000.01, and DOD Financial Management Regulation Volume 2B guidance includes policy requiring consideration of sustainability and technological refreshment. The department also demonstrated that its DOD Directive 5000.01 guidance includes policy to ensure that best systems engineering practices are used in the procurement and deployment of commercial systems, modified commercial systems, and defense-unique systems. In addition, with regard to the requirement that DOD ensure that business systems are reviewed and certified in accordance with U.S. Code Title 10, Section 2222, the Army demonstrated that it had implemented our recommendation that the department improve its guidance for certifying defense business systems. Specifically, the Army issued guidance to require that the systems be certified on the basis of (1) having valid, achievable requirements and a viable plan to implement the requirements; (2) having an acquisition strategy designed to eliminate or reduce the need to tailor commercial off-the-shelf systems to meet unique requirements, incorporate unique requirements, or incorporate unique interfaces to the maximum extent practicable; and (3) being in compliance with DOD’s auditability requirements. The Air Force also demonstrated that it had implemented our recommendation that the department improve its guidance for certifying defense business systems. Specifically, the Air Force issued guidance to require that systems be certified on the basis of (1) having an acquisition strategy designed to eliminate or reduce the need to tailor commercial off-the-shelf systems to meet unique requirements, incorporate unique requirements, or incorporate unique interfaces to the maximum extent practicable; and (2) being in compliance with DOD’s auditability requirements. Further, with regard to this requirement, the Office of the CMO demonstrated that it had addressed the recommendation to implement and use business enterprise architecture and business process reengineering compliance assessments more effectively. Specifically, in September 2019, the Office of the CMO demonstrated that it had reviewed business enterprise architecture and business process reengineering compliance assessments and identified in investment decision memorandums which systems had assessments that required action. Even with the actions taken, however, more remained to be done to implement eight other recommendations relating to the code’s requirements that could help strengthen the department’s management of its business systems. Specifically, with regard to the requirement to ensure that business systems are reviewed and certified in accordance with the code, the department had not implemented two of our related recommendations. For example, it had not implemented our recommendation to ensure that portfolio assessments are conducted in key areas identified in GAO’s IT investment management framework, such as current schedule, project reporting, and risks. In addition, with regard to the requirement to develop and maintain a defense business enterprise architecture and IT enterprise architecture, in accordance with relevant laws and Office of Management and Budget policies and guidance, the department had developed a business enterprise architecture. However, it had not implemented our five related recommendations. For example, it had not implemented our recommendation to integrate its business and IT architectures. According to officials in the Office of the CIO, the office plans to finalize the first increment of version 3 of its DOD Information Enterprise Architecture (i.e., IT enterprise architecture) by the end of December 2019, and intends to integrate the business enterprise architecture into the Information Enterprise Architecture as a part of that effort. Further, with regard to human capital, which supports the other requirements, the department had not implemented our related recommendation to develop a skills inventory, needs assessment, gap analysis, and plan to address identified gaps as part of a strategic approach to human capital planning. In commenting on its status in addressing the recommendation in September 2019, the Office of the CMO stated that it intends to publish a Defense Business Operations Management Workforce Plan by December 31, 2019. According to office officials, the plan is to include skills requirements for both Office of the CMO permanent employees and DOD employees detailed to the Office of the CMO in support of the management and reform of defense business operations management. Table 3 summarizes the status of the 12 recommendations, as of November 2019, relative to the requirements established in the code. Further, appendix II provides additional information about the status of the recommendations, as of November 2019. By taking actions to implement four of the 12 remaining recommendations, DOD made important progress in its efforts to comply with the requirements established in the U.S. Code. Nevertheless, taking further actions to implement all of the recommendations is essential to helping the department comply with all of the requirements—and ultimately, to strengthen the management of its defense business system investments and efforts to effectively transform its business operations. DOD provided written comments on a draft of this report, which are reprinted in appendix III. In its comments, the department stated that it appreciated GAO’s recognition of the progress that DOD has made to strengthen business systems management. The department agreed that more needs to be done to strengthen the management of investments in DOD business operations and stated that, in the coming months, it plans to execute a reform agenda intended to strengthen oversight and improve business performance. Further, while this report made no new recommendations, the department concurred with our assessment of the status of seven of the eight open recommendations that DOD has not yet implemented. In addition, the department suggested that we close one recommendation that it perform specific activities as part of a strategic approach to human capital planning for the Office of the CMO, based on the CMO’s submission of a human capital analysis report to Congress in early January 2020. The department subsequently provided us a copy of the CMO’s human capital analysis report on February 7, 2020. We plan to assess the report to determine if it meets the intent of our recommendation. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget, the Secretary of Defense, and other interested parties. This report also is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions on matters discussed in this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. Since 2005, GAO has issued 12 reports in which we assessed DOD’s actions to respond to business systems modernization requirements contained in U.S. Code, Title 10, Section 2222. These requirements specify how DOD is to manage its business system investments. The reports are listed below: Defense Business Systems: DOD Needs to Continue Improving Guidance and Plans for Effectively Managing Investments, GAO-18-130 (Washington, D.C.: April 16, 2018). DOD Business Systems Modernization: Additional Action Needed to Achieve Intended Outcomes, GAO-15-627 (Washington, D.C.: July 16, 2015). Defense Business Systems: Further Refinements Needed to Guide the Investment Management Process, GAO-14-486 (Washington, D.C. May 12, 2014). DOD Business Systems Modernization: Further Actions Needed to Address Challenges and Improve Accountability, GAO-13-557 (Washington, D.C.: May 17, 2013). DOD Business Systems Modernization: Governance Mechanisms for Implementing Management Controls Need to Be Improved, GAO-12-685 (Washington, D.C.: June 1, 2012). Department of Defense: Further Actions Needed to Institutionalize Key Business System Modernization Management Controls, GAO-11-684 (Washington, D.C.: June 29, 2011). Business Systems Modernization: Scope and Content of DOD’s Congressional Report and Executive Oversight of Investments Need to Improve, GAO-10-663 (Washington, D.C.: May 24, 2010). DOD Business Systems Modernization: Recent Slowdown in Institutionalizing Key Management Controls Needs to Be Addressed, GAO-09-586 (Washington, D.C.: May 18, 2009). DOD Business Systems Modernization: Progress in Establishing Corporate Management Controls Needs to Be Replicated Within Military Departments, GAO-08-705 (Washington, D.C.: May 15, 2008). DOD Business Systems Modernization: Progress Continues to Be Made in Establishing Corporate Management Controls, but Further Steps Are Needed, GAO-07-733 (Washington, D.C.: May 14, 2007). Business Systems Modernization: DOD Continues to Improve Institutional Approach, but Further Steps Needed, GAO-06-658 (Washington, D.C.: May 15, 2006). DOD Business Systems Modernization: Important Progress Made in Establishing Foundational Architecture Products and Investment Management Practices, but Much Work Remains, GAO-06-219 (Washington, D.C.: November 23, 2005). Table 4 summarizes the status of recommendations made to the Department of Defense (DOD), based on GAO’s assessments of DOD’s actions to respond to defense business system investment management requirements contained in U.S. Code Title 10, Section 2222. The recommendations, included in GAO reports issued from 2005 through 2018, were aimed at helping the department strengthen its approach to managing its business system investments. As of June 2019 (when GAO began its current review of DOD’s actions to address the recommendations), the department had not yet addressed 12 of 29 total recommendations. Subsequently, between June 2019 and November 2019, the department implemented four of the recommendations, but did not implement eight other recommendations. In addition to the contact above, individuals who made contributions to this report include Michael Holland (Assistant Director), Tyler Mountjoy (Analyst in Charge), Camille Chaires, Cheryl Dottermusch, William Hutchinson, Monica Perez-Nelson, Priscilla Smith, and Adam Vodraska.", "summary": "DOD spends billions of dollars each year on systems to support its key business areas, such as personnel and logistics. For fiscal year 2020, DOD reported that its business system investments are expected to cost about $8.9 billion. GAO has made many recommendations to DOD aimed at strengthening defense business systems management. Further, U.S. Code Title 10, Section 2222 requires DOD to perform activities aimed at ensuring that these investments are managed efficiently and effectively. The National Defense Authorization Act for Fiscal Year 2016 included a provision for GAO to report on the extent to which DOD is complying with the code's requirements. Accordingly, the objective of this review was to assess the extent to which DOD has taken actions that comply with the code's requirements for ensuring that business system investments are managed efficiently and effectively. To do so, GAO selected 12 recommendations that DOD had not implemented as of June 2019, and assessed the department's subsequent actions on the recommendations (through November 2019) against the requirements in the code. GAO also analyzed DOD's business systems guidance and business enterprise architecture documentation, and interviewed relevant DOD officials. As of November 2019, the Department of Defense (DOD) had taken actions that addressed some, but not all, of the 12 prior GAO recommendations for strengthening defense business systems management. In doing so, the department made progress in complying with related business system investment management requirements contained in U.S. Code Title 10 Section 2222 (the code or U.S. Code). Specifically, as of November 2019, DOD had implemented four of the 12 recommendations (see table). For example, with respect to the requirement associated with investment management guidance, DOD had implemented the recommendation to issue policy requiring full consideration of sustainability and technological refreshment requirements for its business system investments. However, DOD had not yet implemented eight other recommendations relating to the code's requirements. The recommendations that had not been implemented relate to the department's actions to: integrate its business and information technology (IT) architectures, ensure that portfolio assessments are conducted in key areas identified in the GAO Information Technology Investment Management framework. develop a skills inventory, needs assessment, gap analysis, and plan to address identified gaps as part of a strategic approach to human capital planning, among other things. Taking further actions to implement all of the recommendations is essential to helping the department achieve compliance with all of the requirements—and, ultimately, further strengthen the management of its defense business system investments as well as its efforts to effectively transform its business operations. GAO is not making any new recommendations in this report. As of November 2019, DOD had not yet implemented eight of the 12 prior recommendations. GAO will continue to monitor DOD's actions to address the remaining recommendations.", "document_type": "gao"}
{"report": "To the Commissioner of Internal Revenue In our audits of the fiscal years 2019 and 2018 financial statements of the Internal Revenue Service (IRS), we found IRS’s financial statements as of and for the fiscal years ended September 30, 2019, and 2018, are presented fairly, in all material respects, in accordance with U.S. generally accepted accounting principles; although internal controls could be improved, IRS maintained, in all material respects, effective internal control over financial reporting as of September 30, 2019; and no reportable noncompliance for fiscal year 2019 with provisions of applicable laws, regulations, contracts, and grant agreements we tested. The following sections discuss in more detail (1) our report on the financial statements and on internal control over financial reporting, which includes required supplementary information (RSI) and other information included with the financial statements; (2) our report on compliance with laws, regulations, contracts, and grant agreements; and (3) agency comments. In accordance with our authority conferred by the Chief Financial Officers (CFO) Act of 1990, as amended by the Government Management Reform Act of 1994, we have audited IRS’s financial statements. IRS’s financial statements comprise the balance sheets as of September 30, 2019, and 2018; the related statements of net cost, changes in net position, budgetary resources, and custodial activity for the fiscal years then ended; and the related notes to the financial statements. We also have audited IRS’s internal control over financial reporting as of September 30, 2019, based on criteria established under 31 U.S.C. § 3512(c), (d), commonly known as the Federal Managers’ Financial Integrity Act (FMFIA). We conducted our audits in accordance with U.S. generally accepted government auditing standards. We believe that the audit evidence we obtained is sufficient and appropriate to provide a basis for our audit opinions. IRS management is responsible for (1) the preparation and fair presentation of these financial statements in accordance with U.S. generally accepted accounting principles; (2) preparing, measuring, and presenting the RSI in accordance with U.S. generally accepted accounting principles; (3) preparing and presenting other information included in documents containing the audited financial statements and auditor’s report, and ensuring the consistency of that information with the audited financial statements and the RSI; (4) maintaining effective internal control over financial reporting, including the design, implementation, and maintenance of internal control relevant to the preparation and fair presentation of financial statements that are free from material misstatement, whether due to fraud or error; (5) evaluating the effectiveness of internal control over financial reporting based on the criteria established under FMFIA; and (6) its assessment about the effectiveness of internal control over financial reporting as of September 30, 2019, included in the accompanying Management’s Report on Internal Control over Financial Reporting in appendix I. Our responsibility is to express an opinion on these financial statements and an opinion on IRS’s internal control over financial reporting based on our audits. U.S. generally accepted government auditing standards require that we plan and perform the audits to obtain reasonable assurance about whether the financial statements are free from material misstatement, and whether effective internal control over financial reporting was maintained in all material respects. We are also responsible for applying certain limited procedures to RSI and other information included with the financial statements. An audit of financial statements involves performing procedures to obtain audit evidence about the amounts and disclosures in the financial statements. The procedures selected depend on the auditor’s judgment, including the auditor’s assessment of the risks of material misstatement of the financial statements, whether due to fraud or error. In making those risk assessments, the auditor considers internal control relevant to the entity’s preparation and fair presentation of the financial statements in order to design audit procedures that are appropriate in the circumstances. An audit of financial statements also involves evaluating the appropriateness of the accounting policies used and the reasonableness of significant accounting estimates made by management, as well as evaluating the overall presentation of the financial statements. An audit of internal control over financial reporting involves performing procedures to obtain evidence about whether a material weakness exists. The procedures selected depend on the auditor’s judgment, including the assessment of the risk that a material weakness exists. An audit of internal control over financial reporting also includes obtaining an understanding of internal control over financial reporting, and evaluating and testing the design and operating effectiveness of internal control over financial reporting based on the assessed risk. Our audit of internal control also considered IRS’s process for evaluating and reporting on internal control over financial reporting based on criteria established under FMFIA. Our audits also included performing such other procedures as we considered necessary in the circumstances. We did not evaluate all internal controls relevant to operating objectives as broadly established under FMFIA, such as those controls relevant to preparing performance information and ensuring efficient operations. We limited our internal control testing to testing controls over financial reporting. Our internal control testing was for the purpose of expressing an opinion on whether effective internal control over financial reporting was maintained, in all material respects. Consequently, our audit may not identify all deficiencies in internal control over financial reporting that are less severe than a material weakness. An entity’s internal control over financial reporting is a process effected by those charged with governance, management, and other personnel, the objectives of which are to provide reasonable assurance that (1) transactions are properly recorded, processed, and summarized to permit the preparation of financial statements in accordance with U.S. generally accepted accounting principles, and assets are safeguarded against loss from unauthorized acquisition, use, or disposition, and (2) transactions are executed in accordance with provisions of applicable laws, including those governing the use of budget authority, regulations, contracts, and grant agreements, noncompliance with which could have a material effect on the financial statements. Because of its inherent limitations, internal control over financial reporting may not prevent, or detect and correct, misstatements due to fraud or error. We also caution that projecting any evaluation of effectiveness to future periods is subject to the risk that controls may become inadequate because of changes in conditions, or that the degree of compliance with the policies or procedures may deteriorate. In our opinion, IRS’s financial statements present fairly, in all material respects, IRS’s financial position as of September 30, 2019, and 2018, and its net cost of operations, changes in net position, budgetary resources, and custodial activity for the fiscal years then ended in accordance with U.S. generally accepted accounting principles. In accordance with federal accounting standards, IRS’s financial statements do not include an estimate of the dollar amount of taxes that are owed to the federal government but that taxpayers have not reported or that IRS has not identified through its enforcement programs, often referred to as the tax gap, nor do they include information on tax expenditures. Further detail on the tax gap and tax expenditures, as well as the associated dollar amounts, is provided in the other information included with the financial statements. In our opinion, although internal controls could be improved, IRS maintained, in all material respects, effective internal control over financial reporting as of September 30, 2019, based on criteria established under FMFIA. Our fiscal year 2019 audit identified continuing deficiencies concerning IRS’s internal control over unpaid assessments and continuing and new deficiencies concerning IRS’s internal control over financial reporting systems. While not considered material weaknesses, these deficiencies are collectively important enough to merit attention by those charged with governance of IRS. Therefore, we considered these issues affecting IRS’s internal controls over unpaid assessments and financial reporting systems to be significant deficiencies in internal control as of September 30, 2019. These two significant deficiencies are discussed in more detail below. We considered these significant deficiencies in determining the nature, timing, and extent of our audit procedures on IRS’s fiscal year 2019 financial statements. Although the significant deficiencies in internal control did not affect our opinion on IRS’s fiscal year 2019 financial statements, misstatements may occur in unaudited financial information reported internally and externally by IRS because of these significant deficiencies. In addition, because of the significant deficiencies in internal controls over unpaid assessments and financial reporting systems that existed during fiscal year 2019, IRS’s financial management systems did not comply substantially with federal financial management systems requirements as required by the Federal Financial Management Improvement Act of 1996. We will be reporting additional details concerning any new issues relating to these significant deficiencies separately to IRS management, along with recommendations for corrective actions. We also identified other deficiencies in IRS’s internal control over financial reporting that we do not consider to be material weaknesses or significant deficiencies. Nonetheless, these deficiencies warrant IRS management’s attention. We have communicated these matters to IRS management and, where appropriate, will report on them separately along with related recommendations for corrective actions. Further, as we have reported in past audits, IRS continues to face significant ongoing financial management challenges relating to safeguarding taxpayer receipts and associated information, and preventing and detecting fraudulent refunds based on identify theft. Although these challenges do not rise to the level of significant deficiencies in internal control, we believe they are sensitive matters requiring IRS management’s attention. We have made several recommendations to IRS to enhance its internal controls to mitigate these challenges. It is important that IRS continue its efforts to minimize the risks these challenges pose to taxpayers and any associated losses to the federal government. Limitations in the financial systems IRS uses to account for federal taxes receivable and other unpaid assessment balances, as well as other control deficiencies that led to errors in taxpayer accounts, continued to exist during fiscal year 2019. As a result of these deficiencies, IRS’s systems were unable to provide the timely, reliable, and complete transaction-level financial information necessary to enable IRS to appropriately classify and report unpaid assessment balances. As in prior years, IRS used a complex and labor-intensive statistical estimation process to compensate for the effects of its system limitations and other deficiencies on a material portion of its federal taxes receivable balance to help ensure that this balance was free of material misstatement. During fiscal year 2019, IRS recorded adjustments totaling about $17 billion to correct the effects of continued errors in its underlying data that IRS identified during its manual estimation process. While using this process to determine a material portion of taxes receivable has enabled IRS to produce reliable related balances for year- end reporting, it does not provide IRS management with readily available, reliable unpaid assessment information on a daily basis throughout the year in order to effectively manage unpaid assessment balances. Further, errors in taxpayer accounts create a burden for those taxpayers whose accounts were affected. While not collectively considered a material weakness, IRS’s ongoing control deficiencies related to unpaid assessments are important enough to merit attention by those charged with governance of IRS. Therefore, these issues represent a significant deficiency in IRS’s internal control over financial reporting as of September 30, 2019. During fiscal year 2019, IRS documented the key management decisions in the design and use of the estimation process. This step should reduce the risk that IRS may perform sampling procedures inconsistent with management intent or plans. Continued management commitment and sustained efforts are necessary to build on the progress made to date and to fully address IRS’s remaining unresolved issues concerning the management and reporting of unpaid assessments. During our fiscal year 2019 audit, we determined that unresolved information system control deficiencies from prior audits, along with new control deficiencies pertaining to business process application controls and general controls in IRS’s information systems, collectively represent a significant deficiency in IRS’s internal control over financial reporting systems. Specifically, IRS did not correct control deficiencies we reported as of September 30, 2018, concerning (1) unnecessary access rights granted to accounts, (2) inconsistent monitoring of systems and accounts, (3) out-of-date and unsupported hardware and software, (4) change controls over tax and financial management processing on the mainframe, and (5) developing and implementing effective policies and procedures as part of IRS’s security management program. In addition, during this year’s audit, we found new control deficiencies in the following areas: (1) implementing automated financial controls of interfaces between key applications, (2) ensuring that authorized personnel reviewed key documents for external systems, (3) enforcing multifactor authentication, (4) enforcing adequate encryption to protect systems and data, or (5) ensuring that patches installed on systems were current to protect against known vulnerabilities. The potential effect of these continuing and new deficiencies on IRS’s financial reporting for fiscal year 2019 was mitigated primarily by IRS’s compensating management controls designed to detect potential misstatements on the financial statements. Nevertheless, these application and general control deficiencies increase the risk of unauthorized access to, modification of, or disclosure of sensitive financial and taxpayer data and disruption of critical operations, and are therefore important enough to merit the attention of those charged with governance of IRS. According to IRS management, IRS has developed a plan that focuses on strengthening its information system controls. Continued and consistent management commitment and attention will be essential to addressing existing financial reporting system deficiencies. U.S. generally accepted accounting principles issued by the Federal Accounting Standards Advisory Board (FASAB) require that the RSI be presented to supplement the financial statements. Although the RSI is not a part of the financial statements, FASAB considers this information to be an essential part of financial reporting for placing the financial statements in appropriate operational, economic, or historical context. We have applied certain limited procedures to the RSI in accordance with U.S. generally accepted government auditing standards, which consisted of inquiries of management about the methods of preparing the RSI and comparing the information for consistency with management’s responses to the auditor’s inquiries, the financial statements, and other knowledge we obtained during the audit of the financial statements, in order to report omissions or material departures from FASAB guidelines, if any, identified by these limited procedures. We did not audit and we do not express an opinion or provide any assurance on the RSI because the limited procedures we applied do not provide sufficient evidence to express an opinion or provide any assurance. IRS’s other information contains a wide range of information, some of which is not directly related to the financial statements. This information is presented for purposes of additional analysis and is not a required part of the financial statements or the RSI. We read the other information included with the financial statements in order to identify material inconsistencies, if any, with the audited financial statements. Our audit was conducted for the purpose of forming an opinion on IRS’s financial statements. We did not audit and do not express an opinion or provide any assurance on the other information. In connection with our audits of IRS’s financial statements, we tested compliance with selected provisions of applicable laws, regulations, contracts, and grant agreements consistent with our auditor’s responsibility discussed below. We caution that noncompliance may occur and not be detected by these tests. We performed our tests of compliance in accordance with U.S. generally accepted government auditing standards. IRS management is responsible for complying with laws, regulations, contracts, and grant agreements applicable to IRS. Our responsibility is to test compliance with selected provisions of laws, regulations, contracts, and grant agreements applicable to IRS that have a direct effect on the determination of material amounts and disclosures in IRS’s financial statements, and perform certain other limited procedures. Accordingly, we did not test compliance with all laws, regulations, contracts, and grant agreements applicable to IRS. Our tests for compliance with selected provisions of applicable laws, regulations, contracts, and grant agreements disclosed no instances of noncompliance for fiscal year 2019 that would be reportable under U.S. generally accepted government auditing standards. However, the objective of our tests was not to provide an opinion on compliance with laws, regulations, contracts, and grant agreements applicable to IRS. Accordingly, we do not express such an opinion. The purpose of this report is solely to describe the scope of our testing of compliance with selected provisions of applicable laws, regulations, contracts, and grant agreements and the results of that testing, and not to provide an opinion on compliance. This report is an integral part of an audit performed in accordance with U.S. generally accepted government auditing standards in considering compliance. Accordingly, this report on compliance with laws, regulations, contracts, and grant agreements is not suitable for any other purpose. In commenting on a draft of this report, IRS stated that it was pleased to receive an unmodified opinion on its financial statements. IRS also commented on its continued efforts to address its financial reporting systems control deficiencies and improve its internal controls in financial reporting of unpaid assessments. The complete text of IRS’s response is reproduced in appendix II.", "summary": "In accordance with the authority conferred by the Chief Financial Officers Act of 1990, as amended, GAO annually audits IRS's financial statements to determine whether (1) the financial statements are fairly presented and (2) IRS management maintained effective internal control over financial reporting. GAO also tests IRS's compliance with selected provisions of applicable laws, regulations, contracts, and grant agreements. IRS's tax collection activities are significant to overall federal receipts, and the effectiveness of its financial management is of substantial interest to Congress and the nation's taxpayers. In GAO's opinion, the Internal Revenue Service's (IRS) fiscal years 2019 and 2018 financial statements are fairly presented in all material respects, and although controls could be improved, IRS maintained, in all material respects, effective internal control over financial reporting as of September 30, 2019. GAO's tests of IRS's compliance with selected provisions of applicable laws, regulations, contracts, and grant agreements detected no reportable instances of noncompliance in fiscal year 2019. Limitations in the financial systems IRS uses to account for federal taxes receivable and other unpaid assessment balances, as well as other control deficiencies that led to errors in taxpayer accounts, continued to exist during fiscal year 2019.These control deficiencies affect IRS's ability to produce reliable financial statements without using significant compensating procedures. In addition, unresolved information system control deficiencies from prior audits, along with application and general control deficiencies that GAO identified in IRS's information systems in fiscal year 2019, placed IRS systems and financial and taxpayer data at risk of inappropriate and undetected use, modification, or disclosure. IRS continues to take steps to improve internal controls in these areas. However, the remaining deficiencies are significant enough to merit the attention of those charged with governance of IRS and therefore represent continuing significant deficiencies in internal control over financial reporting related to (1) unpaid assessments and (2) financial reporting systems. Continued management attention is essential to fully addressing these significant deficiencies. Based on prior financial statement audits, GAO made numerous recommendations to IRS to address internal control deficiencies. GAO will continue to monitor and will report separately on IRS's progress in implementing prior recommendations that remain open. Consistent with past practice, GAO will also be separately reporting on the new internal control deficiencies identified in this year's audit and providing IRS recommendations for corrective actions to address them. In commenting on a draft of this report, IRS stated that it continues its efforts to improve its financial reporting systems controls and internal controls over unpaid assessments.", "document_type": "gao"}
{"report": "Our nation’s critical infrastructure refers to the systems and assets, whether physical or virtual, so vital to the United States that the incapacity or destruction of them would have a debilitating impact on our security, economic stability, public health or safety, or any combination of these factors. Critical infrastructure includes, among other things, banking and financial institutions, telecommunications networks, and energy production and transmission facilities, most of which are owned and operated by the private sector. Threats to the systems supporting our nation’s critical infrastructures are evolving and growing. These systems are susceptible to unintentional and intentional threats, both cyber and physical. Unintentional, or nonadversarial, threat sources include equipment failures, software coding errors, or the accidental actions of employees. They also include natural disasters and the failure of other critical infrastructures, since the sectors are often interdependent. Intentional or adversarial threats can involve targeted and untargeted attacks from a variety of sources, including criminal groups, hackers, and disgruntled employees. Adversaries can leverage common computer software programs to deliver a threat by embedding exploits within software files that can be activated when a user opens a file within its corresponding program. Due to the cyber-based threats to federal systems and critical infrastructure, the persistent nature of information security vulnerabilities, and the associated risks, GAO first designated federal information security as a government-wide high-risk area in our biennial report to Congress in 1997. In 2003, we expanded this high-risk area to include the protection of critical cyber infrastructure and, in 2015, we further expanded this area to include protecting the privacy of personally identifiable information. We continue to identify the protection of critical cyber infrastructure as a high-risk area, as shown in our March 2019 high- risk update. Because the private sector owns the majority of the nation’s critical infrastructure, it is vital that the public and private sectors work together to protect these assets and systems. Toward this end, federal law and policy assign roles and responsibilities for agencies to assist the private sector in protecting critical infrastructure, including enhancing cybersecurity. Presidential Policy Directive 21 establishes the SSAs in the public sector as the federal entities responsible for providing institutional knowledge and specialized expertise. The SSAs lead, facilitate, and support the security and resilience programs and associated activities of their designated critical infrastructure sectors. The directive identified 16 critical infrastructure sectors and designated the nine associated SSAs, as shown in figure 1. In addition, the directive required DHS to update the National Infrastructure Protection Plan to address the implementation of the directive. The directive called for the plan to include, among other things, the identification of a risk management framework to be used to strengthen the security and resilience of critical infrastructure and a metrics and analysis process to be used to measure the nation’s ability to manage and reduce risks to critical infrastructure. DHS, in response, updated the National Infrastructure Protection Plan in December 2013 in collaboration with public- and private-sector owners and operators and federal and nonfederal government representatives, including SSAs, from the critical infrastructure community. According to the 2013 plan, SSAs are to work with their private-sector counterparts to understand cyber risk and they are to develop and use metrics to evaluate the effectiveness of risk management efforts. To work with the government, the SCCs were formed as self-organized, self-governing councils that enable critical infrastructure owners and operators, their trade associations, and other industry representatives to interact on a wide range of sector-specific strategies, policies, and activities. The SSAs and the SCCs coordinate and collaborate in a voluntary fashion on issues pertaining to their respective critical infrastructure sector. In addition to the directive, federal laws and policies have also established roles and responsibilities for federal agencies to work with industry to enhance the cybersecurity of the nation’s critical infrastructures. These include the Cybersecurity Enhancement Act of 2014 and Executive Order 13636. In February 2013, Executive Order 13636 outlined an action plan for improving critical infrastructure cybersecurity. Among other things, the executive order directed NIST to lead the development of a flexible performance-based cybersecurity framework that was to include a set of standards, procedures, and processes. The executive order also directed SSAs, in consultation with DHS and other interested agencies, to coordinate with the SCCs to review the cybersecurity framework and, if necessary, develop implementation guidance or supplemental materials to address sector-specific risks and operating environments. Further, in December 2014, the Cybersecurity Enhancement Act of 2014 established requirements that are consistent with the executive order regarding NIST’s development of a cybersecurity framework. According to this law, NIST’s responsibilities in supporting the ongoing development of the cybersecurity framework included, among other things, identifying an approach that is flexible, repeatable, performance-based, and cost- effective. Additionally, the Cybersecurity Act requires NIST to coordinate with federal and nonfederal entities (e.g., SSAs, SCCs, and ISACs) to identify a prioritized, performance-based approach to include information security measures to help entities assess risk. In May 2017, Executive Order 13800 directed federal agency heads to use the framework to manage cybersecurity risks. The executive order also required them to provide a risk management report to DHS and the Office of Management and Budget within 90 days of the date of the executive order. The risk management report calls for agencies to document the risk mitigation and acceptance choices including, for example, describing the agency’s action plan to implement the framework. In response to Executive Order 13636, NIST published, in February 2014, the Framework for Improving Critical Infrastructure Cybersecurity, a voluntary framework of cybersecurity standards and procedures for industry to adopt. According to NIST, as of February 2019, the framework had been downloaded more than a half million times since its initial publication in 2014. Additionally, it has been translated into Arabic, Japanese, Portuguese, and Spanish, and has been adopted by many foreign governments. The framework is composed of three main components: the framework core, the implementation tiers, and the profiles. The framework core provides a set of activities to achieve specific cybersecurity outcomes and references examples of guidance to achieve those outcomes. Through the use of the profile, the framework is intended to help organizations align their cybersecurity activities with business requirements, risk tolerances, and resources. The framework core is divided into four elements: functions, categories, subcategories, and informative references. Functions consist of five elements—(1) identify, (2) protect, (3) detect, (4) respond, and (5) recover. When considered together, these functions provide a strategic view of the life cycle of an organization’s management of cybersecurity risk. Categories are the subdivisions of a function into groups of cybersecurity outcomes tied to programmatic needs and particular activities (i.e. asset management). Subcategories further divide a category into specific outcomes of technical and/or management activities (i.e. notifications from detection systems are investigated). Lastly, informative references are specific sections of standards, guidelines, and practices that illustrate a method to achieve the outcomes described and support one or more informative references (i.e. NIST Special Publication (SP) 800-53A). Implementation tiers characterize an organization’s approach to managing cybersecurity risks over a range of four tiers. The four tiers are partial, risk informed, repeatable, and adaptive. They reflect a progression from informal, reactive responses to approaches that are flexible and risk- informed. Profiles enable organizations to establish a road map for reducing cybersecurity risks that is well aligned with organizational and sector goals, consider legal/regulatory requirements and industry best practices, and reflect risk management priorities. Organizations can use the framework profiles to describe the current state (the cybersecurity outcomes that are currently being achieved) or the desired target state (the outcomes needed to achieve the desired cybersecurity risk management goals) of specific cybersecurity activities. In December 2015, we issued our first report on the development and promotion of the framework in response to the 2014 Cybersecurity Act. We reported that the framework met the requirements established in federal law that it be flexible, repeatable, performance-based, and cost- effective. We also reported that SSAs and NIST had promoted and supported adoption of the cybersecurity framework in the critical infrastructure sectors. For example, we reported that DHS had established the Critical Infrastructure Cyber Community Voluntary Program to encourage adoption of the framework and had undertaken multiple efforts as part of this program. These efforts included developing guidance and tools intended to help sector entities that use the framework. However, we noted that DHS had not developed metrics to measure the success of its activities and programs. Accordingly, we concluded that DHS could not determine if its efforts were effective in encouraging adoption of the framework. We recommended that the department develop metrics to assess the effectiveness of its framework promotion efforts. DHS agreed with the recommendation and subsequently took actions to implement it. We also reported in December 2015 that SSAs had promoted the framework in their sectors by, for example, presenting the framework at meetings of sector stakeholders and holding other promotional events. In addition, all of the SSAs, except for DHS and the General Services Administration (GSA), as co-SSAs for the government facilities sector, made decisions, as required by Executive Order 13636, on whether to develop tailored framework implementation guidance for their sectors. However, we noted that DHS and GSA had not set a time frame to determine, as required by Executive Order 13636, whether sector-specific implementation guidance was needed for the government facilities sector. We concluded that, by not doing so, DHS and GSA could be hindering the adoption of the framework in this sector. As a result, we recommended that DHS and GSA set a time frame to determine whether implementation guidance was needed for the government facilities sector. Both DHS and GSA agreed with our recommendations and subsequently took actions to implement them. More recently, in February 2018, we issued our second report on the adoption of the framework. We reported that most of the 16 critical infrastructure sectors had taken action to facilitate adoption of the framework by entities within their sectors. We also reported that 12 of the 16 critical infrastructure sectors had taken actions to review the framework and, if necessary, develop implementation guidance or supplemental materials that addressed how entities within their respective sectors can adopt the framework. We also reported that none of the SSAs had measured the cybersecurity framework’s implementation by entities within their 16 respective sectors. We noted that the nation’s plan for national critical infrastructure protection efforts stated that federal and nonfederal sector partners (including SSAs) were to measure the effectiveness of risk management goals by identifying high-level outcomes and progress made toward national goals and priorities, including securing critical infrastructure against cyber threats. However, we reported that none of the 16 coordinating councils reported having qualitative or quantitative measures of framework adoption because they generally did not collect specific information from entities about critical infrastructure protection activities. As of November 2019, most of the SSAs had not developed methods to determine their level and type of cybersecurity framework adoption, as we previously recommended. The SSAs and SCCs identified a number of impediments to developing a comprehensive understanding of the use of the framework, including the voluntary nature of the framework. However, most SSAs have taken steps to encourage and facilitate use of the framework. Further, the 12 selected organizations we interviewed reported either fully or partially using the cybersecurity framework. Best practices identified in the National Infrastructure Protection Plan recommend that entities, such as SSAs and SCCs, take steps to evaluate progress toward achieving their goals—in this case, to implement or adopt the cybersecurity framework. As we previously reported, until the SSAs had a more comprehensive understanding of the use of the cybersecurity framework by entities within the critical infrastructure sectors, they would be limited in their ability to understand the success of protection efforts or to determine where to focus limited resources for cyber risk mitigation. As a result, we recommended that the SSAs take steps to consult with respective sector partner(s), such as the SCCs, DHS, and NIST, as appropriate, to develop methods for determining the level and type of framework adoption by the entities across their respective sectors. However, as of November 2019, most of the SSAs had not developed methods to determine the level and type of framework adoption. Specifically, only two of the nine SSAs—the Department of Defense (DOD) in collaboration with the defense industrial base sector and GSA in conjunction with DHS’s Federal Protective Service—had methods to determine the level and type of framework adoption across their respective sectors. DOD, in coordination with the defense industrial base sector, had developed a process to monitor the level or extent to which all contracts (not including commercial off-the-shelf contracts) were or were not adhering to the cybersecurity requirements in DOD acquisition regulations. The regulations called for organizations to implement the security requirements in NIST SP 800-171, which is mapped to the functional areas of the cybersecurity framework. By doing so, DOD is able to determine the level at which the sector organizations are implementing the framework and the type of framework adoption through mapping to the functional areas. Additionally, the federal departments and agencies that form the government facilities sector had submitted their risk management reports to DHS and OMB that described agencies’ action plans to implement the framework, as required under Executive Order 13800. The risk management assessments are included as part of OMB’s FISMA Annual Report to Congress. As a result, the reports could be used as a resource to inform the level and type of framework adoption. In addition, two other SSAs had begun taking steps to develop methods to determine the level and type of framework adoption in their sectors. Specifically, in October 2019, DHS, in coordination with its information technology (IT) sector partner, administered a survey to all small and midsized IT sector organizations to gather information on, among other things, framework use and plans to report on the results in 2020. Further, officials in the Department of Transportation’s (DOT) Office of Intelligence, Security, and Emergency Response, in coordination with its co-SSA (DHS), told us that they planned to develop and distribute a survey to the transportation systems sector to determine the level and type of framework adoption. DOT officials stated that the draft survey was undergoing DHS legal review and that the completion of the review and subsequent OMB review would determine when the survey is approved for distribution. The remaining five SSAs did not have efforts underway to determine the level and type of framework adoption: Department of Agriculture, Department of Energy, Department of Health and Human Services (HHS), Environmental Protection Agency (EPA), and Department of the Treasury. These SSAs identified impediments to determining framework adoption but also noted steps taken to encourage use of the framework within their respective sector. Department of Agriculture’s Office of Homeland Security officials stated that their sector is diverse and includes over 500 sector members that can range from small farms that are family operated to large corporations that deal with selling food wholesale. The officials noted that the diversity makes it difficult to develop a method for determining the level and type of framework adoption across the sector that would apply to all their members. The framework, however, is adaptive to provide a flexible and risk- based implementation. Accordingly, the framework can be used with a broad array of cybersecurity risk management processes. Agriculture officials added that the SCC frequently invites DHS to semi-annual meetings to present on both the threat to cybersecurity and resources available to support the needs of the sector. Department of Energy’s Office of Cybersecurity, Energy Security, and Emergency Response officials stated that the voluntary nature of the framework made it difficult to determine the level and type of framework adoption. However, the department published the Cybersecurity Capability Maturity Model in May 2012, with the most recent update (version 1.1) published in February 2014. The model focused on the implementation and management of cybersecurity practices, and was intended to be descriptive, rather than prescriptive, guidance that could be used by organizations of various types and sizes to strengthen their cybersecurity capabilities. The model was designed for organizations to use with a self-evaluation methodology and toolkit to measure and improve their cybersecurity programs and serve as an example for how to implement the framework. In February 2020, officials stated that they were in the process of updating the model and will update the framework implementation guidance once the model has been updated. HHS’s Assistant Secretary for Preparedness and Response (ASPR) officials stated that, since the use of the framework by the private sector is voluntary, organizations were free to choose any cybersecurity framework(s) that they believed to be most effective for their particular environment. However, HHS, in collaboration with NIST, DHS, and the Joint Healthcare and Public Health Cybersecurity Working Group, released a cybersecurity publication (Health Industry Cybersecurity Practices: Managing Threats and Protecting Patients) that contained 10 best practices in December 2018 for the healthcare and public health services sector based on the framework. This publication allowed stakeholders to identify how to use the framework with existing sector resources by raising awareness and providing vetted cybersecurity practices to enable the organizations to mitigate cybersecurity threats to the sector. In addition, officials from HHS’s ASPR stated that the working group discussed the challenges associated with measuring the use and impact of the NIST framework, and approved the establishment of a task group in 2020 to further investigate the issue. ASPR officials added that some of the ideas discussed included the use of surveys and identification of a set of voluntary reporting indicators. EPA officials told us that the agency will coordinate with its SCC to identify appropriate means to collect and report information, such as a survey, to determine the level and type of framework adoption. They explained that, in the past, the water sector had expressed concerns with sharing sensitive cybersecurity information and in developing metrics to evaluate cybersecurity practices. However, EPA officials stated that they have conducted training, webcasts, and outreach related to cybersecurity, including using the framework and tailoring its efforts to sector needs. According to EPA officials, the agency’s goal in doing so was to ensure that sector organizations understood the importance of the framework. Department of the Treasury officials noted the size of the financial services sector as an impediment to determine framework adoption. Specifically, officials stated that, because of the large number of members, it is difficult to survey all 800,000 organizations to determine framework adoption. However, officials stated that the department, in coordination with the Financial and Banking Information Infrastructure Committee, and in consultation with NIST, developed the Cybersecurity Lexicon in March 2018. The lexicon addressed, among other things, common terminology for cyber terms used in the framework. Additionally, the financial services sector, in consultation with NIST, created the Financial Services Sector Cybersecurity Profile (profile) in October 2018, which mapped the framework core to existing regulations and guidance, such as the Commodity Futures Trading Commission System Safeguards Testing Requirements. Officials stated that these efforts will facilitate the use of the framework. While the five SSAs have ongoing initiatives, implementing our recommendations to gain a more comprehensive understanding of the framework’s use by critical infrastructure sectors is essential to the success of protection efforts. Executive Order 13636 directs SSAs, in consultation with DHS and other agencies, to review the cybersecurity framework and, if necessary, develop implementation guidance or supplemental materials to address sector-specific risks and facilitate framework use. Most of the SSAs developed guidance to encourage and facilitate use of the framework. Specifically, SSAs for 13 of the 16 sectors had developed implementation guidance that included mapping the existing sector cybersecurity tools, standards, and approaches to the framework. For example, the implementation guidance for the healthcare and public health sector provides instruction on how to align a host of existing voluntary or required standards (such as those promulgated pursuant to the Health Insurance Portability and Accountability Act of 1996), guidelines, and practices to the framework core functions. Table 1 describes the 13 sectors and the associated cybersecurity framework implementation guidance. The Cybersecurity Capability Maturity Model helps organizations evaluate and potentially improve their cybersecurity practices. Appendix A of the Energy Sector Cybersecurity Framework Implementation Guidance provides a mapping of the model to the framework. The Financial Services Sector Cybersecurity Profile was created for financial institutions of all sizes to use for cyber risk management assessment and a mechanism to comply with various regulatory frameworks and the NIST Cybersecurity Framework. The remaining three sectors (government facilities, food and agriculture, and IT) had not developed implementation guidance. In this regard, DHS’s Federal Protective Service officials stated that, in 2015, the co- SSAs of the government facilities sector (DHS and GSA) decided that implementation guidance was not needed based on a consensus within the government facilities sector. DHS’s Federal Protective Service officials added that this decision was reevaluated in 2017 and they determined that the guide was still not needed. Department of Agriculture officials from the Office of Homeland Security stated that the co-SSAs (Agriculture and HHS) and the SCC for the sector collectively decided that a single implementation guidance document was not sufficient for addressing the needs of the diverse membership of the food and agriculture sector and that the creation of such a document was a low priority for the sector. These officials added that, due to the complexity of operations and large number of entities within the sector, the coordinating councils determined that it was more appropriate to refer sector members to DHS's Critical Infrastructure Cyber Community Voluntary Program. DHS officials representing the SSA for the IT sector stated that the SSA and SCC jointly determined that creating formal implementation guidance within the sector was not necessary. They added that the IT sector continued to play an active role by participating in framework development and promotion across the sectors, to include the development of a small and midsize business cybersecurity survey that was issued in 2019. In addition to the above efforts, NIST officials stated that they took steps to encourage framework adoption through three main mechanisms for federal and nonfederal entities and organizations that were interested in the framework: (1) conferences and speaking engagements, (2) requests for information to solicit ways in which organizations are using the framework to improve cybersecurity risk management and how best practices are being shared, and (3) industry and agency events, such as webcasts. The 12 selected organizations reported either fully or partially using the cybersecurity framework. Specifically, six organizations reported fully using the framework, whereas six others reported partially using the framework. For example, one organization that reported fully using the framework stated that the framework core, profiles, and tiers were implemented across all the components or business units in the organization. In contrast, one organization that reported partially using the framework stated that it used the framework profiles, but did not fully use the framework core and tiers. Two other of the organizations that reported partially using the framework stated that they considered themselves to be using the framework since they use International Organization for Standardization (ISO) 27001, an international standard that has elements that overlap with those in the framework. The 12 selected organizations using the framework reported varying levels of improvements. Such improvements included identifying risks and implementing common standards and guidelines. However, the SSAs have not collected and reported sector-wide improvements as a result of framework use. The SSAs, SCCs, ISACs, and the selected organizations identified impediments to collecting and reporting such improvements, including developing precise measurements of improvement, the voluntary nature of the framework, and lack of a centralized information sharing mechanism. NIST and DHS have identified initiatives to help address these impediments. The 12 selected organizations reported varying levels of improvements as a result of using the framework. Specifically, four of the 12 reported great improvement, six reported some improvement, and two reported little improvement. Examples of each category are described below: Great improvement: One organization stated that the framework allowed it to determine the current state (the cybersecurity outcomes that are currently being achieved) and the desired target state (the outcomes needed to achieve the desired cybersecurity risk management goals). The organization stated that identifying the current and target states enabled the organization to identify risks and implement common policies, standards, and guidelines across their organization. Officials of the organization also stated that the common language provided by the framework made it easier to communicate within the organization when discussing budgets for cybersecurity that resulted in budget increases. Some improvement: One organization explained that the framework is accepted across organizations and that modeling its capabilities against the framework provided assurance that it covered the critical aspects of security. However, the organization noted that, if the framework did not exist, it would have used another framework to protect its critical infrastructure and facilitate decision making. Little improvement: One organization noted that it already had a very robust risk management process through the use of international standards before using the framework. As a result, the organization stated that use of the framework resulted in little improvements. Another organization that reported little improvements stated that use of the framework helped the organization, but there were no specific improvements that it could identify in protecting its critical infrastructure as a result of using the framework. NIST Special Publication 800-55 guidance on performance measurement states that agency heads are responsible for actively demonstrating support for developing information security measures and facilitating performance improvements in their information security programs, which is to include a periodic analysis of data to determine lessons learned. Additionally, the National Infrastructure Protection Plan directed SSAs and their federal and nonfederal sector partners (including SCCs) to measure the effectiveness of risk management goals by identifying high- level outcomes to facilitate the evaluation of progress toward national goals and priorities, including securing critical infrastructure from cybersecurity threats. The SSAs are not collecting and reporting on improvements in the protection of critical infrastructure as a result of using the framework across the sectors. The SSAs, SCCs, ISACs, and organizations reported a number of impediments to identifying sector-wide improvements, including developing precise measurements of improvement, the voluntary nature of the framework, difficulty in measuring the direct impact of using the framework, lack of use cases, and lack of a centralized information sharing mechanism. Figure 2 depicts the number of entities and organizations that identified these five impediments, and is followed by a discussion of each challenge. Two SCCs, two ISACs, and two organizations identified the difficulty of having precise measurements of improvements as a result of using the framework. SCC officials from the communications and healthcare and public health sectors stated that authoritative and precise measurements of improvements are difficult to determine in a consistent and non-subjective manner. For example, the SCC officials for the healthcare and public health sector stated that they were not aware of a direct or precise form of sector-wide measurements to define success in mitigating cybersecurity risk using the framework within the sector. These officials added that future efforts could include methodologies to track sector-wide improvements based on the framework structure or other cybersecurity guidance. However, officials from NIST’s Information Technology Laboratory stated that they were in the early stages of initiating an information security measurement program to facilitate identifying improvements sector-wide. Officials stated that the program aims to provide foundation tools and guidance to support the development of information security measures that are aligned with an individual organization’s objectives. The officials stated that they had not established a time frame for the completion of the measurement program. They added that, once the program is developed, the SSAs are expected to be able to customize the program and work with their respective sector organizations to determine sector-wide improvements based on their unique objectives. Eight SSAs, two SCCs, and four organizations stated that the voluntary nature of using the framework made it difficult to identify sector-wide improvements. Officials stated that private sector framework adoption was voluntary and, therefore, there were no specific reporting requirements to provide information on improvements. For example, DOT officials from the Office of Intelligence, Security, and Emergency Response stated that, while the department and its co-SSA (DHS) intended to develop a survey to determine sector-wide improvements, consolidating voluntarily shared information will not reflect the depth and breadth of sector stakeholders, as organizations that share information will not collectively represent a sector. In April 2019, NIST issued the NIST Roadmap for Improving Critical Infrastructure Cybersecurity, version 1.1, which included a self- assessment tool that provided a mechanism for individual organizations to self-assess how effectively they manage cybersecurity risks in the context of broader enterprise risk management activities and identify improvement opportunities. In addition to the road map, NIST’s framework included a section that encouraged organizations to incorporate measurements of their risks, which can be used to identify sector-wide improvements related to using the framework. In addition, as previously mentioned, DHS, in partnership with its IT sector partners, administered a survey to the small and mid-sized IT sector organizations to gather information on, among other things, framework adoption, challenges, and related improvements. While DHS did not plan to report on the results until 2020, the survey was intended to help the department in identifying improvements across the small and mid-sized IT sector organizations. The survey was administered to the small and mid-sized organizations within the IT sector. DHS officials stated that any small or mid-sized business across all critical infrastructure sectors could complete the survey and that the department had promoted the survey to all sectors. Moreover, among all 16 sectors, only DOT and its co-SSA (DHS) had considered the applicability of a similar approach for their sector organizations. Specifically, DOT, in conjunction with DHS, plans to distribute a survey intended to cover framework adoption, challenges, and related improvements across the sector. DOT officials stated that the survey completion is contingent upon DHS’s Transportation Security Administration’s coordination of the review and approval process to meet Paperwork Reduction Act compliance requirements. Three SSAs, four SCCs, one ISAC, and seven organizations stated that identifying sector-wide improvements as a result of using the framework was difficult due to organizations struggling with determining the direct impact from framework use. For example, the Department of Energy officials from the Office of Cybersecurity, Energy Security, and Emergency Response stated that the sector cannot relate improvements to any one framework or model because the sector organizations are engaged in numerous concurrent public and private cybersecurity initiatives, each of which could impact cybersecurity to varying degrees. In addition, EPA officials from the Office of Groundwater and Drinking Water stated that most organizations will not be able to link improvements directly to the framework because EPA does not exclusively incorporate the framework into the agency’s sector guidance. The officials added that existing industry standards and best practices are also recognized in the development of EPA cybersecurity guidance. Therefore, although an organization might experience improvements from using elements of the framework, it might not be readily apparent that those improvements came directly from the framework. To provide the sector organizations with access to various framework resources, NIST updated its website to include sector-specific implementation guidance and case studies, as well as insights from organizations using the framework. Five organizations identified the lack of use cases as an impediment to determining improvements. For example, one organization stated that small and medium organizations struggled with identifying improvements from using the framework because of the lack of use cases (examples for how to determine or measure improvements as a result of using the framework). To address the challenge, the organization stated that it would be helpful if NIST, in collaboration with federal and nonfederal entities, would share and provide use cases or direction on common scenarios small and medium organizations faced and how these could be addressed through the framework. NIST officials stated that they were in the early stages of developing a cybersecurity framework starter profile for small organizations. NIST officials stated that they did not have a time frame for completing the profile. However, they added that the profile will aim to identify common solutions to a specific challenge, such as threat surface or cybersecurity challenges in cloud computing, using a customized adaptation of the framework. In addition, DHS created a small and midsize business road map for all critical infrastructure sectors in 2018. The road map provided a guide for small and mid-sized businesses to use in enhancing their cybersecurity posture. The road map also included DHS’s cybersecurity information sharing and collaboration program and secure information sharing portal. The purpose of the information sharing and collaboration program was to enable actionable, relevant, and timely unclassified information exchange through trusted public- private partnerships across all critical infrastructure sectors. In addition, the secure information sharing portal served as a forum to share cybersecurity strategies and insights with the critical infrastructure sectors. Five organizations identified the lack of a centralized information sharing mechanism as an impediment. For example, one organization stated that there is a challenge in sharing information among all critical infrastructure sectors in a more open and non-judgmental way. To address this challenge, the organization stated that it would be helpful to establish a centralized information sharing mechanism to share and exchange information in an anonymous manner. Another organization added that the challenge with determining improvements is that there is no centralized information sharing mechanism to obtain information. The organization added that it would be helpful to see how organizations compare with one another in terms of goals through this type of mechanism. DHS, however, identified its homeland security information network as a tool that was intended to be the primary system used by entities to collaborate to protect critical infrastructure. Officials in DHS’s Stakeholder Engagement and Cyber Infrastructure Resilience division stated that the information in its homeland security information network could be used by all sectors to report on best practices, including sector-wide improvements and lessons learned from using the framework. Although NIST and DHS have identified initiatives to help address the impediments, the SSAs have not reported on sector-wide improvements. Until they do so, the extent to which the 16 critical infrastructure sectors are better protecting their critical infrastructures from threats will be largely unknown. Most of the SSAs have not determined the level and type of framework adoption, as we previously recommended. Most of the sectors, however, had efforts underway to encourage and facilitate use of the framework. Even with this progress, implementation of our recommendations is essential to the success of protection efforts. While selected organizations reported varying levels of improvements, the SSAs have not collected and reported sector-wide improvements as a result of framework use. The SSAs and organizations identified impediments to collecting and reporting sector-wide improvements, including the lack of precise measurements of improvement, voluntary nature of the framework, and lack of a centralized information sharing mechanism. However, NIST and DHS have initiatives to help address these impediments. These included an information security measurement program, cybersecurity framework starter profile, information sharing programs, self-assessment tools, and surveys to support SSAs in measuring and quantifying improvements in the protection of critical infrastructure as a result of using the framework. However, NIST has yet to establish time frames for completing the information security measurement program and starter profile. Moreover, the SSAs have yet to report on sector-wide improvements using the initiatives. Until they do so, the critical infrastructure sectors may not fully understand the value of the framework to better protect their critical infrastructures from cyber threats. We are making the following 10 recommendations to NIST and the nine sector-specific agencies. The Director of NIST should establish time frames for completing NIST’s initiatives, to include the information security measurement program and the cybersecurity framework starter profile, to enable the identification of sector-wide improvements from using the framework in the protection of critical infrastructure from cyber threats. (Recommendation 1) The Secretary of Agriculture, in coordination with the Secretary of Health and Human Services, should take steps to consult with respective sector partner(s), such as the SCC, DHS, and NIST, as appropriate, to collect and report sector-wide improvements from use of the framework across its critical infrastructure sector using existing initiatives. (Recommendation 2) The Secretary of Defense should take steps to consult with respective sector partner(s), such as the SCC, DHS, and NIST, as appropriate, to collect and report sector-wide improvements from use of the framework across its critical infrastructure sector using existing initiatives. (Recommendation 3) The Secretary of Energy should take steps to consult with respective sector partner(s), such as the SCC, DHS, and NIST, as appropriate, to collect and report sector-wide improvements from use of the framework across its critical infrastructure sector using existing initiatives. (Recommendation 4) The Administrator of the Environmental Protection Agency should take steps to consult with respective sector partner(s), such as the SCC, DHS, and NIST, as appropriate, to collect and report sector-wide improvements from use of the framework across its critical infrastructure sector using existing initiatives. (Recommendation 5) The Administrator of the General Services Administration, in coordination with the Secretary of Homeland Security, should take steps to consult with respective sector partner(s), such as the Coordinating Council and NIST, as appropriate, to collect and report sector-wide improvements from use of the framework across its critical infrastructure sector using existing initiatives. (Recommendation 6) The Secretary of Health and Human Services, in coordination with the Secretary of Agriculture, should take steps to consult with respective sector partner(s), such as the SCC, DHS, and NIST, as appropriate, to collect and report sector-wide improvements from use of the framework across its critical infrastructure sector using existing initiatives. (Recommendation 7) The Secretary of Homeland Security should take steps to consult with respective sector partner(s), such as the SCC and NIST, as appropriate, to collect and report sector-wide improvements from use of the framework across its critical infrastructure sectors using existing initiatives. (Recommendation 8) The Secretary of Transportation, in coordination with the Secretary of Homeland Security, should take steps to consult with respective sector partner(s) such as the SCC and NIST, as appropriate, to collect and report sector-wide improvements from use of the framework across its critical infrastructure sector using existing initiatives. (Recommendation 9) The Secretary of the Treasury should take steps to consult with respective sector partner(s), such as the SCC, DHS, and NIST, as appropriate, to collect and report sector-wide improvements from use of the framework across its critical infrastructure sector using existing initiatives. (Recommendation 10) We received comments on a draft of this report from the ten agencies to which we made recommendations—the Departments of Agriculture, Commerce, Defense, Energy, Health and Human Services, Homeland Security, Transportation, and the Treasury; and the Environmental Protection Agency and the General Services Administration. Among these agencies, eight agreed with the recommendations, one neither agreed nor disagreed with the recommendation, and one partially agreed with the recommendation. In written comments, the Department of Agriculture generally concurred with the recommendation in our report. The department’s comments are reprinted in appendix II. In written comments, the Department of Commerce concurred with the recommendation in our report. The department stated that the National Institute of Standards and Technology expects to document its cybersecurity measurement program scope, objectives, and approach by about June 2020 and publish two cybersecurity starter profiles by about September 2020. The department’s comments are reprinted in appendix III. In written comments, the Department of Defense concurred with the recommendation in our report and described ongoing steps to evaluate defense organizations’ cybersecurity maturity levels. The department’s comments are reprinted in appendix IV. In written comments, the Department of Energy partially concurred with the recommendation in our report. The department stated that it will coordinate with the energy sector to develop an understanding of sector- wide improvements from use of the framework. The department, however, stated that implementing our recommendation as written prescribes the SCC as a forum for coordination regarding the framework. Our recommendation is not intended to be prescriptive, but rather, to provide suggestions for consideration. Thus, we have revised the wording of the recommendation to emphasize coordination with other entities, as appropriate. The department also stated that the recommendation implies that improvements from the use of the framework could accurately be attributed to a single initiative, which may be misleading. We do not agree. Our report identifies the challenge of determining the direct impact from framework use and notes that NIST’s website provides the sector organizations with access to various framework resources, to include sector-specific implementation guidance and case studies, as well as insights from organizations using the framework. Hence, organizations can report on improvements from use of the framework using multiple initiatives. Further, the department stated that suggesting government collection and reporting of information regarding adoption or improvements erodes the voluntary character of the framework. We do not agree with this statement. Our report recognizes the voluntary character of the framework but also notes that, without collecting and reporting such information, critical infrastructure sectors may not fully understand the benefits and value of the framework to better protect their critical infrastructures from cyber threats. The department’s comments are reprinted in appendix V. In written comments, the Department of Health and Human Services concurred with the recommendation in our report and stated that it would work with the appropriate entities to refine and communicate best practices to the sector. The department’s comments are reprinted in appendix VI. In written comments, the Department of Homeland Security concurred with the recommendation in our report. The department stated that, once it receives the results of the survey on framework adoption that it sent to small- and mid-sized IT sector partners, it will determine the feasibility of issuing similar surveys to other sectors. The department’s comments are reprinted in appendix VII. In written comments, the Department of the Treasury neither agreed nor disagreed with the recommendation in our report. The department stated that it will assess using the identified initiatives and their viability for collecting and reporting sector-wide improvements from use of the framework with input from the SCC and financial regulators. The department added, however, that it does not have the authority to compel financial institutions to respond to inquiries regarding the sector’s use of the framework or resulting improvements. We acknowledge the lack of authority but believe that implementing the recommendation to gain a more comprehensive understanding of the framework’s use by the critical infrastructure sector is essential to the success of protection efforts. The department’s comments are reprinted in appendix VIII. In written comments, the Environmental Protection Agency concurred with the recommendation in our report. The agency stated that it will coordinate with its SCC to investigate options to collect and report sector- wide improvements from use of the cybersecurity framework that are consistent with statutory requirements and the sector's willingness to participate. The agency’s comments are reprinted in appendix IX. In written comments, the General Services Administration concurred with the recommendation in our report and stated that it is working with the Department of Homeland Security to develop a plan to address the recommendation. The agency’s comments are reprinted in appendix X. In comments sent via e-mail, the Department of Transportation’s Director of Audit Relations and Program Improvement stated that the department concurred with the recommendation in our report. In addition to the aforementioned comments, we received technical comments from officials of the Departments of Agriculture, Energy, Health and Human Services, Homeland Security, Transportation, and Treasury. We also received technical comments on the report from the Environmental Protection Agency and General Services Administration. We incorporated the technical comments in the report, where appropriate. We are sending copies of this report to the appropriate congressional committees; the Secretaries of Agriculture, Commerce, Defense, Energy, Health and Human Services, Homeland Security, Transportation, and Treasury; the Administrators of the Environmental Protection Agency and General Services Administration; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6240 or at dsouzav@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix XI. Our objectives were to determine the extent to which (1) agencies with lead roles in critical infrastructure protection efforts, referred to as sector- specific agencies (SSAs), have determined the level and type of National Institute of Standards and Technology Cybersecurity Framework (framework) adoption and (2) implementation of the framework has led to improvements to the protection of critical infrastructure from cyber threats. To address the first objective, we analyzed documentation and evidence, such as implementation guidance and survey instruments that discussed actions federal and nonfederal entities have taken since our report in 2018 to develop methods to determine the level and type of adoption across their sectors, as we previously recommended. These entities included nine SSAs,13 out of the 16 Sector Coordinating Councils (SCC) representing all 16 critical infrastructure sectors established in federal policy, the National Institute of Standards and Technology (NIST), and Information Sharing and Analysis Centers (ISAC). We also analyzed documentation from the SSAs and SCCs, such as the Department of Energy’s Cybersecurity Capability Maturity Model and the Department of the Treasury’s Financial Services Sector Cybersecurity Profile. We compared these to best practices, such as the National Infrastructure Protection Plan and the Standards for Internal Control in the Federal Government to determine efforts to facilitate framework adoption across the sectors. We supplemented our review by interviewing officials from these entities to determine any actions taken to determine framework adoption. In addition, we selected six critical infrastructure sectors identified in the 2018 National Cyber Strategy of the United States of America as having critical infrastructure with the greatest risk of being compromised. The six sectors were (1) communications, (2) financial services, (3) energy, (4) healthcare and public health, (5) information technology, and (6) transportation systems. We asked SCCs, trade associations (e.g., the American Petroleum Institute), and ISACs to provide a list of organizations that were users of the framework. We divided up the list of identified organizations by sector, and we randomly selected one large and one small or medium organization from each sector, resulting in a final list of 12 organizations. We then conducted semi-structured interviews with officials from the selected organizations to understand the extent to which these organizations were using the framework. To address the second objective, we collected and reviewed documentation from NIST and the federal and nonfederal entities, such as NIST’s framework and its April 2019 Roadmap for Improving Critical Infrastructure Cybersecurity, the Department of Homeland Security’s Information Technology Sector Small and Midsize Business Cybersecurity Survey and 2018 Cybersecurity Resources Road Map, and other SSA efforts to determine ongoing efforts to enable the identification and measurement of improvements as a result of using the framework. We compared these efforts to the 2014 Cybersecurity Act and best practices, such as NIST Special Publication 800-55 on performance- based measures to determine the measures the SSAs and SCCs had taken to determine improvements from using the framework. In addition, we interviewed officials from the selected organizations to understand the extent to which they realized improvements as a result of framework adoption and the support the organizations received from federal and nonfederal entities. We also interviewed officials from other federal and nonfederal entities, to include NIST, nine SSAs, 13 of the 16 SCCs, and six ISACs on efforts to measure improvements from use of the framework, and any related challenges. We conducted this performance audit from January 2019 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Neelaxi Lakhmani (assistant director), Kendrick M. Johnson (analyst in charge), Christopher Businsky, Nancy Glover, Douglas Harris, Ceara Lance, Edward Malone, Gabriel Nelson, Harold Podell, and Dana Pon made key contributions to this report.", "summary": "Cyber threats to the nation's critical infrastructure (e.g., financial services and energy sectors) continue to increase and represent a significant national security challenge. To better address such threats, NIST developed, as called for by federal law, a voluntary framework of cybersecurity standards and procedures. The Cybersecurity Enhancement Act of 2014 included provisions for GAO to review aspects of the framework. The objectives of this review were to determine the extent to which (1) SSAs have developed methods to determine framework adoption and (2) implementation of the framework has led to improvements in the protection of critical infrastructure from cyber threats. GAO analyzed documentation, such as implementation guidance, plans, and survey instruments. GAO also conducted semi-structured interviews with 12 organizations, representing six infrastructure sectors, to understand the level of framework use and related improvements and challenges. GAO also interviewed agency and private sector officials. Most of the nine agencies with a lead role in protecting the 16 critical infrastructure sectors, as established by federal policy and referred to as sector-specific agencies (SSAs), have not developed methods to determine the level and type of adoption of the National Institute of Standards and Technology's (NIST) Framework for Improving Critical Infrastructure Cybersecurity (framework), as GAO previously recommended. Specifically, two of the nine SSAs had developed methods and two others had begun taking steps to do so. The remaining five SSAs did not yet have methods to determine framework adoption. Most of the sectors (13 of 16), however, noted that they had taken steps to encourage and facilitate use of the framework, such as developing implementation guidance that links existing sector cybersecurity tools, standards, and approaches to the framework. In addition, all of the 12 selected organizations that GAO interviewed described either fully or partially using the framework. Nevertheless, implementing GAO's recommendations to the SSAs to determine the level and type of adoption remains essential to the success of protection efforts. The 12 selected organizations using the framework reported varying levels of resulting improvements. Such improvements included identifying risks and implementing common standards and guidelines. However, the SSAs have not collected and reported sector-wide improvements. The SSAs and organizations identified impediments to doing so, including the (1) lack of precise measurements of improvement, (2) lack of a centralized information sharing mechanism, and (3) voluntary nature of the framework. NIST and the Department of Homeland Security (DHS) have initiatives to help address these impediments. Precise measurements: NIST is in the process of developing an information security measurement program that aims to provide the tools and guidance to support the development of information security measures that are aligned with an individual organization's objectives. However, NIST has not established a time frame for the completion of the measurement program. Centralized sharing: DHS identified its homeland security information network as a tool that was intended to be the primary system that could be used by all sectors to report on best practices, including sector-wide improvements and lessons learned from using the framework. Voluntary nature: In April 2019, NIST issued its NIST Roadmap for Improving Critical Infrastructure Cybersecurity , version 1.1, which included a tool for organizations to self-assess how effectively they manage cybersecurity risks and identify improvement opportunities. While these initiatives are encouraging, the SSAs have not yet reported on sector-wide improvements. Until they do so, the extent to which the 16 critical infrastructure sectors are better protecting their critical infrastructures from threats will be largely unknown. GAO is making ten recommendations—one to NIST on establishing time frames for completing selected programs—and nine to the SSAs to collect and report on improvements gained from using the framework. Eight agencies agreed with the recommendations, while one neither agreed nor disagreed and one partially agreed. GAO continues to believe that all ten recommendations are warranted.", "document_type": "gao"}
{"report": "Wildfires play an important ecological role on the nation’s landscapes but various management practices over the past century—including fire suppression, timber harvesting, and grazing—have altered the normal frequency of fires in many forest and grassland ecosystems and have reduced these ecosystems’ resilience to wildland fire. This history of fire exclusion and changes in forest management have resulted in a buildup of surface fuels—burnable material found on or near the ground—and the overstocking of some forests with trees and other fuels. In addition, the reduced frequency of wildfire in some ecosystems has resulted in increased amounts of vegetative debris (e.g., dead trees, branches, leaves, and grasses) accumulating on the ground, which serves to increase fuel quantities and can create more continuous fuels. When this occurs, surface fires—fires that occur on the ground—may ignite more quickly and burn with greater intensity, causing fires to spread more rapidly and extensively than they may have in the past. The arrangement of living vegetation also affects the way wildfires burn. For example, an increase in the density of small trees creates a layered forest structure with fuels going from the forest floor into the forest’s canopy. These layers are sometimes referred to as ladder fuels. This arrangement may allow fire that previously would have remained on the ground to climb the ladder fuels and spread into the trees’ crowns, becoming a high-intensity crown fire. In addition, reducing the frequency of fire in fire-adapted forests and other ecosystems can result in changes to the plant species that make up the forest or ecosystem, which may cause the vegetative composition to shift toward species that are not well adapted to fire, including non-native invasive species. For example, many areas with sagebrush ecosystems—that historically had fires only once every few decades—have been invaded by cheatgrass that when dried creates large swaths of fuels that increase rates of fire spread, intensity, and frequency. Approximately 70,000 communities nationwide are considered to be at risk from wildfire, according to the National Association of State Foresters, Communities at Risk, Fiscal Year 2018 Report. Communities face different levels of risk from wildfires depending on such factors as the flammability of vegetation in and around the community, the flammability of materials used in constructing structures, and the location of the structures in relation to vegetation. Structures not located immediately adjacent to wildland vegetation can also be vulnerable to wildfire because winds can transport flaming embers that can ignite homes more than a mile away from a wildfire. In addition to residential housing, other valuable assets and infrastructure that support communities may be located in the WUI, including power lines; highways; and natural resources that provide economic benefits, such as timber, oil and gas wells, and recreational areas. According to the Cohesive Strategy, reducing fuels can help reduce a wildland fire’s intensity, which in turn can help lower the risk fires pose to communities, structures, and other valuable assets and infrastructure. The Forest Service, BLM, FWS, and NPS manage more than 670 million acres of federal land across the country. In addition, BIA is responsible for administering approximately 55 million acres of lands held in trust by the United States for Indian tribes, individuals, and Alaska Natives. Figure 1 shows the lands that these five agencies managed or administered in the contiguous United States. The agencies have estimated that over 100 million of these acres are at high risk from wildfire. Each agency has a unique mission that shapes how it manages or administers its associated lands. Specifically: The Forest Service manages land for multiple uses, such as grazing, timber, recreation, and watershed protection, and to sustain the health, diversity, and productivity of the nation’s forests and grasslands. The agency operates through nine regional offices that manage 154 national forests and 20 national grasslands. BIA provides services, directly or through contracts or compacts, to federally recognized tribes comprising approximately 1.9 million American Indian and Alaska Natives, many of whom live on BIA- administered lands. Tribal forests provide a source of revenue and jobs for many tribal governments and their members, and play an important role in sustaining tribal cultures and traditions, according to BIA documents. The agency operates through 12 regional offices that manage 83 BIA field units. BLM manages land for multiple uses, such as recreation, mining, grazing, timber, and natural scenic values. The agency operates through 12 state offices that manage subsidiary district and field offices. FWS manages the National Wildlife Refuge System, a network of lands and waters that provides for the conservation; management; and, where appropriate, restoration of fish, wildlife, and plants and their habitats, as well as opportunities for wildlife-dependent recreation, including hunting, fishing, and wildlife observation. The refuge system includes approximately 585 refuges. The agency operates through eight regional offices that manage the refuges. NPS manages the National Park System to conserve the scenery, natural and historic objects, and wildlife therein and to leave them unimpaired for the enjoyment of future generations. Individual park units have varied designations corresponding to the natural or cultural features they are to conserve, including national parks, monuments, lakeshores, seashores, recreation areas, preserves, and historic sites. The agency operates through seven regional offices that manage 419 individual park units. Generally, after receiving its annual appropriation, the Forest Service allocates its fuel reduction funds to its nine regional offices, which in turn allocate the funds they receive to individual field units (e.g., national forests and grasslands). Interior, upon receiving its annual appropriation, allocates its fuel reduction funds through its Office of Wildland Fire to BIA, BLM, FWS, and NPS. These agencies then allocate the funds to their regional offices, which, in turn, allocate the funds to individual field units, such as national parks or wildlife refuges. Once the field units receive their allocations, they select fuel reduction projects to implement during the fiscal year. For fiscal years 2009 through 2018, the Forest Service and Interior implemented fuel reduction projects that treated, respectively, approximately 1.4 million and 1.1 million acres per fiscal year on average. Figure 2 illustrates the annual appropriation and allocation processes for fuel reduction funds. From fiscal years 2009 through 2018, Congress appropriated approximately $5 billion in fuel reduction funds to the Forest Service and Interior, with the Forest Service and Interior annually receiving on average about $339 million and $177 million, respectively (see fig. 3). Most development in the WUI occurs on nonfederal lands. Accordingly, state and local government agencies, as well as property owners, play a major role in protecting communities and other development from wildfire. The Forest Service and the National Institute of Standards and Technology have developed publicly available resources that describe ways communities can adapt to wildfire. Specifically, two critical actions for protecting structures from wildfires are (1) reducing vegetation and flammable objects within an area of 30 to 100 feet around a structure, referred to as creating defensible space, and (2) using fire-resistant roofing materials and covering attic vents with mesh screens to block embers from entering the structure. Individuals and communities can also take steps to mitigate fire risk by avoiding development in higher-risk areas. To help protect structures, state and local agencies may conduct, or help fund, fuel reduction projects to protect communities and other nonfederal lands from wildfire. For example, a rural fire department in Montana funds a crew to reduce fuels around private residences to create defensible space for those homes. In addition, individual property owners may reduce fuels around their homes. In previous reports, we found that state and local agencies have adopted laws or ordinances that require homeowners to maintain a specified level of defensible space or have adopted building codes that require the use of fire-resistant building materials in fire-prone areas. For example, in our May 2017 report, we found that under an Oregon law, property owners in certain at-risk areas must reduce excess vegetation around structures and along driveways. According to Forest Service and Interior documents and officials, the Forest Service and the four Interior agencies use various methods to reduce fuels, which have advantages and disadvantages under different conditions. For example: Mechanical treatments. This method entails using equipment such as chainsaws, masticators, bulldozers, or mowers to cut and remove vegetation. Mechanical treatments reduce tree density where there are abnormally dense groups of trees or ladder fuels to help reduce the risk of a wildfire becoming severe. Interior officials said that mechanical treatments are also widely used for removing shrubs and other vegetation in rangeland ecosystems. However, mechanical treatments may also increase the amount of smaller fuels on the ground, including treetops and limbs (referred to as slash) and other debris from thinning, which can in some cases increase a fire’s intensity or rate of spread. Prescribed burns. This method entails using deliberate, planned fires set by land managers to restore or maintain desired ecosystem conditions and reduce fuels. Prescribed burning under specified fuel and weather conditions is designed to enable a fire to burn at a relatively low intensity level within a confined area. Prescribed burns typically work best when combined with previous prescribed burns or mechanical treatments because they are effective in removing smaller vegetation that can fuel a fire—such as grasses, leaves, pine needles, and twigs—which can reduce a fire’s intensity and rate of spread, but are not as effective in removing larger fuel, such as trees. Smoke produced from prescribed burns and the risk of a prescribed burn spreading into other areas can limit the use of prescribed burns around communities, according to the Forest Service’s Fuels Technical Guide. Herbicides and targeted grazing. Herbicides can be used to reduce fuels or when needed to kill fast growing vegetation to maintain an existing fuel reduction project. However, herbicide kills vegetation but does not remove it, potentially increasing an area’s susceptibility to fire if further action is not taken to remove the dead fuel. Targeted grazing—the intentional use of cows, sheep, or goats to eat vegetation in a specified area—can also be used to reduce grasses and other smaller fuels that can fuel fires. One advantage of such methods is that they often can be applied with a greater level of control over the location, timing, and desired outcome of the treatment. These methods can be particularly helpful in removing smaller fuels in areas where prescribed burning is undesirable, such as in proximity to structures. With grazing, however, it may take multiple years before there is a noticeable difference in the fuels, and according to agency officials, moving livestock to different areas for grazing is labor-intensive and can potentially increase the spread of invasive plants if livestock movement is not controlled. While some fuel reduction projects may be completed with a single treatment method, other projects may require multiple treatment methods and may span several years. For example, a project may first use mechanical treatment to thin accumulated vegetation, followed by a prescribed burn to remove remaining slash and litter on the ground. Moreover, once a project is completed, it needs to be maintained over time to retain its effectiveness as vegetation grows back. Depending on the ecosystem, fuels treatment effectiveness can vary in length from only a few years to over a decade. For example, fuel reduction projects are generally effective for 3 to 5 years in southeastern U.S. pine forests given the high rate at which vegetation grows in that region. In contrast, projects are generally effective for 8 to 12 years in dry conifer forests in the western United States. The most appropriate fuel reduction method or methods—as well as how they are applied (i.e., how much vegetation is removed)—depends on the outcomes desired (e.g., protecting communities, restoring ecosystems); the type of forest or other vegetation present; and site-specific factors, such as topography and proximity to communities, according to the Forest Service’s Fuels Technical Guide and agency officials. The Forest Service and Interior have long-standing research programs that are designed to support agency managers’ understanding of how to implement effective fuel reduction projects. As of November 2019, Forest Service research priorities included refining the scientific understanding of how wildfire burns across landscapes and the effects of fuel reduction projects conducted at different scales. In addition, the agencies conduct assessments, known as fuel treatment effectiveness monitoring reports, in cases where a wildfire either starts within or burns into a fuel reduction project area to evaluate the project’s effect on fire behavior and fire suppression actions. Officials believe that such research helps their agencies continue to improve how they design and implement fuel reduction projects to account for site-specific factors. Regardless of the method used, the purpose of fuel reduction projects is to reduce the intensity of future wildfires to help protect communities, restore ecosystems, or both, according to agency documents. The following examples illustrate various fuel reduction methods that the agencies have used to help protect communities and ecosystems: Officials from BIA and the San Carlos Apache Tribe said that they perform prescribed burns and mechanical treatments annually on approximately 1,000 to 1,600 acres of the San Carlos Apache Indian Reservation in Arizona to remove rapidly growing grasses, which could quickly carry a wildfire into the community. The officials said that they primarily use prescribed burns as this allows them to inexpensively treat the most acres. The officials said that they perform these treatments close to the community, to help keep fires from reaching structures and to provide space for firefighters to work more safely in the event of a fire (see fig. 4). An FWS official at the Mississippi Sandhill Crane National Wildlife Refuge said that the refuge uses prescribed burns and mechanical treatments to reduce the wildfire risk to several nearby communities. For example, for a 1,000-acre area near Ocean Springs, Mississippi, the refuge has been doing fuel reduction projects for decades in an effort to protect nearby residential and commercial areas, as well as a highway, railroad, and other infrastructure (see fig. 5). The official said that because the dominant tree species on the refuge is slash pine, which grows very quickly, they have to treat the area every 3 to 5 years to maintain the effectiveness of the project. The official also said that the refuge uses more mechanical treatments than prescribed burns in this area because of concerns about smoke drifting into nearby communities but that they also use prescribed burns when weather conditions are favorable. Santa Fe National Forest officials said that since the early 2000s, they have partnered with the New Mexico State Forestry Division and the New Mexico Department of Game and Fish to conduct a series of fuel reduction projects, including mechanical treatments and prescribed burns, covering 8,000 acres in the Jemez Mountains of New Mexico. These projects were designed to reduce both the likelihood of a fire reaching nearby communities and potential ecosystem damage. The officials said that given the proximity to development and the large accumulation of fuels in that area, they used mechanical treatments first because a prescribed burn would be hazardous until fuel levels were reduced. After the mechanical treatments were completed, they used prescribed burns to remove as much of the remaining fuels as possible. Officials told us that the utility of these projects was demonstrated in July 2018 when the Venado Fire burned from an untreated into a treated area and changed from a high-intensity fire burning the crowns of the trees to primarily a low-intensity fire burning on the ground (see fig. 6). The officials said that while they do not know what the Venado Fire would have done without the fuel reduction projects, they believe that the projects slowed the fire sufficiently to provide firefighters with time to contain the fire before it spread to populated areas and also helped reduce ecosystem damage. NPS officials at the Whiskeytown National Recreation Area near Redding, California, said that many of the fuel reduction projects they undertake are designed to reduce risk to local communities and restore ecosystem health. For example, the officials said that in 2013 they began a 1,000-acre project, consisting primarily of prescribed burns but also some mechanical treatments, located adjacent to privately owned houses and timber land. The officials said that they primarily use prescribed burns because the lower cost of the burns allows them to treat more acres. The project was intended to reduce fire risk to adjacent private property and to help improve the ecological health of old-growth Douglas-fir stands within the recreation area. The officials said that they believed the project helped to reduce the intensity in some areas burned by the 2018 Carr Fire but also noted that the fire was too intense for the treatments to be effective in other areas, as shown in figure 7. Officials at the BLM West Desert District office in Utah said that they have been working on a 4,680-acre fuel reduction project since 2017. The primary purpose of this project is to improve breeding and winter habitat for the greater sage-grouse by removing juniper and other vegetation that pose a wildfire risk to the sagebrush habitat the bird relies on. The project area is home to the largest population of greater sage-grouse in the state. The officials said that they mostly use mechanical treatments, including mastication, because mastication, unlike other fuel reduction methods, allows for the selective removal of juniper trees while still preserving sagebrush. Figure 8 shows the project area before and after treatment, with juniper trees removed and sagebrush remaining. Agency officials told us that in deciding how to allocate their fuel reduction funds in fiscal year 2018, they primarily considered information related to the wildfire hazard potential on lands they manage or administer, the proximity of communities and infrastructure to those potential fires, and ecosystem health. Wildfire hazard potential. To allocate their fuel reduction funds, officials from the five agencies said they considered information regarding the likelihood and severity of wildfires that may occur across the areas they manage and administer. For example, officials said they generally used information incorporated into a national geospatial database that the Forest Service developed to estimate the relative probability a given area faces of experiencing a wildfire that would be difficult for suppression resources to contain and therefore may cause damage to communities or ecosystems. To produce this database, the Forest Service used, among other things, satellite imagery to identify fuel conditions across the landscape. The Forest Service then ran computer models that used this fuel condition information to estimate the potential intensity of future wildfires. The Forest Service’s identification of the likelihood and potential intensity of a wildfire in a given area helps the agencies compare the relative hazard potential different geographic areas face from such fire. The agencies also used information from another national geospatial database that the Forest Service developed on historical fire occurrence data to identify where fires have most frequently occurred, whether because of natural causes (e.g., lightning) or human causes (e.g., accidental ignitions or arson). Figure 9 shows the wildfire hazard potential, as assessed by the Forest Service in July 2018, on lands the five agencies managed and administered in the contiguous United States. Location of communities and infrastructure. Officials from the five agencies told us that they considered the location of communities and important infrastructure, such as municipal watersheds and electrical transmission lines, which could be damaged by wildfires. The officials said they used several information sources to help them identify the locations of these communities and infrastructure. For example, the agencies used a national geospatial database that the Forest Service developed that maps the WUI as defined by the Forest Service and Interior in 2001. Field unit officials said that they also considered local knowledge about areas that are important to protect in or near to a given community when selecting fuel reduction projects to prioritize and implement. For example, officials said that many communities had developed Community Wildfire Protection Plans— plans identifying areas the communities believe are important to protect—and that they would consider these local plans when selecting fuel reduction projects to implement. Ecosystem health and location of natural resources. Officials from four of the five agencies said that they considered information on the locations of particularly valued natural resources, such as rare or otherwise important plants, including those that provide habitat for threatened or endangered species. Using an interagency tool, they also considered information on the overall ecological condition of forests, grasslands, and other vegetation and how current conditions related to historical conditions in given locations. The officials said that this information helped them identify areas where wildfires may be more damaging than they were in the past because of changes in the density, age, and species composition of the vegetation. For example, officials said that in part because of decades of fire suppression, many ponderosa pine forests currently contain more trees than they would have historically, and as a result, today’s wildfires may burn hotter and cause more damage to those forests than fires did in the past. Reducing fuels can help the agencies to restore an area closer to its historical conditions, which in some ecosystems may reduce the risk of wildfire damaging an ecosystem and the resources it contains, according to the Cohesive Strategy. As they considered similar information on potential damage to communities and ecosystems, each agency used a different approach for allocating fuel reduction funds in fiscal year 2018, according to agency documents and agency officials. Officials from each of the agencies said that professional judgment plays an important role in making these decisions. The general approaches each agency used for allocating fuel reduction funds in fiscal year 2018 were: Forest Service. Forest Service headquarters officials said they allocated fuel reduction funds to their regions based primarily on the allocation levels from the previous fiscal year. However, they also said they considered information based on the best available science on the wildfire risk facing the regions and each region’s contributions to meeting the agency’s acreage targets for fuel reduction projects in the previous fiscal year. According to a 2017 Forest Service manual, the agency was to develop national and regional risk assessments to help inform their approach to allocation decisions, but the national assessment had not been finalized for use in fiscal year 2018. Forest Service officials initially allocated approximately 70 percent of the agency’s total fuel reduction funds to the regions, withholding about 30 percent to make available to regions and national forests on a competitive basis later in the fiscal year. The regions and forests then competed for additional fuels funds for projects aligned with specific national priorities as determined by Forest Service headquarters. Interior. Interior’s Office of Wildland Fire officials said they allocated fuel reduction funds to the Interior agencies based primarily on allocation levels from fiscal year 2017. However, late in the third quarter of fiscal year 2018, Interior officials began testing an approach for reviewing each of the four Interior agencies’ planned fuel reduction projects for consistency with the Secretary of the Interior’s priorities for the fiscal year. The agencies’ plans for such projects were to be updated each quarter to keep Interior officials informed on the implementation status for projects underway and of changes to planned projects, according to Interior documents. BIA. BIA headquarters officials told us they allocated fuel reduction funds to their regional offices based on an allocation model that the agency adopted around fiscal year 2012. The model analyzes wildfire hazard potential and agency staffing levels across BIA regions, among other factors. According to a BIA document, the model includes information that captures risk-related information for wildfires on BIA-administered tribal lands. It also captures information on performance and fiscal management for each BIA regional office’s fuel reduction program during the previous fiscal year and each BIA regional office’s contributions to the total number of acres treated overall by the BIA fuel reduction program. BIA officials said the comparative scores for each regional office derived from the model served as a starting point for discussions with BIA senior leadership when determining the fuel reduction allocations to the regions. BLM. BLM headquarters officials said they allocated fuel reduction funds to their state offices based on the results of the 5-year allocation model the agency adopted in 2015. The model analyzes the location of communities, critical infrastructure, and sagebrush habitat, among other factors, as well as wildfire fire hazard potential for the area covered by each BLM state office. According to BLM officials, the model provides a relative ranking for each BLM state office based on acreage at risk, which helps determine the state offices’ respective fuel reduction allocations. For example, BLM state offices that manage more sage-grouse habitat that is at high risk for wildfire received larger allocations than offices in states without such habitat or where the sage-grouse habitat was at lower risk for wildfire. FWS. FWS headquarters officials said they allocated fuel reduction funds to their regional offices based on the results of an allocation system—the Fuels Management Allocation and Accountability System—that they have used since fiscal year 2016. This system generates a risk profile for each FWS region based on, for example, the location of infrastructure, population density, and how fuel conditions may affect wildfires that occur on FWS-managed land. According to FWS officials, this system provides a relative ranking for each FWS region based on acreage at risk, which helps determine the regions’ respective fuel reduction allocations. In general, the FWS regions with the most acreage at risk receive the largest percentage of FWS’s fuel reduction funds. NPS. NPS headquarters officials told us that they allocated fuel reduction funds to their regions based primarily on historical allocation levels from fiscal year 2017. Headquarters officials said they are considering ways to improve their allocation process, such as potentially adopting a model developed in one of their regions. Specifically, officials from the NPS region in our review said that they had developed a model to help analyze the relative risk facing the field units in their region when making allocation decisions. This model is designed to identify highly valued assets in the national parks and other NPS-managed lands in the region and provide relative rankings for those assets requiring protection through fuel reduction projects, according to the officials. Officials we interviewed from the five federal agencies cited a variety of factors affecting their efforts to implement fuel reduction projects. The officials also identified steps they were taking to help mitigate some of the factors. Scale of problem. Officials from all five agencies we interviewed said that the number of acres needing fuel reductions is significantly larger than the number of acres the agencies are able to treat in any given year. As previously noted, the Forest Service estimated in 2018 that there were approximately 63 million acres of national forest lands at high to very high risk from uncharacteristic wildfire, and Interior officials estimated in 2019 that 54 million acres of the lands that they manage or administer were at high or very high risk from wildfire. In fiscal year 2018, the Forest Service and Interior implemented fuel reduction projects that treated approximately 1.7 million and 1.3 million acres, respectively, of lands they manage or administer. Agency officials told us that they recognize that their efforts will not allow them to reduce fuels on all high-risk lands needing treatment but said that in addition to the projects they undertake to reduce fuels, wildfires also serve to reduce fuels in areas burned by such fires. In some circumstances, officials said, wildfires may provide similar fuel reduction benefits as prescribed burns and other fuel reduction methods. To the extent that wildfires reduce fuels in areas that the agencies would otherwise plan to implement fuel reduction projects, such wildfires would serve to reduce fuels on more acreage than they would otherwise be able to treat. Agency officials also said, as previously discussed, that they are working to improve their ability to identify areas to prioritize for treatment. For example, scientists at the Forest Service’s Rocky Mountain Research Station are helping the agency refine its methods for identifying areas most at risk from wildfire and the communities closest to those areas by expanding and updating agency risk assessments to more accurately depict where fuels reduction projects on national forest lands could provide the most protection to communities. This may also allow Forest Service officials to reduce the total number of acres needing treatment through better targeting of the highest-risk acres. According to Forest Service officials, the agency intends to consider this research to help inform its budget requests and funding allocations for fuel reduction efforts in future fiscal years. The Forest Service and Interior are also working to improve their existing fuel reduction project computer simulation software—called the Interagency Fuels Treatment Decision Support System—so that it can be used to model and quantify the risk reduction effects of potential projects across larger geographic areas. Officials said these improvements would help them prioritize areas to treat by allowing agency officials to explore how different combinations of locations and types of treatments affect predicted future wildfire behavior. Operating under continuing resolutions. Officials we interviewed from all five agencies said that operating under continuing resolutions negatively affected their ability to implement fuel reduction projects. Specifically, agency officials said that they tend to budget conservatively until they receive their regular appropriation and therefore implementation of planned projects may be delayed. For example, Forest Service officials said that the weather for doing prescribed burns is often better in the fall and winter and that receiving their annual appropriation later in the fiscal year can reduce their ability to perform these burns in a given year. In addition, the officials said they had delayed hiring and training staff in previous years when the agencies were operating under continuing resolutions, reducing the number of staff available to implement projects. The Forest Service has taken some steps to mitigate the effects of operating under continuing resolutions. For example, officials in one region said they recently adopted an approach that allows them to more readily shift funding from one planned fuels project to another, either within the same national forest or to other national forests in the region, to complete projects as weather conditions and budgets allow. Officials from one national forest in this region said that this approach has facilitated sharing fuels reduction staff among neighboring national forests to plan additional projects, thereby leading to a broader array of projects being ready for implementation when the agency receives its regular annual appropriations. Balancing fuels projects in new areas with maintaining past treatments. Officials from all five agencies said that it can be difficult to balance conducting fuel reduction projects in new areas with maintaining areas that have already had initial fuel reduction projects completed. Some agency officials said that while it is important to conduct projects to reduce wildfire risk in new areas, they also need to conduct projects in previously treated areas to maintain the effectiveness of past treatments. Agency officials said that in balancing their investments between new and previously treated areas, they consider the relative costs of projects. Conducting fuel reduction projects in new areas can be more expensive than conducting maintenance projects because of the type of treatments that need to be done, according to officials. For example, officials from one national forest said that initial mechanical treatments may cost from $300 to $1,500 per acre, depending on the area where the treatment is located, while conducting prescribed burns to maintain a previously treated area may cost from $25 to $100 per acre. Availability of staff. Agency officials from all five agencies said that fuel program staff may be involved in wildfire suppression efforts and therefore may not be available to plan or perform fuel reduction projects, leading to delays in completing such projects. Officials noted that this was largely an unavoidable result of the agencies’ approach to suppression operations, whereby staff from many of the agencies’ program areas, including fuels, are mobilized through temporary emergency assignments to respond to large wildfires across the country as they occur. Agency officials said that they are used to working within staff availability constraints. However, some officials expressed concern about the potential for staff burnout. Specifically, fuel program staff may work many overtime hours when suppressing fires and additional overtime hours when they return to their field units to catch up with planned fuel reduction projects that were delayed because of the emergency suppression assignments. Higher cost of treating WUI areas. Officials we interviewed from four of the five agencies said that costs are a factor when determining which projects to pursue and that it can be more expensive to conduct fuel reduction projects close to homes and infrastructure in the WUI. For example, officials at one national forest said that conducting prescribed burns close to communities in the WUI typically costs almost $250 per acre, whereas it may cost $60 per acre to reduce fuels further away from communities. Agency officials told us that they try to balance their work between WUI and non-WUI areas to ensure treatment of high-risk areas. In balancing between WUI and non-WUI areas, some Forest Service field unit officials noted that Forest Service headquarters annually sets fuel reduction acreage targets for each region; each region then sets targets for each of its national forests and grasslands. Some officials said that as their annual targets for acres of fuel reduction increase, they may feel pressure to choose projects in locations where they can treat more acres to meet their targets, even if those acres may not be located in the areas at highest risk from wildfire damage. Forest Service headquarters officials said that they do not pressure field units to meet the targets but that they are aware that increasing the annual fuels targets, while budgets remain relatively flat, may incentivize field units to select lower cost areas, which may be at lower risk from wildfire. The officials added that the field units, consistent with Forest Service guidance, should be selecting their project locations based on their risk assessments, not cost. Community acceptance of fuel reduction projects. Officials we interviewed from four of the five agencies said that community concerns about the effects of proposed fuel reduction projects have affected their ability to conduct some projects but that they are often able to work with communities to gain their acceptance. For example, the officials said that community members are frequently concerned that smoke from prescribed burns will have negative impacts on their health and quality of life, or that mechanical thinning of vegetation near their communities will be visually unattractive or have negative impacts on wildlife. Agency officials said that they work to minimize these impacts. For example, Forest Service officials schedule prescribed burns at times when weather conditions are not expected to cause a significant volume of smoke to drift into communities. The officials also said that they work with community members to educate them about the benefits of reducing fuels, steps the agencies are taking to reduce negative impacts on the community and wildlife, and steps community members can take to help avoid some impacts. In other instances, agencies partner with various stakeholders to help mitigate negative effects of fuel reduction projects on communities. For example, the Forest Service in New Mexico is part of the Greater Santa Fe Fireshed Coalition, a group that loans air filters to community members who are sensitive to smoke to help them avoid negative health impacts from prescribed burns. Limited economic value of biomass. Officials from three of the five agencies we interviewed said that, in contrast to commercial timber harvests in which contractors pay the agency for the material they remove, fuel reduction projects often produce small trees and other biomass with limited economic value. As a result, fuel reduction projects are unlikely to generate revenues that the agencies could use to help offset the costs of completing such projects. To help mitigate this issue, Forest Service officials said they are working to expand their use of a practice known as stewardship contracting. Through stewardship contracting, the agencies can trade goods—such as timber—for fuel reduction or forest restoration services that the agencies would otherwise pay for with appropriated dollars. Officials we interviewed at two national forests said that the use of stewardship contracts had facilitated their ability to conduct fuel reduction projects, although officials at one of the forests also said they were concerned that the relatively long length of the contracts could slow the rate at which contractors completed the projects. The Forest Service is also researching ways to increase demand for small trees and other biomass—for example, by expanding their use in energy production and building materials—which, if successful, could help to increase the economic value of the material. We provided a draft of this report to the Department of Agriculture and the Department of the Interior for review and comment. In comments reproduced in appendix II, the Forest Service, responding on behalf of the Department of Agriculture, generally agreed with our findings. In addition, the Forest Service and Interior provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Agriculture, the Secretary of the Interior, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Appendix I: Federal Agencies, Agency Units, and Nonfederal Entities Interviewed Regional office (geographic area covered by region) Field unit (state in which unit is located) Southwestern Region (Arizona, New Mexico, Oklahoma, Texas) Cibola National Forest (New Mexico) Santa Fe National Forest (New Mexico) Pacific Southwest Region (California, Hawaii) Cleveland National Forest (California) Shasta-Trinity National Forest (California) Pacific Northwest Region (Oregon, Washington) Deschutes National Forest (Oregon) Southern Region (Alabama, Arkansas, Florida, Georgia, Kentucky Louisiana, Mississippi, North Carolina, Oklahoma, South Carolina, Tennessee, Texas, Virginia, and the territory of Puerto Rico) Francis Marion and Sumter National Forests (South Carolina) Western Region (most of Arizona, Nevada, Utah) San Carlos Agency (Arizona) Utah State Office (Utah) West Desert District (Utah) Southeast Region (Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, and the territories of Puerto Rico and the U.S. Virgin Islands) Mississippi Sandhill Crane National Wildlife Refuge (Mississippi) Pacific West Region (portions of Arizona; California; Hawaii; Idaho; portions of Montana; Nevada; Oregon; Washington; and the territories of American Samoa, Guam, and the Northern Mariana Islands) Whiskeytown National Recreation Area (California) In addition to the contact named above, Jonathan Dent (Assistant Director), David Lysy (Analyst-in-Charge), Aditi Archer, Kathryn Godfrey, Richard Johnson, Gwen Kirby, Anne Rhodes-Kline, Dan Royer, and Kyle Stetler made key contributions to this report.", "summary": "Wildfires have been increasing in size and severity, exacerbated by abnormally dense vegetation, drought, and other climate stressors. Development in and around wildlands also continues to increase, placing more people at risk from wildfires. To reduce vegetation that can fuel such fires, federal land management agencies implement fuel reduction projects on public lands. GAO was asked to examine the federal government's preparedness, response, and recovery efforts following the wildfires and other natural disasters of 2017. This report describes (1) methods federal agencies use to reduce fuels to help protect communities and ecosystems, (2) information the agencies considered in allocating fuel reduction funds in fiscal year 2018, and (3) factors affecting agency efforts to implement fuel reduction projects. GAO examined laws, regulations, and agency policies and budget documents; interviewed federal agency officials at headquarters, as well as in eight regional offices and 10 field units selected based on their locations' high wildland fire hazard potential; and interviewed officials from nonfederal entities, including representatives from the state forestry agencies for the seven states where selected field units were located (three field units were in California and two were in New Mexico). Five federal land management agencies—the Department of Agriculture's Forest Service and the Department of the Interior's Bureau of Indian Affairs, Bureau of Land Management, Fish and Wildlife Service, and National Park Service—use several methods to reduce fuels (vegetation) to help lower the intensity of wildland fires on lands they manage or administer. These methods primarily include mechanical treatments, which use equipment to cut and remove vegetation, and prescribed burns, which are deliberate, planned fires set by land managers. The agencies have long-standing research programs designed to further develop their understanding of how to implement effective fuel reduction projects, including conducting assessments to evaluate project effectiveness. Officials said the research helps the agencies to improve how they design and implement fuel reduction projects to address site-specific conditions. In fiscal year 2018, when allocating fuel reduction funds, the agencies considered information on wildfire hazard potential, the location of communities, and ecosystem health and the location of natural resources. Total fuel reduction appropriations exceeded $5 billion in fiscal years 2009 through 2018 (see figure). Officials from the five agencies cited several factors affecting implementation of fuel reduction projects. A key factor officials cited is that the number of acres needing treatment is significantly larger than the agencies can treat annually. The agencies have estimated that over 100 million acres they manage or administer are at high risk from wildfire, but, for example, in fiscal year 2018 they treated approximately 3 million acres. The agencies are developing risk assessments to help identify areas to prioritize for fuel reductions.", "document_type": "gao"}
{"report": "Coastal communities face hazards from coastal storms and flooding that can cause loss of life, property damage, and damage to the environment. More specifically, coastal communities face threats from erosion and damages from waves, wind, and storm surges. For example, during Superstorm Sandy in 2012, shoreline water levels rose across the East Coast, causing billions of dollars in property damage to homes and businesses. These threats can be exacerbated by several factors, including sea level rise and commercial and residential development, according to Corps documents on coastal risk reduction and resilience. For example, rising sea levels increase the risks from regular tidal flooding and coastal storms and new construction along coastlines can increase the number of people and buildings at risk from the storms. The Corps constructs projects to help reduce the risks from coastal storm hazards and mitigate erosion, wave damage, and flooding, which may include the use of hard structures. The Corps has decades of experience developing projects that use hard structures, such as revetments, seawalls, and storm surge barriers, to reduce the risks from coastal storm hazards, according to a 2014 report by the National Academy of Sciences (see fig. 1). Natural infrastructure can also be designed and developed for coastal storm and flood risk reduction purposes. Natural infrastructure can involve several types of natural features that have the potential to reduce risks to coastal areas from storms (see fig. 2). Diverse natural features occur in different areas of the United States. For example, some areas along the Florida Gulf Coast are host to mangroves—coastal wetlands found in tropical and subtropical regions—that can reduce the impacts of high energy waves from storm surges. The extent to which natural infrastructure can reduce risks to coastal areas from storms and flooding depends on the types of natural features being used. For example, underwater vegetation, such as seagrass, has less capacity to reduce wave energy than a coral reef, which is a hard underwater structure, according to scientific studies. According to a 2014 National Academy of Sciences report, in addition to reducing the risks of storms and flooding for coastal communities, projects using natural infrastructure may provide other benefits, depending on the type of natural feature associated with the project. Among other things, natural infrastructure has the potential to enhance commercial and recreational fisheries and create recreational opportunities. For example, natural infrastructure may support fish habitats, which could enhance a commercial or recreational fishery. In addition, wetlands may improve habitats for birds, which could enhance bird watching activities. Similarly, replenishing beaches may provide more beach area for individuals to use for recreational activities, and provide nesting habitat for birds and sea turtles. The Corps’ Civil Works program—responsible for water resources projects—is organized in three tiers: a national headquarters in Washington, D.C.; eight regional divisions; and 38 districts (see fig. 3). Corps headquarters primarily develops the policies and guidance that the agency’s divisions and districts carry out as part of their oversight responsibilities for the water resources projects under the Corps’ purview. Corps districts are responsible for planning, engineering, constructing, and managing water resources projects in their districts, including projects that consider or use natural infrastructure. The Corps has several programs and initiatives related to using natural infrastructure for water resources projects. For example, the Engineer Research and Development Center, the research organization within the Corps, manages a portfolio of research related to water resources projects that includes research focused on flood risk management and coastal systems. The Corps also has an initiative called Engineering With Nature®, which the Corps’ scientists and engineers developed to facilitate using sustainable practices in Corps projects. The Corps develops water resources projects, including coastal storm and flood risk management projects, in conjunction with nonfederal sponsors, such as state and local governments. According to Corps guidance, the planning process for these projects begins with a nonfederal sponsor identifying a problem and approaching the Corps to help develop a solution. Upon statutory authorization for a study and appropriations to fund it, the Corps and the nonfederal sponsor enter into an agreement to conduct a feasibility study for a potential project. Nonfederal sponsors are to participate in the planning process, as well as remain involved through project design, construction, and post-project operations and maintenance. For example, for projects where the Corps constructs hard infrastructure, such as a seawall, the nonfederal sponsor is to assume responsibility for monitoring and maintenance costs associated with the seawall after its construction. In contrast, for a project that involves replenishing a beach, the Corps and the nonfederal sponsor usually share the cost of replenishment for a specific period of time, typically 50 years. The U.S. Water Resources Council’s 1983 Economic and Environmental Principles and Guidelines for Water and Related Land Resources Implementation Studies (Principles and Guidelines) outline the standards and procedures that the Corps is to follow for planning water resources projects, including those with coastal storm and flood risk management objectives. The Principles and Guidelines establish that the federal objective of water resources projects is to contribute to national economic development while protecting the nation’s environment. The Corps implements the planning process outlined in the Principles and Guidelines by conducting feasibility studies for proposed water resources projects. The Corps’ Planning Guidance Notebook (Planning Guidance) provides detailed guidance on how to implement the general process outlined in the Principles and Guidelines for planning water resource projects. The Corps’ feasibility study process includes four major phases and five milestones, as shown in figure 4. The Corps initiates a feasibility study by forming a project team, comprising Corps engineers, economists, planners, and other specialists, to conduct the study. The Corps project team begins with a scoping phase that specifies the problem, such as the potential for coastal storm and flood damage, and identifies opportunities for a project to address the problem. The project team then inventories conditions in the project area, including physical, economic, and social conditions, and forecasts how these conditions may change over the life of a potential project. As it continues the scoping phase, the project team identifies various measures that could address the problem, such as replenishing an existing beach or constructing a seawall. The project team then develops potential individual measures or combinations of measures (e.g., beach replenishment and seawall construction) into an initial list of alternatives. Since 2016, the Corps has been required by statute to consider natural infrastructure in certain circumstances. With its initial list of alternatives, the Corps project team is to then evaluate each alternative by (1) comparing it to the scenario of proceeding with no project; (2) applying criteria established in the Principles and Guidelines; (3) identifying beneficial and adverse effects of each alternative; and (4) considering other relevant factors, such as compliance with environmental requirements. To identify beneficial and adverse effects of each alternative, the Corps uses four general categories established in the Principles and Guidelines, as shown in table 2. The Corps’ Planning Guidance states that project teams should evaluate alternatives using the four categories of analysis, but the evaluation from two categories—National Economic Development and Environmental Quality—must be included in each feasibility study. According to the Planning Guidance, evaluating projects’ potential costs and benefits through these categories of analysis provides a basis for determining which alternatives should be eliminated from consideration, modified, or selected for further analysis. This evaluation can eliminate alternatives that do not meet planning objectives and may narrow the initial list of alternatives to a final list for more detailed analyses and comparison. Corps officials stated that the process of evaluating alternatives can be iterative and is project specific. The Corps project team then is to conduct detailed analyses of its final list of alternatives to compare them to each other and select a recommended alternative. The project team includes the recommended alternative in a draft report with its analysis. The draft report is made available for review and comment by nonfederal sponsors, federal and state agencies, and other stakeholders. The project team incorporates comments into the report, as appropriate, and determines whether the agency will endorse the recommended alternative. The project team finalizes its feasibility study after internal review. The Corps then prepares a report summarizing the proposed plan—known as the Chief’s report—and submits it to Congress for consideration and potential authorization. Based on our review of Corps guidance and eight selected projects that used natural infrastructure, we found that the Corps typically identified project costs and damage reduction benefits in selecting the alternative, although for some projects it also considered additional benefits, such as recreational benefits. Once a Corps project team develops a final list of alternatives in conducting a feasibility study for a particular project, the project team is to conduct an economic analysis for each alternative. This analysis allows the team to compare costs and benefits directly across the alternatives, including alternatives using natural infrastructure, hard infrastructure, or a combination of the two. Specifically, the project team is to develop estimates for each project alternative’s net economic benefits—benefits minus costs—to identify and select the project alternative with the maximum net benefits. The Corps’ Planning Guidance states that the Corps shall select coastal storm and flood risk management projects determined to have the maximum net benefits. Our review of Corps guidance and eight selected projects identified the following costs and benefits that the Corps generally incorporated into its economic analyses: Project costs. According to the Corps’ Planning Guidance, project costs include three categories: implementation costs, other direct costs, and associated costs. Implementation costs, for example, include planning and design, construction, construction contingency, operations, maintenance, repair, and other costs necessary to implement a project. The eight selected projects that we reviewed included analyses of project costs, which mostly focused on implementation and interest costs. For example, the costs for the Corps’ Jacksonville District Lido Key project included initial construction costs (i.e., beach construction and hard infrastructure designed to reduce shore currents), future beach replenishment costs (i.e., operations related to placing material on beaches to replenish eroding shores), and monitoring costs (e.g., measurement of beach fill, sediment type, and habitat quality). Damage reduction benefits. Reducing damages to existing structures, including homes and commercial buildings, is the primary benefit the Corps considers when identifying benefits for coastal storm risk management project alternatives, according to the Corps’ Planning Guidance. The guidance outlines general steps for estimating damage reduction benefits, which are to be calculated and included in each coastal storm and flood risk management alternative’s economic analysis. In seven of the eight projects we reviewed, the Corps analyzed damage reduction benefits as part of its economic analysis. For example, the Corps’ project team for the New York District Union Beach project determined the potential damage reduction benefits of each alternative by estimating the alternative’s potential to reduce (1) damages to coastal property from flooding and waves, (2) public emergency spending, and (3) administrative costs for the National Flood Insurance Program (see fig. 5). We also found that for some selected projects, the Corps identified and incorporated additional benefits into the projects’ economic analyses, including the following: Incidental recreational benefits. Corps project teams may include in the economic analysis recreational benefits that stem directly from the project alternative but that are incidental to the primary purpose of damage reduction, according to the Corps’ Planning Guidance. Specifically, Corps project teams may include recreational benefits, such as increases in recreational visits because beaches are larger in their economic analysis of project alternatives, but recreational benefits are limited to no more than 50 percent of the total economic benefits used to justify an alternative (i.e., demonstrate that an alternative has greater benefits than costs). After an alternative has been economically justified, the team can use the full estimated recreational benefits with the damage reduction benefits to select the alternative with maximum net benefits. In our review of eight projects, we identified four projects where the Corps project team included recreational benefits in its economic analysis for the project alternative that was selected. For one such project, the Los Angeles District’s Encinitas-Solana Beach project, the Corps’ economic analysis showed that the selected project alternative had lower damage reduction benefits than project costs. However, when the Corps added recreational benefits—as allowed by Corps policy—the combined annual damage reduction and recreational benefits resulted in the alternative having greater benefits than costs (see fig. 6). Other direct incidental benefits. The Corps may also consider other direct incidental benefits in its economic analysis, as appropriate, according to the Principles and Guidelines. In our review of eight projects, we identified three projects that included estimated incidental benefits aside from recreational benefits. The three projects included economic benefits associated with reduced maintenance costs for local communities, whose expenses for maintaining local beaches would decline after the Corps projects were constructed. Other than these reduced maintenance costs, the Corps did not include other types of direct incidental benefits, such as environmental or other social benefits, in the economic analyses for the eight projects we reviewed. According to Corps officials, some project alternatives using natural infrastructure may provide direct incidental benefits that are not included in the economic analysis, such as environmental and social benefits. For example, the draft feasibility study for the New York District’s Jamaica Bay project states that natural infrastructure can provide direct incidental benefits, such as improving ecosystems, filtering water, and improving aesthetics. The Corps acknowledged these incidental benefits and their importance to communities in its draft feasibility study, but did not incorporate these benefits into its economic analysis because they could not be monetized, according to Corps district officials. Corps headquarters officials said incidental benefits that cannot be monetized in the economic analysis are considered in the planning process through the evaluation of other Principles and Guidelines categories. Two reports published by the National Academy of Sciences stated that when assessing project alternatives, the Corps primarily uses qualitative measures to assess benefits that are difficult to monetize but that relegates such effects to secondary status compared to the monetized estimates of costs and benefits. Moreover, a 2004 National Academy of Sciences report found that the Principles and Guidelines outlines a process that focuses on the effects that can be monetized, which does not allow for full consideration of a project’s total economic effects. Nonetheless, for three of the eight projects we reviewed, we found that the Corps modified its approach in selecting the use of natural infrastructure as part of the recommended alternative. For instance, for the Encinitas-Solana Beach project, the Corps granted an exception to its planning process and recommended a locally preferred plan. In certain circumstances, Corps project teams can deviate from the Corps’ Planning Guidance that calls for the Corps to select the project alternative with the maximum net benefits. Corps headquarters officials said that requesting such an exception is the primary method the agency uses for recommending a project alternative that does not meet the Corps maximum net benefits requirement for a project focused solely on coastal storm or flood risk management. For the Encinitas-Solana Beach project, the California Coastal Commission found that the Corps’ proposed alternative with the maximum net benefits was inconsistent with the mission of California’s coastal management program to protect and enhance the state’s coastal environment. In particular, the Commission had concerns about the size of the project and the amount of sand to be added to the beach under the proposed alternative, as well as the potential adverse effects on a nearshore natural reef and marine resources. In response, the Corps’ Los Angeles District worked with the project’s nonfederal sponsors to address the commission’s concerns and revised the project by reducing its size and potentially lessening its environmental impacts. The commission approved the revised project alternative in November 2013. The Corps’ Planning Guidance also allows projects with multiple objectives to incorporate other analyses in selecting a recommended alternative. For the Philadelphia District’s Lower Cape May project, ecosystem restoration was the project’s primary objective, but it also had a coastal storm risk management objective. According to the project’s feasibility study, the project focused on protecting and restoring a freshwater marsh that was being flooded with salt water from storms because of continued beach erosion. The Corps used a cost- effectiveness analysis to meet the primary objective, which compared environmental measures (e.g., the number of acres of habitat restored) with the costs of different alternatives. In addition, this project included a beach component to protect the marsh from saltwater intrusion. In the process of designing beach alternatives, the Corps project team conducted a damage reduction benefit analysis to determine an optimal size for the beach that would provide the greatest net damage reduction benefit to nearby communities. This analysis helped inform the Corps decision to select a beach design option that met the project’s primary objective of protecting the ecosystem, while also providing the most incidental damage reduction benefits to local communities, according to Corps district officials. For the third project, the New York District’s Jamaica Bay project, the Corps incorporated natural features into the project, although it did not directly include the economic benefits of these features in its economic analysis. For the project, the Corps project team recommended an alternative that was designed to address frequent flooding within Jamaica Bay at three locations. The project team incorporated wetlands into the design at one location, along with hard infrastructure. The nonfederal sponsors of the project told us that they advocated for the inclusion of these natural features, where appropriate, because of their risk reduction and ecological benefits. In response to the interests of nonfederal sponsors, the Corps project team developed and recommended the alternative incorporating hard infrastructure, such as floodwalls, along with coastal wetlands. The Corps did not include the risk reduction benefits from the wetlands in the economic analysis, but the draft feasibility study noted that the project was economically justified based on the monetary benefits of the hard infrastructure alone and that the wetlands provided additional benefits that could not be monetized. Based on our literature review, agency documentation, and interviews with Corps officials and other stakeholders, we found that the Corps faces challenges developing cost and benefit information for some natural infrastructure to help inform the process for selecting project alternatives and conducting economic analyses in feasibility studies. Specifically, these challenges related to (1) assessing the performance of some types of natural infrastructure and (2) monetizing the social and environmental benefits associated with using natural infrastructure. The Corps recognizes the need to obtain additional data to better develop cost and benefit information for some types of natural coastal infrastructure, and it has begun taking steps to do so. Information is not readily available on the performance of some types of natural features in reducing coastal storm and flood damages, which makes it challenging for the Corps to develop cost and benefit information for these features and compare them to other alternatives, such as those that use hard infrastructure. For example, Corps headquarters officials said that—in contrast to beaches and dunes—there are significant knowledge gaps about the extent to which wetlands, reefs, and subaquatic vegetation can reduce the risks associated with coastal storms by, for example, moderating wave heights and flooding. In addition, there are knowledge gaps about how these natural features will change over time and how any changes might affect the long-term performance of the features. A Corps report from January 2015 also identified knowledge gaps in understanding how some natural infrastructure, such as wetlands, may perform during coastal storms or floods. According to the report, wetlands may reduce storm surge, but in some instances water can be redirected, potentially causing a storm surge increase elsewhere. Corps officials noted that all structures—whether natural or hard—change over time, requiring maintenance and repair, but said that natural infrastructure may change more dramatically than hard infrastructure and over a shorter period of time. For example, a healthy wetland could restore itself and reduce maintenance costs after a major storm or require the Corps to take action to restore the wetland after the storm event, which could increase the costs of maintaining the wetland. Corps officials also stated that there are knowledge gaps regarding whether wetlands can absorb major storm surges and how these features would perform in the event of recurring coastal storms in a short period of time. Specifically, natural features may be damaged during intense storms (e.g., wetlands can erode and vegetation may be stripped apart), which may degrade the long-term performance of the features. Because the Corps does not have information on performance for some natural features, it has been unable to update engineering guidance to include the use of some natural features, according to Corps officials. A Corps headquarters official explained that the agency must first develop a broader understanding of how some natural features, such as wetlands, perform under various coastal storm scenarios over time before it can begin to develop design guidance for using these features for coastal storm protection and flood risk management projects. A Corps headquarters official said that the agency recognizes the need to obtain additional information on natural infrastructure and has initiated steps to address the challenge related to developing information on the performance of some types of natural infrastructure. In particular, in October 2016, the Engineer Research and Development Center began collaborating with several entities, including other federal agencies, international partners, academic institutions, and nongovernmental organizations, to develop guidelines for using some types of natural infrastructure. According to the scoping document, this effort is to entail developing guidelines to support various phases of building natural infrastructure projects, including conceptualization, design, engineering, construction, and maintenance. According to the Corps official, an anticipated key output from the international effort includes developing information on defining performance for different types of natural infrastructure features and options for measuring performance depending on project objectives. The final product is expected to include chapters with information on analyzing natural infrastructure benefits and related monitoring, maintenance, and adaptive management issues, among others. The Corps official stated that the guidelines will not be official Corps guidance or policy, but Corps project teams and other practitioners can use the guidelines as a resource for identifying best practices in planning projects and assessing potential alternatives. For example, the guidelines will include case studies illustrating design and engineering concepts for certain types of natural features. The guidelines are scheduled for publication in March 2020. The Corps has also developed a separate internal initiative to help fill knowledge gaps regarding how some natural features’ performance can provide benefits relevant to flood risk management, among other benefits. Specifically, the Corps’ Engineering With Nature® Initiative is focused on sharing natural infrastructure best practices that are emerging, and communicating the information to staff in the Corps’ district offices and other key stakeholders. According to a Corps official, the goal of this initiative, among other things, is to help familiarize the Corps’ district staff with existing natural infrastructure information and relevant case studies. The Corps’ Galveston and Philadelphia Districts have projects that may incorporate natural infrastructure. For example, the Galveston District is considering opportunities through the Coastal Texas study to use natural features, such as barrier islands, wetlands, and reefs, in combination with hard infrastructure (e.g., levees), to reduce the risks from storms and floods. Similarly, the Corps’ Philadelphia District is considering a plan to design, construct, and evaluate natural features as part of the New Jersey Back Bays Storm Flood Risk Management study. In addition, in 2018, the Corps’ coastal working group initiated a project within the Corps to help identify natural infrastructure knowledge gaps and prioritize key areas for research based on requests for information received from Corps’ districts. The Corps plans to incorporate information gathered from this project into a strategic plan that is intended to help inform research funding decisions for fiscal year 2020, according to a Corps official. Our review of economic literature identified challenges in estimating the total economic benefits associated with using natural infrastructure features. Several studies noted that data for conducting economic analyses are not readily available. For example, one study noted that there is insufficient information on how restoring wetlands might affect the survival of certain endangered species. Such information is needed, according to the study, to provide insight on the extent to which such features might generate economic benefits. Another study noted that because projects that combine natural features with more traditional structures (i.e., hybrid projects) are relatively new, less is known about their effectiveness or their costs and benefits. Finally, according to another study, estimating recreational benefits associated with natural habitats, such as coastal marshes, can be difficult because there is insufficient information about the extent to which the public visits those sites. In the eight projects we reviewed, Corps project teams did not estimate incidental benefits other than recreational benefits or through avoiding maintenance costs for coastal storm and flood risk management projects. As previously discussed, environmental and social benefits are considered incidental benefits, and Corps guidance indicates that they do not have to be included in the economic analysis. On the other hand, when assessing potential alternatives of coastal storm and flood risk management projects in its feasibility studies, the Corps can quantify or describe the benefits qualitatively and consider these effects during the planning process, outside of the economic analysis. For example, the Corps has measures to quantify changes in habitat, such as number of acres of wetlands restored. The Corps can also qualitatively describe habitat benefits for specific species. However, these nonmonetized benefits may not affect the selection of the recommended alternative, which is generally based on the monetized net benefit estimates of each proposed alternative. The Corps has begun developing a process for identifying, describing, and considering a broader array of potential benefits when assessing natural infrastructure alternatives for specific projects. Specifically, a June 2017 memorandum from the Corps’ Director of Civil Works indicated that projects with coastal storm and flood risk management objectives as well as other objectives should consider social and environmental benefits in the formulation, design, and implementation of projects within existing legislation and Corps policy. A Corps headquarters official said that the agency is not attempting to monetize all potential benefits but is considering options for accounting for potential benefits other than through the traditional monetary assessments of costs and economic benefits. A Corps headquarters planning group is currently working on developing an initiative that would identify a process for using a flexible approach for considering the social and environmental effects of natural infrastructure for coastal storm and flood risk management projects. For example, project teams may have the option of determining whether to incorporate nonmonetized social and environmental benefits, such as enhancing public safety in coastal communities, into the decision-making process for selecting the recommended alternative. The Corps official stated that the agency has begun working on developing guidance for this initiative and expects to issue the guidance in calendar year 2019. We provided a draft of this report for review and comment to the Department of Defense. The department provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Defense, the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions regarding this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to the report are listed in appendix II. This appendix presents information on the eight projects that we selected for review with coastal storm and flood risk management objectives that the U.S. Army Corps of Engineers (Corps) constructed and that included natural infrastructure. We randomly selected eight projects across Corps districts on the Atlantic, Gulf, and Pacific coasts. In seven of the eight projects, the Corps recommended alternatives with either beaches or dunes as the type of natural features to be used for coastal storm and flood risk reduction (see table 3). According to several Corps district officials we interviewed, alternatives featuring beaches are often most appropriate because other natural features, such as wetlands, would not survive the impacts of the high-energy storm waves in open ocean coastal areas where these projects are located. In addition to the contact named above, Alyssa M. Hundrup (Assistant Director), Leo Acosta (Analyst-in-Charge), Mark Braza, Eric Charles, Timothy Guinane, and Jeanette Soares made key contributions to this report. Important contributions were also made by John Delicath and Sara Sullivan.", "summary": "The Corps constructs water resources projects to reduce risks to coastal communities from storm damage, among other things. These projects can involve building hard structures, such as seawalls, to protect against flooding and wave damage. The Corps and some state and local agencies are increasingly considering using natural infrastructure, such as wetlands, to reduce risks from coastal storms and flooding. GAO was asked to review the uses, costs, and benefits of natural coastal infrastructure for the Corps' coastal storm and flood risk management projects. This report describes (1) how the Corps considered costs and benefits for selected projects that used natural infrastructure and (2) challenges the Corps faces in developing cost and benefit information for using natural infrastructure and steps taken to address them. GAO reviewed Corps guidance; obtained information on projects that used natural infrastructure and received funding from fiscal years 2012 through 2017; randomly selected eight coastal storm and flood risk reduction projects from the Atlantic, Gulf, and Pacific coasts; and reviewed each project's planning documentation and economic analyses. Findings from these projects are not generalizable to all Corps' projects. GAO also reviewed economic literature, reviewed Corps documents related to the use of natural infrastructure, and interviewed Corps officials and stakeholders with experience in using natural infrastructure. The U.S. Army Corps of Engineers (Corps) typically identified project costs and damage reduction benefits for the eight projects using natural infrastructure that GAO reviewed. In selecting projects, the Corps is to conduct economic analyses of project alternatives, which may include hard structures, natural infrastructure, or a combination, to compare their costs and benefits. Corps guidance states that for coastal storm and flood risk management projects it is to select the alternative determined to have the maximum net benefits (benefits minus project costs). The Corps calculated project costs for the eight projects, such as planning, design, construction, and maintenance costs. It calculated damage reduction benefits for seven projects by estimating reduced damages to existing structures in the project area, including to homes and commercial buildings. Corps guidance allows the economic analysis to also include incidental benefits of a project, and four projects incorporated recreational benefits of alternatives, such as increases in recreational visits because beaches would be larger. The Corps did not include other types of incidental benefits, such as environmental or other social benefits, for the eight projects. Corps documentation for one project identified environmental benefits of constructing wetlands as part of the project, such as improving ecosystems and filtering water. However, Corps officials said they did not incorporate these benefits into the economic analysis because the benefits could not be monetized. The Corps faces challenges in developing cost and benefit information for some types of natural infrastructure and has initiated steps to address this. For example, a 2015 Corps report identified knowledge gaps in understanding how natural coastal infrastructure, such as wetlands, may perform during coastal storms. These knowledge gaps make it challenging for the Corps to develop cost and benefit information for some natural infrastructure alternatives and compare them to other alternatives, such as those that use hard infrastructure. The Corps recognizes the need to obtain additional data to better develop cost and benefit information and has begun taking steps to do so. For example, in 2018, the Corps initiated a project to help identify natural infrastructure knowledge gaps and prioritize key areas for research. The Corps plans to incorporate information gathered from this project into a strategic plan that is intended to help inform research funding decisions for fiscal year 2020, according to a Corps official.", "document_type": "gao"}
{"report": "In the United States, deaths from all causes are recorded and tracked through a multi-step registration process, which may vary by state, according to CDC officials. According to CDC officials, when an individual dies, a death certificate is filed with the state. Funeral directors are responsible for providing demographic information about the deceased individual, while physicians, coroners, and medical examiners are responsible for providing information on cause of death. The local registrar of vital statistics is responsible for verifying the information, and then transferring copies of the death certificate to the city or county health department in some jurisdictions, and to the state registrar. The state vital registration office is responsible for verifying the information, maintaining official copies, and creating an electronic record. According to CDC officials we interviewed, state vital records offices submit their death certificate information to CDC electronically. In 2003, CDC revised the standard death certificate to include a checkbox to indicate whether a woman was pregnant at the time of her death or up to 1 year after delivery or end of pregnancy. (See fig. 3) The checkbox is part of the medical portion of the death certificate that is completed by a physician, coroner, or medical examiner. According to CDC officials, there has been staggered adoption of the revised death certificate by states, and not every state death certificate included the pregnancy checkbox until 2019. CDC collects maternal mortality data using two national surveillance systems, NVSS and PMSS. In addition, CDC has developed a data application that state or local MMRCs can use to centrally collect information abstracted from various sources about each death. Federal law directs CDC’s National Center for Health Statistics to collect statistics on maternal mortality. According to CDC officials, the National Center for Health Statistics receives copies of electronic records for deaths from all states and jurisdictions, such as the District of Columbia. It uses this death certificate information to assign ICD-10 codes based on the cause of death. According to CDC officials, the National Center for Health Statistics uses these coded records to compile national vital statistics files in NVSS, which is the source of official statistics on mortality in the United States, including maternal deaths. NVSS data are used to identify national trends and make international comparisons. CDC published national maternal mortality rates for deaths that occurred in 2007 based on data collected in NVSS in its report on all deaths in the United States. CDC also made these data publicly available in microdata files that can be downloaded through website applications and, according to officials, in response to specific requests when additional details, such as geography, are sought. However due to staggered implementation of the 2003 revised death certificate by the states and reliability concerns about the use of the pregnancy checkbox, CDC officials said that as of September 2019, they have not published NVSS statistics on maternal mortality in the agency’s annual mortality reports since the report on deaths that occurred in 2007. For example, as specified in the technical notes of a June 2019 CDC report on national vital statistics for deaths in 2017, CDC noted evidence of an increase in false reporting of maternal deaths as a result of incorrect completion of the pregnancy checkbox on death certificates. According to CDC officials, the individual who completed the pregnancy checkbox may have incorrectly noted that a woman was pregnant or had been pregnant within 1 year of her death, and as a result, the death would have been recorded as a maternal death or late maternal death. However, prior to the addition of the pregnancy checkbox, there was a general concern that the United States was not identifying all of the maternal deaths and thus did not have a full picture of maternal mortality. According to CDC officials, the agency has recently taken steps to improve NVSS data on maternal mortality. For example, in 2018, CDC developed training for individuals who complete the cause of death portion of a death certificate, and in 2019 was developing guidance on completing the pregnancy checkbox. The agency also participated in a quality assurance pilot from January 2016 through March 2017 with four states, Georgia, Louisiana, Michigan, and Ohio, to test processes that may improve state-level data on maternal mortality. The results of the pilot were made available in articles in 2019, and CDC officials said the agency is currently disseminating the findings from the pilot to other states. According to agency officials, one of the findings from the pilot was that a greater proportion of deaths where the pregnancy checkbox on the death certificate was marked incorrectly were for women aged 45 and older. As a result, CDC officials said they plan to use additional criteria when classifying deaths for these women. Specifically, they will only code the death as a maternal death or late maternal death if the cause of death is explicitly reported on the death certificate as due to pregnancy or an obstetric cause. CDC officials stated that the changes they made improved the accuracy of data on maternal mortality, and they will implement the new reporting criteria for 2018 data. On January 30, 2020, CDC published maternal mortality statistics in a National Vital Statistics Report. For data on maternal mortality accessible through CDC’s website, CDC officials said they will direct users of these data to its limitations that are noted in the annual report. Officials said they will also publish guidance on the limitations of the maternal mortality statistics, and they will continue to monitor the accuracy of the data. Taking these steps to improve the NVSS data on maternal mortality and noting their limitations should help provide federal, state, and local organizations with accurate data. For example, CDC notes on its CDC WONDER website that state, local, and county health departments rely on this source of publicly accessible data to review their community’s population health trends, evaluate their program’s performance for planning purposes, and compare their community with other locations. In 1986, CDC initiated a second national surveillance system for maternal mortality, PMSS. Unlike NVSS, PMSS is exclusively focused on pregnancy-related deaths. According to CDC, the system was developed because more clinical classification of the causes of these deaths was needed in order to fill data gaps and help clinicians and public health professionals to better understand circumstances surrounding pregnancy- related deaths, including the causes and appropriate actions to prevent them. This collection effort included expanding the scope of deaths under surveillance to those up to 1 year after the end of pregnancy, which is beyond the international standard of up to 42 days after the end of a pregnancy. To collect the additional data, CDC officials annually send requests to vital records offices for all 50 states and other applicable jurisdictions to provide the following: death certificates, linked live birth or fetal death certificates, and any other supporting information for 1) deaths with an ICD-10 Chapter O code for the previous year, and 2) all deaths from any cause (including injury or trauma) among women who were pregnant or were within 1 year of pregnancy as identified by matching the death certificate to a birth or fetal death certificate or by a pregnancy checkbox on the death certificate. According to CDC officials and related literature, linking information on death certificates to information on infant birth or fetal death certificates can help confirm that the pregnancy checkbox on the death certificate was completed accurately. In addition to confirming validity of a pregnancy-related death as indicated by the checkbox, linking information on death certificates to infant birth or fetal death certificates can identify pregnancy-related deaths where the checkbox did not indicate a pregnancy but should have (false negatives). CDC officials said that PMSS data are considered the most reliable source of national data on pregnancy-related deaths because (1) PMSS links death certificates with birth or fetal death certificates and additional information when available (e.g., hospital records), and (2) these files are reviewed by medically trained epidemiologists to determine if the cause and time of death are related to the pregnancy. CDC publishes national data from PMSS on the leading causes of pregnancy-related deaths and pregnancy-related mortality ratios using this system. However, the data are not published annually—such as is generally the case with NVSS data on deaths—and only national level data are made publicly available from PMSS in annual updates on the website and periodically in reports. According to agency officials, states and jurisdictions voluntarily provide the records in response to CDC’s request that specifies that PMSS analyses will only be published at the national and regional level, and those records are subject to confidentiality protections. Additionally, according to CDC officials, because of the time involved in collecting documentation from states, the most recent data available from PMSS as of September 2019 were for deaths in 2016. To improve the timeliness of PMSS data, CDC is taking steps to gain access directly to the records that states have been submitting to the agency. Specifically, CDC entered into a contract, effective August 2019, for a pilot project with the National Association for Public Health Statistics and Information Systems to become an approved user of the State and Territorial Exchange of Vital Events system. According to the National Association for Public Health Statistics and Information Systems, this vital events system provides timely access to state vital records, including records on deaths, to federal and state data partners for use in authorized public health and administrative programs, like those at CDC. According to the contract, over the next 5 years, select CDC staff will receive training on the use of the system and will coordinate phased access to state and jurisdiction vital records. At the conclusion of the contract, authorized CDC staff are expected to have access to electronic vital records data from up to 51 states and jurisdictions. CDC officials said this should allow them to link birth and death certificate information and no longer rely on states and jurisdictions to conduct vital records linkages for PMSS. According to CDC officials, being able to access the vital events system will allow them to confirm and report pregnancy-related deaths with improved timeliness. In 2017, CDC released MMRIA, in which MMRCs— multidisciplinary committees at the state and other jurisdictional level that review pregnancy-related deaths—can collect and review data from various sources (e.g., medical records, social service records, autopsy reports, and vital records) to determine preventability, and identify factors that contributed to these deaths as well as prevention strategies to address these factors. As of June 2019, CDC officials said that 25 states and one other jurisdiction were using this system. While MMRCs provide the information collected in MMRIA, federally published reports only include aggregate information from select states collected through the application. For example, in May 2019, CDC published a study using information from 13 states. In the study, state MMRCs identified an average of three to four contributing factors per pregnancy-related death based on information collected through MMRIA, such as: community factors (e.g., unstable housing and limited access to transportation); health facility factors (e.g., limited experience with obstetric emergencies and lack of appropriate personnel or services); patient factors (e.g., lack of knowledge of warning signs and nonadherence to medical regimens); provider factors (e.g., missed or delayed diagnosis); and system-level factors (e.g., inadequate access to care and poor case coordination). Similar to PMSS data, the most current aggregated data that CDC publishes from the MMRIA can be for deaths that occurred 2 or more years prior to the date of the report. As noted in the May 2019 article, the most recent information from states contributing to the article varied with some state data on these deaths being as recent as 2017 while the most recent data from other states was from 2014. According to CDC officials, in August 2019, CDC awarded 24 cooperative agreements covering 25 states, and under these agreements, the committees will use the system to record review results within 2 years of a death. Our analysis of CDC’s PMSS data shows that from 2007 through 2016, over 6,700 women died of causes related to or aggravated by their pregnancy—either while pregnant or within 1 year of the end of pregnancy. Our analysis also shows that while there was an overall increase in the pregnancy-related mortality ratio during this time frame, the annual mortality ratio in the United States fluctuated. As previously noted, CDC data also show that racial and age disparities exist in the rates of pregnancy-related deaths. For example, from 2007 through 2016, non-Hispanic black women were more than three times as likely to die than non-Hispanic white women, while non-Hispanic American Indian/Alaska Native women were more than two times as likely to die than non-Hispanic white women. Similarly, rates of pregnancy-related deaths for women 35 years old and older are higher than the rates for women under 30 years old. During this time period, the specific causes of death varied by race/ethnicity and age. Further, CDC data show that most of the deaths occurred within 42 days of delivery or the end of pregnancy. CDC’s PMSS data show that among all pregnancy-related deaths, the cause of death varied. In general, what CDC classifies as “other cardiovascular conditions” was the most common cause of pregnancy- related deaths, followed by infection, hemorrhage, and cardiomyopathy. (See fig. 4.) These four leading causes comprised about 50 percent of all pregnancy-related deaths from 2007 through 2016. See appendix I for more information on leading causes of pregnancy-related deaths. CDC data shows that the leading causes of pregnancy-related deaths differed by racial/ethnic groups. Specifically, for non-Hispanic white and black women, the leading cause was other cardiovascular conditions from 2007 through 2016; for non-Hispanic American Indian/Alaska Native and Asian/Pacific Islander women, it was hemorrhage; for Hispanic women, it was infection, as indicated by figure 5. CDC has reported that multiple factors contribute to pregnancy-related mortality and to racial/ethnic disparities, including community, health facility, patient/family, provider, and system factors. Leading causes of pregnancy-related deaths also differed by the age of the woman, as indicated by figure 6. Specifically, the leading cause for women under 25 was infection, while for all other women the leading cause was other cardiovascular conditions. In a 2017 article, the authors noted that maternal morbidity and mortality rates increase with advanced maternal age, due in part to increased prevalence of chronic conditions (e.g., hypertension, diabetes, and chronic heart disease.). This may help explain the variation in the rate of pregnancy-related deaths among women of different ages. Our analysis of CDC’s PMSS data shows that from 2011 through 2016, most pregnancy-related deaths occurred between 0 and 42 days postpartum—meaning that they occurred either on the day of delivery or end of pregnancy up to 42 days after pregnancy. (See fig. 7.) According to CDC officials, understanding the timing of pregnancy-related deaths is important for prioritizing intervention strategies. The officials noted that deaths resulting from cardiomyopathy can occur months after pregnancy but can also be prevented with appropriate interventions. In particular, the American College of Obstetricians and Gynecologists published guidance for managing pregnancy and heart disease that noted that complications are frequently encountered in the days, weeks, and months after delivery in women with known cardiovascular disease and in those with latent cardiovascular disease. Women with multiple risk factors for cardiovascular disease may be particularly at risk of manifesting symptoms for the first time during their postpartum course. CDC’s data show that the leading causes of pregnancy-related death varied depending on when the death occurred. For example, over the period 2011-2016, hemorrhage and amniotic fluid embolism were leading causes of pregnancy-related deaths on the day of delivery or the end of pregnancy, while cardiomyopathy was the leading cause of pregnancy- related deaths between 43 and 365 days postpartum. (See fig. 8.) A recent article on pregnancy-related deaths stated that multiple factors contribute to pregnancy-related deaths during pregnancy, labor and delivery, and the postpartum period. Further, the article notes that no single intervention strategy is sufficient, and reducing these deaths requires reviewing and learning from each death, improving women’s health, and reducing social inequities across the life span, as well as ensuring quality care for pregnant and postpartum women, according to the article. See appendix II for supplemental data on pregnancy-related deaths. State Participation in the Alliance for Innovation on Maternal Health (AIM) Initiative According to the American College of Obstetrics and Gynecology, as of June 2019, 26 states were enrolled in the AIM initiative. The AIM initiative, funded by the Health Resources and Services Administration, engages provider organizations, state-based public health systems, consumer groups and others in a national partnership to assist state- based teams in implementing evidence-based maternal safety bundles. Ten of the 26 states joined in the last year and are beginning to implement maternal safety bundles and collect data. Five of these bundles are being implemented by one or more states. Bundle topics: Maternal Venous Thromboembolism, Postpartum Care Basics for Maternal Safety: From Birth to Comprehensive Postpartum Visit, Obstetric Care for Women with Opioid Use Disorder, Obstetric Hemorrhage, Reduction of Peripartum Racial/Ethnic Disparities, Safe Reduction of Primary Cesarean Birth, Severe Hypertension in Pregnancy, and Postpartum Care Basics for Maternal Safety: Transition from Maternity to Well Woman Care. Preventing Maternal Deaths: Supporting Maternal Mortality Review Committees Cooperative Agreements. Under these cooperative agreements, CDC is providing funding to state agencies and organizations that coordinate and manage MMRCs. As previously mentioned, MMRCs systematically and comprehensively review pregnancy-related deaths in order to identify prevention opportunities. Funding recipients will identify and review deaths within 1 year of death and enter clinical and non-clinical data and committee decisions in MMRIA—a standardized data system managed by CDC—within 2 years of death. As part of the agreement, recipients—in coordination with CDC—analyze data and share findings with stakeholders, such as clinicians, to inform policy and prevention strategies to reduce pregnancy-related deaths, such as screening procedures. According to CDC officials, in August 2019, CDC awarded these 5-year cooperative agreements to 24 recipients covering 25 states. Recipients received different amounts ranging from $150,000 to over $550,000 in the first year to support their MMRC. CDC anticipates awarding a similar level of funding for the 5- year period of performance. Maternal and Child Health (MCH) Services Block Grant Program. HRSA provides funding through this program to 59 states and jurisdictions to improve maternal and child health. According to agency officials, many recipients reported using their MCH Services Block Grant funding to help support or complement other federal initiatives, such as an MMRC, a Perinatal Quality Collaborative (PQC), and Alliance for Innovation on Maternal Health (AIM) maternal safety bundles. For example, according to HRSA officials, in fiscal year 2018, 38 recipients self-reported that the block grant partially or fully funded their MMRCs, and additional states and jurisdictions reported using block grant support for planning activities to begin development of their MMRC. Further, states and jurisdictions cited PQCs, networks of multidisciplinary teams that work to improve measurable outcomes for maternal and infant health, in their block grant narrative. Additionally, implementation of a HRSA-supported AIM maternal safety bundle, sets of actionable, evidence-based practices for improving maternal outcomes was cited. (Appendix III includes more information on funding for PQCs and AIM maternal safety bundles.) Our review of HRSA documentation shows that in fiscal year 2017, total federal expenditures for the block grant program were about $540 million for women and children covered by the program, and expenditures for services for pregnant women from all sources—federal funds, as well as state, local, program income, and other funds—was about $300 million. Indian Health Service (IHS) Implementation of Alliance for Innovation on Maternal Health (AIM) Maternal Safety Bundles In 2017, IHS’s leadership released a request that IHS federal hospitals that provide inpatient obstetric care implement at least one maternal safety bundle—sets of evidence- based practices that when implemented collectively and reliably in the delivery setting may improve patient outcomes and reduce maternal mortality and severe maternal morbidity. According to IHS, since 2014, IHS has had phased implementation of the bundles in federal hospitals that provide inpatient obstetric care. Officials said that for many facilities, Obstetric Hemorrhage was the first bundle implemented. Others have also been implemented, such as the Severe Hypertension in Pregnancy bundle, and the Obstetric Care for Women with Opioid Use Disorder bundle. According to officials we interviewed in five selected states, they use these two efforts—MMRC findings and MCH block grant funding—and other efforts collectively to address pregnancy-related deaths. For example, according to Georgia officials, Georgia’s PQC received funding from CDC and implemented the AIM obstetric hemorrhage maternal safety bundle in 2018 based on the state’s MMRC finding that hemorrhage was a leading cause of pregnancy-related deaths in Georgia. According to officials, Georgia’s MMRC was funded primarily through the MCH Services Block Grant. Similarly, according to Maryland officials, Maryland’s PQC oversees implementation of the state’s AIM initiative. Officials we interviewed from three of the five selected states said that the AIM initiative had an immediate or the largest effect on addressing maternal mortality in their state. Officials from the other two states said they could not identify which efforts had the largest or most immediate effect on addressing maternal mortality. Officials from one state noted the importance of their collaborative approach and the other noted that there is no one contributing factor for maternal mortality. See appendix V for more information about how the selected states we interviewed are using these funding efforts. All five states also mentioned beginning or continuing to address racial/ethnic or other health disparities with block grant funding, through their MMRCs, or other efforts. For example, officials in one state said they use block grant funding to support its Black Infant Health Program, which helps address maternal morbidity and mortality of black mothers in the late maternal period. Additionally, two of the HHS funding efforts awarded in fiscal year 2019 have outcomes related to decreasing racial and ethnic disparities in maternal mortality: the Alliance for Innovation on Maternal Health Community Care Initiative and the State Maternal Health Innovation Program. In addition to those efforts that are exclusive to maternal mortality or have a focus on maternal mortality, HHS agencies have other funding efforts that may reduce maternal mortality by improving maternal health. For example, agency officials also identified the following: HRSA’s Maternal, Infant, and Early Childhood Home Visiting Program supports voluntary, evidence-based home visiting services for at-risk pregnant women and parents with children up to kindergarten entry. Our review of agency documentation shows that in fiscal year 2019, HRSA awarded about $351 million in funding to 56 states, territories, and nonprofit organizations to support communities in providing voluntary evidence-based home visiting services through the Maternal, Infant, and Early Childhood Home Visiting Program. The Substance Abuse and Mental Health Services Administration, which is responsible for leading public health efforts to advance the behavioral health of the nation and reducing the impact of substance abuse and mental illness on America’s communities. The agency funds two programs that provide grants to public and private nonprofit entities and state substance abuse agencies for substance use disorder treatment and recovery services for pregnant and postpartum women. Our review of agency documentation and interviews with agency officials shows that from fiscal year 2017 through 2019, the Substance Abuse and Mental Health Services Administration awarded 41 Services Grant Program for Residential Treatment for Pregnant and Postpartum Women and six State Pilot Grant Program for Treatment for Pregnant and Postpartum Women grants. The Centers for Medicare & Medicaid Services, which administers the Medicare and Medicaid programs, developed the Maternal Opioid Misuse Model. Through this model, state Medicaid agencies will coordinate with care-delivery partners to test whether payments for evidence-based, coordinated care delivery improve outcomes and reduce costs for pregnant and postpartum Medicaid beneficiaries with opioid use disorder and their infants. According to agency officials, funding for cooperative agreements with 10 state Medicaid agencies began in January 2020. We provided a draft of this report to HHS. HHS provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of HHS, and appropriate congressional committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have questions about this report, please contact me at (202) 512-7114 or deniganmacauleym@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. In 1986, the Centers for Disease Control and Prevention (CDC) initiated national surveillance of pregnancy-related deaths in the Pregnancy Mortality Surveillance System (PMSS) because more clinical information was needed to fill data gaps about causes of these deaths. A pregnancy- related death, as defined in statute, is the death of a woman while pregnant or within 1 year of the end of a pregnancy—regardless of the outcome, duration or site of the pregnancy—from any cause related to or aggravated by the pregnancy or its management, but not from accidental or incidental causes. As of September 2019, CDC used 11 categories when coding the cause of death for pregnancy-related deaths in PMSS, and 2016 data were the most recent data available. From 2007 through 2016, there were 6,765 pregnancy-related deaths, according to PMSS data. See table below for information on the 11 cause of pregnancy-related death categories, including PMSS data on leading causes, most common time frame, and most common age group affected. The following tables include supplemental data on pregnancy-related deaths by racial/ethnic and age groups. As of September 2019, the Department of Health and Human Services was providing funding for 13 efforts with a stated outcome, goal, or focus on reducing pregnancy-related deaths. One of the these—Supporting Maternal Mortality Review Committees—is funded by CDC and has an exclusive focus on reducing deaths of the women during pregnancy or up to 1 year of pregnancy, while the other 12 have additional focus areas, such as improving infant health. Two of these efforts are not discrete funding opportunities, but rather a variety of research funding opportunities offered by the Health Resources and Services Administration and the National Institutes of Health. Table 5 lists the 13 efforts, their current awards, funding, purpose, and examples of goals or research. Health Resources and Services Administration (HRSA) and National Institutes of Health (NIH) officials noted research the agencies support, including funding related to maternal health that also includes projects specific to maternal mortality or that can affect maternal mortality. The Maternal and Child Health Bureau supports field-based, applied and translational research through an extramural research program that provides leadership and funding that support innovative research to inform practitioners, the scientific community, and the public. According to HRSA officials, this research program helps to advance the field of maternal and child health; improve the health and well-being of women, children, and families; and address the needs of economically or medically vulnerable maternal and child health populations. According to HRSA officials, in fiscal year 2018, HRSA awarded a total of about $1.2 million in funding for six research projects related to maternal illness. The following HRSA website includes an option for searching for funded projects using key terms, https://mchb.hrsa.gov/research/. Baeva, S., D.L. Saxton, K. Ruggiero, et al. “Identifying Maternal Deaths in Texas Using an Enhanced Method, 2012”, Obstetrics & Gynecology, vol. 131, no. 5 (2018): 762-769. Casey, M.M., P. Hung, C. Henning-Smith, et al. “Rural Implications of Expanded Birth Volume Threshold for Reporting Perinatal Care Measures.” Joint Commission Journal on Quality and Patient Safety, vol. 42, no. 4 (2016): 179-187. Hung, P., K.B. Kozhimannil, M.M. Casey, et al. “Why Are Obstetric Units in Rural Hospitals Closing Their Doors?” Health Services Research, vol. 51, no. 4 (2016): 1546-1560. Kozhimannil, K.B., C. Henning-Smith, P. Hung, et al. “Ensuring Access to High-Quality Maternity Care in Rural America.” Women’s Health Issues, vol. 26, no. 3 (2016): 247-250. Kozhimannil, K.B., P. Hung, M.M. Casey, et al. “Factors Associated with High-Risk Rural Women Giving Birth in Non-NICU Hospital Settings.” Journal of Perinatology, vol. 36, no. 7 (2016): 510-515. Kozhimannil, K.B., M.M. Casey, P. Hung, et al. “Location of Childbirth for Rural Women: Implications for Maternal Levels of Care.” American Journal of Obstetrics and Gynecology, vol. 214, no. 5 (2016): 661e1- 10. Kozhimannil, K.B., C. Henning-Smith, and P. Hung. “The Practice of Midwifery in Rural US Hospitals.” Journal of Midwifery & Women’s Health, vol. 61, no. 4 (2016): 411-418. Kozhimannil, K.B., P. Hung, M.M. Casey, et al. “Relationship between Hospital Policies for Labor Induction and Cesarean Delivery and Perinatal Care Quality among Rural U.S. Hospitals.” Journal of Health Care for the Poor and Underserved, vol. 27, no. 4 (2016): 128-143. Weigel, P.A., F. Ullrich, D.M. Shane, et al. “Variation in Primary Care Service Patterns by Rural-Urban Location.” Journal of Rural Health, vol. 32, no. 2 (2016): 196-203. NIH support research, including funding maternal health research through a number of its institutes and centers, such as the Eunice Kennedy Shriver National Institute of Child Health and Human Development; the National Heart, Lung, and Blood Institute; the National Institute of Alcohol Abuse and Alcoholism; the National Institute of Diabetes and Digestive and Kidney Diseases; the National Institute of Mental Health; the National Institute of Nursing Research, and the Office of Research on Women’s Health. For example, NIH officials noted that The Eunice Kennedy Shriver National Institute of Child Health and Human Development supports essential research designed to overcome many of the complex challenges that women encounter in trying to achieve and maintain healthy pregnancies, and to prevent maternal mortality and severe maternal morbidity. In fiscal year 2018, NIH funded 661 projects totaling almost $303 million that included a focus on maternal health. The following NIH website includes a link to funded research for fiscal years 2015 through 2018 and estimates for fiscal year 2019 and 2020 by category, including maternal health, https://report.nih.gov/categorical_spending.aspx. MacDorman, M.F., E. Declercq , and M.E. Thoma. “Making Vital Statistics Count: Preventing U.S. Maternal Deaths Requires Better Data.” Obstetrics & Gynecology, vol. 131, no. 5 (2018): 759-761. MacDorman, M.F., E. Declercq, H. Cabral, et al. “Recent Increases in the U.S. Maternal Mortality Rate: Disentangling Trends from Measurement Issues.” Obstetrics & Gynecology, vol. 128, no. 3 (2016): 447-455. Thoma, M.E., D.A. De Silva, and M.F. MacDorman. “Examining Interpregnancy Intervals and Maternal and Perinatal Health Outcomes Using U.S. Vital Records: Important Considerations for Analysis and Interpretation.” Paediatric and Perinatal Epidemiology, vol. 33, no. 1 (2019): O60-O72. Brogly, S.B., K.E. Saia, M.M. Werler, et al. “Prenatal Treatment and Outcomes of Women with Opioid Use Disorder.” Obstetrics & Gynecology, vol. 132, no. 4 (2018): 916-922. Dimidjian, S., S.H. Goodman, J.N. Felder, et al. “Staying Well During Pregnancy and the Postpartum: A Pilot Randomized Trial of Mindfulness-based Cognitive Therapy for the Prevention of Depressive Relapse/Recurrence.” Journal of Consulting and Clinical Psychology, vol. 84, no. 2 (2016): 134-145. Hauspurg, A., S. Parry, B.M. Mercer, et al. “Blood Pressure Trajectory and Category and Risk of Hypertensive Disorders of Pregnancy in Nulliparous Women.” American Journal of Obstetrics and Gynecology, vol. 221, no. 3 (2019): 277.e1-277.e8. Liu, T., M. Zhang, E. Guallar, et al. “Trace Minerals, Heavy Metals, and Preeclampsia: Findings from the Boston Birth Cohort.” Journal of the American Heart Association, vol. 8, no. 16 (2019): e012346. Miller, E.C., M. Gallo, E.R. Kulick, et al. “Infections and Risk of Peripartum Stroke during Delivery Admissions.” Stroke, vol. 49, no. 5 (2018): 1129-1134. Sheen, J.J., J. D. Wright, D. Goffman, et al. “Maternal Age and Risk for Adverse Outcomes.” American Journal of Obstetrics and Gynecology, vol. 219, no. 4 (2018): 390.e1-390.e15. To describe how selected states use Department of Health and Human Services (HHS) funds to implement select efforts to reduce maternal mortality, we interviewed officials from five states—California, Georgia, Illinois, Maryland, and Texas—selected because of their geographic diversity and because these state have the following efforts shown in Table 6. Officials from three of the five states we interviewed said that the Alliance for Innovation on Maternal Health (AIM) Initiative had an immediate or the largest effect on addressing maternal mortality in their state. Officials from the other two states said they could not identify which efforts had the largest or most immediate effect on addressing maternal mortality. Officials from one state noted the importance of a collaborative approach and the other noted that there is no one contributing factor for maternal mortality. Some of the HHS-funded efforts previously described in appendix III had not been awarded at the time of our interviews, such as the State Maternal Health Innovation Program cooperative agreements. See table 6 below for information about these states’ efforts. Mary Denigan-Macauley, Director, (202) 512-7114 or deniganmacauleym@gao.gov. In addition to the contact above, Raymond Sendejas (Assistant Director), Natalie Herzog (Analyst-in-Charge), Sam Amrhein, Margaret Cullinan, Kaitlin Dunn, Laura Ann Holland, Diona Martyn, Jennifer Rudisill, and Vikki Porter made key contributions to this report. Other contributors include Jieun Chang, Leia Dickerson, Sandra George, and Amy Leone.", "summary": "Every year in the United States, hundreds of women die of complications related to pregnancy and childbirth. According to CDC data, racial/ethnic disparities exist with regard to these deaths. For example, non-Hispanic black women were more than three times as likely to die as non-Hispanic white women, and non-Hispanic American Indian/Alaska Native women were more than two times as likely to die as non-Hispanic white women. GAO was asked to review issues related to maternal mortality in the United States. In this report, GAO describes, among other things, (1) trends in pregnancy-related deaths in the United States, including trends in causes and timing of these deaths, and (2) HHS funding efforts focused on reducing pregnancy-related deaths. GAO reviewed documentation about HHS's surveillance efforts related to pregnancy-related deaths; and analyzed CDC data on leading causes of pregnancy-related deaths from 2007 through 2016 (the most recent 10-year period available at the time of GAO's review). GAO also reviewed documentation and interviewed HHS and state public health officials in five selected states about HHS's funding efforts aimed at reducing pregnancy-related deaths, including select efforts used in these states. GAO selected these states primarily based on their geographic diversity and their implementation of select efforts to address maternal mortality. GAO provided a draft of this report to HHS. HHS provided technical comments, which GAO incorporated as appropriate. GAO's analysis of the Centers for Disease Control and Prevention's (CDC) Pregnancy Mortality Surveillance System data shows that from 2007 through 2016, over 6,700 women died of causes related to or aggravated by their pregnancy—either while pregnant or within 1 year of the end of pregnancy. While CDC data show an overall increase in the pregnancy-related mortality ratio in the United States during this time frame, the annual ratio fluctuated. Cardiovascular conditions, infection, and hemorrhage were the leading causes of pregnancy-related deaths, and comprised about 50 percent of all pregnancy-related deaths from 2007 through 2016. In addition, CDC data show that the leading causes of pregnancy-related deaths differed by racial ethnic groups. (See figures.) The Department of Health and Human Services has 13 ongoing efforts aimed at reducing pregnancy-related deaths. The following are key examples of these: Supporting Maternal Mortality Review Committees Cooperative Agreements . According to CDC officials, in September 2019, CDC awarded 5-year cooperative agreements to 24 recipients covering 25 states with amounts ranging from $150,000 to over $550,000 in the first year, totaling about $8.4 million. Under these agreements, CDC is providing funding to state agencies and organizations that coordinate and manage Maternal Mortality Review Committees. The committees are responsible for comprehensively reviewing deaths to identify prevention opportunities. Maternal and Child Health (MCH) Services Block Grant Program .The Health Resources and Services Administration provides funding through this program to 59 states and jurisdictions to improve maternal and child health. In fiscal year 2017, total expenditures for services for pregnant women from all sources—federal funds, as well as state, local, program income, and other funds—was about $300 million. According to agency officials, many recipients reported using their block grant funding to help support or complement other federal initiatives, such as their review committee, quality collaborative, and use of maternal safety bundles. According to officials GAO interviewed in five selected states, they use these efforts and others collectively to address pregnancy-related deaths. For example, according to officials in one state, they implemented an obstetric hemorrhage maternal safety bundle in 2018 based on the state's Maternal Mortality Review Committee finding that hemorrhage was a leading cause of pregnancy-related deaths in the state. According to officials, the state's Maternal Mortality Review Committee was funded primarily through the MCH Services Block Grant. All five states also mentioned beginning or continuing to address racial/ethnic or other health disparities with block grant funding, through their Maternal Mortality Review Committees, or other efforts. For example, officials in one state said they use block grant funding to support their Black Infant Health Program, which helps address maternal morbidity and mortality of black mothers in the late maternal period. Additionally, two of the HHS funding efforts awarded in fiscal year 2019 have outcomes related to decreasing racial and ethnic disparities in maternal mortality: the Alliance for Innovation on Maternal Health Community Care Initiative and the State Maternal Health Innovation Program.", "document_type": "gao"}
{"report": "FAA defines an FBO as a business granted the right by the airport to operate fueling facilities, hangars, aircraft tie-downs, aircraft rental, aircraft maintenance, flight instruction, and other aeronautical services at an airport. In addition, FBOs sometimes manage parking ramps for transient aircraft at the airport. FBOs may charge a fee for parking, as they also maintain the ramp areas for the airport. According to FAA, airports, within certain parameters, have the ability to charge or not to charge users for access to airport ramp space. FBOs generally serve pilots who operate general aviation aircraft, but can also support commercial flights. The type of amenities and services any one FBO provides varies. For example, representatives of one FBO said that it provides high-level customer service and offers more services such as catering, pilot lounges, concierge services, and aircraft maintenance and repair facilities for its clients. See figure 1 for an illustration of FBO services. As of March 2019, we identified 3,070 FBOs operating at 3,016 airports located in the contiguous United States; these airports are included in FAA’s National Plan of Integrated Airport Systems (NPIAS). FBOs can be run by the airport, an independent operator, or a network chain with multiple locations. The Transportation Research Board estimated in 2016 that 47 percent of all FBO locations were airport-operated. Most stakeholders with whom we spoke agreed that there are fewer FBOs today than in the past, although estimates vary on the extent of the decline in FBO numbers. Stakeholders we interviewed said that this decline is due to factors such as a drop in general aviation activity that resulted in a reduction in fuel sales. More recent innovations, such as more fuel-efficient aircraft and decision-making software that provides information to pilots on where to purchase fuel to fly more efficiently, also contributed to the decline. Airports that receive federal AIP grants contractually agree to FAA “grant assurances” that require those airports to adhere to certain requirements. One of those key grant assurances is a prohibition of unjust economic discrimination. This assurance requires airports to provide to users equal access to airport facilities. Likewise, tenant businesses (e.g. FBOs) operating at airports are required to make services available and price those services not in an unduly discriminatory fashion. Although FAA is required to ensure that airports, as a condition for accepting federal grants, provide fair and equal access to services and nondiscriminatory pricing, FAA does not regulate prices in the FBO industry. As a condition of accepting federal grants, airports have a responsibility to ensure that they and their contractors and concessionaires abide by the grant assurances. FAA’s Airport Compliance Office oversees airports’ adherence to grant assurances by taking and responding to inquiries and complaints, developing and circulating advisories and guidance documents, and coordinating with airports and industry to conduct compliance training and airport land use inspections. The Department of Justice (DOJ) also plays a role in overseeing the FBO industry under its antitrust responsibilities to preserve competition. For example, in the Final Judgement entered by a federal court in the DOJ’s case regarding the acquisition by BBA Aviation (Signature Flight Support) acquisition of Landmark Aviation, BBA was required to divest FBO facilities in six locations where the transaction would have created a monopoly or duopoly for FBO services. In addition, the court order required BBA to provide advance notice of certain future acquisitions for the 10-year duration of the final judgment. Based on our review of FBO and third party websites, we found that fuel prices at FBOs are readily available to anyone on the internet. Nearly all of the pilots we spoke with told us they use these resources for making flight plans. For example, current prices for fuel are readily available on third party websites such as AirNav and Sky Vector, among others, and on about a third of the websites for FBOs we visited. See figure 2 for a representation of a website providing FBO information. However, fees for other services such as for parking and aircraft handling are less transparent. Our review of FBO and third-party websites found that such fees are not always available online, and that fees may vary by type of aircraft, are sometimes waived, and are called by different terms. According to FBO staff we spoke with, fees for services other than fueling can be lengthy and unwieldy to post on their websites for multiple reasons. First, some fees will vary based on the size and approved weight of the aircraft. For example, the price sheet for services other than fuel at one FBO showed fees varying by the aircraft’s approved weight, so there were 11 different prices for each of those services. The same pricing sheet also included prices for dozens of incidental services, such as aircraft towing and lavatory service that are not based on aircraft weight. Additionally, customers may be eligible for discounts on fuel purchases either by volume or through a membership program. FBOs may also waive fees in some cases—for example, with a qualifying fuel purchase an FBO might waive a parking, ramp, or handling fee. Further, a few stakeholders and pilots we spoke to indicated that FBOs don’t always use the same terms for a fee. For example, a landing fee or a ramp fee might be a fee for doing essentially the same thing. Consequently, to find out how much an FBO visit will cost, 16 of the 18 pilots we interviewed told us they call the FBO in advance. Based on information such as their type of aircraft, length of stay, and other services they might require, the FBO provides an estimate of their total cost. Recently, some industry stakeholders have called for increased price transparency and consistency among FBOs regarding how they characterize their fees, and have taken some actions to increase the transparency of fees. A campaign called, “Know Before You Go,” developed through the cooperation of six aviation associations, encourages FBOs to communicate and expeditiously provide available services and a listing of currently applicable posted fuel prices, as well as fees and charges for other available services. Further, the campaign suggests that these fees and charges should be made accessible to aircraft operators online in a user friendly manner and with sufficient clarity. Additionally, it encourages customers to contact the FBO to ask questions so pilots can make informed decisions. In response, one large- chain FBO began posting fees online for piston aircraft at its locations and another FBO company created a trip calculator on its web site for pilots to calculate the cost of their visit (see fig 3). We also found that a third party company recently began a web site that provides FBO parking ramp fees similar to those providing fuel prices. In addition, AOPA invited FBOs to include their fees in the association’s online airport directory. The association also indicated it categorizes the variety of fees into basic types of fees such as landing, using a hanger, or using lavatory service to help clarify what pilots could be expected to pay. In October 2019, AOPA officials indicated that FBOs’ posting of fees had not increased as much as they hoped. Selected stakeholders we interviewed—including officials from 26 airports, 16 FBOs, as well as 18 general aviation pilots—highlighted key factors that may influence FBO prices at airports across the country. Our statistical model confirmed a correlation between certain key factors identified by stakeholders and FBO prices. Consistent with general economic theory, these factors fall into three groups: (1) an FBO’s costs, (2) demand for an FBO’s services, and (3) competition among FBOs. Selected stakeholders we interviewed highlighted cost factors such as airport leases, infrastructure investment, fuel, labor, and security as influencing FBO prices. They cited the following examples: Airport Leases. Airport leases dictate terms and conditions of contracts between an FBO and an airport and include provisions related to the services an FBO must provide for pilots. Depending on the specific requirements or minimum standards developed at a given airport, FBO lease requirements vary and can affect an FBO’s costs. For example, at one airport we visited, the FBO is required to offer an after-hours self-service fueling option, which necessitates the acquisition and maintenance of additional equipment. In another case, the manager at an FBO we spoke to said that its overhead costs are relatively high because it is required to offer flight training and aircraft maintenance as part of its lease. To offer these services the FBO needs additional hangar space and must pay qualified skilled employees. Infrastructure Investment. As with leasing costs, the greater the investment an FBO makes at an airport, the higher its prices to users may be. According to airport and interest group officials we spoke with, FBOs typically have 20 to 30 year leases during which they may make infrastructure investments such as building hangars or lounges, based on FBO’s assessment of customer demand for its services. For example, according to airport and FBO officials we spoke to, an FBO will choose to invest in high-end facilities and amenities if it determines there is sufficient demand and revenues earned are expected to be sufficient to recoup the costs over the term of the lease. Fuel Transportation Costs. FBOs generally sell two types of aviation fuel for general aviation aircraft: Jet A and 100 low lead (100LL). Jet A is generally delivered over long distances via pipeline. According to one petroleum company, 100LL is generally moved by truck, rail, or barge—less cost-effective methods of transport than pipeline—due to the smaller volumes being produced. Further, there are parts of the United States, specifically on the East Coast, where little or no 100LL is produced and, as a result, transportation costs can significantly affect the cost of fuel to the FBO. Labor costs. FBOs compete in the local labor market for staff. The cost of labor for FBOs may vary across local labor markets around the country. Further, a particular FBO may need specialized skills to provide the services they offer, and this factor can affect the FBO’s costs. For example, some FBOs offer maintenance services, so will have trained mechanics on staff to perform such services. State taxes. Aviation fuel excise taxes on 100LL vary considerably from state to state and may also affect the costs to a consumer. For example, both Oregon and Idaho have a lower state aviation fuel tax compared to neighboring Washington State. An FBO manager told us that in some cases, a pilot will fly over to Idaho to obtain less expensive fuel, even though he or she may base the aircraft in Washington. Security. Some airports—particularly those with commercial service—are responsible for implementing security requirements in accordance with their Transportation Security Administration (TSA)- approved security programs, notably the security of perimeters and access controls protecting restricted areas of the airport, such as ramps and taxiways. We found that some FBOs are responsible for security and access controls on their leased property based on our review of individual lease requirements and the airport security plan. These FBOs might require staff on site 24 hours a day to maintain airfield and perimeter security, a requirement that can increase FBO costs. For example, an FBO operating at an airport with commercial service told us that it is responsible for perimeter security on the land it leases from the airport. In addition, it is subject to unannounced security checks by TSA. In contrast, smaller general aviation airports without commercial service are not required to have as many security requirements. Selected stakeholders told us that the location of an airport may influence demand for FBO services. Economic theory indicates that increased demand for a service will generally result in increased prices, all else equal. In particular, stakeholders cited the following examples of demand factors that may influence prices: Busy and congested airports may have higher prices for FBO services due to greater demand. Prices may be higher during part of the year in locations with significant seasonal traffic, such as beach resorts with a summer high season and ski resorts with a winter high season. The increased demand at FBOs during high seasons results in higher prices than during the off season. An airport’s proximity to the central business district may be associated with higher demand and higher prices in such locations. In addition to cost and demand factors, the extent to which the market for FBO services is competitive may also influence prices. According to the stakeholders we interviewed, competition among FBOs may lead to lower prices than would be the case when only one FBO provides the service at that airport. In our analysis of FAA airport and FBO data, however, we found that nearly 90 percent of NPIAS airports that offer FBO services are served by only one FBO (see table 1). According to a Transportation Research Board report, a strong indicator of the number of FBOs that can be financially viable at an airport can be the amount of fuel sales. For example, two airport managers we spoke to said that there was an insufficient volume of fuel sold at their airports to support more than one FBO. While the majority of NPIAS airports in the contiguous United States have only one FBO, pilots we spoke to said that competition from FBOs at nearby airports can also affect prices. For example, within 30 miles of Spokane International Airport, there are five other airports, each of which is served by an FBO that may compete with the services provided at Spokane International. (See fig. 4) We asked selected managers of FBOs and airports and selected general aviation pilots to describe how off-airport competition may influence FBO pricing. FBO and airport managers told us that they view nearby airports as competitors and monitor the FBO prices at these locations. For example, an FBO manager in Maine told us he regularly checks the prices at the larger international airport that is nearby. This finding suggests that when an FBO sets its prices, it takes into account the extent to which nearby airports may compete for its services. On the buyer’s side of the market, 11 of the 18 general aviation pilots we interviewed told us that they generally “price shop” for aviation fuel. Further, most general aviation pilots we spoke with told us they use online flight-planning tools to map their route and consider the fuel cost and service fees of the airports along that route. Further, four pilots and an FBO manager indicated that on longer trips that require refueling before reaching a destination, pilots may have options that are hundreds of miles from each other. For example, when flying from California to Texas, a pilot could choose to stop either in New Mexico or Arizona to obtain fuel. In this scenario, a pilot would compare prices of many FBOs in those two states and likely choose one with lower fuel prices. Likewise, an FBO manager in Kansas indicated that for these types of customers, he competes with FBOs at airports more than 100 miles away. However, we interviewed some pilots who said that they do not consider every nearby airport as a substitute. To be a true substitute the airport must meet the pilot’s needs to be a viable option. For example, the airport’s runway must be of sufficient length for the aircraft, and some runways may be too short for certain aircraft. Also, pilots take into account the type of fuel offered at an FBO. The pilot of a piston-driven aircraft will be unable to refuel at an FBO that offers only jet fuel. Finally, some pilots said the price differential would need to be sufficiently large to compensate them for any inconvenience. Some mentioned that the price of 100LL would have to be 30 to 40 cents per gallon lower to affect their flight plan, while others put that threshold at a lower point, 25 to 30 cents per gallon. Pilots also told us they take travel time into account. For example, some pilots said that they would consider landing at an alternative airport with lower prices if it were no more than 20 to 30 miles out of their way and if the change in destination were to add no more than 10 to 20 minutes to their trip. As a first step in examining the relationship of FBO competition with pricing, we examined differences in the average posted prices for 100LL and Jet A across NPIAS airports in the contiguous United States where only one FBO sold a fuel type compared to airports at which more than one FBO sold that fuel, without controlling for other factors that might also be correlated with prices. We also calculated the average prices at airports with one FBO and airports with multiple FBOs for a subset of airports with an air traffic control tower. We examined this subset of towered-airports, as they generally have more operations, and thus more demand. As shown in table 2 below, the average price per gallon of aviation fuel was lower at airports with only one FBO than at airports with on-airport competition. For example, at airports with only one FBO, the average price posted for full-service 100LL was $5.01 per gallon, while the average price posted at airports with more than one FBO was about 73 cents higher. However, examining average differences in price fails to control for other factors that might be correlated with prices. In particular, airports that have more than one FBO are likely to be those that have higher traffic volumes and that are located in areas with larger populations and higher per-capita incomes—all factors likely correlated with higher prices. Therefore, to more fully assess the issue, we developed a statistical model that examines how fuel prices may be correlated with measures of competition when controlling for other factors, such as demand, that also may be correlated with prices. Our statistical model confirmed a correlation between selected cost and demand factors and FBO-posted pricing of full-service 100LL and Jet A. It also confirmed a correlation between some of the competition factors described by stakeholders and the price of aviation fuel. Our analysis included information on posted prices for both 100LL as well as Jet A. In addition to running the model for NPIAS airports in the contiguous United States for which posted prices were available (all-airports), we also ran the model for a subset of these airports that have air traffic control towers (towered-airports). See appendix II for a more detailed discussion of the model structure and findings. As we have noted, we expected fuel prices to be correlated with a variety of cost, demand, and competition factors that pertain to characteristics of airports and their locations, as well as characteristics of FBOs operating at airports. We found the following correlations: Airport Characteristics. Our model found that the size of an airport—measured as the total number of operations—was associated with higher prices for both 100LL and Jet A. The operational size of an airport is likely associated with higher demand for airport services and also is likely related to higher costs of providing those services. Specifically, we found that an increase of 10,000 airport operations per year was associated with higher prices of about 2 cents per gallon for both 100LL and Jet A in both the all-airports and towered-airports datasets. The length of the longest runway available at an airport was also correlated with higher fuel prices. The length of the runway is an indicator of the types of aircraft an airport can support. In particular, longer runways are able to accommodate larger and heavier aircraft— aircraft that generally use more fuel and thus may indicate higher demand for fuel at the airport. Specifically, we found that a 1,000-foot increase in runway length was associated with a higher price of about 7 to 8 cents per gallon for both fuels. Demographic Characteristics of Airport Location. Our analysis found that FBOs’ fuel prices were generally higher at airports in areas with higher incomes, but not always at airports in areas with larger population. We found that prices for both types of fuel were higher at airports located in counties with higher per-capita incomes. Where incomes are higher, we would expect the demand for travel to be greater. Moreover, where there are higher incomes, the cost of providing FBO services—particularly labor costs—are likely higher. We found income correlated with fuel prices for both the 100LL and Jet A in the all-airports datasets and for 100LL in the towered-only airport dataset. We also found that 100LL aviation fuel prices were higher at airports located in counties with larger populations. This was expected due to the likely greater demand for air travel in more populous areas. However, county population was not statistically significant in relation to the price of Jet A. Geographic Characteristics of Airport. Our model found that airports located in states on the East Coast tend to have higher 100LL prices. Specifically, the model suggests that 100LL prices are between 22 to 26 cents higher per gallon on average in these states. We expected FBOs operating in East Coast states to have higher 100LL prices due to higher transportation costs, as we found that there is no production of 100LL in these states. As mentioned earlier, 100LL is generally moved by truck, rail or barge due to the smaller volumes being produced—a less cost-effective means of transport than pipeline. Jet A, on the other hand, is transported by pipeline over long distances. We thus expected higher prices for 100LL at airports in these states. We found this geographic differential in all specifications for the 100LL model. Large-Chain FBOs. Our model found that both types of fuel tend to have higher posted prices at airports that have a large-chain FBO operating on the premises, regardless of whether or not there was another competitor on the premises. Specifically we found that when a large-chain FBO operates at an airport, fuel at the airport tends to be more expensive on average—on the order of 60 cents more per gallon for 100LL, and an even greater differential for Jet A, more than $1.20 per gallon. Availability of Self-Serve 100LL fuel. Our model found that when self-serve 100LL is available at an airport, the prices for full-serve 100LL tend to be lower than at airports with no self-serve 100LL available. Specifically, we found that if a self-serve 100LL option is available at an airport, the price of full-service 100LL will be about 10.5 cents per gallon lower, on average, compared to FBOs at airports without self-service 100LL. We expected a self-service option might be correlated with somewhat lower prices for full service 100LL—even if the full service option is provided by the same FBO— because pilots are presented with a lower price option may constrain the prices that FBOs will charge for a full-service option. Competition. Within our statistical model, we examined whether the extent of competition among FBOs had a correlation with fuel pricing in two ways. On-airport competition. On-airport competition occurs when two or more FBOs at an airport sell the same kind of fuel. We estimated that the price of Jet A is lower, on average, at an airport when two or more FBOs provided that fuel at an airport. Specifically, for the all-airports dataset, we found that, on average, the posted price of Jet A was 35 cents per gallon higher if only one FBO sold the fuel at an airport compared to the case when at least one additional competitor also served the airport. In the towered-airports dataset, the posted price of Jet A was about 50 cents higher on average if there were only one FBO at the airport. For 100LL, we did not find a statistical relationship between on- airport competition and prices in the all-airports dataset; however, we did find a statistical relationship between on-airport competition and prices in the towered-airports dataset. The finding of no correlation between on-airport competition and 100LL prices may be linked to the rarity of airports with more than one FBO selling 100LL in the all-airports dataset. In fact, in the all-airports dataset, only 13 percent of FBOs faced on-airport competition in the sale of 100LL while in the towered-airports dataset about one-third of FBOs faced competition in the sale of 100LL. Specifically, we estimated that the price of 100LL is 11 cents lower, on average, if there are at least two FBOs selling that fuel at a towered airport. Nearby competition. Our model also tested whether the availability of additional FBOs at airports within a 30-mile distance from a given airport had any correlation to prices for 100LL. We included this factor because many of the stakeholders we spoke to noted that general aviation pilots will consider using an airport near their preferred airport if prices were more favorable at the alternative location. However, across all model specifications, we did not find that prices for 100LL were correlated with the presence of FBOs at nearby airports. FAA officials told us they primarily rely on airports to self-certify their compliance with federal airport grant assurances when they accept AIP grant funding. This reliance includes the grant assurance that relates to FBO fees—an airport must ensure aeronautical services are available to all users on a reasonable and not unjustly discriminatory basis. FAA officials indicated that airport compliance staff conduct outreach to stakeholders and provide training aimed at ensuring that airports comply with these assurances. One recent outreach effort focused on FBO pricing. Additionally, FAA responds to phone and email inquiries and informal and formal complaints, and conducts periodic airport land use inspections, as discussed below, but none of these efforts has identified FBO pricing as a widespread area of concern. Training and Outreach. According to FAA, compliance staff conducts periodic training and outreach to the airport community on a variety of compliance issues. FAA headquarters annually conducts recurrent compliance training—which includes overseeing airport grant assurances—with regional and other FAA offices. FAA officials told us they use these sessions to address concerns brought up by regional compliance officials and airport compliance staff. One example of FAA’s outreach efforts occurred in December 2017 after AOPA raised questions about FBO pricing earlier that year. To bring clarity to the issue of FBO pricing and the role of FAA, the agency released questions and answers that emphasized: (1) FAA does not regulate FBO prices; and (2) airports are responsible for ensuring FBO prices are reasonable and applied in a non-unjustly discriminatory manner. Furthermore, FAA stated that whether an FBO’s fees are reasonable (i.e., higher than average than other FBOs) involves a number of economic, business, and other factors that vary widely from airport to airport and FBO to FBO and may include underlying costs, market conditions, quality of service, and other factors. Inquiries and Complaints. According to FAA, airport compliance staff respond to (1) phone and e-mail inquiries, (2) informal complaints, and (3) formal complaints. FAA officials told us that, while FAA does not regulate FBO prices, if someone contacts them with inquiries or a concern about a potential grant assurance violation, such as one involving FBO prices, they first refer the issue to the local airport to resolve. If the issue is not resolved, the complainant may file an informal complaint with an FAA regional office. According to FAA guidance, each FAA regional office will review the complaint and issue a letter indicating whether FAA sees a grant violation that the airport should fix or not. If the complainant is dissatisfied with the regional office’s letter, the complainant may then file a formal complaint about a violation of grant assurances with headquarters. Headquarters will then review the circumstances of the complaint and make a formal determination as to whether a grant violation occurred and work with the airport to address the violation. Data on informal and formal complaints filed with FAA headquarters and regional offices indicate FAA has not received many complaints on FBO pricing. Specifically, we reviewed informal complaint data from 2013 through 2018 from each FAA region, and found a total of 142 informal complaints about potential grant violations. Seven of these complaints related to FBO prices, and FAA found one violation later resolved by the airport by providing space for aircraft to do routine maintenance. In addition, we obtained and reviewed FAA’s responses to formal complaints from 2013 through 2018, and found that none of these formal complaint responses dealt with FBO prices. While FAA received few complaints related to FBO prices, there are limitations in relying on complaint data to understand the magnitude of an issue. For example, some pilots we spoke with stated if they have an issue with an FBO, they will use an alternative FBO rather than submit a complaint to FAA. We found that in addition to informal and formal complaints, each FAA regional office independently records inquiries about airport grant assurance issues ranging from inappropriate hanger use to noise complaints to FBO lease arrangements. Further, each region varies in the way it captures airport compliance information such as airport location, dates, and description of an inquiry or concern. For example, some regions indicate the specific grant assurance that was potentially violated while others simply describe the nature of the concern. To help see if there is a pattern of concerns across the country, FAA’s Office of Airport Compliance in headquarters has an initiative to centralize information on inquiries and concerns about grant assurances, including any that may be related to FBO prices. As envisioned, this “Enhanced Information Sharing Initiative” will provide FAA compliance staff with the ability to record and track inquires and complaints in comparable systems and should facilitate information sharing among regions and between regions and headquarters. According to FAA officials, centralizing this information will help identify issues that may be of a concern to airport users. According to FAA officials, this initiative, originally planned to be completed in August 2019, has faced delays due to a government shutdown earlier this year and information technology security difficulties. However, FAA has hired a new contractor and anticipates completion sometime in fiscal year 2020. According to an FAA compliance manager, problems that arise in the regions are brought to the attention of the airport compliance offices and discussed, and whether additional actions should be taken. Airport Land-Use Inspections. FAA is required to conduct a minimum of two airport land-use inspections per year per region, reviewing whether airports are complying with grant assurances such as airport property use requirements and lease agreements. We reviewed FAA’s annual land use inspection reports to Congress from fiscal year 2013 through 2018 and did not identify any FBO pricing concerns. We provided a draft of this report to DOJ and DOT for review and comment. DOJ provided technical comments, which we incorporated as appropriate. DOJ also suggested that we discuss more directly the implications that airport ownership of FBOs might have for fuel prices. We agree that airport-owned FBOs might price fuel differently than privately- owned FBOs. However, we were not able to obtain reliable data on airport ownership of FBOs. DOT did not have any comments. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, the Attorney General, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov If you or your staff any have questions about this report, please contact me at (202) 512-2834 or VonahA@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This appendix describes a model we developed to assess factors that may correlate with fixed base operator (FBO) aviation fuel prices across airports. The model uses data on posted prices for full-service 100LL aviation fuel (100LL) and Jet A (Jet A) fuel at a sample of airports in the contiguous United States that are part of the National Plan of Integrated Airport Systems (NPIAS), along with data on selected other factors that may be correlated with fuel prices. Specifically, this appendix discusses (1) the structure of the model, data sources, and variable definitions, and (2) base-case and alternative model results. Based on our audit work as well as economic reasoning, we hypothesized that a variety of factors may be correlated with aviation fuel prices across airports. Generally, factors that influence the price of any product are the demand for the product, the cost of producing and marketing the product, and the extent of competition among those selling the product. To examine the correlation between these factors and the price of both 100LL and Jet A fuel sold by FBOs, we developed an econometric model. Specifically, our model analyzed the independent correlation of selected key factors with aviation fuel prices. We used several specifications of the model for both full-service 100LL and full-service Jet A. Each specification used airport-level data to analyze variation in the price of a single type of fuel across airports. For each type of fuel, we included only NPIAS airports within the contiguous United States for which our data on aviation fuel prices reported a price for at least one FBO. In addition, we ran the analysis not only on the full dataset of all of the airports for which we were able to obtain fuel-pricing information (which we refer to as the all-airports dataset), but also on a subset of airports limited to those with an air traffic control tower (towered-airports dataset). For each type of aviation fuel, the dependent variable—or the variable to be explained in the model—is the average price of that fuel at an airport, net of state taxes. If an airport has only one FBO selling a fuel, the average price of that fuel at that airport is simply the price charged by the FBO that sells it. At an airport where two or more FBOs compete to sell the same type of fuel, the average price of the fuel is calculated as a simple (unweighted) average across all of the FBOs that sell the fuel at the airport. For 100LL, about 87 percent of airports in the all-airports dataset are served by only one FBO, and for Jet A, the share is lower, at about 84 percent. We obtained data on posted aviation fuel prices from a company that publishes such data online for two separate dates—a Wednesday in October of 2018 and a Wednesday in May of 2019. All of the fuel price data we received had been updated within 30 days of these dates. Independent variables in our model included a variety of demand, cost, and competitive factors that we hypothesized may explain the variation in fuel prices across airports. In particular, these factors relate to characteristics of (1) airports, (2) the locations where an airport resides, (3) the FBOs operating at a given airport, and (4) the availability of competing FBOs. Characteristics of airports. We expected certain characteristics of each airport to be related to the level of fuel prices. Airport size. We measured airport size based on the number of total operations—takeoffs and landings—at the airport. Greater activity at an airport reflects higher demand for services, which we expected to correlate with higher prices. Moreover, it is likely more costly to provide services at these busier airports. As such, based on both demand and cost factors, we expected larger airports to have higher fuel prices. We obtained data on airport operations from the Federal Aviation Administration (FAA). Length of the longest runway. Longer runways can accommodate larger and heavier aircraft, which may increase demand of such traffic. Because larger and heavier aircraft require more fuel, a longer runway may be indicative of greater demand for fuel at the airport. At the same time, longer runways are more costly to construct and maintain. Thus both demand and supply factors related to having a longer runway would suggest that fuel prices could be higher at such airports. We obtained information on runway length, which we measure in thousands of feet, from FAA. Characteristics of locations. We also expected demographic and geographic characteristics of the location of each airport to be correlated with fuel prices. Demographic characteristics of the population living in the area near an airport. Personal income per capita. Areas where per-capita incomes are higher could signal a greater demand for air travel and airport services. At the same time, areas with higher per-capita incomes also suggest that costs for labor and other resources the FBO will need to procure will be higher. Thus, we hypothesize that airports located in counties with higher per-capita incomes will have higher fuel prices. We obtained data on personal income per capita by county from the Bureau of Economic Analysis, Department of Commerce. Square of per-capita income. We also expected that, as income levels rise, the effect of even higher levels of income on fuel prices will attenuate. To account for the possibility of a nonlinear relationship between income and fuel prices, we included a variable equal to the square of personal income per capita. Population. A larger population in the area surrounding an airport would likely indicate higher demand for airport services. We obtained population data by county from the Bureau of Economic Analysis, Department of Commerce. Distance from Source of 100LL aviation fuel production. Following production, 100LL is typically shipped over longer distances by truck, rail, or barge, while Jet A tends to be transported over longer distances by pipeline. As such, long-haul transport is relatively more costly for 100LL. We found that there is no production of 100LL in East Coast states, while most other states in the contiguous United States have production sources for 100LL in closer proximity. Therefore, we controlled for the greater cost of transporting 100LL to states along the East Coast with a dummy variable in the 100LL pricing model. We obtained information on production sources for 100LL from the Energy Information Administration, Department of Energy. Characteristics of FBOs. We included two variables in the model that relate to the services provided by FBOs at the airport Large-Chain FBOs. Based on our audit work, we hypothesized that large-chain FBOs—those with operations at numerous airports—are more likely to focus their business model on meeting the demands of pilots looking for a suite of services and amenities. We thus expected that an airport served by a large-chain FBO may have higher average fuel prices due to the costs of providing such services. We used the data on aviation fuel prices to determine the number of operations run by each FBO. For purposes of the model, we defined an FBO as a large-chain if the owner had at least 25 FBO operations across airports reported in our dataset. Availability of self-service fuel at airport. We hypothesized that the price for full-service 100LL might be lower at an airport where a self- service 100LL is also offered for sale. That is, the ready availability of a cheaper fueling option may influence the pricing of full-service 100LL. Therefore, we included a dummy variable in the 100LL pricing model if self-service 100LL was also available at the airport. In many cases, only one FBO is available at an airport and provides both self- service and full-service 100LL. The variable is derived from the data on aviation fuel prices. Degree of competition among FBOs. Economic theory suggests that market prices for a product will be lower when more firms are selling a product, all else equal. We examined “on-airport” competition among FBOs for both fuels, and for 100LL, we also developed a variable to account for competition at nearby airports. The number of on-airport FBOs selling a given fuel. The most immediate and likely relevant competition among FBOs would occur at a given airport. To examine the correlation between on-airport competition and aviation fuel prices, we used two alternative measures: (1) the number of FBOs selling the fuel at each airport and (2) a dummy variable that equals 1 for airports where more than one FBO sells the fuel and 0 otherwise. These competition measures were derived from the data on aviation fuel prices. Availability of alternative FBOs at airports in the vicinity. Because stakeholders we interviewed said that pilots using 100LL may consider using nearby airports where that fuel is less expensive rather than their intended destination airport, we examined whether the availability of FBO services at nearby airports correlated with 100LL prices. Specifically, we counted the number of different FBOs selling 100LL at airports within a 30-mile radius of each airport where 100LL is sold. We derived this measure of competition from nearby airports by combining geospatial data for each airport with information from our data on aviation fuel prices. As noted, we ran the fuel-pricing model for both 100LL and Jet A aviation fuels. Table 4 provides descriptive statistics for all of the variables included in the models. We report regression results for several specifications in tables 5–9. Specifically these tables provide the extent and direction (plus or minus) of the estimated correlation of each of the independent variables on aviation fuel prices. We also indicate whether each estimated correlation is statistically different from zero. The per-gallon price of aviation fuel (100LL and Jet A)—the dependent variable in our model—is measured in dollars and cents. Some of the independent variables are measured in levels—for example, annual airport operations are measured in tens of thousands, and the length of the longest runway in thousands of feet. For these variables, the regression model results indicate the estimated correlation of a one-unit increase in the level of the independent variable on the price of aviation fuel. For example, as shown in table 4, an increase in runway length of 1,000 feet is associated with an increase in the price of both 100LL and Jet A of about 8 cents, and this estimated correlation is statistically different from zero at the 1 percent level. The model also includes some “dummy” variables—variables that take a value of either 1 or 0, depending on whether a specific attribute does or does not apply. For a dummy variable, the estimated correlation is interpreted as the effect of the attribute on the per-gallon fuel price. Based on the findings in table 5, being located on the East Coast is associated with an increase in the price per gallon of 100LL fuel of about 22 cents. This correlation was also found to be statistically different from zero. In another example, the model specification shown on table 5 uses a dummy variable to indicate the presence of competition at an airport—the variable equals 1 for airports that are served by more than one FBO and 0 for airports that are served by only one FBO. Results in table 5 indicate that the price per gallon of Jet A fuel is about 35 cents lower at airports that are served by more than one FBO than at airports with only one FBO. In addition to the individual named above, Cathy Cowell (Assistant Director); Nick Nadarski (Analyst-in-Charge); Amy Abramowitz; Dave Hooper; Christopher Jones; Ned Malone; Malika Rice; Ardith Spence; and Michelle Weathers; made key contributions to this report.", "summary": "Since 2007, the FAA has provided more than $37 billion in grants to airports to fund capital development and is responsible for ensuring compliance with requirements airports assume when they accept these grants. One such requirement is that the airports provide users equal access to airport services such as fueling and parking. Recently, an industry group and pilots raised concerns about the transparency and reasonableness of prices charged for these and other services at airports. GAO was asked to examine FBOs' pricing and FAA's oversight of related airport grant assurances. This report examines: (1) the transparency of FBO prices, (2) the factors that influence prices, and (3) the extent to which FAA ensures compliance with federal airport grant assurances related to FBO activities. GAO analyzed FAA data related to complaints from 2013 through 2018 and reviewed relevant literature, key laws and regulations, and program documentation. GAO developed a statistical model to analyze variation in fuel prices across airports in the contiguous United States. GAO interviewed FAA compliance staff at headquarters and all regional offices, as well as a non-probability selection of stakeholders. Fixed base operators (FBO) at airports (see figure) offer a variety of services to pilots and passengers. While anyone can view fuel prices offered by FBOs online, other service fees, such as for aircraft parking, can vary by type of aircraft and are not always available online, although they can be obtained by calling the FBO. Recently, industry groups developed the “Know Before You Go” campaign that calls for greater transparency of FBO prices. Some of the FBOs GAO interviewed list their fees online; however, others do not. Stakeholders GAO interviewed––including general aviation pilots, airports, FBOs, and industry groups––said FBOs' costs to build and maintain facilities—such as hangars and fueling facilities—as well as operating expenses such as labor and fuel––influence their prices. Stakeholders also said that demand for FBOs' services can influence prices, such as when seasonal demand affects operations at an airport near a ski resort. Finally, they also said that competition affects FBO's prices. GAO's statistical model confirmed a correlation between many cost and demand factors and aviation fuel prices and found higher prices at airports with higher costs and demand. This model also found that on-airport competition is associated with lower prices at the country's busiest airports: Prices for aviation fuels were lower at such airports with more than one FBO. However, not all airports can support more than one FBO due to, for example, the amount of business each gets. Airports receiving Federal Aviation Administration (FAA) grants must meet “grant assurances” such as charging reasonable and not unjustly discriminatory prices for services, including prices charged by FBOs. FAA officials said FAA oversight relies on (1) airports' consent to adhere to grant assurances; (2) training and outreach; and (3) complaints. Since 2013, in complaints received by FAA, GAO found few complaints about FBOs' prices. GAO found each regional office independently records additional inquiries. FAA is moving to collect regional inquires centrally, and by 2020 that step may allow FAA to stay abreast of apparent nationwide trends or issues with any grant assurance concerns.", "document_type": "gao"}
{"report": "The Food and Nutrition Service at USDA is responsible for overseeing the school meals programs at the federal level, which includes issuing regulations and guidance. The school meals programs are administered at the state level by a designated state agency—generally an education or agriculture agency—that issues guidance to school districts providing the meals. School districts are responsible for certifying students as eligible for free or reduced-price meals, providing children with nutritionally balanced meals each school day, and counting and claiming eligible meals for federal reimbursement, among other things. USDA and state agencies also conduct oversight of the school meals programs. Figure 1 summarizes the responsibilities of the different entities within the school meals programs. All students in schools operating the school meals programs may participate in the programs, and eligible students may be certified to receive free or reduced-price meals. Individual students can be certified into the school meals programs either through household application or direct certification. Household application. A household can submit an application that lists all sources of household income and the names of all household members, among other information. School districts compare this information to income-eligibility guidelines to determine whether the student is eligible for free or reduced-price meals. Alternatively, the household can indicate on the application that it participates in certain public-assistance programs—such as the Supplemental Nutrition Assistance Program (SNAP)—or that the student meets an approved designation, which confers categorical (automatic) eligibility for free meals. No documentation to support income listings—such as tax returns or pay stubs—is required at the time of application. Direct certification. Under the direct certification method, data matching is used to identify and certify students who are categorically eligible for free meals. State agencies are required by statute to match student enrollment records against SNAP records and may also match against records for other public-assistance programs or approved designations. Students who are directly certified into the school meals programs are eligible for free meals without a household application. Alternatively, schools can use certain program provisions to serve meals at no charge to all students (i.e., eligibility is not determined for each student individually on an annual basis). Community eligibility provision. Schools and school districts may apply for community eligibility if their percentage of students identified as eligible for free meals without an application—known as the identified student percentage—is at or above 40 percent. Meals served at schools using the community eligibility provision are reimbursed using a formula based on the identified student percentage. Other special provisions. Under these special provisions, schools generally use standard procedures to certify free and reduced-price eligible students and count meals by eligibility category to establish a base year. Following the base year, schools serve free meals to all students and are reimbursed based on the information collected in the base year. USDA reimburses state agencies, which in turn reimburse school districts, for qualifying meals through the process of meal counting and claiming. A meal is reimbursable if it meets federal nutrition requirements and is not reimbursable if it is missing a required food component or fails to meet the meal pattern requirements. Meals are recorded at the point of sale in a school. Generally, individual meals are recorded as either free, reduced price, or paid based on the student’s certification status, unless the school is operating under community eligibility or a special provision. Reimbursable meals are tallied at each school. School districts aggregate meal tallies from each school and then report to the respective state agencies on a monthly basis. State agencies then aggregate the reports from each district and submit tallies of free, reduced-price, and paid meals to USDA. USDA then reimburses states for the amount reported. Meals in each category (free, reduced-price, and paid) are reimbursed at different rates. Given that the school meals programs are administered on a daily basis at schools across the country, USDA officials stated that the agency relies on two key oversight practices—management evaluations and administrative reviews—to monitor these programs. Management evaluations. USDA conducts management evaluations of state agencies’ administration of the school meals programs. USDA uses risk-based criteria, such as the level of turnover in state agency staff, to select the state agencies to review each year. According to USDA officials, starting in fiscal year 2019, USDA will automatically select a state agency for review if it has not been reviewed in the past 3 years. According to USDA guidance, examples of operations it reviews during management evaluations include (1) state oversight of certification and verification of students into the school meals programs, (2) administrative reviews of school districts conducted by state agencies, (3) claims for reimbursement, and (4) state oversight of compliance with federal meal pattern requirements, among other areas. According to USDA officials, if USDA considers state agencies reviewed in one year as high risk for program noncompliance, those agencies may receive an additional management evaluation in the following year focused on technical assistance. Administrative reviews. USDA develops guidance for administrative reviews in which state agencies review school districts’ administration of the school meals programs. State agencies are required to conduct administrative reviews of each of their school districts at least once in a 3-year review cycle. USDA guidance states that the objectives of administrative reviews include identifying noncompliance, providing technical assistance, and assessing fiscal actions. Among other things, state agency staff are to review a school district’s certification records and its meal counting and claiming data for the most recent month for which a claim for reimbursement was submitted. State agency staff are also to review school meals served while the staff are on-site to determine whether the meals contain the required food components. State agency staff are to record any identified noncompliance and also provide technical assistance to school district staff. The Improper Payments Information Act of 2002 (IPIA), as amended, requires agencies to identify, estimate, and report their improper payment amounts and to develop and implement improper payment reduction plans, among other things. USDA estimates improper payments for the school meals programs through a model based on its APEC study, which is conducted by contractors. The most recent APEC study (APEC II) was released in May 2015 and covered activity during the 2012–2013 school year. USDA conducts an APEC study about every 5 years, with APEC III expected to be released in 2020. Conducting the APEC study involves multiple sampling and data analysis efforts, including the following examples from APEC II. In-person surveys. Contractors conducted in-person surveys of over 3,000 sampled households to collect information on each household’s circumstances at the time of application, including income, household size, and receipt of other public-assistance benefits. Using this information, contractors determined a sampled student’s eligibility status and compared it to the school district’s master benefit list and application or direct certification documentation. Data matching. Contractors assessed the accuracy of the identified student percentage—the figure used to determine reimbursement for schools using the community eligibility provision—for over 100 sampled schools. To do so, contactors used an iterative process to match sampled students to SNAP and Temporary Assistance for Needy Families records, as well as additional data sources if necessary. Observation of meal service. Contractors observed approximately 25,000 lunch transactions and 23,000 breakfast transactions at over 400 sampled schools to identify the food items in each meal at the point of sale, whether the meal was served to a student or nonstudent, and whether the meal was recorded as reimbursable. USDA determined that conducting the APEC study annually would not be feasible. Instead, the APEC study includes a model that allows USDA to use program participation data to report estimated improper payment error rates on an annual basis. Changes in program participation data result in small changes to the estimated improper payment error rates USDA reports during years between APEC studies. According to federal standards and guidance, executive-branch agency managers are responsible for managing fraud risks and implementing practices for combating those risks. Federal internal control standards call for agency management officials to assess the internal and external risks their entities face as they seek to achieve their objectives. The standards state that as part of this overall assessment, management should consider the potential for fraud when identifying, analyzing, and responding to risks. In July 2015, GAO issued the Fraud Risk Framework, which provides a comprehensive set of key components and leading practices that serve as a guide for agency managers to use when developing efforts to combat fraud in a strategic, risk-based way. The Fraud Risk Framework describes leading practices in four components, as shown in figure 2. The Fraud Reduction and Data Analytics Act of 2015, enacted in June 2016, requires the Office of Management and Budget (OMB) to establish guidelines for federal agencies to create controls to identify and assess fraud risks and design and implement antifraud control activities. The act further requires OMB to incorporate the leading practices from the Fraud Risk Framework in the guidelines. In July 2016, OMB published guidance about enterprise risk management and internal controls in federal executive departments and agencies. Among other things, this guidance affirms that managers should adhere to the leading practices identified in the Fraud Risk Framework. Further, the act requires federal agencies to submit to Congress a progress report each year for 3 consecutive years on the implementation of controls established under OMB guidance, among other things. It is important to note that fraud and “fraud risk” are distinct concepts. Fraud is challenging to detect because of its deceptive nature. Additionally, once suspected fraud is identified, alleged fraud cases may be prosecuted. If the court determines that fraud took place, then fraudulent spending may be recovered. Fraud risk exists when individuals have an opportunity to engage in fraudulent activity, have an incentive or are under pressure to commit fraud, or are able to rationalize committing fraud. When fraud risks can be identified and mitigated, fraud may be less likely to occur. Although the occurrence of fraud indicates there is a fraud risk, a fraud risk can exist even if fraud has not yet been identified or occurred. For example, suspicious billing patterns or complexities in program design may indicate a risk of fraud. Information to help identify potential fraud risks may come from various sources, including whistleblowers, agency officials, contractors, law-enforcement agencies, or beneficiaries. USDA has reported various actions aimed at lowering the school meals improper payment error rates in its agency financial reports. These actions—including onetime actions and longer-term efforts—cover multiple aspects of the programs. The actions USDA reported included the creation of a new household application prototype intended to reduce applicant errors and the development of training for food service workers to address administrative errors. USDA also reported on mechanisms to collect information on program errors to support agency analysis and monitoring efforts. Examples of the reported actions are illustrated in figure 3 below. Because the study used to develop improper payment error rates in school meals programs—APEC—is conducted about once every 5 years, the effect of these actions is currently unknown. Estimated improper payment error rates reported in years between APEC studies will generally not reflect the effect of most actions until the next study is released. Currently, the next APEC study is expected to be released in 2020. USDA changed what it considers to be an improper payment in the school meals programs for fiscal year 2018 reporting, resulting in improper payment error rates that are not comparable to those of prior years. Specifically, USDA determined that meal claiming errors do not meet the definition of an improper payment. According to USDA, meal claiming errors occur when meals are incorrectly categorized as reimbursable or nonreimbursable at the point of sale. For example, a meal claiming error occurs when a meal that is missing a required meal component (e.g., the required quantity of a vegetable) is counted as reimbursable. USDA officials reported that the rationale for the change in what constitutes an improper payment is that meal claiming error does not result in the payment of federal funds for services that were not provided or that were provided to ineligible recipients. Agency officials also stated that the remedy for meal claiming error is to add the missing food component to the meal, so correcting the error would not reduce program payments. Although the errors will not be considered in determining the reported estimated improper payment error rates, USDA officials stated that the agency is committed to reducing meal claiming error and will continue to measure it as part of its periodic APEC studies. Prior to fiscal year 2018 reporting, meal claiming errors were considered improper payments. As a result, this change contributed to a significant decrease in the estimated improper payment error rates for the school meals programs reported for fiscal year 2018, as shown in figure 4. Accordingly, the results shown for 2015 through 2018 in figure 4 are not comparable. Although USDA considers certain program integrity risks through specific processes, it has not assessed fraud risks in the school meals programs. As a result, USDA cannot determine whether its key oversight processes—extensive efforts designed for broad monitoring purposes— address areas at risk for fraud. The assess component of the Fraud Risk Framework calls for federal managers to plan regular fraud risk assessments and to assess risks to determine a fraud risk profile. Furthermore, federal internal control standards state that management should consider the potential for fraud when identifying, analyzing, and responding to risks. According to USDA officials, fraud in the school meals programs would look the same as nonfraudulent errors. For example, income listed on an application may be misreported intentionally or unintentionally. Consequently, agency officials stated that they have not established a process to plan or conduct a specific fraud risk assessment for these programs. Instead, fraud risks are considered through the agency’s efforts to assess overall program integrity risk in the programs. We have previously reported that integrating fraud risk management into a larger program integrity approach could limit the amount of resources and attention focused specifically on fraud prevention, detection, and response. The deceptive nature of fraud makes it harder to detect than nonfraudulent errors, potentially requiring control activities that are specifically designed to prevent and detect criminal intent. According to agency officials, USDA’s efforts to assess overall program integrity risks in the school meals programs include researching, monitoring, and reporting activities designed to identify areas of program operations susceptible to improper payments and program error. Specifically, agency officials stated that these efforts include research projects—the APEC study used to estimate improper payment error rates and other smaller-scale, informal projects—and a consideration of specific risks when annually determining which states USDA will review through management evaluations. These efforts to assess overall program integrity risk serve specific purposes and are not designed to identify or address fraud risks in the school meals programs. For example, the purpose of one research project mentioned by USDA officials was to identify challenges related to alternative service models for the School Breakfast Program, which include serving breakfast in locations other than a cafeteria and at a later time in the morning. USDA has not developed a process to consider these disparate efforts to comprehensively assess fraud risks. As a result, USDA’s efforts do not align with the overarching concepts of planning and conducting fraud risk assessments in the Fraud Risk Framework. The Fraud Risk Framework identifies leading practices for planning fraud risk assessments. Specifically, the leading practices include tailoring the fraud risk assessment to the program and planning to conduct the assessment at regular intervals and when there are changes to the program or operating environment. The leading practices also include identifying the tools, methods, and sources for gathering information about fraud risks and involving relevant stakeholders in the assessment process. Information to help identify potential fraud risks may come from various sources, including whistleblowers, agency officials, contractors, law-enforcement agencies, or beneficiaries. Existing oversight efforts— such as USDA’s management evaluations and administrative reviews— may also be a useful source, as information on errors and noncompliance may highlight areas at risk for fraud. The Fraud Risk Framework also identifies leading practices for conducting fraud risk assessments, as illustrated in figure 5. Without a process to plan and conduct regular assessments, USDA cannot identify and assess fraud risks facing the school meals programs. Such information is necessary to appropriately design and implement an antifraud strategy—including specific controls like USDA’s key oversight processes—and evaluate and adapt its strategy and controls to improve fraud risk management in these programs. Historically, the school meals programs have reported high estimated improper payment error rates. USDA has reported various steps to reduce the error rates, though a change in what USDA considers an improper payment in the school meals programs resulted in error rates for fiscal year 2018 that are not comparable to those of prior years. Although the two concepts are different, high improper payment error rates may suggest that the school meals programs may also be inherently vulnerable to fraud. However, USDA has not established a process to plan and conduct regular fraud risk assessments for the school meals programs, and existing efforts to assess specific risks in the school meals programs do not comprehensively consider fraud risks. According to leading practices, such an assessment is a pivotal step in managing fraud risks, helping to ensure that USDA’s key oversight efforts are targeted at areas at greatest risk for fraud in these programs, and helping safeguard the government’s substantial investment in them. The Administrator of the Food and Nutrition Service should establish a process to plan and conduct regular fraud risk assessments for the school meals programs that align with the leading practices in the Fraud Risk Framework. (Recommendation 1) We provided a draft of this report to USDA for review and comment. On April 29, 2019, the Director of the Office of Program Integrity for Child Nutrition provided us with the agency’s oral comments on the draft report. FNS officials generally agreed with the recommendation in the draft report. FNS officials noted that the agency does not currently conduct a formal fraud risk assessment, but they explained that the agency considers fraud risks through multiple existing efforts. These efforts include APEC and other studies, as well as key oversight processes. For example, FNS noted that the APEC study aims to identify the factors that contribute to errors in the school meals programs. Officials explained that this study includes an interview of sampled households, in part to determine whether these households underreported income on their applications and whether such underreporting suggests anything about the applicants’ intent. As noted in our report, we agree that these efforts may be a useful source of information on areas at risk for fraud. However, we continue to believe that additional action is necessary to comprehensively assess fraud risks in the school meals programs, consistent with the Fraud Risk Framework. USDA also provided technical comments, which we incorporated into the report, as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Agriculture, the FNS Administrator, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6722 or bagdoyans@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the contact named above, the following staff members made key contributions to this report: Gabrielle M. Fagan, Assistant Director; James M. Healy, Analyst in Charge; and Matthew L. McKnight. Also contributing to this report were Rachel Frisk, Maria McMullen, Jean McSween, and Sabrina Streagle.", "summary": "In 2018, almost 30 million children participated in the National School Lunch Program and over 14 million participated in the School Breakfast Program, with cash payments totaling almost $17 billion. Historically, the school meals programs have reported high estimated improper payment error rates, which suggest that these programs may also be vulnerable to fraud. GAO was asked to review improper payment error rates and potential fraud in the school meals programs. This report (1) describes steps USDA has reported taking since 2015 to lower improper payment error rates and (2) examines the extent to which USDA has assessed areas of risk for fraud in the school meals programs. GAO reviewed the results of the most recent study USDA uses to estimate improper payments in the school meals programs, as well as the error rates and actions to reduce them reported in USDA's agency financial reports from fiscal years 2015 through 2018. Further, GAO analyzed guidance for key oversight practices and documentation regarding USDA's risk assessment processes. GAO examined these processes against the leading practices in the Fraud Risk Framework for assessing fraud risks. GAO also interviewed agency officials. The Department of Agriculture (USDA) has reported various actions aimed at lowering estimated improper payment error rates in the National School Lunch Program and School Breakfast Program (school meals programs). Examples include a new application prototype intended to reduce applicant errors and training for food service workers to reduce administrative errors. USDA uses a model based on a periodic study to estimate improper payments, and reported error rates will generally not reflect the effect of most actions until USDA's next study is released, likely in 2020. However, in fiscal year 2018, USDA redefined what it considers an improper payment. Specifically, meal claiming errors—for example, meals that are missing a required nutritional component but that are counted as reimbursable—are no longer considered improper payments, resulting in error rates for fiscal year 2018 that are not comparable to prior years. USDA has not assessed fraud risks in the school meals programs, which hinders its ability to ensure that its key oversight practices—extensive processes designed for broad monitoring purposes—address areas at risk for fraud. The assess component of A Framework for Managing Fraud Risks in Federal Programs (Fraud Risk Framework) calls for managers to plan regular fraud risk assessments and to assess risks to determine a fraud risk profile. USDA officials stated that the agency considers fraud risks through efforts to assess overall program integrity risk in the programs, which include research projects and consideration of specific risks when allocating monitoring resources. However, GAO found that USDA's efforts to assess risk do not comprehensively consider fraud risks. As a result, these efforts are not aligned with the overarching concepts of planning and conducting fraud risk assessments in the Fraud Risk Framework. Establishing a process to plan and conduct regular fraud risk assessments that align with the leading practices in the Fraud Risk Framework—including those in the figure below—will help USDA design and implement an antifraud strategy, as well as evaluate and adapt its strategy to improve fraud risk management in the school meals programs. GAO recommends that USDA establish a process to plan and conduct regular fraud risk assessments for the school meals programs that align with the leading practices in the Fraud Risk Framework. USDA generally agreed with the recommendation.", "document_type": "gao"}
{"report": "The Bureau’s address canvassing operation updates its address list and maps, which are the foundation of the decennial census. An accurate address list both identifies all living quarters that are to receive a notice by mail to respond to the census, and serves as the control mechanism for following up with households that fail to respond to the initial request. Precise maps are critical for counting the population in the proper locations—the basis of congressional apportionment and redistricting. Our prior work has shown that developing an accurate address list is challenging—in part because people can reside in unconventional dwellings, such as converted garages, basements, and other forms of “hidden” housing. For example, as shown in figure 1, what appears to be a single-family house could contain an apartment, as suggested by its two doorbells. During address canvassing, the Bureau verifies that its master address file and maps are accurate to ensure the tabulation for all housing units and group quarters is correct. For the 2010 Census, the address canvassing operation mobilized almost 150,000 field workers to canvass almost every street in the United States and Puerto Rico to update the Bureau’s address list and map data—and in 2012 reported the cost at nearly $450 million. The cost of going door to door in 2010, along with the emerging availability of imagery data, led the Bureau to explore an approach for 2020 address canvassing that would allow for fewer boots on the ground. To reduce costs for the 2020 Census, the Bureau took a new approach and some address canvassing work was completed in-office. The Bureau compared current satellite imagery to the contents of its master address file to determine if areas had housing changes, such as new residential developments or repurposed structures. If the satellite imagery and the master address file matched, then the Bureau considered those areas to be resolved or stable and did not canvass them in-field. These areas that were unresolved by the in-office review were sent to in- field address canvassing. Field staff called listers used laptop computers to compare what they saw on the ground to the address list and maps. Listers confirmed, added, and deleted addresses or moved addresses to their correct map positions. The listers were trained to speak with a knowledgeable resident at each housing unit to confirm or update address data, ask about additional units, confirm the housing unit location on the map (known as the map spot), and collect a map spot either using global positioning systems (GPS) or manually. If no one was available, listers were to use house numbers and street signs to verify the address data. The data were then transmitted electronically to the Bureau. The Bureau completed in-field address canvassing on time despite nationwide hiring shortfalls. The Bureau credits this success to better- than-expected productivity. The Bureau conducted “in-field” address canvassing for approximately 35 percent of the housing units (approximately 50 million housing units) across the country (see fig. 2). The Bureau had already determined “in-office” that the other 65 percent of addresses (approximately 93 million housing units) were part of stable blocks. The Bureau began the in-field address canvassing operation at seven of its 39 Area Census Offices on August 4, 2019, and then rolled out the operation to the remaining 32 offices on August 18, 2019. It conducted this phased approach to ensure all operations and systems worked together before commencing the operation nationwide. The total in-field address listing workload was more than 50 million addresses from the Bureau’s address file. Bureau officials reported that listers were generally more productive than expected, thus allowing the Bureau to complete the operation as scheduled on October 11, 2019 (see fig. 3). The actual hourly productivity rate for the operation was 19.8 addresses versus the anticipated rate of 15.8 addresses. According to Bureau officials, listers were more productive due to efficiency gains from the Bureau’s new approach, including an automated time and attendance system, the use of computer laptops to collect census data, and a new operational control system that was used to electronically optimize assignments and transmit work to listers. Bureau officials stated that the high productivity also helped the operation come in under budget. The operation’s cost was $118.6 million—while the anticipated cost was $185 million—a reduction of 36 percent. For in-field address canvassing, listers received online training, which detailed the procedures they were to follow, such as: comparing the housing units they see on the ground to the housing units on the address list, knocking on all doors so they could speak with a resident to confirm the address (even if the address is visible on the mailbox or house) and to confirm that there are no other living quarters such as a basement apartment, looking for hidden housing units, and confirming the location of the housing unit on a map with GPS coordinates collected on the doorstep. In our observations of in-field address canvassing, the majority of listers generally followed these procedures. However, some listers we observed did not always follow procedures. For example, Ten out of 59 listers did not work ground to book (i.e., compare what they saw on the ground to what was on their list). Nine out of 59 listers did not walk up to the doorstep to collect the GPS coordinate. Specifically, we observed listers use mailboxes to confirm address information and collect the GPS coordinates from the mailbox. Following proper procedures is important because getting a GPS reading from the doorstep of every address contributes to the accuracy of the address file. Fourteen of 59 listers did not consistently knock on every door as required to confirm the address and ask about “hidden” housing units. Seventeen of 59 listers did not always look for or ask about “hidden” housing units. Not knocking on doors or asking about hidden housing units represents missed opportunities to potentially add missing addresses to the Bureau’s master address file. Further, not all listers we observed provided the required confidentiality notices to occupants. Seven listers we observed did not provide confidentiality notices. Occupants may be more willing to provide their information if they know their responses will not be shared. We communicated the information regarding our observations to the Bureau, and on August 26, 2019, the Bureau instructed its field offices to remind listers of the appropriate procedures. According to Bureau officials, some amount of temporary staff deviates from following procedures with every decennial census. As such, to control for this, the Bureau implemented a Quality Control (QC) component for in-field address canvassing that is designed to detect and correct deficient production listers’ work. QC started on August 11, 2019, and included a total workload of around 3.4 million addresses. For this operation, an automated system selected the sample of addresses to review; these addresses were assigned to QC listers. QC listers received instructions to begin canvassing at a specified location, usually an intersection, and to continue canvassing addresses until the system identified the work unit as “complete” for QC purposes. An address worked by a production lister was considered to have “failed” QC if the QC lister recorded changes, or if the lister missed the address and the QC lister found it. Depending on the size of the block, after a predetermined number of addresses fail within a block, the system fails the entire block. Once a block fails, the QC lister must recanvas all the addresses in that block. Based on preliminary results, Bureau officials estimate that 4.3 percent, or about 2.2 million addresses, failed. According to Bureau officials, while they did not have a predetermined target for what was an acceptable range for the total number of addresses that failed QC, they nevertheless are reasonably confident that this was in an acceptable range for QC errors encountered during the operation. They further stated that they could not compare 2020 QC results to 2010 because the 2010 Address Canvassing Operation canvassed 100 percent of the addresses in-field, while the 2020 In-field Address Canvassing Operation only covered approximately 35 percent of the addresses across the country. Lister productivity for QC was also higher than expected. The Bureau anticipated the QC productivity at 8.03 addresses per hour compared to the actual rate of 14.05 addresses per hour. Higher-than-expected productivity rates contributed to a reduction in costs and the actual cost of QC production was $10.3 million versus the anticipated cost of $25.6 million, a savings of $15.3 million. Additionally, Bureau officials stated that QC came in so far under budget because the use of laptops increased efficiency and the actual QC workload was lower than the budget estimate. While the Bureau conducted real-time quality control follow-up of selected blocks during address canvassing, it also has two studies underway that will evaluate the re-engineered address canvassing approach, as well as the in-field address canvassing operation. Similar studies conducted by the Bureau in 2010 found that 95.7 percent of addresses were correctly deleted and 83.6 percent of addresses were correctly added. Both studies underway have a set of research questions designed to evaluate the accuracy and effectiveness of address canvassing. For example the Bureau seeks to answer questions such as: What percentage of the housing units added during in-field address canvassing were correctly added (and added-in-error)? What percentage of the housing units identified as deleted or duplicated by the listers during in-field address canvassing were correctly deleted or duplicated (and deleted-in-error)? Answering these and other questions contained in both studies will be critical to determining the quality of the operation, as not all listers followed procedures, which may have led to errors in the address file. It is anticipated that the final report for the 2020 Census In-Field Address Canvassing Operational Assessment study will be available September 2020, and the 2020 Census Evaluation: Reengineered Address Canvassing study will be available March 2023. In addition to completing in-field address canvassing on schedule and under budget Bureau officials highlighted other successes from the operation including: Automated solutions for training staff. Bureau-developed training materials that used a blended training approach including instructor- led, computer-based, and hands-on training. This is a change from the 2010 paper-based and classroom-only training approach. Efficiency gains from conducting reengineered field operations using: New operational control systems, which were used to electronically assign and transmit work to the listers. New automated time and expense reporting (timecards) for employees. In 2010, timecards were paper-based and the listers had to meet with their supervisors to submit them. Enhanced software application for validating and updating addresses. Implementation of rapid response to Hurricane Dorian, which affected areas of the Southeastern United States, resulted in minimal disruptions to the operation. Additionally, the Bureau was able to resolve some unforeseen challenges at the seven Area Census Offices that opened early. For example, the Bureau identified issues with training login and new hires not being on the training roster and rectified those issues before the operation expanded to the rest of the country. The Bureau experienced delays in hiring for its early operations, raising concerns about hiring for peak operations. The Bureau’s target was to hire 40,300 listers by September 7, 2019, but as of September 9, 2019, the Bureau had hired 31,151 listers. Though address canvassing productivity was higher than expected, in some parts of the country the operation was at risk of falling behind because of a shortage of listers. The Bureau told us it filled the gap with listers who lived well outside of the area in which they were supposed to work—in some cases from a different state. This strategy allowed the Bureau to complete the operation on schedule; however, though the operation as a whole was under budget, the Bureau incurred unplanned costs for travel (airfare, personal mileage rates, rental cars, hotel stays, and per diem). As we previously reported, these hiring problems are an early warning for what may occur later in the census during nonresponse follow-up, when the Bureau intends to hire between 320,000 to 500,000 enumerators to follow up with households that did not initially respond to the census. The Bureau said the hiring issues were caused by delays in processing background checks and greater-than-expected attrition. According to the Bureau, these delays arose, in part, due to early shortages of staff to review background checks and because a significant number of applicants did not completely or accurately fill out related forms. In February 2019, the Bureau began to bring on about 130 temporary staff to review forms for accuracy and completeness prior to submission for investigation and to help investigators conduct the pre-employment background checks. Those delays in turn contributed to subsequent challenges in onboarding listers for address canvassing. For example, according to Bureau officials, the delays in early hiring for Area Census Office staff meant some offices did not have enough clerks in place to process paperwork for listers or make reminder phone calls to hire and onboard listers. Regarding attrition, more listers quit than expected at two points in the hiring process: Fingerprinting: The Bureau expected about 15 percent of applicants would leave the hiring process after being selected and before submitting fingerprints. However, the attrition rate was closer to 25 percent. Bureau officials told us they attributed this to selected applicants, in some cases, having to travel long distances to be fingerprinted. Training: The Bureau found that fewer selected and cleared applicants attended training than anticipated. Bureau officials attributed this to fewer clerks being available to call trainees with reminders to attend training due to delays in clerks receiving their own background checks. Bureau officials also attributed some of this attrition to the 60-day period between the selection of applicants and their training. This new time frame was put in place for the 2020 Census to provide adequate time for adjudication of background checks. The Bureau has begun to address these challenges by adapting its hiring and onboarding processes for peak operations, such as nonresponse follow-up, which is to begin May 2020. For example, the Bureau: Increased the number of fingerprinting locations and machines. According to Bureau officials, it added 133 additional sites and 300 additional machines, bringing the total number of vendor sites for fingerprinting to 829. Staffed Area Census Offices to help newly-selected applicants for positions complete their forms and initiate the background check process. Hired additional staff to help clear background checks. The Bureau hired 200 staff at the National Processing Center and an additional 150 at the Regional Census Centers. Changed the recruiting goals due to the attrition experienced during address canvassing. The recruiting goal has increased from 2.3 million to 2.7 million to ensure it has a large enough applicant pool. This increases the ratio of recruited applicants to positions from 5:1 to 6:1. Completed a wage rate study and increased wages in 73 percent of counties by an average of $1.50 per hour for enumerators. Developed an email campaign to maintain contact with individuals in the recruiting pool. Decreased the types, and therefore the number, of positions that required a full background check. Included additional training for replacement hires in the training schedules. A make-up session was added to the nonresponse follow- up training schedule, May 14-19, 2020. If effectively implemented, these steps hold promise for helping to address the hiring issues. To effectively manage address canvassing, the Bureau provides data- driven tools for the census field supervisors to manage listers, including system alerts that identify issues that require the supervisor to follow up with a lister. Operational issues such as listers not working assigned hours or falling behind schedule need to be resolved quickly because of the tight time frames of the address canvassing and subsequent operations. For the address canvassing operation, the system generated codes that covered a variety of operational issues such as unusually high or low productivity (which may be a sign of fraud or failure to follow procedures) and administrative issues such as compliance with overtime and completion of expense reports and time cards. During the operation, more than 621,000 alerts were sent to census field supervisors. Each alert requires the supervisor to take action and then record how the alert was resolved. To assist supervisors, these alerts need to be reliable and properly used. However, nine out of 22 census field supervisors we spoke to indicated the alerts were not always useful. For example, almost 40 percent of those alerts were related to no progress being made on a block. This was due in part to listers opening all of the blocks they were assigned on their laptops in order to manage their workload, triggering the system that work had begun on all assigned blocks when in fact the lister was only working one block. We first heard about this issue from field supervisors in late August. Census field supervisors we spoke to indicated that these alerts took an inordinate amount of time to resolve, in part because almost every lister would open every block to plan his or her day. We alerted Bureau officials in headquarters, and they notified area census offices to remind supervisors to instruct listers not to open all of their blocks at once. After the notification was sent out, Bureau officials reported that the number of alerts due to blocks not being worked declined. Bureau officials further stated that this issue would not impact nonresponse follow-up because enumerators do not receive multiple assignments, but instead receive, work, and transmit only one assignment of housing units for follow-up a day. Another challenge faced by census field supervisors was providing feedback to listers on why addresses failed quality control. Four of 22 census field supervisors we spoke with were not aware that they had access to the reasons why addresses on a block failed quality control. Knowing where to find this information would have allowed census field supervisors to communicate this information to listers, thus improving lister performance as well as the accuracy of the data collected. We shared this information on some census field supervisor’s lack of awareness with the Bureau and on August 26, 2019, the Bureau notified its field offices to remind supervisors that detailed information on why addresses failed quality control was available on their laptops. For nonresponse follow-up, Bureau officials told us QC information about any enumerator with a specified number of failed cases will be sent directly to the Regional Census Center rather than the census field supervisor. The Regional Census Center will decide whether the enumerator should continue working and, if so, what corrective action to take, such as retraining. However, if it is determined that an enumerator falsified data, then the enumerator would not be given new assignments and all of his or her work would then be reinterviewed. We provided a draft of this report to the Secretary of Commerce. In its written comments, reproduced in appendix I, the Bureau noted that our report made no formal recommendations and that we highlighted several successes of the in-field address canvassing operation. The Bureau also described several claims of cost savings and efficiency gains which it attributed to various address list-building activities. While we have previously reported on the Bureau’s 2020 address list-building efforts, we have not audited claims made in the Bureau’s response or elsewhere regarding potential cost savings from innovations for the 2020 Census. The Bureau also provided us with technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Commerce, the Under Secretary of Economic Affairs, the Director of the U.S. Census Bureau, and interested congressional committees. The report also will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report please contact me at (202) 512-3236 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made major contributions to this report are listed in appendix III. J. Christopher Mihm, (202) 512-3236 or mihmj@gao.gov In addition to the contact named above, Lisa Pearson, Assistant Director; Timothy Wexler, Analyst-in-Charge; Margaret Fisher; Robert Gebhart; Richard Hung; Cynthia Saunders; Anna Sorrentino; Kate Sharkey; Dylan Stagner; Jon Ticehurst; Peter Verchinski; and Alicia White made key contributions to this report.", "summary": "The decennial census is a costly and complex undertaking and its success depends largely on the Bureau's ability to locate every person residing in the United States. To accomplish this monumental task, the Bureau must maintain accurate address and map information for every person's residence. If this information is inaccurate, people can be missed, counted more than once, or included in the wrong location. To help control costs and to improve accuracy, the Bureau used new procedures to build its address list for 2020. GAO was asked to review how the in-field address canvassing operation performed. This report (1) determines the extent to which the Bureau followed its plans and schedule for in-field address canvassing, and (2) identifies the successes and challenges that occurred during 2020 Census In-Field Address Canvassing that have potential implications for future operations. To address these objectives, GAO reviewed key documents including the 2020 Census operational plan that discussed the goals and objectives for the operation. GAO observed in-field address canvassing across the country at 18 area census offices, including a mix of rural and urban locations. GAO also interviewed field supervisors, listers, and office management to discuss the operation's successes and challenges. GAO provided a draft of this report to the Bureau. The Bureau provided technical comments, which were incorporated as appropriate. The Census Bureau (Bureau) completed in-field address canvassing as scheduled on October 11, 2019, despite nationwide hiring shortfalls. The Bureau credits this success to better-than-expected productivity—the actual hourly productivity rate for the operation was 19.8 addresses versus the anticipated rate of 15.8 addresses. The total workload included more than 50 million addresses. GAO observations of in-field address canvassing found that a majority of field staff (listers) generally followed procedures, but there were a number of exceptions. For example, 14 of 59 listers we observed did not consistently knock on every door as required to confirm the address and ask about “hidden” housing units. Not knocking on doors or asking about hidden housing units represents missed opportunities to potentially add missing addresses to the Bureau's address file. GAO communicated to Bureau officials that listers were not following procedures and they sent out a nationwide reminder for listers to do so. The Bureau credits efficiency gains to new systems for assigning work and a new reporting mechanism for collecting timecards, but experienced delays in hiring for address canvassing. Though address canvassing productivity was higher than expected, in some parts of the country the operation was at risk of falling behind because of a shortage of listers. The Bureau told GAO that it filled the gap with listers who lived well outside of the area in which they were supposed to work—in some cases from a different state. The Bureau is taking actions to address hiring problems for later operations, including nonresponse follow-up, when the Bureau intends to hire between 320,000 to 500,000 enumerators to follow up with households that did not initially respond to the census. Those actions include increasing wage rates in 73 percent of the counties nationwide.", "document_type": "gao"}
{"report": "Air Force pilots are required to complete various phases of training before they are considered to be mission ready. To assess pilot candidates’ flying aptitude for both traditional (i.e., manned) and remotely-piloted aircraft, the Air Force first requires these candidates to attend initial flight training. Successful candidates for traditional aircraft then attend one of two schools: Euro-NATO Joint Jet Pilot Training, the graduates of which become Specialized Undergraduate Pilot Training, the graduates of which become fighter, bomber, airlift, or tanker aircraft pilots, depending on pilot strengths and Air Force needs. During or following this training, a pilot is assigned to a specific aircraft, and the flight training program proceeds through three stages: Initial Qualification Training. This stage of training qualifies a pilot for basic flying duties associated with the type of aircraft (e.g., an F-16). The pilot accomplishes this stage at a formal training unit before moving to an assigned squadron. Graduates of initial qualification training courses have basic aircraft qualification status. Mission Qualification Training. This stage of training occurs once the basic aircraft qualified pilot is at the assigned unit. The pilot undergoes mission qualification training in the type of aircraft assigned to qualify for the specific missions the unit is required to perform. Continuation Training. This stage of pilot training has two components. Pilots participate in continuation training to (1) in some instances upgrade their qualifications to fill certain positions, such as flight lead, instructor, or forward air controller through specialized continuation training; and (2) in all instances maintain proficiency and improve their capabilities to perform their units’ assigned missions. The Ready Aircrew Program establishes the minimum number of live training events, or “sorties,” and virtual simulator training events, or “simulator missions,” that aircrews of a particular combat aircraft must complete during the annual training cycle to maintain mission readiness. These sorties and simulator missions are aligned with the units’ primary missions, for which the units must maintain “proficiency,” and secondary missions, for which they must maintain “familiarity.” The Air Combat Command, as lead command for the Ready Aircrew Program, with the assistance of other major commands (including the Air Force Global Strike Command, Pacific Air Forces, and U.S. Air Forces in Europe) and associated subordinate organizations (i.e., air wings and squadron commanders), develops tasking memorandums for the Ready Aircrew Program that delineate and specify the annual continuation training requirements for personnel assigned to each of the subordinate combat units. On the basis of our analysis, we found that the RAND report addressed each of the three statutory elements required by Congress in Section 351. First, the RAND report addressed two statutory elements by reviewing and assessing the assumptions underlying annual continuation training requirements for the Ready Aircrew Program and the overall effectiveness of the Ready Aircrew Program in managing aircrew training requirements. These two statutory elements focus on issues raised in our prior report recommendations, which we discuss in more detail later in this report. Table 1 provides detailed information about these statutory elements, our assessment of RAND’s findings, and RAND’s findings associated with each element. The RAND report addressed the third statutory element by making recommendations for the improved management of training requirements. Specifically, the RAND report made nine recommendations, listed in table 2. The RAND report and its underlying analysis is consistent with generally accepted research standards for design, execution, and presentation. Table 3 summarizes our assessment of the extent to which the RAND report conformed with these standards. The RAND study and our previous audit work identified similar deficiencies in the management and operation of the Ready Aircrew Program. Specifically, in September 2016, we reported that the Air Force had used the same underlying assumptions to establish its annual training requirements in the Ready Aircrew Program from 2012 through 2016, which may not reflect current and emerging training needs. We concluded that without fully reassessing the assumptions underlying its training requirements, the Air Force could not be certain that its annual training plans are aligned with its stated goals to ensure a full-spectrum- capable force that can successfully achieve missions across a broad range of current and emerging threats. We recommended that the Air Force comprehensively reassess the assumptions underlying its annual training requirements—and make any appropriate adjustments in future aircrew training plans to ensure that its forces can accomplish a full range of missions. Additionally, in our September 2016 report, we also reported that the Air Force did not systematically evaluate the effectiveness of training performed as part of the Ready Aircrew Program. We recommended that the Air Force establish desired learning objectives and training support elements needed to accomplish the training expectations identified by the Ready Aircrew Program and develop a process to collect data to assess the effectiveness of annual training against these features. We discuss these recommendations and their status in more detail later in this report. Section 351 of the NDAA for Fiscal Year 2017 required the Air Force to report on any actions it plans to take in response to RAND’s recommendations and to estimate the resources required to implement the recommendations. On August 30, 2018, the Air Force provided its report—a one-page transmittal letter from the Secretary of the Air Force with the RAND report incorporated as an enclosure—to congressional committees in fulfillment of the Section 351 requirements. In its report, the Air Force agreed with RAND that more investment is needed in data collection because its current system does not lend itself to analysis that could be used to gain efficiencies. The Air Force also stated that it is addressing the RAND recommendations by working to link readiness to Ready Aircrew Program training requirements. Specifically, the Air Force stated that it was taking the following three actions: building training matrices to help commanders assess their units’ establishing common data architecture through the Air Force’s Chief Data Officer–led effort, and evaluating aspects of the Ready Aircrew Program to increase lethality and improve readiness as the Air Force shifts to executing the mandates of the 2018 National Defense Strategy. However, in its August 2018 report, the Air Force did not provide any additional details to further describe or link these broad actions to the RAND findings and recommendations. Therefore, the extent to which these three actions are responsive to the RAND recommendations is unclear. In October 2019, upon completion of our audit work, Air Force officials provided us with a briefing they described as a corrective action plan that further elaborated on the Air Force’s position with respect to each of RAND’s nine recommendations. Air Force officials conceded that the actions described in that briefing and plan were already underway at the time of the study and were not initiated in response to the study’s recommendations. They stated that, though they generally agreed with the recommendations, the Air Force lacked the manpower, resources, and means to implement them. As such, the Air Force considers each of the recommendations closed and plans no further actions. Further, the Air Force has no plans for future follow-up on implementation of the RAND recommendations. Accordingly, we did not further assess the actions described by the Air Force in relation to the RAND study. Table 4 summarizes the Air Force’s position as provided in its corrective action plan briefing. As described in table 4, the Air Force concurred with three recommendations, partially concurred with five, and did not concur with one. The following summarizes the Air Force position by concurrence category: Concur: In concurring with RAND Recommendations 2 and 8—to invest in data systems for the collection, access, and storage of data to correct deficiencies in current systems and improve analysis and readiness reporting—the Air Force stated that, before the recommendations were made, it had made available $5.15 million in fiscal year 2020 funding to develop an Aircrew Readiness Training Management Module as an upgrade to the Air Force’s Aviation Resource Management System. The Air Force expects that this module will centralize management of the Air Force Ready Aircrew Program Tasking Memorandum at the command level, transfer aircrew training data whenever a member moves to a new station, and improve capability to track the types of flight simulators used for training. Further, because the Air Force does not plan to take additional actions, it estimated no resources are required, beyond the $5.15 million already funded, prior to RAND making these recommendations. In concurring with RAND Recommendation 5—to document training quality to support requests for training resources—the Air Force explained that it is documenting the quality of training prescribed in its annual Air Force Ready Aircrew Program Tasking Memorandum at the squadron level. However, this is not a change based on the RAND recommendation and was being done prior to RAND making this recommendation. The Air Force factors unit training, accomplishments, and readiness inputs into the Air Force Ready Aircrew Program Tasking Memorandum and overall flying hour program. As the flying hour program is the basis for resource training requests and the Air Force factors unit training and accomplishments into its flying hour program, the Air Force explained that it plans to take no additional actions based on this recommendation. Further, because the Air Force does not plan to take additional actions, it estimated no resources were required to implement this recommendation. Partially Concur: In partially concurring with RAND recommendations 1, 3, 4, 6, and 9, Air Force officials explained that the mechanisms reflected in its comments are sufficient to address the intent of the RAND recommendations even though these mechanisms predated RAND’s recommendations. Consequently, according to the Air Force officials who briefed us, the Air Force plans no additional actions based on the recommendations, obviating the need to estimate resources in the case of these five recommendations. Nonconcur: In not concurring with RAND Recommendation 7—to consider changing how Air Force Flying Hour Program requirements affect the flying hour program—the Air Force explained that the flying hour program determination is standard across the Total Air Force, affecting more than combat aircrews alone, and that the centrality of the flying hour program to readiness and combat capability cannot be overemphasized and must be defendable and auditable. Further, based on its not concurring with this recommendation, the Air Force did not estimate resources for this recommendation. Notwithstanding the RAND study, the actions taken by the Air Force may not fully implement our prior recommendations as described previously. Specifically, we recommended that the Air Force comprehensively reassess the assumptions underlying its annual training requirements— including for example the total annual training requirements by aircraft, the criteria for designating aircrews as experienced or inexperienced, and the mix between live and simulator training—and make any appropriate adjustments in future aircrew training plans to ensure that its forces can accomplish a full range of missions. While RAND accomplished such an analysis as part of its review (see table 1, items 1(a) through 1(e) above), RAND concluded in its analysis that the Air Force did not have the objective measures of proficiency needed to determine the minimum and optimum number of sorties and that the combat air forces aviation community lacks consensus on how to define and measure an aircrew member’s proficiency. While the Air Force has defined proficiency in an Air Force manual issued in September 2019, the Air Force has not reassessed its assumptions underlying training requirements and made appropriate adjustments to future training plans per our recommendation. Further, we recommended that the Air Force establish desired learning objectives and training support elements needed to accomplish the training expectations in its annual Ready Aircrew Program tasking memorandums, and develop a process to collect data to assess the effectiveness of annual training. In commenting on RAND’s recommendations 2 and 8, the Air Force stated that, before the recommendations were made, it had made available $5.15 million in fiscal year 2020 funding to develop an Aircrew Readiness Training Management Module as an upgrade to the Air Force’s Aviation Resource Management System. The Air Force’s effort to upgrade this system, while not a result of the RAND recommendations, may meet the intent of our recommendation. For example, the Air Force’s development of the Aviation Resource Management System is expected to centralize management of the Air Force’s Ready Aircrew Program training requirements data at the command level, transfer aircrew training data whenever a member moves to a new station, and improve capability to track the types of flight simulators used for training. When fully implemented, these improvements may ultimately allow the Air Force to assess the effectiveness of annual training, as we recommended. However, it is too early to tell as the actions were under development or had just begun at the end of September 2019, and sufficient data to evaluate the results have not been collected. Fully implementing both of our recommendations would better position the Air Force to ensure that its aircrews receive effective training to achieve a range of missions for current and emerging threats. We provided a draft of this report to DOD for review and comment. In its written comments (reprinted in appendix II), DOD stated that it saw great value in our discussion. DOD added that its position on the two recommendations we made in our 2016 report—with which DOD did not concur—is fundamentally unchanged. However, we continue to believe the recommendations should be implemented by the Air Force, as previously discussed in this report. In its comments, DOD also stated that it is addressing training infrastructure and aircrew proficiency through two initiatives. First, in response to the 2018 National Defense Strategy, the department's Joint Operational Infrastructure Plan is framing the modernization effort of DOD-wide Operational Infrastructure. The Joint Operational Infrastructure Plan specifically addresses areas such as Live Virtual Constructive and aircrew training. Second, the department is pursuing efforts to align training events with the range of current and evolving threats. According to DOD, both efforts will address the underlying assumptions for aircrew training and proficiency with reportable readiness metrics. However, the Joint Operational Infrastructure Plan is a draft and not yet officially issued, according to an official at the Office of the Under Secretary of Defense for Personnel and Readiness. Therefore, details needed to assess this plan are not yet available for us to consider in determining whether the change will help to address the recommendations we made in 2016. Regarding efforts the department is pursuing to better align training events, the DOD comments did not include sufficient details to allow us to state whether the change could be helpful in addressing our recommendations. Nonetheless, to the extent that DOD is successful in completing, issuing, and implementing its new Joint Operational Infrastructure Plan, or takes further actions related to our prior recommendations, we will consider them as we continue to analyze DOD efforts to address our recommendations. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense for Personnel and Readiness, and the Secretary of the Air Force. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. To determine whether the RAND report followed generally accepted research standards, we chose the following criteria, which are based on a review of research literature and Department of Defense (DOD) guidance, from which we identified frequently occurring, generally accepted research standards that are relevant for defense studies, including those related to the presentation of results. These standards have been used in a number of our prior reports, modified as appropriate for each situation. For the purposes of this engagement, we assessed the RAND study against three major elements, listed below, that fall under generally acceptable research standards, as based on our prior work. These generally accepted research standards are consistent with Office of Management and Budget (OMB) Guidelines and DOD guidance on ensuring and maximizing the quality of information disseminated by federal agencies to the public. We discussed these standards with RAND officials, who agreed that they were generally consistent with their own quality standards for research and were applicable to this study. We determined that these standards are still current and relevant for the purposes of this report, based on their consistency with OMB and DOD guidance, discussions with RAND officials, and consideration of prior GAO work applying generally accepted research standards with the assistance of GAO’s Applied Research and Methods Team. The standards include the following: Design—Study is well designed. For the RAND study, we focused on the following elements for design: The study plan, scope, and objectives follow existing guidance. Assumptions and constraints are reasonable and consistent. Execution—Study is well executed. For the RAND study, we focused on the following elements for execution: The methodology is successfully executed. Data used to support study and analyses are validated. Presentation—Results are well presented. For the RAND study, we focused on the following elements for presentation: Timely, complete, accurate, concise, and relevant to stakeholders. Presentation of results supports findings. In addition to the contact named above, Beverly Schladt (Assistant Director), John Strong (Analyst in Charge), John Beauchamp, Vincent Buquicchio, Martin De Alteriis, and Lillian Moyano Yob made key contributions to this report.", "summary": "In September 2016, GAO reported that annual combat aircrew training requirements delineated in the Air Force's Ready Aircrew Program might not address pilot training needs, and that the Air Force did not systematically evaluate the effectiveness of its training. As a result, Congress included a provision in Section 351 of the NDAA for Fiscal Year 2017 for the Air Force to commission an independent review of its Ready Aircrew Program, report on actions it planned to take in response to any recommendations, and provide an estimate of any resources required. Section 351 also included a provision for GAO to assess the Air Force report. This report examines whether (1) the independent review conducted by the RAND Corporation addressed statutory requirements to review and assess the Ready Aircrew Program, and (2) the Air Force has reported on completed or planned actions to implement the RAND report recommendations. To address these objectives, GAO reviewed the RAND and Air Force reports on the Ready Aircrew Program, assessed the study against generally accepted research standards, and interviewed officials at RAND, Air Force Headquarters, and the Air Combat Command. A July 2018 RAND report—commissioned by the Air Force—addressed the statutory requirements of the National Defense Authorization Act (NDAA) for Fiscal Year 2017 to review and assess the Air Force's Ready Aircrew Program and make recommendations for ways to improve it. The Ready Aircrew Program establishes minimum annual training requirements for combat aircrew. RAND's report, entitled Independent Review and Assessment of the Air Force Ready Aircrew Program , made nine recommendations to improve its management: 1. Leverage internal expertise to implement measures for proficiency. 2. Invest resources to design data collection and storage solutions that facilitate analysis and readiness reporting. 3. Document the Ready Aircrew Program Tasking Memorandum development process in Air Force instruction supplements and ensure that the process incorporates squadron-level input and feedback. 4. Establish a more explicit and formal link between proficiency and Ready Aircrew Program requirements. 5. Document training quality to support requests for training resources. 6. Identify the conditions under which Ready Aircrew Program requirements, including mission types, can be accomplished. 7. Consider changing how Ready Aircrew Program requirements affect the Flying Hour Program. 8. Invest in data systems to correct data collection and assess deficiencies. 9. Leverage the Air Force Research Laboratory's performance data work and invest in added analysis to produce enterprise-wide proficiency metrics. The nine RAND recommendations aligned with two GAO recommendations made in 2016 to comprehensively assess the assumptions underlying the annual aircrew training requirements and develop a process to collect data to assess the effectiveness of the training. The Air Force's August 2018 one-page report to Congress included three broad actions in response to RAND's recommendations. The Air Force planned to build training matrices to help commanders assess their units' effectiveness, establish common data architecture through the Air Force's Chief Data Officer–led effort, and evaluate aspects of the Ready Aircrew Program to increase lethality and improve readiness as the Air Force shifts to executing the mandates of the 2018 National Defense Strategy. The Air Force, however, did not explain how these three efforts would specifically address the nine recommendations. Air Force officials said that, though they generally agreed with RAND's recommendations, the Air Force lacked the resources to fully implement them beyond actions that were underway prior to the RAND report, and considers all recommendations as “closed.” In part due to its not fully implementing RAND's recommendations, the Air Force has not fully addressed GAO's two recommendations. Fully implementing GAO's recommendations would better position the Air Force to ensure its aircrews receive effective training to achieve a range of missions. GAO is not making any new recommendations in this report but restates the need to fully implement GAO's two 2016 recommendations. In comments on a draft of this report, DOD did not concur with the two 2016 recommendations. GAO maintains that the Air Force should implement both recommendations. recommendation. GAO will consider these in its annual recommendations follow-up.", "document_type": "gao"}
{"report": "In 1968, Congress created NFIP, with the passage of the National Flood Insurance Act, to help reduce escalating costs of providing federal flood assistance to repair damaged homes and businesses. According to FEMA, NFIP was designed to address the policy objectives of identifying flood hazards, offering affordable insurance premiums to encourage program participation, and promoting community-based floodplain management. To meet these policy objectives, NFIP has four key elements: identifying and mapping flood hazards, floodplain management, flood insurance, and incentivizing flood-risk reduction through grants and premium discounts. NFIP enables property owners in participating communities to purchase flood insurance and, in exchange, the community agrees to adopt and enforce NFIP minimum floodplain management regulations and applicable building construction standards to help reduce future flood losses. A participating community’s floodplain management regulations must meet or exceed NFIP’s minimum regulatory requirements. Insurance offered through NFIP includes different coverage levels and premium rates, which are determined by factors that include property characteristics, location, and statutory provisions. NFIP coverage limits vary by program (Regular or Emergency) and building occupancy (for example, residential or nonresidential). In NFIP’s Regular Program, the maximum coverage limit for one-to-four family residential policies is $250,000 for buildings and $100,000 for contents. For nonresidential or multifamily policies, the maximum coverage limit is $500,000 per building and $500,000 for the building owner’s contents. Separate coverage is available for contents owned by tenants. NFIP also offers Increased Cost of Compliance coverage for most policies, which provides up to $30,000 to help cover the cost of mitigation measures following a flood loss when a property is declared to be substantially or repetitively damaged. Through NFIP, FEMA maps flood hazard zones on a Flood Insurance Rate Map, which participating NFIP communities must adopt. According to FEMA, floodplain management standards are designed to prevent new development from increasing the flood threat and to protect new and existing buildings from anticipated flooding. FEMA has a division responsible for flood mapping activities and policy and guidance, but stakeholders from various levels of government and the private sector participate in the mapping process, as appropriate. A community’s Flood Insurance Rate Map serves several purposes. They provide the basis for setting insurance premium rates and identifying properties whose owners are required to purchase flood insurance. Since the 1970s, homeowners with federally backed mortgages or mortgages held by federally regulated lenders on property in a special flood hazard area have been required to purchase flood insurance. Others may purchase flood insurance voluntarily if they live in a participating community. The maps also provide the basis for establishing minimum floodplain management standards that communities must adopt and enforce as part of their NFIP participation. As of May 2020, 22,487 communities across the United States and its territories voluntarily participated in NFIP by adopting and agreeing to enforce flood-related building codes and floodplain management regulations. FEMA supports a variety of community-level flood mitigation activities that are designed to reduce flood risk (and thus NFIP’s financial exposure). These activities, which are implemented at the state and local levels, include hazard mitigation planning; adoption and enforcement of floodplain management regulations and building codes; and use of hazard control structures such as levees, dams, and floodwalls or natural protective features such as wetlands and dunes. FEMA provides community-level mitigation funding through its HMA grant programs. In addition, FEMA’s Community Rating System is a voluntary incentive program that recognizes and encourages community floodplain management activities that exceed the minimum NFIP requirements. Flood insurance premium rates are discounted to reflect the reduced flood risk resulting from community actions that meet the three goals of reducing flood damage to insurable property, strengthening and supporting the insurance aspects of NFIP, and encouraging a comprehensive approach to floodplain management. At the individual property level, mitigation options include property acquisition—or “buyouts”—to either demolish a building for green space or relocate a building to a low flood risk area, elevation, or floodproofing. Acquisition and demolition (acquisition) is one of the primary methods by which states or localities use FEMA funding to mitigate flood risk. Through this process, a local or state government purchases land and structures that flooded or are at risk from future floods from willing sellers and demolishes the structures. The community restricts future development on the land, which is maintained as open space in perpetuity to restore and conserve the natural floodplain functions. According to FEMA officials, an advantage of property acquisition is that it offers a permanent solution to flood risks, whereas other mitigation methods make properties safer from floods but not immune. Property acquisition and demolition is a voluntary process, and property owners are paid fair market value for their land and structures. Acquisition is typically done on a community-wide scale, purchasing several or all properties in an at-risk neighborhood. Acquisition projects typically require building consensus from property owners and sustained communication and collaboration between residents and the government executing the project. Acquisition and relocation (relocation) refers to purchasing a structure and moving it to another location instead of demolishing it. Through this process, state or local governments use FEMA funding to help purchase land from willing sellers and assist the property owners with relocating the structure. The structure must be sound and feasible to move outside of flood-prone areas. Relocation is a voluntary process and property owners are paid fair market value for their land. Elevation involves raising a structure so that the lowest occupied floor is at or above the area’s base flood elevation. Structure elevation may be achieved through a variety of methods, including elevating on continuous foundation walls; elevating on open foundations, such as piles, piers, or columns; and elevating on fill. Structures proposed for elevation must be structurally sound and capable of being elevated safely. Further, elevation projects must be designed and adequately anchored to prevent flotation, collapse, and lateral movement of the structure from flooding, waves, and wind. Floodproofing falls into two categories: dry floodproofing and wet floodproofing. Dry floodproofing involves sealing a structure to prevent floodwater from entering. Examples of dry floodproofing measures include using waterproof coatings or coverings to make walls impermeable to water, installing waterproof shields, and installing devices that prevent sewer and drain backup. Dry floodproofing is appropriate only where floodwaters do not exceed three feet, the speed of flood waters is low, and the duration of flooding is relatively short because walls and floors may collapse from the pressure of higher water levels. Wet floodproofing involves changing a structure to allow floodwaters to enter and exit with minimal damage. Wet floodproofing is used in parts of a structure that are not used as living space, such as a crawlspace, basement, or garage. Examples of wet floodproofing measures include installing flood openings in the foundation and enclosure walls below the base flood elevation, using flood-resistant building materials and furnishings located below the base flood elevation, and either elevating or floodproofing all utility systems and associated equipment to protect them from damage. FEMA administers three HMA grant programs that can be used to fund flood mitigation projects: the Hazard Mitigation Grant Program (HMGP), Pre-Disaster Mitigation (PDM), and Flood Mitigation Assistance (FMA). Eligible HMA applicants include states, territories, and federally recognized tribal governments. Local communities cannot apply directly to FEMA for HMA funding but instead must collaborate as sub-applicants with their state, territory, or tribal government and then receive funding through that entity. Certain nonprofit organizations can act as sub- applicants but only under HMGP. Generally, individuals may not apply for HMA funding, but they may benefit from a community application. Applicants to all three programs must have FEMA-approved hazard mitigation plans. FEMA evaluates HMA applications based on technical feasibility and cost-effectiveness, among other factors. In fiscal year 2019, HMA awarded $859 million in funding. Eligible activities differ for the three programs but must be consistent with FEMA’s National Mitigation Framework. The Hazard Mitigation Grant Program helps communities implement hazard mitigation measures following a presidential major disaster declaration to improve community resilience to future disasters. HMGP provides funding to protect public or private property through various mitigation measures based on state or tribal priorities. Mitigation project examples include acquisition, relocation, retrofitting structures to minimize damages from various natural hazards, and elevating flood prone structures. HMGP recipients (states, territories, and federally recognized tribal governments) are primarily responsible for prioritizing, selecting, and administering state and local hazard mitigation projects. According to FEMA guidance, although individuals may not apply directly to the state for assistance, local governments engage interested property owners during the application process. A formula based on the size of the presidential disaster declaration determines the amount of money available to HMGP. Pre-Disaster Mitigation seeks to reduce overall risk to the population and structures from future natural hazard events, while also reducing reliance on federal funding in future disasters. PDM grants fund mitigation plans and eligible projects that reduce or eliminate long-term risk to people and property from natural disasters, such as property acquisition, property elevation, earthquake hardening, and construction of tornado and high-wind safe rooms. Generally, local governments (i.e., sub-applicants) submit mitigation planning and project applications to their state, territory, or federally recognized tribal government (i.e., applicants) for review and prioritization. The state, territory, or federally recognized tribal government then submits one PDM grant application to FEMA for consideration. Annual Congressional appropriations fund these grants, and FEMA awards them on a nationally competitive basis. In fiscal year 2019, Congress appropriated $250 million to PDM, which was the program’s final year of funding. In 2018, Congress passed the Disaster Recovery Reform Act, which included amendments to PDM, which FEMA calls the Building Resilient Infrastructure and Communities program. According to FEMA officials, this program is replacing PDM in fiscal year 2020 and will be funded through the Disaster Relief Fund as a 6 percent set-aside from the estimated total amount of grants for each major disaster declaration. FEMA has solicited public input on the program and said it expects to release a notice of funding opportunity in summer 2020. Flood Mitigation Assistance is designed to reduce or eliminate flood insurance claims by funding cost-effective flood mitigation projects that reduce or eliminate long-term risk of flood damage to structures insured under NFIP. Typical projects may include acquisition of RL properties, elevation of buildings, and neighborhood-scale flood defense investment. Generally, local communities will sponsor applications on behalf of homeowners and then submit the applications to their state. A state or federally recognized tribal government must submit the grant applications to FEMA. Annual Congressional appropriations fund FMA grants, and FEMA awards them on a nationally competitive basis. FMA appropriations have remained relatively stable at about $175 million for fiscal years 2016 through 2019. RL properties present a financial challenge for NFIP. FEMA has three definitions for such properties that vary slightly to meet the specific needs of different programs: NFIP Repetitive Loss refers to an NFIP-insured structure that has incurred flood-related damage on two occasions during a 10-year period, each resulting in at least a $1,000 claim payment. FEMA uses the NFIP RL definition for insurance purposes related to the Community Rating System, for local hazard mitigation plans, and for eligibility determinations for preferred risk policies and individual assistance. FMA Repetitive Loss refers to an NFIP-insured structure that (a) has incurred flood-related damage on two occasions in which the cost of repair, on average, equaled or exceeded 25 percent of the value of the structure at the time of each such flood event; and (b) at the time of the second incidence of flood-related damage, the flood insurance policy contained Increased Cost of Compliance coverage. FEMA uses this definition for FMA purposes, as these properties are eligible for the largest federal cost share for mitigation, up to 90 percent. This is also the same definition NFIP uses to approve an Increased Cost of Compliance payment. Severe Repetitive Loss refers to an NFIP-insured structure that has incurred flood-related damage for which (a) four or more separate claims have been paid that exceeded $5,000 each and cumulatively exceeded $20,000; or (b) at least two separate claim payments have been made under such coverage, with the cumulative amount of such claims exceeding the fair market value of the insured structure. FEMA has two severe RL definitions for mitigation and insurance, which are similar except that the insurance definition includes only residential structures, while the mitigation definition includes all structures. FEMA uses the severe RL definition for grant eligibility and cost share, the Community Rating System, and insurance rate setting. HMGP is the largest of FEMA’s three HMA programs and, unlike the others, it is based on the amount of disaster assistance a state or territory receives following a presidential disaster declaration (see table 1). PDM and FMA are smaller grant programs that receive annual appropriations and are not directly tied to an immediately preceding disaster. Because these programs do not require an immediate disaster declaration, FEMA considers them pre-disaster programs, as their intent is to mitigate potential damage before disasters occur. HMGP and PDM can be used for projects that mitigate the risk of many hazards, including flood, wind, fire, earthquake, and drought, but FMA can only be used to mitigate the risk of flood (see table 1). Furthermore, FMA funds can only be used to mitigate properties that are insured by NFIP, but HMGP and PDM funds can be used to mitigate properties without NFIP coverage. Properties mitigated in a special flood hazard area, where the structure remains on the parcel, must maintain a flood insurance policy after project completion. HMA grants fund a variety of methods to mitigate the flood risk of properties, including acquisition, elevation, relocation, and floodproofing. In most cases, HMA grants cover up to 75 percent of the project cost, and the grantee generally must contribute the remainder using nonfederal funds (although there are some exceptions, discussed below). However, PDM will cover up to 90 percent of project costs for communities that meet FEMA’s definition of small and impoverished. Moreover, FMA will cover up to 90 percent for projects that mitigate RL properties and up to 100 percent for severe RL properties. Funding levels for the three programs have varied over time because they have depended on disaster declarations and annual appropriations (see fig. 1). HMGP is the largest of the three programs—adjusted for inflation, annual HMGP grants have reached $2.9 billion, while PDM and FMA have never exceeded $300 million. According to FEMA officials, the estimated annual funding for the Building Resilient Infrastructure and Communities program, the successor to PDM, will average $300 million to $500 million, as it will be funded by a 6 percent set aside of annual estimated disaster grant expenditures. HMA funding also varies by state. Louisiana has obligated the most funding. After adjusting for inflation, it has obligated more than $3.1 billion from all three programs since HMGP was created in 1989, followed by California ($2.0 billion), Texas ($1.8 billion), New York ($1.6 billion), and Florida ($1.5 billion), while the bottom 18 states and territories each obligated less than $50 million (see fig. 2). Because HMGP is the largest program and is tied to presidential declarations, these totals reflect, in part, the extent to which states and territories have experienced natural disasters in this time period. Typically, recipients of federal mitigation grants must use nonfederal funds to meet cost share requirements because federal law prohibits the use of more than one source of federal disaster recovery funding for the same purpose. However, according to FEMA, some federal programs are exempt from these requirements due to authorizing statutes and therefore may be used in concert with HMA funds. Department of Housing and Urban Development’s Community Development Block Grant (CDBG) program. The Department of Housing and Urban Development awards CDBG funds to state and local governments to support a variety of community and economic development needs. According to FEMA’s HMA Cost Sharing Guide, HMA applicants may use several categories of CDBG funds as a source of project cost share, as long as the project meets Department of Housing and Urban Development rules. CDBG Disaster Recovery funds are the most frequently used form of HMGP cost share from a federal agency, according to FEMA. FEMA Increased Cost of Compliance coverage. NFIP offers Increased Cost of Compliance coverage, which provides up to $30,000 for policyholders to fund mitigation efforts on their property if they experience substantial damage or if their structure is an RL property. Between 1997 and 2014, the vast majority (99 percent) of Increased Cost of Compliance claims met the substantially damaged property definition, according to a 2017 report from the University of Pennsylvania. Unlike CDBG, which is awarded to states and local governments, Increased Cost of Compliance is awarded directly to individuals. According to FEMA, it is eligible as an HMA nonfederal cost share because it is considered a direct contract between the insurer and policyholder. FEMA allows recipients to assign their funds to the community as part of a collective mitigation project, and the community is then obligated to provide HMA funding to any property owner who contributed Increased Cost of Compliance dollars toward the nonfederal cost share. As of September 2019, FEMA had closed more than 38,000 Increased Cost of Compliance claims with dates of loss since 1997, totaling more than $877 million. Small Business Administration disaster loans. Small Business Administration disaster loans provide up to $200,000 for repairing or replacing a primary residence and $40,000 for repairing or replacing personal items that have been affected by a disaster. The interest rate cannot exceed 4 percent for applicants unable to access credit elsewhere, and cannot exceed 8 percent for all others. Secondary or vacation homes are not eligible, but qualified rental properties may be eligible under the Small Business Administration’s business disaster loan program, which offers loans of up to $2 million. According to FEMA guidance, these loans can serve as a source of cost share if HMA grants are disbursed early enough; however, the differing award timelines often make these funding sources incompatible. Further, disaster loans may not be eligible in conjunction with HMA funds due to duplication of benefits, but general-purpose Small Business Administration loans are not subject to this restriction, according to FEMA. In addition to FEMA’s three HMA programs, other federal, state, and local programs have helped acquire properties. Community Development Block Grants. In addition to its use as a cost- share complement to HMA grants, states and communities can use CDBG Disaster Recovery funding as a stand-alone source of property acquisition funds, according to the Department of Housing and Urban Development. Availability of CDBG Disaster Recovery funds is subject to supplemental appropriations following a presidential disaster declaration and must be used in response to that specific disaster. CDBG Disaster Recovery funds are disbursed to state and local governments and not to individuals directly. However, the governmental recipient can award CDBG Disaster Recovery funds to private citizens, nonprofits, economic development organizations, businesses, and other state agencies. The Bipartisan Budget Act of 2018 appropriated funding for CDBG, of which the Department of Housing and Urban Development allocated almost $6.9 billion for CDBG mitigation funds for the first time, as a result of the 2015 to 2017 disasters. Unlike CDBG Disaster Recovery funds, which the recipient must use in response to a specific disaster, recipients may use CDBG Mitigation funds to mitigate risks from future disasters. U.S. Army Corps of Engineers’ National Nonstructural Committee. The Army Corps of Engineers (Corps) conducts a range of mitigation measures through the National Nonstructural Committee, including acquisitions, elevations, relocations, and floodplain mapping. Nonstructural refers to measures that attempt to mitigate the consequences of floods, as opposed to structural measures intended to prevent floods from occurring. According to the Corps, except for limited research funding, it does not offer grants for flood risk management projects, and large projects generally require specific authorization from Congress. However, the Corps’ Continuing Authority Program allows it to execute smaller projects at its discretion. For example, for one of the programs, the federal government funds 65 percent of a project’s cost, and the project sponsor must provide all land, easement, rights-of-way, relocations, and disposal areas required for the project. The sponsor’s cost share includes credit for provision of the requirements above and pre-approved work-in-kind, but at least five percent must be provided in cash. Department of Agriculture’s Natural Resources Conservation Service Emergency Watershed Protection Program. The Federal Agriculture Improvement and Reform Act of 1996 enables the Emergency Watershed Protection Program to purchase floodplain easements on residential and agricultural land for flood mitigation purposes and to return the land to its natural state. For agricultural and residential land, this program pays up to the entire easement value and also funds property demolition or relocation, according to the Department of Agriculture. Land generally must have flooded in the past year or twice within the previous 10 years to be considered eligible. State and local acquisition programs. While state and local governments are active participants in federal acquisition projects, some have also developed their own acquisition programs. These programs vary on the extent to which they rely on federal funds, if at all. For example: The Harris County Flood Control District, a special purpose district, in Texas acquired about 3,100 properties between 1985 and 2017, according to a 2018 report from Rice University, using a combination of FEMA grants, Corps funds, and local dollars. Charlotte-Mecklenburg Storm Water Services, a joint city-county utility in North Carolina, has acquired more than 400 homes since 1999. Initially, it primarily used federal funds, but now it uses almost solely stormwater fees and other local revenue to fund acquisitions. The utility’s Quick Buys program allows it to acquire properties soon after a flood, before homeowners invest in repairs, whereas federal acquisitions often occur after property owners have begun rebuilding, according to FEMA officials. New Jersey, through its Blue Acres program, plans to acquire up to 1,300 properties damaged by Superstorm Sandy. The program has used state funds, including $36 million in bonds, as well as more than $300 million in federal funding received from multiple agencies. Since 1989, the primary means by which FEMA has mitigated flood risk at the property level has been by funding property acquisitions. Acquisitions accounted for about 75 percent of FEMA’s $5.4 billion in flood mitigation spending, adjusted for inflation, from 1989 to 2018 (see fig. 3). Most of the remaining spending was used to elevate properties, with smaller amounts used to floodproof and relocate properties. The average federal cost-per-property was $136,000 for acquisitions and $107,000 for elevations, according to 2008-2014 FEMA data. As seen in figure 4, FEMA-funded property acquisitions have fluctuated over time but have generally increased since FEMA’s HMA programs began. For example, from 1989 through 1992—the first four years of HMGP funding and prior to the creation of PDM and FMA—less than $8 million, adjusted for inflation, was obligated for property acquisitions each year, resulting in fewer than 200 acquisitions each year (see fig. 4). The highest acquisition funding generally was associated with years that had significant flood events, such as Superstorm Sandy (2012) and Hurricanes Harvey, Irma, and Maria (2017). From fiscal years 1989-2018, approximately $3.3 billion of property acquisition funding, adjusted for inflation, occurred through HMGP, resulting in the acquisition of 41,458 properties (see fig. 5). HMGP represented about 90 percent of all property acquisitions and 82 percent of all acquisition funding, with PDM and FMA representing the remainder. As a result, most FEMA-funded acquisitions occurred following flood events. Most of the funding, adjusted for inflation, for HMGP’s and PDM’s flood mitigation projects has been for property acquisition (83 percent and 89 percent of total funds, respectively), while most FMA funding has been for elevation (49 percent). Although FEMA mitigated more than 57,000 properties for flood risk from 1989 to 2018, including more than 46,000 through acquisition, the number of nonmitigated RL properties increased from 2009 to 2018. Figure 6 shows that this growth in the number of RL properties has outpaced efforts to mitigate their flood risk. From 2009 through 2018, FEMA’s inventory of new RL properties grew by 64,101. During this period, FEMA mitigated 4,436 RL properties through its three HMA programs, and an additional 15,047 were mitigated through other federal or state programs. As a result, the number of nonmitigated RL properties increased by 44,618—more than double the number of RL properties that were mitigated in that time period. States varied in the extent to which they mitigated high-risk properties, including RL properties, between 1989 and 2018. While FEMA does not require a property to be an RL property to receive flood mitigation funding, the number of properties mitigated by a state relative to its population of RL properties provides context to its flood mitigation progress. For example, some states with large numbers of RL properties, such as Texas, Louisiana, Florida, and New York, mitigated few properties relative to their numbers of RL properties (see table 2). Other states, such as Missouri and North Carolina, have far fewer RL properties but have mitigated more properties relative to their numbers of RL properties. States also varied in their methods for flood mitigation (see table 2). For example, while property acquisition accounted for 81 percent of mitigated properties nationwide, it represented closer to half of mitigated properties in Virginia, New Jersey, and Florida and only 19 percent in Louisiana. According to some FEMA and local officials, high property values in some regions can make acquisitions cost prohibitive and other mitigation methods such as elevation more attractive because they do not incur the cost of purchasing the land. Many other factors could affect mitigation, including homeowners’ preferences. Further, the voluntary nature of FEMA’s HMA programs may limit states’ ability to acquire properties with known flood risk. According to FEMA, acquisition permanently addresses flood risk because, unlike elevation or floodproofing, it moves individuals and structures away from flood risk rather than mitigating a structure in place. In a subsequent report, we plan to explore in more detail the factors, including homeowner demand for acquisition, that have affected the extent to which states have used acquisition to mitigate flood risk. NFIP represents a fiscal exposure to the federal government because its premium rates have not kept pace with the flood risk of the properties it insures. Addressing this imbalance would mean reducing the flood risk of the insured properties, increasing premium revenue, or some combination of both. Despite FEMA’s efforts to mitigate its insured properties’ flood risk, premium rates for many properties do not reflect the full estimated risk of loss. As we have reported previously, mitigation alone will not be sufficient to resolve NFIP’s financial challenges; structural reforms to the program’s premium rates will also be necessary. NFIP’s total annual flood claim payments have grown in recent years, potentially indicating an increase in flood risk. For example, the eight years of the highest annual NFIP claims have all occurred since 2004, with particularly catastrophic flood events accounting for much of these claims: In 2005, claims reached $17.8 billion ($23.3 billion, adjusted for inflation), largely due to Hurricanes Katrina, Rita, and Wilma. In 2012, claims reached $9.6 billion ($10.7 billion, adjusted for inflation), largely due to Superstorm Sandy. In 2017, claims reached $10.5 billion ($11.0 billion, adjusted for inflation), largely due to Hurricanes Harvey, Irma, and Maria. These severe weather events appear to be contributing to the long-term increases in claims paid by NFIP, as would be expected with infrequent but severe events. As seen in figure 7, the amount of claims paid per policy, adjusted for inflation, does not show a steady increase in claims but rather substantial spikes in certain years associated with catastrophic flooding events. RL properties have contributed heavily to NFIP’s claims and, as noted earlier, the number of RL properties continues to rise despite FEMA’s mitigation efforts. Of the $69.7 billion in claims NFIP paid out from 1978 to 2019, $22.2 billion was for flood damage sustained by RL properties (32 percent). The frequency and intensity of extreme weather events, such as floods, are expected to increase in coming years due to climate change, according to the U.S. Global Change Research Program and the National Academies of Sciences. Further, numerous studies have concluded that climate change poses risks to many environmental and economic systems and a significant financial risk to the federal government. For example, according to the November 2018 National Climate Assessment report, the continued increase in the frequency and extent of high-tide flooding due to sea level rise threatens America’s trillion-dollar coastal property market. According to the National Oceanic and Atmospheric Administration, minor flood events (sometimes referred to as nuisance flooding) also are projected to become more frequent and widespread due to climate change. While it is uncertain the exact extent to which flood risk has changed and will continue to change, NFIP’s fiscal exposure will persist as long as premium rates do not keep pace with flood risk. As we have been reporting since 1983, NFIP’s premium rates do not reflect the full risk of loss because of various legislative requirements and FEMA practices. To set premium rates, FEMA considers several factors, including location in flood zones, elevation of the property relative to the community’s base flood elevation, and characteristics of the property, such as building type, number of floors, presence of a basement, and year built relative to the year of the community’s original flood map. Most NFIP policies have premium rates that are deemed by FEMA to be full-risk rates, which FEMA defines as sufficient to pay anticipated losses and expenses. However, FEMA’s overall rate structure may not reflect the full long-term estimated risk of flooding, as discussed below. Subsidized rates. NFIP offers some policyholders subsidized rates—that is, rates that intentionally do not reflect the full risk of flooding. These premium rates are intended to encourage the widespread purchase of flood insurance by property owners and encourage floodplain management by communities. Subsidized rates generally are offered to properties in high-risk locations (special flood hazard areas) that were built before flood maps were created. FEMA staff said they have begun increasing rates for certain subsidized properties as prescribed under the Biggert-Waters Flood Insurance Reform Act of 2012 and the Homeowner Flood Insurance Affordability Act of 2014. In addition, the percentage of subsidized policies is decreasing. According to FEMA data, the percentage of NFIP policies receiving subsidized rates dropped from about 22 percent in July 2013 to about 17 percent in June 2019. In 2013, we recommended that FEMA obtain elevation information to determine full-risk rates for subsidized properties. As of January 2020, FEMA had not fully implemented this recommendation but was in the process of doing so. For example, FEMA had requested proposals from third-party vendors for obtaining the elevation information and was reviewing these proposals. This information remains necessary for FEMA to determine the adequacy of its premium rates and the costs of any subsidization. It will also allow Congress and the public to understand the amount of unfunded subsidization within the program and the federal fiscal exposure it creates. Grandfathered rates. FEMA allows some property owners whose properties are remapped into higher-risk flood zones to continue to pay the premium rate from the lower-risk zone. FEMA data show that about 9 percent of NFIP policies were receiving a grandfathered rate as of June 2019. In 2008, we recommended that FEMA collect data to analyze the effect of grandfathered policies on NFIP’s fiscal exposure. As of February 2020, FEMA officials said they had not fully implemented this recommendation but were in the process of doing so. The officials told us they had finished collecting data on grandfathered policies and that they planned to analyze it as they completed efforts to update their premium rate setting approach. Collection and analysis of data on grandfathered policies will help FEMA understand and communicate the extent to which these policies are contributing to NFIP’s fiscal exposure. Rates designated full-risk. As we reported in 2008 and 2016, it is unclear whether premiums FEMA considers to be full-risk actually reflect the full long-term estimated risk of loss. For example, NFIP full-risk premium rates do not fully reflect the risk of catastrophic losses or the expenses associated with managing them. Private insurers typically manage catastrophic risk using capital, reinsurance, and other instruments, such as catastrophe bonds, and include the associated expenses in premium rates. By contrast, FEMA has traditionally managed catastrophic risk by relying on its authority to borrow from Treasury. In January 2017, FEMA began purchasing reinsurance to transfer some of its flood risk exposure to the private reinsurance market. However, FEMA has not accounted for these expenses in setting its NFIP premium rates. Reinsurance could be beneficial because it would allow FEMA to recognize some of its flood risk and the associated costs up front through the premiums it must pay to the reinsurers rather than after the fact in borrowing from Treasury. However, because reinsurers must charge FEMA premiums to compensate for the risk they assume, reinsurance’s primary benefit would be to manage risk rather than to reduce NFIP’s expected long-term fiscal exposure. Congress has directed FEMA to provide discounted premium rates to promote affordability for policyholders but did not provide FEMA with dedicated funds to pay for these subsidies. As a result, premium revenue has been insufficient to pay claims in some years, requiring borrowing from Treasury to make up for the shortfall. While Congress passed reforms to NFIP in 1994 and 2004, neither set of actions sufficiently addressed program revenue. In 2005, Hurricanes Katrina, Rita, and Wilma hit the Gulf Coast and resulted in NFIP borrowing nearly $17 billion from Treasury to pay claims (see fig. 8). In July 2012, Congress passed the Biggert-Waters Flood Insurance Reform Act, which contained significant reforms to NFIP’s premium rates. But a few months later, Superstorm Sandy occurred, pushing NFIP’s debt to $24 billion. Following policyholders’ concerns about the rate increases authorized by the 2012 act, Congress slowed the pace of many of these rate increases in 2014 with the Homeowner Flood Insurance Affordability Act. In the fall of 2017, Hurricanes Harvey, Irma, and Maria occurred, prompting additional borrowing from Treasury and causing NFIP to reach its borrowing limit. In response, Congress canceled $16 billion of NFIP’s debt in October 2017, which allowed NFIP to pay claims from these storms. Since September 2017, NFIP has been operating under a series of short-term authorizations, the most recent of which expires in September 2020. As of March 2020, NFIP’s debt remained at $20.5 billion. To improve NFIP’s solvency and enhance the nation’s resilience to flood risk, we suggested in 2017 that Congress could make comprehensive reforms that include actions in six areas. We reported that it was unlikely that FEMA would be able to repay its debt and that addressing it would require Congress to either appropriate funds or eliminate the requirement that FEMA repay the accumulated debt. However, eliminating the debt without addressing the underlying cause of the debt—insufficient premium rates—would leave the federal taxpayer exposed to a program requiring repeated borrowing. To address NFIP’s fiscal exposure, there are two general approaches: decrease costs or increase revenue. Decreasing costs to the program in the form of claims involves mitigating insured properties’ flood risks. Mitigation can be very costly, but there will be some properties for which the cost to mitigate will be outweighed by the benefit of reduced flood risk and, ultimately, fiscal exposure. Mitigation may be a cost-effective option for those properties for which full-risk rates would be cost-prohibitive. Increasing revenue would require reforms to NFIP’s premium rates. FEMA has begun increasing rates on subsidized properties. But, as we suggested in 2017, Congress could remove existing legislative barriers to FEMA’s premium rate revisions. Members of Congress and others have raised concerns about such reforms because raising premium rates may make coverage unaffordable for some policyholders. To address these concerns, we suggested that all policies include full-risk premium rates, with targeted, means-based, appropriated subsidies for some policies. This would improve the program’s solvency while also addressing affordability concerns. Assigning full-risk premium rates to all policies would remove subsidies from those who do not need them, helping improve solvency. It would also more accurately signal the true flood risk to property owners and enhance resilience by incentivizing mitigation measures, such as acquisition. Means-based subsidies would ensure that property owners who needed help would get it, and an explicit appropriation for the subsidies would make their true cost transparent to taxpayers. We maintain that a comprehensive approach that includes mitigation and rate reform is needed to address NFIP’s fiscal exposure. Because several categories of NFIP premium rates do not reflect the full risk of flood loss, FEMA has had to borrow $36.5 billion from Treasury to pay claims from several catastrophic flood events since 2005. To address this, some have suggested additional funding to mitigate RL properties. While we acknowledge that mitigation is part of the solution, we maintain that a more comprehensive approach is necessary to address the program’s fiscal exposure. We have made two recommendations to FEMA that, if implemented, could help inform Congress’ efforts to reform NFIP. In 2008, we recommended that FEMA collect information on grandfathered properties and analyze their financial effect on NFIP, and in 2013, we recommended that FEMA obtain elevation information on subsidized properties. By implementing these recommendations, FEMA would better understand NFIP’s fiscal exposure and be able to communicate this information to Congress. Further, we suggested in 2017 that Congress take a comprehensive approach to reforming NFIP. One important first step would be to implement full-risk premium rates for all policies, with appropriated means-based subsidies for some policies. Full-risk premium rates would remove subsidies from those who do not need them, helping improve solvency, and also more accurately signal the true flood risk to property owners and incentivize efforts to mitigate flood risk. Further, means- based subsidies would ensure that property owners who need help will get it, and having Congress explicitly appropriate for the subsidies would make the true cost of the subsidy transparent to taxpayers. While this would be an important step to putting NFIP on a sustainable path, comprehensive reform of the program should also address the other issues we have identified, including mitigating the flood risk of insured properties. We provided a draft of this report to the Department of Homeland Security for its review and comment. The agency provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report addresses the Federal Emergency Management Agency’s (FEMA) National Flood Insurance Program (NFIP). Our objectives were to examine (1) funding programs available for property acquisitions, (2) FEMA’s flood mitigation efforts, and (3) factors contributing to NFIP’s fiscal exposure. To describe funding programs available for property acquisitions, we reviewed authorizing legislation, the Code of Federal Regulations, and FEMA guidance and manuals, including the Hazard Mitigation Assistance Guidance and Cost Share Guide, to identify program characteristics, eligibility requirements, and application guidelines. To identify funding for these programs, we analyzed FEMA’s project-level Hazard Mitigation Assistance (HMA) data from its Enterprise Applications Development Integration and Sustainment system, which FEMA uses to track mitigation projects funded through its HMA grant programs. To summarize Increased Cost of Compliance coverage, which NFIP policyholders can use to fund mitigation efforts, we analyzed FEMA’s NFIP claims database to identify the number and amount of such claims. We also interviewed the FEMA officials responsible for administering these grant programs. Further, we identified other federal agency programs that can fund property acquisitions or meet cost share requirements and reviewed their authorizing legislation and their relevant federal regulations. Finally, to identify examples of state and local programs that have been used to fund property acquisitions, we reviewed academic reports, including from the University of North Carolina and Rice University. To review FEMA’s flood mitigation efforts, we analyzed FEMA’s project- level HMA data from the “Mitigation Universe” of its Enterprise Applications Development Integration and Sustainment system. We analyzed several variables in this dataset, including number of properties, federal share obligated, mitigation type category, grant program area, grant program fiscal year, and state. For the analyses by mitigation type category, we excluded projects (79 percent of the total records) that did not include a flood mitigation activity (those with values of “Other” or “Pure Retrofit”). Of the remaining records, 98 percent were “Pure,” meaning all properties within each project were of a single mitigation method type (acquisition, elevation, floodproof, or relocation). The remaining 2 percent were “Mixed,” indicating a project contained at least one acquisition and at least one elevation but could also contain other mitigation methods. For analyses by grant program area, we treated projects funded through the Severe Repetitive Loss and Repetitive Flood Claims grant programs as being part of the Flood Mitigation Assistance program and projects funded through the Legislative Pre-Disaster Mitigation program as being part of the Pre- Disaster Mitigation program. For data on the number of flood mitigated properties, we used the final number of properties mitigated by a project. For data on funding, we used the federal share of the project’s obligated funding. To analyze mitigated and nonmitigated repetitive loss (RL) properties, we summarized FEMA’s RL property mitigation report, which tracked the cumulative number of RL properties by year from June 2009 through June 2018. To describe the number of RL properties by state, we analyzed FEMA’s list of RL properties as of August 31, 2019, which included every property that at any point FEMA had designated as an RL property under any of its three definitions. The list included properties that had since been mitigated, as well as those that are no longer insured by NFIP. To examine factors contributing to NFIP’s fiscal exposure, we analyzed FEMA’s claims dataset as of September 30, 2019. This dataset includes the more than 2 million claims paid to NFIP policyholders since the beginning of the program. We excluded records whose status was “open” or “closed without payment.” Further, we excluded records whose year of loss was before 1978 because FEMA officials told us that that was the first year they considered their claims data to be reliable and complete. To identify factors that contribute to NFIP’s fiscal exposure and illustrate how this fiscal exposure has materialized and changed over time, we reviewed several of our previous reports and the Department of the Treasury’s statements of public debt. Finally, to summarize how flood risk could change in the future, we reviewed our previous reports on climate change. In general, we adjusted for inflation any dollar figures that we compared or aggregated across multiple years and indicated this accordingly. To do this, we used the Bureau of Labor Statistics’ Consumer Price Index for All Urban Consumers. To assess the reliability of all of the datasets we analyzed for this report, we requested and reviewed preliminary versions of the data and accompanying data dictionaries. We used the data dictionary to identify potential variables for use in our analyses and output statistics on these variables (e.g., frequencies of values, number of blanks or zero values, minimum, maximum, and mean) to identify any potential reliability concerns such as outliers or missing values. We met with relevant FEMA officials to discuss each of the data sets to understand how FEMA collected, used, and maintained the data; the reliability and completeness of key variables; reasons for any potential discrepancies we identified; and whether our understanding of the data and approach to analyzing them were accurate and reasonable. After these meetings, we requested updated versions of the data and updated our analyses accordingly. We determined that all data elements we assessed were sufficiently appropriate and reliable for this report’s objectives. We conducted this performance audit from January 2019 to June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. January 1983: We recommended that FEMA improve its rate-setting process to ensure adequate income for NFIP and suggested that Congress either limit FEMA’s borrowing for extraordinary losses or establish an emergency fund for such losses, and pay for NFIP subsidies with appropriations. March 1994: We found that NFIP’s premium income was insufficient to meet expected future losses because of subsidized rates and suggested that Congress consider how any changes in premium rates would affect policyholder participation. September 1994: National Flood Insurance Reform Act. Developed a mitigation assistance program and expanded the mandatory purchase requirement. June 2004: Flood Insurance Reform Act. Authorized grant programs to mitigate properties that experienced repetitive flooding losses. August-October 2005: Hurricanes Katrina, Rita, Wilma. Caused $17.1 billion in NFIP claims. FEMA debt to Treasury increased to $16.9 billion in fiscal year 2006. March 2006: We added NFIP to our high-risk list. October 2008: We recommended that FEMA collect data to analyze the effect of grandfathered policies on NFIP’s fiscal exposure. November 2008: We identified three options for addressing the financial impact of subsidies: increasing mitigation efforts; eliminating or reducing subsidies; and targeting subsidies based on need. June 2011: We suggested that Congress allow NFIP to charge full- risk premium rates to all property owners and provide assistance to some categories of owners to pay those premiums. July 2012: Biggert-Waters Flood Insurance Reform Act. Required FEMA to increase rates for certain subsidized properties and grandfathered properties; create a NFIP reserve fund; and improve flood risk mapping. October 2012: Superstorm Sandy. Caused $8.8 billion in NFIP claims. FEMA debt to Treasury increased to $24 billion in fiscal year 2013. February 2013: We added limiting the federal government’s fiscal exposure by better managing climate change risks to our high-risk list. July 2013: We recommended that FEMA obtain elevation information to determine full-risk rates for subsidized policyholders. March 2014: Homeowner Flood Insurance Affordability Act. Reinstated certain rate subsidies removed by the Biggert-Waters Flood Insurance Reform Act of 2012; established a new subsidy for properties that are newly mapped into higher-risk zones; restored grandfathered rates; and created a premium surcharge that would be deposited into the NFIP reserve fund. October 2014: We recommended that FEMA amend NFIP minimum standards for floodplain management to encourage forward-looking construction and rebuilding efforts that reduce long-term risk and federal exposure to losses. July 2015: We recommended that the Mitigation Framework Leadership Group establish an investment strategy to identify, prioritize, and guide federal investments in disaster resilience and hazard mitigation-related activities. August-October 2016: Hurricane Matthew and Louisiana floods. Caused $3.1 billion in NFIP claims. FEMA debt to Treasury debt increased to $24.6 billion in early fiscal year 2017. April 2017: We suggested that Congress make comprehensive reforms to NFIP that include actions in six areas: (1) addressing the debt; (2) removing legislative barriers to full-risk premium rates; (3) addressing affordability; (4) increasing consumer participation; (5) removing barriers to private-sector involvement; and (6) protecting NFIP flood resilience efforts. August-September 2017: Hurricanes Harvey, Irma, and Maria. Caused $10 billion in NFIP claims. FEMA reached the limit of its Treasury borrowing authority of $30.4 billion. September 2017: NFIP’s last long-term authorization ended, resulting in a string of short-term reauthorizations. October 2017: Congress canceled $16 billion of NFIP’s debt to enable FEMA to continue paying flood claims. This reduced FEMA’s debt to Treasury to $20.5 billion. March 2020: FEMA’s debt to Treasury remained at $20.5 billion. September 2020: NFIP’s current short-term authorization ends. Alicia Puente Cackley, (202) 512-8678 or cackleya@gao.gov In addition to the contact named above, Patrick Ward (Assistant Director), Christopher Forys (Analyst in Charge), Emily Bond, Christina Cantor, William Chatlos, Eli Dile, Lijia Guo, Holly Halifax, Laura Ann Holland, Yann Panassie, Stephen Ruszczyk, Jessica Sandler, Joseph Silvestri, Jena Sinkfield, and Kelsey Wilson made key contributions to this report.", "summary": "NFIP has faced significant financial challenges over the years, highlighted by a rise in catastrophic flood events and its $20.5 billion debt to Treasury. Contributing to these challenges are repetitive loss properties—those that have flooded and received a claim payment multiple times. Acquiring and demolishing these properties is one alternative to paying for repeated claims, but questions exist about the cost, efficiency, and effectiveness of this approach. GAO was asked to review FEMA's property acquisition efforts as a means of addressing NFIP's financial challenges. This report examines (1) funding programs available for acquisitions, (2) FEMA's flood mitigation efforts, and (3) factors contributing to NFIP's fiscal exposure. To conduct this work, GAO reviewed FEMA guidance and other documentation; analyzed FEMA data sets related to NFIP policies and claims, repetitive loss properties, and mitigation projects; and interviewed FEMA officials. The Federal Emergency Management Agency (FEMA) administers three grant programs that can fund efforts to mitigate the flood risk of properties insured by the National Flood Insurance Program (NFIP). Together, these three programs funded $2.3 billion in mitigation projects from fiscal years 2014 through 2018. The largest program's funding is tied to federal recovery dollars following presidential disaster declarations, while the other two programs are funded each year through congressional appropriations. States and localities generally must contribute 25 percent of the cost of a mitigation project, but some other federal program funds can be used for that purpose. One example of such a project is property acquisition—purchasing a high-risk property from a willing property owner, demolishing the structure, and converting the property to green space. From 1989 to 2018, FEMA has helped states and localities mitigate more than 50,000 properties; however, the number of nonmitigated repetitive loss properties (generally meaning those that flooded at least twice in 10 years) has grown. Mitigation efforts varied by state. Property acquisition accounted for about 80 percent of mitigated properties nationwide, but, in some states, elevation (raising a structure) was more commonly used. In addition, some states (e.g., Missouri and North Carolina) mitigated a high number of properties relative to their numbers of repetitive loss properties, while others (Florida, New York, Louisiana, and Texas) mitigated a low number. While these efforts can reduce flood risk and claim payments, the federal government's fiscal exposure from NFIP remains high because premium rates do not fully reflect the flood risk of its insured properties. NFIP has experienced several catastrophic flood events in recent years, and the frequency and severity of floods is expected to increase. However, NFIP's premium rates have not provided sufficient revenue to pay claims. As a result, FEMA still owed Treasury $20.5 billion as of March 2020, despite Congress cancelling $16 billion of debt in 2017. As GAO has reported in the past (GAO-17-425), Congress will need to consider comprehensive reform, including mitigation and structural changes to premium rates, to ensure NFIP's solvency. GAO suggested in GAO-17-425 that Congress make comprehensive reforms to NFIP to improve the program's solvency. Given NFIP's continued debt growth, GAO maintains that comprehensive reform warrants consideration.", "document_type": "gao"}
{"report": "Each major facilities project has a sponsoring office from within NSF’s seven research directorates. The sponsoring office assesses the scientific merit of a potential project, proposes projects for funding through NSF’s MREFC account, and is responsible for overseeing the project during the following five stages of its life cycle. Development. Initial project ideas emerge, and a broad consensus is built within the relevant scientific community for the potential long-term needs, priorities, and general requirements for research infrastructure that NSF may consider funding. Design. Entrance into this stage occurs when the NSF Director approves the proposed research infrastructure as a national priority and the sponsoring directorate makes an award (either through a cooperative agreement or contract) for developing detailed project cost, scope, and schedule for possible construction. This stage is divided into conceptual, preliminary, and final design phases. According to NSF documentation, the goal of the conceptual design phase is to create a comprehensive design that clearly articulates project elements that NSF will consider, such as a description of research infrastructure and technical requirements, a concept of operations, and an initial risk analysis, among others. The preliminary design phase further develops projects through the formulation of a site-specific scope, an accurate budget estimate, a revised and updated project execution plan, and other deliverables to establish a project baseline. In the final design phase, a candidate project will refine cost and contingency estimates, complete recruitment of key staff needed to undertake construction of the project, and develop the necessary documentation needed to undergo final design review. A candidate project will exit the design stage and enter the construction stage after a successful review by the NSF director and other key stakeholders of its project execution plan and authorization of its not- to-exceed total project cost by the National Science Board, as discussed below. Construction. The construction stage begins when NSF makes awards to external recipients for acquisition or construction of research infrastructure. Such awards generally take the form of cooperative agreements, although NSF occasionally uses contracts, according to agency officials. The policies and procedures in NSF’s Major Facilities Guide apply to research infrastructure projects regardless of the award instrument employed. According to NSF’s Major Facilities Guide, the transition from construction to operations could be a single acceptance event or multiple events depending on the nature of the project, and many projects require an integration and testing phase, followed by a commissioning phase to bring the facility up to the design level of operational readiness. The construction stage ends after final delivery and acceptance of the defined scope of work and facility performance per terms of the award instrument. Operations. The operations stage includes the day-to-day work necessary to operate and maintain the research infrastructure (including refurbishment or upgrade activities) and to perform research. Operations awards, which are separate from construction awards, may be made to the construction award recipients or to a different entity. Depending on the project, initial operations may begin before completion of construction. Integration and testing activities may continue during the operations stage, depending upon the complexity and time needed to reach design specifications. Divestment. Divestment can include the transfer of the research infrastructure to another entity’s operational and financial control or the decommissioning of the research infrastructure, including its complete deconstruction and removal. NSF generally decides to divest when the agency or the scientific community determines that the facility is no longer considered an operational priority with regard to advancing science, according to NSF’s Major Facilities Guide. NSF funding for the development, design, operations, and divestment stages generally comes from the sponsoring directorate. Funding for the construction stage generally comes from the MREFC account. However, if the sponsoring directorate funds construction, the policies and procedures in NSF’s Major Facilities Guide apply if total project costs meet the definition of a major multiuser research facility project under the American Innovation and Competitiveness Act—that is, if the costs exceed $100 million or 10 percent of the responsible directorate’s annual budget, whichever is less. NSF has established an oversight structure for major facilities projects that includes offices from across the agency (see fig. 1). This includes the National Science Board, a policy and advisory body that is part of NSF and consists of the NSF Director and 24 members, drawn from industry and universities, who represent a variety of science and engineering disciplines. The NSF Office of the Director and the National Science Board provide high-level, ongoing oversight of major facilities projects, including the approval of new projects to be included in NSF’s annual budget request. Within NSF’s Office of Budget, Finance, and Award Management, the Large Facilities Office (1) develops business-related oversight policies for all life-cycle stages with a focus on the design and construction stages and (2) provides assistance on nonscientific and nontechnical aspects of project planning, budgeting, implementation, and management. To that end, the office maintains the Major Facilities Guide, which contains NSF policies for agency staff and recipients on the planning, management, and oversight of major facilities. Prior to requesting the National Science Board’s authorization to include a proposed project in a future NSF budget request, the Large Facilities Office provides independent assurance—apart from the sponsoring office and external panels—that NSF oversight processes have been followed, project plans are construction ready, and construction and operations budgets are justified. In addition, it prepares a bimonthly status report for NSF leadership on all ongoing major facilities in construction and candidate projects in design. NSF also uses external panels of experts to review projects at several points during their life cycles. An external panel may first review a project proposal during the development stage. Separate panels then review the project at the culmination of each of its design phases. In addition, an external panel periodically reviews each project during both construction and operations; according to NSF officials, those reviews are generally on an annual basis. According to NSF officials and policy documents, the agency selects panelists based on the questions that need to be addressed and on the type of review taking place. For example, for panels charged with reviewing all aspects of a project, NSF will generally select panelists to represent the academic and broader national or international research community, as well as experts in administrative aspects of facilities and project management, according to NSF’s Major Facilities Guide. Furthermore, the responsible directorate and the Large Facilities Office jointly manage the external panel review process and other NSF staff may attend as observers, according to the agency’s Major Facilities Guide. Each panel is to provide NSF with a report summarizing the review’s findings and any recommendations to NSF. Under NSF’s major facilities construction process, the recipients of design awards develop construction cost and schedule estimates for projects and submit them to NSF for review. In particular, after a project’s final design review, the National Science Board authorizes a not-to-exceed award amount and an award duration. According to NSF officials, this finalizes the initial budget request previously submitted to Congress after the project’s preliminary design review. The not-to-exceed award amount that the National Science Board authorizes is the amount against which NSF measures cost increases to implement its no cost overrun policy. NSF’s Major Facilities Guide defines two components that together make up the total project cost and schedule for the construction of major facilities projects. The total project cost awarded in a project’s construction agreement may be less than the not-to-exceed cost but not more. These components of the total project cost and schedule are the following: Performance measurement baseline. During design, the cost, scope, and schedule are refined and eventually become the project baseline. Once the baseline has been authorized and included in a construction award, it is known as the performance measurement baseline. NSF documents the performance measurement baseline in the terms and conditions of the award instrument and requires that any changes to it be made through a formal change control process. The performance measurement baseline does not include the project’s budget or schedule contingency. Contingency. This is an amount of budget or time for covering the cost increases or delays that would result if foreseen project risks were to occur. During development of a total project cost estimate, the timing and impacts of such risks are uncertain. As a project progresses, the impacts of risks that materialize may exceed the cost or schedule in the performance measurement baseline and lead to use of the project’s budget or schedule contingency. According to NSF’s Standard Operating Guidance on budget contingency, it is likely no contingency will be left over by the end of a project because all of it will have been used during normal execution of the project to manage known risks and uncertainties. NSF approval is needed when use of contingency exceeds certain project-specific thresholds, which are described in the project’s execution plan and codified in the award. In this report, we identify total project costs for the construction of major facility projects which were developed during the design phase based on the latest estimates available from NSF officials; those estimates are subject to change before construction awards are made. For projects under construction, we identify total project costs based on the amounts awarded in the cooperative support agreements for construction and the not-to-exceed amounts authorized by the National Science Board. Only at the end of the projects—when construction is complete and the awards have been closed out—will the final total project costs be known. In addition to the performance measurement baseline and budget contingency, a project’s not-to-exceed cost that the National Science Board authorized may include the following: Fee. NSF may provide recipients the opportunity to earn a fee (formerly referred to by NSF as a management fee) for major facilities projects. According to NSF’s Standard Operating Guidance on negotiation, award, and payment of a fee, such a fee can stimulate efficient performance. Management reserve. NSF, not the award recipient, holds management reserve to manage budget uncertainties, unforeseeable events, and risks that the recipient is not able to manage, according to NSF officials. According to agency officials and the Major Facilities Guide, NSF does not hold a management reserve except in rare circumstances. Since February 2008, NSF has had a policy to manage cost overruns on major facilities projects. Under this policy, the cost estimate developed at the preliminary design review should have adequate contingency to cover all foreseeable risks. Any cost increases not covered by contingency are generally to be accommodated by reductions in scope. Figure 2 provides a breakdown of the total project cost components in relation to the not-to-exceed award amount. NSF officials said that under this policy, they will only request an increase to the not-to-exceed cost that the National Science Board authorized if the recipient cannot address the increase through use of the project’s budget contingency or acceptable reductions to the project’s scope. Accordingly, at the preliminary design review, projects must have a prioritized, time-phased list of options for reducing scope during construction, known as scope contingency, and the potential cost savings associated with those options is to total at least 10 percent of the project’s baseline. As defined by NSF’s Major Facilities Guide, scope contingency is scope that can be removed without affecting the overall project’s objectives but that may still have undesirable effects on facility performance. As of September 2019, NSF continued construction of three major facilities projects with no changes to their authorized total project costs or scheduled completion dates since our March 2019 report. In addition, NSF approved a fourth project to enter the construction stage, completed construction of one project, and advanced two major facilities projects in the design stage. The four major facilities projects under construction have a combined total cost of approximately $1.6 billion (see table 1). Ongoing construction projects. Three projects—the Daniel K. Inouye Solar Telescope, the Rubin Observatory and the Regional Class Research Vessels—continued construction with no changes to their authorized total project costs or scheduled completion dates since our March 2019 report. Instead, NSF managed cost increases on the projects through the use of budget contingency, as specified under its no cost overrun policy, and managed delays through the use of schedule contingency. For example, the Rubin Observatory utilized $11.9 million in budget contingency and 5 months of schedule contingency to better align testing of the camera within the project schedule due to delays associated with the completion of the dome enclosure and telescope mount assembly, among other delays. The project team for the Rubin Observatory is also evaluating scope reduction options in order to complete the project within its total project cost and by its scheduled completion date of October 2022. New construction project. In February 2019, the National Science Board authorized a not-to-exceed total project cost of $410.4 million for the AIMS project and NSF awarded an initial contract modification for construction. We previously reported that in NSF’s fiscal year 2019 budget request, the estimated total project cost for construction of the AIMS project was $355.0 million. By the project’s final design review in October 2018, the AIMS team determined that it could not execute the project with the desired scope for this amount because of changing market conditions. NSF evaluated scope reduction options for the project but decided to maintain the project’s scope at the higher total project cost of $410.4 million. This change in total project cost did not count as an increase under NSF’s no cost overrun policy because the previous amount had not been authorized by the National Science Board as the project’s not-to-exceed cost. Completed construction project. In May 2019, NSF completed construction of the National Ecological Observatory Network project within the $35.5 million cost increase authorized by the National Science Board and a schedule increase of 2.8 years (57 percent). In 2011, NSF made the original award for construction of this nationwide network of ecological observation sites which was planned for completion in July 2016 at a total project cost of $433.8 million. In 2017, NSF increased the not-to-exceed cost for the project to $469.3 million. In accordance with NSF’s no cost overrun policy, the NEON project implemented scope reductions, such as reducing the number of observation sites from 106 to 81 and eliminating certain scientific instruments at the project’s observation sites. The scope reductions resulted in an estimated cost savings of $62.4 million. According to NSF documentation as of November 2019, NSF obligated a total of $458.9 million from the MREFC account for the construction of NEON, $10 million below the authorized total project cost. As of January 2020, NSF extended the construction stage award for NEON to allow for award close-out activities, which NSF officials expected to be complete in August 2020. Projects in design. In addition, in 2019, NSF advanced the design of two major facilities projects in the design stage, the Large Hadron Collider High Luminosity Upgrade (HL-LHC) and the Leadership- Class Computing Facility (LCCF). Under NSF policy, a major facility project’s cost, scope, and schedule are not finalized until after the final design review, when the National Science Board authorizes a not-to- exceed cost and an award duration. The not-to-exceed cost that the National Science Board authorized is the amount against which NSF measures cost increases to implement its no cost overrun policy. In September 2019, NSF convened two external panel reviews for the final design of the two separate detector upgrades that make up the HL-LHC program. According to NSF officials, the panels recommended to the NSF Director that the detector upgrades proceed to the construction stage. According to NSF documentation dated November 2019, the HL-LHC program had an estimated total project cost of $150 million for both upgrade projects. However, this amount was subject to change since the projects had not yet been authorized by the National Science Board to advance to the construction stage. According to NSF officials, the National Science Board authorized the total program cost at $153 million in early February 2020, setting the not-to-exceed costs for both awards. The LCCF project entered the conceptual design phase in March 2019. As of September 2019, the LCCF project had not developed an initial estimated total project cost because it had so recently entered design. Further details on the two projects in design are located in appendix II. NSF has fully implemented two of the six recommendations we made in June 2018 and March 2019—recommendations on policies for estimating the costs of major facilities projects and revising the Rubin Observatory’s schedule to better meet best practices. NSF has taken steps to address but has not fully implemented the remaining four recommendations concerning the agency’s management of major facilities, specifically our recommendations on policies for developing schedules for major facilities projects, project management competencies of the agency’s major facilities project management expertise of award recipients for major facilities ensuring the sharing of lessons learned or best practices on major facilities projects. Cost estimating policies. In our June 2018 report, we found that procedures documented in NSF’s policies for major facilities projects fully or substantially met many best practices and partially or minimally met others identified in GAO’s guide for developing project cost estimates. Specifically, we found that NSF’s procedures fully or substantially met seven of the 12 best practices in GAO’s cost guide and partially or minimally met the remaining five, such as the best practice for conducting a sensitivity analysis to understand which variables most affect the cost estimate. The American Innovation and Competitiveness Act requires that NSF ensure that its policies for estimating and managing costs and schedules are consistent with the best practices in GAO’s cost guide, and NSF requires the same of its recipients. We recommended that NSF revise the agency’s policies for estimating the costs of major facilities projects, and for reviewing those costs, to better incorporate best practices. In response, NSF revised its Major Facilities Guide and certain internal Standard Operating Guidance policies that documented procedures for estimating costs. In our current assessment of these revised guidance and policy documents, we found that NSF fully met the five cost estimating best practices in GAO’s cost guide that we previously found were minimally or partially met. For example, in our 2018 report, we concluded that NSF’s procedures required a sensitivity analysis but did not describe how one is to be conducted. In our updated assessment, we found that NSF’s procedures describe the best practice and how it should be applied to NSF major facility cost estimates. Specifically, the procedures describe, among other things, (1) identifying key variables—cost drivers, ground rules, and assumptions—for inclusion in the analysis, with examples particular to NSF major projects included as part of the procedures; (2) evaluating the effect of these variables on the cost estimate by varying them one at a time; and (3) developing a strategy to deal with the variables to which the estimate is most sensitive. Table 2 provides an overview of our original and updated assessments of NSF’s cost estimating policies. Between our June 2018 assessment and our current assessment, NSF’s policies substantially or fully met all 12 of the best practices in GAO’s cost guide. Rubin Observatory schedule. In our March 2019 report, we found that the Rubin Observatory’s schedule could not be considered reliable because it did not substantially or fully meet all four characteristics of a reliable schedule from GAO’s schedule guide—comprehensive, controlled, well-constructed, and credible, as described in table 3. While the schedule substantially met the comprehensive and controlled characteristics, it partially met five scheduling best practices associated with the well-constructed and credible characteristics. Specifically, we found certain issues related to the construction of the project’s schedule, including (1) the sequencing of activities, (2) the schedule’s critical path— a chain of dependent activities that drive a project’s earliest completion date, and (3) the amount of float calculated in the schedule—the amount of time by which a project activity can slip before the delay affects the project’s estimated completion date. We recommended that NSF ensure that the project’s schedule meets the well-constructed and credible characteristics of a reliable schedule, as defined in GAO’s schedule guide. Our current assessment found that the revised schedule addressed our recommendation. Specifically, the schedule substantially met four of the five best practices that we previously found had been partially met within the well-constructed and credible characteristics of a reliable schedule and partially met the remaining best practice (ensuring reasonable total float). Between our two assessments, the Rubin Observatory project’s schedule substantially or fully met the four characteristics and nine of the 10 best practices in GAO’s schedule guide. We consider NSF’s actions sufficient to address our recommendation. Table 3 provides our original and current assessments of the Rubin Observatory project’s schedule. In addition to implementing two of our recommendations, NSF has taken initial steps to address the other four recommendations from our June 2018 and March 2019 reports, but has not fully implemented them. Once NSF completes the steps discussed below, we will evaluate its actions to determine whether they are sufficient to fully address our recommendations. Policies for developing project schedules. In our June 2018 report, we found that NSF’s procedures for recipients substantially met one of the 10 best practices for developing project schedules—the best practice on conducting a schedule risk analysis. In contrast, NSF’s procedures partially or minimally met six and did not meet three of the remaining best practices. For example, we found that NSF’s procedures did not meet the best practice of establishing the durations of all activities because the NSF documents we reviewed did not include policy or guidance related to this practice, such as guidance on using realistic assumptions in estimating durations. The American Innovation and Competitiveness Act requires that NSF ensure that its policies for estimating and managing costs and schedules are consistent with the best practices in GAO’s schedule guide, and NSF requires the same of its recipients. We recommended that NSF revise its policies for developing schedules for major facilities projects, and for reviewing those schedules, to better incorporate the best practices in GAO’s schedule guide. As of November 2019, NSF had updated its internal guidance on standardized cost analysis to include a new section related to schedule reviews to help address this recommendation. This guidance states that the NSF Large Facilities Office will lead analysis of the schedule for each proposed major facilities project, which will include a technical evaluation by the sponsoring office, and may include input from an independent cost estimate and schedule review, or other reviews. As further steps to implement this recommendation, NSF plans to update two other policy and guidance documents, according to NSF officials. Specifically, NSF plans to: develop a new section of the Major Facilities Guide on schedule development, estimating, and analysis and post the guidance for public comment; and develop new internal guidance to help NSF staff more fully utilize external panels to address elements of schedule—in addition to cost—as part of the panels’ oversight reviews. According to NSF officials, they plan to complete these actions by the end of fiscal year 2020. Once NSF completes these actions, we will re-assess NSF’s procedures against the nine best practices that NSF partially or minimally met or did not meet in the assessment we conducted for our June 2018 report. Project management competencies of NSF’s major facilities oversight workforce. In our March 2019 report, we found that NSF had not (1) assessed potential gaps in how well its key major facilities oversight staff met project management competencies or (2) developed human capital plans for its major facilities oversight staff to address any gaps that may exist. Taking these steps would be consistent with leading principles for strategic workforce planning that we and the Office of Personnel Management have previously identified. Therefore we recommended that NSF assess its major facilities oversight workforce to identify any project management competency gaps, develop a plan to address any gaps and time frames for doing so, and monitor progress in closing them. In September 2019, in response to our recommendation, NSF awarded a contract for a proficiency assessment and workforce gap analysis. NSF expects this analysis to assess the core competencies and necessary proficiency levels of agency staff overseeing the major facilities portfolio and promote long-term workforce development. According to contract documentation, the contractor will take the following actions, among others: conduct a proficiency assessment and gap analysis based on a review of existing workforce materials, such as relevant position descriptions, vacancy announcements, performance plans, and other NSF guidance documents; work with NSF staff to refine competency guidance to better meet needs of the agency; and work with NSF to update training plans as necessary, based on the findings in the gap analysis and a review NSF’s existing training plan. According to contract documentation, NSF anticipates finishing the competency assessment and workforce gap analysis by the second quarter of calendar year 2020 and the implementation of contract tasks by March 2021. According to NSF officials, depending on the results of the assessment and analysis, improvements to address any identified gaps may involve developing standards of performance for the oversight workforce, identifying training opportunities in support or workforce development, and clarifying minimum competency requirements. Project management expertise of award recipients for major facilities projects. In our March 2019 report, we found that NSF had some procedures in place to help ensure that award recipients had project management expertise, but that the agency had not established criteria for the expertise needed by recipients or how they should demonstrate it. We concluded that, as a result, NSF was at risk of making awards to organizations that may not be well qualified to manage construction of major facilities projects. We recommended that NSF establish criteria for the project management expertise of award recipients for major facilities projects and incorporate the criteria in project requirements and external panel reviews. As of November 2019, NSF had drafted new language for the Major Facilities Guide and related supplemental award terms and conditions for major facilities that would require award recipients to document how project management competencies will be met. NSF officials told us they had shared the draft documents with targeted recipient representatives for review and comment in September 2019. NSF officials stated that the supplemental terms and conditions are planned to be published in fiscal year 2020, with an effective date of June 2020. The officials also said that, for existing awards, the agency will work with recipients on a phased implementation of the new guidance and terms and they will automatically be incorporated into future awards. Sharing of lessons learned or best practices on major facilities projects. In our March 2019 report, we found that NSF formalized a process for identifying and sharing lessons learned on major facilities projects. The process, which NSF refers to as its knowledge management program, responded to a 2015 recommendation by the National Academy of Public Administration and to the American Innovation and Competitiveness Act’s requirements that NSF coordinate the sharing of best management practices and lessons learned from major facilities projects. We recommended that NSF ensure, through a requirement or other means, that award recipients for major facilities projects provide information to NSF on any lessons learned or best practices. NSF developed supplemental award terms and conditions for major facilities to require recipients to participate in NSF’s knowledge management program. According to NSF officials, among other things, the requirement can be met by recipients: sending appropriate staff to the annual major facilities workshop that NSF hosts to provide a collaborative forum for continuous learning and information sharing among participants; presenting lessons learned or good practices at the annual workshop; participating in a workshop planning committee; or providing lessons learned or good practices to NSF. According to NSF officials, the draft terms and conditions will be included in the same revision as those related to recipients’ project management expertise, planned for publication in fiscal year 2020. As described above, NSF officials said that for existing awards, the agency will work with recipients on a phased implementation of the new terms and conditions, and they will automatically be incorporated into future awards. According to NSF documentation, NSF requested $45 million for fiscal year 2020 within the MREFC account to fund its first set of mid-scale projects with a total project cost between $20 million and $70 million. In response to a solicitation it issued in December 2018, NSF received approximately 50 preliminary proposals for mid-scale projects from research areas spanning all of NSF’s directorates, according to NSF officials. NSF invited 14 of these applicants to submit a full proposal and received full proposals from 11. The solicitation specified a list of information each full proposal should contain, including a project summary and description, a budget, and a project execution plan. NSF is currently reviewing the full proposals and expects to award its first portfolio of mid-scale projects in August 2020, according to NSF documentation. NSF’s solicitation anticipated that $150 million will be available over five years to fund its first batch of mid-scale projects. According to NSF officials, NSF plans to award subsequent sets of mid- scale projects biennially, depending on the availability of funds for future projects. According to NSF’s solicitation, the agency is seeking prospective mid- scale projects that are innovative and potentially transformative, that include a strong component of student training, and that provide unique research capabilities relative to what currently exists in the research community. Based on the definition of mid-scale projects in the American Innovation and Competitiveness Act, the solicitation stated that NSF would consider upgrades to existing major facilities projects currently in operation as candidates for mid-scale projects. The solicitation required full proposals to describe the full life cycle cost and schedule—including development, design, implementation, operations, and divestment. According to agency officials, NSF is only seeking to fund construction and acquisition costs from the MREFC account but needs to understand potential cost impacts on other life cycle stages. According to NSF officials, the mid-scale program is designed to identify potential projects with shorter implementation timelines and high levels of readiness as compared to the multiyear, incremental refinements to cost, scope, and schedule that occur with major facilities projects. NSF officials also stated that, to assess the readiness of the mid-scale projects for which full proposals were received, the agency will use an internal proposal review process similar to the final design review process used for major facilities projects. In addition, NSF policies state that there can be multiple inputs to the proposal review process, such as external panels or ad hoc reviews, which ensure that the mid-scale projects NSF awards will reflect the needs and interests of the scientific community. To provide guidance on oversight for mid-scale projects, NSF has included a chapter in its September 2019 update of the Major Facilities Guide to outline minimum recipient requirements and NSF oversight activities for mid-scale projects. In addition, NSF has created a management plan for NSF personnel that outlines procedures for reviewing proposals, selecting mid-scale projects, and managing the award process. NSF last updated the plan in November 2019, and according to NSF officials, the agency will continue to update the plan as it leads its initial set of projects from award to execution. According to NSF officials, oversight requirements for mid-scale projects will be dependent upon the technical scope and complexity of each individual project. As a result, NSF has tailored its guidance to provide the level of oversight commensurate with each project’s technical scope, type and mix of work, and risk profile. In addition, NSF is incorporating some aspects of its existing guidance for major facilities projects into its guidance for mid-scale projects. However, NSF officials anticipate that mid-scale projects will be less complex than major facilities projects. The following describes aspects where NSF has adapted its guidance for major facilities projects to the lower level of complexity anticipated for mid-scale projects. Performance measurement baselines. Similar to major facilities projects, NSF requires that the scope, cost, and schedule for mid-scale projects be defined at the time of award. In addition, NSF requires budget management, cost controls, and identification of potential risks and mitigation strategies for mid-scale projects, and its guidance states that budgets should be developed in accordance with GAO’s cost estimating best practices. While NSF officials state that NSF will apply substantial rigor in assessing the defined total project cost, mid-scale projects will not be subject to NSF’s no-cost-overrun policy. As a result, unlike for major facilities projects, NSF will not require all mid-scale projects to include budget contingency and scope reduction options, both of which are necessary for implementing the no-cost-overrun policy, although it may choose to include contingency in the budgets for certain mid-scale projects. For those mid-scale projects that have budget contingency, they must follow guidance for budget contingency laid out in the Major Facilities Guide, such as obtaining approval from NSF for using budget contingency. Monitoring and assessment. Like major facilities projects, NSF will monitor the award progress of mid-scale projects through periodic reports that provide quantifiable measurements on technical progress as well as cost and schedule performance. Depending on the complexity of each project, annual site visits or reviews may also be conducted. However, recipients of mid-scale projects may use alternatives to an earned value management system to report progress, such as reporting on milestone events or expenditure reports. According to NSF officials, the burden of establishing an earned value management system for some mid-scale projects may outweigh the benefits of using such a system, depending on the technical nature of the project. Project execution plan. According to the Major Facilities Guide, NSF will require a project execution plan for all mid-scale projects to demonstrate how recipients will manage the projects. A project execution plan serves as the stand-alone document that explains all of a project’s requirements for execution. According to NSF officials, the project execution plan used for major facilities projects would be excessive for mid-scale projects and may discourage potential proposals. Thus, NSF guidance for mid-scale projects requires only nine of the 16 sections normally required in a project execution plan and allows the recipients to tailor the detail and scope of each section to the specifics of each project. In addition, NSF will not require mid-scale projects to include design and development plans or site and environment information, which are required sections for major facilities projects. Since it is only funding the construction of mid- scale projects and seeking to award projects with high levels of readiness, NSF does not consider these sections to be beneficial in assessing how a recipient would manage a mid-scale project. We provided a draft of this report to NSF for review and comment. In its comments, reproduced in appendix III, NSF stated that our report provides the agency with an independent assessment of its oversight of projects in design and construction and its stewardship of the MREFC account. With regard to our recommendations on policies for estimating the costs of and developing schedules for major facilities projects, NSF stated it is proud of the progress it has made in meeting GAO best practices for cost estimating on major facilities projects and that it recognizes the remaining work needed to codify NSF guidance on project schedules. NSF also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Director of the National Science Foundation, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6888 or neumannj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This appendix provides individual summaries of the National Science Foundation’s (NSF) four major facilities projects under construction: (1) the Daniel K. Inouye Solar Telescope, (2) the Vera C. Rubin Observatory, (3) the Regional Class Research Vessels, and (4) the Antarctic Infrastructure Modernization for Science. Each project’s summary is based on project documents and other information that NSF officials provided and includes the following: An overview of the project and its purpose. A timeline identifying key project dates, including the date of the original construction award, which we report as the start of construction. Project information, such as the project’s estimated completion date for construction (including schedule contingency), the type and latest amounts of the awards for construction, the responsible NSF directorate, project partners, and expected duration of operations. Tables summarizing the project’s current status and its cost, any cost or schedule increases or scope reductions made under NSF’s no cost overrun policy, and changes since our March 2019 report. A summary of the project’s cost and schedule performance history. A chart depicting the latest construction award’s total project cost for construction, including the performance measurement baseline and budget contingency. If applicable, a chart showing the increase in the construction award’s total project cost since the original construction award. Information on remaining project risks and potential for cost or schedule increases, including the amount of remaining contingency and scope reduction options. When completed, the National Science Foundation’s (NSF) Daniel K. Inouye Solar Telescope (DKIST), formerly named the Advanced Technology Solar Telescope, will be the world’s flagship facility for the study of magnetic phenomena in the solar atmosphere. It will help answer fundamental questions in solar physics and enable understanding of solar variability and activity, which can affect Earth through phenomena generally described as space weather. Construction of NSF’s DKIST project was 94 percent complete as of September 2019. The project was in its 10th year of construction and in the integration, testing, and commissioning phase. Since our March 2019 report, the project completed installation of all telescope optics. Testing of the optics, originally planned for October 2019, was delayed until January 2020 to allow the project to replace a key piece of equipment that is essential to safely perform the testing. Despite the delay, the estimated completion of construction and beginning of full operations remained unchanged at June 2020, including 1.5 months of schedule contingency. Estimated construction completion date, including schedule contingency: June 2020. Construction award: Cooperative support agreements with the Association of Universities for Research in Astronomy, Inc., consisting of 42 U.S. institutional members and five international affiliates. Responsible NSF directorate: Mathematical and Physical Sciences. Project partners: More than 20 U.S. and international organizations. Kiepenheuer-Institut für Sonnenphysik (Germany) and Queens University Belfast (Northern Ireland) are supplying additional equipment for the project. Expected duration of operations: 50 years. Legend: ▲ = cost or schedule increase; ▼= scope reduction. NSF’s DKIST project had no changes to its authorized total project cost, June 2020 completion date or project scope since our March 2019 report, which used data as of September 2018. From April to November 2019, NSF approved the project’s use of about $6.2 million in budget contingency, with the largest usage of about $4.6 million in August 2019. Project delays requiring use of 3 months of schedule contingency— primarily because the project faced challenges with the installation and testing of the mirror systems, as described above—accounted for $4.3 million of the $4.6 million. We previously reported that the DKIST project’s risk of delays had the potential to increase costs for such items as labor, utilities, real estate, and equipment. NSF officials stated that most of the activities at risk of further delays would be achieved during the testing planned for January 2020. In 2013, NSF increased DKIST’s total project cost and the not-to-exceed cost that the National Science Board authorized from $297.9 million to $344.1 million, an increase of $46.2 million (16 percent) since 2010. NSF also delayed the project’s estimated completion date by about 2.5 years (31 percent), from December 2017 to June 2020. Prior to the National Science Board’s authorization to increase the total project cost, the recipient also reduced DKIST’s scope, resulting in estimated cost savings of $5.9 million but generally low expected impacts for the project. According to NSF officials, these cost and schedule increases resulted primarily from unforeseeable legal and administrative challenges to the construction site’s environmental permits. Remaining Project Risks and Potential for Cost or Schedule Increases As of September 2019, the DKIST project had $7.8 million of budget contingency remaining—$0.4 million more than the estimated remaining risk exposure of about $7.4 million when weighted for the risks’ probability. The project also had 1.5 months of schedule contingency remaining to help avoid any potential delays in completing construction. According to the project documentation, the largest remaining risk category is project completion and closeout risks. As of October 2019, 10 risks in this category remained, some of which had been partially realized, according to NSF officials, with about $4.0 million in risk exposure when weighted for probability. The remaining risks included staff retention as the construction project nears completion, and damage to or wear of equipment during integration and commissioning. For example, contingency may be needed to make minor repairs to the dome enclosure in preparation for full operations. In accordance with NSF policy, the project maintains a list of scope reduction options, which as of October 2019 included approximately $56,700 in total possible project de-scopes, such as reductions in travel. However, the ability of these remaining de-scope options to reduce costs will continue to decrease as the project continues to spend down remaining funds as it approaches completion. The National Science Foundation’s (NSF) Vera C. Rubin Observatory (Rubin Observatory), formerly named the Large Synoptic Survey Telescope (LSST), is an 8.4- meter, wide-field optical telescope. It will initially be used to image the entire visible southern sky—every 3 days for a decade—using the world’s largest digital camera (3.2 billion pixels). Built on a mountaintop in Chile to take advantage of the location’s pristine skies, the observatory will collect data and images that will allow for charting billions of galaxies as well as increased knowledge about potentially hazardous asteroids, dark matter, and dark energy. The observatory has the potential to advance every field of astronomical study, from the inner solar system to the large-scale structure of the universe. As of September 2019, the Rubin Observatory was 75 percent complete and in its sixth year of construction. NSF made the initial operations award in October 2018, and NSF officials anticipate completion of construction and start of full operations in October 2022, including contingency. Since our March 2019 report, the project has experienced delays related to both the telescope’s dome enclosure and mount assembly, leading NSF to add the project to the Director’s Watch List. Estimated construction completion date, including schedule contingency: October 2022. Construction award: Cooperative support agreement with the Association of Universities for Research in Astronomy, Inc., consisting of 42 U.S. institutional members and five international affiliates. Responsible NSF directorate: Mathematical and Physical Sciences. Project partners: The LSST Corporation, Department of Energy. Expected duration of operations: 50 years. Legend: ▼= scope reduction. Since our March 2019 report, NSF’s Rubin Observatory project had no changes to its authorized total project cost and implemented one scope reduction option valued at $1.4 million to increase available budget contingency. In addition, the project utilized $11.9 million in budget contingency and 5 months of schedule contingency to better align the testing of the camera within the project schedule. According to project documentation, the use of schedule contingency was due to delays with completion of the telescope mount assembly and dome enclosure that will house the telescope and other buildings. NSF officials attributed the delays to contractor performance and adverse weather conditions. For example, due to high winds, the project was able to use a crane to complete dome construction for only two days in September 2019. Remaining Project Risks and Potential for Cost or Schedule Increases Project data on the remaining risks and contingencies and the findings of two recent reviews indicate that the final cost of the Rubin Observatory may exceed the not-to-exceed cost authorized by the National Science Board, unless the project implements scope reduction options under NSF’s no-cost-overrun policy. As of September 2019, the project had an estimated remaining risk exposure of $26.4 million, which is equal to the remaining budget contingency of $26.4 million. In addition, the project had 3.5 months of schedule contingency remaining as of September 2019 to help avoid any potential delays in completing construction by October 2022. According to project documentation, the project’s largest remaining risks included delays in the completion of the telescope’s dome enclosure, the installation of the mount assembly, and delivery of the camera from the Department of Energy (DOE). The project team is modifying activity plans to mitigate these delays. For example, the project plans to complete dome enclosure and telescope mount assembly activities in parallel. As part of the Director’s Watch List, NSF plans to closely track updates on the project, including potential execution of scope reduction options. In August 2019, NSF and DOE jointly convened an external committee of experts to review the project’s construction progress. The committee found that the project may face difficulty in completing the baseline scope within the authorized total project cost. Specifically, the committee expressed concerns with the rate at which schedule contingency has been used (5 months of schedule contingency within the past 18 months), delays in completing the dome due to contractor performance issues, and the risks associated with maintaining an aggressive schedule composed of parallel activities in order to minimize further delays. The review committee recommended that NSF direct the project team to develop a proposal for executing scope reductions in fiscal year 2020 to complete the telescope within an acceptable level of risk at the current total project cost, among other recommendations. Remaining Contingency and Scope Reduction Options As of September 2019 with construction 75 percent complete. Budget contingency: $26.4 million (Equal to the probability- weighted risk exposure of $26.4 million). Schedule contingency: 3.5 months (included in the October 2022 estimated completion date). Estimated value of remaining scope reduction options: $24.8 million. as compared to a separate risk exposure analysis from July 2019 that indicated a 50 percent confidence. The panel recommended that the project report risk based on the analysis with the lower confidence level and conduct more frequent risk exposure analyses based on changes that have occurred, such as the realization or retirement of identified risks, to better inform management decisions. According to NSF documentation, the project team has recently acquired enhanced risk management software for analyzing risk exposure, including the effects of mitigating actions within the schedule. In a July 2019 update to its scope management plan, the project team identified 39 scope reduction items with a total value of $25.0 million. Among them is a de-scope option for reducing the amount of final commissioning surveys that may potentially return $4.3 million of budget contingency and 3.5 months of schedule contingency. According to NSF officials, NSF has yet to evaluate the impact of reducing the surveys to the project’s capabilities or operational costs. According to the external panel review convened by NSF and DOE, the project team identified potential scope reductions options valued at $14 million that the project can exercise in fiscal year 2022. However, the panel questioned the feasibility of executing the project’s scope reduction options and recommended that the project prioritize viable options while pursuing a no-cost extension to complete the project without an increase to the total project cost. The U.S. Department of Energy (DOE), a cosponsor of the Rubin Observatory, is responsible for delivering the observatory’s camera at a cost of $168 million. SLAC National Accelerator Laboratory manages a collaboration of DOE national laboratories and universities to develop, fabricate, and deliver the camera. As of September 2019, the project had the camera integration on the telescope scheduled for September 2021. Budget contingency accounts for the risk of a delayed delivery that would impact integration. The LSST Corporation is a not-for-profit organization representing nearly 40 institutional members and 34 international contributors. It acts as the agent for nonfederal funding contributed to the project and has raised more than $50 million for certain long-lead construction items and additional development efforts. The National Science Foundation’s (NSF) Regional Class Research Vessels (RCRV) project will construct three 199-foot vessels to support the nation’s ability to conduct fundamental scientific research in the coastal zone and continental shelf, including from the ocean’s surface through the water column to the sea floor and subsea floor environment. These vessels will provide enhanced capabilities beyond those of the retiring vessels they will replace. The three vessels’ research locations will depend on locations of the greatest science demand, but NSF planned to operate the first vessel along the west coast, the second along the east coast, and the third along the gulf coast of the United States. As of September 2019, NSF’s RCRV project was 20 percent complete and was in its third year of construction. Since our March 2019 report, the project progressed with construction of the first vessel and began construction of the second vessel in September 2019. NSF also awarded funds for construction of the third vessel, which was scheduled to begin in March 2020, and awarded a cooperative agreement for its future operations to the Gulf-Caribbean Oceanographic Consortium. In February 2019, the RCRV project experienced a partial suspension of work due to the status of necessary production design and modeling deliverables, among other concerns. This resulted in 16 weeks of schedule contingency usage. However, there was no overall increase to the scheduled construction completion date of July 2024. Estimated construction completion date, including schedule contingency: July 2024 for three vessels. Construction award: Cooperative support agreement with Oregon State University, which contracted with Gulf Island Shipyards, LLC. Responsible NSF directorate: Geosciences. Project partners: The U.S. Navy performed initial design for the vessels. Expected duration of operations: 30 years. Construction Status of the Regional Class Research Vessels, as of September 2019 Percentage complete (based on construction of three vessels) aScope changes included are reductions in response to NSF’s policy on cost overruns or as part of a cost increase. As of September 2019, the RCRV project had no changes to its authorized total project cost, no changes to its estimated completion date of July 2024 for all three vessels, and no scope reductions. The National Science Board had authorized a not-to-exceed cost of $365.0 million for construction of three vessels. However, the shipyard bid was ultimately lower than expected, reducing the total project cost of building three vessels to $354.0 million. NSF accepted the project’s earned value management system in May 2019, following a surveillance review of the system. The review team found that the project’s system met the intent of NSF requirements and that its data were reliable. (In our March 2019 report, we reported that NSF conditionally accepted the project’s earned value management system in November 2018.) As of September 2019 with construction of three vessels 20 percent complete. Budget contingency: $44.0 million (exceeded the probability- weighted risk exposure of $24.6 million). Schedule contingency: 6 months (included in the July 2024 estimated completion date for three vessels). Estimated value of remaining scope reduction options: $9.8 million. Beginning in February 2019, the RCRV project utilized 16 weeks of schedule contingency and $2.4 million of budget contingency due to a partial suspension of work issued by the construction award recipient, Oregon State University (OSU). OSU was concerned with Gulf Island Shipyards’s (GIS) project management capacity and its ability to manage subcontractors, such as engineering vendors responsible for providing design specifications. During the work suspension, GIS developed a corrective action plan that identified eight areas of improvement, such as a subcontract management plan and updated schedules that better align the development of necessary design specifications with construction activities. OSU’s management team assessed and monitored GIS’s progress on these areas and subsequently lifted the work suspension in May 2019. However, the project continues to face subcontractor management issues. OSU has requested NSF approval for an estimated $6.1 million of budget contingency and 4 months of schedule contingency to compensate for the delays associated with these issues. According to project documentation, this issue may cause the construction completion date of each vessel to slip. Remaining Project Risks and Potential for Cost or Schedule Increases According to project documentation, the project had an estimated risk exposure of $24.6 million and $44.0 million in remaining contingency as of September 2019. With the utilization of 4 months of schedule contingency in 2019, the RCRV project had 6 months of contingency remaining until construction is scheduled to end in 2024. According to project documentation, 12 options for reducing scope were available as of December 2019, with potential savings estimated at $9.8 million. and schedule expertise, which resulted in a decrease in the impact of the risk. In addition, the RCRV project is closely monitoring two risks related to newer technologies and requirements for regional operability of each vessel. First, the project team identified newer technologies for systems such as communications compared to those specified during the design phase. According to project documentation, the project may utilize contingency to integrate such technologies into the vessels. Second, the project may incur additional engineering, labor, and material costs associated with certain potential design changes that NSF and the operating institutions for the three vessels have identified. These design changes are intended to improve quality and performance within the different regions where the three vessels will be operating. The National Science Foundation’s (NSF) Antarctic Infrastructure Modernization for Science (AIMS) project will modernize the core infrastructure of McMurdo Station in Antarctica, the largest of three stations operated by NSF’s United States Antarctic Program and used by multiple agencies. McMurdo Station serves as a logistics hub for remote field sites and for the Amundsen-Scott South Pole Station. The AIMS project is expected to make environmental and safety upgrades to McMurdo Station and redevelop it into a more compact, energy and operationally efficient core facility to support research. The planned core facility will consolidate critical buildings, such as medical facilities and field science support. Construction of NSF’s AIMS project was about 6 percent complete as of September 2019. The project was in its first year of construction. In February 2019, the National Science Board approved the project’s not-to- exceed cost of $410.4 million, and NSF awarded an initial contract modification for construction equipment and materials to be delivered to California by December 2019, in time for deployment to McMurdo station through two supply vessels. In April 2019, NSF awarded the second contract modification for construction of the first major components of AIMS: the Vehicle Equipment and Operation Center (VEOC) and a new lodging facility structure and exterior shell. According to NSF, the VEOC will facilitate maintenance and repair of both heavy and light equipment ranging from tractors and cranes to trucks, vans, snowmobiles, and field generators. The lodging facility will include space for 285 beds, which the project’s final design review panel expected to be adequate to support short- and long-term plans for McMurdo station, including construction needs. As of September 2019, the start of initial operations for the VEOC and lodging facility were planned for 2022 and 2023, respectively, and completion of both facilities was planned for 2022, according to NSF officials. Later phases of the AIMS project will include construction of central services, emergency operations, field science support, and industrial trades facilities. In November 2018, the U.S. Army Corps of Engineers completed an independent cost estimate (ICE) report for the AIMS project. According to NSF officials, the ICE was critical for negotiations with the contractor as NSF utilized data within the ICE, such as labor rates and cost of materials, to verify costs. Specifically, the ICE assisted NSF in determining the reasonableness of the contractor’s proposed cost estimate and schedule for the project and associated risks. According to NSF officials, NSF and the contractor resolved all recommendations from the ICE report to NSF’s satisfaction prior to setting the not-to-exceed cost. Cost and Schedule Performance History As of September 2019, NSF’s AIMS project had no changes to its authorized total project cost, changes to its estimated completion date, or scope reductions since the National Science Board authorized the project’s not-to-exceed cost of $410.4 million, which included $67.2 million in budget contingency, in February 2019. Remaining Contingency and Scope Reduction Options As of September 2019 with construction about 6 percent complete. $59.2 million ($7.1 million more than the probability-weighted risk exposure of $52.1 million). 18.4 months (included in the 2028 estimated completion date). By the project’s final design review in October 2018, the AIMS team determined that it could not execute the project with the desired scope for the $355.0 million estimate—as was previously presented in NSF’s fiscal year 2019 budget request—because of changing market conditions. In response, NSF convened a review panel, which evaluated scope reduction options such as relocating and reducing bed space in the lodging facility from 285 to 100 beds, which would also entail keeping the current lodging facility in operation instead of demolishing it to make room for a new facility. While it accepted some of these options, such as a reduction of warehouse space within the VEOC, the panel noted that relocation of the lodging facility and a reduction of bed space would have adverse effects on the project. For example, the panel found that constructing a new 100-bed lodging facility in an alternate location would not support the eventual construction of sky bridges. According to the project’s Final Design Review report, these sky bridges would improve efficiency by avoiding the need for personnel to put on Antarctic gear before moving between buildings, reduce energy use by reducing the need to open exterior doors, and significantly improve the quality of life for personnel. NSF therefore decided to maintain the 285-bed plan and finalized the total project cost at $410.4 million. and four of 11 procurements for the lodging facility to the 2021 vessel, but NSF officials do not expect significant construction delays as a result. The officials explained that the VEOC procurements are not required for 2020 construction and that the deferral of lodging procurements is expected to be accommodated by re-sequencing activities on site. Remaining Project Risks and Potential for Cost or Schedule Increases As of September 2019, the AIMS project had a risk exposure of $52.1 million and $59.2 million in remaining contingency, and all of the project’s 18.4 months of schedule contingency remained available. The project had cumulatively used $7.9 million in budget contingency. Of this, $7.8 million was used during initial award for contract modifications for initial construction, with the remainder used for additional equipment purchases and leases in August and September 2019. As of September 2019, the AIMS project had $14.2 million in high- likelihood risks. The largest remaining risk, with an estimated value of $12.5 million and a 23-day delay, was that subcontractor proposals would exceed planned construction costs. Another such risk was an increase in the estimated base price of key construction materials—such as steel, copper wire, concrete, gypsum, and specialty items—before the materials were procured. NSF’s contractor for the project, Leidos Innovations Corporation, was working with one of its subcontractors to ensure material costs were accurate and consistent with market pricing. In accordance with NSF policy, the project maintains a list of scope reduction options, which as of April 2019 included approximately $34.0 million to $43.1 million in total possible project de-scopes. For example, the largest scope reduction option, with an estimated value of up to $19.1 million, is to remove the new trades shop from the AIMS scope and instead use the current facility. Another option, with an estimated value of up to $4.0 million, is to remove the gymnasium from the emergency operations facility and instead continue to use and maintain the existing gymnasium. This appendix provides individual summaries of the two National Science Foundation (NSF) projects that were in design and planned for construction as major facilities projects: (1) the Large Hadron Collider High Luminosity Upgrade and (2) Leadership Class Computing Facility. As of September 2019, no construction funds had been awarded for these projects and all cost, schedule, scope, and design information for these projects was subject to change. Each project’s summary is based on project documents and other information that NSF officials provided and includes the following: An overview of the project and its purpose. A timeline identifying key project dates. Project information, such as the expected date for completion of construction; the anticipated type of awards for construction; the responsible NSF directorate; project partners; and expected duration of operations. A summary of the project’s current status. A summary of the project’s design and construction costs, if available, and the budget account NSF planned to use for construction of the project. Information on potential project risks. The Large Hadron Collider (LHC) is the world’s most powerful particle accelerator. The facility’s four detectors observe new particles that are produced when high-energy protons are accelerated and collided, providing insight into fundamental forces of nature and the condition of the early universe. Through the National Science Foundation’s (NSF) Large Hadron Collider High Luminosity Upgrade (HL-LHC) program, the agency will fund a portion of a larger international effort to upgrade the facility’s accelerator and detectors. Specifically, NSF plans to fund the design and implementation of certain parts of the upgrades as two separate projects for the facility’s detectors, the A Toroidal LHC Apparatus (ATLAS) and Compact Muon Solenoid (CMS) detectors. The Department of Energy (DOE) is also contributing to upgrades to the LHC’s accelerator and to the ATLAS and CMS detectors. As of September 2019, NSF’s HL-LHC program was approaching its fifth year of design. The program has conducted several required activities to complete the design stage. In September 2019, NSF convened an external panel for the final design review of the program. The panel found that both detector upgrades met the readiness criteria within NSF’s Major Facilities Guide to proceed to construction. NSF also convened the internal Facilities Readiness Panel in November 2019 and conducted life cycle cost reviews for each detector upgrade in October 2019, according to NSF officials. Estimated construction completion date, not including schedule contingency: 2026. Construction awards: If approved, planned for 2020 as cooperative agreements with Columbia University (ATLAS detector) and Cornell University (CMS detector). Responsible NSF directorate: Mathematical and Physical Sciences. Project partners: European Organization for Nuclear Research and the Department of Energy. Expected duration of operations: 12 years. According to NSF officials, NSF planned to request National Science Board authorization in February 2020 to make construction awards. As a prerequisite for making the awards in April 2020, NSF received the independent cost estimates for both projects from the Army Corps of Engineers in January 2020. According to NSF documentation, these results align with the current total project cost reviewed during the final design review. According to the Major Facilities Guide, NSF uses independent cost estimates to validate recipient estimates, negotiate awards, check for compliance with GAO best practices and Uniform Guidance cost principles, and inform NSF’s cost analysis. According to NSF officials, the estimated completion for both upgrade projects is 2026. According to program documentation, NSF had obligated a total of $24.3 million for the design of its detector upgrades as of September 2019. Funding for the design has come from NSF’s Research and Related Activities account, rather than the Major Research Equipment and Facilities Construction account. Planned Contingency and Scope Reduction Options As of November 2019, with finalization of the NSF cost analysis still pending. Budget contingency: $38.9 million as follows $20.0 million for the ATLAS detector. $18.9 million for the CMS detector. Schedule contingency: To be determined. Estimated value of scope reduction options: $15.1 million as follows $8.4 million for the ATLAS detector. $6.7 million for the CMS detector. until authorization by the National Science Board. These figures remained subject to change before completion of the final design phase. According to NSF documentation, the total project cost may increase slightly based on a detailed evaluation of both projects’ contingency budgets following the final design review. NSF plans to fund the upgrades with separate cooperative agreements for each detector and to monitor each agreement in accordance with its distinct terms and conditions, total project cost, and earned value management metrics, according to agency officials. In August 2019, NSF initiated independent cost estimates of both projects (ATLAS and CMS) under the HL-LHC program, as required by the American Innovation and Competitive Act for projects in the design phase. The U.S. Army Corps of Engineers is conducting the estimates under an interagency agreement with NSF, with contractor support. In addition, NSF is conducting a cost analysis that will be informed by the final design review panels, internal assessments by the NSF’s Large Facilities Office and other business units, and the independent cost estimates. DOE’s Contributions to Upgrading the Large Hadron Collider DOE’s High Energy Physics program helped fund the construction of the Large Hadron Collider and continues to support researchers using the facility as well as upgrades to it. According to DOE’s fiscal year 2020 budget request, the department planned to support the upgrades to the ATLAS and CMS detectors at an estimated cost range of $149 million to $181 million for the ATLAS detector and $125 million to $155 million for the CMS detector. The scope of DOE’s work on the detectors was to focus on areas where the expertise and infrastructure of the department’s national labs were needed, whereas the scope of NSF’s work was to focus on areas led by university researchers. In addition, DOE approved upgrades to the accelerator itself with a total project cost of $242.7 million, according to DOE’s fiscal year 2020 budget request. NSF plans to fund the construction of the detector upgrades through its Major Research Equipment and Facilities Construction account. While the upgrades would involve separate cooperative agreements for each detector, NSF considers them one program consisting of two distinct projects, according to agency officials. Project Risks and Potential Scope Reduction Options Under NSF policy, a project’s cost should include enough budget contingency to cover all foreseeable risks. Following the preliminary design review, the amount of budget contingency included in the construction cost for the upgrades was approximately $38.9 million, or 26 percent of the planned total project cost. At the time of this report, the NSF cost analysis following the final design review was still pending and therefore the estimated amount of contingency is subject to change. NSF policy also directs a project’s design to include prioritized, time- phased options for reducing its scope during construction if needed. As of the final design review, the project teams had identified a total of $15.1 million of potential scope reduction options for the projects, which are subject to change throughout the design and construction of a project. According to the projects’ scope management plans we reviewed, the ATLAS detector has nine options to reduce scope totaling $8.4 million, with the options ranging in value from $0.6 million to $1.7 million. The CMS detector has 17 scope reduction options with a total value of $6.7 million. According to the project’s scope management plan, both NSF officials and external panels reviewed and provided input to determine the current scope reduction options. The National Science Foundation’s (NSF) Leadership-Class Computing Facility (LCCF) project is intended to provide advanced computational capabilities to enable transformative research in all areas of science and engineering that would not be possible by theory or experiment alone. According to NSF officials, future research using LCCF might include extremely detailed simulations ranging from biological molecules to supernovae and analyses of very large data streams such as satellite images to create high-resolution Earth maps. Project Status As of September 2019, the LCCF project was in its first year of design; consequently, all cost, schedule, scope, and design information for the project was subject to change. In March 2019, the NSF Director approved the project to enter the design stage as a candidate major facilities project. The project represents the final phase of a two-phase deployment of high-performance computing systems. The first phase—known as the Frontera project at the Texas Advanced Computing Center at the University of Texas at Austin—was completed in September 2019. According to NSF, at that time, Frontera was the largest high- performance computing system deployed on a U.S. academic campus. The LCCF project will support the design and construction of an upgrade to the Frontera system as well as to the physical facility that will host it. In project documentation, NSF has described the upgrade as providing a substantial improvement in application performance but has not specified the extent of improvement. Estimated construction completion date, not including schedule contingency: Fiscal Year 2025. Construction award: Planned for 2024. Responsible NSF directorate: Directorate for Computer & Information Science & Engineering. Project partners: None. Expected duration of operations: 10 years. In July 2019, NSF awarded both an overarching cooperative agreement for the LCCF project and a cooperative support agreement for the conceptual design phase to the University of Texas at Austin. As of November 2019, the project was focused on leading and participating in activities with experts within the community for high-performance computing. The purpose of these activities was to document the science, technology, and facilities requirements for LCCF, as well as to shape the design and cost of long-lead items, such as the power and cooling infrastructure to service the facility. NSF plans to conduct the conceptual design review in June 2020. NSF’s Support for High- Performance Computing Systems NSF has supported high-performance computing capabilities for nearly 4 decades. In 2007, NSF awarded $226.6 million for the Blue Waters high- performance computing system through a cooperative agreement with the University of Illinois at Urbana- Champaign. According to NSF, at the time of its deployment in 2013, Blue Waters was one of the most powerful supercomputers in the world and was one of the fastest on a university campus. Scientists and engineers across the country used the computing and data power of Blue Waters to tackle a wide range of problems, including predicting the behavior of complex biological systems and simulating the evolution of the cosmos. Because of the rapid evolution of computer technology, by 2019, NSF no longer considered Blue Waters to be the leadership computing system for fundamental science and engineering research. Anticipating these technological advances, in September 2018, NSF awarded about $63.0 million to the University of Texas at Austin for the follow-on project to Blue Waters. Frontera was intended to provide three to five times the computing capability and twice the storage capacity to support the increased computational requirements for science and engineering research. NSF also anticipated that Frontera would help inform science requirements and reduce risks for LCCF, which is planned to provide substantially more computational capabilities than both Blue Waters and Frontera. obligated $2 million from its Research and Related Activities account for the design of LCCF. According to the project’s cooperative agreement, NSF may provide additional funding to advance the design of LCCF— $3.5 million in fiscal year 2020 and $2.5 million in fiscal year 2022 following successful completion of the conceptual and preliminary design reviews, respectively, subject to availability of appropriations. As of September 2019, NSF had not yet formally identified risks for the LCCF project because the project was early in the design stage. NSF requires recipients to develop and follow formalized risk management during the design and construction stages of major facility projects to identify potential risks, assess the nature of those risks, and identify actions that can be taken to either reduce the probability of those risks occurring or reduce their impact to the project. NSF officials told us that an assessment of risks associated with the LCCF project will be part of the conceptual design review, planned for June 2020. According to NSF officials, one anticipated challenge for the LCCF project is the rapid pace of technological change in the field of high-performance computing. The officials stated that forecasting the technology marketplace in the future can be challenging as technology can change radically because of external market forces. Conversely, the rapid pace of change can also be an opportunity if the LCCF project can incorporate the latest technological advances that result in the most advanced computing capabilities. According to NSF officials, taking advantage of such opportunities as late in the design stage as possible will be important for the success of the project. John Neumann, (202) 512-6888 or neumannj@gao.gov In addition to the contact named above, Joseph Cook (Assistant Director), Sean Manzano (Analyst in Charge), Louise Fickel, Yvette Gutierrez, Patrick Harner, Douglas G. Hunker, Jason T. Lee, Serena Lo, and Anika McMillon made key contributions to this report.", "summary": "NSF supports the design, construction, and operations of major facilities projects–science and engineering research infrastructure such as telescopes and research vessels that typically have construction costs of at least $70 million and may take many years to design and construct. The agency oversees the performance of each project against an authorized total project cost and schedule. NSF currently has four projects under construction at a combined authorized cost of $1.6 billion and two additional projects in design. Prior GAO reports reviewed NSF's cost estimating and schedule policies, as well as project management expertise of its oversight workforce. Senate Report 114-239 and House Report 114-605 included provisions for GAO to review NSF's major facilities projects. Among other objectives, this report (1) describes the cost and schedule performance of NSF's ongoing major facilities projects and (2) assesses the extent to which NSF addressed prior GAO recommendations related to its management of major facilities. GAO analyzed NSF policies and documents for projects in design and construction, interviewed agency officials, and compared NSF's processes to best practices identified in prior GAO work. Since GAO's March 2019 report on the status of its major facilities projects, the National Science Foundation (NSF) had no increases to the authorized total project costs or schedules for its four projects under construction (see figure): The Daniel K. Inouye Solar Telescope was on track to be completed within its $344.1 million cost and June 2020 completion date. NSF was evaluating options for reducing the scope of the Vera C. Rubin Observatory (previously the Large Synoptic Survey Telescope), which it believed might be necessary to keep the project within its $473 million cost and October 2022 completion date. Construction of a second Regional Class Research Vessel began in September 2019 and was anticipated to begin on a third and final vessel in March 2020 at a combined cost of $365 million. The Antarctic Infrastructure Modernization for Science entered the construction phase in February 2019 at a cost of $410.4 million. NSF fully implemented two of the six prior GAO recommendations including revising policies for estimating the costs of major facilities projects and revising the Vera C. Rubin Observatory's schedule to better meet best practices. NSF took steps to address but has not fully implemented the remaining four recommendations on the agency's oversight of major facilities. NSF agreed with and has taken initial steps to address four open recommendations from GAO's prior work, including to revise policies for developing schedules and to ensure the sharing of lessons learned for major facilities projects. NSF needs to complete additional steps to fully address the recommendations.", "document_type": "gao"}
{"report": "Traditionally, federal surface-transportation funding has been primarily delivered through formula grant programs based on distributions prescribed by statute. Discretionary grant programs, such as INFRA, represent an alternative approach for directing federal funding toward national priorities. Through a discretionary grant program, Congress or federal agencies establish desired goals or outcomes—such as improving the condition of critical infrastructure, enhancing economic competitiveness, or reducing fatalities. Generally, federal agencies review grant applications against published selection criteria and statutory and regulatory requirements before selecting projects to receive awards. This approach can help assure accountability for federal investment by more clearly linking program funds to desired outcomes and can support projects of national or regional significance that cross state lines. In prior work, we have recommended that a merit-based competitive approach— like INFRA—be used to direct a portion of federal funds to transportation projects of national and regional significance. The FAST Act authorized over a dozen discretionary transportation-grant programs, and Congress may consider additional programs as it considers reauthorizing DOT’s surface transportation programs in 2020. State, local, and tribal governments, as well as multistate or multijurisdictional groups, are among the entities eligible to receive INFRA funding. Freight or highway projects must meet the statutory requirements outlined in the FAST Act to receive INFRA funding. Notable statutory requirements regarding the distribution of awards include: Ten percent of available funds are reserved for small projects each fiscal year. At least 25 percent of available funds are reserved for rural areas each fiscal year unless DOT does not receive enough qualified rural project applicants. No more than $500 million, in aggregate, over fiscal years 2016 through 2020 may be used to fund freight rail, water (including ports), or other freight intermodal projects. The Secretary must consider geographic diversity during the selection process. Large projects have to meet seven additional statutory requirements to be eligible for selection by the Secretary. Specifically, the Secretary must determine that the project: will generate national or regional economic, mobility, or safety will be cost-effective; will contribute to one or more of the national goals for the transportation system: improved safety, infrastructure maintenance, congestion reduction, system reliability, freight movement, economic vitality, environmental sustainability, and reduced project delivery delays; is based on the results of preliminary engineering; for related non-federal financial commitments, has stable and dependable funding and financing sources to construct, maintain, and operate the project, and contingency amounts to cover unanticipated cost increases; cannot be easily and efficiently completed without other federal funding or financial assistance; and is reasonably expected to begin construction no later than 18 months after the date of obligation of funds for the project. In the July 2017 NOFO for the INFRA program, DOT established four new criteria for INFRA outlining how projects would be evaluated (see table 1). DOT did not require that projects address every criterion. DOT noted that in addition to these criteria, called merit criteria, it would also evaluate a project’s readiness, meaning the likelihood of a project’s successful delivery and that the project will meet statutory deadlines for certain milestones. In December 2018, DOT issued a NOFO in which it called for applications for grants of fiscal year 2019 INFRA funds, and made some changes to the program’s criteria. However, that process is ongoing and is outside the scope of this review. In reviewing applications submitted in response to the July 2017 NOFO, DOT evaluated proposed projects against the statutory and merit criteria using a multiphase review process involving technical and senior management teams. The process had three phases—application intake, technical evaluation, and senior review—each supported by different teams. The process also included a Quality Control and Oversight Team (QCO) that was involved throughout the process and responsible for ensuring consistent reviews and documentation. QCO consisted of team leads from each of the seven technical evaluation teams as well as liaisons from FHWA, MARAD, and FRA (see fig. 1). The application intake phase consisted of two sequential steps performed by two different teams. First, the intake review team assessed each of the projects to: 1. verify that the applicant type, project type, and cost-sharing met the statutory requirements; 2. determine the project’s size as being either small or large; 3. identify the highway and non-highway cost components; 4. determine whether the project is in an urban or rural area; 5. identify which technical evaluation teams should review the 6. which modal administration should perform the Operating Administration screen (described below). DOT received 258 applications for projects in November 2017 and determined that 24 projects did not qualify for INFRA funding. The remaining 234 projects then moved to the Operating Administrations’ screen. As part of the Operating Administrations’ screen, staff from the appropriate modal agency provided input on: 1. the applicant’s history with delivering projects on time; 2. whether the applicant had previously received federal funding from 3. whether the applicant contacted the agency about their INFRA project, the nature of the contact, and the level of technical and financial assistance provided by the agency; 4. whether the project is on the Transportation Improvement Program or the Statewide Transportation Improvement Program; and, 5. any specific issues with the project that evaluators should be aware of. The 234 projects then advanced to the Technical Evaluation phase of the process. The seven technical evaluation teams, made up of experts from across the agency, assessed and rated projects against the merit criteria. Each team was responsible for rating a different merit criterion (as noted in figure 1, the innovation criterion was split into three factors, so there were three innovation teams). Since DOT did not require projects to address all the criteria, teams only reviewed the projects that related to their criterion. The teams used the factors outlined in DOT’s INFRA evaluation plan to assess and rate the projects and documented their rating and a narrative justification for the assigned rating in DOT’s tracking spreadsheet. Generally, raters assigned scores of high, medium, or low for each criterion, with some exceptions. For example, the economic vitality team calculated the project’s benefit-cost ratio and net present value, while also noting whether the uncertainty associated with the rating was high, medium, or low. Similarly, the leveraging team assigned a rating score of high, medium, or low, but also calculated the percentage of non-federal funding, and noted whether the project included private-sector funding. Technical teams did not provide an overall rating of projects (such as not recommended, recommended, or highly recommended), an approach that differs from prior DOT discretionary grant programs we have reviewed. For detailed information about the evaluation factors and possible scores for each criterion, see appendix I. Each technical review team was assigned a team lead, who was responsible for ensuring that the projects were evaluated consistently and per the plan that governed that team’s criterion. All 234 projects received technical evaluation ratings for their merit criteria and then advanced for further review. According to DOT’s evaluation plan, the Quality Control and Oversight Team (QCO) was responsible for ensuring the consistency of reviews and documentation throughout the INFRA process. QCO consisted of team leads from each of the seven technical evaluation teams and liaisons from FHWA, MARAD, and FRA. QCO was also responsible for performing a “large project determination,” in which QCO assessed whether projects met each of the seven statutory requirements for large projects. QCO used information from the technical evaluations and the information provided in the application to determine whether projects met the statutory requirements. In cases where QCO could not definitively determine whether a large project met a statutory requirement, it would note “additional information is necessary” in DOT’s tracking spreadsheet. After QCO recorded its assessment, it submitted the projects to the Senior Review Team for review. The Senior Review Team was responsible for assembling a list of projects for consideration by the Secretary, and consisted of senior officials from the Office of the Secretary, and the Administrators of FHWA, FRA, FTA, and MARAD. The Senior Review Team, with QCO present to answer questions, met to review the projects and their technical evaluation scores for each criterion. The evaluation plan stated the Senior Review Team could, at its discretion, request that QCO seek additional information from applicants to help QCO determine if a large project met the statutory requirements. The final list of projects for consideration developed by the Senior Review Team contained 165 projects (all of the small projects and all of the large projects that QCO and the Senior Review Team determined met the statutory requirements). At the end of the review process, the Secretary received a series of spreadsheets ranking each of the 165 projects according to how well they scored on each merit criteria. According to a member of the Senior Review Team, the Secretary formally met twice with her chief of staff, deputy secretary, and other senior advisors to discuss the projects, first to analyze all of the projects on the list and second to finalize the award decisions. In June 2018, DOT announced it had awarded approximately $1.54 billion in INFRA funding to 26 projects (see fig. 2). For the 26 awarded projects, 44 percent of funds went to rural projects and 5 percent of funds went to small projects. In addition as shown in figure 2, highway projects received the largest percentage of funding (85 percent), and rail projects received the smallest percentage (1 percent). In designing its process for evaluating INFRA applications submitted in response to the July 2017 NOFO, DOT took steps to address issues that we found led to inconsistencies in DOT’s review of FASTLANE applications. Specifically, we reported that technical teams were divided by modal administrations (FHWA, MARAD, and FRA) and lacked clear guidance on how to score applications. This led to inconsistent scoring practices among the FASTLANE teams because one team applied a higher standard than the others. We recommended that DOT develop an evaluation plan for INFRA that clearly defined how all review teams should apply criteria, assess applications, and assign ratings to ensure that all applications are consistently reviewed. In response, DOT developed an INFRA evaluation plan that provided guidance on how to evaluate and assign a rating for each criterion, and in some cases, provided discrete numeric rating categories, allowing for less interpretation by technical review teams when assigning a score. In addition, DOT organized technical review teams by merit criteria and selected staff with the relevant expertise to serve on each team—for example, economists from the various modal agencies served on the economic vitality team. DOT also took steps to improve the transparency of its process by better communicating with unsuccessful applicants. Specifically, DOT formally notified unsuccessful INFRA applicants of selection decisions via email, addressing a concern we raised regarding the FASTLANE process. In our review of FASTLANE, we recommended that DOT notify unsuccessful applicants of DOT’s decision and that the notification should include a brief explanation of the decision. For INFRA, DOT emailed unsuccessful applicants notifying them of its decision. While the email did not include a brief explanation of the decision, it did offer applicants the chance to schedule a debriefing with DOT officials. Some of the selected applicants and consultants we spoke to said that the debriefing was helpful. For example, one applicant told us that during the debriefing, DOT shared how the project was rated by criterion. One applicant we met with said the debriefing was not helpful because the applicant did not receive a substantive answer about why they did not receive an award. Another applicant said he requested a debriefing but did not receive one. A DOT official told us that prior to issuing the fiscal year 2019 INFRA NOFO, DOT contacted all previous applicants to notify them of the upcoming round and again offer debriefs. We found that DOT’s process for following up with applicants lacked consistency and transparency, due to a lack of guidance and documentation. Specifically, DOT followed up with some applicants and not others to request additional information about their projects, and the rationale behind which applicants were selected for follow-up is not clear. We identified similar issues in our review of FASTLANE. As discussed earlier, for large projects to be eligible for an award, DOT must determine that the project meets several statutory requirements, such as generating benefits and demonstrating cost-effectiveness, among others. Our review of DOT documents revealed that DOT staff originally determined that 97 (of 116) applications for large projects did not include sufficient information for DOT to assess if the projects met each of the statutory requirements. At the request of officials on the Senior Review Team, QCO requested more information from 42 of those 97 applicants to help DOT determine if their projects met the requirements. Of the 42 applicants that DOT followed up with, 28 provided information that QCO determined was sufficient to ensure that they met the statutory requirements, and 13 of the projects received an award. Similarly, at the request of officials on the Senior Review Team, DOT staff reduced the scope of a number of projects. QCO staff split 9 projects into “components,” to scope out pieces of projects that could not meet a statutory requirement (for example, cost-effectiveness). Four of these component projects received an award. OMB guidance states that the intent of a NOFO is to make the application review process transparent so applicants can make informed decisions when preparing their applications to maximize fairness of the process. The guidance also states that federal agencies should make clear whether an applicant’s failure to meet an eligibility criterion by the time of an application deadline will result in the awarding agency returning the application without review or, even though an application may be reviewed, will preclude the awarding agency from making an award. Similarly, internal control standards note that federal agencies should communicate with external entities and enable these entities to provide quality information to the agency that will help it achieve its objectives. DOT’s NOFO states that the applications must include sufficient information for DOT to determine whether projects meet the statutory requirements, but also notes that DOT may seek additional information from applicants. The NOFO does not provide information on the basis for why DOT would follow up with one applicant and not another. After reviewing DOT’s documentation, we found that the rationales for following up with specific applicants were insufficient to explain why DOT followed up with certain applicants over others. The documentation, with few exceptions, included generally vague statements that additional information from the applicant could help DOT determine whether the project met the statutory requirements. We asked two officials from the Senior Review Team about several specific projects for which those officials requested additional information. These officials both stated they could not recall their rationale, given that roughly a year had elapsed and the large number of projects reviewed. However, they did provide some reasons why they might have requested additional information, such as the need for more clarity on a project, a high score on a criterion of interest to that official, or the desire to ensure that the list provided to the Secretary included a diverse array of projects (in terms of location, urban or rural status, and project type). Further, one official noted that there was insufficient time to follow up with every applicant. We have previously identified recommended practices for evaluating and selecting discretionary grant awards and noted that in order to align with these practices, it is important to document decisions, including decisions regarding which projects should have the opportunity to advance in the process. When we identified similar issues related to a lack of consistent and transparent follow-up with FASTLANE applicants, we recommended DOT develop an INFRA evaluation plan that clearly defines how all review teams should apply criteria, assess applications, and assign ratings to ensure that all applications are consistently reviewed. DOT’s INFRA evaluation plan states that if QCO has been unable to make an affirmative determination with respect to whether a large project meets a statutory requirement, a Senior Review Team member may direct QCO to seek clarifying information from the applicant or provide the necessary clarifying information themselves to support a determination. However, DOT’s evaluation plan does not require documentation of the reasons why the Senior Review Team asked QCO to follow up with certain projects over others. Without clearly outlining in the NOFO and the evaluation plan the situations in which certain applicants may be asked to provide additional information, as well as clear documentation for why follow-up does occur with specific projects over others, the process lacks transparency and the assurance of fairness. For example, we found examples in which reviewers noted that additional information could help them determine whether a project met the statutory requirements (such as whether the project was cost-effective) but less than half of the projects had the chance to provide such information. Of the 26 awarded projects, half of those projects (13 large projects) were afforded the opportunity to provide additional information to demonstrate that their projects met the statutory requirements. We were unable to determine the rationale for the selection of projects for INFRA awards; an issue we also found with the FASTLANE process. This is due to: inconsistency in the NOFO regarding how merit criteria would be used to select awardees; a large number of applications forwarded for potential award regardless of merit scoring; and limited documentation regarding why 26 projects were ultimately selected out of 165 for award. In the NOFO for INFRA, DOT provided inconsistent and unclear messages regarding the extent to which the merit criteria should be addressed to be competitive for an award, which also reduced transparency and caused confusion for some applicants. OMB guidance states that the intent of a NOFO is to make the application review process transparent so applicants can make informed decisions when preparing their applications to maximize fairness of the process. In the NOFO, DOT stated it would evaluate applications against four merit criteria, but also stated, “The Department is neither weighting these criteria nor requiring that each application address every criterion, but the Department expects that competitive applications will substantively address all four criteria.” In some cases, this approach led to confusion among applicants, as several of the selected applicants and consultants we interviewed noted that it was difficult to address the innovation merit criterion, with some stating the criterion was confusing or unclear and others stating that they faced difficulties adapting their projects to meet the criterion. Compounding this issue, several applicants and consultants also expressed uncertainty as to how DOT determined which projects should receive awards and which factors affected a project’s ability to get an award. For example, representatives for one applicant noted that they spent a considerable amount on a consultant for the benefit-cost analysis (which was common among most of the applicants we interviewed), but it was not clear how the benefit-cost analysis affected DOT’s decision-making. Despite the language in the NOFO, DOT did not use the merit scores— which reflect the extent to which projects addressed all four criteria— when it determined which projects should be provided to the Secretary for consideration. While DOT reviewers did score applications on all four merit criteria, all of the 165 projects that QCO found to be statutorily eligible—47 large projects and 118 small projects—were sent to the Secretary for potential award, regardless of merit criteria scores or whether the applicant substantively addressed all four merit criteria. DOT officials told us that DOT sought a “portfolio” approach in which the Secretary selected projects that scored highly on at least one criterion. Thus, the Secretary received a 25-page spreadsheet showing 14 different lists (7 for small projects and 7 for large projects) sorting all of the projects against the merit criteria, with each list arranged from highest to lowest score for that criterion. This method of presenting information on projects (and the volume of information presented) would make it challenging for any decision maker to compare projects and readily see how 165 projects scored across all criteria and whether all criteria were “substantively addressed.” In addition, projects were provided to the Secretary for consideration—and in some cases awarded—despite concerns raised by technical reviewers and regardless of whether projects addressed all of the merit criteria. For example, we found instances in which awarded projects had: Low cost-effectiveness scores. Over 50 percent (14 of 26) of all awarded projects received a high uncertainty rating related to their benefit-cost ratio and net present value score, meaning that the technical team had a low degree of confidence in the assigned score. Only 38 percent of all projects had this uncertainty rating. Moreover, of the 14 awarded projects with this rating, 11 had benefit-cost ratios of 1.0 to 1.5, which, when combined with the high uncertainty rating, raises the risk that the project would not be cost-effective. For example, for one large project, a technical reviewer noted, “… we conclude that the benefits of this project are reasonably likely to exceed its costs, though the case is very marginal and highly uncertain, as even a small change in some of the key assumptions and parameters could result in a negative finding.” Uncertainty regarding projects’ benefit-cost ratios is particularly important as DOT used these scores to assess whether large projects met the FAST Act requirement to be cost-effective. While comments from technical reviewers were not included in the spreadsheets provided to the Secretary, an official stated that the Senior Review Team reviewed each project in-depth with the Secretary, and other DOT officials noted that the spreadsheet provided to the Secretary included the uncertainty ratings for each project. Low scores on multiple criteria, or did not address all criteria. Two awarded small projects had a benefit-cost ratio of less than one, and one of those projects did not address the innovation criteria at all. Three of the 26 awarded projects (11.5 percent) did not address the innovation criteria at all. In addition, several of the selected applicants and consultants we interviewed expressed confusion regarding how DOT reviewed the applications and moved them forward within DOT. Some of the applicants and consultants thought that DOT used the project scores to determine which projects should move forward to the Secretary (similarly to previous rounds of other DOT grant programs in which projects were sorted into categories such as “highly recommended,” “recommended,” and “not recommended”). One applicant noted that it is important to know how many projects make it to the Secretary in order to understand the extent to which decisions are based on technical scores versus other considerations. DOT’s guidance states that grant recipients should be selected based on technical merit and those projects most likely to achieve the intended purpose. In addition, we have identified recommended practices for awarding discretionary grants, one of which includes documenting the rationale for award decisions. Documenting the rationale for award decisions becomes even more important in light of DOT’s decision to provide every eligible INFRA application to the Secretary, rather than providing the Secretary a list of the projects “most likely to achieve the intended purpose.” However, DOT’s documentation on the final selection of projects states the anticipated benefits of the projects but does not indicate why these projects, according to DOT’s guidance, “best address program requirements and, therefore, are most worthy of funding.” In our review of the FASTLANE process, we also noted that due to limited documentation, we could not determine how DOT selected which projects should receive awards. We recommended that DOT require program teams to document their decision-making rationale throughout all levels of review in the application selection process. DOT agreed with this recommendation; however, it has not yet been implemented. Therefore, it remains unclear whether DOT is awarding discretionary grants on the basis of merit principles or other considerations. An absence of documentation can give rise to challenges to the integrity of the evaluation process and thus the decisions made. Since 2011, we have found similar issues with DOT’s management of other competitive discretionary grant programs, including a lack of documentation of key award decisions, and have made recommendations aimed at increasing consistency and transparency. In some cases, DOT implemented our recommendations for one program, but we subsequently found similar or recurring problems in other DOT programs. In 2011, we reviewed DOT’s Transportation Investment Generating Economic Recovery (TIGER) program and FRA’s High Speed Intercity Passenger Rail program—two discretionary grant programs funded through the American Recovery and Reinvestment Act of 2009. For both programs, we found, among other things, limitations in the agencies’ documentation of the rationale for award decisions. With respect to TIGER, we noted that a lack of documentation could subject DOT to criticism that projects were selected for reasons other than merit. However, we also noted that documenting key decisions could help build confidence in DOT’s ability to administer competitive discretionary grant programs. We recommended that DOT and FRA improve their documentation of key decisions for both programs. DOT implemented these recommendations by updating its TIGER and FRA guidance to require additional documentation. Despite the steps DOT took to address our prior recommendations, in 2014, we found continued issues in the TIGER program and made more targeted recommendations. Specifically, we found that DOT did not document key decisions to, among other things, (1) advance projects with lower technical ratings instead of more highly rated projects, and (2) change the technical ratings of lower-rated projects that had been selected for an award. We recommended that the Secretary of Transportation establish additional accountability measures for management of the TIGER program, to include using a decision memorandum or similar mechanism to document a clear rationale for decisions to: change the technical evaluation rating of an application, not advance applications rated as highly recommended, and advance for senior review applications other than those rated as highly recommended. Subsequently, DOT revised its guidance for the TIGER program to prohibit changes to the technical ratings, require that all highly rated projects be advanced, define the conditions through which lower rated projects may be advanced, and require that all such decisions be fully documented. DOT did not require that these decisions be documented through a decision memorandum or similar mechanism, as we had recommended. However, taken together, we determined DOT’s actions were sufficient to address our recommendation for the TIGER program. In December 2016, we found similar problems during our review of the Hurricane Sandy transit-resilience grant program administered by the Federal Transit Administration (FTA). For example, we found that FTA did not document rationales for changes to project ratings nor did it document how it addressed high-level project concerns raised by reviewers in their evaluation comments. In addition, we found that DOT lacked clear department-wide requirements for what should be documented when evaluating and selecting discretionary grant awards. We noted that internal control standards state that all transactions and significant events need to be clearly documented, and that a recommended practice for evaluating and selecting discretionary grant awards is documenting the rationale for awards decisions, including reasons individual projects were selected or not selected. We also found that FTA did not develop an evaluation plan prior to calling for applications, despite the fact that this was a requirement in DOT’s Financial Assistance Guidance Manual and that recommended practices for administering discretionary grant programs note the importance of having an evaluation plan that describes a method for overseeing the technical review panels to ensure a consistent review. Finally, we found that FTA did not assess projects against the policy priorities it outlined in its notice of funding availability, despite an OMB directive to provide sufficient information to help an applicant make an informed decision about whether to submit a proposal. At this time, we noted a pattern of problems occurring across DOT and its modal administrations’ discretionary grant programs and determined that a department-wide action was needed to address these issues. Specifically, we recommended that the Secretary issue a department- wide directive that should include requirements to: develop a plan for evaluating project proposals in advance of issuing a notice of funding availability that defines the stages of the process, including how the process will be overseen to ensure a consistent review of applications; document key decisions, including the reason for any rating changes and the officials responsible for those changes, and how high-level concerns raised during the process were addressed; and align stated program purpose and policy priorities with the evaluation and selection process. DOT concurred with our 2016 recommendation to develop a department- wide directive and initially stated that it would address it by updating its Financial Assistance Guidance Manual by September 2018 (DOT recently extended this to December 2019). In response, we noted that in order to address our recommendation, DOT needed to issue a directive that incorporates all of the elements identified in our recommendation. In addition, it remains unclear whether updating the manual would have the same effect as issuing the department-wide directive that we recommended. Specifically, we have found that DOT has not always followed its own guidance despite clear language that certain actions are required. For example, in our 2017 review of the FASTLANE program, we noted that the Financial Assistance Guidance Manual required finalization of the evaluation plan prior to soliciting applications for grants, but this guidance was not followed. Since 2017, we have sent letters to the Secretary of Transportation noting that this is a high-priority recommendation that warrants her attention. In March 2019, DOT issued a one-page memo to all offices and departments that administer discretionary grants. This memo directed the offices to update their policies and procedures to implement our 2016 recommendation and to send the updated policies to DOT’s Office of the Senior Procurement Executive by June 30, 2019. DOT officials told us that DOT believes this action has addressed the recommendation. Due to a number of issues, however, it is unclear how this action will address our recommendation to create clear department-wide requirements aimed at improving transparency and consistency. Specifically, we found that the memo was essentially limited to a repetition of our recommendation. That is, DOT did not take steps to ensure that the various affected offices consistently interpret and implement the recommendation. For example, DOT did not define key terms such as “high level concerns,” or “key decisions.” In addition, DOT did not communicate to offices how they should sufficiently document their decisions to ensure that the rationale for those decisions—including the reasons individual projects were selected or not selected—is clear. DOT officials told us they wanted to provide the affected offices flexibility to implement the recommendation and would assess the need for additional guidance based on the completion of the Financial Assistance Guidance Manual. However, the lack of information regarding how offices should implement the memo raises significant questions about whether various offices will interpret and implement the recommendations differently, and enhances the risk that DOT will continue to lack a department-wide approach to ensure that discretionary grant programs are consistently and transparently administered. As DOT continues to try to address these long-standing issues with its discretionary grant programs, Congress has an opportunity through reauthorization legislation, scheduled for 2020, to build requirements for enhanced consistency and transparency into these programs. This is particularly important as DOT has two additional rounds of INFRA funding to award under the FAST Act, and the President’s Budget proposal proposed providing an additional $1 billion to INFRA. Moreover, the FAST Act also authorized over a dozen discretionary transportation grant programs, and Congress may consider additional programs during the reauthorization of DOT’s surface transportation programs. Through legislation, Congress could craft requirements around the administration of DOT’s discretionary grants to improve the processes for awarding grants. Absent effective action by DOT going forward, the recurring and long-standing issues we have identified could continue to affect DOT’s discretionary grant programs. While DOT has taken some steps to improve its reviews of INFRA grant applications since we reviewed the FASTLANE program, issues related to consistency and transparency remain. Specifically, without clear communication from DOT regarding (1) the situations in which DOT may provide certain applicants the opportunity to supplement their applications with additional information, and (2) how merit scoring is used, if at all, to determine whether projects advance to the Secretary for selection and which projects are selected, applicants lack the information needed to make informed decisions about whether to apply. In addition, without documentation outlining why DOT decided to request additional information from certain applicants over others, the process lacks transparency. Since the FAST Act was enacted in 2015, we have been unable to determine the basis for the resulting awards of about $2.3 billion through the FASTLANE and INFRA program. This lack of clarity is significant and is the product of long-standing issues that we have identified with DOT’s discretionary grant programs since 2011. We have previously noted that competitive discretionary grant programs have promise in better targeting federal transportation spending to areas of national and regional significance; however, this promise cannot be fulfilled if DOT’s process and rationale for making awards remains unclear. In 2019, DOT issued a department-wide memo aimed at addressing our 2016 recommendation, but it is unclear how DOT’s approach will improve consistency and transparency in its management of grant programs. We will continue to monitor DOT’s efforts to address our recommendation. However, given the long-standing nature of the issues we identified and the potential that they could continue to affect DOT’s discretionary grant programs, the reauthorization of DOT’s surface transportation programs scheduled for 2020 provides Congress the opportunity to require DOT to take additional action to ensure consistency and transparency in the management of its discretionary grant programs. During the next reauthorization for surface transportation programs, Congress should consider including language in the reauthorization bill that would require DOT to develop and implement transparency measures for DOT’s review and selection process for discretionary grants. Such measures should, at a minimum, help to ensure that the evaluation process is clearly communicated, that applications are consistently evaluated, and that the rationale for DOT’s decisions are clearly documented. Such measures should be developed in line with OMB guidance, federal internal control standards, and recommended practices for evaluating and selecting discretionary grant awards (Matter for Consideration 1) We are making the following three recommendations to DOT: The Secretary of Transportation should ensure that DOT, in its notice of funding opportunity and evaluation plan for each remaining INFRA- funding cycle, clarify the circumstances under which DOT may select applicants to receive requests for additional information. (Recommendation 1) The Secretary of Transportation should develop procedures for each remaining INFRA-funding cycle to ensure that when additional information is requested from an applicant, the specific rationale behind the request is documented (for example, to promote geographic diversity among projects), as well as to ensure that DOT documents the rationale if similar projects were not afforded an opportunity to provide additional information. (Recommendation 2) The Secretary of Transportation should ensure that DOT provides information to applicants in its notice of funding opportunity for each remaining INFRA-funding cycle regarding: (1) how scores on merit criteria are used, if at all, to determine whether projects advance to the Secretary for selection, and (2) how, if at all, DOT plans to use merit scores to determine which projects should receive an award. (Recommendation 3) We provided a draft of this report to DOT for review and comment. In comments, reproduced in appendix II, DOT concurred with our recommendations. DOT noted its efforts to improve the INFRA process for the 2019 round of funding and stated that it looks forward to assisting Congress in addressing the matter for congressional consideration in a manner that is feasible within DOT’s timing and resource constraints. DOT also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Transportation, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Likely NEPA status/type of action required, based on available information (such as expecting the project to be found to have no significant impact on the environment or that the project would be required to be reevaluated) The likelihood the project will be able to be delivered by its obligation timeframe High risk = high likelihood that the project will not be obligated on time Medium risk = some possibility the project will not be obligated on time Low risk = highly likely the project will be obligated on time The Special Experimental Project authorities (SEP 14/15 waiver) is a program that identifies and tests innovative project-delivery methods (such as non-traditional contracting techniques). Susan Fleming, (202) 512-2834 or flemings@gao.gov. In addition to the contact named above, Steve Cohen (Assistant Director); Crystal Huggins (Analyst in Charge); Amy Abramowitz; Melissa Bodeau; Michelle Everett; Geoffrey Hamilton; Joshua Ormond; Oliver Richard; Kelly Rubin; and Charles Truxillo made key contributions to this report.", "summary": "The cost to repair and upgrade the nation's surface transportation system to meet current and future demands is estimated in the hundreds of billions of dollars. In December 2015, Congress established a DOT discretionary grant program to fund nationally significant freight and highway projects. DOT awarded $1.54 billion for such projects for fiscal years 2017 and 2018. GAO was asked to review DOT's process for evaluating and selecting applications for awards. This report discusses the consistency and transparency of DOT's process for evaluating and awarding INFRA grants for the fiscal-year 2017–2018 round of funding, among other objectives. GAO reviewed DOT's documentation of its evaluation process, and interviewed DOT staff and officials, as well as 11 INFRA applicants selected to ensure diversity in projects' size, type, location, and award status, as well as type of applicant. The Department of Transportation's (DOT) process for reviewing applications for grants to fund projects under the Infrastructure for Rebuilding America (INFRA) program lacked consistency and transparency in aspects related to following up with applicants and evaluating applications. Following up with applicants. DOT must determine that an applicant's project meets statutory requirements in order for the project to be eligible for an INFRA award. DOT initially found that 97 applications had insufficient information for an eligibility determination. DOT followed up with 42 of the 97 applicants to request additional information. DOT did not sufficiently document why it followed up with certain applicants over others. If DOT does not clearly communicate and document its process regarding applicant follow-up, the process lacks transparency and the assurance of fairness. Evaluating applications. In addition to the statutory requirements, DOT established merit criteria (e.g., economic vitality) to evaluate projects against, and stated that competitive projects would substantively address all of the criteria. DOT teams scored the projects on how well they addressed each criterion. However, DOT forwarded the information on all 165 projects that were found to be statutorily eligible to the Secretary for potential award, regardless of how well they scored on the merit criteria. In the end, DOT awarded some projects that did not address all of the criteria. Several applicants told GAO they were uncertain how DOT determines which projects should receive awards. In addition, DOT's documentation does not provide insight into why projects were selected for awards, an issue GAO has previously noted and recommended DOT address. The above limitations reflect long-standing issues GAO has identified in DOT's discretionary grant programs. Specifically, since 2011, GAO has recommended actions to increase consistency and transparency. In some cases, DOT implemented the recommendations for one program, but GAO later found similar problems in other programs. After finding repeated issues, GAO recommended in 2016 that DOT develop a department-wide directive that would, among other things, require that key decisions be documented. DOT agreed with the recommendation. In a March 2019 memo, DOT directed offices to implement GAO's recommendation by June 2019. However, it is unclear how this action will improve transparency and consistency because, among other things, DOT did not communicate how offices should sufficiently document decisions to ensure that the rationale for decisions is clear. The next reauthorization of surface transportation programs provides Congress the opportunity to build requirements for greater consistency and transparency into DOT's grant programs. This is particularly important as DOT has two additional rounds of INFRA funding to award under the FAST Act, and the President's Budget proposal proposed providing an additional $1 billion to INFRA. Absent effective action by DOT going forward, the recurring and long-standing issues GAO has identified could continue to affect DOT's competitive discretionary grant programs. GAO is making three recommendations, including that DOT should communicate and document the rationale for asking specific applicants for more information and provide information to applicants on how, if at all, DOT uses merit criteria scores to advance projects through its evaluation and selection process. Also, Congress should consider directing DOT to develop and implement transparency measures in the next surface-transportation reauthorization bill. DOT concurred with GAO's recommendations and provided technical comments that GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "Over the past decade, our prior work has highlighted the evolving nature of individual IDT refund fraud and the challenges IRS faces in keeping up with fraudsters’ tactics. Since 2015, our biennial High-Risk Report has highlighted the challenges associated with IDT refund fraud, the actions IRS needs to take to address them, and the cybersecurity issue of protecting PII amid large-scale data breaches. These challenges are relevant to business IDT and further compounded by the complexity of the business tax environment. According to IRS officials, this complexity stems, in part, from the number of business types or structures, the various taxes that businesses pay, and the different tax forms businesses must file. Further, many businesses file tax returns throughout the year, unlike individual taxpayers who generally file income tax returns once a year. These factors make detecting, researching, and resolving potential business IDT cases more challenging than individual IDT cases. When establishing a business, a business owner must determine the structure of the business for tax purposes, among other things, and may link business entities together in networks with multiple tiers. In addition, unlike individuals, businesses are required to pay different types of taxes depending on the business structure. For example, C corporations and S corporations pay income tax, and may also pay employment taxes and excise taxes on certain products and services such as fuel. Businesses are required to file different forms for each type of tax and may also file forms to claim various tax credits. Table 1 provides examples of business types and associated tax forms, volume, and total refunds for fiscal year 2018. IRS officials said that the complexity of the business tax environment makes it difficult for tax examiners to distinguish between true business IDT and frivolous tax arguments or noncompliance, such as incorrect or missing information on a form. Officials also noted that fraudsters may be attracted to the potential large payout associated with business tax refunds. According to IRS data, the average 2018 tax refund for corporations was about $286,200 and about $24,700 for estates and trusts. In contrast, the IRS Data Book, 2018 reports that the average individual tax refund was about $2,900. Further, business IDT may also lead to other types of tax fraud. In addition to filing false business returns seeking a refund, fraudsters may use stolen EINs and business information to support an individual income tax refund scheme. For example, fraudsters may file fraudulent Forms W- 2, Wage and Tax Statement with information on fictitious employees. These forms could then be used to file fraudulent individual tax returns seeking refunds. According to IRS, there are two ways a fraudster can commit business IDT, both of which involve the fraudulent use of the EIN. 1. Obtain an existing EIN. In this scenario, a fraudster obtains federal tax information from an existing business (see fig. 1). The business may be active or dormant, meaning that the business owner has not filed a tax return for at least two tax periods. The fraudster then uses the EIN and other key business information to file a fraudulent business return, such as Form 1120. 2. Fabricate an EIN. In this scenario, a fraudster steals the identifying information of an individual, such as a Social Security number and uses it to apply for an EIN. The fraudster would then use the fabricated EIN to complete and file false business returns. In June 2016, Congress passed and the President signed into law the Fraud Reduction and Data Analytics Act of 2015 (FRDAA), which created requirements for agencies to establish financial and administrative controls for managing fraud risks. These requirements are aligned with leading practices outlined in our Fraud Risk Framework. In addition, guidance from OMB affirms that managers should adhere to the leading practices identified in the framework. The Fraud Risk Framework provides key components and leading practices for agency managers to use when developing efforts to combat fraud in a strategic, risk-based way. The framework consists of four primary components of fraud risk management: commit, assess, design and implement, and evaluate and adapt, as shown in figure 2. Specifically, the components call for agencies to (1) commit to combatting fraud by creating an organizational culture conducive to fraud risk management, (2) plan regular fraud risk assessments and assess risks to determine a fraud risk profile, (3) design and implement a strategy with specific control activities to mitigate assessed fraud risks, and (4) evaluate outcomes using a risk-based approach and adapt activities to improve fraud risk management. According to the Fraud Risk Framework, the four components are interdependent and mutually reinforcing. For example, fraud response efforts can inform preventive activities, such as using the results of investigations to enhance fraud detection efforts. We have previously reported that preventive activities generally offer the most cost-efficient use of resources, since they enable managers to avoid a costly and inefficient “pay-and-chase” model. The framework also reflects ongoing activities for monitoring and feedback that apply to all four components. IRS uses computerized checks, or fraud filters, to screen incoming tax returns for known or suspected characteristics of fraud. As of September 2019, IRS had implemented 19 unique fraud filters that assess incoming returns on certain business and employment tax forms. These fraud filters help IRS determine if an incoming return exhibits suspicious characteristics. IRS also cross-references these returns against lists of taxpayer identification numbers previously involved in data breaches and at greater risk of tax-related identity theft. IRS officials stated that they plan to implement additional fraud filters for three employment tax forms for the 2020 filing season. Our analysis of IRS’s data shows that from January 2017 to August 2019, business IDT fraud filters stopped about 188,500 incoming business returns as potential IDT, claiming $47.6 billion in refunds. Of these, IRS performed in-depth research on about 182,700 returns claiming $47.3 billion in refunds. IRS determined that about 77 percent of these cases (140,100 cases) claiming $38.3 billion in refunds were not business IDT while about 4 percent (7,900 cases) were confirmed business IDT claiming $384 million in fraudulent refunds. The remaining cases were still under review as of August 2019. However, as we discuss later in this report, these estimates do not capture the full size and scope of business IDT. In addition to developing fraud filters, IRS has established more advanced fraud detection efforts through the Return Review Program (RRP). As of September 2019, IRS was developing and testing fraud detection models in RRP for certain business tax forms. IRS officials said they intend to develop additional models, such as those to address fuel tax credit fraud and entity fabrication. Officials also noted that they will continue to rely on fraud filters to detect potentially fraudulent business returns, even after expanding RRP’s functionality. Further, IRS’s broader fraud detection efforts include working with external partners. For example, IRS collaborates with states and industry partners through the Security Summit Business IDT sub-workgroup. This group has identified business-related data elements that are captured during the tax filing process and analyzed for potential suspicious patterns that could indicate business IDT. During the 2018 filing season, IRS analyzed 37 data elements from incoming business tax returns and 10 data elements on incoming employment tax returns, including, for example, characteristics of the computer used to submit the return. IRS officials also stated that they are working directly with tax practitioners to help improve the quality of the data they collect to better inform future business IDT fraud filters and models. In addition, in December 2017, IRS initiated a pilot project with the Alabama Department of Labor to help detect and prevent business IDT. IRS officials stated that they send the department a data extract on all newly issued EINs from the prior month. The state performs research on these businesses and, in turn, sends IRS a list of businesses that it has determined to be fraudulent. As a result, IRS is able to deactivate the fraudulent EINs before the fraudster files a false business, employment, or individual tax return claiming a refund. This allows IRS to reject returns associated with the fraudulent EINs. According to IRS data, in 2018 IRS identified about 3 percent (1,343 out of 53,826) of new EINs in Alabama as fraudulent. The early results of this collaborative effort indicate that this project shows promise, and IRS officials stated that they are working to determine if they can expand the initiative to other states. One component of our Fraud Risk Framework calls for agencies to create an organizational culture conducive to combating fraud. Such a culture can be created through “tone at the top,” whereby senior-level staff demonstrate commitment to integrity and combating fraud, and actions that involve all levels of the agency in setting an antifraud tone that permeates the organization. In addition, the Fraud Risk Framework calls for agencies to designate an entity to lead fraud risk management activities. Among other things, the designated entity should have defined responsibilities and the necessary authority to perform its role, including managing a fraud risk assessment process and coordinating antifraud activities across the program. Our prior work has shown that when agencies formally designate an entity to design and oversee fraud risk management activities, their efforts can be more visible across the agency, particularly to executive leadership. Consistent with the Fraud Risk Framework, IRS leadership has demonstrated a commitment to identifying and combating overall IDT refund fraud. For example, the agency has recognized the broad and evolving challenge of IDT refund fraud in its fiscal year 2018–2022 strategic plan. Also, as previously discussed, IRS has expanded its fraud detection activities to prevent payment of fraudulent refunds, including refunds on business-related returns. In addition, our 2019 High-Risk Report noted that IRS took significant actions to facilitate information sharing with states and industry partners through the Identity Theft Tax Refund Fraud Information Sharing and Analysis Center. Further, IRS has implemented agency-wide antifraud efforts, including bringing officials together from across the organization to discuss potential fraud risks. These efforts have helped to foster an antifraud tone across IRS, according to IRS officials. At the business unit level, four IRS entities have responsibility for detecting, preventing, and resolving business IDT, as described below. However, IRS has not designated a lead entity to design and oversee business IDT fraud risk management activities across the agency, including a fraud risk assessment, consistent with leading practices. During our interviews with IRS, we found that IRS officials were knowledgeable about the business IDT policies, processes, and outcomes in their individual unit. However, none of the entities has defined responsibilities and the necessary authority to manage fraud risk across the business units. Further, no one we spoke with could articulate an agency-wide view of the problem and its potential impact on IRS. Return Integrity and Compliance Services (RICS) is responsible for detecting potential fraud on incoming business tax returns during the “pre-refund” phase (i.e., the period from when IRS accepts the return but before it issues a refund). About 20 RICS and Integrity and Verification Operations tax examiners are responsible for researching taxpayer accounts to confirm whether or not business IDT occurred. Tax examiners are also responsible for resolving cases to both prevent IRS from paying out fraudulent refunds and ensure that legitimate taxpayers’ returns are released for processing. RICS refers cases to other IRS units if the case shows other signs of fraud, such as a frivolous return. Accounts Management (AM) is responsible for researching and resolving potential business IDT cases identified during the “post- refund” phase (i.e., after a refund has been paid). AM customer service representatives perform in-depth account research and work with taxpayers to determine if business IDT has occurred. In cases of confirmed business IDT, AM corrects related account errors and enters appropriate IDT markers on the taxpayer’s account. According to IRS officials, about five AM staff work on business IDT cases one day a week or as needed. Criminal Investigation (CI) investigates large-scale tax schemes and other financial fraud, including fraud related to IDT. Office of Research, Applied Analytics and Statistics (RAAS) is responsible for supporting RICS and other business units in identifying and developing various business IDT fraud detection capabilities. RAAS also performs analyses to help IRS determine how best to proceed with other fraud detection and prevention efforts. IRS officials stated that representatives from the four business units meet regularly to share information on cases and discuss challenges. Further, IRS officials stated that the IDT Executive Steering Committee—which last met in October 2018—is responsible for providing general oversight and guidance to business units working on IDT-related efforts. However, our review of several sets of Committee meeting minutes indicates that while RICS has briefed committee members on the status of various business IDT efforts, they have not specifically discussed business IDT program priorities, potential fraud risks, or resources. When asked why IRS has not designated an entity to be responsible for overseeing business IDT fraud risk efforts, IRS officials said its business IDT efforts may not require additional oversight because they are significantly smaller than IRS’s individual IDT efforts in terms of both case volume and number of employees. They also said that the business IDT efforts are relatively new. However, with no more than 30 IRS employees working on business IDT issues, each business unit is mainly focused on day-to-day operations. The absence of an entity to lead business IDT fraud risk efforts may contribute to the issues we identify later in this report related to identifying and assessing business IDT fraud risks consistent with leading practices and delays in resolving business IDT cases. The Fraud Risk Framework’s leading practices provide flexibility in structuring the designated entity to best support an agency’s fraud risk management efforts. For example, leading practices note that the designated entity could be an individual or a team, and can vary depending on factors like existing organizational structures and expertise within the agency. In addition, employees across an agency or program, as well as external entities, can be responsible for the actual implementation of fraud controls. For example, IRS could designate one business unit as a lead entity, or leverage existing cooperative relationships between RICS, AM, CI, and RAAS to establish a business IDT leadership team with defined responsibilities and authority for managing fraud risk. A lead entity could help provide a strategic direction, coordination across business units, and oversight for managing IRS’s business IDT fraud risks. Further, without a designated entity, it is not clear which entity would be responsible for assessing business IDT risks and documenting the results, consistent with leading practices. These activities are important to combat the evolving threat of business IDT. The Fraud Risk Framework calls for agencies to regularly plan and perform fraud risk assessments to determine a risk profile. Fraud risk assessments that align with the Fraud Risk Framework involve (1) identifying inherent fraud risks affecting the program, (2) assessing the likelihood and impact of those fraud risks, (3) determining fraud risk tolerance, (4) examining the suitability of existing fraud controls and prioritizing residual fraud risks, and (5) documenting the results (see fig. 3). Such a risk assessment provides the detailed information and insights needed to create a fraud risk profile, which, in turn, is the basis for creating an antifraud strategy for the program. IRS has taken preliminary steps to understand fraud risks associated with business IDT through data analysis efforts and internal discussions with subject matter experts. However, IRS has not fully identified and assessed fraud risks to business IDT consistent with leading practices. These practices include identifying and assessing the likelihood of inherent fraud risks, determining a fraud risk tolerance, and examining the suitability of existing fraud controls to determine if they appropriately address identified risks. IRS business units use current and prior year tax return data and information on known business IDT threats to improve existing fraud detection efforts and develop new efforts. For example, RICS and RAAS officials stated that they regularly collaborate to discuss the feasibility of new fraud filters and identify and prioritize analyses on business IDT data. This effort has resulted in IRS business units identifying 38 discrete projects to, for example, analyze existing fraud filter performance and understand business tax return filing behaviors. RICS officials stated they typically identify two to three projects to begin each year, resources permitting. In addition, IRS officials stated that at the end of each filing season, they review and analyze confirmed business IDT cases to identify any new patterns or trends that may be useful for enhancing existing fraud filters and developing fraud detection models in RRP. Further, RAAS has performed ad hoc data analyses, such as on the characteristics of fabricated entities, to help understand potential risks to the business tax environment. While these are positive steps, IRS has not assessed business IDT fraud risks consistent with leading practices in the Fraud Risk Framework. For example, IRS has not identified and documented inherent fraud risks in the business tax environment, or assessed the likelihood of their occurrence and impact on IRS—the first two steps of a fraud risk assessment process. Further, our review of past GAO, Treasury Inspector General for Tax Administration (TIGTA), and National Taxpayer Advocate reports identified issues that pose inherent risks to IRS’s business IDT efforts. These risks include weaknesses with correspondence-based authentication, EIN vulnerabilities, and the high false detection rates for IDT fraud filters. We consider these to be inherent risks due to the complex nature of the business tax environment and IRS management’s overall limited response to them. Weaknesses with correspondence-based authentication. To help verify whether a suspicious business tax return is legitimate, IRS’s business IDT procedures rely on correspondence-based authentication. This involves the taxpayer answering several brief, written questions about the business and sending this information to IRS via mail. IRS officials stated that they believe correspondence-based authentication is no less secure than other forms of authentication, such as having business owners verify their identity in-person at a Taxpayer Assistance Center or authenticating via telephone. However, unlike other forms of authentication, correspondence-based authentication is inherently less secure because it may not require the taxpayer to verify their identity using a government-issued form of identification. Consequently, IRS has less assurance that the person is the actual business owner and the return in question is legitimate. In June 2018, we reported that IRS had not performed risk assessments to identify, assess, and mitigate risks associated with correspondence- based authentication because it did not have a policy that requires regular assessments and timely mitigation of identified issues. Therefore, without a policy for conducting risk assessments for correspondence- based authentication and a plan for performing an assessment, IRS may underestimate known risks and overlook emerging threats to the tax environment. We recommended that IRS establish a policy for conducting such risk assessments and develop a plan for performing them. IRS agreed with our recommendations and, as of November 2019, had developed a draft policy for conducting risk assessments. However, IRS had not yet developed a plan for performing these assessments. IRS officials stated that they intend to address these recommendations by May 2020. EIN vulnerabilities. In February 2018, TIGTA identified concerns with IRS’s EIN application process and made 18 recommendations, including that IRS improve processes to ensure that the applicant meets the requirements for obtaining an EIN and implement policies to help detect potential abuse of the online EIN application system. IRS agreed with 15 of TIGTA’s recommendations and, as of September 2019, IRS reported that it had addressed 11 recommendations. The four unaddressed recommendations aim to improve data collection and validation in the EIN system, which could help IRS identify suspicious applications. IRS officials stated that these improvements are on hold due to limited resources and competing priorities. In addition, characteristics of the EIN may make it inherently risky and susceptible to fraudsters. According to IRS, a business’s EIN is not considered PII and is not required to be protected like a Social Security number. This may make it easier for a fraudster to obtain an existing EIN and file a fraudulent business tax return. In addition, we have previously reported that fraudsters may target paid preparers, tax software providers, and other third parties to steal taxpayer data to commit IDT refund fraud or other types of financial crimes. These data may include existing EINs or the necessary information to obtain a new EIN, making it easier for fraudsters to file fake business returns. IRS officials stated that they recognize the potential risk of the EIN application process, but must balance the needs of legitimate businesses against IRS’s responsibility to detect and prevent fraud. Officials noted that they have security measures in place to detect potentially suspicious activity in the online EIN application and fraud filters to detect when taxpayers file a return with a dormant EIN. A fraud risk assessment consistent with leading practices would help IRS establish a risk tolerance for the EIN process and determine if its existing fraud controls are sufficient to address the vulnerabilities inherent to the EIN application process. High false detection rates for IDT fraud filters. The National Taxpayer Advocate’s 2018 Annual Report to Congress noted that one of IRS’s most serious problems is a high false detection rate in its fraud detection systems. In general, the false detection rate is the number of legitimate returns selected by the IRS as potentially fraudulent, divided by the total number of returns selected as potentially fraudulent. The National Taxpayer Advocate noted that IRS’s false positive rate for individual IDT filters was 63 percent in 2018. The high rate contributed to increased processing times and delays in issuing refunds for legitimate returns. It also created additional work for IRS. Similarly, our data analysis of BMFIC data shows that IRS’s business IDT fraud filters had about an 85 percent false detection rate for returns screened by fraud filters from mid- January 2017 to December 2018. In September 2019, IRS officials described several factors contributing to the high false detection rate for business IDT fraud filters. These factors include taxpayers and tax preparers failing to update key information with IRS, cross-referenced fraud filters triggering other filters, and changes in taxpayer filing behaviors due to new tax laws. The officials said they are working to reduce the false detection rate. While it is reasonable to expect fraud filters will catch some legitimate returns, IRS has not conducted a risk assessment—or developed a fraud risk tolerance—consistent with leading practices. Determining a fraud risk tolerance would help officials determine how best to balance the risks of missing fraudulent returns with the risks of flagging legitimate returns. Doing so may also help IRS prioritize any needed improvements to existing filters. According to the Fraud Risk Framework, a fraud risk assessment is the basis for developing an antifraud strategy. Among other things, an antifraud strategy considers the benefits and costs of control activities to address risks, such as the inherent business IDT risks described above, and other risks facing the program. As of July 2019, IRS’s Wage and Investment division had identified the overall threat of business IDT as one of 12 risks it is currently facing. However, IRS’s risk documentation does not include important components of a fraud risk assessment consistent with GAO’s Fraud Risk Framework. Specifically, the documentation does not include information on the likelihood or impact of each risk, IRS’s risk tolerance, or clear plans or responsibilities for mitigating risks. A business IDT fraud risk assessment with these key items would position IRS to develop a fraud risk profile and an antifraud strategy for business IDT going forward. In addition, officials from IRS’s Office of the Chief Risk Officer stated that consistent with the Fraud Reduction and Data Analytics Act of 2015 (FRDAA), the agency compiles an annual enterprise-wide fraud risk report based on program-level risks that IRS business units identify and monitor. The Office of the Chief Risk Officer’s October 2019 report acknowledges business IDT as one of 11 enterprise fraud risks for 2019– 2020. A fraud risk assessment and a fraud risk profile on business IDT consistent with leading practices would also help support IRS’s broader efforts to report and monitor enterprise-wide fraud risks. IRS officials stated that they have not performed a formal fraud risk assessment or developed a fraud risk profile for business IDT because they have directed their resources toward identifying and addressing fraud that is occurring right now and improving fraud detection efforts. When asked whether they had plans to further identify and assess inherent fraud risks for business IDT—the first step of the fraud risk assessment process—IRS officials said they thought that the costs of identifying and assessing inherent risks of business IDT would likely outweigh the benefits given the relatively low volume of confirmed business IDT cases, compared with individual IDT refund fraud. Without assessing inherent risks, determining the likelihood, impact, and IRS’s tolerance for each risk, and examining the suitability of existing fraud controls, IRS lacks reasonable assurance that it is aware of the most significant fraud risks facing business IDT. Such an analysis would also help IRS determine whether additional fraud controls are needed and whether to make adjustments to existing controls. Further, without this critical information, IRS will be unable to develop a fraud risk profile consistent with leading practices. A fraud risk profile for business IDT may help IRS make better informed decisions about allocating resources to combat business IDT and minimize financial losses. Consistent with our Fraud Risk Framework, a fraud risk profile that considers the likelihood and impact of fraud risks, IRS’s tolerance for risk, and the suitability of existing fraud detection activities is critical for developing an antifraud strategy and ensuring that IRS has an effective approach to addressing risks to business IDT. The Fraud Risk Framework states that managers may conduct quantitative or qualitative assessments, or both, to help determine the likelihood and impact of inherent fraud risks on the program’s objectives and help estimate fraud losses and frequency. Further, federal internal control standards call for program managers to use quality information to achieve their objectives, address relevant risks, and communicate that information as necessary to internal and external stakeholders. As of September 2019, IRS was collecting fraud filter data for some, but not all, business-related forms that may be susceptible to business IDT. Our analysis of IRS’s data shows that for 2018, business IDT fraud filters covered about 88 percent of business tax forms claiming a refund (14.0 million out of 15.9 million returns) and nearly all employment tax forms claiming a refund (30.7 million out of 31.0 million returns). IRS officials stated that since 2016, they have incrementally implemented business IDT fraud filters for the most commonly filed forms. We recognize that IRS has made progress in implementing filters for commonly filed forms and that the deceptive nature of fraud makes developing accurate fraud estimates challenging. However, our analysis shows that IRS has not developed business IDT fraud filters for at least 25 additional business-related tax forms. In 2018, these forms represented about $10.4 billion in refunds. As a result, IRS is not able to analyze data from these forms for emerging fraud patterns or schemes. Further, while current business IDT fraud filters cover the most commonly filed forms, IRS has not assessed which remaining forms or fraud scenarios pose the greatest risk to IRS and taxpayers. IRS also has not determined a risk tolerance for existing fraud filters, and whether the benefits of expanding existing filters outweigh the risks of flagging legitimate returns. Given the complexity of business tax forms and the evolving nature of fraud schemes, IRS’s existing fraud filters may not be sufficient to detect different business IDT scenarios. For example, IRS has implemented two fraud filters related to business tax credits, but they are each limited to a specific scenario. TIGTA has previously reported that tax credit forms have been found to be attractive to fraudsters. For example, in 2015, TIGTA reported that fraudsters have targeted individual tax credits when filing a fraudulent tax return to increase their refund. In September 2019, TIGTA reported that IRS lacked systematic controls to identify or prevent fraudulent use of an electric motor vehicle tax credit which is available to individuals and businesses. Without additional data on business IDT, IRS cannot estimate the full size and scope of this problem. As we have previously reported, IRS’s annual Identity Theft Taxonomy (Taxonomy) is a valuable tool to inventory, characterize, and analyze available individual IDT refund fraud data and to assess the performance of IRS’s individual IDT refund fraud defenses. Following each filing season, IRS estimates the volume of returns and associated dollar amounts on attempted and prevented individual IDT refund fraud, and on refunds it paid to fraudsters. While we recognize there may be differences in how IRS estimates the extent of individual versus business IDT, the Taxonomy is a useful framework to understand the data IRS needs to estimate the size and scope of business IDT. For example, the Taxonomy estimates the number of identified individual IDT refund fraud cases where IRS prevented or recovered the fraudulent refunds (e.g., returns caught by fraud filters or suspicious refunds returned by banks). In December 2018, IRS developed a draft plan for an initial business IDT taxonomy based on two business tax forms on which IRS has collected data since 2016. IRS officials stated that they intend to begin preliminary work on this effort in December 2019. However, these efforts will be limited until IRS collects additional data. IRS officials stated that they are committed to better understanding business IDT and expanding their fraud detection and data collection efforts. However, officials said that doing so depends on the availability of resources to develop and test new fraud filters prior to each filing season. IRS may address these constraints by, for example, determining which forms or fraud scenarios pose the greatest risk for business IDT based on a fraud risk assessment and profile. This would include determining a risk tolerance for business IDT on these forms and prioritizing new filters or filter enhancements based on its risk assessment. Having additional data to better estimate the size and scope of business IDT is critical in helping IRS understand how fraudsters are evading IRS defenses. Additionally, such data will help IRS identify unknown business IDT fraud risks, allocate limited resources, assess the suitability of its existing fraud control activities, and develop tools such as a business IDT taxonomy. Further information on the size and scope of business IDT could better position IRS to assess the risk of business IDT on tax administration and inform the Congress and the public about the risk. IRS has established procedures for resolving business IDT cases in its Internal Revenue Manual (IRM) and officials described general guidelines for resolving both pre-refund and post-refund business IDT cases. However, IRS does not resolve all cases within these guidelines due to various challenges IRS could potentially address, such as correspondence-based authentication; and challenges which are more difficult to address, such as the overall complexity of business IDT cases. In addition, we found that a lack of customer service-oriented performance goals for resolving cases may also contribute to delays. Key IRS documents highlight both a commitment to combating IDT refund fraud and improving customer service for taxpayers by, for example, reducing case resolution time frames through new technologies, among other things. In addition, Office of Management and Budget guidance highlights that federal program and project managers have an obligation to ensure that their programs deliver efficient and effective services to the public. This includes assessing how well a program is working to achieve intended results, and delivering customer service to align with the program’s goals. Our review of IRS documentation found that business units have developed procedures to manage and resolve business IDT cases identified during different stages of the tax return process. For example, during the pre-refund stage, RICS notifies business taxpayers via mail if their return shows signs of potential IDT refund fraud and has been held for review. Similarly, when a taxpayer notifies IRS about potential IDT refund fraud during the post-refund stage, Accounts Management (AM) may require the taxpayer to submit a form describing how and when the fraud occurred. IRS business units have also established procedures for conducting in-depth research on taxpayer accounts to determine if a case is business IDT or another type of fraud. However, RICS and AM have had some difficulty in resolving cases within their respective guidelines, as described below. Pre-refund cases. In regards to pre-refund business IDT, cases are generally to be resolved within 90 days, according to IRS’s IRM and agency officials. RICS officials stated that they aim to meet this guideline because it provides enough time to reach the correct taxpayer via mail and for the taxpayer to respond. However, RICS has been challenged in resolving cases within 90 days. Our analysis of pre-refund business IDT cases opened from mid-January 2017 through December 2018 shows that RICS did not meet this guideline for about 87 percent of cases, including open cases. RICS also took between 6 months to 2 years to resolve about 29 percent of cases (see fig. 4). Further, our analysis found that this delay was consistent across case outcomes. On average, RICS took 136 days to resolve cases of confirmed business IDT (7,248 cases) and 171 days to resolve cases determined not to be business IDT (58,279 cases). As of August 2019, IRS had not resolved 4,649 cases which had been open for an average of 383 days. RICS officials identified several reasons for the delay in resolving pre- refund cases, including ones rooted in business IDT policies and procedures. Specifically, officials stated that communicating with the taxpayer via correspondence is the primary driver of delays in resolving cases. RICS officials stated that mail-based authentication generally takes more time because letters can get lost, thrown away, or not reach the right person. RICS officials stated that in March 2018, they began making two attempts to correspond with a business with a potentially suspicious return before closing a case, rather than one attempt. RICS made this change because taxpayers were taking longer than 45 days to respond to the letter, often after RICS had closed the case as a nonresponse. Officials stated that while they are aware of IRS’s other methods of authenticating taxpayers for individual IDT refund fraud, such as by phone or in person, they have not explored similar options for the business IDT program. As we reported in June 2018, IRS uses a risk-based approach to determine the ways in which a taxpayer can authenticate his or her identity and what data are required during the authentication process. High risk interactions include those when a taxpayer accesses prior year tax information and other PII, while lower risk interactions include a taxpayer paying a bill online. According to IRS officials, as the risk level of taxpayer interactions increases, the authentication process becomes more rigorous. This approach minimizes risk to both the taxpayer and IRS. In addition, officials identified other challenges that contribute to delays, including incorrect information on the business taxpayer’s account, nonresponses to authentication requests, and the complexity of business IDT cases, which may be more difficult to address. RICS officials noted that taxpayers do not always update the business’s responsible party with IRS when they sell or transfer a business to someone else. This can make it more difficult for IRS to contact the taxpayer when their return has been selected for review. RICS officials stated that IRS reminds business taxpayers to check and update their information each year to avoid unnecessary delays in processing tax returns; however, IRS does not require taxpayers to make updates. IRS officials also stated that a business’s failure to respond to mail-based authentication requests contributes to case resolution delays. Finally, RICS officials noted that the inherent complexity of the business IDT environment may require RICS staff to research cases across multiple IRS business units or refer cases outside of RICS, which can contribute to delays. Post-refund cases. Our review of AM procedures and discussions with officials indicate that post-refund business IDT cases are generally to be resolved within 6 months. AM officials stated they established this guideline for individual IDT refund fraud cases and extended it to business IDT cases when the program started in 2016. We analyzed post-refund cases that AM opened from July 2016 (when IRS began collecting data) through December 2018. We found that AM resolved about 84 percent of post-refund cases within 6 months. However, about 17 percent of these cases—including open cases—took more than 6 months to resolve (see fig. 5). Similar to RICS officials, AM officials cited several reasons for case resolution delays, including the complexity of the business tax environment and the need to research associated businesses, employment, and individual tax returns. AM officials also noted challenges inherent to the case research process, including that staff often pursue multiple lines of inquiry to determine a case outcome. This may involve referring cases to other business units if, for example, AM staff do not have access to a specific IRS system to complete their research. Finally, AM officials stated that AM staff do not always recognize business IDT cases and may initially classify them as an individual IDT case, which results in delays. To help address this issue, AM officials stated that management periodically reviews business IDT operations, and provides refresher training in areas where staff did not follow procedures consistently. While RICS and AM officials have stated that they have general guidelines for resolving business IDT cases, they have not established customer service-oriented performance goals. We have previously found that a fundamental element in an organization’s efforts to manage for results is its ability to set meaningful goals for performance, including customer service standards, and to measure progress toward those goals. Standards that include customer service-oriented performance targets or goals allow agencies to define, among other things, the level, quality, and timeliness of the service they provide to their customers. In the context of IRS’s business IDT efforts, a customer service-oriented goal could be, for example, to resolve a certain percentage of cases within a specific timeframe. This is particularly important for IRS because one of its strategic goals is to empower customers to meet their tax obligations by providing exceptional customer service. Identifying and implementing methods to address challenges that IRS can control—such as reliance on correspondence-based authentication— could help IRS improve its timeliness in resolving business IDT cases and address its overall strategic objective to reduce case resolution time frames. It is also consistent with OMB guidance to deliver efficient and effective services to the public. Further, establishing customer service- oriented performance goals could help IRS measure progress, identify opportunities for improvement, and communicate reasonable time frames for resolving cases to taxpayers. Case resolution performance goals may also help reduce costs to the Treasury. Specifically, IRS has a legal obligation to pay interest on refunds issued after 45 days from the due date of the tax return. This requirement includes incoming tax returns that IRS holds for review for potential business IDT but then later releases for processing. Specific and relevant performance goals for both pre-refund and post-refund cases may help IRS balance its efforts to protect revenue against the burden on legitimate taxpayers and additional costs to the Treasury. IRS has recognized business IDT as a growing threat to both taxpayers and tax administration. The complexity of the business tax environment— including different business types and taxes that businesses must pay— makes detecting, researching, and resolving potential business IDT cases more challenging for IRS compared with individual IDT cases. IRS has taken important steps to prevent business IDT, including using fraud filters to screen incoming business returns on selected forms and collaborating with state and industry partners to identify and respond to potentially suspicious activity. IRS leadership has demonstrated an overall commitment to identifying and combating IDT refund fraud. However, IRS has not designated a lead entity to design and oversee business IDT fraud risk management activities consistent with leading practices. A lead entity could also help IRS ensure its business IDT activities are better coordinated to combat the evolving threat of business IDT. Further, while IRS has taken some steps to understand business IDT fraud risks, it has not developed a fraud risk profile based on an assessment of inherent risks, the likelihood and impact of risks, IRS’s risk tolerance, and an evaluation of existing fraud controls. Assessing inherent fraud risks, such as those that we highlighted—correspondence-based authentication, vulnerability of EINs, and a high false detection rate for IDT fraud filters—would help IRS to establish a fraud risk tolerance and form the basis for an antifraud strategy. IRS has made progress in detecting and preventing business IDT by implementing fraud filters and collecting data on six business-related tax forms. However, without a risk profile, IRS does not have assurance that its existing filters mitigate inherent risks. For example, risks may also be associated with at least 25 other tax forms, and IRS has not determined which forms or fraud scenarios pose the greatest risk to IRS and taxpayers based on an analysis of risk. Collecting additional data by implementing new fraud filters would better position IRS to estimate the full size and scope of business IDT. IRS’s planning documents articulate a commitment to reducing case resolution time frames and improving customer service, but RICS and AM have been delayed in resolving business IDT cases due to various challenges. Identifying and implementing ways to address the challenges IRS can control, such as its methods for taxpayer authentication, and establishing customer service-oriented case resolution performance goals could help IRS better serve taxpayers and minimize additional costs to the Treasury. We are making the following six recommendations to IRS: The Commissioner of Internal Revenue should designate a dedicated entity to provide oversight of agency-wide efforts to detect, prevent, and resolve business IDT, consistent with leading practices. This may involve designating one business unit as a lead entity or leveraging cooperative relationships between business units to establish a business IDT leadership team. This entity should have defined responsibilities and authority for managing fraud risk. (Recommendation 1) The Commissioner of Internal Revenue should develop a fraud risk profile for business IDT that aligns with leading practices. This should include (1) identifying inherent fraud risks of business IDT, (2) assessing the likelihood and impact of inherent fraud risks, (3) determining fraud risk tolerance, and (4) examining the suitability of existing fraud controls. (Recommendation 2) The Commissioner of Internal Revenue should develop, document, and implement a strategy for addressing fraud risks that will be identified in its fraud risk profile. (Recommendation 3) The Commissioner of Internal Revenue should ensure that IRS collects additional data on business IDT by identifying and implementing new fraud filters consistent with its fraud risk profile. This should include prioritizing IDT filters for tax forms determined to be most at risk based on an analysis of risk tolerances. (Recommendation 4) The Commissioner of Internal Revenue should identify and implement methods to address delays in resolving business IDT cases due to correspondence-based authentication. This could involve using different methods for taxpayer authentication based on the risk level of the return. (Recommendation 5) The Commissioner of Internal Revenue should establish customer service-oriented performance goals for resolving business IDT cases. (Recommendation 6) We provided a draft of this report to IRS for review and comment. In written comments, which are summarized below and reproduced in appendix II, IRS’s Deputy Commissioner for Services and Enforcement agreed with five of our six recommendations and neither agreed nor disagreed with one of our recommendations. IRS agreed with our four recommendations to better identify, assess, and manage business IDT fraud risks consistent with leading practices in our Fraud Risk Framework. IRS agreed to designate a dedicated entity to provide oversight of agency-wide business IDT efforts and stated that it will determine the appropriate oversight structure and scope of authority. IRS also agreed with our recommendations to, consistent with leading practices, develop a business IDT fraud risk profile; develop, document, and implement a strategy for addressing fraud risks; and implement and prioritize new fraud filters consistent with its fraud risk profile. IRS did not provide details on the actions it plans to take to address these recommendations. In its written comments, IRS stated that formally implementing leading practices in the Fraud Risk Framework may be helpful, but noted that it has consistently completed business IDT fraud risk assessments and developed risk profiles. However, during our review, IRS did not provide evidence that it had taken such actions. Figure 3 in our report outlines leading practices for performing a fraud risk assessment and developing a risk profile. For example, regarding the leading practice to identify and assess inherent fraud risks, IRS stated that it has found that the risks associated with in-person or telephone authentication are higher for business IDT than correspondence-based authentication. However, we could not verify this assertion, as IRS did not provide evidence during our audit that it had assessed the risks of different authentication options for business taxpayers. Further, IRS stated that our report does not acknowledge that multiple individuals may be authorized to act on behalf of a business, including authenticating a potentially suspicious tax return. We have added this information to our report. IRS also stated that our report implies that it would be acceptable for a percentage of potentially fraudulent returns to be filed, unchecked, solely to reduce false detections or business costs. However, as we indicate in our report, fraud risk tolerance does not mean IRS management tolerates fraud, or that it needs to eliminate controls to detect and prevent fraud. Rather, it means that IRS management accepts a certain amount of risk, based on its assessment of the likelihood and impact of the fraud. Determining a fraud risk tolerance would help IRS management establish appropriate and cost-effective controls that are commensurate with the fraud risk. Relatedly, we agree with IRS’s statement that IDT victims suffer significant financial, social, and emotional hardships. We have updated the report’s introduction to acknowledge these hardships. In addition, IRS stated that its work on business IDT filters is more robust than stated in our report. Our report recognizes various IRS efforts to improve business IDT fraud detection and prevention, including efforts to refine its fraud filters. However, having fraud filters does not preclude IRS from identifying and assessing other potential fraud risks. Further, IRS cannot accurately determine the suitability of its business IDT filters—or other controls—without first identifying inherent fraud risks, assessing the likelihood and impact of those risks, and determining a fraud risk tolerance. Additionally, IRS did not provide evidence that it has examined the suitability of other antifraud controls, including controls to prevent fraudsters from obtaining new EINs using stolen information. IRS neither agreed nor disagreed with our recommendation to establish customer service-oriented performance goals for resolving business IDT cases. However, IRS stated that it will review its customer service- oriented performance goals and modify them, as warranted, to address the resolution of business IDT cases. Doing so would meet the intent of our recommendation. In its written comments, IRS stated that our report does not fully address obstacles that prevent timely case resolution. We have revised our discussion of pre-refund cases to more clearly identify nonresponses from taxpayers as a cause for delays. IRS also said our methodology for determining the time to close business IDT cases does not adequately consider the impact of nonresponses on the agency’s ability to close cases in a timely manner. We have added a note to figure 4 to acknowledge the challenge of nonresponses. However, IRS did not provide evidence during the audit that it collects data on how long a case is suspended while it waits for the taxpayer to respond—information that would provide insight into the challenges associated with resolving business IDT cases in a timely manner. As agreed with your offices, we plan no further distribution of this report until 30 days from the report date. At that time, we will send copies to the Chairmen and Ranking Members of other Senate and House committees and subcommittees that have appropriation, authorization, and oversight responsibilities for IRS. We will also send copies of the report to the Commissioner of Internal Revenue and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff has any questions about this report, please contact me at (202) 512-9110 or mctiguej@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff members who made major contributions to this report are listed in appendix III. Our objectives were to (1) describe the Internal Revenue Service’s (IRS) efforts to detect business identity theft refund fraud (business IDT), (2) evaluate the extent to which IRS’s efforts to prevent business IDT are consistent with selected fraud risk management leading practices, and (3) assess IRS’s efforts to resolve business IDT cases. In this report, business IDT refers to the fraudulent use of both business and employment tax forms. Both of these types of forms require an Employer Identification Number (EIN) when filing with IRS, and a fraudster can file these forms to obtain a refund. To address all of our objectives, we reviewed our prior reports on individual identity theft refund fraud and the Treasury Inspector General for Tax Administration’s (TIGTA) prior reports on business IDT. We also interviewed IRS officials from business units responsible for detecting, preventing, and resolving business IDT cases, specifically from Return Integrity and Compliance Services (RICS), Accounts Management (AM), and Criminal Investigation (CI). In December 2018, we visited IRS’s campus in Ogden, Utah, to interview officials responsible for IRS’s business IDT efforts and to observe how RICS and AM staff process and research business IDT cases using IRS information technology systems and tools. To describe IRS’s current processes to detect business IDT refund fraud, we reviewed documentation describing the business IDT fraud filters IRS implemented from 2017 through 2019, including the logic for each filter and the forms to which they apply. In addition, we analyzed data from IRS’s Dependent Database (DDb) on business IDT fraud filter results for applicable incoming business and employment tax returns IRS received from mid-January 2017 through mid-August 2019. This was the most recent, complete, and available set of data at the time of our review. This analysis showed the volume of returns selected by IRS’s business IDT fraud filters by form, tax processing year, and associated refund amount. We also analyzed data from IRS’s Business Master File Identity Check (BMFIC) system—RICS’s case management system for business IDT returns flagged by DDb—for cases opened from mid-January 2017 through mid-August 2019. These were the most complete set of data available at the time of our review. Our analysis of BMFIC data showed the number of returns that RICS researched as potential business IDT, the outcome of the case, and associated refund amounts. For the purpose of analysis and reporting, we grouped business IDT case outcomes into three categories: confirmed business IDT, not business IDT, and open/unresolved. We assessed the reliability of data from these systems by: (1) testing key data elements, including checks for missing, out-of-range, or logically inaccurate data; (2) reviewing documents for information about the data and IRS’s systems; and (3) interviewing officials knowledgeable about the data to discuss any limitations. We determined that these data were sufficiently reliable to describe the volume of incoming returns stopped by business IDT fraud filters, associated refunds, and the outcome of business IDT cases. To understand IRS’s efforts to collaborate with external partners to detect and prevent business IDT, we interviewed IRS and state officials from the Security Summit’s Business IDT sub-workgroup and reviewed IRS’s 2018 report which analyzed business-related data elements from incoming tax returns. We also interviewed IRS officials about a pilot program with the Alabama Department of Labor to help detect and deactivate potentially suspicious EINs established in that state. For context, we obtained information from January to December 2018 from IRS on the performance of this pilot, including the number of EINs identified as fraudulent. To evaluate the extent to which IRS’s efforts to prevent business IDT are consistent with selected fraud risk management leading practices, we reviewed the Fraud Reduction and Data Analytics Act (FRDAA) of 2015 and leading practices outlined in A Framework for Managing Fraud Risks in Federal Programs (Fraud Risk Framework). We generally focused our review on the first two components of the Fraud Risk Framework: (1) commit to combating fraud by creating an organizational culture and structure conducive to fraud risk management, and (2) plan regular fraud risk assessments and assess risks to determine a fraud risk profile. We reviewed agency documents and information obtained from interviews, as described below, and compared them against leading practices identified in the Fraud Risk Framework related to these two components. We reviewed IRS’s most recent strategic planning documents related to reducing fraud, IRS organizational charts, and relevant Internal Revenue Manual (IRM) sections on business IDT operations and procedures. We interviewed officials from RICS, AM, CI, and the Office of Research, Applied Analytics, and Statistics (RAAS) to understand each business unit’s respective role in detecting, preventing, and resolving business IDT cases and the extent to which business units work together on day-to-day and longer-term efforts. In addition, we reviewed IRS reports on business IDT case workload. We also reviewed meeting notes from IRS’s IDT Executive Steering Committee (July and October 2017, and January and October 2018) to understand the extent to which IRS’s executive-level groups are, for example, involved in helping guide business IDT efforts or made aware of business IDT challenges. We interviewed officials from RICS, AM, CI, and RAAS and reviewed documentation on IRS’s efforts to identify and assess business IDT fraud risks. These included reviewing RAAS’s analyses on business IDT fraud filter performance, descriptions of potential new fraud filters that IRS may implement in the future, and the Wage and Investment Division’s risk register. We also interviewed officials from IRS’s Office of the Chief Risk Officer to understand IRS’s efforts to compile and report on enterprise-wide fraud risks and agency efforts to develop an antifraud culture. Further, we reviewed documentation related to three inherent fraud risks to business IDT that we identified in the course of our work: correspondence-based authentication, EIN vulnerabilities, and high false-detection rates for IDT fraud filters. This included reviewing prior GAO, TIGTA, and National Taxpayer Advocate reports and the status of open recommendations, and relevant IRM sections. We reviewed the methodologies of these reports and found them reasonable for the purpose of describing the inherent risks related to business IDT. In addition, we identified a false detection rate for business IDT fraud filters based on BMFIC cases opened from mid-January 2017 through December 2018. To do so, we compared the number of cases IRS determined were not business IDT, relative to the total number of cases. We did not include BMFIC cases from 2019 because at the time of our analysis, about 27 percent of those cases were unresolved. We also assessed the extent to which IRS is positioned to estimate the size and scope of business IDT. To do so, we reviewed documents and information on IRS’s efforts to collect quality data on incoming business and employment returns. We compared these efforts to leading practices associated with the first two components of the Fraud Risk Framework and Standards for Internal Control in the Federal Government related to using quality information. Specifically, we determined what proportion of incoming business and employment tax forms filed in 2018 would have been screened by business IDT fraud filters, by tax form type. We also reviewed a preliminary plan and interviewed RAAS and RICS officials on their efforts to develop a business IDT taxonomy. To assess IRS’s efforts to resolve business IDT cases, we reviewed IRS procedures for managing, researching, and resolving pre-refund and post-refund business IDT cases. We interviewed officials from RICS and AM to understand the rationale behind their respective current case resolution time frames, and potential reasons for case resolution delays. We compared RICS and AM’s efforts to resolve business IDT cases against Office of Management and Budget guidance on program management and providing customer service. To determine RICS’s performance in resolving business IDT cases identified during the pre- refund phase, we analyzed 181,032 cases from BMFIC, described above. Specifically, we calculated the duration between when RICS opened the case in BMFIC to when the case was closed. In addition, we determined how many cases in RICS’s inventory were open at the time of our analysis in August 2019. For these open cases, we manually added the date we received the data as the date the case was closed. This was an indicator of the minimum amount of time RICS could have taken to close these cases. For this analysis, we did not include cases opened and closed in 2019 because we wanted to ensure there was sufficient time for RICS to research and close a case. We determined that cases opened by the end of December 2018 gave both RICS and AM (discussed below) enough time to resolve a case. In addition, we identified an anomaly in RICS’s 2019 cases. IRS officials stated that a new fraud filter inaccurately flagged incoming returns on one form, and IRS released these returns. Our analysis of RICS’s data showed that these returns accounted for about 65 percent of closed cases in 2019, and that they were resolved in an unusually short time frame (fewer than 45 days) thus skewing the overall data. We also did not include 1,679 cases that were opened and closed in zero or fewer days. To determine AM’s performance in resolving business IDT cases identified during the post-refund phase, we analyzed 1,997 relevant business IDT cases from IRS’s Correspondence Imaging System (CIS) that AM opened from July 2016 through December 2018. As discussed earlier, we did not include cases opened and closed in 2019 to allow AM enough time to research and resolve a case. We calculated the duration between when AM opened the case in CIS to when the case was closed. We also determined how many cases in AM’s inventory were open at the time of our analysis. For these open cases, we manually added the date we received the data as the date the case was closed. This was an indicator of the minimum amount of time AM could have taken to close these cases. We assessed the reliability of the CIS data by reviewing relevant documents, testing key data elements, and interviewing knowledgeable IRS officials. We determined that the data from CIS was sufficiently reliable to determine how long it took AM to resolve post- refund business IDT cases during this time period. We conducted this performance audit from July 2018 to January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Shannon Finnegan (Assistant Director), Heather A. Collins (Analyst-in-Charge), Ann Czapiewski, Michele Fejfar, Robert Gebhart, Tonita Gillich, Bethany Graham, James Andrew Howard, Krista Loose, Jungjin Park, Bryan Sakakeeny, and Rebecca Shea made significant contributions to this report.", "summary": "Business IDT is an evolving threat to both taxpayers and IRS and if not addressed can result in large financial losses to the government. The risk of business IDT has increased due to the availability of personally identifiable information and general ease of obtaining business-related information online. This makes it more difficult for IRS to distinguish legitimate taxpayers from fraudsters. GAO was asked to review IRS's efforts to combat business IDT. This report (1) describes IRS's current efforts to detect business IDT, (2) evaluates IRS's efforts to prevent business IDT against selected fraud risk management leading practices, and (3) assesses IRS's efforts to resolve business IDT cases. GAO reviewed IRS documents and business IDT fraud detection data, evaluated IRS's efforts to combat business IDT against two components of GAO's Fraud Risk Framework , analyzed case resolution data, and interviewed IRS officials. The Internal Revenue Service (IRS) has efforts in place to detect business identity theft refund fraud (business IDT), which occurs when thieves create, use, or try to use a business's identifying information to claim a refund. IRS uses computerized checks, or fraud filters, to screen incoming returns. From January 2017 to August 2019, IRS researched about 182,700 returns stopped by business IDT fraud filters. IRS determined that about 77 percent of returns (claiming $38.3 billion) were not business IDT and about 4 percent of returns (claiming $384 million) were confirmed business IDT. As of August 2019, IRS was reviewing the remaining returns. The Fraud Reduction and Data Analytics Act of 2015 created requirements for agencies to establish financial and administrative controls for managing fraud risks. These requirements are aligned with leading practices outlined in GAO's A Framework for Managing Fraud Risks in Federal Programs ( Fraud Risk Framework) . IRS has taken steps to understand fraud risks associated with business IDT but has not aligned its efforts with selected components within the Fraud Risk Framework . First, IRS leadership has demonstrated a commitment to identifying and combating overall identity theft refund fraud, but has not designated a dedicated entity to design and oversee business IDT fraud risk management efforts agency-wide. This is because the program is relatively new. Without designating an entity to help guide agency-wide business IDT fraud risk efforts, it is not clear which entity would be responsible for assessing business IDT risks and documenting the results. Second, IRS has not conducted a fraud risk assessment or developed a fraud risk profile for business IDT consistent with the Fraud Risk Framework's leading practices. Doing so would help IRS determine the likelihood and impact of risks, the level of risk IRS is willing to tolerate, and the suitability, costs, and benefits of existing fraud risk controls. IRS officials stated that they have not formally performed a fraud risk assessment or developed a risk profile because they have directed their resources toward identifying and addressing business IDT that is occurring right now and improving fraud detection efforts. Documenting a risk profile would also help IRS determine whether additional fraud controls are needed and whether to make adjustments to existing controls. Third, IRS has not assessed which business-related tax forms or fraud scenarios pose the greatest risk to IRS and taxpayers. Current business IDT fraud filters cover the most commonly filed tax forms; however, IRS has not developed fraud filters for at least 25 additional business-related forms that may be susceptible to business IDT. Without additional data on business IDT, IRS cannot estimate the full size and scope of this problem. IRS has procedures for resolving business IDT cases and has described general guidelines for resolving business IDT cases, but it does not resolve all cases within these guidelines. Further, IRS has not established customer service-oriented performance goals for resolving business IDT cases, which is inconsistent with federal guidance. Establishing performance goals may help IRS better serve taxpayers and minimize additional costs to the Treasury. GAO is making six recommendations, including that IRS designate a dedicated entity to manage its business IDT efforts, develop a fraud risk profile consistent with leading practices, implement additional fraud filters consistent with the profile, and establish customer service-oriented performance goals for resolving business IDT cases. IRS agreed with five recommendations. IRS neither agreed nor disagreed with our recommendation to establish customer service-oriented performance goals, but stated it would take actions consistent with the recommendation.", "document_type": "gao"}
{"report": "Mass transit rail operators have primary responsibility for securing their own systems. Unlike the aviation environment, where TSA has operational responsibility for screening passengers and baggage for prohibited items prior to boarding a commercial aircraft, TSA has no operational role for securing mass transit, such as employing screeners or purchasing and acquiring security equipment. Rather, TSA regularly partners with mass transit operators to address their security needs by conducting vulnerability assessments, sharing intelligence information and best practices, and working to mitigate security risks to their systems by assessing commercially available security technologies, among other measures. Mass transit operators which can be public or private entities, administer and manage all transit activities and services, including acquiring and operating any technologies designed to augment their existing security infrastructure. Securing transit systems presents inherent challenges for mass transit operators for numerous reasons. In general, mass transit systems are designed to expedite the movement of large numbers of people through multiple stations, situated along extended routes, and technologies used in the mass transit environment should not disrupt the efficiency of these operations. In addition, individual stations within these systems frequently include multiple points of entrance and exit that vary in the extent to which they may be accessed by passengers. For example, open systems include walk-up platforms with little to no barrier to entry, while other, more closed systems, typically include dedicated points of entry and exit that allow or prohibit entry access through various mechanisms. Given the size and complexity of these systems, it can be difficult for operator personnel to comprehensively monitor them for security threats. Finally, the large number of riders that pass through these systems during peak hours generally makes the sustained use of some security measures and technologies (e.g., metal detectors) difficult because such measures could result in long lines that would disrupt scheduled service. Though mass transit systems are difficult to secure, mass transit operators commonly employ a number of standard security measures for the transit environment, including: public awareness announcements and signage (e.g., reminders to report unattended baggage or suspicious activities to operator personnel immediately); use of canine teams; access controls, such as the use of lasers for intrusion detection; station design (e.g., designing transit stations to limit recess areas where bombs could be hidden, such as under a bench or a ticket machine); and video surveillance, which includes video cameras that transmit a signal to a set of television monitors to display real-time footage of transit system platforms, entrances, exits, etc. (see figure 1). Federal requirements for mass transit R&D efforts guide department and agency efforts. Specifically, the Implementing Recommendations of the 9/11 Commission Act of 2007 requires that S&T, in consultation with TSA and the Federal Transit Administration, carry out an R&D program to improve the security of public transportation systems. Additionally, Executive Order 13416, Strengthening Surface Transportation Security, requires the Secretary of Homeland Security to coordinate research, development, testing, and evaluation of technologies related to the protection of surface transportation, including commercial off-the-shelf products. To implement these requirements and other related activities to help secure mass transit systems, DHS, S&T, and TSA, in coordination with mass transit operators, have assumed the following roles and responsibilities: DHS. DHS carries out its requirement to coordinate department-wide R&D (including that for mass transit and other surface transportation modes) through its Integrated Product Team (IPT) process. The IPTs themselves are comprised of officials across DHS components, and are tasked with identifying DHS technology capability gaps, which are defined as differences between a department or agency’s current capabilities and those capabilities needed to perform its mission. Each IPT is responsible for identifying capability gaps related to a broad security area (such as preventing terrorism) and potential R&D efforts to address those gaps. Capability gaps relevant to surface transportation, including mass transit, fall under the Prevent Terrorism IPT and its Explosive Screening sub-IPT. S&T’s Research Council receives IPT information on security gaps and is responsible for prioritizing them across all IPTs. The results are used to inform which R&D research projects S&T and DHS components will undertake (see figure 2). S&T. Once capability gaps are agreed upon and prioritized by DHS components through the IPT process, S&T undertakes R&D projects intended to address the highest prioritized capability gaps. S&T can undertake R&D projects on behalf of any of the department’s components, including TSA, and will either initiate development of a new technology solution or coordinate or adapt existing technologies to meet the project’s needs. Once S&T’s R&D efforts result in a preliminary technology, or prototype, S&T will begin the testing process. S&T’s technology development process includes developmental and operational testing phases that are carried out by staff at laboratories S&T contracts with. Developmental testing is typically conducted in simulated environments, such as laboratories, test facilities, or engineering centers, which can sometimes be representative of the complex operational environment (i.e., an actual subway station). Operational testing includes field tests performed under realistic conditions by actual users (i.e., the transit operators testing in a subway or rail station) and overseen by S&T in order to determine the effectiveness and suitability of a prototype technology. Once testing is complete, S&T contracts with staff at partner laboratories and works with private sector industry partners on further developing the product for the commercial market, where it can be purchased by mass transit operators to help secure their systems. TSA. While TSA does not conduct R&D for mass transit security, it does sponsor testing of commercially available security technologies for the mass transit environment. This testing can take place in both laboratory environments and in operational, real-world environments. Regarding the latter, TSA uses its Surface Transportation Operational Test Bed Program (Operational Test Bed Program) to place commercially available security technologies in surface transportation environments, such as mass transit systems, for performance testing and evaluation. TSA established the program to assess the effectiveness of emerging and existing security technologies in real-world environments; verify prior laboratory testing performance results versus performance in a TSA mass transit system (or other surface transportation mode serving as a test bed); and develop recommendations for use of certain technologies in surface transportation. As part of the program, TSA establishes memorandums of agreement with surface transportation entities that participate in the program and also provides them with logistical support, such as installing technology and providing personnel to help operators with technology training and operating needs. As of 2019, there are nine mass transit operators participating as test beds in the program (test beds), and TSA officials reported working to add two more mass transit agencies as test beds. In addition, there are five other surface transportation test beds, including two pipeline and two freight rail test beds. Mass Transit Operators. Mass transit operators generally do not have dedicated portions of their budgets for, and therefore do not conduct, R&D. However, selected operators work with TSA to test commercially developed technology solutions intended to enhance their system security. In general, mass transit operators must assume security-related expenses for their systems, including the purchase or acquisition of surface security technologies. In 2010, S&T’s Explosives Division began work on the Surface Transportation Explosive Threat Detection (STETD) program to address the threat of improvised explosive devices (IED) on persons or in objects within a mass transit station. S&T officials stated that, as of fiscal year 2019, the STETD program remains S&T’s sole R&D program related to surface transportation. In addition, it is the only DHS technology development program focused on developing products to address the threat of IEDs in a mass transit security environment. The STETD program consists of four separate technologies designed to address explosive threats within a mass transit station, each of which is in a different stage of development and maturation. Specifically, the four technologies are known as Forensic Video Exploitation and Analysis (FOVEA), Standoff Detection, Real-Time Threat Detection Agent, and Layered Architecture. The technologies are designed to address unique aspects of the mass transit environment (i.e., multiple access points, lack of access barriers, etc.) while also working together to provide IED detection coverage from the point at which the passenger enters a mass transit station, boards a train, and then finally exits the system. These technologies would allow mass transit operators to scan large unstructured crowds to detect concealed explosives worn or carried on a person (person-borne IED), or placed in stationary objects such as baggage, or intentionally deposited in an unnoticed location for detonation by a timer or remote control (leave-behind IED). S&T intends for STETD technologies to perform without requiring checkpoint baggage screening or other measures that could impede the traveling public moving through mass transit systems during periods of high passenger volume (e.g., rush hour). FOVEA is a software suite designed to interface with video management systems already installed in mass transit systems, and is intended to help operators use recently recorded camera footage to quickly determine a person’s movement through the system. S&T began developing the technology in fiscal year 2013, and it is generally directed at helping operators identify responsible parties when objects are left behind in a mass transit system. Specifically, FOVEA includes a number of tools to enable its video analysis (see figure 3). As of December 2018, the FOVEA video suite has been installed in the Washington Metropolitan Area Transit Authority’s Security Operations Control Center. During our site visit to the control center to view the use of the FOVEA suite, officials told us that FOVEA has enhanced the ability of personnel to analyze video footage for active law enforcement investigations. Officials also told us that because of this functionality, FOVEA is a valuable tool. S&T anticipates it will transition the technology to commercial development in fiscal year 2020. S&T is developing a set of imaging sensors designed to scan unstructured crowds to detect hidden potential threat items (e.g., a person-borne IED) on travelers without requiring passengers to open bags or remove outerwear. According to S&T, the sensor technology would be placed in walls, near platforms, or other structures. These screening devices are designed to unobtrusively scan for detailed information on possible person-borne threat objects, such as wires connected to a pressure cooker, and provide alerts to operators via the Real-Time Threat Detection Agent (described below). Development of these sensors began in fiscal year 2014, and since 2017, S&T officials have been using the Massachusetts Bay Transportation Authority training facility to test prototypes of the technology. S&T expects to transition the technology to commercial development in fiscal year 2023. The Real-Time Threat Detection Agent technology is intended to automatically detect abandoned objects that could be potential threats, and notify transit operators of their existence. Specifically, the system is to analyze live video footage to identify, tag (i.e., mark on the video footage), and track left-behind objects (i.e., baggage possibly containing IEDs), as well as individuals associated with the object, without the need for continuous human monitoring of video footage. To track potential IEDs without human monitoring, the system is intended to have the capability to identify abandoned objects that could be potential threats and compare them against defined criteria for person-borne, as well as leave-behind, IED threats. Based on its analysis, the system would then create alerts and send them to operators, as well as to video review software, such as FOVEA. Development of this system began in fiscal year 2013, and S&T began developmental testing of the technology at Washington Metropolitan Area Transit Authority facilities in October 2018. S&T expects to transition the equipment to commercial development in fiscal year 2021. Layered architecture, the final component of the STETD program, is intended to have the capability to integrate information from the various existing security technologies utilized by a mass transit system to enable more accurate threat identification. The goal of this component is to gather input from multiple pieces of technology, such as distributed sensors and tools used across a mass transit environment (to include existing sensors and other STETD technologies), to present a consolidated threat profile to operators in a command center. This technological component is the least developed of the STETD program technologies, with ongoing work focused on experimentation with different prototypes. S&T expects to transition the technology to commercial development in fiscal year 2023. Since 2013, S&T’s funding for mass transit R&D has decreased, delaying the development of associated technologies. Specifically, during fiscal years 2013 through 2017 funding for the STETD program—the only DHS R&D program focused solely on mass transit security—decreased by 78 percent, but then increased again in fiscal year 2018 (see figure 4). According to S&T officials, one reason funding was reduced during this period was to direct additional funds to R&D for a newly-identified threat. Specifically, in fiscal years 2015 and 2016, following the landing of a gyrocopter on the U.S. Capitol grounds and other incidents, unmanned aerial systems became a significant and emerging threat, and a top DHS priority. At the time, S&T had no funding allocated for a related R&D effort, so S&T leadership subsequently redirected funding from the STETD program toward R&D on unmanned aerial systems. S&T officials told us that, due to fluctuations in funding, in addition to other factors, the program’s completion date has shifted from 2017 to 2023, which has delayed efforts to make the technologies commercially available to mass transit operators. Although S&T began increasing funds for the STETD effort in 2018, according to program officials, decreases in program funding have delayed program deliverables and pushed timelines out. For example, according to S&T officials, lower levels of program funding have made it difficult to employ highly-skilled contract staff at the laboratories S&T partners with to carry out STETD R&D, which has slowed the pace of development. S&T officials also stated that a lack of funding and changes in funding slow down what is already a technically challenging development effort. As S&T officials explained, the STETD program is pushing the performance boundaries and capabilities of existing technologies, and in some cases, inventing entirely new technologies for screening highly trafficked environments. S&T is not using milestones that fully adhere to DHS guidance for milestone descriptions to track its progress on developing STETD technologies. Specifically, DHS budget development guidance directs DHS components to develop program milestones that are specific, measurable, results-oriented and relevant, and time-bound. To be results-oriented and relevant, milestones must clearly link to activities in program strategy, budget, or other planning documents. Linking milestones to such activities allows parties reviewing the milestones, such as DHS leadership and Congress, to understand how achieving the milestones move the development process forward overall. We assessed all 22 STETD milestones that S&T has used to report progress on the program from fiscal years 2013 through fiscal year 2018. We found that 17 of the 22 milestones were not results-oriented as required by DHS guidance because they did not clearly link to any key activities described by STETD program documents. Specifically, one STETD program document identified several key activities for completing work on the technologies, including a requirements development phase, developmental testing, and operational testing, but STETD milestones did not clearly link to these activities. For example, One fiscal year 2018 STETD milestone for the layered architecture technology was to conduct a simulation and analysis of layered sensing configurations to optimize sensor placement and system performance, to be completed by the end of fiscal year 2018. While this milestone was specific, measurable, and time-bound, it did not clearly link to a key activity (e.g., developmental or operational testing) identified in the STETD program’s plan, and thus was not results- oriented. Another milestone from 2013 was to demonstrate advanced leave- behind detection software in a mass transit system. The milestone was to be met by the second quarter of fiscal year 2014, and remained unmet as of March 2019. Because the milestone was not results-oriented (i.e., it did not link back to activities in program documents), it was unclear how failing to achieve the milestone impacted, or potentially delayed, the overall technology development process. S&T officials explained because they are dealing with technology innovation and invention, they plan to develop milestones that closely align to program plans after they develop a potential technology solution that is ready for developmental or operational testing. However, according to STETD program plans, it can take several years for STETD technologies to begin developmental testing. For example, one STETD technology (layered architecture) is not expected to begin developmental testing until the fiscal year 2021-2022 time frame. Therefore, under S&T’s current practice, the program would not begin using results-oriented milestones, clearly linked to program plans, for this STETD technology until more than ten years after work on the program was initiated. Furthermore, we found that for one STETD technology (FOVEA), the program did not consistently use results-oriented milestones after the technology began developmental testing. Specifically, in fiscal year 2015 S&T began developmental testing for FOVEA, but, two of three FOVEA milestones reported in fiscal years 2017 and 2018 were not results- oriented. Without milestones for the STETD program that reflect DHS guidance to clearly link the milestone to key events in program planning documents, Congress and DHS decision makers cannot fully assess whether the STETD program is meeting its goals within identified time frames. Additionally, DHS decision makers are not positioned to identify adjustments that may be needed to facilitate the achievement of program goals. We previously recommended in March 2019 that S&T take steps to more fully incorporate DHS’s budget development guidance, to include more results-oriented milestones, for its R&D programs. DHS concurred with this broader recommendation, and as of June 2019, is taking initial steps to ensure its implementation. TSA sponsors tests of commercial products at contracted partner laboratories, as well as at mass transit stations and other surface transportation venues through its Operational Test Bed Program. The purpose of these tests is to inform surface transportation operators about different technological products that could address their security needs and to confirm whether the products will operate effectively. As part of the testing process, TSA officials assist in product installation, hold technical demonstrations, and provide training for mass transit officials. Once product testing concludes, TSA officials document test results in written assessments, which they make available to mass transit and other surface transportation stakeholders to review upon request. TSA officials told us they use a number of methods to identify products that are currently available in the commercial marketplace and could be tested. Officials stated they conduct market research on vendors currently making technology products that could potentially meet the needs of mass transit operators, such as portable screening devices used to detect potential person-borne IEDs. To do so, officials maintain and utilize an existing list of vendors, conduct market research on their products, attend relevant symposiums and university conferences on technological advancements, and solicit information on these products and their capabilities from both vendors and operators who have used them. TSA officials also work with national laboratories to assist with relevant research on specific technological products and their capabilities that could be good candidates for testing. To determine which technology products to test, TSA prioritizes technologies that can be used in the mass transit environment. TSA officials told us that because of the level of risk facing mass transit systems, they generally try to ensure that the commercial products that are tested address capability gaps relevant to the mass transit environment. In addition, TSA officials stated that while they try to address all identified surface transportation capability gaps, due to limited resources, they tend to select, on an annual basis, products for testing related to the following gaps: anomaly and explosive detection, high throughput threat detection, intrusion detection, infrastructure protection, and chemical and biological threat security, all of which have applicability to the mass transit environment. Moreover, TSA officials told us that anomaly and explosive detection and high throughput threat detection are the technology gaps that are critical to securing mass transit systems. TSA officials also identify other criteria used for testing, including performance requirements and vendor claims about their products’ ability to address specific capability gaps. Regarding performance requirements, TSA officials told us any products selected for testing must meet a set of minimum performance requirements in order to be considered appropriate for addressing the unique security needs for different surface transportation modalities. For example, for an explosive screening product used in mass transit, TSA officials established requirements for probability of detection and probability of false alarm. TSA officials stated they work with mass transit operators and others to identify these requirements to ensure that IED detection technologies can effectively identify threats without disrupting mass transit operations. Lastly, TSA officials said they also try to select technologies that can be used to secure multiple surface transportation modes. For example, officials said they may select an intrusion detection sensor for testing that could be adapted for securing pipelines or freight rail yards. From fiscal years 2013 through 2018, TSA sponsored laboratory and field tests of approximately 110 commercial products that are designed to address identified surface transportation capability gaps (such as intrusion detection and explosive detection). These tests take place in either of two environments—laboratory or field (i.e., within a mass transit venue)—and are designed to address surface transportation security capability gaps. Since 2013, 67 percent of the technology products (72 products) assessed by TSA focused on detection-related gaps, most of which were related to intrusion detection. The remainder of products tested addressed interoperable information systems or a combination of capability gaps, such as anomaly and explosive detection and chemical and biological threat security (see figure 5). Of the products that addressed more than one capability gap, 78 percent (14 of 18) of these products could be used for anomaly and explosive detection, as well as high throughput threat detection. TSA officials told us that because anomaly and explosive detection and high throughput threat detection technologies can be easily transported to different locations within a station, they are of particular interest to mass transit operators. On our site visits to two mass transit operators that TSA utilizes to test technology products, we observed the testing and use of commercial products. These products were designed to detect anomalies on the underside of railcars as well as among persons traversing transit platforms, terminals, and stations. The products were being used to support both normal operations and a national security special event (see figure 6). Mass transit officials told us during our site visits that they considered these commercial technologies to be useful additions to their existing security measures. They also said the test results, which TSA made available to them, were helpful in determining whether to invest in purchasing the products for long-term use. In addition to allowing TSA to perform technology assessments, the program also gives transit operators hands-on experience with technologies they are unfamiliar with. For example, an official from one mass transit test bed told us that equipment often performs differently when the manufacturer’s employees are operating it due to their prior experience with, and dedicated training on, the equipment. This official noted that transit system employees, who often do not have similar experience and training with a particular technology, can sometimes have different performance results when operating this equipment. The official told us that TSA’s Operational Test Bed Program gives transit employees an opportunity to develop structured, hands-on experience with certain products with TSA’s assistance, allowing them to understand the full potential and capabilities of the technology. This official said that, in her case, observing fellow transit employees using a particular technology for several hours convinced her of the product’s application in a real-world environment, and subsequently was an important factor in her decision to recommend its purchase for use by her transit agency. Although mass transit operators we spoke with valued the Operational Test Bed Program, TSA has decreased funding for the program since fiscal year 2013. Specifically, our analysis of program funding showed that the program experienced an approximately 70 percent decrease in funding from fiscal years 2013 through 2018 (see figure 7). TSA officials stated that recent decreases in program funding, coupled with projected funding shortfalls for the Operational Test Bed Program for 2019 through 2024, will limit the program’s capacity to conduct testing and assessments of technologies. Specifically, the TSA program manager for the Operational Test Bed Program told us that the recent decreases in funding for the program to its current level will materially impact the operation of the program moving forward. Furthermore, a TSA May 2019 budget planning document shows that, to fully meet project requirements, the program will require approximately $20 million in additional funding for fiscal years 2019 through 2024. Should the program not receive this funding, TSA officials stated they would not be able to test as many products or address as many surface transportation capability gaps through the program. They also stated that the funding shortfalls would limit TSA’s analysis of technology performance. Program managers are in the process of identifying additional funding requirements for the program through TSA’s internal budget review process. S&T, TSA, and mass transit operators regularly collaborate on issues related to identifying mass transit capability gaps and testing security technologies to address those gaps. During the course of our review, we identified four key mechanisms that S&T, TSA, and mass transit operators use to collaborate on mass transit security issues—the DHS IPT process’s sub-IPT focusing on surface transportation capability gaps (including those pertaining to mass transit); the Intermodal Transportation Systems Research and Development Working Group (RDWG); TSA’s Operational Test Bed Program; and the Transit Policing and Security Peer Advisory Group (PAG) (see table 1). We assessed the effectiveness of collaboration in these four mechanisms using our leading collaboration practices (see side bar) and found that each of them generally followed these practices. We reported in March 2019 that DHS-wide R&D collaboration has improved through the IPT process but some challenges remain, such as ensuring all components participate in the process, among other things. However, as discussed below, we found that S&T and TSA collaborate effectively through the IPT process for identifying mass transit security capability gaps and security technologies. sustained over the long-term? If leadership is shared, have roles and responsibilities been clearly identified and agreed upon? Have participating agencies clarified roles and responsibilities? participants been included? Do they have the ability to commit resources for their agency? mechanism be funded and staffed? Have online collaboration tools been developed? Written Guidance and Agreements: If appropriate, have participating agencies documented their agreement regarding how they will be collaborating? Have they developed ways to continually update and monitor these agreements? Prevent Terrorism sub-IPT on Explosive Screening. S&T and TSA collaborate on identifying surface transportation (including mass transit) capability gaps through the Prevent Terrorism’s sub-IPT on Explosive Screening. At the federal level, we found that TSA and S&T’s ongoing use of this mechanism met several leading collaboration practices, including bridging organizational cultures, leadership, clarity of roles and responsibilities, participants, and written guidance and agreements. Specifically, the Explosive Screening sub-IPT has a formal structure that is outlined in the Prevent Terrorism IPT’s charter. This written agreement establishes leadership and clarifies the roles and responsibilities of each of the participants. As key participants and voting members, S&T and TSA possess the necessary expertise to identify capability gaps for mass transit (i.e., S&T has expertise in technology research and development, and TSA has in-depth knowledge of the mass transit and other surface transportation sectors). Moreover, by requiring S&T and TSA to work together to prioritize capability gaps, the process allows them to operate across agency boundaries (i.e., to bridge their respective organizational structures). In 2012, S&T officials stated that they collaborated with TSA as members of the Explosive Screening sub-IPT to identify anomaly and explosive detection and high-throughput threat detection as the highest priority capability gaps for surface transportation. These gaps were the basis for S&T’s STETD program. RDWG. We also found that the RDWG facilitates effective collaboration between S&T, TSA, and surface transportation stakeholders, including 30 mass transit operators. The RDWG generally follows leading collaboration practices related to bridging organizational cultures, leadership, clarity of roles and responsibilities, participants, resources, and written guidance and agreements. Specifically, the RDWG is a working group with an established charter that brings together federal and surface transportation representatives to operate across agency and sector boundaries to identify surface transportation-related capability gaps, including those for mass transit. The RDWG charter clarifies the participant roles and responsibilities and establishes a framework for nominating new members to ensure all relevant participants are included. TSA, as the designated chair, funds the working group and ensures its continuation. In addition to identifying surface transportation security gaps, an S&T official told us that the members of the RDWG review the prior year’s security capability gaps to determine whether they are still relevant and if there are commercially available technologies to address them. The S&T official explained that they then use the results of these reviews to inform their work on the Explosive Screening sub-IPT, specifically using them as basis for R&D project requirements. For example, S&T officials said that anomaly and explosive and high- throughput threat detection were the gaps identified by the participants of the RDWG and then reported through the DHS IPT process, which ultimately helped inform the scope of S&T’s STETD program. In addition, S&T officials told us that they use the RDWG annual meetings to communicate to surface transportation stakeholders the progress they have made to address these gaps through the STETD program. Operational Test Bed Program. TSA’s Operational Test Bed Program facilitates collaboration between S&T, TSA, and mass transit operators on the testing and evaluation of security technologies. The program, through its memorandums of agreement with mass transit operators, generally follows leading collaboration practices related to collaboration criteria for bridging organizational cultures, clarity of roles and responsibilities, participants, resources, and written guidance and agreements. Specifically, memorandums of agreement serve as a mechanism for TSA and S&T to operate outside of their agency boundaries to test technologies in real-world environments (i.e., mass transit systems). The agreements also clarify the roles and responsibilities and serve as guidance for TSA, S&T, and mass transit operators on how testing is to be carried out. For example, the agreements clarify responsibilities for installing and operating test equipment. The program is funded and managed by TSA, and S&T and mass transit operators leverage TSA’s resources through their participation. Collaboration through the program has led to numerous benefits for TSA, S&T, and mass transit operators. According to TSA officials, the program allows TSA to fulfill federal requirements to test and evaluate mass transit security technologies and to expand the market for these products. Additionally, S&T has used the program’s established agreements with transit systems to facilitate the testing of STETD program prototypes. Lastly, transit operators use the program to obtain first-hand information on the performance of security technologies within their system. For example, an official from one mass transit operator participating in the test bed program told us they purchased two different types of passive millimeter wave scanners that they tested through the program and found to be effective. Transit Policing and Security Peer Advisory Group (PAG). The PAG facilitates collaboration among mass transit stakeholders to share information and meets the bridging organizational cultures, leadership, participants, resources, and written guidance and agreements collaboration key features criteria. Specifically, the PAG follows several practices through its charter, which designates a transit police chief as the chair of the group. The charter also outlines the participating mass transit stakeholders and allows them to share information across agency boundaries on security-related issues, including information on incident response, emerging threats, and other best practices on mitigating security issues. Officials from all nine of the mass transit operators we spoke with are members of the PAG, and seven of them stated that the PAG fosters collaboration between mass transit operators by providing a forum for transit officials to connect and share information. Particularly, one mass transit operator stated that the PAG was beneficial to the safety and security of her system because it has allowed mass transit officials to share experiences and disseminate best practices for responding to threats. Another official said that the PAG helped him stay informed about the current risks facing all of the mass transit operators, and without the group, collaboration probably would not happen. TSA engages in a number of collaborative activities to share information on security technologies with mass transit stakeholders to help improve their technology investments. Specifically, TSA disseminates a quarterly newsletter to surface transportation stakeholders (including mass transit operators) which summarizes TSA’s efforts to address various surface transportation security issues. Among other things, the newsletter shares information on the technologies for which TSA has sponsored testing in different mass transit operator systems. Additionally, TSA maintains a collective email account that was created for all mass transit operators to send TSA suggestions for technologies or products to test, among other things. Further, TSA officials stated that they notify and communicate to regional mass transit stakeholders any information on upcoming test bed demonstrations, so these stakeholders can attend in person if they prefer. Finally, according to the TSA program manager, TSA officials utilize the American Public Transportation Association’s annual conference, industry symposiums, and security roundtables to engage with mass transit stakeholders to share information on security threats, capability gaps, and technology. In addition, to assist transit operators, TSA produces a number of assessments and reports (products) that include performance information on a range of technologies. These products include: TSA’s Market Survey. TSA officials regularly update a market survey, which contains a list of commercial vendors who develop technological products capable of addressing surface transportation security issues. This list, which does not contain sensitive information and may be readily shared with operators, includes vendors and their associated products that TSA believes may be applicable to mass transit security; it does not catalog the list of products TSA has sponsored testing for. TSA officials populate this list by attending relevant symposiums and university conferences, as well as soliciting input from partner laboratories. Surface Transportation Sensor Catalog. The Surface Transportation Sensor Catalog documents the technology assessments performed by TSA and contains summaries of various security technologies evaluated since 2007. In addition to evaluated products, it also contains summaries of the vendor product demonstrations received by TSA since fiscal year 2016. TSA officials stated that the catalog is updated each year with new entries for recently evaluated products, and is intended as a resource for both TSA and its stakeholders to have greater awareness of technologies and help them make more informed technology investment decisions. State of Technology Reports. TSA publishes a State of Technology report that provides a detailed overview of a specific challenge to surface transportation stakeholders (such as person-borne IEDs) and gives a high-level summary of available products that could address it, and a technology maturation roadmap (with objectives) that needs to be implemented in order to meet surface transportation stakeholders’ operational and security needs to address those specific threats. Although these TSA products contain technology assessment and other information that would benefit mass transit operators seeking to purchase and implement security technologies, mass transit operators may not be receiving them. Specifically, TSA shares these products with transit operators upon request. However, officials from seven of the nine mass transit operators that we spoke with said they wanted more technical assessment information on commercially available security technologies, indicating that they may not be routinely requesting, and therefore not receiving, the TSA products that would provide this information. In addition, four of the nine said they would like TSA’s assessment information on technologies to be more accessible. Finally, an official from one mass transit system who previously worked for TSA on mass transit issues, and thus has knowledge of the broader mass transit community, stated that many operators would benefit from the Surface Transportation Sensor Catalog, but smaller operators are not aware of this resource and therefore do not know how to request it. TSA officials stated they do not routinely share information on security technologies with mass operators for two reasons. First, TSA officials explained that many of the in-depth reports that result from its testing of security technologies contain sensitive information and cannot be distributed without first assessing whether the requester is eligible to receive it. However, officials from most of the mass transit operators we spoke with said they would like more technical assessment information. Therefore it could be useful for mass transit operators to know when TSA publishes these reports so they can request the full report for review. This notification could consist of non-sensitive, high-level information on technologies assessed so that mass transit operators could request the information in full. Second, the TSA program manager responsible for these assessments told us that his office does not have sufficient resources to develop and maintain a centralized, web-based repository that would allow mass transit operators to search and retrieve sensitive information independently, such as the sensor catalog and technology assessment reports. Despite these limitations, in the past, TSA has shared information related to technology assessments routinely with mass transit operators. For example, TSA used to post a verified technology list on a Federal Emergency Management Agency web page. A TSA official stated that the information posted on the website included summary information on technology evaluations and other technology information. Further, TSA received feedback from surface transportation (including mass transit) stakeholders that the information posted was useful. TSA no longer posts information on the web page because the Federal Emergency Management Agency no longer maintains the website. In addition, TSA officials stated that they had plans to include more comprehensive information about TSA’s technology assessments within an online information resource known as the DHS Responder Knowledge Base—a department-wide database previously developed to house information for first responders. In April 2019 officials from DHS’s Countering Weapons of Mass Destruction Directorate said they had plans to reach out to TSA and other DHS components on how to utilize the Responder Knowledge Base as a repository for their reports and other sensitive information. Furthermore, officials from that directorate told us that the database is about 2 years from being launched, and they do not have a specific completion date. Standards for Internal Control in the Federal Government states that management should communicate externally to their stakeholders through the appropriate means. Further, the 2013 National Infrastructure Protection Plan states that in order to ensure that situational awareness capabilities keep pace with the evolving risk environment, officials should improve practices for sharing information that will improve security and resilience. Until the Responder Knowledge Base is operational, mass transit operators could benefit from TSA routinely sharing appropriate information on the technology assessments and other performance information at its disposal. For example, TSA could leverage the resources of existing coordination mechanisms, like the PAG, or develop a listserv to automatically notify a more comprehensive group of mass transit operators of the existence of a new technology assessment or sensor catalog. Notifying more mass transit operators on an ongoing basis that this information is available would help ensure they have the benefit of all relevant TSA information when making strategic security technology investments. Further, doing so would help mass transit operators to better use their limited resources to acquire proven technologies that could enhance the overall security posture of their systems. Monitoring and securing surface transportation systems continues to present unique challenges. With respect to mass transit systems, for example, operators must balance the need to efficiently move passengers through the system with the need to screen for explosives and other threats. Since 2010, S&T’s STETD program has been the only DHS R&D program that has developed technologies to address these challenges. Although S&T has made progress, as of fiscal year 2019, none of the technologies associated with the STETD program have matured enough to undergo commercial development, and the program’s completion date has been extended from fiscal year 2017 to fiscal year 2023. While fluctuations in the program’s funding have contributed to delays, S&T has not followed DHS guidance for developing milestones that would help officials understand whether the program is achieving key activities identified in planning documents when faced with funding and other challenges. Without milestones that clearly convey an understanding of the program’s progress, DHS decision makers are not positioned to identify any adjustments that may be needed to facilitate the achievement of program goals. S&T, TSA, and mass transit operators effectively collaborate through a number of stakeholder groups to identify mass transit security gaps and to test possible technology solutions that could address them. TSA also supports greater awareness of available technologies by publishing key information on commercially available products (such as technology assessment results) and making it available to mass transit operators upon request. However, TSA does not comprehensively or routinely share this information, and seven of the nine mass transit operators we spoke with stated they wanted more technology assessment information. Without a mechanism to share technology assessments and related information with more mass transit operators and on a routine basis, TSA cannot ensure that mass transit operators will be fully informed about available technologies they could use to secure their systems. Moreover, without this information, mass transit operators may not be positioned to make the best possible use of the limited funding available for purchasing these technologies. We are making two recommendations, one to DHS and one to TSA. The Deputy Secretary of Homeland Security should ensure that S&T take steps to more fully incorporate practices for developing milestones within DHS’s budget preparation guidance, into the Surface Transportation Explosive Threat Detection program. (Recommendation 1) The Administrator of TSA should develop a mechanism to more routinely and comprehensively share appropriate information on the performance of mass transit security technologies (such as the annual sensor catalog and security technology assessments) with mass transit operators and stakeholders until DHS completes work on a more permanent information sharing resource. (Recommendation 2) We provided a draft of this report to S&T and TSA for review and comment. DHS provided written comments which are reprinted in appendix I. In its comments, DHS concurred with both recommendations and described actions planned to address them. S&T and TSA also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the acting Secretary of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact William Russell at (202) 512-8777 or russellw@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. William Russell (202) 512-8777 or RussellW@gao.gov. In addition to the contact named above, Christopher Ferencik (Assistant Director), Mona Nichols Blake (Analyst in Charge), Jason Blake, Frederick K. Childers, Michele Fejfar, Jonathan G. Felbinger, Eric Hauswirth, Tracey King, Kristiana D. Moore, Claire Peachey, Jack Sheehan, Sarah Veale, and Robert Ward made key contributions to this report.", "summary": "Since 2016, bombings of subways and bus systems in foreign cities and attempted attacks in U.S. cities demonstrate continued security threats to mass transit and other surface transportation systems. S&T and TSA are the primary federal entities responsible for researching, developing, and testing technologies designed to address threats to these systems. GAO has previously identified challenges with S&T's oversight of R&D projects. GAO was asked to review S&T and TSA's roles in developing and testing surface transportation security technologies. This report, among other objectives, (1) assesses the extent to which S&T is developing technologies to secure surface transportation systems and progress made, and (2) identifies the key mechanisms that S&T, TSA, and stakeholders use to collaborate and share information on identifying capability gaps and security technologies, and analyzes the extent to which they are effective. GAO assessed S&T's mass transit program because it was the only active R&D effort for surface transportation security. GAO interviewed officials from S&T, TSA, and nine mass transit operators; observed technologies; reviewed documentation; and analyzed budget information from fiscal years 2013 to 2018. GAO also used GAO's leading collaboration practices to assess collaboration on security technologies. The Department of Homeland Security's (DHS) Science and Technology Directorate (S&T) has one research and development (R&D) effort focused on surface transportation, the Surface Transportation Explosive Threat Detection (STETD) program, which is developing technologies to secure mass transit systems (see figure). DHS guidance requires S&T to develop results-oriented milestones to track progress. GAO found, however, that S&T has not used milestones that fully adhered to DHS guidance. For example, most STETD program milestones did not clearly link to key activities described in program plans. As a result, DHS may not have the information needed to determine whether the STETD program is meeting its goals. S&T, TSA, and stakeholders effectively collaborate, but TSA could better share test results with mass transit stakeholders. For example, S&T, TSA, and mass transit operators regularly collaborate on issues related to identifying mass transit capability gaps and testing security technologies to address those gaps. Nevertheless, GAO found TSA's efforts to share information on existing technologies to secure mass transit could be improved. Specifically, TSA regularly assesses commercially available technologies, but does not routinely or comprehensively share its results with mass transit operators. For example, TSA's reports on its testing of commercially available products would provide mass transit operators with technical assessment information. However, seven of the nine mass transit operators GAO spoke with asked for more technical assessment information on existing commercial technologies, indicating that they may not be receiving the TSA products that would provide this information. Sharing this information more routinely and comprehensively with mass transit operators would allow TSA to better inform them about the capabilities of technologies that could be acquired to secure thteir systems. GAO is making two recommendations: that S&T incorporate DHS milestone guidance for its STETD program, and that TSA develop a mechanism to routinely and comprehensively share security technology information with mass transit operators. DHS concurred with both recommendations.", "document_type": "gao"}
{"report": "DHS and its components invest billions of dollars each year to acquire IT and other capabilities to support the department’s critical functions. The department plans to spend approximately $2.3 billion on major IT investments in fiscal year 2020. However, DHS has faced long-standing challenges in acquiring and managing IT. We have highlighted the department’s IT management issues on our high-risk list since 2003 and have made numerous recommendations to improve its IT management practices. For example, in 2013, we testified that, out of 68 major IT investments that the department had in development, 21 had one or more subsidiary projects that were not meeting cost and/or schedule commitments due to technical issues in the development phase, changes in agency priorities, or a lack of understanding of user requirements, among other things. Many federal agencies, including DHS, are accustomed to using a waterfall software development model. This type of model typically consists of long, sequential phases, resulting in product delivery years after program initiation. With many federal IT investments in a development phase, it is important to ensure that agencies are making the most efficient use of their financial resources through effective management practices. However, as we have previously reported and testified, historically federal IT projects often fail—that is, even after exceeding their budgets by millions of dollars and delaying the schedules by years—and the results do not meet requirements. Recognizing the severity of challenges related to the government-wide management of IT, in December 2014, federal IT acquisition reform provisions (commonly referred to as FITARA) were enacted as a part of the Carl Levin and Howard P. ”Buck” McKeon National Defense Authorization Act for Fiscal Year 2015. One of the provisions requires that the Office of Management and Budget (OMB) require in its annual IT capital planning guidance that each covered agency’s chief information officer (CIO) certify that IT investments are adequately implementing incremental development, as defined in capital planning guidance issued by OMB. Agile software development—one form of incremental development— calls for the rapid delivery of software. Probably the most well-known feature of Agile software development is iterative product development and delivery; that is, development of software in segments that are continuously evaluated against requirements. This method is well suited for programs in which the final product is to include distinct features, some of which may be discovered during the process rather than planned at the beginning. These frequent iterations can effectively measure progress and allow developers to respond quickly to feedback from customers, thus reducing technical and programmatic risk. With its emphasis on early and continuous delivery of working software, Agile can be a valuable tool for agencies in mitigating schedule and budget risks. Figure 1 compares requirements, design, development, and testing using Agile software methods versus a traditional waterfall approach; illustrating how requirements, design, development, and testing are performed concurrently in smaller time-boxed iterations for Agile and sequentially in waterfall development. As a result, using an Agile framework should result in producing high-quality software with frequent reviews and customer feedback to ensure that the highest value requirements are being met. The figure assumes that planning for both Agile and waterfall development has already occurred. In February 2016, the DHS Under Secretary for Management announced an effort to pilot the use of Agile development methodologies to improve the department’s execution and oversight of IT acquisitions. This resulted in five Agile pilot programs. Each pilot program was overseen by a component integrated program team. Collectively, the first pilot programs were also overseen and supported by a DHS integrated program team. In April 2016, the department issued an Agile instruction, which identified Agile software development as the preferred approach for all DHS programs and projects that are to deliver an IT, or embedded-IT, capability. The department also set an expectation for its component CIOs to develop plans to increase the use of Agile development and justify any major IT programs that did not intend to use Agile development practices. Many DHS programs were already using Agile or similar incremental development methods before the department identified it as the preferred approach. The DHS CIO, as the individual delegated departmentwide responsibility for approving, managing, and overseeing all of the department’s IT programs, sets the policies and procedures to help ensure Agile practices meet the department’s goals and comply with acquisition management policy. The DHS CIO is supported in this effort by the heads of other major DHS lines of business, such as the Chief Procurement Officer. Table 1 describes the roles and responsibilities that support Agile development within the department. Additionally, DHS established a headquarters-level team—the ITPM COE—to collaborate across the department on improvements to policy, governance, and acquisition guidance. In April 2017, the ITPM COE assumed responsibilities for the department’s transition to Agile development. The Office of the Chief Technology Officer (OCTO) Strategic Technology Management (STM) division within the OCIO facilitates the ITPM COE and serves as the official liaison between other OCIO divisions, other partner headquarters directorate and management offices, and operational components as needed. We have reported on various programmatic and technical challenges that were limiting DHS’ efforts on Agile programs. For example, In 2016, we reported that the U.S. Citizenship and Immigration Services Transformation program, which was using Agile software development to modernize citizenship and immigration benefits processing, needed to improve testing of its software code and ensure its approaches to interoperability and end user testing met leading practices. We made 12 recommendations to improve Transformation program management, including ensuring alignment among policy, guidance, and leading practices in areas such as Agile software development and systems integration and testing. DHS concurred with the recommendations and has thus far implemented eight of them. We reported in October 2017 that the Transportation Security Administration Technology Infrastructure Modernization program had not defined key roles and responsibilities, prioritized system requirements, or implemented automated capabilities that were essential to ensuring effective adoption of Agile. We made 14 recommendations including that DHS should prioritize requirements and obtain leadership consensus on oversight and governance changes. DHS concurred with the recommendations and to date has implemented 13 of them. In November 2018, we reported that the U.S. Secret Service OCIO did not fully measure post-deployment user satisfaction with one project supporting the Information Integration and Technology Transformation investment. We made 13 recommendations to the U.S. Secret Service including that the Secret Service establish a process that ensures the CIO reviews all IT contracts, as appropriate; and identify the skills needed for its IT workforce. DHS concurred with the recommendations but has not yet implemented them. We reported in April 2019 that the Federal Emergency Management Agency Grants Management Modernization program had not yet fully established plans for implementing new business processes or established completed traceability of IT requirements. We made eight recommendations to implement leading practices related to reengineering processes, managing requirements, scheduling, and implementing cybersecurity. DHS concurred with the recommendations and has thus far implemented two of them. According to the Project Management Institute, the practice of change management is a comprehensive, cyclic, structured approach for transitioning individuals, groups, and organizations from a current state to a future state with intended business benefits. It helps organizations to integrate and align people, processes, structures, culture, and strategy. The Project Management Institute and GAO have both described leading practices for effective organizational change management. Leading practices in organizational change management advise an agency to (1) plan for, (2) implement, and (3) measure the impact when undertaking a significant change, such as a transition from one software development approach to another. Since DHS committed to its transition to Agile software development in policy in April 2016, the department has fully developed plans to facilitate the transition. However, DHS has not fully implemented these plans and has experienced challenges in measuring progress against its intended goals. In addition, many of the plans are part of a larger effort to improve overall IT acquisitions rather than specific to a transition to Agile development, an approach that may delay DHS’s execution of these plans. Leading practices for Agile software development adoption advise an agency to focus on three organizational levels of adoption: (1) agency environment, (2) program processes, and (3) team activities and dynamics. DHS has partially adopted practices at all three organizational levels. For example, the agency activities fully supported Agile methods through actions such as developing policies and procedures that called for the alignment of software, program goals, and agency goals. However, the department’s culture can better support Agile methods by, among other things, demonstrating an incentives and rewards structure to incentivize Agile teams. Leading practices that we developed for Agile software development adoption are organized into three areas, called organizational levels: agency environment, program processes, and team activities and dynamics. The organizational levels are further divided into nine leading practices. Table 2 identifies the three organizational levels and nine leading practices associated with these levels (three practices within each area). A detailed assessment of DHS’s implementation of each of the nine leading practices can be found in appendixes III, IV, and V. We refer to the leading practices related to an agency’s processes, culture, and acquisition strategies as agency environment practices. For an agency to successfully transition from an agency that supports traditional development methods, it should ensure that its activities, culture, and acquisition policy and procedures support Agile methods. DHS partially implemented the agency environment practice level by fully implementing two leading practices and partially implementing the remaining one. A more detailed assessment of DHS’s agency environment leading practices can be found in appendix III. Agency activities support Agile methods–fully implemented. DHS established appropriate life cycle activities to support Agile methods. For example, the department has outlined its policies, procedures, and guidance in several documents to assist its components in the acquisition and implementation of Agile software development. The department also developed policies and procedures that called for the alignment of software, program goals, and agency goals. Agency culture supports Agile methods–partially implemented. DHS established an environment that supported Agile development, and senior stakeholders supported its development throughout the agency. However, DHS did not take sufficient steps to ensure that senior stakeholders serving as executive sponsors understood Agile development, as called for by leading practices that are described in further detail in appendix III. The Director of STM stated that Agile sponsors were considered to be chief executive officers (e.g. Executive Director of PARM and the Deputy Under Secretary for Management). These parties oversaw the actions of the ITPM COE and approved the Agile action plans in June 2017. In addition, the department did not require training for senior stakeholders serving as executive sponsors, as called for by leading practices. In a written response, the Office of the Chief Human Capital Officer said that there were no Agile training requirements for officials at this level. By training executive-level sponsors in Agile development, the department can mitigate the risk of setting expectations for programs and projects that do not align with the values and principles of Agile software development. DHS also did not demonstrate that it established an incentives and rewards structure to incentivize Agile teams, as called for by leading practices. Officials from the Office of the Chief Human Capital Officer stated that the department’s existing rewards structure allowed for incentivizing team and individual performance even though it was not focused specifically on Agile methods. These officials stated that they did not believe that additional policy, guidance, or modifications to their existing policy were necessary. The Director of STM within OCIO stated that rewarding Agile teams was not a topic the ITPM COE was currently considering, but that OCIO might be interested in pursuing the topic after completing existing, higher-priority activities. By considering modifications to policy and guidance governing the incentives and rewards structure to promote team performance, DHS could improve team productivity and output. Agency acquisition policies and procedures support Agile methods–fully implemented. DHS guidance for acquisition strategies supported the unique needs of Agile programs. For example, DHS offered guidance for preparing acquisition strategies through its Procurement Innovation Lab and published Agile guidance that discussed contracting and acquisition strategies. Program processes involve staff being appropriately trained in Agile methods, technical environments enabling Agile development, and project planning controls being compatible with Agile development. DHS partially implemented the program processes practice level by fully implementing one leading practice and partially implementing the remaining two. A more detailed assessment of DHS program process leading practices can be found in appendix IV. Staff are appropriately trained in Agile methods–partially implemented. DHS training policy and guidance called for some of the acquisition management program staff to be trained in Agile methods. DHS has also taken steps to incorporate Agile concepts into required training for members of the acquisitions workforce. In addition, DHS offered elective training covering Agile methods and guidance for Agile teams, including contractors, to have the appropriate technical expertise needed to perform their role. The department also took steps to identify the necessary competencies for Agile teams and individuals. In April 2019, the Strategic Workforce Planning team within OCIO published a white paper identifying 27 competencies necessary for teams and individuals to use and training courses associated with the competencies. The white paper also made recommendations to help DHS address challenges in implementing Agile methods, such as establishing communities of practice for Agile practitioners to identify best practices and provide workshops. According to a written response by OCIO, the Strategic Workforce Planning team will create an implementation and communication plan for any deliverables associated with the white paper. However, the department did not provide policy or guidance to ensure that all program staff were trained in Agile methods, as called for by leading practices described in further detail in appendix IV. Existing Agile training requirements covered only the acquisitions workforce. DHS did not establish training requirements for program staff outside of the acquisitions workforce—such as a product owner or other staff—who may be assigned to an Agile program. As a result, individual programs must independently decide on and enforce training requirements if they want to ensure that all staff receive the needed training. DHS officials stated that the department focuses on key acquisition career fields in part because those career fields are defined in policy and procedures. According to the Director of STM, the department also encourages programs to independently find coaching and training because the components are more likely to have funding. By providing policy or guidance to ensure that all personnel staffed to an Agile program or project receive appropriate training, the department can better prepare program staff to plan and execute appropriately, and increase the likelihood of achieving the expected outcomes of the transition to Agile. Technical environments enable Agile development–fully implemented. DHS guidance called for technical and project tools to be available to support Agile development. For example, DHS test and evaluation guidance stated that automated testing should be implemented where practical. In addition, DHS guidance called for system designs that will support iterative delivery. For example, DHS enterprise architecture guidance and supplementary design considerations for acquisition programs discussed loose coupling and different methods for establishing a modular system. Project planning controls are compatible with Agile development–partially implemented. DHS guidance called for defining and incorporating non-functional requirements and critical features throughout development. In addition, DHS provided guidance for establishing a sustainable development pace. For example, the Agile instruction manual identified the benefits of monitoring the amount of work completed by Agile teams across each iteration in order to monitor ongoing team progress. However, DHS was not tracking and monitoring the pace of Agile team development as called for by DHS guidance and described further in appendix IV. According to the Director of STM, programs were not consistently reporting the Agile core metrics associated with development team pace as required. The Director of STM stated that the department was taking steps to begin tracking and monitoring the pace of Agile teams. In addition, the Director stated that he allocated staff to assist programs with consistently reporting the Agile core metrics. According to the Director of STM, the department was in the process of updating the core metrics and intended to publish a new version of them in the future, which would include tracking the pace. Nevertheless, DHS did not provide assurance that the metrics associated with development pace would be included in this revised set of metrics or that programs would consistently report that information in order for the department to track and monitor the pace of Agile teams. Until the department consistently tracks and monitors Agile programs and projects, it will not have the information needed to help ensure the development pace is maintained. Practices at the team activities and dynamics level include team composition supporting Agile methods, work being prioritized to maximize value for the customer, and repeatable processes being in place. DHS partially implemented the team activities and dynamics practice level by fully implementing one leading practice and partially implementing the remaining two. A more detailed assessment of DHS team activity and dynamics leading practices can be found in appendix V. Team composition supports Agile methods—fully implemented. DHS established guidance that called for self-organizing teams and defined the role of a product owner. For example, the Agile instruction and Agile instruction manual both explain that collaborative, self- organizing, and cross-functional teams help achieve the flexibility needed for the iterative development that characterizes Agile development methods. In addition, the Agile instruction manual states that the product owner is responsible for representing stakeholders and should be available to the development team throughout the iteration to answer questions and clarify requirements on behalf of the stakeholders. Work is prioritized to maximize value for the customer—partially implemented. DHS guidance called for Agile teams to craft user stories to define work. The guidance also called for user stories to be prioritized in a backlog based on value. However, the guidance did not describe how Agile teams can estimate the relative complexity of the user stories as called for by leading practices and described in further detail in appendix V. The Director of STM stated that relative estimation is a basic exercise and that guidance on this topic can be found in a number of sources outside of DHS. However, without providing guidance or directing Agile teams to external sources for additional information on relative estimation, OCIO risks that teams supporting Agile projects will not appropriately estimate user stories relative to each other. By providing guidance on estimating the relative complexity of user stories, the department can help Agile teams to effectively commit to an appropriate amount of work during a given iteration. Repeatable processes are in place—partially implemented. DHS guidance addressed holding daily meetings to review progress and discuss impediments, using a demonstration for the acceptance of a user story and conducting a retrospective to evaluate progress. In addition, the department’s guidance called for Agile programs to employ continuous integration and emphasized the need for mechanisms to help ensure code quality. However, DHS did not set expectations for automated testing and code quality, as called for by DHS guidance and described further in appendix V. DHS’s Agile core metrics included a series of metrics that addressed automated testing and code quality. The core metrics included targets but the targets were notional and, therefore, not expectations that DHS required a program to meet. According to the Director of STM, the initial core metrics were intended to assess the level of DHS team achievement without imposing artificial industry- based target measures for each. The Director stated that, on receiving the metrics for a period of time, the department would then adjust the core metrics and begin to include target measures based on the results achieved. According to the Director, this effort is currently underway and an updated set of core metrics will be distributed in early fiscal year 2020. Moreover, the department did not track and monitor automated testing or code quality against expectations. As discussed under project planning controls, DHS intended to track and monitor Agile practices, such as automated testing and code quality, through the Agile core metrics. However, according to the Director of STM, programs and projects were not consistently reporting these core metrics and those that were reporting did not collect data or report on particular metrics. By setting expectations for automated testing and code quality and beginning to track and monitor project performance against these expectations, DHS can increase the likelihood that Agile programs and projects are delivered within cost, schedule, and performance estimates. DHS has taken many positive steps in its transition to Agile software development. It has implemented activities and artifacts that support all levels of adoption, from the department and component offices to Agile programs, projects, and teams. These activities and artifacts include providing opportunities for Agile programs and projects to streamline acquisition and life cycle processes to allow for iterative delivery and exhibiting senior support for the transition to Agile. The department successfully planned for the transition to Agile software development and completed many of its intended implementation activities. However, because DHS did not assess the skills and resources needed to complete deferred activities, it risks continued delays in completing these. In addition, without identifying target measures tied to expected outcomes, the department is limited in determining whether the transition is achieving its desired outcomes. Moreover, until DHS can ensure that all programs are consistently reporting on Agile core metrics, the department will not be able to track programs’ development techniques. Further measuring and communicating the benefits of the transition can enable the department to know whether Agile programs are performing better than those used prior to the transition. DHS has demonstrated significant progress in implementing leading Agile practices. The department can further improve its performance through full execution of the remaining partially implemented practices. At the agency environment level, DHS can mitigate risk and improve productivity through executive level training and modifications to policy to incentivize Agile teams. For program level practices, addressing training requirements for all necessary staff and tracking and monitoring the pace of Agile team development can help ensure teams’ success. With respect to team-level practices, DHS has not established guidance for estimating the relative complexity of user stories. As a result, Agile teams are hampered in effectively committing to an appropriate amount of work during a given period of time. Finally, because DHS has not set expectations for performance metrics for monitoring and tracking the use of automated testing and code quality, DHS is at a greater risk for programs breaching their cost and schedule expectations. We are making the following 10 recommendations to the Secretary of the Department of Homeland Security (DHS). The Secretary should ensure that the Director of Strategic Technology Management (STM), in collaboration with other members of the Information Technology Program Management Center of Excellence (ITPM COE), identifies the skills and resources needed to complete the work intended for the upcoming fiscal year, including the availability of supplementary staff, such as subject matter experts. (Recommendation 1) The Secretary should ensure that the Executive Steering Committee overseeing the activities of the ITPM COE establishes target measures for the department’s desired outcomes of its transition to Agile development. (Recommendation 2) The Secretary should ensure that the DHS Chief Information Officer (CIO) defines a process and associated set of controls to ensure that Agile programs and projects are reporting a set of core required performance metrics for monitoring and measuring Agile adoption. (Recommendation 3) The Secretary should ensure that the ITPM COE, in coordination with the CIO, begins measuring results associated with the transition to Agile and the success of the transition based on its impact on the department. (Recommendation 4) The Secretary should ensure that the CIO, in collaboration with the Chief Procurement Officer, through the Homeland Security Acquisition Institute, establish Agile training requirements for senior stakeholders. (Recommendation 5) The Secretary should ensure that the Chief Human Capital Officer, in collaboration with the CIO, consider modifications to the current employee recognition and performance management governance to ensure that teamwork and team performance of Agile programs and projects are incentivized. (Recommendation 6) The Secretary should ensure that the CIO, in collaboration with the Chief Procurement Officer, through the Homeland Security Acquisition Institute, establish Agile training requirements for staff outside of the acquisition workforce but assigned to Agile programs. (Recommendation 7) The Secretary should ensure that the CIO, upon establishing a set of core performance metrics, tracks and monitors the pace of Agile team development. (Recommendation 8) The Secretary should ensure that the CIO, in collaboration with the Executive Director of the Office of Program Accountability and Risk Management (PARM), update or develop new guidance on Agile methodologies to describe how Agile teams can estimate the relative complexity of user stories. (Recommendation 9) The Secretary should ensure that the CIO, upon establishing a set of core performance metrics, sets expectations for automated testing and code quality, and tracks and monitors against those expectations. (Recommendation 10) DHS provided written comments on a draft of this report. In its comments (reproduced in Appendix VI), the department agreed with our 10 recommendations and described actions that it had completed and planned to address them. Based on the actions DHS said it had taken, the department requested that we close the first three recommendations as implemented. For example, the department described steps it had taken to address our recommendation that it identify the skills and resources needed to complete the work intended for the upcoming fiscal year, including the availability of supplementary staff such as subject matter experts. In addition, the department stated that it had addressed our recommendation to define a process and controls to ensure that Agile programs and projects are reporting a set of core required performance metrics for monitoring and measuring Agile adoption. We plan to follow up with DHS to assess the sufficiency of its actions to address our recommendations. The department also described actions that it plans to take to address the other seven recommendations. For example, DHS stated that it will use the results of its Agile core metrics and Agile Software Delivery Maturity Model to measure the success of the transition to Agile and its impact on the department. According to the department, it expects this action to be completed by June 30, 2021. Further, DHS stated that it will identify Agile training requirements for staff in Agile programs, and will use that to establish Agile training requirements for staff outside of the acquisition workforce but assigned to Agile programs. Specifically, DHS stated that the DHS OCIO will gather requirements from components via its IT workforce planning integrated project team to identify training resources available across the department that also address the skill sets needed for Agile programs. The department added that the DHS OCIO will utilize information from the April 2019 white paper, titled “OCIO Agile White Paper” to inform proposed Agile program training requirements. The department estimated that these actions are to be completed by September 30, 2020. DHS also provided technical comments, which we have incorporated as appropriate. We are sending copies of this report to the Acting Secretary of Homeland Security and interested congressional committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. Our objective was to assess the extent to which the Department of Homeland Security (DHS) addressed selected leading practices for its transition to the use of Agile software development. To accomplish this objective, we assessed the extent to which the department adhered to leading practices in two specific areas: organizational change management and Agile software development adoption. With regard to organizational change management, we reviewed leading practices published by the Project Management Institute and GAO. Based on this review, we identified 15 leading practices. We then grouped these 15 practices in three broad organizational change management areas: planning, implementing, and measuring change. To determine the extent to which DHS addressed leading practices for organizational change management in its transition to Agile development, we assessed DHS policies, procedures, guidance, plans, and other working group artifacts and compared them against leading practices. In particular, we reviewed working group charters for the DHS headquarters Agile Acquisition Integrated Program Team and IT Program Management Center of Excellence (ITPM COE), and any plans developed by these working groups, including the DHS Agile Action Plans and associated implementation plans. We then reviewed working group meeting minutes, presentation slides, and status update charts to assess the progress of the transition to Agile, identified artifacts prepared to support the transition to Agile, and assessed the status of plans for the transition to Agile. We reviewed all Agile artifacts prepared by or supporting the Agile working groups, such as a preliminary software development maturity model, the DHS Agile Acquisition Software Delivery Core Metrics (Agile core metrics), and an updated test and evaluation master plan template for Agile, among other artifacts. We also interviewed officials from DHS headquarters line of business representatives explicitly identified in the Agile Development and Delivery for Information Technology instruction (Agile instruction). This included officials from the Office of the Chief Procurement Officer, Office of the Chief Financial Officer, Office of the Chief Information Officer (OCIO), Office of Program Accountability and Risk Management (PARM), and the Science and Technology Directorate, and offices of Test and Evaluation and Systems Engineering. Within OCIO, we interviewed officials from the Office of the Chief Technology Officer (OCTO) within the Strategic Technology Management (STM) division, among others, as STM is the entity tasked with facilitating the ITPM COE and serves as the official liaison between other OCIO divisions, other partner headquarters directorate and management offices, and operational components. We also interviewed representatives from groups participating in ITPM COE activities but not explicitly called out in the Agile instruction, including the Privacy Office and Joint Requirements Council. In addition, we interviewed representatives from other groups not represented on the ITPM COE but potentially impacted by the transition to Agile. This included officials from the Office of the Chief Readiness Support Officer and Office of the Chief Human Capital Officer. With regard to leading practices for Agile software development adoption, we reviewed work performed by GAO to develop generally accepted leading practices. In developing these leading practices, GAO reviewed information from a variety of sources related to Agile adoption and compiled a draft of leading practices commonly mentioned across these different sources. We then convened a working group of experts from the public and private sectors and academia. This working group met three times a year between August 2016 and August 2019 to review and discuss these leading practices. More than 200 experts participated in the meetings, including more than 20 officials from DHS. GAO received comments from some of these experts both during these meetings and by email after the meetings. Based on this work, GAO developed a set of nine leading practices for Agile adoption. GAO grouped these leading practices into three organizational levels: (1) agency environment, (2) program processes, and (3) team activities and dynamics. The leading practices were further described by a series of core elements and core element expectations that, collectively, can be used to assess the status of an agency’s implementation. To determine the extent to which the department had implemented the leading practices for the adoption of Agile development, we obtained and assessed DHS policies, procedures, guidance, plans, and other documentation and compared them against the nine leading practices. In particular, we reviewed department acquisition policy, procedures, and guidance, such as acquisition management directive 102-01; software engineering life cycle policy, procedures, and guidance, such as those published in the software engineering life cycle guidebook; requirements policy, procedures, and guidance, such as the Joint Requirements Integration and Management System and Requirements Engineering User’s Guide; testing policy, procedures, and guidance, such as the Test and Evaluation Master Plan template and Test and Evaluation Management Guide; technical assessment and enterprise architecture policy, procedures, and guidance; program health assessment policy, procedures, and guidance such as the Acquisition Program Health Assessment instruction and CIO Program Health Assessment Scoring Guideline; and Agile-specific policy, procedures, and guidance, such as the Agile instruction and the Agile Development and Delivery for Information Technology Instruction Manual (Agile instruction manual), among other policy, procedures, and guidance. In addition to reviewing the department policy, procedures, and guidance, we obtained and assessed supplementary Agile documentation. In particular, we reviewed training materials prepared by the Homeland Security Acquisition Institute for acquisition workforce certifications and webinars offered by the Procurement Innovation Lab; ITPM COE Agile artifacts discussed under our assessment of the implementation of organizational change management leading practices, such as the Agile core metrics; and Agile-specific technical review completion letters, such as the release planning review. We also interviewed officials from the components responsible for the associated policy, procedures, and guidance and those specifically cited in the Agile instruction. This included officials from the Office of the Chief Procurement Officer, Office of the Chief Financial Officer, OCIO, PARM, Science and Technology Directorate, offices of Test and Evaluation and Systems Engineering, the Joint Requirements Council, Office of the Chief Readiness Support Officer, and Office of the Chief Human Capital Officer. As with our assessment of DHS implementation of organizational change management practices, within OCIO, we interviewed officials from the OCTO STM division, among others. We assessed a core element as being “met” if the department provided supporting documentation that demonstrated it met all of the expectations associated with the core elements. We assessed a core element as being “partially met” if the department provided supporting documentation that demonstrated some, but not all, aspects of the underlying expectations. We assessed a core element as “not met” if the officials did not provide any supporting documentation for the core element, or if the documentation provided did not demonstrate any aspect of the underlying expectations. The expectations associated with each core element are described more fully in appendixes III, IV, and V. We assessed each leading practice and practice level as being “fully implemented” if DHS provided evidence that it had met all of the core elements. We assessed each leading practice and practice level as being “not implemented” if DHS did not provide evidence that it had met or partially met any of the core elements. We assessed each leading practice and practice level as being “partially implemented” if DHS provided evidence that it had not met all core elements and partially met at least one core element. To supplement our assessment of the department’s implementation of the leading practices for adopting Agile development, we also assessed selected projects’ implementation of selected program process and team activity and dynamics leading practices. We updated the core element test plans to include general control objectives, associated controls, and associated test steps in order to reach a determination on the extent to which these projects implemented a particular aspect of a leading practice. We identified potential case study projects based on data provided by DHS from the Investment Evaluation, Submission, & Tracking system. We determined that the data in the Investment Evaluation, Submission, & Tracking system was sufficiently reliable for our use in selecting projects for our case studies. We selected case study projects, rather than programs, because, according to DHS officials from OCIO, programs report software development life cycle data to the Investment Evaluation, Submission, & Tracking system at the project level only. We selected only the projects supporting programs on the Major Acquisition Oversight List because these programs are expected to comply with the Agile instruction and acquisition management policy. We then further limited the scope of projects to those within components where GAO has not previously assessed a program using Agile methods or was not in the process of assessing such a program. This excluded the U.S. Citizenship and Immigration Services, Federal Emergency Management Agency, Transportation Security Administration, and U.S. Secret Service. We then further refined our selection based on the following criteria: Software development life cycle methodology (iterative development only) Project completion date (in-progress only) DHS component (selection of only one project per component) We then selected a random sample of three projects, with no more than one project selected from a component. The three case study projects we selected were the U.S. Coast Guard (USCG) Command, Control, Communications, and Computers, Intelligence, Surveillance and Reconnaissance (C4ISR) program New Asset Acquisition Offshore Patrol Cutter project, with particular attention to the SeaWatch portion of this project; the U.S. Customs and Border Protection (CBP) Biometric Entry Exit (BEE) program Air Exit project, with particular attention to the Traveler Verification Services portion of this project; and the U.S. Immigration and Customs Enforcement (ICE) Student and Exchange Visitor Information System (SEVIS) program 8001 project, with particular attention to the SEVIS modernization portion of this effort. In preliminary interviews, we confirmed that these projects were applying Agile practices in order to validate data reported to the Investment Evaluation, Submission, and Tracking system. These case studies were used to supplement our findings from our program process and team activity and dynamics-level evaluations of the department’s implementation of leading practices for adopting Agile development. To evaluate case studies’ implementation of these leading practices, we reviewed artifacts from the selected projects. In particular, we reviewed artifacts demonstrating a project’s use of Agile including testing metrics, evidence of Agile ceremonies, the existence of user stories and a backlog, and the availability of Agile coaching and training. We then interviewed officials responsible for program and project management and representatives of groups responsible for software development for the three selected case study projects to discuss gaps we identified. We shared our initial assessment with DHS, USCG, CBP, and ICE and obtained feedback and additional supporting documentation. Regarding our analysis of project implementation of the program process and team activity and dynamics core elements, we followed the aforementioned process in assessing a core element as being “met”, “partially met”, or “not met”. These assessments were used to gain insight into the extent to which DHS policy, procedures, and guidance prepared programs and projects for the successful adoption of Agile leading practices. We did not evaluate the projects in order to make specific recommendations to the individual projects. We conducted this performance audit from December 2017 through April 2020, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In June 2017, Department of Homeland Security (DHS) senior stakeholders endorsed Recommendations Action Plans: Agile Acquisition Pilots, developed by the Agile Acquisitions Working Group. These recommendations were an effort to sustain the success of the information technology (IT) acquisition and delivery pilot program. The action plans were developed in response to the February 18, 2016, Acquisition Decision Memorandum from the Under Secretary for Management, which recognized the expressed need for both components and headquarters directorates to continue driving organizational change and process improvement to DHS IT acquisitions and delivery. The action plans were intended to codify lessons learned and recommendations based on independent interviews and retrospective meetings with those who participated in the five acquisition pilots. These plans were organized by priority: 12 critical, three high, and three moderate. The recommendations were weighted against one another based on impact, level of difficulty, and alignment with the original five goals of the Agile acquisition pilot program charter: reduce risk, increase customer value, faster time to market, economic value, and increased accountability and oversight. Table 3 describes the DHS Agile action plans, including the associated goal, primary organization(s), level of difficulty, impact, and executive priority. Each DHS action plan included a problem statement and recommendation, as detailed in table 4. This appendix describes in detail our evaluation of the three leading practices for agency environment when adopting Agile development, including a further explanation of expectations for each practice as well as some of the findings associated with each practice. We do not present any additional recommendations from these findings; this information is intended to assist the Department of Homeland Security (DHS) in implementing the recommendations described in our report. Establish appropriate life cycle activities Agency activities should support Agile methods by allowing for incremental and iterative software delivery that is tailored to the cadence of Agile software development and by incorporating technical reviews that occur throughout the development process. These activities and supporting policy and guidance should allow for requirements to be changed during development and the requirements change approval process should not impede the cadence of iterative and incremental development. Life cycle activities should also be user-focused and call for collaboration between the development team and users. To manage its multi-billion dollar investments, DHS has established policies, procedures, and guidance for IT program management. These publications govern the complete life cycle of a system, from technology development through integration and testing and, finally, implementation and operations and maintenance. DHS has outlined its policies, procedures, and guidance in several documents to assist its components in the acquisition and implementation of software development. Policies for managing its major acquisition programs are primarily set forth in a directive and supporting instruction. These policies outline an acquisition life cycle framework (ALF) that includes a series of predetermined milestones—known as acquisition decision events—at which the Acquisition Decision Authority reviews a program to assess whether it is ready to proceed to the next phase of the ALF. DHS’s Under Secretary for Management serves as the Acquisition Decision Authority for the department’s major acquisition programs. A separate DHS instruction and associated guidebook outline a framework of major systems engineering activities and technical reviews, collectively considered the systems engineering life cycle (SELC), which should be conducted by all DHS programs, both major and non-major. The SELC helps to ensure that appropriate systems engineering activities are planned and implemented and that a program’s development effort is meeting business needs. The SELC consists of major activities and a set of related technical reviews and artifacts that fit within the acquisition life cycle. Figure 2 depicts the acquisition life cycle and associated technical reviews established in DHS acquisition management policy. DHS provided programs with flexibility in their SELC technical reviews. Within the ALF, Agile processes are applied primarily within the obtain phase, where design, development, testing, and implementation of a system takes place. Prior to entering the obtain phase, a program selects its software development approach, such as Agile. The agreed-upon approach is then codified in an SELC Tailoring Plan, which is approved at acquisition decision event 2A. The SELC Tailoring Plan identifies the technical reviews and artifacts that the program is responsible for completing based on its unique characteristics (e.g., scope, complexity, and risk). To assist in tailoring efforts and further guide the implementation of Agile, DHS published an Agile instruction that includes the scope, definitions, roles and responsibilities, and procedures for establishing an Agile framework for developing all DHS IT acquisitions. DHS supplemented this instruction with an Agile instruction manual and provided a template that Agile programs can follow to tailor their activities. For example, instead of holding a system definition review, an Agile program is encouraged to conduct a release planning review (which encompasses the development and release of a segment of software). This optional approach to tailoring a technical review is depicted in figure 3. Outside of technical reviews, DHS updated acquisition policy in February 2019 and associated guidance in May 2019 to allow programs greater flexibility in the larger ALF. The Director of Strategic Technology Management (STM) stated that, under the previous acquisition policy and guidance, IT programs were using in-house expertise due to limited funding to prepare for the 2B decision, when full program funding was received. He noted that, by the time a contract was awarded for development following a 2B decision, the contractor might or might not have been using planning artifacts developed by the program and instead might have recreated them, thereby rendering 2 to 3 years of work useless. The Director stated that programs were unable to fully flesh out the program architecture and other key aspects of the program because programs did not receive funding until the 2B decision and in-house expertise was limited. For example, if a program had not proven out its architecture prior to a 2B decision, it could continue to refine and modify the architecture during the course of development, thereby impacting productivity and quality. DHS updated acquisition management policy and guidance to modify the requirements for the acquisition decision events and addressed a related GAO recommendation. DHS policy and guidance also allowed for programs to modify requirements over the course of development. The traditional process for requirements may be modified as part of tailoring the SELC in order to allow for increased flexibility. The DHS Requirements Engineering User’s Guide detailed requirements engineering steps, activities, and methods for performing those steps. DHS developed this user guide to supplement SELC policy and guidance. One section of this guide focused on Agile development. According to the guide, requirements are broken down over the course of the ALF and commitments are made at different levels of specificity. Fundamental capability gaps are defined in the mission needs statement presented at acquisition decision event 1. Subsequently, the analyze/select phase would ultimately define the high-level features and functions of each required capability, define the fundamental performance of those high- level features and functions, and establish the business case to support approving the acquisition at an acquisition decision event 2A. Often, a preliminary concept of operations is developed and delivered with the mission needs statement. The guide also states that the activities to evaluate these potential alternatives will ultimately result in a preferred solution with defined business practices, methods, and processes that allow the development of business epics and associated architecture epics. Business epic is an Agile term that defines the high-level “stories” that describe a capability, or what the new system is required to perform. Architectural epic is an Agile term that defines the architecture the system will be incorporated into. In addition, the preferred solution would have defined high-level performance requirements (stated from the operational perspective) in terms of how well the solution must perform to be operationally effective and suitable. Key constraints such as security, Section 508 compliance, privacy, reliability, etc., should also be identified. These top-level requirements will be documented in the operational requirements document. According to the guide, Agile teams capture the capabilities and constraints (essentially the functional and non-functional requirements that reflect the business epic level of performance) in an artifact called the capabilities and constraints document. Requirements statements in this document should follow the standard “shall statement” format for ease of translation between the operational requirements document and the capabilities and constraints document. The capabilities and constraints document and its contents mature over time and, as the document matures, business and architectural epics decompose to features/functions or themes, and ultimately to user stories that reflect the specific tasks that users will perform. Officials within the DHS Joint Requirements Council noted that headquarters involvement occurred at this level to approve the high-level operating requirements. After headquarters oversight and approval, the program may then decompose requirements as part of planning for and executing technical reviews. If tailored into an Agile program, the capabilities and constraints document should drive the development of a backlog. The backlog is a list of all the user stories that describe what the system needs to do. The backlog should become more refined as the program decomposes the high-level features (a service that fulfills a user need) and functions down to specific stories that an individual software developer will code and test during a specific iteration. To prevent the backlog from becoming unmanageable, DHS guidance stated that backlogs may be established at different levels. For example, the business and architectural epics along with the associated operating requirements would constitute the “program backlog.” Sub-epics are usually broken down into “high-level features” with business epics broken down into business features and architectural epics broken down into architectural features. Features or functions are decomposed into detailed stories that are then allocated to a “release”. The list of user stories in a specific release constitutes the release backlog. This process of decomposing stories continues to the iteration backlog. DHS guidance places an emphasis on end user needs. The Requirements Engineering User’s Guide raised the importance of identifying stakeholders, including system users, and capturing the needs of those users via requirements or, in the case of Agile, user stories. The Agile instruction manual placed an emphasis on the importance of users to a program and articulated that the product owner represent the user community and was expected to continually seek ongoing feedback and elicit requirements from users. The Agile core metrics also strongly recommended the use of a net promoter score. This score was one mechanism for measuring customer satisfaction through asking users to rank how likely they would be to recommend the system or application to a friend or colleague, based on a score of 1 to 10. Clearly align goals and objectives Program goals should clearly reflect stakeholder needs and concerns based on input from stakeholders and stakeholder review and approval. Program goals should align with strategic IT objectives. Software-related goals should be defined and clearly aligned with program goals. The agency should collect objective measures that are well defined to track progress towards achieving software goals so the agency knows which features and capabilities have been achieved. The Requirements Engineering User’s Guide described program expectations for tracing from mission needs to operating and functional requirements, or user stories. The guide recognized that, as a program progressed through the ALF and SELC, it was important to trace requirements from the top-level mission needs or capabilities and/or business requirements down to the system/sub-system, component, or configuration item level that enabled those requirements to be met. This helped ensure continuity across various DHS artifacts, such as the program’s mission need statement, concept of operations, and operational requirements document, to vendor specifications (or applicable equivalent artifacts). Although an Agile program will modify the SELC to accommodate its needs, generally programs were expected to follow the same conceptual approach to the requirements of planning, development, and management. The user’s guide stated that collaboration among the various stakeholders was important and the program requirements team must continuously work to establish partnerships and networks. To do so, the guide stated that the program team must identify all individuals and organizations that may be impacted by their program and ensure those stakeholders were engaged throughout development to facilitate understanding of their perspectives and needs. The first step was to identify applicable stakeholders, which would include end users, program sponsors, developers, maintainers, trainers, and other affected individuals or organizations. The program requirements team then solicited input from these stakeholders to understand their needs, policies, processes, and operations to begin the requirements definition effort. It identified some ways a team might begin the process of eliciting requirements from the stakeholders. After collaborating with stakeholders, the stakeholder needs must be translated into the program requirements, or goals. The guide stated that the program requirements team should take the inputs from the various stakeholders and decompose, prioritize, de-conflict, and validate the needs identified. It clarified that a “good” requirement was achievable, testable, clear, concise, technology-independent, feasible, and able to stand alone. The guide grounded all of the requirements elicitation and development process in the overall contribution to the agency mission, recognizing the need for general strategic alignment. In particular, the guide noted that requirements were “mission need” driven as opposed to “solution” driven. Requirements were developed throughout the life of a program, with the first formal requirements being the operating requirements documented in the operational requirements document. To ensure that DHS’s mission or strategic goals were key inputs for decision making, DHS relied, in part, on its enterprise architecture process. DHS policy for enterprise architecture stated that the enterprise architecture program provided a vehicle to tie the strategic mission goals and objectives of DHS to the business processes, information resources, and technology investments necessary to reach key performance outcomes. This methodology was intended to integrate IT into the mission and strategic priorities of DHS, which provided the core foundation for all subsequent processes. DHS capital planning and investment control guidance reinforced this fact, stating that the Federal Acquisition Streamlining Act of 1994 required capital investments to align with mission and strategic goals. This included the framework within which the department formulated, managed, and maintained its portfolio of investments as critical assets for achieving success in the DHS mission and alignment to the DHS IT Strategic Plan and the DHS Strategic Plan. Cascading sponsorship for Agile software development Senior stakeholders should support and model the use of Agile, along with its values and principles, through explicit policy or guidance impacting the business and should take steps to complete responsibilities defined in agency Agile policy or guidance. Agile should also be supported in all relevant areas of the business impacting a software development project through the use of Agile sponsors. These sponsors should represent the lines of business in key agency decisions on Agile. Senior stakeholders at DHS demonstrated support for Agile through the publication of policy and guidance that established Agile development as the department’s preferred approach for software development. As discussed previously, the department published Instruction 102-01-004 Agile Development and Delivery for Information Technology (Agile instruction), which provided the scope, definitions, roles and responsibilities, and procedures to establish an Agile framework for the development of IT acquisitions at DHS. Specifically, the Agile instruction established responsibilities for the CIO, the Chief Procurement Officer, the Chief Financial Officer, the Director, Office of Test and Evaluation within the Science & Technology Directorate, and the Executive Director of PARM. Each of these five stakeholders and their associated components demonstrated their support for Agile development by taking steps to complete their responsibilities defined in the Agile instruction. For example, the Office of the Chief Information Officer (OCIO), the Office of the Chief Procurement Officer, and the Director, Office of Test and Evaluation within the Science and Technology Directorate all had responsibilities related to providing guidance for the implementation of Agile within their specific area of expertise. All three components had taken steps to execute these responsibilities, such as by publishing the Agile instruction manual, providing supplementary guidance for test and evaluation in an Agile environment, and offering elective training on contracting strategies for Agile services. Representatives from offices with a role in software development also supported Agile via membership in the IT Program Management Center of Excellence (ITPM COE). In addition to the stakeholder organizations identified in the Agile instruction, the ITPM COE membership included representatives from the Joint Requirements Council and the Chief Privacy Officer. According to the ITPM COE charter, the ITPM COE served as a cross-functional team to identify and promote best practices, provide tools, and coordinate assistance for programs and projects to maximize the successful management of DHS IT investments. This included making progress towards the 18 Agile action plans that resulted from the Agile acquisition pilots. The ITPM COE membership requirements called for representatives of the member organizations to be involved in key decisions regarding Agile. According to the Director of STM within OCTO, ITPM COE members were selected and approved by their organization’s executives. The ITPM COE charter stated that these representatives must be authorized to represent or make decisions on behalf of their officers or organizations. Officials from all ITPM COE member organizations expressed support for the ITPM COE and confirmed that their component was appropriately represented in decision making. This was represented, in part, by the fact that at least one representative for each ITPM COE member group attended at least half of the meetings. For example, at least one representative from the Science and Technology Directorate attended approximately 95% of the meetings. Sponsor understanding of Agile software development Sponsors should understand and communicate changes resulting from Agile development. Sponsors should attend training or receive coaching on Agile and the agency’s framework, the agency should monitor completion of training, and sponsors should transmit learning from training to staff. Sponsors should also commit to achieving those intended results and sponsor performance should be tied to achieving those intended results. The Director of STM stated that Agile sponsors were considered to be chief executive officers (e.g. Executive Director of PARM and the Deputy Under Secretary for Management). They oversaw the actions of the ITPM COE and approved the Agile action plans in June 2017. DHS did not ensure that Agile sponsors attended training or received coaching in Agile development. The department made training available for Agile, including courses such as those required for acquisition professionals. However, in a written response, the Office of the Chief Human Capital Officer stated, and the Director of STM confirmed, that the department did not administer mandatory training on Agile for Agile sponsors. The department also did not monitor the completion of sponsor training in Agile. Although DHS employees leveraged the Federal Acquisition Institute Training Application System to track their training and certifications, the department was not using this system to monitor sponsor training in Agile. According to a written response from the Office of the Chief Human Capital Officer, the department did not keep a record of whether sponsors completed training in Agile because the department did not require Agile training specifically for sponsors. DHS Agile sponsors exhibited support for achieving the intended results from the transition to Agile. Agile sponsors committed to achieving these results through an endorsement of the 18 Agile action plans and the associated implementation plans. However, DHS did not demonstrate that Agile sponsor performance was tied to achieving the intended results of the transition to Agile. According to a written response from the Office of the Chief Human Capital Officer, the department’s employee performance management policy did not specifically address Agile. This written response further stated that addressing Agile in these policies was unnecessary because the Office of the Chief Human Capital Officer incorporated goals derived from project plans in individual performance plans. DHS policy and guidance for performance management identified individual performance goals as a component of employee performance, but the department did not provide evidence that specific performance plans for the sponsors were linked to such goals. Establish an environment supportive of Agile software development Team dynamics should be facilitated through access to common team rooms and/or modern communication and social media methods and headquarters infrastructure operations should allow for communal spaces and co-location in program offices. A headquarters technical environment should allow access to tools by programs to foster distributed communication, and there should be a process for continuous feedback on the Agile environment and modifications to that process (e.g. communities of practice, routine working group sessions). Agency governance bodies should allow programs greater autonomy and flexibility within existing acquisition processes through the modification of gate reviews and other touchpoints in the acquisition process for Agile projects and increased transparency for governance bodies into project operations when necessary. DHS policy and guidance allowed for team dynamics to be facilitated through access to common team rooms and modern communication methods. In addition, department policy promoted and allowed program offices to support team dynamics through the use of communal spaces and co-location. Specifically, the Director of Systems and Information Integration within the Chief Readiness Support Office confirmed that DHS had modified policy related to infrastructure operations to allow any office to reorganize their space, citing the USCIS Transformation program as an example of this reorganization. The Director of Systems and Information Integration also noted that he was not aware of any restrictions to this practice in policy. With respect to facilitating access to modern methods of communication, DHS offered programs the option of using a suite of tools that included those for distributed communication. DHS took multiple steps to establish a process for continuous feedback related to the department’s Agile environment and process modifications. According to the Director of STM, OCIO built support for Agile through the Centers of Excellence, communities of interest, brown bag lunches, and public speaking engagements. The Director added that these sessions facilitated the discussion of Agile and could be used to compile feedback. The Director of STM explained that, as this feedback came in, it was either addressed immediately or put into a backlog. Efforts to further streamline the acquisition process were tracked via Agile action plan 6. The department’s governance bodies also increased transparency into project operations when necessary. The Agile Development and Delivery for Information Technology Instruction Manual (Agile instruction manual) stated that the program or project manager should coordinate with the various oversight bodies that govern IT development. These bodies varied depending on the level of investment, but, for major programs, executive steering committees were often established to oversee all aspects of program planning and execution between major acquisition decision events. In addition, PARM officials stated that DHS increased the frequency of acquisition review board reviews and modified the content presented at the reviews to allow it to be more actively involved with projects earlier in the acquisition life cycle. Specifically, PARM updated the Acquisition Review Board slide templates and informed us of its intent to update acquisition management policy to require Agile projects to hold Acquisition Review Board reviews once every six months, as opposed to once every 12 months. Align incentives and rewards to Agile methods The agency should establish an incentive and reward structure promoting team successes and the value of individuals within those teams. Management should establish agency goals to align incentives and rewards with Agile methods. Goals for incentives and rewards should align with the agency’s goal(s) and focus on team success. The agency should allocate incentives and rewards based on team success. DHS did not establish an incentives and rewards structure that promoted team successes and did not demonstrate that management had established agency goal(s) to align incentives and rewards with Agile methods. Furthermore, the department did not demonstrate that human resources and others were actively involved in setting goals for incentives and rewards alignment. DHS guidance specifically discussed contract incentives for Agile projects. For example, the Agile instruction manual suggested that consideration be given to address the duration of the base term and options, scalability, deliverables, and pricing with a mindset that contractors need appropriate incentives to encourage them to perform well. The manual also stated that contract award terms could provide a greater incentive for contractors working on longer-term Agile projects. Although the department made efforts to adapt incentives and rewards for contractors supporting Agile projects, it acknowledged that it did not update existing incentives and rewards for federal employees working on Agile projects. Officials within the Office of the Chief Human Capital Officer stated that existing human capital and performance plan policy allowed for rewarding and incentivizing Agile teams as well as individuals. These officials further noted that DHS had numerous opportunities to recognize and reward team or individual performance, regardless of the development methodology a program relied on. Specifically, these officials clarified that the Office of the Chief Human Capital Officer used project plans to set goals and included those goals in employee performance plans. As these officials felt the existing performance plan policy was sufficient, they did not believe additional guidance or modifications to existing policy were necessary. Guidance is appropriate for Agile acquisition strategies Agency acquisition policy and guidance should support awarding contracts for the unique needs of an Agile program. Acquisition strategies should recognize the need for interim delivery of software, allow for close coordination between the contracting office and program office staff, and allow for changing requirements and contract oversight mechanisms to be tailored to support Agile development methods. DHS offered guidance for preparing acquisition strategies through its Procurement Innovation Lab. Webinars offered by the Procurement Innovation Lab on acquisition strategies for Agile programs discussed the need for interim delivery of software, close coordination between contractors and program office staff, contract oversight mechanisms that were tailored to support Agile development, and changing requirements. For example, the “Transportation Security Administration Agile Services Procurement” webinar discussed planning, executing, and de-briefing technical demonstrations used to select the contract recipient, paying particular attention to the value of transparency and modifying contract oversight mechanisms. Officials from the Office of the Chief Procurement Officer clarified that the webinars were available as needed and were not required training. DHS also published Agile guidance that discussed contracting and acquisition strategies. From an oversight perspective, according to the Agile instruction manual, DHS executive steering committees oversee all aspects of program planning and execution between acquisition decision events. This authority extends to assisting programs in developing acquisition strategies where appropriate. The manual included a section that specifically called out Agile contracting considerations that pointed back to Office of Management and Budget Contracting Guidance to Support Modular Development, the TechFAR handbook, the Digital Services playbook, and innovative contracting case studies. Among other useful information in the Agile instruction manual were key contracting considerations for an Agile program or project manager. These considerations included, among other things, frequent, iterative deliveries of software, an ability to monitor changes to maintain contract and project scope, flexibility to accommodate refinement of requirements, transparency and collaboration, and prior experience in the Agile methodology. The manual also highlighted goals for the acquisition to discuss with a contracting officer, such as rapid contracting processes to keep pace with Agile development, contracting to accommodate incompletely defined scope and requirements, and the ability to respond to requirements changes without requiring extensive change orders. According to officials within OCTO and the Office of the Chief Procurement Officer, the department also supported Agile programs in preparing acquisition strategies through the IT acquisition review process. This process was established to provide a mechanism for the DHS Chief Information Officer to review and guide agency IT expenditures. The process was intended to analyze IT acquisitions to ensure alignment with DHS missions, goals, policies, and guidelines. This process relied on subject matter experts to assist in the review of IT acquisitions, including one for Agile reviews. According to the IT Acquisition Review Essentials Guide, Agile reviews occurred where software was being developed to ensure development activities adhered to Agile best practices and DHS SELC guidance. The Agile subject matter expert was expected to review acquisition materials against an established set of criteria for both the acquisition plan and the requirements document. For example, when reviewing the acquisition plan for approval, the subject matter expert should consider if the statement of need adequately addresses Agile or iterative project-specific activities and/or deliverables. The Director of STM stated that there is one staff member in STM who actively participates in the IT acquisition review process and was responsible for ensuring Agile language was correctly implemented in contract statements of work. The Director also added that they were willing to help teams that were having trouble providing explanations of Agile processes in their statements of work. The Director of STM stated that there was no policy to guide his staff member reviewing Agile language in the statement of work, but that he asked his division to put together a checklist review to govern this process. The Director added that the department sent programs and projects requiring assistance with Agile contracting to the Procurement Innovation Lab by request to streamline the acquisition plan. According to the Leader of the Procurement Innovation Lab Team, the Office of the Chief Procurement Officer was primarily focused on supporting Agile pilot programs, such as the Federal Emergency Management Agency Grants Management Modernization program. The team leader noted that, while the procurement office supported these programs, it relied on the program offices to ensure accuracy. For example, the program management office ensures that the requirements are structured and delivered, which could be challenging for Agile programs. The team leader mentioned that a particular focus at the moment was defining the pricing for contract line item numbers in such a way as to afford the flexibility needed for Agile development while still holding contractors accountable. This appendix describes in greater detail our evaluation of the three leading practices for program processes when adopting Agile development. It does not present new findings; rather, the information is intended to assist the Department of Homeland Security (DHS) in implementing the recommendations described in this report. Program processes refer to leading practices related to the program office and technical environment. For programs to successfully transition from processes used for traditional development projects, programs should ensure that staff are appropriately trained in Agile methods by ensuring Agile teams have the appropriate technical expertise needed to perform their roles technical environments enable Agile development through making technical and project support tools available, and designing a system that supports iterative delivery project planning controls are compatible with Agile methods by maintaining a sustainable development pace and tracking and defining and incorporating non-functional requirements in defining and incorporating critical features in development The department develops an environment that supports these processes. Within DHS, program management offices are responsible for planning and executing individual programs and implementing applicable Agile methodologies. In addition, the DHS Office of the Chief Information Officer (OCIO) is responsible for setting policies and procedures to ensure that programs leverage Agile development best practices to meet the department’s goals and are within acquisition policy. The DHS OCIO is also responsible for providing guidance for and reviewing the adoption and execution of Agile development. Train all program staff in Agile methods The agency should provide a training program in Agile for staff and track and monitor the training. All members supporting the team, not only the software development team, should be trained in the specific Agile framework they will be using. DHS required its acquisitions workforce to take training that incorporated Agile methods. DHS Instruction 102-01-006, Acquisition Program Management Staffing, established certifications for key acquisition career fields, which included training requirements. According to the Associate Director for Training from the Homeland Security Acquisitions Institute, the certification requirements included training that has been updated to incorporate Agile methods. Specifically, the department updated course content for AQN 101: DHS Fundamentals of Systems Acquisition to include Agile development concepts, such as small team management and Agile metrics, following the issuance of department policy governing Agile development. This course was required training for seven of the acquisition career fields, including program and project managers, systems engineers, and test and evaluation managers. DHS tracked and monitored the completion of training requirements for the acquisitions workforce. According to DHS Directive 064-04, Acquisition Professional Career Information, component acquisition executives were responsible for ensuring that acquisition personnel met the mandatory training requirements. Officials from the Homeland Security Acquisitions Institute within the Office of the Chief Procurement Officer stated that DHS employees leveraged the Federal Acquisition Institute’s training application system to track their training and certifications. According to the catalog of product services of the institute, members of the DHS acquisition workforce were required to attach copies of their training certificates to request certification of completion of the required training. Because the DHS acquisitions workforce may not cover all personnel staffed to Agile projects, some program staff may not be subject to training requirements that incorporate Agile methods. According to the Director of the Homeland Security Acquisition Institute, certain Agile team members, such as the product owner, were not necessarily classified as part of the acquisitions workforce. For example, according to the U.S. Immigration and Customs Enforcement (ICE) Student and Exchange Visitor Information System (SEVIS) program staffing plan, the product owner role was not part of the acquisitions workforce and did not require any certifications. To help address the Agile training needs of all staff, including those who are not part of the acquisitions workforce, DHS also provided elective training in Agile methods. The department offered commercial training through the Homeland Security Acquisition Institute, such as acquisition of Agile services and Agile requirements for creating user stories. The DHS Agile instruction manual also identified training offered by the U.S. Citizenship and Immigration Services Office of Information and Technology as another resource on Agile concepts, such as user stories and automated testing. In addition to elective training, the Agile Development and Delivery for Information Technology Instruction Manual (Agile instruction manual) encouraged program managers to seek out an Agile coach to help teams adopt Agile methods and supplement training. The instruction manual suggested that program managers should identify an Agile coach to serve as an embedded trainer, consultant, and team advisor. This Agile coach could help the team adapt Agile methods to their environment and work through challenges. An Agile coach could also help individual team members understand the responsibilities of their role on an Agile team. Although DHS did not provide coaches for Agile teams, the department offered resources that could help programs select and obtain an Agile coach. First, the department established a blanket purchase agreement for programs to acquire Agile development support in the form of hands- on coaching services for the design and use of Agile methods. According to Homeland Security Acquisition Institute officials, this agreement would enable programs and projects to acquire Agile coaching. Among other things, this agreement defined the scope of Agile coaching services and their pricing so that programs would not need to develop these terms on their own. Second, the Agile instruction manual included considerations to help program managers select a qualified Agile coach. For example, the instruction manual encouraged program managers to collaborate with contracting officials to identify an Agile coach who had demonstrated successful past performance on projects implementing similar technology and Agile methodologies. The U.S. Customs and Border Protection’s (CBP) Biometric Entry Exit (BEE) program’s Air Exit project provided informal training for new team members that included a discussion of Agile methods. According to the Air Exit project manager in the Office of Information and Technology, new team members received onboarding training that covered CBP’s approach to Agile methods. The project did not track attendance for this onboarding training, but an Air Exit project manager noted that team members were incentivized to attend the training in order to learn how to satisfy their responsibilities. The Scrum master for the Air Exit project stated that this training was also available to the team as a refresher course approximately every fiscal quarter. The BEE program also relied on an Agile coach to support the Agile team. According to the Agile coach supporting the Air Exit project, this role included training for the Agile team on basic Agile topics and working with the team on their use of a project management software tool. According to a project manager within the Office of Field Operations Air Exit project management office, the Agile coach that supported the project was instrumental in designing the CBP Office of Information and Technology’s Agile development program beyond the BEE program. Ensure Agile teams have the appropriate technical expertise needed to perform their roles The agency should have policy or guidance in place to help programs ensure Agile teams have the appropriate technical expertise. A program should also consider Agile-centric skills when forming teams. In addition, programs should define requirements for contractor proposals and evaluate contractor proposals for Agile services (e.g., source selection). DHS guidance provided programs with considerations for forming teams with Agile-centric skills. The DHS Agile instruction manual stated that a development team with experience in Agile practices can mitigate risks to on-time delivery. This experience included Agile processes as well as technical skills, such as automated testing. In the context of the Agile Scrum methodology in particular, the Agile instruction manual stated that teams needed to be cross-functional and have all of the skills required to deliver a project from conception to delivery. To enable teams to deliver a project from conception to delivery, the Agile instruction manual stated that program managers should seek team members with general skills. The manual advised that team members should contribute to routine development activities and possess cross- functional expertise that allows the team to achieve work without depending on individuals outside of the team. For example, in Agile development, testers are part of the development team and should therefore possess both testing and development skills. In addition, the instruction manual stated that, according to industry experts, program managers should seek some overlap in team member’s skillsets to mitigate risks associated with a key person becoming temporarily unavailable. DHS guidance further provided programs with considerations for defining requirements in solicitations for contract proposals for Agile services. For example, DHS supplemental guidance for incorporating testing and evaluation into contract requirements noted that contracts should specify government test and evaluation staff, as well as contractors, on the development team in order to access the test data they need. The DHS IT acquisition review process also helped to ensure that requirements were defined in solicitations for contractor proposals. According to the Information Technology Acquisition Review Essentials Guide, Agile subject matter experts in the department review proposed contracts to ensure that they will enable development activities that adhere to Agile best practices and DHS systems engineering life cycle (SELC) guidance. For example, Agile subject matter experts should assess whether contract requirements documents, such as the statement of work, are prepared in terms that will enable vendors to clearly understand the Agile requirements. The department also provided guidance to assist programs in evaluating contractor proposals for Agile services. The Agile instruction manual noted that programs can consider certifications in various Agile methodologies and recommended that programs coordinate with contracting officials to review vendors’ past performance in implementing Agile methods. In addition, the department established the Procurement Innovation Lab within the Office of the Chief Procurement Officer to help programs address challenges in procuring Agile services, such as validating contractor qualifications. According to a Procurement Innovation Lab team leader, the lab shares lessons learned from Agile services contracts via webinars, which are available to staff on an as-needed basis. Several of these webinars highlighted the value of using technical demonstrations to validate the qualifications of vendors. The ICE SEVIS program provided training for all team members, including contractors, to ensure they had the necessary Agile-centric skills and expertise. A team process agreement for one development module showed that the technical lead, development team, test engineer, and Scrum master roles were filled by contractors, while other positions such as the project manager, product owner, and test automation subject matter expert roles were filled by government employees. According to the ICE SEVIS program manager and Scrum master, the program provided training for contractors that covered Agile processes as well as technical and project management support tools. In addition, some government employees took role-specific training. For example, the program’s test automation subject matter expert completed training in continuous integration and test automation. To further ensure contractors on ICE SEVIS Agile teams had the necessary Agile-centric skills, the ICE SEVIS program defined the Agile methodology and necessary technical expertise for contractors in the contract requirements. For example, the performance work statement for one development module required contractors to use the program’s management software tool to track user stories. The performance work statement also required use of the program’s continuous integration and automated testing tools. The terms and conditions for this contract also identified the required experience for key personnel, such as proven experience working in an Agile environment. The ICE SEVIS program also evaluated contractor qualifications to ensure they had the necessary technical expertise. According to the program manager, contractor qualifications were evaluated in two stages; first, by assessing the contractor’s proposal, and second, by conducting a technical challenge to ensure that contractors could demonstrate the technical skills in the proposal. According to the instructions included in the request for contractor proposals, this technical challenge required the contractor to leverage Agile best practices to design, develop, and demonstrate working software that addressed user stories provided by the program. Although the instructions stated that contractors were required to follow Agile methods, the ICE SEVIS program manager stated that the primary goal of the technical challenge was to assess development skills rather than knowledge of Agile. Agency policy or guidance should call for technical and project tools to be available to support Agile development and for system design that will support iterative delivery. Make technical and project support tools available Project management and technical support tools should be integrated into a program’s technical environment, where appropriate. The tools within this technical environment should be readily available to Agile teams. DHS policy and guidance called for Agile projects’ technical environments to support Agile methods. The department published guidance for standing up technical environments specifically for Agile projects. For example, the DHS Agile instruction manual identified the benefits of using program support tools for tracking program progress, reporting on that progress as part of program governance, and automating tests within an Agile technical environment. The manual stated that a program or project manager is responsible for fostering an environment that enables the Agile team to succeed, including obtaining the appropriate tools. To supplement this guidance, DHS offered a suite of tools that Agile programs could access. The suite of tools was referenced in a checklist of activities for program or project managers in the Agile instruction manual. According to an IT specialist from the Technical Architecture and Engineering division within the Office of the Chief Technology Officer (OCTO), the tools available included program management tools as well as technical tools. The specialist stated that OCTO provided programs with access to this suite of tools to build support for and familiarity with the tools, evaluating any requested plug-ins from programs and doing their best to accommodate them. The ICE SEVIS program defined the technical environment to include technical tools for automated testing and continuous integration. The team process agreement for one of the program’s development modules identified technical tools that supported continuous integration and testing within the program’s technical environment. This included Jenkins for continuous integration as well as MUnit and Soap UI for continuous testing. In addition, the ICE SEVIS Modernization Test and Evaluation Master Plan discussed that tools for helping to ensure code quality, such as an automated code analytics tool, should be used to identify test coverage of code and cybersecurity code vulnerabilities. The program also defined management support tools in the process agreement. Specifically, it identified support tools for tracking and knowledge management, such as JIRA and Confluence. The team process agreement stated that JIRA should be the main knowledge management tool and that all changes, discussion, and history should be tracked in each ticket. This process agreement also stated that JIRA should be the team’s tracking tool with Confluence used to provide transparency. Design a system that supports iterative delivery The agency should adopt policy or guidance that allows project designs to develop modular system components and the program should establish a loosely coupled architecture that allows for modular development. DHS guidance allowed project designs to develop modular system components through upfront architecture planning. The DHS Technical Review Guide advises stakeholders to discuss and approve the technical design of the system, including its top-level architecture, as part of the system definition review. This review should take place prior to development work. For Agile programs, DHS suggested that programs may elect to switch the system definition review with a release planning review. The SELC Tailoring Examples for Selected Types of DHS Acquisition Programs specified that this design discussion should take place as a part of release planning. The department referred to this design as an “architectural runway”, a description that should enable the team to conceptualize how the user stories will be implemented. In exiting the release planning review, the Technical Review Guide noted that programs should answer whether or not an architecture exists, if the architecture enables the deployment of the release, if architecture collaboration is explained and understood for this development process, and if the appropriate resources are available. In addition to transitioning to a release planning review, DHS guidance urged Agile programs to move away from traditional artifacts associated with a system definition review. In this shift from traditional artifacts, the department proposed that programs document software design within a system design document on a release-by-release basis. According to the Requirements Engineering Users Guide, in Agile methodologies detailed design occurs at the iteration level and, as such, the design is documented in an iterative fashion in the system design document. The guide further stated that the system design document allows the development team to communicate the design to others including customers, managers, and other developers and that industry best practice was to represent the design through a series of “design views.” Each software design stakeholder could have a distinct perspective on what are the essential aspects of a software design. Together, these views provide a comprehensive description of the design in a concise and usable form that simplifies information access and assimilation. DHS guidance did not discuss the system design document as a delivered artifact until after the sprint review and demo and a release readiness review had been discussed. At the end of each iteration, DHS guidance stated that the system design document should represent the design of the feature, function, and/or system as it existed at that moment. To facilitate communication between Agile teams and to ensure the most up-to-date description of the design is available, guidance called for the system design document to be developed and maintained in an electronic form using any number of programs or web tools that are available. The Requirements Engineering Users Guide noted that the system design document is to be considered complete when each identified design concern is the topic of at least one design view, all design constraints have been applied, and sufficient detail exists to be an authoritative and primary “code-to” artifact. The system design document should also provide traceability to the feature, epic, and operational requirements document “shall” statements. The SELC Tailoring Examples for Selected Types of DHS Acquisition Programs stated that, prior to releasing software to the production environment, a release readiness review should be conducted. As part of this guidance, the department stated that the intent of this release readiness review included ensuring that all elements of the release were complete, including a system design document. DHS guidance also discussed designing a loosely coupled architecture, another important aspect of project design that facilitates modular development. A member of the contractor support staff for the DHS OCIO stated that the Enterprise Architecture Team was expected to consider modularity and loose coupling generally through consideration of technical complexity. According to DHS Enterprise Architecture principles, technical complexity is to be mitigated in part by the implementation of loose coupling. According to the principles, DHS will incorporate loose coupling into architecture and systems design to minimize the risk resulting from changes within one system necessitating changes within an interoperable system. The BEE Air Exit project design document defined the planned design for the system and addressed design and architectural concerns that could affect the system’s operating environment. As part of this design consideration, the project established a loosely coupled architecture. This loosely coupled architecture was illustrated within the project’s system design document. This system design document defined the Traveler Verification Services software as consisting of two distinct components: 1) traveler verification services core and 2) traveler verification services matcher. The functionality and responsibility of these two components were distinguished throughout the document. Moreover, the document detailed how the Traveler Verification Services software would be delivered as a system of applications, combining an integration layer, business layer, data access layer, and data layer. Agency policy or guidance should call for teams to maintain a sustainable development pace and track and monitor that pace and for non-functional requirements and critical features to be defined and incorporated in development. Maintain a sustainable development pace and track and monitor that development pace The agency should have policy or guidance that calls for Agile projects to establish a sustainable development pace. This guidance should be supplemented by tracking and monitoring the pace. The program should establish a sustainable pace for Agile projects and that pace should be tracked and monitored. DHS guidance called for Agile projects to manage the pace of the software development. The Agile instruction manual stated that Agile projects should consider velocity and burndown rates to track the overall project status and update the project plan to reflect this status. In a separate appendix, the Agile instruction manual also identified metrics for project and program managers and executives to consider in order to monitor how a project was progressing, how Agile was optimizing the use of team members and resources, and where the project stood in terms of key Agile measures. In the list of Agile metrics, DHS highlighted burndown rate and velocity, and offered a description and method of calculation for each. In addition to the Agile instruction manual, the department provided training that spoke to development team pace. For example, the curriculum for lesson six of course APM 350 on managing program execution included a section covering Agile development metrics. Among the metrics discussed were those associated with progress, including velocity and burndown charts. Progress metrics were also covered in other course offerings. However, DHS guidance and training materials did not cover the concept of ensuring a sustainable pace. In order to track and monitor the development team’s pace, the department incorporated several related measures into the Agile core metrics. Among others, programs executing Agile were expected to report on the following pace-related metrics after each iteration: story points planned to be completed, number of production deployment per quarter, and average product deployment lead time. These measures could provide programs and the department with an understanding of the development team’s pace and the extent to which it was or was not sustainable. However, the department was not tracking and monitoring development team pace as intended. The Agile instruction required Agile programs to submit Agile core metrics within six months of the instruction’s publication. However, according to the Director of STM, programs were not consistently reporting these core metrics. According to the Director of STM, the department was still working with programs to ensure they consistently reported the core metrics to the Investment Evaluation, Submission, & Tracking system. The SeaWatch project at the United States Coast Guard (USCG) demonstrated that it was monitoring development pace on a monthly basis. SeaWatch officials stated that they used TAIGA as a tool to manage the overall project and to auto-calculate pace. Additionally, SeaWatch officials stated that contractors delivered a monthly progress report, which contained the accomplishments of each team and a snapshot from the latest TAIGA report. For example, one monthly report for SeaWatch included a burndown chart for the SeaWatch project’s development backlog and the monthly output of user stories and associated story points by development effort that could be used to assess development pace over time. SeaWatch officials stated that the teams used velocity to help plan for the next iteration. Officials added that they tracked the collective velocity of all four teams as they were all working together on the same ship build. In the future, officials stated that this tracking of velocity could also be used to track individual team velocities as necessary. The project demonstrated that it was adapting in order to achieve a sustainable pace. According to the April 2018 monthly report, the team completed 55 user stories worth 500 story points. The following month, in the May 2018 monthly report, the number of user stories dropped from 55 user stories to 17, worth 130 story points. According to the June 2018 monthly report, the team completed a development effort of 31 user stories and 278 story points. According to the SeaWatch acquisition manager, development pace fluctuated because not all sprints were of equal difficulty. The acquisition manager added that the number of completed story points per sprint could also be inconsistent due to inaccurate user story estimates, changes in staff availability from sprint to sprint, and other external factors such as weather. Define and incorporate non-functional requirements in development The agency should have policy or guidance in place for incorporating non-functional requirements for Agile projects and the program should account for non-functional requirements, such as security and privacy, in the program strategy and throughout development. DHS guidance addressed the incorporation of non-functional requirements for Agile projects. According to the Technical Review Guide, non-functional requirements could be governed via a system definition review. According to the guide, this review was required at the end of the requirements definition phase to focus on the completeness of the requirements engineering activities, including the gathering, analysis, and documentation of functional and non-functional requirements. This review assessed the traceability of these requirements to the operational requirements document and concept of operations. In the case of Agile programs, DHS suggested replacing the system definition review with a release planning review. In place of traditional artifacts associated with a system definition review, DHS guidance stated that the capabilities and constraints document, backlogs, and the system design document, which are developed iteratively throughout the release, should document the requirements and provide traceability to the operational requirements document. These artifacts served the function and filled in for the functional requirements document and the system requirements document previously required for a system definition review. The Technical Review Guide noted that, as the capabilities and constraints document matures, business and architectural epics should decompose to features or themes, and, ultimately, user stories that reflect the specific tasks that users will perform. The Technical Review Guide cited as exit criteria that a program or project should answer whether the capabilities and constraints document identified the specific features and non-functional requirements to be addressed in the release. DHS requirements engineering guidance expanded on how Agile programs and projects could manage non-functional requirements. The guidance explained that there were various ways that the constraints or non-functional requirements such as security, Section 508 accessibility, privacy, or reliability could be translated down to the iteration level. It stated that some Agile teams may include these non-functional requirements in the backlog, while other teams may include them as part of acceptance criteria or in an artifact called the “definition of done”. According to officials from the Science and Technology Directorate Office of Systems Engineering, once defined, the day-to-day operations and testing for non-functional requirements were the responsibility of the operational test agent. DHS maintained some governance over non-functional requirements. According to the DHS acquisition management instruction, the operational requirements document should be approved by the Acquisition Decision Authority after validation by the Joint Requirements Council. The operational requirements document should include both the functional and non-functional requirements. Officials from the Office of the Director of Test and Evaluation said that they do not usually provide feedback on the decomposed functional or technical requirements for software development projects, focusing only on the operating requirements, because that is what directly impacts operations. The CBP BEE program’s functional requirements document outlined a series of non-functional requirements as the requirements used to define how the system is to behave as opposed to functional requirements that define what the system should do. The project included 10 non-functional requirements in the functional requirements document. For example, the biometric match service should have an overall availability of greater than or equal to 99%, which included both scheduled and unscheduled downtime. These ten non-functional requirements comprised five related to availability, three related to reliability, one related to scalability, and one related to security. All of these non-functional requirements were scheduled for release as part of the initial operating capability. CBP officials noted that non-functional requirements were also captured within the operational requirements document as measures of effectiveness. According to project officials, measures of effectiveness and other security-related parameters translated into the key performance parameters for the project. Officials noted that these key performance parameters were tracked on a daily basis and that information was fed into a monthly report. The operational requirements document stated that the program’s suitability requirements conformed to the DHS and CBP enterprise architectures and all DHS and CBP infrastructure policies and guidelines. Moreover, it noted that National Institute for Standards and Technology guidance and DHS guidance factored into the development of security related non-functional requirements. For example, system security controls should be compliant with National Institute of Standards and Technology and DHS sensitive system guidelines based on its Federal Information Processing Standard 199 rating for availability, integrity, and confidentiality. Define and incorporate critical features in development The agency should have policy or guidance in place for incorporating critical features for Agile projects. The program should ensure that its strategy considers all mission, architectural, and critical safety components, along with their dependencies, on a regular basis. DHS policy and guidance addressed the incorporation of critical features for Agile projects. As discussed in the non-functional requirements section, programs were expected to document functional requirements via the systems design review or, as recommended for Agile programs, a release planning review. Artifacts associated with these reviews served to capture the functional requirements for the program and should be evaluated as part of the entrance and exit criteria defined in the technical review guide. Additional guidance elaborated on the process for decomposing requirements. Unlike non-functional requirements, applicable exit criteria on critical features expanded into the solution engineering review. This criteria included questions devoted to critical features and how they tied back to performance measures (e.g. key performance parameters). According to the Director of STM, headquarters oversight of critical features was limited to the higher-level requirements defined in the operational requirements and concept of operations documents. The ICE SEVIS program captured critical features in documents required by department acquisition management policy and guidance. The ICE SEVIS Modernization Concept of Operations listed specific functional capabilities associated with mission and mission support scenarios. The ICE SEVIS Modernization Operational Requirements document expanded on these functional capabilities and identified the operational and program-level requirements. These requirements were necessary to achieve the performance goals and mission of the Student and Exchange Visitor Program and the Department of State, the primary sponsors for the program. In particular, the SEVIS Modernization Operational Requirements document identified business capabilities and key performance parameters that measured system capabilities. The core capabilities are long-term initiatives intended to span multiple contracts and deliver the major components necessary for SEVIS modernization. The SEVIS Modernization Operational Requirements document stated that these capabilities must be present for the SEVIS modernization to be considered a success. These business capabilities represented the core SEVIS functions needed to close outstanding SEVIS vulnerabilities. According to the ICE SEVIS Modernization SELC Tailoring Plan, there were 79 sub-capabilities supporting the eight core capabilities. The sub-capabilities generally fulfilled one or more stakeholder needs and were delivered within a release or series of releases. The SEVIS Modernization Operational Requirements document confirmed that the program should prioritize and sequence the capabilities for delivery during the release planning and delivery processes. The program provided a road map for one development module. This road map listed areas for development in the order they were intended to be developed and identified the associated business capabilities. The business capabilities identified in the road map aligned with the sub- capabilities listed in the SEVIS Modernization Operational Requirements document. Examples of business capabilities in the road map that were also sub-capabilities identified in the operational requirements document included: create nonimmigrant record (including supporting forms), align nonimmigrant eligibility information with unique nonimmigrant, update nonimmigrant biographical information, and add/update dependent information. This appendix describes in more detail our evaluation of the three leading practices for team activities and dynamics when adopting Agile development. It does not present new findings; rather, the information is intended to assist the Department of Homeland Security (DHS) in implementing the recommendations described in this report. For teams to successfully transition from processes using traditional software development methods to Agile methods, leading practices for team activities and dynamics recommend that the composition of the team supports Agile methods by defining the role of a product owner work is prioritized to maximize value for the customer through creating user stories to define work prioritizing requirements in a backlog based on value estimating the relative complexity of user stories repeatable processes are in place by meeting daily to review progress and discuss impediments ensuring the quality of code being developed Within DHS, program management offices are responsible for planning and executing individual programs and implementing applicable Agile methodologies. According to Office of the Chief Technology Officer (OCTO) officials, DHS contracts for Agile services, including development, rather than performing development in-house. As a result, Agile teams may be predominantly contractors rather than federal employees. In addition, DHS Office of the Chief Information Officer (OCIO) is responsible for setting the policies and procedures to ensure that programs and, in turn, the teams that make up those programs, leverage Agile development best practices to meet the department’s goals and are within acquisition policy. DHS OCIO is also responsible for providing guidance for and reviewing the adoption and execution of Agile development. Agency policy or guidance should require individual, self-organizing Agile teams for each segment or iteration and define the role and responsibilities of the product owner. Agile teams should be self-organizing, meaning they are empowered to collectively control how to accomplish their work and the resulting product. An Agile team’s authority should include lower-level decision making and team formation and highlight the importance of team stability. The team’s composition should be cross-functional and consist of members who possess all the skills needed to produce working software, including, but not limited to, contract specialists, developers, and testers. DHS provided guidance to Agile teams on self-governance. The Agile Development and Delivery for Information Technology instruction (Agile instruction) and the Agile Development and Delivery for Information Technology Instruction Manual (Agile instruction manual) both explain that collaborative, self-organizing, and cross-functional teams help achieve the flexibility needed for the iterative development that characterizes Agile development methods. The Agile instruction manual notes that most Agile methodologies assume the dedicated involvement of all stakeholder, developer, and integration staff throughout the project. DHS guidance also discusses team formation. The Agile instruction manual recommends that the project team include the roles of the program or project manager, a product owner, a development team of approximately five to nine members, testers, and an Agile coach, and any additional expertise as needed. According to DHS guidance, a program or project manager is responsible for establishing the project team. The program or project manager is supported in this by the component acquisition executive and other component management. At DHS, U.S. Immigration and Customs Enforcement’s (ICE) Student and Exchange Visitor Information System (SEVIS) program had self-organizing teams that defined their own processes for completing work. ICE Agile teams, including those supporting the SEVIS program, were expected to document their processes in a team process agreement, where a team had the authority to define its own operational strategy and make decisions about the product, including when to consider the product completed according to the program’s “definition of done.” According to the ICE Agile principles instruction, a program chooses a baseline set of practices that are documented in a team process agreement and are adjusted over time. ICE SEVIS teams were self-managing and included the roles necessary to deliver what they committed to in a sprint. ICE’s Agile playbook suggested minimum levels of experience, knowledge, and certifications necessary for key personnel to support Agile methodologies. For instance, the playbook suggests that Scrum masters be certified and have a minimum of one year of experience. To help ensure that contractors have the requisite skills necessary, ICE SEVIS officials stated that vendors are required to demonstrate their ability to develop a small software application before a contract is awarded to them. Define the role of a product owner A product owner should understand the business and strategic values of the agency and its alignment with the vision of the product team and support Agile methods. A product owner’s responsibilities include availability to the team, authority for making programmatic decisions, general responsibilities as a member of the team, and the need to possess subject matter expertise related to the business needs. A product owner is an authoritative user who manages the requirements prioritization, communicates operational concepts, and provides continual feedback to the team. DHS provided guidance on the role and responsibilities of a product owner. According to the Agile instruction manual, the product owner is responsible for representing stakeholders. To do so, the product owner should be available to the development team throughout the iteration to answer questions and clarify requirements on behalf of the stakeholders. The manual stated that the product owner is also responsible for ensuring that the product meets user needs and delivers value. This includes, for example, prioritizing user stories in the backlog and serving as an acceptance authority for work completed by the team. The department also provided elective training on the role of a product owner. For example, the U.S. Citizenship and Immigration Services Office of Information Technology offered an elective product owner training course. The USCIS product owner training covered concepts such as the importance of the product owner’s availability to the team and the product owner’s authority for making programmatic decisions. ICE identified a product owner for SEVIS to represent two user communities. The program identified one product owner from ICE’s Student and Exchange Visitor Program and a second product owner from a stakeholder organization within the Department of State. Both product owners were identified in the ICE SEVIS staffing plan. According to a team process agreement for one development module, a product owner is responsible for, among other things: Prioritizing and deciding which user stories will be implemented in each iteration. Making an acceptance decision for each user story based on the story’s acceptance criteria. Ensuring that the intended value of the functionality is delivered. According to program officials, product owners for the ICE SEVIS program prioritized user stories during planning sessions. The Student and Exchange Visitor Program Agile Overview slides stated that the team, including the product owner, attends sprint planning to review the prioritized product backlog and to ensure a common understanding of the product owner’s immediate priorities. Product owners also exercised authority to validate acceptance criteria and subsequently close user stories. The program’s “definition of done” stated that the product owner must test and indicate acceptance of each user story in order for a user story to be considered complete. In a written response, ICE SEVIS officials stated that ICE SEVIS product owners indicated a user story had met the acceptance criteria and could be closed by changing the user story’s status to “closed” using the team’s program management software tool. In addition, product owners were available to the development team to ensure timely input. According to a team process agreement for a development module, the product owner should work closely with the development team to communicate the details of requirements and answer questions about user stories. In an interview, the ICE SEVIS product owner representing the Student and Exchange Visitor Program stated that the role was a full-time position and did not have any competing responsibilities. To ensure availability, the ICE SEVIS product owner representing the Student and Exchange Visitor Program stated that there was a designated backup who had the same authority and responsibilities as the full-time product owner. Agency policy or guidance should call for Agile teams to create user stories to define the work; prioritize requirements in a backlog based on value, including tracking and monitoring the value of work accomplished; and estimate the relative complexity of user stories. Individual Agile teams within the respective programs and projects should implement these aspects of Agile development. Create user stories to define work A user story is to reflect a small segment of work that can be completed in a single iteration. The agency should have policy or guidance in place for writing user stories for Agile projects. The product owner should determine the value of a user story in consultation with the development team, including the acceptance criteria and defining what “done” means. User story value should then be re-evaluated based on requirements to ensure the greatest return on investment. DHS provided guidance that Agile programs and projects could leverage when writing a user story. The Agile instruction manual, Homeland Security Acquisition Institute Agile lessons, such as “Managing Program Execution” and the Requirements Engineering User’s Guide provided a basic format for how to craft a user story. These resources noted that a user story defines where a “role” wants some “goal/desire” accomplished to result in a “benefit”. The Requirements Engineering User’s Guide also discusses the role of acceptance criteria and a definition of done in user story development. The guide highlighted that acceptance criteria defines the boundaries of a user story and confirms when a story has been completed and is working as intended. It specifies that acceptance criteria should be included in an Agile program or project’s capabilities and constraints document, a DHS artifact unique to Agile development and highlighted in the systems engineering life cycle (SELC) tailoring example for Agile. This guide added that the definition of done identifies all of the activities/artifacts besides working code that must be completed for a feature or sub-epic to be ready for deployment or release including testing, documentation, training material development, certifications, etc. The Agile instruction manual places much of the responsibility for defining a user story under the purview of the product owner. The Agile instruction manual stated that the product owner is the individual tasked with providing requirements to the development team and is responsible for determining the features necessary for the product release. The manual also emphasized that the product owner is only responsible for clarifying the user story requirements that would meet his or her needs and not responsible for clarifying how user stories should be implemented to meet those needs. The ICE SEVIS program developed user stories based on business capabilities and other requirements as determined by the product owner and the business stakeholders. The SEVIS Modernization Operational Requirements Document describes eight business capabilities that represent core SEVIS functions. According to ICE SEVIS officials, these business capabilities are addressed through user stories, so there is traceability in the backlog from user stories to epics to business capabilities/operating requirements. The team’s process agreement for one development module—Information Sharing—assigned responsibility for writing user stories to the product owner. This agreement also noted that acceptance criteria would be required for most stories. User stories for the program were managed through a program management software tool. An output of the backlog from the program management software tool for one development module—Managing Nonimmigrant Information—contained 525 user stories. These user stories generally followed DHS and ICE guidance for capturing what a user needs and why. Most of these user stories also included acceptance criteria. The program also developed a “definition of done” for all user stories in the team process agreement. According to the definition, a user story was “done” when the following steps were addressed: All code to meet the story’s needs was written according to the system’s development standards. Unit tests were written and run successfully. All code was checked in and the build completed successfully. All database changes (if required) were complete and checked in (a functional test could be run). The software had been deployed to the system test environment and passed system tests. The product owner agreed that the implementation met the acceptance criteria written in the story as appropriate. All documentation required to support the story was completed (test cases, interface updates, etc.) Prioritize requirements in a backlog based on value Agile teams should pull work from a prioritized backlog and provide frequent deliveries of software with immediate value to the user. The team should determine the value of the user stories, prioritize work in a product backlog, and provide an ongoing assessment of value expected versus value delivered. The value of the work accomplished by Agile projects should be tracked and monitored. DHS guidance called for prioritizing user stories in a backlog. The department published an example of a SELC tailoring plan for Agile development that encouraged programs and projects to prioritize user stories in a backlog as part of each release. To ensure that programs or projects took these steps, the Technical Review Guide exit criteria for the release planning review asks if programs or projects will have a process in place for prioritizing user stories prior to the development of features for each release. Planning sessions were one such process that programs and projects could use to prioritize user stories in the backlog. The DHS Agile instruction manual stated that, during sprint planning, the product owner meets with the development team in order to identify user stories from the backlog that should be prioritized for the upcoming sprint and that prioritization decisions should be made based on value to the users. In addition, the product owner should ensure that prioritization decisions maximize mission values. The Requirements Engineering User’s Guide also states that requirements should be prioritized based on continuous stakeholder input so that programs can prioritize what users need the most. DHS guidance also discussed how to determine the value of individual user stories. While the Director of STM said that the product owner is responsible for interpreting the concept of value as it applies to a user story and the relative prioritization of the backlog, Agile Requirements and Road Mapping Guidance for DHS includes a discussion on how a program can sequence its road map for learning, risk, and economic value. In this section, DHS offers models to consider to assist in user story prioritization decisions and considerations for the product owner, such as seeking to balance between business value and cost. The Director added that there were venues, such as Agile “chat and chews,” where program staff could ask questions and receive informal guidance. DHS modified acquisition procedures to allow for an ongoing assessment of progress, and indirectly the value of work accomplished, via the release road map. DHS guidance stated that the release road map is submitted to the Acquisition Review Board prior to acquisition decision event 2B, as required by the Agile instruction. The Technical Review Guide exit criteria for the release planning review and the release readiness review asked if the development team was following the release road map and making adjustments that supported the successful completion requirements defined at acquisition decision event 2B. Thereafter, programs submitted a road map to the Acquisition Review Board during regular program reviews. In addition to tracking and monitoring the value of work accomplished against a release road map, regular Acquisition Review Board program reviews allowed for the assessment of value expected versus value delivered. The presentation template for Acquisition Review Board program reviews included a slide for programs to report their progress toward planned features. For each review, programs identified a percentage of each capability that they planned to complete by the next review. In addition, programs reported on the percentage of each capability that they had completed since the last review. The U.S. Coast Guard (USCG) SeaWatch Agile teams prioritized requirements in a backlog based on the team’s ability to complete them within a sprint. According to the acquisition manager for the Command, Control, Communications, and Computers, Intelligence, Surveillance and Reconnaissance (C4ISR) program, the SeaWatch product owner for new development determined priorities for new requirements with stakeholders. The product owner then defined those requirements as an epic or as a user story. The C4ISR acquisition manager stated that the user stories were prioritized in the backlog during sprint planning primarily based on whether the Agile team could complete the work in the upcoming sprint rather than on the value assigned by the stakeholders. According to SeaWatch officials, user stories that could not be completed during the current sprint were marked as a priority item for the next sprint. Although the SeaWatch program assessed value to the user for some epics, this did affect how the epic or its associated user stories were prioritized in the backlog. The C4ISR acquisition manager stated that SeaWatch assigned a value (e.g. extra large, large, or medium) to an epic based on the epic’s value to the user. However, the acquisition manager noted that user stories were not typically prioritized by the value of the associated epic. User stories were instead prioritized based on the Agile team’s ability to complete the work within the current sprint. The project reported on its accomplishments via a road map. In May 2018, SeaWatch reported on progress toward milestones in its road map during an annual briefing for the Non-Major Acquisition Oversight Council. The program reported that it had installed SeaWatch v3.0 on 65 out of 70 in-service cutters. Estimate the relative complexity of user stories The agency should have policy or guidance in place for relative estimation practices for Agile projects. Teams should use relative estimation for sizing the effort of work required to satisfy a user story by estimating its complexity based on work of similar size and complexity. Relative estimation enables teams to maintain a sustainable software development pace and predict work commitments. The team should size user stories relative to one another, assess the complexity of the work, refine user stories and estimates over time, and use prior estimates to inform future estimates. The product owner and team should continually revisit the estimates as they learn more about the business priorities and as user stories rise in the order of priority. DHS did not provide policy or guidance for relative estimation. Although the Agile instruction manual identified estimating user story size as an integral part to sprint planning, it did not describe the specific techniques or processes for estimating the relative complexity of user stories. Instead, the Agile instruction manual discussed how programs could successfully apply traditional earned value management and cost estimating principles to Agile projects. DHS guidance noted that programs had largely moved from measuring story points to feature points to help programs quantify incremental progress The U.S. Customs and Border Protection’s (CBP) Biometric Entry Exit (BEE) program defined practices and guidelines for how the program expected to estimate user stories. For example, the Traveler Verification Services process definition document identified a formula for calculating story point values on the basis that one story point would equate to approximately four working hours. Moreover, the process definition document noted that story points must be reconciled to better reflect the level of effort and task completion at the end of a given sprint. However, it was not evident that the BEE program had implemented its own guidance on the estimation of story points. Although the process definition document outlined procedures for estimating user stories, only two of 358 user stories in the Air Exit project backlog were estimated using story points. Agency policy or guidance should call for Agile teams to meet daily to review progress and discuss impediments, and to observe end-iteration demonstrations and end-iteration retrospectives. In addition, agency policy or guidance should call for Agile projects to employ continuous integration and confirm mechanisms are in place to ensure the quality of code being developed. This includes setting expectations for automated testing and code quality and tracking and monitoring against these expectations. Responsibility for these aspects of Agile development should lie with the individual Agile teams. Meet daily to review progress and discuss impediments The agency should have policy or guidance in place for holding the daily stand-up and teams should hold daily meetings in order to stay on track to meet the iteration goals for Agile projects and adjust as necessary. DHS guidance defined the general procedure for holding a daily stand-up. The Agile instruction manual stated that teams should conduct a daily stand-up meeting for all team members. It can be conducted in person or via another method of communication (particularly for remote employees) for a brief, informal meeting every work day. According to the manual, all team members should discuss the work each has accomplished since the last daily stand-up, the work to be accomplished by the next daily stand- up, and highlight any impediments that are preventing the team members from completing their work. Additionally, the manual suggests that it is necessary to conduct the daily stand-up with strict discipline, so that the meetings stick to their allotted brief time and are consistently productive. The Agile instruction manual also highlighted the importance of the daily stand-up meeting to an Agile process. It called this meeting an essential collaboration event during which all team members were expected to participate and discuss their work. The manual suggested that holding these meetings allowed the team to practice discipline that would assist them in their work and foster mentoring and partnering relationships within the team that were reinforced through the constant communication of meeting every single day. The manual added that this activity allowed the team to hold its members accountable and be made aware of issues that may be mitigated through collaboration. The Traveler Verification Services team supporting the BEE program Air Exit project at CBP held daily stand-up meetings. According to project officials and supporting project artifacts, a daily stand-up meeting was held each day at 10:00 a.m. Project officials noted that the daily stand- ups included the entire 40-person team. The agency should have policy or guidance in place for holding demonstrations or other interactions for acceptance of user stories in Agile projects. Teams should hold frequent demonstrations to showcase features that have been implemented and obtain feedback for acceptance of user stories in Agile projects. DHS guidance defined the general procedure for holding an end-iteration demonstration or review. In the SELC Tailoring Plan example for Agile development, DHS recommended a sprint review and demo as one type of technical review at the end of each iteration. The purpose of the review was to demonstrate the working software to end users and other stakeholders and to obtain feedback that could result in additional items being added to the backlog. It stated that this review should also ensure that the software design was documented for inclusion in the system design document, a proposed DHS Agile-specific artifact. The tailoring example noted that this review should formally end the iteration’s work with no further development or testing occurring on any stories. The Agile instruction manual added that this demonstration should confirm the value of the incremental piece of software produced. DHS guidance also encouraged the use of demonstrations. The Agile instruction manual states that a demonstration or review could be used to reach a consensus on whether the work associated with a user story met expectations or not. The manual also recommended that program and project managers ensure that the functional software developed during each iteration was demonstrated to the stakeholder at an iteration review meeting. The ICE SEVIS program held end-iteration demonstrations. The ICE SEVIS Modernization Systems Engineering Lifecycle Tailoring Plan stated that sprint demonstrations were tailored into the program to replace other review activities, such as the preliminary design, critical design, and integration readiness review. The Test and Evaluation Master Plan for SEVIS Modernization stated that standard sprint testing results were to be reported at sprint reviews. According to program artifacts, the sprint demonstration was to be conducted at the completion of each sprint, every other Wednesday from 11:00 a.m. to 12:00 p.m. The agency should have policy or guidance in place for holding a retrospective to adapt and continuously improve on Agile projects. Teams should hold a retrospective at the end of each iteration to identify areas for improvement to adapt and continuously improve Agile practices. DHS guidance defined the general procedure for holding a retrospective. The program or project manager and team reviewed progress after each iteration and release. This included the use of a retrospective to discuss what went well, what didn’t go well, and to identify actions to correct problems. Guidance noted that the team should immediately incorporate feedback from the retrospective into future iterations. The DHS Agile instruction manual highlighted the importance of the retrospective. The manual stated that the end-iteration retrospective is a key part of ensuring that teams following Agile methodologies are able to identify problems and adapt to continuously improve for future sprints. Additionally, the manual stated that end-iteration retrospectives are useful in satisfying governance needs. For example, the Agile instruction manual stated that programs could tailor standard-format SELC artifacts (as codified in the SELC Tailoring Plan) to instead rely on assessment and performance data addressed in end iteration retrospectives. The Traveler Verification Services team supporting the BEE program’s Air Exit project held end-iteration retrospectives. According to the process definition for this team, a retrospective was to be held between the end- iteration review and the subsequent planning session for the upcoming sprint. The process definition defined the goal of the retrospective as obtaining an honest review of the process with a consensus on how to adapt it. In an interview, project officials noted that the team documented the results of retrospectives on a release-by-release basis in a project management software tool. The agency should have policy or guidance that defines and emphasizes the use of automated testing and continuous integration. This guidance should be supplemented by defining expectations for automated testing and tracking and monitoring against these expectations. Agile teams should adopt practices for continuous integration and automated testing to ensure that software handoffs are repeatable and dependable. Automated testing should be tracked and monitored based on established expectations. The DHS Agile instruction manual defined continuous integration as the practice where delivery teams frequently integrate their code into a shared master copy. It noted that these integrations are verified by an automated build process, which performs testing to detect any integration errors quickly and automatically. The manual stated that continuous integration in Agile projects should be planned and recorded on a release-by-release basis. The Agile instruction manual also emphasized the importance of continuous integration and automated testing. With regard to automated testing, the manual set an expectation for program or project managers and stakeholders to consider both automated testing tools and infrastructure support for the Agile software build and test processes as part of general project planning efforts. Moreover, the manual identified continuous integration, automated acceptance testing, and automated unit testing as key practices program or project managers can use for continuously monitoring and reporting project health. These practices could also help to identify opportunities for improving project team performance. DHS officials acknowledged that current DHS programs implemented testing and evaluation inconsistently and that the department’s existing guidance and policies did not effectively support modern best practices in automated testing and continuous integration. To address these gaps, DHS had an Agile action plan that set an expectation for updating DHS acquisition guidance, policy, and practices for testing and evaluation to enable modern best practices in automated testing and continuous integration. In lieu of more explicit guidance, DHS incorporated training as part of a curriculum geared toward test and evaluation managers that discussed both continuous integration and automated testing. According to the Deputy Director of Policy and Workforce Development in the Test and Evaluation Division of the Science and Technology Directorate, an alternative course containing content addressing Agile and continuous integration and automated testing was recently merged with a required test and evaluation course, creating a new course. According to the Deputy Director, the new course was piloted during fiscal year 2019 and will be standard in fiscal year 2020 as the required course for level II test and evaluation certification. In order to track and monitor automated testing, the department incorporated several measures into the Agile core metrics. Programs executing Agile were expected to report on the following testing-related metrics after each iteration: Percentage of unit test coverage, Percentage of automated tests, and Percentage of regression testing coverage. DHS had not established expectations for these Agile core metrics. The Agile core metrics included a target. For example, the department suggested a program strive for seventy percent of tests to be automated. However, the instructions accompanying the Agile core metrics stated that all targets were notional and not expected to be reached. According to the Director of STM, the initial core metrics were intended to assess the level of DHS team achievement without imposing artificial industry- based target measures for each. The Director stated that, on receiving the metrics for a period of time, the department would then adjust the core metrics and begin to include target measures based on the results achieved. According to the Director, this effort was underway and an updated set of core metrics would be distributed in early fiscal year 2020. Moreover, the department was not tracking and monitoring automated testing as intended. The CBP BEE program Air Exit project stood up a technical environment that allowed for continuous integration. This technical environment was outlined within the process definition of the Traveler Verification Services team that was developing software. The Traveler Verification Services process definition identified three operating environments: the development, test, and production environments. All development activities during the sprint were conducted within the development environment. Similarly, all testing activities in preparation for the release were conducted in the test environment. The final approved software would then be deployed to the production environment. CBP officials noted that the BEE program primarily used Jenkins to integrate code for both continuous builds and deployment. The Air Exit systems design document also mentioned the role of Jenkins in continuous integration and continuous deployment for the project. The Traveler Verification Services team incorporated JaCoCo and FindBugs automated tests as part of the continuous delivery process and they were run automatically when the code was checked in. Moreover, the project’s system design document noted that the Traveler Verification Services team integrated JaCoCo with the Eclipse Integrated Development Environment as a code coverage inspection tool for unit testing. Officials also noted that Selenium was used for automating the testing within the technical environment. Ensure the quality of the code being developed The agency should have policy or guidance for an Agile project on ensuring the quality of code being developed. This guidance should be supplemented by defining expectations for code quality and tracking and monitoring against these expectations. Agile teams should adopt practices for code quality, such as having a test-driven development, pair programming, and manual code reviews to supplement automated testing. Agile teams should incorporate refactoring into code quality practices and understand the importance of setting aside time for refactoring. DHS guidance recognizes the importance of ensuring code quality as part of the development and testing process. The SELC Guidebook set an expectation that code review and testing should be part of the software development environment. The guide recommended setting up servers where developers could test code and check whether the developed application runs successfully with that code. The guide suggested another level of tests on application reliability to help ensure that the application did not fail on the production server. The guide stated that the program manager should ensure that the team takes corrective action for any hardware and software deficiencies. In order to find deficiencies early, DHS guidance identified coding and testing practices that could help development teams. The Agile instruction manual cited pair programming as one practice where two programmers work simultaneously on a single task: one programmer observes and reviews each line of code as it is written. DHS guidance also identified test-driven development as a practice that could motivate developers to write effective code. The Supplemental Guidance for Test and Evaluation stated that this approach consists of writing test cases that define a desired improvement, then writing the code to meet the desired functionality, ensuring that the test passes, and refactoring the code as necessary. Refactoring, or re-coding, without changing the way the application functions, is an Agile practice that DHS guidance recommends for correcting deficiencies in the code. The Agile instruction manual stated that refactoring aims to improve code readability and reduce the complexity of previously delivered increments of software. It noted that refactoring is important because development teams are focused on adding the desired functionality with each release and may proceed with making improvements to the code. Refactoring was cited as one way to address this accumulation of needed improvements to the code, which are known as technical debt. The Agile instruction manual further emphasizes the importance of setting aside time for refactoring to address risks associated with technical debt. The manual states that refactoring a previously developed increment of software to improve code quality may force a change in the release schedule. However, if the team does not make these revisions in a timely manner, the effort required to correct them later tends to increase. The manual states that this increasing technical debt is a risk factor to be addressed as soon as feasible. If the technical debt is allowed to accumulate unchecked, or if the project team loses track of the scope of its technical debt, the project could suffer from schedule and performance problems. In order to track and monitor the quality of code being developed, the department incorporated several code quality and testing measures into the Agile core metrics. Among others, programs executing Agile were expected to report on the following quality-related metrics after each iteration: Number of critical or major defects fixed. Number of critical or major defects in the backlog. Number of technical debt issues completed. Number of technical debt issues in the backlog. However, the department was not tracking and monitoring code quality as intended. These measures could provide programs and the department with an understanding of the development team’s ability to address defects and technical debt. In addition to these metrics, programs are also expected to report quarterly on the number of outages requiring a rollback or patch after production deployment. The ICE SEVIS program used manual testing to ensure code quality. The definition of done for the program stated that new code should be peer reviewed to identify risk to the existing code, assess compliance with coding best practices, and evaluate refactoring. According to ICE SEVIS officials, an independent specialist provides internal code reviews and offers feedback on areas for improvement. The ICE SEVIS program also employed automated testing to ensure code quality. The definition of done required that unit tests cover a minimum of 85 percent of code. Program officials stated that vulnerabilities and bugs identified through this process were added to the backlog and classified as technical debt. The program refactored code to address technical debt, but did not set aside time for refactoring each sprint. According to ICE SEVIS officials, the development team refactored code as necessary to improve overall quality but did not set aside time for refactoring unless they were addressing a consistent issue. ICE SEVIS officials stated that the development team could propose refactoring code during sprint planning if there was a specific technical debt they had identified. However, according to the Scrum master for the program, addressing technical debt was additional work for the team to take on beyond the user stories they planned to complete and this additional work incentivized the development team to prevent the accumulation of technical debt. Although DHS allowed Agile programs to tailor the core metrics, ICE SEVIS submitted some of the code quality-related Agile metrics to the department. The program included Agile metrics in June 2018 presentation slides for the Acquisition Review Board. For this initial reporting period, the program reported no critical or major defects in the backlog and no technical debt issues in the backlog. It also provided a screenshot of the Agile core metrics reported to DHS via the Investment, Evaluation, Submission, & Tracking system in February 2019. This reporting period covered two iterations. The program reported that it fixed four critical or major defects during the first iteration and did not have any critical or major defects in the backlog for either iteration. The program also reported that it completed eight technical debt issues in the first iteration, out of 14 technical debt issues in the backlog. The program did not report on the number of outages after deployment as part of the Acquisition Review Board program review or as part of the metrics submitted via the Investment Evaluation, Submission, and Tracking system. In addition to the contact named above, the following staff made key contributions to this report: Michael Holland (assistant director), Mathew Bader (analyst in charge), Lamis Alabed, Jennifer Beddor, Christina Bixby, Hannah Brookhart, Chris Businsky, Alan Daigle, Aryn Ehlow, Nancy Glover, Gina Hoover, Anna Irvine, Hoyt Lacy, Jennifer Leotta, Alexis Olson, Zsaroq Powe, Martin Skorczynski, Natalie Smith, and Daniel Spence.", "summary": "Many of DHS's major IT acquisition programs have taken longer than expected to develop or failed to deliver the desired value. In April 2016, to help improve the department's IT acquisition and management, DHS identified Agile software development as the preferred approach for all of its IT programs and projects. GAO was asked to examine DHS's adoption of Agile software development. The objective of this review was to assess the extent to which DHS has addressed selected leading practices for its transition to the use of Agile software development. GAO identified leading practices for planning, implementing, and measuring organizational change that apply to DHS's transition to Agile through its review of guidance published by the Project Management Institute and GAO. GAO also reviewed work it performed to develop leading practices for Agile software development adoption. GAO analyzed DHS documentation, such as policies, guidance, plans, and working group artifacts and assessed them against the selected leading practices. GAO also reviewed the implementation of selected practices within individual IT projects. Finally, GAO interviewed DHS officials to discuss any practices that were not fully implemented. The Department of Homeland Security (DHS) has taken steps to implement selected leading practices in its transition from waterfall, an approach that historically delivered useable software years after program initiation, to Agile software development, which is focused on incremental and rapid delivery of working software in small segments. As shown below, this quick, iterative approach is to deliver results faster and collect user feedback continuously. DHS has fully addressed one of three leading practice areas for organization change management and partially addressed the other two. Collectively, these practices advise an organization to plan for, implement, and measure the impact when undertaking a significant change. The department has fully defined plans for transitioning to Agile development. DHS has partially addressed implementation—the department completed 134 activities but deferred roughly 34 percent of planned activities to a later date. These deferred activities are in progress or have not been started. With respect to the third practice, DHS clarified expected outcomes for the transition, such as reduced risk of large, expensive IT failures. However, these outcomes are not tied to target measures. Without these, DHS will not know if the transition is achieving its desired results. DHS has also addressed four of the nine leading practices for adopting Agile software development. For example, the department has modified its acquisition policies to support Agile development methods. However, it needs to take additional steps to, among other things, ensure all staff are appropriately trained and establish expectations for tracking software code quality. By fully addressing leading practices, DHS can reduce the risk of continued problems in developing and acquiring current, as well as, future IT systems. GAO is making 10 recommendations to DHS to implement selected leading practices for its transition to Agile software development. DHS agreed with GAO's recommendations and described actions taken and planned to address them.", "document_type": "gao"}
{"report": "Civil aviation, including U.S.-bound flights, remains a target of coordinated terrorist activity. In the last 2 years, we issued reports on TSA’s foreign airport and air carrier inspection programs (December 2017), assessments of Cuban aviation security (July 2018), and TSA’s process for reviewing security directives and emergency amendments that apply at last point of departure airports (October 2019). Foreign airport assessments and air carrier inspections. In December 2017, we reported that TSA had taken steps to enhance its foreign airport assessments and air carrier inspections since 2011, including aligning resources based on risk, resolving airport access issues, making evaluations more comprehensive, and creating operational efficiencies. For example, we found that TSA had implemented targeted foreign airport assessments in locations where risk was high and developed a system to strengthen its data analysis capabilities. However, we also found that TSA’s database for tracking the resolution status of security deficiencies did not have comprehensive data on security deficiencies’ root causes and corrective actions. In addition, the database lacked adequate categorization mechanisms such as capturing subcategories that would better explain the root causes of security deficiencies. We recommended, among other things, that TSA fully capture and more specifically categorize data on the root causes of security deficiencies that it identifies and corrective actions. To implement this recommendation, TSA developed a tool to capture airport vulnerability data and provided training to staff in the use of the tool and developed guidance that delineates updated categories for root causes in its data systems. Cuban aviation security. In July 2018, we reported on TSA’s efforts to ensure the security of air carrier operations between the United States and Cuba. We found that TSA’s inspections and assessments in Cuba generally followed standard operating procedures, but TSA did not inspect all air carriers at its own established frequency. We recommended that TSA improve its ability to identify certain air carriers requiring inspection in Cuba and develop and implement a tool that more reliably tracks their operations between the United States and Cuba. In response to our recommendation and as required under the TSA Modernization Act, TSA developed several tools and processes that corroborate and validate flight schedule data. For example, TSA developed a tool to analyze aggregate flight data and validate or identify service to the United States from international locations and began issuing monthly reports on unscheduled operations to its inspectors responsible for Cuba. By taking these steps, TSA is better able to identify operations requiring inspection and corroborate and validate flight schedule data. Security directives and emergency amendments. When threat information or vulnerabilities at foreign airports indicate an immediate need for air carriers to implement additional security measures, TSA may issue new or revise existing security directives (for domestic air carriers) and emergency amendments (for foreign air carriers). The TSA Modernization Act includes a provision for us to review the effectiveness of the TSA process to update, consolidate, or revoke security directives, emergency amendments, and other policies related to international aviation security at last point of departure airports. As of March 2019, there were 46 security directives and emergency amendments (i.e., directives) in effect related to air carrier operations at foreign airports. Earlier this month, we reported that TSA reviews directives, but its process does not fully define how to coordinate with industry representatives and TSA has not determined if it is appropriate to incorporate the security measures of many longstanding directives into air carrier security programs in accordance with TSA policy. Representatives from four domestic air carriers stated that coordination with TSA on directives has improved. However, representatives from six air carriers and two associations indicated that TSA has issued revised directives that are vague or difficult to implement because TSA did not sufficiently involve them in the review process. This contributed to TSA officials offering different interpretations of aircraft cabin search requirements. Further, TSA policy states that directives are not intended to be permanent and are expected to eventually be canceled or incorporated into security programs. Our analysis found that TSA issued more than one half (25) of the directives prior to 2014, meaning they have been in effect for more than 5 years. Several have been in effect for more than 10 years. We recommended, among other things, that TSA better define how to coordinate with air carriers when reviewing directives and when to cancel or incorporate longstanding security directives and emergency amendments into security programs. TSA agreed with our recommendations and plans to develop a process for more formal and consistent coordination with air carrier and industry association stakeholders and consideration of directives for cancellation or incorporation into security programs. Public area security. In November 2013, an armed individual entered the Los Angeles International Airport, firing multiple shots killing a transportation security officer and injuring two others and a passenger. As a result of this and subsequent airport attacks, TSA co-hosted a series of security summits with stakeholders and published the Public Area Security National Framework in May 2017 outlining a series of best practices and recommendations to secure airport pubic areas. The TSA Modernization Act requires TSA and the DHS Cybersecurity and Infrastructure Security Agency to establish a public area security working group to promote collaboration between TSA and public and private stakeholders to develop non-binding recommendations for enhancing security in public areas of transportation facilities. The Act also requires TSA to periodically share best practices developed by TSA and transportation stakeholders related to protecting public spaces of transportation infrastructure from emerging threats. In March 2019, TSA officials established the public area security working group to engage with stakeholders to validate and update the best practices that were developed in the 2017 Public Area Security National Framework. The working group consisted of security stakeholders from both aviation and surface transportation modes. In October 2019, TSA officials told us that they plan to issue an updated list of best practices in the fall of 2019. Insider threats. Recent incidents involving aviation workers misusing their access privileges have heightened concerns regarding the risk of insider threats at airports. TSA estimated in 2018 that there were approximately 1.8 million people with unescorted access to secured areas of the nation’s airports. We have ongoing work examining the actions TSA, airport operators, and air carriers have taken to mitigate concerns regarding insider threats at airports and the extent to which TSA’s Insider Threat Program is guided by a strategic plan. Additionally, the TSA Modernization Act requires TSA, in consultation with the Aviation Security Advisory Committee to conduct a study examining the cost and feasibility to airports, airlines, and TSA of implementing enhanced employee inspection measures at all access points between non-secured areas and secured areas of certain airports. We will review this study once submitted by TSA. Screening rule changes. In 2010, TSA began identifying passengers for enhanced screening who are not known or suspected terrorists, but who fall within the scope of screening rules. Specifically, TSA identifies passengers for enhanced screening through the application of screening rules, which TSA develops by considering current intelligence and other factors. TSA refers to these rules and lists as Silent Partner and Quiet Skies. Silent Partner rules identify passengers for enhanced screening on inbound flights to the United States. Quiet Skies rules—a subset of the Silent Partner rules—identify passengers for enhanced screening on subsequent domestic and outbound flights. The TSA Modernization Act includes a provision for GAO to review the oversight mechanisms and effectiveness of Silent Partner and Quiet Skies. We found that TSA coordinates reviews of Silent Partner and Quiet Skies through quarterly meetings and notifies an expanded set of DHS and TSA stakeholders—including DHS Traveler Redress Inquiry Program and the Federal Air Marshal Service—of rule changes as required under the Act. We also found that TSA has not identified a means to comprehensively measure rule effectiveness. TSA officials explained that they had not yet fully assessed the rules’ effectiveness because it was difficult to measure. TSA has access to data—such as the outcomes of enhanced screening of Silent Partner and Quiet Skies passengers at airport checkpoints—that could be explored to better assess rule effectiveness. Exploring additional data sources could help TSA refine and supplement the agency’s existing efforts to measure program effectiveness. In our draft report, we recommended that TSA explore additional data sources for measuring the effectiveness of Silent Partner and Quiet Skies rules. TSA is currently reviewing the draft report and is scheduled to provide any comments by early November 2019. To protect the U.S. aviation sector, including the roughly 440 airports it regulates, TSA deploys technologies to screen passengers and their carry-on and checked baggage for homemade explosives and other prohibited items that could, among other things, cause catastrophic damage to an aircraft. The ongoing threat of terrorism requires TSA to continually assess the effectiveness of its screening operations and, when necessary, develop and deploy new screening technologies. The TSA Modernization Act includes a provision for us to review whether TSA allocates resources appropriately based on risk at TSA-regulated airports, among other things. Our review of TSA acquisition documents found that TSA considers risk at the beginning of the screening technologies acquisition process. However, TSA officials could not provide an example of when risk information for specific airports had directly influenced decisions about where and in what order to deploy screening technologies to airports in the recent past. Fully disclosing what risk factors are weighed and how decisions are made could better ensure that TSA’s deployment of screening technologies matches potential risks. We recommended that TSA officials document their assessments of risk and the rationale behind decisions to deploy screening technologies. We also found that TSA does not ensure that screening technologies continue to meet detection requirements after they have been deployed to airports, when performance can degrade over time. According to officials, the agency uses certification—a step in the test and evaluation process— to confirm that technologies meet detection requirements before they are deployed to airports, and calibration of the technologies to confirm that technologies are at least minimally operational while in use at airports. They stated that these processes are sufficient to assure TSA that screening technologies are operating as intended. While these processes serve important purposes, they do not ensure that screening technologies continue to meet detection requirements after they have been deployed because performance can degrade over time. Developing and implementing a process to ensure technologies continue to meet detection requirements after deployment would help ensure that TSA screening procedures are effective and enable TSA to take corrective action if needed. In our draft report, we recommended that TSA develop and implement a process to ensure technologies continue to meet detection requirements after deployment. TSA is currently reviewing the draft report and is scheduled to provide any comments by early November 2019. The TSA Modernization Act includes a provision that we review resources provided to TSA surface transportation programs and the coordination between relevant entities related to surface transportation security. According to our analysis, TSA Surface Programs received $123 million in fiscal year 2017 and $129 million in fiscal year 2018. The surface program appropriation represented about 1.6 percent of TSA’s total appropriation in both fiscal years, according to DHS data. We also found that in fiscal years 2017 through 2019, TSA reported using surface program resources for non-surface activities. For example, in fiscal year 2018, TSA reprogrammed $5 million from the Surface Programs account to Mission Support activities to address security requirements and increase hiring of transportation security officers. Further, we found that TSA could improve internal coordination roles and responsibilities for planning and implementing its voluntary Intermodal Security Training and Exercise Program (I-STEP)—a program intended to engage with system operators and governmental security partners to enhance surface transportation security. For example, officials from TSA’s office that provides intelligence briefings during program exercises stated that they do not typically participate in planning meetings because they are not consistently invited to attend. In our draft report, we recommended that TSA clarify roles and responsibilities for all offices involved in the coordination of surface transportation exercises, including when these offices are to coordinate. TSA is currently reviewing the draft of this report and is scheduled to provide any comments by early November 2019. More than 2.7 million miles of pipelines transport and distribute the natural gas, oil, and other hazardous liquids that the people and businesses within the United States depend on to operate vehicles and machinery, heat homes, generate electricity, and manufacture products. Responsibility for safeguarding these pipelines is shared by TSA; the Pipeline and Hazardous Materials Safety Administration (PHMSA), within the Department of Transportation (DOT); and pipeline operators. TSA oversees the security of all transportation modes, including pipelines. PHMSA oversees pipeline safety. DHS and DOT signed a memorandum of understanding (MOU) on their roles across all transportation modes in 2004, and an Annex to the MOU in 2006 to further delineate their pipeline security-related responsibilities. The TSA Modernization Act includes a provision for GAO to review DHS and DOT roles and responsibilities for pipeline security. We reported in June 2019 that key pipeline security documents need to better reflect the current operating environment. For example, the MOU Annex has not been reviewed to consider pipeline security developments since 2006. As a result, the MOU Annex may not fully reflect the agencies’ pipeline security and safety-related activities. We reported that by developing and implementing timeframes for reviewing the MOU and updating it, as appropriate, TSA and PHMSA could better ensure any future changes to their respective roles and responsibilities are clearly delineated and updated on a regular basis. In addition, TSA’s Pipeline Security and Incident Recovery Protocol Plan, issued in March 2010, defines the roles and responsibilities of federal agencies and the private sector, among others, related to pipeline security incidents. For example, in response to a pipeline incident, TSA coordinates information sharing between federal and pipeline stakeholders and PHMSA coordinates federal activities with an affected pipeline operator to restore service. However, TSA has not revised the plan to reflect changes in at least three key areas: pipeline security threats (e.g., cybersecurity threats), incident management policies, and DHS’s terrorism alert system. By periodically reviewing and, as appropriate, updating its plan, TSA could better ensure it addresses changes in pipeline security threats and federal law and policy related to cybersecurity, incident management and DHS’s terrorism alert system, among other things. We made five recommendations to address these issues, including for TSA and DOT to develop and implement a timeline for reviewing and updating the 2006 MOU Annex and for TSA to periodically review and update its 2010 pipeline incident recovery plan, as appropriate. TSA and PHMSA have actions under way to address our recommendations. For example, PHMSA officials stated that PHMSA and TSA continue to collaborate on updates to the 2006 MOU Annex. TSA has also developed and provided pipeline operators with voluntary security guidelines, and evaluates the vulnerability of pipeline systems through security assessments. However, in December 2018 we identified some weaknesses and made recommendations to strengthen TSA’s management of key aspects of its pipeline security program. For example, we reported that the number of TSA security reviews of pipeline systems has varied considerably over time. TSA officials stated that staffing limitations— ranging from 1 full-time equivalent in 2014 to 6 from fiscal years 2015 through 2018—within its Pipeline Security Branch have prevented TSA from conducting more reviews. Further, TSA does not have a strategic workforce plan to help ensure it identifies the skills and competencies—such as the required level of cybersecurity expertise— necessary to carry out its pipeline security responsibilities. We recommended that TSA develop a strategic workforce plan. As of October 2019, TSA has not yet fully addressed this recommendation. We will continue to monitor progress. Chairman Correa, Ranking Member Lesko, and Members of the Subcommittee, this concludes my prepared statement. I would be happy to respond to any questions you may have at this time. If you or your staff members have any questions about this testimony, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals making key contributions to this work include Kevin Heinz, Assistant Director; Paul Hobart, Analyst-in-Charge; Josh Diosomito; Amber Edwards; Michele Fejfar; Melissa Greenaway; Barbara Guffy; Winchee Lin; Tom Lombardi; Michelle Serfass; and Adam Vogt. Key contributors to the previous work discussed in this statement are listed in each of the cited reports. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Threats to the nation's transportation systems persist and continue to evolve. Within DHS, TSA is the federal agency with primary responsibility for the prevention of and defense against terrorist and other threats to the United States' civil aviation, and rail, public transit, pipeline, and other surface transportation systems. The TSA Modernization Act includes provisions intended to enhance security across this broad range of systems and further calls on GAO to review TSA's progress in these areas. This statement summarizes past and ongoing work related to TSA's actions to address selected aviation and surface transportation security areas covered by the TSA Modernization Act. This statement is based on products GAO issued from December 2017 through October 2019 and draft reports currently with TSA for comment. To perform this work GAO reviewed TSA program documents, visited domestic and foreign airports, and interviewed TSA officials, DHS officials, and transportation industry stakeholders, including associations and air carriers. The Department of Homeland Security's (DHS) Transportation Security Administration (TSA) has made initial progress in certain security areas mandated by the TSA Modernization Act, but additional actions are needed. International aviation security. In December 2017, GAO reported that TSA has taken steps to enhance its foreign airport assessments. Since that time, TSA has developed a tool to better track and address foreign airport vulnerabilites. In addition, TSA reviews security directives and emergency amendments it issues to address security concerns. However, TSA's review process does not fully define how to coordinate with industry representatives and it has not determined if it is appropriate to incorporate the security measures of many longstanding directives into air carrier security programs in accordance with TSA policy. In October 2019, GAO recommended, and TSA officals agreed, that TSA better define how to coordinate with air carriers when reviewing directives and when to incorporate directives into security programs. Passenger screening rules. TSA develops screening rules by considering current intelligence and other factors to identify passengers who fall within the scope of the rules for enhanced screening. GAO found that TSA coordinates rules reviews through quarterly meetings and notifies an expanded set of DHS and TSA stakeholders of rule changes as called for by the Act. TSA tracks some data on rule implementation but does not comprehensively measure rule effectiveness. In its draft report, GAO recommended that TSA explore additional data sources for measuring the effectiveness of its rules. TSA is currently reviewing this recommendation. Aviation screening technologies. GAO found that TSA does not ensure that screening technologies continue to meet detection requirements after they have been deployed to airports. According to officials, the agency uses certification—a step in the test and evaluation process—to confirm that technologies meet detection requirements before they are deployed to airports, and calibration of the technologies to confirm that technologies are at least minimally operational while in use at airports. While these processes serve important purposes, performance can degrade over time. In its draft report, GAO recommended that TSA implement a process to ensure technologies continue to meet detection requirements after deployment. TSA is currently reviewing this recommendation. Surface transportation pipeline security . In December 2018, GAO identified some weaknesses and made recommendations to strengthen TSA's management of key aspects of its pipeline security program. For example, TSA does not have a strategic workforce plan to help ensure it identifies the skills and competencies—such as the required level of cybersecurity expertise—necessary to carry out its pipeline security responsibilities. GAO recommended, and TSA concurred, that TSA develop a strategic workforce plan. As of October 2019, TSA has not yet fully addressed this recommendation. We will continue to monitor progress. GAO has made recommendations designed to address the challenges discussed in this statement. TSA concurred with recommendations from prior work and is currently reviewing recommendations from our draft reports, including those regarding passenger screening rules and aviation screening technologies.", "document_type": "gao"}
{"report": "The FHLBank System comprises 11 federally chartered banks. The FHLBanks represent 11 districts and are headquartered in Atlanta, Boston, Chicago, Cincinnati, Dallas, Des Moines, Indianapolis, New York City, Pittsburgh, San Francisco, and Topeka (see fig. 1). Each FHLBank is cooperatively owned by its members––such as commercial and community banks, thrifts, credit unions, and insurance companies. As previously noted, the FHLBank System also includes the Office of Finance, which is the fiscal agent for the 11 banks. As of December 31, 2018, the total amount of assets each FHLBank held varied widely, as did the number of member institutions in each district (see table 1). FHLBanks primarily obtain funding to provide loans to their member institutions by issuing debt. The Office of Finance (also regulated by FHFA) issues the debt on behalf of the FHLBanks. FHLBanks’ debt products include discount notes (short-term debts) and bonds (short- to long-term debts). The debt transactions can vary in size and be conducted by one or more broker-dealers. Additionally, FHLBanks individually invest in permissible securities, including mortgage-backed securities, which generate additional income for the banks. Broker- dealers are compensated in fees for certain transactions they conduct. Generally, the fees a broker-dealer can earn for capital markets transactions depend on the type of the transactions or the broker-dealer’s role in transactions. The Office of Finance identifies and approves broker-dealers for the banks’ debt issuance transactions, including diverse broker-dealers that are minority-, women-, disabled-, and veteran-owned. As part of its approval process, the Office of Finance assesses the broker-dealers based on their track record in conducting certain debt transactions, and reviews documents including broker-dealers’ audited financial statements, documentation on capital sustainability, legal or regulatory issues, diversity certification, procedures, and any other issues that may affect their eligibility or performance. For investment transactions, the banks approve broker-dealers for their own investment needs according to their own qualification requirements. Similar to the Office of Finance’s requirements, the banks’ qualification requirements can include financial performance and capital requirements. To implement requirements in HERA, in December 2010, FHFA issued the Minority and Women Inclusion rule to set forth minimum requirements for FHLBank diversity programs and reporting, as previously noted. Among other things, the 2010 rule required each bank to create its own OMWI or designate an office to perform duties related to the bank’s diversity efforts, and establish policies related to diversity and inclusion, including workforce and business activities (which can include suppliers and broker-dealers used for capital markets activities). The 2010 rule also requires FHLBanks to submit an annual report to FHFA that describes the gender and racial/ethnic composition of the bank’s workforce and of the suppliers and broker-dealers used in business activities and past and future diversity and inclusion efforts in these areas. The 2010 rule also requires the banks to report on businesses owned by individuals with disabilities that enter into contracts with the FHLBank and the number of individuals with a disability or disabilities for certain workforce data, including the number of individuals who separated from the bank and the number of employees promoted. In 2017, FHFA added the requirement for each FHLBank to develop a standalone strategic plan on diversity and inclusion or incorporate a diversity and inclusion plan into their general strategic plan. FHFA conducts annual examinations and off-site monitoring of FHLBanks. FHFA’s examination includes reviewing the banks’ diversity and inclusion efforts, financial reporting, and corporate governance by bank board directors. We previously reported on diversity in the financial services sector, including FHLBank board governance and board diversity. In 2017, we reported that representation of women and minorities at the management level in the financial services sector showed marginal or no increase during 2007–2015. In a 2015 report on FHLBank board governance, we found that FHFA and FHLBanks had taken steps to increase board diversity, including creating regulations that encouraged the banks to consider diversity in board candidate selection and developing processes to identify and nominate independent directors. In a 2019 report on FHLBank board diversity, we found that since 2015, FHLBanks increased the share of women and minority directors on FHLBank boards, but the banks continued to face challenges in increasing diversity among directors elected from member institutions. We recommended that FHFA, in consultation with FHLBanks, review the banks’ data collection processes for board demographic information and communicate effective practices to banks. FHFA agreed with our recommendation. The agency stated that it planned to engage with FHLBank leadership in 2019 to discuss board data collection issues and explore the feasibility and practicability for FHLBanks to adopt processes that could lead to more complete data on board director demographics. Across the 11 FHLBanks, the share of women in senior-management positions increased from 2011 to 2017, while the share of minorities remained about the same. In the FHLBank workforce overall, the share of female and minority employees was similar in 2011 and 2017. Individual FHLBanks reported a number of challenges in recruiting and retaining a diverse workforce, including limited hiring opportunities due to low employee turnover and a small workforce and competition for diverse talent from larger and better-known companies. Despite these challenges, banks have been taking steps to help maintain or increase a diverse workforce. Female representation. The share of women in senior management across all 11 FHLBanks increased by about 7 percentage points from 2011 to 2017 based on the most recently available EEOC data. As shown in figure 2, the percentage of women across 11 banks was about 21 percent in 2011 (35 individuals) and 28 percent in 2017 (47 individuals). While female representation in senior management collectively increased for the 11 FHLBanks from 2011 to 2017, there was substantial variation among the individual banks. We discuss representation at individual banks in more detail later in this section. Six banks increased the share of women in senior management during this time period (ranging from about 10 to 20 percentage points); three banks decreased (from about 6 to 13 percentage points); and the share for two banks did not change. One FHLBank decreased its number of senior-management positions between 2011 and 2017 by reclassifying those positions, while another bank increased the number of senior-management positions through reclassification, according to staff from each bank respectively. Bank staff noted some banks have fewer senior-management positions because they interpret EEOC’s definition of senior management more narrowly, while others have more senior-management positions because they interpret the definition more broadly. These differences could have affected the comparability of the share of women and minorities in senior management among individual banks in the period we reviewed. Also, because of the relatively small number of senior managers at the FHLBanks, a small change in the number of such managers can result in a larger change in the associated percentage. Minority representation. The share of minority senior management across all 11 FHLBanks was approximately 14 percent (23 individuals) in both 2011 and 2017. Five banks increased the share of minority senior management (from about 1 to 23 percentage points); three banks decreased (from about 6 to 13 percentage points); and three banks did not change. Four banks did not have any minorities in senior management in 2017 (see fig. 3). The largest racial/ethnic group among senior management in 2017 was Asian (about 5 percent) followed by African-American and Hispanic (both at about 4 percent). See figure 4. Female and minority representation combined. The combined share of female and minority senior management across 11 banks increased 7 percentage points—from about 32 percent (54 employees) in 2011 to 39 percent (65 employees) in 2017. At individual FHLBanks, the percentage of female and minority senior management increased at seven banks by a range of about 3 to 29 percentage points, decreased at three FHLBanks by about 11 to 13 percentage points, and stayed the same at one bank. While combined female and minority representation increased overall, eight of the 11 banks did not have any female minorities in senior- management positions in 2017 (see fig. 5). Using EEOC data, we also examined the composition of the senior- management workforce in the financial services industry to determine how FHLBank senior management compares with the broader financial services industry. The percentage of women, minorities, and women and minorities combined in senior management in FHLBanks overall was similar to the corresponding share of senior management in the financial services industry in 2017. Specifically, the respective percentages for the FHLBanks and the financial services industry in 2017 were approximately 28 percent and 30 percent for female senior management; 14 percent and 13 percent for minority senior management; and 39 percent and 38 percent for female and minority senior management. Shares of women and minorities in senior management for individual FHLBanks varied more in comparison with the financial services industry in their districts. Four banks had a higher share of women in senior management than the financial services industry in their respective bank districts (from about 1 to 14 percentage points higher); and seven banks had a lower share (from about 3 to 15 percentage points lower). Five FHLBanks had a higher share of minorities in senior management than the financial services industry in their respective districts (from about 2 to 27 percentage points higher); the other six had a similar or lower share (from about 1 to 8 percentage points lower). Four banks had a higher percentage of women and minorities combined in senior management than the financial service industry in their respective bank districts (from about 7 to 27 percentage points higher), two banks had a modestly lower share (no more than 3 percentage points), and the remaining five banks had a share that was lower by more than 8 percentage points. We also reviewed the representation of women and minorities in the overall FHLBank workforce and for each FHLBank. Although the difference in the share of female and minority employees across 11 FHLBanks in 2011–2017 was not large, both women and minorities were better represented in first- and mid-level management and professional positions than in senior management. Female representation. Across the 11 FHLBanks, the overall share of female employees in 2017 (about 45 percent) was somewhat lower than the share in 2011 (about 47 percent), although the total number of female employees increased from 1,317 in 2011 to 1,355 in 2017 (see fig. 6). In 2017, the share of women in job categories below the senior- management level was higher than the share of women in senior management. Specifically, the share of women in first- and mid-level management positions was about 41 percent and in professional positions about 44 percent, both higher than the percentage of women in senior management (about 28 percent). Employees in these positions can be potential candidates for the banks’ management. Minority representation. The share of racial/ethnic minority employees in the overall FHLBank workforce in 2017 (about 33 percent) was slightly higher than the share in 2011 (about 31 percent), and the number of racial/ethnic minorities increased during this period from 864 employees in 2011 to 1,007 employees in 2017. During this time period, the share of minorities in first- and mid-level management positions increased by approximately 6 percentage points (from about 21 percent in 2011 to 27 percent in 2017); the share of professionals increased by about 3 percentage points (from about 34 percent in 2011 to 37 percent in 2017), and the share of minorities in other job categories, such as administrative, decreased by about 3 percentage points (from about 41 percent in 2011 to 38 percent in 2017). Similar to the share of female employees, the share of minority employees in first- and mid-level management (about 27 percent) and professional positions (about 37 percent) was higher than that for senior management (about 14 percent) in 2017. Among these employees in 2017, Asians accounted for the largest share (about 16 percent), followed by African-Americans (about 11 percent) and Hispanics (about 5 percent), as shown in figure 7. Female and minority representation. When looking at the combined representation of women and minorities in the overall FHLBank workforce, the share of women and minorities was similar in 2011 and 2017 at about 61 percent (1,704 employees in 2011 and 1,847 employees in 2017). The number of female and minority employees increased by 143 (about 8 percent) during this period. Similarly, the number of total employees increased by about 200 (about 8 percent) from 2011 to 2017. At the individual bank level, the share of female and minority employees increased at six banks (from about 1 to 6 percentage points) and decreased at the remaining five banks (from about 2 to 4 percentage points) from 2011 to 2017. In 2017, the percentage of female and minority employees across the 11 banks ranged from about 48 to 77 percent of the workforce at the individual banks (see fig. 8). Women generally were less represented among FHLBank employees than in the financial services industry (overall and by FHLBank district) and in college-educated populations in selected metropolitan areas. Minorities were similarly represented across the categories. First, we compared the representation of women and minorities among FHLBank employees (overall and by bank) with such representation in the financial services industry (overall and by FHLBank district) in 2017 to help determine how similar the FHLBank workforce was to that of other financial institutions. The workforce in the financial service industry can represent a pool of potential employees for the FHLBanks. The share of female employees in FHLBanks overall was about 14 percentage points lower than the corresponding share in the financial services industry (about 45 percent in FHLBanks and 59 percent in the financial services industry) in 2017. Each of the FHLBanks had a lower share of female employees than the financial services industry in the bank’s district (by about 2 to 27 percentage points). The share of racial/ethnic minority employees was about 33 percent across the FHLBanks and the financial services industry in 2017. Six FHLBanks had a similar or higher share of minority employees than the financial services industry in their respective districts (by about 1 to 21 percentage points); the other five had a lower share (by less than 1 percentage point to about 11 percentage points). The combined percentage of female and minority employees across the FHLBanks was about 10 percentage points lower than the corresponding percentage in the financial services industry in 2017 (about 61 percent and 71 percent, respectively). All FHLBanks except one had a lower share in this combined category than the financial services industry in their districts. Second, we compared the share of women and minorities among each FHLBank’s employees in 2017 with the population with at least a bachelor’s degree in the metropolitan statistical areas associated with each bank’s headquarters city in 2018 (see table 2). This population can provide potential employees for the banks’ workforce. The percentage of female employees in seven banks was lower than the estimated share of females with at least a bachelor’s degree in their respective metropolitan areas (smaller than the lower end of the range of the estimated percentage for each area). For the remaining four banks, the share of female employees was similar to that of the estimated share for their respective metropolitan areas (within the range of the estimated percentage for each area). The percentage of minority employees in nine banks was similar to the estimated share of minorities in the population in the respective metropolitan areas around each bank’s headquarters. In two banks, the share of minority employees exceeded the corresponding estimated percentage for this population (larger than the upper end of the range of the estimated percentage for each area). To provide additional context on the demographic composition of the population served by the FHLBanks, we compared the share of female and minority employees at each FHLBank in 2017 with the share of the female and minority population in each bank district (all of which are multistate areas). All FHLBanks except two had a lower share of female employees than the female share of the population in their respective bank districts (by at least 5 percentage points). The percentage of minority employees at one bank was higher than the share of the minority population in its bank district (by about 11 percentage points); the individual percentages at five banks were similar (no more than 3 percentage points difference); and the individual percentages at the remaining five banks were lower by at least 4 percentage points. FHLBanks reported continuing challenges to recruiting and retaining a diverse workforce, including the following: Low turnover rates and small workforce. Staff of four banks said that low turnover rates have limited opportunities for hiring or promoting diverse candidates. For example, the percentage of employees leaving individual banks in 2017 ranged from about 5 to 12 percent. In comparison, the average estimated separation rate for the financial services industry as a whole was about 25 percent in 2017. Additionally, staff of four banks noted that the size of their workforce is relatively small, which also limits opportunities for hiring and promotion. The number of employees in individual FHLBanks ranged from 202 to 462 in 2017 (see table 3). Population in geographic location not diverse. Staff of four banks stated that their geographic location makes it challenging to recruit diverse talent because the population in the area is relatively undiverse. Two banks indicated it can be difficult to attract potential candidates to work in their geographic location. Despite these stated challenges, as we previously noted, the 2017 share of minorities in each bank was similar to or exceeded the 2018 share of minorities with at least a bachelor’s degree in their respective metropolitan areas. Competition for women and minority candidates. Staff of five banks said that competition for diverse talent is high because banks compete with other companies in their districts that are larger or have better brand recognition, such as large investment banks and technology companies. For example, staff of four banks noted that the FHLBanks are not well known compared with these larger organizations. Staff from one bank noted that the compensation the bank offered was lower than that of larger firms—including for internships—which can make it difficult to attract diverse candidates. Staff of two banks also noted that relatively low unemployment rates in their areas mean that diverse candidates have other employment options, making it more challenging to attract such candidates. Difficulty aligning bank needs and requirements with skillsets of diverse candidates. Staff of six banks said that there may be few women or minority candidates who meet specific skill or job requirements. For example, staff of five of these banks noted that it can be challenging to find diverse candidates in certain technical fields, such as information technology. Staff of three banks also noted that diversity in the financial services industry overall is limited, which contributes to a limited pool of diverse candidates. We found that FHLBanks implemented and continue to implement a variety of practices to maintain and increase diversity in their workforces, based on our review of the FHLBanks’ annual Office of Minority and Women Inclusion (OMWI) reports, FHFA examination documents, FHLBank diversity and inclusion strategic plans, and interviews with FHLBank staff from all 11 banks. These practices align with leading practices we previously identified on diversity management. The leading practices can help the banks address some of the challenges described previously and recruit and retain a diverse workforce, which also can contribute to a more diverse pipeline for management positions. Bank leadership commitment to diversity and inclusion. All 11 FHLBanks have implemented practices intended to demonstrate leadership’s commitment to diversity and inclusion, which included the following examples. All 11 FHLBanks include workforce diversity objectives in their diversity and inclusion strategic plans and generally established goals that were quantitative, qualitative, or both related to their workforce diversity programs, based on our review of the banks’ annual reports. Examples of such goals included increasing employee awareness of diversity and inclusion and percentage targets for workforce diversity composition and recruitment. The FHLBanks also incorporate diversity and inclusion into their incentive compensation goals or performance competencies. An example of such goals relates to participation in diversity and inclusion training and other events. All 11 FHLBanks track data on the diversity composition of their workforce; external and internal applicants for open positions, new hires, promotions, and separated employees; and progress in meeting diversity and inclusion goals and objectives. The OMWI officers at all 11 banks report directly to the bank’s chief executive officer/president or the equivalent of the chief operating officer. The board of each bank also receives periodic updates on the bank’s diversity and inclusion efforts. Staff of eight banks said that senior leaders, such as the chief executive officer, express their commitment to diversity and inclusion through participation in internal diversity and inclusion events, in written materials, and by sponsoring employee groups that represent diverse employees. Targeted recruitment. All 11 banks reported several targeted diversity recruitment efforts to increase recognition and build a potential pipeline of diverse employees. For example, all banks conducted outreach to colleges that have diverse student populations, according to banks’ annual reports and staff, and FHFA examination documents. All banks also conducted outreach to local and national professional and other organizations that represent diverse communities. Seven banks indicated that they engage with their communities, such as by participating in community events and volunteer activities, and partnering with community organizations. They noted that these efforts can help enhance their bank’s brand recognition and in turn can help recruit and retain diverse employees. To build a pipeline of diverse employees, all 11 banks offered a college internship program and six banks offered a high school internship or work study program for which they try to recruit diverse candidates. The banks also engaged in efforts to build the potential pipeline of diverse employees in the long term, such as by participating in programs or activities to increase skillsets among young women and minorities in technical or financial services fields. Employee involvement/feedback. All 11 banks described efforts to create a more inclusive environment for employees, according to banks’ annual reports and bank staff. For example, nine banks have an employee resource group or other organization representing employees, and can engage in diversity and inclusion activities, such as professional development and cultural events, according to the banks’ annual reports. Nine banks reported that they conducted employee surveys or meetings to obtain feedback from employees on diversity and inclusion efforts, according to the banks’ annual reports and staff. Staff from one bank told us that when conducting interviews with employees leaving their organization, they include a question specifically on diversity and inclusion to identify potential employee retention practices. Training on diversity and inclusion topics. Ten banks offered training courses on diversity and inclusion topics for all employees, according to the banks’ annual reports and FHFA examination documents. Ten banks hosted events or informal training related to diversity and inclusion, including events sponsored by employee groups, according to bank documents and staff. Development of succession plans that address diversity and inclusion. All 11 banks engaged in succession planning, but FHFA’s 2018 diversity and inclusion examination found that the banks addressed diversity and inclusion in their succession planning to varying degrees. FHFA staff explained that banks should evaluate potential successors on their demonstrated ability to manage diversity and inclusion using performance competencies. Examples of such competencies include assessing candidates on their ability to include diverse groups when making team decisions and supporting the bank’s diversity and inclusion efforts. FHFA worked with FHLBanks and developed instructions and templates for more consistent reporting of 2018 data on the banks’ use of diverse suppliers and broker-dealers, including those that are minority- and women-owned. FHLBanks’ use of minority- and women-owned suppliers and broker-dealers in 2018 varied among the banks. Banks also told us there are challenges that may slow or limit their use of diverse suppliers and broker-dealers. They generally implemented key practices to help ensure they consider diverse suppliers and broker-dealers in searches for business partners. Data reporting. Before 2018, FHFA had not issued a standardized data reporting template for FHLBank data on use of diverse suppliers and broker-dealers; therefore, data were not comparable across banks or years. As part of the requirements of FHFA’s 2010 Minority and Women Inclusion regulation, in 2012 the banks and the Office of Finance began reporting data on their business activities with diverse businesses (minority-, women-, and disabled-owned) in the preceding year. However, the data prior to 2018 were not comparable across years and banks because the banks did not use consistent methods or definitions in their data reporting. To develop a common understanding and make the data more consistent, FHFA and the banks began working together in 2017 to develop a data dictionary and data templates. FHLBanks used the new templates to report their 2018 data. Minority- and women-owned suppliers. In 2018, FHLBanks varied in their use of minority- and women-owned suppliers (see fig. 9). The 11 banks entered into more than 2,900 supplier contracts overall in 2018 (ranging from 60 to 477 per bank). Of the total number of contracts, about 10 percent (279 contracts) were with minority-owned suppliers and about 12 percent (340 contracts) were with women-owned suppliers. Among the individual banks, the share of contracts entered into with minority- owned suppliers in 2018 ranged from about 1 percent to 38 percent and from about 4 percent to 25 percent for contracts with women-owned suppliers. In 2018, FHLBanks’ total supplier expenditure was about $453 million, of which about 8 percent and 13 percent, respectively, went to minority- and women-owned suppliers. Among the individual banks, the percentage of the total annual 2018 expenditure that went to minority-owned businesses varied from about 3 percent to 15 percent, and to women-owned businesses from about 2 percent to 31 percent (see fig. 10). According to FHFA staff, annual expenditure paid to suppliers can vary from year to year. More specifically, an increase in a bank’s annual supplier expenditure in any one year is usually related to long-term, large investments made during that year, such as construction costs or investment in technology products and services. FHFA staff noted that these one-time increases in expenditures can provide opportunities to increase the use of diverse suppliers. FHFA staff said bank data showed that for example, in 2018, three FHLBanks each had large one time investments in construction or building maintenance. Diverse broker-dealers in debt transactions. FHLBanks conduct capital markets transactions with broker-dealers that meet certain qualifications (such as capital sustainability and financial performance), including those that have been approved as diverse broker-dealers. As previously mentioned, these transactions include debt issuance and investments. The Office of Finance acts as an agent to the banks and primarily functions to issue and service all debt transactions. In addition, it identifies and approves broker-dealers for the banks’ debt issuance transactions, including dealers that are minority-, women-, disabled-, and veteran-owned. As of December 31, 2018, the Office of Finance had 64 approved broker- dealers, 16 of which were diverse broker-dealers, including seven minority-owned and five women-owned firms. In 2019, the Office of Finance added two additional diverse broker-dealers, one minority-owned and one disabled veteran-owned, bringing the total to 18. This represents an increase in the number of approved diverse broker-dealers from 10 in 2014 (see table 4). A total of 69 broker-dealers conducted at least one debt transaction with the Office of Finance in 2018. Ten percent of these broker-dealers were minority-owned and 7 percent were women-owned. In 2018, FHLBanks issued about $8 trillion in debt transactions. Of this total volume, approximately 3 percent of transactions were conducted with minority-owned broker-dealers and less than 1 percent with women- owned broker-dealers. Similarly, minority-owned broker-dealers received approximately 5 percent of the fees paid to broker-dealers overall on these transactions, and women-owned broker-dealers received approximately 0.5 percent. While the Office of Finance reports debt volume data and other data, such as number of transactions conducted by diverse broker-dealers, to FHFA, staff noted that they also use two other performance goals to measure their capital markets diversity efforts. These two goals are the utilization of diverse broker-dealers in debt issuance programs and the number of outreach engagements with diverse broker-dealers (such as marketing and investor meetings). According to Office of Finance staff, as of June 2019, diverse broker- dealers had the opportunity to participate in all FHLBank debt issuance programs. However, as discussed later, some practices that the Office of Finance implements to control risk may limit diverse broker-dealers from taking a more substantial role in certain types of transactions. Diverse broker-dealers in investment transactions. FHLBanks make investments based on their investment needs and identify and approve broker-dealers for their investment needs according to their own qualification requirements. As shown in figure 11, the number of minority- owned broker-dealers approved by the FHLBanks ranged from five to 12, and the number of approved women-owned broker-dealers ranged from one to seven as of December 2018. Of the total number of broker-dealers that conducted at least one investment transaction with FHLBanks in 2018, the share of minority- or women-owned broker-dealers varied among banks (see fig. 12). Of the 10 banks that made investment transactions in 2018, shares for individual banks ranged from 0 percent to about 22 percent for minority-owned broker-dealers and from 0 percent to 10 percent for women-owned broker-dealers. In 2018, FHLBanks conducted about $12 trillion in investment transactions, less than 1 percent of which was conducted with minority- owned broker-dealers or with women-owned broker-dealers. Of the total number of transactions conducted by the banks, minority-owned dealers and women-owned dealers each accounted for less than 1 percent. FHLBank staff reported some challenges that may slow or limit their use of diverse suppliers and broker-dealers, such as those owned by minorities, women, and individuals with a disability. For example, staff from seven banks said that the bank’s needs for goods and services are small or can fluctuate from year to year. Staff from two banks added that this can make it difficult to consistently increase or maintain the use of diverse suppliers. For example, staff from one bank described a building construction project that increased the use of diverse suppliers during the year in which the construction took place, but had no effect in the subsequent year because construction had been completed. In addition, staff from five banks said some bank procurement needs are fulfilled by continuing contracts with existing vendors. For example, staff from two banks said that some bank needs, such as existing information system support, are offered by continuing suppliers that may not be diverse suppliers. Additionally, staff from five banks said that there are not always diverse suppliers that can meet the bank’s needs. For instance, staff from one bank said it can be difficult to find diverse suppliers to fill some contracting needs that require specific skills or expertise, such as the vendors used to review technical risk-assessment models. FHLBank staff also reported challenges that may slow or limit their use of diverse broker-dealers. For example, staff from six banks and the Office of Finance said that diverse broker-dealers often do not have the level of capital required by the banks to make the capital markets transactions the banks need. Staff from five banks said this is because diverse broker- dealers generally are smaller firms. In addition, staff from seven banks said that diverse broker-dealers may be limited in the services and products they can offer the banks. For example, staff from one bank told us that some diverse broker-dealers have fewer financial resources, which limits them to basic transactions as opposed to more complex transactions. Staff from six banks also said that their capital markets transactions are dependent on membership needs or market conditions for funding, which can lead to year-by-year fluctuations in transaction levels. Staff from four banks said that the fluctuations affect the bank’s need for broker-dealers overall, and make it challenging for the bank to maintain or increase use of diverse broker-dealers. Office of Finance staff also noted this challenge, adding that many factors, such as underwriting capacity and experience, may affect a broker-dealer’s ability and desire to participate in the FHLBanks’ debt issuance programs. Staff further said that individual broker-dealers are responsible for identifying investors to be able to participate in debt transactions; the Office of Finance cannot control whether a broker-dealer can identify an investor. We previously identified key practices for increasing opportunities for minority- and women-owned asset managers. We found these practices can be applied to diverse suppliers and broker-dealers and used by organizations, such as FHLBanks, to help ensure they consider qualified diverse suppliers and broker-dealers in their selection process. Diverse suppliers and broker-dealers include businesses owned by minorities, women, and individuals with a disability. The key practices are Demonstrate top leadership commitment: Demonstrate commitment to increasing opportunities for diverse businesses. Conduct outreach: Conduct outreach to inform diverse businesses about opportunities and the selection processes. Communicate priorities and expectations: Explicitly communicate priorities and expectations about inclusive practices to staff and ensure those expectations are met. Remove potential barriers: Review policies and practices to remove barriers that limit the participation of diverse businesses. We found that FHLBanks generally implemented the four key practices in their supplier management programs, based on our review of the FHLBanks’ 2017 and 2018 annual OMWI reports and interviews with OMWI staff from all 11 banks, and with bank staff with responsibility for vendor management. Demonstrate top leadership commitment. The FHLBanks demonstrated top leadership commitment to supplier diversity through strategic plans, goals, and reporting. All 11 FHLBanks include supplier diversity as a component of their diversity and inclusion strategic plans. In addition, the banks generally established quantitative or qualitative goals related to their supplier diversity programs; for example, the percentage of total expenditure with diverse suppliers. FHFA told us that they have been working with the banks to assess these goals and ensure they are outcome-based. Furthermore, the 11 banks track their progress in meeting diversity and inclusion objectives and goals. Each FHLBank’s OMWI director reports to the bank’s chief executive officer/president or the equivalent of the chief operating officer. The FHLBank boards also receive periodic updates on the bank’s diversity and inclusion efforts. Conduct outreach. FHLBanks use a variety of methods to reach potential diverse suppliers. All 11 banks work with local or national industry organizations, such as the National Minority Supplier Development Council and Women’s Business Enterprise National Council, to identify potential suppliers. Nine banks described attending events hosted by these organizations, such as matchmaking sessions or business fairs, and at least two banks reported using their databases to search for diverse suppliers. Staff from seven banks said that they proactively meet with potential vendors to educate them on bank needs and processes. Staff from one bank said they have invited a number of diverse vendors to the bank to meet bank managers and discuss their goods and services. This has resulted in contract proposals from five different diverse vendors. Staff from five banks also described using advertising and social media to reach a broad base of potential diverse suppliers; for example, one bank described placing advertisements in publications that target diverse businesses. Communicate priorities and expectations. FHLBanks communicated priorities and expectations on supplier diversity to bank staff through policies and training. All 11 banks have a written policy that outlines the requirements for bank staff to include a diverse supplier in their search whenever the need for a new contract is identified. In addition, all 11 banks conducted staff training on supplier diversity or on their vendor management policy. Remove potential barriers. According to our interviews with external stakeholders knowledgeable about working with diverse suppliers, diverse businesses may face barriers as they seek to obtain contracts. FHLBanks took steps that could ameliorate these barriers. Supplier contracts are often made through existing relationships and networks. Diverse suppliers may not have access to these networks and therefore miss opportunities to apply for contracts. As previously mentioned, FHLBanks conducted targeted recruitment of diverse suppliers. By actively seeking to build relationships with these suppliers, the FHLBanks have been working to address this barrier. The procurement process itself can be complicated and difficult to understand. Smaller diverse businesses may have limited staff and skill to navigate the process. Staff from seven banks have addressed this barrier by conducting one-on-one meetings with potential vendors to walk them through the bank’s procurement processes. Third- Party Certification of Supplier Diversity Status According to the Federal Housing Finance Agency’s (FHFA) 2010 Minority and Women Inclusion regulation, a firm qualifies as a minority- or women-owned business when it is more than 50 percent owned and controlled by one or more minority individuals or women and more than 50 percent of net profit or loss accrues to a member of those groups. Businesses can submit documentation to approved third-party certifiers to obtain a certification of their diversity status. Certifiers then review the documentation and sometimes conduct site visits to confirm the diversity status of the business. The process to certify as a diverse business with a third party can be confusing or costly, according to two external stakeholders we interviewed. The preambles to the 2010 Minority and Women Inclusion rule and its 2017 amendments state that while FHFA prefers reliance on certifications from qualified, independent third parties, FHFA also allows for reliance on self-certifications by the businesses. The FHLBanks each confirmed that they allow businesses to self- certify their diversity status. A small diverse supplier may not be able to fulfill a large contract requiring multiple services. An external stakeholder we interviewed told us that this barrier can be overcome when diverse suppliers join forces to fulfill multipart contracts. For example, a supplier that provides pens can join with a supplier that provides paper to fulfill a single office supplies contract. To do this, suppliers need advance notice of bank needs to create a business plan. Three banks told us that before meeting with potential suppliers, they work with various business departments in the bank to identify upcoming purchasing needs and share those with the suppliers. All 11 banks have a representative on the systemwide OMWI Council Procurement Sub-Working Group, which meets monthly. The subgroup spent the majority of 2017 addressing the challenge of FHFA data reporting. An OMWI Council representative told us that the members use the subgroup as forum to discuss key issues. In addition, the representative said the subgroup plans to focus on improving its outreach efforts in 2019. Two banks reported that they hold internal meetings to discuss trends and potential barriers and make updates to their supplier diversity program. We found that the FHLBanks and the Office of Finance generally implemented the four key practices for their capital markets programs, based on our review of the FHLBanks’ 2017 and 2018 annual OMWI reports and interviews with OMWI staff from all 11 banks, the Office of Finance, and with bank staff with responsibility for capital market activities. Demonstrate top leadership commitment. Similar to the banks’ supplier management programs, FHLBanks demonstrated top leadership commitment to capital markets diversity through strategic plans, goals, and reporting. All 11 FHLBanks and the Office of Finance include capital markets diversity as a component of their diversity and inclusion strategic plans. They also generally established quantitative or qualitative goals related to their capital markets diversity programs, such as the percentage of transactions conducted with diverse broker-dealers. The 11 banks and the Office of Finance track their progress in meeting diversity and inclusion objectives and goals. In addition, the OMWI director reports to the chief executive officer/president or the equivalent of the chief operating officer. The FHLBank and the Office of Finance boards also receive periodic updates on the bank’s and the Office of Finance’s diversity and inclusion efforts, respectively. Conduct outreach. FHLBanks and the Office of Finance interacted with diverse broker-dealers through regular communication and face-to-face meetings. Staff from all 11 banks and the Office of Finance reported some form of regular communication with diverse broker-dealers to keep the broker-dealers informed on bank capital markets activities and needs. In addition, all 11 banks and the Office of Finance reported attending events to interact with diverse broker-dealers. For example, on behalf of the OMWI Council Capital Markets Subgroup, the FHLBank of New York hosts an annual Diverse Dealer Reception at which the banks and the Office of Finance can interact with current approved diverse broker- dealers and those in the pipeline for potential approval. At the 2017 reception, diverse broker-dealers were provided with contact information for all capital market staff in the FHLBank System and a brochure listing examples of ways diverse broker-dealers could engage with the system. Staff from all 11 banks and the Office of Finance also told us they hold one-on-one meetings with diverse broker-dealers to explain bank processes and needs. In addition, Office of Finance staff told us they help diverse broker-dealers build relationships with investors by accompanying them to one-on-one meetings with investors to introduce FHLBank securities products. Communicate priorities and expectations. FHLBanks communicate priorities and expectations on capital markets diversity to bank staff through policies and by sharing practices systemwide. Ten of the banks and the Office of Finance have a written policy or procedure related to the use of diverse broker-dealers, which outlines the importance of engagement with diverse broker-dealers or how bank staff should interact with them. The OMWI Council Capital Markets Subgroup also developed a list of aspirational practices for the 11 banks. These practices include many activities related to the four key practices we identified, such as engaging in regular communication and periodically examining capital market operations to identify potential obstacles to increasing business with diverse broker-dealers. Staff from one bank told us that having the practices codified in a document helps ensure consistent expectations across the system. Remove potential barriers. FHLBanks and the Office of Finance made changes to their capital markets practices and certain features of their debt products to increase access for diverse broker-dealers. For example, the Office of Finance, with input from the OMWI Council Capital Markets Subgroup, reduced the capital requirements on some types of debt transactions to bring in more diverse broker-dealers. The Office of Finance also told us that in the case of a new type of debt product created in November 2018, they allow multiple broker-dealers to participate in various roles, including diverse broker-dealers. However, FHFA staff told us that although the FHLBanks and the Office of Finance made changes to debt and investment products offered to diverse dealers, the banks and the Office of Finance have not always used a systematic process to review and evaluate debt and investment policies and procedures for potential changes that may expand participation by diverse broker-dealers. In 2017 and 2018, FHFA asked the banks and the Office of Finance, respectively, to develop such a process to facilitate opportunities for diverse broker-dealers. FHFA determined that 10 banks had addressed this request. FHFA staff noted that they will review the remaining entities’ progress in addressing this request in 2019 examinations. In addition, each bank and the Office of Finance has a representative on the systemwide OMWI Council Capital Markets Subgroup. According to staff from four banks, the subgroup’s monthly meetings provide them with an opportunity to discuss barriers and practices. In 2018, the subgroup administered a survey to the approved diverse broker-dealers to solicit suggestions and feedback on their interactions with the banks. Bank staff told us the survey did not result in any program changes, but provided information on how they could communicate more effectively. However, diverse broker-dealers still may face barriers in some areas, according to three diverse broker-dealers and two industry stakeholders, with whom we spoke. The Office of Finance has been taking steps to address these barriers where possible. Some practices that the Office of Finance implements to control risk can limit diverse broker-dealers from taking a more substantial role in transactions associated with certain debt products. For example, according to Office of Finance staff, only the top eight broker-dealers (ranked by the Office of Finance based on performance) can lead transactions for certain longer-term and larger-size debt products. Two diverse broker-dealers told us this requirement limits their ability to participate in these transactions. For example, one broker-dealer told us that because diverse broker-dealers generally are newer firms with less capacity relative to the top eight broker-dealers, it would be unlikely that they would ever be one of the top eight. Office of Finance staff told us these products accounted for less than 1 percent of the FHLBank’s total debt issuance in 2018 based on net proceeds received. According to the Office of Finance, they rely on these top eight broker-dealers because they can better cover any risk posed by the transactions. The Office of Finance allows diverse broker-dealers to serve as co-managers on these larger transactions, but diverse broker-dealers told us that acting as a co-manager did not noticeably increase the share of transactions they could execute or their own fee revenue. Staff from the Office of Finance said the office does not implement quotas as a way to maintain or increase the use of diverse broker- dealers, but rather focuses on providing diverse broker-dealers with opportunities and on implementing outreach opportunities. According to these staff, the percentage of transactions conducted and fees received by diverse broker-dealers as a whole has increased over time. The Office of Finance told us they recently met with diverse broker- dealers about another risk-management practice that may limit the participation of diverse broker-dealers. According to Office of Finance staff, this practice requires broker-dealers to have at least $100 million in capital to conduct certain complex debt transactions. According to external stakeholders, a higher capital requirement may limit the participation of diverse broker-dealers, who generally have less capital, in these transactions. Staff from the Office of Finance said that they have been evaluating whether it is appropriate to modify the requirement. Their evaluation is part of the office’s continual process to evaluate debt issuance programs. Three diverse broker-dealers and one industry stakeholder we interviewed said that increased transparency by FHLBanks and the Office of Finance in information provided to broker-dealers could help diverse broker-dealers identify opportunities and better understand the banks’ needs. For example, one diverse broker-dealer and one industry stakeholder said that access to information on the fees paid to broker-dealers on different types of capital markets transactions could help them take advantage of areas of greater opportunity. This information is reported by the Office of Finance and FHLBanks to FHFA, but generally is not released publicly. The preamble of FHFA’s Minority and Women Inclusion rule notes that FHFA treats this information as confidential because it can affect the agency’s oversight of the banks. However, FHFA does not prohibit FHLBanks and the Office of Finance from publishing their diversity information if they so choose. Office of Finance staff told us that they do not publish data on fees or broker-dealer transactions for certain debt product because they consider this proprietary and competitive information. They said publishing the data could increase the leverage of broker- dealers and also could lead to an adverse impact on investor participation and support. FHFA’s oversight of FHLBanks includes annual examinations, development of instructions and templates to improve data quality, incorporation of bank data in oversight, and communication of agency expectations for diversity and inclusion efforts to the banks through various mechanisms. Began examining FHLBanks’ diversity and inclusion efforts in 2017. In 2017, FHFA started reviewing FHLBanks’ diversity and inclusion efforts in its annual examinations of the banks. FHFA developed a separate examination module (to add to its examination manual) in 2016 for reviewing the banks’ diversity and inclusion efforts and the banks’ oversight of these efforts. The areas that FHFA reviews include strategic planning and associated goals for diversity and inclusion, board oversight, organizational structure of diversity and inclusion programs, workforce, suppliers (which encompasses broker-dealers in the capital markets program), reporting structure and processes, and internal audit and compliance. In the 2017 and 2018 examinations, FHFA found the banks generally took steps to promote and maintain diversity and inclusion in their workforce and use of diverse suppliers and broker-dealers. FHFA also identified some areas for improvement. Specifically, in the 2017 examinations, FHFA recommended that all 11 banks improve their reporting and program goals on workforce diversity and use of diverse suppliers and broker-dealers. For example, FHFA specifically found that six banks needed to improve performance measurement of their supplier diversity goals. In the 2018 examinations, FHFA recommended that seven banks enhance their succession planning to ensure that potential successors are assessed on how well they manage and implement diversity and inclusion. As previously discussed, FHFA also asked FHLBanks and the Office of Finance to develop a more systematic process to review and determine potential changes to their debt and investment policies that could expand participation by diverse broker-dealers. Based on our review of FHFA’s examination documentation, FHFA followed its processes to document, communicate, and resolve examination findings related to diversity and inclusion in its 2017 and 2018 examinations. For example, FHFA examiners prepared memorandums to document the assessment and findings of each individual bank’s diversity and inclusion efforts and communicated findings to bank management and boards. Consistent with the examination manual, FHFA followed up on 2017 examination findings and banks’ remediation actions during the 2018 examinations. As of March 2019, FHFA determined that 10 banks satisfactorily remediated findings from the 2017 examination related to goals and reporting issues, among other things. For the remaining bank, management has not completed all remediation steps to address FHFA’s examination findings, according to FHFA staff. FHFA staff added that they will review the bank’s actions again in the 2019 examination and assess the banks’ progress in addressing the 2018 examination findings. According to FHFA staff, they plan to make some changes to the diversity and inclusion examination module. For example, in the module FHFA plans to more explicitly separate the information on the review of diversity efforts related to use of diverse broker-dealers from use of diverse suppliers (they are currently under one examination component). Developed instructions and templates to improve data quality. To enhance the quality of the data and information submitted by FHLBanks on their workforce diversity and use of diverse suppliers and broker- dealers, FHFA worked with FHLBanks and developed instructions and templates to help FHLBanks submit more consistent data on a quarterly basis. During 2018, FHFA requested that banks submit quarterly diversity data on their workforce and the use of diverse suppliers and broker- dealers. FHFA also developed a data reporting manual that includes a data dictionary, and templates for the quarterly and annual data and for the annual report to help FHLBanks more consistently report diversity data for their workforces and use of diverse suppliers and broker-dealers. FHFA staff told us that they reviewed the banks’ 2018 data to identify any discrepancies, and they worked with the banks to clarify data definitions and correct the discrepancies. For example, some banks had used an incorrect definition to account for their diverse supplier expenditures. Because 2018 was the first year in which the banks used the new templates, FHFA staff said they had expected some discrepancies in the data as the banks became familiar with the data definitions. Staff said FHFA plans to continue to work with the banks to help them achieve a common understanding of the data definitions. Incorporated bank data in oversight. According to FHFA staff, in 2018 they began to use the banks’ quarterly data for ongoing monitoring of the banks’ diversity and inclusion efforts in workforce, procurement, and capital markets. For example, the FHFA OMWI office assesses each bank’s diversity performance in these three areas using the quarterly data, and has been considering developing benchmarks. FHFA staff said the quarterly data provide more detailed information on the banks’ use of diverse businesses; for example, the types of goods or services for which the banks contract with diverse businesses. FHFA staff noted that the additional data not only inform FHFA’s oversight but also can help the banks’ internal reporting on diversity and inclusion efforts. Additionally, FHFA plans to review the banks’ data reporting systems as part of its annual examinations to help ensure banks have the appropriate controls for data reporting. FHFA staff said that the agency expects the banks to establish the appropriate data system to ensure the quality of data reported to FHFA and for internal reporting. Communicated with FHLBanks, including on data templates and expectations. FHFA provided clarification on the roles and duties of the banks’ OMWI officers and the scope of diversity regulations. FHFA collected the banks’ feedback and responded to questions on the new quarterly data reporting and the new data instructions and templates. Subsequently, FHFA modified the data templates in 2019 to allow the banks to more efficiently report their diversity data on a quarterly and annual basis. For example, FHFA consolidated data fields common to quarterly and annual reporting, among other things. Additionally, FHFA provided responses to the banks on their questions on the data and annual report templates when the templates were first introduced in 2018 and revised in 2019. FHFA staff said the annual report template helped clarify FHFA’s expectation on annual report content. In addition, FHFA staff noted that since 2015, FHFA’s OMWI director has met with the bank presidents and board of directors of most of the FHLBanks, and began in 2018 to have at least one visit for each bank every other year. The FHFA OMWI director also generally attends the semi-annual conferences of the banks’ OMWI officers, during which she has the opportunity to meet with the banks’ presidents individually. During these meetings, the OMWI director or staff discussed diversity issues such as strategic planning, results of the banks’ annual reports, and examinations. We provided a draft of this report to FHFA, each of the 11 FHLBanks, and the Office of Finance for review and comment. FHFA, six FHLBanks, and the Office of Finance provided technical comments, which we incorporated as appropriate. The other five FHLBanks did not have any comments. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Director of FHFA, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or ortiza@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. Anna Maria Ortiz, (202) 512-8678, ortiza@gao.gov. In additional to the individual named above, Kay Kuhlman (Assistant Director), Anna Chung (Analyst in Charge), Meghana Acharya, Laurie Chin, Kaitlan Doying, Jill Lacey, Moon Parks, Barbara Roesmann, Jessica Sandler, and Jena Sinkfield made key contributions to this report.", "summary": "The FHLBank System consists of 11 regionally based banks that are cooperatively owned by member institutions (such as community banks and credit unions) and of the Office of Finance. The banks, which are regulated by FHFA, provide liquidity for their member institutions to use in support of housing finance and community lending. GAO was asked to review FHLBanks' implementation of diversity and inclusion matters in workforce and business activities (including the use of suppliers and broker-dealers). This report examines (1) trends in gender, race, and ethnicity in FHLBank workforces, and challenges faced and practices used to maintain and increase a diverse workforce; (2) use of minority- and women-owned suppliers and broker-dealers in 2018, and challenges faced and practices used to increase and maintain their use; and (3) FHFA oversight of FHLBank diversity and inclusion efforts. GAO analyzed FHLBank and Equal Employment Opportunity Commission data on the banks' workforce, suppliers, and broker-dealers. GAO also reviewed FHFA and FHLBank policies and regulations and previous GAO work on these issues. GAO interviewed FHFA and FHLBank staff and a nongeneralizable sample of external stakeholders knowledgeable about supplier and broker-dealer diversity. From 2011 to 2017, the share of women in senior management in Federal Home Loan Banks (FHLBank) increased from about 21 percent (35 individuals) to 28 percent (47 individuals). The share of minority senior management remained the same at about 14 percent (23 individuals). The overall share of women employees slightly decreased and minority employees slightly increased during this period, but gender and minority representation varied by individual bank. FHLBanks identified challenges to maintaining and increasing workforce diversity, such as limited hiring opportunities due to low turnover. FHLBanks have been taking steps to promote workforce diversity, such as outreach to organizations that represent women or minorities and incorporation of diversity and inclusion in incentive compensation goals or performance competencies. In 2018, use of minority- and women-owned suppliers (for goods and services) and broker-dealers varied among individual FHLBanks. Overall, minority- and women-owned suppliers accounted for 8 percent and 13 percent of procurement expenditures, respectively. Minority- and women-owned broker-dealers accounted for about 3 percent and less than 1 percent of the debt issuance amount, respectively. FHLBanks and the Office of Finance (which issues debts on behalf of the banks) have been taking steps to increase diversity in these business activities, such as conducting outreach to diverse entities. However, external stakeholders said such suppliers and broker-dealers may continue to face some barriers—for example, capital requirements that limit participation by diverse broker-dealers, which generally have fewer resources. In 2017, the Federal Housing Finance Agency (FHFA) started reviewing the diversity and inclusion efforts of FHLBanks in its annual bank examinations. In the 2017 and 2018 examinations, FHFA found the banks generally took steps to promote diversity and inclusion but also identified areas for improvement, such as improving goals for workforce and supplier diversity. In 2018, FHFA issued a manual and templates for reporting of quarterly and annual diversity data to help ensure consistent reporting of the data. FHFA also began using the quarterly data for ongoing monitoring of the banks' diversity and inclusion efforts.", "document_type": "gao"}
{"report": "We reported in September 2017 that while estimates of the economic effects of climate change are imprecise due to modeling and information limitations, they can convey useful insight into broad themes about potential damages in the United States. We also reported that according to the two national-scale studies available at the time that examined the economic effects of climate change across U.S. sectors, potential economic effects could be significant and these effects will likely increase over time for most of the sectors analyzed. For example, for 2020 through 2039, one of the studies estimated from $4 billion to $6 billion in annual coastal property damages from sea level rise and more frequent and intense storms. In addition, the national-scale studies we reviewed and several experts we interviewed for the September 2017 report suggested that potential economic effects could be unevenly distributed across sectors and regions. For example, one of the studies estimated that the Southeast, Midwest, and Great Plains regions will likely experience greater combined economic effects than other regions, largely because of coastal property damage in the Southeast and changes in crop yields in the Midwest and Great Plains (see fig. 1). This is consistent with the findings of the Fourth National Climate Assessment. For example, according to that assessment, the continued increase in the frequency and extent of high- tide flooding due to sea level rise threatens America’s trillion-dollar coastal property market and public infrastructure sector. As we reported in September 2017, information on the potential economic effects of climate change could help federal decision makers better manage climate risks, according to leading practices for climate risk management, economic analysis we reviewed, and the views of several experts we interviewed. For example, such information could inform decision makers about significant potential damages in different U.S. sectors or regions. According to several experts and our prior work, this information could help federal decision makers identify significant climate priorities as an initial step toward managing climate risks. Such a first step is consistent with leading practices for climate risk management and federal standards for internal control. For example, leading practices from the National Academies call for climate change risk management efforts that focus on where immediate attention is needed. As noted in our September 2017 report, according to a 2010 National Academies report, other literature we reviewed, and several experts we interviewed, to make informed choices, decision makers need more comprehensive information on economic effects to better understand the potential costs of climate change to society and begin to develop an understanding of the benefits and costs of different options for managing climate risks. The federal government faces fiscal exposure from climate change risks in a number of areas, and this exposure will likely increase over time, as we concluded in September 2017. In the March 2019 update to our High-Risk List, we summarized our previous work that identified several of these areas across the federal government, including programs related to the following: Disaster aid. The rising number of natural disasters and increasing reliance on federal assistance are a key source of federal fiscal exposure, and this exposure will likely continue to rise. Since 2005, federal funding for disaster assistance has been at least $450 billion. In September 2018, we reported that four hurricane and wildfire disasters in 2017 created an unprecedented demand for federal disaster resources and that Hurricanes Harvey, Irma, and Maria ranked among the top five costliest hurricanes on record. Subsequently, the fall of 2018 brought additional catastrophic disasters such as Hurricanes Florence and Michael and devastating California wildfires, with further needs for federal disaster assistance. Disaster costs are projected to increase as certain extreme weather events become more frequent and intense due to climate change—as USGCRP observed and projected. We reported in July 2015 that the federal government’s fragmented and reactive approach to funding disaster resilience presented challenges to effective reduction of climate-related risks. In addition, our prior work found that the Federal Emergency Management Agency’s (FEMA) primary indicator for determining whether to recommend that a jurisdiction receive disaster assistance—which was set in 1986—is artificially low because it does not accurately reflect the ability of state and local governments to respond to disasters. Without an accurate assessment of a jurisdiction’s capability to respond to a disaster without federal assistance, we found that FEMA runs the risk of recommending that the President award federal assistance to jurisdictions that have the capability to respond and recover on their own. Federal insurance for property and crops. The National Flood Insurance Program (NFIP) and the Federal Crop Insurance Corporation are sources of federal fiscal exposure due, in part, to the vulnerability of insured property and crops to climate change. These programs provide coverage where private markets for insurance do not exist, typically because the risk associated with the property or crops is too great to privately insure at a cost that buyers are willing to accept. From 2013 to 2017, losses paid under NFIP and the federal crop insurance program totaled $51.3 billion. Federal flood and crop insurance programs were not designed to generate sufficient funds to fully cover all losses and expenses, which means the programs need budget authority from Congress to operate. NFIP, for example, was about $21 billion in debt to the Department of the Treasury as of April 2019. Further, the Congressional Budget Office estimated in May 2019 that federal crop insurance would cost the federal government an average of about $8 billion annually from 2019 through 2029. Operation and management of federal property and lands. The federal government owns and operates hundreds of thousands of facilities and manages millions of acres of land that could be affected by a changing climate and represent a significant federal fiscal exposure. For example, the Department of Defense (DOD) owns and operates domestic and overseas infrastructure with an estimated replacement value of about $1 trillion. In September 2018, Hurricane Florence damaged Camp Lejeune and other Marine Corps facilities in North Carolina, resulting in a preliminary Marine Corps repair estimate of $3.6 billion. One month later, Hurricane Michael devastated Tyndall Air Force Base in Florida, resulting in a preliminary Air Force repair estimate of $3 billion and upwards of 5 years to complete the work. In addition, we recently reported that the federal government manages about 650 million acres of land in the United States that could be vulnerable to climate change, including the possibility of more frequent and severe droughts and wildfires. Appropriations for federal wildland fire management activities have increased considerably since the 1990s, as we and the Congressional Research Service have reported. As we reported in October 2019, our past work shows an absence of government-wide strategic planning for climate change. Specifically, our past work identifies limitations related to strategic planning for climate change that include a lack of coordination, prioritization, and consolidation of strategic priorities. For example, we reported in October 2009 that the federal government’s emerging climate resilience activities were carried out in an ad hoc manner and were not well coordinated across federal agencies. In May 2011, we reported that federal officials did not have a shared understanding of strategic government-wide priorities related to climate change. In the same report, we found that there was not a consolidated set of strategic priorities integrating climate change programs and activities across the federal government. In our March 2019 High-Risk Update, we reported that one area of government-wide action needed to reduce federal fiscal exposure is in the federal government’s role as the leader of a strategic plan that coordinates federal efforts and informs state, local, and private sector action. For our 2019 High-Risk Update, we assessed the federal government’s progress since 2017 related to climate change strategic planning against five criteria and found that the federal government had not met any of the criteria for removal from the high-risk list. Specifically, since our 2017 high-risk update, four ratings regressed to “not met” and one remained unchanged as “not met.” (See fig. 2.) We have made 62 recommendations related to the climate change high-risk area, 17 of which address improving federal climate change strategic planning. As of August 2019, no action had been taken toward 14 of those 17 recommendations—one dating back to 2003. Although the federal government faces fiscal exposure to climate change, its investments in resilience to climate change impacts have been limited. One way to reduce federal fiscal exposure is to enhance resilience by reducing or eliminating long-term risk to people and property from natural hazards. For example, in September 2018 we reported that elevated homes and strengthened building codes in Texas and Florida prevented greater damages during the 2017 hurricane season. In addition, one company participating in a 2014 forum we held on preparing for climate- related risks noted that for every dollar it invested in resilience efforts, the company could prevent $5 in potential losses. Finally, a 2018 interim report by the National Institute of Building Sciences examined a sample of federal grants for hazard mitigation. The interim report estimated approximate benefits to society (i.e., homeowners and communities) in excess of costs for several types of resilience projects through the protection of lives and property, and prevention of other losses, though precise benefits are uncertain. According to the interim report, for every grant dollar the federal government spent on resilience projects, over time, society is estimated to accrue benefits amounting to the following: About $3 on average from projects addressing the effects of fire in the wildland urban interface, with most benefits (approximately 70 percent) coming from the protection of property (i.e., avoiding property losses). About $5 on average from projects to address hurricane-force and tornado-force winds, with most benefits (approximately 90 percent) coming from the protection of lives. This includes avoiding deaths, nonfatal injuries, and causes of posttraumatic stress. About $7 on average from projects that buy out buildings prone to riverine flooding, with most benefits (approximately 65 percent) coming from the protection of property. The interim report also projected that society could accrue benefits amounting to about $11 on average for every dollar invested in designing new buildings to meet the 2018 International Building Code and the 2018 International Residential Code—the model building codes that the International Code Council developed—with most benefits (46 percent) coming from the protection of property. We reported in October 2009 that the federal government’s activities to build resilience to climate change were carried out in an ad hoc manner and were not well coordinated across federal agencies. We reported similar findings in October 2019. Federal agencies have included some of these activities within existing programs and operations—a concept known as mainstreaming. For example, the Fourth National Climate Assessment reported that the U.S. military integrates climate risks into its analysis, plans, and programs, with particular attention paid to climate effects on force readiness, military bases, and training ranges. However, according to the Fourth National Climate Assessment, while a significant portion of climate risk can be addressed by mainstreaming, the practice may reduce the visibility of climate resilience relative to dedicated, stand-alone approaches and may prove insufficient to address the full range of climate risks. In addition, as we reported in March 2019, the Disaster Recovery Reform Act of 2018 (DRRA) was enacted in October 2018 and could improve state and local resilience to disasters. DRRA, among other things, allows the President to set aside, with respect to each major disaster, a percentage of the estimated aggregate amount of certain grants to use for predisaster hazard mitigation and makes federal assistance available to state and local governments for building code administration and enforcement. However, it is too early to tell what impact implementing the act will have on state and local resilience. The federal government has made some limited investments in resilience, and DRRA could enable additional improvements at the state and local levels. However, we reported in October 2019 that the federal government does not have a strategic approach for investing in climate resilience projects—that is, an intentional, crosscutting approach in which the federal government identifies and prioritizes projects for the purpose of enhancing climate resilience. Federal agencies may take actions to invest in projects with potential climate resilience benefits related to their own mission areas using funds from federal programs designed for other purposes. In addition, the National Climate Assessment provides high- level information on what is known about observed and projected climate risks in the United States. However, no federal entity looks holistically at the federal government’s investments to strategically prioritize projects to ensure that they address the nation’s most significant climate risks and provide the highest net benefits relative to other potential projects. Further, we reported in September 2017 that the federal government had not undertaken strategic government-wide planning to manage significant climate risks before they become fiscal exposures. As an initial step in managing climate risks, most of the experts we interviewed for the September 2017 report told us that federal decision makers should prioritize risk management efforts on significant climate risks that create the greatest fiscal exposure. Moreover, several stakeholders told us that the federal government’s emphasis has been on funding postdisaster efforts instead of funding resilience projects before a disaster occurs. This is consistent with findings from our July 2015 report that most federal funding for hazard mitigation is only available after a disaster. In addition, according to FEMA officials, some of the agency’s hazard mitigation programs are designed to empower state and local governments to determine their mitigation funding priorities, and these state and local priorities may or may not align with the federal interest. Finally, although we did not identify a government-wide strategic approach specifically for investing in climate resilience projects, the National Mitigation Investment Strategy—a national effort under way to plan for predisaster resilience investments—represents a potential cross- agency vehicle for climate resilience planning. However, the strategy does not specifically address climate change or identify and prioritize specific climate resilience projects. As we reported in March 2019, the federal government could reduce its fiscal exposure to climate change by focusing and coordinating federal efforts. However, the federal government is currently not well organized to address the fiscal exposure presented by climate change, partly because of the inherently complicated and crosscutting nature of the issue. We have made a total of 62 recommendations related to limiting the federal government’s fiscal exposure to climate change over the years, 12 of which have been made since February 2017. As of December 2018, 25 of these recommendations remained open. In describing what needs to be done to reduce federal fiscal exposure to climate change, our March 2019 High-Risk Report discusses many of the open recommendations.Implementing these recommendations could help reduce federal fiscal exposure. Several of them, including those highlighted below, identify key government-wide efforts needed to help plan for and manage climate risks and direct federal efforts toward common goals, such as improving resilience. Develop a national strategic plan: In May 2011, we recommended that appropriate entities within the Executive Office of the President (EOP), including the Office of Management and Budget, work with agencies and interagency coordinating bodies to establish federal strategic climate change priorities that reflect the full range of climate- related federal activities, including roles and responsibilities of key federal entities. Use economic information to identify and respond to significant climate risks: In September 2017, we recommended that the appropriate entities within EOP use information on the potential economic effects of climate change to help identify significant climate risks facing the federal government and craft appropriate federal responses. Such federal responses could include establishing a strategy to identify, prioritize, and guide federal investments to enhance resilience against future disasters. Provide decision makers with the best-available climate information: In November 2015, we reported that federal efforts to provide information about climate change impacts did not fully meet the climate information needs of federal, state, local, and private sector decision makers, which hindered their efforts to plan for climate change risks. We reported that these decision makers would benefit from a national climate information system that would develop and update authoritative climate observations and projections specifically for use in decision-making. As a result, we recommended that EOP (1) designate a federal entity to develop and periodically update a set of authoritative climate observations and projections for use in federal decision-making, which other decision makers could also access, and (2) designate a federal entity to create a national climate information system with defined roles for federal agencies and nonfederal entities with existing statutory authority. Consider climate information in design standards: In November 2016, we reported that design standards, building codes, and voluntary certifications established by standards-developing organizations play a role in ensuring the resilience of infrastructure to the effects of natural disasters. However, we reported that these organizations faced challenges in using forward-looking climate information that could help enhance the resilience of infrastructure. As a result, we recommended in the November 2016 report that the Department of Commerce (Commerce), acting through the National Institute of Standards and Technology—which is responsible for coordinating federal participation in standards organizations— convene federal agencies for an ongoing government-wide effort to provide the best-available forward-looking climate information to standards-developing organizations for their consideration in the development of design standards, building codes, and voluntary certifications. In addition, in October 2019, we recommended that Congress consider establishing a federal organizational arrangement to periodically identify and prioritize climate resilience projects for federal investment. We also identified six key steps the federal government could use to prioritize climate resilience investments and opportunities to increase the climate resilience impacts of federal funding options that Congress could use in designing the arrangement. In October 2019 we also issued the Disaster Resilience Framework to serve as a guide for analysis of federal action to facilitate and promote resilience to natural disasters. The framework identifies three key principles that can help federal efforts to promote disaster resilience, including building resilience to climate change. First, authoritative and understandable information can help decision makers identify current and future risks and the impact of risk-reduction strategies. Second, integrated analysis and strategic planning can help decision makers take coherent and coordinated resilience actions. Third, financial and nonfinancial incentives can help make long-term, forward-looking risk-reduction investments more viable and attractive among competing priorities. In conclusion, the effects of climate change have already posed and will continue to pose risks that can create fiscal exposure across the federal government, and this exposure will continue to increase. The federal government does not generally account for such fiscal exposure to programs in the budget process, and it has not undertaken strategic efforts to manage significant climate risks that could reduce the need for far more costly steps in the decades to come. To reduce its fiscal exposure, the federal government needs a cohesive strategic approach with strong leadership and the authority to manage risks across the entire range of related federal activities. The federal government could make further progress toward reducing fiscal exposure by implementing the recommendations we have made. Chairman Rouda, Ranking Member Comer, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact J. Alfredo Gómez, Director, Natural Resources and Environment, at (202) 512-3841or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Joseph Dean Thompson (Assistant Director), Micah McMillan (Analyst in Charge), Holly Halifax, Caitlin Jackson, Richard Johnson, Joe Maher, Oliver Richard, and Kiki Theodoropoulos. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Since 2005, federal funding for disaster assistance is at least $450 billion, including approximately $19.1 billion in supplemental appropriations signed into law on June 6, 2019. In 2018 alone, there were 14 separate billion-dollar weather and climate disaster events across the United States, with a total cost of at least $91 billion, according to the National Oceanic and Atmospheric Administration. The U.S. Global Change Research Program projects that disaster costs will likely increase as certain extreme weather events become more frequent and intense due to climate change. The costs of recent weather disasters have illustrated the need for planning for climate change risks and investing in resilience. Resilience is the ability to prepare and plan for, absorb, recover from, and more successfully adapt to adverse events, according to the National Academies of Science, Engineering, and Medicine. Investing in resilience can reduce the need for far more costly steps in the decades to come. Since February 2013, GAO has included Limiting the Federal Government's Fiscal Exposure by Better Managing Climate Change Risks on its list of federal program areas at high risk of vulnerabilities to fraud, waste, abuse, and mismanagement or most in need of transformation. GAO updates this list every 2 years. In March 2019, GAO reported that the federal government had not made measurable progress since 2017 to reduce fiscal exposure to climate change. This testimony—based on reports GAO issued from October 2009 to October 2019—discusses 1) what is known about the potential economic effects of climate change in the United States and the extent to which this information could help federal decision makers manage climate risks across the federal government, (2) the fiscal exposure facing the federal government due to climate risks and current efforts to address that exposure, (3) the extent to which the federal government has invested in resilience to climate change impacts, and (4) how the federal government could reduce fiscal exposure to the effects of climate change. GAO had made 62 recommendations related to the Limiting the Federal Government’s Fiscal Exposure by Better Managing Climate Change Risks high-risk area. As of December 2018, 25 of those recommendations remained open. The estimated economic effects of climate change, while imprecise, can convey useful insight about potential damages in the United States. In September 2017, GAO reported that the potential economic effects of climate change could be significant and unevenly distributed across sectors and regions (see figure). This is consistent with the 2018 findings of the U.S. Global Change Research Program's Fourth National Climate Assessment, which concluded, among other things, that the continued increase in the frequency and extent of high-tide flooding due to sea level rise threatens America's trillion-dollar coastal infrastructure. Information about the potential economic effects of climate change could inform decision makers about significant potential damages in different U.S. sectors or regions. According to prior GAO work, this information could help decision makers identify significant climate risks as an initial step toward managing them. The federal government faces fiscal exposure from climate change risks in several areas, including: Disaster aid: due to the rising number of natural disasters and increasing reliance on federal assistance. GAO has previously reported that the federal government's fragmented and reactive approach to funding disaster resilience presented challenges to effective reduction of climate-related risks. GAO has also reported that, due to an artificially low indicator for determining a jurisdiction's ability to respond to disasters that was set in 1986, the Federal Emergency Management Agency risks recommending federal assistance for jurisdictions that could recover on their own. Federal insurance for property and crops: due, in part, to the vulnerability of insured property and crops to climate change impacts. Federal flood and crop insurance programs were not designed to generate sufficient funds to fully cover all losses and expenses. The flood insurance program, for example, was about $21 billion in debt to the Treasury as of April 2019. Further, the Congressional Budget Office estimated in May 2019 that federal crop insurance would cost the federal government an average of about $8 billion annually from 2019 through 2029. Operation and management of federal property and lands: due to the hundreds of thousands of federal facilities and millions of acres of land that could be affected by a changing climate and more frequent extreme events. For example, in 2018, Hurricane Michael devastated Tyndall Air Force Base in Florida, with a preliminary repair estimate of $3 billion. As we reported in October 2019, our past work shows an absence of government-wide strategic planning for climate change. Specifically, our past work has identified limitations related to strategic planning for climate change that includes a lack of coordination, prioritization, and consolidation of strategic priorities. In our March 2019 High-Risk Update, we assessed the federal government's progress since 2017 related to climate change strategic planning against five criteria and found that the federal government had not met any of the criteria for removal from the high-risk list. Federal investments in resilience to reduce fiscal exposures have been limited. As GAO has reported, enhancing resilience can reduce fiscal exposure by reducing or eliminating long-term risk to people and property from natural hazards. For example, a 2018 interim report by the National Institute of Building Sciences estimated approximate benefits to society in excess of costs for several types of resilience projects. While precise benefits are uncertain, the report estimated that for every dollar invested in designing new buildings to particular design standards, society could accrue benefits amounting to about $11 on average. GAO's March 2019 High-Risk report identified a number of recommendations GAO has made related to fiscal exposure to climate change. The federal government could reduce its fiscal exposure by implementing these recommendations. Among GAO's key government-wide recommendations are: Entities within the Executive Office of the President (EOP) should work with partners to establish federal strategic climate change priorities that reflect the full range of climate-related federal activities; Entities within EOP should use information on potential economic effects from climate change to help identify significant climate risks and craft appropriate federal responses; Entities within EOP should designate a federal entity to develop and update a set of authoritative climate observations and projections for use in federal decision making, and create a national climate information system with defined roles for federal agencies and certain nonfederal entities; and The Department of Commerce should convene federal agencies to provide the best-available forward-looking climate information to organizations that develop standards and building codes to enhance infrastructure resilience. Further, in October 2019, GAO reported that Congress could consider establishing a federal organizational arrangement to periodically identify and prioritize climate resilience projects for federal investment. GAO also issued the Disaster Resilience Framework to serve as a guide for analysis of federal action to facilitate and promote resilience to natural disasters, including resilience to climate change.", "document_type": "gao"}
{"report": "Reducing transportation-related fatalities and serious injuries has consistently been DOT’s top priority. Traffic fatalities and serious injuries may result from unsafe driver behaviors, such as speeding and alcohol- or drug-impaired driving, or from the design or condition of the road and its accompanying infrastructure. Within DOT, both NHTSA and FHWA are charged with reducing fatalities and serious injuries on the nation’s highways and, respectively, provide grant funding to states to mitigate the behavioral and infrastructure-related causes of vehicular crashes. NHTSA provided over $600 million in fiscal year 2018 to state highway safety offices through the Highway Safety Grants Program for activities designed to improve traffic safety by modifying driver behavior. For example, states may use NHTSA grant funding for efforts to increase seatbelt use, or to reduce impaired driving. FHWA provided about $2.6 billion in fiscal year 2018 to state departments of transportation through the Highway Safety Improvement Program (HSIP) for projects to improve safety on all public roads. HSIP funds can be used for infrastructure projects, such as rumble strips, and other projects such as road safety audits, safety planning, and improving safety data. States are allowed to transfer up to 50 percent of their HSIP safety apportionment made available each fiscal year to the other core FHWA highway programs. For example, from 2013 through 2018, 24 states transferred HSIP safety funding totaling over $1 billion to other core programs and three states transferred approximately $600 million into their HSIP safety program from other core programs. Over the last decade, the federal government has taken steps to move toward a performance-based framework for traffic safety funding. Historically, most federal surface transportation funds were distributed through formulas that often had no relationship to outcomes or grantees’ performance. In 2008, we recommended that Congress consider integrating performance-based principles into surface transportation programs such as NHTSA’s Highway Safety Grants Program and FHWA’s HSIP to improve performance and accountability in states’ use of federal funds. In particular, we noted that tracking specific outcomes that are clearly linked to program goals can provide a strong foundation for holding grant recipients responsible for achieving federal objectives and measuring overall program performance. The Moving Ahead for Progress in the 21st Century Act, enacted in 2012, formally required the Secretary of the Department of Transportation to, among other things, establish performance measures for states to use to assess fatalities and serious injuries to ensure further accountability for federal traffic safety funding provided to states. See table 1 for a complete list of mandatory performance measures. States are also required to establish targets annually for each of the performance measures and measure progress toward these targets. NHTSA first required states to develop targets for their performance measures as part of their planning for fiscal year 2014, and FHWA first required states to establish targets for their performance measures set in 2017 for calendar year 2018. Starting with these targets, state highway safety offices and departments of transportation were required by both NHTSA and FHWA to set identical targets for the three common performance measures in both frameworks. Both NHTSA’s and FHWA’s frameworks provide flexibility to states in how they may establish targets and emphasize using data to develop realistic and achievable targets rather than aspirational ones that reflect a long-term vision for future performance. Because the frameworks do not require a specific reduction in fatalities or serious injuries, states may set targets that are higher or lower than their historical averages depending on state-specific factors, such as population increases or economic conditions. As a result, targets may reflect either an anticipated increase or decrease in fatalities or serious injuries. NHTSA and FHWA require states to submit annual plans and reports to establish targets and describe their use of federal funds to improve safety and the results they have achieved relative to their targets. (See table 2.) NHTSA requires that states submit an annual Highway Safety Plan to, among other things, set targets, identify projects they will implement in the upcoming fiscal year, and describe how they will use funds from the Highway Safety Grants Program. States are also required to submit an Annual Report to NHTSA that includes an assessment of the state’s progress in achieving safety performance targets in the previous fiscal year. States are required to submit an HSIP report to FHWA that describes, among other things, how they have used federal HSIP funding for highway safety improvement projects during the prior reporting period as well as performance targets for the upcoming calendar year. In addition to the annual requirements, FHWA requires a Strategic Highway Safety Plan from states every 5 years that identifies a state’s key safety needs and long-term goals, and guides investment decisions to reduce fatalities and serious injuries. NHTSA and FHWA rely on states and localities to collect and report fatality and serious injury data used in the performance framework. In addition to providing information through annual plans and reports, states report traffic fatalities to NHTSA’s FARS database, which tracks all fatal traffic crashes nationwide. When a fatal crash occurs, a state or local police officer completes a crash report form unique to each state. These forms can include a variety of data fields, such as the time of the crash, weather conditions, and the number of killed or injured persons. FARS analysts—state employees who are trained by NHTSA’s data validation and training contractors—use the data in crash report forms to compile a record of the fatal crash. However, NHTSA’s collection and validation of these data may take up to 24 months following the end of a calendar year before it is finalized. FARS also contains serious injury data associated with fatal crashes, though neither NHTSA nor FHWA maintain a database of all serious injuries. Rather, the agencies rely on states and localities to collect and store records of serious injuries resulting from traffic crashes and report this information to them each year. Based on data the states and localities provide, NHTSA estimates the number of total injuries resulting from crashes to track overall national trends. From 2014 through 2017, states did not achieve about two-thirds of the targets they set for the required fatality performance measures, according to our analysis of state-reported NHTSA data. In addition, for a majority of the fatality performance measures required by NHTSA, these data show that the number of targets states achieved generally decreased from 2014 through 2017. (See table 3.) Over this same time, fatalities increased nationwide by 13 percent from about 33,000 in 2014 to over 37,000 in 2017. NHTSA officials said that fewer states achieved their targets over this time because fatalities increased nationwide over the same period due to increases in vehicle miles traveled and corresponding exposure to driving-related risks. Officials from the 10 states we selected said that achieving targets often depends on factors outside of their control, such as demographic and economic factors, as well as changes to state laws. Demographic factors. Officials from eight of the 10 selected states said that demographic factors such as increases or decreases in population affect traffic safety. For example, officials from one state said that when companies expanded in the state, the population increased rapidly and the economy improved and led to more driving. Officials from another state noted that the increasing population in the state’s urban areas has increased the number of pedestrian fatalities. Economic factors. Officials from seven of the 10 selected states noted that economic factors such as low unemployment can affect traffic safety. For example, officials in one state said that fatalities decreased during the 2009 recession, but when the economy began to improve and more people were employed, fatalities increased. These officials noted that the number of people driving is also affected by gas prices because when prices increase, people drive less. Changes to state laws. Officials from eight of the 10 selected states said that changes in state laws can affect whether a state meets its targets. For example, officials from one state said fatalities increased beginning in 2012 when the state legislature passed a law allowing the operation of a motorcycle without a helmet, and continued to increase through 2017 when the state legislature increased the speed limit on some roads from 70 to 75 miles per hour. These officials also noted that they expect fatalities in their state to further increase as a result of the recent legalization of the recreational use of marijuana. However, the extent to which states achieve targets does not necessarily reflect whether the number of fatalities has increased or decreased over time. First, states that achieved fatality targets did not necessarily experience reduced traffic fatalities. For example, for the 2017 targets, state-reported NHTSA data shows that 10 of 52 states achieved their target for the pedestrian fatalities performance measure, but five of these 10 states also experienced an increase in pedestrian fatalities compared to their 2012 through 2016 historical average. These data also show that the remaining 42 states did not achieve their total fatality target. Second, some states have experienced a decrease in traffic fatalities while not achieving their targets. For example, state-reported NHTSA data shows that 31 states did not achieve their targets for the speeding-related fatalities performance measure. However, these same data show that 11 of these 31 states decreased the total of number of these fatalities over their 2017 target period compared to their 2012 to 2016 average. Further, states that established targets that represented an increase in fatalities from historical averages (increasing targets) were more likely to achieve them than states that established targets that represented a decrease or no change in fatalities compared to their historical averages (decreasing targets), according to state-reported NHTSA data. Specifically, in 2017, for all of the required fatality performance measures, these data show that states that set increasing fatality targets relative to their historical 2012 to 2016 average achieved them at a higher rate than states that set targets that represented a decrease or no change to the number of fatalities (See fig.1.) For example, for the total fatality performance measure, eight states set increasing targets relative to their historical 2012 to 2016 average, while 44 states set decreasing or unchanged targets relative to their averages. However, these data show that six of the eight states with increasing targets for the total fatalities performance measure achieved them, while only three of the 44 states with decreasing or unchanged targets achieved theirs. In response to statute, NHTSA requires states to assess and report progress in achieving targets in the following year’s Highway Safety Plan and the NHTSA Annual Reports each year. Such an approach is consistent with federal standards for internal control, which state that agencies should communicate quality information, including about activities and achievements. According to NHTSA officials, state evaluations of their progress in these plans and reports are designed to be an interim assessment of a state’s progress. For example, because fatality data can take up to 2 years to be recorded by states in FARS and validated by NHTSA, final FARS data are not available when states are required to report on the achievement of the prior fiscal year’s targets in their Highway Safety Plans. Therefore, NHTSA encourages states to use state data to conduct this assessment or provide a qualitative analysis of the progress made in achieving these targets when FARS data are not available. Upon review of these reports, NHTSA publishes them on its website. While NHTSA has established requirements for states to provide assessments of their progress on achieving the prior year targets in their Highway Safety Plans and Annual Reports, we found that many states have not done so. For example, in the 2019 Highway Safety Plans submitted to NHTSA in July 2018, a third of states (19 of 52) did not provide an assessment of the progress they had made in achieving the fatality targets established in their 2018 Highway Safety Plans. Similarly, in the 2018 Annual Reports, submitted to NHTSA in December 2018, half of states (26 of 52) did not provide an assessment of whether they had made progress toward achieving the fatality targets established in their 2018 Highway Safety Plans. Instead, many of these states assessed progress for an earlier year or performance period. NHTSA officials acknowledged that some states are not clear on which target years to assess in their Highway Safety Plans and Annual Reports. NHTSA officials stated that they work closely with states to review the contents of the Highway Safety Plans and Annual Reports. To do so, NHTSA has developed guides to help its staff review Highway Plans and the Annual Reports to ensure states meet requirements to provide assessments of their progress. NHTSA officials stated they expect most states to comply with the requirements to assess progress in future Annual Reports and Highway Safety Plans because states will be more familiar with the reporting requirements. However, NHTSA has had similar requirements for states to provide in-progress assessments in these documents for a number of years. For example, the requirement to report on progress achieving highway safety performance measure targets identified in the Highway Safety Plans in the Annual Report was introduced in 2013. Similarly, NHTSA’s regulations have also required states to include an assessment of their progress in meeting state performance targets in their Highway Safety Plans since 2013. Without additional clarification from NHTSA to states on which target years to assess in their Highway Safety Plans and Annual Reports, NHTSA and other stakeholders may lack a timely understanding of the progress states have made in achieving their targets. NHSTA could provide such clarification through outreach to states, or by providing guidance on NHTSA’s website. Beyond the required interim state assessments of progress contained in the Annual Reports and Highway Safety Plans, NHTSA does not communicate to the public and other stakeholders about whether states eventually achieve their fatality targets. Federal standards for internal control state that agencies should communicate quality information, including about activities and achievements, so that external parties–such as Congress and other stakeholders–can help realize agency goals and objectives. NHTSA officials said that they have reported on states’ achievement of fatality targets in the past. For example, NHTSA previously reported to Congress in 2017 on states’ achievement of the fatality targets established in the 2014 and 2015 Highway Safety Plans in response to a statutory requirement. However, NHTSA did not provide this report to other stakeholders, and it has not subsequently reported to Congress or the general public on whether states achieved targets. NHTSA officials told us they did not have any plans to develop a similar report in the future because the requirement to report to Congress was repealed in January 2019. NHTSA was directed by statute in January 2019 to provide information on its website on state performance relative to the targets in the Highway Safety Plan. The statute broadly directs NHTSA to report on state performance and does not specifically direct NHTSA to communicate whether states eventually achieve their performance targets. NHTSA officials told us that this effort was in its initial stages and NHTSA is still in the process of determining how to meet the statutory requirement. By improving external communication of states’ achievement of fatality targets, NHTSA could give stakeholders better insight into the results states and NHTSA have achieved in their efforts to reduce fatalities and hold states more accountable for their use of federal safety funds. NHTSA could provide such information to all stakeholders through its planned website or by developing an alternative mechanism to convey this information. We were not able to determine the extent to which states achieved NHTSA serious injury targets from 2014 through 2017 because states’ definitions of “serious injury” have changed over time. As a result, state serious injury data used to set targets and analyze results may not be comparable year to year over this time period. NHTSA officials noted that changes to serious injury definitions can affect the total number of serious injuries recorded by the states. Similarly, officials from the Association of Transportation Safety Information Professionals told us that based on their experience, when there is a change to how serious injury data are defined or collected by states, total serious injury numbers in that state may change by up to 15 percent the following year. In some cases, changes to serious injury totals may be more extensive. For example, in 2016, one state changed its definition as part of implementing a new database to store crash records. After this change, the number of serious injuries nearly doubled from the previous year. NHTSA and FHWA have taken steps to standardize how states define and report serious injury data. In 2016, both FHWA and NHTSA set out requirements for all states to use a specific definition of serious injury by April 15, 2019, establishing a single national standard definition that will be used under both NHTSA’s and FHWA’s performance management framework. This standard includes requirements for states to integrate this definition into their practices for collecting and recording serious injury data. According to NHTSA and FHWA, this standard will ensure consistent, coordinated, and comparable data at the state and national levels and will assist stakeholders in addressing highway safety challenges. Moreover, according to officials from the Association of Transportation Safety Information Professionals, adoption of this standard will be an improvement upon the previous approaches used by states to define serious injuries. However, it will take time for states to adopt this standard and collect consistent data under the new national standard for serious injuries to use in the NHTSA’s and FHWA’s performance management frameworks. First, NHTSA’s and FHWA’s regulations require that states establish 5-year averages for serious injury targets; however, according to states’ most recent reporting, many states have only recently adopted NHTSA and FHWA’s national standard for defining serious injuries. Specifically, based on our review of information submitted by states in their 2018 HSIP reports, we found that 18 states had reported that they were fully compliant with the national standard as of the end of August 2018. FHWA officials told us that, based on their review of the information in the 2018 HSIP reports, they estimated that an additional 22 states planned to fully align their serious injury definition with requirements in the national standard by April 2019, and that the remaining 12 states had not indicated if they would be compliant with the national standard by that time. FHWA officials said they would conduct a compliance assessment in fall 2019 to determine whether states fully adopted the national standard. Second, data collected under previous, differing definitions cannot be retroactively converted to equivalent data under the definition established by the national standard, and thus it will take time to develop a consistently defined set of serious injury data. Specifically, for those states that have adopted the new standard in the last year, it may be 4 to 5 years until a 5-year average of serious injury data under the new standard can be reported, while the transition period may be longer for those states that have yet to adopt the standard. For example, the American Association of State Highway and Transportation Officials noted that if a state was not currently using the national standard, it would take a lengthy and resource-intensive effort to adopt the standard, including changing reporting processes, guidance, and training. State officials we interviewed also said the costs of updating software and paper forms to collect and store serious injury information, and of training state officials to collect serious injury data using the national standard, could further delay implementation. NHTSA and FHWA have taken steps to assist states with the transition to the new national standard for serious injuries. For example, in preparation for issuing the regulations, NHTSA and FHWA published state-specific guidance to help states adopt an interim standard before the national standard took effect in 2019. According to NHTSA and FHWA officials, this guidance, which aligned states’ existing definitions with a scale for injury severity, helped states provide more consistent serious injury statistics prior to implementing the new national standard in the FHWA rulemaking. While this interim standard helps improve consistency of the definition of serious injury within a state, it does not standardize the specific definition across all states as does the new national standard. In addition, NHTSA and FHWA developed an outreach program and training to help states adapt to the new requirement prior to implementation in 2019. While the transition occurs and until states have collected 5 years of data under the new national standard for serious injuries, NHTSA and FHWA plan to take different approaches to assessing states’ progress toward serious injury targets and communicating the results of their assessments. NHTSA officials told us that they would wait to assess progress until the states had adopted a consistent set of data under the national standard for serious injuries. NHTSA officials also noted that they did not assess whether states achieved their serious injury targets in NHTSA’s 2015 and 2017 reports to Congress, because of limitations with the data that the new standard seeks to mitigate. However, once the transition to the new national standard for serious injuries is complete, similar to state fatality targets, NHTSA does not have a formal mechanism for communicating whether states eventually achieve their serious injury targets. Communication of states’ achievement of both fatality and serious injury targets could help NHTSA hold states more accountable for their use of federal funds. In contrast, as directed by statute and regulations, FHWA plans to evaluate whether each state has met or made “significant progress” toward meeting both the fatality and serious injury-related targets by improving upon the state’s historical 5-year baseline for four of the five required performance measures. As directed by statute and FHWA’s regulations, states that FHWA determines either have not met their 2018 targets or not made significant progress are required to develop an implementation plan to describe how they will achieve targets in future years. Further, these states must use a portion of these states’ fiscal year 2021 HSIP funding exclusively for HSIP projects and may not transfer this portion of their HSIP funding to other core highway programs. Once FHWA’s evaluation of state progress is complete, it plans to communicate the extent to which states achieve these targets on its website, which contains information on the 5-year averages that make up the baseline, targets, and results, and tracks this information over time. FHWA officials said that, as states transition to the new national standard for serious injuries, the use of data collected under multiple definitions in a state may occur in future assessments of significant progress as states collect 5 years of data under the national standard. However, FHWA officials said that states will be able to take the limitations in the data into consideration and adjust targets each year as needed to minimize the risk that states’ results will vary significantly from their targets. An official from the Association of Transportation Safety Information Professionals said that he expects states may recalculate targets to account for changes in the data over the transition to the national standard for serious injuries, but that states have not expressed concerns about doing so. More broadly, FHWA officials also stated that modifying its approach for the transition period would require additional rulemakings by both FHWA and NHTSA, which could be a lengthy process and thus may not be completed before most states collect 5 years of data under the new standard. Officials from a majority of the states we surveyed reported that the performance measures and targets in the NHTSA framework influenced which projects they selected to fund to improve traffic safety and reduce fatalities and serious injuries. (See fig. 2.) For example, officials from two states we surveyed reported that the performance measures helped them identify emerging traffic safety trends, such as higher rates of speeding; as a result, the states directed more funding to projects addressing those issues. Officials from another state noted that the performance measures have led them to develop new projects to reduce cyclist and pedestrian fatalities, in addition to their traditional projects targeting impaired driving or seat belt use. In addition, other state officials responded that setting targets influenced their project selection by requiring staff to identify and fund projects that would have a positive effect on the targets established. When NHTSA developed the performance measures for states, it noted that, in addition to helping states monitor and evaluate their progress, performance measures can be used to allocate resources towards the most pressing safety issues. Officials from 19 states we surveyed said that the performance measures in the NHTSA framework did not influence their project selection. Similarly, officials from 23 states said the targets did not influence their project selection. Officials we surveyed cited a variety of reasons for why they did not use this performance information to select projects. For example, officials from three of these states said their states already had a data-driven or performance-based approach to project selection. Officials from one state explained that the NHTSA performance measures provide them with a general overview of safety trends in the state, but that they rely on more detailed data analysis of safety trends in different localities to select projects. Officials from another state said they do not use the specific targets to select projects, because they look for ways to decrease fatalities, not to achieve a specific number of fatalities in a given year. Officials from another state explained that they receive limited safety funding and therefore select projects to make sure they are eligible to qualify for NHTSA grants. NHTSA officials acknowledged that the performance management framework can pose challenges for some states, but noted that they provide technical assistance and guidance to help states make the best use of their performance information. State officials reported other safety benefits from NHTSA’s performance framework in addition to improved project selection. Specifically, officials from almost three-quarters of states we surveyed said the NHTSA framework helped them to improve highway safety in their state. For example, officials from five states we surveyed reported that the framework has improved how they identify highway safety problems, such as by formalizing a data-driven approach to highway safety in their state. Officials we surveyed also noted that by requiring states to reach agreement on some NHTSA and FHWA targets, the framework helped them to increase collaboration with other highway safety stakeholders in the state. For example, officials from one state reported that the collaboration between the state department of transportation and highway safety office has increased their awareness of how physical road improvements and behavioral projects can work together to improve safety in the state. Officials from the 14 states who reported that the framework has not helped them improve safety cited various reasons, including that they used data-driven approaches prior to NHTSA’s framework and that the framework has increased their administrative burden. NHTSA officials agreed that the framework imposed some administrative burdens on states, but stated that the benefits of using a performance-based approach to manage state highway safety programs outweighed any costs for states. To ensure that the framework helps states to improve traffic safety, NHTSA regulations require states to include at least one performance measure (and associated target) for each program area contained in their Highway Safety Plans. These requirements are consistent with federal standards for internal control that agencies should establish and operate activities to monitor the internal control system. Such monitoring activities should be built into the agency’s operation. We found 49 states included performance measures with all the program areas in their 2019 Highway Safety Plans. For example, one state uses the number of motorcyclist fatalities and unhelmeted motorcyclist fatalities as performance measures for its motorcycle safety program area. The remaining three states included performance measures for at least 80 percent of their program areas. By requiring states to establish performance measures for their program areas, NHTSA can help ensure states have appropriate performance measures in place to evaluate whether they are achieving the objectives of their highway safety programs. NHTSA’s regulations also require states to describe the linkage between the countermeasure strategies—the safety initiatives a state plans to fund to address highway safety problems—and the performance targets in their Highway Safety Plans. Requiring states to link their funding decisions with their targets aligns with a leading practice for performance management we have previously identified: that agencies should use performance information to allocate resources. We examined the sections of 2019 Highway Safety Plans where states are prompted to provide this linkage, and found, however, that less than a third of states (12 of 52) described all the linkages between their performance targets and the countermeasure strategies in those sections. NHTSA officials noted that states are directed to submit similar information in other locations throughout the plans, and that NHTSA’s review of the 2019 plans credited states with making these linkages by considering information in other sections of the plan. NHTSA has taken steps this year to improve states’ reporting and its own review of the 2020 Highway Safety Plans. For example, NHTSA officials told us that they have held in-person meetings with state highway safety officials to emphasize the need to provide linkages between their targets and countermeasures in their 2020 Highway Safety Plans. NHTSA officials said they have also held training in 2019 for staff who review these plans to ensure states adhere to reporting requirements. Specifically, during the training, NHTSA officials said they provided guidance to staff on reviewing Highway Safety Plans; this guidance prompts reviewers to check whether states link their countermeasure strategies with targets, and to provide feedback to states that have not provided these linkages. As a result of these actions, NHTSA anticipates that states will more clearly identify linkages in their 2020 plans. While states recently began setting performance measure targets under FHWA’s framework in 2017, officials from about a third of states we surveyed reported that performance measures in FHWA’s framework influenced their decisions about which infrastructure-based safety projects to fund. (See fig. 3.) Slightly fewer respondents said the targets they set influenced their project selection. These states reported that this performance information influenced their decision making in different ways. For example, officials from one state reported funding more pedestrian and bicycle safety projects as a result of the trends indicated by the performance measures. Officials from another state said they have shifted to selecting projects that can be constructed quickly in order to reach their annual safety targets. Officials from about two-thirds of states we surveyed said the performance measures and performance targets did not influence their HSIP project selection. Instead, many of these state officials reported that the FHWA performance framework has not changed their project selection methodology, and that they used alternative data-driven approaches to select highway projects. For example, officials from four states reported that they used their 5-year Strategic Highway Safety Plans, which highlight traffic safety issues to guide project selection. In other cases, state officials reported that they continued to use a data- driven approach, such as cost-benefit analysis or crash data analysis, to maximize safety benefits and select the most cost-effective highway safety projects. This approach is consistent with a recent FHWA survey of state departments of transportation, which reported that most states used their 5-year Strategic Highway Safety Plans and cost to prioritize projects. Federal guidelines, including those at FHWA, encourage the use of cost- benefit analysis for selecting infrastructure projects. We have also previously reported that such analysis can lead to better-informed transportation decisions. According to FHWA officials, performance management is not intended to supplant the use of other data-driven project selection methods, but to complement and be integrated into existing methods. To help further this synthesis, FHWA officials told us that they are developing a guide to better explain how states can incorporate the use of performance measures into existing methods, such as cost-benefit analysis, to select projects and achieve their safety targets. FHWA officials expect to issue this guide by January 2020. Overall, a slight majority of states we surveyed (27 of 52) reported that FHWA’s performance framework assisted them in improving safety. Officials cited safety benefits beyond improved project selection, such as increased awareness of highway safety issues for state leaders and the public; and increased collaboration with other highway safety agencies within the state. State officials who did not find the framework helpful cited various reasons. For example, some state officials we surveyed said they were already using performance measures prior to FHWA’s framework. Other officials surveyed said FHWA’s performance framework was not helpful because they have a “Vision Zero” or a “Toward Zero Deaths” policy in their state. According to these officials, under such a policy, the state’s goal is to achieve zero traffic fatalities. Officials from a state with such a policy explained that setting a target to achieve any fatalities was not acceptable to the public or the state because it suggests that not every life is important. FHWA officials said that setting annual targets, however, can ensure states are on track to reach their long-term goals, such as to reduce fatalities to zero. To encourage states to integrate the performance framework into their other safety plans, FHWA regulations require states to link their performance measure targets to the long-term goals in their 5-year Strategic Highway Safety Plans. States must provide a description in their HSIP reports of how each target supports these goals. FHWA has developed and issued a template for the HSIP report that prompts states to describe the link between their targets and their Strategic Highway Safety Plans’ goals. However, about half of the states did not describe how all of their targets support their Strategic Highway Safety Plans’ goals in their 2018 HSIP report, and thirteen of these states did not describe these linkages for any of their targets. In response to our analysis, FHWA officials have taken additional actions to improve states’ HSIP reporting. Specifically, FHWA officials provided training to staff and state officials that referenced our analysis that states did not describe the linkages between targets and long-term goals in their HSIP reports. During the training, FHWA officials emphasized the importance of including such information as states prepare their 2019 HSIP reports. Additionally, FHWA officials said they are updating the guide its staff uses to review HSIP reports to ensure states are describing how the targets they set support their Strategic Highway Safety Plan’s goals. In light of the large number of fatalities that occur each year on the nation’s highways and the billions of federal dollars DOT provides annually to states to improve traffic safety, the ability to assess the outcomes of federal surface transportation safety programs and hold grant recipients accountable for results is critical. NHTSA and FHWA have made great strides over the last decade in moving to a performance-based approach for traffic safety funding to improve accountability for federal funds. The results, however, that states have achieved under these frameworks are not always clear. For example, NHTSA has required states to report on their interim progress achieving targets, but states have not had clear direction on what results to assess. In addition, NHTSA lacks a formal mechanism to communicate whether states have been achieving the targets set under their framework. Without improved communication of progress, Congress will be limited in its ability to hold NHTSA and states accountable for their use of federal funds. Moreover, improved reporting of states’ achievements under NHTSA’s framework could help provide insight into the effectiveness of the overall federal traffic safety program. We are making two recommendations to NHTSA: The NHTSA Administrator should provide direction and clarification to states to ensure compliance with requirements to assess and report progress made in achieving fatality targets. (Recommendation 1) The NHTSA Administrator should develop and implement a mechanism that communicates to Congress and other stakeholders whether states achieve their fatality and serious injury targets. (Recommendation 2) We provided a draft of this report to DOT for comment. In its comments, reproduced in appendix III, DOT stated that it concurred with our recommendations. DOT also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Transportation, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Susan Fleming at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. The questions we asked in our survey of state Highway Safety Offices and the aggregate results of the responses to the closed-ended questions are shown below. Our survey was comprised of closed- and open-ended questions. We do not provide results for the open-ended questions. We sent surveys to 52 state highway safety offices about the National Highway and Traffic Safety Administration’s (NHTSA) performance framework from the 50 states, Puerto Rico and the District of Columbia. We received responses from 50 state highway safety offices, for a 96 percent response rate. For more information on our survey methodology, see page 4 of this report. Q1a. NHTSA has implemented a performance management framework that requires states to set targets for highway safety performance measures and to track their progress towards meeting those targets. Generally speaking, has NHTSA’s highway safety performance framework assisted you in improving highway safety in your state? Q1b. Why has NHTSA’s highway safety performance framework assisted or not assisted you in improving highway safety in your state? (Written responses not included.) Q2a. Each year, states use Highway Safety Plan (HSP) funding and select projects to address identified highway safety problems. How much, if at all, has NHTSA’s highway safety performance framework changed your state’s current approach to selecting HSP projects? Q2b. In what ways, if any, has NHTSA’s highway safety performance framework changed your state’s current approach to selecting HSP projects? (Written responses not included.) Q3a. Thinking about your state’s current HSP program, how much, if at all, did NHTSA’s required highway safety performance measures influence which projects your state selected? Q3b. In what ways, if any, have NHTSA’s required performance measures influenced which HSP projects your state selected? (Written responses not included.) Q4a. Thinking again about your state’s current HSP program, how much, if at all, did the specific targets your state set for NHTSA’s required performance measures influence which projects your state selected? Q4b. In what ways, if any, have the specific targets your state set for NHTSA’s required performance measures influenced which HSP projects your state selected? (Written responses not included.) The questions we asked in our survey of state departments of transportation and the aggregate results of the responses to the closed- ended questions are shown below. Our survey was comprised of closed- and open-ended questions. We do not provide results for the open-ended questions. We surveyed 52 state departments of transportation about the Federal Highway Administration’s (FHWA) performance framework from the 50 states, Puerto Rico and the District of Columbia. We received responses from all 52 state departments of transportation, for a 100 percent response rate. For more information on our survey methodology, see page 4 of this report. Q1a. FHWA has implemented a performance management framework that requires states to set targets for highway safety performance measures and to track their progress towards meeting those targets. Generally speaking, has FHWA’s highway safety performance framework assisted you in improving highway safety in your state? Q1b. Why has FHWA’s highway safety performance framework assisted or not assisted you in improving highway safety in your state? (Written responses not included.) Q2a. Each year, states use Highway Safety Improvement Program (HSIP) funding and select projects to address identified highway safety problems. How much, if at all, has FHWA’s highway safety performance framework changed your state’s current approach to selecting HSIP projects? Q2b. In what ways, if any, has FHWA’s highway safety performance framework changed your state’s current approach to selecting HSIP projects? (Written responses not included.) Q3a. Thinking about your state’s current HSIP program, how much, if at all, did FHWA’s required highway safety performance measures influence which projects your state selected? Q3b. In what ways, if any, have FHWA’s required performance measures influenced which HSIP projects your state selected? (Written responses not included.) Q4a. Thinking again about your state’s current HSIP program, how much, if at all, did the specific targets your state set for FHWA’s required performance measures influence which projects your state selected? Q4b. In what ways, if any, have the specific targets your state set for FHWA’s required performance measures influenced which HSIP projects your state selected? (Written responses not included.) In addition to the contact named above, Sara Vermillion (Assistant Director); Matt Voit (Analyst-in-Charge); Carl Barden; Caitlin Cusati; Timothy Guinane; Geoffrey Hamilton; Georgeann Higgins; Catrin Jones; Jesse Mitchell; Joshua Ormond; Kelly Rubin; and Laurel Voloder made key contributions to this report.", "summary": "Over 37,000 people were killed in traffic crashes on the nation's highways in 2017. Within the U.S. Department of Transportation (DOT), two agencies—NHTSA for behavioral factors and FHWA for highway infrastructure—provide about $3 billion annually to states for programs to improve traffic safety. To ensure that states are held accountable for these funds, NHTSA and FHWA developed performance management frameworks that require states to use performance measures and targets in tracking traffic fatalities and serious injuries. GAO was asked to review NHTSA's and FHWA's traffic safety performance management frameworks. This report examines the extent to which: (1) states have met fatality and serious injury targets, and NHTSA's and FHWA's approaches to assessing states' achievements, and (2) states have used performance measures and targets to make traffic safety funding decisions. GAO analyzed state-reported targets and NHTSA data from 2014 through 2017—the most recent data available—for all 50 states, the District of Columbia, and Puerto Rico; surveyed these states on the use of performance measures and targets; reviewed requirements in NHTSA's and FHWA's frameworks; and interviewed officials from NHTSA, FHWA, and 10 states, selected to obtain a mix of population sizes, geographic locations, and other factors. From 2014 through 2017, states did not achieve most of the fatality-related targets they set under the National Highway Traffic Safety Administration's (NHTSA) performance management framework (see table), and the number of serious injury targets states achieved during this period is unclear. GAO did not assess whether states achieved targets they set under the Federal Highway Administration's (FHWA) framework because the data were not yet available. State officials we interviewed said that achieving fatality targets may depend on factors outside their control, such as demographic, economic, and legislative changes. GAO's analysis of states' reports showed that nearly half of states did not provide the required assessment of progress to NHTSA on their most recent set of fatality targets. While NHTSA has taken steps to improve its review of these reports, officials acknowledged states are not clear on which target years to assess. Further, NHTSA lacks a mechanism to report whether states eventually achieve these targets. As a result, NHTSA and other stakeholders have limited insight into the results states have achieved from their use of federal safety funds. The extent to which states achieved serious injury targets is unclear because states have changed their definitions of serious injury over time. To ensure the consistency of these data, NHTSA and FHWA established a standard definition for reporting serious injuries, which states are in the process of adopting. In a survey that GAO administered, officials from a majority of states said that performance measures informed how they selected projects under NHTSA's framework. GAO found, however, that in the 2019 plans submitted by states to NHTSA, less than a third of states reported how performance targets and funded projects were linked. Since the submission of those plans, NHTSA has provided training and guidance to its staff to ensure future plans will more clearly identify these links. Under FHWA's framework, about one-third of states reported in GAO's survey that performance measures influenced their project selection; the remaining two-thirds reported using an alternative data-driven approach, such as cost-benefit analysis. FHWA officials said they are developing guidance to help states integrate performance measures and targets into methods that states are currently using to select highway safety projects. GAO recommends that NHTSA (1) provide additional direction and clarification to ensure states assess and report progress in meeting fatality targets, and (2) report on states' final achievement of targets. DOT concurred with the recommendations.", "document_type": "gao"}
{"report": "VA’s mission is to promote the health, welfare, and dignity of all veterans in recognition of their service to the nation by ensuring that they receive medical care, benefits, social support, and lasting memorials. In carrying out this mission, the department operates one of the largest health care delivery systems in America, providing health care to millions of veterans and their families at more than 1,500 facilities. The department’s three major components—the Veterans Health Administration (VHA), the Veterans Benefits Administration (VBA), and the National Cemetery Administration (NCA)—are primarily responsible for carrying out its mission. More specifically, VHA provides health care services, including primary care and specialized care, and it performs research and development to address veterans’ needs. VBA provides a variety of benefits to veterans and their families, including disability compensation, educational opportunities, assistance with home ownership, and life insurance. Further, NCA provides burial and memorial benefits to veterans and their families. The use of IT is critically important to VA’s efforts to provide benefits and services to veterans. As such, the department operates and maintains an IT infrastructure that is intended to provide the backbone necessary to meet the day-to-day operational needs of its medical centers, veteran- facing systems, benefits delivery systems, memorial services, and all other systems supporting the department’s mission. The infrastructure is to provide for data storage, transmission, and communications requirements necessary to ensure the delivery of reliable, available, and responsive support to all VA staff offices and administration customers, as well as veterans. Toward this end, the department operates approximately 240 information systems, manages approximately 314,000 desktop computers and 30,000 laptops, and administers nearly 460,000 network user accounts for employees and contractors to facilitate providing benefits and health care to veterans. These systems are used for the determination of benefits, benefits claims processing, patient admission to hospitals and clinics, and access to health records, among other services. VHA’s systems provide capabilities to establish and maintain electronic health records that health care providers and other clinical staff use to view patient information in inpatient, outpatient, and long-term care settings. The department’s health information system—VistA—serves an essential role in helping the department to fulfill its health care delivery mission. Specifically, VistA is an integrated medical information system that was developed in-house by the department’s clinicians and IT personnel, and has been in operation since the early 1980s. The system consists of 104 separate computer applications, including 56 health provider applications; 19 management and financial applications; eight registration, enrollment, and eligibility applications; five health data applications; and three information and education applications. Within VistA, an application called the Computerized Patient Record System enables the department to create and manage an individual electronic health record for each VA patient. In June 2017, the former VA Secretary announced that the department planned to acquire the same Cerner electronic health record system that the Department of Defense (DOD) has acquired. VA’s effort—the Electronic Health Record Modernization (EHRM) program—calls for the deployment of a new electronic health record system at three initial sites in 2020, with a phased implementation of the remaining sites over the next decade. In addition, VBA relies on the Veterans Benefits Management System (VBMS) to collect and store information such as military service records, medical examinations, and treatment records from VA, DOD, and private medical service providers. In 2014, VA issued its 6-year strategic plan, which emphasizes the department’s goal of increasing veterans’ access to benefits and services, eliminating the disability claims backlog, and ending veteran homelessness. According to the plan, the department intends to improve access to benefits and services through the use of enhanced technology to provide veterans with access to more effective care management. The plan also calls for VA to eliminate the disability claims backlog by fully implementing an electronic claims process that is intended to reduce processing time and increase accuracy. Further, the department has an initiative under way that provides services, such as health care, housing assistance, and job training, to end veteran homelessness. Toward this end, VA is working with other agencies, such as the Department of Health and Human Services, to implement more coordinated data entry systems to streamline and facilitate access to appropriate housing and services. Since 2007, VA has been operating a centralized organization, OI&T, in which most key functions intended for effective management of IT are performed. This office is led by the Assistant Secretary for Information and Technology—VA’s Chief Information Officer (CIO). The office is responsible for providing strategy and technical direction, guidance, and policy related to how IT resources are to be acquired and managed for the department, and for working closely with its business partners—such as VHA—to identify and prioritize business needs and requirements for IT systems. Among other things, OI&T has responsibility for managing the majority of VA’s IT-related functions, including the maintenance and modernization of VistA. As of January 2019, OI&T was comprised of about 15,800 staff, with more than half of these positions filled by contractors. VA’s fiscal year 2020 budget request includes about $5.9 billion for OI&T and its new electronic health record system. Of this amount, about $4.3 billion was requested for OI&T, which represents a $240 million increase over the $4.1 billion enacted for 2019. The request seeks the following levels of funding: $401 million for new systems development efforts to support current health care systems platforms, and to replace legacy systems, such as the Financial Management System; approximately $2.7 billion for the operations and maintenance of existing systems, which includes $327.3 million for infrastructure readiness that is to support the transition to the new electronic health record system; and approximately $1.2 billion for administration. Additionally, the department requested about $1.6 billion for the EHRM program. This amount is an increase of $496 million over the $1.1 billion that was enacted for the program for fiscal year 2019. The request includes the following: $1.1 billion for the contract with the Cerner Corporation to acquire the $161,800 for program management, and $334,700 for infrastructure support. In 2015, we designated VA Health Care as a high-risk area for the federal government and noted that IT challenges were among the five areas of concern. In part, we identified limitations in the capacity of VA’s existing systems, including the outdated, inefficient nature of certain systems and a lack of system interoperability—that is, the ability to exchange and use electronic health information—as contributors to the department’s IT challenges related to health care. Also, in February 2015, we added Improving the Management of IT Acquisitions and Operations to our list of high-risk areas. Specifically, federal IT investments were too frequently failing or incurring cost overruns and schedule slippages while contributing little to mission- related outcomes. We have previously reported that the federal government has spent billions of dollars on failed IT investments, including at VA. Our 2017 update to the high-risk report noted that VA had partially met our leadership commitment criterion by involving top leadership in addressing the IT challenges portion of the VA Health Care high-risk area; however, it had not met the action plan, monitoring, demonstrated progress, or capacity criteria. We have also identified VA as being among a handful of departments with one or more archaic legacy systems. Specifically, in our May 2016 report on legacy systems used by federal agencies, we identified two of VA’s systems as being over 50 years old—the Personnel and Accounting Integrated Data system and the Benefits Delivery Network system. These systems were among the 10 oldest investments and/or systems that were reported by 12 selected agencies. Accordingly, we recommended that the department identify and plan to modernize or replace its legacy systems. VA addressed the recommendation in May 2018, when it provided a Comprehensive Information Technology Plan that showed a detailed roadmap for the key programs and systems required for modernization. The plan included time frames, activities to be performed, and functions to be replaced or enhanced. The plan also indicated that the Personnel and Accounting Integrated Data system and the Benefits Delivery Network system are to be decommissioned in quarters 3 and 4 of fiscal year 2019, respectively. Our March 2019 update to our high-risk series noted that the ratings for leadership commitment criterion regressed, while the action plan criterion improved for the IT Challenges portion of the VA Health Care area. The capacity, monitoring, and demonstrated progress criteria remained unchanged. Our work continued to indicate that VA was not yet able to demonstrate progress in this area. Since its 2015 high-risk designation, we have made 14 new recommendations in the VA Health Care area, 12 of which were made since our 2017 high-risk report was issued. For example, in June 2017, to address deficiencies we recommended that the department take six actions to provide clinicians and pharmacists with improved tools to support pharmacy services to veterans and reduce risks to patient safety. VA generally concurred with these recommendations; however, all of them remain open. Congress enacted FITARA in December 2014 to improve agencies’ acquisitions of IT and enable Congress to better monitor agencies’ progress and hold them accountable for reducing duplication and achieving cost savings. The law applies to VA and other covered agencies. It includes specific requirements related to seven areas, including agency CIO authority, data center consolidation and optimization, risk management of IT investments, and government-wide software purchasing. Agency CIO authority enhancements. CIOs at covered agencies are required to (1) approve the IT budget requests of their respective agencies, (2) certify that IT investments are adequately implementing incremental development, as defined in capital planning guidance issued by the Office of Management and Budget (OMB), (3) review and approve contracts for IT, and (4) approve the appointment of other agency employees with the title of CIO. Federal data center consolidation initiative. Agencies are required to provide OMB with a data center inventory, a strategy for consolidating and optimizing their data centers (to include planned cost savings), and quarterly updates on progress made. The law also requires OMB to develop a goal for how much is to be saved through this initiative, and provide annual reports on cost savings achieved. Enhanced transparency and improved risk management in IT investments. OMB and covered agencies are to make detailed information on federal IT investments publicly available, and department-level CIOs are to categorize their major IT investments by risk. Additionally, in the case of major investments rated as high risk for 4 consecutive quarters, the act required that the department- level CIO and the investment’s program manager conduct a review aimed at identifying and addressing the causes of the risk. Government-wide software purchasing program. The General Services Administration is to enhance government-wide acquisition and management of software and allow for the purchase of a software license agreement that is available for use by all executive branch agencies as a single user. Additionally, the Making Electronic Government Accountable by Yielding Tangible Efficiencies Act of 2016, or the “MEGABYTE Act,” further enhanced CIOs’ management of software licenses by requiring agency CIOs to establish an agency software licensing policy and a comprehensive software license inventory to track and maintain licenses, among other requirements. In June 2015, OMB released guidance describing how agencies are to implement FITARA. This guidance is intended to, among other things: assist agencies in aligning their IT resources with statutory establish government-wide IT management controls that will meet the law’s requirements, while providing agencies with flexibility to adapt to unique agency processes and requirements; clarify the CIO’s role and strengthen the relationship between agency CIOs and bureau CIOs; and strengthen CIO accountability for IT costs, schedules, performance, and security. The federal approach and strategy for securing information systems is prescribed by federal law and policy. The Federal Information Security Modernization Act (FISMA) provides a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support federal operations and assets. In addition, the Federal Cybersecurity Enhancement Act of 2015 requires protecting federal networks through the use of federal intrusion prevention and detection capabilities. Further, Executive Order 13800, Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure, directs agencies to manage cybersecurity risks to the federal enterprise by, among other things, using the National Institute of Standards and Technology Framework for Improving Critical Infrastructure Cybersecurity (cybersecurity framework). Federal agencies, including VA, and our nation’s critical infrastructures— such as energy, transportation systems, communications, and financial services—are dependent on IT systems and electronic data to carry out operations and to process, maintain, and report essential information. The security of these systems and data is vital to public confidence and national security, prosperity, and well-being. Because many of these systems contain vast amounts of personally identifiable information, agencies must protect the confidentiality, integrity, and availability of this information. In addition, they must effectively respond to data breaches and security incidents when they occur. The risks to IT systems supporting the federal government and the nation’s critical infrastructure are increasing, including insider threats from witting or unwitting employees, escalating and emerging threats from around the globe, and the emergence of new and more destructive attacks. Cybersecurity incidents continue to impact federal entities and the information they maintain. According to OMB’s 2018 annual FISMA report to Congress, agencies reported 35,277 information security incidents to DHS’s U.S. Computer Emergency Readiness Team in fiscal year 2017. VA has made limited progress toward addressing the IT management challenges for three critical initiatives: VistA, the Family Caregiver Program, and VBMS. Specifically, the department has recently initiated its fourth effort to modernize VistA, but uncertainty remains regarding the program’s governance. In addition, although VA has taken steps to address our recommendations for the Family Caregiver Program and VBMS, the department has not fully implemented most of them. VA has pursued four efforts over nearly 2 decades to modernize VistA. These efforts—HealtheVet, the integrated Electronic Health Record (iEHR), VistA Evolution, and EHRM—reflect varying approaches that the department has considered to achieve a modernized health care system. Figure 1 shows a timeline of the four efforts that VA has pursued to modernize VistA since 2001. In 2001, VA undertook its first VistA modernization project, the HealtheVet initiative, with the goals of standardizing the department’s health care system and eliminating the approximately 130 different systems used by its field locations at that time. HealtheVet was scheduled to be fully implemented by 2018 at a total estimated development and deployment cost of about $11 billion. As part of the effort, the department had planned to develop or enhance specific areas of system functionality through six projects, which were to be completed between 2006 and 2012. In June 2008, we reported that the department had made progress on the HealtheVet initiative, but noted concerns with its project planning and governance. In June 2009, the Secretary of Veterans Affairs announced that VA would stop financing failed projects and improve the management of its IT development projects. Subsequently in August 2010, the department reported that it had terminated the HealtheVet initiative. In February 2011, VA began its second VistA modernization initiative, the iEHR program, in conjunction with DOD. The program was intended to replace the two separate electronic health record systems used by the two departments with a single, shared system. In addition, because both departments would be using the same system, this approach was expected to largely sidestep the challenges that had been encountered in trying to achieve interoperability between their two separate systems. Initial plans called for the development of a single, joint iEHR system consisting of 54 clinical capabilities to be delivered in six increments between 2014 and 2017. Among the agreed-upon capabilities to be delivered were those supporting laboratory, anatomic pathology, pharmacy, and immunizations. According to VA and DOD, the single system had an estimated life cycle cost of $29 billion through the end of fiscal year 2029. However, in February 2013, the Secretaries of VA and DOD announced that they would not continue with their joint development of a single electronic health record system. This decision resulted from an assessment of the iEHR program that the secretaries had requested in December 2012 because of their concerns about the program facing challenges in meeting deadlines, costing too much, and taking too long to deliver capabilities. In 2013, the departments abandoned their plan to develop the integrated system and stated that they would again pursue separate modernization efforts. In December 2013, VA initiated its VistA Evolution program as a joint effort of VHA and OI&T. The program was to be comprised of a collection of projects and efforts focused on improving the efficiency and quality of veterans’ health care, modernizing the department’s health information systems, increasing the department’s data exchange and interoperability with DOD and private sector health care partners, and reducing the time it takes to deploy new health information management capabilities. Further, the program was intended to result in lower costs for system upgrades, maintenance, and sustainment. However, VA ended the VistA Evolution program in December 2018 to focus on its new electronic health record system acquisition. In June 2017, VA’s Secretary announced a significant shift in the department’s approach to modernizing VistA. Specifically, rather than continue to use VistA, the Secretary stated that the department would acquire the same electronic health record system that DOD is implementing. In this regard, DOD awarded a contract to acquire a new integrated electronic health record system developed by the Cerner Corporation. According to the Secretary, VA decided to acquire this same product because it would allow all of VA’s and DOD’s patient data to reside in one system, thus enabling seamless care between the department and DOD without the manual and electronic exchange and reconciliation of data between two separate systems. According to the Secretary, this fourth VistA modernization initiative is intended to minimize customization and system differences that currently exist within the department’s medical facilities, and ensure the consistency of processes and practices within VA and DOD. When fully operational, the system is intended to be a single source for patients to access their medical history and for clinicians to use that history in real time at any VA or DOD medical facility, which may result in improved health care outcomes. According to VA’s Chief Technology Officer, Cerner is expected to provide integration, configuration, testing, deployment, hosting, organizational change management, training, sustainment, and licenses necessary to deploy the system in a manner that meets the department’s needs. To expedite the acquisition, in June 2017, the Secretary signed a “Determination and Findings,” for a public interest exception to the requirement for full and open competition, and authorized VA to issue a solicitation directly to Cerner. Accordingly, the department awarded a contract to Cerner in May 2018 for a maximum of $10 billion over 10 years. Cerner is to replace VistA with a commercial electronic health record system. This new system is to support a broad range of health care functions that include, for example, acute care, clinical decision support, dental care, and emergency medicine. When implemented, the new system will be expected to provide access to authoritative clinical data sources and become the authoritative source of clinical data to support improved health, patient safety, and quality of care provided by VA. Further, the department has estimated that, as of November 2018, an additional $6.1 billion in funding, above the Cerner contract amount, will be needed to fund additional project management support supplied by outside contractors, government labor costs, and infrastructure improvements over a 10-year implementation period. Deployment of the new electronic health record system at three initial sites is planned for March 2020, with a phased implementation of the remaining sites over the next decade. Each VA medical facility is expected to continue using VistA until the new system has been deployed at that location. After VA announced in June 2017 that it planned to acquire the Cerner electronic health record system, we testified in June 2018 that a governance structure had been proposed that would be expected to leverage existing joint governance facilitated by the Interagency Program Office. At that time, VA’s program officials had stated that the department’s governance plans for the new program were expected to be finalized in October 2018. However, the officials had not indicated what role, if any, the Interagency Program Office was to have in the governance process. This office has been involved in various approaches to increase health information interoperability since it was established by the National Defense Authorization Act for Fiscal Year 2008 to function as the single point of accountability for DOD’s and VA’s electronic health record system interoperability efforts. In September 2018, we recommended that VA clearly define the role and responsibilities of the Interagency Program Office in the governance plans for acquisition of the department’s new electronic health record system. The department concurred with our recommendation and stated that the Joint Executive Committee, a joint governance body comprised of leadership from DOD and VA, had approved a role for the Interagency Program Office that included providing expertise, guidance, and support for DOD, VA, and joint governance bodies as the departments continue to acquire and implement interoperable electronic health record systems. However, the department has not yet provided documentation supporting these actions and how they relate to VA’s governance structure for the new acquisition. In addition, the role described does not appear to position the office to be the single point of accountability originally identified in the National Defense Authorization Act for Fiscal Year 2008. We continue to monitor the department’s governance plans for the acquisition of the new electronic health record system and its relationship with the Interagency Program Office. In May 2010, VA was required by statute to establish a program to support family caregivers of seriously injured post-9/11 veterans. In May 2011, VHA implemented its Family Caregiver Program at all VA medical centers across the country, offering caregivers an array of services, including a monthly stipend, training, counseling, referral services, and expanded access to mental health and respite care. In fiscal year 2014, VHA obligated over $263 million for the program. In September 2014, we reported that the Caregiver Support Program office, which manages the program, did not have ready access to the types of workload data that would allow it to routinely monitor the effects of the Family Caregiver Program on VA medical centers’ resources due to limitations with the program’s IT system—the Caregiver Application Tracker. Program officials explained that this system was designed to manage a much smaller program and, as a result, the system has limited capabilities. Outside of obtaining basic aggregate program statistics, the program office was not able to readily retrieve data from the system that would allow it to better assess the scope and extent of workload problems at VA medical centers. Program officials also expressed concern about the reliability of the system’s data. The lack of ready access to comprehensive workload data impeded the program office’s ability to monitor the program and identify workload problems or make modifications as needed. This runs counter to federal standards for internal control which state that agencies should monitor their performance over time and use the results to correct identified deficiencies and make improvements. We also noted in our report that program officials told us that they had taken initial steps to obtain another IT system to support the Family Caregiver Program, but they were not sure how long it would take to implement. Accordingly, we recommended that VA expedite the process for identifying and implementing a system that would fully support the Family Caregiver Program. VA concurred with our recommendation and subsequently began taking steps to implement a replacement system. However, the department has encountered challenges related to the system implementation efforts. We have ongoing work to evaluate VA’s effort to acquire a new IT system to support the Family Caregiver Program. In September 2015, we reported that VBA had made progress in developing and implementing VBMS—its system for processing disability benefit claims—but also noted that additional actions could improve efforts to develop and use the system. Specifically, VBA had deployed the initial version of the system to all of its regional offices as of June 2013. Further, after initial deployment, it continued developing and implementing additional system functionality and enhancements to support the electronic processing of disability compensation claims. Nevertheless, we pointed out that VBMS was not able to fully support disability and pension claims, as well as appeals processing. While the Under Secretary for Benefits stated in March 2013 that the development of the system was expected to be completed in 2015, implementation of functionality to fully support electronic claims processing was delayed beyond 2015. In addition, VBA had not produced a plan that identified when the system would be completed. Accordingly, holding VBA management accountable for meeting a time frame and demonstrating progress was difficult. Our report further noted that, even as VBA continued its efforts to complete the development and implementation of VBMS, three areas were in need of increased management attention: cost estimating, system availability, and system defects. We also noted in our report that VBA had not conducted a customer satisfaction survey that would allow the department to compile data on how users viewed the system’s performance and, ultimately, to develop goals for improving the system. We made five recommendations to improve VA’s efforts to effectively complete the development and implementation of VBMS. VA agreed with four of the recommendations. In addition, the department has addressed one of the recommendations—that it establish goals for system response time and use the goals as the basis for reporting system performance. However, the department has not yet fully addressed our remaining recommendations to (1) develop a plan with a time frame and a reliable cost estimate for completing VBMS, (2) reduce the incidence of system defects present in new releases, (3) assess user satisfaction, and (4) establish satisfaction goals to promote improvement. Continued attention to these important areas can improve VA’s efforts to effectively complete the development and implementation of VBMS and, in turn, more effectively support the department’s processing of disability benefit claims. FITARA included provisions for federal agencies to, among other things, enhance government-wide acquisition and management of software, improve the risk management of IT investments, consolidate data centers, and enhance CIOs’ authorities. Since its enactment, we have reported numerous times on VA’s efforts toward implementing FITARA. VA’s progress toward implementing key FITARA provisions has been uneven. Specifically, VA issued a software licensing policy and has generated an inventory of its software licenses to inform future investment decisions. However, the department did not fully address requirements related to IT investment risk, data center consolidation, or CIO authority enhancement. VA has made progress in addressing federal software licensing requirements. In May 2014, we reported on federal agencies’ management of software licenses and stressed that better management was needed to achieve significant savings government-wide. Specifically regarding VA, we noted that the department did not have comprehensive policies that included the establishment of clear roles and central oversight authority for managing enterprise software license agreements, among other things. We also noted that it had not established a comprehensive software license inventory, a leading practice that would help the department to adequately manage its software licenses. The inadequate implementation of these and other leading practices in software license management was partially due to weaknesses in the department’s policies related to licensing management. Thus, we made six recommendations to VA to improve its policies and practices for managing licenses. For example, we recommended that the department regularly track and maintain a comprehensive inventory of software licenses and analyze the inventory to identify opportunities to reduce costs and better inform investment decision making. Since our 2014 report, VA has taken actions to implement all six recommendations. For example, the department implemented a solution to generate and maintain a comprehensive inventory of software licenses using automated tools for the majority of agency software license spending and/or enterprise-wide licenses. Additionally, the department implemented a solution to analyze agency-wide software license data, including usage and costs; and it subsequently identified approximately $65 million in cost savings over 3 years due to analyzing one of its software licenses. VA has made limited progress in addressing the FITARA requirements related to managing the risks associated with IT investments. In June 2016, we reported on risk ratings assigned to investments by CIOs. We noted that the department had reviewed compliance with risk management practices, but had not assessed active risks when developing its risk ratings. VA determined its ratings by quantifying and combining inputs such as cost and schedule variances, risk exposure values, and compliance with agency processes. Metrics for compliance with agency processes included those related to program and project management, project execution, the quality of investment documentation, and whether the investment was regularly updating risk management plans and logs. When developing CIO ratings, VA chose to focus on investments’ risk management processes, such as whether a process was in place or whether a risk log was current. Such approaches did not consider individual risks, such as funding cuts or staffing changes, which detail the probability and impact of pending threats to success. Instead, VA’s CIO rating process considered several specific risk management criteria: whether an investment (1) had a risk management strategy, (2) kept the risk register current and complete, (3) clearly prioritized risks, and (4) put mitigation plans in place to address risks. As a result, we recommended that VA factor active risks into its CIO ratings. We also recommended that the department ensure that these ratings reflect the level of risk facing an investment relative to that investment’s ability to accomplish its goals. VA concurred with the recommendations and cited actions it planned to take to address them. VA has reported progress on consolidating and optimizing its data centers, although this progress has fallen short of targets set by OMB. Specifically, VA reported a total inventory of 415 data centers, of which 39 had been closed as of August 2017. While the department anticipated another 10 data centers would be closed by the end of fiscal year 2018, these closures fell short of the targets set by OMB. Further, while VA reported $23.61 million in data center-related cost savings and avoidances from 2012 through August 2017, the department did not realize further savings from the additional 10 data center closures. In addition, as of February 2017, VA reported meeting one of OMB’s five data center optimization metrics related to power usage effectiveness. Also, the department’s data center optimization strategic plan indicated that VA planned to meet three of the five metrics by the end of fiscal year 2018. Further, while OMB directed agencies to replace manual collection and reporting of metrics with automated tools no later than fiscal year 2018, the department had only implemented automated tools at 6 percent of its data centers. We have recommended that VA take actions to address data center savings goals and optimization performance targets identified by OMB. The department has taken actions to address these recommendations, including reporting data center consolidation savings and avoidance costs to OMB and updating its data center optimization strategic plan. However, the department has yet to address recommendations related to areas that we reported as not meeting OMB’s established targets, including implementing automated monitoring tools at its data centers. VA has made limited progress in addressing the CIO authority requirements of FITARA. Specifically, in November 2017, we reported on agencies’ efforts to utilize incremental development practices for selected major investments. We noted that VA’s CIO had certified the use of adequate incremental development for all 10 of the department’s major IT investments. However, VA had not updated the department’s policy and process for the CIO’s certification of major IT investments’ adequate use of incremental development, in accordance with OMB’s guidance on the implementation of FITARA, as we had recommended. As of October 2018, a VA official stated that the department was working to draft a policy to address our recommendation, but did not identify time frames for when all activities would be completed. In January 2018, we reported on the need for agencies to involve CIOs in reviewing IT acquisition plans and strategies. We noted that VA’s CIO did not review IT acquisition plans or strategies and that the Chief Acquisition Officer was not involved in the process of identifying IT acquisitions. Accordingly, we recommended that the VA Secretary ensure that the office of the Chief Acquisition Officer is involved in the process to identify IT acquisitions. We also recommended that the Secretary ensure that the acquisition plans or strategies are reviewed and approved in accordance with OMB guidance. The department concurred with the recommendations and, in a May 2018 update, provided a draft process map that depicted its forthcoming acquisition process. However, as of March 2019, this process had not yet been finalized and implemented. In August 2018, we reported that the department had only fully addressed two of the six key areas that we identified—IT Leadership and Accountability and Information Security. The department had partially addressed IT Budgeting, minimally addressed IT Investment Management, and had not at all addressed IT Strategic Planning or IT Workforce. Thus, we recommended that the VA Secretary ensure that the department’s IT management policies address the role of the CIO for key responsibilities in the four areas we identified. The department concurred with the recommendation and acknowledged that many of the responsibilities provided to the CIO were not explicitly formalized by VA policy. In December 2018, we reported on the effectiveness of the government’s approach and strategy for securing its systems. The federal approach and strategy for securing information systems is prescribed by federal law and policy, including FISMA and the presidential executive order on Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure. Accordingly, federal reports describing agency implementation of this law and policy, and reports of related agency information security activities, indicated VA’s lack of effectiveness in its efforts to implement the federal approach and strategy. Our December 2018 report identified that the department was deficient or had material weaknesses in all four indicators of departments’ effectiveness in implementing the federal approach and strategy for securing information systems. Specifically, VA was not effective in the Inspector General Information Security Program Ratings, was found to have material weaknesses in the Inspector General Internal Control Deficiencies over Financial Reporting, did not meet CIO Cybersecurity Cross-Agency Priority Goal Targets, and had enterprises that were at risk according to OMB Management Assessment Ratings. We reported on federal high-impact systems—those that hold sensitive information, the loss of which could cause individuals, the government, or the nation catastrophic harm—in May 2016. We noted that VA had implemented numerous controls, such as completion of risk assessments, over selected systems. However, the department had not always effectively implemented access controls, patch management, and contingency planning to protect the confidentiality, integrity and availability of these high-impact systems. These weaknesses existed in part because the department had not effectively implemented elements of its information security program. We made five recommendations to VA to improve its information security program. The department concurred with the recommendations and, as of March 2019, had implemented three of the five recommendations. Our March 2019 report on the federal cybersecurity workforce indicated that VA was not accurately categorizing positions to effectively identify critical staffing needs. The Federal Cybersecurity Workforce Assessment Act of 2015 required agencies to assign the appropriate work role codes to each position with cybersecurity, cyber-related, and IT functions. Agencies were to assign a code of “000” only to positions that did not perform IT, cybersecurity, or cyber-related functions. As we reported, VA had assigned a “000” code to 3,008 (45 percent) of its 6,636 IT positions. Human resources and IT officials from the department stated that they may have assigned the “000” code in error and that they had not completed the process to validate the accuracy of their codes. We recommended that VA take steps to review the assignment of the “000” code to any of the department’s positions in the IT management occupational series and assign the appropriate work role codes. VA concurred with the recommendation and indicated that it was in the process of conducting a cyber coding review. In conclusion, VA has long struggled to overcome IT management challenges, which have resulted in a lack of system capabilities needed to successfully implement critical initiatives. In this regard, VA is set to begin deploying its new electronic health record system in less than 1 year and questions remain regarding the governance structure for the program. Thus, it is more important than ever for the department to ensure that it is managing its IT budget in a way that addresses the challenges we have identified in our previous reports and high-risk updates. If the department continues to experience the challenges that we have previously identified, it may jeopardize its fourth attempt to modernize its electronic health record system. Additionally, the department has been challenged in fully implementing provisions of FITARA, which has limited its ability to improve its management of IT acquisitions. Until the department implements the act’s provisions, Congress will be unable to effectively monitor VA’s progress and hold it accountable for reducing duplication and achieving cost savings. Further, the lack of key cybersecurity management elements at VA is concerning given that agencies’ systems are increasingly susceptible to the multitude of cyber-related threats that exist. As VA continues to pursue modernization efforts, it is critical that the department take steps to adequately secure its systems. Chair Lee, Ranking Member Banks, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staffs have any questions about this testimony, please contact Carol C. Harris, Director, Information Technology Management Issues, at (202) 512-4456 or harrisc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony statement. GAO staff who made key contributions to this testimony are Mark Bird (Assistant Director), Eric Trout (Analyst in Charge), Justin Booth, Rebecca Eyler, Katherine Noble, Scott Pettis, Christy Tyson, and Kevin Walsh. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The use of IT is crucial to helping VA effectively serve the nation's veterans. Each year the department spends billions of dollars on its information systems and assets. However, VA has experienced challenges in managing its IT programs, raising questions about its ability to deliver intended outcomes needed to help advance the department's mission. To improve federal agencies' IT acquisitions, in December 2014 Congress enacted FITARA. GAO has previously reported on IT management challenges at VA, as well as its progress in implementing FITARA and cybersecurity requirements. GAO was asked to summarize key results and recommendations from its work at VA that examined systems modernization efforts, FITARA implementation, and cybersecurity efforts. To do so, GAO reviewed its recently issued reports and incorporated information on the department's actions in response to GAO's recommendations. The Department of Veterans Affairs (VA) has made limited progress toward addressing information technology (IT) system modernization challenges. From 2001 through 2018, VA pursued three efforts to modernize its health information system—the Veterans Health Information Systems and Technology Architecture (VistA). However, these efforts experienced high costs, challenges to ensuring interoperability of health data, and ultimately did not result in a modernized VistA. Regarding the department's fourth and most recent effort, the Electronic Health Record Modernization, GAO recently reported that the governance plan for this program was not yet defined. VA has not fully implemented GAO's recommendation calling for the department to define the role of a key office in the governance plans. The Family Caregiver Program, which was established to support family caregivers of seriously injured post-9/11 veterans, has not been supported by an effective IT system. Specifically, GAO reported that, due to limitations with the system, the program office did not have ready access to the types of workload data that would allow it to routinely monitor workload problems created by the program. GAO recommended that VA expedite the process for identifying and implementing an IT system. Although the department concurred with the recommendation, VA has not yet fully addressed it. VA had developed the Veterans Benefits Management System—its system that is used for processing disability benefit claims; however, the system did not fully support disability and pension claims, as well as appeals processing. GAO made five recommendations for VA to improve its efforts to effectively complete the development and implementation of the system. The department concurred with the recommendations but has implemented only one thus far. VA has demonstrated uneven progress toward fully implementing GAO's recommendations related to key Federal Information Technology Acquisition Reform Act (FITARA) provisions. Specifically, VA has implemented all six recommendations in response to GAO's 2014 report on managing software licenses, leading to, among other things, savings of about $65 million over 3 years. However, the department has not fully addressed two recommendations from GAO's 2016 report on managing the risks of major IT investments. Further, the department has not implemented (1) two of four recommendations related to its effort to consolidate data centers and (2) GAO's four recommendations to increase the authority of its Chief Information Officer. VA's management of cybersecurity has also lacked key elements. For example, GAO reported in May 2016 that VA had established numerous security controls, but had not effectively implemented key elements of its information security program. In addition, as GAO reported in March 2019, the department had not accurately categorized positions to effectively identify critical staffing needs for its cybersecurity workforce. VA has implemented three of six cybersecurity-related recommendations from these two reports. GAO has made numerous recent recommendations to VA aimed at improving the department's IT management. VA has generally agreed with the recommendations and has taken steps to address them; however, the department has fully implemented less than half of them. Fully implementing all of GAO's recommendations would help VA ensure that its IT effectively supports the department's mission.", "document_type": "gao"}
{"report": "VHA’s Family Caregiver Program is designed to provide support and services to family caregivers of post-9/11 veterans who have a serious injury that was incurred or aggravated in the line of duty. The program provides approved primary family caregivers with a monthly financial stipend as well as training and other support services, such as counseling and respite care. The Family Caregiver Program has a series of eligibility requirements that must be satisfied in order for family caregivers to be approved. To meet the program’s initial eligibility criteria, the veteran seeking caregiver assistance must have a serious injury that was incurred or aggravated in the line of duty on or after September 11, 2001. According to the program’s regulations, a serious injury is any injury, including traumatic brain injury (TBI), psychological trauma, or other mental disorder, that has been incurred or aggravated in the line of duty and renders the veteran or servicemember in need of personal care services. The veteran must be in need of personal care services for a minimum of 6 continuous months based on any one of the following clinical eligibility criteria: (1) an inability to perform one or more activities of daily living, such as bathing, dressing, or eating; (2) a need for supervision or protection based on symptoms or residuals of neurological or other impairment or injury such as TBI, post-traumatic stress disorder, or other mental health disorders; (3) the existence of a psychological trauma or a mental disorder that has been scored by a licensed mental health professional, with a Global Assessment of Functioning score of 30 or less, continuously during the 90-day period immediately preceding the date on which VHA initially received the application; or (4) the veteran has been rated 100 percent service connected disabled for a qualifying serious injury and has been awarded special monthly compensation that includes an aid and attendance allowance. To be considered competent to care for the veteran, family caregivers must meet certain requirements including (1) having the ability to communicate and follow details of the treatment plan and instructions related to the care of the veteran; (2) not determined by VA to have abused or neglected the veteran; (3) being at least 18 years of age; and (4) either being a family member—such as a spouse, son or daughter, parent, step-family member, or extended family member— or an unrelated person who lives or will live full-time with the veteran. Family caregivers must also complete required training before being approved for the program. VHA’s Caregiver Support Program office is responsible for developing policy and providing guidance and oversight for the Family Caregiver Program. It also directly administers the program’s stipend, provides support services such as a telephone hotline and website, and arranges coverage through the Civilian Health and Medical Program of the Department of Veterans Affairs (CHAMPVA) for eligible caregivers if they have no other coverage. Further, the office provides funding to VAMCs to cover certain program costs. These costs may include the salaries of the caregiver support coordinators (CSC), who implement and administer the Family Caregiver Program at the local VAMC level, and the costs VAMCs incur for having their clinical staff, such as nurses, conduct the program’s required in- home visits to approved caregivers and their veterans. CSCs are generally licensed social workers or registered nurses, and they have both clinical and administrative responsibilities. Their clinical responsibilities may include identifying and coordinating appropriate interventions for caregivers or referrals to other VA or non-VA programs, such as mental health treatment, respite care, or additional training and education. Their administrative responsibilities may include responding to inquiries about the program, overseeing the application process, entering information about applications and approved caregivers into IT systems, and facilitating the processing of appeals. As of May 2014, there were 233 CSCs assigned to 140 VAMCs or health care systems across the country. Additionally, each regional VISN office has a VISN CSC lead for the program, who provides guidance to CSCs and helps address their questions or concerns. CAT, which was deployed in May 2011, is a web-based system that was designed to facilitate the exchange of information about approved caregivers between VAMCs and other VHA entities. Such entities include the Health Administration Center, which processes the caregiver stipend payments and administers CHAMPVA. In 2014, we reported that the Caregiver Support Program office was not able to easily retrieve data from CAT that would allow officials to better assess workload trends at individual VAMCs—such as the length of time applications are delayed or the timeliness of home visits—even though these data were already captured in the system. Caregiver Support Program officials only retrieved workload data on an ad hoc, as-needed basis, which limited their ability to assess the scope and extent of workload problems comprehensively at individual VAMCs and on a system-wide basis. Program officials also expressed concern about the reliability of the system’s data. As we noted in our report, program officials also identified the need for a more capable and flexible system that could interface with other departmental systems. The officials told us that they had taken initial steps to obtain another IT system to support the Family Caregiver Program; however, the officials were not sure how long it would take to implement the system. Accordingly, we recommended that VA expedite the process for identifying and implementing a system that would fully support the Family Caregiver Program. VA concurred with our recommendation and subsequently began taking actions in 2015 to implement a replacement system. These actions included taking steps toward implementing short-term improvements to CAT that were to be followed by the implementation of a long-term replacement system. The recommendation continues to remain open. The John S. McCain III, Daniel K. Akaka, and Samuel R. Johnson VA Maintaining Internal Systems and Strengthening Integrated Outside Networks Act of 2018 (VA MISSION Act), which was enacted in June 2018, included provisions directing VA to implement an IT system to support the Family Caregiver Program and the incremental expansion of program eligibility. Specifically, the act required VA to implement an IT system to fully support the Family Caregiver Program by October 1, 2018. According to the act, the system is to allow for data assessment and comprehensive monitoring of the program. In particular, the system is to have, among other things, the ability to (1) retrieve data to monitor workload trends at the medical center and aggregate levels; (2) manage an increased number of caregivers as the program expands; and (3) integrate with other relevant IT systems at VHA. The act also stated that VA was to submit an initial report to Congress regarding the status of the planning, development, and deployment of this system within 90 days of enactment of the VA MISSION Act, and that the department is to submit a final report to Congress by October 1, 2019. The final report is to include a certification by the VA Secretary that the system has been implemented, along with a description of how the Secretary is using the system to monitor the workload of the program. Although we previously recommended that VA expedite implementation of a replacement for CAT, and the MISSION Act directed the department to implement an IT system to support the Family Caregiver Program, VA has not yet been successful in its multiple efforts to implement such a system. Specifically, VA has faced a number of difficulties in developing and implementing short-term improvements as well as a long-term replacement system for CAT. In July 2015, VHA and the Office of Information and Technology (OIT) initiated a joint acquisition project, called CAT Rescue, to update CAT and improve the system’s data reliability. However, the department reported in January 2017 that this project had experienced delays and identified a large number of defects during system testing. VA terminated the project in April 2018 before any new system capabilities were implemented. A companion project to CAT Rescue that VA initiated in September 2015 was to develop the Caregivers Tool (CareT), a new system intended to be a long-term replacement for CAT. As envisioned, this system was to use the improved data from CAT Rescue while also adding new system capabilities. However, the user acceptance testing of CareT identified the need for the department to develop more system capabilities than originally planned. Further, the department determined that the time period needed to perform additional system development would have extended beyond the term of the development contract, which ended in April 2017. VA subsequently awarded a new CareT development contract in July 2017. However, after additional system development, the department determined during user acceptance testing that the system was not performing as expected and implementation of CareT was further delayed. In October 2018, the department reported to congressional committees that implementing a system to fully support the Family Caregiver Program by the VA MISSION Act deadline was not feasible. Subsequently, the department determined that CareT was not a viable solution and VHA and OIT terminated work on the system in February 2019. VHA and OIT began a third effort in March 2019 to acquire a replacement system that is to be based on an existing commercial product. According to OIT officials, the new IT solution, referred to as the Caregiver Record Management Application (CARMA), is intended to replace CAT. However, the department has not yet established a date for completing CARMA. Thus, VA’s efforts to implement an IT system that supports the Family Caregiver Program have been continuing with no end in sight. We have ongoing work to further evaluate the status and progress of the department’s efforts to implement a system to support the Family Caregiver Program consistent with the VA MISSION Act requirements. Figure 1 provides a timeline of the various IT projects that VA has undertaken to support the program. Our prior work has determined that successfully overcoming IT acquisition challenges can best be achieved when critical success factors are applied. Specifically, we reported in 2011 on common factors critical to the success of IT acquisitions, based on seven agencies having each identified the acquisition that best achieved the agency’s respective cost, schedule, scope, and performance goals. These factors remain relevant today and can serve as a model of best practices that agencies can apply to enhance the likelihood that the acquisition of an IT system such as CARMA will be successfully achieved. Among the agencies’ seven IT investments, agency officials identified nine factors as having been critical to the success of three or more of the seven investments. These nine critical success factors are consistent with leading industry practices for IT acquisition. The factors are: Active engagement of program officials with stakeholders. Qualified and experienced program staff. Support of senior department and agency executives. Involvement of end users and stakeholders in the development of requirements. Participation of end users in testing system functionality prior to formal end user acceptance testing. Consistency and stability of government and contractor staff. Prioritization of requirements by program staff. Regular communication maintained between program officials and the prime contractor. Sufficient funding. Officials for all seven selected investments cited active engagement with program stakeholders—individuals or groups (including, in some cases, end users) with an interest in the success of the acquisition—as a critical factor to the success of those investments. Agency officials stated that stakeholders, among other things, reviewed contractor proposals during the procurement process, regularly attended program management office sponsored meetings, were working members of integrated project teams, and were notified of problems and concerns as soon as possible. Further, officials from two investments noted that actively engaging with stakeholders created transparency and trust, and increased the support from the stakeholders. Additionally, officials for six of the seven selected investments indicated that the knowledge and skills of the program staff were critical to the success of the program. This included knowledge of acquisitions and procurement processes, monitoring of contracts, large-scale organizational transformation, Agile software development concepts, and areas of program management such as earned value management and technical monitoring. Finally, officials for five of the seven selected investments identified having the end users test and validate the system components prior to formal end user acceptance testing for deployment as critical to the success of their program. Similar to this factor, leading guidance recommends testing selected products and product components throughout the program life cycle. Testing of functionality by end users prior to acceptance demonstrates, earlier rather than later in the program life cycle, that the functionality will fulfill its intended use. If problems are found during this testing, programs are typically positioned to make changes that would be less costly and disruptive than ones made later in the life cycle. In conclusion, VA has invested considerable time in multiple efforts toward improving and replacing its IT system to better serve the Family Caregiver Program. However, even with these efforts, the department has not yet implemented a system and the program is not prepared for expansion. Going forward, it is important that VA take steps to improve its efforts to implement a replacement IT system for the Family Caregiver Program. In this regard, the department could benefit from applying critical success factors we previously reported as leading to successful federal IT acquisitions. These factors can serve as a model of best practices that the department can apply to enhance the likelihood that its effort to replace the IT system for the Family Caregiver Program will be successful. Chairs Lee and Brownley, Ranking Members Banks and Dunn, and Members of the Subcommittees, this completes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staffs have any questions about this testimony, please contact Carol C. Harris, Director, Information Technology Management Issues, at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony statement. GAO staff who made key contributions to this testimony are Mark Bird (Assistant Director), Rebecca Eyler, Jacqueline Mai, Monica Perez-Nelson, Scott Pettis, and Jennifer Stavros-Turner (Analyst in Charge). This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "To provide greater support for caregivers of post-9/11 veterans, Congress and the President enacted legislation requiring VA to establish a program to assist caregivers with the rigors of caring for seriously injured veterans. In May 2011, the Veterans Health Administration (VHA), which operates VA's health care system, established the Family Caregiver Program at each of its VA medical centers across the United States. At that time, the department implemented an IT system, called CAT, to help support the program. Subsequently, the VA MISSION Act was enacted in June 2018, requiring VA to implement an IT system to fully support the Family Caregiver Program by October 1, 2018. Further, VA's Secretary is to certify the system by October 1, 2019. GAO was asked to discuss its September 2014 report that examined how VHA is implementing the Family Caregiver Program. In addition, the statement includes relevant information VA provided on its actions toward addressing GAO's prior recommendation. The statement also discusses critical success factors related to IT acquisitions as identified in GAO's prior work. The reports cited throughout this statement include detailed information on the scope and methodology of GAO's prior reviews. In September 2014, GAO reported on the Department of Veterans Affairs' (VA) Program of Comprehensive Assistance for Family Caregivers (Family Caregiver Program) and found that the program office had limitations with its information technology (IT) system—the Caregiver Application Tracker (CAT). Specifically, the program did not have ready access to workload data that would allow it to monitor the effects of the program on VA medical centers' resources. VA has initiated various projects since 2015 to implement a new system, but has not yet been successful in its efforts. (See figure.) Specifically, in July 2015 VA initiated a project to improve the reliability of CAT's data, called CAT Rescue. However, the department reported in January 2017 that it had identified numerous defects during system testing. The project ended in April 2018 before any new system capabilities were implemented. A companion project was initiated in September 2015 to develop the Caregivers Tool (CareT), a new system intended to replace CAT. The CareT project was expected to use improved data from CAT Rescue, while also adding new system capabilities. However, the user acceptance testing of CareT identified the need for the department to develop more system capabilities than originally planned. Further, VA reported that implementing a system by October 1, 2018, as specified in the Maintaining Internal Systems and Strengthening Integrated Outside Networks Act of 2018 (MISSION Act), was not feasible. Subsequently, VA terminated CareT in February 2019. The department initiated another project in March 2019 to implement a new system, the Caregiver Record Management Application (CARMA). GAO has ongoing work to evaluate the department's efforts to implement an IT system to support the Family Caregiver Program as required by the MISSION Act. GAO's prior work has determined that successfully overcoming IT acquisition challenges can best be achieved when critical success factors are applied. These factors can serve as a model of best practices that VA could apply to enhance the likelihood that the acquisition of a replacement IT system for the Family Caregiver Program will be successfully achieved. Examples of these critical success factors include, maintaining active engagement of program officials with stakeholders, involving end users and stakeholders in the development of requirements, and ensuring participation of end users in testing system functionality prior to formal end user acceptance testing. GAO recommended in 2014 that VA expedite the process for identifying and implementing an IT system that would fully support the Family Caregiver Program. VA concurred with the recommendation and subsequently began taking steps to implement a replacement system. The recommendation remains open.", "document_type": "gao"}
{"report": "Most private employers subject to Title VII of the Civil Rights Act of 1964 with 100 or more employees, and all federal contractors who have 50 or more employees and meet certain other requirements, must submit data to the EEOC on the racial/ethnic and gender characteristics of employees by occupations for a range of industries, including financial services. Employers are required to submit these data to EEOC every year using the EEO-1 report. EEOC requires employers to use the North American Industry Classification System to classify their industry. Under this system, the financial services industry includes the following five sectors: Credit intermediation and related activities (banks and other credit institutions), which include commercial banks, thrifts, and credit unions; Securities and other activities, which includes firms that bring together buyers and sellers of securities and commodities and offer financial advice; Insurance firms and agents that provide protection against financial risks to policyholders; Funds and trusts, which include investment trusts and holding Monetary authorities, including central banks. Beginning in 2007, EEOC changed its requirements for reporting data on managers. Specifically, employers were required to report separately on senior-level management positions rather than combining data on senior- level managers with data for first- and mid-level managers, as had been the practice until 2007. Employers are required to review EEOC guidance describing the two management positions and determine how their firm’s job positions fit into these classifications. In a January 2005 report, we identified a set of nine leading practices that should be considered when an organization is developing and implementing diversity management. They are (1) commitment to diversity as demonstrated and communicated by an organization’s top leadership; (2) the inclusion of diversity management in an organization’s strategic plan; (3) diversity linked to performance, making the case that a more diverse and inclusive work environment could help improve productivity and individual and organizational performance; (4) measurement of the impact of various aspects of a diversity program; (5) management accountability for the progress of diversity initiatives; (6) succession planning; (7) recruitment; (8) employee involvement in an organization’s diversity management; and (9) training for management and staff about diversity management. In 2017, we reported that industry representatives confirmed that these nine practices are still relevant. As we reported in November 2017, at the overall management level, representation of minorities in the financial services industry increased from 2007 through 2015, though representation varied by individual minority groups (see fig. 1). Specifically, minorities’ representation in overall management positions increased by 3.7 percentage points. Asians had the largest gains since 2007, increasing their representation among managers from 5.4 percent to 7.7 percent. Hispanics made smaller gains; their representation among managers increased from 4.8 percent to 5.5 percent. In contrast, the proportion of African-Americans in management positions decreased from 6.5 percent to 6.3 percent. Representation of minorities also increased between different levels of management from 2007 through 2015 (see fig. 2). Minority representation among first-and mid-level managers increased by 3.7 percentage points. In contrast, representation of minorities among senior- level management increased at a slower pace during this period (1.7 percentage points). Minority representation among senior-level managers remained considerably lower than among first- and mid-level managers. Among first- and mid-level managers, representation of Asians experienced the largest increase from 2007 through 2015 (2.6 percentage points). Hispanic representation increased by less than 1 percentage point, while African-American representation slightly decreased by 0.3 percentage point. In addition, among senior-level managers, representation of each racial and ethnic group changed by less than 1 percentage point. We also reported in November 2017 that representation of women at the overall management level had generally remained unchanged. From 2007 through 2015, women represented about 45 percent of overall management. Representation of each racial and ethnic group varied by gender during this time period. For example, among minority women, African-American women consistently had the highest representation in overall management (about 4 percent of managers per year). Among minority men, Asian men consistently had highest representation in overall management (3.1 percent to 4.6 percent of all managers). The proportion of men and women within various levels of management remained unchanged from 2007 through 2015, though there were some increases in the representation of both minority women and minority men. During this timeframe, women represented around 48 percent of first-and mid-level managers and about 29 percent of senior-level managers. Among first- and mid-level management positions, the representation of minority women increased by 1.6 percentage points and the representation of minority men increased by 2.2 percentage points (see fig. 3). Among senior-level management positions, representation of minority women and minority men increased by smaller amounts (0.3 percentage points and 1.5 percentage points, respectively). In November 2017, we reported that management-level diversity varied across sectors within the financial services industry. Minorities’ representation in overall management increased in all four sectors of the financial services industry (see fig. 4). For example, representation of minorities in the banks and other credit institutions sector increased by 3.1 percentage points and 4.3 percentage points in the funds and trusts sector. Also, the representation of minorities in overall management was consistently the greatest in the banks and other credit institutions and lowest in the insurance sector. The representation of women in overall management also varied by financial services sector (see fig. 5). The insurance sector consistently had the highest proportion of women in management positions, followed by banks and other credit institutions. The proportion of women in management decreased in each sector except for the insurance sector where it increased by 1.9 percentage points from 47.7 percent to 49.6 percent. Our November 2017 report found that the representation of minorities in overall management positions increased as firm size (number of employees) increased, whereas the representation of women in management generally remained the same across firm size. More specifically, in 2007, the representation of minorities in overall management was nearly 5 percentage points greater in firms with 5,000 or more employees compared to firms with 100–249 employees. By comparison, in 2015, the representation of minorities in overall management was about 6 percentage points greater in firms with 5,000 or more employees compared to firms with 100–249 employees. Across firms of different sizes, the representation of women in management positions in 2015 was generally the same as it was in 2007. Our November 2017 report found that from 2007 through 2015, representation of minorities in all levels of management increased in the financial services sector, the professional services sector, and the overall private sector. However, among first- and mid-level managers, representation of minorities increased at a lower rate in the financial services sector during this time period (3.7 percentage points) than in the professional services sector (7.5 percentage points) and slightly lower than the overall private sector (3.8 percentage points) . In addition, the financial services sector generally had a greater proportion of women in management compared to the overall private sector and professional services sector. For example, women represented 36.7 percent and 38.2 percent of first- and mid-level managers in the professional services sector and overall private sector, respectively, in 2015. As previously mentioned, women represented about 48 percent of first- and mid- level managers in the financial services sector from 2007 through 2015. Potential employees for the financial services industry can come from a range of academic and professional backgrounds. Financial firm representatives we spoke to for our November 2017 report told us that undergraduate or graduate degrees are an important consideration for employment. Some firm representatives also told us that while graduates with Master of Business Administration (MBA) degrees are an important pool of talent, firms seek students with a variety of degrees. We also found that from 2011 through 2015, about one-third of the external pool of potential talent for the financial services industry—that is, those obtaining undergraduate or graduate degrees—were racial/ethnic minorities (see fig. 6). Additionally, rates of attainment of bachelor’s, master’s, and MBA degrees by racial/ethnic minorities all increased during this time period. For example, minorities’ representation among those who attained an MBA increased from 35.6 to 39.2 percent. Furthermore, from 2011 through 2015, minority women consistently earned a greater proportion of master’s and MBA degrees compared to minority men. Additionally, we found that from 2011 through 2015, a majority of those obtaining undergraduate or graduate degrees have been women (see fig. 7). For example, women consistently earned about 58 percent of bachelor’s degrees, just over 60 percent of master’s degrees, and about 45 percent of MBA degrees during this time period. As we reported in November 2017, the internal pool of potential talent for the financial services industry is known as the “internal pipeline” of staff that could potentially move into management positions. There are two nonmanagement job categories in the financial services sector that are considered to be part of the internal pipeline: professional and sales positions. From 2007 through 2015, EEOC data show that minorities’ representation in professional and sales positions had changed over time, but had generally been greater than minorities’ representation in overall management positions. Similarly, EEOC data over the same timeframe show that representation of women in professional positions in the financial services industry had generally been greater than women’s representation in overall management. For example, from 2007 through 2015, women consistently represented about 50 percent of all employees in professional positions and about 45 percent of overall management. The percentage of women in sales positions in the financial industry had generally been lower, at about 40 percent. Representatives from financial services firms and organizations that advocate for women or racial/ethnic minorities who we spoke to for our November 2017 report described a variety of challenges to recruiting a diverse workforce for the financial services sector. These challenges included negative perceptions of the financial services industry that might discourage potential candidates and a lack of awareness of career paths in the industry. Research we reviewed and representatives we spoke with identified several practices believed or found to be effective for recruiting women and racial/ethnic minorities, which included: Recruiting students from a broad group of schools and academic disciplines. Representatives from three firms stated that they were increasingly hiring and interested in recruiting students from a variety of academic disciplines, such as liberal arts or science and technology. For example, representatives from one firm said that they were interested in candidates with critical thinking skills, and that technical skills could be taught to new employees. Additionally, representatives from several firms noted the importance of recruiting at a broad group of schools, not just a small number of elite universities. Offering programs to increase awareness of careers in financial services. Several representatives of financial firms told us that they had established relationships with high school students to expose diverse students to the financial services field. For example, representatives from one firm described a program that pairs high school students with a mentor from the firm. Additionally, a 2016 consulting firm report on women in financial services organizations in 32 countries found that a majority of asset managers who were interviewed thought it was important for financial services firms to educate students about careers in financial services. Financial services firms and other sources also noted challenges to retaining women and racial/ethnic minorities. For example, some representatives of financial firms noted that employee resistance, particularly from middle-managers, poses a challenge to diversity efforts. In addition, officials from some organizations we interviewed noted that unconscious bias can negatively affect women and minorities. As we noted in our November 2017 report, according to reports on diversity, representatives from financial services firms and other stakeholders, certain practices that may help improve the retention of women and racial/ethnic minorities, included: Establishing management-level accountability. Representatives from three financial services firms told us that management should be held accountable for workforce diversity goals. For example, two representatives discussed the use of a “diversity scorecard,” which is a set of objectives and measures derived from a firm’s overall business strategy and linked to its diversity strategy. Additionally, one firm representative noted that tying senior managers’ compensation to diversity goals had been an effective practice for retaining women and minorities. Researchers have noted that efforts to establish organizational responsibility for diversity have led to the broadest increases in managerial diversity. Assessing Data on Workforce Diversity. Financial services firms and organizations we talked to generally agreed that assessing demographic data to understand a firm’s diversity is a useful practice. All of the financial services firms we interviewed agreed on the importance of analyzing employee data. Several firms stated that it is important for organizations to understand their progress on workforce diversity–and, if data trends indicate problems, such as retention issues, they then can take steps to address them. Representatives of firms and organizations that advocate for diversity differed on the benefits of making demographic data public. Representatives of one organization said requiring businesses to be transparent about their workforce data creates incentives to improve the diversity of their workforce. However, representatives of two financial firms expressed concerns that publicly disclosing firm-level employee characteristics would not be beneficial to businesses. For example, one representative noted that publicly disclosing that firms are not diverse could damage their reputation and make improvement of workforce diversity more difficult. In closing, I would like to thank you for the opportunity to discuss trends in management-level diversity in the financial services industry. I look forward to working with this subcommittee on these important issues. Chairwoman Beatty, Ranking Member Wagner, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. For further information on this testimony, please contact Daniel Garcia- Diaz at (202) 512-8678 or GarciaDiazD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Lisa Moore (Assistant Director), Christine Ramos (Analyst in Charge), Kay Kuhlman, Jill Lacey, Tovah Rom, Jena Sinkfield, and Tyler Spunaugle. Financial Services Industry: Trends in Management Representation of Minorities and Women and Diversity Practices, 2007—2015. GAO-18-64. Washington, D.C.: November 8, 2017. Investment Management: Key Practices Could Provide More Options for Federal Entities and Opportunities for Minority- and Women-Owned Asset Managers. GAO-17-726. Washington, D.C.: September 13, 2017. Corporate Boards: Strategies to Address Representation of Women Include Federal Disclosure Requirements. GAO-16-30. Washington, D.C.: December 3, 2015. Federal Home Loan Banks: Information on Governance Changes, Board Diversity, and Community Lending. GAO-15-435. Washington, D.C.: May 12, 2015. Diversity Management: Trends and Practices in the Financial Services Industry and Agencies after the Recent Financial Crisis. GAO-13-238. Washington, D.C.: April 16, 2013. Federal Reserve Bank Governance: Opportunities Exist to Broaden Director Recruitment Efforts and Increase Transparency. GAO-12-18. Washington, D.C.: October 19, 2011. Financial Services Industry: Overall Trends in Management-Level Diversity and Diversity Initiatives, 1994—2008. GAO-10-736T. Washington, D.C.: May 12, 2010. Financial Services Industry: Overall Trends in Management-Level Diversity and Diversity Initiatives, 1993—2004. GAO-06-617. Washington, D.C.: June 1, 2006. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The financial services industry is a major source of employment that affects the economic well-being of its customers and the country as a whole. As the makeup of the U.S. workforce continues to diversify, many private sector organizations, including those in the financial services industry, have recognized the importance of recruiting and retaining minorities and women in key positions to improve business or organizational performance and better meet the needs of a diverse customer base. However, questions remain about the diversity of the workforce in the financial services industry. This statement is based on GAO's November 2017 report on changes in management-level diversity and diversity practices in the financial services industry. This statement summarizes (1) trends in management-level diversity in the financial services industry, (2) trends in diversity among potential talent pools, and (3) challenges financial services firms identified in trying to increase workforce diversity and practices they have used to address those challenges. In November 2017, GAO reported that overall management representation in the financial services industry increased marginally for minorities and remained unchanged for women from 2007 to 2015. Similar trends also occurred at the senior-level management of these firms. For example, women represented about 29 percent of senior-level managers throughout this time period. As shown below, representation of minorities in senior management increased slightly, but each racial/ethnic group changed by less than 1 percentage point. The diversity of overall management also varied across the different sectors of the financial services industry. For example, the banking sector consistently had the greatest representation of minorities in overall management, whereas the insurance sector consistently had the highest proportion of women in overall management. As GAO reported in November 2017, potential employees for the financial services industry, including those that could become managers, come from external and internal pools that are diverse. For example, the external pool included those with undergraduate or graduate degrees, such as a Master of Business Administration. In 2015, one-third of the external pool were minorities and around 60 percent were women. The internal talent pool for potential managers included those already in professional positions. In 2015, about 28 percent of professional positions in financial services were held by minorities and just over half were held by women. Representatives of financial services firms and other stakeholders GAO spoke to for its November 2017 report described challenges to recruiting and retaining members of racial/ethnic minority groups and women. They also identified practices that could help address those challenges. For example, representatives from several firms noted that an effective practice is to recruit and hire students from a broad group of schools and academic disciplines. Some firms also described establishing management-level accountability to achieve workforce diversity goals. Firm representatives and other stakeholders agreed that it is important for firms to assess data on the diversity of their employees but varied in their views on whether such information should be shared publicly.", "document_type": "gao"}
{"report": "The FSM and RMI are independent countries about 3,000 miles southwest of Hawaii. The FSM is a federation of four semiautonomous states—Chuuk, Kosrae, Pohnpei, and Yap—whose population and income vary widely. Chuuk, the largest state by population, has the lowest per-capita gross domestic product (GDP). Overall, the FSM had a 2016 population of approximately 102,000 and a GDP per capita of about $3,200. The RMI’s 2016 population was approximately 54,000 with a GDP per capita of about $3,600. The RMI’s most recent census, in 2011, found that approximately three-quarters of the population lived in Majuro, the nation’s capital, and on the island of Ebeye in the Kwajalein Atoll. Table 1 shows the FSM’s, FSM states’, and RMI’s estimated populations and annual GDP per capita in fiscal year 2016. U.S. relations with the FSM and the RMI began during World War II, when the United States ended Japanese occupation of the region. Starting in 1947, the United States administered the region under a United Nations trusteeship. In 1986, after a period of negotiations, the United States entered into a compact of free association with the FSM and RMI that provided for economic assistance to the two countries, secured U.S. defense rights, and allowed FSM and RMI citizens to migrate to the United States. In 2003, after a period of negotiations, the United States approved separate amended compacts with the FSM and the RMI that went into effect on June 25, 2004, and May 1, 2004, respectively. The amended compacts’ implementing legislation authorized and appropriated direct financial assistance to the FSM and the RMI in fiscal years 2004 through 2023, with the base amounts decreasing in most years. The legislation also provided for partial inflation adjustment of the base amount of compact sector grants and trust fund contributions each year. As the base amount of compact sector grants decreases, the trust fund contributions generally increase by an equivalent amount. Because the annual inflation adjustment is less than full inflation, the value of compact sector grants declines in real terms. Figure 1 shows the amount of compact sector grants and trust fund contributions each fiscal year from 2004 through 2023. The amended compacts and associated fiscal procedures agreements require that compact sector grants support the countries in six core sectors—education, health, infrastructure, environment, private sector development, and public sector capacity building—with priority given to the education and health sectors. These grants are described in section 211(a) of each compact and are referred to as compact sector grants or 211(a) grants. Section 211(b) of the RMI compact further states that the RMI must target a specified amount of grants to Ebeye and other Marshallese communities within Kwajalein Atoll. The RMI military use and operating rights agreement (MUORA) states that the Kwajalein- related funds provided to the RMI in the compacts shall be provided through fiscal year 2023 and thereafter for as long as this agreement remains in effect. The amended compacts and their subsidiary trust fund agreements provided that each trust fund is to be managed by a compact trust fund committee. Each compact trust fund committee includes representatives from both the United States and the respective country, but the United States is required by the terms of the trust fund agreements to hold the majority of votes on each committee. The Director of Interior’s Office of Insular Affairs serves as the chair of each committee. Trust fund committee responsibilities include overseeing fund operation, supervision, and management; investing and distributing the fund’s resources; and concluding agreements with any other contributors and other organizations. As part of this oversight, the committees are to establish an investment and distribution policy. The committees are also to determine fiscal procedures to be used in implementing the trust fund agreements on the basis of the fiscal procedures used for compact grant administration, unless otherwise agreed by the parties to the agreement. The U.S.–FSM and U.S.–RMI trust fund agreements allow for the agreements to be amended in writing at any time, with mutual consent of the governments. However, the U.S. legislation implementing the amended compacts requires that any amendment, change, or termination of all, or any part, of the compact trust fund agreements shall not enter into force until incorporated into an act of Congress. The compact trust fund agreements state that no funds, other than specified trust fund administrative expenses, may be distributed from the funds before October 1, 2023. From fiscal year 2024 onward, the maximum allowed disbursement from each compact trust fund is the amount of the fiscal year 2023 annual grant assistance, as defined by the trust fund agreement, with full inflation adjustment. In addition, the trust fund committees may approve additional amounts for special needs. The RMI compact trust fund agreement excludes Kwajalein-related assistance, defined in section 211(b) of the RMI compact, from the calculation of the allowed disbursement. Although the compact trust fund agreements state the maximum allowable disbursement level, they do not establish or guarantee a minimum disbursement level. Each country’s compact trust fund consists of three interrelated accounts: the “A” account, the “B” account, and the “C” account. The A account is the trust fund’s corpus and contains the initial, and any additional, U.S. and FSM or RMI contributions; contributions from other countries; and investment earnings. No funds, other than specified trust fund administrative expenses, may be disbursed from the A account. The B account is the trust fund’s disbursement account and becomes active in fiscal year 2023. All income earned in 2023 will be deposited in the B account for possible disbursement in 2024. Each subsequent year’s investment income will similarly be deposited in the B account for possible disbursement the following year. If there is no investment income, no funds will be deposited in the B account for possible disbursement the following year. The C account is the trust fund’s buffer account. Through 2022, any annual income exceeding 6 percent of the fund balance is deposited in the C account. The size of the C account is capped at three times the amount of the estimated annual grant assistance in 2023, including estimated inflation. From 2023 onward, if annual income from the A account is less than the previous year’s disbursement, adjusted for inflation, the C account may be tapped to address the shortfall. After 2023, any funds in the B account in excess of the amount approved for disbursement the following fiscal year are to be used to replenish the C account as needed, up to the maximum size of the account. If there are no funds in the C account and no prior-year investment income in the B account, no funds will be available for disbursement to the countries the following year. Figure 2 shows the compact trust fund account structure and associated rules. According to the U.S. trust fund agreements with the FSM and the RMI, contributions from other donors are permitted. In May 2005, Taiwan and the RMI reached an agreement that Taiwan would contribute a total of $40 million to the RMI’s compact trust fund A account between 2004 and 2023. A “D” account may also be established to hold any contributions by the FSM and the RMI governments of revenue or income from unanticipated sources. According to the trust fund agreements, the D account must be a separate account, not mixed with the rest of the trust fund. Only the RMI has a D account, governed in part by the agreement between Taiwan and the RMI. The amended compacts’ implementing legislation incorporates, by reference, related agreements extending programs and services to the FSM and RMI. The programs and services agreement with each country identifies the following programs and services as being available to each country: U.S. postal services, weather services, civil aviation, disaster preparedness and response, and telecommunications. Each programs and services agreement extends for 20 years from the compact’s entry into force. The agreement with the FSM ends on June 24, 2024, and the agreement with the RMI ends on April 30, 2024. The amended compacts’ implementing legislation (Pub. L. No. 108-188) and other U.S. legislation authorize other U.S. grants, programs, and services for the FSM and RMI. Pub. L. No. 108-188 authorized an annual supplemental education grant (SEG) for the FSM and RMI in fiscal years 2005 through 2023, to be awarded in place of grants formerly awarded to the countries under several U.S. education, health, and labor programs. The FSM and RMI are not eligible for the programs replaced by the SEG during these years. Unlike the compact sector grants, the amended compacts’ implementing legislation authorized the SEG but did not appropriate funds for it. Funding for the SEG is appropriated annually to the U.S. Department of Education (Education) and is transferred to Interior for disbursement. Other provisions of the amended compacts’ implementing legislation, as well as other U.S. law, make the FSM and RMI eligible for a number of additional programs. As of fiscal year 2016, compact sector grants and the SEG, each of which end in 2023, supported a substantial portion of government expenditures in the FSM and RMI. Compact sector grants and the SEG supported about one-third of all FSM government expenditures. The four FSM states relied on these grants to a greater extent than the FSM national government does. In the RMI, compact sector grants and the SEG supported about one-quarter of all government expenditures. The end of the compacts’ programs and services agreements in 2024 would also require the FSM and RMI to bear additional costs to provide services currently provided by the United States under the agreements. Compact sector grants, the SEG, and other U.S. grants supported almost half of FSM national and state government expenditures in fiscal year 2016. Compact sector and supplemental education grants that end in 2023 supported approximately one-third of total FSM national and state government expenditures in fiscal year 2016, while other U.S. grants supported an additional 15 percent of total FSM government expenditures (see fig. 3). While the supplemental education grant ends in 2023, the FSM would be eligible for some of the programs that the supplemental education grant replaced after 2023. A small number of other U.S. grants also end in 2023. See GAO-19-648T, app. I, for a discussion of grants and programs that do and do not end in 2023. In fiscal year 2016, compact sector and supplemental education grants that end in 2023 supported a larger proportion of FSM state governments’ expenditures than of the FSM national government’s expenditures. Compact sector grants and the SEG supported 8 percent of national government expenditures but supported 50 percent or more of each state’s government expenditures. Among the FSM states, Chuuk, which has both the largest population and the lowest per-capita income in the FSM, had the highest percentage of expenditures supported by U.S. grants. (See table 2 for a summary of FSM national and state government expenditures supported by compact sector grants and the SEG and by other U.S. grants.) Compact sector and supplemental education grants that end in 2023 supported approximately 25 percent of the RMI’s $123.5 million in government expenditures in fiscal year 2016, while other U.S. grants supported an additional 8 percent. Kwajalein-related compact grants that do not end in 2023 supported an additional 3 percent (see fig. 4). While the supplemental education grant ends in 2023, the RMI would be eligible for some of the programs that the supplemental education grant replaced after 2023. A small number of other U.S. grants also end in 2023. See GAO-19-648T, app. I, for a discussion of grants and programs that do and do not end in 2023. FSM and RMI budgets would be further affected if the countries assumed responsibility for providing programs and services currently provided by the United States. The following describes the status after 2023 of U.S. grants, programs, and services in the FSM and RMI under current law: Compact sector grants are scheduled to end in 2023, but the RMI MUORA extends the time frame of Kwajalein-related compact grants for as long as the MUORA is in effect. The SEG and additional grants identified in the amended compacts’ implementing legislation are scheduled to end in 2023. Also, after fiscal year 2023, the FSM and RMI will no longer be eligible for some programs that the SEG replaced, including Head Start (early childhood education, health, and nutrition services for low-income children and their families). The compact-related programs and services agreements with each country will end in 2024. However, some U.S. agencies, such as the National Weather Service, Federal Aviation Administration, and U.S. Agency for International Development, may continue to provide programs and services similar to those provided in the agreement under other authorities. The FSM and RMI will generally remain eligible for other programs identified in the amended compacts’ implementing legislation. These programs include U.S. Department of Agriculture (USDA) Rural Utilities Service grant and loan programs and U.S. Department of Education Pell grants for higher education and grants under Part B of the Individuals with Disabilities Education Act for children with disabilities. The FSM and RMI will remain eligible for additional programs we identified that have been provided under other current U.S. laws. Examples of these programs include USDA housing assistance programs and multiple public health, medical, and disease control and prevention grants provided by the U.S. Department of Health and Human Services. See appendix I for more information about the status after 2023 of U.S. grants, programs, and services in the FSM and RMI under current law. Our May 2018 projections for the compact trust funds showed that after fiscal year 2023, the funds are unlikely to provide maximum annual disbursements and may provide no disbursements at all in some years. The risk of disbursements below the maximum and the risk of zero disbursements increase over time for both funds. Potential strategies we analyzed in our May 2018 report would reduce or eliminate the risk of the compact trust funds’ experiencing years of zero disbursement. However, all of the potential strategies would require the countries to exchange a near-term reduction in resources for more-predictable and more- sustainable disbursements in the longer term. Our May 2018 projections for the FSM and RMI compact trust funds after 2023 indicated that, given their balance at the end of fiscal year 2017 and current compact trust fund rules—the baseline scenario—the funds will be unable to provide maximum disbursements (equal to the inflation- adjusted amount of annual grant assistance in 2023) in some years and unable to provide any disbursement at all in some years, with the likelihood of zero disbursement in a given year increasing over time. The compact trust funds’ C account—designed as a buffer to protect disbursements from the B account in years when the funds do not earn enough to fund the disbursement—could be exhausted by a series of years with low or negative annual returns. Since current rules do not allow disbursements from the compact trust fund corpus (the A account), exhaustion of the C account would result in zero disbursement in years when fund returns are zero or negative. Thus, there may be no funds available to disburse even if the funds’ A accounts have a balance. As a result of low or zero disbursements, the countries could face economic and fiscal shocks and significant challenges in planning programs and budgets. Since we published our May 2018 report, an additional year of compact trust fund performance data and updated estimates of future inflation have become available; however, the updated information does not alter the conclusions we presented in May 2018. The updated data and inflation estimates change our model’s assumptions about the current compact trust fund balance, size of future U.S. contributions to the FSM and RMI compact trust funds, annual grant assistance in fiscal year 2023, and C account balance—each of which are relevant variables for our analysis. However; the updated variables would result in only slight changes to our 2018 report’s projections of future compact trust fund performance presented in this testimony and do not alter our broader conclusions about future risks to the compact trust funds. FSM compact trust fund projections. In May 2018, our model projected that, given the baseline scenario and a 6 percent net return, the FSM compact trust fund will experience declining disbursements relative to the maximum allowable disbursements and an increasing chance of zero disbursements. (See app. I of GAO-18-415 for a full description of our methodology, and see app. V of GAO-18-415 for the baseline results with alternative net returns.) Projected disbursements. We projected that the FSM compact trust fund will, on average, be able to provide disbursements equal to 82 percent of the maximum allowable disbursement—the inflation- adjusted amount of 2023 annual grant assistance—in its first decade of disbursements. The likely average disbursement falls to 49 percent of the maximum in the next decade and falls further in subsequent decades. In addition, the amount available for disbursement may fluctuate substantially from year to year. Depending on the compact trust fund’s performance in the previous year, disbursements may be higher or lower than the average amount if the balance in the C account is not sufficient to provide additional disbursements. Likelihood of providing zero disbursement. We projected a 41 percent likelihood that the FSM compact trust fund will be unable to disburse any funds in 1 or more years during the first decade of trust fund disbursements. This likelihood increases over time, rising to 92 percent in fiscal years 2054 through 2063. Figure 5 shows our May 2018 projections of the FSM compact trust fund’s average disbursements as a percentage of maximum disbursement and the likelihood of 1 or more years of zero disbursement, given the baseline scenario and a 6 percent net return. RMI compact trust fund projections. In May 2018, our model projected that, given the baseline scenario and a 6 percent net return, the RMI compact trust fund will experience declining disbursements relative to the maximum allowable disbursements and an increasing chance of zero disbursements. Projected disbursements. We projected that in its first decade of disbursements, the RMI compact trust fund will, on average, be able to provide disbursements nearly equal to the inflation-adjusted amount of 2023 annual grant assistance as defined by the trust fund agreement—the maximum allowable. However, in each subsequent decade, the projected disbursements as a percentage of the maximum disbursements decline by about 10 percentage points. In addition, from year to year, the amount available to disburse may fluctuate substantially. Depending on the compact trust fund’s performance in the previous year, disbursements may be higher or lower than the average amount if the balance in the C account is not sufficient to provide additional disbursements. Likelihood of providing zero disbursement. We projected a 15 percent likelihood that the RMI compact trust fund will be unable to disburse any funds in 1 or more years during the first decade of trust fund disbursements. This likelihood increases over time, rising to 56 percent in fiscal years 2054 through 2063. Figure 6 shows our May 2018 projections of the RMI compact trust fund’s average disbursements as a percentage of maximum disbursement and its likelihood of 1 or more years of zero disbursement, given the baseline scenario and a 6 percent net return. For our May 2018 report, we conducted a series of simulations to determine the likely effects of potential strategies for improving the outlook of the FSM and RMI compact trust funds. For example, we developed and analyzed potential strategies in which: annual disbursements are reduced below the maximum allowable additional annual contributions are made to the trust fund prior to the end of fiscal year 2023, and the trust fund agreement disbursement policies are modified to limit the annual disbursement to a fixed percentage of the fund’s moving average balance over the previous 3 years, up to the maximum disbursement amount defined by the current trust fund agreement. All of the potential strategies we analyzed would reduce or eliminate the risk of the compact trust funds experiencing years of zero disbursement. However, some of the potential strategies may require changing the trust fund agreements and all of the potential strategies would require the countries to exchange a near-term reduction in resources for more- predictable and more-sustainable disbursements in the longer term. (See app. VII of our May 2018 report for detailed results of our analysis.) The trust fund committees have not taken the actions we recommended in 2018 to prepare for the 2023 transition to trust fund income. The compact trust fund committees have not yet prepared distribution policies, required by the trust fund agreements, which could assist the countries in planning for the transition to trust fund income. In addition, the committees have not established fiscal procedures for oversight of compact trust fund disbursements as required by the trust fund agreements. Further, the committees have not yet addressed a potential misalignment between the timing of their annual calculation of the amounts available to disburse and the FSM’s and RMI’s budget timelines, potentially complicating each country’s planning and management. The compact trust fund committees have not yet developed, as the compact trust fund agreements require, policies to guide disbursements from the trust funds after fiscal year 2023. Under the agreements, each trust fund committee must develop a distribution policy, with the intent that compact trust fund disbursements will provide an annual source of revenue to the FSM and RMI after the scheduled end of compact grant assistance. The trust fund committees could use distribution policies to address risks to each fund’s sustainability. For example, the committees have the discretion to disburse an amount below the established maximum. Our analysis of potential strategies for improving the funds’ outlook shows that reducing the size of disbursements would improve each compact trust fund’s long-term sustainability. Without a distribution policy that provides information about the size of expected disbursements, the FSM and RMI are hampered in their current and ongoing efforts to plan for the potential reduction in U.S. compact assistance after 2023. The compact trust fund committees have not yet established fiscal procedures for compact trust fund disbursements after fiscal year 2023. Each trust fund agreement requires the respective committee to determine the fiscal procedures to be used in implementing the trust fund agreement. The committees are to base their procedures on the compact fiscal procedures agreements, unless the parties to the trust fund agreement agree to adopt different fiscal procedures. No compact trust fund disbursements are to be made unless the committee has established such trust fund fiscal procedures. Without fiscal procedures in place, the trust fund committees will not be able to provide disbursements and the United States, the FSM, and the RMI will lack clear guidance to ensure oversight for trust fund disbursements. The timing for the trust fund committees’ calculation of the amounts available for annual disbursement to the FSM and the RMI after fiscal year 2023 does not align with the countries’ budget and planning timelines. The amounts available for disbursement in a given fiscal year cannot be determined until each fund’s returns have been determined at the end of the prior year. Further, if the disbursement amounts are calculated from audited fund returns as determined by annual audits required by the trust fund agreements, the amounts may not be determined until as late as March 31, 6 months into the fiscal year for which the disbursement is to be provided. However, both the FSM and the RMI government budget cycles are completed before the annual amounts available for disbursement will be known. As a result, the FSM and RMI would have to budget without knowing the amount to be disbursed, complicating their annual budget and planning processes. The trust fund committees, chaired by Interior, have discussed potential actions to address the recommendations in our May 2018 report. In May 2018, we made six recommendations to Interior—three parallel recommendations regarding each country’s trust fund. We recommended that the Secretary of the Interior ensure that the Director of the Office of Insular Affairs work with other members of the trust fund committees to: develop distribution policies, develop the fiscal procedures required by the compact trust fund address the timing of the calculation of compact trust fund disbursements. Interior concurred with our recommendations and has stated that it plans to implement them before the FSM and RMI transition to trust fund income in 2023. The FSM and RMI also concurred with our recommendations to Interior. According to the Trust Fund Administrator and Interior officials, the distribution policy was discussed at subsequent trust fund committee meetings—including the most recent, in May 2019. According to the trust fund administrator, trust fund representatives met with FSM and RMI representatives in January 2019 to discuss the status of the trust fund and future scenarios for its management. Interior officials further stated that discussions about trust fund policies and controls were frequent and ongoing among committee members and staffers as well as the trust fund manager and investment advisers. The FSM’s and RMI’s transition to relying on income from the compact trust funds will likely require significant budgetary choices. However, the lack of trust fund distribution policies, and the lack of alignment between the trust fund committees’ annual disbursement calculations and the countries’ budget cycles hamper the countries’ ability to plan for the transition. In addition, without the required fiscal procedures governing trust fund actions after 2023, the trust fund committees will be unable to make disbursements and the United States, the FSM, and the RMI will not have assurance of necessary oversight. However, as of June 2019, Interior had not implemented our recommendations to address these issues. Further, while Interior has continued to discuss possible actions to address our recommendations with the trust fund committees, it targeted implementation of our recommendations for 2023. Chairwoman Murkowski, Ranking Member Manchin, and Members of the Committee, this concludes my statement. I would be pleased to respond to any questions you may have. If you or your staff have any questions about this testimony, please contact David Gootnick, Director, International Affairs and Trade, at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Emil Friberg (Assistant Director), Ming Chen, Neil Doherty, Mark Dowling, Christopher Keblitis, Reid Lowe, Moon Parks, and Michael Simon. The amended compacts, compact–related agreements, the amended compacts’ implementing legislation, and other U.S. laws provide grants or eligibility for U.S. programs and services for the Federated States of Micronesia (FSM) and Republic of the Marshall Islands (RMI). The amended compacts provide compact sector, Kwajalein-related, and audit grants. Under current law, compact sector and audit grants are each scheduled to end in 2023, but the RMI military use and operating rights agreement (MUORA) extends the time frame of Kwajalein-related compact grants for as long as the agreement is in effect. The amended compacts’ implementing legislation provides additional grants, including authorizing a supplemental education grant (SEG), and identifies several specific U.S. programs as available to the FSM and RMI. Under current law, the additional grants end in 2023, but the statutory authorizations for some programs identified in Pub. L. No. 108-188 provide for the continued eligibility of the FSM and RMI to receive benefits under the programs. However, after fiscal year 2023, the FSM and RMI will no longer be eligible under current U.S. law for some programs that the SEG replaced. The compact-related programs and services agreements with each country identify additional programs and services that the United States makes available to the FSM and RMI. While these agreements will end in 2024, under current law, some U.S. agencies may continue to provide programs and services similar to those provided in the agreement under other authorities. Based on the status of current law, the FSM’s and RMI’s eligibility for other programs we identified that have been provided under other current U.S. laws will not change after fiscal year 2023. Under current law, compact sector grants provided to the FSM and the RMI under section 211(a) of the amended compacts are scheduled to end in 2023. However, the RMI is scheduled to continue to receive $7.2 million, partially inflation adjusted, related to the U.S. military base in Kwajalein Atoll and provided under section 211(b) of its compact. Under the terms of the RMI MUORA, the United States agreed to provide these Kwajalein-related grants for as long as the MUORA is in effect. The MUORA continues until 2066 and may be extended at the discretion of the United States until 2086. The amended RMI compact provides for $18 million, partially inflation adjusted, in annual payments to the RMI government to compensate for impacts from the U.S. Army Garrison– Kwajalein Atoll. These payments will continue for as long as the MUORA is in effect. Annual compact grants of up to $500,000 (not inflation adjusted) to each country to pay for required annual audits of compact grants are scheduled to end in 2023. See table 3 for a summary of compact sector, Kwajalein-related, and audit grants. The supplemental education grant (SEG) authorized by the amended compacts’ implementing legislation is scheduled to end in fiscal year 2023 and, under current law, FSM and RMI eligibility for most programs that the SEG replaced will not resume after fiscal year 2023. Absent changes to current law, the FSM and RMI will not be eligible after fiscal year 2023 for the following programs that the SEG replaced during fiscal years 2005 through 2023: U.S. elementary and secondary education grant programs, adult education and literacy programs, career and technical education programs, job training programs, and Head Start early education programs. However, under other provisions of current law, qualifying individuals in the FSM and RMI will be eligible after fiscal year 2023 for undergraduate education grants and work-study programs that the SEG replaced. See table 4. Although the programs and services agreements with the FSM and RMI will end in fiscal year 2024, current U.S. law enables U.S. agencies to continue providing some programs and services now provided under the agreements. After the agreements end, no current provisions of U.S. law will enable the Federal Emergency Management Agency (FEMA) to provide disaster response funding, enable the Federal Deposit Insurance Corporation to provide deposit insurance, or enable the U.S. Postal Service to provide the services that it currently provides to the FSM and RMI. However, the National Weather Service, the U.S. Department of Transportation’s (DOT) Federal Aviation Administration (FAA), and the U.S. Agency for International Development (USAID) could, under other legal authorities, provide services similar to those they now provide under the programs and services agreements. National Weather Service. The programs and services agreements authorize the National Weather Service to fund the operations of weather stations in the FSM and RMI, which it can continue to fund after the end of the agreements under other authorities, according to Department of Commerce officials. Federal Aviation Administration. The programs and services agreements authorize DOT’s FAA to provide technical assistance in the FSM and RMI, which it can continue to provide after the end of the agreements under other provisions of current U.S. law. However, DOT officials stated that FAA would require new bilateral agreements with the FSM and the RMI in order for the countries to continue to receive the civil aviation safety services that FAA currently provides under the programs and services agreements. The FAA would also seek reimbursement for any technical assistance it provides to the FSM and RMI. With regard to the civil aviation economic services provided under the programs and services agreements, DOT officials stated that, while the FSM and RMI could voluntarily decide to allow U.S. air carriers to continue operations in the FSM and RMI, new bilateral agreements would be needed to assure that result. U.S. Agency for International Development. Following a U.S. presidential disaster declaration, FEMA provides the funding for disaster relief and reconstruction, which is programmed through USAID. Under current law, FEMA funds will no longer be available to the FSM and RMI for this purpose once the agreements end; however, USAID will be able to provide foreign disaster assistance funding to the two countries under the same terms as it provides this assistance to other countries. After the programs and services agreements end, FEMA will be able to support disaster relief efforts only if USAID or the countries request such support on a reimbursable basis. In addition, according to State and Interior officials, telecommunications- related services that the two agencies provide to the FSM and RMI under the programs and services agreements will continue as long as the FSM and RMI provide appropriate authorization for such services. Table 5 shows the status after the programs and services agreements end of programs and services currently provided to the FSM and the RMI under the agreements. Although additional grants provided to the FSM and the RMI under the amended compacts’ implementing legislation will end in fiscal year 2023, the countries’ eligibility for programs now provided under that legislation will generally continue under current U.S. law. Grants provided under the amended compacts’ implementing legislation for (1) judicial training in the FSM and the RMI, and (2) agricultural and planting programs on the RMI’s nuclear-affected Enewetak Atoll are scheduled to end. However, under current U.S. law, legal authorities permitting the operation of other programs will remain available to the FSM and RMI after fiscal year 2023. Eligibility under these legal authorities continues either because the amended compacts’ implementing legislation does not specify an ending date or because other provisions in current U.S. law make the FSM and RMI eligible for the program. Programs provided in the amended compacts’ implementing legislation include U.S. Department of Agriculture Rural Utilities Service grant and loan programs; U.S. Department of Education Pell grants for higher education and grants under Part B of the Individuals with Disabilities Education Act for children with disabilities; programs for nuclear-affected areas in the RMI; and additional programs provided by the Departments of Commerce and Labor as well as law enforcement assistance provided by the U.S. Postal Service. See table 6 for a summary of the programs identified in the amended compacts’ implementing legislation and their status as of the end of fiscal year 2023. In addition to being eligible for the programs provided through the compact, its associated agreements, and the amended compacts’ implementing legislation, the FSM and RMI are also eligible for a number of programs under other provisions of current U.S. law. The FSM and RMI have each received funds from the U.S. Department of Agriculture for forestry and rural housing programs, multiple U.S. Department of Health and Human Services public health program grants, U.S. Department of the Interior technical assistance and historic preservation programs, and the DOT FAA airport improvement program, among others. Under current U.S. law, the legal authorities permitting the provision of these programs in the FSM and RMI would not necessarily change after 2023. Table 7 shows the FSM’s and RMI’s eligibility for these additional grants and programs under current law after fiscal year 2023.", "summary": "In 2003, the United States approved amended compacts of free association with the FSM and RMI, providing a total of $3.6 billion in economic assistance in fiscal years 2004 through 2023 and access to several U.S. programs and services. Compact grant funding, overseen by the Department of the Interior (Interior), generally decreases annually. However, the amount of the annual decrease in grants is added to the annual U.S. contributions to the compact trust funds, managed by joint U.S.-FSM and U.S.-RMI trust fund committees and chaired by Interior. Trust fund earnings are intended to provide a source of income after compact grants end in 2023. This testimony summarizes GAO's May 2018 report on compact grants and trust funds ( GAO-18-415 ). In that report, GAO examined (1) the use and role of U.S. funds and programs in the FSM and RMI budgets, (2) projected compact trust fund disbursements, and (3) trust fund committee actions needed to address the 2023 transition to trust fund income. For this testimony, GAO also reviewed key variables for its trust fund model as of June 2019 to determine whether these variables had substantially changed. In addition, GAO reviewed the status of Interior's response to GAO's May 2018 recommendations. The Federated States of Micronesia (FSM) and the Republic of the Marshall Islands (RMI) rely on U.S. grants and programs, including several that are scheduled to end in 2023. In fiscal year 2016, U.S. compact sector grants and supplemental education grants, both scheduled to end in 2023, supported a third of the FSM's expenditures and a quarter of the RMI's. Agreements providing U.S. aviation, disaster relief, postal, weather, and other programs and services are scheduled to end in 2024, but some U.S. agencies may provide programs and services similar to those in the agreements under other authorities. GAO's 2018 report noted that the FSM and RMI compact trust funds face risks and may not provide disbursements in some future years. GAO projected a 41 percent likelihood that the FSM compact trust fund would be unable to provide any disbursement in 1 or more years in fiscal years 2024 through 2033, with the likelihood increasing to 92 percent in 2054 through 2063. GAO projected a 15 percent likelihood that the RMI compact trust fund would be unable to provide any disbursement in 1 or more years in fiscal years 2024 through 2033, with the likelihood increasing to 56 percent in 2054 through 2063. Potential strategies such as reduced trust fund disbursements would reduce or eliminate the risk of years with no disbursement. However, some of these strategies would require changing the trust fund agreements, and all of the strategies would require the countries to exchange a near-term reduction in resources for more-predictable and more-sustainable disbursements in the longer term. Interior has not yet implemented the actions GAO recommended to prepare for the 2023 transition to trust fund income. The trust fund committees have not developed distribution policies, required by the agreements, which could assist the countries in planning for the transition to trust fund income. The committees have not developed the required fiscal procedures for oversight of disbursements or addressed differences between the timing of their annual determinations of the disbursement amounts and the FSM's and RMI's annual budget cycles. In its May 2018 report, GAO made three recommendations to Interior regarding each country's trust fund to address trust fund disbursement risks. Interior concurred with GAO's recommendations and discussed actions in response at subsequent trust fund committee meetings, with implementation targeted for 2023.", "document_type": "gao"}
{"report": "NNSA largely executes its missions at eight sites that comprise the nuclear security enterprise and that are managed by seven M&O contractors. These eight sites are three national security laboratories—Lawrence Livermore National Laboratory in California, Los Alamos National Laboratory in New Mexico, and Sandia National Laboratories in New Mexico and other locations; four nuclear weapons production plants—the Pantex Plant in Texas, the Y-12 National Security Complex in Tennessee, the Kansas City National Security Complex in Missouri, and tritium operations at DOE’s Savannah River Site in South Carolina; and the Nevada National Security Site, formerly known as the Nevada Test Site. As shown in figure 1, each of NNSA’s eight sites has specific responsibilities within the nuclear security enterprise. NNSA also executes portions of its missions across several other DOE sites, such as the Pacific Northwest National Laboratory in Washington and the Oak Ridge National Laboratory in Tennessee. At this time, NNSA’s common financial reporting efforts are focused on the eight sites, as required by the National Defense Authorization Act for Fiscal Year 2017. NNSA’s sites are owned by the federal government but managed and operated by M&O contractors. According to DOE, the use of M&O contracts is supported by an underlying principle: the federal government employs highly capable companies and educational institutions to manage and operate government-owned or -controlled scientific, engineering, and production facilities because these companies and educational institutions have greater flexibility than the government in bringing scientific and technical skills to bear. As we previously found, an M&O contract is characterized by, among other things, a close relationship between the government and the contractor for conducting work of a long-term and continuing nature. To support its missions, NNSA is organized into program offices that oversee the agency’s numerous programs. For example, the Office of Defense Programs oversees the B61-12 Life Extension Program, and the Office of Defense Nuclear Nonproliferation oversees the Nuclear Smuggling Detection and Deterrence Program. NNSA’s program offices are Defense Nuclear Nonproliferation; Safety, Infrastructure, and Operations; Defense Nuclear Security; Counterterrorism and Counterproliferation; and Naval Reactors. Mission-related activities are primarily overseen by these program offices, which are responsible for integrating the activities across the multiple sites performing work. NNSA field offices, co-located at the sites, oversee the day-to-day activities of the contractors as well as mission support functions such as safety. NNSA is subject to different cost accounting requirements than its seven M&O contractors. NNSA is required to follow Managerial Cost Accounting Standards. The principal purpose of Managerial Cost Accounting Standards is to determine the full cost of delivering a program or output to allow an organization to assess the reasonableness of this cost or to establish a baseline for comparison. The standards state that federal agencies should accumulate and report the costs of their activities on a regular basis for management information purposes. The standards also state that agencies should allow flexibility for agency managers to develop costing methods that are best suited to their operational environment. Such information is important to Congress and to NNSA managers as they make decisions about allocating federal resources, authorizing and modifying programs, and evaluating program performance. Separate standards—referred to as federal Cost Accounting Standards—govern how NNSA’s M&O contractors structure and account for their costs. Federal Cost Accounting Standards provide direction for the consistent and equitable distribution of a contractor’s costs to help federal agencies more accurately determine the actual costs of their contracts and the contractor’s costs associated with specific projects and programs. To comply with federal Cost Accounting Standards, M&O contractors classify costs as either direct or indirect when they allocate these costs to programs. Direct costs are assigned to the benefitting program or programs. Indirect costs—costs that cannot be assigned to a particular program, such as costs for administration and site support—are to be accumulated, or grouped, into indirect cost pools. The contractor is to estimate the amount of indirect costs to distribute to each program (accumulated into indirect cost pools) and make adjustments by the end of the fiscal year to reflect actual costs. The contractor is then to distribute these costs proportionally across all programs based on a rate in accordance with the contractor’s cost allocation model. The final program cost is the sum of the total direct costs plus the indirect costs distributed to the program. In implementing these allocation methods, federal Cost Accounting Standards provide contractors with flexibility regarding the extent to which they identify incurred costs directly with a specific program and how they collect similar costs into indirect cost pools and allocate them among programs. Therefore, different contractors may allocate similar costs differently because the contractors’ cost allocation models differ—that is, a cost classified as an indirect cost at one site may be classified as a direct cost at another. Because each contractor can allocate similar indirect costs differently and contractors may change the way they allocate indirect costs over time, it is difficult to compare contractors’ costs among sites and accurately calculate total program costs when work for a program is conducted at multiple sites. The seven NNSA M&O contractors and NNSA’s program offices account for and track costs differently. We previously found that NNSA’s M&O contractors have historically developed their own processes to manage and track costs for work at each site even when their work contributes to the same program. These processes have generally differed from the ones NNSA program offices have developed to describe the scope of its programs. This makes it difficult for NNSA and others to track and compare costs for analogous activities across programs, contractors, and sites. For example, in May 2018, we found that NNSA’s work breakdown structure for the B61-12 Life Extension Program and its $7.6 billion cost estimate (at that time) did not include $648 million in activities that were undertaken by other NNSA programs, such as research and development, test and evaluation activities, and infrastructure elements. Leading practices for developing work breakdown structures state that a work breakdown structure should include all activities that contribute to a program’s end product, and should not treat contributing activities separately. DOE’s and NNSA’s financial management and accounting system—the Standard Accounting and Reporting System (STARS)—provides budget execution, financial accounting, and financial reporting capabilities for the department. STARS is also integrated with other agency systems for procurement, funds distribution, travel, and human resources. The M&O contractors’ financial systems must be able to directly provide cost reports to NNSA’s financial management system. The primary source of cost data contained in STARS comes from summary-level cost reports provided by M&O contractors, which they report for NNSA’s appropriations at the budget and reporting code level. Program offices access STARS financial data through the DOE Office of the Chief Financial Officer’s integrated data warehouse. While financial data collected through STARS represent DOE’s official financial data, the data are not detailed and therefore may not satisfy the information needs of NNSA’s program offices. For example, STARS financial data do not differentiate labor costs from other programmatic costs, nor do they provide detailed information about the costs of activities that contribute to program costs. In addition, according to M&O contractor representatives, if one M&O contractor provides funding to another contractor, such as to conduct testing, NNSA does not have the ability in STARS to identify that funding was transferred. In the absence of an automated managerial cost accounting system that collects data from financial systems and relevant operating systems to consistently and uniformly produce useful cost information, NNSA’s program offices developed various systems, tools, and spreadsheets to track relevant cost information. Specifically, NNSA’s program offices separately collect cost information from M&O contractors that is more detailed than costs reported through STARS. Collecting these data requires M&O contractors to map, or “crosswalk,” their cost data to the work breakdown structures of one or more of NNSA’s program offices. Some program offices collect financial data through ad hoc data calls, rather than regular data calls. Some tools the program offices use include program management systems or spreadsheets designed to meet each program office’s programmatic, budgetary, and project requirements. For example, the Office of Defense Programs built the Enterprise Portfolio Analysis Tool in 2007 to capture financial data from the M&O contractors for its programs. Also, in 2007, officials from the Office of Defense Nuclear Nonproliferation developed a program management system designed to integrate and manage data such as scope, schedule, budget, and cost at the program level with greater detail than the data in STARS. The Office of Safety, Infrastructure, and Operations later adopted this system and called it the G2 program management system. M&O contractors use the G2 system to upload crosswalks of financial data for those program offices’ work breakdown structures after the costs were incurred. This process allows M&O contractors to report detailed financial data to the respective program offices every month. The process to track cost information is different for each program office and depends on the tool used and the information collected. However, for all program offices the process to track cost information is in addition to the financial reporting that M&O contractors provide for STARS (see fig. 2). To implement common financial reporting and standardize financial reporting by the M&O contractors across programs and sites, NNSA is pursuing an approach in which the agency collects M&O contractors’ financial data in a common reporting framework using an NNSA-wide data reporting and analysis tool. M&O contractors produce crosswalks of their financial data and submit the data to NNSA using a data reporting and analysis tool called CostEX. NNSA then stores the reported financial data in the DOE Office of the Chief Financial Officer’s integrated data warehouse. The Office of Defense Programs has used this process to collect financial data from the M&O contractors for its programs since fiscal year 2017. NNSA implemented this process for the broader common financial reporting effort in fiscal year 2018. Figure 3 illustrates NNSA’s data management process for common financial reporting. To implement common financial reporting, NNSA established a common reporting framework using agreed-upon work breakdown structures and common cost elements and definitions. However, in January 2019, we found that NNSA did not establish a common work breakdown structure for all of the participating program offices, although the agency had established 22 common cost elements and definitions. Specifically, the Offices of Defense Programs, Emergency Operations, Defense Nuclear Security, and Counterterrorism and Counterproliferation used NNSA’s common work breakdown structure, while the Offices of Safety, Infrastructure, and Operations and Defense Nuclear Nonproliferation used their own programmatic work breakdown structures. The M&O contractors crosswalk their internal financial data into a work breakdown structure for each of the participating program offices (either NNSA’s common work breakdown structure or a programmatic work breakdown structure) using common cost elements and definitions. The M&O contractors’ business systems capture their financial data at a more detailed level than is needed for common financial reporting. Each M&O contractor tracks financial data for its site based on how it manages the work using projects, tasks, and expenditure types. For example, M&O contractors collect time and attendance data from their employees based on the number of hours spent working on a project for the pay period. The M&O contractors aggregate this information across multiple employees to report on labor costs for a project. When the M&O contractors prepare their data for common financial reporting, site managers identify the component(s) of the applicable work breakdown structure and cost elements with which the project aligns and crosswalk their financial data to the NNSA structure using professional judgment. Figure 4 shows an example of how an M&O contractor crosswalks its financial data into an NNSA work breakdown structure in CostEX. After the M&O contractors submit their financial data in CostEX, NNSA performs data quality and accuracy checks of the M&O contractors’ data, referred to as “data validation” and “data reconciliation.” NNSA performs data validation using CostEX, which automatically checks each row for data quality—such as confirming that the correct contractor is entering data for the site—and formatting based on 45 validation checks. CostEX identifies data that do not pass the validation check as errors and rejects them, and the M&O contractor corrects and resubmits the data until it passes the validation check. NNSA performs data reconciliation with STARS using CostEX at the budget and reporting code level. CostEX extracts STARS data for selected budget and reporting codes and compares it with the data the M&O contractors submitted for common financial reporting. CostEX identifies data that differ from the STARS data by more than $1 as an error and rejects the data, and the M&O contractor corrects and resubmits the data until it passes the reconciliation check. According to NNSA officials, it is important for the agency to perform these data validation and reconciliation checks prior to accepting the M&O contractors’ financial data to ensure data quality. NNSA has made progress toward implementing common financial reporting across the nuclear security enterprise since our last report in January 2019, but it faces challenges in fully implementing the effort. We identified seven steps related to NNSA’s efforts to implement common financial reporting in our January 2019 report: (1) identifying an approach and developing a tool to implement common financial reporting, (2) developing a policy, (3) establishing common cost elements and definitions, (4) identifying and reporting costs for programs of record and base capabilities, (5) implementing a common work breakdown structure, (6) collecting financial data from the M&O contractors, and (7) publishing and analyzing data. To date, the agency has completed three steps but has not yet completed four others, as shown in table 1. As required by the National Defense Authorization Act for Fiscal Year 2017, NNSA is to implement common financial reporting by December 23, 2020, to the extent practicable. NNSA’s progress to implement common financial reporting in these seven steps since our January 2019 report is described below: Identify an approach and develop a tool to implement common financial reporting. NNSA identified an approach and developed a tool to implement common financial reporting prior to our January 2019 report. NNSA continues to use CostEX to collect financial data from the M&O contractors and stores the data in DOE’s integrated data warehouse. Develop a policy. NNSA developed a policy for common financial reporting. NNSA began developing the policy in October 2016 and approved it in February 2019. Establish common cost elements and definitions. NNSA established common cost elements and definitions prior to our January 2019 report. An NNSA official said NNSA established the cost elements and definitions based on data that the M&O contractors could readily provide from their business systems. In fiscal year 2019, NNSA used the established cost elements to collect the M&O contractors’ data and added a requirement for the contractors to report data on unpaid commitments. NNSA officials are considering adding cost elements in the future, such as additional details on labor categories. NNSA is working with the M&O contractors to ensure they can provide the additional data. Identify and report costs for programs of record and base capabilities. NNSA has not yet identified and reported costs for all programs of record or costs for base capabilities. The National Defense Authorization Act for Fiscal Year 2017 required NNSA to establish definitions and methodologies for identifying and reporting costs for programs of record and base capabilities as part of its efforts to implement common financial reporting. According to the program director for financial integration, NNSA establishes its programs of record in its congressional budget justification and other documents to align with agency appropriations, which include Weapons Activities, Defense Nuclear Nonproliferation, and Federal Salaries and Expenses. Through common financial reporting in fiscal year 2018, NNSA collected financial data from the M&O contractors for $8.9 billion of $13 billion from these appropriations. In May 2018, NNSA issued guidance that identified 25 base capabilities that the M&O contractors used to develop their site strategic plans. We reviewed the M&O contractors’ site strategic plans for 2018 and found that the contractors identified base capabilities for their sites, but did not include information about the costs to maintain each sites’ base capabilities. NNSA is working to determine whether or how to collect information on the cost of base capabilities through the M&O contractor site strategic planning process in coordination with the common financial reporting effort. We will continue to monitor NNSA’s progress in addressing this requirement. Implement a common work breakdown structure. NNSA has not yet implemented a common work breakdown structure across the program offices in the nuclear security enterprise, but plans to assess the feasibility of implementing a common structure in fiscal year 2020. The National Defense Authorization Act for Fiscal Year 2017 requires NNSA to develop a common work breakdown structure as part of its efforts to implement common financial reporting. In January 2019, we found that NNSA decided not to pursue a common work breakdown structure. Rather, NNSA collected financial data from the M&O contractors using a common work breakdown structure for four program offices and used different, programmatic work breakdown structures for two other program offices. As we found in January 2019, these two offices did not want to change their work breakdown structures to the common structure. For example, the Office of Safety, Infrastructure, and Operations did not want to change its work breakdown structure because it uses the structure for scope, schedule, and risk management, in addition to budget and cost. We recommended that NNSA implement a common work breakdown structure across its participating program offices because without doing so, NNSA could not ensure that its efforts would result in the collection of reliable, enterprise-wide financial data that satisfies the needs of Congress and enables NNSA to report the total costs of its programs. At the time of that report, NNSA neither agreed nor disagreed with the recommendation. The agency stated that it would continue to use its current approach, while focusing on enhancing analysis and reporting to provide comparative data across the enterprise. Once this was completed, NNSA planned to assess the effectiveness of the approach and evaluate what changes, if any, were necessary to the work breakdown structures to meet the overarching objectives of common financial reporting. In May 2019, in response to our recommendation, NNSA changed its approach and decided to conduct an assessment in fiscal year 2020 of the feasibility of implementing a common work breakdown structure across all participating program offices. To do so, NNSA plans to collect M&O contractors’ financial data in fiscal year 2020 using both the common work breakdown structure for all program offices and— specifically for the Offices of Safety, Infrastructure, and Operations and Defense Nuclear Nonproliferation—the programmatic work breakdown structures while it assesses the feasibility of a common work breakdown structure. NNSA decided to take this approach to assess the potential benefits while mitigating potential risks to the program offices that use the data collected through the programmatic work breakdown structures to oversee their programs. NNSA officials said that reporting the same data using two different work breakdown structures will require additional resources for the M&O contractors to prepare their data submissions, which NNSA does not view as a long-term solution for common financial reporting. NNSA planned to collect data using these two approaches in parallel starting in November 2019 and make a decision on whether to implement a common work breakdown structure across the nuclear security enterprise in March 2020. NNSA plans to assess the feasibility of implementing a common work breakdown structure using criteria such as (1) whether using a common work breakdown structure reduces burden on the M&O contractors, (2) how much it will cost NNSA to update other program management systems, (3) whether NNSA can collect financial data quickly enough to meet the needs of the program offices, and (4) whether financial data collected using the common work breakdown structure provides program offices with comparable data to support existing program analysis. Collect financial data from M&O contractors. Since our January 2019 report, the M&O contractors submitted their financial data for fiscal years 2018 and 2019 for the participating program offices using CostEX. However, NNSA and the M&O contractors faced challenges in collecting accurate and consistent financial data for common financial reporting across the nuclear security enterprise. Specifically, NNSA faced challenges in (1) fully implementing its data validation and reconciliation process, (2) collecting financial data from each M&O contractor for all of the program offices, and (3) communicating information about changes in a timely manner. First, NNSA faced challenges fully implementing its data validation and reconciliation process for fiscal year 2018. NNSA designed CostEX to automatically validate the M&O contractors’ data to check data quality and formatting and perform data reconciliation with STARS. However, according to an NNSA official, for fiscal year 2018, the agency manually reconciled the M&O contractors’ fiscal year 2018 data with STARS to identify and fix issues with the process prior to automation. For example, an NNSA support contractor manually submitted and reconciled data for one M&O contractor that manages two sites because the M&O contractor submits combined data for the two sites into STARS, but NNSA collects financial data for common financial reporting by site. For the fiscal year 2019 data collection effort, NNSA officials said they corrected the submission issue and CostEX was able to automatically reconcile the M&O contractors’ data with STARS. Another M&O contractor’s fiscal year 2018 financial data did not reconcile each month with STARS. NNSA officials and representatives from the M&O contractor said the reconciliation issue was due to timing differences between when the contractor reported data into STARS and CostEX. Specifically, M&O contractor representatives for the site said that when NNSA is delayed in collecting data for common financial reporting in CostEX, the relationships between the data reported into STARS and CostEX will have changed, which may result in reconciliation errors. During that time, the site changed how it tracked some of the data, which led to differences in how the data were provided for STARS and common financial reporting, and which caused the reconciliation errors. NNSA officials said they resolved the issue with the M&O contractor for fiscal year 2019 and completed data collection in October 2019. Second, NNSA faced challenges in collecting data from each M&O contractor for all of the participating program offices. Specifically, the Office of Defense Nuclear Nonproliferation made ongoing changes to its work breakdown structure templates throughout the fiscal year 2018 data collection effort. This resulted in challenges for the M&O contractors when reporting data for this program office. NNSA did not collect complete fiscal year 2018 financial data for this office, in part because one of the contractors had significant data validation and reconciliation errors, resulting in data that NNSA could not validate and reconcile. Third, NNSA faced challenges in communicating information about changes to the work breakdown structure in a timely manner to M&O contractors. Leading project management practices emphasize the importance of establishing and implementing change control processes, which include reviewing and approving all change requests, documenting the changes, and communicating the decisions. In fiscal years 2018 and 2019, not all NNSA programs consistently ensured that changes to the work breakdown structure were approved, documented, or communicated to the M&O contractors in a timely manner because NNSA had not established and implemented a work breakdown change control process. NNSA established aspects of such a process, in which program offices submitted changes to the work breakdown structures to the financial integration team so the team could upload the changes into CostEX and notify the M&O contractors of the changes prior to their data submissions. However, according to officials with the financial integration team, the federal program managers did not always follow the process. Officials with the financial integration team said that in some instances, the sites’ program managers contacted the M&O contractors directly to request changes to their work breakdown structures. The financial integration team identified issues with the program offices’ work breakdown structures when the M&O contractors’ data could not be validated and reconciled. In such instances, the financial integration team contacted the program managers to request the updated work breakdown structures for CostEX. Further, the existing process does not include some aspects of change control processes that are consistent with leading practices. Approving changes. Under the existing process, the financial integration team does not check whether changes that federal program managers submit to them have been reviewed and approved, at a minimum, by program office management prior to making changes to the work breakdown structures in CostEX. The program director for financial integration said that they defer to the program offices to ensure that program office management review and approve changes to the work breakdown structure before the program managers submit these changes to the financial integration team. Documenting changes. NNSA officials said that not all program offices have tracked changes to their work breakdown structures over time. NNSA’s Office of Defense Programs has a process for tracking changes to its work breakdown structure, but that process—or a similar process—was not utilized consistently by all of NNSA’s other program offices. If the program offices do not track the changes to their work breakdown structures over time, they cannot ensure the data are comparable across fiscal years. According to officials, NNSA built a tool in CostEX to track work breakdown structure changes across fiscal years. NNSA officials said the tool was tested at the end of fiscal year 2019 by the Office of Defense Programs. NNSA plans to test using the tool to track changes for the other program offices in fiscal year 2020. Communicating decisions. NNSA did not always communicate changes to the work breakdown structure to the M&O contractors in a timely manner. Representatives from the seven M&O contractors stated that they encountered challenges in submitting their data in CostEX on multiple occasions throughout fiscal years 2018 and 2019 because federal program managers in some offices made frequent changes to the work breakdown structures that often were not communicated to the M&O contractors in a timely manner. When work breakdown structures change, representatives from the seven M&O contractors said they have to redo the crosswalk of their financial data to the new work breakdown structures before they submit the data— this takes time and additional resources and may result in delayed data submissions. Representatives from three of the M&O contractors said the frequency of changes to the work breakdown structures decreased for the fiscal year 2019 data collection effort, but representatives from six M&O contractors said they continued to encounter challenges when changes were made to the work breakdown structures. Without establishing and systematically implementing a work breakdown structure change control process, NNSA will not be able to verify that, at a minimum, program office management has approved changes to the work breakdown structure or that these changes have been documented, potentially leading to challenges in ensuring that the data are comparable over time. Furthermore, NNSA cannot ensure that changes to the work breakdown structures are communicated to the M&O contractors in a timely manner, which results in contractors using additional time and resources to address validation or reconciliation errors. Publish and analyze data. NNSA has published the M&O contractors’ financial data for fiscal years 2018 and 2019, but NNSA has not conducted agency-wide analysis of the data. The NNSA financial integration team has a website for common financial reporting from which the program offices can download financial data. However, an NNSA official stated that agency-wide analysis of the data was not feasible for fiscal years 2018 or 2019 because NNSA did not use a common work breakdown structure for all participating program offices. In addition, an NNSA official stated that the agency needs to collect at least 3 years of data to produce useful NNSA-wide findings. Some of the NNSA program offices have started to analyze the financial data collected through the common financial reporting effort. For example, the Office of Defense Programs is using financial data collected through common financial reporting for program evaluation and to make budgetary decisions. In addition, an NNSA official from the Office of Counterterrorism and Counterproliferation stated that the office has used financial data from common financial reporting to identify and address accounting issues, such as identifying previously unidentified unspent funds carried over from prior fiscal years and redirecting these funds to support program activities in fiscal year 2019. However, some of the program offices have not used the data collected through common financial reporting for various reasons. For example, officials from the Office of Safety, Infrastructure, and Operations stated that the fiscal year 2018 data were not useful for analysis because they were not collected in a timely manner. NNSA officials said they completed data validation and reconciliation of the M&O contractors’ fiscal year 2018 financial data in February 2019—nearly halfway through the following fiscal year—making the data late and not useful for that office’s purposes. Additionally, officials from the Office of Defense Nuclear Security stated that they have not used the data collected through the common financial reporting effort because they want to ensure that the data are accurate and consistent before using it for decision-making. As discussed previously, M&O contractors crosswalk their financial data into a reporting framework using work breakdown structures and common cost elements and definitions, and they submit their data to NNSA using CostEX. To help ensure the accuracy of the data, NNSA performs data quality checks of the M&O contractors’ financial data submitted using CostEX. If NNSA cannot validate and reconcile the submitted data using the agency’s processes, it rejects and returns the data to the M&O contractor to correct the errors. NNSA also provides the M&O contractors with error reports from CostEX that they can use to identify and correct errors. Each M&O contractor has established processes to check data quality prior to submitting the data to NNSA in CostEX. For example, representatives from all of the M&O contractors said they reviewed their data for missing information and errors before submitting the data into CostEX. In addition, all of the M&O contractors performed checks to compare their data submissions for common financial reporting with their STARS submissions before submitting the data into CostEX. After the M&O contractors complete their internal data quality checks, they submit their financial data into CostEX. At most sites, M&O contractor representatives said the way their site tracks financial data does not align with how NNSA requests the data be reported in the work breakdown structure and cost elements. Officials from NNSA’s Office of Cost Estimating and Program Evaluation said that because the M&O contractors do not track their financial data using NNSA work breakdown structures, the contractors have to make decisions using professional judgment as to how to crosswalk their project costs, raising concerns that each M&O contractor may make different decisions about how to allocate costs. The officials said this may result in data that are not accurate or comparable for conducting agency- wide analysis. We identified several limitations to the approach NNSA uses to collect common financial data that could affect the accuracy and consistency of the data: NNSA’s data reconciliation process does not ensure M&O contractors’ financial data are accurate. M&O contractors identified potential issues with using STARS for reconciliation to ensure data accuracy. For example, two M&O contractors said that errors can sometimes occur in their monthly STARS reporting. Errors in STARS can be created when a number is mistyped or corrections are made to purchase card or time sheet information. Once the M&O contractor submits its data to STARS, errors cannot be corrected until at least the following month. However, because the common financial reporting data must reconcile with STARS, the M&O contractor has to submit financial data into CostEX that includes the error. The program director for financial integration said a process is in place for the M&O contractors to identify any issues with STARS reporting and correct their reported data in the future. More significantly, some M&O contractors said they make changes to their data before submitting it into CostEX to ensure that the data reconcile. Specifically, representatives from two M&O contractors said they compare their financial data for common financial reporting with their STARS data submission. If data from the two systems do not match for small dollar amounts, the contractors manually make adjustments to the data for common financial reporting rather than making the corrections in their business systems. The representatives also said they do not notify NNSA officials of the manual changes. NNSA requires that financial data for common financial reporting reconcile with STARS. Specifically, NNSA rejects M&O contractor financial data that differs from the STARS data by more than $1. According to federal standards for internal control, management should define objectives clearly to enable the identification of risks and define risks tolerances. For the fiscal year 2018 data collection effort, NNSA documentation indicated that M&O contractors reported financial data for $8.9 billion of costs and reconciled the data with their STARS cost reporting to a total difference of $5.03. According to an NNSA official, M&O contractors reported financial data for $10.2 billion of costs and reconciled the data with STARS to a total difference of $8.97 for fiscal year 2019. However, NNSA has limited assurance that the financial data provided internally reconcile as required because the agency does not know the extent of changes that M&O contractors made to ensure the data reconcile with STARS or the potential effects of those changes on the accuracy of the data. Assessing the extent to which M&O contractors make manual changes to ensure reconciliation with STARS for common financial reporting and determining the effect of these changes could provide additional assurance that the financial data collected through common financial reporting are accurately reported. M&O contractors crosswalk site projects and tasks to NNSA work breakdown structures, resulting in the potential for differences in how costs are allocated. Each M&O contractor tracks financial data for its site based on how it manages the work using projects and tasks, as allowed by federal Cost Accounting Standards. When a site’s projects and tasks do not align with NNSA’s work breakdown structure, site program managers identify the component of the NNSA work breakdown structure with which the project and tasks best align and crosswalk their financial data to the NNSA structure using professional judgment. One site program manager said it is sometimes challenging to identify which of their internal projects and tasks aligns with the NNSA work breakdown structure, especially when internal projects have similar names to describe different project scopes. Another site program manager said the site’s projects and tasks closely align with the NNSA work breakdown structure approximately 30 to 40 percent of the time, and contractor representatives use professional judgment to crosswalk the remaining 60 to 70 percent of their projects and tasks. To create the crosswalk, site program managers consider which NNSA program the project mostly supports. It can be difficult to crosswalk the site data into NNSA’s work breakdown structure, especially for work that benefits multiple weapons programs. For example, a site program manager said that the site’s project to develop inert material for NNSA’s high explosives activities supports multiple weapons programs. The site tracks that work as one project, but NNSA’s work breakdown structure requires that the costs be reported across multiple programs. When M&O contractors make decisions to crosswalk their financial data using professional judgment, the contractors do not provide information to NNSA on how the costs are allocated. By verifying this information, NNSA could ensure that allocation decisions are made consistently across the nuclear security enterprise. M&O contractors provided different financial data for the same projects. M&O contractors continue to report financial data for some program offices into multiple systems, including the G2 program management system, WebPMIS, and spreadsheets. For fiscal year 2018, NNSA compared financial data that the M&O contractors reported, for two NNSA program offices, into the G2 program management system and the CostEX tool used for common financial reporting and found differences between the data reported for the same budget and reporting codes and levels of the work breakdown. The program director for financial integration said he worked with the program offices and identified the cause of the differences in the data. NNSA cannot ensure the accuracy of the data submitted for common financial reporting because NNSA does not have an internal process to verify whether M&O contractors crosswalk their financial data accurately from their business systems to the NNSA work breakdown structure. According to federal standards for internal control, management should use quality information to achieve the agency’s objectives. Under the financial integration policy, the program director for financial integration is responsible for executing a plan for NNSA to achieve enterprise-wide financial integration to collect standardized financial management data; increase transparency of financial accountability; and improve cost analysis, comparability, and reporting consistency among programs and M&O contractors. The program director for financial integration said that verifying whether the M&O contractors properly crosswalk their data to the work breakdown structure is an area in which the agency should improve its common financial reporting effort. NNSA officials stated that the common financial reporting effort does not have a process to validate financial data that are more detailed than STARS and indicated that until the agency has assurances the reported data are accurate, NNSA should not use that more detailed data for agency decision-making. By developing an internal process for NNSA to verify the M&O contractors’ crosswalks, the agency will have better assurance that the data collected through common financial reporting will produce accurate, enterprise- wide financial data that is comparable across the M&O contractors and that satisfies the needs of Congress and other stakeholders. Further, this would help address long-term issues with NNSA’s ability to report the total costs of its programs, in accordance with Managerial Cost Accounting Standards. As part of common financial reporting, M&O contractors crosswalk their financial data to NNSA’s cost elements. Cost elements capture discrete costs of a particular activity of work and include direct costs such as labor and equipment and indirect costs such as general and administrative costs. In March 2018, NNSA established 22 cost elements and definitions—including 10 indirect cost elements—that the M&O contractors use to report financial data. As we found in our January 2019 report, NNSA officials said this was a critical step toward implementing common financial reporting because without common cost elements, the agency was limited in its ability to report lower-level costs consistently across programs and sites. In addition, having the M&O contractors report financial data across common cost elements would allow NNSA to improve its management of programs across the enterprise. NNSA developed the cost elements and definitions in consultation with the M&O contractors based on the data they could provide because officials said it is important for the contractors to report accurate financial data using the NNSA cost elements. M&O contractors manage their sites’ financial data using expenditure types to track the costs of their projects. These expenditure types capture similar costs as the cost elements, but at a more detailed level, and are specific to each individual M&O contractor based on how the contractor manages its expenses. M&O contractors have flexibility to determine how they structure their work and the expenditures they track in their financial systems consistent with Cost Accounting Standards. Based on our review of M&O contractor documents, M&O contractors varied significantly in the number of expenditure types they tracked. For example, the M&O contractor for one of the national laboratories tracked its financial data using over 900 expenditure types, while another national laboratory used around 50 expenditure types. NNSA officials said that the number of expenditure types at the sites varies based on the nature of the work performed at each site. Most of the M&O contractors cannot crosswalk their expenditures to certain NNSA cost elements because of how they track costs in their systems. Specifically, representatives from five of the M&O contractors said they cannot accurately crosswalk their indirect expenditure types to NNSA’s indirect cost elements because their systems do not capture the data in the way that NNSA wants these data reported. M&O contractors have discretion to classify which costs are considered indirect, and costs for similar activities can be allocated differently by each contractor. In fiscal year 2018, NNSA’s M&O contractors reported spending $3.5 billion on indirect activities. Generally, in cases in which the M&O contractors could not crosswalk their indirect costs to specific NNSA cost elements, representatives from one of the M&O contractors said they allocated their indirect costs to NNSA’s cost elements using percentages, while others said they reported data that did not adhere to the NNSA cost elements. Below are examples of situations in which M&O contractors were not able to accurately report expenditures into NNSA’s indirect cost elements: Representatives from one M&O contractor said they could not accurately report financial data for the general and administrative cost element and site support from other overhead cost elements because the site did not capture its data in that way. As a result, the M&O contractor allocated its indirect costs using formulas and composite rates, rather than reporting actual cost data to NNSA. Representatives from two M&O contractors said they could not accurately report financial data across the site support and infrastructure support cost elements because the structure of their indirect cost pool did not allow them to track those expenditures separately. As a result, representatives from one of the M&O contractors said they reported all of their infrastructure expenditures to the site support cost element. NNSA officials said they were aware of the M&O contractors’ issues with reporting their expenditure types using the NNSA cost elements. Although M&O contractors are required to provide financial data using NNSA’s cost elements, the program director for financial integration said he was aware that M&O contractors report some indirect costs for separate cost elements to a single cost element in CostEX, meaning that they do not accurately report some indirect costs based on NNSA’s definitions. Additionally, the financial integration team identified differences between indirect cost data collected from the M&O contractors for common financial reporting and data reported to another group in NNSA’s Office of Management and Budget. NNSA plans to conduct a review of the data reported through the two efforts to determine the cause of the differences. Officials from the Office of Safety, Infrastructure, and Operations stated that it is important that the common financial reporting effort is able to collect accurate information on M&O contractors’ costs related to infrastructure spending. NNSA is aware of the challenges its M&O contractors have with accurately reporting their expenditure types against the NNSA cost elements. However, NNSA cannot ensure that the agency collects accurate financial data because NNSA does not have a process to verify how the M&O contractors crosswalk their expenditure types to NNSA’s cost elements, consistent with the previously described information quality standard under the federal standards for internal control and NNSA’s financial integration policy. M&O contractors reporting data based on allocated—as opposed to actual—costs is not ideal because NNSA cannot ensure that each M&O contractor is consistently applying the allocation and because the data may not be standardized and comparable across the sites, which affects the quality of the data. By developing an internal process for NNSA to verify how the M&O contractors crosswalk their expenditure types, the agency could better ensure that the data collected through common financial reporting will produce accurate financial data across the nuclear security enterprise that satisfies the needs of Congress and other stakeholders. Further, this would help address long-term issues with NNSA’s ability to report the total costs of its programs. NNSA continues to make progress toward implementing agency-wide common financial reporting. However, NNSA faces challenges in fully implementing the effort. For example, NNSA has not consistently ensured that changes to the work breakdown structure are approved, documented, and communicated to the M&O contractors in a timely manner because NNSA has not established and implemented a change control process for the changes. Without establishing and fully implementing a work breakdown structure change control process, NNSA will not be able to verify that the changes to the work breakdown structure are approved by program office management, at a minimum; documented and tracked for accurate data analysis and comparison over time; and communicated to the M&O contractors on a timely basis. NNSA’s approach to implementing common financial reporting relies on M&O contractors to crosswalk their internal financial data into a common reporting framework using a work breakdown structure and common cost elements and definitions, with certain quality checks to help ensure the accuracy of the data. However, NNSA has limited assurance that the financial data that the M&O contractors provide for common financial reporting are accurate because the agency does not know the extent of the changes the M&O contractors make to the data so that the data reconcile to the agency’s accounting system or the potential effects of these changes. By determining the extent of these changes and whether these changes affect the accuracy of the data, NNSA could have greater assurance that the financial data collected through common financial reporting are accurate. Additionally, NNSA cannot ensure that M&O contractors accurately crosswalk their financial data to either the NNSA work breakdown structure or the common cost elements because NNSA has not established processes to verify the information. By developing internal processes that would allow NNSA to verify how the M&O contractors crosswalk their data to the work breakdown structure and common cost elements, NNSA could better ensure that the data collected through common financial reporting will produce accurate enterprise-wide financial data that is comparable across the M&O contractors and that satisfies the needs of Congress and other stakeholders. Further, this would help to address long-term issues with NNSA’s ability to report the total costs of its programs. We are making four recommendations to NNSA: The Program Director for Financial Integration, with input from NNSA’s Office of Management and Budget and respective program offices, should establish and implement a work breakdown structure change control process for common financial reporting that ensures changes are approved by program office management, at a minimum; documented; and communicated to M&O contractors on a timely basis. (Recommendation 1) The Program Director for Financial Integration should assess the extent to which M&O contractors make manual changes to their financial data to reconcile with STARS and determine whether it has an effect on the accuracy of the data collected for common financial reporting. (Recommendation 2) The Program Director for Financial Integration should develop and implement an internal process for NNSA to verify how the M&O contractors crosswalk financial data from their systems to the appropriate NNSA work breakdown structure to ensure the reported data are accurate and consistent. (Recommendation 3) The Program Director for Financial Integration should develop and implement an internal process for NNSA to verify that the M&O contractors are consistently applying common cost element definitions at their sites and across the nuclear security enterprise. (Recommendation 4) We provided a draft of this report to NNSA for comment. In its written comments, which are reproduced in appendix II, NNSA agreed with the report’s four recommendations and described actions it intends to take to address them. NNSA also provided technical comments that we incorporated into the report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, the Administrator of NNSA, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or bawdena@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in appendix III. In our January 2019 report on the National Nuclear Security Administration’s (NNSA) efforts to implement common financial reporting, we made seven recommendations. Table 2 describes NNSA’s progress to implement these recommendations, as of December 2019. In addition to the individual named above, key contributors to this report included Hilary Benedict (Assistant Director), Amanda K. Mullan (Analyst in Charge), Colette Alexander, Antoinette Capaccio, Jennifer Echard, Cindy Gilbert, Michael LaForge, Jason Lee, Holly Sasso, and Sheryl Stein.", "summary": "NNSA has long faced challenges in determining and comparing the costs of its programs, which are principally performed by M&O contractors across eight sites. Congress needs this information to provide effective oversight and make budgetary decisions. The National Defense Authorization Act for Fiscal Year 2017 required NNSA to implement a common financial reporting system, to the extent practicable, across all sites by December 2020. NNSA's efforts began in 2016 and are ongoing. The Senate report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2018 includes a provision for GAO to periodically review NNSA's implementation of common financial reporting. This is GAO's second report on this issue. This report examines (1) the steps NNSA has taken to implement common financial reporting since GAO's January 2019 report, and (2) the extent to which NNSA's approach to data collection aligns with the purpose of common financial reporting, including collecting accurate and consistent data from its M&O contractors. GAO reviewed NNSA documents about implementing common financial reporting, including policy and briefing documents, and interviewed NNSA officials and M&O contractor representatives. The National Nuclear Security Administration (NNSA)—a separately organized agency within the Department of Energy (DOE)—is required to implement common financial reporting, to the extent practicable, across its sites to better understand the total costs of its programs. NNSA has taken additional steps to implement such reporting since January 2019 but faces challenges in fully implementing the effort (see table). For example, for fiscal years 2018 and 2019, NNSA used separate work breakdown structures—a method of dividing a project into successive levels of detail—to collect data for some offices. Without a common work breakdown structure, NNSA cannot ensure that it can collect reliable financial data across its sites. NNSA plans to assess the feasibility of implementing a common work breakdown structure, in response to GAO's January 2019 recommendation. In fiscal years 2018 and 2019, NNSA also faced challenges in collecting financial data from management and operating (M&O) contractors, including collecting complete data for all program offices. NNSA is working to resolve these issues. NNSA's approach to data collection provides limited assurance that the data collected for common financial reporting are accurate and consistent across the M&O contractors. At most sites, the M&O contractors track their financial data in a way that does not align with how NNSA requests the contractors report the data. M&O contractors use professional judgment to crosswalk, or map, the financial data from their business systems to the NNSA structures to report the data. NNSA's data quality checks on the M&O contractors' financial data focus on data formatting and ensuring the data match the agency's accounting system. NNSA does not have a process to verify whether the contractors accurately crosswalk their financial data. Under NNSA's financial integration policy, the program director for financial integration is to, among other things, execute a plan to improve cost analysis, comparability, and reporting consistency among programs and M&O contractors. By developing an internal process for NNSA to verify how the M&O contractors crosswalk their financial data to the work breakdown structures, NNSA will have better assurance that it is collecting accurate financial data that are comparable across the M&O contractors, that satisfy the needs of Congress and other stakeholders, and that address long-term issues with its ability to report the total costs of its programs. GAO is making four recommendations, including that NNSA implement an internal process to verify the M&O contractors' crosswalks of their financial data to NNSA's work breakdown structures for reporting information. NNSA agreed with the four recommendations.", "document_type": "gao"}
{"report": "Established by the Communications Act of 1934, FCC regulates interstate and international communications by radio, television, wire, satellite, and cable in all 50 states, the District of Columbia, and U.S. territories. FCC is responsible for, among other things, making available nationwide worldwide wire and radio communication service. More recently, it has been responsible for promoting competition and reducing regulation of the telecommunications industry in order to secure lower prices and higher quality services for consumers. FCC’s functions include: issuing licenses for broadcast television and radio; overseeing licensing, enforcement, and regulatory functions of carriers of cellular phones and other personal communication services; regulating the use of radio spectrum and conducting auctions of licenses for spectrum; investigating complaints and taking enforcement actions if it finds that there have been violations of the various communications laws and commission rules that are designed to protect consumers; addressing issues related to public safety, homeland security, emergency management, and preparedness; educating and informing consumers about communications goods and reviewing mergers of companies holding FCC-issued licenses. FCC relies extensively on computerized systems to support its mission- related operations, and on information security controls to protect the commission’s data. FCC’s Information Technology Center, within the Office of the Managing Director, uses IT to perform the commission’s business operations. Through its computer network and systems, the commission collects and maintains nonpublic information, including proprietary information of businesses regulated by the commission, as well as information available to the public through rulemaking proceedings. FCC’s Chairman, chief information officer (CIO), and chief information security officer (CISO) each have specific responsibilities for information security. Specifically, the FCC Chairman has responsibility for, among other things: 1. providing information security protections commensurate with the risk and magnitude of harm resulting from unauthorized access, use, disclosure, disruption, modification, or destruction of the commission’s information systems and information; 2. ensuring that senior officials provide security for the information and systems that support the operations and assets under their control; and 3. delegating to the CIO the authority to ensure compliance with the information security requirements imposed on the commission. In addition, the CIO is responsible for establishing and enforcing policies and procedures for protecting information resources. Toward this end, the CIO has designated and assigned responsibilities to the CISO for managing the cybersecurity program. The CISO, among other things, is responsible for providing information security protections commensurate with the risk and magnitude of the harm resulting from unauthorized access, use, disclosure, disruption, modification, or destruction of information and information systems that support the operations and assets of the commission. The Federal Information Security Modernization Act of 2014 (FISMA) provides a comprehensive framework for information security controls over information resources that support federal operations and assets. The law also requires each agency to develop, document, and implement an agency-wide information security program to provide risk-based protections for the information and information systems that support the operations and assets of the agency. Such a program should include assessing risks; developing and implementing policies and procedures to cost-effectively reduce risks; developing and implementing plans for providing adequate information security for networks, facilities, and systems; and providing security awareness and specialized training. Further, the program should include testing and evaluating the effectiveness of controls; planning, implementing, evaluating, and documenting remedial actions to address information security deficiencies; developing and implementing procedures for detecting, reporting, and responding to security incidents; and ensuring continuity of operations. FISMA requires agencies to comply with the federal information processing standards (FIPS) publications issued by NIST and Office of Management and Budget (OMB) Circular A-130 requires agencies to comply with the information security guidelines prescribed in NIST special publications. Consequently, NIST FIPS publications and special publications contain many of the cybersecurity-related requirements for federal agencies. For example, NIST FIPS Publication 199 requires agencies to categorize their information and information systems according to the potential harm and impact to agency assets, operations, or individuals should the confidentiality, integrity, or availability of its information and information systems be compromised through unauthorized access, use, disclosure, disruption, modification, or destruction. In addition, NIST FIPS Publication 200 requires agencies to meet minimum security requirements by selecting the appropriate security controls, as described in NIST Special Publication 800-53. This special publication provides a catalog of 18 security control areas for federal information systems and a process for selecting controls to protect organizational operations and assets. The publication provides baseline security controls for low-, moderate-, and high-impact systems, and agencies have the ability to tailor or supplement their security requirements and policies based on agency mission, business requirements, and operating environment. Further, in May 2017, the President issued an executive order requiring agencies to immediately begin using NIST’s cybersecurity framework for managing their cybersecurity risks. The framework, which provides guidance for cybersecurity activities, is based on five core security functions: Identify: Develop an organizational understanding to manage cybersecurity risk to systems, people, assets, data, and capabilities. Protect: Develop and implement the appropriate safeguards to ensure delivery of critical infrastructure services. Detect: Develop and implement the appropriate activities to identify the occurrence of a cybersecurity event. Respond: Develop and implement the appropriate activities to take action regarding a detected cybersecurity incident. Recover: Develop and implement the appropriate activities to maintain plans for resilience and to restore any capabilities or services that were impaired due to a cybersecurity incident. According to NIST, these five functions occur concurrently and continuously, and provide a strategic view of the life cycle of an organization’s management of cybersecurity risk. Within the five functions are 23 categories and 108 subcategories that include information security program-related controls and technical controls for achieving the intent of each function. Appendix II provides a description of the framework categories and subcategories of controls. On May 7 and 8, 2017, FCC experienced a dramatic surge in the number of comments sent to the commission through its ECFS during a public comment period. This surge led to a disruption of services, which prevented the system from being able to accept additional comments for a period of time. The FCC Office of Inspector General determined that the system service disruption was likely due to a combination of the sudden increase in traffic from commenters all trying to access the system’s website over a short period of time and system design deficiencies that negatively impacted the capacity and performance of the system to collect and process the increase in traffic. Figure 1 presents a timeline of the May 2017 ECFS service disruption and subsequent related events. Additional details on the timeline are provided in appendix III. In response to the ECFS service disruption that occurred on May 7 and 8, 2017, FCC Information Technology Center officials took four key actions to reduce the risk of future service disruptions to the system. 1. Conducted Internal Assessments In response to the service disruption, in early May 2017, the FCC CIO initially stated that the cause was a cyberattack on the ECFS. However, upon further assessment, FCC Information Technology Center officials later determined that the disruption was caused by a surge in comment traffic to the system and existing system performance and capacity deficiencies. In response to multiple congressional inquiries, in late July 2017, FCC Information Technology Center officials assessed the extent to which malicious intent was involved in causing the disruption based on whether: (a) internet protocol (IP) addresses from foreign sources were present on the commission’s network at the time of the May 2017 event; (b) comment submissions were denied (i.e., dropped) from the commission’s network, (c) observable botnet traffic was present; and (d) duplicate comment submissions were accepted into ECFS. The assessment concluded that the commission did not have sufficient information and tools to determine whether there was any malicious intent. 2. Deployed Additional Virtual Hardware Following the disruption, in early May 2017, FCC deployed additional virtual hardware to address system performance issues and support system stabilization efforts of ECFS during the period in which service was disrupted. In early July 2017, the commission installed security sensors and forwarding agents on the ECFS virtual servers. These devices are intended to provide additional layers of security capability for the system. In mid-July 2017, FCC automated the process for deploying virtual hardware resources to support system availability subsequent to the May 2017 service disruption. 3. Optimized and Acquired System Software From late May 2017 to early June 2017, FCC acquired a diagnostic tool to measure system performance. According to the commission, this tool is used to determine the maximum amount of simultaneous user capacity within ECFS during periods of high web traffic. In early June 2017, the commission optimized the search functionality within the ECFS database to reduce the system response time. In mid-June 2017, FCC removed redundant internal processes for ECFS web requests to increase the responsiveness of the system. During late July 2017, the commission acquired a security information and event management tool to collect and analyze security-related events that may indicate a cybersecurity incident. In late August 2017, FCC established rate control limits within ECFS to safeguard against potential distributed denial-of-service attacks aiming to flood one target with network traffic. 4. Updated Incident Response Policy and Procedures In January 2018 and March 2018, during its annual policy review, FCC Information Technology Center officials updated the commission’s incident response and reporting policy and procedures to incorporate lessons learned from the May 2017 ECFS service disruption and clarify their processes. For example, FCC Information Technology Center officials revised the commission’s incident response procedures to document internal escalation time frames for notifying management of potential security incidents and reporting the incidents to the United States Computer Emergency Readiness Team within 1 hour of identification of an incident. Figure 2 shows a chronological sequence of the hardware and software improvements that FCC officials implemented after the May 2017 event. FCC provided evidence that indicated its actions to add additional hardware and software resources increased ECFS’s capacity and performance and demonstrated that the system was stable from June 2017 through December 2017. For example, FCC acquired a performance diagnostic tool in late May 2017, which was designed to determine the maximum number of potential simultaneous public users within ECFS during periods of high web traffic. Using the diagnostic tool, FCC Information Technology Center officials determined in June 2017, that the system became unstable when the number of simultaneous simulated public users reached 500. However, by December 2017, the system had demonstrated that it could accept a capacity of over 3,000 simultaneous public users without a service disruption. FCC data showed that the increased capacity and improved performance of the ECFS prevented further service disruptions during periods of sharp spikes in the volume of comments received. For example, on May 8, 2017, service was disrupted on the system when it received a peak of about 249,000 comments in 1 day, whereas on July 12, 2017, the system accepted and processed at least 1.4 million comments in 1 day without a reported service disruption. Similar spikes in traffic volumes that occurred through December 2017 also did not result in service disruptions. Figure 3 shows the daily comment submissions to ECFS from May 2017 through December 2017 and demonstrates FCC’s ability to accept a higher volume of comments without a service disruption. We reported in September 2019 that FCC had implemented numerous security controls for the three systems we reviewed, but it had not consistently implemented the NIST cybersecurity framework’s five core security functions to effectively protect the confidentiality, integrity, and availability of these systems and the information maintained on them. Deficiencies existed in the FCC information security program and technical controls for the five core functions that were intended to (1) identify risk, (2) protect systems from threats and vulnerabilities, (3) detect cybersecurity events, (4) respond to these events, and (5) recover system operations when disruptions occur. These deficiencies increased the risk that sensitive information could be disclosed or modified without authorization or be unavailable when needed. As shown in table 1, deficiencies existed in all five core security functions for the FCC systems we reviewed. Also shown are the numbers of recommendations we made to FCC to rectify the deficiencies. Activities associated with the identify core security function are intended to help an agency to develop an understanding of its resources and related cybersecurity risks to its organizational operations, systems, and data. Essential elements of a FISMA-mandated information security program include assessing risks, developing system security plans, and authorizing information systems to operate. NIST guidance states that agencies should assess risks and authorize systems on an ongoing basis. Additionally, FCC requires that security plans, risk assessments, and system authorizations be reviewed annually or whenever significant changes occur to the information system, computing environment, or business operations. Consistent with its guidance, FCC had developed system security plans for each of the three systems we reviewed and had updated the risk assessments for two of the systems in 2017 and 2018, respectively. However, as of March 2019, the commission had not reviewed or updated the risk assessment for the third system reviewed since May 2017—a lag of about 22 months. Commission officials stated that they had not reviewed or updated the system’s risk assessment because the commission had implemented a new risk assessment process and officials had not yet had time to review and update documentation for this system. In addition, FCC continued to operate two of the three selected systems on expired authorizations to operate. Although FCC granted a full authorization to operate to one system in May 2018, the commission allowed the authorizations for the other two systems we reviewed to expire. Both of these systems had received a conditional authorization to operate so that the systems could continue to operate while the commission mitigated known system vulnerabilities. However, in December 2018, the conditional authorizations for both systems expired because, according to FCC officials, the commission had not mitigated the vulnerabilities. Nevertheless, FCC continued to operate the systems. By not regularly updating the risk assessment of one system and continuing to operate another system without a current authorization to operate, FCC unnecessarily exposed the information on these systems to increased risks of unauthorized changes and access to information. Subsequent to our September 2019 report, FCC reviewed and updated the system’s risk assessment in accordance with its new risk assessment process. In addition, FCC granted a full authorization to operate to one of the systems in October 2019, but does not expect to grant a full authorization to operate for the other system until later in 2020. NIST SP 800-144, Guidelines on Security and Privacy in Public Cloud Computing, states that a service-level agreement should define the terms and conditions for access and use of the services offered by the cloud service provider. In addition, FedRAMP Control Specific Contract Clauses provides security control specifications that may need to be included in the task order for the service and specified in the service level agreement. These contract clauses include specifications related to data jurisdiction, audit records storage, time frames for reporting security incidents, and system boundary protection. FCC’s task order and service level agreement with its cloud service provider specified activities the provider was to perform, such as providing access and support for products and services, and completing performance deliverables to ensure service availability. However, FCC had not documented specific contract clauses associated with implementing security control requirements related to retaining audit records, meeting reporting incident time frames, and protecting system boundaries in accordance with FedRAMP. According to FCC’s associate chief information officer, the commission relied on FedRAMP’s oversight to ensure that its cloud provider implemented security controls that comply with federal data requirements. However, FedRAMP assesses and monitors only the security controls that the program and cloud service provider agree that the provider will implement. These agreed-upon controls may not include an agency’s specific security requirements. Thus, responsibility falls on FCC to ensure that its information security requirements are being implemented in cloud computing environments. Nevertheless, by not specifying its specific control requirements when procuring services from its cloud provider, FCC increased the risk that its data and sensitive regulatory information will not be adequately protected in the event that its cloud service provider experiences a security breach. Subsequent to our September 2019 report, FCC developed a plan of action and milestones (POA&M) for this deficiency and stated that it plans to rectify the deficiency by May 2020. Activities associated with the protect core security function are intended to help agencies develop and implement appropriate system safeguards. These activities include limiting access to computing resources to authorized users, processes and devices; encrypting data to protect its confidentiality and integrity; configuring devices securely; and updating software to protect systems from known vulnerabilities. FCC implemented activities that established multiple layers of technical controls, including access controls and firewalls, encryption of sensitive data, and system configuration management. However, we reported in September 2019 that implementation of these technical controls were not consistent. For example, 37 technical control deficiencies and an information security program-related deficiency diminished the effectiveness of the controls protecting the systems we reviewed. A brief summary of the results of our tests of FCC’s controls for protecting the three systems we reviewed follows. FCC policy states that, in accordance with NIST SP 800-53 guidelines, users should not share the same identifier and the commission should configure its information systems to require users to create complex passwords. FCC’s policy also stipulates that the commission employ the principle of “least privilege” and enforce approved authorizations for controlling the flow of information within the system and between interconnected systems. However, FCC did not consistently implement technical controls to effectively limit access to the systems we reviewed, as the following examples illustrate. Although FCC policy states that individual user accounts are not to be shared, the commission allowed multiple users to share the credentials of several privileged accounts. While FCC policy established minimum requirements for password complexity and account lock-out provisions, the commission did not routinely enforce these requirements. While FCC policy requires limiting access rights for users to only those they need to perform their work, the commission inappropriately granted excessive permissions to users to access server configuration files. Although FCC established a policy for monitoring and controlling access between systems, it did not securely configure network devices to effectively control access and communications between systems. Access control deficiencies existed primarily because FCC network administrators did not adequately monitor configuration settings and did not implement sufficient controls to enforce consistent authentication and authorization across all of the commission’s systems that we reviewed. However, until FCC fully implements those actions and remediates related technical deficiencies, the commission remains at increased risk that unauthorized individuals or attackers could obtain inappropriate access to its network devices, firewalls, and servers, and compromise its network. As of November 2019, FCC had acted to address several technical control deficiencies related to access control. NIST SP 800-53 recommends that organizations employ cryptographic mechanisms to prevent the unauthorized disclosure of information during transmission and establish a trusted communications path between users and security functions of information systems. NIST also requires that, when agencies use encryption, they use an encryption algorithm that complies with FIPS Publication 140-2. In addition, FCC’s System and Communication Protection Policy states that confidentially sensitive data must be encrypted before being transmitted using any nonprotected communication method and that all passwords must be encrypted. However, in seven instances, the commission did not consistently deploy strong encryption capabilities to protect sensitive data or establish a secure communications path between users and information systems. For example, FCC sometimes sent data in clear text over the network and did not enable FIPS 140-2 compliant encryption algorithms on certain devices. These deficiencies existed primarily because commission personnel did not adequately monitor configuration settings. By not consistently deploying strong encryption capabilities, FCC limits its ability to protect the confidentiality and integrity of its sensitive information. According to Information Technology Center officials, as of November 2019, the commission was still working toward full compliance with federal encryption standards. NIST SP 800-53 states that agencies should configure security settings to the most restrictive mode consistent with operational requirements and disable services within the information system deemed to be unnecessary or non-secure. FCC policy on risk assessment states that systems and devices should be scanned periodically and software patches should be applied for all known critical security vulnerabilities. In addition, OMB Circular A-130 states that agencies are to implement current updates and patches for all software components of information systems, and prohibit the use of unsupported systems and system components. Although FCC established policies for applying software patches on a prescribed basis, it did not update software in a consistent or timely manner to effectively protect the three systems we reviewed. For example, FCC did not apply software patches in a timely manner to resolve known security vulnerabilities, and used unsupported or out-of- date system software on multiple network devices, firewalls, and servers. Patching control deficiencies existed because FCC did not adequately monitor configuration settings of devices on its network. According to Information Technology Center officials, as of February 2019, the commission was in the process of (1) migrating and modernizing its systems’ portfolio and (2) implementing an application monitoring and testing tool to reduce patching times. However, until FCC applies software patches in a timely manner, and replaces unsupported software and devices, it will remain at increased risk that individuals could exploit known vulnerabilities to gain unauthorized access to its computing resources. As of November 2019, FCC had taken corrective actions to address certain technical control deficiencies related to configuring servers securely and updating software in a timely manner. Developing, documenting, and implementing information security policies and procedures are essential elements of an agency’s FISMA-mandated information security program. FCC’s Policy for Information Security and Privacy states that FCC shall implement procedures and controls at all levels to protect the confidentiality and integrity of information stored and processed on the commission’s systems, and to ensure that the systems and information are available to authorized persons when required. Although FCC developed and documented commission-wide policies addressing the 18 control areas—such as access control, configuration management, security awareness training, and contingency planning— identified in NIST SP 800-53, the commission had not fully developed or documented the detailed operating procedures that are needed to effectively implement its security policies. For example, FCC had not documented detailed procedures for implementing the following NIST- specified control areas: (1) access control, (2) configuration management, (3) identification and authentication, (4) system maintenance, (5) media protection, (6) physical and environmental protection, (7) information security program management, (8) risk assessment, (9) system and services acquisition, (10) system and communication protection, and (11) system and information integrity. The lack of detailed operating procedures likely was an underlying cause for many of the technical control deficiencies we identified. According to the FCC CISO, as of February 2019, the commission was in the process of reviewing and revising its information security policies and had issued POA&Ms to develop and document the missing procedures. Nevertheless, until FCC fully develops and documents detailed operating procedures for implementing its security policies, the commission faces increased risks that it will not effectively protect its information systems and information from cyber threats. The detect core security function is intended to allow for the timely discovery of cybersecurity events and deficiencies. Controls associated with this function include logging and monitoring system activities, and assessing security controls in place. NIST SP 800-53 states that agencies should enable system logging features and retain sufficient audit logs to support the investigations of security incidents and monitoring of select activities for significant security-related events. Additionally, NIST SP 800-53 and industry leading practices state that organizations should increase their situational awareness through enhanced monitoring capabilities to analyze network traffic data over an extended period of time at external boundaries and inside their internal network to identify anomalous, inappropriate, or unusual malicious activities. Lastly, FISMA requires each agency to periodically test and evaluate the effectiveness of its information security controls in place applicable to policies, procedures, and practices. In September 2019, we reported that FCC had implemented security monitoring controls, such as performing regular vulnerability scanning and deploying a system information and event management tool, to detect the presence of potential malicious threats. However, six technical control deficiencies in these capabilities diminished the effectiveness of the controls to detect cybersecurity events in the systems we reviewed. For example, FCC did not fully capture system log data on certain devices and had limited network monitoring visibility into portions of its data center environment. According to Information Technology Center officials, FCC had deficiencies in logging, retention, and monitoring because the commission had not fully configured its security information and event monitoring tool to capture and monitor sufficient system log and network traffic data to adequately detect cybersecurity events. As a result, FCC may not be able to detect or investigate anomalous activities inside its network. In addition, although the commission established a process for assessing the effectiveness of the security controls for its systems, its control tests and evaluations were not sufficiently robust. For example, the commission’s evaluations did not identify many of the security control deficiencies we identified. Consequently, FCC had limited assurance that the security controls were in place and operating as intended. As of November 2019, FCC had acted to address several technical control deficiencies, and associated recommendations, such as capturing network traffic data and providing for real-time network monitoring; however, other technical control deficiencies remain. The respond core security function is intended to support the ability to contain the impact of a potential cybersecurity event. Controls associated with this function include implementing an incident response capability and remediating newly identified deficiencies. We reported in September 2019 that, as part of its information security program, FCC had implemented controls for incident response by developing, documenting, and annually updating its incident handling policy and procedures, along with its guidelines for remediating deficiencies. However, two information security program-related deficiencies and a technical control deficiency diminished the effectiveness of the controls to respond to cybersecurity events for the systems we reviewed. For example, the commission did not adequately address security incidents and mitigate known deficiencies in a timely manner. NIST SP 800-53 and SP 800-61 state that agencies should develop, document, and implement incident response policy and procedures, and keep them updated according to agency requirements. FCC incident response policy also states that all employees are required to report suspected security incidents to the FCC Network Security Operations Center (NSOC) group within 1 hour of discovery or detection, and all other incidents within 24 hours of discovery. Further, FCC’s incident response procedures require internal escalation and external notification to the United States Computer Emergency Readiness Team (US-CERT) within 1 hour. FCC had developed, documented, and updated its incident response policy and procedures on an annual basis to address security incidents. The commission also established a NSOC group as the single point of contact for potential security incidents. However, FCC did not report internally to the NSOC group or externally to US-CERT in a timely manner for three of 10 security incidents we reviewed. Specifically, A FCC employee took 2 days to report the existence of an information spillage incident to the NSOC instead of the required 1-hour reporting time frame. The NSOC group took approximately 4 hours to report a December 2017 distributed denial-of-service attack incident and a February 2018 malicious attack incident to the US-CERT, instead of the 1 hour required for each. According to the FCC CISO, the commission plans to review its incident response policy and procedures, as well as re-train its staff, to ensure that staff consistently follow the commission’s policy and US-CERT incident notification guidelines. Subsequent to the issuance of our September 2019 report, FCC indicated that it plans to address these matters by October 2020. Until it does so, the commission may impede its ability to receive timely assistance from appropriate federal agencies and mitigate any harm. NIST 800-53 states that agencies are to develop a POA&M for an information system to document the agencies’ planned remedial actions to correct identified deficiencies. FCC’s Plan of Action and Milestone Guide also states that the maximum completion time frames for implementing POA&M items related to critical and high severity level deficiencies are 30 and 60 days, respectively. Although FCC developed a remedial action process and maintained a management system to document and track the status of POA&M items, it did not complete remedial actions in a timely manner for the three systems we reviewed. Specifically, FCC did not remedy critical and high severity level deficiencies within the required time frames as stated in its policy. For example, FCC took an average of approximately 3 months to implement four critical severity level POA&M items for one system. FCC took an average of more than 1 year to remediate three critical and nine high severity level POA&M items for another system. Additionally, as of October 2018, this system had seven open critical and four open high severity level POA&M items that exceeded the remediation threshold on average by 1 year, 4 months, and 5 months, respectively. FCC took an average of more than 3 years to implement two critical and seven high severity level POA&M items for the third system. FCC officials attributed these delays to operational priorities and resource constraints, such as financial, personnel, and technological factors. However, such longstanding delays in remediating weaknesses pose a significant threat to the overall security posture of the commission, since the delays could allow intruders to exploit critical and high severity level deficiencies to gain access to FCC’s information resources. As of November 2019, FCC stated that it planned to address security program deficiencies related to remediating weaknesses in a timely manner by October 2020. The recover core security function is intended to support timely recovery of system operations to reduce the impact from a cybersecurity event. Controls associated with this function include developing and testing contingency plans to ensure that, when unexpected events occur, critical operations can continue without interruption or can be promptly resumed, and that information resources are protected. In September 2019, we reported that, as part of its information security program, FCC had developed contingency plans for selected systems and established priorities for application disaster recovery. However, two information security program-related deficiencies diminished the effectiveness of the controls to recover the systems we reviewed. Specifically, the commission did not document detailed procedures for restoring two of the three systems conduct an annual test of its disaster recovery plan for the three selected systems in fiscal year 2018. NIST SP 800-34 Contingency Planning Guide for Federal Information Systems states that an information system contingency plan should provide detailed procedures to restore the information system or components to a known state. In addition, FCC’s Policy for Contingency Planning states that system contingency plans should reflect the restoration activities required for information systems to recover after an incident. FCC developed and documented a contingency plan for one system that specified detailed procedures for restoring system operations, data, and supporting applications. However, FCC did not include detailed procedures for restoring the other two systems we reviewed in their respective contingency plans—both of which are major application systems. For example, the contingency plans for these two systems did not specify procedures for restoration activities such as restoring critical operating system, application software, and system data to a known state. According to Information Technology Center officials, they did not consider the two systems as supporting mission essential functions, which would necessitate the inclusion of the applications in the detailed restoration procedures. However, both of the systems are major application systems and support mission essential functions at FCC. Subsequent to our September 2019 report, FCC documented detailed restoration procedures in the two other systems’ contingency plans that included activities associated with restoring critical operating system, application software, and system data to a known state. By doing so, FCC increased the likelihood that it will be able to restore operations to its mission essential functions in the event of a disaster. NIST SP 800-84 states that a disaster recovery test should assess the ability of an agency to restore IT processing capabilities in the event of a disruption. Moreover, FCC’s policy for contingency planning states that all information system and facility disaster recovery plans should be tested annually to determine the effectiveness of the plan and the organizational readiness to execute the plan. In September 2019, we reported that FCC did not conduct test exercises of the disaster recovery plans for the three systems we reviewed during fiscal year 2018, nor did it test system backup, recovery, restoration, and reconstitution procedures for these systems. According to FCC officials, the test exercise did not take place in fiscal year 2018 because other business operation activities took precedence over the exercise since the test exercise requires all mission-essential function applications to be unplugged. As a result, FCC had limited assurance that it would be able to recover from unexpected disruptions in a timely and efficient manner. While it did not complete the exercise in fiscal year 2018, FCC did subsequently conduct a disaster recovery exercise at the beginning of fiscal year 2019. By doing so, FCC increased its assurance that it would be able to recover use of its systems from unexpected disruptions in a timely and efficient manner. In our September 2019 report, we made 136 recommendations to FCC to bolster its agency-wide information security program and strengthen its technical security controls. Specifically, we recommended that FCC take nine actions to improve its information security program by, among other things, authorizing systems to operate, documenting operating procedures, resolving known vulnerabilities and reporting security incidents in a timely manner, and testing disaster recovery plans. We also recommended that FCC take 127 actions to address technical control deficiencies by implementing stronger access controls, encrypting sensitive data, configuring network devices securely, strengthening firewall rules, implementing audit and monitoring controls more effectively, among other actions. Since the issuance of our September 2019 report, FCC has made significant progress in implementing the recommendations we made to improve its information security program and resolve the technical control deficiencies in the information systems we reviewed. Specifically, as of November 2019, FCC had implemented 85 (63 percent) of the 136 recommendations we made in the September 2019 report and had effectively resolved the underlying deficiencies associated with the recommendations. The commission also had partially, but not fully, implemented 10 recommendations. In these instances, FCC provided evidence that it had resolved a portion of the underlying control deficiency, but had not completed all of the actions necessary to fully resolve the underlying control deficiencies. FCC did not provide any evidence that it had begun implementing the remaining 41 (30 percent) recommendations. The status of our recommendations to FCC is illustrated in figure 4. Table 2 provides additional details on the status of FCC’s actions to implement our recommendations to improve its information security program and the technical controls for the systems we reviewed. By implementing 85 recommendations, FCC (as of November 2019) had reduced risks associated with certain key activities. Specifically, FCC’s actions to implement four information security program-related recommendations included conducting a disaster recovery test exercise, documenting detailed system restoration procedures, and updating risk assessments to reflect the commission’s current computing environment. Regarding the technical controls, the commission had implemented 81 of our recommendations to rectify technical control-related deficiencies. For example, FCC strengthened firewall rules and access controls on its information system servers and internal networks—that we highlighted in our September 2019 report as being particularly vulnerable and requiring the commission to take immediate corrective actions. FCC also had developed a POA&M for each of the identified information security program-related and technical control deficiencies that remained open as of November 2019. The POA&M items contained required elements, such as severity levels (i.e., high, medium, and low) for identified weaknesses; identified estimated costs; designated points of contact; and established time frames for resolving those weaknesses and fully implementing the related recommendations. The commission’s plans called for it to implement the majority of the remaining information security program and technical control-related recommendations by May 1, 2020, and all recommendations by April 30, 2021, as shown in figure 5. Fully implementing the remaining recommendations is essential to ensuring that the commission’s systems and sensitive information are adequately protected from cyber threats. Key actions that remain include: documenting operational procedures, applying security patches and software updates, and enhancing network monitoring capabilities. Until FCC fully implements all of our recommendations and resolves the associated deficiencies, its information systems and information will remain at increased risk of misuse, improper disclosure or modification, and loss. We received written comments on a draft of this report from FCC. In its comments, which are reprinted in appendix IV, the commission expressed its commitment to protecting the confidentiality, integrity, and availability of its information systems. FCC noted our evaluation of its efforts to implement 85 of the 136 recommendations made in our September 2019 report and stated that it had also addressed nine additional recommendations. The commission further stated that it plans to address the remaining recommendations over the next 14 months with full mitigation anticipated by April 2021. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees. We are sending copies of this report to the appropriate congressional committees, the Federal Communications Commission, the commission’s Office of the Inspector General, and interested congressional parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, our primary point of contact is Vijay A. D’Souza at (202) 512-6240 or dsouzav@gao.gov. You may also contact Seto J. Bagdoyan at (202) 512-4749 or bagdoyans@gao.gov. GAO staff who made key contributions to this report are listed in appendix V. Our objectives were to determine 1) the actions FCC took to respond to the May 2017 event that affected the Electronic Comment Filing System (ECFS), and 2) the extent to which FCC implemented security controls to effectively protect the confidentiality, integrity, and availability of selected systems. In September 2019, we issued a report which detailed the findings from our work in response to these two objectives. In the report, we made 127 recommendations to FCC to resolve the technical security control deficiencies in the information systems we reviewed and nine additional recommendations to improve its information security program. We designated that report as “limited official use only” (LOUO) and did not release it to the general public because of the sensitive information it contained. This report publishes the findings discussed in our September 2019 report, but we have removed all references to the sensitive information. Specifically, we deleted the names of the information system software, network devices, and resource tools that we examined, disassociated identified control deficiencies from named systems, deleted certain details about information security controls and control deficiencies, and omitted an appendix that was contained in the LOUO report. The appendix contained sensitive details about the technical security control deficiencies in the FCC’s information systems and computer networks that we reviewed, and the 127 recommendations we made to mitigate those deficiencies. We also provided a draft of this report to FCC officials to review and comment on the sensitivity of the information contained herein and to affirm that the report can be made available to the public without jeopardizing the security of the commission’s information systems and networks. In addition, this report addresses a third objective that was not included in the September 2019 report. Specifically, this objective was to determine the extent to which FCC had taken corrective actions to address the previously identified security program and technical control deficiencies and related recommendations for improvement that we identified in the earlier report. To address the first objective, we reviewed FCC’s security and incident response policies and procedures, examined related reports prepared by the commission and its Office of Inspector General, reviewed an internal assessment of the May 2017 event that was performed by the FCC Information Technology Center, and reviewed artifacts associated with system enhancement and performance such as change requests and email. We also extracted comment submission data derived from the data.gov application programming interface between May 1, 2017 and December 31, 2017 to identify the peak periods of increased comment submissions during and after the May 2017 event. In addition, we examined the aforementioned documents to assess whether the updated incident response policy and procedures, along with system enhancement and performance artifacts, were directly related to changes made subsequent to the May 2017 event. Lastly, we interviewed FCC Information Technology Center officials, including system and security staff, and Office of Inspector General officials to identify FCC’s actions to respond to the May 2017 event. To address the second objective, we reviewed FCC’s overall network environment, identified interconnectivity and control points, and examined controls for the commission’s networks and facilities. We performed this work at FCC facilities located in West Virginia, Pennsylvania, and Washington, D.C. As noted in our September 2019 report, we determined the extent to which FCC had implemented security controls to effectively protect the confidentiality, integrity, and availability of selected systems. To do so, we selected three of the commission’s information systems for review. We selected these systems because they (1) are essential to FCC’s mission and (2) were assigned a Federal Information Processing Standards Publication 199 rating of moderate or high impact. The results of our review of these systems is not generalizable to the commission’s other systems. To evaluate FCC’s controls for its information systems, we used GAO’s Federal Information System Controls Audit Manual, which contains guidance for reviewing information system controls that affect the confidentiality, integrity, and availability of computerized information. We based our assessment of controls on requirements of the Federal Information Security Modernization Act of 2014 (FISMA), which establishes key elements for an effective agency-wide information security program; National Institute of Standards and Technology (NIST) guidelines and standards; FCC policies and procedures; and standards and guidelines from relevant security organizations, such as the National Security Agency, and the Center for Internet Security. For reporting purposes, we categorized the security controls that we assessed into the five core security functions described in the NIST cybersecurity framework. The five core security functions are: Identify: Develop the organizational understanding to manage cybersecurity risk to systems, assets, data, and capabilities. Protect: Develop and implement the appropriate safeguards to ensure delivery of critical infrastructure services. Detect: Develop and implement the appropriate activities to identify the occurrence of a cybersecurity event. Respond: Develop and implement the appropriate activities to take action regarding a detected cybersecurity event. Recover: Develop and implement the appropriate activities to maintain plans for resilience and to restore any capabilities or services that were impaired due to a cybersecurity event. These core security functions are described in more detail in appendix II. For each of the five core security functions, we examined selected FCC security controls and related documentation: For the identify core security function, we examined FCC’s reporting for its hardware and software assets; analyzed risk assessments for the three selected systems to determine whether threats and vulnerabilities were being identified; analyzed FCC policies and procedures to determine their effectiveness in providing guidance to personnel responsible for securing information and information systems; and analyzed security plans for the three selected systems to determine if those plans had been documented and updated according to federal guidance. For the protect core security function, we examined access controls for the three systems. These controls included the password complexity and settings to determine if password management was being enforced; administrative users’ system access permissions to determine whether their authorizations exceeded the access necessary to perform their assigned duties; and firewall configurations, among other things, to determine whether system boundaries had been adequately protected. We also examined configurations for providing secure data transmissions across the network to determine whether sensitive data were being encrypted. In addition, we examined configuration settings for routers, network management servers, switches, and firewalls to determine if settings adhered to configuration standards, and we inspected key servers and network devices to determine if critical patches had been installed and/or were up to date. For the detect core security function, we analyzed security control assessments, and centralized logging and network traffic monitoring capabilities for key assets connected to the network. For the respond core security function, we reviewed FCC’s implementation of incident response practices, including an examination of incident tickets for 10 incidents the commission considered most significant from January 1, 2017 to May 29, 2018; and examined the commission’s process for correcting identified deficiencies for the three selected systems. For the recover core security function, we examined contingency and disaster recovery plans for the three selected systems to determine whether those plans had been developed and tested. For the core security functions, as appropriate, we evaluated elements of FCC’s information security program. For example, we analyzed risk assessments, security plans, remedial action plans, and contingency plans for each of the three selected systems. We also evaluated FCC’s security policies and procedures. In assessing FCC’s controls associated with these core functions, we interviewed FCC’s Information Technology Center personnel, chief information officer, chief information security officer, general counsel, inspector general, and Public Safety and Homeland Security Bureau officials, as needed. To determine the reliability of FCC’s computer-processed data for incident response records, we evaluated the materiality of the data to our audit objective and assessed the data by various means, including reviewing related documents, interviewing knowledgeable FCC officials, and reviewing internal controls. Through a combination of these methods, we concluded that the data were sufficiently reliable for the purposes of our work. To accomplish our third objective—on FCC’s actions to address the previously identified security program and technical control deficiencies and related recommendations—we requested that the commission provide a status report of its actions to implement each of the recommendations. For each recommendation that FCC indicated it had implemented as of November 2019, we examined supporting documents, observed or tested the associated security control or procedure, and/or interviewed the responsible agency officials to assess the effectiveness of the actions taken to implement the recommendation or otherwise resolve the underlying control deficiency. Based on this assessment and FCC status reports, we defined the status of each recommendation according to three categories: fully implemented—FCC had implemented the recommendation (i.e., the commission provided evidence showing that it had effectively resolved the underlying control deficiency); partially implemented—FCC had made progress toward, but had not completed implementing the recommendation (i.e., the commission provided evidence showing that it had effectively resolved a portion of the underlying control deficiency); and not started—FCC did not provide evidence that it had acted to implement the recommendation (i.e., the commission provided no evidence showing that it had effectively resolved the underlying control deficiency). We conducted the performance audit for the first two objectives from February 2018 through September 2019 in accordance with generally accepted government auditing standards. We conducted work supporting the third objective and, where applicable, included updates to our work in the second objective, from October 2019 through March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings. The National Institute of Standards and Technology’s cybersecurity framework consists of five core functions: identify, protect, detect, respond, and recover. Within the five functions are 23 categories and 108 subcategories of security-related controls (see table 3). Below is a timeline of the Federal Communications Commission’s (FCC) May 2017 Electronic Comment Filing System (ECFS) event and subsequent related events: On April 27, 2017, FCC issued the Restoring Internet Freedom Notice of Proposed Rulemaking in the Federal Register. The notice directed interested parties to submit comments via FCC’s ECFS. On the evening of May 7, 2017, a late night talk show aired a segment on the Restoring Internet Freedom notice and encouraged viewers to submit comments via ECFS. On the evening of May 7, 2017, according to a report by the FCC Office of Inspector General (IG), ECFS experienced a significant increase in the level of comment traffic attempting to access the system, resulting in the disruption of system availability. A contractor providing web performance and cloud security solutions to FCC identified a 3,116 percent increase in traffic to ECFS between May 7 and May 8, 2017. In the early morning of May 8, 2017, ECFS became unavailable to commenters. FCC’s vendor sent automated alerts indicating a spike in network traffic, in addition to preliminary network statistical data, to FCC. During the mid-morning of May 8, 2017, FCC’s Information Technology Center responded to the alerts from the vendor and initiated stabilization efforts to ECFS. During the afternoon of May 8, 2017, FCC issued a press release in which FCC’s chief information officer (CIO) at that time provided a statement about the cause of delays experienced by commenters trying to file comments on the ECFS. The CIO’s statement said that FCC was subjected to multiple distributed denial-of-service attacks. He further stated that, “these were deliberate attempts by external actors to bombard the FCC’s comment system with a high amount of traffic.” During May 9-10, 2017, FCC restored ECFS but still experienced response-time problems relating to system performance. On May 10, 2017, FCC’s Information Technology Center responded to inquiries from the Federal Bureau of Investigations and FCC OIG via email and phone. On June 21, 2017, the FCC OIG opened a full investigation into the event because of, according to the OIG, the importance of FCC’s cybersecurity posture and the possibility that cybercrimes had been committed that had the potential of being ongoing threats to the integrity of FCC’s computer systems. On January 4, 2018, FCC OIG referred the investigation to the Justice Department. On August 7, 2018, the FCC OIG published an investigative report on the ECFS event. According to the OIG report, the allegations of multiple distributed denial-of-service attacks alleged by the FCC CIO at that time were not substantiated. The FCC OIG concluded that the spikes in web traffic to ECFS had coincided exactly with the timing of the late night television show where the host discussed the FCC’s Restoring Internet Freedom proceeding and encouraged viewers to visit the commission’s website and file comments. The FCC OIG’s report also indicated that the commission did not define the event (i.e., any observable occurrence in a network or system) as a cybersecurity incident (i.e., an imminent threat or violation of computer security policies, or security practices). Therefore, according to the OIG report, FCC did not take actions to: refer the matter to the United States Computer Emergency Readiness Team (US-CERT) in accordance with federal policy, implement internal incident handling procedures in accordance with its incident handling policy, or conduct a thorough analysis before or after the event to determine if it was an incident. On August 16, 2018, the FCC Chairman testified at a Senate Committee on Commerce, Science and Transportation oversight hearing on the conclusions of the FCC OIG investigative report on the ECFS event. In addition to the contacts named above, Gary Austin, David Bruno, Tammi Kalugdan, Duc Ngo, and Christopher Warweg (assistant directors); David Hong (analyst-in-charge); Breanne Cave; Chris Businsky, Jr.; Saar Dagani; Marshall Williams, Jr.; Corey Evans; Andrew Howard; Elizabeth Kowalewski; Priscilla Smith; Henry Sutanto; and April Yeaney made significant contributions to this report.", "summary": "FCC relies extensively on information systems to accomplish its mission of regulating interstate and international communications in the United States. FCC uses one such system, ECFS, to receive public comments about proposed changes in FCC regulations. In May 2017, a surge in comments caused a service disruption of ECFS during a public comment period. GAO was requested to review ECFS and the reported disruption. In September 2019, GAO issued a limited official use only report on the actions FCC took to respond to the May 2017 event, and the extent to which FCC had effectively implemented security controls to protect the confidentiality, integrity, and availability of selected systems. This current report is a public version of the September 2019 report with sensitive information removed. In addition, for this public report, GAO determined the extent to which FCC has taken corrective actions to address the previously identified security program and technical control deficiencies and related recommendations for improvement. In the prior report, GAO compared FCC's policies, procedures, and reports to federal cybersecurity laws and policies. GAO examined logical access controls and security management controls for three systems selected based on their significance to FCC. For this report, GAO examined supporting documents regarding FCC's actions on previously identified recommendations, observed controls in operation, and interviewed personnel at FCC. As GAO reported in September 2019, the Federal Communications Commission (FCC) bolstered the capacity and performance of the Electronic Comment Filing System (ECFS) to reduce the risk of future service disruptions. FCC also implemented numerous information security program and technical controls for three systems that were intended to safeguard the confidentiality, integrity, and availability of its information systems and information. systems from threats and vulnerabilities, detecting and responding to cyber security events, and recovering system operations. GAO made 136 recommendations to address these deficiencies (see table). As of November 2019, FCC had made significant progress in resolving many security deficiencies by fully implementing 85 (about 63 percent) of the 136 recommendations GAO made in September 2019. FCC had also partially implemented 10, but had not started to implement the remaining 41 recommendations (see figure). Additionally, FCC has created remedial action plans to implement the remaining recommendations by April 2021. Until FCC fully implements these recommendations and resolves the associated deficiencies, its information systems and information will remain at increased risk of misuse, improper disclosure or modification, and loss.", "document_type": "gao"}
{"report": "DOD’s policy is to ensure that eligible personnel and their families have access to affordable, quality housing facilities and services consistent with grade and dependent status, and that the housing generally reflects contemporary community living standards. From the inception of MHPI, the military departments were provided with various authorities to obtain private-sector financing and management to repair, re novate, construct, and operate military housing in the United States and its territories. Through these authorities, the military departments have entered into a series of agreements with private partners to provide housing to servicemembers and their families. The military departments have flexibility in how they structure their privatized housing projects, but typically the military departments lease land to private developers for 50-year terms and convey existing housing located on the leased land to the developer for the duration of the lease. The developer then becomes responsible for renovating and constructing new housing and for the daily management of these housing units. At the end of fiscal year 2017, 14 private partners were responsible for 79 privatized military family housing projects—34 for the Army, 32 for the Air Force, and 13 for the Navy and the Marine Corps—in the United States, each of which includes housing at one or more military installation. Each privatized housing project is a separate and distinct entity governed by a series of legal agreements that are specific to that project. However, there are some common elements in how projects invest and use funds. Every project takes in revenue, which consists mostly of rent payments. Projects then pay for operating expenses, including administrative costs, day-to-day maintenance, and utilities, among other things. After that, projects generally allocate funds for taxes and insurance, followed by debt payments. In the typical funding structure for a privatized housing project, once debt payments are made, funds are allocated to accounts that fund scheduled maintenance, such as repair and replacement of items like roofs, heating and cooling systems, and infrastructure. After that, funds are then allocated to a series of management incentive fees, such as the property management fee. Finally, the project divides these remaining funds according to a fixed percentage between accounts that (1) fund major renovations and rebuilds and (2) are provided to the developer. The Deputy Assistant Secretary of Defense for Facilities Management, under the authority, direction, and control of the Assistant Secretary of Defense for Sustainment, is responsible for all matters related to MHPI and is the program manager for all DOD housing, whether DOD-owned, DOD-leased, or privatized. In this capacity, the Deputy Assistant Secretary is to provide both guidance and general procedures related to military housing privatization, as well as required annual reports to Congress on the status of privatized military housing projects. However, it is the responsibility of the military departments to execute and manage privatized housing projects, including conducting financial management and monitoring their portfolio of projects. Each military department has issued guidance that outlines its responsibilities for privatized housing, such as which offices are responsible for overseeing privatized housing projects. In our draft report, currently with DOD for review and comment, we found that each military department conducts a range of oversight activities— some more extensive than others—for its privatized housing projects. For example, among other things, military departments review sample work order requests and inspect housing during the change–of-occupancy process. DOD guidance states that because privatization creates a long- term governmental interest in privatized housing, it is essential that projects be attentively monitored. Through its guidance, DOD delegates oversight responsibility of the individual privatized housing projects to each of the military departments. In our draft report, we noted that OSD and the military departments’ oversight efforts have been limited in the following key areas. Specifically, we found that (1) the scope of oversight of the physical condition of privatized housing has been limited; (2) performance metrics focused on quality of maintenance and resident satisfaction do not accurately reflect private partner performance related to the condition of privatized housing; (3) there is a lack of reliable or consistent data on the condition of privatized housing; and (4) past DOD reports to Congress on resident satisfaction are unreliable due to the inconsistent handling and calculation of the data and therefore may be misleading. DOD delegates oversight responsibilities of the individual privatized housing projects to each of the military departments, and each military department has subsequently issued guidance outlining oversight roles and responsibilities. Military department oversight activities generally fall into two categories—(1) daily oversight of management and operations and (2) periodic reviews of compliance with each project’s business agreements. Daily oversight of management and operations. Daily oversight of a project’s management and operations is to be conducted by each installation’s military housing office. Military housing officials told us that activities to monitor the physical condition of housing units generally include reviewing sample work order requests, following up with a sample of residents to check on their experience with recently completed work, and inspecting housing units during the change-of-occupancy process. As we noted in our draft report, the implementation and scope of these activities varies and can be limited. For example, during our site visits conducted from June through August 2019, we observed that the rate of inspections of homes following change-of-occupancy maintenance varied at the installations we visited. Military housing office officials at one Air Force installation told us that they inspect 100-percent of homes that have completed change-of-occupancy maintenance, while officials from a different Air Force installation stated they inspect 10 to 20 percent of these homes. Military department officials told us that in spring 2019, each department conducted a “100-percent” review of privatized housing by directing installation commanders to contact all residents of privatized housing and offering a visual inspection of their privatized housing unit. In addition, in March 2019 the Army issued an order directing military housing office officials to inspect 100-percent of homes where change-of-occupancy maintenance has been completed. Officials from Army installations we visited noted that this was an increase from previous practices, and for one installation was a change in practice from only conducting inspections during the move-out process, which occurs prior to change- of-occupancy maintenance. Similarly, in November 2019, Air Force officials told us they were moving to a 100 percent inspection policy. Periodic reviews of compliance with each project’s business agreements. Periodic reviews of compliance with a project’s business agreements are a joint effort between the local military housing office, the private partners, military department installation commands, and other echelons of command. These reviews can include neighborhood tours to view project amenities such as community centers, playgrounds, and pools, all of which are owned, maintained, and operated by the private partner companies, as well as exteriors of housing units. However, our draft report showed these annual reviews have been narrow in the scope of their assessment of the physical condition of the housing units, as interior walk-throughs were, at times, focused on just a few homes at each installation. According to military department officials, each department has completed initiatives and is undertaking initiatives to revise guidance and standardize daily oversight activities in an effort to provide consistent oversight across projects and installations, and to increase the focus on the physical condition of housing. In addition, the military departments have initiatives to increase staffing levels, improve training for military housing office officials, and ensure that military department housing officials have independent access to work order data, to strengthen their oversight activities. However, each military department is working to implement service-specific initiatives with only limited guidance from OSD on the level of oversight expected of the services as it relates to the condition of the housing. Specifically, OSD guidance is focused on the oversight of the implementation of projects, the construction of new housing units, and project financial monitoring. The guidance stipulates that after privatized housing projects are awarded, monitoring should include descriptions of deal structure and strategies for project monitoring. In contrast, OSD guidance for military-owned housing provides clearly defined objectives to the military departments for oversight, including the physical condition of the homes. Unless OSD updates its guidance on the oversight of privatized housing with objectives for overseeing the physical condition of housing units, it cannot be assured that the military departments’ oversight activities will be sustained over time or be sufficiently consistent across projects, raising the risk that private partners may not provide adequate quality housing. The military departments each use a range of project-specific performance metrics to monitor private partner performance, but as we note in our draft report, the metrics designed to focus on resident satisfaction and on the quality of the maintenance conducted on housing units do not provide meaningful information or reflect the actual condition of the housing units. Most, but not all, of the private partners are eligible to receive performance incentive fees based on generally meeting the performance metrics established in each individual project’s business agreement. Private partner performance is commonly measured through four key metrics—resident satisfaction, maintenance management, project safety, and financial management. To determine how well the private partners are performing under the metrics, military housing office officials told us they rely on a range of specific indicators established in the project business agreements. However, the indicators themselves do not provide meaningful information on the private partner’s performance in maintaining quality housing units. For example, we identified the following in our draft report: Maintenance management. One indicator of performance of maintenance management that is regularly included in project business agreements measures how often the property manager’s response time to work orders meets required timeframes established in the project’s business agreements. While this indicator measures the timeliness of the private partner’s response, it does not measure or take into account the quality of the work that was conducted or whether the resident’s issue was fully addressed. As such, a property manager may fully meet the metric for maintenance management even if a given repair has not been adequately completed. Residents in 13 of our 15 focus groups noted that they typically have had to submit multiple work order requests before an individual maintenance issue has been fully addressed. Some projects include indicators that aim to more directly measure quality, such as the number of work orders placed during the first 5 business days of residency, which may indicate the extent to which all of the change-of-occupancy maintenance was completed. Resident satisfaction. One example of an indicator of resident satisfaction is whether a project has met target occupancy rates established in the project’s business agreements. An OSD official and private partner representatives told us they use occupancy as an indicator of satisfaction based on the assumption that residents would move if they were dissatisfied with their home’s condition. However, based on our focus groups, this may not be a reliable assumption. Although most residents are not required to live in military housing, residents in each of our 15 focus groups indicated a variety of reasons for choosing to live in privatized housing, many of which did not have to do with their satisfaction with the quality or condition of their homes. For example, residents in our focus groups cited other factors influencing their decision to live in privatized housing, such as living in close proximity to military medical or educational services for children or other family members that are part of the military’s Exceptional Family Member Program, a lack of safe and affordable housing in the surrounding community, and access to quality schools. OSD and military department officials have recognized that the current indicators for measuring performance do not consistently focus on or prioritize the private partners’ performance with maintaining housing units and ensuring resident satisfaction. For example, Army officials told us they are no longer using occupancy rates as an indicator of resident satisfaction and have taken steps to standardize performance indicators across all Army projects, while still allowing for flexibility at the installation level to modify the weight of indicators to provide incentives reflective of the specific needs of the installation. Limitations to the current indicators may hinder the military departments’ ability to accurately determine private partner performance. However, OSD and military department officials told us they have not yet reevaluated the specific indicators used to determine whether a private partner has met a specific metric because doing so will require negotiation with each of the private partners for each project. Nonetheless, without reviewing the specific indicators used to award performance incentives, OSD and the military departments do not have assurance that the information the military departments are using to award these incentives reflects the actual condition of the housing. The housing projects’ business agreements typically include a requirement for the private partner to maintain a records management system to record, among other things, maintenance work requested and conducted on each housing unit. According to private partner officials, each company uses commercial property management software platforms for activities such as initiating maintenance work orders and dispatching maintenance technicians. Some private partner representatives stated that while data from the work order tracking systems are primarily used to prioritize and triage maintenance work, the data were never intended to monitor the overall condition of privatized housing units. While data from these work order tracking systems may be useful for point-in-time assessments of work order volume at a given installation, military department officials told us that efforts are underway to monitor work order data to increase the military departments’ oversight and the accountability of the private partners for providing quality housing. However, as we noted in our draft report, we found that these data are not captured reliably or consistently for use in the ongoing monitoring of the condition of privatized housing units. We received and reviewed data from each of the 14 private partners’ work order tracking systems covering each of the 79 privatized family housing projects. Based on our review of these data and discussions with private partner representatives for our draft report, we found two primary factors that would limit the reliability or consistency of using these data for ongoing monitoring of the condition of privatized housing units over time—(1) inconsistent use of terminology in work order records and (2) differing practices for opening and closing work orders: Inconsistent use of terminology. Based on our review of the data provided by the private partners and discussions with private partner officials, we noted cases where work orders were inconsistently entered into the work order tracking systems with respect to two primary factors—(1) how the request is described by the resident or interpreted by the official entering the data, which can differ for each work order, and (2) the existing range of pre-established service category options in the private partner’s work order tracking system, which differ among the partners. Differing practices for opening and closing work orders. At some installations we visited, private partners noted changes in practices for opening and closing work orders, limiting the usefulness of the data in monitoring the status of work orders over time and thus the condition of privatized housing. In addition, we identified other anomalies in work order data from each of the 14 partners. For example, we identified instances of, among other things, duplicate work orders, work orders with completion dates prior to the dates that a resident had submitted the work order, and work orders still listed as in-progress for more than 18 months. According to military department officials, efforts to review data from the private partners’ work order tracking systems have increased, and military department officials told us they have found similar limitations. However, neither OSD nor the military departments have identified minimum data requirements, established consistent terminology or practices for data collection, or developed processes for the military departments to validate the work order data collected by the private partners. Without direction from OSD to establish minimum data requirements and consistent terminology or practices for data collection, as well as a requirement for the military departments to validate data, the military departments’ ability to use data from the private partners’ work order tracking systems to monitor the condition of privatized homes over time will remain limited and may vary across projects. DOD is statutorily required to provide reports to Congress that include, among other things, information about military housing privatization projects’ financial health and performance and backlog, if any, of maintenance and repairs. These reports have included information on resident satisfaction with privatized housing based on the results of the annual military department satisfaction surveys. As we state in our draft report, we determined that information on resident satisfaction in these reports to Congress on privatized housing have been unreliable and are misleading due to (1) variances in the data the military departments collect and provide to OSD and (2) OSD’s calculation and presentation of the data. In May 2019, OSD issued its report for fiscal year 2017, which stated that overall resident satisfaction for calendar year 2017 was 87 percent. For OSD’s fiscal year 2017 report, the military departments provided data on resident satisfaction based on information from the annual resident satisfaction surveys. Specifically, OSD’s instructions to the military departments required the military departments to report satisfaction based on resident responses to the question that asks: “Would you recommend privatized housing,” with results indicating how many tenants responded “yes,” “no,” or “don’t know.” However, the military departments’ approaches for collecting data in their annual resident satisfaction surveys vary, which limits their ability to assess whether residents would recommend privatized housing. Instead of asking whether residents would recommend privatized housing, the military departments’ annual resident satisfaction survey asks residents the following: “How much do you agree or disagree with the following statement, ‘I would recommend this community to others.’” A resident’s satisfaction with his or her community and inclination to recommend it to others may not be reflective of satisfaction with either the privatized housing unit or privatized housing in general. Residents are then provided the following response categories on a scale of five to zero: (5) strongly agree, (4) agree, (3) neither agree nor disagree, (2) disagree, (1) strongly disagree, and (0) not applicable, no opinion, don’t know, or no answer. Through our analysis, we have identified variances in the methods that each of the military departments use to translate the residents’ responses into the “yes,” “no,” or “don’t know” categories. The variances in how the military departments calculate “yes,” “no,” or “don’t know” resulted in inconsistencies in how resident satisfaction is ultimately reported to Congress. For example, for the fiscal year 2017 report, Navy and Army officials told us they counted responses reported in category 3 (neither agree nor disagree) as “don’t know.” For the same time period, however, Air Force officials told us they counted responses in category 3 (neither agree nor disagree) as “yes.” If the Air Force had not counted category 3 as “yes,” reported resident satisfaction rates would have been lower. For example, for one Air Force installation, if officials had not counted responses in category 3 as “yes,” the resident satisfaction rate for newly constructed units would have been more than 20 percent lower than what was reported. In our draft report, we also identified instances of errors and inaccuracies in how OSD calculates these data and reports on resident satisfaction to Congress. Specifically, we found missing data points and incorrect formulas, among other errors, in OSD’s calculation of the data submitted by the military departments for OSD’s fiscal year 2017 report to Congress. For example: The formula used by OSD to calculate overall resident satisfaction for the fiscal year 2017 report did not include data for several projects, including for four Army projects that, as of September 30, 2017, accounted for over 18 percent of the Army’s total housing inventory. For one Air Force project, OSD reported identical resident satisfaction data for the fiscal year 2015, 2016, and 2017 reports, despite the fact that Air Force officials had noted in their submissions to OSD that the resident satisfaction data were from the annual resident satisfaction survey conducted in December 2013. In our draft report, we also found that presentation of data in OSD’s report to Congress may be misleading because OSD did not explain the methodology it used to calculate the overall resident satisfaction percentage or include caveats to explain limitations to the data presented. Specifically, OSD did not include information on overall response rates to the annual satisfaction survey for each military department, nor did it include response rates by project. Low response rates can create the potential for bias in survey results. For example, in the report for fiscal year 2017, OSD reported that 25 percent of residents living in renovated housing units for one privatized housing project were satisfied with their housing, but we found that only four residents had provided responses to this question. Thus, only one resident reported being satisfied. In addition, we found that OSD did not provide an explanation in the report for why five projects were listed as “not applicable.” According to OSD officials, this error was a quality control issue that they plan to address. According to OSD officials, there are no plans for quality control in development at this time. The National Defense Authorization Act for Fiscal Year 2020 (fiscal year 2020 NDAA) includes a provision requiring each military installation to use the same satisfaction survey for tenants of military housing— including privatized military housing—the results of which are not to be shared with private partners until reviewed by DOD. Until OSD makes changes to the data collection and calculation efforts that make up the department’s report to Congress and provides explanations of the data in the reports, OSD will not be able to provide Congress with an accurate picture of resident satisfaction with privatized housing. Military housing office officials, located at each installation, are available to provide resources to servicemembers experiencing challenges with their privatized housing, among other services. However, as we stated in our draft report, we found that these offices have not always clearly and systematically communicated this role to residents of privatized housing. The military housing office is to provide new residents with information on their local housing options, to include referral services for housing options. According to some military housing office officials, the military housing office then works with the private partner to identify the eligibility and type of home the servicemember qualifies for, if the resident chooses to live in privatized housing. According to some residents we spoke with in one of our focus groups, beyond this initial interaction, military housing office officials generally do not interact with residents on a regular basis. Additionally, residents who participated in our focus groups noted they were sometimes confused about the military housing offices’ roles and responsibilities with regard to the maintenance of their home; there was a perception that the military housing office was not working independently of the partner in the residents’ best interest; or they did not know the military housing office existed. The military department oversight agencies have also found that the military departments have not clearly and systematically communicated their roles to residents, and resident confusion and a lack of awareness regarding the role of the military housing offices is an issue. In April 2019 the Air Force Inspector General reported that less than half of the residents interviewed used their military housing office to resolve complaints, and at some installations officials visited, many residents did not know the military housing office had an oversight role. Similarly, in May 2019, the Army Inspector General reported to the Secretary of the Army that at 82 percent of Army installations with privatized housing, residents did not know how to escalate issues to either the private partner or the Army housing office. Additionally, the Army Inspector General reported that installation command teams and staff cited multiple circumstances where military housing offices and tenant advocacy roles and responsibilities were unclear. Further, some military housing office officials with whom we spoke during our site visits acknowledged the gap in resident awareness regarding the existence and purpose of the military housing office. Some military housing officials also noted that some residents are unaware of the difference between the military housing office and the private partner office, due in part to their physical co- location and unclear building signage. Each military department has issued information that establishes that its housing offices can assist in the resident dispute resolution process. Specifically, if servicemembers are experiencing a dispute with a private partner, military department guidance establishes varying roles for their respective military housing office officials. For example, Army policy states that each installation should have an official tasked with supporting servicemembers regarding resident issues that cannot be resolved by the private property manager. This individual is also responsible for resolving every resident complaint and the military housing office, if required, can request mediation by the garrison commander. OSD has recognized that the military departments’ communication with residents about their role as a resource for them has been limited. In February 2019, the Assistant Secretary of Defense for Sustainment testified before Congress that a way forward in addressing resident concerns would require focus in three key areas: communication, engagement, and responsiveness. Some military housing office officials told us they have taken steps to increase resident awareness, such as increasing the advertising of the military housing office’s role and contact information, conducting town hall meetings, and rebranding their military housing offices to differentiate them from the private partners. For example, a Marine Corps housing office official stated that the housing office established a document, which is distributed to residents by the private partner, informing residents of housing office contact information and the service’s three-step dispute resolution process, but efforts have not been standardized across all projects. Moving forward, having plans in place to clearly and systematically communicate the difference between the military housing office and the private partners—including the military departments’ roles, responsibilities, and military housing office locations and contact information—will better position the military departments to achieve the intended objectives of their initiatives aimed at improving residents’ experience. OSD, the military departments, and the private partners have identified and begun collaborating on a series of initiatives aimed at improving residents’ experiences with privatized housing, but as we state in our draft report currently with DOD for review and comment, these efforts face challenges. In addition, in the fiscal year 2020 NDAA, Congress established several requirements regarding privatized military housing reform. Several of the statutory requirements provide specific provisions that DOD will need to incorporate into its development and implementation of existing MHPI initiatives, as well as additional requirements aimed at improving oversight of privatized housing. In our draft report, we discuss several of these key initiatives, including the following. Development of a resident bill of rights. DOD has been working to develop a resident bill of rights intended to provide clarity to residents on their rights and responsibilities while living in privatized military housing. The fiscal year 2020 NDAA includes specific requirements to be included in the bill of rights, for example, ensuring residents have the right to have their basic allowance housing payments segregated and held in escrow, with approval of a designated commander, and not used by the property owner, property manager, or landlord pending completion of the dispute resolution process. In January 2020, DOD officials told us that they were in the process of updating their existing resident bill of rights to include these provisions. In February 2020, the Secretary of Defense signed the resident bill of rights, noting that the rights would be available to residents on May 1, 2020. Implementation of a common (enterprise) dispute adjudication process that will apply to all projects. The military departments and private partners have been working to develop a common dispute resolution process that would apply to all privatized housing projects. The fiscal year 2020 NDAA includes requirements reinforcing this initiative, specifically stating that the military department Secretary concerned shall implement a standardized formal dispute resolution process to ensure the prompt and fair resolution of disputes between landlords providing housing units and tenants residing in housing units concerning maintenance and repairs, damage claims, rental payments, move-out charges, and such other issues relating to housing units as the Secretary determines appropriate. Additionally, the statute requires that each military department Secretary designate the installation or regional commander in charge of oversight of housing as the deciding authority under the dispute resolution process. Reviewing MHPI resident satisfaction data collection process and the process by which DOD measures and reports on resident satisfaction data. According to OSD officials, the department is reviewing the process by which it measures and reports resident satisfaction data, and has plans to review the survey questions used to measure resident satisfaction. In line with these planned efforts, the fiscal year 2020 NDAA further requires that DOD’s reports to Congress include additional information, such as the results of residence surveys and other factors related to the condition of privatized housing. Standardizing Performance Incentive Fee Ranges. In October 2019, OSD issued new guidance standardizing the performance incentive fee ranges across the military departments. The fiscal year 2020 NDAA requires that DOD publically report information regarding the use of performance incentive fees. The statute also requires that DOD take into consideration any decision a commander renders in favor of the tenant in the formal dispute resolution process in determining whether to pay or withhold all or part of any incentive fees for which a private partner may otherwise be eligible under the contract. In addition to requirements impacting current DOD initiatives, the fiscal year 2020 NDAA included requirements for increased oversight of the physical condition of privatized housing. For example, the statute requires the Secretary of Defense to designate a Chief Housing Officer to oversee housing units, including the creation and standardization of policies and processes regarding housing units. The statute also requires the Secretary of Defense to establish a uniform code of basic standards for privatized military housing, as well as plans to conduct inspections and assessments of the condition of privatized homes. However, both DOD and private partner representatives have cited several challenges that could affect their ability to implement initiatives aimed at improving MHPI. Specifically: Timeliness of implementation due to the need to collaborate with and obtain input and agreement from the large number of stakeholders involved in privatized housing. According to DOD officials and private partner representatives, many of the initiatives designed to improve privatized housing not only require agreement between DOD and the private housing partners, but also discussion with and, in some cases, approval by the project bond holders. Because DOD does not have the ability to unilaterally make changes to existing business agreements, this need for stakeholder agreement limits DOD’s control over the implementation timeline of any initiative that requires changes to a project’s business agreement. The need for more military department staff with targeted expertise. The military departments reduced their involvement in daily privatized military housing operations as part of the overall privatization effort, to include reducing staffing levels at the installations. Military housing office officials at over half of the installations we visited stated that reduced staffing levels impacted their ability to carry out oversight duties, such as work order data analysis and housing inspections. Each of the military departments has plans to increase the military housing office staffing at each installation to allow for enhanced oversight. In particular, according to military department officials, these positions will focus on quality control and quality assurance of the maintenance of privatized homes. The potential for unintended negative financial impacts on the projects that could outweigh the intended benefits of the initiatives. OSD officials and private partner representatives have expressed concern that some initiatives could result in unintended financial consequences for the housing projects. For example, increased frequency of change-of-occupancy inspections could result in homes remaining vacant longer than planned and therefore not collecting rent. This could unintentionally impact a project’s cash flow. Some of the private partners noted that the financial impact of unfunded requirements to projects that are already experiencing financial distress could result in even fewer funds available to reinvest in improvements to the current and future physical condition of the homes. Without assessing risks to the financial viability of the MHPI projects associated with the implementation of these initiatives aimed at improving privatized housing, DOD’s efforts to improve the privatized housing program over the long term could be compromised. In summary, as we state in our draft report, we found that while DOD and the private partners have taken steps to address concerns raised about their ability to adequately maintain and oversee the condition of these housing units and provide quality housing for servicemembers, the extent to which the efforts will be sustained and result in improvements remains unclear. Our draft report includes several recommendations to OSD to strengthen oversight of MHPI, such as updating oversight guidance and assessing the risks to the financial viability of housing projects. Our draft report also includes recommendations to the military departments to enhance monitoring of privatized housing projects, such as improving processes used for data collection; reviewing private partner performance; collecting and reporting resident satisfaction data; and communicating with residents. Chairwoman Wasserman Schultz, Ranking Member Carter, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff members have any questions about this testimony, please contact Elizabeth A. Field, Director, Defense Capabilities and Management, at (202) 512-2775 or FieldE1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony include Kristy Williams (Assistant Director), Tida Reveley (Analyst in Charge), Austin Barvin, Ronnie Bergman, William Carpluk, and Jordan Tibbetts. In addition, key support was provided by Vincent Buquicchio, Juliee Conde- Medina, Mae Jones, Kelly Rubin, Monica Savoy, John Van Schaik, Madeline Welter, and Kelsey Wilson. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Congress enacted the Military Housing Privatization Initiative in 1996 to improve the quality of housing for servicemembers. DOD is responsible for general oversight of privatized housing projects. Private-sector developers are responsible for the ownership, construction, renovation, maintenance, and repair of about 99 percent of military housing in the United States. Recent reports of hazards, such as mold and pest infestation, have raised questions about DOD's oversight. This statement summarizes GAO's draft report on privatized housing, which is currently at DOD for review and comment. Specifically, the statement discusses, among other objectives, OSD and the military departments' (1) oversight of privatized military housing and (2) development and implementation of initiatives to improve privatized housing. For its draft report, GAO reviewed policies and guidance; visited a non-generalizable sample of 10 installations representing each military department, among other factors; analyzed work order data; and interviewed DOD officials and private partner representatives. The Office of the Secretary of Defense (OSD) and the military departments conduct a range of oversight activities, but some of these activities have been more extensive than others. Specifically, GAO's draft report notes: The military departments conduct some oversight of the physical condition of housing, but some efforts have been limited in scope. Military departments have authority to conduct oversight of the condition of privatized housing; that oversight generally consists of reviewing a sample of work order requests, visual inspections of housing during change of occupancy, and other point in time assessments. However, GAO found that these efforts are limited in scope. For example, annual interior walk-throughs are limited to just a few homes at some installations, which may not comprehensively reflect the condition of the housing units at those installations. Military departments use performance metrics to monitor private partners, but metrics do not provide meaningful information on the condition of housing. OSD has recently issued guidance to ensure consistency in the framework used to measure project performance. However, the specific indicators used to determine if the metrics are being met do not accurately reflect private partner performance related to the condition of the home. For example, a common indicator is how quickly the private partner responded to a work order, not whether the issue was actually addressed. The military departments and private partners collect maintenance data on homes, but these data are not captured reliably or consistently. The Department of Defense (DOD) is expanding its use of work order data to monitor and track the condition of privatized housing. However, based on GAO's analysis of data provided by all 14 private partners, these data cannot reliably be used for ongoing monitoring of privatized housing because of data anomalies and inconsistent business practices in how these data are collected. DOD provides reports to Congress on the status of privatized housing, but some data in these reports are unreliable, leading to misleading results. DOD provides periodic reports to Congress on the status of privatized housing, but reported results on resident satisfaction are unreliable due to variances in the data provided to OSD by the military departments and in how OSD has calculated and reported these data. OSD and the military departments have made progress in developing and implementing a series of initiatives aimed at improving privatized housing. In addition, Congress established several requirements addressing privatization housing reform. However, DOD officials and private partner representatives have identified challenges that could affect implementation of these various initiatives. These include concerns that implementation could have unintended negative impacts on the financial viability of the privatized housing projects. GAO's draft report includes several recommendations, including that DOD take steps to improve housing condition oversight, performance metrics, maintenance data, and resident satisfaction reporting as well as to assess the risk of initiatives on project finances.", "document_type": "gao"}
{"report": "Companies that offer executive retirement plans typically do so to supplement benefits provided under qualified retirement plans or to provide retirement benefits in lieu of a qualified retirement plan. In an executive retirement plan, a select group of managers or highly compensated employees defer the receipt of compensation earned in one year to be paid in a future year, generally at or after retirement. Executive retirement plans are not subject to certain statutory limits that apply to qualified retirement plans, such as limits on the annual amount of benefits received, the annual amount of contributions made to the plan, or the annual compensation level used to determine benefits and contributions. Executive retirement plans can be structured as defined benefit plans or defined contribution plans but generally must defer compensation to a future year. For executive retirement plans structured as a defined contribution plan, executives’ benefits are based on a plan account balance. During the deferral period, companies will typically allow executives to select from among a menu of market indices (e.g., of stock, or bond performance or of interest rates) or other investment options and base the plan account balance on the performance of those selections. The company generally credits plan contributions and changes in the value of the plan account balance to executives, but does not have to make actual investments that correspond to executives’ selections because companies are not obligated to designate funds for the plan before distributions are made. For executive retirement plans structured as a defined benefit plan, executives are typically paid based on a formula that accounts for salary and years of employment. Distributions from all executive retirement plans are made from company assets. In the first objective of this report, we discuss and illustrate the defined contribution form of executive retirement plans, except as otherwise indicated. ERISA contains various provisions intended to protect the interests of plan participants and beneficiaries in workplace retirement plans. These protections include requirements related to reporting and disclosure, participation, vesting, and benefit accrual, as well as plan funding. Generally, most of the substantive protections of ERISA do not apply to executive retirement plans. Specifically, ERISA requirements pertaining to participation, vesting, funding, and fiduciary responsibilities do not apply to executive retirement plans. The policy underlying the executive retirement plan exemption from the substantive provisions of ERISA has been described by DOL as based on a recognition by Congress that “certain individuals, by virtue of their position or compensation level, have the ability to affect or substantially influence, through negotiation or otherwise, the design and operation of their deferred compensation plan.” Additionally, ERISA grants DOL the authority to prescribe alternative methods of compliance for the reporting and disclosure provisions under Part 1 of Title I for any plan or class of plans, which includes executive retirement plans. Using this authority, DOL issued a regulation permitting administrators of executive retirement plans to submit a one- time single page filing statement to satisfy ERISA reporting requirements in 1975, according to DOL. DOL’s executive retirement plan filing statement includes: the name and address of the employer, the employer identification number (EIN) assigned by the IRS, a declaration that the employer maintains a plan or plans primarily for the purpose of providing deferred compensation for a select group of management or highly compensated employees, and a statement of the number of such plans and the number of employees in each plan. In addition, plan administrators are required to provide plan documents to DOL upon request. The Internal Revenue Code (IRC) provides preferential tax treatment for workplace retirement plans that meet certain qualification requirements set out in the IRC. The structure of tax incentives and certain limits on qualified retirement plans are intended to balance encouraging employers to establish and maintain voluntary, tax-qualified pension plans with ensuring lower-income employees receive an equitable share of the tax-subsidized benefits. Although executives may benefit from tax deferral under an executive retirement plan, these plans are not eligible for the same preferential tax treatment afforded to qualified retirement plans under the IRC. For the executive to be eligible for the tax deferral, executive retirement plans must be an “unfunded and unsecured” company promise to pay benefits in the future. Generally, for an executive retirement plan to be considered unfunded and unsecured, the executive’s rights to receive plan distributions will be no greater than the rights of an general unsecured creditor in the event of company bankruptcy or insolvency. Companies are not permitted to fund (i.e., set aside assets for the exclusive benefit of participants that are separate from company assets and beyond the reach of creditors) executive retirement plans while maintaining the benefits of tax-deferral for executives. However, companies are able to “informally fund” executive retirement plans by transferring amounts to a trust that remains part of the company’s general assets—often referred to as a “Rabbi Trust”—to help keep its promise to pay benefits. Because executive retirement plans are unfunded, executives’ benefits in these plans can be subject to credit risk of non-payment, such as in the event of a company bankruptcy, according to IRS officials. The IRC provides rules regarding deferring compensation in executive retirement plans, including restrictions on the timing of distributions, restrictions on payment acceleration, and restrictions on the timing of deferral elections. At the time of deferral, the amount of compensation deferred under the plan is generally excluded from executives’ income for tax purposes and not tax deductible for the company (see fig. 1). During the deferral period, because any assets associated with the executive retirement plan remain company assets (and subject to creditor claims), the company is subject to applicable taxes on any investment earnings attributable to the assets. Executives are subject to federal income taxes on their executive retirement plan distributions when they are received. However, if an executive retirement plan fails to meet the applicable requirements at any time during a taxable year, all of the compensation deferred, including investment earnings associated with the deferred compensation, is included in each executive’s gross income for the taxable year to the extent it is vested, along with an additional 20 percent tax on the compensation to be included in gross income plus additional income tax. Companies must defer taking their tax deductions, up to statutory limits, for plan contributions they make until the executive is taxed on those benefits. In addition to DOL’s role under ERISA and IRS’s role administering the IRC requirements related to executive retirement plans, other federal agencies may have roles related to executive retirement plans. For example, SEC requires public companies to provide an annual proxy statement that includes information on the amount and type of executive compensation—including benefits from executive retirement plans—paid to their Chief Executive Officer (CEO), Chief Financial Officer (CFO), and the next three most highly compensated executive officers. Other federal agencies that play a role with respect to qualified retirement plans, such as the PBGC, may monitor the status of executive retirement plans in certain circumstances, such as in bankruptcy proceedings involving a company with both an executive retirement plan and a qualified single-employer defined benefit plan (see table 1). According to our analysis, more than 400 of the 500 largest U.S. public companies provided executive retirement plans to almost 2,300 top executives, totaling about $13 billion in accumulated plan benefits in 2017 (see fig. 2). Although DOL collects limited data on the prevalence of executive retirement plans, public companies subject to SEC reporting requirements for executive retirement plans must report the benefits provided to the Chief Executive Officer (CEO), Chief Financial Officer (CFO), and the next three most highly compensated executive officers. Industry experts we interviewed said that most large companies offer executive retirement plans to help executives and highly compensated employees save more for retirement because most executives have reached the contribution and income limits imposed on savings in qualified retirement plans. Top executives at large public companies generally accumulated more executive retirement plan benefits than top executives at smaller companies. The most recent available data from 2017 show that the average accumulated plan benefit among the top five executives in large companies was about $5.7 million, about twice as much as their counterparts in smaller companies, where the average was about $2.8 million. The average and median accumulated plan benefits generally remained consistent for large and smaller companies from 2013 to 2017 (see fig. 3). In addition, our analysis showed that, among the top five executives at large public companies, accumulated plan benefits are concentrated among a subset of these top executives based on their job title, company contributions, and plan type. The average accumulated plan benefit among top executives in large companies was consistently greater than the median accumulated plan benefit from 2013 to 2017 (see fig. 3). For example, as of 2017, the average accumulated plan benefit among top executives was more than four times the median, indicating that plan benefits for a smaller subset of executives is greater than a majority of other individual executives. CEOs accumulated more executive retirement plan benefits than the next four highest compensated executives. As of 2017, the CEOs had accumulated, on average, about $14 million in executive retirement plan benefits. In contrast, CFOs had accumulated, on average, about $3 million and the next three most highly compensated executive officers with other titles accumulated an average of about $3.4 million in accumulated plan benefits. Our analysis also showed that, for each of the three job title categories (CEO, CFO, and the next three most highly compensated executive officers), the average accumulated plan benefits were at least twice the median amount from 2013 to 2017 (see fig. 4). From 2013 to 2017, about 80 percent of large companies that offered an executive retirement plan made company contributions to the plan. As of 2017, the average accumulated plan benefit for top executives among companies providing company contributions was more than $6.5 million. This was more than twice the average of nearly $2 million for executives in about 20 percent of the remaining companies that offered an executive retirement plan that did not include company contributions. Our analysis showed that plan benefits are also concentrated among a subset of executives as the average amount of accumulated plan benefits for executives in plans that received company contributions were several times greater than the median from 2013 to 2017 (see fig. 5). The top five executives with defined benefit executive retirement plans generally accumulated more plan benefits than those with defined contribution executive retirement plans alone. As of 2017, about 30 percent of large companies that sponsored an executive retirement plan offered a defined benefit plan, as compared with about 70 percent that only offered a defined contribution plan. In 2017, the top five executives at large companies with a defined benefit plan had accumulated plan benefits of nearly $9 million on average, more than twice the average of about $4.4 million for top five executives with defined contribution executive retirement plans alone. Our analysis showed that plan benefits are concentrated among a subset of executives as the average accumulated plan benefits for top five executives with a defined benefit plan was several times more than the median from 2013 to 2017 (see fig. 6). However, industry experts told us the number of companies offering defined benefit executive retirement plans has declined over time. Executive retirement plans can help executives reduce their potential tax liability, increase retirement savings, and provide financial planning advantages through: (1) tax substitution of investment earnings, (2) additional company compensation for investment earnings, (3) additional company compensation for personal income taxes, and (4) allowable distributions during working years. Treasury officials and some industry experts told us that executives who participate in executive retirement plans may be able to reduce their potential federal tax liability on plan investment earnings and increase their savings because these plans substitute the executive’s applicable individual tax rate on investment earnings with the company’s corporate tax rate (see fig. 7). In an executive retirement plan, the company defers compensation for the executive, but investment earnings on associated assets during the deferral period are taxed to the company at the company’s applicable corporate tax rate (see “Executive defers compensation” at top of fig. 7). In contrast, the executive who chooses not to defer compensation and instead takes the current compensation (paying income taxes) and invests the balance will pay taxes on investment earnings at the individual tax rate (see “Executive does not defer compensation” at bottom of fig. 7). The actual taxes paid under either scenario—deferring compensation or not—will depend on a number of factors, including the type of investments, if any, selected by the executive or the company, length of time invested, and applicable tax rates. For example, an executive who does not defer compensation and invests outside of the plan might select investments that are expected to produce long-term capital gains, which are taxed at lower individual rates than short-term capital gains. This same executive, if deferring compensation through the plan, might elect to invest in short-term bonds or investment earnings based on a market interest rate, which are taxed at a lower corporate tax rate inside the plan than outside. As another example, a company might invest deferred compensation in a tax-favored vehicle such as corporate- owned life insurance. According to Treasury officials and some industry experts, by participating in an executive retirement plan, executives may be able to effectively reduce their potential federal income tax liability during the deferral period because investment earnings on associated plan assets are taxed at the company’s corporate rate that may be lower than the executive’s individual tax rate. This tax substitution of investment earnings may allow the plan account to grow over time at a higher rate of investment return than if an executive invested in the same or similar assets outside the plan. Further, any such tax advantages may allow companies to reduce their total compensation costs. Conversely, Treasury officials told us the IRC may effectively disadvantage executive retirement plans to the extent the tax on an executive’s investment earnings outside the plan is lower than the tax the company would pay if invested through the plan. In this circumstance, the tax disadvantage may increase the cost of companies’ total compensation. However, our analysis of tax rates suggests that the corporate tax rate may be lower than the individual tax rate on several forms of investment income. In this case, the company may be able to achieve a higher after-tax rate of return on investments than the executive can, depending on the type of investment and amount of time invested. The lower the applicable corporate tax rate is relative to the applicable individual tax rate, the greater the tax benefit for the executive or the company. Treasury officials and some industry experts told us that, in this scenario, the potential tax advantage resulting from tax substitution of investment earnings is effectively a federal subsidy because the federal government receives less in tax revenue. And due to the effects of compounding, the tax advantage is also greater the longer the deferral period (and higher the investment return). Treasury officials and experts whose published work we reviewed and interviewed told us the potential effective federal tax subsidy for executive retirement plan investment earnings can be greater when companies have effective tax rates that are lower than statutory tax rates. This can occur, for example, when a company’s losses from the current year or losses carried over from prior years offset all other company income, including any investment earnings associated with their executive retirement plan. In these instances, the federal government could effectively subsidize the plan investment earnings because it receives no taxes on those earnings until funds are distributed. Companies also provide executives with additional executive retirement plan compensation that increases their overall savings by not passing along taxes paid on investment earnings during the deferral period, according to Treasury officials and some industry experts. In this scenario, a company’s assets associated with the executive retirement plan are reduced for taxes it pays on investment earnings, but the executive’s corresponding plan account balance is unaffected by tax because the company provides the executive with additional plan compensation in the same amount as the taxes the company pays. Unaffected by taxation on investment earnings, the account balance accumulates over time at a pre-tax investment rate of return, rather than at the company’s potentially lower after-tax investment rate of return, until those funds are distributed to the executive. In this manner, this additional compensation provided by the company allows the account balance of an executive retirement plan to accumulate in the same way as in a qualified defined contribution retirement plan (e.g., a 401(k) plan). The additional compensation can result in a substantial benefit for an executive, and due to the effects of compounding, the benefit is greater the longer the deferral period (and higher the investment return). Lastly, industry experts said some companies provide additional executive retirement compensation to pay for the personal income taxes that executives expect to pay when plan benefits are distributed. This practice is known as a tax “gross-up” because the company increases the amount of gross or pre-tax executive retirement plan benefits to pay for the executive’s anticipated income taxes at distribution. As a result, the executive effectively receives the total amount of the initial pre-tax benefit at distribution. For example, a company that wants an executive who is in the 37 percent income tax bracket to receive $1,000 from the plan on an after-tax basis would provide an additional $588 in plan compensation (for a total of $1588) to cover the executive’s anticipated taxes at distribution. Treasury officials said that while tax gross ups and other similar executive compensation practices provide an economic benefit to executives, these practices by companies to offset executives’ tax burden is a corporate governance issue for shareholders to decide and that tax law does not address their appropriateness. Some industry experts told us that it has become less common for public companies to offer tax gross-ups, mostly due to shareholder concerns about their appropriateness in light of required public disclosures. Executive retirement plans can also provide executives with financial planning benefits through allowable distributions during their working years. Treasury officials and industry experts said that while executive retirement plans are intended for retirement purposes, plans typically also allow executives to take distributions while still working. These distributions generally are allowed if they comply with applicable statutory requirements. Industry experts told us that executives can align distributions during their working years with income needs, such as to pay for a child’s college expense, or for specific goals, such as buying a home. Industry experts said that the ability to structure pre-retirement distributions can allow executives to smooth out their overall income over time to better coordinate use of other income sources during their working years and retirement, which they said can lead to overall tax savings. Executive retirement plans can provide tax advantages that may have revenue effects for the federal government, but the extent of those effects currently is unknown. Treasury is responsible for providing economic analysis and revenue estimates of tax legislation for the executive branch, and Treasury officials said that the Congressional Joint Committee on Taxation prepares official revenue estimates of all tax legislation considered by the Congress. Treasury officials told us that while executive retirement plans do not receive the preferential tax treatment afforded to qualified retirement plans, these arrangements can result in tax advantages that may have revenue effects for the federal government. These officials explained that executive retirement plans are tax revenue neutral when corporate tax rates and individual tax rates (or taxes paid) are the same because the federal government would generally receive the same amount of taxes regardless of the executive’s decision to defer compensation. Treasury officials also told us that executive retirement plans could have federal revenue effects to the extent corporate and individual tax rates (or taxes paid) diverge from each other. Among the 38 Chapter 11 corporate bankruptcy cases we reviewed, 30 cases showed that participants in executive retirement plans expected to receive general unsecured creditor status when settling their plan benefit claims. As a general unsecured creditor, executives in these plans are part of what is typically the last creditor class to be paid in bankruptcy, and only if funds remain after claims from all other creditors with payment priority have been paid in full (see fig. 8). Our review of bankruptcy cases showed that executives’ expected losses and recoveries varied among the 30 Chapter 11 cases we reviewed where all or some plan participants were expected to receive general unsecured creditor status for their plan benefit claims (see fig. 9). In 21 of the 30 cases, plan participants were expected to sustain losses of more than 75 percent of their plan benefit claims, and in 17 of these 21 cases, participants were estimated to lose 90 percent or more. However, the remaining nine cases showed that participants were expected to recover more than half of their plan benefit claims with six of those cases expecting a full recovery and one case expecting a 99 percent recovery. Companies generally file for bankruptcy when they do not have sufficient assets to pay off their debts. Bankruptcy and industry experts said that executive retirement plan participants as general unsecured creditors may expect to sustain a significant or even a total loss of their deferred compensation in a company bankruptcy. However, bankruptcy and industry experts noted that the level of losses or recoveries depends on the facts and circumstances of each case, including the type of bankruptcy the company filed. Our review of bankruptcy cases showed differences in expected benefit losses and recoveries based on whether the bankrupt company intended to continue to operate by filing a reorganization plan or sell all of its assets to pay creditors by filing a liquidation plan. Among the 30 Chapter 11 bankruptcy cases where participants in executive retirement plans were expected to receive general unsecured creditor status, 14 filed a reorganization plan and 16 filed a liquidation plan. Among the bankruptcy cases we reviewed, executives were generally estimated to sustain less severe claims losses and recover more of their plan benefits if their company filed a reorganization plan to continue to operate and restructure its debts. In seven of 14 reorganization cases we reviewed, executive retirement plan participants were estimated to recover about 80 percent or more of their plan benefit claims, with participants in six of those cases expected to fully recover their benefits. In contrast, participants in the remaining seven of 14 cases were estimated to sustain benefit claims losses of about 20 percent or more, with participants in five cases expected to lose 90 percent or more. Industry experts told us plan participants are more likely to sustain fewer losses when their bankrupt company reorganizes because it has a plan to emerge from bankruptcy and pay its debts as it continues to operate. Bankruptcy and industry experts noted that in some reorganization cases, general unsecured creditors can receive full recoveries. Executives were generally estimated to sustain greater plan benefit claim losses if their company filed a liquidation plan. In 15 of 16 liquidation cases we reviewed, executive retirement plan participants were estimated to sustain losses of nearly 50 percent or more of their plan benefit claims. Participants in the remaining case were expected to nearly fully recover their benefits. Industry experts told us that whether a company has a viable post-bankruptcy future affects its ability to fulfill its debt obligations, including paying promised plan benefits to executive retirement plan participants. Bankruptcy experts said the severity of plan benefit claims losses for participants is generally greater when a bankrupt company liquidates because it signals the end of a company and is a last resort after it has exhausted all other options to restructure its debts and continue to operate. Among the 38 Chapter 11 bankruptcy cases we reviewed, 11 involved the situation where all or some of the executive retirement plan participants were not expected to receive general unsecured creditor status for their benefit claims. Although the circumstances varied among these 11 cases, the expected outcome was that some of these participants’ plan benefits which were accrued at or around the time the company filed for bankruptcy were expected to be preserved or paid. Among the 11 cases we reviewed in which executive retirement plan benefits were expected to be maintained, eight occurred with a bankrupt company that filed a reorganization plan. In three of the eight cases, benefits for all plan participants were expected to be preserved; in five cases participants were divided into different groups where some were expected to have their benefits preserved and others were not. Bankruptcy and industry experts said that, paying plan benefit claims in a bankruptcy often depends on the financial health of the company and the value of the executive to the future of the company. These experts also said that not all executive retirement plan participants receive the same treatment for their claims. These experts added that a common scenario is to preserve in some manner the benefits for key executives who are retained, while giving executives who are not retained, or former executives no longer with the company, less favorable treatment as a general unsecured creditor. Industry experts also told us that some executive retirement plan participants’ benefits may be preserved or the participants may be provided with more favorable treatment because they are key executives who need to be retained to help ensure their company successfully reorganizes and emerges from bankruptcy. These experts explained that key executives may not be willing to risk staying on without assurances that accrued plan benefits will be preserved or made up in some manner. Bankruptcy and industry experts said that because key high-level executives can be integral to the success of a company reorganization, its major creditors are more likely to agree to preserve plan benefits for them because it will likely result in increased overall recoveries and greater benefits for their stake in the company. Lastly, bankruptcy and industry experts said that in order for bankrupt companies to retain key executives, they typically need to provide assurances that, in addition to executive retirement plan benefits, executives will receive other forms of compensation. Bankruptcy and industry experts noted that because various forms of executive compensation may be interchangeable to the executive, informal agreements may be arranged so that executive retirement plan benefit losses that may occur as a general unsecured creditor are made-up through other forms of compensation. However, they told us these types of arrangements are not discernable from bankruptcy filings. In three of the 11 cases we reviewed in which executive retirement plan benefits were expected to be preserved, the companies filed a Chapter 11 liquidation plan. Court filings indicated executive retirement plan participants in two of the three cases received distributions shortly before the company filed bankruptcy. In one case, the bankruptcy estate chose not to seek to recover those funds despite restrictions for early distributions before a bankruptcy in part because the costs to recover the monies outweighed the benefits. Bankruptcy and industry experts said that while there are restrictions and penalties for early distributions before a bankruptcy, the costs and time associated with suing to recover monies can discourage bankruptcy estates from pursuing legal action. IRS oversees executive retirement plans for compliance with the IRC during audits of companies who offer such plans. The Pension Protection Act of 2006 amended the IRC to provide that, during a restricted period, which includes bankruptcy, if a company that sponsors a qualified single- employer defined benefit plan sets aside or reserves assets in a trust for the purposes of paying nonqualified deferred compensation (which includes executive retirement plan compensation) to applicable covered employees (key executives), the key executives are required to include the amount of assets in their gross income for the taxable year. A restricted period is defined as: (1) any period in which the plan sponsor is a debtor in bankruptcy; (2) any period when the qualified single-employer defined benefit plan of the company is in at-risk status; or (3) the 12- month period that begins 6 months before the date the qualified single- employer defined benefit plan is terminated if, as of the termination date, the plan’s assets are not sufficient to cover benefit liabilities. In general, a company’s qualified single-employer defined benefit plan is in at-risk status if it is less than 80 percent funded. As part of its oversight effort, IRS officials said that its examiners can use IRS’s Nonqualified Deferred Compensation Audit Techniques Guide (the guide) to audit these plans for compliance with the IRC, including the relevant provision, which was added by the Pension Protection Act of 2006. The guide describes the requirements in section 409A of the IRC related to deferred compensation set aside during a restricted period. While the guide is designed to provide guidance for IRS employees, the guide is publicly available and also useful for businesses and tax professionals who prepare returns. However, the guide does not instruct examiners or other users on how to determine compliance with the relevant provision. For example, the guide does not instruct examiners or other users to determine if the company has set aside assets—such as by making contributions of funds to a Rabbi Trust—to pay deferred compensation during bankruptcy. It also does not require examiners or other users to obtain data sufficient to determine whether there exists a restricted period with respect to the company’s qualified single-employer defined benefit plan. Lastly, it does not provide instructions regarding the type of data to collect or questions to ask to determine whether a company’s defined benefit plan is in a restricted period. When asked if additional instructions were available to examiners on auditing companies with these plans for compliance with the relevant provision, IRS officials pointed us to sections of the Internal Revenue Manual (IRM), IRS’s primary source of instructions to staff, and other internal training manuals. However, we found no specific instructions in these sources related to the relevant IRC provision or its oversight. IRS officials said examiners can also review SEC filings to determine whether there exists a restricted period with respect to a company’s qualified single-employer defined benefit plan. However, SEC filing requirements do not apply to many privately-held companies, limiting the usefulness of this information source for IRS audit examiners for this purpose. IRS officials also said that Form 5500, Annual Return/Report of Employee Benefit Plan, and the attached schedules are available on the DOL website and that examiners can download and review these data during their examinations. For example, officials said information on the 5500 Form’s Schedule SB, Single-Employer Defined Benefit Plan Actuarial Information, can be used to verify the income tax deduction for contributions to pension plans. Specifically, the schedule’s Item 4 box, Part I Basic Information, will be marked if the plan is in at-risk status. The form, however, does not capture whether companies set aside assets for the purpose of paying deferred compensation or elicit information about a company’s bankruptcy. Moreover, the IRM, the guide, and the IRS training manuals provide no instruction to examiners regarding how to review this information during audits of companies with executive retirement plans. IRS also may be able to use non-confidential information that PBGC collects to monitor the financial condition of companies that sponsor single-employer defined benefit plans. In its capacity to provide plan termination insurance, PBGC monitors single-employer defined benefit plans—including companies’ financial condition and plans’ at-risk status— through a variety of reporting requirements and initiatives. For example, because PBGC represents itself and the pension plan and participants as a creditor when companies (publicly and privately-held) sponsoring single-employer defined benefit plans file for bankruptcy, it is aware of such bankruptcy filings. PBGC also uses data that companies are required to report on Form 5500, describing the assets and liabilities of their single-employer defined benefit plans, to identify when a defined benefit plan is underfunded or in at-risk status. IRS may be able to use the timely, non-confidential information PBGC possesses to help IRS identify whether companies with single-employer defined benefit plans are setting aside assets for the purpose of paying deferred compensation under an executive retirement plan during a restricted period. Federal standards for internal control require federal agencies to obtain and use quality information and to communicate this information to internal and external parties that can help the agency achieve its objectives and address related risks. Without providing specific instruction to its examiners to collect and evaluate information that describes company actions relative to this requirement limiting tax deferral for key executives for amounts deferred under an executive retirement plan and set aside by the company during a restricted period, IRS cannot sufficiently determine if companies are including these amounts in the executives’ gross income as required by the IRC provision. Without taking steps to improve the sufficiency of its audit instructions to help strengthen its oversight, IRS cannot know if companies are reporting the correct amount of income for taxation for these key executives and if the correct amount of tax is being paid by the executives in these instances. IRS also may not be collecting additional taxes and interest due from key executives who participate in executive retirement plans. Absent improved IRS oversight in this area, companies may be failing to report assets set aside to pay deferred compensation to key executives while in a restricted period as income for these employees. To the extent some companies are failing to report this income, they may continue to do so at the cost of foregone federal tax revenues while lacking an important incentive from IRS to cease this practice. Another aspect of executive retirement plan oversight is ensuring that only eligible executives are allowed to participate since these plans are excluded from most of ERISA’s substantive protections. DOL requires companies to report on their executive retirement plans, but the reporting lacks important information that could allow the agency to identify plans that may be including ineligible employees. Currently, under its alternative reporting method regulation, DOL regulations require the administrator of the executive retirement plan, typically the sponsoring company, to submit a one-time single page filing statement within 120 days of the executive retirement plan being established to satisfy ERISA reporting requirements (see fig. 10). According to DOL officials, no other filings are required for executive retirement plans to comply with Part 1 of Title I of ERISA. The information provided in the filing statement does not describe the job title or salary of executives participating in the plan, the percentage of the company’s workforce that is eligible to participate, or the actual percentage of employees who participate in the plan; nor does it compare the salaries of executives with rank-and-file workers. Because DOL only requires companies to submit the filing statement once within 120 days of plan formation, the agency is not aware when participation in the plan changes over time or if plans are terminated. When asked if these additional data would be useful to the agency, one DOL official said that they could be used to increase oversight of executive retirement plans. For example, the official said if the filing statement included the percentage of the company’s workforce that participated in such a plan, a high participation percentage could signal to DOL that the company might be permitting employees to participate in the plan who do not meet the “select group” requirements, and that such information could prompt a DOL audit. However, the DOL official said the agency would need to evaluate how the data would be used and the collection costs before determining the data’s overall value. The preamble to DOL’s regulation states that the agency chose to require limited reporting because these plans are for executives who generally have access to information concerning their rights and obligations under the plan and do not need ERISA protections. Moreover, DOL officials said there is no statutory requirement specifically directing the agency to collect executive retirement plan data and no requirement for companies to file an amended filing statement to report substantive plan changes. However, ERISA authorized DOL to prescribe an alternative method of reporting and the agency chose to require a limited one-time single page filing statement for executive retirement plans. DOL officials said the data currently collected can only be used for simple analysis or to facilitate the agency’s ability to respond to requests from Congress, the media, or the public. This limited usefulness regarding eligibility is due to the age and limits of the original data submitted. However, officials told us there currently is no plan to place executive retirement plan reporting on DOL’s regulatory project agenda. Federal standards for internal control state that agencies should (a) use quality information to achieve its objectives; (b) obtain data from reliable sources in a timely manner based on identified information requirements; and (c) process the data into quality information—information that is appropriate, current, complete, accurate, accessible, and timely—to support its internal control system. Without reviewing or clarifying its reporting requirements to allow the agency to collect more useful information on executive retirement plans, DOL will continue to lack insight into the composition of these plans and, as a result, may be missing opportunities to ensure that companies with executive retirement plans are meeting the eligibility requirements for the plan. Many industry experts we spoke to said that eligibility requirements for executive retirement plans are not clearly defined and that companies are unclear on how to establish eligibility. DOL has acknowledged that at least in one case a company may have denied ERISA protections to rank- and-file employees by allowing them to participate in executive retirement plans. DOL officials also said the agency has issued guidance on the executive retirement plan provisions in ERISA. For example, DOL pointed us to Advisory Opinion 90-14A, which DOL officials said is the agency’s most recent advisory opinion on provisions related to plan participant eligibility. The Advisory Opinion restates that executive retirement plans are excluded from most of ERISA’s substantive protections and describes DOL’s view that the term “primarily,” as used in the statute, refers to the purpose of the plan—the benefits provided—rather than the participant composition of the plan (see fig. 11). The Advisory Opinion further states DOL’s view that executive retirement plans that include employees who are not from a select group of management or highly compensated would fail to constitute a “select group” under ERISA, which would subject the plan to all of the requirements of Title I. Despite the information in the Advisory Opinion, several industry experts expressed the view that DOL’s current policy lacks specific information on the factors companies should consider when establishing eligibility for participation in these plans. Recent industry surveys we reviewed have suggested some companies may be extending employee eligibility to a relatively high percentage of their workforce—in some cases, more than 30 percent—and to relatively lower-paid or lower-ranked employees. For example, results from a recent survey of executive retirement plan sponsors suggested that just over 8 percent of respondents offer eligibility to between 20 to 30 percent of their workforce and just over 4 percent offer eligibility to more than 30 percent of their employees. Further, over 20 percent of respondents indicated that over 15 percent of their workforce was considered highly compensated employees and eligible to participate in an executive retirement plan. Industry experts pointed to court cases that they identified as contributing to the confusion regarding executive retirement plan eligibility, including cases that have suggested a limit on the percentage of employees who may participate in an executive retirement plan and still constitute a select group. Several industry experts suggested that DOL could help to address this issue in the future by providing a safe harbor that describes limits or thresholds companies could follow to establish eligibility. Two industry experts identified a range of possible information DOL could provide, such as a ceiling on the percentage of the company’s workforce permitted to participate, job titles that could be eligible for participation, or a compensation threshold. Industry experts also suggested more detailed information on factors to consider for eligibility, rather than a “one-size-fits-all” design, would help to ensure the information would be flexible enough for a variety of companies to apply. We asked DOL officials about issuing clarifying information on the statutory requirements under ERISA for eligibility into these plans. DOL officials stated that the agency has the authority to do so but has no plans to issue guidance because it has not encountered eligibility problems during plan audits and enforcement actions. Rather, DOL officials said that in light of resource constraints, other high priority guidance projects, and the absence of systematic abuses involving these plans, it does not believe it advisable to shift resources from other projects to undertake a guidance project in this area. DOL officials said the agency no longer renders decisions on the status of “select group” eligibility for executive retirement plans in advisory opinions or in response to external inquiries because such determinations involve factual questions that are not well suited to an advisory opinion or informal participant assistance process. Federal standards for internal control require federal agencies to communicate quality information externally through reporting lines so that external parties can help the entity achieve its objectives and address related risks. By exploring ways it may be able to help reduce the incidence of ineligible employees participating in executive retirement plans, DOL could help ensure ineligible rank-and-file employees are not participating in these plans and are receiving the applicable protections under ERISA. One such way may be by providing information to companies on factors to consider when determining a “select group” to aid companies in establishing plan eligibility. A related issue that companies can face is dealing with eligibility decisions that turn out to be in error. DOL officials told us they have not issued any guidance on how companies are to correct eligibility errors found in executive retirement plans. Officials referred us to a 2015 amicus brief DOL filed in a particular case that described the department’s views on how companies might consider addressing eligibility errors. The amicus brief suggests that the company could modify the plan to exclude the ineligible rank-and-file employees and award them the full vesting and other protections under ERISA while maintaining the plan’s status under ERISA as an executive retirement plan for those executives who do qualify. However, the amicus brief states that DOL took no position on the form of equitable relief appropriate under ERISA to redress an employer’s violation of vesting requirements by including rank-and-file employees in an executive retirement plan. The amicus brief also suggests that this approach would avoid providing a windfall gain to executives who properly could have been included in such a plan, because they possess sufficient bargaining power to protect their rights, and are not the intended beneficiaries of the substantive provisions under Parts 2, 3, and 4 of Title I of ERISA. When asked about this remedy, DOL officials said that funds from the executive retirement plan could be distributed to a qualified retirement plan for rank-and-file employees, with their benefits immediately fully vested and receiving ERISA protections. When we discussed the possible remedy described in the amicus brief with IRS officials, they said that while 409A regulations were being drafted, they were aware that applying strict distribution rules could have adverse tax consequences for rank-and-file employees participating in executive retirement plans. IRS officials said that removing these employees from these plans and awarding them full vesting of their benefits under Title I of ERISA could violate section 409A, raising concerns that the possible remedy noted in DOL’s amicus brief may be inadequate for companies seeking a method to correct plan errors. Officials also said that there are certain exceptions under section 409A when accelerated payments may be permitted; however, IRS officials said there is no current exception permitting an accelerated payment to be made to a rank-and-file employee in order to correct a violation of Title I of ERISA. IRS officials said they are willing to work with DOL to promulgate new section 409A regulations to create an exception to the accelerated payment rule for plans that seek to remove ineligible rank-and-file employees from the plan and make distributions to an employee’s qualified retirement plan in order to maintain the plan’s ERISA exemption. However, IRS officials said that prescribing corrective action in these situations is under DOL’s purview and that DOL first would need to further delineate the meaning of an executive retirement plan employee and then decide the proper approach for removing ineligible rank-and-file employees from a plan before any new regulations under section 409A could be considered. As mentioned above, federal standards for internal control require federal agencies to externally communicate necessary quality information to achieve their objectives. Without additional information from DOL on what companies can do to reduce the incidence of ineligible rank-and-file employees participating in these plans, some ineligible employees may continue to participate in some instances, potentially subjecting them to unexpected tax consequences such as if they are removed from the plan and the payment of their deferred compensation is accelerated. Further, without knowing how to properly remove ineligible rank-and-file employees when they are found participating in executive retirement plans, companies may be uncertain on how to re-establish an executive retirement plan’s exemption from the substantive provisions of Title I of ERISA for otherwise eligible participants. Although executive retirement plans are an important retirement savings vehicle for corporate executives and other highly compensated employees, little is known about certain key aspects of these arrangements. While some federal regulatory data exist on plans provided to the top five executives of publicly owned companies, information about the design, participation, and benefits provided under plans offered by privately owned companies or offered to employees beyond top five executives are largely unknown, as is their net revenue effect on the federal government. In addition, IRS has not taken steps nor collected adequate information to know if companies under audit with a qualified single-employer defined benefit plan are setting aside assets for the purpose of paying benefits deferred under executive retirement plans while the companies are in at- risk status—a practice the law intended to discourage. Through effective oversight, IRS can help ensure that it is collecting the appropriate amount of income taxes as a result of this potential practice. Another important consideration with respect to executive retirement plans is their potential to permit ineligible rank-and-file employees to participate in the plan, thereby leaving such employees without the protections of ERISA. Little information is available at the federal level about who is included in executive retirement plans because companies provide minimal information to DOL only once when they implement such a plan. By revisiting its reporting requirements, DOL can help ensure that only executives who can bear the risks inherent in these plans are permitted to participate. DOL has other opportunities to diminish this risk by providing assistance to companies, such as additional information describing plan eligibility, which could help companies reduce the incidence of rank-and-file employees participating in these plans. In addition, DOL can provide direction that companies can follow to remove rank-and-file employees found participating in these plans to ensure their benefits are protected and coordinate with the IRS so that these employees do not incur unexpected tax consequences that could result from erroneous inclusion in an executive retirement plan. We are making a total of four recommendations, including one to IRS and three to DOL. The IRS Commissioner should develop specific instructions within the Internal Revenue Manual, the Nonqualified Deferred Compensation Audit Techniques Guide, or other IRS training material to aid examiners in obtaining and evaluating information they can use to determine whether there exists a restricted period with respect to a company with a single- employer defined benefit plan and if a company with a single-employer defined benefit plan has, during a restricted period, set aside assets for the purpose of paying deferred compensation under an executive retirement plan. (Recommendation 1) The Secretary of Labor should review and determine whether its reporting requirements for executive retirement plans should be modified to provide additional information DOL could use to oversee whether these plans are meeting eligibility requirements. (Recommendation 2) The Secretary of Labor should explore actions the agency could take to help companies prevent the inclusion of rank-and-file employees in executive retirement plans and determine which, if any, actions should be implemented. (Recommendation 3) The Secretary of Labor should provide specific instructions for companies to follow to correct eligibility errors that occur when rank-and-file employees are found to be participating in executive retirement plans, and should coordinate with other federal agencies on these instructions, as appropriate. (Recommendation 4) We provided a draft of this report to DOL, IRS, PBGC, SEC, Treasury, and the United States Trustee Program within the Department of Justice for review and comment. DOL, IRS, PBGC, SEC, and Treasury provided technical comments, which we have incorporated where appropriate. IRS and DOL also provided formal comments, which are reproduced in appendices II and III, respectively. In response to our recommendation to develop specific instructions to aid IRS examiners in monitoring executive retirement plans for compliance with federal tax law, IRS stated that they would review and consider developing further specific instructions within the Internal Revenue Manual, the Nonqualified Deferred Compensation Audit Techniques Guide or other IRS training material to aid examiners. GAO continues to maintain that implementing this recommendation will help ensure that IRS is aware of when companies with at-risk single-employer defined benefit plans are reporting assets set aside to pay deferred compensation to key executives while in a restricted period as income for those employees. DOL stated that it does not have plans to issue guidance or regulations regarding executive retirement plans, citing, among other considerations, existing resource constraints and priority regulatory and guidance projects in development, and that it would not be advisable to shift resources from other projects. GAO continues to maintain that DOL’s one-time single page alternative reporting for executive retirement plans lacks important information sufficient to help the agency identify whether companies may be including ineligible employees in its plan and DOL’s current data on executive retirement plans has limited usefulness due to the age and limits of the original data submitted. DOL also stated that the agency has not encountered evidence of systematic abuses involving executive retirement plans or that ERISA’s claims procedure rules and judicial remedies are inadequate to protect participants’ benefit rights. As we report, industry surveys indicate that some companies may be extending employee eligibility to high percentages of their workforce who are lower- paid and lower-ranked employees who may not be considered a part of a select group. Industry experts also told us that plan eligibility requirements for executive retirement plans are not clearly defined and that companies are unclear on how to establish eligibility, and they identified court cases that contribute to the confusion regarding plan eligibility. Additionally, the remedy DOL suggested in an amicus brief for companies to follow to correct eligibility errors in these plans could have unintended consequences for participants because, according to IRS officials, it could result in violations of federal tax law and additional tax for participants. Without implementing our recommendations, DOL will continue to be unable to ensure that only executives who can bear the risks inherent in these plans are participating. We urge DOL to develop instructions to correct eligibility errors, in coordination with other federal agencies, as needed, in a way that does not adversely affect rank-and-file employees participating in these plans. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees; the Secretaries of the Departments of the Treasury, Labor, and Justice; the Commissioner of the Internal Revenue Service; the Chairman of the Securities and Exchange Commission; and the Director of the Pension Benefit Guaranty Corporation. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or jeszeckc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made contributions to this report are listed in appendix IV. This report examines (1) what is known about the prevalence, key advantages, and revenue effects of executive retirement plans; (2) the potential outcomes of executive retirement plan benefits in company bankruptcy; and (3) how federal agency oversight protects benefits and prevents ineligible participation in executive retirement plans. To address these objectives, we reviewed relevant federal laws, regulations, guidance, and other agency documents related to executive retirement plans. We reviewed relevant research on executive retirement plans, which we identified with the help of a GAO librarian, through stakeholder interviews, by reviewing sources cited in documents we obtained, and through limited internet searches driven by stakeholder and documentary evidence. This research included published research on the costs of executive retirement plans on the companies that offer them and the revenue effects on the federal government. We interviewed a non- generalizable sample of executive retirement plan experts representing different roles in the industry, including plan consultants, plan providers (including record keepers and insurers), attorneys, investment advisors, actuaries, proxy advisors, and researchers. We also interviewed an array of bankruptcy experts—including those with experience in executive compensation—to understand bankruptcy procedure and the treatment of executive retirement plans in company bankruptcy. We selected executive retirement plan and bankruptcy experts to interview based on a combination of published work, breadth and depth of experience, as well as peer referrals. We interviewed representatives from industry associations representing a diverse range of stakeholder groups, such as those that offer, provide services to, or conduct research on executive retirement plans. As part of this effort, we contacted the American Institute of Certified Public Accountants to discuss their perspective on the use of executive retirement plans but they declined to meet with us. We also interviewed agency officials from the Department of Labor’s (DOL) Employee Benefits Security Administration (EBSA), Department of the Treasury’s Office of Tax Policy, the Internal Revenue Service (IRS), the Securities and Exchange Commission, the Pension Benefit Guaranty Corporation (PBGC), and the United States Trustee Program within the Department of Justice. To understand the prevalence of executive retirement plans, we analyzed data provided by the Main Data Group (MDG), an executive compensation benchmarking and corporate governance analytics firm. MDG compiled the data provided from required SEC disclosures from filing years 2013 to 2017 (the most recent data available at the time of our analysis) for executive retirement plan benefits provided to top executives in Standard & Poor’s (S&P) 500 and Russell 3000 companies as reported in the annual 10-K, proxy statement, and other documents. Companies listed in the S&P 500 are generally also listed in the Russell 3,000. The SEC generally requires public companies to disclose executive compensation information—including executive retirement plan benefits— provided to the Chief Executive Officer, Chief Financial Officer, and the next three most highly compensated executive officers. These data are principally found in the annual proxy statement within the Summary Compensation Table, Pension Benefits Table, and Nonqualified Deferred Compensation Table. The data include executive retirement plan benefits offered as a defined benefit plan and defined contribution plan. For a given year, the total accumulated value of executive retirement plans structured as a defined benefit provided to top executives are based on the “present value of accumulated benefit” and “payments during the last fiscal year” as reported in the Pension Benefits Table. For defined contribution plans, the total accumulated values are based on the “aggregate balance at last fiscal year end” and the “aggregate withdrawals/distributions” for the reporting period as disclosed in the Nonqualified Deferred Compensation Table. To determine the average level of plan benefits for top executives, we summed the total accumulated plan benefits for all top executives in a given year and divided them by the total number of executives. For the median, we sorted the total accumulated plan benefits for all executives in a given year and determined the midpoint. To assess the reliability of the data provided, we interviewed MDG officials regarding their data collection processes. We also independently compared executive retirement plan data from a random sample of SEC filings obtained from Edgar (the SEC’s public database for required disclosures) with data for the same companies as reported by MDG. We found the data to be sufficiently reliable for the purpose of describing the prevalence of executive retirement plans among companies subject to SEC’s disclosure requirements. To understand the expected outcomes for executive retirement plan benefits during company bankruptcy, we analyzed data collected from our non-generalizable review of a random sample of companies that offered an executive retirement plan and filed for bankruptcy during the period from October 17, 2005—the effective date for most of the provisions of the Bankruptcy Abuse Prevention and Consumer Protection Act of 2005 (2005 Bankruptcy Act)—through November 30, 2017— the most recent at the time of our analysis. The 2005 Bankruptcy Act made significant changes to federal bankruptcy law, including provisions limiting executive compensation in corporate bankruptcy. Using the unique Employer Identification Number (EIN) the IRS assigns to companies, we matched corporate Chapter 7 and Chapter 11 bankruptcy cases with DOL’s database of executive retirement plans to obtain lists of companies that filed for bankruptcy and offered at least one executive retirement plan. We obtained lists of corporate bankruptcy filings from New Generation Research Inc.’s (NGR) online database. NGR is a provider of data on corporate bankruptcies and companies in financial distress. We obtained from DOL its comprehensive list of executive retirement plans as filed with the agency from July 1982 to August 2017. The NGR and DOL data are not exclusive to public or private companies. To assess the reliability of the NGR and DOL data, we corresponded with officials regarding their respective data collection processes and requirements. We found the data to be sufficiently reliable for our purposes. The results of our data matching produced 138 Chapter 7 cases and 594 Chapter 11 cases of companies that filed for bankruptcy and offered an executive retirement plan. We reviewed a random selection of 151 cases (30 Chapter 7 and 121 Chapter 11) from a total of 732 relevant bankruptcy cases. To review bankruptcy court cases, we developed a standardized protocol to review each identified case and data collection instrument to input the data. The protocol included step-by-step instructions for reviewers to follow, including prescribed court documents to review and data to be collected. We obtained feedback on our case review protocol and data collection instrument from two outside bankruptcy experts—an attorney with expertise in the tax aspects of corporate bankruptcies and a bankruptcy law professor and former attorney who previously served as a federal bankruptcy judge—and incorporated their technical feedback on the documents. We also worked with a GAO methodologist to pretest our case review protocols and data collection instruments on a review of a select sample cases from the matched list to ensure our review process could collect reliable data between different reviewers. To obtain bankruptcy case documents to review, we used court filings obtained from PACER exclusively and did not rely on other data sources. PACER is an electronic public access service provided by the Federal Judiciary that allows users to obtain case and docket information online from federal appellate, district, and bankruptcy courts. Case documents are available on PACER as they are filed or entered into the court’s case system. Based on our case review protocol, we reviewed (where available), the court docket, case summary, bankruptcy petition, first day motions, management affidavit, schedule of assets and liabilities, statement of financial affairs, court-approved disclosure statement, court- approved plan (of reorganization or liquidation), and settlement agreements, among other documents with information relevant to executive retirement plans and their expected resolution in bankruptcy. We reviewed cases based on documents available in PACER between April and May 2018. Our review of 151 cases (30 Chapter 7 and 121 Chapter 11) from the matched lists resulted in 38 Chapter 11 cases where we identified executive retirement plan benefits in existence at or around the time of company bankruptcy and were able to determine the expected resolution of those benefits for employees as a result of the bankruptcy proceeding. As part of our review, we excluded cases if: (1) we were unable to confirm the presence of an executive retirement plan through review of court documents, (2) the case did not have a court-approved disclosure statement with estimated recovery percentages for various creditor classes in the case docket, or (3) if the case was open (i.e., not terminated) and had a reorganization or liquidation plan confirmed on or after May 2016, about 2 years from the start of our review. For the foregoing reasons, we were unable to identify expected outcomes in any of the Chapter 7 cases reviewed. For Chapter 11 cases, we were unable to ascertain actual outcome information for any of the cases we reviewed, but based the expected outcome of the executive retirement plan benefits on estimates provided in the court-approved disclosure statement, bankruptcy plan (reorganization or liquidation), or settlement agreement, which may differ from actual recoveries. To determine the expected resolution of executive retirement plan benefits, we reviewed case filings for evidence of specific treatment provided to employees with these claims. To the extent we did not find evidence of specific treatment for executive retirement plan benefits, we relied on estimated recovery information for the class of general unsecured creditors. Because the nature of bankruptcy proceedings depends on the facts and circumstances of each individual cause, the results of our analysis are not generalizable but provide illustrative examples of the potential outcomes of such cases. We reviewed selected court cases related to employee eligibility in executive retirement plans as identified by DOL, industry experts, and other literature. We also reviewed executive retirement plan surveys produced by industry firms, including plan sponsor organizations, benefit consultancies, record keepers, and other plan providers. We also interviewed representatives from many of these organizations regarding the use of executive retirement plans and determined that their survey data generally accorded with these discussions. We found the data to be sufficiently reliable for our purposes. In addition to the contact named above, the following individuals made important contributions to this report: Tamara Cross (Assistant Director), David Lin (Analyst-in-Charge), Ted Burik, Dan Powers, and David Reed. Also contributing to this report were James Bennett, Joanna Berry, Colenn Berracasa, Sherwin Chapman, Nina Daoud, Sarah Gilliland, Laura Hoffrey, Angie Jacobs, Kirsten Lauber, Ted Leslie, Avani Locke, Sheila R. McCoy, James R. McTigue Jr., Jeffrey Miller, Ed Nannenhorn, Oliver Richard, Marylynn Sergent, Frank Todisco, Walter Vance, Kathleen Van Gelder, and Adam Wendel.", "summary": "Some types of employers offer executive retirement plans to help select employees save for retirement. There are no statutory limits on the amount of compensation that executives can defer or benefits they can receive under these plans. However, employees in these plans do not receive the full statutory protections afforded to most other private sector employer-sponsored retirement plans, such as those related to vesting and fiduciary responsibility, among other things. These plans can provide advantages but they also have disadvantages because plan benefits are subject to financial risk, such as in a company bankruptcy. GAO was asked to review these plans. This report examines, among other objectives, (1) the prevalence, key advantages, and revenue effects of executive retirement plans and (2) how federal oversight protects benefits and prevents ineligible participation. GAO analyzed industry-compiled Securities and Exchange Commission plan data for 2013 to 2017 (the most recent data available at the time of our analysis); reviewed relevant federal laws, regulations, and guidance; and interviewed officials from IRS and DOL, among others. Executive retirement plans allow select managers or highly compensated employees to save for retirement by deferring compensation and taxes. As of 2017, more than 400 of the large public companies in the Standard & Poor's 500 stock market index offered such plans to almost 2,300 of their top executives, totaling about $13 billion in accumulated benefit promises. Top executives at large public companies generally accumulated more plan benefits than top executives at the smaller public companies in the Russell 3000 stock market index. Advantages of these plans include their ability to help executives increase retirement savings and potentially reduce tax liability, but the plans come with risks as well. To receive tax deferral, federal law requires the deferred compensation to remain part of a company's assets and subject to creditor claims until executives receive distributions (see figure). Department of Treasury officials and industry experts said executive retirement plans can be tax-advantaged and may have revenue effects for the federal government; however, the revenue effects are currently unknown. The Internal Revenue Service (IRS) oversees executive retirement plans for compliance with federal tax laws. For example, IRS must ensure that key executives are taxed on deferred compensation in certain cases where that compensation has been set aside, such as when a company that sponsors a qualified defined benefit retirement plan is in bankruptcy. However, IRS audit instructions lack sufficient information on what data to collect or questions to ask to help its auditors know if companies are complying with this requirement. As a result, IRS cannot ensure that companies are reporting this compensation as part of key executives' income for taxation. The Department of Labor (DOL) oversees these plans to ensure that only eligible employees participate in them since these plans are excluded from most of the federal substantive protections that cover retirement plans for rank-and-file employees. DOL requires companies to report the number of participants in the plan; however, the one-time single page filing does not collect information on the job title or salary of executives or the percentage of the company's workforce participating in these plans. Such key information could allow DOL to better identify plans that may be including ineligible employees. Without reviewing its reporting requirements to ensure adequate useful information, DOL may continue to lack insight into the make-up of these plans and will lack assurance that only select managers and highly compensated employees are participating. GAO is making four recommendations, including that IRS improve its instructions for auditing companies that offer these plans, and that DOL consider modifying reporting by companies to better describe participants in these plans. IRS and DOL neither agreed nor disagreed with our recommendations.", "document_type": "gao"}
{"report": "Key requirements related to improper payments during the period of our audit were included in IPIA, OMB M-18-20, and OMB Circular A-136. Federal agencies were required to take various steps regarding improper payments under IPIA and as directed by OMB M-18-20. The steps include the following: 1. Review all programs and activities and identify those that may be susceptible to significant improper payments (commonly referred to as a risk assessment), 2. For those programs and activities that agency risk assessments, OMB, or statute identifies as being susceptible to significant improper payments, agencies should develop statistically valid improper payment estimates, as well as analyze the root causes of improper payments and develop corrective actions to reduce them, 3. Report on the results of addressing the foregoing requirements. According to OMB officials, agencies are responsible for maintaining the documentation to demonstrate that these steps, if applicable, were satisfied. Figure 1 illustrates these steps, as well as the major components of conducting an improper payment risk assessment. IPIA required that agencies conduct improper payment risk assessments for all federal programs and activities at least once every 3 years and identify any program or activity that may be susceptible to significant improper payments. OMB M-18-20 provides guidance for implementing the IPIA requirements and covers agencies’ responsibilities for improper payment risk assessments, estimation, and reporting. According to the OMB guidance, agencies must institute a systematic method of reviewing all programs and activities to identify those that may be susceptible to significant improper payments. This systematic method can be a quantitative evaluation based on a statistical sample or a qualitative method, such as a risk-assessment questionnaire. Regardless of which method of review is used, IPIA required agencies to consider seven risk factors during the risk assessment. (See table 1.) OMB is also required to designate a list of high-priority programs for greater levels of oversight and review. The threshold for high-priority program determinations for fiscal year 2018 reporting and subsequent years is $2 billion in estimated improper payments, regardless of the improper payments rate estimate. In addition, OMB may determine that a program is high-priority for reasons other than exceeding the $2 billion threshold. High-priority programs are subject to additional requirements, such as submitting information about semi-annual or quarterly actions taken to reduce improper payments that can be used as a tool for tracking progress. According to OMB M-18-20, another fundamental requirement that agencies must meet is to recover any federal dollars that are a monetary loss to the government, unless legislation specifically prevents such recovery. Specifically, the Improper Payments Elimination and Recovery Improvement Act of 2012 (IPERIA) requires any program that expends at least $1 million during the year to implement payment recapture audits, if cost-effective to the agency, in order to recover improper payments. The requirement to conduct payment recapture audits applies to all agencies regardless of whether they have a program susceptible to significant improper payments. A payment recapture audit is a review and analysis of an agency’s or program’s accounting and financial records, supporting documentation, and other pertinent information supporting its payments, that is specifically designed to identify overpayments. It is not an audit that is performed in accordance with government auditing standards. OMB M-18-20 also states that for high-priority programs the agency shall report any action it has taken or plans to take to recover improper payments and intends to take to prevent future improper payments. If an agency has determined that performing payment recapture audits for any applicable program or activity is not cost-effective, a justification for that determination must be reported. Further, OMB M-18-20 states that agencies should report a justification for that determination through AFRs, Performance Accountability Reports, or in the format required through data requests from OMB. DOE’s 15 field Chief Financial Officers, in cooperation with DOE contracting officers, are responsible for overseeing contractor and other activities in the field, and they assist the OCFO in implementing improper payment reporting requirements. The OCFO issues Annual Payment Integrity Requirements and Guidance that transmits DOE’s instructions for meeting improper payments reporting and recapture requirements prescribed by OMB M-18-20. This guidance includes instructions for completing an attached template for reporting risk assessments and improper payments and payment recapture information. Using this template, 48 payment reporting sites provide information that is the basis for DOE’s department-wide improper payment risk assessment and reporting. These payment reporting sites consist of four types of federal entities and two types of contractors. (See appendix I for more information about DOE’s payment reporting sites.) In addition to the completed template, sites are directed to submit a signed certification that attests to the accuracy of the improper payment information and risk assessment and, if applicable, a justification for why payment recapture audits were not conducted. The OCFO completes a quality assurance checklist for each site and consolidates and reports the data as one program in DOE’s annual AFR. DOE reports on its improper payments 1 year in arrears; meaning, for example, that DOE’s fiscal year 2019 AFR included information on its improper payments identified in fiscal year 2018. In addition to reporting payment recovery information, as required by OMB Circular A-136, DOE has optionally reported some information it collected about improper payments that it identified at the time of the AFR issuance each year for fiscal years 2015 through 2019. (See table 2.) Specifically, DOE reported the amount of improper payments that had been made and identified in the preceding year—not based on a statistically valid estimate of improper payments but, rather, on reported amounts of known improper payments from individual payment reporting sites. DOE also reported improper payment rates that it calculated based on these reported amounts. DOE was not required to report a statistically valid estimate of improper payments in its AFRs because it determined it was at low risk of susceptibility to significant improper payments. See appendix II for additional details about improper payments in the data that DOE collected from the sites. In previous years, DOE reported statistical estimates of its improper payments. Specifically, DOE’s Performance and Accountability Reports and AFRs from fiscal years 2004 and 2007 through 2011 indicated that DOE used statistical sampling to produce projected improper payment estimates for certain payment categories. During these years, DOE reported estimated improper payment rates of less than 1 percent in these categories. However, in 2012 the DOE OIG determined that the estimated improper payment rate presented in DOE’s fiscal year 2011 AFR was not based on a statistical method. According to OCFO officials, DOE discontinued the use of statistical sampling to produce estimates in fiscal year 2012 because it was not required to do so, due to DOE’s determination that it was at low risk for significant improper payments. Since 2012, the DOE OIG has found DOE to be compliant with requirements for improper payment reporting and risk assessments as part of its required review. Specifically, the DOE OIG reported each year that DOE met the requirements for publishing improper payment information in its AFRs and performed the required risk assessments. According to an OIG official, the OIG is not required to perform evaluative procedures to determine the adequacy and completeness of DOE’s risk assessment and reporting in its AFR, and they have not optionally performed these procedures. The DOE OIG and other federal agencies or external audit organizations conduct periodic incurred cost audits and assessments of DOE’s cost- reimbursement contracts. The purpose of incurred cost audits is to determine whether such incurred costs are reasonable; applicable to the contract; allowable under generally accepted accounting principles and cost accounting standards applicable in the circumstances; and not prohibited by the contract, statute, or regulation. If, as a result of these audits or assessments, improper payments are identified—such as reimbursements for costs determined to be unallowable under the contract—DOE will question these costs, indicating that there is a possibility the costs are improper. DOE may then negotiate or otherwise work with the contractor to resolve the questioned costs. Sometimes, this can result in DOE recovering funds. According to DOE’s fiscal year 2019 annual payment integrity requirements and guidance, for the purpose of improper payment reporting, a questioned cost is not deemed an improper payment until it has been determined by the contracting officer to be unallowable. In addition, investigations conducted by DOE OIG, the Department of Justice, and other federal agencies may identify potentially unallowable DOE payments. Upon their resolution, these investigations may find such DOE payments to have been improper. According to OIG officials, improper payments identified through OIG investigations may be recovered through civil or administrative processes, and some of the improper payments identified through OIG investigations may lead to government-run criminal investigations. OCFO officials told us that recovered amounts may differ from the monetary loss associated with the original payments because of fees or fines, among other reasons. IPERIA requires agencies to include all identified improper payments in their reported estimate, regardless of whether the improper payment has been or is being recovered. According to DOE’s fiscal year 2019 annual payment integrity requirements and guidance, if the terms of a settlement require repayment to DOE, then the settlement amount would be considered an unallowable cost. Furthermore, the 2019 guidance states that due to the timing of when a settlement is reached, it is not possible to report these costs as an improper payment in the current year of reporting. Beginning in fiscal year 2018, DOE reported information in its AFR on improper payments made in prior years that were identified for recapture in the current reporting year. For example, in its fiscal year 2018 AFR, DOE reported that $92.69 million in prior years’ improper payments had been identified for recapture in fiscal year 2017. Similarly, in its fiscal year 2019 AFR, DOE reported $14.18 million in prior-year payments identified for recapture in fiscal year 2018. DOE’s reporting sites generally identify prior years’ improper payments identified for recapture through audits and investigations, among other strategies. DOE did not provide information on the years in which the prior-year improper payments were made, and the prior year improper payments identified for recapture were reported separately but not included in DOE’s reported improper payment amount and rate in any of its AFRs. The improper payment amounts that DOE reported in its AFRs for fiscal years 2015 through 2019 may not be accurate or complete and are likely understated, for two key reasons. First, we found that some DOE payment reporting sites did not correctly identify, track, and report their improper payments. Second, DOE reported improper payment amounts and rates for the current year, but did not report that the amounts and associated rates do not include a substantial amount of improper payments that may be identified in the years following the year in which the payment took place. The information in DOE’s AFRs for fiscal years 2015 through 2019 may not be accurate or complete, in part because DOE does not ensure that payment reporting sites correctly identify, track, and report their improper payments to the OCFO. Our review of documentation and interviews with officials at the selected payment reporting sites found some instances in which payment reporting sites’ processes for identifying and tracking improper payments did not always result in accurate and complete financial reporting as required. Specifically, we identified the following errors in reporting improper payments to the OCFO at three of the 10 sites we selected for review: Officials at one site told us that they resolve a portion of their improper payments by adjusting future invoices to account for the error. Site officials told us that in such cases, they do not track or report the amounts to the OCFO as improper payments. While adjusting future invoices is an efficient way to recapture improper payments, not tracking such adjustments as improper payments results in understated improper payments reported to the OCFO. The total amount of the understatement of these improper payments is not known. In addition, the site may have overstated other improper payments. In particular, site officials told us that they were unsure whether some of the annual adjustments from its indirect cost reconciliation process were reported as improper payments, even though OCFO officials told us that such adjustments are routine and are not considered improper payments. This could have resulted in an overstatement of improper payments, but the amount overstated is unknown. Officials at another site told us that they mistakenly included almost $1 million in questioned costs in their fiscal year 2017 improper payment reporting to the OCFO. Because questioned costs are not considered improper until they are determined to be unallowable, this means that the site overstated its improper payments by almost $1 million for that year. Additionally, this site subtracted its underpayments from its overpayments for its fiscal year 2015 reporting, resulting in an understatement of improper payments. Improper payments, regardless of whether they are over- or underpayments, should be added together and not netted, as both amounts are considered improper. Officials at the site told us that these issues had been corrected as of fiscal year 2018. Officials at a third site told us that they do not closely track underpayments and cannot state with certainty that all underpayments are included in the site’s annual improper payments report. The site therefore may be understating its improper payments each year. DOE’s Financial Management Oversight order states that financial management processes must include procedures and methods for ensuring that financial managers provide accurate, relevant financial reporting to customers, such as Congress and OMB. Additionally, federal internal control standards state that management should implement control activities through policies, including documenting policies in the appropriate level of detail to allow management to effectively monitor the control activity. However, not all of the payment reporting sites have fully documented their procedures for correctly identifying, tracking, and reporting their improper payments or ensuring the quality of their data, in part because there was no requirement to do so. Specifically, officials from all 10 selected payment reporting sites we interviewed told us they have procedures for tracking their identified improper payments. However, three of the 10 selected sites had not documented their procedures and two sites had documented some of their procedures but not others, including two of the sites mentioned in the examples above. By requiring payment reporting sites to document their procedures for identifying, tracking, and reporting their improper payments to ensure the quality of their data, the OCFO could better ensure that each payment reporting site maintains consistent procedures and provides comparable information about that site’s improper payments over time. Furthermore, the OCFO cannot ensure that sites are correctly identifying, tracking, and reporting improper payments and ensuring the quality of their data because the OCFO does not have a process to monitor that sites have documented—and are implementing—procedures to do so. The OCFO has taken some steps to help ensure the quality of the improper payments data that the sites report to the OCFO. For example, OCFO officials said they confirm that sites provide accurate information by requiring sites to self-certify the accuracy and completeness of the data, but does not take steps to verify the certification. Further, four of the five contractor payment reporting sites we interviewed told us the DOE field sites that oversee the contractors review the contractors’ submissions before sending the information to the OCFO; OCFO officials told us that the field sites do not formally approve these submissions. Additionally, OCFO staff complete a quality assurance checklist for each site’s submission. The checklist contains a series of questions to determine whether a site has submitted the required documentation and whether certain elements of that documentation are complete. OCFO quality assurance reviews also include simple mathematical checks for internal consistency, such as ensuring that the amount for total identified improper payments is the same across multiple tables. These steps, however, are not sufficient to ensure that sites are correctly identifying, tracking, and reporting improper payments and ensuring the quality of their data. For example, the quality assurance checklist does not include any tests to verify the accuracy of the procedures sites used to generate that data to ensure the sites’ data are reliable. By developing a monitoring process to ensure that payment reporting sites have developed and implemented procedures for identifying, tracking, and reporting their improper payments to the OCFO and ensuring the quality of their data, the OCFO could better ensure that the information it reports about improper payments in its AFR is accurate and complete. The amount of current year improper payments DOE reports in each fiscal year, as well as the improper payment rate DOE calculates based on this amount and reports in its AFRs, is not accurate or complete because it does not disclose that there are additional improper payments that are (1) not identified or that DOE’s OCFO is not aware of until a later date, or (2) potential improper payments that may be identified at a later date. Additionally, DOE does not conduct payment recapture audits, which may identify additional improper payments that could be recovered. DOE identifies many of its improper payments after the end of the fiscal year in which the payments occur and does not identify some improper payments until several years after they occur. These improper payments are identified through processes such as post-payment reviews, audits and assessments, and investigations that do not conclude until after the end of the fiscal year in which DOE made the payments. As a result, there is a known lag in identifying certain improper payments. The current year improper payment amount and associated rate DOE reported in its AFR excludes any improper payments that are identified after the end of the fiscal year in which the payments occurred. For example, in its fiscal year 2018 AFR, DOE reported $32.86 million of current reporting year (fiscal year 2017) improper payments, with an associated improper payment rate of 0.09 percent. In its fiscal year 2018 AFR DOE also reported $92.69 million of improper payments made earlier than fiscal year 2017; however DOE did not disclose that the amount of improper payments originally reported for any prior fiscal year had subsequently increased as a result of improper payments identified after the end of the fiscal year. While it is not possible for DOE to report on the specific amount of improper payments it has not yet identified, DOE also did not disclose in its fiscal year 2018 AFR that it expected to complete audits and investigations in subsequent years that could increase the amount of improper payments reported for fiscal year 2017. See figure 2 for categories of improper payments and the extent to which they are included in DOE’s improper payment amount and rate. Specifically, the OCFO excludes some known improper payments from the annual amount and associated rate it reports in its AFRs for the following reasons: Post-payment reviews may not conclude in the same fiscal year the reviewed payments were made. We have previously found that DOE identifies some improper payments through post-payment reviews. For example, DOE has not required its contractor payment reporting sites—most of which are M&O contractors—to submit invoices before DOE makes payments; instead DOE uses a “payments cleared funding arrangement,” which authorizes the contractors to withdraw funds directly from federal accounts. OCFO officials told us that improper payments made by DOE to contractors without such an agreement would be reported by the responsible federal site, and improper payments made by M&O contractors would be reported by the M&O contractor. DOE policies and procedures do not require that DOE site officials monitor M&O contractor withdrawals to determine the appropriateness of their incurred costs. DOE officials do not review M&O contractor withdrawal of funds to determine the appropriateness of M&O contract costs, and thus can only identify improper payments associated with these contracts through post-payment reviews of contractor costs that may occur after the end of the fiscal year. However, such post-payment reviews, such as monthly or quarterly reviews of invoices, may not identify certain improper payments— including improper payments that occurred late in a given fiscal year—leading DOE to exclude them from their annual reported improper payment amount and associated rate. For example, according to a document describing improper payments that one selected payment site reported to the OCFO, the site identified about $103,000 in fiscal year 2016 improper payments associated with travel during that same fiscal year. Additional reviews of fiscal year 2016 travel payments conducted in fiscal year 2017 identified further improper payments for travel of more than $35,000. Because the contractor identified these additional travel payments as improper through quarterly reviews that did not conclude until after the end of fiscal year 2016, this increase of about 35 percent in the site’s known improper travel payments was not included in the OCFO’s reported improper payment amount or rate for that year. In our March 2017 report, we recommended that DOE help ensure that necessary data are available to employ data analytics—which can identify improper payments more quickly than post-payment reviews can, increasing the likelihood that DOE will include them in its reported amount and rate for each fiscal year—as a tool to perform contractor cost-surveillance activities. Specifically, we recommended that DOE require contractors to maintain sufficiently detailed transaction-level cost data that are reconcilable with amounts charged to the government, including (1) cost data that, at a minimum, represent a full data population; and (2) the details necessary to determine the nature of each cost transaction. DOE disagreed with the recommendation. According to DOE officials, DOE is now developing plans to begin to use data analytics in fiscal year 2021. We continue to believe it is important for DOE to employ data analytics as a cost surveillance tool so DOE can better identify improper payments to its contractors in a timely manner and look forward to reviewing DOE’s plans and actions to address our prior recommendation. Audit coverage of DOE payments is limited, and some audits are not completed until several years after the audited payments were made. As we also found in March 2017, DOE uses incurred cost audits and assessments to identify contractors’ improper payments. However, our review of DOE OIG and other external entities’ audits and assessments of incurred costs for DOE’s 24 largest contractors for this report shows that, historically, these audits are infrequent and may occur several years after the costs have been incurred. For example, our updated analysis shows that as of September 2019, only about $25 billion—or 23 percent—of the nearly $108 billion in costs incurred during fiscal years 2014 through 2018 by DOE’s 24 largest contractors had been audited or assessed (see table 3). Although there is no requirement for how often contractors should be audited, the Contract Disputes Act of 1978 imposes a 6-year statute of limitations for the government to seek recovery of unallowable costs that could be identified through audits. According to our review of DOE reporting and documentation, known DOE improper payments amounts for a given fiscal year can increase in later years as more costs are audited. For example, one payment reporting site reported to the OCFO nearly $164,000 in improper payments made and identified in fiscal year 2017, and OCFO included this amount in the improper payment rate it reported in its fiscal year 2018 AFR. According to site documentation, the same site also identified, as the result of an audit in fiscal year 2017, an improper payment of nearly $920,000 that had occurred in a prior year. This improper payment was substantially more than the total amount of improper payments that the site reported in the fiscal year 2018 AFR. However, because the improper payment occurred prior to fiscal year 2017, the OCFO did not include it in its current year improper payment amount or rate for any fiscal year. For fiscal year 2020, DOE’s OIG has planned several assessments of costs the contractors incurred in prior fiscal years. However, contractor costs the OIG plans to review in the fiscal year 2020 planned assessments were incurred as early as fiscal year 2015. Therefore, any improper payments identified through the planned assessments will not be included in DOE’s reported improper payment rate using the current reporting methods and will instead be included in an overall lump sum amount of prior year improper payments, which has no effect on DOE’s reported improper payment rate. DOE does not track questioned costs centrally, and such costs can take several years to resolve. Audits and assessments can identify questioned costs that require additional review before they are either allowed or deemed improper. In its Semiannual Report to Congress, DOE’s OIG reported nearly $700 million of unresolved, questioned costs identified through its own audits and investigations as of September 30, 2019. Our analysis of the DOE OIG’s reporting found that a substantial portion of questioned costs the OIG identified were ultimately determined to be allowable once they were resolved; however, our analysis also found that DOE has not consistently resolved questioned costs in a timely manner. For example, some of the questioned costs that the DOE’s OIG identified—such as potential state gross receipts tax overpayments of $15.1 million that a DOE payment site made in fiscal years 2010 and 2011—have remained unresolved for nearly a decade. Large amounts of unresolved costs reported by DOE’s OIG add uncertainty about the completeness of the OCFO’s improper payment reporting. Moreover, the nearly $700 million of unresolved questioned costs that the DOE OIG reported does not include questioned costs identified through external audits of non-M&O contractors, such as those conducted by the Defense Contract Audit Agency or nongovernmental entities. Questioned costs identified through these external audits can be substantial, like those the DOE OIG has reported. For example, a 2017 incurred cost audit of a DOE contractor’s fiscal year 2010 costs conducted by an external firm identified nearly $280 million in questioned and unresolved DOE payments to the contractor. In November 2019, DOE officials told us that these questioned payments were resolved when DOE reached a settlement agreement with the contractor. DOE disallowed $34 million of the questioned costs as part of the settlement agreement, according to DOE officials. DOE’s Financial Management Oversight order states that financial management processes must include procedures and methods for ensuring that financial managers provide accurate, relevant financial reporting to customers. DOE customers include Congress and OMB. Additionally, federal internal control standards state that management should implement control activities through policies, including documenting policies in the appropriate level of detail to allow management to effectively monitor the control activity. According to OCFO officials, DOE does not have a mechanism for tracking questioned costs identified through external audits. Instead, OCFO officials said they rely on payment reporting sites to track these costs to resolution. However, the office does not require payment reporting sites to document policies for such tracking. DOE officials from two selected sites told us that their sites do not have policies for tracking questioned costs identified through external audits, including questioned costs that may later be deemed improper. As a result, the OCFO may not be aware of all potentially improper payments identified through external audits or know the status of their resolution. Without a requirement for sites to have policies to track questioned costs to their resolution, the OCFO cannot ensure that payment reporting sites are tracking—and ultimately reporting—all improper payments, and thus cannot ensure that it is including all improper payments in the amount it reports as actual in its AFRs. Investigations that identify DOE improper payments may not conclude until years after the payments were made. Investigations by DOE’s OIG, the Department of Justice, and other federal agencies can also identify DOE improper payments. However, similar to improper payments identified through audits, these improper payments—which can be substantial—may not be identified until years after they occur due to the length of time it takes to investigate and resolve criminal, civil, or administrative cases. For example, in fiscal year 2018, DOE reported $60.6 million of improper payments identified through a fiscal year 2017 settlement with a contractor. DOE made some of these improper payments as early as 2001. Also, in fiscal year 2018, a DOE payment site reported that no improper payments were made or identified in fiscal year 2017, but the site reported a $4.6 million prior-year improper payment associated with a subcontractor’s false claims that were settled with the subcontractor in fiscal year 2017. The OCFO reported these two cases, along with other DOE improper payments identified through investigations, as lump sum prior-year improper payments identified for recapture in its fiscal year 2018 AFR. However, the OCFO did not include these known improper payments in the improper payment amounts used to calculate its improper payment rates for the years in which DOE incurred the disallowed costs. Furthermore, some DOE improper payments are not reported as current or prior-year improper payments because the investigations of the payments were resolved in a manner that prevented DOE from formally considering the payments improper. For example, in 2015, DOE’s OIG reported that a company received a loan guarantee of more than $500 million from DOE after it “provided the Department with statements, assertions, and certifications that were inaccurate and misleading, misrepresented known facts, and, in some instances, omitted information that was highly relevant to key decisions in the process to award and execute” the loan guarantee. The company later declared bankruptcy and did not repay the loan. However, because DOE did not determine this payment to be improper through a legal case or any other process, the $500 million of known monetary loss was not included in DOE’s improper payments reporting in its AFR for any fiscal year. Also, in fiscal year 2017, DOE excluded a six- figure settlement with an outside party from its improper payment reporting. OCFO officials told us that they excluded payments associated with this case from their office’s reporting due to certain aspects of the settlement agreement. DOE’s Information Quality Guidelines state that information disseminated to the public, such as information on improper payments reported in DOE’s AFRs, should be presented in an accurate, complete, unbiased, and clear manner and should be useful to the intended users of the information. As previously noted, agencies with programs that are susceptible to significant improper payments—defined to include improper payments exceeding $100 million in a year—are required to develop improper payment estimates and corrective action plans. However, the OCFO cannot determine whether improper payments in a given year exceeded the $100 million threshold because the OCFO does not track information on the year that payments were made for all known improper payments for a given fiscal year—including improper payments identified in later years through resolution of questioned costs or conclusions of audits or investigations. By tracking information on the year the payment occurred for all improper payments identified, to include those identified in later years, and determining and disclosing in its AFR whether improper payments in a given year exceeded the $100 million threshold, DOE could better inform Congress, OMB, and the public about whether it has made significant improper payments. Additionally, DOE sites perform some payment recapture activities, but does not conduct payment recapture audits, which could identify additional improper payments that could be reported, and potentially recovered. As previously discussed, IPIA required any program that expended at least $1 million annually to conduct payment recapture audits, if cost-effective to the agency, or to provide justification if such audits are determined not to be cost-effective. A payment recapture audit is a review and analysis of an agency’s or program’s accounting and financial records, supporting documentation, and other pertinent information supporting its payments, that is specifically designed to identify overpayments. As such, payment recapture audits are tools to identify improper payments, in addition to an avenue for recovering those overpayments. DOE included a justification for its decision not to conduct payment recapture audits in its AFRs for fiscal years 2015 through 2019. For example, in its fiscal year 2019 AFR, DOE cited its improper payment rate of 0.09 percent and recapture rate of 97 percent to support the department’s determination that it was not cost-effective to perform payment recapture audits. DOE also cited other activities it employed to identify and recapture improper payments, such as prepayment review and approval of invoices, post-payment reviews, contractor internal audits, results of cost allowability audits of integrated contractors, and results from travel audits, among others. The OCFO fiscal year 2018 payment integrity guidance included a list of seven criteria that sites were to use to determine whether payment recapture audits are cost-effective. For fiscal year 2018, 42 of DOE’s 48 payment reporting sites submitted a justification stating that it would not be cost-effective to employ payment recapture auditors. Our review of the 42 justifications found that the quality of the justifications varied by site. We found that 40 of the 42 justifications did not demonstrate consideration of any of the seven criteria in support of their determinations that payment recapture audits would not be cost- effective. One DOE field site’s justification included three bullet points, as shown in figure 3, none of which aligned with the criteria. The OCFO uses a quality assurance checklist to review payment sites’ improper payment reports that includes verifying that the site submitted a justification and that the justification is “adequate.” The checklist does not define “adequate,” and the OCFO approved all of the justifications submitted, even those that did not demonstrate consideration of any of the seven criteria from the payment integrity guidance. DOE’s Financial Management Oversight order states that financial management processes must include procedures and methods for ensuring that financial managers provide accurate, relevant financial reporting to customers. Furthermore, under OMB M-18-20, agencies are required to recover any federal dollars that are a monetary loss to the government, unless legislation specifically prevents such recovery. By clarifying guidance to define the factors for assessing the adequacy of the justifications, and reviewing sites’ justifications for not performing or arranging for payment recapture audits, DOE could better ensure that the justifications it reports have a sound basis and that DOE is not missing opportunities to identify and recover improper payments. Additionally, DOE may be missing opportunities to recover federal dollars that are a monetary loss to the government, as required under OMB M- 18-20, because it has not evaluated whether sites could identify additional improper payments through payment recapture audits. Our analysis of information provided by DOE shows that in fiscal year 2003 the department conducted payment recapture audits and that the improper payments identified through these audits far exceeded the costs of conducting the audits. According to OCFO officials, the information on payment recapture efforts was from a payment recapture audit at one site; it was not an OCFO recovery audit program. The OCFO officials reiterated that the majority of the payment reporting sites have not performed payment recapture audits because they believe existing efforts are effective in recovering identified improper payments. However, payment recapture audits are designed to identify additional improper payments not previously identified. By evaluating whether it could identify enough additional improper payments to make payment recapture audits cost-effective, such as performing audits at a limited number of sites, DOE would have an opportunity to identify and recover additional improper payments or have better information to justify that payment recapture audits are not cost-effective. In its fiscal year 2018 improper payment risk assessment, DOE assessed its risk of susceptibility to significant improper payments as low. However, DOE did not provide sufficient documentation to support how it conducted its risk assessment and made this low-risk determination. Consequently, we could not determine if the process DOE used to perform its improper payment risk assessment provided a reasonable and reliable basis for making its risk determination. DOE’s process to conduct its fiscal year 2018 risk assessment may not be adequate to support its low-risk determination of susceptibility to significant improper payments. DOE has a decentralized process for conducting its statutorily required improper payment risk assessment every 3 years. For fiscal year 2018, DOE developed and provided each payment reporting site with an improper payment risk assessment template to complete. DOE directed all of its payment reporting sites to consider the seven risk factors listed in IPIA, as well as four additional risk factors that DOE developed. Table 4 lists the additional DOE-developed risk factors that sites were to consider in their risk assessments. DOE’s improper payment risk assessment template included a variable scale for rating each of the risk factors. The OCFO provided guidance instructing payment reporting sites to, when populating the template, consider the site’s exposure to the risk factors and to rate them by applying a numerical score to each risk factor. Each payment site totaled its numerical scores to calculate the site’s overall level of susceptibility to significant improper payments. DOE then consolidated all of the payment site assessments into an overall department-wide risk assessment. However, DOE could not explain, and did not provide us documentation to support, its rationale for the variable scales used to score such risk factors in its fiscal year 2018 assessments—both in the 10 payment-site risk assessments we reviewed and in DOE’s department-wide risk assessment—and how the scores assigned for each risk factor affected DOE’s susceptibility to significant improper payments. As a result, we could not determine if DOE’s process for conducting its fiscal year 2018 improper payment risk assessment provided a reasonable basis for DOE’s overall risk determination. Furthermore, the OCFO weighted all of the payment reporting sites equally in terms of overall risk when it aggregated the risk ratings into an overall assessment of susceptibility to significant improper payments. However, DOE did not provide an explanation or documentation of why the sites were weighted equally in the overall department-wide improper payment risk assessment, even though the payment types and dollar amounts of outlays processed by the sites varied widely. For example, a payment site processing $3 million of outlays in fiscal year 2017 had the same weight in the aggregated assessment as a payment site processing $5.7 billion of outlays. Finally, the OCFO did not provide evidence that it considered the known lag in identifying certain improper payments as an inherent risk during its fiscal year 2018 department-wide improper payment risk assessment process. This inherent risk relates to certain limitations affecting DOE’s ability to determine the extent of improper payments until several years after they occur, such as those identified through incurred cost audits and investigations, as previously discussed. For example, in its fiscal year 2018 AFR, DOE reported that a total of $124.35 million in payments were identified for recapture during fiscal year 2017, including $31.66 million made in fiscal year 2017 and $92.69 million made in years prior to fiscal year 2017. However, DOE did not provide us documentation to support how it considered the $92.69 million in improper payments made during years prior to fiscal year 2017—which could represent an inherent risk to the department—when assessing its risk of susceptibility to significant improper payments. As discussed earlier, some of the $92.69 million of improper payments identified for recapture occurred in fiscal year 2016. Thus, the amount of fiscal year 2016 improper payments that DOE reported in its fiscal year 2017 AFR is understated. Federal internal control standards state that management should design control activities to achieve objectives and respond to risks, and should implement control activities through policies. To contribute to the effective design and implementation of such control activities, management should clearly document internal control and all transactions and other significant events in a manner that allows the documentation to be readily available for examination. Additionally, management should periodically review policies, procedures, and related control activities for continued relevance and effectiveness. Further, although OMB does not direct agencies to demonstrate how each risk factor contributes to the agency’s overall susceptibility of risk for significant improper payments, OMB M-18-20 states that if a qualitative method is used during an improper payment risk assessment, it must be designed to accurately determine whether the program is susceptible to significant improper payments. DOE may not have an adequate process to support its risk determination because it did not properly document how it developed and considered risk factors during its fiscal year 2018 risk assessment. Until DOE revises its department-level process for conducting improper payment risk assessments, it cannot ensure that the process produces a reliable assessment of whether it is susceptible to significant improper payments. Specifically, without documenting its rationale for the variable scale used to score risk factors and weighting of the payment reporting sites, and consideration of the known lag in identifying the extent of total improper payments each fiscal year to support the development of its department- level risk assessment, DOE cannot demonstrate that its process for determining its low risk of susceptibility to significant improper payments is reasonable. Addressing these issues may result in DOE determining that it is susceptible to significant improper payments, and therefore subject to additional requirements—such as developing a statistically valid estimate of improper payments and reporting on actions to reduce improper payments, including a description of the root causes, and developing corrective actions to reduce them, including program-specific improper payment reduction targets. We also found that DOE’s OCFO did not sufficiently review the reasonableness of the selected payment reporting sites’ improper payment risk assessments. When we reviewed the risk assessments of the 10 selected sites, we found a lack of consistency in how the sites applied DOE guidance, as well as inadequate documentation supporting how the sites considered improper payment risk factors. Specifically, we found that the OCFO review process did not identify instances in which these sites did not adequately support certain ratings or did not adhere to DOE instructions for completing the improper payment risk assessment template. Staff from the OCFO used a quality assurance checklist to review the sites’ fiscal year 2018 improper payment risk assessments. However, the extent to which the OCFO reviewed documentation supporting payment sites’ risk assessments is unclear. Although the reviewer guidance provided in the quality assurance checklist directs reviewers to ensure that the documentation supporting the payment site’s risk rating adequately supports the risk factor being evaluated, a payment site official told us that OCFO reviewers did not consistently request to view their supporting documentation. Eight out of 10 payment reporting sites we reviewed had documentation to support that they followed DOE’s guidance to consider the results of prior GAO and DOE OIG audit reports and OMB Circular A-123-related assessment results. However, we found that two sites did not have such documentation. One site rated itself as having no significant deficiencies despite audit reports that indicated some deficiencies and findings for that site. Another payment site did not discuss the OMB Circular A-123 assessment results in its improper payment risk assessments, despite OCFO guidance to include such results when conducting improper payment risk assessments. However, quality assurance checklists completed by OCFO staff for these two sites did not indicate that documentation supporting the sites’ consideration of these prior reports and assessments in their risk assessments was missing. Further, five of the 10 payment reporting sites we reviewed did not provide sufficient explanation or documentation supporting their ratings for several of the risk factors they considered in their improper payment risk assessment, despite instructions in DOE’s guidance to do so. For example, one site cited “discussions with team lead” as the primary source of support for its ratings assigned for several risk factors. However, the site did not have documentation to support the results of these discussions and how such discussions supported the ratings for each risk factor. Federal internal control standards state that management should design control activities to achieve objectives and respond to risks, and should implement control activities through policies. To contribute to the effective design and implementation of these control activities, management should clearly document internal controls and other significant events in a manner that allows the documentation to be readily available for examination. We also found that OCFO staff did not document any potential changes to the payment sites’ risk ratings in the 10 quality assurance checklists we reviewed. However, the process to be followed in the event OCFO reviewers find that payment site risk ratings are not reasonable is unclear because DOE has not defined and documented in its policies and procedures the process for OCFO reviewers to override these risk ratings. DOE’s Financial Management Oversight order directs business units to evaluate and assess the effectiveness of their financial management oversight activities and other internal controls, such as the OCFO’s oversight of the payment reporting sites’ risk assessments. Further, the order charges the OCFO with reviewing and analyzing activities throughout DOE to evaluate the adequacy of established policies, procedures, and standards governing accounting and related reporting functions; evaluating the performance of internal controls over those functions; and recommending corrective actions as needed. In addition, according to federal internal control standards, management should also establish and operate monitoring activities to monitor the internal control system and evaluate the results. Such monitoring includes regular management and supervisory activities, comparisons, reconciliations, and other routine actions. Additionally, management should periodically review policies, procedures, and related control activities for continued relevance and effectiveness. Further, federal internal control standards state that management should use quality information to achieve the entity’s objectives. By developing, documenting, and implementing policies and procedures to require OCFO to review documentation supporting payment site risk assessments and define the process for overriding their risk determinations, DOE would enhance its ability to adequately monitor its decentralized improper payment risk assessment process and help ensure the accuracy and reliability of payment reporting sites’ risk assessments and DOE’s assessment of overall risk of susceptibility to improper payments. DOE’s OCFO relies on its 48 payment reporting sites to provide information about improper payments that DOE reports in its AFR; however, we identified several reasons that the information in DOE’s AFRs for fiscal years 2015 through 2019 may not be accurate or complete. First, DOE’s improper payments information may not be accurate or complete because the OCFO does not require the payment reporting sites to document their procedures for correctly identifying, tracking, and reporting their improper payments. By doing so, the OCFO could better ensure that the payment reporting sites provide consistent and comparable information about their improper payments over time. Second, the OCFO cannot ensure that sites are correctly identifying, tracking, and reporting improper payments to the OCFO and ensuring the quality of their data because OCFO does not have a process to monitor that sites have—and are implementing—procedures to do so. By developing such a monitoring process, the OCFO could better ensure that the information it reports about improper payments in DOE’s AFRs is accurate and complete. Third, DOE may not be reporting additional improper payments in the form of unallowable costs claimed by some contractors because, as we have previously found, DOE policies and procedures do not require that DOE sites monitor M&O contractor withdrawals to determine the appropriateness of costs incurred by the contractor. Under this arrangement, DOE does not use prepayment reviews to determine the appropriateness of M&O contract costs and, thus, can only identify improper payments associated with these contracts through post-payment reviews that typically occur after the end of the fiscal year. We previously recommended that DOE ensure data are available to employ data analytics—which can identify improper payments more quickly than post-payment reviews can—but DOE has not fully implemented the recommendation. We continue to believe it is important for DOE to employ data analytics as a cost surveillance tool so DOE can better identify improper payments to its contractors in a timely manner. DOE only includes improper payments that occur and are identified in the same fiscal year in its reported improper payment amount and rate in the AFR. However, DOE does not identify a substantial amount of improper payments in the same fiscal year due to the known lag in identifying such payments. Audits and assessments of DOE’s contractors can identify questioned costs that require additional review before they are either allowed or deemed improper, but DOE has not consistently resolved questioned costs in a timely manner because the OCFO does not direct payment reporting sites to document policies for tracking questioned costs to resolution. Without a requirement for sites to have policies to track questioned costs to their resolution, the OCFO cannot ensure that payment reporting sites are tracking—and ultimately reporting—all improper payments, and thus cannot ensure the accuracy and completeness of improper payments reported in DOE’s AFRs. Additionally, the OCFO cannot determine whether improper payments in a given year exceeded the $100 million threshold because the OCFO does not track information about the year that payments were made for all known improper payments for a given fiscal year. By tracking and disclosing information about all improper payments identified and the year in which these payments were made in its AFR, DOE would have better information to provide to Congress, OMB, and the public about whether it has made significant improper payments. Although DOE’s sites submitted individual justifications for not completing payment recapture audits, the quality of the justifications varied and did not meet DOE requirements. By clarifying guidance to define the factors for assessing the adequacy of the justifications, and reviewing sites’ justifications for not performing or arranging for payment recapture audits to ensure that the justifications meet requirements and are supported by appropriate analysis that considers the costs and benefits of performing the audits, DOE can better ensure that the justifications it reports have a sound basis and that DOE is taking advantage of all opportunities to both identify and recover improper payments, which in turn will help reduce the monetary loss to the government. Further, DOE may be missing opportunities to recover federal dollars that are a monetary loss to the government because it has not evaluated whether sites could identify additional improper payments through payment recapture audits. DOE has concluded that based on its self-assessed low improper payment rate and recapture rate, it is not cost effective to perform payment recapture audits. By evaluating whether it could identify enough additional improper payments to make payment recapture audits cost-effective, such as performing audits at a limited number of sites, DOE would have an opportunity to identify and recover additional improper payments or have better information to justify that payment recapture audits are not cost- effective. Finally, DOE may not have an adequate process to support its risk determination because it did not properly document how it developed and considered risk factors during its fiscal year 2018 risk assessment. Until DOE revises its department-level process for conducting improper payment risk assessments, it cannot ensure that the process produces a reliable assessment of whether it is susceptible to significant improper payments. Further, the process for the OCFO to oversee the accuracy of payment site risk ratings is unclear because DOE has not defined and documented, in its policies and procedures, the process for OCFO reviewers to override a payment site’s risk ratings in the event the reviewer finds that the rating was not reasonable. By developing, documenting, and implementing department-wide policies and procedures, DOE would enhance its ability to adequately monitor its decentralized improper payment risk assessment process and help ensure that individual payment reporting sites accurately score their risk factors—leading DOE to obtain a more accurate and reliable assessment of its overall risk of susceptibility to improper payments. We are making the following nine recommendations to DOE: The Office of the Chief Financial Officer should require payment reporting sites to document their procedures for identifying, tracking, and reporting improper payments to ensure they provide consistent and comparable information about their improper payments over time. (Recommendation 1) The Office of the Chief Financial Officer should develop a monitoring process to ensure that payment reporting sites document and implement procedures that will enable them to correctly identify and report improper payments to the OCFO. (Recommendation 2) The Office of the Chief Financial Officer should require payment reporting sites to document policies for tracking questioned costs to resolution. (Recommendation 3) The Office of the Chief Financial Officer should track information on the year the payment occurred for all improper payments, regardless of when they are identified, and determine and disclose in DOE’s AFR whether the department’s total annual improper payments exceeded $100 million in any given year. (Recommendation 4) The Office of the Chief Financial Officer should clarify guidance to (1) define the factors for assessing adequacy of payment reporting sites’ justifications that conducting recapture audits would not be cost-effective, and (2) require that the Office of the Chief Financial Officer review the sufficiency of these justifications against the criteria defined. (Recommendation 5) The Office of the Chief Financial Officer should evaluate whether payment reporting sites could identify enough additional improper payments through payment recapture audits to make those audits cost- effective, such as by performing audits at selected sites. (Recommendation 6) The Office of the Chief Financial Officer should revise DOE’s department- level process for conducting improper payment risk assessments to include (1) developing and documenting the rationale for the variable scale used to score risk factors and weighting of the payment reporting sites; and (2) documenting DOE’s consideration of the inherent risk associated with the lag in identifying certain improper payments subsequent to the fiscal year they occurred to ensure that the process results in a reliable assessment of whether the department is susceptible to significant improper payments. (Recommendation 7) The Office of the Chief Financial Officer should revise DOE’s department- level policies and procedures for reviewing risk assessments submitted by payment reporting sites to require a review and approval of the documentation supporting these assessments to help ensure the accuracy of the sites’ assessments. (Recommendation 8) The Office of the Chief Financial Officer should revise DOE’s department- level policies and procedures for conducting improper payment risk assessments to define the process for overriding a payment reporting site’s risk determination, when appropriate. (Recommendation 9) We provided a draft of this report to DOE for review and comment. DOE concurred with six of our recommendations and said that it plans to complete actions from November 2020 through December 2021 to address these recommendations. DOE did not concur with three of our recommendations; however, we believe that these recommendations remain valid. DOE’s written response is reproduced in appendix III and summarized below. In addition, DOE provided technical comments, which we incorporated as appropriate. DOE did not concur with our sixth recommendation to evaluate whether its payment reporting sites could identify enough additional improper payments through payment recapture audits to make those audits cost- effective, such as by performing audits at selected sites. In response to this recommendation, DOE stated in its comments that it has an ongoing Fraud Risk Management Working Group and that officials have developed a Fraud Risk Management and Data Analytics Implementation Plan to strengthen DOE’s capability to prevent, identify, and recover improper payments and fraud. However, DOE’s plan is still in draft form, and according to DOE’s technical comments, they will not begin using data analytics until fiscal year 2021. In addition, DOE stated in its comments that existing payment recapture activities such as pre- and post-payment reviews, contractor internal audits, use of the results of cost allowability audits of integrated contractors, and interim and close-out reviews of contracts and financial assistance awards are sufficient. As we discuss in the report, DOE determined that it does not need to conduct payment recapture audits based on justifications submitted by the reporting sites. However, most of the sites’ justifications did not include consideration of the OCFO criteria for making determinations about the cost-effectiveness of conducting payment recapture audits. We continue to believe that by evaluating whether it could identify enough additional improper payments to make payment recapture audits cost-effective, such as by performing audits at a limited number of sites, DOE would have an opportunity to identify and recover additional improper payments or have better information to justify that payment recapture audits are not cost-effective. DOE did not concur with our seventh recommendation to (1) develop and document the rationale for weighting risk factors, including the weighting of all payment reporting sites; and (2) document its consideration of the inherent risk associated with the lag in identifying certain improper payments subsequent to the fiscal year they occurred to ensure that the process results in a reliable assessment of whether the agency is susceptible to significant improper payments. Regarding the weighting of risk factors, DOE said that its risk assessment evaluates the volume and dollar amount of payments by payment category, payments subject to manual controls, and fluctuations in volume and dollar amounts. We recognize that DOE’s risk assessment template asks each site to assess its risk with regard to payment amounts and fluctuations. However, we are recommending that the OCFO document the weighting of all its risk factors, including its decision to consider as equal the risks identified by all sites—regardless of the dollar amount of outlays. While assessing the risk of improper payments at an individual site is important, it does not address the intent of our recommendation. We continue to believe that, because DOE did not properly document how it developed and considered risk factors during its fiscal year 2018 risk assessment, it cannot ensure that the process produces a reliable assessment of whether DOE is susceptible to significant improper payments. Regarding the consideration of inherent risk, DOE stated in its comments that the Payment Integrity Risk Assessment directs payment reporting sites to consider inherent risk as part of DOE’s Internal Control Program. We recognize that sites are to assess the inherent risk that an improper payment may occur. However, even if none of the sites identifies the known lag in identifying improper payments as a risk, based on our review of DOE’s AFRs, this lag is a risk to DOE as a whole. Therefore, we continue to believe that DOE should document in its risk assessment process its consideration of the known lag in identifying improper payments. Finally, DOE did not concur with our eighth recommendation to revise DOE’s department-level policies and procedures for reviewing risk assessments. Specifically, we recommended a policy revision to require OCFO review and approval of documentation submitted by payment reporting sites in support of their risk assessments to help ensure the accuracy of these sites' assessments. DOE stated in its comments that sufficient processes are in place for ensuring the accuracy of payment reporting sites’ risk assessments. DOE also stated that OCFO’s Payment Integrity Guidance instructs payment reporting sites to maintain detailed information supporting risk assessments, which is available to the OCFO and DOE’s auditors upon request, and that review and approval of the documentation occurs during periodic payment reporting site visits by OCFO staff. Further, DOE stated that as part of the OCFO’s quality assurance reviews, the OCFO evaluates the documentation used to support risk assessment ratings and directs updates to risk assessments if documentation listed does not support the stated risk ratings. As we discuss in the report, five of the 10 sites we reviewed did not provide sufficient explanation or documentation supporting their ratings for several of the risk factors. This includes one site that cited “discussions with team lead” as the primary source of support for the ratings it assigned for several risk factors. We continue to believe that by developing, documenting, and implementing policies and procedures to require the OCFO to review documentation supporting payment site risk assessments, DOE would enhance its ability to adequately monitor its decentralized improper payment risk assessment process and help ensure that individual payment reporting sites accurately score their risk factors, leading DOE to obtain a more accurate and reliable assessment of its overall risk of susceptibility to improper payments. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, the Administrator of the National Nuclear Security Administration, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact Allison Bawden at (202) 512-3841 or bawdena@gao.gov; or Beryl Davis at (202) 512-2623 or davisbh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. The Department of Energy (DOE) has 48 payment reporting sites that are responsible for conducting improper payment risk assessments and annually providing data on actual improper payments to DOE’s Office of the Chief Financial Officer (OCFO). The 48 sites consist of six types, four of which are types of federal entities and two of which are types of contractors. The four types of federal entities are Headquarters, DOE field sites, Power Marketing Administrations, and the Federal Energy Regulatory Commission. The two types of contractors are management and operating (M&O) contractor and non-M&O contractor. Table 5 lists the 48 payment reporting sites and provides the fiscal year 2017 outlays and improper payments data they reported to the OCFO for DOE’s fiscal year 2018 Agency Financial Report (AFR). The Department of Energy’s (DOE) Office of the Chief Financial Officer (OCFO) requires the payment reporting sites to provide some details about their improper payments that were not required to be included in the department’s Agency Financial Report (AFR) during the period under review. These details include information about how the improper payments were identified and the reasons why the payments were determined to be improper. As shown in figure 4, two methods accounted for most of the current year improper payments identified by DOE in fiscal year 2017: post-payment review (57.6 percent) and self-reporting (22.1 percent). As shown in figure 5, there was a broader range of reasons payments were determined to be improper in fiscal year 2017, although the majority (54.7 percent) were attributable to settlements as the result of litigation. In addition to the contacts named above, Hilary Benedict (Assistant Director), Michelle Philpott (Assistant Director), Kathy Pedalino (Analyst in Charge),Taya Tasse (Auditor in Charge), Perry Chen, Andy Furillo, Isabella Guyott, Latesha Love, Laura Pacheco, and Farrah Stone made key contributions to this report. Also contributing to this report were Kevin Bray, John Delicath, James Kernen, Jason Kirwan, Dan C. Royer, and Anne Thomas.", "summary": "Improper payments—payments that should not have been made or were made in an incorrect amount—are a significant problem in the federal government. Agencies are required to perform risk assessments to identify programs that are susceptible to significant improper payments. House Report 115-697 included a provision for GAO to review DOE's system for tracking improper payments. This report examines the extent to which (1) the amounts reported in DOE's AFRs for fiscal years 2015 through 2019 were accurate and complete, and (2) its fiscal year 2018 risk assessment provided a reasonable basis for its risk determination. GAO reviewed DOE's improper payment reporting for fiscal years 2015 through 2019 and its fiscal year 2018 risk assessment, and reviewed documents and interviewed officials from 10 of 48 reporting sites selected to provide a range of sites and about half of fiscal year 2018 reported improper payments. The improper payments amounts that the Department of Energy (DOE) reported in its annual agency financial reports (AFR) for fiscal years 2015 through 2019 may not be accurate or complete. Agencies with programs that are susceptible to significant improper payments—including those with more than $100 million of improper payments in a year—are required to report statistically valid estimates of their improper payments. DOE determined these requirements did not apply, but optionally reported information on actual improper payments it made and identified in the prior year. For example, in its fiscal year 2019 AFR, DOE reported fiscal year 2018 improper payments—such as those made to contractors for unallowable costs—totaling about $36 million, less than 0.1 percent of its outlays. However, DOE did not disclose that these amounts do not include improper payments identified through reviews, audits, and investigations completed several years after it issues its AFR (see figure). For example, as of September 2019, DOE had not audited $23.8 billion of its $38.5 billion in fiscal year 2018 outlays. Such audits may increase the improper payments in a year by millions of dollars. For example, based on a 2017 audit, DOE identified $34 million in fiscal year 2010 improper payments. DOE does not always track information on the year improper payments were made that would allow it to determine whether improper payments identified later would increase the total to more than $100 million. By tracking and disclosing such information, DOE could better inform Congress, the public, and others about whether it exceeded the $100 million threshold and should be subject to additional reporting requirements. DOE determined that its risk of significant improper payments was low in its fiscal year 2018 risk assessment. However, GAO found that the risk assessment may not provide a reasonable basis for DOE's determination. DOE did not provide sufficient documentation to support that it considered the known lag in identifying improper payments as an inherent risk, nor did it provide sufficient documentation to support its rationale for the scale it used to score risk factors or for weighting risk ratings of payment reporting sites. For example, a payment site processing $3 million of outlays had the same weight in the overall assessment as a payment site processing $5.7 billion of outlays. As a result, DOE cannot demonstrate that its low-risk determination is reasonable and that its risk assessment process produces reliable results. GAO is making nine recommendations to DOE, including to track and disclose information on improper payments identified later and determine whether these payments exceeded $100 million in any year, and to revise its risk assessment process to ensure the process has a reasonable basis and reliable results. DOE agreed with six of the recommendations, but did not agree with three recommendations, including to revise its risk assessment process. GAO maintains that the recommended actions are valid.", "document_type": "gao"}
{"report": "IRS began posting information on the internet in the 1990s. In the early 2000s, IRS launched its first two interactive services which allowed taxpayers to (1) check on the status of a refund, and (2) set up a payment plan to pay taxes they may owe over time. Since then, irs.gov has expanded to include other online services, such as personal informational accounts and tax transcript request services. In addition to online services, the irs.gov website contains information on various topics, including forms and publications offered on static web pages. While these static web pages do not provide taxpayers with personalized support or information, taxpayers seeking more targeted information may consult online calculators irs.gov offers (e.g., a tax withholding estimator). Two distinctions between these calculators and the online services our report examines are that taxpayers do not have to establish their identity before using a calculator and the calculators can be used to explore hypothetical tax planning situations. IRS reports that its website, which includes online services, static web pages, and calculators received more than 600 million visits in fiscal year 2018. With respect to taxpayer services, OLS is tasked with leading IRS’s business transformation efforts related to online services and improving the online experience for taxpayers. To improve online services for individual taxpayers, OLS primarily works with IRS’s relevant business operating divisions—Wage and Investment and Small Business/Self- Employed —which assist individual taxpayers in fulfilling their tax obligations. On the operations support side, Information Technology is responsible for delivering services and solutions related to technology and one of its responsibilities is to support IRS’s online services. IRS’s Research, Applied Analytics, and Statistics division conducts research related to taxpayer burden, which is defined as the time and money taxpayers spend complying with their tax obligations. Both Congress and presidential administrations have set the expectation that agencies provide high-quality customer service. Starting in the 1990s, they required agencies to develop plans for improving their services and regularly report on the progress they are making. In recent years, Congress and the executive branch have emphasized the importance of improving online services. In December 2018, Congress passed and the President signed the 21st Century Integrated Digital Experience Act (IDEA Act) that includes requirements for agencies when they are creating or redesigning a website or digital service that is intended to be used by the public. Among these requirements are to design the website or digital service around user needs, with data-driven analysis influencing management and development decisions, using qualitative and quantitative data to determine user goals, needs, and behaviors. The IDEA Act also requires agencies to ensure that any paper form related to serving the public is made available in a digital format by December 2020. In July 2019, the Taxpayer First Act became law. It includes a requirement that the Secretary of the Treasury (or designee) submit a comprehensive customer service strategy to Congress by July 2020 including, among other things, a plan to provide assistance to taxpayers that is designed to meet reasonable taxpayer expectations. This plan is to include online services. The act also requires this customer service strategy to identify metrics and benchmarks for quantitatively measuring progress in implementing it. Similarly, the administration has established a cross-agency priority goal called “improving customer experience with federal services” intended to improve the usability and reliability of the most important online services, which contains requirements related to IRS’s online services that will be discussed in more detail later in this report. In addition to these expectations for a high quality user experience, the GPRA Modernization Act of 2010 (GPRAMA) requires, among other provisions, strategic plans identifying Treasury’s and other cabinet departments’ and other executive agencies’ most important goals. It also requires annual performance plans that identify specific targets and reports to Congress and the public on results achieved. While GPRAMA is applicable to the department or agency level (e.g., Treasury), we have previously reported that these requirements should serve as leading practices at other organizational levels, such as component agencies, offices, programs, and projects, and are therefore applicable to IRS. IRS requires taxpayers whose income, filing status, and age fall within specified parameters to file a tax return. Taxpayers have five choices for filing a return: (1) hire a tax practitioner to file a return on their behalf with IRS, which the practitioner generally does electronically; (2) obtain tax preparation and filing services on the internet or download software, which allows for assisted preparation in addition to online filing; (3) file on paper for free; (4) use the Free File program; or (5) seek assistance from IRS’s Volunteer Income Tax Assistance or the Tax Counseling for the Elderly programs in which IRS provides funding to IRS-certified volunteers who meet in person with eligible taxpayers to help them prepare their return and the completed return is filed electronically. To encourage taxpayers to file electronically, IRS advertises that it will deliver refunds more quickly to those who file electronically than those who file on paper. In 2002, IRS signed a memorandum of understanding (which we will refer to as an agreement) with a consortium of tax preparation companies now known as Free File, Inc. Initially, the participating companies agreed to provide free electronic tax preparation and filing services for eligible taxpayers. In return, IRS stated that it would not offer its own free, online tax return preparation and filing services. This agreement has been periodically renewed, most recently in October 2018, when IRS and Free File, Inc. extended the terms of the agreement to October 2021. This requirement that IRS not offer its own online filing services has remained the same. The income limit for taxpayers to participate has evolved over time. In 2005, IRS and the consortium of tax preparation companies amended the agreement to provide for coverage for 70 percent of taxpayers based on the taxpayers’ adjusted gross income beginning in filing season 2006. They further agreed that while the percentage of taxpayers covered would remain the same throughout the agreement, the income limit for taxpayers would be adjusted each filing season. For the 2020 filing season—which IRS expects will begin in January 2020—taxpayers will be required to have an income below $69,000 and meet other eligibility requirements to use the participating companies’ preparation and filing services. For married taxpayers, the $69,000 threshold applies to their combined income if the two individuals file a joint return. Taxpayers whose income exceeds $69,000 are ineligible to use the participating companies’ software for free, but are allowed to use Free Fillable Forms (FFF), a service provided by one of the Free File, Inc. participating companies each year, with a link on irs.gov. FFFs are designed to be the online equivalent of paper forms, on which users can type information into a fillable field corresponding to each line item on the paper form. IRS.gov explains that to use FFFs, users “must know how to do your taxes yourself” and states that “only basic guidance” is provided. After a taxpayer completes the FFFs, the Free File, Inc. participating company that provides this service electronically transmits the return to IRS. IRS provides 10 online services for individual taxpayers (see table 1). We organized these online services into four categories based on common interactions between individual taxpayers and revenue agencies. IRS officials told us and we verified that all services are accessible from personal computers and nondesktop devices, such as smart phones and tablets. IRS also offers an app for mobile devices which taxpayers can use as a portal for accessing “Where’s My Refund” and for making payments. Usage across all online services in fiscal year 2018 was overwhelmingly concentrated within “Where’s My Refund?” (see figure 1). In 2018, taxpayers completed more than 300 million queries on “Where’s My Refund?”, IRS’s most used online service, attempting to determine when their anticipated refunds would arrive. To put this figure in perspective, IRS processed approximately 150 million individual tax returns and approximately 120 million individual income tax refunds in fiscal year 2018, meaning that some taxpayers are making multiple online inquiries about their refund. IRS.gov directs taxpayers to use this online service to follow up on their refund and suggests that taxpayers only call IRS in certain circumstances. “View Your Account Information” in fiscal year 2018 was IRS’s fifth most utilized online service, but it has experienced recent growth. Usage more than tripled between fiscal years 2017 and 2018. IRS has continued to add capabilities and make other improvements to “View Your Account Information” since it first launched in 2016, such as links to IRS’s online payment and transcript services. Further, officials told us in September 2019 that they plan to add information about the status of an installment agreement and additional payment history. As of October 2019, the capabilities of “View Your Account Information” focus on providing information to taxpayers about how much money they may owe IRS and payments made. For example, taxpayers who paid their full tax bill for prior tax years may log into “View Your Account Information” and see they have a $0 balance. Similarly, taxpayers who receive only refunds will log on and see a $0 balance, but not information about their refund. Officials noted that even a $0 balance may be of value to taxpayers who want reassurance that they and IRS have a common understanding of their tax situation. We identified three revenue agencies in other countries that offer online services that IRS does not: the Australian Taxation Office (ATO), New Zealand’s Inland Revenue Department (IRD), and the United Kingdom’s (U.K.) Her Majesty’s Revenue and Customs (HMRC) (see figure 2). In each of the selected countries, we found that taxpayers are offered a single online account that integrates the many different online services each revenue agency offers. As noted above, “View Your Account Information” on irs.gov provides links to other online services and taxpayers do not have to log in again to use those services. However, the remaining online services are not connected to “View Your Account Information” and taxpayers must leave the “View Your Account Information” platform to access those services. IRS officials told us that over the long term they would like to integrate additional services into “View Your Account Information,” but they explained that they have prioritized the development of new online services over connecting existing services to “View Your Account Information.” Further, IRS officials told us they also want taxpayers to be able to check their refund without having to establish an account. A second difference is that taxpayers in the three countries can complete their filing obligations on the revenue agency’s website. We examined the extent to which contextual differences between the U.S. income tax system and the three countries’ tax systems could enable or inhibit offering electronic filing on the revenue agency’s website. We found that the U.S. income tax system and the three other countries’ tax systems have similar definitions of income, employers withhold income taxes employees owe, and subsidies for certain social goals are channeled through the tax system. All three selected countries offer taxpayers the ability to communicate electronically with agency employees via the revenue agencies’ websites. For example, Australian taxpayers who are working on preparing their return in their account can communicate through an electronic chat with ATO employees about questions they may have, such as regarding deductions and the capital gains tax. In New Zealand, taxpayers can upload documents requested by IRD to their accounts, whereas American taxpayers generally must mail these documents. IRS’s pilots of electronic communication capabilities between taxpayers and its employees will be discussed later in this report. In addition to the three countries reviewed, we also selected three states that have integrated more services into a single online taxpayer account than IRS has done. Additionally, all three states provide taxpayers with two-way secure electronic communication (see figure 3). For example, Alabama and California taxpayers can log into their respective accounts for a secure electronic chat. California and New York taxpayers can share documents. We found that two of the states—Alabama and California—offer taxpayers the capability to file their tax return on the revenue agencies’ websites. However, officials in both states told us that few taxpayers have used this option. Alabama officials believed this was because many taxpayers prefer to use the same method to file their state tax return as their federal tax return, and as we have previously discussed, IRS does not currently offer this service. New York officials said they previously offered this service, but decided to stop offering it because they did not believe the benefits were sufficient to justify continuing it. Our review of the revenue agency websites for the states that have income taxes (43 states and the District of Columbia) found that 21 of the state revenue agencies, including Alabama and California, allow taxpayers to file their state tax return on the revenue agencies’ website (see figure 4). While IRS regularly surveys taxpayers who visit the static pages of irs.gov, the surveys do not provide information on the extent to which all of IRS’s online services meet taxpayer needs. All three of the foreign revenue agencies we reviewed collected survey information that allowed them to more clearly explain the extent to which they believe their online services are meeting taxpayer needs. In addition, IRS has long-running research seeking to estimate the time and money taxpayers spend complying with their tax obligations, but the implications of expanding online services for taxpayer burden have not yet been assessed. IRS seeks feedback from a randomly selected sample of users of the static pages of its website and we reviewed an example of feedback IRS had collected between February 28, 2019, and March 31, 2019. This feedback mechanism is not designed to measure taxpayers’ experiences with individual online services or the extent to which services meet their needs. The invitation to participate appears on the static pages of irs.gov, not when a taxpayer is logged into an online service. However, this does not necessarily exclude taxpayers using online services from providing feedback. IRS officials explained that it does allow IRS to capture feedback across irs.gov static pages and related applications, even though it does not provide feedback on any single online service. A hypothetical example would be a taxpayer starts the process of applying for a student loan by logging into IRS’s “Data Retrieval Tool” and then after completing that task peruses IRS’s publications on tax benefits for higher education on the static pages of irs.gov and is then invited to participate. IRS officials stated that one intent of the survey is to measure a taxpayer’s entire experience, including instances when a taxpayer is visiting for multiple reasons. If a user goes to a static page they may be invited to participate, however if they use the online services without visiting a static page, they will not have the opportunity to provide feedback. IRS officials told us that this feedback may provide insight into taxpayers who report coming to irs.gov to do a task, such as obtaining tax records, but then do not successfully complete the task. IRS told us that this information could alert IRS officials to challenges taxpayers may face in locating online services, but officials agreed that this method does not provide specific feedback on individual online services. IRS officials told us that they would like to combine this survey with surveys focused on specific online services, but face resource constraints. In addition to the survey of users from the static web pages, IRS collects feedback from taxpayers who access “View Your Account Information.” For example, IRS selected a random sample of “View Your Account Information” users between January 2019 and March 2019 who successfully logged into their accounts. OLS officials explained that IRS updated this survey to ask four questions that Office of Management and Budget (OMB) guidance directs agencies to use in assessing their customers’ experiences with the agency’s “highest-impact customer journeys.” IRS asked users: 1. If the online tax account tool met their needs. 2. About their overall satisfaction with irs.gov. 3. Whether this experience increased their confidence in IRS. 4. Whether they could find what they needed easily and quickly. IRS’s summary of the results of the “View Your Account Information” taxpayer experience survey identifies potential limitations. IRS states that users who have experienced challenges logging into protected services, such as “View Your Account Information,” provide negative feedback on the survey administered to users of static pages. However, the “View Your Account Information” experience survey is not designed to capture such negative feedback because a taxpayer must log into his or her account to be selected to participate in this survey. IRS officials agreed that our analysis is accurate, but had a different view on the implications. In their view, the purpose of the “View Your Account Information” taxpayer experience survey is to assess taxpayers’ experiences using this particular service. They believe that challenges legitimate taxpayers may experience in logging into “View Your Account Information” have broader implications and affect taxpayers’ experiences using other online services. And as noted above, IRS officials noted that the survey of users of static pages captures negative feedback from users who have had difficulty accessing online services. However, successfully passing the security checks is the first step in the journey legitimate taxpayers must take to use “View Your Account Information.” The result is a potential knowledge gap in the extent to which “View Your Account Information” is providing taxpayers with a satisfactory experience and very little knowledge on the extent to which the other online services are meeting taxpayers’ needs. OMB has directed agencies to ask two additional questions to gauge user experience with agency services: (1) Did it take a reasonable amount of time; and (2) Does the customer believe he or she was treated fairly. IRS asks the first question of a sample of users of static pages, but neither question is currently asked of a sample of “View Your Account Information” users. IRS told us that the first of these questions was covered by their question about whether taxpayers could find what they needed to easily and quickly, although OMB guidance considers these as separate questions. IRS does not believe the second question is relevant because its online services are automated. The OMB guidance authorizes agencies to request exemptions and modifications to the requirements. We confirmed with OMB staff that IRS had discussed its approach with OMB and that staff concurred with it. While OMB staff said they recognize that variation presently exists across agencies in the required survey questions, they told us that they would like to continue working with agencies to bring greater consistency to surveys so that comparable data will be collected by fiscal year 2021. Both IRS officials and OMB staff told us that that IRS is participating in an interagency working group focused on consistent implementation of this guidance. In addition to surveying customers, OMB’s Circular A-11 section 280 establishes government-wide guiding principles for all executive branch agencies which contain the following requirement: “Agency annual performance plans should include indicators for outcomes related to customer experience.” Our review of IRS’s congressional budget justification and performance plan and report for fiscal year 2020 found no performance measures or indicators summarizing the only taxpayer experience information that IRS collects on one of its online services—the “View Your Account Information” survey discussed above. In regards to the “View Your Account Information” survey, IRS officials told us they are not allowed to publicly share the results because of the process they used to obtain approval to administer this survey pursuant to the Paperwork Reduction Act. The act contains requirements that agencies justify the necessity of collecting information from the public and publish notices informing the public of the planned information collection. In addition the Office of Information and Regulatory Affairs (OIRA) within OMB must review and approve planned information collections. For authorization to administer the “View Your Account Information” survey, IRS officials used approval the Department of the Interior had obtained from OIRA for multiple agencies to administer customer satisfaction surveys for government websites. However, our review of the notice the Department of the Interior published found that while the contractor administering the survey must obtain permission from the agency before releasing the information, there is no prohibition on the agency choosing to release the information. Further to facilitate government-wide comparisons of the customer experiences different agencies are providing, the General Services Administration published a notice in the Federal Register in September 2019 stating that Treasury and other agencies will be publishing relevant data on Performance.gov. This notice does not prohibit agencies from publishing this same data in other publications (e.g., the agency’s performance plan and report). Until it collects more specific feedback on the other online services, it will not be possible for IRS to summarize and report information about the taxpayer experience with online services and the extent to which those services are meeting taxpayer needs. Without information about how effectively IRS’s online services are meeting taxpayer needs, it is difficult for decision makers to appreciate the potential value of these services and help ensure IRS has the necessary resources to maintain and improve these services. The three foreign revenue agencies reviewed all report to their parliaments on the extent to which they believe their online services are meeting taxpayer needs and use that information to help target areas for improvement: Australia: The Australian Taxation Office’s (ATO) annual reports for 2015-2016, 2016-2017, and 2017-2018 tracked “community satisfaction with ATO performance,” which combined more specific measures tracking satisfaction with different service channels—online, telephone, and mail—and satisfaction levels among different groups of taxpayers, such as individuals and small businesses. ATO publishes the more detailed satisfaction levels on its website with the most recent report presenting 2018 results. In its 2017-2018 report, ATO reported that the information collected showed that declining satisfaction with online services was negatively affecting its overall performance and stated that the office plans to use this feedback to improve online services. In its 2018-2019 report, ATO introduced a new measure—”community confidence in the ATO” —which it says is based on surveys of clients who have recently interacted with ATO and surveys of the general community. New Zealand: The Inland Revenue Department’s (IRD) annual reports for 2018 and 2019 present detailed results of its customer satisfaction and perceptions survey, including the overall percentage of customers satisfied with online services, as well as how satisfied subgroups of individual taxpayers are with online services, such as those receiving a tax credit for working families. United Kingdom: Her Majesty’s Revenue and Customs’ (HMRC) annual report presents a quantitative measure of customer satisfaction with online services. HMRC has published a more detailed explanation of how the agency measures customer satisfaction with online services. IRS states that the development of new online services should reduce taxpayer burden—referring to the time and money taxpayers spend to comply with their tax obligations—and one of IRS’s strategic goals states that IRS will reduce taxpayer burden. To help IRS officials and policy makers measure the progress they are making in achieving their goal of reducing taxpayer burden, IRS’s Research, Applied Analytics, and Statistics (RAAS) office has periodically surveyed taxpayers since 1984 about the time and money they spend to complete their tax obligations and uses the responses along with information from those taxpayers’ tax returns to estimate the total compliance burden for individual taxpayers. A RAAS official told us that IRS has not conducted any burden research specifically related to online services. IRS’s Data Book for fiscal year 2018 describes the magnitude of the taxpayer assistance provided through online services. More than 300 million electronic transactions took place through online services for individual taxpayers. IRS is missing an opportunity because taxpayers are already making extensive use of IRS’s online services for such tasks as setting up payment plans and obtaining records. Online services will likely continue to assist taxpayers in fulfilling their tax obligations. Two of the case study countries’ revenue agencies—in Australia and New Zealand—have conducted research on taxpayer burden. For example, New Zealand’s IRD’s annual report for 2019 stated that it has made progress in making taxes easier and simpler for its customers. To track its progress, IRD added online services to its taxpayer burden research in 2016 and updated this study in 2018. IRD compared the 2016 and 2018 survey results to burden research conducted in 2013 prior to expanding online services. As a result of this research, IRD found that 20 percent of taxpayers who run small businesses and participated in the survey reported that expanded online services and an improved IRD website have overall reduced their compliance burden. IRD’s report notes that additional online services will be launched in April 2019 and a follow-up survey is planned for 2020 to compare the reported burden with the earlier surveys and, thereby, track the progress it is making. A series of long-term planning documents establishes priorities to guide IRS decision-making and identify new online services, but does not contain evidence that taxpayer input was used to help identify the highest priority services. In April 2019, IRS published the IRS Integrated Modernization Business Plan (modernization plan). One of the plan’s goals is to modernize the taxpayer experience. To do this, IRS proposes to develop new services including delivering taxpayer notices electronically, modernizing online installment agreements, and establishing omni-channel communication capabilities provided that IRS continues to receive the requested resources from Congress. IRS does not currently incorporate taxpayer input into its prioritization process because it prioritizes services primarily based on their potential to benefit IRS’s operations or because they can be developed quickly. An OMB memorandum directs agencies to understand what their customers want by engaging in research to understand their goals, needs, and behaviors before beginning to develop new services. Going forward, the IDEA Act requires that new digital services be “designed around user needs with data-driven analysis influencing management and development decisions”. This requirement took effect in June 2019. IRS documents describing how new services were prioritized show that IRS did not incorporate taxpayer research or input into the score it assigns to each proposed service. Instead, IRS officials estimated the potential taxpayer value of a new service. For example, supporting documentation for one proposed project from the modernization plan to allow taxpayers to receive notices electronically states that IRS expects taxpayers to receive less paper mail and have easier online access to recent or historical notices if the project is developed. IRS expects that electronic delivery of notices will increase the timeliness of its service, which would improve the taxpayer experience. After a new online service is selected and approved, IRS does obtain input from taxpayers during the development phase, for example, to improve usability of the service and fine-tune technical capabilities. OLS officials provided us with documentation of user experience research they conducted on how to improve specific design elements of existing online services. For example, IRS reworded a button within “View Your Account Information” to access the online payment agreement service from “Need more time to pay?” to “Go to payment plans” to improve clarity. As a result of the change, the rate of taxpayers accessing the online payment agreement service from their online account has doubled, according to data IRS provided. While IRS’s modernization plan outlines new online services it plans to develop, the modernization plan also states that it expects additional services to be added over time as technology advances and customer expectations evolve. The Taxpayer First Act requires IRS to expand an online service—currently offered to taxpayers in nine states and the District of Columbia—to provide taxpayers with Identity Protection Personal Identification Numbers and an online platform to prepare and file a Form 1099 to report independent contractor earnings or other miscellaneous income. In May 2019, IRS officials told us that they will continue to reprioritize new service development based on available resources. As IRS reprioritizes the new services it plans to develop, IRS runs the risk of developing online services which may be of lower priority to taxpayers or that taxpayers do not utilize if IRS does not include input from taxpayers on what new services IRS should prioritize. By contrast, the United Kingdom’s HMRC has conducted taxpayer research to understand user needs and taxpayer preferences. For example, HMRC conducted taxpayer research in 2016 to inform decisions about which services to include in development of the “Personal Tax Account” which as noted above provides taxpayers with integrated access to various online services. HMRC’s research included workshops and interviews with taxpayers who use online services as well as an online survey of 4,000 taxpayers. The online survey asked taxpayers to rank the top five services they would like HMRC to develop from a list of 14 potential services and asked taxpayers to explain their rationale. HMRC then assessed preferences among taxpayers and concluded that secure electronic messaging was one of the services most highly sought, according to the report. As a result, HMRC incorporated taxpayer input into its new service prioritization process and began developing services that it knew taxpayers desired and were more likely to use. IRS’s modernization plan states that IRS intends to measure the success of its efforts to improve taxpayers’ experience consistent with the administration’s government-wide goal to improve customer experience with federal services. OMB’s guidance to agencies on this topic states that they should measure customer perceptions of the ease, efficiency, and equity in the process of obtaining the service. The modernization plan also states that IRS will measure taxpayer burden hours, which would capture changes in the amount of time taxpayers spend doing their taxes as a result of the modernization of information services and the development of new online services. The GPRA Modernization Act (GPRAMA) requires agencies’ annual performance goals to be expressed in an objective, quantifiable, and measurable form and this principle is relevant to IRS’s modernization plan. In prior work identifying leading practices related to this requirement, we explained that expressing goals in a quantifiable form provides an objective way to assess the agency’s performance. The Taxpayer First Act, enacted in July 2019, similarly requires the Secretary of the Treasury (or designee) to identify metrics and benchmarks for IRS for quantitatively measuring progress in implementing a customer service strategy that the act requires IRS to develop. That strategy must be submitted to Congress within 1 year of enactment. While Treasury and IRS are not required under the act to submit the strategy containing the metrics and benchmarks for quantitatively measuring progress until July 2020, we found that IRS’s modernization plan is not well positioned to help Treasury and IRS implement this new requirement. IRS’s modernization plan states that IRS intends to measure progress towards its customer experience goal through promoting ease and simplicity in taxpayer interactions. To measure that, IRS stated that it plans to increase its “American Customer Satisfaction Index” score, although IRS did not set a numerical target for improvement. The survey for this index score is administered by researchers outside the government and focuses on taxpayers’ experiences filing their tax return, which is not a service offered on irs.gov, making it of little use in assessing taxpayer satisfaction with IRS’s online services. IRS’s modernization plan does set numerical targets for output measures such as the percentage of notices available in an electronic format for taxpayers, but these targets are not aligned with any of IRS’s taxpayer experience feedback mechanisms, including those that come from the feedback mechanism discussed above administered to users of IRS’s “View Your Account Information.” For example, IRS asks a sample of “View Your Account Information” users if the online service met his or her needs, but IRS’s modernization plan does not set a target or desired level of performance for this question or for any other survey question. Our finding that IRS lacks targets for improving taxpayer experience is consistent with our prior work. In April 2013 we reported that previous IRS planning efforts to expand online services had not set a clear target for improving taxpayer experience and we recommended that IRS establish a numerical or other measureable goal to improve taxpayer satisfaction and a time frame for achieving it. While IRS neither agreed nor disagreed with this recommendation, in 2016 IRS said it would consider the development of numerical or other measurable goals related to taxpayer experience. Our current review shows that IRS has not developed such measures. We continue to believe this recommendation is valid and that the issue will continue to grow in importance along with the use of IRS’s online services. While IRS believes that the planned online services will promote “ease and simplicity,” no target is set in the modernization plan for reductions in taxpayer burden hours. As noted above, IRS has not started to examine the implications of expanding online services on taxpayer burden. Without targets for reducing taxpayer burden, IRS cannot determine the success of new online services in helping drive progress towards this goal. All three of the foreign revenue agencies we reviewed set numerical targets for performance measures related to improving online services and used these goals to target areas for further improvement. New Zealand’s IRD stated in 2015 that its business transformation program should improve the percentage of customers “who find it easy to comply” to between 90 and 95 percent by 2023/2024 to assess progress in its goal to reduce taxpayer burden. IRD’s annual report for 2019 states that the business transformation program remains on track and is delivering benefits and this section also provides an update on the percentage of customers “who find it easy to comply.” While the annual report does not refer to the target for 2023/2024, the percentage reported in the 2019 annual report is lower than the target identified for 2023/2024. The 2019 report explains that taxpayers are “getting used to our new systems and processes” and describes additional improvements IRD is planning. In addition, the United Kingdom’s HMRC set a target for 80 percent of taxpayers to report satisfaction with online services for 2018. HMRC published performance towards its goal in its 2018-2019 annual report, finding that 80.4 percent of taxpayers reported satisfaction with online services. IRS’s modernization plan states that taxpayers will be able to sign up to receive notices electronically by fiscal year 2021 and to have text or video chats with IRS employees by fiscal year 2024. IRS currently sends taxpayers notices via mail for identity verification, balance due, or if IRS needs additional information about a tax return. IRS plans to allow taxpayers to access certain notices electronically via a taxpayer’s online account. In July 2019, IRS Information Technology (IT) officials told us that they have established a team to start developing the capability to make notices available to taxpayers electronically, which is IRS’s first step towards developing full-scale digital communication capabilities. IT officials told us in October 2019 that they plan to conduct customer testing to pilot the service before it launches and gather customer feedback after launching the service. While IRS has just begun development of full-scale digital communication capabilities, IRS has experience providing a subset of taxpayers secure messaging capabilities through two pilot programs that we reviewed. Specifically, under the coordination of OLS, IRS began testing digital communication services in December 2016 to allow for secure and personalized correspondence between taxpayers and IRS employees through three pilot programs, as described below: An active pilot within the Small Business/Self Employed (SB/SE) business unit is testing digital messaging for examinations, which have traditionally been done by mailing questions and documents back and forth between the examiner and taxpayer. SB/SE began this pilot in fiscal year 2017 for a subset of taxpayers selected for examination for returns related to itemized deductions, the child care deduction, and education tax credits. Interested taxpayers must successfully complete security checks to verify their identity and then can exchange messages electronically with IRS employees and share requested documents through a platform accessed through irs.gov. A Taxpayer Advocate Service (TAS) pilot that tested the ability for taxpayers to send documents in electronic form began in fiscal year 2017 and ended in fiscal year 2019. TAS designed the pilot for the purpose of helping two sets of taxpayers: (1) those who were facing the prospect of IRS seizing their property to pay a tax debt, and (2) those who were facing an audit of their claim of the Earned Income Tax Credit and had sought TAS’ assistance. IRS officials told us that an authenticated chat pilot to assist taxpayers in completing an Online Payment Agreement was introduced in June 2019. Results for this pilot were unavailable as of October 2019. We evaluated the SB/SE and TAS pilots against leading practices. We found that the two digital communication pilots mostly addressed leading practices our prior work identified for designing a well-developed and documented pilot program. These leading practices are to: (1) establish objectives; (2) develop an assessment plan; (3) assess scalability; (4) evaluate results; and (5) ensure stakeholder communication. These practices enhance the quality, credibility, and usefulness of evaluations and help ensure that time and resources are used effectively. Although we found both pilots to be generally aligned with the leading practices to develop an assessment plan, evaluate results, and ensure stakeholder communication, neither pilot fully established objectives or assessed scalability. These leading practices are also relevant for testing of future capabilities of the electronic messaging platform that IRS plans to develop. Our leading practices state that objectives for pilot evaluations should be well defined, appropriate, clear, and measurable. We found differences in stated objectives between OLS and the participating offices. OLS set a target for the SB/SE pilot to reduce total case time from greater than 200 days to fewer than 100 days, and for the TAS pilot to improve the relief rate to taxpayers by 5 percent, which OLS officials explained were ambitious goals. However, SB/SE officials told us that they believed the magnitude of OLS’s goal for reduction in case time to be unrealistic. The National Taxpayer Advocate told us that she had narrower objectives for the pilot including testing the viability of sending documents electronically and assessing taxpayer willingness to participate. Having officials from relevant offices with different understandings of the quantitative target they are trying to achieve is not fully consistent with the leading practice and makes it more difficult for officials to evaluate the performance of the pilots. We previously reported that key features of an assessment methodology include a strategy for comparing the pilot’s implementation and results with other efforts; a clear plan that details the type and source of the data necessary to evaluate the pilot; and methods for data collection, including the timing and frequency. Our review found that the implementing offices for both pilots developed plans to conduct periodic assessments to assess the objectives. For example, the assessment plan for SB/SE’s pilot included measurements of average case time and participation levels. The TAS assessment plan also included measurements of average case time and participation levels as well as the reasons taxpayers provided for not enrolling in the online pilot. The purpose of a pilot is generally to inform a decision on whether and how to implement a new approach in a broader context. Identifying criteria or standards for identifying lessons about the pilot will help inform an agency’s decisions about scalability and when to integrate pilot activities into overall efforts. A common challenge that both of IRS’s communication pilots experienced was that only a small proportion of eligible taxpayers participated. Among the taxpayers selected for an SB/SE exam for whom the pilot was offered, approximately 11 percent of taxpayers participated in the digital communication pilot and sent their exam responses and supporting documentation through a secure, electronic messaging platform while approximately 51 percent of taxpayers sent their exam responses to IRS via mail. In contrast, 1 percent of invited taxpayers participated in the TAS pilot, according to the TAS report, which found that most taxpayers opted to communicate with TAS through more traditional methods such as telephone, mail, or fax. Officials involved in each pilot reported that additional taxpayers expressed interest in participating, but experienced challenges in getting through the identity verification requirements for enrollment. Of the potential participants in the SB/SE pilot, only 44 percent of those who began the secure enrollment process successfully enrolled. Pilot participants told TAS that they found the secure enrollment system for the TAS pilot to be too complicated to use and preferred instead to fax documents to avoid the burdensome sign up process. The National Taxpayer Advocate concluded that the participation rate was so low that it did not make sense to continue the pilot. In conjunction with a clearly articulated assessment methodology, a detailed data-analysis plan identifies who will analyze the data as well as when and how data will be analyzed to assess the pilot’s performance and draw conclusions about how to improve procedures moving forward. SB/SE’s pilot report found that use of the Taxpayer Digital Communications (TDC) electronic platform did reduce the number of days to complete an examination compared to paper. However, IRS examiners spent more hours on average on exams conducted through TDC than paper exams because taxpayers sent more attachments in their electronic messages, on average, than in the mail. TAS found that those taxpayers who enrolled in the pilot were able to successfully communicate with them.TAS officials expressed optimism that enrollment in secure communication could help reduce case processing time among those seeking assistance avoiding an IRS seizure of their property to pay a tax debt. A leading practice is that agencies identify who the relevant stakeholders are and communicate frequently to obtain feedback on the successes and challenges of the pilot. IRS identified taxpayers and participating business units as the relevant stakeholders for each pilot. We found that both the SB/SE and TAS pilots obtained feedback from stakeholders including employees and taxpayers who participated in a pilot as well as taxpayers who chose not to participate. In January 2018, SB/SE developed a web-based survey which sends taxpayers a voluntary survey upon closing of an exam, when communication ceases with a taxpayer. In addition, SB/SE called taxpayers who did not participate in the pilot program to discuss why they chose not to participate. Of the 262 taxpayers who were successfully contacted, taxpayer reasons for not signing up included that they did not remember seeing the invitation to enroll, they could not pass the secure enrollment process, and they thought it was a scam. TAS held focus groups with employees participating in its pilot and found that many employees raised concerns that the digital communication platform was not user friendly and discouraged uptake. The Office of Appeals conducted a pilot between fiscal years 2017 and 2018 using video conferencing software as a way for Appeals Officers who volunteered to participate to conduct video conferences with taxpayers in lieu of a telephone conference. The Office of Appeals concluded the pilot demonstrated the viability of this technology and allowed all Appeals officers who are willing to use this technology to offer it to the taxpayers they are working with as of October 1, 2018. We did not assess the pilot against our leading practices for conducting pilots because we had recently completed a review of IRS’s Office of Appeals, including its video conferencing capabilities, and the Treasury Inspector General for Tax Administration (TIGTA) recently published a review of the pilot. TIGTA identified a risk related to the scalability of videoconferencing. As of September 30, 2018, Appeals’ officials told us that less than 4 percent of invited taxpayers chose to participate in its pilot. They said that some of the taxpayers who declined to use videoconferencing thought it easier to have a phone call rather than go through the steps involved in setting up a videoconference. Unlike the digital communication pilots described earlier, participants in the Appeals pilot were not required to verify their identity through Secure Access, which is a multifactor authentication process for which taxpayers provide personal and financial information and then IRS verifies that the taxpayer has a mobile phone in his or her name by texting a code to the phone or mailing an activation code. Instead, Appeals officers verified taxpayer identities at the beginning of the videoconference. Despite the difference in security requirements, the participation rate for the Appeals pilot was also low. IRS’s modernization plan’s discussion of future video chats between IRS employees and taxpayers makes no mention of the challenges Appeals has experienced in getting taxpayers to use the videoconferences it already offers selected taxpayers. The same concerns about clear objectives and scalability that we found in the TAS and SB/SE digital communication and videoconferencing pilots also have implications for the full-scale services that IRS plans to develop. The modernization plan states that one of IRS’s objectives through development of an online notification service is to reduce mailing costs by sending fewer notices via mail. However, IRS officials told us in September 2019 that they plan to continue to mail all notices once an online notice service is developed. This suggests that IRS will move forward with development of an online notification service without clear objectives such as cost savings. In December 2019, IRS officials noted that while this may be true of initial deployment, future iterations could potentially allow users to change their delivery preferences. The services outlined in IRS’s modernization plan include delivery of tax credit qualification notices electronically to taxpayers, which IRS officials explained would be limited to low-income taxpayers who IRS believes to be eligible for, but not claiming, the Earned Income Tax Credit (EITC). TAS’s pilot of secure messaging with a similar set of taxpayers— taxpayers subject to an audit of their EITC claim—showed that many low- income taxpayers did not have the access to technology to properly enroll. IT officials told us that they plan to conduct customer testing before and after the introduction of the new electronic notice service. Because IRS is just beginning development of its new digital communication platform, it has not yet provided evidence that it plans to consider concerns and limitations identified in prior digital communication pilots. Without developing a pilot to test its new services and incorporate the lessons learned from prior pilots, IRS risks developing a full-scale service targeted to taxpayers with a low potential for uptake. Our discussions with IRS officials confirmed that they continue to address security and human capital challenges, which we have evaluated in recent reports. IRS’s modernization plan states that IRS faces increasingly sophisticated and frequent efforts by cybercriminals to steal taxpayer data. For example in 2015, IRS temporarily suspended online transcript services after fraudsters used personal information obtained from sources outside IRS to pose as legitimate taxpayers and access tax return information from up to 724,000 accounts. IRS relaunched this service in 2016 with the requirement that taxpayers go through Secure Access. IRS also uses Secure Access for “View Your Account Information”. The remaining online services require different levels of authentication. For example, taxpayers must provide their Social Security numbers or individual taxpayer identification numbers, filing status, and exact refund amounts to access “Where’s My Refund?” The information provided is limited to tracking IRS’s receipt of a return, approving the refund, and sending the refund. Users of this service cannot, for example, redirect the refund from the destination specified on the tax return or access the detailed personal information contained in transcripts. If IRS makes the authentication process too stringent, it may adversely affect legitimate taxpayers, but too easy of an authentication process presents security risks. In June 2018, we recommended 11 actions IRS should take to improve taxpayer authentication, including developing a plan to fully implement new federal guidelines for online authentication and IRS agreed with our recommendations. In June 2019, IRS officials stated that they have identified an approach for improving the security of online authentication consistent with new federal guidelines. However, additional work remains to fully address our recommendations. Further, the Taxpayer First Act, enacted in July 2019, requires IRS to verify the identity of any individual opening an “e-Services account” by January 2020 before the individual can use the e-Service tools. IRS also continues to face human capital challenges. The former Acting Director of OLS told us in March 2019 that her office has faced several challenges in recruiting and hiring: (1) competition with technology companies for employees with the necessary skills; (2) challenges in crafting position descriptions to inform job seekers of openings; and (3) delays in IRS’s Human Capital Office processing of applications. These challenges are similar to IRS-wide challenges we recently identified, including skill gaps in mission critical occupations and limited capacity by the Human Capital Office to hire employees. In March 2019, we recommended IRS take six actions, including improving its workforce planning and addressing delays in the hiring process. IRS agreed with our recommendations. In September 2019, the Deputy Commissioner for Operations Support reported that IRS is working to address our recommendations, including a plan to reduce the hiring backlog, increase hiring capacity, and improve monitoring and reporting capabilities. In November 2019, we determined IRS had addressed two of our recommendations by developing a strategy to address current and future hiring requirements and issuing guidance to business units’ executives on streamlining the hiring approval process. As noted above, the Deputy Commissioner for Operations Support reported that IRS is working to address our remaining recommendations. Taxpayers cannot file their tax returns on irs.gov and IRS officials told us they have no plans to develop such a capability. As noted above, the absence of electronic filing services on irs.gov is a notable difference between the services IRS provides and those provided in the three countries and two of the three states we reviewed. We found that IRS’s Free File agreement benefits a small proportion of taxpayers, but that the full benefits and costs of this agreement are uncertain. IRS has renewed the nearly 20-year old agreement eight times since its inception in 2002 without sufficient consideration of how this agreement relates to its growing portfolio of online services, such as the development of the capability for taxpayers to file amended returns electronically. IRS officials do not regard the absence of electronic filing capabilities on irs.gov to be a shortcoming. Rather, they believe that the Free File agreement has served both taxpayers and IRS well. Officials noted that eligible taxpayers can receive free access to electronic tax preparation and filing services provided by the companies which make up the Free File, Inc. consortium. Additional benefits accrue to IRS, according to officials, by encouraging electronic filing which reduces the costs of processing returns. Further, IRS officials noted that having industry assist taxpayers with electronic filing allows them to focus on providing other online services, such as the informational and payment services described above. Officials representing Free File, Inc. expressed similar views on what they consider to be the benefits of the agreement. However, IRS data show that less than 2 percent of all individual tax returns were filed using Free File in fiscal year 2018 (see figure 5). IRS’s annual data books started tracking the number of returns filed using Free File in fiscal year 2009. As the data show, excluding paper returns, approximately 2 to 3 percent of all electronically filed returns were filed through Free File for the 10 years with available data. IRS’s data include taxpayers who were eligible for and used free commercial software or websites and those who used Free Fillable Forms discussed earlier (see table 2). As the data show, the vast majority of taxpayers filing electronically either hired a practitioner to do so on their behalf or obtained commercial tax preparation and filing services outside of the Free File program. The low usage of Free File is one of the topics that has been reviewed in more detail in reports by the National Taxpayer Advocate and IRS’s Advisory Council. Usage was also cited in separate letters that the Chairman and Ranking Member of the Senate Committee on Finance and the Chairman and Ranking Member of the House of Representative’s Committee on Ways and Means sent to the IRS Commissioner in May 2019 requesting a review of Free File. In June 2019, IRS hired the MITRE Corporation to review, among other objectives, the Advisory Council’s findings and recommendations. In an October 2019 report submitted to IRS, the MITRE Corporation examined: (1) The extent to which eligible taxpayers were using Free File; (2) the participating companies’ compliance with the agreement between Free File, Inc. and IRS; and (3) researchers’ observations of taxpayers’ experiences using the software provided by companies participating in Free File. The report found that the program generally appeals to taxpayers who prefer a “do-it-yourself” method of tax preparation and filing and taxpayers’ preferences should be taken into account when interpreting IRS’s usage data. It found that participating companies had generally complied with the terms of the agreement but that taxpayers experienced challenges in navigating the Free File program. The report made a number of recommendations on these and other topics to IRS. Under the terms of the Free File agreement, IRS does not pay Free File, Inc. companies for the services provided. Rather, participating companies benefit from continuing this agreement because they stand to potentially lose business should IRS develop its own online filing capabilities. Although the agreement does not have a direct monetary cost to IRS, our review found there are indirect costs. Specifically, the agreement states that “the federal government has pledged to not enter the tax preparation software and e-filing services marketplace.” IRS’s decision not to develop and offer electronic filing on its website is a contrast to the capabilities offered by some other countries and U.S. states. The Free File agreement in its current form could potentially constrain the development of new online services such as allowing taxpayers to file amended returns on irs.gov. Online services have the potential to decrease both taxpayer burden and costs for revenue agencies in the long term. IRS’s efforts to assist taxpayers with amending previously filed tax returns illustrate the potential costs of renewing the Free File agreement without consideration of IRS’s long-term plans for online services. Irrespective of the method used to file the original return, taxpayers must file an amended return (Form 1040X) on paper. Officials in IRS’s Wage and Investment (W&I) business operating division provided a business case they drafted proposing to give taxpayers the option of filing an amended return electronically. The business case makes clear that IRS officials believe the current paper-based process is inconvenient for taxpayers because the vast majority of taxpayers are filing the original return electronically. It is also costly and challenging for IRS to process these paper forms with more than 3 million of these amended returns received in processing year 2017. The business case says IRS has been assessing a potential online service in this area for more than 10 years, but has not moved forward due to technical and resource challenges. One approach IRS is exploring is to allow private sector tax preparation and filing companies and practitioners to file an amended return on behalf of a client, which would be similar to the current arrangement for original returns established by the Free File agreement. The second approach IRS is exploring is allowing taxpayers to correct the return with a new online service on irs.gov. The business case states IRS currently prefers the first approach of having taxpayers work with industry or a tax practitioner because of a combination of cost and technical considerations. The business case states that IRS officials have had discussions with officials affiliated with Free File, Inc. According to IRS’s account of these discussions, industry is supportive of the first potential approach of having taxpayers electronically file amended returns through their industry. Further, IRS notes that some taxpayers who used software to prepare the original return may find it convenient to use the same software to prepare and file an amended return. However, the business case also states that, “the costs are not insignificant” for IRS in working with industry, even though IRS plans to leverage existing systems as much as possible. A further complication is how IRS’s plans to work with industry on electronic filing of amended returns relate to the Free File agreement. IRS’s business case states that Free File, Inc. officials told them participating companies would be willing to provide electronic filing of amended returns for free, but as noted above less than 2 percent of original returns are filed through Free File. While individual tax preparation and filing companies could choose to offer electronic filing of amended returns for free or include that capability in paid packages they offer for filing an original return, the agreement in its current form would not guarantee free access to electronically filing amended returns for the vast majority of taxpayers who file an original return outside of Free File. If IRS were to return to its earlier idea of offering the capability to file an amended return on irs.gov, that approach also comes with potential risks for IRS regarding the Free File agreement. IRS officials noted that the agreement states that, “this agreement does not limit IRS from providing phone-based, web-based, or electronic interaction between the IRS and a taxpayer (or a taxpayer’s representatives) regarding issues in a previously filed return after such a return has been accepted by IRS.” IRS officials told us they believe this language allows Form 1040X-type actions by IRS. However, as noted above, IRS made a commitment to “not enter the tax preparation software and e-filing services marketplace”. Our analysis determined that the Form 1040X is nearly identical to the Form 1040, with the difference being that a taxpayer notes which lines he or she needs to correct. IRS’s instructions for the Form 1040X state, “When you file Form 1040X for a tax year, it becomes your new tax return for that year. It changes your original return to include new information.” Therefore, the capability for taxpayers to file a Form 1040X on irs.gov would put irs.gov closer to having an online filing capability for original returns. In written documents that officials from Free File, Inc. provided to us, they said they would need to see a specific proposal for electronic filing of amended returns before they could comment. However, they made clear that they believe any future development of IRS’s online services should continue to leave the task of preparing and electronically filing a tax return to industry. Officials provided a copy of a letter the Executive Director of their organization had sent the W&I Commissioner in March 2019 recommending that IRS consider enabling the electronic acceptance of amended returns through the system industry uses to electronically file original returns on behalf of taxpayers. Circumstances, including IRS’s technical capabilities, have changed since the Free File agreement was first established in 2002. Today’s irs.gov provides “View My Account Information” and other online services which did not exist in 2002, and as discussed above, IRS is exploring allowing taxpayers to file amended returns electronically. Another potential new online service serves as a second example of a way that IRS might soon interact directly with taxpayers online without the use of private sector intermediaries in the tax preparation and filing industry. In the Taxpayer First Act, Congress directs IRS to develop a new online service for taxpayers to report miscellaneous payments. Specifically, Congress directed IRS to develop no later than January 1, 2023, an internet platform for persons to prepare and file the Form 1099, which is used by persons to report payments made to taxpayers for such things as rent and services performed by someone other than an employee. The House Committee on Ways and Means’ report accompanying the act explains that the committee believes that having IRS provide this online service will improve compliance with the reporting requirements and reduce the administrative burden for taxpayers who run small businesses. OLS coordinates the development of new online services and W&I oversees the Free File agreement. OLS officials referred our questions about the Free File agreement to W&I. W&I officials provided no documentation they had coordinated renewal of the Free File agreement in 2018 with OLS. IRS also could not provide us any evidence that it has analyzed the full costs and benefits of the Free File agreement to IRS, the participating private sector companies, and the public. For example, the MITRE Corporation report discussed above states that the researchers “assume that industry will continue to be the entity that provides free tax return preparation and filing offerings to taxpayers.” IRS’s approach to renewing the Free File agreement is not consistent with leading practices we identified in our prior work stating that decision makers should periodically review government programs, tax provisions, and regulations to ensure they are achieving desired goals. Among the leading practices we identified is that the costs and benefits should be assessed. Without more rigorous examination of costs and benefits to all parties of future renewals of the Free File partnership, IRS runs the risk of not being fully aware of the effects of the agreement; including the effects of constraints on new services that IRS could provide to taxpayers. The internet has reshaped how citizens interact with businesses and government agencies. IRS.gov has contributed to this by giving taxpayers access to detailed information about their taxes and allowing them to make arrangements to pay money they owe. Our comparison of IRS to other countries’ and states’ revenue agencies highlights areas for potential future development. IRS has told Congress and the public that developing electronic communication capabilities is an area of focus. Our review identified a number of challenges IRS will need to address as it moves forward, including measuring taxpayers’ experiences with the online services IRS already offers—including the extent to which those services meet taxpayer needs—and how these services may affect taxpayer burden. Summarizing and reporting that information would help decision makers appreciate the potential value of these services and help ensure IRS has the necessary resources to maintain and improve these services. Likewise, including input from taxpayers when prioritizing new services would help IRS reduce the risk of developing online services that taxpayers do not use. While IRS has recently published a modernization plan which outlines its vision for expanding online services, no targets are set for improving taxpayer experience or reducing taxpayer burden, which hinders decision-making. As we recommended in April 2013, we continue to believe that IRS should establish a numerical or other measureable goal to improve taxpayer satisfaction and a time frame for achieving it. IRS also faces a risk that plans for full-scale digital communication services will encounter enrollment challenges similar to those that IRS has experienced in prior digital communication pilots. In 2002, IRS established a Free File agreement in which participating tax preparation companies agreed to provide free electronic tax preparation and filing services for low- and middle-income taxpayers, provided that IRS does not enter the tax preparation software and e-filing services marketplace. IRS’s Free File agreement benefits a small proportion of taxpayers, but the full benefits and costs of this agreement are uncertain. IRS is currently constrained in providing the online services that are part of its long-term plans for taxpayers, including allowing electronic filing of amended tax returns. We are making the following seven recommendations to IRS: The Commissioner of the IRS should ensure that information is collected on taxpayers’ experiences with all online services and the extent to which the services are meeting taxpayers’ needs. (Recommendation 1) The Commissioner of the IRS should ensure that information collected on taxpayers’ experiences with online services is summarized in the document serving as IRS’s performance plan and report. (Recommendation 2) The Commissioner of the IRS should direct the Director of OLS and the Chief Research and Analytics Officer to work together to analyze the potential effects of online services on taxpayer burden. (Recommendation 3) The Commissioner of the IRS should ensure that taxpayer input is included as an element of IRS’s identification and prioritization process for new online services. (Recommendation 4) The Commissioner of the IRS should work with relevant officials to set a target to reduce taxpayer burden through the development of new online services. (Recommendation 5) The Commissioner of the IRS should direct the Chief Information Officer and the Director of OLS to ensure that planned future capabilities of digital communication platforms are tested or piloted before deployment with a particular focus on mitigating the risks that were identified in prior pilots of digital communication services, such as challenges in establishing common objectives and enrolling taxpayers. (Recommendation 6) The Commissioner of the IRS should direct the Commissioner of W&I to work with the Director of OLS to ensure that future decisions regarding whether to renew the Free File agreement incorporate findings from a comprehensive examination of the benefits and costs of the agreement as it relates to long term plans for IRS’s online services, including plans to file amended returns electronically. (Recommendation 7) We provided a draft of this report to IRS for review and comment. In written comments provided by IRS’s Deputy Commissioner for Services and Enforcement (reproduced in appendix II and summarized below), IRS agreed with six of our seven recommendations. IRS also provided technical comments, which we incorporated as appropriate. IRS agreed with our recommendations to ensure that information is collected on taxpayers’ experiences with online services, ensure that such information is summarized in IRS’s performance plan and report, analyze the potential effects of online services on taxpayer burden, include taxpayer input in its identification and prioritization of new online services, ensure that planned future capabilities of digital communication platforms are tested or piloted, and ensure that future decisions regarding renewal of the Free File agreement incorporate findings from a comprehensive examination of the benefits and costs of the agreement as it relates to long term plans for IRS’s online services. IRS indicated general steps it plans to take to address these recommendations but did not provide time frames for doing so. IRS disagreed with our recommendation that it set a target to reduce taxpayer burden through the development of new online services. IRS stated that it will continue to look for opportunities to reduce burden through the development of new online services, but believes that a measurable target cannot be set. We recognize that it may take time for the relevant IRS offices to review changes in individual taxpayer burden estimates over multiple years and begin to collect the necessary data to set a measurable target for burden reduction. However, as established in our prior work, goals should be expressed in as specific terms as possible and be expressed in a form which allows the agency and external audiences to assess the progress being made. As noted in our report, IRS has a strategic goal for reducing taxpayer burden and its strategic plan identifies expanding online services as one of the strategies it will use to drive progress on its goal. Further, IRS agreed with our related recommendation that relevant offices analyze the potential effect of online services on taxpayer burden, which should provide a starting point for IRS in working to identify a burden reduction target. In its response, IRS also stated that its methodology for estimating taxpayer burden is not designed to capture the effect of specific program improvements on taxpayer burden. We agree and are not suggesting IRS resurvey taxpayers and re-estimate burden for every new online service it may introduce. Rather, our recommendation refers to total burden reduction from all the online services IRS offers individual taxpayers. As noted above, IRS’s strategic plan anticipates that taken together all these different online services should make it easier over time for taxpayers to fulfill their tax obligations. We continue to believe that this recommendation has merit. We provided relevant sections of this report to OMB staff concerning information they provided us regarding the applicability of customer experience requirements to IRS. Staff confirmed we accurately summarized their statements. We provided relevant sections of the draft report to the revenue agencies and national audit offices in the three countries reviewed and to the revenue agencies in the three states reviewed. Two foreign revenue agencies, three national audit offices, and three state revenue agencies provided technical comments, which were incorporated as appropriate. One foreign revenue agency did not respond as of December 10, 2019. We also contacted 19 additional state revenue agencies which our draft report identified as offering electronic filing of a state income tax return on the revenue agency’s website and verified that 15 of them offer this service. The remaining four did not respond as of December 12, 2019. We provided relevant sections of the draft report to officials representing Free File, Inc.; specifically, sections describing the agreement between their organization and IRS and the views of Free File, Inc., officials towards IRS plans for allowing electronic filing of amended returns. An official representing the organization provided technical comments, which were incorporated as appropriate. We are sending copies of this report to the relevant congressional committees, the Secretary of the Treasury, the Commissioner of the IRS, and other interested parties. In addition, this report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-9110 or lucasjudyj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. The four countries in this review—the United States and Australia, New Zealand, and the United Kingdom—define income similarly and tax investment income. For wages, all four revenue agencies require employers to withhold taxes from their employees’ paychecks. Withholding also means that the four revenue agencies have processes for sending refunds to taxpayers should the government end up collecting more money than the taxpayer owes in taxes. Further, all four countries use tax expenditures to channel subsidies through their tax systems to further social goals such as supplementing the wages of lower income workers. The requirement to file a tax return depends on a combination of factors: gross income, filing status such as whether a taxpayer is single or married, age, and whether a taxpayer is a dependent. For example, a single taxpayer who is under the age of 65 and has a gross income of at least $12,000 is required to file. Taxpayers who meet the specified criteria must file a return even if the government owes them a refund. A taxpayer married to another taxpayer can choose to file a joint return which reports the two individuals’ combined income and deductions. Selecting this option may provide a higher standard deduction and access to other tax benefits for married couples. The Australian Taxation Office’s (ATO) website states that most taxpayers need to lodge a tax return each year and taxpayers can choose among using their online account to electronically file a return, submit a paper return, hire a registered tax agent, or taxpayers meeting specified eligibility criteria may seek assistance from ATO-trained volunteers who help taxpayers complete their tax returns online. Spouses file separate returns, although taxpayers are required to report details about their spouse’s tax situation, such as their taxable income. After a return is lodged, ATO issues a notice of assessment informing the taxpayer whether he or she is entitled to a refund or owes tax. A taxpayer can correct errors on the original return electronically, on paper, or through a registered tax agent. An Inland Revenue Department (IRD) publication and the IRD website explain that the requirement to file depends on the sources of the taxpayer’s income. Taxpayers who received income other than salary, wages, interest, dividends, or taxable Maori authority distributions must file a return. For example, the filing requirement applies to taxpayers who had self-employed income, rental income, cash jobs, or income derived overseas exceeding 200 New Zealand dollars. Taxpayers required to file can log into their online account to electronically file a return or choose to file on paper or hire a tax agent to prepare their return. After receiving the return, IRD informs the taxpayer of any refund or tax they must pay. Taxpayers can correct errors on their return by logging into their online account or calling IRD. In regards to the remaining taxpayers whose income is from sources IRD is aware of, such as wages, the department sends them an assessment informing them of whether they owe taxes or are owed a refund and directing them to report any additional income over 200 New Zealand dollars IRD does not know about, such as cash jobs. To support this, an IRD official explained that his department receives information from employers on income paid and taxes withheld every pay cycle. Further, the official reported that beginning in April 2020 IRD will receive at least monthly information from financial institutions on investment income, which should help IRD further refine its calculations of the tax positions of taxpayers. Regarding married taxpayers, each spouse is required to file his or her own return or receives their own assessment, although the tax return states that the amount a spouse or partner (or ex-spouse or ex- partner) received for a tax credit for working families may affect the other spouse’s tax situation. The United Kingdom’s government website explains that Her Majesty’s Revenue and Customs (HMRC) uses a system—called pay as you earn—in which taxes are deducted automatically from wages and pensions. This means that taxpayers whose only income is from wages or pensions are generally not required to file a return. HMRC mails these taxpayers an annual tax summary (or taxpayers can view it online in their account) informing them of their taxable income and amount collected. If HMRC determines that they owe the taxpayer money or the taxpayer owes the government money, HMRC sends a separate form explaining how it will pay or collect this money. Taxpayers with untaxed income (e.g., renting a property, tips and commissions, income from investments and dividends, and foreign income) may be required to file a return (also referred to as a self-assessment). Even if a taxpayer is not required to file a return, he or she may choose to file a return to claim “income tax reliefs” for such activities as making pension or charitable contributions. Before filing a return, taxpayers who did not file a return in the previous tax year must register with HMRC, which can be done online. HMRC will mail an identification number and activation code and set up an online account for the taxpayer to use to complete the self-assessment. Once taxpayers confirm they are registered, they complete their tax return online using their personal account or choose among using commercial software, hiring an accountant or someone else to help them, filing on paper, or taxpayers meeting specified eligibility criteria may be able to get free professional advice. However, HMRC advises there are certain tax situations, such as a taxpayer receiving income from a partnership, in which its website cannot be used and taxpayers in these situations must use commercial software or file on paper. Regarding correcting errors on filed returns, taxpayers who submitted their return on HMRC’s website can make corrections there, while paper filers must mail a corrected form. Married taxpayers file separate returns, but the tax form has a marriage allowance section which allows a taxpayer to transfer a portion of his or her personal allowance to their spouse or civil partner under certain conditions. The ATO commenced development of new online services for individual taxpayers in 2015 under the direction of its modernization plan, titled “Reinventing the ATO”. The Australian National Audit Office (ANAO) reviewed ATO’s modernization plan in a 2017 report and found that the ATO’s modernization plan provided “clear road maps outlining program intent, deliverables and timing” but identified challenges for ATO in conformance to those processes, specifically in completing cost estimates for development of all new services. In addition, the ANAO found that the costs and benefits associated with the “Reinventing the ATO” program and most of its projects had not been tracked. ANAO reported that ATO collects survey information from taxpayers about the ease of accessing services and information, doing business with the ATO, and measures of timeliness in processing complaints. However, ANAO noted that ATO’s online services have experienced periods of outages, but ATO has not monitored the impact of service outages on satisfaction with its services. In 2017, ANAO reported that ATO had successfully implemented its recommendation to develop an overarching cross-channel strategy that detailed how the ATO plans to transition to an improved online service environment, while also continuing to provide and improve the performance of other service channels. In 2011, IRD began a long-term business transformation program, which plans to modernize tax administration in New Zealand and offer new online services to taxpayers. New Zealand’s Office of the Auditor General (OAG) reviewed IRD’s governance of the business transformation program in 2015 and found IRD to be providing clear direction and supporting clear and effective decisions, but recommended that IRD continue to manage risks, including identifying clear benefit estimates to decision makers. OAG also recommended that IRD manage risks by improving its outreach to stakeholders and taxpayers in advance of the release of new services. As a result, IRD stated in its 2018 program update that it intends to be more proactive in engaging with individual taxpayers. In regards to IRD’s procurement of goods and services for the business transformation program, OAG found instances in which IRD did not consistently comply with relevant rules and policies and made recommendations for improvement. OAG’s report, however, noted that IRD restructured its procurement function and brought in procurement specialists with appropriate skills and resources and OAG intends to follow up on the progress IRD is making in addressing its recommendations. An OAG official reported in November 2019 that his office is currently doing a performance audit of the measurement of benefits from the IRD business transformation program. OAG anticipates submitting the report to the House of Representatives in the first half of calendar year 2020. HMRC outlined a strategy in 2014 (it refers to as a transformation program) to improve its online services for individual taxpayers, which included goals of promoting voluntary tax compliance, designing services to meet customer needs, and improving ease and convenience to taxpayers. The United Kingdom’s National Audit Office (NAO) has reviewed HMRC’s customer service performance, including online services. A 2016 NAO review found that HMRC had reduced the cost of its personal tax operations between 2010-2011 and 2014-2015 in part by moving customers from traditional service channels to less expensive service channels, including online services. Initially, HMRC maintained or improved its customer service performance, but HMRC ended up releasing too many customer service staff and wait times for telephone service started to increase in 2015-2016. While HMRC’s performance improved after it recruited additional staff, NAO concludes that the sustainability of HMRC’s cost reductions will depend on the success of new online services in reducing demand for telephone and mail service. Another review by NAO in 2017 credited HMRC for exceeding its target set for customer satisfaction for digital services, which includes both existing services and new services. Moving forward, NAO recommended that HMRC continue to reevaluate its priorities for its transformation program at least annually, including by measuring the impact on customers, to ensure that new services are delivering the anticipated benefits. In addition, NAO recommended that HMRC be clearer about the way it tracks the costs and benefits of its transformation program. In 2019, NAO found that HMRC had reprioritized its plans due to other demands related to the agency’s preparations for the United Kingdom’s planned exit from the European Union, which has resulted in deferment of development of new online services. Jessica Lucas-Judy, (202) 512-9110 or lucasjudyj@gao.gov. In addition to the individual named above, Tara Carter (Assistant Director), Michael O’Neill (Analyst in Charge), Michael Bechetti, Jacqueline Chapin, Rianna Jansen, Edward Nannenhorn, Andrew Olson, Julia Robertson, Kayla Robinson, Cynthia Saunders, Stewart W. Small, Andrew J. Stephens, Robyn Trotter, and Christopher Woika made key contributions to this report.", "summary": "IRS recognizes that taxpayers want more choices in how they interact with IRS, including through online services. GAO was asked to review IRS's online services—those which allow IRS and individual taxpayers to exchange personalized information electronically. This report (1) examines what is known about how IRS's current online services are meeting taxpayers' needs, and provides information about selected foreign and state revenue agencies' online services; (2) evaluates the extent to which IRS's strategy for identifying and prioritizing the development of new online services is consistent with relevant requirements and leading practices; and (3) examines how IRS is addressing key challenges in providing online services. GAO assessed IRS's online services against relevant requirements, agency goals, and leading practices; interviewed IRS officials; and identified additional services and practices from six foreign and state revenue agencies selected for offering multiple online services for exchanging personalized information with taxpayers. The Internal Revenue Service's (IRS) online services for individual taxpayers primarily provide taxpayers one-way communication of key information derived from their tax return, such as when an anticipated refund should arrive, or allow taxpayers to pay money owed or make payment arrangements. IRS has done little research or reporting on the extent to which its online services are satisfying taxpayers' needs. Also, IRS has not set a target for using online services to help reduce taxpayer burden. Selected foreign and state revenue agencies' online services have developed online filing and communication capabilities, such as filing a tax return on the agency's website and offering electronic chats between revenue agency employees and taxpayers (see figure). IRS has long-term planning documents which detail online services it intends to develop, which include services to communicate digitally with taxpayers, to achieve its goal of modernizing the taxpayer experience. However, GAO found that IRS has not sufficiently considered taxpayer input in the prioritization process for these new services and instead prioritizes services primarily based on the potential benefit to IRS operations or how quickly a service might be developed. Without considering taxpayer input on user needs and preferences, IRS risks developing services that taxpayers do not use. A group of private sector tax preparation companies known as Free File, Inc., has a long-standing agreement with IRS in which the companies provide free electronic tax preparation and filing services to eligible taxpayers in exchange for IRS not offering its own filing capability. However, few taxpayers use these services and GAO found that IRS has given inadequate consideration to the full benefits and costs of the Free File agreement to all parties. Not considering these costs and benefits has implications for the future evolution of IRS's online services, including helping taxpayers electronically file amended returns. GAO is making seven recommendations to IRS, including measuring and reporting on the effect of online services on satisfaction and taxpayer burden and setting a target for reducing burden, considering taxpayer input when prioritizing new online services, and ensuring that any renewal of the Free File agreement reflects benefits and costs. IRS agreed with six recommendations, but disagreed on setting a target to reduce burden. GAO continues to believe IRS should set such a target.", "document_type": "gao"}
{"report": "According to the Summary of the 2018 National Defense Strategy and the Army, the character of warfare is changing. For decades, the United States enjoyed uncontested or dominant superiority in every operating domain—land, air, sea, cyber, and space—but today every domain is likely to be contested by other great-power competitors and potential regional adversaries. Figure 1 below describes these operating domains. Since at least 2012, DOD began shifting its focus from counterinsurgency operations in Iraq and Afghanistan to adversaries who possess more sophisticated capabilities. For example: In 2012, DOD issued strategic guidance that cited efforts by Iran and China to pursue cyber and electronic warfare capabilities with the ability to counter U.S. power projection and limit operational access. The 2014 Quadrennial Defense Review acknowledged the efforts of China and others to counter U.S. strengths using anti-access and area-denial approaches and using new cyber and space control technologies. The 2014 Quadrennial Defense Review also addressed the rapid evolution of modern warfare, including increasingly contested battlespaces in the air, sea, space and cyber domains. In 2016, an Army study of Russia’s operations and doctrine concluded that Russia employs formations, operational concepts, and capabilities that overmatch U.S. capabilities in range and lethality, thus challenging the Army’s ability to conduct operations and win battles. The 2017 National Security Strategy stated that U.S. advantages are shrinking as rival states modernize their forces. The 2017 National Security Strategy identified many of the challenges that China and Russia pose, including Russia’s use of offensive cyber efforts to influence public opinion, and how cyberattacks have become a key feature of modern warfare. A classified National Defense Strategy followed in January 2018, and the unclassified summary cited challenges to the U.S. military advantage as a shift in the global environment. The Army’s multi-domain operations concept originates from an Army effort to rethink how it will fight in the new, more complex operating environment. The Army defines multi-domain operations as ways for confronting adversaries in contested environments by presenting them with multiple challenges through the combining of multiple capabilities. This means that ground forces should be able to operate freely in other warfighting domains and, if necessary, be able to overwhelm an adversary’s forces by combining capabilities across different domains, such as land, air, sea, cyber, and space simultaneously. According to Army officials, in 2014 the then-Deputy Secretary of Defense tasked the Army to update its warfighting concept to deal with the threats and challenges posed by great-power competitors in the future operating environment. The Army officials added that around the same time, the Army began developing and running a wargame scenario focused on a threat that employed similar doctrine, tactics, and capabilities as those used by Russia in Ukraine. In 2016, the Army also assessed the increasingly sophisticated Russian military capabilities and identified specific multi-domain challenges that the Army would face if it came into conflict with Russia. Army officials said that its analysis highlighted the urgency of updating how it would fight such an adversary. In beginning to develop this concept, the Army reached out to the Marine Corps, as both services face similar problems in ground-combat operations. Since the Army began developing its concept, the Army established a framework for assessing how the adversary operates and the problems the Army needs to resolve as a ground force. For example, early on the Army developed an expanded battlefield that stretches far beyond the front lines, or “close area”, where ground forces face off against each other. Under this expanded battlefield, adversaries can use more sophisticated weapons and cyber capabilities that are based in distant and protected territories, potentially reaching targets that are located well behind the front lines, even within the continental United States. Figure 2 below depicts the Army’s new expanded battlefield for multi-domain operations, including a description of each area of the battlefield. The Army is changing aspects of its doctrine, organizations, and training simultaneously to develop a force that can effectively engage great-power competitors, such as Russia and China, across multiple domains, and expects this process to continue through the 2020s. Army concepts propose new approaches for the Army to develop capabilities against emerging challenges. The new Army Operating Concept built around multi-domain operations is intended to drive capability development, which is addressed through changes to the Army’s doctrine, organizations, and training, among other areas. The Army’s goal is to field a more lethal and capable force by 2028 that is able to dominate adversaries in a multi-domain environment. Figure 3 below summarizes how the Army uses validated concepts to drive changes in capabilities and the force. Doctrine. Given the Army’s attention to multi-domain operations, it has updated or is in the process of updating doctrine that guides how the Army fights. Primary among this effort is updating the Army’s overarching operations field manual, which establishes how the Army conducts large- scale ground combat operations against the threat posed by a great- power competitor, among other things. In its most recent revision to its doctrine, the Army incorporated several aspects of multi-domain operations, such as the expanded battlefield that includes cyber and the electromagnetic spectrum. TRADOC officials stated that they are also in the process of updating doctrine related to cyber operations and field artillery operations in order to build a force that can integrate both cyber capabilities and long-range fires—such as artillery, rockets, and missiles—for multi-domain operations. The officials added that the Army is developing or is planning to develop specific doctrinal guidance for new Army units that will focus on multi-domain operations in the areas of intelligence, cyber, electronic warfare, and space. Organizations. The Army wants to ensure that its warfighting organizations have the engineering, artillery, air defense, and other enabling capabilities needed to conduct multi-domain operations. For example, the Army believes that formations above the brigade level, such as division headquarters and corps headquarters, must have the ability to conduct electronic warfare and cyber operations. To that end, the Army is creating several new organizations focused on cyber and electronic warfare (discussed later in the report). Additionally, the Army is trying to align its multi-domain operations concept with a complementary concept focused on the roles and responsibilities of these organizations above the brigade level. Expanding the roles and responsibilities of formations above the brigade level signifies a departure from the Army’s modular force, which was implemented beginning in 2004. At that time, the Army embedded “key enablers” such as military intelligence, reconnaissance, and logistics functions, as well as other specialized personnel and equipment, into brigade combat teams to provide them independent capabilities. Moving forward, the Army envisions enhancing the capabilities of brigade combat teams for multi-domain operations, as well as providing additional key capabilities to formations above the brigade level. For example: Brigade combat teams. Brigade combat teams are the Army’s primary tactical unit, composed of around 4,400-4,700 soldiers. They are being adjusted to conduct operations in the cyber domain, including new platoons focused on electronic warfare. Army division headquarters. Army divisions command multiple brigade combat teams. The Army expects division headquarters to manage the electromagnetic spectrum and to be the primary echelon for integrating aviation, fires, and electronic warfare into ground maneuver to defeat enemies in a close fight. Army corps headquarters. Army corps command multiple divisions. Under the Army’s concept, the Army corps headquarters will be the primary echelon for defeating mid- and long-range enemy artillery fires. The Army corps will also integrate artillery rockets and missiles, as well as cyber capabilities in support of division or brigade ground operations. Field armies. Field armies, which have the ability to command two or more Army corps, are forward-stationed in regions with capable threats posed by great-power competitors. They will conduct campaigns to compete with adversaries short of armed conflict, and manage the transition to armed conflict should it be needed. The field army will also direct deception operations and provide long-range artillery and fires support. Theater armies. Theater armies are also forward-stationed forces and will be responsible for managing and combining Army capabilities in support of information environment operations and space operations. The theater army must be able to protect joint bases and networks and enable access to the theater. Training. The Army is also updating its training across a broad range of efforts. Army training officials stated that there is a need to train units collectively under multi-domain operations conditions against great-power competitors like Russia and China, per guidance from the Chief of Staff of the Army. The commander of Army Forces Command also issued guidance for fiscal year 2019 to help train and prepare soldiers to conduct multi-domain operations. This guidance included increasing the realism and rigor of every unit rotation to one of the Army’s combat training centers, as well as designing warfighter exercises that focus on units conducting operations in contested electronic warfare, cyber, and space environments. Additionally, the training officials stated that in recent years the Army has updated its decisive-action training scenarios to include regional versions for Europe, the Pacific, and Africa that comply with the multi-domain operations concept. The officials added that, in future years, several Army organizations will be collaborating to modernize the Army’s home-station training and combat training centers in support of fielding a force capable of conducting multi-domain operations. All of this builds upon the Army’s earlier efforts to shift its training focus to large-scale combat after a decade of training for counterinsurgency operations, as we testified to Congress in February 2019. The Army is also taking steps to revise the training for cyber and electronic warfare personnel. These steps include revising the U.S. Army Cyberspace Operations Training Strategy so that it accounts for new equipment and doctrine, but also for the new organizations being created and the tasks those units will be expected to perform, according to Army cyber officials. Additionally, the Army Cyber School is revising its cyber and electronic warfare training so that personnel will be able to conduct multi-domain operations. Furthermore, the Army is working on a joint solution for training cyber personnel on behalf of U.S. Cyber Command, according to Army Cyber Command officials. The Army’s goal is to provide the total cyber force with the ability to conduct joint cyber training, including exercises and mission rehearsals by developing a virtual training environment that simulates realistic cyber threats. This cyber training solution, called the Persistent Cyber Training Environment, will allow for experimentation, unit certification, and assessment and development of the cyber mission force in a virtual training environment. The Army’s goal is that the environment will decrease training time, increase throughput of personnel, and improve training quality. One of the stated operational imperatives of the Persistent Cyber Training Environment is to become integrated with multi-domain exercises. The Army is seeking to quickly create or design several new cyber and electronic warfare units in order to execute multi-domain operations; however, Army leadership is activating some units at an accelerated pace due to the sense of urgency imposed by the growing capabilities of potential great-power competitors. Some of these new Army units are more narrowly focused on a particular domain or skill set, such as the recently activated 915th Cyber Warfare Support Battalion based out of Fort Gordon, Georgia, and new Electronic Warfare Companies and platoons. The 915th Cyber Warfare Support Battalion will focus on providing offensive cyber capabilities consistent with its authorities to conduct offensive operations. The battalion is designed to fit with various Army formations—such as corps, divisions, or brigade combat teams—as assigned by the Army. The Electronic Warfare Companies, which are scheduled to be fielded during fiscal years 2023 through 2025 according to Army officials, will be attached to an Army corps and will be capable of planning and conducting electronic warfare operations. Electronic Warfare platoons, which Army officials said are scheduled to be fielded during fiscal years 2020 through 2022, will provide similar capabilities to brigade combat teams and other Army tactical-level formations. Other units are being designed to plan and conduct operations in and across multiple domains, with specialists in cyber, electronic warfare, space, and intelligence assigned to the same unit. For example, a recently activated Intelligence, Cyber, Electronic Warfare, and Space (ICEWS) unit will be capable of planning and directing operations in any or all of those areas. The ICEWS unit will function as part of a larger Multi-Domain Task Force, which will be capable of expanding those operations into other domains such as land and air. The Army plans to field at least two of these ICEWS units by the end of fiscal year 2020. Additionally, the Army is restructuring or creating Cyber, Electromagnetic Activities planning sections in the headquarters of more than 125 Army formations, from special forces units up to theater-level Army headquarters. This restructuring effort will take place during fiscal years 2020 through 2022, according to Army officials. Army guidance states that a unit’s activation date should be identified 1 to 2 years in advance, according to Army officials, in order to provide time to build up trained personnel and equipment in the unit before it is activated and available to be deployed. As a result of accelerating the activation of these units, the Army is facing interrelated challenges in terms of staffing, equipping, and training the units, as discussed below. Accelerated pace creates challenges filling positions. The Army has had difficulty filling its ICEWS unit and the 915th Cyber Warfare Support Battalion with personnel to conduct operations. See table 1 below. By accelerating the activation of the ICEWS unit in October 2018 as a pilot, or test, program, the Army activated the unit with only 32 percent of its personnel in place, and Army headquarters officials report that filling the unit with personnel with the right skills has been a slow process. The 915th Cyber Warfare Support Battalion is facing similar staffing challenges. As of the end of March 2019, the unit was understaffed by more than 80 percent as it filled 30 of 171 authorized positions for fiscal year 2019, according to an Army headquarters official. The official acknowledged that the 915th Cyber Warfare Support Battalion may not meet the authorized staffing levels for fiscal year 2019 if higher priorities arise for the service. Looking ahead, Army officials said that filling all of these new cyber and electronic warfare units could be challenging because cyber personnel are in high demand, with competition for these skilled personnel existing between the Army, other government entities, and the private sector. Army headquarters officials said they are exploring options to address the challenges and have taken steps to retain the personnel that they have, mostly in the form of retention bonuses and incentive pay. Some of those incentives are targeted at the senior enlisted levels, which are some of the personnel that Army officials indicated are in the most demand and of which they have a shortage. Accelerated pace creates equipping challenges. Officials with both Army headquarters and the Army Cyber School cited equipment challenges as one of the key issues that must be addressed when activating a unit on an accelerated basis. For example, in November 2018, an Army headquarters’ official responsible for building the ICEWS unit stated that the Army was having a difficult time identifying where the unit’s equipment would be coming from. By the end of January 2019, the official said the situation was improving and that 55 percent of the equipment had been identified, but the Army was trying to find a source for the remaining 45 percent. However, most of this is common Army equipment, such as firearms, according to an Army official; those percentages do not include the specialized cyber equipment that the unit will need to perform its missions, such as a communications system designed to transfer data beyond the line of sight during air defense operations. An Army headquarters’ official stated that the Army is prototyping different types of specialized equipment in order to expedite the acquisition of such capabilities. Revisions to training not keeping up with activation of units. Army officials acknowledged the need to update its cyber training, in part because the doctrine for new units is still being written. Officials with the Army Cyber School and the Army’s Combined Arms Center stated that the current U.S. Army Cyberspace Operations Training Strategy did not foresee all of the new cyber and electronic warfare organizations the Army now intends to create, including the Cyber Electromagnetic Activities sections attached to various formations. Army headquarters officials stated that they are working on a revision to the U.S. Army Cyberspace Operations Training Strategy to address these issues. However, the first ICEWS unit and the 915th Cyber Warfare Support Battalion were activated without this updated training strategy. With other units scheduled to be activated in fiscal year 2020, it is possible others may be activated without the training strategy as well. Without the updated doctrine and subsequent training strategies that will result from it, TRADOC officials said they would have difficulty designing training for the new units, and soldiers will not have a clear understanding of their tasks and missions. Obtaining equipment also could be a challenge for training servicemembers before they are assigned to cyber or electronic warfare units, according to some Army officials. Officials with the Army Cyber School stated that it could end up growing and producing a workforce that outpaces its ability to procure equipment. However, Army headquarters’ officials stated that equipping operational units is a higher priority than providing equipment to the schools for training, and the Army ensures that those units receiving the equipment get the training they need upon fielding the equipment. If the Army does not acquire new equipment quickly enough, the result could be that soldiers in the Army Cyber School will be trained on outdated equipment, which they will not use when they get to the field. In the process of creating some new units, the Army assessed the risk of whether it can meet the units’ staffing, equipping, and training requirements before the units’ activation date, but it did not do so for those units activated at an accelerated pace. For example, the Army conducted risk assessments for some new Electronic Warfare platoons and Cyber Electromagnetic Activities sections that it plans to begin activating in fiscal year 2020. Those assessments identified issues and mitigation strategies for the Army to consider when making fielding and resource decisions. For example, the risk of finding a sufficient number of qualified personnel for the Electronic Warfare platoons and Cyber Electromagnetic Activities sections would be mitigated by spreading the activations over a minimum of 3 years. The assessment for the Electronic Warfare platoons also identified some equipping issues that will require either more senior-level input or extending timeframes for completion. In contrast, the Army activated the ICEWS unit and the 915th Cyber Warfare Support Battalion in an accelerated manner because of the urgent need to develop these organizations, given the growing capabilities of potential great-power competitors. However, the Army did so without completely assessing the staffing, equipping, and training risk to those units over the long term. For example: According to Army officials, the Army did not perform a risk assessment for the ICEWS unit currently assigned to and participating in exercises in the Pacific, because the Army initiated the unit as a pilot, or test, program. According to Army officials, a risk assessment was unnecessary prior to activating the unit because the Army expects to refine the unit’s personnel, equipping, and training requirements during the pilot program. However, the ICEWS unit is expected to become part of a larger Multi-Domain Task Force in fiscal year 2020. Until that occurs, the ICEWS unit is attached to another active Army unit and, according to Army officials, eligible to be deployed if needed based on its current capabilities. Unless the Army assesses the staffing, equipping, and training risks of the ICEWS unit, the unit may be unable to provide the expected capabilities, either currently or as part of the larger task force to which it will belong. The Army performed an initial risk assessment for the 915th Cyber Warfare Support Battalion before the unit was activated in December 2018. However, Army officials told us that the Army has plans to grow the unit to as many as 627 personnel by 2024, at which point it would be considered fully operational. Unless the Army performs a more complete risk assessment of the 915th Cyber Warfare Support Battalion’s staffing, equipping, and training requirements prior to achieving full operational capability, the Army may be poorly positioned to make decisions about how to use and support the battalion. Army guidance states that the Army should assess its ability to support a new unit’s staffing, equipping, and training requirements, among other things, so that senior Army leaders can evaluate proposed organizational changes. For example, under a force integration functional area analysis, the Army staff evaluates all proposed organizational changes to ensure that they meet the intent of senior Army leaders, have the resources available to accomplish their mission, and that their projected benefits justify increased resources. These assessments analyze the proposed organization in nine areas, such as staffing, structuring, equipping, and training, and are intended to give senior Army leaders an understanding of whether the organizations are affordable, supportable, and sustainable. According to Army officials, the force integration functional area analysis is similar to a risk assessment. In addition, Standards for Internal Control in the Federal Government state that management should identify, analyze, and respond to the risks related to achieving the defined objective—in this case quickly fielding a cyber force to deal with current threats. Because the Army has not completely assessed the risk of organizing the ICEWS unit and the 915th Cyber Warfare Support Battalion, senior Army leaders may be left with an incomplete picture of the challenges in affording, supporting, and sustaining these units over the long term. Moreover, senior Army leaders lacked key information needed to understand the capability and capacity of the units at the time they were activated. For example, these units currently do not have what they need in terms of personnel and equipment to conduct their missions successfully. Further, according to some Army officials, without such an assessment, the Army does not know whether accelerated activation was the best course of action; what challenges they may face in staffing, equipping, and training the units; or how to mitigate challenges that may arise in other areas, such as deploying and sustainment. Army officials stated that there is a lot of informal discussion between relevant Army offices to try to identify and deal with challenges for these units. However, they also acknowledged the problems inherent in activating a unit by accelerating timelines. Such risk assessments also could inform future Army decisions as it activates new units for multi-domain operations. Given the Army’s perception of the threat environment, the Army may decide to activate other multi-domain operations units in an accelerated manner. For example, the Army is exploring ideas for creating several new units in future years to enhance its capability in multi-domain operations, such as a Theater Space Warfare Battalion. The Army also has been running wargames to see how they would operate new types of units at the division, corps, and theater level for commanding and operating long- range missiles and rockets. Army officials stated that as these units grow and evolve, it is uncertain when more comprehensive risk assessments would take place. If the Army does not perform a risk assessment for the activated ICEWS unit before it joins the larger Multi-Domain Task Force, or a more complete risk assessment for the 915th Cyber Warfare Support Battalion as that unit matures, the Army may end up fielding units that are not capable of providing the needed capabilities. Moreover, these risk assessments could provide vital lessons that could inform future Army decisions on the development, activation, and fielding of other units focused on enhancing the Army’s capability to conduct multi-domain operations. The Army engaged with the Joint Staff and other services to develop its Army Operating Concept and envisions opportunities for further coordination in the future. The Army’s overarching objective is to field a multi-domain-capable force by 2028, and it considers further engagement with the Joint Staff and other services as essential to accomplishing that goal. According to Army plans, the Army needs to finalize the next version of its Army Operating Concept by the fall of 2019 in order to incorporate multi-domain operations into all levels of Army leadership, training, and education by 2020. The Army plans indicate that maintaining this schedule is important to have a ready, lethal, and modern force for multi- domain operations by 2028. From the outset, the Army engaged with the Marine Corps to begin its concept development. Together the Army and Marine Corps published a white paper in January 2017 where they unveiled “Multi-Domain Battle” as a new concept for combat operations against a sophisticated great- power competitor. This white paper highlighted the need for ground forces to focus on all five warfighting domains and was intended as a first step toward further multi-domain concept development, wargaming, experimentation, and capability development. Once the white paper was written, the Army engaged with the Joint Staff and the other services in several ways to refine its concept: Joint Staff collaboration. The Army engaged with the Joint Staff on an Army-led study of recent contingency operations and used the lessons to refine the Army Operating Concept’s description of the emerging operational environment. Based on that study, the Army also refined some solutions for addressing threats posed by great- power competitors. Joint Staff officials reported that the Army engaged with the Joint Staff through other collaborative events as well, including tabletop exercises that tested and refined multi-domain concept ideas. Marine Corps collaboration. As the Army moved forward from the white paper, the Marine Corps’ input informed the concept’s development in various ways. This included changing the concept’s title from multi-domain battle to multi-domain operations in April 2018 to better reflect the scope of competition and conflict, as well as the inherent joint nature of modern warfare. The Marine Corps also hosted a multi-domain symposium in April 2018 that was attended by the Army, Air Force, Navy, and Joint Staff. Air Force collaboration. The Army initially collaborated with the Air Force Air Combat Command to inform concept-development efforts, and more recently began working with the Air Force Warfighting Integration Capability under Air Force headquarters. Also, the Army and Air Force collaborated on tabletop exercises focused on simulating multi-domain operations. Army officials told us that this helped them refine their thinking on how to enhance the maneuverability of its land forces by combining Army and Air Force capabilities across domains. Navy collaboration. The Army and Navy principally collaborated by testing multi-domain capabilities during real-world exercises. For example, the Army joined the Navy’s 2018 Naval Rim of the Pacific exercise to demonstrate capabilities for multi-domain operations in a real world environment. While the Army took steps to engage with the Joint Staff and the other services, it made the decision to move forward with the latest version of its Army Operating Concept in order to meet its overarching objective to develop a multi-domain operations-capable force by 2028. Given this urgency, Army officials told us that they may have missed opportunities to further refine its Army Operating Concept in 2018 with the perspectives of the Joint Staff and other military services. Joint Staff officials told us that by not fully including the Joint Staff in some tabletop exercises, the Army may have missed the Joint Staff’s perspective on key issues related to multi-domain operations, such as joint command and control. As the Army continues to revise its Army Operating Concept, the Army recognizes the need to continue to engage with the Joint Staff and other services. Joint Staff officials told us that the Joint Staff has initiated its own plans to engage with the services to refine key ideas of multi-domain operations in joint concepts, including logistics, intelligence, and command and control. Army officials told us that they recognize the importance of not getting too far ahead of these efforts, or the efforts of other services related to multi-domain operations. Army officials told us that the mechanisms built into the Joint concept-development framework would provide opportunities to engage the services and Joint Staff as the Army revises its own concept. Army officials added that beginning in the fall of 2019 the Army will participate with the Joint Staff in a wargame designed, in part, to analyze how the Army Operating Concept works with the other military service operating concepts. As a result, the current concepts are likely to evolve in the future as the Army synchronizes its efforts with those of the Joint Staff and other services. Rising threats posed by great-power competitors, particularly China and Russia, prompted the Army to initiate a profound and fundamental transformation to the way it plans to fight. The refinement of the Army’s Operating Concept is beginning to drive changes across the Army. The Army is making near-term changes by incorporating multi-domain operations into its doctrine, organizations, and training, which includes the accelerated creation of new cyber and electronic warfare units. However, these units are short of both people and equipment. While Army leadership believes that the urgency to confront threats justifies its decision to accelerate the development of those units, the Army did not assess the risks associated with staffing, equipping, and training its existing ICEWS unit prior to activation to determine whether it is affordable, supportable, and sustainable, and officials said it was uncertain when a more comprehensive assessment would take place. The Army plans to incorporate this unit into the first Multi-Domain Task Force by the end of Fiscal Year 2020, but in the meantime the unit could be deployed if needed. The Army did prepare a preliminary risk assessment for the 915th Cyber Warfare Support Battalion prior to activation, but it is unclear whether the Army will perform a more comprehensive risk assessment as the unit matures and nears full operational capability. For the units already activated, a risk assessment could benefit the Army by providing insights about the ability to deploy and sustain the units. It is important for the Army to assess its efforts before committing resources to activate new units. By formally assessing the risk of all new units activated in an accelerated manner, the Army will have the key information its leaders need for making decisions related to the activation of those units and other related units going forward. We are making the following three recommendations to the Secretary of the Army. The Secretary of the Army should ensure that the Deputy Chief of Staff, G-3/5/7 assess the risk associated with staffing, equipping, and training the existing ICEWS unit prior to its incorporation into the first Multi- Domain Task Force in fiscal year 2020. (Recommendation 1) The Secretary of the Army should ensure that the Deputy Chief of Staff, G-3/5/7 conduct a comprehensive risk assessment associated with staffing, equipping, and training the 915th Cyber Warfare Support Battalion prior to approving the expansion of the unit to its full operational capability. (Recommendation 2) The Secretary of the Army should ensure that the Deputy Chief of Staff, G-3/5/7 assess the risk associated with staffing, equipping, and training of new units that it plans to activate in an accelerated manner for the purposes of conducting multi-domain operations, taking into consideration the assessments performed on the first activated ICEWS battalion and the 915th Cyber Warfare Support Battalion. (Recommendation 3) We provided a draft of this report to DOD for review and comment. In its written comments, reproduced in appendix I, the Army partially concurred with the first two recommendations and concurred with the third recommendation. The Army partially concurred with the first recommendation for it to conduct a risk assessment, such as a force integration functional area analysis, for the first activated ICEWS unit. The Army stated in its comments that it does not perform force integration functional area analyses for experimental or pilot organizations, and that because the first ICEWS was activated as a pilot, no such assessment was performed. The Army added that it would conduct a risk assessment at the conclusion of the pilot if and when the Army decides to establish such a unit. We met with Army officials to discuss their comments, during which they provided additional information and clarification regarding how they were assessing risks for the unit. Based on this information, we modified the report to reflect the Army’s position that a risk assessment was unnecessary prior to activating the unit because the Army plans on using the pilot period to determine the staffing, equipping, and training requirements for the unit. We also incorporated additional information on the status of the ICEWS unit. As a result, we clarified our recommendation to state that the Army should assess the risk associated with staffing, equipping, and training the existing ICEWS unit prior to its incorporation into the first Multi-Domain Task Force in fiscal year 2020. Army officials generally agreed with the revised recommendation. Moving forward, it will be important for the Army to implement this recommendation to ensure the ICEWS unit, which is active and eligible to be deployed, will be prepared to carry out its mission effectively. The Army partially concurred with the second recommendation for it to conduct a risk assessment, such as a force integration functional area analysis, for the 915th Cyber Warfare Support Battalion. The Army stated in its comments that it does not perform force integration functional area analyses for force generating units such as the 915th Cyber Warfare Support Battalion. Instead, it develops a concept plan, which applies rigor and analysis to determine the most efficient and effective way of fielding a new unit. We met with Army officials to discuss their comments, during which they provided additional information related to assessing risks for the 915th Cyber Warfare Support Battalion. Specifically, Army officials said that prior to activating the battalion, leadership approved the battalion’s concept plan, which included an initial risk assessment. We reviewed the concept plan for the battalion and found that the assessment only addressed the risk of not having the unit’s capabilities activated and in the field for operations. We incorporated this additional information on this initial risk assessment for the 915th Cyber Warfare Support Battalion into the report. As a result of this additional information, we clarified our recommendation to state that the Army should conduct a comprehensive risk assessment associated with staffing, equipping, and training the 915th Cyber Warfare Support Battalion prior to approving the expansion of the unit to its full operational capability. Army officials generally agreed with this. It will be important for the Army to implement the revised recommendation to ensure the 915th Cyber Warfare Support Battalion, which is active and performing operations, will be prepared to carry out its mission effectively. The Army concurred with the third recommendation for it to ensure that a risk assessment is conducted before activating any new organizations it plans to field in an accelerated manner for the purposes of conducting multi-domain operations. The Army added that any lessons learned from the activation of the first ICEWS unit and the 915th Cyber Warfare Support Battalion will be taken into consideration when assessing the risk before the activation of these new organizations. It will be important for the Army to implement the recommendation to ensure that any new organizations are prepared to carry out their missions, while potentially avoiding some of the challenges that the ICEWS and 915th Cyber Warfare Support Battalion have experienced. Lastly, the Army also recommended that we change the title of our report; however, we did not accept the title offered by the Army. We believe the title accurately reflects the issues and recommendations highlighted in the report. We are sending copies of this report to the appropriate congressional committees and to the Secretary of Defense; the Acting Under Secretary of Defense for Personnel and Readiness; the Chairman of the Joint Chiefs of Staff; the Acting Secretaries of the Departments of the Air Force and the Army; the Secretary of the Navy; and the Chief of Staff of the Army. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3489 or pendletonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix II. In addition to the contact named above, Kevin O’Neill (Assistant Director), Matt Spiers (Analyst-in-Charge), Tracy Barnes, Shannon Finnegan, Christopher Gezon, Ruben Gzirian, J. Kristopher Keener, Alberto Leff, Joshua Leiling, Amie Lesser, Jon Ludwigson, Ned Malone, and Clarice Ransom made key contributions to this report.", "summary": "The rise of great-power competitors, such as China and Russia, prompted the Army to transform the way it plans to fight. The Army is developing a new warfighting concept to guide how its forces will engage jointly with other services in multiple domains, especially in cyber and space. The House Armed Services Committee included a provision in House Report 115-200 accompanying a bill for the National Defense Authorization Act for Fiscal Year 2018 for GAO to review the Army's implementation of the concept. Among its objectives, this report addresses (1) how the Army is changing its doctrine, organizations, and training in order to execute multi-domain operations; and (2) the extent to which the Army has established new cyber and electronic warfare units, including any challenges faced by these units, and whether the Army assessed risks associated with its plan to establish these units. GAO reviewed Army concepts, doctrine, force design, and training documents concerning multi-domain operations. GAO also interviewed Army and Department of Defense officials. The Army is changing aspects of its doctrine, organizations, and training to develop a force that can effectively engage great-power competitors—Russia and China—through multi-domain operations by 2028. Multi-domain operations present adversaries with multiple challenges across multiple domains (land, air, sea, cyber, and space) in contested environments. To this end, the Army is revising its doctrine to guide how the force and specific units will function. The Army is also reorganizing its force by creating new units to conduct missions in multiple domains and by updating the responsibilities of key Army formations, such as Army divisions. Also, the Army is training its combat forces for multi-domain operations in part by increasing the focus on cyber operations. The Army is establishing new cyber and electronic warfare units for multi-domain operations, but did not fully assess the risk of activating some units at an accelerated pace and is experiencing staffing, equipping, and training challenges. For example, the Army activated a cyber battalion in December 2018, and as of March 2019, this unit was understaffed by more than 80 percent. Army guidance directs the Army staff to conduct assessments on new units to determine whether the Army can staff, equip, and train these organizations. However, Army leadership believed the threats justify developing these units at an accelerated pace. Consequently, the Army did not assess the staffing, equipping, and training risk before activating one unit, and only conducted an initial risk assessment before activating a second unit. As a result, senior Army leaders may not know what other challenges could arise, such as sustainment, as the units grow in capabililty. Army officials told GAO that as these units evolve, it is uncertain when more comprehensive risk assessments would take place. The Army has previously accelerated the activations of other units when it saw fit to do so, and is considering creating other new units for multi-domain operations. If the Army does not assess risks for units activated at an accelerated pace, those units may be unable to effectively conduct multi-domain operations. GAO is making three recommendations, including that the Army comprehensively assess the risk of staffing, equipping, and training the cyber and electronic warfare units that it has activated at an accelerated pace, and to do so for new organizations it plans to activate in an accelerated manner for multi-domain operations. The Army concurred with one recommendation and partially concurred with two recommendations. GAO clarified the recommendations, as discussed in the report.", "document_type": "gao"}
{"report": "VHA recommends that all veterans who receive VHA services be screened for HIV as part of routine medical care, including those who do not think they are at risk for acquiring the virus. The aim is to ensure that veterans who are infected with the virus can be diagnosed as early as possible, receive life-saving care, and avoid passing the virus on to others. VHA has made earlier diagnosis of HIV a priority for the agency and established certain requirements for VAMC providers that aim to achieve early diagnoses and rapid linkages to HIV care for veterans. HIV screening at VAMCs involves three stages, and related VHA policy sets forth providers’ requirements related to each of these stages. (See fig 1.) Stage one: providing HIV tests to consenting veterans. A provider in a primary care clinic, a specialty care setting (such as an infectious disease clinic), or other outpatient setting (such as a women’s health clinic) offers a voluntary HIV test to an eligible veteran. In accordance with Centers for Disease Control and Prevention (CDC) recommendations, VHA policy requires providers to offer a one-time test to all veterans; annual tests to veterans with known higher risk factors for acquiring the virus, such as injection drug use; and tests every 3 months to veterans with known higher risk factors who are prescribed preventive medication known as pre-exposure prophylaxis (PrEP). Once a provider obtains consent from the veteran to be tested for HIV, the provider initiates an HIV test order with the laboratory. Although VHA policy previously required that providers document that they obtained veterans’ verbal consent to be tested for HIV, as of April 2019, providers must obtain, but no longer need to document, such consent. In addition, under VHA policy, providers must order the most current CDC- recommended HIV test (which detects HIV antigens and antibodies) when clinically indicated, and laboratories must follow the CDC-recommended HIV testing algorithm (see text box). A blood sample is collected from the veteran, and the laboratory processes the HIV test. Officials from the five selected VAMCs reported using information technology solutions and other strategies to facilitate each of the three stages of HIV screening: providing HIV tests to consenting veterans (stage one), communicating HIV test results to veterans (stage two), and linking HIV-positive veterans to care (stage three). Officials from multiple VAMCs in our review stated their providers use information technology solutions, such as clinical reminders, to fulfill their requirements related to the first stage of HIV screening: offering HIV tests to veterans, obtaining veterans’ verbal consent to be tested, and ordering the most current recommended HIV test. Offering HIV tests to veterans. Officials from three VAMCs in our review told us that providers often use clinical reminders that were developed and implemented by the VAMC or associated VISN to prompt them to offer HIV tests to veterans. (See fig. 2.) According to these officials, clinical reminders are used to prompt providers to offer a one-time HIV test to veterans who have not been tested. They can also be used to facilitate providers’ identification of veterans who are at higher risk for acquiring HIV and subsequently prompt them to offer these veterans an HIV test on an annual, rather than a one-time, basis. For example, officials at two of these three VAMCs indicated that the reminders include prompts for determining if veterans are at higher risk of acquiring HIV or fields to document identified risk factors. One of these officials told us that the recurrence of these clinical reminders can subsequently be increased or decreased to prompt providers to offer an HIV test to veterans who are at higher risk of acquiring HIV on a more or less frequent basis, depending on the risk factors identified over time. Obtaining veterans’ verbal consent to be tested. According to officials from the three VAMCs that discussed the use of clinical reminders, this technology prompts providers to obtain veterans’ verbal consent to be tested for HIV before ordering tests. Further, the reminders give providers a way to document that consent was obtained. For example, officials at one of the three VAMCs stated that providers can access the laboratory menu, which they use to order an HIV test, through the clinical reminder. The officials stated that providers must either (a) document that they obtained veterans’ verbal consent within the clinical reminder before accessing the menu, or (b) document that verbal consent was obtained once they have accessed the menu. Ordering recommended HIV tests. Officials from four of the VAMCs in our review reported that the facilities’ laboratory menus are designed to make it easier for providers to order the most current CDC-recommended HIV test. For example, officials from two VAMCs told us that the most current CDC-recommended HIV test is either the first result that appears when searching for an HIV test within the laboratory menu or the first HIV test that appears within a list of different types of HIV tests. According to officials from another VAMC, the facility’s laboratory menu includes a prompt that explains that an HIV viral load test (a test that is primarily used to monitor an active HIV infection) is not recommended solely to be used for diagnostic purposes if a provider attempts to order such a test for this purpose. Officials at each of the five VAMCs in our review told us that staff contact veterans to schedule non-routine, in-person appointments within the 7 day time frame to inform them that they have tested positive for HIV. According to officials at four VAMCs, staff first place phone calls to veterans and request that the veterans schedule face-to-face visits with providers. Officials at two VAMCs explained that providers attempt to inform veterans of positive HIV test results in person given the sensitive nature of the diagnosis, as recommended by VHA policy. If staff cannot reach the veterans by phone, officials at these two VAMCs indicated that they send letters to the veterans asking them to contact their providers to obtain their test results. Further, officials at three VAMCs stated that staff send letters to veterans to inform them of negative HIV test results within the required 14 day time frame. Officials from all five VAMCs in our review also reported using various, additional approaches to communicating negative HIV test results to veterans, including notifying them by phone, informing them of test results during face-to-face visits, or uploading test results into veterans’ personal electronic health records (EHR). In addition, all five VAMCs in our review have developed protocols to prevent delays in the communication of positive HIV test results to veterans when the provider who ordered the test is unavailable. These protocols are generally outlined in facility-specific policies, which we reviewed, that require that a designee communicate positive HIV test results to veterans in lieu of the ordering provider. According to officials at three VAMCs, these protocols apply when the ordering provider is unavailable for a certain number of consecutive days (typically 3 days). Officials told us that if the designee is not available, their facility’s protocol requires that VAMC leadership (such as the Chief of Medicine) communicate the results to the veteran. Officials from all five VAMCs in our review indicated that providers may refer eligible HIV-positive veterans to care within the community to ensure that treatment occurs in a timely manner. According to officials at two of these VAMCs, these referrals are often made based upon veterans’ preferences or primary care providers’ comfort levels in providing HIV care to veterans who are also eligible for community care. An official at another of these VAMCs told us that eligible veterans who live further distances from the VAMC may ask to be referred to community care. According to officials from multiple VAMCs in our review, providers may also use telecommunications to provide HIV care to veterans. For example, officials from two VAMCs told us that their facilities offer telehealth consultations with an infectious disease provider to veterans who live outside the city in which the VAMC is located or who otherwise find it inconvenient to be seen in-person by an infectious disease provider at the facility. Telehealth allows infectious disease providers to care for veterans who would otherwise receive HIV care from primary care providers or in the community. Officials at another VAMC reported that infectious disease providers are available via cell phone or Skype (software that can be used to make one-to-one or group voice or video- based calls from a cell phone or computer) to assist primary care providers who assume responsibility for veterans’ HIV care. VHA facilitates monitoring of the first stage of HIV screening by providing information to VAMCs that include data on the number of veterans who have been tested for the viral infection. While VHA does not collect data on the timeliness with which HIV test results are communicated to veterans, data resulting from VHA’s monitoring of the communication of other test results may indicate whether veterans are informed of HIV test results within recommended time frames. However, HIV lead clinicians may not be aware that they have access to this information. VHA does not currently monitor whether veterans who test positive for HIV are linked to care within recommended time frames; however, VHA has taken steps to collect and disseminate data that can be used to monitor this stage of screening. According to HHRC officials, the office collects and disseminates annual and biannual data to each VAMC’s HIV lead clinician on the offering of HIV tests to veterans. (See table 2 for information related to VHA’s monitoring activities.) This includes data on (1) the number of veterans who are eligible to receive one-time HIV tests, as well as the number of eligible veterans who were tested, for each VAMC and VISN; and (2) the number of veterans who are prescribed PrEP who are tested for HIV every 3 months to document that they are still HIV negative as recommended by the CDC. HHRC officials told us that they share the one-time testing rate data with HIV lead clinicians on an annual basis, and that these clinicians can use the data to calculate their VAMCs’ one- time HIV test rates and, subsequently, compare their rates regionally or to VAMCs that offer the same complexity of services. According to HHRC officials, they upload these data to an internal data sharing website and notify HIV lead clinicians that the data are available via email and during regularly scheduled conference calls that facilitate the discussion of issues related to HIV screening. HHRC officials also told us that VHA uses the same method to share with HIV lead clinicians on a biannual basis data on the HIV test rate for veterans who are prescribed PrEP. VISNs and VAMCs have used VHA’s data on the offering of HIV tests to veterans to support local efforts to improve HIV screening. For example, HHRC officials told us that VISNs have used data on the number of veterans who are eligible to receive one-time HIV tests, and who were tested, to support applications for VHA-sponsored grants intended to improve the offering of such tests to homeless veterans. Officials from four VAMCs in our review told us that they have used these data to identify the need to increase testing, which led to the implementation of new strategies, such as clinical reminders that prompt providers to offer one-time and risk-based HIV tests to veterans. While VHA recently monitored the documentation of verbal consent by collecting data that VAMCs used to make related improvements, such monitoring is no longer needed due to a change in VHA policy. Between fiscal years 2013 and 2016, NCEHC (the VHA office responsible for VHA’s policy on informed consent) oversaw a system-wide review that led to improvements in the number of VAMC providers that documented in veterans’ medical records that they obtained veterans’ verbal consent to be tested for HIV. In 2019, VHA amended its policy and no longer requires providers to document that they obtained verbal consent. In addition, VHA recently monitored VAMC laboratory protocols for HIV testing, but HHRC noted that this monitoring is no longer needed, because the recommended testing technologies have been implemented. In 2018, VHA conducted a one-time review of VAMC laboratory protocols to ensure that CDC recommendations for the use of HIV tests were followed at each VAMC, such as recommendations related to the type of HIV test that providers should order for diagnostic purposes. VAMCs were required to submit verification to VHA showing that their laboratories had implemented the most current CDC-recommended testing technologies. According to HHRC officials, this provided assurance that providers were ordering the most current CDC-recommended HIV test and that laboratories were following the CDC-recommended HIV testing algorithm. VHA’s Director of Pathology and Laboratory Medicine Service reviewed the verification submitted by each VAMC, and VAMCs were required to develop action plans to address any identified deficiencies. As of August 7, 2018, VHA found that all VAMCs were following CDC’s recommendations related to the availability and use of HIV tests. According to HHRC officials, VHA does not need to continue its monitoring effort in this area, since the implementation of recommended testing technologies by VAMCs was a one-time effort. Further, officials from the five VAMCs in our review told us that the VAMCs were using the CDC-recommended HIV test, and nothing inconsistent came to our attention during our medical records review. OPC and RAPID (the VHA offices responsible for VHA’s policy on the communication of test results and related performance measurement) make data available to VAMC staff that may indicate the timeliness with which HIV test results are communicated to veterans. OPC and RAPID publish a quarterly report on the timeliness with which results from the eight tests that are included in its review of veterans’ medical records are communicated to veterans at each VAMC. While HIV tests are not one of the eight tests included in the OPC and RAPID review, VAMC officials we interviewed told us that VAMC procedures for communicating results are generally the same for all tests. OPC officials stated that VAMC officials could use the data to identify needed performance improvement efforts related to the communication of test results. OPC officials added that while it is not the primary goal of the OPC and RAPID review, data on the eight tests included in the review may serve as a sample, providing some indication as to whether VAMC procedures promote the timely communication of results of any test to veterans. Although OPC and RAPID publish a quarterly report on the timeliness of communicating test results, HIV lead clinicians may not be aware they have access to this information. OPC and RAPID officials told us that VAMC staff responsible for serving as liaisons for OPC’s medical records review are notified by RAPID via email of the report’s availability. RAPID officials added that any VAMC staff may opt in to the email group that officials use to notify liaisons that the timeliness data have been published. HIV lead clinicians we interviewed reported that they did not know that they can opt in to this email group. According to RAPID officials, the main mechanism for making VAMC staff aware that they can join this email group is through their VAMC colleagues. VHA has not taken steps to more systematically communicate the availability of these timeliness data to all VAMC staff (including HIV lead clinicians). Standards for internal control in the federal government require that agencies communicate necessary information throughout all agency reporting lines to achieve the agencies’ objectives and respond to identified risk. VHA policy requires that HIV lead clinicians serve as VAMC points of contact on HIV testing, diagnosis, and care, which may include monitoring HIV care. An HIV lead clinician we interviewed also noted that these data could be used as an indicator as to whether HIV test results are being communicated to veterans in a timely manner. Further, having these data could help staff determine if delays in communicating test results pose risks to the timely completion of HIV screening, such as whether veterans who test positive for HIV are linked to care for their diagnosis as expeditiously as possible. If there are unnecessary delays in communicating positive HIV test results to veterans, providers may be at risk of delaying the start of needed HIV treatment. According to VHA policy, and confirmed by RAPID officials, the timely communication of test results to veterans is essential for high quality care, and the timely follow- up of positive test results may help veterans achieve favorable health outcomes. Linking Veterans to Preventive Care for Human Immunodeficiency Virus (HIV) In addition to linking veterans who test positive for HIV to care for their diagnosis, Department of Veterans Affairs (VA) medical centers link veterans who test negative for HIV to preventive care. The use of preventive medication, or pre-exposure prophylaxis (PrEP), reduces the risk of acquiring HIV in adults. Officials from VA’s HIV, Hepatitis, and Related Conditions Programs (HHRC) told us that they implemented a PrEP quality improvement initiative in September 2016, which focuses on increasing the use of PrEP among veterans who live in areas of the country with a higher prevalence of HIV compared to the national average. HHRC officials told us that the initiative focuses on providing high quality care to veterans in accordance with current recommendations on the use of PrEP. For example, the Centers for Disease Control and Prevention (CDC) has recommended that providers prescribe PrEP medications to individuals who test negative for HIV within one week of documenting the test result. HHRC officials told us that they monitor the time frames in which veterans are prescribed PrEP medication by collecting data on a biannual basis on the date on which veterans’ blood was drawn for the purposes of conducting an HIV test and the date on which veterans’ were prescribed the medication. HHRC officials told us that these data are disseminated to VA medical center staff responsible for improving HIV screening to improve the appropriate use of PrEP as needed. source of information to determine whether veterans are linked to care specifically for their HIV diagnosis within the recommended time frame. According to officials, the data tool was implemented in October 2018, and as of early November 2019, they were in the process of building the capacity to generate a report based on these data showing the time frames in which veterans are linked to HIV care. HHRC officials initially indicated that they expected to begin monitoring linkage to HIV care in August or September 2019, but they were not able to do so for various reasons. According to HHRC officials, the process of building the new data tool and the capacity to generate a report has been lengthy due to competing priorities related to VHA’s ongoing development of a new EHR system. These officials added that they have been simultaneously focused on implementing required improvements in the diagnosis and treatment of veterans with Hepatitis C. According to officials, the time frame to develop the new data tool and report has been extended due to these competing priorities. HHRC officials told us that once monitoring begins, they will report on the number of veterans who are linked to HIV care within the recommended 30-day time frame for each VAMC on an annual basis, retroactive to fiscal year 2018. According to HHRC officials, the data will be disseminated by publishing them on an internal data sharing website that each VAMC’s HIV lead clinician can access. The officials explained that these clinicians will be notified when the data have been published via email and during regularly scheduled conference calls with HHRC. HHRC officials also told us that the data may be used to inform any needed improvements in the timeliness of linking newly diagnosed veterans to HIV care. Standards for internal control in the federal government require that agencies perform ongoing monitoring activities and evaluate results to remediate any identified deficiencies on a timely basis. VHA policy requires that HHRC develop data reports for monitoring the quality of HIV care that are to be disseminated to the VISNs or VAMCs, among other entities and individuals, and lead VHA efforts toward meeting the NHAS’s recommendations. However, until HHRC disseminates data on the timeliness with which veterans are linked to HIV care, VAMCs are limited in their ability to identify any delays and take the necessary steps to ensure that this occurs within recommended time frames, now and in the future. In our nongeneralizable review of the 38 medical records for veterans who tested positive for HIV, we observed some instances of delay. Specifically, we found that six veterans were first seen by an infectious disease provider, who typically treats HIV, more than 30 days after being informed of their positive test results. We were unable to identify a documented explanation in the six medical records for why linkages to care exceeded 30 days. Delays in linking veterans to HIV care can increase the risk that veterans are not promptly beginning treatment to help achieve favorable health outcomes. According to the 2015 NHAS, evidence shows that earlier treatment reduces the risk that an individual with HIV will develop AIDS or transmit the virus to others. Veterans who are voluntarily tested for HIV at VAMCs, informed of positive HIV test results in a timely manner, and expeditiously linked to care before their infections progress further have improved health outcomes, a longer life expectancy, and a reduced risk of transmitting the virus to, for example, a sexual partner. VHA has monitored the provision of HIV tests to veterans and reported related improvements resulting from these monitoring efforts, ensuring that, for example, veterans are receiving the most current CDC-recommended test. However, VHA’s dissemination of data on the time frames in which test results are communicated to veterans and monitoring of the time frames in which HIV-positive veterans are linked to care specific to their diagnosis needs improvement. We are making the following two recommendations to VA: The Under Secretary for Health should take steps to improve communication to VAMC staff (including HIV lead clinicians) about the availability of data on the time frames in which test results are communicated to veterans. (Recommendation 1) The Under Secretary for Health should disseminate data to HIV lead clinicians on the extent to which veterans who test positive for HIV are linked to care within recommended time frames. (Recommendation 2) We provided a draft of this report to VA for review and comment. In its written comments, which are reproduced in appendix I, VA concurred with our recommendations. VA stated that it will communicate to VAMC staff, including HIV lead clinicians, how providers may be notified when the data on the time frames in which test results are communicated to veterans have been published. Further, VA stated that HIV test results will be added to the OPC and RAPID quarterly review of such time frames beginning in the second quarter of fiscal year 2020. VA also indicated that as of December 2019, the agency began annual monitoring of whether veterans are linked to HIV care within recommended time frames and will notify HIV lead clinicians of the availability of the data during conference calls scheduled to take place in January and March 2020. We are sending copies of this report to the appropriate congressional committees and the Secretary of Veterans Affairs. In addition, this report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at DraperD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Hernán Bozzolo (Assistant Director), Karen Belli (Analyst-in-Charge), Hannah Grow, Cathy Hamann, and Tatyana Walker made key contributions to this report. Also contributing were Jacquelyn Hamilton, Diona Martyn, and Vikki Porter.", "summary": "VHA is the largest single provider of medical care to HIV infected individuals in the nation. In 2018, VAMCs tested approximately 240,000 veterans for HIV and provided HIV care to over 31,000 veterans. Early diagnosis and timely treatment is important for achieving favorable health outcomes and reducing the risk of transmitting the virus to others. The accompanying Joint Explanatory Statement for the Consolidated Appropriations Act, 2018 included a provision for GAO to examine how VAMCs have implemented VHA's HIV screening policy. This report examines (1) approaches that selected VAMCs use to facilitate HIV screening, and (2) the extent to which VHA monitors HIV screening. GAO analyzed VHA documents, including VHA directives and a nongeneralizable sample of 103 veterans' medical records, to understand how providers made decisions and documented actions related to HIV screening. GAO also interviewed VHA and VAMC officials, the latter from five facilities selected based on factors such as the range of HIV prevalence rates. Officials from five selected Department of Veterans Affairs (VA) medical centers (VAMC) reported using various approaches to facilitate human immunodeficiency virus (HIV) screening, which involves three stages. For example, for the first stage of HIV screening (providing HIV tests to consenting veterans), officials told GAO that VAMCs use information technology solutions, such as clinical reminders that prompt providers to offer HIV tests to veterans who have not been tested. These clinical reminders can also prompt providers to offer an HIV test on a repeated, rather than a one-time, basis to veterans with known higher risk factors for acquiring HIV. The Veterans Health Administration (VHA) monitors the first stage of HIV screening by collecting and disseminating data that VAMCs can use to calculate and, if necessary, improve facility HIV testing rates. VHA also collects data on the time frames in which results for eight types of tests are communicated to veterans; these data could indicate how timely test results are being communicated generally (stage two of HIV screening). However, VHA has not effectively communicated the availability of these data to HIV lead clinicians. In addition, VHA does not currently monitor whether VAMCs link veterans who test positive for HIV to care in a timely manner (stage three of HIV screening). VHA officials indicated that they are in the process of building the capacity to collect and disseminate to HIV lead clinicians data on the number of veterans at each VAMC who are linked to HIV care within 30 days, as recommended. However, the time frames for completing these efforts have been extended due to competing priorities, such as implementing required improvements in the diagnosis and treatment of veterans with Hepatitis C. Until VHA improves VAMC staff's access to, or provides them with, these data, it increases its risk that HIV-positive veterans do not receive timely treatment. Such treatment can improve veterans' health outcomes and prevent the transmission of the virus to others. VA should (1) improve communication regarding the availability of data on the timeliness with which test results are communicated to veterans, and (2) disseminate data to HIV lead clinicians on the timeliness with which veterans are linked to HIV care. VA concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Federal agencies can respond to a disaster when effective response and recovery are beyond the capabilities of affected state and local governments. In such cases, the President can declare a major disaster in response to a request by the governor of a state or territory or the chief executive of a tribal government. The SBA Administrator also can issue an agency physical disaster declaration for events that do not rise to the level of a major Presidential disaster declaration in response to a timely request by a state governor. The National Response Framework governs any type of federal disaster or emergency response. Under this framework, FEMA has lead responsibility and offers disaster assistance. At least 30 other federal agencies, including SBA, also administer disaster assistance programs and activities. The Small Business Act authorizes SBA to make direct loans to help businesses, nonprofit organizations, homeowners, and renters repair or replace property damaged or destroyed in a declared disaster (declared by the President or SBA). ODA is responsible for administering the SBA disaster loan program, primarily through the following offices: Customer Service Center is a single nationwide point of contact for disaster survivors who have questions about SBA disaster loans. It provides a call center, email response, disaster application mailings, and pre-application entry. Field Operations Centers coordinate disaster field operations and publicize ODA’s Disaster Loan Program before and after disasters. Field Operations Centers establish, staff, and maintain field operations in declared disaster areas, including Disaster Recovery Centers, Business Recovery Centers, and Disaster Loan Outreach Centers. Processing and Disbursement Center screens all applications, reviews and processes those that are complete, closes all approved loans, and disburses loan proceeds. In addition, ODA works with resource partners, such as SBDCs, to provide disaster assistance to businesses. SBDCs help SBA by conducting local outreach to disaster victims; assisting those who have had their business loan application denied or withdrawn with applications for reconsideration or re-acceptance; assisting declined applicants in remedying issues that initially precluded loan approvals; and providing business loan applicants with technical assistance, including to reconstruct business records, to better understand requirements to complete a loan application, and for compiling financial statements and collecting required documents. The Small Business Act authorizes SBA to make available two different types of direct disaster loans to survivors located in a declared disaster area (see table 1): Physical disaster loans are for the permanent rebuilding and replacement of uninsured or underinsured disaster-damaged property. They are available to homeowners, renters, most types of businesses regardless of size, and nonprofit organizations. Businesses in agriculture-related industries—also known as agricultural enterprises—are not eligible. Economic injury disaster loans help meet working capital needs (until normal operations resume) for most disrupted small businesses and private nonprofit organizations that utilized all reasonable available funds and were not able to obtain credit elsewhere after a disaster declaration. The loans cover operating expenses the businesses could have paid had the disaster not occurred. Business eligibility for the loans also differs. Certain businesses of all sizes are eligible for physical disaster loans, but only small businesses are eligible for economic injury disaster loans. Small businesses that did not sustain physical damage from a disaster cannot apply for physical disaster business loans but can apply for economic injury disaster loans. And, applicants seeking both types of loans may be approved for one type of disaster loan but denied for the other. SBA procedures generally require that applications for a physical disaster loan be submitted no later than 60 days following the disaster declaration and no later than 9 months after this date for an economic injury disaster loan. SBA may authorize an extension of the filing period, and did so for each of the 2017 hurricanes. Applicants can apply for a loan (1) at a FEMA Disaster Recovery Center, (2) at a Business Recovery Center or SBA Disaster Loan Outreach Center, (3) by mail, or (4) online through SBA’s electronic loan application system. SBA requires applicants seeking any disaster loan to submit documents with their application package, including permission for SBA to access applicants’ tax return information. SBA’s regulations contain underwriting criteria that require reasonable assurance of repayment. The regulations state that SBA must have reasonable assurance that all disaster loan applicants can repay their loans based on SBA’s analysis of the applicants’ credit or personal or business cash flow. Processing of disaster loan applications involves several stages (see fig. 1). For example, loss verifiers conduct desktop or on-site inspections for physical disaster loan applications to estimate the cost of restoring damaged property to predisaster condition. The verified loss becomes the basis for the loan amount. Loan officers determine eligibility during processing, taking into consideration any insurance or other recoveries, and make the decision to approve or decline an application. Case managers work with borrowers through the disbursement process (until the loan is fully disbursed). To plan for its response to Presidentially declared disasters, SBA maintains long-term disaster planning documentation, conducts disaster simulations, maintains an outreach plan, and creates action plans for specific disasters. Disaster planning documents. SBA’s major planning documents for disasters include its Disaster Preparedness and Recovery Plan and its Disaster Playbook. The Disaster Preparedness and Recovery Plan outlines SBA’s responsibilities as part of federal disaster response efforts. The plan is intended to ensure a broad scope of coordination, awareness, and support throughout the organization, and describes how SBA conducts its disaster-related missions. The plan also includes appendixes describing different aspects of SBA’s disaster response operations, including SBA’s forecasting and modelling to predict internal resource requirements such as staff. The document is revised annually. The Disaster Playbook outlines the roles and responsibilities of ODA departments, resource partners, and other private-sector partners at each major phase of the disaster recovery process. For example, it describes the steps taken by each ODA department to receive and process incoming disaster loan applications. According to SBA officials, the Disaster Playbook that guided the disaster response for the 2017 hurricanes was issued in 2014. Disaster simulations. SBA also participates in external and internal disaster simulation exercises. SBA participates in the FEMA-led National Exercise Program, which is a 2-year cycle of exercises that examine and validate capabilities in the National Preparedness System mission areas. SBA’s internal disaster simulations occur at SBA’s biennial Senior Leadership Summit. SBA is required by law to conduct an internal disaster simulation exercise at least once every 2 fiscal years. The exercises are designed to improve understanding of preparedness concepts, identify opportunities or problems in SBA’s response to disasters, and discuss solutions for improvement. Outreach plan. SBA maintains a marketing and outreach plan with the goal of continuing and strengthening lines of communication with stakeholders to build strong and productive partnerships. The plan describes how SBA promotes awareness of the risk of natural disasters, the need to be prepared, and SBA’s role in disaster recovery. The plan also outlines areas of responsibility, disaster event scenarios, strategies for achieving its outreach goals, and implementation of outreach efforts. According to the plan, implementation of outreach efforts should include distribution of SBA disaster information products by public information officers and SBA’s resource partners, including SBDCs. Action plans. SBA Field Operations Centers create disaster-specific action plans to guide disaster responses. According to officials from the centers, action plans outline projected resource requirements, such as staff, and may include descriptions of the impending disaster scenario, SBA leadership responsibilities, and operational support requirements. SBA has designated response levels that correlate staffing levels with anticipated numbers of applications. SBA has four response levels based on the number of loan applications it expects to receive (see fig. 2). Each level has different goals for staffing and processing applications. The 2017 hurricanes triggered a level III response from SBA. SBA established the Express Bridge Loan Pilot Program to supplement the agency’s disaster response capabilities. The pilot program became available for use on October 16, 2017, and is set to expire on September 30, 2020. The pilot allows 7(a) lenders with SBA Express lending authority to deliver SBA-guaranteed financing for disaster-related purposes to eligible small businesses on an emergency basis while the businesses apply for and await long-term financing. The businesses must be located in, or contiguous to, communities with Presidentially declared disasters and have a pre-existing banking relationship with the 7(a) lender. Loan applications are subject to a streamlined underwriting process to ensure that small business owners receive a quick decision. SBA’s major disaster planning documentation lacks an in-depth discussion of risks SBA could face when responding to disasters. The Disaster Preparedness and Recovery Plan states that SBA must assess risk from two perspectives. First, it must address risk to its own resources and capabilities through its employee safety and continuity of operations plans to preserve its mission-essential functions, including the Disaster Loan Program. Second, SBA must evaluate the demands various disaster scenarios can place on the agency regardless of the impact to its own assets. However, the Disaster Preparedness and Recovery Plan does not include a description of potential risks that may prevent SBA from successfully executing the Disaster Loan Program or discuss how SBA is to conduct risk analysis. Furthermore, SBA’s Disaster Playbook does not include any mention of risk and risk-mitigation techniques. Federal internal control standards state that management should identify, analyze, and respond to risks related to achieving defined objectives. For example, the types of challenges SBA faced in executing its response to Hurricane Maria in Puerto Rico and the U.S. Virgin Islands represent potential risks to the program because they could compromise SBA’s ability to successfully execute the program. These challenges included Loss of electricity: Electrical outages restricted SBA’s ability to quickly establish operations and help survivors access disaster loans. According to SBA officials, ODA established its base of operations at the SBA District Office in San Juan (which had generator power) because they had difficulty locating other usable office space due to power outages. In a report issued after the 2017 hurricanes, SBA noted that periodic outages also prevented applicant access to the online Disaster Loan Application Portal. Other infrastructure damage: Physical infrastructure damage restricted SBA’s mobility around Puerto Rico and the U.S. Virgin Islands following Hurricane Maria, delaying SBA’s execution of the Disaster Loan Program. According to SBA officials, there were no flights into or out of the islands, and driving was difficult due to damage to and debris on the roads. No or intermittent communications services: According to SBA officials, telephone lines were down and cellular phone service was intermittent in Puerto Rico immediately after Hurricane Maria, which made communicating with staff and disaster survivors more difficult. We present applicant, other stakeholder, and SBA perspectives on these and other challenges later in this report. SBA officials told us that the agency contracts with an outside firm to conduct monthly portfolio analysis reports. Officials also told us that they use this report to identify areas of credit risk and make adjustments to program parameters, such as minimum acceptable credit scores. However, officials told us that they do not formally document other types of risks, including operational risks, or the steps taken to mitigate these risks. For example, SBA’s major planning documentation does not include any discussion of steps taken by ODA to mitigate the risk to the program posed by the major infrastructure damage that occurred in Puerto Rico after Hurricane Maria. Without identifying risk elements associated with its disaster response and documenting how it plans to mitigate these risks, SBA may not be adequately prepared to respond to challenges that arise during its disaster response efforts. The action plans created for the 2017 hurricanes contained varied information and in some cases did not discuss certain resource requirements for responding to the disaster, such as equipment needs. For example, the Hurricane Harvey action plan is less detailed than the plans for Hurricanes Irma and Maria and only discusses the anticipated staffing needs and strategies to meet those needs. The action plans for Hurricanes Irma and Maria identified additional resource needs and issues unique to a particular location. FEMA provides guidance on developing disaster plans for local, state, and federal government planners to guide their on-site response to disasters. These plans should focus on managing personnel, equipment, and resources that play a direct role in the response. The plans are to define how specific actions will be performed to achieve a planned outcome and include the “who, what, where, and when” in describing deployment and direction of resources. SBA officials told us they created a template to help in estimating staffing needs, but SBA has not identified whether any other elements, such as those defined by FEMA, should be included in these plans. Moreover, SBA has not provided guidance to the Field Operations Centers that is specific to preparing action plans for impending disasters. In addition, SBA’s major disaster planning documents, the Disaster Preparedness and Recovery Plan and the Disaster Playbook, do not mention action plans. According to SBA officials, these plans are internal documents, and the template and the Field Operations Center director’s prior experience creating action plans are sufficient guidance. Without identifying the key elements of a disaster action plan and providing additional guidance to staff on how to incorporate these elements in future action plans, SBA’s Field Operations Centers may miss opportunities to better tailor their response to individual disasters, decreasing the effectiveness of their responses. As discussed previously, SBA participates in external and internal disaster simulation exercises to help it prepare for disasters. For instance, SBA participated in the 2019 National Level Exercise, which simulated the interagency response to a large earthquake in Tennessee. Prior to the 2017 hurricanes, SBA conducted an internal disaster simulation at its biennial leadership seminar in 2016. SBA simulated a series of three progressively more challenging events, including a simulated earthquake that, according to SBA officials, took out power to two major West Coast cities. According to SBA officials, this exercise helped them prepare for the widespread power outages encountered during the disaster response in Puerto Rico after Hurricane Maria. SBA officials told us that after an internal simulation, SBA prepares a report that summarizes the events and recommends disaster response improvements. For example, after the 2016 simulation, SBA determined it did not currently have the resources to adequately respond to a large event. Suggested improvements to increase SBA’s response capacity included establishing timelines for screening and training of new hires. In response to the 2017 hurricanes, SBA forecasted the need for additional staff and hired staff as outlined in its Staffing Strategy. SBA also encountered several challenges associated with its hiring and training processes. The Staffing Strategy used by SBA in its response to the 2017 hurricanes describes the types of staff SBA hires for disaster response, how SBA determines and manages staffing levels, and factors to consider when hiring new or returning staff. For example, the guidance describes business needs SBA should consider when making hiring decisions, such as workload, special skills required, and cost-saving measures. Before an impending disaster, SBA uses a model to forecast staffing levels. According to SBA officials, the model helps identify the expected number of staff needed in each ODA office and peak need based on the anticipated number of applications received. The officials noted that predicted applications typically peak about 3–4 weeks into the disaster response. Key assumptions and inputs to the model include the target application review and decision time frame; the requirements for specialized staff skills such as loss verification, loan processing, and legal review in the application process; staff productivity and training requirements; the total expected loan volume; and the type of disaster. As shown in figure 3, SBA’s actual staffing lagged behind its predicted levels for the first 4 months following the 2017 hurricanes. Officials told us that SBA encountered challenges quickly hiring and getting new staff on board immediately following the 2017 hurricanes. Due to the rapid need for qualified staff to respond to the 2017 hurricanes, SBA faced a variety of self-identified challenges in hiring the staff. Recruitment: ODA lacked a national recruitment strategy and vehicle for advertising vacant positions. In addition, SBA processed resumes manually, without the ability to effectively match resumes with required skillsets. Due in part to its manual processes, it took SBA time to hire qualified human resources specialists to process the large number of applicants for positions, which affected the ability to hire and deploy new staff to disaster areas. Hiring: Because of the size of the 2017 hurricanes, ODA initially was unable to effectively support staffing needs of centers. In addition, SBA officials said that the agency had to hire Puerto Rican attorneys because secured loans must be signed by a notary and in Puerto Rico, notaries must be lawyers. “On-boarding”: ODA’s on-boarding processes were manual, heavily paper-based, and inefficient and caused delays in processing new staff. In addition, not enough time was devoted to the on-boarding process, and delays, disruptions, and unforeseen issues routinely extended the process. Training: New staff required significant training that SBA did not have time to provide. Shortened or omitted training created a greater need to provide on-the-job training. In response to these challenges, SBA’s after-action report for the 2017 hurricanes and its 2018 Staffing Strategy made recommendations to improve SBA’s hiring processes. These recommendations included establishing regular intermittent postings; creating an enhanced recruitment toolkit that includes social media, advertisement, third-party partnerships, and an outreach plan; establishing interagency agreements for detailee staff; developing a template for deployment of staff; reviewing and updating position descriptions; building an applicant pipeline; developing an on-boarding plan; and implementing an integrated technology solution that includes recruitment, hiring, on-boarding, benefits, training, performance management, and “off-boarding.” SBA officials told us they have begun implementing these recommendations. For example, the agency consolidated its hiring processes for new staff through USA Staffing, a federal online recruitment, evaluation, and hiring system. SBA establishes outreach locations for businesses and individual disaster survivors, including Business Recovery Centers, in disaster areas. The SBA OIG noted that while the centers are designed to assist business owners, staff also were available to assist homeowners and renters. Based on our analysis of SBA data, SBA operated six such centers in response to Hurricane Harvey, 18 in response to Hurricane Irma, and 85 in response to Hurricane Maria. According to SBA officials, the number of Business Recovery Centers opened is dependent on two key data points—the number of declared counties with the greatest impact from the disaster and the business and population density. Officials told us that SBA opened the most centers in response to Hurricane Maria in Puerto Rico because of the high number of affected municipalities. Staffing levels at the centers are based on the number of businesses categorized as major and on population density, and then refined based on actual daily activity at the center. See figure 4 for a photograph of SBA staff at a center in a municipal sports complex in Bayamón, Puerto Rico. In some locations we visited, SBA worked with its district office or local SBDCs to set up and staff Business Recovery Centers. For example, the Field Operations Center–East and the Florida SBDC Network created a staffing and deployment plan that outlined center locations in the Florida counties most affected by Hurricane Irma. The Florida centers were jointly staffed by SBA and SBDC employees, and included a mobile center. According to SBA officials, SBA and SBDC representatives drove the mobile center to locations without a dedicated Business Recovery Center, such as the Florida Keys. Similarly, the Field Operations Center–East created a staffing and deployment strategy for Business Recovery Centers with SBA’s District Office in Puerto Rico. They established fixed and mobile centers and each week staff traveled to multiple municipalities without fixed centers. The SBA District Office coordinated with local mayors and other officials to select the locations and ensure the public was informed. SBA officials noted that staff visited all the Puerto Rican municipalities over about 7 months. SBA’s outreach for the 2017 hurricanes included local partners, generally was favorably regarded by stakeholders we interviewed, and met statutory requirements. But SBA’s current guidance on outreach efforts does not incorporate region-specific risks (such as those encountered after the 2017 hurricanes), and SBA has not evaluated its efforts. According to SBA officials, SBA relied on public information officers, staff at recovery centers, local partners such as SBDCs and local government, and media contacts to disseminate information about the Disaster Loan Program to disaster survivors. When communicating with disaster survivors was difficult immediately after Hurricane Maria struck Puerto Rico, SBA officials told us they worked with the only operational radio station on the island to broadcast information about the program. During our site visits to selected areas in the states and territories most affected by the 2017 hurricanes, stakeholders—including business owners and local SBA partners—told us they generally were satisfied with SBA’s outreach efforts to disaster survivors. Officials at one SBDC told us that they believed SBA public information officers did a good job reaching out to local businesses and keeping the SBDC informed about upcoming SBA disaster loan presentations and media outreach and went door-to- door to reach small businesses. Similarly, officials at an SBDC in Puerto Rico told us that SBA staff visited all the municipalities in their region, and that they participated in presentations about disaster assistance with SBA and FEMA. Nine of the 24 business owners with whom we talked told us they first learned about SBA’s disaster loans from prior experience with the loans or through the agency’s outreach efforts (presented online or on other media). As discussed in more detail below, SBA does not have metrics that allow it to assess the effectiveness of its outreach efforts. Outreach materials SBA provided to us contained required statutory elements. By statute, if a disaster is declared, SBA must make “every effort to communicate through radio, television, print and web-based outlets, all relevant information needed by disaster loan applicants.” SBA must include (1) the date of the declaration; (2) cities and towns in the areas of the declaration; (3) loan application deadlines related to the disaster; (4) all relevant contact information for victim services available through SBA (including links to SBDC websites); (5) links to relevant state and federal disaster assistance websites, including links that provide information on assistance available through FEMA; (6) information on eligibility criteria for SBA loan programs, including where applications can be found; and (7) application materials that clearly state SBA’s function as the federal source of disaster loans for homeowners and renters. We reviewed SBA Disaster Loan fact sheets available for Hurricanes Harvey, Irma, and Maria and other outreach materials distributed to disaster survivors and found they collectively included the required elements (see fig. 5 for examples). For example, the Hurricane Harvey fact sheet contains the date of the declaration and counties included in the declaration, as well as information about SBA’s function to provide disaster loans, the types of loans available, loan eligibility requirements, deadlines, and contact information for SBA disaster assistance. SBA’s guidance on outreach does not include steps on identifying regional disaster risks. In 2009, we recommended that SBA develop procedures to enable it to meet the region-specific requirements of the Small Business Act. Specifically, we recommended that SBA include likely scenarios for certain regions prone to disasters. In 2012, SBA completed a marketing and outreach plan that stated SBA would develop webinars for specific regional risks. However, SBA’s 2018 Marketing and Outreach Plan did not mention or incorporate regional challenges such as those SBA encountered responding to the 2017 hurricanes in Puerto Rico and the U.S. Virgin Islands. SBA officials told us that they experienced challenges in conducting outreach to the territories. As previously mentioned, SBA used the only operational media outlet in Puerto Rico—a radio station—immediately after Hurricane Maria to broadcast information. SBA officials told us that language barriers also presented a challenge during the response to Hurricane Maria. Outreach materials had to be printed in both Spanish and English (see fig. 6). Federal internal control standards state that management should internally communicate the necessary quality information to achieve the agency’s objectives. However, SBA’s guidance on outreach does not identify regional disaster risks. SBA officials told us that they have not considered documenting these challenges in their outreach guidance, although they may do so in the future. Without updating its outreach guidance to discuss region-specific challenges, such as those faced in responding to disasters in the U.S. territories, SBA misses a key opportunity to better ensure staff are adequately prepared to conduct outreach in similar situations and locations. SBA does not have metrics for how well its outreach efforts informed disaster survivors about the Disaster Loan Program. Although SBA surveyed a sample of disaster loan applicants in August 2018, the survey did not include questions specific to applicants’ perception of SBA’s outreach efforts that the agency could use to measure the success of its efforts. Officials told us the survey was primarily used to evaluate SBA’s loan processing and not its outreach efforts. In past work, we convened an expert panel to discuss challenges with consumer education and key planning components to overcome these challenges. One of the key practices identified in the expert panel was the need to establish metrics to measure success in achieving the objectives of an outreach campaign. For example, process metrics can help ensure the quality, quantity, and timeliness of a campaign, and outcome metrics evaluate how well the campaign influenced the attitudes and behaviors of the target audience. Without metrics evaluating its disaster outreach efforts, SBA will not be able to determine how well and to what extent its outreach efforts have informed disaster survivors about the Disaster Loan Program. Since 2005, SBA has streamlined its loan application and review process and recognized such changes resulted in a need for earlier staff activation. SBA’s approval rate for all disaster loan applications following the 2017 hurricanes was approximately 49 percent and Hurricane Maria had the highest approval rate (62 percent). Disaster loan applicants, SBA resource partners, and SBA officials identified challenges that affected application or review processes, including burdensome documentation requests and translation issues. Since 2005, SBA has made changes to its disaster loan program to streamline the loan application and review process. Some of the changes SBA implemented include Using electronic loan applications. In 2008, SBA created an online portal for the Disaster Loan Program, which eliminated the need for applicants to mail in applications or visit a recovery center. According to SBA officials, they typically receive paper applications within 14 days and electronic loan applications within 1–2 days after a disaster. They also indicated that increased usage of electronic loan applications has reduced data entry errors and improved loan processing times. The vast majority (96 percent) of the approximately 340,000 applications SBA accepted after the 2017 hurricanes were submitted electronically. Expediting declines for applicants with poor credit. According to SBA officials, in 2005 SBA established automatic declination for applicants with poor credit (instead of moving forward with full processing of all applications). SBA refers homeowners who are automatically declined to FEMA for a potential grant and refers such businesses to resource partners for assistance. Overall, more than half (55 percent) of the approximately 146,000 applications declined after the 2017 hurricanes were automatically declined. Expediting approvals for applicants with strong credit. In 2014, SBA revised its disaster loan program regulations for physical disaster loans to allow it to consider an applicant’s credit instead of only looking at a full cash flow analysis when determining an applicant’s ability to repay. Overall, SBA processed 28 percent of the applications approved after the 2017 hurricanes using this revised method. Using desktop verification. In 2017, SBA began conducting desktop verification to evaluate the cause and extent of property damage. The process involves an initial loss verification through interviews with the applicant and use of third-party information (such as from a tax assessor’s website or Google maps) to estimate the cost of repairs. SBA is to conduct a post-desktop review following the initial disbursement using FEMA’s inspection report, SBA’s on-site inspection, or supporting documentation to validate the initial estimate. In a September 2019 report, the SBA OIG found that the use of desktop loss verification contributed to SBA meeting its timeliness goals for processing loan applications for the 2017 hurricanes. Standardizing loan terms. In 2017, SBA established 15- and 30-year fixed terms for loans, which streamlined the loan process by using the loan amount (instead of income) to determine repayment. According to SBA officials, the use of fixed loan terms is consistent with standard private-sector lending practices and therefore is easier for borrowers to understand. SBA also has continued to make technological changes to streamline DCMS and the web portal. The DCMS version SBA used for the 2017 hurricanes supported up to 10,000 concurrent users. According to SBA officials, they have been transitioning to DCMS 2.0, which is expected to support more concurrent users. As noted in SBA’s fiscal year 2019 Congressional Budget Justification and its 2017 Annual Performance Report, SBA anticipates that DCMS improvements will increase loan officer productivity from processing three to processing six loan applications per day. Similarly, SBA integrated new features into the online portal to improve applicants’ access to information resources during the application process. For example, applicants can readily access general questions and information, check the status of their applications, receive status notifications, and electronically upload and sign documents such as Internal Revenue Service Form 4506-T. According to SBA officials, electronic loan application and other changes have reduced SBA’s processing times. For Hurricanes Harvey and Irma, the number of applications to be processed peaked about 2 months after each disaster; applications for Hurricane Maria peaked nearly 3 months after the disaster (see fig. 7). The number of business applications to be processed peaked more than 3 months after Hurricane Sandy made landfall on October 29, 2012. Our analysis of SBA data found that despite a high volume of applications, SBA exceeded its goal for the 2017 hurricanes of processing at least 85 percent of applications from receipt to decision within 45 days (see fig. 8). More specifically, SBA processed 96 percent of its applications within 45 days. As noted previously, SBA’s processing goal varies based on expected application volume. The average processing times for loans submitted after Hurricanes Harvey, Irma, and Maria were 16, 16, and 18 days, respectively. In general, SBA processed loans for homeowners faster than loans for businesses after the 2017 hurricanes. In contrast, SBA did not meet its 21-day processing goal after 2012’s Hurricane Sandy and developed a backlog of more than 6,000 applications that lasted approximately 4 weeks. The SBA OIG noted issues with and recommended two improvements to how SBA calculates processing time from acceptance to decision. In a June 2014 report, the OIG found that SBA included times for automatically declined applications in its average processing times. The OIG recommended SBA report processing times for automatically declined applications separately from applications requiring more processing. In the summary data SBA provided for the 2017 hurricanes, SBA continued to include automatically declined applications, which require significantly less time to process than other accepted loans, in its average processing times. The OIG also found SBA’s computation did not include all the processing time for applications previously submitted and withdrawn, but later reaccepted. SBA used only the days elapsed between the reacceptance and the decision date. As discussed in more detail later in the report, SBA and applicants can withdraw and later resubmit applications. The OIG recommended that SBA establish processing time goals that consider the full processing time for withdrawn applications that later are reaccepted. According to OIG officials, the OIG closed this recommendation although SBA did not implement it because SBA officials stated that system limitations would not allow for this measurement to be readily accomplished and reaccepted loans require a level of analysis and diligence that justifies separate measurement of processing time. Although SBA officials told us there are no timeliness goals associated with closing a loan, approved borrowers have up to 60 calendar days from the date of the loan authorization and agreement to sign and return all loan closing documents to close the loan. For Hurricanes Harvey and Irma, SBA took less than 50 days on average to close a loan, with business loans taking the longest. For Hurricane Maria, it took more than 53 days on average to close loans. In comparison, for Hurricane Sandy SBA took 66 days on average from approval to close a physical disaster business loan and 43 days for an economic injury loan. After the 2017 hurricanes, SBA also met its disbursement goal of providing initial disbursements to 95 percent of borrowers within 5 days of loan closing. On average, SBA provided initial disbursements to approved borrowers within about 4 days of closing. SBA also had met its initial disbursement goal following Hurricane Sandy. As a result of SBA’s changes to its loan processing and review functions, SBA reduced its overall processing times. For the 2017 hurricanes, SBA took about 70 days on average to go from acceptance to initial disbursement, which is much less time than it took for Hurricane Sandy and the 2005 Gulf Coast hurricanes (see fig. 9). According to SBA officials, increased usage of electronic loan applications and expedited loan processing has resulted in the need to activate staff earlier, although SBA still faced challenges doing so. The number of applications that SBA received peaked at 95,000 applications in October 2017. SBA began adding staff immediately after the hurricanes, but the number of staff in ODA’s Processing and Disbursement Center did not peak until December 2017 (see fig. 10). In contrast, following Hurricane Sandy, application numbers peaked in December 2012 but ODA’s Processing and Disbursement Center did not reach peak staffing until March 2013. While hiring processes were ongoing, SBA temporarily utilized staff from other SBA divisions to assist with reviewing applications. ODA then hired additional loan officers and also used other SBA employees and detailees from other federal agencies to process loans. By October 2017, ODA had more than quadrupled processing staff (from 536 in August to 2,302 in October), and the workload for those staff had peaked (see fig. 11). Each loan officer averaged more than 50 new loan applications in September 2017, but by January 2018 the average decreased to about six or fewer new applications per month. As discussed earlier, SBA anticipates its DCMS update will double loan officer productivity. In response to the 2017 hurricanes, SBA accepted about 340,000 applications and approved about 141,000 of them, making about $7.2 billion in loans (see fig. 12). SBA accepted the most applications for Hurricane Irma, but about half of the total approved loan amount was for applicants affected by Hurricane Harvey. Overall, SBA’s approval rate for all disaster loan applications following the 2017 hurricanes was approximately 49 percent (see fig. 13). Of the three 2017 hurricanes, Hurricane Maria had the highest approval rate, 62 percent. The approval rate for physical disaster loans for homeowners was higher for Hurricane Maria (64 percent) than for Harvey (46 percent) and Irma (42 percent). There was not much variation in approval rates for physical and economic injury disaster loans for businesses. As shown in figure 14, the overall approval rate for loan applications was lower for Hurricanes Harvey, Irma, and Maria than for Hurricane Sandy, but higher than the combined rate for Hurricanes Katrina, Rita, and Wilma. Following the 2017 hurricanes, SBA declined about 146,000 loan applications. The primary reasons for declining applications were lack of repayment ability and unsatisfactory credit history. Other common reasons included unsatisfactory history on a federal obligation and that the damaged property was not an applicant’s primary residence or a qualified rental property. Declined applicants can request that SBA reconsider their applications. SBA received about 15,000 such requests after the 2017 hurricanes. Of the requests SBA accepted, about half had their applications approved, and about 30 percent were denied. Applicants who have their reconsideration requests denied can appeal SBA’s decision. Ninety-one percent of applicants who were denied after the 2017 hurricanes and subsequently appealed won their appeal. We also discuss withdrawal and cancellation rates for the 2017 and prior hurricanes in appendix II. Disaster loan applicants, SBA resource partners, and SBA officials we interviewed in Florida, Puerto Rico, Texas, and the U.S. Virgin Islands identified a number of challenges that affected the application or review processes following the 2017 hurricanes. Disaster loan applicants and SBA resource partners we interviewed identified the following challenges that applicants had when applying for SBA disaster loans following the 2017 hurricanes: Providing required loan documentation. Eight (of the 24) business loan applicants and officials from 10 entities, including from five SBDCs, felt that meeting SBA’s loan documentation requirements was time-consuming or burdensome. Following the hurricanes, applicants experienced difficulty readily producing required documentation (such as insurance policies, property titles, and tax returns) because of extensive physical damage and power issues or outages. Additionally, 11 applicants reported that follow-up requests from SBA for additional documentation delayed processing, added confusion, or led some to withdraw their applications. According to the results of a 2018 customer satisfaction survey conducted for SBA, the satisfaction of business loan applicants with the disaster loan application process had decreased by 9 percentage points since the previous survey in 2017. The lowest-rated aspects of the application process for businesses were “ease of attaining information required for completing the application,” the “amount of paperwork involved,” and the “overall ease of filling out the application.” According to SBA officials, they are statutorily required to request certain information from disaster victims who apply for a disaster loan. Although more documentation is requested for business loans than for home loans, the officials believe the information requested from business applicants is similar to information requested from individuals applying for a loan at a commercial bank. In response to previous recommendations we made, SBA recently has taken steps to streamline the application process by improving accessibility and consistency of loan-related information and requirements in paper and electronic resources. For example, SBA added a list of frequently asked questions to its Disaster Loan Application Portal with a list of documents required to file an application. The paper applications for both home and business loans list potential additional documents that SBA may request as well as when they may be required. Officials from all three SBDCs in Puerto Rico told us that applicants in Puerto Rico particularly had difficulty accessing tax documentation in a timely manner. SBA has an automated process to request and obtain tax transcripts from the Internal Revenue Service for final approval of loans. However, in Puerto Rico SBA must use a manual process with the Departamento de Hacienda (Puerto Rican taxing authority). Further delays occurred due to widespread physical damage to infrastructure, including the Departamento de Hacienda. As a result, SBA permitted applicants in Puerto Rico to postpone submitting tax documentation until later in the application process. Instead of submitting tax documentation during the loan decision process, SBA allowed applicants to submit such documentation during conditional commitment (the point at which SBA’s recommendation for loan approval is contingent on the applicant submitting additional required documents). In addition, ODA co- located loan officers with staff from the commitments department to facilitate prompt access to the tax transcripts they were awaiting from the Departamento de Hacienda. Meeting flood insurance requirements. Officials from two Florida entities told us many of the small businesses with which they worked had problems submitting insurance documentation as part of their application. Officials from one SBDC in Puerto Rico told us they had a client who was unable to obtain flood insurance. By law, SBA requires borrowers whose damaged or collateral property is located in a special flood hazard area to obtain and maintain appropriate flood insurance for the term of the loan. Delays also occurred as a result of miscommunication between SBA and loan applicants about flood insurance requirements, according to an applicant we interviewed. That business owner told us that when he submitted a disaster loan application for physical and economic injury damages, SBA informed him the insurance requirements applied only to the physical damage portion of the loan. However, SBA later told the individual that insurance coverage was necessary for the entire loan, not just physical damages. Frequent changes in loan officers or case managers. More than two-thirds (17 of 24) of the business owners we interviewed and officials at four of the nine SBDCs told us they (or their clients) worked with more than one case manager or loan officer during the loan application process. Two small business owners said they had interacted with as many as nine. These changes led to the applicants having to repeat or resubmit information. In addition, they sometimes received different answers to questions when a new loan officer or case manager was assigned. SBA officials told us that SBA does not track the extent to which applicants work with multiple loan officers and case managers but that applicants should generally have only one of each. According to SBA officials, one loan officer is typically assigned to an application until a loan decision is made. Once a loan is approved, one case manager is assigned until the loan is fully disbursed. However, according to SBA officials, an applicant may interact with more than one loan officer and case manager for reasons such as staff turnover and staff downsizing. In addition, whenever an applicant requests reconsideration of a declined application, the application is to be assigned to a new loan officer for processing. According to SBA officials, a newly assigned loan officer or case manager should be able to use DCMS to access an applicant’s loan file, including records of past communication between prior loan officers and the applicant, loss verification reports, and previously submitted documents. When an applicant electronically submits a document through the web portal, ODA scans and uploads it into DCMS (typically in 24–48 hours), at which point it is viewable in DCMS. The officials explained that document storage methods in DCMS are uniform, which should minimize the possibility of documents being improperly stored. Poor customer service and translation issues. Five of the applicants (of 24) and officials from four entities we interviewed said disaster loan applicants perceived a lack of SBA responsiveness after submitting their applications. Staff from two of the entities attributed this lack of responsiveness to SBA having an inadequate amount of staff to handle the number of applications. SBA officials indicated that Processing and Disbursement Center staff strive to contact each applicant within 3 days of assigning a loan officer and within 5 days of assigning a case manager. They also stated the center should contact each applicant at least every 30 days using whatever available forms of communication, including mail, email, and telephone. But two applicants told us they only heard from their case manager after the applicant initiated the contact. Nine applicants also stated that when they did talk to their loan officer or case manager that person was not very helpful. For example, officials from one SBDC told us they talked to some loan officers on behalf of their clients and those loan officers were not able to clarify what documentation their clients needed to provide to SBA to complete their applications. One Puerto Rican applicant and officials from all three Puerto Rican SBDCs told us that applicants in Puerto Rico faced translation issues. They believed SBA had insufficient staff who spoke Spanish, which made it difficult to communicate with applicants regarding status updates and requests for additional documentation. The SBA OIG reported that SBA officials estimated that some disaster survivors waited more than 45 minutes for an interpreter or experienced dropped calls. In addition, officials from one Puerto Rican SBDC told us SBA only would accept official responses in English and believed that many applicants were denied in part because the documents with applicants’ personal and financial information were in Spanish. The SBDC officials were told only an SBA translator could translate official documentation between SBA and applicants from English to Spanish. According to SBA officials, they tried to hire as many Spanish- speaking staff as possible and relied on a contractor to provide interpretation services during telephone communications with applicants. But the volume of calls was higher than anticipated, resulting in long wait times and many lost telephone connections. Following Hurricane Maria, SBA replaced the contractor with three new language service providers, which SBA used during its response to Hurricanes Florence and Michael in 2018. Disbursement delays. Eight business disaster loan applicants and officials from three SBDCs told us that applicants experienced delays receiving disbursements after their initial disbursement. Although applicants said they received initial disbursements within expected time frames, subsequent disbursements took longer than anticipated. For example, three small business owners told us it took more than a year from the time they applied to receive their full loan disbursement, and two others had to contact their federal representative to help get their disbursements because of delays. SBA data show that after the 2017 hurricanes it took, on average, about 141 days after applying for a loan for businesses to receive their full disbursement, including an average of 63 days from closing to final disbursement. Similarly, business respondents to the 2018 customer satisfaction survey conducted for SBA expressed concerns about the timeliness of disbursements following loan closing. The lowest-rated aspect of the loan closing process was “timeliness of receiving loan funds after the closing was complete.” As a result, the firm that conducted the survey recommended that SBA examine data on the timing of loan disbursements over the past several years, determine whether the timing had changed significantly, determine the root causes of any notable changes, and develop plans to address any root causes. SBA guidance requires approved borrowers to arrange for and obtain all loan funds within 6 months from the date of the loan authorization and agreement. However, SBA may extend the time frame on a case- by-case basis for ongoing projects. According to SBA officials, there are no timeliness goals for subsequent disbursements because the time frame for receiving further disbursements is contingent on the borrower’s ability to meet insurance requirements, secure a contractor to repair damages, and submit receipts to SBA. SBA also has improved features within the web portal so that borrowers can now use direct deposit to receive disbursements and commence repairs more quickly. Additionally, the portal now enables borrowers to electronically submit receipts for repairs or invoices to loan officers, who in turn can verify the use of disbursed funds and make additional disbursements much sooner than before. During interviews and in an after-action report, SBA also identified challenges it experienced after the 2017 hurricanes that included a prolonged loss of electricity, DCMS performance issues, and unique loan closing requirements in Puerto Rico. Loss of electricity. As discussed previously, a prolonged loss of electricity adversely affected application submission and loan processing, especially in Puerto Rico. DCMS performance issues. SBA officials said the agency increased staffing to process incoming loan applications and the multitude of concurrent users caused technical issues and delays. Loan applicants also encountered periodic system outages. SBA released DCMS updates throughout the first few months of its response to the 2017 hurricanes to address system performance issues. As previously mentioned, SBA expects that DCMS 2.0 will address concurrent user issues. Unique loan closing requirements in Puerto Rico. SBA officials said they were initially unaware of loan closing requirements unique to Puerto Rico, which led to processing delays. They told us that secured loans must be signed by a notary, who in Puerto Rico must be a Puerto Rican-licensed attorney. As a result, SBA had to hire additional locally licensed attorneys and devote resources toward training attorneys and other staff involved in processing and closing loans in Puerto Rico, a process hampered by the previously discussed hiring challenges for the 2017 hurricanes. SBA has been implementing changes to its hiring process for future disasters. Number of loans. As of September 2019, lenders had issued two loans totaling $50,000 under SBA’s Express Bridge Loan Pilot Program. The loans were issued by one lender in 2018 to small businesses in North Carolina and South Carolina recovering from Hurricane Florence. Outreach. SBA generally has not targeted its outreach for the program to disaster-prone areas. According to SBA officials, to market the pilot program OCA issued two Federal Register notices and a program guide, and encouraged district office staff to notify area lenders of the program. Additionally, SBA officials told us that before anticipated disasters, they have mentioned the program on quarterly telephone conferences attended by 1,000–1,500 7(a) lenders, and referred to the pilot in press releases. One lender told us that it was notified of the pilot program through an SBA policy notice with program guide attached, while another noted receiving an email promoting the program. According to SBA officials, this same email was sent to OCA field staff to be shared with all 7(a) Express lenders before the 2019 hurricane season. Internally, according to SBA officials, OCA has made information on the Express Bridge Loan Pilot Program available to all SBA employees on an internal website. However, OCA has not marketed the Express Bridge Loan Pilot Program to ODA. OCA officials were unaware of any conversation with ODA (whose staff are on the ground after disasters and therefore most likely to interact with small business owners) about the pilot program, but assumed that ODA officials had high-level information about the pilot. However, staff we interviewed from both Field Operations Centers were unaware of the program. OCA also has not targeted its external marketing to partners, such as 7(a) Express lenders and SBDCs, in disaster-prone areas. Although SBA officials told us that they made presentations at a 2019 lender conference in Florida, they did not point to similar outreach in other disaster-prone areas. The small business owners and SBDC officials with whom we spoke in Florida, Puerto Rico, Texas, and the U.S. Virgin Islands generally were unaware of the SBA Express Bridge Loan Program. During our February 2019 interview with the Florida Department of Economic Opportunity (which helps administer the state’s bridge loan program), officials stated they were unaware of program specifics and asked if it had started. Evaluation. SBA officials do not currently plan to evaluate the program or determine why so few loans have been issued. SBA officials explained that the Express Bridge Loan Pilot Program’s performance during the 2019 hurricane season would determine whether SBA would continue or terminate the program after its pilot period. When SBA announced the pilot, SBA stated it planned to evaluate the program using three principal measures: (1) the number of small businesses served, (2) the percentage of loans made that were paid off or down using lower fixed-rate disaster loans versus those held to term, and (3) the default rate on Express Bridge Loans compared to regular SBA Express loans of similar size in the 7(a) portfolio. SBA officials told us they were not planning to conduct an evaluation because only two loans had been made, which the officials believe is not a large enough sample size to conduct a meaningful evaluation of the program. Although SBA has guaranteed only two loans issued under the Express Bridge Loan Pilot Program, the program received 93 applications. While most of the applications were not completed, they suggest a potentially larger demand for the program than initially indicated by the two completed loans. We discuss demand for bridge loans in more detail in the following section. SBA has guidelines for evaluating pilot programs. SBA issued a Policy Notice on September 29, 2016, that called for it to evaluate any pilot program. SBA subsequently incorporated this requirement in one of its standard operating procedures. In addition, our guide for designing evaluations states that an evaluation gives an agency the opportunity to refine the design of a program and provides a useful tool to determine whether program operations have resulted in the desired benefits for participants. We also previously reported that an evaluation can be valuable in determining why goals were not met, and can provide feedback on both program design and execution. Such an evaluation can help determine what program changes might be warranted to achieve the desired impact. In the absence of loan data, an evaluation could include consideration of feedback from lenders in disaster-prone areas on the pilot or their experiences with other bridge loan programs, such as the Florida program discussed in the following section. According to SBA officials, SBA had not actively solicited lender feedback on the current pilot since it became operational. However, SBA sought feedback from lenders on prior efforts to develop a bridge loan program. SBA officials told us that lenders commented on a prior proposal for a bridge loan pilot (the Immediate Disaster Assistance Program) and indicated that private lenders were not interested in participating in such a pilot because they considered disaster relief to be a governmental responsibility. Additionally, SBA officials told us during our previous work looking at the Immediate Disaster Assistance Program that they performed initial outreach to lenders—such as those who participated in SBA’s Gulf Opportunity Pilot Loan Program in the aftermath of Hurricanes Katrina and Rita—to obtain their reaction to and interest in the program. The current pilot provides a similar opportunity to obtain feedback from lenders on loan terms that could affect their willingness to participate in this or a potentially redesigned program. For example, one 7(a) lender with whom we spoke did not like the Express Bridge Loan Pilot Program due to the maximum loan amount and the SBA guaranteed percentage. Without evaluating the pilot program, including assessing potential demand and why so few loans have been made and the sufficiency of its outreach efforts, OCA will have limited information to inform its decision on the future of the pilot, including loan terms it may offer. Additional feedback, such as from lenders, could help SBA determine if design changes are warranted. The very low level of loans guaranteed under SBA’s bridge loan pilot contrasts with the desire for such loans indicated by our interviewees and the experience of the Florida Small Business Emergency Bridge Loan program. We interviewed small business owners and those who work with small businesses. Almost two-thirds of all the entities (16 of 26) we interviewed and six small business owners told us that businesses needed immediate financial support after a disaster (for example, to help remove debris, make repairs, and replace inventory). Although many businesses ultimately receive insurance payments to help cover losses, the payments may not be received for several months. Representatives from one municipality told us a lot of people, including the city itself, still were awaiting their insurance payments more than a year after the disaster. To help fund their immediate recovery, small business owners with whom we talked often relied on their own savings, credit cards, or other sources of credit. However, these financing sources typically have interest rates higher than those offered by the Express Bridge Loan Pilot Program. In addition, it can take applicants approved for SBA Disaster Loans months to receive funds—time that could determine whether a business remains in operation. Florida’s Small Business Emergency Bridge Loan program provides small business owners with interim disaster financing, similar to the Express Bridge Loan Pilot Program, but some structural differences exist (see table 2 for a comparison of principal terms and features). In particular, the Florida program uses public funds rather than being funded by financial institutions. Officials of the Florida Department of Economic Opportunity, which administers the program in partnership with the Florida SBDC Network and a third-party fiscal administrator, told us the Florida program is not focused on generating a return on investment. Rather, the program is focused on helping small businesses bridge the gap between the time their businesses incur damages and the time they secure other financial resources. Although SBA officials told us that the State of Florida’s program is not a valid comparison to SBA’s Express Bridge Loan Pilot Program because the loans for SBA’s program are made by private, for-profit lenders, the Florida program demonstrates demand for bridge financing for small business owners. Officials from the Florida Department of Economic Opportunity stated that when the Governor’s office activates the Florida Small Business Emergency Bridge Loan program, the department uses various outreach media, including social media, public-private partners, and state emergency response to disseminate information about the program. The program’s fiscal administrator and the Florida SBDC Network also circulate information about the program, including by providing hyperlinks to the loan application on their websites. Finally, the program has been in place since 1992, which likely contributed to word-of-mouth about the program. According to officials of the Florida Department of Economic Opportunity, following Hurricane Irma the program received 1,167 applications, of which 883 were approved, totaling $35 million in disaster bridge financing. Following Hurricane Michael, the program received 742 applications, of which 590 were approved, totaling $34.3 million in disaster bridge financing. Florida Department of Economic Opportunity officials attributed the wide utilization of the Florida Small Business Emergency Bridge Loan to expedited access to funding and favorable terms. For example, the program offers survivors a zero percent interest rate for the term of the loan, which is generally 1 year. Given the difference in loan terms between Florida’s program and SBA’s pilot program, the demand for SBA’s higher-interest loans may not equal the demand for Florida’s loans. However, in disaster areas in which there is no state bridge loan program, SBA’s pilot program could meet at least some of the need for bridge financing. Following Hurricanes Harvey, Irma, and Maria in 2017, SBA faced difficulties in delivering its disaster loans due to the magnitude of the storms and resulting infrastructure damage, especially in Puerto Rico and the U.S. Virgin Islands. Nevertheless, SBA accepted about 340,000 applications and approved about 141,000 of them, making more than $7 billion in loans to help business owners, homeowners, renters, and nonprofits recover. However, the planning documents and guidance to the field that SBA used to guide its response to the 2017 hurricanes did not identify risks and focus on risk assessments or include detailed instructions on how to prepare disaster-specific action plans. And after the 2017 hurricanes, the documents and guidance, including for outreach on disaster loans, has not incorporated lessons learned—particularly as they related to region- specific risks that could hamper the operations of the Disaster Loan Program. By identifying risks and enhancing guidance on actions plans and outreach efforts, SBA can better design its plans for and implementation of disaster response efforts and help ensure staff are adequately prepared to conduct operations in situations such as those encountered in Puerto Rico and the U.S. Virgin Islands. SBA also has not established metrics to measure the success of its outreach efforts. By establishing metrics, such as including questions when surveying disaster loan applicants on their perception of SBA’s outreach efforts, SBA would be better able to determine how well its outreach efforts have informed disaster survivors about the Disaster Loan Program and make adjustments to improve the effectiveness of such efforts. SBA also can improve its Express Bridge Loan Pilot Program, which offers small businesses the opportunity to quickly receive funding after disasters. The success of a Florida program offering a similar product and our own work suggest considerable demand for bridge loans. SBA has no plans to evaluate the program because it has issued so few loans. While SBA does not have the information needed to conduct a loan performance evaluation, it has the opportunity to conduct an evaluation of the pilot program’s design and implementation. Such an evaluation could help determine why so few loans were issued, what role program design and internal and external outreach may have played, and what, if any, changes to the pilot might be warranted. By evaluating the Express Bridge Loan Pilot Program, including obtaining lender feedback, OCA will be able to make an informed decision about the program’s future. We are making the following five recommendations to SBA: The Associate Administrator for the Office of Disaster Assistance should identify and document risks associated with its disaster response and plans to mitigate these risks in its disaster planning documentation. (Recommendation 1) The Associate Administrator for the Office of Disaster Assistance should identify the key elements of a disaster action plan and provide additional guidance to staff on how to incorporate these elements into future action plans. (Recommendation 2) The Associate Administrator for the Office of Disaster Assistance should update its outreach plan to include information on region-specific risks or challenges, such as those encountered after the 2017 hurricanes. (Recommendation 3) The Associate Administrator for the Office of Disaster Assistance should establish metrics to measure the success of its outreach efforts during the response to a disaster. (Recommendation 4) The Associate Administrator for the Office of Capital Access should evaluate the implementation of the Express Bridge Loan Pilot Program to determine why so few loans have been made and if any design changes may be warranted before the end of the pilot. Such an evaluation could include assessing SBA’s outreach efforts and seeking feedback from lenders. (Recommendation 5) We provided a draft of this report to SBA for review and comment. In written comments, which are reproduced in appendix III, SBA stated that overall it agreed with the report’s recommendations and provided comments, as summarized below. SBA described actions it planned to take to address the first four recommendations, which if implemented as planned would address them. For the first recommendation to identify and document risks—and plans to mitigate those risks—in its disaster planning documents, SBA noted that ODA would work with SBA’s Office of Continuous Operations and Risk Management, which updates and publishes the annual Disaster Preparedness and Recovery Plan, to identify and document known risks associated with SBA’s disaster response and implement a risk-informed approach to its direct response and recovery operations. SBA expected that its fiscal year 2021 plan would include this information. For the second recommendation to identify the key elements of a disaster action plan and provide related guidance to staff, SBA agreed and noted that ODA would develop the key elements of and templates for a disaster action plan, provide guidance to Field Operations Centers, and coordinate with the Office of Continuous Operations and Risk Management on including the information in appendixes to the Disaster Preparedness and Recovery Plan. For the third recommendation to update its outreach plan to include information on region-specific risks or challenges, SBA agreed and stated that ODA would update its outreach plan to include risks and challenges experienced during the 2017 hurricane season. For the fourth recommendation to establish metrics to measure its disaster outreach efforts, SBA agreed and noted that it would explore potential new metrics to measure the effectiveness of its outreach efforts, such as adding a question about the efforts to the annual American Customer Satisfaction Index survey. For the fifth recommendation to evaluate the implementation of the Express Bridge Loan Pilot Program to determine why so few loans have been made and if any design changes may be warranted before the end of the pilot (which could include assessing SBA’s outreach efforts and seeking feedback from lenders), SBA described some actions it planned to take in response to the recommendation that would partially address it. The agency also commented on some of our findings, which we discuss below. We maintain our recommendation. In its comments, SBA noted it would seek feedback from SBA Express Lenders during the upcoming spring National Association of Government Guaranteed Lenders conference on their interest and participation in the pilot. SBA also noted that it would provide ODA with information and a set of frequently asked questions about the pilot program that ODA could distribute to small business owners at Business Recovery Centers. Although beginning to seek lender feedback and improving its internal outreach are good first steps, SBA would still need to conduct an evaluation to determine if any design changes were warranted to fully address the recommendation. SBA stated it did not currently have adequate data to evaluate the effectiveness of the program because only two loans had been funded. It also noted that while there were more than 90 incomplete applications for the Express Bridge Loan Pilot Program, the agency had concluded that there were not sufficient data to suggest the applications were actual attempts to originate loans and that lenders likely started the loans in error. In the draft report, we acknowledged only two loans had been issued and included SBA’s views about why other applications were not completed. As we note in the report, in the absence of loan data an evaluation instead could focus on the pilot’s design and implementation, and include feedback from lenders. Therefore, we maintain that SBA can and should evaluate the pilot program. SBA noted it previously sought feedback from lenders on the Immediate Disaster Assistance Program (a prior proposal for a bridge loan program) and at that time, there was very little lender interest in the program. The draft report included this information and noted that the current pilot provides a similar opportunity to solicit lender feedback, including on loan terms. In response to this, SBA stated that making significant changes to the size and guarantee on the pilot loans based on lender feedback would affect the subsidy for the program. However, SBA will not know how lenders feel about the current pilot or what types of changes they might recommend until it seeks lender feedback. Lenders could call for changes to the program that were minor but still would improve their willingness to participate. In addition, if lender feedback suggested that the program would not work without significant changes and additional subsidy, such information could inform future actions to address borrower demand for bridge loans. SBA also stated that it did not believe our comparison of the Express Bridge Loan Pilot Program to the Florida Small Business Emergency Bridge Loan was appropriate. SBA stated the Florida program provides direct loans in contrast to SBA’s limited guarantee to lenders, and had been in existence since 1992 and was well known (versus a 2017 program start and less knowledge about SBA’s program). We highlighted both differences in the draft report. SBA also stated the Florida program has a larger pool of eligible applicants because it is not limited to Presidentially declared disasters or delivered only by certain lenders, as is the case with the SBA program. However, we note that the SBA program is available nationwide and the Florida program is limited to the state. In summary, SBA stated in its comments that the characteristics of the Florida program contributed to the demand for the program. We acknowledged this in the draft report, stating that given the differences between the programs, the demand for SBA’s loans may not be equivalent to the demand for Florida’s loans. Despite the program differences, we continue to believe that including a discussion of the Florida Small Business Emergency Bridge Loan program in the report was appropriate because its use indicates demand for bridge financing. In disaster areas in other states with no state bridge loan program, SBA’s pilot program, with changes determined to be appropriate, could meet at least some of the demand for bridge financing identified by people we interviewed. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees and the Administrator of SBA. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact William B. Shear at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix IV. Our objectives were to examine the Small Business Administration’s (SBA) (1) planning for and initial response to Hurricanes Harvey, Irma, and Maria (hereinafter referred to as the 2017 hurricanes); (2) loan application and review process (including changes since Hurricane Katrina in 2005), timeliness, and approval rates; and (3) implementation of the Express Bridge Loan Pilot Program. We focused our review on Florida, Texas, Puerto Rico, and the U.S. Virgin Islands—the states and territories most directly affected by the 2017 hurricanes. For all of our objectives, we visited two areas each in Florida, Texas, and the U.S. Virgin Islands, and three areas in Puerto Rico. We selected these areas based on the high numbers of approved homeowner and business disaster loans, high population, geographic diversity, presence of a Small Business Development Center (SBDC), and presence of an SBA district office. To help ensure geographic diversity, we selected locations from different metropolitan statistical areas in each state and Puerto Rico. In each location, we conducted a group interview with small business owners who applied for a disaster loan with SBA to learn about their experiences applying for a loan and working with SBA while their loan was processed and funds disbursed, and their need for bridge financing. These small businesses were recruited by representatives of the local SBDC and were selected to represent a range of outcomes (approved, denied, and withdrawn applications). Overall, we interviewed 24 small business owners across the nine areas we visited. In addition to small business owners, we met with officials from the SBDC and the local SBA district office responsible for each of the areas we visited. We also met with officials from seven local governments that oversaw eight of the areas we visited—either from a mayor’s office or economic development office. Lastly, we interviewed officials from six chambers of commerce or business associations, one in every area except one. While the results of these interviews are not generalizable to all areas affected by the 2017 hurricanes, they provided insight into the experiences of small businesses and local communities with the SBA Disaster Loan Program. In addition to the site visit interviews, we interviewed officials from SBA’s Office of Disaster Assistance, including officials from both Field Operations Centers and the Processing and Disbursement Center, to discuss the Disaster Loan Program and SBA’s response to the 2017 hurricanes. For our analysis of open-ended responses from these interviews, we used a software program designed for analyzing qualitative information. For each open-ended response, we coded, organized, and analyzed responses under a number of relevant themes. Specifically, team members independently reviewed a segment of the interview transcripts to code the responses to identify themes. Once each analyst had completed coding his or her respective sections, the documents were merged into a master file that was reviewed by the engagement’s methodologist. Possible alternative categorizations of the material were discussed and resolved jointly. This code-based method constituted our primary approach to validating the results of our analysis. All categorizations were sourced to the original interviews through the use of the qualitative analysis software. To evaluate SBA’s planning for and initial response to the 2017 hurricanes, we reviewed SBA planning documents in effect for the hurricanes—SBA’s 2017 Disaster Preparedness and Recovery Plan and its 2014 Disaster Playbook—as well as the targeted action plans SBA created for each of the hurricanes and summaries of exercises SBA conducted. We also reviewed the Office of Disaster Assistance’s after- action report following Hurricanes Harvey, Irma, and Maria; SBA’s 2017 and 2018 Staffing Strategies; models SBA used to guide its response; and reports issued by the SBA Office of Inspector General (OIG). We used these sources to identify challenges SBA faced planning for and responding to the 2017 hurricanes, including those it faced responding to Hurricane Maria in Puerto Rico and the U.S. Virgin Islands and with its hiring. We reviewed SBA data on hiring and compared SBA’s staffing modeling efforts to the actual staff activated after the 2017 hurricanes. We determined that these data were sufficiently reliable for the purposes of discussing SBA’s staffing after the 2017 hurricanes by interviewing knowledgeable officials about the data. We compared SBA’s planning efforts against federal internal control standards for identifying, analyzing, and responding to risks, and against Federal Emergency Management Agency guidance on disaster planning. To examine SBA outreach on disaster loans after the 2017 hurricanes, we reviewed marketing and outreach plans from 2012 and 2018 and a sample of outreach materials SBA used during its response to Hurricanes Harvey, Irma, and Maria. We compared SBA’s outreach efforts against federal internal control standards for internal communication and against key consumer education practices. To evaluate SBA’s loan application and review process, we reviewed prior GAO and SBA OIG reports on SBA’s Disaster Loan Program to help identify changes SBA made since Hurricane Katrina in 2005. We also analyzed summary data from SBA’s Disaster Credit Management System for applications submitted between August 31, 2017, and September 24, 2018. The August date corresponds to the date SBA received the first applications from the 2017 hurricanes and the September date to the one-year anniversary of Hurricane Maria, the last of the three 2017 hurricanes. SBA completed processing more than 99 percent of the applications for each hurricane during the period covered by our data. Because the hurricanes occurred in quick succession and SBA treated them as one event, we compared the combined data for the 2017 hurricanes to those from Hurricane Sandy and the combined data for Hurricanes Katrina, Rita, and Wilma, which SBA also treated as one event. We determined these data were sufficiently reliable for describing characteristics associated with SBA’s processing of applications by reviewing related documentation, interviewing knowledgeable agency officials, and reviewing related internal controls. We used the results from our thematic qualitative analysis and interviews with SBA officials to identify challenges associated with applying for or processing applications following the 2017 hurricanes. To evaluate SBA’s implementation of the Express Bridge Loan Pilot Program, we reviewed Federal Register notices and a program guide SBA published about the program. We interviewed SBA officials from its Office of Capital Access about their implementation of the program, and officials from three lenders to discuss their perspectives on and awareness of the program. We selected one lender because it had made loans under the pilot program, another because it had begun the most applications for the program, and the third because it was a 7(a) lender located in a state affected by Hurricane Florence or Michael. We chose those hurricanes because they occurred about 1 year after SBA began its pilot (in September and October 2018, respectively), allowing time for SBA to fully launch the pilot. We also interviewed officials from the Florida Department of Economic Opportunity to discuss the Florida Small Business Emergency Bridge Loan Program, which it helps administer. We compared SBA’s plans to evaluate the Express Bridge Loan Pilot Program against guidance for designing evaluations. We conducted this performance audit from September 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix presents withdrawal and cancellation rates for disaster loan applications the Small Business Administration (SBA) accepted after Hurricanes Harvey, Irma, and Maria (the 2017 hurricanes), after Hurricane Sandy in 2012, and after Hurricanes Katrina, Wilma, and Rita (the 2005 Gulf Coast hurricanes). For the 2017 hurricanes, withdrawal rates were generally comparable for each type of disaster loan, as shown in figure 15. Of the disaster loan applications SBA accepted for Hurricanes Harvey, Irma, and Maria, about 52,000 were withdrawn from consideration by SBA or the applicant. Of these, approximately 32,000 were withdrawn by SBA, and the rest were withdrawn at the applicant’s request. The leading reason for an SBA withdrawal was a lack of contact with the applicant to discuss disaster damages (inability to verify losses). Other common SBA-initiated reasons for withdrawals were the Internal Revenue Service having no record of the applicant filing taxes in the required time period and an applicant’s failure to provide requested additional information. The main reason for withdrawal by applicant was a change of plans. Applicants who withdrew their applications or had applications withdrawn by SBA could request that SBA reaccept their application. Nearly half (48 percent) of the approximately 52,000 withdrawn applications were submitted for reacceptance. Of those, 67 percent were approved, 20 percent were declined, and 13 percent were withdrawn again. As shown in figure 16, the overall withdrawal rate for the 2017 hurricanes was lower than that for Hurricane Sandy and the combined rate for the 2005 Gulf Coast hurricanes. Across each of the disasters, home disaster loan applications had the lowest withdrawal rates while nonprofit applications had the highest withdrawal rates. Among the 2017 hurricanes, Hurricane Harvey had the most cancelled disaster loans (10,945) with an overall cancellation rate of 25 percent (see fig. 17). More than half (56 percent) of the cancelled loans were due to the borrower’s decision, including receiving sufficient support from other sources and reluctance to incur additional debt. Of the loans SBA cancelled, the most common reasons included the borrower’s failure to complete and return all loan closing documents and an adverse change in the borrower’s credit or financial condition that required a referral to the Federal Emergency Management Agency’s Individuals and Household Program. As shown in figure 18, the overall cancellation rate for the 2017 hurricanes was lower than that for Hurricane Sandy and the combined rate for the 2005 Gulf Coast hurricanes. In addition to the contact name above, Paige Smith (Assistant Director), Daniel Newman (Analyst in Charge), Laura Gibbons, Marshall Hamlett, Marc Molino, Patrick Netherclift, Barbara Roesmann, Jessica Sandler, Cynthia Saunders, and Nina Thomas-Diggs made significant contributions to this report.", "summary": "SBA assists most types of businesses regardless of size and others affected by natural and other declared disasters through its Disaster Loan Program. Disaster loans can be used to help rebuild or replace damaged property or continue business operations. GAO was asked to review SBA's response to three 2017 hurricanes (Harvey, Irma, and Maria). This report examines SBA's (1) planning for and response to the 2017 hurricanes; (2) disaster loan application and review process; and (3) implementation of the Express Bridge Loan Pilot Program. GAO analyzed SBA planning documents; summary data from SBA's Disaster Credit Management System for applications submitted between August 31, 2017, and September 24, 2018 (the period in which SBA processed nearly all loan applications for each hurricane); and SBA guidance on the bridge loan program. GAO interviewed small business owners and officials from local governments, business advocacy organizations, and Small Business Development Centers in Florida, Texas, Puerto Rico, and the U.S. Virgin Islands. The Small Business Administration's (SBA) Office of Disaster Assistance, which administers the Disaster Loan Program, regularly develops disaster plans but does not discuss risks and risk mitigation in detail in its planning documents. Specifically, SBA's current Disaster Preparedness and Recovery Plan lacks an in-depth discussion of risks (including extended power and communications outages) that could affect its disaster response. SBA's disaster response includes deploying staff to and establishing centers in disaster areas to accept loan applications. The aftermath of the 2017 hurricanes (Harvey, Irma, and Maria) illustrates how the risks affected SBA's disaster loan operations. For example, because of widespread power outages (particularly in Puerto Rico), loan applicants often could not submit applications electronically and SBA often could not call or e-mail applicants. As a result, SBA may not be adequately prepared to respond to challenges that arise during its disaster response efforts. Changes SBA made to the loan application process since 2005 (such as implementing electronic applications) improved timeliness. For the 2017 hurricanes, SBA processed more than 90 percent of all loan applications (including those quickly declined or withdrawn) within its 45-day goal, averaging less than 18 days for each hurricane. Overall, about 49 percent of applications submitted after the 2017 hurricanes were approved (see figure). Applicants and others with whom GAO spoke noted some application challenges, including frequent changes to SBA contact staff and having to resend documents. According to SBA officials, staff changes resulted from turnover, among other reasons. Many applicants in Puerto Rico also encountered translation challenges during interactions with SBA. SBA has no plans to evaluate its Express Bridge Loan Pilot Program, a loan guarantee program that began in October 2017 and is set to expire on September 30, 2020, and is intended to offer small businesses quicker funding after disasters. As of September 2019, SBA had received 93 applications, but most of them were incomplete and SBA had guaranteed only two loans. The Office of Capital Access, which manages the pilot, had not sought feedback from lenders on why so few loans had been made. Without evaluating program design and implementation, SBA's ability to make an informed decision on the program's future, including assessing potential demand for bridge loans, is limited. GAO is making five recommendations to SBA, including that it more comprehensively document risks and plans to mitigate these risks and evaluate the implementation of the Express Bridge Loan Pilot Program. Overall, SBA agreed with the recommendations, but described actions that partially address the recommendation for evaluating the pilot. GAO maintains it should be fully implemented, as discussed in the report.", "document_type": "gao"}
{"report": "DOD currently relies on more than 900 commercial industry transportation service providers (TSPs) to move and store servicemembers’ and their families’ household goods. About 40 percent of DOD’s annual household goods moves occur during the 14-week annual peak moving season, which runs from May 15 through August 31. As figure 1 shows, servicemember survey satisfaction scores declined in 2018 during the peak moving season. According to TRANSCOM officials, this decline in satisfaction scores during the peak moving season has persisted for years. Under the current DP3, the military services own, operate, and staff most of the infrastructure involved in managing and overseeing the movement and storage-in-transit of DOD’s household goods, including the personal property processing offices and personal property shipping offices. The military services’ processing offices, among other things, ensure that the servicemembers’ reassignment orders and paperwork authorizing a move are in order, and that their application is accurate and complete. The processing offices also provide counseling, including advising servicemembers on the DP3 process, entitlements, and restrictions so they can make informed decisions about their moves. The military services’ shipping offices work with TSPs to schedule moves, monitor TSP performance, and take or recommend punitive action against poor performing TSPs. DOD has conducted and sponsored several studies to help address persistent DP3 performance issues, including those related to servicemember satisfaction and challenges with meeting capacity demands during the peak summer months. For example, LMI and the Institute for Defense Analyses (IDA) produced reports on behalf of DOD in 2012 and 2018, respectively, to analyze DP3 performance issues and make recommendations to improve DP3. In its report, LMI outlined the impact of a range of options for contracting out some or all of the management of DOD’s household goods movement and storage activities. IDA recommended that DOD create consistent performance metrics, unify its DP3 operating structure, and improve servicemember support. Concurrent with pursuing its Global Household Goods Contract, TRANSCOM has several ongoing initiatives intended to improve servicemember satisfaction in general, and household goods loss and damage specifically. According to survey data, loss of and damage to household goods is the number one reason servicemembers cite for dissatisfaction. Ongoing TRANSCOM initiatives to improve servicemembers’ moving experiences that are independent of the Global Household Goods Contract include the following: Increasing the use of shipping containers in domestic household goods moves to reduce the number of servicemember loss and damage claims. According to TRANSCOM officials, in the past TRANSCOM has primarily used shipping containers only for international moves. Replacing the internet-based system used to manage DOD household goods moves, the Defense Personal Property System, with a new, mobile-friendly information technology operating system platform called MILMOVE. MILMOVE is intended to provide servicemembers and their families with more reliable information about their planned and ongoing household goods moves through access to all phases of the move process via their personal smartphones or tablets. Tracking whether quality inspections occur in person or via telephone, and requiring that at least 50 percent of all shipments have an in- person quality inspection. Increasing the liability limits for damaged or lost household goods shipments to $6 per pound and the total loss cap to $75,000 per shipment, when claims are filed within a specified time. In November 2018, TRANSCOM, in coordination with OSD and the military services, began planning and developing requirements for the Global Household Goods Contract. According to TRANSCOM officials, the goal was to have the contract in place in time for the 2021 peak moving season. TRANSCOM announced that in April 2020 it would award a Global Household Goods Contract to a single commercial move manager to oversee DP3 activities that relate to the movement and storage-in-transit of household goods. The proposed contract includes an initial 9-month transition period that will commence in May 2020, a 3-year base period, three 1-year option periods, two 1-year award terms, and an option to extend the contract 6 months. Including all option periods and award terms, the contract is expected to be completed at the end of January 2029 or, if the option to extend services is exercised, by the end of July 2029. The 9-month transition period is intended to give the Global Household Goods contractor sufficient time to, among other things, integrate its information technology systems with existing DOD information technology systems. During the initial years of the contract, the contractor’s volume of workload will incrementally increase, including the volume of household goods moves that it will handle. The Global Household Goods Contract is not intended to replace DP3 in its entirety. Activities that are planned to be a part of the contract include the movement and storage-in-transit of household goods, direct procurement method shipments, and some level of servicemember counseling. DP3 activities that are not to be a part of the Global Household Goods Contract include the long-term storage of household goods (referred to as non-temporary storage) and the movement and storage of servicemembers’ privately owned vehicles. Additionally, while the contract is expected to include counseling to servicemembers, the military services will also perform some counseling. According to TRANSCOM and military service officials, they are working to determine the amount of counseling that will be retained by the military services. The movement and storage-in-transit of household goods in the current DP3 differs in key ways from the approach under the planned Global Household Goods Contract, as shown in table 1. TRANSCOM has developed preliminary and refined cost estimates for determining the cost implications associated with moving to a DP3 that incorporates the Global Household Goods Contract. According to TRANSCOM officials, they began developing preliminary cost estimates in December 2018 in order to (1) create a baseline cost estimate for the movement and storage of household goods under the current DP3, (2) create a cost estimate for those DP3 activities that will be a part of the Global Household Goods Contract, and (3) serve as a point of comparison between the two estimates to determine the cost implications of moving to the Global Household Goods Contract. TRANSCOM’s preliminary cost estimates for the current DP3 and for the activities that will be a part of the Global Household Goods Contract included costs associated with the movement and storage of household goods and government personnel costs associated with the military services’ personal property processing and shipping offices. TRANSCOM developed these preliminary cost estimates in part by requesting and collecting information from the military services about their current costs under DP3. Some DP3 costs were not included in the preliminary cost estimates, such as infrastructure and vehicle costs because these assets are used to support multiple programs and could not be easily isolated. However, our assessment of the preliminary DP3 cost estimates associated with moving to the Global Household Goods Contract found that TRANSCOM may not have accurately calculated some costs because of unanswered questions about how certain activities will be performed. TRANSCOM’s cost estimates associated with the number of government personnel that (1) will counsel servicemembers and (2) oversee contractor performance under the contract had weaknesses. Specifically, these estimates relied on assumptions that may have resulted in over- or underestimating some costs. First, TRANSCOM’s preliminary DP3 cost estimates associated with moving to the Global Household Goods Contract assume that DOD’s costs related to government personnel will be equivalent to those under the current DP3. However, TRANSCOM officials acknowledge that this assumption is based on discussions with military service officials that occurred before the Global Household Goods Contract draft request for proposals was issued in April 2019. Furthermore, military service officials told us they have not decided how much of the counseling function their service has provided will be moved to the planned contract, and the services’ approaches will likely differ. TRANSCOM officials pointed out that while the Global Household Goods contractor will perform some level of servicemember counseling, the contract will allow the individual military services to decide how much of the counseling responsibility to retain. Second, TRANSCOM’s cost estimates do not account for the number of DOD contracting officer’s representatives and quality assurance evaluators required to oversee contractor performance under the contract. According to TRANSCOM officials, the command’s preliminary cost estimates assume that government personnel who have been relieved of servicemember counseling responsibility will perform contract oversight and quality assurance responsibilities under the planned contract. For example, TRANSCOM officials told us that quality assurance inspectors under the current DP3 will transition to serve as quality assurance evaluators once the contract is in place. However, TRANSCOM officials acknowledge that it is possible that workload (and personnel costs) for the military services under the current DP3 will change under the planned contract. Moreover, if government personnel do switch roles and responsibilities, TRANSCOM did not take into account the cost to transition and train personnel to execute their contracting officer’s representative and quality assurance evaluator responsibilities under the contract. TRANSCOM officials told us that they were unable to determine with any precision the number and associated costs of government personnel required to counsel servicemembers and oversee the contract because of the fractured nature of the current DP3. Nonetheless, in September 2019, TRANSCOM tasked LMI with developing a BCA with refined cost estimates. LMI finalized the BCA and we received it on January 17, 2020. When we assessed the BCA, we found that like us LMI determined that it did not have complete information to fully calculate the cost of a DP3 that incorporates the Global Household Goods contract. Specifically, LMI acknowledged some of the same limitations in its cost estimates that we identified in TRANSCOM’s preliminary cost estimates, such as uncertainty about the number of government personnel required to oversee the Global Household Goods Contract. For example, LMI states in its BCA that the Global Household Goods Contract may reduce DOD staffing requirements for functions such as counseling; however, roles such as quality assurance evaluators and contracting officer’s representatives to oversee contractor performance may increase DOD staffing requirements. When we raised concerns about these unanswered questions and their potential cost implications with TRANSCOM officials, they told us that the move to the Global Household Goods Contract is less about saving money than it is about representing the best value to DOD when both cost and program performance are considered. They also told us they have developed a plan of action for the phase-in of the Global Household Goods Contract, and that plan includes conducting a manpower study during the third year of the contract. As described earlier in this report, the initial 3 years of the contract involve a gradual phase-in of household goods move volume for the contractor, and in the third year the contractor will be responsible for all of DOD’s household goods move volume. According to these officials, by waiting until year 3 of the contract to conduct a manpower study, DOD will have data to more precisely determine the number (and cost) of government personnel required to counsel servicemembers and oversee the contract, such as contracting officer’s representatives and quality assurance evaluators. However, we have determined that TRANSCOM does not have a process in place to track data that would inform its manpower study in year 3, such as how many personnel within each military service are needed to perform contract oversight duties and the costs associated with these personnel. We have reported that organizations should determine their personnel requirements as part of a systematic requirements- determination process that includes (1) identifying an organization’s mission, functions, and tasks and (2) determining the minimum number and type of personnel—military, civilian, and contractor—needed to fulfill those missions, functions, and tasks by conducting a workforce analysis. Without a way to track key data, DOD risks conducting a manpower study that does not allow it to fully understand the personnel and cost implications of its move to a DP3 that incorporates a Global Household Goods Contract. Notably, in its BCA LMI recommended that DOD reexamine the BCA when additional relevant information becomes available, such as when DOD completes its manpower study. TRANSCOM officials told us they are in agreement with this recommendation. However, if TRANSCOM does not have a process in place to collect key pieces of data during the first 3 years of the contract, a reexamination of the BCA will be less fruitful than it otherwise might be. First, TRANSCOM has developed performance metrics—referred to as performance indicators—for its Global Household Goods Contract, and has developed performance metrics for some, but not all, activities that fall outside of the contract. TRANSCOM’s draft quality assurance surveillance plan for the Global Household Goods Contract, which was developed in coordination with the military services, outlines how the contractor’s performance will be assessed against performance indicators. These performance indicators set measurable standards for, among other things, information technology systems’ availability, claims settlement timeliness for lost and damaged goods, and the timeliness of household goods pick-up and deliveries. Examples of performance indicators include: settling 90 percent of all loss and damage claims valued at less than $1,000 within 30 days and settling 95 percent of all claims, regardless of value, within 60 days; delivery of household goods on the scheduled date 95 percent of the time per month; and overall customer satisfaction rating of satisfactory at least 95 percent per month. Based on a DP3 briefing TRANSCOM provided us, the command has also developed metrics for assessing other contracted DP3 activities, such as the transport of privately owned vehicles and non-temporary storage, which TRANSCOM captures under contracts separate from the Global Household Goods Contract. For example, TRANSCOM tracks the number and cost of non-temporary storage lots maintained annually to support its storage requirements. However, TRANSCOM officials acknowledge that the command has not developed metrics for other activities that government personnel will continue to perform in DP3, once a Global Household Goods Contract is in place, such as servicemember counseling and claims resolution. Although TRANSCOM’s draft quality assurance surveillance plan includes a performance indicator to assess the contractor’s performance with respect to servicemember counseling, TRANSCOM officials told us the command has not developed commensurate metrics to assess the military services’ performance in providing servicemember counseling. TRANSCOM officials also noted that while the military services’ claims offices will be responsible for handling unresolved loss and damage claims between the servicemember and the contractor, the command has not developed metrics associated with this DP3 activity. Second, we found that, while TRANSCOM has articulated overarching DP3 goals, it has not clearly articulated how its performance metrics align with each of these goals. According to TRANSCOM officials, they decided to move certain DP3 activities to the Global Household Goods Contract because doing so could positively impact the program’s five goals regarding cost, quality, capacity, accountability, and responsibility, terms described in LMI’s BCA as shown below: Cost considers the potential financial impact, including either opportunities or risks to the future-state costs and any savings the program is likely to achieve. Quality encompasses the value of a move, typically measured by customer (i.e., servicemember) satisfaction. Quality may improve with on-time performance and minimized loss or damage. Capacity is the availability of industry providers to meet the program’s demands at a given time. This can be during peak or non-peak season. Accountability refers to the government’s ability to affix responsibility to the contractor and their supplier network for performance. Responsibility encompasses the authority of the government and how each government stakeholder will be held responsible for accomplishing their assigned tasks. Planned contract performance indicators and metrics are intended to assess contractor performance, but they may also provide information that TRANSCOM could use to more broadly assess the extent to which DP3 is meeting its overarching program goals. However, TRANSCOM has not clearly articulated which of the performance metrics it has established under the Global Household Goods contract align with which program goals. For example, one performance indicator for the Global Household Goods Contract is for contractor delivery of household goods to be on-time at least 95 percent of the time. While it appears that this performance indicator could relate to the quality and capacity goals, TRANSCOM has not established linkage between this performance indicator and either of these goals. Determining how it will assess improved capacity is particularly important. When we spoke with them about the move to the Global Household Goods contract, some military service officials, representatives of the moving and storage industry, and members of TRANSCOM’s Personal Property Relocation Advisory Panel told us that they doubt whether the Global Household Goods Contract will, indeed, improve capacity. Moreover, none of the existing performance indicators appear to relate to TRANSCOM’s responsibility goal. It is not surprising that the Global Household Goods Contract does not include performance indicators on how government stakeholders are to be held responsible for various program activities, because indicators under the contract are intended to evaluate contractor performance. However, we would expect to see performance metrics outside of the contract to measure government stakeholders’ performance. When we spoke with TRANSCOM officials about the gaps we identified in their performance assessment approach, they agreed that clarifying the linkage between performance metrics and the overarching program goals and ensuring there are performance metrics for assessing each of the goals would improve the command’s ability to assess overall program performance. While the performance indicators and metrics related to the various contracts, including the Global Household Goods Contract, are intended to assess contractor performance, they could also provide information that TRANSCOM could use to more broadly assess the extent to which DP3 is meeting its overarching program goals. DOD Instruction 4500.57, Transportation and Traffic Management, calls for TRANSCOM, in coordination with DOD components, to conduct annual program reviews to ensure the overall effectiveness of the DP3. The instruction calls for these reviews to include a metrics-based evaluation of the program, and assessments of TSP and service-provider cost and performance, information technology systems and contracts that support DP3, and external factors that impact the program such as industry capability and changes to servicemember shipping entitlements. Moreover, the Standards for Internal Control in the Federal Government call for management to define objectives (or metrics) in terms of what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement so that performance toward achieving those objectives (or metrics) can be assessed. We have also previously reported on the importance of linking lower-order performance metrics and higher-order strategic goals to achieve desired outcomes. Without performance metrics that account for those DP3 activities that fall outside of the Global Household Goods Contract and other DP3 contracts and without clearly articulating the linkage between performance metrics and program goals, TRANSCOM’s ability to assess progress toward, and take actions to help achieve, its overarching program goals will be hindered. The need to take these steps is particularly important, given some of the skepticism that TRANSCOM faces with its move to a DP3 that incorporates the Global Household Goods Contract. Further, because TRANSCOM officials have determined that the move to the Global Household Goods contract is about delivering better value, rather than just saving money, it is imperative that they have a robust performance assessment approach in place. DOD has experienced long-standing quality issues moving and storing the household goods of servicemembers and their families, despite numerous reform efforts. To address persistent quality-of-service issues, such as late pick-up and deliveries and high claims costs for lost and damaged goods, TRANSCOM intends to award a multi-year Global Household Goods Contract, under which a single commercial move manager would oversee the movement and storage of household goods shipments. DOD has been working to award its Global Household Goods Contract in time to meet the fiscal year 2021 peak season demand, and will award the contract without precise information on the number and cost of government personnel required to counsel servicemembers and oversee the contract. Given this determination, it is particularly important that DOD put in place a process to track key data during the first 3 years of the contract to inform its planned manpower study, so that it can fully determine the cost implications of this shift in DP3. Moreover, because DOD has stated that the purpose of the move to the Global Household Goods contract is to provide better value for the customers, DOD should ensure that it has in place performance metrics to assess all DP3 activities, including those that fall outside of the Global Household Goods contract, and clearly articulate the linkage between performance metrics and overarching program goals. Without doing so, TRANSCOM will be hindered in its ability to assess whether a DP3 that incorporates the Global Household Goods Contract is an improved program, particularly as it relates to the moving experiences of servicemembers and their families. We are making the following three recommendations to the Secretary of Defense: The Secretary of Defense, in coordination with the Chairman of the Joint Chiefs of Staff, should ensure that the TRANSCOM Commander, in coordination with the military services and the Coast Guard, develop a process for tracking data during the first 3 years of the Global Household Goods Contract to inform its planned manpower study during the third year of the contract to more precisely determine DP3 manpower needs and associated costs. (Recommendation 1) The Secretary of Defense, in coordination with the Chairman of the Joint Chiefs of Staff, should ensure that the TRANSCOM Commander develop performance metrics for those DP3 activities that will not be a part of the Global Household Goods Contract, such as servicemember counseling and claims resolution that will, at least in part, continue to be performed by the military services. (Recommendation 2) The Secretary of Defense, in coordination with the Chairman of the Joint Chiefs of Staff, should ensure that the TRANSCOM Commander articulate the linkage, where appropriate, between DP3 performance metrics, including Global Household Goods Contract performance indicators, and overarching program goals. (Recommendation 3) We provided a draft of this report to DOD for comment. In its comments, which are reproduced in Appendix III, DOD concurred with all of our recommendations and described ongoing and planned actions to address them. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense, the Chairman of the Joint Chiefs of Staff, and the Secretaries of the Army, the Navy, and the Air Force; and the Commandant of the Marine Corps. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our review focused on household goods movement and storage-in-transit for servicemembers in the Army, the Navy, the Air Force, the Marine Corps, and the Coast Guard. We did not evaluate Department of Defense (DOD) Defense Personal Property Program (DP3) activities that will not be a part of the Global Household Goods Contract, which include the long-term storage of household goods and the movement and storage of servicemembers’ privately owned vehicles. We did, however review metrics being tracked for these activities to determine the U.S. Transportation Command’s (TRANSCOM) ability to assess performance for the broader DP3. During our review, we coordinated with officials from the DOD Office of the Inspector General to gain an appreciation for the objectives, scope, and methodology of their ongoing audit on the timeliness of household goods deliveries and claims resolution under the current DP3. The resulting audit report contained several recommendations, including that TRANSCOM issue warnings or letters of suspension to transportation service providers (TSPs) within 14 days of missing the agreed-upon delivery date from storage, and that TRANSCOM help servicemembers and their families file inconvenience claims with TSPs within 14 days of a missed delivery date. To inform both of our objectives, we met with officials from the Office of the Secretary of Defense (OSD), TRANSCOM, the military services, the Coast Guard, and the DOD Office of the Inspector General. We also met with external stakeholders, including associations representing the moving and storage industry, the Small Business Administration, the American Federation of Government Employees, and members of TRANSCOM’s Personal Property Relocation Advisory Panel. For objective one, we assessed preliminary cost estimates TRANSCOM developed in connection with the move to a DP3 approach that incorporates the planned Global Household Goods Contract against best practices in the GAO Cost Estimating and Assessment Guide. The guide states that valid and useful historical data are important in developing sound cost estimates, and that risk and uncertainty and sensitivity analyses should be performed to mitigate the effects of changing assumptions. We analyzed TRANSCOM’s preliminary cost estimates, and discussed with TRANSCOM officials assumptions that were used to develop the estimates and techniques that were applied to account for variability in the assumptions. Given that TRANSCOM’s cost estimates were preliminary and likely to change based on the Logistics Management Institute’s (LMI) ongoing business case analysis (BCA), we did not conduct a reliability assessment of TRANSCOM’s cost estimates. We instead focused on TRANSCOM’s treatment of evolving assumptions in its preliminary cost estimates. We met with representatives from LMI to discuss their process for updating TRANSCOM’s preliminary cost estimates as part of a BCA that it was preparing for the command. We evaluated these cost estimates, including the cost estimates in the BCA that was issued in January 2020, using criteria from GAO’s Assessment Methodology for Economic Analysis that outlines five key elements for an economic analysis. Our assessment of LMI’s BCA is included in appendix II. For objective two, we reviewed relevant DOD guidance, including DOD Instruction 4500.57, Transportation and Traffic Management. The instruction requires TRANSCOM, in coordination with the DOD components, to annually evaluate the effectiveness of the DP3. We also reviewed the Global Household Goods request for proposals and TRANSCOM’s draft quality assurance surveillance plan. The draft quality assurance surveillance plan outlines how DOD will oversee the move manager’s performance under the Global Household Goods Contract and describes the performance indicators that will be used to assess contractor performance. Using the Standards for Internal Control in the Federal Government, which calls for management to define objectives (or metrics) in terms of what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement so that performance toward achieving those objectives can be assessed, we also assessed DOD’s DP3 performance metrics. Additionally, we discussed with TRANSCOM officials their approach for overseeing the Global Household Goods Contract, including the linkage between performance measures for those DP3 activities that will and will not be a part of the contract and DP3’s overarching goals. We conducted this performance audit from May 2019 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: GAO’s Assessment of the Logistics Management Institute’s (LMI) Business Case Analysis (BCA) We assessed the cost and benefit information in LMI’s BCA against our Assessment Methodology for Economic Analysis. Our assessment methodology identifies the following five key components of an economic analysis: We found that the BCA informs decision-makers and stakeholders, with caveats, about the economic effects of the proposed Global Household Goods Contract. Further, we found that the BCA fully met four and partially met one of the five key components of an economic analysis. A summary of our rationale for these assessments is outlined in table 2. Elizabeth A. Field, (202) 512-2775 or fielde1@gao.gov. In addition to the contact named above, GAO staff who made key contributions to this report include Marc Schwartz, Assistant Director; Pedro Almoguera, John Bumgarner, William Cordrey (retired), Tim DiNapoli, Jennifer Echard, Christopher Gezon, Mae Jones, Jason Lee, Ned Malone, Tara Porter, Oliver Richard, Mike Shaughnessy, Susan Tindall, Nate Tranquilli, John Van Schaik, and Mary Weiland.", "summary": "DOD, through its DP3, arranges for the movement and storage of about 400,000 personal property shipments of servicemembers and their families annually—40 percent of them during peak moving season. DOD has identified problems meeting peak season demand and addressing long-standing quality-of-service issues. TRANSCOM announced that in April 2020 it would award a Global Household Goods Contract to a single commercial move manager to oversee DP3 activities that relate to the movement and storage-in-transit of household goods. GAO was asked to evaluate matters related to DOD's plans to implement the Global Household Goods Contract. GAO assessed the extent to which TRANSCOM has (1) determined the cost implications of moving to a DP3 that incorporates the Global Household Goods Contract and (2) developed metrics to assess program activities and that relate to overarching DP3 goals. GAO evaluated TRANSCOM's cost estimates against the GAO Cost Estimating and Assessment Guide and a DOD business case analysis against GAO's Assessment Methodology for Economic Analysis. The U.S. Transportation Command (TRANSCOM) has developed cost estimates to assess the cost implications of adjusting the Defense Personal Property Program (DP3), its program to move and store servicemembers' household goods, to incorporate a single move manager approach through the Global Household Goods Contract. However, TRANSCOM may not have accurately calculated some Department of Defense (DOD) costs because of unanswered questions about how tasks related to counseling servicemembers and overseeing contractor performance will be performed. DOD plans to conduct a manpower study in the third year of the contract to determine the number and cost of government personnel required to perform these tasks. However, TRANSCOM does not have a process in place to track data over the initial years of the contract to inform its manpower study, such as the number and associated cost of military service personnel needed to perform contract oversight. We have reported that organizations should determine their personnel requirements by identifying the minimum number and type of personnel needed to fulfill their missions, functions, and tasks by conducting a workforce analysis. Without a way to track key data, DOD risks conducting a manpower study that would result in less than a full understanding of the personnel and cost implications of the move to the Global Household Goods Contract. TRANSCOM has developed performance metrics for assessing some, but not all, DP3 activities. For example, TRANSCOM has developed indicators for assessing contractor performance, including the timeliness of household goods deliveries under the Global Household Goods Contract. However, TRANSCOM has not developed metrics for other activities that DOD personnel will continue to perform at least partially once the contract is in place, such as servicemember counseling. Further, TRANSCOM has not articulated how existing metrics link to TRANSCOM's program goals that relate to servicemembers' household goods movement and storage experience (see fig.). Without developing performance metrics for all DP3 activities, and articulating the linkage between metrics and goals, TRANSCOM will have limited ability to assess whether a DP3 incorporating the new contract is an improved program for servicemembers. GAO makes three recommendations–that DOD collect and track data to more precisely determine DP3's manpower needs and costs, develop performance metrics for DP3 activities not part of the contract, and articulate the linkage between performance metrics and program goals. DOD concurred with all three GAO recommendations.", "document_type": "gao"}
{"report": "USPS’s mission is to provide universal postal service while operating as a self-financing entity, but USPS’s current financial position is not sustainable. To achieve its mission, USPS must cover its expenses through revenues generated from the sale of its products and services. However, USPS’s total operating expenses have exceeded total operating revenue each year since fiscal year 2007, including a $2.6 billion loss from operations in fiscal year 2017 alone (see fig. 1). Moreover, we have reported that USPS’s overall financial condition is deteriorating. For example, in August 2018 we reported that USPS had about $149 billion in unfunded liabilities and debt at the end of fiscal year 2017. As a result, USPS’s financial condition remains on our list of high- risk areas needing attention by Congress and the executive branch. According to USPS financial documents, its ability to sell innovative products and services will be a key factor in improving its financial condition. Thus, USPS established a strategic goal to “innovate faster to deliver value” to its customers, by making investments in innovations that respond to rapidly evolving customer needs. A key element of this effort is to accelerate testing of innovative products and services to better serve these needs, according to USPS. While USPS is allowed to develop certain new postal products and services, there are statutory restrictions that currently limit the range of innovations USPS can offer. For example, under current statute, USPS is not permitted to ship alcoholic beverages. Similarly, although USPS is explicitly authorized to provide services to federal executive agencies (e.g., passport services), such authorization does not include services to state, local, and tribal governments. Legislation has been introduced in previous sessions of Congress that would permit USPS to deliver alcoholic beverages and allow USPS to provide property and services to state, local, and tribal governments under certain conditions. According to USPS officials, USPS supports these legislative proposals, which could enhance its ability to offer innovative products and services. According to USPS officials, two USPS handbooks include policies applicable to piloting key innovations. Specifically, the first handbook includes requirements, procedures, and responsibilities for all types of investment programs and projects undertaken by USPS, regardless of size, cost, or complexity. This handbook, for example, requires the identification and documentation of lessons learned for all investments and projects. The second handbook includes requirements and procedures specifically for major operating expense investments. This handbook, among other things, establishes requirements and procedures meant to ensure that new and enhanced products and services consistently meet customer needs, generate new revenue, and strengthen USPS as a business. This handbook further states that USPS has a responsibility to subject new initiatives to rigorous financial analysis, testing, and measurement, to determine whether these initiatives will make a positive financial contribution to the organization and ensure that USPS’s leadership has appropriate information for effective decision- making. USPS’s Office of Product Innovation generally has lead responsibility for piloting innovations. According to USPS’s policies, a project manager is responsible for establishing and coordinating a cross-functional team to design and evaluate the pilot. This team typically includes officials from a variety of USPS departments, such as finance, general counsel, information technology, marketing, and operations. The project manager, with support from the cross-functional team, is responsible for preparing a proposal for the pilot that includes key information, such as the pilot’s objectives and performance measures, and overseeing pilot implementation and communication with key stakeholders. In some cases, PRC has a role in overseeing postal innovation pilots. For example, USPS must notify PRC before it pilots any postal product innovation for which it will impose a price (i.e., a pilot that generates revenue for USPS) and must subsequently report quarterly revenue, volume, and cost data. PRC also ensures that certain safeguards are maintained during the pilot, such as limitations on the pilot’s duration and revenue. However, according to PRC officials, the commission has limited involvement in other areas of USPS’s efforts to develop innovations. USPS piloted 24 key innovations from fiscal years 2013 through 2017. For example, USPS piloted an Identity Verification Service that allows users to verify their identity either remotely (i.e., online) or in person at a postal facility. Similarly, USPS piloted an innovation to allow mailers to print shipping labels, track packages, and schedule package pick-ups by accessing USPS data. The primary goal of the majority of these key innovations (16 of 24) was to generate revenue, while the primary goal for the remaining innovations was generally to improve customers’ experience using USPS products or services (see fig. 2). Appendix I includes a complete list of key innovations USPS piloted from fiscal years 2013 through 2017. The following discussion provides additional information about the 4 key innovations we selected as illustrative examples of USPS’s efforts to pilot innovative products and services. Same-Day Delivery: From December 2012 to December 2015, USPS piloted same-day delivery for consumer e-commerce purchases (see fig. 3). According to the pilot proposal, this innovation was intended to generate revenue for USPS by allowing it to leverage its existing delivery infrastructure to capture part of the growing e-commerce market. To determine the potential scalability of same-day delivery, USPS first tested its operational feasibility and potential demand in several major metropolitan areas, including San Francisco, New York, and Phoenix. During the pilot, USPS delivered photos, chocolates, water, electronics, and other goods from 38 participating mailers to consumers in these areas. At the pilot’s conclusion, USPS decided to continue offering same-day delivery to interested participating mailers under Priority Mail contracts. Grocery Delivery: From November 2014 to October 2017, USPS piloted a grocery delivery product in nine selected metropolitan areas. According to USPS, the innovation was intended to generate additional revenue by taking advantage of the growing market for grocery delivery. To test the innovation’s operational feasibility, USPS required the pilot’s sole participating mailer to bring totes containing groceries and other prepackaged goods ordered by customers directly to post offices (see fig. 4). USPS was then responsible for sorting the totes and delivering them to customers. According to its proposal for this pilot, USPS expected grocery delivery to provide a substantial revenue generation opportunity. At the pilot’s conclusion, like same- day delivery, USPS decided to continue offering grocery delivery with the participating mailer under a Parcel Select contract. Informed Delivery: From spring 2014 through July 2016, USPS piloted a notification service called Informed Delivery in Northern Virginia and New York. According to USPS, this innovation is intended to bridge the gap between the physical and digital worlds by, for example, emailing customers with a scanned image of the exterior address side of letter-sized mail they should receive later that day (see fig. 5). Informed Delivery can also allow mailers to conduct marketing campaigns by integrating other elements—such as hyperlinks to mailers’ websites—into the email and other notifications that customers receive. In its proposal to pilot Informed Delivery, USPS stated the pilot was intended to help USPS understand the service’s business opportunity and increase the certainty of its potential benefits, which included retaining mail volume and generating new revenue from large advertisers. In addition, the pilot aimed to generate “statistically valid data” on how subscribers respond to marketing campaigns that mailers conduct. According to USPS, more than 70,000 customers were actively using the service at the pilot’s conclusion. In July 2016, USPS decided to end the pilot and launch the service nationally. According to USPS, about 13 million customers were subscribed to the service as of October 2018. USPS aims to have 40 million customers subscribed to the service by 2020. Keyless Parcel Lockers: Since October 2013, USPS has piloted keyless parcel lockers that allow customers to independently pick up packages in 98 selected post offices. According to USPS, among other things, this innovation is intended to reduce the number of missed package deliveries to customers’ post office boxes and thereby reduce USPS’s delivery costs (see fig. 6). The purpose of the pilot is to assess the performance and use of the lockers and to assess their performance. In October 2013, USPS began pre-testing the technical performance of 10 prototype keyless parcel locker units at post offices in New York City and Northern Virginia. Following this pre-test, in February 2015, USPS approved the installation of 50 made-to-order locker units in selected post offices across the country. Finally, in May 2016 USPS expanded the pilot to include an additional 50 units, including 2 units that a senior USPS official told us were not yet installed. As of November 2018, the pilot is still ongoing. USPS’s policies applicable to piloting key innovations fully reflect two of the five leading practices for pilot design and evaluation that we identified in prior GAO work and relevant standards for internal control (see table 1). These policies do not, however, fully reflect the other three leading practices due to policy gaps. Further, we found that USPS had not consistently followed its policies to document lessons learned at the conclusion of each pilot, as discussed below. Senior USPS officials acknowledged that gaps exist in its policies for pilot design and evaluation because they were not developed by USPS to fully reflect all leading practices. These policy gaps limit the extent to which USPS can ensure that it is making good resource allocation decisions based on pilot experiences. Below we further discuss the extent to which USPS’s policies reflect the five leading pilot practices we identified as well as how USPS applied these leading practices among the four piloted innovations that we reviewed. Establish appropriate and measurable objectives linked with identified performance measures: We found USPS’s policies do not fully reflect this leading practice. While USPS policies require that project managers establish pilot objectives and performance measures, they do not require that each objective be linked with identified performance measures. As a result, some pilots may have objectives without an associated performance measure. For example, although USPS established both objectives and performance measures for each of the four innovations we selected for review, it did not consistently link each established objective to performance measures. USPS’s proposal to pilot same-day delivery, for example, had objectives of generating new revenue and improving customers’ experience. However, while the proposal included performance measures associated with generating new revenue—i.e., package volume, gross revenue, and net revenue—it did not identify and link any performance measures with its objective of improving customer experience. Similarly, USPS’s proposal to pilot keyless parcel lockers included improving customers’ experience as one of its objectives. However, while the proposal included a variety of performance measures—reduction in the number of missed deliveries to post office boxes, locker rate utilization, and on-time locker installation—it did not identify and link any performance measures with its improving customers’ experience objective. Absent such measures, USPS may not know whether customers have experienced an improvement using keyless parcel lockers compared to using manual, keyed parcel lockers. Linking all objectives to performance measures could help ensure that USPS has the performance information to assess the extent to which a pilot has achieved all of its objectives. USPS officials told us that it can be difficult to measure performance for some objectives related to customer experience. While measuring customers’ experience can be challenging, it is important to understand the extent to which a pilot has achieved all of its objectives. Further, USPS has demonstrated that it can measure improvement in customers’ experience. For example, during its Informed Delivery pilot, USPS conducted a consumer survey with approximately 5,500 Informed Delivery subscribers to collect data on consumer adoption and satisfaction. In the survey, USPS found that over 80 percent were satisfied or very satisfied with the service. According to USPS officials, this data helped USPS to measure the pilot’s success in meeting its objective of improving customers’ experience. Articulate a methodology for evaluating pilot performance: We found that USPS’s policies fully reflect this leading practice because the policies require officials to develop and communicate a methodology for evaluating pilot performance. Articulating such a methodology helps managers to identify the types and sources of performance information necessary to evaluate the pilot. USPS’s policies require project managers to work with the pilot’s cross-functional team to develop and reach consensus on the methodology. These policies also require the project manager and cross-functional team to identify data needs, data sources, and how the data will be evaluated. For the four innovations we reviewed, we found that USPS articulated a methodology for evaluating the pilot’s performance. For example, for the Informed Delivery pilot, USPS identified its customer registration system as the method for tracking progress toward performance measures related to the number of Informed Delivery subscribers. Similarly, in its proposal to pilot keyless parcel lockers, USPS identified its central parcel locker monitoring system as a method of tracking progress toward performance measures related to utilization of keyless parcel lockers. Evaluate pilot performance and identify and document lessons learned: We found that USPS’s policies fully reflect this leading practice, but USPS did not consistently follow its policy that requires documenting lessons learned. Specifically, the policies require project managers to evaluate performance and document lessons learned at the conclusion of each pilot. Doing so can enable USPS to identify information needed to make conclusions about the pilot’s scalability and ensures that such information will be accessible to inform future related efforts. However, among the key innovations we selected for review USPS had not consistently documented lessons learned (see table 2). USPS officials told us that they discussed lessons learned during ongoing monitoring of pilot performance for these innovations, but had only documented lessons learned for its Informed Delivery pilot. Specifically, USPS identified lessons learned in its July 2016 proposal to launch its Informed Delivery service nationally. In this proposal, we found that USPS identified some lessons learned about the pilot related to user satisfaction and adoption rates. USPS officials told us that this information helped to inform USPS’s decision to launch the service nationally. However, USPS officials acknowledged that USPS did not document lessons learned for the other two concluded pilots that we selected for review (same-day delivery and grocery delivery). Senior USPS officials told us that USPS had not consistently documented lessons learned at the conclusion of pilots across the 24 key innovations because it had not developed tools, such as a template, or training that could help ensure such consistency. Without consistently documenting lessons learned for all of its pilots, USPS risks losing information garnered during pilot implementation that could be relevant to future innovation efforts. Doing so can be particularly important because, according to a senior USPS official, officials responsible for pilot projects sometimes retire or leave USPS for employment elsewhere, creating a gap in knowledge of pilot experiences. Standards for internal control underscore the importance of maintaining documentation in order to retain organizational knowledge and mitigate the risk of having knowledge limited to a few personnel. Draw and document conclusions about scalability based on pilot results: USPS’s policies do not fully reflect this leading practice. These policies require that project managers draw conclusions based on the results and lessons learned from the pilot. According to USPS officials, conclusions may include determining scalability—i.e., whether, how, and when to integrate pilot activities into overall efforts. However, USPS’s policies do not specifically require that officials document these conclusions. Documenting conclusions about scalability based on pilot results helps to ensure retention of organizational knowledge related to the pilot that may inform future decisions. Among the three innovations that we selected for review for which the pilots had concluded (i.e., same-day delivery, grocery delivery, and Informed Delivery), USPS officials told us that senior leadership discussed the results and lessons of the pilots and made determinations regarding whether, how, and when to launch them more broadly, but that they did not document these decisions or the rationale for them. By not documenting conclusions, USPS risks losing information that could affect the success of future related efforts and that could inform future USPS leadership of the rationale for maintaining investments in activities upon which pilots were based. Documenting conclusions for innovation pilots can be especially important in cases in which USPS decides to continue or expand pilot activities even when the pilots do not meet all of their intended objectives. For example, USPS’s same-day delivery and grocery delivery pilots had revenue objectives, along with associated performance measures; however, neither pilot achieved these objectives. For the same-day delivery pilot, costs exceeded revenue in 12 of the 13 fiscal year quarters in which the pilot was conducted, according to data USPS reported to PRC. Likewise, USPS data indicate it did not reach its annual revenue target for its grocery delivery pilot. Similarly, USPS’s pilot of Informed Delivery was intended to generate “statistically valid data” on how consumers respond to mailer marketing campaigns. However, according to a senior USPS official, the pilot did not generate the data as intended, because no such campaigns were conducted during the pilot. As discussed earlier in this report, USPS did not discontinue any of these three selected innovations when their pilots concluded. Although USPS may have had good reasons to continue with, or more broadly launch, these innovations despite the pilots not meeting all of their objectives, the lack of documentation regarding its reasoning and decisions limits information relevant to whether USPS is making judicious use of limited resources. Ensure appropriate two-way communication at all stages of the pilot with key internal and external stakeholders in order to understand and address their views: USPS’s policies do not fully reflect this leading practice. USPS’s policies require the involvement of key internal stakeholders in pilots. Specifically, USPS’s policies require the involvement of cross- functional teams—which include legal, finance, and other departments— and varying levels of review during the design and implementation of pilot proposals. However, USPS’s policies do not address communication with key external stakeholders. According to USPS officials, some pilot projects may be confidential or have limited or no direct effect on external stakeholders and, thus, communication with external stakeholders may not be appropriate. While external stakeholder communication may not be appropriate with some pilots, such communication, as appropriate, can help to ensure that issues critical to the success of a pilot activity are identified and addressed. Among the innovations that we selected for review, USPS officials explained various steps taken to involve internal stakeholders in the design and evaluation of the pilots, such as the involvement of cross functional teams to develop pilot proposals. Further, while USPS’s policies do not address external stakeholder communication, we found that USPS employed strategies for some of the innovations we selected for review to communicate with some external stakeholders—i.e., industry associations and mailers. For example, a representative of a postal association told us that USPS shared information and sought input about its Informed Delivery pilot during a quarterly meeting with industry groups. Similarly, a mailer we interviewed told us that USPS had shared information and sought input on the Informed Delivery pilot through direct outreach with the mailer. However, USPS did not consistently employ strategies to communicate with some key external stakeholders among the innovations that we selected for review. Specifically, USPS did not design or implement strategies to obtain feedback from consumers on its pilots for same-day delivery, grocery delivery, or keyless parcel lockers, despite the fact that each of these innovations directly affected consumers. In contrast, as previously discussed, for its Informed Delivery pilot, USPS planned and conducted a survey to obtain consumer feedback, the results of which helped USPS project managers support the proposal to expand the service nationally. Absent communication with all key stakeholders, USPS risks not having a complete understanding of perspectives that could inform the viability of its innovations. In recent years, USPS has sought to compete in a challenging business environment by piloting innovations intended primarily to generate revenue and enhance customers’ experience. The policies that USPS uses for piloting key innovations fully reflect some leading practices for pilot design and evaluation, such as articulating a methodology for evaluating pilot performance. However, addressing gaps between USPS’s policies and leading practices related to linking objectives and performance measures, documenting conclusions, and communicating with key external stakeholders would enable USPS leadership to better assess the outcomes of its pilots, understand the rationale for conclusions about scalability based on pilot results, and gauge customers’ reactions to innovative products and services. Moreover, developing tools or training to ensure that USPS consistently implements its policy of documenting lessons learned from pilots would provide USPS with key information to inform future related efforts. We are making the following two recommendations to USPS: The Postmaster General should direct the Vice President of Product Innovation to develop policies that fully reflect leading practices for pilot design and evaluation in areas such as linking objectives and performance measures; documenting conclusions about scalability based on pilot results; and communicating with key external stakeholders, as appropriate. (Recommendation 1) The Postmaster General should direct the Vice President of Product Innovation to develop tools, such as a template, or training to help ensure USPS consistently documents lessons learned at the conclusion of pilots, as required by USPS policies. (Recommendation 2) We provided a draft of this product to USPS and PRC for comment. USPS provided a written response, which is reproduced in appendix II of this report. In its response, USPS did not state whether it agreed with our recommendations, but described actions that it plans to take related to each. These actions, if fully implemented, would meet the intent of our recommendations. For example, USPS stated that it would develop policies specifically for pilot design, and would reflect leading practices for pilot design and evaluation based upon best practice research. USPS also noted that it would develop training to ensure consistent documentation of lessons learned from its pilots. USPS added that this planned training would cover best practices for pilot tests. Regarding our first recommendation USPS said that pilots are only one step in a larger process for developing innovations. We agree with this and noted in our report that piloting is one key element of USPS’s efforts to innovate. Nonetheless, given USPS’s financial position, effectively piloting innovations is a critical step to ensure that USPS invests its limited resources on innovations that are most likely to improve its long- term viability. USPS also stated that flexibility is important in innovation pilots, particularly as it pertains to linking pilot objectives with performance measures. We continue to believe that linking objectives with performance measures is key to effectively evaluating pilots. In so doing, however, there is flexibility to adjust pilot objectives and performance measures as new information is gleaned during the pilot. Finally, with regard to communication with external stakeholders during pilots, USPS said that it communicates consistently with external stakeholders regarding pilots at Mailers’ Technical Advisory Committee meetings (MTAC). In our report, we noted that USPS employed strategies to communicate with some external stakeholders—i.e., industry associations and mailers. We continue to believe, however, in the importance of communication with all key external stakeholders, which may include stakeholders, such as consumers, that do not participate in MTAC meetings. USPS and PRC also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committee, the Postmaster General, Chairman of PRC, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or RectanusL@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. The U.S. Postal Service (USPS) piloted 24 key innovations from fiscal years 2013 through 2017 (see table 3). In addition to the individual named above, Derrick Collins (Assistant Director); William Colwell and James Leonard (analysts in charge); Barbara El Osta; Geoffrey Hamilton; Gina Hoover; Anthony Jackson; and Laurel Voloder made key contributions to this report.", "summary": "USPS faces a challenging business environment that has led to reduced demand for its traditional services and significant financial losses. USPS aims to address this challenge by offering innovative products and services. The success of these efforts will depend, in part, on how effectively USPS tests each innovation's performance on a small scale to determine whether, how, and when to launch an innovation more broadly—a practice known as “piloting.” GAO was asked to review USPS's efforts to develop postal innovations. This report (1) describes key innovations that USPS recently piloted and (2) examines the extent to which USPS's policies reflect leading practices for pilot design and evaluation. GAO analyzed information on USPS pilots from fiscal years 2013 through 2017; compared USPS policies for piloting innovations to leading practices for pilot design and evaluation in prior GAO work and relevant standards for internal control; and selected four key innovations based on various characteristics (e.g., innovation type) to serve as illustrative examples of USPS's piloting efforts. From fiscal years 2013 through 2017, the U.S. Postal Service (USPS) piloted 24 key innovations intended primarily to generate revenue or improve customers' experience. The following four selected innovations illustrate these efforts: Same-Day Delivery: USPS delivered goods consumers bought online or in stores. The pilot sought to test the product's feasibility and revenue potential. Grocery Delivery: USPS delivered groceries to consumers in metropolitan areas. The pilot sought to test the product's feasibility and revenue potential. Informed Delivery: USPS emailed customers an advance image of the mail they would receive. The pilot sought to test the service's potential benefits, such as generating new revenue from advertisers that may use the service. Keyless Parcel Lockers: USPS is testing lockers where customers can independently pick up packages at post offices. The pilot seeks to test the service's operation and potential benefits for USPS and customers. USPS's policies for piloting innovations do not fully reflect the five leading practices for pilot design and evaluation identified in GAO's prior work. The policies fully reflect two of the leading practices because they require articulating a methodology for evaluating pilot performance and documenting lessons learned. The policies do not fully reflect the other three practices because they do not require: (1) linking pilot objectives to identified performance measures; (2) documenting conclusions based on pilot results; or (3) communicating with key external stakeholders, as appropriate. These policy gaps limit the extent to which USPS can ensure that it is making good resource allocation decisions based on pilot experiences. For example, GAO found that USPS did not document its conclusions based on the results of its pilots of same-day delivery, grocery delivery, and Informed Delivery. Documenting conclusions can be especially important when USPS continues to offer the product or service after the pilot has concluded, even though the pilot did not achieve all of its objectives, as was the case with these three innovations. Further, while USPS's policies require documenting lessons learned from its pilots, USPS did not do so for some pilots GAO reviewed. Senior USPS officials said that USPS did not consistently follow this policy because it had not developed tools or training that could help ensure such consistency. As a result, USPS risks losing information that could be relevant to future innovation efforts. GAO recommends that USPS (1) develop policies that fully reflect leading practices for pilot design and evaluation and (2) develop tools or training to ensure consistent documentation of lessons learned from pilots. USPS neither agreed nor disagreed with the recommendations but described actions it plans to take related to each.", "document_type": "gao"}
{"report": "Ballistic missiles, which foreign adversaries generally use as a deterrent or instrument of coercion, are becoming increasingly important weapons to support military and political objectives. These weapons continue to proliferate and show advances in mobility, reliability, in-flight maneuverability, accuracy, and ability to reach longer distances. According to the defense intelligence community, there has been a dramatic increase in ballistic missile capabilities over the last decade, and the over 20 countries that already possess ballistic missiles are likely to pursue further expansions in their quantities and capabilities. Figure 1 shows the lineup of operational ballistic missiles from North Korea and Iran, two of the various countries that pose threats to the United States and its allies and are of concern to the BMDS. Ballistic missile threats are generally categorized by their range (i.e., ground distance covered between the launch point and impact of the missile) as shown in figure 2 below. The configuration of a ballistic missile is also largely determined by the range a missile is expected to travel. For example, longer range ballistic missiles typically have two or three distinct sections, known as stages, that separate during flight and each has an independent propulsion system to ensure the warhead reaches its target. Shorter range ballistic missiles generally only have one section, or a single stage, that remains intact until the warhead reaches its intended target and detonates. Ballistic missiles may also carry countermeasures or adversaries may employ tactics and techniques, both of which are intended to confuse missile defense systems. For example, countermeasures can include penetration aids that are released during flight, such as decoys, which are intended to complicate the ability of missile-tracking sensors and missile defense interceptors to identify the warhead among the multiple objects. Challenging tactics and techniques can include structured attacks, such as simultaneously launching a number of missiles or outfitting a single missile with multiple warheads. In addition, some newer missiles are capable of traveling at greater speeds, performing maneuvers during all phases of flight, and remaining in the atmosphere for longer durations of their flight. These newer missiles, generally referred to as hypersonics, possess a combination of high speed, maneuverability, and relatively low altitude that can make them a challenging target for missile defense systems to track and engage. According to a publicly released intelligence assessment, nearly all adversaries that possess ballistic missiles have devised various means to confuse missile defense systems. In November 2010, the Defense Intelligence Agency (DIA) established the Defense Intelligence Ballistic Missile Analysis Committee (DIBMAC) to oversee and coordinate intelligence analysis and threat assessment production activities pertaining to foreign ballistic missile developments. Under the leadership of this committee, the defense intelligence community performs important stakeholder, advisor, and oversight functions in support of MDA’s acquisitions by (1) producing threat assessments; (2) providing advice on important threat-related issues pertaining to BMDS acquisition; and (3) validating threat models and reports. Table 1 provides further explanation of these roles and additional information on the defense intelligence community is in appendix I. In November 2013, DOD’s acquisition leadership issued a memorandum that requested DIA work with the acquisition community to produce more timely, relevant, and dynamic defense intelligence community threat assessments for DOD acquisition programs. The memorandum notes that DOD acquisition program officials expressed concerns about the timeliness of threat assessments due to the lengthy process and varying timelines that sometimes left them with threat assessments that did not contain the most up-to-date information. In addition, the defense intelligence community noted its concerns with the significant duplication in producing certain threat assessments, which placed a huge burden on its manpower and resources. Consequently, DOD leadership directed the acquisition customers and defense intelligence community to work together to improve threat assessments and in 2016 the defense intelligence community set forth its planned revisions to threat assessment processes and products. Subsequent revisions include creating a library of threat modules and replacing a former type of threat assessment with a new Validated Online Lifecycle Threat (VOLT) report, among others. These revisions were codified in the defense intelligence community’s policies in September 2016 and in DOD policy in August 2017. However, defense intelligence community officials noted that they are still in the process of implementing these revisions. MDA is developing a variety of missile defense systems, known as elements, including sensors, interceptors, and battle management and communication capabilities. The ultimate goal is to integrate these various elements to function as a layered system called the Ballistic Missile Defense System (BMDS). The BMDS elements, when integrated, are designed to destroy enemy missiles of various ranges, speeds, sizes, and performance characteristics in different phases of flight, as seen in figure 3 below. When MDA was established in 2002, the agency was granted exceptional flexibilities to diverge from DOD’s traditional acquisition lifecycle and defer the application of acquisition policies and laws designed to facilitate oversight and accountability until a mature capability is ready to be handed over to a military service for production and operational use. In particular, MDA was exempted from DOD’s standard requirements-setting process and instead uses a unique and flexible requirements-setting process that is intended to enable MDA to quickly develop and field useful but limited capabilities, which can be incrementally improved over time and adapted to address changes in the threat. MDA also implemented a tailored process that is intended to use defense intelligence community threat assessments in a way that enables the BMDS to defend against a broad range of uncertain and evolving threats. MDA uses defense intelligence community threat assessments as the foundation for developing threat models and establishing wide-ranging critical threat parameters upon which to design, develop, and test the BMDS. Specifically, MDA’s process includes the following: Design: MDA uses threat assessments to select a set of threat models in which it incrementally designs BMDS capabilities to defend against. MDA combines the capabilities from the selected threat models into parameters, forming what MDA refers to as the “parametric threat space.” MDA assigns subsets of the threat space to each of the BMDS elements to inform the design of their respective systems. Development: MDA assigns specific threat models to each of the elements for use in simulations as they are undergoing development. MDA uses these threat models to verify that the element’s system design has the capability necessary to defend against its assigned threat space. Test: Toward the end of BMDS element development, MDA coordinates with the warfighter and test and evaluation communities to select specific threat models for use in testing to assess the performance of the BMDS elements. MDA also uses its threat models to prepare for flight tests to help ensure that the BMDS elements have a high probability of achieving their test objectives, such as successfully intercepting the target. Operational capability: MDA uses its threat models as the foundation for algorithms, which are embedded into the BMDS to enable its sensors and interceptors to determine which object(s) amongst a group of objects (e.g., countermeasures, debris, etc.) is lethal. This capability is referred to as “discrimination.” Various challenges have recently emerged that have affected the availability of the threat assessments MDA needs to inform the agency’s acquisition decisions. Challenges include an upsurge in threat missile activity, which has increased the overall demand for threat assessments; a transition period as the defense intelligence community works through how to implement recent revisions to its processes and products; and MDA’s request for accelerated support from the defense intelligence community. Defense intelligence community officials say they are contending with all of these challenges without the provision of additional manpower or resources. Consequently, defense intelligence community officials have stated that their manpower and resources are constrained, which can affect the timely delivery of threat assessments to customers, such as MDA. If MDA does not have the threat information it needs when it is needed, the delay of information could result in setbacks for the agency’s weapon system design, development, and testing, or could put the agency in the position of moving forward without the requisite information, thereby increasing the risk of performance shortfalls and costly retrofits. However, MDA has opportunities to mitigate these challenges by collectively prioritizing its threat assessment requests and working through existing venues with the defense intelligence community to determine what additional resources may be needed to secure the accelerated support that it needs. One challenge for the defense intelligence community is a recent upsurge in threat missile activity, which has increased MDA’s requests for threat assessments. For example, ballistic missile flight testing has more than doubled from 2005 to 2016, from about 70 tests in 2005 to nearly 180 tests in 2016, and the most notable increases have occurred since 2010 (see figure 4). This upsurge of threat missile activity increases the urgency for the defense intelligence community to provide the requisite type of threat assessments to MDA to enable the agency to counter and defeat such threats; however, defense intelligence community officials have said that manpower and resource constraints have limited their ability to do so. In 2016, we reported on how the defense intelligence community’s manpower and resource constraints have impacted its ability to provide threat assessments. Since then, defense intelligence community officials have said that the manpower and resource constraints have not been resolved, but threat missile activity has increased. For example, some countries have recently displayed or flight tested new threat missiles capable of reaching the United States. When new threat missiles emerge, MDA requests missile-specific threat assessments—known as reference documents—from the defense intelligence community to understand their size, performance characteristics, and signature when detected by a sensor. This detailed information on the threat missiles enables MDA to build the threat models used to design, develop, and test BMDS weapon systems. Defense intelligence community officials have said that, although important, missile-specific threat assessments utilize considerable manpower and resources because they can be labor-intensive, lengthy, and take months, and at times a year or longer, to prepare. According to these officials, one way to minimize the workload and shorten the preparation timeframe is for MDA to differentiate the specific information that it needs from anything that might be extraneous. As a simplified and hypothetical example, defense intelligence community officials explained that MDA may only need some simple, general information about a missile or conversely it may need complex, highly-detailed information on everything about the missile from tip to tail. The amount of time and effort it would take defense intelligence community officials to gather the information in these two scenarios would vary significantly. MDA officials have acknowledged that some extraneous information may be gathered and included in these threat assessments but noted that, at the time they request a threat assessment, they may not yet fully understand what information is essential for their purposes. Therefore, they prefer to have as much information as possible, with the ability to determine whether and how to use it. Defense intelligence community officials, however, told us that they believe this is an inefficient use of their manpower and resources, especially given current constraints. Another challenge for the defense intelligence community is the implementation of recent revisions to its threat assessment processes and products, which apply to all DOD acquisition programs. In 2016, in response to the November 2013 memorandum from DOD’s acquisition leadership, the defense intelligence community began overhauling its threat assessment processes and products to produce more timely, efficient, and relevant information. See table 2 for an overview of these revisions. While each of these revisions has potential benefits, defense intelligence community officials have said that implementing the revisions has been more time-consuming and difficult than anticipated, which has affected their ability to provide certain threat assessments to MDA when needed. For example, MDA and the defense intelligence community were initially uncertain about the responsibilities and processes for creating a VOLT report for the BMDS. Although it took some time to resolve these uncertainties, MDA is now compiling its own country-specific threat assessments—known as the BMDS VOLT report—which DIA then validates. The military services generally have their own defense intelligence production centers, and therefore, a means for compiling VOLT reports. MDA, however, uses information from multiple defense intelligence production centers and does not possess its own production center. In September 2017, MDA reached out to DIA on this matter and DIA responded that, per the DOD policy update, it does not see anything that would preclude MDA, as a DOD component, from compiling VOLT reports. DIA stated that MDA compiling its own VOLT report aligns the agency with the rest of the DOD acquisition community. MDA is waiting on threat modules from the defense intelligence community to prepare its preliminary BMDS VOLT report, which MDA will use to inform acquisition decisions. MDA needs specific threat modules from the defense intelligence community, including those for six specific countries, in order to compile its preliminary BMDS VOLT report. However, defense intelligence community officials have said that they are still in the process of creating some of the digitized threat modules MDA needs, because it has taken more time and effort than they expected to standardize the threat modules’ content and coordinate production across multiple defense intelligence community production centers. Consequently, MDA is planning to publish its preliminary BMDS VOLT report in 2019 (table 3). In the meantime, without the preliminary BMDS VOLT report or digitized threat modules used to compile the BMDS VOLT report, MDA is reliant on threat assessments written between 2014 and 2016 for some of its acquisition decisions. For example, MDA recently made design decisions for certain BMDS elements using these threat assessments, although these threat assessments have not yet been updated. Consequently, these weapon systems that MDA recently made design decisions for could have capability gaps or performance shortfalls that present risks for the warfighter. MDA has attempted to fill the void for digitized threat modules and the preliminary BMDS VOLT report by submitting ad hoc requests for threat assessments to the defense intelligence community, but this has only added to the defense intelligence community’s workload and exacerbated delays. Moving forward, MDA has asked the defense intelligence community to provide the digitized threat modules on an accelerated schedule to ensure the agency can compile BMDS VOLT reports in a timely manner to inform its acquisition decisions; however, some defense intelligence production centers have said that an accelerated schedule will be difficult, if not impossible, without additional manpower and resources. Specifically, MDA wants the defense intelligence community to provide the digitized threat modules every year, as opposed to every two years as required by DOD policy. MDA has stressed the importance of having these digitized threat modules on an accelerated schedule in order to be responsive to threat advancements and mitigate the potential for capability gaps or performance shortfalls in its weapon systems. Defense intelligence community officials have acknowledged MDA’s need to have the digitized threat modules on an accelerated schedule but are concerned about their ability to provide them due to personnel and resourcing issues at some defense intelligence production centers. For example, two defense intelligence production centers have said that MDA’s request for an accelerated schedule is currently unrealistic due to their manpower and resource levels. Defense intelligence officials have said that once the initial digitized threat modules are created, the threat modules will be easier and quicker to update, but whether they can provide them annually is still being determined. Although MDA has the capability to centrally and collectively prioritize its threat assessment requests submitted to the defense intelligence community, it currently prioritizes its threat assessment needs through two distinct, individual lanes—country-specific and missile-specific— supplemented by informal discussions with the defense intelligence community. According to MDA, the individual lanes are as follows: 1. Country-specific threat assessments (i.e., threat modules for BMDS VOLT reports) are prioritized via the VOLT Threat Steering Group, which is co-chaired by MDA and DIA. The VOLT Threat Steering Group’s objectives are to determine MDA’s threat module requirements, to achieve concurrence on the threat modules used in the BMDS VOLT report, and to review the BMDS VOLT production schedule. The first VOLT Threat Steering Group meeting was held in April 2018 and during that meeting, MDA presented its prioritized list of threat assessments by adversary country to the defense intelligence community personnel in attendance. 2. Missile-specific threat assessments (i.e., reference documents used to build threat models) are prioritized via an annual intelligence mission data process managed by the Joint Staff. Through the intelligence mission data process, MDA prioritizes the data it needs for threat missiles by most to least critical—119 total threat missiles in 2018. With these two individual lanes for prioritization, MDA treats each type of threat assessment as independent and unrelated. According to MDA, the agency maintains these individual lanes for prioritizing its threat assessment requests because the requests can be more easily managed by the defense intelligence community components that develop the threat assessments. For example, MDA stated that requests for missile-specific threat assessments are often routed to intelligence production centers while requests for country-specific threat assessments are often routed to DIA’s regional centers (see appendix I for more information on defense intelligence community components). According to MDA, the vast majority of new requirements submitted to the defense intelligence community are also accompanied by an informal verbal discussion and if MDA’s priorities shift because a new threat emerges, MDA stated that it can convey that shift to the defense intelligence community in an effort to work out the best path forward. If the defense intelligence community cannot meet MDA’s needs, MDA stated that it works with the defense intelligence community to determine the best course of action for resolving prioritization issues. For example, MDA cited a recent example where it had worked with the U.S. Navy’s Office of Naval Intelligence to develop a threat model production schedule for two threat systems; however, the emergence of a new threat shifted MDA’s priorities. MDA was able to understand the effect of choosing one system ahead of the others based on the priority and projected production timelines. MDA cited another recent example where it had similarly worked with the U.S. Air Force’s National Air and Space Intelligence Center to prioritize production of a threat model for a new, unique threat. After some initial informal discussions and questions about whether the threat model production effort was a top priority for MDA, both agreed in a meeting in January 2019 to lower the priority for the model production effort. The specific threats referenced in the examples above have been omitted because they are classified. However, MDA’s approach of prioritizing its threat assessment needs through individual lanes creates the potential for unresolved, competing priorities because the defense intelligence community produces threat assessments collaboratively rather than disparately. Defense intelligence community officials told us that the underlying analyses that support both country-specific and missile-specific threat assessments are developed and reviewed by many of the same subject matter experts and managers within the defense intelligence community. Defense intelligence community officials told us that they have no way of knowing whether the information to build a specific threat model is a greater or lesser priority than updating a particular threat module needed to support the BMDS VOLT report. Our prior best practices work found that successful commercial companies employ a formal process for prioritizing their investments collectively rather than as independent and unrelated initiatives. MDA instead stovepipes its threat assessment prioritization through individual lanes and informally discusses its collective priorities with the defense intelligence community. Consequently, MDA’s requests, and resulting output from the defense intelligence community, may not be based on the collective order of importance, as depicted in figure 5. MDA relies on both country- and missile-specific threat assessments for its acquisitions, as each characterizes threats in unique ways and for different purposes, and it uses other requests to fill information gaps, as needed. Thus, all of MDA’s requests are important, but one among them may be the most important or urgent due to the timing of an upcoming design or testing decision. In the example illustrated above in figure 5, the most important request is for a country-specific threat assessment; however, it will not likely be the next one out of the defense intelligence community’s queue because there is a missile-specific request ahead of it. Hence, MDA may have the information it needs to build the threat model used to test one weapon system’s performance, but it may delay the country-specific information it needs to make design decisions for another. This delay in the country-specific information could put MDA in a position of moving forward with design decisions without the requisite information or relying on outdated information, which increases the risk for performance shortfalls and costly retrofits. One opportunity that MDA has to address the availability of threat assessments from the defense intelligence community is to collectively prioritize its threat assessment requests based on the order of importance. We have previously identified collective prioritization as a best practice—specifically, that it is important for an agency to regularly evaluate the totality of its needs or tasks, to determine whether specific ones should be prioritized ahead of others, based on the costs, benefits, and risks. While MDA has no formal requirement to collectively prioritize its threat assessment requests, defense intelligence community officials said that they have had discussions with MDA through existing venues and requested that it do so to ensure it has the most urgently needed information. MDA has the capability to collectively prioritize its threat assessment requests because all of the requests go through a centralized intelligence requirements group within the agency’s engineering directorate. This group has insight into the totality of the agency’s threat assessment requests and is uniquely positioned to make determinations about the order of importance among them. As the group submits requests to the defense intelligence community, the defense intelligence community responds to the requests in the order that they were received, because, as we previously found, the defense intelligence community is not required to prioritize the requests, does not currently possess the capability to do so, and would not be in a position to dictate to an agency what is most important. Another opportunity for MDA to address the availability of threat assessments is through further collaboration with the defense intelligence community to determine the extent of additional resources that would be needed to enable accelerated support. When intelligence support requirements exceed the defense intelligence community’s responsibilities, DOD acquisition programs are generally required to account for resources to augment intelligence support. For example, according to defense intelligence community officials, the Air Force is providing one of the defense intelligence community’s production centers with additional resources to collect data and devise tools primarily to support a specific major defense acquisition program via a military interdepartmental purchase request because the program’s request exceeds the defense intelligence community’s responsibilities. According to MDA, intelligence mission data shortfalls are currently identified through an annual departmental review process. MDA stated that in fiscal year 2019 DOD approved budgeting additional funding in the future to help address intelligence mission data shortfalls for all of the military services, including MDA. MDA has not provided the defense intelligence community with additional resources for an accelerated schedule to update threat modules more frequently. MDA has requested that the defense intelligence community update the digitized threat modules it needs to compile a BMDS VOLT report every year to ensure that it has the updated threat information needed for acquisition decisions; however, the defense intelligence community is only required to update the digitized threat modules every two years. Some defense intelligence community officials have acknowledged MDA’s need to have an accelerated schedule, but have communicated to MDA that given its current manpower and resource constraints, the accelerated schedule is unrealistic without additional resources. Thus, MDA’s request for the defense intelligence community to update the digitized threat modules faster exceeds what the defense intelligence community is currently able to do given its manpower and resource constraints. With existing venues, like the VOLT Threat Steering Group, MDA and the defense intelligence community have a forum to further collaborate and identify what additional resources are needed and the potential funding scenarios to support an accelerated schedule for threat module production. Without collaboration through these existing venues, MDA and the defense intelligence community may not be utilizing an available method to ensure their individual needs are met. According to our best practices for inter-governmental agency collaboration, it is important for the inter-reliant agencies to collaboratively identify the resources— information, manpower, and funding—needed to accomplish their respective missions. Doing so enables the agencies to have a common understanding and explore opportunities to leverage each other’s resources; thus, realizing benefits that would not be available if they were working separately. Therefore, working together, MDA and the defense intelligence community would be better positioned to determine how to best meet their respective needs. MDA uses defense intelligence community threat assessments to inform its acquisitions, but the agency has not fully engaged the defense intelligence community on challenges in preparing the BMDS for existing and emerging threats. According to MDA, the rapid pace of threat evolution presents significant challenges for the agency to sufficiently plan for emerging threats. Although the defense intelligence community is uniquely positioned to assist MDA in addressing these challenges, the agency generally limits the defense intelligence community’s insight into and input on critical threat-related BMDS acquisition processes and decisions, such as establishing the BMDS threat space and assigning threat parameters and threat models to BMDS elements. Major defense acquisition programs are generally required to engage the defense intelligence community on how to design and test weapon systems, but MDA generally does not, due to the acquisition flexibilities DOD has granted to the agency. Moreover, DIA is currently unable to validate MDA’s threat models, as required by DOD policy, because MDA does not follow the department’s best practices on models and simulations. MDA has steadily increased its outreach to the defense intelligence community and other stakeholders over the past few years, but opportunities remain for more comprehensive engagement on key challenges the agency faces with keeping pace with the threat. According to MDA, the rapid pace of threat evolution presents significant challenges for the agency to sufficiently plan for emerging threats. MDA currently faces some difficult choices regarding what steps it needs to take and in what order to address recent threat advancements. In making these decisions, MDA has an opportunity to engage the defense intelligence community on whether and how it should make changes to the BMDS threat space, threat parameters, and threat models the agency uses as design requirements and test cases for BMDS elements. As previously noted, the defense intelligence community plays important stakeholder, advisor, and oversight roles for MDA’s acquisitions. Although the department has provided MDA with flexibilities on following many of the requirements that specifically define when and how major defense acquisition programs are to engage the defense intelligence community, DOD policy requires MDA to vet its threat models and consult with the defense intelligence community on threat-related acquisition matters. DOD, senior defense officials, and expert panels supported by DOD have consistently maintained that the defense intelligence community’s direct involvement in MDA’s acquisitions is critical to staying ahead of the threat: In a written response following a 2002 congressional hearing, a senior defense official stated that every effort was being made to coordinate development of the document establishing the BMDS threat space with the defense intelligence community and that the defense intelligence community’s participation was critical to the agency’s success. In 2010, DOD’s Ballistic Missile Defense Review similarly found the need to maintain a strong focus by the defense intelligence community on the ballistic missile threat and that accurate and timely intelligence should play a vital role in informing BMDS planning. In 2010, an expert panel known as JASON (not an acronym) found that MDA lacked sufficient plans for improving discrimination and that the agency risked falling behind the evolution of the threat’s countermeasure capabilities. The study recommended that DOD form stronger two-way connections between MDA and defense intelligence agencies. In 2012, the National Research Council found that MDA did not follow through on efforts to improve discrimination and that much of the agency’s expertise on discrimination was lost in the late 2000s. The study recommended that MDA seek assistance from experts with experience in understanding sensor data for threat missiles. In 2018, DOD’s National Defense Strategy stated that modernizing missile defense, among other items, was necessary to keep pace with adversaries and that the department must expand the role of intelligence analysis throughout the acquisition process in order to streamline rapid, iterative approaches for delivering performance at what DOD refers to as “the speed of relevance.” During a 2018 congressional hearing, the Under Secretary of Defense for Research and Engineering stated that catching up to near-peer adversaries in missile defense can be achieved by exceeding their technical capabilities and that the intelligence community was critical to making sure that we are outpacing our adversaries. Although MDA uses defense intelligence community threat assessments to inform BMDS acquisition, the defense intelligence community generally has limited insight into the BMDS, which is unprecedented among major defense acquisition programs. When MDA was established in 2002, DOD granted the agency exceptional flexibilities to diverge from the standard acquisition framework that most major defense acquisition programs follow. These flexibilities enable MDA to forego obtaining the defense intelligence community’s input on some critical threat-related BMDS acquisition processes and decisions, such as how MDA establishes the: threat space that informs overall BMDS design and development; threat parameters assigned to each BMDS element as design requirements; and threat models assigned to each BMDS element as test cases for design reviews and testing. However, according to MDA, the new BMDS VOLT report will serve as the source document for specific details on the BMDS threat space, threat parameters, and threat models. Although MDA may leverage the defense intelligence community’s threat assessments, MDA has not included the defense intelligence community in these key threat-related BMDS acquisition processes and decisions. For example, in response to a questionnaire we sent to MDA in May 2018, agency officials stated that decisions related to the threat parameters it assigns to the different BMDS elements should be left to MDA, as it is within the agency’s purview and authority to design threats as it deems necessary for research, development, test, and evaluation purposes. Moreover, MDA indicated that the defense intelligence community should provide the agency with the best intelligence information on adversary missile capabilities, in a timely manner, to support the agency’s mission. As such, MDA stated it does not support obtaining the defense intelligence community’s concurrence on the threat parameters it assigns to the BMDS elements. MDA has provided the defense intelligence community with some insight into the BMDS but not to the same extent DOD generally requires of major defense acquisition programs. For example, MDA has held a number of “immersion days” over the past nine years, which allow the defense intelligence community to receive briefings from MDA programs on priorities, future developments, and weapon system operations. According to MDA, it also assigns intelligence portfolio managers to BMDS elements and their mission, among other items, is to keep the defense intelligence community informed on key program developments and how intelligence feeds into the agency’s threat-related acquisition processes and decisions. In addition, MDA has briefed the DIBMAC on how it uses threat assessments to inform BMDS acquisition. However, defense intelligence community officials stated that they generally lack fundamental information on the BMDS and have no visibility into the BMDS threat space, threat parameters, or test cases MDA assigns to the BMDS elements. In contrast, for most major defense acquisition programs, the defense intelligence community is integrally involved in determining the: threat(s) of record upon which requirements of the weapon system are based; key performance parameters and attributes of the weapon system; threat parameters that could critically degrade or negate the weapon system; and operational threat environment the weapon system is tested against. These insights, enabled by DOD’s standard requirements-setting process and acquisition framework, are intended to provide the defense intelligence community with in-depth knowledge of the design and performance requirements for most major DOD weapon systems. Officials from other various organizations we met with, such as the Joint Staff, contractors, warfighters, and test and evaluation, expressed concerns about MDA’s ability to unilaterally define the threats it designs the BMDS against. As one MDA prime contractor told us, what really matters is how the BMDS would perform in the real world against real threats. Defense intelligence community officials acknowledged that MDA, as the BMDS developer, has a legitimate need to explore threat capabilities beyond those that the intelligence community has observed from specific adversaries. However, defense intelligence community officials rejected a sentiment expressed to us by MDA officials that the defense intelligence community lacks expertise in understanding the bounds of threat capabilities. To the contrary, according to defense intelligence community officials, this is exactly the type of analysis at which the defense intelligence community excels. In choosing not to engage the defense intelligence community on these key threat-related BMDS acquisition processes and decisions, MDA runs the risk of not sufficiently planning for existing and emerging threats. MDA’s reluctance to provide the defense intelligence community with insight into or input on some threat-related BMDS acquisition processes and decisions is consistent with how MDA has engaged other DOD stakeholders and oversight groups. Our prior work on defense acquisitions has shown that establishing buy-in from decision makers is a key enabler of achieving better acquisition outcomes because DOD components provide varying perspectives due to their unique areas of expertise and experience. However, in May 2017, we found that MDA generally limits the warfighter’s input on the requirements it pursues and overlooked stakeholder concerns on the acquisition strategy for a redesigned kill vehicle for the Ground-based Midcourse Defense system. We made recommendations aimed at increasing stakeholder engagement and oversight in BMDS acquisition, such as coordinating operational requirements with the warfighter and obtaining input from DOD’s Office for Cost Assessment and Program Evaluation (CAPE) on acquisition strategies for new efforts. DOD’s acting Assistant Secretary of Defense (Acquisitions) did not concur with the recommendations, stating that warfighters lacked the skillset to determine operational BMDS requirements and existing DOD policy does not require MDA to obtain CAPE’s concurrence on acquisition policies. We continue to maintain that DOD should implement the recommendations. MDA builds its own threat models to support BMDS design, development, and testing but it does not validate its threat models with DIA, which is inconsistent with DOD policy and best practices. Although the defense intelligence community builds threat models, MDA cannot currently use those models as-is because they are generally not compatible with MDA’s modeling and simulation framework. Even with MDA using its own threat models, DOT&E has found that integrating the various BMDS models and presenting them with a common threat scene has been an extremely challenging task for MDA. Moreover, MDA’s BMDS modeling and simulation architecture requires highly detailed threat models for simulations to function properly. Defense intelligence community officials stated that they generally do not need the same level of detail MDA requires for the types of analyses the defense intelligence community performs. In addition, according to a March 2018 MDA memorandum, the agency was previously told by representatives of the DIBMAC that they do not have the staff or resources to produce the high volumes of detailed threat models that MDA needs to support BMDS development and testing. Therefore, MDA continues to build its own threat models for use in BMDS development and testing. MDA uses defense intelligence community threat assessments to build its threat models, but independent evaluators have not been able to fully trace MDA’s threat models to defense intelligence community threat assessments. According to a briefing MDA presented to the defense intelligence community in September 2018, every target, model, and test can be traced back to defense intelligence data. However, in August 2018, the U.S. Army issued a memorandum for MDA stating that the BMDS Operational Test Agency (OTA)—the agency responsible for independently analyzing the verification and validation data for models used in operational testing—was only able to certify some of the threat models used in a recent ground test. In other ground tests, though, the BMDS OTA was able to trace MDA’s threat models back to defense intelligence community threat assessments. In February 2019, DOT&E reported that (a) credible threat models are the linchpins of BMDS models and simulation; (b) reducing threat model uncertainty is a high priority; and (c) MDA and the BMDS OTA should ensure that MDA-developed threat models are representative of the defense intelligence community’s understanding of the threat. MDA also has not implemented best practices established by DOD’s Models and Simulation Coordination Office that would enable DIA to be in a position to validate MDA’s threat models. According to DOD best practices on modeling and simulation, the validation agent should: (1) be brought on in the beginning of the modeling and simulation development process; (2) work closely with the model developers as the models are built and tested; and (3) perform validation as a continuing activity of the overall process of developing and preparing a model for use or reuse in a simulation. Conversely, defense intelligence community officials stated that they lack sufficient insight into and input on how MDA builds and uses threat models. For example, the defense intelligence community has emphasized to MDA that caveats need to be carried through with the model data and voiced concerns about the engineering judgments the agency makes in its threat models, because these judgments could lead to the BMDS performing well or poorly for reasons not based on the actual threat. Given these uncertainties and the defense intelligence community’s lack of insight into the purposes for which MDA uses its threat models, DIA lacks the insight and input necessary to validate MDA’s threat models. Although MDA has previously expressed interest in validating its threat models with the defense intelligence community, long-standing obstacles remain. During a May 2018 meeting between MDA and the DIBMAC, defense intelligence community officials identified the lessons they have learned from working with other acquisition programs to validate threat models. Model validation can be achieved if the acquisition program: establishes a partnership with the defense intelligence community; prioritizes its threat modeling needs; recognizes there are limits to how many threat models can be built in a given time; provides in-depth insight into its threat modeling needs and weapon system’s capabilities; discusses how the models will be applied; jointly defines model acceptance criteria early in the process; provides resources, including funding and staff; and invests in the defense intelligence community’s capability and capacity. MDA officials stated that the agency desires to have its threat models validated but noted that the defense intelligence community does not validate models produced by other organizations. MDA officials also emphasized that the defense intelligence community cannot meet MDA’s timeline for building threat models, whereas the agency can. In addition, MDA officials indicated to us that they do not believe it is practical to provide the amount of insight defense intelligence community officials told us they would need in order to validate MDA’s threat models. MDA officials told us that the only way in which the defense intelligence community could obtain such insight is by being co-located with MDA’s threat modelers as the models are being built. However, the 2010 JASON study found that this type of close working arrangement between MDA engineers and defense intelligence analysts is necessary to effectively plan for emerging threats. Defense intelligence community officials also clarified for MDA that the defense intelligence community can validate models produced by another agency but it would require the defense intelligence community having detailed knowledge of everything used to produce the model. As a result, although DOD policy generally requires that threat models used to support acquisition decisions be validated by DIA, MDA has yet to validate any of the numerous threat models it has developed since 2004. Without independent validation, MDA runs the risk that DOD and congressional decisionmakers may not have confidence that the agency’s plans and proposals for developing the BMDS are appropriate and sufficient to address the threat because any flaws or bias in MDA’s threat models can have significant implications on the BMDS’s overall performance. According to a Federally Funded Research and Development Center publication describing its efforts supporting MDA threat modeling, acquisition influences can place pressure on MDA threat modelers to tailor the missile threats to suit the currently feasible BMDS design. In May 2017, we found a parallel circumstance where, in the absence of warfighter validation of MDA-established requirements, the agency made critical design choices for three new BMDS efforts. These design choices reflected the needs and preferences of MDA ahead of the warfighter, potentially compromising performance to the extent of not being able to defeat current and future threats. MDA has undertaken a number of efforts over the past few years to generally increase stakeholder involvement in BMDS acquisition. The engagement efforts, in large part, are a result of efforts led by MDA’s previous director to improve the agency’s relationship with department stakeholders. In addition to previously serving as the Deputy Director for MDA, the Director also held a variety of assignments in operational, acquisition, and staff units within DOD. When we met with the MDA Director in March 2018, he told us that he wanted to change the agency’s culture of limiting stakeholder input, noting that he had recently provided updated guidance to his leadership team and agency personnel on bringing stakeholders in early, engaging them more frequently and substantively, and ensuring that the agency has obtained their buy-in on major undertakings. The MDA Director also stated that he was willing to take some actions that could effectively address a recommendation we made in May 2017 intended to provide the warfighter with greater input on operational requirements for ballistic missile defense. Officials from several DOD organizations we met with over the course of our review observed that MDA’s engagement with their respective organizations was improving. In 2018, MDA began working with the defense intelligence community to determine a more appropriate level of involvement for the defense intelligence community throughout MDA’s acquisition activities. MDA and defense intelligence community officials agreed during a May 2018 meeting that processes could be put in place to develop intelligence-based countermeasure assessments if adequate resources are provided. MDA officials also acknowledged that the defense intelligence community would benefit from having a better understanding of how the BMDS responds to threats and agreed to work towards providing such information. Defense intelligence community officials stated that increased insight would allow them to better focus their intelligence collection, analysis, and production by knowing which threat parameters MDA most often uses and the specificity of those parameters. The defense intelligence community and MDA also agreed that providing defense intelligence community engineers with MDA program-level access would improve the support the defense intelligence community provides to MDA. MDA has also recently increased its outreach to the defense intelligence community on some early BMDS planning decisions, although opportunities for more comprehensive engagement remain. For example, MDA engaged the defense intelligence community on an analysis of alternatives the agency completed in February 2017 that assessed future sensor options for the BMDS. According to MDA officials, they are also engaging the defense intelligence community on another analysis of alternatives pertaining to defense against hypersonic missiles. In addition, MDA worked with the defense intelligence community to establish threat space parameters for some specific threat systems. Also, as noted earlier, over the last nine years, MDA has held 18 “immersion day” events with the defense intelligence community, half of which occurred in the last two years. Moving forward, MDA has opportunities to more comprehensively engage the defense intelligence community on updating the BMDS threat space and determining threat parameters and threat models assigned as design requirements and test cases for BMDS elements. In addition, MDA has recently begun placing greater emphasis on ensuring its models are credible. According to an internal MDA memorandum signed by the MDA Director in April 2018, a culture exists within the agency that generally tolerates the use of models that have not been sufficiently vetted and is too willing to accept the associated risk. The memorandum states that the agency’s goal is for all MDA personnel to help address this culture problem and that model verification, validation, and accreditation is a high priority for MDA. During a meeting with the BMDS OTA in October 2018, officials confirmed that MDA is taking steps to address the challenges raised in the memorandum. MDA also increased its outreach to the defense intelligence community in 2016 to coordinate on threat modeling efforts. In the past three years, MDA and the defense intelligence community have collaborated to quickly model several newly-observed threat missiles, according to MDA. Figure 6 below shows that MDA held 93 threat model coordination meetings with the defense intelligence community over the last four years, with more frequent meetings occurring in early 2016 and again in early-to-mid 2018. In addition, MDA is working with the defense intelligence community to address compatibility issues that currently prevent MDA from directly using the defense intelligence community’s threat models in BMDS ground testing. MDA plans to include a few missile trajectory models produced by the defense intelligence community in the models and simulation framework for the agency’s upcoming Ground Test-08 campaign. The Technical Interchange Meetings and pathfinder efforts for MDA directly using defense intelligence community threat models are improving collaboration between MDA and the defense intelligence community on threat modeling efforts. However, they do not provide MDA with a pathway for validating its threat models with DIA. Even if compatibility issues that currently prevent MDA from using defense intelligence community threat models could be resolved, the defense intelligence community is currently not resourced to build threat models for MDA. Moreover, although MDA has indicated that the Technical Interchange Meetings can include any topic of interest, the meetings do not provide defense intelligence officials with sufficient insight into how MDA builds its models, including the assumptions, caveats, or intended use of the models. According to MDA, the agency continues to hold discussions with the defense intelligence community and explore process improvements, as well as technical and resource requirements, to ensure the creation of valid, threat-representative models for BMDS development. In March 2018, the MDA Director told us that one of his priorities was to ensure that the agency was using appropriately validated models and acknowledged the importance of ensuring its threat models are sufficiently representative. In April 2018, MDA subsequently began holding meetings with the DIBMAC to define the issues preventing the defense intelligence community from validating MDA’s threat models. MDA and the defense intelligence community met five times in 2018 to identify actions that would facilitate working together to develop threat models the defense intelligence community would be comfortable validating. During these meetings, both organizations agreed on specific actions intended to increase the defense intelligence community’s involvement in MDA’s threat modeling process. To achieve threat model validation, an initial plan was developed that included a combination of (a) MDA directly using aspects of defense intelligence community threat models; and (b) MDA partnering with the defense intelligence community to build threat models. MDA and the defense intelligence community plan to hold follow-on meetings in 2019 to further discuss the plan and review actions. MDA is reliant on threat assessments from the defense intelligence community, as they inform what weapon systems the agency pursues, the design of those systems, and how those systems are tested prior to being delivered to the warfighter for operational use. However, the defense intelligence community has been facing a variety of challenges that are affecting its ability to provide MDA the threat assessments it needs, when it needs them. If MDA does not have the threat assessments it needs, when needed, the agency’s weapon systems are at risk of being designed or tested against irrelevant or outdated information, which could result in performance shortfalls and costly retrofits. MDA has opportunities to mitigate these challenges and risks by collectively prioritizing its threat assessment requests and working through existing venues with the intelligence community to determine if and to what extent additional resources may be needed to secure the support that it needs. If MDA does not take advantage of these opportunities, the defense intelligence community’s challenges will likely continue, which will impact the availability of threat assessments and increase the likelihood that MDA’s weapon systems will not be designed or tested against the most up-to-date threat information. In addition, MDA faces a steep challenge in developing the BMDS and fielding capabilities at a rate that keeps pace with the threat. MDA was previously informed by expert panels and senior defense leaders that it needed to work more closely with the defense intelligence community to better prepare for future threats or risk falling behind the threat. Given these challenges, it is imperative for MDA to make the most out of its available resources. Aside from providing MDA with threat assessments, the defense intelligence community is a resource MDA has yet to fully tap into. The defense intelligence community is uniquely qualified to assist MDA on fundamental and critically important BMDS acquisition processes and decisions, such as establishing the BMDS threat space and the threat parameters and models it assigns to the BMDS elements. Moreover, after nearly 15 years of building numerous threat models, MDA has yet to fully implement a plan for DIA to validate these threat models, as generally required by DOD policy. However, MDA has recently begun laying the groundwork for more comprehensive engagement with the defense intelligence community through efforts which have the potential to address long-standing obstacles that have prevented DIA from validating MDA’s threat models. Resolving these issues would help MDA keep pace with emerging threats and improve the BMDS’s viability to defend against the complex missile threats of the future. We are making a total of three recommendations to DOD: The Director, MDA should coordinate with the defense intelligence community on the agency’s collective priorities for threat assessments and work with the defense intelligence community to determine if additional resources are needed to support the agency’s threat assessment needs. (Recommendation 1) The Director, MDA should fully engage the defense intelligence community on key threat-related missile defense acquisition processes and decisions, including providing insight into and obtaining input from the defense intelligence community on the threat space MDA establishes for the BMDS and the threat parameters and threat models MDA assigns to BMDS elements as design requirements and test cases. (Recommendation 2) The Secretary of Defense should require the Director, MDA and the Director, DIA to coordinate on establishing a process for MDA to obtain validation of its threat models. (Recommendation 3) DOD provided written comments in response to the classified version of this report (GAO-19-92C), indicating that the department concurred with all three of our recommendations. An edited version of DOD’s comments is reprinted in appendix II as some information had to be omitted due to classification. In addition, the summarized version of DOD’s comments below is reflective of the content in the classified version. DOD provided us with technical comments and a significant amount of new information in response to the classified version of this report. We incorporated this information into our report, as appropriate, but the new information did not substantively change our findings and did not alter our recommendations. Although DOD concurred with our third recommendation, DOD also raised concerns about statements in our report related to our third recommendation that the department believes are inaccurate. We do not believe DOD’s concerns are warranted because our findings are based on evidence we obtained during our review—evidence that we believe is sufficient and appropriate and provides a reasonable basis for our findings and conclusions. We address this in further detail below. DOD concurred with our first recommendation that the Director, MDA should coordinate with the defense intelligence community on the agency’s collective priorities for threat assessments and determine whether additional resources are needed. In its response, DOD stated that MDA will continue to follow established processes to identify threat assessment needs and to determine if additional resources are required. However, our review found that these established processes—prioritizing exclusively through distinct, individual threat assessment lanes—have not proven entirely effective. In addition, although MDA has participated in the department’s intelligence mission data review process since 2016, the agency has yet to provide the defense intelligence community with additional resources to address known funding and manpower shortages. Moreover, this review process is limited to intelligence mission data and does not cover all of the other types of threat assessments that MDA needs. As such, we maintain that MDA should take additional steps beyond continuing existing processes to address the challenges MDA currently faces in obtaining the threat assessments it needs, when it needs them. DOD also concurred with our second recommendation that the Director, MDA should provide insight into and obtain input from the defense intelligence community on the threat space MDA establishes for the BMDS and the threat parameters and threat models the agency assigns to BMDS elements as design requirements and test cases. DOD stated in its response that MDA has and will continue to fully engage the defense intelligence community on key threat-related missile defense acquisition processes and decisions. The efforts MDA has recently undertaken to expand its outreach to the defense intelligence community are positive steps. However, we have yet to see MDA provide the defense intelligence community with further insight into or input on the threat space the agency has established for the BMDS or the assignment of threat models and threat parameters to BMDS elements. We will continue to monitor MDA’s ongoing efforts to see whether it takes this next step toward more fully engaging the defense intelligence community. DOD concurred with our third recommendation that the Secretary of Defense should require the MDA and DIA Directors to coordinate on establishing a process for MDA to obtain validation of its threat models. In its response, DOD stated that the department will re-examine the most cost-effective approach to meet the intent of DIA validation to support development and fielding of effective BMDS elements. More specifically, DOD stated that MDA and the DIBMAC are currently having extensive discussions regarding how the defense intelligence community can best support MDA’s threat modeling requirements. As noted in our report, the discussions MDA has had with the defense intelligence community over the course of 2018 demonstrate that the department is beginning to consider substantive measures to address the long-standing issue of MDA not using DIA-validated threat models. However, MDA and defense intelligence community officials have also cautioned that obstacles remain and that alternative solutions may need to be explored. We will continue to monitor these ongoing discussions and any results that emerge. DOD also stated in its response that it was concerned that statements in our report pertaining to our third recommendation imply that MDA has not coordinated with DIA on validating its threat models and that our report could be interpreted as saying MDA does not internally conduct threat model validation. To be clear, our review did, in fact, find that, until recently, MDA did not sufficiently coordinate with DIA on establishing a process for creating valid threat models for use in MDA simulations. Furthermore, we explain in our report that MDA was told that the defense intelligence community can validate MDA’s threat models if it has sufficient insight into how MDA builds its models—insight which MDA officials previously told us was unnecessary. Additionally, although MDA may internally validate its threat models for each ground test, the BMDS OTA was not able to certify many of those threat models, in part, because some models could not be traced back to the defense intelligence community’s threat assessments. We therefore excluded MDA’s internal threat model validation process from our report, as it is not a comparable substitute for DIA threat model validation. We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix Ill. In its entirety, the intelligence community is a federation of 17 agencies and organizations that span the executive branch of the U.S. government. The defense intelligence components responsible for assessing foreign ballistic missile threats are headed by the Defense Intelligence Agency and overseen and coordinated by the Defense Intelligence Ballistic Missile Analysis Committee. Table 4 below identifies each component and its respective focus areas. In addition to the contact named above, LaTonya Miller (Assistant Director), Rose Brister, Lori Fields, Laura Greifner, Kurt Gurka, Helena Johnson, Kevin O’Neill, Jay Tallon, Brian Tittle, Hai Tran, Alyssa Weir, and Robin Wilson made key contributions to this report.", "summary": "MDA is developing missile defense capabilities to defend the United States, deployed forces, and regional allies from missile attacks. However, missile threats continue to emerge, as adversaries continue to improve and expand their missile capabilities. The National Defense Authorization Act for Fiscal Year 2012 included a provision that GAO annually assess and report on the extent to which MDA has achieved its acquisition goals and objectives, and include any other findings and recommendations. This report is a public version of a classified report GAO issued in May 2019, which addresses (1) the challenges MDA and the defense intelligence community face in meeting the agency's threat assessment needs and (2) the extent to which MDA engages the defense intelligence community on missile defense acquisitions. GAO reviewed MDA's threat-related acquisition processes and interviewed relevant officials from the defense intelligence community, MDA, test community, and warfighters. Information deemed classified by DOD has been omitted. The Missile Defense Agency (MDA) is experiencing delays getting the threat assessments needed to inform its acquisition decisions. Officials from the defense intelligence community—intelligence organizations within the Department of Defense (DOD)—told GAO this is because they are currently overextended due to an increased demand for threat assessments from a recent upsurge in threat missile activity, as well as uncertainties related to their transition to new threat processes and products. The delays are exacerbated because MDA does not collectively prioritize the various types of threat assessment requests submitted to the defense intelligence community or provide resources for unique requests, as other major defense acquisition programs are generally required to do. Without timely threat assessments, MDA risks making acquisition decisions for weapon systems using irrelevant or outdated threat information, which could result in performance shortfalls. MDA has increased its outreach to the defense intelligence community over the past few years, but opportunities remain for further engagement on key threat­related processes and decisions. Specifically, MDA provides the defense intelligence community with limited insight into how the agency uses threat assessments to inform its acquisition decisions. MDA is not required to obtain the defense intelligence community's input, and instead has discretion on the extent to which it engages the defense intelligence community. However, the defense intelligence community is uniquely positioned to assist MDA and its involvement is crucial for helping MDA keep pace with rapidly emerging threats. Moreover, this limited insight has, in part, prevented the defense intelligence community from validating the threat models MDA builds to test the performance of its weapon systems. Without validation, any flaws or bias in the threat models may go undetected, which can have significant implications on the performance of MDA's weapon systems. MDA and the defense intelligence community recently began discussing a more suitable level of involvement in the agency's acquisition processes and decisions. Note: the threat missile coverage depicted is notional and not representative of MDA's actual threat coverage. GAO is making three recommendations to improve how MDA: prioritizes and resources its threat assessment needs; obtains input from the defense intelligence community on key threat-related processes and decisions for missile defense acquisitions; and validates its threat models. DOD concurred with all three recommendations, citing actions it is already taking. While DOD has taken some positive steps, GAO believes more action is warranted.", "document_type": "gao"}
{"report": "As of May 2019, the federal government recognized 573 Indian tribes as distinct, independent political communities with certain powers of sovereignty and self-government, including power over their territory and members. The tribes can vary greatly in terms of their culture, language, population size, land base, location, and economic status. As of the 2010 U.S. Census, about 21 percent, or 1.1 million, of all American Indians lived on tribal lands. Tribal lands include many land types (see table 1). According to BIA, the federal government holds about 46 million acres in trust for tribes (tribal trust land) and more than 10 million acres in trust for individual Indians (individual trust land). Some tribes also have reservations. According to BIA, there are approximately 326 Indian land areas in the United States administered as federal Indian reservations (including reservations, pueblos, rancherias, missions, villages, and communities). The land within the reservation may include a mixture of tribal trust land, individual trust land, restricted fee land, allotments, and land without trust or restricted status (that is, fee- simple land), which may be owned by tribes, individual Indians, or non- Indians. Agricultural producers (farmers, ranchers, or producers or harvesters of aquatic products) on tribal lands can be individual tribal members, the tribe itself, or non-Indians who lease the land from the tribe or Indian owner. According to USDA’s 2012 Census of Agriculture, about 75 percent of farms and ranches on 76 selected Indian reservations were operated by agricultural producers that identified as American Indian or Alaska Native (see table 2). On these reservations, Indian producers held 61 percent of total farm and ranch acreage. However, the total market value of agricultural products sold from Indian-operated farms and ranches was just over a tenth of that of non-Indian operated farms and ranches on the 76 selected reservations. In 2011, USDA, which operates several agricultural programs targeted to traditionally underserved populations, settled a class action lawsuit brought by Native American farmers and ranchers for $760 million (Keepseagle v. Vilsack). The lawsuit alleged that USDA discriminated against Native Americans in its farm loan and farm loan servicing programs. In 2018, $266 million of the remaining settlement proceeds were used to establish the Native American Agriculture Fund. The Fund will begin awarding grants in 2019 to fund the provision of business assistance, agricultural education, technical support, and advocacy services to Native American farmers and ranchers. Like other businesses, agricultural producers generally require financing to acquire, maintain, or expand their farms, ranches, or agribusinesses. Types of agricultural loans as categorized by their purpose or maturity may vary by lender but generally include the following: Short-term loans. These loans are used for operating expenses and match the length and anticipated production value of the operating or production cycle. They are typically secured by the product (crops or livestock). Intermediate-term loans. These loans are typically used to finance depreciable assets such as equipment, which serves as the loan collateral. The loan terms usually range from 18 months to 10 years. Long-term loans. These loans are used to acquire, construct, and develop land and buildings with terms longer than 10 years. They are secured by real estate and may be called real estate loans. Several types of lenders provide credit to U.S. agricultural producers. According to USDA’s Economic Research Service, in 2017, FCS and commercial banks provided most agricultural credit in the United States, with respective market shares of 40 and 41 percent. USDA’s Farm Service Agency—a lender that focuses on assistance to beginning and underserved farmers and ranchers and also guarantees the repayment of loans made by other lenders—provided 3 percent, and the remainder was provided by individuals, life insurance companies, and other lenders. FCS is a government-sponsored enterprise, established in 1916 to provide sound, adequate, and constructive credit to American farmers and ranchers. FCS is regulated by FCA, an independent federal agency. FCS’s statutory mission includes being responsive to the needs of all types of creditworthy agricultural producers, and in particular, young, beginning, and small farmers and ranchers. According to FCA, FCS is not statutorily mandated to focus on providing financial opportunities to any other group. FCS lends money to eligible agricultural producers primarily through its 69 lending associations (FCS associations), which are funded by its four banks (FCS banks). All are cooperatives, meaning that FCS borrowers have ownership and control over the organizations. As of 2017, FCS had approximately $259 billion in loans outstanding, of which 46 percent were long-term real estate-based loans; 20 percent were short- and intermediate-term loans (such as for farm equipment or advance purchases of production inputs); and 16 percent were for agribusiness activities, such as agricultural processing and marketing. FCS associations are not evaluated under the Community Reinvestment Act, which requires certain federal banking regulators to assess whether financial institutions they supervise are meeting the credit needs of the local communities. FCS receives certain tax exemptions at the federal, state, and local level. Little data exists on the credit needs of tribes and their members. One measure of unmet credit needs is the difference between the amount applied for and the amount received. However, we could not determine the amount of agricultural credit that Indian tribes and their members applied for or received. These data were limited in part because federal regulations historically have prohibited lenders from asking about the race of applicants for nonresidential loans, including agricultural loans. Additionally, even if data were available, the unmet need could be greater than that indicated by information on those who may have applied for and did not receive credit. Four tribal stakeholders and experts told us that tribal members may choose not to apply for agricultural credit because they were directly discouraged by loan officers, had problems completing paperwork, or had heard of other tribal members being denied loans. Two tribal agricultural experts told us that on some level, the agricultural credit needs of Indian tribes and their members are the same as other agricultural producers’ credit needs. In particular, tribal stakeholders and experts told us that the tribal members need short-term loans for operating expenses and intermediate-term loans for equipment. One difference between the agricultural credit needs of tribal members and other producers is that tribal members may have a greater unmet need for long-term loans, which are typically secured by real estate, because of difficulties in using tribal lands as collateral, as discussed later in this report. Credit needs vary based on the type of operation or borrower. Type of operation. Some tribal stakeholders we interviewed told us that members of their tribes were more likely to participate in ranching than farming, partly because farming has higher start-up costs. For example, one tribal agricultural expert told us a rancher can start with a few head of cattle and grow the herd over time, but a beginning farmer may need to purchase equipment. Additionally, several tribal stakeholders told us that land on their reservations was more suitable for ranching than farming. Type of borrower. Some tribes have agricultural businesses, which have credit needs different from those of individual tribal members, according to experts and BIA officials we interviewed. For example, they may be greater or more complex. According to an expert and a tribal stakeholder, established agricultural businesses likely would be able to receive credit from commercial lenders because they have more resources to pledge as collateral or stronger credit histories. Additionally, if a tribe has other profitable businesses, it likely will have less difficulty obtaining credit or financing agriculture with those other resources than those without such resources. According to tribal stakeholders, experts, and BIA officials we interviewed, tribal members who obtain agricultural credit likely receive it from USDA’s Farm Service Agency, other USDA programs, or Native CDFIs. Some tribal members receive agricultural credit from local private lenders, but they are typically larger, more established borrowers. One expert told us that tribal members who are smaller or beginning agricultural producers and cannot access commercial banks instead may borrow money from family members. A 2017 report found that Native business owners were less likely than other business owners to obtain start-up capital from banks. Some experts we interviewed cited Native CDFIs as growing providers of agricultural credit to tribal members. A 2014 survey of 41 Native CDFIs— credit unions, community banks, and loan funds—found more than 40 percent provided credit and training to farmers and ranchers. In total, these CDFIs made almost $6 million in agricultural loans annually. However, Native CDFIs are limited in how much agricultural credit they can provide. In the 2014 survey, 56 percent of the Native CDFIs that made agricultural loans reported not having enough capital for such loans, with a total unmet need of at least $3 million in the previous year. One Native CDFI we interviewed said its agricultural loans averaged about $100,000 per borrower, and another said its operating loans were about $50,000–$75,000 and its intermediate-term loans about $100,000. Selected literature we reviewed and interviews with some tribal stakeholders found that tribes have a growing interest in agriculture, motivated by concerns over tribal members’ access to food, health, and employment opportunities. Food access. A 2014 USDA study found that about 26 percent of individuals in tribal areas lived within 1 mile of a supermarket, compared to about 59 percent of all Americans. Health. According to the Centers for Disease Control and Prevention, American Indians and Alaska Natives have higher rates of obesity and diabetes than white Americans. Employment. A 2014 Interior report found that, on average, only about 50 percent of Native American adults in tribal statistical areas were employed either full or part-time. Two commissioned reports on tribal agriculture say that Indian tribes’ vast land base represents an untapped opportunity for tribes to increase agricultural production, including growing their own healthful foods and economic development. But, as previously discussed, for reservations featured in USDA’s 2012 Census of Agriculture, non-Indian producers received a large share of the agricultural revenue. Additionally, the agricultural products grown on tribal lands typically do not feed tribal members and instead are sold into the general agriculture commodity system. Furthermore, these reports and experts we interviewed noted that the growth of agriculture on tribal lands could require access to credit. For example, one tribal agriculture expert told us some tribes are interested in transitioning to “value-added” agriculture, which aims to help the community that produces raw agricultural materials capture the value of the products as they progress through the food supply chain (for example, by processing crops they grow or transitioning to more profitable products, such as organic). Value-added agriculture initiatives might require building facilities or acquiring more expensive inputs, and tribes likely would need financing to support these initiatives. According to some experts and a study we reviewed, if tribes and their members cannot access affordable credit, it could limit the growth of these initiatives. Tribes and their members face several barriers to obtaining agricultural credit, including land tenure issues, administrative challenges, lenders’ legal concerns, and loan readiness issues. As a result, there is limited commercial lending on tribal lands. Ten tribal stakeholders and experts we interviewed cited difficulties in using tribal lands as collateral as a barrier to obtaining credit because of federal laws or other constraints. Tribal trust and restricted fee lands. Federal law generally prohibits lenders from obtaining an ownership interest in tribal trust and restricted fee lands. As a result, tribes are not able to use their 46 million acres of tribal trust or restricted fee lands as collateral for a loan. However, tribes can lease such lands to other parties, including a tribal business or tribal member who wishes to use the land for agricultural purposes (lessees). These lessees can then pledge their “leasehold interest” in the lands as collateral for a loan, but may face challenges in doing so. For example, in general, leases of tribal trust and restricted fee lands must be approved by BIA and comply with its leasing regulations, which stipulate that agricultural leases generally have a maximum term of 10 years. While BIA generally allows leased tribal trust and restricted fee lands to be subject to a leasehold mortgage, three tribal stakeholders and experts we interviewed said that BIA’s maximum term for agricultural leases often was insufficient for obtaining an agricultural loan. Individual trust and restricted fee lands. Unlike tribal trust and restricted fee lands, the owners of individual trust and restricted fee lands can use these lands as collateral for a loan with permission of the Secretary of the Interior. However, many tracts of individual trust and restricted fee lands are allotments with fractionated ownership. According to nine tribal stakeholders and experts we interviewed, fractionated land is a barrier to agricultural activity and obtaining credit. Fractionated land occurs when an allottee dies without a will and ownership is divided among all the heirs, but the land is not physically divided. Thus, multiple owners (in some cases thousands) can have an ownership interest in the land and may have different ideas about how the land should be used. Interior estimated that out of the 92,000 fractionated tracts (representing more than 10 million acres), more than half generated no income in 2006–2011. For agricultural leases and leasehold mortgages on fractionated lands, BIA regulations require consent from owners of a majority interest in such lands. However, according to Interior, some allotments have thousands of co-owners, some of whose whereabouts are unknown, which could make it difficult to obtain their permission for an agricultural lease or a leasehold mortgage. Additionally, as a result of allotment, many Indian reservations contain different land ownership types, creating a “checkerboard” pattern of lands that can make the establishment and financing of large-scale agricultural projects difficult. For example, in addition to tribal and individual trust and restricted fee lands, reservations also may include lands that passed out of trust during the allotment period and were bought by non-Indians. Thus, multiple tracts within a large-scale agricultural project may need to be leased and financed separately because they have different owners and may be subject to different laws. This can also make legal jurisdiction unclear, which is a concern for private lenders financing projects on such lands, as discussed below. Experts and tribal stakeholders we interviewed reported that the barriers to collateralizing various types of tribal lands make it difficult for tribes and tribal members to access different types of agricultural loans. Most long- term loans—typically used for larger projects—generally need to be secured by real estate, which make these inaccessible to tribes and tribal members who do not have land that can be encumbered. For example, an Indian agricultural producer who operates on trust land and wants to build an agricultural facility for a value-added operation may not be able to obtain a long-term loan unless he or she has other unrestricted land to pledge as collateral. In addition, according to the former Executive Director of the Intertribal Agriculture Council, when most agricultural producers face economic distress, they can pledge land as security and receive an extended period of time (20–40 years) to pay off the debt. Tribal members may not have that option, making it difficult to obtain credit in an emergency (such as adverse weather). In addition, according to a tribal agriculture expert and three tribal stakeholders, tribal trust land is not counted as an asset on balance sheets, which may affect an agricultural lender’s assessment of a borrower’s creditworthiness for various types of loans. Processes at Interior—particularly at BIA—can increase the amount of time it takes to obtain a loan, which can discourage both lenders and borrowers, according to tribal stakeholders and experts. Most of the tribal stakeholders and experts we interviewed told us that tribal members often encounter delays when seeking necessary documentation from BIA. For example, for loans involving trust or restricted fee lands, BIA needs to provide a title status report to the lender that identifies the type of land ownership and current owners. Two tribal stakeholders told us that BIA takes months to produce a certified title status report. By that time, the growing season could be over. A representative from a Native CDFI serving a tribe in the Great Plains said it can take years to receive these reports. BIA reported that in fiscal year 2017, it certified 95 percent of land titles within 48 hours. However, BIA’s performance on this measure has varied considerably over the last several years, and BIA officials told us that it can take significantly longer to process title status reports for complicated cases. Tribal members also can encounter administrative challenges at other points in the process. One Native CDFI representative told us she found out that BIA did not record a leasehold mortgage when the CDFI attempted to foreclose on the loan, which almost prevented the CDFI from recovering the loan collateral. In other cases, Interior’s Appraisal and Valuation Services Office might need to conduct an appraisal, such as for an agricultural lease. According to Interior policy, these appraisals should be completed within 60 days, but one tribal economic development expert said they routinely take much longer. As a result of the unique legal status of tribes, some lenders, including FCS associations, reported concerns about their ability to recover loan collateral if the borrower defaulted on a loan involving tribal lands. Seven of the 11 FCS associations we contacted told us that they had legal concerns of this nature, and six of the associations said they had experienced the issues themselves. These concerns primarily arise from the following issues: Tribal sovereign immunity. Tribes are distinct, independent political communities with certain inherent powers of self-government and, as a result of this sovereignty, have immunity from lawsuits. A lender cannot sue to enforce the terms of a loan agreement with a tribe unless the tribe waives its sovereign immunity in connection with the agreement. Private lenders therefore might be hesitant to make a loan because they would not be able to sue the tribe if any disputes arose. We previously reported that tribes may waive sovereign immunity in agreements or contracts on a case-by-case basis and some tribes have formed separate companies to conduct business that are not immune from lawsuits. However, tribal government officials may decide that waiving the tribe’s sovereign immunity for purposes of enforcing the loan agreement is not in the tribe’s best interest. Additionally, tribal sovereign immunity would not bar lenders from seeking to foreclose on loans made to individual tribal members. Legal jurisdiction. Loans made to Indian tribes or their members and secured by tribal lands or collateral located on tribal lands may be subject to tribal laws, rather than state laws. In addition, it is sometimes unclear whether federal, state, or tribal courts would have jurisdiction in the event of a default or foreclosure. If tribal laws govern but do not adequately provide for the lender’s foreclosure, or if there is not a legal forum to hear the foreclosure lawsuit, lenders may be unable to recover the loan collateral. To address these types of concerns, some tribes have adopted secured transaction codes modeled after the Uniform Commercial Code, which can help to assure lenders of their ability to recover collateral in the event of default. Unfamiliarity with tribal laws. Laws and court systems vary among the nation’s 573 tribes, making it more difficult and costly for lenders to learn tribal laws. For example, one FCS association noted that it has many federally recognized tribes in its region, each of which may have different laws. If lenders have concerns regarding their ability to recover loan collateral in the event of a default, lenders may not make loans involving tribal lands due to concerns that the loan would not meet safety and soundness requirements. Five tribal stakeholders we interviewed said some tribal members may need assistance—such as credit repair and technical assistance for loan applications—to become ready for agricultural loans. Some tribal members have no credit history, which can be a barrier to obtaining a loan. One study found that compared to off-reservation counterparts, reservation residents were more likely to have no credit history and when credit scores were available, they were lower on average. Many Native CDFIs provide credit builder or credit repair products to help tribal members qualify for larger loans, such as small business loans. Four tribal stakeholders we interviewed said members of their tribes sometimes need technical assistance to complete the paperwork required for agricultural loans, such as a business plan. One tribal member who owns a ranch told us that the first time he tried to apply for a loan, he had trouble completing the required paperwork and ultimately chose not to apply. He felt tribal members seeking credit would benefit from assistance in completing loan applications. One Native CDFI representative told us that her organization provides technical assistance to its borrowers to help them complete loan paperwork but noted that commercial lenders often did not provide these services. We and others have noted that the barriers described above have depressed commercial lending on tribal lands. In 2010, we found that banks were reluctant to do business on tribal lands because of the cumbersome procedures and their lack of experience. More recently, a report for the Department of Housing and Urban Development surveying lenders found that BIA processing times were a major challenge in making mortgage loans involving tribal lands. A Native CDFI representative told us that lenders have little incentive to engage in a lengthy underwriting process, particularly if the loan is for a small amount and if other potential borrowers have less complicated circumstances. Some experts have described tribal lands as “credit deserts.” For example, one study of three different areas of tribal lands found that few financial institutions or automated teller machines were located on these reservations. One Native CDFI representative told us that in her experience, many people on her reservation never had a bank account. She noted that when people do not have a bank account, it can be challenging for them to see themselves as potential borrowers. Similarly, our analysis found that the land tenure issues, administrative process delays, lenders’ legal concerns, and loan readiness issues can make agricultural loans involving tribal lands more time-consuming and costly to underwrite. For example, one FCS association told us that loans involving tribal lands require specialized legal analysis, which may be an additional expense that it would not incur for otherwise comparable loans. These same issues can increase a lender’s exposure to the risks inherent in agricultural lending because they can affect the borrower’s ability to repay the loan, the adequacy of the collateral to secure the loan, and the lender’s ability to recover the collateral in the event of a default. According to FCA, consistent with the purposes of the Farm Credit Act of 1971, the ability of a lender to collect loans is an important element of the institution’s safety and soundness, and the continued availability of credit. Finally, some stakeholders said they believe that discrimination also contributes to the lack of commercial lending on tribal lands. Four experts, a tribal stakeholder, and a BIA representative told us that they believe that some commercial lenders do not want to make loans involving tribal lands because of bias. As previously discussed, the plaintiffs in the Keepseagle case that USDA settled for $760 million alleged that USDA discriminated against Native American farmers and ranchers in certain programs. According to a tribal economic development expert, tribal members who face discrimination or other negative experiences with commercial lenders may share these experiences with other tribal members and deter them from applying for credit. We found that FCS generally has authority to make loans involving tribal lands. Of the 11 FCS associations we contacted with tribal lands in their territories, some reported that they had recently made loans to Indian tribes or their members, and their outreach to these populations included support for agricultural education. Generally, FCS has authority to provide a broad range of credit services to eligible agricultural producers, which may include tribes, tribal businesses, and individual tribal members operating on various types of tribal lands. However, borrowers must meet various eligibility and underwriting criteria that are required by law. For example, applicants for agricultural loans must be determined to be eligible borrowers, which means they must own agricultural land or be engaged in the production of agricultural products, including aquatic products. Also, long-term real estate loans (which have terms of up to 40 years) made by FCS institutions must be secured by a first-position lien on interests in real estate, thus enabling FCS to obtain ownership or control of the land in the event of default. FCA has determined that this statutory requirement can be satisfied, for example, with leasehold interests in real estate—such as that held by a tribal member leasing reservation land from a tribe—provided that the lease grants the borrower significant rights to the land, and the loan is made on a safe and sound basis. As noted earlier, BIA regulations often limit agricultural leases of tribal lands to a term of up to 10 years. In such cases, FCS associations similarly may limit the term of the related loan (to less than 10 years). According to FCA, when loans are for shorter terms than the leases, the FCS association’s first lien is preserved, as required by law, and the loan is prudent from a safety and soundness perspective. FCA has not issued written guidance indicating whether interests in other types of tribal lands—such as individual trust or restricted fee lands—also satisfy the requirement for a first-position lien on interests in real estate. However, FCA has the authority to determine what types of interests in real estate will satisfy this requirement. Also, according to FCA, there is no statutory requirement that short- and intermediate-term loans be secured with interests in real estate; such loans instead can be secured by other collateral, such as equipment, crops, livestock, and business revenues. In addition to making direct loans to agricultural producers, FCS has authority to lend to non-FCS institutions, such as commercial banks and credit unions, which in turn make agricultural loans to FCS-eligible borrowers. These other financing institutions are known as OFIs. According to FCA, the OFI lending authority allows FCS banks to fulfill their mission as a government-sponsored enterprise by enhancing the liquidity of OFIs, thereby lowering the cost of agricultural credit. As noted earlier, FCS is required to establish programs to serve young, beginning, and small farmers and ranchers, but it is not statutorily mandated to focus on providing financial opportunities to any other group of eligible agricultural producers. Notwithstanding the authorities described above, FCS must comply with other applicable laws and requirements. For example, FCS institutions are subject to safety and soundness oversight by FCA, including with respect to loan underwriting. FCS institutions also must comply with applicable federal, state, and tribal laws governing any tribal lands or property thereon used as loan collateral. FCS associations may obtain Farm Service Agency guarantees on loans to borrowers who otherwise may not meet FCS underwriting requirements. However, by law, loans made by FCS associations are not eligible for a similar BIA loan guarantee program. Based on information from selected FCS associations located near tribal lands, some FCS associations have lent to Indian tribes or their members in the last 2 years. Of the 11 FCS associations we contacted with tribal lands in their territories, representatives of eight told us they had loaned to tribes or their members in the last 2 years—primarily to individual tribal members. We made the following observations based on the associations’ responses: Limited data on lending amounts. Representatives of 10 of the 11 FCS associations we queried stated that they either do not collect or do not maintain data on lending to specific racial populations, thus making it difficult to provide more detailed information on lending to Indian tribes and their members. However, four representatives provided estimates of their recent lending to this population on tribal lands. One association cited more than $25 million in total loans outstanding to a small number of tribes and tribal entities. Another association reported making about $5.5 million in new loans to tribes or their members on tribal lands in the last 2 years. A third reported a $3 million revolving line of credit to a family farm, and the fourth said it had made approximately $150,000 in five separate loans to two tribal members. Loan purposes. Seven associations reported on the type of credit they extended to Indian tribes and their members on tribal lands. In general, they made short-term operating loans and short- and intermediate-term loans for the purchase or refinance of items such as machinery and equipment, livestock, vehicles, or buildings and improvements. Two associations also reported making long-term real estate loans. The other association that reported lending to tribes or their members did not report on the types of loans it made. Type of collateral. Representatives of the eight associations that reported lending to tribes or their members all indicated that the associations secured loans with personal property, such as crops, livestock, or equipment. In addition, the associations that reported making real estate loans said they secured the loans with fee-simple land. Representatives of three FCS associations said they had not loaned to Indian tribes in the past 2 years. One association had not received any credit applications from tribal members, and another could not say if it had served tribal members because of a lack of racial data on borrowers. The third association had not provided loans to tribal members in the past 2 years, but the representative stated that it provided several letters of credit to guarantee the payments of BIA leases on tribal land. Although the FCS associations we contacted stated they have the resources to lend to tribes and their members on tribal lands, a few key factors affect their lending decisions. Representatives of all 11 FCS associations stated their associations had adequate financial capacity and resources to make potentially more complicated or time-consuming loans, such as those involving tribal lands. In general, they stated that the factors they consider in deciding whether to loan to Indian tribes or their members on tribal lands are the same as for any comparable loan—for example, creditworthiness, loan purpose, and the ability to secure a lien on collateral. However, as described earlier, some FCS association representatives described challenges related to tribal law, jurisdiction, tribal sovereign immunity, and recovery of collateral as complicating the lending process to Indian tribes and their members on tribal lands. Although three of the 11 FCS associations we queried reported making loans to tribes that had waived their sovereign immunity for those contracts, most loans the associations reported were to individual tribal members and secured by personal property or fee-simple land. According to two tribal stakeholders we interviewed, Indian tribes or tribal members who received loans from FCS or other commercial lenders may have larger agricultural operations, a longer credit history, and property that can be more easily used as collateral. For example, an established rancher may be able to secure operating loans with his or her cattle herd or interests in fee-simple land, thus preventing the need to rely on trust land as collateral. At the national level, FCS—through its trade association, the Farm Credit Council—conducts and facilitates outreach to tribes and tribal stakeholder groups. According to a representative of the Farm Credit Council, the Council and representatives of associations with tribal lands in their territories participate in an informal FCS working group focused on outreach and lending on tribal lands. One association representative described the group as sharing examples of lending success or reasons for missed opportunities; local, regional or national sponsorship opportunities; local or regional agricultural education events; and relevant legal proceedings, such as the Keepseagle settlement. At the institution level, FCS associations must prepare annual marketing plans describing, among other things, how they will be responsive to the credit needs of all eligible and creditworthy agricultural producers in their respective territories, with a focus on diversity and inclusion. The marketing plan must detail strategies and actions to market their products and services to potential borrowers who may not have been considered previously for reasons other than eligibility or creditworthiness. However, FCS associations are not required to achieve specific outcomes or quantifiable results. Our nongeneralizable review of the marketing plans of the 11 selected FCS associations with tribal lands in their territories and our analysis of their written responses to our queries for additional information found that outreach to tribes and their members focused on educational and charitable initiatives and direct marketing about agricultural lending, or did not directly target tribal populations. Seven of the 11 associations discussed actual or planned outreach to Indian tribes or their members in their marketing plans or written responses. Four of those seven associations cited financial support of specific agricultural education activities for tribes and their members. Two associations reported making charitable donations that benefited tribal members. Four of the seven associations reported direct marketing to potential tribal borrowers. However, in one case, the marketing was a one-time conversation with a tribe regarding financing for a new facility. The other three associations reported that they called potential Indian borrowers, sought referrals from existing tribal member customers, or conducted meetings with tribal government officials. In general, the four remaining associations, in their marketing plans and written responses, addressed outreach to minority producers through broader methods, such as participation in ethnic group organizations or through inclusion in the association’s overall outreach and marketing efforts. In addition, five of the 11 associations discussed outreach to minority producers in conjunction with their statutorily-mandated outreach to young, beginning, and small farmers. According to FCA officials, FCA’s guidance on providing credit to young, beginning, and small farmers, as well as to local food producers, would be broadly applicable to socially disadvantaged or minority populations that fall within the program definitions. Most of the tribal stakeholders with whom we spoke either were not familiar with FCS or did not know of the tribe or any of its members receiving FCS loans. One Native CDFI representative noted that although he was not familiar with any members of his tribe receiving FCS loans, he thought other nearby tribes or their members had worked with FCS. FCA also encouraged FCS associations to develop underwriting procedures to facilitate lending on Indian reservations. FCA identified one FCS association that developed such procedures, and another one of the associations we queried noted that they had such procedures. The first association provided an overview of its procedures, which identified links to information on borrower and collateral eligibility and actions that require BIA approval, among other topics. According to representatives of the second association, its procedure manual directs loan officers to treat tribal members’ applications for loans secured by personal property the same as any other applications. In addition, they said the manual contains instructions for working with BIA for real estate loans to tribal members on trust land and for making direct loans to tribes. Our review of literature and interviews with experts, tribal stakeholders, FCS associations, Farm Credit Council representatives, and FCA officials identified the following options for improving access to agricultural credit on tribal lands. Partnerships with local lenders. Tribal economic development experts and tribal stakeholders cited the importance of commercial or government lenders partnering with Native CDFIs and other Indian- owned lenders, which are the most capable of navigating the challenges related to Indian agricultural credit. According to these experts and stakeholders, if larger commercial or government lenders worked with Native CDFIs or other tribal lenders (such as tribal banks or economic development corporations) to provide funds or conduct outreach, the tribal organizations could more efficiently reach Indian tribes and their members. They noted these organizations are familiar with tribal members and the administrative processes for obtaining loans on tribal land. Partnership with tribal lenders and other tribal businesses also could support tribes’ efforts to improve members’ loan readiness, according to literature we reviewed and a tribal economic development expert and a Native CDFI representative we interviewed. Commercial and government lenders may need to clarify whether tribal lenders with which they might partner meet their lending requirements. For example, although FCS banks have authority to lend to OFIs, which in turn can lend to FCS-eligible borrowers, only certain types of CDFIs may qualify as OFIs. In addition, this authority does not extend to long-term funding, and thus cannot be used to fund agricultural real estate loans made by OFIs. One FCS bank that commented on a 2004 FCA rule noted the latter statutory limitation as a major impediment to OFI program expansion. Flexibility with collateral requirements. As noted earlier, multiple stakeholders we interviewed discussed the challenges related to collateralizing trust land. In addition, FCA officials cited the need for a statutory change or clarification of the requirement that long-term loans made by FCS be secured by a first lien on interests in real estate. They said that by removing or clarifying this requirement, lenders would have authority to provide larger, longer-term loans to creditworthy tribes or tribal members who cannot mortgage their tribal lands. Guarantees. Some stakeholders we interviewed mentioned loan guarantees as an option to improve access to agricultural credit on tribal lands. For instance, FCA officials and Farm Credit Council representatives told us they had spoken with leadership of the Native American Agriculture Fund (created as part of the Keepseagle settlement) regarding the potential establishment of a loan guarantee fund, such as a first-loss fund, which would step in to purchase a loan in default (thus substantially reducing credit risk to the lender). In addition, three of the 11 FCS associations we queried identified guarantees as a possible way to increase FCS lending to Indian tribes and their members on tribal lands. FCS associations still face challenges in using guarantees. With regard to the first-loss loan guarantee fund, FCS associations still must adhere to the FCS statutory requirement for a first-position lien on interests in real estate for long-term loans. According to an FCA official, although the first-loss loan guarantee fund could mitigate repayment risk, a statutory change or clarification would be necessary for FCS associations to accept guarantees in lieu of real estate for long-term loans. And as noted earlier, FCS loans are statutorily ineligible for BIA’s loan guarantee program. Two FCS associations noted that removal of this restriction could increase FCS lending on tribal lands. Finally, FCA officials stated that challenges FCS associations face in making loans involving tribal lands also can extend to Farm Service Agency guarantees on those loans. In other words, to obtain such guarantees, FCS associations must navigate issues around land tenure, legal jurisdiction, and tribal laws. Tribal options. In addition, stakeholders discussed the following tribal actions that could increase credit access for tribes and their members: Representatives of two FCS associations noted that waivers of sovereign immunity (limited to specific contracts) by tribes may increase lending involving tribal lands, as it helps to enable lenders to enforce the terms of loans made to tribes. According to the Office of the Comptroller of the Currency, some banks have negotiated limited waivers of sovereign immunity (restricted to a specific transaction). As noted earlier, tribes may decide that waiving sovereign immunity is not in their best interest. In addition to the limited waivers of sovereign immunity, representatives of three FCS institutions stated that increased adoption of uniform commercial laws (such as the Uniform Commercial Code) by tribes could increase lending involving tribal lands. One tribal economic development expert told us that tribes that adopted their own leasing regulations under the HEARTH Act have seen substantially increased economic development. As noted earlier, the HEARTH Act provides tribes with greater flexibility to enter into leases for agriculture or other purposes. Once a tribe’s leasing regulations have been approved by the Secretary of the Interior, tribes may negotiate and enter into agricultural leases with 25-year terms without further approval by the Secretary. The combination of longer lease terms and the ability to conduct business outside of the BIA approval process can expedite the process of obtaining a leasehold mortgage on tribal trust and restricted fee land. As of May 1, 2019, the Secretary had approved agricultural leasing regulations for seven tribes under the HEARTH Act. We provided a draft of this report to FCA, Interior, and USDA for review and comment. FCA and USDA provided technical comments, which we incorporated as appropriate. In comments provided in an email, Interior officials noted that efforts to simplify the Secretary of the Interior’s approval process could provide faster mortgage determinations and thus may result in expanded lending and production opportunities for Indian agricultural producers. We are sending copies of this report to the appropriate congressional committees, the Chairman and Chief Executive Officer of the Farm Credit Administration, the Secretary of the Interior, and the Secretary of Agriculture. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Our objectives in the report were to describe (1) what is known about the agricultural credit needs of Indian tribes and their members on tribal lands, (2) the barriers stakeholders and experts identified that Indian tribes and their members on tribal lands face in obtaining agricultural credit to meet their needs, (3) the Farm Credit System’s (FCS) lending authority and lending and outreach activities on tribal land, and (4) suggestions stakeholders have discussed to improve access to agricultural credit on tribal lands. For the purpose of this report, we use the term “tribal lands” to refer to reservations (including all land within the reservations’ boundaries), trust land, allotments, and restricted fee land. In general, our report focuses on the agricultural credit needs of tribes and their members in the lower 48 states. To describe what is known about the agricultural credit needs of Indian tribes and their members on tribal lands, we explored various potential data sources on agricultural loans that Indian tribes and their members applied for or received. We reviewed available data from the Consumer Financial Protection Bureau and Department of Agriculture (USDA). For example, we obtained borrower-reported loan data from USDA’s Agricultural Resource Management Survey, but for several data fields related to Indian producers on tribal lands, sample sizes were too small or the coefficients of variation were too high to produce reliable estimates. We also reviewed provisions of the Equal Credit Opportunity Act, federal regulations, and other legal documentation pertaining to collection of data regarding the personal characteristics of applicants for nonresidential loans. To describe what is known about Indian tribes and their members’ agricultural credit needs and the barriers they face in obtaining agricultural credit, we conducted a literature review. We conducted searches of various databases, such as EBSCO, ProQuest, Google Scholar, and Westlaw to identify sources such as peer-reviewed academic studies; law review articles; trade and industry articles; reports from government agencies, nonprofits, and think tanks; and Congressional transcripts related to tribal agriculture, barriers to accessing credit on tribal lands, and FCS. We identified additional materials through citations in literature we reviewed. In addition, we reviewed statutes and the Department of the Interior’s Bureau of Indian Affairs’ (BIA) regulations related to use and ownership of tribal lands, including leasing. To describe FCS’s authority and lending and outreach activities on tribal lands, we reviewed statutes and regulations governing FCS, as well as written guidance issued by the Farm Credit Administration (FCA). We also reviewed the marketing plans of a nongeneralizable sample of 11 FCS associations (16 percent of the 69 FCS associations that lend directly to agricultural producers) whose territories included large tribal land areas with high levels of agricultural activity, including the tribes we interviewed (described below). We selected an additional FCS association but on closer review realized it did not have a significant amount of tribal land in its territory; we therefore excluded this association from our analysis. For comparison purposes, we also reviewed three marketing plans from FCS associations that did not have significant tribal populations in their territories. In addition to reviewing the marketing plans, we sent the 11 FCS associations a questionnaire about their lending and outreach to tribes and their members and any challenges in making loans involving tribal lands. We also asked these associations about any suggestions to improve access to agricultural credit on tribal lands. We received responses from all 11 FCS associations, and followed up with some associations to clarify information they provided. While the sample allowed us to learn about many important aspects of FCS associations’ lending and outreach to tribes and their members on tribal lands, it was designed to provide anecdotal information, not findings that would be representative of all of 69 FCS lending associations. To address all four objectives, we attempted to interview representatives of six tribes. First, we selected these tribes to represent five regions (Great Plains, Rocky Mountain, Northwest, Southwest) and a state (Oklahoma) that—according to experts we interviewed—have tribes engaged in agricultural activity. Within these regions, we generally selected large tribal land areas that have high levels of agricultural activity, as indicated by the USDA 2012 Census of Agriculture data. Specifically, we selected tribes based on number of farms, land in farms, and market value of agricultural products. In addition, we selected one of the six tribes because two experts recommended that we speak with them. For the six tribes, we contacted tribal government leaders and employees of the relevant government offices, such as the agriculture or tribal lands departments. For two of the six tribes, we interviewed employees of the tribal agriculture department. One of these interviews also included representatives of the Native Community Development Financial Institution (Native CDFI) that serves the reservation. For the third tribe, we received written responses from a tribal farm. For the fourth tribe, we interviewed a representative of the Native CDFI that serves the reservation. For this series of interviews, we only received information relating to four tribes. We did not obtain meetings with relevant tribal government officials for the last two tribes. We also contacted farms or Native CDFIs associated with an additional three tribes based on USDA data or recommendations from experts we interviewed. For one of these tribes, we interviewed a tribal farm employee and a representative of the tribe’s community development corporation. For the second tribe, we interviewed a tribal farm employee. For the third tribe, we interviewed a representative of the Native CDFI that serves the reservation. In summary, we interviewed employees of two tribal agriculture departments, employees of three tribal farms, and representatives of three Native CDFIs and one tribal community development corporation. Throughout this report, we refer to tribal government employees, tribal farm employees, or representatives of Native CDFIs or community development corporations serving a tribe as “tribal stakeholders.” Although the information we obtained from the tribal agriculture employees allowed us to provide anecdotal tribal perspectives, it is not generalizable to the 573 federally recognized Indian tribes. In addition, the views of tribal farm employees and Native CDFI and community development corporation representatives cannot be generalized to tribes but illustrate views on needs, barriers, and other issues from the perspectives of the organizations. In addition, for all four objectives, we interviewed the following: Experts on agricultural and economic development on tribal lands. We interviewed subject matter experts on tribal agriculture and economic development from various organizations, including advocacy and academia. Specifically, we interviewed representatives of the following organizations: the Center for Indian Country Development at the Federal Reserve Bank of Minneapolis, First Nations Oweesta Corporation, the Indian Land Tenure Foundation, the Indigenous Food and Agriculture Initiative at the University of Arkansas, the Intertribal Agriculture Council, and the Native American Agriculture Fund. We selected these organizations based on relevant publications, testimonies before Congress, or recommendations from other experts. These organizations work with a number of tribes and thus could speak to general trends or commonalities in tribal agriculture and economic development. Throughout the report, we refer to the representatives of these organizations as “experts.” Agency and trade group representatives. We interviewed officials from FCA, USDA (including the Farm Service Agency, Economic Research Service, and National Agricultural Statistics Service), and BIA. We also interviewed representatives of the Farm Credit Council, the national trade association for the Farm Credit System. We conducted this performance audit from December 2018 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Karen Tremba (Assistant Director), Lisa Reynolds (Analyst in Charge), Miranda Berry, Tom Cook, Anne-Marie Fennell, John Karikari, Marc Molino, Kirsten Noethen, Barbara Roesmann, Jeanette Soares, and Farrah Stone made significant contributions to this report.", "summary": "About 46 million of the 56 million acres of the land that the federal government holds in trust for the benefit of Indian tribes and their members has an agricultural purpose. However, tribal agriculture and economic development experts have noted that Indian tribes and their members may need improved access to agricultural credit. Congress included a provision in statute for GAO to review the ability of FCS to meet the agricultural credit needs of Indian tribes and their members on tribal lands. This report describes (1) what is known about the agricultural credit needs of Indian tribes and their members, (2) barriers stakeholders identified to agricultural credit on tribal lands, (3) FCS authority and actions to meet those agricultural credit needs, and (4) stakeholder suggestions for improving Indians' access to agricultural credit on tribal lands. GAO explored potential data sources on Indians' agricultural credit needs, conducted a literature review, and reviewed statutes and regulations governing tribal lands and FCS. GAO also reviewed the marketing plans and written responses of a nongeneralizable sample of 11 FCS associations whose territories included tribal lands with high levels of agricultural activity. GAO interviewed stakeholders from a sample of seven tribes (generally selected based on tribal region and agricultural activity), experts in tribal agriculture and economic development (selected based on relevant publications, Congressional testimonies, and others' recommendations), and representatives from FCS and its regulator, the Farm Credit Administration, and other relevant government agencies. Limited data are available on the needs of Indian tribes and their members for agricultural credit, such as operating or equipment loans, to develop and expand agricultural businesses on tribal lands. Federal regulations have generally prohibited lenders from inquiring about the personal characteristics, such as race, of applicants on nonresidential loans. Some tribal stakeholders and experts said that tribal members may not have applied for agricultural credit because they heard of other tribal members being denied loans. They said that tribal members likely obtain agricultural credit from Department of Agriculture programs or tribal lenders. Another potential source of agricultural credit is the Farm Credit System (FCS), a government-sponsored enterprise that includes 69 associations that lend to farmers and ranchers. Tribal stakeholders and experts reported a general lack of commercial credit on tribal lands due to the following factors: Land use restrictions. Most tribal lands only can be used as loan collateral in certain circumstances or with federal permission. Administrative process delays. Tribal members reported often encountering delays obtaining necessary federal loan documents. Legal challenges. Lenders reported concerns about their ability to recover loan collateral due to the unique legal status of tribes. Loan readiness. Tribal members may have no or poor credit histories and be unfamiliar with the paperwork required for an agricultural loan, such as a business plan. FCS is authorized to provide a range of credit services to eligible agricultural producers, which may include Indian tribes, tribal businesses, and tribal members. FCS associations must obtain land as collateral for long-term real estate loans, but are not required to do so for shorter-term loans, such as for operating costs or equipment purchases. Some FCS associations GAO contacted reported making loans to Indian tribes or their members. In a sample of 11 FCS associations with tribal lands in their territory, eight said they have loaned to tribes or their members in the past 2 years. GAO's review of these 11 associations' marketing plans and written responses to GAO follow-up questions found that seven noted outreach—such as support for agricultural education activities—targeted to tribes and their members. The other four reported broad and general outreach efforts that also included minority groups. To improve access to agricultural credit on tribal lands, stakeholders discussed several options. For example, some stakeholders discussed the potential for partnerships between commercial or government lenders and tribal lenders (such as Native Community Development Financial Institutions) and increased use of loan guarantees. Some stakeholders also discussed actions tribes could take to ease barriers to lending, such as adopting their own leasing procedures to reduce administrative processing time with federal agencies for certain loans.", "document_type": "gao"}
{"report": "OMB provides guidance to federal managers on how to improve accountability and effectiveness of federal programs and operations by identifying and managing risks. OMB updated its Circular No. A-123 in July 2016 to establish management’s responsibilities for ERM. As part of the overall governance process, ERM calls for the consideration of a risk across the entire organization and how it may interact with other identified risks. When used appropriately, ERM is a decision-making tool that allows agency leadership to view risks across an organization and helps management understand an organization’s portfolio of top risk exposures, which could affect achievement of the agency’s goals and objectives. In December 2016, we issued a report that provided an overall framework for agencies to build an effective ERM program. In July 2016, OMB also updated Circular No. A-11, Preparation, Submission, and Execution of the Budget. In Circular No. A-11, OMB referred agencies to Circular No. A-123 for requirements related to ERM implementation, including for developing a risk profile as a component of the agency’s annual strategic review. A risk profile is a prioritized inventory of the most significant risks identified and assessed through the risk assessment process. It considers risks from a portfolio perspective, identifies sources of uncertainty that are both positive (opportunities) and negative (threats), and facilitates the review and regular monitoring of risks. Together, these two OMB circulars constitute the ERM policy framework for executive agencies by integrating and operationalizing specific ERM activities and helping to modernize existing risk management efforts. Standards for Internal Control in the Federal Government describes internal control as a process put in place by an entity’s oversight body, management, and other personnel that provides reasonable assurance that objectives related to performing operations effectively and efficiently, producing reliable internal and external reports, and complying with applicable laws and regulations will be achieved. Internal control serves as the first line of defense in safeguarding assets. Its importance to federal agencies is further reflected in permanent requirements enacted into law. The internal control processes required by FMFIA and the Standards for Internal Control in the Federal Government help to form an integrated governance structure designed to improve mission delivery, reduce costs, and focus corrective actions toward key risks. OMB Circular No. A-123 precludes agencies from concluding that their internal control is effective if there are one or more material weaknesses identified from its assessment. As a component of DOD, the Air Force is required to (1) identify and manage risks, (2) establish and operate an effective system of internal control, (3) assess and correct control deficiencies, and (4) report on the effectiveness of internal control through an annual Statement of Assurance. In addition, the Chief Financial Officers Act of 1990 (CFO Act), as amended by the Government Management Reform Act of 1994 and implemented by guidance in OMB Bulletin No. 19-03, Audit Requirements for Federal Financial Statements (August 27, 2019), requires the Air Force to annually undergo a financial statement audit. However, since 1990, the Air Force has continued to be unable to demonstrate basic internal control that would allow it to pass a financial statement audit, which has contributed to DOD’s financial management remaining on the GAO High-Risk List since 1995. For fiscal year 2018, the Air Force reported 11 material weaknesses in internal control over operations and 14 material weaknesses in internal control over reporting in its Statement of Assurance. For fiscal year 2019, it reported the same number of operations-related material weaknesses, and its reporting-related material weaknesses increased to 25. During the Air Force’s fiscal years 2018 and 2019 financial statement audits, independent auditors specifically considered the Air Force’s internal control over financial reporting in order to determine appropriate audit procedures to perform in order to express an opinion on the financial statements. The independent auditors disclaimed an opinion on the Air Force’s fiscal years 2018 and 2019 financial statements, stating that the Air Force continued to have unresolved accounting issues, and for each year, the auditors reported 23 material weaknesses in internal control over financial reporting. These material weaknesses included control deficiencies in processes related to the Air Force’s mission-critical assets and involved a lack of policies and procedures, inadequate financial information systems and reporting, and inaccurate and incomplete information in its accountability records and financial reports. The Air Force’s efforts to implement ERM are in the early stages, and accordingly, it has not fully incorporated ERM into its management practices. Since the July 2016 update to OMB Circular No. A-123 required agencies to implement ERM, the Air Force has been leveraging and relying on its existing risk management practices. To date, these practices have focused on the organizational unit level and not at the entity level, as required by OMB Circular No. A-123. The Air Force plans to integrate ERM increasingly into its management practices over the next several years, with expectations of a fully developed ERM approach after fiscal year 2023. The Air Force has taken the initial steps to establish an ERM governance structure, define risk classifications, and develop its ERM framework. For instance, the Air Force has drafted charters updating responsibilities for two senior management advisory councils—(1) the Enterprise Productivity Improvement Council (EPIC) and (2) the Executive Steering Committee (ESC)—to implement OMB Circular No. A-123. EPIC will oversee the agency’s risk management function, with a specific emphasis on overseeing the regular assessment of risk and approving risk responses and the Air Force’s risk profile. ESC will lead the implementation, assessment, and documentation of risk management over financial reporting, financial systems, all associated activities, and oversight with respect to the Air Force’s internal control program. EPIC is designed to focus exclusively on potential operational material weaknesses, and ESC will focus on potential financial reporting and financial systems material weaknesses. Air Force officials informed us that both councils would share responsibility for compliance objectives and resulting material weaknesses. During our audit, we analyzed the Air Force’s financial reports beginning with those for fiscal year 1999 and noted that the agency and the external auditors have generally reported material weaknesses each year involving the tracking, reporting, location, accountability, and cost of certain mission-critical assets. These weaknesses identified risks that decreased the Air Force’s ability to perform operations efficiently, prepare reliable financial reports, and comply with applicable laws and regulations. EPIC and ESC currently assess proposed material weaknesses that the primary reporting elements (PRE) submit and determine whether to recommend them to the Secretary of the Air Force for reporting in the annual Statement of Assurance. However, the Air Force’s governance structure does not include a mechanism for EPIC or ESC to oversee the management of risk associated with material weaknesses and consider its effect across the entire agency. Based on our review of the draft charters and documentation from governance meetings, the Air Force included provisions for ESC to identify material weaknesses related to financial reporting and financial systems and EPIC to identify material weaknesses related to operations objectives. However, there were no charter provisions for either council to identify, assess, respond to, and report on the risks associated with those material weaknesses or material weaknesses identified through external audits. A material weakness, reported by either the agency or an external auditor, by definition indicates a significant decrease in an agency’s ability, during the normal course of operations, to achieve objectives and address related risks. Under OMB Circular No. A-123, an agency’s risk management governance structure helps ensure that the agency identifies risks that have the most significant effect on the mission outcomes of the agency. Without a thorough and integrated ERM governance structure that includes oversight responsibilities managing risks associated with material weaknesses in internal control, there is an increased risk that the Air Force will not properly identify, assess, and respond to significant entity-level risks. The Air Force’s current internal control assessment process is not designed to facilitate the timely identification and correction of internal control deficiencies or to be used to support the Air Force’s annual Statement of Assurance. Specifically, Air Force management has not designed an adequate process for assessing internal control. Further, the process does not focus on areas with the greatest risk, such as mission- critical assets. In addition, the reviews of mission-critical assets in fiscal years 2018 and 2019 in support of the financial statement audit did not result in adequate assessments of internal control. The Air Force’s policy for assessing the effectiveness of its internal control system and for preparing the agency’s annual Statement of Assurance is based on DOD Instruction 5010.40, Managers’ Internal Control Program Procedures, dated May 2013. The Air Force’s policy is outlined in Air Force Policy Directive 65-2, Managers Internal Control Program. This policy is supported by the procedures outlined in Air Force Instruction (AFI) 65-201, Managers Internal Control Program Procedures, dated February 2016, which the Air Force currently is revising to address the July 2016 OMB Circular No. A-123 update. The Air Force provides additional guidance to supplement AFI 65-201 in its Statement of Assurance Handbook and its Internal Control Playbook. The Air Force’s OMB Circular No. A-123 program comprises 17 designated PREs, including the Secretariat and Air Force staff offices, major commands, the Army and Air Force Exchange Service, and direct- reporting units. The Air Force subdivides each PRE along organizational lines into more than 6,500 organizational assessable units (organizational units), such as a squadron or wing, and other specific programs and functions, where it evaluates internal controls per AFI 65-201. Each of the organizational units has an assessable unit manager (unit manager) who has authority over the unit’s internal control, including continual monitoring, testing, and improvement. Figure 1 illustrates how the Air Force’s organizational structure informs its overall annual Statement of Assurance. The Air Force requires each unit manager to submit an annual supporting statement of assurance providing the manager’s opinion on whether the unit has reasonable assurance that its internal controls are effective. The units submit the statements to the Assistant Secretary of the Air Force, Financial Management and Comptroller (SAF/FM), the office responsible for OMB Circular No. A-123 implementation and compilation of the annual Statement of Assurance. Based on discussions with Air Force officials, SAF/FM uses the unit managers’ supporting statements of assurance to develop the overall Air Force annual Statement of Assurance. The Air Force’s internal control assessment process does not require (1) an assessment of all required elements of an effective internal control system; (2) test plans that specify the nature, scope, and timing of procedures to conduct; and (3) management validation of results. In addition, existing policies and procedures that staff follow to perform the assessments do not fully implement OMB Circular No. A-123. Further, the Air Force provided inadequate training to those responsible for conducting and concluding on the internal control assessments. Although not required by policy, the Air Force performed its first assessment of the five components of internal control during fiscal year 2019 through an SAF/FM review of entity-level controls, which are controls that have a pervasive effect on an entity’s internal control system and may pertain to multiple components. Based on this assessment, SAF/FM concluded in the Air Force’s Statement of Assurance for fiscal year 2019 that three components of internal control (i.e., risk assessment, control activities, and information and communication) were not designed, implemented, or operating effectively. Although SAF/FM performed this assessment in 2019, the assessment did not include a determination of whether each internal control principle was designed, implemented, and operating effectively. Also, there was no indication that the Air Force designed the assessment of entity-level controls to be pertinent to all Air Force objectives, such as those related to operations, reporting, or compliance. In addition, SAF/FM did not provide the assessment results to the unit managers for input or consideration in their unit-specific control assessments and supporting statements of assurance. The Air Force’s Internal Control Playbook directs unit managers to assess the design and operating effectiveness of the relevant entity-level controls within their purview. However, for fiscal year 2019, SAF/FM performed this assessment, and officials informed us that it was not their intent for unit managers to assess entity-level controls. According to OMB Circular No. A-123, management must summarize its determination of whether each of the five components and 17 principles from Standards for Internal Control in the Federal Government are designed, implemented, and operating effectively and components are operating together in an integrated manner. The determination must be a “yes/no” response. If one or more of the five components are not designed, implemented, and operating effectively, or if they are not operating together in an integrated manner, then an internal control system is ineffective. AFI 65-201 states, as part of its discussion on assessing internal control over financial reporting, that OMB Circular No. A-123 prescribes a process to evaluate controls at the entity level for the five components of internal control (i.e., control environment, risk assessment, control activities, information and communication, and monitoring). The Air Force’s assessment lacked required determinations related to internal control principles because the Air Force lacked policies or procedures for the following: Clearly delineating who within the Air Force (e.g., unit managers or SAF/FM) is responsible for assessing the components and principles of internal control, how often assessments are performed, at what level (e.g., entity or transactional) components and principles are to be evaluated, what objectives are covered in the assessment of entity-level controls, to whom to communicate the results if the results are relevant to others performing assessments of internal control, and what Air Force guidance to follow. Documenting management’s summary, whether performed by the unit managers as outlined in the guidance or by SAF/FM as performed during fiscal year 2019, of its determination of whether each component and principle is designed, implemented, and operating effectively and whether components are operating together in an integrated manner. By not ensuring that management is assessing whether each internal control component and principle is designed, implemented, and operating effectively, the Air Force cannot determine whether internal control is effective at reducing the risk of not achieving its stated mission and objectives to an acceptable level. Moreover, given the entity-wide relevance of SAF/FM’s conclusions, unit managers may not be aware of all the necessary information with which to draw conclusions about the effectiveness of their organizational units’ internal control. Further, management’s assurances on internal control effectiveness, as reported in the Statement of Assurance, may not appropriately represent the effectiveness of the Air Force’s internal control. The Air Force did not have a process in place to base its annual assessment of internal control and Statement of Assurance preparation on uniform testing performed across its agency. Although the Air Force had standard test plans for reviews associated with financial reporting objectives, SAF/FM could not demonstrate what procedures are performed to support its assessment of internal control over its operational, internal reporting, and compliance objectives. Specifically, for these objectives, the Air Force did not develop guidance for those responsible for assessing internal controls on which tests to conduct to obtain the best evidence of whether controls are designed, implemented, and operating effectively; how much testing is needed in each area; when to conduct the tests; how to ensure that current year conclusions are based on current year how assessment procedures are to be adjusted or amended to reflect a consideration of prior year self-identified control deficiencies and internal and external audit results. Additionally, standard test plans for the reviews conducted as part of the Air Force’s financial statement audit remediation efforts did not include guidance on how to consider prior year self-identified control deficiencies and internal and external audit results in determining the nature, timing, and extent of procedures to be conducted for the current year. Further, although the Air Force outlines 20 overall objectives in its 2019 through 2021 Business Operations Plan (dated January 2019), it did not document the specific procedures the Air Force planned and performed to support an evaluation of its internal control over these 20 objectives. According to Standards for Internal Control in the Federal Government, management should establish and operate activities to monitor the internal control system and evaluate the results and should remediate identified internal control deficiencies on a timely basis. For example, as part of its monitoring activities, agency management responsible for the OMB Circular No. A-123 program could design a test plan or establish a baseline to monitor the current state of the internal control system and compare that baseline to the results of its internal control tests. The Air Force’s assessment of internal control and Statement of Assurance are not clearly supported by completed test plans or other documented monitoring activities because SAF/FM does not have a policy or procedures for conducting internal control assessments that require documented test plans that (1) tie back to specific objectives included in the Business Operations Plan; (2) specify the nature, scope, and timing of procedures to conduct under the OMB Circular No. A-123 assessment process; and (3) reflect a consideration of prior year self- identified control deficiencies and results of other internal and external audits. By not ensuring that its more than 6,500 unit managers are evaluating internal control based on the agency’s established baseline, the Air Force cannot ensure that it is consistently and effectively assessing its internal control in order to timely identify and correct deficiencies or that its design of internal control reduces, to an acceptable level, the risk of not achieving agency operational, reporting, and compliance objectives. As a result, Air Force management’s assurances on internal control, as reported in the overall agency Statement of Assurance, may not appropriately represent its internal control effectiveness. Air Force management did not have a process to validate whether its unit managers appropriately performed and documented their internal control assessments. During our review, Air Force management was uncertain about how many internal control assessments were being performed or by whom. SAF/FM officials initially stated that there were 5,567 organizational units responsible for assessing internal control, but officials later informed us that the actual number was more than 6,500. Furthermore, Air Force officials were unable to provide information on how many organizational unit managers failed to report on their specific internal control assessments or received waivers from performing such assessments. Finally, management lacked a process to ensure that results used to compile the current year Statement of Assurance are based upon current fiscal year assessments. The Air Force requires unit managers to assess internal control and submit results to SAF/FM through the automated statement of assurance submission system. SAF/FM then compiles the supporting statements of assurance submissions and prepares the Air Force’s annual Statement of Assurance. However, we found that the automated system that collects the annual assessments from more than 6,500 unit managers allows these managers to import internal control testing activities from the prior fiscal year. Air Force officials were unable to provide information about how they ensure that unit managers were not importing prior year results without performing current year testing. OMB Circular No. A-123 requires documentation to demonstrate and support conclusions about the design, implementation, and operating effectiveness of an entity’s internal control system, and requires agencies to consider carefully whether systemic weaknesses exist that adversely affect internal control across organizational or program lines. The Air Force’s process lacks management validation of results because it has not developed a documented policy or procedures to ensure that management can readily review and validate the results of its internal control testing. The Air Force has not required SAF/FM to validate (1) the number of organizational units reporting for its overall internal control assessment; (2) how it tested control procedures, what results it achieved, and how it derived conclusions from those results; and (3) whether it based the results used to compile the current year Statement of Assurance on current fiscal year assessments. Additionally, when PRE management waives assessments, SAF/FM does not have a process to track waivers and assess how they affect the current year assessment of internal control, determination of systemic weaknesses, and compilation of the Air Force’s overall Statement of Assurance. By not validating the internal control assessment results, Air Force management cannot ensure that the assessment was performed as expected to support related conclusions and timely identify internal control deficiencies. Further, management’s assurance on internal control, as reported in the overall Statement of Assurance, may not appropriately represent the internal control effectiveness. Air Force guidance for its assessment of internal control neither accurately nor completely reflects definitions included in OMB Circular No. A-123. For example, AFI 65-201 and the Statement of Assurance Handbook provided to unit managers for conducting internal control assessments, and the Internal Control Playbook that the Air Force developed in August 2019 to address internal control over reporting objectives, do not include the complete definitions of the four material weakness categories for deficiencies related to (1) operations, (2) reporting, (3) external financial reporting, and (4) compliance objectives, consistent with guidance in OMB Circular No. A-123. Additionally, the handbook does not define internal control as a process that provides reasonable assurance that objectives will be achieved or an internal control system as a continuous built-in component of operations, affected by people, that provides reasonable assurance that an entity’s objectives will be achieved. Although the playbook does adequately define internal control and a system of internal control, the Air Force developed this guidance after we initiated our review, and the guidance only addresses internal control over reporting objectives and not operational and compliance objectives. These inaccuracies and incomplete descriptions occurred because the Air Force did not provide its internal control assessment guidance preparers or reviewers with training to assist them in writing and reviewing the guidance to ensure proper application of the fundamental concepts of internal control and OMB Circular No. A-123, such as those related to definitions of internal control and material weakness. By not ensuring that Air Force guidance reflects accurate and complete definitions included in OMB Circular No. A-123, the Air Force is at increased risk that its officials performing internal control assessments will not properly conclude on the results; therefore, management’s assurances on internal control, as reported in the Statement of Assurance, may not appropriately represent the effectiveness of internal control. Among other things, OMB Circular No. A-123 requires staff to identify objectives, assess related risks, document internal controls, evaluate the design of controls, conduct appropriate tests of the operating effectiveness of controls, report on the results of these tests, and appropriately document the assessment procedures. However, the Air Force’s training provided to unit managers responsible for assessing internal control lacks sufficient instructions on how to perform such assessments. Specifically, the current annual training provided by SAF/FM lacks instruction on how to prepare documentation to adequately support conclusions, identify and test the key internal controls, and evaluate and document test results; limits discussion of OMB Circular No. A-123 internal control assessments to internal control over external financial reporting objectives and does not cover internal control over operational, compliance, and internal reporting objectives; lacks adequate definitions of material weaknesses included in OMB Circular No. A-123; lacks instruction on how to interpret, respond to, and correct self- identified deficiencies (control deficiencies, significant deficiencies, and material weaknesses); and is not required for individuals performing reviews related to external financial reporting. SAF/FM officials informed us that the definitions of material weakness and instructions on how to interpret, respond to, and correct deficiencies were included in other guidance documents, such as the newly created Internal Control Playbook. However, the Air Force did not provide the playbook to PREs during the fiscal year 2019 training, and it is not officially named as guidance in the Air Force’s policy for assessments of internal control. Although the Air Force has described the playbook as supplemental guidance, it does not refer to the playbook as such in its policy for assessing the effectiveness of its system of internal control to provide reasonable assurance that operational, reporting, and compliance objectives are achieved. These inadequacies occurred because SAF/FM has not fully evaluated and incorporated the requirements for assessing an internal control system into its training and has not designed training that (1) enhances skills in evaluating an internal control system and documenting the results; (2) reflects all OMB Circular No. A-123 requirements, such as those related to assessing controls for all objectives and determining material weaknesses; and (3) is provided to all who are responsible for performing internal control assessments. According to federal internal control standards, management should demonstrate a commitment to developing competent individuals. For example, management could provide training for employees to develop skills and competencies needed for key roles and responsibilities in assessing internal control. Without appropriate training, those responsible for assessing internal control may not do so adequately enough to identify internal control deficiencies timely and support the agency’s internal control assessments with appropriate documentation and summarization of the results. OMB Circular No. A-123 requires an agency to evaluate whether a system of internal control reduces the risk of not achieving the entity’s objectives using a risk-based assessment approach. However, the Air Force’s current AFI 65-201 approach calls for assessing internal control at more than 6,500 organizational units without regard to quantitative or qualitative risks. As previously discussed, the Air Force lacks procedures to verify whether its unit managers are performing internal control assessments as intended and does not provide guidance for uniform testing across the organization. Therefore, the Air Force’s current approach for assessing internal control does not ensure that areas of greatest risk are addressed, such as mission-critical assets, and instead may unnecessarily focus on areas of lower risk. As a result, the Air Force may not be using resources efficiently. The Air Force’s current design of assessing internal control does not ensure, at a minimum, the evaluation of internal control over areas key to meeting its mission. Specifically, the Air Force does not have a policy requiring evaluation of whether its internal control over processes related to areas of highest risk—such as processes related to mission-critical assets, including equipment, government-furnished equipment, and weapons-system spare parts managed and held by contractors and working capital fund inventory—reduces the risk of not achieving specific operation, reporting, or compliance objectives to an acceptable level. The Acting Secretary of Defense, during fiscal year 2019, emphasized two of these areas—government property in the possession of contractors, which includes government-furnished equipment, and working capital fund inventory—as high priority for corrective actions related to financial statement audit remediation. The Air Force’s current approach for assessing internal control calls for more than 6,500 organizational units to perform assessments without regard to risk because the Air Force has not developed a policy or procedures providing guidance on how to perform the assessment using a risk-based approach. A risk-based approach provides a methodology for Air Force management to focus and prioritize its internal control assessments on areas and activities of greater risk and importance to accomplishing mission and strategic objectives. By not evaluating internal control with a risk-based approach, Air Force management lacks the assurance that resources are used efficiently to assess key controls associated with achieving Air Force objectives subject to the highest risks along with those designated as high priority by agency management, such as controls over accounting for, managing, and reporting on mission-critical assets. Although the Air Force has not designed a process for performing OMB Circular No. A-123 internal control assessments based on risk, it did review certain business process assessable units, such as mission-critical assets, as part of its financial statement audit remediation efforts. However, Air Force’s reviews of internal control over processes related to mission-critical assets did not meet OMB Circular No. A-123 requirements or federal internal control standards for evaluating a system of internal control. During fiscal years 2018 and 2019, the Air Force engaged the Air Force Audit Agency (AFAA) to review control activities for five processes related to mission-critical assets and instructed business process assessable unit leads to conduct additional internal control reviews for select mission-critical asset areas during fiscal year 2019. However, the organizational unit managers did not formally consider the results of these reviews when concluding on their assessments of internal control. For fiscal year 2018, AFAA performed certain agreed-upon procedures to confirm current transactional processes and related internal control over external financial reporting for five mission-critical asset areas as documented in the related business process cycle memorandums. In order to perform the procedures, AFAA used SAF/FM-prepared templates to confirm certain processes and key controls included in the respective process cycle memorandums. However, the procedures SAF/FM instructed AFAA to perform in 2018 did not meet the requirements of an assessment of an internal control system as prescribed in OMB Circular No. A-123. Specifically: Procedures to test design of controls did not include steps for evaluating whether the controls individually or in combination with other controls would achieve objectives or address related risks. Instead, SAF/FM instructed AFAA to confirm whether the process cycle memorandums accurately reflected the controls and processes in place. Procedures to test operating effectiveness of controls were conducted even though there was no determination of whether the controls were designed to achieve objectives or address related risks. Procedures performed involved the use of process cycle memorandums as a baseline, which, as noted by the Air Force’s auditor, did not always reflect the current process, and there was no process in place for management to assess whether the differences related to an inaccurate cycle memorandum or improper implementation of the process. For fiscal year 2019, tests continued to (1) address operating effectiveness without first determining if the controls were designed to meet objectives and reduce risks and (2) involve the use of process cycle memorandums as a baseline that did not always reflect the current business process. For fiscal year 2019, business process assessable unit leads conducted the additional internal control reviews for select processes related to mission-critical assets based on the templates for tests of design and tests of operating effectiveness in Internal Control Playbook appendixes. Similar to the procedures developed for AFAA, the Air Force did not devise the fiscal year 2019 playbook’s template procedures to support conclusions on the design, implementation, and operating effectiveness of internal control over processes that are key to achieving Air Force operational, internal reporting, and compliance objectives. For example, the procedures that the Air Force used to assess the design of internal control over a process related to spare engines at one air base only considered controls related to external financial reporting objectives. The Air Force did not provide evidence that it tested additional controls key to achieving internal reporting, operating, and compliance objectives, such as improving and strengthening business operations and harnessing the power of data for timely decision-making and mission success, or evidence that the Air Force would test such controls during future reviews. Additionally, the Air Force lacked a process for the organizational unit managers or PREs to consider the results of internal control reviews performed at the business process assessable unit level in assessing internal control when they assess and report on the status of internal control for the overall Air Force Statement of Assurance (see fig. 2). Specifically, the current and draft AFI 65-201 and Statement of Assurance Handbook do not include procedures for how information gathered from AFAA agreed-upon procedures or business process unit leads’ testing of internal control over processes related to mission-critical assets is considered in the conclusions reported through the organizational unit managers’ supporting statements of assurance. OMB Circular No. A-123 requires that management, in accordance with federal standards for internal control, evaluate whether a system of internal control reduces the risk of not achieving the entity’s objectives related to operations, reporting, or compliance to an acceptable level. According to the federal internal control standards, when evaluating the design of internal control, management determines if controls individually and in combination with other controls are capable of achieving an objective and addressing related risks. A control cannot be effectively operating if it was not properly designed and implemented. Further, management should establish and operate monitoring activities to monitor the internal control system and evaluate the results. For example, once established, management can use the baseline, or current state of the internal control system, as criteria in evaluating the internal control system and make changes to reduce the difference between the criteria (what is expected) and condition (what Air Force staff did do instead of what was expected). Also, per OMB Circular No. A-123, an agency may document its assessment of internal control using a variety of information sources, such as management reviews conducted expressly for the purpose of assessing internal control (e.g., AFAA agreed-upon procedures and Internal Control Playbook procedures). Air Force reviews of internal control over processes related to mission- critical assets were inadequate because SAF/FM did not include in the agreed-upon procedures or the Internal Control Playbook tests of design to determine if controls individually and in combination with other controls are capable of achieving an objective and addressing related risks, tests of implementation and operating effectiveness only after a favorable assessment of the design of control, and a baseline that has accurate descriptions of business processes and identifies key internal controls as designed by management to respond to risks. Further, SAF/FM did not document its approach for using results from the AFAA agreed-upon procedures in assessing the Air Force’s internal control over processes related to mission-critical assets because the Air Force did not provide guidance establishing the process and reporting lines of all the sources of information that it considered in preparing its overall Statement of Assurance. Also, SAF/FM did not have a documented process for integrating the results of internal control reviews performed at the business process assessable unit level into the organizational units’ assessment of internal control. Moreover, Air Force did not have guidance describing how often, through which conduit, or when the results from the business process internal control reviews were to be provided to relevant organizational units, or how this information would affect conclusions made in a unit’s respective assurance statement. By not comprehensively evaluating internal control over processes related to mission-critical assets, the Air Force is at increased risk that it may not timely identify internal control deficiencies and may lack reasonable assurance over the effectiveness of internal control over processes accounting for mission-critical assets. In addition, without performing internal control assessments in accordance with requirements or having a formal process to consider the results of the AFAA agreed-upon procedures and the Internal Control Playbook procedures in the organizational unit managers’ assessment process, the Air Force increases the risk that its assessment of internal control and related Statement of Assurance may not appropriately represent the effectiveness of internal control. Air Force senior leaders work to achieve complex and inherently risky objectives across the agency, while managing over $230 billion in mission-critical assets available to carry out its mission. To reduce the risk of not achieving its objectives or efficiently managing its resources, the Air Force needs to implement an ERM capability that is integrated with an effective system of internal control, as outlined in OMB Circular No. A-123 and federal standards for internal control. Although the Air Force has been working to improve its risk management and internal control practices, including remediation of deficiencies in its internal control over financial reporting related to mission-critical assets, it still faces significant challenges. For example, the agency continues to have difficulties with tracking and reporting, with reasonable accuracy, financial information about its mission-critical assets that directly affect its ability to efficiently support the warfighter, achieve its objectives, and accomplish its mission through reliable, useful, and readily available information. Without an effective ERM governance structure, there is an increased risk that the Air Force will not properly identify, assess, and respond to significant entity-level risks. In addition, by not comprehensively implementing and evaluating its internal control system, the Air Force cannot ensure that it is timely identifying and correcting internal control deficiencies or effectively reducing, to an acceptable level, the risk of not achieving its objectives. Further, Air Force management’s assurances on internal control, as reported in the overall agency Statement of Assurance, may not appropriately represent its internal control effectiveness. We are making the following 12 recommendations to the Air Force: The Secretary of the Air Force should develop and implement procedures for an ERM governance structure that includes oversight responsibilities for identifying, assessing, responding to, and reporting on the risks associated with agency material weaknesses from all relevant sources. These procedures should clearly demonstrate that risks associated with material weaknesses are considered by Air Force governance, as a whole, and are mitigated appropriately to achieve goals and objectives. (Recommendation 1) The Secretary of the Air Force should develop policies or procedures for assessing internal control to require (1) clearly delineating who within the Air Force is responsible for evaluating the internal control components and principles, how often they are to perform the evaluation, the level (e.g., entity or transactional) of the evaluation, what objectives are covered in the assessment, to whom to communicate the results if they are relevant to others performing assessments of internal control, and what guidance to follow; (2) documenting management’s determination of whether each component and principle is designed, implemented, and operating effectively; and (3) documenting management’s determination of whether components are operating together in an integrated manner. (Recommendation 2) The Secretary of the Air Force should develop policies or procedures for assessing internal control to require the use of test plans that (1) tie back to specific objectives to be achieved as included in the Business Operations Plan; (2) specify the nature, scope, and timing of procedures to conduct under the OMB Circular No. A-123 assessment process; and (3) reflect a consideration of prior year self-identified control deficiencies and results of internal and external audits. (Recommendation 3) The Secretary of the Air Force should develop policies or procedures for assessing internal control to require SAF/FM to validate (1) the number of organizational units reporting for its overall internal control assessment; (2) how control procedures were tested, what results were achieved, and how conclusions were derived from those results; and (3) whether the results used to compile the current year report are based on current fiscal year’s assessments. (Recommendation 4) The Secretary of the Air Force should develop policies or procedures for assessing internal control to require SAF/FM to assess how waivers affect the current year assessment of internal control, the determination of systemic weaknesses, and the compilation of the Air Force’s overall Statement of Assurance. (Recommendation 5) The Secretary of the Air Force should require that developers of the policy and related guidance associated with designing the procedures for conducting OMB Circular No. A-123 assessments receive recurring training and are appropriately skilled in conducting internal control assessments and are familiar with Standards for Internal Control in the Federal Government. (Recommendation 6) The Secretary of the Air Force should analyze all definitions included in Air Force ERM and internal control assessment policy and related guidance to ensure that all definitions and concepts are defined correctly. (Recommendation 7) The Secretary of the Air Force should require SAF/FM to design recurring training for those who will assess internal control that (1) includes enhancing their skills in evaluating the internal control system and documenting results; (2) reflects all OMB Circular No. A-123 requirements, such as those related to identifying objectives, evaluating deficiencies, and determining material weaknesses; and (3) is provided to all who are responsible for performing internal control assessments. (Recommendation 8) The Secretary of the Air Force should develop policy or procedures consistent with OMB Circular No. A-123 to assess the system of internal control using a risk-based approach. (Recommendation 9) The Secretary of the Air Force should develop procedures to assess internal control over processes related to mission-critical assets, including (1) tests of design that evaluate whether controls are capable of achieving objectives, (2) tests of effectiveness only after a favorable assessment of the design of the control, and (3) a baseline that has accurate descriptions of business processes and identifies key internal controls as designed by management to respond to risks. (Recommendation 10) The Secretary of the Air Force should establish a process and reporting lines of all the sources of information, including reviews performed of internal control processes related to mission-critical assets, that will be considered in the Secretary’s Statement of Assurance. (Recommendation 11) The Secretary of the Air Force should develop procedures to require coordination between business process leads and the Air Force’s unit managers to ensure that mission-critical asset–related internal control deficiencies are considered in the unit managers’ assessments of internal control and related supporting statements of assurance. These procedures should include how, when, and with what frequency the results from the business process internal control reviews should be provided to relevant organizational units for consideration in their respective assurance statements. (Recommendation 12) We provided a draft of this report to the Air Force for review and comment. In written comments, the Air Force concurred with all 12 of our recommendations and cited actions to address them. Air Force’s comments are reproduced in appendix I. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense (Comptroller)/Chief Financial Officer, the Secretary of the Air Force, the Assistant Secretary of the Air Force (Financial Management and Comptroller), and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2989 or kociolekk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, John Sawyer (Assistant Director), Russell Brown, Anthony Clark, Oliver Culley, Eric Essig, Patrick Frey, Jason Kelly, Aaron Ruiz, and Vanessa Taja made key contributions to this report.", "summary": "OMB Circular No. A-123 requires agencies to provide an annual assurance statement that represents the agency head's informed judgment as to the overall adequacy and effectiveness of internal controls related to operations, reporting, and compliance objectives. Although the Air Force is required annually to assess and report on its control effectiveness and to correct known deficiencies, it has been unable to demonstrate basic internal control, as identified in previous audits, that would allow it to report, with reasonable assurance, the reliability of internal controls, including those designed to account for mission-critical assets. This report, developed in connection with fulfilling GAO's mandate to audit the U.S. government's consolidated financial statements, examines the extent to which the Air Force has incorporated ERM into its management practices and designed a process for assessing internal control, including processes related to mission-critical assets. GAO reviewed Air Force policies and procedures and interviewed Air Force officials on their process for fulfilling ERM and internal control assessments. The Air Force's efforts to implement Enterprise Risk Management (ERM) are in the early stages, and accordingly, it has not fully incorporated ERM into its management practices as outlined in Office of Management and Budget (OMB) Circular No. A-123. As a result, the Air Force is not fully managing its challenges and opportunities from an enterprise-wide view. Until it fully incorporates ERM—planned for some time after 2023—the Air Force will continue to leverage its current governance and reporting structures as well as its existing internal control reviews. The Air Force has not designed a comprehensive process for assessing internal control, including processes related to mission-critical assets. GAO found that existing policies and procedures that Air Force staff follow to perform internal control assessments do not accurately capture the requirements of OMB Circular No. A-123. For example, the Air Force does not require (1) an assessment of each internal control element; (2) test plans that specify the nature, scope, and timing of procedures to conduct; and (3) validation that the results of internal control tests are sufficiently clear and complete to explain how units tested control procedures, what results they achieved, and how they derived conclusions from those results. Also, Air Force guidance and training was not adequate for conducting internal control assessments. In addition, GAO found that the Air Force did not design its assessment of internal control to evaluate all key areas that are critical to meeting its mission objectives as part of its annual Statement of Assurance process. Furthermore, GAO found that procedures the Air Force used to review mission-critical assets did not (1) evaluate whether the control design would serve to achieve objectives or address risks; (2) test operating effectiveness after first determining if controls were adequately designed; (3) use process cycle memorandums that accurately reflected the current business process; and (4) evaluate controls it put in place to achieve operational, internal reporting, and compliance objectives. GAO also found that the results of reviews of mission-critical assets are not formally considered in the Air Force's assessment of internal control. Without performing internal control reviews in accordance with requirements, the Air Force increases the risk that its assessment of internal control and related Statement of Assurance may not appropriately represent the effectiveness of internal control, particularly over processes related to its mission-critical assets. GAO is making 12 recommendations to the Air Force, which include improving its risk management practices and internal control assessments. The Air Force agreed with all 12 recommendations and cited actions to address them.", "document_type": "gao"}
{"report": "IPIA requires executive branch agencies to take various steps regarding improper payments in accordance with guidance issued by OMB, including the following: 1. reviewing all programs and activities and identifying those that may be susceptible to significant improper payments; 2. developing improper payment estimates for those programs and activities that agency risk assessments, OMB, or statutes identify as being susceptible to significant improper payments; 3. analyzing the root causes of improper payments and developing corrective actions to reduce them; and 4. reporting on the results of addressing the foregoing requirements. Figure 1 illustrates these steps, as well as the major components of analyzing root causes of improper payments and developing corrective action plans to remediate them. IPIA requires agencies with programs susceptible to significant improper payments to report a description of the causes of the improper payments identified, actions that the agency has planned or taken to correct those causes, and the planned or actual completion dates of those actions. It also requires agencies to report program-specific improper payment reduction targets that OMB has approved. OMB M-18-20 provides guidance to agencies for implementing IPIA requirements, including their responsibilities for preventing and reducing improper payments. The guidance directs agencies that have developed estimates for improper payments to categorize them by root causes, including the percentage of the total estimate for each category. According to the guidance, this level of specificity helps lead to more effective corrective actions and more focused prevention strategies. Table 2 summarizes OMB’s root cause categories. OMB M-18-20 directs agencies with programs deemed susceptible to significant improper payments to implement a corrective action plan that responds to their root causes to prevent and reduce them. As such, OMB directs that an agency must understand the true root cause of its improper payments in order to develop targeted, effective corrective actions, which are proportional to the severity of the associated amount and rate of the root cause. OMB M-18-20 also directs agencies to annually measure the effectiveness and progress of individual corrective actions by assessing results, such as performance and outcomes. In performing such measurements, OMB states that agencies should determine if any existing corrective actions can be intensified or expanded to further reduce improper payments and to identify annual benchmarks for corrective actions that agencies implement over multiple years. Agencies may use these benchmarks to demonstrate progress in implementing the actions or their initial effect on preventing and reducing improper payments. The eight programs we reviewed serve a variety of purposes and are administered by various agencies across the federal government, as discussed below. The Department of Agriculture’s (USDA) Supplemental Nutrition Assistance Program (SNAP) is the largest federally funded nutrition assistance program, providing benefits to about 40 million people in fiscal year 2018. SNAP is intended to help low-income households obtain a more nutritious diet by providing them with benefits to purchase food from authorized retailers nationwide. SNAP recipients receive monthly benefits on an Electronic Benefit Transfer (EBT) card and redeem them for eligible food at authorized food stores. The Food and Nutrition Act of 2008 established SNAP as a federally funded, state-administered program. States, following federal guidelines, are responsible for program administration. States determine applicant eligibility, calculate benefit amounts, issue EBT cards to recipients, and investigate possible recipient program violations. USDA’s Food and Nutrition Service (FNS) pays the full cost of SNAP benefits and shares 50 percent of administrative costs with the states. As part of oversight responsibilities, FNS develops program regulations and monitors states to ensure that they comply with program rules. FNS is also directly responsible for authorizing and monitoring retail food stores where recipients may purchase food. In accordance with IPIA, USDA has annually reported an improper payment estimate for SNAP since fiscal year 2004. In its fiscal year 2019 AFR, USDA reported an improper payment estimate of approximately $4 billion, or 6.8 percent of SNAP outlays of $59.1 billion. The Department of Education’s (Education) William D. Ford Federal Direct Loan (Direct Loan) program authorizes Education to make loans, through participating schools, to eligible undergraduate and graduate students and their parents. The Direct Loan program comprises four types of loans: Subsidized Stafford, Unsubsidized Stafford, PLUS, and Consolidation loans. Evidence of financial necessity is required for an undergraduate student to receive a Subsidized Stafford loan; however, borrowers at all income levels are eligible for the other three types. Education originates the loans and disburses them through each borrower’s school. Once a loan is disbursed, Education assigns a servicer responsible for communicating with the borrower, providing information about repayment, and processing payments from the borrower. Education first reported an improper payment estimate for the Direct Loan program in fiscal year 2013. In its fiscal year 2019 AFR, Education reported an improper payment estimate of approximately $483 million, or 0.5 percent of Direct Loan program outlays of $92.9 billion. Education’s Pell Grant program—the single largest source of grant aid for postsecondary education—awards federally funded grants to low-income undergraduate and certain post-baccalaureate students who are enrolled in a degree or certificate program and have a federally defined financial need. Students are eligible to receive Pell Grants for no more than 12 semesters (or the equivalent). To qualify, an applicant must, in addition to satisfying other requirements, demonstrate financial need and not have obtained a bachelor’s degree or a first professional degree. Grant amounts depend on the student’s expected family contribution, the cost of attendance (as determined by the institution), the student’s enrollment status (full-time or part-time), and whether the student attends for a full academic year or less. Education first reported an improper payment estimate for the Pell Grant program in fiscal year 2004. In its fiscal year 2019 AFR, Education reported an improper payment estimate of approximately $646 million, or 2.2 percent of Pell Grant program outlays of $28.9 billion. The Department of Health and Human Services’ (HHS) Children’s Health Insurance Program (CHIP) expands health coverage to uninsured children who are ineligible for Medicaid but cannot afford private coverage. The states and the federal government jointly fund CHIP benefit payments and administrative expenses. HHS’s Centers for Medicare & Medicaid Services (CMS) oversees the program; however, each state administers the program and sets its own guidelines regarding eligibility and services according to federal guidelines. HHS first reported an improper payment estimate for CHIP (based on one-third of the states) in fiscal year 2008. In its fiscal year 2019 AFR, HHS reported an improper payment estimate of approximately $2.7 billion, or 15.8 percent of CHIP outlays of $17.3 billion. The Earned Income Tax Credit (EITC) administered by the Department of the Treasury (Treasury) is a credit that offsets taxes owed by eligible taxpayers, and because the credit is refundable, EITC recipients need not owe taxes to receive a benefit. If the taxpayer’s credit exceeds the amount of taxes due, the Internal Revenue Service (IRS) issues a refund of the excess to the taxpayer. To claim the EITC, the taxpayer must work and have earnings that do not exceed the phaseout income of the credit. Additional eligibility rules apply to any children that a taxpayer claims for calculating the credit. Among other criteria, a qualifying child must meet certain age, relationship, and residency requirements. Treasury first reported an improper payment estimate for EITC in fiscal year 2003. In its fiscal year 2019 AFR, Treasury reported an improper payment estimate of approximately $17.4 billion, or 25.3 percent of EITC outlays of $68.7 billion. Through its Prosthetic and Sensory Aids Service (PSAS), the Department of Veterans Affairs’ (VA) Veterans Health Administration (VHA) provides prosthetics to veterans who have experienced the loss or permanent impairment of a body part or function. The items VA provides include those worn by the veteran, such as an artificial limb or hearing aid; those that improve accessibility, such as ramps and vehicle modifications; and devices surgically placed in the veteran, such as hips and pacemakers. In general, veterans enrolled in the VA health care system with a medical need for a prosthetic service or item are eligible; however, additional eligibility criteria for certain services or items may apply. PSAS officials in VA’s central office provide overall administration of VA’s provision of prosthetic items, including allocating funding among various networks, monitoring spending, and establishing and monitoring mechanisms to evaluate the agency’s performance. PSAS processes prescriptions and provides the prescribed items to individual veterans. PSAS government credit card holders, typically at VA medical centers, perform administrative actions—such as obtaining additional information from the prescribing clinician, obtaining price quotes from contractors, and creating purchase orders—to process prescriptions. PSAS also has staff who provide clinical services to veterans, such as evaluating prosthetic needs and designing and fitting artificial limbs. VA first reported an improper payment estimate for PSAS in fiscal year 2017. In its fiscal year 2019 AFR, VA reported an improper payment estimate of approximately $60 million, or 2.1 percent of PSAS outlays of $2.9 billion. The Social Security Administration’s (SSA) Old Age, Survivors, and Disability Insurance program (OASDI), collectively referred to as Social Security, provides cash benefits to eligible U.S. citizens and residents. OASDI is financed largely on a pay-as-you-go basis. Specifically, OASDI payroll taxes, paid each year by current workers, are primarily used to pay benefits provided during that year to current beneficiaries. OASDI consists of two separate insurance programs that SSA administers under the Social Security Act. Old Age and Survivors Insurance (OASI) provides benefits to retired workers, their families, and survivors of deceased workers. The monthly benefit amount depends on a worker’s earnings history and the age at which he or she chooses to begin receiving benefits, along with other factors. Benefits are paid to workers who meet requirements for the time they have worked in covered employment—that is, jobs through which they have paid Social Security taxes. Disability Insurance (DI) provides cash benefits to working-age adults who are unable to work because of long-term disability. SSA generally considers individuals to have a disability if (1) they cannot perform work that they did before and cannot adjust to other work because of their medical condition(s) and (2) their disability has lasted or is expected to last at least 1 year or is expected to result in death. Further, individuals must have worked and paid into the program for a minimum period of time to qualify for benefits. To ensure that only beneficiaries who remain disabled continue to receive benefits, SSA is required to conduct periodic continuing disability reviews in certain circumstances. SSA first reported an improper payment estimate for OASDI in fiscal year 2004. In its fiscal year 2019 AFR, SSA reported an improper payment estimate of approximately $2.7 billion, or 0.3 percent of OASDI program outlays of $948 billion. SSA’s Supplemental Security Income (SSI) is a federal income supplement program funded by general tax revenues (not Social Security taxes). The program provides payments to low-income aged, blind, and disabled persons—both adults and children—who also meet financial eligibility requirements. For adults, a disability is defined as the inability to engage in any substantial gainful activity because of any medically determinable physical or mental impairment(s) that can be expected to result in death or has lasted or can be expected to last for a continuous period of not less than 12 months. To ensure that only recipients who remain disabled continue to receive benefits, SSA is required to conduct periodic continuing disability reviews in certain circumstances. To be eligible to receive monthly SSI payments, the adult individual’s (or married couple’s) or child’s (and parent’s) monthly countable income has to be less than the monthly federal SSI benefit amount. The amount of the monthly SSI payment is then determined based on the countable income. In most cases, countable income received in the current month affects the SSI payment amount 2 months later. Furthermore, countable resources—such as financial institution accounts—must not exceed the maximum allowable threshold. While recipients are required to report changes in their income and financial resources, SSA also conducts periodic redeterminations to verify that recipients are still eligible for SSI. SSA first reported an improper payment estimate for SSI in fiscal year 2004. In its fiscal year 2019 AFR, SSA reported an improper payment estimate of approximately $5.5 billion, or 9.7 percent of SSI program outlays of $56.9 billion. We found that five out of six agencies—USDA, Education, HHS, VA, and SSA—used the results of their improper payment estimation methodologies as the basis for identifying the root causes of improper payments for the selected programs we reviewed. Specifically, the agencies generally used a two-step process to identify root causes of improper payments. First, the agencies reviewed a sample of payments to identify which payments were improper and to establish an improper payment rate. Second, the agencies analyzed the improper payment results to determine the causes of error. Further details on each agency’s process are provided below. USDA: According to USDA’s fiscal year 2018 AFR, FNS used SNAP’s Quality Control System to identify improper payments and determine improper payment rates for fiscal year 2018. According to agency officials, SNAP improper payment root causes occur at the state level. According to agency officials, as required by the Food and Nutrition Act of 2008 and subsequent program regulations, FNS requires states to conduct root cause analyses and develop corrective action plans because of the unique circumstances in each state owing to flexibilities under statute and regulations. SNAP’s Quality Control system uses a two-tier approach to report improper payments. In the first tier, each month, state agencies follow federal sampling requirements to select samples of households that participated in SNAP in their states and conduct quality control reviews to determine whether each selected household was eligible and received the right amount of benefits. In the second tier of the process, Federal SNAP staff select a subsample of the state data for review to confirm the validity of the states’ findings. Federal SNAP staff use that subsample data to aggregate the root cause information at a nationwide level in order to categorize the data into the OMB root cause categories for fiscal year 2018 reporting. Education: According to Education’s fiscal year 2018 AFR, Education conducted a risk-based, nonstatistical sample and estimation methodology, which OMB approved, to estimate Pell Grant and Direct Loan improper payment rates for fiscal year 2018 reporting. As part of this estimation process, Education analyzed identified improper payments to determine improper payment root causes. HHS: According to HHS’s fiscal year 2018 AFR, HHS estimated the CHIP improper payment rate for fiscal year 2018 reporting through the Payment Error Rate Measurement (PERM) program. CHIP improper payment root causes were identified at both the agency and state levels. Specifically, to determine improper payment root causes at the agency level, HHS analyzed the issues identified during the PERM review and identified primary drivers of the national PERM rate for CHIP. HHS also provided improper payment results to each state and required them to conduct more in-depth state-level root cause analyses as part of developing their corrective action plans. VA: According to VA’s fiscal year 2018 AFR, VA conducted a statistical sample and estimation methodology to estimate the PSAS improper payment rate for fiscal year 2018 reporting. VA then analyzed the improper payments identified during testing to determine improper payment root causes. SSA: According to SSA’s fiscal year 2018 AFR, SSA conducts stewardship reviews each fiscal year to estimate the improper payment rates for OASDI and SSI. Although SSA considers the stewardship review data sufficient to provide statistically reliable data on the overall payment accuracy of OASDI and SSI, SSA considered deficiency data from the most recent 5 years of stewardship reviews to determine improper payment root causes for each program for its fiscal year 2018 reporting. Treasury identified the root causes of EITC improper payments for fiscal year 2018 reporting based on the most recent detailed 3-year EITC compliance study IRS conducted, using data from tax years 2006 through 2008. IRS officials acknowledged that using older data creates additional potential for error; however, they stated that IRS is only able to conduct in-depth compliance studies on major refundable income tax credits, including EITC, on a rotating basis. IRS also conducted in-depth EITC compliance studies for tax years 1997 and 1999. These studies and IRS’s 2006 through 2008 compliance study, identified income misreporting and qualifying child errors as the main sources of errors. Therefore, agency officials indicated that Treasury is comfortable with using the 2006 through 2008 data as the basis for determining the root causes of fiscal year 2018 EITC improper payments. However, Treasury has reported changes to the tax environment since 2008, including legislative revisions that may have affected taxpayer compliance behavior. Specifically, EITC-related changes include expanding the credit to a third child, establishing new criteria for claiming a qualifying child, and amending the “age test” for qualifying children, among others. Furthermore, the 2006 through 2008 compliance study did not take into account the Protecting Americans from Tax Hikes Act of 2015 program integrity provisions that required tax filers to provide Form W-2 payer information to IRS for verification earlier than in previous tax years. Federal internal control standards state that management should use quality information to achieve the entity’s objectives. As part of these standards, management obtains relevant data from reliable internal and external sources in a timely manner and uses quality information to make informed decisions and evaluate the entity’s performance in achieving objectives and addressing risks. Quality information is appropriate, current, complete, accurate, accessible, and provided on a timely basis. Although a specific delivery date has not been set, agency officials stated that IRS plans to conduct another in-depth EITC compliance analysis within the next 2 years. We agree with Treasury’s plan to conduct another EITC compliance analysis using more timely data. However, until Treasury conducts an EITC improper payment root cause analysis using more timely data, it will be uncertain whether identified root causes are sufficiently relevant to inform decisions and evaluate risks. Specifically, continued use of outdated information to evaluate EITC improper payments increases the risk that Treasury may not be identifying these payments’ true root causes and therefore will lack quality information needed to develop appropriate corrective actions and reduce them. Four out of six agencies—Education, HHS, VA, and SSA—developed corrective actions that correspond to identified root causes of improper payments for the selected programs we reviewed, in accordance with OMB guidance. Specifically, we found that Education and VA developed corrective actions corresponding to each root cause of improper payments identified for fiscal year 2018 in Education’s Direct Loan and Pell Grant programs and VA’s PSAS, respectively. In addition, HHS stated that it developed corrective actions that corresponded to the root causes it determined to be significant to CHIP improper payments for fiscal year 2018, prioritizing large dollar over smaller dollar value root cause categories. Corrective action plans for CHIP improper payments were developed at both the agency and state levels. According to agency officials, CMS helped individual states develop and implement state-specific PERM corrective action plans to address the errors identified in each state. In addition, because each state’s errors do not necessarily represent errors that are the main drivers of the national PERM rate, CMS developed agency-level corrective action plans focused on those drivers, which typically occurred across multiple states. We also found that SSA’s corrective actions corresponded to root causes of improper payments identified in OASDI and SSI for fiscal year 2018. However, SSA did not develop corrective actions corresponding to three of the six major root causes it identified for OASDI improper payments based on its stewardship review findings. Agency officials explained that SSA’s corrective action development process was decentralized among the different SSA components, and therefore, there was no formalized process for components to develop corrective actions for all identified root causes. SSA has since developed a new standardized improper payment strategy and updated procedures to implement the strategy for fiscal year 2020. Although the scope of our review focused on processes in place for fiscal year 2018, we found that the updated procedures, if effectively implemented, will address our concerns because they include control activities designed to help ensure that corrective actions that SSA develops and implements correspond to the identified root causes of improper payments, as directed by OMB guidance. Specifically, the updated procedures direct SSA components to identify root causes of improper payments and develop mitigation strategies for each; conduct cost-benefit analyses for such strategies; and after considering these analyses, determine and prioritize necessary corrective actions. In contrast to HHS, which developed both agency- and state-level corrective actions for its state-administered CHIP, USDA did not develop agency-level corrective actions corresponding to the root causes of SNAP improper payments. USDA’s IPIA corrective action plan guidance directs its components, including FNS, to develop corrective actions that correspond to the identified root causes of improper payments for programs that are susceptible to significant improper payments. Instead of developing agency-level SNAP corrective actions, FNS requires the states to develop state-level corrective actions. Additionally, FNS provided technical assistance and support to the individual states to help them improve payment accuracy. As part of this assistance, agency officials stated that FNS regional offices provided routine formal training and guidance to the states and conducted site visits. According to agency officials, FNS did not develop agency-level corrective actions corresponding to the root causes of SNAP improper payments because FNS requires the states to develop individual state- level corrective actions. Additionally, because of varying root causes and the uniqueness of issues identified among the states, agency officials believe that state corrective actions may not easily aggregate to the state level. However, FNS’s procedures did not include a process to analyze state-level root causes to identify similarities and develop agency-level corrective actions, if warranted, to help address them. According to agency officials, FNS has made significant improvements in the last few years regarding its controls over SNAP. The officials said that FNS has also implemented major changes in oversight in the last few fiscal years to address previously identified deficiencies among the states. While these changes may be valuable in improving agency oversight and states may have unique circumstances that could lead to varying state-identified root causes of improper payments, FNS is ultimately responsible for preventing and reducing improper payments within SNAP. OMB guidance directs agencies to develop and implement appropriate corrective actions that respond to the root causes of improper payments to prevent and reduce them. OMB guidance also directs agencies to ensure that managers; programs; and, where applicable, states are held accountable for reducing improper payments. Additionally, federal internal control standards state that management should establish and operate activities to monitor the internal control system and evaluate the results and remediate identified internal control deficiencies on a timely basis. As part of these standards, management retains responsibility for monitoring the effectiveness of internal control over the assigned processes that external parties, such as state agencies, perform. Without considering similarities of root causes of SNAP improper payments among the states, USDA will be uncertain whether developing and implementing agency-level corrective actions (in addition to state-level actions) would also help to effectively reduce them. Instead of developing corrective actions corresponding to the identified root causes of EITC improper payments for fiscal year 2018, Treasury addressed improper payments through IRS’s compliance programs and through outreach and education efforts to taxpayers and preparers. According to agency officials, although some of the outreach efforts are indirectly related to root causes identified, it is difficult to link those efforts to the reduction of errors that result from being unable to authenticate eligibility—which Treasury considers the biggest issue in the EITC program—because of the complexity of statutory eligibility requirements. Although Treasury uses information from SSA and HHS to help IRS verify residency and relationship information for parents and children, Treasury’s strategy for addressing the root causes of EITC improper payments does not include continuing efforts to identify and reach out to additional agencies to (1) determine how they verify information for certain eligibility-based programs and whether they use strategies that Treasury could adopt or (2) identify other potential data sources that could be used to verify EITC information or confirm that other data sources do not exist. According to agency officials, such inquiries are not included because the eligibility requirements for EITC are not always the same as requirements for other government programs. Additionally, Treasury’s fiscal year 2018 AFR states that because of the nature of EITC, corrective actions implemented by IRS alone will not significantly reduce EITC improper payments. For example, according to Treasury officials, legislative changes are needed to help address certain EITC improper payments. While Treasury has made certain legislative proposals related to providing IRS greater flexibility to address correctable errors and increasing oversight of paid tax return preparers, it has not made proposals to help address EITC eligibility criteria issues. Additionally, Treasury’s strategy does not include identifying and proposing legislative changes needed to help reduce EITC improper payments related to these or other issues, such as those related to the inability to authenticate taxpayer eligibility discussed above. OMB guidance directs agencies to develop and implement appropriate corrective actions that respond to the root causes of improper payments to prevent and reduce them. Further, federal internal control standards state that management should use quality information to achieve the entity’s objectives. As part of these standards, management designs a process that uses the entity’s objectives and related risks to identify the information requirements needed to achieve the objectives and address the risks and obtains relevant data from reliable internal and external sources in a timely manner based on the identified information requirements. While we recognize the unique eligibility requirements for EITC, until Treasury coordinates with other agencies to identify potential strategies or data sources that may help in determining eligibility, it will be uncertain whether Treasury can leverage additional sources to help verify data. Additionally, without identifying and proposing legislative changes to help resolve such issues, Treasury will be at risk of continuing to be unable to significantly reduce EITC improper payments. All six agencies responsible for the programs we reviewed communicated with internal agency stakeholders regarding their improper payment corrective action plan information, in accordance with OMB guidance and federal internal control standards. However, as shown in table 3, three of the four agencies—Education, HHS, and SSA—that developed corrective actions corresponding to the identified root causes either did not establish planned completion dates, monitor the progress, or measure the effectiveness of their corrective actions. In fact, we found that VA was the only agency that measured the effectiveness of each corrective action for the selected program (PSAS) that we reviewed. As previously discussed, USDA and Treasury did not develop agency corrective actions corresponding to the identified root causes of improper payments for their selected programs and therefore did not establish planned related completion dates, monitor progress, or measure the effectiveness of such corrective actions. All six agencies we reviewed communicated information regarding the selected programs’ corrective action plans to internal stakeholders, consistent with OMB guidance and federal internal control standards. OMB M-18-20 directs agencies to ensure that managers, accountable officers (including the agency head), and program officials are held accountable for reducing improper payments. Additionally, federal internal control standards state that management should internally communicate the necessary quality information to achieve the entity’s objectives. As part of these standards, management communicates quality information down, across, up, and around reporting lines to all levels of the entity. We found that the six agencies communicated information, at least annually, to such internal stakeholders, including the relevant agency head, chief financial officer (CFO), and program managers. For example, some selected agencies—Education, HHS, VA, and SSA—provided briefings to the agency head and the CFO’s office regarding the status of the selected program’s improper payment corrective action activities during fiscal year 2019 for the corrective actions reported for fiscal year 2018. USDA and Treasury required their components to annually submit deliverables to the office of the CFO and coordinate accordingly with the Office of the Secretary as part of their fiscal year 2018 AFR reporting process. We found that two of the six agencies we reviewed—Education and VA— established planned completion dates for the selected programs’ corrective actions. Two agencies—HHS and SSA—did not consistently establish planned completion dates for all the selected programs’ corrective actions, as required by IPIA. Two agencies—USDA and Treasury—did not develop agency corrective actions corresponding to the identified root causes of improper payments for their selected programs and therefore did not establish planned completion dates for such corrective actions. Further details on each agency’s process are provided below. USDA: As previously discussed, FNS did not develop corrective actions at the agency level to address SNAP’s root causes of improper payments and, as a result, did not have planned completion dates for such corrective actions. However, in the event that FNS develops agency-level corrective actions, USDA’s IPIA corrective action plan guidance includes a directive for each corrective action to have an estimated completion date. Education: Education established planned completion dates for all Direct Loan and Pell Grant corrective actions that were not legislative proposals. For example, in fiscal year 2018, Education did not report a planned completion date for Federal Student Aid’s (FSA) corrective action related to proposed legislative changes, as the timeline for the legislative process is subject to external factors outside of Education’s control. HHS: HHS did not consistently establish planned completion dates for agency-level CHIP corrective actions. According to agency officials, most agency-level CHIP corrective actions are unlikely to have completion dates because the work is ongoing. We agree with HHS’s determination that establishing completion dates for ongoing corrective actions was not relevant. HHS provided a spreadsheet of CHIP’s corrective actions, which included a column of target completion dates. However, this column was not consistently filled out for actions that were not considered either ongoing or voluntary state processes. HHS officials stated that although HHS has a process for its improper payment corrective action plans, this process is not documented in formal policies and procedures. Instead, HHS uses OMB guidance as its policies and procedures. Lack of formally documented policies and procedures may have contributed to the inconsistencies in HHS establishing planned completion dates for agency-level CHIP corrective actions. Treasury: As previously discussed, instead of developing corrective actions to address root causes of EITC improper payments, Treasury addressed improper payments through IRS’s compliance programs and through outreach and education efforts to taxpayers and preparers. According to agency officials, Treasury did not establish planned completion dates for its compliance programs and outreach efforts because these activities were ongoing in nature and completed every year as part of IRS operations. We agree with Treasury’s determination that establishing completion dates for EITC ongoing compliance activities was not relevant. In the event that Treasury develops corrective actions for EITC improper payments, Treasury’s corrective action plan guidance includes a directive for each corrective action to have an estimated completion date. VA: VA established relevant planned completion dates for each PSAS corrective action. In addition, each task associated with each corrective action had a planned completion date. SSA: SSA did not consistently establish relevant completion dates for each OASDI and SSI corrective action. For example, SSA’s corrective action plans included sections for “target completion.” However, based on our review, these sections were not filled out consistently. According to agency officials, the process for developing and implementing its corrective actions was inconsistent because of SSA’s decentralized corrective action plan process. As previously discussed, SSA developed a new standardized improper payment strategy that if effectively implemented will address these concerns. Specifically, SSA’s procedures to implement this strategy include control activities designed to help ensure that the agency establishes planned completion dates for each corrective action, as required by IPIA. IPIA requires agencies to report on the planned or actual completion date of each action taken to address root causes of improper payments. Federal internal control standards state that management should design control activities to achieve objectives and respond to risks and implement control activities through policies. Further, federal internal control standards state that management should remediate identified internal control deficiencies on a timely basis. As part of these standards, management monitors the status of remediation efforts so that they are completed on a timely basis. Additionally, federal internal control standards state that management should implement its control activities through policies. Without documented policies and procedures for its improper payment corrective action plan process, including the establishment of planned completion dates, HHS lacks assurance that corrective action plan–related activities will be performed consistently. Additionally, without planned completion dates, HHS cannot demonstrate that it is effectively implementing and completing corrective actions timely and therefore cannot ensure that they will help reduce improper payments. Three of the four agencies—Education, HHS, and VA—that developed corrective actions corresponding to the identified root causes monitored the progress of the selected programs’ corrective actions, in accordance with OMB guidance. However, HHS’s process was not documented in policies and procedures. SSA did not monitor the progress for all relevant OASDI and SSI corrective actions but has since implemented policies and procedures to monitor such progress. USDA did not develop corrective actions at the agency level that corresponded to the identified root causes of improper payments for SNAP and therefore did not monitor the progress of such corrective actions. In addition, USDA’s corrective action plan guidance does not direct the agency to monitor the progress of its corrective actions. Although Treasury did not have corrective actions that corresponded to the root cause of improper payments, it did monitor the progress of its compliance and outreach efforts that are intended to help reduce EITC improper payments. Further details on each agency’s process are provided below. USDA: As previously discussed, FNS did not develop corrective actions at the agency level to address SNAP’s root causes of improper payments and, as a result, did not monitor the progress of such corrective actions. In addition, USDA’s IPIA corrective action plan guidance does not direct the agency to monitor the progress of its corrective actions. Without agency-level corrective actions to address the root causes of SNAP improper payments and a documented process to monitor the progress of implementing such agency-level corrective actions, USDA may miss opportunities to reduce SNAP improper payments. Education: Education monitored the progress of implementing each Direct Loan and Pell Grant corrective action. We found that Education maintained a spreadsheet to track the implementation status of each corrective action annually. Specifically, the status of each corrective action was updated to either “complete” or “open” for the annually recurring and long-term, multiyear corrective actions. The actions marked as “complete” had actual completion dates. Actions that Education considered ongoing, such as needed updates to help clarify verification requirements to the “Question and Answer” section of FSA’s website, were updated as “not applicable.” HHS: HHS monitored the progress of implementing each of its agency-level CHIP corrective actions. Specifically, HHS tracked the progress of implementing the corrective actions in a spreadsheet that included status updates for each agency-level corrective action. Agency officials stated that this information was updated approximately two to three times each fiscal year through an online interface; however, this process was not documented in policies and procedures. Without a properly documented process and related control activities, HHS is at increased risk that it may not consistently monitor the progress of CHIP corrective actions and has less assurance that such actions are implemented and completed timely. Treasury: Treasury did not develop corrective actions that corresponded to the root causes of EITC improper payments and, as a result, did not monitor the progress of such corrective actions. However, Treasury did monitor its compliance programs and outreach efforts that are intended to help reduce EITC improper payments during fiscal year 2018. VA: VA monitored the progress of implementing each PSAS corrective action. Specifically, we found that VA monitored the progress for each corrective action each month by calculating a completion percentage based on the status of tasks associated with each corrective action. SSA: SSA did not monitor the progress of implementing each OASDI and SSI corrective action. According to agency officials, the monitoring of corrective actions was inconsistent and evaluation of corrective actions was limited because of SSA’s decentralized corrective action plan process. As previously discussed, SSA developed a new standardized improper payment strategy that if effectively implemented will address these concerns. Specifically, SSA’s procedures to implement this strategy include control activities designed to help ensure that the agency monitors the progress of its corrective actions, as directed by OMB guidance. OMB guidance directs agencies to measure the progress of each individual corrective action annually. Federal internal control standards state that management should establish and operate activities to monitor the internal control system and evaluate the results and remediate identified internal control deficiencies on a timely basis. As part of these standards, management monitors the status of remediation efforts so that they are completed on a timely basis. Additionally, federal internal control standards state that management should implement its control activities through policies. Without monitoring the progress of its corrective actions, USDA cannot demonstrate that it is effectively implementing and completing its corrective actions timely and therefore cannot ensure that they will contribute to a reduction in improper payments. Further, unless HHS documents its process in policies and procedures, it will lack assurance that the progress of its corrective actions is monitored consistently and that such actions are implemented and completed timely. We found that one out of six agencies we reviewed—VA—measured the effectiveness of the selected programs’ corrective actions, including the establishment of reduction targets in accordance with OMB guidance. Education, HHS, and SSA did not measure the effectiveness of their corrective actions for the selected programs. In addition, USDA and Treasury did not develop agency corrective actions corresponding to the identified root causes of improper payments for their selected programs and therefore did not measure the effectiveness of such corrective actions. Further details on each agency’s process are provided below. USDA: As previously discussed, FNS did not develop agency-level corrective actions to address root causes of SNAP improper payments. Instead, FNS provided technical assistance and support to the individual states. According to agency officials, FNS cannot link each technical assistance initiative it provides to the states to the effect these efforts have on reducing payment integrity errors, as the technical assistance provided to the states can vary significantly. Additionally, USDA’s IPIA corrective action plan guidance did not include direction for the agency to measure the effectiveness of its corrective actions. Without agency-level corrective actions to address the root causes of SNAP improper payments and a documented process to measure the effect that agency actions have on improper payments, USDA will be unable to demonstrate whether such actions are effective in reducing improper payments and may risk continuing ineffective actions. In addition, as permitted by OMB, USDA did not establish a reduction target for SNAP improper payments because it lacked a sufficient baseline to accurately project future improper payment rates. USDA plans to reestablish reduction targets for fiscal year 2021 reporting. Education: Education’s policies and procedures state that to measure the effectiveness of the corrective actions, FSA solicits input from the corrective action owner, including, among other items, whether measuring and monitoring of the effectiveness of the corrective action has been established and a description of anecdotal evidence available to confirm the effectiveness of the corrective action. However, based on the procedures, it is unclear how the corrective action owners will conduct this analysis to demonstrate effectiveness. Education provided an example of communication to a corrective action owner requesting, among other items, that the corrective action owner (1) confirm that existing actions are focused on the true root causes of the improper payments and are actually reducing improper payments and (2) verify that existing corrective actions are achieving the intended purposes and results. Education officials informed us that although these items were discussed in stakeholder meetings, FSA was unable and did not attempt to quantify the direct effect of any one corrective action on the improper payment estimates. Education’s fiscal year 2018 AFR states that FSA does not attempt to quantify the reduction of the improper payment estimates in terms of percentage or amount due to Pell Grant and Direct Loan corrective actions. It further states that quantifying of results is not feasible because Education uses a nonstatistical alternative estimation methodology. However, according to Education’s fiscal year 2019 AFR, Education implemented a statistical estimation methodology for the fiscal year 2019 estimates. Education believes that the new methodology will allow FSA to better measure the effectiveness of corrective actions over time as FSA collects a baseline of statistically valid improper payment estimates. According to agency officials, FSA is currently refining its process for measuring the effectiveness of corrective actions based on its new statistical estimation methodology. However, until Education revises and documents its process to include measuring the direct effect that its Pell Grant and Direct Loan corrective actions have on improper payments, it will be unable to demonstrate whether the corrective actions are effective in reducing the associated improper payments and may risk continuing ineffective actions. As part of its overall payment integrity reporting in fiscal year 2018, Education established program-wide reduction targets for Pell Grant and Direct Loan. However, according to agency officials, because it used an OMB-approved nonstatistical methodology, Education’s confidence in using these results to establish reduction targets for the upcoming fiscal year was limited. Specifically, Education’s fiscal year 2018 AFR states that imprecision and volatility in the improper payment estimates continue to limit its ability to establish accurate out-year reduction targets. Therefore, for fiscal years 2016 through 2018, Education set the upcoming fiscal year reduction targets to match the current fiscal year reported improper payment rate for each program. According to agency officials, Education plans to consider the feasibility of setting meaningful reduction targets moving forward with its new statistical methodology. HHS: HHS did not measure the effectiveness of its corrective actions for CHIP improper payments. In addition, as discussed above, HHS does not have formal documented policies and procedures for its improper payment corrective action plan process. According to agency officials, establishing a one-to-one relationship between specific corrective actions and resulting changes in the improper payment rates is difficult because of the complexity of factors involved that lead to them. However, until HHS develops and implements a documented process to measure the effect that CHIP corrective actions have on improper payments, it will be unable to demonstrate whether the corrective actions are effective in reducing the associated improper payments and may risk continuing ineffective actions. As permitted by OMB’s implementing guidance, HHS did not establish a program-wide reduction target for CHIP improper payments for fiscal years 2019 or 2020, and does not anticipate setting one for 2021 because it lacks a sufficient baseline to accurately project future improper payment rates. According to agency officials, HHS plans to establish a CHIP reduction target for fiscal year 2022 reporting. Treasury: Treasury did not develop specific corrective actions to address root causes of EITC improper payments, so it could not measure the effectiveness of its corrective actions. Agency officials recognized that the current actions on their own will be unable to significantly reduce the amount of EITC improper payments. As approved by OMB, Treasury did not establish a program-wide reduction target for EITC improper payments for fiscal year 2018 reporting. However, Treasury set a reduction target for EITC improper payments in its fiscal year 2019 AFR, per OMB guidance. VA: VA has documented procedures in place to measure the effectiveness of its corrective actions for PSAS improper payments. As part of this process, VA set reduction targets and timelines for reducing the errors associated with each corrective action. VA maintained a timeline spreadsheet showing the corrective action reduction targets by year and the percentage of improper payments it expects to be reduced once each corrective action is fully implemented. VA updated the spreadsheet at the end of fiscal year 2019 with the current results of the effectiveness measure for corrective actions reported in fiscal year 2018. In addition, VA also set a program-wide reduction target for PSAS improper payments. SSA: SSA did not measure the effectiveness of its corrective actions for OASDI and SSI improper payments. According to agency officials, SSA did not have procedures to collect the necessary data and therefore was unable to measure the effectiveness of its corrective actions. SSA’s procedures for its new standardized improper payment strategy (discussed above) direct responsible components to define the metrics and information necessary to evaluate the corrective actions and to determine if the actions are effectively reducing improper payments. However, it is still unclear which metrics will be used to determine the effect that OASDI and SSI corrective actions have on the corresponding root causes to demonstrate effectiveness. Until SSA develops and implements a documented process to measure the effect that the OASDI and SSI corrective actions have on improper payments, it will be unable to demonstrate whether the corrective actions are effective in reducing the associated improper payments and may risk continuing ineffective actions. As part of its overall payment integrity reporting in fiscal year 2018, SSA established program-wide reduction targets for both programs. However, some of SSA’s reduction targets have remained constant since fiscal year 2004 reporting. Agency officials stated that although SSA believes OASDI’s payment accuracy rate is exceptionally high, if SSA’s mitigation strategies help decrease improper payments, it would consider changing the reduction target. For SSI, agency officials stated that SSA believes that SSI’s program complexity and reliance on self-reporting have made meeting the current accuracy goal challenging. Agency officials further stated that if planned mitigation strategies help decrease improper payments, SSA would consider changing the SSI reduction target. OMB guidance directs agencies to measure the effectiveness of each individual corrective action annually. Agencies may measure the effectiveness of corrective actions by assessing the results of actions taken to address the root causes, such as the performance and outcomes of these processes. In addition, OMB guidance states that for long-term, multiyear corrective actions, agencies should identify annual benchmarks used to demonstrate the initial effect on improper payment prevention and reduction. For corrective actions already in place, agencies should be able to describe how they evaluate these actions’ effectiveness and the results. Federal internal control standards state that management should establish and operate activities to monitor the internal control system and evaluate the results. As part of these standards, management performs ongoing monitoring of the design and operating effectiveness of the internal control system as part of the normal course of operations. Additionally, federal internal control standards state that management should implement its control activities through policies. Unless USDA, Education, HHS, and SSA develop and implement a process that clearly links corrective actions to effectively addressing improper payments, they will be uncertain whether the actions are actually reducing improper payments and the agencies may risk continuing ineffective actions. Further, unless these processes are documented in policies and procedures, agencies will lack assurance that the effectiveness of their corrective actions is measured consistently. Developing corrective action plans that respond to identified root causes of improper payments is a critical component in government-wide efforts to reduce improper payments. Agency processes to monitor the progress and measure the effectiveness of such plans are also essential to evaluating their efforts to address improper payments. However, certain agencies have not effectively taken these steps for the selected programs we reviewed. For example, USDA and Treasury have not developed agency-wide corrective actions that correspond to the identified root causes of improper payments in their SNAP and EITC programs, respectively, that would better position these agencies to reduce and prevent them. Also, HHS lacks important information to monitor its efforts to address CHIP improper payments because it does not consistently establish planned completion dates for agency-level corrective actions. Additionally, USDA, Education, HHS, and SSA do not have sufficient processes in place to measure the effectiveness of corrective actions to address improper payments for the selected programs we reviewed. Unless agencies develop corrective action plans that correspond to the root causes of improper payments and implement processes to effectively monitor progress and measure their effectiveness, their ability to ensure that their actions will reduce improper payments will be limited. We are making the following seven recommendations—one each to Education, HHS, and SSA and two each to USDA and Treasury. The Administrator of FNS should develop and implement a process, documented in policies and procedures, to analyze SNAP state-level root causes to identify potential similarities among the states and develop and implement SNAP agency-level corrective actions, if appropriate, to help address them. (Recommendation 1) The Secretary of Agriculture should revise USDA’s procedures to include processes for monitoring the progress and measuring the effectiveness of improper payment corrective actions. The process for measuring the effectiveness of corrective actions should clearly demonstrate the effect USDA’s corrective actions have on reducing improper payments. (Recommendation 2) The Secretary of Education should revise and document Education’s process for measuring the effectiveness of its corrective actions based on its new statistical estimation methodology for Direct Loan and Pell Grant improper payments. This process should clearly demonstrate the effect Education’s corrective actions have on reducing improper payments. (Recommendation 3) The Secretary of Health and Human Services should document in policies and procedures HHS’s improper payment corrective action plan process. As part of these procedures, HHS should include processes for (1) establishing planned completion dates, (2) monitoring the progress of implementing corrective actions, and (3) measuring the effectiveness of improper payment corrective actions. The process for measuring the effectiveness of corrective actions should clearly demonstrate the effect HHS’s corrective actions have on reducing improper payments. (Recommendation 4) The Secretary of the Treasury should determine whether Treasury’s current improper payment root cause analysis provides sufficiently relevant information that can be used as a basis for proposed corrective actions in reducing EITC improper payments and, if not, update the analysis using more timely data to ensure their reliability for identifying root causes of EITC improper payments. (Recommendation 5) The Secretary of the Treasury should update Treasury’s strategy for addressing the root causes of EITC improper payments to include (1) coordinating with other agencies to identify potential strategies and data sources that may help in determining EITC eligibility and (2) determining whether legislative changes are needed, and developing proposals as appropriate, to help reduce EITC improper payments, such as those related to the inability to authenticate taxpayer eligibility. (Recommendation 6) The Commissioner of SSA should develop and implement a process, documented in policies and procedures, to measure the effectiveness of SSA’s corrective actions for OASDI and SSI improper payments. This process should clearly demonstrate the effect SSA’s corrective actions have on reducing improper payments. (Recommendation 7) We provided a draft of this report for comment to OMB, USDA, Education, HHS, Treasury, VA, SSA, and the Council of the Inspectors General on Integrity and Efficiency (CIGIE). We received written comments from five agencies—USDA, Education, HHS, VA, and SSA—which are reproduced in appendixes I through V and summarized below. The Assistant Director of Treasury’s Risk and Control Group also provided comments in an email, which are summarized below. Treasury, HHS, VA, and SSA also provided technical comments, which we incorporated as appropriate. CIGIE and OMB liaisons informed us that CIGIE and OMB had no comments on the report. In its written comments, USDA stated that it generally agrees with our findings and recommendations. USDA stated that FNS has agency-level corrective actions that correspond to the identified root causes and establishes planned completion dates, monitors the progress, and measures the effectiveness of SNAP’s corrective actions. However, USDA officials did not provide documentation or other information supporting such agency-level corrective actions and efforts. Rather, as discussed in our report, FNS provides technical assistance and support to the states to help them improve payment accuracy and requires them to develop state-level corrective actions. Because FNS’s initiatives do not address specific root causes, we continue to believe that USDA does not have agency-level corrective actions that correspond to the identified root causes of SNAP improper payments. In regard to our recommendation to FNS to develop and implement a process to analyze SNAP state-level root causes and take other related actions, FNS stated that it already has an existing process and recommended that we revise our recommendation to indicate that its existing process should be formalized. In our report, we acknowledge that under statutory requirements and program regulations, FNS requires the states to identify the root causes and develop corrective actions that address them. However, USDA did not provide any evidence that FNS analyzes the states’ root causes to identify similarities and develop corrective actions at the agency level. Therefore, we continue to believe that our recommendation to FNS to develop and implement this process is valid to help ensure that it develops corrective actions at the agency level, if appropriate, and to help reduce improper payments within SNAP. In regard to our recommendation to revise USDA’s procedures, USDA stated that it will develop a proposed action plan to revise its procedures for monitoring the progress and measuring the effectiveness of improper payment corrective actions and the revised process will focus on the impact corrective actions have on the corresponding root causes of improper payments. The actions USDA described, if implemented effectively, would address our recommendation. In its written comments, Education neither concurred nor disagreed with our recommendation, stating that FSA will continue to evaluate and refine its processes to measure corrective actions and the effectiveness of these actions. Further, Education stated that FSA’s measurement of corrective action effectiveness and root cause identification will gain additional precision as FSA collects annual improper payment data and builds upon the new baseline of statistically valid improper payment estimates. Education stated that FSA annually measures the overall effectiveness of its corrective action plans collectively against the improper payment reduction targets, rather than measuring the effectiveness of each individual corrective action. However, as discussed in our report, OMB guidance directs agencies to measure the effectiveness of each individual corrective action annually. We continue to believe that our recommendation to Education is valid to help ensure that Education’s corrective actions are effective in reducing improper payments. In its written comments, HHS stated that it does not concur with our recommendation. Specifically, HHS stated that the portion of our recommendation providing that HHS’s process for measuring the effectiveness of corrective actions should clearly demonstrate their impact on the corresponding root causes of improper payments is operationally impossible and not required by OMB guidance. We acknowledge that given the unique circumstances across federal agencies concerning improper payments, OMB guidance provides some flexibility for how agencies are to measure the effectiveness of their corrective actions. However, if agencies’ corrective actions are effective, they should ultimately reduce improper payments. Without being able to demonstrate whether corrective actions are effective in reducing the associated improper payments, agencies will be uncertain if their actions are actually reducing improper payments and may risk continuing ineffective actions. While we acknowledge that OMB guidance does not explicitly require agencies to demonstrate the impact corrective actions have on the corresponding root causes of improper payments, agencies are required to analyze the root causes of improper payments and develop corrective actions to reduce improper payments. As such, we clarified this portion of our recommendation to indicate that HHS’s process should clearly demonstrate the effect corrective actions have on reducing improper payments, to better align with the purpose of corrective action plans. We also made this revision to our recommendations to USDA, Education, and SSA. In its written comments, VA stated that PSAS supported improper payments statutory requirements by completing annual audit reviews, identifying root causes, and developing a national program action plan to reduce improper payments. VA also stated that PSAS reduced improper payments from 39.7 percent in fiscal year 2018 to 2.1 percent in fiscal year 2019 and continues to make improvements through enhanced audit reviews and consultation with PSAS sites. In its written comments, SSA stated that it concurs with our recommendation and will determine the most cost-effective strategies to remediate the underlying causes of payment errors and monitor, measure, and revise the strategies as needed. The actions SSA described, if implemented effectively, would address our recommendation. In emailed comments, the Assistant Director of Treasury’s Risk and Control Group neither concurred nor disagreed with our recommendations. In regard to our recommendation to update its strategy for addressing root causes of EITC improper payments, Treasury stated that each year it indicates in its corrective action plan that IRS will continue to work with Treasury to develop legislative proposals that will improve refundable credit compliance and reduce erroneous payments. Treasury also stated that its fiscal year 2020 budget request included two legislative proposals that may improve refundable credit compliance and reduce erroneous payments and that both proposals have been in the President’s Budget for several years now. We acknowledge these legislative proposals in our report, and note that although Treasury has made certain legislative proposals, it has not made proposals to specifically help address EITC eligibility criteria issues. Additionally, as noted in the report, Treasury’s strategy does not include identifying and proposing additional legislative changes needed to help reduce EITC improper payments. Therefore, we continue to believe that our recommendation to Treasury is valid to help ensure that Treasury addresses EITC eligibility issues, which Treasury identifies as the primary root cause for EITC improper payments. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget, the Secretary of Agriculture, the Secretary of Education, the Secretary of Health and Human Services, the Secretary of the Treasury, the Secretary of Veterans Affairs, the Commissioner of the Social Security Administration, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-2623 or davisbh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. In addition to the contact named above, Matthew Valenta (Assistant Director), Stephanie Adams (Auditor in Charge), William Beichner, Susanna Carlton, Virginia Chanley, Anthony Clark, Lindsay Hollup, James Kernen, and Diana Lee made key contributions to this report.", "summary": "Improper payments, estimated at almost $175 billion for fiscal year 2019, are a significant problem in the federal government. IPIA and OMB guidance directs agencies to analyze the root causes of improper payments and develop corrective actions to reduce improper payments. This report examines (1) actions that agencies took to identify root causes of improper payments for selected programs, (2) the extent to which their corrective action plans correspond to identified root causes, and (3) the extent to which they monitored progress and evaluated the effectiveness of corrective actions. GAO analyzed corrective action plans reported in fiscal year 2018 for the following eight programs: Department of Education's Direct Loan and Pell Grant; HHS's Children's Health Insurance Program; SSA's Old Age, Survivors, and Disability Insurance and Supplemental Security Income; Treasury's EITC; USDA's SNAP; and VA's Prosthetic and Sensory Aids Service. GAO selected these programs based, in part, on those programs with at least $1 billion in fiscal year 2018 improper payment estimates. Five out of six agencies used their improper payment estimation results to identify the root causes for the eight programs GAO reviewed. However, the Department of the Treasury (Treasury) used 2006 through 2008 taxpayer data to identify root causes of fiscal year 2018 Earned Income Tax Credit (EITC) improper payments. Without timely data on the true root causes of EITC improper payments, Treasury will lack quality information needed to develop appropriate corrective actions to reduce them. In addition, only one agency we reviewed—the Department of Veterans Affairs (VA)—adhered to relevant Improper Payments Information Act of 2002, as amended (IPIA), requirements and Office of Management and Budget (OMB) guidance. The Department of Agriculture (USDA) and Treasury did not develop agency corrective action plans corresponding to the identified root causes of improper payments for the Supplemental Nutrition Assistance Program (SNAP) and EITC, respectively. In addition, the remaining three agencies did not have processes in place to either establish planned completion dates, monitor progress, or measure the effectiveness of their corrective actions in reducing improper payments. Unless agencies develop corrective action plans that correspond to root causes of improper payments and implement processes to monitor progress and measure their effectiveness, their ability to ensure that their efforts will reduce improper payments will be limited GAO is making seven recommendations: one each to Education, HHS, and SSA and two each to USDA and Treasury to improve their processes for addressing root causes of improper payments and measure their effectiveness. In their responses, SSA agreed, USDA generally agreed, Education and Treasury neither agreed nor disagreed, and HHS disagreed with GAO's respective recommendation(s). GAO clarified four recommendations and continues to believe all the recommendations are valid.", "document_type": "gao"}
{"report": "The federal government relies on commercial communications networks to obtain various services, including video conferencing, local and long- distance telephone calls, email, text messages, file transfers, and more. Much of the communications infrastructure is owned or operated by commercial entities. Similarly, federal agencies rely on call centers (also known as contact centers) to handle public inquiries on government programs and services, such as Medicare. These centers utilize automated and live telephone response systems, websites, and trained customer service representatives to provide information to the public. Agencies that contract with industry to meet their telecommunications and call center needs report information about these contracts and their obligations in FPDS-NG—the federal government’s primary database for contract information at the prime contract level. When reporting contract data, agencies report information on the type of product or service being purchased as well as the NAICS code that best describes the principal purpose of the product or service being acquired. See table 1 for a description of the industry categories for businesses that provide telecommunications and call center goods or services. In addition to FPDS-NG, the federal government has developed other contract reporting systems to collect contracting information related to subcontracting. The Electronic Subcontract Reporting System (eSRS) was created in 2005 to streamline contractors’ reporting of progress toward meeting the small business subcontracting goals in their subcontracting plans and to facilitate agency oversight. The Federal Acquisition Regulation (FAR) generally requires that contractors be required to submit an acceptable subcontracting plan when they are awarded a contract that exceeds $700,000 and is expected to have subcontracting possibilities. Depending on the individual contract, the system may contain subcontracting information reported by both the prime contractor as well as multiple subcontractors. The Federal Funding Accountability and Transparency Act Subaward Reporting System (FSRS) was created in 2010 to provide transparency about federal spending. Prime contractors must register and report subcontract information for first-tier subcontractors, as applicable. Information on subcontracts awarded by first-tier subcontractors to other entities, or lower-tier subcontractors, is not required. USASpending.gov was created in 2007 to promote transparency by providing the public with information about where and how federal dollars are spent. USASpending.gov contains prime contract award data from FPDS-NG and subcontract information from FSRS. Telecommunications and information technology (IT) fields have been merging in recent years due to integration of the technologies and combined operational management of their functions. Federal telecommunications systems can include a multitude of IT equipment and products, as well as services, such as managed network services and IT security services. In addition, telecommunications include such broadband internet services. The Federal Information Security Modernization Act (FISMA) of 2014 provides a comprehensive framework for ensuring that effective information security controls are put in place for information resources and assets that support federal operations and for ensuring the effective oversight of the security of the information. Under FISMA, the Office of Management and Budget (OMB) is responsible for overseeing agency information security policies and practices. To implement FISMA, the National Institute of Standards and Technology (NIST)—a component within the Department of Commerce—developed standards and guidelines for agencies to use to help manage information security risks. Both FISMA and OMB require agencies to comply with applicable NIST standards and guidelines. The NIST framework has many components, but generally provides guidance to agencies to manage information security risks for communication and information technology networks. The framework emphasizes that an organization needs to develop and implement appropriate safeguards to ensure delivery of critical services. To accomplish this goal an agency generally must be able to develop an organizational understanding to manage cybersecurity develop and implement appropriate safeguards to ensure delivery of mitigate those events, and restore system capabilities or services that were impaired due to a cybersecurity event. NIST publications can help agencies mitigate potential risks by providing approaches on how to manage or resolve information technology risks. For example, NIST states that agencies should conduct continuous threat monitoring and suggests control activities to implement to help manage supply chain risks, among other things. Some of the controls that NIST recommends are access controls—authentication requirements and physical access controls to limit or detect inappropriate access to data, equipment, and facilities; security management controls—establish a framework and continuous cycle for assessing data systems for security weaknesses, implementing security procedures, and monitoring the procedures to ensure adequate protection of sensitive or critical resources; and contingency planning and restoration of services—planning for how to provide continued or restored services when system interruptions or problems occur. Various federal laws exist to protect workers, establishing requirements related to wages, hours worked, and worker safety and health, among other things. Some of these laws apply specifically to federal contractors, although the requirements may vary depending on factors such as the type and size of the contract. For example, the Service Contract Act establishes minimum wage, fringe benefit, and safety and health requirements for covered federal service contractors. Telecommunications service contracts are exempt from the Service Contract Act, but call center contracts may be subject to it. Similarly, the Walsh-Healey Act establishes minimum wage, overtime, and workplace safety and health requirements for covered federal supply contractors. Contractors are also generally subject to a number of non-discrimination and equal employment opportunity requirements under an executive order and federal laws. For example, covered contractors and subcontractors are prohibited from discriminating in employment based on race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran. In addition, covered contractors and subcontractors generally are prohibited from discriminating against applicants or employees because they inquire about, discuss, or disclose their compensation or that of others, subject to certain limitations. Along with laws that apply specifically to federal contractors, worker protection requirements of other federal laws may also apply, such as the Fair Labor Standards Act or the Occupational Health and Safety Act. Federal contractors are also generally subject to the requirements set forth in the FAR, which provides uniform policies and procedures for acquisition by executive agencies. Specifically, Part 22 of the FAR, Application of Labor Laws to Government Acquisitions, establishes various labor-related requirements for federal contractors and implements applicable requirements, as described above. Federal contractors may also be subject to specific department or agency regulations. For example, when contracting with DOD, contractors must comply with applicable contract provisions and clauses from the Department of Defense Federal Acquisition Regulation Supplement (DFARS), such as clauses incorporated pursuant to DFARS Part 222, Application of Labor Laws to Government Acquisitions. FAR clauses in the prime contract can indicate whether the contractor’s requirements will flow down to its subcontractors. FAR flow-down clauses may be mandatory or discretionary, and are subject to other considerations such as whether a subcontract is performed extraterritorially. Federal agencies reported obligating a total of over $30 billion to acquire telecommunications products and services during fiscal years 2014 through 2018. Telecommunications spending accounted for 1.2 percent of total federal obligations for the 5-year period. Over these five years, the majority of the government-wide telecommunications obligations—84 percent—were awarded for services, such as internet and satellite services with the remainder going to products. In fiscal year 2018, federal agencies reported obligating $6.2 billion to acquire telecommunications products and services—an amount that is consistent with the preceding 4 fiscal years. DOD accounted for about two-thirds of this amount and civilian agencies for roughly one-third. These obligation levels are consistent with the previous 4 fiscal years. Within DOD, DISA—which has responsibility for providing, operating, and assuring command and control and information-sharing capabilities across the full spectrum of military operations—had the highest obligations for telecommunications services and products. Among civilian agencies, the National Aeronautics and Space Administration, the Department of Transportation, and the Department of Veterans Affairs had the highest obligations. These three agencies consistently had the highest obligations in each of the previous 4 fiscal years. Defense and civilian agencies’ obligations for telecommunications for the 5-year period are shown in figure 1. Agencies procured telecommunications products and services from an average of 1,500 vendors each year across the five telecommunications industry categories. A little more than half of these contractors were classified as small businesses. Ten contractors accounted for 52 percent of total federal telecommunications obligations for fiscal year 2018, which is generally consistent with obligation levels in the preceding 4 fiscal years. Appendix III provides additional information on the top federal telecommunications contractors based on dollars obligated. For the 5- year period we reviewed, our analysis shows that agencies reported the majority of dollars obligated were for purchases for wired telecommunications, as illustrated in figure 2. Agencies reported an average of $800 million annually for call center obligations for fiscal years 2014 through 2018, with HHS accounting for at least 80 percent of total spending. Call center spending accounted for 0.2 percent of all federal spending during the 5-year period we reviewed. Almost all—an average of 99.7 percent—of call center contract obligations were awarded for services each year, such as professional and administrative support, help desk, and technical assistance services. For example, the CMS contract in our sample was awarded to acquire management and staffing services for a call center that handles Medicare beneficiary inquiries for 1-800 MEDICARE and consumer inquiries for the Health Insurance Marketplace. Total government-wide call center obligations for fiscal years 2014 through 2018 are shown in figure 3. An average of 133 different contractors had contracts with obligations for call centers during the 5 years we reviewed and about half were classified as small businesses. One contractor accounted for the majority of all obligations with obligation levels ranging from 80 to 84 percent for fiscal years 2014 through 2018. Appendix IV provides additional information on the top call center contractors based on dollars obligated. Three federal reporting systems provide limited information about subcontracting and no information about offshoring because the systems were not designed to capture the extent of these activities. While FPDS-NG captures data on contracts entered into by federal agencies, it was not designed to include subcontracting data. The system has a field to indicate whether the prime contractor has developed a subcontracting plan, but does not have a field for contracting officials to specify details about what or how much of the products or services will be obtained through subcontracting. In addition, FPDS-NG was not designed to collect data on the extent to which prime contractors may offshore work performed on a federal contract. No field exists for contracting officials to indicate whether the contract involves business activities that include offshoring, regardless of what type of products or services are being acquired. eSRS collects information from prime contractors on their planned use of subcontractors. The FAR generally requires that contractors be required to submit an acceptable subcontracting plan when they are awarded a contract that exceeds $700,000 if subcontracting opportunities exist, and impose subcontracting plan requirements on subcontractors that receive subcontracts above certain thresholds. However, as we previously reported in December 2014, eSRS was not designed to provide a list of subcontractors associated with a particular contract. As a result, the utility of eSRS in linking reported subcontractors to prime contracts is limited. Additionally, in general, prime contractors are not required to report in eSRS if a subcontractor’s services are being performed outside of the United States or its territories. Contracting officials told us that they have limited insight into whether prime contractors subcontract with foreign entities. FSRS is used to collect award and entity information, such as subcontractor names and award amounts, from prime contractors on their subcontract awards. Prime contractors obtain and report information provided by their subcontractors into FSRS. However, in June 2014, we reported that we could not verify the subcontract data in FSRS as agencies frequently do not maintain the records necessary to verify the information reported by the awardees. In light of this, we recommended that the Director of OMB, in collaboration with Treasury’s Fiscal Service, clarify guidance on agency maintenance of records to verify the accuracy of required data reported. OMB generally agreed with our recommendation. As of our latest report in April 2017, OMB had not yet taken action to implement our recommendation. We identified several examples of worker protection requirements in our review of the five selected contracts. We categorized those requirements into three areas: wages and hours, workplace safety and health, and protections against certain employer actions. Wages and Hours. These protections ensure the payment of minimum wage rates and authorize overtime pay, as appropriate, among other things. For example: The GSA Networx services contract and the DISA contract for DOD Information Network operations include requirements to ensure that covered contractor employees are to be paid wages at least at the federal minimum wage rate. The CMS call center operations contract and the DISA emergency telecommunications services contract authorize the contractor to provide overtime pay to certain employees if they work more than their standard hours. The CMS call center operations contract and the DISA contract for DOD Information Network operations identify classes of workers and state the minimum wage rate and fringe benefits that may be or are payable to them. For example, the CMS call center operations contract reflects Department of Labor rates for federal hires. The CMS call center operations contract also includes an HHS- specific requirement related to salary rate limitations that specifies that the contractor shall not use contract funds to pay the direct salary of an individual at a rate that exceeds the Federal Executive Schedule Level II in effect on the date the funding was obligated. Workplace Safety and Health. These protections address dangers in the workplace that might affect the workplace safety or health of contractor employees. All five contracts reviewed contain requirements aimed at promoting or ensuring safe behaviors in the work environment, among other things. For example: All five contracts require the contractor to promote a drug-free workplace environment. DISA’s DOD Information Network operations contract requires the contractor to establish specific safeguards to protect the health of its workers who might work in a federal building complex that is known to be a toxic location, since asbestos and toxic metals have been located in the soil. DISA’s DOD Information Network operations contract also includes requirements for the contractor to ensure its employees have health screenings and vaccinations as applicable to ensure they are physically and psychologically fit to perform the work at specific locations, such as those in military operation zones. All five contracts encourage the contractor to establish policies to ban text messaging while driving. Protections against Certain Employer Actions. These protections are intended to protect workers from potentially harmful actions undertaken by employers—such as discrimination in hiring practices, retaliation for reporting company violations, and participation in human trafficking. For example: All five selected contracts included equal employment opportunity provisions that prohibit discrimination in employment based on specific characteristics, such as being a veteran or a person with a disability. The CMS call center operations contract also included an agency requirement for the contractor to cooperate in any investigations into allegations of employment discrimination. The CMS call center operations contract, the two DISA contracts, and the GSA Networx services contract incorporate clauses requiring their contractors to provide whistleblower protections that protect an employee from reprisal when they inform authorities of fraud, waste, abuse, or violations of contract law by the contractor. All five selected contracts include the clause that prohibits the contractor and its employees from any involvement in trafficking in persons. In addition, DISA’s DOD Information Network operations contract requires the contractor to offer employment to specific groups of people under certain circumstances. Specifically, the contractor is to employ local residents when work is to be performed in Hawaii. In addition, the contractor is to offer employment to former federal employees first when work is to be performed at a military base that is closing. Observations on Offshoring. We did not identify offshoring of the products or services being acquired in the five contracts we reviewed. Generally, if a prime contractor awards a subcontract, the contractor will flow down applicable requirements to the first-tier subcontractor and other subcontractors at lower tiers, unless otherwise specified. We identified only one worker protection clause that would flow down to the subcontractor in the event of offshoring—the requirement to prohibit involvement in trafficking in persons. The five selected contracts we reviewed include examples of various safeguards—such as limiting access to data systems and data, system management controls, contingency planning and restoration of services, and restrictions on the use of equipment—to protect data systems and personally identifiable information from unauthorized access and use. These safeguards are all part of NIST standards. Access Controls. Physical access controls and authentication requirements limit, block, or detect inappropriate access to data, equipment, and facilities. These controls help to reduce the chances of data systems being used for malicious purposes and protect the systems from unauthorized modification, loss, or disclosure. For example: The GSA Alaska telecommunications services contract states that the physical access point to the telecommunications closet must be limited to personnel with appropriate identification. In addition, this contract requires the contractor to follow agency security procedures, such as having personnel sign into and out of physical locations and abide by escort procedures. Further, the contractor is required to ensure that all employees have identification that meets specific federal guidelines. The contract also states that subcontractors are subject to personal identity verification, and are to comply with applicable standards. The CMS call center operations contract requires a multifactor authentication—which requires two pieces of identifying information to log in—for call center employees to remotely access sensitive government-owned data on computer systems. In addition, the contract requires all employees who have access to data systems and personally identifying information to pass a background check. Further, the contract reduces the ability of employees to copy or transmit a customer’s personal information by requiring the contractor to ensure a secure floor that prohibits cell phone usage or note taking on paper. According to the CMS officials, the call center employees are required to leave all personal items, such as cell phones, in lockers, and the scripts they reference during calls are laminated. In addition, the supervisor on duty checks desks to ensure personal items are not present. According to CMS officials these steps help protect callers’ sensitive data, such as their medical information. The DISA contract for the day-to-day operations for the DOD Information Network states that the contractor must have a plan in place that includes physical security and protection of the system infrastructure. Security Management Controls. These controls establish a framework and continuous cycle for assessing data systems for security weaknesses, implementing security procedures, and monitoring the procedures to ensure adequate protection of sensitive or critical resources. A variety of security management control requirements were included in the selected contracts. For example: The GSA Networx contract, which provides a variety of network services to the federal government, states that a contractor must comply with FISMA and NIST standards. According to a GSA contracting official, contractors have to show that their information systems are adequately protected against cybersecurity threats before performing any services on a task order. Government officials will certify the system once they agree the system is adequately protected. This certification occurs after a contract has been awarded, but before work begins. According to a government official, these systems are periodically reviewed and monitored to ensure the systems stay protected. The DISA contract for the day-to-day operations for the DOD Information Network requires the contractor to assist the government to ensure that all networks and information systems are accredited in accordance with DOD’s Certification and Accreditation Program, which requires certain cybersecurity protections are in place. This contract also requires that the contractor or any subcontractor implement safeguarding requirements to protect covered contractor information systems, such as limiting access to authorized users, verifying and controlling connections to and use of external information systems, authenticating the identities of users before allowing access to information systems, and limiting physical access to systems and equipment. The contract also requires that the government have access to the contractor’s databases in order to carry out vulnerability testing and audits to safeguard against threats to the integrity, availability, and confidentiality of data or to the functions of information technology systems operated on behalf DISA or DOD. The DISA contract that provides priority telecommunications for executive branch staff in case of an emergency requires that the contractor must identify and analyze threats to the system on a 24- hours-a-day, 7-days-a-week basis, and offer solutions to fix identified weaknesses. DISA contracting officials stated that threats to the data systems are mitigated before contract award because the government is trying to prevent attacks and not just react to threats. Additionally, the contractor has to provide periodic maintenance of the installed networking infrastructure to certify proper functioning of the equipment. The CMS call center operations contract requires that the contractor perform annual vulnerability assessments, which includes tests that attempt to break into the contractor’s systems, the contractor’s system programs, and the contractor’s facility in accordance with agency specific standards. Contingency Planning and Restoration of Services. Planning for how to provide continued or restored services when system interruptions or problems occur is necessary because even a minor interruption can result in lost or incorrectly processed data. NIST has published guidance on the contingency planning process. Several of the contracts we reviewed required the contractor to have contingency plans in place in case of any disruption of services and specified how quickly services are to be restored if disrupted. For example: The GSA Alaska telecommunication services contract requires that the contractor restore service within 4 hours of any system disruption. According to the contracting officer, not restoring the system within 4 hours, unless a longer time is agreed to by the contracting officer, would be considered a performance issue and would count against the contractor during its performance review. This includes restoring any equipment, transmission station, circuit, or area that the government deems critical. The DISA DOD Information Network services contract requires the contractor to ensure that there is no disruption of services on the government networks during routine maintenance of systems, during system upgrades, or while the system has vulnerability testing, among others. The CMS call center contract requires that the contractor develop a business continuity plan that identifies and prioritizes critical systems and recovery strategies, as well as a consolidated business continuity plan. The consolidated plan needs to account for the interdependence between applications and operations and address procedures for sustaining essential business operations while recovering from significant disruptions, including contingencies for a catastrophic loss of equipment required to deliver its services. Restricting the Purchase and Use of Equipment from Identified Countries or Manufacturers. As we reported in July 2018, reliance on a global supply chain introduces multiple risks to federal information systems, including the installation of intentionally harmful hardware or software, reliance on malicious service providers, or installation of hardware or software containing unintentional vulnerabilities such as defective code. NIST published several guidelines to help federal agencies select controls and activities relevant to managing supply chain risk. Our selected contracts included several requirements related to mitigating supply chain risks. For example: Under the CMS call center contract, certain government-provided systems are supplied to the contractor to meet the requirements of the contract. By providing the systems, the government controls what type of equipment is being used and reduces the risk that any compromised equipment is introduced in its network. The DISA contract for the day-to-day operations for the DOD Information Network requires the contractor to use the DISA-approved products list for purchasing equipment for use in repair and similar functional activities. According to a DISA contracting official, this list is continuously updated to make sure that vulnerable products are not being purchased. In addition, this contract specifically prohibits contractors from using certain Chinese-manufactured equipment or services utilizing that equipment. This requirement extends to any equipment or services provided by subcontractors. According to the contracting officer, the contractor requests confirmation from its subcontractors that they are not using prohibited equipment. The contractor then notifies the contracting officer that prohibited equipment is not used on the contract. All five contracts include a restriction on purchases of most goods and services from specific countries, such as Cuba and Iran. The contract requires this restriction to flow down to any subcontractor. Privacy for Personally Identifiable Information. The CMS call center contract involves handling personally identifiable information, such as private medical information. As part of the contract terms, contractor personnel are required to follow specific health care privacy requirements to protect customers’ personal health information. In addition, the contract includes agency-specific requirements to protect personally identifiable information and personal health information. Observations on Offshoring. The five contracts we reviewed included requirements that limited the contractors’ opportunity to use offshoring for labor. The DISA contract for the day-to-day operations for the DOD Information Network stipulates that only U.S. citizens can be hired to perform services. According to the contracting officer, the DISA contract that provides priority telecommunications for executive branch staff in case of an emergency also requires that the contractor hire only U.S. citizens. In addition, the GSA Networx services contract states that work on some orders may require U.S. citizenship. The GSA Alaska telecommunications services contract states that contractor personnel may be required to successfully pass a background check to work in controlled areas under the contract. The CMS call center contract requires that the call center be located in a facility within the continental United States. According to officials, this requirement helps protect data and privacy information. CMS officials stated that generally for call center contracts the contractor must obtain prior approval from the agency’s contracting officer in writing if it wants to subcontract or move operations to a location outside of the United States or its territories. According to CMS contracting officials, they have never received a request to offshore call center operations. We provided a draft of this product to DOD, DOL, GSA, and HHS for review and comment. DOL, GSA, and HHS provided technical comments, which we incorporated as appropriate. DOD informed us that it had no comments on the draft report. We are sending copies of this report to the appropriate congressional committees and the Secretaries of Defense, Labor, and Health and Human Services and the Administrator of General Services. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact William T. Woods at (202) 512-4841 or woodsw@gao.gov or Cindy S. Brown Barnes art (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Our review of data from the Bureau of Labor Statistics (BLS) Quarterly Census of Employment and Wages shows that employment in the telecommunications sector overall declined 12 percent from calendar years 2014 through 2018, as illustrated in figure 4. In contrast, during the same 5-year period, total employment across all industries in the United States grew by 7 percent. According to BLS data, the decline in telecommunications employment has been underway since at least 2009. BLS projects this decline will continue through at least 2028. Department of Labor (DOL) data series on trends in employment are not designed to identify causes of employment changes. However, BLS officials and other researchers cited the role of technology as a possible cause of the decline in telecommunications employment. For example, BLS officials said the move toward newer technologies, such as satellite transmissions, has had an adverse impact on employment. Additionally, representatives of a major telecommunications contractor told us that technological advances either resulted in fewer employees being needed to perform specific functions or replaced previous manual operations with automated processes. In addition, the representatives stated that uses of artificial intelligence, such as smart networks and machine learning, facilitate tasks that in the past relied extensively on human labor. Finally, the effect of technology on employment in telecommunications has also been noted by an industry analyst. Employment in the telecommunications industry has been marked by job gains as well as losses in the last 5 years, although generally, losses have exceeded gains. BLS’s Business Employment Dynamics data capture the gross number of job gains from establishment openings and expansions and job losses from establishment closings and contractions across the U.S. economy. In the last 5 years, new jobs in telecommunications have been generated; however, job losses have exceeded job gains in almost every quarter since 2014, as shown in figure 5. The effect of offshoring on employment in telecommunications, if any, is unknown due to the absence of data. Although U.S. employment in telecommunications has declined, the role of offshoring as a potential contributor to the decline is unclear, due to a lack of data and because offshoring is one of many factors that can affect employment levels. According to BLS officials, no public or private data exist that estimate the extent of offshoring in this or any industry sector. BLS officials told us that little interest has been expressed in collecting data on offshoring. They noted that if BLS were to develop a new survey aimed at measuring the extent of offshoring, technical issues—including determining what data should be collected that would give such insight—would need to be resolved. Furthermore, the BLS officials stated that they did not identify offshoring as a factor contributing to recent employment declines in telecommunications based on their industry research, which included interviews with industry specialists. According to BLS researchers, offshoring is one of many factors that can affect job gains and losses for occupations within an industry. In a 2008 article estimating the susceptibility of different occupations to offshoring, BLS researchers cautioned that “no attempt should be made to attribute growth rates in an occupation, or differences between occupations, to offshoring.” Overall, employment in call centers has fluctuated recently, but appears relatively stable over the period of calendar years 2014 through 2018, though it remains higher than during the previous 5 years. As shown in figure 6, after rising for a few years, in 2018 employment returned to a level slightly below that reached in 2015. According to BLS officials, employment in the business support services industry, which includes call centers, is projected to increase modestly through 2028. BLS officials said it is not clear why employment in call centers declined in 2018. Although GAO and others have identified call centers as potentially subject to offshoring, the full extent of offshoring occurring within the U.S. call center industry is unknown. Some anecdotal evidence exists about the purported growth of offshore call centers that serve U.S. companies. However, we found no analyses in our literature review regarding the effects of offshoring on call center jobs overall. Just as with telecommunications, many other factors potentially affect call center employment, such as technological advances. For example, interactive voice response technology has been used to provide responses to simple inquiries, which to some extent may reduce or eliminate some call center work. Although we did not find studies that address the effects of offshoring on telecommunications and call centers specifically, the literature we reviewed discussed some characteristics of workers, services, and companies that potentially influence offshoring decisions, in general, across industries. As such, offshoring decisions may involve, but are not limited to, considerations of the presumed interchangeability of U.S.- based and overseas workers, workers’ languages and cultures, technical requirements for the services being offshored, and companies’ ability to manage offshoring. This report addresses: (1) total federal obligations for telecommunications and call center contracts; (2) worker protections identified in selected telecommunications and call center contracts; and (3) data security and privacy protection requirements identified in these contracts. This report also includes observations on the extent and effect of offshoring. For the purposes of this report we define “telecommunications” to encompass the preparation, transmission, communication, or related processing of information that can be in the form of voice, video, or data; “call centers” to include centers handling inquiries via multiple channels such as telephone, Web page, e-mail, and postal mail; and “offshoring” to mean the obtaining of goods or services from non-U.S.-based employer subcontractors located outside of the United States and its territories that use non-U.S. citizen employees. In addition, we gathered information on employment trends for the telecommunications and call center industries for calendar years 2014 through 2018. To determine the level of federal obligations for telecommunications and call centers, we used data from the Federal Procurement Data System- Next Generation (FPDS-NG) for fiscal years 2014-2018. We identified obligations for telecommunications and call center contracts by using the associated North American Industry Classification System (NAICS) codes for these industry sectors. As defined in the NAICS manual, telecommunications contracts are identified as having a NAICS code starting with the prefix 517, and call center contracts are identified as having a NAICS code starting with the prefix 56142. To identify examples of worker protections and data security and privacy protections in federal contracts, we selected a nongeneralizable sample of five contracts from three agencies with some of the highest obligations for telecommunications and call center contracts during fiscal years 2014 through 2018. Specifically, we selected (1) the Department of Defense (DOD) because it obligated the highest amount for telecommunications contracts; (2) the Department of Health and Human Services (HHS) because it obligated the highest amount for call center contracts; and (3) the General Services Administration (GSA), which is among the top ten agencies with the highest amounts for telecommunications contracts, because it provides a government-wide contract available for agencies to place orders for telecommunications and call centers. We then identified the component within each agency that obligated the most for these services or that provides a large government-wide contract vehicle. The components were DOD’s Defense Information Systems Agency, GSA’s Federal Acquisition Service, and HHS’s Center for Medicare and Medicaid Services (CMS). We selected contracts that included a large call center and a large government-wide telecommunications contract vehicle. We also selected a variety of telecommunications contracts that were among the highest obligations during fiscal years 2014 through 2018, and represented different types of telecommunications services procured during the period, such as wired and wireless services. Table 2 provides a synopsis of the 5 contracts included in our review. We reviewed documentation from the five selected contracts, along with the relevant federal acquisition regulations for worker protections, data security and privacy protections, subcontracting, and offshoring. We interviewed cognizant contracting officials to clarify our understanding of the contract requirements we identified related to worker protection and data security and privacy protections. We also met with representatives from three contractors to obtain their insights into contracting with the government, relevant contract requirements, and industry trends. The purpose of our contract review was to illustrate the different worker protections and data security and privacy protections that may be included in these types of contracts. To address the employment trends in telecommunications and call centers and how they were affected by offshoring, we reviewed employment data from the Quarterly Census of Employment and Wages published by the Bureau of Labor Statistics (BLS) within the Department of Labor. The Quarterly Census of Employment and Wages program publishes a quarterly count of employment and wages reported by employers that covers more than 95 percent of U.S. jobs and is supported by quarterly reports from all private sector employers. We also reviewed data from BLS’ employment projections program, which draws from several BLS data collections as well as interviews with industry specialists and reviews of relevant articles to develop information about the labor market for the nation as a whole for 10 years in the future. In addition, we reviewed BLS’ Business Employment Dynamics data, which consist of a quarterly series of statistics on gross job gains and gross job losses for the entire economy. Gross job gains and gross job losses reveal some aspects of business dynamics, including establishment openings and expansions, and establishment closings and contractions. The quarterly data series include the number and percent of gross jobs gained by opening and expanding establishments, and the number and percent of gross jobs lost by closing and contracting establishments. Furthermore, we reviewed DOL data on layoffs collected by the Trade Adjustment Assistance program that are considered to be caused by trade through shifts in production or services to a foreign country. While the data include layoffs in telecommunications, DOL officials did not believe the data would be useful for this report. Specifically, the data do not necessarily reflect all layoffs in a given sector, but only those associated with requests for investigations by DOL as to the role of trade in the layoff, and initial estimates of affected workers—those facing layoffs and those threatened by layoffs—are not representative. To review the potential effect of offshoring on employment trends, we performed a literature review of selected economic research and other relevant articles, and discussed the results with DOL officials. To identify relevant material—including reports, dissertations, working papers, and journal articles—we searched databases including the National Bureau of Economic Research, Bureau of Economic Analysis, Business Source Corporate Plus, EBSCO, EconLit, ProQuest (including dissertations and theses), Social SciSearch, Public Affairs Information Service via DIALOG, Lexis Trade files, SSRN, WorldCat, National Academies Press, and National Technical Information Service. We used search terms that included variations on “telecommunications” and “call centers,” as well as “offshoring,” “offshore outsourcing,” “labor market impact,” “worker displacement,” “layoffs,” “employment trends,” and “hiring trends.” From our initial literature search we selected 13 documents for more in-depth review. We excluded references that addressed the effects of offshoring on non-U.S. economies and labor forces, or were otherwise beyond our scope, such as working conditions, work flow, collective bargaining, customer service, service quality, or training in call centers. Based on this research, we identified reasonable observations about employment trends and offshoring in telecommunications and call centers. As noted in this report, research on the questions addressed in this report reaches different conclusions. The relevant research that we reviewed provided some insights on how offshoring could potentially affect the telecommunications and call center industries, but provided no information regarding the extent of the impact. Because of this and other data limitations, we were unable to determine the extent to which offshoring may be occurring and what effect offshoring is having on the telecommunications and call center industries. We conducted this performance audit from March 2019 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Agencies reported obligations for approximately 1,500 different contractors that provided telecommunications products and services each year during fiscal years 2014 through 2018. Ten of these contractors accounted for 52 percent of obligations for telecommunications contracts in fiscal year 2018. The top contractor received 10 percent of the total telecommunications obligations in fiscal year 2018, and was also one of the top three contractors in the preceding fiscal years. Figure 7 shows the top 10 telecommunications contractors’ based on total obligations during fiscal years 2014 through 2018. Agencies reported obligations for approximately 133 different call center contractors each year during fiscal years 2014 through 2018. Ten contractors accounted for 94 percent of obligations for call centers in fiscal year 2018; with one contractor accounting for 83 percent of total obligations for the 5-year period we reviewed. Figure 8 shows the top 10 telecommunications contractors’ based on total obligations during fiscal years 2014 through 2018. Although the amount of obligations each year changed, these contractors were generally among the top 10 across all 5 fiscal years. In addition to the contacts named above, Candice Wright (Assistant Director), Blake Ainsworth (Assistant Director), R. Eli DeVan (Analyst-in- Charge), Pedro Almoguerra, Sarah Cornetto, Lorraine Ettaro, Suellen Foth, Stephanie Gustafson, Victoria Klepacz, Chris Morehouse, Patricia Powell, Miranda Riemer, Roxanna Sun, Alyssa Weir, and April Yeaney made key contributions to this report.", "summary": "The federal government relies on an extensive global telecommunications network to carry out operations and provide information to the public. These networks and call centers, which handle public inquiries, are often maintained or supported by contractors. Concerns have been raised about the extent to which federal contractors are subcontracting or offshoring work, and have in place worker protections and mechanisms to secure the technologies and the data they handle. GAO was asked to review aspects of contracting for federal telecommunications and call centers, including the extent of subcontracting and offshoring. This report provides information on, among other things (1) federal obligations on telecommunications and call center contracts, (2) worker protections identified in selected contracts, and (3) data security and privacy protections identified in selected contracts. GAO analyzed federal procurement data for fiscal years 2014 through 2018 (the most recent available), reviewed a nongeneralizable sample of five contracts from three agencies with significant telecommunications and call center procurements to identify worker protections and data security and privacy protections; and interviewed relevant officials and federal contractors about contracting and industry trends. The federal government obligated over $30 billion for telecommunications contracts and almost $4 billion for call center contracts from fiscal years 2014 through 2018. On average for the 5-year period, telecommunications and call center obligations were a nominal portion of total federal spending—accounting for 1.2 percent and less than 0.2 percent, respectively. Defense agency obligations accounted for the majority of federal telecommunications spending to support a range of information capabilities across the full spectrum of military operations. The Department of Health and Human Services accounted for the majority of call center obligations to support customer inquiries about Medicare and the health insurance marketplace, among other services. Federal procurement data systems do not collect information that can provide insight into the extent of subcontracting or offshoring—including for telecommunications and call center contracts—because they were not designed to do so. GAO's review of selected contracts found that four of the five contracts expressly stated that some or all work must be performed within the continental United States or by U.S. citizens. GAO identified several examples of worker protection requirements in the five selected contracts, generally falling into the areas of wages and hours, workplace safety and health, and protections against certain employer actions. With regard to data security and privacy protections, the five selected contracts GAO reviewed included requirements to limit access to data systems and data maintained, establish security management procedures for and monitoring of data systems, or establish contingency plans for how to provide continued or restored services when system interruptions or problems occur.", "document_type": "gao"}
{"report": "The MHS is a complex organization in which responsibility for health care delivery is shared among the military departments—the Army, the Navy, and the Air Force—and the Defense Health Agency (DHA), with oversight from the Office of the Secretary of Defense and advice from the Joint Staff. As such, several leaders have responsibility for DOD’s medical workforces, their readiness, and the MTFs to which many of them are assigned. Specifically: The Under Secretary of Defense for Personnel and Readiness is the principal staff assistant and advisor to the Secretary and Deputy Secretary of Defense for health-related matters and, in that capacity, develops policies, plans, and programs for health and medical affairs. The Secretaries of each military department are responsible for organizing, training, and equipping military forces as directed by the Secretary of Defense as well as responsibilities related to ensuring the readiness of military personnel, and providing military personnel and other authorized resources in support of the combatant commanders and the DHA. The Surgeon General of each respective military department serves as the principal advisor to the Secretary of the military department concerned on all health and medical matters of the military department. The Assistant Secretary of Defense for Health Affairs (ASD(HA)) serves as the principal advisor for all DOD health-related policies, programs, and activities. He or she has the authority to develop policies, conduct analyses, provide advice, and make recommendations to the Secretary of Defense and others; issue guidance; and provide oversight on matters pertaining to the MHS. Further, the ASD(HA) prepares and submits a DOD unified medical program budget which includes, among other things, the Defense Health Program budget to provide resources for MTFs and the TRICARE Health Program. The Director of the DHA manages, among other things, the execution of policies issued by the ASD(HA) and manages and executes the Defense Health Program appropriation. The Director of the DHA is also responsible for the TRICARE Health Program. In December 2016, Congress expanded the role of the DHA by directing the transfer of responsibility for the administration of each MTF from the military departments to the DHA. By no later than September 30, 2021, the Director of the DHA will be responsible for the administration of each MTF. Specifically, the Director of the DHA will be responsible for budgetary matters, information technology, health care administration and management, administrative policy and procedure, and military medical construction, among other things. As of October 2019, the DHA had assumed administration and management responsibilities for all MTFs within the United States. In fiscal year 2019, DOD’s Defense Health Program-funded workforce numbered over 174,000 personnel, comprising active-duty servicemembers from each military department (the Army, the Navy, and the Air Force), federal civilian employees of DOD, and private-sector contractors. These personnel included health-care providers, such as physicians (both primary and specialty care providers), nurses, and enlisted specialists who assist with medical procedures, and administrative and support personnel. MTFs vary in size and capabilities from small clinics, to ambulatory surgery centers, hospitals, and medical centers. Clinics generally provide primary-care services, which may include pediatrics at some locations. Other health-care services at clinics range from urgent care, women’s health, occupational health, and behavioral health, to orthopedics and other specialty services depending on location and population demand, according to MHS officials. Some clinics treat only active-duty servicemembers. Other clinics, along with hospitals and medical centers, also serve other eligible beneficiaries, including military family members, retirees, and some civilian employees of DOD. DOD’s hospitals provide emergency medicine, inpatient care and other specialty care services depending on population demand, according to MHS officials. For example, they generally offer surgical capabilities and labor and delivery services. According to DOD Instruction 6000.19, the primary purpose of MTFs is to support the readiness of the military services. In addition, the guidance states that the size, type, and location of MTFs must further this readiness objective. Further, each MTF must spend most of its resources supporting wartime skills development and maintenance for military medical personnel, or the medical evaluation and treatment of servicemembers. To that end, MTFs serve as training and readiness platforms for active-duty medical providers in two respects. First, many MTFs host graduate medical and dental education programs for physicians and dentists, and other training and education programs for medical providers. Graduate medical education (GME) programs train physician specialties through internships, residencies, and fellowships, thereby helping maintain the necessary pipeline of physicians to staff the MTFs and to deploy in support of military operations. The MTFs host non-GME training and education programs for other medical personnel, such as physician assistants, nurses, and enlisted technicians, which help them attain and maintain their skills. Second, day-to-day patient care at MTFs helps maintain the clinical skills and readiness of medical providers. The military departments track clinical readiness for providers using a series of checklists for deployable medical specialties. In addition, since 2018, DOD has piloted a clinical readiness metric for select physician specialties that provide combat casualty care. To meet the metric, a physician must attain a minimum threshold of points that indicate the complexity, diversity, and volume of patient care they provided. Finally, the MTFs also maintain data on physicians’ clinical workloads to measure their productivity against benchmarks and thereby approximate their clinical readiness. These clinical workload data are recorded as work Relative Value Units (wRVU), a metric of the level of professional time, skill, training, and intensity to provide a given clinical service. Under TRICARE, DOD maintains a purchased-care system of civilian providers to augment MTF capabilities. In each TRICARE region (East and West), DOD contracts with private-sector companies—referred to as managed-care support contractors—to develop and maintain networks of civilian providers and perform other customer service functions, such as processing claims, enrolling beneficiaries, and assisting beneficiaries with finding providers. The Director of the DHA awards and oversees the managed-care support contracts. TRICARE’s non-Medicare-eligible beneficiaries generally obtain coverage through two health plan options—TRICARE Prime (a managed-care option) and TRICARE Select (a self-managed, preferred provider option). All active-duty servicemembers are required to enroll in the Prime option, while other TRICARE beneficiaries may choose it. Prime enrollees receive most of their care from MTFs and also may receive purchased care from network civilian providers. Prime has the lowest out- of-pocket costs for beneficiaries, as care provided at MTFs does not have a copayment. TRICARE Prime has five access standards that set requirements for (1) travel time to provider sites, (2) appointment wait time, (3) availability and accessibility of emergency services, (4) composition of network specialists, and (5) office wait time. TRICARE Select beneficiaries are able to obtain health care from network and non-network providers. They can also receive care from MTFs, but they have a lower priority for receiving care than TRICARE Prime beneficiaries and are seen on a space-available basis. Section 703(a) of the NDAA for Fiscal Year 2017 added section 1073d to title 10, United States Code, which set forth various requirements for MTFs. To support the medical readiness of the armed forces and the readiness of medical personnel, the Secretary of Defense is required to maintain three types of MTFs—medical centers, hospitals, and ambulatory care centers (or clinics). All of these MTFs are required to provide specific health services required to maintain medical readiness. Hospitals are to be located in areas where civilian health care facilities are unable to support the health care needs of members of the armed forces and covered beneficiaries. Both hospitals and clinics are to provide limited specialty care that is cost-effective or is not available at civilian health care facilities in the area. In 2017, DOD appointed a Reform Leader for Health Care Management. Among other responsibilities, the Reform Leader led a work group to address Section 703 (hereafter, we refer to this as the 703 Work Group). The 703 Work Group included representatives from the Office of the ASD(HA), DHA, Joint Staff, the military services, and the TRICARE Health Plan. Together, the 703 Work Group members led DOD’s efforts to address section 703(c) of the NDAA for Fiscal Year 2017 by updating its 2016 Report on Military Health System Modernization (“the Modernization Study”) to address the future restructuring of MTFs pursuant to 10 U.S.C. § 1073d; determine the scope of its review of MTFs in the United States (i.e., identify which MTFs to evaluate for the Plan, as opposed to those to evaluate at a later date); develop MTF-specific recommendations for whether to restructure an MTF and in what ways to do so by developing and applying a methodology to assess each MTF in accordance with 10 U.S.C. § 1073d; and draft the final section 703(d) Plan to Congress delineating the restructuring actions it determined. In making determinations for selected MTFs, the 703 Work Group drafted a “Use Case” for each MTF summarizing potential restructuring actions and their analytical basis. The Work Group presented each MTF “Use Case” for review to a team of senior DOD leaders, including the Under Secretary of Defense for Personnel and Readiness, the ASD(HA), the undersecretaries of the military departments, and military service leaders, among others. When the senior leaders agreed on the restructuring actions for the MTFs, the 703 Work Group presented those determinations to the Secretary of Defense for approval. In recent years, DOD leaders have taken steps to refocus the MTFs as platforms for sustaining high-quality combat casualty care and the operational readiness of active-duty medical providers while increasing efficiency, in part by responding to congressional mandates. In 2016, for example, DOD submitted the Modernization Study to Congress in response to section 713 of the Carl Levin and Howard P. “Buck” McKeon National Defense Authorization Act (NDAA) for Fiscal Year 2015. Its main goals were to increase medical force readiness to support military operations and achieve cost savings. The Modernization Study included an MTF analysis of 24 hospitals to determine whether they should maintain inpatient capabilities or birthing centers. It recommended changes for 10 of the 24 hospitals, including closing inpatient services in whole or part at eight of them. In September 2016, we reported that the Modernization Study’s recommendations positioned DOD to improve the effectiveness and efficiency of the MHS, but there were shortcomings in its methodology. To strengthen any future assessments of MTF changes, we recommended that DOD describe steps taken to assess the reliability of supporting data. DOD concurred with the recommendation and has taken some steps to implement it. In response to other provisions in the NDAA for Fiscal Year 2017, DOD has made reforms aimed at improving the MHS focus on readiness. Our prior work has made recommendations to address gaps in those reforms. For example, in February 2019 we reported that DOD had not determined the required size and composition of its operational medical and dental forces who support the wartime mission or submitted a complete report to Congress, as required by section 721 of the NDAA. We recommended that DOD establish joint planning assumptions and a method for assessing efficiencies and risk, use them to determine its operational medical and dental requirements, and report the requirements to Congress. DOD concurred but had not implemented the recommendations as of May 2020. We also reported in February 2019 that DOD had begun initiatives to maintain the wartime readiness of medical providers in response to section 725 of the NDAA for Fiscal Year 2017. However, DOD’s methodology was limited with respect to a key initiative—the use of a metric to assess medical providers’ clinical readiness. We made three recommendations to improve DOD’s application of the metric. DOD concurred but had not implemented them as of May 2020. According to MHS leaders, efforts to identify the required number of operational medical personnel and the level of readiness they must maintain (pursuant to sections 721 and 725) were foundational steps toward section 703 of the NDAA for Fiscal Year 2017. A list of other related products is also included at the end of this report. The DOD 703 Work Group’s methodology for determining MTF restructuring actions was thoroughly documented and prioritized cross- cutting statutory elements, including support for military readiness, adequate nearby civilian health care, and cost-effectiveness. However, the group based key parts of its methodology on some incomplete and inaccurate information. In reviewing the 703 Work Group’s methodology for determining MTF restructuring actions, we found that the group prioritized cross-cutting elements from 10 U.S.C. § 1073d to guide its approach. Its methodology to evaluate each selected MTF consisted of data analyses and interviews with officials from the MTF and its host installation. The 703 Work Group based its MTF evaluation determinations on, by order of priority, the (1) support each MTF provides to servicemembers’ medical readiness and the readiness of military medical providers, (2) adequacy of civilian health care facilities and providers to support the health care needs of servicemembers and other beneficiaries through purchased care near where each MTF is located, and (3) the cost-effectiveness of direct care services at the MTF relative to purchased care in the area. In addition to thoroughly documenting this methodology for evaluating the MTFs, the 703 Work Group documented the basis for the resulting conclusions. Servicemembers’ and military medical providers’ readiness. According to 703 Work Group leaders, as a first step in developing a methodology for evaluating MTFs for restructuring actions, they decided on a strategy they believed would prioritize MTF support to servicemembers’ medical readiness and the readiness of military medical personnel. To that end, the Work Group established minimum criteria to determine an MTF’s level of support to readiness. In most cases, the Work Group determined that MTFs should maintain certain minimum capabilities for servicemembers’ individual medical readiness, including primary care and, on a case-by-case basis, specialty services such as behavioral health and physical therapy. In addition, through site visits and interviews with MTF and installation personnel, the Work Group determined that certain MTFs should maintain urgent and emergency care services if they support a large training component on an installation. The other element of the 703 Work Group’s strategy to prioritize readiness was to evaluate the contribution of each MTF toward the clinical readiness of military medical providers and recommend restructuring actions on the basis of attaining minimum standards therein. In particular, our review of methodology documents revealed the Work Group prioritized MTFs’ support to the readiness of combat casualty care physicians. In doing so, the Work Group analyzed clinical workloads (e.g., wRVUs) and readiness metrics to identify which MTFs supported the physicians’ attainment of minimum thresholds. Finally, the Work Group determined that MTFs that host a GME program for training a combat casualty care or other physician specialty, or a graduate dental education program, should preserve the inpatient services required to continue the program. Adequacy of nearby civilian health care. A secondary criterion the 703 Work Group applied in its methodology for evaluating MTFs for restructuring was its determination of whether civilian health care facilities and providers in proximity to a given MTF (i.e., TRICARE network providers as well as non-network providers) were adequate to absorb an increased demand for certain health care services from the MTF—that is, whether DOD could use purchased care from civilian providers to replace care divested from a restructured MTF. According to Work Group officials, their strategy in applying this criterion was to reduce or eliminate health care services from MTFs if those capabilities (1) were not needed for readiness purposes and (2) could be adequately replaced with civilian facilities and providers through purchased care. In making this determination, the 703 Work Group conducted two assessments for each evaluated MTF. These assessments applied different criteria and assumptions to determine the adequacy of civilian health care, as shown in table 1. In addition, the Work Group supplemented the assessments by interviewing MTF and installation personnel to gain their perspectives about the availability of civilian care nearby. Cost-effectiveness of direct-care services at MTFs. Last in order of priority was the 703 Work Group’s determination of whether the MTF- delivered health care is cost-effective relative to nearby purchased care. Specifically, the Work Group assessed whether the cost per unit of health care delivered at each evaluated MTF was less than, equal to, or greater than the unit cost of purchased care. According to Work Group officials, these assessments supplemented the criteria on readiness and civilian health care adequacy by lending a resource-informed perspective to the overall methodology. A determination that an MTF’s unit costs exceeded those of nearby purchased care would confirm to the Work Group that it should consider replacing health care services from that MTF with purchased care, provided that the group had first determined that the corresponding capabilities (1) were not needed to support readiness, and (2) could be adequately replaced with purchased care from civilian health care providers nearby. Table 2 below illustrates an example of the 703 Work Group’s calculation of cost-effectiveness at one MTF it evaluated—the 2nd Medical Group outpatient clinic at Barksdale Air Force Base in Louisiana. In this case, the MTF’s unit cost in fiscal year 2017 was more than two times the cost of purchased care. These results confirmed the Work Group’s recommendation that some of the MTF’s capabilities should be replaced with purchased care—specifically, the health care services that it provided to non-active-duty servicemembers. Notwithstanding the positive aspects of DOD’s methodology for evaluating MTFs previously discussed, our review found that information the 703 Work Group considered was sometimes limited in completeness, accuracy, or both. Specifically, the Work Group (1) conducted limited assessments of MTFs’ readiness support to military primary care and nonphysician medical providers, and did not include, as part of its methodology, (2) complete and accurate data about the quality of and access to purchased care from civilian providers, or (3) alternative assumptions that could affect the perceived cost-effectiveness of MTF- provided direct care. The 703 Work Group conducted limited assessments of MTFs’ support for the readiness of military primary care physicians and nonphysician medical providers, including nurses, physician assistants, and enlisted medical and surgical specialists, which constitute a substantial portion of DOD’s medical forces. As discussed previously, the Work Group prioritized assessments of MTFs’ support to combat casualty care physicians’ readiness. For military primary care providers, the Work Group determined whether a minimum amount of patient care workload (i.e., RVUs) was available at each MTF to support productivity goals. This was due in part to the fact that DOD has not developed a clinical readiness metric for primary care and nonphysician providers as it has for combat casualty care providers, according to Work Group officials. Unlike a clinical readiness metric, a productivity goal does not account for the types of workload needed for readiness. According to MTF medical providers, they could meet their productivity goal as their MTF restructures, but doing so would not ensure that they addressed diverse and complex medical issues needed to maintain their clinical skills. MHS senior leaders and MTF officials, including providers, expressed concern that opportunities to treat diseases and nonbattle injuries will be limited in MTFs that restructure to serve only active-duty servicemembers. We also found that the 703 Work Group did not assess the support that certain training and education programs provide to the readiness of medical personnel at evaluated MTFs. The Work Group surveyed each MTF within its scope to identify any graduate medical and dental education and non-GME training and education programs the facility hosts. The Work Group determined that nonprimary care GME and graduate dental education programs are essential to maintain at MTFs, but did not evaluate the readiness benefits of primary care GME and non- GME training to MTFs. We found that half of the MTFs identified for restructuring as active-duty clinics or for closure host one or more non- GME training program for nurses, nurse practitioners, and enlisted medical personnel, among others. Four MTFs that were deferred or identified for reduction in capabilities or closure either host or support a GME program for primary care physicians. MTF officials we interviewed expressed concerns about the effects on military providers’ readiness from reducing or displacing the programs. However, DOD’s Plan states that any effects on GME and non-GME training programs will be addressed later in a next phase of executing MTF restructuring transitions, as discussed later in this report. Although DOD assessed the availability of civilian health care providers and facilities in proximity to MTFs, as described above, our review of DOD’s assessments found that information gathered and applied in the course of its methodology was sometimes incomplete and inaccurate. Specifically, the information we reviewed did not consistently account for the quality of available civilian health care providers in proximity to MTFs and the extent to which those providers meet access-to-care standards, as described in detail below. As a result, DOD’s assessments may have included providers of lower quality health care and those who do not meet DOD’s access-to-care standards. Including such providers in its assessments means that DOD’s conclusions could be overestimated regarding the adequacy of civilian health care providers in proximity to some MTFs. Quality of available care near MTFs. The 703 Work Group’s assessments did not consistently account for the quality of available providers located in proximity to each MTF. Although the TRICARE Health Plan assessments documented and considered patient satisfaction scores and quality ratings for hospitals from the Centers for Medicare & Medicaid Services, its assessments of individual providers did not contain information about quality of care. The independent contractor’s assessments did not include any information about the quality of available providers it identified. Instead, DOD generally assumed that all identified providers were of sufficient quality. Officials from the 703 Work Group stated that they sometimes discussed the quality of available civilian health care during their site visits and interviews with MTF officials. However, our review found that quality of care was not consistently documented or considered for decision-making purposes. For example, in our review of 11 selected MTFs, we found that the Work Group documented and considered the quality of available civilian health care in proximity to one of the 11 MTFs—Bayne-Jones Army Community Hospital at Fort Polk, Louisiana. In this instance, the Work Group’s information about the variable quality of civilian health care near Fort Polk led to their determination that available care was not yet adequate to restructure the MTF. Other MTF officials discussed with us concerns they had about the quality of purchased care from some civilian providers. Similarly, a recent study found that TRICARE-insured families were less likely to report accessible or responsive care compared to civilian peers, whether commercially or publicly insured or uninsured. We have previously reported on concerns about DOD’s information about the quality of purchased care. In September 2018, we reported that the MHS does not monitor and report on quality measures for individual civilian providers, although it does so for purchased care networks as a whole. In contrast, the MHS maintains numerous measures for its MTFs to track, assess, and report the quality of care and related patient outcomes. Access to an accurate and adequate number of current civilian health care providers in the TRICARE networks. DOD’s assessments of available civilian health care surrounding MTFs did not consistently apply complete and accurate information about patients’ access to care in terms of the number of available TRICARE providers in proximity to MTFs. DOD’s assessments relied on the directories of network providers (primary and specialty care) that are maintained by each of the regional TRICARE contractors. In November 2019, we reported on problems with the accuracy of these provider directories. Specifically, we reported that as of June 2019, the TRICARE West region contractor’s directory of network providers was 76 percent accurate and the East region’s was 64 percent accurate, according to DHA officials. However, we found that the TRICARE Health Plan verified the accuracy of the directory entries of network providers in proximity to only one of 77 MTFs—the Army’s Farrelly Health Clinic at Fort Riley, Kansas. In this instance, the list of available health care providers in proximity to Farrelly clinic was overstated by 26 percent because of duplicate listings and practices that had closed, among other factors. Likewise, MTF officials we interviewed stated that the TRICARE network directories in their area contained inaccurate information, such as outdated provider listings, and overstated the number of providers who were accepting new TRICARE patients. Access to providers within standards for patients’ drive time. DOD’s independent contractor assessments of available civilian health care providers (both TRICARE network and non-network providers) used some inaccurate information about those providers, especially their locations. For 11 selected MTFs, we found that about 56 percent of primary care providers and 42 percent of specialty care providers that an independent contractor identified in its assessment exceeded DOD’s drive-time standards for TRICARE Prime patients by varying degrees, as shown in figure 1. A certain portion of the providers listed for each of the 11 selected MTFs were outside the drive-time standards, based on our analysis. In addition, for each of the 11 selected MTFs, there was one or more inaccuracies in the provider listing, such as providers that were no longer in practice, duplicate providers, or those that were mischaracterized as a medical provider. MTF officials we interviewed also expressed concerns that the assessments did not account for traffic, including bridges and tunnels that create traffic chokepoints. In other words, they believed that even providers that appeared to be within drive- time standards based on mileage could actually exceed the standard depending on their location and time of day. Appendix III illustrates the results of our analysis in detail. DOD guidance states that beneficiaries should have a choice of health care providers that is sufficient to ensure access to appropriate, high- quality health care. In addition, Standards for Internal Control in the Federal Government require the use of quality information that is appropriate, current, complete, accurate, accessible, and timely to inform decisions. Such standards also require that an agency’s management define objectives clearly to enable the identification of risk and risk tolerances, to include defining objectives in specific and measurable terms to allow for the assessment of performance toward achieving objectives. Applied to DOD’s analysis of civilian health care available in proximity to MTFs, such information would include (1) health care quality and (2) accurate and complete access-to-care data for civilian providers identified in its assessments. DOD’s assessments were missing complete and accurate information about the adequacy of purchased care through civilian health care providers because 703 Work Group officials stated that their analyses were detailed enough for the purposes of decision-making about restructuring. Furthermore, they stated that they plan to “test” the purchased-care networks of civilian providers during the transition of MTFs to their restructured end states. Officials stated they believe such a test will reveal that the supply of providers will increase over time to meet an increased demand for care from DOD beneficiaries. However, recent research has reported concerns about growing nationwide shortages of physicians, including primary care providers—a type of civilian health care provider that will be in high demand from DOD beneficiaries as MTFs restructure. For example, a 2019 study projected physician demand will continue to grow faster than supply, leading to a projected nationwide shortfall of between 46,900 and 121,900 physicians by 2032. DOD officials stated they expect to monitor health care quality and patients’ access during the implementation of MTF transitions. While this will be a positive step, a better understanding of the quality of civilian health care providers and patients’ access to an adequate supply of such providers within drive-time standards could help DOD in its implementation planning for MTF transitions and its tests of network capabilities by illustrating areas of highest risk. Until DOD consistently captures and assesses information about the quality of available civilian health care and the extent to which such care has met and will continue to meet patients’ access standards, DOD leaders may not fully understand risks to the achievement of their objectives in restructuring future MTFs. DOD applied a single set of assumptions in comparing the cost- effectiveness of direct care delivered at MTFs to that of purchased care, as previously discussed. On the basis of our analysis of the assumptions and related data elements, and interviews with DOD officials, we found that the assumptions do not account for uncertainties that could affect conclusions about an MTF’s cost-effectiveness. Specifically, DOD made assumptions about the costs of military personnel salaries, the workload performed at MTFs, and the reimbursement rates for TRICARE that individually and collectively likely result in the underestimation of the cost- effectiveness of MTFs, as described in more detail below. Including the full cost of military medical personnel does not account for their value outside of MTFs in support of military operations. DOD included the full cost of active-duty medical personnel salaries when calculating the unit-level cost of MTF health care. This approach assumed that military personnel spend all their time in MTFs. However, military personnel who staff MTFs sometimes spend half or more of their time contributing to other military work activities, according to MHS officials. These personnel are essential for military operations outside the MTFs. Accordingly, DOD referred to its medical personnel as a “fixed cost” in the Modernization Study. In its interim report to Congress for section 721 of the NDAA for Fiscal Year 2017, DOD determined that about 111,000 active-duty personnel are essential to support its war plans as part of the operational medical force. By including the full cost of military personnel salaries in calculations of the unit-level cost of MTF-provided care, DOD has likely underestimated the cost-effectiveness of MTFs given the dual purpose of active-duty medical personnel who staff MTFs but spend time on other military duties and deploy to support operations. According to MTF officials, some portion of the cost of military personnel salaries could be considered an approximation of the “cost of medical force readiness” for the wartime mission, though an imperfect one. Units of health care may underreport workload performed at MTFs. DOD calculated the cost of delivering a single unit (e.g., wRVU), of care at its MTFs. Doing so likely underestimates the cost- effectiveness of MTFs given concerns that wRVUs may be underreported. MTF officials at all 11 locations and 703 Work Group members agreed that wRVUs are likely underreported within MTFs for various reasons. For example, some MTF services are not recorded in wRVUs, such as telehealth consultations, which comprise a growing share of patient encounters, according to MTF officials. In addition, in February 2019 we reported that source data for DOD’s clinical readiness metric for physicians—the same data MTFs use to record wRVUs—had not passed DOD audits for at least 3 years. Likewise, in April 2016, we reported concerns that providers’ workload at MTFs was not being accurately recorded. TRICARE reimbursement rates for purchased care will likely need to increase. In comparing the cost-effectiveness of direct care at MTFs to purchased care from civilian providers in the TRICARE networks, DOD applied current TRICARE reimbursement rates in its calculations. MTF and 703 Work Group officials, along with senior MHS leaders, agreed that DOD may need to pay higher reimbursement rates in the future to attract new, quality network providers as its reliance on purchased care for beneficiaries increases in proportion to the decrease in access to health care services at many MTFs. In addition, MTF officials and MHS leaders stated that utilization of some purchased-care services from civilian providers may be higher than utilization of like services at MTFs because civilian providers are not incentivized to manage health services and costs the way the MHS does. This means that the cost of purchased care could increase by more than expected if utilization rates increase. For example, a research study completed in 2017 found that an estimated 21 percent of purchased medical care in the United States is attributed to unnecessary costs associated with overtreatment. In 2010, the Institute of Medicine reported that unnecessary services are the largest contributor to waste in the U.S. health care system, and could have accounted for about $210 billion in excess spending in 2009. By applying a single set of assumptions as described above, DOD’s assessment of the cost-effectiveness of MTFs was not consistent with a key practice in economic analysis. Our assessment methodology for economic analysis states that a sensitivity analysis is an essential element of a high-quality analysis of cost-effectiveness. Likewise, a DOD instruction on economic analysis states that analyses of investment alternatives should include, among other things, a sensitivity analysis, accounting for uncertainties by testing the sensitivity of the economic analysis results against various factors. A sensitivity analysis examines the effects that changes to key assumptions have on the analytic outcome and are helpful to understanding risk. To demonstrate the effect of a single set of assumptions versus an analytical approach that explored other assumptions, we adjusted some of the assumptions for illustrative purposes. Our analysis found that for two of seven MTFs we evaluated in detail, changing DOD’s assumptions in only one respect—by subtracting military personnel salaries—would have materially affected DOD’s assessment about whether direct care at the MTF was more cost-effective than purchased care. Further, if military personnel salaries are excluded from the assessments and TRICARE reimbursement rates increase by 5 percent, three of the seven MTFs would be more cost-effective than purchased care. For illustrative purposes, figure 2 shows how alternative assumptions could change both the data (i.e., costs and wRVUs) and the results of the assessments as to whether an MTF is more or less cost effective than purchased care. According to officials from the 703 Work Group, they did not apply alternative assumptions to analyze cost-effectiveness because readiness and the adequacy of civilian health care were more important in their methodology, and they generally assumed that purchased care is less costly. However, DOD could still maintain its prioritization sequence while augmenting its cost-effectiveness analyses with a sensitivity analysis to help provide more complete information for decision-making and, in the future, for executing MTF transitions. Without doing so, DOD leaders may further jeopardize their understanding of risks to the achievement of their objectives in restructuring future MTFs. Through its section 703(d) Plan to Congress, DOD has identified actions that will be needed to facilitate MTF restructuring. These actions include 17 recommendations for enterprise-wide changes across MTFs, and various MTF-specific steps to mitigate risks at a local level. However, DOD’s Plan also poses challenges for the military departments and the DHA related to medical provider readiness and MTF staffing. DOD does not have a process for monitoring MTF restructuring transitions to address these challenges. Through its Plan, DOD has taken preliminary steps toward transition planning by identifying actions needed to facilitate the restructuring and MTF transitions. Specifically, in the Plan DOD (1) recommended certain actions across the collective enterprise of MTFs to facilitate their restructure to reduced health-care delivery capabilities, and (2) identified risks and potential mitigation strategies specific to each MTF identified for restructuring. According to the Plan and 703 Work Group officials, DOD will begin to plan these transitions in detail after a congressional review period is completed in May 2020. Enterprise-wide actions for transition planning for restructured MTFs. In its plan, DOD recommended 17 enterprise-wide actions to facilitate MTF restructuring transitions. The Plan noted that the actions apply across various installations and MTFs, and were not specific to any certain region, military service, or population size. According to 703 Work Group officials, these actions will be critical to the successful transition of MTFs to their restructured end states. We found that the actions described in the Plan are interdependent and have implications for military readiness, the adequacy of civilian health care in proximity to MTFs, and the cost-effectiveness of MTF health care, which are discussed throughout our report. Moreover, we found that the recommendations require actions and coordination from multiple organizations and stakeholders. For example, the Plan recommends structuring health care operations to support patients from the military’s Exceptional Family Member Program in relevant markets. The military departments are responsible for oversight of this Program, and their coordination with the DHA, MTF officials, and with military commands will be needed to ensure those patients’ medical needs are met. This and the other 16 actions are listed below in table 3, along with the stakeholders who may be needed to implement them. MTF-specific actions for transition planning. A second step the 703 Work Group has taken toward preliminary transition planning is to identify, for each MTF, certain salient risks and potential mitigation strategies. The Work Group documented these risks and mitigations in the “Use Case” for each MTF, which are included in appendices to the Plan. The “Use Cases” summarize, among other things, the recommended restructuring actions and the related analytical basis. Specifically, the “Use Cases” list risks and related mitigation strategies, noting that the lists summarize observations from the Work Group’s analyses but are not exhaustive. For some risks, the “Use Cases” noted that a mitigation strategy should be determined later. For example, a risk associated with patients’ changes in expectations—from getting health care at the MTF to getting care outside the base (from a civilian provider)— will need to be monitored and managed. Other risks and mitigation strategies identified in the MTF “Use Cases” are specific to an MTF’s concerns based on local considerations, such as the health care services they deliver, the type of active-duty population they serve, and their knowledge of the nearby civilian health care providers. For example, at Langley Air Force Base, where DOD is recommending that the hospital transition to an ambulatory surgery center (which would not have inpatient care or an emergency room), the “Use Case” for the MTF notes that the elimination of inpatient capabilities would decrease the MTF’s support to readiness. This means that future numbers and types of patients and health care services delivered at Langley’s MTF, once it becomes an ambulatory surgery center, may not sustain the clinical readiness requirements of the active-duty medical personnel assigned to work there—requirements that they must meet for deployments. Accordingly, the “Use Case” notes that a related mitigation strategy would create partnerships across area hospitals where Langley’s medical personnel may be able to supplement their MTF workload and maintain their readiness. As another example, the “Use Case” for Fort Polk’s MTF in Louisiana— where DOD is recommending the MTF maintain inpatient care in the short term but monitor the expansion of local hospitals to determine when inpatient services can be replaced with purchased care—noted several specific risks related to patients’ access to care given the remote, rural location. One risk pertains to labor and deliveries, and the “Use Case” noted that the two nearest off-base hospitals in the TRICARE network have had problems with fiscal solvency and there is an insufficient number of obstetricians. Accordingly, DOD’s mitigation plan is for the MTF to initially maintain pre- and postnatal care services to expectant mothers, while (1) monitoring the TRICARE network hospitals and (2) ensuring the MTF’s obstetricians have privileges at those hospitals for labor and delivery until the number of network obstetricians is sufficient. Despite DOD’s preliminary steps toward transition planning, including the mitigation strategies for risks previously discussed, its Plan poses other challenges for the military departments and the DHA in executing MTF restructuring. In particular, our review highlighted two interrelated challenges with respect to medical providers’ clinical readiness and MTF staffing levels. Military departments’ efforts to maintain the clinical readiness of primary care and nonphysician medical providers. As discussed earlier in this report, MTF officials stated that MTFs that restructure will not likely present the diverse and complex medical conditions needed to sustain the readiness of military primary care and nonphysician providers. A senior enlisted leader at one MTF observed that the staff would “have to be creative” to find the right mix of opportunities for enlisted personnel to gain the clinical experience they need to be ready to deploy as the clinic transitions to seeing only active-duty patients. MHS leaders we interviewed agreed with these concerns. To address MTF and MHS leaders’ concerns, 703 Work Group officials stated that the MHS would need to develop better metrics for primary care and nonphysician providers’ clinical readiness requirements, as the MHS has done for combat casualty care physicians. The officials also stated that another mitigation plan will be to allow MTFs that become active-duty clinics to diversify the patient population available to providers by treating some family members and retirees. MTF officials we spoke with were encouraged that continuing to treat some family members and retirees could help address the provider readiness shortfalls they believe are inherent to becoming an active-duty clinic. However, they and senior MHS leaders were concerned about the prospect of differentiating among such beneficiaries in terms of who may be eligible for MTF care at an active-duty clinic. To that end, officials stated that having the DHA clarify its roles and responsibilities in executing this flexibility will be a helpful step. To address challenges in maintaining the clinical readiness of medical providers assigned to MTFs that restructure, 703 Work Group officials stated that existing MHS partnerships with civilian hospitals and the Department of Veterans Affairs should be sufficient for MTF providers along with other available mechanisms, such as temporary duty at other MTFs. However, MHS leaders stated that existing civilian partnerships, in particular, may not have sufficient capacity to take on additional military medical personnel. As a result, the leaders believe they may need to expand partnerships to accommodate the expected increase in demand from military providers for on-the-job readiness training as MTF capabilities decrease during restructuring transitions. Furthermore, MTF officials we interviewed had mixed opinions about the readiness benefits they derived from their experiences with civilian hospital partnerships and training at other MTFs. DHA and the military departments’ ability to fully staff the MTFs. According to MTF officials, sending their medical providers to work outside their assigned MTF in support of clinical readiness, though temporary, creates another challenge by reducing providers’ availability to the MTFs. As more providers require such experience due to MTF capabilities’ decrease from restructuring, MTF officials we interviewed noted that staffing gaps could complicate their ability to execute the transitions and ensure the continuity of care for patients. Furthermore, MTF officials stated that active-duty medical personnel reductions that occurred in fiscal year 2019 have also created shortfalls in staffing that could pose challenges for them in executing the MTFs’ transitions. According to these officials, this is because they expect their administrative workload to increase while transitions are ongoing, while clinical workload for patient care would not decrease soon enough to mitigate any shortfalls in providers. DOD’s Plan states that the DHA should collaborate with the military departments to establish standard staffing models to facilitate MTF transitions, and transition plans must specify reductions in personnel and resources for the future state of the MTFs. However, the continuation of a phased transfer of MTF administration and management to the DHA from the military departments may present challenges to the DHA’s ability to concurrently accomplish new tasks related to restructuring the MTFs and facilitating their transitions. Likewise and more broadly, we reported in November 2018 that the transfer of MTF administration and management to the DHA may present challenges to the management of military personnel given that the military departments are responsible for medical personnel readiness, not the DHA, while DHA assumes responsibility for staffing the MTFs. DOD has not established a process for monitoring MTF restructuring transitions to address the aforementioned challenges. Yet, the MHS plans to move forward with restructuring actions beginning in June 2020. While officials expect that transitions of certain smaller clinics to their restructured end state may be relatively simple, they acknowledged that other MTF transitions could be complex and take several years. According to the ASD(HA) and Work Group officials, DOD will readjust its plans by reversing or slowing an MTF transition, if needed, to address any challenges that arise with ensuring patients’ ability to access health care—one of the restructuring objectives. DOD’s Plan does not discuss conditions that would warrant slowing or reversing an MTF’s restructure, or how that would be determined. According to senior MHS leaders and MTF officials, the potential need to reverse or slow transitions will make monitoring the transitions important, and they are awaiting such decisions, along with associated roles and responsibilities from the DHA. However, the Plan does not establish a process for monitoring MTF restructuring transitions, as this was not within the scope of efforts, according to 703 Work Group officials. Rather, officials stated that decisions about monitoring should occur in a next phase of execution for MTF transitions after completion of the Plan. Accordingly, after DOD submitted its Plan to Congress in February 2020, the ASD(HA) issued a memorandum tasking the Director of the DHA to implement the changes specified in the Plan (i.e., the MTF restructuring actions) and providing high-level guidance. For example, the memorandum states that: MTF transitions are not authorized to start before May 19, 2020 (i.e., 90 days after the Plan was provided to Congress) but should be completed no later than October 1, 2025; transition planning may begin at the DHA Director’s discretion (but not later than the beginning of fiscal year 2021) and should include all impacts from ongoing personnel reductions and realignments; and detailed transition plans should include clear mechanisms for stakeholder tracking of activities and progress, and be arranged in a manner that addresses the needs of multiple stakeholders from the local to the national levels. Regarding the transition plans, the memorandum requested that that the DHA Director provide the ASD(HA) with a point of contact within 5 days, and a timeline, milestones, initial resource requirements, and task organization for the effort within 2 weeks—i.e., by February 26 and March 6, 2020, respectively. The DHA missed these milestones, having not yet provided the requested information, although an official from the Office of the ASD(HA) stated that, as of March 2020, the DHA response was being drafted. MHS reform and the DHA’s progress in achieving goals are longstanding challenges on which we have previously reported. In April 2012, before the DHA was established, we reported that DOD did not consistently employ key management practices in implementing initiatives to change its MHS governance structure. We recommended that the ASD(HA) and the Surgeons General implement a monitoring process across DOD’s portfolio of initiatives for overseeing progress and identify accountable officials and their roles and responsibilities for all of its initiatives. DOD implemented this recommendation by assigning each initiative a working group, an initiative leader, and executive sponsor to help ensure that the initiative remained on schedule, on budget, and achieved performance goals. After DOD established the DHA, we reported in November 2013 and later in September 2015 on its progress. In both reports, we identified deficiencies and made recommendations to provide decision makers with more complete information on the implementation, management, and oversight of the DHA. DOD concurred with the 10 related recommendations and implemented all but one. We reported in March 2004 that a process for monitoring progress is key to successful results-oriented management. However, DOD does not have such a process for the MTF restructuring transitions, in part because MHS officials stated they would first need to establish detailed roles and responsibilities for executing the transitions generally. Beyond the DHA Director’s role of transition leader, other roles and responsibilities have not been established, such as what involvement MTF officials will have in monitoring and tracking progress or challenges, and how the military departments will share responsibilities with the DHA. The Senior Military Medical Advisory Council could sufficiently monitor the transitions at a high level, according to the DHA Director. Other MHS leaders we spoke with believed that involvement from additional military department and Office of the Secretary of Defense leaders could also be needed. As we reported in October 2005, agreement on roles and responsibilities is a key step to successful collaboration when working across organizational boundaries, such as the military services. Committed leadership by those involved in the collaborative effort, from all levels of the organization, is needed to overcome the many barriers to working across organizational boundaries. Our prior work has also shown that a dedicated team vested with necessary authority and resources to help set priorities, make timely decisions, and move quickly to implement decisions is critical for a successful transformation. DOD also has not defined objectives in a measurable way with related thresholds and goals to enable monitoring of progress and challenges. For example, as previously discussed, DOD’s three general priorities, or objectives, for restructuring MTFs include ensuring (1) the medical readiness of servicemembers and readiness of medical providers, (2) that civilian health care facilities and providers adequately support the health care needs of beneficiaries near each MTF, and (3) the cost-effectiveness of MTF and purchased care. However, DOD has not decided how to define and measure any of those objectives. Furthermore, DOD has not established thresholds or goals in relation to the objectives. By first establishing clear roles and responsibilities for executing and monitoring restructuring transitions, DOD can be better positioned to navigate and overcome organizational boundaries between the DHA, which manages the MTFs, and the military departments that provide staff. In doing so, DOD could also be better positioned to address challenges in executing transitions, such as those that arise with mitigating providers’ clinical readiness challenges and MTF staffing gaps during transitions. Then, by defining measurable objectives, goals, and thresholds for tracking the progress of MTF transitions—such as the clinical readiness of providers, quality and accessibility of quality health care, and cost- effectiveness—DOD could better ensure its objectives are being met and help facilitate timely adjustments to the transitions, as needed. As MHS leaders have acknowledged, correctly aligning MTF infrastructure to the size of the armed forces, the medical forces, and their desired readiness levels is essential to balancing mission requirements within available resources. DOD’s substantial work over the past 2 years on its Plan for MTF restructuring is a positive step toward meeting statutory requirements and prioritizing MTF readiness outcomes in a resource-informed manner. Notwithstanding the work DOD has undertaken in making a series of analytically-based determinations for restructuring in its Plan, our review highlighted several gaps in DOD’s methodology. Until DOD takes action to address these gaps by using more complete and accurate information about civilian health care quality, access, and cost-effectiveness, DOD leaders may not fully understand risks to the achievement of their objectives in restructuring future MTFs. DOD officials agree that some MTF restructuring actions may be more challenging than others. These challenges could be exacerbated by concurrent MHS reform efforts, including the transition of MTF administration and management to the DHA. However, by establishing clear roles and responsibilities for executing and monitoring the transitions, DOD can be better positioned to overcome the difficulties in navigating organizational boundaries between the DHA and the military departments, and make timely adjustments to their transition plans, as needed. In addition, by defining measurable objectives, thresholds, and goals for restructuring transitions, and applying them to evaluate progress and challenges, DOD can be better positioned to execute the transition of its MTFs and ensure that the objectives are being met. We are making the following six recommendations to DOD: The Secretary of Defense should ensure that the Assistant Secretary of Defense for Health Affairs, in coordination with the Surgeons General of the military departments and the Director of the DHA, consistently collect complete and accurate information about the quality of available civilian health care in proximity to its MTFs (such as ratings from the Centers for Medicare and Medicaid Services and perceptions from MTF officials who regularly coordinate with civilian providers, among other means) and assess that information to inform recommendations for future MTF restructuring decisions. (Recommendation 1) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Health Affairs, in coordination with the Surgeons General of the military departments and the Director of the DHA, consistently collect complete and accurate information about the extent to which current health care providers within the TRICARE networks meet access-to-care standards, and assess that information to inform recommendations on future MTF restructuring decisions. (Recommendation 2) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Health Affairs, in coordination with the Surgeons General of the military departments and the Director of the DHA, consistently collect complete and accurate information about the extent to which non-network civilian health care providers that could be incorporated into the TRICARE network meet access-to-care standards in terms of drive time, and assess that information to inform recommendations on future MTF restructuring decisions. (Recommendation 3) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Health Affairs, in coordination with the Surgeons General of the military departments and the Director of the DHA, conducts a sensitivity analysis of the relative cost-effectiveness of MTF-provided care compared to civilian-provided care under varying assumptions, and document that information for decision makers to inform recommendations on future MTF restructuring decisions. Varying conditions could include different types of health care services, reducing the cost of military personnel salaries, and increasing estimated MTF wRVUs and civilian reimbursement rates. (Recommendation 4) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Health Affairs, in coordination with the Surgeons General of the military departments and the Director of the DHA, establishes clear roles and responsibilities for executing and monitoring transitions for MTFs identified for restructuring. (Recommendation 5) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Health Affairs, in coordination with the Surgeons General of the military departments and the Director of the DHA, defines measurable objectives for MTF restructuring transitions, establishes thresholds and goals for each objective, and applies them to evaluate progress and challenges. For example, measurable objectives, thresholds, and goals, should include an evaluation of medical providers’ clinical readiness, civilian health care provider adequacy, and the cost-effectiveness of MTF and purchased care. (Recommendation 6) We provided a draft of this report to DOD for review and comment. In its written comments, reproduced in appendix IV, DOD concurred with two of our recommendations and partially concurred with four recommendations. DOD also provided technical comments on the draft report, which we incorporated as appropriate. DOD concurred with our recommendations to establish roles and responsibilities for executing and monitoring MTF restructuring transitions (recommendation 5), and to define measurable objectives with thresholds and goals and apply them to evaluate progress and challenges for the transitions (recommendation 6). In its response, the department described actions it is taking and plans to take to implement both recommendations. DOD partially concurred with our first recommendation—to collect complete and accurate information about the quality of available civilian health care in proximity to its MTFs and assess that information to inform recommendations for future MTF restructuring. DOD stated that complete and accurate information on the quality of available care would require substantial resources to accomplish on a routine basis. To that end, DOD stated that until standardized quality data becomes readily available, it intends to collect this level of information as needed to support actions at a particular MTF. As noted in our report, we have previously reported that standardized information about hospital and outpatient care quality is available through the Centers for Medicare & Medicaid Services and has been widely adopted by major private insurers. As the restructuring of the MTFs continues and DOD relies to a greater extent on civilian-provided care, it will be important for the department to monitor the quality of care it purchases on behalf of beneficiaries. Thus, we continue to believe that DOD should make it a priority to collect and assess such information. DOD partially concurred with our second recommendation—to consistently collect complete and accurate information about the extent to which current health care providers within the TRICARE networks meet access-to-care standards, and assess that information to inform recommendations on future MTF restructuring decisions. In its response, DOD stated that each month, TRICARE contractors report, by specialty, average wait times from referral placement to patient appointment. Further, DOD stated that it is piloting centralized booking of MTF and network appointments, which, if successful, will result in more complete, accurate, and timely network access information. In cases where access standards are not being met, DOD explained that it works to mitigate the access shortfall either through MTF or expanded network resources. We agree that TRICARE’s monthly reports on patient wait times for appointments are a helpful tool for DOD in monitoring access to care, and that the pilot for centralized appointment booking is also a promising step. As we noted in our report, however, DOD’s analyses of the adequacy of civilian health care in proximity to MTFs were based on network provider directories that are of questionable accuracy and can overstate the number of available providers. MTF officials we interviewed stated that TRICARE directories in their area overstated the number of providers accepting new TRICARE patients. Even if the provider directory issues have not led to access-to-care challenges in the past in terms of patients’ wait times to appointments, such issues could cause challenges in the future with increasing numbers of DOD patients needing TRICARE network care. Accordingly, we continue to believe that it will be important for DOD to collect complete and accurate information about the extent to which current health care providers within the TRICARE networks meet access-to-care standards as DOD moves forward with its restructuring plans. DOD partially concurred with our third recommendation—to consistently collect complete and accurate information about the extent to which non- network civilian health care providers that could be incorporated into the TRICARE network meet access-to-care standards in terms of drive time, and assess that information to inform recommendations on future MTF restructuring decisions. DOD stated that drive times for non-network providers were assessed in the development of the recommendations for its Plan. DOD added that the approach used in the Plan included assessing drive times and distances from the beneficiaries’ homes, rather than the MTF, yielding a more accurate assessment of access, availability, and convenience. However, our review of DOD’s methodology workpapers showed that its analyses measured the distance between providers and a single location point that corresponds with the center of the zip code boundary in which a majority of beneficiaries reside. While a perfect methodology and information for projecting actual drive times may not be possible to achieve, the alternative method in our report illustrates that a portion of the providers DOD identified for potential network expansion would exceed access-to-care standards for some of its beneficiaries—especially those who live or work near an MTF (such as those who live on a military installation). According to DOD’s comments on the report, future restructuring efforts will be informed by the section 703 approach, and DOD will adjust the approach as needed by future analysis and conditions. As DOD moves forward with restructuring efforts in the future, we continue to believe that more accurately measuring the distance between providers’ locations and beneficiaries’ residences would improve the quality of DOD’s information about access-to-care. Accordingly, we continue to believe that DOD should fully implement our recommendation. DOD partially concurred with our fourth recommendation—to conduct a sensitivity analysis of the relative cost-effectiveness of MTF-provided care compared to civilian-provided care under varying assumptions, and to document that information for decision makers to inform recommendations on future MTF restructuring decisions. In response, DOD stated that there is value in the use of sensitivity or scenario analysis to inform decisions where a range of possibilities exist and a clear analytical question can be formed as a guide to both the analysts and decision-makers. However, DOD stated that it does not support the generic use of this analysis suggested by the recommendation. We disagree that our recommendation suggests the “generic use” of a sensitivity analysis. A sensitivity analysis is appropriate for evaluating restructuring opportunities for MTFs for two reasons. First, the evaluation of each MTF presents decision makers with a range of possibilities—from reducing or expanding the capabilities of an MTF, to closing it entirely or maintaining the status quo. Second, a sensitivity analysis would address uncertainties in DOD’s analytic assumptions about costs and workload for each MTF, which our report identified. As our own sensitivity analysis— conducted using minimal resources and available DOD data— demonstrated, changing the assumptions also changes the resulting conclusions about whether MTF or civilian care is less expensive. Therefore, we continue to believe that analyzing the relative cost- effectiveness of MTF-provided care compared to civilian-provided care under varying assumptions would provide more complete information for decision-making and executing MTF transitions. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense for Personnel and Readiness, the Assistant Secretary of Defense for Health Affairs, the Director of the Defense Health Agency, and the Secretaries of the Army, the Navy, and the Air Force. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. In February 2020, the Department of Defense (DOD) submitted its implementation plan (“the Plan”) to Congress in response to section 703(d) of the National Defense Authorization Act (NDAA) for Fiscal Year 2017. This report addresses the extent to which (1) DOD’s methodology for determining military medical treatment facility (MTF) restructuring actions in the Plan prioritized cross-cutting elements from 10 U.S.C. § 1073d and considered complete information, and (2) DOD has positioned itself to execute transition planning for restructuring its MTFs. For both objectives, we selected a nongeneralizable sample of MTFs and applied a case study approach to review DOD’s methodology for determining restructuring actions and actions that may be needed for transition planning. From DOD’s initial list of 73 MTFs included in its scope, we selected 11 of them as case studies to represent a variety of characteristics, including a mix of hospitals and clinics from each military department, different recommendations for how they should be restructured, different conclusions about network adequacy, and urban and rural areas located in proximity to one another in terms of driving distance. Appendix II identifies the names and locations of each MTF within the scope of the DOD Plan. The 11 MTFs we selected are listed in table 4. For objective one, we reviewed DOD’s draft and final Plan and related documentation of the methodologies used to assess all 77 MTFs within its scope. We compared this information with cross-cutting elements for MTFs from 10 U.S.C. § 1073d. These elements include the (1) support an MTF provides to servicemembers’ medical readiness and the readiness of medical personnel, (2) adequacy of civilian health care facilities and providers in the proximity of the MTF to support the health care needs of servicemembers and other beneficiaries through purchased care, and (3) cost-effectiveness of direct care services at MTFs versus purchased care in the nearby civilian provider networks. We discussed the methodological approaches for assessing MTFs for possible restructuring actions, including assumptions, data sources, and any limitations, with representatives from relevant organizations across DOD, including the Office of the Assistant Secretary of Defense for Health Affairs; DOD’s Section 703 Work Group; Defense Health Agency; Office of the Assistant Secretary of the Army for Manpower and Reserve Affairs; U.S. Army Medical Command; Navy Bureau of Medicine; Air Force Medical Readiness Agency; and officials from 11 selected MTFs and from their host installations. Specifically, in analyzing DOD’s methodology for assessing MTFs’ support to readiness, we reviewed information DOD used to estimate MTFs’ readiness value in terms of support to servicemembers’ medical readiness and to medical force readiness. We compared this information with findings from our prior work regarding DOD’s methodology for assessing clinical readiness. We also reviewed records of interviews that DOD officials held with MTF, installation, and command officials during their visits to MTFs, noting the readiness-related effects and concerns that were documented. For DOD’s methodology for assessing available civilian health care services in proximity to each MTF, we reviewed reports on the results of DOD’s assessments to identify their findings, recommendations, and assumptions. For the civilian health care providers that DOD identified in proximity to each of 11 MTFs we selected, we verified the address of each listed provider by searching for each provider’s website and making phone calls to verify addresses and specialty types, and whether the practice was open or closed. We then used R software and data from openstreetmap.org to calculate to calculate the driving distance between the provider and the MTF, comparing the distance with DOD’s access-to- care standards. We evaluated the extent to which the assessment reports considered information about quality of health care services and access-to-care standards, comparing the information with DOD guidance for patients’ access to quality and timely health care services, and with federal internal control standards on the use of quality information to inform decision-making. Regarding DOD’s methodology for evaluating cost-effectiveness, we reviewed DOD’s work papers and interviewed officials about the calculations and source data they used. We compared DOD’s methodology with our assessment methodology for economic analysis and with DOD guidance for economic analysis. We also obtained the fiscal years 2017 and 2018 data DOD used to calculate the cost- effectiveness of MTF-provided direct care relative to civilian-provided purchased care. Using these data, we performed a sensitivity analysis by recalculating relative cost-effectiveness under different assumptions. Specifically, for seven of our 11 case study MTFs, we recalculated their cost-effectiveness relative to purchased care by (1) omitting military personnel salaries, given that DOD has characterized these as a fixed cost, (2) increasing the work Relative Value Units to adjust for potential underreporting of those data, and (3) increasing the reimbursement rate of purchased care to account for future increases that are likely necessary, according to DOD officials. We also assessed the reliability of each data source for DOD’s and our calculations of cost-effectiveness by administering questionnaires about the data to those who have quality control responsibilities, interviewing responsible DOD officials, reviewing the data for outliers and missing values, and reviewing our prior reports about the data. We determined that DOD’s data on the costs of MTF care and civilian health care were sufficiently reliable for the purpose of calculating the total costs of health care services. However, DOD’s data on units of health care delivered in fiscal year 2018 were of undetermined reliability for the purpose of calculating a unit-level health care cost. To provide additional information on DOD’s methodology and supplement our understanding of available data, we conducted a literature review of research articles. We conducted a search of the literature on military health system clinical readiness, trends in physician supply and demand across the United States, and cost analyses of military health care published from 2014 through 2019 to identify articles on key challenges and methodological alternatives. To identify relevant articles, we searched a variety of databases with the assistance of a research librarian, limiting our review to papers that were included in peer-reviewed publications, as well as government reports, trade and industry articles, and publications by associations, nonprofits, or think tanks. We then reviewed the results and excluded any that were technical in nature or did not have wide applicability across MTFs or to health care analyses. For objective two, we reviewed DOD’s draft and final Plan, including detailed appendices on the MTFs within the scope of the plan, noting any aspects of transition planning described or recommended, and the agencies and organizations that would be responsible for managing those transition aspects. We also interviewed MTF officials from the selected case study locations regarding steps they had begun taking and actions they believed would be needed to facilitate a restructuring of the facility. In addition, we reviewed our prior, related work on MHS reform and the establishment of the Defense Health Agency to identify related themes and challenges. We corroborated our understanding of transition planning steps described in the Plan by interviewing 703 Work Group officials, the Director of the Defense Health Agency, and the Surgeons General of the Army and the Air Force to better understand what roles and responsibilities and monitoring mechanisms they had considered. We compared this information from the Plan and from our interviews with practices identified in our prior work on results-oriented government. Specifically, our prior work has found that agreement on roles and responsibilities and committed leadership by those involved are key steps to successful collaboration when working across organizational boundaries. A dedicated team vested with necessary authority and resources to help set priorities, make timely decisions, and move quickly to implement decisions, along with a process for monitoring progress, are also key to success. Finally, to assess the extent to which DOD’s section 703(d) Plan addressed all requirements of section 703(d), we compared the Plan with the elements from the statute. Examples of those elements included, for each MTF, whether it will be restructured, whether its functions will be expanded or consolidated, and the related costs. Some of the elements required that multiple items be addressed. We considered an element “addressed” if it included all of the items listed in the NDAA; “partially addressed” if it included some, but not all, of the items; and “not addressed” if it did not include any of the items. We conducted this performance audit from February 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. A work group of representatives from across the Department of Defense (DOD) led efforts to address section 703(d) of the National Defense Authorization Act for Fiscal Year 2017—the restructure or realignment of military medical treatment facilities (MTF). The 703 Work Group developed a methodology to address section 703(d) and determined the scope of its review of MTFs in the United States by identifying which of those to evaluate for the mandated implementation plan (the “Plan”). Figures 3 through 5 below identify the name and location of each of the 77 MTFs within the scope of DOD’s Plan, which it submitted to Congress in February 2020. The Department of Defense’s (DOD) 703 Work Group based its military medical treatment facility (MTF) restructuring determinations for its implementation plan to Congress, in part, on its assessments of the adequacy of civilian health care facilities and providers to support the health care needs of DOD beneficiaries near each MTF. In one component of the assessments, DOD identified civilian health care facilities and providers in proximity to each of its 77 evaluated MTFs. For each provider DOD identified in proximity to 11 MTFs—which we selected from the 77 MTFs DOD evaluated—we verified the provider’s address, specialty type, and whether the practice was open or closed. We then calculated the driving distance between each MTF and the respective listed providers. Figures 6 through 16 below show that a certain portion of the providers listed for each of the 11 selected MTFs were outside DOD’s access-to-care standards for travel time to provider sites for TRICARE Prime patients, based on our analysis. In addition, for each of the 11 selected MTFs, there was one or more inaccuracies in the provider listing, such providers that were no longer in practice, duplicate providers, or those that were mischaracterized as a medical provider. In addition to the contact named above, Lori Atkinson (Assistant Director), Melissa Blanco (Analyst in Charge), John Beauchamp, Timothy Carr, Alexandra Gonzalez, Hannah Hubbard, David Jones, Amie Lesser, John Mingus, Jr., Oliver Richard, Terry Richardson, Guiovany (Geo) Venegas, and Lillian M. Yob made key contributions to this report. Defense Health Care: Opportunities to Improve Future TRICARE Managed Care Support Contract Transitions. GAO-20-39. Washington, D.C.: November 21, 2019. Defense Health Care: DOD’s Proposed Plan for Oversight of Graduate Medical Education Programs. GAO-19-338. Washington, D.C.: March 28, 2019. Defense Health Care: Actions Needed to Determine the Required Size and Readiness of Operational Medical and Dental Forces. GAO-19-206. Washington, D.C.: February 21, 2019 Defense Health Care: Additional Assessments Needed to Better Ensure an Efficient Total Workforce. GAO-19-102. Washington, D.C.: November 27, 2018. Defense Health Care: DOD Should Demonstrate How Its Plan to Transfer the Administration of Military Treatment Facilities Will Improve Efficiency. GAO-19-53. Washington, D.C.: October 30, 2018. Defense Health Care: Expanded Use of Quality Measures Could Enhance Oversight of Provider Performance. GAO-18-574. Washington, D.C.: September 17, 2018. Defense Health Reform: Steps Taken to Plan the Transfer of the Administration of the Military Treatment Facilities to the Defense Health Agency, but Work Remains to Finalize the Plan. GAO-17-791R. Washington, D.C.: September 29, 2017. Defense Health Care Reform: DOD Needs Further Analysis of the Size, Readiness, and Efficiency of the Medical Force. GAO-16-820. Washington, D.C.: September 21, 2016. Defense Health Care Reform: Actions Needed to Help Ensure Defense Health Agency Maintains Implementation Progress. GAO-15-759. Washington, D.C.: September 10, 2015. Military Health System: Sustained Senior Leadership Needed to Fully Develop Plans for Achieving Cost Savings. GAO-14-396T. Washington, D.C.: February 26, 2014. Defense Health Care Reform: Additional Implementation Details Would Increase Transparency of DOD’s Plans and Enhance Accountability. GAO-14-49. Washington, D.C.: November 6, 2013. Defense Health Care Reform: Applying Key Management Practices Should Help Achieve Efficiencies within the Military Health System. GAO-12-224. Washington, D.C.: April 12, 2012. Military Personnel: Enhanced Collaboration and Process Improvements Needed for Determining Military Treatment Facility Medical Personnel Requirements. GAO-10-696. Washington, D.C.: July 29, 2010.", "summary": "DOD's MTFs are critical to the medical readiness of servicemembers and providing readiness training for about 107,000 active-duty medical providers. About 9.6 million beneficiaries are eligible for DOD health care through MTFs and civilian network providers. To further support readiness, the National Defense Authorization Act (NDAA) for Fiscal Year 2017 required DOD to plan to restructure MTFs. DOD's February 2020 Plan included decreasing capabilities at 43 MTFs and closing five. The NDAA included a provision for GAO to review the Plan. This report addresses the extent to which 1) the Plan's methodology prioritized statutory elements and considered complete information, and 2) DOD is positioned to execute MTF restructuring transitions. GAO reviewed DOD's Plan, MTF workload and cost data, and interviewed DOD leaders and officials at 11 MTFs selected on the basis of military department, restructuring action, and location. The Department of Defense's (DOD) methodology to determine Medical Treatment Facilities' (MTF) restructuring actions in its implementation plan (the Plan) prioritized statutory elements. These included military readiness, adequacy of nearby civilian health care, and cost-effectiveness. However, DOD based part of its methodology on incomplete and inaccurate information. Civilian health care assessments did not consistently account for provider quality. DOD generally assumed that identified providers were of sufficient quality. GAO found that DOD considered the quality of nearby civilian providers for one of 11 selected MTFs. In this instance, information from the MTF about the variable quality of nearby civilian health care led to DOD's determination that such care was not yet adequate to support MTF restructuring. Officials GAO interviewed from other MTFs discussed concerns about quality of care from nearby civilian providers. Civilian health care assessments did not account for access to an accurate and adequate number of providers near MTFs. DOD may have included in its assessments providers who do not meet DOD's access-to-care standards for certain beneficiaries. For 11 selected MTFs, GAO found that about 56 percent of civilian primary care providers and 42 percent of civilian specialty providers that DOD identified as being nearby exceeded DOD's drive-time standards. Including such providers in its assessments means that DOD could have overestimated the adequacy of civilian health care providers in proximity to some MTFs. Cost-effectiveness assessments were based on a single set of assumptions. DOD concluded that civilian health care was more cost-effective than care in its MTFs without considering other assumptions that could affect its conclusions. For example, DOD applied assumptions about the cost of military personnel salaries, MTF workloads, and reimbursement rates for TRICARE that likely underestimated the cost-effectiveness of MTFs. GAO also found that DOD conducted limited assessments of MTFs' support to the readiness of military primary care and nonphysician medical providers—an issue DOD officials stated they will address during MTF transitions. Until DOD resolves methodology gaps by using more complete and accurate information about civilian health care quality, access, and cost-effectiveness, DOD leaders may not fully understand risks to their objectives in restructuring future MTFs. DOD's Plan identified actions needed to facilitate MTF restructuring, but the department is not well positioned to execute the transitions. DOD's Plan poses challenges for the military departments and the Defense Health Agency (DHA) related to MTF providers' readiness. Yet, DOD plans to move forward with restructuring without a process to monitor progress and challenges. By establishing roles and responsibilities for executing and monitoring MTF restructuring transitions, DOD can be better positioned to navigate organizational boundaries between the DHA that manages the MTFs and the military departments that provide staff. Additionally, by defining measurable objectives and progress thresholds, DOD can better ensure it is meeting objectives and facilitating timely adjustments to MTF restructuring transitions, as needed. GAO is making six recommendations, including that future MTF assessments use more complete and accurate information about civilian health care quality, access, and cost-effectiveness; and that DOD establish roles, responsibilities, and progress thresholds for MTF transitions. DOD partially concurred with four recommendations and concurred with two. As discussed in the report, GAO continues to believe that all six recommendations are warranted.", "document_type": "gao"}
{"report": "Numerous organizations have roles and responsibilities in formulating and executing the Marine Corps’ budget. Specifically: Office of Management and Budget. OMB directs federal agencies, including DOD and the military services, to develop, among other things, civilian personnel budget requests by calculating workload requirements, the time needed to complete the workload and the number of FTEs needed to perform the work for the upcoming fiscal year. Every year, OMB releases an update to OMB Circular A-11, which provides guidance for budget formulation for the upcoming fiscal year. OMB Circular A-11 states that federal agencies should take steps to assess, and as appropriate, restructure, retain, and resize their FTE counts to achieve missions as effectively and efficiently as possible. Office of the Under Secretary of Defense (Comptroller). OUSD(C) issues the DOD’s Financial Management Regulation for budget formulation and execution. The Financial Management Regulation directs statutory and regulatory financial management requirements, systems, and functions for all organizational entities within the DOD. Department of the Navy. According to Department of Navy budget officials, they provide annual budget formulation guidance to all of the budget submitting offices, including the Marine Corps, through the Program Budget Information System, the Navy’s Financial Management & Budget web portal. According to these officials, the guidance contains detailed instructions and updates for budget formulation, worksheets for checking the accuracy of submitted data, and points of contact for budget formulation questions. Marine Corps Deputy Commandant for Programs and Resources (DC P&R). DC P&R is the lead for determining and allocating civilian labor budgets. The DC P&R delegates this responsibility to the Fiscal Director who monitors major subordinate command and Headquarters Marine Corps labor budget execution to ensure compliance with Manage to Payroll budget controls. DC P&R sub-allocates the portion of the budget that will be executed against civilian labor by each major subordinate command and Headquarters Marine Corps. In order to hold the major subordinate commands and Headquarters Marine Corps accountable for exercising prudent Manage to Payroll authority, DC P&R monitors civilian labor execution and issues Mange to Payroll reports on a monthly basis. Marine Corps Director for Civilian Human Resources (DCHR). DCHR provides program direction, technical advice, guidance and assistance to major subordinate commands, Headquarters Marine Corps staff agencies, and servicing human capital resource offices in carrying out Manage to Payroll responsibilities, with regard to position classification. DCHR also holds individuals with delegated classification authority accountable for carrying out effective Manage to Payroll responsibilities, when it comes to classification of position descriptions. In June 2018, the DOD Inspector General reported that, among other things, the Marine Corps did not justify or fully provide supporting documentation for how it determined its civilian personnel pay requirements for the Marine Corps’ fiscal year 2017 budget request. The Department of the Navy concurred with a DOD Inspector General recommendation that the Department of Navy establish and implement controls for the civilian pay budget process. The Marine Corps stated in an unpublished written response that it is reviewing its current budget formulation and other metrics through the fiscal year 2020 budget formulation process, with a plan of implementing an updated budget documentation process for the fiscal year 2021 budget request. The DOD Inspector General also found that the Marine Corps did not determine civilian pay funding levels using FTEs calculated from projected hours to be worked, as required by OMB. As a result, the DOD Inspector General recommended that the Marine Corps determine its budgeted civilian pay funding levels using FTEs calculated based on projected hours to be worked, as required by OMB. In its unpublished written response to the DOD Inspector General report, the Marine Corps stated that it determines FTEs in accordance with OMB and is working to provide greater emphasis on FTEs in its budget documents. See appendix II for more details on the DOD Inspector General’s report. The Marine Corps formulates its annual civilian personnel budget request using prior fiscal year budget execution data as a baseline, then makes adjustments for the upcoming fiscal year based on inputs from various sources like the Department of the Navy and OUSD(C). First, according to Marine Corps officials, the Marine Corps gathers information from several sources to start the budget formulation process including previous fiscal year budget information, information from the Program Objective Memorandum (POM) process, feedback from Marine Corps commands, OUSD(C) guidance, Department of Navy guidance, and the National Security Strategy. To develop its civilian personnel budget request for fiscal year 2020, the Marine Corps used fiscal year 2018 information as a starting point and incorporated changes made through the POM process and other inputs, according to Marine Corps Programs and Resources officials. Second, Marine Corps officials explained during the budget formulation process, the Department of the Navy publishes a Civilian Pricing Tool that financial management and budget officials use to evaluate civilian personnel pricing estimates. The Marine Corps has access to the Civilian Pricing Tool throughout the budget process. After all the inputs are included and the pricing calculations are completed, the Marine Corps officials explained, the Deputy Commandant, Programs and Resources presents the POM to the Commandant of the Marine Corps for approval and inclusion in the Department of the Navy’s overall budget request. Third, according to Department of Navy officials, they conduct a thorough analytic review of each line item’s dollar amount and FTE request in the Marine Corps’ budget request before submitting its total budget to OUSD(C). Once the Department of the Navy has reviewed and approved of the Marine Corps budget request, including its request for civilian personnel funding, the Department of the Navy sends the budget request to OUSD(C) for review. Fourth, according to Department of the Navy officials, OUSD(C) reviews the Marine Corps’ civilian personnel budget request as part of the Department of the Navy’s overall budget request submission, including any change in year-to-year FTE growth, to determine if the Marine Corps properly justified these changes. Once OUSD(C)’s review of the Department of the Navy’s budget submission is complete, the Navy’s budget request is submitted with the other military department’s budget requests to OMB for review. During this period, according to Department of the Navy officials, OMB has monthly conversations with the military departments, including the Department of the Navy, about its budget formulation process. OMB has the authority to raise concerns with a particular military department’s budget request during a passback period. Once the passback period with OMB is complete and the budget request is approved by OUSD(C), the entire Department of Navy budget is submitted to the Office of the Secretary of Defense. Finally, according to Department of the Navy officials, the Office of the Secretary of Defense makes adjustments to the Department of the Navy’s budget request according to DOD and OMB priorities, and after OUSD(C) provides feedback to each military service, the military services’ budget request, as modified, is incorporated into the President’s budget request. The Marine Corps manages the execution of its civilian personnel budget based on the dollar amount, not FTE workload forecasts, through an approach referred to as “Manage to Payroll.” Specifically, the Manage to Payroll approach places an emphasis on spending the amount of dollars or funding available for civilian personnel and not on executing a calculated full-time equivalent civilian personnel workload. Further, Marine Corps Order 12510.2D requires officials delegated Manage to Payroll authority to be accountable for establishing positions to accomplish the mission with maximum efficiency and productivity balanced against the civilian labor budget. The Manage to Payroll approach is comprised of three separate functions: (1) position management, (2) position classification, and (3) compensation management. Position management. The process of organizing and structuring organizations to accomplish their mission with maximum economy, efficiency, and productivity. Managers and supervisors determine the type of organizational structure needed to fulfill the functions assigned to a particular unit, how many positions are needed, how positions should be designed, and the most cost effective way of filling the requirement. Position classification. The function that assigns an individual position to the appropriate pay plan, occupational series, title, and grade. Compensation management. For positions where funding levels are prescribed by the Deputy Commandant for Programs and Resources, the major subordinate commands and Headquarters Marine Corps staffing agencies must ensure salary costs and other cost drivers (i.e. overtime, awards, incentives, etc.) do not exceed the civilian labor funding levels. To implement this approach, the Marine Corps’ Director for Civilian Human Resources, among other things, provides program direction, technical advice, guidance and assistance to the major subordinate commands, Headquarters Marine Corps staff agencies, and servicing human resources offices for carrying out Manage to Payroll responsibilities. In order to hold individuals with delegated classification authority accountable for carrying out effective Manage to Payroll responsibilities, the Director for Civilian Human Resources also conducts consistency reviews in coordination with the human resources offices to validate proper classification of positions. These reviews involve verifying positions are classified in accordance with Office of Personnel Management classification standards and within sound position management principles. Additionally, the Marine Corps Deputy Commandant for Programs and Resources is the lead for determining and allocating the Marine Corps’ civilian personnel budget and sub-allocates the portion of the budget that will be spent against civilian labor by each major subordinate command and Headquarters Marine Corps. According to Marine Corps officials, there are benefits to using the Manage to Payroll approach for managing its civilian personnel budget execution. These benefits, according to Marine Corps officials, include: Flexibility in spending. This approach provides flexibility to the major subordinate commanders by allowing them the ability to prioritize their own current mission requirements and functions rather than spending their civilian personnel budget on workload requirements used to formulate a civilian personnel budget request during the previous fiscal year. Management of personnel requirements. The process allows commanders to manage their personnel requirements to fit with mission priority rather than adhering to the FTE-based workload requirements used to formulate their civilian personnel budget request. Visibility. This process enables officials at Marine Corps Programs and Resources and Headquarters Marine Corps to have direct control over and closely monitor the major subordinate commands’ civilian personnel budget execution. However, weaknesses exist with the Marine Corps’ Manage to Payroll approach to managing its civilian personnel budget execution. Our analysis of the Department of the Navy’s annual budget request, which includes Marine Corps civilian personnel FTE data, found that the number of civilian FTEs the Marine Corps reported does not match the number of civilian FTEs it requested. This discrepancy between the number of FTEs requested and the number of FTEs reported is a result of the Marine Corps managing civilian personnel to dollar amounts and not to FTEs. Specifically, funding provided by Congress annually for the Marine Corps to manage its civilian personnel is based on the number of FTEs the Marine Corps requested for a particular fiscal year. However, during the budget execution process, Headquarters Marine Corps distributes funding to the major subordinate commands by dollar amount and not by FTEs requested. The President’s budget request, which is the sole single document with budget information for the entire government, contains (1) a record of actual receipts and spending levels for the fiscal year just completed, (2) a record of current-year estimated receipts and spending, and (3) estimated receipts and spending for the upcoming fiscal year and 9 years beyond, as proposed by the President. Additionally, OMB Circular A-11 requires that current year FTE estimates should be consistent with previous year actuals, should be fully funded, and should be very close to the actual usage reported at the end of the fiscal year. For example, the estimates in the previous year’s budget should be very close to the actuals published in the current budget. Table 1 shows the difference between the Marine Corps FTEs estimates in its annual civilian personnel budget request and its reported usage of actual FTEs for the previous year’s budget execution contained in the budget requests from fiscal years 2013 through 2018. When asked about the difference between estimates and reported usage of actual FTEs, a Headquarters Marine Corps official stated that they were generally unaware of the importance of the budget data in measuring the degree to which an agency was exceeding or not reaching its requested FTEs. However, the official also acknowledged that a possible consequence of not managing to estimated FTEs could be a reduction in future funding for civilian personnel because budget data provided to Congress for civilian personnel is based on FTE workload and not the amount of dollars spent on civilian personnel. Therefore, a result of the Marine Corps managing to dollar amounts and not to FTEs may result in Congressional decision-making based on incorrect data, which may result in the major subordinate commands having to eliminate civilian positions. Department of the Navy officials confirmed this and told us that the Marine Corps’ use of Manage to Payroll puts Department of the Navy resources at risk of reduction. Additionally, according to Department of the Navy officials, the Navy’s financial management branch does not support the Marine Corps’ use of Manage to Payroll and recommends that the Marine Corps begin to manage its civilian personnel resources using the same process as the Navy’s other budget submitting offices, by formulating civilian personnel funding requests with estimated FTE requirements and then monitoring execution of the budget by actual FTEs. Additionally, by formulating and monitoring civilian personnel budgets by FTEs, there would be more transparency in how the Marine Corps is actually executing its civilian personnel budget. Our review of the Marine Corps’ policy for managing its civilian personnel budget execution found that it does not provide guidance for major subordinate commands to manage actual civilian FTEs to requested amounts. Specifically, Marine Corps Order 12510.2D provides budget execution requirements but does not include requirements on managing civilian FTEs. According to officials with Marine Corps Programs and Resources, Marine Corps Order 12510.2D is a Manpower and Reserve Affairs Department document that focuses on personnel actions rather than explicitly establishing cost controls. Specifically, the order states that individuals delegated Manage to Payroll authority are accountable for establishing positions to accomplish the mission with maximum efficiency and productivity balanced against the labor budget. Federal internal control standards state that management should document the organization’s internal control responsibilities in its policies at the appropriate level of detail to allow management to monitor the control activity effectively. Without updated guidance for major subordinate commands to formulate and execute its civilian personnel budget to estimated FTEs, the Marine Corps risks overspending or underspending on its personnel requirements. Additionally, decision makers may not have sufficient information to effectively and efficiently provide funding for Marine Corps civilian personnel. Our analysis of Marine Corps internal spending data found that the dollar amount the Marine Corps is projecting to spend on civilian personnel for fiscal year 2019 is $1,749,444,000, which is in line with the $1,750,500,000 provided in its budget request for managing civilian personnel. However, our analysis of 15 major subordinate commands found that four of them are either exceeding or not reaching their requested dollar amounts by $5 million dollars or more. For example, for fiscal year 2019, Marine Corps Systems Command is projected to overspend its requested civilian personnel dollar amount by $24.7 million and Marine Corps Cyber Command is projected to fall short of its requested civilian personnel dollar amount by $7.9 million dollars in fiscal year 2019. Table 2 shows the variation in projected dollar amount by major subordinate commands for fiscal year 2019. Marine Corps Programs and Resources officials stated that when a major subordinate command exceeds its requested dollar amount, the major subordinate command is not compensated from larger Marine Corps accounts, but the command must find additional funding from within its other accounts, which may mean a funding cut to another program to make up the difference. Our review of Marine Corps civilian personnel data also found that Marine Corps data on civilian FTEs is not consistent with data that OUSD(C) uses to formulate DOD’s request for civilian personnel FTEs in its annual budget submission. The Marine Corps uses a database called SABRS to maintain and track civilian personnel FTE data for fiscal years 2013 through 2019. To develop DOD’s defense-wide annual civilian personnel budget request, OUSD(C) uses data from the Program Resources Collection Process system. Our review of Marine Corps data maintained in SABRS found that it does not match the data DOD provided in its annual budget request submissions, which comes from OUSD(C)’s Program Resources Collection Process system, for fiscal years 2013 through 2018. Table 3 shows the difference between the Marine Corps’ internal civilian FTE data in SABRS and the civilian FTE data provided in the annual budget request. An OUSD(C) official told us that the two sources should match because all data, including civilian personnel dollar amounts and FTEs, should be submitted and processed through the Program Resources Collection Process system. Marine Corps officials explained that they were aware of the variations between their internal FTE data and the DOD budget request data. The officials further explained that one instance of variation in the data occurs because the Department of the Navy uses another database, the Work Year Personnel Cost system, to prepare its annual budget request, which includes the Marine Corps’ request for civilian personnel FTEs, before submitting the request to OUSD(C). According to the Marine Corps officials, the Work Year Personnel Cost system automatically deletes error transactions while SABRS does not, as error transactions in SABRS are manually edited. These types of error transactions occur because they did not pass one or more edits in SABRS, did not find the required matching transaction, or have some other issue that is keeping the transaction from processing. The Marine Corps corrects error transactions manually while errors transactions in the Work Year Personnel Cost System are deleted automatically without correction, which creates differences in the data between the two systems. Marine Corps officials stated that, as a result, the Work Year Personnel Cost database typically underestimates Marine Corps civilian FTEs, and that variations in actual FTE data between the SABRS and Work Year Personnel Cost databases explains why it appears the Marine Corps is not reaching its requested civilian personnel FTEs. The Marine Corps has not identified or reconciled differences between internal Marine Corps civilian personnel FTE data compared to data submitted in the annual budget request. Additionally, federal internal control standards state that management should use quality information to achieve the entity’s objectives; that reliable internal and external sources provide data that are reasonably free from errors and faithfully represent what they purport to represent; and that information is appropriate, current, complete, accurate, accessible, and provided on a timely basis. Congress and DOD leadership rely on information presented in an agency’s annual budget request to help determine policies and to make financial decisions. Our prior work has found that civilian FTEs, by themselves, may not be reliable measures of the cost of the civilian personnel workforce and changes in civilian FTEs may not achieve commensurate changes in monetary spending. However, in that report we also noted that, according to OUSD(C) officials, FTEs are typically the primary measure OUSD(C) uses in managing and reporting on DOD’s civilian workforce. Therefore, without reliable Marine Corps civilian personnel data, senior leaders in DOD and decision makers in Congress may not have sufficient and appropriate information to make informed planning and spending decisions and may risk funding not accurately tracking with actual needs. To develop its annual civilian personnel budget request, the Marine Corps uses a process that relies on prior fiscal year budget execution data instead of calculating civilian personnel workload requirements and the number of FTEs needed to perform the work. Without updated guidance for the major subordinate commands to manage their respective civilian personnel budget execution to requested FTEs, the Marine Corps risks overspending or underspending on its personnel requirements. Further, assessments of Marine Corps civilian personnel FTE and DOD budget data found that the Marine Corps’ data does not match comparable data DOD reported in its annual budget request documentation. Congress and DOD leadership rely on FTE information presented in the Marine Corps’ annual budget request to help determine policies and to make financial decisions. Without civilian personnel data that are free from errors and are consistent with how the Marine Corps is managing its civilian personnel, senior leaders in DOD and decision-makers in Congress will not be able to make informed planning and spending decisions and may risk funding not accurately tracking with actual needs. As a result, the Marine Corps risks having its annual civilian personnel funding reduced. We are making the following two recommendations to the Secretary of the Navy: The Secretary of the Navy should ensure that the Commandant of the Marine Corps updates the Marine Corps’ civilian personnel budget formulation and execution policies to include guidance for the major subordinate commands to manage civilian personnel to FTEs. (Recommendation 1) The Secretary of the Navy should ensure that the Commandant of the Marine Corps identifies and reconciles any differences between the Marine Corps’ internal civilian personnel data and the civilian personnel data the Department of the Navy uses to support its annual budget request. (Recommendation 2) We provided a draft of this report to DOD for review and comment. In written comments, DOD concurred with both recommendations and noted actions that the Marine Corps plans to take. DOD’s comments are reprinted in appendix III. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of the Navy, and the Commandant of the Marine Corps. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. For our first objective, to determine how the Marine Corps develops its civilian personnel budget request, we obtained and reviewed Marine Corps, Department of Defense (DOD), Department of Navy, and Office of Management and Budget policies that guide the budget process in order to determine what requirements exist across each level of oversight and discussed these requirements with agency officials. To further analyze the Manage to Payroll process, we also obtained and reviewed Marine Corps Program Objective Memorandum requests, budget formulation and execution documents, budget analyses, financial spreadsheets, and budget presentations. We also interviewed responsible officials about these processes and requirements. For our second objective, to assess the Marine Corps’ management of its civilian personnel budget and FTEs, we reviewed the Marine Corps’ process for managing its civilian personnel budget and FTEs and compared it against Marine Corps orders and administrative messages. To examine Marine Corps policies for budget data entry, coordination, and management, we compared them to the Marine Corps Financial Management Standard Operating Procedure Manual. To determine how the Marine Corps manages its civilian personnel funding, how the Marine Corps measures budget execution by major subordinate command, and how the Marine Corps monitors and collects data on budget execution, we interviewed officials from Marine Corps Programs and Resources. To identify Manage to Payroll requirements and Marine Corps policies for managing its civilian personnel we interviewed officials from Marine Corps Manpower and Reserve Affairs. We also interviewed officials in the Marine Corps Programs and Resources office, Marine Corps Systems Command, Marine Corps Cyber Command, the Department of the Navy, and the Office of the Under Secretary of Defense (Comptroller) about their perceptions of the benefits and weaknesses of the Manage to Payroll process. To measure whether the Marine Corps exceeded or fell below its civilian personnel full-time equivalents (FTE) budget request, we compared the Marine Corps’ civilian personnel budget execution data to its budget request data from the same fiscal year to determine if Marine Corps actual FTEs matched this data from year to year. To measure whether the Marine Corps was consistently exceeding or falling below its FTE budget request at each major subordinate command, we reviewed Marine Corps budget execution spreadsheets displaying end-of-year budget and FTE projections. To provide a statistical measurement of the Marine Corps’ efforts to manage its civilian personnel budget, we obtained and analyzed fiscal year 2013-18 budget execution data from the Marine Corps’ Standard Accounting, Budgeting, and Reporting System (SABRS) that tracks both dollar amount and FTE allotment across the major subordinate commands. We also used SABRS data to determine how funds are allocated to the major subordinate commands within the Marine Corps. To determine if differences in reported FTE totals existed between the Marine Corps’ internal data and publically available data, we compared SABRS’ civilian personnel FTE actual data from fiscal year 2013-18 to its requested FTEs. To assess the reliability of the Marine Corps’ data, we reviewed policies and procedures related to data collection and entry, evaluated Marine Corps internal data reliability checks, and interviewed cognizant officials. Based on this, we determined that the data were sufficiently reliable for the purposes of our reporting objectives. We conducted this performance audit from November 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In June 2018, the Department of Defense (DOD) Inspector General issued a report on the civilian personnel budget formulation process of the Department of the Navy, which included information on the Marine Corps. In its report, the DOD Inspector General found that the Marine Corps could not justify or support how it determined its civilian personnel pay requirements for fiscal year 2017’s Marine Corps budget request. Specifically, the report noted that Marine Corps budget officials could not fully explain the rationale for their civilian pay budget adjustments. According to the DOD Inspector General report, the Marine Corps did not maintain documentation to support these budget adjustments or material showing how it calculated average basic compensation amounts and benefit rates. The DOD Inspector General also reported that Marine Corps officials were unable to explain or provide support regarding the calculation of the civilian pay dollars and full-time equivalents (FTE) associated with these adjustments. In its unpublished written response to the DOD Inspector General’s report, the Marine Corps concurred with the DOD Inspector General’s recommendation that the Marine Corps determine budgeted civilian pay funding levels using full-time equivalents calculated based on projected hours to be worked, as required by Office of Management and Budget Circular A-11. The Marine Corps’ written response also acknowledged that having source data, assumptions, calculations, and better documentation related to budget formulation would provide for retention of institutional knowledge and would benefit budget officials formulating future budgets. The Marine Corps’ unpublished written response also stated that it is reviewing other command metrics within the department, which will be performed throughout the remainder of the present budget cycle with a plan of implementation during the next budget cycle. The DOD Inspector General’s report stated that the Marine Corps did not determine civilian pay funding levels using FTEs calculated from projected hours to be worked, as directed in Office of Management and Budget (OMB) Circular A-11. According to the DOD Inspector General’s report, in its fiscal year 2017 budget formulation, Marine Corps officials stated that they considered FTEs to be the same as end strength for budget formulation purposes, assuming that one person would be on board for an entire year even though these officials acknowledged that this was not expected to be the reality during budget execution. As a result, the DOD Inspector General recommended that the Marine Corps determine its budgeted civilian pay funding levels using FTEs calculated as required by OMB Circular A-11 requirements. In its unpublished written response to the DOD Inspector General’s report, the Marine Corps stated that it determines FTEs in accordance with OMB Circular A-11. Marine Corps Programs & Resources officials told us that they use cumulative hours paid in a fiscal year divided by the number of work hours in that fiscal year to generate the number of FTEs executed. These officials told us that they then divide the cumulative amount paid by the number of FTEs to receive the average work year cost for that fiscal year. Marine Corps officials stated that Programs & Resources budget officials then adjust the civilian personnel budget request during the Department of Navy pricing tool time frame, typically reducing FTEs by the recommendation provided by the tool, which uses 18 months of execution data. Marine Corps officials told us that, as result of the DOD Inspector General’s report, they are working to provide greater emphasis on FTEs in their budget formulation documents. Brenda S. Farrell, (202) 512-3604 or farrellb@gao.gov. In addition to the contact named above, Vincent Balloon (Assistant Director), Timothy Carr, Brian Pegram, Clarice Ransom, Aaron Safer- Lichtenstein, Shari Nikoo, Michael Silver, Carter Stevens, John Van Schaik and Gregory Wong made key contributions to this report. Defense Health Care: Additional Assessments Needed to Better Ensure an Efficient Total Workforce. GAO-19-102. (Washington, D.C.: Nov. 27, 2018). DOD Civilian and Contractor Workforces: Additional Cost Savings Data and Efficiencies Plan Are Needed. GAO-17-128. (Washington, D.C.: Oct. 12, 2016). Civilian and Contractor Workforces: Complete Information Needed to Assess DOD’s Progress for Reductions and Associated Savings. GAO-16-172. (Washington, D.C.: Dec. 23, 2015). Defense Headquarters: DOD Needs to Reassess Personnel Requirements for the Office of Secretary of Defense, Joint Staff, and Military Service Secretariats. GAO-15-10. (Washington, D.C.: Jan. 21, 2015). Human Capital: DOD Should Fully Develop Its Civilian Strategic Workforce Plan to Aid Decision Makers. GAO-14-565. (Washington, D.C.: July 9, 2014). Human Capital: Opportunities Exist to Further Improve DOD’s Methodology for Estimating the Costs of Its Workforces. GAO-13-792. (Washington, D.C.: Sept. 25, 2013). Human Capital: Additional Steps Needed to Help Determine the Right Size and Composition of DOD’s Total Workforce. GAO-13-470. (Washington, D.C.: May 29, 2013).", "summary": "The Marine Corps requested $1.81 billion to pay for approximately 16,000 civilian employees in its fiscal year 2020 budget request. The Office of Management and Budget directs federal agencies to develop civilian personnel budgets by calculating workload requirements, the time needed to complete the work, and the number of FTEs needed. The Marine Corps uses a unique budget formulation process that relies on prior fiscal year budget data to calculate FTE estimates for future civilian personnel budget requests. Senate Report 115-290, accompanying a bill for the DOD Appropriations Act, 2019, included a provision for GAO to review how the Marine Corps develops its civilian labor requirements for both FTEs and funding and examine the benefits and shortfalls of the Manage to Payroll process. This report (1) describes how the Marine Corps formulates its civilian personnel budget request and (2) assesses the Marine Corps' management of its civilian personnel budget and FTEs, including the benefits and weaknesses of the process. GAO reviewed DOD civilian personnel budget policies, analyzed fiscal years 2013 through 2018 Marine Corps budget data that tracks spending and FTE allotment, and compared 2013 through 2018 budget execution data to budget request data. The Marine Corps develops its civilian personnel budget request using prior fiscal year budget execution data with adjustments based on input from sources such as the Office of the Undersecretary of Defense (Comptroller) [OUSD(C)] and the Department of the Navy. As part of the Department of the Navy, the Marine Corps' budget request is added to the Navy's overall budget request, which is incorporated into the Department of Defense's (DOD) overall budget request. The Marine Corps manages its civilian personnel based on dollar amounts—not full-time equivalent (FTE) workload like the other military services—through an approach called Manage to Payroll. Specifically, while the Marine Corps requests a certain number of FTEs each year as required by policy, the Marine Corps distributes the funds it receives to its commands by dollar amount and not based on the FTEs requested. This approach has benefits, such as providing flexibility to employ civilians based on current mission requirements. However, under this approach, for fiscal year 2019, internal Marine Corps' data show that four of its commands are either exceeding or not reaching their requested dollar amounts. Marine Corps policy does not provide guidance to its commands to manage FTEs to requested amounts. Without such updated guidance the Marine Corps risks overspending or underspending on its personnel requirements. In addition, internal Marine Corps civilian FTE data for fiscal years 2013 through 2018 is not consistent with data that OUSD(C) used to formulate DOD's overall civilian personnel budget request, as shown in the figure below. The Marine Corps has not identified or reconciled differences between its internal data compared to data submitted in the annual budget request. If information in the Marine Corps' budget request does not reflect internal Marine Corps data, then Congress and DOD leadership may not have sufficient and appropriate information to make informed planning decisions. GAO recommends that the Marine Corps 1) updates its budget policies to include guidance for commands to manage civilian personnel to FTEs and 2) identifies and reconciles differences between its internal data and data OUSD(C) uses to formulate the Marine Corps' annual budget request. DOD concurred with both recommendations.", "document_type": "gao"}
{"report": "In fiscal year 2019, states are required to spend at least 8 percent of CCDF funding for “quality activities”—activities that are designed to improve the quality of child care services the state provides. These activities may include supporting the professional development of the child care workforce and improving the supply and quality of child care programs and services for infants and toddlers. Table 1 describes examples of quality activities states may choose to fund with their required quality set-aside, as well as requirements for states to carry out certain activities from the CCDBG Act of 2014, where applicable. After setting aside funds for quality activities and administrative activities, states must spend at least 70 percent of discretionary funds that remain on subsidies for eligible families. They provide subsidies to eligible families through the CCDF program in the form of certificates or vouchers to use for child care in homes, child care centers, and classrooms, or through grants or contracts to child care providers. Children receiving CCDF subsidies may receive care alongside nonsubsidized children— that is, children who may be eligible for child care subsidies but who do not receive them, or who may be ineligible for child care subsidies. A majority of states used fiscal year 2017 CCDF funds to entirely or mostly fund 7 of 10 major state child care activities, according to our survey of CCDF administrators in the 50 states and D.C. (see fig. 1). The 10 child care activities included in our survey, components of which are also required by CCDF, are key means through which states may choose to improve the quality of their child care services (i.e., quality activities). They also represent diverse aspects of a state’s child care system. Among states that relied on CCDF funding to support the quality activities, we found that, on average, states funded 6 of the 10 activities entirely or mostly with CCDF. Nearly one-third of states (16) funded at least 8 of the 10 to that degree. States reported that they relied on CCDF funding most frequently for the following activities: child care resource and referral systems, consumer education, and health and safety standards establishment and training. Child care resource and referral systems. More than three-quarters of states (40) reported in our survey that all (22) or most (18) of the funding they used for their child care resource and referral systems in fiscal year 2017 came from CCDF. Statewide systems of child care resource and referral agencies can serve an important role in supporting state quality improvement efforts, though not all states have them, according to HHS. For example, child care resource and referral agencies may provide training and technical assistance to child care providers and share consumer education with families, among other things. States may use CCDF funds to establish or support a system of local or regional agencies that is coordinated by a lead child care resource and referral organization. Officials in the states we interviewed described various ways in which their child care resource and referral agencies support child care providers and parents, such as: Delivering professional development, including training and technical assistance, to child care providers, regardless of whether or not the providers accept subsidized children, according to several CCDF administrators interviewed. Supporting parents by determining eligibility for subsidies, providing referrals for care, and offering information on child care quality, according to state officials. For example, one state houses eligibility specialists in regional child care resource and referral agencies, through which families apply for subsidies, while another state uses these agencies to refer families to child care providers and support families with specialists, including mental health consultants and infant specialists, as needed. Consumer education. About 70 percent of states (36) reported that all (12) or most (24) of the funding they used for consumer education activities in fiscal year 2017 came from CCDF. Consumer education activities are intended to help parents seeking child care make informed decisions and improve access to information that supports child development. States must certify that they have policies to make public the results of child care providers’ monitoring and inspection reports, as well as certify that they will collect and disseminate information on child care services available through CCDF, research and best practices concerning child development, and state policies regarding the social- emotional and behavioral health of children, among other requirements. Moreover, many of the 15 states we interviewed used child care resource and referral agencies to do this. Examples from our state interviews illustrate that: One state promotes awareness of its quality rating and improvement system for child care providers through materials available from the state’s child care resource and referral agencies, according to its CCDF administrator. Another state’s child care resource and referral system has a public awareness campaign aimed at the parents of infants, toddlers, and preschoolers to help families understand and identify quality child care, according to the head of the state’s child care resource and referral network. Parents in a third state can obtain information on child development through resources available from lending libraries, according to the state’s CCDF administrator. Health and safety standards. About 70 percent of states (36) also reported entirely funding (15) or mostly funding (21) the development or deployment of training for health and safety standards with CCDF in fiscal year 2017. According to the CCDBG Act, states are required to certify that they have health and safety standards in specific topic areas, such as the use of safe sleeping practices and pediatric first-aid, and certify that all CCDF providers will receive minimum health and safety training in these areas. Most of the 15 states that we interviewed went beyond CCDBG Act requirements and elected to apply their health and safety training requirements to all licensed child care providers in the state, and in some cases, to child care providers that are exempt from licensing. In doing so, officials described how their requirements served to elevate the health and safety of children in care regardless of whether they receive CCCF subsidies. Several state officials specifically credited the CCDBG Act as the impetus for their states’ changes. State officials we interviewed also described taking various approaches, including offering financial incentives, to facilitate child care providers in meeting training requirements. Examples of these approaches and their impact include the following: One state official said that while the state child care agency had wanted to increase health and safety requirements for child care providers for years, the reauthorization of the CCDBG Act propelled the state forward in its efforts to increase child care quality and require the same health and safety training of all licensed and license-exempt providers. One state offers health and safety grants to child care providers to meet these requirements, while another is considering increasing child care provider payment rates to a level that will allow them to meet the updated health and safety requirements, according to state officials. CCDF administrators in two states told us they are developing online training modules for the required health and safety training so child care providers can access the modules more easily and for free or have mailed training DVDs to every child care program in the state. CCDF administrators in almost all of the 15 states we interviewed told us their states set aside more than the minimum amount that CCDF required to support quality in 2017. They described how their states use quality set-aside funds to support child care licensing programs, accreditation, and quality rating systems for child care providers, among other things. Some state officials we interviewed also described specific supports for infants and toddlers, such as partnerships to provide training for child care providers around the care of this age group, and increases in provider payment rates for infant and toddler care, which is costly to provide, from the infant and toddler-specific set-aside. According to one state CCDF administrator, the ability to divert funds to activities that benefit infants and toddlers is critical as this is the neediest age—a time when children and parents need the most support. A range of CCDF quality activities, including consumer education, child care licensing, and professional development of the child care workforce affect the care of children not receiving subsidies (nonsubsidized children), according to our 51-state survey of CCDF administrators (see fig. 2). On average, states reported that 9 of the 10 activities included in our survey affect nonsubsidized children receiving child care in the state, with more than 40 percent of states (22) reporting that all of the activities affect nonsubsidized children, according to our analysis of the survey data. As previously noted, the activities serve as key supports for building quality in state child care systems. Of these activities, CCDF administrators unanimously cited three in our survey as affecting nonsubsidized children: consumer education; licensing, monitoring or background checks for child care; and professional development. Below are some specific examples of the way nonsubsidized children are affected by these activities, as discussed with CCDF administrators in our 15 state interviews. Consumer education. During our interviews, state officials discussed ways in which their CCDF programs share important information on child care quality and child development with all families, including those not receiving subsidies. As previously noted, many of the 15 states we interviewed rely on their child care resource and referral agencies to provide such information to the public. HHS requires states to have a website that includes, among other things, a searchable list of licensed child care providers and information about the provider’s quality rating, if available. States we interviewed use these and other consumer education tools, such as billboards, public service announcements, and commercials in an effort to reach a wide-ranging audience. Licensing, monitoring, or background checks. According to the CCDBG Act, states must certify they have policies to annually conduct unannounced inspections of all licensed CCDF providers for compliance with all child care licensing standards, including health, safety, and fire standards, with at least one pre-licensure inspection. But most of the 15 states we interviewed have elected to apply certain CCDBG Act requirements for CCDF providers, including those pertaining to monitoring and inspections, to all licensed providers in the state, according to their states’ CCDF administrators. Officials in several states suggested that updating their requirements for all licensed providers with the CCDF requirements establishes a high-quality foundation for child care that reflects the importance of a healthy and safe environment for all children receiving care, regardless of whether children receive a subsidy. Examples from some states that we interviewed are: In one state, where subsidized children make up about 20 percent of children in licensed care, the state’s CCDF administrator estimated that significant numbers of nonsubsidized children benefit from higher quality care, including from more extensive monitoring of all licensed providers. Another state applied the requirements for CCDF providers to all license-exempt child care providers (including those not serving subsidized children), which helps ensure that all children in care benefit from the updated monitoring, health and safety, and background check requirements. An official from another state that does not apply CCDBG Act requirements more broadly said that, because nonsubsidized children share classrooms with subsidized children, requirements that apply to subsidized providers, in turn, also still benefit the nonsubsidized children in their classrooms. In particular, he said that the requirement that all CCDF providers that serve subsidized children be inspected has opened up child care centers that previously, as license-exempt providers, were not inspected, and has resulted in improvements in some centers. Professional development. CCDF administrators we interviewed recognize professional development as key to high-quality child care for all children, including nonsubsidized children. The CCDBG Act requires states to describe the training and professional development requirements designed to enable CCDF providers to promote the social, emotional, physical, and cognitive development of children. According to HHS, states must also require ongoing training for CCDF providers that is accessible, appropriate to the age and setting of the children served, and aligned to a progression of professional development that includes a minimum number of annual hours of training for the child care workforce. As with certain other CCDBG Act requirements, a majority of states we interviewed have established the same professional development requirements for all licensed child care providers, whether or not they care for subsidized children, according to state officials. One CCDF administrator said that the updated CCDF requirements for subsidized providers were an impetus for her state to raise the training requirements for providers that do not care for children receiving subsidies and that are unlicensed. She said the updated, more comprehensive training requirements help ensure that all children are in care with child care providers that parents can trust. CCDF administrators also highlighted characteristics of their states’ professional development activities that serve to positively impact nonsubsidized children as well—namely, availability, accessibility, and affordability of professional development opportunities to child care providers. For example, state officials told us of making these opportunities available to all child care providers through online training courses, training and onsite consultation from child care resource and referral agencies, technical assistance and coaching, and resource lending libraries. Nearly all states we interviewed use their states’ quality set-aside funds to support such training, technical assistance, and/or coaching opportunities. Where training may not be free, CCDF administrators told us of financial incentives that assist child care providers in their efforts to increase quality through professional development. For example, several states use their quality set-aside funds to offer scholarship grant programs available to child care providers to help increase their qualifications, whether or not they care for subsidized children. One state offers incentive payments based on a provider’s level of attainment in the state’s career ladder, for which all providers are eligible to apply, according to its CCDF administrator, and can result in provider development that benefits the nonsubsidized children in their care. In addition to spending on quality activities, states reported through our interviews that nonsubsidized children also indirectly benefit from state spending on subsidies. According to officials in many of the 15 states that we interviewed, states’ spending on subsidies helps increase the economic stability of CCDF providers, which, in turn, also benefits nonsubsidized children in their care. Officials said that subsidizing providers to help pay for the cost of care for eligible families can provide a consistent source of revenue for CCDF providers that allows them to continue stable operations, invest in professional development, and increase teacher pay, for example. Such spending, in turn, can lead to improved child care quality as well as access (i.e., by helping providers stay in business) to nonsubsidized children, too, who are in their care, according to state officials. However, officials in many states we interviewed also noted that CCDF subsidies or related policies may negatively impact nonsubsidized children and families. For example, several said that state increases in payment rates for CCDF providers may lead providers to similarly increase the rates they charge for the nonsubsidized children in their care, which, some noted, could drive families for whom such care is no longer affordable to alternative, unregulated providers that may have lower quality standards. Rate increases can be particularly difficult for middle-income families who do not qualify for CCDF and are struggling to meet the current market rate of child care, according to one state’s CCDF administrator. CCDF administrators from several other states also noted a drop in CCDF child care providers in recent years due to various factors, including low payment rates, extensive CCDF requirements for inspections and background checks, and an insufficient number of children to sustain operating costs, for example. In much of one state’s neediest areas, local elementary schools often provide the highest quality care, according to the state’s CCDF administrator; however, with the addition of background checks that some school districts have found administratively burdensome and duplicative, the official said that many districts have dropped out of the CCDF program. Among quality activities, states most often reported plans to spend the new discretionary CCDF funding from the Consolidated Appropriations Act, 2018 on three—licensing, consumer education, and professional development—the same activities that all states reported affect nonsubsidized children, according to our survey of CCDF administrators in the 50 states and D.C. (see fig. 3). Licensing, monitoring, or background checks. More than two thirds of states we surveyed (34) plan to spend the new CCDF funds on child care licensing or related activities of monitoring and background checks. During our interviews, many state CCDF administrators provided examples of how they plan to use new funds on licensing-related activities, such as hiring or increasing pay of licensing staff or making administrative or system improvements to facilitate the interstate background checks required under the CCDBG Act. For example, one state plans to enhance its online background check portal to streamline interstate coordination while another state plans to help providers pay for the interstate background check fees by offsetting the increased cost for the next 1 or 2 years. A third state, which has been operating under an HHS waiver that allowed for delayed implementation of the interstate background check requirements, now plans to use new funds to conduct the required checks, according to the state’s CCDF administrator. Without the new funds, officials from two states said that they may have had to reduce funding to other child care activities, including subsidies, in order to allocate the additional resources needed to comply with licensing, monitoring, or background check requirements. Consumer education. More than half of states we surveyed (30) said they plan to spend new funds on consumer education activities. Some state officials we interviewed described plans to enhance public state child care websites to make them more user-friendly or available in other languages, such as Spanish. For example, one state plans to improve online access to provider information by featuring a dashboard with a snapshot of each provider’s license history, including inspection violations. Officials from another state said they plan to use new funds to launch a public engagement campaign to provide timely and important information about child care and state-based child care services. In the absence of the new funding, officials from two states said they would likely need to reduce their efforts to better educate families statewide about important child development information and the states’ publicly- available tools that can help parents identify high-quality child care. Specifically, officials from one state said they would have to forgo plans to make their public child care website more sophisticated and consumer- friendly and officials from another state said they would not be able to conduct their planned public education campaigns. Professional development. More than half of states we surveyed (30) said they plan to fund professional development activities for child care providers. Officials we interviewed in several states told us about plans to use the new funds to implement or improve online professional development systems, such as by increasing online course offerings or creating training applications accessible by cellphone, which can improve accessibility for all child care providers. We also heard about plans in five states to use some new funds to provide specialized training, including training focused on infant and toddler-specific topics, caring for children exposed to trauma, and emergency planning and response. CCDF administrators from two other states described plans to fund more scholarships for child care workforce training and certification programs, including Child Development Associate credential programs. Lastly, officials in one state told us they plan to create a mentorship program whereby high-quality licensed providers mentor licensed-exempt providers in order to help providers who are interested in becoming licensed improve their quality and professional development qualifications. Without the new funds, officials from one state said they would not have been able to continue to support as many professional development opportunities that support both subsidized and nonsubsidized children, such as conferences, networking events, and coaching. Another CCDF administrator expressed concern that in the absence of the new funds, her state may have struggled to implement a new workforce registry system that tracks child care providers’ education and credentials. Most states reported plans to allocate the new funds to multiple state child care activities, according to our analysis of the survey data. Specifically, we found that more than two thirds of states plan to fund at least three of the activities, and half of states plan to fund at least five activities. Moreover, according to our survey, about 40 percent of states (20) also plan to spend at least some of the new funds to increase the proportion of funding set aside for quality activities beyond the required minimum for the year—which, as described earlier, they can use to fund these activities. During our interviews, we heard about states’ plans to spend new funds on a variety of qualifying activities, including child care resource and referral systems, accreditation of child care providers, and development of high-quality program standards. In the absence of the new funds, one state CCDF administrator told us that the state would likely have had to eliminate some optional quality activities, such as financial support to help providers become accredited. She further explained that the state is more willing to cut back on quality activities when there is insufficient funding than to disenroll families from the CCDF program. Aside from quality activities, states we surveyed also reported plans to spend new CCDF funds toward subsidies. More than half of states (31) plan to spend at least some of the new funds on increasing payment rates for CCDF providers or lowering parental copayments. For example, one state official we interviewed told us about plans to increase payment rates for infant and toddler care, with a goal to increase access to child care for infants and toddlers across the state. In addition, about half of states (25) we surveyed reported plans to spend new funds to implement two requirements that allow families to continue receiving subsidies for a longer period of time—the 12-month eligibility period and the graduated phase-out of assistance. Lastly, nearly one-third of surveyed states (16) reported plans to use new funds to pay for subsidies for children on their wait lists to receive child care. CCDF administrators in all of the states we interviewed that use a wait list (5) stated that they might have had to expand their wait lists in the absence of the new funds. However, several state CCDF administrators expressed uncertainty about their states’ plans for using the new CCDF funds in interviews (conducted in May and June 2018). Officials from more than a third of the 15 states we interviewed (6) said their spending plans were still in flux. In some of these states, officials said they were still developing and reviewing their funding proposals as part of their state’s legislative and budgeting process and they were awaiting future legislative approval or spending authorization. For example, in one state, the CCDF administrator said she was awaiting information on how much money the state would receive before she planned to convene stakeholder groups to discuss potential funding proposals. In another state, the CCDF administrator said her office needed to wait for other local budget appropriation decisions before her office could commit the new CCDF funds to specific priorities. Officials in more than half of the 15 states we interviewed also told us they faced challenges making spending decisions because they were unclear whether the new funds would be provided on an ongoing basis. For example, CCDF administrators in two states that plan to expand subsidies to children on their wait lists expressed concerns about having to disenroll children from the program if funding is discontinued. Officials from several states suggested that they are proceeding cautiously with spending decisions given there is no guarantee that the increased funds will be provided in the future, while an official from another state told us they are operating under the assumption that the new funds will be provided on an ongoing basis and do not have a contingency plan in the event that the funds are not continued. We provided a draft of this report to HHS for review and comment. HHS provided technical comments only, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of the Department of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix II. In addition to the contact named above, Janet Mascia (Assistant Director), Avani Locke (Analyst-in-Charge), and Elizabeth Hartjes made key contributions to this report. Also contributing to the report were Seto Bagdoyan, James Bennett, Randy De Leon, Kirsten Lauber, Sheila R. McCoy, Jonathon Oldmixon, Jessica Orr, James Rebbe, Almeta Spencer, and Amy Sweet.", "summary": "CCDF is the primary source of federal funding for child care subsidies. States administering CCDF are subject to requirements that improve the quality of child care for all children, nonsubsidized as well as subsidized. In March 2018, the Consolidated Appropriations Act, 2018 was enacted, which provided $5.2 billion in additional CCDF discretionary funding for fiscal year 2018, approximately twice the amount provided in fiscal year 2017. GAO was asked to review state use of CCDF funds and their potential impact on nonsubsidized children. GAO examined (1) the extent to which states use CCDF funds to support their child care system, (2) the kinds of CCDF–related activities states engage in that affect children who are not receiving CCDF subsidies, and (3) how states plan to use the increase in CCDF funding from the Consolidated Appropriations Act, 2018. GAO collected information from state CCDF administrators through a survey to the 50 states and the District of Columbia (D.C.) and interviews with officials in 15 states, including D.C., selected to reflect diverse characteristics and locations. GAO also reviewed relevant federal laws, regulations, and guidance, and interviewed Department of Health and Human Services officials. GAO makes no recommendations in this report. A majority of states used funding from the Child Care and Development Fund (CCDF) in fiscal year 2017 to entirely or mostly support 7 of 10 major state child care activities GAO identified in its survey of 51 state CCDF programs. These activities, components of which are also required by CCDF, represent diverse aspects of state child care systems and are a key means through which states may choose to improve the quality of their child care. States reported that they relied primarily on CCDF funding for child care resource and referral systems, consumer education, and health and safety standards establishment and training more frequently than for other activities. States reported in GAO's survey that a range of CCDF quality activities affect the care of children not receiving CCDF subsidies (nonsubsidized children), including three activities cited by all states—consumer education, child care licensing, and professional development of the child care workforce. CCDF administrators in most of the 15 states GAO interviewed said they have elected to apply certain requirements for caregivers subsidized under CCDF to all state licensed child care providers. For example, child care providers may be subject to monitoring and professional development requirements, whether or not they care for children receiving subsidies. CCDF administrators also stated that, as a result, all children in the care of licensed providers in these states—including nonsubsidized children—benefit from the enhanced requirements. States most often reported in GAO's survey that they plan to spend new CCDF funds provided in the Consolidated Appropriations Act, 2018, on quality activities that benefit all children in child care including licensing, consumer education, and professional development. For example, officials GAO interviewed in several states described plans to enhance public state child care websites to make them more user-friendly for all families or available in other languages, such as Spanish. However, more than a third of the interviewed states said their spending plans were still in flux, and more than half said they faced challenges making spending decisions because it was unclear whether the new funds would be provided in the future.", "document_type": "gao"}
{"report": "As the landlord for the federal government, GSA acquires space on behalf of federal agencies through new construction and leasing. In this capacity, GSA leases space in 8,681 buildings or other assets and maintains a total inventory of more than 370 million square feet of workspace for 1.1 million federal employees, plus support contractors. Furthermore, GSA is authorized by law to enter into lease agreements for up to 20 years and is permitted to obligate funds for its multiyear leases one year at a time. GSA can delegate its leasing authority to agencies if GSA determines it is in the government’s best interest. Agencies may request this delegation of authority when they believe they can obtain the lease more efficiently than GSA. GSA grants three types of delegations of leasing authority, depending on the intended use of the leased space: General purpose – types of space that might be needed by almost any agency, such as office or warehouse space; Categorical – specific types of space that might be needed by some agencies, such as for antennas, depots, or docks; and Special purpose – types of space designated for 13 specified agencies, such as laboratories for the Department of Health and Human Services or office space in or near stockyards for USDA. GSA’s FMR Bulletin C-2, issued in 2014, (the 2014 Bulletin) provides usage and reporting requirements for delegations of leasing authority. Many of these requirements restate or elaborate on various requirements in statute and regulation. All delegations of leasing authority, including general purpose, categorical, and special purpose space delegations, are covered by the 2014 Bulletin. Agencies are responsible for compliance with all applicable requirements when using delegated leasing authority. Agencies must also conform with the requirements of any delegation approval from GSA. The requirements can include limits on square footage or the length of the lease. Although GSA delegates its leasing authority to other agencies, it acts as a guarantor for the leases in the event of a default by an agency. GSA officials said that there have not been any defaults to date. The process to apply for delegated leasing authority and then obtain a delegated lease is outlined in figure 1 below. In 2007, GAO found that GSA’s delegated leasing program documentation was incomplete, inconsistent, unclear, and outdated. Specifically, we found that GSA’s lease delegation process lacked certain management controls, such as current written policies and procedures. In addition, the GSA OIG found that some delegated leases had excessive rental rates and inadequately documented lease files, primarily due to customer agencies’ lack of expertise. Further, 56 percent of the lease files reviewed by the OIG contained insufficient documentation to support that the federal government received a fair and reasonable price. In response to problems identified in GAO and GSA reviews, GSA reformed its lease delegation program by clarifying requirements, documenting policies and procedures, and centralizing data management. In 2007, GSA issued new requirements for the delegated leasing program in the FMR Bulletin 2008-B1 (2008 Bulletin). For example, the 2008 Bulletin instructed GSA and the agencies on the proper submission of documents to GSA; and required agencies to have an organizational structure in place to support the delegation of authority, and to ensure compliance with all applicable laws, regulations, and GSA directives governing the lease acquisition. In 2014, GSA began using a new electronic system—G-REX—to review and process applications for delegations of leasing authority. Requesting agencies began electronically submitting pre-authorization and post award documents to G-REX. In 2014, GSA re-emphasized and updated the requirements applicable to GSA leasing delegations in its 2014 Bulletin, which continued to be in effect when this report was issued. GSA continues to address data quality issues that persist in spite of its reform efforts. These data quality issues affect GSA’s ability to monitor its delegated leasing program. First, we found that when information is compiled, the G-REX system overstates the total delegated lease contract values by 12 times higher than they actually were for every delegated lease in the G-REX system. This occurred because it multiplied annual rents by the number of months of the lease, instead of by the number of years. For example, for a lease with an annual rent of $2,300,000 and a lease term of 48 months, the calculated total contract value was $110,400,000 instead of the $9,200,000 total contract value it should have had for the 4 year lease. GSA officials confirmed this error and corrected it during the course of our review. Second, we also found data errors in G-REX resulting in approved delegated leasing projects with annual rental rates higher than they actually were. For example, we found a data entry within G-REX for an approved delegated lease with a total lease rental rate several times higher than the average annual rent rate. After reviewing the lease file, GSA officials confirmed that the rental rate was incorrectly entered by the user into G-REX. We also found two G-REX data entries for approved delegated leasing projects with 25 year lease terms. General purpose delegated leases can only be for terms of up to 20 years. GSA officials confirmed that both identified leases were within the authorized delegated leasing parameters but that the data entries were inaccurate due to a system error within G-REX that incorrectly calculated the renewal options. GSA officials said that they are aware of some data quality issues with the G-REX system and are working to address them in an updated version, which they plan to launch later in 2019. Officials said that the new version of G-REX will include more business rules to prevent missing data and identify anomalies. Further, uploading required post award documents is not currently a mandatory action in G-REX. Instead, G-REX sends automatic reminder emails to agencies if these documents have not been uploaded. To address this issue, GSA officials said that the new version of G-REX would improve the post award document upload process. As we discuss later in this report, we found that selected agencies did not always submit all required post award documents. While GSA is taking steps to improve the G-REX system, it does not reconcile FRPP and G-REX data. Specifically, the 2014 Bulletin states that GSA will perform an annual reconciliation of data between FRPP and G-REX. GSA officials described the annual reconciliation as an oversight procedure that would help ensure that GSA has an accurate listing of delegated leases by comparing FRPP data with the centralized records on delegated leases (currently stored in G-REX). According to GSA officials, they tried to fully reconcile the two databases in 2014 but were unable to do so. GSA officials stated that while they could identify certain specific discrepancies between FRPP and G-REX, conducting a full reconciliation of the two databases has many degrees of complexity. Specifically, G-REX does not include all delegated leases, in part, because not all existing delegated leases migrated into G-REX from the prior GSA leasing system. In addition, GSA officials said FRPP and G-REX do not directly match because each database serves different purposes. Specifically, FRPP is a single comprehensive database that contains information on federal real property worldwide, updated annually. In contrast, G-REX is considered a business process management software application and is primarily used by GSA to process and capture lease delegation applications, according to GSA officials. GSA officials now report that, even though the 2014 Bulletin still calls for the annual reconciliation of data in G-REX and FRPP, they believe fully reconciling the two datasets would have little, if any, value, and currently have no intentions to do so. The Standards for Internal Control in the Federal Government state that improving the reliability of data could help agencies better manage programs. For example, in this case, agencies could utilize real property data to measure performance and inform decision-making to ultimately improve the cost effectiveness and efficiency of their real property portfolio. Moreover, although FRPP data quality could be improved, FRPP can still provide reliable background information on GSA’s federal real property portfolio. Since agencies are required to report data to FRPP on all leased assets acquired under a delegation from GSA, FRPP may provide GSA with useful information on an agency’s delegated leases, in addition to what is included in G-REX. We recognize the challenges posed by attempting to fully reconcile G- REX and FRPP. However, the 2014 Bulletin does not explicitly state GSA will perform a full reconciliation. GSA could partially reconcile G-REX and FRPP by doing some cross-data comparison. For example, had GSA cross-verified G-REX and FRPP data, even on a case-by-case basis, it could have potentially caught and addressed the data quality issues we found in G-REX earlier. Some comparison of G-REX with the relevant data in FRPP could improve the reliability, and thereby the usefulness, of both data sets. For example, GSA officials said that GSA could, in theory, begin comparing leases reported in FRPP as being awarded with delegated authority against G-REX’s record of delegated leases. A partial reconciliation like this could identify leases possibly acquired without delegated leasing authority or other data quality issues and GSA could then take steps to increase the reliability of the G-REX data. Until GSA clarifies its position on what efforts it will take to reconcile G-REX and FRPP, GSA is potentially losing opportunities to enhance its oversight and is operating at odds with its own procedures. We found that GSA has not designed control activities that would allow it to regularly determine the adequacy of requesting agencies’ policies and procedures to manage their delegated leasing activities. Instead, GSA officials said that they expect agencies to have the capacity to manage their delegated leases until evidence suggests otherwise and said GSA assesses agencies’ activities on an ad hoc basis. For example, GSA officials said that GSA audited USDA and Bureau of Indian Affairs (BIA) because of tips from outside sources. Agencies requesting a delegation of leasing authority must submit, among other things, an organizational structure and staffing plan to support the delegation that identifies trained and experienced staff to support delegated leasing activities. In our review, we found that not all selected agencies had sufficient policies and procedures to manage their own delegated leases. For example, GSA’s ad hoc review of USDA’s delegated leases found significant oversight issues. Specifically, GSA found that USDA had awarded seven leases without a delegation of authority. In addition, USDA was unable to locate the executed lease for one of the delegated leases we reviewed. USDA officials said the agency has learned from experiences like this one and is currently developing better policies and procedures to prevent this from happening again. For example, USDA has centralized leasing oversight between two bureaus and plans to annually review selected delegated leases. Moreover, GSA’s ad hoc review of BIA’s delegated leases found that BIA had also leased property without delegated authority. Further, GSA’s 2012 audit of post award documents found that BIA had some delegated leases that had expired, and some exceeded the space threshold of 19,999 square feet. As a result of its review, GSA did not grant BIA any new delegated leasing authority until its OIG completed its findings and BIA responded with a corrections plan that corrected these deficiencies, according to GSA. GSA’s 2014 Bulletin states that GSA will review the adequacy of the requesting agency’s organizational structure and staffing proposed for the delegation; and whether the requesting agency has complied with all applicable laws, executive orders, regulations, OMB Circulars, and reporting requirements under previously authorized delegated leases. Further, according to federal standards for internal control, management should design control activities to achieve objectives and respond to risks. Control activities are the actions management establishes through policies and procedures to achieve objectives and respond to risks in the internal control system. Accordingly, agencies with delegated leasing authority should have an appropriate organizational structure and effective policies and procedures to support the delegation and to ensure compliance with applicable laws and other requirements, both of which help agencies manage their delegated leasing activities. If GSA had designed control activities to regularly review each agency’s policies and procedures for managing its delegated leases, GSA officials could have known earlier that an agency lacked the ability to manage its delegated leases and possibly delayed granting additional delegations of leasing authority until the agency had demonstrated their ability to manage its delegated leasing activities. GSA officials said assessing an agency’s policies and procedures to manage delegated leasing activities when reviewing the agency’s individual application for a delegation of leasing authority is not practical. GSA officials noted that it would become a repetitive and unproductive process to review an agency’s policies and procedures each time they applied for delegated leasing authority as the same agencies are requesting delegated leasing authority for many leases and an agency’s policies and procedures would not change with each new application. However, GSA could assess agencies’ policies and procedures for managing delegated leasing activities at regular intervals, such as annually or biennially. Because GSA is not following its own procedures set out in the 2014 Bulletin, or designing control activities that would allow it to assess, at regular intervals, agencies’ ability to manage their own delegated leasing activities, GSA cannot ensure that it is providing this authority to agencies that can manage it effectively. GSA does not track agencies’ performance toward meeting GSA’s management goals, which is inconsistent with the 2014 Bulletin and GSA policy. GSA has three key management goals for tracking the success of the delegated leasing program: 1. Delegated leases should have lease rates that are at or below private sector rates over half the time, according to GSA’s annual performance plan. The 2014 Bulletin states that, prior to granting the agency’s request for a leasing delegation, GSA will consider the demonstrated ability of the requesting agency to meet or exceed this published performance measure for the cost of leased space, among other things. 2. Delegated leases should not extend into holdover status. The 2014 Bulletin states that a lease in holdover status, or an agency occupying a building or space with no lease because it has expired, is in violation of the lease delegation authority. 3. Delegated leases should not be extended unless necessary to avoid a holdover. GSA’s leasing desk guide states that short-term lease extensions should only be used as a last resort because they typically cost more, among other reasons. The post award documents that agencies submit into G-REX do not allow GSA to track agencies’ performance in meeting these management goals. For example, G-REX does not calculate when lease rates are at or below private sector rates. GSA officials said that GSA does not track the performance of agencies with delegated leasing authority against these three management goals because it is primarily the agencies’ responsibility to ensure they meet them. However, the four agencies with delegated leases that we reviewed did not always meet GSA’s three goals. Officials from two of the agencies we interviewed said that they were unaware of GSA’s performance cost metric for negotiating lease rates at or below private sector rates or that it applied to delegated leases. Consequently, the agency officials did not know if they met it. Since neither G-REX nor the agencies with delegated authority track lease rates in this way, GSA does not know if agencies are meeting GSA’s performance cost metric or, more simply stated, if agencies are negotiating cost-effective lease rates. Regarding holdovers, we found all four agencies in our review were experiencing holdovers, which raises questions about how effective their policies are to prevent them. For example, USDA does not use its lease expiration data in an effective manner to track expiring leases to submit lease delegation applications, according to GSA’s audit of USDA delegated leases. Consequently, USDA had one quarter (1,100 of 4,000) of its delegated leases in holdover status in the past 24 months, according to the GSA report. Furthermore, according to our analysis of agency data, all four selected agencies have expired delegated leases where the agency either has a standstill agreement with the landlord, or is simply in holdover status. For example, VA had approximately 10 percent of its delegated leases in holdover status in fiscal year 2018. Regarding extensions, according to G-REX data, almost half of all approved delegated lease authority requests from fiscal year 2016 to fiscal year 2018 were for lease extensions, which goes against GSA’s goals. Officials from three of the four agencies in our review said that they use extensions because they need more time to develop the agency’s space need requirements for a new delegated lease, and they might not have the time to do so before the current delegated lease’s expiration date. GSA staff stated that if an agency has a large number of extensions or holdovers, it denotes that the agency may not be monitoring its leases and as a result is not fully aware of expiring delegated leases. Tenant agencies agree that lease extensions are often not in the best financial interest of the federal government because they are not open to competition, according to this previous work. For example, the USDA’s delegated lease site in Coquille, Oregon was extended without competition for 45 years. USDA officials agreed this was not in the best financial interest of the federal government and that delegated leases should be opened for competition after 20 years. Lease extensions and expired leases in holdover or standstill status are inefficient and costly for the federal government for two reasons. First, without competition among landlords, an agency may be unable to meet the goal of negotiating a lease rate at or below the private sector rate. Second, we have previously reported that the short-term nature of holdovers and standstill agreements creates uncertainties, which can make it challenging for agencies to plan and budget for space needs and difficult for lessors to secure financing. Moreover, we have reported that holdovers can create an adversarial relationship with building owners, prompt concerns about an agency’s portfolio management, and create unnecessary uncertainty for relevant stakeholders. We also noted that holdovers and standstills occur for a variety of reasons, including challenges finalizing space requirements, tenant agency labor shortages, and the sometimes lengthy duration of the leasing process. Absent procedures to regularly track the performance of agencies with delegated leasing authority to ensure cost effectiveness and limit the use of extensions, holdovers, and standstill agreements, GSA cannot ensure that these agencies are meeting the management goals of the delegated leasing program. When previously reviewing GSA’s management of its own portfolio, we found that tracking and monitoring several measures over the life cycle of the lease acquisition process may help reduce the overall number of holdovers and extensions. For example, using a tracking tool to alert management of delegated leases approaching their expiration date could help to reduce the reliance on extensions and to prevent holdovers and standstill agreements. Regularly tracking agencies’ ability to meet key management goals would alert GSA to holdovers and heavy use of extensions that are not cost effective and may warrant additional oversight. GSA requires that agencies submit an acquisition plan for their lease when requesting delegated leasing authority, but GSA does not systematically ensure that the subsequently executed leases follow those plans and meet program requirements. Agencies submit an acquisition plan along with other documents in order to request delegated leasing authority. GSA officials told us that they review requests for delegated leasing authority by verifying that all required information and documents are uploaded into G-REX and that a lease consistent with the acquisition plan would meet program requirements. GSA officials noted, however, that the acquisition plan is strictly a planning tool and that the terms and conditions are subject to change when finalizing the lease. When approving a request for delegated leasing authority, GSA issues an executive summary and approval letter to the agency identifying the parameters of the leasing authority delegated, such as space limits. Once the agency with delegated leasing authority awards the lease, the agency is required to upload to G-REX certain post award documentation, including the executed lease, within 30 days. These documents provide insight on final lease terms such as square footage, lease expiration date and cost, which may differ from the acquisition plans agencies submitted when applying for delegated leasing authority. We have previously identified risk-based assessment and mitigation as leading practices for providing assurances to managers that they are complying with existing legislation, regulations, and standards and effectively preventing, detecting, and responding to potential fraud, waste, and abuse. Assessing a selection of delegated leases’ post award documents could serve as an early warning system for managers to help mitigate or promptly resolve issues through corrective actions and ensure compliance with existing legislation, regulations, and standards. However, GSA officials said that they do not have a process in place to systematically review post award documents from delegated leases to determine whether the lease awarded met program requirements and were within the authority granted in the approval letter. We found that as of November 2018, GSA had reviewed approximately one percent of the post award documents agencies submitted into G-REX, according to G- REX data. GSA officials told us they had not developed a system for reviewing post award documents because GSA views it as primarily the responsibility of the agency with the delegated authority to ensure they comply with the 2014 Bulletin’s post award requirements. Further, according to GSA officials, GSA’s primary role in the lease delegation process is to review and approve requests for delegated leasing authority. As a result, GSA officials have determined that regularly reviewing post award documents is not the best use of their already constrained resources. However, GSA’s reliance on agencies to comply with all requirements absent any mechanism to ensure post award accountability could allow agencies to lease space outside of the delegated authority granted to them. GSA’s previously mentioned, ad hoc audits of USDA and BIA delegated leases reinforced the need for strengthened oversight to ensure that leases meet requirements, as both audits found problems. For example, the DOI’s OIG confirmed in 2014 GSA’s findings that BIA approved $32.7 million in delegated lease agreements that exceeded GSA square footage and purchase approval limits. GSA’s review of USDA’s delegated leases also found that approximately 540 lease files were missing the awarded lease documents in G-REX. In addition, the review found that no file, in its sample of 27 lease files, had all the required documents uploaded in G-REX. Furthermore, among our selected delegated leases, we found instances of agencies not uploading post award documents to G-REX after the lease was awarded. For example, one delegated lease file in our sample was still missing the executed lease over 2 years after the lease was signed. If post award documents are not uploaded as required, GSA may not even have the documentation necessary to determine if a delegated lease met program requirements and was within the authority granted. Even if all post award documents are uploaded, GSA still cannot verify that the leases were executed within the parameters of the granted delegated leasing authority and in accordance with program requirements without a systematic process for reviewing post award documentation. For example, as noted above, if GSA assessed a selection of delegated leases’ post award documents, it may have identified the missing executed lease and other deficiencies noted above and been able to notify the agency. Further, GSA cannot ensure that agencies are preventing fraud, waste, or abuse. GSA oversees the delegated leasing program and is a guarantor of the government’s monetary obligations under a delegated lease in the event of default. However, if not properly managed, delegated leases run the risk of not being cost effective for the federal government. GSA has taken some actions to address previously identified issues with the program, but its current oversight and management of the program is compromised by a lack of key processes that make it unable to ensure the program is working as intended. Because GSA has not determined how to reasonably reconcile G-REX and FRPP data, pursuant to its own procedure, it is missing oversight opportunities, such as finding leases with annual rent or lease terms that do not meet program requirements. Additionally, without a way to regularly assess agencies’ policies and procedures to manage their delegated leasing activities or track their performance in meeting key management goals, GSA cannot be sure agencies can sufficiently manage their leases or secure cost-effective rates. Periodic reviews of an agency’s ability to manage its delegated leasing activities would help GSA ensure that it is providing this authority to agencies that can manage it effectively and efficiently. Finally, without a systematic process for monitoring a selection of submitted post award documents to help identify and promptly resolve issues and ensure compliance with existing legislation, regulations, and standards, GSA cannot ensure that delegated leases comply with the terms of the delegation and the program is free from fraud, waste, and abuse. We are making the following four recommendations to GSA The Administrator of GSA should take steps to reconcile G-REX and FRPP to the extent practical. (Recommendation 1) The Administrator of GSA should develop a process for assessing at regular intervals, such as annually, agencies’ policies and procedures for managing their delegated leasing activities. (Recommendation 2) The Administrator of GSA should develop a process that would allow GSA to track agencies’ progress in meeting GSA management goals, such as cost effective lease rates, and avoiding holdovers. (Recommendation 3) The Administrator of GSA should develop a systematic, risk-based process for monitoring a selection of submitted post award documents. (Recommendation 4) We provided a draft of this product to GSA, VA, USDA, Interior, and Commerce for review and comment. In its comments, reproduced in appendix I, GSA concurred with the recommendations. GSA and USDA provided technical comments, which we incorporated as appropriate. VA, Interior, and Commerce did not have comments. We are sending copies of this report to the appropriate congressional committees, the Administrator of the General Services Administration, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or RectanusL@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Lori Rectanus, (202) 512-2834 or RectanusL@gao.gov. In addition to the individual named above, other key contributors to this report were Keith Cunningham, Assistant Director; Sarah Jones, Analyst in Charge; Eli Albagli; Lacey Coppage; Josh Ormond; Colleen Taylor; Michelle Weathers; and Elizabeth Wood.", "summary": "As the federal government's landlord, GSA is authorized to lease property to accommodate federal agencies. It can also delegate this authority to other agencies, though GSA is still responsible for overseeing the delegated leasing program. However, prior audits found problems with delegated leasing, including excessive rental rates and insufficient documentation to support that the government received a fair and reasonable price for the lease. GAO was asked to review GSA's delegated leasing program. This report examines: 1) GSA's efforts to reform its delegated leasing program; 2) the extent to which GSA assesses agencies' policies, procedures, and performance in managing their delegated leasing activities; and 3) the extent to which GSA ensures delegated leases meet requirements. GAO reviewed federal statutes and regulations, and GSA's guidance and data on delegated leases. To illustrate how GSA approves and oversees delegated leases, GAO judgmentally reviewed 17 delegated leases selected to include lease contract value, type of lease, and agencies with high number of delegated leases. GAO interviewed officials from GSA and the four agencies associated with GAO's selected delegated leases. The General Services Administration (GSA) has taken steps to reform its delegated leasing program, but data reliability issues remain. For example, GSA created GSA's Real Estate Exchange (G-REX) to centralize delegated lease requests and approvals, but GAO found G-REX had incorrect information on lease rental values and rates—reporting rates 12 times higher than they actually were. Moreover, GAO found that GSA was not annually reconciling data between G-REX and the government-wide real property database, per GSA's own procedures. GSA officials said that their past efforts to fully reconcile the data were unsuccessful but acknowledged there may be ways to compare the data to improve the reliability of both datasets. Until GSA clarifies what it can do to partially reconcile the data sets, it is not obtaining the intended benefits of this data validation exercise. GSA does not know if agencies have the ability to manage their delegated leasing activities because it does not regularly assess their policies and procedures, or their performance in meeting GSA's management goals, such as avoiding extensions. GSA procedures state that GSA will consider the agency's organizational structure and ability to meet certain GSA performance measures prior to granting requests for delegated leasing authority. Moreover, federal internal control standards call for agencies to design control activities to better manage the program. However, GSA officials said that GSA relies on the agencies to oversee their own delegated leases. Nevertheless, GAO found instances of inadequate policies and procedures at one agency in managing its delegated leasing activities. Further, all 4 agencies had delegated leases that were in holdover status (occupying a space beyond the expiration of the lease term), which violates program requirements. Because GSA does not regularly assess agencies' procedures or performance, it cannot ensure that agencies are effectively managing their delegated leasing activities. GSA cannot ensure that the leases agencies execute under delegated authority meet program requirements and are within the authority granted because it lacks key procedures to do so. GAO found that GSA had only reviewed 1 percent of the post lease award documents agencies had submitted, and in some cases, agencies had not submitted required documentation. GSA officials said the agencies are responsible for ensuring that documents are submitted and requirements are met. However, a risk-based assessment of a selection of delegated leases' post award documents can provide assurances that agencies comply with existing regulations and prevent potential fraud, waste, and abuse. Because GSA did not have a process to systematically review these documents, GSA is unable to ensure that delegated leases meet requirements and that agencies are positioned to prevent fraud, waste, or abuse. GAO recommends that GSA (1) reconcile its databases; (2) regularly assess agency procedures for managing delegated leasing, (3) track agency performance, and (4) develop a review process for post lease award documents. GSA agreed with the recommendations.", "document_type": "gao"}
{"report": "The Airline Deregulation Act of 1978, which established the EAS program, specifies that if DOT determines that air service will not be provided without subsidy, DOT shall use EAS program funds to award a subsidy to a carrier willing to provide service. As of October 1, 2018, 108 communities within the contiguous United States (as well as 65 in Alaska and Hawaii) were receiving EAS (see fig.1). To be eligible for EAS, a community must: be located more than 70 miles from the nearest large or medium hub require a subsidy per passenger of $200 or less, unless the community is more than 210 miles from the nearest large or medium hub airport or unless DOT decides to issue a waiver; have a subsidy per passenger of less than $1,000 during the most recent fiscal year at the end of each EAS contract, regardless of the distance from a hub airport; have had an average of 10 or more enplanements per service day during the most recent fiscal year, unless the community is more than 175 driving miles from the nearest medium or large hub airport or unless DOT is satisfied that any decline below 10 enplanements is temporary; and have received subsidized EAS in fiscal year 2011 or were provided a 90-day termination notice by an air carrier, and the Secretary required the air carrier to continue such service to the community. EAS is funded through appropriations from a combination of discretionary funding provided through annual appropriations acts, and overflight fees, which are collected by the Federal Aviation Administration (FAA) from foreign aircraft traveling over U.S. airspace without taking off or landing in the United States. Historically, the amount of overflight fees provided to EAS has been $50 million per year, but the FAA Modernization and Reform Act of 2012 directed that all overflight fees be directed to EAS, an action that which resulted in an increased proportion of the program being funded by overflight fees (see fig. 2). The minimal level of service each community is required to receive—the minimum number of roundtrips and passenger seats that must be provided, certain characteristics of aircraft to be used, and the maximum number of permissible stops to a medium or large hub airport—are all established in law. In general, current law requires that an EAS carrier provide the following: service to a hub airport, defined as an FAA-designated medium- or large-hub airport; two daily round trips, 6 days a week, with not more than one intermediate stop to the hub; flights at reasonable times taking into account the needs of passengers with connecting flights and at prices that are not excessive compared to prices of other air carriers for like service between similar places; service in an aircraft with an effective capacity of at least 15 passengers, under certain circumstances, unless the affected community agrees in writing to the use of smaller air craft; service in an aircraft with at least two engines and using two pilots; and service with pressurized aircraft under certain circumstances. DOT awards contracts to individual air carriers to serve EAS communities on a rolling basis throughout the year. According to DOT officials, DOT takes the following steps: DOT issues a request for proposals to all carriers to provide air service to an eligible community. Air carriers submit proposals that include the size of the aircraft to be used, the frequency of service, potential hubs, and the amount of subsidy required. Air carriers request subsidies at a level to cover the difference between their projected revenues and expenses, and to provide a profit. While there are no limits on the amount of subsidy that a carrier can request in its proposal, a community can become ineligible for EAS if the annual subsidy exceeds $1,000 per passenger regardless of distance from the nearest hub airport or $200 per passenger if it is located fewer than 210 miles from the nearest large or medium hub airport. DOT reviews the proposals and selects an air carrier to provide air service to the community, generally for a contract period ranging from 2 to 5 years. When selecting air carriers to provide service to EAS communities, DOT is directed by statute to consider five factors: service reliability, contracting and marketing arrangements with a larger carrier at the hub, “interline agreements” with a larger carrier at the hub, whether the air carrier has included a plan in its proposal to market its service to the community, and user preferences. In addition, the Secretary may consider the relative subsidy requirements of the carriers. By statute, the subsidy is set at an amount to cover the difference between the carrier’s projected costs of operation and its expected passenger revenues, while providing the carrier with a profit element typically equal to 5 percent of total operating expenses. DOT awards a contract and pays air carriers based on the number of flights completed in the prior month. Air fares on EAS routes are set at the air carrier’s discretion without input from DOT. In 2003, the Vision 100—Century of Aviation Reauthorization Act established the AEAS, which allows communities to forgo subsidized EAS for a prescribed amount of time in exchange for a grant to spend on options that may better suit their transportation needs. For example, a community under AEAS may use the grant to purchase an aircraft to meet transportation needs or may receive some flexibility on operating requirements. Under AEAS, the community must still adhere to EAS eligibility requirements, and the maximum annual grant amount may not exceed the annual EAS subsidy at the time of application to the program or what DOT would pay to maintain EAS at the eligible community. For example, if an air carrier received a subsidy of $1 million per year to serve a community and the community decides to leave EAS and enter AEAS, then the grant amount to the community under AEAS may not be more than $1 million per year. As of September 2019, 8 of the 108 EAS communities in the contiguous United States were participating in the AEAS. In addition, federal funds are available to support airports—including airports that receive subsidized EAS—through the Airport Improvement Program (AIP). AIP grants are awarded to public entities to make capital improvements—such as runway and taxiway improvements. The level of AIP funding that an airport receives is based on the number of annual enplanements at the airport. For fiscal year 2018, airports with 10,000 or more passengers were entitled to at least $1 million; airports with between 8,000 and 10,000 passengers were entitled to $600,000, and airports with fewer than 8,000 passengers were eligible for $150,000. Thus, the number of enplanements at an airport receiving subsidized EAS may affect the amount of AIP funds for which the airport is eligible. Officials from the 14 communities receiving EAS that we interviewed cited several economic benefits of the local air service they receive: Economic development, including the ability to attract and retain businesses and professionals: When asked what benefits they received from local air service, officials from all 14 communities mentioned that having access to reliable air service through EAS was crucial for economic development in their community, including the ability to attract and retain businesses and professionals. In three of the communities, officials told us that the first question a business asks when deciding to locate to the area is if air service is available. Increased tourism to the community: When asked about benefits, officials in 6 of the 14 communities mentioned that EAS helps to bring tourists to the community. One community official told us that having access to air service through EAS was a key factor in the community’s being selected to host the Boy Scout Jamboree, which brought 8,000 volunteers and 45,000 Boy Scouts to the area. Creation of jobs related to air service: Officials from 4 of the 14 communities also mentioned that EAS brought jobs related to air service to the community, including TSA personnel, airport employees, airline employees, and concessionaire employees such as those at fixed-based operators and airport restaurants. In addition, some community officials told us that having air service in the community creates other types of jobs and supports area industries, such as hotels, restaurants, and rental car companies. Further, community officials told us that EAS provides other benefits in addition to economic benefits. Officials from 11 of the 14 communities mentioned that EAS allows residents to more easily travel and be connected to the rest of the world. Officials in 3 communities said that residents use EAS to travel to larger cities for medical services that are not available locally, such as procedures and appointments with specialists. Officials whom we interviewed in three communities that lost eligibility for subsidized EAS told us that losing air service has had a negative economic effect. For example, officials in one community told us that the lack of air service has decreased the ability of local businesses, hospitals, and colleges to recruit for professional-level jobs, such as physicians and professors, who have travel needs to maintain proficiency in their field. An official from another community told us that losing EAS led to decreased enplanements, which, in turn, reduced the amount of AIP funding that the airport receives. With less AIP funding, the airport is not able to pay for improvements that would attract or enable air carriers to serve the community. Most of the studies we reviewed found there to be a correlation between aviation activity and economic development. Specifically, several of the findings indicate that greater aviation activity in a region is correlated with some increase in the growth in population, employment, or per capita incomes. The size of the influence in these findings was relatively small but statistically significant. For example, one study found that a 1 percent rise in passengers per capita was associated with 0.055 percent rise in output per capita and another study found that a 10 percent increase in number of nonstop destinations served from an airport was associated with a 0.13 percent increase in employment and a 0.2 percent increase in average wage. One study that specifically examined the effect of subsidized air service found that the availability of EAS was related to a small but statistically significant increase in per-capita income in the local market. Specifically, this study found that a 1 percent increase in traffic at an airport receiving subsidized EAS was related to a 0.12 percent increase in per-capita income. Further, another study that focused solely on small airports found airport activity was associated with higher per-capita income, while another study found that more rural areas experienced an even greater benefit of nearby aviation activity than did more urban areas. However, two of the studies we reviewed found that the effect of aviation activity on local economic factors may be greater in areas with larger airports, which tend to be in larger metro areas, than in areas with smaller airports. Since 2010, four statutory changes and a change in DOT’s enforcement policy have limited the number of communities that are eligible to receive EAS. (See app. II for a detailed list of statutory changes.) The Airport and Airway Extension Act of 2011 prohibited DOT from continuing to provide subsidies to communities with annual per- passenger EAS subsidies of over $1,000, regardless of their distance from the nearest hub airport. The FAA Modernization and Reform Act of 2012 removed eligibility for communities within 175 miles of a large- or medium-hub airport that do not have an average of least 10 enplanements per day during the most recent fiscal year, unless DOT grants them a waiver. The FAA Modernization and Reform Act of 2012 removed EAS eligibility for communities that did not receive EAS between September 30, 2010, and September 30, 2011, thus preventing further growth of the program. This limitation does not apply to Alaska and Hawaii. The number of communities that would otherwise be eligible for service if not for this provision is unknown. We are aware of at least one community that lost eligibility based on this requirement. However, DOT has not been able to determine how many communities fall into this category due to a number of complicating factors, including an unclear count of the number of communities that were initially eligible for EAS in January 1979 and changes in eligibility in the intervening years. The Consolidated Appropriations Act of 2014 and subsequent appropriations acts required the Secretary of Transportation to negotiate a local cost share with communities located less than 40 miles from the smallest hub airport before entering into a new contract using EAS subsidies. Two communities in the contiguous United States—Pueblo, Colorado and Lancaster, Pennsylvania—were initially subject to this provision. Currently, Lancaster, Pennsylvania is the only community in the contiguous United States subject to the provision. In October 2014, DOT issued a Notice of Enforcement Policy stating that it would start enforcing the annual subsidy-per-passenger cap of $200 for communities located less than 210 miles from a medium- or large-hub airport after September 30, 2015, thereby limiting the number of communities eligible for EAS in 2016. However, DOT may grant a waiver to communities that have not met the cap. We also identified two statutory changes since 2010 that increased the flexibility of air carriers’ operations for the EAS program, and one that automatically grants waivers for the $200 subsidy-per-passenger cap to communities that meet certain requirements. The Consolidated and Further Continuing Appropriations Act of 2012 and subsequent appropriations acts eliminated the requirement that aircraft providing service under the EAS program have a minimum 15- seat passenger capacity. Officials from about half (8 of 17) of the communities that we interviewed were in favor of the elimination of this requirement. As a result of this change, the number of EAS communities in the contiguous United States receiving service with eight- or nine-seat aircraft increased from 23 percent (25 of 107 communities) in 2010 to 47 percent (50 of 107 communities) in 2019. The FAA Reauthorization Act of 2018 explicitly allowed the Secretary of Transportation to consider the flexibility of current operational dates and airport accessibility when issuing requests for proposal of EAS at seasonal airports. DOT had already been considering seasonal service for some communities. Two of the communities that we interviewed—Bar Harbor, ME, and Cody, WY—have seasonal EAS because the number of passengers fluctuates during different times of the year. The FAA Reauthorization Act of 2018 required DOT to automatically grant waivers for annual subsidy-per-passenger cap of $200 if (1) a community’s subsidy per passenger for a fiscal year is lower than any of the previous 3 fiscal years or (2) if the subsidy per passenger for a fiscal year is less than 10 percent higher than the highest subsidy per passenger for the previous 3 fiscal years. The Secretary may only waive this subsidy cap once per community. According to DOT, it began implementing this provision in 2019 using fiscal year 2018 data. As described earlier, DOT is allowed to waive some eligibility requirements. DOT can grant waivers to communities for (1) not meeting the 10-enplanements per-day requirement or (2) exceeding the $200 subsidy-per-passenger cap in the prior fiscal year. There are several steps that DOT generally follows when granting EAS waivers: DOT collects information from the prior fiscal year to determine which communities no longer meet EAS eligibility requirements. DOT issues a “show cause” order that directs the EAS community or other interested persons to submit information to show why DOT should not terminate the eligibility of the community. The communities that are listed in the “show cause” order may provide DOT with information demonstrating that they met EAS requirements or submit a petition to DOT that demonstrates that the community’s failure to meet eligibility requirements is a temporary situation in order to retain eligibility. If the community does not provide new information to demonstrate that they met EAS requirements or submit a petition, then the community’s eligibility for EAS is terminated. DOT then issues a final order that changes its initial determination, grants a waiver to the community, or terminates the community’s eligibility for EAS. If a community disagrees with DOT’s decision to terminate eligibility, it may submit a petition for restoration. As a result of these changes in statute and enforcement policy, 12 communities lost eligibility for EAS since 2010 and either were not eligible for a waiver, did not apply for one, or applied for a waiver and were not granted one (see table 1). While some communities lost eligibility for EAS, many communities that did not meet eligibility requirements since 2014 continue to receive EAS because they were granted at least one waiver from DOT. From fiscal year 2014 through fiscal year 2019, DOT granted a total of 110 waivers to 37 communities—about one-third of the number of communities currently in the program (see fig. 3). The number of communities that received waivers in recent years has increased during this time period, in part due to DOT’s decision to enforce the $200 subsidy-per-passenger cap. DOT granted waivers to 15 communities because they experienced a hiatus in service during the year that resulted in the community’s not meeting the 10 average daily enplanements requirement or exceeding the $200 subsidy-per-passenger cap. Of the communities that petitioned for waivers, DOT granted waivers to all but three—Jamestown, NY; Franklin/Oil City, PA; and Hagerstown, MD. Jamestown did not meet the 10 enplanements per-day requirement and exceeded the $200 subsidy cap in fiscal year 2016. DOT officials did not grant a waiver to Jamestown because they did not think there was sufficient evidence that Jamestown would ever have enough service to meet eligibility requirements. Franklin/Oil City has not met the 10 enplanements per-day requirement in each year since fiscal year 2013 and has exceeded the $200 subsidy cap in each year since fiscal year 2015. DOT did not grant a waiver to Franklin/Oil City because of its continued non- compliance with these requirements and its proximity to a medium hub airport. Pittsburgh International Airport is 85 driving miles away. In September 2019, Franklin/Oil City filed a petition to DOT for reconsideration. DOT denied the petition. Hagerstown has not met the 10 enplanements per-day requirement since fiscal year 2013 (except fiscal year 2016), and has exceeded the $200 subsidy cap each fiscal year since fiscal year 2015. DOT did not grant a waiver to Hagerstown because of its proximity to a large hub airport— Hagerstown is less than 70 miles from Washington Dulles International Airport—and the fact that there was not sufficient evidence to indicate that Hagerstown would be able to meet eligibility requirements in the future. In August 2019, Hagerstown filed a petition to DOT for reconsideration. DOT denied the petition, and Hagerstown filed suit to challenge the decision in federal court. Athens, GA, which did not meet the 10-enplanements per-day requirement, was eligible to submit a waiver request but did not do so. The number of communities in the contiguous United States receiving EAS changed little since the beginning of fiscal year 2010 to the beginning of fiscal year 2018—from 104 on October 1, 2009, to 109 on October 1, 2017. However, program expenditures for EAS communities in the contiguous United States have increased from approximately $161.3 million in fiscal year 2010 to $276.9 million in fiscal year 2018—an increase of nearly 72 percent (see fig.4). Some of the increased program expenditures were due to increased costs of certain critical resources over the last several years, such as pilots’ salaries. However, even when total expenditures are adjusted for the effect of inflation, expenditures still rose substantially. Notably, we found a nearly 50 percent increase in spending that is not accounted for by the general rise in prices over these years, despite a roughly consistent number of communities served by the program. According to DOT officials, some of the cost increase is related to factors that also affected the rest of the airline industry, such as increased costs for pilots, flight crew, and mechanics. For example, in 2018 we found that compensation for commercial airline pilots has increased in recent years, most noticeably in new-hire compensation at regional airlines. Our analysis of Bureau of Labor Statistics data from 2012 through 2017 showed that the median wages in the pilot occupation increased by approximately 2.4 percent per year, while wages for all occupations increased by about 1 percent per year over this period. DOT officials told us that other factors contributing to increased program costs are more specific to EAS. For example, some regional airlines that serve EAS communities have experienced financial difficulties, and in some cases, contracts with new carriers have increased in price to factor in costs associated with replacing the previous carrier’s service. DOT officials noted that larger air carriers that serve many markets have more options available to help offset industry-wide cost increases, such as increasing fares on more commercially viable routes, whereas some of the smaller carriers that primarily service EAS markets have fewer options on the revenue side to offset cost increases. Community officials and air carriers that we interviewed described several challenges they face with regard to maintaining viable service. Many of these challenges compound each other. Quality of Service: According to officials from the communities we interviewed, an air carrier provides good quality service to an EAS community when the service is reliable (i.e., flights are on time, at convenient times, and are not frequently cancelled), offers connections to multiple locations, and includes benefits such as the ability to easily catch a connecting flight and check bags to the final location. Some community officials also said good quality service involves seamless connections to large hubs with regional jets. When a carrier does not provide what communities and passengers see as quality service, the number of enplanements decreases because people stop using the service. As a result, the carrier may decrease the number of flights per day to make the service financially viable. However, the reduction in frequency could further degrade the quality of service. Carrier representatives explained that many factors affect the quality of service carriers are able to provide and communities explained that unreliable service can result in several problems for them. Decline in Enplanements: Officials in most of the communities (15 of 17) said that a lack of quality service from the carrier had been a challenge and in many instances (14 of 17) had led residents to opt to travel to an alternative larger airport for service. The resulting decline in the number of enplanements can put a community at risk of losing EAS eligibility because it may not be able to achieve an average of at least 10 enplanements per service day or stay under the $200 subsidy-per-passenger cap. Officials from one community said that its EAS carriers’ cancelled flights and lack of interline agreements with mainline airlines had resulted in customers choosing to drive 80 miles to fly out of a large hub airport rather than use the local airport. Providing Service within Subsidy Caps: Four of the carriers we interviewed said that increased costs—such as those resulting from increased pilot wages—make it difficult to provide service within the subsidy caps, which have not been increased to account for inflation. An official from one carrier said that factors such as the increasing costs for pilots and an insufficient number of aircraft operating with less than 50 seats make it difficult for a community airport to comply with the $200 subsidy-per-passenger cap. According to representatives of the carrier, in some instances, they are paying their pilots 75 percent more than they were 5 years ago. They said that to compensate, the carrier may have to raise fares, a step that could lead to losing passengers and potentially put communities at risk of losing eligibility for EAS. Loss of Customers’ Confidence: Three of the carriers we interviewed said that when they were selected to replace carriers that had not provided reliable service to a community, it took time to regain the community’s confidence and attract people to use their EAS air service. If these air carriers had not been able to regain the community’s confidence and increase enplanements, the community may have lost eligibility for EAS. Loss of AIP Funding: A decline in the number of enplanements may also lead to a reduction in AIP funding available to the airport. AIP funding is important for small communities that have fewer financial resources than large- or medium-sized airports. AIP funding can help airports make improvements that could attract more business, such as from commercial and business aviation. Pilot shortage: Aviation stakeholders have voiced concerns that there is an insufficient supply of qualified pilots to support current and future demand from U.S. regional and mainline airlines. In May, 2017, the Working Group on Improving Air Service to Small Communities found that as a result of the pilot shortage, there were too few pilots to fly all the EAS routes. In June 2018, we found that labor market indicators for the pilot occupation were consistent with the existence of a pilot shortage. Carriers and community officials that we interviewed cited the following as issues related to the pilot shortage. Difficulty Retaining Pilots: Officials from 6 of the 10 carriers we interviewed said that it has been a challenge to retain sufficient pilots to provide the air service they have committed to providing under EAS. Pilots often start their careers with smaller air carriers that may serve EAS communities, and after a few years in the business, pilots are hired by larger airlines offering higher pay and more opportunities for advancement. Officials from 3 of the 10 carriers we interviewed said that they have responded to the pilot shortage by operating eight- or nine-seat aircraft under Part 135 regulations, which allows them to use pilots that have less flight time as first-officers. This increases the pool of pilots who can fill first-officer positions and gives these pilots the opportunity to build flight hours toward their Airport Transport Pilot license. Reduced Service Quality: Officials from 15 of the 17 communities we spoke with said that a shortage of pilots has been a challenge. Specifically, the pilot shortage has resulted in a reduction in service quality for some EAS communities because the air carrier has not been able to attract enough pilots to provide reliable service. Six of 17 communities told us that their enplanements declined and that some had lost service for a period of time due to a lack of pilots. For example, an official from one community said their carrier ended service to the community in 2014 due the industry-wide pilot shortage. Airport costs: Air carriers must pay fees to use airport facilities. Fees are charged for landing, counter and gate space, parking, and other airport facilities. These varied fees are part of carriers’ operating costs. Officials from 3 of the 10 carriers we talked to said that these airport costs may be difficult to cover because carriers serving the EAS program use relatively smaller aircraft with fewer passengers, and therefore, the carrier must charge more per passenger to cover the costs. For example, an official from one carrier we interviewed said that a community wanted to have an EAS flight that flew into Las Vegas; however, the airport in Las Vegas charged a single-aisle 9-seat aircraft the same landing fee as any other single-aisle aircraft, some of which can hold hundreds of passengers. Production and supply of small aircraft: Because there is a lack of availability of aircraft between 19 and 50 seats, in some cases, DOT, airlines, and communities have to choose service with a plane that is either too small or too large for demand. Manufacturers have said they are generally not producing this size aircraft because there is less demand and higher costs since they must certify them under Part 25 regulations for scheduled commercial service as opposed to the lower costs incurred under Part 23 regulations. Insufficient or Excess Capacity: Officials from 12 of the 17 communities we interviewed said that the declining production and supply of 19- to 50-seat aircraft has been a challenge for the EAS program. Officials from 2 communities we interviewed said they have moved to larger 50-seat aircraft, which means the communities might have too much capacity. On the other hand, officials from 11 of the 17 communities we interviewed expressed concerns about receiving service from a carrier that operates aircraft with less than 15 seats because, according to six communities we spoke with receiving air service from a carrier that only operates eight- or nine-seat aircraft may not provide sufficient capacity to allow the community to fulfill the EAS annual enplanement requirements, and thus, the community could lose eligibility for EAS. In addition, officials from 5 of the 17 communities were concerned that some people have an aversion to or difficulty getting into small aircraft that could deter them from using the service. Financial Effects on Air Carriers: Officials from 5 of the 10 carriers we interviewed said that the lack of available aircraft between 19 and 50 seats is a challenge. For example, an official from one carrier was concerned that operating eight- or nine-seat aircraft may limit their ability to serve EAS communities whose enplanements are increasing because the carriers would have to add seat capacity either through increased frequency of flights or larger aircraft they do not currently own in order to decrease the subsidy-per-passenger costs. However, if the carrier uses an aircraft with 50 or more seats, the carrier must have sufficient increasing demand to fill that plane on a regular basis to justify the capital expenditure and increased costs to operate. Furthermore, according to officials from another air carrier, eight- or nine-seat aircraft were not designed to operate with the frequency that small carriers are using them, which can reduce reliability and increase maintenance and operating costs. Driving Distance Calculation: While communities that we interviewed cited several specific benefits of the local air service they receive, as previously discussed, some expressed concerns about specific aspects of the program. Officials from 5 of the 17 communities we interviewed said that DOT’s calculation of the shortest driving distance between the community and the nearest large- or medium-hub can affect their eligibility requirements. DOT relies on the driving distance calculation to determine which communities are subject to the 10-enplanement and $200 subsidy-cap requirements. According to community officials, the easiest, safest, and quickest route from the community to the airport may be further than what DOT has calculated as the shortest driving distance, which could make the community exempt from these requirements. For example, one community official we spoke with told us that most people in the community take the expressway to the nearest hub airport, which is further from the center of the community to the airport than the two-lane route DOT uses in its calculation. An official from another community we interviewed said that DOT should take into account the time required to drive the route and the safety of the roadway when calculating the distance for EAS eligibility. The official explained that the route should take 2 hours to drive but often takes much longer due to traffic and delays, and expressed concerns that the route is very dangerous. Carrier Contracts: Contracts in the EAS program are in the form of DOT Orders announcing the carrier selected to serve a route and the subsidy awarded to the carrier. The Orders contain information such as the annual subsidy rate, the time frame for service, and various carrier requirements. Officials from 6 of the 17 communities we interviewed said that the structure of DOT’s contracts with EAS carriers can present a challenge because the communities feel they provide little to no leverage over a carrier that provides unreliable service. Officials from five communities said that EAS contracts do not include performance requirements or have penalties if the carrier does not meet service quality standards or targets. As previously discussed, officials from 15 of the 17 communities we interviewed told us that they had not received quality service at some point in the EAS program, which can result in declining enplanements and, ultimately, the community losing eligibility for the EAS program. However, if a community wants to have DOT cancel a contract, the community might lose air service if there is not another carrier interested in providing service. DOT has stated that the EAS program already provides financial incentives for carriers to provide reliable service. For example, DOT states that its “no fly, no pay” policy encourages carriers to complete flights because DOT reimburses carriers only for flights that they actually operated. Further, DOT also believes that carriers have financial incentives to increase completion rates above the rate estimated in their proposals. Because carriers frequently account for predictable flight cancellations they have an incentive to beat their estimate. Furthermore, carriers have the financial incentive to provide quality service to avoid losing enplanements and maintain a financially viable service. The communities and air carriers we interviewed suggested potential reforms to EAS that they believed would improve service to their communities. Several of these changes would likely result in increased program costs. Change the subsidy cap: Officials from two communities and four carriers we interviewed said that the $200 per-passenger-subsidy cap should be changed, either by indexing the cap to inflation or increasing the cap temporarily for a community to allow a carrier more flexibility to develop a market for new service in a community or to account for higher labor costs. Since the subsidy cap is established in statute, revising it would require a legislative change. An official from one community said that increasing the cap for inflation would allow a carrier to use a larger aircraft, thereby improving use of the airport. One air carrier official said the cap needs to be increased to reflect rising labor costs. In its October 2014 notice of enforcement policy, DOT said that while it recognized the cap has not kept pace with inflation, the requirements of the statute did not provide DOT with the discretion to adjust the subsidy cap amount or refrain from enforcement. However, DOT issued waivers to 34 communities that did not meet the $200 subsidy cap from 2014 through 2019. If the subsidy cap were tied to inflation since its inception in 2000, the cap would be $283 in 2018. Of the 55 communities that were subject to the subsidy cap in 2018 because they are within 210 miles of a medium- or large-hub airport, 39 were under the subsidy cap and 16 exceeded it. Our analysis shows that if the subsidy cap were adjusted for inflation, an additional 10 communities would fall under the subsidy cap, and only 6 communities would exceed it. See figure 5. Renegotiate EAS agreements: Officials from 3 of the 10 carriers we interviewed said they should be permitted to request additional funds from DOT during the course of a contract. In 2009, we reported that allowing air carriers to renegotiate EAS contracts in response to rising costs would enable carriers to continue rather than file a Notice of Termination. As previously discussed, carriers we interviewed cited airport and operating costs as challenges they have encountered over the course of an EAS contract. Legislation passed in 2003 explicitly provided DOT with the option of adjusting the subsidy paid to an EAS carrier if the carrier’s expenses substantially increased. However, DOT officials said that to date no carrier has petitioned for such an increase. Revise DOT’s calculation of the driving distance: As mentioned earlier, to determine whether an EAS community is subject to the 10- enplanement-per-day and subsidy-cap requirements, DOT must determine the shortest driving distance from the center of the community to the nearest large- or medium-hub airport. Officials from four of 17 communities we interviewed suggested that DOT adjust its calculation to account for local factors, such as the time required to drive the shortest route, the condition of the road, and the most common route that members of the community use to get to the nearest large- or medium-hub airport. Considering these factors could result in communities not being subject to the limit on eligibility of requiring an annual subsidy per passenger of $200 or less, if the more commonly used or faster route is more than 210 miles from the nearest large or medium hub airport. Allow communities to regain eligibility: Officials from two communities and two carriers we interviewed suggested that subject to the availability of funds, communities that lost eligibility for the EAS program should be allowed to regain it if they are having difficulty obtaining air service without a subsidy. Officials from one community and one carrier we interviewed said communities that lost EAS eligibility as a result of unreliable service from their carrier should not be penalized by losing EAS program eligibility. According to DOT, they consider such circumstances when deciding to grant a community a waiver. In other instances, communities lost eligibility because they were not receiving EAS in fiscal year 2011. An official from one carrier suggested communities that regain eligibility could pay a co-share of the subsidy costs, possibly limiting the effect on the cost of the program. Some of the options that communities and carriers suggested, such as revising DOT’s process for carrier selection and restructuring DOT’s contracts with carriers could address the challenges in the EAS program but not necessarily increase program costs. Revise DOT’s process for carrier selection: Officials from 3 of the 17 communities and 4 of the 10 carriers we interviewed suggested that DOT adjust its method for carrier selection to account for factors such as the carrier’s financial viability, ability to comply with enplanement requirements, and agreements with mainline carriers, as well as the number of available pilots and mechanics in order to ensure that carriers are capable of providing good service to EAS communities. In addition, officials from one community also suggested that DOT give more weight to community preferences regarding carrier selection. While DOT is required to consider factors such as service reliability, interline agreements, and carrier financial and operating fitness when selecting a carrier, most of the communities we interviewed cited the quality of service they have received through the EAS program as a challenge. Include performance measures in DOT’s contracts with air carriers: Officials from four communities and one carrier suggested that DOT include performance measures in EAS contracts to ensure carriers are held accountable for providing a given level of service and subject to penalties for not meeting service quality targets. For example, one community official suggested that on-time performance and percentage of flights cancelled could be included as performance measures for EAS carriers. Officials from three communities and one carrier suggested that DOT include more requirements for service to EAS communities. For example, DOT could require that EAS carriers provide service to large-hub airports and have agreements with mainline carriers that could enhance quality of service; however, an official from one air carrier told us the carrier was reluctant to enter into agreements with smaller air carriers that serve EAS communities because they did not want their reputation to be negatively affected if the air carrier did not provide reliable service. An official from another carrier suggested that it is beneficial for carriers to enter into longer contracts because they can spend more time building the air service market for the communities they serve rather than renewing contracts. The officials said that for longer contracts DOT should include performance measures that require the carrier to provide a minimum level of reliable service or lose the route. Limit airport fees for EAS carriers: Officials from 3 of the 10 carriers we interviewed thought DOT should limit fees airports charge to EAS flights, such as landing fees and gate charges in order to increase the financial viability of EAS routes. Airport fees can be based on any number of factors including weight and number of seats on the aircraft. According to FAA’s policy on establishing airport charges, it recognizes airports are allowed to charge fees to help ensure their financial viability and at the same time those fees should be reasonable and not unjustly discriminatory. FAA’s policy further indicates that the issue of rates and charges is best addressed at the local level by agreement between users and airports. Change EAS from a carrier subsidy program to a community grant program: Officials from three communities we interviewed thought that similar to AEAS, DOT could consider providing a grant to a community in lieu of traditional EAS to allow the community more control over the service they receive. For example, an official from one community said that they liked the additional control the AEAS program has given the community over the service and that AEAS gives the community more weight with the carrier when there is a complaint about the service. Officials from three air carriers told us that a potential downside to this option is that it would be more complicated because carriers would need to work with individual communities for payment instead of just DOT. In addition, officials from three communities told us that they lack the technical expertise needed to effectively administer such a program. We provided a draft of this report to DOT for review and comment. DOT provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Transportation, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report please contact me at 202-512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In addition to the contact above, Cathy Colwell (Assistant Director); Stephanie Purcell (Analyst in Charge); Amy Abramowitz; David Hooper; Bonnie Pignatiello Leer; John Mingus; Dominic Nadarski; Malika Rice; Pamela Snedden; Laurel Voloder; and Elizabeth Wood made key contributions to this report.", "summary": "Congress established EAS as part of the 1978 deregulation of the U.S. airline industry. Through the EAS program, DOT provides subsidies to airlines to make service available to communities that airlines would otherwise not serve. Since 2010, several statutory changes have limited eligibility for EAS subsidies by, among other things, changing eligibility requirements. In spite of these changes, program costs have continued to rise, prompting questions about whether additional modifications should be made. A provision in the Federal Aviation Administration Reauthorization Act of 2018 directed GAO to examine several aspects of the EAS program. This report discusses, among other objectives, (1) how federal laws enacted since 2010 have affected air service to communities funded through the program; and (2) challenges that communities and air carriers face with EAS, and options for reform. GAO reviewed relevant federal laws, DOT orders, and DOT program data. GAO also interviewed representatives, such as airport managers and local government officials, from 17 communities that have participated in EAS; representatives from 10 of the 11 air carriers that participate in the program; and DOT officials. This report focuses on the EAS program as it operates in the contiguous United States, as there are different rules for EAS in Alaska and Hawaii. Statutory changes since 2010 have reduced the number of communities eligible for subsidized air service through the Essential Air Service (EAS) program; however, the Department of Transportation (DOT) granted waivers to most of the communities that applied, resulting in little change in the number of communities receiving EAS. In 2012, statutory changes limited eligibility for the program in the contiguous United States to those communities receiving EAS in fiscal year 2011. Further statutory changes set a maximum average per-passenger subsidy, and a minimum number of passengers, that some communities would have to meet to retain eligibility. DOT also resumed enforcing the $200 per-passenger subsidy cap for certain communities. While these changes limited eligibility, in some cases the changes also gave DOT the option of providing waivers—most of which DOT granted. Thus, as noted, the overall number of communities receiving EAS remained about the same; however, EAS expenditures increased from $161 million in fiscal year 2010 to $277 million in fiscal year 2018 (see figure). DOT officials said this increase was due, in part, to factors affecting the entire airline industry, such as increased labor wages. Community officials and air carriers GAO interviewed cited several challenges associated with EAS and suggested options for reform. For example, some carriers said it was difficult to find and retain pilots due to an insufficient supply of qualified pilots. At the same time, pilot wages have increased, making it difficult to provide quality service without exceeding the subsidy caps. Some carriers and community officials noted that the $200 subsidy cap has not changed for several years to account for inflation or these increased costs. To address these and other challenges, stakeholders suggested a number of options, such as indexing the $200 subsidy cap to inflation or allowing communities that lost eligibility to re-apply for the program. Several of these reforms would result in additional program costs.", "document_type": "gao"}
{"report": "The U.S. government implements export controls to manage risks associated with exporting sensitive items while ensuring that legitimate trade can still occur, and to advance U.S. national security and foreign policy objectives. These export controls are governed by a complex set of laws, regulations, and processes that multiple federal agencies administer to ensure compliance. State and Commerce each play a significant role in the implementation of U.S. export controls. State controls the export of sensitive military items, known as defense articles and defense services, such as tanks, fighter aircraft, missiles, and military training, which it lists on the U.S. Munitions List (USML). Commerce controls the export of U.S.-origin items with both commercial and military applications (known as “dual-use” items), such as computers, sensors and lasers, and telecommunications equipment, as well as less sensitive military items, which it lists on the Commerce Control List (CCL). Items subject to State and Commerce jurisdiction are governed by separate laws and regulations. The Arms Export Control Act of 1976, as amended, provides the statutory authority to control the export of defense articles and services, which the President delegated to the Secretary of State. State’s ITAR implements this authority and identifies the specific types of items subject to control in the USML. Within State, the Directorate of Defense Trade Controls (DDTC) is responsible for implementing controls on the export of these items. The Export Control Reform Act of 2018 provides the statutory authority for Commerce to control the export of less sensitive military items not on the USML, dual- use items, and basic commercial items. In general, items subject to the EAR include commodities, software, and technology. Commerce’s EAR, which contains the CCL, implements this authority. Commerce’s Bureau of Industry and Security (BIS) is responsible for administering these export controls. DDTC and BIS control the export of items within their respective jurisdictions by requiring, in certain instances, a license or other authorization to export an item. Whether a license is required will generally depend on the intended destination, end-use and end-user, and the item’s classification. Generally, unless a license exemption or exception applies, exporters submit a license application to DDTC if their items are controlled on the USML, or to BIS if their items are controlled on the CCL. In addition to the shipment of tangible commodities or the tangible or intangible transfer of software or technology outside of the United States, export control regulations also consider the transfer or release of certain U.S. technology or source code to a foreign person in the United States to be an export. These transfers or releases are commonly referred to as “deemed exports” and can take the form of written, oral, or visual disclosure of technology or source code. Under the ITAR, technical data is controlled for all exports, including deemed exports. Under the EAR, technology and source code are controlled for the purpose of deemed exports. Export-controlled items or source code used in U.S. universities’ research activities may be subject to export controls. Such activities could include shipping an export-controlled item—such as certain biological samples or research equipment—overseas. Additionally, the release of export- controlled items or source code in connection with research activities to a foreign student or scholar could qualify as a deemed export requiring a license. U.S. universities may be exempt from or not subject to export controls if the information they are planning to release falls into one of three categories: published information or information in the public domain, certain academic information, or fundamental research. Published information or information in the public domain: Under the ITAR, information that is published and generally available in the public domain through specific methods is not considered to be technical data, and is therefore not subject to ITAR export licensing requirements. Under the EAR, unclassified technology or software that has been made available to the public without restrictions upon its further dissemination is considered to be published and is therefore not subject to the EAR. Certain academic information: Under the ITAR, information regarding general scientific, mathematical, or engineering principles commonly taught in schools is not included in the definition of technical data and is not subject to ITAR export controls. Similarly, information that is taught in catalog-listed courses or associated teaching laboratories of academic institutions is not subject to the EAR. Fundamental research: Fundamental research is not subject to the ITAR or the EAR. The ITAR defines fundamental research as basic and applied research in science and engineering where the resulting information is ordinarily published and shared broadly within the scientific community, as distinguished from research the results of which are restricted for proprietary reasons or specific U.S. government access and dissemination controls. The EAR defines fundamental research as research in science, engineering, or mathematics, the results of which ordinarily are published and shared broadly within the research community, and for which the researchers have not accepted restrictions for proprietary or national security reasons. Under the EAR, software and technology that arise during or result from fundamental research that is intended to be published is also not subject to the EAR. For example, a foreign person may be able to read research reports or view presentations that result from fundamental research and are intended to be published without the university obtaining a license. However, if that research involves software or technology that is subject to the ITAR or the EAR and is not intended to be published or produces an item that is subject to the ITAR or the EAR, the foreign person generally could not participate in the research without the university securing an export license. According to the FBI and DOD, as foreign adversaries use increasingly sophisticated and creative methodologies to exploit America’s free and open education environment, the United States faces an ever-greater challenge to strike a sustainable balance between unrestricted sharing and sufficient security within the U.S. university research environment. According to a 2019 FBI white paper, the inclusion of foreign students and scholars at U.S. universities entails both a substantial benefit and a notable risk. Specifically, the FBI reported that while many of these foreign students and scholars contribute to advanced research, the development of cutting-edge technology in an open research environment puts academia at risk for exploitation by foreign actors who do not follow U.S. laws and regulations. Additionally, a DOD report from September 2019 stated that research targeted by foreign talent programs includes topics relevant to U.S. national defense. According to the FBI, while the majority of foreign students and scholars do not pose a threat to their host institution, fellow classmates, or research fields, some foreign actors seek to illicitly or illegitimately acquire U.S. academic research and information to advance their home countries’ scientific, economic, and military development goals. By doing so, they can save their home countries significant time, money, and resources while achieving generational advances in technology. The U.S. government, including GAO, has long identified vulnerabilities in U.S. agencies’ efforts to protect U.S. research from foreign entities who might seek to exploit the openness of the U.S. academic environment. In prior GAO reports, we identified weaknesses in the deemed export control system that could allow the unauthorized transfer or release of export-controlled items to foreign persons in the United States. Moreover, since 2007, we have identified the protection of technologies critical to U.S. national security interests—including through U.S. export controls—as a high-risk area. More recently, the Senate Homeland Security and Governmental Affairs Committee reported that federal agencies need to do more to mitigate the threat to American universities by foreign persons seeking to undermine the integrity of the American research enterprise and endanger our national security. More than 1.2 million foreign students and 21,000 foreign scholars studied or worked at U.S. universities in 2018. Nearly a third of foreign students studying in the United States are from China, and a large proportion of Chinese students major in science, technology, engineering and mathematics (STEM) fields (see table 1). In addition, 10 countries accounted for about 70 percent of the more than 21,000 foreign scholars who worked at U.S. universities in 2018 (see table 2). The federal government obligated approximately $33 billion for U.S. universities for research and development in fiscal year 2017. The National Institutes of Health obligated approximately 54 percent of federal research and development funding provided to U.S. universities that year. The Department of Energy, DOD, and the National Aeronautics and Space Administration also obligated significant funding for universities for research (see fig. 1). State’s DDTC and Commerce’s BIS have developed export compliance- related guidance and conducted outreach to support all exporters’ understanding of and compliance with the regulations. However, university and association officials raised concerns that DDTC and BIS guidance and outreach does not adequately address university-specific export compliance issues. In addition, DDTC’s export compliance guidelines do not explicitly promote risk assessments, identified by GAO as a key element for determining whether an entity’s processes address current threats. State’s DDTC and Commerce’s BIS have developed various forms of written guidance and conducted outreach to support all exporters’ understanding of export control regulations. The ITAR and the EAR regulations apply to all exporters, whether universities, private entities, non-profits, or government entities, and according to DDTC and BIS officials, the guidance and outreach materials they have developed are similarly applicable to all potential exporting entities, including universities. Both DDTC and BIS provide written guidance intended to (1) increase awareness of applicable export control regulations, (2) provide specific instructions or tools for complying with those regulations, and (3) dispense transaction or entity-specific information or guidance for all exporters. For example, DDTC’s and BIS’s websites include general information about their respective export control regulations, including guidance on when an export license is needed and how such a license can be procured. DDTC highlights useful resources available on its website in a letter it sends to entities, including universities, when those entities register with DDTC as potential exporters of ITAR-controlled items. BIS’s website includes information about deemed exports, which one BIS official said is particularly relevant to universities. Both websites also include sets of frequently asked questions. DDTC and BIS have also developed guidance that provides specific instructions or tools for complying with the agencies’ regulations, including export compliance guidelines (see below for more information about these guidelines) and decision tools for classifying items subject to the ITAR and the EAR. For example, DDTC offers exporters an online tool to help them identify steps to follow in reviewing the USML and in classifying items subject to the ITAR. Similarly, BIS provides exporters with (1) online tools to help them classify items subject to the EAR and (2) guidelines for completing the license application for both deemed exports and tangible exports, such as chemical and biological items. Finally, both DDTC and BIS offer several mechanisms for obtaining transaction- or entity-specific information or guidance. For example, DDTC and BIS provide advisory opinions when an exporter requests a formal answer to an export control-related question, and both agencies operate a hotline to provide informal guidance to potential exporters. In addition, BIS reviews and provides feedback on export compliance manuals adopted by exporting entities, including universities, when requested. Exporters may also request a commodity jurisdiction classification from DDTC and BIS to determine whether a commodity is subject to the ITAR or the EAR. Both agencies also provide training, present at conferences, and conduct site visits to further educate exporters. For example, DDTC provides in- house seminars on export licensing basics approximately twice a year. BIS has developed and conducts various types of training related to export control compliance, including training videos that are publicly available on its website. BIS also hosts regional seminars and an annual conference in Washington, D.C., on export controls and export compliance. Both DDTC and BIS participate in various conferences. For example, DDTC and BIS participate in an annual conference affiliated with the Association of University Export Control Officers, where agency officials discuss topics such as regulatory updates, license statistics, and export compliance best practices. In fiscal year 2019, DDTC participated in 52 outreach events, two of which were university-specific. During that year, BIS conducted or participated in over 80 outreach events, six of which were university-specific. DDTC and BIS also conduct site visits to learn more about a given entity’s export compliance program and provide feedback, among other things. According to officials, DDTC conducted three university site visits from 2015 through 2019. Similarly, according to officials, BIS conducted two university site visits from 2013 through 2019. Further, officials at both agencies stated that they share information at outreach events about export compliance program strengths and weaknesses they identified during site visits. Officials from universities in our sample and university association officials told us that most DDTC and BIS export control-related guidance and outreach does not address those issues most relevant to the university export compliance environment and that additional guidance and outreach efforts would be useful. For example, according to association officials and officials at six of the nine universities we visited, it is sometimes difficult to understand how to implement in the university environment what they perceive to be industry-focused guidance developed by DDTC and BIS. Some of these officials further noted that the export compliance environment for industry typically differs from that for academia. Specifically, university and association officials noted that companies are typically focused on developing proprietary technologies, whereas universities are primarily focused on expanding knowledge through fundamental and collaborative research. In addition, officials from two universities stated that researchers typically do not see themselves as exporters, which makes it difficult to explain to them how export control regulations pertain to university research. For example, one official told us that it is difficult to explain the concept of a deemed export within an open, academic setting to university researchers. Officials at two universities also noted that the term “defense service,” a type of export subject to the ITAR, is a difficult concept to explain to university researchers who do not consider their work to be a “service.” Officials at four universities told us that they rely on university associations to develop a common understanding or interpretation of the regulations for the university context. For example, officials from one university said that university associations are a resource for sharing information and best practices regarding export compliance in the university environment. An official from another university stated that although she reviews the DDTC and BIS websites periodically for regulatory updates, she relies on university associations to explain how any updates affect universities. Some university officials stated that some agency outreach efforts are useful, but others said that more outreach is needed. Specifically, five university officials mentioned specific agency training and outreach efforts as being useful. For example, the officials said they appreciate that BIS conducts regional seminars for all exporters, which they said are easier to get to than events in Washington, D.C. One of these officials further noted that these seminars discuss how to set up an effective compliance program. However, four university officials stated that additional outreach efforts by both DDTC and BIS were needed. For example, two of these officials suggested that agencies consider additional training for universities, such as webinars or videos providing examples of simple export scenarios for university audiences, to clarify the intent of the export control regulations and explain how regulatory requirements pertain to university research. In discussing additional guidance needs, university and association officials told us that a set of all-encompassing, university-specific guidance is not necessary, but that additional guidance addressing specific topics that are relevant to universities would be useful. For example, one university association told us that additional DDTC and BIS guidance could take the form of frequently asked questions regarding issues of interest to universities, such as deemed exports and fundamental research. Similarly, one university export control officer stated that additional sets of frequently asked questions focused on issues most relevant to university export compliance, examples of university export compliance best practices, and examples of export control violations committed by universities would be particularly helpful. This export control officer explained that such guidance would help her and her colleagues (1) explain why the export control regulations are relevant for university researchers and (2) better explain the need for additional compliance resources to university management. University and association officials further stated that it would be helpful if DDTC and BIS would work with university associations to develop guidance that would support universities’ efforts to interpret the regulations consistently. These officials said that a stronger partnership between the regulatory agencies and universities would support agencies’ understanding of the university environment and result in better guidance for universities. They noted, for example, that soliciting university input on existing guidance and suggestions for additional guidance could provide DDTC and BIS with helpful information about the challenges that universities face in complying with export control regulations in their distinct environment. DDTC officials acknowledged that additional guidance addressing university-specific issues could be helpful and agreed that it may be difficult for university export control officers to explain export control regulations to researchers. They told us that it could be useful for the department to draft white papers, sets of frequently asked questions, or tip sheets specifically addressing issues most relevant to universities. For example, officials suggested that DDTC could develop tips on what may constitute a defense service in the university context. DDTC officials explained that they had not drafted such guidance because of resource constraints and other priorities. When we asked BIS officials about the potential need for university- specific guidance, one official identified some currently available guidance that could be most useful to universities. For example, BIS maintains a set of frequently asked questions and a YouTube webinar concerning deemed exports, and has guidance related to fundamental research available on its website. According to BIS, it regularly updates guidance related to deemed exports and fundamental research, including in connection with regulatory changes that affected both areas in 2016. GAO’s Standards for Internal Control in the Federal Government state that management should communicate with, and obtain information from, external parties using established reporting lines. Although BIS has provided written guidance that is relevant to universities and both DDTC and BIS conduct university-specific outreach, officials at universities we visited and associations we interviewed raised concerns about the adequacy of this guidance and outreach for the university research environment. Without additional guidance and outreach from DDTC and BIS that addresses issues most relevant to universities, some universities may utilize guidance, training, or other resources developed by other entities that may not facilitate compliance with export control regulations in the way that DDTC and BIS intended. Hence, universities may be at risk of failing to comply with export control regulations and properly safeguard export-controlled items from foreign students and scholars who are not authorized under deemed export licenses to receive such items. In addition, such university-focused guidance is consistent with the Export Control Reform Act of 2018, which requires the President to enforce export controls by providing guidance in a form that facilitates compliance by academic institutions and other entities. Although State’s DDTC and Commerce’s BIS officials identified their respective export compliance guidelines, available on the agencies’ websites, as key sources of written guidance for supporting exporters’ compliance with each agency’s export control regulations, DDTC’s compliance guidelines do not explicitly promote risk assessments. Both sets of export compliance guidelines include similar elements that the agencies consider critical for an effective export compliance program. For example, both sets of guidelines include elements related to management commitment, recordkeeping, and training. However, DDTC’s guidelines do not advise entities on how to assess risk, which GAO has identified as a key element for determining whether an entity’s processes address current threats. BIS Guidelines. BIS’s export compliance guidelines identify eight elements of an effective export compliance program. BIS officials stated that the agency’s guidelines provide a useful compliance framework for all exporters, including universities. These guidelines include information about recordkeeping, conducting internal audits, performing risk assessments, and training, among other elements. BIS’s guidelines also provide templates, checklists, specific examples, and other tools exporters may use to develop an export compliance program or enhance an existing program. For example, the guidelines include a summary of potential risks involved in each phase of the exporting process with a list of tools to mitigate such risks. The guidelines also include an audit module tool to help exporters review and revise their current compliance program with a set of checklists for each of the eight elements. DDTC Guidelines. DDTC’s export compliance guidelines include nine elements that it has identified as important aspects of an effective export compliance program. According to DDTC, its guidelines are also applicable to all exporters, including universities, and the agency references them in a confirmation letter when entities register as exporters. The guidelines include information about organizational structure, corporate commitment and policy, internal monitoring, and training, among other elements. The guidelines also provide examples of questions a compliance program should address for some elements. However, DDTC’s export compliance guidelines lack a risk assessment element. Risk assessments provide entities with an opportunity to review their processes to determine whether the processes in place address current threats. According to DDTC, the agency has not added guidance related to risk assessments to the export compliance guidelines because it assumes that exporters conduct a risk assessment for each compliance element as a matter of course. GAO’s Standards for Internal Control in the Federal Government state that management should communicate quality information externally so that external parties can help the entity achieve its objectives and address related risks. Further, according to an Office of Management and Budget bulletin, agencies increasingly have relied on guidance documents to inform the public and to provide direction to their staffs as the scope and complexity of regulatory programs have grown. Exporters, including universities, may not conduct periodic risk assessments if DDTC’s guidance does not encourage them to do so. As such, they may be unaware of potential threats and may not take appropriate measures to protect export- controlled items. University and association officials we interviewed identified challenges working with and obtaining guidance from federal agencies that fund research and monitor threats to the United States, including threats to research security. Specifically, university and association officials identified the following three challenges working with and obtaining guidance from these agencies: (1) federal agencies are developing different requirements for reporting financial conflicts of interest to address foreign influence issues, (2) some agencies provide briefings and other forms of guidance related to export controls and foreign threats that do not sufficiently address universities’ needs, and (3) DOD officials inconsistently interpret export control regulations and misunderstand what constitutes fundamental research. Agencies are taking steps to address some of these challenges. For example, an interagency working group established by the White House Office of Science and Technology Policy and individual federal agencies are undertaking efforts to address university concerns regarding inconsistent financial conflict of interest reporting requirements and the lack of relevant, university-specific resources to address threats identified by some agencies. However, the actions that DOD plans to take to address agency officials’ inconsistent interpretation of the regulations and their misunderstanding of the term fundamental research may not fully address the challenge identified by university and association officials. University and association officials expressed concerns that federal agencies are developing different requirements for reporting financial or other conflicts of interest, such as foreign funding, but some of these differences in reporting requirements may be necessary to address varying agency-specific legal requirements. For example, recent reporting guidance from the National Institutes of Health reminds researchers to report all sources of support, including support for laboratory personnel and the provision of materials that are not freely available, whereas the most recent guidance from DOD does not include such clarification for what constitutes “support.” Although each agency has a separate mission and separate legal authorities, which may require agencies to have different financial or other conflict of interest reporting requirements, officials at several universities and associations discussed the challenges they face in complying with these varied reporting requirements. Representatives from one university association explained that these new requirements are especially challenging for universities because they typically accept funding from multiple agencies. In addition, officials from one university stated that the variation across the agencies’ reporting requirements makes it difficult to develop one process to support researchers’ efforts to comply with them. According to university and association officials, universities will need to spend more time and resources to understand and comply with each set of requirements. Moreover, one association official told us there is more room for universities to make mistakes when each agency develops different requirements to deal with the same issue. An interagency working group established by the White House Office of Science and Technology Policy is undertaking efforts to address university concerns regarding inconsistent financial conflict of interest reporting requirements. In May 2019, the Office of Science and Technology Policy established the Joint Committee on the Research Environment (JCORE), an interagency effort to address research security and other related issues. According to officials in the Office of Science and Technology Policy, JCORE has drafted one set of coordinated guidance for funding agencies to ensure that funding agencies consistently require researchers to report the same types of information regarding potential conflicts of interest. In addition, JCORE has drafted a set of non-binding guidelines for universities to support their efforts to comply with conflict of interest reporting requirements. Officials stated that the draft guidance for funding agencies and the non-binding guidelines for universities were under review as of January 2020. Officials further stated that JCORE is developing a set of case studies and other materials that federal agencies will be able to use to educate researchers and universities about the types of situations that represent a potential conflict of interest. Agencies such as the FBI, DHS, BIS’s Office of Export Enforcement, and DOD’s Defense Counterintelligence and Security Agency provide briefings and other forms of guidance related to export controls and foreign threats. For example, officials at these agencies provide briefings to individual universities or to groups of universities during university association events, such as the annual Association of University Export Control Officers conference and the annual Academic Security Conference hosted by the Texas A&M University System. In addition, DHS identified the 11 universities with the largest number of foreign students studying in STEM fields in 2018 to target university outreach efforts in late 2018 and early 2019. DHS developed a template presentation for DHS field offices to use during their outreach to these universities to increase awareness of export control laws. According to DHS, it plans to expand this effort to target the top 60 universities with foreign students in STEM fields. The Department of Justice and BIS’s Office of Export Enforcement have both published reports summarizing recent major U.S. export enforcement-related criminal and administrative prosecutions. Some university officials told us that the briefings and other information that some agencies provide are helpful for improving their awareness of threats. However, officials at five of the nine universities we visited and officials from three university associations said that these briefings and other information are not as useful as they could be. Some of these officials cited the following reasons for why they did not find such information to be useful: Classified information cannot be shared widely: Some university officials and an association representative stated that some agencies often provide classified briefings and materials that they cannot share widely with the university community. One university official said that it would be helpful if agencies, where possible, could also provide some unclassified information with clear examples that could then be shared with researchers about current threats and what these threats may look like in a university setting. Without such information, university officials are restricted in how they can use the threat-related information they obtain for raising awareness on campus, according to a university association official. Moreover, another university official stated that if export control officers cannot share relevant threat information with the university’s administration because of classification issues, the university may not get the resources it needs to improve its compliance programs and properly comply with export control regulations. Guidance and threat information does not address the university environment or utilizes outdated examples: Representatives from three university associations and one university stated that some federal agencies do not provide guidance and threat information that address the university research environment, and two associations said that any university-specific examples federal agencies provide during briefings are outdated, which limits the relevancy of guidance and threat information to the university environment. For example, an official from one association explained that in 2015 the FBI provided a threat briefing at an association meeting and requested that university officials contact the FBI if a researcher had, among other things, published in an international scientific journal or attended an international conference, or if any graduate students worked in university laboratories late at night. This official noted that these FBI officials did not understand that researchers must undertake such activities to obtain tenure and that it is common for students to work late at night. In addition, according to an official from one association, when university officials ask the FBI to provide recent examples of foreign students stealing sensitive or export-controlled items from U.S. universities, the FBI often cites cases that occurred more than 10 years ago. He further stated that federal agencies are raising alarms that universities are vulnerable to foreign theft of export-controlled items without any concrete, recent examples. FBI threat briefings lack actionable guidance: University officials told us that many FBI threat briefings are not helpful because they do not provide actionable guidance for addressing identified threats, which limits universities’ understanding of how to address them. For example, one university official stated that the FBI briefings do not provide any detailed information about what attendees should do with the information they obtain. He further stated that the briefings would be more beneficial if the FBI provided prescriptive guidance on how to use the information. DOD and the FBI are taking steps to partner with academia to address challenges regarding information sharing. DOD is undertaking several collaborative efforts with academia in response to Section 1286(d) of the 2019 National Defense Authorization Act, which directed the Secretary of Defense to establish an initiative to support protection of national security academic researchers from undue influence and other security threats. For example, DOD partnered with the National Academy of Engineering to establish the “Roundtable on Linking Defense Basic Research to Leading Academia Research and Engineering Communities,” or the “Deans’ Roundtable.” The Deans’ Roundtable brings DOD leadership together with deans from U.S. university engineering programs to facilitate dialogue between DOD and the academic research community on research protection. The roundtable’s objectives are to better understand major issues in the defense research community and to form working groups to help craft potential solutions to challenges identified by the roundtable. The roundtable is expected to help address issues of research espionage by foreign governments on university campuses and inform senior DOD officials about technological developments on university campuses, among other efforts. The FBI partnered with the Academic Security and Counter Exploitation Program, a university-led association focused on research security, to produce a series of unclassified “awareness-raising” materials for university audiences. According to FBI officials and a member of the Academic Security and Counter Exploitation Executive Committee, the FBI recognized that university officials were frustrated that relevant FBI documents regarding the foreign threat to U.S. research were classified. The association’s Executive Committee member further explained that this created significant restrictions on the way university officials could use the materials for awareness and training efforts on campus. He further noted that many of these “awareness-raising” materials were tone- deaf to the needs of academia and did not explain how the threats were related to university researchers’ work. The Academic Security and Counter Exploitation Executive Committee worked with the FBI to revise existing FBI handouts to create a series of academic-focused, unclassified documents suitable for inclusion in awareness and training programs on university campuses. For example, they revised a FBI handout regarding the threat that China poses to corporate America to instead focus on the threat that China poses to academia. Officials from multiple universities and associations stated that DOD officials inconsistently interpret export control regulations and misunderstand the term fundamental research and its implications when providing funding for university research, which some officials said leads to confusion, results in contract delays, and may limit universities’ ability to conduct research for DOD. DOD officials acknowledged that some officials have inconsistently interpreted the regulations. Moreover, DOD reported to Congress in September 2019 that it is mindful of the fact that reducing the quantity and competitiveness of early ideas flowing through the university system to the department by non-judicious use of controls could have negative consequences. Officials at four of the nine universities we visited identified DOD officials’ inconsistent interpretation of the regulations and their misunderstanding of what constitutes fundamental research as a challenge they face in complying with export control regulations. For example, officials at three universities asserted that DOD includes contract clauses, such as export control-related clauses, that are not relevant to or conflict with other stated terms in the contract, in some cases. Officials at two universities further stated that there appears to be an internal disagreement between the program officers and contracting officers about how to interpret some aspects of export control regulations. One university official said the university tries to negotiate with DOD when contracts that the university perceives as only containing fundamental research include export control- related clauses; however, the official said these types of delays slow the pace of research. Moreover, university association officials noted that member universities are reporting that DOD is increasingly including publication restrictions in research contracts for projects that the universities believe only entail fundamental research. Research does not qualify as fundamental research if the researcher accepts any restrictions on the publication of the information resulting from the research. Officials from one association stated that DOD is reluctant to remove publication restrictions from award contracts even when it acknowledges that the work may only involve fundamental research. As a result, universities that only accept contracts for fundamental research may decline an awarded contract if the conditions for the award vary from initial expectations, which may lead to a loss in research funding for many universities focused on fundamental research. In 2008 and 2010, DOD issued memoranda to its personnel providing clarifying guidance concerning fundamental research and directed that information about contracted fundamental research be included in general training modules for research program personnel. For example, these memoranda state that DOD must not place restrictions on subcontracted unclassified research that has been scoped, negotiated, and determined to be fundamental research within the definition of National Security Decision Directive 189 according to the prime contractor and research performer and certified by the contracting component, except as provided in applicable federal statutes, regulations, or executive orders. These memoranda also state that the effective implementation of the guidance requires that all DOD personnel involved in the acquisition and monitoring of fundamental research have a clear and common understanding of the relevant statutes, regulations, and policies, including the definitions of key terms. To implement these memoranda, DOD also amended the defense federal acquisition regulations in 2013 to update the relevant contract clause for inclusion in DOD contracts. The Deputy Director for Basic Research at DOD stated that most program officers and contracting officers are familiar with the export control regulations and understand the term fundamental research and how to interpret it in the context of university research, but acknowledged that some officials have inconsistently interpreted the regulations and misinterpreted the term fundamental research. Specifically, DOD officials stated that program officers and contracting officers who frequently work with universities through basic research grants understand what constitutes fundamental research; however, program officers and contracting officers working with applied research contracts may not be as familiar with it or with engaging with universities. Furthermore, DOD officials acknowledged that although DOD has developed export control-related training, it does not require program officers and contracting officers to take this training. Officials stated that not all program officers and contracting officers work with universities, so they do not all need to take training on export control regulations. To address these and other research-related concerns, DOD’s Office of Basic Research convened a workshop for basic research program officers in October 2019 to facilitate the sharing of best practices and identify any concerns. According to DOD, program officers raised a concern that they need to constantly ensure that the research being conducted is properly categorized as basic or fundamental research and has not transitioned into applied or non-fundamental research in the course of the contract. DOD’s Office of Basic Research is planning to develop a checklist based on input from program officers that program officers can use when determining whether the scope of a research project meets the definition of fundamental research. Following this workshop, a DOD official stated that program officers are best suited to make technical and nuanced fundamental research determinations because program officers have first-hand knowledge about the scope of the research project. These actions, however, may not address the concerns universities raised, because they do not include any effort to further educate contracting officers. Contracting officers may add export control clauses or publication restrictions to a contract award after the program officer writes the original solicitation. Additionally, contracting officers are the individuals with regulatory authority for defense contracts to certify that research is fundamental research. Hence, a checklist for program officers may not fully address program officers’ and contracting officers’ inconsistent interpretation of the regulations, including determining whether university research constitutes fundamental research. Without additional efforts to educate all relevant DOD officials on a clear and common understanding of the relevant statutes, regulations, and policies, as identified by the department’s 2010 memorandum, universities may continue to perceive that DOD officials inconsistently interpret the regulations and misunderstand whether research constitutes fundamental research, potentially hindering DOD-funded research at universities. The nine universities we visited have generally developed export compliance policies and practices to safeguard export-controlled items that align with State’s DDTC and Commerce’s BIS export compliance guidelines, but some of the universities’ compliance efforts have weaknesses in certain areas (see fig. 2). We reviewed DDTC’s and BIS’s export compliance guidelines to identify common elements and developed a list of eight elements that the agencies classified as critical for an effective compliance program, such as recordkeeping and training, among others. See table 3 for a description of the eight elements we identified for this assessment. We then interviewed officials at nine universities about their universities’ export compliance policies and practices. We selected universities with annual average expenditures for research and development during the 2013 through 2017 period that ranged from $15 million to over $750 million. In addition, we selected universities on the basis of a number of factors, including total research and development expenditures, number of graduate students, research funding received from certain federal agencies, and geographic dispersion (see app. I for more information about our selection methodology). Finally, we assessed the university officials’ responses against the eight elements of an effective export compliance program to determine the extent to which these universities’ policies and practices align with DDTC’s and BIS’s export compliance guidelines. See appendix III for a detailed description of our assessment of each university’s policies and practices against these elements and a description of the export compliance policies and practices the selected universities have in place. In addition, we reviewed the websites of a generalizable sample of 100 U.S. universities to determine the extent to which these universities provide publicly available information about export control regulations, training, and other topics pertinent to the campus community. In general, the universities with larger research and development expenditures provided more export control-related information on their websites. See appendix IV for the results of this analysis. The seven universities with the highest research expenditures among the nine we visited have export compliance policies and practices that generally align with the eight elements we identified from DDTC’s and BIS’s export compliance guidelines, while the two universities with the lowest expenditures among the nine have more weaknesses in their compliance programs. Most of the universities we visited have robust export compliance practices in the following four areas: Management commitment and organizational structure: All nine of the universities we visited have developed policies and practices that fully or partially align with this element. For example, management at seven of the nine universities we visited issued public statements supporting the university’s export compliance program. These statements briefly described export control regulations, discussed the importance of the universities’ compliance with export control regulations, and emphasized the universities’ commitment to compliance efforts. Export authorization and tracking export-controlled items: All but one of the nine universities we visited have developed policies and practices that fully align with this element. For example, officials at all nine of the universities we visited stated that their universities require researchers to submit research proposals to an office charged with reviewing proposals and awards for grants and contracts. When reviewing research proposals or awards, this office will flag those proposals and awards that may be subject to export control regulations for further review, either by the export control officer or another authorized university entity. In addition, officials at seven of the universities said they had developed mechanisms to track any export-controlled items being used or developed by the university. The universities we visited also employ various security mechanisms to safeguard export-controlled items. These include physical security mechanisms, as shown in figure 3, as well as information technology security mechanisms, such as setting up separate networks for researchers using export-controlled data in their research. Recordkeeping: Officials at all nine universities we visited have developed policies and practices that fully align with this element to ensure that they maintain appropriate export control-related records. For example, at least five of the nine universities we visited maintain their export compliance-related records in an electronic database or other electronic system. One of the universities utilizes a system that tracks each research project from start to finish. This system enables officials to search for all export control-flagged research proposals, awards, and technology control plans, among other documents. One of the officials also told us that the system will alert the export control officer to any technology control plans with an upcoming expiration date. Two of the remaining four universities maintained some files electronically and some in hard copy. The other two universities did not discuss how they maintained their files, but identified who is responsible for export control-related recordkeeping and the types of documents they maintain. Reporting violations: All nine universities we visited have developed policies and practices that fully align with this element. Specifically, these universities have developed clear procedures outlining the actions employees should take in the event that potential noncompliance is identified. For example, officials at seven universities told us that they have a compliance hotline that people can use to report suspected violations. Some of the universities we visited have weaknesses in their export compliance programs, particularly in the following four areas: Risk assessment: Four of the nine universities we visited do not currently conduct risk assessments to assess and identify potential risks in their export compliance programs, which may limit their ability to identify potential risks or build safeguards in their export compliance program to address potential risks. Three of these four universities are in the lowest tier for annual research and development expenditures. Training: Two of the nine universities we visited do not provide any formal training for researchers and other officials involved in implementing export control regulations. However, an official from one of the universities said that the university provides access to online export control-related trainings developed by a for-profit entity. The export control officer at the other university said that although the university does not conduct formal training, he conducts frequent outreach and provides materials to increase university officials’ awareness of export control regulations. Internal audits: Four of the nine universities we visited either partially conducted, or did not conduct, internal audits of their export compliance programs. The three universities that partially conducted internal audits have an export control officer who periodically reviews some internal processes but did not have a university audit group outside of the export control office that had reviewed the export compliance program. However, officials from two of these universities stated that their audit office plans to conduct an audit of the export compliance program soon. Export compliance manual: Four of the nine universities we visited have not developed an export compliance manual. According to DDTC and BIS guidelines, exporters are encouraged to develop a manual to document export control-related roles and responsibilities of various offices and officials. The manuals should also describe export control procedures, development of technology control plans for export-controlled work, training requirements, and processes for reporting potential violations, among other topics. Research conducted by U.S. universities and supported by visiting foreign students and scholars makes critical contributions to U.S. national security and economic interests. However, the relative openness of the university environment also presents a vulnerability that can be exploited by foreign adversaries. State’s DDTC and Commerce’s BIS administer systems of export controls to minimize these vulnerabilities while allowing legitimate business to occur, and the agencies provide guidance and conduct outreach to facilitate universities’ compliance with these controls. While DDTC and BIS provide some guidance and conduct outreach to universities, university officials told us that this guidance does not adequately address university-specific issues. The universities we visited primarily rely instead on guidance and training provided by other entities, which may not always facilitate compliance with the export control regulations as DDTC and BIS intended. We found that the nine universities we visited have generally developed export compliance policies and practices that align with agency guidance, but some of the universities’ compliance efforts have gaps. Improved guidance and outreach based on feedback from university stakeholders could further strengthen universities’ efforts to identify and protect export-controlled items from unauthorized transfers or releases to foreign students and scholars. This is especially important in light of continued reports of foreign entities’ exploitation of university research. Moreover, DDTC’s export compliance guidelines do not include information concerning risk assessments, a key element for determining whether an entity’s processes address current threats. Four of the nine universities we visited did not conduct risk assessments. Including information about risk assessments in DDTC’s written guidelines regarding the elements of an effective export compliance program would enable DDTC to remind universities and other exporters that conducting risk assessments is a beneficial practice. If exporters, including universities, do not conduct periodic risk assessments, they may be unaware of new threats and, consequently, may not take appropriate measures to protect export-controlled items. Furthermore, universities reported challenges working with DOD because of DOD officials’ inconsistent interpretation of export control regulations, including how to assess whether a university is engaging in fundamental research. DOD officials acknowledged this challenge, but DOD has not taken sufficient action to educate its personnel on the regulations. Without additional action, DOD may continue contributing to confusion and contract delays that hinder legitimate research. We are making four recommendations, including two to State, one to Commerce, and one to DOD. Specifically: The Secretary of State should ensure that the Deputy Assistant Secretary for Defense Trade Controls, in consultation with university representatives, provides additional or revises existing guidance and outreach to address university-specific export control issues to further support universities’ understanding and compliance with the International Traffic in Arms Regulations. (Recommendation 1) The Secretary of Commerce should ensure that the Under Secretary for Industry and Security, in consultation with university representatives, provides additional or revises existing guidance and outreach to address university-specific export control issues to further support universities’ understanding and compliance with the Export Administration Regulations. (Recommendation 2) The Secretary of State should ensure that the Deputy Assistant Secretary for Defense Trade Controls revises existing export compliance guidelines to include information concerning periodic risk assessments to remind exporters that it is beneficial to periodically identify, analyze, and respond to new risks as part of an effective International Traffic in Arms Regulations compliance program. (Recommendation 3) The Secretary of Defense should ensure that the Under Secretary of Defense for Research and Engineering takes steps to ensure that its program officers and contracting officers are interpreting export controls consistent with regulations and guidance and consistently determining whether university research constitutes fundamental research. (Recommendation 4) We provided a draft of this report to Commerce, DHS, DOD, FBI, State, and the White House Office of Science and Technology Policy for comment. In their comments, reproduced in appendixes V and VI, State and DOD concurred with the recommendations directed to them. State also provided information about the actions it plans to take to address recommendations 1 and 3. With respect to recommendation 1, State noted that it is already expanding its outreach to university representatives and planning to issue additional guidance to further support universities’ understanding of the ITAR. With respect to recommendation 3, State noted that it plans to revise existing export compliance guidelines to include information concerning periodic risk assessments. DOD also provided information about actions it plans to take to address recommendation 4. Specifically, DOD stated that it will develop new guidance for DOD personnel to clarify the process for identifying fundamental research, funding contracts containing fundamental research, and monitoring those contracts to ensure that they are performed in compliance with export control regulations and fundamental research policies. DOD also stated that it plans to work with State and Commerce to ensure that the new guidance is consistent with the ITAR and the EAR, respectively. Commerce concurred with recommendation 2, but it did not provide a comment letter in time for publication in the report. DHS, FBI, and the White House Office of Science and Technology Policy informed us that they had no comments. Commerce, DOD, and State provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees; the Secretaries of Commerce, Defense, and State; the Acting Secretary of Homeland Security; the Attorney General of the United States; the White House Office of Science and Technology Policy; and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8612 or gianopoulosk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. Our report examines (1) the extent to which the Departments of State (State) and Commerce (Commerce) have provided guidance and outreach that supports U.S. universities’ understanding of and compliance with both agencies’ export control regulations, (2) export control-related challenges that U.S. universities face while working with or obtaining guidance from other federal agencies, and (3) the extent to which export compliance policies and practices developed by U.S. universities align with State’s and Commerce’s export compliance guidelines. In addition to the methods discussed below, we reviewed government reports concerning (1) previously identified gaps in the U.S. export control system and (2) the threat that some foreign persons pose to U.S. universities to provide context for all three objectives, and reviewed relevant federal laws and regulations to address all three objectives. We also attended a conference in March 2019 hosted by Association of University Export Control Officers member universities to better understand how universities administer export control regulations and those aspects of the regulations most relevant to universities. We used the information we collected during the conference to inform our planning for our site visits. To provide context for all three objectives, we examined federal data concerning (1) the number of foreign students and scholars studying or working at U.S. universities, (2) federal agencies’ research and development funding provided to universities, and (3) U.S. universities’ export license applications. We examined data identifying the country of citizenship for foreign students and scholars studying or working at U.S. universities from 2013 through 2018. We received the foreign student data from the Department of Homeland Security (DHS), which pulled data from its Student and Exchange Visitor Information System. We used these data to identify the top 10 countries sending foreign students to U.S. universities in 2018. DHS also provided data identifying foreign scholars working at U.S. universities based on I-129 filings. The I-129 form is typically filed by a U.S. employer on behalf of a nonimmigrant worker to come to the United States to temporarily perform services or labor or to receive training. We used these data to identify the top 10 countries sending foreign scholars to U.S. universities in 2018. We utilized data collected by the National Science Foundation to determine the amount of research and development funding U.S. universities received from federal agencies in fiscal year 2017. The National Science Foundation collects funding information from federal agencies through its Survey of Federal Funds for Research and Development. We downloaded the data from the agency’s website and analyzed the data to determine how much funding selected federal agencies and the federal government as a whole provided to universities and university-administered Federally Funded Research and Development Centers. Finally, we analyzed State and Commerce data for export license applications received in calendar years 2014 through 2018 to identify trends in U.S. university export license applications and determine the percentage of export license applications from U.S. universities as a share of all export license applications. For both data sets, we reviewed each applicant to verify whether it was a U.S. university, because both agencies provided some data that included license applications submitted by entities that are not U.S. universities, such as associations or foreign universities. We then analyzed the data to determine trends in application results, identify the top 10 destination countries for approved U.S. university export license applications, and identify the top five categories of export-controlled items for export license applications submitted by U.S. universities. We determined that all of these data sources were sufficiently reliable for providing context for our report. To address our first objective, we interviewed relevant State and Commerce officials from the Directorate of Defense Trade Controls and Bureau of Industry and Security and reviewed the guidance and outreach materials these agencies developed related to export controls. We also analyzed information regarding their outreach efforts for fiscal year 2019 to identify the number of university-specific outreach events. In addition, we attended (1) the March 2019 Association of University Export Control Officers conference, at which both State and Commerce officials presented to university officials, and (2) Commerce’s annual conference on export controls in Washington, D.C., at which State officials also presented. To address our second objective, we interviewed officials from several agencies that provide research funding to universities, including the Departments of Defense (DOD) and Energy, the National Institutes of Health, and the National Aeronautics and Space Administration, to learn how they work with universities that receive research funding. Additionally, we met with a number of security agencies, including DOD’s Defense Counterintelligence and Security Agency, DHS, and the Federal Bureau of Investigation, and reviewed reports, handouts, and outreach materials regarding either export control regulations or the threat environment to learn how these agencies educate U.S. universities. Finally, we met with the White House Office of Science and Technology Policy to discuss an interagency effort to address research security and other related issues. To identify university perspectives for all three of our objectives, we interviewed (1) representatives from four university associations and (2) officials at nine U.S. universities. Specifically, for our first and second objectives, we interviewed representatives from the Association of University Export Control Officers, Association of American Universities, and Council on Governmental Relations. The Association of University Export Control Officers is a member organization composed of over 270 export control and other compliance officers at U.S. academic institutions to provide a forum for the exchange of information regarding higher education and export, import, and trade sanctions policies. The Association of American Universities represents 65 research universities and seeks to shape policy for higher education, science, and innovation. According to a representative, the association’s membership is composed of university presidents and chancellors. The Council on Governmental Relations provides information to over 185 member universities regarding research administration and compliance, financial oversight, and intellectual property. The association’s membership is mainly composed of Vice Presidents for Research and Directors of Sponsored Research, according to a representative. For our second objective, we also interviewed a representative from the Academic Security and Counter Exploitation Program, whose executive committee includes representatives from 11 universities and university systems. This university-led association is focused on providing a forum within academia for discussions concerning the protection of intellectual property, controlled information, key personnel, and critical technologies at U.S. universities conducting research relevant to national security. For all three of our objectives, we interviewed officials at nine U.S. universities. See below for our selection methodology. To inform all three of our objectives, we conducted site visits to nine U.S. universities to speak with various university officials. We selected a non- generalizable sample of nine U.S. universities on the basis of a number of factors, including total research and development expenditures, number of graduate students, research funding received from certain federal agencies, and geographic dispersion. To identify a sample of U.S. research universities, we first examined U.S. university research and development expenditures data collected by the National Science Foundation for the 2013 through 2017 period. The National Science Foundation collects this data from universities through its annual Higher Education Research and Development Survey and we downloaded the data from the agency’s website. We then calculated the average annual research and development expenditures for each university on this list for this period. We limited our scope of universities to those with an annual average total research and development expenditures of over $15 million. This resulted in a total sample size of 292 U.S. universities. To assess the reliability of the data, we reviewed related documentation on the National Science Foundation’s web page regarding the Higher Education Research and Development Survey and dataset. We determined these data to be sufficiently reliable for the purposes of our report. We then reviewed a number of other factors for each of these universities. First, we categorized each of the 292 universities in our sample as public or private. We then identified the number of full-time graduate students for each university on the basis of results from the National Science Foundation’s annual Survey of Graduate Students and Postdoctorates in Science and Engineering (2016), because federal officials told us that graduate students were more likely to conduct research involving items subject to export control regulations than undergraduate students. We also reviewed universities’ security clearance level and membership in a number of associations to identify those universities that may be more aware of research security-related issues. Finally, we downloaded data from the Federal Procurement Data System to identify the total amount of federal contracts for research and development each university in our sample had received from four main funding agencies—DOD, the Department of Energy, the National Institutes of Health, and the National Aeronautics and Space Administration. These four agencies represent four of the five major funding agencies for university research and development in fiscal year 2017. In addition, they represent the four agencies that we determined, in consultation with GAO stakeholders and State and Commerce officials, are most likely to provide funding for research involving items that may be subject to export control regulations. We grouped the universities in our sample into six geographic regions and initially selected 35 universities across these six regions that represented a cross-section of universities, on the basis of the factors discussed above. Ultimately, we selected nine universities for site visits from four of these regions on the basis of university officials’ availability and scheduling considerations. While we sought to include a range of university experiences regarding export control compliance in our non- generalizable sample, the university officials’ views stated in this report do not represent the entirety of the U.S. academic community. During our site visits, we conducted semi-structured interviews with about 80 university officials involved in export compliance on the main campus of nine universities, including officials in the following relevant positions: vice presidents for research, export compliance officers, facility security officers, and officials charged with reviewing grants and contracts, among others. During these interviews, we asked officials about the export control-related policies and practices their university had developed; their roles in implementing those practices; their perspectives concerning guidance and threat-related information from federal agencies; and any challenges they face in complying with export control regulations, among other topics. We also conducted seven focus groups with 44 faculty in Science, Technology, Engineering and Mathematics (STEM) fields. However, we were not able to meet with all of the same types of officials at each university we visited. To address our third objective, we assessed university officials’ responses concerning export compliance policies and practices against a set of eight elements of an effective export compliance program. We reviewed State’s and Commerce’s guidelines to identify a list of eight common elements that the agencies classified as critical for an effective compliance program. We then assessed the responses of university officials from the nine universities we visited against these eight elements. Within some of the elements, we identified sub-elements for assessing university policies and practices. For example, within the element for management commitment and organizational structure, we identified five sub-elements against which we reviewed university officials’ responses. For each element, we developed a scale for determining whether each university’s export compliance policies and practices fully aligned, partially aligned, or did not align with that element. For example, for the management commitment and organizational structure element, we defined the extent to which each university’s policies and practices aligned with this element as (1) “fully aligned” if policies and practices were in place for at least four out of five sub-elements, (2) “partially aligned” if they were in place for two or three out of five sub-elements, and (3) “not aligned” if they were in place for one or zero of five sub- elements. We conducted this performance audit from February 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Although U.S. universities generally promote an open learning environment that is focused on the free exchange of information through fundamental research, some U.S. universities conduct research involving export-controlled items and have applied for export licenses for deemed exports (releases within the United States to foreign persons) and exports of tangible items out of the United States. The Departments of State (State) and Commerce (Commerce) both control the export of items within their respective jurisdictions by requiring a license or other authorization prior to the export of an item. Within State, the Directorate for Defense Trade Controls (DDTC) is responsible for implementing export controls. Similarly, within Commerce, the Bureau of Industry and Security (BIS) is responsible for implementing export controls. State’s DDTC received 597 license applications from U.S. universities in calendar years 2014 through 2018. DDTC provides one of four decisions for each license application—approved, approved with provisos, denied, or returned without action. DDTC approved roughly 79 percent of license applications it received from U.S. universities during this period. Commerce’s BIS reviewed 680 license applications from U.S. universities during this same time period. BIS provides one of three decisions for each license application—approved, denied, or returned without action. BIS approved 74 percent of these license applications. DDTC and BIS denied a small number of license applications submitted by U.S. universities in calendar years 2014 through 2018. Specifically, DDTC denied five applications for exports to Mexico, Sri Lanka, and the United Kingdom, as well as one application involving various destination countries. BIS denied eight applications for exports to China, Iran, and Russia during this same period. See figure 4 for more information regarding the status of U.S. university export license applications submitted to DDTC and BIS in calendar years 2014 through 2018. In calendar years 2014 to 2018, approximately 70 percent of the license applications submitted by U.S. universities that DDTC approved were for exports (including tangible exports and deemed exports) to 10 destination countries or multiple countries. This total included applications that involved various destination countries, which on their own represented 26 percent of total approved applications during this period (see table 4). Similarly, 57 percent of the license applications submitted by U.S. universities that BIS approved in calendar years 2014 through 2018 were for exports (including tangible exports and deemed exports) to 10 countries (see table 5). The top five U.S. Munitions List (USML) categories for which U.S. universities applied for export licenses from DDTC accounted for 77 percent of all applications for calendar years 2014 through 2018. These include license applications for exports controlled under USML categories related to spacecraft, night vision, and missiles (see table 6). The top five categories for which U.S. universities applied for export licenses from BIS accounted for 85 percent of all license applications for calendar years 2014 through 2018. These include license applications for exports specified on the Commerce Control List (CCL) under categories related to chemicals, aerospace, and sensors and lasers, as well as the export of items designated as EAR99 (see table 7). The Departments of State (State) and Commerce (Commerce) have each developed a set of export compliance guidelines (guidelines), which agency officials identified as key sources of written guidance for supporting exporters’ compliance with the agency’s export control regulations. Both sets of guidelines include similar elements that the agencies have identified as being critical for an effective export compliance program. We reviewed both agencies’ guidelines and developed one set of eight elements of an effective export compliance program, which we then used to assess universities’ export control compliance practices. The eight sections below include descriptions of each element. We selected a non-generalizable sample of nine U.S. universities for site visits on the basis of a number of factors, including total research and development expenditures, number of graduate students, research funding received from certain federal agencies, and geographic dispersion. To learn more about our methodology for selecting universities for site visits, see appendix I. We visited these nine universities to learn about the export control policies and practices that they had developed. During our site visits, we conducted semi-structured interviews with about 80 university officials involved in export compliance, including officials in the following relevant positions: vice presidents for research, export compliance officers, facility security officers, and officials charged with reviewing grants and contracts, among others. We also conducted focus groups with 44 faculty in Science, Technology, Engineering and Mathematics (STEM) fields at seven of the nine universities we visited. During our university site visits, we asked officials about the export control-related policies and practices their universities had developed; their roles in implementing those practices; and the roles and responsibilities of others involved in implementing the university’s export compliance policies and practices, among other topics. We did not independently verify universities’ implementation of the export compliance policies and practices that university officials described during our site visits. We found that the nine universities we visited had generally developed export compliance policies and practices to safeguard export-controlled items that aligned with State and Commerce export compliance guidelines, but that some of the universities’ compliance efforts had weaknesses in certain areas (see fig. 5). In the following sections, we provide a (1) description of each element and (2) summary of the results of our assessment of each university’s policies and practices against each element. For this element, we assessed universities’ activities within five sub- elements: (1) public management support for the export compliance program, (2) management’s understanding of export control regulations, (3) whether the university had designated an export control officer, (4) sufficiency of resources and authority to conduct export compliance activities, and (5) whether the university had created a clear organizational structure identifying individuals responsible for compliance. See figure 6 for the results of our assessment. Management commitment and organizational structure Entities should have public management support for their compliance program, sufficient resources to conduct compliance activities, and a clear organizational structure identifying individuals responsible for compliance. All nine of the universities we visited have developed policies and practices that fully or partially align with this element concerning management commitment and organizational structure. Specifically, seven universities had practices that fully aligned and two had practices that partially aligned with this element. Below, we provide additional detail on universities’ activities within the following five sub-elements: Provides public management support for export compliance program. Seven of the nine universities we visited have issued public statements from university management supporting the export compliance program. These statements briefly describe export control regulations, discuss the importance of the universities’ compliance with export control regulations, and emphasize university management’s commitment to compliance efforts. In addition, university researchers who participated in our focus groups said that their universities had created an environment in which they felt comfortable reaching out to university staff with compliance-related questions. For example, participants in one of the focus groups told us that compliance officials are not trying to find violations, but are instead focused on building stronger compliance programs and stronger relationships with faculty. Understands export control regulations. Export control officers at all nine of the universities we visited said that university management understands and is knowledgeable about export control regulations and the implications of these regulations on the university’s research and development activities. For example, one export control officer stated that increasing awareness among the administrators, faculty, and staff has taken time, but that the administration now has a good knowledge of export control requirements following the outreach and training that the export control office provided over the last few years. Designates an export control officer position. Eight of the nine universities we visited have export control officers, and of those eight, five have had an export control officer position for over 10 years. The only university we visited that did not have an export control officer position had such a position prior to our visit. Among the universities we visited, this university had the lowest average research and development expenditures from 2013 through 2017—less than $30 million. Provides sufficient resources and authority to conduct export compliance activities. Officials at eight of the nine universities we visited stated that their university had sufficient resources and that relevant officials had adequate authority to conduct export compliance activities. Officials at one university said that they did not have adequate authority to conduct compliance activities, but that this condition might be changing because the export control officers now report directly to the Vice President of Research, which is giving them greater access to university management. Creates a clear organizational structure for export compliance. Officials at seven of the nine universities we visited identified individuals who are involved in export control compliance, including researchers and officials working in procurement, shipping, and contracting, among other things. Five of these seven universities also have export compliance manuals that specifically describe various officials’ export compliance roles and responsibilities. For this element, we assessed the extent to which the university conducted risk assessments of its export compliance program. See figure 7 for the results of our assessment. Five of the nine universities we visited have developed policies and practices that fully align with this element concerning risk assessments, while the other four have not developed such policies and practices. Below, we provide additional detail on universities’ risk assessment activities. Of the five universities that told us they conduct risk assessments, three stated that the export control officers periodically or annually conduct internal risk assessments of their export compliance efforts, while the other two described university groups that conduct periodic or annual, university-wide risk assessments that include an assessment of the export compliance program. For example, one university’s export control officer said that her office periodically reviews the university’s export compliance policies and practices to determine whether any gaps exist within the program. She also recently started reviewing her university’s export compliance policies and practices against those of other universities to determine whether other universities had developed any export compliance practices that would be appropriate for her university to emulate. She found, for example, that other universities had implemented a centralized loaner laptop program for researchers traveling abroad to minimize the risk of the theft of sensitive data from personal laptops, and said she hopes to implement such a program at her university. Officials at a university that periodically conducts university- wide risk assessments said they had conducted two such risk assessments since 2015 and were conducting a third assessment during our visit. During one assessment, reviewers recommended that the university increase export control training and staffing, which the export control office is working to address. Another university that conducts annual risk assessments has a research oversight committee that is made up of many subcommittees, including one for export controls. Each subcommittee conducts an annual risk assessment for its compliance area and reports any recommendations for optimizing compliance program effectiveness to the vice president for research. For this element, we assessed universities’ activities within seven sub- elements: whether the university (1) had processes in place to identify research involving export-controlled items, (2) had processes in place to monitor research to determine whether a license might be required at a later time, (3) tracked any export-controlled items being used or developed, (4) had developed any policies or practices for safeguarding export-controlled items, (5) used technology control plans to document and safeguard export-controlled items, (6) screened and monitored foreign visitors, and (7) screened all foreign parties associated with research projects prior to any export activities. See figure 8 for the results of our assessment. Export authorization and tracking export- controlled items Entities should develop processes to (1) ensure the organization makes correct export decisions, including identifying when U.S. government authorization is required prior to exporting; (2) track and protect any export- controlled items being used or developed by the organization; and (3) screen all parties associated with an export transaction against the U.S. proscribed/restricted parties lists prior to exporting. All but one of the nine universities we visited have developed policies and practices that fully align with this element concerning export authorization and tracking export-controlled items. Below, we provide additional detail on universities’ activities within the seven sub-elements, which fall under three process categories: making export decisions, tracking and safeguarding export-controlled items, and screening foreign parties. Under this category, we assessed universities’ activities in the following two areas: Identifies research involving export-controlled items: Officials at all nine of the universities we visited stated that they had, to varying degrees, developed policies and practices for identifying research projects that might involve items that are subject to export control regulations. Policies and practices for identifying research involving export-controlled items. All nine of the universities we visited require the lead researcher on a project to submit research proposals to an office charged with reviewing proposals and awards for grants and contracts, which we refer to as the Office of Grants and Contracts. The office also reviews the terms and conditions for awards—contracts, grants, or cooperative agreements—to ensure there is nothing in the paperwork that necessitates additional negotiation or that raises a concern related to export controls. When reviewing research proposals or awards, the Office of Grants and Contracts will flag those proposals and awards that may involve items subject to export control regulations for further review, either by the export control officer or another authorized university entity. Tools developed to support officials’ identification of research involving export-controlled items. The universities we visited have developed a variety of tools to support officials’ export control reviews of proposals and awards. For example, seven of the nine universities we visited require the lead researcher on a project to complete a questionnaire that includes export control- related questions when submitting research proposals for review. This questionnaire identifies research proposals that may be subject to export control regulations earlier in the process. In addition, at least four of the universities’ export control officers have developed flowcharts or checklists to help the Office of Grants and Contracts understand when to flag research proposals or awards for further review by the export control officer. In addition, seven of the nine universities we visited require that researchers obtain university approval to conduct research involving export-controlled items. For example, one university’s export control officer said that flagged proposals are sent to an export control review committee for review and approval. The committee reviews the risk associated with each of these research projects and determines whether the university is willing to accept the export control-related risks for that project. Another university requires the lead researcher to obtain approval from the university’s board before accepting an award for research involving export-controlled items. Monitors research to determine whether a license is required after the project starts. Officials at five of the nine universities described practices they had developed to monitor research projects in order to determine whether an export license is required after a research project is underway. For example, one university’s export control officer said her department monitors all research teams that intend to develop hardware or technology during their research because the resulting hardware or technology could be subject to export control regulations. These projects are flagged in the electronic system used to track research projects and the export control officer checks in with the lead researcher periodically to determine the status of the research. An official at another university explained that the university conducts periodic audits of timecards to see if any foreign persons have started charging time to ongoing projects involving export-controlled items. In contrast, one official at another university stated that the university relies on the lead researcher to alert the compliance office of any changes to the research team or research objectives, which may then require a license before continuing research. This official suggested that the lead researchers are better positioned than the export control officer to identify changes to the research that might necessitate obtaining an export license. Tracking and Safeguarding Export-Controlled Items Seven of the nine universities we visited used a variety of mechanisms to track and safeguard export-controlled items, including manual locks, electronic access systems, and other physical security systems, as well as separate computer networks to protect data subject to export control regulations. Under this category, we assessed universities’ activities in the following three areas: Tracks export-controlled items used at the university. Officials at seven of the nine universities we visited said they had developed mechanisms to track any export-controlled items being used or developed by the university. These mechanisms range from maintaining paper files to using electronic systems to track such information. For example, some of the universities maintain physical copies of documents they use to identify and track export-controlled items on campus. Other universities have developed electronic databases to track this information. One university maintains all records related to research projects in one electronic system, including technology control plans. Electronic databases and systems allow the export control officer to quickly identify the on-campus location of export-controlled items and who is working with these items. Safeguards export-controlled items. Eight of the nine universities we visited employ various security mechanisms to protect export- controlled items, including physical and information technology security mechanisms. For example, officials at seven of the nine universities we visited said their university protects export-controlled items by limiting access to spaces where these items are housed with locks or access cards, depending on the space. Three of these universities also require researchers to store export-controlled items in a locked box or storage space, in a locked room, when it is not in use. Some universities also use signs to indicate which spaces are restricted; however, officials at one university said that they do not use signage to indicate restricted spaces because it would draw more attention to the space. Some university officials also described information technology security mechanisms in place to protect data that may be subject to export control regulations. For example, officials at two universities noted the use of isolated or separate networks for researchers working with such data to limit access to this data. Uses technology control plans to document and safeguard export-controlled items. Officials at all nine of the universities we visited stated that researchers used export-controlled items on campus, and officials at eight of these universities said that their universities had developed and implemented technology control plans to safeguard such items. According to Commerce’s export compliance guidelines, organizations that possess or work with export-controlled items and either employ foreign persons or have frequent meetings with foreign persons should create a technology control plan. These plans should include a physical security plan, an information security plan, and training programs, among other components. According to the university officials we interviewed, the export control officer typically works with the lead researcher to develop the technology control plan. Six of the nine universities we visited require the lead researchers to sign the technology control plan to acknowledge that they understand their responsibilities for protecting the export- controlled items identified in the plan, and at least four of these universities require all the members of the research team to sign it as well. In addition, some of the universities we visited conduct annual audits of the technology control plans to ensure proper implementation. For example, an official at one of these universities explained that the university’s annual audit of the technology control plans verifies that security practices outlined in the plan are being followed by the research team and that only those researchers who signed the technology control plan have access to the export- controlled items. An official at another university said he reviews the human resources account information for projects involving export- controlled items annually to verify that only those individuals who have signed the technology control plan are working on those projects. Under this category, we assessed universities’ activities in the following two areas: Screens and monitors foreign visitors. All but one of the nine universities we visited screen and monitor foreign visitors to some extent. Specifically, four of these universities conduct restricted party screenings on all foreign visitors prior to their visit to verify that potential visitors are not on any U.S. government list of restricted or proscribed parties. The other four universities conduct restricted party screenings on some foreign visitors. Three of these four universities said that they do not have a formal process for reviewing foreign visitors and that the effort to invite and review visitors is decentralized. Some of the universities we visited also described how they monitor foreign visitors on campus. For example, officials at two universities said that the foreign visitors’ sponsor is responsible for monitoring their access. The export control officer at a third university told us that he briefs foreign persons visiting restricted spaces on the rules of their visit, including restrictions on camera usage. Screens foreign parties associated with research projects. All nine of the universities we visited use restricted party screening software, which searches several lists that U.S. agencies continually update to screen for restricted or denied parties. Universities and other exporters may be prohibited or restricted from doing business with any individuals or entities identified on one of these lists. Eight of the nine universities we visited screen all foreign individuals and entities associated with a research project using such software. Entities associated with a research project may include foreign researchers on the research team, foreign sponsors, or foreign collaborators, among others. Officials at the ninth university stated that they conduct ad hoc screening for research collaborations with foreign entities. Additionally, one of the universities has compiled a list of all the foreign entities the university works with and conducts weekly restricted party screenings of the foreign entities on this list. Although we focused our assessment on universities’ export compliance policies and practices in place to limit unauthorized deemed exports to foreign persons, officials at some of the universities we visited discussed their efforts to conduct restricted party screenings for other process areas, such as shipping, procurement, and gifts. We found that individuals or offices responsible for these processes at some universities manually screened entities. In one case, this was because the other offices did not have access to the restricted party screening software that the export control officer used. For this element, we assessed the extent to which the university had developed processes for maintaining relevant export control-related records. See figure 9 for the results of our assessment. All nine of the universities we visited have developed policies and practices that fully align with this element concerning recordkeeping. Below, we provide additional detail on universities’ recordkeeping activities. At least five of the nine universities we visited maintain their export compliance-related records in an electronic database or other electronic system. For example, one university’s system tracks each research project from start to finish and enables officials to search for all export control-flagged research proposals and awards, technology control plans, and other documents. One of the officials also told us that the system will alert the export control officer to any technology control plans with an upcoming expiration date. Officials at another university explained that their system also enables them to track all the approved technology control plans to quickly identify who is working under a technology control plan on campus at any point in time. Five of the nine universities we visited have written export compliance program manuals, and all of those universities’ manuals include information concerning recordkeeping requirements. For example, four of the five manuals specifically note that export control-related files must be maintained for at least 5 years, and four identify the types of records that need to be maintained, including export reviews, contracts, licenses, technology control plans, and shipping documents, among others. For this element, we assessed universities’ activities within two sub- elements: whether the university (1) provided export control-related training to all employees involved in exports and (2) required any individuals to complete mandatory export control-related training. See figure 10 for the results of our assessment. Seven of the nine universities we visited have developed policies and practices that fully align with this element concerning training, while the other two have not. Below, we provide additional detail on universities’ activities within the following two sub-elements: Provides export control-related training to all employees involved in exports. Seven of the nine universities we visited stated that they provide export control-related trainings to researchers and other officials involved in the implementation of export control regulations. The export control-related training available to various university officials at the universities we visited varies depending on officials’ level of interaction with export controls. For example, at least five of the universities’ export control officers we interviewed provide export control-related training tailored to the needs of staff whom the university relies on to identify requests for export-controlled items or research involving export-controlled items, including the procurement office and the Office of Grants and Contracts. One export control officer stated that he provides annual training to officials in the Office of Grants and Contracts and provides biannual training to officials in the procurement office. He noted that he spends the most time training officials responsible for reviewing grants and contracts because they are the “gate keepers” for all research proposals and research funding coming into the university. The two universities that do not provide export control-related training to all employees involved in exports do make some export control-related information available. An official from one of the universities said that the university provides access to online export control-related trainings developed by a for- profit entity. The export control officer at the other university said that although the university does not conduct formal training, he conducts frequent outreach and provides materials to increase university officials’ awareness of export control regulations. Conducts mandatory training for researchers conducting research involving export-controlled items. Seven of the nine universities we visited require researchers conducting research involving export-controlled items to complete training with the export control officer prior to beginning their project. Furthermore, researchers at four of these universities are required to complete additional periodic training to refresh their understanding of their compliance roles and responsibilities every 1 to 3 years. Most of the universities that conduct required export control training have varying systems in place to document attendance. For example, three of the nine universities we visited require attendees to sign a form certifying that they have completed the technology control plan training and understand their responsibilities. For this element, we assessed the extent to which the university conducted periodic audits of its export control compliance program to assess its effectiveness and integrity. See figure 11 for the results of our assessment. Eight of the nine universities we visited have developed policies and practices that fully or partially align with this element concerning internal audits, while one of the universities’ policies and practices did not align with this element. Below, we provide additional detail on universities’ efforts to conduct periodic audits of their export control compliance programs to assess their effectiveness and integrity. Eight of the nine universities we visited conduct some type of internal audit to assess the export compliance program’s effectiveness. For example, five export control officers at these universities review all technology control plans annually. One official said her office conducts these annual reviews to ensure that researchers are properly implementing the technology control plans and to determine if the plans need to be updated to address any changes to the export control regulations. In addition, seven of the nine universities we visited have an internal audit group, and four of these audit groups had conducted an audit of the export compliance program within recent years. One university official explained that the audit group’s periodic review of the export compliance program once found that the project management system did not provide enough transparency, and on the basis of this finding, the export control officer was able to petition the university for additional funding to further improve the system in place to track all research projects. According to an official at another university, a quality assurance official at his university audits a sample of research awards each month. Every few months, this official identifies a mistake, such as a failure to screen a foreign party against the lists of restricted parties. When a mistake is identified, the export control officer then screens the foreign party and counsels the person who missed this step. These audits provide universities with an opportunity to identify any potential gaps and continually improve their programs. For this element, we assessed the extent to which the university had developed clear procedures outlining the actions employees should take in the event that potential noncompliance is identified. See figure 12 for the results of our assessment. All nine of the universities we visited have developed policies and practices that fully align with this element concerning the reporting of violations. For example, officials at seven universities told us that they have a compliance hotline that people can use to report suspected violations. Two of these seven universities described additional actions they have taken to further educate their university community about the need to report potential export control violations by adding such information to flyers for the university compliance hotline and advertising this information online. Officials at three of the universities also discussed escalation procedures they have in place to investigate a potential export control violation. For example, one export control officer explained that he is responsible for investigating and reporting any violations. If he needs to initiate an investigation, he will select a team of university officials to enquire about the violation and determine whether a violation has occurred. Following the investigation, the Vice President for Research is responsible for determining whether the university needs to self-disclose a violation to the relevant federal regulatory agency. Five of the nine universities we visited had written export compliance program manuals, and all of those universities’ manuals included information concerning export control violations. For example, some of the manuals include a discussion about the legal and criminal penalties associated with export control violations and emphasize the importance of reporting any potential violations. In addition, two of the universities’ manuals describe the need to develop corrective action plans to prevent recurrence of any violations arising from systemic institutional practices or procedures. Three of the nine universities we visited had voluntarily disclosed export control violations. For example, one university disclosed information regarding a foreign person’s unauthorized access to ITAR-controlled technology because the lead researcher on the project and the procurement office did not know the technology was controlled. According to the export control officer at this university, her office is working with the procurement office to ensure that the future procurement of controlled technologies is flagged for review by the export control officer prior to ordering. This updated procedure will enable the export control officer to work with the lead researcher to develop a technology control plan if the university agrees to support the procurement of such a technology. For this element, we assessed the extent to which each university documented export control compliance processes, roles and responsibilities, and other relevant information in a manual to help the university implement its compliance program. See figure 13 for the results of our assessment. Five of the nine universities we visited have developed export compliance manuals, consistent with this element, while the other four have not. These manuals describe the export control-related roles and responsibilities of various offices and officials on campus, including the export control officer and university researchers, among others. In general, the manuals also describe a number of export control compliance procedures, including the initial review of research proposals, development of technology control plans for research involving export- controlled items, training requirements, and processes for investigating potential violations, among others. Four of the five universities developed manuals in 2015 or earlier, and one university developed a manual in 2018. Three of the universities that published manuals in or before 2015 have updated their manuals at least once, but one of these universities has not updated its manual since 2013. We reviewed the public websites of a statistically generalizable sample of 100 U.S. universities expending more than $15 million for research and development annually, on average, to determine the extent to which universities publicly share export control-related information with their campus community. Using research expenditure data collected by the National Science Foundation for 2013 through 2017, we identified 292 public and private U.S. universities that expended more than $15 million on research and development, on average, over a 5-year period. We selected a stratified, random sample of 100 universities from this list to provide representation from a diverse set of universities in our sample. Next, we created a top and bottom stratum based on total research and development expenditures. The top stratum included universities with expenditures above $250 million (85 universities) and the bottom stratum included universities with expenditures between $15 million and $250 million (207 universities). The sample included 55 universities from the bottom stratum and 45 from the top stratum. Of the 55 universities from the bottom stratum, 30 are public and 25 are private. Of the 45 universities from the top stratum, 25 are public and 20 are private. We assessed the information on the selected universities’ websites against six of the eight elements of an effective export compliance program: 1. Management commitment and organizational structure 2. Export authorization and tracking export-controlled items 5. Reporting and addressing violations 6. Export compliance manual We did not review information related to risk assessments or internal audits on the selected universities’ websites because we did not expect universities to publicly publish this type of information. Of the 100 universities in our sample, 77 maintained a dedicated web page for export control-related information, and 79 provided contact information for the person or office responsible for complying with export control regulations on their website. However, only about half of the universities’ websites identified an export control officer or similar official, and only 24 included a public statement from university management supporting the export compliance programs. See table 8 for additional results from our website analysis. Management commitment and organizational structure Entities should have public management support for their compliance program, sufficient resources to conduct compliance activities, and a clear organizational structure identifying individuals responsible for compliance. related information? Export Control Officer or similar title identified? Export control roles and responsibilities of researchers described? Export authorization and tracking export- controlled items Entities should develop processes to (1) ensure the organization makes correct export decisions, including identifying when U.S. government authorization is required prior to exporting; (2) track and protect any export- controlled items being used or developed by the organization; and (3) screen all parties associated with an export transaction against the U.S. proscribed/restricted parties lists prior to exporting. A majority of the 100 universities’ websites included information about relevant export regulations and a definition of exports, and almost half provided additional resources or tools for researchers to better understand how or whether their research involves items subject to export control regulations; however, a limited number provided information about practices the university may employ to protect export-controlled items. For example, 74 of the 100 universities published information about the International Traffic in Arms Regulations (ITAR) and the Export Administration Regulations (EAR) on their websites. About half of the universities also maintained a frequently asked questions section concerning export control regulations and about half provided tools such as decision tree matrices to help researchers determine whether an export may require a license. However, less than a third of the universities’ websites included any information about technology control plans or guidance regarding foreign visitors, which are practices that universities may undertake to protect export-controlled items used in university research or other academic activities. For example, only 27 of the 100 universities’ websites contained explanations of when a technology control plan would be necessary. See table 9 for additional results from our website analysis. Twenty of the 100 universities’ websites provided information regarding export compliance recordkeeping requirements. See table 10 for these results. About half of the universities’ websites provided information about export control trainings available online, developed by the university, associations, or for-profit organizations, among others. However, only 21 of the 100 universities’ websites provided information about how to request university-provided, in-person training regarding export compliance. See table 11 for additional results from our website analysis. Only about a quarter of the universities’ websites provided guidance about when to report potential violations, but about half of the universities’ websites provided information about the potential administrative or criminal penalties associated with export control violations. See table 12 for additional results from our website analysis. Reporting and addressing violations Entities should develop clear procedures outlining the actions employees should take in the event that potential noncompliance is identified. Entities should also develop processes for identifying and addressing the root cause of any noncompliant activity. Guidance on when to report a potential export control violation? Less than half of the universities in our sample published an export compliance manual on their website. See table 13 for these results. Kimberly Gianopoulos, (202) 512-8612 or gianopoulosk@gao.gov. In addition to the contact named above, Juan Gobel (Assistant Director), Drew Lindsey (Assistant Director), Amanda Bartine (Analyst-in-Charge), Taylor Bright, Debbie Chung, Neil Doherty, Tina Huang, Kathryn Long, Sulayman Njie, and Jina Yu made key contributions to this report. Ashley Alley and Justin Fisher provided technical assistance.", "summary": "Over 1.2 million foreign students studied at U.S. universities in 2018 (see fig.). Although foreign students and scholars contribute to U.S. research, there is a risk that they will “export” sensitive knowledge they gain to their home countries. To mitigate this risk, the U.S. government implements export controls. GAO was asked to review agency guidance and universities' security practices. This report examines (1) the extent to which State and Commerce have provided guidance and outreach that supports U.S. universities' understanding of export control regulations; (2) challenges U.S. universities face working with other federal agencies, such as DOD; and (3) the extent to which universities' export compliance practices align with State and Commerce guidelines. GAO reviewed related laws, regulations, and guidance, and interviewed officials from relevant federal agencies and four university associations. GAO also visited nine universities—selected, in part, on the basis of research expenditures and geography—and assessed their compliance practices against agency guidelines. The Departments of State (State) and Commerce (Commerce) have each provided guidance and outreach to support exporters' understanding of and compliance with their separate export control regulations. Exporters, including universities, are subject to these regulations if they ship export-controlled items overseas or if they share such items, including technology or source code, with foreign persons in the United States. University and association officials raised concerns that State and Commerce guidance and outreach does not adequately address export compliance issues that are more common to universities than to industry, such as fundamental research—i.e., research that is ordinarily published and not subject to export control regulations. Without additional guidance and outreach that addresses such issues, universities may not have the information they need to adequately comply with these regulations and properly safeguard export-controlled items. Officials from selected universities and university associations identified three export control-related challenges in working with other federal agencies. For example, university and association officials asserted that Department of Defense (DOD) officials misunderstand the term fundamental research, which may limit universities' ability to conduct research for DOD. DOD acknowledged that some officials have inconsistently interpreted the regulations and that it has not yet fully addressed this challenge. Additionally, university and association officials expressed concerns that threat briefings and other guidance that the Federal Bureau of Investigation (FBI) and Department of Homeland Security provide are not helpful because, for example, they do not contain unclassified information that can be shared widely. To address these concerns, the FBI partnered with a university association to produce a series of unclassified “awareness-raising” materials for university audiences, among other efforts. Seven of the nine universities GAO visited have export compliance policies and practices that generally align with State's and Commerce's export compliance guidelines. For example, most have demonstrated a strong management commitment to export compliance and have robust practices for tracking export-controlled items, recordkeeping, and reporting potential violations. However, GAO identified gaps in some universities' practices in four areas—risk assessments, training, internal audits, and export compliance manuals. GAO is making four recommendations, including that State and Commerce should improve their export control guidance and outreach, which may help address gaps in university export control compliance practices. GAO also recommends that DOD take steps to ensure its officials consistently interpret export control regulations. State, Commerce, and DOD concurred with the recommendations.", "document_type": "gao"}
{"report": "In a 2006 report, we stated that there was no generally agreed-upon definition of insurance and most definitions in the private sector differed because they were developed for specific purposes or had changed over time. However, the definitions share key elements of risk transfer and risk spreading, and include other elements such as indemnification, which is payment for losses actually incurred; the ability to make reasonable estimates of future losses; the ability to express losses in definite monetary amounts; and the possibility of adverse, random events occurring outside the control of the insured. The Financial Accounting Standards Board establishes generally accepted accounting principles for private-sector entities. The Board defines an insurance contract as “a contract under which one party (the issuing entity) accepts significant insurance risk from another party (the policyholder) by agreeing to compensate the policyholder or its designated beneficiary if a specified uncertain future event (the insured event) adversely affects the policyholder.” Generally, private insurers offer several types of insurance products for individuals, families, or businesses, including health, disability, life, annuity, and property and casualty insurance products. Per the terms of the contracts, insurers generally offer coverage for losses from specified events in exchange for premium payments. Private insurers spread risk over a large enough group of insured parties to reasonably predict total losses, then set risk- based premium rates to make a profit and cover costs, including claim payments for covered losses, administrative expenses, and other expenses associated with selling and servicing policies. Similarly, there is no generally agreed-upon definition of insurance as it relates to federal activities. FASAB, which establishes generally accepted accounting principles for federal entities, provides a definition for financial reporting purposes. FASAB defines an insurance program as “a general term used to refer to a program that is authorized by law to financially compensate a designated population of beneficiaries by accepting all or part of the risk for losses incurred as a result of an adverse event.” In addition to activities that may fall under the FASAB definition of insurance, the federal government undertakes other activities that compensate or provide benefits to individuals or other third parties that suffer losses from an adverse event. Unlike private insurance policies, these federal activities do not necessarily utilize a contract or charge premiums or fees in exchange for goods, services, or benefits. Even when premiums or fees exist, they may not cover all costs, as federal expenditures can be driven more by policy goals or agency missions than a desire to achieve fiscal solvency. For example, by design, premiums collected through the Federal Crop Insurance program do not cover its costs. One of the program’s goals is to help farmers manage the risks inherent in farming, such as the risk of poor crop yields and declines in prices, and it does so while subsidizing more than 60 percent of the premiums. In addition, Congress created the National Flood Insurance Program (NFIP) to address the increasing cost of federal disaster assistance by providing flood insurance to property owners in flood-prone areas, where such insurance either was not available or prohibitively expensive through the private sector. Subsidized premium rates, among other things, have precluded NFIP from achieving rates that reflect the full risk of loss, and the program has not had sufficient funds to pay claims. Similarly, the federal government uses the Disaster Relief Fund to provide disaster relief assistance without collecting premiums or other fees from the entities receiving the funds before or after an event occurs and without knowing beforehand who might receive compensation. The President’s Budget and the Financial Report present complementary perspectives on the federal government’s financial position and condition. As illustrated later in the report, differences in when costs are recognized for budgetary or financial reporting purposes can provide substantially different measures of cost in a given year for some federal activities, including federal insurance. First, the federal budget process serves as the primary financial plan of the federal government and thus plays a critical role in the decision- making process for all federal expenditures, including those for insurance programs and other activities that transfer risk or losses to the federal government. The President, federal agencies, and Congress use the annual budget process, in part, to plan how federal funds should be spent for federal activities and track budget approval and execution. Besides the President’s proposed appropriations for a given fiscal year, the President’s Budget also reports actual data for two prior fiscal years and estimated data for the prior fiscal year, such as the budget authority, unpaid obligations, and outlays (liquidated obligations). Generally, receipts are recorded when the federal government receives the cash and spending is recorded when outlays (payments) are made. We refer to the President’s Budget as primarily “cash-based.” The Secretary of the Treasury, in coordination with the Director of OMB, prepares the annual Financial Report, which consolidates and summarizes financial information from federal agencies and departments. The Financial Report provides an overall view of the annual financial results of operations, condition, and position of the federal government. In particular, it provides the net operating cost of the federal government by comparing its revenues and costs. The report follows FASAB accounting standards and generally records transactions on an accrual basis—not on a cash basis—to recognize and track assets, liabilities, revenues, and expenses. That is, expenses and liabilities are recorded when they are incurred, even if payment is due at a later date, and revenues (other than taxes and other nonexchange revenues) and related assets, such as receivables, are generally recorded when amounts are earned, even if actual receipt occurred at an earlier time. Our analysis allowed us to identify 148 federal activities that transfer risk or losses from adverse events to the government (see table 1). We broadly categorize the federal activities that met our criteria as follows: federal insurance programs; federal loan guarantee programs; senior preferred stock purchase agreements with two government-sponsored enterprises (enterprises), Fannie Mae and Freddie Mac; federal employee and veterans benefits (excluding education benefits and burial benefits); and other activities, such as those that provide property damage or financial loss compensation and those that offer life, health, or disability benefits to nonfederal employees. We generally only provide cost and exposure information for activities as available in the Financial Report and agency documents. See appendix I for more information on our scope and methodology. Through our analysis of the Financial Report, we identified five federal insurance programs. While there is no universally accepted definition of what constitutes a federal insurance program, federal agencies have reported insurance and guarantee liabilities and related note disclosures in the Financial Report for the following programs: Department of Agriculture, Risk Management Agency’s (RMA) Federal Crop Insurance Program; Department of Homeland Security, Federal Emergency Management Agency’s (FEMA) National Flood Insurance Program (NFIP); Pension Benefit Guaranty Corporation’s (PBGC) single-employer and multiemployer pension insurance programs; Federal Deposit Insurance Corporation’s (FDIC) deposit insurance program; and National Credit Union Administration’s (NCUA) share insurance program. See table 2 for descriptions of the programs as well as related funding and cost or exposure information. All five federal insurance programs collect premiums, assessments, or fees, but the programs differ in the extent to which they are designed to fund their liabilities using only these sources of income. Federal crop insurance premiums are subsidized by the federal government by law. In addition, RMA automatically receives a permanent indefinite appropriation each fiscal year for this premium subsidy and other expenses, and it returns unobligated balances to the U.S. Treasury at the end of the fiscal year. The other programs are generally intended to use premiums or assessments collected and other nonfederal assets and income to pay claims or guarantees. The programs also differ in their authority to borrow funds from the U.S. Treasury to pay claims and other expenses, as well as their use of such authority. As of September 30, 2018, NFIP had about $20.5 billion of outstanding debt with the U.S. Treasury (after Congress granted $16 billion in debt cancellation to NFIP in October 2017). Before 2004, NFIP was able to cover most of its claims and repay occasional loans from the U.S. Treasury with premiums it collected, but it has not been able to do so since, partly due to extraordinary catastrophic loss years resulting from Hurricane Katrina and Superstorm Sandy. According to FEMA, the program as currently designed is unlikely to be able to repay this debt. PBGC is expected to fund itself entirely through premiums, other nonfederal assets and income (such as investment income), and assets from underfunded, terminated single-employer plans it takes over, as it currently does not receive taxpayer funds and does not have authority to borrow funds from the U.S. Treasury. The balance in FDIC’s deposit insurance fund (DIF) fell to negative $20.9 billion as a result of bank failures triggered by the 2007–2009 financial crisis. As required by law, FDIC implemented a plan to replenish the DIF and raise the reserve ratio to its designated minimum in the time limits prescribed by the Federal Deposit Insurance Act. Pursuant to the plan, FDIC raised assessment rates and imposed a one-time special assessment to recapitalize the DIF. To meet the projected liquidity needs for failures of FDIC-insured depository institutions during the financial crisis, FDIC required the banking industry to prepay its quarterly risk-based assessments for the fourth quarter of 2009 and for the following 3 years. FDIC did not use its authority to borrow funds from the U.S. Treasury. On the other hand, NCUA’s Share Insurance Fund borrowed funds from the U.S. Treasury during the 2007–2009 financial crisis but has since repaid the loans. Lastly, the programs also differ in their expected ability to cover future losses. RMA receives a permanent indefinite appropriation each fiscal year to pay for its commitments, so certainty exists that the program will be able to pay future losses using such appropriations. According to FEMA, as currently designed, NFIP likely will not have enough funds to cover all future program expected losses. NFIP also would not have enough funds to cover a single super-catastrophic year, in which NFIP could experience as much as $40 billion in claims, according to FEMA. While PBGC programs have been able to pay all guaranteed benefits and financial assistance to date, PBGC forecasts a very high likelihood of insolvency for the multiemployer program in the next several years if there are no changes in law. DIF’s ability to pay future claims depends on whether the fund has sufficient assets. Congress sets a minimum ratio of assets to insured deposits for the DIF (called the reserve ratio), which by statute must be at least 1.35 percent by September 30, 2020. In addition, FDIC sets a target ratio (called the designated reserve ratio), currently set at 2 percent. FDIC views the designated reserve ratio as a minimum goal that will allow the fund to grow sufficiently large in good times to increase the likelihood of the fund remaining positive during bad times. DIF’s reserve ratio was 1.36 percent as of September 30, 2018. To comply with the statutory requirement that large banks—those with total assets of $10 billion or more—bear the responsibility of increasing the DIF reserve ratio from 1.15 percent to 1.35 percent, FDIC imposed a quarterly surcharge on large banks. According to FDIC officials, the surcharge began in the third quarter of 2016, the quarter after the reserve ratio first reached or exceeded 1.15 percent, and ended in the third quarter of 2018, the quarter in which the reserve ratio first reached or exceeded 1.35 percent. The fund ratio for NCUA’s Share Insurance Fund is called the equity ratio, and has a statutory minimum of 1.20 percent. NCUA’s target equity ratio is called the normal operating level and was set at 1.38 percent by the NCUA Board of Directors on December 13, 2018. NCUA’s equity ratio was 1.46 percent as of December 31, 2017, both above the statutory minimum and the normal operating level. We identified 33 federal loan guarantee programs that transfer risk or losses to the federal government (see full list in app. III). The federal government uses loan guarantees as tools to support specific social and public policy objectives, such as those for housing and small businesses. Federal loan guarantees are any guarantees, insurance, or other pledges with respect to the payment of all or a part of the principal or interest on any debt obligation of a nonfederal borrower to a nonfederal lender. Thus, the federal guarantee transfers some or all of the risks of borrower default from private lenders to the federal government. The Federal Credit Reform Act of 1990 requires agencies to estimate the cost to the government of guaranteeing credit in the President’s Budget, beginning in fiscal year 1992. This cost, the loan guarantee subsidy cost (referred to in this report as “subsidy cost”), equals the net present value of the following cash flows at the time a loan guarantee is disbursed by the lender: (1) the estimated payments by the government to cover defaults, delinquencies, interest subsidies, or other payments; and (2) the estimated payments to the government, including origination and other fees, penalties and recoveries. If the present value of estimated cash outflows exceeds cash inflows, there is a positive subsidy cost. If the present value of estimated cash inflows exceeds cash outflows, there is a negative subsidy cost, referred to as subsidy income. Every fiscal year, subsidy costs are (1) estimated for the loan guarantees obligated during that year and (2) reestimated for loan guarantees obligated in previous fiscal years to update costs for actual loan performance and to incorporate any changes in assumptions about future loan performance. If reestimates increase subsidy costs, an agency would need additional funds. If they decrease subsidy costs, an agency generally would return funds to the general fund of the Treasury. Regardless of whether credit programs are discretionary or mandatory, agencies do not need to request additional appropriations to cover upward reestimates because the Federal Credit Reform Act provides permanent indefinite budget authority for this purpose. In addition, the loan guarantee liability in the Financial Report is the present value of estimated net cash outflows. Thus, this liability is an estimate of the exposure to the federal government because of all outstanding loan guarantees. This liability is based on all loan guarantees obligated in a given fiscal year and previous years that are outstanding as of the end of a fiscal year. It takes into account the subsidy costs of these guarantees estimated as of the time the loan was obligated and subsequent adjustments such as modifications and reestimates. As is the case with federal insurance, federal loan guarantee liabilities are publicly reported in the Financial Report and related note disclosures. Table 3 presents some of the measures reported in the note disclosures as of September 30, 2017. The estimated subsidy cost to the government for loan guarantees, including reestimates, was $12.5 billion during fiscal year 2017, as reported in the Financial Report. This was largely attributable to guarantees under Federal Housing Administration (FHA) loans administered by the Department of Housing and Urban Development, the largest of which is FHA’s Mutual Mortgage Insurance (MMI) program. The program provides mortgage insurance to encourage lenders to make credit available to borrowers not adequately served by the conventional market, such as first-time homebuyers, minorities, and lower-income families. Similarly, federal loan guarantee liabilities were approximately $43 billion, with FHA loan guarantees accounting for about 48 percent of all guarantee liabilities. Current senior preferred stock purchase agreements between Treasury and the enterprises, Fannie Mae and Freddie Mac, transfer risk or losses to the federal government. The enterprises purchase mortgage loans that meet certain criteria for size, features, and underwriting standards, known as “conforming” loans. After purchasing mortgages, the enterprises create mortgage-backed securities and guarantee investors in these securities that they will receive timely payments of principal and interest. In 2008, because of the enterprises’ poor financial condition, the Federal Housing Finance Agency (FHFA) placed the enterprises into conservatorship and Treasury agreed to provide capital assistance in part to ensure timely payment to investors in exchange for shares of senior preferred stock, thus transferring risk to the federal government. Under the agreements, Treasury has committed to providing up to $445.6 billion in capital support to the enterprises while they are in conservatorship. If Fannie Mae or Freddie Mac has a net worth deficit at the end of a financial quarter, Treasury will provide capital support to eliminate the deficit. Under the most recent agreement, the enterprises must pay Treasury a dividend of all their quarterly positive net worth above a $3 billion capital reserve that each enterprise is allowed to retain. As of December 31, 2018, the enterprises had paid $292.3 billion in cumulative dividends to Treasury. Since the second quarter of 2012— with the exception of the first quarter of 2018 during which the enterprises required Treasury support due to devaluation of certain assets—Fannie Mae and Freddie Mac have not required additional support from Treasury. As of August 2018, Treasury had provided the enterprises with $191.4 billion of this amount since they were placed under conservatorship in 2008, leaving $254.1 billion in potential taxpayer exposure should Treasury need to provide any additional support. The latter represents the maximum amount of potential future federal spending under the current agreements. According to Treasury, based on their assessments, there were no probable future funding draws as of September 30, 2018, but it was reasonably possible that market volatility or non-recurring events could cause the enterprises to generate quarterly losses and, therefore, result in future funding draws against Treasury’s funding commitment. We identified 13 large federal employee and veterans benefit activities that transfer at least some of the risk or losses to the federal government (see app. IV for more information). The federal government offers its civilian and military employees health and life insurance, defined benefit pension and other retirement benefits (such as post-retirement health insurance and life insurance), and other benefits. Many of these benefit programs exchange current services for a guarantee of lifetime annuity payments or the continuation of health insurance coverage, inherently transferring at least part of the risk of an adverse experience—such as people living longer than expected, or health care costs rising faster than expected—from the employees to the federal government. For example, the following three pension activities account for more than 80 percent of all federal employee and veterans benefit liabilities: On the civilian side, the Office of Personnel Management (OPM) administers the Civil Service Retirement System and the Federal Employees Retirement System, which are the largest civilian pension plans covering nearly all full-time, permanent civilian federal employees. The Department of Defense and the Department of Veterans Affairs administer the largest military plans. The Department of Defense administers the Military Retirement System, and Veterans Affairs provides for the payment of compensation, pension, and burial benefits to veterans and survivors. Federal employee and veterans benefit liabilities are publicly reported in the Financial Report and related note disclosures. Generally, these liabilities are recorded as employee services are rendered. Table 4 presents liabilities of the government for certain federal employee and veterans benefit activities. Such benefits include deferred compensation that generally commit the federal government to provide cash compensation and health insurance following a term of service and to accept certain risks regarding the ultimate costs of those benefits. These liabilities were approximately $7.6 trillion as of September 30, 2017, and represented about 32 percent of all federal liabilities (which were $23.9 trillion). An analysis of the President’s Budget, the Catalog of Federal Domestic Assistance (CFDA), and the U.S. Code yielded 95 additional activities that met our criteria of transferring risks or losses from adverse events to the federal government. These activities can be broadly categorized into those that provide compensation for property or financial losses— including losses resulting from adverse environmental or manmade events—and those that offer life, health, or disability benefits to nonfederal employees. See appendix V for information on all 95 activities. Some of these federal activities provide compensation to specific third parties if they suffer certain losses from future adverse events, but the federal government may not always charge premiums for accepting this risk of loss. For example, the Department of Agriculture’s Price Loss Coverage Program provides payments to farmers of certain crops when the effective price of the commodity is less than a reference price for that commodity. Farmers can apply to receive such assistance and do not pay premiums to receive benefits. We also found other activities in which the beneficiaries who receive government compensation for their losses are known only after an adverse event occurred. This was generally the case for activities that provide compensation for property or financial losses to victims of unforeseen adverse environmental or manmade events, such as activities funded by the Disaster Relief Fund. Lastly, we found activities that offer life, health, or disability benefits. These include federal grants to states for Medicaid, which assists states in providing medical care to generally low-income individuals, and activities that support mental health services, treatment for substance abuse, or child health insurance services. For example, the Department of Health and Human Services administers the Children’s Health Insurance Program, which provides funds to states to help them maintain and expand health assistance to uninsured, low-income children and, at a state’s option, low-income pregnant women. Table 5 has information on the budget accounts we found with more than $10 billion in total new obligations for fiscal year 2017 that funded activities that transferred risk or losses to the federal government. While obligations are a legal liability for the federal government, they may not necessarily reflect an activity’s fiscal exposure if, for example, the activity has dedicated payment streams that reduce the government’s fiscal exposure. We found five additional activities authorized in law that have not yet caused financial liabilities to the federal government but may do so if certain adverse events occurred. Such events include acts of terrorism, nuclear power plant incidents, or catastrophic space launch-related incidents. In all five cases, the federal government is generally authorized to help finance third-party liability claims related to the event, if needed, after private-sector insurers have paid a certain level of claims. As seen in table 6, some of these activities could require large, previously unbudgeted expenditures by the federal government if an event occurred. As of December 31, 2018, these activities had not triggered losses to the federal government. As mentioned earlier, we listed federal activities that meet the following criteria: (1) a risk of financial loss or actual financial loss to a third party exists that stems from an adverse event; and (2) through the activity, the federal government accepts some or all of the risk of financial loss or actual financial loss from the adverse event by indemnifying, guaranteeing, or providing benefits to the affected entity or beneficiary. Our categorization of such activities as federal insurance activities, federal loan guarantees, senior preferred stock purchase agreements with Fannie Mae and Freddie Mac, certain federal employee and veterans benefits, or other programs, was driven by the sources we used, in particular the Financial Report. Our results were based solely on the criteria we developed for this report and the sources and methodologies we used. Other criteria, sources, or methodologies might yield lists that differ from ours in number and composition of activities. Expert opinions sometimes differed on which types of activities met our criteria. We acknowledge the different opinions. In updating our 2005 catalog of federal insurance activities, our efforts are aimed at providing Congress with an expanded list that helps convey the wide variety of activities that may not necessarily be considered federal insurance but share important aspects of insurance. We also intended to highlight laws that authorize the federal government to cover third-party liabilities from specific adverse events—such as terrorist attacks or nuclear accidents— and that have not yet resulted in liabilities to the federal government but could do if the events occurred. Our catalog may not be appropriate for other purposes. In our previous work, we found challenges in relation to measuring and reporting fiscal exposures caused by certain federal activities, including federal insurance programs. We previously reported that the primarily cash-based budget may not accurately reflect the costs the government incurs and the payments the government may be expected to make for some activities that transfer risk or losses to the government. In addition, the amount of the exposure to the federal government can be hard to measure for some activities. These challenges still exist, and to illustrate them, we reviewed six activities from among those we identified that transfer risk or losses to the federal government. Federal activities that transfer risk or losses to the federal government have a range of fiscal exposures in which the extent of the government’s legal commitment varies (see fig. 1). In 2003, we developed a conceptual framework for fiscal exposures that notes fiscal exposures may be explicit or implicit. Explicit exposures are commitments that the government is legally required to fund, while implicit exposures arise not from a legal commitment, but from current policy, past practices, or other factors that may create the expectation for future spending. Some federal activities have a combination of explicit and implicit exposures. For example, the government is not legally required to cover PBGC insurance claim losses in excess of PBGC’s available resources. Therefore, claims up to the statutory limit are explicit exposures, and losses in excess of PBGC’s available resources represent an implicit exposure for the federal government to the extent there is an expectation that the government would step in and cover losses beyond the program’s reserves. In contrast, loan guarantees under the MMI Fund represent an explicit exposure only, because the government has a legal commitment to pay claims if the borrower defaults on a loan. Implicit exposures may not be evident in the budget, because the primarily cash-based budget records spending only when payments are made. For example, as part of the Commercial Space Launch Insurance Program, the federal government is potentially liable for damages from commercial space launch accidents, subject to appropriation, up to $3.1 billion per licensed space launch in 2017. This program represents an implicit exposure because a new appropriation—which would represent the federal government’s legal commitment to pay for this program—is required to fund damages. Because there has never been such an event or appropriation, this fiscal exposure has not been included in the budget. According to Federal Aviation Administration officials, the agency has not designed internal processes or procedures to address these potential costs, such as estimating the costs, in part because the agency cannot presume the government will provide funds until such an appropriation were made. While implicit exposures do not present a legal commitment to the government, the federal government historically has shown a willingness to fund them in some cases. For example, NFIP has authority to borrow funds from the U.S. Treasury. To the extent there is an expectation that the federal government will cover claims exceeding the amount NFIP has been authorized to borrow from the U.S. Treasury, NFIP represents an implicit exposure. In October 2017 when NFIP was about to exhaust its borrowing authority, Congress demonstrated its willingness to fund NFIP implicit exposures by passing a supplemental appropriation, which the President signed into law, that cancelled $16 billion of NFIP’s past borrowing from the U.S. Treasury. This allowed NFIP to borrow an additional $6.1 billion that would have exceeded its borrowing authority without this intervention, while also reducing its overall debt. Additionally, in fiscal years 2005–2018 the federal government designated a total of $138 billion in supplemental appropriations to the Disaster Relief Fund for declared disasters (see fig. 2). These costs indicate that there was an implicit exposure because Congress must pass a supplemental appropriation to cover them. Congress passed at least one supplemental appropriation for a major disaster in 9 of the 14 years during 2005–2018. As we previously found, the federal budget may not accurately reflect the government’s costs or the likely claim on federal resources from activities that transfer risk or losses to the government. Again, except for loan guarantees, the federal government’s primarily cash-based budget generally does not record the full cost of commitments incurred in the present until corresponding payments are made in the future. However, for some claims, such as pension and post-retirement life insurance, the federal commitment occurs years before the related cash consequences are reflected in the budget. For example, the cost of pension plan insurance accrued in a given year is not reflected in the budget; rather, premiums are shown as receipts when they are collected and payments are shown as outlays when they are made. In fiscal year 2017, the budget showed PBGC’s annual receipts exceeded its outlays by $4.8 billion. But in the same year, the program also had a $76 billion negative net position, which is one measure of the magnitude of the government’s fiscal exposure and is not included in the budget (see fig. 3). Similarly, the budget may not indicate the government’s long-term exposure from weather-related events. Like PBGC, NFIP reports premiums in the budget as receipts in the year they are collected and payments as outlays in the year they are made. The budget does not currently include information on NFIP’s liabilities, assets, or net position. In fiscal year 2017, the budget showed a deficit (outlays exceeded receipts) for NFIP of $2.2 billion, a modest deficit compared to NFIP’s net position in the same year, which fell by $11.5 billion to negative $37.4 billion (see fig. 4). NFIP’s $30.425 billion in debt to the U.S. Treasury at that time, which was included in its liabilities, contributed to its net negative position. We previously reported that FEMA is required by law to charge many policyholders less than full-risk rates, otherwise known as subsidized rates. We found that FEMA’s budget does not recognize the subsidy, making it difficult to analyze the effect of subsidized premium rates on the overall financial stability of NFIP. In 2017, we recommended that Congress consider comprehensive reform to improve NFIP solvency and enhance national resilience to floods. At that time, we developed five policy goals for evaluating options for reforming NFIP that included requiring transparency of the program’s federal fiscal exposure. Congress is still considering various reforms related to NFIP. In addition, the budget request for the Disaster Relief Fund, which provides a significant portion of the total federal response to major disasters, traditionally has been intended to cover each fiscal year’s costs for previously declared disasters and the predictable cost of noncatastrophic events. It does not pre-fund anticipated needs from disasters that have yet to occur or seek funding for potential catastrophic events. As previously noted and shown in figure 2, extreme weather events have cost the nation more than $100 billion through supplemental appropriations to the Disaster Relief Fund since fiscal year 2005. According to the Analytical Perspectives of the President’s 2019 Budget, inflation, urbanization, and other factors are expected to contribute to increasing future disaster response and recovery costs. Additionally, climate change could increase the costs of severe weather events as more frequent and extreme weather and climate-related events are expected to continue to damage infrastructure, ecosystems, and social systems, according to the United States Global Change Research Program. This is one reason we added the federal government’s fiscal exposure created by climate change to our 2013 High Risk List. However, the increased fiscal exposures are not reflected in the Disaster Relief Fund’s financial measures in the budget. According to FEMA officials, the agency does not forecast costs or exposures for catastrophic disasters, in part because each catastrophe is different and presents its own set of risks that would be very difficult to predict with reasonable certainty. While the primarily cash-based budget may not represent an activity’s likely claim on federal resources, members of Congress and the public have access to information on the fiscal health of the activities through other avenues. Many of the activities we reviewed track exposures and other relevant data internally and provide that information to Congress and the public through a variety of reports outside the budget (see table 7). These include actuarial reviews, one-time analyses of various subjects, and annual reports that provide detailed financial information. For example, each year an independent actuarial contractor conducts two separate actuarial reviews of the MMI Fund—one for forward mortgages and one for Home Equity Conversion Mortgages (reverse mortgages)—to estimate the economic value of the two portfolios. FHA then compiles statutorily required annual reports for Congress based on the results of the actuarial analyses. The annual report includes the calculation of the MMI Fund’s overall capital ratio and some additional analyses of the MMI Fund’s financial condition. While these reports provide additional financial information, we previously reported that decision-making is best informed if the government includes in the budget the costs of its commitments at the time it makes them. If the full cost of a spending decision is included in the budget when the decision is made, then decision makers can consider the total costs when setting priorities, compare the cost of an activity with its benefits, or assess the cost of one method of reaching a specified goal against another. Decision makers’ ability to make informed choices would be improved by increased transparency about the impact of policy decisions on the expected path of spending and revenue. We previously recommended that Congress consider expanding the use of accrual-based information to other activities, such as insurance, because accrual measurement would advance the recognition of costs for these commitments, especially those that involve cash flows over many years. We determined that, for many programs, adopting accrual-based information selectively within the current, primarily cash-based budget might improve information while preserving up-front control. PBGC, Federal Employees’ Group Life Insurance (FEGLI), and NFIP officials stated that adding limited accrual accounting information, such as the balance sheet, to the President’s Budget would be relatively easy. NFIP officials agreed that while the financial statements and various reports show the full liability of the program, including this information in the budget would consolidate it in one place. The President’s Budget (Appendix) already includes a balance sheet from FHA’s MMI Fund, along with information on credit subsidy reestimates in the Federal Credit Supplement of the budget as required by the Federal Credit Reform Act of 1990. According to FHA officials, the main benefit of reporting balance-sheet information in the budget is that the public, OMB, and Department of Housing and Urban Development personnel have more data that can be used to make decisions. FHA officials also noted that creating a crosswalk between the financial reports and the budget has been a challenge because there is not always a one-to-one relationship, which is due to different reporting elements and concepts underlying their measurement. Given the variation in fiscal exposures, we previously concluded that while accrual budgeting better recognizes long-term costs, a uniform, across-the-board approach to make fiscal exposures more apparent in the budget may not be appropriate. One of several factors that should be considered is the extent to which the magnitude of the exposure can be reasonably estimated. The complexity and uncertainty surrounding some exposures create significant cost estimation challenges, while other activities are easier to estimate. For example, OPM considers various factors that are fairly stable and easily known or can be reasonably estimated (such as changes in the mortality of federal employees, federal salaries, and interest rates) when calculating FEGLI’s liability for current and future life insurance coverage. Because of this, FEGLI officials noted that they did not face significant challenges in estimating the program’s fiscal exposure. In contrast, exposures related to natural disasters are especially hard to estimate. According to NFIP officials, the extreme variability of flood losses is the single biggest challenge in estimating the program’s exposure. Similarly, officials from the Disaster Relief Fund said it is not possible to forecast catastrophic disasters because each is different. According to FEMA officials, the agency has begun working with catastrophe modeling firms and others in developing better estimates of loss exposures. While several components of the models are still in development, FEMA officials believe they show promise to be useful tools in the future. Agencies developed models to make estimates of fiscal exposures for several of the activities we reviewed, although the agencies noted that generating reasonably reliable estimates is difficult for a variety of reasons, such as the sufficiency of data on potential losses and the nature of the risks insured by the government. For example, PBGC developed the Pension Insurance Modeling System to help the agency better understand and quantify its long-term risk and exposure to loss under different economic conditions and policy alternatives. Agency officials stated that obtaining current, complete, comprehensive, and reliable data on the company pension plans likely to present claims was one of the most significant challenges in these estimates. FHA also uses economic assumptions and historical data to estimate and reestimate the net lifetime costs of the mortgages it insures. Agency officials noted that it is difficult to produce these estimates because risk can vary based on a variety of factors that are uncertain, volatile, or sensitive, such as the economy and housing market. In recognition of this difficulty, the Federal Credit Reform Act of 1990 provides permanent and indefinite budget authority for upward reestimates, so that FHA can receive additional funds when needed if reestimates increase subsidy costs. We previously recommended to Congress in 2007, and reiterated in 2013, that it consider requiring increased reporting of accrual-based cost information where appropriate alongside cash-based budget numbers for both existing and proposed activities—where accrual-based cost information includes significant future cash resource requirements not yet reflected in the primarily cash-based budget. From 2009 through 2014, several bills were introduced to budget for certain activities on an accrual basis, but none were signed into law. The Bipartisan Budget Act of 2018 created the Joint Select Committee on Budget and Appropriation Process Reform, but no bills were signed into law as a result. The committee was terminated by statute by December 2018. We continue to support this recommendation to improve budget recognition of these fiscal exposures, because, as shown, challenges remain in identifying and measuring fiscal exposures. The government undertakes a wide range of activities that create fiscal exposures by obligating the government to future spending or creating an expectation for such spending. The federal budget both allocates and controls resources, but does not provide complete information about some significant fiscal exposures. Failure to understand and address these exposures can have significant consequences. These fiscal exposures will require future federal spending and will absorb resources available for other activities. Not capturing the long-term costs of current decisions limits Congress’s ability to control federal fiscal exposures at the time decisions are made. Presenting accrual information alongside cash-based budget numbers, particularly in areas where it would enhance up-front control of budgetary resources, would be useful to policymakers when debating current activities and considering new legislation. We provided a draft of this report to OMB and Treasury. The agencies had no comments on the draft report but provided technical comments that we incorporated as appropriate. We sent relevant portions of the draft to the following agencies: Department of Agriculture’s RMA, Department of Homeland Security’s FEMA, Department of Housing and Urban Development’s FHA, Department of Transportation’s Federal Aviation Administration, FDIC, NCUA, OPM, and PBGC. All the agencies (except the Department of Transportation, Department of Housing and Urban Development, and Department of Agriculture) provided technical comments that we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Treasury, and the Director of OMB. This report will also be available at no charge on our website at http://www.gao.gov. Should you or your staff have questions concerning this report, please contact me at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VII. In this report, we (1) identify federal insurance and other activities that transfer risk or losses to the federal government and, where possible, identify cost- and exposure-related information on these activities; and (2) use selected activities to illustrate some of the challenges that we identified in past reports with measuring and reporting fiscal exposures in budget documents. For this objective, we updated our 2005 catalog of federal insurance activities and used additional sources of information on federal activities to obtain cost- and exposure-related information and identify additional activities. To compile the 2005 catalog, we developed the following criteria for identifying federal insurance activities: (1) The federal government must accept the risk of financial loss in providing protection against specific types of losses, events, or conditions whose timing, magnitude, or duration, are uncertain or unknown; and (2) by accepting this insurance risk, the federal government must be obligated to pay compensation or provide benefits if the losses, events, or conditions occur. In addition, we verified that the activities we cataloged as federal insurance also were recognized lines of insurance in the private sector. We applied the criteria to the Appendix of the Budget of the United States Government (President’s Budget) for Fiscal Year 2005 to identify budget accounts that funded federal insurance activities per our criteria. In this report, we first developed our own criteria for activities that transfer risk or losses to the federal government using definitions of federal insurance from the Federal Accounting Standards Advisory Board’s (FASAB) Statement of Federal Financial Accounting Standards (SFFAS) 5: Accounting for Liabilities of the Federal Government and Statement of Federal Financial Accounting Standards SFFAS 51: Insurance Programs. Activities were assessed against the following two criteria: (1) a risk of financial loss or actual financial loss to a third party exists that stems from an adverse event; and (2) through the activity the federal government accepts some or all of the risk of financial loss or actual financial loss from the adverse event by indemnifying, guaranteeing, or providing benefits to the affected entity or beneficiary. We also reviewed our new criteria internally with input from GAO experts, including accountants, actuaries, and budget law experts. We then used the following sources to identify federal activities that met our criteria: (1) 2017 Financial Report of the United States Government (Financial Report) and underlying account-level data from the Government-wide Treasury Account Symbol Adjusted Trial Balance System (GTAS) of the Department of the Treasury (Treasury), (2) the Office of Management and Budget’s (OMB) President’s Budget; (3) OMB’s Catalog of Federal Domestic Assistance (CFDA) administered by the General Services Administration; and (4) the Code of Laws of the United States (U.S. Code). The resulting catalog is based solely on the criteria we developed for this report and the sources and methodology we used. Other criteria, sources, or methodologies might yield a list that differs from ours in both number and composition of activities. Because we use different criteria, sources, and methodologies, our results are not directly comparable to results in our 2005 catalog. First, we identified certain categories of federal activities that met our criteria using the note disclosures of the 2017 Financial Report: We identified the following federal insurance programs by analyzing the note disclosure on federal insurance and guarantee liabilities as well as our internal audit documents on that note disclosure for fiscal years 2014–2017: Federal Crop Insurance Program, National Flood Insurance Program (NFIP), Pension Benefit Guaranty Corporation’s (PBGC) single-employer and multiemployer pension insurance programs, Federal Deposit Insurance Corporation’s Deposit Insurance Fund, and National Credit Union Administration’s Share Insurance Fund. To find information on the programs, the role of the government in the administration of the programs, and cost- and exposure-related information on the programs, we analyzed GAO reports, individual agency annual, financial, or other reports, and reports from the Congressional Budget Office. We also identified the following categories of activities that met our criteria: federal loan guarantee programs, senior preferred stock purchase agreements with two government-sponsored enterprises— Fannie Mae and Freddie Mac; federal employee and veterans benefits excluding veterans’ burial and education benefits; and social insurance. We used GTAS to identify Treasury accounts for the federal loan guarantee and the federal employee and veterans benefit categories. We generally presented liabilities for these activities as available in the Financial Report and GTAS for fiscal year 2017. Second, we conducted searches of key words in the names of budget accounts reported in the Appendix of the President’s Budget using OMB’s MAX system. We analyzed the results and identified budget accounts that funded additional federal activities that met our criteria and reported budget obligation data for those accounts for fiscal year 2017. We also analyzed more than 2,200 federal activities in CFDA as of September 30, 2017. To do this, at least two analysts verified that a budget account or a CFDA program met or did not meet our criteria (with review from an additional analyst, if needed). We included a budget account or CFDA program in our catalog if all analysts reviewing the program agreed the account or program met our criteria. The additional federal activities we found through the President’s Budget and CFDA can be broadly categorized as providing compensation for property or financial losses— including losses resulting from adverse environmental or manmade events—or providing life, health, or disability benefits to nonfederal employees. We also crosschecked this list of federal activities with those we identified in our 2005 report. Lastly, we conducted a search of key words in the table of contents of the U.S. Code and found additional activities that provided compensation for property damage or financial loss and thus transferred risk or losses to the federal government. We conducted this search to identify potential federal activities that currently only exist in law but met our criteria. According to officials from OMB, activities may not appear in the President’s Budget for different reasons, including if the activities ended or expired or if they have no expected expenditures or proposed appropriations. In reviewing the U.S. Code, two analysts verified whether a search result represented a new federal activity that met our criteria. In addition, a senior attorney from our Office of General Counsel verified that the additional activities were accurately stated based on the language of the U.S. Code. We shared our catalog with and obtained expert opinion from officials from Treasury, OMB, the General Services Administration, the Congressional Budget Office, and FASAB, as well as from key GAO staff with relevant expertise (including accountants, actuaries, and experts on budget appropriation and other federal activities). We added an additional federal activity to our catalog that an outside expert brought to our attention and that we had not identified through the methodologies described above. To illustrate some of the challenges in measuring and reporting fiscal exposures that we identified in past reports, we reviewed prior reports on fiscal exposures, the federal budget, and accrual budgeting to identify key challenges relevant to insurance and other activities. We then selected six activities from those we identified for this report that illustrate these key challenges. We selected the (1) Disaster Relief Fund, (2) Federal Aviation Administration’s Commercial Space Launch Insurance Program, (3) Federal Employees’ Group Life Insurance program, (4) Federal Housing Administration’s Mutual Mortgage Insurance Fund, (5) NFIP, and (6) PBGC pension insurance programs. We reviewed the Financial Report and the President’s Budget for information and financial measures on these activities, such as receipts, outlays, and net position for NFIP and PBGC. We also reviewed the appropriation and supplemental appropriation laws for the Disaster Relief Fund from fiscal years 2005 through 2018. We analyzed the agencies’ financial and budget documents and conducted interviews with agency officials. We conducted this performance audit from October 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Federal Crop Insurance program is administered by the Risk Management Agency (RMA) of the Department of Agriculture. It helps farmers manage the risks inherent in farming by allowing them to insure against losses caused by poor crop yields, declines in prices, or both. Crop insurance premiums are subsidized in part to achieve high participation and coverage levels, which may reduce or eliminate the need for potentially costly disaster assistance payments from congressionally authorized ad hoc disaster programs. RMA partners with private insurers that sell and service policies. The program insures farmers against losses on more than 100 crops, which include corn, soybeans, wheat, cotton, and grain sorghum, as well as nursery crops and certain fruits and vegetables. In crop year 2018, RMA estimated it sold 1.1 million policies for a total of about $109.1 billion in insurance protection. The federal government subsidizes crop insurance policies by charging participating farmers less than the full amount of the policy premium. Congress sets the programs’ premium subsidy rates—the percentage of the premium paid by the government. RMA subsidized approximately 63 percent of total premiums in crop years 2017 and 2018 (or $6.36 billion and $6.27 billion, respectively), while farmers paid the remaining 37 percent. The federal government also reimburses participating private- sector insurance companies for administrative and operating expenses. The reimbursements are based on a percentage of crop insurance premiums and are intended to cover the companies’ expenses to sell and service policies, such as employee salaries; fees paid to insurance adjusters to verify claims; and sales commissions and other compensation (profit sharing) paid to the insurance agents who sell the crop insurance to farmers. The federal government is also the primary reinsurer for the private insurance companies that take on the risk of covering losses to insured farmers, allowing private insurers and the federal government to share in the risk of loss and opportunity for gain associated with the policies. The insurance companies retain part of the premiums and associated risk, and RMA holds the remaining premiums and risk. In addition, each company cedes to RMA a percentage of its underwriting gains or losses. The Federal Crop Insurance program is funded through mandatory spending authority, so that RMA receives a permanent indefinite appropriation each fiscal year for premium subsidy and other expenses and returns unobligated balances to the U.S. Treasury at the end of the fiscal year. According to RMA, the net cost of operations for the program was $5.5 billion and $6.8 billion for fiscal years 2018 and 2017, respectively (see table 8). Lastly, in April 2018, the Congressional Budget Office projected that federal crop insurance would cost the federal government an average of about $7.9 billion per year in 2018–2028. In 2017, we recommended that RMA and Congress consider improving the calculations related to the payments to and risk-sharing agreements with participating insurance companies to reduce year-to-year fluctuations in the subsidy costs of the program. In 2015, we also reported on the need for RMA to obtain more information on program costs and improve its premium setting calculations. We recommended that Congress consider reducing premium subsidies for the highest-income participants to reduce the cost of the program. As of December 31, 2018, these recommendations remained unaddressed. Administered by the Federal Emergency Management Agency (FEMA) of the Department of Homeland Security, the National Flood Insurance Program (NFIP) makes federally backed flood insurance available to residential property owners and businesses. By design, NFIP does not operate for profit. Instead, the program must meet a public policy goal—to provide flood insurance in flood-prone areas to property owners who otherwise would not be able to obtain it. Under NFIP, the federal government assumes the liability for the insurance coverage and sets rates and coverage limitations, while the private insurance industry sells the policies and administers the claims. As of September 30, 2018, NFIP provided about $1.3 trillion of insurance coverage on 5.1 million policies. NFIP generally is expected to cover its claim payments and operating expenses with the premiums it collects, but it has had to use its authority to borrow funds from the U.S. Treasury to cover large shortfalls. Since 2004, Congress increased NFIP’s initial borrowing limit from $1.5 billion to $30.4 billion, which was passed into law in 2013. Until 2004, NFIP was able to cover most claims with premiums it collected and occasional loans from the U.S. Treasury that it repaid. Cumulative debt increased substantially from 2005 to 2016 due to extraordinary catastrophic loss years resulting primarily from Hurricane Katrina and Superstorm Sandy. By September 2017, NFIP exhausted its borrowing authority following hurricane season, which prompted Congress to grant $16 billion in debt cancellation. NFIP then borrowed $6.1 billion to cover incurred and anticipated expenses for the 2018 hurricane season. As of September 30, 2018, NFIP had $20.5 billion of outstanding debt with the U.S. Treasury. According to FEMA, as currently designed, the program likely will not be able to repay this debt. According to FEMA, from October 1, 2017 through September 30, 2018, NFIP’s total expenses were more than $12 billion, which was more than twice its total revenue of $5.6 billion. In that time, NFIP collected $3.51 billion in premium revenues and $1.04 billion in reinsurance collections, but paid $9.21 billion in claims through the National Flood Insurance Fund. For fiscal year 2018, expenses also exceeded revenues by about $6.64 billion for the National Flood Insurance Reserve Fund. FEMA has produced estimates of its ability to pay claims and of annual maximum probable losses. FEMA calculated NFIP’s capacity to pay claims, which includes almost $10 billion in remaining borrowing authority from Treasury, at $15.82 billion as of September 30, 2018 (see table 9). Although FEMA entered into a reinsurance contract in 2018 for $1.5 billion, it projects it will not be able to cover potential future fiscal exposure from a single, low-probability, super-catastrophic event, which it estimates could cost as much as $40 billion in claims. In 2017, we again reported that NFIP premiums do not reflect the full risk of loss, which increases the federal fiscal exposure created by the program, obscures that exposure from Congress and taxpayers, contributes to policyholder misperception of flood risk (they may not fully understand the risk of flooding), and discourages private insurers from selling flood insurance (they cannot compete on rates). We concluded that eliminating rate discounts by requiring all rates to reflect the full risk of loss would help address these problems, but also would make policies less affordable and could reduce consumer participation. We stated that the decreases in affordability could be offset by other actions such as providing means-based assistance. We recommended that Congress consider comprehensive reform to improve NFIP's solvency and enhance the nation's resilience to flood risk. As of December 31, 2018, Congress still was considering various reforms as it worked to reauthorize the program. The Pension Benefit Guaranty Corporation (PBGC) is a wholly-owned government corporation established to insure the pension benefits of participants in and beneficiaries of private-sector defined benefit plans. The corporation operates a single-employer program and a multiemployer program. The single-employer program covers defined benefit pension plans that generally are sponsored by one employer. When an underfunded single-employer plan terminates, PBGC becomes the trustee and administers the plan. As of September 30, 2018, the single- employer program insured about 26 million people in approximately 23,400 plans and approximately 861,000 people were receiving benefits payments from PBGC. The multiemployer program insures plans arranged through collective bargaining between labor unions and employers, with two or more employers participating in a plan. PBGC provides financial assistance to multiemployer plans when they become insolvent. According to PBGC, as of September 30, 2018, the multiemployer program insured about 10.6 million people in approximately 1,400 plans and about 62,300 people were receiving benefits payments from plans receiving financial assistance from PBGC. Premium rates are set in law by Congress and plan sponsors or plans pay per-participant flat premiums under both programs. In addition, under the single-employer program, a plan sponsor or plan pays a variable-rate premium based on its plan underfunding. PBGC receives no funds from general tax revenue and assets from one program cannot be used to support the other, so both programs must pay claims primarily from nonfederal sources. The single-employer program had positive cash flow during fiscal year 2018 and both programs have been able to maintain enough assets to pay guaranteed benefits and financial assistance to date. But historically, PBGC’s statutory premium structure has not reflected significant risks PBGC insures against—for example, the risk that a single-employer plan sponsor becomes bankrupt, forcing the termination of an underfunded plan, or the risk that a multiemployer plan’s financial condition deteriorates, causing it to become insolvent—imposing claims on PBGC programs. As shown in table 10, PBGC’s multiemployer program had a negative net position (that is, total liabilities exceeded total assets) at the end of fiscal year 2018. The single-employer program reached a positive net position (for the first time since 2001) by the end of fiscal year 2018. PBGC projects a positive net position in 10 years for the single-employer program, but a negative net position for the multiemployer program (negative $68.9 billion by the end of fiscal year 2027), although there is inherent uncertainty around such a projection. PBGC’s forecasts for the following decade and beyond based on current economic conditions project a very high likelihood of insolvency for the multiemployer program before the end of fiscal year 2025 if there are no changes in law. In 2013, PBGC officials told us that once the multiemployer fund’s cash balance was depleted, the agency would have to rely solely on annual insurance premium receipts to pay financial assistance to plans. The precise effect that the insolvency of the multiemployer insurance fund would have on retirees receiving the PBGC guaranteed benefit would depend on a number of factors—primarily the number of guaranteed benefit recipients and PBGC’s annual premium income at that time. However, the impact likely would be severe. In 2012, we recommended that Congress consider redesigning PBGC’s premium structure to more fully reflect the risks posed by plans and sponsors to the agency and improve PBGC’s access to additional information needed to assess risk and assist in setting premiums. In 2013, we also recommended that Congress consider comprehensive and balanced structural reforms to reinforce and stabilize the multiemployer program. As of December 31, 2018, Congress had yet to authorize a redesign of PBGC's premium structure. However, in December 2014, Congress enacted the Multiemployer Pension Reform Act of 2014, which substantially established in law certain key structural reforms to the multiemployer system, including allowing severely distressed multiemployer plans to reduce accrued pension benefits; expanding PBGC’s ability to assist financially distressed plans; and raising multiemployer insurance premiums to provide PBGC with additional resources. The Federal Deposit Insurance Corporation (FDIC) insures the deposits of commercial banks and savings associations up to the statutory limit of $250,000. According to FDIC, as of September 30, 2018, there were 5,486 insured depository institutions with total insured deposits of $7.4 trillion. FDIC administers the federal deposit insurance program through its management of the Deposit Insurance Fund (DIF), which has two primary purposes: (1) to insure the deposits and protect the depositors of insured banks and (2) to resolve failed banks. FDIC manages the DIF by determining the size of the fund and of the DIF reserve ratio (the ratio of the fund balance to estimated insured deposits). The DIF is funded mainly through quarterly risk-based assessments on insured depository institutions, and it also earns interest income on its securities. The DIF is reduced by the amount of losses and expenses associated with failed banks and by FDIC operating expenses. The financial strength of the DIF can be gauged by comparing the fund’s actual reserve ratio to the minimum reserve ratio, and by measuring its progress to the designated, or desired, reserve ratio. Section 334 of the Dodd-Frank Wall Street Reform and Consumer Protection Act increased the minimum reserve ratio from 1.15 percent to 1.35 percent and required that the reserve ratio reach that level by September 30, 2020. To meet these requirements, large banks paid temporary surcharges from the third quarter of 2016 through the third quarter of 2018. In addition, under the long-term DIF management plan, the FDIC Board of Directors set the designated reserve ratio at 2.0 percent, with the goal of helping FDIC maintain a stable insurance assessment rate and sustain a positive DIF balance even during a serious economic downturn. In November 2018, FDIC announced that the DIF reserve ratio had reached 1.36 percent (as of September 30, 2018), exceeding the statutorily required minimum reserve ratio of 1.35 percent ahead of the statutory deadline (September 30, 2020). Obligations of FDIC are backed by the full faith and credit of the U.S. government. In addition, FDIC is authorized to borrow up to $100 billion from the U.S. Treasury and issue and sell up to $100 billion in obligations to the Federal Financing Bank (see table 11). A statutory formula, known as the maximum obligation limitation, limits the amount of obligations the DIF can incur to the sum of its cash, 90 percent of the fair market value of other assets, and the amount authorized to be borrowed from the U.S. Treasury. The maximum obligation limitation for the DIF was $191.5 billion as of December 31, 2017. FDIC did not use its authority to borrow funds from the U.S. Treasury when the DIF was depleted and fell to negative $20.9 billion, its lowest point in history, as a result of the 2007–2009 financial crisis. Instead, FDIC first replenished the DIF through increased assessments and a one- time special assessment. These actions were taken pursuant to a restoration plan established to replenish the DIF and raise the reserve ratio to its designated minimum within the time limits prescribed by the Federal Deposit Insurance Act. Finally, FDIC was able to improve the liquidity of the DIF by requiring the banking industry to prepay its quarterly risk-based assessments for the fourth quarter of 2009 and for the next 3 years. The National Credit Union Administration (NCUA) administers the National Credit Union Share Insurance Fund and provides up to $250,000 of insurance per depositor. According to NCUA, by the end of calendar year 2017, the Share Insurance Fund insured the deposits of more than 111 million members in 5,573 credit unions with $1.38 trillion in assets, and the fund insured $1.1 trillion of member shares, or dollars deposited. The Share Insurance Fund is primarily funded by contributions of 1 percent of the insured shares or deposits from each member credit union. Other sources of income include premiums, when assessed as explained below, and investment income. The financial performance of the Share Insurance Fund can be measured by comparing the equity ratio to the normal operating level (or desired equity ratio). The equity ratio is the total of credit unions’ contributions to the fund, less any gain or loss on investments, plus accumulated retained earnings, divided by total insured shares. By law, the equity ratio of the Share Insurance Fund cannot decline below 1.20 percent and may not exceed 1.50 percent. If NCUA expects the equity ratio to fall below this threshold, it must establish and implement a restoration plan to rebuild the equity ratio, which must include a premium assessment to each insured credit union. The reported equity ratio at the end of 2017 was 1.46 percent, which is above the normal operating level, set at 1.39 percent as of 2017. According to NCUA, a normal operating level of 1.39 percent was set with the goal of ensuring that the Share Insurance Fund could withstand a moderate recession without the equity ratio declining below 1.20 percent over a 5- year period. The Share Insurance Fund is backed by the full faith and credit of the U.S. government and, according to NCUA, has $6.0 billion in borrowing authority from the U.S. Treasury, all of which was available as of December 31, 2017. The fund also has the ability to borrow from the NCUA’s Central Liquidity Facility up to the amount of the liquidity facility’s unused borrowing authority, which was $6.6 billion as of December 31, 2017. As of December 31, 2017, the Share Insurance Fund had $12.6 billion in total available borrowing capacity, which is the combination of the borrowing authority from the U.S. Treasury and the liquidity facility (see table 12). The recent equity ratio contrasts with low points reached during and after the 2007–2009 financial crisis. NCUA had to take a number of steps to stabilize credit unions, stemming primarily from the failure of five large corporate credit unions. NCUA established the Temporary Corporate Credit Union Stabilization Fund, which replaced the Share Insurance Fund as the primary source to absorb the corporates’ losses. Congress increased NCUA’s borrowing authority with the U.S. Treasury up to $6 billion through a revolving loan fund to be shared between the Stabilization Fund and Share Insurance Fund. The Stabilization Fund borrowed and repaid a total of $11.2 billion from the U.S. Treasury from its inception in 2009 through its closure in October 1, 2017. The highest amount of total borrowing outstanding was $5.1 billion in October 2012. However, the Share Insurance Fund’s equity ratio fell below 1.20 percent in both 2009 and 2010, and two premiums totaling $1.7 billion were necessary to restore the equity ratio. NCUA stated that without the premiums the equity ratio would have fallen to 1.07 percent. Federal loan guarantees are any guarantees, insurance, or other pledges with respect to the payment of all or a part of the principal or interest on any debt obligation of a nonfederal borrower to a nonfederal lender. Thus, the federal guarantee transfers some or all of the risks of borrower default from private lenders to the federal government. Table 13 lists the 33 federal guaranteed loan activities that presented liabilities to the federal government as of September 30, 2017. Table 14 lists 13 large federal employee and veterans benefit activities— such as pension, health, life, and disability benefits—that transfer risk or losses to the federal government. Each of the activities listed represented $10 billion or more in benefit liabilities payable for the fiscal year ending September 30, 2017. Combined, the activities accounted for 99 percent of the total federal employee and veterans benefit liabilities of $7.7 trillion. Tables 15, 16, and 17 list a total of 95 federal activities that met our criteria of transferring risk or losses from adverse events from third parties to the federal government and that we found in the Budget of the United States Government (President’s Budget), the Catalog of Federal Domestic Assistance (CFDA), or the Code of Laws of the United States (U.S. Code). These activities can be broadly categorized into activities that, at least in part, provide compensation for property or financial losses—including losses resulting from adverse environmental or manmade events—and activities that offer life, health, or disability benefits to nonfederal employees. Table 15 has information on 39 budget accounts from the President’s Budget, generally organized by amount of obligations for fiscal year 2017. While budget obligations create a legal liability for the federal government, they may not necessarily reflect an activity’s fiscal exposure if, for example, the activity has dedicated payment streams. Table 16 has information on an additional 51 activities found through our sources that met our criteria. Table 17 has information on five activities authorized in law that had not triggered losses to the federal government as of December 31, 2018. With the exception of the Terrorism Risk Insurance Program, we identified these programs through an analysis of the U.S. Code, since the programs have not had liabilities or appropriations and could not be found in the Financial Report or the President’s Budget. We were able to find some financial and budgetary information on the Terrorism Risk Insurance Program because administrative expenses and potential projected payments under the program are identified in the President’s Budget on an annual basis. Federal social insurance programs are Social Security, Medicare (Parts A, B, and D), Railroad Retirement, and Black Lung. These programs provide eligible individuals with benefits, such as health insurance, disability, and retirement benefits, thus transferring risk to the federal government. Fiscal exposures from the four programs are discussed annually in the Statement of Social Insurance (SOSI) in the Financial Reports of the United States Government. Specifically, the SOSI details the present value of the estimated future revenues and expenditures for scheduled benefits over the next 75 years. The amounts in the SOSI and presented below are not considered liabilities in an accounting context. Future benefit payments will be recognized in the Financial Report as expenses and liabilities as they are incurred based on the continuation of the social insurance programs' provisions contained in current law. While future social insurance benefit payments that are not due and payable are not treated explicitly as legal liabilities to the federal government, the SOSI’s forward-looking projections are intended to help citizens understand the long-term sustainability of these programs and the fiscal exposures they present. The social insurance programs are mainly funded by taxes and premiums. Contributions and dedicated taxes consist of: payroll taxes from employers, employees, and self-employed persons; revenue from federal income taxation of Old-Age Survivors and Disability Insurance (OASDI) and railroad retirement benefits; excise tax on the domestic sale of coal; premiums from, and state transfers on behalf of, participants in Medicare; and reimbursements from the General Fund to the OASDI and Medicare Trust Funds. The social insurance trust funds account for all related program income and expenses, and have automatic funding authority to pay future benefits to the extent that funds are available. Taxes, premiums, and other income are credited to the funds, while benefit payments and program administrative costs are paid from the funds. However, as of January 1, 2017, based on information from the SOSI , the present value of federal expenditures for social insurance programs over 75 years was projected to exceed program revenues by about $19.0 trillion (see table 18). This represents about 1.5 percent of the present value of the gross domestic product over 75 years. To illustrate the sustainability of current benefits, the Social Security and Medicare Part A SOSI projections assume that scheduled social insurance benefit payments would continue after related trust funds are projected to be depleted, contrary to current law. The projections for Medicare Parts B and D for fiscal year 2017 include $30 trillion in transfers of general revenues that, under current law, are used to finance the remainder of the expenditures in excess of revenues. We have reported that there are significant uncertainties related to the achievement of projected reductions in Medicare cost growth assumed in the SOSI projections that have prevented us from expressing an opinion on the sustainability of the financial statements in the Financial Report. We previously reported on the fiscal problems presented by these programs, in particular Social Security and Medicare. The Social Security and Medicare programs are projected to face financial challenges. In June 2018, we noted that fiscal spending increases in 2017 were driven by Social Security, Medicare, Medicaid, and interest on debt held by the public. The spending increases were largely a result of the aging of the population and increasing health care costs rather than legislative changes to these programs. Spending on Social Security and these health programs is expected to continue to increase because of long- standing demographic and economic trends. The 2017 Financial Report of the United States Government, Congressional Budget Office, and our projections all show that, absent policy changes, the federal government’s fiscal path is unsustainable and that the ratio of debt to the gross domestic product would surpass its historical high of 106 percent within 14–22 years. All the projections also note that the longer action is delayed, the greater and more drastic changes will have to be. Alicia Puente Cackley, (202) 512-8678 or cackleya@gao.gov. In addition to the contact name above, Patrick Ward (Assistant Director), Silvia Arbelaez-Ellis (Analyst in Charge), Katherine Carter, Robert F. Dacey (Chief Accountant), Rachel DeMarcus (Assistant General Counsel), Jill Lacey, Janice Latimer (Assistant Director, Strategic Issues), Scott McNulty, Marc W. Molino, Angela Pun, Barbara Roesmann, Jessica Sandler, Dawn Simpson (Director, Financial Management and Assurance), and Frank Todisco (Chief Actuary) made significant contributions to this report.", "summary": "The federal government conducts many activities that protect parties from the effects of adverse events—for instance, by providing flood insurance, guaranteeing mortgage loans, or making payments to beneficiaries of deceased military personnel. Identifying these activities and understanding the fiscal exposures they create can be a challenge, making it difficult for Congress to oversee them through the budget and appropriation processes. GAO was asked to update information on federal insurance activities it created in 2005 ( GAO-05-265R ) and identify opportunities for improving budgeting for such activities. This report (1) identifies and provides cost- and exposure-related information on federal activities that transfer risk or losses to the government, and (2) illustrates challenges GAO identified in past reports with measuring and reporting fiscal exposures in budget documents. GAO primarily reviewed government-wide financial and budget data, the Catalog of Federal Domestic Assistance, and the U.S. Code. GAO also drew on previous work, conducted interviews with the Office of Management and Budget, Department of the Treasury, and other agencies, and reviewed agency financial and budget documents. Through analysis of sources containing government-wide information on federal activities, GAO identified 148 federal insurance and other activities that transfer risk or losses from adverse events to the government (see fig.). Unlike private insurance, the activities do not necessarily have a contract or charge premiums or fees in exchange for assuming risk. Even when premiums or fees exist they may not cover all costs, as federal expenditures can be driven by policy goals or agency missions rather than the aim of fiscal solvency. GAO generally was able to provide financial or budget information for the activities. Source: GAO . | GAO-19-353 Note: GAO's results are based solely on the criteria GAO developed for this report and the sources and methodology it used. Other criteria, sources, or methodologies might yield lists that differ from GAO's in number and composition of activities. a GAO identified 13 Treasury accounts that accounted for 99 percent of all federal employee and veterans benefits liabilities to the federal government as of September 30, 2017. These include accounts that fund retirement benefits, disability insurance, health insurance, and life insurance programs for civilian and military employees. The government's primarily cash-based budget generally does not record the full cost of commitments incurred until corresponding payments are made in the future. Therefore, the budget may not accurately reflect federal costs or the likely claim on federal resources for such activities. For some claims, such as pension and life insurance, the federal commitment occurs years before payments are reflected in the budget. Additionally, payments the government may be expected to make based on policies or past practices (but is not legally required to make) may not be evident in the budget. For example, the Commercial Space Launch Insurance Program created a potential liability to the government of up to $3.1 billion per licensed space launch in 2017 but never has been included in the budget. GAO previously recommended ( GAO-08-206 , and reiterated in GAO-14-28 ) that Congress consider expanding the use of accrual-based information in the budget documents submitted to Congress. However, this recommendation has not been implemented. Accrual measurement would provide enhanced control over future spending by recognizing long-term costs when decisions are made. This analysis provides additional support for GAO's 2007 recommendation that Congress consider improving recognition of fiscal exposures in budget documents such as by expanding use of information on expected future spending arising from present-day commitments.", "document_type": "gao"}
{"report": "Title VII of the Civil Rights Act of 1964 and Section 501 of the Rehabilitation Act of 1973 mandate that all federal personnel decisions be made without discrimination on the basis of race, color, religion, sex, national origin, or disability and require that agencies establish a program of equal employment opportunity for all federal employees and applicants. EEOC has oversight responsibility for federal agencies’ compliance with EEOC regulations, which direct agencies to maintain a continuing affirmative program to promote equal opportunity and to identify and eliminate discriminatory practices and policies. In order to implement the programs described above, each federal agency is required to designate an EEO director. The EEO director’s responsibilities include, among others, providing for counseling of aggrieved individuals, providing for the receipt and processing of individual and class complaints of discrimination, and advising agency leadership regarding equal employment opportunity matters. EEOC calls for federal agencies to conduct a continuing campaign to eradicate every form of prejudice or discrimination from the agency’s personnel policies, practices, and working conditions. EEOC’s Management Directive 715 (MD-715) calls for agencies to take appropriate steps to ensure that all employment decisions are free from discrimination and provides policy guidance and standards for establishing and maintaining effective affirmative programs of equal employment opportunity. The directive also sets forth the standards by which EEOC will review the sufficiency of agencies’ Title VII and Rehabilitation Act programs, including periodic agency self-assessments and the removal of barriers to free and open workplace competition. MD- 715 guidance further requires agencies to report annually on the status of activities undertaken pursuant to their equal employment opportunity programs and activities. Federal agencies are required to submit an annual MD-715 report to EEOC on the status of their EEO programs. In addition to including employee demographic data, among other things, the MD-715 reports are to include an agency self-assessment checklist, plans to correct any program deficiencies, and a description of any barrier analysis conducted and any plans to eliminate identified barriers. As part of a model EEO program to prevent unlawful discrimination, federal agencies are to regularly evaluate their employment practices to identify barriers to EEO in the workplace, take measures to eliminate identified barriers, and report annually on these efforts to EEOC, according to MD-715. EEOC’s MD-715 defines a barrier as an agency policy, procedure, practice, or condition that limits, or tends to limit, employment opportunities for members of a particular gender, race, or ethnic background or for individuals on the basis of disability status. According to EEOC’s MD-715 instructions, many employment barriers are built into the organizational and operational structures of an agency and are embedded in the agency’s day-to-day procedures and practices. USAID’s Office of Civil Rights and Diversity (OCRD) administers programs intended to promote equal opportunity, foster diversity at all levels and occupations, and sustain an inclusive workforce. According to USAID, OCRD strives to maintain a model EEO program. As table 1 shows, OCRD consists of the Complaints and Resolution Division, the Reasonable Accommodations Division, the Diversity and Inclusion Division, and the Program Operations Division. OCRD collaborates with the Office of Human Capital and Talent Management (HCTM) to develop and implement recruitment strategies intended to support a diverse and well-qualified workforce; consults with agency officials such as the Executive Diversity Council; partners with USAID employee resource groups to extend outreach opportunities and develop strategies of inclusion within USAID; and addresses allegations of discrimination, harassment, or retaliation. According to a June 2019 testimony by USAID’s Chief Human Capital Officer, OCRD collaborates with HCTM on the following recruitment programs intended to increase diversity: Donald Payne International Development Fellowship. Launched in 2012, the Donald Payne International Development Fellowship targets underrepresented groups in USAID’s Foreign Service. According to USAID officials, the purpose of the Payne Fellowship is to enhance diversity in the Foreign Service through outreach and strategic efforts focused on minority serving institutions. USAID provides support for selected candidates for 2 years of graduate school as well as an internship on Capitol Hill and another at a USAID mission overseas. On completion of the graduate program and internships, the selected candidate is appointed as a Foreign Service officer with a 5-year service agreement. According to USAID, each year the Payne Fellowship supports 10 fellows entering USAID’s Foreign Service. Development Diplomats in Residence. Established in 2016, the Development Diplomats in Residence program aims to educate, recruit, and channel talent to USAID by placing senior USAID officials at universities. These officials provide guidance and advice on careers, internships, and fellowships to students, professionals, and faculty members at minority-serving institutions. Two USAID career Senior Foreign Service officers serve in this role at California State University, Long Beach, and at Morehouse College, respectively. Pathways Internship Program. The Pathways Internship Program provides targeted diversity recruitment, salaries, and payments for Pathways Interns, according to the USAID Chief Human Capital Officer’s June 2019 testimony. The testimony states that the overall racial or ethnic minority representation rate in fiscal year 2018 for the Pathways Internship Program was 69 percent and that Hispanics, at 31 percent, represented the largest minority demographic. USAID officials said that the agency views its internship programs as a succession-planning tool designed to convert as many internships as possible into full-time positions. According to USAID, the agency had no Pathways Interns in 2019, as a result of funding limitations, but as of April 2020 was planning 21 internships for 2020. USAID provides training as well as a formal mentoring program intended to support diversity and inclusion, according to USAID officials. OCRD is responsible for providing mandatory agency-wide training on diversity awareness and equal opportunity. USAID officials stated that the agency has mandatory and nonmandatory training on diversity and inclusion issues. For example, USAID provides online mandatory training classes on the No FEAR Act and sexual harassment. According to USAID data, 326 people took versions of these courses in 2019. USAID also offers nonmandatory in-person classes such as EEO counselor training and unconscious bias training. In 2019, 17 people took EEO counselor training, and 36 people took USAID’s in-person unconscious bias training. Additionally, USAID officials said that external partners of USAID have developed training related to diversity and inclusion, to which OCRD refers employees on request. According to USAID, the agency’s mentoring programs build on informal mentoring efforts and support strategic human capital initiatives for recruitment and retention, employee development, succession planning, and diversity. USAID officials stated that the mentoring program includes a facilitated process for matching mentors and mentees, formal mentoring training, an established tracking system, and goals used to measure success. According to the officials, the mentoring program is open to all employees. USAID reported to Congress on its workforce categories in 2018. USAID defines its core workforce as those who have an employer– employee relationship with the agency. This includes the following employment categories: Civil Service employees. USAID’s Civil Service employees are U.S. U.S. Personal Services Contractors U.S. personal services contractors represent a significant and growing proportion of USAID’s workforce whose demographic composition is not included in USAID’s Management Directive 715 reports. As we reported in 2017, USAID uses personal services contractors for a broader range of functions than other agencies, as its regulations permit (see GAO-17-610). Those regulations provide that personal services contractors who are U.S. citizens may be delegated or assigned any authority, duty, or responsibility that direct-hire government employees might have, although they generally cannot supervise direct-hire government employees or sign obligating documents except when specifically designated as a contracting officer. Until recently, when looking to fill a vacancy through outside hiring or by promotions and reassignments, USAID bureaus and offices had to submit that action to USAID’s Hiring and Reassignment Review Board for review. The board’s guidelines exempted personal services contracts from review and approval. In April 2020, USAID officials told us that hiring decisions no longer required the board’s approval. From June 2016 to September 2018, U.S. personal services contractors were USAID’s fastest growing workforce category, increasing from 759 to 1,015 according to USAID’s staffing reports to Congress. During this period, USAID’s Civil and Foreign Service employees decreased from 3,548 to 3,002. contractors are non–direct hire U.S. citizens on contract for the specific services of those individuals. As we reported in 2017, USAID uses personal services contracts for a broad range of functions, such as program management, security analysis, and logistics. According to its staffing report to Congress, USAID had 1,015 U.S. personal services contractors at the end of fiscal year 2018. Foreign nationals. USAID’s foreign national employees are non–U.S. citizens who are locally employed at posts abroad. They may be direct hires or personal services contractors. USAID uses foreign nationals to manage mission operations and oversee development activities. According to its staffing report to Congress, USAID had 4,712 foreign national employees at the end of fiscal year 2018. While USAID collects demographic data on U.S. personal services contractors for its payroll processor, it does not analyze this information. USAID does not report these data, because USAID does not regard personal services contractors as U.S. government employees. USAID officials noted that current reporting requirements call only for demographics of direct-hire employees, which excludes a considerable portion of the agency’s workforce. Other categories of staff not directly employed by USAID, including institutional support contractors and staff detailed from other organizations and U.S. government agencies, also perform a wide range of services in support of the agency’s programs. According to its staffing report to Congress, USAID had 1,681 institutional support contractors at the end of fiscal year 2018. EEOC has determined that contractors are a vulnerable group because of confusion as to where such personnel should seek redress for EEO matters. However, according to OCRD officials, OCRD is responsible for EEO matters for both direct and non– direct hires, including contractors. Figure 1 shows the total number of staff in each of USAID’s workforce categories in fiscal year 2018. In fiscal year 2018, USAID had 2,964 full-time, permanent, career employees (i.e., direct-hire U.S.-citizens) in its Civil and Foreign Services, according to National Finance Center data. This number reflects an increase of more than 54 percent from fiscal year 2002. Figure 2 shows the numbers of full-time, permanent, career employees in USAID’s Civil and Foreign Services in fiscal years 2002 through 2018. USAID’s Civil Service made up 44 percent of the agency’s full-time, permanent, career workforce in fiscal year 2018. Civil Service employees are ranked in the GS classification system from GS-1 (lowest) to GS-15 (highest), followed by the executive rank. Civil Service promotions are filled through competitive procedures and noncompetitive career-ladder positions. To be eligible for a promotion, Civil Service candidates must meet minimum qualification standards such as fulfilling time-in-grade requirements and receiving sufficiently positive ratings on their most recent performance appraisals. For competitive promotion positions, USAID uses an automated system to evaluate and rate all eligible candidates and develop referral lists of employees eligible for the promotions. Officials interview all direct-hire USAID employees from the promotion referral lists and select employees for promotion on the basis of the announcement. Career-ladder positions are intended to prepare employees for successive, noncompetitive promotions up to the full performance of the positions. For career-ladder positions, USAID officials select employees for noncompetitive promotions and are responsible for developing individual learning and training plans, offering developmental work, and providing feedback regarding employees’ performance. Each year, USAID promotes varying numbers of Civil Service employees. Promotion generally becomes more competitive for higher ranks. For example, in fiscal year 2018, 45.3 percent of employees ranked GS-11 in fiscal year 2017 were promoted to GS-12, while 1.0 percent of employees ranked GS-15 in fiscal year 2017 were promoted to the executive rank. Table 2 shows the number and percentage of employees in each Civil Service rank as well as the rate of promotion from each GS level for promotions effective in fiscal year 2018. Foreign Service employees made up 56 percent of USAID’s full-time, permanent, career workforce in fiscal year 2018. Foreign Service officers enter at Class 4, 5, or 6, depending on their education and experience. Officers can be promoted from each level up to Class 1, after which they can apply for the executive rank. Foreign Service promotions are based on employee eligibility, a rank- ordered list prepared by a performance board, and the number of promotions authorized by USAID management. To be promoted to the next class, Foreign Service employees must meet eligibility requirements, such as time in their current class and overseas experience. Each year, performance boards evaluate the performance of eligible employees in Class 4 and higher, develop a rank-ordered list of employees recommended for promotion, and submit the list to HCTM. According to USAID policy, performance boards primarily consist of Foreign Service employees and, to the extent possible, include members of groups that are underrepresented in the service. The Chief Human Capital Officer, the Director of OCRD, and a representative of the American Foreign Service Association review the list before finalizing promotion decisions. USAID promotes varying numbers of its Foreign Service employees each year. Promotion generally becomes more competitive for higher ranks. For example, in fiscal year 2018, 33.2 percent of employees ranked Class 4 in fiscal year 2017 were promoted to Class 3, while 3.9 percent of employees ranked Class 1 in fiscal year 2017 were promoted to the executive rank. Table 3 shows the number and percentage of employees in each Foreign Service rank in fiscal year 2018 as well as the rate of promotion from each rank for promotions effective in that fiscal year. According to USAID’s Chief Human Capital Officer, USAID established the Hiring and Reassignment Review Board (HRRB) in July 2017 as a mechanism to allow USAID to prioritize positions during the government- wide hiring freeze and a subsequent period when all USAID external hires required approval from the Secretary of State. In fiscal years 2017 through 2019, the HRRB met regularly and was responsible for prioritizing U.S. direct-hire positions, monitoring attrition levels, and identifying gaps in national security and other key positions. According to June 2019 guidelines, the HRRB was required to review certain hiring and reassignment actions. Such actions included filling vacancies externally by hiring individuals from outside the agency, using operating expense funding, and filling vacancies internally by reassigning operating expense–funded Civil Service staff between the bureaus and independent offices. Hiring and reassignment actions exempted from HRRB review included, among others, hiring to compensate for attrition in certain defined high-risk mission-critical occupations, hiring into program- funded positions, Foreign Service limited appointments, personal services contracts, and institutional support contracts. According to USAID’s strategic workforce plan for fiscal years 2019 through 2021, USAID planned to have the HRRB, the Office of the Administrator, HCTM, and the Bureau for Management set broader staffing levels for the agency’s bureaus and independent offices beginning by the first quarter of fiscal year 2020. The workforce plan also states that a renamed HRRB would shift to serving as a strategic human capital governance board rather than performing position-by- position reviews. In April 2020, USAID officials told us that hiring decisions no longer required HRRB approval. From fiscal year 2002 to fiscal year 2018, the proportion of racial or ethnic minorities among USAID’s full-time, permanent, career employees increased from 33 percent to 37 percent, as figure 3 shows. This increase in the proportion of racial or ethnic minorities at USAID overall was driven by an increase in the proportion of racial or ethnic minorities in the Foreign Service. During this period, the proportion of racial or ethnic minorities in the Civil Service decreased slightly, from 49 to 48 percent and the proportion of racial or ethnic minorities in the Foreign Service increased from 18 to 27 percent. We compared the proportions of racial or ethnic minorities in USAID’s workforce with those in the federal workforce and relevant civilian labor force. Our comparison of USAID workforce data for fiscal year 2018 with federal workforce data for fiscal year 2017—the most recent available— found that the proportion of racial or ethnic minorities was 37 percent both at USAID and in the federal workforce. For more details, see appendix III. The proportion of racial or ethnic minorities at USAID increased from 33 percent in fiscal year 2002 to 37 percent in fiscal year 2018. In comparison, the proportion of racial or ethnic minorities in the federal workforce increased from 31 percent in fiscal year 2002 to 37 percent in fiscal year 2017. Our comparison of USAID workforce data from fiscal year 2018 with data for the relevant civilian labor force from 2006 through 2010 (the most recent available data) found larger proportions of racial or ethnic minorities at USAID than in the relevant civilian labor force for three occupational groups: (1) officials and managers, (2) professional workers, and (3) technical workers and technologists. For more details, see appendix III. Although the overall proportion of racial or ethnic minorities at USAID increased from fiscal year 2002 to fiscal year 2018, the direction of change for specific racial or ethnic minority groups varied—the proportions of Hispanics, Asians, and other racial or ethnic minorities rose, while the proportion of African Americans fell. As figure 3 shows, from fiscal year 2002 to fiscal year 2018, the proportion of Hispanics at USAID rose from 3 to 6 percent; Asians, from 4 to 7 percent; and other racial or ethnic minorities, from 1 to 2 percent of USAID employees. In contrast, during the same period the proportion of African Americans fell from 26 to 21 percent of the agency’s employees. Our analysis found that the overall decline in the proportion of African Americans at USAID reflected a substantial decline in the proportion of African Americans in USAID’s Civil Service. The proportion of African Americans in USAID’s Civil Service decreased from 42 percent in fiscal year 2002 to 32 percent in fiscal year 2018. The proportion of African Americans in USAID’s Foreign Service increased from 11 percent to 12 percent over the same period. In contrast to the proportion of African Americans, the proportions of Hispanics, Asians, and other racial or ethnic minorities at USAID increased in both the Civil and Foreign Services from fiscal year 2002 to fiscal year 2018. Our analysis of USAID data for fiscal year 2018 found that the proportions of racial or ethnic minority employees generally decreased as rank increased. As figure 4 shows, the proportions of racial or ethnic minorities in the Civil Service in fiscal year 2018 were progressively smaller in each rank above GS-12, except at the executive rank, where the proportion of racial or ethnic minorities was larger than in GS-15. Specifically, the proportions of racial or ethnic minorities decreased from 77 percent in GS-12 to 31 percent in GS-15. Our analysis similarly found that, in general, the proportions of racial or ethnic minorities in the Foreign Service in fiscal year 2018 were progressively smaller in all ranks above Class 6. In fiscal year 2002, the proportion of racial or ethnic minorities was also generally smaller at higher ranks in both the Civil and Foreign Services. From fiscal year 2002 to fiscal year 2018, the proportion of women at USAID increased from 51 to 54 percent, as figure 5 shows. Our analysis found that the overall increase in the proportion of women at USAID reflected a growth in the proportion of women in the Foreign Service. Specifically: The proportion of women in the Civil Service decreased from 66 percent in fiscal year 2002 to 61 percent in fiscal year 2018. The proportion of women in the Foreign Service increased from 38 percent in fiscal year 2002 to 49 percent in fiscal year 2018. We compared the proportion of women at USAID with the proportions of women in the federal workforce and relevant civilian labor force. Our comparison of USAID workforce data for fiscal year 2018 with federal government workforce data for 2017 found the following: The proportion of women at USAID in fiscal year 2018 (54 percent) was higher than the proportion of women in the federal workforce in fiscal year 2017 (43 percent). The proportion of women at USAID increased from 51 percent in fiscal year 2002 to 54 percent in fiscal year 2018. In contrast, the proportion of women in the federal workforce decreased slightly, from 44 percent in fiscal year 2002 to 43 percent in fiscal year 2017. Our comparison of USAID workforce data for fiscal year 2018 with data from the relevant civilian labor force for 2006 through 2010 (the most recent available data) found that the proportions of women were higher at USAID than in the relevant civilian labor force for two occupational groups—(1) officials and managers and (2) technical workers and technologists. However, the proportion of women was lower at USAID than in the relevant civilian labor force for professional workers. For more details, see appendix III. As figure 6 shows, our analysis of USAID data for fiscal year 2018 for the Civil Service found progressively smaller proportions of women in each rank above GS-11. The proportions of women ranged from 75 percent in GS-11 or lower ranks to 43 percent in the executive rank. Additionally, data for fiscal year 2018 for the Foreign Service show overall smaller proportions of women in the higher ranks. Specifically, women made up 55 percent of employees in Class 4 or lower ranks but 48 percent of Foreign Service executives. In fiscal year 2002, the proportion of women was also generally smaller in higher ranks in both the Civil and Foreign Services. Our analyses of USAID data on promotions in fiscal years 2002 through 2017 found lower promotion outcomes for racial or ethnic minorities than for whites in early to mid career. We found these differences when conducting descriptive analyses, which calculated simple average promotion rates, as well as adjusted analyses, which controlled for certain individual and occupational factors other than racial or ethnic minority status that could influence promotion. Promotion rates were generally lower for racial or ethnic minorities than for whites in both the Civil and Foreign Services, although the differences shown by our adjusted analyses were generally statistically significant only in the Civil Service. However, our analyses do not completely explain the reasons for differences in promotion outcomes, which may result from various unobservable factors. Thus, our analyses do not establish a causal relationship between demographic characteristics and promotion outcomes. Both our descriptive analysis and adjusted analysis of data for USAID’s Civil Service found that promotion rates were lower for racial or ethnic minorities than for whites in early to mid career, as table 4 shows. In addition, our adjusted analysis found that racial or ethnic minorities in USAID’s Civil Service had lower odds of promotion than their white counterparts. As table 4 shows, our descriptive analysis of the data for USAID’s Civil Service found that the average percentage of racial or ethnic minorities promoted from ranks GS-11 through GS-14 was lower than the average percentage of whites promoted from the same ranks. For example, our descriptive analysis found that in fiscal years 2002 through 2017, an average of 38.9 percent of racial or ethnic minorities were promoted from GS-11 to GS-12, compared with an average of 69.9 percent of whites. This difference of 31.0 percentage points indicates that the average rate of promotion from GS-11 to GS-12 was 44.4 percent lower for racial or ethnic minorities than for whites. In addition, our analysis of yearly promotion rates in the Civil Service for fiscal years 2013 through 2017 showed that the rate of promotion from GS-11 and higher ranks was greater for whites than for racial or ethnic minorities for every rank and year, except for promotions from GS-15 to the executive class in fiscal years 2013, 2014, and 2016. However, our descriptive analysis does not account for the variety of factors besides racial or ethnic minority status, such as occupation, that may affect promotion rates. Our adjusted analysis of the data for USAID’s Civil Service, controlling for certain factors other than racial or ethnic minority status that could influence promotion, found that racial or ethnic minorities had lower adjusted rates and lower odds of promotion from each rank from GS-11 through GS-14 than their white counterparts. Specifically, our adjusted analysis of USAID data on promotions in fiscal years 2002 through 2017 found the following: The average adjusted rate of promotion from GS-11 to GS-12 for racial or ethnic minorities was 46.8 percent, compared with an average of 55.8 percent for whites. This statistically significant difference indicates that the odds of promotion from GS-11 to GS-12 in the Civil Service were 41.4 percent lower for racial or ethnic minorities than for whites. Our estimates of the adjusted rates and odds of promotion from GS- 12 to GS-13 and from GS-13 to GS-14 were also statistically significantly lower for racial or ethnic minorities than for whites. There was no statistically significant difference in the odds of promotion from GS-14 to GS-15 or from GS-15 to the executive rank for racial or ethnic minorities relative to whites in the Civil Service. Compared with our descriptive analysis, our adjusted analysis found smaller percentage differences in promotion outcomes for racial or ethnic minorities relative to whites in the Civil Service. Figure 7 shows key results of our descriptive and adjusted analyses of USAID data for racial or ethnic minorities and whites in USAID’s Civil Service. As table 5 shows, our descriptive analysis of data for USAID’s Foreign Service found that the rate of promotion was generally lower for racial or ethnic minorities than for whites. In addition, our adjusted analysis found differences between the promotion rates for racial or ethnic minorities and those for whites. These differences were not statistically significant for promotions from Class 4 to Class 3, from Class 2 to Class 1, or from Class 1 to the executive rank. However, the differences between promotion rates for racial or ethnic minorities and whites were statistically significant for promotions from Class 3 to Class 2. As table 5 shows, our descriptive analysis of the data for USAID’s Foreign Service found that for Class 4 and higher ranks, a lower average percentage of racial or ethnic minorities than of whites was promoted from each rank except Class 1. For example, our descriptive analysis found that in fiscal years 2002 through 2017, an average of 31.5 percent of racial or ethnic minorities were promoted from Class 4 to Class 3, compared with an average of 33.7 percent of whites. This difference of 2.2 percentage points indicates that the average rate of promotion from Class 4 to Class 3 was 6.4 percent lower for racial or ethnic minorities than for whites. However, our descriptive analysis does not account for the variety of factors besides racial or ethnic minority status, such as occupation, that may affect promotion rates. Our adjusted analysis of the data for USAID’s Foreign Service, controlling for certain factors other than racial or ethnic minority status that could influence promotion, found that racial or ethnic minorities had lower adjusted rates and odds of promotion than their white counterparts but that these differences were generally not statistically significant. Specifically, our adjusted analysis of USAID data on promotions in fiscal years 2002 through 2017 found the following: On average, the adjusted rate of promotion from Class 3 to Class 2 for racial or ethnic minorities was 11.0 percent, compared with 13.1 percent for whites. This statistically significant difference indicates that the odds of promotion from Class 3 to Class 2 in the Foreign Service were 21.5 percent lower for racial or ethnic minorities than for whites. The adjusted rates and odds of promotion for racial or ethnic minorities relative to whites were also lower for promotion from Class 4 to Class 3 and from Class 2 to Class 1 and were higher for promotion from Class 1 to the executive rank, but these differences were not statistically significant at the 95 percent confidence level. That is, we could not conclude that there was a statistical relationship between racial or ethnic minority status and promotion from these ranks. Compared with our descriptive analysis, our adjusted analysis found a larger percentage difference in promotion outcomes at all levels from Class 4 to the executive rank for racial or ethnic minorities relative to whites. Figure 8 shows key results of our descriptive and adjusted analyses of USAID data for racial or ethnic minorities and whites in the Foreign Service. Our analyses of USAID data on promotions in fiscal years 2002 through 2017 found differences between promotion outcomes for women relative to men, but these differences were generally not statistically significant. We found these differences when conducting descriptive analyses, which calculated simple average promotion rates, as well as adjusted analyses, which controlled for certain individual and occupational factors other than gender that could influence promotion. In particular, we found that average promotion rates for women in the Civil Service varied relative to men, but the differences were not statistically significant. In the Foreign Service, average promotion rates varied for women relative to men, but these differences were statistically significant only for promotion from Class 4 to Class 3. Our analyses do not completely explain the reasons for differences in promotion outcomes, which may result from various unobservable factors. Thus, our analyses do not establish a causal relationship between demographic characteristics and promotion outcomes. As table 6 shows, our descriptive analysis of USAID data on promotions in fiscal years 2002 through 2017 found that the rate of promotion in USAID’s Civil Service was generally lower for women than for men at GS- 13 and lower ranks. However, our adjusted analysis did not find any statistically significant differences in the rates or odds of promotion for women relative to men in the Civil Service. Our descriptive analysis of the data for USAID’s Civil Service found that the average percentage of women promoted from GS-11 through GS-13 was lower than the average percentage of men. For example, our descriptive analysis found that in fiscal years 2002 through 2017, an average of 47.4 percent of women were promoted from GS-11 to GS-12, compared with an average of 58.7 percent of men. This difference of 11.3 percentage points indicates that the average rate of promotion from GS- 11 to GS-12 was 19.3 percent lower for women than for men. However, our descriptive analysis does not account for the variety of factors besides gender (e.g., occupation) that may affect promotion rates. Our adjusted analysis of the USAID data, controlling for certain factors other than gender that could influence promotion, found no statistically significant differences in the rates or odds of promotion for women compared with men in the Civil Service. Specifically, the adjusted analysis for promotions in fiscal years 2002 through 2017 found the following: The adjusted rates and odds of promotion from GS-12 to GS-13, from GS-13 to GS-14, and from GS-14 to GS-15 were lower for women than for men. Our estimates of the odds of promotion from GS-11 to GS-12 and from GS-15 to the executive rank were higher for women than for men. In all cases, we found no statistically significant differences at the 95 percent confidence level in the odds of promotion from any rank for women relative to men in the Civil Service. That is, we could not conclude that there was a statistical relationship between gender and promotion from these ranks. Figure 9 shows key results of our descriptive and adjusted analyses of USAID data for men and women in USAID’s Civil Service. Our descriptive and adjusted analyses of data on promotions in fiscal years 2002 through 2017 for USAID’s Foreign Service both found that the rate and odds of promotion were generally higher for women than for men, as table 7 shows. Our descriptive analysis of the data for USAID’s Foreign Service found that higher average percentages of women, relative to men, were promoted from Class 4 to Class 3, from Class 2 to Class 1, and from Class 1 to the executive rank. For example, our descriptive analysis found that in fiscal years 2002 through 2017, an average of 33.9 percent of women were promoted from Class 4 to Class 3, compared with an average of 32.2 percent of men. This 1.7 percentage point difference indicates that the average rate of promotion from Class 4 to Class 3 was 5.2 percent higher for women than for men. However, our descriptive analysis does not account for the variety of factors besides gender (e.g., occupation) that may affect promotion rates. Our adjusted analysis of the data for USAID’s Foreign Service, controlling for certain factors other than gender that could influence promotion, found that the adjusted rates and odds of promotion varied for women relative to men in the Foreign Service. Specifically, our adjusted analysis of data on promotions in fiscal years 2002 through 2017 found the following: On average, the adjusted rate of promotion from Class 4 to Class 3 for women in the Foreign Service was 34.7 percent, compared with 31.5 percent for men. This statistically significant difference indicates that the odds of promotion from Class 4 to Class 3 were 20.2 percent higher for women than for men. While the adjusted rates of promotion from Class 3 to Class 2 and from Class 1 to the executive rank were lower for women than for men, there were no statistically significant differences in the odds of promotion from these ranks for women relative to men in the Foreign Service. Thus, we could not conclude that there was a statistical relationship between gender and promotion from these ranks. Compared with the descriptive analysis, our adjusted analysis found a smaller percentage difference in promotion outcomes from Class 3 to Class 2 and from Class 2 to Class 1 for women relative to men. Our adjusted analysis also found positive, rather than negative, percentage differences in promotion outcomes from Class 4 to Class 3 and from Class 2 to Class 1 for women relative to men. Figure 10 displays key results of our descriptive and adjusted analyses of USAID data for men and women in USAID’s Foreign Service. USAID has determined that specific groups, such as Hispanics and African Americans, are underrepresented in its workforce, and the agency has a strategic plan that identifies goals, activities, and measures to support diversity and inclusion. However, staffing gaps stemming in part from a lack of leadership attention have prevented OCRD from conducting required equal employment opportunity functions. Specifically, staffing gaps have prevented OCRD from responding to EEO complaints within mandated timeframes, analyzing USAID’s workforce for trends and potential barriers to equal employment, and completing the annual MD- 715 reports on the agency’s diversity efforts. USAID has identified specific groups that are underrepresented in its workforce relative to the national civilian labor force. In each of its MD- 715 reports to EEOC for fiscal years 2013 through 2017, USAID identified the following groups as being underrepresented in its workforce: (1) Hispanic males and females in both the Civil Service and the Foreign Service; (2) individuals with a targeted disability; and (3) Hispanic, African American, and Asian American males and females in certain major occupations in areas such as health, contracting, program or project development, auditing, and management and program analysis. According to USAID officials, these groups remain underrepresented in USAID’s workforce. In fiscal years 2010 through 2016, USAID completed analyses intended to identify barriers that could contribute to underrepresentation of specific groups and other diversity issues and described such barriers in its MD- 715 reports. For example, in its report for fiscal year 2010, USAID stated that its recruitment and outreach efforts had failed to attract a representative pool of qualified applicants. In its report for fiscal year 2011, USAID stated that it had no executive development program to prepare employees to enter the senior executive service. In its report for fiscal year 2016, USAID reported on three barrier analyses examining the underrepresentation of, respectively, Hispanics; people with targeted disabilities; and African Americans, Asian Americans, and Hispanics in major occupations. Additional diversity issues may exist at USAID. For example, in 2014, EEOC found that black and Asian females may encounter barriers to equal employment when attempting to enter USAID’s Senior Foreign Service. Further, representatives from 10 of 11 employee resource groups told us that they believed members of their communities have fewer career prospects at USAID than members of other USAID communities. USAID outlined planned efforts to support diversity and inclusion in its June 2016 Human Resource Transformation Strategy and Action Plan, 2016-2021 (HR Transformation Strategy) as well as its 2017 Diversity and Inclusion Strategic Plan. According to the HR Transformation Strategy, USAID envisioned an environment in which diversity recruiting is targeted and strategic, selection bias does not prevent diverse candidates from being hired, all staff and supervisors are trained regularly in diversity and inclusion topics, and agency leaders incorporate diversity into staffing decisions. The HR Transformation Strategy included an objective focused on diversity and inclusion, with planned activities to work toward this goal. The 2017 Diversity and Inclusion Strategic Plan was developed concurrently with, and folded into, the HR Transformation Strategy’s diversity and inclusion objective. Shortly into the first year of the HR Transformation Strategy implementation, USAID narrowed its scope and suspended the diversity and inclusion objective. USAID’s 2017 Diversity and Inclusion Strategic Plan identifies three goals: (1) diversify the federal workforce though active engagement of leadership, (2) include and engage everyone in the workplace, and (3) optimize inclusive diversity efforts using a data-driven approach. The plan also identifies priorities, activities, and measures intended to meet USAID’s diversity goals, several of which cite, and overlap with, the original diversity and inclusion–related elements of the HR Transformation Strategy. HCTM and OCRD officials indicated that that the Diversity and Inclusion Strategic Plan includes some of the areas that would no longer be addressed through the HR Transformation Strategy. In addition, the officials noted that USAID has implemented some aspects of the plan. For example, according to the officials, its employee resource groups have participated in various outreach and recruitment events, as called for by the plan. HCTM and OCRD officials told us that USAID was drafting an update to the Diversity and Inclusion Strategic Plan, which it aimed to finish in June 2020. The officials stated that, although OCRD and HCTM will remain the plan’s primary implementers, the new plan will give USAID bureaus and offices more responsibility for diversity and inclusion activities. Additionally, the officials stated that working groups from USAID’s employee resource groups had begun reviewing the draft. The officials stated that OCRD expected to submit the draft to the Executive Diversity Council for comment after these reviews. OCRD has faced persistent staffing gaps stemming in part from a lack of management attention to the agency’s EEO programs. Moreover, the office has experienced turnover among its directors. OCRD officials stated that the staffing gaps and turnover challenges have prevented the office from completing required EEO functions. As figure 11 shows, the number of OCRD’s filled positions has consistently been less than its allocation. According to OCRD and EEOC officials, the office needs to fill its allocated positions to effectively perform its duties and responsibilities. These staffing gaps generally correspond to times when USAID reported that OCRD could not perform EEO investigations within mandated timeframes, conduct barrier analyses of the agency’s workforce, or complete an MD-715 report. OCRD cannot effectively perform its duties and responsibilities without sufficient staff, according to OCRD officials. Federal equal employment regulations require federal agencies to provide sufficient resources to their EEO programs to ensure efficient and successful operation. However, as table 8 shows, OCRD has faced staffing gaps since fiscal year 2010. According to USAID officials, vacancies have a greater effect on smaller offices such as OCRD, where fewer staff are available to take on the resulting extra work. The officials said that this can in turn affect morale, which can increase staff turnover. Such turnover is observable in USAID’s employee data showing the number of employees and new hires in OCRD. Specifically, while OCRD added new hires to the office each fiscal year, the number of filled positions generally stayed the same or decreased. For example, the number of filled positions in OCRD decreased from 10 to nine in fiscal year 2016, despite the addition of a new hire. Similarly, in fiscal years 2017 and 2018, OCRD’s filled positions remained constant at 10 despite four new hires during that period. As a result, OCRD’s vacancy rate remained near or above 30 percent from October 2015 through April 2020. EEOC similarly noted OCRD’s insufficient staffing in compliance letters to USAID in 2017 and 2019. In both letters, EEOC outlined its expectation that USAID establish a plan to allocate sufficient resources to its EEO program and demonstrate meaningful progress toward correcting this deficiency. USAID officials stated that these staffing gaps have limited OCRD’s capacity to carry out required EEO functions. For example, in November 2019, most of OCRD’s divisions had vacancy rates of 50 percent or more. At that time, all three allocated positions in the Reasonable Accommodation Division and five of six positions in the Diversity and Inclusion Division were vacant. In February 2020, OCRD officials reported that the division’s Affirmative Employment Program had no staff to implement the MD-715 report for fiscal year 2019. Additionally, OCRD reported that the Complaints and Resolution Division’s Anti-Harassment Program continued to receive cases while working through backlogs. Without sufficient staff, OCRD is unable to effectively perform its duties and responsibilities, according to OCRD officials. As part of its response to EEOC’s October 2019 compliance letter, USAID increased the number of positions approved for OCRD to 24. However, the office has struggled to fill those positions. HCTM and OCRD officials stated that, although they are working to resolve the staffing gaps in OCRD, high demand for staff with the specialized skills OCRD requires, as well as unexpected recent turnover in OCRD due to illness and retirement, have hindered this effort. According to USAID officials, long security clearance processes also caused several candidates to withdraw from the hiring process when they found other employment. As table 9 shows, OCRD continued to have staffing gaps of 30 to 50 percent in April 2020. Representatives from nine of the 13 USAID employee groups we spoke with echoed the concern that OCRD lacked sufficient staffing resources to do its job effectively. For example, one group attributed OCRD’s lack of responsiveness to information requests to a lack of sufficient staffing resources. Another group said that there was an implicit understanding in USAID that OCRD had to prioritize reacting to negative events rather than undertaking proactive efforts to increase diversity. Without sufficient staffing resources, USAID will lack the capacity to perform required functions such as responding to EEO complaints, analyzing demographic data, or completing annual MD-715 reports. According to EEOC MD-715 instructions to federal agencies, model EEO programs must have sufficient budget and staffing to support the success of the EEO program, including sufficient staffing to ensure thorough and fair processing of EEO complaints in a timely manner. According to USAID, a lack of staffing resources has prevented the agency from meeting required time frames for EEO investigations. In four of its six MD- 715 submissions for fiscal years 2010 through 2018, USAID reported that it did not have sufficient staffing to implement a successful complaint process. In recent years, USAID has consistently reported being unable to complete EEO counseling, EEO investigations, or final agency decisions on EEO complaints in a timely manner, as required by federal equal employment regulations. For example, in fiscal year 2013 and fiscal years 2015 through 2019, USAID reported being unable to complete EEO investigations within prescribed time frames. Further, in an October 2019 compliance letter, EEOC stated that in fiscal year 2018, USAID completed 67 percent of EEO counseling, 14 percent of EEO investigations, and none of the final agency decisions in a timely manner. As table 10 shows, USAID reported that it did not complete any stages of the EEO complaints response process in a timely manner for fiscal years 2016 through 2019, with the exception of EEO counseling in fiscal year 2016. In fiscal year 2019, the agency continued to lack sufficient funding and qualified staffing to process EEO complaints in a timely, thorough, and fair manner, according to USAID documentation. Representatives of three USAID employee groups also stated that OCRD lacked the capacity to address EEO issues in a timely manner and attributed this to understaffing. Representatives of the first group said that, at a certain point, USAID had a single EEO investigator for the entire agency and that investigations took more than a year. Representatives of the second group stated that because OCRD was short-staffed, it had a backlog of complaints of harassment and bullying. Representatives of the third group said that they had observed the reasonable-accommodation process taking longer than a year. They speculated that this had resulted from USAID’s assigning the handling of reasonable-accommodation requests across the worldwide portfolio to a single person. According to USAID, OCRD has made progress in reducing complaint backlogs. In February 2020, OCRD officials said that the timeliness requirement had been met for the EEO complaint process and that the office no longer had a backlog of complaints. However, OCRD officials said that backlogs remained in processing anti-harassment cases. Further, the officials said that the Reasonable Accommodation Program continued to be affected by a lack of staff. In an April 2020 compliance letter to EEOC, USAID reported that OCRD had developed metrics and new internal procedures for complaint processing. The letter further stated that thus far in fiscal year 2020, OCRD had been 100 percent timely with EEO counseling, EEO investigations, and final agency decisions. While USAID has noted recent improvement in its ability to conduct timely EEO counseling and investigations, without the capacity to consistently perform these functions, USAID cannot meet mandated timeframes for responding to EEO complaints and risks being unable to achieve its goal of a diverse and inclusive workforce environment. According to EEOC MD-715 instructions to federal agencies, model EEO programs must have sufficient budget and staffing to, among other things, conduct self-assessments of possible program deficiencies and conduct thorough barrier analyses of their agency’s workforce. Although USAID has previously completed barrier analyses of its workforce, the agency reported insufficient personnel resources to conduct annual agency self- assessments and self-analyses for its MD-715 submissions for fiscal years 2010, 2013, 2016, and 2017. For example, the fiscal year 2017 MD-715 report stated that USAID did not conduct trend analyses of the effects of management or personnel policies, procedures, and practices and that the agency lacked sufficient resources to enable it to conduct a thorough barrier analysis of its workforce. According to USAID officials, OCRD lost its staff member assigned to manage barrier analyses and was unable to fill that position during the hiring freeze. Further, OCRD continues to lack sufficient personnel to conduct barrier analyses. In November 2019, OCRD’s Diversity and Inclusion Division consisted of one supervisor and five vacant positions. Despite subsequent efforts to hire more staff, OCRD reported in February 2020 that it still lacked staff to perform its data analysis responsibilities. EEOC officials expressed concern regarding OCRD’s lack of capacity to analyze and address diversity issues. For example, EEOC officials said that USAID had not adequately used applicant flow data to identify potential barriers in fiscal year 2017. Despite having collected applicant data, USAID did not submit applicant flow data as part of its MD-715 submission for fiscal year 2017, the most recent year for which it submitted this report. According to the EEOC officials, OCRD told them that it lacked staff with sufficient technical expertise to conduct a barrier analysis of these data. Without the capacity to perform self-analysis, USAID is unable to proactively identify and address barriers to diversity in its workforce. EEOC MD-715 requires federal agencies to submit their MD-715 reports to the EEOC annually. The report is due by February 28 following the end of the fiscal year that is being reported, although EEOC has the discretion to grant extensions. However, OCRD did not complete the MD-715 report in fiscal years 2011 or 2012 and has not submitted an MD-715 report for fiscal year 2018. Despite being granted submission extensions, USAID had not submitted its MD-715 report for fiscal year 2018 by the certification deadline of September 30, 2019, according to EEOC’s October 2019 compliance letter. The letter stated that EEOC expected USAID to submit the MD-715 report for fiscal year 2018 and to ensure that the MD-715 report for fiscal year 2019 would be submitted by the deadline of February 28, 2020. In November 2019, USAID officials told us that OCRD lacked the staff needed to complete the fiscal year 2018 MD- 715 report by this deadline and therefore intended to concentrate on submitting a report for fiscal year 2019. However, in February 2020, USAID officials told us that OCRD’s Affirmative Employment Program continued to lack any staff to monitor and implement the MD-715 effort. In April 2020, USAID officials reported that they were using a contractor to complete the fiscal year 2019 MD-715 report. Without OCRD capacity to submit required reports on the agency’s diversity and inclusion efforts, USAID leadership will lack sufficient insight into the EEO program to ensure that its activities meet agency goals. Furthermore, inconsistent reporting could hamper EEOC’s oversight of USAID’s EEO programs. OCRD’s staffing gaps stem in part from a lack of leadership attention to USAID’s equal employment opportunity programs at both the office and agency levels. We have previously identified top leadership commitment as a leading practice for diversity management. Leaders and managers within organizations are primarily responsible for the success of diversity management, because they must provide the visibility and commit the time and necessary resources. Both USAID and EEOC officials attributed OCRD’s staffing problems to frequent management turnover within OCRD. According to information provided by USAID officials, OCRD has had five directors (permanent and acting) since 2013. USAID officials stated that this turnover made it difficult for any director to provide sufficient office-level leadership attention to sustain efforts to improve OCRD’s capacity. EEOC officials also expressed concern regarding this level of director turnover and asserted that without consistent office leadership that could effectively advocate for scarce personnel resources within USAID, OCRD would continue to face staffing shortages. EEOC officials said that OCRD could not draw sufficient attention from senior USAID leadership without a permanent director. According to EEOC MD-715 instructions to federal agencies, model EEO programs have a reporting structure for the EEO program that provides the principal EEO official with appropriate authority and resources to effectively carry out a successful EEO program. This includes, but is not limited to, an annual State of the Agency briefing given by the EEO Director (in USAID’s case, the Director of OCRD) to the agency head and other senior management officials after the submission of a MD-715 report. According to MD-715 instructions to federal agencies, the briefing must thoroughly cover all components of the agency’s MD-715 report, including an assessment of the agency’s performance in each of the six elements of a model EEO program, as well as a report on the agency’s progress in completing its barrier analysis. However, OCRD has not presented a State of the Agency briefing to the head of USAID and other senior leadership for 3 consecutive fiscal years. In April 2020, OCRD officials told us that the office planned to provide the briefing to USAID’s Executive Diversity Council once the MD-715 for fiscal year 2019 was completed, which they anticipated would occur in May 2020. HCTM and OCRD officials also told us that since receiving the EEOC’s October 2019 compliance letter, senior USAID leadership had been more engaged than previously. Without senior USAID leadership attention to diversity, OCRD will continue to lack the staffing resources necessary to build its capacity to support USAID’s diversity and inclusion efforts as well as operate an effective and efficient EEO program. Although USAID has made some progress in increasing representation of diverse groups in its Civil and Foreign Service workforces, continued underrepresentation and generally lower promotion outcomes for racial or ethnic minorities suggest that additional efforts are needed. Addressing these issues requires an effective and efficient EEO program. However, OCRD, which operates the agency’s EEO program, is currently unable to perform its key functions because of significant staffing gaps and turnover. USAID’s recent efforts to fill staff vacancies within various OCRD divisions could help increase OCRD’s capacity to perform its required EEO functions. However, such capacity will not be fully demonstrated until OCRD can consistently ensure timely processing of EEO complaints and investigations, regular analysis of workforce demographics for trends, and regular submission of required MD-715 reports. Further, sustained attention to diversity efforts from USAID’s senior leadership would help ensure that OCRD has the capacity to perform its required EEO functions. Without capacity to perform these functions, USAID cannot consistently respond to allegations of discrimination in a timely manner, identify potential barriers to equal employment opportunity, or maintain accountability for the progress of its diversity and inclusion efforts. We are making the following four recommendations to USAID: 1. The USAID Administrator should ensure that OCRD consistently responds to EEO complaints in a timely manner. (Recommendation 1) 2. The USAID Administrator should ensure that OCRD consistently analyzes USAID workforce demographic data for trends and potential barriers to equal employment opportunity. (Recommendation 2) 3. The USAID Administrator should ensure that OCRD submits required MD-715 reports to EEOC in a timely manner. (Recommendation 3) 4. The USAID Administrator should demonstrate senior leadership attention to diversity by ensuring that OCRD has the capacity to perform required EEO functions. (Recommendation 4) We provided a draft of this report to USAID, EEOC, and OPM for comment. USAID provided comments, which we have reproduced in appendix XV. EEOC and OPM stated they did not have comments. In its comments, USAID concurred with our four recommendations and described actions planned or underway to address them. For example, in response to recommendations 2 and 3, USAID stated that it is in the process of establishing an Affirmative Employment Program in OCRD to, among other things, analyze and report on workforce data and prepare and submit the agency’s annual MD-715 Report. USAID indicated that it expects to finish implementing actions addressing our EEO-related recommendations in 2020. We believe that, to demonstrate consistent capacity to perform its EEO functions, USAID will need to successfully complete these functions for at least two consecutive cycles. We are sending copies of this report to the appropriate congressional committees, the Administrator of USAID, the Chair of EEOC, and the Director of OPM. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6881 or at bairj@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix XVI. This report examines (1) the demographic composition of USAID’s workforce in fiscal years 2002 through 2018, (2) differences in promotion outcomes for racial or ethnic groups in USAID’s workforce, (3) differences in promotion outcomes for men and women in USAID’s workforce, and (4) the extent to which USAID has identified workforce diversity issues and worked to address them. For this report, we analyzed National Finance Center data on USAID’s full-time, permanent, career workforce (direct-hire U.S. citizen Civil and Foreign Service employees) for fiscal years 2002 through 2018. For each fiscal year, we analyzed record-level status data for USAID’s employees as of September 30 (the end of the fiscal year). This included demographic and administrative data for each employee, such as race, ethnicity, gender, grade or class, age, date of entry to USAID, years of service, veteran’s status, occupation, location or duty station, and the employee’s unique identifier. We also analyzed record-level dynamic data that included personnel actions, such as promotions or separations. In addition, we obtained “Post (Hardship) Differential Percentage of Basic Compensation” data from the Department of State’s website for fiscal years 2002 through 2018. Following guidance from the U.S. Equal Employment Opportunity Commission, we used data for nine federal job categories and their correspondence to specific occupation codes to match federal job categories to the occupations of USAID’s employees. We assessed the reliability of these data sets and of other data critical to our analyses through documentation review, electronic testing, and interviews with knowledgeable agency officials. We determined that these data were sufficiently reliable for our purposes. To examine the demographic composition of USAID’s workforce over time, we analyzed National Finance Center data for USAID’s full-time, permanent, career workforce for fiscal years 2002 through 2018. For each year, we calculated the demographic composition of the workforce by racial or ethnic group and by gender for USAID overall and for USAID’s Civil and Foreign Services. We also analyzed these numbers and percentages by occupation and rank, including General Service (GS) grade for the Civil Service, salary class for the Foreign Service, and executive rank (i.e., Senior Executive Service or Senior Foreign Service). We excluded political appointees and Office of Inspector General employees from our overall analysis because, according to agency officials, USAID’s Office of Human Capital and Talent Management does not have authority over these hires. We also compared the demographics of USAID’s workforce in fiscal year 2018 with the most recent available data on demographics of (1) the federal workforce, as reported by the Office of Personnel Management (OPM), and (2) the relevant civilian labor force, from the Census Bureau’s equal employment opportunity (EEO) tabulation. Because of USAID’s involvement in disability-related litigation during the course of this engagement, we did not analyze the numbers and percentages of employees with disabilities. Additionally, because the National Finance Center data we used did not include information about employees’ sexual orientation, we were unable to analyze the data on that basis. For the purposes of our report, racial or ethnic minorities exclude non- Hispanic whites; Hispanics include Hispanics of all races; and the remaining non-Hispanic racial or ethnic groups include white, African American, Asian, and other. Our analysis for the category we report as “other” includes non-Hispanics identified as American Indian or Alaskan Native, Native Hawaiian or other Pacific Islander, and individuals identifying as two or more races. For instances where an employee’s reported racial, ethnic, or gender category changed, we assigned the most recently recorded category to all available years. To examine promotion outcomes for racial or ethnic minorities and women in USAID’s workforce, we conducted two types of analyses— descriptive and adjusted—using USAID’s National Finance Center data for its full-time, permanent, career workforce in fiscal years 2002 through 2018. For both analyses, we considered promotion to be an increase in rank between 2 consecutive fiscal years. We included in these analyses all individuals in the original rank and did not distinguish between individuals who did or did not apply for promotion or who were eligible or ineligible. We conducted a descriptive analysis of USAID data, comparing annual promotion rates for racial or ethnic minorities and whites and for women and men. For each rank and fiscal year, we calculated these rates as the number of newly elevated employees in the next-higher rank in the following fiscal year divided by the number of employees in the given rank in the current year. We conducted adjusted analysis using a multivariate statistical method (i.e., duration analysis), which accounted for certain individual and occupational factors other than racial or ethnic minority status and gender that could influence promotion. Specifically, we used a discrete-time multivariate statistical logit model to analyze the number of yearly cycles it took to be promoted up to the executive level from GS-11 in the Civil Service and from Class 4 in the Foreign Service. We examined the statistical relationship between promotion and racial or ethnic minority status and gender, including adjusted promotion rates, odds ratios, and percentage differences in relative odds of promotion. Because a variety of factors besides racial or ethnic minority status and gender may influence promotion outcomes, we incorporated various individual and position-specific characteristics in our regression models to control for other potential factors. These included an employee’s (1) time in each rank before promotion; (2) years of prior federal government experience; (3) age when entering USAID; (4) receipt of veterans’ preference points; (5) having transferred between the Civil and Foreign Services; (6) having worked overseas in the previous year (for the Foreign Service); (7) having worked in at a location where the hardship differential was 20 percent or more in the previous year (Foreign Service only); (8) proficiency in two or more languages other than English (Foreign Service only); and (9) occupation as well as (10) fiscal years. We identified these attributes as being relevant to promotion by reviewing relevant literature and interviewing agency officials. Our primary model was a pooled model that included all employees whose records we used to determine summary statistics for USAID’s full-time, permanent, career workforce in fiscal years 2002 through 2018. Additionally, we conducted a number of sensitivity analyses, such as examining the robustness of our models to the inclusion of various sets of control variables (see app. XIII) and applying the multivariate statistical method for various permutations of racial or ethnic minority status (see app. XIV). To examine the extent to which USAID has identified workforce diversity issues and worked to address them, we reviewed all annual Management Directive 715 reports that it submitted to EEOC from fiscal year 2011 through fiscal year 2019. We also reviewed policies, guidance, and other USAID documentation related to diversity. Additionally, we met with relevant USAID officials from the Office of Civil Rights and Diversity and the Office of Human Capital and Talent Management as well as officials from EEOC. We also conducted interviews with representatives of 13 employee groups representing current employees in USAID’s Civil and Foreign Services to obtain their perspectives on diversity efforts at USAID. These groups included two unions: the Association of Federal Government Employees and the American Foreign Service Association. The 13 groups also included 11 employee resource groups: Arab- Americans in Foreign Affairs Agencies, the Asian Pacific American Employees Committee, Blacks in Government, Employees with Disabilities, Gender and Sexual Minorities, the Hispanic Employees Council of Foreign Affairs Agencies, the Jewish Affinity Group, the Native Americans in Foreign Affairs Council, the Personal Services Contractor Association, the USAID Muslims Employee Resource Group, and Women@AID. We conducted this performance audit from October 2018 to June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The following figures and tables present numbers and proportions of employees in racial, ethnic, and gender groups in the U.S. Agency for International Development (USAID) overall and in USAID’s Civil and Foreign Services in fiscal years 2002 through 2018. We compared summary statistics for the U.S. Agency for International Development’s (USAID) workforce overall with summary statistics for the federal government and relevant civilian labor force. We compared summary statistics calculated from USAID personnel data for fiscal year 2018 with summary statistics for the federal government for fiscal year 2017, published in the Federal Equal Opportunity Recruitment Program (FEORP) report. Our comparison of USAID personnel data with data from the Office of Personnel Management’s FEORP report for the federal government found differences between the proportions of racial or ethnic minorities at USAID and those in the federal workforce. In particular, the proportions of African Americans and Asians were higher at USAID in fiscal year 2018 than in the federal workforce in fiscal year 2017, but the proportion of Hispanics was lower at USAID than in the federal workforce for those years. The proportion of women at USAID was higher than in the federal workforce (see table 17). We compared summary statistics for USAID’s workforce with summary statistics for the relevant civilian labor force from the Census Bureau’s equal employment opportunity tabulation for three of the Equal Employment Opportunity Commission (EEOC) occupational classification system’s nine categories. Using an EEOC table that cross-classifies Office of Personnel Management occupation codes and federal sector occupational categories, we classified each USAID employee into one of the nine categories. We compared USAID and relevant civilian labor force statistics for the following three categories, corresponding to 99 percent of USAID’s full-time, permanent employees in fiscal year 2018: officials and managers, professional workers, and technical workers and technologists. Our comparison of USAID workforce data with relevant civilian labor force data found generally larger proportions of racial or ethnic minorities at USAID than in the relevant civilian labor force for officials and managers, professional workers, and technical workers and technologists (see tables 18 through 20). The proportions of women were lower at USAID than in the relevant civilian labor force for professional workers but were higher for officials and managers and for technical workers and technologists. To compare U.S. Agency for International Development (USAID) and federal government workforce data, we contrasted summary statistics on executive employees calculated from USAID personnel data for fiscal year 2018 with summary statistics on executives from federal government workforce data for fiscal year 2017 that were published in the Federal Equal Opportunity Recruitment Program (FEORP) report. As table 21 shows, our comparison of USAID workforce data with the FEORP data found a slightly higher proportion of white executives and a slightly lower proportion of racial or ethnic minority executives at USAID than in the federal workforce overall. We analyzed U.S. Agency for International Development (USAID) data on employees hired with veterans’ preference in fiscal years 2002 through 2018. The following tables present the numbers and percentages of employees hired with or without veterans’ preference in USAID’s workforce overall and in USAID’s Civil and Foreign Services during that period. Table 25 shows the proportions of permanent employees with a disability in the U.S. Agency for International Development’s (USAID) Civil and Foreign Services in fiscal years 2009 through 2017. The data shown are summary statistics from USAID’s Management Directive 715 (MD-715) reports to the Equal Employment Opportunity Commission. As the table shows, the proportion of permanent employees with disabilities increased in the Civil Service and remained constant in the Foreign Service in the years for which USAID reported these data. In addition to analyzing the demographic composition of the U.S. Agency for International Development’s (USAID) workforce, we analyzed USAID personnel data to determine summary statistics on political appointees in fiscal years 2002 through 2018. We considered employees to be political appointees if they were on the executive pay plan or the administratively determined pay plan. This includes Senate-confirmed political appointees as well as political appointees that did not require Senate confirmation. The following figures and tables present the numbers and proportions of political appointees in racial or ethnic and gender groups in USAID overall and USAID’s Civil Service and Foreign Service in fiscal years 2002 through 2018. We also analyzed USAID personnel data to determine summary statistics on employees of the agency’s Office of Inspector General in fiscal years 2002 through 2018. The following tables present the numbers and percentages of the office’s employees in racial or ethnic and gender groups in fiscal years 2002 through 2018. We analyzed data for applicants to the U.S. Agency for International Development’s (USAID) Civil Service in fiscal years 2012 and 2018 and applicants to USAID’s Foreign Service in fiscal years 2012 and 2016. According to USAID’s guidance on personnel recruitment, an applicant is considered eligible when USAID’s online application evaluation system, using the applicant’s online responses to standardized questions, determines that the applicant meets eligibility requirements and the minimum qualifications defined in the vacancy announcement. USAID’s Civil Service staffing guidance provides that officials may interview and make selections on the basis of referral lists of eligible applicants. USAID’s personnel recruitment guidance for the Foreign Service also notes that an applicant is considered selected when the applicant’s score is above the cut-off total score and the applicant has passed the onsite assessment to advance to the reference-check stage of the hiring process. We considered an applicant to have been rated eligible if the applicant data showed that the applicant had not been rated ineligible. We considered an applicant to have been selected if the applicant data showed that the applicant was either hired or selected. Tables 30 through 32 show the percentages of eligible applicants and selected eligible applicants to, respectively, USAID overall in fiscal years 2012 and 2018, USAID’s Civil Service in fiscal years 2012 and 2018, and USAID’s Foreign Service in fiscal years 2012 and 2016. In addition to analyzing the demographic composition of the U.S. Agency for International Development (USAID) workforce, we analyzed USAID personnel data to determine summary statistics on employees hired in fiscal years 2003 through 2018. We considered an employee to have been hired in a given fiscal year if the employee first appeared in USAID’s personnel data for that year. Because the USAID data we reviewed began in fiscal year 2002, we were unable to identify employees who were hired in that fiscal year; thus, fiscal year 2003 is the first for which we were able to identify newly hired employees. Figure 21 shows the number of newly hired employees at USAID from fiscal year 2003 to fiscal year 2018. The following figures and tables present the numbers and proportions of newly hired employees in racial, ethnic, and gender groups in USAID overall and USAID’s Civil Service and Foreign Service in fiscal years 2003 through 2018. In addition to analyzing the demographic composition of the U.S. Agency for International Development’s (USAID) workforce, we analyzed USAID personnel data to determine summary statistics for employees who left USAID in fiscal years 2003 through 2018 for reasons other than retirement or death. Figures 24 and 25 show the percentages of such employees in various racial, ethnic, and gender groups at USAID overall and in USAID’s Civil Service and Foreign Service in fiscal years 2003 and 2018. Table 35 presents attrition rates for white and racial or ethnic minority employees who left USAID in fiscal years 2003 through 2018 for reasons other than retirement or death. Table 36 presents attrition rates for men and women who left USAID in fiscal years 2003 through 2018 for reasons other than retirement or death. As table 37 shows, our analysis of yearly promotion rates for fiscal years 2013 through 2017 at the U.S. Agency for International Development (USAID) found that promotion rates for white employees exceeded those for racial or ethnic minority employees for Civil Service promotions from GS-11 and every higher rank in every year, except from GS-15 to executive in 3 years, and Foreign Service promotions from Class 4 and higher ranks for 11 of the 20 possible year-rank combinations. Table 38 shows the promotion rates for white employees and racial or ethnic minority employees in USAID’s Civil and Foreign Services in fiscal years 2013 through 2017. from Class 4 and higher ranks for 12 of the 20 possible year-rank combinations in the Foreign Service. Table 40 shows the promotion rates for men and women in USAID’s Civil and Foreign Services in fiscal years 2013 through 2017. Our analysis of U.S. Agency for International Development (USAID) workforce data found that racial or ethnic minorities generally spent more years in each rank than whites did in USAID’s Civil Service in fiscal years 2002 through 2018. Table 41 shows the average years in rank for whites and racial or ethnic minorities in USAID’s Civil and Foreign Services. Our analysis also found that in the Civil Service, women generally spent more years than men in early- to mid-career ranks (GS-13 and below) before being promoted. However, women spent fewer years than men in later career ranks (GS-14 and above) before being promoted. In the Foreign Service, women generally spent fewer years than men in early- to mid-career ranks (Class 2 and below) before being promoted. Table 42 shows the average years in rank for men and women in USAID’s Civil and Foreign Services in fiscal years 2002 through 2018. Tables 43, 44, 50, and 51 provide summaries of the multivariate statistical regression results (specifically, duration regression results) for our estimates of the percentage differences in odds of promotion for racial or ethnic minorities compared with whites and for women compared with men in the U.S. Agency for International Development’s (USAID) Civil and Foreign Services. Our analyses do not completely explain the reasons for differences in promotion outcomes, which may result from various unobservable factors. Thus, our analyses do not establish a causal relationship between demographic characteristics and promotion outcomes. promotion slots (and therefore promotion outcomes) may be affected by budget constraints that vary across fiscal years. Model 6 used data for fiscal years 2011 through 2018 only. In addition to controlling for the same variables as model 5, model 6 controlled for use of long-term leave in the prior year. Tables 43 through 55 provide the regression results of these six models for all promotion stages that we analyzed in the Civil and Foreign Services. Tables 43, 44, 50, and 51 present the consolidated regression results for all six models and all promotion stages, presented as estimates of percentage differences. Tables 45 through 49 and tables 52 through 55 provide the full regression results of the first five models, presented as odds ratios. Odds ratios that are statistically significant and lower than 1.00 indicate that individuals with the given characteristic were less likely to be promoted. Odds ratios that are statistically significant and greater than 1.00 indicate that individuals with the given characteristic were more likely to be promoted. To convert the values in tables 45 through 49 and tables 52 through 55 to the values in tables 43, 44, 50, and 51, we linearly transformed the estimates. That is, the values for the estimates in tables 43, 44, 50, and 51 are equal to the values in tables 45 through 49 and in tables 52 through 55 multiplied by 100, minus 100. The values for the standard errors in tables 43, 44, 50, and 51 are equal to the values in tables 45 through 49 and in tables 52 through 55 multiplied by 100. For example, in table 45, the estimate for model 1a is 0.463; we arrived at the percentage difference of negative 54 percent in table 43 by 0.463*100-100. Additionally, in table 45, the estimate for the standard error for model 1a is (0.0624); we arrived at the converted standard error of (6) in table 45 by (0.0624)*100. Table 43 summarizes the regression results for our estimates of the percentage differences in odds of promotion for racial or ethnic minorities compared with whites in the Civil Service. We observed that racial or ethnic minorities’ lower odds of promotion from GS-11 through GS-14 were consistently statistically significant across all of our models examining combinations of factors that could influence promotion (i.e., models 1a through 5). In addition, our results were generally statistically significant when we examined the more recent time period fiscal years 2011 through 2018 (see model 6). (0.0624) (0.0629) (0.0950) (0.0850) (0.0972) 0.872 (0.200) 0.962 (0.229) 1.016 (0.255) Age at entry, squared (0.000627) (0.000640) (0.000659) 0.932*** (0.0197) 0.927*** (0.0200) 0.909*** (0.0211) Years of government service, squared (0.317) (0.362) (0.340) (0.393) (0.399) 0.764 (0.174) 0.802 (0.193) (1.043) (2.707) 1.750** (0.393) 1.919*** (0.459) (0.568) (0.559) 1.987** (0.662) 2.096** (0.728) Odds ratio (standard error) (0.718) (0.747) 1.147 (0.228) 1.235 (0.262) (5.832) (5.407) (6.191) (1.395) (1.142) (0.802) Legend: GS = General Schedule, — = not applicable, √ = controls applied, *** = statistically significant at p-value < 0.01, ** = statistically significant at p- value < 0.05, * = statistically significant at p-value < 0.1. Odds ratio (standard error) Model 3 Model 2 0.876 0.825* — (0.103) Model 4 0.833 (0.100) Model 5 0.806* (0.102) (0.150) (0.149) (0.163) 1.020 (0.0400) 1.022 (0.0410) 1.032 (0.0427) Age at entry, squared (0.000528) (0.000540) (0.000554) Odds ratio (standard error) Model 2 Model 3 0.937*** — (0.0170) (0.0174) (0.0184) Years of government service, squared 1.000 (0.000678) 1.000 (0.000687) 1.000 (0.000703) (0.158) (0.150) (0.152) (0.278) (0.316) 2.299 (3.283) (0.321) (0.306) 0.781 (0.261) 0.878 (0.303) (0.141) (0.142) 1.518 (0.482) 1.515 (0.500) 2.194*** (0.579) 2.050*** (0.548) 2.457*** (0.669) 2.315 (1.734) 1.564 (1.199) 0.648 (0.538) Legend: GS = General Schedule, — = not applicable, √ = controls applied, *** = statistically significant at p-value < 0.01, ** = statistically significant at p- value < 0.05, * = statistically significant at p-value < 0.1. Appendix XIII: Full Promotion Regression Results For example, the estimated odds ratio for racial or ethnic minority employees for promotion from GS- 12 to GS-13 is 0.640 (model 5), which means that the odds of promotion for racial or ethnic minority employees are about 64 percent of the odds for white employees. We conducted discrete-time duration analysis using logit models to analyze the time duration (number of years) before promotion from each GS grade shown. In all models, we controlled for the time that employees spent in each grade before promotion. The overall baseline population for the duration analysis represents individuals who possessed none of the characteristics indicated by the list of control variables. These analyses do not completely explain why differences in odds of promotion exist. While various independent variables capture and control for many characteristics across demographic groups, unobservable factors may account for differences in odds of promotion; thus, our regression results do not establish a causal relationship between demographic characteristics and promotion outcomes. Model 1b 0.912 (0.0897) Model 2 0.988 (0.0986) Model 3 0.855 (0.0906) Model 4 0.832* (0.0900) Model 5 0.838 (0.0926) (0.116) (0.120) (0.139) 1.021 (0.0385) 1.028 (0.0395) 1.034 (0.0405) Age at entry, squared (0.0158) (0.0165) (0.0170) Years of government, service squared 1.000 (0.000575) 1.000 (0.000594) 1.000 (0.000603) (5.518) (4.311) 0.865 (0.235) 1.013 (0.281) (0.159) (0.160) 0.788 (0.242) 0.703 (0.219) Odds ratio (standard error) (0.163) (0.180) 0.580 (0.210) 0.612 (0.223) (0.277) (0.284) 0.844 (0.263) 0.932 (0.295) 0.198*** (0.0621) 0.164*** (0.0520) 0.199*** (0.0636) 0.243* (0.183) 0.221* (0.172) 0.187** (0.154) Legend: GS = General Schedule, — = not applicable, √ = controls applied, *** = statistically significant at p-value < 0.01, ** = statistically significant at p- value < 0.05, * = statistically significant at p-value < 0.1. (0.0822) (0.0823) (0.0910) (0.0988) (0.111) 0.634* (0.150) 0.633* (0.151) 0.722 (0.175) Age at entry, squared (0.000670) (0.000673) (0.000690) 1.018 (0.0214) 1.024 (0.0221) 0.987 (0.0224) Years of government service, squared (0.327) (0.352) (0.473) (1.609) (1.437) 1.157 (0.497) 1.401 (0.612) (1.302) (1.511) 0.553 (0.229) 0.557 (0.232) (0.172) (0.146) 1.170 (0.451) 1.575 (0.617) Odds ratio (standard error) (0.227) (0.230) 0.856 (0.361) 1.009 (0.433) 0.0232*** (0.0103) 0.0202*** (0.00903) 0.0227*** (0.0102) 0.00362*** (0.00377) 0.00317*** (0.00336) 0.00526*** (0.00584) Legend: GS = General Schedule, — = not applicable, √ = controls applied, *** = statistically significant at p-value < 0.01, ** = statistically significant at p- value < 0.05, * = statistically significant at p-value < 0.1. (0.418) (0.413) (0.332) (0.360) (0.455) 4.827*** (2.417) 5.221*** (2.569) 6.051*** (3.095) (0.147) (0.141) (0.148) Odds ratio (standard error) Control variable Age at entry, squared (0.00180) (0.00173) (0.00174) 1.120* (0.0751) 1.143* (0.0785) 1.054 (0.0768) Years of government service, squared (0.963) (1.082) (1.282) (0.472) (0.514) (0.459) (0.485) (0.274) (0.242) 1.712 (1.907) 2.195 (2.500) — 1.287 (1.431) — 1.708 (1.964) (0.446) (0.414) Odds ratio (standard error 0.00117*** (0.00181) 0.00110*** (0.00171) 0.00109*** (0.00170) 0.000580** (0.00178) 0.000754** (0.00229) 0.00114** (0.00361) Legend: GS = General Schedule, — = not applicable, √ = controls applied, *** = statistically significant at p-value < 0.01, ** = statistically significant at p- value < 0.05, * = statistically significant at p-value < 0.1. Table 50 summarizes the regression results for our estimates of the percentage differences in odds of promotion for racial or ethnic minorities compared with whites in the Foreign Service. We found that racial or ethnic minorities had lower estimated odds of promotion than whites in early to mid career (Class 4 through Class 1), but these differences were generally not statistically significant. However, we observed statistically significantly lower odds of promotion for racial or ethnic minorities from Class 3 through Class 2. These results were consistently statistically significant across all of our models examining combinations of factors that could influence promotion (i.e., models 1a through 5). including in the more recent period fiscal years 2011 through 2018 (see models 5 and 6). Tables 52 through 55 present full regression results for models 1a through 5 for each rank in the Foreign Service. The results are presented as odds ratios. (0.0732) (0.0725) (0.0746) (0.0733) (0.0831) 0.910 (0.229) 0.819 (0.209) 1.010 (0.278) Age at entry, squared (0.000598) (0.000602) (0.000645) 1.104*** (0.0310) 1.104*** (0.0312) 1.098*** (0.0327) Years of government service, squared (0.126) (0.129) (0.128) 2.035*** (0.182) 2.018*** (0.182) 3.664*** (0.442) (0.217) (0.216) (0.346) (3.089) (3.039) 1.338 (0.266) 1.557** (0.339) (0.151) (0.157) 0.935 (0.151) 1.190 (0.209) Odds ratio (standard error) (0.124) (0.147) 1.592** (0.362) 1.391 (0.334) Duration controls Fiscal year controls (0.169) (0.212) Legend: — = not applicable, √ = controls applied, *** = statistically significant at p-value < 0.01, ** = statistically significant at p-value < 0.05, * = statistically significant at p-value < 0.1. (0.0722) (0.0745) (0.0817) (0.0831) (0.0911) 0.752*** (0.0769) 0.767** (0.0790) 0.767** (0.0803) 0.748*** (0.0801) 0.785** (0.0895) (0.154) (0.147) (0.201) Odds ratio (standard error) Control variable Age at entry Age at entry, squared (0.0679) (0.0688) (0.0650) 0.998** (0.000764) 0.998** (0.000769) 0.999 (0.000799) Years of government service, squared (0.00140) (0.00141) (0.00148) 1.077 (0.109) 1.078 (0.110) 0.855 (0.0970) (0.0868) (0.0868) (0.121) 1.033 (0.168) 1.024 (0.167) 0.929 (0.164) 2.641 (3.278) 2.499 (3.189) (0.185) (0.182) 1.063 (0.177) 1.166 (0.208) (0.126) (0.221) 1.534** (0.295) 1.857*** (0.386) (0.160) (0.215) Odds ratio (standard error) 0.000459*** (0.000257) 0.000467*** (0.000262) 0.000494*** (0.000277) 0.000022*** (0.000028) 0.000017*** (0.000022) 0.000417*** (0.000588) Legend: — = not applicable, √ = controls applied, *** = statistically significant at p-value < 0.01, ** = statistically significant at p-value < 0.05, * = statistically significant at p-value < 0.1. Odds ratio (standard error) Model 2 Model 3 1.122 0.997 (0.0951) (0.0986) (0.0922) (0.0948) (0.0970) 0.837* (0.0890) 0.820* (0.0882) 0.842 (0.0922) 0.830* (0.0922) 0.863 (0.0979) Age at entry, squared (0.0659) (0.0665) (0.0642) 0.998*** (0.000750) 0.998*** (0.000754) 0.998** (0.000772) Years of government service, squared (0.000815) (0.000821) (0.000859) Odds ratio (standard error) Model 2 Model 3 — 1.417*** (0.135) (0.137) (0.130) 1.480*** (0.165) 1.437*** (0.164) 1.777*** (0.227) (0.141) (0.144) (0.172) (1.409) (3.636) 1.181 (0.241) 1.203 (0.249) (0.200) (0.202) 1.050 (0.197) 1.280 (0.247) (0.274) (0.268) 0.966 (0.229) 1.003 (0.242) (0.130) (0.137) Legend: — = not applicable, √ = controls applied, *** = statistically significant at p-value < 0.01, ** = statistically significant at p-value < 0.05, * = statistically significant at p-value < 0.1. Appendix XIII: Full Promotion Regression Results (0.117) (0.117) (0.124) (0.127) (0.127) 1.028 (0.160) 1.034 (0.162) 1.089 (0.173) 1.220 (0.202) 1.321 (0.226) (0.0613) (0.0577) (0.0612) Age at entry, squared 1.002** (0.00104) 1.003*** (0.00101) 1.003** (0.00106) Years of government service, squared (0.000948) (0.00101) (0.00114) 0.954 (0.125) 0.996 (0.137) 0.918 (0.130) (0.139) (0.154) (0.200) 0.546* (0.182) 0.654 (0.223) 0.827 (0.306) 4.998*** (1.128) 3.924*** (0.907) Odds ratio (standard error) (0.326) (0.321) 0.586* (0.185) 0.514** (0.165) (0.418) (0.413) 1.168 (0.551) 1.139 (0.552) (0.189) (0.145) 1.156 (0.451) 0.882 (0.354) 0.00502*** (0.00303) 0.00513*** (0.00310) 0.00510*** (0.00308) 0.211 (0.312) 0.547 (0.813) 0.00422** (0.0100) Legend: — = not applicable, √ = controls applied, *** = statistically significant at p-value < 0.01, ** = statistically significant at p-value < 0.05, * = statistically significant at p-value < 0.1. Tables 56 and 57 summarize the multivariate statistical regression results (specifically, duration regression results) for our estimates of the percentage differences in odds of promotion for two groupings of racial or ethnic minorities in the U.S. Agency for International Development’s (USAID) Civil and Foreign Services. We examined odds of promotion for African Americans and non– African American racial or ethnic minorities compared with whites. We examined odds of promotion for the individual racial or ethnic groups—African Americans, Hispanics, Asians, and other racial or ethnic minorities—compared with whites. Our analyses do not completely explain the reasons for differences in promotion outcomes, which may result from various unobservable factors. Thus, our analyses do not establish a causal relationship between demographic characteristics and promotion outcomes. Veteran’s status Transferring between the Foreign and Civil Services Having a hardship assignment in the prior year (Foreign Service only) Having an overseas post in the prior year (Foreign Service only) Proficiency in two or more languages other than English (Foreign Service only) Fiscal year fixed effects (indicator variables representing the fiscal year) The third model, which was limited to fiscal years 2011 through 2018, controlled for the same variables as the second model and also controlled for use of long-term leave in the previous year. Table 56 summarizes the regression results for our estimates of the percentage differences in odds of promotion for the two groupings of racial or ethnic minorities compared with whites in the Civil Service. For the first grouping, we found statistically significantly lower odds of promotion from GS-11 through GS-15 for African Americans than for whites in fiscal years 2002 through 2018 (model 2). The odds of promotion from GS-12 to GS-13 were also statistically significantly lower for non–African American racial or ethnic minorities during the same period. For the second grouping, we found statistically significantly lower odds of promotion from GS-12 to GS-13 for Asians than for whites in fiscal years 2002 through 2018. (46) (24) (24) (31) (74) -49 (22) -30 (22) -2 (31) 2 (38) Other racial or ethnic minority (198) (27) (26) (68) Legend: GS = General Schedule, *** statistically significant at p-value < 0.01, ** statistically significant at p-value < 0.05, * statistically significant at p- value < 0.1, — = not applicable. Table 57 presents the summary of the regression results for our estimates of the percentage differences in odds of promotion for the two groupings of racial or ethnic minorities compared with whites in the Foreign Service. For the first grouping, we found statistically significantly lower odds of promotion from Class 4 to Class 3 for African Americans than for whites in fiscal years 2002 through 2018 (model 2). For the second grouping, we found statistically significantly lower odds of promotion from Class 3 to Class 2 for members of the “Other” racial or ethnic minority group than for whites in fiscal years 2011 through 2018 (model 3). In addition to the contacts named above, Mona Sehgal (Assistant Director), David Hancock (Analyst-in-Charge), Cody Knudsen, Moon Parks, Nisha Rai, Deirdre Sutula, and Melinda Cordero made key contributions to this report. Reid Lowe, Justin Fisher, Nicole Willems, and Chris Keblitis provided technical assistance.", "summary": "USAID has a stated commitment to fostering an inclusive workforce that reflects the diversity of the United States and has undertaken efforts to increase diversity in its Civil and Foreign Services. However, concerns about the demographic composition of USAID's workforce are longstanding. GAO was asked to review issues related to the diversity of USAID's workforce. This report examines, among other things, the demographic composition of USAID's workforce in fiscal years 2002 through 2018, differences between promotion outcomes for racial or ethnic minorities, and the extent to which USAID has identified workforce diversity issues and worked to address those issues. GAO analyzed USAID's personnel data for its full-time, permanent, career workforce for fiscal years 2002 through 2018—the most recent available data. GAO's analyses do not completely explain the reasons for differences in promotion outcomes, which may result from various unobservable factors. Thus, GAO's analyses do not establish a causal relationship between demographic characteristics and promotion outcomes. GAO also reviewed USAID documents and interviewed USAID officials and members of 13 employee groups. The overall proportion of racial or ethnic minorities in the U.S. Agency for International Development's (USAID) full-time, permanent, career workforce increased from 33 to 37 percent from fiscal year 2002 to fiscal year 2018. The direction of change for specific groups varied. For instance, the proportion of Hispanics rose from 3 to 6 percent, while the proportion of African Americans fell from 26 to 21 percent. The proportions of racial or ethnic minorities were generally smaller in higher ranks. During this period, the overall proportion of women increased from 51 to 54 percent, reflecting their growing proportion in USAID's Foreign Service. 8 Promotion outcomes at USAID were generally lower for racial and ethnic minorities than for whites in early to mid career. When controlling for factors such as occupation, GAO found statistically significant odds of promotion in the Civil Service were 31 to 41 percent lower for racial or ethnic minorities than for whites in early and mid career. In the Foreign Service, average promotion rates were lower for racial or ethnic minorities in early to mid career, but differences were generally not statistically significant when GAO controlled for various factors. USAID has previously identified underrepresentation of specific groups in its workforce, but staffing gaps, partly due to a lack of senior leadership attention, prevent the agency from consistently performing required Equal Employment Opportunity (EEO) activities. The Office of Civil Rights and Diversity (OCRD), responsible for USAID's EEO program, has been significantly understaffed. Vacancy rates in most OCRD divisions were 50 percent or higher in November 2019 and, despite attempts to hire more staff, remained at 30 to 50 percent as of April 2020. These staffing gaps have limited OCRD's capacity to process EEO complaints and investigations within mandated timeframes and analyze USAID's demographic data. Staffing gaps also prevented OCRD from submitting required reporting on the status of its EEO program in fiscal year 2018. A lack of consistent leadership in OCRD as well as a lack of senior USAID leadership attention to diversity has contributed to OCRD's staffing gaps. As a result, USAID lacks the capacity to respond to allegations of discrimination, identify potential barriers to equal employment opportunity, and submit required annual reports on the progress of its diversity and inclusion efforts in a timely manner—all of which are required EEO functions. GAO is making four recommendations to USAID, including three to perform required EEO activities and one to demonstrate senior leadership attention to diversity efforts. USAID concurred with the recommendations.", "document_type": "gao"}
{"report": "The Improper Payments Information Act of 2002 (IPIA), as amended by IPERA and the Improper Payments Elimination and Recovery Improvement Act of 2012, requires executive branch agencies, among other things, to (1) review all programs and activities and identify those that may be susceptible to significant improper payments (commonly referred to as conducting a risk assessment), (2) publish improper payment estimates for those programs and activities that the agency identified as being susceptible to significant improper payments, (3) implement corrective actions to reduce improper payments and set reduction targets, and (4) report on the results of addressing the foregoing requirements. IPERA also requires executive agencies’ IGs to annually determine and report on whether their respective agencies complied with six IPERA- related criteria. If an agency does not meet one or more of the six IPERA criteria for any of its programs or activities, the IG considers the agency to be noncompliant overall. The six criteria are as follows: 1. publish a financial report in the form and including all content required by OMB—typically an AFR or a PAR—for the most recent fiscal year, and post that report on the agency website; 2. conduct a program-specific risk assessment, if required, for each program or activity that conforms with IPIA, as amended; 3. publish improper payment estimates for all programs and activities deemed susceptible to significant improper payments; 4. publish corrective action plans for those programs and activities assessed to be susceptible to significant improper payments; 5. publish and meet annual reduction targets for all programs and activities assessed to be at risk for significant improper payments; and 6. report a gross improper payment rate of less than 10 percent for each program and activity for which an improper payment estimate was published. As described above, not all criteria are applicable to every agency. For example, if an agency publishes a financial report and conducts a risk assessment and determines that none of its programs or activities are susceptible to significant improper payments, then the remaining criteria would not be applicable. OMB plays a key role in implementing laws related to improper payment reporting. As required by statute, OMB has established guidance for federal agencies on estimating, reporting, reducing, and recovering improper payments. Such guidance includes OMB Circular A-123 Appendix C, Requirements for Payment Integrity Improvement, which also includes guidance to IGs on determining agency compliance with IPERA. The Council of the Inspectors General on Integrity and Efficiency (CIGIE) also published guidance in July 2019 to assist IGs who are required to conduct an annual improper payment review under IPERA. We continued to report improper payments as a material weakness in internal control in our audit report on the U.S. government’s consolidated financial statements for fiscal years 2018 and 2017 because of the federal government’s inability to determine the full extent to which improper payments occur and reasonably ensure that appropriate actions are taken to reduce them. We have also reported that estimation of improper payments is key to understanding the extent of the problem and to developing effective corrective actions to address it. However, the government’s ability to understand the full scope of its improper payments is hindered by incomplete, unreliable, or understated estimates; risk assessments that may not accurately assess the risk of improper payment; and noncompliance with criteria listed in IPERA. For example, we previously reported that issues and inconsistencies we identified in selected agencies’ processes for estimating improper payments may affect the quality of their estimates. In addition, certain IGs have reported issues with their agencies’ reported improper payment estimates that were caused by insufficient sampling methods and flawed estimation methodologies for calculating and reporting improper payment estimates. Based on agencies that reported improper payment estimates in their AFRs and PARs, government-wide estimated improper payments for fiscal years 2019 and 2018 totaled about $175 billion and $151 billion, respectively. See appendix I for the reported amounts by agency and program for fiscal years 2019 and 2018. As shown in figure 1, of the $175 billion for fiscal year 2019, about $121 billion (approximately 69 percent) is concentrated in three program areas: (1) Medicaid, totaling about $57.4 billion (approximately 32.8 percent); (2) Medicare (comprised of three reported programs: Fee-for-Service (Parts A and B), Advantage (Part C), and Prescription Drug (Part D)), totaling about $46.2 billion (approximately 26.5 percent); and (3) Earned Income Tax Credit (EITC), totaling about $17.4 billion (approximately 9.9 percent). Key information contained in agency AFRs and PARs regarding the types and causes of fiscal year 2019 estimates of improper payments, and reasons for significant changes in reported estimates from fiscal year 2018, are summarized as follows: The $175 billion total reported government-wide estimates for fiscal year 2019 is broken down per OMB’s Paymentaccuracy.gov Data Call Instructions by type as follows: overpayments, totaling about $79.1 billion (approximately 45.2 underpayments, totaling about $12.9 billion (approximately 7.4 unknown, totaling about $74.1 billion (approximately 42.4 technically improper due to statute or regulation, totaling about $8.7 billion (approximately 5 percent). About $74.6 billion (approximately 42.7 percent) of the government- wide estimates was reported as monetary loss. About $151.2 billion (approximately 86.6 percent) of the reported government-wide improper payment estimates for fiscal year 2019 related to root causes that occurred in the three areas below. See appendix II for details on the root causes that agencies identified for their reported improper payment estimates for fiscal year 2019. Insufficient documentation to determine payment accuracy. About $74.1 billion (approximately 42.4 percent) resulted from situations where the agency lacked supporting documentation necessary to verify the accuracy of the payments. Administrative or process error. About $39.1 billion (approximately 22.4 percent) resulted from incorrect data entry, classifying, or processing of applications or payments. Inability to authenticate eligibility. About $38 billion (approximately 21.8 percent) resulted from the agency not being able to authenticate eligibility criteria. The fiscal year 2019 total reported government-wide estimated improper payments, among programs that reported estimates, increased by about $24 billion from the fiscal year 2018 total reported. While decreases in estimated improper payments were reported for several programs, these were offset by increases for certain other programs. Between fiscal years 2018 and 2019, six programs had an increase and five programs had a decrease of over $1 billion in estimated improper payments. Appendix III provides information on all the programs that had a substantial change in estimated improper payments between fiscal years 2018 and 2019 and the reasons for those changes as reported in agency AFRs. Examples of substantial changes in improper payments and the reasons for such changes that agencies provided in their AFRs include the following: Department of Health and Human Services (HHS) reported an increase in the total estimated improper payments for the Medicaid program in excess of $21.1 billion for fiscal year 2019. The majority of the increase in the total estimated improper payments for the Medicaid program was due to HHS’s reintegration of the eligibility component of the Payment Error Rate Measurement (PERM) for Medicaid for fiscal year 2019. From fiscal years 2015 through 2018, HHS did not estimate improper payments attributed to eligibility determinations, but did include a proxy estimate, which was the last reported rate in fiscal year 2014 for the eligibility component, while HHS worked to update this component. For fiscal year 2019, HHS estimated improper payments attributed to eligibility determinations in 17 states (about one-third of all states). HHS’s national eligibility estimated improper payment rate still includes a proxy estimate for 34 remaining states that have not yet been measured since the reintegration of the PERM eligibility component. HHS reported that most eligibility errors identified through the new measurement process were due to insufficient documentation to verify eligibility or noncompliance with eligibility redetermination requirements. HHS also reported that these insufficient documentation situations were related primarily to income or resource verifications. HHS’s fiscal year 2019 AFR noted that another significant cause for estimated Medicaid improper payments is errors resulting from state noncompliance with provider screening and enrollment requirements. The Department of the Treasury (Treasury) began reporting improper payment estimates for fiscal year 2019 for two programs deemed newly susceptible to significant improper payments. Specifically, Treasury reported about $7.2 billion and $2.1 billion in improper payment estimates for Additional Child Tax Credit and American Opportunity Tax Credit, respectively. In addition, HHS reported a decrease in the total estimated improper payments for the Medicare Fee-for-Service (Parts A and B) program of about $2.7 billion. According to HHS’s fiscal year 2019 AFR, the decrease in the estimate is due to a reduction in estimated improper payments for home health; Medicare Fee-for- Service Part B; and Durable Medical Equipment, Prosthetics, Orthotics, and Supplies claims. As stated earlier, the federal government’s ability to understand the full scope of its improper payments is hindered by incomplete, unreliable, or understated agency estimates and risk assessments that may not accurately assess the risk of improper payment. For example, certain federal programs and activities that agencies determined to be at risk for significant improper payments did not report estimates of improper payments for fiscal year 2019, including the Premium Tax Credit and Temporary Assistance for Needy Families programs, and as we previously reported, the Department of Defense (DOD) lacks quality assurance procedures to ensure the completeness and accuracy of the payment populations from which it develops improper payment estimates. Eight years after the implementation of IPERA, half of the 24 CFO Act agencies were compliant with IPERA overall for fiscal year 2018, as reported by their IGs. See appendix IV for each CFO Act agency’s overall compliance with IPERA. With regard to the six IPERA criteria, as shown in figure 2, IGs reported all agencies as compliant with the requirement to conduct program-specific risk assessments if it was applicable to the agency. In addition, 22 of 24 agencies (92 percent) met the requirement to publish a PAR or AFR. Based on the IGs’ fiscal year 2018 compliance reports, agencies were most frequently reported as noncompliant with the IPERA requirement to publish and meet annual targets for improper payment reduction. Out of the 14 agencies for which this requirement was applicable, IGs for eight agencies (57 percent) reported that their agencies were noncompliant. The second most-frequently reported area of noncompliance related to the IPERA requirement for agencies’ reported improper payment rates to be below 10 percent for programs that published estimates. Out of the 15 agencies for which this requirement was applicable, IGs for five agencies (33 percent) reported that their agencies were noncompliant. See appendix IV for additional details on each CFO Act agency’s compliance with the six IPERA criteria for fiscal year 2018, as reported by their IG. In addition, IGs for certain CFO Act agencies reported quality issues in their agencies’ reporting of improper payment data. Although the issues did not result in noncompliance with the related IPERA criterion, the IGs noted these as areas that need improvement. For example, one agency reported inaccurate amounts for identified and recaptured improper payments in its AFR. However, the IG reported that the agency was compliant with the IPERA criterion for publishing financial information in a PAR or AFR. Another agency’s IG reported that its agency did not accurately evaluate its corrective actions’ effectiveness in recapturing improper payments. However, the IG reported that the agency was compliant with the IPERA criterion to publish corrective action plans. As we stated above pertaining to the IGs’ determination of compliance with IPERA criteria, these determinations are based on whether the agency met the requirements and is not a judgment on the quality of the work conducted in order to meet those requirements. As stated above, IGs for 12 of the 24 CFO Act agencies reported that their agencies were compliant with IPERA overall for fiscal year 2018. As shown in figure 3, this is an increase from 10 agencies reported as compliant for fiscal year 2017, and 11 agencies reported as compliant for fiscal year 2016. The improvement in IPERA compliance is attributable to the Departments of Commerce and Education, which were reported by their IGs as noncompliant in fiscal year 2017 but compliant in fiscal year 2018. No agencies that IGs reported as compliant in fiscal year 2017 were reported as noncompliant in fiscal year 2018. In addition, the IGs reported that 21 programs within these agencies were noncompliant with IPERA for each of the past 3 fiscal years (2016–2018). Improper payment estimates for these programs totaled about $78 billion, representing approximately 52 percent of the $151 billion government- wide reported improper payment estimates for fiscal year 2018. As shown in table 1, this includes improper payment estimates for Medicaid of about $36 billion and for EITC of about $18 billion. As shown in figure 4, the number of programs reported as noncompliant with IPERA for 3 or more consecutive years has increased since fiscal year 2016. Specifically, the number of programs reported as noncompliant for 3 or more consecutive years increased from 14 programs in fiscal year 2016 to 18 programs in fiscal year 2017 and 21 programs in fiscal year 2018. The reported improper payment estimates for these programs totaled about $109 billion for fiscal year 2016, $74 billion for fiscal year 2017, and $78 billion for fiscal year 2018. The total improper payment estimates for programs reported as noncompliant for 3 or more consecutive years decreased for fiscal 2017 primarily because the Medicare Fee-for-Service program, with about $41 billion of improper payments in fiscal year 2016, was reported as compliant beginning fiscal year 2017. We provided a draft of this report to OMB and CIGIE for review and comment. CIGIE stated that it had no comments. OMB did not provide any comments. We also provided the full draft for review and comment to agencies and respective IG offices we met with throughout the course of this work. In addition, we sent summary facts to other agencies that had substantial changes in reported improper payment estimates between fiscal years 2018 and 2019 (as shown in app. III), and provided the full draft for review and comment, upon request, to those agencies. We received written comments from the U.S. Agency for International Development, which is reproduced in appendix V. The Department of Health and Human Services, Department of Veterans Affairs, and the Social Security Administration’s Office of Inspector General provided technical comments, which we incorporated in the report as appropriate. The remaining agencies and IG offices informed us that they had no comments. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget, the Chairman of the Council of the Inspectors General on Integrity and Efficiency, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2623 or davisbh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Table 2 details the improper payment estimates and rates that federal agencies reported to the Office of Management and Budget or in their agency financial reports or performance and accountability reports for fiscal years 2019 and 2018. In addition, as shown in the table 2, 17 programs had a substantial change in their reported improper payment estimates or rates between fiscal years 2018 and 2019. The reasons for the changes, as reported in the agency financial reports, are detailed in appendix III. Table 3 shows the government-wide agency-reported improper payment estimates and rates for fiscal year 2019, grouped by Office of Management and Budget (OMB) improper payment root cause categories. OMB defines the root cause categories as follows: Insufficient documentation to determine: For this category, there is a lack of supporting documentation necessary to verify the accuracy of a payment identified in the improper payment testing sample. For example, a program does not have documentation to support a beneficiary’s eligibility for a benefit, and without that particular documentation, the agency is unable to discern that the payment was for the correct amount or went to the right recipient. Administrative or process errors: In this category, errors were caused by incorrect data entry, classifying, or processing of applications or payments. For example, an eligible beneficiary receives a payment that is too high or too low because of a data entry mistake (such as transposing a number) or an agency enters an incorrect invoice amount into its financial system. Inability to authenticate eligibility: In this category, an improper payment is made because the agency is unable to authenticate eligibility criteria. These types of errors include but are not limited to (1) inability to access data and (2) data needed do not exist. Program design or structural issue: For this category, improper payments result from the design of the program or a structural issue. For example, a scenario in which a program has a statutory (or regulatory) requirement to pay benefits when due, regardless of whether all the information has been received to confirm payment accuracy. Medical necessity: For this category, a medical provider delivers a service or item that does not meet coverage requirements for medical necessity (for example, providing a power wheelchair to a patient whose medical record does not support meeting coverage requirements for a power wheelchair). Failure to verify data: In this category, the agency (federal, state, or local), or another party administering federal dollars, fails to verify appropriate data to determine whether a recipient should be receiving a payment, even though such data exist in government or third-party databases. In these situations, the data needed exist, and the agency or other party administrating federal dollars had access to them but did not check the payment against those data prior to making the payment. Other reason: This category covers when the improper payment does not meet any of the above categories. Table 4 shows the 17 programs that had a substantial change in the improper payment estimates or rates between fiscal years 2018 and 2019, and the reasons for those changes, as reported in the agency financial reports. Figure 5 details the Chief Financial Officers Act of 1990 (CFO Act) agencies’ overall compliance with the Improper Payments Elimination and Recovery Act of 2010 (IPERA), as well as the agencies’ compliance with each of the six IPERA criteria for fiscal year 2018, as reported by their inspectors general. Beryl H. Davis, (202) 512-2623 or davisbh@gao.gov In addition to the contact named above, Matt Valenta (Assistant Director), Cherry Vasquez (Auditor in Charge), Pat Frey, Jason Kelly, Jim Kernen, Anne Thomas, Judy Tsan, and Landon Western made key contributions to this report.", "summary": "Improper payments—payments that should not have been made or that were made in incorrect amounts—continue to be an area of fiscal concern in the federal government. Improper payments have been estimated to total almost $1.7 trillion government-wide from fiscal years 2003 through 2019. From fiscal year 2003 through 2016, a government-wide estimate and rate had been included in government-wide financial reports based on the programs and activities that reported estimates. However, financial reports for fiscal years 2017 and 2018 did not include a government-wide improper payment estimate or rate. Agency-reported improper payment estimates are posted on the Office of Management and Budget's Paymentaccuracy.gov website. IPERA requires IGs to annually determine and report on whether executive branch agencies complied with six IPERA criteria, such as conducting risk assessments and publishing and meeting improper payment reduction targets. This report summarizes (1) federal agencies' reported improper payment estimates for fiscal years 2018 and 2019, and reasons for substantial changes between years, and (2) CFO Act agencies compliance with IPERA criteria for fiscal year 2018, as determined by their IGs, and overall compliance trends for fiscal years 2016 through 2018. GAO summarized (1) improper payment estimates from agency financial reports and Paymentaccuracy.gov and (2) information on CFO Act agencies' IPERA compliance reported in IGs' fiscal year 2018 IPERA compliance reports and prior GAO reports. Agency-reported improper payment estimates for fiscal year 2019 totaled about $175 billion, based on improper payment estimates reported by federal programs, an increase from the fiscal year 2018 total of $151 billion. Of the $175 billion, about $121 billion (approximately 69 percent) was concentrated in three program areas: (1) Medicaid, (2) Medicare, and (3) Earned Income Tax Credit. About $74.6 billion (approximately 42.7 percent) of the government-wide estimate was reported as monetary loss, an amount that should not have been paid and in theory should or could be recovered. However, the federal government's ability to understand the full scope of its improper payments is hindered by incomplete, unreliable, or understated agency estimates; risk assessments that may not accurately assess the risk of improper payment; and agencies not complying with reporting and other requirements in the Improper Payments Elimination and Recovery Act of 2010 (IPERA). Eight years after the implementation of IPERA, half of the 24 Chief Financial Officers Act of 1990 (CFO Act) agencies—whose estimates account for over 99 percent of the federal government's reported estimated improper payments—complied with IPERA overall for fiscal year 2018, as reported by their inspectors general (IG). Based on the IGs' fiscal year 2018 compliance reports, agencies were most frequently reported as noncompliant with the requirement to publish and meet annual targets for improper payment reduction. Out of the 14 agencies for which this requirement was applicable, eight agencies were noncompliant. The second most-frequently reported area of noncompliance related to the requirement for agencies' reported improper payment rates to be below 10 percent for programs that published estimates. Out of the 15 agencies for which this requirement was applicable, five agencies were noncompliant. Chief Financial Officers Act of 1990 Agencies' Fiscal Year 2018 Compliance with IPERA Criteria, as Reported by Their IGs The IGs reported that 21 programs were noncompliant with IPERA for each of the past 3 fiscal years (2016–2018). These programs represented about $78 billion, or approximately 52 percent of the $151 billion government-wide reported improper payment estimates for fiscal year 2018.", "document_type": "gao"}
{"report": "DOD began the F-35 development program in October 2001 with plans to produce next-generation aircraft to replace aging aircraft in the military services’ inventories. Figure 1 shows the F-35 in flight. The program has developed and is delivering three variants of the F-35 aircraft; the F-35A conventional takeoff and landing variant for the Air Force, the F-35B short takeoff and vertical landing variant for the Marine Corps, and the F-35C carrier-suitable variant for the Navy. The characteristics of the services’ variants are similar, but each service’s variant also has unique operating requirements. For example, the Marine Corps requires that the F-35B be capable of operating from aircraft carriers, amphibious ships, and main and austere operating bases alike, requiring the ability to conduct short take offs and vertical landings. In March 2005, we found that the F-35 program had started development without adequate knowledge of the aircraft’s critical technologies or a solid design. Further, DOD’s acquisition strategy called for high levels of concurrency between development and production, which runs counter to best practices for major defense acquisition programs. In our prior work, we identified the F-35 program’s lack of adequate knowledge and high levels of concurrency as the major drivers of the program’s significant cost and schedule growth, and other performance shortfalls. Since the development program began in 2001, it has been restructured three times with revised cost and schedule estimates. The most recent restructuring was initiated in 2010 when the program’s cost estimates exceeded certain thresholds established by statute—a condition known as a critical Nunn-McCurdy breach. DOD subsequently certified to Congress in June 2010 that the program was essential to national security and needed to continue. DOD then established a new acquisition program baseline in 2012 that added $162.7 billion to the program’s cost estimate and extended the original delivery schedule by 5-6 years. Since then, the program’s cost and schedule estimates, as well as the expected number of aircraft to be delivered, have remained relatively stable, as shown in table 1. Of the F-35’s $406 billion estimated acquisition cost, DOD needs a majority of the funding ($270.3 billion) to purchase aircraft over the next 26 years. Of that future funding, the program plans to spend between $9.6 billion and $14 billion each year through fiscal year 2031. In addition, the program’s sustainment costs to operate and maintain the F-35 fleet over the next 52 years are estimated to be $1.12 trillion. Though the program’s total planned quantities have been relatively stable, the program’s timeframes for procuring these aircraft have changed multiple times. Since the start of development, the program has pushed the procurement of more than half of the total aircraft planned into the future, mostly due to significant concurrency between development and production. Specifically, the program office had originally planned to procure almost 2,000 aircraft by fiscal year 2019. However, according to the current plan, by the end of 2019, the program will have procured just over 500 aircraft. The F-35 baseline aircraft development program was complete in April 2018, when developmental testing concluded. As we reported in June 2018, the program office reported it had met all nine of its capability thresholds—or the minimum acceptable value for each capability—and delivered three of those nine capabilities. However, we also reported that the program has to complete operational testing before DOD can determine if the six remaining capabilities have been delivered. The purpose of operational testing is to assess the effectiveness, suitability, survivability, lethality, and mission capability of the F-35, including the information systems and the air vehicle, in an operationally representative environment. Operational testing includes cyber security assessments, some of which have been conducted. Production of the F-35 began in 2007 while development was in its early stages and before developmental flight testing had started. As a result of this concurrent development, the 357 aircraft delivered through 2018 will need retrofits to fix deficiencies and design issues found during testing. The program’s total estimated cost of concurrency is $1.4 billion. The program office plans for over 500 aircraft to be procured by the time operational testing is completed. Until operational testing is complete, there is a risk that additional problems with the aircraft may be identified. As a result, the concurrency costs of retrofitting delivered aircraft could increase. The F-35 program started formal operational testing in December 2018 after a 3-month delay. This testing was delayed for two main reasons: (1) to resolve critical deficiencies and (2) to accommodate an unexpected grounding following the crash of an F-35B in September 2018. According to a test official, the program expects to complete testing in December 2019, about three months later than planned due to delays with the simulator that is used for more complex testing. Figure 2 shows the program’s planned end to developmental testing and planned timeframes for operational testing for 2012 and the past four years and the delays the program has realized each year since the program was re-baselined in 2012. The operational testing was delayed for the following two main reasons. Resolution of deficiencies: First, before the program could begin operational testing, it had to resolve critical deficiencies with the aircraft that were identified during development testing. The program categorizes deficiencies according to their potential impact on the aircraft’s performance. Category 1 deficiencies are considered critical and could jeopardize safety, security, or another requirement. Category 2 deficiencies are those that could impede or constrain successful mission accomplishment. In January 2018, the F-35 program had 966 open deficiencies—111 category 1 and 855 category 2. At that time, the program planned to move forward before resolving all of them. In June 2018, we recommended that the Secretary of Defense direct the F-35 program to resolve all these deficiencies before the program’s October 2019 full-rate production decision. According to DOD officials, over the past year, the program has made progress in reducing the number of open deficiencies by resolving, re-categorizing, closing, or combining them. For example, in 2018, the program resolved nearly 50 category 1 deficiencies and re- categorized over 50 others to category 2. As a result, the program received approval from the Under Secretary of Defense for Acquisition and Sustainment to begin formal operational testing with 13 category 1 deficiencies and almost 900 category 2 deficiencies. According to the Program Executive Officer, none of the open category 1 deficiencies are a safety of flight concern, and all of them have operational workarounds. A current example of an open category 1 deficiency is with lines on the F- 35’s landing gear, which can rupture when a tire blows, potentially causing loss of a major aircraft system such as the brakes. Such an event requires some repair work to the landing gear, but contractor officials explained that it is not a safety concern. According to the program office, it is not a safety concern because the current workaround for this deficiency is pilot training to avoid braking on the side of the blown tire. Program test officials said that testing with deficiencies is not uncommon and they will continue to work to address them, but some may not be fully resolved for several years. Unexpected grounding: In October 2018, the F-35 fleet was grounded after the program identified a manufacturing fault with an engine fuel tube—a component in the F-35 engine produced by Pratt & Whitney. The fault was found in an inspection that stemmed from an F-35B crash in September 2018. This was the first crash of an F-35. Of the 23 operational test aircraft, the program replaced the fuel tubes on 18 aircraft by the start of operational testing in December 2018, which contributed to the 3-month delay. This and other key technical risks are described in more detail in appendix III. In addition to starting operational testing and the unexpected grounding, the program and the airframe contractor Lockheed Martin experienced other major events over the past year, as shown in figure 3. For example, the United States completed its first F-35 combat mission in September 2018 when an F-35B successfully hit a target in Afghanistan. The program took steps to mitigate delays to the start of operational testing. For example, the program office, in coordination with DOT&E, received approval to conduct some preoperational testing events starting in January 2018, before the official start date in December. According to DOT&E officials, the outcome of these preoperational test events should count towards the completion of operational testing. This included cold weather testing in Alaska, which took advantage of appropriate weather conditions. Despite the 3-month delay, program officials stated that they consider the F-35 operational test schedule to be adequate for addressing schedule risks, which pertain to unresolved deficiencies and potential problems with the availability of test and support aircraft, ground systems, test ranges, and necessary test models and simulations. According to a test official, as of April 2019, some of these risks have been realized, such as the delay with the simulator, and as a result, the end of operational testing is now planned for December 2019. In addition, there is the possibility of new deficiencies emerging from operational testing. Unresolved deficiencies: Existing or new deficiencies could negatively affect test results. According to DOT&E officials, since the start of operational testing, four new category 1 deficiencies have been identified, bringing the total to 17. According to DOD officials, it would not be unexpected during the course of operational testing for the program to discover additional deficiencies that may require resolution and re-testing. Availability of test and support aircraft: According to test officials, F-16s and F-18s are needed to represent adversaries during F-35 operational tests. These assets may not be available because they also support other test programs. According to officials, the F-35 program does not have control over the availability of these aircraft and must work with the Navy and Air Force to negotiate their use. In addition, the limited availability of F-35 test aircraft, in part due to R&M issues and shortages of replacement parts, may also pose a challenge to completing test events, according to officials. Availability of ground systems: Ground systems required for operational testing, such as the DOT&E developed Radar Signal Emulators, are late in development and may not be available when required. According to DOT&E officials, the emulators imitate modern threat radar capabilities of adversarial nations but their integration with the test range is approximately a year behind schedule. The program is currently using other threat simulators. DOT&E officials stated that they are working to have the radars ready by the spring of 2019, when needed. Availability of test ranges: Test officials at Edwards Air Force Base expressed concern about the availability of test ranges, which the F- 35 program shares with other programs. According to test officials, the F-35 was the fifth in line, in terms of priority, to use the range at Edwards Air Force Base, as of October 2018. DOT&E officials, however, stated that they did not observe any range availability issues during the F-35’s first month of operational testing. Availability of test models and simulations: According to program officials, the program’s testing simulator, which runs the F-35’s mission systems software and provides test scenarios that cannot be replicated in a real-world environment, will not be complete until at least November 2019. Completion of the testing simulator was originally scheduled for the end of 2017. Any additional delays in operational testing could affect another upcoming program decision: DOD’s decision to begin full-rate production in December 2019. This decision is typically made after operational testing is completed. The F-35 program has made slow, sustained progress in improving the F- 35’s R&M. R&M determines the likelihood that the aircraft will be in maintenance rather than available for operations. Each F-35 aircraft variant is measured against eight R&M metrics, four of which are in part of the contract. All F-35 variants are generally performing near or above targets for half of the R&M metrics while the other four are still falling short, which is the same as last year. While the program is on track to meet the targets for half of the metrics, the program has not taken adequate steps to ensure the targets for the others will be met. While DOD has an action plan to improve R&M, its guidance does not define specific, measurable objectives for what the desired goals for the F-35’s R&M performance should be. Furthermore, the program office has not prioritized funding for projects that will improve the R&M metrics that are not meeting their targets. All F-35 variants are measured against eight R&M metrics’ targeted performance levels, and all variants are generally performing near or above targets for four of the eight R&M metrics. This represents little change from their overall performance last year. All eight R&M metrics are described in the program’s Operational Requirements Document (ORD)—the document that outlines the requirements DOD and the military services agreed the F-35 should meet. However, in December 2018, DOT&E reported that, although performance for the four under- performing metrics has shown slow growth over the years, none of these metrics were meeting interim goals needed to reach requirements at each variant’s maturity. Each F-35 variants’ R&M performance against these metrics’ targets is shown in table 2. Since the program began tracking R&M performance in 2009, the program has seen small, annual improvements. Over the past year, all variants showed a slight improvement in targeted performance levels for one metric, the mean flight hours between failures (design controlled), but saw little or no discernable improvement for the four metrics not meeting targets. However, based on current performance, the program does not expect to meet those targets by full aircraft maturity. According to F-35 program officials, the ORD R&M metrics should be re-evaluated to determine more realistic R&M performance metrics, but they have not yet taken actions to do so. Until it does so, the program office remains accountable for ensuring those ORD R&M metrics are achieved. In June 2018, we recommended that the F-35 program identify what steps it needs to take to ensure the F-35 aircraft meet R&M requirements before each variant reaches maturity and update its R&M Improvement Program (RMIP)—DOD’s action plan for prioritizing and funding R&M improvement projects—with these steps. DOD concurred with our recommendation but has yet to take substantive actions to address it. It did, however, complete 16 improvement projects since we last reported on this. Despite completing these projects, there were not significant gains in the R&M metrics not meeting targets. Program officials advised, however, that measurable improvements in R&M can take time to manifest. To speed this process, the program is accelerating planned upgrades to older aircraft where appropriate, which officials stated should translate to an overall improvement in the program’s R&M performance. The F-35 program office has estimated that implementing all of the identified improvement projects currently contained in its RMIP could result in potential life cycle cost savings of over $9.2 billion by improving the F-35’s R&M. As of December 2018, the guidance the F-35 program office has used to implement the RMIP does not define specific, measurable objectives for what the desired goals for the F-35’s R&M performance should be or align improvement projects with R&M goals. Furthermore, the RMIP has not been a funding priority. Federal internal control standards state that programs should define objectives when implementing programs such as the RMIP. Although the F-35 program RMIP’s guidance has a general goal of improving R&M, it does not identify achieving the targets for the eight R&M metrics the program tracks as an objective. Program officials acknowledged that the RMIP’s guidance does not include such an objective. Instead, officials are using the RMIP to prioritize and fund projects that will improve aircraft availability and mission capability—neither of which are included in the eight R&M metrics, but are necessary and important initiatives. Officials stated that by prioritizing these projects, they will eventually improve performance under all R&M metrics, including the four that are not meeting targets. The RMIP’s guidance, however, does not discuss these priorities or align improvement projects with the eight R&M metrics. In our prior work on weapon system acquisitions, we have identified a number of best practices for improving program outcomes, such as clearly establishing well-defined requirements and securing stable funding that matches resources to requirements. The F-35 program office has not prioritized or dedicated funding in its budget to improve R&M in part because program officials explained that they have been focused on initiatives intended to lower the cost of the aircraft. Further, any current funding for R&M improvement projects comes from the program’s operation and maintenance funds, which are only available for one fiscal year. Officials further explained that, if such funding runs out or is used by the program for other efforts, then R&M projects will go unfunded or be suspended until new funding is available. In fiscal year 2018, for example, while some R&M improvement projects were completed, several other projects were suspended when that year’s funding ran out. According to officials, these projects may not be started back up until fiscal year 2019. In addition, most of the R&M improvement projects that were approved in fiscal year 2018 were not funded. For example, as of December 2018, according to a contractor representative, all of the identified improvement projects currently unfunded in the program’s RMIP would cost about $30 million to implement, but are on hold and waiting to be funded. Program officials stated that they are in the process of revising the RMIP and have considered including more specific objectives, such as a focus on improving aircraft availability and mission capability and a focus on improving R&M performance where the ORD R&M metrics’ targets are not being met. Additionally, in its 2019 annual lifecycle sustainment plan, the program office noted that a dedicated annual budget for R&M improvement projects would benefit the program. According to the program, any revisions to the RMIP and changes to how it will be funded, however, will not be complete until April 2019 or later. Without defining measurable objectives in its RMIP guidance for meeting all eight R&M metrics and aligning which improvement projects will ensure those metrics are met, the program is at risk of not fully meeting its R&M goals. Further, without prioritizing funding for improving R&M, projects may continue to be either prematurely suspended or never get underway. As a result, the warfighter may accept aircraft that (1) are less reliable than originally described in the program’s ORD, and (2) have operation and sustainment costs that may raise affordability questions. With development of the baseline program complete, the program is transitioning to early development and testing for modernization efforts known as Block 4, which are expected to cost about $10.5 billion. The F- 35 program plans to award Block 4 development contracts starting in May 2019, before completing a business case—a baseline cost and schedule estimate to track the program’s performance going forward. In doing so, the program will commit resources without adequate knowledge of Block 4’s full cost, schedule, and level of technology maturity, putting Block 4 at risk of experiencing cost and schedule overruns similar to those experienced by the baseline program during its development. The National Defense Authorization Act for Fiscal Year 2017 required DOD to submit a report containing certain elements of an acquisition program baseline—in essence, a full program business case—to include the cost, schedule, and performance information for Block 4. In 2018, we found that DOD’s report to Congress was incomplete but included information on some elements of the Block 4 acquisition program baseline. In its report, DOD stated that the acquisition program baseline would continue to be refined over the next year. As a result, we presented a matter for congressional consideration to restrict Block 4 funding until the program established a complete business case. DOD’s report to Congress also outlined the F-35 program office’s new development approach to deliver Block 4 capabilities—new requirements beyond the baseline aircraft capabilities to address evolving threats. As we reported in June 2018, this new approach, meant to deliver capabilities to the warfighter faster, is referred to as Continuous Capability Development and Delivery (C2D2). This approach consists of 6-month development cycles in which small groups of capabilities will be developed, tested, and delivered as they are matured. In January 2018, the F-35 program started using this C2D2 approach to develop and test software updates to address deficiencies identified during testing. According to the contractor, the first two software updates also established a foundation for new Block 4 capabilities to be fully developed later. According to program officials, as of December 2018, the program has executed contract actions valued over $1.4 billion to establish testing facilities and support early Block 4 development of capabilities the program plans to deliver through 2024. According to DOD’s January report, results from this work will help the program inform its Block 4 business case. The F-35 program plans to award Block 4 development contracts without knowledge of the effort’s full cost or the maturity of critical technologies. Over the past year, the program has been working to complete its business case for Block 4, including incorporating Block 4 activities into its acquisition strategy—which was approved in October 2018. However, three key Block 4 business case documents will not be ready before the program’s planned May 2019 contract awards for development efforts. Independent technology readiness assessment: Although the contracts for Block 4 development efforts are planned to be awarded in May 2019, the program will not conduct an independent technology readiness assessment by that time. A technology readiness assessment is a systematic, evidence-based process that evaluates the maturity of hardware and software technologies critical to the performance of a larger system or the fulfillment of the key objectives of an acquisition program. According to a program official, the program will conduct its own assessments on a rolling basis as initial capabilities are developed. The official stated that technologies will not be integrated into the aircraft until they are adequately mature. The program office plans to conduct a partial assessment of initial capabilities sometime between October and December 2019 with additional assessments to follow. However, without an independent technology readiness assessment, the program has not identified potential critical technology elements and as a result, may be at risk of delaying the delivery of new capabilities. Test and evaluation master plan: Although the F-35 program has begun testing Block 4 capabilities, it does not have an approved test and evaluation master plan. The test and evaluation master plan documents the overall structure, strategy, and objectives of the test program as well as the associated resources needed for execution. It provides a framework for the program office to provide detailed test plans and subsequently determine the resources needed. Test officials have expressed concerns about the lack of an approved test plan, uncertain funding, the number of test aircraft available, and the draft test schedule, among other things. Officials were also concerned as to whether the Block 4 test aircraft would be in the same configuration as fielded aircraft, which are in earlier configurations than the test fleet. Further, DOT&E stated in its annual report that it considers the current Block 4 schedule to be high risk due to the large amount of planned capabilities that will be developed and tested in 6- month development cycles. An approved, properly resourced test plan is essential for planning and preparing for adequate testing of the Block 4 capabilities. Without an approved test and evaluation master plan, the F-35 program is providing the test authorities with capabilities to be tested without giving them the necessary direction on how to adequately prepare to conduct the tests. Specifically, test officials stated the F-35 program office has not provided details on which capabilities are planned for each testing development cycle making it difficult to execute testing. While this is still a concern, F-35 program officials explained that over the past 3 months they have been providing the test authorities with the direction needed to conduct testing. Independent cost estimate: The Block 4 independent cost estimate, which details the program’s total estimated life cycle cost, is not complete. In August 2017, we reported that DOD estimated the development funding needed for the first phase of modernization for Block 4 to be over $3.9 billion through 2022. Since then, the program incorporated more scope and fidelity into the Block 4 cost estimate, which has increased to $10.5 billion for Block 4 capabilities planned through 2024. The program office has provided its Block 4 cost estimate to the Cost Assessment and Program Evaluation office (CAPE) for an independent cost estimate. According to CAPE officials, they will provide the independent cost estimate between October and December 2019 to support the program’s full-rate production decision, but this would occur several months after the program plans to award the Block 4 development contracts. Without an independent cost estimate, Congress does not have insight into the full potential cost of the Block 4 effort. The expected completion dates for these documents are between October and December 2019, at the earliest. Figure 4 shows key Block 4 dates, the planned development contract awards, and planned completion dates for the remaining business case documents. Major defense acquisition programs generally follow DOD acquisition policy, which states that prior to the release of a development contract request for proposal, program officials should have confidence that program requirements are firm. Program officials should also clearly state that the risk of committing to development has been reduced or will be adequately reduced prior to contract award. According to best practices identified by GAO, without several of the business case documents completed, program officials cannot have a high level of confidence that the requirements are firm and that the risk to committing an estimated $10.5 billion in funding to Block 4 has been adequately reduced. According to program officials, business case documents have not been completed because they took a step back to re-examine their approach and the cost estimate for Block 4 that DOD established in 2017. Counter to acquisition best practices, the program plans to initiate additional development work before they acquire the requisite knowledge of the necessary levels of technology maturity and funding. Program officials have reported the planned modernization contracting efforts shown in table 3. If program officials move ahead with awarding Block 4 contracts without gaining the knowledge that a full business case could would provide, Block 4 modernization efforts will be at risk of experiencing the same kind of cost and schedule growth the baseline development program experienced. With a few exceptions, the negotiated prices for all F-35 variants have generally been decreasing with each production lot, and more aircraft are being procured in each lot. In particular, the F-35A’s price has decreased in each subsequent production lot, with the most recent price per aircraft at $89 million in lot 11, as shown in figure 5 below. In 2018, we reported that while the F-35 program faces affordability challenges, it was investing in several projects to reduce production and sustainment costs. According to DOD, to improve production affordability, the F-35 program office is continuing to make investments to lower the price of an F-35A to below $80 million by lot 13. To realize this goal, the F-35 program office and the prime contractor are increasing the production rate and investing in various initiatives to lower production costs. For example: According to the program office, it has invested a total of $320.3 million in efforts to improve manufacturing processes that it estimates could result in up to $7.9 billion in savings over the life of the program. In addition, the prime contractor has invested $90 million and plans to invest an additional $25 million to lower its production costs. DOD issued a contract announcement for economic order quantity purchases for use in production lots 13-14. This approach involves making large purchases of components that will be used across multiple procurement lots of aircraft to reduce production costs by buying components in bulk and achieving economies of scale. The program had expected $1.2 billion in cost savings from this effort, but according to estimates from the CAPE, cost savings will more likely be $595 million. In addition, according to program officials, once the program achieves full-rate production, it plans to utilize a multi-year procurement strategy, beginning in fiscal year 2021. This strategy is intended to have similar benefits as the economic order quantity purchases by providing industry with a stable, long-term demand. According to Pratt & Whitney, the cost of the engine is also declining. For example, the price of the F-35A and C engine dropped by $100,000 per engine over the past year. The most recent negotiated price is $11.9 million per engine. The F-35 airframe and engine contractors saw a significant increase in their production rates in 2018, but faced some production challenges as well. The airframe contractor—Lockheed Martin—increased its production rate by 50 percent and delivered a total of 91 aircraft in 2018, with a total of 267 aircraft on its production floor or in contract negotiations as of December 2018, as shown in figure 6. In addition, Lockheed Martin delivered more aircraft on time. In 2012, none of the planned aircraft deliveries were on time whereas in 2018, 58 percent were on time. To incentivize the contractor to improve on-time deliveries, the program office has added a performance incentive fee to the lot 11 production contract. Table 4 shows some improvements in Lockheed Martin’s production metrics since 2012 and over the past 2 years. Between 2012 and 2017, Lockheed Martin saw some improvement for all variants’ production metrics, with the F-35A showing improvements through 2018. However, over the past year, several metrics for the F- 35Bs and F-35Cs saw a decline. According to Lockheed Martin, it faced several challenges with the increased production rate which led to these declines. For example, since January 2018, the contractor hired around 900 new personnel, nearly 30 percent of its workforce, all of whom needed training. According to officials, this influx of new personnel led to an increase in the average labor hours for the F-35C and the number of hours required for scrap, rework, and repair of the F-35B and F-35C. According to the contractor, as the newly hired personnel gain more experience in the production processes, the average labor hours it takes to build an F-35C should start decreasing again. The contractor faced several production quality issues and parts delays, which it worked to address over the past year. For instance, we reported last year that due to a fault in the production process, Lockheed Martin halted deliveries after the Air Force identified corrosion between the aircraft’s surface panels and the airframe because Lockheed Martin did not apply primer when the panels were attached. The program office stated that Lockheed Martin and the F- 35 Program Executive Officer reached a mutual agreement on the cost to resolve this issue, the details of which have not been disclosed publicly. With the production rate increase, the supply chain was strained to deliver parts on time, which led to increases in material shortages for key components, such as the radar. Pratt & Whitney has also increased production over the past year and has shown similar manufacturing performance for the F-35 engine as in past years; however, it had fewer on-time deliveries in 2018 due to the challenges it faced, including an increase in the average number of quality issues per engine. Pratt & Whitney’s production rate increased by 10 percent over the past year, with 81 engines delivered in 2018. Table 5 shows the trends in Pratt & Whitney production metrics’ performance. According to Pratt & Whitney, its late engine deliveries increased in 2018 partially due to a subcontractor that did not have all of the needed tooling in place to produce more F-35B engines. To address this and other issues causing the late deliveries, Pratt & Whitney is taking lessons learned from its other production facilities and applying them to the F-35’s engine production. The F-35 program has overcome significant hurdles in its 18 years of development of the baseline aircraft, which was completed last year. One recent hurdle that it overcame was resolving many critical deficiencies found during developmental testing, which allowed the program to begin operational testing this past December. Other hurdles remain, including with the F-35’s reliability and maintainability (R&M). Four of the eight R&M metrics continue to fall short of meeting performance targets. Program officials stated that the Operational Requirements Document (ORD) R&M targets need to be re-evaluated to determine more realistic R&M performance metrics but have not yet taken actions to do so. Until the program re-evaluates the targets, it is accountable for achieving those requirements. Furthermore, funding improvement efforts have not been a priority for the program. As a result, over the past year, some projects were started, several were halted while underway, and others are on hold, waiting for funding. As long as targets under all of the R&M metrics continue to fall short, the U.S. military services and the taxpayer will have to settle for aircraft that are less reliable and more costly to maintain than originally planned. Also, with continuing concerns about the program’s long-term affordability, the program is missing a prime opportunity to infuse affordability into the aircraft’s future with better R&M performance. As the program is considering revisions to its R&M Improvement Program (RMIP), it is in a good position to clearly define and communicate its R&M objectives for the aircraft to meet the targets under all of its eight R&M metrics. Until it does so, the program office will not know whether the steps it is taking now are sufficient to ensure each F-35 variant achieves its R&M requirements in the future. As we have reported in the past, the F-35 program started its development before it was ready. It is now at risk of doing the same thing with the Block 4 modernization effort. Since we last reported in June 2018, the program has still not established a solid business case to commit funding and other resources to developing new capabilities for the aircraft. This could result in the program delivering technologies late and over cost estimates. Finally, the program has committed a significant amount of funding to support Block 4, but it has not completed an independent cost estimate of the life-cycle cost. Consequently, Block 4 may follow in the footsteps of the F-35’s baseline program which saw significant cost and schedule growth during its development. This approach leaves the F-35 program, DOD, Congress, and the U.S. military services without key information to make decisions regarding Block 4. We are making the following five recommendations to the Department of Defense: The Secretary of Defense should ensure that the F-35 program office assesses whether the ORD R&M targets are still feasible and revise the ORD accordingly. (Recommendation 1) The Secretary of Defense should ensure that the F-35 program office, as it revises its RMIP, identifies specific and measurable R&M objectives in its RMIP guidance. (Recommendation 2) The Secretary of Defense should ensure that the F-35 program office, as it revises its RMIP, identifies and documents which RMIP projects will achieve the identified objectives of the RMIP guidance. (Recommendation 3) The Secretary of Defense should ensure that the F-35 program office prioritizes funding for the RMIP. (Recommendation 4) The Secretary of Defense should ensure that the F-35 program office completes its business case, at least for the initial Block 4 capabilities under development, before initiating additional development work, to include: an independent cost estimate; an approved test and evaluation master plan which addresses resources, aircraft shortfalls, and funding; and an independent technology readiness assessment. (Recommendation 5) We provided a draft of this report to DOD for review and comment. Our initial draft report contained only recommendations 2 through 5 above. During the comment period, DOD officials provided additional information about the program’s R&M performance concerning whether the ORD targets continue to be feasible and should be re-examined. As a result, we added our first recommendation above—that the F-35 program office assess whether the ORD R&M targets are still feasible and revise the ORD accordingly. DOD provided written comments on our report, which are reprinted in appendix IV. DOD concurred with our four recommendations on R&M but did not concur with our last recommendation on the Block 4 modernization. DOD also provided technical comments, which were incorporated as appropriate. In concurring with our four R&M recommendations, DOD stated that it would review its R&M requirements and possibly revise them, update its RMIP guidance, and plan for R&M funding going forward. DOD officials did not concur with our recommendation that the F-35 program office complete its business case before initiating additional development work. DOD stated that the F-35 program office has adequate cost, schedule, and technical maturity knowledge to begin the development of initial Block 4 capabilities. DOD also outlined when some of the remaining Block 4 business case documents would be complete. As we stated in our report, these documents will not be complete until after the contracts to initiate additional Block 4 development work will be awarded. We maintain that completing its business case before initiating additional development work would put DOD and the program in a better position to effectively and successfully develop Block 4 capabilities. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense for Acquisition and Sustainment, the Secretary of the Air Force, the Secretary of the Navy, and the Commandant of the Marine Corps. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Key program event Start of system development and demonstration approved. Primary GAO conclusions/recommendations Critical technologies needed for key aircraft performance elements are not mature. We recommended that the program should delay start of system development until critical technologies are matured to acceptable levels. DOD response and actions DOD did not concur with our recommendation. DOD did not delay the start of system development and demonstration stating technologies were at acceptable maturity levels and that it will manage risks in development. Program sets in motion plan to enter production in 2007 shortly after first flight of the non-production representative aircraft. The program was entering production with less than 1 percent of testing complete. We recommended that the program delay investing in production until flight testing shows that the Joint Strike Fighter performs as expected. DOD partially concurred but did not delay start of production because it believed the risk level was appropriate. The program was restructured to reflect findings from a recent independent cost team and independent manufacturing review team. As a result, development funds increased, test aircraft were added, the schedule was extended, and the early production rate decreased. Costs and schedule delays inhibited the program’s ability to meet needs on time. We recommended that the program complete a comprehensive cost estimate and assess warfighter and initial operational capability requirements. We suggested that Congress require DOD to tie annual procurement requests to demonstrated progress. DOD continued restructuring, increasing test resources, and lowering the production rate. Independent review teams evaluated aircraft and engine manufacturing processes. Cost increases later resulted in a Nunn- McCurdy breach. Military services completed the review of capability requirements, as we recommended. The program incorporated positive and more realistic restructuring actions taken since 2010, including more time and funding for development and deferred procurement of more than 400 aircraft to future years. The program was moving in the right direction but needed to fully validate design and operational performance and at the same time make the system affordable. We did not make recommendations to DOD in this report. DOD agreed with GAO’s observations. Key program event The services established initial operational capabilities dates in 2013. The Marine Corps and Air Force planned to field initial operational capabilities in 2015 and 2016, respectively, and the Navy planned to field its initial capability in 2018. Primary GAO conclusions/recommendations Delays in developmental flight testing of the F-35’s critical software may hinder delivery of the warfighting capabilities to the military services. We recommended that DOD conduct an assessment of the specific capabilities that can be delivered and those that will not likely be delivered to each of the services by their established initial operational capability dates. DOD response and actions DOD concurred with our recommendation. On June 22, 2015, the Under Secretary of Defense for Acquisition, Technology, and Logistics issued a Joint Strike Fighter software development report, which met the intent of GAO’s recommendation. DOD planned to begin what it refers to as a block buy contracting approach that was anticipated to provide cost savings. In addition, DOD planned to manage the follow-on modernization program under the current F-35 program baseline and not as its own separate major defense acquisition program. The terms and conditions of the planned block buy and managing follow-on modernization under the current baseline could present oversight challenges for Congress. We recommended that the Secretary of Defense hold a milestone B review and manage follow-on modernization as a separate major defense acquisition program. DOD did not concur with our recommendation. DOD viewed modernization as a continuation of the existing program and the existing oversight mechanisms, including regularly scheduled high-level acquisition reviews, would be used to manage the effort. The DOD F-35 program office was considering contracts for economic order quantity of 2 years’ worth of aircraft parts followed by a separate annual contract for procurement of lot-12 aircraft with annual options for lot-13 and lot-14 aircraft. However, as of January 2017, contractors stated they were still negotiating the terms of this contract; therefore, the specific costs and benefits remained uncertain. Program officials projected that the program would only need $576.2 million in fiscal year 2018 to complete baseline development. At the same time, program officials expected that more than $1.2 billion could be needed to commit to Block 4 and economic order quantity in fiscal year 2018. GAO recommended DOD use historical data to reassess the cost of completing development of Block 3F, complete Block 3F testing before soliciting contractor proposals for Block 4 development, and identify for Congress the cost and benefits associated with procuring economic order quantities of parts. DOD did not concur with the first two recommendations and partially concurred with the third while stating that it had finalized the details of DOD and contractor investments associated with an economic order quantity purchase and would brief Congress on the details, including costs and benefits of the finalized economic order quantity approach. Key program event The program office determined that it could not resolve all open deficiencies found in developmental testing within the development program, and they would need to be resolved through post-development contract actions. DOD provided a report to Congress outlining preliminary plans to modernize the F-35. It stated it planned to develop a full acquisition program baseline for the modernization effort in 2018 and provide a report to Congress by March 2019. Primary GAO conclusions/recommendations The program office plans to resolve a number of critical deficiencies after full-rate production. We recommended that the F-35 program office resolve all critical deficiencies before making a full- rate production decision, and identify steps needed to ensure the F-35 meets reliability and maintainability requirements before each variant reaches maturity. We also suggested that Congress consider providing in future appropriations that no funds shall be available for obligation for F-35 Block 4 until DOD provides a report setting forth its complete acquisition program baseline for the Block 4 effort to the congressional defense committees. DOD response and actions DOD concurred with both recommendations and identified actions that it would take in response. The National Defense Authorization Act for fiscal year 2019 included a provision limiting DOD from obligating or expending more than 75 percent of the appropriations authorized under the Act for the F-35 continuous capability development and delivery program until 15 days after the Secretary of Defense submits to the congressional defense committees a detailed cost estimate and baseline schedule. The National Defense Authorization Act for fiscal year 2015 included a provision for GAO to review the F-35 acquisition program annually until the program reaches full-rate production. This is the fourth report under that provision. In this report, we (1) provide information on the program’s progress toward completing testing of the baseline aircraft; (2) assess the aircraft’s current reliability and maintainability (R&M) status; (3) assess the program’s modernization efforts (to add new aircraft capabilities), known as Block 4; and (4) provide information on the program’s production costs and efficiency initiatives. To provide information on progress in the F-35’s development, we reviewed the program’s costs, schedule, and performance plans and compared the actual progress in each area with the goals established in its 2012 baseline to identify any significant trends. We also reviewed the F-35’s selected acquisition report and its fiscal year 2019 budget request. To assess progress in testing, we reviewed test results and associated reports, program briefings, and internal DOD program analyses. We interviewed officials from the program office, military test authorities, and contractors—Lockheed Martin (airframe) and Pratt & Whitney (engine)— on key aspects of F-35 development progress, including flight testing, future test plans, and recent findings from test events. We also interviewed the Director, Operational Test and Evaluation office and F-35 program developmental and operational test pilots. To assess the program’s progress in achieving its R&M targets, we obtained and analyzed its monthly reports on R&M performance from January 2018 through December 2018. We compared these to the program’s R&M targets documented in the F-35 Operational Requirements Document and the Joint Contract Specification. We examined program data for the metrics’ performance across 12 months to identify any trends. We assessed the reliability of this data by reviewing supporting documentation and interviewing program office officials who track reliability metrics and other knowledgeable DOD officials. We also reviewed the program’s Reliability and Maintainability Improvement Program’s guidance to determine if it contained specific and measurable objectives and the projects needed to meet those objectives. We determined that the R&M metric data were sufficiently reliable for our purposes of determining whether the program will meet its targets. To assess the program’s Block 4 modernization plans, we reviewed documents that GAO best practices identify should be completed prior to awarding a development contract. We interviewed DOD and program office officials, and contractor representatives regarding the program’s Block 4 activities to date and future plans. We compared the program’s accomplishments over the past year and its future plans to the product development best practices identified by GAO. We reviewed the fiscal year 2019 budget request to identify costs associated with the Block 4 effort. We obtained contract documents for Block 4 activities between March 2014 and December 2018 to determine the total amount of funding that has been obligated to Block 4 and the scope of work that has been contracted. To provide information on ongoing manufacturing performance and the program’s plans to achieve full rate production, we obtained and analyzed the prime contractor’s production metrics and its aircraft delivery rates and from 2012 through 2018. We compared this performance to the program’s procurement plans from its selected acquisition reports since 2003. We reviewed metrics and briefings provided by the program office, Lockheed Martin, Pratt & Whitney, and the Defense Contract Management Agency to identify progress in improving manufacturing processes. We analyzed changes in delivery dates for lot 10 aircraft delivered in 2018. We discussed reasons for any delivery delays and plans for improvement with officials from Lockheed Martin and Pratt & Whitney. We obtained cost investment and savings estimates and discussed cost and manufacturing efficiency initiatives, such as the economic order quantity purchases, with the contractors and program office officials to understand potential cost savings and plans. We also obtained and analyzed metrics on parts and aircraft quality through December 2018 and discussed steps taken to improve quality and deliveries with Lockheed Martin and Pratt & Whitney officials. We determined that the contractor’s production metrics and delivery dates were sufficiently reliable for our purposes of determining production efficiency and deliveries. We conducted this performance audit from June 2018 to April 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The F-35 program continues to address technical risks discovered in testing. Since our 2018 report, the program identified new risks with the canopy, fuel tubes, and cockpit, described below. The program has also incorporated design changes that have mitigated technical risks that we previously highlighted. The status of the Department of Defense’s (DOD) efforts to address these issues follows. Canopy Coating De-laminations and Corrosions: The F-35 fleet has experienced approximately 20 incidents of the canopy transparencies delaminating after less than 100 flight hours. The contractor is currently testing numerous solutions for the de-laminations, with intentions of completing testing by January 2019. F-35 aircraft are also experiencing canopy corrosion resulting from moisture intrusion due to the aircraft’s adhesive cracking under pressure and insufficient tape adhesion. The program has identified the need to modify over 173 canopies over 4 years. The contractor has begun to incorporate alternative material and tape into production, and released standardized repair procedures to mitigate this issue. Engine Fuel Tubes: In September 2018, a manufacturing fault in an engine fuel tube caused an in-flight failure, which resulted in an F-35B crash. The investigation identified several other life-limited fuel tubes in each F-35 variant. The fleet was grounded while all aircraft were inspected, and any fuel tubes identified were replaced or will be replaced by June 2019. Cockpit Display: In November 2018, operational test pilots experienced the cockpit display freezing and blanking, and identified the problem as a category 1 deficiency. The display issues occurred after a software update. The start of operational testing was delayed until the contractor could provide a software update to correct the problem, which was accomplished with a work-around in December 2018. Helmet Mounted Display: During low-light flights, the Helmet Mounted Display’s technology cannot display pure black, causing a green glow on the screen which makes it difficult to see the full resolution of the night vision video feed. The contractor is developing a new system to avoid this effect, and the contractor delivered this system to the test fleet in September 2018 with final flight testing planned through January 2019. Figure 7 is a photograph of the Helmet Mounted Display. Aerial refueling probes: The F-35B and F-35C variants use a “hose and drogue” system in which an aerial refueling tanker aircraft extends a long, flexible refueling hose and a parachute-like metal basket that provides stability, the receiving aircraft then connects to the drogue basket with its extendable refueling probe, as shown in figure 8. The refueling probe tips are meant to break in the event there is a stress occurring during refueling. However, the breaking is occurring more often than expected. Since April 2014, more than 20 incidents have occurred where the F-35’s aerial refueling probes broke off while conducting aerial refueling, leading to a restriction of aerial refueling operations. Tire service life: We reported in June 2018, the average service life of tires on the F-35B is below 10 landings. Lockheed Martin is currently working with three tire manufacturers to develop a new design with the goal of 20 landings. Testing of the new tires will occur throughout 2019. Figure 9 shows an F-35B during a landing. Life support system: The program has identified over 35 pilot physiological events, of which nearly 30 occurred in-flight. An action team made of government officials, contractors, and doctors completed their work by May 2018. A root cause investigation did not identify any F-35 system deficiencies, but reported it was difficult to fully determine the problem due to a lack of real-time data. Contracting officials stated that this is partially because the technology has not yet been developed to monitor pilot’s health in flight, in real time. The prime contractor continues to try to develop a means to monitor pilot health. In addition to the contact named above, the following staff members made key contributions to this report: Justin Jaynes (Assistant Director), Jennifer Baker, Emily Bond, Brandon Booth, Erin Butkowski, Matthew T. Crosby, Desirée E. Cunningham, R. Eli DeVan, Laura Jezewski, Jennifer Leotta, Meghan Perez, Hai Tran, Abby Volk, Mary Weiland, Alyssa Weir, and Robin M. Wilson. F-35 Joint Strike Fighter: Development is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved. GAO-18-321. Washington, D.C.: June 5, 2015. Warfighter Support: DOD Needs to Share F-35 Operational Lessons Across the Military Services. GAO-18-464R. Washington, D.C.: April 25, 2018. F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency. GAO-18-75. Washington, D.C.: October 26, 2017. F-35 Joint Strike Fighter: DOD’s Proposed Follow-on Modernization Acquisition Strategy Reflects an Incremental Approach Although Plans Are Not Yet Finalized. GAO-17-690R. Washington, D.C.: August 8, 2017. F-35 Joint Strike Fighter: DOD Needs to Complete Developmental Testing Before Making Significant New Investments. GAO-17-351. Washington, D.C.: April 24, 2017. F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities. GAO-16-390. Washington, D.C.: April 14, 2016. F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016. F-35 Joint Strike Fighter: Preliminary Observations on Program Progress. GAO-16-489T. Washington, D.C.: March 23, 2016. F-35 Joint Strike Fighter: Assessment Needed to Address Affordability Challenges. GAO-15-364. Washington, D.C.: April 14, 2015. F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.: September 23, 2014. F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26, 2014. F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014. F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013. F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013. Fighter Aircraft: Better Cost Estimates Needed for Extending the Service Life of Selected F-16s and F/A-18s. GAO-13-51. Washington, D.C.: November 15, 2012. Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks. GAO-12-437. Washington, D.C.: June 14, 2012. Joint Strike Fighter: Restructuring Added Resources and Reduced Risk, but Concurrency Is Still a Major Concern. GAO-12-525T. Washington, D.C.: March 20, 2012.", "summary": "In 2018, DOD sent an F-35 aircraft to its first combat mission and started initial operational testing. DOD now plans to spend over $270 billion to buy more than 2,000 F-35 aircraft over the next 26 years. Since 2011, GAO has found the need for more attention to the F-35's R&M performance to achieve an operationally suitable system. The National Defense Authorization Act for Fiscal Year 2015 included a provision for GAO to review the F-35 acquisition program until it reaches full-rate production. This is GAO's fourth report under this provision. This report assesses, among other objectives, (1) the program's progress in meeting R&M requirements (such as mission reliability) and (2) its plans for spending on new capabilities. GAO reviewed and analyzed management reports and historical test data; discussed key aspects of F-35 development with program management and contractor officials; and compared acquisition plans to DOD policies and GAO acquisition best practices. The F-35 program has made slow, sustained progress in improving the aircraft's reliability and maintainability (R&M). The F-35 aircraft (see figure) are assessed against eight R&M metrics, which indicate how much time the aircraft will be in maintenance rather than operations. Half of these metrics are not meeting targets. While the Department of Defense (DOD) has a plan for improving R&M, its guidance is not in line with GAO's acquisition best practices or federal internal control standards as it does not include specific, measurable objectives, align improvement projects to meet those objectives, and prioritize funding. If the R&M requirements are not met, the warfighter may have to settle for a less reliable and more costly aircraft than originally envisioned. In 2019, the F-35 program will start modernization efforts—estimated to cost $10.5 billion—for new capabilities to address evolving threats, without a complete business case, or a baseline cost and schedule estimate. Key documents for establishing the business case, such as an independent cost estimate and an independent technology assessment, will not be complete until after the program plans to award development contracts (see figure). Without a business case—consistent with acquisition best practices—program officials will not have a high level of confidence that the risk of committing to development has been reduced adequately prior to contract awards. Moving ahead without a business case puts F-35 modernization at risk of experiencing cost and schedule overruns similar to those experienced by the original F-35 program during its development. GAO is making five recommendations to DOD, including that it identify specific and measurable R&M improvement objectives, align improvement projects, and prioritize resources to meet them. In addition, DOD should complete its business case for modernization before beginning additional development efforts. DOD did not concur with this recommendation, but did concur with the R&M recommendations and plans to take action to address them.", "document_type": "gao"}
{"report": "Under traditional Medicare, hospitals are paid for the inpatient and outpatient services they provide under two distinct payment systems. Inpatient stays, including services incurred after being admitted to the hospital, are paid under the IPPS. Under this system, Medicare pays hospitals a flat fee per beneficiary stay, set in advance, with different amounts generally based on the beneficiary’s condition. Payment rates are also influenced by hospital-specific factors such as the relative hourly wage in the area where the hospital is located, and whether the hospital qualifies for other case- or hospital-specific additional payments. Outpatient services, including services obtained through the emergency department or other services incurred without being admitted to the hospital, are paid under the outpatient prospective payment system. Under this system, Medicare pays hospitals a flat fee per service, set in advance, with different amounts for each type of service. As with the IPPS, payment rates are adjusted for geographic factors. Congress has established payment adjustments for certain hospitals under the IPPS by changing the qualifying criteria for IPPS payment categories, creating and extending exceptions to IPPS rules, or exempting certain types of hospitals from the IPPS. These adjustments may help ensure beneficiary access to care or to help hospitals recruit and retain physicians and other medical professionals. Created through the Omnibus Budget Reconciliation Act of 1989, the MDH designation is an example of how Congress can enhance payments to certain hospitals. To qualify as an MDH, a hospital must demonstrate that it is: Medicare-dependent, defined as having at least 60 percent of their inpatient days or discharges attributable to Medicare beneficiaries; small, defined as having 100 or fewer beds; and rural, defined as being located in a rural area, though hospitals can also be eligible if they are located in a state without any rural areas. CMS regulations provide that hospitals can meet the requirement of demonstrating a 60 percent Medicare share of days or discharges using two of the three most recently settled cost reports, or using cost reports from 1987 or 1988. We refer to hospitals that meet this criterion using 1987 or 1988 cost report data as “legacy MDHs.” Some, but not all, MDHs are eligible to receive additional payment each year if they meet the payment criterion. Specifically, MDHs are assigned a payment rate—known as the hospital-specific rate (HSR)—based on their historic reported inpatient operating costs, trended forward to adjust for inflation and other factors, from one of three years (1982, 1987, or 2002). If the payment based on the HSR is higher than what the MDH would have otherwise received under IPPS, the MDH receives an additional payment. In this case, the MDH additional payment is calculated as 75 percent of the difference between the HSR and the IPPS amount. If the IPPS amount were higher than the HSR, the MDH would receive no additional payment. (See fig. 1.) Hospitals with an MDH designation are also eligible to receive other benefits. For example, MDHs are eligible for a separate additional payment if the hospital experiences at least a 5 percent decline in inpatient volume due to circumstances beyond its control. The MDH program does not provide for additional payments for outpatient services. In addition to the MDH designation, four other rural hospital designations exist: (1) critical access hospitals (CAH), (2) sole community hospitals (SCH), (3) low-volume adjustment hospitals (LVA), and (4) rural referral centers (RRC). Our review of CMS documentation shows that the MDH payment designation differs from the other rural payment designations in terms of eligibility criteria, financial benefit, extent of legislative permanence, and size—that is, the number of hospitals receiving the designation. (For detailed information on the five rural payment designations, see app. II.) Eligibility Criteria. The MDH designation differs from the other designations in terms of eligibility criteria. As noted earlier, MDHs must have at least 60 percent of their inpatient days or discharges attributed to Medicare patients, must be small and, with few exceptions, rural. In contrast, both the SCH and CAH designations require hospitals to be remote rural hospitals (i.e., located a specified distance from the nearest hospital). Similarly, LVAs are generally required to be more than 15 miles from the nearest hospital. Rural hospital designations also differ in terms of eligibility criteria related to bed size. CAH-designated hospitals are required to have 25 inpatient beds or fewer, while MDHs must have 100 beds or fewer. RRCs must have at least 275 beds or meet other criteria, such as serving a high proportion of remote patients, among other things. Financial Benefit. The MDH designation has a relatively small financial benefit compared to most of the other rural hospital designations, and the benefit only applies to costs associated with inpatient services. MDHs generally can only receive 75 percent of the difference between payment based on their HSR and the payment they would have otherwise received based on the IPPS rate as an additional payment added to their IPPS rate payment. In contrast, the SCH and CAH designations have both inpatient and outpatient payment benefits. Hospitals with an SCH designation can receive an additional payment added to their IPPS rate payment equal to 100 percent of the difference between payment based on the HSR and what the hospital would otherwise receive as payment based on the IPPS rate, as well as a 7.1 percent addition to their outpatient payments. The CAH designation results in the highest financial benefit by generally providing 101 percent of the hospital’s reported costs in the current year for both inpatient and outpatient Medicare services. LVAs generally can receive up to 25 percent in additional payments, and while RRCs receive no direct financial benefit, they are exempt from certain requirements related to geographic reclassification (as are SCHs). Legislative Permanence. Unlike all but one other rural payment designation, the MDH program is a temporary program and must be extended periodically by Congress in order to continue. Historically, the extension by Congress has sometimes occurred after the program has expired and as a result there were temporary lapses in payments to MDH designated hospitals. The Bipartisan Budget Act of 2018 included a provision to extend the MDH program through fiscal year 2022. The only other designation that must be extended is the LVA designation. In 2010, the Patient Protection and Affordable Care Act temporarily expanded the LVA designation eligibility criteria to include hospitals with a higher volume of discharges and located closer to other hospitals than in previous years. These expanded eligibility criteria have been amended and extended through fiscal year 2022. If Congress does not extend the expanded eligibility criteria beyond fiscal year 2022, the LVA designation will return to the narrower eligibility criteria that were in place prior to the Patient Protection and Affordable Care Act. Relative size and overlap. Of the 2,204 rural hospitals in fiscal year 2017, a relatively small share of these hospitals were MDHs. (See fig. 2.) In total, 138 hospitals, or 6.3 percent of those rural hospitals with at least one designation, were MDHs. In contrast, CAHs comprised the largest proportion of rural hospitals with a designation. In fiscal year 2017, 1,246 rural hospitals—or 56.5 percent of those rural hospitals with at least one designation—were CAHs. Of the five designations, three—CAHs, MDHs, and SCHs—are exclusive to each other, meaning a hospital can only have one of the three designations at any time. Hospitals designated as MDHs and SCHs may also be designated as LVAs, RRCs, or both. Approximately 75 percent of MDHs and 81 percent of SCHs had at least one concurrent designation in fiscal year 2017; in contrast, none of the CAHs received a secondary designation because CAHs are not eligible to receive other designations. Those MDHs with a concurrent designation consisted of 88 that had an LVA designation, 14 that had an RRC designation, and 2 that had both an LVA and RRC designation. (For detailed information on the 5 rural payment designations including LVA and RRC eligibility and financial benefit, see app. II.) From fiscal years 2011 through 2017, the number of MDHs declined, as well as the number of MDHs that received an additional payment under the program. In addition, during this period MDHs varied on other operational and financial metrics, including the share of Medicare revenue coming from inpatient care, various measures of Medicare dependence, and profit margins. Our analysis of CMS data shows that the number of MDHs declined from 193 to 138—a 28 percent decrease over the 7-year period from fiscal year 2011 through fiscal year 2017. (See fig. 3.) This decline can be due to a number of factors, including hospital closures, mergers, or changes in designation. For example, we previously reported that 16 MDHs closed between 2013 and 2017. Moreover, our review of Medicare Administrative Contractor documentation found that some MDHs became ineligible for the program due to no longer meeting eligibility criteria. In addition, the number of MDHs that received an additional annual payment also declined, from 92 MDHs in fiscal year 2011 to 78 MDHs in fiscal year 2017—a 15 percent decrease. Among MDHs that received an additional payment, the amount received and the share of the hospital’s total revenue this payment represented varied widely across the years, though the average amount generally increased over time. (See table 1.) For example, in fiscal year 2017, one hospital received around $1,000 in additional payment while another received almost $10.5 million. While the trend was not uniform among all MDHs, the median additional payment increased from about $695,000 in fiscal year 2011 to about $812,000 in fiscal year 2017. Our analysis of CMS data also shows that the average additional payment MDHs received ranged from less than 0.1 percent up to 8.7 percent of total facility revenue, with a fairly consistent average of 1.2 to 1.6 percent. (See table 2.) This underscores that the additional payment under the MDH program can be small relative to the overall revenue that the hospital receives. Our analysis of CMS data also shows that from fiscal years 2011 through 2017, MDHs varied on selected operational and financial metrics: the mix of Medicare revenue that came from inpatient versus outpatient care, various measures of Medicare dependence, and profit margins. On average, MDHs experienced a decline in the share of Medicare revenue that came from inpatient services. (See fig. 4.) In fiscal year 2011, around 66 percent of MDH Medicare revenue came from inpatient services compared to 58 percent in fiscal year 2017—a 13 percent decrease. This trend was slightly greater than that for all rural hospitals (an 11 percent decrease) and all hospitals (a 10 percent decrease). The trends across three measures of Medicare dependence varied for MDHs over time. Looking at the Medicare share of total revenue for MDHs, we found this share decreased when comparing fiscal years 2011 and 2017, from 25 to 22 percent. (See fig. 5.) In contrast, in terms of the number of inpatient days and discharges attributable to Medicare beneficiaries, we found these measures both increased slightly over time. Specifically, the median share of MDH inpatient days attributable to Medicare beneficiaries increased, although by less than a percentage point, and the median Medicare share of inpatient discharges increased by about 2 percentage points, when comparing fiscal years 2011 and 2017. (See figures 6 and 7.) To obtain additional context on the relationship between MDH eligibility criteria and the various measures of Medicare dependence, we ran regression models to identify the extent to which hospitals’ bed size and rural status were associated with the Medicare share of days, discharges, and total care revenue for all hospitals from fiscal years 2011 through 2017. We found that rural hospitals with fewer beds were associated with higher Medicare shares of inpatient days and discharges, holding all other factors constant. This indicates that by targeting smaller, rural hospitals in its eligibility criteria, the MDH program is targeting hospitals that are Medicare-dependent defined in terms of inpatient volume. At the same time, rural hospitals with fewer beds generally received a smaller share of their total care revenue from Medicare compared with other hospitals. This suggests that hospitals associated with high Medicare inpatient volume may not have relatively high shares of total care revenue coming from Medicare. For more technical detail on our regression analyses and findings, see appendix III. Our analysis of self-reported data from hospitals shows that Medicare profit margins and total facility profit margins declined for MDHs from fiscal year 2011 through 2017. (See table 3.) The degree to which Medicare margins declined for MDHs during this time period (6 percentage points) was greater than the degree to which they declined for rural hospitals (4 percentage points) and all hospitals (3 percentage points). The self-reported data show that unlike rural and all hospitals, MDHs were not profitable in 2017—meaning that the revenue they received from Medicare and other payers was less than their reported costs for providing services. Specifically, the total facility profit margin turned from positive to negative and dropped almost two percentage points between fiscal years 2011 and 2017. We also ran regression models to examine the relationship between all hospitals’ total profit margins and the various measures of Medicare dependence. We found that hospitals with a higher Medicare share of total-care revenue had lower total facility margins on average, holding all other factors constant; in contrast, there was no significant relationship between total facility margins and the inpatient volume-based measures of Medicare dependence. This indicates that a higher volume of inpatient services was not associated with lower profitability. We provided a draft of this report to the Department of Health and Human Services for comment. The Department of Health and Human Services provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of the Department of Health and Human Services. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or farbj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This appendix explains the quantitative scope and methodology used to examine how the Medicare-dependent hospital (MDH) designation differs from the other Medicare rural hospital designations. This appendix also explains the scope and methodology used to describe changes in the number and selected metrics of MDHs and other hospital types, including those used for a regression analysis to provide information on the relationship between MDH program criteria and Medicare dependence. To describe how the MDH designation differs from other rural hospital designations, we used CMS data—specifically, the Provider Specific File (PSF)—to identify the number of MDHs, critical access hospitals (CAH), sole community hospitals (SCH), rural low-volume adjustment hospitals (LVA), and rural referral centers (RRC) in fiscal year 2017. We then identified all rural hospitals without a designation in 2017 using the 2018 CMS Inpatient Prospective Payment System (IPPS) Impact File because those data are prepared in the middle of the year preceding the fiscal year. We define rural hospitals using the CMS MDH programmatic definition; that is, those hospitals that are not located in metropolitan statistical areas, as well as those hospitals that reclassified as rural for CMS payment purposes. We next identified the number of hospitals with each designation and the value of additional payments received under the rural designations that each hospital had in that year using data provided by each hospital through their Medicare Cost Report (MCR). The MCR is submitted to CMS by hospitals each fiscal year and contains information such as facility characteristics, utilization data, and costs to provide services to Medicare beneficiaries and all patients. Because CAHs are paid based on cost under a different payment system than the other hospitals, we did not have complete data to estimate what those hospitals would have been paid under the inpatient prospective payment system and thus could not identify the additional payments received by CAHs. In addition, RRCs only receive indirect payment benefits, and thus we could not calculate a comparable additional payment for that group of hospitals. For all analyses, we excluded hospitals within the Indian Health Service, as well as hospitals in Maryland and those outside of the remaining 49 states and the District of Columbia. We also excluded hospitals with reporting periods greater than 14 months or less than 10 months and those that reported zero or negative Medicare revenue. To describe changes in the number and select metrics of MDHs and other hospital types, we examined MCR data for fiscal years 2011 through 2017. To first identify the universe of MDHs, rural hospitals, and all acute care inpatient prospective payment system (IPPS) hospitals, we used the PSF and MCR for fiscal years 2011 through 2017, as well as CMS Impact Files for fiscal years 2012 through 2018. Then, we used the MCR to calculate the number of MDHs that received the MDH payment adjustment and the distribution of additional payments among MDHs in each year. Using those same data sources, we then calculated several metrics and examined trends for MDHs as compared to all rural hospitals and all hospitals overall. The first metric is the median proportion of total Medicare payments—referred to as revenue—each hospital group received from providing inpatient and outpatient care to Medicare beneficiaries. The second metric is hospitals’ median profit margins—a profitability measure calculated as the amount of revenue the hospital received minus reported costs, divided by the amount of revenue received. We calculated profit margins specific to Medicare revenue and costs (Medicare profit margins) but also for revenue and costs beyond Medicare (total facility profit margins), including payments for treating non-Medicare (including privately insured) patients. We calculated Medicare and total facility profit margins at the hospital level using hospital-reported costs and revenues from the MCRs, and reported the median margins for each hospital group. The Medicare margin reflects only payments and costs received for inpatient and outpatient services (about 90 percent of total Medicare revenue, according to CMS officials) and excludes payments and costs for other hospital-based services, such as those for skilled nursing and home health care. Third, we calculated hospitals’ degree of Medicare dependence using three separate definitions, or measures, of dependence: (1) the amount of revenue the hospital received from Medicare as a share of all the revenue the hospital received for inpatient and outpatient services (total care revenue), (2) the share of inpatient days of care the hospital provides that are attributed to Medicare beneficiaries, and (3) the share of inpatient discharges that are attributed to Medicare beneficiaries. We also calculated these metrics separately for those MDHs that were eligible for the program based on data from the 1980s—legacy MDHs. To do so, we used data provided by Medicare Administrative Contractors—third-party entities that administer Medicare program payments and determine MDH eligibility. To provide additional context on the relationship between MDH eligibility criteria and the various definitions of Medicare dependence, we developed an econometric model to analyze the association between bed size, rural status, and the three measures of Medicare dependence. We conducted the regression analysis using data from the CMS IPPS Impact Files and MCRs from fiscal years 2011 through 2017. We used the following measures as dependent variables: (1) the amount of revenue the hospital received from Medicare as a share of all the revenue the hospital received for inpatient and outpatient services (total care revenue), (2) the share of inpatient days of care the hospital provides that are attributed to Medicare beneficiaries, and (3) the share of inpatient discharges that are attributed to Medicare beneficiaries. 𝑌𝑌𝑖𝑖𝑖𝑖=log (𝑅𝑅𝑖𝑖𝑖𝑖) . Where 𝑅𝑅𝑖𝑖𝑖𝑖 represents the Medicare share of revenue, inpatient days or discharges, and the i and t subscripts represent the hospital and year, respectively. This formulation has the advantage of restricting the models’ predicted values to be positive and also allows for a relatively straightforward interpretation of the parameter estimates. We included hospital capacity or size as measured by the number of hospital beds. The number of beds is itself one of the criteria for MDH eligibility, and we were interested in whether hospitals of smaller sizes have more or less Medicare dependency. We included an indicator variable flagging whether the hospital is in a rural location. Rural location is one of the criteria for MDH program eligibility, and so this was a key variable in our model. We included the ownership category of the hospital, such as whether a hospital is for-profit or not for-profit, or whether it is a public or private institution. This organizational category may determine institutional characteristics, which affects the likelihood that the hospital serves either more or fewer Medicare beneficiaries. We included the degree of proximity to other hospitals of substantive size; specifically, the distance from the closest hospital with at least 100 beds. In addition to our rural indicator variable, this controlled for whether more remote hospitals are more likely to be more dependent on Medicare. We included whether the state in which the hospital is located has expanded Medicaid to provide coverage to low-income, non-elderly adults, because it is possible that an increased number of Medicaid- eligible patients may affect the number of Medicare patients using hospital services. This variable may be associated with less Medicare dependence if Medicaid becomes a relatively larger payer source, or it may be associated with more Medicare dependence if Medicaid eligibility brings Medicare-eligible people into the health care system. We included the percent of population in the hospital’s county over age 65, because areas with larger numbers of people over age 65 may be more likely to have a higher proportion of Medicare beneficiaries using health care services. We included the percent growth in county population, which allowed us to control for areas with declining populations that may be more likely to contain Medicare-dependent hospitals. Our model included time fixed effects (a dummy variable for each year in the analysis). The time fixed effects controlled for factors affecting hospitals nationally in as given year—in particular, those factors for which data were unavailable. We included a set of state fixed effects (a dummy variable for each of the states in the analysis) to control for effects that are common to a specific area, but for which data may have been unavailable. We estimated specifications that included interactions between our bed size categories and rural location. This allowed us to determine whether bed size had the same impact on Medicare dependence for hospitals in rural locations compared with those in urban locations. ln (𝑅𝑅𝑖𝑖𝑖𝑖)=�𝑓𝑓𝑖𝑖𝐹𝐹𝑖𝑖 +𝑋𝑋𝑖𝑖𝑖𝑖𝛽𝛽+𝐶𝐶𝑠𝑠𝑖𝑖𝛾𝛾+𝜀𝜀𝑖𝑖𝑖𝑖,𝑡𝑡=1,…,𝑇𝑇; 𝑖𝑖=1,…,𝐻𝐻. The dependent variable is the logarithm of our measure of Medicare 𝑋𝑋𝑖𝑖𝑖𝑖 is a 1 x k vector of hospital characteristics and possible 𝛽𝛽 is a k x 1 vector of parameters associated with the hospital interactions of these characteristics, where i denotes the ith hospital ∑ ∑ 𝐶𝐶𝑠𝑠𝑖𝑖 is a 1 x m vector of time-varying county-level variables hospital and their associated (lower case) parameters. 𝛾𝛾 is an m x 1 vector of parameters associated with the state-level characteristics such as the percent of the population over 65 and the dependence, 𝑅𝑅𝑖𝑖𝑖𝑖. and t denotes the year. 𝑋𝑋𝑖𝑖𝑖𝑖 contains key explanatory variables such as characteristics, 𝑋𝑋𝑖𝑖𝑖𝑖. ownership type, the number of beds, rural or urban location, whether 𝑇𝑇𝑖𝑖=2a hospital receives MDH program monies and other characteristics. represents the set of time (year) dummy variables (upper 𝑆𝑆𝑠𝑠=2 represents the set of state dummy variables (upper case) case) and their associated (lower case) parameters. characteristics such as the percent of the population over 65 and the county population growth rate. characteristics, 𝐶𝐶𝑐𝑐𝑖𝑖. Our model includes an interaction effect between the rural dummy variable and each of the characteristics except the geographic fixed effects. 𝜀𝜀𝑖𝑖𝑖𝑖 is a well-behaved Gaussian random error term that may have a heteroskedastic and/or clustered structure. We used Stata® to estimate the regression model, using fixed effects at the state-level to account for unobserved heterogeneity and clustering at the county-level. Our focus was on the main criteria for MDH eligibility—namely hospital size as measured by number of beds and rural versus no-rural hospital location. We divided the hospitals into five bed number categories: 50 beds or fewer Over 50 beds to 100 beds Over 100 beds to 300 beds Over 300 beds to 400 beds This categorization strikes a balance between having too many categories, which would reduce the statistical power of our analysis, and having too few categories, which would fail to identify any non-linear pattern in the statistical relationship. These categories also contain the 100 bed criterion as one of the cut-off points. Our analysis controls for location and possible heterogeneity by using geographic fixed effects but we also want to identify the impact of rural location. Selecting too detailed a level of geographic fixed effect such as county or zip code would limit our ability to identify the rural effect so we used states. We recognized that state fixed effects may not identify more localized effects; this is a limitation of our model. We also modeled the effects of hospital characteristics on total facility profit margins; that is, the difference between revenue and costs as a percent of revenue. For MDHs in our analysis, we excluded any MDH additional payment from the margin calculation in order to isolate and remove the program impact on financial status. We used the same explanatory factors in our econometric model of hospital margins as in our models of Medicare dependence but we supplement these factors with our three measures of Medicare dependence—a separate model for each measure. This allowed us to assess how our different measures of Medicare dependence are associated with financial well-being. We assessed the reliability of the relevant fields in each of the data sets we used for these analyses by interviewing CMS officials, reviewing related documentation, and performing data checks. On the basis of these steps, we concluded that the data were sufficiently reliable for the purposes of our reporting objective. We identified five Medicare rural hospital payment designations and categorized them into two categories: (1) primary payment designations and (2) secondary payment designations. Primary designations include critical access hospitals (CAH), sole community hospitals (SCH), and Medicare-dependent Hospitals (MDH). Each designation has distinct eligibility requirements and payment methodologies. This appendix describes the full results for our modeling of Medicare dollars as a percentage of total revenue, the percent of inpatient days, the percent of inpatient discharges, and total hospital profit margins. We tested for the hypothesis that key groups of parameters were significantly different between urban and rural locations. We performed a k-parameter post-estimation Wald linear restriction where 𝛽𝛽𝑘𝑘𝑢𝑢 and 𝛽𝛽𝑘𝑘𝑟𝑟 are matrices of the estimated urban and rural parameters, respectively, for each of the k categories (bed-size, ownership type, etc.). We rejected the null hypothesis of parameter equality for bed-size, ownership types, Medicaid expansion, and year dummies at the 5 percent level. The miles distance parameters rejected the hypothesis at marginally above the 5 percent level. Rural hospitals generally were associated with larger Medicare shares of revenue than urban hospitals. In every bed-size category, the parameters for rural hospitals were significantly greater than for urban hospitals. In addition, controlling for urban-rural location, with the exception of the largest hospital category (over 400 beds) hospitals with fewer beds had a smaller Medicare share of revenue, as shown in figure 8. Hospitals in counties with higher percentages of people over age 65 were significantly associated with greater Medicare dependence. Our Wald tests rejected the null that parameters rural and urban were equal in the bed-number categories and in the ownership categories. As with the Medicare share of total revenue, our model for Medicare share of inpatient days showed, controlling for bed numbers, that rural hospitals generally had significantly greater Medicare dependence than urban hospitals. In most bed-size categories, the parameters for rural hospitals were greater than for urban hospitals. The pattern for bed size was different for Medicare dependence measured in revenue in that for rural hospitals, dependence fell as bed numbers rose, but, for urban hospitals, we observed a hump- shape distribution with the middle bed-number categories having higher dependence than the smallest and largest categories, as shown in figure 9. Hospitals located in counties with higher percentages of people over age 65 had higher dependence. Our model for the Medicare share of inpatient discharges showed that, controlling for bed numbers, rural hospitals generally had greater Medicare dependence than urban hospitals. In most bed-size categories, the parameters for rural hospitals were significantly greater than for urban hospitals. Our Wald tests rejected the null hypothesis that parameters for rural and urban were equal in the bed-size categories, Medicaid expansion variables, and in the ownership categories. The pattern for bed numbers was also different to Medicare dependence measured in revenue. The urban hospitals had a hump- shape distribution with the middle bed-number categories having higher dependence than the smallest and largest categories, whereas the rural showed largest effects at the smallest and the larger intermediate categories, as shown in figure 10. Hospitals located in counties with higher percentages of people over age 65 had higher dependence. The Medicare share of total revenue was significantly associated with smaller total facility profit margins and was the only statistically significant measure of Medicare dependence in the margin models. In general, hospitals with small numbers of beds—fewer than 100—were associated with smaller hospital margins relative to our reference category of large urban hospitals. However, there was no significant difference in any of the bed-number categories between urban and rural hospitals. Jessica Farb, (202) 512-7114 or farbj@gao.gov In addition to the contact named above, Gregory Giusto (Assistant Director), Kate Nast Jones (Analyst-in-Charge), Britt Carlson, Rachel Gringlas, Michael Kendix, Vikki Porter, Caitlin Scoville, Jennifer Rudisill, and Jeffrey Tamburello made key contributions to this report.", "summary": "The MDH program was enacted in 1989, providing a financial benefit to some small, rural hospitals with high shares of Medicare patients. The original MDH program was established through statute for 3 years, and Congress has extended it on several occasions. The Bipartisan Budget Act of 2018 included a provision to extend the MDH program through 2022, as well as a provision for GAO to review the MDH program. This report describes, among other things, the changes that occurred in the number of MDHs and selected metrics over time. GAO analyzed data submitted to CMS by hospitals from fiscal years 2011 through 2017—the most recent year for which consistent data were available at the time of GAO's analysis—among other CMS data. GAO also reviewed CMS regulations and other agency documents. The Department of Health and Human Services provided technical comments on a draft of this report, which GAO incorporated as appropriate. The Centers for Medicare & Medicaid Services (CMS) operates the Medicare-dependent Hospital (MDH) program, which assists hospitals that have 60 percent or more of inpatient days or discharges from Medicare patients, 100 or fewer beds, and that are generally located in a rural area. MDHs receive an additional payment if their historic costs in one of three base years adjusted for inflation, among other things, are higher than what the hospital would have otherwise received under the inpatient prospective payment system (IPPS). In contrast, if the IPPS amount was higher than historic costs, the MDH would receive no additional payment. In fiscal year 2018, CMS paid approximately $119 million in additional payments to MDHs. From fiscal years 2011 through 2017, the number of MDHs declined by around 28 percent. (See figure.) In addition, the number of MDHs that received an additional payment declined by around 15 percent. Over this period of time, MDHs also experienced a 13 percent decrease in the share of their Medicare revenue that came from inpatient services. In addition, there was a decline in the share of total MDH revenue that was attributed to Medicare patients, and a decline in Medicare profit margins by about 6 percentage points.", "document_type": "gao"}
{"report": "FAR Part 15 describes negotiated contracting, which includes the use of several competitive source selection processes. The processes are associated with the best value continuum, which includes the LPTA process on one end and the trade-off process on the other (see figure 1). Federal agencies may elect to use the LPTA process where the requirement is clearly defined and the risk of unsuccessful contract performance is minimal. In such cases, agencies can determine that cost or price should play a dominant role in the source selection. When using the LPTA process, the agency specifies the evaluation factors that establish the requirements of acceptability in the solicitation. Firms submit their proposals and the agency determines which of the proposals meet those requirements. No trade-offs between cost or price and non-cost factors (for example, technical capabilities or past performance) are permitted. Non-cost factors are rated on an acceptable or unacceptable basis. The award is made based on the lowest priced, technically acceptable proposal submitted to the government. In contrast, agencies may elect to use the trade-off process in acquisitions where the requirement is less definitive, more development work is required, or the acquisition has a greater performance risk. In these instances, non-cost factors may play a dominant role in the source selection process. Trade-offs between price and non-cost factors allow agencies to accept other than the lowest priced proposal. The FAR requires the solicitation to state whether all evaluation factors other than cost or price, when combined, are significantly more important than, approximately equal to, or significantly less important than cost or price. Contracting officials have broad discretion in the selection of the evaluation criteria that will be used in an acquisition. When one is required, a written acquisition plan generally should include a description of the acquisition’s source selection process and the relationship of the evaluation factors to the acquisition objectives. The FAR does not explicitly require contracting officials to document the reasons why the specific source selection process was chosen. The defense and civilian provisions required the DFARS and FAR, respectively, be revised to require that the LPTA process only be used if certain criteria are met, as described in table 1. The defense and civilian provisions also required that the use of the LPTA process be avoided, to the maximum extent practicable, in procurements that are predominantly for the products and services identified in table 2. The process for revising the FAR and DFARS is governed by statute, which generally requires agencies to issue a proposed rule in the Federal Register. Agencies are also required to provide at least a 30-day public comment period following publication of the proposed rule. Figure 2 illustrates the basic process that is generally used to revise the FAR and the DFARS. We have issued two reports in response to the defense provisions requiring us to review DOD’s use of the LPTA process. In November 2017, we found that the Army, Navy, and Air Force used the LPTA process for information technology and other services in 9 out of 133 instances when awarding contracts valued at $10 million or more in the first half of fiscal year 2017. Contracting officials stated that the LPTA process was used in these instances, in part, because the requirements were well-defined, noncomplex, or recurring. We also found that contracting officials’ use of the LPTA process was generally consistent with the criteria listed in the defense provisions. In November 2018, we estimated that about 26 percent of DOD’s contracts and orders valued at $5 million or more in fiscal year 2017 were competitively awarded using the LPTA process. We found that DOD used the LPTA process to buy equipment, fuel, information technology services, and construction services, among other things. We also found that contracting officials used the LPTA process for reasons consistent with the criteria in the defense provisions. Specifically, contracting officials associated with the 14 contracts and orders we selected used the LPTA process, in part, because they determined there was no trade-off available or determined that DOD would not derive any benefit from paying a premium for offers that exceeded the minimum capabilities. Finally, we found that some contracting officials were confused about how to apply two of the criteria included in the defense provisions. Specifically, contracting officials were confused regarding how to assess life cycle costs associated with their procurements (shown as criterion 6 in table 1) or whether the products and services they were acquiring would be considered expendable in nature (criterion 8). Absent clarification on how to consider these two criteria, we found there was potential for increased risk that DOD contracting officials would not consistently apply the criteria of the defense provisions. Accordingly, we recommended that DOD address how contracting officials should apply these two criteria when using the LPTA process. DOD concurred with our recommendations, and plans to address them by issuing guidance concurrent with publication of the final rule at the end of fiscal year 2019. In December 2018, DOD issued a proposed DFARS rule for public comment to address the defense provisions for using the LPTA process. The December 2018 proposed rule reflected the criteria and limitations for using the LPTA process set forth in the defense provisions, and provided further clarification that these provisions were applicable to both contracts and orders. The public comment period ended on February 4, 2019, during which time the Defense Acquisition Regulations Council received 15 comments. In commenting on the proposed rule, industry representatives generally indicated their support for the proposed rule. On June 19, 2019, the Council agreed to move forward with the process for issuing a final rule revising the DFARS. Defense Pricing and Contracting officials stated that DOD expects to finalize the rule by the end of fiscal year 2019. The time required to develop and finalize the revisions to the DFARS has been longer than provided for under the NDAA for fiscal year 2017, which required the DFARS be revised within 120 days after enactment, which would have been in April 2017. In July 2019, we found that it can take a year or longer to issue a final DFARS rule. For this DFARS case, a Defense Pricing and Contracting official cited several reasons why the revisions have been delayed, including the need to address LPTA-related provisions in two separate NDAAs and the need to resolve a backlog of DFARS changes. In addition to ongoing efforts to update DFARS regulations, DOD officials plan to update the DFARS Procedures, Guidance and Information to provide defense contracting officers with supplemental guidance on applying the new criteria for using the LPTA process. A Defense Pricing and Contracting official stated that this update would be finalized by the end of fiscal year 2019 to coincide with the issuance of the final DFARS rule. The FAR Council has also initiated efforts to incorporate the civilian provisions for using the LPTA process into the FAR. The NDAA for Fiscal Year 2019 required that the FAR be revised to incorporate the civilian provisions within 120 days after enactment, which would have been in December 2018. Officials from the Office of Federal Procurement Policy told us, however, that it generally takes much longer than 120 days to revise the FAR. According to an analysis provided by DOD, it takes 483 days on average to issue a FAR rule. The FAR case to implement the civilian LPTA provisions was initiated in August 2018—the same month the NDAA for Fiscal Year 2019 was enacted. Office of Federal Procurement Policy officials stated that a proposed FAR rule is scheduled to be published in the Federal Register in September 2019. The public comment period for the proposed rule is scheduled to end in November 2019. Figure 3 shows when the defense and civilian provisions were enacted, when the rules were required to be implemented, and some of the efforts associated with revising both the DFARS and the FAR. Of the six agencies we reviewed, we found that DOD and DHS had existing source selection guidance that already reflected some of the criteria for using the LPTA process identified in the defense and civilian provisions. The other four civilian agencies did not have source selection guidance specific to using the LPTA process. Table 3 shows the status of selected agencies’ existing guidance related to the LPTA process. We found the following: DOD’s March 2016 Source Selection Procedures generally includes five of the eight criteria for using the LPTA process. A Defense Pricing and Contracting official stated that this guidance could be updated after the DFARS rule is implemented and the Procedures, Guidance, and Information resource is updated. The DHS September 2013 Source Selection Guide generally includes the first four of the six criteria for using the LPTA process. DHS officials stated that they plan to update their guidance after the FAR is amended to reflect the criteria and limitations for using LPTA. Acquisition policy officials from VA, GSA, USDA, and HHS stated that they do not have agency-specific guidance for using the LPTA process beyond what is currently provided for under the FAR. These officials stated that they were waiting for regulations to be finalized before determining if there is a need to develop any new guidance. Based on the results of our generalizable samples, we estimate that the selected DOD components used the LPTA process for about 25 percent of competitive contracts and orders valued at $5 million or more in fiscal year 2018, compared to about 7 percent of such contracts and orders at selected civilian agencies, as shown in Table 4. Our findings regarding how often DOD uses the LPTA process are consistent with what we found in our prior work. In November 2018, for example, we reported that Army, Navy, Air Force, and DLA awarded about 26 percent of contracts and orders using the LPTA process in fiscal year 2017. In November 2017, we reported that officials told us the LPTA process was used in instances where the requirements were well- defined, noncomplex, or recurring. This is the first year we were required to evaluate civilian agencies’ use of the LPTA process. Civilian agency officials we interviewed provided various perspectives on the extent to which their agency used the LPTA process. HHS officials told us that their acquisitions are generally complex, so the LPTA process is not often deemed the appropriate mechanism for determining best value. USDA officials told us that they have few acquisitions valued at more than $5 million, and that those acquisitions are likely to have more complex requirements. In such cases, the officials told us, technical and performance considerations generally would be more important than price factors. In analyzing FPDS-NG data, we found that 1 percent of USDA’s fiscal year 2018 contracts and orders were valued at more than $5 million. GSA officials told us their agency often procures services where it is beneficial for industry to propose solutions to a stated need, rather than GSA dictating the solution, such as professional services or information technology systems for a secure network solution. In these cases, officials said they would not have the technical specifications that an LPTA process would require. Officials from DHS and VA stated that they do not centrally track the source selection method used and they do not have sufficient information to say why their agencies use LPTA less frequently than other source selection methods. Within the sample of contracts we reviewed, we found DOD and the five selected civilian agencies bought a variety of products and services using the LPTA process in fiscal year 2018 (see table 5). We found that four of these DOD contracts and orders and one civilian agency order were for services that could be considered within the categories for which the defense and civilian provisions place limitations on, but do not prohibit, use of the LPTA process. In November 2018, we found that DOD contracting officers generally justified the use of the LPTA process for products and services in these categories. As described earlier in this report, the DFARS and FAR are in the process of being revised and do not currently address the limitations on the use of LPTA for these products and services. We provided a draft of this report to OFPP, DOD, VA, HHS, GSA, DHS, and USDA for review and comment. OFPP, DOD, GSA, DHS and HHS provided technical comments, which we incorporated as appropriate. VA and USDA told us that they had no comments on the draft report. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget, the Secretary of Defense, the Administrator of General Services, the Secretary of Veterans Affairs, the Secretary of Homeland Security, the Secretary of Agriculture, and the Secretary of Health and Human Services. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or dinapolit@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Section 813 of the National Defense Authorization Act (NDAA) for Fiscal Year 2017, as amended, included a provision that we report on the number of instances where Department of Defense (DOD) used the lowest price technically acceptable (LPTA) process for contracts exceeding $5 million, as well as provide an explanation of how acquisition officials considered the new criteria in making a determination to use the LPTA process. We have previously issued two reports in response to this provision. Subsequently, Section 880 of the NDAA for Fiscal Year 2019 included a provision that we report on the number of instances where civilian agencies used the LPTA process for contracts exceeding $5 million, as well as provide an explanation of how acquisition officials considered the six criteria in making a determination to use the LPTA process. This report, which addresses both provisions, describes (1) the status of regulatory changes required by the defense and civilian provisions for using the LPTA process and (2) the extent to which DOD and selected civilian agencies used the LPTA process to competitively award contracts and orders valued at $5 million or more in fiscal year 2018, and what they bought using this process. To address both objectives and select the DOD components and civilian agencies included in our scope, we used data from the Federal Procurement Data System-Next Generation (FPDS-NG) to identify the population of DOD and civilian agency contracts and orders that were reported as competitively awarded and valued at $5 million or more in fiscal year 2018. For DOD, we focused our review on the top four DOD components—Army, Navy, Air Force, and Defense Logistics Agency (DLA)—because they accounted for about 5,400—or about 88 percent— of all DOD contracts and orders valued at $5 million or more that were reported as competitively awarded in fiscal year 2018. Similarly, we focused our analysis on the top five civilian agencies—the Departments of Veterans Affairs (VA), Health and Human Services (HHS), Homeland Security (DHS), and Agriculture (USDA) and the General Services Administration (GSA)—which accounted for about 3,000—or about 66 percent—of all civilian agency contracts and orders valued at $5 million or more that were reported as competitively awarded in fiscal year 2018. To describe the status of regulatory changes governing the use of the LPTA process, we obtained information on agency officials’ efforts to amend the Defense Federal Acquisition Regulation Supplement (DFARS) and the Federal Acquisition Regulation (FAR). To do this, we met with DOD and Office of Federal Procurement Policy officials responsible for overseeing the regulatory changes. We also reviewed DOD’s December 2018 proposed rule to revise the DFARS and the 15 public comments DOD received on the proposed rule. Because revisions to the FAR and DFARS have not been finalized, regulations do not yet require or provide guidance to acquisition officials on how to consider the new criteria. Therefore, we also analyzed agency guidance and interviewed acquisition and contracting policy officials at DOD and each of the selected civilian agencies to determine whether agencies had existing guidance that addressed the defense and civilian provisions, in whole or in part. Specifically, we reviewed agency-specific source selection guidance from DOD, DHS, and VA. GSA, USDA, and HHS do not have source selection guidance that specifically addresses the LPTA process. According to officials, DOD and the selected civilian agencies do not maintain centralized data on whether the LPTA process is used to award contracts and orders. Consequently, to describe the extent to which DOD and civilian agencies used the LPTA process in competitively awarded contracts and orders valued at $5 million or more in fiscal year 2018, we used FPDS-NG to select two generalizable random samples of contracts and orders to estimate the use of LPTA by the DOD components and the civilian agencies within our scope. This resulted in samples of 102 contracts and orders for the four selected DOD components and 100 for the five selected civilian agencies. We removed five contracts and orders from our DOD sample: two contracts and one order because they were incorrectly reported by the agency in FPDS-NG as having been competitively awarded, and two contracts because they were classified. We removed three contracts and orders from our civilian agency sample: two orders because they were incorrectly reported by the agency in FPDS-NG as having been competitively awarded, and one contract because it was incorrectly reported as having an estimated value of more than $5 million. After removing these contracts and orders, our generalizable sample consisted of 97 DOD contracts and orders and 97 civilian agency contracts and orders. For each contract and order in our sample, we requested that the selected agencies identify whether the LPTA process was used. We independently verified agency responses by reviewing the solicitations for each of the contracts and orders within our two samples. We also verified relevant FPDS-NG data on estimated value and competition using agency-provided documentation for the contracts and orders we reviewed. Based on this, we determined these data were sufficiently reliable for us to estimate the percentage of contracts and orders valued at $5 million or more that the four components within DOD and the five selected civilian agencies competitively awarded in fiscal year 2018 using the LPTA process. We also used FPDS-NG product and service codes to identify whether the LPTA contracts and orders in our sample could be considered to be within one of the categories that the defense and civilian provisions direct agencies to avoid use of the LPTA process to the maximum extent practicable. The regulatory changes required by the defense and civilian provisions are not yet in place, and the defense and civilian provisions do not explicitly prohibit use of the LPTA process to acquire these categories of products and services. Therefore, we did not evaluate the reasons why an agency may have used the LPTA process in these instances. The findings based on our review of the product and services codes for the LPTA contracts and orders in our sample are not generalizable. We conducted this performance audit from February 2019 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Justin Jaynes (Assistant Director), Heather B. Miller (Analyst-in-Charge), Sarah Cantatore, Matthew T. Crosby, Lorraine Ettaro, Lori Fields, Stephanie Gustafson, Julia Kennon, Sarah Martin, Alyssa Weir, and Khristi Wilkins made key contributions to this report.", "summary": "When awarding a contract competitively, agencies can evaluate proposals using a best value, LPTA process that assesses which firm offered the lowest priced technically acceptable proposal. Section 813 of the NDAA for Fiscal Year 2017, as amended, included limitations on DOD's use of the LPTA process and required DOD to revise its acquisition regulation to reflect new criteria for use of the LPTA process. Section 880 of the NDAA for Fiscal Year 2019 required the FAR to be updated with similar requirements for civilian agencies. Sections 813 and 880 also included provisions for GAO to report on the number of instances where the LPTA process was used for contracts exceeding $5 million. This report describes (1) the status of regulatory changes governing the use of the LPTA process; and (2) the extent to which DOD and selected civilian agencies used the LPTA process to competitively award contracts and orders valued over $5 million in fiscal year 2018. GAO interviewed DOD and civilian agency officials involved in revising the DFARS and the FAR. GAO used data from the Federal Procurement Data System-Next Generation to select the top four DOD components and the top five civilian agencies based on the total number of contracts and orders valued at $5 million or more and competitively awarded in fiscal year 2018. Using this data, GAO developed generalizable samples to estimate these components' and agencies' use of the LPTA process in fiscal year 2018. Defense and civilian agencies are in the process of revising acquisition regulations to include criteria and limitations for using the lowest price technically acceptable (LPTA) process, as established under the National Defense Authorization Acts (NDAA) for Fiscal Years 2017 and 2019. While the Acts required revised regulations to be in place within 120 days of enactment, officials involved in revising the regulations stated that this process typically takes at least a year. The Department of Defense (DOD) issued a proposed Defense Federal Acquisition Regulation Supplement (DFARS) rule in December 2018 and expects the rule to be finalized by the end of fiscal year 2019. Officials responsible for revising the Federal Acquisition Regulation (FAR) have drafted a proposed FAR rule. The proposed FAR rule is scheduled to be published in the Federal Register in September 2019. See the figure below for the time frames and actions taken to update the DFARS and the FAR. Based on the results of GAO's generalizable samples, DOD used the LPTA process more frequently than selected civilian agencies in fiscal year 2018 for competitive contracts and orders valued at $5 million or more (see table).", "document_type": "gao"}
{"report": "In creating the military justice system, Congress established three types of military courts, called the summary, special, and general courts-martial, to adjudicate UCMJ violations. Each of these types of military courts respectively is intended to deal with progressively more serious offenses, and each court-martial type may adjudicate more severe maximum punishments as prescribed under the UCMJ. In addition, an accused servicemember can receive nonjudicial punishment under Article 15 of the UCMJ, by which a commander can punish a servicemember without going through the court-martial process. There are several steps in the discipline of a servicemember who allegedly commits a crime under the UCMJ, which are summarized in figure 1 below. The military justice process begins once an offense is alleged and an initial report is made, typically to law enforcement, an investigative entity, or the suspect’s chain of command. The commanding officer, law enforcement, or a military criminal investigative organization (MCIO) will conduct an inquiry or investigation into the accusation and gather all reasonably available evidence. Investigations are recorded in MCIO databases when a servicemember is the subject of a criminal allegation; for the purposes of our report, we say the servicemember had a “recorded investigation” to describe these cases. Following an investigation, the first step toward initiation of a court-martial is when the accused is presented with a list of charges signed by the accuser under oath, which is called preferral of charges. After charges are preferred, the charges are forwarded to an officer with sufficient legal authority to convene a court-martial, also known as the “convening authority.” The convening authority in receipt of preferred charges may, among other actions, refer the case to its own court or forward the case to a superior commander for disposition. Once referred to a general or special court- martial, an accused servicemember may be tried by a military judge alone or by a military judge with a military jury. In summary courts-martial, a single commissioned officer who is not a military judge adjudicates minor offenses and a sentence. Convictions at the general and special court- martial level are subject to a post-trial process and may be appealed to higher courts in cases where the sentence reaches a certain threshold. The military justice system, like the civilian criminal justice system, provides avenues for accused servicemembers to raise allegations of discrimination, improprieties in investigations, improprieties in disposition, and improprieties in the selection of the military jury at the court-martial proceeding, before a judge and on appellate review. The military services do not collect and maintain consistent information regarding race and ethnicity in their investigations, military justice, and personnel databases. Specifically, the number of potential responses for race and ethnicity within the 15 databases across the military services ranges from 5 to 32 options for race and 2 to 25 options for ethnicity, which can complicate cross-service assessments. For example, the Army’s personnel database maintains 6 options for race and 23 options for ethnicity, whereas the Coast Guard’s personnel database maintains 7 options for race and 3 for ethnicity. Table 1 below summarizes how the databases used by the military services vary in how the servicemember’s race is entered and the number of potential race options. Table 2 shows that the military services’ databases also vary in how information about servicemembers’ ethnicity is entered into the databases and the number of potential ethnicity options that are collected. Although the data collected and maintained was not consistent within and across the military services, each of the military services’ databases maintained race and ethnicity data for at least 99 percent of the servicemembers, with the exception of the Coast Guard. The Coast Guard did not track information about race or ethnicity in its military justice database, Law Manager. Coast Guard officials stated that this is because Law Manager was designed to determine the status of court- martial cases, and captures attributes that are needed to generate relevant UCMJ documents, such as court pleadings. Demographic information such as race and ethnicity is not included in these official documents, so this information is not input into Law Manager. Further, four of the databases we reviewed—including both of the Army’s military justice databases, and the Navy and the Marine Corps’ military justice databases—collect information on race and ethnicity in a combined data field as shown in table 2 above, whereas the other databases collect and maintain race and ethnicity information in two separate fields. These inconsistencies limit the military services’ ability to collectively or comparatively assess these demographic data to identify any racial or ethnic disparities in the military justice system within and across the services. Recommendations to collect and maintain race and ethnicity information in investigations and personnel databases. To address these inconsistencies, in our May 2019 report, we made four separate recommendations to each of the military departments and to the Secretary of Homeland Security for the Coast Guard. We recommended that these entities develop the capability to present servicemembers’ race and ethnicity data in their investigations and personnel databases using the same categories of race and ethnicity established in the uniform standards for the military justice databases that were issued in December 2018. As part of these uniform standards, the military services were directed to collect data related to race and ethnicity in their military justice databases, to collect race and ethnicity data in separate data fields, and to standardize the reporting of the data into categories identified in the standards. However, DOD applied these December 2018 standards only to the military justice databases and not to the investigations and personnel databases. DOD officials stated that the investigations and personnel databases do not fall under the charter of the DOD General Counsel, which issued the standards for the military justice databases. DOD and the Department of Homeland Security (DHS) concurred with these four recommendations. As of October 2019, officials from each of the military departments said that they were working to implement the uniform standards for race and ethnicity and the ability to aggregate the data, and they expected to implement these categories in December 2020. Similarly, as of May 2019, the Coast Guard expected to implement such modifications by September 2020. Although some military services report demographic information about the subjects of military justice actions internally, the military services have not externally reported data that provides visibility into, or would enable an analysis of, the extent of racial or ethnic disparities in the military justice system. Officials from all of the military services told us that they compile internal quarterly or monthly staff judge advocate reports, which include the total number of each type of court-martial handled by their legal offices and of nonjudicial punishments. According to military service officials, the Air Force and the Army reports include demographic information about servicemembers involved in these cases, such as the total number of each type of case broken out by the subject’s race and ethnicity. However, the Navy, Marine Corps, and Coast Guard reports do not include this demographic information, and there was no requirement to do so at the time of our May 2019 report. Regarding external reporting, the UCMJ directs the Court of Appeals for the Armed Forces, the Judge Advocates General, and the Staff Judge Advocate to the Commandant of the Marine Corps to submit annual reports on the military justice system to the Congressional Armed Services Committees, the Secretary of Defense, the secretaries of the military departments, and the Secretary of Homeland Security. These reports are to include information on the number and status of pending cases handled in the preceding fiscal year, among other information. The annual reports include the total number of cases each military service handled for each type of court-martial and for nonjudicial punishments. However, prior to our review, these annual reports did not include demographic information about servicemembers who experienced a military justice action, such as breakdowns by race, because the reporting requirement did not direct the military services to include such information. Recommendation to require military services to include data about race and ethnicity in annual reports about military justice actions. In our May 2019 report, we recommended that the Joint Service Committee on Military Justice, which is responsible for reviewing the UCMJ annually, consider an amendment to the UCMJ’s annual military justice reporting requirements to require the military services to include demographic information, including race and ethnicity, for all types of courts-martial. DOD concurred with this recommendation. According to a memorandum from the Joint Service Committee on Military Justice, in September 2019 the committee proposed an action item as part of its annual review. Specifically, the committee was considering an amendment to the UCMJ’s annual military justice reporting requirements to require the military services to include demographic information, including race and ethnicity, for all types of courts-martial. However, in December 2019, the National Defense Authorization Act for Fiscal Year 2020 included a provision directing the Secretary of Defense to include data on race, ethnicity, and gender in the annual military justice reports. We believe that this statutory change meets the intent of our recommendation. By requiring the military services to report this information, servicemembers and the public will have greater visibility into potential disparities, which will help build confidence that DOD is committed to a military justice system that is fair and just. DOD has not issued guidance that establishes criteria to specify when any data indicating possible racial or ethnic disparities in the investigations, trials, or outcomes of cases in the military justice system should be further reviewed, and to describe what steps should be taken to conduct such a review if it were needed. While equal employment opportunity enforcement is a very different context than the military justice system, other federal agencies have developed such criteria in the equal employment opportunity context that can indicate when disparities should be examined further. For example, the Department of Justice, the Department of Labor, the Equal Employment Opportunity Commission, and the Office of Personnel Management use a “four-fifths” test to determine when differences between subgroups in the selection rates for hiring, promotion, or other employment decisions are significant. These criteria, though inexact, provide an example of the type of criteria that DOD could consider using as a basis for determining when disparities among racial groups in the military justice process could require further review or analysis. Recommendation to issue guidance to establish criteria that determines when racial and ethnic disparities should be reviewed. In our May 2019 report, we recommended that the Secretary of Defense, in collaboration with the Secretaries of the military departments and the Secretary of Homeland Security, issue guidance that establishes criteria to specify when data indicating possible racial, ethnic, or gender disparities in the military justice process should be further reviewed, and that describes the steps that should be taken to conduct such a review. In commenting on a draft of our report, DOD partially concurred with this recommendation, agreeing with the content, but requesting that we modify the recommendation to direct it to more appropriate entities. That change was made before our report was issued. In October 2019, DOD officials said that the department was exploring the feasibility of conducting relevant research to inform implementation of this recommendation. At that time, they estimated that this research might be concluded in March 2021. In December 2019, the National Defense Authorization Act for Fiscal Year 2020 included a provision directing the Secretary of Defense to issue guidance consistent with our recommendation. DOD was directed to commence or carry out these activities by June 2020. We believe that issuing guidance that establishes criteria for determining when data indicating possible racial disparities in the investigations, trials, or outcomes of cases in the military justice system should be further examined, and describes the steps that should be taken to conduct such further examination, would better position DOD and the services to monitor the military justice system to help ensure that it is fair and just, a key principle of the UCMJ. Racial disparities exist in investigations, disciplinary actions, and punishment of servicemembers in the military justice system. Our analysis of available data from fiscal years 2013 through 2017, which controlled for attributes such as race, gender, rank, education, and years of service, found racial disparities were more likely in actions that first brought servicemembers into the military justice system, but we identified fewer statistically significant racial disparities in case outcomes— convictions and punishment severity. Black and Hispanic servicemembers were more likely than White servicemembers to be the subjects of recorded investigations in all of the military services, and were more likely to be tried in general and special courts-martial in the Army, the Navy, the Marine Corps, and the Air Force, as shown in figure 2 below. We could not analyze Coast Guard cases due to the small number of general and special courts-martial adjudicated in the Coast Guard from fiscal years 2013 through 2017. When separating general and special court-martial cases into those that either were or were not preceded by an investigation recorded in an MCIO database, we found fewer statistically significant racial disparities in most of the military services in general and special courts-martial that were preceded by a recorded investigation. However, as shown in figure 3 below, statistically significant racial disparities were also present in general and special courts-martial that did not follow a recorded investigation in all military services included in this analysis, which would include cases where the investigation was performed by the servicemember’s command. Specifically, as shown in figure 3 above, we found that: General and special courts-martial following a recorded investigation. Black, Hispanic, and servicemembers in the Other race category in the Army, and Hispanic servicemembers in the Marine Corps were more likely than White servicemembers to be tried in general and special courts-martial following a recorded investigation, after controlling for other attributes. We generally found fewer statistically significant differences compared to the results of our analyses for all special and general courts martial. General and special courts-martial without a recorded investigation. Black servicemembers in all of the military services were more likely than White servicemembers to be tried in general and special courts-martial without a recorded investigation after controlling for other attributes. These differences were consistent with the differences we identified for general and special courts-martial overall, as shown in figure 2 above. Hispanic servicemembers in the Army were more likely than White servicemembers to be tried in general and special courts-martial without a recorded investigation, but we found no statistically significant differences in the likelihood of Hispanic servicemembers to be tried in general and special courts-martial without a recorded investigation in the Marine Corps, the Navy, or the Air Force. Black servicemembers were more likely than White servicemembers to be tried in summary courts-martial and to be subjects of nonjudicial punishment in the Air Force and the Marine Corps, as shown in figure 4. The Army and the Navy did not maintain complete summary court-martial or nonjudicial punishment data, and the Coast Guard had too few summary courts-martial for us to analyze, and did not maintain complete nonjudicial punishment data. We could not determine whether disparities existed among servicemembers tried in summary courts-martial or subject to nonjudicial punishments in the Army and the Navy because the Army and the Navy did not collect complete summary courts-martial or nonjudicial punishment data in their investigations, military justice, or personnel databases. Specifically, as part of our data reliability checks, we identified the total number of summary courts-martial that the Army and the Navy reported in the Court of Appeals for the Armed Forces annual reports for fiscal years 2013 through 2017, and compared these totals to the number of cases we identified in their military justice databases. While our comparisons are not exact, due to differences in the dates we used to count the number of cases, we found that approximately 60 percent of the Army’s reported summary courts-martial cases and less than 50 percent of the Navy’s reported summary courts-martial cases were included in their military justice databases. The absence of complete summary court-martial data in the military justice databases of the Army and the Navy limits these services’ visibility into any disparities that may exist among servicemembers involved in these types of military justice proceedings. On December 17, 2018, the General Counsel of the Department of Defense issued the uniform standards and criteria required by article 140a of the Military Justice Act of 2016. As part of these uniform standards, the military services were directed to collect certain information about all cases in their military justice databases, which a DOD official said includes summary court- martial cases. The DOD General Counsel directed that military services are to implement the Secretary’s direction no later than December 23, 2020. Similarly, we identified the total number of nonjudicial punishments that the Army, the Navy, and the Coast Guard reported in the Court of Appeals for the Armed Forces annual reports for fiscal years 2013 through 2017, and compared these totals to the number of cases we identified in their military justice and personnel databases. As shown in figure 5 below, we found that 65 percent of the Army’s reported nonjudicial punishments, 8 percent of the Navy’s reported nonjudicial punishments, and 82 percent of the Coast Guard’s reported nonjudicial punishments were recorded in their military justice databases. Recommendation to include benefits and drawbacks of collecting and maintaining complete information for nonjudicial punishment. In our May 2019 report, we made separate recommendations to the Army, the Navy, and the Coast Guard to consider the feasibility, to include the benefits and drawbacks, of collecting and maintaining complete information for all nonjudicial punishment cases in one of the military service’s databases, such as information on the servicemembers’ race, ethnicity, gender, offense, and punishment imposed. DOD and DHS concurred with these recommendations. As of October 2019, Army and Navy officials said that they were developing the capability to collect data on race, ethnicity, gender, offense and punishment imposed for nonjudicial punishments. They expected to complete this action in December 2020. As of May 2019, the Coast Guard stated that it would consider the feasibility of collecting and maintaining complete information for all nonjudicial punishments cases through a military justice and personnel work group. The estimated completion date for this action had not been determined at that time. We identified fewer statistically significant racial disparities in case outcomes—convictions and punishment severity. Among the servicemembers convicted in general and special courts-martials, we found no statistically significant differences regarding the likelihood of conviction among racial groups in the Army, the Navy, the Marine Corps, and the Air Force, while controlling for other attributes, as shown in figure 6 below. In the military services that maintained complete punishment data—the Army, the Navy, the Marine Corps, and the Air Force—we found that minority servicemembers were either less likely to receive a more severe punishment in general and special courts-martial compared to White servicemembers, or there were no statistically significant differences in punishments among racial groups. Specifically, as shown in figure 7, Black servicemembers were less likely to receive a more severe punishment in general and special courts-martial compared to White servicemembers in the Navy, but there was no statistically significant difference for Black servicemembers in the Marine Corps, the Army, and the Air Force. Additionally, there were no statistically significant differences for Hispanic servicemembers in the Navy, the Marine Corps, the Army, or the Air Force. We could not determine disparities in case outcomes—convictions and punishment severity—in the Coast Guard’s general and special courts- martial for fiscal years 2013 through 2017 because the Coast Guard did not collect and maintain complete conviction and punishment data in its military justice database. Specifically, 16 percent of all Coast Guard cases were missing conviction and punishment data. When broken down by court-martial type, 20 percent of general court-martial cases, 15 percent of special court-martial cases, and 4 percent of summary court- martial cases were missing conviction and punishment data. Coast Guard officials acknowledged that incomplete conviction and punishment data entry is a consistent problem. They said that data entry had improved recently. On December 17, 2018, the General Counsel of the Department of Defense issued the uniform standards and criteria required by article 140a of the Military Justice Act of 2016. As part of these uniform standards, the military services were directed to collect information about the findings for each offense charged, and the sentence or punishment imposed. The DOD General Counsel directed that the military services are to implement the Secretary’s direction no later than December 23, 2020. DOD and the military services have taken some steps to study racial disparities in the military justice system over the last several decades, but they have not comprehensively studied the causes of any disparities. We previously reported in 1995 on DOD studies on discrimination and equal opportunity, and found DOD and the military services conducted seven reviews of racial disparities in discipline rates between 1974 and 1993. Since our 1995 report through 2016, DOD and military service assessments of military justice disparities have been limited. Officials in the Office of Diversity, Equity and Inclusion noted DOD has not conducted any department-wide assessments of racial disparities in military justice during this period. The military services’ diversity offices also were not able to identify any service-specific reviews of disparities in military justice. However, DOD has conducted climate surveys to address servicemembers’ perceptions of bias. In addition, the military services have some initiatives to examine and address disparities in military justice. For example, the Air Force routinely analyzes military justice data using a rates-per-thousand analysis to identify whether certain demographic groups are tried by courts-martial or subject to nonjudicial punishments at higher rates than others. These Air Force analyses found that Black servicemembers were more likely than White servicemembers to be subject to courts-martial and nonjudicial punishments from fiscal years 2013 through 2017, which is consistent with what we found. However, the other services do not routinely conduct such analyses. Officials from DOD and the military services acknowledged that they do not know the cause of the racial disparities that have been identified in the military justice system. This is because they have not conducted a comprehensive evaluation to identify potential causes of these disparities and make recommendations about any appropriate corrective actions to remediate the cause(s) of the disparities. Recommendation to identify causes of racial disparities in the military justice system. In our May 2019 report, we recommended that the Secretary of Defense, in collaboration with the Secretaries of the military services and the Secretary of Homeland Security, conduct an evaluation to identify the causes of any disparities in the military justice system, and take steps to address the causes of these disparities as appropriate. DOD partially concurred with this recommendation, agreeing with the content, but requesting that we modify the recommendation to direct it to more appropriate entities. We made that change before the report was issued. In October 2019, DOD officials said that the department was exploring the feasibility of conducting a research project to delve into the differences in military justice data to inform implementation of this recommendation. At that time, they estimated that this research might be concluded in March 2021. In December 2019, the National Defense Authorization Act for Fiscal Year 2020 included a provision directing the Secretary of Defense to conduct an evaluation consistent with our recommendation. DOD was directed to commence or carry out these activities by June 2020. We believe that conducting a comprehensive analysis into the causes of disparities in the military justice system, would better position DOD and the military services to identify actions to address disparities, and thus help ensure that the military justice system is fair and just, a key principle of the UCMJ. In conclusion, our analysis of available data identified racial disparities in all of the military services for servicemembers with recorded investigations, and for four of the military services for trials in special and general courts-martial, but these disparities generally were not present in the convictions or punishments of cases. These findings show an association for disparities at particular stages of the military justice process, but are inconclusive regarding other stages for the period covered by our analysis. However, our findings of racial disparities, taken alone, do not establish whether unlawful discrimination has occurred, as that is a legal determination that would involve other corroborating information along with supporting statistics. The absence of complete nonjudicial punishment data in the Army, the Navy, and the Coast Guard limits their visibility into the vast majority of legal punishments imposed on servicemembers under the UCMJ every year. Without such data, these three military services will remain limited in their ability to assess or identify disparities among populations subject to this type of punishment. Our May 2019 report included several recommendations with specific actions that can be taken to better position DOD and the military services to identify and address disparities, such as (1) developing the capability to present race and ethnicity data from the military services’ personnel and investigations databases using the same categories as the military justice databases; (2) establishing criteria to determine when possible disparities among racial or ethnic groups should be further reviewed, and describing the steps that should be taken in such a review; and, importantly, (3) conducting a comprehensive evaluation of the causes of these disparities and taking steps to address them. To help build confidence that DOD is committed to a military justice system that is fair and just, and for the system of military law to be recognized as fair and just by both members of the armed forces and by the American public, DOD and the military services need to take actions to address these recommendations. Madam Chairwoman Speier, Ranking Member Kelly, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you or other Members of the Subcommittee may have at this time. If you or your staff have any questions about this testimony, please contact Brenda S. Farrell, Director, Defense Capabilities and Management, who may be reached at (202) 512-3604 or farrellb@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Kimberly C. Seay, Assistant Director; Christopher Allison; Renee S. Brown; Vincent M. Buquicchio; Christopher Gezon; Won (Danny) Lee; Serena C. Lo; Dae B. Park; Samuel J. Portnow; and Clarice Ransom. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Uniform Code of Military Justice (UCMJ) was established to provide a statutory framework that promotes fair administration of military justice. Every active-duty servicemember is subject to the UCMJ, with more than 258,000 individuals disciplined from fiscal years 2013-2017, out of more than 2.3 million unique active-duty servicemembers. A key principle of the UCMJ is that a fair and just system of military law can foster a highly disciplined force. This statement provides information on 1) the collection of race and ethnicity information in the military services' databases, 2) the extent of racial disparities in investigations, disciplinary actions, and case outcomes in the military justice system, and 3) steps taken by DOD to study any identified disparities. This statement is based on GAO -19-344 issued on May 30, 2019. As part of that work, GAO analyzed data from the investigations, military justice, and personnel databases from the military services, including the Coast Guard, from fiscal years 2013-2017 and interviewed agency officials. In May 2019, GAO found that the military services did not collect consistent information about race and ethnicity in their investigations, military justice, and personnel databases. Thus, the military services are limited in their ability to identify disparities (i.e., instances in which a racial or ethnic group was overrepresented) in the military justice system. The military services were not required to, and thus did not, report demographic information that would provide greater visibility into potential disparities in their annual military justice reports. GAO's analysis of available data identified disparities in how likely servicemembers of different races were to be subjects of investigations recorded in military criminal investigative organization databases and tried in general and special courts-martial in particular. For example, in three military services, Black servicemembers were about twice as likely as White servicemembers to be tried in general and special courts-martial. Racial disparities generally were not present in convictions or punishments. These findings show an association for disparities at particular stages of the military justice process, but are inconclusive regarding other stages. However, GAO's findings of racial disparities, taken alone, do not establish whether unlawful discrimination has occurred, as that is a legal determination that would involve other corroborating information and supporting statistics. Note: These analyses, taken alone, should not be used to make conclusions about the presence of unlawful discrimination. These multivariate regression analysis results estimate whether a racial group is more likely or less likely to be the subject of an investigation or a trial in general or special courts-martial after controlling for race, gender, rank, and education, and in the Air Force, years of service. GAO made all racial comparisons to White servicemembers, and grouped individuals of Hispanic ethnicity together, regardless of race. The Other race category includes individuals who identified as American Indian/Alaska Native, Asian, Native Hawaiian/Other Pacific Islander, and multiple races. The Department of Defense (DOD) has taken some steps to study disparities but has not comprehensively evaluated the causes of racial disparities in the military justice system. Doing so would better position DOD to identify actions to address disparities and to help ensure the military justice system is fair and just. GAO made 11 recommendations in prior work, including that the military services develop the capability to present consistent race and ethnicity data, and DOD and the Coast Guard include demographic information in military justice annual reports and evaluate the causes of disparities. DOD and the Coast Guard generally concurred. Progress has been made in addressing some of the recommendations. Continued attention is needed to ensure that the remainder of these recommendations are addressed.", "document_type": "gao"}
{"report": "Approximately 51,000 drinking water systems and 15,000 public wastewater systems provide clean and safe water to communities nationwide. About 9,000 drinking water systems provide service to 92 percent of the total population, or approximately 273 million people nationwide. The remaining 8 percent of the population is served by small systems that according to the American Society of Engineers frequently do not have the financial, managerial, and technical capabilities necessary to meet state and federal requirements for safe drinking water, such as limits in the levels of specific contaminants in drinking water. Drinking water and wastewater facilities include infrastructure such as tanks, pipes, pumps, and buildings that contain electrical, chemical, and mechanical equipment to treat and test water. The infrastructure is often built to last for over 50 years or longer, depending on the equipment. Many utilities in the country were built decades ago and therefore have existing and aging infrastructure that they must operate and maintain. Utilities generally develop long-term capital plans to identify the infrastructure they will need to replace or rebuild in the future. Utilities generally use historic records of seasonal precipitation, runoff, water temperature, and snow pack levels to determine how their systems should be designed and operated. According to the Water Research Foundation’s 2014 study, utilities have designed their infrastructure based on the expectation that future climate conditions will remain the same and have used historical climate or other data within a 100-year range. Generally, the study reported that utility infrastructure is designed and operated to convey or treat water up to a specific threshold amount based on these historic records. As they plan to rebuild or replace their infrastructure, utilities employ or contract with engineers to ensure that their infrastructure treats and transports water appropriately to meet standards under the Safe Drinking Water Act or the Clean Water Act. Under the Safe Drinking Water Act, EPA, among other actions, sets standards to protect the nation’s drinking water from contaminants, such as lead and arsenic. The Clean Water Act generally prohibits the discharge of pollutants from “point sources”—such as discharge pipes from industrial facilities and wastewater treatment facilities—without a permit. Drinking water and wastewater infrastructure remain the largest financial investment by communities nationwide, according to the Water Research Foundation’s 2014 Study. To pay for operations, maintenance, repair, and replacement of their infrastructure, drinking water and wastewater utilities generally raise revenues by charging their customers for the services they provide. In addition, the federal government invests in drinking water and wastewater infrastructure, as we reported in September 2017. In 2017, the most recent year for which data were available, state and local governments spent approximately $109 billion on their drinking water and wastewater infrastructure, according to Congressional Budget Office data. During the same time period, the federal government spent approximately $4 billion on drinking water and wastewater infrastructure. Agencies across the federal government, such as NOAA and the National Aeronautics and Space Administration, collect and manage many types of climate information and provide technical assistance to make this information more meaningful to federal, state, local, and private decision makers. Decision makers from all levels of government and the private sector use different types of climate information in their planning processes to reduce the potential impacts of climate change. To be useful, climate information must be tailored to meet the needs of each decision maker, such as an engineer responsible for building a bridge in a specific location, a county planner responsible for managing development in a large region, or a federal official managing a national-scale program. Decision makers also need climate information at different timescales corresponding to the short-, medium-, or long-term nature of their planning processes. A 2011 World Meteorological Organization report stated that decision makers need access to expert advice and support to help them select and properly apply climate information. According to a 2010 National Research Council report on making informed decisions about climate change and our November 2015 report on climate information, most decision makers need a basic set of information to understand and make choices about how to adapt to climate change. The set of information includes the following: Information and analysis about observed climate conditions. This includes information on, for example, temperature, precipitation, drought, storms, and sea level rise and how they may be changing in a local area. Information about observed climate impacts and vulnerabilities. This includes site-specific and relevant information on environmental, social, and economic impacts and vulnerabilities, resulting from observed changes in the climate against which past and current decisions can be monitored, evaluated, and modified over time. Projections of what climate change may mean for a local area. This includes, for example, projections based on easily understandable best- and worst-case scenarios with confidence intervals and probability estimates and examples of potential climate impacts. The primary source is NOAA’s online Climate Explorer, which provides climate projections in a range of climate variables relevant to decision makers for every county in the contiguous United States, enabling users to compare historical climate observations under two possible climate change scenarios that could occur this century. Information on the economic and health impacts of climate change. Observed and projected local impacts must be translated into costs and benefits, as this information is needed for many decision-making processes. Agencies across the federal government collect and manage many types of climate information, including observational records from satellites and weather monitoring stations on temperature and precipitation, among other things; projections from complex climate models; and other tools to make this information more meaningful to decision makers. Presidential Policy Directive 21 directs federal agencies to work with owners and operators and state, local, tribal, and territorial entities to manage risks and strengthen the security and resilience of critical infrastructure against all hazards. The directive, issued in 2013, identifies 16 critical infrastructure sectors whose assets, systems, and networks—either physical or virtual—are considered so vital to the United States that their incapacitation or destruction would have a debilitating effect on the nation’s security, economy, and public health or safety. One of the sectors is the Water and Wastewater Sector. The directive established a national policy on critical infrastructure security and resilience and made DHS the lead agency to coordinate the overall federal effort to promote security and resilience of the nation’s critical infrastructure. The directive assigned protection responsibilities to selected federal government agencies and departments, called Sector Specific Agencies, and designated EPA as the Sector Specific Agency for the Water and Wastewater Sector. As the Sector Specific Agency, EPA organized a Water and Wastewater Government Coordinating Council, including federal, state, and local decision makers. In turn, water utility owners and operators organized the Water and Wastewater Sector Coordinating Council. EPA and the councils work together and are responsible for planning and implementing the sector’s security and resilience activities. Presidential Policy Directive 21 also directed the DHS to update the National Infrastructure Protection Plan to provide a framework for how federal, state and local decision makers and private sector stakeholders can coordinate to improve the security and resilience of critical infrastructure. The DHS updated the National Infrastructure Protection Plan in 2013 and EPA issued the Water and Wastewater Sector-Specific Plan in 2015. In 2016, the Water and Wastewater Government Coordinating Council and the Water and Wastewater Sector Coordinating Council chartered the Water and Wastewater Sector Strategic Roadmap Work Group to review key threats and vulnerabilities of the sector, identify gaps in the sector’s capabilities relative to the key threats and vulnerabilities, and develop priorities and associated actions to address those gaps. In 2017, the work group issued the report, Roadmap to a Secure and Resilient Water and Wastewater Sector (Roadmap), in part, to help inform utilities’, industry groups’, and government agencies’ planning processes and to support collaboration and leverage resources between stakeholders in the sector. The resulting report identified weather-related disasters, such as floods and earthquakes, and long-term climate-related hazards, such as drought and sea level rise, as among the most significant risks to drinking water and wastewater infrastructure. Our previous work on climate change found that the federal government could improve the way that it provides information to facilitate more informed local infrastructure adaptation decisions. In November 2015, we reported that federal agencies could help local infrastructure decision makers by providing the best available climate-related information and by clarifying federal sources of technical assistance for incorporating climate- related information into their planning. In November 2015, we found that federal efforts to provide climate information could be improved by incorporating key organizational and data elements, including (1) a focused and accountable organization; (2) authoritative data that define the best available information for decision makers; and (3) technical assistance to help decision makers assess, translate, and use climate information in planning. We recommended that the Executive Office of the President direct a federal entity to develop a set of authoritative climate change projections and observations and create a national climate information system with defined roles for federal and nonfederal entities. The Executive Office of the President neither agreed nor disagreed with the recommendations and, as of May 2018, had not implemented them. Our previous work on natural disasters found that disaster costs are a key source of federal fiscal exposure. In our July 2015 report on Hurricane Sandy, we found that there was no comprehensive, strategic approach to identifying, prioritizing, and implementing investments for disaster resilience, which increases the risk that the federal government and nonfederal partners will experience lower returns on investments or lost opportunities to strengthen critical infrastructure. We recommended that the Mitigation Framework Leadership Group—an interagency group chaired by FEMA that organizes mitigation efforts across the federal government and assesses the effectiveness of mitigation strategies— establish an investment strategy to identify, prioritize, and guide federal investments in disaster resilience and hazard mitigation-related activities and make recommendations to the President and Congress on how the nation should prioritize future disaster resilience investments. The Mitigation Framework Leadership Group agreed and issued the National Mitigation Investment Strategy in August 2019. In September 2018, we reported that four near-sequential disasters in 2017—Hurricane Harvey, Hurricane Irma, Hurricane Maria, and the California wildfires—created an unprecedented demand for federal disaster response and recovery resources and that Hurricanes Harvey, Irma, and Maria ranked among the top five costliest hurricanes on record. As of June 2018, Congress had appropriated over $120 billion in supplemental funding for response and recovery related to the 2017 hurricanes and wildfires. In October 2019, we issued a Disaster Resilience Framework that identifies federal actions and opportunities to enhance and promote disaster and climate change resilience nationwide focusing on three principles where the federal government can influence decision-making. First, the framework states that federal action can help ensure that decision makers at all levels of government and across industrial sectors can access, understand, and use information on current and future disaster risk. As part of this, federal agencies can use risk reduction strategies, such as providing technical assistance to help decision makers use climate information in their infrastructure investment decisions. Second, the framework stated that federal agencies can help decision makers use risk reduction strategies and prioritize all types of risk. For example, federal agencies can ensure that federal programs and policies that support disaster risk reduction are well coordinated. Third, the framework stated that federal agencies can provide decision makers at all levels of government and across sectors with incentives to make long- term, forward-looking risk reduction investments and remove barriers to such investments. Projected increases in the frequency, severity, and duration of extreme temperature changes or precipitation events, as well as rising sea levels, are among the potential impacts of climate change that may affect drinking water and wastewater infrastructure. The type and severity of these potential impacts on drinking water and wastewater infrastructure will vary by region. EPA, the USGCRP, NOAA, and other federal agencies have identified a variety of potential climate change impacts that may affect drinking water and wastewater infrastructure, as well as other critical and interconnected industries. EPA’s Adaptation Strategies Guide for Water Utilities (Guide) identifies five general categories of climate change impacts that can affect drinking water and wastewater utilities: ecosystem changes, droughts, floods, water quality degradation, and changes in service demand and use. Within these five general categories, EPA has identified specific climate change impacts that may affect drinking water and wastewater infrastructure systems. For example, degraded water quality from decreased stream flows may lead to higher treatment costs and the need for capital improvements to treat wastewater before discharging it from wastewater treatment facilities to meet more stringent regulatory requirements. Additionally, projected sea level rise can lead to saltwater intrusion in coastal groundwater aquifers and in estuaries. This may degrade water quality and increase treatment costs for drinking water treatment facilities or require new desalination facilities to treat water supplies with higher salt content. According to the Fourth National Climate Assessment, compound extreme events—the combination of two or more hazard events or climate variables (e.g., extreme rainfall and storm surge) that occur simultaneously or consecutively that lead to an extreme impact—have a multiplying effect on the risk to drinking water and wastewater infrastructure systems. Compound extreme events can also increase the risk of cascading infrastructure failure since some infrastructure systems rely on others and the failure of one system can lead to the failure of interconnected systems. This includes a water infrastructure system relying on the energy sector for power to operate pump stations and drinking water and wastewater treatment facilities. For example, during Hurricane Sandy in 2012, extreme rainfall coincided with high tides creating a storm surge. Hurricane Sandy caused power outages and flooding at eight of New York City’s 14 wastewater treatment facilities and 42 of the city’s 96 pumping stations. Further, power outages and flooding of wastewater treatment facilities and the large influx of floodwater in the sewer system resulted in the release of approximately 562 million gallons of untreated and diluted sewage into local waterways, as shown in figure 1. The Fourth National Climate Assessment states that drinking water and wastewater infrastructure in every region in the United States are sensitive to weather- and climate-related events and noted that the effects of such events will vary in severity and type by region, meaning different measures will be required to make infrastructure more resilient. The Fourth National Climate Assessment established 10 climate regions to better address the risks and needs of specific regions across the United States. Further, EPA’s Guide states that the type and severity of potential climate impacts on utilities will vary by region, and identifies the impacts that have the greatest likelihood of affecting utilities in the different regions and along the U.S. coast. For example, in the Southwest, increased duration and intensity of drought may stress water supplies and increase water demand for agricultural uses, increase energy requirements to treat and cool drinking water and wastewater effluent, and require investments in new water sources and options for reusing water. In the Northwest, increased water temperatures, as well as wildfires that create increased nutrient runoff, may degrade drinking water quality from higher levels of harmful toxins and algal blooms, and require drinking water utilities to develop increased treatment capabilities. The interactive map in figure 2 displays the 10 climate regions as established in the Fourth National Climate Assessment and the U.S. coasts, the most relevant potential climate change impacts for each region and the coast, and examples of the potential effects on drinking water and wastewater utilities, according to EPA’s Guide. One federal program—EPA’s Creating Resilient Water Utilities initiative— is designed to provide technical assistance to drinking water and wastewater utilities for planning climate resilient infrastructure, although the 15 selected utilities used a mix of sources, including other federal programs, to obtain technical assistance with understanding climate impacts and designing resilient infrastructure. To provide additional technical assistance for climate resilience, selected experts generally supported the option of developing a coordinated network of technical assistance providers including federal and state agencies, universities, consultants, and industry groups. Our review of the programs federal agencies’ used to provide technical assistance to 15 selected utilities to help make drinking water and wastewater infrastructure more climate resilient found that one program— EPA’s Creating Resilient Water Utilities initiative (CRWU)—was specifically designed to provide drinking water, wastewater, and stormwater utilities with the practical tools, resources, training, and technical assistance needed to increase resilience to extreme weather events. The initiative provides web-based tools and resources in the form of an interactive guide, a case studies map, a risk assessment tool, climate scenario projection maps, and storm surge inundation maps to help drinking water and wastewater utilities understand potential long- term risks and options to enhance their resilience to climate impacts, including extreme weather events. Furthermore, CRWU provides direct utility technical assistance and training through workshops and onsite exercises with utilities. As part of the initiative, EPA developed the Climate Resilience Evaluation and Assessment Tool (CREAT), a web- based application to assist drinking water and wastewater utilities in understanding potential climate change impacts and assessing the related risks to their systems. EPA also developed a Resilient Strategies Guide for Water Utilities, a web-based interactive guide to help drinking water and wastewater utilities identify resilience strategies to prepare for droughts, protect water quality, build flood protections, preserve ecosystems, maintain service levels, improve energy efficiency, implement green infrastructure, and conserve water. In addition, from 2010 through 2013, EPA collaborated with the Water Research Foundation and NOAA to publish the results of a series of workshops assessing the information and tools necessary to incorporate climate risks into utility planning. As part of a pilot program to help EPA develop CREAT, most of the drinking water and wastewater utilities we reviewed used CREAT to conduct climate risk assessments of their systems. Utility representatives said the tool was a helpful starting point for thinking about potential climate risks and vulnerable infrastructure qualitatively. For example, Bozeman Water and Sewer (Bozeman, Montana) used CREAT to assess potential consequences of drought, water quality changes, and wildfires on their drinking water assets and operations to better understand their systems’ vulnerabilities and start thinking about potential resilience measures. Keene Public Works (Keene, New Hampshire) also used CREAT to assess potential climate change impacts from extreme precipitation events on their water supplies and drinking water system and evaluate the performance and costs of additional short-term and long-term resilience measures. However, representatives from a few drinking water and wastewater utilities said they used additional assistance from consulting firms to help them use CREAT, and to complete assessments on the current and future climate risks to their infrastructure systems. A few other federal agencies have been involved in efforts to help utilities incorporate climate resilience into their planning, but their programs were not specifically designed to provide technical assistance to water utilities. NOAA’s Regional Integrated Science Assessments (RISA) program and the National Center for Atmospheric Research (NCAR) worked with utilities and EPA, through an effort called the Water Utility Climate Alliance which aims to enhance the quality and accessibility of regional climate change data to help improve water resource planning, develop adaptation strategies, and assist overall decision-making for water-related policies. The alliance, which was formed in 2007 and includes 12 of the nation’s largest drinking water utilities, provides leadership and collaboration on climate change issues affecting the country’s water agencies. The alliance collaborates with member agencies, federal agencies, industry groups, academia, and consulting firms to provide workshops on planning for climate change uncertainty for drinking water and wastewater sector professionals. Representatives from the New York City Department of Environmental Protection (New York City, New York) and the San Diego County Water Authority (San Diego, California), stated that through their membership in the alliance, they have used technical assistance from NOAA’s RISA program research teams and the Water Research Foundation to manage their climate risks. Specifically, in 2010, four Water Utility Climate Alliance members, including the New York Department of Environmental Protection (New York City, New York), contributed to a pilot project to better understand how climate change might affect their water systems through collaboration between climate experts and utilities, with the goal of improving the process of producing climate information utilities need for decision-making. Two RISA research teams, the Consortium on Climate Risk in the Urban Northeast at Columbia University and the Pacific Northwest Climate Impacts Research Consortium at Oregon State University, provided technical assistance on climate information and modeling to support the effort. In 2013, three Water Utility Climate Alliance members, including the San Diego County Water Authority (San Diego, California), contributed to a research study to increase the adaptive capacity of water utilities in planning for and responding to pressures that may result from climate change, particularly related to the demand for water. The Water Research Foundation led the study. We found that other federal programs offer technical assistance, but the assistance is either not targeted to drinking water and wastewater utilities or it is not specific to climate impacts. For example, San Diego Public Utilities (San Diego, California) worked with the Bureau of Reclamation to assess the region’s water supply and demand, determine the potential effects from climate change impacts within the region, and explore alternatives for addressing future water management challenges. Utilities in Estes Park, Colorado and Iowa City, Iowa worked with FEMA after flood events to develop long-term recovery plans that made their river pipeline crossings stronger and moved a wastewater treatment plant from the floodplain, respectively. In addition, several of the selected utilities worked with NOAA or the U.S. Geological Survey to collect data necessary for planning efforts, including monitoring weather and storms, rainfall levels from stream gauges, and salt water intrusion into water supplies. Houston Water (Houston, Texas) also used NOAA’s Atlas 14 Precipitation-Frequency Atlas to update its floodplain regulations and redefine the amount of rainfall it takes to qualify as a 100-year or 1,000- year flood event (see fig. 3 for pictures of flooded infrastructure). To date, federal efforts to provide technical assistance to help drinking water and wastewater utilities manage climate risks have been small- scale or pilot efforts to develop tools and information. For example, EPA’s CRWU has developed a number of tools and guides for utilities and has provided training and assisted a number of utilities, but the number of utilities that EPA helped directly is small—about 50—and EPA does not have the resources to provide assistance to all utilities, according to EPA officials. Similarly, the Water Utility Climate Alliance’s membership consists of 12 utilities in large metropolitan areas, and has focused on large utilities when developing examples of how drinking water utilities can plan for climate risks according to Water Utility Climate Alliance representatives; however, these alliance members are large enough to have in-house climate expertise and have established relationships with federal or university-based climate services providers. According to industry group officials, the majority of the 70,000 utilities across the country are small and do not have resources to work with consultants or research climate information. While water utilities used federal technical assistance, we found that almost all of the selected drinking water and wastewater utilities, regardless of size, used a mix of technical assistance providers including consultants, industry groups, academia, or federal programs to help them plan for resilience projects, as shown in appendix IV. Most of the selected utilities said they used a mix of assistance because they needed help understanding what climate information and climate models were appropriate to use for their regions and locales. For example, Anacortes Public Works (Anacortes, Washington) worked with the Skagit Climate Science Consortium—a nonprofit organization—to conduct a climate risk assessment for their drinking water system. Anacortes Public Works used the initial climate risk assessment to implement projects that will increase their resilience to the most significant effects from climate- related impacts, flooding, and increased sediment levels in their water supply (see fig. 4). Anacortes Public Works plans to work with the consortium again to better understand how rising sea levels and increasing salinity levels will affect their drinking water supply in the future. A few utilities said that technical assistance efforts should be a collaborative process between the utilities using climate information to make decisions and the scientists providing the technical assistance to ensure that climate information and models are what drinking water and wastewater utilities need to plan for climate resilience. All 10 of the selected experts we interviewed said that drinking water and wastewater utilities need additional technical assistance to manage climate risks. Specifically, these experts stated that utilities need technical assistance to use key climate information to incorporate climate resilience into their planning and operations. This information includes the following: forward-looking climate information and models to identify vulnerabilities to specific geographic regions; potential climate change impacts on regional and local socioeconomic and demographic trends for utility users; hydrologic information on the movement, distribution, and quality of water at the local, regional, and/or watershed level; and estimates of benefits and costs of incorporating resilience into utility projects. According to several of the selected experts we interviewed, such information is provided through a mix of sources, depending on what is available, and all sources are needed. Several selected experts also said that the utilities could obtain forward-looking climate information and models from federal agencies, such as NOAA, and could obtain information on potential climate change impacts from CREAT. In addition, several experts stated that they could obtain local socioeconomic and demographic data, hydrologic information, and benefit-cost information from industry sources, universities, and consultants. Several of the experts we interviewed also said that such assistance is not a one-time event, but requires consistent and continuous collaborative efforts between utilities and technical assistance providers. For example, several experts said that utilities need technical assistance on an ongoing basis to reevaluate their planning and operations regularly given the uncertainty associated with the severity of some potential climate risks. In addition, several experts said that individual utilities need help understanding which climate information and analytical tools are appropriate for assessing the climate risks specific to their regions or localities, and how to use them to manage climate risks to their infrastructure. Almost all of the experts said that small and rural utilities would need additional technical assistance to collect and use the information necessary to enhance their resilience to climate change impacts. Specifically, several experts said that, as opposed to many large utilities, small utilities lack the technical capacity to use climate information and do not have the financial resources to hire consultants or develop the internal expertise necessary to manage climate risks to their drinking water and wastewater infrastructure. Further, most of the selected experts we interviewed stated that a network of providers would be needed to provide assistance to water utilities. This is consistent with what we and others have previously reported. For example, we reported in November 2015 that clearly organized technical assistance would improve federal climate information efforts by helping different types of decision makers—ranging from those who can define their needs to those who have limited experience using climate information—access, translate, and use climate information. We also found that key stakeholders and relevant studies generally called for a system of nonfederal technical assistance providers, with federal leadership to help federal, state, and local decision makers, including utility decision makers, use climate information. In addition, a 2014 task force of state, local, and tribal leaders stated that the greatest need for enhancing climate resilience is often not the creation of new data or information, but assistance and tools for decision makers, including utility managers, in navigating the wide array of resources already available. Further, in August 2019, the National Mitigation Investment Strategy recommended that the federal government increase investment in hazard mitigation by building the capacity of communities to address their risks, including climate-related risks. To implement the recommendation, the strategy said that the federal government should create a professional network to encourage collaboration and information sharing across different levels of government and the water and wastewater sector, and that the federal government and its nonfederal partners should work together to develop a pool of skilled mitigation professionals. The following is a list of options for providing a network of technical assistance providers that selected experts we interviewed discussed, as well as the advantages and disadvantages of each. Existing utility technical assistance providers. A strengthened and expanded network of existing federal technical assistance providers, including EPA’s Environmental Finance Centers, USDA’s Rural Utilities Service, the National Rural Water Association, and the Rural Community Assistance Partnership, could help consolidate climate information and provide technical assistance to utilities to improve their resilience. Most experts said that a network of existing utility technical assistance providers would have the advantage of established relationships with communities and utilities or could ensure that small and rural utilities obtain needed information and assistance to improve their resilience to climate change. However, several experts said that the network may lack the expertise necessary to effectively identify or develop climate information and planning tools to provide the technical assistance necessary to meet the specific needs of utilities to improve their resilience. See appendix V for additional information on these programs. Existing federal climate services providers. A strengthened and expanded network of existing federal climate services providers, such as USDA’s Climate Hubs, Interior’s Landscape Conservation Cooperatives, Interior’s Climate Science Centers, and NOAA’s RISA program could provide technical assistance to utilities to improve their resilience. Several experts said that a network of existing federal climate services providers would have a good understanding of the available climate information and would, for example, be best positioned to develop the specific tools and guidance necessary to provide the technical assistance utilities need to improve their resilience. In contrast, several experts said that federal climate services providers may not have the established relationships with utilities necessary to understand and tailor technical assistance to the needs of individual utilities. In addition, one expert said that the climate services providers may not have the funding to provide these services to utilities in a comprehensive way. See appendix V for additional information on these programs. Universities and university-based research centers. A new network of academic or university-based technical assistance providers, such as NCAR, organized by state, region, or watershed could provide technical assistance to all types of utilities to improve their resilience. According to several experts, this option would be advantageous because many universities and centers already have the technical capacity to use climate information to provide risk assessment and planning tools necessary to provide technical assistance to utilities at the local or regional level. Several experts also said that it would be cost-effective to expand this option because some universities and centers are already providing technical assistance. However, several experts said that without a clear shift in federal incentives to prioritize the applied research necessary to provide the technical assistance that utilities need, universities and centers are unlikely to provide sustained assistance nationwide. Similarly, several experts said that federal coordination would be needed to ensure that the universities and centers were consistently providing information, planning tools, and assistance that meet the specific needs of utilities. See appendix V for additional information on these programs. Industry groups and private-engineering consultants. A new network of nonfederal industry and nonprofit groups, such as the American Water Works Association and the Association of Municipal Water Utilities, could provide technical assistance to utilities to improve their resilience. Several experts said that this option would be advantageous because it could leverage existing relationships, for example, to strengthen information sharing between utilities regarding the best available climate information and approaches to resilience planning. In addition, several experts said that industry groups and private engineering consultants would have a better understanding of utility operations and management when compared to other options for providing technical assistance. In contrast, half of the experts said that this network would need additional federal oversight and coordination. For example, several experts said that there would need to be a certification process for industry groups and private consultants to ensure that the technical assistance being provided to utilities was sufficient and transparent. In addition, several experts said that the network would not be effective unless it was coordinated among stakeholders from, for example, the private sector; industry groups; and federal, state, and local governments. A network of utilities. A network of utilities, similar to the Water Utility Climate Alliance, could consolidate and update information and provide technical assistance for all types of utilities to improve their resilience. Similar to a network of industry groups and consultants, several experts said a network of utilities could help coordinate and strengthen information sharing between utilities on best practices and lessons learned from resilience planning. However, several other experts said that it would be difficult to develop and expand a network of utilities that was capable of providing technical assistance to utilities of different sizes or geographic locations. One expert also said that utilities that provide technical assistance would need to be certified by the federal government, academics, or industry groups to ensure the technical assistance being provided to utilities was sufficient. When asked how they would design a network to provide technical assistance, most experts supported an approach in which federal agencies organized a network of technical assistance providers for drinking water and wastewater utilities, a network that would include federal and state agencies, universities, consultants, and industry groups. For example, one expert said that EPA and other relevant federal agencies could provide guidance and leadership for a network of (1) university and federal climate services providers that would assess the risks that potential climate impacts pose to utilities and (2) utility technical assistance providers, including consultants and industry groups, to help utilities apply those assessments to their infrastructure to make it more resilient. Another expert said that existing networks of universities, industry groups and consultants, or utilities would not be as effective unless they were part of a larger networked effort with clear leadership that provides continuous technical assistance to utilities. Similarly, several experts stated that it was important that the network be a collaboration of different technical assistance providers to be able to tailor the technical assistance to the needs of different types of utilities, in different locations, with differing technical capabilities. For example, one expert said that universities, industry groups, and federal programs have different levels of resources and expertise in different regions of the country and a coordinated network could help utilities identify the sources of technical assistance in their regions or localities. Further, another expert said that it was important that the network have the capability to help utilities understand and respond to climate risks that other types of infrastructure create. Specifically, the expert said that while EPA has a role in regulating drinking water and wastewater infrastructure, the agency does not regulate larger-scale infrastructure, such as dams and reservoirs that need to be operational to reduce risks to utilities. Under Presidential Policy Directive 21, EPA, as the Sector Specific Agency for the water and wastewater systems sector, is to work to enable efficient information exchange between federal agencies and infrastructure owners and operators, and to implement an integration and analysis function to inform planning and operational decisions regarding critical infrastructure. In addition, one of the key activities of the Water Sector Government Coordinating Council, which EPA chairs, is to facilitate information sharing between federal, state, and local decision makers on critical infrastructure protection. This is consistent with our disaster resilience framework, which states that federal efforts should improve the availability of authoritative, understandable, and comprehensive information on disaster risks and risk reduction strategies to help entities effectively assess their climate risks, determine what viable alternatives are available to increase resilience to those risks, and better understand and measure the impact of resilience strategies. Our framework also states that federal efforts can help by providing technical assistance and capacity building to nonfederal partners. To date, however, federal efforts to provide technical assistance to drinking water and wastewater utilities do not provide the ongoing technical assistance that according to experts utilities need to plan and build climate resilient infrastructure. In addition, current efforts may not be widespread enough to provide comprehensive coverage of the drinking water and wastewater utilities across the nation. The 2017 Roadmap shows actions for the short term (2 years) and midterm (5 years), but it does not include actions such as developing guidance on technical assistance, building networks of technical assistance providers, or developing other methods to help utilities build capacity to manage their climate change risks and plan for resilient infrastructure. According to EPA officials we interviewed, the agency has worked within its existing authorities and available resources to prioritize developing voluntary guidance, tools, training, and webinars that utilities can use to identify potential risks from climate change and plan to improve their resilience. Further, EPA officials said that while the agency has collaborated closely with key federal, state, and local decision makers; industry groups; and utilities in its role as chair of the Water Sector Government Coordinating Council, the council has focused on other short-term threats to utilities, such as disasters and terrorism, and has not assessed how it could develop and coordinate a network to effectively provide the technical assistance that utilities need to enhance their climate resilience. By identifying and engaging existing technical assistance providers in a network to help drinking water and wastewater utilities incorporate climate resilience into their projects and planning on an ongoing basis, EPA would have more reasonable assurance that climate information was effectively exchanged among federal agencies and infrastructure owners and operators. Supporting the need for a broader collaborative approach, several of the selected utilities are already members of organizations that coordinate and collaborate among members and various technical assistance providers, including federal agencies, to understand the potential climate impacts for their regions, use similar climate models, and share best practices for projects to enhance climate resilience. For example, the Southeast Florida Regional Climate Change Compact is a decade-old partnership between Miami-Dade, Broward, Monroe, and Palm Beach Counties to coordinate mitigation and adaptation activities across county lines in response to the effects of climate change, including sea level rise, flooding, and economic and social disruptions. The compact and its partners work with various federal, state, regional, municipal, nonprofit, academic, and private sector partners to provide technical assistance and support for utilities in southeast Florida to help the region identify emerging issues and all move in one direction for resilience planning efforts. The supporting federal agencies include NOAA, EPA, and the Army Corps of Engineers. Another example is Charleston Water (Charleston, South Carolina), a member of the Charleston Resilience Network—a collaborative group of public, private, and nonprofit organizations in the region that work to increase resilience of communities, critical infrastructure, and the economy to natural disasters and chronic coastal hazards, such as rising sea levels. The network provides a forum to share science-based information, educate stakeholders, and enhance long-term resilience planning decisions. The network also works to provide consistent information for planning decisions. The federal agencies that advise the Charleston Resilience Network include NOAA, DHS, and the Army Corps of Engineers. The four selected federal agencies in our review provide broad financial assistance to help drinking water and wastewater utilities plan and build infrastructure projects. The agencies have taken some actions to promote climate resilience when providing financial assistance for water infrastructure projects, but they do not consistently include the consideration of climate resilience when funding such projects. Most selected experts we interviewed suggested that requiring the consideration of climate change risks in the planning and design of all federally funded water and wastewater infrastructure projects could help enhance climate resilience and limit future federal fiscal exposure. The four federal agencies we reviewed have nine programs that provide broad financial assistance, through loans or grants, for drinking water and wastewater infrastructure (see table 1). However, federal programs generally do not have selection criteria or requirements for utilities to incorporate climate resilience in the planning and design of projects that receive federal financial assistance. Each of the programs used different selection criteria for providing financial assistance to drinking water and wastewater utilities. EPA’s Drinking Water State Revolving Fund, Clean Water State Revolving Fund, and Water Infrastructure Finance and Innovation Act (WIFIA) programs generally provide financial assistance to projects that address the most serious risks to human health and ensure compliance with the Safe Drinking Water Act or Clean Water Act. Other programs, such as FEMA’s Public Assistance and Hazard Mitigation Grant programs provide financial assistance to repair or replace infrastructure damaged during natural disasters, or to enhance disaster resilience against future damage. HUD’s Community Development Block Grant-Disaster Recovery funding is used for, among other things, projects to help cities, communities, and states recover from presidentially-declared disasters or enhance disaster resilience of damaged infrastructure, especially in low- income areas. USDA provides financial assistance for drinking water and wastewater infrastructure in small and rural communities. According to EPA, FEMA, HUD, and USDA officials we interviewed, drinking water and wastewater utilities can use financial assistance from their programs to pay for projects that, in addition to other benefits, can help enhance climate resilience. We have previously reported that the federal government invests billions of dollars annually in infrastructure—such as roads, bridges, and wastewater infrastructure—but faces increasing risks from climate change. When the climate changes, infrastructure—typically designed to operate within past climate conditions—may not operate as well or for as long as planned, leading to economic, environmental, and social impacts. We have also reported that some federal agencies have made efforts to manage climate change risk within existing programs and operations—a concept known as mainstreaming—and these efforts may convey some climate resilience benefits. For example, an agency planning to build a seawall to protect a coastal facility might build it higher to account for rising sea level projections, but may not track this spending as related to climate change. Representatives of several of the drinking water and wastewater utilities we reviewed reported using selected federal financial assistance programs to help fund projects for fiscal years 2011 through 2018 that, in addition to other benefits, enhanced their climate resilience. For example, Iowa City Public Works used financial assistance from HUD Community Development Block Grant-Disaster Recovery funding and FEMA’s Public Assistance grant program to increase their resilience to floods by relocating a flood-prone wastewater treatment facility after flooding in 2008, as shown in figure 5. Similarly, as of December 2018, Houston Water was working with FEMA to use Public Assistance grants and Hazard Mitigation grants to increase the utility’s resilience to floods and extreme storm events when rebuilding the wastewater infrastructure damaged by Hurricane Harvey in 2017, according to Houston Water representatives. In addition, the San Diego Public Utilities Department received an EPA WIFIA loan to increase its resilience to droughts by building a new recycled wastewater treatment facility that will provide an additional source of drinking water and reduce the need for water imported from the Colorado River Basin (see app. VI for details on completed and ongoing infrastructure projects that utilities undertook to enhance their climate resilience, according to selected drinking water and wastewater utility representatives). The remaining selected utilities relied on other sources of funding such as municipal bonds and funds raised primarily through user rates and fees for fiscal years 2011 through 2018 to enhance their climate resilience (see app. VII for details on the financial assistance drinking water and wastewater utilities used for infrastructure projects). However, making the nation’s drinking water and wastewater infrastructure resilient will be expensive, costing anywhere from $448 billion to $944 billion, including operations and maintenance through 2050, according to a 2009 Association of Metropolitan Water Agencies study, the most recent such study. These costs would likely be in addition to the EPA-estimated $774 billion in costs for replacing and repairing existing infrastructure over the next 20 years. According to representatives of several of the selected utilities in our review, additional financial assistance will be necessary to enhance the resilience of drinking water and wastewater infrastructure. Representatives from several utilities said they would not be able to make the necessary upgrades to incorporate climate resilience into their drinking water or wastewater systems without additional grant assistance. Based on estimates from one of the selected utilities, the costs to enhance their resilience will be high. For example, in 2013, the New York Department of Environmental Protection estimated that it would cost about $315 million to build the protective measures necessary to make its wastewater treatment facilities and pump stations resilient to future flood projections. Officials from EPA, FEMA, HUD, and USDA said that federal agencies have taken action to change program requirements or selection criteria to provide financial assistance for projects that enhance climate resilience. However, according to federal officials, some federal agencies are providing financial assistance to utilities for projects that do not consider climate resilience in their planning and design consistently. In addition, federal officials stated that their ability to require that climate resilience be incorporated in the projects they fund is limited by requirements specific to their programs. Examples of their efforts, and limited authorities, include the following: EPA. EPA provides annual grants to states to capitalize their state- level drinking water and wastewater state revolving fund programs. The states use the revolving funds to provide low-cost loans or other financial assistance to communities for, among other things, a wide range of drinking water and wastewater infrastructure projects. According to EPA officials, states establish program criteria and do not consider climate resilience consistently in planning and designing projects that receive financial assistance from state revolving fund programs. Specifically, EPA officials said that despite agency efforts to promote climate resilience, states have discretion in setting project funding criteria and priorities for their state revolving fund programs, and that the agency does not have the authority to require that states prioritize projects that incorporate climate resilience. EPA continued to encourage the states to incorporate resilience planning in their priority systems. In documents released in May 2016, September 2016, and June 2017, the EPA described the types of climate resilience projects eligible for drinking water and clean water state revolving fund assistance. The September 2016 document also describes how programs can encourage resilient infrastructure through financial incentives. According to fiscal year 2015 data that EPA provided, 17 state clean water revolving fund programs have created additional financial incentives that utilities could use to fund climate resilience projects, and only New York’s program requires that climate risks from sea level rise be incorporated into the projects that receive financial assistance. In addition, utilities have discretion in whether to incorporate climate resilience into their state revolving fund project applications, and EPA cannot require utilities to incorporate climate resilience into the planning and construction of projects that states fund, according to EPA officials. Similarly, while EPA manages the WIFIA program and its application process and criteria, EPA officials said that the 2018 and 2019 program guidance did not prioritize protection against the impacts of climate change in its selection criteria, and that the agency does not require that applicants incorporate climate resilience into project planning and design. FEMA. FEMA’s Public Assistance Grant Program provides grants to state, tribal, territorial, and local governments, and nonprofits that can be used to repair and replace damaged infrastructure, including drinking water and wastewater infrastructure. In addition, FEMA’s Pre- Disaster Mitigation and Hazard Mitigation Grant Programs can provide financial assistance to states, communities, or tribes that can be used to reduce the risks to drinking water and wastewater infrastructure from future disasters. FEMA officials said they have developed guidance for states and communities to incorporate climate resilience into the planning for projects funded by all three programs. However, officials said that states and utilities do not consider climate change resilience consistently in planning and designing of projects that use financial assistance from FEMA. Specifically, according to FEMA officials, funding through the Public Assistance Program and the Hazard Mitigation Grant Program is limited to states and localities with a presidentially-declared disaster and generally is not provided for projects that incorporate climate resilience into their planning and design. In addition, according to FEMA officials, states and localities have discretion over the projects they choose to submit for funding and FEMA cannot require them to incorporate climate resilience into the planning and construction of projects that states fund without a change to program requirements. HUD. HUD provides grants to states and local governments through its Community Development Block Grant program to fund housing; economic development; neighborhood revitalization; and other community development activities, including drinking water and wastewater infrastructure. In addition, HUD can provide grants that can be used for reconstruction of drinking water and wastewater infrastructure to help communities recover from presidentially declared disasters through its Community Development Block Grant program. According to HUD officials, the agency has taken action to encourage states and local governments to incorporate climate resilience planning in the projects they fund after disasters. Officials also said that HUD provides guidance on how financial assistance requirements for states and entitlement communities can be waived so that states and communities can use Community Development Block Grant funding for disaster recovery and resilience in presidentially-declared disaster areas. In addition, in 2016, HUD finalized rules requiring states and localities to consider incorporating resilience to natural hazard risks and climate change into their planning documents for Community Development Block Grant funding in low- and moderate-income communities. However, officials said that states do not consider climate change resilience consistently when planning and designing projects using financial assistance from HUD. Specifically, according to HUD officials, the agency can only directly provide financial assistance to projects that enhance climate resilience using Community Development Block Grant-Disaster Recovery Grants if climate change resilience is specified in disaster relief appropriations language. Further, states and localities have discretion regarding whether to incorporate climate resilience into their project applications, and HUD cannot require them to incorporate climate resilience into the planning of projects that receive financial assistance, according to HUD officials. USDA. USDA’s Rural Utilities Service provides grants and loans for drinking water, wastewater, and stormwater projects in rural areas— defined as any area not in a city or town with a population in excess of 10,000 inhabitants. According to USDA officials, the agency has promoted climate resilience planning through its Water and Waste Disposal Program by requiring small and rural utilities to complete planning and vulnerability assessments for natural disasters. In addition, USDA officials said the agency has collaborated with EPA to develop guidance and training through the Sustainable Rural and Small Utility Management Initiative to help small and rural utilities create plans for improving their sustainability, including planning to help make the utilities resilient to potential climate impacts. According to USDA officials, utilities have discretion in whether to incorporate climate resilience into their Water and Waste Disposal project applications, and USDA cannot under its current regulations require them to incorporate climate resilience into the planning and construction of projects that receive financial assistance. As a result, according to officials, utilities do not consider climate resilience consistently when planning and designing projects that receive financial assistance from USDA. According to most selected experts, requiring the consideration of climate risks in projects that receive financial assistance will help limit the future fiscal exposure of the federal government and help enhance the climate resilience of drinking water and wastewater infrastructure. Specifically, most of the experts we interviewed said that a federal requirement that potential climate impacts be considered and, if necessary, incorporated into the design of all new drinking water and wastewater infrastructure projects that receive federal financial assistance, should be a high or very high priority for the federal government. Several of the experts said that this option would be advantageous because it could help ensure more effective and efficient use of federal dollars on drinking water and wastewater infrastructure. For example, several experts said that this option would help ensure that infrastructure funded by the federal government incorporated climate risks during the planning stages, helping avoid expensive retrofits or the abandonment of federally funded infrastructure that was not climate resilient. Several other experts said that such a federal requirement could help make consideration of future climate risks to enhance resilience a standard industry practice within the drinking water and wastewater sector. Several of the selected utilities said that a federal requirement for potential climate impacts to be considered and, if necessary, incorporated into the design of all new drinking water and wastewater infrastructure projects that receive federal financial assistance, would be moderately to extremely effective in helping utilities enhance their resilience. These selected utilities also said that it would be at least moderately feasible to implement. Several of the selected utilities are already required to consider some potential climate risks in the planning and design of their drinking water and wastewater infrastructure. For example, according to representatives from the Miami-Dade County Water and Sewer Department, Miami-Dade County adopted an ordinance requiring that potential climate risks be considered in the design of county-funded infrastructure. According to the same officials, this requirement has shifted the culture of the Miami-Dade County Water and Sewer Department to emphasize potential future climate change risks in the planning and design of all of the county’s drinking water and wastewater infrastructure. Representatives from a few selected utilities also said that a requirement could make it easier to access federal financial assistance programs for projects that enhance climate resilience. Several selected experts cautioned that many utilities do not have the climate information and technical capacity to carry out such requirements or that the uncertainty of the available climate science would make it difficult to implement for some utilities. In addition, several experts said that such a requirement may force utilities with limited funding to prioritize planning and investment in projects to improve climate resilience over more pressing concerns, such as repairing and replacing damaged or obsolete infrastructure. Several selected utilities said that it will be difficult to implement these new requirements, but added that additional technical and financial assistance could help. For example, representatives from Cottage Grove Public Utilities said that the federal government will need to provide additional financial and technical assistance opportunities for small and medium-sized public utilities that do not have the capacity to plan, implement, and fund large climate resilience projects. However, if the federal agencies do not require the incorporation of climate resilience into the projects that receive financial assistance, they may continue to fund drinking water and wastewater infrastructure projects that may be damaged or incapacitated by future floods, drought, water quality problems, and other climate change impacts. This increases the risk that critical infrastructure will not be well protected and drinking water and wastewater utilities will not be able to continue operations that provide critical public health and environmental services to the public. EPA and other federal, state, local, and sector-level officials, recognizing the need to incorporate climate resilience into drinking water and wastewater infrastructure, have taken action to promote climate resilience but generally do not require it to be incorporated in these projects. Specifically, the 2017 Roadmap calls for the Water Government Coordinating Council and the Water Sector Coordinating Council to promote eligibility criteria for financial assistance programs to support resilience activities by 2019. In addition, in a 2019 report, EPA’s Environmental Finance Advisory Board recommended that EPA create a coordination group to set priorities and reduce gaps in funding predisaster resilience for drinking water and wastewater infrastructure, and that EPA consider expanding the state revolving fund program to include financial assistance for flooding and storm-related damages. Further, the National Mitigation Investment Strategy, issued in draft in January 2018, and finalized in August 2019, states that successful mitigation of natural hazard risks requires shared priorities, consistent approaches, aligned funding, expanded incentives, and coordination between the federal government and nonfederal partners. It also states that the federal government and nonfederal partners should look at risk and resilience consistently by, for example, having similar requirements for assessing risk and rebuilding for long-term resilience. It emphasizes the need to focus on critical infrastructure in communities, such as drinking water and wastewater infrastructure. Incorporating climate resilience likely decreases the risk that water and wastewater infrastructure, some of which is paid for with federal financial assistance, will fail during extreme events. According to the National Research Council, as the climate changes and historical patterns—in particular, those related to extreme weather events—no longer provide reliable predictions of the future, infrastructure designs may underestimate the climate-related impacts to infrastructure over its design life, which can range as long as 50 to 100 years. In April 2013, we reported that according to one set of commonly used design standards, wastewater treatment plant components are typically designed for 25-, 50-, or 100-year storms. We reported that changes in characteristics of strong storms—for instance, a storm that historically occurred once every 100 years may occur every 50 years in the future—could cause wastewater management systems to be overwhelmed more frequently. Incorporating climate resilience into drinking water and wastewater infrastructure projects also likely decreases the risk that the federal government will need to pay to repair and replace damaged facilities. In our previous work, we said that building resilience can help reduce the federal fiscal exposure. As we reported in April 2013, such resilience means reducing potential future losses rather than waiting for an event to occur and paying for recovery afterward. We said that enhancing resilience can create additional up-front costs, but can also reduce potential future damage from climate-related events that—given expected budget pressures—would otherwise constrain federal programs. In 2018, the National Institute of Building Sciences found that every dollar spent on infrastructure hazard mitigation to enhance resilience to wind- and flooding-related disasters resulted in 7 to 8 dollars in avoided future losses, respectively. This potential can be considered in light of recent costs that the federal government incurred to address losses. In particular, from fiscal year 2011 through fiscal year 2018, we estimate that FEMA’s Public Assistance program and HUD’s Community Development Block Grant-Disaster Recovery Grants have obligated at least $2.3 billion and at least $1.4 billion, respectively, in federal disaster recovery funding on drinking water and wastewater infrastructure-related projects. Drinking water and wastewater utilities face challenges in using climate information to identify actions that they can take to enhance their climate resilience. At the moment, utilities obtain technical assistance and use climate information from a mix of sources and that assistance is not organized to help ensure more comprehensive coverage of the more than 70,000 drinking water and wastewater utilities across the nation. As designated lead agency for the resilience and security of the drinking water and wastewater sector and as chair of the Water Sector Government Coordinating Council, EPA is tasked with coordinating federal and sector efforts to provide the information and assistance that state and local decision makers—including utilities—need to enhance their climate resilience. The councils have identified a number of actions to support the drinking water and wastewater sector, but EPA, other federal agencies, and the water and wastewater sector, have not assessed how they could organize a network of technical assistance providers to effectively provide the assistance that utilities need to enhance their resilience to climate change. By identifying existing technical assistance providers and engaging them in a network to help drinking water and wastewater utilities consider climate resilience in the planning and design of projects on an ongoing basis, EPA, as chair of the Water Sector Government Coordinating Council, would have more reasonable assurance that climate information was effectively exchanged among federal agencies and infrastructure owners and operators. In recognition of the federal interest in protecting the health and economic benefits that clean and safe water provide, federal programs provide funding to support drinking water and wastewater infrastructure. In 2013, Presidential Policy Directive 21 identified the water and wastewater sector as critical infrastructure, with important implications for protecting and investing in that sector. Federal agencies such as EPA, FEMA, HUD, and USDA provide financial assistance to help ensure the long-term success of drinking water and wastewater utilities. These agencies have taken action to promote climate resilient infrastructure projects with the financial assistance they provide, but their abilities to ensure that projects receiving financial assistance are resilient are limited. To enable agencies to further drive climate resilient investments by drinking water and wastewater utilities, changes would be needed to programs that EPA, FEMA, HUD, and USDA administer to require that climate resilience be incorporated into planning for projects that receive federal financial assistance. Such changes could help ensure that drinking water and wastewater infrastructure projects that receive federal financial assistance adequately address risks from climate change and ensure that utilities carry out their critical operations. Such changes could also help limit the fiscal exposure to the federal government for future recovery costs. We are making the following matter for congressional consideration: Congress should consider requiring that climate resilience be incorporated in the planning of all drinking water and wastewater projects that receive federal financial assistance from programs that EPA, FEMA, HUD, and USDA administer. (Matter for Consideration 1) We are making one recommendation to EPA: The Director of Water Security of EPA, as Chair of the Water Sector Government Coordinating Council, should work with the council to identify existing technical assistance providers and engage these providers in a network to help drinking water and wastewater utilities incorporate climate resilience into their projects and planning on an ongoing basis. (Recommendation 1) We provided a draft of this report to EPA, DHS, HUD, NOAA, and USDA for review and comment. EPA provided written comments, which are reproduced in appendix VIII. The other four agencies did not provide comments on our draft report. EPA and USDA provided technical comments, which we incorporated as appropriate. In its written comments, EPA neither agreed nor disagreed with our recommendation that the Administrator, as Chair of the Water Sector Government Coordinating Council, should work with the council to identify existing technical assistance providers and engage these providers in a network to help drinking water and wastewater utilities incorporate climate resilience into their projects and planning on an ongoing basis. The agency noted in its technical comments that the Director of Water Security is the chair of the Water Sector Council, not the administrator. We made this change in the report. In its written response, EPA made two points related to the recommendation. First, it stated that it will, consistent with our recommendation, continue to work with its wide-ranging, existing technical assistance providers and coordinate with its stakeholders to identify additional providers as applicable. We agree with this approach and highlighted several of these efforts in our report. For example, EPA noted that it provides annual training to over 5,000 water utilities, state officials, and federal emergency responders on how to become more resilient to natural or manmade incidents that could endanger water and wastewater services. Second, in response to the part of the recommendation that EPA engage the providers in a network, the agency noted that states serve as a coordinating entity under its Small System Training and Technical Assistant grants. Further, EPA also noted that the providers work with states to identify the systems in greatest need of assistance and identify the training topics of greatest need for small public water systems. We agree that this could be a helpful approach, but note that EPA remained silent on how it plans to work with the states and the water and wastewater sector to develop a network of technical assistance providers. Our report showed that utilities obtain technical assistance from a number of different sources and that they could benefit from a larger network with continuous technical assistance. The Water Sector Coordinating Council functions as a forum to coordinate members of existing networks, and to ensure they have the most current and relevant information as they provide assistance to utilities. As EPA works with its wide-ranging technical assistance providers, consistent with our recommendation, we would encourage it to also work with the Water Sector Coordinating Council to ensure the coordination of the different networks that exist in the water and wastewater sector. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees; the Administrator of the Environmental Protection Agency; and the Secretaries of Homeland Security, Housing and Urban Development, Commerce, and Agriculture. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made key contributions to this report are listed in appendix IX. The objectives of our review were to examine (1) the potential impacts of climate change and the effects of these impacts on drinking water and wastewater infrastructure; (2) technical assistance selected federal agencies provided to selected utilities to help make drinking water and wastewater infrastructure more resilient to the impacts of climate change, and options experts identified for providing additional technical assistance to utilities; and (3) financial assistance federal agencies provided to selected utilities to help make drinking water and wastewater infrastructure more resilient to the impacts of climate change, and options experts identified for providing additional financial assistance to utilities. For the first objective, we reviewed the Fourth National Climate Assessment; the Environmental Protection Agency’s (EPA) Adaptation Strategies Guide for Water Utilities, Climate Resilience Evaluation and Awareness Tool Methodology Guide, and Climate Scenarios Projection Map, and the U.S. Climate Resilience Toolkit, which the Department of Commerce’s National Oceanic and Atmospheric Administration (NOAA) manages and hosts with oversight from the U.S. Global Change Research Program. Based on our review of these sources, we first identified different categories of potential climate change impacts, and how those impacts may vary in the different climate regions identified in the Fourth National Climate Assessment. For both the second and third objectives, we reviewed the efforts of and interviewed five federal agencies and 15 drinking water and wastewater utilities. We reviewed our previous reports to identify agencies that provide technical assistance or financial assistance, or both, to drinking water and wastewater utilities and identified five agencies: EPA, NOAA, the Department of Homeland Security’s (DHS) Federal Emergency Management Agency (FEMA), the Department of Housing and Urban Development (HUD), and the Department of Agriculture’s (USDA) Rural Utilities Service. For the second and third objectives, we also selected a nongeneralizable sample of 15 drinking water and wastewater utilities in 13 communities using a stratified purposeful sampling approach. We selected utilities to obtain variation in their size and climate region to capture similarities and differences among utilities. We classified utilities into small, medium, and large utilities based on the sizes of the populations that they serve. We defined small utilities (serving populations of 10,000 or less), medium utilities (serving populations of 10,001 to 999,999), and large utilities (serving populations of 1 million or more) for this report to capture utilities with the greatest resources available for climate resilience efforts. In order to ensure geographic diversity, we selected small, medium, and large utilities from different climate regions identified in the Fourth National Climate Assessment. Because this was a nonprobability sample, the findings related to the 15 utilities cannot be generalized to all drinking water and wastewater utilities but provide illustrative examples of how the selected utilities used federal technical assistance and financial assistance. Further, for the second and third objectives, we selected 10 experts in the climate change and disaster fields to interview about options for providing additional technical and financial assistance to drinking water and wastewater utilities. To identify experts on the resilience of water infrastructure to climate change, we searched Elsevier’s Scopus database for peer-reviewed articles published from January 2003 through September 2018 searching titles, abstracts, and keywords for “drinking water” or “wastewater” in close proximity to terms such as “infrastructure,” “climate change,” and “resiliency.” We identified approximately 300 studies from this search, identified the relevant studies from that group, and then found an additional eight studies from their citations. We reviewed the abstracts of these studies and found 96 that were within the scope of our objectives. To develop a list of potential experts, we extracted the names of the authors of these studies and the names of authors cited in these studies using the Python programming language and the Scopus Application Programming Interface. Next, we used statistical software to calculate the number of times that each author cited every other author. Using these calculations, we arrayed the authors into a network graph, in which authors who frequently cited each other were situated closer together and authors who did not cite each other were situated further apart. We analyzed this network using social network analysis techniques. Specifically, to measure each author’s prominence in the network, we calculated the number of times that each author was cited in the articles written by other authors in the network. To divide the network into groups, we used an algorithm known as hierarchical clustering. This algorithm allowed us to identify groups of authors who cited each other frequently and who cited authors in the rest of the network infrequently. We sorted authors by group and by the number of times they were cited. For the most frequently cited authors in the largest groups in the network, we examined biographical details and publication details via web searches, such as their geographic location and the relevance of their publications to our research topic. We selected a final list of 15 frequently cited experts who were primarily from the largest clusters in the network, who were based in North America, whose research was topically relevant, and who were still active in the field. Eight of these experts agreed to be interviewed and we included them in our final sample. We supplemented this list with two experts who served as lead authors for the water chapter of the Fourth National Climate Assessment. While these 10 experts are prominent researchers and correspond to a range of major fields of research on the topic, their views do not represent the views of all experts on the resilience of drinking water and wastewater infrastructure to climate change. To examine the first part of the second objective, the technical assistance selected federal agencies provided to selected utilities, we reviewed relevant laws, regulations, and planning guidance about programs that can provide technical assistance to drinking water and wastewater utilities to help enhance climate resilience for each selected federal agency. We also interviewed federal officials at each agency. To examine the first part of the third objective, the financial assistance selected federal agencies provided to selected utilities, we reviewed project eligibility criteria and appropriation amounts for EPA’s Clean Water State Revolving Fund, Drinking Water State Revolving Fund, and Water Infrastructure Finance and Innovation Act Programs; HUD’s Community Development Block Grant Program and Community Development Block Grant-Disaster Recovery Fund; and USDA’s Water and Wastewater Disposal Program for fiscal years 2011 through 2018. We also interviewed federal officials at each agency. As part of analyzing the federal financial assistance to drinking water and wastewater utilities, we estimated FEMA’s pre- and post-disaster spending to help such utilities recover from natural disasters. To identify federal disaster recovery and hazard mitigation obligations on drinking water and wastewater infrastructure, we analyzed federal financial assistance that FEMA’s Public Assistance, Hazard Mitigation, and Pre- disaster Mitigation Programs provide for disaster recovery for drinking water and wastewater infrastructure. Specifically, using a list of search terms associated with drinking water and wastewater infrastructure, we queried FEMA’s disaster recovery spending database to identify a list of drinking water and wastewater infrastructure disaster recovery and hazard mitigation projects funded from fiscal years 2011 through 2018. After we queried FEMA’s disaster recovery spending database, we manually reviewed records from a stratified sample to ensure that each project was related to water and wastewater infrastructure. We reviewed all 25 records with the highest obligated amounts, 15 records in which a project was associated with more than one site, and 35 records in which a project was associated with just one site. We chose this sample design to ensure that we were capturing projects with the highest dollar amounts as well as all other projects, while also ensuring that if one site in a project was water related, the rest of the sites under the project were also water related (manual review showed that if one site in a project was water related, 98 percent of the other sites in the project were also water related). After manual review, we generated an estimate of total obligated funds from the ratio of number of projects that we reviewed that were related to water and wastewater infrastructure to the total number of projects in our sample. The estimate we used was the lower bound of a 95 percent confidence interval. We chose this estimate in order to give a conservative estimate of the amount that FEMA’s public assistance program has obligated. The relative error was 0.07. To assess the reliability of the disaster recovery obligations data, we (1) performed electronic testing for errors in accuracy and completeness, (2) reviewed related documentation about the data and the system that produced them, (3) interviewed agency officials knowledgeable about the data, and (4) worked closely with agency officials to identify and resolve data discrepancies before conducting our analyses. We determined that the data were sufficiently reliable for the purposes of our reporting objectives. To examine what technical assistance and financial assistance selected drinking water and wastewater utilities used for the second and third objectives, we provided a short questionnaire and interviewed utility representatives from the 15 selected drinking water and wastewater utilities to understand what technical and financial assistance they used to enhance their climate resilience for fiscal years 2011 through 2018. In the questionnaire and interviews, we discussed their efforts to plan for climate resilience and the technical and financial assistance they used for such efforts, which could include the five agencies we selected to review or other federal and nonfederal entities we did not review, but knew could potentially be sources of technical and financial assistance for utilities based on our prior work. Specifically, the federal agencies we did not review, but included in our questionnaire were: NOAA, the Department of Defense’s U.S. Army Corps of Engineers, and the Department of the Interior’s Bureau of Reclamation (Reclamation). To examine the second parts of the second and third objectives, the options experts identified for providing additional technical and financial assistance to utilities, we conducted semistructured interviews with the 10 climate change and disaster resilience experts. To develop the semistructured interview documents, we assessed the content of the 96 articles identified in our literature review to develop a list of actions that the federal government could take to make drinking water and wastewater infrastructure more resilient to the effects of climate change. The articles used to develop this list of actions were identified by searching resources such as Agricola, ProQuest’s Environmental Databases, Policyfile, Harvard’s Think Tank Search, and Scopus. We searched for both peer-reviewed articles and reports from nonprofits and think tanks published between January 2003 and September 2018 searching titles, abstracts and keywords for “water” in close proximity to “climate change,” “utilities,” and terms such as “project,” “program,” “policy,” or “recommendation.” We asked the 10 experts about the list of actions during our interviews (see table 2). We conducted semistructured interviews with the 10 selected experts and asked the experts to rate the effectiveness of the nine actions we provided for making drinking water and wastewater infrastructure more resilient to the impacts of climate change, describe the advantages and disadvantages of each action, and describe how the actions could be implemented. We also asked experts to rate the administrative feasibility and cost of the actions. Finally, we asked the experts if any additional actions should be added to our list. We then analyzed the results of our interviews to identify five options to provide technical assistance and developed a follow-up questionnaire. The questionnaire asked the 10 selected experts to rate the effectiveness of the five options for providing additional technical assistance, describe the advantages and disadvantages of each option, and describe how the options could be implemented (see table 3). We also asked experts to rate the overall effectiveness, administrative feasibility, and cost of the options. We also requested written responses from the 15 selected utilities on the 5 technical assistance options and the 4 financial assistance options identified in our interviews with experts. We conducted this performance audit from October 2017 to January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 4 corresponds with figure 2 in the report, which is an interactive figure and contains the text for drinking water utilities that is not accessible to readers of print copies of this report. As readers scroll over the water-drop icons in the figure, separate pop-up boxes appear describing specific regional impacts. Table 5 corresponds with figure 2 in the report, which is an interactive figure and contains the text for wastewater utilities that is not accessible to readers of print copies of this report. As readers scroll over the water- drop icons in the figure, separate pop-up boxes appear describing specific regional impacts. Table 6 provides additional information on the selected drinking water and wastewater utilities and the sources of technical assistance they used for climate resilience planning for fiscal years 2011 through 2018. The following federal programs have the potential to help drinking water and wastewater utilities, in particular smaller utilities that do not have the resources to conduct climate risk assessments and plan for measures to help make their drinking water and wastewater infrastructure more resilient to climate change impacts. Several of the federal efforts we reviewed provide general assistance with planning and operating drinking water and wastewater infrastructure. Specifically: Environmental Protection Agency’s (EPA) Environmental Finance Centers. The Environmental Finance Centers provide targeted technical assistance to, and partner with states and the private sector to help manage the costs of environmental financing. Environmental Finance Centers can provide technical assistance for financing drinking water and wastewater infrastructure and its operations and maintenance. EPA’s Training and Technical Assistance for Small Systems Grants. EPA’s Training and Technical Assistance to Small Systems grants provide funding to nonprofit organizations to provide training and technical assistance to small public water systems, small wastewater systems, and private well owners, located in urban and rural communities in the U.S. and its territories. According to EPA officials, training and technical assistance to small systems facing drought, flooding, and other weather-related challenges is an eligible activity for the grants. Department of Agriculture’s (USDA) Rural Water and Wastewater Technical Assistance and Training Program. USDA’s Rural Water and Wastewater Technical Assistance and Training Program provides grants to nonprofits such as the National Rural Water Association and the Rural Community Assistance Partnership to provide training and technical assistance to small and rural utilities for operating, managing, and financing drinking water and wastewater infrastructure. USDA’s Rural Water and Wastewater Circuit Rider Program. USDA contracts with a qualified national organization, through its Circuit Rider program, to provide technical assistance to rural water and wastewater systems to provide technical assistance to rural utilities for operating, managing, and financing water and wastewater infrastructure. Circuit riders also provide critical assistance in disaster response and recovery. The circuit rider contract was awarded to the National Rural Water Association in fiscal year 2019. Other federal efforts help decision makers use climate information in existing planning processes. Specifically: USDA Climate Hubs. USDA established regional Climate Hubs to deliver science-based knowledge and practical information to farmers, ranchers, and forest landowners to support decision-making related to climate change. Department of the Interior’s (Interior) Landscape Conservation Cooperatives. Interior developed a network of collaborative Landscape Conservation Cooperatives composed of federal, state, local, and tribal governments; nongovernmental organizations; universities; and interested public and private organizations to, manage large landscapes such as national forests, grasslands, and wetlands. As part of this program, the groups develop and provide the science and technical expertise needed to apply climate data in natural resources decision-making. U.S. Geological Survey’s Climate Adaptation Science Centers. Climate Adaptation Science Centers partner with natural and cultural resource managers to provide science that helps fish, wildlife, ecosystems, and the communities they support adapt to climate change by, among other things, providing climate, water, and ecosystem information to decision makers. National Oceanic and Atmospheric Administration’s (NOAA) Regional Integrated Sciences and Assessments (RISA) Program. NOAA’s RISA program supports a network of 11 regional research teams that work with public and private decision makers to identify and provide specific climate information and models to identify risks and adaptation options to increase resilience to climate variability and change. One area of emphasis for the RISA teams is conducting research on climate and water management issues while engaging with a range of water management organizations, including some water utilities. National Center for Atmospheric Research (NCAR). NCAR carries out interdisciplinary research on adaptation to climate change by generating scenarios of projected climate change, developing scientific tools and methods for analyzing current and future vulnerability, and conducting integrated analyses of climate change impacts and adaptation. An important component of NCAR’s program is the integration of decision makers and users of climate information, including water utilities, into its research activities. NCAR provides the atmospheric research community in academia, government, and the private sector with the shared resources necessary to conduct their research. Table 7 presents examples of drinking water and wastewater capital improvement projects to enhance climate resilience, according to utility representatives, from fiscal years 2011 through 2018. Table 8 presents additional information on financial assistance used by utilities we reviewed for capital improvement projects to enhance their climate resilience for fiscal years 2011 through 2018. In addition to the contact named above, Susan Iott (Assistant Director), Micah McMillan (Analyst-in-Charge), Jim Ashley, Mark Braza, Colleen Candrl, Caitlin Cusati, John Delicath, David Dornisch, Kathryn Godfrey, Holly Halifax, Karen Howard, Rob Letzler, Jon Melhus, Patricia Moye, Eve Nealon, Sam Portnow, Dan Royer, Kiki Theodoropoulos, Joe Thompson, Seyda Wentworth, and Melissa Wolf provided key contributions to this report.", "summary": "Human health and well-being require clean and safe water, according to the Water Research Foundation. The Fourth National Climate Assessment states that the potential impacts of extreme weather events from climate change will vary in severity and type and can have a negative effect on drinking water and wastewater utilities. GAO's previous work on climate change and resilience to extreme weather and disasters has shown how the federal government can provide information and technical and financial assistance to promote and enhance climate resilience. In 2015, GAO reported that enhancing climate resilience means taking action to reduce potential future losses by planning and preparing for climate-related impacts, such as extreme rainfall. This report examines federal technical and financial assistance to utilities for enhancing climate resilience, and options experts identified for providing additional assistance, among other things. GAO reviewed relevant federal laws, regulations, and guidance from four federal agencies—EPA, FEMA, HUD, and USDA—and interviewed federal officials, representatives from 15 water utilities selected for diversity of size and geography, and 10 experts selected to represent different views. Four federal agencies—the Environmental Protection Agency (EPA), the Federal Emergency Management Agency (FEMA), and the Departments of Housing and Urban Development (HUD) and Agriculture (USDA)—provide technical and financial assistance (e.g., loans and grants), to drinking and wastewater utilities. Technical assistance. EPA provides technical assistance to drinking water and wastewater utilities to enhance their infrastructure's resilience to climate change. However, according to EPA officials, EPA's program is small and cannot assist utilities nationwide. All of the selected experts GAO interviewed stated that utilities need additional technical assistance on an ongoing basis to manage climate risks, and most experts said that organizing a network of existing technical assistance providers, including federal and state agencies, universities, and industry groups, would be needed to provide such assistance. Under a presidential policy directive, EPA is to work to enable efficient information exchanges among federal agencies and to help inform planning and operational decisions for water and wastewater infrastructure. By identifying existing technical assistance providers and engaging them in a network to help utilities incorporate climate resilience into their infrastructure projects on an ongoing basis, EPA would have better assurance that climate information was effectively exchanged among federal agencies and utilities. Financial assistance. Federal agencies have taken some actions to promote climate resilience when providing financial assistance for water infrastructure projects, but agencies do not consistently include the consideration of climate resilience when funding such projects. Most selected experts suggested that federal agencies should require that climate information be considered in the planning of water infrastructure projects as a condition of providing financial assistance. Moreover, representatives from several utilities said that such a requirement could be an effective and feasible way to help enhance utilities' climate resilience. A requirement would ensure that utilities consider climate resilience in planning for water infrastructure projects and potentially limit future fiscal exposures. For example, from fiscal years 2011 through 2018, the federal government provided at least $3.6 billion in disaster recovery financial assistance for drinking water and wastewater infrastructure related projects (see figure). GAO recommends that EPA identify technical assistance providers and engage them in a network to help water utilities incorporate climate resilience into infrastructure projects. Also, Congress should consider requiring that climate resilience be considered in planning for federally funded water infrastructure projects. EPA neither agreed nor disagreed. GAO believes the recommendation is still warranted.", "document_type": "gao"}
{"report": "USPS is one of the largest civilian employers in the United States. In fiscal year 2018, USPS reported that it employed approximately 634,000 people and retirement benefits were paid to over 600,000 retirees and their survivors. According to USPS, it is one of the leading employers of minorities, women, veterans, and disabled veterans; for example, USPS reports on its website that it currently employs about 100,000 military members and veterans, nearly one-sixth of its workforce. Ninety-two percent of the USPS workforce is comprised of employees who are represented by four unions that are roughly organized along occupation type (see table 1 for the unions and member representation). These employees are also divided into “career”, and “non-career” employees. Career employees are considered permanent and are entitled to a range of benefits (e.g., health and retirement) and privileges. Non- career employees are generally considered temporary and hired, for example, during times of large mail volume such as holidays. As discussed later, non-career employees receive fewer benefits and lower pay than career employees. The Postal Reorganization Act (PRA) established USPS as an independent establishment of the executive branch of the government of the United States. PRA also established a compensation system where career postal employees and officers generally receive the same benefits as federal government employees, but also authorizes employees to collectively bargain over pay. Pay at many federal agencies is not subject to collective bargaining. Instead, pay at those entities is set through the General Schedule, which is developed and updated by the Office of Personnel Management (OPM). Additionally, PRA established that USPS should maintain compensation and benefits “on a standard of comparability to the compensation and benefits paid for comparable levels of work in the private sector of the economy.” Reform bills have been introduced in Congress that would amend some of the current compensation requirements, but none have passed. USPS costs are concentrated in employee compensation, which accounted for approximately 72 percent of total operational costs in fiscal year 2018 (see fig. 1). The majority of compensation costs are payments to current employees, which include an employee’s hourly pay and benefits such as contributions to retirement and healthcare plans and USPS’s share of payroll taxes for Social Security and Medicare. USPS contributions for retirement benefits are made to OPM administered funds that pay out USPS retiree pension and health benefits, as well as to the Thrift Savings Plan (TSP). USPS negotiates contracts that include terms for the compensation of the 92 percent of employees represented by unions through a collective bargaining process. This process may entail a three-step process for USPS: negotiation, mediation, and interest arbitration (as described below). If USPS and its unions cannot reach agreement during initial negotiations, a federal mediator is appointed, unless both parties waive mediation. If no agreement is reached with the mediator, or if the parties waive mediation, the contract goes to impasse. An impasse then proceeds to final and binding interest arbitration. In interest arbitration, the dispute goes before a three-member panel, which determines factors impacting compensation, such as pay increases. The total cost of compensation for current USPS employees was about $9 billion less in fiscal year 2018 than in fiscal year 2009, when adjusted for inflation. However, most costs decreased between fiscal year 2009 and fiscal year 2014, and costs have generally risen since (see fig. 2). Without adjusting for inflation, USPS compensation costs for current employees are still lower—by almost $1 billion—when compared to 2009, but costs have been rising since 2014, and USPS has reported an anticipated total compensation cost increase for fiscal year 2019. Over the same time period, the number of employees followed a similar pattern of decline from fiscal years 2009 through 2013 and then generally increased. Overall, compared to fiscal year 2009, USPS has reduced its total number of employees as of fiscal year 2018 by over 77,000. One key reason for the decline in USPS compensation costs was the decrease of 90 million work hours over this period. The largest decrease in work hours was from fiscal years 2009 through 2013, when work hours declined about 12 percent. We reported in 2014 that this was accomplished in part through attrition and separation incentives. Recent trends, however, show total work hours are increasing, from a combination of new hiring and increased work hours for current employees. From fiscal years 2014 through 2018, work hours increased by 5.4 percent. Additionally, the number of work hours associated with higher costs—overtime and penalty overtime—have also been increasing. USPS reported that the recent increase in work hours and overall compensation costs is a result of increases in the number of delivery addresses and increases in more labor intensive package volume. USPS adds about one million new delivery points each year. Although overall mail volume declined from fiscal years 2009 through 2018, package volume increased almost 200 percent during the same period. However, package volume growth has slowed in recent months, largely due to significant competition among delivery providers, according to USPS. Generally, USPS compensation grew more slowly over the last decade than in the private sector and federal government. Based on our review of USPS data for fiscal years 2009 through 2018, USPS employee compensation has increased on average by 1.0 percent per year. According to Bureau of Labor Statistics data, average employee compensation increased by approximately 2.3 percent per year for workers in private industry. In a prior review of federal civilian compensation trends, we found average compensation increased 1.2 percent per year for the federal workforce from 2004 through 2012. Based on a review of publically available Office of Management and Budget data, we found this trend of about a 1.2 percent annual increase continued through 2018. Although USPS decreased compensation costs paid to current workers, its unfunded liabilities for retirement benefits significantly increased during the same time period. By law, USPS employees are entitled to participate in the federal retirement health benefits and pension programs. USPS is required to make annual payments into the OPM administered pension and retiree health benefits funds that support postal employee retirement benefits; however, USPS has failed to make a significant portion of these payments. Retiree Health Care Liabilities: OPM administers the Postal Service Retiree Health Benefits Fund, which pays USPS’s share of premiums for retired postal employee health care coverage. As of September 30, 2018, USPS had contributed $20.9 billion to the fund, and missed payments on an additional $33.9 billion in required payments to the fund for 2012-2016. For fiscal years 2017 and 2018, OPM billed USPS for required payments to the fund of $3.3 billion and $3.7 billion respectively and USPS did not make either payment. As of September 30, 2018, USPS reported the unfunded retiree health benefit liability to be $66.5 billion. Pension Liabilities: OPM also administers federal pension benefits through the Civil Service Retirement System (CSRS) and the Federal Employees Retirement System (FERS). USPS employees participate in one or the other of these plans. Both plans are funded through the Civil Service Retirement and Disability Fund (CSRDF). In 2018, USPS failed to make required payments to the CSRDF totaling approximately $2.4 billion; $958 million for FERS and $1.4 billion for CSRS. USPS reported the unfunded pension benefit liability, as of September 30, 2018, to be $25.1 billion for CSRS and $18.4 billion for FERS. As the total unfunded liabilities for health care and pension benefits owed to current and future retirees are about $110 billion, we have previously reported on the significant risk posed by these financial liabilities to USPS’s long-term sustainability. We have also reported that Congress should consider passing legislation to put postal retiree health benefits on a more sustainable financial footing, and recently provided options for proposed legislative changes related to retiree health costs in particular. For the remainder of this report we will focus mainly on those costs USPS incurs related to current employee services. In addition to decreasing the number of employees and work hours, USPS also implemented three major changes to decrease employee compensation: (1) lowering pay for new career employees, (2) increasing use of non-career employees, and (3) reducing USPS contributions to health insurance premiums for active employees. These changes were negotiated with the four unions representing the majority of postal employees and established in CBAs. According to USPS management officials, these actions were intended to decrease compensation costs and increase workforce flexibility, which were necessary responses to declining letter mail volume and revenue, growth in more labor-intensive package volume, and increases in the number of delivery addresses. We report USPS’s estimates, and our estimates, of how much these changes saved in employee compensation costs below; we further describe the differences between the two estimates in the next section. For additional technical details about our analysis, see appendix I. 1. Lowering Pay for New Career Employees: Beginning in 2010, USPS implemented a negotiated lower starting pay for new career employees. More specifically, career employees hired after a specified date have lower starting pay than previously hired career employees. For example, a city carrier hired in January 2016 would make about $37,640 a year compared to $48,406 a year if hired before the new starting pay agreement. USPS estimated about $2.3 billion in savings for fiscal years 2016 through 2018 as a result of this effort. We were not able to substantiate the estimated savings because USPS could only provide individual data for fiscal years 2016 through 2018, which were not enough data to develop comparison groups for employees hired before and after the pay rate change. 2. Increasing Use of Non-career Employees: In 2010 and 2011, USPS negotiated the ability to hire up to 20 percent of the workforce as non- career employees; the prior limit had been 10 percent for most employee types. USPS officials told us they also changed some work rules so that USPS could use non-career employees for some tasks previously only allowed for career employees. Non-career employees are less costly because they generally have lower pay rates and are not entitled to the full federal benefits received by career workers. According to USPS officials, non-career employees are also “more flexible” because there are fewer restrictions on their tasks and schedules. For example, USPS management officials told us that they use non-career employees for much of the Sunday package delivery service and to make extra trips needed to deliver packages to meet service targets. USPS estimated that increased use of non- career employees saved about $8.2 billion in compensation costs since fiscal year 2016, but our analysis found that USPS likely saved about $6.6 billion from fiscal years 2016 through 2018 from this effort. 3. Reducing Contribution for Employee Health Insurance Premiums: USPS decreased its contribution percentage for employee health insurance premiums from 84 and 85 percent in 2008 to 74 percent in 2018. Based on its recent agreement with NRLCA, USPS’s contribution will decrease from 73 percent in 2019 to 72 percent in 2020. In past CBAs with the three other unions (APWU, NALC, and NPMHU), negotiations over USPS contributions to health insurance premiums followed those agreed to by NRLCA. USPS officials estimated that USPS’s reduced contribution percentage to employee health insurance premiums has saved about $1.6 billion across the types of postal employees from fiscal years 2016 through 2018. However, our analysis found that USPS likely saved about $1.4 billion for the three-year period. Although USPS was able to decrease its share of the health insurance premium to achieve a larger saving in fiscal year 2018 than in fiscal year 2017, overall USPS expenditures for its share of employee health insurance premiums did not decrease due to annual increases in premiums. USPS reported that employee health benefits expenses increased from $5.0 billion in 2016 to $5.2 billion in 2018, even as its share of premium costs decreased from 76 percent to 74 percent for employees covered by the CBAs during the same period. Across all of its efforts, USPS estimated it saved approximately $12 billion for fiscal years 2016 through 2018. While there are multiple valid approaches for estimating cost savings based on policy changes, we found that USPS did not account for some significant factors and, therefore, potentially overstated the savings achieved. Specifically, USPS did not account for the effects of changes in work hours or tenure of employees. When we accounted for these additional factors, we were able to substantiate $8 billion, of USPS’s estimated $9.7 billion, in savings over the last three fiscal years for changes to the number of non-career employees and health insurance contributions. As noted above, we were not able to substantiate the estimated savings of lowering pay for new career employees. We summarize the specific factors below and include additional details about our review of USPS’s estimates and the effect of each factor in appendix I. As previously discussed, in recent years, USPS employees have worked significantly more overtime and other “premium” pay hours. According to USPS, use of overtime and premium hours enables it to meet irregular work demands (for example, spikes in volume resulting from holidays) or delivery performance targets, particularly for Sunday package delivery. Also, to incentivize or compensate employees for working extra or traditionally less desirable hours, USPS routinely uses overtime and other premium pay, such as additional pay for work at night and on Sundays. As a result, these types of work hours cost more to compensate than regular work hours (i.e., straight time hours) on a per hour basis. When calculating the savings it achieved from using lower-paid employees, USPS compared what it actually paid in compensation to estimates of what it would have paid in the absence of having non-career or lower-paid workers, using average pay rates and not individual level employee data. USPS’s method therefore did not fully account for the mix of types of hours worked in its estimates. As a result, USPS underestimated how much the lower-paid employees are compensated in its cost estimates. Our analysis of the last three years of data found that lower-paid employees work a different mix of hours, and overall they work more hours and more premium hours, factors that USPS does not capture in its estimates. For example, from our analysis of USPS payroll data, we found that, on average: a non-career employee worked 30 more straight hours, 73 more overtime hours, and 23 more night and Sunday hours per year than a career employee, and a lower-paid career employee worked a higher number of straight time hours and, depending on the craft, also may work more overtime, night work, and Sunday hours than a higher-paid career employee. USPS officials said it was not necessary to factor in work hours because the amount of work hours was not changed by introducing lower-paid employees. For example, USPS officials told us that, to meet the increase in packages, more carrier work hours were needed in recent years, to make deliveries on Sundays for instance. USPS officials also noted that to save costs, it is preferable that these hours go to lower-paid employees. However, our analysis suggests that lower-paid employees may work different amounts and mixes of work hours than higher paid employees. For example, newer, lower-paid employees may be more willing to work extra hours, and being newer, their inexperience could mean that they take longer to complete their work on average. USPS management officials said that they do not believe newer employees are less productive than more experienced employees, nor do they lead to increases in overall work hours. USPS management officials also told us that employees cannot opt into working more hours because overtime hours are assigned as necessary by supervisors. Our analysis did not include information that would allow us to determine whether management was pre-approving all overtime hours. However, in June 2019, the USPS OIG reported $136.6 million in unauthorized overtime— which occurs when an employee’s clock time exceeds eight hours without prior approval—for mail processing alone. USPS has saved billions by using a less costly and more flexible workforce. Indeed, based on fiscal year 2018 data, we calculated that USPS could potentially save up to an additional $4.4 billion a year if the current cap on non-career employees was doubled to 40 percent. However, USPS did not fully evaluate the impact of pay rates and work hours by employees. Given the growth in work hours, particularly overtime and premium pay hours, USPS risks overestimating savings and making ill-informed changes to employee compensation by not including information about employee work hours. USPS employees with longer tenure generally receive higher pay than similar employees with less tenure. Based on our analysis of USPS payroll data for fiscal year 2018, the average pay of career employees is driven in part by the high median tenure of those employees, which was 20 years (with a median age of 54 years old) in fiscal year 2018. However, when calculating its savings estimates for non-career employees, USPS did not factor in the effect of employee tenure. Specifically, USPS’s savings estimate for non-career employees compared what it was paying for a newly hired non-career employee against the average pay for a career employee, rather than the starting pay for a career employee. When we accounted for tenure in our analysis, we found that some of the savings from hiring new employees could be explained by the shorter tenure of the lower-paid employees. USPS officials told us that they agreed that tenure should have been taken into account and that they would recalculate these estimates. Without adjusting for mix of hours worked and tenure, we found the difference in pay between career and non-career employees to be, on average, $25 per hour. After adjusting for tenure and mix of workhours, we found the difference in pay to be, on average, $8.27 per hour. There are a variety of acceptable methods for conducting cost savings estimates, but all estimates should include all the relevant factors driving costs and be clearly documented. The GAO Cost Estimating and Assessment Guide—a best practices guide for developing and managing program costs—states that estimates should include a common set of agreed-upon estimating standards and ensure that assumptions are not arbitrary. USPS officials said that they do not have guidance for how to develop these estimates, including what significant factors should be considered. Given that USPS regularly evaluates and manages employee compensation in its labor negotiation, as well as overall budget planning, without guidance on what factors are necessary to consider when developing employee compensation cost estimates, USPS risks making ill-informed decisions about whether to maintain, or make additional, changes to compensation. Based on interviews with USPS and postal employee union officials, as well as recent research by the USPS OIG, we identified additional costs that USPS did not factor into its cost savings estimates related to lowering employee pay and benefits. Specifically, USPS did not include the impact of the changes on recruitment and turnover of non-career employees in its cost saving estimates, both of which could have a significant impact on the overall level of savings. Both USPS management and postal union representatives discussed the impact of lower pay on recruitment of non-career employees. Officials from two unions told us USPS is having a harder time recruiting and retaining some non-career employees, especially minorities and veterans because of the lower pay. In July 2019, USPS OIG reported that a post office in Denver had constant challenges filling letter carrier vacancies due in part to USPS’s inability to offer competitive compensation. The report noted that a high number of vacancies affected carriers’ ability to complete their routes on time, contributing to excess overtime and penalty overtime. USPS officials stated that with very few exceptions USPS has had little trouble attracting applicants to non-career positions. One example of an exception USPS officials provided was that in fiscal year 2018 USPS increased pay for non-career seasonal holiday workers to make it more competitive. USPS and postal union officials also told us that USPS had trouble specifically hiring truck drivers at the non-career pay scale. In addition to lower pay compared to the private sector, a high demand for drivers and low unemployment rates across the industry has made it challenging for USPS to find enough qualified drivers. In addition to the lower wages, USPS and postal union officials stated that the unpredictable non-career employee work schedules, as well as low unemployment rates, have created additional challenges for recruiting qualified non-career employees. In contrast, USPS officials told us that implementing a lower pay rate for new career employees has not affected recruitment because employees are generally recruited from the non- career employee pool, so these employees get an increase in compensation from their current position. USPS and postal stakeholders also raised concerns about the effect of lower pay on retaining non-career employees and the associated costs. USPS officials told us they expected the turnover rate among non-career employees to increase with the reduction in starting pay, but stated that recent turnover was higher than expected. According to USPS, the average monthly turnover rate for non-career employees has decreased from 3.57 percent in fiscal year 2016 to 3.08 percent in fiscal year 2017 and 3.02 percent in fiscal year 2018. USPS officials told us that USPS strives to keep the turnover rate as low as possible and that overall, postal employees voluntarily leave their jobs at a lower rate than in the private sector. According to two postal union estimates, it costs USPS about $4,000 to $7,000 to hire and train new employees. USPS OIG has reported that turnover costs USPS about $95.1 million in fiscal year 2015, with an additional $23.1 million in fiscal year 2016 and could be $29.8 million in fiscal year 2017. According to USPS officials, the lower pay rate for new career employees has not had a significant impact on employee turnover. New career employees were converted from the pool of non-career employees, who had a lower pay rate than the new career employees, thus getting an equivalent of a pay raise and more benefits. However, USPS provided us with its analysis showing that the turnover rate for career employees with a lower pay rate (average of 4.21 percent per month) was higher than for career employees with a higher pay rate (average of 0.36 percent per month) in fiscal year 2018. According to postal union officials and USPS OIG, decreases in pay and lack of work schedule flexibility have resulted in some negative effects on morale that increased turnover of non-career employees. Union officials told us that some managers have abused the flexibility of non-career employees, such as requiring them to work many consecutive days. In addition, non-career employees do not have regular work schedules and can be laid off for lack of work. Officials at one union told us that non- career employees would prefer being hired as career employees, but have to start as non-career employees before being converted to a career position, and getting such a conversion can take from one to seven years. USPS OIG reported that in exit surveys, non-career employees stated that the lack of schedule flexibility, low pay, and lack of benefits are among the most cited reasons for leaving their job. The report also stated that managers realized cost benefits by using non-career employees to provide coverage for vacation days, sick days, and unscheduled leave for career employees because their hourly rates are less than those of their career counterparts. Union officials also told us that some non-career employees do not receive the necessary training but are expected to perform their jobs correctly from the start. They also said that these new employees are less experienced and are more likely to make mistakes. In addition, they said that managers become upset that new employees cannot do their jobs correctly from the start, which leads to morale issues among employees. USPS officials told us that when they implemented the compensation changes discussed above, they expected higher rates of employee turnover, especially among non-career employees. USPS officials told us that they are developing an assessment of the cost of turnover and the preliminary results have not been validated. Specifically, USPS officials also told us that they have not yet determined how to accurately apply the turnover estimates to the population of employees who leave because some turnover is necessary and preferable. For example, there are seasonal needs for increases in labor hours, such as major holidays or in some vacation areas, and when these employees exit, it is often because the season ends and their employment is not needed. In contrast, other employees leave USPS voluntarily for higher paid, or less difficult, work elsewhere. USPS officials told us they recently began to develop estimates of employee turnover costs, estimates that include costs such as training, background checks, and drug screenings for new employees, and the estimates are preliminary. We have reported that legislative reform and additional cost-cutting are needed for USPS to achieve sustainable financial viability. As noted above, compensation costs are about three-quarters of USPS’s annual expenditures and many aspects of how USPS compensates its employees are defined in law. As a result, changes to the current statutory requirements for employee compensation are one way to alter USPS’s operational costs. A variety of reviews of USPS have also recommended legislative action to help address USPS’s long-term sustainability. We examined four broad reviews of USPS and found 12 recommendations that could impact employee compensation costs by amending statutes governing three areas: employee work hours, benefits, and pay. The four reviews we analyzed are: (1) Task Force Review of 2018, (2) Presidential Commission Review of 2003, (3) USPS 2010 Comprehensive Statement, and (4) PRC 2016 Analysis. The recommendations in these four reviews are not exhaustive of all possible statutory changes that could impact employee compensation costs. The recommendations we reviewed also do not include changes to the fundamental business model of USPS, such as privatization, or a return to annual appropriations to finance its operations. We are also not recommending or endorsing the adoption of any of these recommendations, in part, because our cost estimates and limitations discussed below are based on broad policy options and do not take into account many of the specific factors that would need to be determined when implementing any of these options. This information is meant to describe the potential for savings from increasing flexibility related to work hours, benefits, and pay, as well as highlight some potential challenges of implementing those changes. A major driver of USPS’s operating costs is delivering mail to nearly every mailing address, regardless of volume, six days per week. USPS’s mission to serve, as nearly as practicable, the entire population of the United States, requires a significant, continual use of employee work hours. This is particularly true of the mail carriers who visit addresses each delivery day. Based on USPS payroll data, we found mail carrier compensation in fiscal year 2018 was approximately $24.4 billion, or about 50 percent of compensation costs for current employees. Two of the twelve recommendations we reviewed suggest legislative changes to increase USPS’s authority to determine delivery frequency, which would enable USPS to manage work hours more closely to volume. Specifically, the 2018 Task Force report recommended that USPS be given more flexibility to determine delivery frequency. USPS recommended in its 2010 Comprehensive Statement that Congress change the current delivery requirement from six days a week to five days a week. Changing the frequency of USPS’s deliveries could reduce its employee compensation costs significantly by allowing USPS to reduce work hours, particularly for carriers. Reducing delivery by one day could potentially reduce carrier work hours by a maximum of one sixth—or 16.7 percent. Our analysis shows that, based on fiscal year 2018 payroll data, if USPS decreased the current mail carrier hours by one sixth, it could save up to $2.6 billion in compensation costs. This estimate assumes that USPS would reduce work hours from both the career and non-career carrier employee pools. If USPS reduced mail carrier hours from only the non- career carrier workforce by 16.7 percent, it could save approximately $1.96 billion. USPS officials agreed that USPS could potentially save work hours and associated costs due to a reduction in delivery frequency. However, they noted that, even if USPS went to 5-day delivery, it would still deliver packages seven days a week. Under that scenario, USPS has reported estimated savings of $1.4 to $1.8 billion a year. The 2018 Task Force report recommended that USPS be given increased delivery flexibility by allowing USPS to determine delivery frequency. Additional flexibility could result in a range of alternatives. For example, USPS could deliver to addresses every other day (three or four days a week) with optional dynamic routing as necessary up to an additional two days a week, and could potentially save more than 16 percent. USPS has begun to introduce technology and other options within its package handling that might alleviate undue burden caused by such a large decrease in service. For example, USPS now offers informed delivery, which is an email that is sent to mail recipients with pictures of their mail that is to be delivered, and enables people to have better insight into what to expect and when. In 2018, we reported that USPS was piloting keyless parcel lockers where customers could independently pick up their packages; it is possible similar types of backups could be provided for letters. Regardless of the delivery frequency, reducing mail carrier hours is more likely to come through a decrease in non-career employee hours. USPS reports that non-career employees are temporary in nature and can be laid off; therefore, it would be easier to implement hour reductions in this pool of employees. In contrast, if USPS were to reduce hours for career employees, savings would accrue more slowly over time, because career employees’ are usually covered by no lay-off clauses in their CBAs, and with low turnover rates, USPS would need employees to leave voluntarily. A large portion of the career mail carriers, however, are aging, and it is possible that many will leave through retirement in the next five years. Specifically, in analyzing the USPS payroll data, we found that, in fiscal year 2018, approximately 16 percent of career city carriers are currently 60 or older, with an additional 38 percent between the ages of 50 and 59. Approximately 21 percent of career rural carriers are 60 or older and 38 percent are between the ages of 50 and 59. Based on our analysis of the recommendations, we identified three potential challenges to reducing delivery frequency: (1) management of work hours (2) redistribution of mail volume; and (3) meeting delivery needs. Management of work hours: Realizing cost savings from a decrease in delivery frequency largely depends on USPS being able to reduce work hours accordingly. Prior USPS OIG and GAO reports have found that in two previous efforts USPS has not successfully decreased labor hours commensurate with a decreased level of service. Beginning in January 2015, USPS revised its First-Class Mail service standards, eliminating single-piece overnight service and shifting some mail from a 2-day to a 3-day service standard. According to USPS officials, these revisions were intended to, among other things, allow USPS to process mail on fewer machines and decrease the need for overnight work hours, which are paid at a higher rate than day time hours. USPS OIG, in its review of this service change, found that mail processing overtime costs increased by $68.4 million, or 9 percent, rather than decreasing. USPS OIG conducted a follow-up study that found USPS was not effectively managing mail processing overtime in fiscal year 2018. USPS management’s official response partially agreed with the recommended steps USPS OIG outlined to better manage overtime. In 2012, USPS implemented Post Office Structure Plan (POStPlan), which was intended to reduce work hours at some retail facilities, which USPS estimated would result in about $500 million in annual cost savings. In 2016, we found that this effort likely resulted in less savings than USPS had estimated. According to USPS officials, this is, in part, because USPS had to revise its plan after a union grievance and arbitrator award required it to change the way it was staffing post offices. We also reported concerns with USPS’s methodology for determining work hour and compensation savings. Overall, while USPS likely achieved an overall reduction in work hours at thousands of post offices, we found the accuracy of the saving may have been limited by errors we identified. For example, among other issues, we found that USPS had not included the increase in workload, and associated costs, from increasing the number of remotely managed post offices. Distribution of volume: Any reduction of delivery frequency would require USPS to re-distribute its mail volume to the remaining delivery days. In 2010, USPS recommended eliminating Saturday delivery and re- distributing the mail volume from Saturday to the delivery days Monday through Friday, though USPS continues to deliver mail on Saturdays. USPS stated that the additional volume in the remaining delivery days would result in higher mail carrier productivity. However, we reported in 2011 that USPS’s ability to efficiently absorb the cost of transferred workload from Saturday to weekdays is a key factor in determining potential cost savings. Meeting Delivery Needs: Another challenge to reducing delivery frequency is that it could reduce the demand and value of USPS products if customers are not getting their delivery needs met. Some stakeholders have raised concerns that a reduction in mail delivery frequency will decrease demand from mailers because products may not reach households in a timely manner. Other stakeholders, however, have stated that reducing delivery frequency is worth pursuing as long as it results in significant cost savings. Federal law requires USPS to provide certain benefits to its employees, which cost USPS billions each year to satisfy. Further, as noted above, USPS is mandated to pre-fund its retiree health benefits, which USPS has failed to do in recent years. Four of the twelve recommendations we reviewed suggest legislative changes to the funding mechanism and requirements for USPS retiree health benefits. In addition, in 2018, we issued a report with options for postal retiree health benefits and noted that it is up to Congress to consider the merits of different approaches and determine the most appropriate action to take. Here, we focus on the other three of the twelve recommendations we reviewed that suggest legislative changes to other USPS retiree benefits. USPS’s 2010 Comprehensive Statement and the 2003 Presidential Commission Report both have broad recommendations suggesting that USPS should be allowed to make changes to its retirement benefits package. USPS pays toward the following retirement benefits for current employees: contribution to retirement pension (FERS or CSRS) and contributions to the TSP, Medicare, and Social Security. For the purposes of estimating the impact of decreases in retirement contributions, we estimated savings based on decreases to the cost of pensions (CSRS and FERS) and TSP, and assume no changes to Medicare and Social Security costs. Decreases to Retirement Contributions: If USPS was able to decrease its cost of retirement payments made on behalf of current employees by 1 percent, 5 percent, and 10 percent, then we estimate based on fiscal year 2018 payroll data, the potential savings would be about $35 million, $175 million, and $350 million respectively. Implementation of such decreases could include USPS offering lower cost benefits by increasing the employee contribution or lowering the promised benefits. The 2018 Task Force report recommended reform for postal employees under FERS to move away from a “defined benefit” system—where the payment received in retirement is a specified amount—towards a defined contribution system—where the contribution into the system is a specified amount. There are many different ways to implement this kind of change, and we have not outlined potentially restructuring options. However, CBO has calculated potential savings of increasing civilian employees’ contribution for FERS and estimates it would save about $20 billion over five years. Decreases to Healthcare Contributions: If USPS was able to decrease its cost of health care payments for current employee coverage by 1 percent, 5 percent, and 10 percent, we estimate, based on fiscal year 2018 payroll data, the potential savings would be about $45 million, $224 million, and $449 million, respectively. If USPS no longer had to offer federal health care coverage for current employees, it is possible that USPS could substitute a less costly alternative. In 2013, we conducted a review of a specific USPS proposal for restructuring its health care benefits, and reported that the change could result in cost savings, but that other issues should be considered, such as exposure of health care funding if investments are made outside of Treasury securities. Based on our analysis of the recommendations, we identified three challenges to achieving cost savings from changes to employee benefits: (1) union agreement; (2) cost savings timeline; and (3) impact on federal benefit programs. Union Agreement: According to USPS officials, if there was a legislative change that allowed for USPS to alter the current retirement benefits, USPS would need to negotiate future benefits offerings with the unions. Savings, therefore, depend on the ability of USPS and the unions to develop alternative options that meet the needs of the current workforce, but also cost less than the current options. Implementation Timeline: Cost savings are not likely to be realized in the short-term because changes likely will not apply to current career employees. In the past, when Congress has made changes to benefits— as when Congress increased the required retirement contribution levels for federal employees under FERS, which also applied to USPS employees—it only applied to employees hired after the change was implemented. Therefore, savings would only occur as new employees replace current employees. This is also consistent with the lower pay for new career workers that USPS negotiated with the unions we discussed previously—it only applied to new career employees. Federal Benefit Impact: The Presidential Commission Review of 2003 stated that USPS should work with the Department of the Treasury and OPM to determine the impact that separating USPS’s pension and retiree health care programs would have on the existing federal systems. With over 600,000 USPS employees, the Presidential Commission review stated that it had concerns about the impact of removing USPS employees might have on the OPM administered fund, which also pays out retirement benefits for other federal employees. Two of the twelve recommendations we reviewed suggest legislative changes to collective bargaining rights, which could result in decreased pay rates. The 2018 Task Force report recommended the elimination of the right of USPS employees to bargain over compensation and that employee pay rates be frozen in the short term, which would lead to a slower rate in growth over time. If USPS was provided authority to determine pay rates for its employees without going through collective bargaining, it could reduce employee compensation costs through pay cuts. We estimated the potential annual cost savings associated with USPS implementing cuts for all current employee pay by 1 percent, 5 percent, and 10 percent across all current employees based on fiscal year 2018 payroll data. We find the potential savings to be about $321 million, $1.6 billion, and $3.2 billion, respectively. Based on our analysis of the recommendations, we identified three challenges to obtaining savings through reductions in pay: (1) difficulty of cutting current workers’ pay; (2) trade-offs of lower wage rates; and (3) history of collectively bargaining pay. Difficult to cut pay: As discussed previously, pay has been the area in which USPS has made progress in reducing employee compensation costs in the recent past. However, as discussed, the savings mostly comes from implementing lower pay rates for new employees. It is difficult to implement changes that decrease the current pay of workers below what they have previously received. In the private sector, a company can restructure and turnover a portion of the workforce as an effort to decrease compensation. USPS cannot easily turnover and restructure its workforce because of the no layoff clauses. Trade-off effects: Pay cuts to current employees could result in a variety of negative consequences for USPS. According to literature on labor economics, workers who face pay cuts may exhibit behavioral responses including adjusting worked hours, adjusting level of effort for each hour of work, or dropping out from the workforce altogether. Workers may adjust the hours of work from changes in pay for two reasons. First, a pay cut may reduce the incentive for employees to work because each hour of work generates less money than it did before, holding income constant. Second, a pay cut that reduces the income of the worker may induce an employee to work more hours because the employee feels poorer. Changes in pay rates may also change workers’ morale, and consequently the effort workers exert during worked hours. Economic literature has found that wage cuts can impact the effort workers provide, and that productivity may fall. For example, workers may exert less effort in an attempt to punish the employer for the wage cut, or they may be less worried about job loss because the cost of losing a job is lower after a wage cut. These consequences may be of particular concern as USPS has reported that productivity has stagnated in recent years, and USPS is currently not meeting its standards for on-time delivery service. Finally, pay cuts may also induce some individuals to leave the workforce altogether. Studies have found that the share of the population that is working may be influenced by pay rates. As we discussed previously, USPS has already experienced some difficulty in recruitment and retention as a result of the lower pay for new employees. History of Collective Bargaining: Elimination of collective bargaining rights—which could facilitate changes to USPS employee pay—would be a major change in management-labor relations at USPS, with possible negative effects on employee commitment and productivity. Unions representing USPS employees have been bargaining over pay since the 1970s. Prior to that time, USPS employees had major strikes over low compensation levels. USPS has a high approval rating from the public, which it attributes, in part, to its employees feeling a sense of duty related to their work. All of the union officials we spoke with said that they would not support the removal of collective bargaining rights over pay. One union official stated that both parties are better served through working as a team to meet the needs of postal customers at reasonable cost. USPS has made changes to employee compensation and saved billions through these efforts. USPS, however, has not achieved financial sustainability. USPS overestimated its cost savings from the employee compensation changes because it did not include significant factors such as tenure and mix of work hours when developing its estimates. In addition, USPS did not weigh the costs of tradeoffs, such as an increase in turnover, which likely further limits cost savings. Cost estimates that include the significant factors driving compensation costs would help USPS make better informed decisions about how to use, and potentially change, its workforce. Quality estimates are also important for USPS to make a business case for additional employee compensation changes, which it does regularly as it negotiates employee contracts and will be doing as it develops future strategic plans. Additionally, as Congress considers USPS reform legislation, comprehensive cost estimates will improve policymakers’ ability to fully assess savings, as well as costs associated with any efforts and associated implications for managing USPS compensation. We are making the following recommendation to USPS: The Postmaster General should direct executive leaders to develop guidance for cost savings estimates related to employee compensation specifying that analysis used to calculate estimates should, to the extent possible, include significant factors, such as work hours, tenure, and turnover. (Recommendation 1) We provided a draft of this report to USPS for review and comment. USPS provided written comments that are summarized below and reprinted in appendix III. In its written comments, USPS agreed that quality decision-making rests upon quantitative analysis using the best available data. In that respect, it stated that it accepts the recommendation to more formally articulate internal guidance for developing cost savings estimates to ensure appropriate factors—such as work hours, tenure, and turnover—are taken into account when evaluating potential business outcomes. USPS disagreed, however, that the lack of formal guidance adversely affected USPS’s ability to develop appropriate cost estimates. As discussed in detail in the report, we found that USPS’s analysis potentially overestimates savings because it did not take certain factors into account. Specifically, in its comments, USPS identified issues related to our findings about their cost-estimation analysis. For example, our analysis, which relied on USPS payroll data, showed that non-career employees have generally worked more overtime hours when compared to career employees. Although USPS did not dispute this finding, it said it believed our analysis reflected erroneous assumptions about the source and administration of overtime because we described some possible reasons for the overtime patterns we saw based on our analysis and other research. We did not intend to determine the full cause of overtime hours and how they are distributed among employees, rather, our analysis sought to identify important factors in employee compensation costs, and found that the mix of work hours was important and varied across types of employees. We also found that USPS estimates had not taken differences in the mix of work hours into account and in assuming that career and non-career employees work similar types of hours, USPS potentially overstates the savings from non-career employees. USPS agreed to take work hours into consideration in future cost estimates and this may provide a more accurate assessment of costs, and better opportunity to target future efforts to manage workforce costs. Regarding our analysis related to recruitment and retention issues among non-career employees, USPS stated that it disagreed with our statements regarding wages for non-career workers and their purported impact on recruitment of certain employees. It said that USPS has little trouble attracting applicants to non-career positions, and we made changes to the report to reflect this view. Regarding turnover, USPS acknowledged that turnover among certain groups will be higher and they account for these turnover rates in their analysis. However, we found that USPS did not fully account for costs associated with these turnover rates in the analysis they provided us. With a higher than expected turnover rate among non-career employees, which have become a significantly larger percentage of its workforce, USPS should be accounting for the additional costs of on-boarding of employees, like recruitment and training. USPS stated that, in response to the recommendation, it will incorporate the cost of turnover into future analysis. We are sending copies of this report to the appropriate congressional committees, the Postmaster General, Chairman of PRC, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff making key contributions to this report are listed in appendix IV. To support all parts of this review, we requested and received U.S. Postal Service (USPS) national payroll data for fiscal years 2009 through 2018. We specifically requested individual level payroll data for all ten years; however, individual employee level data before fiscal year 2016 were not readily available. For fiscal years 2009 through 2015, we received aggregate data at the post office level, and for fiscal years 2016 through 2018, we received the data at the individual level. For both sets of data, USPS provided point-in-time data at the end of the fiscal year. We conducted a data reliability assessment and found that, for describing general trends, the fiscal years 2009 through 2015 payroll data provided at the post office level was sufficient for our purposes. However, for evaluating policy changes to employee compensation, only fiscal years 2016 through 2018 payroll data provided at the individual level were appropriate. To assess the reliability of the payroll data for fiscal years 2009 through 2018, we reviewed technical documentation for the dataset, related publications, and information on USPS and Bureau of Labor Statistics websites about employee compensation. We performed several analyses in order to validate that these data were appropriate to use for the purposes of our work. We spoke with USPS payroll data specialists to discuss known limitations and issues with the data. USPS officials informed us that they do not keep a data dictionary for the entire payroll system because it is a conglomerate data system with over 40 sub- databases. However, we were able to obtain documentation on the variables relevant to our analysis to understand the definitions and limitations of those variables. We found the individual level payroll data provided for fiscal years 2016 through 2018 reliable for the purpose of examining policy changes to manage employee compensation, and to determine the effect of potential legislative changes to USPS employee compensation. To describe recent trends in USPS employee compensation, we analyzed high-level trends in the payroll data for fiscal years 2009 through 2018. We compared our analysis of USPS national payroll data with USPS annual reports to Congress and financial forms filed as a result of Securities and Exchange Commission requirements, from fiscal years 2009 through 2018. While we found that the data do not match exactly, we found that our estimates are close to reported USPS numbers for each year, (see table 2). We had several discussions with USPS payroll data specialists to clarify how to use the payroll data and ensure that the payroll data were reliable for reporting on changes in hours and overall compensation. To examine the results of recent actions taken by USPS to manage employee compensation costs, we identified three major changes implemented through collective bargaining agreements aimed a decreasing the cost of employee compensation. To evaluate the impact of these changes, we analyzed USPS payroll data. Using these data we developed models to determine trends in compensation based on worker characteristics, including pay rate, participation in various benefits, and career or non-career status. We analyzed data to determine the costs savings accrued by USPS from having undertaken changes to compensation in recent years. We also analyzed the data to determine the effect of potential legislative changes to USPS employee compensation. We received USPS National Payroll data from fiscal years 2016 through 2018 for individual employees with a detailed summary of a worker’s pay, benefits, and hours worked. Pay data include pay for straight time, overtime, and other time with pay differentials (Sundays, nights, holidays, and Christmas), and leave, including annual, sick, holiday, military and other types of leave. See table 3 for a summary description of the types of pay and hours. For each pay category (e.g., straight time, overtime), USPS provided information on the number of hours worked by each worker in a given fiscal year. Benefits include health insurance payments, pension contributions (FERS, CSRS and Dual CSRS with Social Security) and Thrift Savings Plan (TSP), life insurance, Social Security, and Medicare (see table 4). The data contained detailed information on the worker’s earnings, benefits, and hours of work and some characteristics, including age, and the worker’s start and separation dates, if the worker has separated from the USPS. We examined postal employees classified as career or non-career within each of the four main crafts based on the type of work performed. We separated employees into their respective craft and career or non-career status based on their Designation Activity Code. We used these individual-level data to estimate total compensation costs based on observable characteristics of the workers. This section discusses the quantitative analysis methods we used to determine the results of recent actions taken by USPS to manage employee compensation. Table 5 presents the numbers of employees in the postal workforce for fiscal years 2016 through 2018, within four crafts – city carriers, rural carriers, mail handlers, and clerks – and other employees not in the four crafts. Table 6 presents average pay, average benefits, average compensation, and the median age for postal workers by craft and career status. We examined the mix of straight and premium hours between the higher pay (Tier-1) and new career employees hired at the lower starting pay (Tier-2). We used USPS individual-level payroll data for fiscal years 2016 through 2018. Table 7 summarizes the effective dates for the lower starting pay for new career employees by craft. Table 8 summarizes differences in hours between these two groups among full-time equivalent employees. We examined several types of work hours. Straight time hours include all reported straight time hours in a fiscal year. Overtime hours include overtime work, penalty overtime, holiday work, and Christmas hours. Premium hours include hours worked in night work and Sunday work hours. Our analysis does not adjust for characteristics that can affect hours such as age or tenure. We found that Tier-2 employees worked a higher number of straight hours. Furthermore, carriers who are Tier-2 employees also worked a larger number of overtime hours. Among mail handlers, Tier-2 employees worked a higher number of night work and Sunday work hours (see table 8). To the extent that Tier-2 employees work more overtime hours, and assuming a similar productivity between the two tiers of employees, USPS may be miscalculating the effect of the lower pay rate on costs. To analyze the results of hiring more non-career employees, we examined differences in hourly compensation (pay and benefits) between career and non-career postal employees, and estimated cost savings from moving to a workforce with more non-career employees. The analysis examined the entire workforce, within the four different crafts (i.e., city carrier, clerk, mail handlers, and rural carriers) and the remainder of the workforce excluding employee from the four crafts (termed as “other”). 𝑦𝑦𝑖𝑖𝑖𝑖𝑖𝑖=𝛼𝛼+𝛽𝛽1𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑖𝑖+𝛽𝛽𝑋𝑋𝑖𝑖𝑖𝑖𝑖𝑖+𝛾𝛾𝑁𝑁𝑖𝑖+𝛿𝛿𝐹𝐹𝐹𝐹𝑖𝑖+𝜁𝜁𝑖𝑖+𝜑𝜑𝑖𝑖+𝜀𝜀𝑖𝑖𝑖𝑖𝑖𝑖 (2) Equation (2) predicts work hours, hourly pay, benefits, and compensation as a function of individual characteristics. All models are estimated using Ordinary Least Squares (OLS). (1) We examined several types of work hours. Straight hours include all reported straight hours in a fiscal year. Overtime hours include overtime hours, penalty overtime, holiday work hours, and Christmas work hours. Premium hours include hours worked in night shift differential and Sunday premium. Total work hours include straight time, overtime, holiday hours, Christmas work, and penalty overtime. We do not include night shift differentials and Sunday work in the calculations for total worked hours as these hours are already captured under straight hours. We also exclude from work hours any leave. (2) Per hour pay is defined as earnings for worked hours divided by total worked hours. Earnings for worked hours includes payments for straight time, overtime, holiday hours, Christmas work, penalty overtime, and premiums for night and Sunday work. (3) Hourly benefits include USPS contribution to health insurance, life insurance, retirement, TSP, Social Security and Medicare, and dollars associated with leave (see table 4). These variables were calculated by summing over all benefits and dividing by total work hours. (4) We calculated hourly total compensation by summing over hourly wage compensation and benefits that USPS paid to employees. The value of total compensation is divided by total work hours which include total work hours. The NonCareer variable (𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 ) is a dummy taking the value of describing the employee’s position. The parameter 𝛽𝛽1 identifies how 1 for non-career employees and zero for career employees. We were 1 for non-career employees and zero for career employees. We were able to generate this variable based on designation activity codes different (if at all) non-career employees’ hours, pay, benefits and compensation are relative to career employees. To account for other variables that could be driving differences in pay, benefits and compensation between career and non-career employees, we included the variables described below in the estimation. Employee craft (𝑁𝑁𝑖𝑖,) are a series of binary indicators for city carriers, Finance unit binary indicators (𝜁𝜁𝑖𝑖), control for level difference in clerks, mail handlers, rural carriers, and the other employees category. City carrier is the benchmark category. These indicators were excluded in the results by craft, but were included in results for the entire workforce. demand between finance units (usually post offices), but also implicitly account for level differences in local level unemployment rates during the period of our study. Age, and age squared capture differences in potential labor market year as fiscal year minus the year of birth for the employee. Tenure and tenure squared describe years of experience with USPS began working for the USPS and the fiscal year in the data. To account for time effects that could affect compensation and The error term (𝜀𝜀𝑖𝑖𝑖𝑖𝑖𝑖 ) captures differences in earnings and experience and were included in (𝑋𝑋𝑖𝑖𝑖𝑖𝑖𝑖). We defined age in each fiscal (𝑋𝑋𝑖𝑖𝑖𝑖𝑖𝑖). We defined tenure as the difference between the year a worker 2018( 𝐹𝐹𝐹𝐹𝑖𝑖). earnings we included year indicators for fiscal years 2016 through compensation that are not observed in our data. We adjusted earnings and compensation by the Consumer Price Index for Urban Wage and Clerical Workers (CPI-W), and present values in 2018 constant dollars. We cluster all errors at the finance unit level to allow for correlation in errors between those within the same finance unit. Indexes i, c, and t designate the individual (i), cohort of hire (c) and time (t). Many non-career employees work a limited number of hours each year, because employees who work on short contracts may have different preferences about the number of work hours that they are willing to supply. Therefore, comparing the outcomes of those who work a limited number of hours to those who work on a full-time basis does not generate a valid comparison. We do not observe preferences for flexible work- schedules in our data, and as a result, analysis comparing individuals with different work schedules would be subject to omitted variables that could bias our estimates of cost differences between career and non- career employees. To facilitate a closer comparison between career and non-career employees, we restricted the sample to those employees who work more than 1,820 each year (excluding leave) which we computed by assuming a 35 hour work week. Furthermore, we excluded those who worked more than 80 hours a week, or the equivalent of working 4,160 hours each year. We refer to these employees as full-time equivalent employees, because their hours are equal to or exceed the full-time equivalent of 35 hours per week. Our analysis relied on payroll data, and these data have not been collected for research purposes. We limited our analysis to those with positive benefits and compensation per hour. We excluded from the sample those earning wages below the federal minimum wage ($7.25 per hour) in the study period. Our data on hourly wage, benefits, and compensation included values in the thousands of dollars. We considered these values to be aberrations related to adjustments in the payroll system. To address these values, we excluded observations that were above the 99.5 percentile for hourly wages, benefits, and compensation for a particular craft and in a fiscal year. Our final analytical sample included 1,373,717 observations over the three years of data, from fiscal years 2016 through 2018. The analysis described adjusted differences in components of compensation, but does not adjust for many characteristic differences among the different categories of employees that may matter in determining outcome variables (e.g., aptitude, gender, and race). The tenure variable is likely mis-measured because non-career employees may have previously been employed by USPS in some other capacity, but given that our individual-level dataset goes back to fiscal year 2016, we do not have previous USPS job experiences for these employees. Because the analysis restricted the data to those working between 1,820 and 4,160 hours a year, we were modeling the USPS workforce that is employed with USPS on what would be considered a full-time basis. While non-career employees on short-term contracts are expected to work a full-time schedule, we do not observe start and separation dates for these non-career employees in our data. As a result, the analysis including these employees cannot be conducted as we would be making assumptions about unobserved preferences of individuals who work on a full-time basis and those who do not. Unadjusted differences in work hours and hourly pay, benefits, and compensation can be found in table 9. We note that these differences compare full-time equivalent employees with non-career and career status. For the entire workforce, differences in work hours existed between career and non-career employees. For example, the average non-career employees worked 115 more straight time hours, 94 more overtime hours, and 6 more night and Sunday premium hours. Regarding differences in the hourly wage, benefits and compensation, we observe a difference of $11 in hourly pay, $14 in benefits per hour, and $25 in overall compensation between career and non-career employees. While these unadjusted differences capture the overall rate differentials between career and non-career employees, they do not account for the differences in characteristics in the career and non-career workforce. The majority of the USPS career workforce is comprised of employees who have been with USPS for a long time. In contrast, the non-career workforce, by its very function, is more flexible and comes and goes based on demand for postal products and services. Consequently, the non-career workforce may have less of an opportunity to accumulate on- the-job experience (tenure) with USPS. Previous literature finds that wages rise with tenure. As such, these large unadjusted differences between career and non-career employees can be attributed in part to the extensive on-the-job experience of career employees. To account for these differences, and other differences in labor market experience, we present adjusted estimates among the career and non-career workforce in table 10. Results from the analysis that controls for differences in employee characteristics are summarized in table 10. Differences in work hours indicate that full-time equivalent non-career employees perform 30 more straight time hours, 73 more overtime hours, and 23 more night and Sunday work hours. We observe variation in these estimated effects by craft. For example, differences in straight time hours are largest for clerks and mail handlers, followed by others, and city carriers. In contrast, the highest differences in overtime hours are observed for city and rural carriers, no differences in hours worked between career and non-career clerks. We also found that non-career mail handlers and other employees work fewer overtime hours relative to their respective career counterparts. The use of night and Sunday differential is higher among non-career city carriers and clerks, while it is lower for non-career mail handlers. While we found a small statistically significant effect for non-career rural carriers, this effect is relatively small given that rural carriers perform a very limited number of hours at night and on Sundays. We found that non-career employees receive $2.10 less in pay per hour, $6.17 less in benefits per hour and $8.27 less in compensation per hour. Examining the differences in pay by different types of work hours reveals that the largest difference in pay exists for overtime hours, where pay is $3.42 less among non-career relative to career workers. The difference in pay is $2.23 per hour, and while differences for night shift and Sunday differential are present, they are smaller at 13 cents per hour. Examining the effect across crafts, we find that clerks have the largest difference in per hour compensation at $8.43, followed by $8.04 for rural carriers, $7.72 for mail handlers, and $7.25 for city carriers. We also describe differences for the category of employees designated as other, which includes all other employees not designated as carriers, clerks and mail handlers. Among these employees, non-career employees receive $10.79 less in hourly compensation relative to career employees. These findings contrast with the findings in table 9, where we do not adjust for differences in characteristics between career and non-career employees. Adjusted differences are approximately 19 percent of the unadjusted difference in hourly pay, 43 percent of the unadjusted difference in benefits, and 33 percent of the difference in hourly compensation, highlighting the importance of controlling for employee characteristics in estimating difference in pay between career and non- career employees. We estimated savings USPS generate by hiring non-career employees, by calculating all hours serviced by non-career employees and multiplying this number by the difference in compensation estimate for the entire workforce ($8.27 per hour). Our calculations indicate that USPS was able to save $2.3 billion in fiscal year 2016, $2.1 billion in fiscal year 2017 and $2.2 billion in fiscal year 2018 from using non-career employees. To analyze the result of USPS reducing its contributions to health insurance premium for active employees, we examined the differences in cost of these contributions. We assumed that in the absence of decreases in the contribution percentage each year, USPS would continue to contribute health insurance premiums at the 2008 rate of 85 or 84 percent (see table 11). We examined employees with positive health insurance premiums contributions and generated average contributions per employee. We then calculated health insurance costs in the absence of any contributions by dividing the cost paid by USPS by the percentage contribution in each year (see table 11). We generated per employee savings by comparing the dollar values between what USPS paid each year and what it would have paid under an 85 or 84 percent contribution. We generated total savings by multiplying the number of employees who took up these plans by the savings per employee. We present these results in table 12. Our results indicated that the reduction in USPS health insurance contributions generated savings of $429.45 million in fiscal year 2016, $438.14 million in fiscal year 2017, and $513.77 million in fiscal year 2018, or $1.38 billion for the entire three year period. Our analysis does not model changes in health insurance participation arising from workers who drop insurance as a result of having to contribute a higher percentage of their health insurance costs. To understand the potential effects of changes to USPS employee compensation that would require legislative or statutory change, we conducted several analyses to estimate potential costs and savings from these changes. We examined the impact of (1) eliminating one day of delivery, (2) reducing benefits, such as shifting additional costs of health insurance premium to active employees, and (3) cutting employee pay across the board. Eliminating some of the current mail delivery would have varied effects on employees based on their craft. If all delivery were stopped for one day that USPS currently delivers mail, work hours for carriers may be reduced by a maximum of one-sixth but work hours for clerks, mail handlers, and other employees would not be affected in the same way. We examined two ways to reduce employee compensation costs, by cutting hours (1) across all career and non-career carriers (city and rural) and (2) only for non-career carriers. Cutting work hours for career and non-career carriers by one-sixth: We aggregated over all work hours for career and non-career city and rural carriers for fiscal year 2018. We multiplied the number of hours under a one-sixth hours reduction by the average pay for hours worked for city and rural carriers. We generated yearly savings for rural and city carriers. Our analysis suggests that cutting mail carrier hours by one-sixth, through a reduction in delivery frequency may have saved USPS $2.6 billion in fiscal year 2018. Cutting work hours for non-career carriers by one-sixth: We aggregated over all work hours for career and non-career city and rural carriers. We then reduced this amount of work by one-sixth, to roughly approximate a cut in hours on average equivalent to cutting one day of delivery for all carriers regardless of career status. We multiplied the hours by the pay of non-career carriers (i.e., rural and city carriers) and estimated savings generated if the one-sixth cut was applied only to non-career carriers. We expected the savings to be less than the scenario presented above, since non-career carriers receive a lower per hour wage rate. We also calculated the percent reduction of work hours for non-career workers that would be necessary to eliminate one day of delivery. Our analysis suggest that reducing non-career hours by the equivalent of cutting one-sixth of all carrier hours may have saved USPS $1.96 billion in fiscal year 2018. These cuts would imply a decrease in non-career hours of 49 percent for rural carriers and 69 percent for city carriers. This analysis does not account for substitution between hours worked. For example, cutting Saturday delivery may shift workers to work more overtime or premium time pay categories. The analysis assumed carrier productivity per hour does not vary with career status, and is not affected by cuts. Finally, it assumed that benefits do not change, but to the extent that benefits are a function of income they may also be reduced. We examined the overall effect of cutting benefits by 1.0 percent, 5.0 percent, and 10.0 percent for all employees. We considered the entire workforce and examined the total payments USPS contributed to health insurance and retirement accounts (e.g., FERS, CSRS, and TSP). Reducing Health Insurance Premiums: We examined the effect of reducing USPS’s contributions toward employee health insurance premiums by 1.0 percent, 5.0 percent, and 10.0 percent, by aggregating all health insurance contributions that USPS made on behalf of all employees in fiscal year 2018 and applying these respective cuts. The analysis does not account for the fact that health insurance participation may fall because the USPS contribution cuts would shift a higher proportion of the cost of insurance to workers. We present the results in table 15. Reducing Retirement Contributions: To examine the effect of cuts of 1.0 percent, 5.0 percent, and 10.0 percent in USPS retirement contributions (FERS normal cost) and USPS TSP contributions, we aggregated over all such USPS contributions for all USPS employees participating in these plans, and applied these respective cuts to contributions for fiscal year 2018. One limitation of this analysis is that it does not account for the change in work hours and, as a result, in compensation from the cut in these benefits. We present results in tables 16 and 17. To examine the effect of cuts of 1.0 percent, 5.0 percent, and 10.0 percent in USPS employee pay, we determined the total work hours (straight, overtime, other hours) and average pay rates for all USPS employees. We performed the following calculations. 1.0 percent reduction in pay: hours*Δwage= hours*(wage*0.01) 5.0 percent reduction in pay: hours* Δwage = hours*(wage*0.05) 10.0 percent reduction in pay: hours* Δwage= hours*(wage*0.10) We found the cost savings associated with cuts of 1.0 percent, 5.0 percent, and 10.0 percent across all current employees are $322 million, $1.61 billion, and $3.22 billion respectively for fiscal year 2018. Across the board wage cuts will produce both direct and indirect effects on overall compensation costs for USPS. In the section above, we provided a calculation of the savings that USPS may realize from the direct effect of a policy that cuts worker pay. The direct effect implies that a pay cut reduces the total wages paid. For example, a 10 percent reduction in pay should reduce the total wage paid by 10 percent, other things constant. However, workers who face pay cuts may exhibit responses to wage cuts that include: adjusting hours worked, adjusting their level of effort for each hour of work. These responses may negate or reinforce additional savings from a wage cut and are examples of indirect effects of wage cuts. Workers may adjust the hours of work from changes in wages, as these wage changes produce to both substitution and income effects on hours worked. The Substitution Effect implies that a pay cut reduces the incentive for employees to work because each hour of work generates less money than it did before, decreasing the opportunity cost of leisure (the time spent not working for pay), holding income constant. Thus a reduction in pay could lead to additional reductions in work hours that employees are willing to supply. Consequently, the reduction in pay may lead to additional savings for USPS in its labor costs from the substitution effect. In contrast, the Income Effect implies that a pay cut reduces the income of the worker; this reduction in income induces the employee to work more hours because the employee feels poorer. The income effect would therefore increase the hours worked, and could lead to reductions in savings for USPS. Income and substitution effects generally run in opposite directions, and uncertainty regarding which effect will dominate determines the overall effect on hours worked. A recent review of the literature from the Congressional Budget Office finds that combined hour elasticities that incorporate both income and substitution effects range between 0 and 0.2. These combined hours elasticities would suggest that a 10 percent cut in wages would reduce hours between zero and 2 percent (0.2*10%). While we are aware that the estimates from the general population may not extend to the USPS workforce, we provide the above example for illustrative purposes. While the effect on workhours from a change in wages may appear small, given the overall hours serviced by USPS each year – adjustments in hours arising from wage cuts may produce nontrivial changes in USPS compensation costs. Changes in wage rates may change workers’ morale, and consequently the effort workers exert during work hours. Economic literature finds that wage cuts can impact on the effort workers provide, and that productivity may fall. For example, workers may exert less effort in an attempt to punish the employer for the wage cut, or they may be less worried about job loss because the cost of losing a job is lower after a wage cut. It is important to note that wage cuts may also induce some individuals to leave the USPS workforce altogether. Estimates of participation elasticities in the literature range between zero and 0.2. Participation elasticities capture the percentage change in the share of the population that is working resulting from a 1 percent change in wage rates. These estimate elasticities would imply that a 10 percent wage cut could be met with a reduction in the USPS workforce between 0 and 2 percent. As we previously noted, it is not clear that these population estimates extend to the USPS workforce, thus we believe examining these effects from USPS workforce data may be an important step in understanding the potential changes in workforce that wage cuts may generate. We reviewed four reports and identified twelve recommendations that proposed legislative changes that relate to USPS employee compensation (see table 19). We categorized these recommendations as having the potential to impact wages, benefits, or required work hours. The reports we reviewed were: 1) 2018 Task Force Report: United States Postal Service: A Sustainable Path Forward. Report from the Task Force on the United States Postal System, U.S. Department of the Treasury. December 2018. 2) 2016 PRC Analysis: Section 701 Report: Analysis of the Postal Accountability and Enhancement Act of 2006, Postal Regulatory Commission, November 2016 3) 2010 USPS Comprehensive Statement: Foundation for the Future: 2010 USPS Comprehensive Statement on Postal Operations United States Postal Service. 4) 2003 Presidential Commission Report: Embracing the Future: Making the Tough Choices to Preserve Universal Mail Service. Report of the President’s Commission on the United States Postal Service, July 2003. In addition to the individual named above, Kyle Browning (Assistant Director); Jade Winfree (Analyst-in-Charge); Thanh Lu, Silda Nikaj; Josh Ormond; Steven Putansu; Oliver Richard, Frank Todisco, Michelle Weathers; Seyda Wentworth; and Crystal Wesco made key contributions to this report.", "summary": "USPS faces major financial challenges. In the last 11 years it has lost over $69 billion; an issue for an organization that is to be self-sufficient. Significant USPS expenses are concentrated in employee compensation—72 percent of its costs in fiscal year 2018—and USPS has taken actions to decrease these costs. GAO was asked to review issues related to USPS's employee compensation. This report examines: (1) recent trends in postal employee compensation, (2) the results of recent USPS efforts to manage compensation and (3) potential effects of proposed changes to employee compensation that would require legislative change. GAO analyzed USPS employee payroll data from fiscal years 2009 through 2018 to determine compensation trends and impacts of management efforts to manage compensation. GAO reviewed relevant legal documents, USPS policy documents and collective bargaining agreements. GAO assessed four broad reviews of USPS including recommendations for legislative change related to pay, benefits and required workhours. GAO also interviewed USPS officials, officials representing USPS employee unions, and industry and mailer stakeholders. Compensation costs for current United States Postal Service (USPS) employees are $9 billion lower than 10 years ago, when adjusted for inflation (see fig). Most of the decline happened in fiscal years 2009 through 2014 as a result of reductions in the number of USPS employees and the hours they worked. While compensation costs have increased in recent years, USPS reports that more work hours were necessary to handle growth in delivery points and labor intensive packages. In recent years, USPS has also failed to make required payments for retiree health and pension benefits—a total unfunded liability of about $110 billion. Compensation Costs for Current USPS Employees for Fiscal Years 2009 through 2018 USPS estimates a savings of about $9.7 billion from fiscal years 2016 through 2018 as a result of paying new employees less, among other efforts. GAO substantiated about $8 billion in savings, and found that USPS's cost savings estimates are likely overstated because they do not fully account for changes in work hours or tenure of employees. Also, USPS did not account for other costs such as increased turnover rates among lower-paid employees. USPS lacks guidance on what factors to consider in its cost savings estimates, and as a result may make future changes to employee compensation based on incomplete information. Changes to employee compensation that would require legislative change could save USPS billions, but the amount saved is dependent on USPS overcoming implementation challenges. If USPS could reduce delivery frequency and associated work hours, GAO estimated USPS could save billions a year. However, other recent USPS reductions in service have not fully achieved planned work hour reductions due to, among other things, issues with management of work hours and lack of union agreement. Changing employee pay and benefit requirements could also achieve significant long-term savings, but saving depends on USPS overcoming challenges, such as potential increases in turnover and reduced productivity resulting from decreases in pay and benefits. GAO recommends that USPS develop guidance that specifies that cost estimates include important factors, such as turnover. USPS accepted this recommendation stating it would formally articulate internal guidance to ensure appropriate factors are taken into account when developing cost estimates and evaluating outcomes.", "document_type": "gao"}
{"report": "NNSA uses professional SSCs in its program offices, headquarters offices, and field offices. Program offices plan and oversee NNSA’s numerous programs and projects and are generally responsible for integrating activities across the agency. NNSA’s program offices are: Defense Programs, Safety, Infrastructure, and Operations, Defense Nuclear Security, Counterterrorism and Counterproliferation, and Naval Reactors. Headquarters offices generally provide leadership, develop policy and budgets, or provide other functional support across NNSA. The headquarters offices include the offices of: Acquisition and Project Management, Cost Estimating and Program Evaluation, Information Management and Chief Information Officer, Management and Budget, and Policy. NNSA also has seven field offices across the country. The field offices are responsible for overseeing NNSA’s management and operating (M&O) contractors, including ensuring compliance with federal contracts. To provide oversight of the M&Os, each field office employs subject matter experts in areas such as emergency management, physical security, cybersecurity, safety, nuclear facility operations, environmental protection and stewardship, radioactive waste management, quality assurance, business and contract administration, public affairs, and project management. NNSA’s field offices are generally located at the sites they oversee. NNSA’s field offices are: Kansas City Field Office in Missouri, Livermore Field Office in California, Los Alamos Field Office in New Mexico, Nevada Field Office, NNSA Production Office in Tennessee and Texas, Sandia Field Office in New Mexico, and Savannah River Field Office in South Carolina. After an office determines that it has an unmet work need, officials are to consult an agency document that outlines the procedures to determine whether to hire a federal employee or use another hiring option, such as an SSC, to meet the office’s need. If, upon consulting the document, officials determine that an SSC is appropriate for their needs, they are then required to contact a representative from NNSA’s Office of Acquisition and Project Management to begin the procurement process. This office is responsible for acquisition support and contracting oversight for the agency throughout the acquisition lifecycle. NNSA’s Office of Management and Budget also has responsibilities for SSCs through, among other things, assisting offices in determining the appropriate funding source for contracts and providing advice on the development of performance work statements. Performance work statements provide a clear description of the activities that the contractor is expected to undertake and how the contractor’s performance will be assessed. NNSA guidance describes the performance work statement as the most important document in a procurement package, as the performance work statement is considered to be the binding agreement under which the contractor must perform. In addition, officials must submit a selection justification form to NNSA’s Office of Management and Budget for approval. In 2012, NNSA implemented the use of a form specific to SSCs—referred to as a determination form—to help mitigate the risk of awarding SSCs for activities that must be performed by federal employees. The form includes a series of questions to help officials from the office that plans to use the SSC and contracting officers to identify inherently governmental functions when reviewing a performance work statement. According to the determination form, if functions contemplated are closely associated with inherently governmental functions, an official must determine that NNSA has sufficient capacity to give special management attention to the contractor’s performance to preclude unauthorized personal services. If the support needed includes inherently governmental functions, the agency would not procure the service by contract. After officials confirm the services to be procured do not include work that must be performed by federal employees, officials sign the determination form to indicate that they have sufficient capacity and capability to, among other things, give special management attention to contractor performance, and include the completed form in the contract file. Once an SSC is awarded, NNSA relies on certain key personnel in various offices to oversee the contractor’s performance and ensure that the contractors comply with the terms of a contract. These include: Contracting officers. Contracting officers work within NNSA’s Office of Acquisition and Project Management and have the authority to enter into, administer, and terminate contracts. Contracting officers, along with program office officials, are responsible for determining the level of risk associated with a contract. Further, as part of the acquisition process, the office that identified the need for the SSC works with a contracting official to develop the performance work statement. Contracting officer’s representatives (COR). CORs are nominated by the program office and approved by the contracting officer. CORs are authorized representatives of the contracting officer and have the primary responsibility of overseeing the contractor, assessing performance, accepting deliverables, and reviewing invoices. Task monitors. Normally assigned by a program office, task monitors assist the COR with oversight of contractor performance. During the life of a contract, contracting officers and CORs regularly monitor contractors’ performance to ensure the contractors are complying with the terms of the contract. This monitoring varies across contracts and can include, for example, reviewing the contractor’s monthly invoices or reports and conducting formal annual evaluations. The monitoring activities can also vary based on the types of tasks included in the contract. For example, a contract requiring advanced technical analysis may warrant monitoring that is different from a contract that requires office administrative support. This difference is because the former is a more complex task that may include the review and approval of technical reports or other deliverables. Contracts for office support may not generate such deliverables. NNSA uses three appropriations accounts—or funding sources—to fund its SSCs. The first is NNSA’s Federal Salaries and Expenses appropriations account. Funding from this account is also referred to as program direction funding in NNSA’s annual budget justification materials. This account is generally used to pay for costs associated with NNSA’s federal employees, such as salaries, benefits, travel costs, and training, regardless of whether those federal employees work in headquarters, program, or field offices. The annual congressional budget justification materials define the Federal Salaries and Expenses account as used mostly to support the federal workforce. NNSA also uses this account to fund SSCs personnel who provide advice and assistance to a federal employee or in lieu of a federal employee. Because Federal Salaries and Expenses is the appropriations account used for most costs associated with federal employees, the amount of appropriations for this account helps determine the size of NNSA’s federal workforce. In addition, NNSA is subject to a statutory FTE cap on the total number of NNSA employees for each fiscal year. Congress and the President established a statutory cap in fiscal year 2013 that limited the total number of NNSA employees to up to 1,825 by October 1, 2014, and decreased that number in fiscal year 2015 to up to 1,690, where the number remains. NNSA can exceed the number of FTEs in the cap by submitting to the congressional defense committees a report justifying such excess. The other two sources NNSA uses to fund its SSCs are NNSA’s Weapons Activities and Defense Nuclear Nonproliferation appropriations accounts. Funding from these two accounts is referred to in NNSA’s annual congressional budget justification materials as program funding. Weapons Activities account. NNSA uses the Weapons Activities appropriation account to fund programs that provide for: (1) the maintenance and refurbishment of nuclear weapons to continue sustained confidence in their safety, reliability, and performance; (2) the investment in scientific, engineering, and manufacturing capabilities for certification of the enduring nuclear-weapons stockpile; and (3) the manufacture of nuclear weapon components. This account is also used to fund program offices other than the Office of Defense Nuclear Nonproliferation and Naval Reactors. Defense Nuclear Nonproliferation account. NNSA uses the Defense Nuclear Nonproliferation appropriation account to fund programs: (1) that provide, for example, policy and technical leadership to prevent or limit the spread of materials, technology, and expertise related to weapons of mass destruction; (2) that develop technologies to detect nuclear proliferation; (3) that secure or eliminate inventories of nuclear weapons-related materials and infrastructure; and (4) that ensure a technically trained emergency- management response is available both domestically and worldwide to nuclear and radiological incidents. Table 1 provides information on the three funding sources and the types of SSCs funded with each source. In recent years, we have reported concerns with federal agencies’ use of SSCs. In December 2011, we found that while agencies increasingly relied on contractors to provide professional and management support services, agencies generally did not consider and mitigate risks of acquiring such services, including the risk that contractors may inappropriately influence the government’s authority, control, and accountability for inherently governmental decisions. Additionally, in September 2018, we found that contracts requiring increased management attention, such as contracts for professional and management support services, have posed contractor oversight challenges for federal agencies. In that report, we found that there was an increased risk that contractors may perform tasks reserved for the government under contracts like those for management support services. We also found that the Office of Management and Budget (OMB) had taken steps to help agencies reduce some of the risks associated with contracts warranting increased management attention. For example, in September 2011, OMB’s Office of Federal Procurement Policy issued a policy letter to executive agencies to provide guidance on managing the performance of contractors performing work that is closely associated with inherently governmental and critical functions. The letter directed agencies to employ and train a sufficient number of qualified government personnel to provide active and informed management and oversight of contractors’ performance where contracts have been awarded for functions closely associated with the performance of inherently governmental functions. The September 2011 policy letter also provided guidance intended to clarify when governmental outsourcing for services is and is not appropriate. The letter identifies the need to increase management attention to using federal employees when functions that generally are not considered to be inherently governmental approach being in that category because of the nature of the function and the risk that performance may impinge on a federal official’s performance of an inherently governmental function. In addition, the policy letter calls for agencies to ensure that they have sufficient internal capability to control their missions and operations for managing critical functions. In 2013, NNSA’s Office of Defense Programs conducted an internal review of its use of nonfederal personnel to accomplish its missions. The study resulted in nine recommendations related to SSCs, including: developing policy on when to use each of the funding sources for SSCs and policy and guidelines on roles and responsibilities for federal employees; providing training for all NNSA employees on the proper use and management of SSCs; and evaluating current practices for the appearance of inherently governmental functions and terminating any inappropriate practices. As of July 2019, NNSA officials said the agency was working to finalize a policy on when to use each of the funding sources for SSCs. To address the recommendations on the two latter issues, NNSA developed training and guidance documents intended to assist staff in managing and working alongside contractors’ staff. Specifically, with regard to training, NNSA developed training for all NNSA’s federal employees to ensure that those employees understand the role of SSCs in the offices. This training covers, among other things, appropriate behavior and activities for federal staff who work alongside contractor personnel. With regard to guidance, NNSA developed documents that explain appropriate interactions with contractor personnel. For example, NNSA prepared a tip sheet for all staff to assist with maintaining proper relationships with SSC personnel; the tip sheet includes respecting the relationship between a contractor and its employees. NNSA also developed a contracting guide in 2014 that provides information on requirements, policies, and procedures, and that covers contracts for different purposes, including SSCs. The guide also includes descriptions of inherently governmental functions. In addition, NNSA’s Office of Management and Budget prepared a memorandum in 2014 for NNSA’s program offices to clarify the process for approving the funding source for SSCs. A July 2015 DOE Inspector General review of NNSA’s use of SSCs also found potential issues with the management of SSCs, particularly related to contractors’ performance of inherently governmental functions. For example, the review found that half of the 20 contracts in its sample included contracted services that approached being inherently governmental. The Inspector General’s review reiterated the recommendations in the Office of Defense Programs’ study and recommended that NNSA track the corrective actions to respond to the recommendations in that study to their completion. According to agency officials and documentation, NNSA has been tracking progress on these recommendations. In 2018, NNSA completed two workforce studies related to its use of SSCs. A joint workload and organizational analysis by NNSA and the Office of Personnel Management reviewed all program offices’ current workloads and federal staffing levels to assess the workforce needs to execute NNSA’s missions. The analysis concluded that NNSA did not have enough federal personnel to meet its mission requirements and called for an increase in the number of federal government employee FTEs by 238 over the agency’s current statutory cap of up to 1,690, for a total of 1,928. The analysis also concluded that the need for additional federal FTEs was driven, in part, by new mission requirements. NNSA’s Office of Cost Estimating and Program Evaluation also conducted an assessment of the number of federal personnel and contractors’ FTE personnel working on SSCs within each of NNSA’s program offices, as well as the appropriate workforce balance between federal and contractor FTEs, among other things. This assessment concluded that NNSA should rebalance its workforce by increasing the number of federal personnel to meet current and future missions. NNSA’s fiscal year 2020 budget justification materials request 1,753 federal FTEs, an increase of 63 FTEs over the current cap, in order to meet its mission requirements. In our March 2019 High-Risk Update, we stated that Congress should consider working with NNSA to ensure that the statutory cap on staffing is re-examined and is consistent with NNSA’s human capital needs, as evaluated in these two studies. NNSA increasingly used professional SSCs for a variety of functions in fiscal years 2010 through 2018. Specifically, based on our analysis of data from FPDS-NG, NNSA’s obligations for SSCs increased from about $139 million in fiscal year 2010 to about $193 million in fiscal year 2018 (see fig. 1). This is an increase of $54 million, or nearly 40 percent, in current dollars. The largest increase in NNSA’s obligations for SSCs occurred from fiscal year 2013 to 2014 when obligations for SSCs increased by about $26 million in current dollars—or about 16 percent, when adjusted for inflation to constant 2018 dollars. As discussed previously, in fiscal year 2013, Congress established a cap on the number of NNSA’s federal FTEs of up to 1,825 by October 1, 2014. After declining from a high of nearly 200 contracts in fiscal year 2010 to 160 in fiscal year 2011, the number of contracts did not fluctuate as much from fiscal years 2011 through 2018. NNSA used SSCs in nearly all of its offices in recent years (see table 2). The Offices of Defense Programs, Acquisition and Project Management, and Defense Nuclear Security together accounted for more than half of the FTE contractor personnel funded through professional SSCs in fiscal years 2015 through 2018. To understand how NNSA used these SSCs, we analyzed the product service codes associated with each of the SSCs. NNSA categorizes each of its SSCs using product service codes that provide some information on the types of tasks to be performed under the contract. NNSA identified 77 codes that define its professional SSCs when it started reporting information on SSCs in its congressional budget justification materials. These codes are arranged in five broad categories: (1) information technology and telecommunications support; (2) environmental consulting and legal support; (3) professional support; (4) administrative support; and (5) management support. Within each category, there are codes for specific activities, as well as a code for “other” support. For example, within the administrative support category, there are specific codes for word processing/typing, paper shredding, and transcription, and there is a separate code for “other” administrative services that is for tasks that do not fit within the other codes. According to several contracting officers and CORs we interviewed, officials try to select the code that best addresses all of the tasks included in the contract; however, most SSCs encompass a variety of tasks, so contracting officers often select the “other” category. Further, according to officials, if NNSA awards a task order under an existing contract, the task order has the same product service code as was assigned to the existing contract. As shown in figure 2, based on our analysis of FPDS-NG data, NNSA used three of the 77 product service codes—other professional services, engineering and technical services, and other administrative services—for more than 80 percent of its obligations to SSCs in fiscal year 2018. Because the product service codes encompass a wide range of activities, we reviewed the performance work statements for the 12 contracts in our sample to gain a greater understanding of the types of activities these codes may represent. The 12 contracts in our sample used five product service codes. Within those five product service codes, activities in the performance work statements for the 12 contracts in our sample include: Other professional services. Budgeting and evaluation analyses, technical support in training emergency response personnel, technical assessments and reviews, and policy analysis. One performance work statement included managing and maintaining databases, statistical analysis of budgetary data for decision makers, and programmatic assessments of data management systems for various programs. Engineering and technical services. Feasibility studies, acquisition planning, analysis of technical alternatives, project planning, risk analysis, general design support, and document preparation. One performance work statement included providing technical training support to the training program manager in a field office. Other administrative services. Analyzing the economic aspects of foreign nuclear programs, analyzing and producing reports on nuclear security issues in one region, processing correspondence, and data entry. One performance work statement included providing administrative and clerical support for functions such as responding to freedom of information act inquiries and providing support for training procurement, development, and evaluation. Other management support services. Providing technical coordination and document-editing services and reviewing, assessing, and linking government requirements to project documents. One performance work statement included support for maintaining an effective security program, including revising both federal and contractor sites’ requirements and procedures for two facilities and the field office. Program management and support services. Providing technical and advisory assistance in the design, construction, and operation of NNSA facilities for a certain program, technical evaluations, and technical and analytical support. One performance work statement included expert technical and advisory assistance related to the design, construction, and operation of facilities related to a certain program, including working with M&O contractors in engineering, equipment fabrication, construction, and tests. According to NNSA officials, NNSA increased its use of SSCs in fiscal years 2010 through 2018 due to: (1) increases in appropriations under the Weapons Activities appropriations account for additional work and (2) a decrease in the number of authorized federal employee FTEs due to a decrease in the statutory cap from fiscal years 2014 to 2015. First, as shown in figure 3, NNSA’s total appropriations increased from about $9.9 billion in fiscal year 2010 to $14.7 billion in fiscal year 2018 in current dollars. The increase in NNSA’s appropriations occurred mainly in the Weapons Activities appropriations account, which increased from $6.4 billion in fiscal year 2010 to $10.6 billion in fiscal year 2018 in current dollars. During the same period, NNSA’s appropriations for Defense Nuclear Nonproliferation generally remained around $2 billion per fiscal year in current dollars, and appropriations for Federal Salaries and Expenses—which covers the costs of all federal employees, including those working on Weapons Activities and Defense Nuclear Nonproliferation programs—remained around $400 million per fiscal year in current dollars. The increases in appropriations for the Weapons Activities account generally reflect the increasing workload to modernize the nuclear weapons stockpile and its associated infrastructure, as described in the 2010 and 2018 Nuclear Posture Reviews. According to an official in the Office of Defense Programs, that office has increased its use of SSCs because of the increase in refurbishment activities in the nuclear stockpile. Similarly, the internal review by NNSA’s Cost Estimating and Program Evaluation office attributed the increase in NNSA’s use of SSCs since 2012 to an increase in appropriations through the Weapons Activities account. According to an official from that office, the increased appropriations were for additional work related to weapons refurbishment and infrastructure modernization. Second, according to several NNSA officials, offices have increasingly used SSCs because of a decline in federal FTEs. As figure 4 shows, the number of NNSA’s federal FTEs funded through the Federal Salaries and Expenses account decreased from 1,897 in fiscal year 2010 to 1,608 in fiscal year 2018, a decrease of 15 percent. According to an NNSA official, this decline in federal FTEs is due, in part, to the annual statutory cap on federal FTEs that was to be implemented by October 1, 2014. An official explained that, by using SSCs, program offices have been able to accomplish the agency’s missions while remaining under the cap. Although the number of NNSA’s federal FTEs has generally decreased since fiscal year 2010, the change in federal FTEs has differed across program offices. From fiscal years 2013 through 2018, the number of federal FTEs in offices that support programs funded through the Defense Nuclear Nonproliferation appropriations account decreased, while those that support programs funded through the Weapons Activities appropriations account increased. For example, as shown in table 3, federal FTEs in the Office of Defense Nuclear Nonproliferation decreased by 22 percent from fiscal years 2013 through 2018. In contrast, the number of federal FTEs in the Office of Defense Programs increased 4 percent during the time period. In general, the number of federal FTEs supporting Defense Nuclear Nonproliferation activities has decreased, while appropriations for that office’s activities have remained consistent. In contrast, appropriations for Weapons Activities account have increased substantially, while the number of federal FTEs supporting those activities has increased by about 1.5 percent. According to some NNSA officials, SSCs provide the agency with flexibility to address new work needs that are episodic or specialized. This has led NNSA offices to use SSCs more frequently with the increased available appropriations and workload for Weapons Activities while remaining within the statutory FTE cap. Starting in fiscal year 2017, NNSA reported information on SSCs in its annual congressional budget justification materials, but the information was not complete because NNSA did not include data on (1) all of its professional SSCs or (2) the number of FTE contractor personnel who worked under an SSC for more than 2 years, as required by the fiscal year 2016 NDAA. Additionally, some of the information NNSA reported was not fully useful to support congressional decision-making because it did not present the cost of SSCs in terms of obligations for 1 fiscal year and did not identify the specific appropriations accounts used to fund SSCs. The NDAAs for fiscal years 2016 and 2017 require NNSA to report annually certain information on its use of SSCs in its congressional budget justification materials. NNSA reported information on its SSCs in its annual congressional budget justification materials for fiscal years 2017 through 2020, its most recent justification. Figure 5 shows an excerpt of the SSC information NNSA reported in its fiscal year 2020 annual congressional budget justification materials. NNSA obtained data for the first six columns of the information on SSCs reported in the fiscal year 2020 congressional budget justification materials from its accounting and contracting systems, called the Standard Accounting and Reporting System (STARS) and Strategic Integrated Procurement Enterprise System (STRIPES), respectively. The vendor name column identifies the name of the contractor performing the work. The contract number and order number columns provide the unique identifier that NNSA uses for the contract. If an SSC is a task order pursuant to an indefinite delivery/indefinite quantity contract, an order number is listed; otherwise the information is listed as “Unavailable.” The fund description column identifies the funding source for the contract—either (1) “Program” funding or (2) “FSE,” the latter of which represents SSCs funded through the Federal Salaries and Expenses appropriations account, which is also referred to as program- direction funding. In a few instances, the budget justification identifies the funding source as “both”—meaning both program funding and Federal Salaries and Expenses funding was combined to fund the contract. The “obligations to date” column provides the amount that NNSA has obligated on the contract since it was awarded. The “maximum value” column provides the total amount that could be obligated on the contract through the contract term and any options. NNSA collected the data on the number of FTE contractor personnel under each SSC—presented in the last column of figure 5—manually. Each year, the Office of Acquisition and Project Management requests information from contracting officers—in collaboration with program offices, CORs, and contractor staff, if needed—on the number of FTE contractor personnel working under contracts for professional SSCs. The information that the Office of Acquisition and Project Management provided to contracting officers states that each FTE represents 2,080 hours, each full-time employee is 1 FTE, and those who are less than full- time should be a portion of an FTE. According to NNSA officials, the agency uses this methodology for reporting FTE contractor personnel because the contracts do not require vendors to use a specific number of personnel to complete the work. Rather, the contractors determine the amount of labor needed to complete the scope of work under the contract. The information that NNSA reported on its professional SSCs in its annual congressional budget justification material was not complete because NNSA did not report information on all of its professional SSCs or on the number of FTE contractor employees who worked on the contract for more than 2 years, as required by the fiscal year 2016 NDAA. Reporting this information could provide some insight into how NNSA is using its SSCs and whether any of these contracts present increased risk for performance of personal services. Among other information, the NDAA for fiscal year 2016 required NNSA to include annually in its congressional budget justification materials a report on the number of its SSCs, as of the date of the report. Rather than report the number of SSCs, NNSA reported the names of vendors in its budget justifications. In its fiscal year 2017 congressional budget justification materials, NNSA reported the names of vendors but did not list the number of contracts it awarded to each vendor. In its congressional budget justification materials for fiscal years 2018 through 2020, NNSA reported the names of vendors and the contract number for each contract with a vendor. A count of the contracts included in NNSA’s annual congressional budget justification materials for this period showed NNSA used from 127 to 152 SSCs in fiscal years 2017 through 2020. NNSA officials involved with preparing the information included in the annual congressional budget justification materials said they made decisions on which SSCs to include and which to exclude based on the statutory language. According to these officials, because the requirements in the NDAA specified that NNSA was to report the data on the number of SSCs “as of the date of the report,” the officials interpreted that to mean they should only include contracts that were active on the date they queried their accounting and contracting databases. The officials said they excluded SSCs for which the contracts expired before NNSA officials prepared the information for the annual congressional budget justification materials. To prepare the information, the officials said that they obtained data on all contracts that were active on the day they queried the database, which was in mid- to late-October. The officials said that if a contract’s performance period ended prior to that date, they did not include the contract in the annual congressional budget justification materials, even if NNSA obligated funds to the contract in that year. For example, if a professional SSC reached the end of its 5-year term on September 15, 2018, that contract would not be included in NNSA’s reporting on SSCs for fiscal year 2018. However, according to the officials, information on the contract would have been included in the annual congressional budget justification materials in the 4 prior fiscal years. Although NNSA reported on SSCs that were active as of the date the officials queried the database in its congressional budget justification materials, this information is not complete because NNSA did not report information on all of the professional SSCs to which it obligated funds in those years. According to our analysis of data from FPDS-NG, NNSA excluded from 31 to 42 contracts each year from its annual congressional budget justification materials for fiscal years 2017 through 2020. These unreported contracts accounted for from about $10 million to $31 million in obligations for SSCs each year, as shown in table 4. The SSCs NNSA reported in the annual congressional budget justification materials align with the reporting requirements in the NDAAs for fiscal years 2016 and 2017. However, this information does not provide complete information on the number of SSCs that NNSA used or for which the agency obligated funds at some point during the fiscal year and does not disclose which contracts were excluded. For each SSC that NNSA excludes from its annual congressional budget justification materials, Congress does not have information, such as the amount obligated, number of FTE contractor personnel, or funding source— information that could assist congressional decision-making about NNSA’s workforce and annual appropriations levels. By reporting information on all professional SSCs to which funds were obligated during the fiscal year in its annual congressional budget justification materials, NNSA could provide more complete information on the number of SSCs used to meet mission requirements, assisting Congress in making better informed decisions about NNSA’s annual appropriations levels. The NDAA for Fiscal Year 2016 requires NNSA to report annually in its congressional budget justification materials on the number of FTE contractor personnel who have been working under each SSC for more than 2 years. NNSA did not provide this information in its annual congressional budget justification materials for fiscal years 2017 through 2020 because, according to the budget justification materials, NNSA does not collect information on individual contractor personnel from vendors. Specifically, NNSA included statements in its annual congressional budget justification materials for fiscal years 2017 through 2020 that the agency does not have information to address this requirement and that it is the responsibility of each individual contractor to determine who will perform the scope of work required by the terms and conditions of each contract. According to NNSA’s Office of the General Counsel, NNSA does not collect information on an individual contractor’s personnel because the vendor—not NNSA—is the employer for contractor’s employees and NNSA does not want to appear as if the agency is also their employer. Additionally, NNSA officials said that the agency does not have access to the personnel systems of its vendors and would not have information on whether contractor personnel worked on a contract for more than 2 years available to include in the annual congressional budget justification materials. NNSA officials also stated that they do not want to collect the names of individual contractors, although the NDAA for fiscal years 2016 and 2017 do not require NNSA to collect or report the names of individual contractor personnel working on contracts for more than 2 years. NNSA officials currently have access to information, such as employee badge records and office organizational charts, that can be used to develop notional, or approximate, information on the number of FTE contractor personnel who have worked on an SSC for more than 2 years. For example, we reviewed current organizational charts for several NNSA organizations that included the names of SSC personnel. Additionally, NNSA officials said that they could require vendors to track and report data on FTE contractor personnel assigned to an SSC for more than 2 years to NNSA on an annual basis. However, in addition to raising concerns about the perception of being a co-employer of the contractor personnel, the officials said that this additional requirement could increase contract costs and could be an administrative burden for NNSA and the contractors. Further, NNSA officials said it would be difficult to obtain the FTE data from vendors because, among other things, vendors’ methods for calculating FTE contractor personnel may vary from contract to contract and contractor personnel may work on a contract for only part of the year. The officials said the information would therefore need to be caveated significantly and may not be reliable. We understand the challenges in collecting the information; however, Congress has not modified or eliminated this reporting requirement in statute. In addition, the FAR identifies one element that may indicate a personal services contract as a service that can reasonably be expected to last more than 1 year. In a July 2015 report, the DOE’s Inspector General identified 14 contracts out of its sample of 20 that exhibited one or more characteristics of a personal services contract. According to the report, this situation could lead observers to question NNSA’s management of its SSCs, although the report did not find any clear violations. The report also stated that the Office of Defense Programs’ self-assessment found that many contractor employees appeared to be assigned to particular organizations for multiple years. However, NNSA cannot know the number of FTE contractor personnel who have been working under each SSC for more than 2 years because it does not collect this information. By collecting the information as required by law, NNSA could provide Congress—as well as its own decision makers—with greater insight into how NNSA is using its SSCs, including whether these SSCs display any of the characteristics of personal services contracts. NNSA reported information on obligations and funding sources used for SSCs in its annual congressional budget justification materials for fiscal years 2018 through 2020. However, some of the information is not fully useful to support congressional decision-making because it presents obligations for SSCs over multiple fiscal years, instead of presenting such obligations annually, and does not identify the specific program’s appropriation accounts, such as Weapons Activities and Defense Nuclear Nonproliferation, used to fund the contracts, as required by the fiscal year 2017 NDAA. The NDAA for fiscal year 2017 directs NNSA to report annually in its congressional budget justification materials on the cost of each SSC, as of the date of the report. According to NNSA officials who prepared the information, in the absence of specific guidance from Congress on the information to report, NNSA reported the obligations to date and the maximum value for each contract in its annual congressional budget justification materials for fiscal years 2018 through 2020 (see fig. 5). According to NNSA officials, the obligations-to-date column in the annual congressional budget justification materials represents the cumulative obligations on each contract from when it was awarded through the October prior to the submission of the materials, and the maximum value column represents the maximum amount that NNSA can obligate on the contract over the contract’s base term and any options. NNSA officials told us they reported the obligations to date and maximum value of the contracts because they determined that these measures met the definition for reporting information on the cost of the contracts, as required by the NDAA. According to the officials, they determined that obligations by fiscal year did not provide the total cost of an SSC because NNSA obligates funds on SSCs over multiple years, but the officials could provide obligations data by fiscal year if directed by Congress to do so. Additionally, NNSA officials said that the NDAA did not prescribe how the information was supposed to be reported, and they made a professional judgment on how best to report it. According to DOE’s information quality guidelines, the quality of information is measured, in part, by its utility, which the guidelines defined as the usefulness of the information to intended users. Because the information on the costs of SSCs is required to be included in NNSA’s report in its annual congressional budget justification materials, the intended users of the SSC information are the congressional appropriations and authorizing committees. However, staff from the Senate and House Armed Services Committees told us that the information on the cost of SSCs in the annual congressional budget justification materials was not fully useful because NNSA reported the amounts obligated over multiple fiscal years. By reporting information in this way, the cost data are not consistent across contracts and are not consistent with other information presented in the budget justification. Specifically: Cost data are not consistent across contracts. For fiscal years 2018 through 2020, NNSA presented the data on obligations to date and maximum value of the contract without identifying the period of time included for each individual contract. This period of time, particularly for the obligations-to-date data, could vary significantly and could represent a period of a few months if the contract was awarded late in the year or multiple years if a contract was reaching the end of its term and option periods. For example, NNSA reported obligating about $3.5 million on one SSC in its fiscal year 2019 annual congressional budget justification materials. Based on our analysis, NNSA obligated this amount over 4 years in amounts ranging from about $170,000 to about $1.2 million per year. Cost data are not consistent with other information in the budget justification. Other information in the annual congressional budget justification materials—which is used to support annual appropriations decisions or the budget request for the coming year—is subject to requirements in OMB’s Circular A-11, which states that agencies should generally present financial information in terms of budgetary resources by year in the annual congressional budget justification materials. As presented, users of the annual congressional budget justification materials could be unintentionally misled by the information that NNSA reported on its SSCs. For example, NNSA reported in its annual congressional budget justification materials for fiscal year 2020 that the maximum contract value for its SSCs in fiscal year 2018 totaled about $824 million and included 884 FTE contractor personnel, as shown in figure 5. Although the columns are labelled appropriately, users of the annual congressional budget justification materials could misinterpret the information to include obligations over a single year, and the user could—incorrectly—assume that NNSA spent an average of about $930,000 per contractor FTE. The NDAA for Fiscal Year 2016 directs NNSA to report annually in its congressional budget justification materials whether program or program- direction funds supported each SSC as of the date of the report. NNSA identified whether it funded each SSC through “program” or “Federal Salaries and Expenses” (which is program direction) accounts in its congressional budget justification materials for fiscal years 2017 through 2020 and totaled the cost data—which, as discussed earlier, represent multiple fiscal years of contract obligations—included in the table across all reported contracts (see fig. 5). As previously discussed, according to DOE’s information quality guidelines, quality information is measured by the usefulness of the information to the intended users. Staff from the Senate and House Armed Services Committees told us that the information on the funding source reported in the annual congressional budget justification materials was not fully useful because the budget justifications did not specify which program appropriation account—“Weapons Activities” or “Defense Nuclear Nonproliferation”—NNSA used to fund the SSCs and did not total the obligations by funding source. According to NNSA officials, they reported what was required by law. The NDAA directs NNSA to identify the funding source—either program or program direction accounts—for each SSC but does not specify that NNSA must report on the specific appropriations account or total the amount obligated by account. Based on our analysis of FPDS-NG data, NNSA’s obligations to SSCs varied significantly across the three appropriations accounts. For example, in fiscal year 2018, 84 percent of NNSA’s obligations for SSCs (about $162 million of the $194 million obligated for SSCs in that year) were from program appropriations and 15 percent (over $29 million) were from the Federal Salaries and Expenses account (see fig. 6). Of the amounts obligated for SSCs from program accounts in fiscal year 2018, 65 percent were from the Weapons Activities account, with the remaining 35 percent from the Defense Nuclear Nonproliferation account. These amounts represent about 1 percent of the total appropriations for Weapons Activities and about 3 percent for Defense Nuclear Nonproliferation. NNSA is reporting whether program or program direction funds support the contracts, as required. As previously discussed, NNSA guidance states that offices should use program funding for SSCs that produce deliverables and short-term, specific program-related technical support. However, by reporting in NNSA’s annual congressional budget justification materials the specific program appropriations account— Weapons Activities or Defense Nuclear Nonproliferation—used to fund each SSC and totaling the amounts obligated by appropriations account, NNSA would have more reasonable assurance that Congress had insight into which programs the SSCs supported. This reporting could facilitate congressional oversight of NNSA’s use of funds for SSCs by account and could assist NNSA in workforce planning should Congress reevaluate its FTE cap. NNSA identifies SSCs that are more likely to have the potential of including inherently governmental functions in its input to DOE’s annual service contract inventory analysis and its determination forms, but the agency may not be effectively managing the potential risks of SSCs that it determines may include such functions. The Consolidated Appropriations Act, 2010, requires civilian agencies to submit to OMB annual inventories of their service contracts. According to OMB guidance, the service contract inventory is a tool to assist an agency in better understanding how contracted services are being used to support mission and operations. NNSA’s input to DOE’s annual service contract inventory for fiscal years 2015 through 2017 identified a significant number of SSCs that included functions that approached being inherently governmental. For example, NNSA’s 2017 inventory analysis reported that contract specialists identified 621 of 775 contract actions, totaling over $170 million in obligations in that year, that were more likely to have the potential to include inherently governmental functions. The analysis identified 194 contract actions as closely associated with inherently governmental functions, 10 as critical functions, and 51 as both closely associated with inherently governmental functions and related to critical functions. Based on our analysis of data in FPDS-NG for fiscal year 2018, NNSA identified 37 of its 166 professional SSCs as closely associated with inherently governmental functions and 4 contracts as related to critical functions. Additionally, as discussed previously, prior to awarding an SSC, officials in the office for which the SSC will provide services and the contracting officer fill out a determination form that includes questions about whether the draft performance work statement includes tasks related to the parts of the FAR that identify inherently governmental functions and functions that can approach being inherently governmental. Tasks included in the performance work statements for SSCs vary widely and could present unique risks for including inherently governmental functions. The purpose of the determination form is to mitigate the risk of awarding an SSC that includes inherently governmental functions. The determination forms include a statement that, among other things, the agency has sufficient capacity and capability to give special management attention to contractor performance, limit or guide the contractor’s exercise of discretion, and avoid or mitigate conflicts of interest. To better understand how NNSA manages the risks of SSCs including inherently governmental functions, we reviewed the performance work statements for SSCs in our sample and, for contracts that had the potential to include inherently governmental functions, discussed how the contracting officials oversee contracts. For one contract we reviewed, the performance work statement called for the contractor to award contracts on behalf of NNSA with foreign organizations and review deliverables and technical performance. The FAR lists awarding contracts and administering contracts as two examples of functions considered to be inherently governmental. The contracting officials overseeing this contract said they do not typically see such a task in a performance work statement but noted that the contract was originally awarded in 2012, prior to those officials’ oversight of the contract. Contract oversight can change throughout the life of a contract—which can extend to 5 years and beyond—and the contracting officials assigned to manage an SSC can change throughout the contract. The contracting officials also told us that they were not concerned that the contract could include inherently governmental functions, as the program office supported by this contract was heavily involved in the activity. The FAR, however, states that awarding contracts and administering contracts are considered to be inherently governmental functions. In another contract we reviewed, the performance work statement included activities that, among other things, involved contractors conducting annual visits to a foreign country to meet and confer with military and governmental officials to develop opportunities for greater access by NNSA to foreign officials. The FAR lists the conduct of foreign relations and the determination of foreign policy among functions considered to be inherently governmental. The contracting officials for the contract said that the program office reviews information to be presented during the visits in advance of the meetings and that federal officials attend some of the meetings, allowing NNSA to ensure that the functions performed by the contractor do not become inherently governmental. In 2011, the Office of Federal Procurement Policy issued a policy letter that states agencies should review, on an ongoing basis, the functions being performed by their contractors, paying particular attention to the way in which contractors are performing, and agency personnel are managing, contracts involving functions that are closely associated with inherently governmental functions and contracts involving critical functions. According to the policy letter, these reviews should be conducted in connection with the development and analysis of inventories of service contracts. The policy letter also calls for agencies to ensure that they have sufficient internal capability to control their missions and operations. Additionally, according to the Consolidated Appropriations Act, 2010, after submitting the service contract’s inventories, the agency must review the contracts and information in the inventory and ensure that, among other things: the agency is giving special management attention to functions that are closely associated with inherently governmental functions; the agency is not using contractor employees to perform inherently governmental functions; the agency has specific safeguards and monitoring systems in place to ensure that work that contractors are performing has not changed or expanded during performance to become an inherently governmental function; the agency is not using contractor employees to perform critical functions in such a way that could affect the agency’s ability to maintain control of its mission and operations; and there are sufficient internal agency resources to manage and oversee contracts effectively. DOE’s service contract inventory analysis for fiscal year 2017 stated that NNSA offers training on inherently governmental contracts on a periodic basis and also uses the determination form, which is completed before the contract is awarded, to ensure that all contracts with inherently governmental potential receive proper attention. However, these steps may not allow NNSA to effectively manage the potential risks of contractors performing inherently governmental functions throughout the life of the contract. First, officials complete the required determination forms prior to awarding an SSC, and NNSA does not take steps to ensure that contracting officers document the steps that they plan to take to oversee specific SSCs, including those the agency determined carry a risk for the performance of inherently governmental functions. This is, in part, because the determination form does not require the contracting officers to include such information on the form. By documenting on the determination form specific steps that the contracting officer plans to take to address the risks of the particular contract, NNSA can better ensure that the functions contractors are performing and the way they perform them do not evolve into inherently governmental functions. Second, NNSA has no process—in connection with the development and assessment of the service contract inventory or another process—to verify that contracting officers are performing planned oversight. Under federal internal control standards, management should design control activities to achieve objectives and respond to risks, such as by comparing actual performance to planned or expected results and analyzing significant differences. By developing a process to verify that the contracting officer has implemented the planned oversight steps for SSCs that have a high risk of including inherently governmental functions throughout the term of the contract, NNSA would have better assurance that planned oversight was being carried out. Taking these actions could also help NNSA better ensure that planned oversight steps continue, even if the contracting officer or other oversight official changes during the term of the contract. Since 2010, NNSA has increasingly used professional SSCs across the agency to meet the demands of its increasing workload at a time when the size of its federal workforce has decreased. However, the use of SSCs can also prove challenging, as many of the services categorized as professional and management may be closely aligned with inherently governmental functions, increasing the risk that contractors may inappropriately influence the government’s authority, control, and accountability for decisions. We identified four ways NNSA could improve the completeness and usefulness of its reporting on its SSCs in its annual congressional budget justification materials. Such efforts could assist with congressional decision-making. First, NNSA did not include data on all professional SSCs to which funds were obligated during the fiscal year. By including such data, NNSA could provide more complete information on the number of SSCs used to meet mission requirements, assisting Congress in making better informed decisions about NNSA’s annual appropriations levels. Second, NNSA did not report information on the number of FTE contractor personnel working under the same contract for more than 2 years. NNSA officials identified difficulties in collecting the information. Collecting the information, as required by law, could provide Congress and NNSA’s own decision-makers with greater insight into how NNSA is using its SSCs. Third, NNSA did not present the cost of its SSCs in terms of obligations for 1 fiscal year. By reporting annual obligations data for each SSC, NNSA could more accurately represent its annual budgetary needs for the support needed to perform its missions. Fourth, NNSA did not identify the specific appropriations accounts used to fund SSCs. By identifying such accounts, NNSA would have more reasonable assurance that Congress had insight into which programs the SSCs supported, facilitating congressional oversight of NNSA’s use of funds for SSCs by account and assisting NNSA in workforce planning should Congress reevaluate NNSA’s FTE cap. Additionally, we identified two ways that NNSA could better manage the potential risks of contractors performing inherently governmental functions over the life of a contract. First, NNSA has not taken steps to ensure that contracting officers document the steps that they plan to take to oversee SSCs identified as at high risk of including inherently governmental functions on the determination forms. Second, NNSA does not have a process to verify that contracting officers are performing planned oversight for contracts that NNSA has identified as more likely to have the potential of including inherently governmental functions. By taking steps to document and verify that contracting officers have implemented the planned oversight steps for SSCs that may include inherently governmental functions throughout the term of the contract, NNSA would have better assurance that planned oversight was being carried out. We are making the following six recommendations to NNSA: The Associate Administrator for Acquisition and Project Management should report information on all professional SSCs to which funds were obligated during the fiscal year in its annual congressional budget justification materials. (Recommendation 1) The Associate Administrator for Acquisition and Project Management should collect and report all required data regarding the number of FTE contractor personnel employed under an SSC for more than 2 years. (Recommendation 2) The Associate Administrator for Acquisition and Project Management, in coordination with NNSA’s Office of Management and Budget, as appropriate, should report annual obligations data by fiscal year, as part of its reporting on SSCs in annual congressional budget justification materials. (Recommendation 3) The Associate Administrator for Acquisition and Project Management should report in NNSA’s annual congressional budget justification materials the program appropriations account—Weapons Activities or Defense Nuclear Nonproliferation—used to fund each SSC and total the amounts obligated by appropriations account. (Recommendation 4) The Associate Administrator for Acquisition and Project Management should take steps to ensure that contracting officers document—in the required determination form or elsewhere in the contract file—information on the steps that the contracting officers plan to take to oversee SSCs that NNSA has determined to be at high risk of including inherently governmental functions. (Recommendation 5) The Associate Administrator for Acquisition and Project Management should develop a process to verify that contracting officers are carrying out the steps identified to oversee contracts at risk of including inherently governmental functions throughout the term of the contract. (Recommendation 6) We provided a draft of this report to NNSA for review and comment. In its written comments, which are reproduced in full in appendix II, NNSA generally agreed with the report’s six recommendations and described actions that it intends to take in response to them. With regard to the second recommendation to collect and report required data on the number of full-time equivalent contractor personnel employed under an SSC for more than 2 years, we recognize the difficulties in collecting this information and appreciate that the agency intends to meet with congressional staff to discuss ways to address this issue. We continue to believe that collecting this information will provide NNSA and congressional decision-makers with greater insight into how NNSA uses its SSCs, including whether these SSCs display any of the characteristics of personal services contracts. With regard to the fifth recommendation to take steps to ensure that contracting officers document information on the steps the contracting officers plan to take to oversee SSCs that are determined to be at high risk of including inherently governmental functions, NNSA stated that it considers the recommendation closed based on processes already in place as well as the complementary activities discussed in response to our sixth recommendation. We continue to believe that documenting planned oversight activities in the contract files is important to ensure that planned oversight is consistent throughout the duration of the contract, particularly in light of OMB’s call for agencies’ ongoing review of the functions performed by its contractors and the potential for contracting officers to change over the life of the contract. The agency also provided technical comments, which we incorporated into our report, as appropriate. We are sending copies of this report to appropriate congressional committees, the Administrator of NNSA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or bawdena@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines the extent to which: (1) the National Nuclear Security Administration (NNSA) used professional support service contracts (SSC) in fiscal years 2010 through 2018, (2) the information about SSCs in NNSA’s annual congressional budget justification materials for fiscal years 2017 through 2020 is complete and useful to support congressional decision-making, and (3) NNSA manages the potential risks of SSCs that it determines are at high risk for providing inherently governmental functions. Overall, our review focused on NNSA’s use of professional SSCs. For the purposes of this report, we define professional SSCs to include contracts for activities such as program management support, administrative assistance, technical assistance, and engineering and technical services, consistent with NNSA’s definition of professional SSCs used to report the required information in its annual congressional budget justification materials. We excluded NNSA’s Office of Naval Reactors from our review because it is managed as a separate entity within NNSA. To address the first objective, we obtained and analyzed data on NNSA’s professional SSCs for fiscal years 2010 through 2018 from the Federal Procurement Data System –Next Generation (FPDS-NG), including the contract number, the amounts obligated to the contract in the fiscal year, the funding source, and the product service code assigned to the contract. We performed electronic testing of the data to identify missing data, obvious errors, or outliers and reviewed documentation and determined the data were sufficiently reliable to summarize the number of SSCs, amounts obligated, funding sources, and product service codes for NNSA’s SSCs in fiscal years 2010 through 2018. Unless otherwise specified, we report dollar figures as current dollars. In selected places, we also report inflation-adjusted dollars that are in constant 2018 dollars and were computed using a gross domestic product price deflator. To determine the kinds of tasks for which NNSA used its SSCs, we reviewed performance work statements for a nongeneralizable sample of 12 contracts. We selected contracts from the 407 SSCs NNSA reported in its annual congressional budget justification materials for fiscal years 2017 through 2019. We selected contracts that ranged in award amounts and represented work performed for different NNSA offices. In addition, to understand changes in NNSA’s use of SSCs, we analyzed data on NNSA’s appropriations and the number of federal full-time equivalent (FTE) employees for fiscal years 2010 through 2018. NNSA provided data on FTEs as of the last day of the last pay period of each fiscal year. We did not include federal FTE data by program office prior to fiscal year 2013 because NNSA restructured the organization, and the organizational structure prior to 2013 was not comparable to the current organization structure. We reviewed the data for obvious errors or outliers and compared the federal FTE data to other sources and discussed the data with officials and determined the data were sufficiently reliable to show changes in the size of NNSA’s work force over the time period. We also obtained and analyzed data by program office on the number of FTE contractor personnel from fiscal years 2015 through 2018. According to an NNSA official, NNSA did not collect data on FTE contractor personnel prior to fiscal year 2015. We reviewed the data for obvious errors or outliers and interviewed NNSA officials knowledgeable about the process to collect the data and NNSA officials that completed an internal study that, among other things, independently collected and verified the number of FTE contractor personnel by program office. Although we identified that NNSA did not report data on all of its SSCs, we determined the data were sufficiently reliable to illustrate changes in the number of FTE contractor personnel by program office for fiscal years 2015 through 2018. Further, to determine how NNSA uses its SSCs, we also reviewed two NNSA workforce studies and interviewed agency officials in program offices that used SSCs in fiscal years 2015 through 2018. To address the second objective, we compared the information on SSCs in NNSA’s annual congressional budget justification materials for fiscal years 2017 through 2020 with the requirements in the NDAA for fiscal years 2016 and 2017. We also reviewed documentation and interviewed NNSA officials from the Office of Acquisition and Project Management to determine how they prepared the information included in the annual congressional budget justification materials. We compared NNSA’s process for reporting information on SSCs to DOE’s information quality guidelines, particularly the sections related to completeness and usefulness of the information. Additionally, we compared the data on SSCs included in NNSA’s annual congressional budget justification materials to data in FPDS-NG to determine whether NNSA included all of its SSCs in the budget justification. To perform this analysis, we obtained data from FPDS-NG on all of NNSA’s active SSCs for fiscal years 2015 through 2018. We assessed the reliability for these data as described previously. For each fiscal year, we included only the SSCs that met NNSA’s definition of professional SSCs using the 77 product service codes. We also removed from the data any contracts listed that had $0 obligations or negative obligations for the fiscal year. For the remaining contracts, we compared the task order or contract numbers included in the FPDS-NG data to the task order or contract numbers that NNSA reported in its annual congressional budget justification materials. For those contracts where there was not a match between the annual congressional budget justification materials data and the FPDS-NG data on the task order or contract number, we reviewed the data manually to ensure there was not an error in the formula used or an error in the data that was easily identifiable, such as a transposed or missing digit in the task order or contract number. We discussed the list of contracts that was not included in NNSA’s annual congressional budget justification materials with officials responsible for the reporting to determine why the contracts were excluded. To address the third objective, we reviewed documents, such as applicable Federal Acquisition Regulation (FAR) provisions and NNSA policy documents, and interviewed officials from NNSA’s Office of Acquisition and Project Management, Office of Management and Budget, and Office of General Counsel to determine how NNSA oversees its SSCs. We also reviewed performance work statements for the nongeneralizable sample of 12 contracts discussed above to identify oversight activities and determine whether they included examples of tasks that could have characteristics of inherently governmental functions. We reviewed determination forms for eight of the 12 SSCs in our sample for which NNSA could provide the forms. We also interviewed NNSA’s contracting officers or contracting officer’s representatives and representatives from 11 of the 12 contractors in our sample to learn how NNSA and the contractors manage the contracts. When referring to the findings from these interviews, we use “some” to refer to 3 to 4 interviews, “several” to refer to 5 to 6 interviews, “many” to refer to 7 to 9 interviews, and “most” to refer to 10 to 11 interviews. In addition, we reviewed NNSA’s service contract inventory analysis reports from fiscal years 2015 through 2017 to obtain information on contracts that NNSA had identified as having the potential to include inherently governmental functions. We conducted this performance audit from October 2017 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Hilary Benedict (Assistant Director); Bridget Grimes (Analyst in Charge); Ellen Fried; Cindy Gilbert; Elizabeth Jimenez; Julia Kennon; Dan C. Royer; Sylvia Schatz; and Tatiana Winger made key contributions to this report.", "summary": "The Department of Energy's NNSA relies on federal employees and contractor personnel to carry out its mission. SSCs fill essential needs, and their use requires special diligence to ensure applicable statutes, regulations, and management practices are followed. The House report on the National Defense Authorization Act for Fiscal Year 2018 included a provision for GAO to report on NNSA's use of SSCs. This report examines the extent to which: (1) NNSA used SSCs for professional support in fiscal years 2010 through 2018; (2) the information about SSCs in NNSA's annual congressional budget justification materials for fiscal years 2017 through 2020 is complete and useful to support congressional decision-making; and (3) NNSA manages the potential risks of SSCs that it determines are at high risk for providing inherently governmental functions. GAO analyzed agency data; reviewed documentation; and interviewed federal and contractor officials representing a non-generalizable sample of 12 SSCs out of 407, selected to represent a range of years and contract obligations. The National Nuclear Security Administration (NNSA) obligated about $193 million in fiscal year 2018 for support service contracts (SSC), an increase of nearly 40 percent since 2010. These contracts provide a variety of professional support services, such as program management support. Officials attribute the increased use of SSCs to increases in appropriations and workload for the modernization of nuclear weapons and related infrastructure and decreases in the number of authorized federal staff due to the decrease in the statutory cap from fiscal year 2014 to 2015. Information on SSCs in NNSA's congressional budget justification materials is not complete or fully useful for congressional decision-making because, among other things, NNSA did not include information on all of its professional SSCs. NNSA is required to report annually certain information about SSCs, including the number and cost of SSCs, in its materials. NNSA reported information on its SSCs in its materials for fiscal years 2017 through 2020. However, NNSA's reporting was not complete because NNSA excluded information on 31 to 42 contracts each year (see fig. for fiscal year 2020). According to officials, they excluded contracts that expired during the fiscal year. By reporting information on all professional SSCs to which funds were obligated during the fiscal year, NNSA could provide more complete information to Congress that it could use to make better informed decisions about NNSA's annual appropriations levels. NNSA may not be effectively managing the potential risks of contractors performing inherently governmental functions—those that must be performed by a government employee—for contracts NNSA identifies as having the potential for providing such functions. NNSA identifies such SSCs through required assessments. However, contracting officers are not required to document planned steps to oversee these contracts, and the agency does not verify that planned oversight is performed. Contracting officers who oversee SSCs can change during the life of a contract. By documenting steps that contracting officers plan to take to oversee contracts with a high risk of including inherently governmental functions—and verifying that the planned oversight occurs—NNSA can better ensure over the life of the contract that the functions contractors are performing do not evolve into inherently governmental functions and that planned oversight is completed. GAO is making six recommendations to NNSA, including that NNSA: (1) report information on all professional SSCs to which funds were obligated during the fiscal year; (2) document plans to oversee SSCs that have a high risk of including inherently governmental functions, and (3) verify that the planned oversight occurs. NNSA generally agreed with the recommendations.", "document_type": "gao"}
{"report": "In April 2018, Facebook disclosed that a Cambridge University researcher may have improperly shared the data of up to 87 million of Facebook’s users with a political consulting firm. This followed other incidents in recent years involving the misuse of consumers’ personal information from the Internet, which about three-quarters of Americans use. These types of incidents have raised public concern because Internet-based services and products, which are essential for everyday social and economic purposes, often collect and use various forms of personal information that could cause users harm if released. The federal privacy framework for private-sector companies is comprised of a set of tailored laws that govern the use and protection of personal information for specific purposes, in certain situations, or by certain sectors or types of entities. Such laws protect consumers’ personal information related to their eligibility for credit, financial transactions, and personal health, among other areas. We reported in 2013 that no overarching federal privacy law governs the collection and sale of personal information among private-sector companies, including information resellers—companies that collect and resell information on individuals. We found that gaps exist in the federal privacy framework, which does not fully address changes in technology and the marketplace. We recommended that Congress consider legislation to strengthen the consumer privacy framework to reflect the effects of changes in technology and the marketplace. Such legislation has not been enacted. As we reported in January 2019, FTC is primarily a law enforcement agency with authority to, among other things, address consumer concerns about Internet privacy, both for Internet service providers and content providers. It does so using its general authority under section 5 of the FTC Act, which prohibits “unfair or deceptive acts or practices in or affecting commerce.” Even though the FTC Act does not speak in explicit terms about protecting consumer privacy, the Act authorizes such protection to the extent it involves practices FTC defines as unfair or deceptive. According to FTC, an act or practice is “unfair” if it causes, or is likely to cause, substantial injury not reasonably avoidable by consumers and not outweighed by countervailing benefits to consumers or competition as a result of the practice. FTC has used this “unfairness” authority to address situations where a company has allegedly failed to properly protect consumers’ data, for example. According to FTC, a representation or omission is “deceptive” if it is material and is likely to mislead consumers acting reasonably under the circumstances. FTC has applied this “deceptiveness” authority to address deceptions related to violations of written privacy policies and representations concerning data security, for example. FTC staff investigate Internet privacy complaints from various sources and also initiate investigations on their own. If FTC staff have reason to believe that an entity is engaging in an unfair or deceptive practice, they may forward an enforcement recommendation to the commission. The commission then determines whether to pursue an enforcement action. With certain exceptions, FTC generally cannot directly impose civil monetary penalties for Internet privacy cases. Instead, FTC typically addresses Internet privacy cases by entering into settlement agreements requiring companies to take actions such as implementing reasonable privacy and security programs. If a company then violates its settlement agreement with FTC, the agency can request civil monetary penalties in court for the violations. In addition, FTC can seek to impose civil monetary penalties directly for violations of certain statutes and their implementing regulations, such as the statute pertaining to the Internet privacy of children and its corresponding regulations. FTC has not promulgated rules under section 5 specific to Internet privacy. According to FTC staff, the process the agency must use to issue such rules—known as the Magnuson-Moss procedures—includes steps that add time and complexity to the rulemaking process. FTC has not promulgated any regulations using the Magnuson-Moss procedures since 1980. Although FTC has not implemented its section 5 authority by issuing regulations regarding internet privacy, it has issued regulations when directed and authorized by Congress to implement other statutory authorities using a different set of rulemaking procedures. These procedures, spelled out in section 553 of the Administrative Procedures Act (APA), are those that most federal agencies typically use to develop and issue regulations. APA section 553 establishes procedures and requirements for what is known as “informal” rulemaking, also known as notice-and-comment rulemaking. Among other things, section 553 generally requires agencies to publish a notice of proposed rulemaking in the Federal Register. After giving interested persons an opportunity to comment on the proposal by providing “data, views, or arguments,” the statute then requires the agency to publish the final rule in the Federal Register. In contrast, the rulemaking procedures that FTC generally must follow to issue rules under the FTC Act are the Magnuson-Moss procedures noted above. These are required by the Magnuson-Moss Warranty Act amendments to the FTC Act and impose additional rulemaking steps beyond APA section 553. These steps include providing the public and certain congressional committees with an advance notice of proposed rulemaking (in addition to the notice of proposed rulemaking). FTC’s rulemaking under Magnuson-Moss also calls for, among other things, oral hearings, if requested, presided over by an independent hearing officer, and preparation of a staff report after the conclusion of public hearings, giving the public the opportunity to comment on the report. FTC has promulgated regulations using the APA section 553 notice-and- comment rulemaking procedures when authorized or directed by specific statutes. For example, the 1998 Children’s Online Privacy Protection Act (COPPA) required FTC to issue regulations concerning children’s online privacy; promulgate these regulations using the APA section 553 process; and, in determining how to treat a violation of the rules, to treat it as an unfair or deceptive act or practice in most cases. COPPA governs the online collection of personal information from children under the age of 13 by operators of websites or online services, including mobile applications. COPPA contained a number of specific requirements that FTC was directed to implement by regulation, such as requiring websites to post a complete privacy policy, to notify parents directly about their information collection practices, and to obtain verifiable parental consent before collecting personal information from their children or sharing it with others. Laws and regulations may be enforced in various ways, for example, by seeking civil monetary penalties for non-compliance. As mentioned, FTC has authority to seek civil monetary penalties when a company violates a settlement agreement or certain statutes or regulations. For example, in March 2018, FTC announced that it is investigating whether Facebook’s current privacy practices violate a settlement agreement that the company entered into with FTC. In the case that resulted in the 2012 settlement, FTC had charged Facebook with deceiving consumers by telling them they could keep their information private, but then allowing it to be shared and made public. FTC also has authority to seek civil monetary penalties for violations of the COPPA statute as well as FTC’s COPPA regulations. In our January 2019 Internet privacy report, we found that during the last decade, FTC filed 101 Internet privacy enforcement actions to address practices that the agency alleged were unfair, deceptive, a violation of COPPA, a violation of a settlement agreement, or a combination of those reasons. Most of these actions pertained to first-time violations of the FTC Act for which FTC does not have authority to levy civil monetary penalties. In nearly all 101 cases, companies settled with FTC, which required the companies to make changes in their policies or practices as part of the settlement. Various stakeholders we interviewed for our January 2019 Internet privacy report said that opportunities exist for enhancing Internet privacy oversight. Most industry stakeholders said they favored FTC’s current approach—direct enforcement of its unfair and deceptive practices statutory authority, rather than promulgating and enforcing regulations implementing that authority. These stakeholders said that the current approach allows for flexibility; that regulations could hinder innovation, create loopholes, and become obsolete; and that rulemakings can be lengthy. Other stakeholders, including consumer advocates and most former FTC and FCC commissioners we interviewed, favored having FTC issue and enforce regulations. Stakeholders said that regulations can provide clarity, flexibility, and act as a deterrent, and may also promote fairness by giving companies notice of what actions are prohibited. Those stakeholders who believe that FTC’s current authority and enforcement approach is unduly limited identified three main actions that could better protect Internet privacy: (1) enactment of an overarching federal privacy statute to establish general requirements governing Internet privacy practices of all sectors, (2) APA section 553 notice-and- comment rulemaking authority, and (3) civil penalty authority for any violation of a statutory or regulatory requirement, rather than allowing penalties only for violations of settlement agreements or consent decrees that themselves seek redress for a previous statutory or regulatory violation. Stakeholders from a variety of perspectives—including academia, industry, consumer advocacy groups, and former FTC and FCC commissioners—told us that a statute could enhance Internet privacy oversight by, for example, clearly articulating to consumers, industry, and privacy enforcers what behaviors are prohibited. Some stakeholders suggested that such a framework could either designate an existing agency (such as FTC) as responsible for privacy oversight or create a new agency. For example, in Canada, the Office of the Privacy Commissioner, an independent body that reports directly to the Parliament, was established to protect and promote individuals’ privacy rights. Some stakeholders also stated that the absence of a comprehensive Internet privacy statute affects FTC’s enforcement. For example, a former federal enforcement official from another oversight agency said that FTC is limited in how it can use its authority to take action against companies’ unfair and deceptive trade practices for problematic Internet privacy practices. Similarly, another former federal enforcement official from another agency said that FTC is limited in how and against whom it can use its unfair and deceptive practices authority noting, for example, that it cannot pursue Internet privacy enforcement against exempted industries. In addition, some stakeholders said FTC’s section 5 unfair and deceptive practices authority may not enable it to fully protect consumers’ Internet privacy because it can be difficult for FTC to establish that Internet privacy practices are legally unfair. Because of this difficulty, some stakeholders said that FTC relies more heavily on its authority to take enforcement action against deceptive trade practices compared with the agency’s unfair trade practices authority. This is consistent with the results of our analysis of FTC cases, which showed that in a majority of the actions FTC settled, FTC alleged that companies engaged in practices that were deceptive. Furthermore, a recently decided federal appeals court case illustrates potential limits on FTC’s enforcement remedies. The court found that FTC could not direct the company, which was accused of unfair practices, to create and implement comprehensive data security measures for the personal information the company stored on its computer networks as a remedy for the practices alleged. Instead, the court ruled that FTC’s authority was limited to prohibiting specific illegal practices. Various stakeholders said that there are advantages to overseeing Internet privacy with a statute that provides APA section 553 notice-and- comment rulemaking authority. Officials from other consumer and worker protection agencies we interviewed described their enforcement authorities and approaches. For example, officials from CFPB and FDA, both of which use APA section 553 notice-and-comment rulemaking, said that their rulemaking authority assists in their oversight approaches and supports their enforcement actions. EEOC officials said that regulations are used to guide investigations that establish whether enforcement action is appropriate. Some stakeholders suggested that FTC’s ability to levy civil penalties could also be enhanced. As noted, FTC can levy civil penalties against companies for violating certain regulations, such as COPPA regulations, or for violating the terms of a settlement agreement already in place. According to most former FTC commissioners and some other stakeholders we interviewed, FTC should be able to levy fines for initial violations of section 5 of the FTC Act. An academic told us that the power of an agency to levy a fine is a tangible way to hold industries accountable. Recent data breaches at federal agencies, retailers, hospitals, insurance companies, consumer reporting agencies, and other large organizations highlight the importance of ensuring the security and privacy of personally identifiable information collected and maintained by those entities. Such breaches have resulted in the potential compromise of millions of Americans’ personally identifiable information, which could lead to identity theft and other serious consequences. For example, the breach of an Equifax online dispute portal from May to July 2017 resulted in the compromise of records containing the personally identifiable information of at least 145.5 million consumers in the United States and nearly 1 million consumers outside the United States. We reported in August 2018 that Equifax’s investigation of the breach identified four major factors— identification, detection, segmenting of access to databases, and data governance—that allowed the attacker to gain access to its network and extract information from databases containing personally identifiable information. In September 2017, FTC and CFPB, which both have regulatory and enforcement authority over consumer reporting agencies such as Equifax, initiated an investigation into the breach and Equifax’s response. Their investigation is ongoing. According to a 2017 National Telecommunications and Information Administration (NTIA) survey conducted by the U.S. Census Bureau, 24 percent of American households surveyed avoided making financial transactions on the Internet due to privacy or security concerns. NTIA’s survey results show that privacy concerns may lead to lower levels of economic productivity if people decline to make financial transactions on the Internet. Consumers who were surveyed indicated that their specific concerns were identity theft, credit card or banking fraud, data collection by online services, loss of control over personal information, data collection by government, and threats to personal safety. Recent data breaches and developments regarding Internet privacy suggest that this is an appropriate time for Congress to consider what additional actions are needed to protect consumer privacy, including comprehensive Internet privacy legislation. Although FTC has been addressing Internet privacy through its unfair and deceptive practices authority and FTC and other agencies have been addressing this issue using statutes that target specific industries or consumer segments, the lack of a comprehensive federal privacy statute leaves consumers’ privacy at risk. Comprehensive legislation addressing Internet privacy that establishes specific standards and includes APA notice-and-comment rulemaking and first-time violation civil penalty authorities could enhance the federal government’s ability to protect consumer privacy, provide more certainty in the marketplace as companies innovate and develop new products using consumer data, and provide better assurance to consumers that their privacy will be protected. In our January 2019 report, we recommended that Congress consider developing comprehensive legislation on Internet privacy that would enhance consumer protections and provide flexibility to address a rapidly evolving Internet environment. Issues that should be considered include: which agency or agencies should oversee Internet privacy; what authorities an agency or agencies should have to oversee Internet privacy, including notice-and-comment rulemaking authority and first-time violation civil penalty authority; and how to balance consumers’ need for Internet privacy with industry’s ability to provide services and innovate. Chairman Portman, Ranking Member Carper, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions you may have at this time. For further information regarding this testimony, please contact Alicia Puente Cackley at (202) 512-8678 or cackleya@gao.gov or Mark Goldstein at (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony include Andrew Huddleston, Assistant Director; Kay Kuhlman, Assistant Director; Bob Homan, Analyst-in-Charge; Melissa Bodeau; John de Ferrari; Camilo Flores; Nick Marinos, and Sean Standley. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "This testimony summarizes the information contained in GAO's January 2019 report, entitled Internet Privacy: Additional Federal Authority Could Enhance Consumer Protection and Provide Flexibility ( GAO-19-52 ). cackleya@gao.gov goldsteinm@gao.gov The United States does not have a comprehensive Internet privacy law governing the collection, use, and sale or other disclosure of consumers' personal information. At the federal level, the Federal Trade Commission (FTC) currently has the lead in overseeing Internet privacy, using its statutory authority under the FTC Act to protect consumers from unfair and deceptive trade practices. However, to date FTC has not issued regulations for Internet privacy other than those protecting financial privacy and the Internet privacy of children, which were required by law. For FTC Act violations, FTC may promulgate regulations but is required to use procedures that differ from traditional notice-and-comment processes and that FTC staff said add time and complexity. In the last decade, FTC has filed 101 enforcement actions regarding Internet privacy; nearly all actions resulted in settlement agreements requiring action by the companies. In most of these cases, FTC did not levy civil penalties because it lacked such authority for those particular violations. The Federal Communications Commission (FCC) has had a limited role in overseeing Internet privacy. From 2015 to 2017, FCC asserted jurisdiction over the privacy practices of Internet service providers. In 2016, FCC promulgated privacy rules for Internet service providers that Congress later repealed. FTC resumed privacy oversight of Internet service providers in June 2018. Stakeholders GAO interviewed had varied views on the current Internet privacy enforcement approach and how it could be enhanced. Most Internet industry stakeholders said they favored FTC's current approach—direct enforcement of its unfair and deceptive practices statutory authority, rather than promulgating and enforcing regulations implementing that authority. These stakeholders said that the current approach allows for flexibility and that regulations could hinder innovation. Other stakeholders, including consumer advocates and most former FTC and FCC commissioners GAO interviewed, favored having FTC issue and enforce regulations. Some stakeholders said a new data-protection agency was needed to oversee consumer privacy. Stakeholders identified three main areas in which Internet privacy oversight could be enhanced: Statute . Some stakeholders told GAO that an overarching Internet privacy statute could enhance consumer protection by clearly articulating to consumers, industry, and agencies what behaviors are prohibited. Rulemaking . Some stakeholders said that regulations can provide clarity, enforcement fairness, and flexibility. Officials from two other consumer protection agencies said their rulemaking authority assists in their oversight efforts and works together with enforcement actions. Civil penalty authority. Some stakeholders said FTC's Internet privacy enforcement could be more effective with authority to levy civil penalties for first-time violations of the FTC Act. Comprehensive Internet privacy legislation that establishes specific standards and includes traditional notice-and-comment rulemaking and broader civil penalty authority could enhance the federal government's ability to protect consumer privacy.", "document_type": "gao"}
{"report": "Information on consumers is exchanged through a consumer reporting process that includes consumers, CRAs, furnishers of consumer information, and users of consumer reports (see fig. 1). Consumers are individuals whose information is collected by CRAs and shared by CRAs with users of consumer reports to make decisions about eligibility, such as for credit, insurance, or employment. Information about consumers is generated through their participation in markets for goods and services—such as the use of banking or insurance products. CRAs are companies that regularly assemble or evaluate consumer information for the purpose of providing consumer reports to third parties. CRAs obtain data from a wide variety of sources, including data furnishers, such as banks and mortgage lenders, and public records. They can generate revenue by selling consumer reports to third parties. In 2012, CFPB estimated that the consumer reporting market consisted of more than 400 CRAs. CFPB estimated in 2015 that the three nationwide CRAs—which also are the three largest CRAs—held information on about 208 million Americans. Data furnishers are companies that report consumer information to CRAs. Examples of furnishers include banks, payday lenders, mortgage lenders, collection agencies, automobile-finance lenders, and credit card issuers. The information provided by furnishers may include personally identifiable information such as names, addresses, Social Security numbers, and employment data and account status and credit histories. A furnisher may provide CRAs with consumer information on multiple types of products. For example, a financial institution may provide account information on student loans as well as bank deposits. Furnishing of information to CRAs is generally voluntary; therefore, a furnisher is not required to submit all of the records about a consumer’s activity on an account to CRAs. Some accounts may only be reported when the payment history turns negative, such as when the debt is transferred to debt collection. Users of consumer reports include banks, credit card companies, landlords, employers, and other entities that use consumer reports to determine individual consumers’ eligibility for housing, employment, or products and services such as credit and insurance. Companies use consumer information compiled in consumer reports to screen for consumer risks and behaviors. For example, banks and credit unions may rely on consumer reports to assess the risk of opening new accounts. Some companies may act as both furnishers and consumer report users. During the consumer reporting process, a consumer does not necessarily interact with the CRA; however, if consumers discover inaccurate or incomplete information on their consumer reports as a result of, for example, being denied credit, they can file a dispute with the CRA, the furnisher, or both. Consumers may also request copies of their consumer reports from CRAs directly, and CRAs may provide consumers with certain disclosures about how their information is being shared. Different types of CRAs compile different types of reports using the data they obtain. The three nationwide CRAs produce credit reports and credit scores that can be used to qualify consumers for credit. Credit reports generally contain personally identifiable information, employment information, account status and credit histories, and inquiries made about consumers’ credit histories (see fig. 2). Other CRAs, called specialty CRAs, provide a variety of specialized reports used for making decisions on employment, rental housing, or other purposes. For example, reports from a specialty background-screening CRA may include some of the same information as a credit report but may also contain criminal history, education verification, and employment history. Several federal laws govern the consumer reporting industry, including the accuracy, security, use, and sharing of consumer report information. The Fair Credit Reporting Act (FCRA) is the primary federal law governing the collection, assembly, and use of consumer reports. FCRA was enacted to improve the accuracy and integrity of consumer reports, and promote the consumer reporting agencies’ adoption of reasonable procedures regarding the confidentiality, accuracy, relevancy, and proper use of consumer information. FCRA has been amended several times since it was enacted in 1970. When FCRA was originally enacted, FCRA imposed certain requirements on CRAs and users of consumer reports. Amendments to FCRA, pursuant to the Consumer Credit Reporting Reform Act of 1996 and the Fair and Accurate Credit Transactions Act of 2003, expanded the duties of CRAs, including requirements for dispute investigations, and imposed duties on data furnishers, such as requirements related to data accuracy and dispute investigations. FCRA requires CRAs and furnishers to take steps regarding the accuracy of the information contained in consumer reports. In addition, FCRA’s implementing regulation—Regulation V—as well as FTC’s Furnisher Rule more specifically outline furnishers’ responsibilities regarding accuracy. FCRA requires CRAs to follow reasonable procedures to assure “maximum possible accuracy” of the information concerning the individual to whom the report relates when preparing consumer reports. FCRA prohibits furnishers from reporting information that they know or have reasonable cause to believe is inaccurate, unless the furnisher has clearly and conspicuously specified to consumers an address whereby consumers can notify the furnisher that specific information is inaccurate. Regulation V and FTC’s Furnisher Rule require furnishers to have reasonable written policies and procedures in place regarding the accuracy and integrity of the information they provide to a CRA, where accuracy means that the information is for the right person and reflects the terms of the account and the consumer’s performance on the account. They also require furnishers to consider and incorporate, as appropriate, guidelines such as internal controls for accuracy and integrity of furnished information. FCRA requires CRAs and furnishers to address disputes consumers submit to them about the completeness or accuracy of information in their consumer reports. FCRA requires CRAs and Regulation V and FTC’s Furnisher Rule require furnishers to conduct reasonable investigations of a consumer’s dispute to determine the accuracy of the disputed information. As part of the process, CRAs and furnishers are required to consider all relevant information, including information provided by the consumer. The Gramm-Leach-Bliley Act (GLBA), provisions in the Federal Trade Commission Act, and provisions of the Dodd-Frank Wall Street Reform and Consumer Protection Act (Dodd-Frank Act), among other laws, govern the security of consumer data. Congress enacted GLBA in part to protect the privacy and security of nonpublic personal information that individuals provide to financial institutions. Many financial institutions furnish consumer data to CRAs. In a prior report, FTC staff told us that CRAs themselves might be considered financial institutions under GLBA if they collect, maintain, and report on consumer information. GLBA includes a provision directing FTC and certain federal regulators—including the Federal Reserve, FDIC, and OCC—to establish standards relating to administrative, technical, and physical safety for customer records. Specifically, GLBA directs these federal agencies to establish appropriate standards for financial institutions under their jurisdiction to ensure the security and confidentiality of customer records and information; protect against any anticipated threats or hazards to the security or integrity of such records; and protect against unauthorized access to or use of such records or information that could result in substantial harm or inconvenience to any customer. Provisions in the Federal Trade Commission Act prohibiting unfair or deceptive acts or practices and provisions in the Dodd-Frank Act prohibiting unfair, deceptive, or abusive acts or practices also may apply to CRAs’ protection of consumer data. Specifically, section 5 of the Federal Trade Commission Act prohibits “unfair or deceptive acts or practices” in or affecting commerce. In the context of privacy and security, these provisions require companies to represent practices to consumers in a truthful manner. For example, we reported previously that FTC has found companies that alleged they were following certain security protections, but did not in fact have such security features, to have engaged in unfair or deceptive practices. Similarly, the Dodd- Frank Act prohibits providers of consumer financial products or services from engaging in “unfair, deceptive, or abusive acts or practices.” For example, we reported previously that CFPB has alleged that claims to consumers that transactions are safe and secure while simultaneously lacking basic security practices can constitute “unfair, deceptive, or abusive acts or practices.” FCRA, GLBA, and the Economic Growth, Regulatory Relief, and Consumer Protection Act govern how consumer information may be used and shared. However, as we have previously reported, consumers have limited legal rights to control what personal information is collected and how it is maintained, used, and shared. For example, consumers generally cannot exercise choice in the consumer reporting market—such as by choosing which CRAs maintain their information—and do not have legal rights to delete their records with CRAs. FCRA permits CRAs to provide users of consumer reports the report only if the user has a “permissible purpose,” such as to process a credit application, screen a job applicant, or underwrite an insurance policy, subject to additional limitations where the credit or insurance transaction is not initiated by the consumer. FCRA also prohibits a person from using or obtaining a consumer report for any purpose other than that specified to the CRA when the user obtained the report. Further, FCRA requires that CRAs take steps to validate the legitimacy of users and their requests for consumer report information. FCRA and Regulation V also allow consumers to opt out of allowing CRAs to share their information with third parties for prescreened offers and limits the ability of affiliated companies to market products or services to consumers using shared consumer data. GLBA contains provisions regarding the use and sharing of consumer information that apply to CRAs. GLBA restricts the sharing of nonpublic personal information collected by or acquired from financial institutions. In particular, generally a nonaffiliated third party that receives nonpublic personal information from a financial institution faces restrictions on how it may further share or use the information. For example, a third party that receives nonpublic personal information from a financial institution to process consumer account transactions may not use the information for marketing purposes or sell it to another entity for marketing purposes. Consumers can prevent certain users from accessing their consumer reports by placing a security freeze on their consumer reports, which generally prevents the opening of new lines of credit in the consumer’s name (provided the creditor checks the consumer’s credit). Consumers may place a credit freeze at the three nationwide CRAs free of charge. Federal and state agencies share oversight of CRAs and furnishers. At the federal level, CFPB has supervisory authority over certain CRAs and shares enforcement and rulemaking authority with FTC for certain statutes applicable to all CRAs. At the state level, state Attorney General offices have enforcement authority to oversee CRAs, and some state agencies have limited supervisory authority under state laws. Federal agencies that have oversight authority for data furnishers are CFPB, FTC, and prudential regulators—the Federal Reserve, FDIC, NCUA, and OCC. Their oversight authority depends on the size as well as the type of the furnisher, such as if the furnisher is a nonbank institution, depository institution, or credit union. CFPB is the only federal agency with supervisory authority over CRAs, but it generally shares enforcement authority over CRAs with FTC as well as rulemaking authority for certain statutes applicable to all CRAs (see table 1). CFPB’s supervisory authority includes the authority to perform examinations to assess compliance with FCRA and other Federal consumer financial laws and to detect and assess risk to consumers and markets. CFPB may issue matters requiring attention (MRA) based on its examinations. MRAs identify corrective actions that result from examination findings that require the attention of the supervised institution’s board of directors or principals, including violations of Federal consumer financial laws. According to CFPB, MRAs are not legally enforceable, but CFPB can use them to determine future supervisory work or the need for potential enforcement actions. CFPB’s supervisory authority is generally limited to CRAs that qualify as larger participants in the consumer reporting market. In 2012, CFPB defined larger participants of the consumer reporting market to include CRAs with more than $7 million in annual receipts resulting from consumer reporting activities. CFPB’s authority does not extend to CRAs that do not participate in activities involving the use of consumer information to make decisions regarding financial products or services. For example, a specialty CRA that only provides consumer reports regarding a consumer’s employment history may not be considered a larger participant for the purposes of CFPB supervision, even if the CRA’s annual receipts from this activity are more than $7 million. In the preamble to its 2012 rule defining larger participants, CFPB stated that the threshold of more than $7 million is consistent with the objective of supervising market participants that have a significant impact on consumers and is appropriate in light of the highly concentrated nature of the consumer reporting market. In particular, CFPB estimated that out of about 410 CRAs, 30 CRAs met the threshold. Of those 30 CRAs, CFPB estimated that the six largest CRAs generated approximately 85 percent of industry receipts. While CFPB generally has supervisory authority over only larger- participant CRAs, CFPB and FTC generally share enforcement authority over CRAs. For example, they both enforce CRA compliance with most provisions of FCRA and provisions in other laws related to unfair or deceptive acts or practices. Both agencies have similar enforcement tools, including investigation, civil penalties, monetary relief for consumers, and requirements for a company to conduct or refrain from conducting certain acts. CFPB and FTC entered into a memorandum of understanding to coordinate their enforcement efforts, and staff from both agencies told us they take additional actions to coordinate their enforcement activities. For example, FTC staff said that CFPB and FTC maintain a log of each agency’s investigations to avoid duplication. Additionally, CFPB and FTC staff said they hold periodic coordination meetings to discuss their enforcement activities. FTC staff told us that because CFPB possesses supervisory authority over the three largest CRAs, FTC has focused its FCRA enforcement efforts on other CRAs. However, FTC staff said that to the extent that the largest CRAs offer nonfinancial products or services, such as employment or tenant background screening, FTC will also investigate these activities. CFPB and FTC each have certain rulemaking authority in connection with statutes that may apply to CRA activities, but generally CFPB has broader authority than FTC. Generally, CFPB has broad authority to issue regulations for Federal consumer financial laws, including most provisions of FCRA, which are applied to all CRAs. FTC has specific rulemaking authority that may apply to CRAs under FCRA, the Federal Trade Commission Act, and GLBA. For example, FTC’s rule related to safeguarding the security and confidentiality of customer records under GLBA applies to CRAs. State agencies, such as state Attorney General offices, have enforcement authority to oversee CRAs, and some state agencies have limited supervisory authority under state laws. Federal laws establish enforcement authority for state agencies over CRAs. Under FCRA and the Dodd-Frank Act’s provisions prohibiting unfair, deceptive, or abusive acts and practices, state Attorney General offices (or another official or agency designated by the state) have certain enforcement authority over some companies, including certain CRAs. However, states are required to coordinate enforcement actions with CFPB and FTC. In addition to enforcement authority under federal laws, state agencies may have enforcement authority under their state laws that apply to CRAs. Staff from state agencies in four selected states—Ohio, New York, Maine, and Maryland—told us that their states’ Attorney General offices have enforcement authority over CRAs under state laws prohibiting unfair or deceptive acts or practices. In addition, according to the National Consumer Law Center, every state has a consumer protection law that prohibits deceptive acts or practices and many states prohibit unfair acts or practices, and the enforcement of such state laws typically is the responsibility of a state enforcement agency, such as the state Attorney General offices. Some state Attorney General offices have used their enforcement authority under FCRA and state laws prohibiting unfair or deceptive acts or practices to investigate and take enforcement actions against CRAs. For example, the three nationwide CRAs entered into two separate settlements with 30 state Attorney General offices in 2015 in which the CRAs agreed to implement a number of specific reforms, including reforms related to consumer report accuracy and dispute processes. Under these settlements, the state Attorney General offices claimed the CRAs violated FCRA and the states’ laws prohibiting unfair or deceptive acts or practices. Additionally, representatives of several states’ Attorney General offices told us in connection with a prior report that they launched a joint investigation into whether a nationwide CRA violated state laws in a 2017 data breach, including state laws prohibiting unfair or deceptive practices. In addition to the enforcement authority state Attorney General offices have under state laws prohibiting unfair or deceptive acts or practices, some state laws provide state agencies, such as financial regulators and consumer protection bureaus, with oversight authority over CRAs. Our interviews with staff from four selected states’ agencies—Ohio, New York, Maine, and Maryland—indicated that CRA oversight authority given to state agencies under state laws varies. Staff from Ohio’s Office of the Attorney General told us that Ohio does not have specific laws that provide Ohio state regulators with supervisory, rulemaking, or enforcement authority over CRAs, apart from Ohio laws prohibiting unfair or deceptive acts or practices that provide the Office of the Attorney General with enforcement authority. New York’s financial regulator told us that state laws provide the agency with supervisory, enforcement, and rulemaking authority over institutions that provide financial products and services, including certain CRAs. The agency issued a rule in 2018 requiring CRAs reporting on consumers within the state to register with the agency annually and provide information as required by the agency. Staff from Maine’s consumer protection agency told us that under Maine law, the agency has supervisory and enforcement authority over CRAs operating within the state. Agency staff told us that the agency examines certain CRAs every 2 years for compliance with Maine’s consumer reporting laws, such as by reviewing records and documents provided by CRAs. Maryland’s financial regulator told us that Maryland’s laws provide the agency with enforcement and rulemaking authority over CRAs but not supervisory authority. The agency can adopt regulations in order to administer provisions of Maryland statutes, such as procedures for ensuring accuracy in consumer reports. Additionally, agency staff said that the agency can investigate CRAs using its enforcement authority but cannot conduct supervisory examinations of CRAs. Representatives from several CRAs we interviewed told us that their supervision by state regulators has been limited. Representatives from two CRAs told us that a state agency has examined them. Representatives from three other CRAs we interviewed said they had limited encounters with state-level agencies. However, as previously stated, CFPB, FTC, and state agencies generally have enforcement authority over CRAs regarding consumer financial protection. CFPB, FTC, and the prudential regulators—the Federal Reserve, FDIC, NCUA, and OCC—share federal oversight of data furnishers for compliance with FCRA, among other Federal consumer financial laws. These furnishers include insured depository institutions and credit unions and nondepository institutions, such as student and mortgage loan servicers. Federal agencies generally split oversight of furnishers based on their charter type and asset size. Oversight of furnishers that are depository institutions or credit unions. CFPB and the prudential regulators have supervisory and enforcement authority over insured depository institutions and credit unions for compliance with FCRA and other federal consumer financial laws (see table 2). The Dodd-Frank Act generally divided authority between CFPB and the prudential regulators based on an institution’s charter type and the value of an institution’s total assets. Assets of more than $10 billion. In general, CFPB has enforcement and supervisory authority for insured depository institutions and credit unions (as well as their affiliates) that have more than $10 billion in total assets for compliance with many Federal consumer financial laws.However, a prudential regulator that is authorized to enforce a Federal consumer financial law may recommend that CFPB initiate an enforcement action, and if CFPB does not, the prudential regulator may initiate an enforcement action. Assets of $10 billion or less. In general, the four prudential regulators have enforcement and supervisory authority over insured depository institutions or credit unions with total assets of $10 billion or less. If, however, CFPB believes that an institution in this category has violated a Federal consumer financial law, it must notify the appropriate prudential regulator in writing and recommend action. Additionally, regardless of an institution’s asset size, CFPB generally has rulemaking authority for many Federal consumer financial laws that apply to insured depository institutions and insured credit unions. However, prudential regulators have limited rulemaking authority as related to furnishing activities for certain provisions specifically retained pursuant to the Dodd-Frank Act and FCRA. CFPB generally has supervisory and enforcement authority over insured depository institutions and insured credit unions, as well as their affiliates, that have more than $10 billion in total assets, for compliance with Federal consumer financial laws as defined under the Dodd-Frank Wall Street Reform and Consumer Protection Act. CFPB has broad rulemaking authority under many Federal consumer financial laws that apply to depository institutions and credit unions, with limited exceptions. The Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation, National Credit Union Administration, and Office of the Comptroller of Currency (collectively called the prudential regulators) generally have supervisory and enforcement authority for Federal consumer financial laws (as defined under the Dodd-Frank Wall Street Reform and Consumer Protection Act) for insured depository institutions and credit unions that have $10 billion or less in total assets. The prudential regulators also have limited rulemaking authority related to furnishing activities under certain Federal consumer financial laws, including parts of FCRA. Oversight of furnishers that are nondepository institutions. CFPB and FTC share oversight of nondepository institutions. In general, CFPB has supervisory authority over certain types of nondepository financial institutions for compliance with FCRA and other Federal consumer financial laws (see table 3). Such institutions include certain kinds of mortgage market participants, private student lenders, and payday lenders. CFPB also has supervisory authority over institutions in markets for consumer financial products or services that it defines as larger participants. For example, CFPB has issued rules defining larger participants for automobile-financing and consumer debt-collection markets. CFPB and FTC share enforcement authority for many different types of nondepository institutions, such as mortgage lenders, payday lenders, debt collectors, and telecommunication companies. FTC additionally has enforcement authority over other nondepository institutions for which CFPB does not have enforcement authority, such as automobile dealers. FTC staff told us that, similar to their coordination efforts for CRAs, FTC and CFPB coordinate their enforcement activities with respect to furnishers. CFPB has rulemaking authority for most consumer financial laws, including FCRA, that may apply to furnishers that are nondepository institutions. In comparison to CFPB, FTC has specific rulemaking authority under FCRA, the Federal Trade Commission Act, and GLBA to promulgate rules that may apply to nondepository institution furnishers. CFPB has supervisory authority over certain nondepository institutions. It shares enforcement authority with FTC for certain nondepository institutions and has broad rulemaking authority for Federal consumer financial laws which apply to many institutions, including those for which it has supervisory jurisdiction. According to CFPB, in its oversight of the consumer reporting market, CFPB has prioritized CRAs representing the greatest potential risks to consumers. Additionally, CFPB has generally focused on certain compliance areas, particularly data accuracy and investigations conducted in response to consumer disputes. On an annual basis, CFPB updates its plans for supervision of CRAs and furnishers for the next 1 to 2 years. According to CFPB, it assesses specific risks in the market and determines entities and compliance areas to examine. In making these determinations, CFPB stated that it considers factors such as market presence, consumer complaints, its prior supervisory examinations and findings, and its resources. According to CFPB, since the start of its supervisory program for the consumer reporting market in 2012, CFPB has prioritized CRAs that pose the greatest risks to consumers and the marketplace for examinations. Specifically, CFPB staff told us that CFPB has prioritized CRAs that represent a significant share of the market and the largest volume of consumer complaints submitted to CFPB’s complaint database. CFPB has also examined one or more specialty CRAs. CFPB stated that in determining which specialty CRAs to examine, it considered factors such as the CRAs’ market share in the particular consumer reporting products they offer. According to CFPB, in setting supervisory priorities, supervision staff also consulted with stakeholders and considered CFPB’s resources and findings from prior examinations that may have indicated weaknesses. CFPB staff said that when CFPB began examining CRAs, its supervisory approach was to examine their compliance management systems first before focusing on other compliance areas. The staff said that the compliance management system reviews helped CFPB to learn about how CRAs operate. Based on the compliance management reviews, CFPB determined that it could review data accuracy, dispute investigations, and other compliance areas by examining the mechanisms CRAs use to comply with FCRA. After examining compliance management systems, CFPB prioritized examining other aspects of compliance related to data accuracy (including processes for monitoring furnishers) and dispute investigations, as well as performing follow-up examinations in those areas. CFPB staff stated that they have chosen to focus on data accuracy and dispute investigations because these were the largest problem areas based on CFPB’s assessment of complaint data. Additionally, CFPB identified compliance with the FCRA obligations regarding data accuracy and effective and efficient dispute resolution as agency priorities for the consumer reporting market. CFPB has also examined other CRA compliance areas, including procedures related to suppression and reinsertion of information that CRAs found to be inaccurate, unverifiable, or obsolete; procedures for ensuring a permissible purpose for obtaining consumer reports; and compliance management systems related to data security. According to CFPB, when determining compliance areas for examination, the agency considered factors such as its data on complaints, the extent to which it had previously examined the areas, and concerns identified in prior examinations. In February 2019, we found that CFPB’s examination process did not routinely include an assessment of CRAs’ data security risks, and we recommended that CFPB’s prioritization specifically account for data security risk. In conducting its examinations, CFPB has focused on assessing CRA procedures for complying with FCRA rather than on the extent of inaccuracy in consumer reports. For example, according to a 2017 CFPB report, CFPB directed one or more CRAs to establish quality control programs to regularly assess the accuracy of information included in consumer reports and to develop systems to measure the accuracy of consumer reports and identify patterns and trends in errors. CFPB staff said CFPB has not monitored the extent of inaccuracy in consumer reports produced by the CRAs it examines. They stated that FCRA requires CRAs and furnishers to follow reasonable procedures with regard to accuracy but does not require or identify acceptable thresholds for accuracy. CFPB staff explained that CFPB’s supervisory program has therefore focused on evaluating CRAs’ compliance with FCRA requirements for reasonable procedures with regard to accuracy and identifying weaknesses in such procedures. According to CFPB, in prioritizing examinations of data furnishers, the agency has primarily considered the furnishers’ market shares, the number of disputes CRAs received about the furnishers, and the number of complaints CFPB received in its complaint database. CFPB has prioritized large furnishers within their respective markets. For example, CFPB identified one or more student loan servicers furnishing data to CRAs that had large shares of the student loan servicing market. CFPB has also prioritized furnishers with high dispute rates relative to other furnishers within their markets. For example, CFPB identified one or more credit card issuers with higher dispute rates compared to their peers, based on CFPB’s review of dispute data provided by CRAs. According to CFPB, it has also considered the results of prior CFPB examinations and input from agency stakeholders. As with CRAs, CFPB’s examinations of furnisher activities have focused on accuracy and dispute investigations. In its Supervisory Highlights from March 2017, CFPB stated that the accuracy of consumer report information is a CFPB priority and that furnishers play an important role in ensuring the accuracy of consumer report information through the dispute process. For example, CFPB stated that furnishers’ timely response to consumer disputes may reduce the effect that inaccurate negative information on a consumer report may have on the consumer. From 2013 through 2018, CFPB examined several CRAs. Many of these examinations evaluated CRA compliance with accuracy and dispute investigation obligations under FCRA, such as by assessing data governance systems, quality control programs, and furnisher oversight and data monitoring. Additionally, some examinations evaluated other FCRA compliance areas, including ensuring that users had permissible purposes for requesting consumer reports and preventing reinsertion of previously deleted information. CFPB’s examinations related to data accuracy and dispute investigation obligations resulted in supervisory findings that CFPB directed CRAs to take actions to address. CFPB found that one or more CRAs had minimal compliance mechanisms in place to meet requirements for data accuracy and for dispute investigations (see table 4 for examples of CFPB’s supervisory findings and directed actions in these areas). For example, CFPB found that one or more CRAs lacked quality control policies and procedures to test compiled consumer reports for accuracy and had insufficient monitoring and oversight of furnishers that provided information used in the reports. CFPB also found that one or more CRAs did not review evidence that consumers provided to support their disputes and relied entirely on the furnishers to investigate the disputes. CFPB directed specific changes in some CRAs’ policies and procedures for ensuring data accuracy and conducting dispute investigations, including increasing oversight of incoming data from furnishers, developing internal processes to monitor furnisher dispute responses to detect those that may present higher risk of inaccurate data, and enforcing the CRAs’ obligation to investigate consumer disputes, including review of relevant information provided by consumers. In addition, CFPB directed one or more CRAs to establish quality control programs that regularly assess the accuracy and integrity of compiled consumer reports. In follow-up reviews of some of its supervisory findings, CFPB found that one or more CRAs took actions that resulted in improvements in policies and procedures. For example, CFPB has found that one or more CRAs established quality control programs, including developing tests to identify the extent to which consumer reports are produced using information for the wrong consumer. For other findings, CFPB determined that one or more CRAs had not taken actions to address the findings, or CFPB had not yet conducted follow-up examinations to determine if they had been addressed. From 2013 through 2018, CFPB conducted examinations of several data furnishers. These furnishers were involved in various consumer financial markets, such as automobile loan servicing, debt collection, mortgage servicing, and student loan servicing. CFPB staff told us that until 2017, CFPB generally examined furnishers’ compliance with FCRA as part of its assessment of compliance with other Federal consumer financial laws and regulations. CFPB staff said that in 2017, CFPB began conducting examinations specifically focused on furnishing activities under FCRA and Regulation V. CFPB stated that this change was made because the review of furnishers’ practices under FCRA and Regulation V was resource-intensive and merited dedicated resources. In a 2017 report, CFPB stated that it had found numerous furnisher violations of FCRA and Regulation V related to data accuracy and dispute investigations and that it directed furnishers to take corrective actions (see table 5 for examples of CFPB’s supervisory findings and directed actions). For example, CFPB found that certain furnishers failed to establish, implement, and maintain reasonable written policies and procedures consistent with Regulation V regarding the accuracy and integrity of the information furnished; provided information to CRAs despite having reasonable cause to believe the information was inaccurate; and lacked policies for their employees on how to conduct reasonable investigation of consumer disputes. In some cases, CFPB’s furnisher examinations conducted from 2013 through 2018 resulted in findings related to FCRA and Regulation V that CFPB directed the furnishers to take actions to address. For example, CFPB directed furnishers to develop reasonable written policies and procedures regarding accuracy, to promptly update the information provided to CRAs after determining that the information was not complete or accurate, and to update and implement policies and procedures to ensure disputes are handled in accordance with FCRA requirements. CFPB staff told us that the agency decides whether to investigate based on consideration of factors such as consumer complaints, extent of effects on consumers, and severity of misconduct. CFPB staff told us that, in many cases, CFPB has chosen to identify and correct FCRA violations and weaknesses in compliance management systems at CRAs through supervisory activity rather than enforcement investigations. However, CFPB has also investigated and used enforcement remedies, such as civil penalties and injunctive relief, against CRAs and furnishers that violated FCRA or Regulation V. From 2012 through 2018, CFPB settled 26 enforcement actions for violations related to FCRA and Regulation V, including four settlements involving CRAs and 16 settlements involving furnishers. Although CFPB found other FCRA violations in its investigations of these companies, such as those related to permissible purpose for obtaining consumer reports and disclosure issues, most of the violations related to data accuracy and dispute investigations. For example, two of the four FCRA-related settlements with CRAs involved dispute investigations or data accuracy procedures. Of the 16 settlements with furnishers for alleged violations related to FCRA and Regulation V, all contained violations related to the furnishers’ obligations regarding data accuracy or dispute investigations. CFPB’s settlements contained findings similar to its supervisory examination findings. For example, CFPB found that a CRA failed to investigate consumer disputes, and another CRA failed to take steps to ensure its consumer reports were accurate. For furnishers, CFPB found violations including furnishers that failed to establish and implement reasonable written policies and procedures regarding the accuracy and integrity of information provided to CRAs, as well as furnishers that provided inaccurate or incomplete information about consumers to CRAs or failed to conduct reasonable investigations of consumer disputes. CFPB has not defined its expectations—including views on appropriate practices—for how CRAs can comply with key FCRA requirements. Among other provisions, FCRA requires CRAs to (1) follow reasonable procedures when preparing a consumer report to assure maximum possible accuracy of consumer report information and (2) conduct reasonable investigations of consumer disputes. However, FCRA does not define what would constitute such reasonable policies and procedures or investigations or stipulate the types of procedures or investigations that would meet the requirements for CRAs. While CFPB has not defined its expectations for these two key FCRA requirements for CRAs, it has adopted Regulation V, which, as required by statute, includes information on CFPB’s requirements and guidelines in these areas for furnishers. Regulation V includes requirements and guidelines for reasonable policies and procedures concerning the accuracy and integrity of furnished consumer information and requirements for reasonable investigations of consumer disputes filed directly with the furnishers. In its supervision of furnishers, CFPB has examined furnishers for compliance with the requirements of Regulation V—for example, it has found in examinations that furnishers violated Regulation V’s requirement to establish written policies and procedures regarding the accuracy of consumer information furnished to a CRA. Although CFPB has not similarly established guidelines or otherwise provided information on its supervisory expectations for CRAs, CFPB has found specific weaknesses in CRA practices. In particular, CFPB has cited one or more CRAs for specific deficiencies related to determinations of noncompliance with FCRA provisions regarding reasonable procedures for accuracy and dispute investigations. For example, CFPB has directed one or more CRAs to take specific actions to improve their accuracy procedures. In addition, CFPB found one or more CRAs’ data governance programs to be decentralized and informal, and it directed the CRAs to develop and implement written policies and procedures to formalize the programs. However, CFPB has not issued any information on its supervisory expectations indicating that “reasonable procedures” include having formal written policies and procedures. CFPB also has identified FCRA violations related to reasonable dispute investigations. For example, CFPB determined that one or more CRAs failed to review and consider documentation attached by consumers to disputes and relied entirely on furnishers to investigate a dispute—therefore violating FCRA requirements for reasonable investigations and for reviewing and considering all relevant information submitted by the consumer—and directed the CRAs to independently investigate consumer disputes. However, CFPB has not issued any information on its supervisory expectations to help interpret FCRA’s requirement for CRAs to conduct a reasonable investigation of disputes and to review and consider all relevant information, including the expectation that CRAs investigate consumer disputes independently. Based on the FCRA requirements alone, it may be unclear to CRAs and others that these FCRA requirements include performing independent investigations. For example, representatives from one large CRA we interviewed stated that the company is not required to conduct an independent investigation. FCRA instructs CFPB to enact regulations that are necessary to carry out the purposes of the act, which could include issuing implementing regulations for CRAs regarding data accuracy and dispute investigations. Additionally, a 2018 policy statement issued by CFPB and the prudential regulators explains that information on supervisory expectations serves to articulate an agency’s general views regarding appropriate practices. The policy statement further states that it is important for such information to provide insight to industry, as well as to supervisory staff, in a transparent way that helps to ensure consistency in the supervisory approach. According to CFPB’s Supervisory Highlights from March 2017, CFPB’s vision for the consumer reporting system is a system in which furnishers provide and CRAs maintain and distribute data that are accurate, supplemented by an effective dispute management and resolution process for consumers. According to the same CFPB publication, this vision is rooted in the obligations and rights set forth in FCRA and Regulation V. One reason why accuracy guidelines exist for furnishers but not CRAs is that the Fair and Accurate Credit Transactions Act of 2003 added a provision to FCRA requiring the prudential regulators and FTC to establish and maintain guidelines for furnishers regarding the accuracy of consumer data furnished to CRAs and to prescribe regulations requiring furnishers to establish reasonable policies and procedures for implementing the guidelines. In 2011, CFPB adopted these regulations as part of Regulation V after assuming rulemaking authority from the other agencies. Neither the Fair and Accurate Credit Transactions Act of 2003 nor any other statutory provision within FCRA includes a similar provision for the agencies to establish and maintain guidelines or provide information concerning supervisory expectations regarding the accuracy of consumer data CRAs maintain, and CFPB has not established guidelines or defined supervisory expectations for CRAs. Since 2015, CFPB’s long-term rulemaking agenda has stated that it will evaluate possible policy responses to consumer reporting issues, including potential additional rules or amendments to existing rules governing consumer reporting accuracy and dispute processes. However, as of May 2019, CFPB had not conducted any rulemaking related to these topics. CFPB staff said that a substantial body of case law exists to guide CRAs regarding what practices may be considered compliant with FCRA requirements, including with respect to provisions for reasonable procedures for accuracy and performing reasonable dispute investigations. The staff also said that CFPB staff look to relevant case law when assessing CRA compliance with FCRA during examinations, and that supervisory findings serve to communicate to a supervised CRA how CFPB has applied FCRA during an examination. Providing information to CRAs about its supervisory expectations for these two key FCRA requirements—and ways in which CRAs could comply—could help CFPB to facilitate CRA compliance with FCRA and achieve agency objectives for the consumer reporting system. Without information about its expectations, CFPB’s supervision lacks transparency about what practices it considers appropriate or expects CRAs to adopt to comply with key FCRA requirements. Absent such information from CFPB, representatives from four CRAs we interviewed told us that they look to other sources to understand what CFPB will consider to be noncompliant during examinations and to determine if they are meeting FCRA requirements for maintaining reasonable procedures. These sources include publicly available information such as court cases, presentations from industry associations, CFPB publications highlighting supervisory actions, and public enforcement actions. While CFPB can communicate with individual CRAs during examinations and by directing corrective actions, the impact of such interactions is limited to specific CRAs rather than helping to ensure consistency in its supervisory approach by providing transparent insights to the industry. While relevant case law could provide CRAs with some information regarding practices that have been determined to be compliant with FCRA requirements, there may be a lack of clarity about the extent to which all case law fully reflects CFPB’s expectations. By communicating information about its expectations and ways in which CRAs could comply, CFPB could help ensure that CRAs receive complete and consistent information about how to interpret key FCRA requirements. Further, defining its expectations regarding how CRAs can meet key FCRA requirements could help CFPB promote consistency in its supervisory approach by providing examiners with information on the agency’s interpretation of FCRA provisions. FTC’s enforcement actions since 2010 have targeted smaller CRAs. FTC staff told us that because CFPB has supervisory authority over the larger CRAs, FTC has focused its FCRA enforcement efforts on other CRAs. Additionally, our review of FTC’s enforcement actions showed that FTC generally took enforcement actions against specialty CRAs that are smaller than the nationwide CRAs, such as CRAs conducting background screening. However, FTC staff also told us that they do not use a specific size threshold to initiate investigations against CRAs or furnishers and that they conduct their enforcement on a case-by-case basis, focusing on violations or potential violations of applicable laws. Prior to taking an enforcement action against a company, FTC conducts an investigation to determine if the company has violated a law. Using its investigative authority, FTC can compel companies to produce documents, testimony, and other materials to assist in its investigations. To determine whether to initiate investigations, FTC staff said they consider several sources, including leads from consumer advocacy groups and other FTC investigations, tips from whistleblowers, and monitoring of media reports. FTC staff also said that FTC regularly monitors its consumer complaint database to identify the types of complaints that consumers file and to determine if the activity described in the complaint indicates potential noncompliance with laws and regulations. FTC also can start investigations based on requests, such as by a member of Congress. FTC staff said that the agency targets its investigations based on the extent to which the potential noncompliance may affect a large number of consumers. For example, staff said FTC targets companies for investigation where inaccuracies may be occurring on a large scale. In addition, as we reported in February 2019, FTC staff said that when determining whether to initiate an investigation related to privacy and data security matters, they consider factors such as the companies’ size and the sensitivity of the data in the companies’ networks. FTC staff said that the consumer reporting market is a high priority for FTC, and that the accuracy of consumer reports and CRA activities has been a large part of FTC’s enforcement priorities. FTC staff said that they initiated about 160 FCRA investigations from 2008 through 2018. FTC staff stated that of the approximately 160 investigations, about 70 related to CRAs or companies, such as data brokers and companies selling public records, that FTC investigated to determine if they were engaged in conduct that would render them CRAs. Additionally, the staff said that about 20 of the approximately 160 investigations related to furnishers, about 55 related to users of consumer reports, and about 15 related to companies that fall under provisions of FCRA that do not require that the entity be a CRA, furnisher, or user. FTC staff stated that among these investigations, FTC investigated specialty CRAs, such as background- screening and check-authorization companies, and furnishers, such as debt collectors, lenders, and telecommunications companies. After an investigation, FTC may initiate an enforcement action if it has reason to believe that a law is being or has been violated. From 2010 through 2018, FTC took 30 enforcement actions related to FCRA, including against 14 CRAs, six furnishers, and two companies that acted as both a CRA and furnisher. Of the 30 enforcement actions, 14 contained issues related to data accuracy or disputes and two contained issues related to data security. In total, 20 of the 30 enforcement actions contained issues related to other consumer reporting topics, such as provision of consumer reports without permissible purpose. FTC staff told us that all of the enforcement actions related to FCRA involved injunctive relief. Additionally, some enforcement actions involved civil penalties. For example, in one action, a CRA was ordered to pay civil penalties for failing to use reasonable procedures to ensure the maximum possible accuracy of information it provided to its customers, and for failing to reinvestigate consumer disputes, as required by FCRA. FTC alleged that the CRA failed to take reasonable steps to ensure that the information in the reports was current and reflected updates, such as the expungement of criminal records. FTC staff said that there is no overarching definition regarding the FCRA provision for reasonable procedures for assuring maximum possible accuracy and that FTC determines on a case-by-case basis whether a violation has occurred. FTC staff also said that FTC’s enforcement actions provide industry with information on unacceptable practices and that the enforcement actions are closely monitored by the consumer reporting industry. In addition to enforcement actions related to FCRA, FTC staff told us that FTC took five actions against CRAs for unfair or deceptive acts or practices related to data security in the past 10 years. FTC alleged that all five CRAs failed to employ reasonable and appropriate security measures to protect sensitive consumer information. As discussed previously, the prudential regulators have supervisory and enforcement authority for FCRA over depository institutions and credit unions with total assets of $10 billion or less, some of which act as furnishers. The four prudential regulators told us they do not perform standalone examinations of these financial institutions for FCRA compliance. Rather, they examine for FCRA compliance in conjunction with other consumer financial laws and regulations and as part of examining an institution’s compliance with federal consumer protection laws and regulations. For example, OCC staff told us that if an examiner reviews an institution’s general compliance management system and identifies compliance, procedural, or other weaknesses related to FCRA, then the examiner would look at those issues more closely. Staff from the four prudential regulators told us they take a risk-based approach to determine the scope of examinations. They said that the approach includes consideration of factors such as an institution’s asset size, record of FCRA compliance, and trends in consumer complaints. As part of their compliance examinations from 2013 through 2018, staff from FDIC, the Federal Reserve, and NCUA said their agencies identified multiple FCRA- and Regulation V-related findings, including findings not related to financial institutions’ furnishing activities. FDIC staff said that examiners identified more than 1,200 violations related to FCRA and Regulation V at around 800 institutions, but found that the majority of the violations posed a low level of concern to the institution and consumers. Of these violations, FDIC staff stated that 106 related to furnisher obligations under Regulation V regarding the accuracy and integrity of information furnished to CRAs and that those types of violations were among the five most frequently cited violation topics related to FCRA and Regulation V. Federal Reserve staff said that in examinations that reviewed compliance with FCRA and Regulation V, Federal Reserve examiners cited FCRA and Regulation V about 210 times for an aggregate of about 4,200 related violations. Of these, Federal Reserve staff said the agency cited FCRA and Regulation V provisions related to furnisher accuracy about 20 times and cited an aggregate of about 3,600 violations. NCUA staff stated that NCUA identified 160 FCRA violations at around 150 credit unions. NCUA staff explained that 20 of the violations related to furnisher accuracy and that these types of violations were not among the five most frequently cited violation topics related to FCRA. OCC staff told us that OCC identified no findings related to FCRA or Regulation V from 2013 to 2018. Three prudential regulators stated that they consider the risk that a FCRA or Regulation V violation poses to the depository institution, including risk to consumers. For example, FDIC staff stated that the violations they cited may have had a small but negative effect on consumers, or may have the potential to have a negative effect in the future if uncorrected. FDIC staff added that such violations may also pose compliance and legal risks to the institution. NCUA staff stated that they require corrective action for any FCRA violation, and that they consider the pervasiveness of violations—particularly a risk of systemic or repeated violations—in determining the appropriate supervisory action. CFPB, FTC, and industry stakeholders attributed inaccuracies in consumer reports to several causes, including (1) CRAs matching data to the wrong consumer files due to missing, inaccurate, or inconsistent personally identifiable information; (2) errors in furnished data; (3) timing of data updates; and (4) identity fraud or theft. In particular, CFPB, FTC, and industry stakeholders most frequently cited CRAs mismatching data and errors in furnished data as the primary causes of consumer report inaccuracies. Several industry stakeholders identified CRAs’ mismatching of furnished data or public records to consumer files as a major source of inaccuracies in consumer reports. Two of the consumer groups we interviewed— Consumers Union and the National Consumer Law Center—also cited mismatching of data to consumer files as a source of inaccuracies in reports they published. In addition, FTC and CFPB reported in separate studies in 2012 that mismatching is a key source of inaccuracies in consumer reports. When CRAs do not correctly match data to the appropriate consumer files, the consumer’s file may contain data pertaining to another consumer. Alternatively, data can be excluded from the “correct” consumer’s file. For example, if one consumer’s report contains information about a different consumer’s debt payment history or collections activity, this information would also be missing from the file of the consumer who generated that activity. CFPB reported in its 2012 study that inconsistent, inaccurate, or incomplete personally identifiable information can cause errors in matching furnished data to the correct consumer’s file. CFPB, FTC, and industry stakeholders—three CRAs, a CRA industry group, and a consumer group—identified multiple reasons why personally identifiable information in data furnished to CRAs may be inconsistent, inaccurate, or incomplete, including the following examples: Consumers may use variations of their names when establishing an account with financial institutions (such as Kathy and Katherine). Consumers may change their names as a result of divorce or marriage, but the name change may not be reflected in furnished data. Consumers with suffixes in their names (such as junior or senior) may not consistently use suffixes in their applications. Furnishers may omit personally identifiable information. Furnishers may input consumers’ information incorrectly during data entry. In addition, CFPB stated in its 2012 report that matching public records to consumers’ files can be particularly challenging for CRAs because public records rarely contain Social Security numbers. The processes CRAs have in place to match data to consumers’ files may also contribute to inaccuracies in consumer reports. Generally, CRAs use various combinations of personally identifiable information to match data to consumers. For example, representatives from one CRA said the CRA uses at least the name and address to conduct matches. These representatives said that where only name and address are used, the address is required to be an exact match while the name can be a logical variation determined by the CRA’s algorithm. Representatives from another CRA said that the CRA matches public record information using at least the full name and date of birth but not the Social Security number because it is difficult to obtain. According to a CFPB report, the three nationwide CRAs—as part of their settlements with multiple state Attorney General offices—now require name, address, and Social Security number or date of birth to be present in public records furnished to them and use that personally identifiable information to conduct matches. Representatives from three consumer groups attributed consumer report inaccuracies to how CRAs make such matches. For example, representatives of two consumer groups said that CRAs could reduce inaccuracies arising from mismatching by using stricter requirements, such as requiring both Social Security number and date of birth, in addition to names and addresses, or only matching data to consumers if all nine digits of the Social Security number are present. Altogether, the errors originating from consumers or furnishers, as well as processes that CRAs have in place for matching, affect the accuracy of consumer reports (see fig. 3). CFPB and representatives from several industry stakeholders identified errors in furnished data as a primary cause of consumer report inaccuracies. Even when a CRA matches data to the correct consumer file, the consumer report can still contain inaccuracies if the information a furnisher provided to the CRA regarding the consumer contained errors (see fig. 4). CFPB has reported and a few CRAs told us that CRAs conduct quality checks to identify issues including blank fields or logical inconsistencies in furnished data, such as reporting of new account balance for closed consumer accounts. The CRA can reject furnished data or ask furnishers to provide corrected data. However, a CFPB report and a few industry stakeholders we interviewed identified weaknesses in furnisher and CRA processes as contributing to errors in furnished data. Two of the consumer groups we interviewed—Consumers Union and the National Consumer Law Center—also cited weaknesses in furnisher and CRA processes as contributing to errors in furnished data in reports they published. Processes for handling consumer transactions. CFPB reported that problems with processes used by furnishers include failing to update records, failing to post a payment, and misattributing ownership of an account to an individual who is only an authorized user. Processes for handling data accuracy. CFPB also reported and a few stakeholders told us that some furnishers lack processes for ensuring the accuracy of data submitted to CRAs and some CRAs lack processes for ensuring the accuracy of furnished data. CFPB reported and representatives from a few industry stakeholders said that timing of data updates in furnished data and court records could be a source of potential inaccuracies. For example, representatives from one CRA said that an address or name change can take up to two billing cycles to be reflected in a consumer report. Additionally, representatives from a CRA industry group told us that online court records, where CRAs may obtain data, often lag behind paper court records. Representatives from one consumer group pointed to the timing of when furnishers report debt as a source of potential inaccuracies. CFPB, the National Consumer Law Center, and Consumers Union have reported that identity fraud and theft are causes of inaccuracies in consumer reports. Additionally, representatives from one CRA also told us that identity fraud and theft are primary causes of inaccuracies. For example, identity thieves can create new credit accounts in a consumer’s name and let the debt go unpaid. Such debts then may be reflected in the consumer’s account and be reported to CRAs if not identified by the furnisher as resulting from fraudulent activity. Consumers can dispute the accuracy or completeness of their consumer reports with the CRAs that produced the consumer reports, with the data furnishers, or both. As stated previously, FCRA requires CRAs to conduct reasonable investigations of consumer disputes; FCRA, Regulation V, and FTC’s Furnisher Rule, as applicable, generally also require furnishers to conduct reasonable investigations of consumer disputes. If consumers are dissatisfied with the results of the investigations conducted by the CRAs or furnishers, they have a few options, discussed in detail below. FCRA requires CRAs and furnishers to take specific steps to respond to consumer disputes. When a consumer files a dispute with the CRA, the CRA must investigate the dispute internally, and once the CRA notifies the furnisher of the dispute, the furnisher must also investigate the disputed information (see fig. 5). If the CRA’s internal investigation or the furnisher’s investigation finds that the disputed item is inaccurate, incomplete, or cannot be verified, the CRA must delete the disputed item from the consumer’s file or modify the information and notify the furnisher of the action taken. The CRA must notify the consumer of the investigation results. Representatives from six of the CRAs we interviewed said that they consider disputes resolved when they or the furnishers complete their investigations and notify consumers of the results, even if the consumer does not agree with the results. If a furnisher does not conduct an investigation and report to the CRA within the time frame required by FCRA, then the CRA must remove the disputed information from the consumer’s file. Certain furnisher processes for investigating a dispute received from a CRA and a dispute received directly from the consumer are similar under FCRA. When a furnisher investigates a dispute received from a CRA, the furnisher must report the results of the investigation to the CRA that forwarded the dispute. If the furnisher receives the dispute directly from a consumer, then it must investigate the dispute and report the results of the investigation to the consumer, generally within 30 days (see fig. 6). In both cases, the furnisher must provide corrected information to every CRA to which it provided the information. CRAs may have differing dispute investigation processes in place because of regulatory requirements or because of how they obtained their data. Under FCRA, the nationwide CRAs are required to maintain an automated system through which furnishers can report incomplete or inaccurate information in a consumer’s file. The nationwide CRAs share the use of an automated system that sends disputes to furnishers and receives furnishers’ responses to the disputes. Other CRAs are not required by FCRA to use an automated system. Representatives from one CRA told us that the CRA uses email and phone calls to send disputes to and receive responses from furnishers. Representatives from a CRA industry group, as well as representatives from a background- screening CRA, said that compared to CRAs that obtain information from furnishers, background-screening CRAs generally obtain records from courts and therefore conduct their dispute investigations by confirming court records and contacting court officials. Consumers have several options to address potential inaccuracies in their consumer reports if they disagree with the results of a CRA or furnisher investigation, but these options have potential limitations, according to the stakeholders we interviewed. Placing a consumer statement on the report. Under FCRA, if the investigation does not resolve the dispute (where the dispute is filed with a CRA), the consumer may place a statement regarding the nature of the dispute on the consumer report, such as why the consumer disagreed with the reported item. According to the three nationwide CRAs, such statements alert creditors to the consumer’s disagreement. However, the statement does not modify or remove the information that the consumer perceived to be inaccurate from the consumer report, and users of the consumer report may or may not consider the consumer’s statement in their decision-making. Resubmitting disputes to CRAs or furnishers. Consumers who believe their disputes have not been satisfactorily resolved may choose to resubmit disputes regarding the same items that they disputed previously to CRAs or to the furnishers. If a consumer submits a dispute and does not provide sufficient information to investigate the disputed information or resubmits a dispute and does not provide additional or new supporting information, a CRA or furnisher may determine that the dispute is frivolous or irrelevant and does not warrant an investigation. Representatives from one CRA told us that if the CRA receives a dispute from a consumer about an item that was previously disputed, it would review consumer records to see if it has verified the consumer’s information previously. If so, the CRA would ask the consumer to provide additional documentation or to contact the furnisher to obtain support for the dispute. In some cases, consumers may turn to third parties that submit disputes on their behalf. Representatives from one CRA said that the CRA does not investigate disputes that certain third parties submit on behalf of consumers because these third parties dispute the same items repeatedly. Representatives from another CRA said that the CRA reviews third-party dispute requests to determine if the third party has proper authorization from consumers to act on their behalf. Submitting complaints to federal and state agencies. Consumers can submit complaints about inaccuracies in their consumer reports to federal and state agencies, such as CFPB and state Attorney General offices. CFPB has stated that it forwards these complaints to CRAs and works with them to obtain responses within 15 days. Staff from several state agencies we interviewed generally told us that after receiving complaints, they contact CRAs about the complaints to obtain responses but do not compel CRAs to take specific actions. CFPB has reported that CRAs handle complaints similarly to consumer disputes. As a result, although complaints are separate from the dispute process required under FCRA, the effectiveness of this option also depends on the same CRA processes for addressing inaccuracies. However, representatives from two consumer groups said that submitting complaints to CFPB through its consumer complaint database has helped consumers resolve inaccuracies in their reports. Representatives from one consumer group said the publication of complaints in CFPB’s database helps to hold CRAs accountable and incentivizes CRAs to respond. Taking private legal action. Under FCRA, consumers have private rights of action—or ability to litigate—against CRAs and furnishers, under certain provisions. Consumers have brought legal claims against CRAs and furnishers for failure to follow reasonable procedures to assure maximum possible accuracy or conduct a reasonable investigation of a dispute. Under FCRA, consumers can sue a furnisher for failure to conduct a proper investigation when notified by a CRA that a consumer has disputed information provided by the furnisher. However, before initiating suit, the consumer must first dispute the information with the CRA. A consumer may initiate a dispute through a CRA even if the consumer has previously initiated a dispute with the furnisher. Representatives from two consumer groups and one state agency told us that in general, consumer barriers to litigation include that it is time-consuming and has potentially high legal costs and that consumers might be unaware of their legal rights. As a result of CFPB and FTC oversight and settlements with multiple state Attorneys General, the nationwide CRAs and several other CRAs have made changes in their policies and procedures to improve data accuracy and processes for addressing inaccuracies in consumer reports. However, CFPB and a few industry stakeholders said that challenges to improving accuracy in consumer reports remain. According to CFPB and nationwide CRAs, examples of the changes that CRAs have made as a result of oversight include the following: Changes as a result of CFPB supervision. According to CFPB, as a result of supervisory findings, one or more CRAs have implemented or changed policies and procedures related to ensuring accuracy and dispute investigations. These include (1) establishing a data- governance structure to oversee furnisher monitoring, such as by developing policies and procedures for ongoing and systemic screening of furnishers; (2) implementing systems to forward relevant dispute documents submitted by consumers to furnishers; and (3) implementing policies and procedures to ensure consideration of all supporting material submitted by consumers. Changes as a result of CFPB and FTC enforcement. As a result of CFPB’s and FTC’s enforcement, the two agencies directed a few CRAs to revise the procedures they use to match data using personally identifiable information. For example, CFPB directed two background-screening CRAs to revise procedures for assuring accuracy, such as by using algorithms to distinguish records by middle name and to match common names and nicknames. In another example, FTC directed a background-screening CRA that required an exact match of a consumer’s last name and a nonexact match of first name, middle name, and date of birth to put in place reasonable procedures to ensure maximum possible accuracy. Changes as a result of state oversight. According to the three nationwide CRAs, they have implemented measures as a result of their 2015 settlements with multiple state Attorneys General. For example, they stated they monitor data furnishers’ dispute responses and take corrective actions against data furnishers for noncompliance with their dispute investigation responsibilities. Additionally, they established special handling procedures for disputes involving mixed files, fraud, and identity theft and provided CRA employees with discretion to resolve such disputes, rather than relying on furnishers’ responses. In addition to the changes described above, representatives at various CRAs said they had quality assurance processes in place to help ensure that furnished data are accurate and that furnishers are responsive to disputes. Monitoring of furnished data. Representatives from four CRAs said that they use various mechanisms to monitor furnished data to detect potential inaccuracies and take corrective actions against furnishers that do not comply with data furnishing standards. For example, representatives from three CRAs told us they compare data submissions against industry patterns and historical trends—such as data submission history over the past 6 months—to identify anomalies that would suggest erroneous data and take actions such as rejecting incoming data and returning data for correction. Representatives from one of these CRAs said that they analyze why a furnisher deviates from industry trends and help the furnisher identify and implement changes. Representatives from four CRAs told us that they provide regular reports, such as monthly reports, on data quality to furnishers. We reported previously that such steps may improve the quality of the information received from furnishers but cannot ensure the accuracy of such data. Monitoring of dispute investigations. Representatives from four CRAs said they have processes in place to help ensure that furnishers are responsive to disputes. For example, representatives from one CRA said that the automated system they use to correspond with furnishers about disputes automatically identifies illogical furnisher responses; the CRA contacts the furnisher to confirm the accuracy of those responses. Representatives from four CRAs told us that they monitor furnisher responses to disputes, such as dispute trends by furnisher type and the rate at which furnishers do not respond to disputes. Although CRAs have made changes to improve processes for ensuring accuracy and addressing inaccuracies, CFPB and industry stakeholders said that challenges remain in these areas. First, CFPB staff told us that the consumer reporting market has historically had comparatively less regulatory intervention than other regulated markets. As a result, the staff said that it has been challenging to change CRAs’ approach to a proactive one, whereby the CRAs proactively address compliance and change practices, as opposed to a defensive, reactive approach in response to consumer disputes and lawsuits. CFPB staff explained that this has been a focus of CFPB’s supervision and said that its examination findings have demonstrated that CRAs can take actions to improve accuracy. Further, representatives from three consumer groups said that consumer report inaccuracy remains a challenge because CRAs lack incentives to be responsive to consumers, in part because the CRAs’ customers are the users of consumer reports, such as banks and employers, rather than the consumers themselves. Additionally, two industry stakeholders identified gaps in furnisher responsibilities for ensuring accuracy as a challenge. Representatives from one of these stakeholders, a state agency, said that furnishers often do not know their responsibilities for ensuring the accuracy of their data. Representatives from the other stakeholder, a CRA, said that while the CRA has implemented policies and procedures to ensure accuracy in response to CFPB’s supervision, furnishers might not have implemented similar policies and procedures to ensure the accuracy of the data provided. Consumer reports affect the lives of millions of Americans because of the role they play in many important decisions, such as whether a lender decides to extend credit and at what terms or whether an employer offers a candidate a job. Therefore, it is important for CRAs to produce reports that are accurate and for consumers to have appropriate procedures available to correct any inaccuracies in their consumer reports, including disputing inaccuracies. We found that opportunities exist for CFPB to improve its oversight of CRAs. As part of its supervision, CFPB has directed CRAs it has examined to make specific changes based on examination findings related to FCRA requirements for (1) reasonable procedures for assuring accuracy and (2) reasonable investigation of consumer disputes. However, CFPB has not defined its expectations for how CRAs can comply with these requirements. Providing additional information to CRAs about its expectations for key FCRA requirements could help CFPB achieve its vision of promoting a consumer reporting system where CRAs maintain and distribute accurate data, supplemented by effective dispute resolution processes. Additionally, such information could help to promote consistency and transparency in CFPB’s supervisory approach. We are making two recommendations to CFPB: The Director of CFPB should communicate to CRAs its expectations regarding reasonable procedures for assuring maximum possible accuracy of consumer report information. (Recommendation 1) The Director of CFPB should communicate to CRAs its expectations regarding reasonable investigations of consumer disputes. (Recommendation 2) We provided a draft of this report to CFPB, the Federal Reserve, FDIC, FTC, NCUA, and OCC for review and comment. We received written comments from CFPB, which are summarized below and reprinted in appendix II. CFPB, the Federal Reserve, FDIC, and FTC provided technical comments, which we incorporated as appropriate. In email responses, officials indicated that NCUA and OCC did not have any comments on the draft of this report. In its written comments, CFPB neither agreed nor disagreed with the recommendations. CFPB stated that it has made oversight of the consumer reporting market a top priority and that its supervisory reviews of CRAs have focused on evaluating their systems for assuring the accuracy of data used to prepare consumer reports. CFPB noted that CRAs have made significant advances to, among other things, promote greater accuracy. With respect to the first recommendation—that CFPB should communicate to CRAs its expectations regarding reasonable procedures for assuring maximum possible accuracy—CFPB noted that case law includes interpretations of the reasonableness standard and provides guidance to CRAs about how the standard applies to various factual scenarios. CFPB also noted that it and FTC have settled enforcement actions regarding the reasonableness standard in which each agency provided examples of how it applied the standard and the relevant case law to the facts of each matter and described a consent order with two background-screening companies that made clear that a lack of certain written procedures was not reasonable. Additionally, CFPB noted that its examination procedures discuss factors that would be considered in evaluating compliance with the reasonable procedures standard and that it publishes “Supervisory Highlights” that document key examination findings. While we agree that case law may provide information to CRAs regarding how courts have interpreted the reasonableness standard in specific circumstances, as we note in the report, there may be a lack of clarity about the extent to which all case law fully reflects CFPB’s expectations. Absent additional information from CFPB, the current case law and case- by-case enforcement actions may not best serve to enable CRAs to proactively address compliance practices. More direct communication of CFPB’s expectations can provide CRAs with clearer information on what they should be doing and what actions might constitute a FCRA violation. Similarly, while FTC and CFPB have settled actions with certain CRAs regarding reasonable procedures, such settlements may be applicable only to the specific facts and circumstances and the parties involved in those cases. CFPB’s examination procedures provide information on factors that would be considered in evaluating compliance and areas that may be reviewed in examinations, but they do not provide information on CFPB’s oversight expectations regarding how CRAs may comply with the FCRA requirement for reasonable procedures. Likewise, while CFPB’s Supervisory Highlights provide information on key examination findings, the Supervisory Highlights do not represent CFPB’s expectations for how CRAs may or should comply with the reasonableness standard. For example, the Supervisory Highlights state that the legal violations described are based on particular facts and circumstances and may not lead to such findings under different facts and circumstances. With respect to the second recommendation—that CFPB should communicate to CRAs its expectations regarding reasonable investigations of consumer disputes—CFPB stated that what qualifies as a “reasonable investigation” has been articulated in court cases and noted that an FTC report summarizes how the reasonable investigations standard has been interpreted by courts and FTC. While we acknowledge that FTC may have interpreted and the courts may have ruled on this issue, CFPB has not communicated to CRAs specific information on what may and may not qualify as a “reasonable investigation.” CFPB also stated that it issued a bulletin in September 2013 that is relevant to this recommendation. However, in that bulletin, CFPB restated FCRA requirements and emphasized their importance, but it did not provide further information on what practices may represent a “reasonable investigation” or what it expects of CRAs. CFPB noted that it has and will continue to communicate its expectations to CRAs. As stated in our report, communicating information about CFPB’s compliance expectations and ways in which CRAs could comply could help to ensure that CRAs receive complete and clear information about how to comply with key FCRA requirements. CFPB could provide such information in several ways; for example, CFPB has put consumer reporting issues on its rulemaking agenda since 2015. We maintain that providing additional information to CRAs about its expectations for key FCRA requirements could help CFPB to promote consistency and transparency in its supervisory approach and that the recommendations should be addressed. We are sending copies of this report to the appropriate congressional committees and financial regulators, and other interested parties. This report will also be available at no charge on our website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or OrtizA@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Our objectives for this review were to (1) describe the current oversight framework for consumer reporting agencies (CRA), (2) examine how the Consumer Financial Protection Bureau (CFPB) has overseen CRAs and entities that furnish consumer data, (3) examine how other federal agencies, including the Federal Trade Commission (FTC) and the prudential regulators, have overseen CRAs and entities that furnish consumer data, and (4) identify what is known about the causes of inaccuracies in consumer reports and the processes that are in place to help ensure accuracy. Some information has not been included in this public report because CFPB determined it was information prohibited by law from public disclosure. This report omits such information, but we will be issuing a nonpublic version of this report that includes all the information. Although the information provided in this report is more limited, it addresses the same objectives as the sensitive nonpublic report and uses the same methodology. To describe the oversight framework for CRAs, we identified and reviewed relevant federal laws and their application for CRAs and institutions that furnish data to CRAs (called furnishers). We identified and reviewed laws focused on the accuracy of consumer reports, the security of consumer information, and the use and sharing of consumer reports. These laws include the Fair Credit Reporting Act (FCRA) and its implementing regulation, Regulation V, the Gramm-Leach-Bliley Act, the Dodd–Frank Wall Street Reform and Consumer Protection Act, the Federal Trade Commission Act, and the Economic Growth, Regulatory Relief, and Consumer Protection Act. We interviewed staff from CFPB, FTC, and the prudential regulators—the Board of Governors of the Federal Reserve System, the Federal Deposit Insurance Corporation, the National Credit Union Administration, and the Office of the Comptroller of the Currency—about applicable laws and regulations for CRAs and furnishers and their oversight authority over CRAs and furnishers. Additionally, we interviewed five categories of stakeholders to learn about federal and state oversight over CRAs: state agencies such as Attorney General offices and regulators, CRAs, groups representing state agencies, industry groups representing CRAs, and consumer groups. We selected four states—Maine, Maryland, New York, and Ohio—for a more in-depth review. We chose these states because they had laws and regulations related to consumer reporting or had oversight activities involving CRAs, such as prior enforcement actions. We interviewed staff from state regulatory agencies in Maine, Maryland, and New York, as well as staff from the New York Office of the Attorney General. In addition, we received written responses to our questions from the Ohio Office of the Attorney General. In each case, we asked questions about state oversight of CRAs, including the relevant state laws and state enforcement, rulemaking, and supervisory authorities. We interviewed three nationwide CRAs and four smaller or specialty CRAs that produce or compile consumer reports covering the credit and background-screening markets about federal and state oversight, including applicable laws. We selected these CRAs because of potential differences in oversight based on their size and market. In our selection, we considered the size of the CRA and the number of consumer complaints in CFPB’s database. We also interviewed two industry groups representing CRAs (the Consumer Data Industry Association and the National Association of Professional Background Screeners); two groups representing states (the Conference of State Bank Supervisors and the National Conference of State Legislatures); and four consumer groups (Consumers Union, the National Association of Consumer Advocates, the National Consumer Law Center, and U.S. Public Interest Research Group). We asked these groups about federal and state authorities for overseeing CRAs. We selected these groups because, based on our analysis of publicly available information and interviews with federal agencies, they are the primary organizations representing stakeholders in our review, such as CRAs, or have existing work, such as reports or testimonies, related to CRAs. The groups we included and the views they represent reflect a range of stakeholders but do not necessarily reflect the full scope of the industry. To examine how CFPB has overseen CRAs and furnishers, we interviewed CFPB staff about CFPB’s supervision and enforcement strategies and activities, and we reviewed relevant documents, including supervisory and examination documents. To examine CFPB’s supervisory strategies and activities, we reviewed CFPB’s supervisory plans that document how CFPB determined which CRAs and furnishers to examine and which compliance areas to examine. We also reviewed CFPB’s public reports, such as Supervisory Highlights, and nonpublic examination documents to evaluate CFPB’s supervisory activities for both CRAs and furnishers. To learn about CFPB’s enforcement strategies and enforcement activities in the consumer reporting market, we reviewed the types of enforcement actions available to CFPB for violations of relevant laws, and we identified specific enforcement actions CFPB brought against CRAs and furnishers for violations related to FCRA and Regulation V from 2012 through 2018. We identified these enforcement actions by reviewing CFPB’s publicly available enforcement activities on its website, and we corroborated our results with CFPB. We also interviewed stakeholders, including CRAs, consumer groups, state agencies, and state groups, to obtain their views on CFPB’s oversight. To examine how FTC and the prudential regulators have overseen CRAs and furnishers, we interviewed staff from FTC and the prudential regulators to discuss the agencies’ oversight and enforcement activities. To learn about FTC’s enforcement strategies and activities in the consumer reporting market, we reviewed the types of enforcement actions available to FTC for violations of relevant laws, interviewed FTC staff regarding the process for initiating investigations and the investigations FTC conducted, and identified specific enforcement actions brought against CRAs and furnishers for violations related to FCRA, Regulation V, and FTC’s Furnisher Rule from 2010 through 2018. We identified these enforcement actions by reviewing FTC’s publicly available enforcement activities on its website, and we corroborated our results with FTC. To learn about prudential regulators’ activities, we reviewed the prudential regulators’ policies and procedures for examining furnishers and interviewed regulators’ staff. We also collected information from the regulators about their FCRA-related findings for furnishers from 2013 through 2018. To identify what is known about the causes of inaccuracies in consumer reports and the processes that are currently in place to help ensure accuracy, we conducted interviews with stakeholders. In particular, we interviewed staff from CFPB, FTC, the prudential regulators, and the state agencies to learn about what they believe are the causes of inaccuracies in consumer reports and the options available to consumers to address inaccuracies. Similarly, we interviewed staff at three nationwide CRAs and four smaller or specialty CRAs about the causes of inaccuracies and the processes they have in place for ensuring accuracy and addressing inaccuracies, including the processes in place to meet FCRA requirements for addressing consumer disputes about consumer report information. Additionally, we spoke with staff from four consumer and two industry groups (described above) to gain their perspectives on the causes of inaccuracies and processes in place to address them. We also conducted a literature search on the causes of inaccuracies in consumer reports and processes in place to help ensure accuracy. The search covered academic literature and court cases from 2008 through 2018 and used subject and keyword searches of various databases, such as ProQuest, Westlaw, and CQ. The literature search resulted in limited relevant information. However, we identified reports from CFPB and FTC that included information on the causes of inaccuracies in consumer reports, as well as information CFPB has published, such as Supervisory Highlights, on the processes CRAs have in place to help ensure accuracy. Additionally, through our interviews, we identified information that stakeholders, such as the National Consumer Law Center, have published on these issues. We conducted this performance audit from July 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Kevin Averyt (Assistant Director), Weifei Zheng (Analyst in Charge), Yue Pui Chin, Sergio Enriquez, Marc Molino, Stephen Ruszczyk, Kelsey Sagawa, Jessica Sandler, Jennifer Schwartz, and Farrah Stone made key contributions to this report.", "summary": "CRAs collect data from various sources, such as banks and credit card companies, to create consumer reports that they sell to third parties. The three largest CRAs hold information on more than 200 million Americans. The Economic Growth, Regulatory Relief, and Consumer Protection Act, enacted in 2018, included a provision for GAO to examine issues related to the consumer reporting market. This report examines, among other objectives, the causes of consumer report inaccuracies and how CFPB has overseen CRAs. To answer these questions, GAO reviewed relevant laws, regulations, and agency documents related to CRA oversight. GAO interviewed representatives of federal agencies and stakeholders, including a nongeneralizable selection of state agencies from four states that had laws or oversight activities involving CRAs and seven CRAs selected based on size and the type of consumer reports produced. GAO also interviewed groups representing state agencies, consumers, and CRAs selected to reflect a range of stakeholders or based on their work related to CRAs. Businesses and other entities use consumer reports to make decisions about consumers, such as whether they are eligible for credit, employment, or insurance. Consumer report inaccuracies can negatively affect such decisions. The Consumer Financial Protection Bureau (CFPB) and other stakeholders identified various causes of consumer report inaccuracies, such as errors in the data collected by consumer reporting agencies (CRA) and CRAs not matching data to the correct consumer. In 2010, CFPB was granted supervisory and enforcement authority over CRAs. In using its oversight authorities, CFPB has prioritized CRAs that pose the greatest potential risks to consumers—such as those with significant market shares and large volumes of consumer complaints—for examination. CFPB's oversight has generally focused on assessing compliance with Fair Credit Reporting Act (FCRA) requirements regarding accuracy and the investigations CRAs conduct in response to consumer disputes. For example, since 2013, CFPB has conducted examinations of several CRAs and directed specific changes in CRAs' policies and procedures for ensuring data accuracy and conducting dispute investigations. CFPB has not defined its expectations for how CRAs can comply with key statutory requirements. FCRA requires CRAs (1) to follow reasonable procedures for ensuring maximum possible accuracy and (2) to conduct reasonable investigations of consumer disputes. CFPB has identified deficiencies related to these requirements in its CRA examinations, but it has not defined its expectations—such as by communicating information on appropriate practices—for how CRAs can comply with these requirements. Absent such information, staff from four CRAs GAO interviewed said that they look to other sources, such as court cases or industry presentations, to understand what CFPB will consider to be noncompliant during examinations. A 2018 policy statement issued by CFPB and other regulators highlighted the important role of supervisory expectations in helping to ensure consistency in supervision by providing transparent insight to industry and to supervisory staff. By providing information to CRAs about its expectations for complying with key FCRA requirements, CFPB could help achieve its goal of accurate consumer reporting and effective dispute resolution processes. Such information also could help to promote consistency and transparency in CFPB's supervisory approach. CFPB should communicate to CRAs its expectations regarding (1) reasonable procedures for assuring maximum possible accuracy and (2) reasonable investigations of consumer disputes. CFPB described actions it has taken to provide information to CRAs. GAO maintains that communicating expectations in these two areas is beneficial, as discussed in the report.", "document_type": "gao"}
{"report": "As part of our audit of the fiscal years 2018 and 2017 CFS, we considered the federal government’s financial reporting procedures and related internal control. Also, we determined the status of corrective actions Treasury and OMB have taken to address open recommendations relating to their processes to prepare the CFS, detailed in our previous reports that remained open as of the completion of our fiscal year 2017 audit. A full discussion of our scope and methodology is included in our March 2019 report on our audit of the fiscal years 2018 and 2017 CFS. We have communicated each of the control deficiencies discussed in this report to your staff. We performed our audit in accordance with U.S. generally accepted government auditing standards. We believe that our audit provides a reasonable basis for our findings and recommendations in this report. During our audit of the fiscal year 2018 CFS, we identified three new internal control deficiencies in Treasury’s processes used to prepare the CFS. Specifically, we found that (1) Treasury did not have sufficient procedures to analyze and determine whether appropriate disclosures related to new federal accounting standards were included in the draft fiscal year 2018 Financial Report; (2) Treasury did not have sufficient procedures to properly support and consistently report restatements, reclassifications, and adjustments to beginning net position reported in the draft fiscal year 2018 Financial Report; and (3) Treasury and OMB did not have adequate processes and procedures for reporting appropriate information regarding legal contingency losses in the fiscal year 2018 CFS. During our fiscal year 2018 CFS audit, we found that Treasury did not have sufficient procedures to analyze and determine whether appropriate disclosures related to new federal accounting standards were included in the draft fiscal year 2018 Financial Report. Treasury uses working groups, disclosure checklists, and other tools to assist in determining appropriate reporting in accordance with new standards. Treasury’s procedures include working with federal entities during the fiscal year to determine the reporting required based on new standards. Treasury presents the new standards for discussion at monthly Central Reporting Team meetings that include financial reporting representatives from federal entities. Treasury also establishes working groups for certain new standards, such as the working group established for the Federal Accounting Standards Advisory Board’s (FASAB) Statement of Federal Financial Accounting Standards (SFFAS) 47, Reporting Entity, to work with federal entities on their implementation. Treasury also utilizes a financial reporting disclosure checklist to help determine that all disclosures required by FASAB standards are included in the CFS. Treasury updates this disclosure checklist throughout the fiscal year and completes the checklist as a key focus of the CFS compilation process. Although Treasury has processes in place for the implementation of new standards, certain disclosures required by new standards were not included in the draft fiscal year 2018 Financial Report. For example, the draft fiscal year 2018 Financial Report did not include disclosures related to federal government land assets, such as the number of acres held at the end of each reporting period, explanations of federal entities’ election to include or exclude land and land rights from their opening balances, and a reference to the component reporting entity’s financial report, as required by SFFAS 50, Establishing Opening Balances for General Property, Plant, and Equipment. In addition, the draft fiscal year 2018 Financial Report did not include disclosures related to significant component entity amounts included in certain CFS line items that were determined in accordance with Financial Accounting Standards Board (FASB) standards rather than FASAB standards, in accordance with SFFAS 47. SFFAS 34, The Hierarchy of Generally Accepted Accounting Principles, Including the Application of Standards Issued by the Financial Accounting Standards Board, provides that general purpose federal financial reports prepared in conformity with accounting standards issued by FASB also may be regarded as in conformity with U.S. generally accepted auditing principles (U.S. GAAP). SFFAS 47 permits the consolidation of amounts determined in accordance with FASAB and FASB standards into a single line item without conversion for differences in accounting policies and also provides application guidance that emphasizes the need for disclosures of the different accounting policies and the related amounts to aid financial statement users’ understanding of the information provided. Treasury’s disclosure checklist was not updated in sufficient detail for Treasury accountants to identify appropriate disclosures for inclusion in the draft fiscal year 2018 Financial Report in accordance with these new federal accounting standards. The updated disclosure checklist used for fiscal year 2018 did not include (1) specific details about disclosures required by SFFAS 50 for land assets, such as the number of acres added or disposed of during the reporting period, and (2) questions to help determine the need for disclosures to communicate the effect on certain CFS line items that include material amounts determined using accounting policies in accordance with FASB standards rather than FASAB standards, as SFFAS 47 allows. Also, Treasury did not calculate in aggregate the amounts that are reported in the CFS on a FASB basis by line item in order to determine line items where the disclosures were needed. We communicated these matters to Treasury officials, who conducted a comprehensive analysis and included disclosures in the final fiscal year 2018 Financial Report, as appropriate. In addition, although Treasury established a working group to help implement SFFAS 47, Treasury’s procedures did not provide for sufficient consultation with technical experts in interpreting new standards and updating the disclosure checklist to reasonably assure that all requirements related to the new standard were incorporated during implementation. Actively consulting with technical experts, such as members of FASAB, the body designated as the source of U.S. GAAP for federal reporting entities, would help minimize the risk of misinterpreting the standards or presenting and disclosing information in the Financial Report that is incorrect, inconsistent, or incomplete. Standards for Internal Control in the Federal Government states that one of the key objectives of an organization’s internal control over financial reporting is to provide reasonable assurance as to the reliability of its financial reporting, including its financial statements and note disclosures. Accompanying notes are an integral part of financial statements and provide additional disclosures that are necessary to make the financial statements more informative. Without sufficient procedures for analyzing and determining the appropriate reporting of disclosures required by new federal accounting standards, Treasury cannot reasonably assure that disclosures included in the Financial Report are reliable and complete. We recommend that the Secretary of the Treasury should ensure that the Fiscal Assistant Secretary develops and implements procedures to enhance Treasury’s processes for reasonably assuring that the Financial Report includes disclosures required by new federal accounting standards, such as conducting an appropriate level of analysis to determine whether disclosures are needed, consulting with technical experts, and including additional details on these requirements in its financial reporting disclosure checklists. (Recommendation 1) During our fiscal year 2018 CFS audit, we found that Treasury did not have sufficient procedures to properly support and consistently report restatements, reclassifications, and adjustments to beginning net position in the draft fiscal year 2018 Financial Report. Treasury identifies changes to prior year amounts (either restatements or reclassifications) and adjustments to current year beginning net position based on its review of information that significant component entities submit to Treasury and applicable FASAB standards. Treasury also performs procedures to determine the consistency of such reporting with significant component entities’ audited financial statements. Treasury’s Subject Matter Analysis standard operating procedure includes steps for Treasury’s staff to obtain financial information from significant component entities and to prepare the draft Financial Report. According to these procedures, Treasury’s staff uses third quarter financial data that significant component entities report and other information to obtain a preliminary understanding of potential changes to prior year amounts and adjustments to beginning net position. Treasury’s staff compares these preliminary results to the entities’ year-end information and compares beginning net position amounts reported for the current year to ending net position amounts reported the prior year to identify changes to prior year amounts. The staff then prepares a table categorizing such changes as restatements, reclassifications, or adjustments to beginning net position. This table includes a separate line for each of the significant component entities but does not include separate lines for each line item or note disclosure affected. Treasury uses this table to prepare a summary analysis of its conclusions for reporting restatements, reclassifications, and adjustments to beginning net position and related note disclosures presented in the draft Financial Report. The subject area manager reviews the results of the procedures that Treasury’s staff performed and documented. Although Treasury’s staff followed these procedures for fiscal year 2018, we found that Treasury did not always maintain sufficient support for restatements, reclassifications, and adjustments to beginning net position included in the draft fiscal year 2018 Financial Report. For example, Treasury reported an adjustment to beginning net position but did not identify errors made in prior years to support such an adjustment. Treasury also reported that restatements affected the Statement of Changes in Cash Balance from Budget and Other Activities, but supporting documentation provided by Treasury for the draft fiscal year 2018 Financial Report did not clearly indicate how this statement was affected by restatements. Treasury included a summary of significant accounting policies in Note 1 to the CFS as required by U.S. GAAP, which contained information about restatements, reclassifications, and adjustments to beginning net position. However, Treasury did not disclose consistent information in related line item notes, such as those for loans receivable and loan guarantee liabilities, federal employee and veteran benefits payable, and funds from dedicated collections in the draft fiscal year 2018 Financial Report. Treasury included information in its summary analysis and supporting documentation that was not consistent with information that significant component entities reported. Although Treasury’s processes did not identify these inconsistencies, Treasury corrected them in response to our questions. We found that Treasury did not identify these inconsistencies, in part, because its subject matter review procedures did not include steps for coordinating with Treasury managers of other subject matter areas to reasonably assure consistency and appropriate support for conclusions. Also, although Treasury’s process involved preparing a summary of analyses performed, Treasury’s process did not include steps or other tools to reasonably assure that consistent information was communicated in all financial statement line items and note disclosures affected by restatements, reclassifications, and adjustments to net position. SFFAS 21, Reporting of Corrections of Errors and Changes in Accounting Principles, requires that reporting entities restate prior period financial statements for material errors discovered in the current period, if such statements are provided for comparative purposes and if the effect of an error would be material to the financial statements in either period. If not material, corrections should be made to the beginning balance of cumulative results of operations in the statement of changes in net position. A reclassification is the movement of a prior year amount in comparative financial statements in order to conform to the current year presentation. Standards for Internal Control in the Federal Government states that management should (1) design control activities to achieve objectives and respond to risks, such as procedures to help reasonably assure that financial information is completely and accurately reported, and (2) implement control activities, such as documenting responsibilities through policies and procedures. Management should periodically review policies, procedures, and related control activities for continued relevance and effectiveness in achieving objectives. Without sufficient procedures for reporting restatements, reclassifications, and adjustments to beginning net position, there is an increased risk of presenting information that is inaccurate or incomplete in the Financial Report. We are making the following two recommendations to Treasury: The Secretary of the Treasury should ensure that the Fiscal Assistant Secretary enhances existing procedures for Treasury management to perform additional reviews for restatements, reclassifications, and adjustments to beginning net position to reasonably assure that they are properly supported and accurately reported. (Recommendation 2) The Secretary of the Treasury should ensure that the Fiscal Assistant Secretary develops and implements steps to reasonably assure that restatements, reclassifications, and adjustments to beginning net position are consistently reported in the Financial Report, such as developing a tool that identifies all affected financial statement line items and note disclosures. (Recommendation 3) During our fiscal year 2018 CFS audit, we found that Treasury and OMB did not have adequate processes and procedures for reporting appropriate information regarding legal contingency losses in the fiscal year 2018 CFS. Significant component entities are responsible for properly accounting for and reporting legal contingency losses in their entity-level financial statements and submitting this information to Treasury for inclusion in the CFS. For each entity-level financial statement audit, U.S. generally accepted government auditing standards require that component entity auditors obtain written legal representations as part of the audit. The significant component entities provide interim and final legal representation letters, along with management schedules, to Treasury, the Department of Justice (DOJ), and GAO. According to DOJ’s established process, its legal counsel reviews individual cases included in these legal representation letters for which the potential loss exceeds $500 million individually or in the aggregate for similar cases. DOJ is responsible for preparing and submitting an interim and final government-wide legal representation letter to Treasury and GAO, containing its assessment of the potential litigation losses, including whether there are litigation, claims, or assessments that were not addressed in the significant component entities’ legal representation letters that DOJ believes should have been reported or which DOJ believes should have been reported differently. Treasury’s procedures call for it to determine whether the financial statement information that the significant component entities submitted and Treasury used to compile the fiscal year 2018 CFS is consistent with the significant component entities’ management schedules, legal representation letters, and the government-wide legal representation letter. For fiscal year 2018, Treasury identified various inconsistencies among the significant component entities’ financial statement information, management schedules, and legal representation letters as well as inconsistencies between the government-wide legal representation letter and the significant component entities’ management schedules and legal representation letters. There may be appropriate reasons for these differences. For example, although management often relies on advice of legal counsel on the likelihood of loss and estimate of the amount or range of potential loss, as reflected in the legal representation letter, management is ultimately responsible for determining whether the loss should be recognized as a liability or disclosed in the notes to the financial statements. As such, management may make a different determination as to the likelihood of loss or estimated loss amounts than those in the legal counsel’s assessment. Also, differences between the government-wide legal representation letter and the significant component entities’ legal representation letters can occur in situations in which DOJ has more current information on the likelihood of loss and estimated loss amounts. However, Treasury was not always able to timely determine whether there were appropriate reasons for the differences it identified or whether adjustments were needed to the legal contingency loss information reported in the fiscal year 2018 CFS. Based on our work, we found that certain of these differences required correction in the fiscal year 2018 CFS. For example, Treasury noted that one significant component entity included estimated loss amounts for reasonably possible cases in its management schedule and that such amounts were not reported in the financial information provided to Treasury for consolidation. Because Treasury was unable to timely resolve the issue, the fiscal year 2018 CFS was not appropriately adjusted to include these amounts. Further, DOJ did not provide us the final government-wide legal representation letter as of our audit completion date. Although Treasury has procedures for reviewing and analyzing the significant component entities’ legal contingency loss information and the government-wide legal representation letter, we found that Treasury lacked effective processes and procedures to reasonably assure that appropriate information regarding legal contingency losses was reported in the fiscal year 2018 CFS. Specifically, Treasury did not have sufficient processes and procedures to obtain the needed information in a manner that would facilitate the timely compilation of the legal contingency loss information for inclusion in the fiscal year 2018 CFS or for timely resolving issues identified during its review. For example, as part of Treasury’s procedures, it compares the estimated loss amounts for reasonably possible and probable cases included in the significant component entities’ management schedules with the financial statement information that the significant component entities report for inclusion in the fiscal year 2018 CFS. If discrepancies are greater than 10 percent, Treasury’s procedures call for significant component entities to provide an explanation. However, because of the extensive amount of time needed to perform this analysis, Treasury was not always able to timely follow up and resolve the differences with the significant component entities. SFFAS 5, Accounting for Liabilities of The Federal Government, as amended by SFFAS 12, Recognition of Contingent Liabilities Arising from Litigation, contains accounting and reporting standards for loss contingencies, including those arising from litigation, claims, and assessments. An entity should recognize a liability and a related charge to expense for an estimated loss from a loss contingency only when (1) a past event or exchange transaction has occurred, (2) a future outflow or other sacrifice of resources is probable, and (3) the future outflow or sacrifice of resources is measurable. A contingent liability should be disclosed if any of the conditions for liability recognition are not met and there is at least a reasonable possibility that a loss or an additional loss may have been incurred. Disclosure should include the nature of the contingency and an estimate of the possible liability, an estimate of the range of possible liability, or a statement that such an estimate cannot be made. OMB Bulletin 19-01 provides guidance to federal agencies on the preparation of legal letters and management schedules, and the Treasury Financial Manual provides federal agencies the legal letter reporting requirements. Also, Standards for Internal Control in the Federal Government provides that management should design control activities to achieve objectives and respond to risks. For example, management should design control activities so that financial information is completely and accurately reported. Until the federal government implements effective processes and procedures to obtain and assess information regarding legal contingency losses, the reliability of amounts and disclosures associated with legal loss contingencies reported in the CFS and the ability to assess their potential effect on the financial condition of the federal government will be limited. We recommend that the Secretary of the Treasury should ensure that the Fiscal Assistant Secretary, working in coordination with the Controller of OMB, establishes effective processes and procedures to reasonably assure that appropriate information regarding legal contingency losses is reported in the CFS. (Recommendation 4) At the completion of our fiscal year 2017 audit, 14 recommendations were open from our prior reports regarding control deficiencies in the processes used to prepare the CFS. During our fiscal year 2018 CFS audit, we found that Treasury, in coordination with OMB, implemented corrective actions that resulted in significant progress in resolving certain of the control deficiencies addressed by our prior recommendations. For two recommendations, the corrective actions resolved the related control deficiencies, and we closed the recommendations. While progress was made, 12 recommendations from our prior reports remained open as of March 20, 2019, the date of our report on the audit of the fiscal year 2018 CFS. These continuing control deficiencies contributed to the three material weaknesses that relate to the federal government’s processes used to prepare the CFS. Consequently, a total of 16 recommendations need to be addressed—12 remaining from prior reports and the four new recommendations we are making in this report. Appendix I summarizes the status as of March 20, 2019, of the 14 open recommendations from our prior years’ reports according to Treasury and OMB, as well as our own assessment and additional comments, where appropriate. Various efforts are under way to address these recommendations. As part of our fiscal year 2019 CFS audit, we will continue to monitor Treasury’s and OMB’s progress in addressing our recommendations. We provided a draft of this report to Treasury for comment. In its written comments, reproduced in appendix II, Treasury concurred with our four new recommendations. Treasury agreed that new processes and procedures would enhance its internal controls over the processes used to prepare the CFS. It also described actions it will take, and has taken, to address these recommendations as well as certain open recommendations from our prior reports that are summarized in appendix I of this report. Treasury also provided technical comments, which we incorporated as appropriate. Treasury stated that it implemented process improvements that addressed three of the new recommendations, which resulted from our review of the draft fiscal year 2018 Financial Report, prior to the publication of the final report. These three recommendations are aimed at enhancing Treasury’s processes related to (1) including appropriate disclosures required by new federal accounting standards in the Financial Report and (2) supporting and consistently reporting restatements, reclassifications, and adjustments to beginning net position. In our report, we acknowledged that Treasury addressed the need for certain additional disclosures related to new federal accounting standards and inconsistencies related to restatements, reclassifications, and adjustments to beginning net position in the final fiscal year 2018 Financial Report. However, addressing the specific issues we identified in the draft fiscal year 2018 Financial Report does not fully address our recommendations and Treasury did not provide sufficient documentation supporting its efforts to develop and implement or enhance procedures or other steps to reasonably assure that the Financial Report is complete and accurate related to these areas. Treasury also stated that it will work on addressing the remaining recommendation. Regarding five open recommendations from our prior year reports related to treaties and other international agreements, Treasury stated that its existing controls provide reasonable assurance that there are no material misstatements in the Financial Report and that it worked with federal entities in fiscal year 2018 to obtain reasonable assurance that proper amounts and disclosures are reported as commitments and contingencies. As noted in appendix I, Treasury established three broad categories to help agencies in classifying treaties and other international agreements with respect to their potential financial implications. The establishment of three standard categories provides some guidance for identifying and reporting treaties and other international agreements; however, as stated in appendix I, we continue to believe further guidance is needed to determine whether additional disclosure in the CFS is required by U.S. GAAP. The guidance should be consistent with FASAB standards and provide procedures for identifying and assessing all treaties and other international agreements for potential contingencies. Treasury also stated that its efforts to validate the material completeness of budgetary information included in the Financial Report and verify the consistency of such information with federal entity reports are sufficient to address the two open recommendations from our prior year reports related to the Reconciliations of Net Operating Cost and Budget Deficit and the Statements of Changes in Cash Balance from Budget and Other Activities (Reconciliation Statements). As noted in appendix I, Treasury has improved its process by documenting its detailed analyses related to the accrual-based and cash-based effects of federal entities’ transactions included in net operating cost and the budget deficit and by continuing to identify items needed to prepare the Reconciliation Statements. However, as noted in appendix I, we continue to believe additional work is needed to (1) reconcile line items to audited federal entity financial statements and (2) determine the appropriate presentation for the reconciling items, which could affect the line items included. Treasury stated that it appreciates our perspective and will continue to focus its efforts on cost-beneficial solutions to resolve the material conditions that preclude having an opinion rendered on the CFS. Treasury also indicated that it plans to work with GAO as it fulfills its commitment to improving federal financial reporting. As part of our fiscal year 2019 CFS audit, we will determine the status of corrective actions to address the four new recommendations made in this report and the 12 remaining open recommendations from our prior year reports. We provided a draft of this report to OMB for comment. In oral comments, OMB staff in the Office of Federal Financial Management stated that OMB generally agreed with the draft report and Treasury’s written response. OMB noted that the current administration is committed to continuing to work with Treasury and federal agencies to achieve sound financial management across the federal government. This report contains three recommendations to the Secretary of the Treasury and one recommendation to the Secretary of the Treasury, working in coordination with the Controller of OMB. The head of a federal entity is required by 31 U.S.C. § 720, to submit a written statement on actions taken or planned on our recommendations to the Senate Committee on Homeland Security and Governmental Affairs and to the House Committee on Oversight and Reform, the congressional committees with jurisdiction over the programs and activities that are the subject of our recommendations, and GAO not later than 180 days after the date of this report. A written statement must also be sent to the Senate and House Committees on Appropriations with the entity’s first request for appropriations made more than 180 days after the date of this report. Please send your statement of actions to me at simpsondb@gao.gov. We are sending copies of this report to appropriate congressional committees, the Fiscal Assistant Secretary of the Treasury, the Controller of the Office of Management and Budget’s Office of Federal Financial Management, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov/. We acknowledge and appreciate the cooperation and assistance that Treasury and OMB staff members provided during our audit. If you or your staffs have any questions or wish to discuss this report, please contact me at (202) 512-3406 or simpsondb@gao.gov. Contacts points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff members who made major contributions to this report are listed in appendix III. Table 1 shows the status of GAO’s prior year recommendations related to the processes used to prepare the consolidated financial statements of the U.S. government. The abbreviations used are defined in the legend at the end of the table. In addition to the contact named above, the following individuals made major contributions to this report: Carolyn M. Voltz and Paul F. Foderaro (Assistant Directors), LaTasha L. Freeman (Auditor-in-Charge), Youssef R. Amrani, Maria Hasan, W. Stephen Lowrey, Fabian J. Mendive, Maria M. Morton, and Kristine A. Papa.", "summary": "The Secretary of the Treasury, in coordination with the Director of OMB, prepares the Financial Report of the United States Government , which contains the CFS. Since GAO's first audit of the fiscal year 1997 CFS, certain material weaknesses and other limitations on the scope of its work have prevented GAO from expressing an opinion on the accrual-based consolidated financial statements. As part of the fiscal year 2018 CFS audit, GAO identified material weaknesses and other continuing control deficiencies in the processes used to prepare the CFS. The purpose of this report is to provide (1) details on new control deficiencies GAO identified related to the processes used to prepare the CFS, along with related recommendations, and (2) the status of corrective actions that Treasury and OMB have taken to address GAO's prior recommendations related to the processes used to prepare the CFS that remained open as of the completion of GAO's audit of the fiscal year 2017 CFS. During its audit of the fiscal year 2018 consolidated financial statements of the U.S. government (CFS), GAO identified control deficiencies in the Department of the Treasury's (Treasury) and the Office of Management and Budget's (OMB) processes used to prepare the CFS. These control deficiencies contributed to material weaknesses in internal control that involve the federal government's inability to adequately account for intragovernmental activity and balances between federal entities; reasonably assure that the consolidated financial statements are (1) consistent with the underlying audited entities' financial statements, (2) properly balanced, and (3) in accordance with U.S. generally accepted accounting principles; and reasonably assure that the information in the (1) Reconciliations of Net Operating Cost and Budget Deficit and (2) Statements of Changes in Cash Balance from Budget and Other Activities is complete, properly supported, and consistent with the underlying information in the audited entities' financial statements and other financial data. During its audit of the fiscal year 2018 CFS, GAO identified three new internal control deficiencies. Treasury did not have sufficient procedures to analyze and determine whether appropriate disclosures related to new federal accounting standards were included in the draft fiscal year 2018 Financial Report of the United States Government . Treasury did not have sufficient procedures to properly support and consistently report restatements, reclassifications, and adjustments to beginning net position in the draft fiscal year 2018 Financial Report of the United States Government . Treasury and OMB did not have adequate processes and procedures for reporting appropriate information regarding legal contingency losses in the fiscal year 2018 CFS. In addition, GAO found that various other control deficiencies identified in previous years' audits with respect to the processes used to prepare the CFS either were resolved or continued to exist. Specifically, Treasury, in coordination with OMB, implemented corrective actions that resolved the control deficiencies related to two of the 14 recommendations open as of the completion of GAO's fiscal year 2017 CFS audit, and as a result, GAO closed these recommendations. While progress was made, 12 of the 14 recommendations remained open as of March 20, 2019, the date of GAO's report on its audit of the fiscal year 2018 CFS. GAO will continue to monitor the status of corrective actions to address the four new recommendations made in this report as well as the 12 open recommendations from prior years as part of its fiscal year 2019 CFS audit. GAO is making four new recommendations—three to Treasury and one to both Treasury and OMB—to address the control deficiencies identified during the fiscal year 2018 CFS audit. In commenting on GAO's draft report, Treasury concurred with the four new recommendations and noted its ongoing commitment to improving federal financial reporting. OMB generally agreed with the draft report and noted its continuing commitment to achieving sound financial management across the federal government.", "document_type": "gao"}
{"report": "The Partnership Program is part of the Bureau’s Integrated Partnership and Communications (IPC) operation, which contains the Bureau’s varied partnership and outreach activities aimed at spreading the word about the importance of participating in the census. Specifically, the IPC operation is intended to: engage and motivate people to self-respond; raise and keep awareness high throughout the entire 2020 Census to encourage response; support field recruitment efforts for a diverse, qualified census workforce; and support dissemination of census data to stakeholders and the public. Several components of the IPC operation, including the Partnership Program, are described in more detail below. Figure 1 also demonstrates these components in example situations. The Partnership Program has two main components that deliver outreach at the national and local levels, respectively: the National Partnership Program (NPP) and the Community Partnership and Engagement Program (CPEP), as shown in table 1. NPP and CPEP are intended to complement and leverage their respective expertise to help maximize participation by partners. According to Bureau officials, the Bureau’s Census Open Innovation Labs is another important component that aims to integrate these partnership activities and provide guidance at the national and local levels on various strategies for messaging with partners. The roughly 1,500 Partnership specialists hired for 2020 are temporary Bureau employees who implement CPEP and are responsible for, among others things, establishing local partnerships and engaging those partners to host activities and events (known as “commitments”) within the communities they serve. These commitments could include activities and events such as conducting knowledge-sharing seminars, issuing press releases, providing questionnaire assistance, producing pamphlets, and hosting field recruiting events, among others. For the 2020 Census, the Bureau had a goal of establishing 300,000 community partners nationwide. In 2010, the Bureau secured approximately 256,000 local partners. The Partnership program also supports inter-governmental census outreach activities through state complete count commissions (SCCCs) at the state level and complete count committees (CCCs) at the tribal and local levels. These groups foster a collaborative partnership between political, business, and community leaders to deliver messages on the importance of participation in the census. The Bureau had a goal of encouraging every state to create an SCCC, formed at the highest level of state government, such as the governor’s office. SCCCs seek to leverage the state’s vested interest in a timely and complete population count. According to Bureau planning documentation, CCCs are usually formed by the highest elected official in that jurisdiction, such as a tribal leader, a mayor, county commissioner, or regional chairman. The Bureau notes that partnership specialists are the primary contact between the Bureau and the CCC. Partnership specialists may conduct workshops, train CCC members, and provide or direct CCCs to census promotional materials. Bureau officials noted that initial partnership specialists were hired earlier in the decennial cycle for 2020 in part to help form CCCs. Partnership specialists are also to play a role in assisting CCCs with identifying hard- to-count populations within their respective communities and assist in developing strategies to reach those communities. Similar to previous censuses, for 2020, the Bureau plans to use multiple paid media outlets such as radio, television, newspapers, magazine, and billboards as one of its means to promote awareness and encourage participation in the census. Along with the traditional advertising, the Bureau also plans to use digital advertising such as web banner ads, video ads, digital extensions of traditional outlets, and social media channels. According to Bureau officials, they plan to be able to use paid advertising modes to target specific audiences including hard-to-count populations that may be more apt to trust such communication. For example, the Bureau’s 2020 communication plan cites research that states that the majority of Asian-Americans use traditional media— whether in print, online, or from a mobile device—as their primary source of local news. The Bureau plans to implement its 2020 paid media campaign in four phases (see table 2). As of early May 2020, the Bureau has not yet indicated the revised timeframes for the phases of paid advertising to reflect the COVID-19 disruptions. Bureau officials told us that they are planning to spend an additional $160 million on advertising as part of its public education and outreach campaign, originally estimated to cost over $500 million. According to the Bureau, its paid advertising campaign is designed to reach more than 99 percent of the nation’s 140 million households. For the 2020 Census, to distribute these messages, the Bureau will conduct outreach to the media including providing talking points, media lists, news releases, fact sheets, frequently asked questions, and other scripts and messaging for multiple media platforms, such as a variety of radio, podcasts, and special events. Media-focused materials such as press kits and press releases will be available in the media section of 2020Census.gov. As part of its media relations effort, the Bureau has also developed a crisis communications plan for handling major events and potential messaging disturbances. According to the Bureau, its crisis communication plan is intended to allow it to respond quickly and effectively to any events or actions that jeopardize the public’s confidentiality or reduce its willingness to respond to the 2020 Census. For example, the Bureau stated that in response to the COVID-19 outbreak, it adapted part of its advertisement messaging to re-emphasize the importance of responding to the census online, avoiding the need for in-person follow-up interviews. The Bureau’s social media strategy for 2020 reflects the increased number of social media platforms that were not available during the last census, such as Snapchat, Instagram Live Stories, and Facebook Messenger. According to the Bureau, social media will play a critical role in raising awareness of the census—particularly among hard‐to‐count audiences—as well as enhancing customer service efforts, promoting recruiting efforts, driving online completion of the census, assisting with data dissemination, and mitigating disinformation. The Bureau’s social media outreach efforts are to leverage source material on the Bureau’s website. Such material includes prepared language for posts and graphics, lists of influential partners, messages for those partners, customer service-themed frequently asked questions that are tailored for social media, methods for sharing content across the Bureau’s regional offices, and social media events. In March 2020, the Bureau kicked off its Statistics in Schools (SIS) program. During the week of March 2-6, 2020, prior to the widespread closure of schools due to the COVID-19 outbreak, the Bureau provided daily modules to educators for dissemination to students. These modules included such topics as an introduction to the census, a virtual tour of the Bureau headquarters, and a take-home assignment in which students and their families were asked to summarize a discussion of what they want to see in their communities. According to the Bureau planning documentation, the SIS materials were to encourage students to pass along the importance of counting everyone to an adult in their home who will complete the census. SIS was also intended to raise awareness among students themselves, which can be important in instances where the presiding adult(s) have limited English proficiency and have to rely on children to translate or interpret information from English into their native language. We, the Bureau, and others, such as the Bureau’s National Advisory Committee, have previously identified several challenges the Bureau has faced related to its partnership and outreach efforts in prior censuses. While the Bureau has taken important actions to address these challenges, current events such as the COVID-19 outbreak provide a salient basis for which to continue to monitor these challenges and any effects they may have on the census. Nature of Challenge. The Bureau strives to conduct an accurate count of the nation’s population. However, some degree of inaccuracy is inevitable. While the Bureau reported that the 2010 Census did not have a significant net undercount or net overcount nationally, the Bureau also reported that errors in coverage were unevenly distributed through the population, as figure 2 shows. In addition to the undercounted groups shown below, prior censuses, such as the 1990 Census, also showed statistically significant undercounts of Non-Hispanic Asians, American Indians off (as well as on) Reservations, and Native Hawaiian or Pacific Islanders. These errors are problematic because certain groups such as minorities, young children, and renters are more likely to be missed in the census, while other groups such as those who may own a second, seasonal home are more likely to be counted more than once. The Bureau has noted these historical trends and has classified these and other subpopulations as hard to count. According to the Bureau, hard-to-count groups can share some or all of the following characteristics: Hard to Locate: Some groups are hard to locate because where they live is unknown, or they move frequently. Hard to Persuade: Other groups are hard to persuade to participate in the census. Hard to Interview: Some groups may have communication barriers, such as limited English proficiency. Hard to Contact: Other groups may live in places with access barriers, such as residents of gated communities or renters with doormen and locked buildings. Adding to the challenge in 2020 is the possibility that, as the Bureau adapts its timing and procedures for outreach and promotion, as well as data collection, there is a possibility of uneven effects resulting on the participation rates of different groups. For example, messaging and operations that emphasize the importance of filling out the census online, in response to the COVID-19 outbreak, may not be applicable to communities or groups with limited internet access. If social distancing measures result in fewer successful interviews during Non-Response Follow-Up, for instance, then these groups with less internet access will be at relatively greater risk of being missed by the census. Adapting field enumeration procedures to implement social distancing might also be less effective in addressing respondent concerns about interacting with strangers in apartment buildings or other densely populated areas if a census worker cannot practically distance themselves from the door. Actions Taken. As part of its integrated communications plan, the Bureau’s lead communications contractor segmented the market and developed a series of strategic frameworks targeting advertising and messaging to hard-to-count subpopulations through demographic profiles. These groups include: persons experiencing homelessness; households with young children; lesbian, gay, bisexual, transgender, queer, and questioning persons; persons with disabilities; the young and mobile. In addition to demographic profiles for each of these groups, the contractor compiled lists of relevant community partners and consulted with stakeholders to construct a “day-in-the-life” analysis and develop advertising strategies to align tailored census advertising with the experiences of each group. According to Bureau documentation, these strategic frameworks provide the NPP data to decide how to best target related outreach resources. The Bureau also developed for 2020 a publicly-available tool that displays the areas of the country that are considered hard to count, according to an index of demographic indicators known as the Low Response Score. Using this index, the Bureau is able to monitor incoming census data for those areas, such as response rates and hiring and recruitment numbers, to see if efforts to reach hard-to-count areas are succeeding amid challenging current events, such as the COVID-19 outbreak, which could affect willingness of workers to participate in work taking them door-to- door to nonrespondents. Using funds appropriated under the Further Continuing Appropriations Act, 2020, the Bureau also developed a mobile questionnaire assistance (MQA) initiative to deploy census workers to specific locations, such as grocery stores, houses of worship, and community centers, or at specific events, such as festivals, to assist residents of low-response areas in filling out the census. For the beginning of the self-response period, partnership specialists identified the initial locations and times for the initiative within hardest-to-count census tracts. The Bureau plans to then monitor actual self-response data later in the operation to target those areas reporting the lowest response rates. In April 2020, the Bureau announced an indefinite delay of this latter stage of MQA in response to the outbreak of COVID-19. Basis for Continued Monitoring. Participation in the census and availability of nonrespondents for follow-up will help indicate whether the Bureau is successfully reaching hard-to-count groups. With self-response and follow-up for nonrespondents still ongoing, it is too early to know the effectiveness of the Bureau’s outreach efforts. However, multiple streams of data will provide indications of Bureau success in enumerating areas and demographic groups considered hard to count: As during the 2018 Census Test, the Bureau is monitoring active data on self-response rates at the local level, which it can compare across areas it deems hard to count. The Non-Response Follow-Up operation will yield data on interview rates—namely, the rate at which census workers are able to complete interviews with residents who had not yet responded to the census. These rates can also be compared across areas deemed hard to count. Demographic evaluation data and the Bureau’s post-enumeration survey will further provide insight into whether racial, ethnic, and other demographic groups were counted accurately. Nature of Challenge. For its partnership and outreach strategies to be effective, the Bureau must have the necessary people and resources in the right places to execute those strategies. In prior censuses, we have reported on issues related to staffing and promotional materials faced by the Partnership Program. Specifically, during the 2000 Census we noted that partnership resources were stretched thin and in some cases took effect too late in the decennial cycle. We recommended that the Bureau review its staffing levels to ensure adequate support to partners, a recommendation the Bureau subsequently implemented. During the 2010 Census, partnership specialists expressed concern about the timely availability of promotional materials, and the impact on their ability to build relationships with potential partners. We recommended in December 2010 that the Bureau ensure that promotional materials are provided to partnership staff when they are hired, a recommendation that the Bureau agreed with but that remained open at the beginning of our audit work in December 2019. Similarly, in a 2012 Bureau evaluation of the 2010 NPP, the Bureau found that some national partners felt that promotional materials needed to be better tailored to target their audiences’ needs and that these partners had difficulty accessing relevant census data for their audiences. The Bureau also observed that improvements were needed in the distribution of promotional materials by region. Actions Taken. The Bureau increased its partnership specialist hiring from roughly 800 in 2010 to a little more than 1,500 in 2020. As figure 3 shows, collectively these partnership specialists were able to secure more than 307,000 community partners by the end of February 2020—higher than the Bureau’s goal of 300,000 and the roughly 256,000 local partners the Bureau had by the end of the 2010 Census. Partnership staff were also able to encourage the creation of state complete count commissions (SCCCs) in every state and territory except Nebraska and South Dakota. These SCCCs are complemented across the country by more than 10,000 complete count committees (CCCs) at the local level. The Bureau also set a priority of getting partners in the right places. In addition to the Bureau’s overall goal for community partners, the Bureau also set out to get at least one community partner in 100 percent of census tracts that, according to their predicted indicator of low response, are hardest to count. As of late March 2020, the Bureau had made progress towards this goal, establishing partnerships in 85 percent of such areas. The Bureau also provided partnership specialists a centralized website where they can access promotional materials for distribution to partners. These materials included fact sheets, brochures, and marketing messages translated into 13 languages. Bureau officials indicated that these materials were developed and published on the website prior to the completion of partnership specialist training. According to officials, the website was made available to partnership specialists when they started work. In addition to national and community partners, tribal, state, and local CCCs also have access to these materials. The Bureau also provided guidance on how to order hard copies of such materials through the U.S. Government Publishing Office. We found during the course of our audit work that these actions implemented our December 2010 recommendation. Basis for Continued Monitoring. The Bureau encountered some issues in onboarding partnership specialists and tracking the establishment of partnerships in hard-to-count areas. We will continue to monitor any effects of these issues as part of our ongoing work: The Bureau experienced delays in onboarding partnership specialists, which resulted in less time to form partnerships and meant less time for community engagement and education activities leading up to the census. While the Bureau was able to increase the number of partnership specialists from 2010, the Bureau did not get all of its more than 1,500 partnership specialists on board until November 2019, more than 4 months later than its initial goal. The Bureau successfully surpassed its nationwide goal for registering more than 300,000 community partners. However, we reported in February 2020 that the Bureau missed interim goals for getting at least 200,000 partners in place by January 1, 2020, and at least 250,000 partners in place by February 1, 2020. We reported during the 2000 Census on the benefits of having Partnership Program resources on the ground earlier in the decennial cycle. The Bureau varied by region in terms of getting community partners located in hard-to-count areas. As noted, the Bureau had a goal of getting at least one partner in each of the hardest-to-count census tracts by March 2020. Nationwide, the Bureau was able to achieve this in 85 percent of such areas as of late March 2020. According to Bureau data, five of the six regional offices reached at least 82 percent of this goal, with the Los Angeles region the farthest along at 90 percent as of late March 2020. On the other hand, only 70 percent of the hardest-to-count tracts in the Philadelphia region, which covers the Mid-Atlantic States, had at least one community partner. Bureau officials noted that partners in adjacent tracts can provide relevant services to the hardest-to-count tracts. Officials indicated that they would continue monitoring progress using this measure. Given the effect of the COVID-19 outbreak on partner activities, having partnership specialists on boarded and partnerships formed later than anticipated meant several months less time for in-person community engagement and education activities leading up to the census in some areas. In April 2020, Bureau officials told us that partnership specialists had been instructed to continue interactions with partners via conference calls, text, email, and other virtual means. Officials also cited the ability of partnership specialists to support virtual partner commitments such as radio interviews and virtual town halls, and officials noted that the Government Publishing Office could continue to directly supply hard copies of promotional materials to partners. Going forward, response rates the Bureau achieves in areas where the Bureau lagged in registering partners may shed light on whether or not having full partnership coverage in certain hard-to-count areas is associated with lower response rates. Additionally, the Bureau is conducting a survey of public awareness of and sentiment toward the 2020 Census, with a goal of evaluating the effectiveness of public communication efforts. This survey will also provide data that could help answer the question of whether having less time on the ground than anticipated for the full complement of partnership specialists and community partners affected community awareness of the census. Nature of Challenge. Partnership and outreach activities, along with local enumeration activities, span numerous decennial operations and phases of data collection, and the Bureau has at times struggled to fully integrate these efforts. For instance, we reported in July 2018 that, during the planning stages for the 2020 Census, the Bureau’s management of initiatives aimed at enumerating hard-to-count groups was decentralized and not fully integrated across operations. We recommended that the Bureau take steps to ensure greater integration of efforts going forward. Coordination across regions of the country can also be a challenge. Bureau officials overseeing CPEP told us that during the initial staffing of partnership specialists, they observed variations in how job expectations were communicated across regional offices through training. The COVID-19 outbreak further complicates coordination across the country where state, tribal, and local public health rules, guidance, and enforcement may affect how partners in different locations need to interact with the census, and messaging to Bureau partnership staff and partners may not be amenable to a one-size-fits-all approach nationwide. The challenge of coordination can also have effects on how local managers of census activities perceive the efficiency of census efforts. Partnership specialists are expected in 2020 to work on initiatives, such as MQA, to which area census office staff also contribute. However, partnership specialists do not report to managers within the area census office, but directly report to the Bureau’s regional offices. As part of our December 2010 report on hard-to-count populations, we found that about half of all local census office managers we surveyed were dissatisfied with the level of coordination between local census office staff and partnership staff, noting duplication of effort in some cases. We recommended that the Bureau develop mechanisms to increase coordination between partnership and local census office staff, a recommendation that remains open. Actions Taken. In January 2020, in response to our July 2018 recommendation cited above, the Bureau provided us evidence of ongoing, multi-team discussions focused on integrating perspectives on decisions and planning for hard-to-count enumeration activities. In closing the July 2018 recommendation, we noted that the Bureau’s use of such an integrated approach will help ensure that the Bureau’s otherwise decentralized efforts to address hard-to-count challenges will be more effective. For field operations, the Bureau is holding a series of weekly teleconferences, to which area census office managers and all partnership specialists are invited to hear about updates from Bureau management on field operations and partnership and outreach activities. Bureau officials told us that having these calls, along with providing emailed summaries of the calls, helps to standardize any updates to guidance on field procedures or messaging across the area census office and partnership staff. For example, the Bureau sent an email in mid- March 2020 to partnership staff that included guidance on emphasizing the importance of internet self-response when responding to inquiries on the COVID-19 outbreak. The Bureau indicated taking steps to standardize partnership staff training and integrate partnership staff into field enumeration operations. Specifically, officials told us they convened a nationwide, standardized training curriculum after the Bureau had hired all of its partnership specialists. This curriculum replaced what had been a regional office- specific approach. The Bureau developed guidance for partnership specialists on how to provide direct feedback to area census offices when they become aware of a potential facility or location that should be added to the Bureau’s database for enumeration. Specifically, if partnership staff become aware of soup kitchens, homeless shelters, and other locations potentially being missed, they are instructed to submit information including the address, type of living quarters, and any applicable contact information to the relevant area census office. Moreover, the Bureau indicated it had assigned at least one partnership specialist to every area census office to improve coordination between the Partnership Program and local census operations. In addition, the Bureau created a rumor control page on its website to quickly disseminate factual content in response to misinformation. Examples of misinformation that the Bureau has detected include the false suggestion of a citizenship question being on the census and the posting of a potentially fraudulent census job website. The Bureau’s rumor control page includes the email address rumors@census.gov for the public to report possible misinformation and disinformation. The web page also has a set of frequently asked questions on topics such as data confidentiality and ways to participate in the census, thus helping to ensure that partnership staff, partners, and the public alike with access to the internet have access to accurate online information. Basis for Continued Monitoring. As the Bureau experiences inevitable turnover in partnership specialists and needs to update its messaging and outreach to reflect changing conditions on the ground, such as the COVID-19 outbreak, ensuring consistent training and guidance across its partnership staff and keeping all staff abreast of changes will be important. As noted above, our December 2010 recommendation for the Bureau to increase coordination between the Partnership Program and its area census offices remains open. While partnership specialists were assigned to area census offices, the Bureau has not put in place formal expectations for how partnership staff should support area census office staff and their activities. To fully implement this recommendation, the Bureau would need to document how partnership specialists and area census office staff are expected to work together. We will also continue to monitor how area census office managers express their perspectives on the effectiveness of the Partnership Program in supporting implementation of census field operations. For example, we surveyed area census office managers in March 2020 and found that nationally a plurality (42 percent) who responded said they were dissatisfied with the level of communication and coordination between the Partnership Program and their offices, while 41 percent said they were satisfied. A plurality of respondents (44 percent) were also dissatisfied with the level of clarity of roles and responsibilities between the Partnership Program and their offices, while 38 percent of respondents were satisfied. Open-ended comments we received from survey respondents contained ideas for improving the level of communication between partnership staff and area census office staff, such as having partnership staff be more integrated within the area census office structure. We are sharing such suggestions with the Bureau in near- real time as we receive them for the Bureau to consider moving forward. We will continue to periodically survey the office managers throughout the summer of 2020. Nature of Challenge. We have previously reported on challenges the Bureau has faced across operations in measuring outcomes. Namely, during the 2018 Census Test we reported that the Bureau’s primary measure of operational progress during Non-Response Follow-Up overstated the amount of completed workload because it emphasized the number of follow-up attempts made instead of the number of cases in which the Bureau got the data it needed—the preferred outcome of the operation. Similarly, our 2017 review of in-office address canvassing—when the Bureau attempted to use aerial imagery to identify areas of the country called blocks that did not require fieldwork to validate the address list— found that the Bureau’s performance measures looked at how many of the addresses in the address list were reviewed in-office, rather than how many addresses did not require additional fieldwork. Separately measuring the contributions of partnership and outreach activities on the desired outcome of maximizing the number of residents who respond to the census is inherently difficult. In one of its evaluations of the 2010 Census, though, the Bureau noted that the Partnership Program needed to do a better job of measuring and rewarding the intensity of partner effort throughout census operations, rather than simply looking at the number of contacts made or the number of partners. As Bureau officials noted to us, the quality of partners ultimately matters more than the quantity. Actions Taken. In its performance measures of the Partnership Program, the Bureau has taken steps beyond measuring the number of partners nationwide. For example, as noted previously, the Bureau prioritizes and tracks the extent to which it secures partnerships in hard-to-count areas, acknowledging that partnerships are needed everywhere but are particularly critical in areas that may otherwise be less likely to have high response rates. Additionally, the Bureau tracks the number of events partners complete and the number of commitments partners signed up for but did not fulfill by their planned dates (known as overdue commitments). As of early March 2020, community partners had fulfilled more than 270,000 committed events nationwide, while nearly 14,000 commitments were overdue. Basis for Continued Monitoring. Going forward, Bureau evaluations of the 2020 Census will demonstrate what, if any, lessons learned the Bureau is able to draw regarding the quality of its 2020 partnership and outreach efforts. The Bureau has multiple sources of data that it is, and will be, collecting to better understand how success can be defined: While the Bureau monitors and reports the number of overdue partnership commitments (cited above), it is not clear what the current level of nearly 14,000 overdue commitments means in the context of the universe of more than 270,000 total committed events. When we asked about this issue, Bureau officials indicated that they monitor the status of overdue commitments mostly to see if partners are staying active in their communities. Officials also said they use the metric to verify that partnership specialists are recording any follow-up they do with partners that have not fulfilled their commitments. Similarly, the Bureau’s plan for the MQA initiative cites the number of census questionnaire responses received through MQA as an indicator of how successful the Bureau is in connecting partners and other census questionnaire assistance resources to areas that need them. Yet, it is unclear from Bureau documentation what levels of MQA uptake, if any, would constitute success. While low MQA uptake, for instance, could mean that partnership specialists and census response representatives were not successful in identifying the right times and locations for MQA, it could also mean that residents were successfully able to respond to the census via other means, or that they did not attend MQA events due to the COVID-19 outbreak. When we followed up with Bureau officials, they indicated that they have planned a series of required debriefings and exit interviews with partnership specialists, which will include MQA effectiveness as a topic. Such debriefings and exit interviews will likely be particularly important, given the MQA change necessitated by the COVID-19 outbreak. Lastly, the COVID-19 outbreak could further complicate the Bureau’s ability to determine the ultimate effectiveness of its partnership and outreach—how well the Bureau achieves the goal of counting everybody once, only once, and in the right place. The Bureau plans to estimate census quality by relying in part on interviews conducted door-to-door in a nationwide sample of households, scheduled for the summer and fall of 2020. The COVID-19 outbreak could prompt the Bureau to delay the related field operation to collect the data or affect household responsiveness to in-person visits. As the timing of this report coincides with ongoing implementation of the self-response period and field enumeration operations, planned Bureau evaluations and assessments will be best positioned to identify any lessons learned and determine whether, in light of the challenges cited above, the Bureau’s partnership and outreach efforts were successful in maximizing participation in the census. We provided a draft of this report to the Department of Commerce. In its written comments, reproduced in appendix I, the U.S. Census Bureau said it agreed with the findings of our report and would continue to work to implement related open recommendations from our prior reports. We are sending copies of this report to the Secretary of Commerce, the Undersecretary of Economic Affairs, the Director of the U.S. Census Bureau, and the appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. J. Christopher Mihm, (202) 512-6806 or mihmj@gao.gov In addition to the contact named above, Ty Mitchell (Assistant Director), Devin Braun (Analyst-in-Charge), Mark Abraham, Carole J. Cimitile, Alexandra Edwards, Amalia Konstas, Kerstin Meyer, Cynthia Saunders, Kate Sharkey, Farrah Stone, Jon Ticehurst, Peter Verchinski, and Alicia White made key contributions to this report.", "summary": "The decennial census is used to apportion seats in Congress, redraw congressional districts, and allocate hundreds of billions of dollars in federal assistance annually and helps to guide public policy decisions based on social, economic, and demographic data. While recent censuses appear to have been increasingly accurate, measurement errors are not evenly distributed across the population. Given the uses of census data, ensuring an accurate count is important. As part of its partnership and outreach efforts, the U.S. Census Bureau's (Bureau) Partnership Program works with local and national organizations, businesses, and governments to promote awareness of and participation in the census, as well as to help recruit census workers. GAO was asked to review the Bureau's partnership and outreach efforts, including paid advertising and targeted communications. This report examines the Bureau's progress in addressing selected prior census challenges in these areas. GAO reviewed relevant Bureau planning documentation, collected regular Bureau reports on progress, and interviewed Bureau officials responsible for partnership and outreach efforts. GAO provided a draft of this report to the Bureau. The Bureau agreed with the report's findings. The Partnership Program, a core component of the Bureau's partnership and outreach activities, delivers outreach to partnering organizations at the national and local levels in order to ensure a more complete and accurate count. These partners include retail associations, tribal, state, and local governments, local businesses, and non-profit organizations, among others. Roughly 1,500 partnership specialists, who are temporary Bureau employees responsible for building relationships with and obtaining commitments from these partners, help to implement the Partnership Program, which exists alongside several other components of the Integrated Partnership and Communications operation, as shown below. The Bureau experienced delays, however, in getting these employees onboarded. The Bureau has taken important actions to address challenges that GAO, the Bureau, and others have previously identified. These challenges include: (1) Enumerating hard-to-count groups; (2) Mobilizing partnership and outreach resources; (3) Coordinating outreach across the Bureau's organization and operations; and (4) Measuring outcomes. Events taking place during implementation of partnership and outreach activities, such as the COVID-19 outbreak, provide a salient basis for which to continue to monitor these challenges and any effects they may have on the census. Moreover, continued monitoring of the Bureau's survey of public awareness of and sentiment toward the census, for example, will provide information on whether difficulties experienced in getting partnership specialists onboarded had an effect on the success of the Bureau's outreach.", "document_type": "gao"}
{"report": "More than a dozen federal agencies—known as National Drug Control Program agencies—have responsibilities for drug prevention, treatment, and law enforcement activities. For example, the Department of Health and Human Services (HHS) has led efforts to expand access to drug treatment, and the Departments of Justice (DOJ) and Homeland Security (DHS) have taken lead roles in limiting the availability of illicit drugs through criminal investigations and prosecutions. The Anti-Drug Abuse Act of 1988 established ONDCP to enhance national drug control planning and coordination. In this role, the office is responsible for (1) leading the national drug control effort, (2) coordinating and overseeing the implementation of national drug control policy, (3) assessing and certifying the adequacy of National Drug Control Programs and the budget for those programs, and (4) evaluating the effectiveness of national drug control policy efforts. Until its 2018 reauthorization, ONDCP had been operating under the provisions of the ONDCP Reauthorization Act of 2006 pursuant to annual appropriations acts. In October 2018, the Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment for Patients and Communities Act (the SUPPORT Act) was enacted and reauthorized ONDCP and a number of its programs. The SUPPORT Act aims to address overprescribing and opioid misuse in the United States and includes provisions involving law enforcement, public health, and healthcare financing and coverage. Under both the ONDCP Reauthorization Act of 2006 and the SUPPORT Act, the Director of ONDCP is required to promulgate the National Drug Control Strategy and work with National Drug Control Program agencies to develop an annual National Drug Control Program Budget. Prior to the SUPPORT Act, the Director was required to promulgate a National Drug Control Strategy on an annual basis, while under the SUPPORT Act, generally, the National Drug Control Strategy is required to be developed every two years. ONDCP did not issue a National Drug Control Strategy for 2017 or 2018 despite the statutory requirement. Under both the ONDCP Reauthorization Act of 2006 and the SUPPORT Act, the National Drug Control Strategy is to set forth a comprehensive plan to reduce illicit drug use and the consequences of such illicit drug use in the United States by limiting the availability of and reducing the demand for illegal drugs. As part of our ongoing work, we are reviewing the 2019 National Drug Control Strategy that ONDCP issued on January 31, 2019. Agency officials told us that they began preparing the National Drug Control Strategy in early 2018—prior to the enactment of the SUPPORT Act in October 2018. Officials stated that the National Drug Control Strategy was intended to respond to the requirements of the ONDCP Reauthorization Act of 2006, which was the applicable law at that time. In light of ONDCP’s stated approach, we based our preliminary analysis of the National Drug Control Strategy on the ONDCP Reauthorization Act of 2006. However, the SUPPORT Act retained certain similar requirements for the National Drug Control Strategy contained in the prior law. According to ONDCP, the 2019 National Drug Control Strategy provides a high-level vision of federal drug control efforts by focusing on prevention, treatment and recovery, and reducing the availability of illicit drugs. The 2019 National Drug Control Strategy designates one overarching strategic objective—to reduce the number of lives lost to drug addiction— and provides some general descriptions of federal agencies’ activities. However, our preliminary observations related to the 2019 National Drug Control Strategy indicate that it does not include several pieces of required information, including the following: Quantifiable and measurable objectives. The National Drug Control Strategy was required to include “annual quantifiable and measurable objectives and specific targets to accomplish long-term quantifiable goals that the Director determines may be achieved during each year beginning on the date on which the National Drug Control Strategy is submitted.” However, our work showed that the 2019 National Drug Control Strategy does not include this information. While it lists seven items that it designates as measures of performance or effectiveness, it does not indicate how they would be quantified or measured, or targets to be achieved each year. For example, one of the measures of performance relates to educating the public, especially adolescents, about drug use. However, it lacks information on how ONDCP would measure its efforts to educate adolescents, as well as targets ONDCP hopes to achieve, such as the number of adolescents educated or specific knowledge gains. Further, none of the seven measures has a baseline of current performance or annual targets, and four of the seven measures do not have associated timelines—which are important ways that results could be quantified. For example, one of the Strategy’s measures of effectiveness is that evidence-based addiction treatment, particularly medication-assisted treatment for opioid addiction, is more accessible nationwide for those who need it. However, there is no information on the current level of treatment access, any targets for expanding access, or any associated timeline by which ONDCP hopes to achieve desired results. Program and budget priorities. The National Drug Control Strategy was required to include “a 5-year projection for program and budget priorities.” While the 2019 National Drug Control Strategy outlines several high-level priorities, including a top priority to address the current opioid crisis and its associated deaths, it does not include such a 5-year projection. Specific assessments. The National Drug Control Strategy was required to include specific assessments related to illicit drug use. For example, the National Drug Control Strategy was required to include “an assessment of the reduction of illicit drug availability.” This assessment was to be measured by, among other things, the quantities of cocaine, heroin, marijuana, methamphetamine, ecstasy, and other drugs available for consumption in the United States; the amount of marijuana, cocaine, heroin, methamphetamine, ecstasy, and precursor chemicals and other drugs entering the United States; and the number of illicit manufacturing laboratories seized and destroyed as well as the number of hectares of marijuana, poppy, and coca cultivated and destroyed domestically and in other countries. The 2019 National Drug Control Strategy does not include this information. In addition, the National Drug Control Strategy was required to include “an assessment of the reduction of the consequences of illicit drug use and availability.” This assessment was to include the burden illicit drug users placed on hospital emergency departments; the annual national health care cost of illicit drug use; and the extent of illicit drug-related crime and criminal activity. Similarly, the 2019 National Drug Control Strategy does not include this information. Performance measurement system. The ONDCP Director was required to submit “as part of the National Drug Control Strategy a description of a national drug control performance measurement system.” Among other things, this system was to describe the sources of information and data that would be used for each performance measure incorporated into the performance measurement system. This system was also to coordinate the development and implementation of national drug control data collection and reporting systems to support policy formulation and performance measurement. Further, the system was to monitor consistency across the drug-related goals and objectives of the National Drug Control Program agencies and ensure that each agency’s goals and budgets support are fully consistent with the National Drug Control Strategy. The 2019 National Drug Control Strategy does not contain a description of such a national drug control performance measurement system. As part of our ongoing work, we also asked ONDCP for information regarding how ONDCP officials determined the adequacy of National Drug Control Program agencies’ budget submissions without a National Drug Control Strategy in effect for 2017 and 2018. The National Drug Control Program Budget is to provide information on federal drug control funding requested by the executive branch to implement the National Drug Control Strategy. National Drug Control Program agencies are required to submit to ONDCP the portion of their annual budget requests dedicated to drug control activity undertaken by the department, agency, or program. Agencies are to prepare these as part of their overall budget submission to the Office of Management and Budget for inclusion in the President’s annual budget request. ONDCP is required to review and certify whether these budgets are sufficient to support the relevant goals and objectives outlined in the National Drug Control Strategy and then include these budgets in the consolidated National Drug Control Program Budget, which the President issues alongside his budget each year. As of March 4, 2019, ONDCP had not provided information on how it accomplished the required determination. In addition, as of March 4, 2019, the President’s FY 2020 budget, and the accompanying National Drug Control Program Budget, had not been released. We will continue to consider ONDCP’s activities in this area as part of our ongoing work. As part of our ongoing work, we will also discuss the 2019 National Drug Control Strategy with ONDCP and plan to examine how ONDCP intends to address additional requirements in the SUPPORT Act. The lack of information in the 2019 National Drug Control Strategy on assessing progress toward its objective to reduce lives lost reflects findings in our previous reports. Our prior work in general, and our work on federal drug control efforts in particular, has consistently emphasized the importance of setting clear priorities through measurable and quantifiable goals, and assessing progress toward those goals over time, in order to achieve results. For example: In March 2018, we reported on federal agencies’ efforts—including those of ONDCP—to limit the availability of and enhance their response to illicit opioids, such as heroin and fentanyl. We reviewed five strategies related to combating illicit opioids and determined that only one included outcome-oriented performance measures that aim to assess the effectiveness of its efforts—ONDCP’s Heroin Availability Reduction Plan (HARP). In contrast, we found that ONDCP’s High Intensity Drug Trafficking Areas (HIDTA) programs’ Heroin Response Strategy did not include any outcome-oriented performance measures. Outcome measures address the results of programs and services, such as reductions in overdose deaths, and they can help in assessing the status of program operations, identifying areas that need improvement, and ensuring accountability for results. Among other things, we recommended in March 2018 that ONDCP coordinate with the HIDTAs to establish outcome-oriented performance measures to assess progress towards the goals set out in the Heroin Response Strategy. While ONDCP neither agreed nor disagreed with our recommendation, ONDCP told us in June 2018 that they had engaged with leaders from the HIDTAs participating in the Heroin Response Strategy to develop performance measures, including some outcome performance measures. As of March 4, 2019, this recommendation has not yet been addressed and ONDCP has not provided additional information on these efforts. We continue to believe that establishing these measures would enhance the HIDTAs’ ability to assess whether these efforts are producing intended results. In October 2017, we reported on HHS’s efforts to reduce the prevalence of opioid misuse and the fatalities associated with it by expanding access to medication-assisted treatment (MAT) for opioid use disorder. These efforts included four grant programs that focus on expanding access to MAT in various settings (including rural primary care practices and health centers) and implementing regulatory and statutory changes that expanded treatment capacity by increasing patient limits for a MAT medication—buprenorphine—and by expanding the types of practitioners who can prescribe it in an office-based setting. We found that HHS had not established performance measures with targets that would specify the results that HHS hoped to achieve through its efforts, and by when. We concluded that without this information, HHS would not have an effective means to determine whether its efforts are helping to expand access to MAT or whether new approaches are needed. Among other things, we recommended that HHS establish performance measures with targets related to expanding access to MAT for opioid use disorders. HHS concurred with the recommendation and in February 2019, provided information that the agency had established performance measures with targets to increase the number of prescriptions for certain MAT medications, one of the potential ways to measure access to MAT. However, the recommendation has not yet been fully addressed, in part because HHS did not provide information on measures related to the treatment capacity of providers who prescribe or administer MAT medications, which HHS had identified as another way to measure access. We continue to believe that fully implementing this recommendation will help ensure that invested resources in the program are yielding intended results. As our prior work shows, using data—such as information collected by performance measures and findings from program evaluations and research studies—to drive decision-making can help federal agencies improve program implementation, identify and correct problems, and make other management decisions. Although agencies may struggle to effectively use this approach, regular performance reviews and evidence- based policy tools can help them incorporate performance information into federal decision-making. Without effective long-term plans, such as a national strategy, that clearly articulate goals and objectives and without specific measures to track performance, federal agencies cannot fully assess whether taxpayer dollars are invested in ways that will achieve desired outcomes. ONDCP’s responsibility to develop the National Drug Control Strategy and coordinate among federal agencies offers the agency an important opportunity to guide federal activities to address the unprecedented number of drug overdose deaths. We are continuing with ongoing and planned work to monitor and assess federal drug control efforts. Chairman Cummings, Ranking Member Jordan, and Members of the Committee, this concludes our prepared statement. We would be happy to respond to any questions you may have at this time. If you or your staff has any questions concerning this testimony, please contact Triana McNeil at (202) 512-8777 (McNeilT@gao.gov) or Mary Denigan-Macauley at (202) 512-7114 (DeniganMacauleyM@gao.gov). Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contacts named above, Joy Booth (Assistant Director), Will Simerl (Assistant Director), Michelle Loutoo Wilson (Analyst in Charge), Billy Commons, Helen Desaulniers, Wendy Dye, Jane Eyre, Kaitlin Farquharson, and Jan Montgomery made key contributions to the testimony. Other staff who made key contributions to the reports cited in the testimony are identified in the source products. Drug Control: DOD Should Improve Its Oversight of the National Guard Counterdrug Program. GAO-19-27. Washington, D.C.: January 17, 2019. Colombia: U.S. Counternarcotics Assistance Achieved Some Positive Results but State Needs to Review the Overall U.S. Approach. GAO-19-106. Washington, D.C.: December 12, 2018. Opioid Crisis: Status of Public Health Emergency Authorities. GAO-18-685R. Washington, D.C.: September 26, 2018. Prescription Opioids: Medicare Needs Better Information to Reduce the Risk of Harm to Beneficiaries. GAO-18-585T. Washington, D.C.: May 29, 2018. VA Health Care: Progress Made Towards Improving Opioid Safety, but Further Efforts to Assess Progress and Reduce Risk Are Needed. GAO-18-380. Washington, D.C.: May 29, 2018. Illicit Opioids: Office of National Drug Control Policy and Other Agencies Need to Better Assess Strategic Efforts. GAO-18-569T. Washington, D.C.: May 17, 2018. Illicit Opioids: While Greater Attention Given to Combating Synthetic Opioids, Agencies Need to Better Assess their Efforts. GAO-18-205. Washington, D.C.: March 29, 2018. Substance Use Disorder: Information on Recovery Housing Prevalence, Selected States’ Oversight, and Funding. GAO-18-315. Washington, D.C.: March 22, 2018. Substance-Affected Infants: Additional Guidance Would Help States Better Implement Protections for Children. GAO-18-196. Washington, D.C.: January 19, 2018. Opioid Use Disorders: HHS Needs Measures to Assess the Effectiveness of Efforts to Expand Access to Medication-Assisted Treatment. GAO-18-44. Washington, D.C.: October 31, 2017. Counternarcotics: Overview of U.S. Efforts in the Western Hemisphere. GAO-18-10. Washington, D.C.: October 13, 2017. Preventing Drug Abuse: Low Participation by Pharmacies and Other Entities as Voluntary Collectors of Unused Prescription Drugs. GAO-18-25. Washington, D.C.: October 12, 2017. Prescription Opioids: Medicare Needs to Expand Oversight Efforts to Reduce the Risk of Harm. GAO-18-15. Washington, D.C.: October 6, 2017. Newborn Health: Federal Action Needed to Address Neonatal Abstinence Syndrome. GAO-18-32. Washington, D.C.: October 4, 2017. Opioid Addiction: Laws, Regulations and Other Factors Can Affect Medication-Assisted Treatment Access. GAO-16-833. Washington, D.C.: September 27, 2016. Anti-Money Laundering: U.S. Efforts to Combat Narcotics-Related Money Laundering in the Western Hemisphere. GAO-17-684. Washington, D.C.: August 22, 2017. International Mail Security: Costs and Benefits of Using Electronic Data to Screen Mail Need to Be Assessed. GAO-17-606. Washington, D.C.: August 2, 2017. Drug Control Policy: Information on Status of Federal Efforts and Key Issues for Preventing Illicit Drug Use. GAO-17-766T. Washington, D.C.: July 26, 2017. Medicaid Expansion: Behavioral Health Treatment Use in Selected States in 2014. GAO-17-529. Washington, D.C.: June 22, 2017. Drug-Free Communities Support Program: Agencies Have Strengthened Collaboration but Could Enhance Grantee Compliance and Performance Monitoring. GAO-17-120. Washington, D.C.: February 7, 2017. Highlights of a Forum: Preventing Illicit Drug Use. GAO-17-146SP. Washington, D.C.: November 14, 2016. Drug Enforcement Administration: Additional Actions Needed to Address Prior GAO Recommendations. GAO-16-737T. Washington, D.C.: June 22, 2016. State Marijuana Legalization: DOJ Should Document Its Approach to Monitoring the Effects of Legalization. GAO-16-419T. Washington, D.C.: April 5, 2016. Firearms Trafficking: U.S. Efforts to Combat Firearms Trafficking to Mexico Have Improved, but Some Collaboration Challenges Remain. GAO-16-223. Washington, D.C.: January 11, 2016. Prenatal Drug Use and Newborn Health: Federal Efforts Need Better Planning and Coordination. GAO-15-203. Washington, D.C.: February 10, 2015. Office of National Drug Control Policy: Office Could Better Identify Opportunities to Increase Program Coordination. GAO-13-333. Washington, D.C.: March 26, 2013. Prescription Pain Reliever Abuse: Agencies Have Begun Coordinating Education Efforts, but Need to Assess Effectiveness. GAO-12-115. Washington, D.C.: December 22, 2011. This is a w ork of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety w ithout further permission from GAO. How ever, because this w ork may contain copyrighted images or other material, permission from the copyright holder may be necessary if you w ish to reproduce this material separately.", "summary": "Over 70,000 people died from drug overdoses in 2017, according to the most recently available Centers for Disease Control and Prevention data. Overdoses have become the leading cause of death due to injuries in the United States, and most of these deaths involve opioids. GAO has a body of work on drug policy and ongoing work on ONDCP's efforts, including issuance of the National Drug Control Strategy. GAO also noted in its March 2019 High Risk report that federal efforts to prevent drug misuse is an emerging issue requiring close attention. This statement includes preliminary GAO observations on the 2019 National Drug Control Strategy and related findings from select GAO reports on federal opioid-related efforts. It is based on ongoing GAO work, two reports that GAO issued in March 2018 and October 2017, and selected updates on recommendations from these reports as of February 2019. For ongoing work and recommendation updates, GAO assessed the 2019 National Drug Control Strategy against statutory requirements, reviewed ONDCP and HHS documents, and interviewed ONDCP officials. The Office of National Drug Control Policy (ONDCP)—responsible for coordinating and overseeing efforts by more than a dozen federal agencies to address illicit drug use—issued the 2019 National Drug Control Strategy on January 31, 2019. ONDCP describes the strategy as a high-level vision of federal drug control efforts, focused on prevention, treatment and recovery. The strategy designates one overarching objective to reduce the number of lives lost to drug addiction, and provides some description of federal agencies' activities, including steps to reduce the availability of illicit drugs. However, it does not include certain information required by law, such as annual objectives that are quantifiable and measurable, or a 5-year projection for program and budget priorities. This required information could help prioritize activities across federal agencies and measure progress over time, which previous GAO work has shown to be important for achieving results. GAO will continue to assess the strategy as part of ongoing work, and make recommendations as appropriate. The lack of information in the 2019 National Drug Control Strategy on measuring progress toward its objective to reduce lives lost is particularly concerning in light of previous GAO reports. These reports found that individual agencies could do more to assess their particular efforts related to opioids. For example, GAO reported in March 2018 on five agency-specific strategies to combat illicit opioids, and also reported in October 2017 on the Department of Health and Human Services' (HHS) efforts to expand access to medication-assisted treatment for opioid use disorder. In these reports, GAO recommended, among other things, that federal agencies establish performance measures to better determine progress toward their goals. While federal agencies have taken some action to implement these recommendations, such as establishing performance measures for access to medication-assisted treatment, additional actions to measure the effectiveness of related drug control efforts would further help to gauge agencies' success, determine if new approaches are needed, and efficiently target resources. GAO has made prior recommendations to ONDCP, HHS, and other federal agencies to address drug misuse, including establishing performance measures with targets to better gauge progress toward achieving agency goals.", "document_type": "gao"}
{"report": "The U.S. freight rail network is a private industry that moves about 40 percent of U.S. freight across about 140,000 miles of track. Freight railroads are responsible for the safety of their trains, tracks, and personnel. These railroads rely on their revenues for funding to perform track inspections and maintenance. Close to 600 freight railroads operate in the United States, and they are divided into three classes according to revenues. The seven Class I freight railroads, those with approximately $447.6 million or more in annual revenue over 3 consecutive years, account for more than 90 percent of the annual revenues of the railroad- freight industry and employ 90 percent of railroad employees. Class I freight railroads move freight over long-haul routes and face competition from each other and from other freight-shipping modes, such as trucks and barges. Class II and III railroads tend to operate over much smaller geographic areas than Class I railroads and employ fewer people. Trains operate on different types of train tracks, including main line tracks—the primary rail arteries trains use to travel—and sidings, which are primarily auxiliary tracks for trains to pass one another. Trains can be stored, sorted, and assembled in railyards. Railroads use main line and sidings to enable trains to enter and leave the yard. Outside railyards, some sections of main line track have sidings that lead to a parallel set of rails to allow trains to pass one another. In addition, some portions of main line track consist of two or more parallel sets of track to allow trains to pass one another or travel in opposite directions. See figure 1 for an illustration of different track types and siding. Train makeup refers to the placement of individual railcars that make up a train. A typical train consists of a locomotive—the power and control unit where the train operator sits at the front of the train—followed by connected railcars. The lead locomotive pulls the train and provides control for other functions, including braking. Freight trains carry a variety of freight using different types of railcars that vary in capacity, length, and weight. When assembling a train, railroads consider a variety of factors— such as each car’s weight, length, freight, and whether it is loaded or empty—when determining its position in the train. Train make up is also dependent on external conditions, such as variations in terrain and weather conditions. Railroads can place radio-controlled locomotives, called distributed power (DP) units, throughout trains to spread out pulling and pushing power, which we discuss in more detail below. Proper train makeup is critical for ensuring a train is able to effectively negotiate track and prevent derailment, according to FRA. Improperly assembled trains are more susceptible to derailment, in part because of vertical, longitudinal, and lateral forces throughout the train—also known as “in-train” forces—that can affect the stability of a train on its tracks, depending on a variety of factors, including the train’s speed and terrain. For example, excessive “in-train” forces can cause a long, heavy train to pull apart or climb off the track upon a change of grade (e.g., going up or down hills) or when the train enters a curve. “Unit trains”—which carry a single commodity, such as coal or oil, to one destination—experience in-train forces that are easier for railroads to model and engineers to predict because the railcars are generally uniform in size and weight. In comparison, determining train makeup is more complex in mixed freight trains, which can experience more unpredictable in-train forces resulting from railcars of different weights, lengths, and freight (e.g., bulk goods such as grain and coal, consumer goods such as automobiles, or hazardous materials). For example, if a train is assembled in a manner in which empty railcars alternate with loaded, heavy railcars, braking can create compression at the couplers and cause “buckling”—when an empty train car is compressed between heavier railcars and derails from the train tracks. Freight trains in the United States utilize air-braking systems to control speed and stop. A conventional air-braking system is controlled by an air pressure signal from the leading locomotive, which sends a signal through the train to engage brakes. Because each railcar receives this signal sequentially, it takes multiple seconds for railcars at the end of the train to receive the air pressure signal and begin braking, depending on the train’s length. The application of air brakes generates in-train forces, as railcars at the front of the train that have applied brakes will be pushed by railcars further back that have not yet received the air signal. Other technologies, including two-way end-of-train devices and DP, are frequently used by U.S. railroads in conjunction with conventional brakes to provide improved braking performance or other benefits. End-of-train devices measure brake pressure and transmit this information via radio signal to the front of the train. An end-of train-device can also engage air brakes at the rear end of a train in an emergency to decrease the time required to apply the brakes on all cars. As previously mentioned, railroads can also place radio-controlled locomotives, called DP units, throughout trains to spread out pulling and pushing power and improve braking. For example, engineers can engage a DP locomotive’s air brakes at the same time as a leading locomotive to decrease the time needed to activate brakes throughout the train. (See fig. 2.) Engineers can also use a locomotive’s dynamic brake system, which uses traction generated by the engine, to slow a train. While most railroads employ conventional brakes, railroads can also employ electronically controlled pneumatic brakes—which provide an electronic brake signal instantaneously throughout a train—allowing railcars to brake faster than with conventional air brakes. As we previously reported, electronically controlled pneumatic brakes reduce the in-train forces that occur during braking when individual cars push and pull against one another. Freight trains in the United States generally operate with two crew members—the conductor and the engineer. The conductor is responsible for the train, freight, and crew. The engineer operates the locomotive, including application of air brakes, dynamic brakes, and any radio- controlled DP locomotives. Train crews use hand-held radios to communicate when they are working in different parts of the train. For example, if the crew detects a train maintenance issue, the conductor may need to leave the locomotive and walk the length of the train to address the problem. In these situations, the conductor may use a hand- held radio to communicate. The U.S. Department of Transportation (DOT) is responsible for ensuring the safety of the transportation system. Within DOT, FRA is the primary federal agency responsible for formulating railroad safety policies and regulations and for monitoring and enforcing railroads’ compliance with requirements. FRA’s mission is to enable the safe, reliable, and efficient movement of people and goods. FRA provides regulatory oversight of the safety of U.S. railroads, including both passenger and freight. FRA issues and enforces safety regulations including requirements governing track, signal, and train control systems, grade-crossing warning systems, mechanical equipment such as locomotives and railcars, and railroad-operating practices. In developing most of its regulations, FRA seeks input from the railroad industry and other organizations through its Railroad Safety Advisory Committee. FRA provides oversight of railroad safety through a variety of activities, including periodic inspections and enforcement actions. FRA has safety inspectors and specialists in eight regional offices that are primarily responsible for the enforcement of federal laws and regulations related to railroad safety. FRA conducts inspections of railroads to monitor compliance with safety regulations, such as those governing the transport of hazardous materials, among other issues. In addition, 31 states conduct inspections for compliance with federal safety regulations. FRA trains state inspectors to enable them to conduct inspections according to FRA’s standards. In addition to these activities, FRA conducts other types of safety oversight to reduce train accidents, such as analyzing railroad safety data, investigating accidents, and reviewing complaints. FRA also funds research and development to support its safety oversight. FRA’s Office of Research, Development, and Technology conducts research to understand railroad safety risks and improve safety. This work contributes information used to inform FRA’s development of regulations, standards, and best practices. Other federal agencies also have roles in overseeing freight railroads, such as through promoting safety or regulating railroad industry economics. For example, NTSB is an independent federal agency that produces safety studies and investigates transportation-related accidents across all transportation modes to determine probable causes, identify safety issues, and make recommendations to prevent recurrences. STB oversees significant rail-service matters and resolves rate and service disputes between railroads and their customers, known as “shippers.” Class I railroads report data to STB on the amount and type of freight they transport. STB produces and releases statistical data derived from the railroad’s submitted data. According to officials from all seven Class I freight railroads and representatives from AAR, FRA, STB, and other stakeholders we interviewed, freight train-length has increased in recent years; however, the data are limited. According to data that two Class I railroads provided to us, their average train length increased over the 10-year period of 2008 through 2017 by about 1,500 feet for one railroad (from about 6,000 to 7,500 feet, or up to about 1.4 miles) and about 1,200 feet for the other railroad (from about 4,900 to 6,100 feet, or up to about 1.2 miles). These data represent an increase in the average length of a train of about 25 percent for both railroads. Two additional Class I railroads reported average train lengths between about 5,800 and 6,600 feet for the year 2017. However, we were not able to verify increases more broadly because FRA, STB, and AAR do not collect comprehensive data on train- length in feet, and while such data are collected by Class I railroads, they are not publicly available. Officials from two Class I railroads stated that operating longer trains is not a new practice, and one official noted that the railroad has been operating trains in excess of 10,000 feet in selected rail corridors for almost 30 years. Officials from AAR added that increases in train length over time have likely been gradual. While two Class I railroads provided data on average freight train-length over time, officials from each of the seven Class I railroads stated that they operate longer trains. Railroad officials said they operate these trains in certain rail corridors that have the capacity to accommodate longer trains, and not over their entire rail networks. For example, officials from one Class I railroad said they are running on a daily basis a 12,000-foot train—which is about 2.3 miles long—and another reported that twice weekly it operates a 16,000-foot train—which is about 3 miles long—on a route linking the mid-west to the west coast. Both of these Class I railroads noted that longer trains such as these are a small percentage of the trains they operate. More specifically, one of these railroads reported that over the previous 24 months, about 1 percent of its train-miles were traveled by trains over 10,000 feet long, and the other reported that about 2 percent of its current train-miles were traveled by trains over 10,000 feet long. Other data describing the average number of railcars per train and average weight of trains indicate an overall increase over the past 10 years. However, these measures are not proxies for freight train-length since the length and weight of railcars can vary significantly depending on their design and freight. Class I railroads are required to report data, such as the freight car-miles, to STB annually, and AAR aggregates and makes this information publicly available in various publications. We analyzed these data and found that the average number of railcars per freight train across all Class I railroads increased from 71.0 to 73.2 railcars per train (an increase of about 3 percent) from 2008 through 2017. Additionally, FRA and some Class I railroad officials stated that railroads operate freight trains that have more than twice this average number of railcars—including trains with 150 railcars or more. Similarly, the average train weight increased from about 5,978 tons to 6,577 tons per train (an increase of about 10 percent) from 2008 through 2017. Class I railroad officials said that there are advantages to operating longer freight trains in some rail corridors and that operating longer trains is part of strategic planning for many railroads for a variety of reasons. Officials from all Class I railroads stated that they operate longer trains in some rail corridors as a way to increase efficiencies, such as fuel efficiency, and decrease costs by reducing the number of train crew and other costs. Additionally, railroad officials said that running longer trains can mean that they do not need to operate as many trains—officials from six Class I railroads specifically indicated they are operating fewer shorter trains as a result of operating longer trains. Further, Class I railroad officials stated that market forces, such as competition from the trucking industry, create an incentive for them to increase efficiency. Class I railroad officials also stated that the use of certain technologies, such as DP locomotives, enables them to operate longer trains more safely. Other Class I railroad officials attributed their increased usage of longer trains to capital improvements on railroad tracks, such as lengthening the sidings to accommodate longer trains. While the need for proper train makeup and handling are not unique to longer trains, it is particularly important for their safe operation, according to stakeholders we spoke with. As previously discussed, the length of each train and its makeup—the manner in which its cars and locomotives are arranged—can affect the forces involved on a moving train. Stakeholders we spoke with said that the consequences of improper train makeup may be more pronounced in longer trains—especially in situations with extremes in track grade, curvature, or weather conditions—and may add to the challenges of operating longer trains. For example, FRA has investigated accidents in which it determined that train makeup and handling were the probable cause and contributing factors in train derailments of longer freight trains. According to officials from FRA, NTSB, railroad employee unions, and other stakeholders, longer mixed- freight trains may be more difficult to handle than unit trains in certain circumstances due to variations in car length and weight and the extent to which additional DP locomotives are employed. Stakeholders noted that placing additional DP locomotives within a train can improve train handling and prevent train separations and derailments. Stakeholders added that using DP can also help improve air brake performance and reduce braking response time, as previously discussed. In addition, according to stakeholders, use of properly positioned DP locomotives can improve radio communication between the lead locomotive and rear DP locomotives on longer trains. Union representatives added that in their view, the safest train-braking operations are when DP locomotives are used in conjunction with electronically controlled pneumatic brakes. According to representatives from AAR and Class I railroads, however, freight railroads have faced challenges with these braking systems, including reliability issues, as we have noted in a previous report. While there are no comprehensive federal regulations that govern train makeup, including use of DP locomotives, representatives of Class I railroads told us they consider a variety of factors when determining train makeup to ensure safe operation of all of their trains, including tonnage, train-length, and terrain. According to one railroad, using software to determine train makeup and predict train handling needs is an industry standard and critical best practice. Another railroad told us they use computer simulations to develop rules for train makeup in order to operate longer and heavier trains. Some railroads told us they impose length and weight restrictions on specific routes to ensure safe train operation and manage corridor capacity. Union representatives and rail experts we spoke with told us that in their view, railroads do not always properly assemble their longer trains, for example placing heavy railcars behind lighter railcars, a practice that can increase the likelihood of derailment. These stakeholders also said that railroads do not always use DP with longer trains, which experts attributed to the extra cost of deploying additional locomotives. We did not independently verify how the railroads we spoke with assemble or operate longer trains. Stakeholders we interviewed said that it is essential that crews are properly trained to operate longer freight trains. FRA regulations require railroads to train and certify their train crews. More specifically, FRA requires qualified locomotive engineers to demonstrate proficiency in operating trains in the most demanding type of service they may be permitted to perform, which includes operating longer trains. Railroads are required to conduct annual performance evaluations of engineers to ensure that they can safely operate trains according to federal railroad safety requirements. Representatives of Class I railroads told us they train their crews on trains and simulators with various routes, scenarios, and train lengths. However, union representatives said that some railroads do not provide sufficient training for crews to operate longer trains, and that some locomotive engineers and conductors lack the necessary training and experience to handle longer trains, a situation that can be challenging even for properly trained crew. As discussed later, FRA is planning to review this issue when it performs planned audits of Class I railroads’ training programs. Stakeholders we interviewed identified additional challenges for crews when operating longer freight trains related to crew members’ fatigue. For example, according to FRA, union representatives, and other stakeholders, a longer train may require crew members to walk a long distance if the train stops unexpectedly. For example, when there is a mechanical or other problem that causes a train to stop, the conductor may have to walk from the lead locomotive to the problem area and back again. This could mean walking 4 miles to the end and back on a 2-mile- long train. Also, according to FRA officials, as with any train that is left unattended, the crew must apply a sufficient number of handbrakes to prevent unintended movement. With longer and heavier trains, railroads may require additional handbrakes to be applied. According to union representatives, such physically demanding tasks can increase crew fatigue. Stakeholders we interviewed expressed divergent views about whether longer trains may increase or decrease blockages at grade crossings. Our prior work has noted a connection between the volume of freight rail traffic and the potential for grade-crossing blockages to increase. In 2014, we found that trends in freight flows, if they continue as expected, may exacerbate congestion issues in communities, particularly along certain corridors. FRA officials told us that complaints about blocked highway-rail grade crossings have increased in recent years. They noted that blocked crossings are a local concern and it is not clear the extent to which longer freight trains are contributing to increases in reporting about such blockages. According to FRA, trains sometimes block crossings for a limited time or for hours if an accident or mechanical problem occurs. They noted that such blockages can be created by trains of any length and that in their experience, railroads prioritize movement of longer trains, making it less likely that such trains would be responsible for prolonged blockages of crossings. Furthermore, officials from FRA and Class I railroads and others we spoke with pointed out that longer trains may decrease the frequency of blocked crossings, as railroads may run fewer trains. In contrast, officials from the National League of Cities, as well as state and local officials we spoke with, expressed concerns over increased frequency of longer trains and their impact on grade crossing safety. Although they also acknowledged that trains of any length can block grade crossings, they raised concerns that longer trains prolong the duration of a blockage and can block more crossings concurrently, making it harder for vehicles to find an alternate route around the train. Consequently, these stakeholders are concerned that longer trains create increased delays for emergency responders and increase the likelihood of unsafe behavior among motorists and pedestrians, as outlined below. Delayed emergency response. According to national, state, and local officials we interviewed, longer trains pose concerns about the potential for emergency response delays if responders encounter a train blocking one or more crossings and cannot quickly find an alternate route around it. (See fig. 3.) For example, officials in Mount Victory, Ohio, reported that 22 freight trains travel through their town daily, including a 16,000-feet train, which is nearly 3 miles long. This train blocks 4-to-5 grade crossings concurrently, which increases the time to access parts of the town, according to local officials. Our prior work has found that blocked highway-railroad grade crossings can have significant impacts on emergency response time and outcomes. For example, we reported an instance of a fire that destroyed a house while train traffic blocked the only two crossings in the town and prevented fire crews from responding in time. In another example, a local official in Texas said that one Class I railroad assembles trains and conducts brake checks on the main line tracks because the trains are too long to fit into sidings and railyards. Executing such procedures on mainline track has blocked grade crossings for up to several hours and poses safety challenges for surrounding communities, according to this local official. As a result of situations like these, communities are looking for ways to mitigate delays in emergency services when emergency vehicles must find ways around blocked grade crossings. For instance, some local officials in Washington and Ohio said they have revised their emergency response plans to avoid grade crossings that are likely to be blocked. Motorist and pedestrian behavior. Stakeholders we spoke with expressed concerns that longer trains may increase the likelihood of unsafe behavior among motorists and pedestrians. For example, fatalities can occur when motorists or pedestrians engage in risky behavior such as trying to make it across the tracks before an approaching train reaches the crossing. Moreover, pedestrians have been known to crawl over, through, or under stopped trains (see fig. 4). For example, local officials in Ohio and Texas told us that they have witnessed children crawling through stopped trains to get to school. Research sponsored by FRA has identified driver behavior as the main cause of highway-rail grade crossing collisions, but other factors such as train and traffic volume can contribute to the risk of a crash occurring. Although there are no current federal regulations that directly address blocked crossings or limit the amount of time trains can block grade crossings, some states and localities have attempted to address this issue. For example, some states and localities have passed laws limiting the duration of blocked crossings and proposed fines for railroads, but state and local officials and other stakeholders we spoke with said that federal law preempts such efforts. Other states and local communities have attempted to address blocked crossings through studies and communication with federal agencies and railroads, with mixed success. For example, the Texas Department of Transportation has undertaken mobility studies for the towns of El Paso and Laredo to identify options to address blocked crossings, such as constructing bridges or underpasses. According to officials with the Texas Department of Transportation, these studies identified alternatives that may help alleviate some of the vehicular/rail conflicts if they were implemented; however, the implementation of alternatives for any potential projects are constrained by the availability of funds. In other examples, local officials from Ohio and Illinois told us they have contacted Class I railroads and FRA to find solutions when idle trains lead to blocked crossings, especially when emergency access is a concern but continue to face challenges. Class I railroads and FRA officials said they work with local communities to find solutions to these issues. Additionally, state and local officials noted that they do not have access to information on the length of trains that travel through their communities. Some added that freight railroads are not required to provide such data and that local efforts to gather this information, such as through videotaping train movements and analyzing data, are costly. This circumstance makes it challenging for state and local officials to assess the extent to which longer trains may or may not be contributing to blocked crossings. In the fall of 2017, FRA began a study to understand operational risks of long freight trains. The study is examining issues related to train makeup and handling, including the use of DP locomotives, crew training and fatigue, and braking performance for longer trains. The study intends to identify strategies to reduce any risks identified. According to FRA, as the railroad industry has increased the length of freight trains, past accepted practices for train makeup and handling may not be appropriate for longer trains. For example, according to FRA, the current performance standard for air brakes was last updated in 1947 and based on tests for trains with up to 150 cars. FRA officials stated the study will conduct air brake tests to evaluate brake performance for trains with 150-to-250 railcars and use this data to conduct computer simulations of trains in a variety of configurations—for example, with and without DP and with DP locomotives at different locations throughout a train—to evaluate in-train forces. According to FRA officials, this information will help FRA determine whether rail safety issues exist for trains with over 150 railcars and if regulatory actions are necessary. The study employs a two-phase approach that includes data analysis, literature review, computer simulations, and brake testing. FRA officials said the agency plans to complete the first phase of its study and issue a report by the end of 2020 and issue a report on the second phase by the end of 2021. Table 1, below, outlines specific tasks of the study by phase. As we previously mentioned, FRA provides oversight of railroad safety through a variety of activities to ensure compliance with regulations, such as conducting inspections of railroad operations and reviewing and approving new and materially modified railroad crew training programs. According to FRA officials, these activities address safety for all freight trains, including longer trains. In addition to these activities, FRA plans to begin new, more in-depth audits of Class I railroads’ training programs on a systematic basis in 2019 to determine whether engineers are being adequately trained to operate longer trains and perform other types of demanding service. According to FRA, these audits will determine whether locomotive engineer certification programs are in compliance with federal rail safety regulations. For example, federal regulations require that railroads provide training to their engineers—through classroom lessons and in trains or simulators—on the most demanding type of service they may be called upon to perform. According to FRA officials, this would include operation of longer freight trains in challenging terrain. FRA plans to audit the training programs of three Class I railroads by the end of 2019, selected based upon safety risk factors, with additional audits of other railroads planned for the following year. Once the audits are complete, FRA plans to discuss its findings with each audited railroad and make recommendations for improvements, as needed. While FRA’s study to assess operational safety risks of longer trains is under way, the agency lacks a current, documented strategy for how it will use and share the results of its research with relevant stakeholders. According to FRA officials, after internal review and approval, the agency routinely shares its research results at conferences and on its website. However, FRA’s strategic plan for research and development, which outlined how the agency shares research results and engages with internal and external stakeholders in support of FRA’s rail safety mission, expired in 2017. More specifically, this plan outlined key internal and external stakeholders and their roles—including labor and industry partners—and specific outreach strategies, such as holding periodic, public events to present FRA’s research and development. This plan also stated that FRA’s research provides the scientific and technological basis for its rulemaking and regulation enforcement and that effectively sharing the results of its studies increases the likelihood that its research will have “real world” impacts. According to FRA officials, the agency is currently updating its strategic plan for research and development, which will outline FRA’s goals and objectives for its research, and expects to finalize the plan by the summer of 2019. FRA does not have any other documented policies in place for how it will use or disseminate the results of its study. Federal internal control standards call for management to communicate quality information—using appropriate methods—both internally and externally in order to achieve an entity’s objectives and respond to risks. Further, our work on best practices for strategic planning has found that formulating specific strategies and linking them with goals and objectives is critical for agencies to achieve these goals and objectives. In addition, we previously identified generally accepted research standards for sound studies, standards that include presentation of results. These standards call for relevant stakeholders to be informed of research results and any recommendations upon completion of a study. The Transportation Research Board—a part of the National Academies of Sciences, Engineering, and Medicine which provides research-based solutions to improve transportation, among other things—has found that organizations that develop processes and a systematic approach to implementing research are more effective and efficient at applying research results. FRA’s study is a first step for determining how, if at all, makeup and handling for longer trains as well as their crews’ needs may differ from shorter trains. If study results are effectively shared with relevant stakeholders, then those best situated to act on the results may be more likely to do so. For example, FRA officials—who have rulemaking and enforcement authority—could identify and implement changes needed to improve the safety of longer train operations, such as by issuing relevant guidance, rulemaking, or other actions. Similarly, external stakeholders, such as Class I railroads and workers, would have the opportunity to use study results to inform their practices and policies, such as making changes to internal train-makeup rules or operators’ training programs for longer trains. As FRA updates its strategic plan for research and development, formulating specific strategies for how it will share its research results with internal and external stakeholders would help to ensure FRA is in the best position to achieve its research goals and objectives in support of the agency’s mission of enabling the safe, efficient, and reliable transportation of people and goods. While FRA is taking steps to assess operational safety risks of longer trains through its study and other efforts, it is not assessing whether longer freight trains impact communities by blocking more grade crossings. Safety at grade crossings has been a longstanding issue in the United States, and according to stakeholders we spoke with, some of these issues may be exacerbated by longer trains. In 2006, as part of its report on the impacts of blocked grade crossings on emergency response services, FRA stated that future growth in rail and highway traffic will likely increase blocked crossings, and more recently FRA officials stated that this is still the case. In addition, while collisions at grade crossings have declined over time, FRA also expects the risk of grade-crossing incidents to grow as both rail and highway traffic increase during the next decade. However, FRA officials also stated that there is no evidence that more blocked crossings results in more grade-crossing incidents. Further, according to FRA, the agency is not in a position to address community- specific public safety issues. We have previously reported that the amount of time that grade crossings are blocked depends on a number of factors and is typically a function of the number, speed, and length of trains. Although there are no federal regulations directly addressing blocked grade crossings, to gauge the extent of reported instances of blocked crossings, in early 2018, FRA began to track data on the location of blocked-crossing complaints from state rail-safety managers in nine states. FRA officials stated they intend to use this data to identify communities where frequently blocked grade crossings are reported and work with the railroads and communities for resolution. However, FRA officials said they do not plan to explore any potential impacts of longer trains on grade crossings in communities, as FRA officials have stated they do not believe that longer trains are having an impact on blocked crossings. For example, FRA does not plan to use any of the information gathered in its longer train study—which will include a sampling of the routes longer trains travel—to inform the agency’s work on blocked crossings because FRA officials stated that they do not expect the study will yield relevant information. State and local officials we spoke with, as previously mentioned, expressed concerns about the potential for longer trains to increase the number of blocked grade crossings, causing delays for emergency responders and affecting the behavior of motorists and pedestrians. Federal internal control standards state that effective use of information and communication are vital for an entity to achieve its objectives. These standards call for management to use quality information—relevant, reliable information that is current, complete, accurate, accessible, and timely—to achieve an agency’s objectives and respond to risks. Further, we previously identified essential practices for agencies to help manage risks and identify opportunities that could impact the achievement of agencies’ goals. These risk-management practices call on agencies to systematically identify risks and use the best information available to assess them. Community officials acknowledged that while they believe longer trains are making blocked crossings worse, they do not have access to information needed to confirm this observation. As previously discussed, some local communities continue to face challenges after reaching out to FRA and Class I railroads to find solutions to issues related to grade crossings. As these issues continue to evolve and FRA works to identify locations where blocked crossings are reported, working with railroads and local communities to identify any potential impacts of longer trains on grade crossings would help FRA to determine whether and how longer trains are affecting these communities and what could be done to address those impacts. In addition, it would allow FRA to determine whether it should take additional action to ensure that longer trains are operating safely and to work with railroads to minimize their impact to the communities through which they travel. FRA faces a challenging task in assessing the safety impacts of longer trains and has taken some important steps to collect needed information through its study of longer trains’ operations. However, without documented strategies for how it plans to communicate the results of its research, FRA may lose an opportunity to effectively work with internal and external stakeholders—such as railroads, railroad workers, and local communities—to address any risks of operating longer trains in support of the agency’s mission of enabling the safe, efficient, and reliable transportation of people and goods. In addition, local community officials we spoke with raised concerns that longer trains are creating safety risks by causing emergency response delays and exacerbating dangerous motorist and pedestrian behavior, but acknowledged that they lack access to information on longer trains. FRA, however, is uniquely positioned to assess whether these concerns have merit. As FRA has stated, it expects that future growth in rail and highway traffic will increase incidences of blocked crossings and the risk of grade-crossing incidents. As traffic continues to grow—including railroads’ potential increased use of longer trains—having better information could be useful to FRA and other stakeholders. Without examining the potential impacts of longer trains on local communities, including on blocked grade crossings, FRA may lose an opportunity to identify what, if any, additional actions should be taken to ensure the safety of longer trains and the communities through which they travel. We are making the following two recommendations to FRA: The Administrator of FRA should develop a strategy for sharing FRA’s research results with internal and external stakeholders and implement that strategy for its research on the safety impacts of very long trains. (Recommendation 1) The Administrator of FRA should work with railroads to engage state and local governments to (a) identify community-specific impacts of train operations, including longer trains, where streets and highways cross railroad rights-of-way and (b) develop potential solutions to reduce those impacts. (Recommendation 2) We provided a draft of this report to DOT, NTSB, and STB for their review and comment. In its comments, reproduced in appendix I, DOT concurred with the recommendations. DOT and STB also provided technical comments, which we incorporated as appropriate. NTSB had no comments. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Transportation, the Chairman of NTSB, the Chairman of STB, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or FlemingS@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Andrew Huddleston (Assistant Director); Jean Cook (Analyst in Charge); Jason Coates; Philip Farah; David Hooper; Rosa Leung; Gail Marnik; John Mingus; Madhav Panwar; Malika Rice; Kelly Rubin; and Michelle Weathers made key contributions to this report.", "summary": "In 2017, the U.S. freight rail system moved over 1.5-billion tons of goods. The largest freight railroads—Class Is—dominate the industry and account for more than 90 percent of its annual revenue. In recent years, railroad workers and local communities have expressed safety concerns related to longer freight trains, and recent accidents involving such trains are currently under investigation by FRA. FRA does not currently place limits on freight train length. GAO was asked to review the safety and other impacts of longer freight trains. This report examines: (1) changes in freight train length over time, (2) safety considerations for operating longer freight trains, and (3) the extent to which FRA is assessing any safety risks. GAO reviewed relevant statutes, regulations, and federal agencies' reports and plans; analyzed available data on freight train length from railroads; and interviewed federal officials and various stakeholders, including state and local officials and first responders from five states (selected to represent different railroads and regions), and officials from the railroad industry, unions, and advocacy groups. Freight train length has increased in recent years, according to all seven Class I freight railroads. Data on train length are not publicly available; however data provided to GAO by two Class I railroads indicated that their average train length has increased by about 25 percent since 2008, with average lengths of 1.2 and 1.4 miles in 2017. Officials from all seven Class I railroads said they are currently operating longer than average trains on specific routes, although some said such trains are a small percentage of the trains they operate. One railroad said it runs a 3-mile-long train twice weekly. Officials identified increased efficiencies and economic benefits among the advantages of longer freight trains. Stakeholders said that the arrangement of train cars and locomotives—known as “train makeup”—and the potential for blocking highway-railroad crossings are issues to consider to safely operate longer freight trains. To prevent derailment, stakeholders said it is important that longer trains are arranged appropriately and that crews are trained to operate them. While Class I railroads and others said that longer trains may decrease the frequency of blocked crossings, some state and local officials said these trains can prolong their duration, posing challenges for emergency responders unable to cross the tracks. The Federal Railroad Administration (FRA) is studying the safety risks of and strategies for operating longer trains. As part of the study, FRA plans to analyze train-handling and braking capabilities under varying conditions. FRA officials said they plan to share their research results with relevant stakeholders; however, FRA currently has no documented strategy for sharing the results of its research. FRA officials are also analyzing which parts of the country are reporting frequently blocked crossings. However, FRA officials said they do not plan to use information from either of these efforts to determine whether longer freight trains might contribute to increases in blocked crossings, and the officials believe the issues are unrelated. Developing and implementing a strategy for sharing FRA's research results and identifying any potential impacts of longer freight trains on highway-railroad crossings would enable FRA and stakeholders to better determine what, if any, actions are needed to ensure the safe operation of longer freight trains. GAO recommends that FRA develop and implement a strategy to share the results of its study on longer trains and work with railroads to engage state and local governments to identify and reduce impacts of longer freight trains on highway-railroad crossings. FRA concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "Enacted on January 23, 1995, the CAA, as amended, applies 13 federal civil rights, workplace, and labor laws to legislative branch employees who were previously exempted from such coverage. Table 1 lists the 13 laws included under the CAA. The CAA contained a series of specific requirements for the Office of Compliance to meet as it carried out its responsibility to administer and enforce the act. Toward this end, the Office of Compliance took a number of actions, such as administering a dispute resolution process; conducting investigations and inspections to ensure compliance with safety, health, and disability access standards; investigating and managing matters concerning labor management relations, and educating both employees and employing offices about their rights and responsibilities under the CAA. The Reform Act expanded the office’s duties and responsibilities, as well as the number of employees covered by the CAA. These new duties and responsibilities include, among other things: changing the name of the office to OCWR; substantially modifying the administrative dispute resolution process under the CAA, including creating additional procedures for preliminary hearing officer review of claims; appointing one or more advisers to provide confidential information to legislative branch employees about their rights under the CAA; extending CAA protections to unpaid staff, including interns, detailees, and fellows, as well as previously unprotected legislative branch employees; conducting a workplace climate survey; significantly expanding OCWR reporting obligations; creating a program to permanently retain records of investigations, mediations, hearings, and other proceedings; and establishing an electronic system to receive and keep track of claims. The act mandated that OCWR institute some of these requirements, such as changing the name of the office, immediately. Other requirements, such as establishing an electronic system to receive and keep track of claims, were to be met no later than 180 days after the implementation of the act, or by June 19, 2019. To implement its statutory requirements, OCWR currently has 28 full-time equivalent positions, which includes five part-time members of OCWR’s Board of Directors (counted as one full-time equivalent) appointed by congressional leadership. This represents an increase of five full-time equivalents since April 2018. OCWR relies extensively on IT services and systems provided by external parties to support its mission-related operations and protect claims data. For example, the Library provides network and end-user computing services for OCWR, including email; network services such as Internet access and file sharing; and end-user services and support, such as desktop support and software management. OCWR also relied on an external contractor to develop and maintain its legacy claims management system, known as the Case Management System (CMS). Since 2014, the office used CMS to manage claims submitted by covered legislative branch employees using one of four ways: in person at OCWR’s office; or by mail, email, or fax. After a claim was received, an OCWR employee would manually enter the claim information into CMS and update the information as it progressed through the dispute resolution process. In response to the Reform Act enacted in December 2018, OCWR initiated the SOCRATES project to meet the requirement of implementing an electronic system for claims. SOCRATES is intended to enable covered legislative branch employees to file a claim via a web-based form, and an OCWR employee to electronically manage the workflow of claims as they progress through the dispute resolution process. Specifically, the system is expected to maintain and track claim deadlines, generate correspondence, as well as update and store claim information. OCWR relied on both the Library and an external contractor to upgrade CMS to SOCRATES. As part of its SOCRATES implementation efforts, OCWR first moved the CMS application and claim data from its office to the Library, which began hosting the system in April 2019. Between April 2019 and June 2019, OCWR’s external contractor continued work to develop and implement new and updated components for CMS to facilitate the electronic filing and management of claims. In addition, the external contractor worked to develop and implement the web-based form to electronically capture claims. According to OCWR, SOCRATES is comprised of three components that are hosted by the Library: SOCRATES web-based form: This form is intended to be used by covered legislative branch employees to submit a claim alleging a violation of civil rights, workplace, or labor laws during their employment. Secure information sharing platform: This platform is intended to be a web-based, secure workflow file collaboration application. The platform allows for the sharing of claim related information between OCWR, the covered employee, the employee’s office, and any other relevant parties (e.g., employee representatives). SOCRATES internal CMS console: Based on updated functionality from OCWR’s CMS, this console is intended to provide secure, detailed workflow management of each claim that is submitted. Specifically, the console introduces new workflows based on the Reform Act’s updated requirements for a claim and allows OCWR employees to internally manage a claim. Figure 1 shows the updated claim filing process using SOCRATES. According to OCWR, testing of SOCRATES the week prior to its June 19, 2019, due date revealed numerous problems with the system. For example, if a user did not submit his or her claim within a certain amount of time, the system refreshed the webpage without saving the user’s data, forcing the user to restart the claim. As a result, OCWR delayed the deployment 7 days to allow time to resolve this issue and others. On June 26, 2019, OCWR deployed SOCRATES and began accepting claims via the web-based form. In addition to SOCRATES, OCWR relies on the external contractor to provide hosting and application support for FMA. FMA is used by OCWR to document reported violations of the Occupational Safety and Health Act. The CAA requires OCWR to conduct biennial inspections of the legislative branch to ascertain compliance with the act and to report its findings to Congress. The office also reports its findings to the legislative branch agency that is reportedly in violation of the act in a Hazard Summary Report. The agency is responsible for responding, and providing verification of the abatement of violations and hazards documented in the findings, to OCWR. IT systems supporting federal agencies are inherently at risk. These systems are highly complex and dynamic, technologically diverse, and often geographically dispersed. This complexity increases the difficulty in identifying, managing, and protecting the numerous operating systems, applications, and devices comprising the systems and networks. Compounding the risk, federal systems and networks are also often interconnected with other internal and external systems and networks, including the internet. This increases the number of avenues of attack. Information and systems are subject to serious threats that can have adverse impacts on organizational operations and assets, individuals, other organizations, and the nation. These threats can include purposeful attacks, environmental disruptions, and human/machine errors, and may result in harm to the national and economic security interests of the United States. In recognition of the growing threat, we have designated information security as a government-wide high-risk area since 1997. In 2003, we expanded the information security high-risk area to include the protection of critical cyber infrastructure. We further expanded the information security high-risk area in 2015 to include protecting the privacy of personally identifiable information. Cybersecurity incidents continue to impact federal agencies, including those entities in the federal executive and legislative branch. For example, in fiscal year 2017, federal executive branch civilian agencies reported 35,277 incidents to the U.S. Computer Emergency Readiness Team. These incidents included web-based attacks, phishing, and the loss or theft of computing equipment. These incidents and others like them can pose a serious challenge to economic and national security and personal privacy. The following examples highlight the impact of incidents from legislative and executive branch entities: In January 2019, the Department of Justice announced that it had indicted two Ukrainian nationals for their roles in a large-scale, international conspiracy to hack into the Securities and Exchange Commission’s computer systems and profit by trading on critical information they stole. The indictment alleges that the two hacked into the commission’s Electronic Data Gathering, Analysis, and Retrieval system and stole thousands of files, including annual and quarterly earnings reports containing confidential, nonpublic, financial information, which publicly traded companies are required to disclose to the commission. In July 2016, the Library announced that it had experienced a significant distributed denial-of-service attack that affected multiple internal and external Library systems and services. Specifically, the attack successfully disrupted services to multiple Library systems and services including email, databases, and public web domains, such as Congress.gov. According to the Library, the attack was sophisticated in both the size of the attack and methods that the attack employed. In June 2015, the Office of Personnel Management reported that an intrusion into its systems had affected the personnel records of about 4.2 million current and former federal employees. Then, in July 2015, the agency reported that a separate, but related, incident had compromised its systems and the files related to background investigations for 21.5 million individuals. In total, the Office of Personnel Management estimated that 22.1 million individuals had some form of personally identifiable information stolen, with 3.6 million being a victim of both breaches. Recognizing the importance of information security and privacy, Congress enacted the Federal Information Security Modernization Act of 2014 (FISMA), which requires federal agencies in the executive branch to develop, document, and implement an information security program and to evaluate the program for effectiveness. The act retains many of the requirements for federal agencies’ information security programs previously set by the Federal Information Security Management Act of 2002. As legislative branch entities, OCWR and the Library are not subject to FISMA. However, OCWR’s Executive Director and the Library’s Chief Information Officer have chosen to follow aspects of the law’s requirements. For example, an interagency agreement between OCWR and the Library describes plans to protect OCWR’s CMS application and claim data using NIST guidance that is intended to satisfy FISMA requirements and relates to managing risks to the information system. The 2002 act also assigns certain responsibilities to NIST, which is tasked with developing standards and guidelines for systems other than national security systems. These standards and guidelines must include, at a minimum, (1) standards to be used by all agencies to categorize all of their information and information systems based on the objectives of providing appropriate levels of information security, according to a range of risk levels; (2) guidelines recommending the types of information and information systems to be included in each category; and (3) minimum information security requirements for information and information system in each category. Accordingly, NIST developed a risk management framework of standards and guidelines for agencies to follow in developing information security programs. The framework addresses broad information security and risk management activities to be followed in developing information systems, including categorizing the system’s impact level; selecting, implementing, and assessing security controls; authorizing the system to operate (based on progress in remediating control weaknesses and an assessment of residual risk); and monitoring the efficacy of controls on an ongoing basis. In December 2019, we reported that OCWR faced management challenges in implementing its new requirements under the Reform Act, such as establishing a program to permanently retain records of investigations, mediations, hearings, and other proceedings. Specifically, we determined that OCWR did not always use project schedules to manage the implementation of the requirements of the Reform Act. For example, we noted that the office used a project schedule for developing the workplace climate survey, but did not use a project schedule to manage the SOCRATES project. We also determined that OCWR did not address risks associated with its records retention program. For example, we noted that the office had not yet developed policies and procedures to address the risks associated with permanently retaining sensitive records, such as ensuring they remain confidential when stored in multiple locations. Our report also identified weaknesses in OCWR’s IT planning, including that the office did not develop long-term strategies for recruiting and retaining staff with critical skills and competencies needed to achieve current and future agency goals. Accordingly, our report included six recommendations for the office related to incorporating key management practices into project planning and ensuring that it has the necessary skills and capacity to meet its mission. OCWR agreed with our recommendations and described plans to address them. We have also previously reported on weaknesses with the Library’s information security program, as well as specific security controls that support OCWR’s systems and services. In March 2015, we issued a report that identified weaknesses in the Library’s information security program. We made 10 recommendations to the Library aimed at better protecting IT systems and reducing the risk that the information they contain will be compromised. These recommendations included, among other things, developing contingency plans for all systems and conducting comprehensive and effective security testing for all systems within the time frames called for by Library policy. The Library generally agreed with our recommendations and described planned and ongoing actions to address them. As of January 2020, the Library fully implemented nine of the 10 recommendations and has taken steps to implement the remaining recommendation. We have work underway to determine whether the steps taken by the Library fully address the remaining recommendation. In a related June 2015 limited official use only report, we made 74 detailed security recommendations aimed at addressing specific weaknesses in the Library’s security controls. The Library generally agreed with our security recommendations and described planned and ongoing actions to address them as well. As of January 2020, the Library had fully implemented 72 of 74 detailed security control recommendations from this report and had plans to implement the remaining two recommendations by February 2020. Effectively managing a project entails, among other things, developing a project schedule, defining and managing requirements, and effectively managing project risks. Project scheduling. The success of a program depends, in part, on having an integrated and reliable master schedule that defines, among other things, when work activities will occur, how long they will take, and how they relate to each other. A reliable schedule provides a road map for systematic execution of a program and a means by which to gauge progress, identify and address potential problems, and promote accountability. GAO’s Scheduling Assessment Guide lists 10 best practices associated with a high-quality and reliable schedule, including capturing and sequencing all activities, as well as establishing the duration of all activities. Requirements management. Requirements establish what the system is to do, how well it is to do it, and how it is to interact with other systems. The Software Engineering Institute’s Capability Maturity Model Integration® for Acquisition (CMMI-ACQ) and Capability Maturity Model Integration® for Development (CMMI- DEV) note that requirements management processes are important for enabling programs to ensure that their set of approved requirements is managed to support planning and execution needs. This should include steps to obtain stakeholder’s review and commitment to the requirements and to manage changes to requirements as customer needs evolve. Project risk management. The discipline of risk management is important to help ensure that projects are delivered on time, within budget, and with the promised functionality. According to leading practices for acquisition, the purpose of risk management is to identify potential issues that could endanger achievement of critical objectives before they occur. A continuous risk management approach effectively anticipates and mitigates risks that can have a critical impact on a project. Organizations that plan to acquire IT products and services for a project should also identify and assess risks associated with the acquisition process. Incorporating cybersecurity management activities (such as the selection and implementation of security controls) into each of these project planning areas can help to reduce cybersecurity risks and better protect critical assets. For example, according to NIST’s risk management framework, integrating system security requirements into a project’s planning activities, such as scheduling, can help to ensure that resources are available when needed and that project milestones are met. In addition, the framework notes that defining the system security requirements early and integrating them with other system requirements can result in a system having fewer deficiencies, and therefore, fewer security vulnerabilities that can be exploited in the future. The framework also describes the importance of identifying security risks early in a system project and addressing such risks on an ongoing basis. However, OCWR did not effectively manage the SOCRATES project because it did not establish a schedule, develop and manage requirements, and manage risks. Consequently, the office did not incorporate key cybersecurity management activities into each of these project planning areas. Specifically: OCWR did not manage the SOCRATES project using an established, approved project schedule that identified when cybersecurity activities would be completed. As discussed earlier, we previously reported that OCWR did not establish a project schedule to manage the SOCRATES project. Although the office drafted a project schedule in January 2019, this schedule was not finalized and used during the project. According to OCWR’s Director of the IT Governance, Risk Management, and InfoSec Compliance Program, the schedule was not used due to, among other things, challenges encountered in managing the interdependencies of SOCRATES development with the implementation of other Reform Act requirements (e.g., modifying the administrative dispute resolution process). Consequently, OCWR did not use a project schedule to manage key SOCRATES cybersecurity activities, including those to be completed by OCWR, the Library, and the contractor. To its credit, the Library provided an early project schedule with certain cybersecurity activities they performed related to CMS. For example, the Library’s project schedule documented initial activities the Library was to perform that related to procurement of equipment, installation of software, security testing, and vulnerability remediation in order to move CMS from OCWR to the Library. However, OCWR did not use a project schedule for the upgrade of CMS to SOCRATES that included the time frames for key cybersecurity management activities, such as selecting and documenting security controls, implementing controls, and assessing controls. The lack of a project schedule likely hindered OCWR’s ability to respond to changes during the project and execute key cybersecurity management activities in a timely manner. For example, in May 2019, OCWR made a decision to use a locally hosted platform at the Library for its secure information sharing platform instead of a cloud-based solution. Without a project schedule, OCWR was unable to assess the impact of this late change on the time available for completing remaining cybersecurity management activities. OCWR did not establish a requirements management process or develop a set of detailed system requirements, including cybersecurity requirements. OCWR did not establish a requirements management process that included steps to obtain stakeholders’ review and commitment to the requirements and to manage changes to the requirements. Instead, the office established a set of business flow diagrams, which identified how claim information would move within OCWR and SOCRATES. Further, OCWR did not establish a set of detailed system requirements, including the cybersecurity requirements (e.g., what cybersecurity controls were to be implemented). OCWR did not document and manage risks to the SOCRATES project, including those related to cybersecurity. OCWR did not document and manage risks for the SOCRATES project. Specifically, the office did not document and manage risks related to cybersecurity and did not mitigate those risks that could have had a critical impact on the project. For example, OCWR was not able to ensure that the Library tested all moderate-level security controls for the SOCRATES web-based form and secure information sharing platform before the system was deployed. However, this was not documented or managed by OCWR as a risk. In addition, as discussed later in this report, there were also risks associated with OCWR’s reliance on the Library and its external contractor that were implementing cybersecurity responsibilities on its behalf. For example, we identified shortfalls in the OCWR’s oversight of the planning and conducting of system security assessments. However, no risks related to the office’s reliance on external parties were documented or managed throughout the project. According to the Director of the IT Governance, Risk Management, and InfoSec Compliance Program, the office did not complete key project planning activities and documentation, in part, because of the compressed time frame associated with the project and the need to complete it by its mandated June 19, 2019, completion date. In aiming to meet this date, the OCWR official added that they held frequent discussions with the contractor and made changes “on the fly” to ensure that OCWR met the mandate. However, frequent discussions with the contractor does not negate the need to document and manage cybersecurity activities using leading project planning practices, including a project schedule, a requirements management process, and a risk management process. OCWR’s project management weaknesses also occurred, in part, because the office lacked policies and procedures for IT project scheduling, requirements management, and risk management. Such policies and procedures are critical to have in place as OCWR plans future IT projects. For example, as of October 2019, the office was planning to move its other key system, FMA, to the Library in 2020. Until OCWR develops and implements policies and procedures for incorporating cybersecurity management activities into its IT project planning using a project schedule, a requirements management process, and a risk management process, it will continue to have a limited ability to effectively manage and monitor the completion of cybersecurity activities and will face increased cybersecurity risks. The responsibility for adequately mitigating risks arising from the use of externally-operated systems remains with the agency itself. NIST Special Publications 800-53 and 800-53A guide agencies in selecting security and privacy controls for systems and assessing them to ensure that the selected controls are in place and functioning as expected. Additional NIST special publications on IT security services and risk management (Special Publications 800-35 and 800-37) identify several key activities important for assessing the security and privacy controls of information systems operated by external entities. The key activities and the steps included in NIST Special Publications 800-35 and 800-37 are shown in table 2. For the two selected systems—SOCRATES and FMA—OCWR either partially implemented, or did not implement, system oversight activities. Table 3 details the extent to which OCWR implemented system oversight activities and is followed by a discussion of each activity. Establish security and privacy requirements. OCWR partially implemented this oversight activity for both SOCRATES and FMA. Communicate requirements to external entities. OCWR communicated certain security and privacy requirements to its external partners for these two systems. For example, the office’s agreements with the Library for SOCRATES stated that the system will be secured in accordance with NIST security guidelines, including Special Publication 800-37, and the Library’s security policy guidelines. However, OCWR did not always include language in agreements in sufficient detail to ensure that requirements were communicated effectively. For example, the office did not always provide sufficient language to communicate privacy requirements related to the protection of personally identifiable information within its SOCRATES or FMA agreements. Further, OCWR’s agreements—related to FMA—expired during our review and contained references to retired Library guidelines that are no longer applicable or enforceable with regard to OCWR’s external contractor. Select and document security and privacy controls. OCWR worked with the Library to select and document about 300 security and privacy controls and control enhancements for SOCRATES within a system security plan. Further, the office worked with the Library to support the selection of controls by documenting privacy risks and impacts to SOCRATES within a privacy impact assessment—as called for by NIST to assess the privacy risks associated with collecting and using personal information—that was referred to in the system security plan. However, OCWR did not adequately oversee the selection and documentation of security and privacy controls in the system security plan that was used to plan and conduct initial control assessments for SOCRATES. In particular, the office did not always ensure that the system security plan for SOCRATES provided an appropriate description of controls to be implemented to meet the security and privacy requirements. For example, in certain instances, the system security plan described SOCRATES as a low-impact system when describing the security controls used to protect the system. These descriptions differed though from its actual classification as a moderate-impact system, as documented within an interagency agreement between OCWR and the Library. As another example, the system security plan for SOCRATES incorrectly described a security control related to the maintenance of SOCRATES as not applicable to moderate- impact systems. However, NIST’s classification of this control describes it as applicable to moderate-impact systems. For the FMA system, OCWR relied on its external contractor to document a system security plan that generally described security requirements for the system. However, the plan did not document the privacy requirements or the specific security and privacy controls that were expected to be implemented for FMA as a low-impact system. For example, the plan did not specify an authority to report information to in the event of a security incident. Further, the plan did not include or refer to other necessary security and privacy documentation, such as a privacy impact assessment. As a result, OCWR did not adequately oversee the completion of this key step for its FMA system. Plan assessment of security controls. OCWR partially implemented this oversight activity for SOCRATES and did not implement it for FMA. Select an independent assessor. OCWR relied on the Library to select an assessor for SOCRATES who was independent. For example, for SOCRATES, the Library used an external contractor to initially assess the system and reported taking steps to verify that the assessor was independent from the Library. However, the office did not adequately oversee the completion of this key step for SOCRATES and did not ensure that the assessor used for the system was independent from the office. Specifically, OCWR allowed the Library to select the assessor for SOCRATES and did not take steps to verify the assessor’s independence. Further, for FMA, OCWR did not select an assessor to review the system. Develop a test plan. Although OCWR relied on the Library to develop a test plan for SOCRATES, the test plan used to conduct initial control testing was not approved by the office and did not specify the procedures that were to be followed to test each control from the SOCRATES system security plan. For example, the SOCRATES test plan specified a high-level procedure for collecting relevant artifacts but did not specify what particular documentation would be collected or reviewed for each control identified in the system security plan. Regarding FMA, OCWR and its external contractor did not develop a test plan. Conduct assessment. OCWR partially implemented this oversight activity, which includes executing the test plan, for SOCRATES and did not implement it for FMA. Specifically, OCWR worked with the Library to perform initial control testing for SOCRATES and document the results in an online tracking system; however, as previously mentioned, the office did not ensure that a test plan with detailed procedures to test each control was developed and approved prior to the initial testing of SOCRATES. As a result, the office did not adequately oversee the execution of the test plan by the Library to ensure that controls that were assessed as implemented were effectively operating as intended. For FMA, OCWR and its external contractor did not execute a test plan or document the results of any tests for the system. Review assessment. OCWR partially implemented this oversight activity, which includes developing POA&Ms for remediation of weaknesses, for SOCRATES and did not implement it for FMA. Specifically, OCWR worked with the Library to develop POA&M data for SOCRATES that included many of the recommended NIST elements, such as estimated completion dates and issue identification. For example, following initial control testing in March 2019, OCWR and the Library worked to develop POA&M data for 62 security control weaknesses, including 24 high-risk and 38 moderate- risk weaknesses. As of November 2019, there were seven POA&Ms, including six categorized as high-risk and one as moderate-risk, that OCWR and the Library had not yet addressed. However, as previously mentioned, the office did not ensure that a test plan that included detailed procedures to test each control was developed and approved prior to the initial testing of SOCRATES. Therefore, the office could not ensure that controls were tested appropriately to identify necessary remedial actions in POA&Ms. As a result, OCWR did not adequately oversee the completion of this step and ensure that key POA&Ms were appropriately documented. For FMA, without an executed test plan, OCWR and its external contractor could not complete or update POA&Ms for the system. According to OCWR officials, including the office’s Deputy Executive Director, part of the reason for these shortfalls was that the office did not obtain expertise in security to aid in the completion of these oversight activities until September 2018 when the office hired a new IT Manager. In addition, OCWR officials, including the Deputy Executive Director, could not explain why the contractor did not produce key oversight related artifacts, such as those related to the security testing of controls, as agreed upon in contracts covering FMA during the performance period. However, a key contributing reason that we identified for the shortfalls in OCWR’s oversight of external partners was that OCWR had not documented procedures to direct the office in performing such oversight activities effectively. The lack of documented oversight procedures and shortfalls in OCWR’s oversight of its external partners contributed to concerns with the deployment of SOCRATES. For example: As previously discussed, OCWR did not ensure that all moderate- level security controls for the SOCRATES web-based form and secure information sharing platform were tested before the system was deployed in June 2019. For example, a control related to testing contingency plans for the SOCRATES web-based form was not assessed until August 2019, approximately 2 months after the system was deployed. Although penetration testing of the CMS portion of SOCRATES was completed in May 2019, OCWR did not ensure that penetration testing of the SOCRATES web-based form and secure information sharing platform was conducted before deployment. Penetration testing for the SOCRATES web-based form and secure information sharing platform was subsequently completed in December 2019, approximately 6 months after the system was deployed. Until OCWR develops and implements effective oversight procedures over its external partners, it may not be able to mitigate risks that could result in the loss of sensitive data or compromise of the office’s external systems. We also assessed selected security controls in place for SOCRATES and FMA including, but not limited to, configuration management, patch management, and personnel security. We intend to issue a separate limited official use only report that discusses the results of this review. NIST’s cybersecurity framework is intended to support federal agencies as they develop, implement, and continuously improve their cybersecurity risk management programs. In this regard, the framework identifies cybersecurity activities for achieving specific outcomes over the lifecycle of an organization’s management of cybersecurity risk. According to NIST, the first stage of the cybersecurity risk management lifecycle—which the framework refers to as “identify”—is focused on foundational activities for effective risk management that provide agencies with the organizational understanding to manage cybersecurity risk to systems, assets, data, and capabilities. Additional NIST guidance, including its risk management framework, provides information on implementing foundational activities and achieving desired outcomes that calls for, among other things, the following: A risk executive in the form of an individual or group that provides agency-wide oversight of risk activities and facilitates collaboration among stakeholders and consistent application of the risk management strategy. This functional role helps to ensure that risk management is institutionalized into the day-to-day operations of organizations as a priority and integral part of carrying out missions. A cybersecurity risk management strategy that articulates how an agency intends to assess, respond to, and monitor risk associated with the operation and use of the information systems it relies on to carry out the mission. The strategy should, among other things, make explicit an agency’s risk tolerance, accepted risk assessment methodologies, a process for consistently evaluating risk across the organization, risk response strategies, approaches for monitoring risk over time, and priorities for investing in risk management. Risk-based policies and procedures that act as the primary mechanisms through which current security requirements are communicated to help reduce the agency’s risk of unauthorized access or disruption of services. If properly implemented, these policies and procedures may be able to effectively reduce the risk that could come from cybersecurity threats such as unauthorized access or disruption of services. For example, establishing policies and procedures that incorporate NIST’s risk management framework can help to ensure that a consistent approach is used to conduct a complete security assessment before a system is deployed and that a designated agency official certifies the system for operation based on progress in remediating control weaknesses and an assessment of residual risk. To its credit, OCWR’s strategic plan for fiscal years 2019 through 2023 includes a goal of developing, among other things, cybersecurity risk policies and procedures. The strategic plan also describes the office’s plans to ensure compliance with applicable IT and cybersecurity standards. Nevertheless, OCWR has not yet fully established an effective approach to organization-wide cybersecurity risk management that includes foundational elements. Specifically, although the office’s Director of the IT Governance, Risk Management, and InfoSec Compliance Program stated that he was serving as the risk executive, this role and its related responsibilities are not documented in OCWR’s policies. In addition, OCWR has not developed an organization-wide cybersecurity risk management strategy or determined a time frame for when the policies and procedures discussed in its strategic plan will be implemented. According to the Director of the IT Governance, Risk Management, and InfoSec Compliance Program, the reason for these shortfalls in risk management was that the office’s top priority was completing work on the SOCRATES system, and then it planned to work on its cybersecurity policies and procedures. Additionally, the official stated that OCWR considers development of documentation to be a continual process, and that the office would like to develop and build procedures to lay a foundation for effective risk management. However, until OCWR establishes the role and responsibilities of the risk executive function in policy, the office will lack an understanding of who is ultimately responsible for overseeing the cybersecurity risk activities of the organization and what those responsibilities include. Further, until OCWR establishes and implements a strategy for managing its cybersecurity risks using NIST’s framework, its ability to make operational decisions that adequately address security risks and prioritize IT security investments will be hindered. Finally, until OCWR establishes a time frame for developing and implementing risk-based policies and procedures, it will lack assurance that consistent steps are being taken to categorize systems; select, implement, and assess system security controls; and make risk-based decisions on authorizing systems to operate. Although OCWR completed the upgrade of its legacy claims management system through the SOCRATES project, the office did not incorporate cybersecurity activities into the project during planning. As a result, OCWR was left without a complete understanding of potential schedule issues, the system’s planned security requirements, and cybersecurity- related risks to the success of the project. These shortcomings existed, at least in part, because of a lack of OCWR policies and procedures that required cybersecurity management activities be incorporated into project scheduling, requirements management, and risk management. Until OCWR develops and implements such policies and procedures, future IT projects—such as the office’s planned transition of its FMA system to the Library—may face unnecessary cybersecurity risks and may not be carried out in an efficient and effective manner. OCWR made initial efforts to assess the implementation of security and privacy controls for the two selected externally-operated systems, but did not fully implement critical oversight activities. A contributing reason for these shortfalls is that OCWR had not documented procedures for the office to follow in order to perform such oversight of its external entities effectively. This ultimately contributed to OCWR not being able to first test important system security controls for ensuring the confidentiality, integrity, and availability of the system before it was deployed. Until OCWR establishes and implements specific procedures for overseeing external entities, it will have reduced assurance that external entities are adequately securing and protecting the office’s information. In addition, the office will face increased risks that system weaknesses may go undetected and unresolved, which could result in the loss of sensitive data or compromise of its systems. Given the increasing number and sophistication of cyber threats facing federal agencies, it is critical that organizations such as OCWR are well positioned to make consistent, informed risk-based decisions in protecting their systems and information against these threats. To its credit, OCWR has recognized the need for an improved organization-wide approach to its cybersecurity policies and IT governance in its most recent strategic plan. However, important elements of an effective organization-wide cybersecurity approach have not been fully implemented, including establishing the roles and responsibilities for the risk executive function in policy, a cybersecurity risk management strategy, and policies and procedures for managing cybersecurity risks. Until OCWR fully addresses these organization-wide cybersecurity risk management practices, its ability to ensure effective oversight and management of IT will remain limited. Moreover, OCWR may be limited in its ability to strengthen its risk posture, including ensuring effective cybersecurity across its relationships with external entities that are critical to its ability to provide IT services and systems needed to meet its mission. We are making five recommendations to the Office of Congressional Workplace Rights: The Executive Director should ensure the development and implementation of policies and procedures for incorporating key cybersecurity activities into IT project planning, including scheduling, requirements management, and risk management. (Recommendation 1) The Executive Director should ensure the development and implementation of oversight procedures for each externally-operated system that include establishing security and privacy requirements, planning the assessment of security controls, conducting the assessment, and, reviewing the assessment. (Recommendation 2) The Executive Director should ensure the establishment of roles and responsibilities for a risk executive function. (Recommendation 3) The Executive Director should ensure the development and implementation of a cybersecurity risk management strategy. (Recommendation 4) The Executive Director should ensure commitment to a time frame for developing and implementing policies and procedures for managing cybersecurity risk. (Recommendation 5) We provided a draft of this report to OCWR, the Library, and the third- party contractor for review and comment. In response, we received written comments from OCWR, which are reproduced in appendix II. In its comments, the office did not state whether it agreed or disagreed with our recommendations, but described initial actions taken and planned to address them. Specifically, OCWR noted that it has initiated several actions, such as revising the office’s IT systems project planning to ensure the development and implementation of policies and procedures incorporating key cybersecurity activities. Further, OCWR stated that it intends to implement additional changes, such as developing and implementing oversight procedures for each externally-operated system. Going forward, OCWR stated that it intends to update us on its progress in implementing the recommendations. We also received technical comments from the Library’s Deputy Chief Information Officer via email, which we incorporated as appropriate. In addition, the third-party contractor indicated via email that it had no concerns about, and worked with OCWR in responding to, the draft report. We are sending copies of this report to the appropriate congressional committees, the Executive Director of the Office of Congressional Workplace Rights, the Librarian of Congress, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9342 or marinosn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. The objectives of our review were to examine the extent to which the Office of Congressional Workplace Rights (OCWR) (1) incorporated key cybersecurity management activities into the project planning for its claims management system upgrade, (2) performed oversight of security controls and mitigated risks for selected systems operated by external parties on its behalf, and (3) established an effective organization-wide approach for managing cybersecurity risk. To assess OCWR’s incorporation of key cybersecurity management activities into the project planning for its claim management system upgrade (known as the Secure Online Claims Reporting and Tracking E- filing System, or SOCRATES), we reviewed available OCWR project planning documentation related to establishing a project schedule, requirements management process, and risk management process. This documentation included, for example, a draft SOCRATES project schedule, contract information, and business flow diagrams. We then compared OCWR’s documentation to leading practices for project planning, including those identified by the Software Engineering Institute. Three key areas needed to effectively managing projects are developing a project schedule; managing project requirements; and managing project risks. We also analyzed OCWR’s available project planning documentation to determine the extent that it incorporated key cybersecurity management activities, as identified by the National Institute of Standards and Technology (NIST) risk management framework. These key activities are: obtaining a system categorization, selecting and implementing security controls, assessing security controls, obtaining an authority to operate, and monitoring of security controls. Further, we conducted interviews with OCWR officials, including the General Counsel and the Director of the Information Technology (IT) Governance, Risk Management, and InfoSec Compliance Program, to assess the extent to which the office incorporated key cybersecurity management activities into its SOCRATES project planning. To assess the extent to which OCWR performed oversight of security controls and mitigated risks for selected externally-operated systems, we chose two systems—SOCRATES and the Facility Management Assistant (FMA). We chose these two systems because they process and maintain OCWR’s most sensitive information, including claims related to alleged violations of employee rights and protections and reported occupational safety and health violations. We then collected and reviewed cybersecurity policies, procedures, and documentation (e.g., system security plans) from the office and its external partners that related to protecting the security and privacy of information and systems. To assess the reliability of the SOCRATES system security plan and its security control testing data obtained from the Library’s online repository, we reviewed related documentation (e.g., security assessment results briefings), reviewed the data for obvious omissions (i.e., fields left blank), and performed electronic testing to identify outliers. We also interviewed Library officials to discuss the reliability of the data. Based on our assessment, we determined that the data were sufficiently reliable for the purpose of our reporting objectives. We then examined whether OCWR and its external partners implemented—for each selected system—four oversight activities important for assessing the security and privacy controls of information systems operated by external entities, as specified in federal requirements and guidance, including NIST Special Publications 800-35, and 800-37. The four oversight activities we examined were: (1) establishing security and privacy requirements, (2) planning the assessment of security controls, (3) conducting the assessment, and (4) reviewing the assessment. We chose these activities because of their importance to providing effective oversight of systems operated by external entities. Further, we assessed whether OCWR implemented policies and procedures set forth by the office, including contractor oversight activities performed by the responsible official. We also conducted interviews with officials from OCWR, including the General Counsel, Deputy Executive Director, and Director of the IT Governance, Risk Management, and InfoSec Compliance Program. In addition, we also interviewed key personnel from OCWR’s external partners, such as the Library’s Deputy Chief Information Officer and the President of the external contractor, to assess the extent of OCWR’s oversight activities for SOCRATES and FMA. We assessed selected security controls in place for SOCRATES and FMA including, but not limited to, configuration management, patch management, and personnel security. We intend to issue a separate limited official use only report that discusses the results of this review. To assess OCWR’s efforts to establish an effective organization-wide approach for cybersecurity risk management activities, we used NIST’s cybersecurity framework, which identifies foundational components of effective cybersecurity risk management. We also used additional guidance provided by NIST for implementing the foundational components and achieving desired outcomes. These components included the establishment of a risk executive function, cybersecurity risk management strategy, and risk-based security policies and procedures. We then evaluated OCWR’s organization-wide cybersecurity risk management approach by, among other things, analyzing available policies and plans, management reports, and strategic planning documentation against the foundational cybersecurity risk management components identified in NIST guidance. Further, we conducted semistructured interviews with relevant OCWR officials with responsibilities for managing their efforts to establish an approach for managing cybersecurity risk, including the General Counsel and the Director of the IT Governance, Risk Management, and InfoSec Compliance Program. We conducted this performance audit from January 2019 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Jon Ticehurst (Assistant Director), Lisa Hardman (Analyst in Charge), Edward Alexander, Jr., Angela Bell, Christina Bixby, David Blanding, Hannah Brookhart, Kisa Bushyeager, Christopher Businsky, West Coile, Linda Erickson, Rebecca Eyler, Kaelin Kuhn, Sukhjoot Singh, Eugene Stevens, and Adam Vodraska made key contributions to this report. Giny Cheong, Edda Emmanuelli-Perez, Elizabeth Fan, Steven Lozano, Rebecca Woiwode, and Edith Yuh also provided valuable assistance.", "summary": "OCWR is an independent, nonpartisan office that administers and enforces various provisions related to fair employment, and occupational safety and health within the legislative branch. To meet its mission, OCWR relies extensively on external parties, such as the Library of Congress, for IT support. In December 2018, Congress passed the Congressional Accountability Act of 1995 Reform Act (Reform Act) which, among other things, required OCWR to create a secure, online system to receive and keep track of claims related to employee rights and protections, such as sexual harassment and discrimination. To meet this requirement, OCWR initiated the SOCRATES project to upgrade its legacy claims management system. The Reform Act included a provision for GAO to review OCWR's cybersecurity practices. This report examines the extent to which OCWR (1) incorporated key cybersecurity management activities into project planning for its claims management system upgrade, (2) performed oversight of security controls and mitigated risks for selected systems operated by external parties on its behalf and, (3) established an effective approach for managing organization-wide cybersecurity risk. To address these objectives, GAO compared OCWR IT policies, procedures, strategic plans, and documentation for two selected systems to leading IT project planning, system oversight, and cybersecurity management practices. The Office of Congressional Workplace Rights (OCWR) did not incorporate key cybersecurity management practices into the planning for its Secure Online Claims Reporting and Tracking E-filing System (SOCRATES) project. While OCWR drafted a SOCRATES project schedule, the office did not finalize and use this schedule to manage cybersecurity activities, such as the time frames for conducting information technology (IT) system security assessments. In addition, the office did not document project cybersecurity risks, such as the office's reliance on external parties to implement responsibilities on its behalf. These weaknesses were due, in part, to a lack of policies and procedures for IT project planning. Until OCWR establishes and implements such policies and procedures, it will continue to have a limited ability to effectively manage and monitor the completion of cybersecurity activities for its IT projects. OCWR did not fully implement important oversight activities for two selected systems—SOCRATES and the system used to document occupational safety and health violations known as the Facility Management Assistant (FMA)—operated by external entities (see table). These shortfalls contributed to concerns with the deployment of SOCRATES in June 2019. For example, important security controls needed to ensure the confidentiality, integrity, and availability of the system were not fully tested before the system was deployed. In addition, penetration testing—where evaluators mimic real-world attacks in an attempt to identify ways to circumvent the security features of the system—was not fully completed before deployment. GAO plans to issue a separate report with limited distribution on its assessment of security controls intended to, among other things, prevent successful attacks. Although OCWR's strategic plan includes a goal of developing cybersecurity policies and procedures, the office had not fully established an effective approach for managing organization-wide cybersecurity risk. For example, OCWR designated an executive to oversee risk, but had not established the responsibilities of the official in the office's policies. Until OCWR improves its appoach to managing cybersecurity risks, its ability to make operational decisions that adequately address security risks will be hindered. GAO is making five recommendations to OCWR to address weaknesses in cybersecurity management and oversight. OCWR did not state whether it agreed or disagreed with GAO's recommendations, but described actions planned or taken to address them.", "document_type": "gao"}
{"report": "As shown in table 1 the cost of counting the nation’s population has been escalating with each decade. The 2010 Census was the most expensive in U.S. history at about $12.3 billion, and was about 31 percent more costly than the $9.4 billion 2000 Census (in 2020 dollars). According to the Bureau, the total cost of the 2020 Census in October 2015 was estimated at $12.3 billion and in October 2017 that cost estimate grew to approximately $15.6 billion, approximately a $3 billion increase. Additionally, Bureau officials told us that while the estimated cost of the census had increased to $15.6 billion, it was nevertheless managing the 2020 Census to a lower cost of $14.1 billion. Bureau officials explained that the $14.1 billion includes all program costs and contingency funds to cover risks and general estimating uncertainty. The remaining $1.5 billion estimated cost is additional contingency for “unknown unknowns”—that is, low probability events that could cause massive disruptions—and several what-if scenarios such as an increase in the wage rate or additional supervisors needed to manage field operations. Moreover, as shown in figure 1, the average cost for counting a housing unit increased from about $16 in 1970 to around $92 in 2010 (in 2020 constant dollars). At the same time, the return of census questionnaires by mail (the primary mode of data collection) declined over this period from 78 percent in 1970 to 63 percent in 2010. Declining mail response rates have led to higher costs because the Bureau sends temporary workers to each non-responding household to obtain census data. Achieving a complete and accurate census has become an increasingly daunting task, in part, because the population is growing larger, more diverse, and more reluctant to participate in the enumeration. In many ways, the Bureau has had to invest substantially more resources each decade to conduct the enumeration. In addition to these external societal challenges that make achieving a complete count a daunting task, the Bureau also faces a number of internal management challenges that affect its capacity and readiness to conduct a cost-effective enumeration. Some of these issues—such as acquiring and developing IT systems and preparing reliable cost estimates—are long-standing in nature. At the same time, as the Bureau looks toward 2020, it has faced emerging and evolving uncertainties. For example, on March 26, 2018, the Secretary of Commerce announced his decision to add a question to the decennial census on citizenship status which resulted in various legislative actions and legal challenges. Ultimately, the case was heard by the U.S. Supreme Court, which, in a June 26, 2019, ruling, prevented the addition of the question because the Court found that the evidence Commerce provided in the case did not match the Secretary’s explanation. In addition, the Fourth Circuit Court of Appeals remanded other legal challenges to the district court on June 24, 2019, for further legal action, which is yet to be resolved. According to Bureau officials, on June 28, 2019, Commerce asked the Bureau to put its scheduled July 1 start date for printing questionnaires on hold while it considered legal implications of the Supreme Court ruling. On July 2, 2019, Commerce told the Bureau to proceed with printing questionnaires and other materials without the citizenship question on them. On July 5, 2019, the Department of Justice (DOJ) indicated that, although printing was continuing without the citizenship question, DOJ was evaluating legal options to include the question. However, on July 11, 2019, the President signed Executive Order 13880 stating that the Attorney General and Secretary of Commerce had informed him that the logistics and timing necessary to carry out the census, combined with delays from litigation, left no practical mechanism for including the question on the 2020 Decennial Census. Instead of collecting this information from the census questionnaire, the Executive Order requires all federal agencies to provide data on citizenship status to Commerce using legally available federal records. On the same day, DOJ notified the District Court of the issuance of the Executive Order and the Attorney General’s prepared statement that “as a practical matter, the Supreme Court’s decision closed all paths to adding the question to the 2020 decennial census.” DOJ advised the court of its intent to confer with opposing counsel regarding appropriate next steps in the proceedings. We have not analyzed these recent developments or their implications, if any, for how the Bureau will tabulate its official counts. We will continue to monitor developments for Congress. The Bureau also faced budgetary uncertainties that, according to the Bureau, led to the curtailment of testing in 2017 and 2018. However, the Consolidated Appropriations Act, 2018 appropriated for the Periodic Censuses and Programs account $2.544 billion, which more than doubled the Bureau’s request in the President’s Fiscal Year 2018 Budget of $1.251 billion. According to the explanatory statement accompanying the act, the appropriation, which is available through fiscal year 2020, was provided to ensure the Bureau has the necessary resources to immediately address any issues discovered during operational testing, and to provide a smoother transition between fiscal year 2018 and fiscal year 2019. The availability of those resources enabled the Bureau to continue preparations for the 2020 Census during the 35 days in December 2018 to January 2019 when appropriations lapsed for the Bureau and a number of other federal agencies. Moreover, the Consolidated Appropriations Act, 2019 appropriated for the Periodic Censuses and Programs account $3.551 billion. According to Bureau officials, this level of funding for fiscal year 2019 is sufficient to carry out 2020 Census activities as planned. Importantly, the census is conducted against a backdrop of immutable deadlines. In order to meet the statutory deadline for completing the enumeration, census activities need to take place at specific times and in the proper sequence. Thus, it is absolutely critical for the Bureau to stay on schedule. Figure 2 shows some dates for selected decennial events. The Bureau has begun to open its area census offices (ACO) for the 2020 Census. It has signed leases for all 248 ACOs, of which 39 of the offices will be open for the address canvassing operation set to begin in August 2019 where staff verifies the location of selected housing units. The remaining 209 offices will begin opening this fall. In 2010 the Bureau opened 494 census offices. The Bureau has been able to reduce its infrastructure because it is relying on automation to assign work and to record payroll. Therefore there is less paper—field assignments, maps, and daily payroll forms—to manually process. For the 2020 Census, the Bureau is refining its recruiting and hiring goals, but tentatively plans to recruit approximately 2.24 million applicants and to hire over 400,000 temporary field staff from that applicant pool for two key operations: address canvassing, and nonresponse follow-up, where they visit households that do not return census forms to collect data in person. In 2010 the Bureau recruited 3.8 million applicants and hired 628,000 temporary workers to conduct the address canvassing and nonresponse follow-up field operations. According to Bureau officials, it has reduced the number of temporary staff it needs to hire because automation has made field operations more efficient and there is less paper. As of July 2019, the Bureau reported that for all 2020 Census operations it had processed just over 500,000 applicants. In addition, the Bureau was seeking to hire approximately 1,500 partnership specialists by the end of June 2019 to help increase census awareness and participation in minority communities and hard-to-reach populations. The Bureau reported that as of July 6, 2019, it had hired 903 partnership specialists, and as of July 17, 2019, another 872 applicants were waiting to have their background checks completed. According to Bureau officials, hiring data are based on payroll dates generated biweekly, while background check data are tracked internally and can be updated daily. The Bureau did not meet its June 30 hiring goal, and told us that it expected to have all partnership specialists on board by September 1, 2019. Among other things, partnership specialists are expected to either provide or identify partners to help provide supplemental language support to respondents locally in over 100 different languages. We will continue to monitor the Bureau’s progress in meeting its partnership specialist staffing goals and addressing any turnover that takes place. Hiring partnership specialists in a timely manner and maintaining adequate partnership specialist staffing levels are key to the Bureau’s ability to carry out its planned outreach efforts, especially to hard-to-count communities. Moreover, Bureau officials also stated that the current economic environment (i.e., the low unemployment rate compared to the economic environment of the 2010 Census) has not yet impacted their ability to recruit staff. The Bureau will continue to monitor the impact of low unemployment on its ability to recruit and hire at the local and regional levels. For the 2020 Census, the Bureau is substantially changing how it intends to conduct the census, in part by re-engineering key census-taking methods and infrastructure, and making use of new IT applications and systems. For example, the Bureau plans to offer an option for households to respond to the survey via the internet and enable field-based enumerators to use applications on mobile devices to collect survey data from households. To do this, the Bureau plans to utilize 52 new and legacy IT systems, and the infrastructure supporting them, to conduct the 2020 Census. A majority of these 52 systems have been tested during operational tests in 2017 and 2018. For example, the Bureau conducted its 2018 End-to- End test, which included 44 of the 52 systems and was intended to test all key systems and operations in a census-like environment to ensure readiness for the 2020 Census. Nevertheless, additional IT development and testing work needs to take place before the 2020 Census. Specifically, officials from the Bureau’s Decennial Directorate said they expect that the systems will need to undergo further development and testing due to, among other things, the need to add functionality that was not part of the End-to-End test, scale system performance to support the number of respondents expected during the 2020 Census, and address system defects identified during the 2018 End-to-End test. To prepare the systems and technology for the 2020 Census, the Bureau is also relying on substantial contractor support. For example, it is relying on contractors to develop a number of systems and components of the IT infrastructure, including the IT platform that is intended to be used to collect data from households responding via the internet and telephone, and for non-response follow-up activities. Contractors are also deploying the IT and telecommunications hardware in the field offices and providing device-as-a-service capabilities by procuring the mobile devices and cellular service to be used for non-response follow-up. In addition to the development of technology, the Bureau is relying on a technical integration contractor to integrate all of the key systems and infrastructure. The contractor’s work is expected to include, among other things, evaluating the systems and infrastructure and acquiring the infrastructure (e.g., cloud or data center) to meet the Bureau’s scalability and performance needs; integrating all of the systems; and assisting with technical, performance and scalability, and operational testing activities. In February 2017, we added the 2020 Decennial Census as a high-risk area needing attention from Congress and the executive branch. This was due to significant risks related to, among other things, innovations never before used in prior enumerations, the acquisition and development of IT systems, and expected escalating costs. Among other things, we reported that the commitment of top leadership was needed to ensure the Bureau’s management, culture, and business practices align with a cost-effective enumeration. We also stressed that the Bureau needed to rigorously test census-taking activities; ensure that scheduling adheres to best practices; improve its ability to manage, develop, and secure its IT systems; and have better oversight and control over its cost estimation process. Our experience has shown that agencies are most successful at removal from our High-Risk List when leaders give top level attention to the five criteria for removal and Congress takes any needed action. The five criteria for removal that we identified in November 2000 are as follows: Leadership Commitment. The agency has demonstrated strong commitment and top leadership support. Capacity. The agency has the capacity (i.e., people and resources) to resolve the risk(s). Action Plan. A corrective action plan exists that defines the root causes and solutions, and that provides for substantially completing corrective measures, including steps necessary to implement solutions we recommended. Monitoring. A program has been instituted to monitor and independently validate the effectiveness and sustainability of corrective measures. Demonstrated Progress. The agency has demonstrated progress in implementing corrective measures and in resolving the high-risk area. These five criteria form a road map for efforts to improve, and ultimately address, high-risk issues. Addressing some of the criteria leads to progress, while satisfying all of the criteria is central to removal from the list. As we reported in the March 2019 high-risk report, the Bureau’s efforts to address the risks and challenges for the 2020 Census had fully met one of the five criteria for removal from the High-Risk List—leadership commitment—and partially met the other four, as shown in figure 3. Additional details about the status of the Bureau’s efforts to address this high-risk area are discussed later in this statement. The 2020 Census is on our list of high-risk programs because, among other things, (1) innovations never before used in prior enumerations are not expected to be fully tested, (2) the Bureau continues to face challenges in implementing IT systems, (3) the Bureau faces significant cybersecurity risks to its systems and data, and (4) the Bureau’s cost estimate for the 2020 Census was unreliable. If not sufficiently addressed, these risks could adversely impact the cost and quality of the enumeration. Moreover, the risks are compounded by other factors that contribute to the challenge of conducting a successful census, such as the nation’s increasingly diverse population and concerns over personal privacy. The basic design of the enumeration—mail out and mail back of the census questionnaire with in-person follow-up for non-respondents—has been in use since 1970. However, a lesson learned from the 2010 Census and earlier enumerations is that this traditional design is no longer capable of cost-effectively counting the population. In response to its own assessments, our recommendations, and studies by other organizations, the Bureau has fundamentally re-examined its approach for conducting the 2020 Census. Specifically, its plan for 2020 includes four broad innovation areas: re-engineering field operations, using administrative records, verifying addresses in-office, and developing an internet self-response option (see table 2). If they function as planned, the Bureau initially estimated that these innovations could result in savings of over $5 billion (in 2020 constant dollars) when compared to its estimates of the cost for conducting the census with traditional methods. However, in June 2016, we reported that the Bureau’s initial life-cycle cost estimate developed in October 2015 was not reliable and did not adequately account for risk. As discussed earlier in this statement, the Bureau has updated its estimate from $12.3 billion and now estimates a life-cycle cost of $15.6 billion, which would result in a smaller potential savings from the innovative design than the Bureau originally estimated. According to the Bureau, the goal of the cost estimate increase was to ensure quality was fully addressed. While the planned innovations could help control costs, they also introduce new risks, in part, because they include new procedures and technology that have not been used extensively in earlier decennials, if at all. Our prior work has shown the importance of the Bureau conducting a robust testing program, including the 2018 End-to-End test. Rigorous testing is a critical risk mitigation strategy because it provides information on the feasibility and performance of individual census-taking activities, their potential for achieving desired results, and the extent to which they are able to function together under full operational conditions. To address some of these challenges we have made numerous recommendations aimed at improving reengineered field operations, using administrative records, verifying the accuracy of the address list, and securing census responses via the internet. The Bureau has held a series of operational tests since 2012, but according to the Bureau, it scaled back its most recent field tests because of funding uncertainties. For example, the Bureau canceled the field components of the 2017 Census Test including non-response follow-up, a key census operation. In November 2016, we reported that the cancelation of the 2017 Census Test was a lost opportunity to test, refine, and integrate operations and systems, and that it put more pressure on the 2018 End-to-End test to demonstrate that enumeration activities will function under census-like conditions as needed for 2020. However, in May 2017, the Bureau scaled back the operational scope of the 2018 End-to-End test and, of the three planned test sites, only the Rhode Island site would fully implement the 2018 End-to-End test. The Washington and West Virginia sites would test just one field operation. In addition, due to budgetary concerns, the Bureau delayed ramp up and preparations for its coverage measurement operation (and the technology that supports it) from the scope of the test. However, removal of the coverage measurement operation did not affect testing of the delivery of apportionment or redistricting data. Without sufficient testing, operational problems can go undiscovered and the opportunity to improve operations will be lost, in part because the 2018 End-to-End test was the last opportunity to demonstrate census technology and procedures across a range of geographic locations, housing types, and demographic groups under decennial-like conditions prior to the 2020 Census. We reported on the 2018 End-to-End test in December 2018 and noted that the Bureau had made progress addressing prior test implementation issues but still faced challenges. As the Bureau studies the results of its testing to inform the 2020 Census, it will be important that it addresses key program management issues that arose during implementation of the test. Namely, by not aligning the skills, responsibilities, and information flows for the first-line supervisors during field data collection, the Bureau limited its role in support of enumerators within the re-engineered field operation. The Bureau also lacked mid-operation training or guidance, which, if implemented in a targeted, localized manner, could have further helped enumerators navigate procedural modifications and any commonly encountered problems when enumerating. It will be important for the Bureau to prioritize its mitigation strategies for these implementation issues so that it can maximize readiness for the 2020 Census. To manage risk to the 2020 Census the Bureau has developed hundreds of risk mitigation and contingency plans. Mitigation plans detail how an agency will reduce the likelihood of a risk event and its impacts, if it occurs. Contingency plans identify how an agency will reduce or recover from the impact of a risk after it has been realized. In May 2019, we reported that the Bureau had identified 360 active risks to the 2020 census as of December 2018—meaning the risk event could still occur and adversely impact the census. Of these, 242 met the Bureau’s criteria for requiring a mitigation plan and, according to the Bureau’s risk registers, 232 had a plan (see table 3). In addition, 146 risks met the Bureau’s criteria for requiring one contingency plan and, according to the Bureau’s risk registers, 102 had a plan. Bureau guidance states that these plans should be developed as soon as possible after a risk is added to the risk register, but it does not establish a clear time frame for doing so. Consequently, some risks may go without required plans for extended periods. We found that, as of December 2018, some of the risks without required plans had been added to the Bureau’s risk registers in recent months, but others had been added more than 3 years earlier. We reviewed the mitigation and contingency plans in detail for six risks which the Bureau identified as among the major concerns that could affect the 2020 Census. These included cybersecurity incidents, late operational design changes, and integration of the 52 systems and 35 operations supporting the 2020 Census. We found that the plans did not consistently include key information needed to manage the risk. For example, the Bureau’s contingency plan for late operational design changes did not include activities specific to the three most likely late operational design changes—including removal of the citizenship question as a result of litigation or congressional action—that the Bureau could carry out to lessen their adverse impact on the enumeration. We found that gaps stemmed from either requirements that were missing from the Bureau’s decennial risk management plan, or that risk owners— the individuals assigned to manage each risk—were not fulfilling all of their risk management responsibilities. Bureau officials said that risk owners were aware of these responsibilities but did not always fulfill them given competing demands. Bureau officials also said that they are managing risks to the census, even if that is not always reflected in their mitigation and contingency plans. However, if such actions are reflected in disparate documents or are not documented at all, then decision makers are left without an integrated and comprehensive picture of how the Bureau is managing risks to the census. We made seven recommendations to improve the Bureau’s management of risks to the 2020 Census, including that the Bureau develop mitigation and contingency plans for all risks that require them, establish a clear time frame for plan development, and ensure that the plans have the information needed to manage the risk. Commerce agreed with our recommendations and said it would develop an action plan to address them. We have previously reported that the Bureau faces challenges in managing and overseeing IT programs, systems, and contractors supporting the 2020 Census. Specifically, we have noted challenges in the Bureau’s efforts to manage, among other things, the schedules and contracts for its systems. As a result of these challenges, the Bureau is at risk of being unable to fully implement the systems necessary to support the 2020 Census and conduct a cost-effective enumeration. To help improve its implementation of IT for the 2020 Census, the Bureau revised its systems development and testing schedule. Specifically, in October 2018, the Bureau organized the development and testing schedule for its 52 systems into 16 operational deliveries. Each of the 16 operational deliveries has milestone dates for, among other things, development, performance and scalability testing, and system deployment. According to Bureau officials in the Decennial Directorate, the schedule was revised, in part, due to schedule management challenges experienced, and lessons learned, while completing development and testing during the 2018 End-to-End test. The Bureau has made initial progress in executing work against its revised schedule. For example, the Bureau completed development of the systems in the first operational delivery—for 2020 Census early operations preparations—in July 2018, and deployed these systems into production in October 2018. However, our current work has determined that the Bureau is at risk of not meeting several near-term systems testing milestones. As of June 2019, 11 systems that are expected to be used in a total of five operational deliveries were at risk of not meeting key milestones for completing system development, performance and scalability testing, and/or integration testing. These 11 systems are needed for, among other things, data collection for operations, business and support automation, and customer support during self-response. Figure 4 presents an overview of the status for all 16 operational deliveries, as of June 2019. The at-risk systems previously discussed add uncertainty to a highly compressed time frame over the next 6 months. Importantly, between July and December 2019, the Bureau is expected to be in the process of integration testing the systems in 12 operational deliveries. Officials from the Bureau’s integration contractor noted concern that the current schedule leaves little room for any delays in completing the remaining development and testing activities. In addition to managing the compressed testing time frames, the Bureau also has to quickly finalize plans related to its IT infrastructure. For example, as of June 2019, the Bureau stated that it was still awaiting final approval for its Trusted Internet Connection. Given that these plans may impact systems being tested this summer or deployed into production for the address canvassing operation in August 2019, it is important that the Bureau quickly addresses this matter. Our past reporting noted that the Bureau faced significant challenges in managing its schedule for system development and testing that occurred in 2017 and 2018. We reported that, while the Bureau had continued to make progress in developing and testing IT systems for the 2020 Census, it had experienced delays in developing systems to support the 2018 End-to-End test. These delays compressed the time available for system and integration testing and for security assessments. In addition, several systems experienced problems during the test. We noted then, and reaffirm now, that continued schedule management challenges may compress the time available for the remaining system and integration testing and increase the risk that systems may not function or be as secure as intended. The Bureau has acknowledged that it faces risks to the implementation of its systems and technology. As of May 2019, the Bureau had identified 17 high risks related to IT implementation that may have substantial technical and schedule impacts if realized. Taken together, these risks represent a cross-section of issues, such as schedule delays for a fraud- detection system, the effects of late changes to technical requirements, the need to ensure adequate time for system development and performance and scalability testing, contracting issues, privacy risks, and skilled staffing shortages. Going forward, it will be important that the Bureau effectively manages these risks to better ensure that it meets near-term milestones for system development and testing, and is ready for the major operations of the 2020 Census. The risks to IT systems supporting the federal government and its functions, including conducting the 2020 Census, are increasing as security threats continue to evolve and become more sophisticated. These risks include insider threats from witting or unwitting employees, escalating and emerging threats from around the globe, and the emergence of new and more destructive attacks. Underscoring the importance of this issue, we have designated information security as a government-wide high-risk area since 1997 and, in our most recent biennial report to Congress, ensuring the cybersecurity of the nation was one of nine high-risk areas that we reported needing especially focused executive and congressional attention. Our prior and ongoing work has identified significant challenges that the Bureau faces in securing systems and data for the 2020 Census. Specifically, the Bureau has faced challenges related to completing security assessments, addressing security weaknesses, resolving cybersecurity recommendations from DHS, and addressing numerous other cybersecurity concerns (such as phishing). Federal law specifies requirements for protecting federal information and information systems, such as those systems to be used in the 2020 Census. Specifically, the Federal Information Security Management Act of 2002 and the Federal Information Security Modernization Act of 2014 (FISMA) require executive branch agencies to develop, document, and implement an agency-wide program to provide security for the information and information systems that support operations and assets of the agency. In accordance with FISMA, National Institute of Standards and Technology (NIST) guidance, and Office of Management and Budget (OMB) guidance, the Bureau’s Office of the Chief Information Officer (CIO) established a risk management framework. This framework requires system developers to ensure that each of the Bureau’s systems undergoes a full security assessment, and that system developers remediate critical deficiencies. According to the Bureau’s risk management framework, the systems expected to be used to conduct the 2020 Census will need to have complete security documentation (such as system security plans) and an approved authorization to operate prior to their use. As of June 2019, according to the Bureau’s Office of the CIO: Thirty-seven of the 52 systems have authorization to operate, and will not need to be reauthorized before they are used in the 2020 Census Nine of the 52 systems have authorization to operate, and will need to be reauthorized before they are used in the 2020 Census Five of the 52 systems do not have authorization to operate, and will need to be authorized before they are used in the 2020 Census One of the 52 systems does not need an authorization to operate before it is used in the 2020 Census. Figure 5 summarizes the authorization to operate status for the systems being used in the 2020 Census, as reported by the Bureau in June 2019. As we have previously reported, while large-scale technological changes (such as internet self-response) increase the likelihood of efficiency and effectiveness gains, they also introduce many cybersecurity challenges. The 2020 Census also involves collecting personally identifiable information (PII) on over a hundred million households across the country, which further increases the need to properly secure these systems. Thus, it will be important that the Bureau provides adequate time to perform these security assessments, completes them in a timely manner, and ensures that risks are at an acceptable level before the systems are deployed. We have ongoing work examining how the Bureau plans to address both internal and external cyber threats, including its efforts to complete system security assessments and resolve identified weaknesses. FISMA requires that agency-wide information security programs include a process for planning, implementing, evaluating, and documenting remedial actions (i.e., corrective actions) to address any deficiencies in the information security policies, procedures, and practices of the agency. Additionally, the Bureau’s framework requires it to track security assessment findings that need to be remediated as a plan of action and milestones (POA&M). These POA&Ms are expected to provide a description of the vulnerabilities identified during the security assessment that resulted from a control weakness. As of the end of May 2019, the Bureau had over 330 open POA&Ms to remediate for issues identified during security assessment activities, including ongoing continuous monitoring. Of these open POA&Ms, 217 (or about 65 percent) were considered “high-risk” or “very high-risk.” While the Bureau established POA&Ms for addressing these identified security control weaknesses, it did not always complete remedial actions in accordance with its established deadlines. For example, of the 217 open “high-risk” or “very high-risk” POA&Ms we reviewed, the Bureau identified 104 as being delayed. Further, 74 of the 104 had missed their scheduled completion dates by 60 or more days. According to the Bureau’s Office of Information Security, these POA&Ms were identified as delayed due to technical challenges or resource constraints to remediate and close them. We previously recommended that the Bureau take steps to ensure that identified corrective actions for cybersecurity weaknesses are implemented within prescribed time frames. As of late May 2019, the Bureau was working to address our recommendation. Until the Bureau resolves identified vulnerabilities in a timely manner, it faces an increased risk, as continuing opportunities exist for unauthorized individuals to exploit these weaknesses and gain access to sensitive information and systems. The Bureau is working with federal and industry partners, including DHS, to support the 2020 Census cybersecurity efforts. Specifically, the Bureau is working with DHS to ensure a scalable and secure network connection for the 2020 Census respondents (e.g., virtual Trusted Internet Connection with the cloud), improve its cybersecurity posture (e.g., risk management processes and procedures), and strengthen its response to potential cyber threats (e.g., federal cyber incident coordination). Federal law and related standards describe practices for strengthening cybersecurity by documenting or tracking corrective actions. As previously mentioned, FISMA requires executive branch agencies to establish a process for planning, implementing, evaluating, and documenting remedial actions to address any deficiencies in their information security policies, procedures, and practices. Standards for Internal Control in the Federal Government calls for agencies to establish effective internal control monitoring that includes a process to promptly resolve the findings of audits and other reviews. Specifically, agencies should document and complete corrective actions to remediate identified deficiencies on a timely basis. This would include correcting identified deficiencies or demonstrating that the findings and recommendations do not warrant agency action. Since January 2017, DHS has been providing cybersecurity assistance (including issuing recommendations) to the Bureau in preparation for the 2020 Census. Specifically, DHS has been providing cybersecurity assistance to the Bureau in five areas: management coordination and executive support, including a CyberStat Review; cybersecurity threat intelligence and information sharing enhancement through, among other things, a DHS cyber threat briefing to the Bureau’s leadership; network and infrastructure security and resilience, including National Cybersecurity Protection System (also called EINSTEIN) support; incident response and management readiness through a Federal Incident Response Evaluation assessment; and risk management and vulnerability assessments for specific high value assets provided by the Bureau. In the last 2 years, DHS has provided 42 recommendations to assist the Bureau in strengthening its cybersecurity efforts. Among other things, the recommendations pertained to strengthening cyber incident management capabilities, penetration testing and web application assessments of select systems, and phishing assessments to gain access to sensitive PII. Of the 42 recommendations, 10 recommendations resulted from DHS’s mandatory services for the Bureau (e.g., risk management and vulnerability assessments for specific high value assets). The remaining 32 recommendations resulted from DHS’s voluntary services for the Bureau (e.g., Federal Incident Response Evaluation assessment). Due to the sensitive nature of the recommendations, we are not identifying the specific recommendations or specific findings associated with them in this statement. In April 2019, we reported that the Bureau had not established a formal process for documenting, tracking, and completing corrective actions for all of the recommendations provided by DHS. Accordingly, we recommended that the Bureau implement a formal process for tracking and executing appropriate corrective actions to remediate cybersecurity findings identified by DHS. As of late May 2019, the Bureau was working to address our recommendation. Until the Bureau implements our recommendation, it faces an increased likelihood that findings identified by DHS will go uncorrected and may be exploited to cause harm to agency’s 2020 Census IT systems and gain access to sensitive respondent data. Implementing a formal process would also help to ensure that DHS’s efforts result in improvements to the Bureau’s cybersecurity posture. The Bureau faces other substantial cybersecurity challenges in addition to those previously discussed. More specifically, we previously reported that the extensive use of IT systems to support the 2020 Census redesign may help increase efficiency, but that this redesign introduces critical cybersecurity challenges. These challenges include those related to the following: Phishing. We have previously reported that advanced persistent threats may be targeted against social media web sites used by the federal government. In addition, attackers may use social media to collect information and launch attacks against federal information systems through social engineering, such as phishing. Phishing attacks could target respondents, as well as Bureau employees and contractors. The 2020 Census will be the first one in which respondents will be heavily encouraged to respond via the internet. This will likely increase the risk that cyber criminals will use phishing in an attempt to steal personal information. According to the Bureau, it plans to inform the public of the risks associated with phishing through its education and communication campaigns. Disinformation from social media. We previously reported that one of the Bureau’s key innovations for the 2020 Census is the large-scale implementation of an internet self-response option. The Bureau is encouraging the public to use the internet self-response option through expanded use of social media. However, the public perception of the Bureau’s ability to adequately safeguard the privacy and confidentiality of the 2020 Census internet self-responses could be influenced by disinformation spread through social media. According to the Bureau, if a substantial segment of the public is not convinced that the Bureau can safeguard public response data against data breaches and unauthorized use, then response rates may be lower than projected, leading to an increase in cases for follow-up and subsequent cost increases. To help address this challenge, the Bureau stated that it plans to inform the public of the risks associated with disinformation from social media through its education and communication campaigns. Ensuring that individuals gain only limited and appropriate access to 2020 Census data. The Bureau plans to enable a public- facing website and Bureau-issued mobile devices to collect PII (e.g., name, address, and date of birth) from the nation’s entire population— estimated to be over 300 million. In addition, the Bureau is planning to obtain and store administrative records containing PII from other government agencies to help augment information that enumerators did not collect. The number of reported security incidents involving PII at federal agencies has increased dramatically in recent years. Because of these challenges, we have recommended, among other things, that federal agencies improve their response to information security incidents and data breaches involving PII, and consistently develop and implement privacy policies and procedures. Accordingly, it will be important for the Bureau to ensure that only respondents and Bureau officials are able to gain access to this information, and enumerators and other employees only have access to the information needed to perform their jobs. Ensuring adequate control in a cloud environment. The Bureau has decided to use cloud solutions as a key component of the 2020 Census IT infrastructure. We have previously reported that cloud computing has both positive and negative information security implications and, thus, federal agencies should develop service-level agreements with cloud providers. These agreements should specify, among other things, the security performance requirements—including data reliability, preservation, privacy, and access rights—that the service provider is to meet. Without these safeguards, computer systems and networks, as well as the critical operations and key infrastructures they support, may be lost; information—including sensitive personal information—may be compromised; and the agency’s operations could be disrupted. Commerce’s Office of the Inspector General recently identified several challenges the Bureau may face using cloud-based systems to support the 2020 Census. Specifically, in June 2019, the Office of the Inspector General identified, among other things, unimplemented security system features that left critical 2020 Census systems vulnerable during the 2018 End-to-End Test and a lack of fully implemented security practices to protect certain data hosted in the 2020 Census cloud environment. Officials from the Bureau agreed with all eight of the Office of Inspector General’s recommendations regarding 2020 Census cloud-based systems and identified actions taken to address them. Ensuring contingency and incident response plans are in place to encompass all of the IT systems to be used to support the 2020 Census. Because of the brief time frame for collecting data during the 2020 Census, it is especially important that systems are available for respondents to ensure a high response rate. Contingency planning and incident response help ensure that, if normal operations are interrupted, network managers will be able to detect, mitigate, and recover from a service disruption while preserving access to vital information. Implementing important security controls, including policies, procedures, and techniques for contingency planning and incident response, helps to ensure the confidentiality, integrity, and availability of information and systems, even during disruptions of service. Without contingency and incident response plans, system availability might be impacted and result in a lower response rate. The Bureau’s CIO has acknowledged these cybersecurity challenges and is working to address them, according to Bureau documentation. In addition, we have ongoing work looking at many of these challenges, including the Bureau’s plans to protect PII, use a cloud-based infrastructure, and recover from security incidents and other disasters. Since 2015, the Bureau has made progress in improving its ability to develop a reliable cost estimate. We have reported on the reliability of the $12.3 billion life-cycle cost estimate released in October 2015 and the $15.6 billion revised cost estimate released in October 2017. In 2016 we reported that the October 2015 version of the Bureau’s life-cycle cost estimate for the 2020 Census was not reliable. Specifically, we found that the 2020 Census life-cycle cost estimate partially met two of the characteristics of a reliable cost estimate (comprehensive and accurate) and minimally met the other two (well-documented and credible). We recommended that the Bureau take specific steps to ensure its cost estimate meets the characteristics of a high-quality estimate. The Bureau agreed and has taken action to improve the reliability of the cost estimate. In August 2018 we reported that while improvements had been made, the Bureau’s October 2017 cost estimate for the 2020 Census did not fully reflect all the characteristics of a reliable estimate. (See figure 6.) In order for a cost estimate to be deemed reliable as described in GAO’s Cost Estimating and Assessment Guide and thus, to effectively inform 2020 Census annual budgetary figures, the cost estimate must meet or substantially meet the following four characteristics: Well-Documented. Cost estimates are considered valid if they are well-documented to the point they can be easily repeated or updated and can be traced to original sources through auditing, according to best practices. Accurate. Accurate estimates are unbiased and contain few mathematical mistakes. Credible. Credible cost estimates must clearly identify limitations due to uncertainty or bias surrounding the data or assumptions, according to best practices. Comprehensive. To be comprehensive an estimate should have enough detail to ensure that cost elements are neither omitted nor double-counted, and all cost-influencing assumptions are detailed in the estimate’s documentation, among other things, according to best practices. The 2017 cost estimate only partially met the characteristic of being well- documented. In general, some documentation was missing, inconsistent, or difficult to understand. Specifically, we found that source data did not always support the information described in the basis of estimate document or could not be found in the files provided for two of the Bureau’s largest field operations: Address Canvassing and Non- Response Follow-Up. We also found that some of the cost elements did not trace clearly to supporting spreadsheets and assumption documents. Failure to document an estimate in enough detail makes it more difficult to replicate calculations, or to detect possible errors in the estimate; reduces transparency of the estimation process; and can undermine the ability to use the information to improve future cost estimates or even to reconcile the estimate with another independent cost estimate. The Bureau told us it would continue to make improvements to ensure the estimate is well- documented. The 2017 life-cycle cost estimate includes much higher costs than those included in the 2015 estimate. The largest increases occurred in the Response, Managerial Contingency, and Census/Survey Engineering categories. For example, increased costs of $1.3 billion in the response category (costs related to collecting, maintaining, and processing survey response data) were in part due to reduced assumptions for self- response rates, leading to increases in the amount of data collected in the field, which is more costly to the Bureau. Contingency allocations increased overall from $1.35 billion in 2015 to $2.6 billion in 2017, as the Bureau gained a greater understanding of risks facing the 2020 Census. Increases of $838 million in the Census/Survey Engineering category were due mainly to the cost of an IT contract for integrating decennial survey systems that was not included in the 2015 cost estimate. Bureau officials attribute a decrease of $551 million in estimated costs for Program Management to changes in the categorization of costs associated with risks. Specifically, in the 2017 version of the estimate, estimated costs related to program risks were allocated to their corresponding work breakdown structure (WBS) element. Figure 7 shows the change in cost by WBS category for 2015 and 2017. More generally, factors that contributed to cost fluctuations between the 2015 and 2017 cost estimates include: Changes in assumptions. Among other changes, a decrease in the assumed rate for self-response from 63.5 percent in 2015 to 60.5 percent in 2017 increased the cost of collecting responses from nonresponding housing units. Improved ability to anticipate and quantify risk. In general, contingency allocations designed to address the effects of potential risks increased overall from $1.3 billion in 2015 to $2.6 billion in 2017. An overall increase in IT costs. IT cost increases, totaling $1.59 billion, represented almost 50 percent of the total cost increase from 2015 to 2017. More defined contract requirements. Bureau documents described an overall improvement in the Bureau’s ability to define and specify contract requirements. This resulted in updated estimates for several contracts, including for the Census Questionnaire Assistance contract. However, while the Bureau has been able to better quantify risk; in August 2018 we also reported that the Secretary of Commerce included a contingency amount of about $1.2 billion in the 2017 cost estimate to account for what the Bureau refers to as “unknown unknowns.” According to Bureau documentation these include such risks as natural disasters or cyber attacks. The Bureau provides a description of how the risk contingency for “unknown unknowns” is calculated; however, this description does not clearly link calculated amounts to the risks themselves. Thus, only $14.4 billion of the Bureau’s $15.6 billion cost estimate has justification. According to Bureau officials, the cost estimate remains at $15.6 billion; however, they stated that they are managing the 2020 Census at a lower level of funding—$14.1 billion. In addition, they said that, at this time, they do not plan to request funding for the $1.2 billion contingency fund for unknown unknowns or $369 million in funding for selected discrete program risks for what-if scenarios, such as an increase in the wage rate or additional supervisors needed to manage field operations. Instead of requesting funding for these contingencies upfront the Bureau plans to work with OMB and Commerce to request additional funds, if the need arises. According to Bureau officials they anticipate that the remaining $1.1 billion in contingency funding included in the $14.1 billion will be sufficient to carry out the 2020 Census. In June 2016 we recommended the Bureau improve control over how risk and uncertainty are accounted for. This prior recommendation remains valid given the life-cycle cost estimate still includes the $1.2 billion unjustified contingency fund for “unknown unknowns”. Moreover, given the cost growth between 2015 and 2017 it will be important for the Bureau to monitor cost in real-time, as well as, document, explain and review variances between planned and actual cost. In August 2018 we reported that the Bureau had not been tracking variances between estimated life-cycle costs and actual expenses. Tools to track variance enable management to measure progress against planned outcomes and will help inform the 2030 Census cost estimate. Bureau officials stated that they already have systems in place that can be adapted for tracking estimated and actual costs. We will continue to monitor the status of the tracking system. According to Bureau officials, the Bureau planned to release an updated version of the 2020 Census life-cycle estimate in the spring of 2019; however, they released the update on July 15, 2019. We will review the documentation to see whether the revised estimate will address our recommendations. To ensure that future updates to the life-cycle cost estimate reflect best practices, it will be important for the Bureau to implement our recommendation related to the cost estimate. The difficulties facing the Bureau’s preparation for the decennial census in such areas as planning and testing; managing and overseeing IT programs, systems, and contractors supporting the enumeration; developing reliable cost estimates; prioritizing decisions; managing schedules; and other challenges, are symptomatic of deeper organizational issues. Following the 2010 Census, a key lesson learned for 2020 that we identified was ensuring that the Bureau’s organizational culture and structure, as well as its approach to strategic planning, human capital management, internal collaboration, knowledge sharing, capital decision- making, risk and change management, and other internal functions are aligned toward delivering more cost-effective outcomes. The Bureau has made improvements over the last decade, and continued progress will depend in part on sustaining efforts to strengthen risk management activities, enhancing systems testing, bringing in experienced personnel to key positions, implementing our recommendations, and meeting regularly with officials from its parent agency, Commerce. Going forward, we have reported that the key elements needed to make progress in high-risk areas are top-level attention by the administration and agency officials to (1) leadership commitment, (2) ensuring capacity, (3) developing a corrective action plan, (4) regular monitoring, and (5) demonstrated progress. Although important steps have been taken in at least some of these areas, overall, far more work is needed. We discuss three of five areas below. The Secretary of Commerce has successfully demonstrated leadership commitment. For example, the Bureau and Commerce have strengthened this area with executive-level oversight of the 2020 Census by holding regular meetings on the status of IT systems and other risk areas. In addition, in 2017 Commerce designated a team to assist senior Bureau management with cost estimation challenges. Moreover, on January 2, 2019, a new Director of the Census Bureau took office, a position that had been vacant since June 2017. With regard to capacity, the Bureau has improved the cost estimation process of the decennial when it established guidance including: roles and responsibilities for oversight and approval of cost estimation processes, procedures requiring a detailed description of the steps taken to produce a high-quality cost estimate, and a process for updating the cost estimate and associated documents over the life of a project. However, the Bureau continues to experience skills gaps in the government program management office overseeing the $886 million contract for integrating the IT systems needed to conduct the 2020 Census. Specifically, as of June 2019, 14 of 44 positions in this office were vacant. For the monitoring element, we found to track performance of decennial census operations, the Bureau relied on reports to track progress against pre-set goals for a test conducted in 2018. According to the Bureau, these same reports will be used in 2020 to track progress. However, the Bureau’s schedule for developing IT systems during the 2018 End-to-End test experienced delays that compressed the time available for system testing, integration testing, and security assessments. These schedule delays contributed to systems experiencing problems after deployment, as well as cybersecurity challenges. In the months ahead, we will continue to monitor the Bureau’s progress in addressing each of the five elements essential for reducing the risk to a cost-effective enumeration. Over the past several years we have issued numerous reports that underscored the fact that, if the Bureau was to successfully meet its cost savings goal for the 2020 Census, the agency needed to take significant actions to improve its research, testing, planning, scheduling, cost estimation, system development, and IT security practices. As of July 2019, we have made 107 recommendations related to the 2020 Census. The Bureau has implemented 74 of these recommendations, 32 remain open, and one recommendation was closed as not implemented. Of the 32 open recommendations, 10 were directed at improving the implementation of the innovations for the 2020 Census. Commerce generally agreed with our recommendations and is taking steps to implement them. Moreover, in April 2019 we wrote to the Secretary of Commerce, providing a list of the 12 open 2020-Census-related recommendations that we designated as “priority.” Priority recommendations are those recommendations that we believe warrant priority attention from heads of key departments and agencies. We believe that attention to these recommendations is essential for a cost-effective enumeration. The recommendations included implementing reliable cost estimation and scheduling practices in order to establish better control over program costs, as well as taking steps to better position the Bureau to develop an internet response option for the 2020 Census. In addition to our recommendations, to better position the Bureau for a more cost-effective enumeration, on March 18, 2019, we met with OMB, Commerce, and Bureau officials to discuss the Bureau’s progress in reducing the risks facing the census. We also meet regularly with Bureau officials and managers to discuss the progress and status of open recommendations related to the 2020 Census, which has resulted in Bureau actions in recent months leading to closure of some recommendations. We are encouraged by this commitment by Commerce and the Bureau in addressing our recommendations. Implementing our recommendations in a complete and timely manner is important because it could improve the management of the 2020 Census and help to mitigate continued risks. In conclusion, while the Bureau has made progress in revamping its approach to the census, it faces considerable challenges and uncertainties in implementing key cost-saving innovations and ensuring they function under operational conditions; managing the development and testing of its IT systems; ensuring the cybersecurity of its systems and data; and developing a quality cost estimate for the 2020 Census and preventing further cost increases. For these reasons, the 2020 Census is a GAO high-risk area. Going forward, continued management attention and oversight will be vital for ensuring that risks are managed, preparations stay on track, and the Bureau is held accountable for implementing the enumeration, as planned. Without timely and appropriate actions, the challenges previously discussed could adversely affect the cost, accuracy, schedule, and security of the enumeration. We will continue to assess the Bureau’s efforts and look forward to keeping Congress informed of the Bureau’s progress. Chairman Raskin, Ranking Member Roy, and Members of the Subcommittee, this completes our prepared statement. We would be pleased to respond to any questions that you may have. If you have any questions about this statement, please contact Robert Goldenkoff at (202) 512-2757 or by email at goldenkoffr@gao.gov or Nick Marinos at (202) 512-9342 or by email at marinosn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other key contributors to this testimony include Jon Ticehurst (Assistant Director); Kate Sharkey (Assistant Director); Ty Mitchell (Assistant Director); Lisa Pearson (Assistant Director); Andrea Starosciak (Analyst in Charge); Christopher Businsky; Jackie Chapin; Jeff DeMarco; Rebecca Eyler; Adella Francis; Scott Pettis; Kayla Robinson; Robert Robinson; Cindy Saunders; Sejal Sheth; Emmy Rhine Paule; and Umesh Thakkar. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Bureau is responsible for conducting a complete and accurate decennial census of the U.S. population. The decennial census is mandated by the Constitution and provides vital data for the nation. A complete count of the nation's population is an enormous undertaking as the Bureau seeks to control the cost of the census, implement operational innovations, and use new and modified IT systems. In recent years, GAO has identified challenges that raise serious concerns about the Bureau's ability to conduct a cost-effective count. For these reasons, GAO added the 2020 Census to its High-Risk list in February 2017. GAO was asked to testify about the reasons the 2020 Census remains on the High-Risk List and the steps the Bureau needs to take to mitigate risks to a successful census. To do so, GAO summarized its prior work regarding the Bureau's planning efforts for the 2020 Census. GAO also included preliminary observations from its ongoing work examining the IT systems readiness and cybersecurity for the 2020 Census. This information is related to, among other things, the Bureau's progress in developing and testing key systems and the status of cybersecurity risks. The 2020 Decennial Census is on GAO's list of high-risk programs primarily because the Department of Commerce's Census Bureau (Bureau) (1) is using innovations that are not expected to be fully tested, (2) continues to face challenges in implementing information technology (IT) systems, and (3) faces significant cybersecurity risks to its systems and data. Although the Bureau has taken initial steps to address risk, additional actions are needed as these risks could adversely impact the cost, quality, schedule, and security of the enumeration. Innovations. The Bureau is planning several innovations for the 2020 Census, including allowing the public to respond using the internet. These innovations show promise for controlling costs, but they also introduce new risks, in part, because they have not been used extensively, if at all, in earlier enumerations. As a result, testing is essential to ensure that key IT systems and operations will function as planned. However, citing budgetary uncertainties, the Bureau scaled back operational tests in 2017 and 2018, missing an opportunity to fully demonstrate that the innovations and IT systems will function as intended during the 2020 Census. To manage risk to the census, the Bureau has developed hundreds of mitigation and contingency plans. To maximize readiness for the 2020 Census, it will also be important for the Bureau to prioritize among its mitigation and contingency strategies those that will deliver the most cost-effective outcomes for the census. Implementing IT systems. The Bureau plans to rely heavily on IT for the 2020 Census, including a total of 52 new and legacy IT systems and the infrastructure supporting them. To help improve its implementation of IT, in October 2018, the Bureau revised its systems development and testing schedule to reflect, among other things, lessons learned during its 2018 operational test. However, GAO's ongoing work has determined that the Bureau is at risk of not meeting near-term IT system development and testing schedule milestones for five upcoming 2020 Census operational deliveries, including self-response (e.g., the ability to respond to the 2020 Census through the internet). These schedule management challenges may compress the time available for the remaining system development and testing, and increase the risk that systems will not function as intended. It will be important that the Bureau effectively manages IT implementation risk to ensure that it meets near-term milestones for system development and testing, and that it is ready for the major operations of the 2020 Census. Cybersecurity. The Bureau has established a risk management framework that requires it to conduct a full security assessment for nearly all the systems expected to be used for the 2020 Census and, if deficiencies are identified to determine the corrective actions needed to remediate those deficiencies. As of the end of May 2019, the Bureau had over 330 corrective actions from its security assessments that needed to be addressed, including 217 that were considered “high-risk” or “very high-risk.” However, of these 217 corrective actions, the Bureau identified 104 as being delayed. Further, 74 of the 104 were delayed by 60 or more days. According to the Bureau, these corrective actions were delayed due to technical challenges or resource constraints. GAO recently recommended that the Bureau take steps to ensure that identified corrective actions for cybersecurity weaknesses are implemented within prescribed time frames. Resolving identified vulnerabilities more timely can help reduce the risk that unauthorized individuals may exploit weaknesses to gain access to sensitive information and systems. To its credit, the Bureau is also working with the Department of Homeland Security (DHS) to support its 2020 Census cybersecurity efforts. For example, DHS is helping the Bureau ensure a scalable and secure network connection for the 2020 Census respondents and to strengthen its response to potential cyber threats. During the last 2 years, as a result of these activities, the Bureau has received 42 recommendations from DHS to improve its cybersecurity posture. GAO recently recommended that the Bureau implement a formal process for tracking and executing appropriate corrective actions to remediate cybersecurity findings identified by DHS. Implementing the recommendation would help better ensure that DHS's efforts result in improvements to the Bureau's cybersecurity posture. In addition to addressing risks which could affect innovations and the security of the enumeration, the Bureau has the opportunity to improve its cost estimating process for the 2020 Census, and ultimately the reliability of the estimate itself, by reflecting best practices. In October 2017, the 2020 Census life-cycle cost estimate was updated and is now projected to be $15.6 billion, a more than $3 billion (27 percent) increase over its earlier estimate. GAO reported in August 2018 that although the Bureau had taken steps to improve its cost estimation process for 2020, it needed to implement a system to track and report variances between actual and estimated cost elements. According to Bureau officials, they planned to release an updated version of the 2020 Census life-cycle estimate in the spring of 2019; however, they released the update on July 15, 2019. GAO will review the released documentation to see whether the revised estimate will address the recommendations. To ensure that future updates to the life-cycle cost estimate reflect best practices, it will be important for the Bureau to implement GAO's recommendation related to the cost estimate. Over the past decade, GAO has made 107 recommendations specific to the 2020 Census to help address these risks and other concerns. The Department of Commerce has generally agreed with these recommendations and has taken action to address many of them. However, as of July 2019, 32 of the recommendations had not been fully implemented. While all 32 open recommendations are important for a high-quality and cost-effective enumeration, 10 are directed at managing the risks introduced by the Bureau's planned innovations for the 2020 Census. To ensure a high-quality and cost-effective enumeration, it will be important for the Bureau to address these recommendations. Over the past decade, GAO has made 107 recommendations specific to the 2020 Census to help address issues raised in this and other products. The Department of Commerce has generally agreed with the recommendations. As of July 2019, 32 of the recommendations had not been fully implemented.", "document_type": "gao"}
{"report": "Congress created a multi-agency framework that established agency responsibilities for securing the nation’s transportation systems. Following the terrorist attacks of September 11, 2001, the Aviation and Transportation Security Act, enacted in November 2001, established TSA as the federal agency with primary responsibility for transportation security. Within this framework, two components of DHS—TSA and Coast Guard—are responsible for most transportation security activities. TSA is the primary federal agency responsible for security in all modes of transportation, including civil aviation, passenger and freight rail, highway and motor carrier transportation, and pipeline transportation systems. Coast Guard is the lead federal agency responsible for maritime transportation security, though TSA plays a role in managing, for example, credentialing for workers at seaports. TSA and Coast Guard’s regulatory authorities vary across modes, which affects how transportation security activities are planned for and implemented. For example, TSA and Coast Guard exercise more regulatory authority over (and, in some cases, have operational responsibility for) the aviation and maritime modes pursuant to their respective statutory authorities. In the aviation mode, TSA has operational responsibility for the screening of passengers and property transported on aircraft, but also imposes and enforces security requirements established through regulation on air carriers and other industry stakeholders. Similarly, Coast Guard has responsibility for ensuring that maritime vessels and facilities are compliant with applicable security requirements. TSA’s statutory responsibilities for the surface transportation modes, however, are generally less prescriptive. With respect to these modes, TSA works with transportation operators on a broad set of risk-based activities such as training, information sharing, and community outreach within a collaborative environment. For example, in freight rail TSA and its partners undertake collaborative efforts to establish security priorities, identify vulnerabilities and capability gaps, and reduce risks. Freight rail operators, meanwhile, engage in cooperative and independent security initiatives to assess risks and refine security plans. Other federal agencies are involved in transportation security, but to varying degrees. At the department level, DHS is responsible for providing strategic guidance, directing a national unity of effort, and coordinating security across critical infrastructure sectors. CBP manages programs designed to secure cargo and ensure intermodal transportation security, among other things. CBP activities include programs to encourage trade partners to implement security best practices and identify high-risk shipments and travelers before they reach U.S. ports of entry. DOT also has some transportation security responsibilities, which we describe below. Figure 1 illustrates agencies’ activities across transportation modes. Federal policies and plans establish specific coordination mechanisms and activities for transportation security. Specifically, in accordance with the Homeland Security Act of 2002, as amended, DHS created the National Infrastructure Protection Plan to guide the national effort to manage risk to the nation’s critical infrastructure, including through coordination of agencies and various critical infrastructure sectors, including transportation systems. Under this structure, DHS and DOT are co-Sector-Specific Agencies for the Transportation Systems Sector. DHS delegated its sector responsibilities to TSA and Coast Guard. Within the transportation systems sector, agencies and stakeholders charter councils for individual transportation modes as well as the sector as a whole. Sector Coordinating Councils and Government Coordinating Councils for each critical infrastructure sector provide forums for promoting efficient collaboration within the sectors. Further, the Sector- Specific Agencies are to develop, in close collaboration with Sector Coordinating Councils and other sector partners, a sector-specific plan that tailors the National Infrastructure Protection Plan to the specific characteristics and landscape of each critical infrastructures sector. Under the Transportation Systems Sector-Specific Plan, DOT and DHS, through TSA, and Coast Guard, coordinate with infrastructure owners and operators, provide technical assistance, and carry out incident management responsibilities. CBP is also a permanent member of the Aviation Government Coordinating Council. The Final Report of the National Commission on Terrorist Attacks Upon the United States (9/11 Commission Report), released in July 2004, identified concerns with aspects of transportation security planning, including the lack of an integrated strategic plan for the transportation sector. The Commission found that the screening of passengers and their property at airports accounted for the majority of transportation security investments, leaving vulnerable other facets of transportation security, such as cargo, general aviation, and surface transportation. The Commission recommended that the U.S. government identify and evaluate the transportation assets that need to be protected, set risk- based priorities for defending them, select the most practical and cost effective means of doing so, and then develop a plan, budget, and funding source to implement the effort. Congress subsequently passed the Intelligence Reform and Terrorism Prevention Act of 2004 (Intelligence Reform Act), which directed the Secretary of DHS to develop, prepare, implement, and update, as needed, a National Strategy for Transportation Security and transportation modal security plans. The statute further directs the Secretary of Homeland Security to work jointly with the Secretary of Transportation to develop, revise, and update the national strategy and transportation modal security plans. Within DHS, responsibility for such strategic planning had been delegated by the Secretary of Homeland Security in May 2003 to TSA for transportation security across all modes of transportation and to Coast Guard for maritime security, specifically. The Intelligence Reform Act called for a national strategy that was to include elements that aligned with the Commission’s recommendation. Table 1 illustrates parallels among the Commission’s multi-part recommendation, the Intelligence Reform Act, as amended, and the 2018 national strategy. Consistent with its underlying statute, the national strategy states that it is the governing document for federal transportation security efforts, and lays out a number of areas where it can govern those efforts. For example, the national strategy states that it contributes to departmental budgetary processes by applying multiple information sources to determine priorities and capability gaps that influence resource allocation decisions and budget projections across federal agencies. Further, the national strategy is intended to support out-year programming and budgeting by measuring progress toward achieving the security outcomes for funded activities. The national strategy states that its risk-based priorities help to narrow capability gaps and raise the security baseline. The risk-based priorities in the national strategy are also intended to inform security decisions about the types of activities government and industry modal security officials should pursue to address terrorism risks. The national strategy includes modal security plans as appendixes—also consistent with its underlying statute—and other, separate, statutorily required national strategy documents as annexes TSA determined were appropriate to include. The 2018 National Strategy for Transportation Security is generally consistent with desirable characteristics of an effective national strategy. In 2004, we reported that national strategies are not required to address a single, consistent set of characteristics, and they contain varying degrees of detail based on their different scopes. We have previously identified a set of desirable characteristics that we believe would provide additional guidance to responsible parties for developing and implementing the strategies—and to enhance their usefulness as guidance for resource and policy decision-makers and to better ensure accountability. Our analysis of the 2018 National Strategy for Transportation Security found that it is fully consistent with two of the six desirable characteristics of an effective national strategy and partially consistent with four, as summarized in table 2. We found that supporting documents of the national strategy (such as a planning guide, project plan, and budget document) include additional elements of desirable characteristics that are not currently included in the strategy. For example, the national strategy’s guidance document describes the methodology for developing the strategy. TSA officials indicated that as they develop the 2020 national strategy, they will take steps to incorporate additional elements of desirable characteristics. The national strategy plays a limited role in guiding federal transportation security efforts. Agencies rely instead on various agency- or mode- specific documents that DHS and DOT officials stated overlap with the national strategy. Similarly, agencies do not consult the national strategy to allocate resources for their federal transportation security efforts. They instead make such decisions based on various strategy documents and department and agency guidance, which the national strategy may inform to varying degrees. TSA identifies the national strategy as the governing document for federal transportation security efforts, consistent with its underlying statute; however, agency officials generally do not use it to guide their efforts and had disparate views about its functional role given overlapping strategic documents. The 2018 national strategy states: “While the strategy presents a whole community plan for reducing the risks to transportation from terrorist attacks, it is, as mandated, the governing document for federal transportation security efforts.” Officials representing TSA aviation, Coast Guard, TSA intermodal, and DOT stated that they did not use the national strategy to guide their efforts; TSA surface officials stated that it generally did guide surface transportation activities. Officials from TSA’s Strategy, Policy Coordination, and Innovation office, which coordinates the national strategy’s development, said that although the national strategy does not drive transportation security activities, it does inform such activities as they related to risk-based priorities. Although the national strategy states that it is to be the governing document for transportation security efforts, TSA strategy officials described it as a catalogue of transportation security activities. The vast majority of the activities and performance measures reported in the national strategy came from ongoing reporting mechanisms such as the DHS Annual Performance Report and TSA voluntary surface security assessments, according to TSA and Coast Guard officials. Therefore, the national strategy did not affect the number of activities or types of programs that agencies undertook, according to TSA and Coast Guard officials. Instead, the national strategy summarized information about current transportation security goals and performance as opposed to guiding such decisions. TSA surface and aviation, Coast Guard, and DOT officials stated that several different strategies and planning documents with similar areas of focus resulted in redundancy or overlap with the National Strategy for Transportation Security. We have reported that when overlap exists there may be opportunities to increase efficiency. For example, communicating the use of overlapping documents could promote efficiency in creating and using strategies to make transportation security- related decisions. Figure 2 shows the National Strategy for Transportation Security and numerous other documents, including several identified in the 2018 national strategy, that guide transportation security decisions. For specific examples of strategies used by each component, see appendix I. As shown in figure 2, the National Strategy for Transportation Security exists among more than a dozen other national-level strategic documents without a hierarchical alignment indicating how they interact or supersede each other. Officials from TSA’s strategy office stated that they view the functional role of the national strategy as informing transportation modes’ activities where applicable, and that transportation officials should use it to ensure consistency of effort across activities. Transportation officials had differing views on the varying role of the national strategy, as described below: TSA Aviation: The national strategy keeps security operations on track and aligned with priorities, but officials used the national strategy more for reference than to guide program or planning decisions. TSA officials stated that aviation policy is regulatory in nature, meaning policy is driven by requirements established through statute and regulation rather than by the national strategy. TSA aviation officials also stated that they could not provide an example of where the national strategy was used to make specific decisions or actions. Coast Guard: The national strategy informs federal partners of Coast Guard’s maritime transportation security activities, but Coast Guard officials stated that the national strategy does not require them to take on activities they are not already doing; instead, it puts those transportation security activities in context. Coast Guard officials stated that the national strategy did not drive decisions or activities. TSA Surface: The national strategy generally guides transportation security activities and drives a common understanding around goals for both TSA officials and industry partners. TSA surface officials stated that the need for voluntary cooperation and engagement makes the alignment of priorities with national strategy more valuable in the surface mode. TSA surface officials stated that they use the national strategy to guide their implementation of federal transportation security programs. Specifically, TSA surface officials stated that they use the national strategy to determine areas of focus for training and exercise programs. DOT: The national strategy delineates the transportation roles and responsibilities through the lens of terrorism, giving it value as a tool for communicating and coordinating within the transportation systems sector rather than as a planning tool. DOT officials stated that they did not use the national strategy as a major factor to prioritize budget decisions and cannot assign a causal relationship between the national strategy and policy. Officials from TSA’s strategy office stated that they created the national strategy to respond to legislative requirements; however, they had not fully considered or communicated to key stakeholders how the national strategy would functionally guide federal efforts. Officials acknowledged that it could be helpful to communicate this information to stakeholders as they develop future iterations of the national strategy. Such communication would be consistent with federal internal control standards, which state that management should externally communicate the necessary quality information to achieve the entity’s objectives. TSA has made efforts over the years to streamline and consolidate reporting requirements of the national strategy with similar documents. For example, in August 2010, TSA sent a letter to notify Congress that it was streamlining the national strategy and several other documents by incorporating them into the Transportation Systems Sector Annual Report. The letter stated that streamlining strategic planning and reporting requirements improves their usefulness and reduces federal government and stakeholder confusion. Similarly, TSA surface officials stated that they have attempted to consolidate their reporting requirements by integrating two strategies focused on mass transit and freight rail into the national strategy. Officials stated that those strategies were published as separate annexes in the 2018 National Strategy for Transportation Security in response to feedback, but had been integrated into the 2016 iteration of the national strategy. Officials from TSA’s strategy office said they believed the national strategy has value in providing a whole-of-government strategy for transportation security with a counterterrorism view. However, we have previously reported that the ultimate measure of the value of national security strategies is the extent to which they are useful as guidance in balancing homeland security priorities with other important, non- homeland security objectives. Though the national strategy lays out a number of areas where it can govern federal transportation security efforts, its unclear position among numerous strategic documents limits its ultimate value. For example, the risk-based priorities in the national strategy are intended to inform security decisions about the types of activities government and industry modal security officials should pursue to address terrorism risks. Instead, according to officials, the national strategy summarizes current transportation security activities within each mode and they generally use other documents to guide their transportation security decisions. By communicating to key stakeholders how the national strategy aligns with related strategies to guide federal efforts, stakeholders would be in a better position to use the strategy as a whole-of-government approach to preventing terrorist attacks. Officials representing TSA, Coast Guard, and DOT identified various documents and strategies as guiding resource decisions. TSA budget representatives stated that specific budgetary decisions and trade-offs result from other strategy documents, such as the TSA Strategy and Administrator’s Intent. TSA budget officials indicated a link between the National Strategy for Transportation Security and the budget process because other strategy documents incorporated the national strategy. Similarly, Coast Guard officials stated that they broadly consider the national strategy, the DHS Resource Planning Guidance, and other documents during their budgeting process. However, Coast Guard officials could not speak to the influence of the national strategy in particular. When asked about how the national strategy influences resource decisions, agency officials explained: TSA Aviation: The national strategy has not influenced any specific resource allocation decisions. Coast Guard: The national strategy is part of the broader budget process, but officials could not speak to its particular influence or provide examples of the national strategy changing the direction of maritime security activities. TSA Surface: The national strategy does not provide specific direction on resource allocation decisions. The national strategy provides a guidepost for where TSA wants to expend effort and can provide guidance during times of limited budgets and personnel, though officials did not provide specific examples of cases in which this occurred. DOT: The national strategy does not play any role in the department’s budget process. The national strategy identifies the creation of out-year budgets as a challenge. For example, the statute under which TSA develops the national strategy provides that it is to include both a 3-year and 10-year budget for federal transportation security programs that will achieve the priorities of the national strategy. However, the national strategy recognizes that it does not provide 3-year and 10-year budget information due to the challenge of anticipating future transportation security programming needs and aligning budget projections across multiple departments and agencies. To address this challenge, the national strategy aims to contribute to budgetary processes by applying multiple information sources to determine priorities and capability gaps that influence resource allocation decisions and budget projections. Further, the national strategy is to support budgeting by measuring progress towards achieving the security outcomes for funded activities. TSA officials explained that, rather than provide the 3-year and 10-year budget, TSA designed its budget process to align with, and be consistent with, the department’s five-year budget cycle set out in the Homeland Security Act. The national strategy explains that, accordingly, agency budget information will continue to be reported through their regular budget processes. TSA officials told us that they have been reporting budget information to Congress this way since before they produced the initial national strategy, and that Congress has not raised concerns with this approach. TSA, Coast Guard, and DOT officials told us they did not use the national strategy to make specific budget or resource allocation decisions because they did not believe the national strategy should direct those decisions. Officials from TSA’s strategy office confirmed that, in their view, the national strategy was not intended to guide resource decisions. TSA officials collaboratively developed the 2018 National Strategy for Transportation Security, which generally reflected risks identified in existing TSA and Coast Guard documents. TSA managed the creation of the national strategy by seeking input from stakeholders with responsibilities in each of the three transportation modes as well as intermodal transportation. Specifically, TSA officials sent out three data calls for information and feedback to officials at TSA, Coast Guard, and DOT responsible for providing information. Each data call built upon the prior one and provided the modal officials multiple opportunities to revise and edit their data. In addition, TSA officials sent the data calls to the Transportation Modal Government Coordinating Councils and recounted sending them to other groups, such as Sector Coordinating Councils. Because of the numerous agencies involved and the length of the development and review process, TSA began development of the 2018 national strategy before they submitted the 2016 national strategy to Congress. TSA planning officials stated that they encouraged officials responsible for overseeing implementation of transportation programs to help develop the strategy so TSA could leverage the expertise of each individual mode. TSA delegated the responsibility of identifying performance measures, activities, and related information to officials in each of the modes. These modal officials in turn contacted officials implementing transportation security programs to gather information and metrics related to their programs in the mode-specific appendixes, as well as coordinate general feedback on the national strategy’s base plan. TSA recommended that modes leverage activity and performance information already reported, to the extent possible. This allowed the national strategy to be efficiently updated according to TSA planning officials, which is crucial to TSA’s planning timeline of developing the national strategy every two years. Officials representing TSA surface and aviation, Coast Guard, and DOT confirmed their participation in the data calls and national strategy development. TSA surface officials also stated that they leveraged existing collaboration and coordination mechanisms to provide industry and stakeholder feedback, such as government coordinating councils and sector coordinating councils. Senior leadership then reviewed the information to ensure that it did not conflict with other strategies that agencies use to guide activities, according to TSA and DHS officials. We compared TSA’s work collaborating with other agencies to produce the national strategy with key practices we have identified for collaboration and found that TSA generally aligned the national strategy development with selected key practices. Specifically, selected leading practices call for agencies to collaborate by identifying 1) leadership, 2) clear roles and responsibilities, and 3) participants. TSA’s leadership developing the national strategy, working jointly with DOT, is identified in agency documentation and a DHS memo which delegates this authority. TSA officials provided clear roles and responsibilities to agencies asked to provide data through the data calls that supported the 2018 national strategy development. In addition, they included all relevant participating agencies in the process and provided a clear method of decision-making. TSA officials stated that it was a challenge to get input from agencies that do not consider their main function to be transportation security, such as CBP. Officials from CBP—which is responsible for carrying out multiple activities related to air cargo and intermodal security in the 2018 national strategy—stated that they were not involved with the 2018 national strategy. CBP officials acknowledged that their programs to inspect cargo played a role in transportation security; however, they said they viewed their responsibilities as separate. For example, CBP officials stated that they are responsible for verifying the security of some cargo transported on planes but not the security of the planes themselves. However, TSA officials stated that they involved two individuals from CBP and will continue to reach out to CBP for information and involvement in the development of the 2020 national strategy. TSA officials stated that they are committed to collaborative development of the national strategy, and have taken an extra measure to seek comments from the public to inform the development of the 2020 national strategy using the Federal Register. In addition to agency collaboration, the development of the 2018 national strategy centered on agencies incorporating risks listed in their risk assessments. TSA officials from surface and aviation modes stated that they relied primarily on the Transportation Sector Security Risk Assessment; while Coast Guard officials relied on the National Maritime Strategic Risk Assessment and the National Maritime Terrorist Threat Assessment. TSA officials stated that they did not have documentation of the risks considered for the intermodal information for the 2018 strategy because the TSA official responsible for its development was no longer with the agency. However, officials stated that they are considering risks for the 2020 national strategy that are described in the Transportation Sector Security Risk Assessment and National Risks Estimate and provided documentation of these considerations. We found that, in general, the risk-based priorities highlighted in the national strategy aligned with the risks identified in the assessments. For example, the 2018 national strategy identified the prevention of insider threats as part of a risk-based priority in its base plan and aviation- specific appendix. In addition, the aviation-specific appendix identified an activity, outcome, and performance measure aimed at addressing this threat. This aligns with the identification of insider threats as a key part of risks specified in TSA’s 2017 risk assessment. In addition, TSA and Coast Guard officials stated that they also considered and included emergent threat information—for example, new threats presented by cybersecurity. They decided to include these threats as a result of ongoing development of strategy documents both in TSA and across the interagency community, according to TSA officials. The development of risk information in the 2018 national strategy remained within the context of each mode. TSA’s Transportation Sector Security Risk Assessment does provide information to compare risks across aviation and surface modes; however, that information is not included in the 2018 national strategy. Similar information related to Coast Guard risks is also not included in the 2018 national strategy, though available in Coast Guard risk assessments. The national strategy lays out areas where it could inform decision-making across modes; however, the information about transportation activities’ effectiveness does not currently lend itself to meaningful comparisons. For example, transportation security activities in the 2018 national strategy report outcome and performance measures, but not targets or results. TSA officials stated that they are developing the 2020 national strategy to include performance measures for activities to respond to risks, which will be the second iteration of measures in the national strategy. Corresponding performance results on activities that respond to risk- based priorities will be directly reported to Congress through annual reports on the progress of the national strategy’s implementation. Though this is not the same as providing cross-modal risk information, it would enable decision-makers to hold risk reduction activities accountable for results that they were intending to achieve, according to TSA officials. In accordance with statutory requirements, the National Strategy for Transportation Security is to be the governing document for federal transportation security efforts. However, its unclear position among numerous related strategies has clouded its value in guiding federal efforts. In light of other strategies and governance documents, DHS, in consultation with DOT, can better communicate the applicability of the National Strategy for Transportation Security so that key stakeholders have clear direction on how to rely on the national strategy. As TSA develops future iterations of the national strategy, key stakeholders would be better positioned to use it if the departments communicate how the national strategy aligns with related strategies. In the absence of such communication, transportation security stakeholders may continue to miss opportunities to use the national strategy as part of a whole-of- government approach to preventing terrorist attacks. The Secretary of Homeland Security should, in consultation with the Secretary of Transportation, communicate to key stakeholders how the National Strategy for Transportation Security aligns with related strategies to guide federal efforts as it develops future iterations of the national strategy. (Recommendation 1) We provided a draft of this report to DHS and DOT for review and comment. In written comments, which are included in appendix II and discussed below, DHS concurred with our recommendation and described actions taken to address it. DHS and DOT also provided technical comments, which we have incorporated into the report, as appropriate. DHS stated that the 2020 national strategy will elevate alignment language from the 2018 national strategy modal plans and better explain how the national strategy relates to newly issued strategies, among other things. These updates to the 2020 strategy are a positive step, and DHS should ensure that it further clarifies alignment language in the modal plans and communicates both newly issued and previous strategies alignment with the national strategy. Further communication about related strategies will provide better direction for key stakeholders on how to use the national strategy in relation to other strategies. We are sending copies of this report to the appropriate congressional committees, the acting Secretary of the Department of Homeland Security, and the Secretary of the Department of Transportation. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-8777 or RussellW@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in Appendix III. In addition to the contact named above, Kevin Heinz (Assistant Director), Michelle Serfass (Analyst-in-Charge), Chuck Bausell, Benjamin Crossley, Elizabeth Dretsch, Andrew Lobel, Tom Lombardi, Sarah Veale, and Adam Vogt made key contributions to this report.", "summary": "In recent years, the nation's transportation systems facilitated over 5 trillion miles of passenger travel annually while moving billions of tons of cargo. The scale and scope of these systems make them targets for terrorist attacks. Congress directed DHS to work jointly with DOT to develop, revise, and update a biennial National Strategy for Transportation Security that governs federal transportation security efforts. The FAA Reauthorization Act of 2018 includes a provision for GAO to evaluate the extent to which the most recent strategy is reflected in relevant federal transportation security efforts. This report examines the extent to which the 2018 strategy (1) guides relevant federal transportation security efforts, including resource allocation, and (2) incorporates input across transportation modes and risk information, among other things. To conduct this work, GAO reviewed relevant transportation security documentation, interviewed officials within DHS and DOT on the development and use of the strategy, evaluated interagency collaboration during the development of the national strategy, and analyzed the national strategy's incorporation of risk information. The 2018 National Strategy for Transportation Security generally does not guide federal efforts due in part to its unclear alignment with several strategies that also inform federal transportation security efforts. The Department of Homeland Security (DHS)—primarily through the Transportation Security Administration (TSA)—developed the national strategy, consistent with congressional direction, to govern federal transportation security efforts. However, TSA and Department of Transportation (DOT) officials all identified some degree of redundancy or overlap regarding the role of the strategy in light of other transportation security strategies such as the National Strategy for Aviation Security. Agencies reported using the national strategy for reference, context, and general coordination, but not for driving program activities. Agencies instead use separate strategies and plans to guide program and resource decisions. Similarly, agencies in DHS and DOT (key stakeholders of the strategy) use various strategy documents to allocate resources for federal efforts, which the strategy may inform. However, DHS has not communicated how the strategy aligns with related strategies to guide these efforts. By doing so, federal stakeholders would be better positioned use the national strategy as part of a whole-of-government approach to preventing terrorist attacks. TSA effectively incorporated input from stakeholders and considered risk information to develop the 2018 National Strategy for Transportation Security. TSA iteratively updated the biennial strategy by incorporating input across transportation modes and feedback from stakeholders in a manner that generally met GAO's leading practices for collaboration. For example, TSA clearly communicated roles and responsibilities regarding the strategy development process for participating agencies. In addition, the strategy compiles risks identified for each transportation mode in other strategic planning documents. TSA strategy development officials stated that they also included emergent risk information, for example cybersecurity risks. The security risks identified in these risk assessments, in general, aligned with the risk-based priorities highlighted in the strategy. GAO recommends that DHS should, in consultation with DOT, communicate to key stakeholders how the National Strategy for Transportation Security aligns with related strategies to guide federal efforts. DHS concurred with the recommendation.", "document_type": "gao"}
{"report": "The CSBG program is intended to focus on three overall (national) goals: reducing poverty, empowering low-income families and individuals to become self-sufficient, and revitalizing low-income communities. The program is administered by OCS within the Administration for Children and Families (ACF) at HHS. CSBG was an outgrowth of the War on Poverty of the 1960s and 1970s, which established the Community Action program under which the nationwide network of local community action agencies was developed. The federal government had direct oversight of local agencies until 1981, when the CSBG program was established and states were designated as the grant recipients. OCS and states now share responsibility for oversight of CSBG grantees. In fiscal year 2019, states received approximately $700 million of the total $725 million CSBG appropriation. Appendix II provides the funding amounts for each state. OCS distributes CSBG funding to states and they, in turn, distribute funds to over 1,000 local agencies. Most of these local agencies receive funding from a variety of federal, state, and private sources. In fiscal year 2017, the latest data available, local agencies received about $9 billion from all federal sources, including about $700 million from CSBG. Other federal programs providing funding include Head Start, the Low Income Home Energy Assistance Program (LIHEAP), the Community Development Block Grant (CDBG), the Child Care and Development Block Grant, Temporary Assistance for Needy Families, and the Social Services Block Grant (see fig. 1). Programs administered by ACF contributed about $6.6 billion of the funds provided to local agencies. CSBG funding can be used broadly, allowing state and local agencies flexibility to provide services tailored to organizational and community needs. CSBG funds can be used by local agencies to provide services to participants in their programs and fill gaps in the funding provided by other means. For example, local agencies may use CSBG funds to support a position for a staff member who determines the service needs of potential participants and connects them with the appropriate services—a position that would not be an allowable expense under the funding rules of other federal programs, according to a local agency official we interviewed. Local agencies have also used CSBG funding to leverage other public and private resources to support a variety of initiatives, such as Head Start programs, low-income energy assistance programs, and low-income housing. OCS monitors all states receiving grant funds to ensure that they are meeting the standards for federal grant programs set by the Office of Management and Budget (OMB) and the specific expenditure requirements for the program. The CSBG Act requires that states submit plans to OCS describing how they intend to use the funds to address the needs of the local community and annual reports detailing the actual use of funds, including information on state performance results and populations served. OCS is required by the CSBG Act to conduct compliance evaluations of several states each fiscal year to review the states’ use of CSBG funds, report to states on the results of these evaluations, and make recommendations for improvements. However, the CSBG Act does not specify the number of states subjected to an evaluation each year or the timeframe each state must undergo such evaluations. Following a compliance evaluation, states are required to submit a plan of action in response to any OCS recommendations. In addition to conducting compliance evaluations to assess states’ use of CSBG funds, OCS is required to submit an annual report to Congress. This annual report must include a summary of how states and local agencies had planned to use CSBG funds; how funds were actually spent, data on the number and demographics of those served by local agencies, and other information. The CSBG Act requires OCS to provide training and technical assistance to states and to assist them in carrying out corrective action activities and monitoring. OCS must reserve 1.5 percent of annual appropriations (in fiscal year 2019, this percentage totaled about $11 million of the total appropriation) for many activities, including training and technical assistance; planning, evaluation, and performance management; assisting states with carrying out corrective action activities; and oversight including reporting and data collection activities. The CSBG Act also requires that states complete several steps before terminating an underperforming entity. The state agency is required, among other things, to provide training and technical assistance, if appropriate, to help the agency correct identified deficiencies, review the local agency’s quality improvement plan, and provide an opportunity for a hearing. The entity can request a federal review of the state’s decision to reduce or terminate funding, which must be completed within 90 days of OCS’s receipt. During this period, the state is required to continue funding the entity until OCS responds to the request. The CSBG Act requires each state to designate a lead state agency to administer CSBG funds and provide oversight of local agencies that receive funds. States are required to award at least 90 percent of their federal block grant allotments to eligible local agencies, and to determine how CSBG funds are distributed among local agencies. States may use up to $55,000 or 5 percent of their CSBG allotment, whichever is higher, for administrative costs. States may use remaining funds for the provision of training and technical assistance, and other activities. In addition, states and local agencies that expend $750,000 or more in total federal awards are required to undergo an audit annually and submit a report to the Federal Audit Clearinghouse. The CSBG Act requires states to determine if local agencies meet the performance goals, administrative standards, and financial management requirements for the CSBG program. For each local agency, the CSBG Act requires the state to conduct: a full onsite review at least once during each 3-year period; an onsite review of each new local agency following the completion of the first year receiving CSBG funds; followup reviews including prompt return visits to local agencies that fail to meet goals, standards, and requirements established by the state; and other reviews as appropriate, including reviews of local agencies found to have had other grants terminated for cause. For states to receive CSBG funding, they must submit an application and state plan at least biennially describing, among other things, how they will use CSBG funds to accomplish various things such as helping families and individuals to achieve self-sufficiency, find and retain meaningful employment, and obtain adequate housing. Within their state plan, states must attest that (1) funds will be used to address the needs of youth in low-income communities; (2) funds will be used to coordinate with related programs; and (3) local agencies will provide emergency food-related services. States must also complete annual reports that include fiscal, demographic, and performance data. In their state plans, states must provide an assurance that all local agencies will submit a community action plan that includes a community needs assessment for the community served. In addition, local agencies must administer the CSBG program through a three-part board, consisting of one-third elected public officials and at least one-third representatives of the low-income community, with the balance drawn from officials or members of the private sector, labor, religious, law enforcement, education or other groups in the community served. The Government Performance and Results Act of 1993 (GPRA) as enhanced by the GPRA Modernization Act of 2010 (GPRAMA) focuses federal agencies on performance by, among other things, requiring agencies (including HHS) to develop outcome-oriented goals and a balanced set of performance indicators, including output and outcome indicators as appropriate, to assist agencies in measuring or assessing their progress toward goals. OMB provides guidance to federal executive branch agencies on how to prepare their strategic plans in accordance with GPRA requirements. We have reported that strategic planning requirements established under GPRA and GPRAMA can also serve as leading practices for strategic planning at lower levels within federal agencies. Federal standards for internal control help to ensure efficient and effective operations, reliable financial reporting, and compliance with federal laws. Internal controls help government program managers achieve desired results through effective stewardship of public resources. Such interrelated controls comprise the plans, methods, and procedures used to meet missions, goals, and objectives. Internal controls support performance-based management and should provide reasonable assurance that an organization achieve its objectives of (1) effective and efficient operations, (2) reliable reporting, and (3) compliance with applicable laws and regulations. With regard to performance measurement for state and local agencies, the CSBG Act requires OCS, in collaboration with states and local agencies, to facilitate the development of one or more model performance measurement systems which may be used by states and local agencies to measure their performance in fulfilling CSBG requirements. Each state receiving CSBG funds is required to participate in and ensure that all local agencies in the state participate in either a performance measurement system whose development was facilitated by OCS or in an alternative system approved by OCS. OCS developed the Results Oriented Management and Accountability (ROMA) performance management approach that states and local agencies follow when overseeing programs and measuring their performance in achieving their CSBG goals. In 2012, OCS began four initiatives to update how it oversees the performance of the CSBG program, and as of April 30 2019, OCS had implemented all four of the initiatives, which include: an updated ROMA process for program management, 58 organizational management standards for local agencies, new federal and state accountability measures, and an updated annual report format where oversight and performance information from states is collected in an automated online data system. In addition, OCS developed the CSBG Theory of Change which illustrates how the core principles of the CSBG program, the performance management framework, and services and strategies offered with CSBG funds relate. The three national goals established under the CSBG Theory of Change are similar to the three national goals identified in the CSBG Act, but are not identical. The three goals under the CSBG Theory of Change are: 1. individuals and families are stable and achieve economic security, 2. communities where low-income people live are healthy and offer 3. people with low incomes are active in their community. OCS and states are responsible for conducting oversight activities to ensure that CSBG recipients use the funds in accordance with the CSBG Act, which includes ensuring that the funds are used in line with the grant’s three national goals related to addressing the causes and conditions of poverty. Our review of oversight efforts during fiscal years 2016 and 2017 for the select states showed that OCS and states conducted required oversight activities, as well as additional oversight activities, and provided training and technical assistance to help CSBG recipients meet CSBG program requirements. Our review of file documentation for six selected states where OCS conducted compliance evaluations during fiscal years 2016 and 2017, and six selected states where OCS conducted routine oversight, showed that OCS identified primarily administrative issues, but in some instances identified non- compliance and other more serious issues that required corrective actions that states took action to resolve. We largely found similar results in our review of the selected states’ onsite and routine oversight activities for local CSBG funds recipients for the same time period. Beyond findings of an administrative nature, a fiscal year 2017 OCS compliance evaluation found that one state did not conduct required monitoring of its eligible entities during fiscal year 2015. Also, one state identified financial mismanagement, which resulted in termination of a local grantee from the CSBG program. Additionally, we found that OCS and states provided training and technical assistance to help CSBG recipients meet requirements. OCS officials conducted onsite compliance evaluations, in addition to other oversight activities, for 12 states using a risk assessment and prioritization process during fiscal years 2016 and 2017. We reviewed six of these 12 states and found that a majority of errors identified by OCS were administrative. The CSBG Act requires OCS to conduct compliance evaluations for several states each year. Since fiscal year 2009, OCS has conducted onsite compliance evaluations in five to seven selected states each year, in addition to the routine oversight it conducts for all the states. According to OCS officials, the number of states visited each year depends upon available resources. OCS primarily bases its selection of states for onsite compliance evaluations on a risk assessment conducted using a scoring tool. The scoring tool generates a risk score of 1 to 5 for each state using a number of measures, as shown in figure 2. The various factors used in developing the total risk score are weighted to ensure the most significant risk indicators and prioritization factors have the most impact on the selection of states for onsite monitoring. The list of risk factors was developed by OCS in response to a recommendation from our 2006 report in which we found that OCS did not systematically use available information to assess risk to focus its monitoring resources on states with the highest risk. According to OCS officials, OCS rarely visits states that they identify as low risk or states that have very few local agencies as grantees, and they try to not visit the same state within 3 years of their last visit. OCS officials told us that monitoring resources limit their ability to reach all of the states for onsite review. We found that, since fiscal year 2008, eight states have not received an onsite evaluation and 10 had been visited twice. According to agency officials, the risk assessment is part of a larger risk assessment and prioritization process designed to direct monitoring resources over multiple years. After determining risk under the scoring tool, OCS considers several other factors and may place a higher priority on states with lower risk scores when selecting states for onsite compliance evaluations. Agency officials said such factors include: size of the CSBG award, findings from single audits, the rate at which the state spends its CSBG funds, time since the last OCS visit, and feedback from the OCS program manager using information gathered from the quarterly calls with the states. For states selected for onsite compliance evaluations, we found that OCS conducts a comprehensive review of each of the state’s plan and annual reports and examines the state’s supporting documents to determine if that state is meeting the requirements of the CSBG program. Although OCS reviews the plans for all 56 states as part of its routine oversight efforts, during the onsite visit the agency also conducts interviews with staff and examines state statutes or regulations and supportive information, such as financial ledgers and oversight procedural manuals. OCS also reviews the state’s grant funding to determine if the state allocated the funds in accordance with the requirements of the CSBG program. Additionally, OCS reviews each state’s fiscal controls and accounting procedures and associated documents to assess the financial integrity of the state’s process for drawing down federal funds, providing funds to local agencies, and reporting financial information. For example, OCS officials may review the state agency’s bookkeeping system and accounting software. In our review of OCS’s file documentation for the six selected states, we found OCS generally identified administrative errors, but in some instances identified issues of non-compliance and other issues that the states took action to resolve. For example, during its fiscal year 2017 onsite visit to Louisiana, OCS found that Louisiana did not implement procedures to monitor and track prior year single audit findings for corrective action and issue management decisions as required. To address this concern, the state assigned a member of its staff to execute these duties and submitted a copy of the Single Audit Process and audit log to OCS. Additionally, OCS found that Louisiana did not visit any of its 42 local agencies in fiscal year 2015 because of limited capacity such as staffing shortages, among other non-compliance issues. OCS determined that Louisiana addressed this issue by visiting all of the local agencies before the end of fiscal year 2017. Also, in a fiscal year 2016 onsite visit to Indiana, OCS found that the state agency did not submit a required financial report to account for CSBG expenditures within established timeframes in two consecutive fiscal years—2014 and 2015—due to the lack of a process to ensure the timely submission of the report. OCS also found that the financial report for fiscal year 2014 contained incorrect amounts for certain expenditures. The Indiana state agency responded to the issues by developing formal written procedures regarding the preparation and submission of financial reports. In addition, for the six selected states, we found that OCS had assessed state plans and annual reports to ensure that the states were complying with the programmatic, financial, and administrative requirements of the CSBG program, as outlined in the CSBG Act. In our review of the selected states, we found that during fiscal years 2016 and 2017, OCS conducted routine reviews and other oversight activities to assess states’ use of CSBG funds. We selected six states (Alaska, Colorado, Kentucky, Mississippi, North Dakota, and Rhode Island) for our review of file documentation of OCS’s routine reviews. We found that for these six states, the routine reviews consisted of OCS reviewing all state plans and annual reports to determine if the state completed all sections of the plan and provided information about how it would achieve the goals of the program. In our review of file documentation for the six states, we found that OCS requested states to provide additional details about their plans; however, like the issues identified in the onsite compliance evaluations, the issues on which OCS commented were primarily administrative. For example, in fiscal year 2016, OCS reviewed Colorado’s 2016 annual plan and requested that the state provide additional details on plans to modify its organizational standards. Also, in its fiscal year 2017 review, OCS requested that Alaska provide additional information in its annual plan to explain how the state would prioritize providing services to individuals based on their income. We found that the states addressed OCS’s comments. OCS officials told us that they used quarterly calls as a part of their routine oversight. Agency officials told us that they generally use quarterly calls to discuss the state plans and the CSBG program broadly, and review the annual reports. OCS officials also told us that OCS uses these calls to update states on issues that have significant impact or importance on the successful operation of the CSBG grantees. In some cases, OCS program specialists may use the quarterly calls to identify areas where the state may be struggling and to discuss ways to address those issues. In addition, OCS officials stated that OCS program specialists will work with states to assist with developing work plans or reviewing corrective action procedures for high-risk local agencies. All three states we visited (New York, North Dakota, and Texas) conducted onsite visits to local agencies at least once every 3 years as required by the CSBG Act, and conducted routine oversight activities. In response to our June 2006 recommendation, OCS issued guidance clarifying that states must conduct an onsite review of each local agency at least once every 3 years. Besides the triennial onsite reviews, the law requires states to conduct: (1) follow up reviews including prompt return visits to local agencies that fail to meet state goals, standards, and requirements, (2) an onsite review of new local agencies following the completion of the first year receiving CSBG funds, and (3) other reviews as appropriate, including reviews of local agencies found to have had other grants terminated for cause. Each of the states we visited had developed oversight policies and procedures that included information on how often CSBG programs should be reviewed onsite and what program operations should be covered during onsite visits; two states provided sample forms or instructions on what forms to use to record findings. For example, each state’s policies and procedures established the frequency of onsite visits: New York and Texas conduct the visits at least once every 3 years and North Dakota conducts them once every 2 years (see table 2). The selected states’ policies and procedures also specified that state officials assess local agency financial controls, review financial records and client files, and review local agency governance. They also described information about actions state officials were required to take when they identified deficiencies in a local agency’s operations. For example, in all three of the states we visited the policies and procedures required state officials to notify local agencies of deficiencies in writing. Our findings from the two local agencies we visited in each of the three states showed that state officials identified a variety of issues during their reviews, but none that required those local agencies to lose their CSBG funding (see table 3). Generally, we found that the issues identified could be characterized as fiscal, governance, or administrative. Fiscal issues included improper use of funds. For example, state officials in one selected state found that a local agency had improperly used a small amount of CSBG funds to purchase a grill for agency activities. Governance-related findings included issues with both the composition and manner of selecting the local agency’s CSBG Board of Directors members. For example, in Texas, state officials cited one local agency for not complying with the CSBG Act’s requirement regarding the structure of its Board. Also, North Dakota cited a local agency for not having the required representation of low-income individuals on its Board. Administrative issues included recordkeeping of information on participants. For example, Texas cited a local agency for inaccurately reporting a program participant as having transitioned out of poverty. The state agency found that the participant’s file did not contain all of the required documentation needed to show that the participant had maintained a certain income level for a 90-day period. The state agency officials we spoke with told us that their reviews sometimes identified more serious issues that resulted in local agencies being terminated from the program. For example, Texas terminated two local agencies’ CSBG funding due to financial mismanagement that was uncovered during state monitoring of the local agencies. Texas officials noted that the process for terminating local agencies with deficiencies was, for them, a prolonged process, in part because of the steps they took to provide technical assistance and work with agencies in an attempt to resolve issues before terminating them from the program. They told us they found it difficult to establish sufficient grounds for termination and, for one of the terminations, Texas officials continued to work with the agency for two years while also working with OCS. Texas officials told us that they found the guidance on terminations to be unclear. OCS officials acknowledged that the information memorandum they have developed on terminations provides broad guidance that covers a range of issues states might encounter, and may not have detailed guidance covering each situation. However, they noted that they work with states on a case by case basis, as they did with Texas, to provide guidance that is specific to each situation. State officials in the selected states told us that local agencies identified as having deficiencies are notified of those deficiencies and provided information on how to correct them. Further, our review of corrective actions required of selected local agencies by the states we visited showed that the local agencies addressed the concerns raised by the states. For example, Texas required a local agency that it found did not comply with CSBG Board requirements concerning membership to fill the vacancies on the Board and to provide the state a timeline for completing the required corrective actions. In addition to taking corrective actions, local agencies may be required to submit fiscal and programmatic reports more frequently when monitoring uncovers problems. For example, North Dakota’s policies and procedures indicate that monthly reports may be required of local agencies that have been found to have financial recordkeeping problems. We also found that state agency officials in our three selected states conducted onsite reviews more frequently than the once every 3 years requirement, as well as routine offsite reviews. For example, New York conducted quarterly onsite visits to all local agencies, where each quarterly visit involved a targeted review of a specific aspect of a local agency’s CSBG program. For example, during the third quarterly visit of the year, state officials focused on local agency planning efforts for the next funding year, including the community needs assessment, while during the last quarterly visit of the year, state officials focused on grant closeout activities. New York, like North Dakota and Texas, also conducted routine offsite reviews of local agencies’ activities and finances. In our three selected states, these reviews included examining fiscal and program reports periodically submitted by local agencies to state officials, periodic meetings and conference calls between state and local agency staff, and reviewing audit reports. These oversight activities also included fiscal audits conducted by the state auditor or independent auditors when a local agency’s funding met the threshold for such review. Our review of single audits and interviews with each state’s auditor’s office in the three states we visited showed that none of the state audit agencies focused specifically on CSBG funding during the period of our review. Texas last conducted an audit focusing on CSBG in 2014 and North Dakota did so in 2011; neither state reported findings as a result of those audits. Officials from the state auditor offices in North Dakota and Texas said CSBG funding levels are below the federally-established threshold for programs that must be audited. New York state audit officials told us that they had not conducted any audits focused on CSBG. OCS and states provided training and technical assistance through a variety of methods to help CSBG recipients meet program requirements. In fiscal years 2016 and 2017, OCS designated nearly $14 million over the 2-year period for such efforts. OCS officials told us that they determine what training is needed through input from OCS program specialists, information obtained through a data task force, and requests from state and local agencies. OCS officials stated that the OCS’s program specialists use the quarterly calls to identify the types of support that states need. For example, a specialist may notice that the states need additional guidance on using their customer survey results. In response, the specialist may share a guide on how states can use the survey results to set reasonable performance improvement goals. In addition, OCS sponsors a CSBG Data Task Force to recommend strategies for building network capacity for collecting, analyzing, reporting and using performance data as well as identifying on-going training and technical assistance needs. OCS officials told us that they also conducted focus groups in 2016 to gather states’ perspectives on their training and technical assistance needs. From these focus groups, OCS issued guidance stating its technical assistance priorities and strategy for meeting identified needs for training and technical assistance in areas including: performance management, governance, effective state oversight, and results-oriented services and strategies. In 2017, OCS issued guidance laying out the agency’s 3-year training and technical assistance strategy to guide the development and delivery of training and technical assistance for the CSBG network. OCS officials said that once they establish the standards for the training and technical assistance and identify specific training needs, the agency awards cooperative agreements to organizations that focus on developing and providing training to build upon guidance already provided. During the period of our review, we found that each agreement focused on a specific type of training. For example, the National Association for State Community Services Programs (NASCSP) has a cooperative agreement with OCS to provide the orientation and oversight training for new state officials overseeing the CSBG program, and collects and coordinates the analysis of the data provided in the state plans and annual reports. OCS has also worked closely with NASCSP in the transition to the new performance framework. OCS officials told us that they are currently reviewing their training and technical assistance portfolio and may issue additional guidance on its strategy and coordination efforts during fiscal year 2020. In addition, OCS uses various methods to provide guidance to states to help them meet CSBG requirements, but state officials differed in their views on the usefulness of the guidance. OCS provides guidance to states through informational memorandums, letters, webinars, and communications with program specialists. Some of the state agency officials in two of the states we visited said that the guidance that OCS has provided to help states ensure compliance with program requirements is not always clear and up to date. For example, officials in North Dakota said that they did not understand the information requirements for a form used to gather information from applicants for local programs. State agency officials in Texas said that OCS issued guidance on the new information requirements just weeks before the reporting deadline, and that this did not allow states sufficient time to set up their data systems to meet the new requirements. OCS officials acknowledged that they were aware of the issues raised by state agency officials and explained that some states have difficulty with the guidance because it is written at a high level so that it can apply to all states. They also acknowledged the delays in getting new information requirements to states and said that such delays were related to troubleshooting the new smart forms and online database. They said that they do not anticipate such delays in the future. As previously discussed, Texas state officials also said that they found the guidance for terminating a deficient agency’s CSBG funding confusing. However, officials in New York said that they found the guidance to be clear. They said that the informational memorandum on terminating agencies’ CSBG funding is more prescriptive than previously issued CSBG guidance. OCS officials stated that the agency is continuously seeking opportunities to work with its technical assistance centers to identify the best means of delivering guidance to states and to eligible entities. OCS officials also said that they must continue to refresh training efforts when there is turnover among key staff in a state agency and work with new state administrators to transition into their new roles. State agency officials in all three states we visited told us that they used some of their state’s discretionary funding for training and technical assistance to help local agencies meet CSBG requirements. The CSBG Act allows states to use a maximum of 10 percent of their CSBG funds for training and technical assistance and other specified purposes. In the selected states, officials spent from $65,000 to over $400,000 for training and technical assistance for local agencies (see table 4). Across the three selected states, we found that the training provided to local agencies addressed what local agencies need to do to meet a wide variety of CSBG requirements, from planning community needs assessments to implementing performance management requirements. In addition, some funds states provided for training were used by local agencies to send their staff to regional or national conferences for training (see table 5). State officials in two of the three states we visited said that they determine what training they need to offer based on analysis of feedback and specific requests from local agencies. For example, Texas identified training needs for local agencies through a Training and Community Affairs group that gathered information from local agencies about their training needs. Texas officials said they analyzed assessment results, feedback, and requests from local agencies and other sources to determine the training needs of individual state and local agencies. State officials said that they then met with the state association to develop the Joint State Training and Technical Assistance Plan and, ultimately, to provide trainings at the annual state conference, and to identify workshops, webinars, and online resources (guides, tools, best practices, and links to other training resources) that need to be added or changed. Similarly, state officials in North Dakota reported working closely with the state association of community action agencies to plan and conduct training for local agency staff. State and local agency officials also said that they have relied on the OCS-funded national resource centers for assistance. Officials in the states we visited all reported being helped by information provided by the national centers on topics such as the new organizational standards and how to submit data in the new annual report. Local agency officials told us that they send staff to the conferences sponsored by the national resource centers to obtain training when funding is available for that purpose. In addition to training, state officials in the states we visited cited a variety of practices that contribute to effective oversight. Both New York and North Dakota officials emphasized the importance of frequent, ongoing communication with local agencies as crucial to successful oversight. New York also identified frequent visits to local agencies and immediate action in response to problems as additional key factors for effective oversight. OCS uses outcome data from state agencies that collect and aggregate data from local CSBG recipients to provide an indication of CSBG’s progress in meeting the three national program goals. As previously discussed, the three national goals of the CSBG program as established under the CSBG Act are to (1) reduce poverty, (2) empower low-income families and individuals to become self-sufficient, and (3) revitalize low- income communities. State agencies report data from a menu of more than 100 performance measures established by OCS and grouped by service types such as employment, early childhood programs, and education. OCS sets annual targets for the overall performance of the CSBG program and uses the aggregated state data as an indicator of CSBG’s national effectiveness to inform budget decisions consistent with federal requirements for performance management. Until fiscal year 2018, OCS used one performance measure—the number of barriers to economic security that the local agencies receiving CSBG funds eliminated for individuals, families, and communities—to provide an indication of CSBG’s national effectiveness. To do this, OCS combined the outcome data from 10 of the more than 100 performance measures from the state annual reports to derive a cumulative total number of barriers overcome. OCS selected the 10 measures as a way to track outcomes from services that range from emergency services to more comprehensive and coordinated services. The 10 measures included outcomes such as the number of participants who obtained a job, maintained employment, maintained an independent living situation, reached the goals of enrichment programs, or obtained emergency assistance. While this one performance measure of barriers eliminated was intended to provide OCS with an indication of how the program was meeting CSBG national goals, several weaknesses with this measure limited OCS’s ability to do so. First, the measure included duplicative counts. For example, an individual may overcome a number of different barriers to reach the outcome of obtaining a job. As a result, by tracking the number of barriers, an outcome may be counted multiple times when combining data from multiple measures. Second, it is also difficult to know which CSBG funded program or service caused the positive outcome or if one service helped achieve multiple outcomes. Third, OCS officials clarified that when calculating this and other outcome measures, the removal of barriers to economic security is not solely the result of CSBG funds, but of all funding administered to local agencies that received CSBG funds. As such, they said that it is difficult to isolate the effects of CSBG funding. In its agency wide budget justification for fiscal year 2020, HHS reported that in fiscal year 2017 local agencies eliminated 32.2 million barriers to economic security, well above the 27.6 million it set as its goal for the year. In the same year, 16.2 million individuals received support through local agencies receiving CSBG funds. While the performance measure aided OCS in providing some indication of how the CSBG program contributes to the goal of improving self-sufficiency, it still did not provide information on the program’s progress in meeting the other two national program goals. Leading practices in performance management stress that performance measures should be tied to the specific goals of the program. However, no such linkage existed between the performance measure OCS used to report on the progress of the CSBG program and the program’s three national goals. required to annually report, among other things, a summary of certain information the states provide and its findings on state compliance to Congress. While OCS does submit such reports, we found that there has historically been a multi-year lag in OCS providing these reports to Congress. In May of 2019, OCS released its fiscal year 2015 CSBG report to Congress (see sidebar on data reported in the CSBG fiscal year 2015 report to Congress). Over the last decade, this type of reporting lag has been common and OCS has taken an average of more than 3 years from the end of the federal fiscal year until the time the Congress received the final report. OCS officials told us that they submitted the draft annual report for fiscal year 2016 for internal review by HHS in October 2018, but said that they could not project when the final report would be issued to Congress. They said they are currently drafting the fiscal year 2017 report. OCS has taken steps to redesign its performance management approach, but several elements of the new approach do not align with federal performance management and internal control standards. OCS has been redesigning how it oversees and manages the performance of the CSBG program to better align with GPRAMA, according to OCS officials. Since fiscal year 2016, OCS has been implementing new performance management tools for the CSBG program, including updating what data it collects and how it collects it on the services and outcomes, or performance measures, of the CSBG program. OCS officials stated that the changes are necessary to be able to provide more information and analysis on CSBG funded programs and their outcomes. They also noted the importance of these updates given a tightening federal budget. As part of these changes, OCS updated its more than 100 performance measures by revising the language of some and adding new measures that state and local agencies can report on, including measures more focused on outcomes in the communities they serve. State and local agency officials told us that the increased emphasis on outcomes in the new measures was an improvement and increased their own focus on connecting CSBG funds to traceable results. In addition, OCS transitioned to an online data reporting system that allows state agencies to directly report and access CSBG program data. However, OCS is still revising how it will use the data provided by state and local agencies to reflect nationwide results. OCS is using the data collected in state annual reports to develop a new national measure intended to provide a national total count of individuals who achieve at least one positive outcome through programs and services offered by local agencies that receive CSBG funds. Unlike the prior measure on the number of barriers to economic security eliminated by local CSBG recipients that could include duplicative counts, the new measure will be a count of individuals. OCS stopped using the prior measure after fiscal year 2017. Until OCS finalizes the new measure, it does not have a performance measure in place with targets and results that it can report to Congress. As such, it is unclear if OCS will report national performance outcomes for fiscal year 2018 or how useful the new measure will be while it is still in development through fiscal year 2022. While OCS has taken steps to redesign its performance management approach, several elements of the new approach do not align with federal performance management and internal control standards. Specifically, OCS has not established (1) how the new national measure will be used to assess CSBG goals, (2) the relationship between state and local measures and program goals, and (3) how OCS will monitor the reliability of state and local agencies’ program data. How the newly developed national measure will assess CSBG program goals. As discussed, OCS is developing a new national measure intended to provide a total number of individuals who achieved at least one positive outcome from CSBG funded program or services. However, it is unclear which of the three program goals—reducing poverty, empowering low-income families and individuals to become self- sufficient, or revitalizing low-income communities—the new national measure is being used to assess. As noted previously, OCS officials have stated that they are working to establish ways to provide more information and analysis on programs and their outcomes. OCS officials also told us that they are using GPRAMA as a guide for these changes and in our prior work we have reported that these requirements can serve as leading practices for strategic planning at lower levels within federal agencies. GPRAMA requires agencies to establish performance goals and a balanced set of performance indicators, including output and outcome indicators as appropriate, in measuring or assessing progress toward those goals. Additional leading performance management practices state that performance measures should be tied to the specific goals of the program. However, OCS’s new measure which is intended to provide a count of the number of individuals that achieve one or more positive outcomes does not specify which of the three national program goals the new measure will address, nor how the other two national program goals will be addressed. OCS officials told us that the new measure is related to two of the three goals because it is aggregated data from some of the outcome measures focused on individual and family outcomes. However, officials acknowledged that the agency has not yet developed a national measure for revitalizing low-income communities. Officials stated they plan to report on progress toward developing these measures and that it will provide examples of community-level outcomes in upcoming reports to Congress. Without clearly linking the measure to the goals, there is no way to tell if, and to what degree, the services local agencies are providing through CSBG grant funds are having the desired effect on their communities, even if examples are included in the shared results. How state and local performance measures are related to the three program goals. It is unclear how the large number of updated state and local performance measures under OCS’s redesigned approach aligns with CSBG’s three national program goals. OCS still collects data on more than 100 measures but it is unclear which of these measures will be analyzed at a national level. According to OCS officials, these data are most useful to state and local agencies for assessing outcomes against their unique goals and numerous measures are necessary to capture the variety of services and outcomes across the 1,000 local agencies. In our prior work on ways that agencies could improve performance management, we have stated that using a minimal number of critical measures is a leading practice. We have found that organizations that seek to manage an excessive number of performance measures may risk creating confusing, excess data that will obscure rather than clarify performance issues. The large number of measures can also further complicate OCS’s efforts to align the measures with CSBG’s three national program goals. How OCS will assess data reliability long-term. Although OCS is taking steps to assess data collected from state and local agencies for its new national measure, it does not have a written plan for how it will assess the data’s reliability for future years. As previously discussed, OCS is using a new data reporting system to collect the data it will subsequently use for its new national measure and this data will now be received directly by OCS instead of a third party. However, OCS does not have written plans in place for how the agency will determine if the new data collected will be a valid measure of the national program’s effectiveness or if the data will be reported reliably by the states into OCS’s online data system. OCS received its first round of performance data for the new measure for fiscal year 2018 on April 30, 2019, and is working with its cooperative agreement grantees and contactors to compile results and conduct quality assurance tests for the new performance data using a multi-step process that involves: OCS staff comparing data provided in the annual report to information previously provided in the state plans; OCS conducting quality assurance reviews, with assistance from the organizations the office has cooperative agreements with, that include checks for discrepancies and identifying items requiring clarification, and conducting follow-up with the states; and, OCS soliciting feedback from state officials and consulting with performance management experts within HHS about refinements to assist OCS in establishing a baseline that will be used in setting future targets. OCS officials also told us that the next steps will be to make any necessary modifications to the measure, such as adjusting how states calculate positive outcomes, and establishing a baseline to set future targets. On October 2, 2019, OCS announced via a Federal Register Notice that it was requesting a three year extension with minor changes of the CSBG Annual Report. OCS plans to make only minor changes to the current data collection tool for 2 years to allow state and local agencies time to assess current information and intends to begin a longer term planning process starting in fiscal year 2020. OCS officials told us that they plan to implement and maintain a quality assurance process to ensure the accuracy of the data based on data from previous years. While the process OCS has put in place to ensure data reliability for the first round of data collected for the new measure is a step in the right direction, OCS does not have a plan for assessing future years’ data. OCS officials told us that they will use selected cooperative agreements and contracts to develop a written plan for how the agency will monitor state and local agency data reliability going forward, but did not provide a timeframe for when this would be completed. Leading practices established by federal internal control standards state that agencies should use quality information that is appropriate, current, complete, and accurate to make informed decisions and evaluate the entity’s performance in achieving key objectives. OCS officials reported that they and contractors are working with the states to adjust and finalize data for fiscal year 2018 by November 2019. By not aligning its redesigned performance management approach with federal performance management leading practices related to program goals, performance measures, and data reliability, OCS cannot properly assess its progress in meeting CSBG’s three national goals. Poverty erodes the well-being of individuals, families, and communities. The CSBG program is intended to reduce poverty, empower low-income individuals and families to become self-sufficient, and revitalize low- income communities. The CSBG program allows local agencies to use funds in a wide variety of ways to reduce the causes of poverty in the communities they serve. However, the inherent flexibility of the program also makes it difficult to assess the program’s performance. OCS recently redesigned its performance management approach to better understand how well the CSBG program is progressing toward meeting national goals. However, several elements of the redesigned approach do not align with leading practices in federal performance management. Inconsistencies with these practices, such as having an excessive number of performance measures and lacking a plan for assessing the reliability of state and local performance outcome data, limit OCS’s ability to demonstrate the national effectiveness of the CSBG program. As such, OCS cannot assure the Congress and the American public that the funding is meeting its intended purpose to reduce the causes of poverty. The Director of OCS, in developing the new performance management approach for the CSBG program, should ensure that its performance framework includes information on (1) details for how the national measure is linked to and used to assess the three national program goals, (2) descriptions of how the updated state and local performance outcome measures align with national program goals, and (3) a written plan for how OCS will assess the reliability of state performance outcome data. (Recommendation 1) We provided a draft of this report to HHS for review and comment. We received written comments from HHS, which are reprinted in appendix III. HHS concurred with our recommendation, and stated that it plans to take actions to better align its performance measures with the three national performance goals outlined in the new CSBG Theory of Change. While we commend HHS for its plans to address our recommendation, we urge HHS to focus on aligning its performance outcomes with the three national goals of the CSBG program as established by the CBBG Act, which are similar but not identical to the three goals outlined in the new CSBG Theory of Change. HHS also stated that it would implement additional actions to assess the reliability of state performance outcome data. In addition, HHS provided technical comments which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees and the Secretary of HHS. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to this report is listed in appendix IV. This appendix discusses in detail our methodology for addressing our two research objectives examining (1) the activities that the Department of Health and Human Services (HHS) and states conduct to oversee the state and local agencies that receive Community Services Block Grant (CSBG) funds and (2) the extent to which HHS assesses the outcomes of the CSBG program. We scoped our review of the CSBG program to include the 50 states, American Samoa, the District of Columbia, Guam, Northern Mariana Islands, Puerto Rico, and the United States Virgin Islands, which are defined as states under the CSBG Act. In addition to the methods we discuss below, to address both our research objectives, we reviewed relevant federal laws, federal grants management guidance, and agency documents that describe the federal requirements and responsibilities for overseeing states’ CSBG programs and assessing program outcomes. We interviewed HHS, Office of Community Services (OCS) officials; and reviewed relevant research from OCS and the HHS Office of Inspector General, as well as our prior work on the CSBG and other federal grant programs. Further, we interviewed representatives of the National Association for State Community Service Programs (NASCSP); state officials from state agencies that oversee the CSBG program in New York, North Dakota, and Texas; and six local agencies that receive CSBG funds. We also analyzed CSBG annual reports to Congress and NASCSP data on local agency allocations. To address the federal oversight aspect of our first objective, we reviewed available information on OCS’s policies and procedures, including the risk assessment criteria OCS uses to select states for onsite compliance evaluations and interviewed OCS officials about their oversight efforts. We also selected 12 states for an in-depth review of OCS’s oversight activities. These included six states (Indiana, Louisiana, Michigan, New York, North Carolina, and Texas) for which OCS conducted onsite compliance evaluations during fiscal years 2016 and 2017. We selected the six states where OCS had conducted onsite compliance evaluations based on which of the visited states OCS had prioritized as those in highest need of onsite reviews for fiscal years 2016 and 2017. We also randomly selected five states (Alaska, Colorado, Kentucky, Mississippi, and Rhode Island) where OCS did not conduct such evaluations, but conducted routine reviews. We also selected a sixth state—North Dakota—because OCS had not visited the state in several years. We compared the results to see if there were any notable differences between the two sets. While our findings are non-generalizable, they provide insight into the different levels of review OCS conducts and examples of OCS oversight actions. Our file documentation reviews included a review of: OCS’s comments on each section of the states’ program documents, including the state plan and annual reports; actions the states took to address OCS’s comments; and state’s fiscal controls, financial and program oversight documents. Table 6 provides a summary of the characteristics of the 12 states we selected for review. To address the state and local oversight aspect of objective one, for a more in-depth look at state oversight practices, including promising practices and challenges, we visited three states: two states (New York and Texas) for which OCS conducted onsite compliance evaluations and one (North Dakota) for which OCS conducted a routine review. We selected these states using several criteria, including state grant amounts, number of local agencies, whether the HHS Office of Inspector General findings had reviewed the state’s use of CSBG funds, the time since the state was last visited by OCS for a compliance evaluation visit, and recommendations from experts at NASCSP and at OCS, who based their recommendations, in part, on states that had promising practices for overseeing local agencies (see table 7). Our final state selections comprise a diverse sample based on these criteria. For example, our selected states include a state with a low number of local agencies, one with a large number of local agencies, states with high and medium amounts of funding, and a state with a low amount of funding. During our state site visits, we interviewed and collected information from state and local agency officials about state oversight efforts from fiscal years 2016 through 2017. For each of the three states, we interviewed state program officials and reviewed related documentation including state policies and procedures, state single audits, onsite oversight guides and reports, and reporting forms for local agencies. We also visited two local agencies in each state and interviewed staff to learn more about state oversight efforts, including fiscal and performance reporting, onsite visits, training and technical assistance, and promising practices and challenges to such oversight. We conducted these visits in November and December 2018. In each state we visited, we reviewed program files for the two local agencies we visited, including oversight, financial, and performance reports; and follow up correspondence concerning the findings from state agency visits to those local agencies. Information collected from state and local agency officials during our site visits are not generalizable to all state CSBG programs. In addition, we obtained information on state audit findings related to CSBG and met with state auditors during site visits to learn more about additional state oversight of CSBG and local agencies to learn whether any coordination occurred between the different federally funded programs offered by the local agencies to support state oversight efforts. We reviewed the Single State Audit findings for fiscal years 2016 through 2017 for each of three states and six local agencies we visited. We reviewed these audit reports to determine if there were findings pertaining to CSBG and if so, the nature of those findings. To address our second objective, we reviewed the program performance indicators OCS uses to measure program outcomes in relation to the stated goals of the CSBG program. We also reviewed OCS’s design and implementation plans for a new performance management approach, including revised performance measures for assessing program outcomes. We compared OCS’s previous performance management approach to its new one, including the types of data it collected and its methods of collecting data from state and local agencies. In conducting our work, we also interviewed OCS officials about the goal of, and changes to, the performance management approach and reporting requirements. Additionally, we interviewed state officials on their experience with CSBG program performance. We reviewed leading practices in grant performance management identified in federal guidance and in GAO reports and assessed OCS’s approach against federal performance and internal control standards. We conducted this performance audit from to May 2018 to November 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This table includes all states as defined by the CSBG Act, which was the focus of our review. In addition to the contact named above, Mary Crenshaw (Assistant Director), Melissa Jaynes (Analyst-In-Charge), Sandra Baxter and Stacy Spence made key contributions to this report. Also contributing to this report were James Bennett, Grace Cho, Alex Galuten, Danielle Giese, Corinna Nicolaou, Monica Savoy, and Almeta Spencer.", "summary": "CSBG is one of the key federal programs focused on reducing poverty in the United States. In fiscal year 2019, CSBG provided about $700 million in block grants to states. In turn, states provided grants to more than 1,000 local agencies, which used the funding to provide housing and other services to program participants. HHS is responsible for overseeing states' use of this funding, and states have oversight responsibility for local agencies. GAO was asked to review CSBG program management. This report examines (1) how HHS and selected states conduct their oversight responsibilities and (2) how HHS assesses the effectiveness of the CSBG program. GAO reviewed files for six of the 12 states where HHS conducted onsite compliance evaluations during fiscal years 2016 and 2017, and six states where HHS conducted routine monitoring—five of which were randomly selected. GAO visited three states, selected based on their CSBG funding amount and other factors, to conduct in-depth reviews of their monitoring activities. GAO also reviewed agency documents and interviewed HHS and selected state and local officials. The Department of Health and Human Services (HHS) and the selected states GAO reviewed provided oversight of the Community Services Block Grant (CSBG) program through onsite visits and other oversight activities to assess grant recipients' use of funds against program requirements. Specifically, GAO found: HHS and the selected states conducted required oversight activities. The Community Services Block Grant Act requires HHS to conduct compliance evaluations for several states each year and requires states to conduct onsite visits to local CSBG recipients at least once every 3 years to evaluate whether recipients met various goals. During fiscal years 2016 and 2017, HHS conducted onsite compliance evaluations for 12 states that it deemed most at risk of not meeting CSBG requirements. GAO's visits to three states found that all three had conducted onsite visits to local grantees during the same fiscal years. HHS and the selected states also conducted additional oversight activities. This included routine reviews and quarterly calls. HHS and state monitoring activities primarily identified administrative errors, instances of non-compliance and other issues, which grant recipients took steps to address. For example, a HHS fiscal year 2017 compliance evaluation found that in fiscal year 2015 one state neither implemented procedures to monitor and track findings from a state audit, nor monitored eligible entities as required. HHS uses state outcome data to report on CSBG's national effectiveness, but these data are not aligned with the national program goals to reduce poverty, promote self-sufficiency, and revitalize low-income communities. HHS recently redesigned its' performance management approach to improve its ability to assess whether the CSBG program is meeting these three goals, but several elements of the approach do not align with leading practices in federal performance management. GAO found that HHS's redesigned approach does not demonstrate: How the agency's newly developed national performance measure—intended to provide a count of the number of individuals who achieved at least one positive outcome through CSBG funds—will assess the program in meeting national program goals. How the state outcome data, consisting of more than 100 state and local program measures, relate to CSBG's three national goals. How data collected from state and local agencies will be assessed for accuracy and reliability. Without aligning its redesigned performance management approach with leading practices, OCS cannot properly assess its' progress in meeting CSBG's three national goals. GAO is recommending that HHS's new performance management approach include information on how its performance measure and state outcome measures align with program goals and how it will assess data reliability. HHS agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "Hospice care helps patients who are terminally ill—as well as the families of those patients—maintain their quality of life. Hospice care focuses on the comfort of patients (palliative care), not curing the illness. Patients are eligible for hospice care under Medicare if they have a life expectancy of six months or less. Most patients get hospice care at home, which typically includes use of controlled substances, including opioids such as oxycodone, to provide pain relief. According to CMS, hospice teams are required to include a physician, nurse, social worker, and pastoral or other counselor; and may also include hospice aides, trained volunteers, and speech, physical, and occupational therapists. The patient selects a primary caregiver when first admitted into home hospice, and this person becomes a member of the home hospice team. The primary caregiver, often a family member, provides most of the care for the patient in home hospice, including most of the physical care for the patient, keeping records of symptoms and other problems, and communicating with the hospice team. The Controlled Substances Act regulates the manufacture, distribution, use, and disposal of controlled substances. In general, the Controlled Substances Act was enacted to facilitate the use of controlled substances for legitimate medical, scientific, research, and industrial purposes while preventing them from being diverted for illegal uses. DEA is the primary agency with responsibility for administering and enforcing the law, and DEA provides oversight of all persons or entities required to register with DEA. The Controlled Substances Act has been amended twice to clarify federal requirements for patient disposal of controlled substances. 2010. The Secure and Responsible Drug Disposal Act of 2010 amended the Controlled Substances Act to allow a patient who has lawfully obtained a controlled substance to deliver the controlled substance to another person for the purpose of disposal, without being registered with DEA. Any person lawfully entitled to dispose of a deceased patient’s property may also deliver the patient’s controlled substances to another person for the purpose of disposal. The person receiving the controlled substances must be legally authorized to do so and the disposal has to take place in accordance with DEA regulations, which DEA issued in 2014. Among other things, the DEA rulemaking clarified that home hospice personnel could not dispose of a deceased patient’s controlled substances unless authorized to dispose of the patient’s property by a state or local law. Instead, the rulemaking encouraged home hospice personnel to assist patients and their families in disposing of controlled substances in accordance with the Controlled Substances Act, and partner with authorized collectors to promote or jointly conduct mail-back programs. 2018. The SUPPORT Act amended the Controlled Substances Act to allow employees of qualified hospices, whether or not registered with DEA, to dispose of a patient’s unused controlled substances onsite and in accordance with all applicable laws after the patient’s death or the controlled substance expires. The employee must be a physician, physician assistant, nurse, or other person who is: employed by a qualified hospice; licensed to perform medical or nursing services by the jurisdiction in which the patient is receiving hospice care; acting within the scope of their employment in accordance with applicable state law; and trained on the disposal of controlled substances by the qualified hospice. If the hospice patient no longer requires the controlled substances because of a change in his or her care plan, only the patient’s DEA- registered physician may dispose of the patient’s unused controlled substances. The authority to dispose of unused controlled substances under the SUPPORT Act applies only to qualified hospices. Such hospices must have written policies and procedures for assisting in the disposal of controlled substances after the patient’s death, and must document that they provided and discussed these policies and procedures in an understandable manner with the patient and family. In addition, the hospice must document the type of controlled substance, dosage, route of administration, and quantity disposed, as well as the time, date, and manner in which the disposal occurred. The SUPPORT Act also allows the Attorney General to issue guidance to hospices regarding the disposal of controlled substances in patients’ homes. According to DEA officials, DEA’s oversight of the disposal of controlled substances in home hospices is limited to instances of suspected or actual diversion. This is because the SUPPORT Act allows employees of qualified home hospices to dispose of unused controlled substances in patients’ homes without registering with DEA. CMS regulates Medicare-certified home hospices through the Hospice Conditions of Participation, which are intended to protect the health and safety of individuals under hospice care. Hospices must be in compliance with the Hospice Conditions of Participation to participate in the Medicare program. CMS oversees compliance with the Hospice Conditions of Participation primarily through inspections, which are conducted by state survey agencies contracted by CMS or CMS-approved national private accrediting organizations. Among other things, the Hospice Conditions of Participation require hospices to have written policies and procedures for the management and disposal of controlled substances in the patient’s home, discuss the hospice policies and procedures for managing the safe use and disposal of controlled substances with the patient and family in a manner that they understand, and document that the written policies and procedures for managing controlled substances were provided and discussed. CMS does not oversee the disposal process. According to a national hospice association official, each hospice had a different approach to disposal prior to the DEA rulemaking in 2014. Some hospices asked their employees to dispose of controlled substances to prevent diversion and others did not. After the DEA rulemaking, some states enacted laws granting authority to hospice employees to dispose of patients’ unused controlled substances when the medications were no longer needed, upon death of the patient, or both. Requirements under states’ laws vary (see appendix I). Hospices in some states without laws on the disposal of controlled substances in home hospice began, or resumed, disposing controlled substances in patients’ homes following the enactment of the SUPPORT Act in 2018. Officials from six selected home hospices expressed support for the authority to dispose of controlled substances granted by the SUPPORT Act, and officials we interviewed from two hospices operating in states without disposal laws told us their hospices had resumed disposing of controlled substances in patients’ homes under the authority granted by the SUPPORT Act. In contrast, an official from one hospice told us that their hospice had not begun disposing of these medications with the enactment of the SUPPORT Act because the state department of health directed it not to do so until a state law granting disposal authority to hospices had been enacted. For now, the hospice has continued with its practice of educating patients’ family members on how to dispose of controlled substances themselves. Medicare-certified hospices, including the seven we selected for our review, are required by CMS’s Hospice Conditions of Participation to have written policies and procedures for the safe disposal of controlled substances in a patient’s home. The policies and procedures may include best practices, such as measures for assessing and mitigating the risk of diversion in a patient’s home, and if and how the hospice will conduct controlled substance disposal. According to officials we interviewed from the selected hospices and state hospice associations, hospices utilize various strategies or best practices to attempt to mitigate diversion risks, including, but not limited to, the following: Education on controlled substance use and disposal. Hospice policies may include disposal education for patients and their caregivers. Specifically, officials from five hospices and five state hospice associations said that patient and family member education on controlled substances and their disposal begins or should begin upon the patient’s admission into hospice care or as soon as possible thereafter. According to officials from three hospices, their staff may use written agreements or acknowledgements that must be signed by the patient or their caregiver. An official from one hospice association told us the association made an agreement template available to their hospice members that can be used to ensure patients understand how to properly use prescribed controlled substances, agree to use them properly, and will not give them to anyone else. Prescription drug counts. Officials from four hospices and two state hospice associations told us that, in general, nurses conduct prescription drug counts at every visit to check if the proper amounts of medications remain. Officials from two of these hospices said that drug counts should require the family’s acknowledgement or be witnessed. Officials from three other state hospice associations mentioned that their members use drug counts as well but did not specify if this occurred at every visit. These counts can be used to recognize possible drug misuse or diversion. If there is an indication that diversion may be the cause of an incorrect count, hospices can put additional drug diversion risk reduction practices in place. Lockboxes. If diversion is suspected to be a risk or if there are children present in the patient’s home, a hospice may choose to use a lockbox to store the patient’s medications and limit access to only an alert patient or their caregivers. Officials from five hospices and five state hospice associations mentioned that their employees and members use lockboxes for such purposes. One hospice official explained that lockboxes may also be used as an accountability tool so that those with access cannot accuse others of stealing if drugs are unaccounted for. Pharmacy cooperation. A hospice may choose to have the pharmacy mail a prescribed controlled substance in smaller quantities and with greater frequency. For example, an official from one of the selected hospices explained that the pharmacy they use will deliver medications as often as daily if needed to reduce the risk of controlled substances being diverted. Similarly, an official from a state hospice association explained that some pharmacy managers and benefits managers note when a refill for a prescription is requested sooner than it should have been and alert the hospice. Witnessed disposal or assisted disposal. Pursuant to some state disposal laws and according to officials from five hospices and four state hospice associations, controlled substance disposal and assisted disposal must or should be performed with a witness present. The state disposal laws may specify who the witness must be. For example, according to two state laws, a family member or a second hospice employee may witness disposal. In-home disposal products. Hospices may have varying preferences for how they dispose of controlled substances; officials from four of the selected hospices mentioned using in-home disposal products, and two specifically explained they believed these to be the safest disposal method, even though, according to the officials, it can be costly. Officials from another hospice told us they receive their in- home disposal products through a grant program. Documentation. Officials from four selected hospices told us their employees document the completion of certain tasks, such as diversion risk assessments, drug counts, drug disposal, and the refusal of drug disposal. An official from one of these hospices told us their staff perform and document a diversion risk assessment of the patient’s home. While officials from four hospices told us their employees perform drug counts, only one official specified that employees from their hospice document the drug counts. Another hospice official explained that disposal documentation includes the name, dosage, form, and administration method of the medication. One hospice official told us that if a patient’s family members refuse disposal, they must sign a form stating they declined to allow the nurse to dispose of the patient’s remaining drugs. Officials from selected hospices and state hospice associations in our review described various challenges associated with disposing of controlled substances in patients’ homes. The challenges described by the selected hospices and state hospice associations include but are not limited to the following: Certain disposal methods may be too costly. An official from one state hospice association said that most of its members do not use a drug disposal process, which may include a mail-back program or in- home disposal product, because it is an extra expense, and an official from one hospice said that after pricing an in-home disposal product, their hospice decided it was cost prohibitive. One state hospice association official explained that although most of its in-home hospice members use an in-home disposal product, in instances when the hospice employee is disposing of 40 to 50 vials, the expense of this disposal method can be burdensome. The official told us the product costs approximately $6 each and fits four to five vials of pills in each. Disposal can be time consuming. One hospice official said that disposal can sometimes be a time-consuming and resource-intensive activity. According to two state hospice association officials, sometimes a patient’s family will ask the disposing hospice employee to dispose of all of the patient’s unused prescription drugs that remain in the home, not only controlled substances or drugs prescribed under hospice care. Officials from two of our selected hospices and two state hospice associations told us that it is not atypical for a hospice patient to have bags or boxes full of unused medications, though the officials did not describe this as a disposal challenge for hospices. Lack of a witness. One hospice official told us that it is a challenge when a witness is not available or is unwilling to participate in a drug count or disposal. Another hospice official indicated that the patient’s primary caregiver is not always the family member present at the time of a drug count or disposal. This can create a challenge, as the hospice employee must either wait for the patient’s primary caregiver to arrive, or for that person to agree to witness a count, disposal, or both. Family members and caregivers sometimes refuse to dispose of controlled substances. Officials from two hospices and three state hospice associations indicated that a family’s refusal to dispose of a patient’s remaining medications can be a challenge, though one hospice official said it occurs infrequently. An official from another hospice said that if a family initially refuses disposal, hospice staff return after two weeks to complete the disposal process. Inconsistencies between state laws and federal law. Hospices must comply with applicable federal and state laws governing controlled substances, and to the extent state law is inconsistent with the Controlled Substances Act, hospices must follow federal law. Hospice officials told us that inconsistencies between state laws and federal law can cause challenges. For example, the SUPPORT Act limits disposal to only physicians, physician assistants, nurses, or other hospice employees who are licensed to provide medical or nursing services. An official from one hospice stated that the hospice used the help of social workers and volunteers to dispose of controlled substances. Regulations in this state do not specify which types of hospice employees are permitted to assist with disposal. According to the official, social workers and volunteers helped dispose of patients’ controlled substances when disposal occurred at a later time, rather than immediately following a patient’s death. Since the SUPPORT Act limits disposal to home hospice personnel with specific qualifications, it is unclear whether hospices are able to allow social workers and volunteers to help in that capacity. As another example, under the SUPPORT Act, only a hospice patient’s DEA-registered physician can dispose of the patient’s controlled substances if the plan of care has been modified. However, some state laws allow other types of hospice employees to perform disposal in this circumstance. Officials from two hospices in these states indicated it will be a challenge for disposal to be limited to physicians when a patient’s plan of care is modified. Similarly, officials from two other selected hospices in states without disposal laws also stated that this would be a challenge. For example, one hospice official explained that their hospice does not have many physicians, and it would be unlikely for a physician to be able to visit a patient’s home solely to handle disposal. We provided a draft of this report to the Departments of Justice and Health and Human Services for review. The Departments of Justice and Health and Human Services provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Attorney General of Justice, the Secretary of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at CosgroveJ@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix II. To describe what is known about selected hospices’ experiences disposing of and preventing the diversion of controlled substances in home settings, we selected five states with laws on the disposal of controlled substances in home hospices and six states without such laws. The five states with disposal laws were chosen for this review because they had state hospice associations that were involved in disposal discussions with a national hospice association or they had higher opioid-related death rates than most states. The summaries in Table 1 below reflect our reviews of the five states’ laws. In addition to the contact name above, Martin T. Gahart (Assistant Director), Deborah J. Miller (Analyst in Charge), Samuel G. Amrhein, Kaitlin M. Farquharson, and Christina C. Murphy made key contributions to this report.", "summary": "Misuse of controlled substances continues to be a serious public health problem in the United States. Most commonly misused controlled substances include opioids (such as oxycodone), which are used to treat pain, and central nervous system depressants (such as diazepam), which are used to treat anxiety and sleep disorders. These types of drugs are commonly prescribed for patients in hospice care. The SUPPORT Act included a provision for GAO to examine disposal of controlled substances in home hospice settings. This report describes selected home hospices' controlled substances disposal practices and the challenges they face in disposing of these substances. GAO reviewed the SUPPORT Act and other related statutory and regulatory provisions. GAO also interviewed officials from the Centers for Medicare & Medicaid Services, the Drug Enforcement Administration, three national hospice trade associations, two national nurse trade associations, 11 state hospice associations, and seven hospices. Hospice care helps patients who are terminally ill maintain their quality of life. Most patients get hospice care at home, which typically includes use of controlled substances, including opioids such as oxycodone, to provide pain relief. When hospice patients die at home, they often leave behind unused controlled substances, which can be diverted and misused by anyone with access to them. The Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment for Patients and Communities Act (SUPPORT Act), enacted in 2018, allows employees of qualified hospice programs to dispose of unused controlled substances by collecting and destroying the drugs in patients' homes. In addition, some states had laws allowing hospice employees to dispose of patients' unused controlled substances prior to 2018. Three of the seven hospices GAO contacted operate in states without such laws. Officials from two of these hospices told us their hospices began disposing of patients' controlled substances in their homes following the enactment of the SUPPORT Act in 2018. However, one hospice had not begun disposing of these medications because the state department of health directed it not to do so until a state law granting disposal authority to hospices had been enacted. An official from that hospice said that it continued the practice of leaving the controlled substances in the home and educating family members about how to dispose of the drugs themselves. Hospice officials we spoke to identified best practices for preventing diversion and disposing of controlled substances. Best practices include prescription drug counts performed by hospice employees to determine if controlled substances are being used properly, use of lock boxes to limit access to controlled substances in situations where diversion is suspected to be a risk, and having a witness for the disposal of unused controlled substances. The officials also identified challenges their hospice employees have faced when disposing of controlled substances in patients' homes. Challenges include the cost of certain disposal methods, a lack of a witness to the disposal process, and inconsistencies between state laws and federal law concerning which hospice employees may dispose of controlled substances. The Departments of Justice and Health and Human Services provided technical comments on a draft of this report, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "Headquartered in Washington, D.C., the Corps has eight divisions established generally according to watershed boundaries and 38 districts that carry out its Civil Works program. Corps headquarters primarily develops policies and provides agency oversight. The Assistant Secretary of the Army for Civil Works, appointed by the President, sets the strategic direction for the agency and has principal responsibility for the overall supervision of functions relating to the Civil Works program. The Chief of Engineers—a military officer—oversees the Corps’ civil works and military missions. The eight divisions—Great Lakes and Ohio River, Mississippi Valley, North Atlantic, Northwestern, Pacific Ocean, South Atlantic, South Pacific, and Southwestern—coordinate Civil Works projects in the districts within their respective divisions. Corps districts are responsible for planning, engineering, constructing, and managing Civil Works projects. Congress established the Section 219 program in the 1992 WRDA, which authorized the Corps to provide planning and design assistance to nonfederal sponsors in carrying out 18 environmental infrastructure projects, located in certain specified locations around the United States. For example, the 1992 WRDA authorized the Corps to provide assistance for a combined sewer overflow treatment facility for the city of Atlanta, Georgia. In subsequent acts, Congress authorized the Corps to provide construction assistance for Section 219 projects, in addition to planning and design, and significantly expanded the number of authorized projects. From 1992 through 2007, Congress authorized a total of 310 Section 219 projects, with the most recent and largest number of project authorizations occurring in 2007 (see table 1). For Section 219 projects, Congress specifies the geographic location (e.g., city, county), amount of authorized dollars, and purpose or scope of the project (e.g., development of drainage facilities to alleviate flooding problems). In general, Section 219 projects fall into one or more of the following types of projects: Drinking water treatment and distribution. Projects that build water treatment plants, water storage tanks, and water distribution lines. Wastewater treatment. Projects that build sewage treatment plants, wastewater collection systems, and facilities that purify treated wastewater for irrigation and other purposes. Stormwater management. Projects that help improve the management of storm sewers, eliminate or control sewer overflows, and address flooding. According to Corps data, of the 310 originally authorized Section 219 projects, 58 have been deauthorized and were no longer active, as of November 2018. The Corps is required by the 1986 WRDA, as amended, to annually identify all authorized projects that have not received obligations in the preceding 5 full fiscal years and submit that list to Congress. If funds are not obligated for planning, design, or construction of a project on that list during the next fiscal year, the project is deauthorized. The Secretary of the Army publishes a list of deauthorized projects in the Federal Register. Based on this process, the Corps considered deauthorizing 197 additional Section 219 projects in its fiscal year 2017 report to Congress. However, the 2018 WRDA provided that the projects identified for deauthorization in the Corps’ fiscal year 2017 report were generally not to be deauthorized unless they met certain additional requirements. The Corps allocates funding for Section 219 projects and other environmental infrastructure programs from the construction account. That account generally receives no-year appropriations through the Energy and Water Development Appropriations Act—meaning the appropriation remains available for obligation for an indefinite period of time. Prior to fiscal year 2012, the conference reports accompanying the annual Energy and Water Development Appropriations Acts generally listed individual Section 219 projects and specific allocations of funding for each project. However, since fiscal year 2012, Congress has not provided allocation direction for individual projects, but instead generally has designated an amount in reports and joint explanatory statements for environmental infrastructure overall, ranging from about $30 million to $55 million annually. According to Corps data, from fiscal years 1992 through 2017, the Corps expended over $440 million on Section 219 projects. Similar to other Civil Works projects, the Corps generally becomes involved in Section 219 projects when a nonfederal sponsor contacts the Corps for assistance on an authorized project. Corps districts gather additional information on the project from the nonfederal sponsor and determine if it is ready to be initiated. Once the Corps receives an appropriation from Congress, the agency decides whether to allocate funding to the project. If the project is selected to receive funding, it enters the preconstruction engineering and design phase. The purpose of this phase is to complete any additional planning studies and all of the detailed technical studies and designs—such as environmental impact studies—needed to begin construction. During this phase, the Corps also completes an environmental assessment of the proposed project. To initiate construction, the Corps and the nonfederal sponsor sign a project partnership agreement that specifies how the parties will collaborate, their respective roles and responsibilities, and the terms and conditions under which they will execute their responsibilities. The project partnership agreement typically requires the sponsor to provide without cost to the U.S. government all lands, easements, rights-of-way, relocations, and disposal areas necessary for the construction and subsequent maintenance of the project; maintain and operate the project after completion without cost to the provide cash or work-in-kind contributions to make the sponsor’s total contribution equal to 25 percent if the value of the sponsor’s land contribution does not equal or exceed 25 percent of the project cost. The Corps manages the construction phase, contracting out construction work to private engineering and construction contractors. Throughout the construction phase, the Corps oversees the contractors’ work, performing routine inspections to ensure it meets the Corps’ design and engineering specifications. During construction, the Corps, the nonfederal sponsor, and the private contractor typically appoint representatives to a project coordination team that meets regularly until the period of construction ends. Upon notification by the District Engineer that construction is complete, the nonfederal sponsor is responsible for operations and maintenance. Figure 1 shows the major steps in managing a Section 219 project. The Corps expended about $81 million on 29 Section 219 projects from fiscal years 2013 through 2017, which included various types of projects such as drinking water treatment and distribution, wastewater treatment, and stormwater management. Examples of these projects include the following: Drinking Water Treatment and Distribution. The Corps manages a Section 219 project that includes the development of water desalination infrastructure in various sections of the South Perris community, located east of Los Angeles, California. In general, the South Perris area relies on a mixture of groundwater and water imported from different sources, including the Colorado River. According to the Corps’ environmental assessment, various factors, such as drought, caused the community to supplement its drinking water supply through increased use of groundwater; however, the groundwater in the area historically contained high salt content. Since the project’s authorization in fiscal year 2001 through fiscal year 2017, the Corps has helped construct groundwater wells and pipelines, which connect to drinking water treatment facilities that reduce the amount of salt in the water (see fig. 2). According to the nonfederal sponsor for the South Perris project, the overall project has provided benefits such as creating a local potable water source to meet anticipated population growth and reducing the community’s dependence on imported water. Wastewater Treatment. The Corps manages a Section 219 project that includes the rehabilitation of sewer lines within the metropolitan area of St. Louis, Missouri. The city’s wastewater system dates back to the 1800s and lacks the capacity to handle large flows. From the project’s authorization in fiscal year 1999 through fiscal year 2017, the Corps has assisted the community, among other things, in sewer rehabilitation of deep tunnels. According to documentation from the Corps’ St. Louis District, the rehabilitation of sewers is important in protecting the health and safety of the public, given the risk of untreated sewage being discharged into the environment. Stormwater Management. The Corps manages a Section 219 project that involves the development of stormwater infrastructure, among other things, across a five-county region (Calumet region) in northern Indiana. For example, flooding is a widespread problem in the region and it has affected commercial corridors, including within Gary, Indiana. From the project’s authorization in fiscal year 1999 through fiscal year 2017, the Corps has been assisting the region with measures to alleviate flooding, such as constructing stormwater storage areas under the street (see fig. 2). According to a nonfederal sponsor we interviewed, the Corps’ efforts in the Calumet region have offered benefits to local communities by, among other things, improving storm drainage in an area that experienced flooding during heavy rainfall. The 29 Section 219 projects with expenditures from fiscal years 2013 through 2017 were located in different parts of the country and managed by six Corps divisions, although the majority of the projects were under the South Pacific Division (10 of the 29 projects) and Great Lakes and Ohio River Division (eight of the 29 projects). The five states with the largest number of projects during this period were California, with nine Section 219 projects; Virginia, with three Section 219 projects; and Michigan, Pennsylvania, and Mississippi, each with two Section 219 projects. These projects varied in terms of the geographic area covered, such as a city, county, or region (e.g., multiple counties). Based on the project descriptions we reviewed, 10 of the projects focused on the environmental infrastructure needs of cities, nine focused on counties and 10 on regions. Projects that cover a broad geographic area, such as those at the county or regional level, generally consist of different types of subprojects. For example, the Cook County, Illinois Section 219 project included several subprojects, such as the construction of water mains and sewer improvements in different areas across the county. Most of the Section 219 projects (24 of the 29 projects) were authorized in 2000 or earlier and were ongoing as of November 2018. Only one of the 29 projects was completed; the project in St. Croix Falls, Wisconsin, was completed in fiscal year 2014. For the St. Croix Falls project, the Corps assisted with improvements to a wastewater treatment plant, such as installing equipment to screen out large solids that otherwise would be released into the St. Croix River. Of the 28 remaining projects that were ongoing, as of November 2018, 17 were in the construction phase, and 11 were in the preconstruction engineering and design phase. Table 2 summarizes information on the 29 projects with expenditures from fiscal years 2013 through 2017 by division and district. See appendix I for additional information on each project, including a detailed description and the total amount of expenditures from fiscal years 2013 through 2017. As previously noted, the Corps spent about $81 million on these 29 Section 219 projects from fiscal years 2013 through 2017. During that period, expenditures by fiscal year ranged from about $11 million to $22 million. Divisions with the largest percentage of overall expenditures from fiscal years 2013 through 2017 were the South Atlantic Division (36 percent) and Mississippi Valley Division (25 percent). The divisions with the smallest percentage of overall expenditures during the period were the North Atlantic Division (less than 1 percent) and Southwestern Division (4 percent). Table 3 summarizes overall expenditures from fiscal years 2013 through 2017 by division and fiscal year. Of the 29 projects with expenditures from fiscal years 2013 through 2017, 15 projects expended less than $1 million each, representing a total of $2.3 million. The majority of these projects (10 of the 15 projects) were in the preconstruction engineering and design phase. For example, as part of the Cambria, California, project, the Corps expended about $244,000 on preconstruction engineering and design activities, such as evaluating the environmental impacts of constructing a seawater desalination facility. In addition, for the Cumberland County, Tennessee, project, the Corps expended about $261,000 on planning and design for water supply projects. In comparison, 14 of the 29 projects expended more than $1 million each over the same time period, representing a total of $78.2 million. In particular, the Corps spent over half of the funding during this time period on four projects: Calumet Region in Indiana; DeSoto County, Mississippi; Jackson County, Mississippi; and Lakes Marion and Moultrie in South Carolina (see fig. 3). These projects generally consisted of multiple subprojects and covered a wide geographic area. For example, the Calumet Region project has involved over 25 subprojects since its authorization in fiscal year 1999 through August 2018. Through these subprojects, the Corps has managed various activities, including replacing drinking water lines, improving wastewater treatment plants, and installing stormwater infrastructure in a number of cities across Indiana. Additionally, the Lakes Marion and Moultrie project in South Carolina has included a range of subprojects, such as construction of a water treatment plant, construction of a water tower, and installation of water transmission lines across six counties. The Corps generally follows its standard budget process for prioritizing funding for the Section 219 program. This process involves ranking Section 219 projects for funding by all three levels of the Corps’ organization—districts, divisions, and headquarters. District officials identify Section 219 projects, including subprojects, and other environmental infrastructure projects for potential funding; enter a numerical ranking for each project in the Civil Works Integrated Funding Database; and submit the information to the division through the database. Division officials receive the rankings from each of the multiple districts in the division. Division officials then re-rank the Section 219 and other environmental infrastructure projects from all of their districts against one another. Division officials enter the numerical ranking for all projects across all their districts into the Civil Works Integrated Funding Database and submit the information to headquarters through the database. Headquarters officials receive the rankings from each division. They re-rank the projects from all divisions against each other to generate the final nationwide rankings. Based on the final rankings, not all Section 219 and other environmental infrastructure projects that the divisions submitted will receive funding. Headquarters officials then determine a funding amount for each Section 219 project selected to receive funding and publish these decisions in the agency’s annual work plan. After headquarters publishes the annual work plan, headquarters officials begin to allocate funding to Section 219 projects. However, the Corps does not have written criteria to guide the ranking of Section 219 projects, in contrast to other types of projects. Specifically, in our December 2018 report, we found that the Corps uses written criteria—such as the rate of economic return, populations at risk, and economic impact—to prioritize funding for core mission areas, such as flood risk management and navigation projects. While Corps budget guidance indicates the criteria each core mission area should use in the ranking process, it does not specify criteria for Section 219 or other environmental infrastructure projects. In the absence of written criteria, Corps officials use their discretion on how to rank Section 219 projects for funding, according to Corps headquarters officials. When ranking Section 219 projects for funding, officials in each of the districts we interviewed generally consider whether Section 219 projects can be completed within the fiscal year. However, we found that the districts vary in terms of whether other factors are considered and what those factors are. Specifically, One district considers the level of congressional support and the potential public health impacts of the project. Another district considers the level of congressional support and the dollar value of the project. A third district only considers whether the project can be completed within the fiscal year. At the division level, officials we interviewed stated that they consider, among other things, the level of congressional support for the projects; however, to a large extent they rely on the rankings provided by their respective districts. Headquarters officials said that they primarily focus on ensuring that projects are geographically dispersed across the divisions when assigning final rankings for Section 219 projects. In recent years, congressional direction has indicated that the Corps, when allocating funding, is to consider giving priority for environmental infrastructure projects that have certain characteristics. For example, the Joint Explanatory Statement accompanying the Consolidated Appropriations Act in 2017 directed the Corps to consider characteristics such as projects: with the greater economic impact; in rural communities; in communities with significant shoreline and instances of runoff; in or that benefit counties or parishes with high poverty rates; and in financially distressed municipalities. Corps headquarters, division, and district officials we interviewed said that while they are generally aware of this congressional direction, they do not use it to guide the Section 219 ranking process. According to a division official, written criteria would be helpful for ranking projects across multiple districts and would clarify procedures for new staff. Officials we interviewed in the three districts said, in general, written criteria would clarify the ranking process. For example, one Corps district official stated that written criteria would provide standardization to the ranking process, ensuring that each district is focused on the highest priorities of the agency. According to Corps headquarters officials, although they see value in having written criteria to prioritize Section 219 funding, they have not developed such criteria because the agency considers Section 219 projects to be outside the agency’s core mission areas, such as flood control. According to a 2008 Corps report to Congress, “Funds provided to the Corps for wastewater treatment and municipal and industrial water supply projects necessarily reduce the amount of funds that instead could be used for the primary mission areas of the Corps. Thus, provision of Civil Works funding for these environmental infrastructure programs negatively affects the Corps’ ability to meet critical mission needs…such as restoring nationally significant ecosystems.” Headquarters officials confirmed that this report accurately reflects the agency’s current position. Corps officials also stated that developing written criteria has not been a priority because Section 219 projects represent a small percentage of the agency’s overall Civil Works budget. Federal standards for internal control states that agencies should use quality information to achieve their objectives by identifying information requirements. The federal standards also call for agencies to design control activities to achieve objectives and respond to risks, such as by clearly documenting internal control in management directives, administrative policies, or operating manuals. By establishing written criteria, the Corps would have greater assurance that its project selections align with a clear set of priorities, such as the characteristics identified in recent congressional direction for the agency to consider when selecting Section 219 projects for funding. Since the inception of the Section 219 program in 1992, the Corps has spent over $440 million on water infrastructure projects across its divisions and districts. However, the Corps has not developed written criteria for ranking Section 219 projects for funding as it has for other Civil Works programs within the agency’s core mission areas. Consequently, officials at the district, division, and headquarters levels are using their discretion regarding which factors to consider in ranking Section 219 projects for funding. Further, Congress has provided direction to the Corps on which characteristics to consider in prioritizing Section 219 funding; however, Corps officials stated that they do not use it to guide their ranking of Section 219 projects. By establishing written criteria, the Corps would have greater assurance that its project selections align with a clear set of priorities, such as the characteristics identified in recent congressional direction for the agency to consider when selecting Section 219 projects for funding. The Secretary of the Army should direct the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers to develop written criteria for ranking Section 219 projects for funding, taking into account a clear set of priorities, such as those identified by recent congressional direction. We provided a draft of this report to the Department of Defense for review and comment. In its written comments, reprinted in appendix II, the department concurred with our recommendation and described the actions they plan to take. Specifically, the Corps will develop and document a more rigorous set of priorities in line with those identified by recent Congressional direction. The department also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Appendix I: Description of U.S. Army Corps of Engineers Section 219 Projects and Expenditures from Fiscal Years 2013 through 2017 Project description as authorized by statute Water-related environmental infrastructure, Allegheny County, Pennsylvania. A combined sewer overflow treatment facility for the city of Atlanta, Georgia and watershed restoration and development in the regional Atlanta watershed including Big Creek and Rock Creek. Water-related infrastructure for the parishes of East Baton Rouge, Ascension, and Livingston, Louisiana. Water-related infrastructure projects in the counties of Benton, Jasper, Lake, Newton, and Porter, Indiana. Desalination infrastructure, Cambria, California. Water and wastewater infrastructure for the Contra Costa Water District, California. Water-related infrastructure and resource protection and development, Cook County, Illinois. Water supply projects in Cumberland County, Tennessee. Desert Hot Springs, California Resource protection and wastewater infrastructure, Desert Hot Springs, California. Wastewater treatment project in the county of DeSoto, Mississippi. Water supply and wastewater infrastructure projects in the counties of Accomack, Northampton, Lee, Norton, Wise, Scott, Russell, Dickenson, Buchanan, and Tazewell, Virginia. Water-related infrastructure and resource protection, including stormwater management, and development, El Paso County, Texas. Wastewater infrastructure assistance to reduce or eliminate sewer overflows, Genesee County, Michigan. Industrial water reuse project for the Harbor/South Bay area, California. Water infrastructure, Inglewood, California. Provision of an alternative water supply for Jackson County, Mississippi. Wastewater treatment and water supply treatment and distribution projects in the counties of Berkeley, Calhoun, Clarendon, Colleton, Dorchester, and Orangeburg, South Carolina. A project to provide water facilities for the Fox Field Industrial Corridor, Lancaster, California. Alleviation of combined sewer overflows for Lynchburg, Virginia, in accordance with combined sewer overflow control plans adopted by, and currently being implemented by, the non-Federal sponsor. Water-related infrastructure in the counties of Lackawanna, Lycoming, Susquehanna, Wyoming, Pike, Wayne, Sullivan, Bradford, and Monroe, Pennsylvania, including assistance for the Montoursville Regional Sewer Authority, Lycoming County, Pennsylvania. Project description as authorized by statute Water and wastewater infrastructure in Hancock, Ohio, Marshall, Wetzel, Tyler, Pleasants, Wood, Doddridge, Monongalia, Marion, Harrison, Taylor, Barbour, Preston, Tucker, Mineral, Grant, Gilmer, Brooke, and Ritchie Counties, West Virginia. A project to eliminate or control combined sewer overflows in the cities of Berkley, Ferndale, Madison Heights, Royal Oak, Birmingham, Hazel Park, Oak Park, Southfield, Clawson, Huntington Woods, Pleasant Ridge, and Troy, and the village of Beverly Hills, and the Charter Township of Royal Oak, Michigan. Recycled water transmission infrastructure, Eastern Municipal Water District, Perris, California. Alleviation of combined sewer overflows for Richmond, Virginia, in accordance with combined sewer overflow control plans adopted by, and currently being implemented by, the non-federal sponsor. San Ramon Valley, California A project for recycled water for San Ramon Valley, California. Water supply desalination infrastructure, South Perris, California. Wastewater infrastructure, St. Croix Falls, Wisconsin. Projects to eliminate or control combined sewer overflows in the city of St. Louis and St. Louis County, Missouri. Anne-Marie Fennell, (202) 512-3841 or fennella@gao.gov. In addition to the contact named above, Vondalee R. Hunt (Assistant Director), Anthony C. Fernandez (Analyst-In-Charge), Patricia Moye, Gloria Ross, and Sheryl Stein made significant contributions to this report. Important contributions were also made by Patricia Donahue, Tim Guinane, Susan Murphy, Sara Sullivan, Kiki Theodoropoulos, and Walter Vance.", "summary": "Under Section 219 of the 1992 Water Resources Development Act, as amended, Congress authorized the Corps to provide assistance for the design and construction of environmental infrastructure projects, known as Section 219 projects. Such projects include the development of water transmission lines. Congress typically provides a lump sum appropriation for the Corps' construction account, out of which Section 219 and other environmental infrastructure projects are funded. GAO was asked to review projects carried out under the Section 219 program. This report examines (1) the number and type of Section 219 projects and expenditures from fiscal years 2013 through 2017, and (2) how the Corps prioritizes funding for Section 219 projects. GAO reviewed relevant federal laws and agency guidance; analyzed agency data for fiscal years 2013 through 2017, the most recent time period for which data were available; and interviewed agency officials at headquarters, three divisions, and three districts–selected based on geographic distribution and the amount of Section 219 project expenditures. From fiscal years 2013 through 2017, the most recent available data, the U.S. Army Corps of Engineers (Corps) spent approximately $81 million on 29 Section 219 projects to develop drinking water, wastewater, and stormwater infrastructure. For example, through the St. Croix Falls, Wisconsin Section 219 project, the Corps assisted with improvements to a wastewater treatment plant. Of the 29 projects, the Corps spent over half of the funding during this period on four projects: (1) Calumet Region, Indiana; (2) Desoto County, Mississippi; (3) Jackson County, Mississippi; and (4) Lakes Marion and Moultrie, South Carolina. The Corps generally follows its standard budget prioritization process—which involves districts, divisions, and headquarters ranking each project and headquarters making final funding decisions—to prioritize Section 219 funding. However, the Corps has not developed criteria to guide this process. GAO found the Corps varies in the factors it uses to rank Section 219 projects. For example, one district considers whether a project can be completed within the fiscal year, while another considers the level of congressional support and dollar value of the project. Headquarters officials said the agency views Section 219 projects as outside its core mission areas and therefore has not developed written criteria. Congressional direction has indicated that the Corps is to consider characteristics—such as projects with the greater economic impact—in prioritizing Section 219 project funding. While aware of this direction, Corps officials said they do not consider it when ranking projects. Federal standards for internal control states that agencies should use quality information to achieve their objectives. By establishing written criteria, the Corps would have greater assurance that its Section 219 project selections align with a clear set of priorities, such as those identified by recent congressional direction. GAO recommends that the Corps develop written criteria for ranking Section 219 projects for funding, taking into account a clear set of priorities, such as those identified by recent congressional direction. The agency concurred with the recommendation.", "document_type": "gao"}
{"report": "OECA has a range of compliance monitoring, compliance assistance, and enforcement tools available to elicit compliance with laws and regulations from regulated entities, as shown in table 1. Enforcement actions can result in, among other things, the imposition of penalties, requirements to remedy the violation of law or regulation, or both. OECA has developed policies and guidance for EPA staff that describe the agency’s recommended responses to noncompliance based on a number of factors and the escalation of enforcement responses to continuing noncompliance. EPA guidance on informal and formal enforcement actions provides an example related to the Resource Conservation and Recovery Act. In that example, if a regulated entity does not return to compliance or notify the state or EPA that it cannot return to compliance within a certain number of days after an informal enforcement action, the state or EPA may take a formal enforcement action. Generally, according to this same 2010 EPA guidance, informal enforcement actions address small or isolated problems, and formal enforcement actions can address bigger problems. OECA stores and manages a range of compliance monitoring and enforcement data in ICIS. For example, ICIS includes descriptive information about regulated entities, violations, and the outcome of enforcement actions. ECHO, the public access website that integrates data from multiple agency databases, has an internal component for staff and other federal agencies and publicly available components. Staff in EPA’s 10 regional offices, OECA headquarters staff, and states input data into ICIS, which feeds data into ECHO. Regional office staff and OECA headquarters staff also use statute-specific databases to maintain data on compliance with a particular law or office-specific databases built to maintain data, according to the preferences of a particular regional or headquarters office. EPA requires regional offices to collect and enter a range of information on its compliance monitoring and enforcement activities—such as permit, inspection, and violations data—into the agency’s national databases. The agency uses these data to manage its oversight efforts and assess how well the efforts are meeting the agency’s strategic objectives. In addition, EPA is piloting an effort to collect data on coordination with states. However, EPA regional offices do not consistently collect or maintain data on informal enforcement actions. In addition, EPA does not require regional offices to collect and maintain data on their compliance assistance activities; therefore, it has no requirements for regional offices to enter data into the agency’s national databases. EPA requires regional offices to collect information from various data sources and enter it into national databases to monitor regulated entities’ compliance with environmental laws and track the agency’s enforcement actions. The information generally includes permit data on limits on emissions or for discharge of pollutants into waters, inspection or other evaluation data, violations data (e.g., failure to take or submit results for drinking water samples); informal enforcement actions, and formal enforcement actions, as shown in figure 1. OECA uses the data in its databases to manage the overall enforcement and compliance program and assess how well its efforts are meeting the objectives outlined in the agency’s strategic plan, according to EPA officials. For example, officials in one regional office told us that regional managers typically review ICIS data (for example, the number of inspections conducted) to monitor their progress toward meeting strategic objectives at the regional level. These regional officials said that staff in their office conduct monthly reviews of ICIS data to understand how their current efforts on certain indicators compare to prior years. OECA headquarters officials told us that the agency had begun to pilot a mechanism to collect data that can help measure agency progress in coordinating with states, one of the agency’s strategic objectives. Specifically, OECA officials told us that in 2018 the agency began a pilot effort to track instances in which regional office staff provide assistance with state enforcement actions, also characterized as “state assists.” According to agency guidance issued in June 2019, a state assist is defined as any instance in which the state could not or would not take the action without OECA’s help or any instance in which a state explicitly requests that OECA take over a case after OECA has identified a violation. During the pilot effort, state assists are documented as such when a regional office has expended substantial resources to identify a violation, develop the injunctive relief, or help the state take an action to obtain a remedy for the violation. According to OECA guidance, the pilot effort, which OECA officials expect to continue through 2021, will help the agency better track its efforts in this area. As of June 2019, according to our analysis of written responses, officials in eight of the 10 regional offices described having documented a state assist as defined by OECA. For example, officials in one regional office stated in their written response to our questions that one specific case against a company located in three different states would have been handled by the regional office. Instead, the regional office agreed to let two of the states take the lead for the cases in those states, and the regional office handled the case in the third state and documented this as two state assists. OECA collects data on some informal enforcement actions, such as the number of warning letters sent to regulated entities, but EPA regions do not always collect data about these actions, according to EPA headquarters officials. As a result, the data do not tell the full story of OECA’s enforcement efforts, according to OECA’s Assistant Administrator in testimony during a February 26, 2019, congressional hearing. Furthermore, OECA headquarters officials we interviewed said that data on EPA and state informal enforcement actions are incomplete in EPA’s ECHO website in part because EPA policy and related guidance for each of the various programs defines informal enforcement differently and these definitions can differ from the definitions in ECHO. In a 2010 document, EPA explained how the various agency policy guidance and ECHO define formal and informal enforcement actions differently. For example, the document states that policy guidance for the Clean Air Act defines notices of violation as formal enforcement actions, but that policy guidance for the Clean Water Act and the Resource Conservation and Recovery Act defines notices of violation as informal enforcement actions. Similarly, this same 2010 document states that administrative penalty orders of field citations are considered informal enforcement actions in the policy guidance for the Clean Water Act, but formal enforcement actions in the policy guidance for the Clean Air Act and Resource Conservation and Recovery Act. In addition, the document states that ECHO characterizes notices of violations under the Clean Air Act as informal enforcement actions even though the policy guidance defines them as formal enforcement actions. OECA headquarters officials highlighted two issues that affect the agency’s ability to consistently maintain data on informal enforcement actions: (1) using different definitions of informal enforcement actions across programs and (2) maintaining data on such actions inconsistently. OECA headquarters officials said that they were addressing the first issue of not having one clear definition of informal enforcement actions that applies across all of the air, water, and hazardous waste programs. In September 2019, OECA headquarters officials said EPA was finalizing a single definition of informal enforcement actions for the purpose of collecting more consistent information. In January 2020, EPA provided us with a September 30, 2019, memorandum that defines enforcement response tools, including a definition of informal enforcement action across all programs. Regarding maintaining data inconsistently, while most of the regional offices collect data on some informal enforcement actions, they use different mechanisms to maintain these data. According to our analysis of written responses, officials in nine of the 10 regional offices stated that their offices collect data on some informal enforcement actions such as warning letters, notices of noncompliance, notices of violation, and notices of determination. However, the officials described using different mechanisms for maintaining the data they collect on informal enforcement. For example, officials in five of the nine regional offices that collect data on some informal enforcement actions stated that they maintain the data in ICIS. As we described, ICIS data feeds into ECHO, which has components available to the public. In three of the nine regional offices that collect data on some informal enforcement actions, staff collect data on such actions in a database other than ICIS, such as a statute- or office-specific database, according to our analysis of written responses. Finally, one of the nine regional offices that collect data on some informal enforcement actions maintains those data in paper records, according to an official in that office. In our October 2017 report on key considerations for agency enforcement decisions, we reported that transparency and availability of data are important to promoting compliance and achieving regulatory objectives. As described earlier, EPA changed the focus of its national priorities from enforcement to compliance and increased its use of informal enforcement actions to achieve its regulatory objectives. Having complete information about informal enforcement actions is essential because EPA has elevated the role of such activities in its overall enforcement efforts. EPA often works informally with regulated entities to help them comply with environmental laws and regulations, according to its 2018 EPA Enforcement Annual Results report. However, the agency does not have complete information on those actions for evaluating its compliance monitoring and enforcement performance. Moreover, more complete and consistent information about OECA’s informal enforcement actions would provide a fuller picture of EPA’s overall enforcement efforts. This, in turn, would better enable EPA and OECA to assess whether they are achieving the agency’s regulatory objectives and improve the transparency of OECA’s informal enforcement actions for Congress and the public. Guidance can help agencies communicate expectations and ensure consistency with a standard. While EPA has issued guidance on how various agency policies and ECHO define formal and informal enforcement actions, the agency has not provided guidance to regional offices on how they should collect or maintain data on informal enforcement actions. According to federal standards for internal control, management should design control activities to achieve objectives and respond to risks, such as by clearly documenting internal control in management directives, administrative policies, or operating manuals. On September 30, 2019, EPA issued a memorandum that provides definitions for enforcement response tools, including informal enforcement actions, and instructions on how to report such actions. Now that the agency has finalized its definition of informal enforcement actions and specified which mechanisms to use to maintain data on such actions, by clearly documenting in guidance to the regional offices how they should use the definition to collect data on these actions, EPA would have better assurance that the regional offices consistently collect and maintain these data. According to EPA headquarters officials, OECA stopped requiring regional offices to collect data and report on their compliance assistance activities around 2012. Prior to that time, each regional office had a full- time staff member dedicated to coordinating compliance assistance activities, according to these officials. However, the staff member’s activities were the only compliance assistance data that regional offices collected and maintained. EPA officials stated that the regional offices stopped collecting the compliance assistance activities associated with this position when the agency redirected the funding for the full-time staff position to compliance monitoring and other enforcement efforts. As a result, EPA officials told us that the agency does not have consistent data about its compliance assistance activities. EPA officials told us that the agency made a policy decision to stop dedicating funding to compliance assistance but encouraged staff to continue conducting compliance assistance activities as part of the agency’s outreach for other programs. EPA headquarters officials said that as of September 2019, the agency had no plans to require regional offices to collect and report data on compliance assistance. However, according to these officials, although the agency stopped funding the compliance assistance coordinator position, regional staff continue to conduct a range of compliance assistance activities as part of their regular enforcement duties. Figure 2 shows the types of compliance and enforcement data that EPA collects, including that the agency does not require regional offices to collect information about compliance assistance. According to our analysis of written responses, officials in nine of the 10 regional offices reported that they collect some data on the compliance assistance activities their offices conduct. Officials in one office said that they do not collect data on compliance assistance activities because it is not required. The types of data on compliance assistance that the nine regional offices collect and the methods those offices use for maintaining the data differ, according to our analysis of written responses. For example, some regional officials described collecting data on compliance assistance provided over the telephone, and other officials described collecting data on on-site compliance assistance provided during inspections. According to our analysis of written responses, officials in two regional offices described providing on-site compliance assistance for minor issues during inspections and tracking the number of times such assistance was provided. Officials in the nine regional offices that still collect data on some compliance assistance activities described storing the data differently, either in region-specific databases or in paper files. Officials in two of these regional offices said that regional staff decide how to document telephone calls from regulated entities for assistance. Officials in one region stated that they no longer conduct large-scale compliance assistance activities such as conducting workshops or developing informational materials because EPA eliminated the reporting requirement. Having complete information about its compliance assistance activities is essential because EPA has elevated the role of such activities in its overall enforcement efforts. However, EPA does not have complete information on its compliance monitoring and enforcement activities, partly because the agency does not require the collection of data on compliance assistance activities. EPA’s lack of complete information on its compliance assistance activities is inconsistent with its change in policy. In addition, in our October 2017 report on key considerations for agency enforcement decisions, we reported that transparency and availability of data are important to promoting compliance and achieving regulatory objectives. Having complete information about its compliance assistance activities may provide more complete information on those activities for evaluating its compliance monitoring and enforcement performance. As discussed earlier, most of the regional offices continue to collect some information on compliance assistance even though they are not required to do so and use varying mechanisms to maintain the information. Because EPA does not direct the regional offices to collect data on compliance assistance activities, the agency would not have issued guidance instructing regional offices to collect such data and specifying which mechanism to use to maintain them. However, according to federal standards for internal control, management should design control activities to achieve objectives and respond to risks, such as by clearly documenting internal control in management directives, administrative policies, or operating manuals. Without clearly documenting in guidance to the regional offices that they should collect data on compliance assistance activities and specifying which mechanism to use to maintain the data, such as ICIS, EPA will not have the information it needs to track progress toward its strategic objective of increasing the agency’s use of compliance assistance activities to help regulated entities comply with laws and regulations. EPA communicates the results of its compliance monitoring activities and enforcement actions by making data available to the public and Congress through its website and annual reports. EPA’s ECHO website allows the public to view data over time, such as the number of facilities inspected by an authorized state or EPA from fiscal years 2011 to 2019. To help the public understand the data presented on its ECHO and other websites, EPA websites list a number of national and state-specific known data limitations concerning the data collected for its environmental programs. For example, the ECHO website identifies whether certain years of data are not appropriate for analyzing trends, such as its data on penalties under the Clean Water Act prior to 2015. EPA issues annual performance reports that include data on compliance monitoring and enforcement to fulfill requirements under the Government Performance and Results Act and other requirements. These reports describe progress toward the three strategic goals and related objectives in EPA’s Fiscal Year 2018-2022 Strategic Plan. In addition, since 2017, EPA has published a Year in Review report that outlines the agency’s accomplishments, including in the area of enforcement, using data on its compliance and enforcement actions to present the results of its efforts. In addition, at the end of each fiscal year, OECA publishes a Fiscal Year EPA Enforcement and Compliance Annual Results report and companion data graphs that provide enforcement data over a selected time period on such topics as the number of EPA inspections conducted, cases initiated, and value of fines and penalties collected. Environmental groups and media outlets have used EPA’s data to develop analyses, conclusions, and inferences about changes in EPA’s enforcement results. In December 2018, we reported that providing information about a dataset—for example, known limitations of the data in that dataset— allows users to determine whether the database is suitable for their intended purpose and make informed decisions about whether and how to use it. For example, EPA’s 2000 EPA Quality Manual for Environmental Programs states that published reports with environmental data shall be accompanied by a readily identifiable section or appendix that discusses the quality of the data and any limitations on the use of the data with respect to their original intended application. It also states that the agency’s reports should include applicable statements about possible misuse of the data for other purposes. EPA’s Fiscal Year 2018 Annual Performance Report includes a link to companion reports on its website that describe, among other things, the sources of the data used in the report and the known limitations of those data. Specifically, the companion reports include information such as the definition of terms used, units of measurement, data sources, method for analyzing the data, and the known limitations of the data. However, neither of EPA’s other 2018 annual reports we reviewed fully disclosed known limitations to the data the agency included in each report: Year in Review 2018. OECA’s Year in Review 2018 report, the most recent report available at the time of our review, includes a range of data—such as number of actions taken, monetary results, the reduction of emissions in tons, and data over selected time periods— to accompany its statements about the agency’s accomplishments. However, the report does not include any information about data sources or known limitations of the data. Fiscal Year 2018 EPA Enforcement and Compliance Annual Results. EPA’s Fiscal Year 2018 EPA Enforcement and Compliance Annual Results report, also the most recent at the time of our review, includes data sources and some known limitations of the data. For example, the report states that the data on results do not include state and local inspections or enforcement actions. Additionally, the report includes statements about changes in how the agency stores data that may prevent the data from being comparable across years. The report lists the various sources of the data used to create the report’s charts and graphs. EPA has published known limitations of these data on its ECHO website and indicated that broad data issues may affect the completeness, timeliness, or accuracy of the data in its various systems. However, based on our review of the report, it does not include information about known limitations of all of the data in the report. In addition, neither the Year in Review 2018 report nor the Fiscal Year 2018 EPA Enforcement and Compliance Annual Results report includes a readily identifiable section or appendix that discusses the known limitations of the data, as called for by leading practices for transparently reporting government data and as exemplified in EPA’s manual governing environmental data quality. In commenting on our assessment of the annual reports, EPA officials did not provide a reason why the reports do not discuss known data limitations but told us in a prior meeting that the current documentation on the ECHO website includes the current known data limitations. Furthermore, EPA’s Fiscal Year 2018 EPA Enforcement and Compliance Annual Results report does not fully describe how the data in the report should be interpreted given the known data limitations the report contains. For example, the 2018 annual results report provides a partial picture of overall enforcement of environmental laws because the data exclude state enforcement actions. In addition, for the yearly data across years (2008 through 2018 or 2012 through 2018), EPA does not fully provide information on any limitations in how the data should be analyzed; for example, whether the data are appropriate for the purpose of identifying trends or providing a snapshot of an activity for a single year. EPA does, however, include information on the impact of one or two large cases on the data presented for some data in the report such as the volume of contaminated soil and water to be cleaned up or the treatment and disposal of hazardous and nonhazardous waste. In our November 2019 report on data transparency, we concluded that without the transparent disclosure of known data limitations, users may view or analyze data without full knowledge of the extent to which the data are timely, complete, accurate, or comparable over time. Our November 2019 report also concluded that this could lead users to inadvertently draw inaccurate information or conclusions from the data. OECA’s Assistant Administrator has discussed the known limitations of EPA’s data in the annual reports. In a February 26, 2019, testimony before Congress, OECA’s Assistant Administrator stated that the averages for some of the metrics used in EPA’s annual results report cannot be interpreted to represent a statistical trend. OECA’s Assistant Administrator also stated that changes in the number of enforcement actions may be a function of changes in programmatic decisions and may not be reflective of changes in the underlying compliance of regulated entities with environmental statutes. By including the known limitations of data in its annual reports and providing information on the intended use of EPA’s data, as called for by leading practices for transparently reporting government data and as exemplified in existing EPA guidance for environmental data, EPA would have better assurance that Congress and the public are informed about the data presented and how the data should be interpreted. EPA collects a range of information and uses the information to manage its enforcement and compliance program and assess how well its efforts are meeting the objectives outlined in the agency’s strategic plan and other documents. However, while most of the regional offices collect data on some informal enforcement actions, they use different mechanisms to maintain these data, and the agency has not provided guidance to regional offices on how they should collect or maintain the data. Without documenting in guidance to the regional offices how they should collect data on informal enforcement actions and specifying which mechanism to use to maintain the data, EPA lacks assurance that the regional offices will consistently collect and maintain these data. On September 30, 2019, EPA issued a memorandum that provides definitions for enforcement response tools, including informal enforcement actions and instructions on how to report such actions. We view this as a step in the right direction. Now that the agency has finalized its definition of informal enforcement actions and provided instructions on how regional offices should report such actions, by clearly documenting in guidance on how regional offices should use the definition to collect data on these actions, EPA would have better assurance that the regional offices consistently collect and maintain these data. Similarly, EPA does not have complete information on its compliance monitoring and enforcement activities because the agency does not require the collection of data on compliance assistance activities. As a result, the agency has not issued guidance instructing regional offices to collect such data and specifying which mechanism to use to maintain them. Without clearly documenting in guidance to the regional offices that they should collect data on compliance assistance activities and specifying which mechanism to use to maintain the data, such as ICIS, EPA will lack key information. Such information is needed to track progress toward its strategic objective of increasing the agency’s use of compliance assistance activities to help regulated entities comply with laws and regulations. While EPA communicates the results of its compliance monitoring activities and enforcement actions through its website and annual reports, neither of its 2018 annual reports includes a readily identifiable section or appendix that discusses the known limitations of the data. The 2018 annual results report also does not fully describe how the data in the report should be interpreted, given the known data limitations the report contains. By including the known limitations of the data in its annual reports and providing information on the intended use of EPA’s data, EPA would have better assurance that Congress and the public are informed about the data presented and how the data should be interpreted. We are making the following three recommendations to EPA: The Assistant Administrator for EPA’s Office of Enforcement and Compliance Assurance should clearly document in guidance to the regional offices how they should use the definition of informal enforcement actions to collect data on these actions. (Recommendation 1) The Assistant Administrator for EPA’s Office of Enforcement and Compliance Assurance should clearly document in guidance to the regional offices that they should collect data on compliance assistance activities and specify which mechanism to use to maintain the data, such as ICIS. (Recommendation 2) The Assistant Administrator for EPA’s Office of Enforcement and Compliance Assurance should include the known limitations of data in its annual reports and provide information on the intended use of EPA’s data. (Recommendation 3) We provided a draft of this report to EPA for review and comment. In its written comments, reproduced in appendix I, EPA stated that it agreed with all three of our recommendations and many of our findings and conclusions. EPA also provided technical comments, which we incorporated into the report, as appropriate. In response to our first recommendation to clearly document in guidance how regional offices should use the definition of informal enforcement to collect data on these actions and specify a mechanism to maintain the data, EPA said that the agency issued a September 30, 2019, memorandum for headquarters and regional enforcement offices to implement. This memorandum provides guidance on EPA definitions for enforcement response tools, to promote consistency and clarity in the use of enforcement terms, according to EPA. EPA also said that the guidance defines “informal enforcement action.” The guidance includes instructions on how to report such actions. The guidance states that, with two exceptions, headquarters and regional offices are expected to report, in ICIS, all informal enforcement actions across all programs that meet the new definition. In addition, the guidance states that because it is only a definitional document and does not include guidance on appropriate use of the enforcement response policy tools, the agency will work to identify the specific changes in practice needed (i.e., changes in use and reporting). The guidance states that EPA anticipates that informal enforcement actions meeting the new definition will be included in the agency’s certified annual enforcement results beginning in fiscal year 2020. We view EPA’s guidance as a step in the right direction, and the guidance states that EPA will provide training and additional guidance for enforcement staff to ensure consistent implementation across regional offices and headquarters. Additional guidance will provide EPA with an opportunity to specify how regional offices are to use the definition of informal enforcement to collect data on these actions. We modified our recommendation because EPA’s recent guidance specifies mechanisms for EPA employees to maintain data on informal enforcement actions. In response to our second recommendation to clearly document in guidance that regional offices should collect data on compliance assistance activities and specify a mechanism to maintain the data, EPA said that it would collect data on compliance assistance for each of the National Compliance Initiatives and maintain those data in ICIS. In response to our third recommendation to include known data limitations in annual reports and provide information on intended use of its data, EPA stated that it acknowledges the importance of providing information about a dataset to facilitate proper interpretation. For that reason, EPA said that, in time for its fiscal year 2020 report, the agency will create a webpage to describe how best to interpret the data presented in the agency’s Fiscal Year EPA Enforcement and Compliance Annual Results report and include a reference to that webpage in the report itself as well as the Year in Review report. In technical comments related to our third recommendation, EPA stated that several of the limitations we identified in the report do not affect the data included in its Fiscal Year EPA Enforcement and Compliance Annual Results report. In considering EPA’s technical comments, we modified the text of the report concerning examples of the annual report’s data limitations, as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Administrator of EPA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. In addition to the contact named above, Chad M. Gorman (Assistant Director); Tahra Nichols (Analyst in Charge); Mark Braza; Courtney Carroux; Tara Congdon; Jazzmin Cooper; Matthew Hunter; Caroline Prado; Dan Royer; Jeanette Soares; Kiki Theodoropoulos, Sonya Vartivarian, and Michelle R. Wong made key contributions to this report.", "summary": "Enforcing environmental laws and regulations, including those governing water, air, and hazardous waste, is a central part of EPA's mission. In partnership with states, EPA oversees compliance with these requirements for about 800,000 regulated entities, such as refineries and sewage treatment plants. OECA carries out much of EPA's compliance and enforcement responsibilities through the agency's 10 regional offices. OECA has a range of compliance assistance, compliance monitoring, and enforcement tools available to elicit compliance with laws and regulations from regulated entities. These tools include conducting on-site inspection, training staff and providing technical assistance, developing cases, and issuing warning letters. GAO was asked to review EPA's enforcement efforts. This report examines, among other objectives, the types of information EPA collects on its compliance assistance, compliance monitoring, and enforcement actions. GAO analyzed written responses to its questions from all 10 regional offices, reviewed agency documents and databases, and interviewed EPA officials in headquarters and regional offices. The Environmental Protection Agency (EPA) collects a range of information on compliance and enforcement such as data on inspections, violations, and enforcement actions. The agency uses these data to manage its efforts and assess progress in meeting the agency's strategic objectives. In an August 2018 memorandum, EPA's Office of Enforcement and Compliance Assurance (OECA) reported a key strategic change to increase compliance assistance activities (e.g., training) and informal enforcement actions (e.g., warning letters). However, the agency does not consistently collect or maintain data on either type of action (see figure). Specifically, OECA has not directed regional offices to collect or report data on compliance assistance activities since 2012 and, consequently, does not have guidance instructing regional offices to collect such data and specifying which mechanism offices should use to maintain these data. Also, the agency did not provide guidance to those offices defining informal enforcement actions or how to maintain data on them until September 30, 2019, but the guidance does not specify how to collect data on such actions. By clearly documenting in guidance how the offices should use the definition to collect data on such actions, EPA could more consistently collect these data. As the figure shows, OECA does not require regional offices to collect data on compliance assistance or complete data on informal enforcement actions. Having complete information about its compliance assistance activities and informal enforcement is essential because EPA has elevated the role of such activities in its overall enforcement efforts. However, because EPA is not consistently collecting these data, the agency cannot be sure it is achieving its strategic objectives. EPA would have better assurance it has the information it needs by clearly documenting in guidance to the regional offices that they should: collect data on compliance assistance activities and informal enforcement actions and specify which mechanism to use to maintain compliance assistance data. By doing so, EPA would have better assurance that the regional offices consistently collect and maintain these data in order to track progress toward the agency's strategic objective of increasing the use of such activities and actions. GAO is making three recommendations to EPA, including that it should clearly document in guidance to its regional offices that they should collect data on compliance assistance activities and informal enforcement actions and specify which mechanism to use to maintain compliance assistance data. EPA agreed with GAO's recommendations and stated that the agency has either begun to or plans to implement them.", "document_type": "gao"}
{"report": "The Goldwater-Nichols Department of Defense Reorganization Act of 1986, in part, was intended to improve joint officer management policies, otherwise enhance the effectiveness of military operations, and improve DOD’s management and administration. With the Goldwater-Nichols Act, Congress also intended to, consistent with the congressional declaration of policy in section 2 of the National Security Act of 1947 and among other things, reorganize DOD and strengthen civilian authority in DOD. The Goldwater-Nichols Act, as amended, also: established various joint officer management policies, including requiring JPME for certain joint assignments and promotion categories; required officers to successfully complete an appropriate program at a JPME school, among other things, to be designated as joint qualified—a prerequisite for promotion to brigadier general or rear admiral lower half rank except under certain circumstances; and required the Secretary of Defense, with the advice and assistance of the Chairman of the Joint Chiefs of Staff, to periodically review and revise the curriculum of JPME schools to enhance the education and training of officers in joint matters. In addition, the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 required the Secretary of Defense to implement a comprehensive framework for officer JPME. The PME continuum consists of five military educational levels that correspond to the five phases of a military officer’s career: (1) precommissioning, (2) primary, (3) intermediate, (4) senior, and (5) general/flag officer. As figure 1 indicates, intermediate- and senior-level PME and JPME programs—the focus of our review—are designed for officers at pay grades O-4 through O-6. As identified in figure 2 below, the military services’ intermediate- and senior-level PME programs tailor curricula according to their respective services’ needs. For example, the Army, Navy, and Marine Corps PME programs focus on land, maritime, and maneuver warfare, respectively. Further, the Chairman’s instruction concerning officer PME and JPME (hereinafter referred to as the Officer Professional Military Education Policy, or “OPMEP”) requires that JPME be integrated across a diverse array of academic topics, including history and political science, and, where appropriate, be offered in conjunction with PME. Collectively, PME and JPME prepare officers, throughout their careers, to increase their knowledge and develop the necessary skills to operate in joint environments, such as a combatant command. PME and JPME also are offered through distance learning and satellite education programs for non-resident students. The OSD, Chairman, and military services are responsible for overseeing the services’ PME and JPME programs. OSD: Within OSD, the Secretary of Defense has delegated responsibility for, among other things, military readiness, total force management, and military and civilian personnel training to the Under Secretary of Defense for Personnel and Readiness. Under DOD Directive 5124.02 the Under Secretary is responsible for, among other things, developing education policies, plans, and programs for the education of all DOD personnel, including PME and JPME programs. Within OUSD(P&R), the Deputy Assistant Secretary of Defense for Force Education and Training (DASD(FE&T)) was established in 2015. The DASD(FE&T) is responsible for developing policies, plans, programs, budgets, and other activities necessary to develop, guide, measure, implement, assess, and oversee all aspects of education and training for military personnel following basic officer and enlisted training, which includes PME and JPME programs. The USD(Comptroller) is the principal staff assistant and advisor to the Secretary of Defense on budgetary and financial matters. The USD(Comptroller) focuses on budgetary formulation and execution; financial management and oversight; and accounting policy; among other things. The USD(Comptroller), among other things, directs the formulation and presentation of DOD budgets; and establishes and supervises the execution of uniform DOD policies, principles, and procedures, including terminologies and classifications, as necessary for certain budgetary and financial matters. Chairman: With the advice and assistance of the Chairman, the Secretary of Defense periodically reviews and revises the JPME curriculum to enhance the education and training of officers in joint matters. The OPMEP outlines the Chairman’s roles and responsibilities as they relate to PME and JPME. According to the OPMEP, the Chairman formulates polices for coordinating military education and advises and assists the Secretary of Defense through the designation and certification/accreditation of JPME. The Chairman accredits military service programs through periodic Process for the Accreditation of Joint Education (PAJE) reviews. Further, the Joint Staff Directorate for Joint Force Development is responsible for, among other things, reviewing the Chairman’s PME policies, overseeing the Military Education Coordination Council, and coordinating PAJE reviews. Military services: The military services provide PME to develop officers with expertise and knowledge appropriate to their grade, branch, and occupational specialty. Each military service is responsible for funding, developing curriculum for, and administering their respective PME programs. In addition, for programs accredited to award JPME, each military service is responsible for meeting the Chairman’s PAJE accreditation requirements and providing qualified military students and faculty to the other military services’ PME programs in accordance with the OPMEP. Membership on PAJE teams, which accredit military services’ PME programs, will be tailored to provide the appropriate balance of expertise in JPME learning areas, objectives, criteria, and standards. All of the military services’ intermediate- and senior-level PME programs are accredited to award master’s degrees. Each program undergoes a Department of Education-governed civilian accreditation process generally every 10 years, depending on the accreditor and the program. Civilian accreditation for the military services’ PME programs occurs at the institution level and includes multiple programs. For example, the civilian accreditation of Marine Corps University includes the Marine Corps’ intermediate- and senior-level PME programs, as well as other programs such as its School for Advanced Warfighting. According to PME program and civilian accreditation officials, the civilian accreditation process starts with the institution conducting a detailed self-evaluation of its performance, and preparing and providing a self-evaluation report to the accreditation officials. This is followed by a site visit by the accreditation officials and a report describing the institution’s compliance with applicable academic quality standards. The accreditation process concludes with the accreditor’s decision on the institution’s accreditation status. Table 1 shows when each of the military services’ intermediate- and senior-level PME program was last accredited (at the institutional- level) for civilian accreditation. Accreditation bodies assess academic quality by applying and enforcing standards in the following areas required, generally, by the Department of Education: (1) success with respect to student achievement; (2) curricula; (3) faculty; (4) facilities, equipment, and supplies; (5) fiscal and administrative capacity; (6) student support services; (7) recruiting and admissions practices; (8) measures of program length and objectives of the degrees or credentials offered; (9) record of student complaints received by, or available to, the accreditation body; and (10) record of compliance with certain federal student loan program responsibilities. Within these areas, civilian accreditation bodies develop their own accreditation standards, which can vary (see table 2). The military services’ intermediate- and senior-level PME programs are assessed against the applicable accreditation standards to enable the PME programs to award master’s degrees. There is no Chairman or OSD requirement for the military services’ PME programs to have civilian accreditation status, but officials reported several benefits related to civilian accreditation. Specifically, DOD and civilian accreditation officials stated that civilian accreditation provides additional assurance from a recognized external authority that the military services’ PME programs are meeting educational standards required of DOD and non-DOD programs alike. In addition, we previously reported that the U.S. accreditation system’s use of peer review offers the relevant expertise to assess academic quality and provides institutions with feedback for improvement as a key strength of the system. Furthermore, DOD officials said that the ability to award master’s degrees from an accredited program helps the programs attract and retain high- quality faculty. All of the military services’ PME programs have been accredited by the Chairman to award JPME credit. The OPMEP outlines the JPME program accreditation requirements and processes that are to occur at least every 6 years. DOD’s process for accrediting the military services’ JPME programs is through the Chairman’s PAJE. The PAJE is based on accepted civilian accreditation standards and practices. According to the OPMEP, the PAJE serves three purposes: (1) oversight, (2) assessment, and (3) improvement. Once JPME programs are initially accredited, accreditation is reaffirmed through subsequent PAJEs every 6 years. In advance of a PAJE accreditation, the military service PME program submits an OPMEP-required self-assessment, which the PAJE team reviews prior to conducting the on-site accreditation. The PAJE team prepares a report on its findings, and includes a full, conditional, or no accreditation determination. PME programs receiving a conditional accreditation or reaffirmation must demonstrate improvements in particular areas within a specific timeframe in order maintain their accreditation. Any program that fails to achieve accreditation, reaffirmation, or conditional accreditation/reaffirmation is no longer a JPME provider. According to the OPMEP, accreditation or reaffirmation is awarded when programs are judged satisfactory overall and have no significant weaknesses. Table 3 shows the date of the most recent JPME accreditation for each of the military services’ intermediate- and senior-level PME programs. Additionally, the military services’ PME programs have (1) met or partially met all of the required joint learning areas, such as joint command and control; and (2) met or partially met all required common educational standards, such as periodically assessing their JPME programs. First, the OPMEP requires intermediate- and senior-level PME programs to fulfill the appropriate joint learning areas and objectives and common educational standards, and generally have a curriculum that includes the required JPME content prescribed in statute. The PAJE review of the joint learning areas and common educational standards includes a combination of objective and subjective assessment based on peer expertise. Specifically, the OPMEP requires intermediate-level PME programs to fulfill the following six joint learning areas: (1) National military capabilities strategy, (2) Joint doctrine and concepts, (3) Joint and multinational forces at the operational level of war, (4) Joint planning and execution processes, (5) Joint command and control, and (6) Joint operational leadership and the profession of arms. The OPMEP requires senior-level PME programs to fulfill the following five joint learning areas: (1) National strategies; (2) Joint warfare, theater strategy and campaigning for traditional and irregular warfare in a joint, interagency, intergovernmental and multinational environment; (3) National and joint planning systems and processes for the integration of joint, interagency, intergovernmental and multinational capabilities; (4) Command, control and coordination; and (5) Strategic leadership and the profession of arms. According to the most recent Joint Staff PAJE accreditation reports, all of the military services’ intermediate- and senior-level PME programs met all of these mandatory joint learning areas, with the exception of the Marine Corps intermediate-level PME program which received a partially meets in the joint learning area for joint planning and execution processes. Second, the OPMEP also requires intermediate- and senior-level PME programs to meet seven common educational standards that the Chairman considers essential in awarding JPME credit. Table 4 describes these seven common educational standards. The most recent Chairman’s accreditation review found that each of the military services’ PME programs met or partially met all seven OPMEP- required common educational standards, as shown in table 5. According to Joint Staff officials, to be assessed as “met,” the program must meet all of the criteria for that common educational standard. On the other hand, if a program does not meet all of the criteria then it “partially met” the criteria for the accreditation standard. When a PAJE team determines that a program “partially met” a standard, the team suggests corrective actions for the program to consider. Receiving a “partially met” on a particular standard does not exclude a program from being accredited, as accreditation is based on the program being judged satisfactory overall and having no significant weaknesses. We identified the following examples of common educational standards that were met or partially met by the military services’ intermediate- and senior-level PME programs during our analysis of the Chairman’s most recent accreditation reports for those programs. Standard 2: Employ Predominantly Active and Highly Effective Instructional Methods – The PAJE team found that the College of Naval Warfare met this standard during its most recent review in May 2015. This standard states that instructional methods should be appropriate to the subject matter and desired levels of learning, and should employ active student learning whenever feasible. In addition, the standard requires that the goals of the educational offerings be rigorous and challenging, requiring that students engage in critical thinking and active interaction. Specifically, the PAJE team found that the College of Naval Warfare employed a preponderance of active instructional methods to achieve desired learning outcomes. The team found that the effective combination of Socratic discussion, case studies, practical exercises, written assignments, and lectures followed by seminar discussions, engaged students in critical thinking and were appropriate to the desired levels of learning. The PAJE team also found that active student discourse occurred both inside and outside of seminars. Lastly, the team found that the effectiveness of the curriculum in refining critical thinking skills was reflected in both student and alumni surveys. Standard 3: Assess Student Achievement – The PAJE team found that the Marine Corps Command and Staff College met this standard during its most recent review in September 2014. This standard states that each college should aggressively assess its students’ performance, clearly state educational goals and objectives, and measure students’ performance against defined standards using direct and indirect assessment tools to identify whether desired educational outcomes are being achieved. Specifically, the PAJE team found that the Marine Corps Command and Staff College clearly identified program outcomes, student learning outcomes, and lesson educational objectives. The PAJE team also found that student assessments were directly linked to student learning outcomes, joint learning areas, and joint learning objectives. Additionally, the team found that results were carefully tracked and used for educational outcome achievement verification, curriculum improvement, and faculty development feedback. Lastly, the PAJE team found that the College used a variety of student assessments—including research papers, exams, staff papers, oral presentations, exercises, practicums, oral defenses, and seminar participation—to provide feedback and verify learning outcome achievement. Standard 4: Assess Program Effectiveness – The PAJE team found that the Army’s Command and General Staff College partially met this standard during its most recent review in February 2014. This standard states that colleges should survey students, graduates, and their supervisors to determine curricula and educational effectiveness of their academic programs. The standard also states that leadership should periodically assess the intended educational outcomes of programs for currency, relevancy, and completeness, and the results of these analyses should be used to refine or develop curricula that continue to meet evolving mission requirements in the context of an ever-changing world. Specifically, the PAJE team found that there is a robust evaluation and assessment process for the common core courses but that neither the electives nor the Command and General Staff College-level outcomes were assessed. Additionally, the PAJE team found that there did not appear to be a process for evaluating the overall curriculum either directly or indirectly. The PAJE team suggested that the Army’s Command and General Staff College develop a capstone evaluation to assess outcomes of its common core curriculum. Army’s Command and General Staff College officials told us that in 2016 the college developed a capstone evaluation for its common core curriculum, consisting of an online examination and a faculty member oral examination. Standard 5: Conduct Quality Faculty Recruitment: Selection, Assignment, and Performance Assessment Program – The PAJE team found that the Air War College partially met this standard during its last review in October 2014. This standard states that faculty should have the academic credentials, teaching skills, and experience in joint and professional matters necessary to teach in the colleges. This standard also states that faculty roles and responsibilities should be clearly documented, and that colleges should hold faculty accountable to clearly defined and measurable performance criteria and standards. Specifically, the PAJE team found that the Air War College did not meet the OPMEP standard for its student-to-faculty ratio, but acknowledged that the college had a plan to meet this requirement by the spring of 2015. The Air War College met the student-to-faculty ratio in academic year 2015. The review also found that delays in hiring presented challenges in maintaining the requisite number of qualified faculty. The PAJE team suggested that the Air War College continue its efforts to reduce the time to complete civilian hiring actions. Air War College officials stated that as part of a wider Air University effort to streamline the civilian hiring process they were able to ameliorate this challenge by making the process more transparent, predictable, and shorter. Most of the military services’ senior-level PME programs met the OPMEP JPME seminar student mix accreditation requirement, which is part of the develop joint awareness, perspective, and attitude common educational standard (Standard 1) that pertains to joint acculturation. However, not all of the military services’ intermediate-level PME programs met the seminar student mix accreditation requirement. The OPMEP requires that each intermediate- and senior-level JPME seminar contain at least one student from each of the two non-host military departments: the Department of the Army, the Department of the Navy (which includes the Marine Corps), and the Department of the Air Force. DOD defines joint acculturation as the process of understanding and appreciating the separate military service cultures resulting in joint attitudes and perspectives, common beliefs, and trust. All but one of the military services’ senior-level PME programs met the seminar student mix accreditation requirement from academic years 2014 through 2018. During that timeframe there were approximately 300 senior-level seminars, and only one did not meet the requirement. Specifically, during academic year 2017, the Air Force’s senior-level PME program lacked sufficient Navy representation for one seminar. However, not all of the military services’ intermediate-level PME programs met the seminar student mix accreditation requirement. Specifically, the Air Force’s and the Army’s intermediate-level PME programs had less than the required Sea Service representation for 3 years between academic years 2014 and 2018. For academic years 2016 and 2018, the Air Force’s intermediate-level PME program had less than the OPMEP-required Sea Service representation for about 24 percent of its seminars (totaling 288 students), as shown in table 6 below. During the 3- year timeframe, the Army’s intermediate-level PME program had less than the required Sea Service representation for about 22 percent of its seminars (totaling 664 students). On the other hand, the Navy’s and the Marine Corps’ intermediate-level PME programs generally met their respective seminar student mix accreditation requirement for each of the last 5 academic years (2014 – 2018). According to Navy officials and documentation, the Navy stated that it was unable to provide the other military services’ intermediate-level PME programs with the required numbers of officers during academic years 2016 – 2018 because of competing staffing priorities, such as its forward presence mission. However, we found that the Navy provided sufficient officers to its own intermediate-level PME program (College of Naval Command and Staff) during each of these academic years so that it could have instead assigned the required number of officers to the Air Command and Staff College and the Army’s Command and General Staff College to meet their respective Sea Service requirements. For example, the Navy sent 121 Navy officers to the College of Naval Command and Staff in academic year 2018 for 27 seminars when the Air Command and Staff College and the Army’s Command and General Staff College needed a cumulative total of 32 officers to meet their OPMEP seminar student mix requirement. As a result, most of the College of Naval Command and Staff’s seminars would have only been reduced by one Navy Officer. Officials from all of the military service PME programs told us that students interacting with students from other military departments is critical for joint acculturation. Officials from the Joint Staff Directorate for Joint Force Development reinforced the importance of the seminar student mix requirement, stating that satisfying the OPMEP common educational standard of developing joint awareness, perspective, and attitude (Standard 1) is dependent on time and intensity of student interaction with students from other military departments. Military service and Joint Staff officials stated it was difficult for Air Force and Army officers to gain a full appreciation of the Navy’s contribution to joint military operations when there were no Sea Service students in the seminar. In the situations where a seminar did not have Sea Service representation, Joint Staff officials told us that a decision was made to award students JPME credit. Furthermore, officials told us that it was decided to not “punish” military service PME programs for not meeting the OPMEP’s JPME seminar student mix requirement as military services’ programs cannot control the number of in-bound students assigned by the other military services. Officials from the Air Force’s and the Army’s intermediate-level PME programs told us that when they are unable to meet the OPMEP seminar student mix requirement, they take steps to compensate for the lack of Sea Service student representation, such as using faculty to provide Sea Service perspectives. Similarly, a 2010 Congressional report noted the value of in-residence officer PME programs because of the acculturation opportunities that they offer. Other than Joint Staff officials requesting that the Navy meet the OPMEP’s JPME seminar student mix requirement, no other actions have been taken by the Chairman, OSD, or the Navy to resolve the issue concerning Navy participation in the Air Force’s and Army’s intermediate- level PME programs. Specifically, according to DASD(FE&T) officials, as of November 2019, OSD has not been involved in addressing the Navy’s failure to meet the OPMEP’s JPME seminar student mix requirement. Additionally, Joint Staff officials told us that the Chairman cannot direct a Secretary of a military department to comply with provisions of a Chairman’s publication. However, Standards for Internal Control in the Federal Government state that management should identify, analyze, and respond to risks related to achieving defined objectives. Given that joint acculturation is a key component of intermediate-level PME programs, the lack of action to resolve or mitigate the issues at hand has the potential to negatively affect students’ opportunities to increase their knowledge and develop the necessary skills to operate in joint environments. Without DOD taking steps to determine whether the appropriate number of Navy officers can be assigned to intermediate-level PME programs of the Air Force and Army, the officers participating in these programs lack the perspectives of Sea Service participants, which diminishes the quality of the educational experience. Furthermore, neither the Chairman nor OUSD(P&R) has evaluated or approved the mitigation steps, either before or after-the-fact, when a PME program lacks representation to meet the joint acculturation requirement. Although the OPMEP requires that each intermediate- and senior-level JPME seminar contain at least one student from each of the two non-host military departments, the OPMEP does not contain guidance on what PME programs should do when they do not meet this requirement. Developing guidance concerning actions, if any, the military services can take to mitigate JPME seminar student mix shortfalls and still meet the intent of the OPEMP’s joint awareness common educational standard could better position DOD and the military services to ensure that DOD’s JPME programs are meeting their objectives. OSD has had PME and JPME statutory oversight responsibilities for more than 30 years; however, while it has taken some steps to strengthen its oversight, it is not well-positioned to assess the effectiveness of the military services’ PME programs. The Goldwater-Nichols Act, as amended, states that the Secretary of Defense shall, with the advice and assistance of the Chairman of the Joint Chiefs of Staff, periodically review and revise the curriculum of JPME schools, and require that the PME schools periodically review and revise their intermediate- and senior-level PME curriculums to strengthen the focus on joint matters and preparing officers for joint duty assignments. Moreover, DOD Directive 5124.02 requires the Under Secretary of Defense for Personnel and Readiness to develop policies, plans, and programs for educating DOD personnel. According to several DOD officials with whom we spoke, prior to the establishment of DASD (FE&T), OUSD(P&R) unofficially relinquished its responsibilities for PME and JPME to the Chairman, whose office issued the first version of the OPMEP in 1996. As mentioned earlier, the OPMEP outlines the Chairman’s process for meeting statutory responsibilities for overseeing officer JPME, which is a subset of PME. For example, the OPMEP states that JPME is a Chairman approved body of objectives, outcomes, policies, procedures, and standards supporting the educational requirements for joint officer management. As recently as 2017, OUSD(P&R) reported to Congress that it had no formal process for exercising its authority to periodically review and revise the curricula of officer JPME. In the same report, OUSD(P&R) stated that DOD was reviewing JPME and the DOD Joint Officer Management Program. OUSD(P&R) also reported that with the reorganization of its office to include a Deputy Assistant Secretary of Defense for Force Education and Training (DASD(FE&T)) in 2015, OSD was now organized to exercise its statutory authorities with respect to PME and JPME and would do so in line with the Secretary of Defense’s direction in the National Defense Strategy. According to the 2015 implementation plan detailing the reorganization, the Deputy Assistant Secretary’s responsibilities include measuring, assessing, and overseeing all aspects of education and training, which includes PME and JPME. In 2019, DOD issued guidance stating that the Assistant Secretary of Defense for Readiness is the principal advisor to the Under Secretary of Defense for Personnel and Readiness on all matters related to the readiness of the Total Force, including by developing policies and plans, providing advice, and making recommendations for PME to include alignment to the National Defense and National Military Strategies and talent management and utilization. OUSD(P&R) is drafting its first DOD instruction (the draft instruction) that covers PME and JPME, which DASD(FE&T) officials told us it plans to issue in February 2020. According to DASD(FE&T) officials, once issued, the DOD instruction will be the prevailing policy document for PME and JPME at the OSD-level. While we believe these steps will improve OSD’s oversight of the military services’ PME and JPME programs, we identified areas that could continue to impede DOD’s ability to assess the effectiveness of these programs. Specifically: DOD lacks a mission statement and performance measures for its PME and JPME programs. DASD(FE&T) officials stated that prior to the draft policy OUSD(P&R) had not developed a mission statement and performance measures for PME, but told us that the draft instruction would include a mission statement and examples of performance measures. However, we did not identify a mission statement for PME that clearly defines the respective key purposes for this program when we reviewed the draft instruction. According to leading training and development practices, a mission statement is important to an organization’s success because it explains the organization’s purpose and goals and is the basis for goal-directed performance measures. The draft instruction proposes the performance measures the military services should track and assess as part of their required annual program reviews, such as graduate assignments and retention rates. Performance measures are important because they assess an organization’s progress toward achieving results that are aligned with its mission. However, without a department-wide mission statement for PME and JPME, OUSD(P&R) is not well-positioned to propose performance measures for the military services to track and enable OUSD(P&R) to assess the effectiveness of these programs. Further, our review of the draft instruction found no examples of cost- related performance measures. DASD(FE&T) officials confirmed that cost-related performance measures were not included in the draft instruction, but told us that they planned to coordinate with officials from the Joint Staff Directorate for Joint Force Development to refine the performance measures sometime in the future. DOD’s Financial Management Regulation states that performance measurement is a means of evaluating efficiency, effectiveness, and results, and that a balanced performance measurement scorecard includes nonfinancial and financial measures focusing on quality, cycle time, and cost. Moreover, leading training and development practices state that performance measures should include both qualitative and quantitative measures to assess training results, and include the identification and tracking of costs. These same leading practices state that organizations should compare associated costs and monetized benefits of training programs to determine return on investment. DASD(FE&T) officials told us that having cost information on the military services’ PME and JPME programs to determine return on investment would enable their office to compare and make well- informed decisions about these programs. DOD lacks a requirement for the military services to report periodically on PME and JPME programs. OUSD(P&R) has not established a requirement for the military services’ to periodically report information to its office on the military services’ respective PME and JPME programs. For example, the Chairman’s PAJE reports that document accreditation findings and include a full, conditional, or no accreditation determination are not provided to OUSD(P&R). According to the OPMEP, PAJE reports will be forwarded to the Chief of the applicable military service, the Director of the Defense Intelligence Agency, or the President of the National Defense University for appropriate action. A Joint Staff official confirmed that PAJE reports are not provided to OUSD(P&R). Our review of the draft instruction found no requirement for the Chairman to provide PAJE reports to OUSD(P&R), nor is there a requirement for the military services to report information on their PME and JPME programs—such as their annual program reviews—to OUSD(P&R). According to DASD(FE&T) officials, reporting requirements were omitted from the draft instruction because their office lacks the personnel to review and assess the information the military services would be required to collect and report. However, without a requirement for the military services’ to periodically report information on their PME and JPME programs, OUSD(P&R)’s ability to assess the effectiveness of these programs and perform meaningful oversight will continue to be limited. Leading training and development practices state that organizations should collect appropriate performance data during implementation and establish accountability for the results of these efforts. Additionally, Standards for Internal Control in the Federal Government state that management relies on quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks. These same standards state that management should receive quality information about the entity’s operational processes to help management achieve the entity’s objectives. Because OUSD(P&R) does not require the military services to periodically report information on their respective PME programs, it does not have information that would help it assess the effectiveness of these programs. We believe that addressing these limitations will enhance the ability of OUSD(P&R) and its subordinate office (i.e., DASD(FE&T)) to oversee and assess the effectiveness of the military services’ PME programs. USD(Comptroller’s) ability to monitor the military services’ PME programs is limited because the military services’ budget request data are incomplete and lack uniformity. DOD’s Financial Management Regulation requires the military services to submit separate budget request data on PME programs in support of DOD’s annual budget request, and this data is included in DOD’s annual congressional budget justification exhibits. While the Financial Management Regulation requires the military services to submit separate annual budget request data exhibits for most of their intermediate- and senior-level PME programs, it does not require the Marine Corps to submit an exhibit for its senior-level PME program, the Marine Corps War College. Based on our review of the Marine Corps’ fiscal years 2014 through 2020 budget request data exhibits and according to the USD(Comptroller) and Marine Corps officials, the Marine Corps did not submit a budget request data exhibit for the Marine Corps War College during this 7-year period. USD(Comptroller) and Marine Corps officials could not explain why the Marine Corps War College was omitted from the DOD Financial Management Regulation, where DOD last updated the chapter requiring this submission in September 2008. In addition, the data the military services include in their annual budget requests varies. DOD Directive 5118.03 outlines USD(Comptroller) responsibilities, requiring the Comptroller to, among other things: (1) direct the formulation and presentation of DOD budgets; and (2) establish and supervise the execution of uniform DOD policies, principles, and procedures, including terminologies and classifications, as necessary, for budget formulation, presentation, and execution, and certain other topics. Additionally, section 2162 of title 10, U.S. Code, requires the Secretary of Defense, with the advice and assistance of the Chairman of the Joint Chiefs of Staff, to promulgate a uniform cost accounting system for use by the Secretaries of the military departments in preparing budget requests for the operation of PME schools. However, the DOD Financial Management Regulation does not specify how the military services should account for the data required for the military services’ budget request data submissions. Consequently, the budget request data reported by the military services varies. For example, in their fiscal year 2020 budget request data submissions the Army and the Air Force combined distance education and in-residence education programs, the Navy reported this data in separate exhibits, and the Marine Corps omitted distance education costs for its intermediate-level PME program. Additionally, according to DOD officials, the extent to which the military services accounted for costs to operate and maintain their PME colleges—such as security, facility maintenance, and information technology support—varies. In 1987, the year following the passage of the Goldwater-Nichols Act, the House Armed Services Committee established a panel on PME led by Representative Ike Skelton (the Skelton Panel). The Skelton Panel undertook a comprehensive congressional review of PME, and published its findings and recommendations in a 1989 report (the Skelton Report). Although the Skelton Panel did not take a comprehensive look at how well PME institutions were funded to accomplish their mission, the panel inquired into cost per student at each school and reported receiving from OSD raw data submitted by each PME institution, which included considerable differences in scope and cost methodology used by the PME institutions. The Skelton Report recommended that DOD establish a uniform cost accounting system for the PME schools, and that the annual report of the Secretary of Defense provide data on PME costs beginning in 1990. A 2010 congressional report focused on PME developments since the Skelton Panel’s review, investigated whether a uniform cost accounting system existed, among other things. The congressional report found that DOD did not have a uniform cost accounting method for PME schools, and that it had not provided cost data to support useful comparisons among PME schools. The report included a recommendation for DOD to report its PME funding to Congress using a standardized accounting method for cost per student at each of the PME institutions, as recommended by the Skelton Panel in 1989. According to DASD(FE&T) and Joint Staff officials, the department has not collected or reported PME program cost information to Congress as the 1989 Skelton Report and the 2010 congressional report both recommended. Without complete and uniform budget request data, USD(Comptroller)’s ability to monitor the military services’ PME programs, identify program trends within the Marine Corps and among the other military services’ PME programs, and formulate meaningful inter-service comparisons is limited. DOD relies on PME to prepare its military personnel for the intellectual demands of complex contingences and major conflicts that typically involve more than a single military service. While all the military services’ intermediate- and senior-level PME programs have met or partially met the accreditation requirements established by civilian accreditation bodies and the Chairman to award master’s degrees and JPME credit, respectively, not all service programs have met the seminar student mix requirement. The Navy, for example, has not provided the requisite representation of officers in Army and Air Force intermediate-level seminars during the 2016 – 2018 academic years. Requiring DOD to determine whether the requisite number of Navy officers can be assigned to the military department’s JPME programs and to develop policy to mitigate student mix shortfalls would address persistent student mix imbalances and align with the joint acculturation goal of JPME. OUSD(P&R)’s draft DOD instruction, expected to be finalized in February 2020, will be the prevailing policy document for PME and could improve OSD’s oversight of the military services’ PME and JPME programs. However, OUSD(P&R)’s ability to assess the effectiveness of the military services’ PME programs is limited by the absence of a department-wide mission statement that explains the purpose and goals of PME that aligns with the proposed performance measures in the draft instruction; the absence of a requirement for the military services to track program costs as a performance measure; and the absence of a requirement for the military services to report data on their PME and JPME programs—such as their annual reviews of PME programs. Addressing these limitations would better position OUSD(P&R) to oversee and assess the effectiveness of the military services’ PME and JPME programs. Finally, USD(Comptroller)’s ability to monitor the military services’ PME programs is limited because the services’ budget request data are incomplete and lack uniformity. Although the military services are required to submit separate budget request data exhibits for most PME institutions, the Financial Management Regulation does not require the Marine Corps to submit an annual budget request data exhibit for its senior-level PME program. Moreover, the data the military services include in their annual budget requests vary because the Financial Management Regulation does not specify how to account for costs. Requiring the Marine Corps to report budget request data on its senior-level PME program annually, and specifying how to account for costs in the exhibits would enhance the USD(Comptroller)’s ability to monitor the military services’ PME programs and also enhance Congress’s ability to identify trends among these programs. We are making a total of seven recommendations to the Secretary of Defense. Specifically: The Secretary of Defense should ensure that the Under Secretary of Defense for Personnel and Readiness, in coordination with the Chairman of the Joint Chiefs of Staff and the Secretary of the Navy, determine whether it can assign the required number of Navy officers to the other military departments’ JPME programs, consistent with Chairman of the Joint Chiefs of Staff guidance. (Recommendation 1) The Secretary of Defense should ensure that the Chairman of the Joint Chiefs of Staff, in coordination with the Under Secretary of Defense for Personnel and Readiness and the military services, develop policy concerning actions, if any, the military services can take to mitigate JPME seminar student mix shortfalls and still meet the intent of the OPMEP’s joint acculturation requirement. (Recommendation 2) The Secretary of Defense should ensure that the Under Secretary of Defense for Personnel and Readiness, in coordination with the Chairman of the Chiefs of Staff, develop and issue a department-wide mission statement for PME that will explain the program’s purpose and goals, and serve as a basis for performance measures. (Recommendation 3) The Secretary of Defense should ensure that the Under Secretary of Defense for Personnel and Readiness, in coordination with the Chairman of the Joint Chiefs of Staff, issue and implement performance measures—to include the tracking of costs—that align with the department-wide mission statement for PME. (Recommendation 4) The Secretary of Defense should ensure that the Under Secretary of Defense for Personnel and Readiness, in coordination with the Chairman of the Joint Chiefs of Staff, require the military services to periodically report information to its office about the military services’ PME and JPME programs—such as results of program reviews. (Recommendation 5) The Secretary of Defense should ensure that the Under Secretary of Defense(Comptroller) updates the DOD Financial Management Regulation to require the Marine Corps to include a budget request data exhibit for the Marine Corps War College in support of DOD’s annual budget request. (Recommendation 6) The Secretary of Defense should ensure that the Under Secretary of Defense (Comptroller), in coordination with the military services and the Chairman of the Joint Chiefs of Staff, issue guidance to standardize the cost data that the military services should include in their annual PME budget request data submissions. (Recommendation 7) We provided a draft of this product to DOD for comment. In its comments, reproduced in Appendix II, DOD concurred with all of our recommendations and stated that it will be implementing our recommendations by issuing policy, among other actions. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense for Personnel and Readiness, the Chairman of the Joint Chiefs of Staff, and the Secretaries of the Army, Navy, and Air Force. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. The Joint Special Operations University (JSOU) was established in September 2000 and is located at MacDill Air Force Base, Florida. The mission of JSOU is to prepare special operations forces to shape the future strategic environment by providing specialized Joint Professional Military Education (JPME); developing applicable undergraduate- and postgraduate-level equivalent curriculum; and fostering special operations research, analysis, and outreach in support of Special Operations Command objectives. JSOU staff and faculty include active duty, active reserve, and temporary duty reserve military personnel; government civilians; civilian contractors; private consultants; and guest lecturers and speakers. JSOU’s active duty military personnel are assigned to the university by Special Operations Command and the military services. JSOU’s professional military education vision is to prepare warfighters to solve ambiguous, complex problems across the spectrum of conflict by providing dynamic and adaptive professional education opportunities. In August 2015, the Accrediting Council for Continuing Education and Training accredited JSOU through December 2019. As of January 2020, officials stated that they are currently undergoing reaccreditation and expect reaffirmation notification by the end of February 2020. While JSOU offers a number of courses, seminars, and programs, officials from JSOU and the Office of the Assistant Secretary of Defense for Special Operations/Low-Intensity Conflict stated the university has no near-term plans to award master’s degrees; therefore, no additional civilian accreditation is necessary. JSOU officials said that they are contemplating offering senior-level JPME in the future, but stated that such an endeavor would take approximately at least 10 years to accomplish. Consistent with its mission of preparing special operations forces to shape the future strategic environment, JSOU laid out the following seven goals in its 2019 academic guidance: 1. Continue to refine target audiences in all courses, assuring the right curricula is provided to the right student at the right time. 2. Implement a title 10, U.S. Code, civilian faculty hiring process that leverages the DOD professional military education community, fully supports the JSOU vision, and retains control to rapidly hire faculty with expertise in required disciplines. 3. Establish and complete a comprehensive building improvement plan that provides a quality learning environment conducive to educational excellence and student success. 4. Establish and complete a comprehensive education technology plan that brings all classrooms and auditoriums up to planned capability inherent in a state-of-the-art learning institution. 5. Facilitate the Technology Review Committee to define and develop the JSOU advanced classroom concept, capable of a wide variety of innovative teaching methodologies. 6. Develop and sustain academic programs in the emerging mission areas of artificial intelligence/machine learning, countering weapons of mass destruction, cyberspace, sensitive activities, and joint unconventional warfare that directly support special operations. 7. Develop highly effective academic instructors and distinguished experts in their individual fields of knowledge. Remain sensitive to individual needs and career development as JSOU embarks on new hiring processes and classroom innovations. According to the JSOU Fact Book for 2018, the newly authorized title 10, U.S. Code, civilian faculty hiring authorities will allow JSOU faculty to attain new heights of excellence with expertise not normally found within the military or civil service communities. The handbook states that the title 10, U.S. Code, faculty hiring authority will have a major impact on shaping JSOU’s curriculum, and will directly add to special operations forces’ readiness and capability. Brenda S. Farrell, (202) 512 -3604 or farrellb@gao.gov. In addition to the contact named above, Marc Schwartz (Assistant Director), Norris “Traye” Smith (Analyst in Charge), Rebecca Guerrero, Edward Malone, Stephanie Moriarty, Patricia Powell, Carter Stevens, and Lillian M. Yob made significant contributions to this report. Higher Education: Expert Views of U.S. Accreditation. GAO-18-5. (Washington, D.C.: December. 22, 2017). Higher Education: Education Should Strengthen Oversight of Schools and Accreditors. GAO-15-59. Washington, D.C.: December 22, 2014. Joint Professional Military Education: Opportunities Exist for Greater Oversight and Coordination of Associated Research Institutions. GAO-14-216. Washington, D.C.: March 10, 2014. Joint Military Education: Actions Needed to Implement DOD Recommendations for Enhancing Leadership Development. GAO-14-29. Washington, D.C.: October 23, 2013.", "summary": "DOD relies on PME and JPME to prepare its military personnel throughout their careers for the intellectual demands of complex contingences and major conflicts that typically involve more than a single military service. However, according to DOD's summary of the 2018 National Defense Strategy, PME “has stagnated, focused more on the accomplishment of mandatory credit at the expense of lethality and ingenuity.” The Conference Report accompanying the John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to evaluate DOD PME and JPME institutions. This report examines the extent to which (1) the military services' PME programs have met civilian and JPME accreditation requirements, (2) OSD has assessed the effectiveness of the military services' PME programs, and (3) USD (Comptroller) has monitored the military services' PME program budget data. GAO analyzed applicable laws and policy, analyzed accreditation and budget information, and interviewed officials from the military services' intermediate- and senior-level resident PME programs. All of the military services' intermediate- and senior-level officer Professional Military Education (PME) programs have met civilian and met or partially met Joint PME (JPME) accreditation requirements. However, not all of the military services' PME programs met the JPME seminar student mix requirement of at least one student from the nonhost military department. For example, the Army's intermediate-level PME program did not meet its Sea Service (i.e., Navy, Marine Corps, and, in certain instances, Coast Guard) requirement (see table). GAO's analysis found that the Navy could have assigned officers to Air Force and Army programs while not harming participation in its own seminars. Without taking steps to improve Sea Service participation, students lose opportunities to interact with students from other military departments, which officials have identified as critical to joint acculturation. The Office of the Secretary of Defense (OSD) has taken steps to improve its oversight of the military services' PME programs, but is limited in its ability to assess their effectiveness. Department of Defense (DOD) guidance states that performance measurement is a means of evaluating efficiency, effectiveness, and results and that a balanced performance measurement scorecard includes nonfinancial and financial measures focusing on quality, cycle time, and costs. While OSD is in the process of developing some performance measures, it is not planning to require the military services to track program costs. Implementing its planned measures and establishing costs as a performance measure will better position OSD to assess the effectiveness of PME programs. The Under Secretary of Defense (USD) (Comptroller's) ability to monitor the military services' PME programs is limited by incomplete and inconsistent reporting of service budget request data. DOD guidance does not require the Marine Corps to submit an annual budget request data exhibit for its senior-level PME program and existing guidance for programs that are reported does not specify how to uniformly account for costs. Without complete and uniform budget request data, USD(Comptroller) is challenged in monitoring these programs. GAO is making seven recommendations, including that DOD take steps to determine its ability to assign Navy officers to PME programs of other services, implement performance measures–including tracking of costs, and issue guidance for service reporting of PME budget request data. DOD concurred with all of GAO's recommendations.", "document_type": "gao"}
{"report": "Section 501 of the Internal Revenue Code provides for tax-exempt status of certain corporations, trusts, and other organizations. This status allows qualifying organizations to claim exemption from federal income taxes. Subsection (c) of section 501 recognizes 28 categories of tax- exempt organizations, ranging from cemetery companies to multiemployer pension plan trusts. Section 501(c)(3), the section that recognizes charitable organizations, applied to approximately 1.3 million organizations in fiscal year 2017. These groups represent the largest number of 501(c) organizations. Federal tax law permits individual taxpayers and organizations to reduce their tax liability by deducting contributions to charitable organizations on their income tax returns. Individual taxpayers may deduct the amount of a contribution to charitable organizations from their gross income if they itemize their deductions. Charitable organizations provide many types of assistance, such as services for the aging or food and shelter for those in need. Taxpayers may support these activities by making contributions in the form of financial donations or in-kind gifts to qualified organizations. Federal law allows taxpayers to deduct charitable contributions from their adjusted gross income (AGI). This policy has been in place since 1917. An individual taxpayer may deduct up to 60 percent of his or her AGI for cash contributions, with 20 percent to 30 percent limits applying in some cases. A corporation may claim a limited deduction for charitable contributions made in cash or other property up to 10 percent of its taxable income for the year. An entity seeking tax-exempt status under 501(c)(3) from IRS must submit either a completed Form 1023, Application for Recognition of Exemption Under Section 501(c)(3) of the Internal Revenue Code, along with organizing documents, or a completed Form 1023-EZ. Both Form 1023 and Form 1023-EZ require the entity seeking recognition of its tax- exempt status to provide information regarding its charitable purpose, as well as certain financial data. IRS employees then review the forms to determine the entity’s eligibility for tax-exemption status. Most tax-exempt charitable entities are required to file an annual information return from the Form 990 series. Certain small entities with gross receipts that are normally $50,000 or less may file Form 990-N Electronic Notice providing abbreviated information. Although the entity is filing its information return as a tax-exempt organization, the entity must pay employment taxes and taxes on unrelated business income, if applicable. IRS provides programs and products to help the entity understand specific issues related to its tax responsibilities. IRS personnel can audit an organization’s or individual’s submitted tax returns and financial information to verify that the reported tax is correct. IRS personnel audited 933,785 individual income tax returns in fiscal year 2017, according to IRS data. This was 0.6 percent of individual returns filed in calendar year 2016. From fiscal year 2006 to fiscal year 2017, the largest number of individual returns IRS audited was 1,564,690 in fiscal year 2010. There was a decrease in audits of individual tax returns after fiscal year 2011, which occurred about the same time that IRS’s budget declined by about $2.1 billion (15.7 percent) from fiscal years 2011 through 2018, after adjusting for inflation. Concurrent with IRS’s declining resources were increasing responsibilities, such as implementing aspects of the Foreign Account Tax Compliance Act and the Patient Protection and Affordable Care Act. We reported in 2014 that budget cuts had resulted in a significant staffing decline and uneven performance at IRS. In March 2019, we reported that IRS was in the early stages of defining and addressing its workforce needs, but IRS officials stated that there was room for improvement in implementing its workforce plans, and that it was working on a corrective action plan that would address deficiencies noted in our report. The operating divisions that, along with conducting audits, carry out service and enforcement, and that deal most often with abusive tax schemes or tax-exempt entities are TE/GE, SB/SE, and LB&I. These divisions interact with taxpayers and entities that file tax returns. In particular, each of the three divisions may audit taxpayers or entities to determine whether information filed was reported accurately. IRS has set one of its cross-divisional objectives as identifying “new types of tax transactions or promotions that are either abusive or potentially abusive requiring different levels of coordination and varying strategies.” Another of TE/GE’s audit objectives is to “promote the highest degree of voluntary compliance with the statutes governing qualification of plans and exemption of certain types of organizations from tax and to determine the extent of compliance and the causes of noncompliance with the tax laws by plans and organizations.” TE/GE accomplishes this objective by auditing charitable organizations’ compliance with the tax code through its Exempt Organizations unit. In addition to this function, Exempt Organizations also reviews organizations’ tax-exempt status applications and makes tax-exempt status determinations. It also coordinates with other state and federal agencies. Additionally, it audits entities to identify and address noncompliance, where it may propose tax assessments or changes to the tax-exempt status of the audited entity. TE/GE uses various enforcement processes, such as referrals from the public and other parts of IRS and data-driven approaches, to select tax- exempt organization for possible audits. IRS projects that Exempt Organizations will receive approximately 1.6 million filings from tax- exempt and government entities in fiscal year 2019, primarily Form 990 series information returns. SB/SE mainly oversees small businesses and self-employed taxpayers and all other businesses with assets of less than $10 million. Examples of the types of businesses that SB/SE covers include small-business start-ups, small businesses with or without employees, taxpayers with rental properties, taxpayers with farming businesses, and individuals investing in businesses such as partnerships. Overall, IRS projects that SB/SE will receive approximately 59.4 million tax returns in fiscal year 2019. The Lead Development Center, an office within SB/SE, receives referrals from and facilitates communication between SB/SE and TE/GE on the subject of abusive tax schemes. LB&I oversees tax compliance of large partnerships, S Corporations, and C corporations with assets of $10 million or more, as well as individuals with high wealth (those with tens of millions of dollars in assets or earnings) or international tax issues. IRS projects that LB&I will receive approximately 400,000 corporate tax-return filings in fiscal year 2019. LB&I has developed a compliance strategy to identify potential issues that arise during audits of tax returns. LB&I also oversees the processing of reportable transaction disclosure filings by those involved in reportable transactions. A transaction includes all the factual elements relevant to the expected tax treatment of any investment, entity, plan, or arrangement. It also includes any series of steps carried out as part of a plan. Transactions become “reportable” (meaning a taxpayer must report it to IRS) when they fall under one or more of the following categories: listed transactions, confidential transactions, contractual protection transactions, loss transactions, and transactions of interest. A listed transaction is any transaction that IRS has identified as an abusive tax avoidance transaction and has identified in published guidance as a listed transaction. Taxpayers that have engaged in transactions that have tax consequences or tax strategies described in published IRS guidance are required by law to disclose the transaction to IRS. The fact that a transaction must be reported does not mean IRS will disallow the tax benefit, but IRS uses the reports to assess compliance. Appendix IV discusses reportable transaction types in greater detail. Taxpayers are required to disclose all types of reportable transactions on Form 8886, Reportable Transaction Disclosure Statement. Similarly, advisers helping taxpayers conduct reportable transactions are required to file Form 8918, Material Advisor Disclosure Statement. Tax-exempt entities are required to file Form 8886-T, Disclosure by Tax- Exempt Entity Regarding Prohibited Tax Shelter Transaction, when the entity is a party to a listed, confidential, or contractual protection transaction, and the entity knows the identify of any other party in the transaction. Tax-exempt entities that are party to a listed or confidential transaction may be subject to an excise tax of 100 percent of the income from the transaction. Transactions that require the filing of form 8886-T constitute a different, smaller range of activity than transactions requiring the filing of Form 8886. The Office of Tax Shelter Analysis, a unit within LB&I, supports LB&I’s work by coordinating its tax shelter planning and operations. This office also analyzes information collected from disclosure forms. According to IRS policy, if the Office of Tax Shelter Analysis determines a formal investigation is warranted, it presents the information to the LB&I Technical Tax Shelter Promoter Committee, an office within LB&I that has sole authority to approve any proposed investigations. Taxpayers seeking to reduce their tax liability through charitable donations may participate in legal tax planning strategies that allow them to maximize their deductions while giving to charitable organizations. In contrast to these legal tax planning strategies involving charitable donations, abusive tax schemes occur when taxpayers conduct transactions that are not supported by established law to improperly claim tax benefits, or that have no economic significance or business purpose other than the avoidance of tax, among other factors. IRS has long recognized that some charitable donors and tax-exempt organizations have engaged in abusive tax schemes. One such scheme can consist of a donor grossly overvaluing a charitable contribution to obtain a larger deduction on his or her filed tax returns. Another abusive tax scheme can entail a tax-exempt organization providing benefits to a private shareholder or individual. As we previously have reported, the abusive transactions that comprise abusive tax schemes have been a long-standing, ever-changing, and often hidden problem for IRS. The following three examples illustrate various ways that an entity’s tax- exempt status can be used in transactions that are not supported by law or are inconsistent with the law’s intent, and how otherwise legitimate tax- exempt activity can be exploited improperly. A conservation easement is a legal agreement that grants an organization the right to restrict the development and use of property for conservation purposes with the intent of preserving the land or buildings. If statutory requirements are met, taxpayers may donate an easement to a qualified organization and receive a charitable income tax deduction for the appraised value of the easement. A conservation easement becomes “syndicated” if a person or company promoting the easement (a promoter) offers multiple investors in a partnership or pass-through entity the opportunity to claim charitable deductions based on the value of the easement in return for cash. The Brookings Institution estimated that investments in syndicated conservation easements totaled $623 million in 2016, an increase of 29 percent from $484 million in 2015. It further estimated that because tax deductions from syndicated conservation easement contributions generate a benefit greater than the value of the investments themselves, the tax deductions resulted in federal tax revenue loss between $1 billion and $1.9 billion in 2015 and between $1.3 billion and $2.4 billion in 2016. According to IRS, in a syndicated conservation easement, promoters purchase land and convey ownership to a pass-through entity, such as a partnership. The promoters offer interests in the pass-through entity to prospective investors who are then able to deduct their share of the value of the easement as a charitable contribution. In its guidance, IRS said the conservation easement becomes noncompliant if, for example, the promoters obtain an appraisal that purports to be a qualified appraisal, but that greatly inflates the value of the conservation easement based on unreasonable assumptions about the development potential of the real property. Because the promoters inflate the value of the property, the investors may benefit by claiming a charitable deduction on their tax returns that exceeds their initial investment. Figure 1 shows the steps in the formation of a syndicated conservation easement and the point at which the easement becomes noncompliant when promoters obtain an inflated value for the easement. IRS has indicated its concern about the potential for abuse of conservation easements, whether syndicated or otherwise, when used in ways not supported by the law. In December 2016, the Department of the Treasury (Treasury) and IRS issued Notice 2017-10 designating syndicated conservation easements as listed transactions. This notice provides that certain syndicated conservation easements promoted with a return on investment of at least 250 percent will be identified as listed transactions. It also provided details on how Treasury and IRS view these transactions as forms of abuse. Although promoters who abuse syndicated conservation easements exploit tax-exempt entities, the law does not treat the tax-exempt entity as a participant, meaning that even when a promoter is found to use a syndicated easement in a noncompliant manner, the tax-exempt entity associated with the scheme may still be considered compliant. In addition to the potential for overvaluation of easements, Treasury and IRS considered that syndicated conservation easements may become problematic because of the potential they have to involve transactions that violate the economic substance doctrine. Because of its concerns, IRS has identified taxpayer abuse of conservation easements as a risk area for noncompliance. Syndicated easements also illustrate how noncompliance can cross the areas of responsibility of IRS’s audit divisions. In this case, the beneficiary of the scheme may be a small-business taxpayer (SB/SE’s responsibility) or a corporation (LB&I’s responsibility), even though the scheme hinges on an inflated appraisal and being able to donate to the tax-exempt recipient (TE/GE’s responsibility). A donor-advised fund is a fund or account held by a charity that receives contributions from donors who may advise, but not control, how the organization uses the money. The Pension Protection Act of 2006 defined donor-advised funds in the Internal Revenue Code and subjected the funds to new requirements. Because donor-advised fund accounts are operated by charities, contributions to these funds are deductible at a higher percentage of adjusted gross income (generally 50 percent or 60 percent for cash contributions) than donations to private foundations (generally 30 percent). Some donors may use the donor-advised funds in ways that IRS considers improper. For example, prior to tax-law changes in 2006, IRS said that abusive donor-advised funds are those that appear to be established to generate questionable charitable deductions, and provide impermissible economic benefits to donors and their families (including tax-sheltered investment income for the donors). Figure 2 illustrates how donor-advised fund accounts operate and highlights where in the process the parties involved could abuse the funds or raise policy concerns about how donor advised funds have been used. Donor-advised funds have grown in various measures in recent years, according to data compiled by the National Philanthropic Trust. For example, it reports that from 2013 to 2017, the total grants made by donor-advised funds grew from $9.83 billion to $19.08 billion, and contributions grew from $17.24 billion to $29.23 billion. Total assets held in donor-advised funds increased from $57.1 billion to $110.01 billion as well, according to the organization’s study. In 2017, about 463,000 donor- advised funds existed in the United States, with an estimated $110 billion in assets, according to the National Philanthropic Trust. Some of the largest of these funds in terms of assets are sponsored by financial institutions, religious groups, and community foundations, while others are independent, according to our review of selected donor-advised funds’ sponsoring organizations’ websites and data from the National Philanthropic Trust. Patient assistance programs help patients afflicted with certain medical ailments obtain financial assistance for medical care or free drug products and these programs may qualify for tax-exempt status. Pharmaceutical companies may establish their own patient assistance programs or make monetary donations to independent charities’ patient assistance programs. In addition to financial support, pharmaceutical companies may donate medication (through in-kind product donations) to patient assistance programs. Donations such as these allow pharmaceutical companies to claim a limited tax deduction for charitable contributions. If they claim deductions, the deductions may be up to 10 percent of the corporations’ taxable income when donating to charities. The possibility of donors receiving private benefits in excess of the charitable deduction creates potential risks to participating pharmaceutical companies and compliance challenges for IRS, according federal regulators. For example, because independent charity patient assistance programs may be 501(c)(3) tax-exempt organizations, pharmaceutical manufacturers’ profits generated from sales of their products to individuals receiving help from patient assistance programs that they donate to may raise issues of inurement. Figure 3 summarizes how a hypothetical patient assistance program works and highlights points in the process where potential abuse of the program may occur. The federal government has investigated cases of potential private benefit by pharmaceutical companies and patient assistance programs. For example, IRS filed a court summons in May 2017 in an ongoing investigation of a patient assistance program over concerns that it spent the majority of its donations on copayment support that went to patients who were prescribed medication from companies that had donated money to the patient assistance program. As shown in the tax scheme examples previously discussed, abusive schemes with tax-exempt entities can involve the tax-exempt entity directly or leverage an entity’s tax-exempt status indirectly to reduce taxes. Consequently, the characteristics of audits involving abusive tax schemes, such as which IRS operating division is responsible for the audit, will differ according to the type of scheme. In addition, IRS generally presents information about abusive tax schemes under a category it calls abusive tax avoidance transactions. The abusive tax schemes we have been discussing in this report are a subset of abusive tax avoidance transactions in which the transaction or arrangement involves multiple types of entities. IRS data do not allow us to identify separately the transactions involving multiple entities. The discussion that follows describes trends under the assumption that over time abusive transactions involving multiple entities would closely track total abusive transactions. TE/GE audited 2,294 tax-exempt entities with what IRS identified as abusive tax avoidance transactions in the 10-year period from fiscal year 2008 through 2017. As shown in figure 4, the number of abusive- transaction audits fell from a high of 886 in fiscal year 2009 to 10 or less in fiscal year 2017. This decline represented at least a 98.9 percent decrease in audits performed by TE/GE (see appendix V, table 6). The decline in abusive-transaction audits generally corresponds with the overall decrease in audit activity by IRS over recent years (see appendix V, tables 2, 3, and 4). During the same 10-year period, TE/GE assessed a total tax increase of $107 million based on its audits of tax-exempt entities and the average tax increase per audit was $46,804. The amount assessed for the tax increase declined from 45.3 million in 2008 to 1.2 million in the merged years of 2016 and 2017. The effectiveness and efficiency of the audit process may be reflected in the no-change rate and staff days associated with the audits. The no- change rate—the percentage of audits that results in no tax change—was 13.9-percent (see appendix V, table 11). IRS uses this ratio as an indicator of how effectively IRS identifies noncompliant taxpayers (a lower no-change rate on its audits is consistent with more effective audit selection methods). The lower rate may also reflect higher economic efficiency because less IRS and taxpayer time and other resources are used for auditing compliant returns. On average, TE/GE spent 70 hours per audit of tax-exempt entities from fiscal year 2008 through 2017 (see appendix V, table 9). Audits involving abusive schemes where taxpayers leverage an entity’s tax exempt status—but the tax-exempt entities are not the subject of the audit—are the responsibility of SB/SE and LB&I. To determine the minimum number of audits these divisions conducted on abusive schemes involving tax-exempt entities, we used IRS project codes that IRS agreed were relevant. For these project codes, SB/SE and LB&I conducted 4,207 audits over the 10-year period. The numbers of audits generally decreased over the period except for increases in 2012, 2015, and 2017 for LB&I audits and increases in fiscal year 2015 and 2017 for SB/SE audits. Combined SB/SE and LB&I audits fell from 1,176 in fiscal year 2008 to 99 in fiscal year 2017, a 91.6 percent decrease (see appendix V, table 13). SB/SE and LB&I recommended about $8.3 billion in tax changes over the 10-year period. As shown in figure 5, the average recommended amount was larger for LB&I, but tended to fluctuate more than the SB/SE amounts. The average tax change amount per audit over the 10-year period recommended by SB/SE was $89,399. The average amount recommended by LB&I was $8.6 million. Figure 5 also shows how both divisions had a surge in recommended tax amount changes for 2017 compared to prior years. SB/SE’s recommended changes increased from $270,131 in fiscal year 2016 to $127 million in fiscal year 2017. LB&I recommended changes increased from $299 million in 2016 to $555 million in 2017. IRS officials could not provide an explanation for the surge in 2017 (see appendix V, table 14). Again, the divisions’ resource use may be reflected in staff days and the no-change rate. SB/SE and LB&I combined spent 218 hours, on average, per audit for the audits involving tax-exempt entities identified by project codes (see appendix V, table 15). The no-change rate for SB/SE audits we examined involving tax-exempt entities identified by project code was 10.9 percent. LB&I audits involving tax-exempt entities had a no-change rate of 15.5 percent (see appendix V, table 16). Numbers of audits of all types of abusive transactions showed a pattern of decline similar to audits involving tax exempt entities. SB/SE and LB&I conducted a total of 155,467 audits involving all types of abusive transactions from fiscal year 2008 to fiscal year 2017. As shown in figure 6, the total number of these audits conducted by each of the operating divisions fell in most years. Abusive transaction audits conducted by SB/SE and LB&I fell from 26,519 in fiscal year 2008 to a low of 4,248 in fiscal year 2017, an 84 percent decrease in audits closed during this period (see appendix V, table 5). Audits involving tax-exempt entities resulted in higher average tax changes than audits for the total of all abusive transactions. Combined, SB/SE and LB&I recommended a total of $39 billion in tax changes for the total of all for abusive-transaction audits. As shown in figure 7, SB/SE recommended tax amount changes that averaged $40,834 per audit and LB&I recommended tax amount changes that averaged $3 million per audit. The recommended tax change per abusive-transaction audit was larger for audits involving tax-exempt entities than for the total of all abusive-transaction audits in both operating divisions which were (as described above) $89,399 for SB/SE and $8.6 million for LB&I. The total recommended tax amount change for SB/SE decreased from $1.4 billion to $339 million, a 75 percent decrease over the period. For LB&I, the recommended tax amount change decreased from $7.5 billion to $866 million, an 89 percent decrease (see appendix V, table 7). We estimated audits involving tax-exempt entities identified on the basis of project codes led to SB/SE and LB&I recommending about $8.3 billion in tax changes over the 10-year period. The no-change rate for all SB/SE abusive transaction audits over the period was 8.8 percent. The no-change rate for all LB&I abusive- transaction audits was 14 percent (see appendix V, table 10). Combined, SB/SE and LB&I spent a total of 6.6 million staff hours for the total of all abusive transaction audits from fiscal year 2008 to 2017, spending, on average, 42 hours per audit for all abusive-transaction audits (see appendix V, table 8). As described above, SB/SE and LB&I spent more in resources, 218 hours, on average, per audit of tax-exempt schemes, than the average for the total of all abusive-transaction audits. The majority (88 percent) of taxpayer audits involving tax-exempt entities identified on the basis of project codes for both SB/SE and LB&I had an Adjusted Gross Income (AGI) of more than $50,000, with about 40 percent of the audits involving the taxpayers with AGI falling between $100,000 and $500,000. The SB/SE audits had an average AGI of $1.2 million and median AGI value between $200,000 and $500,000. LB&I audits had an average AGI of $6.2 million and a median AGI value between $1.0 million and $1.5 million. The majority of business taxpayers with abusive-transaction audits involving tax-exempts (about 70 percent) reported zero gross receipts (see appendix V, tables 17 and 18). While the audit data examined above show the noncompliance IRS has found regarding abusive schemes with tax-exempt entities, information about the taxpayers involved in the transactions can also be derived from the IRS disclosure forms. Most of the taxpayers identified partnerships as the entities involved in the listed transactions that they reported. Of the taxpayer disclosures identifying a tax-exempt entity on Form 8886, 97.8 percent identified the type of reportable transaction as a listed transaction and 95.5 percent listed a partnership for type of entity involved in the transaction. Further, 98.1 percent of taxpayers claimed a deduction from their AGI as the benefit generated by the transaction and 5 percent claimed an ordinary loss as the tax benefit. The different disclosure reports that IRS receives from tax-exempt entities, taxpayers, and tax advisors contain data that identify the potential involvement of tax-exempt entities with reportable transactions. However, there are differences in the legal filing requirements, the types of information supplied, and the number of disclosure forms filed. Few tax-exempt entities directly disclose their involvement in prohibited transactions to IRS. Regulations require that certain tax-exempt entities disclose information on a prohibited tax shelter transaction to which the entity is a party. For calendar years 2004 through 2016, IRS received 71 Form 8886-T disclosures from tax-exempt entities that were a party to a prohibited transaction. Moreover, the actual number of filers making disclosures was smaller, only 33, because some submitted multiple forms during the period. Many more tax-exempt entities were identified by taxpayers filing the Form 8886, which requires a different, broader range of transactions to be reported than the Form 8886-T. For calendar years 2000 through 2017, IRS received more than 979,900 Form 8886 disclosure reports from taxpayers. Of that number the taxpayer identified a tax-exempt entity as part of the reportable transaction on 32,847 disclosures or 3.4 percent of all Form 8886 reports. A smaller number was identified by tax advisors on Form 8918. For calendar years 2007 through 2018, out of the 16,477 Form 8918 disclosure statements received from tax advisors, 155 submissions identified a tax-exempt entity as part of a reportable transaction. While detail about the transactions themselves—when they appear in the form narratives—is not readily available from the Form 8886 disclosure databases. IRS’s Research, Applied Analytics and Statistics Division has created an analytic tool for analyzing narrative information that it has tested on the Form 8886. When we performed a test analysis using this tool on the narrative fields on the Form 8886, we identified keywords that may help isolate tax-exempt organization involvement in potentially abusive schemes and ultimately help select returns for more detailed review. This more detailed review is required because transactions reported on the Form 8886 are not necessarily noncompliant. For our test analysis, we selected certain terms related to known abusive tax schemes involving tax-exempt entities such as “conservation easement” or related to the tax-exempt sector such as “charitable organization” and counted the number of times the terms appeared in the narrative field of 26,632 Form 8886 disclosures made in fiscal year 2017. For example, the term “conservation easement” occurred in the narrative field of 6,767 disclosure forms and the term charitable organization occurred on 17 disclosure forms. Through further searching on terms that might relate to charitable organizations, such as “charity,” “sports,” “children,” “animals,” “foundation,” and “scientific,” we identified 211 occurrences. IRS is not undertaking this type of analysis of taxpayer disclosures, which would expand its ability to identify tax-exempt entities and evaluate their potential involvement with reportable transactions, as discussed later in this report. IRS operates various programs to identify abusive tax schemes involving tax-exempt entities. Not all of these programs exclusively address abusive tax schemes with tax-exempt entities but nevertheless can provide relevant information on that issue. For example, the Office of Tax Shelter Analysis processes disclosures of reportable transactions, including those related to tax-exempt entities, and the Lead Development Center may collect information about abusive schemes related to tax- exempt entities as part of its role in dealing with abusive tax transactions in general. As figure 8 illustrates, several of these programs in practice are linked by the Service-wide Compliance Strategy Executive Steering Committee. This committee is responsible for collecting input from the operating divisions (TE/GE, SB/SE, and LB&I), as well as other parts of IRS, about abusive tax schemes that cross divisional responsibilities, including schemes involving tax-exempt entities. The Executive Steering Committee also may make decisions about how to address abusive tax schemes that cross the operating divisions’ responsibility. IRS officials said that the operating divisions are individually responsible for monitoring the committee’s performance. Therefore, the committee’s decisions depend on what information the operating divisions provide. As figure 8 also shows, the operating divisions pass information about abusive schemes among themselves through referrals, making clear communication among the operating divisions critical for IRS in identifying abusive tax schemes. An IRS office that more directly addresses potential abusive schemes with tax-exempt organizations is TE/GE’s Compliance Planning and Classification office (CP&C). This office has several responsibilities relating to identifying abusive tax schemes and communicating with other parts of IRS, as well as coordinating with other operating divisions on potential noncompliance. For example, CP&C is responsible for reviewing emerging abusive tax schemes, conducting research, and reviewing suggestions from a computer portal through which staff can raise potential issues about compliance. The portal also serves as the foundation to TE/GE’s compliance issue identification process. We found that IRS maintains a variety of programs to identify tax schemes involving tax exempt entities agency-wide, and these programs together fully met seven of our 10 criteria. Appendix I contains more information about the criteria we used in our analysis and a table that summarizes the results of our analysis. One criterion that IRS fully met was identifying areas of authority. All of the programs we reviewed had documentation showing the responsibilities the program was to fulfill and the roles it was to perform. IRS’s programs also fully met the criterion for ensuring competency by having documented procedures for training to enhance staff’s responsibilities across the programs we reviewed and met the communication criterion by, for example, having coordination meetings among officials representing the different operating divisions. In addition, IRS met the criterion for conducting monitoring activities by, for example, having inventory reports on TE/GE’s issue submission portal and maintaining a monitoring group over TE/GE’s audit plans. Finally, IRS met all three of our fraud-related criteria with programs or procedures that specifically identify fraud, such as TE/GE’s Fraud Investigation Unit, or that assist auditors in identifying fraud on returns, such as IRS’s Fraud Handbook. Reviewing whether auditors assessed fraud risk is also part of TE/GE’s quality review system. In the following sections, we discuss how IRS did not meet the other three internal control criteria. A relatively low number of tax exempt entities filing Forms 8886-T combined with our analysis of audit data raises questions about whether tax-exempt entities are filing these forms as often as they should. As we discussed above, tax-exempt entities filed only 71 Forms 8886-T over a 12-year period from fiscal year 2004 through 2016, where they listed prohibited transactions. At about the same time, taxpayers in general filed thousands of Forms 8886 annually where they identified tax-exempt entities as part of their reportable transactions. In addition, when we compared Form 8886 filings that identified tax- exempt entities as part of the reportable transaction with SB/SE and LB&I audit data, again for the same time period, we found 432 closed cases with tax changes. Although we did not determine whether the subject of these audits was the abusive scheme involving a tax-exempt entity, the result of 432 closed audit cases suggests that tax-exempt entities may be part of more prohibited transactions than those reported on the 71 Form 8886-T filed during the period. The audit cases identified in SB/SE and LB&I data resulted in about $1.9 billion in tax changes. The average per audit tax change recommended by SB/SE was $65,143 and by LB&I was $19 million. A similar analysis could be conducted comparing audit results with data from Form 8918, which is filed by tax advisors. IRS officials said the disparity between the number of Form 8886 filings and the small number of 8886-T filings has not raised concerns because the legal requirements for tax-exempt entities filing Form 8886-T are narrower than the requirements taxpayers must follow to file Form 8886, as we discussed earlier. However, IRS has not undertaken a risk assessment to test whether this explanation—that the lower number of filings should be expected because the filing requirement is narrower—is valid, which is inconsistent with the internal control standards for risk assessment. The Office of Tax Shelter Analysis sends Form 8886-T filings it receives to TE/GE, and the Compliance Planning and Classification office reviews these filings, but no documented process exists to determine whether all tax-exempt entities that should file Form 8886-T were filing the form as required. In addition, IRS provided us with no studies investigating the causes and consequences of such a small number of filings. While IRS has adopted processes to help ensure proper filing for other disclosures, such as Form 8886, it has not extended these to Form 8886- T. In 2011, we recommended that IRS establish a process to periodically check whether Form 8886 filers met their reporting obligations. In response to that recommendation, IRS implemented a new indicator and matching process to review whether filers met their obligations. IRS officials told us that similar controls do not exist for 8886-T filings. TE/GE officials said one way that they ensure forms are filed is through penalties, yet they said they have never assessed the penalty for nonfiling of Form 8886-T. TE/GE officials also said that another way they ensure proper filing is through education and cited such documentation as IRS Publication 557, Tax-Exempt Status for Your Organization. IRS said it provides other information through its website informing charities of their responsibilities. Despite this education effort, it may still be the case that a lack of knowledge about filing requirements reduces the number of tax-exempt entities that file. An IRS official suggested that charities may not have the financial sophistication to realize that they are involved in a prohibited tax shelter transaction and therefore are required to file a Form 8886-T. Without a better understanding of the reasons behind the low filing, IRS cannot be reasonably certain that tax-exempt entities are following the law on filing Form 8886-T and ensuring tax-exempt entities’ compliance. We were able to use the IRS audit and disclosure data to perform certain analyses on abusive tax schemes with tax-exempt entities for this report, but data deficiencies prevented us from undertaking more complete analysis and hinder management’s use of the data. These deficiencies— which are inconsistent with internal control standards for quality information—weaken divisions’ ability to identify abusive tax schemes involving tax-exempt entities as well as the Executive Steering Committee’s ability to make decisions about how to address abusive tax schemes across divisions and develop compliance strategies. First, the descriptions of project codes in audit data do not always clearly identify abusive tax schemes across operating divisions. For example, one code LB&I uses to identify abusive transactions in audit data is “domestic tax shelters.” TE/GE uses two codes both titled, “Abusive Tax Avoidance Transactions,” and SB/SE uses a code titled, “Tax Shelter List Projects.” IRS officials provided no additional documentation on what these codes mean. The lack of specificity hinders analyses of abusive tax schemes involving tax-exempt entities. IRS officials said that they do not keep an overall list of project codes that cover abusive schemes involving tax-exempt entities. This limits their ability to readily assess and manage audits of abusive tax schemes involving tax-exempt entities. However, they did say such a list, which would be effective in certain circumstances or operating divisions, might be possible to produce. Cross-operating division analysis could enhance the Executive Steering Committee’s objective to assess emerging issues and develop policy responses. Second, we found that there were no project codes consistently identifying abusive schemes involving tax-exempt entities that crossed operating divisions. Instead, IRS officials said each operating division assigned its own project codes that identify abusive tax schemes. Having no uniform way to identify abusive schemes across the operating divisions makes analysis of schemes that overlap with different operating divisions’ responsibilities problematic and inhibits IRS from accomplishing its objectives. The lack of cross-divisional project codes echoes findings from our 2011 report on abusive tax avoidance transaction data, where we found that some abusive tax avoidance transaction data were reported inconsistently across IRS divisions. We said in that report that without comprehensive or consistent information, IRS does not have the best information to decide how to evaluate the results of its audits. Our recommendation to separately track the tax amounts recommended, assessed, and collected between abusive tax avoidance transaction issues and nonabusive transaction issues remains open because IRS said resource and capability constraints preclude it from capturing information in this way. Similarly, IRS officials told us it would be costly and logistically prohibitive to create new project codes identifying abusive schemes involving tax- exempt entities that crossed divisions. However, as we said in our previous report, tracking audit results for abusive and nonabusive transactions would provide IRS management with the data needed to make more informed decisions about program effectiveness and resource allocation. If, as IRS indicated above, it would be possible to make an overall list of codes, such a list could be used to achieve the same results as adjusting the database system. Although IRS does not identify some data that would facilitate analysis of abusive tax schemes involving tax-exempt entities spanning the operating divisions, we found evidence that TE/GE’s Returns Inventory and Classification System (RICS) could at least partially support analysis and monitoring of audit data across the operating divisions. For example, the RICS user manual states that RICS can access a variety of forms outside of TE/GE’s purview, such as Form 1065 and the Form 1120 series tax returns, which typically are handled by SB/SE or LB&I respectively. While TE/GE uses RICS, officials we spoke with at LB&I, for example, were not familiar with RICS’ capabilities. TE/GE officials said IRS would have to study whether using RICS in other divisions would generate productive audits. As we discussed earlier in this report, IRS’s Research, Analysis and Statistics office also has developed the capability to analyze narrative information, which it has tested on the Form 8886. However, this analytical tool is not being used operationally to review the Form 8886 or any other disclosure report. Our analysis shows that the tool has the potential to help IRS better search disclosure reports for additional information about transactions that could help IRS identify potentially abusive schemes involving tax-exempt entities. For example, it can be used to identify keywords in disclosure reports that could help determine whether a tax-exempt entity was a party to a reportable transaction that warrants further investigation for compliance. However, IRS officials told us they have no plans to use this tool but agreed that it may be beneficial. IRS officials also told us that TE/GE does not routinely review Form 8886 filings that show tax-exempt entities as being part of the reported transaction because the data are not clear indicators of noncompliance. However, by not using these data for possible leads, IRS may be missing opportunities to identify known abusive schemes, which is inconsistent with internal controls on using quality information. Again, our analysis of the 8886 filings combined with audit results suggests that there is potential for IRS to use the Form 8886 to identify potential noncompliance. Without conducting such an analysis, IRS may be missing opportunities to identify leads on tax-exempt entities in abusive tax schemes. We previously showed that abusive tax schemes involving tax-exempt entities can involve multiple types of entities that cross IRS’s operating divisions’ areas of responsibility. We also showed that IRS relies on auditors to refer potentially noncompliant entities involved in an abusive scheme to the responsible operating division. Consequently, IRS needs assurance that auditors’ make referrals when appropriate. However, IRS lacks a control to ensure that auditors make referrals correctly. An IRS audit official said that managers are tasked with reviewing auditors’ work and identifying referrals that should have been made during case closings. However, there is no documented guidance specifically directing managers to assess whether auditors correctly identified referrals involving abusive tax schemes, reducing assurance that such auditors will make such identifications correctly and route them appropriately. IRS’s audit quality review systems, which generally measure how well auditors follow procedures from a random sample of audits, also do not assess whether referrals of abusive schemes involving tax-exempt entities are properly identified and routed. The lack of guidance to ensure auditors make referrals across the operating divisions increases risk that the responsible division will not be alerted to potential noncompliance to make further assessments for enforcement action. Absent specific guidance, there also is increased risk that even when one entity in an abusive tax scheme is audited, other entities in the scheme may go unexamined. This is inconsistent with internal controls standards for control activities. Abusive tax schemes involving tax-exempt entities pose enforcement challenges for IRS, as schemes can cross IRS’s operating divisions’ areas of responsibility and evolve over time. While IRS has established programs to help identify new abusive schemes, opportunities exit to better ensure that IRS accomplishes its objectives of identifying existing and emerging schemes. In particular, opportunities exist for IRS to improve the quality of its data and how it is using the data it has in managing its programs. Because IRS uses codes to identify abusive schemes that are not consistent across the operating divisions, its efforts to formulate policy across operating divisions may be made more difficult. Also, IRS may not be making the best use of its data by not using existing tools that may be helpful in analyzing data to identify abusive schemes involving tax-exempt entities. Next, IRS has an opportunity to reduce the risk that tax-exempt entities are noncompliant by assessing the number of Form 8886-T filings. Finally, referrals across divisions play an important role in IRS’s ability to identify schemes with tax-exempt entities, but IRS’s internal control activities over referrals are limited. By taking actions to further strengthen its internal controls, IRS could enhance its efforts to identify and combat abusive tax schemes that involve tax-exempt entities. We are making the following five recommendations to IRS: The Commissioner of Internal Revenue should undertake a risk assessment of tax-exempt entity Form 8886-T filings. Based on the findings of the risk assessment, IRS should then determine whether steps are needed to increase compliance, such as, for example, through increased outreach to tax-exempt entities or assessment of nonfiling penalties. (Recommendation 1) The Commissioner of Internal Revenue should link audit data on abusive tax schemes involving tax-exempt entities across operating divisions and use the linked data to assess emerging issues and develop policy responses. (Recommendation 2) The Commissioner of Internal Revenue should test the ability of the Return Inventory Classification System to facilitate analysis and monitoring of audit data across the operating divisions and to support the IRS’s enforcement objectives. (Recommendation 3) The Commissioner of Internal Revenue should use existing data analytic tools to further mine Form 8886 and Form 8918 data, which could be used to find audit leads on tax-exempt entity involvement in potentially abusive tax schemes. (Recommendation 4) The Commissioner of Internal Revenue should develop guidance to help managers ensure referrals about abusive schemes involving tax-exempt entities are made across operating divisions. This could be accomplished by, for example, adopting specific guidance for audit managers to look for referral accuracy in their reviews of case closings. (Recommendation 5) We provided a draft of this report to the Commissioner of Internal Revenue for review and comment. On August 16, 2019, the IRS Deputy Commissioner for Services and Enforcement provided written comments stating that IRS agreed with GAO’s recommendations. In the letter, which is reproduced in appendix VII, the Deputy Commissioner said that GAO’s recommendations would provide IRS with additional opportunities for improving the identification of tax schemes involving exempt entities. IRS also sent us technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of the Treasury, the Commissioner of Internal Revenue, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9110 or mctiguej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. To describe ways in which taxpayers have abused a tax-exempt entity through abusive tax schemes, we conducted interviews with knowledgeable professionals and researchers. We chose the interview sources by reviewing relevant articles from academic databases and reaching out to professional organizations. We narrowed our list of examples of abusive tax schemes by focusing on those professionals and researchers who: had recent professional experience as an attorney, accountant, or other industry professional with a firm specializing in tax-exempt entities or tax shelters; had recent professional experience in nonprofit management or affiliation with professional associations specializing in nonprofit organization or oversight; had published books, articles, or other research on tax-exempt entities or tax shelters within the last 10 years; were recommended to us by a relevant professional organization, such as the American Bar Association or the American Institute of Certified Public Accountants; work for or previously worked in charity tax enforcement at the state previously worked for the Internal Revenue Service (IRS), specifically in the Tax Exempt and Government Entities Division (TE/GE); or would (in our professional judgment) be able to speak on the topics of abusive tax avoidance schemes or IRS investigations of tax-exempt entities. We conducted literature and court case reviews using academic and legal databases and covered years 2008 through 2018 using search terms such as “tax avoidance,” “tax-exempt,” and “shelter.” We combined the information found in interviews with reviews of relevant literature and court cases. We categorized the observations from our research by the following criteria. Involved multiple entities, at least one of which was tax exempt, and Involved a transaction or scheme already known to IRS, such as a listed transaction or transaction of interest or Involved a transaction mentioned in expert interviews. We then applied the following factors to make the final three choices for the examples: how representative the example was of abusive tax- schemes involving tax-exempt entities; how well-documented we found the example to be in literature reviews; how recent the example had been used by abusers; and how much impact the example had in terms of prevalence and tax revenues. To examine trends in IRS’s compliance and the characteristics of taxpayers audited for using abusive tax schemes involving tax-exempt entities, we collected data from the following IRS business operating divisions that conduct audits on abusive transactions: (1) TE/GE, (2) Small Business/Self-Employed (SB/SE), and (3) Large Business and International (LB&I). We received data extracts from the following computer data systems (1) the Returns Inventory and Classification System data extracts from TE/GE; (2) the Automated Information Management System Centralized Information System (A-CIS), utilized by SB/SE and LB&I; and (3) the Compliance Data Warehouse (CDW) utilized by SB/SE and LB&I. IRS performs a number of quality control steps to verify the internal consistency of the Return Inventory Classification System, A-CIS, and CDW data. Additionally, we reviewed documentation from the operating divisions on the data, discussed the data with IRS officials, and conducted electronic reliability testing. For example, we verified the completeness of analysis variables and the date ranges for our analysis. We excluded 178 records from our analysis of SB/SE data because they were not within our date range. Based on our review, we believe the data are sufficiently complete and accurate for our purposes. We identified audits with potential tax exempt entities by selecting audits based on IRS project codes that IRS agreed were relevant to determine the minimum number of audits conducted on abusive schemes involving tax-exempt entities. We also matched the SB/SE and LB&I data with IRS’s Form 8886, Reportable Transaction Disclosure Statement, data file of the tax-exempt records. We used these data to produce descriptive statistics on audit and taxpayer characteristics and IRS compliance efforts for 2008 through 2017. Tax return information came from Form 1040, U.S. Individual Income Tax Return; Form 1120, U.S. Corporation Income Tax Return; and Form 990, Return of Organization Exempt from Income Tax. Dollars amounts reported for the 10-year period have been adjusted for inflation in 2018 dollars based on a Fiscal Year, Gross Domestic Product Price index. Separately, we compiled descriptive statistics on disclosures of reportable transactions that also involved tax-exempt entities from Form 8886 and Form 8918, Material Advisor Disclosure Statement. IRS’s Office of Tax Shelter Analysis provided the data for Forms 8886 and 8918. We also performed an analysis of the narrative portions of Form 8886 from tax year 2017 to identify more information about the descriptions of the reported transactions. We identified that IRS could conduct Python optical character recognition (OCR) analysis of the text fields on IRS Form 8886. We worked with officials at IRS’s Research, Analysis and Statistics office on using Python computer programming language to conduct the analysis. IRS ran the OCR using keywords associated with 29 different tax-exempt organizations we identified. The keywords we used were based on characteristics of tax-exempt entities, such as “charity” and “foundation”— terms found in 26 U.S.C. Section 501. We received summary tables and copies of PDFs of all Form 8886-T, Disclosure by Tax-Exempt Entity Regarding Prohibited Tax Shelter Transaction, for tax years 2004 through 2016. We checked the reliability of IRS’s summary tables and manually reviewed the PDF submissions to generate descriptions of the Form 8886-T data. We conducted reliability testing for all of the data we used for this objective. For the audit and tax return data, we interviewed relevant IRS officials and compared our statistical runs with publicly available statistics. For the Form 8886 and Form 8918 disclosure data, we interviewed relevant IRS officials. For the 8886-T data, we compared the summary tables IRS provided with the PDFs of the original Form 8886-T submissions. To assess how IRS identifies emerging abusive tax schemes and to identify potential improvements, we reviewed documentation on programs that help IRS identify possible abusive tax schemes involving tax-exempt entities. We identified the programs by reviewing IRS documentation, including the Internal Revenue Manual, in combination with IRS’s determination of relevant programs (see appendix VI for more details about these programs). We then identified criteria appropriate for assessing the programs’ alignment selected principles from Standards for Internal Control in the Federal Government (Green Book) and leading practices from our Fraud Risk Management Framework. To select these criteria, we reviewed the Green Book and Fraud Risk Management Framework to identify principles relevant to specific aspects of IRS’s programs for identifying and initiating enforcement actions against abusive tax schemes involving tax exempt entities. IRS agreed that these criteria were appropriate. The following list shows the criteria we selected through this process. Green Book Principle 3: Establish structure, responsibility, and Green Book Principle 4: Demonstrate a commitment to recruit, develop, and retain competent individuals Green Book Principle 7: Identify, analyze, and respond to risks Green Book Principle 8: Assess fraud risk GAO Fraud Risk Management Framework Overarching Concept 1.2 (structure) GAO Fraud Risk Management Framework Overarching Concept 2.1 (plans exist to assess fraud) Green Book Principle 10: Design control activities Green Book Principle 13: Use quality information Green Book Principle 14: Communicate Internally Green Book Principle 16: Perform Monitoring Activities After establishing appropriate criteria, two analysts independently reviewed appropriate evidence and determined whether the evidence aligned with the criteria for the programs was based on the attributes for the Green Book criteria and Fraud Risk Management Framework guidance. We also considered how the programs met TE/GE’s objective to “promote the highest degree of voluntary compliance with the statutes governing qualification of plans and exemption of certain types of organizations from tax and to determine the extent of compliance and the causes of noncompliance with the tax laws by plans and organizations,” and IRS’s objective to “identify new types of tax transactions or promotions that are either abusive or potentially abusive requiring different levels of coordination and varying strategies.” We determined the criterion was met only if all of the programs under review offered sufficient support. Table 1 shows how we assessed the programs we reviewed on the criteria. The federal tax code provides a variety of tax benefits to organizations often referred to as “tax exempt.” This appendix focuses on organizations or entities qualifying for a tax-exempt status under 26 U.S.C. § 501. We discussed the tax benefits and requirements for different types of tax- exempt organizations in our 2014 report on oversight of charitable organizations. In addition to section 501, there are various other scattered provisions which give a full or partial tax exemption to certain specific types of entities and income. Section 501 distinguishes between charitable organizations, also known as 501(c)(3) organizations (after the subsection in which they are defined) from all other organizations qualifying for an exemption under section 501. Organizations that qualify for an exemption under section 501, but are not charitable organizations have been referred to as mutual benefit organizations or non-charitable nonprofits. Section 509 further divides charitable organizations between those that are private foundations and all other charitable organizations, and private foundations are divided between operating and nonoperating foundations in section 4942. Figure 9 shows the declines in the Internal Revenue Service’s Tax- Exempt and Government Entities Division’s budget since an increase from fiscal years 2009 through 2011. The Internal Revenue Service (IRS) defines a transaction as one that includes all the factual elements relevant to the expected tax treatment of any investment, entity, plan, or arrangement, and it includes any series of steps carried out as part of a plan. Department of the Treasury (Treasury) regulations require that certain transactions be registered and that lists of investors be maintained by parties who organize or sell interests in the transaction. A transaction becomes “reportable” (i.e., a taxpayer must disclose it to IRS on Form 8886) when it falls under one or more of the following categories: listed, confidential, contractual protection, loss transactions, and transactions of interest. Listed transactions: A listed transaction is reportable when it is the same or substantially similar to one of the types of transactions that IRS has determined to be an avoidance transaction. IRS provides a detailed list of the 36 recognized listed transactions on its website. Confidential transactions: A confidential transaction is offered to a taxpayer or a related party under conditions of confidentiality and is a type of transaction for which a taxpayer has paid a minimum advisor fee. A transaction is considered offered under conditions of confidentiality for two reasons: the advisor places a limitation on the taxpayer’s disclosure of the tax treatment or tax structure of the transaction, and the limitation on disclosure protects the confidentiality of the advisor’s tax strategies. The transaction is treated as confidential even if the conditions of confidentiality are not legally binding on the taxpayer. Contractual protection transactions: A contractual protection transaction is a transaction for which a taxpayer or a related party has the right to full or partial refund of fees if all or part of the tax consequences from the transaction are not sustained. It also includes a transaction for which fees are contingent on a taxpayer’s realization of tax benefits from the transaction. Loss transactions: A loss transaction is a transaction that results in a taxpayer claiming a loss. The type of taxpaying individual or entity determines the applicable amount of the loss. The types of loss transactions IRS has described are as follows for: Individuals: at least $2 million in any single tax year or $4 million in any combination of tax years. Corporations (excluding S corporations): at least $10 million in any single tax year or $20 million in any combination of tax years. Partnerships with only corporations (excluding S corporations) as partners: at least $10 million in any single tax year or $20 million in any combination of tax years, whether or not any losses flow through to one or more partners. All other partnerships and S corporations: At least $2 million in any single tax year or $4 million in any combination of tax years, whether or not any losses flow through to one or more partners or stakeholders. Trusts: At least $2 million in any single tax year or $4 million in any combination of tax years, whether or not any losses flow through to one or more beneficiaries. Transactions of interest: A transaction of interest is one that IRS and Treasury believe to have the potential for tax avoidance or evasion, but which lacks enough information for IRS and Treasury to determine whether the transaction should be identified as a tax avoidance transaction. Tables 2 and 3 below show Internal Revenue Service (IRS) data for forms filed and audited, and the audit coverage rate, for individual income tax and corporate tax returns during fiscal years 2006 to 2017. Table 4 shows the number of returns processed and audited, and the audit coverage rate, for tax-exempt organizations during fiscal years 2006 to 2015. All three tables show declines in audit coverage rates: the decline occurred for individual income tax returns after fiscal year 2011 and for corporate income tax returns after fiscal year 2012. The audit coverage rate for tax-exempt organizations’ returns declined from fiscal years 2013 to 2015, the last fiscal year for which we have complete data on tax- exempt organization returns. Table 21 lists 10 programs that the Internal Revenue Service (IRS) operates that may identify or conduct enforcement action on abusive tax schemes that involve a tax-exempt entity. James R. McTigue, Jr. (202) 512-9110 or mctiguej@gao.gov. In addition to the contact named above, Kevin Daly (Assistant Director); Susan Baker; Jehan Chase; Sara Daleski; Steven Flint; Eric Gorman; Gina Hoover; Andrew Howard; Edward Nannenhorn; Kevin Newak; Carolyn Ours; Robert Robinson; Dylan Stagner; and Elwood White made significant contributions to this review. Also contributing to this report were Toni Gillich; Sarah Gilliland; John Hussey; Jessica Lucas-Judy; Cynthia Saunders; Stewart Small; Rebecca Shea; and Janet Temko-Blinder.", "summary": "Abusive tax schemes contribute to the tax gap and threaten the tax system's integrity. When abusive tax schemes involve tax-exempt entities, they also can erode the public's confidence in the charitable sector. GAO was asked to review what is known about abusive transactions involving tax-exempt entities and how IRS addresses them. This report, among other things, (1) describes ways in which taxpayers have abused an entity's tax-exempt status; (2) examines trends in IRS's compliance efforts; and (3) assesses how IRS identifies emerging abusive tax schemes involving tax-exempt entities. GAO reviewed research on tax schemes involving tax-exempt entities, and interviewed relevant professionals and researchers about tax schemes involving tax-exempt entities; compiled statistics from IRS audit and disclosure data; and compared documentation and testimony from IRS officials on IRS programs and guidance from its operating divisions with certain internal control and GAO fraud framework criteria. Taxpayers have used a variety of abusive tax schemes involving tax-exempt entities. In some schemes, the tax-exempt entity is complicit in the scheme, while in others it is not. For example, an abusive tax scheme could involve multiple donors grossly overvaluing charitable contributions, where the tax-exempt entity is not part of the scheme. Conversely, some patient assistance programs—which can help patients obtain medical care or medications—have been used by pharmaceutical manufacturers to make charitable donations that can be viewed as furthering private interests. Internal Revenue Service (IRS) audits of abusive tax schemes are trending downward, as the figure below shows audits by IRS's Large Business and International division. This trend has occurred amid generally declining IRS resources and corresponds with an overall decrease in audit activity by IRS over recent years. IRS has a variety of programs working collectively to identify abusive tax schemes involving tax-exempt entities, but some internal control weaknesses exist in its approach. For example, GAO found three ways that IRS data or programs were inconsistent with internal control standards for using quality information. First, database project codes used for identifying data on abusive tax schemes are not linked across IRS's audit divisions and do not consistently identify whether a tax-exempt entity was involved. Second, IRS has not leveraged a database with cross-divisional information to facilitate its analysis and monitoring of audit data across divisions. Finally, IRS has not used existing analytic tools to mine the narrative fields of tax forms. Doing so could provide audit leads on abusive schemes involving tax-exempt entities. These deficiencies inhibit IRS's ability to identify abusive tax schemes and develop responses to those schemes. GAO is making five recommendations to IRS to strengthen its internal controls, including that it link data across operating divisions, test the ability of a database to facilitate analysis of audit data, and use existing analytic tools to further mine information on tax forms. In commenting on a draft of this report, IRS agreed with all of GAO's recommendations.", "document_type": "gao"}
{"report": "ONDCP was established by the Anti-Drug Abuse Act of 1988 as a component of the Executive Office of the President, and its Director is to assist the President in the establishment of policies, goals, objectives, and priorities for the National Drug Control Program. ONDCP is responsible for (1) leading the national drug control effort, (2) coordinating and overseeing the implementation of national drug control policy, (3) assessing and certifying the adequacy of National Drug Control Programs and the budget for those programs, and (4) evaluating the effectiveness of national drug control policy efforts. About a dozen National Drug Control Program agencies, as identified by ONDCP, have responsibilities for drug prevention, treatment, and law enforcement activities. Among other responsibilities, the Director of ONDCP is required to develop and promulgate the National Drug Control Strategy. The National Drug Control Strategy is to set forth a comprehensive plan to reduce illicit drug use and the consequences of such illicit drug use in the United States by limiting the availability of and reducing the demand for illegal drugs. Many of the SUPPORT Act’s requirements for the National Drug Control Strategy are the same as, or similar to, those that applied under the ONDCP Reauthorization Act of 2006. For example, both laws require the National Drug Control Strategy to include a 5-year projection for the National Drug Control Program and budget priorities. However, there are certain differences, and the SUPPORT Act includes a wide range of detailed new requirements that were not included under the ONDCP Reauthorization Act of 2006. One of these is that the National Drug Control Strategy include a description of how each comprehensive, research-based, long-range quantifiable goal established in the Strategy for reducing illicit drug use and the consequences of illicit drug use in the United States will be achieved. Other examples of new requirements include creating plans to increase data collection and expand treatment of substance use disorders. The SUPPORT Act also requires the Director to release a statement of drug control policy priorities in the calendar year of a presidential inauguration (but not later than April 1). The President is then required to submit to Congress a National Drug Control Strategy not later than the first Monday on February following the year in which the term of the President commences, and every two years thereafter. The Director of ONDCP is also responsible for developing a consolidated National Drug Control Program budget proposal for each fiscal year, which is designed to implement the National Drug Control Strategy and inform Congress and the public about total federal spending on drug control activities. As part of this effort, the Director of ONDCP is required to assess and certify National Drug Control Program agencies’ drug control budgets on an annual basis to determine if they are adequate to meet the goals and objectives of the National Drug Control Strategy. Figure 1 illustrates ONDCP’s budget certification process. ONDCP did not issue a National Drug Control Strategy for 2017 or 2018. Pursuant to the ONDCP Reauthorization Act of 2006, the Director of ONDCP was required to promulgate the National Drug Control Strategy annually and the President was to submit the National Drug Control Strategy to Congress by February 1 of each year. According to ONDCP officials, ONDCP did not issue a National Drug Control Strategy for these years because (1) ONDCP did not have a Senate-confirmed Director during those years; and (2) 2017 was the administration’s inaugural year, and previous administrations also did not issue a Strategy during their first years. By statute, in the absence of a Director, the Deputy Director of ONDCP is to perform the functions and duties of the Director temporarily in an acting capacity. ONDCP had officials serving as Acting Director beginning in January 2017. The current Director of ONDCP was appointed Deputy Director beginning in February 2018 and served as Acting Director from February 2018 until April 2018. As of April 2018, the current Director continued in his role as Deputy Director until he was confirmed by the Senate as Director of ONDCP in January 2019. The previous administration also did not issue a National Drug Control Strategy in its inaugural year—2009—but it did issue a National Drug Control Strategy in its second year, as shown in table 1. On January 31, 2019, ONDCP issued its National Drug Control Strategy for 2019, which we discuss in more detail later in the report. The ONDCP Reauthorization Act of 2006 required the Director of ONDCP to issue drug control funding guidance to the heads of departments and agencies with responsibilities under the National Drug Control Program by July 1 of each year. ONDCP is to issue funding guidance for agency budget proposals for the fiscal year two years in the future. For example, ONDCP was to issue funding guidance to agencies in 2017 for development of the 2019 budget, and issue funding guidance in 2018 for development of the 2020 budget. Such funding guidance was required to address funding priorities developed in the National Drug Control Strategy. National Drug Control Program agencies are to submit their budget requests to ONDCP in the summer of each year (before submission to the Office of Management and Budget) and in the fall of each year (at the same time as submission to the Office of Management and Budget).The Director of ONDCP then determines whether National Drug Control Program agencies’ summer budget requests are adequate to meet the goals of the National Drug Control Strategy and certifies whether fall budget submissions include the funding levels and initiatives identified during the summer budget review. Since ONDCP did not issue a Strategy in 2017 or 2018, ONDCP could not develop and issue funding guidance, nor could it review and certify budget requests and submissions of National Drug Control Program agencies, in accordance with the statutory requirement. ONDCP officials stated that—in lieu of a Strategy—they used other sources to formulate the administration’s priorities, which served as the basis for drug control funding guidance in 2017 and 2018. For example, for the development of the fiscal year 2019 drug control budget in calendar year 2017, ONDCP officials stated that they relied upon the following sources for drug policy guidance: Initial development of the President’s Initiative to Stop Opioid Abuse and Reduce Drug Supply and Demand; Draft recommendations from the President’s Commission on Combating Drug Addiction and the Opioid Crisis; policy statements made by the President as a candidate; and policy priorities identified in the fiscal year 2018 President’s Budget. Additionally, for the development of the fiscal year 2020 funding guidance in calendar year 2018, ONDCP officials stated that they relied upon the following sources for drug policy priorities: the interim and final Report of the President’s Commission on Combating Drug Addiction and the Opioid Crisis; the President’s Initiative to Stop Opioid Abuse and Reduce Drug Supply and Demand; the draft National Security Council Strategic Framework; and a draft 2018 National Drug Control Strategy that ONDCP officials told us they drafted but did not issue. These sources may have provided ONDCP officials with some information about policy priorities and actions. However, ONDCP officials stated they did not consider these documents to be the National Drug Control Strategy, and none of the sources fulfill the statutory requirements under the ONDCP Reauthorization Act of 2006, which require funding guidance to address priorities from the National Drug Control Strategy. ONDCP officials told us that they provided drug control funding guidance to the heads of departments and agencies with responsibilities under the National Drug Control Program in 2017 and 2018. As described by ONDCP officials, drug control funding guidance identifies key program goals and the programs and activities that require agency funding to achieve the objectives of the National Drug Control Strategy. ONDCP has since issued the 2019 National Drug Control Strategy which states that it establishes the administration’s drug control priorities. The Strategy also states that the priorities provide federal drug control departments and agencies strategic guidance for developing their own drug control plans and strategies, and that the Strategy is intended to ensure federal drug control budget dollars are allocated in a manner consistent with the administration’s priorities. ONDCP officials told us that the agency intends to issue the next National Drug Control Strategy in February 2020 in accordance with the SUPPORT Act. The 2019 National Drug Control Strategy and companion documents include information to address some but not all selected requirements under the ONDCP Reauthorization Act of 2006. ONDCP issued multiple documents that together were intended to address the requirements for the National Drug Control Strategy. The first document, the 2019 National Drug Control Strategy, was issued January 31, 2019, with three companion documents issued later in April and May 2019. These companion documents were the 2019 Data Supplement, the 2019 Performance Reporting System, and the 2019 Budget and Performance Summary. In our March 2019 testimony, we reported that the first document—the National Drug Control Strategy, which was the only one of the four documents available at the time of our testimony—did not include certain information required under the ONDCP Reauthorization Act of 2006. These selected requirements included: annual quantifiable and measurable objectives and specific targets; a 5-year projection for program and budget priorities; specific drug trend assessments; and a description of a performance measurement system. Following our March 2019 testimony, we reviewed the three companion documents and found that while they provide some additional information to address these same selected requirements, they do not completely address the requirements. As stated earlier, we based our analysis of the 2019 National Drug Control Strategy and companion documents on the ONDCP Reauthorization Act of 2006, which was the applicable law at the time ONDCP began drafting the Strategy. Current law is reflected in the SUPPORT Act, which includes some of the same requirements from the ONDCP Reauthorization Act of 2006 and some new or different requirements. In the paragraphs below, we identify which selected requirements from the ONDCP Reauthorization Act of 2006 were retained under the SUPPORT Act, and therefore represent current law, and which selected requirements were not retained. For those selected requirements that were not retained, we identify comparable current requirements in the SUPPORT Act. Annual quantifiable and measurable objectives and specific targets. Pursuant to the ONDCP Reauthorization Act of 2006, the National Drug Control Strategy was required to include “annual quantifiable and measurable objectives and specific targets to accomplish long-term quantifiable goals that the Director determines may be achieved during each year beginning on the date on which the National Drug Control Strategy is submitted.” The SUPPORT Act retained this requirement. We testified in March 2019 that while the 2019 National Drug Control Strategy lists seven items it designates as measures of performance or effectiveness, the document did not indicate how these would be quantified or measured. The document also did not include targets to be achieved each year. Our subsequent analysis of the three companion documents showed that one additional document provided more information related to this requirement. The 2019 Performance Reporting System includes 9 goals and 17 quantifiable and measurable objectives with specific targets for certain years. Specifically, the goals and objectives identified in the 2019 Performance Reporting System included educating the public about the dangers of drug use; expanding access to evidence-based treatment; decreasing the over-prescribing of opioid medications; and reducing the availability of illicit drugs in the United States through reduced production, increased seizure trends, and increased prices and reduced drug purity, among other things. The document states that each goal “is accompanied by aggressive, but achievable, objectives with two- and five-year targets from a baseline of 2017.” However, the 2019 Strategy does not meet the statutory requirement because it does not have annual targets that may be achieved each year. Instead, the Performance Reporting System states that 16 of the 17 objectives in the Strategy have 2-year targets to be achieved in 2019, and 14 of the 17 objectives have 5-year targets to be achieved in 2022. The objectives do not include annual targets for the other intervening years— 2018, 2020, and 2021, as required. The Performance Reporting System states that while ONDCP assumes a linear progression from the baseline year—2017, in most cases—to the 2022 target, the trajectory may not actually be linear, “but rather it may occur at varying rates over the 5-year period due to multiple factors which influence the ability to achieve each of the stated goals and objectives.” In contrast, other information ONDCP provided to us stated that annual targets can readily be determined from the linear paths between the 2- and 5-year targets. Without identifying annual targets, the 2019 National Drug Control Strategy and companion documents do not meet the statutory requirement. Further, annual targets would better position ONDCP to monitor progress in intervening years and make any needed changes to achieve its goals and objectives. The SUPPORT Act continues to require ONDCP to establish annual quantifiable and measurable objectives and specific targets in future Strategy iterations. By taking steps to address this requirement ONDCP could further demonstrate whether it is making meaningful progress every year toward the targets it sets. A 5-year projection for program and budget priorities. Pursuant to the ONDCP Reauthorization Act of 2006, the National Drug Control Strategy was required to include “a 5-year projection for program and budget priorities.” The SUPPORT Act retained this requirement. As we testified in March 2019, the 2019 National Drug Control Strategy did not include this information. Our subsequent analysis of the three companion documents showed that one document—the 2019 Performance Reporting System—provided more information about ONDCP’s program priorities but not ONDCP’s budget priorities. Specifically, 14 of the 17 objectives ONDCP included in the 2019 Performance Reporting System contain various 5-year targets, such as to reduce the rates of illicit drug and opioid use among youth by 15 percent. According to ONDCP officials, the objectives and targets in the 2019 Performance Reporting System satisfy the requirement for 5-year program and budget priorities. However, the document does not include how these objectives and targets relate to 5-year budget priorities. The SUPPORT Act continues to require ONDCP to include a 5-year projection of program and budget priorities in future Strategy iterations. By taking steps to address this requirement, ONDCP and National Drug Control Program agencies will be better positioned to plan for the resources needed to achieve the efforts that will have the greatest impact. Specific drug trend assessments. Pursuant to the ONDCP Reauthorization Act of 2006, the National Drug Control Strategy was required to include assessments of the reduction of the consequences of illicit drug use and availability and the reduction of illicit drug availability. We testified in March 2019 that the 2019 National Drug Control Strategy did not include these assessments. Our subsequent analysis of the three companion documents showed that the 2019 Data Supplement provided more information to address the required assessments but did not address all of the requirements. For example, the assessment of the reduction of the consequences of illicit drug use and availability was to include, among other things, the annual national health care cost of illicit drug use. However, the most recent national health care cost data in the 2019 Data Supplement is from 2007, and ONDCP did not indicate in the supplement whether more recent data were available. In another example, the assessment of the reduction of illicit drug availability was to be measured by, among other things, the number of illicit drug manufacturing laboratories seized and destroyed and the number of hectares of marijuana, poppy, and coca cultivated and destroyed domestically and in other countries. The 2019 Data Supplement provided data for marijuana and poppy until 2016 and for the quantity of coca eradicated until 2015. The SUPPORT Act no longer requires these specific assessments. However, the SUPPORT Act does include a new requirement that the National Drug Control Strategy provide “ description of the current prevalence of illicit drug use in the United States, including both the availability of illicit drugs and the prevalence of substance use disorders.” The SUPPORT Act also contains a new requirement—which we describe later in this report—for ONDCP to describe how each comprehensive, research-based, long-range quantifiable goal in the National Drug Control Strategy was determined, including data, research, or other information used to inform the determination. We address ONDCP’s implementation of this new requirement under the SUPPORT Act later in the report. A description of a performance measurement system. Pursuant to the ONDCP Reauthorization Act of 2006, the National Drug Control Strategy was required to include a “description of a national drug control performance measurement system” that: develops 2-year and 5-year performance measures and targets; describes the sources of information and data that will be used for identifies major programs and activities of the National Drug Control Program agencies that support the goals and annual objectives of the National Drug Control Strategy; evaluates the contribution of demand reduction and supply reduction activities implemented by each National Drug Control Program agency in support of the Strategy; monitors consistency between the drug-related goals and objectives of the National Drug Control Program agencies and ensures that each agency’s goals and budgets support and are fully consistent with the National Drug Control Strategy, among others; and coordinates the development and implementation of national drug control data collection and reporting systems to support policy formulation and performance measurement, including certain assessments. We testified in March 2019 that the 2019 National Drug Control Strategy did not include a description of a performance measurement system pursuant to the ONDCP Reauthorization Act of 2006. Our subsequent analysis of the three companion documents showed that the 2019 Performance Reporting System provides information about some of the elements the performance measurement system is required to do. For example, the 2019 Performance Reporting System includes 2-year and 5- year targets for many of its objectives and describes some of the sources of data that will be used to measure each target. However, it does not include a description of the system that will accomplish each of the requirements in the ONDCP Reauthorization Act of 2006. For example, it does not describe a performance measurement system that identifies major programs and activities of the National Drug Control Program agencies that support the goals and annual objectives of the National Drug Control Strategy. Such programs and activities could indicate how ONDCP expects to achieve these objectives, such as how to educate the public about the dangers of drug use, or how to expand access to evidence-based treatment. Additionally, it does not describe how the performance measurement system monitors consistency between the drug-related goals and objectives of the National Drug Control Program agencies and ensures that each agency’s goals and budgets support and are fully consistent with the National Drug Control Strategy. ONDCP officials stated they believe the 2019 Performance Reporting System meets the statutory requirement for a description of a performance measurement system. The SUPPORT Act, as originally enacted in October 2018, no longer required a description of a performance measurement system. However, the ONDCP Technical Corrections Act of 2019, enacted in November 2019, amended the SUPPORT Act to reinstate the requirement for a description of a performance measurement system. Therefore, this requirement will apply to the 2020 National Drug Control Strategy and future Strategy iterations. As of August 2019, ONDCP filled all five coordinator positions described in the SUPPORT Act, two of which are substantively new positions. Specifically, ONDCP officials stated that they have designated officials for the new positions of performance budget coordinator and emerging and continuing threats coordinator. By filling each of these positions, ONDCP is better positioned to fulfill the responsibilities for which each position is accountable, as described in figure 2 below. As of October 2019, ONDCP officials could not provide in writing or otherwise describe key planning elements to ensure ONDCP can meet selected new requirements in the SUPPORT Act related to the development of the 2020 and future National Drug Control Strategy iterations, and related to the development and implementation of the Drug Control Data Dashboard. Figure 3 outlines the selected requirements for the Strategy, which were effective upon enactment of the SUPPORT Act in October 2018. Each of the four selected SUPPORT Act requirements described in Figure 3 requires ONDCP to include specific information in the 2020 and future National Drug Control Strategy iterations. For example, for each comprehensive, research-based, long-range, quantifiable goal, the National Drug Control Strategy must contain (1) a description of how each goal will be achieved; (2) a performance evaluation plan for each goal; and (3) a description for how each goal was determined. The National Drug Control Strategy must also include a plan to expand treatment for substance use disorders. Officials from ONDCP and selected agencies told us that in spring 2019 ONDCP requested that the National Drug Control Program agencies determine how their existing programs and activities align with the 2019 National Drug Control Strategy, including the goals and objectives articulated in the 2019 Performance Reporting System. In October 2019, ONDCP officials told us that the 2020 Strategy would be issued in accordance with the SUPPORT Act, by the first Monday in February (February 3, 2020). ONDCP also provided us with two documents to describe its approach for meeting this deadline. One document includes a table that lists SUPPORT Act requirements along with the ONDCP component(s) responsible for implementation and the deadline. The other document provides a high-level summary of the National Drug Control Strategy development and interagency review process. For example, the plan to monitor progress on the drafting of components’ sections of the Strategy notes that it is to occur through “as- needed (but frequent)” meetings with the deputy chief of staff and the components and their heads. The extensive nature of the new SUPPORT Act requirements, as described above, indicates that significant implementation steps may be necessary, such as, a description of the specific steps necessary to accomplish this overarching task, identification of who will be responsible for each step, and a schedule of interim milestones. However, neither of these documents describes such critical implementation steps. Further, neither specifies what resources or processes, for example, would be needed and by what specific milestone date ONDCP would accomplish any particular step to complete the overall work in a timely manner. For example, the document that includes the table indicates that the deadline for all requirements related to the National Drug Control Strategy is February 2020. However, some requirements associated with the development of the Strategy, such as consultation requirements, would need to be completed before the Strategy’s due date—February 2020. According to Standards for Internal Control in the Federal Government under Internal Control Principle 6, to achieve an entity’s mission, management should define objectives in specific terms so they are understood at all levels of the entity. This involves clearly defining what it is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement—in other words, key planning elements. Standards for project management also state that managing a project involves developing a plan with specific actions and milestone dates. Defining these key planning elements will help provide assurance that ONDCP’s efforts will result in a National Drug Control Strategy—for 2020 and future years—that fully addresses the requirements of the SUPPORT Act. In addition, developing and documenting these planning elements would help ONDCP structure its planning efforts through consideration of resource investments, time frames, and any necessary processes, policies, roles, and responsibilities to address each requirement. Furthermore, implementing these planning elements will help ensure that ONDCP follows a routine planning process going forward, and that future iterations of the National Drug Control Strategy that ONDCP develops are consistent with the law. Additionally, as of December 2019, ONDCP has not documented key planning elements to ensure it will meet the SUPPORT Act’s requirements for the Drug Control Data Dashboard, to make timely information publicly available on the scope and complexity of drug use and drug control activities. The SUPPORT Act includes requirements for what data is to be included in the Drug Control Data Dashboard as well as its functionality, to ensure it is searchable and sortable. Figure 4 outlines the requirements for the Drug Control Data Dashboard, which were effective upon enactment of the SUPPORT Act in October 2018. In August 2019, ONDCP posted a public version of the Drug Control Data Dashboard that included information from the 2019 Data Supplement in spreadsheet format, but did not provide all of the data required by the SUPPORT Act. For example, the Drug Control Data Dashboard does not include required data on the extent of the unmet need for substance use disorder treatment. ONDCP officials shared information regarding potential data sources they may use to fulfill the additional required data elements. In addition, ONDCP officials told us that some data requirements listed in the statute do not exist at this time. For example, ONDCP officials stated that data do not exist regarding the known and estimated flow of substances into the United States for the current calendar year and each of the three previous years. ONDCP officials stated that there was more work necessary to ensure all the required data are incorporated into the Drug Control Data Dashboard. At that time, they also stated that they expected to address all required elements by the end of 2019. However, we found that they do not have key planning elements, such as a specific timeline with interim milestones or documented plans for when and how they would complete this work. ONDCP subsequently posted an updated version of the Drug Control Data Dashboard, which we reviewed in December 2019. While the updated Drug Control Data Dashboard identifies required data elements that are unavailable, ONDCP has not addressed how or when ONDCP planned to provide them, such as by identifying alternative data sources or identifying additional resources that may be necessary for enhanced data collection efforts. The SUPPORT Act also requires the Drug Control Data Dashboard to be machine-readable and searchable by year, agency, drug, and location, to the extent practicable. Officials stated in September 2019 they planned to add this functionality to the Drug Control Data Dashboard in the fall of 2019. In written comments on a draft of this report in December 2019, ONDCP indicated that the data have been posted in a machine-readable, sortable, and searchable format. However, as of December 2019, we found that the Drug Control Data Dashboard is still not fully searchable by year, agency, drug and location. We have previously reported on key practices for agencies to follow when reporting government data. These practices describe, for example, that agencies should ensure their website’s data search functions and overall interface is intuitive to users. While effective implementation of such functions can be a significant undertaking, ONDCP does not have plans to account for timing, content, functionality, or any additional resources required to fully implement this requirement. ONDCP officials stated in September 2019 they may need to consult Congress about additional resources to fulfill all of the requirements related to the Drug Control Data Dashboard, but stated that they do not have specific plans for what resources they may request. Internal control standards call for agencies to define key planning elements, including how a task will be accomplished and associated timeframes. Developing and documenting key planning elements— including resource investments, time frames, and any necessary processes, policies, roles, and responsibilities—will better position ONDCP to fully implement all of the law’s requirements for the Drug Control Data Dashboard. Once implemented, the Drug Control Data Dashboard will help enable ONDCP to capitalize on available data to better understand the scope and nature of the drug crisis. ONDCP is responsible for leading the nation’s fight against a persistent drug epidemic that continues to devastate Americans’ lives. However, the 2019 National Drug Control Strategy does not fully comply with the law, and the agency has not developed key planning elements to help ensure it will meet its significant additional responsibilities under the SUPPORT Act. These responsibilities include issuing the National Drug Control Strategy in accordance with statutory requirements to help prioritize and measure key efforts to address the drug epidemic and creating a Drug Control Data Dashboard that contains timely information about the scope and complexity of the drug epidemic. These responsibilities also extend beyond the upcoming 2020 Strategy, with requirements to complete future Strategy iterations on a regular basis. Developing and documenting key planning elements, such as resource investments, time frames, and any necessary processes, policies, roles, and responsibilities—will help ONDCP structure its ongoing efforts. Implementing this approach will then better position ONDCP to meet statutory requirements for the next Strategy, due in February 2020, and satisfy all requirements related to the Drug Control Data Dashboard. Implementing this approach over time will also help ONDCP ensure it is meeting statutory requirements for future iterations of the National Drug Control Strategy. We are making 4 recommendations to ONDCP. The Director of ONDCP should develop and document key planning elements to help the agency meet the SUPPORT Act requirements for the 2020 National Drug Control Strategy and future Strategy iterations. These planning elements should include descriptions of resource investments, time frames, and any processes, policies, roles, and responsibilities needed to address each requirement. (Recommendation 1) The Director of ONDCP should—after developing and documenting key planning elements to meet the SUPPORT Act requirements—routinely implement an approach, based on these planning elements, to meet the requirements for the 2020 National Drug Control Strategy and future Strategy iterations. (Recommendation 2) The Director of ONDCP should develop and document key planning elements to help the agency meet the SUPPORT Act requirements to establish a Drug Control Data Dashboard that would include descriptions of resource investments, time frames, and any processes, policies, or roles, and responsibilities needed to address this requirement. (Recommendation 3) The Director of ONDCP should—after developing and documenting key planning elements—implement an approach, based on these planning elements, to meet the SUPPORT Act requirements to establish a Drug Control Data Dashboard. (Recommendation 4) We provided a draft of this report for review and comment to ONDCP, DHS, DOJ, and HHS. ONDCP provided written comments, which are summarized below and reproduced in appendix I. ONDCP, DHS, and DOJ also provided technical comments, which we incorporated, as appropriate. In an email, an HHS official stated that HHS did not have any comments on the report. In its written comments, ONDCP stated that it accepted the first two recommendations regarding the need for a robust internal planning process for National Drug Control Strategies. Specifically, the first recommendation is for ONDCP to develop and document key planning elements to help the agency meet the SUPPORT Act requirements for the 2020 National Drug Control Strategy and future Strategy iterations. The second recommendation is for ONDCP to routinely implement an approach to meet these requirements. In particular, ONDCP agreed to implement key planning elements for future Strategies that will include detailed descriptions of planned steps, identifying which ONDCP component will be responsible for each step, resource investments, interim milestones, and overall time frames. If implemented as planned, these actions would address the intent of these recommendations. Regarding the third and fourth recommendations related to the Drug Control Data Dashboard, ONDCP noted that these recommendations have been rendered moot because the agency has already fully complied with posting the Data Dashboard to its website. ONDCP also stated that it has posted to the Data Dashboard all of the drug-related data required by ONDCP’s statute that currently exists. Further, ONDCP stated that the data has been posted in machine-readable, sortable, and searchable format as required and it will be updated on a continuous basis throughout the year as new data become available. While ONDCP has included additional information on the Dashboard, the two recommendations are to develop, document, and implement key planning elements for the Dashboard to fully meet the law’s requirements, which ONDCP has not yet done. For example, ONDCP identifies in the Dashboard which of the required data elements are unavailable, such as required data on the extent of the unmet need for substance use disorder treatment. However, as stated in the report, ONDCP has not documented key planning elements for how it will address these missing data. Such planning elements could include approaches for collecting the missing data, such as articulating a plan to work with Congress to identify alternative data sources or to identify additional resources that may be necessary for enhanced data collection efforts. Furthermore, ONDCP has not developed or implemented key planning elements to ensure the Drug Control Data Dashboard has the search features noted in the statute. In its current format, the Dashboard is not fully searchable by year, agency, drug, and location. While the statute indicates that search features should have been implemented “to the extent practicable,” ONDCP did not explain why it was not practical to implement them. Therefore, we continue to believe that developing, documenting, and implementing key planning elements for the Dashboard to fully meet the law’s requirements will help enable ONDCP to capitalize on available data to better understand the scope and nature of the drug crisis. ONDCP also noted several points related to our specific findings, as discussed below. First, ONDCP noted that it did issue robust drug budget guidance to National Drug Control Program agencies during 2017 and 2018. The report acknowledges that ONDCP provided this guidance. However, as explained in the report, the guidance is statutorily required to address funding priorities developed in the National Drug Control Strategy. Since ONDCP did not issue a Strategy in 2017 or 2018, it could not meet this statutory requirement. In addition, ONDCP stated that it maintains that the 2019 National Drug Control Strategy met all statutory requirements, and therefore does not agree with our analysis of its adherence to those requirements. ONDCP also noted that the four requirements we assessed constitute only a small portion of the many requirements for the 2019 National Drug Control Strategy and that the report gives the misleading impression that ONDCP did not comply with some significant number of requirements. We recognize that there are a number of requirements for the Strategy; however, as stated in the report, our review focused on these four provisions because we determined them to be significant to ONDCP’s role in setting a strategic direction to oversee and coordinate national drug control policy, and because they are critical to ensuring a framework for measuring results. Specifically, the provisions related to including information in the Strategy related to annual quantifiable and measurable objectives and specific targets, a 5-year projection for program and budget priorities, specific drug trend assessments, and a description of a performance measurement system. As detailed in the report, we found that the 2019 Strategy addressed some—but not all—of these four statutory requirements. For example, we found that the Strategy did not include a 5-year projection for budget priorities and included only some information related to specific drug trend assessments. In its written comments, ONDCP provided additional explanation for why it did not agree with our characterization of the requirements. For example, ONDCP stated that it is not able to provide quantitative fiscal year projections for future years because this would go against long-standing Office of Management and Budget policy. Related to drug trend assessments, ONDCP noted that it reports data generated by other government agencies, and that policy research funding for ONDCP has not been appropriated since fiscal year 2011. We made recommendations, which ONDCP agreed to implement, focused on developing and implementing key planning elements such as descriptions of resource investments; timeframes; and processes, policies, and responsibilities needed to address each requirement. Implementing these planning elements could, for example, help ensure that ONDCP addresses any policy considerations or additional resources needed to help ensure that future iterations of the Strategy fully meet all statutory requirements. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of National Drug Control Policy, the Secretary of the Department of Health and Human Services, the Acting Secretary of the Department of Homeland Security, the Attorney General, and other interested parties. In addition, this report is available at no charge on GAO’s website at http://www.gao.gov. If you or your staff has any questions concerning this testimony, please contact Triana McNeil at (202) 512-8777 or McNeilT@gao.gov or Mary Denigan-Macauley at (202) 512-7114 or McNeilT@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contacts named above, Joy Booth (Assistant Director), Will Simerl (Assistant Director), Michelle Loutoo Wilson (Analyst-in- Charge), Billy Commons, Wendy Dye, Jane Eyre, Kaitlin Farquharson, Susan Hsu Michael, Amanda Miller, and Jan Montgomery made key contributions to this report.", "summary": "Almost 70,000 people died from drug overdoses in 2018, according to the latest Centers for Disease Control and Prevention data. The 2018 SUPPORT Act reauthorized ONDCP and imposed new requirements. GAO noted in its March 2019 High Risk report that the federal effort to prevent drug misuse is an emerging issue requiring close attention. Pursuant to 21 U.S.C. § 1708a(b), GAO has periodically assessed ONDCP's programs and operations. This report assesses the extent to which ONDCP (1) met selected statutory requirements related to the National Drug Control Strategy in 2017, 2018, and 2019, and (2) has planned or implemented actions to meet selected new requirements in the SUPPORT Act. GAO assessed the 2019 Strategy and companion documents against four key statutory requirements that were consistent with or similar to ONDCP's ongoing responsibilities under the SUPPORT Act. GAO also assessed ONDCP's progress in addressing seven new SUPPORT Act requirements, and interviewed ONDCP officials. The Office of National Drug Control Policy (ONDCP) is responsible for overseeing and coordinating the development and implementation of U.S. drug control policy across the federal government. However, ONDCP did not issue a National Drug Control Strategy for either 2017 or 2018, as required by statute. ONDCP was also required to assess and certify federal agencies' drug control budgets to determine if they were adequate to meet Strategy goals and objectives. Without a Strategy in 2017 and 2018, ONDCP could not complete this process according to statutory requirements. ONDCP issued a 2019 Strategy and companion documents that addressed some but not all of the selected statutory requirements GAO reviewed. For example, the Strategy and companion documents did not include the required 5-year projection for budget priorities. The October 2018 Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment for Patients and Communities Act (SUPPORT Act) retained some requirements and introduced new ones for ONDCP. ONDCP met some SUPPORT Act requirements GAO reviewed. For example, ONDCP filled all five coordinator positions described in the SUPPORT Act. However, its approach to meeting other requirements does not incorporate key planning elements. For example, the SUPPORT Act requires that future iterations of the Strategy include a description of how each goal will be achieved, performance evaluation plans, and a plan for expanding treatment of substance use disorders. ONDCP could not provide in writing or otherwise describe its planned steps, interim milestones, resource investments, or overall timeframes—all key planning elements—that would provide assurance it can meet these requirements by the deadline for the next Strategy—February 2020. The SUPPORT Act also required ONDCP to publish an online searchable Data Dashboard of drug control data, with information including quantities of drugs and frequency of their use. While ONDCP published (and later updated) a public version of this resource on its website, as of December 2019, it was not complete (e.g., lacked required data on the unmet need for substance use disorder treatment). Further, ONDCP officials had no information on next steps for fully meeting the requirements. Developing, documenting, and implementing key planning elements to meet these requirements—including resource investments, time frames, and any processes, policies, roles, and responsibilities—would be consistent with key principles for achieving an entity's objective and standards for project management. Importantly, doing so would help ONDCP structure its planning efforts and comply with the law. GAO is making 4 recommendations to ONDCP to develop, document, and implement key planning elements to meet certain requirements in the SUPPORT Act. ONDCP agreed to implement 2 recommendations related to the Strategy, but disagreed with 2 related to the Drug Control Data Dashboard, noting that recent updates satisfy the law. GAO maintains that they do not fully do so, and that implementing key planning elements would help address the law, as discussed in the report.", "document_type": "gao"}
{"report": "Part of the Mariana Islands Archipelago, the CNMI—one of five U.S. territories—consists of 14 islands in the western Pacific Ocean, just north of Guam and about 3,200 miles west of Hawaii. In 2018, the CNMI had an estimated total population of 51,994, according to the U.S. Census Bureau. According to the CNMI’s 2016 Household, Income, and Expenditures Survey, 89 percent of the population lived on the island of Saipan, with an additional 6 percent on the island of Tinian and 5 percent on the island of Rota (see fig. 1). On October 24, 2018, Super Typhoon Yutu made landfall in the CNMI causing widespread damage to the islands. Saipan, Tinian, and Rota experienced heavy rainfall and extremely high winds, which caused damage to homes, businesses, and critical infrastructure. The typhoon severely damaged utility infrastructure on all three islands, including downed power lines, transformers, and poles, which caused power outages across all the islands (see fig. 2). Damage from Yutu also closed the Saipan International Airport, which was unable to restore full flight services until March 2019, according to the Marianas Visitors Authority. The United States captured the Northern Mariana Islands from Japan during the latter part of World War II. After the war, the U.S. Congress approved a trusteeship agreement making the United States responsible to the United Nations for the administration of the islands. In 1976, the District of the Mariana Islands entered into the Covenant with the United States establishing the island territory’s status as a self-governing commonwealth in political union with the United States. This Covenant grants the CNMI the right of self-governance over internal affairs and the United States complete responsibility and authority for matters relating to foreign affairs and defense affecting the CNMI. The Covenant initially made many federal laws applicable to the CNMI, including laws that provide federal services and financial assistance programs. However, the Covenant preserved the CNMI’s exemption from certain federal laws that had previously been inapplicable to the Trust Territory of the Pacific Islands, including certain federal minimum wage provisions and immigration laws, with some limited exceptions. Under the terms of the Covenant, the federal government has the right to apply federal law in these exempted areas without the consent of the CNMI government. In 2008, the CNRA amended the joint resolution approving the U.S.– CNMI covenant to apply federal immigration law to the CNMI, with a transition period for foreign workers that would end on December 31, 2014, unless extended by the U.S. Secretary of Labor. To provide for an orderly transition from the CNMI immigration system to the U.S. federal immigration system under the immigration laws of the United States, DHS, through USCIS, established the CW-1 program in 2011. The transition period was previously extended through December 31, 2019, under the Consolidated and Further Continuing Appropriations Act, 2015. Through the program, employers petition for nonimmigrant CW-1 permits that allow foreign workers who meet certain requirements to work temporarily in the CNMI. Since 2008, Congress has amended the CNRA several times, with provisions that affected the length of the transition period, the number of CW-1 permits allocated, and the distribution of permits. The CNRA, as amended by the Northern Mariana Islands U.S. Workforce Act of 2018, extends the CW-1 program through December 31, 2029, defines the number of permits DHS may issue annually, and reduces that number each year until the end of the transition period. In addition, the Northern Mariana Islands Long-Term Legal Residents Relief Act of June 2019 established a new category of long-term residents in the CNMI, assuming they met certain qualifications (see table 1). Figure 3 shows the numerical limits on CW-1 permits established by DHS and the numerical limits for permits specified in the Northern Mariana Islands U.S. Workforce Act of 2018. The limits shown are the maximum number of permits available for each fiscal year through the end of the transition period and may not reflect the number of permits for which employers would petition and that DHS would approve. The CNMI’s GDP, adjusted for inflation, grew every year from 2012 to 2017, but declined in 2018, according to BEA. GDP, in 2018 inflation- adjusted dollars, grew from $1.022 billion in 2015, to $1.311 billion in 2016, and to $1.646 billion in 2017, before contracting to $1.323 billion in 2018. See figure 4 below for CNMI inflation-adjusted gross domestic product over this time. BEA estimates that the CNMI’s GDP, adjusted for inflation, increased by 28.4 percent in 2016 and by 25.5 percent in 2017 (see fig. 5). BEA attributes this economic growth to exports of services, which reflected continued growth in visitor spending, particularly for casino gambling. In 2018, inflation-adjusted GDP fell by 20 percent, which reflected decreases in exports of services and private fixed investment. According to BEA, exports of services decreased 39 percent, due to a drop in visitor spending, in particular spending on casino gambling where revenues fell over 50 percent in 2018. BEA data on the value added to GDP by individual industries show the change in the composition of the CNMI economy as accommodations and amusement became the largest component of the economy and garment manufacturing declined. In particular: From 2007 to 2017, the contribution to GDP by accommodations and amusement, which partially includes the tourism sector, grew from less than 12 percent to 45 percent. From 2007 to 2017, the contribution to GDP by manufacturing declined from 19 percent to 1 percent, according to BEA. This reflects the decline of the garment manufacturing industry. Between 2007 and 2017, the contribution to GDP by government declined from about 24 percent to 16 percent of GDP. See figure 6 for value added by industry as a percentage of CNMI GDP. Following a period of growth in visitor arrivals—from about 338,000 in fiscal year 2011 to more than 653,000 in fiscal year 2017—visitor arrivals dropped in fiscal year 2018 to about 607,000 and in fiscal year 2019 to less than 425,000 (see fig. 7). According to BEA, the decline in visitors in early fiscal year 2019 was attributable to Super Typhoon Yutu, which devastated the CNMI in October 2018. In November 2018, following Super Typhoon Yutu, visitor arrivals in the CNMI plummeted from the previous month’s total of 32,108 to 5,595. This drop also represented an 88 percent decline from November 2017, when 48,039 visitors arrived in the CNMI. See figure 8 below, which compares monthly visitor arrivals for fiscal years 2018 and 2019, which started on October 1, 2017 and October 1, 2018, respectively. The composition of visitors by country of residence has also significantly shifted since 2005. Data from the Marianas Visitors Authority show that the decline in Japanese arrivals from fiscal years 2005 to 2019 was offset by the increase in arrivals from China and South Korea (see fig. 9). In particular, Japanese arrivals declined from about 376,000 in 2005 (71 percent of total visitors) to about 12,000 in 2019 (3 percent). South Korean arrivals increased from about 65,000 in 2005 (12 percent) to about 192,000 in 2019 (45 percent). Chinese arrivals increased from about 32,000 in 2005 (6 percent) to 186,000 in 2019 (44 percent). While eligible Japanese and South Korean visitors enter the CNMI under a visa waiver program, Chinese visitors are ineligible for the program but can remain temporarily in the CNMI under DHS’s discretionary parole authority, according to DHS officials. DHS exercises parole authority to allow, on a case-by-case basis, eligible nationals of China to enter the CNMI temporarily as tourists when there is significant public benefit, according to DHS. U.S. Customs and Border Protection, a DHS component, recently announced a reduction in the length of stay for Chinese citizens from 45 to 14 days for their entry into the CNMI under discretionary parole. CNMI’s Comprehensive Economic Development Strategy 2019 Update indicates that visa-free access to Chinese visitors serves as the linchpin for the CNMI casino investment. On January 29, 2020, the Governor of the CNMI issued an executive order that declared a state of significant emergency in the Commonwealth related to the spread of the coronavirus from China. Among other measures, the Governor suspended the arrival of travelers from mainland China for a period of 30 days. The Governor also directed the CNMI Secretary of Finance and the CNMI Office of Management and Budget to undertake a cost-impact analysis on the effects the ban will have on the economy. Within the tourism sector, the CNMI government has provided for the licensing of casinos on Tinian and Saipan, but both casinos have faced challenges. Tinian: Tinian Dynasty Hotel and Casino was established in 1998 to boost economic development. In operation for over a decade, the casino was investigated and cited by several federal agencies and closed in 2015. Most recently, following an Internal Revenue Service investigation, the U.S. Department of Justice filed a criminal complaint against the casino operator and two individuals on April 19, 2013, alleging that between September 2009 and April 2013 the casino failed to file reports on currency transactions greater than $10,000, and engaged in a pattern of accommodating gamblers in conducting transactions greater than $10,000. The U.S. Department of Treasury Financial Crimes Enforcement Network reported on June 3, 2015, that it had assessed a $75 million civil money penalty against the casino operator for willful and egregious violations of the Bank Secrecy Act. Saipan: In March 2014, while needing a new revenue source to fund government policies, such as a generous government retirement program, the CNMI government passed a public law that authorized and established an exclusive casino license in Saipan, which was awarded to Imperial Pacific International Holdings of Hong Kong. The operator began construction of a casino and hotel complex originally scheduled for completion no later than 36 months from the date of the casino license, or by August 2017. After facing construction challenges, the CNMI Casino Commission approved delays in the completion schedule. The new casino opened for business on July 6, 2017. As of August 2019, the casino was operating, but hotel construction had not progressed beyond the structural frame and a partial facade. According to a casino representative, labor shortages and Super Typhoon Yutu have delayed construction. Figure 10 shows the casino and hotel tower in August 2019. Several federal agencies have investigated and cited the casino operator and its construction contractors. The casino operator and its contractors have been fined for unfair labor practices: On May 30, 2017, the U.S. Department of Labor’s Occupation Safety and Health Administration reported proposed penalties of $193,750 against three contractors that exposed workers to numerous workplace hazards at the casino site in Saipan. On March 5, 2018, the U.S. Department of Labor announced it had finalized a series of settlements with contractors that would pay $13.9 million in back wages and damages to thousands of Chinese employees who had come to build the Saipan casino and hotel. On April 25, 2019, the U.S. Department of Labor announced it had secured a $3.3 million consent judgment against the casino’s developer for minimum wage, overtime, and recordkeeping violations of the Fair Labor Standards Act. On September 24, 2019, the U.S. Equal Employment Opportunity Commission filed suit against the casino, alleging the casino operator had violated federal law by subjecting female employees to sexual harassment, other sex-based discrimination, and retaliation. Financial reporting from the casino operator in 2019 included warnings about losses in 2018 and 2019. Specifically: On April 29, 2019, the casino operator released its 2018 Annual Report. In this report, independent auditors found that the casino operator had incurred a net loss of almost $3 billion in Hong Kong dollars, or about $379 million in US dollars, and had accumulated current liabilities greater than this amount, for calendar year 2018. The auditors concluded that these conditions, along with others noted in the report, indicate the existence of a material uncertainty, which may cast significant doubt on the operator’s ability to continue in business. On August 9, 2019, the casino operator issued a warning to shareholders and potential investors that it expected to record a loss for the first 6 months of 2019 as compared to a profit for the same period in 2018. On August 30, 2019, the casino operator released its 2019 Interim Report. An independent auditor noted that during the 6-month period ending June 30, 2019, the casino operator incurred a net loss of almost $1.9 billion Hong Kong dollars, or more than $240 million in U.S. dollars. The auditor included the same warning of a material uncertainty reported in the 2018 Annual Report. On November 7, 2019, the casino operator posted an announcement to the Hong Kong Stock Exchange that it had assisted in an investigation at the request of local enforcement authorities, and provided relevant information and documents as required by the enforcement authorities. Between 2014 and 2018, the ratio of United States to foreign workers in the CNMI remained close to 50 percent, according to CNMI Department of Finance tax data that identified the citizenship of workers. In 2018, United States workers constituted 49 percent of the workforce. These workers included U.S. citizens and nationals, and citizens from the Freely Associated States—the Federated States of Micronesia, the Republic of the Marshall Islands, and the Republic of Palau. The size of the CNMI workforce grew every year from 2014 through 2017 before contracting by about 2,000 workers, or 5.6 percent, in 2018, according to CNMI tax data. While the ratio between United States workers and foreign workers has remained steady over the past 5 years, the number and share of foreign workers in the overall CNMI workforce fell significantly from 2001 through 2018. Over this same period, the number of United States workers remained more stable, dropping from about 15,500 workers in 2001, to about 13,700 workers in 2018, or a 12 percent decline in total United States workers. United States workers represented 30 percent of the workforce in 2001 and 49 percent in 2018 (see fig. 11). On 2018 tax forms, the CNMI government started collecting information from employers on workers’ visa type, in response to the Northern Mariana Islands U.S. Workforce Act of 2018. According to a CNMI government report, the new information will help identify workers lawfully admitted for permanent residence. However, about one-third of the 2018 CNMI tax forms collected did not include information on the worker’s visa type. According to the report, the missing data may be attributed to the new reporting procedure for the 2018 tax form. The report indicated that after Super Typhoon Yutu devastated the islands of Saipan and Tinian, there was very little time to adequately inform and prepare employers of the new procedure for the tax form before the end of tax year 2018. Although the CNMI Department of Labor conducted a training presentation in December 2018, not all employers attended and so were unaware of the new procedure. The overall number of approved CW-1 permits fell from a high of 13,581 for fiscal year 2016 to 9,016 for fiscal year 2018. The number of approved permits rose by 23 percent for fiscal year 2019 to 11,093. However, the number of approved CW-1 permits for 2019 was about 2,000 below the updated 2019 cap established in 2018. As figure 12 shows, the number of CW-1 permits approved by USCIS for fiscal years 2012 to 2015 remained well under the annual numerical limits and exceeded or neared those limits for fiscal years 2016 through 2018. According to USCIS data, most individuals with approved CW-1 permits for fiscal year 2019 were born in the Philippines or China. In addition, as table 2 shows, the number of permits approved for workers born in China was four times higher for fiscal years 2016 and 2017 than for fiscal year 2015, although that number fell by more than half for fiscal year 2018. As we reported in 2017, firms involved in building the casino in Saipan have primarily employed Chinese workers. CW-1 permit data for fiscal year 2019 show that the CW-1 permit holders most commonly worked in building service or food service. See table 3 for the top 10 occupations for CW-1 permits for 2015 through 2019 based on 2019’s top 10 occupations. In 2017, Congress amended the CNRA to, among other things, restrict future CW-1 permits for workers in construction and extraction occupations (as defined in the U.S. Department of Labor’s Standard Occupational Classification system) to allow extensions only of those permits first issued before October 1, 2015. This restriction was later modified in 2018 to only allow permits for construction and extraction occupations to be issued for those who qualified as long-term workers, those being workers who were admitted as CW-1 workers during fiscal year 2015 and during every subsequent fiscal year beginning before July 24, 2018. The number of CW-1 permits for construction trades fell from 3,119 for fiscal year 2017 to 347 for fiscal year 2019 (see table 3 above). According to CNMI officials, the islands continue to rebuild following the devastation of Super Typhoon Yutu in late 2018. These officials noted that one of their challenges is the limited number of construction workers. We have previously reported on the limited number of construction workers in the CNMI. In 2017, when Congress restricted the use of CW-1 permits for the construction trade, employers could continue to petition for construction workers using H-2B visas. In January 2019, because of concerns about overstays and human trafficking, DHS removed the Philippines from the list of countries eligible for the H-2B program. CNMI government officials, among others, had previously voiced concerns that the removal of the Philippines from the list would make it more difficult to hire construction workers in the aftermath of Super Typhoon Yutu. On September 24, 2019, a bill, H.R.4479—the Disaster Recovery Workforce Act, was introduced in the House of Representatives that would increase by 3,000 the number of CW-1 permits available for construction and extraction occupations for fiscal years 2020 through 2022, and also included an exception to the restriction on issuing such permits to individuals other than long-term workers for those fiscal years. On December 20, 2019, an amended version of this bill, which retained the 3,000 permit increase and the exception, was signed into law as part of the Further Consolidated Appropriations Act, 2020. As provided in Public Law 115-218, long-term workers may obtain CW-1 permits valid for up to 3 years and may renew their permits for up to 3 years during the transition program. About 23 percent of FY 2019 CW-1 permit holders had maintained continuous employment in the CNMI since 2015. USCIS CW-1 permit data for fiscal years 2015 through 2019 show that, of the 11,093 foreign workers with CW-1 permits approved by USCIS for fiscal year 2019, 2,517 workers (22.7 percent) had maintained continuous employment in the CNMI since fiscal year 2015, as shown in table 4. Public Law 115-218 defines a long-term worker as an alien who was admitted to the CNMI as a CW-1 worker during fiscal year 2015 and every subsequent fiscal year prior to enactment of the law in 2018. We provided a draft of this product to the CNMI government, and the U.S. Departments of Commerce, Homeland Security, and the Interior for comment. The CNMI government and the Department of the Interior told us they had no comments on the draft report. The Departments of Commerce and Homeland Security provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Governor of the CNMI, the Secretary of Commerce, the Secretary of Homeland Security, and the Secretary of the Interior. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you and your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. The Northern Mariana Islands U.S. Workforce Act of 2018 included a provision for GAO to biennially report on the ratio between United States workers and other workers in the Commonwealth of the Northern Mariana Islands (CNMI) workforce during each of the previous 5 calendar years. This report examines (1) economic trends in the CNMI and (2) trends in the composition of the CNMI workforce, including the ratio of United States workers to foreign workers in the CNMI during the previous 5 calendar years. To examine the trends in the CNMI economy, we reviewed prior GAO reports and we obtained and analyzed data from the Department of Commerce’s Bureau of Economic Analysis (BEA) on the Gross Domestic Product (GDP) of the CNMI, including contributions to GDP by select industries, for calendar years 2007 through 2018. We converted the GDP figures from 2009 base year dollars to 2018 base year dollars. We also obtained and analyzed data on visitor arrivals from the Marianas Visitor Authority for fiscal years 2005 through 2019. We compared the data against data we have previously reported. As it relates to visitor arrivals, we discussed with officials from the Mariana Visitors Authority whether Super Typhoon Yutu disrupted the collection of data. All data were deemed reliable for our purposes. To examine the CNMI casinos, we reviewed annual reports from the Saipan casino operator, U.S. and CNMI government documents, press releases and news reports. We also interviewed CNMI government officials from the Departments of Finance and Labor, the Commonwealth Casino Commission, and a casino representative in Saipan, and reviewed documents from U.S. government agencies, to understand potential challenges that could affect the CNMI economy. To examine the trends in the CNMI workforce, we obtained and analyzed data from the CNMI government and the Department of Homeland Security (DHS). Specifically: We obtained summary level tax data from the CNMI government on December 18, 2019, which included information on the number of workers in the CNMI and their citizenship, to examine the ratio between United States and foreign workers in the CNMI workforce. These data were compiled by the CNMI Department of Finance, and were rolled up to provide counts of workers based on the workers’ reported citizenship. The data available for inclusion in this report do not match the definition of United States worker established in the 2018 Act. The Act defines a United States worker as any worker who is: a citizen or national of the United States; an alien who has been lawfully admitted for permanent residence; or a citizen of the Marshall Islands, the Federated States of Micronesia, or the Republic of Palau who has been lawfully admitted to the United States pursuant to their respective compacts of free association. In 2018, the CNMI government began collecting data on worker visa status recorded on employee tax documents filed by the employer. But about one-third of collected tax forms did not include information about the visa type or status of the worker. Therefore, with incomplete data, we could not identify people lawfully admitted for permanent residence who remain foreign citizens. The summary-level citizenship data have been used in prior GAO reports. We reviewed those reports to ensure that the data were being collected using the same procedures as in the past, and we were using the data in the same manner. We also interviewed knowledgeable CNMI officials about the data collection methods and how the data were extracted from CNMI government data systems, and checked available documentation from those prior GAO reports to confirm our use of them. We found the data were sufficiently reliable for our purposes of summarizing the numbers of United States workers and foreign workers. We obtained record-level data (such as worker’s name, worker’s date of birth, and petition receipt number) from DHS’s U.S. Citizenship and Immigration Services (USCIS) for fiscal years 2012 through 2019 to examine CNMI-Only Transitional Worker (CW-1) program information on workers since the program began. We compared the annual number of approved CW-1 permits with the annual numerical limit, or cap, on CW-1 permits that USCIS set for fiscal years 2012 through 2019. Using computerized algorithms, we analyzed the data for key characteristics of workers who were granted CW-1 permits, such as years of continuous employment in the CNMI. To assess the reliability of the USCIS data, we tested the data electronically to identify and resolve inconsistencies in personally identifiable information for permit holders and to ensure accuracy in tracking these individuals over time, and we discussed our results with USCIS officials. We have previously used the same methods for assessing the reliability of this data, and USCIS had agreed with that methodology. We determined that the USCIS data were sufficiently reliable for our purposes of reporting on characteristics of CW-1 permit holders for fiscal year 2019 and for identifying trends over time. We conducted this performance audit from August 2019 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Emil Friberg (Assistant Director), Joshua Akery, Kathryn H. Bernet, Martin de Alteriis, Christopher Hayes (Analyst in Charge), Christopher Keblitis, Andrew Kurtzman, Moon Parks, Aldo Salerno, and Alexander Welsh made key contributions to this report.", "summary": "The Consolidated Natural Resources Act of 2008, which amended the 1976 covenant between the United States and the CNMI, established federal control of CNMI immigration beginning in 2009. Under the act, the Department of Homeland Security began implementing a foreign worker permit program that was specific to the CNMI. The Northern Mariana Islands U.S. Workforce Act of 2018 extended the CNMI-Only Transitional Worker (CW-1) program for 10 additional years, through the end of 2029. The Northern Mariana Islands U.S. Workforce Act of 2018 included a provision for GAO to examine the ratio of United States workers to other workers over the 5 previous calendar years in the CNMI. This report examines (1) recent economic trends in the CNMI through 2018, and (2) recent trends in the composition of the CNMI workforce from 2001 through 2018, including the ratio of United States workers to foreign workers for each of the 5 previous calendar years. GAO analyzed CNMI government and U.S. agency data, prior GAO reports, and interviewed officials from the CNMI government, and the U.S. Departments of Commerce, Homeland Security, the Interior, and Labor. Although the Commonwealth of the Northern Mariana Islands (CNMI) economy grew in 2016 and 2017, it declined in 2018. The U.S. Department of Commerce's Bureau of Economic Analysis (BEA) reports that the CNMI's gross domestic product (GDP) grew 28.4 percent in 2016 and 25.5 percent in 2017, which reflected continued growth in visitor spending, particularly for casino gambling. However, BEA estimates that GDP in the CNMI fell by 20 percent in 2018, with a sharp drop in tourist spending and casino gambling revenues following the severe damage of Super Typhoon Yutu, which made landfall in October 2018. According to BEA, revenue from casino gambling dropped over 50 percent in 2018. In August 2019, the parent company of the casino in the CNMI warned shareholders and potential investors that it expected to record a loss for the first 6 months of 2019 as compared with a profit for the same period in 2018. The company's independent auditor also concluded that the financial information for the first 6 months of 2019 might cast significant doubt on the ability of the company to continue as a going concern. The ratio of United States workers to foreign workers in the CNMI remained close to 50 percent from 2014 through 2018, with United States workers making up 49 percent of the workforce in 2018, according to CNMI tax data. The size of the workforce grew each year from 2014 through 2017, before contracting by almost 2,000 workers in 2018. For 2018, the Department of Homeland Security approved about 9,000 CW-1 foreign worker permits, and approved more than 11,000 permits for 2019.", "document_type": "gao"}
{"report": "Congress and executive branch agencies have made consistent progress in addressing many of the actions we have identified since 2011, as shown in figure 2 and table 4. As of March 2020, Congress and executive branch agencies had fully or partially addressed nearly 80 percent of the actions we identified from 2011 to 2019. See GAO’s online Action Tracker for the status of all actions. As a result of steps Congress and executive branch agencies have taken to address our open actions, we have identified approximately $429 billion in total financial benefits, including $166 billion identified since our last report. About $393 billion of the total benefits accrued between 2010 and 2019, while approximately $36 billion are projected to accrue in 2020 or later, as shown in figure 3. Since our first annual report in 2011, these benefits have contributed to missions across the federal government, as shown in figure 4. Table 5 highlights examples of results achieved in addressing actions we identified over the past 10 years. Our suggested actions, when implemented, often result in benefits such as strengthened program oversight; improvements in major government programs or agencies; more effective and equitable government; and increased international security. The following recent examples illustrate these types of benefits. Housing Assistance (2012-28): The federal government and state and local entities provide both rental assistance and affordable housing through a wide variety of programs. In February 2012, we found instances of fragmentation and overlap among federal rental assistance program. We recommended that the Secretary of the Department of Housing and Urban Development (HUD) work with states and localities to develop an approach for compiling and reporting on the collective performance of federal, state, and local rental assistance programs. In 2019, Executive Order 13878 established the White House Council on Eliminating Regulatory Barriers to Affordable Housing. The establishment of the council and the actions taken by HUD are positive steps for reaching out to states and localities and allowing Congress, decision makers, and stakeholders to evaluate collective performance data and provide mechanisms for setting priorities, allocating resources, and restructuring efforts, as needed, to achieve long-term housing goals. Military and Veterans Health Care (2012-15): The Departments of Defense (DOD) and Veterans Affairs (VA) play key roles in offering support to servicemembers and veterans through various programs and activities. In 2012, we found that the departments needed to improve integration across care coordination and case management programs to reduce duplication and better assist servicemembers, veterans, and their families. We recommended that the Secretaries of Defense and Veterans Affairs develop and implement a plan to strengthen functional integration across all DOD and VA care coordination and case management. The departments took several steps between 2012 and 2019 to address this, including establishing a Care Coordination Business Line within their joint Health Executive Committee. This function is intended to develop mechanisms for making joint policy decisions, involve the appropriate decision makers for timely implementation of policy, and ensure that outcomes and goals are identified and achieved, among other things. By taking these steps, DOD and VA strengthen their oversight and more closely integrate care coordination efforts. Tax Policies and Enforcement (2015-17): Since 1980, partnerships’ and S corporations’ share of business receipts increased greatly. These entities generally do not pay income taxes; instead, income or losses (hundreds of billions of dollars annually) flow through to partners and shareholders on their personal income tax returns. In 2014, we found that the full extent of partnership and S corporation income misreporting is unknown. Electronic filed (e-filed) tax returns provide the Internal Revenue Service (IRS) with digital information to improve enforcement operations and service to taxpayers. We recommended that Congress consider expanding the mandate that partnerships and corporations e-file their tax returns to cover a greater share of filed returns. In 2018, Congress passed and the President signed legislation lowering the e-file threshold for partnership and corporation returns. Requiring greater e-filing of tax return information will help IRS identify which partnership and corporation tax returns would be most productive to examine, and could reduce the number of compliant taxpayers selected for examination. Further, expanded e-filing will reduce IRS’s tax return processing costs. Coordination of Overseas Stabilization Efforts (2019-12): The United States has a national security interest in promoting stability in countries affected by violent conflict. We looked at how three federal agencies and an independent institute support conflict prevention, mitigation, and stabilization efforts, such as removing explosives hidden near homes. In 2019, we found that although these entities have worked together in Iraq, Nigeria, and Syria, they had not documented their agreement on key areas of collaboration, such as clarifying roles and responsibilities for stabilization efforts. We recommended that the Departments of State and Defense and the U.S. Agency for International Development should document their agreement to coordinate U.S. stabilization efforts. In 2019, the agencies took several steps to address this such as publishing a directive with the agreed upon definition of stabilization, description of agency roles and responsibilities, and related policies and guidance. Articulating their agreement in formal documents should help strengthen the agencies’ coordination of U.S. stabilization efforts and mitigate the risks associated with fragmentation, overlap, and duplication. Congress and executive branch agencies have made progress toward addressing the 1,076 actions we have identified since 2011. However, further efforts are needed to fully address the 467 actions that are partially addressed, not addressed, or new. We estimate that at least tens of billions of dollars in additional financial benefits could be realized should Congress and executive branch agencies fully address open actions, and other improvements can be achieved as well. In our 2011 to 2020 annual reports, we directed 110 actions to Congress, including the three new congressional actions we identified in 2020. Of the 110 actions, 58 (about 53 percent) remained open as of March 2020. Appendix V has a full list of all open congressional actions. We also directed 966 actions to executive branch agencies, including 165 new actions identified in 2020. As shown in figure 5, these actions span the government and are directed to dozens of federal agencies. Six of these agencies—DOD, IRS, OMB, VA, and the Departments of Health and Human Services (HHS) and Homeland Security, have more than 20 open actions. Of the 966 actions, 409 (42 percent) remained open as of March 2020. A significant number of open actions are directed to four agencies that made up about 79 percent of federal outlays in fiscal year 2019—HHS, the Social Security Administration, the Department of the Treasury (Treasury), and DOD. Figure 6 highlights agencies with open actions as well as their fiscal year 2019 share of federal outlays. We identified potential financial benefits associated with many open areas with actions directed to Congress and the executive branch. These benefits range from millions of dollars to tens of billions of dollars. For example, DOD could potentially save hundreds of millions of dollars annually by accurately measuring and reducing excess funded, unfinished work at military depots. In another example, IRS should establish a formal collaborative mechanism with the Department of Labor to better manage fragmented efforts and enhance compliance for certain individual retirement accounts that engaged in prohibited transactions, and thereby potentially increase revenues by millions of dollars. Table 6 highlights examples of areas where additional action could potentially result in financial benefits of $1 billion or more. Table 7 shows selected areas where Congress and executive branch agencies can take action to achieve other benefits, such as increased public safety, and more effective delivery of services. This report was prepared under the coordination of Jessica Lucas-Judy, Director, Strategic Issues, who may be reached at (202) 512-9110 or lucasjudyj@gao.gov, and J. Christopher Mihm, Managing Director, Strategic Issues, who may be reached at (202) 512-6806 or mihmj@gao.gov. Specific questions about individual issues may be directed to the area contact listed at the end of each summary. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Section 21 of Public Law 111-139, enacted in February 2010, requires us to conduct routine investigations to identify federal programs, agencies, offices, and initiatives with duplicative goals and activities within departments and government-wide. This provision also requires us to report annually to Congress on our findings, including the cost of such duplication, with recommendations for consolidation and elimination to reduce duplication and specific rescissions (legislation canceling previously enacted budget authority) that Congress may wish to consider. Our objectives in this report are to (1) identify potentially significant areas of fragmentation, overlap, and duplication and opportunities for cost savings and enhanced revenues that exist across the federal government; (2) assess to what extent have Congress and executive branch agencies addressed actions in our 2011 to 2019 annual reports; and (3) highlight examples of open actions directed to Congress or key executive branch agencies. For the purposes of our analysis, we used the term “fragmentation” to refer to circumstances in which more than one federal agency (or more than one organization within an agency) is involved in the same broad area of national need. We used the term “overlap” when multiple agencies or programs have similar goals, engage in similar activities or strategies to achieve them, or target similar beneficiaries. We considered “duplication” to occur when two or more agencies or programs are engaged in the same activities or provide the same services to the same beneficiaries. While fragmentation, overlap, and duplication are associated with a range of potential costs and benefits, we include them in this report only if there may be opportunities to improve how the government delivers these services. This report presents 18 new areas of fragmentation, overlap, or duplication where greater efficiencies or effectiveness in providing government services may be achievable. The report also highlights 11 other new opportunities for potential cost savings or revenue enhancements. In addition to these 29 new areas, we identified 88 new actions related to 10 existing areas presented in our 2011 to 2019 annual reports. To identify what actions, if any, exist to address fragmentation, overlap, and duplication and take advantage of opportunities for cost savings and enhanced revenues, we reviewed and updated our prior work and recommendations to identify what additional actions Congress may wish to consider and agencies may need to take. For example, we used our prior work identifying leading practices that could help agencies address challenges associated with interagency coordination and collaboration and with evaluating performance and results in achieving efficiencies. To identify the potential financial and other benefits that might result from actions addressing fragmentation, overlap, or duplication, or taking advantage of other opportunities for cost savings and enhanced revenues, we collected and analyzed data on costs and potential savings to the extent they were available. Estimating the benefits that could result from addressing these actions was not possible in some cases because information about the extent and impact of fragmentation, overlap, and duplication among certain programs was not available. Further, the financial benefits that can be achieved from addressing fragmentation, overlap, or duplication or taking advantage of other opportunities for cost savings and enhanced revenues were not always quantifiable in advance of congressional and executive branch decision- making. In addition, the needed information was not readily available on, among other things, program performance, the level of funding devoted to duplicative programs, or the implementation costs and time frames that might be associated with program consolidations or terminations. As possible, we used partial data and conservative assumptions to provide rough estimates of potential savings magnitude, when more precise estimates were not possible. Appendix VI provides additional information on the federal programs or other activities related to the new areas of fragmentation, overlap, duplication, and cost savings or revenue enhancement discussed in this report, including budgetary information when available. We assessed the reliability of any computer-processed data that materially affected our findings, including cost savings and revenue enhancement estimates. The steps that we take to assess the reliability of data vary but are chosen to accomplish the auditing requirement that the data be sufficiently reliable given the purposes for which they are used in our products. We review published documentation about the data system and inspector general or other reviews of the data. We may interview agency or outside officials to better understand system controls and to assure ourselves that we understand how the data are produced and any limitations associated with the data. We may also electronically test the data to see whether values in the data conform to agency testimony and documentation regarding valid values, or we may compare data to source documents. In addition to these steps, we often compare data with other sources as a way to corroborate our findings. For each new area in this report, specific information on data reliability is located in the related products. We provided drafts of our new area summaries to the relevant agencies for their review and incorporated these comments as appropriate. To examine the extent to which Congress and executive branch agencies have made progress in implementing the 908 actions in the approximately 325 areas we have reported on in previous annual reports on fragmentation, overlap, and duplication, we reviewed relevant legislation and agency documents such as budgets, policies, strategic and implementation plans, guidance, and other information between April 2019 and March 2020. We also analyzed, to the extent possible, whether financial or other benefits have been attained, and included this information as appropriate (see discussion below on the methodology we used to estimate financial benefits.) In addition, we discussed the implementation status of the actions with officials at the relevant agencies. Throughout this report, we present our counts as of March 2020 because that is when we received our last updates. The progress statements and updates are published on GAO’s Action Tracker. We used the following criteria in assessing the status of actions: In assessing actions suggested for Congress, we applied the following criteria: “addressed” means relevant legislation has been enacted and addresses all aspects of the action needed; “partially addressed” means a relevant bill has passed a committee, the House of Representatives, or the Senate during the current congressional session, or relevant legislation has been enacted but only addressed part of the action needed; and “not addressed” means a bill may have been introduced but did not pass out of a committee, or no relevant legislation has been introduced. Actions suggested for Congress may also move to “addressed” or “partially addressed” with or without relevant legislation if an executive branch agency takes steps that address all or part of the action needed. At the beginning of a new congressional session, we reapply the criteria. As a result, the status of an action may move from partially addressed to not addressed if relevant legislation is not reintroduced from the prior congressional session. In assessing actions suggested for the executive branch, we applied the following criteria: “addressed” means implementation of the action needed has been completed; “partially addressed” means the action needed is in development or started but not yet completed; and “not addressed” means the administration, the agencies, or both have made minimal or no progress toward implementing the action needed. Since 2011, we have categorized 80 actions as “other” and are no longer assessing these actions. We categorized 48 “other” actions as “consolidated or other.” In most cases, “consolidated or other” actions were replaced or subsumed by new actions based on additional audit work or other relevant information. We also categorized 32 of the “other” actions as “closed-not addressed.” Actions are generally “closed-not addressed” when the action is no longer relevant because of changing circumstances. To calculate the total financial benefits resulting from actions already taken (addressed or partially addressed) and potential financial benefits from actions that are not fully addressed, we compiled available estimates for all of the actions from GAO’s Action Tracker, from 2011 through 2019, and from reports identified for inclusion in the 2020 annual report, and linked supporting documentation to those estimates. Each estimate was reviewed by one of our technical specialists to ensure that estimates were based on reasonably sound methodologies. The financial benefits estimates came from a variety of sources, including our analysis, Congressional Budget Office estimates, individual agencies, the Joint Committee on Taxation, and others. Because of differences in time frames, underlying assumptions, quality of data and methodologies among these individual estimates, any attempt to generate a total will be associated with uncertainty that limits the precision of this calculation. As a result, our totals represent a rough estimate of financial benefits, rather than an exact total. For actions that have already been taken, individual estimates of realized financial benefits covered a range of time periods stretching from 2010 through 2029. To calculate the total amount of realized financial benefits that have already accrued and those that are expected to accrue, we separated those that accrued from 2010 through 2019 and those expected to accrue between 2020 and 2029. For individual estimates that span both periods, we assumed that financial benefits were distributed evenly over the period of the estimate. For each category, we summed the individual estimates to generate a total. To account for uncertainty and imprecision resulting from the differences in individual estimates, we present these realized savings to the nearest billion dollars, rounded down. There is a higher level of uncertainty for estimates of potential financial benefits that could accrue from actions not yet taken because these estimates are dependent on whether, how, and when agencies and Congress take our recommended actions, or due to lack of sufficiently detailed data to make reliable forecasts. As a result, many estimates of potential savings are notionally stated using terms like millions, tens of millions, or billions, to demonstrate a rough magnitude without providing a more precise estimate. Further, many of these estimates are not tied to specific time frames for the same reason. To calculate a total for potential savings, with a conservative approach, we used the minimum number associated with each term. To account for the increased uncertainty of potential estimates and the imprecision resulting from differences among individual estimates, we calculated potential financial benefits to the nearest $10 billion, rounded down, and presented our results using a notional term. This report is based upon work we previously conducted in accordance with generally accepted government auditing standards. Generally accepted government auditing standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix presents 18 new areas in which we found evidence of fragmentation, overlap, or duplication among federal government programs. GAO provided a draft of this report section to DOD for review and comment. In its response, DOD officials stated that Army Futures Command is taking the steps necessary to implement GAO’s recommendations, as reflected above. Army Modernization: Army Futures Command Should Take Steps to Improve Small Business Engagement for Research and Development. GAO-19-511. Washington, D.C.: July 17, 2019. GAO provided DOD with a draft of this report section for comment. DOD provided technical comments, which GAO incorporated as appropriate. DOD Utilities Privatization: Improved Data Collection and Lessons Learned Archive Could Help Reduce Time to Award Contracts. GAO-20- 104. Washington, D.C.: April 2, 2020. GAO provided a draft of this report section to SBA for review and comment. SBA stated it plans to continue to explore opportunities for collaboration with USDA and Treasury. SBA also provided technical comments, which GAO incorporated as appropriate. SBA Microloan Program: Opportunities Exist to Strengthen Program Performance Measurement, Collaboration, and Reporting. GAO-20-49. Washington, D.C.: November 19, 2019. GAO provided a draft of this report section to FinCEN for review and comment. In its comments, FinCEN continued to disagree with the recommendations and stated that no futures industry association had applied for BSA advisory group membership and that it advised CFTC staff on the areas that the National Futures Association should include as part of a request for direct BSA data access. GAO maintains that the recommendations are both valid, believes that FinCEN advising CFTC is a good first step, and will continue to monitor the implementation of these recommendations. Bank Secrecy Act: Agencies and Financial Institutions Share Information but Metrics and Feedback Not Regularly Provided. GAO-19-582. Washington, D.C.: August 27, 2019. GAO provided a draft of this report section to OMB for review and comment. OMB did not provide comments on this report section. DATA Act: Quality of Data Submissions Has Improved but Further Action Is Needed to Disclose Known Data Limitations. GAO-20-75. Washington, D.C.: November 8, 2019. DATA Act: Data Standards Established, but More Complete and Timely Guidance Is Needed to Ensure Effective Implementation. GAO-16-261. Washington, D.C.: January 29, 2016. DATA Act: Progress Made in Initial Implementation but Challenges Must be Addressed as Efforts Proceed. GAO-15-752T. Washington, D.C.: July 29, 2015. GAO provided a draft of this report section to the Office of Management and Budget (OMB), CNCS, Education, HHS, DOL, and USAID for review and comment. In its response, HHS provided documentation in February 2020 about the actions it plans to take—as part of its implementation of the Foundations for Evidence-Based Policymaking Act—to address the two recommendations directed to it. GAO will monitor HHS’s actions, which GAO believes would likely address its recommendations, if effectively implemented. CNCS and DOL informed GAO they had no comments on this report section. USAID provided technical comments, which GAO incorporated, as appropriate. OMB and Education did not provide comments. Evidence-Based Policymaking: Selected Agencies Coordinate Activities, but Could Enhance Collaboration. GAO-20-119. Washington, D.C.: December 4, 2019. GAO provided a draft of this report section to IRS and DOL for review and comment. In their March 2020 responses, IRS and DOL stated that they agreed to formalize collaboration on IRA prohibited transactions. The new information sharing process will be documented in forthcoming DOL procedures. In addition, IRS provided technical comments, which GAO incorporated as appropriate. Individual Retirement Accounts: IRS Could Better Inform Taxpayers About and Detect Noncompliance Related to Unconventional Assets, GAO-20-210. Washington, D.C.: January 27, 2020. Individual Retirement Accounts: Formalizing Labor’s and IRS’s Collaborative Efforts Could Strengthen Oversight of Prohibited Transactions, GAO-19-495. Washington, D.C.: June 7, 2019. GAO provided a draft of this report section to IRS for review and comment. IRS did not provide comments on this report section. Taxpayer Information: IRS Needs to Improve Oversight of Third Party Cybersecurity Practices. GAO-19-340. Washington, D.C.: May 9, 2019. GAO provided a draft of this report section to IRS for review and comment. In February 2020, IRS sent GAO a response saying the agency is working to eliminate conditions that inhibit the agency’s ability to identify abusive tax schemes by evaluating existing database project codes to link data across audit divisions and improve the analysis of data monitoring and mining. Tax-Law Enforcement: IRS Could Better Leverage Existing Data to Identify Abusive Schemes Involving Tax-Exempt Entities. GAO-19-491. Washington, D.C.: September 5, 2019. GAO provided a draft of this report section to the Department of Health and Human Services (HHS) for review and comment. HHS commented that ASPR will explore funding opportunities to support an exercise of its federal patient movement framework with its support agencies. In addition, HHS officials stated that ASPR would continue to support interagency liaison officers to provide updates on available resources. While GAO agrees that HHS should continue this practice, the misalignment GAO identified underscores that this was not adequate during the response to Hurricanes Irma and Maria in the U.S. Virgin Islands and Puerto Rico. Moreover, ASPR officials acknowledged that more needs to be done to better understand the resources available. Finally, HHS commented that ASPR has implemented air transportation contracts to begin decreasing its reliance on DOD. GAO will continue to monitor the implementation of these recommendations. Disaster Response: HHS Should Address Deficiencies Highlighted by Recent Hurricanes in the U.S. Virgin Islands and Puerto Rico. GAO-19- 592. Washington, D.C.: September 20, 2019. GAO provided a draft of this report section to VA for review and comment. VA provided technical comments, which GAO incorporated as appropriate. VA Health Care: Veterans’ Use of Long-Term Care Is Increasing, and VA Faces Challenges in Meeting the Demand. GAO-20-284. Washington, D.C.: February. 19, 2020. GAO provided a draft of this report section to Coast Guard, through DHS, for review and comment. The Coast Guard, through DHS, provided technical comments, which GAO incorporated as appropriate. The Coast Guard did not agree with the recommendation in its November 2019 response to GAO’s draft report. At that time, DHS further stated that GAO’s conclusions illustrate a fundamental misunderstanding of the corresponding missions of Specialized Forces units. GAO continues to maintain that overlapping capabilities among units could indicate inefficiencies in how units are used as well as missed opportunities for use in others. In its technical comments provided in March 2020, the Coast Guard indicated that as of February 2020 it had not conducted the analysis necessary to fully identify potential overlap among the units. The Coast Guard stated that it is planning to begin analyzing the units this fiscal year. In line with GAO’s recommendation to analyze potential overlap in capabilities, the Coast Guard should include the cost savings of shutting down a unit from each Specialized Force type and explain the impacts. Coast Guard: Assessing Deployable Specialized Forces’ Workforce Needs Could Improve Efficiency and Reduce Potential Overlap or Gaps in Capabilities. GAO-20-33. Washington, D.C.: November 21, 2019. GAO provided a draft of this report section to DHS for review and comment. DHS provided technical comments, which GAO incorporated as appropriate. Southwest Border: Actions Needed to Address Fragmentation in DHS’s Processes for Apprehended Family Members. GAO-20-274. Washington, D.C.: Feb. 19, 2020. GAO provided a draft of this report section to DHS for review and comment. TSA provided technical comments, which GAO incorporated as appropriate. Transportation Security: DHS Should Communicate the National Strategy’s Alignment with Related Strategies to Guide Federal Efforts. GAO-20-88. Washington, D.C.: November 19, 2019. GAO provided a draft of this report section to DHS for review and comment. DHS officials said DHS has taken initial actions to address GAO’s recommendation, including updating the related Standard Operating Procedure. GAO believes this is a good first step and will continue to monitor the implementation of this recommendation. Surface Transportation: TSA Should Improve Coordination Procedures for Its Security Training Program. GAO-20-185. Washington, D.C.: November 20, 2019. GAO provided a draft of this report section to State for review and comment. In its February 2020 response, State did not comment specifically on whether it agreed with GAO’s revised recommendation. However, it reiterated its disagreement with aspects of the underlying GAO report’s objectives, scope, and methodology. GAO addressed this disagreement in detail in its report and maintains that GAO’s approach provided a reliable and reasonably comprehensive review of the results of U.S. assistance to the Northern Triangle toward achieving key U.S. objectives set forth in the Strategy. U.S. Assistance to Central America: Department of State Should Establish a Comprehensive Plan to Assess Progress toward Prosperity, Governance, and Security. GAO-19-590. Washington, D.C.: September 26, 2019. GAO provided a draft of this report section to Defense, Energy, and the Departments of Commerce (for the National Oceanic and Atmospheric Administration) and Health and Human Services (for the National Institutes of Health), as well as the National Science Foundation and OSTP for review and comment. The National Institutes of Health and the National Science Foundation provided technical comments, which GAO incorporated as appropriate. OSTP said it had no further comments, and the National Oceanic and Atmospheric Administration, Defense, and Energy did not provide comments. Federal Research: Additional Actions Needed to Improve Public Access to Research Results. GAO-20-81. Washington, D.C.: November 21, 2019. GAO provided a draft of this report section to USDA for review and comment. USDA officials took issue with the characterization of their nutrition education efforts as fragmented, stating that coordination must consider the legislative authority each program has to deliver nutrition education to meet the needs of program target populations and audiences. GAO agrees that a consideration of each program’s legislative authority is important. However, GAO believes that USDA could address the fragmentation GAO identified, which refers to the involvement of multiple USDA agencies and programs in administering the department’s nutrition education efforts, consistent with a consideration of program authority. USDA officials continue to agree that the department needs to improve coordination of its nutrition education efforts. USDA officials described initial actions the department has taken to address GAO’s recommendation, including establishing a nutrition education working group that represents agencies across the department and planning an intradepartmental workshop that will include a focus on nutrition education. In addition, USDA issued the USDA Science Blueprint to outline the department’s nutrition science implementation strategies and nutrition and health promotion objectives. GAO will continue to monitor implementation of this recommendation. Further, GAO will monitor the role of the nutrition education working group going forward and consider the extent to which it provides cross- department leadership for USDA’s nutrition education efforts. Nutrition Education: USDA Actions Needed to Assess Effectiveness, Coordinate Programs, and Leverage Expertise. GAO-19-572. Washington, D.C.: July 25, 2019. This appendix summarizes 11 new areas for Congress or executive branch agencies to consider taking action that could either reduce the cost of government operations or enhance revenue collections for the Treasury. This page is intentionally left blank. GAO provided a draft of this report section to DOD for review and comment. DOD commented that the department will continue to work on improving its monitoring and evaluation of its efficiency and reform initiatives. DOD Needs to Address Inefficiencies and Implement Reform across Its Defense Agencies and DOD Field Activities. GAO-18-592. Washington, D.C.: September 6, 2018. GAO provided a draft of this section to the DOD for review and comment. DOD did not provide comments on this report section. Depot Maintenance: DOD Should Adopt a Metric That Provides Quality Information on Funded Unfinished Work. GAO-19-452. Washington, D.C.: July 26, 2019. GAO provided a draft of this report section to Ginnie Mae for review and comment. Ginnie Mae did not have any comments on the draft but noted that it is working diligently on the recommendations. Ginnie Mae: Risk Management and Staffing-Related Challenges Need to Be Addressed. GAO-19-191. Washington, D.C.: April 3, 2019. GAO provided a draft of this report section to IRS for review and comment. IRS’s technical comments were incorporated above. Tax Debt Collection Contracts: IRS Analysis Could Help Improve Program Results and Better Protect Taxpayers. GAO-19-193. Washington, D.C.: March 29, 2019. GAO provided a draft of this report section to IRS for review and comment. In its response, IRS stated that it is working with Treasury on guidance to address third-party reporting on certain taxable transactions involving virtual currency. GAO will review this guidance when it is available. Virtual Currencies: Additional Information Reporting and Clarified Guidance Could Improve Tax Compliance. GAO-20-188. Washington, D.C: February 12, 2020. GAO provided a draft of this report section to CMS for review and comment. In its written comments, CMS provided an update on its actions to address the first recommendation, which GAO incorporated. CMS did not provide information on the second recommendation. GAO will continue to monitor CMS’s implementation of these recommendations. Medicaid Providers: CMS Oversight Should Ensure State Implementation of Screening and Enrollment Requirements. GAO-20-8. Washington, D.C.: October 10, 2019. GAO provided a draft of this report section to VA for review and comment. VA did not provide comments on this report section. Veterans Health Care: VA Needs to Improve Its Allocation and Monitoring of Funding. GAO-19-670. Washington, D.C.: September 23, 2019. GAO provided a draft of this report section to DOD for review and comment. DOD explained that current policy allows and encourages the use of open source software where it meets agency needs. In addition, DOD stated that GAO’s recommendations focus on DOD's role as a producer, rather than as a consumer, of open source software. A DOD official explained that it is not reasonable to conclude that the projected savings will result from the implementation of GAO’s recommendations. However, DOD entities can consume open source software that other DOD entities produce. GAO maintains that this very consumption of open source software developed elsewhere in DOD could reduce development costs and potentially produce overall cost savings. Information Technology: DOD Needs to Fully Implement Program for Piloting Open Source Software. GAO-19-457. Washington, D.C.: September 10, 2019. GAO provided a draft of this report section to DOD for review and comment. DOD did not provide comments on this report section. Defense Logistics Agreement: DOD Should Improve Oversight and Seek Payment from Foreign Partners for Thousands of Orders It Identifies as Overdue. GAO-20-309. Washington, D.C.: March 4, 2020. GAO provided a draft of this report section to CBP for review and comment. CBP provided technical comments, which GAO incorporated, as appropriate. Customs and Border Protection: Risk Management for Tariff Refunds Should Be Improved. GAO-20-182. Washington, D.C.: December 17, 2019. GAO provided a draft of this report section to Education for review and comment. Education provided technical comments, which GAO incorporated. Federal Student Loans: Education Needs to Verify Borrowers’ Information for Income-Driven Repayment Plans. GAO-19-347. Washington, D.C.: June 25, 2019. We are adding 88 new actions based on GAO reports that fall within the scope of 10 existing areas identified in prior annual reports. In March 2020, GAO identified 12 actions for the Navy to improve its acquisition practices and ensure ships can be efficiently sustained, potentially saving billions of dollars. addressed, one action has been partially addressed, and one action has not been addressed. See the Action Tracker for more information. GAO reported in March 2020 on challenges identifying, evaluating, and mitigating ship sustainment risks during the acquisition process for every new warship class—such as aircraft carriers and submarines—that, if fixed, could save billions of dollars. GAO found 150 examples of systemic maintenance problems, such as failed engines and non-functional plumbing. To correct just 30 percent of these problems, GAO found that it would cost the Navy $4.2 billion. Many of these problems could have been prevented with some attention to future maintenance concerns when designing and building the ships. GAO also found that the Navy underestimated the costs to maintain some ships by $130 billion. GAO made 11 recommendations to help the Navy focus on maintenance earlier and one suggestion to Congress to enhance oversight. New Actions: GAO recommended in March 2020 that the Department of Defense (DOD) improve its policy for setting sustainment requirements and the Navy then revisit its requirements to comply with the new policy. GAO also recommended that DOD and the Navy take steps to improve sustainment in the acquisition process. GAO also asked Congress to consider developing an oversight mechanism for evaluating shipbuilding programs sustainment cost estimate growth during the acquisition process. While GAO cannot precisely estimate the financial benefits from these actions, if the Navy could eliminate some of the sustainment problems and even 1 percent of the maintenance cost growth GAO identified, it could amount to billions of dollars in savings. Agency Comments and GAO’s Evaluation: DOD agreed with eight and partially agreed with three recommendations. GAO provided a draft of this report section to DOD for comment. DOD provided technical comments, which GAO incorporated as appropriate. In September 2019, GAO identified two new actions to improve the Department of the Interior’s valuations of offshore oil and gas resources, each of which could increase the amount of revenue collected by tens of millions of dollars annually. partially addressed, and two actions have not been addressed. See the Action Tracker for more information. Production of oil and gas in federal waters generated about $90 billion in revenue from 2006 through 2018 including from industry bids for leasing rights. However, GAO found that the Department of the Interior’s (Interior) Bureau of Ocean Energy Management (BOEM) undervalues federal offshore oil and gas resources, leading it to collect less bid revenue than it otherwise would. Specifically, the bureau (1) forecast unreasonably high levels of depreciation on lease value between lease sales, which lowered bid revenue by about $873 million from March 2000 through June 2018; and (2) adjusted some valuations downward to justify accepting bids, which lowered bid revenue by about $567 million over the same time period. New Actions: The bureau Director should (1) enlist an independent third party to examine the extent to which the bureau's depreciation forecasts assure the receipt of fair market value, and make changes as appropriate; and (2) take steps to ensure that the bureau’s bid valuation process is not biased toward adjusting valuations downward. In its comments on the report, Interior disagreed with the first recommendation and partially agreed with the second, disagreeing with GAO’s characterization of BOEM’s delayed valuations and valuation process, respectively. GAO maintains that taking each of the recommended actions would better ensure a fair return on the sale of offshore oil and gas leases by better ensuring BOEM’s thresholds for accepting bids are sound and unbiased. Agency Comments and GAO’s Evaluation: GAO provided a draft of this report section to Interior for review and comment. In its March 2020 response, Interior indicated that (1) although it disagrees with the first recommendation, it will conduct an in-house review and have it peer-reviewed; and (2) it now agrees with the second recommendation. In April 2019, GAO identified two actions the U.S. Army Corps of Engineers and U.S. Coast Guard can take to improve fragmented interagency coordination of lessons learned following disasters. and two actions have been partially addressed. See the Action Tracker for more information. GAO found that the U.S. Army Corps of Engineers (USACE) and U.S. Coast Guard (USCG) had fragmented approaches to identifying interagency challenges and lessons learned related to disaster contracting, resulting in these findings not being communicated to the Federal Emergency Management Agency’s (FEMA) Emergency Support Function Leadership Group—the group tasked with identifying interagency lessons learned following disasters. FEMA officials stated that it is up to each agency to elevate issues to the group; however, GAO found that neither USACE nor USCG had formal processes for doing so. Identifying and communicating lessons learned would help better manage fragmentation and enhance agencies’ abilities to address weaknesses in disaster response. New Actions: To help address fragmentation and ensure that challenges are communicated across departments, GAO recommended in April 2019 that the Secretary of the Army should direct the Commanding General of USACE to, and that the Commandant of USCG should, establish formal processes to solicit input from officials directly involved in the agencies' response and recovery following a disaster and to share that input with the Emergency Support Function Leadership Group. Agency Comments and GAO’s Evaluation: USACE and USCG concurred with GAO’s recommendations and planned to implement them this year. GAO provided a draft of this report section to USACE and USCG for review and comment. USCG said it is reviewing lessons learned and the after-action reporting process to update its policy. USACE indicated it updated its guidance to incorporate specific steps to communicate lessons learned with FEMA’s Emergency Support Function Leadership Group and that the guidance would be finalized spring 2020. GAO will continue to monitor the implementation of these recommendations. In March 2020, GAO identified a new action to improve the Department of Housing and Urban Development’s working capital fund and better position it to achieve over $1 million in previously identified potential annual savings. Three actions have not been addressed, and one action has been partially addressed. See the Action Tracker for more information. Housing and Urban Development’s (HUD) Working Capital Fund (WCF) provides a mechanism to centralize and fund federal shared services used across offices and agencies within HUD. One of the WCF’s goals is to support the efficient delivery of goods and services. GAO found that HUD does not assess the results of the WCF’s business process analyses, which are used to identify opportunities for efficiencies. For example, these analyses identified actionable ways to reduce high volumes of transactions for certain services, such as calls to help desks to manually reset passwords, which contribute to increased costs. Assessing the results of these analyses would help HUD better understand how the WCF’s efforts contribute to its goal. New Action: GAO recommended that the Secretary of HUD, in conjunction with the Office of the Chief Financial Officer, should ensure that the results of the business process analyses are assessed to better determine how these analyses contribute to its goal of efficient delivery of goods and services. While GAO cannot estimate the potential savings that would result, taking this action could help the WCF achieve over $1 million in potential annual savings already identified by WCF recommendations and to identify additional potential savings. Agency Comments and GAO’s Evaluation: GAO provided a draft of this report section to HUD for review and comment. HUD agreed and said it would address this recommendation in 2020, including adding the results of the business process analyses to its performance measures. In June 2019, GAO identified a new action that could improve oversight of disaster relief funds and long- standing problems of improper payments, which could result in significant cost savings. consolidated, one action has been partially addressed, and two actions have not been addressed. See the Action Tracker for more information. Agencies must distribute disaster relief aid quickly following hurricanes, wildfires, or other natural disasters, but quickly spending billions of dollars can increase the risk of improper payments. In June 2019, GAO reported that one of six selected agencies did not submit required internal control plans to Congress for funds appropriated following the 2017 disasters. Of the five agencies that did submit the required plans, four were not timely and all lacked necessary information, such as how they met Office of Management and Budget (OMB) guidance and federal internal control standards. These issues were caused, in part, because OMB lacked an effective strategy for helping agencies develop internal control plans for overseeing these funds. New Action: GAO recommended in June 2019 that the Director of OMB, after consulting with key stakeholders, should develop a strategy for ensuring that agencies communicate sufficient and timely internal control plans for effective oversight of disaster relief funds. Agency Comments and GAO’s Evaluation: OMB disagreed with this recommendation and stated that it does not believe timeliness and sufficiency of internal control plans present material issues that warranted OMB action; however, GAO continues to believe that future internal control plans could serve as a critical transparency tool for controls over disaster funds. GAO provided a draft of this report section to OMB for review and comment. In its response, OMB continued to disagree that this recommendation is needed. GAO believes this action is needed for oversight of disaster funds. In January 2020, GAO identified three new actions to help the Internal Revenue Service prevent refund fraud associated with identity theft. If implemented, these actions could potentially save millions of dollars. more information. Business identity theft refund fraud (business IDT) occurs when thieves create, use, or try to use a business’ identifying information to claim a tax refund. Between January 2017 and August 2019, the Internal Revenue Service’s (IRS) fraud detection tools helped prevent $384 million from being paid to fraudsters. However, GAO found IRS could do more to combat business IDT. In January 2020, GAO found that, inconsistent with leading practices, IRS had not designated an entity to design and oversee business IDT fraud risk management efforts, conducted a fraud risk assessment, or developed a fraud risk profile to document the results of its risk assessment. Addressing these issues could help IRS identify and implement more effective controls to detect and prevent business IDT. While GAO cannot precisely estimate the financial benefits associated with this action, even a 1 percent increase in fraud prevention could amount to millions in financial benefits. New Actions: In January 2020, GAO recommended that consistent with leading practices, IRS (1) designate a dedicated entity to oversee agency-wide business IDT efforts; (2) develop a fraud risk profile for business IDT; and (3) document and implement a strategy for addressing fraud risks identified in its fraud risk profile. Agency Comments and GAO’s Evaluation: GAO provided a draft of the January 2020 report to IRS for review and comment. IRS generally agreed, but did not provide details on the actions it plans to take to address these recommendations. IRS also did not provide comments on this report section. In January 2020, GAO identified a new action to help the Department of Veterans Affairs assess duplication in its medical supply program. partially addressed. See the Action Tracker for more information. In January 2020, GAO found that the Medical Surgical Prime Vendor program duplicates parts of the Department of Veterans Affairs (VA) Federal Supply Schedule program. VA spends billions of dollars annually on procurement of medical supplies to support care for veterans at its 170 medical centers but has not assessed whether its efforts are duplicative. VA procures medical supplies through both its own Medical Surgical Prime Vendor program and through the Federal Supply Schedule program—a government-wide program, parts of which the General Services Administration has long delegated to VA. However, VA has not assessed whether duplication across these programs is necessary or if efficiencies could be gained. GAO cannot estimate the savings that might be associated with this action because such savings will be dependent on whether, when, and how VA takes action. New Action: GAO recommended that the Secretary of Veterans Affairs should take steps to assess duplication between VA’s Medical- Surgical Prime Vendor and Federal Supply Schedule programs to determine if this duplication is necessary or if efficiencies can be gained. Agency Comments and GAO’s Evaluation: VA agreed with this recommendation. GAO provided a draft of this report section to VA for review and comment. VA provided technical comments which GAO incorporated as appropriate. In November 2019, GAO identified two new actions to help reduce the risk of duplicate funding in emergency relief assistance for transit agencies. addressed, two actions have been partially addressed, and one action has not been addressed. See the Action Tracker for more information. In 2017, Hurricanes Harvey, Irma, and Maria caused hundreds of millions of dollars in damage to U.S. public transit facilities. Both the Federal Transit Administration (FTA) and the Federal Emergency Management Agency (FEMA) have the authority to provide disaster assistance funding to transit agencies, but FTA has primary responsibility if it receives an appropriation from Congress for its Public Transportation Emergency Relief program. FTA did not receive an appropriation until roughly 6 months after the first hurricane’s landfall, thus transit agencies could initially apply to FEMA for assistance. In November 2019, GAO found that although FTA and FEMA coordinated efforts, both agencies still approved about $35,000 to one applicant for the same expenses in 2019. While the amount of funding in question was relatively small, without addressing the challenge of identifying transit expenses in FEMA applications, FTA and FEMA will continue to face the risk that both agencies will approve funding for the same expense in the future. New Actions: GAO recommended in November 2019 that FTA and FEMA identify and develop controls, such as methods to more easily identify transit expenses within applications FEMA receives, to address the risk of duplicate funding. Agency Comments and GAO’s Evaluation: The Department of Transportation (DOT) and the Department of Homeland Security (DHS) agreed with this recommendation and outlined steps they plan to take to address it. GAO provided a draft of this report section to DOT and DHS for review and comment. DOT said it did not have comments on this report section. DHS provided technical comments, which GAO incorporated as appropriate. In April 2019, GAO identified 28 new actions to help agencies save millions of dollars through better planning and implementation of cloud-based computing solutions. Two actions have been addressed. See the Action Tracker for more information including applicable agencies. Beginning in 2012, federal agencies were required to assess all IT investments for cloud computing services, and from 2014 to 2018, agencies reported $291 million in cloud-related savings. For example, agencies reported saving as much as $15 million migrating email systems to cloud services. However, GAO reported that 12 of the 16 agencies reviewed had not completed their assessments and that savings data were unavailable for 84 percent of the 488 cloud investments reviewed. Improving the assessment of investments for cloud services and tracking related savings can help agencies make better decisions regarding cloud acquisitions and potentially save millions of dollars from implementing cloud services. New actions: GAO made 28 recommendations in April 2019 to all 16 agencies, including that (1) 12 agencies should complete an assessment of all of their IT investments for suitability for migration to a cloud computing service, in accordance with OMB guidance; and (2) 16 agencies should ensure that their respective Chief Information Officers establish a consistent and repeatable mechanism to track savings and cost avoidances from the migration and deployment of cloud services. Fourteen agencies agreed with all recommendations, the Department of the Treasury neither agreed nor disagreed, and the Department of Defense agreed with the recommendation on completing assessments, but not with the recommendation on tracking savings. Agency comments and GAO’s evaluation: GAO provided a draft of this report section to the 16 agencies for review and comment. One agency agreed, 13 agencies had no comments, and two neither agreed nor disagreed. Additionally, seven of the 16 agencies are taking actions to address GAO’s recommendations. In April 2019, GAO identified 36 new actions to help federal agencies meet the Office of Management and Budget’s data center consolidation and optimization goals, resulting potentially in hundreds of millions of dollars in savings. original two actions in this area. See the Action Tracker for more information. Federal agencies operate thousands of data centers and, since 2010, have been required to close unneeded facilities and improve the performance of the remaining centers. This effort is currently known as the Data Center Optimization Initiative (DCOI). Since 2010, agencies have closed 6,250 centers and reported $4.2 billion in savings. However, only two of 24 agencies in GAO’s review planned to fully meet the Office of Management and Budget’s (OMB) September 2018 government-wide optimization goals, such as determining how much time data servers sit unused and how effectively data centers use power. New actions: GAO made 36 recommendations in April 2019 to 22 of the 24 agencies in its review, including that (1) 11 agencies should meet DCOI’s data center closure targets; (2) four agencies should meet DCOI’s data center-related cost savings targets and one should identify additional cost savings opportunities; and (3) 20 agencies should meet DCOI’s data center optimization metric targets. While GAO cannot precisely estimate the potential savings of taking these actions, combined estimates from agencies for similar prior actions exceeded $100 million per year, suggesting potential for hundreds of millions of dollars in additional savings over time. In June 2019, OMB significantly revised DCOI’s goals and performance measures and GAO continues to monitor agencies’ progress against these new targets. Agency comments and GAO’s evaluation: GAO provided a draft of this report section to 22 agencies for review and comment. Two agencies agreed, seven neither agreed nor disagreed, and 13 agencies had no comments. Additionally, two agencies have taken action to fully address the recommendations and the remaining 20 agencies are taking actions to address GAO’s recommendations. In our 2011 to 2020 annual reports, we directed 110 actions to Congress, of which 58 remain open. Thirty-five have been addressed and 17 were closed as not addressed or consolidated. Of the 58 open congressional actions, 15 are partially addressed and 43 are not addressed, as of March 2020 (see figure 10). The tables below have more information on the 58 open congressional actions. Our Action Tracker downloadable spreadsheet (available in XLSX or CSV formats) has information on all actions. This appendix provides additional information on the federal programs or other activities related to the new areas of fragmentation, overlap, duplication, cost savings, or revenue enhancement discussed in this report, including budgetary information when available. “Programs” may include grants, initiatives, centers, loans, and other types of assistance or projects. This information can provide useful context for the issues we identified, but limitations should be noted. It is not always possible to report budgetary information at the specific program or activity level because agency budgets are not organized by programs, but rather by appropriations accounts. In those instances, we reported the most reliable and available data for the most recent fiscal year or we did not report budgetary information. Further, because this report discusses various programs or activities, each table may report different types of budgetary information, such as obligations, collections, or outlays. Because of the limitations described above, the budgetary information reported in this appendix should not be totaled and does not represent potential cost savings for all programs.", "summary": "The federal government has made an unprecedented financial response to the COVID-19 pandemic. At the same time, opportunities exist for achieving billions of dollars in financial savings and improving the efficiency and effectiveness of a wide range of federal programs in other areas. Congress included a provision in statute for GAO to identify and report on federal programs, agencies, offices, and initiatives—either within departments or government-wide—that have duplicative goals or activities. GAO also identifies areas that are fragmented or overlapping and additional opportunities to achieve cost savings or enhance revenue collection. This report discusses the new areas identified in GAO’s 2020 annual report—the 10th report in this series; the progress made in addressing actions GAO identified in its 2011 to 2019 reports; and examples of open actions directed to Congress or executive branch agencies. To identify what actions exist to address these issues, GAO reviewed and updated prior work, including matters for congressional consideration and recommendations for executive action. GAO’s 2020 annual report identifies 168 new actions for Congress or executive branch agencies to improve the efficiency and effectiveness of government in 29 new mission areas and 10 existing areas. For example: The Department of Defense could potentially save hundreds of millions of dollars annually by accurately measuring and reducing excess funded, unfinished work at military depots. The Centers for Medicare & Medicaid Services could better ensure that states implement Medicaid provider screening and enrollment requirements, which could potentially save tens of millions of dollars annually . The Government National Mortgage Association could enhance the efficiency and effectiveness of its operations and risk management and reduce costs or enhance federal revenue by tens of millions of dollars annually . The Internal Revenue Service should establish a formal collaborative mechanism with the Department of Labor to better manage fragmented efforts and enhance compliance for certain individual retirement accounts that engaged in prohibited transactions, and thereby potentially increase revenues by millions of dollars . Improved coordination and communication between the Department of Health and Human Services’ Office of the Assistant Secretary for Preparedness and Response and its emergency support agencies—including the Federal Emergency Management Agency and Departments of Defense and Veterans Affairs—could help address fragmentation and ensure the effective provision of public health and medical services during a public health emergency. The Department of Education should analyze data and use it to verify borrowers’ income and family size information on Income-Driven Repayment plans to safeguard the hundreds of billions of dollars in federal investment in student loans and potentially save more than $2 billion . The Internal Revenue Service could increase coordination among its offices to better manage fragmented efforts to ensure the security of taxpayer information held by third-party providers. GAO identified 88 new actions related to 10 existing areas presented in 2011 through 2019 annual reports. For example: The Department of the Navy could achieve billions of dollars in cost savings by improving its acquisition practices and ensuring that ships can be efficiently sustained. The Office of Management and Budget could improve oversight of disaster relief funds and address government-wide improper payments, which could result in significant cost savings. The U.S. Army Corps of Engineers and the U.S. Coast Guard could better identify and communicate lessons learned in contracting following a disaster to improve fragmented interagency coordination. Significant progress has been made in addressing many of the 908 actions that GAO identified from 2011 to 2019 to reduce costs, increase revenues, and improve agencies’ operating effectiveness. As of March 2020, Congress and executive branch agencies have fully or partially addressed 79 percent of all actions (721 of 908 actions)—57 percent (519 actions) fully addressed and 22 percent (202 actions) partially addressed. This has resulted in approximately $429 billion in financial benefits. About $393 billion of these benefits accrued between 2010 and 2019, and $36 billion are projected to accrue in future years. This is an increase of $166 billion from GAO’s 2019 annual report. These are rough estimates based on a variety of sources that considered different time periods and utilized different data sources, assumptions, and methodologies. While Congress and executive branch agencies have made progress toward addressing actions that GAO has identified since 2011, further steps are needed. GAO estimates that tens of billions of additional dollars could be saved should Congress and executive branch agencies fully address the remaining 467 open actions, including the new ones identified in 2020. Addressing the remaining actions could lead to other benefits as well, such as increased public safety, and more effective delivery of services. For example:", "document_type": "gao"}
{"report": "Different types of organizations can serve as representative payees, from residential facilities where beneficiaries live, to organizations that only manage individuals’ Social Security retirement, disability, or other benefits. SSA’s organizational payees include social service agencies, mental institutions (federal, state or local, non-profit, private), non-mental institutions (federal, state or local, non-profit, private), financial organizations, and entities represented by public officials (such as public guardians, officers of the court, and other similar positions). For certain oversight purposes, such as periodic reviews, SSA categorizes organizational payees into several groups, including: (1) fee- for-service organizations, which charge beneficiaries a monthly fee for expenses incurred in providing services; (2) organizations that serve 50 or more beneficiaries and do not charge a fee for their services, referred to in this report as “high-volume”; (3) organizations that serve fewer than 50 beneficiaries and do not charge a fee for their services, referred to in this report as “low-volume”, and (4) state mental institutions participating in the State Onsite review program. SSA data from fiscal year 2018 indicate that the vast majority (86 percent) of organizational payees are low-volume payees, which serve 34 percent of beneficiaries (see fig. 1). Organizational payees decide how to spend beneficiaries’ funds, but must do so for the beneficiary’s use and benefit in a manner the payee determines to be in the beneficiary’s best interest. SSA considers it acceptable if the funds are used for the beneficiary’s current maintenance, which includes the costs of food, shelter, clothing, and medical care. After the representative payee has used the funds consistent with these guidelines, any remaining amounts must be conserved or invested on behalf of the beneficiary. Organizational payees are responsible for keeping records of SSA payments made to them on behalf of each beneficiary and the expenditures for each beneficiary. All organizational payees participate in onsite reviews and—except for state mental institutions participating in State Onsite reviews—are also required to file an annual accounting form to show how benefit payments were used and any amounts that were saved. Organizational payees also are required to notify SSA of certain changes or situations, including: changes that may affect the beneficiary’s eligibility for benefits or the benefit amounts, such as when the beneficiary, or the beneficiary’s spouse, dies; and when the beneficiary moves or is unable to be contacted or located, starts or stops working, or no longer needs a payee; when organizational payees learn that one or more of their employees has stolen a beneficiary’s funds or determines they can no longer serve as the payee for any reason; and when payees establish an account that mingles funds from multiple beneficiaries in one account—referred to as collective accounts— because these accounts must be approved before use. The process for administering the representative payee program is guided largely by requirements in statute and SSA regulations, which SSA communicates through its Program Operations Manual System (POMS). Recent changes to the program—including the new onsite review process—reflect requirements established by the Strengthening Protections for Social Security Beneficiaries Act of 2018. For example, the Commissioner of SSA must now: (1) reassess representative payee selection and replacement policies, (2) award annual grants (totaling at least $25 million nationwide) to each state’s protection and advocacy system to conduct onsite reviews of representative payees, and (3) award annual grants to a national association that can provide state protection and advocacy systems with training, technical assistance, and administrative oversight. In addition, the Act requires SSA to present the results of reviews—including information on representative payees’ misuse of benefits—in an annual report to Congress. SSA administers and manages the representative payee program through three dedicated data systems: The Electronic Representative Payee System (eRPS) is the system used to process payee applications; record poor payee performance; process changes (such as new addresses); and document misuse allegations, significant information about the payee, and why applications were approved or denied. The Electronic Representative Payee Accounting (eRPA) system is used to capture and review annual accounting forms that all organizational payees except state mental institutions participating in SSA’s State Onsite review program, are required to submit for each beneficiary they represent. Field office staff also use this system to track progress in resolving problems identified during reviews of the form, such as representative payees failing to submit complete information. The newly created Representative Payee Monitoring Tool, which is used to track and oversee the updated onsite review process. SSA operates the representative payee program primarily through its network of field offices. Field offices review and approve organizations’ applications to become representative payees, serve as the point of contact when organizations report changes to beneficiary or organization information, and play a role in monitoring and overseeing representative payees. SSA policy describes the required process for designating a representative payee for a beneficiary whom SSA staff have determined to be incapable of managing his or her benefits. First, organizations apply to serve as a payee for specific individuals. Second, SSA staff review applications to assess if the organization is qualified to serve as a payee and is the most suitable payee for the individual beneficiary. Additional qualifications are assessed when organizations apply to collect fees for their payee services. For example, SSA requires that all fee-for- service applicants have already served at least five beneficiaries for a full calendar month or more, and that non-governmental agencies be licensed and bonded. Once approved, organizational payees are subject to ongoing SSA oversight. SSA reviews annual accounting forms from organizational payees on each of the beneficiaries they represent. The accounting forms are used to monitor how the payee spent or saved benefits on behalf of the beneficiary; identify situations where payment to a payee may no longer be appropriate; or determine if the payee is no longer suitable. In addition to reviewing accounting forms annually, every 3 years SSA must review collective accounts established by organizational payees. Whether or not additional oversight in the form of an onsite review is provided, and the frequency of that oversight, generally depends on the organizations’ characteristics (see table 1). Certain types of organizational payees—such as high-volume, fee-for-service, and some state mental institutions—receive onsite reviews every 3 or 4 years. Low- volume organizational payees do not receive periodic reviews; rather SSA selects some of these payees for onsite reviews based on their likelihood of misusing beneficiaries’ funds and may target additional organizational payees because of an event that raises a question about the payee’s performance or suitability or because a protection and advocacy grantee thinks that a review is warranted. SSA’s purpose in conducting onsite reviews is to: (1) ensure organizational payees perform their duties satisfactorily, (2) deter misuse, (3) keep lines of communication open between the organizational payee and the servicing field office, (4) reinforce to the organizational payee their duties and responsibilities, and (5) proactively address the needs of organizational payees. We identified several gaps in SSA’s process for approving organizational payees, including insufficient detail in SSA’s policies, insufficient documentation of suitability decisions, and absence of background or credit checks on most organizational payees—gaps that may increase the risk of approving an unsuitable payee. We also identified challenges that field offices may face when approving replacement representative payees, such as a lack of local organizational payees and difficulty locating some beneficiaries. When an organization applies to serve as a payee, SSA’s policy stipulates that field office staff evaluate whether the organization is suitable. All payee applicants—individuals and organizations—are subject to the same general suitability factors, and organizations are evaluated on an additional set of suitability factors. Organizations are generally evaluated on the same suitability factors, whether they are a first-time applicant, or applying to serve an additional beneficiary. Additional requirements apply to organizations applying to collect fees (see appendix II). Two factors used to determine the suitability of organizations that are applying to be representative payees are straight-forward: (1) whether the payee agrees to receive benefits via direct deposit and (2) whether the payee uses protected accounts for beneficiary funds. However, other suitability factors are more complex, such as whether the applicant: demonstrates sound financial management policies (i.e., has a history of being current in its own financial obligations), demonstrates effective internal communication (i.e., good communication between case management and financial management components), has adequate recordkeeping systems to ensure that the client’s needs are met and benefits are properly administered. We found that SSA’s policies on how to evaluate more complex issues do not provide sufficient detail to ensure staff can fully assess an organization’s suitability. Staff at one of the eight field offices we visited told us the policies can leave room for interpretation, and staff at three field offices use additional guidance developed by field and regional offices that elaborates on how to assess some of the more complex issues in SSA’s policies. For example, SSA’s policy on what constitutes sound financial management states that an organization should have a history of being current in its own financial obligations. However, it generally does not provide direction on how to verify that an organization meets that requirement. Moreover, the policy lacks details on what staff should do to conduct a deeper assessment of an organization’s financial management practices if they think further assessment is warranted. Similarly, SSA policy directs staff to consider whether an organization has effective internal communication, which it defines as good communication between the organization’s case management and financial management components. However, SSA’s policy does not specify what actions constitute effective communication, such as the frequency and method of communication, type of information to be shared, and time frames for transmitting information. According to federal internal control standards, agencies should establish policies to document responsibilities for a process’s objectives and related risks and communicate these policies to personnel so they can fulfill their assigned responsibilities. Although SSA officials were able to point us to sections of agency policy that went into more detail about some of these complex topics, these policies pertain only to the few organizations that are applying to collect fees. In the absence of specific guidance on how to consider factors when assessing the suitability of all organizational payee applicants, SSA staff may be approving some of them without a complete picture of their financial health and ability to be good stewards of vulnerable beneficiaries’ money. According to central office officials, regions are generally given leeway to create their own supplemental guidance documents based on SSA policy to assist with training—documents that may also serve as resources to help staff interpret SSA policy. Officials in field and area offices told us this supplemental guidance is generally made available to staff on internal websites maintained by area or regional offices. Staff in three field offices told us they use supplemental guidance to evaluate organizations. For example, staff in one field office told us they use a supplemental list of questions to interview organizational payee applicants. These supplemental interview questions address some suitability factors in greater detail than SSA policy. For example: SSA policy directs staff to consider whether the organization “has adequate staff and resources to serve its clients.” The supplemental guidance from a regional office includes five questions on the number, type, relationships, and responsibilities of the staff; training and skills of staff dealing with finances; and documentation. SSA policy directs staff to consider whether the organization “has adequate recordkeeping systems to ensure that the client’s needs are met and benefits are properly administered.” The supplemental guidance from a regional office includes nine questions on the systems, records, procedures, and safeguards related to recordkeeping. Staff in another field office told us they created a desk guide on a range of topics related to individual and organizational payees that includes supplemental guidance documents and excerpts from SSA policy. The desk guide is a reference for all employees that work on payee issues and is also used to train new employees. However, SSA lacks a process to ensure that supplemental guidance is reviewed for compliance at the national level and that such guidance is updated by the regional office in a timely manner. Officials told us that because all regions are expected to follow SSA policy, central office staff only review supplemental guidance when the regions request it. Furthermore, SSA central office officials told us that although there is a protocol for communicating policy updates to regional, area, and field office staff, it is up to regions to refresh their own guidance. These officials did not know how long it takes regions to incorporate policy changes into regional guidance documents. As a result, field offices may be using supplemental guidance that has not been updated to reflect policy changes. For example, in a desk guide we reviewed, we identified a policy excerpt that was not the most recent version of that policy. Federal internal control standards stipulate that management should periodically review policies and procedures for continued relevance and effectiveness. Without processes to ensure that supplemental guidance documents are reviewed for compliance or updated in a timely manner when policy changes, decisions to approve organizational payees may be made inconsistently across different regions and field offices. SSA officials told us in May 2019 that they are currently reevaluating the agency’s representative payee approval policies and procedures based on feedback gathered through a forum hosted in September 2018 by the Social Security Advisory Board and in response to a Federal Register notice published in December 2018. However, SSA did not provide additional information on the nature, scope, or timeframes of this effort. SSA policy requires field office staff to document their assessment of an applicant’s suitability as a payee and the rationale for deciding to approve or deny an application. In addition, before approving a payee in eRPS, the system SSA uses to manage representative payee information, field office staff are to enter notes in accordance with the eRPS user guide. Specifically, staff are directed to document their determination regarding the beneficiary’s capability to manage their own finances and the organization’s suitability as a payee for the beneficiary. In certain situations, SSA policy directs staff to enter an additional note to document the relationship between the beneficiary and the payee. However, we found that staff in field offices we visited did not always fully document their decisions before approving organizational payees for the first time. Specifically, of the 21 first-time application files we reviewed, 16 did not contain a note about the organization’s suitability. Of the five files that did contain such notes, three provided limited detail. For example, two of the approved applications contained a note documenting that the beneficiary currently lived in the facility applying to serve as payee. However, notes in these two approved applications did not include any details regarding the prospective payee’s suitability, such as information about the facility or organization itself. Moreover, in two cases where the payee was a creditor for the beneficiary, we found that SSA staff had not documented why they approved these payees even though they were creditors for the beneficiary. Applicants who are creditors for beneficiaries are generally prohibited from serving as payees. Although exceptions are allowed in certain situations—such as when the organization is a care facility licensed or certified by the state, poses no risk to the beneficiary, and whose financial relationship with the beneficiary presents no substantial conflict of interest—staff are required to document why a creditor was selected as the payee. Although being a creditor could affect a payee’s suitability, we found that field office staff had not recorded information about why they selected these two creditors as the beneficiaries’ payees. We found that SSA staff might not fully document their decisions to approve organizational payees in part because eRPS, the system SSA uses to process payee applications, lacks safeguards for certain information entered into the system. As previously noted, staff use eRPS notes to document their assessment of: the beneficiary’s capability, the payee applicant’s suitability, and, in some cases, the beneficiary- payee relationship. However, while eRPS prevents field office staff from approving a payee without first documenting their assessment of a beneficiary’s capability in a note, this automated safeguard does not extend to the other note type. According to federal internal control standards, agencies should clearly document significant events so that that they are available for examination, and design their information systems to obtain and process information that responds to the agency’s objectives and risks. Because eRPS allows SSA staff to approve a payee without fully documenting the decision, SSA staff may not be able to reference that information to inform future decisions about the organizational payee. Specifically, SSA staff will not be as well-prepared to make fully informed decisions about an organizational payee’s continuing eligibility, or whether the organizational payee should be approved to manage benefits for additional vulnerable beneficiaries. This creates a risk that SSA staff may unwittingly approve an inappropriate organizational payee to serve other beneficiaries. SSA uses two types of external screening—background and credit checks—to identify potential concerns regarding the suitability of certain payee applicants. Whether such checks are required depends on the type of applicant, but most organizational payees do not receive either check. Background checks for individual representative payee applicants: According to law and SSA policy, staff should conduct background checks on individual payee applicants to determine if they have a criminal history that would disqualify them from serving as a payee. As part of the background check, policy directs staff to use applicant interviews and tools embedded in eRPS to gather information about the individual payee applicant’s criminal history, including prison time or unsatisfied felony warrants. Unless the payee is exempted by SSA policy, SSA staff will request the payee’s permission to conduct a background check and, if permission is granted, will then obtain a criminal report from eRPS.32, Credit checks for some fee-for-service applicants: According to SSA policy, staff are directed to obtain and review a credit report from Dun & Bradstreet for all non-governmental organizational payees that are applying to collect fees for payee services. These credit reports include information on bankruptcies, pending or completed legal judgments, liens, payment history and risk, credit use, and how the applicant compares to other organizations in its industry. According to POMS GN 00502.113, certain family members with custody of the beneficiary are exempt from the background check. For non-exempt individual representative payee applicants, field office staff must obtain the applicant’s permission before conducting part of the background check. SSA policy provides additional detail about the specific steps staff are directed to take to obtain information on individual representative payee applicants’ past criminal history. If the applicant does not give permission for a background check, their application to serve as payee will be denied. For more information, see POMS GN 00502.113, “Interviewing the Payee Applicant.” potential risk factors that create payee business losses due to fraud, failure, or severe delinquency, and (2) may provide an indication of any risk involved in the organization’s current or future performance as a payee. However, SSA does not assess these risk factors for most organizational applicants because SSA policy generally does not require staff to conduct background checks for organizational payees, and SSA only conducts credit checks for organizational payees that apply to collect fees. According to SSA data, as of July 2018, only 4 percent of organizational payees were authorized to collect fees and, therefore, may have undergone a credit check. Moreover, those credit checks that are conducted for organizations occur after their initial approval—when they are already serving beneficiaries—because organizations can not apply to collect fees until they have regularly served as payee for at least five beneficiaries for 1 calendar month or more. SSA officials told us the agency does not conduct background checks on organizations, in part because the process is more complicated than for individuals. SSA recommends that organizational payees screen employees who deal with beneficiary funds—identifying this as a best practice—but officials told us this is not required. However, in addition to employees who handle beneficiary funds, the criminal history of an organization’s principals (e.g., chief executive and operating officers, director, president, etc.) may also help inform SSA’s assessment of an organizational payee’s suitability, as these individuals may exert great influence over the tone and structure of the organization. Without conducting credit or background checks, SSA risks unknowingly approving questionable organizational applicants, therefore increasing the risk that beneficiary funds may be misused. In May 2019, SSA officials informed us that, while the agency has been focused on implementing criminal background checks on non-exempt individual representative payees, it is also exploring whether to conduct background checks on organizational payees’ employees or require organizational payee applicants to conduct background checks on their employees. In addition, they told us that SSA has also been considering whether to conduct credit checks on additional organizational payees, but has yet to make a decision on this matter. However, SSA did not provide information on the expected timeframes for this decision-making process. Further, SSA lacks a comprehensive plan for evaluating if and how to expand background and credit checks to organizational payees. When an organizational payee closes or is terminated, SSA must ensure that all affected beneficiaries can continue to access their benefits, either by finding a replacement payee or—when a beneficiary is deemed capable of managing their own finances—paying the individual directly. SSA officials told us they strive to avoid temporarily suspending benefits. However, temporarily suspending benefits may be necessary to avoid sending beneficiary funds to a former payee that is no longer able or willing to manage them. SSA’s policies delineate when temporarily suspending benefits may be necessary, such as when a beneficiary’s whereabouts are unknown. In 2017, according to SSA data, 427 organizational payees closed or were terminated by SSA. According to SSA data, SSA suspended benefits for more than 13,000 beneficiaries affected by payee closures and terminations in fiscal year 2017; their benefits were suspended for an average of 2.28 months. SSA policy describes the steps that SSA staff must take when dealing with the closure or termination of an organizational payee serving multiple beneficiaries, but SSA’s level of involvement in finding replacement payees varies depending on the situation. Staff at one field office said that, for the only organizational payee that closed in the last several years, they were involved in finding replacement payees for affected beneficiaries before they terminated the organizational payee. However, staff from two field offices told us that SSA is not always involved in finding new payees. For example, staff at one of these field offices said that when the state closed a nursing home in their jurisdiction, it was state officials and not SSA who found new facilities for affected beneficiaries. When these new facilities applied to serve as payee for their new residents, SSA processed the applications (see sidebar). Staff at another field office told us that before closing, some organizational payees identified prospective payees for affected beneficiaries. In those cases, payee staff submitted proposed payee changes to SSA, and SSA told these prospective payees they must file an application to become the approved payee. Officials in SSA’s central office told us that staff determine if the applicant is the most suitable payee before approving them. According to SSA officials, in 2015, SSA enhanced its policy on what to do when beneficiaries are affected by an organizational payee’s closure or termination. Specifically, national officials told us SSA added new procedures for appointing a new payee in cases of immediate payee termination and emphasized the narrow circumstances when it is appropriate to temporarily suspend benefits. Officials told us these changes were in response to a challenging experience terminating a large organizational payee in 2014 that served nearly 1,000 beneficiaries. Despite this change to agency policy on replacing organizational payees that are terminated or closed, SSA continues to face some challenges in approving replacement payees. Specifically, SSA staff we interviewed cited a number of challenges they had encountered, such as shortages of local organizational payees and difficulties obtaining information from terminated organizational payees. While these challenges may not apply to all field offices, they provide examples of circumstances that can complicate the process of reassigning beneficiaries. Lack of local organizational payees. Officials in some field and regional offices said they lack sufficient organizational payees in their local area. For example, staff in three field offices said many organizational payees in their area only serve certain types of beneficiaries, such as the elderly or individuals with developmental disabilities or specific medical conditions. Staff in two field offices told us they had unsuccessfully tried to recruit additional organizational payees in their jurisdiction. Similarly, a member of an SSA managers association noted that it has been several years since a new organizational payee was approved in her state. Difficulty ensuring community presence for fee-for-service organizational payees. Officials from SSA’s regional, area, and field offices told us that it can be challenging to meet the agency’s requirement that non-governmental fee-for-service organizational payees be community based. For example, staff at an SSA area office told us that finding payees within the community is challenging in sparsely populated and remote areas, such as along Maine’s border with Canada, where beneficiaries may not live near any approved organizational payees. In March 2019, SSA updated the policy on community presence for non-governmental fee-for-service organizational payees to better specify what is required for a payee to establish community presence, but it is not yet clear the extent to which this update resolves field office concerns about remote areas. Difficulty locating beneficiaries. Officials in some field and regional offices noted that they sometimes struggle to locate beneficiaries, which hinders reassignment. Homeless beneficiaries, in particular, can be difficult to find, according to staff in one regional office. Difficulty obtaining information from terminated organizational payees. Officials in some SSA offices told us that they may lack information necessary to complete the transfer of an affected beneficiary to another payee. For example, staff in a regional office said that terminated organizational payees may not always be forthcoming about unspent beneficiary funds. Staff in another field office told us that because a terminated organizational payee had not maintained adequate records of beneficiaries’ guardians, SSA staff had to go to court to identify them before approving replacement payees. SSA staff communicate with organizational payees at various points. According to SSA policy, field offices should communicate with organizational payees when they initially apply, and field office staff may communicate with payees as part of periodic oversight activities—such as through record change reporting requirements or following up on annual accounting forms. During the application process, SSA field office staff should explain the responsibilities and duties of a payee. For example, they should explain that payees must submit an annual accounting form and that payees must keep detailed records of how benefits are used in order to provide an accurate report to SSA when requested. Field office staff also should explain when payees must contact SSA, such as when a payee’s address changes. Monitoring and oversight activities, such as reviews of annual accounting forms, also provide opportunities for SSA field staff to communicate with organizational payees. Similarly, SSA’s ongoing reporting requirements—such as to update certain beneficiary or payee information—provide another opportunity to interact. According to field staff we interviewed, staff frequently communicate with organizational payees regarding changes to a beneficiary’s address. Finally, according to SSA officials, SSA also communicates with organizational payees by providing information online and providing guidance documents when payees are approved. While all field offices communicate with their organizational payees, how field offices communicate with payees can vary. Four of the eight field offices we visited had designated specific staff either to work with each organizational payee or with high-volume payees. In the other four field offices, payees talk to whichever staff member is available. SSA officials told us that the different workforce arrangements stem from varying workflows and staffing resources at individual offices. Similarly, we found variation across field offices regarding whether SSA staff reach out to organizational payees even if changes do not need to be made or problems addressed. For example, staff at four of eight field offices also said that they have held training sessions for groups of organizational payees. Further, staff at three field offices told us that SSA provides training to specific organizational payees at their request, such as when an organization experiences staff turnover. Seven of the eight organizational payees we spoke with expressed frustration either with SSA’s follow-through on communications or with its processes for receiving information from payees. Application status updates. Three payees said that SSA staff did not tell them how long it would take to review their application. They also said SSA staff had not provided updates during the process, which took 2 to 3 months or longer to complete. Follow-up calls. These three payees also said that they were not told how long it would take for SSA staff to return their calls, and two said that sometimes they never received a call back. Wait times. Two payees said that it takes too long to provide information in person at SSA field offices. For example, after signing in at a kiosk, a payee may have to wait for hours until their number is called. This payee said that they often bring beneficiaries to the SSA office and that long wait times can be very difficult for them, particularly those with mental illness. In some cases, beneficiaries have walked out or passed out while waiting in the SSA office, according to the payee. The payee also said that long wait times are sometimes compounded when field office staff require them to return to the queue for each successive case rather than handling all the payee’s cases at once. However, because field offices are allowed to establish their own workflow processes, this issue may not apply to other field offices. Faxing documents versus sending them electronically. Three payees said that having to fax documentation to SSA rather than send this information electronically creates additional work. SSA officials said that the agency has a plan to allow individual payees to securely transmit personally identifiable information electronically, but has not established a timeframe for allowing organizational payees to do so. At the field offices we visited, managers had different expectations regarding time frames for responding to payee requests. Three managers we interviewed said that staff should respond to payees as soon as possible, three managers said that staff should respond within 24-48 hours, and two managers said staff should respond within 7-14 days. SSA officials told us that SSA has not set timeliness standards for field offices because doing so could affect other workloads in unanticipated ways and it is the agency’s goal to provide service and support to all payees on an ongoing basis. SSA may receive feedback from organizational payees through various mechanisms. Officials from SSA’s central office told us that organizational payees can provide feedback either by contacting their local field office or calling SSA’s national customer service number. Some field office staff also said that they provide informal opportunities for payees to offer feedback. For example, one field office manager told us that he spends time building relationships with organizational payees, solicits feedback by asking how things are going, and sometimes visits organizational payees when he is nearby. Another manager emphasized the importance of gathering and responding to organizational payee feedback. This manager said that she established quarterly calls with multiple payees to discuss issues and solicit feedback. Managers of two field offices told us that they provide standardized SSA customer comment cards in their waiting areas, although the cards do not ask respondents to identify whether they are organizational payees. However, SSA does not have a mechanism for payee feedback to be systematically collected, compiled, and analyzed across field offices to determine if programmatic changes are warranted. SSA officials said they do not have or plan to develop a formal mechanism for collecting and analyzing organizational payee feedback because the current process allows field offices to respond to all public contacts in a consistent and timely manner. However, federal internal control standards state that management should establish reporting lines that allow the agency to receive quality information from external stakeholders and specify that quality information, among other things, should be complete, current, and provided on a timely basis. Without a formal mechanism to systematically collect and analyze payees’ feedback and ideas for program improvement, SSA cannot be sure that it is receiving complete or current impressions from organizational payees on how efficient its processes are or how timely it responds to their needs. Being aware of and responding to payees’ concerns might help the agency retain and attract organizations to serve as payees and ensure it is well-positioned to meet future challenges. SSA uses several methods to oversee organizational payees, including conducting onsite reviews, and reviewing annual accounting forms and collective accounts. However, each of these methods has shortcomings in its design and implementation, weakening SSA’s ability to effectively oversee payees and prevent fraud. SSA officials said they plan to conduct an over-arching assessment of fraud risks to the representative payee program in 2019, but the robustness of such a plan is yet to be determined. State protection and advocacy agencies (“state grantees”), the national association grantee (which is currently the National Disability Rights Network, or NDRN), and SSA regional offices play key roles in the new onsite review process for organizational payees. Given the extent to which onsite reviews uncover misuse and other problems, the onsite review is a crucial control for the representative payee program. Under the new process, state grantees generally interview selected payees, beneficiaries, and legal guardians or third parties; review financial records for selected beneficiaries over a 12-month period; transmit findings from their reviews to SSA; and, in some cases, follow up on deficiencies they identify. State grantees also suggest additional payees to review (beyond those targeted by SSA) if they think such a review is warranted. According to SSA, the national association grantee’s responsibilities include: (1) training state grantees; (2) ensuring the quality of onsite reviews; (3) serving as the first point of contact for state grantee communication and questions; and (4) providing state grantees with technical assistance, administrative support, and data collection services. According to SSA, the regional offices are responsible for compiling information to facilitate grantees’ onsite reviews that is not automatically provided through SSA’s system and for clarifying procedural and technical information for the grantees. Regional offices also address and resolve all deficiencies the grantees do not resolve, according to SSA. Lastly, under the new system, state grantees, the national association grantee, and SSA input information from reviews and track progress towards completing their assigned reviews using SSA’s new Representative Payee Monitoring Tool, which is used to manage and control the new onsite review process. According to six NDRN representatives, transitioning to the new onsite review system involved challenges with grantees gaining access to equipment, working through bottlenecks at some regional offices, responding to unanticipated workloads, and receiving timely responses to feedback. Specifically, NDRN representatives said that while the process of clearing grantees to access beneficiaries’ personally identifiable information has been efficient, there have been delays providing grantees with access to SSA laptops, printers, and scanners. As a result of these equipment delays, grantees started to conduct reviews on paper and then input the information later, according to NDRN representatives, thus using less efficient, manual processes. NDRN representatives also said that the new onsite review process involves multiple handoffs between grantee and regional office staff, which has contributed to bottlenecks at some regional offices. Moreover, NDRN representatives noted that, in addition to the reviews SSA originally assigned to the grantees, regional offices have tasked them with conducting quick response checks. Because these reviews have generally involved assessing a large number of financial records and conducting many beneficiary interviews, and were not anticipated in SSA’s initial plan, NDRN representatives believe they may affect the ability of some state grantees to complete the other reviews SSA had initially planned. Lastly, an NDRN representative said that the timeliness of SSA’s responses to grantee feedback and concerns (communicated from state grantees via NDRN) has diminished in recent months. Specifically, the NDRN representative said that the computer program SSA staff developed to enable NDRN to submit questions to the agency was initially working well. However, recently, as the volume of NDRN’s questions has increased, the system is not working as well, and NDRN has asked for clarification on some important issues, to which SSA has not yet responded. As of May 2019, SSA reported to us on progress state grantees had made towards reaching the total number of reviews SSA had planned for the fiscal year. Specifically, as of May 21, 2019, state grantees had conducted 112 of 852 planned high-volume reviews; 45 of 461 planned fee-for-service reviews; and 0 out of 60 planned state mental institution reviews. Although SSA initially assigned 2,800 low-volume reviews in grant year 2019, SSA estimated in July 2019 that it will have initiated around 1600 low-volume reviews by the end of the first grant year—about the same number as completed in fiscal year 2018 (1,691). SSA officials acknowledged these challenges and said they have been addressing them, and will continue to address them and to monitor progress. SSA officials cited significant improvements in issuing laptops since they began the process in September of 2018. Regarding delays in distributing printers and scanners, SSA reported that they are in the final stages of procuring printers but that as of May 2019, they had not identified an acceptable scanner model. SSA officials also said they are developing a policy to govern grantee use of printers. SSA acknowledged that workflow bottlenecks involving regional offices may exist, and said that they will continue to monitor all actions required to be taken by regional office staff. SSA staff also acknowledged having initiated more quick response checks than originally anticipated, and said they are researching options to alleviate the impact of these reviews on NDRN and state grantee resources. Finally, SSA staff said that they will continue to evaluate how SSA collects and responds to state grantees’ feedback, and to hold weekly discussions with NDRN to identify ways to improve the new onsite review process. GAO is not making recommendations in this area because the onsite review process is new and SSA continues to implement it and work to address implementation challenges. Onsite reviews are resource intensive because they involve examining organizational payee financial records and interviewing payee staff and beneficiaries; therefore, SSA uses a risk-based approach to select which organizational payees receive onsite reviews and how frequently such reviews occur. SSA reviews all fee-for-service, high-volume payees, and certain state mental institutions—which together account for around 67 percent of all beneficiaries and about 14 percent of all organizational payees—at regular intervals of every 3 or 4 years, depending on the type of organization. However, for the vast majority of organizations that are low-volume payees (29,082 of around 33,700), SSA selects a subset of payees to receive onsite reviews each year. As shown in figure 2, more than half of the onsite reviews SSA conducted in fiscal year 2018 were for low-volume payees (1,619 of 2,774 reviews). However, because there are so many low-volume payees, only about 6 percent of these payees received an on-site review. In contrast, the lower number of high-volume onsite reviews conducted (767) covered about 25 percent of high-volume payees. Given that only a fraction of low-volume payees are selected for onsite review each year, it is critical that SSA effectively prioritize which payees should receive onsite reviews so SSA can effectively allocate resources. To this end, SSA uses a predictive statistical model it first implemented in 2012 to rank low-volume organizations based on their chance of misusing beneficiary funds and selects for onsite reviews those organizations identified as having the highest risk. SSA staff told us they determine how many reviews to conduct based on available resources. However, we were unable to fully assess SSA’s decisions in developing its model, or the model’s accuracy at predicting misuse compared to alternative targeting methods, because SSA did not fully document or retain documentation that described in sufficient detail important decisions it made when developing it. For example, the available documentation does not explain in sufficient detail how SSA assembled data on the target population; how SSA sampled organizational payees for assessing characteristics; which variables SSA considered using to help predict misuse but ultimately decided not to include; how, if at all, it assessed or assured itself of the reliability of the data the model used; and how it decided to account for multiple beneficiaries with the same payee. An SSA official responsible for using the model said he was not sure whether documentation existed but was not retained, because the individuals who developed the model are no longer with the agency. Office of Management and Budget (OMB) standards for federal censuses and surveys—which contain accepted practices (but not requirements) for federal statistical efforts not officially covered by the standards—call for documentation that “includes those materials necessary to understand how to properly analyze data” and “replicate and evaluate” statistical estimates. Moreover, federal internal control standards state that effective documentation enables agencies to retain organizational knowledge, mitigate the risk of having knowledge limited to a few personnel, and communicate knowledge as needed to external parties, such as external auditors. Due to the absence of key documentation, neither SSA itself nor an external party is able to affirm whether, in comparison to other approaches, SSA’s predictive model is the optimal approach to identify low-volume payees and beneficiaries with the highest risk of misuse. SSA officials told us that they will revise the model at some point in the future—at which point they could improve the documentation—but that they do not have a formal plan to do so. SSA officials said they do not have imminent plans to update the model because the pool of identified misuse cases, which is driven by the number of onsite reviews conducted, is too small. Finally, SSA officials said they are hesitant to re-evaluate the organizational payee predictive model because they believe the current model is working effectively. However, seven years have passed since SSA first developed the model, and SSA cannot be assured that the current model remains as effective as when it was last formally validated and compared to alternative models or targeting methods. Accepted practices for developing predictive statistical models call for periodic re-estimation and re-validation, using data that are current and applicable to the conditions in which the model will be applied. Moreover, federal internal control standards call for agencies to conduct ongoing monitoring of the design and effectiveness of the internal control system including evaluations of control design. SSA reported conducting ongoing assessments of the model’s continued effectiveness, and provided us with aggregate performance data for 2012 to 2016. However, inclusion of older data and absence of more recent misuse data in aggregated results provide limited assurance of the model’s ongoing effectiveness. In addition, a recent report by SSA’s Office of the Inspector General (OIG) suggests that it may be possible to assess the ongoing suitability of nursing home payees by using additional data, although we did not evaluate the validity of the study’s conclusions. The SSA OIG’s report expressly looked at how data from the Centers for Medicare & Medicaid Services (CMS) might be used to evaluate the suitability of nursing homes and found that these data would help SSA more effectively assess the ongoing suitability of existing nursing home payees. Specifically, the OIG used CMS data reflecting penalties and other signs of underperformance to identify poorly-performing nursing homes that might also be poorly executing their duties as payees. For instance, according to SSA policy documents, the form can help SSA identify previously unreported changes to beneficiaries’ addresses; identify unapproved collective accounts; determine if certain beneficiaries’ savings are too high to qualify for benefits; or determine whether the organizational payee is authorized to charge a fee, if the payee reports charging one. outcomes would significantly dilute the model’s ability to detect misuse, which they consider to be the most important goal of the representative payee review process. Developing additional models to predict other types of poor payee performance besides misuse (such as poor recordkeeping or payees’ failing to meet beneficiary needs, which were identified in the OIG study) could reduce SSA’s reliance on a model for which the low number of misuse findings affects the efficacy of ongoing performance assessments and prevents timely updates. Since SSA has only identified 31 misuse cases using the predictive model since 2012, decades may pass before SSA has the approximately 5,300 misuse cases it wants in order to formally evaluate the model, and before SSA and others can be assured that low-volume payees are being optimally targeted for review. Without re-evaluating whether the current model remains predictive, and periodic assessments about whether it predicts high-risk payees better than an alternative model or targeting method, it is unknown whether SSA has maximized its ability to target low-volume payees. The annual accounting form is a key oversight tool because it touches most organizational payees, and reviewing the annual accounting form helps SSA to maintain current beneficiary and payee information and to identify and resolve potential problems. For instance, according to SSA policy documents, the form can help SSA identify previously unreported changes to beneficiaries’ addresses; identify unapproved collective accounts; determine if certain beneficiaries’ savings are too high to qualify for benefits; or determine whether the organizational payee is authorized to charge a fee, if the payee reports charging one. electronic processing indicates a potential problem, field offices sometimes follow up with the payee to resolve the issue. However, SSA has not established time frames within which field offices must initiate this follow-up. For example, SSA guidance states that when organizational payees do not submit the form timely, field offices should contact the payee by phone to find out why the required form was not completed. However, the guidance does not establish time frames within which field offices should initiate the call. Similarly, SSA told us they do not have time frames within which staff should follow up to resolve potential problems flagged during electronic testing. In the absence of national guidance, area offices we interviewed varied in the extent to which they established time frames for the field offices in their purview to follow up with organizational payees that did not submit an annual accounting form or whose form was flagged for potential errors. One area office we talked with expected staff to follow up with payees within 30 days but did not track time frames, another area office had not established time frames, and officials from one field office told us that their area office considered follow-up over 120 days to be untimely. Given the absence of SSA guidance and variation in area office practices related to establishing timeframes, field offices may not ensure that this oversight mechanism is attended to in a timely manner. Officials at one field office we visited told us that they had a backlog of forms needing follow-up because the designated point person had left the agency. Officials from another field office attributed the backlog to multiple factors, including a staff person being out sick and their workload not being reassigned, and the office taking on a special project. While we heard from several field offices that the majority of follow-up on annual accounting forms is for clerical errors or mistakes, staff from one field office said that when staff must follow up with organizational payees to ensure they submit a simple accounting form, it raises concerns about whether those payees are fulfilling their other duties. Federal internal control standards state that managers should use quality information to achieve the entity’s objectives and that they should ensure information is complete and provided on a timely basis. In May 2019, SSA officials told us that they are now exploring approaches to implement a nation-wide time frame to address these forms because a 2018 law— which reduced the volume of annual accounting forms SSA has to process—allows staff to focus on problematic forms more expeditiously. SSA officials said that they had not previously established a time frame because they expected organizational payees to have routine contact with field offices and expected field offices to re-evaluate the payee’s suitability if the payee did not cooperate when conducting SSA business. In addition, SSA expects state grantees to follow up on accounting forms as part of their onsite reviews. At the same time, one of SSA’s stated purposes for using the annual accounting form is to evaluate payee suitability on a regular basis rather than relying on ad hoc interactions between the payee and field office, or relatively infrequent periodic and targeted reviews. Until SSA establishes time frames within which staff must follow up on issues identified during annual accounting reviews, the agency cannot ensure that it is taking timely action to resolve potential problems and maximize this monitoring tool. Although the accounting form is a key oversight tool for SSA, shortcomings exist in the form’s content and design. For example, SSA’s annual accounting form does not ask or remind organizational payees about all collective account requirements, and as a result does not fully support SSA’s oversight efforts. Collective accounts are permitted under SSA policy, but SSA reviews and approves them to ensure that payees comply with SSA’s policies and procedures. While the annual accounting form asks payees whether they put any saved funds into a collective account, the form does not ask whether the payee uses a collective account for day-to-day expenses. Payees should disclose the use of any collective account to SSA independent of the form but may have neglected to, and SSA does not use the form to fully ascertain the use of collective accounts. Consequently, SSA may not have up-to-date information about all of the collective accounts that an organizational payee might be using—information that could help place these risk-prone accounts on SSA’s radar to initiate the approval process and provide ongoing oversight. Federal internal control standards state that agencies should design control activities to achieve objectives and respond to risks. When SSA officials were asked why the annual accounting form does not ask about all collective accounts, the officials said this would be unnecessary because payees are required to notify the field office if they wish to open such accounts. SSA also indicated that its periodic and targeted onsite reviews will uncover collective account issues for the highest risk payees. However, SSA finds many instances of unapproved collective accounts during its onsite reviews, suggesting that organizational payees might not be proactively reporting opening such accounts to SSA as required. For example, in fiscal year 2018, SSA found unapproved collective accounts in nearly 17 percent of the onsite reviews it conducted of organizational payees (in 477 instances out of 2,882 reviews). Staff we interviewed from one field office also said they have identified organizational payees with unapproved collective accounts. Specifically, staff said they have identified at least three payees with unapproved accounts, one of which they identified when reviewing the payee’s annual accounting form. This suggests that some payees may be willing to report they have a collective account, but not remember or understand their responsibility to seek approval from SSA when they open such accounts. Although SSA’s accounting form includes reminders of various payee responsibilities, the form does not include a reminder to all payees that they should notify SSA when they establish collective accounts. Reminding payees of these responsibilities could serve as a regular reminder for payees to notify SSA about the existence of these accounts, and thereby help ensure SSA provides regular oversight of them. Stakeholders have also identified shortcomings in the content and design of the accounting form. For example, SSA currently provides payees’ total benefit amounts in the form, and asks payees to report how they spent those benefits. In a 2007 review of SSA’s representative payee program, the National Academy of Sciences (NAS) reported that because SSA preprints total annual benefit amounts on the annual accounting form, it is easy for payees to report spending that matches the total provided by SSA. Even if the amounts the payee reported were incorrect, SSA’s electronic check would not trigger further review of these responses as long as the numbers added up. NAS further suggested that omitting this information would reinforce payees’ responsibility for keeping and consulting their records. In light of this and other findings, NAS broadly recommended redesigning the form to collect more meaningful data—a recommendation echoed by the Social Security Advisory Board in 2018. When asked why SSA did not adopt NAS’ recommendation, SSA indicated to us that it believed that NAS signaled that other recommendations were more important, and cited NAS’ statement that “no form, by itself, is going to detect program misuse.” At the same time, NAS restated its recommendation to redesign the form twice in its report and in each instance noted how the form could complement other oversight efforts. Research also suggests that agencies can improve the quality of the data they collect via forms by applying behavioral science insights. For example, behavioral science research has shown that requiring a signature at the beginning of an online form helps promote honest self- reporting and can lead to government savings.77, Moreover, the Internal Revenue Service has identified approaches based on behavioral science insights for improving compliance and honest self-reporting, and for encouraging people to make good choices when providing information. Given the importance of the annual accounting forms for oversight of payees, considering and applying, where appropriate, behavioral science insights while redesigning the accounting forms could help SSA achieve more reliable and accurate reporting. Executive Office of the President. National Science and Technology Council. Social and Behavioral Sciences Team Annual Report (September 2015). See ep 31-32. Lisa L. Shu, Nina Mazar, Francesca Gino, Dan Ariely, and Max H. Bazerman, Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end.” Proceedings of the National Academy of Sciences of the United States of America. (Nov. 18, 2012). In contrast, the signature field in SSA’s accounting forms is located at the end of the form. accounts without oversight. A regional office analyst referred to this as a glitch in the system and told us this issue was recently raised during a meeting with the central office. In response to our inquiry about disappearing alerts and collective account information, SSA staff indicated that removing alerts and collective account information after approval expires is appropriate because field offices should always renew collective accounts before this occurs. SSA further explained that the alerts are not deleted from eRPS, but rather removed from the system’s “Workload Action Center” 30 days after the collective account expiration date. Similarly, SSA reported that collective account information is not deleted from eRPS, but rather no longer displayed as an active account. However, removing information on accounts that were not renewed timely weakens the efficacy of its collective account review process to the extent that accounts are operating without SSA approval and oversight. SSA has taken steps to address risk associated with payee oversight, but to date has not continuously assessed and responded to potential risks. Federal internal control standards state that to manage risk, agencies should identify risks that might prevent the agency from achieving its objective; assess the significance of those risks; and design responses so that analyzed risks are within the agency’s risk tolerance level. In June 2013, SSA formed a task team to conduct a comprehensive review of the representative payee program and develop recommendations. This effort resulted in, for example, a new process of sharing misuse information with the Department of Veterans Affairs. While this was a positive step, the task team disbanded in 2014 because it had generated a set of recommendations and SSA wanted to shift to implementing those recommendations, according to agency staff. However, resulting actions did not include a forum or system for continuously assessing lessons learned from audits and reviews or identifying solutions that might have addressed gaps we identified in this report. For example, we found that SSA discovers many instances of unapproved collective accounts during onsite reviews, but we have not seen documentation that SSA has assessed the risk of unapproved collective accounts existing among low- volume payees that do not receive any regular scrutiny. Having a process for continuously assessing and responding to potential risks could better position the agency to respond to pressure placed on the payee program due to an aging population. As of May 2019, SSA reported it was in the early stages of planning a fraud risk assessment of the representative payee program (for both individual and organizational payees). In January 2019, a staff person within SSA’s Office of Anti-Fraud Programs, which provides centralized accountability and oversight for the agency’s anti-fraud activities, told us they had identified the representative payee program as one that might benefit from a risk assessment, and that they were currently developing a strategy for conducting such risk assessments for a number of programs. At that time, the staff person did not know whether they would be doing a fraud risk assessment of the representative payee program specifically. SSA subsequently reported in May 2019 that the agency has established a schedule and business process for conducting its risk assessments, including one on the representative payee program. According to SSA, the fraud risk assessment will provide a comprehensive and strategic look at the fraud risks facing the representative payee program and the controls SSA has in place to mitigate those risks. SSA also reported it plans to begin the assessment of the representative payee program in October 2019, and update it every 3 years beginning in 2024 to determine whether there have been any changes to the risks and whether additional actions are required. While promising, SSA plans have yet to take shape. Ensuring that its fraud risk assessments periodically examine the results of onsite reviews and audits will be an important element in the design of SSA’s risk assessment efforts. Organizational payees play a critical role in ensuring beneficiaries’ basic needs are met. The beneficiaries these payees serve—individuals who cannot manage their own finances and lack a family member or friend to do so on their behalf—are dependent on their representative payees and thus extremely vulnerable to financial abuse. It is therefore crucial that SSA take steps to shore up a range of gaps in how the agency evaluates, supports, and oversees payees to better ensure beneficiaries are protected. Carefully screening organizations applying to be representative payees is key to proactively avoiding potential abuse. However, in the absence of detailed and centrally-approved policy guidance on how to assess complex suitability factors for approving payees, SSA cannot be sure that field office staff are consistently and appropriately evaluating applicants’ suitability. Also, until SSA updates its electronic system to ensure staff’s rationale for approving or denying payees is captured in accordance with policy, SSA may not have the benefit of information to better monitor payees and inform future suitability decisions. Lastly, without a comprehensive plan, including timeframes, for evaluating if and how to conduct background and credit checks to help staff vet organizational payees—as it does for individual payee applicants—SSA may forgo potentially valuable safeguards for further protecting vulnerable beneficiaries. Once approved, organizational payees rely on SSA for information or action in order to effectively carry out their responsibilities. Absent a formal mechanism whereby feedback from payees on SSA services and processes can be collected, compiled, and analyzed, SSA may not be sufficiently aware of payee needs and frustrations, which in turn could result in lost opportunities to either retain or recruit organizations willing to serve this critical function, or make program improvements. To ensure payees are managing beneficiary funds appropriately, SSA relies on a number of monitoring mechanisms, including onsite reviews. Onsite reviews represent SSA’s most thorough and resource-intensive monitoring tool and must be appropriately targeted. Until SSA develops a plan to periodically review the predictive model’s design, considers inclusion of additional relevant data in the current model or an alternative model that predicts outcomes other than misuse, and documents any subsequent design changes, the model’s efficacy cannot be fully assessed or ultimately improved upon, and SSA may not be effectively targeting high-risk, low-volume payees for review. SSA may detect payee performance problems by reviewing annual accounting forms for all organizational payees; however, without a process to ensure prompt follow-up, SSA cannot be sure staff resolve problems in a timely manner. Moreover, mingling beneficiaries’ funds in collective accounts can mask misuse, and until SSA addresses gaps in the annual accounting form and issues with eRPS, SSA cannot effectively monitor payees’ use of such accounts. Addressing gaps in existing processes could improve the integrity of SSA’s representative payee program and reduce risks to SSA’s most vulnerable beneficiaries, but may not be sufficient in light of challenges posed by the nation’s aging population, which could swell the number of vulnerable beneficiaries that need payees. Carrying through with its plan to develop initial and periodic fraud risk assessments for the representative payee program—and ensuring that the assessments reflect consideration of findings from onsite reviews and audits—could help SSA anticipate potential problems and develop strategies to mitigate their impact. We are making the following nine recommendations to SSA: The Commissioner of the Social Security Administration should ensure that (a) the agency’s policies and guidance are specific enough so field office staff know how to apply complex suitability criteria for assessing payee suitability, such as by providing a minimum set of specific questions; and (b) additional regional guidance that is made available to staff is centrally reviewed for compliance and completeness. (Recommendation 1) The Commissioner of the Social Security Administration should create safeguards in the eRPS system to ensure that field office staff fully document all required information, such as the rationale for their decision, before approving an application. (Recommendation 2) The Commissioner of the Social Security Administration should complete a plan, including timeframes, for comprehensively evaluating if and how to leverage external sources of information on organizations’ suitability, such as by conducting background checks or credit checks on organizations or key staff that handle beneficiaries’ funds or requiring organizations to conduct their own background checks on key staff. (Recommendation 3) The Commissioner of the Social Security Administration should develop and implement mechanisms to systematically obtain and review feedback from organizational payees and communicate findings to SSA management. (Recommendation 4) The Commissioner of the Social Security Administration should (a) establish a plan and time frame for periodically reviewing the predictive model’s design; (b) consider additional data sources that would allow for additional screening or modeling of potentially high-risk organizational payees; and (c) ensure that subsequent design decisions are documented in sufficient detail so the development process can be more fully understood and replicated, either by SSA or a knowledgeable third party, with minimal further explanation. (Recommendation 5) The Commissioner of the Social Security Administration should require field offices to contact payees about missing or problematic annual accounting forms within a specific time frame. (Recommendation 6) The Commissioner of the Social Security Administration should revise the annual accounting form to enhance its effectiveness. Such revisions could include (but not be limited to) more fully ascertaining the use of collective accounts, adopting stakeholders’ recommendations on using the form to collect more meaningful data, and reflecting best practices from behavioral science insights in the design of the form. (Recommendation 7) The Commissioner of the Social Security Administration should enhance the eRPS system to ensure that field offices are (a) alerted when collective accounts are due to be reviewed; and (b) able to take action on expired collective account information and thereby avoid payees’ continued use of these accounts without oversight. (Recommendation 8) The Commissioner of the Social Security Administration should, as it carries through with its plan to develop a risk assessment for the organizational payee program, ensure that that the plan reflects periodic consideration of findings from onsite reviews and audits. (Recommendation 9) We provided a draft of this report to SSA for review and comment. In written comments, reproduced in appendix IV, SSA agreed with all nine of our recommendations and outlined its planned actions to address several of the recommendations. SSA also provided technical comments that we incorporated into the report, as appropriate. SSA provided additional comments on its plans to address four of our recommendations. Specifically, with respect to our second recommendation that SSA create safeguards in its Electronic Representative Payee System (eRPS) to ensure that field office staff fully document decisions to approve organizational payee applications, SSA reported that, as part of implementing the Strengthening Protections for Social Security Beneficiaries Act of 2018, planned changes to eRPS will improve documentation of selection decisions. SSA also reported it will also consider additional enhancements to eRPS in the future. We welcome SSA’s intentions to improve documentation of selection decisions and consider additional enhancements to eRPS. With respect to our third recommendation that SSA complete a plan, including timeframes, for evaluating if and how to leverage external sources of information on organizations’ suitability, such as by conducting background checks or credit checks on organizational payee applicants, SSA officials reiterated that SSA is first focusing on implementing provisions of the Strengthening Protections for Social Security Beneficiaries Act of 2018 related to background checks for certain individual payees. After completing this work, SSA plans to evaluate conducting criminal background checks and credit checks on organizational payees and their staff. While we agree that implementing background screening pursuant to the law should take precedence, SSA should seek opportunities to implement screening for organizational payees at the earliest opportunity. With respect to our fifth recommendation related to SSA reviewing, enhancing and documenting its model for selecting low-volume organizational payees for on-site reviews, SSA reported that it will pursue other data sources to develop additional screening tools and models to identify potentially high-risk organizational payees, but that it is unable to incorporate additional data into the existing model. We recognize that the current model, which focuses on misuse findings and is based on historical data, presents challenges for both updating and including new data sources. Therefore, as SSA considers additional screening tools and models to identify high-risk, low-volume organizational payees, SSA should develop a plan for revising the existing model that allows for more timely updates and results in documentation of related design decisions. With respect to our eighth recommendation that SSA enhance the eRPS system to more effectively address expiring collective accounts, SSA officials reported that they would work with staff to ensure staff know where to find alerts for expiring accounts and enhance how eRPS displays information on collective accounts that have already expired. We agree with SSA’s proposed actions. However, we adjusted our recommendation to clarify that SSA should enhance eRPS in a manner that ensures staff take action on expired accounts and that payees do not continue to use expired accounts without oversight. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Commissioner of the Social Security Administration, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4040 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. The three objectives examined in this report are how the Social Security Administration (SSA): (1) approves organizations to be representative payees, (2) communicates with organizational payees, and (3) oversees these organizations. To address our three objectives, we reviewed relevant federal laws and SSA policies and guidance. We interviewed SSA officials in its central office and staff in four regional offices that we selected to reflect a range in the number of states and organizational payees they collectively oversee and to achieve diversity in geographic location. Within those regions, we visited eight field offices covering seven states, which were selected to include both metropolitan and non-metropolitan areas that maximized the number of files we would have available for our review (see next paragraph). We also interviewed officials in one area office per region—two representing metropolitan area field offices, and two representing non-metropolitan area offices that we visited. These interviews with regional, area, and field office staff are intended to obtain perspectives from SSA officials in different parts of the country and are not intended to be representative of all SSA field offices and staff. We also analyzed program data, including the number and type of organizational payees and the number of beneficiaries they serve. We assessed the reliability of these data by reviewing relevant documentation and interviewing SSA staff knowledgeable about the systems used to collect and maintain the data and determined the data were sufficiently reliable for our use. To determine how organizations are approved to be representative payees, we reviewed SSA’s policies and relevant federal laws and regulations. At each field office we visited, we (1) interviewed managers about their role in the application process and (2) reviewed up to six organizations’ electronic files in the Electronic Representative Payee System (eRPS), the primary data system SSA uses to track representative payees. Specifically, at each field office we reviewed up to two applications that SSA had approved (either initial applications to serve as representative payee or initial applications to collect fees); up to two applications that SSA had denied (initial or to collect fees); and files for up to two organizations that were terminated or closed in the past 5 years. In some cases, field offices we visited did not have the full number of cases available, and we reviewed fewer files in those offices. We selected the most recent approval, denial, and termination files that were available. In all, we reviewed 15 recently approved applications, six recently denied applications, and three recent terminations. We also interviewed cognizant SSA officials at the central office and the four regional and area offices we selected. We conducted background checks on a stratified random sample of 205 current organizational payees. The sample was selected to include fee- for-service organizations with 50 or more beneficiaries, fee-for-service organizations with fewer than 50 beneficiaries, non-fee-for-service organizations with 50 or more beneficiaries, and non-fee-for-service organizations with fewer than 50 beneficiaries. We entered information on selected organizations into a database called CLEAR and reviewed the resulting reports for any indication of criminal history. Many of these reports included the criminal history of individuals who are or may be associated with the organizational payee, and we reviewed these with particular focus on the crimes that bar individuals from serving as individual payees. For those reports that contained an indication of criminal history, we selected reports that indicated there may have been federal crimes or felonies at the state or local level and attempted to obtain court records to provide further insight into the nature of the crimes and the outcome of the cases. However, because we lacked information that would have made it possible for us to definitively link a conviction to staff in an organization—such as Social Security numbers for payee staff that are in leadership or financial management roles—the results of our analysis were not reliable enough to report. SSA collects Social Security numbers for individual payee applicants but not for any principals or staff from organizational payee applicants. Without this information, it is impossible to definitively link criminal convictions to individuals associated with organizational payees. To help understand how SSA communicates with organizational payees, we reviewed program guidance and interviewed representatives of eight organizational payees—one in the local area of each field office we visited, in addition to interviewing officials in each field office. We also interviewed cognizant SSA officials at the central office and the four regional and four area offices we selected. To review SSA’s overall oversight of organizational payees—including onsite reviews and reviews of the annual accounting form and payees’ use of collective accounts—we reviewed relevant federal laws and regulations, program policies, and relevant SSA documents; analyzed data; and interviewed SSA officials at the central office, the four regional and four area offices we selected, and the eight field offices we visited. To further understand SSA’s new onsite review process, we reviewed agency documents that describe the roles and responsibilities of key players in SSA’s new onsite review process. We also interviewed SSA officials and representatives of the National Disability Rights Network (NDRN) about the status of its implementation. To determine the extent to which different types of organizational payees receive onsite reviews, we analyzed SSA program data for fiscal year 2018. We assessed the reliability of these data by reviewing relevant documentation and interviewing knowledgeable agency officials and determined they were sufficiently reliable for our purposes. To learn about the outcomes of onsite reviews, such as how frequently unapproved collective accounts were identified, we reviewed SSA’s annual reports to Congress. We determined SSA data on the number of onsite reviews conducted and SSA data reported to Congress on unapproved collective accounts were sufficiently reliable for our purposes. We did not assess the efficacy of the new onsite review process or the quality of onsite reviews because we determined it was too soon to evaluate recent program changes. Instead, we described the roles and responsibilities of key players in the new process and interviewed SSA and NDRN to provide information on the status of implementation. To assess the predictive model SSA uses to select low-volume organizational payees for onsite reviews, we analyzed available documentation and interviewed SSA officials knowledgeable about the predictive model. This information included: (1) a list of variables; (2) the code SSA uses to execute the model; and (3) a brief description of how SSA developed the model, including a high-level description of its methodology and an analysis of the predictive power of the model compared to random chance. We compared the documentation SSA provided us with accepted practices for maintaining documentation of statistical models. For detailed results on the findings of this analysis, see appendix III. To obtain a range of perspectives on the organizational payee program, we interviewed staff of the Social Security Advisory Board, representatives of an SSA managers’ association, an organizational representative payee association, and NDRN. In addition, we interviewed representatives of advocacy groups for the aged, persons with physical disabilities, and persons with mental illness regarding their constituents’ experiences with SSA’s organizational payee program. Per Social Security Administration (SSA) policy, field office staff should consider certain factors when evaluating organizations’ suitability to serve as payees. Some factors apply to all applicants, including both individuals and organizations, while others apply only to organizational payee applicants (see table 2). In addition, there are some requirements for organizational payees applying to collect fees for their payee services. According to SSA policy, organizational payees that are applying to collect fees must meet the following requirements: Be regularly serving as a payee for at least five beneficiaries for at least 1 calendar month; Generally not be a creditor of the beneficiaries it serves; and Be a state or local agency with a qualified mission, or a non-profit social service agency that is community-based, bonded, and licensed. The Social Security Administration (SSA) uses a predictive statistical model it implemented in 2012 to rank low-volume organizations based on their chance of misusing beneficiary funds and selects for onsite reviews those organizations identified as having the highest risk. The predictive model uses a logistic regression to estimate the chance that each payee will misuse benefits, given the characteristics of the beneficiary and payee, such as the length of time served as a payee and whether the beneficiary received a large lump sum payment from the payee. SSA takes the predictive model output, which is calculated for every payee and beneficiary pair, and uses it to rank payees. SSA assigns organizations for review depending on (a) their rank (organizations that have a higher likelihood of misusing benefits are more likely to be selected); and (b) available resources. To review the predictive model, we interviewed SSA officials knowledgeable about the model and reviewed available documentation. This documentation included: (1) a list of variables; (2) the computer code SSA uses to execute the model; (3) a brief explanation of how SSA periodically assesses the model and related performance statistics; and (4) 2 documents (totaling 5 pages) describing how SSA developed the model. We compared this documentation to accepted practices for maintaining documentation of statistical analysis, such as standards published by the Office of Management and Budget (OMB). The documents describe, at a high level, SSA’s methodology for developing the model. It also includes an analysis of the predictive power of the model compared to random chance. of predictor variables, and ultimately selected a final model using a step- wise selection process. However, available documentation does not include information necessary to evaluate how SSA assessed other candidate models or understand the rationale for SSA’s decision to accept its final model. For example, there is limited documentation to: Reproduce SSA’s Target Population: The documentation does not describe in detail how SSA identified all organizational payees that served from 1993 to 2009 (such as how SSA queried the Representative Payee System), nor does it explain in detail how SSA linked beneficiary and organizational level data, such as to count the number of beneficiaries that each payee served. SSA subsequently explained in its technical comments that it used Social Security numbers to link information among several systems. However, SSA did not describe steps it took to establish the linkages, or steps taken to identify organizational payees that served from 1993 to 2009, in enough detail for an independent analyst to reproduce the work. Moreover, SSA did not provide this written documentation upon our original request, which suggests that SSA did not maintain complete records of the work. Reproduce SSA’s Sample Design: The documentation does not describe in detail how SSA designed the probability sample it used to develop the model or how, if at all, it weighted the sample to account for varying probabilities of selection in the sample. Selecting the appropriate sampling method for a model and applying appropriate weights generally increases its predictive accuracy. Reproduce SSA’s Process for Assembling the Data and Selecting the Final Model: The documentation provides limited information about the input variables and models that SSA tested but ultimately did not use. In addition, the documentation does not show how SSA assessed and addressed potential correlations between the variables it selected. For example, we could expect certain variables, such as receipt of a lump sum payment and receipt of a lump sum payment over $1,000, to be highly correlated. Although highly correlated variables do not necessarily impair the model’s predictive accuracy, they can influence which individual variables test as being predictive during the model’s development. The documentation also does not describe how SSA chose to split continuous variables into categorical variables—a choice which can influence predictive accuracy. Understand How SSA Assessed Data Reliability: Available documentation does not indicate whether SSA assessed the reliability of data used in its model. The reliability of the outcome variable— misuse—is particularly important. Unreliable data regarding whether misuse occurred, either due to incorrect data entry or other errors, would compromise the model’s ability to accurately predict the likelihood of misuse. In contrast, the reliability of variables that could signal risk of misuse—such as whether the beneficiary received a large disbursement of funds—is less critical. Even variables prone to measurement error may still predict misuse accurately. Nevertheless, assessing their reliability remains important, since reducing measurement error can increase the model’s predictive power. Such assessments could range from limited testing of the data—e.g., for outliers, illogical values, and missing data—to broader, independent verification of data reliability. Regardless of the approach used, documenting all data reliability assessments allows internal and external stakeholders to assess, and possibly improve, the model. Explain whether, or how, SSA’s model addressed potential patterns of misuse for beneficiaries served by the same payee: Statistical models typically assume that estimates can be generated independently for each unit of analysis—in this case, unique pairs of beneficiaries and payees. However, in cases where multiple beneficiaries are served by the same payee, this may not be the case. Patterns of misuse might be similar for all beneficiaries served by a given payee, such as if the payee were systematically defrauding all of its beneficiaries. Accurately modeling data with this kind of nested structure—which conflicts with typical statistical assumptions—often requires multi-level modeling methods. However, SSA’s documentation does not specify how or whether it applied these methods, or otherwise assessed or adjusted for the nesting of beneficiaries within payees. Reproduce SSA’s process for ranking organizations: With the current model, which assigns a score for each payee-beneficiary pair, SSA uses the predictive model’s output to then rank payees. However, there are various approaches for ranking payees, ranging in sophistication, and SSA does not have sufficient documentation to determine whether the approach currently being used best predicts risk to beneficiaries. In addition to the contact named above, Michele Grgich (Assistant Director), Isabella P. Anderson, Dan Meyer and Amy E. MacDonald (Analysts-in-Charge), Daniel Bibeault, Ted Burik, Daniel Concepcion, Jennifer Cook, Gus Fernandez, Alex Galuten, Sheila R. McCoy, Arthur Thomas Merriam, Jr., Mimi Nguyen, Ramon J. Rodriguez, Margie K. Shields, Joy Solmonson, Almeta Spencer, Jeff M. Tessin, Walter K. Vance, Kathleen van Gelder, Srinidhi Vijaykumar, and Khristi A. Wilkins made significant contributions to this report. In addition, Seto J. Bagdoyan, Joy Booth, Gabrielle M. Fagan, Robert H. Graves, Rosalind C. Romain, and Helina P. Wong contributed to the report.", "summary": "Nearly a million individuals relied on organizational payees to manage their Social Security benefits in 2018. Due to an aging population more beneficiaries may need organizational payees in the future. These beneficiaries are among the most vulnerable because, in addition to being deemed incapable of managing their own benefits, they lack family or another responsible party to assume this responsibility. SSA reports that misuse of benefits by payees is rare, but its Office of Inspector General has identified cases of misuse that have harmed vulnerable beneficiaries. GAO was asked to review SSA's organizational payee program. This review examines, among other things SSA's process for approving payees and its monitoring efforts. GAO reviewed relevant federal laws, regulations, policies, and guidance; analyzed SSA data from fiscal year 2018; analyzed the predictive statistical model SSA uses to select low-volume payees for on-site reviews; and interviewed SSA central office staff and regional, area, and field office staff in four regions selected for geographic diversity. The Social Security Administration (SSA) approves organizational payees—such as nursing homes or non-profits that manage the Social Security benefits of individuals unable to do so on their own—by assessing a range of suitability factors, such as whether the organizations have adequate staff to manage benefits for multiple individuals. However, GAO found that SSA's policy does not specify how to assess more complex suitability factors, such as whether an organization demonstrates sound financial management. Without clearer guidance, unqualified or ill-prepared organizational payees could be approved to manage benefits. Also, SSA does not currently require background checks for key employees of an organizational payee. In contrast, SSA requires background checks for individual payees—such as a relative or friend of the beneficiary. A comprehensive evaluation could help SSA determine whether and how to expand their use of background checks to organizational payees. To ensure organizational payees are managing funds appropriately, SSA uses several monitoring tools, including resource-intensive onsite reviews. Certain organizational payees, such as those that charge fees for their services or have 50 or more beneficiaries (high-volume), receive onsite reviews every 3 to 4-years. In contrast, payees that serve fewer than 50 beneficiaries (low-volume)—the vast majority—are selected for review based on their estimated likelihood of misusing beneficiary funds, and a relatively low percent of them receive onsite reviews (see figure). SSA uses a predictive statistical model to identify higher risk low-volume payees, but the model's effectiveness cannot be fully assessed by GAO or others due to missing documentation on how it was designed. SSA officials said they will update the model in the future, but do not have a time frame for doing so. Establishing such a time frame and documenting design decisions are key steps toward assessing the model's effectiveness. Another way SSA oversees organizational payees is by reviewing their annual accounting forms, but shortcomings exist in SSA's review of the form and in the form's content and design. For example, SSA lacks timeframes for following up on missing or problematic forms. Also, the accounting form does not capture complete information on whether payees co-mingle beneficiaries' funds in collective accounts, which can limit SSA's ability to monitor those risk-prone accounts. Establishing timeframes and revising the form could enhance the effectiveness of the annual accounting form as an oversight tool. GAO is making nine recommendations in this report, including that SSA: clarify how to assess complex suitability factors; assess requiring background checks for organizational payees; establish a timeframe for reviewing the predictive model and document design decisions resulting from that review; and establish timeframes for, and conduct revisions of the accounting form. SSA agreed with all nine recommendations and provided technical comments that GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "Signed into law on May 9, 2014, the DATA Act expands on previous federal transparency legislation. It requires federal agency expenditures to be disclosed and agency spending information to be linked to federal program activities so that policymakers and the public can more effectively track federal spending. The DATA Act also requires government-wide reporting on a greater variety of data related to federal spending, such as budget and financial information, as well as tracking these data at multiple points in the federal spending life cycle. To accomplish these goals, among others, the act gives OMB and Treasury responsibility for establishing government-wide financial data standards for any federal funds made available to or expended by federal agencies. These standards specify the data to be reported under the DATA Act and define and describe what is to be included in each data element with the aim of ensuring that information reported will be consistent and comparable. As Treasury and OMB implemented the DATA Act’s requirement to create and apply data standards, the overall data standardization effort has been divided into two distinct, but related, components: (1) establishing definitions which describe what is included in each data element with the aim of ensuring that information will be consistent and comparable, and (2) creating a data exchange standard with technical specifications which describe the format, structure, tagging, and transmission of each data element. In the implementation of the DATA Act, OMB took principal responsibility for the definitions, while Treasury took principal responsibility for the technical standards that express these definitions, which federal agencies use to report spending data for publication on USAspending.gov. The act also holds agencies accountable for submitting complete and accurate data to USAspending.gov and requires that agency-reported award and financial information comply with the data standards established by OMB and Treasury. One of the purposes of the DATA Act is to establish government-wide data standards to provide consistent, reliable, and searchable spending data that are displayed accurately for taxpayers and policymakers on USAspending.gov (or a successor website). As we have reported previously, establishing a data governance structure—an institutionalized set of policies and procedures for providing data governance throughout the life cycle of developing and implementing data standards—is critical for ensuring that the integrity of the standards is maintained over time. The need for a data governance structure is underscored by our previous analyses of the quality of the federal spending data available on USAspending.gov and inconsistencies we identified in how agencies report data according to data standards. A data governance structure could be useful for adjudicating revisions, monitoring, and ensuring compliance with the standards over time. As we have noted, such a structure, if properly implemented, would greatly increase the likelihood that the data made available to the public will be accurate. A data governance structure can also provide consistent data management during times of change and transition. We have previously reported that gaps in leadership can occur as administrations change. This can impair the effectiveness and efficiency of complex government- wide efforts, potentially resulting in delays and missed deadlines. Accordingly, in 2015, we recommended that OMB, in collaboration with Treasury, establish a set of clear policies and processes for developing and maintaining data standards that are consistent with leading practices. OMB and Treasury did not comment on our recommendation. We plan to conduct work intended to help inform OMB’s and Treasury’s efforts. This work may include the development of a maturity model that could provide a framework for assessing data governance activities related to federal spending data. This work may also have broader government-wide implications as agencies begin implementing the requirements of the Foundations for Evidence-Based Policymaking Act enacted on January 14, 2019, including designating Chief Data Officers with data governance and implementation responsibilities. In December 2018, OMB staff told us that they are transitioning from the governance structure used for initial DATA Act implementation to a new structure for managing data standards within the broader context of efforts to establish a federal data strategy. According to OMB staff, the initial data governance structure reflected OMB’s and Treasury’s focus on creating the design and build functions required to meet the statutory requirements of the DATA Act. The President’s Management Agenda (PMA), released in March 2018, outlines a long-term vision for modernizing federal operations. To address the issues outlined in the PMA, the administration established a number of cross-agency priority (CAP) goals. These goals, required by the GPRA Modernization Act of 2010, are to address issues in a limited number of policy areas requiring action across multiple agencies, or management improvements that are needed across the government. According to OMB staff, several of the 2018 goals relate to data standardization, and a new governance structure is needed to achieve those goals. OMB staff informed us in July 2018 that the governance structure used for initial implementation efforts, which included the DATA Act Interagency Advisory Committee and Data Standards Committee, had been disbanded, and that the advisory roles of these groups were assumed by the Chief Financial Officers Council’s DATA Act Working Group (CFOC Working Group). According to OMB staff, the working group includes four subgroups, which focus on Policy, Internal Controls and Data Quality, Audit Coordination, and the DATA Act Information Model Schema (DAIMS), respectively. OMB staff also told us that by December 2018 an interagency board and council, both led by the General Services Administration (GSA), had begun to advise OMB on policy matters. According to an action plan that OMB and GSA released along with the March 2018 PMA, the new interagency Shared Solutions Governance Board (SSGB) and Business Standards Council (BSC) are responsible for setting goals and providing advice to promote a government-wide marketplace for shared services. Specifically, they cover mission-support services such as human resources and financial management that a small number of providers offer to many agencies. According to OMB staff, this oversight function involves creating and administering government-wide data standards, including data standards established to support the DATA Act. The SSGB includes executives from across the federal government and is responsible for making recommendations to OMB on shared services policy. The BSC provides expertise on various subject matter areas (e.g., procurement and financial assistance) to promote the development of common business capabilities and data standards. The action plan does not discuss how the functions carried out by the SSGB and BSC apply specifically to the data standards established under the DATA Act. In commenting on a draft of this report, OMB staff told us that the “Governance Ecosystem” page on the website of Unified Shared Services Management (USSM) describes the SSGB and links functions of the SSGB and BSC to the DATA Act. They said it does this by showing that OMB and Treasury have key roles in all three entities. However, this common membership does not, in itself, provide the transparency and clarity of documented policies and procedures for governing DATA Act standards. Treasury officials said that the CFOC Working Group is involved in aligning DATA Act data standards across various functional communities, including procurement and financial assistance. Further, the group is considering making recommendations to OMB regarding changes to data definitions and other policy matters. For example, Treasury officials told us that in fall 2018, the DAIMS Subgroup identified difficulties in aligning different definitions of Period of Performance Start Date used for procurement and in financial assistance awards, and plans to elevate this issue to the Policy Subgroup for review. Specifically, the DAIMS Subgroup found that it is not always clear whether the start date should be reported as the date when a specific transaction occurred or the date when the original underlying award was made. This choice about how to interpret the data element can have substantial consequences for the consistency of the data reported. For example, in some cases, the underlying awards for recent transactions were made in the 1960s or 1970s. According to OMB staff and Treasury officials, at the center of this shifting array of advisory bodies, the DATA Act Executive Steering Committee (ESC) has continued to meet regularly and to serve as the top-level governance body for DATA Act implementation. OMB staff told us that the ESC is chiefly responsible for setting government-wide policy for the data standards based on the recommendations from various advisory bodies. In addition to the ESC, Treasury has continued to maintain and update the DAIMS and DATA Act Broker, following a set of change control procedures that involve consultation with stakeholders and public release of information about updates. Although OMB has taken some steps to address our recommendation, efforts are still needed to establish a clear set of policies and procedures for governing the data standards established under the act. The key practices for data governance that we identified in our previous work are shown in table 1. In the specific context of the DATA Act standards, Treasury and OMB have taken steps to enforce the use of data standards by directing agencies to develop and maintain data quality plans and requiring agencies to submit data through the DATA Act Broker. The broker performs validations to improve data quality and ensure the consistent application of data standards. However, because the approach to governing DATA Act data standards has continued to evolve during the past few years, and because a set of data governance policies and procedures is not documented, we were unable to conduct a comprehensive assessment of OMB’s and Treasury’s data governance efforts against leading practices. While some data governance activities have been undertaken within the specific context of DATA Act data standards, others are part of broader efforts under the PMA. In July 2018, OMB staff told us that governance over the DATA Act data standards would be accomplished within the broader context of the CAP goals established under the PMA. For example, OMB established a governance structure to achieve the objectives of CAP goals related to “Results-Oriented Accountability for Grants.” As part of this broader effort to standardize grants management business processes and data to increase efficiency and reduce reporting burden, OMB, the Department of Health and Human Services, and other federal agencies have published a list of draft grants management data standards for public comment. However, published documents describing this effort do not explain how the process for developing grants management standards under this CAP goal would apply specifically to the data standards established under the DATA Act. Nor do they address if or how these new standards align with those established under the act. Further, none of the documentation on the PMA’s governance structure for grants management mentions the DATA Act. In commenting on a draft of this report, OMB staff told us that the staff members from OMB and Treasury who are responsible for the grants management standards are the same people involved in managing the DATA Act standards. While this connection between the two efforts may provide adequate communication in the short term, staffing is likely to change over time, and there is no assurance that the same people will always be involved. As we have reported previously, having documented policies in place that delineate clear roles and responsibilities for decision-making could help to ensure continuity into the future. As the Comptroller General testified in 2015, in the absence of a clear set of institutionalized policies and processes for developing standards and for adjudicating necessary changes, the ability to sustain progress and maintain the integrity of established data standards may be jeopardized as priorities and data standards shift over time. Managing and controlling changes to data standards is a key activity for data standardization and effective data governance. The DATA Act requires OMB and Treasury, in consultation with the heads of federal agencies, to establish government-wide financial data standards that include common data elements for financial and payment information required to be reported by federal agencies and entities receiving federal funds. Among other requirements, these standards, to the extent reasonable and practicable, must be capable of being continually upgraded as necessary. According to key practices for data governance that we identified in our previous work, organizations should have documented policies and procedures for making decisions about changes to existing data standards. In June 2018, OMB staff changed certain data definitions in the publicly accessible website that serves as the official repository for the data definitions. However, OMB does not have a documented procedure for updating or making changes to these definitions. In commenting on a draft of this report, OMB staff stated that the DATA Act Information Model Schema (DAIMS) change control procedures were the method for updating data standards. However, OMB’s website for data definitions is maintained separately from the DAIMS, and the DAIMS procedures only address changes to the DAIMS, and do not address this separate repository of data definition standards. OMB staff said that the June 2018 revisions were made in response to the findings of our November 2017 report. Specifically, OMB revised the Primary Place of Performance Address definition to no longer include a street address or county. OMB amended the definition of Record Type to clarify that it applies to financial assistance awards only. As shown in figure 1, OMB also amended the explanatory text preceding the definitions to revise and clarify its policy regarding agencies’ use of data definitions. OMB staff described the changes to definitions as minor technical corrections to align with the reporting instructions in the DAIMS. In December 2018, OMB staff informed us that OMB’s procedure for making changes to the data definitions it maintains in the official repository can be found on the “Governance Ecosystem” page of the website of Unified Shared Services Management (USSM). However, our review of that page in January 2019, including the links it provides to other pages, found no evidence that the website provides any documentation related to the DATA Act. In particular, we found no evidence of a documented procedure for making changes to the data definitions in OMB’s official repository. The staff were unable to provide documentation to show that any standard procedure was followed in making the June 2018 changes, or that the DATA Act Executive Steering Committee approved the changes. As discussed earlier in this report, that committee is the top-level governing body for DATA Act implementation and is responsible for approving changes to data standards. The evolution of OMB’s approach to developing a governance structure to maintain the integrity of the DATA Act data standards could in part explain the lack of a documented procedure for updating the definitions. As discussed above, OMB has created and disbanded various advisory bodies for DATA Act data standards and has only recently decided on an approach for formalizing governance over the standards, namely the decision to integrate governance of these standards with the governance processes administered by the SSGB. In 2015, we reported that establishing a formal framework for providing data governance throughout the life cycle of developing and implementing these standards is critical for ensuring that the integrity of the standards is maintained over time. Without established written procedures for making revisions to data definitions, needed changes may not be made in a timely manner, which could impair data quality. For example, if the definitions in the DATA Act official repository and definitions in other sources are not aligned, then agency staff responsible for DATA Act compliance and reporting may make inconsistent choices about which definitions to apply when creating and submitting data. As we have previously reported, the current data governance structure did not prevent inconsistencies between the DAIMS and the official repository for data definitions. Changes to data standards for federal spending data should be transparently communicated to stakeholders, including the public. The DATA Act requires OMB and Treasury to consult with public and private stakeholders in establishing data standards. In addition, according to key practices for data governance that we identified in 2016, organizations should have documented policies and procedures for managing, controlling, monitoring, and enforcing consistent application of data standards and for obtaining input from stakeholders and involving them in key decisions, as appropriate. Standards for internal control in the federal government state that management should externally communicate the necessary quality information to achieve the entity’s objectives. These objectives can include those relating to the release of reliable information in accordance with appropriate standards, applicable laws and regulations, and expectations of stakeholders. In the context of standards for transparently reporting federal spending data, stakeholders include the general public as well as staff at federal agencies. OMB did not transparently communicate the June 2018 revisions. OMB staff said that the changes were communicated in OMB Memorandum M- 18-16, which was issued on June 6, 2018. As shown in figure 2, a footnote in that memorandum contains a link to the official web page for OMB’s Office of Federal Financial Management. That page includes a link, labeled “DATA Act Data Standards,” to the public MAX.gov page that serves as the official repository for the data definition standards. However, neither this footnote nor other text in the memorandum makes reference to changes made to the definitions and policy. As of March 18, 2019, the official repository did not indicate that any changes have been made since the initial creation of the definitions in 2015. OMB did not provide documentation showing that the revisions were communicated to the public or to specific categories of stakeholders, such as users of the data standards within the federal government. As described below, the procedures that Treasury has implemented for managing changes to technical guidance, including publishing revision histories for guidance documents, represent one potentially effective approach to informing stakeholders, including the public, about changes to data standards. OMB staff viewed the revisions made in June 2018 as minor technical corrections that were needed to align the definitions with other OMB policies and with the consensus view of stakeholders at the time the data standards were first established. Consequently, they did not believe it was necessary to communicate these revisions publicly or indicate in the official repository that changes had been made. However, these revisions required significant changes in some federal agencies’ use of data definitions. As we reported in November 2017, some agencies applied DATA Act definitions directly when generating data to be reported to USAspending.gov. The new explanatory text added to the data definition repository instructs agencies not to apply these definitions directly, but instead to apply the more detailed definitions contained in regulations and policies governing the making and management of awards. Without transparent communication of changes to data definition standards, stakeholders—including staff at federal agencies required to report data according to these definitions—may miss important information relating to changes in how, when, and by whom data definitions are to be applied. The staff may then report data that are not consistent and comparable across the federal government. Such inconsistent reporting can undermine the transparency goals of the DATA Act, particularly when it affects key data elements, such as those describing geographical information. For example, we found in November 2017 that inconsistent data were reported about the locations where the federal government spends money, because some agencies used OMB’s DATA Act definition of the Primary Place of Performance data element, while other agencies used definitions from other sources, such as the data dictionary for the Federal Procurement Data System – Next Generation (FPDS-NG). In addition, a revision history showing when clarifications of policy and corrections to data standards were made could assist users of federal spending data, including historical data, in interpreting those data and assessing their reliability and quality. Treasury has established procedures for consulting with and informing stakeholders, including the public, about changes to technical guidance and reporting processes. Treasury’s stakeholder engagement process includes regular review of and revisions to its technical guidance. Before revisions to guidance are put into effect, Treasury staff circulate proposed changes through an email list that any member of the public can subscribe to, discuss these changes at frequent meetings with federal agency staff responsible for DATA Act reporting, and provide opportunities for agencies to test reporting under the new rules and provide feedback from this testing to Treasury. In addition, the guidance documents provide logs of all changes that have been made since the documents were created. According to Treasury staff, the most important tools for ensuring that agencies report consistent and comparable data are the DATA Act Information Model Schema (DAIMS) and the DATA Act Broker. Treasury’s documentation states that the DAIMS is “the data standard of the DATA Act” and contains standardized data elements that are complete and reflect the requirements of the act. The DAIMS includes reporting guidance that provides agencies with a complete listing of data elements they must report as well as a complete listing of data elements that will be extracted from government-wide systems, such as FPDS-NG. The DAIMS also includes a validation rules document that describes the business rules employed by the DATA Act Broker, which is Treasury’s system for collecting and validating agency data. Treasury provides federal agencies with detailed procedures for submitting DATA Act data to the broker. Before making changes to the DAIMS and DATA Act Broker, Treasury provides stakeholders with information about the planned changes and an opportunity to comment on them. For example, in June 2018, Treasury released DAIMS 1.3, an updated version of the DAIMS to be implemented during fiscal year 2019. Before releasing the final version of DAIMS 1.3, on June 29, 2018, Treasury shared its plans for the release with stakeholders through the Chief Financial Officers Council’s DATA Act Working Group (CFOC Working Group) and office hours calls. Treasury also transmitted a notice of proposed changes to federal agencies, collected comments from agencies during a designated comment period, and included responses to these comments in the final version of the release. Before implementing any of the changes in DAIMS 1.3 in the DATA Act Broker, Treasury provided agencies with a testing environment that allowed agency staff to identify any issues with the changes before those changes were applied to data published on USAspending.gov. Treasury’s documentation for the public and for federal agencies includes detailed information about the history of changes to the DAIMS. Each of the DAIMS guidance documents includes a change log that shows revisions made since the document was first created. The detailed information Treasury provides about changes to technical guidance can promote data quality and transparency by ensuring that federal staff are aware of reporting requirements, and that users of the data understand how those data are created and reported. Since 2014, OMB and Treasury have made significant strides to address the DATA Act’s requirements for standardization of federal spending data. As they move forward, appropriately and effectively managing changes to data standards will be critical to ensuring the quality and comparability of the data across the federal government. Treasury has instituted regular procedures for making changes to technical data standards, including procedures for consulting with stakeholders and for recording and communicating changes. OMB has taken responsibility for maintaining an official list of data definition standards separate from the technical data standards maintained by Treasury. However, OMB lacks comparable procedures for maintaining these data definition standards. OMB made changes to some of the definitions and clarified policies about how they are to be applied, but did not communicate those changes to stakeholders, including the public. Definitions of data elements and policies about how those definitions are to be applied are a key component of the management of federal spending data under the DATA Act. Although OMB has completed the task of creating an initial set of definitions, it has not formally and explicitly documented a consistent approach for maintaining the integrity of the data definition standards over time as we previously recommended. Until OMB establishes procedures to ensure that changes are controlled, it will continue to be a challenge to apply and interpret these definitions consistently, presenting risks to data quality. In addition, clearly identifying the changes that have already been made in the official repository could aid agency officials in reporting data and users in understanding the context in which past data have been reported. These actions would be important steps toward improving control over the data standards, creating an effective governance structure, and ultimately improving the consistency and quality of federal spending data. We are making two recommendations to the Office of Management and Budget: The Director of OMB should clarify and document OMB’s procedure for changing official data definition standards for DATA Act reporting, for example, by explicitly describing how change procedures developed for other government-wide initiatives apply to DATA Act definition standards in a public source of guidance or information. (Recommendation 1) The Director of OMB should ensure that the June 2018 policy changes regarding DATA Act data definition standards are clearly identified and explained in the official repository or another authoritative public source of DATA Act standards and guidance, such as by including a revision history along with the current version of the definitions. (Recommendation 2) We provided a draft of this report to OMB and Treasury for review and comment. OMB neither agreed nor disagreed with our recommendations, and OMB staff from the Office of Federal Financial Management provided oral comments, which are summarized below and incorporated as appropriate in the report. Treasury informed us that they had no comments on the draft report. In their oral comments, OMB staff stated that on the whole, the report correctly described the complex ecosystem of governance over data standards for federal spending data. However, OMB staff stated that in certain places the report did not fully capture the extent of OMB’s actions related to data governance for the DATA Act data standards. According to OMB staff, the Shared Solutions Governance Board (SSGB), under OMB’s direction, plays an important role in governing DATA Act data standards. OMB staff said that this relationship exists because the same agencies and staff participate in both the SSGB and the governance of the DATA Act data standards. In addition, OMB staff confirmed that descriptions of the specific roles and responsibilities of the SSGB, CFOC Working Group, and the Treasury office that manages the DAIMS have not been documented. OMB staff said that many of the same agency personnel participate in all of these groups, and therefore work closely together on a regular basis. OMB staff stated that this close involvement results in effective communication and a consistent approach to governance, and ensures an understanding of the procedures for changing data standards even though those procedures are not formally documented. We acknowledge OMB’s assertion that the various groups for creating and administering government-wide data standards (including data standards established to support the DATA Act) share many of the same staff. However, OMB’s approach relies on the continued participation of the same staff in order to maintain continuity rather than relying on documented policies, procedures, roles, and responsibilities for data governance functions. A key benefit of having a robust, institutionalized data governance structure is to provide consistent data management during times of change and transition, such as during staffing transitions or administration changes. It is important for OMB to clearly delineate roles and responsibilities so stakeholders understand how governance of the DATA Act standards is accomplished within the broader context of the PMA and other efforts. OMB staff also said they have communicated all changes to DATA Act data standards that have been made since the standards were created. OMB staff told us that the DAIMS is the official data standard for DATA Act reporting and, as such, includes logs that record all changes to the standards. According to OMB staff, OMB updated its public data standards web page on www.max.gov in June 2018 to fix an error and ensure that the page matched the DAIMS. Staff stated their belief that such a correction did not represent an actual change to a data standard and therefore did not need to be recorded in the DAIMS change log or communicated publicly. However, guidance issued in June 2018, OMB Memorandum M-18-16, identifies the MAX.gov web page as the official repository of the data standards. Specifically, the guidance directs agencies to report data in accordance with the standards maintained by OMB and Treasury pursuant to FFATA, as amended by the DATA Act, and provides a link to the OFFM website’s listing of data standards definitions. If OMB chose to identify the DAIMS—instead of the MAX.gov page—as the official source of data standards, then the issue about changes not being identified on the MAX.gov page would not be important. Although OMB made conforming changes based on our input to align the definition of Primary Place of Performance on the MAX.gov web page, having clearly documented procedures for making changes to the data standards and for ensuring that changes are communicated widely is important for ensuring the consistent and comparable reporting envisioned under the act. Additionally, in June 2018, OMB made an important change to the explanatory text that precedes these official data definitions as posted on the MAX.gov website, clarifying OMB’s policy regarding the use of the DATA Act data definitions. OMB staff acknowledged that that this clarification could have been publicized more effectively, which is why we continue to believe that our second recommendation—to include a revision history along with the current version of the DATA Act data definitions—remains valid. We are sending copies of this report to the Secretary of the Treasury and the Acting Director of OMB, as well as interested congressional committees and other interested parties. This report will be available at no charge on our website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Michelle Sager at 202-512-6806 or SagerM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. Key contributors to this report are listed in appendix III. Appendix I: Interagency Groups Responsible for DATA Act Governance GitHub is a web-based software repository hosting service. The Federal Spending Transparency website can be found at: http://fedspendingtransparency.github.io/. JIRA is an online software development tool that Treasury uses to provide responses to stakeholder questions and comments related to the development of the broker. In addition to the contact named above, Peter Del Toro, Assistant Director, and Kathleen M. Drennan, Analyst-in-Charge, supervised the development of this report. Theodore Alexander and Sherrice Kerns made major contributions to this report. Also contributing to this report in their areas of expertise were David M. Ballard, Ann Czapiewski, Jenny Chanley, Robert Gebhart, Michael LaForge, Carl Ramirez, Andrew J. Stephens, and James Sweetman, Jr. Open Data: Treasury Could Better Align USAspending.gov with Key Practices and Search Requirements. GAO-19-72. Washington, D.C.: December 13, 2018. DATA Act: Reported Quality of Agencies’ Spending Data Reviewed by OIGs Varied Because of Government-wide and Agency Issues. GAO-18-546. Washington, D.C.: July 23, 2018. DATA Act: OMB, Treasury, and Agencies Need to Improve Completeness and Accuracy of Spending Data and Disclose Limitations. GAO-18-138. Washington, D.C.: November 8, 2017. DATA Act: As Reporting Deadline Nears, Challenges Remain That Will Affect Data Quality. GAO-17-496. Washington, D.C.: April 28, 2017. DATA Act: Office of Inspector General Reports Help Identify Agencies’ Implementation Challenges. GAO-17-460. Washington, D.C.: April 26, 2017. DATA Act: Implementation Progresses but Challenges Remain. GAO-17-282T. Washington, D.C.: December 8, 2016. DATA Act: OMB and Treasury Have Issued Additional Guidance and Have Improved Pilot Design but Implementation Challenges Remain. GAO-17-156. Washington, D.C.: December 8, 2016. DATA Act: Initial Observations on Technical Implementation. GAO-16-824R. Washington, D.C.: August 3, 2016. DATA Act: Improvements Needed in Reviewing Agency Implementation Plans and Monitoring Progress. GAO-16-698. Washington, D.C.: July 29, 2016. DATA Act: Section 5 Pilot Design Issues Need to Be Addressed to Meet Goal of Reducing Recipient Reporting Burden. GAO-16-438. Washington, D.C.: April 19, 2016. DATA Act: Progress Made but Significant Challenges Must Be Addressed to Ensure Full and Effective Implementation. GAO-16-556T. Washington, D.C.: April 19, 2016. DATA Act: Data Standards Established, but More Complete and Timely Guidance Is Needed to Ensure Effective Implementation. GAO-16-261. Washington, D.C.: January 29, 2016. DATA Act: Progress Made in Initial Implementation but Challenges Must Be Addressed as Efforts Proceed. GAO-15-752T. Washington, D.C.: July 29, 2015. Federal Data Transparency: Effective Implementation of the DATA Act Would Help Address Government-wide Management Challenges and Improve Oversight. GAO-15-241T. Washington, D.C.: December 3, 2014. Government Efficiency and Effectiveness: Inconsistent Definitions and Information Limit the Usefulness of Federal Program Inventories. GAO-15-83. Washington, D.C.: October 31, 2014. Data Transparency: Oversight Needed to Address Underreporting and Inconsistencies on Federal Award Website. GAO-14-476. Washington, D.C.: June 30, 2014. Federal Data Transparency: Opportunities Remain to Incorporate Lessons Learned as Availability of Spending Data Increases. GAO-13-758. Washington, D.C.: September 12, 2013. Government Transparency: Efforts to Improve Information on Federal Spending. GAO-12-913T. Washington, D.C.: July 18, 2012. Electronic Government: Implementation of the Federal Funding Accountability and Transparency Act of 2006. GAO-10-365. Washington, D.C.: March 12, 2010.", "summary": "The DATA Act required OMB and Treasury to establish data standards for the reporting of federal government spending and required agencies to report spending data using these standards beginning in May 2017. GAO's prior work examining the quality of the data reported under the act found significant challenges that limit the usefulness of the data for Congress and the public. These data quality challenges underscore the need for OMB and Treasury to make progress on addressing GAO's prior recommendation to establish a set of clear policies and processes for developing and maintaining data standards. The DATA Act includes a provision for GAO to report on the implementation and use of data standards, and on the quality of the data reported using those standards. This report (1) describes the status of OMB's and Treasury's efforts to establish policies and procedures for governing data standards; and (2) evaluates the extent to which procedures for changing established data standards are consistent with key practices for data governance. The Office of Management and Budget (OMB) and the Department of the Treasury (Treasury) have established some procedures for governing the data standards established under the Digital Accountability and Transparency Act of 2014 (DATA Act), but a formal governance structure has yet to be fully developed. Since enactment, OMB has relied on a shifting array of advisory bodies to obtain input on data standards. As of December 2018, some governance procedures are in place, but others continue to evolve. OMB staff told us that the governing bodies involved in initial implementation efforts had been disbanded, and that the functions previously performed by these advisory bodies over governance of DATA Act data standards would be accomplished within the broader context of the cross-agency priority goals established under the 2018 President's Management Agenda (PMA). However, the documentation of the governance structure established for these goals does not make explicit how it would apply to the data standards established under the DATA Act. Clarifying the connection between this governance structure and the DATA Act could help stakeholders understand how governance of the DATA Act standards is accomplished within the broader context of the PMA. With regard to one specific data governance function—making changes to existing standards—GAO found that OMB does not have procedures for managing changes to the web page it identifies in guidance as the authoritative source for data definition standards. The DATA Act requires, to the extent reasonable and practicable, that data standards be capable of being continually upgraded. In addition, key practices for data governance state that organizations should document policies and procedures for making decisions about changes to standards. In June 2018, revisions were made to the Primary Place of Performance Address data element without following a documented process. OMB staff described these revisions as minor technical corrections to align the definitions with the technical guidance agencies were already using to report data. However, without documented procedures for revising the definitions, needed changes may not be made in a timely manner, which could lead to inconsistent reporting. OMB also did not transparently communicate to stakeholders these changes to data definition standards. Along with the corrections to definitions, in June 2018 OMB changed introductory text on the data definitions web page to clarify policy about how agencies should use DATA Act definitions. However, OMB did not publicly announce this clarification or identify on the website that changes had been made. Without transparent communication of changes to data definition standards, stakeholders—including staff at federal agencies required to report data according to these definitions—may miss important information relating to changes in how, when, and by whom data definitions are to be applied. Although OMB lacks procedures governing changes to DATA Act data definitions, Treasury has established a process for changing related technical guidance in consultation with stakeholders. Treasury's procedures contribute to the objectives of data quality and transparency by helping to ensure that agencies are aware of reporting requirements and users understand how those data are created and reported. GAO makes two recommendations to OMB to (1) document its procedure for changing data definition standards for DATA Act reporting, and (2) ensure that changes made in June 2018 to clarify policy regarding data definitions are identified in an authoritative public source of DATA Act standards and guidance. OMB neither agreed nor disagreed with the recommendations, but provided comments, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "VA provides or pays for nursing home care through three separate programs, one for each of the nursing home settings in which VA provides or pays for care. In general, the three settings provide similar nursing home care, in which veterans receive skilled nursing care, recreational activities, and other services. However, some of the nursing homes may provide care to veterans on a short-term basis, such as rehabilitation after a hospitalization for a period of 90 days or less (“short stay”), or on a long-term basis, which is a period of 91 days or more (“long stay”). Further, officials told us that some of these homes may also provide certain special needs care for a limited number of residents, such as dementia or rehabilitative care, which may require additional specialized equipment or trained staff. Federal oversight of care provided to veterans within the three settings is conducted by VA only or a combination of VA and CMS. See table 1 for key characteristics on the three nursing home settings. Depending on a veteran’s eligibility status, VA pays the full or partial cost of nursing home care in each setting. For example, VA is required by law to provide the full cost of nursing home care for veterans who need nursing home care for a service-connected disability—which is an injury or disease that was incurred or aggravated while on active duty—and for veterans with service-connected disabilities rated at 70 percent or more. For all other veterans, VA provided nursing home care is based on available resources. Veterans and their families are responsible for making decisions about nursing home care that will best meet their needs. At the national level, VA provides information about nursing homes on its Access to Care website; according to VA, the website is intended to help inform veterans and their families’ about the quality of care in nursing homes. According to VA central office officials, the responsibility for helping veterans make decisions about nursing home care is decentralized to local VAMCs. In consultation with veterans and their families, VAMC social workers and clinical care providers can discuss factors such as the veteran’s eligibility for care in each setting, health needs, the type of care provided at different homes, space availability, and the veteran’s geographic preference. VAMC staff may also encourage veterans to take a tour of the prospective home. VA models its oversight of nursing home services provided to veterans on the methods used by CMS. CMS defines the quality standards that approximately 15,600 nursing homes nationwide must meet in order to participate in the Medicare and Medicaid programs. To monitor compliance with these standards, CMS contracts with state survey agencies to conduct inspections of each home not less than once every 15 months. During these inspections the state survey agency might identify deficiencies—or instances in which the nursing home does not meet an applicable quality standard. To address identified deficiencies, CMS generally requires nursing homes to implement corrective action plans. CMS also monitors—by conducting observational assessments of state agencies during inspections or conducting its own comparison inspections on a sample of homes each year—the state agencies that inspect CNHs to ensure that these inspections accurately identify whether the homes meet quality standards. In addition, CMS collects data on various clinical quality measures and calculates nursing home staffing ratios. CMS assigns each nursing home ratings in three components— inspections, quality measures, and staffing ratios—and an overall quality rating. CMS places the greatest weight on inspections in its calculations of each home’s overall quality rating. CMS publicly reports a summary of the information it collects on the quality of nursing homes on its Nursing Home Compare website, which uses a five-star quality rating system. As we previously reported, this website facilitates public comparison of nursing home quality. Within VA central office, the Office of Geriatrics and Extended Care is responsible for overseeing the quality of nursing home care provided to veterans in each of the three settings—CLCs, SVHs, and CNHs. The key mechanism VA uses to assess quality in each of these settings is regular inspections—generally occurring annually—that determine the extent to which homes meet relevant quality standards. VA’s use of inspections and other methods to ensure the quality of care in each of the three nursing home settings differs: CLCs. VA owns, operates, and oversees the quality of CLCs, and conducts regular unannounced inspections to determine the extent to which CLCs meet quality standards. VA central office contracts with the Long Term Care Institute to conduct these inspections, and VA central office reviews the results of all inspections. CLCs receive an initial inspection when they open and then periodic, unannounced inspections thereafter. The frequency of these inspections depends on the number and severity of deficiencies identified during the prior year’s inspection, but they generally occur every 11 to 13 months. CLCs are required to develop and implement corrective action plans for each deficiency identified that detail how it will be addressed. VA central office approves these plans, and the VISN and VA central office monitor the CLC’s actions until each deficiency is addressed. Per VA’s contract, VA monitors the Long Term Care Institute to ensure that inspections are conducted within required timeframes and to conduct quarterly assessments of the contractor’s performance, among other things. In addition, for each CLC, VA also collects information on quality measures and staffing ratios and uses this information, along with the inspection results, to assign a star rating from 1 to 5 stars. In June 2018, VA central office consolidated the ratings for all of the individual CLCs—modeled after CMS’s Nursing Home Compare—into its Access to Care website. SVHs. States own and operate SVHs and, as a result, in most cases SVHs are inspected by state agencies to determine the extent of their compliance with state requirements. About two-thirds of SVHs are inspected by CMS; however, VA is the only entity that conducts annual inspections for all SVHs. Although, VA does not exercise any supervision or control over the administration, personnel, maintenance, or operation of any state home, VA conducts these annual reviews for all SVHs and is prohibited from making payments to SVHs until it determines that they meet applicable quality standards. VA central office contracts with Ascellon to conduct these inspections and reviews the results of the inspections. The inspections first occur when an SVH initially seeks to become eligible for VA payments, and, once the SVH is eligible, unannounced inspections occur on an annual basis to verify that an SVH is eligible to continue to receive VA payments. For these annual inspections, the contractor generally cites deficiencies when SVHs are not in compliance with applicable quality standards. SVHs develop and implement corrective action plans for each deficiency identified, and the VAMC director approves the plan. VA should monitor the contractor’s performance annually, for example, to ensure that inspections are conducted within certain timeframes. VA’s Office of Geriatrics and Extended Care maintains a database of all corrective action plans, and VISN and VAMC staff monitor the SVHs’ actions until each deficiency is addressed. VA also collects VA prescribed quality measure and staffing data from SVHs as part of its survey process. However, VA does not currently assign a quality rating to SVHs. CNHs. CNHs can be publicly or privately owned and operated, and, CMS provides federal oversight for all CNHs that receive Medicare or Medicaid payments. VA requires CNHs under contract to be certified by CMS, and, unlike the other two settings, VA is not required to conduct regular inspections of CNHs. Instead, VA requires VAMC staff to conduct veteran care assessments on a monthly basis and annually review information CMS collects on the homes’ quality, including CMS inspection results, to evaluate whether to initiate or continue a contract with a CNH. The annual reviews use seven criteria established by VA’s Office of Geriatrics and Extended Care, including whether the CNH’s total number of health deficiencies from the most recent CMS inspection is twice the average of the state in which it is located. According to VA officials, CNHs that fail to meet four out of VA’s seven criteria during the annual reviews of CMS data are excluded from participation in its CNH program unless the VAMC seeks a waiver from VA central office to allow the home to participate. If VAMC staff are considering seeking a waiver to allow a CNH to continue participating in the CNH program, or have any other concerns about a home, they have the option of conducting their own onsite reviews of the home to assess care quality. Our analysis of VA data shows that veterans’ utilization of VA nursing home care—across CLCs, SVHs, and CNHs—increased 3 percent from fiscal year 2012 through 2017, from an average daily census of 37,687 to 38,880 veterans. VA projects that nursing home utilization will increase another 16 percent, to an average of 45,279 per day by fiscal year 2022, with varying increases projected for each of the nursing home settings. (See fig. 1.) Moreover, VA projects that overall demand for VA nursing home care will continue to increase through 2034, driven by the aging of the cohort of Vietnam War veterans. VA projects that Vietnam veterans will increasingly rely on VA’s health care system for care and will use more health care services, including nursing home care. As figure 1 shows, SVHs accounted for the largest percentage (53 percent) of the average number of veterans who received nursing home care each day in fiscal year 2017. However, the number of veterans in CNHs has increased and is projected to continue to increase. For example, the average number of veterans receiving nursing home care in CNHs increased 35 percent from fiscal year 2012 to 2017, from an average of 6,875 to 9,251 per day. Over the same period, the number of veterans in CLCs fell 9 percent, and in SVHs it fell 1 percent. VA officials told us that they are prioritizing the use of CLCs for short-term care, and that CNHs have the greatest capacity to meet the future long-term needs of veterans. VA projects that by 2034 the number of veterans receiving nursing care in these homes will exceed 17,000. In addition, VA projects that demand for nursing home care in CLCs and CNHs will decrease after 2034, and VA has not projected care in SVHs beyond 2022. VA officials also said that VA has limited flexibility to expand the number of beds in CLCs and SVHs to accommodate the projected number of veterans needing care. While VA expects to continue placing more of the veterans needing nursing home care into CNHs, officials noted some challenges contracting with these homes. Specifically, VA central office officials said that about 600 CNHs had decided to end their contracts with VA over the last few years for a variety of reasons. For example, officials from four of the VAMCs we interviewed told us about CNH concerns that contract approvals can take 2 years, homes have difficulties meeting VA staff requirements, and VA’s payment rates were very low. Officials said provisions in the VA MISSION Act of 2018 may alleviate some of these difficulties. Specifically, the Act consolidates various VA community care programs into the Veterans Community Care Program and authorizes VA to enter into veterans care agreements with certain providers, including nursing homes. In contrast to contracts, such agreements may not require providers to meet certain wage and benefit requirements. Officials told us that they are in the process of replacing CNH contracts with veterans care agreements, which may alleviate some of those challenges. In addition, VA officials told us that most nursing homes—including homes in each of the three settings—have limited capacity to serve veterans with special needs, such as those needing dementia, ventilator, or behavioral care. For example, they said that homes may not have any of the necessary specialized equipment or trained staff, or may not have as many of these beds as needed, to meet certain veterans’ special care needs. VA officials told us that they are working to expand the availability of special needs care in each of the three settings. Our analysis of VA data also shows that VA nursing home care expenditures have increased in recent years, reflecting increases in the number of veterans receiving such care. Specifically, VA’s nursing home expenditures across all three settings increased 17 percent from fiscal years 2012 through 2017, from $4.9 billion to $5.7 billion. These expenditures are expected to increase to $7.3 billion in fiscal year 2022 as utilization is projected to increase. VA officials told us that expenditures for nursing home care are projected to increase due to the rising costs of care as well as higher utilization of services. (See fig. 2.) Of the three settings, CLCs accounted for the largest share of VA nursing home expenditures; however, this reflects differences in the costs of care and the extent to which VA pays for these costs in each of these settings: For CLCs, VA pays the full cost of care for veterans in these homes and, according to VA officials, VA expenditures for care provided in CLCs are greater compared to the other settings, because CLCs are able to provide acute care that requires higher staffing levels and more specialized equipment. In addition, VA officials indicated that CLC expenditures also include the overhead costs of being associated with VAMC hospitals. For SVHs, 80 percent of veterans receive VA’s partial daily rate that covers only about a quarter of their care costs. For example, in fiscal year 2017, VA’s average SVH per diem was $106 for veterans without eligible service connected disabilities. VA also pays the full cost of care for the remaining 20 percent of veterans with service-connected disabilities. In fiscal year 2017, the full rate for these veterans was $397 per day. For CNHs, VA pays the full cost of care for veterans; however, more of these veterans receive long-term care, at a lower cost per day, than the short-term care that many veterans receive in CLCs, such as for rehabilitation after surgery, at a higher cost per day. As a result of these differences, in fiscal year 2017, VA paid, on average, $1,074 per day per veteran for care in CLCs, $268 for CNHs, and $166 for SVHs. During the contract year completed in 2018, VA’s two contractors conducted the required annual inspections of CLCs and SVHs to determine the extent to which the homes met quality standards. However, VA has opportunities to enhance its oversight of the contractors’ inspections by regularly monitoring both contractors’ performance inspecting CLCs and SVHs through observational assessments and by citing all SVH deficiencies. Although VA’s plans call for quarterly observational assessments, they have not been consistently conducted and documented. Similarly, VA has not provided guidance for the optional onsite reviews of CNHs that VAMCs may perform thus limiting their potential impact. Our review found that during the contract year completed in 2018, VA’s CLC contractor performed the required annual inspections for 126 CLCs. (See table 2.) Through these inspections, VA’s contractor determined the extent to which each CLC met applicable quality standards and issued deficiencies when standards were not met. The most common areas of deficiencies were those in which 1) the facility did not provide quality care for its residents, for example, in its treatment and prevention of pressure ulcers or managing its residents’ pain; 2) the facility did not adequately prevent and control infections, for example, by providing residents influenza and pneumococcal immunizations; and 3) the facility did not provide adequate care and services to sustain the highest possible quality of life for its residents, for example, by providing residents unable to carry out activities of daily living with adequate assistance to maintain good nutrition, grooming, and personal and oral hygiene. (See appendix I for more information on the types of deficiencies identified.) To address deficiencies, VA required CLCs to produce corrective action plans and tracked the CLCs’ progress until the deficiencies were resolved. In addition, for some of the most common deficiencies among CLCs, VA officials said VA took steps such as developing additional VAMC policies to facilitate improvement. For example, to reduce the number of CLC deficiencies related to pain management and improve CLCs’ performance in this area, VA officials said they developed specific guidelines for CLCs to use to assess pain in patients with dementia who were unable to provide numeric pain scores. While VA has monitored and determined that CLC inspections occurred as stipulated in its contract and tracked the results of the inspections, it has an opportunity to enhance its oversight. According to its contract, VA will monitor contractor performance on a quarterly basis, and VA central office officials told us their intention has been to meet this stipulation by observing the contractor as it conducts some inspections—an approach consistent with CMS’s inspection oversight process. However, VA officials told us that they have not been completing these observations quarterly and did not conduct any observations for the April 2017 to April 2018 contract year. VA officials said they had not performed this quarterly observation due to competing demands. For example, the three- person team at VA central office responsible for CLC oversight has overseen a number of recent initiatives, including the rollout of CLC quality ratings in 2018. Officials also told us they conducted one observation for the current contract year in December 2018 (during the course of our review). However, we were not able to confirm the December 2018 observation or any other observations of the CLC inspections because VA has not documented the results. A VA official said that developing an approach for documenting the quarterly observations is something VA needs to work on. VA’s failure to monitor the CLC contractor’s performance through observational assessments is inconsistent with its own goals of assessing the contractor’s performance quarterly and modeling its oversight after CMS’s approach to its own contractors’ inspections. It is also inconsistent with federal internal control standards that state that management should establish and operate monitoring activities to monitor the internal control system and evaluate the results. By not conducting these quarterly observations for more than a year, VA does not know whether, or to what extent, the contractor is effectively assessing CLC compliance with quality standards and is unable to hold the contractor accountable for its inspections. Without effective monitoring of the contractor’s performance inspecting CLCs, VA risks that quality concerns in some CLCs could go overlooked, placing veterans at risk. Our review found that during the contract year completed in 2018, VA’s SVH contractor performed required annual inspections for all 148 SVHs. (See table 3.) As with CLCs, VA’s SVH contractor determined through these inspections the extent to which each SVH met applicable quality standards and cited deficiencies when they were not met. The most common areas of deficiencies were those in which 1) the facility’s physical environment did not adequately protect the health and safety of its residents, for example, by ensuring their safety from fires; 2) the facility did not provide quality care for its residents, for example, by adequately managing their pain; and 3) the facility did not assess residents’ health sufficiently, for example, within 14 days of residents’ admission and on an annual basis thereafter. (See appendix II for more information on the types of deficiencies identified.) To address deficiencies, VA required SVHs to produce corrective action plans and tracked the SVH’s progress until they were resolved. In addition, VA officials said they took steps to address deficiencies common among SVHs. For example, to reduce SVH deficiencies related to physical environment standards for fire safety and improve SVH performance in this area, VA central office staff told us they held SVH town halls with a fire safety engineer and created reference guides for SVH administrators about regulatory changes in fire safety codes. However, while VA has monitored that its contractor conducted the required SVH inspections and tracked the results of these inspections, VA has not monitored the SVH contractor’s performance of these inspections through regular observational assessments to ensure that contractor staff effectively determine whether SVHs are meeting required standards. Specifically, VA officials told us they intended to observe the SVH contractor’s inspections on a quarterly basis, which would be consistent with VA’s approach to CLCs and its goal of modeling its oversight on CMS’s. VA officials told us that although they have a goal of performing this monitoring on a quarterly basis; they could not recall when VA last observed the SVH contractor’s inspections. When asked, VA officials did not provide specific reasons why they had not performed the observational assessments; in prior discussions, these officials noted that VA’s oversight of SVHs is less involved than its oversight of CLCs because VA does not exercise any supervision or control over the administration, personnel, maintenance, or operation of any state home. However, VA pays for veterans to receive care in SVHs, and states that oversee these homes may or may not conduct their own oversight. Furthermore, as CMS conducts oversight of only those SVHs that receive Medicare or Medicaid payments (about two-thirds of all SVHs), for some SVHs, VA is the only federal agency with oversight over the quality of those homes care. For example, VA is the only entity that conducts regular inspections of SVHs in Missouri and New Hampshire. VA is missing another opportunity to enhance its oversight of SVHs by not requiring the SVH contractor to identify all failures to meet quality standards as deficiencies during its inspections. While CMS requires its inspectors to cite all deficiencies, VA directed its contractor to cite low- level deficiencies—deficiencies considered by the contractor to pose no actual harm but with potential for minimal harm—as “recommendations” rather than deficiencies. For example, during one SVH inspection, the contractor recommended that “to ensure nutritional adequacy, the facility should follow the menus, which are planned in advance.” VA officials told us that unlike deficiencies, they do not track or monitor the nature of the recommendations or whether the recommendations have been implemented. In contrast, state survey agencies under contract with CMS are required to cite all failures to meet quality standards as deficiencies. In addition to not citing recommendations as deficiencies, according to the VA contractor’s 2016-2017 annual summary report, SVHs can fix issues identified by the SVH contractor while the inspectors are still onsite to avoid being cited on the inspection. As a result, these issues are also not documented as deficiencies. Officials at four of the six SVHs we interviewed specifically reported being able to make on-site corrections to avoid being cited for deficiencies—for instance, officials at one SVH told us that the SVH was able to relocate handwashing stations before the end of the inspection in order to avoid being cited for a deficiency by the VA inspectors. According to VA, VA does not require its SVH contractor to identify all failures to meet quality standards as deficiencies in its inspections, VA officials said this practice reflects policy and a negotiated position with SVHs. VA officials reiterated that because SVHs are owned and operated by the states, VA is less involved with their oversight than CLCs. Our review of the VA contractor’s annual summary report showed that almost 50 percent of SVHs inspected between August 2017 and July 2018 (the contract year completed in 2018), zero deficiencies were identified through inspections. VA officials cited VA’s ‘collegial approach’ and willingness to make onsite corrections as factors contributing to the decline in recent years. Furthermore, while VA and CMS subject SVHs to slightly different standards, our review of VA and CMS inspection reports from a sample of five SVH inspection reports shows that VA identified a total of seven deficiencies and made four recommendations from these homes. In contrast, CMS identified a total of 33 deficiencies for these homes for approximately the same time period. By not performing observational assessments of SVH inspections, VA does not know whether, or to what extent, VA’s contractor needs to improve its ability to identify SVHs’ compliance with quality standards, which increases the possibility that quality concerns in some SVHs could go overlooked, potentially placing veterans at risk. Further, by not requiring the contractor to cite all failures to meet quality standards as deficiencies on its inspections, VA does not have complete information on deficiencies identified at SVHs and therefore cannot track this information to help identify trends in quality across these homes. Further, it is inconsistent with federal internal control standards that state that management should establish and operate monitoring activities to monitor the internal control system and evaluate the results. We found that in 2017 the six selected VAMCs annually reviewed CMS data on the quality of all the CNHs with which they contract, which is a VA requirement. Specifically, the VAMCs reviewed the CMS data to determine whether the CNHs met VA criteria for contract renewal. (See table 4.) The top three criteria from the annual reviews that VAMCs failed to meet were 1) whether total registered nursing staff ratios per resident day fell below the state average, 2) whether total nursing staff ratios per resident day fell below the state average; and 3) whether six or more of selected CMS quality measures fell above the state average. In addition, we found that all six of our selected VAMCs conducted their own onsite CNH reviews—which, according to VA policy, VAMC officials have the option of performing if they have quality concerns about CNHs with which they contract or are determining whether to seek a waiver. The CNH onsite reviews conducted by these VAMCs focused on many of the categories for quality standards, such as food and nutrition services, quality of care, quality of life, and physical environment. While conducting onsite reviews of CNHs is optional under VA policy, officials at many of the VAMCs we interviewed told us that these onsite reviews—which the VAMCs we interviewed referred to as CNH inspections—are valuable in conducting CNH oversight as they provide important information about a home’s quality that VAMC staff would not have known otherwise. For example, officials from one VAMC shared with us results from an onsite review in which they found moldy and expired food in a CNH’s kitchen— food storage had been identified as an issue during a previous state survey for CMS and was purported to have been corrected 5 months prior. Furthermore, some VAMC staff said that they would suspend placement of veterans in certain CNHs and may not renew a CNH contract based on their findings from these onsite reviews. However, VA could strengthen its support for the optional onsite reviews by providing guidance to VAMC staff conducting these reviews. Officials at some VAMCs expressed concerns that VA did not provide the guidance they needed to conduct the optional onsite reviews, and that they would like to have more information from VA’s central office. As one VAMC official said, “without training or guidance from VA , it is difficult for VAMC staff, especially new staff, to know how to conduct these inspections.” VAMC officials at the six selected VAMCs told us that in the absence of guidance from VA, they had each independently developed their own tools and processes. Furthermore, officials at these VAMCs had differing understandings of the steps they can take if they identify quality concerns during onsite reviews. For example, staff at some VAMCs required CNHs to write corrective action plans and monitored the CNHs’ implementation until the deficiencies were addressed; in contrast, staff at other VAMCs did not monitor implementation, because they did not think they had the authority to hold CNHs accountable to correct deficiencies they identified. VA central office officials who oversee the CNH program told us that they do not provide training or guidance because CMS and the states, not VA, are responsible for regulating the quality of care in these nursing homes. However, in the absence of guidance from VA central office on the optional CNH onsite reviews—guidance that could be developed, for example, by collecting and disseminating best practices—VA has missed an opportunity to leverage efficiencies across VA’s network of VAMCs and empower VAMC officials with knowledge about the steps they can take to hold CNHs accountable for correcting problems. Furthermore, it is inconsistent with federal internal control standards that state that management should design control activities to achieve its objectives—in this case, to ensure that VAMCs contract with CNHs that provide high quality care. As part of its efforts to help veterans find placement into a nursing home, VA publicly provides information on care quality for CLCs and CNHs through its Access to Care website, but VA does not provide information on the quality of SVHs. Specifically, the website allows users to enter a location—such as a city and a surrounding distance—to produce a map with a list of CLCs and VA-contracted CNHs in their preferred area (see fig. 3). For each of the homes on the list, VA reports quality information it collects through its own inspections for CLCs and information CMS collects for CNHs. As previously noted, veterans and their families are responsible for making decisions about the nursing home care that will best meet their needs. Their decision-making can be aided by discussions with VAMC staff and information provided on VA’s Access to Care website, among other sources. The ability for veterans and their families to access information on nursing home quality through the Access to Care website—such as the currently available quality information on CLCs and CNHs—is particularly critical as VAMC officials do not always discuss quality information in their consultations with veterans and their families. As figure 3 shows, VA’s Access to Care website does not provide any information to the public about the quality of the 148 SVHs that provide nursing home care. Specifically, VA does not currently provide any information on SVHs on its Access to Care website—including information on the location of SVHs or CMS information on care quality that VA could easily provide on SVHs using information obtained from CMS’s website, Nursing Home Compare, as VA does now for CNHs. VA has explored activities that could provide veterans and their families with information about SVHs. For example, as stated in VA’s SVH strategic plan for fiscal years 2017 to 2022, VA considered an initiative to create a five-star program for SVHs. Additionally, VA has collaborated with SVHs to produce some data on quality measures. For example, during the course of this review, VA provided to us a quality measures report for SVHs by state that they developed in partnership with the National Association of State Veterans Homes. VA is able to develop this information because it has access to information on SVH quality—in fact, as the only entity that conducts regular inspections, it is the only source for quality information on all SVHs. Specifically, VA collects VA prescribed inspection, quality measure, and staffing data as part of its survey process that could be used to develop and distribute quality information for each home. Some of this information is available to the public at the local level, but it is not currently provided by VA. For example, SVHs are required to make the results of the most recent VA inspection of the home available for examination in a place accessible to residents. According to VA officials, there is no requirement to provide information on SVH quality on the Access to Care website, as SVHs are owned and operated by the states. However, the website is an important tool for veterans and their families to help inform their decision making on nursing home placement. VA has stated goals to provide useful and understandable information to veterans. The VA website could be the only readily accessible source of quality care information publicly available to veterans and their families for certain SVHs. As the SVH strategic plan indicates, VA sees the value in developing SVH ratings that could be used to provide quality information to veterans and their families. Furthermore, officials from three of the SVHs we spoke with told us that they supported having quality information available about their homes that would allow comparisons between SVHs or between SVHs and other homes, such as information contained in Nursing Home Compare. Without information about SVHs on VA’s Access to Care website, veterans and their families are limited in their ability to effectively evaluate all of their options when selecting a nursing home. Our prior work has shown that effective transparency tools—such as websites that allow consumers to compare the quality of different providers—provide highly relevant information to consumers. However, the limited information VA provides on its Access to Care website is inconsistent with VA’s articulated commitment to veteran-centric care, a component of which is ensuring that veterans are well informed about their options for care. The website’s limited information is also inconsistent with federal internal control standards, which state that management should externally communicate the necessary quality information to achieve an entity’s objective—in this case, providing important information to veterans on the quality of nursing homes. Action to inform veterans about the quality of SVHs would better enable veterans and their families to compare the quality of their nursing home care options across all three settings. In the coming years, VA projects an increase in the number of veterans receiving nursing home care. This makes it particularly important that VA ensure veterans receive quality care, regardless of the setting—CLC, SVH, or CNH—in which this care is provided. Inspections are a key oversight tool used to ensure veterans receive quality care. VA relies primarily on annual inspections to oversee the quality of nursing home care at CLCs and SVHs, and our review shows that VA’s two contractors conducted these required inspections during the period we reviewed. However, our review also shows that VA has opportunities to enhance this oversight. First, VA has not regularly monitored the contractors’ performance conducting these inspections by conducting observational assessments as intended and therefore does not know whether the contractors need to improve their ability to determine the homes’ compliance with quality standards. Second, VA does not require inspectors of SVHs to identify all failures to meet quality standards as deficiencies, which limits VA’s ability to track all deficiencies identified at SVHs and identify trends in quality across homes. Third, VA has not provided guidance for VAMC staff for instances in which they may conduct onsite reviews of CNHs directly. As a result, VA has missed an opportunity to leverage efficiencies across VA’s network of VAMCs and empower VAMC officials with knowledge about the steps they can take to hold CNHs accountable for correcting problems. By making enhancements to its oversight of inspections across all three settings, VA would have greater assurance that the inspections are effective in ensuring the quality of care within each setting. VA also seeks to ensure that each veteran chooses a nursing home placement that best meets his or her preferences and needs. To enable veterans to evaluate their care options, VA uses its Access to Care website. However, this website provides no information about SVHs, which is where most veterans are currently receiving VA-funded nursing home care. Since VA is the only entity that inspects and collects quality information on all SVHs, VA possesses quality information that is not available elsewhere. However, because VA’s website lacks information on the quality of SVHs, veterans and their families are limited in their ability to compare the quality of the available nursing home care options. We are making the following four recommendations to the Veterans Health Administration: The Under Secretary of Health should develop a strategy to regularly monitor the contractors’ performance in conducting CLC and SVH inspections, ensure performance results are documented and any needed corrective actions are taken. (Recommendation 1) The Under Secretary of Health should require that all failures to meet quality standards are cited as deficiencies on SVH inspections. (Recommendation 2) The Under Secretary of Health should develop guidance for VAMC staff conducting optional onsite CNH reviews. (Recommendation 3) The Under Secretary of Health should provide information on the quality of all SVHs that is comparable to the information provided on the other nursing home settings on its Access to Care website. (Recommendation 4) VA provided written comments on a draft of this report, which are reprinted in appendix III. In its written comments, VA generally concurred with all four recommendations. With respect to our recommendation on regularly monitoring contractor performance in conducting CLC and SVH inspections, VA concurred and stated they would develop a procedure for observational inspections. VA also concurred with our recommendation requiring all failures to meet quality standards to be cited as deficiencies on SVH inspections and that “any regulation assessed to be incompliant at the time of the survey will be rated as either provisional or not met, which requires a corrective action plan from the SVH.” VA concurred in principle with our other two recommendations and described actions it plans to take to address them. Specifically, regarding our recommendation to develop guidance for VAMC staff conducting optional CNH onsite reviews, VA stated that it will issue a memo to clarify and provide guidance related to CNHs. VA also noted that, although we found the VAMC staff we interviewed discussed and considered these onsite reviews “inspections,” VA does not. Based on these technical comments, we adjusted our terminology. Further, we reiterate the value that VAMC officials placed on these reviews for assessing the quality of care veterans receive in the report. Accordingly, we believe that VA has the opportunity when developing the memo to clarify and provide guidance related to these optional CNH onsite reviews. With respect to our recommendation that VA provide information on the quality of all SVHs that is comparable to the information provided on the other nursing home settings, VA stated it plans to evaluate the feasibility of providing SVH data. VA noted challenges with developing their own five star ratings for SVHs since VA does not have all the required data for SVHs that is needed. We acknowledge that developing comparable information will take time and have adjusted some language in our report to reflect that VA had considered developing an SVH five-star program. VA also stated that we inaccurately portrayed VA’s oversight authority, because each state oversees its own SVH and VA does not have the authority to regulate the business or clinical practices of the SVH. Both our draft and final reports stated that “VA does not exercise supervision or control over the administration, personnel, maintenance, or operation of any state home.” However, as stated in the report, federal law prohibits payments to SVHs that do not meet standards the VA prescribes and authorizes VA to inspect any SVH at such times as VA deems necessary to ensure that such facility meets those standards. Further, we reiterate that as VA is the only entity to conduct inspections for all SVHs—it uniquely possesses information that is not available elsewhere. Accordingly, we believe that VA has the opportunity to help veterans and their families by providing quality information for SVHs as it does for the other nursing home settings. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of the Department of Veterans Affairs, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or silass@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Number of deficiencies (percent) 1 (0) Admission, Transfer, and Discharge Rights 0 (0) 1 (0) 19 (3) 87 (15) 1 (0) 14 (2) 3 (1) 1 (0) 279 (48) 67 (12) 51 (9) Number of deficiencies (percent) 6 (3) 4 (2) 0 (0) 2 (1) 93 (48) 1 (1) 36 (19) 4 (2) 33 (17) Resident Behavior and Facility Practices 10 (5) 3 (2) 0 (0) 192 The total number of deficiencies may include deficiencies from one SVH that VA does not consider a skilled nursing facility. In addition to the contact named above, Karin Wallestad (Assistant Director), Jim Melton (Analyst-in-Charge), Kye Briesath, Krister Friday, and Mandy Pusey made key contributions to this report. Also contributing were Vikki Porter and Jennifer Whitworth.", "summary": "VA provides nursing home care for veterans whose health needs are extensive enough to require skilled nursing and personal care in an institutional setting. VA provides or pays for the cost of nursing home care for eligible veterans. GAO was asked to examine VA nursing home care. In this report, GAO 1) describes utilization of and expenditures for VA-funded nursing home care, 2) examines VA's use of inspections to assess the quality of nursing home care and its oversight of the process, and 3) examines the information VA publicly provides through its website on the quality of nursing home care. To perform this work, GAO reviewed VA policies and information on inspections and interviewed VA officials. GAO also selected six VA medical centers based on factors such as their participation with CLCs, SVHs, and CNHs and location. For each, GAO interviewed medical center officials and officials from corresponding VA regional offices, CLCs, SVHs, and CNHs. According to the Department of Veterans Affairs (VA), veterans' use of nursing home care increased 3 percent, from an average daily census of 37,687 to 38,880 veterans, from fiscal years 2012 to 2017. VA projects that use will increase 16 percent from fiscal years 2017 to 2022 with the aging of Vietnam War veterans. VA's nursing home expenditures increased 17 percent (8 percent adjusted for inflation), from $4.9 billion to $5.7 billion, from fiscal years 2012 to 2017. During the contract year completed in 2018, VA contractors conducted required inspections of community living centers (CLC) (VA-owned and -operated) and state veterans homes (SVH) (state-owned and -operated) to ensure they complied with quality standards. Selected VA medical centers also completed required annual reviews of Centers for Medicare & Medicaid Services data and conducted optional onsite reviews for community nursing homes (CNH), with which VA contracts. However, VA has opportunities to enhance its oversight. For example, VA did not conduct the quarterly monitoring of contractor performance as stipulated in its contract for CLC inspections from April 2017 to April 2018. VA officials also said they intended to regularly observe contractors conducting inspections to ensure they effectively determine compliance with standards, but have not done so due to competing demands. Officials also said they had performed these observational assessments in the past but were unable to provide documentation of them occurring. Conducting and documenting the quarterly observational assessments would allow VA to identify areas for improvements and to take any needed corrective actions. VA's Access to Care website provides publicly available information about the quality of CLCs and CNHs based on inspections. Veterans and their families can use the website to help inform their decisions on nursing home placement. However, the website does not include any SVH information. Although VA has access to SVH quality information, according to VA officials, they are not required to publicly report it. For some SVHs, VA is the only source for quality care information. Some of the quality information is available locally, but the VA website is an important tool for veterans and their families. Providing SVH information on its website could enhance veterans and their families' ability to evaluate all nursing home options. GAO is making four recommendations, including recommendations for VA to enhance its oversight of the quality of care provided to veterans in CLCs, SVHs, and CNHs and include on its website information on the quality of care for SVHs that is comparable to what it provides on CLCs and CNHs. VA concurred with two recommendations and concurred in principle with two recommendations.", "document_type": "gao"}
{"report": "USDA conducts the Census of Agriculture every 5 years, most recently in 2012 and 2017. The census provides a detailed picture of farms and the people who operate them. The census identifies several categories of farmers, including the following: Producers. Producers are individuals involved in farm decision- making. A single farm may have more than one producer. Primary producers. The primary producer is the individual on a farm who is responsible for the most decisions. Each farm has only one primary producer. The 2017 Census questionnaire substantially revised the way it collected certain data in order to better capture the contributions of all persons involved in farm decision-making. For example, the 2017 questionnaire asked for the names and demographic information of up to four producers per farm (compared to three in 2012) and used a series of questions on specific types of farm decisions to determine the primary producer (the 2012 questionnaire did not include these questions). Therefore, comparisons between the two censuses regarding the number and personal characteristics of producers and primary producers should be considered with the 2017 revisions in mind. While some changes may be the result of actual changes in the population of farmers and ranchers, other changes may be the result of changes in census methodology. USDA’s 2017 Census counted about 3.4 million producers across the roughly 2 million farms nationwide, compared to 3.2 million in 2012. This represents an approximately 7 percent increase over 2012 in the number of reported producers, despite a slight drop in the number of farms reported. Many of these additional producers were SDFRs. In 2017, SDFRs accounted for 41 percent (1,390,449) of all producers, compared to 36 percent (1,133,163) in 2012. The number of reported SDFR primary producers also grew between 2012 and 2017. Among SDFR subgroups, women accounted for the largest increase in producers and primary producers. In the 2017 Census, women also made up the largest group of SDFR producers and primary producers (see table 1). Women accounted for 88.3 percent of all SDFR producers and 81.0 percent of SDFR primary producers. Hispanic, Latino, or Spanish-origin producers were the next largest group, accounting for 8.1 percent of all SDFR producers and 11.0 percent of SDFR primary producers. On average, farms for which an SDFR was the primary producer (SDFR farms) were smaller and brought in less revenue than non-SDFR farms in 2017. While representing 30 percent of all farms, SDFR farms operated 21 percent of total farm land and accounted for 13 percent of the market value of agricultural products sold in 2017 (see table 2). About 55 percent of SDFR farms had fewer than 50 acres, and 88 percent had less than $50,000 in total sales and government payments. Additionally, a lower proportion of SDFR-operated farms (21 percent) received government payments compared to non-SDFR farms (36 percent). Agricultural producers generally require financing to acquire, maintain, or expand their farms, ranches, or agribusinesses. Agricultural loans generally fall into two categories: Farm ownership loans. These loans are used to acquire, construct, and develop land and buildings and have terms longer than 10 years. They are secured by real estate and are sometimes referred to as real estate loans. Farm operating loans. These loans are generally short-term or intermediate-term loans that finance costs associated with operating a farm. Short-term loans are used for operating expenses and match the length and anticipated production value of the operating or production cycle. Intermediate-term loans are typically used to finance depreciable assets such as equipment and usually range from 18 months to 10 years. These loans may also be referred to as non- real-estate loans. Several types of lenders provide credit to agricultural producers, including, but not limited to, the following: Farm Credit System. The Farm Credit System is a government- sponsored enterprise, established, in part, to provide sound, adequate, and constructive credit to American farmers and ranchers. The Farm Credit System includes a national network of 73 banks and associations. The Farm Credit System lends money to eligible agricultural producers primarily through its 69 lending associations, which are funded by its four banks. All are cooperatives, meaning that Farm Credit System borrowers have ownership and control over the organizations. The Farm Credit System is regulated by the Farm Credit Administration, an independent federal agency. The Farm Credit System’s statutory objectives include being responsive to the needs of all types of creditworthy agricultural producers having a basis for credit, with additional requirements to serve young, beginning, and small farmers and ranchers. According to the Farm Credit Administration, the Farm Credit System is not statutorily mandated to focus on providing financial opportunities to any other group. Commercial banks. Commercial banks are regulated by the federal depository institution regulators. They vary in size and the type of credit they provide. In a January 2013 report, we found that large banks were more likely to engage in transactional banking, which focuses on highly standardized products that require little human input to manage and are underwritten using statistical information. We also found that small banks were more likely to consider not only data models but information acquired by working with the customer over time. Additionally, we found that by using this banking model, small banks may be able to extend credit to customers who might not receive a loan from a larger bank. The American Bankers Association reported that in 2017, the majority of farm banks—those that made more agricultural loans than the industry average—were small institutions with a median asset size of $125 million. USDA Farm Service Agency. USDA’s Farm Service Agency makes direct loans to farmers and ranchers and guarantees loans made by commercial lenders and Farm Credit System associations. The Farm Service Agency is a lender that focuses on assistance to beginning and underserved farmers and ranchers who are unable to obtain credit elsewhere. For its guaranteed loans, the agency typically guarantees 90 percent of losses the lender might incur in the event that a borrower defaults, although the agency may guarantee up to 95 percent for qualifying loans to certain groups, including SDFRs. Guaranteed loan terms and interest rates are set by the lender, though USDA has established maximum rates and terms. Agricultural loans guaranteed by the Farm Service Agency generally account for about 4–5 percent of outstanding loans made by the Farm Credit System and commercial banks and credit unions. Other lenders. A variety of other businesses and individuals provide agricultural credit to farmers and ranchers, including credit unions, life insurance companies, farm implement dealers, and family members. According to the National Credit Union Administration, agricultural lending represents a small portion (less than several basis points) of credit union lending. Historically, life insurance companies have used agricultural real estate mortgages as part of their investment portfolios. Farm implement dealers sell machinery, parts, and services and offer financing for those products. According to USDA survey data, implement dealers currently provide almost one-third of the agricultural sector’s farm operating debt with terms longer than 1 year and are an increasing source of agricultural credit. According to USDA’s Economic Research Service, in 2017, the Farm Credit System and commercial banks accounted for the bulk of agricultural lending in the United States, comprising about 80 percent of the total outstanding farm debt. The remaining debt was USDA Farm Service Agency direct loans and loans made by other lenders. Information on the types and amount of agricultural credit to SDFRs is limited. Regulation B, which implements the Equal Credit Opportunity Act (ECOA), generally prohibits lenders from collecting data on the personal characteristics (such as sex, race, and national origin) of applicants for loans other than certain mortgages. Therefore, financial institutions and their regulators generally do not have information on the types or amount of agricultural lending to SDFRs. In contrast, USDA collects and maintains personal characteristic data on applicants for the farm loans it makes or guarantees in order to target loans to traditionally underserved populations and fulfill statutorily mandated reporting requirements. The lack of personal characteristic data on a large portion of agricultural loan applications limits the ability of regulators, researchers, and stakeholders to assess potential risks for discrimination. In a July 2009 report, we found that federal enforcement agencies and depository institution regulators faced challenges in consistently, efficiently, and effectively overseeing and enforcing fair lending laws due in part to data limitations. Additionally, we found that such data would enhance transparency by helping researchers and others better assess the potential risk for discrimination. For our current review, some federal depository institution regulators we spoke with said that additional data on nonmortgage lending would allow them to perform additional assessments of financial institutions’ compliance with fair lending laws. Some SDFR advocates we spoke with also expressed concern about the lack of accurate public information on lending to SDFRs, which they said forces them to rely on anecdotal evidence in attempts to monitor potential discrimination. A rulemaking pursuant to Section 1071 of the Dodd-Frank Wall Street Reform and Consumer Protection Act (Dodd-Frank Act) would modify the Regulation B prohibition for certain loans, including possibly some agricultural loans. Section 1071 amended ECOA, requiring financial institutions to report information on credit applications made by women- owned, minority-owned, and small businesses. However, in April 2011, CFPB issued a letter stating that the requirements under Section 1071 do not go into effect until CFPB issues implementing regulations. The purpose of Section 1071 is “to facilitate enforcement of fair lending laws and enable communities, governmental entities, and creditors to identify business and community development needs and opportunities of women-owned, minority-owned, and small businesses.” Section 1071 is consistent with our 2009 report on fair lending issues, which said Congress should consider requiring additional data collection and reporting for nonmortgage loans. Section 1071 did not specify a time frame for CFPB to complete its rulemaking. As of June 2019, CFPB had not yet completed a rulemaking implementing Section 1071 of the Dodd-Frank Act. In 2017, CFPB issued a request for information seeking public comment on topics related to the collection of data on small business lending. However, in November 2018, CFPB announced that it was delaying the rulemaking due to resource constraints and other priorities. CFPB reported in the Spring 2019 Unified Agenda of Federal Regulatory and Deregulatory Actions that it plans to resume pre-rulemaking activities later in 2019. USDA’s annual survey of farm producers, the Agricultural Resource Management Survey, provides some insights into agricultural lending to SDFRs but has limitations when used for this purpose. The limitations fall into two main categories, as follows: First, the sample size used in the survey does not allow for capturing potential differences in the credit needs and challenges of specific socially disadvantaged subgroups. The relatively small proportion of SDFRs in the survey’s sample population renders estimates of SDFR farm debt less precise. To increase the precision of its estimates, USDA averaged 3 years of survey data (2015–2017) to increase the sample size of SDFRs available for analysis. Due to the small size of several SDFR subgroups, we analyzed SDFRs as a single combined group. Second, the survey may underrepresent the total outstanding farm debt of socially disadvantaged groups and should be interpreted with caution, according to USDA officials. As previously discussed, the 2017 Census questionnaire included revisions that better captured the role of SDFRs in farm operations, and its results suggest that the 2012 Census and the 2015–2017 surveys (which used similar methodologies) may have underreported the number of SDFRs designated as primary producers, particularly women. Specifically, in the 2015–2017 surveys, SDFRs represented 17 percent of primary producers, whereas in the 2017 Census, SDFRs accounted for 30 percent of primary producers. However, the potential underrepresentation issue should not affect the statistical significance of comparisons between the SDFR and non-SDFR subgroups within the survey. With these caveats in mind, the 2015–2017 survey data suggest that SDFR primary producers had annual average outstanding farm debt of $20.0 billion ($17.5–$22.6 billion at the 90 percent confidence level). This estimate represents debt used specifically for farm purposes. Farm ownership debt was a larger share of SDFR outstanding farm debt than it was for all other farmers and ranchers. Among SDFR primary producers, farm ownership debt was estimated to account for 67 percent of outstanding farm debt, compared to an estimated 59 percent for non-SDFR primary producers (see fig. 1). Farm operating debt accounted for the remaining 33 percent and 41 percent of outstanding SDFR and non-SDFR farm debt, respectively. SDFRs received proportionately fewer loans and less agricultural credit overall than non-SDFRs. Specifically, SDFRs accounted for an estimated 17 percent of primary producers in the survey but only 13 percent of farms with loans and 8 percent of total outstanding farm debt. SDFR debt represented an estimated 9 percent of total farm ownership debt and 7 percent of total farm operating debt (see table 3). Therefore, even though farm ownership debt comprised most outstanding SDFR farm debt (67 percent), SDFR primary producers were still less likely to have outstanding farm ownership debt than all other farmers and ranchers. While the survey data show that SDFRs had proportionately less agricultural credit than non-SDFRs, the survey does not provide information on the reasons why. However, a number of factors may help explain these differences. For example, the 2017 Census shows that SDFRs are more likely than non-SDFRs to operate smaller farms with less market value, and smaller farms may require less credit to operate. In addition, as discussed later in this report, SDFRs may have greater difficulty qualifying for agricultural loans or may be dissuaded from applying for credit. SDFR primary producers generally borrowed from the same type of lenders as non-SDFRs and reported using a range of agricultural credit providers. The distribution of SDFR and non-SDFR farm debt by lender type in the survey was roughly similar, with all differences within the margin of error (at the 90 percent confidence level). According to the survey data, an estimated 51 percent of SDFRs’ outstanding farm debt was lent by commercial banks and savings associations. Lending by Farm Credit System institutions (28 percent), USDA’s Farm Service Agency (6 percent), and other lenders, such as individuals and equipment dealers (15 percent), comprised the remainder. SDFRs received a larger share of their operating credit, compared to ownership credit, from lenders in the “other” category. This was true for non-SDFR operating debt as well. These results should be interpreted cautiously because the information is self-reported and respondents may not have known the specific types of lenders they used. The survey results for all farms appear to overrepresent debt from commercial banks and savings associations when compared with data collected by USDA’s Economic Research Service on farm-sector balance sheets. It is possible some respondents mischaracterized some debt from Farm Credit System institutions as debt from commercial banks. While loans guaranteed by USDA’s Farm Service Agency make up a small percentage of overall agricultural lending, the agency tracks how much of this lending goes to SDFRs and the purpose of the loans (ownership or operating). In fiscal year 2018, the Farm Service Agency guaranteed $3.2 billion in new agricultural loans. About $340 million (10.8 percent) of this amount went to SDFRs (see fig. 2). By dollar volume, farm ownership loans accounted for about 71 percent of the guaranteed loans to SDFRs. Farm operating loans accounted for the remaining 29 percent. Guaranteed farm ownership loans to SDFRs averaged about $519,000, while farm operating loans averaged about $279,000. A 1988 amendment to the Consolidated Farm and Rural Development Act states that USDA should establish annual target participation rates for SDFRs on a county-wide basis for farm ownership loans and, to the greatest extent practicable, reserve funds for certain loans it makes or insures under these targets. However, in August 2007, USDA’s Office of General Counsel provided a legal opinion that stated that the statute could be read to apply only to the direct loan program. As a result, officials at the Farm Service Agency told us it does not set annual target participation rates by county or reserve funds for guaranteed loans. Over the last 5 fiscal years (2014–2018), the Farm Service Agency guaranteed an increasing number of loans to SDFRs each year. The agency guaranteed 489 loans to SDFRs in fiscal year 2014 and 817 loans in fiscal year 2018—a 5-year high. Over that period, the total dollar amount of guaranteed loans to SDFRs increased by 69.6 percent when adjusted for inflation. The increase was similar for farm ownership and farm operating loans (see fig. 3). While the total dollar amount of guaranteed loans to SDFRs increased each year, the percentage of guaranteed loans that went to SDFRs, by dollar volume, decreased from fiscal years 2014 through 2016 (see fig. 4). This percentage started increasing in fiscal year 2017, when SDFRs accounted for 8.7 percent of guaranteed loans by dollar volume. However, guaranteed loans to SDFRs still accounted for a slightly smaller portion of all guaranteed loans in fiscal year 2018 (10.8 percent) than in fiscal year 2014 (11.0 percent). In fiscal year 2018, the dollar amount and percentage of guaranteed loan funds that went to SDFRs differed substantially by state (see table 4). Hawaii and Puerto Rico were the only two states or territories where SDFRs received more than one-half of all guaranteed loans (farm ownership and operating loans combined). However, Hawaii and Puerto Rico received 0.1 percent of all guaranteed loans. For several states where SDFRs received a large dollar amount of guaranteed loans, these loans represented less than 20 percent of the state’s guaranteed loan funds (for example, Arkansas, Missouri, and South Dakota). In contrast, several states with the largest proportions of guaranteed loans to SDFRs had less guaranteed loan funds overall (for example, Florida, Wyoming, and Maryland). The Farm Service Agency did not guarantee any loans to SDFRs in Alaska, Connecticut, New Hampshire, or Rhode Island in fiscal year 2018. According to representatives from some SDFR advocacy groups, federal depository institution regulators, and lending industry associations we interviewed, SDFRs can have difficulty obtaining agricultural credit from private-sector lenders because they operate smaller farms and in some cases do not meet standards for farm revenue, applicant credit history, and collateral. Farm size. As previously discussed, SDFRs are more likely than other farmers and ranchers to operate small farms, which can make it difficult for them to qualify for private credit. According to data from the 2017 Census of Agriculture, SDFRs represented 30 percent of primary producers but operated 39 percent of farms smaller than 50 acres and 16 percent of farms 500 acres or larger. Some SDFR advocates and lending industry association representatives we interviewed said lenders have several incentives to lend to larger farms. First, one advocate noted that operators of smaller farms typically need smaller loans, and making many small loans is more time- and resource-intensive than making fewer, larger loans. Second, one industry association and one SDFR advocate noted that large farms often produce major commodities such as corn, soybeans, and beef cattle, while small farms often produce specialty crops. The SDFR advocate said underwriting loans to large farms that produce major commodities is easier and less risky because more data are available on the market for those products. Third, representatives of one SDFR advocacy group and one industry association noted that programs such as crop insurance are geared toward large, major-commodity farmers. They said these programs mitigate repayment risk and make lenders more likely to approve a loan or provide more favorable terms, such as lower interest rates. In contrast, representatives from the Office of the Comptroller of the Currency noted that the Community Reinvestment Act can provide incentives for banks to lend to smaller farms. Farm revenue. Consistent with their smaller size, SDFR farms also generate less revenue on average than non-SDFR farms. As previously noted, SDFR primary producers accounted for a disproportionally small portion (13 percent) of total agricultural product sales in 2017 relative to their overall representation among primary producers (30 percent). Additionally, according to one SDFR advocate, SDFRs may have more difficulty than other farmers and ranchers in documenting their revenue because they are more likely to sell their products through informal cash transactions. Operating a lower-revenue farm and having limited documentation of revenue can be hurdles to obtaining private credit because these factors may negatively affect a lender’s assessment of the applicant’s repayment ability. Federal depository institution regulators have noted that farm revenue is critical to demonstrating a borrower’s capacity to repay an agricultural loan. For example, in its risk management expectations for agricultural credit, the Board of Governors of the Federal Reserve System says banks should review borrower-prepared cash-flow statements to identify potential repayment-ability problems. Lenders consider farm revenue when calculating an applicant’s debt-to-income ratio (the percentage of income that goes to recurring debt payments), which is a central underwriting criterion. In general, having lower income relative to recurring debt payments indicates weaker repayment ability. Consistent with this principle, Farm Credit Administration regulations require Farm Credit System associations to have written policies and procedures that include underwriting standards that demonstrate an applicant’s repayment capacity when approving a loan. Additionally, representatives of one industry lending association said that revenue is the most important factor that banks consider in underwriting agricultural loans. Credit history. Some SDFRs may have relatively low credit scores or limited credit histories, which can make it difficult to obtain agricultural credit. Some SDFR advocates and lending industry association representatives we interviewed said that some SDFR subgroups are more likely than members of nondisadvantaged groups to have difficulty meeting credit score standards for agricultural loans. Prior research provides some evidence to support this view. For example, the Board of Governors of the Federal Reserve System reported in 2007 that African Americans and Hispanics had lower credit scores on average than non- Hispanic whites and Asians, although the study did not specifically examine farmers and ranchers. While private agricultural lenders are not subject to federal statutory or regulatory credit score requirements for approving agricultural loans, federal depository institution regulators emphasize the importance of evaluating applicants’ creditworthiness in their lending guidelines. For example, the Office of the Comptroller of the Currency’s handbook on agricultural lending states that current credit information is essential to a bank’s ability to evaluate borrowers’ creditworthiness. Lending industry association representatives we interviewed also noted that underwriting for agricultural lending is increasingly standardized and reliant on credit scores. For example, representatives from the Farm Credit Council (the trade association for the Farm Credit System) said approval decisions for about one-half of the loans that Farm Credit System associations make each year are made using credit scorecards. Credit scorecards are algorithms that statistically quantify a borrower’s probability of repayment using inputs such as the borrower’s credit score. Additionally, participation in the secondary market for agricultural loans may require lenders to comply with credit score criteria. For example, the Federal Agricultural Mortgage Corporation (commonly known as Farmer Mac)—a federal government-sponsored enterprise that purchases and securitizes agricultural loans—has minimum credit score standards that range from 660 to 720. Collateral. Some SDFRs face challenges using their agricultural land as collateral. Many long-term agricultural loans require the borrower to pledge land as collateral to secure the transaction. For example, long- term loans (up to 40 years) made by Farm Credit System associations must be secured by a first-position lien on interests in real estate, generally enabling the Farm Credit System to obtain ownership or control of the land in the event of default. Federal regulators, lending industry association representatives, and SDFR advocates we spoke with identified several reasons why SDFRs, especially African Americans and American Indians on tribal lands, have difficulty using agricultural land as loan collateral. Some SDFRs do not have a clear title to their agricultural land because the land was passed down informally from generation to generation without a will. In addition, land passed down in this manner can result in numerous heirs—thousands in some cases—owning the land in common (that is, not physically divided among them). These circumstances can limit use of the land as collateral because of lending requirements or conventions that require formal proof of ownership or that disallow the use of a partial ownership interest as security for a loan. SDFR advocates and officials from the Farm Credit Administration told us these issues have particularly affected African American farmers due to historical factors that limited their access to legal services. In our May 2019 report about lending on tribal lands, we discussed how these issues also have posed problems for American Indian farmers. As we also reported in May 2019, American Indian farmers on tribal lands face additional challenges in using tribal land as collateral for agricultural loans because of statutory restrictions and some lenders’ concerns about their ability to enforce a foreclosure. SDFR advocates we spoke with said that in addition to difficulty meeting loan underwriting standards, SDFRs face challenges related to historical discrimination, ongoing unfair treatment by lenders, and a lack of familiarity with some programs and technologies when trying to obtain private agricultural credit. As the Congressional Research Service reported in 2013, allegations of unlawful discrimination against SDFRs in the management of USDA programs are long-standing and well-documented. For example, in 1965, the U.S. Commission on Civil Rights found evidence of discrimination in the delivery of USDA farm programs, including loan programs. A subsequent report by the commission in 1982 and a report by the USDA Civil Rights Action Team in 1997 found continuing problems with the experience or treatment of SDFRs in USDA programs. USDA has also settled several class action lawsuits that SDFRs filed for, among other things, discrimination in the agency’s farm assistance programs. The allegations in these lawsuits included that USDA systematically denied SDFRs agricultural credit and other program benefits in violation of ECOA and failed to investigate complaints of discrimination, as required by USDA regulations. The settlements made more than $4 billion in awards available to farmers and ranchers whose claims were approved through administrative procedures. Some SDFR advocates told us that historical discrimination in agricultural lending adversely affects SDFRs’ current ability to obtain private credit in several ways. First, they said SDFRs who were unfairly denied USDA loans and other program benefits in the past have not been able to develop their farms in the same ways as farmers and ranchers who did receive loans, thus reducing their ability to obtain private credit today. The advocates elaborated that USDA agricultural credit allows recipients to expand operations and to purchase land and equipment that can later be used as collateral, making it easier to get subsequent and larger loans. Some SDFR advocates also stated that historical exclusion from credit markets and farm programs has limited SDFRs’ familiarity with lending standards and resulted in less formal recordkeeping, which impairs their ability to obtain private-sector credit. Finally, advocates said that historical discrimination has led generations of SDFRs to distrust institutional lenders, making them less likely to apply for credit. Some SDFR advocates we spoke with said that unfair treatment by private lenders is also a barrier to SDFRs obtaining private agricultural credit. One SDFR advocate said some lenders discriminate against SDFRs in loan approval decisions but that they more frequently treat SDFRs unfairly with respect to loan terms and conditions (for example, interest rates, fees, and collateral requirements) and loan servicing (for example, restructuring and foreclosure mitigation actions). Another noted that adverse loan terms and conditions and servicing practices can increase the risk that borrowers will lose their farm, house, and other property by making the loan unaffordable or reducing the chances that borrowers will catch up on payments if they fall behind. For example, this SDFR advocate said they were aware of cases in which (1) lenders required SDFRs to pledge potentially excessive collateral for loans, such as the borrower’s home in addition to the farm land, and (2) loan servicers moved more quickly to foreclose on SDFR borrowers who were behind on loan payments than on other borrowers and did not provide repayment options that may have allowed them to continue their operations. One SDFR advocate also stated that some SDFRs report not feeling welcome at lending institutions based on the perception of having been repeatedly dismissed by lender staff, while another said that in some cases, SDFRs have not been provided timely or helpful information on the loan application process. One SDFR advocate we spoke with said these practices are prevalent in some agricultural credit markets and that they had been or were currently involved in litigation related to these types of practices. However, banking industry association representatives said they did not believe that SDFRs are being treated unfairly and that denying loans to qualified applicants would cause lenders to decrease profits in a competitive market. They noted that lenders face significant competition, which incentivizes them to make loans to all qualified borrowers, and that lending decisions and loan terms are based only on the applicant’s ability to repay a loan and other underwriting criteria. We did not attempt to independently verify claims of unfair treatment of SDFRs by private-sector lenders, in part because data limitations discussed earlier limit the identification and analysis of possible discriminatory practices. Some SDFR advocates also said that some SDFRs may not be obtaining private agricultural credit because they are not aware of all potential credit options and related programs and are not always familiar with the technology needed to access them. For example, one advocate told us some SDFRs may not be aware that they could qualify for private agricultural loans, especially if they are recent immigrants or new to agriculture. This problem may be particularly true for loans from the Farm Credit System associations. Two advocates said SDFRs are not familiar with these lenders, and representatives of the Farm Credit Council told us people who did not grow up in farming tended not to know about the Farm Credit System. SDFR advocates we spoke with said this issue is exacerbated by limited outreach by private lenders to SDFRs, as discussed in more detail later in this report. Advocates also noted that historically disadvantaged groups are less likely to have access to or be familiar with computer technology and the internet, and that credit applications and related financial education programs are now provided online. The Farm Credit System does not have a specific mandate to serve SDFRs, but its associations conduct some outreach to SDFRs in implementing the following statutory requirements and Farm Credit Administration regulations. The Farm Credit Act of 1971 was amended in 1980 to require the Farm Credit System to serve young, beginning, and small farmers. Related Farm Credit Administration regulations require the associations to implement effective outreach programs to these groups. While these requirements do not mandate outreach to SDFRs specifically, Farm Credit Administration officials said that many SDFRs qualify as young, beginning, or small farmers and, therefore, that Farm Credit System outreach efforts reach SDFRs to some extent. In 2012, the Farm Credit Administration amended its regulations on business planning to help ensure the Farm Credit System is responsive to the credit needs of all eligible and creditworthy persons. The regulations, which first applied to 2013 business plans, require Farm Credit System associations to develop marketing plans describing, among other things, (1) the demographic groups in their service areas, (2) ways to market their services to all qualified farmers and ranchers, and (3) specific outreach toward diversity and inclusion in each market segment. The supplementary information included with the publication of the final rule cites the perception of some SDFR advocates that Farm Credit System associations are not accessible to underserved farmers and have not conducted sufficient outreach to those populations about programs and services. The full extent of the Farm Credit System associations’ outreach to SDFRs is unknown. Neither the Farm Credit Administration nor the Farm Credit Council maintains aggregated information on the number or type of completed outreach activities involving SDFR participants. However, our nongeneralizable review of recent marketing plans from six Farm Credit System associations in areas with relatively high proportions of SDFRs identified some examples of outreach to SDFRs. For instance, some associations have partnered with a nonprofit organization to provide educational programs designed to strengthen women’s roles in the modern farm enterprise. Associations have also participated in agricultural conferences at historically black colleges and universities and translated marketing materials for non-English speakers. Despite some outreach, some SDFR advocates we spoke with said that Farm Credit System associations’ outreach has had limited effects on the amount of credit provided to SDFRs and SDFRs’ familiarity with the system. One SDFR advocate we spoke with said that while some Farm Credit System associations engage with socially disadvantaged communities, the outreach has not increased the diversity of the system’s borrowers. Others said that Farm Credit System outreach to SDFR communities has been insufficient and that some SDFRs are still not aware of the Farm Credit System. However, one SDFR advocate noted that the Farm Credit System’s outreach to young, beginning, and small farmers has been beneficial for those populations. The impact of Farm Credit System associations’ outreach to SDFRs is also not known. The marketing plan requirement does not oblige Farm Credit System associations to meet specific lending goals or favor any type or group of agricultural producers in their underwriting. Accordingly, the associations are not expected to quantify the extent to which they are meeting their diversity and inclusion outreach plans in the information they provide to their boards of directors. Moreover, Farm Credit Administration officials said Regulation B, discussed earlier, prevents the associations from collecting data on the race, ethnicity, and sex of loan applicants that would be needed to assess the effects of outreach efforts on lending to socially disadvantaged groups. In contrast, the officials noted that Farm Credit System associations are required to set lending targets for young, beginning, and small farmers; monitor outreach to those groups; and report on performance results of their young, beginning, and small farmer programs. In 2018, the Farm Credit System reported that all direct-lender institutions with young, beginning, and small farmer programs within the system were in compliance with these requirements. While the Farm Credit Administration has not evaluated the impact of outreach by Farm Credit System associations, its reviews of association marketing plans have found that most of the plans comply with requirements for outreach toward diversity and inclusion but that some lack specificity. The Farm Credit Administration told us it examines all of the associations’ marketing plans for regulatory compliance every 3 years. Farm Credit Administration officials reviewed their examinations from 2014 and 2017, the two scheduled examination cycles after the new requirements were implemented in 2012. They found that 85 percent of the 78 Farm Credit System associations examined in 2014 complied with the marketing and outreach requirements, and 94 percent of the 71 associations examined in 2017 complied. In cases where examiners identified deficiencies in marketing plans, the agency said it prescribed corrective actions, including requiring associations to do the following: obtain sufficiently detailed information to analyze and understand develop specific action plans and outreach strategies to market the institution’s products and services to potentially underserved markets; and ensure appropriate reporting on progress in accomplishing marketing plan strategies and actions. Farm Credit Administration officials said they hold periodic discussions with managers of Farm Credit System associations to monitor the status of corrective actions and conduct follow-up examinations to determine the adequacy of the corrective actions and, if applicable, the need for additional enhancements. The results of our review of a nongeneralizable sample of association marketing plans were broadly consistent with the Farm Credit Administration’s findings. We reviewed the most recent available plans of the six Farm Credit System associations noted previously for evidence of demographic information on the institution’s service area and for diversity and inclusion outreach efforts. Among the plans we reviewed, five included demographic information, but one did not. Farm Credit Administration officials said they also had identified that deficiency in their examination of that marketing plan. Additionally, five of the plans had examples of planned outreach efforts to SDFRs, but another one did not. According to representatives of lending industry associations we interviewed, commercial banks generally do not target outreach for agricultural lending to specific demographic groups. Officials from the federal depository institution regulators noted that commercial banks and credit unions are not required to conduct outreach on agricultural lending, and that the extent to which any lender conducts outreach is a private business decision. However, officials from one federal depository institution regulator noted that some lenders have participated in conferences organized by SDFR groups. They also said that in fulfilling responsibilities under the Community Reinvestment Act, lenders engage with community groups in their assessment areas to help identify credit needs. The officials said these efforts would likely engage SDFRs in areas where agriculture was prevalent and where agricultural lending was part of a bank’s business model. Some SDFR advocates we interviewed said that outreach and engagement by commercial banks was insufficient. For example, despite their familiarity with agricultural lending, some noted that they did not know of any specific outreach to SDFRs by private-sector lenders. They also noted that additional outreach is needed because some SDFRs are not familiar with agricultural lending products offered by commercial banks. Federal depository institution regulators do not monitor outreach to SDFRs by the institutions they supervise but have conducted some additional outreach themselves. Officials from the regulatory agencies told us they do not collect data on the amount of, types of, participation in, or impact of outreach conducted by their regulated institutions. However, as part of their efforts to promote the availability of credit and other services, the federal depository institution regulators have engaged in some outreach to SDFRs, as shown in the following examples. The Office of the Comptroller of the Currency has established an Office of Minority and Women Inclusion and an Office of External Outreach and Minority Affairs, which help to address fair credit access issues affecting minority communities and have worked with some national SDFR groups to coordinate, facilitate, and implement conferences, roundtables, and seminars. The Federal Deposit Insurance Corporation’s Community Affairs Branch has engaged bankers, nonprofits, and other stakeholders to provide small business training for SDFRs. This training provides examples of small business lending and has highlighted programs for which participants may qualify. In 2017, the Federal Reserve Bank of St. Louis and the Board of Governors of the Federal Reserve System engaged with federal agencies, businesses, and groups representing SDFRs to develop and publish a guide titled Harvesting Opportunity, which focuses on how credit can provide greater support for local food-related businesses and farmers. USDA facilitates and provides outreach to SDFRs that some SDFR advocates say has been beneficial, but outreach on USDA-guaranteed farm loans is just one component of this broad-based effort. USDA’s Office of Partnerships and Public Engagement implements the Outreach and Technical Assistance for Socially Disadvantaged and Veteran Farmers and Ranchers Program, referred to as the Section 2501 program. The program is designed to enhance coordination of outreach, technical assistance, and education efforts authorized under agricultural programs to improve SDFR and veteran farmer and rancher participation in the full range of USDA programs, including guaranteed farm loans. USDA officials said this program primarily provides grants and technical assistance to community-based organizations and develops materials describing best practices for national, state, and local outreach efforts. Two SDFR advocates we interviewed said outreach programs coordinated through the Section 2501 program have improved SDFRs’ understanding of USDA’s farm lending programs, and that the program’s efforts to engage SDFRs in programs and services are better now than they have been historically. USDA officials said they track these outreach activities but do not maintain data on activities that specifically address guaranteed loans because the outreach is generally intended to connect socially disadvantaged groups with any USDA program that may be appropriate. In addition to department-level outreach activities, USDA’s Farm Service Agency conducts outreach to increase SDFR participation in its programs through activities targeted to underserved populations. Farm Service Agency outreach efforts are conducted by the agency’s field offices and overseen by the Outreach Office. The outreach includes lender trainings and partnerships with community-based and tribal organizations to engage socially disadvantaged communities. Farm Service Agency officials said that they have partnered with private-sector lenders to conduct some outreach events specifically related to the guaranteed farm loan program but that most of the outreach is more general. Farm Service Agency officials told us they use data on guaranteed loans to SDFRs to target outreach to underserved communities. As previously discussed, unlike other providers of agricultural credit, USDA generally collects data on the personal characteristics of guaranteed loan applicants and borrowers. Farm Service Agency officials told us that state executive directors, farm loan chiefs, and outreach coordinators plan their outreach in annual strategy sessions. As part of this planning, state offices review the state’s lending goals for SDFRs, Census of Agriculture data on the state’s farmer population, and data on Farm Service Agency direct and guaranteed loans made to farmers belonging to different socially disadvantaged groups to target outreach to underserved communities. While the outreach is planned by state offices, the Farm Service Agency’s Director of Outreach said the Outreach Office has emphasized the use of lending goals and loan data in targeting outreach efforts. Although it maintains data on guaranteed loans made to SDFRs, USDA generally does not evaluate whether SDFR outreach participants go on to use Farm Service Agency lending programs or otherwise evaluate the impact of its outreach on lending to SDFRs. Farm Service Agency officials said that they track outreach activities at the national level by monitoring the number of activities, the groups engaged, and the number of participants, but that they face challenges evaluating the impact of outreach efforts. The officials said any personal or demographic information on outreach participants must be voluntarily provided by the participants, but that many of them are reluctant to do so. As a result, data on the characteristics of outreach participants are limited. The lack of data, in turn, makes it difficult to assess how effectively the outreach was targeted and whether it could be expected to increase lending to socially disadvantaged groups. Representatives from one SDFR advocacy organization said that while outreach programs may increase SDFRs’ understanding of USDA’s loan programs, it is unclear how much outreach programs help SDFRs obtain credit because USDA does not track participant outcomes. Farm Service Agency officials said that some of their state offices have begun trying to track the progress of individual outreach participants in obtaining loans through Farm Service Agency programs (using voluntarily provided information), but that these efforts were in the early stages. We provided a draft of this report to USDA, the Farm Credit Administration, the Consumer Financial Protection Bureau, the Office of the Comptroller of the Currency, the Federal Deposit Insurance Corporation, the Board of Governors of the Federal Reserve System, and the National Credit Union Administration for their review and comment. The Board of Governors of the Federal Reserve System and the National Credit Union Administration did not provide comments. USDA, the Farm Credit Administration, the Consumer Financial Protection Bureau, the Office of the Comptroller of the Currency, and the Federal Deposit Insurance Corporation provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Agriculture, the Acting Chairman and Chief Executive Officer of the Farm Credit Administration, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or ortiza@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. The objectives of this report were to examine (1) what is known about the amount and types of agricultural credit to socially disadvantaged farmers and ranchers (SDFR), (2) challenges SDFRs reportedly face in obtaining agricultural credit, and (3) outreach efforts to SDFRs regarding agricultural credit and related services. In this report, we use the term SDFR as defined in the Consolidated Farm and Rural Development Act, as amended, and related U.S. Department of Agriculture (USDA) regulations. The act defines a socially disadvantaged group as one whose members have been subject to racial, ethnic, or gender prejudice because of their identity as members of a group without regard to their individual qualities. USDA regulations further define SDFRs as belonging to the following groups: American Indians or Alaskan Natives, Asians, Blacks or African Americans, Native Hawaiians or other Pacific Islanders, Hispanics, and women. Although the act and USDA regulations defined SDFR for purposes of classifying participants in USDA programs, in this report, we use USDA’s definition to identify SDFRs both in USDA programs and in the broader population of agricultural producers, consistent with the statutory provision this report responds to. Additionally, based on the language of the statutory provision, we excluded USDA direct loans from the scope of our review and focused on lending by private entities. The provision defines an agricultural credit provider as a Farm Credit System institution, a commercial bank, the Federal Agricultural Mortgage Corporation, a life insurance company, and any other individual or entity as determined by the Comptroller General of the United States. For the background section of this report, USDA’s National Agricultural Statistics Service provided estimates from the 2012 and 2017 Censuses of Agriculture on the number of farm and ranch operations (which we refer to as farms) whose primary producer—that is, main decision maker—qualified as an SDFR, broken down by different SDFR subgroups. The service also provided estimates on the characteristics of farms whose primary producer was an SDFR, including the total acreage and market value of products sold. We compared the 2017 Census estimates of SDFR primary producers to analogous estimates from the 2012 Census and calculated numerical and percentage differences. We reviewed documentation on the methodologies used by the 2012 and 2017 Censuses to identify the main decision maker on a farm. We also interviewed National Agricultural Statistics Service officials about methodological differences between the two censuses and their likely effects on the number of reported SDFR primary producers. The 2012 Census used the term “principal operator” rather than “primary producer” to identify the main farm decision maker, but for ease of presentation we use the term primary producer in reference to both the 2012 and 2017 Censuses because the terms generally have the same meaning. To examine what is known about the amount and types of agricultural credit to SDFRs, we reviewed requirements in the Equal Credit Opportunity Act and its implementing regulation (Regulation B) governing the collection of data on the personal characteristics of loan applicants. We interviewed officials from the Consumer Financial Protection Bureau (CFPB), which has primary responsibility for issuing Equal Credit Opportunity Act regulations, about these requirements and the status of a related rulemaking pursuant to a provision in the Dodd-Frank Wall Street Reform and Consumer Protection Act. We also interviewed officials from the federal depository institution regulators—the Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation, Office of the Comptroller of the Currency, and National Credit Union Administration—about the extent of information available on agricultural lending to SDFRs and about data restrictions stemming from Regulation B. We also drew upon information and analysis from our June 2008 and July 2009 reports on data limitations in nonmortage lending. Additionally, we analyzed data from USDA’s Agricultural Resource Management Survey. The survey is a multiphase series of interviews that uses a multiframe, stratified, probability-weighted sampling design. The survey does not include Hawaii or Alaska. USDA’s Economic Research Service provided us customized summary statistics from the 2015, 2016, and 2017 surveys combined. Specifically, the service averaged survey data for those 3 years to provide a robust sample size of surveyed SDFRs. The service provided estimates and associated confidence intervals on the proportion of primary producers who were and were not SDFRs; the annual average amount of outstanding farm debt each group had over the 3-year period, by type of debt (ownership or operating); and the lending source for this debt (USDA Farm Service Agency, Farm Credit System institution, commercial bank and savings associations, or other). The service adjusted debt information for inflation. Specifically, to create standard errors for the 3-year averages, the service adjusted outstanding debt to 2017 dollars using the chain-type gross domestic product deflator. We compared and contrasted survey statistics for SDFRs and non-SDFRs, focusing on the volume and percentage of total outstanding farm debt, farm ownership and operating debt, and lender type. We interviewed Economic Research Service officials about limitations of the survey data. The limitations include the small size of several SDFR subgroups (which prevented more detailed analysis of different demographic groups), the potential underrepresentation of SDFRs in the survey, and potential overreporting of debt from commercial lenders. With regard to lender type, respondents may not have known the specific types of lenders they used. The survey results for all farms appear to overrepresent debt from commercial banks and savings associations when compared with data collected by the service on farm- sector balance sheets. It is possible some survey respondents mischaracterized some debt from Farm Credit System institutions as debt from commercial banks. These issues and their implications are discussed in the body of this report. To assess the reliability of the survey data, we reviewed methodology and quality review documents and compared results to other publicly available sources, such as farm balance-sheet data and the 2017 Census. We concluded that the data were sufficiently reliable for describing the amount and types of agricultural credit SDFRs received, the sources of this credit, and how SDFRs and non-SDFRs compared along these dimensions. We also analyzed USDA data on farm ownership and farm operating loans guaranteed by the Farm Service Agency in fiscal years 2014 through 2018. We focused on guarantees issued by the Farm Service Agency because it operates the primary federal agricultural credit programs. For the 5-year period, we analyzed the annual amount and percentage of guaranteed loans (by dollar volume and adjusted for inflation) that went to SDFRs. We also separately examined trends in guaranteed farm operating and farm ownership loans to SDFRs. Finally, we analyzed the volume of guaranteed loans to SDFRs by state. We used this analysis to identify the top 10 states (or territories) in terms of (1) the dollar amount of guaranteed loans that went to SDFRs and (2) the proportion of guaranteed lending to the state or territory that went to SDFRs. To assess the reliability of data from USDA, we conducted electronic testing—including checks for missing data and erroneous values—and compared the data to publicly available sources. The loan guarantee data we present are somewhat different than publicly available information on USDA’s website because we used loan closing dates to group loans by fiscal year, while the publicly available data used the dates on which USDA obligated commitment authority for the loans. According to USDA officials, the closing date is a more accurate representation of the actual amount of loans guaranteed in a fiscal year, because some loans for which commitment authority is obligated may close in the following fiscal year or not close at all. We also interviewed USDA officials about interpretations of data fields and robustness of estimated values, among other things, and reviewed USDA internal policies and procedures for data entry. We concluded that the data were sufficiently reliable for describing the amount and proportion of farm lending guaranteed by the Farm Service Agency that went to SDFRs and non-SDFRs nationwide and by state. Finally, we reviewed documents and interviewed officials from the Farm Service Agency on the agency’s performance goals and target participation rates for farm lending to SDFRs. We also reviewed a 2007 USDA Office of General Counsel legal opinion on a statutory provision concerning establishment of target participation rates for SDFRs. However, an evaluation of the legal opinion was outside the scope of our study. To examine challenges SDFRs face in obtaining agricultural credit and outreach efforts to SDFRs regarding agricultural lending, we conducted searches of government and academic literature for research on private agricultural lending to socially disadvantaged groups. We searched the internet and various databases, such as AGRICOLA, EconLit, ProQuest Newsstand Professional, and Social SciSearch. Using broad search terms, we identified articles related to our research objectives that provided useful context and discussion topics for interviews with stakeholders. We did not identify any government or peer-reviewed academic literature that directly addressed private agricultural lending to socially disadvantaged groups, barriers those groups may face when trying to obtain agricultural credit, or outreach to disadvantaged groups by private agricultural lenders. We also solicited expert recommendations for academic literature on agricultural lending to socially disadvantaged groups. Several SDFR advocates identified the Socially Disadvantaged Farmers and Ranchers Policy Research Center as a potential source for academic literature on the subject. We found that the center had conducted some potentially relevant research but that the work had yet to be published in academic journals or government publications. To review efforts by agricultural lenders and their regulators to provide and oversee credit-related services to SDFRs—including marketing, outreach, and education activities—we reviewed data and documents from the Farm Credit System, USDA, and the federal depository institution regulators. We reviewed summary statistics from the Farm Credit Administration’s 2014 and 2017 examinations of Farm Credit System association marketing plans to determine the extent to which the associations had met requirements for outreach for diversity and inclusion. We supplemented this effort by reviewing marketing plans from a sample of six Farm Credit System associations in areas with substantial proportions of SDFRs from each of the socially disadvantaged groups identified in USDA regulations. While we included associations from different geographic regions of the country, the sample was not intended to be representative of all associations. We documented the extent to which the marketing plans we reviewed contained information on the demographic characteristics of the population in the associations’ service areas and planned outreach activities for diversity and inclusion. We also documented examples of outreach to SDFRs that were ongoing or that they had completed. Further, we also reviewed illustrative examples of outreach materials to SDFRs developed by USDA and the federal depository institution regulators, and we interviewed officials from these agencies about their outreach efforts. To gain further insight into challenges faced by and outreach efforts to SDFRs, we interviewed (1) SDFR advocacy and research organizations, (2) industry group representatives, and (3) federal agency officials. We refer collectively to the entities we interviewed as stakeholders. To select SDFR advocacy and research organizations, we used a snowball sampling technique that identified organizations based on referrals obtained during prior GAO studies and referrals from stakeholder interviews during this study. We limited our interviews to organizations that are national in scope and that focus on one or more socially disadvantaged populations and on agricultural credit or finance. Based on the snowball sampling, we identified and interviewed representatives from the following five groups: Socially Disadvantaged Farmers and Ranchers Policy Research Center, National Sustainable Agriculture Coalition, National Black Farmers Association, Rural Coalition, and Rural Advancement Foundation International-USA. The snowball sampling did not identify a national advocacy organization focused on women farmers—the largest SDFR subgroup—but we identified American Agri- Women based on an internet search, and we interviewed representatives from that organization as well. Because the group of organizations we interviewed was a nonprobability sample, the information they provided is not generalizable. We also interviewed representatives from lending industry groups—the American Bankers Association, the Independent Community Bankers of America, and the Farm Credit Council—that we selected to cover the major types of private institutional lenders that make agricultural loans, including large commercial banks, community banks, and the Farm Credit System. Additionally, we contacted industry associations representing insurance companies and community development financial institutions— both of which provide some agricultural credit—but representatives from these associations said they did not have information directly related to our research topic. Finally, we interviewed officials from USDA and its Farm Service Agency, the Farm Credit Administration, CFPB, and the federal depository institution regulators. For our work on credit challenges faced by SDFRs, we also drew upon information and analysis from our May 2019 report on agricultural lending on tribal lands. Among other things, that report describes (1) what is known about the agricultural credit needs of Indian tribes and their members, (2) barriers stakeholders identified to agricultural credit on tribal lands, and (3) Farm Credit System authority and actions to meet those agricultural credit needs. We conducted this performance audit from January 2019 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Steve Westley (Assistant Director); Jeremy Anthony (Analyst in Charge); Katherine Carter; William Chatlos; Tom Cook; Sam Portnow; Jennifer Schwartz; Jena Sinkfield; Tyler Spunaugle; and Farrah Stone made key contributions to this report.", "summary": "In 2017, there were about 2 million farm and ranch operations nationwide. Farmers and ranchers often require loans to buy agricultural real estate, make capital improvements, and purchase supplies and equipment. However, minorities and women comprise a disproportionately small share of agricultural producers, and certain minority groups have alleged discrimination in obtaining agricultural credit. Most agricultural lending is done by either commercial banks or the Farm Credit System, a network of lenders regulated by the Farm Credit Administration. USDA accounts for a small share of agricultural credit, but it makes direct loans and guarantees loans made by private lenders. USDA and Farm Credit System lenders have responsibilities to expand credit access. Congress included a provision in statute for GAO to study agricultural credit services provided to SDFRs. USDA direct loans were outside the scope of GAO's review. This report examines (1) what is known about the amount and types of agricultural credit to SDFRs, (2) challenges SDFRs reportedly face in obtaining agricultural credit, and (3) outreach efforts to SDFRs regarding agricultural credit and related services. GAO analyzed survey, census, and other USDA data; reviewed statutes and regulations governing collection of personal data on borrowers; and reviewed Farm Credit Administration and USDA documentation on outreach to SDFRs. GAO also interviewed SDFR advocacy groups, lending industry groups, and officials from the Farm Credit Administration, USDA, and the federal depository institution regulators. Information on the amount and types of agricultural credit to socially disadvantaged farmers and ranchers (SDFR)—which the U.S. Department of Agriculture (USDA) defines as members of certain racial and ethnic minority groups and women—is limited. Comprehensive data on SDFRs' outstanding agricultural debt are not available because regulations generally prohibit lenders from collecting data on the personal characteristics of applicants for loans other than certain mortgages. A Consumer Financial Protection Bureau rulemaking pursuant to a provision in the Dodd-Frank Wall Street Reform and Consumer Protection Act that requires collection of such data in certain circumstances would modify this prohibition for certain loans, possibly including some agricultural loans. The bureau delayed the rulemaking in 2018 due to stated resource constraints and other priorities, but reported that it plans to resume work on the rule later in 2019. An annual USDA survey of farmers provides some insights into agricultural lending to SDFRs but, according to USDA, may underrepresent SDFRs compared to more inclusive estimates from the 2017 Census of Agriculture. In the 2015–2017 surveys, SDFRs represented an average of 17 percent of primary producers in the survey, but they accounted for 8 percent of outstanding total agricultural debt. Loans to purchase agricultural real estate accounted for most of SDFRs' outstanding debt (67 percent). SDFRs reportedly face a number of challenges that hamper their ability to obtain private agricultural credit. According to SDFR advocacy groups, lending industry representatives, and federal officials, SDFRs are more likely to operate smaller, lower-revenue farms, have weaker credit histories, or lack clear title to their agricultural land, which can make it difficult for them to qualify for loans. SDFR advocacy groups also said some SFDRs face actual or perceived unfair treatment in lending or may be dissuaded from applying for credit because of past instances of alleged discrimination. Additionally, they noted that some SDFRs may not be fully aware of credit options and lending requirements, especially if they are recent immigrants or new to agriculture. Private lenders and federal agencies conduct outreach to SDFRs, but the effectiveness of these efforts in increasing lending is unknown. For example, lenders have sponsored educational events targeted to SDFRs and translated marketing materials for non-English speakers. Farm Credit Administration regulations require Farm Credit System lenders to prepare marketing plans that include specific outreach actions for diversity and inclusion. The Farm Credit Administration examines these plans and indicated that it has prescribed corrective actions in some cases. However, the Farm Credit Administration does not require lenders to meet specific lending goals, and the regulatory data restrictions noted previously constrain the Farm Credit Administration's ability to assess the effect of outreach efforts. USDA conducts outreach to SDFRs and lenders about its loan programs and collects data on the personal characteristics of loan applicants. However, USDA officials said they face challenges evaluating the impact of their outreach efforts, in part because outreach participants are reluctant to provide their demographic information.", "document_type": "gao"}
{"report": "Through its ATON mission, the Coast Guard promotes safe waterways and an efficient Marine Transportation System. The Coast Guard has statutory responsibility to operate and maintain a system of maritime aids to facilitate navigation and to prevent disasters, collisions, and wrecks. To fulfill this mission, the Coast Guard operates and maintains ATON that are placed along coasts and navigable waterways as guides to mark safe water and to assist mariners in determining their position in relation to land and hidden dangers. As mentioned earlier, this report focuses on two categories of ATON: fixed ATON that include lighthouses, towers, and other structures that are directly affixed to the ground or seabed; and floating ATON that include buoys and markers anchored to the sea bed by a concrete or metal sinker connected by a metal chain or mooring. See figures 1 and 2 for examples of fixed and floating ATON. The Coast Guard uses several types of vessels to place and service fixed and floating ATON. These ATON vessels include buoy tenders, construction tenders, and boats. As of October 2019, the Coast Guard had a fleet of 79 ATON cutters and 190 ATON boats—which varied in size from a 240-foot Great Lakes Icebreaker to 16-foot ATON boats. (See appendix I for additional details on the Coast Guard’s fleet of ATON vessels.) The Coast Guard’s ATON program consists of several offices and units that work together to carry out the ATON mission: Office of Navigation Systems: Based at Coast Guard headquarters in Washington, D.C., the primary ATON-related roles and responsibilities of Office of Navigation Systems officials include providing oversight and approval for ATON operations and policy. Specifically, the Aids to Navigation and Positioning, Navigation, and Timing Division within the Office of Navigation Systems is responsible for establishing requirements and policy; providing program level guidance; and coordinating processes, platforms, and personnel necessary to establish, maintain, and operate the U.S. ATON system. Office of Civil Engineering: Based at Coast Guard headquarters in Washington, D.C., the primary ATON-related roles and responsibilities of Office of Civil Engineering officials include providing oversight and approval for ATON engineering and logistics policy, including supervision of the Shore Infrastructure Logistics Center. Shore Infrastructure Logistics Center (SILC): Based in Norfolk, VA, SILC supervises the Civil Engineering Units that execute fixed ATON depot-level maintenance and recapitalization projects; as well as the Waterways Operations Product Line. Waterways Operations Product Line (WOPL): A division of the Coast Guard’s Shore Infrastructure Logistics Center, WOPL was established by the Coast Guard in 2016 with the goal of serving as the focal point for implementing engineering and logistics solutions for ATON in order to enhance the mission while reducing costs. To do this, WOPL is to support the ATON mission by providing centralized guidance and oversight covering such issues as ATON acquisition, ATON configuration management (the proper mix of ATON) across the Coast Guard’s nine districts, ATON production and delivery, and ATON logistics and maintenance for the Coast Guard-wide inventory of ATON equipment and systems. WOPL’s support encompasses the entire lifecycle of ATON equipment and systems, from acquisition through disposal. Coast Guard Districts and Sectors: The Coast Guard has nine districts, which have overall responsibility for administration of the ATON within their district. Each district oversees the coordination of operations at the sectors and individual ATON units, which includes cutters, boats, and Aids to Navigation Teams. Figure 3 shows a map of the Coast Guard’s nine districts and the numbers of fixed and floating ATON in each district as of November 2019. The ATON units are responsible for the servicing and maintenance of ATON by conducting both routine servicing based on the last-service dates of the ATON and non-routine servicing of ATON within their area of responsibility. The non-routine servicing process includes responding to and addressing discrepant ATON, which are aids that are not functioning properly due to, for example, a weather-related event such as a hurricane, or an equipment failure. Timely response to and correction of discrepant ATON is a high-priority task for the Coast Guard. According to internal guidance, the Coast Guard has a tiered approach to address ATON discrepancies that accounts for the importance of the ATON relative to the waterway and the nature of the discrepancy. In particular, according to Coast Guard guidance, the servicing unit response ranges from immediately after notification up to 72 hours or as soon thereafter as weather and resources permit. In some cases, the determining factors do not require responding within 72 hours and the servicing unit is to advise the district of future plans to correct the discrepancy. Coast Guard guidance states that during the routine servicing process for floating ATON (buoys), the primary purpose of the ATON units is to check the buoys’ positions, their condition, and ensure the correct operation of the buoys’ signal hardware. As part of this process, the Coast Guard may extract the buoys from the water and bring them onboard an ATON vessel to check the condition of their mooring chain, hull, and lighting equipment. If necessary, the mooring chains are cleaned and repaired and non-functioning lanterns (lights) are replaced. After the planned repairs are made, the buoys are placed back in their assigned position in the water. See figure 4 for an example of the process used by an ATON unit to service a steel buoy. When ATON units conduct routine or non-routine servicing of fixed and floating ATON, they also collect data on the condition of the ATON. These data provide a “snapshot” of the ATON’s condition at the time of servicing and include the aid’s geographic position; the last date that the ATON was serviced; the next-scheduled service date; and other detailed information about the aid, such as an assessment of the physical integrity of the ATON. If warranted, ATON units can initiate action for repair or replacement of ATON if necessary. The information gathered by ATON units during their servicing activities is entered into a Coast Guard database—the Integrated ATON Information System (I-ATONIS)—that is used to track and monitor fixed and floating ATON. A hardcopy record containing detailed information about each aid is subsequently generated from I-ATONIS and stored in local unit files to track and schedule future fixed and floating ATON servicing dates. According to Coast Guard officials, based on the multi-mission nature of its assets and workforce, the Coast Guard does not budget for, request, or receive funding organized by specific missions or program activities. In addition, Coast Guard financial systems are not structured to collect accounting data by specific missions or program activities, and the Coast Guard does not report expenditures by mission. Rather, the ATON mission receives funding through various sources within the Coast Guard’s annual budget. Specific to repairs and recapitalization of fixed ATON, in fiscal year 2018, $300,000 was allocated from Procurement, Construction, and Improvement funding while $10 million was allocated from the Coast Guard’s Operations and Support funds for depot-level ATON maintenance. We previously reported on the Coast Guard’s management and maintenance of its shore infrastructure, which—in addition to fixed and floating ATON—encompasses over 20,000 shore facilities such as piers, docks, boat stations, air facilities, and housing units at more than 2,700 locations. In July 2018, we found that the Coast Guard had not been able to address many shore infrastructure projects, primarily due to lack of funding, longstanding acquisition management challenges, and that previous Coast Guard leadership prioritized the acquisition of new operational assets to replace aging vessels and aircraft over maintaining and repairing shore infrastructure. We recommended, among other things, that the Coast Guard’s annual Capital Investment Plans reflect acquisition trade-off decisions and their effects. The Coast Guard agreed with this recommendation, and estimated implementing actions by March 2020. In February 2019, we found that almost half of the Coast Guard’s shore infrastructure is beyond its service life, and its current backlogs of maintenance projects, as of 2018, will cost at least $2.6 billion to address. We found that the Coast Guard’s process to manage its shore infrastructure recapitalization and deferred maintenance backlogs did not fully meet 6 of 9 leading practices we previously identified for managing public sector maintenance backlogs. We recommended, among other things, that the Coast Guard establish shore infrastructure performance goals, measures, and baselines to track the effectiveness of maintenance and repair investments and provide feedback on progress made; develop and implement a process to routinely align Coast Guard’s shore infrastructure portfolio with mission needs, including by disposing of all unneeded assets; and employ models for its asset lines for predicting the outcome of investments, analyzing trade-offs, and optimizing decisions among competing investments. The Coast Guard agreed with our recommendations and is taking steps to implement them. The condition of fixed and floating ATON Coast Guard-wide declined slightly from fiscal years 2014 through 2018, as determined by the Coast Guard’s key ATON condition metric. In particular, according to data provided by the Coast Guard, the aid availability rate—the percentage of time ATON are functioning correctly—declined from 98.0 percent in fiscal year 2014 to slightly below the Coast Guard’s performance target percentage of 97.5 percent in fiscal years 2017 (97.4 percent) and 2018 (97.1 percent), as shown in figure 5. While the aid availability rate metric indicates that the condition of fixed and floating ATON Coast Guard-wide declined slightly from fiscal year 2014 through fiscal year 2018, other factors—such as the age of many ATON—have contributed to more significant declines in the condition of ATON for some locations. For example, an internal Coast Guard report states that, as of 2018, nearly a quarter (24 percent) of all floating ATON and over half (59 percent) of all fixed ATON are operating past their designed service lives. On a district level, the conditions of fixed and floating ATON differ from one geographical area to the next, and varying weather conditions often have an impact on the physical condition of ATON. For example, the frigid weather conditions of the Great Lakes in certain months frequently erode the condition of both fixed and floating ATON. Coast Guard officials stated that ATON with large steel hulls many times cannot withstand the pressure and weight of ice that can form on them in the winter months. They also stated that the icy waters delay routine servicing trips for personnel to adequately address ATON, which can contribute to the deterioration of the aids. In District 8’s area of responsibility, which includes much of the Gulf of Mexico, Coast Guard officials said that severe storms and hurricanes can adversely impact the condition of fixed and floating ATON and delay servicing trips for safety reasons. Extended periods of exposure to saltwater is another factor that contributes to the degraded condition of ATON in District 8 and elsewhere, as water salinity often corrodes the hulls of steel buoys. In addition to weather, geographic factors can affect the condition of ATON as well. Coast Guard officials in District 1 (headquartered in Boston) stated that the hard, rocky coast in their district makes it difficult to secure fixed ATON structures to the seabed. As a result, this district requires a higher percentage of floating ATON to mark the location of these hazards and these floating ATON are often damaged by the rocks. See figure 6 for examples of the deteriorating condition of some fixed and floating ATON. Our analysis of Coast Guard data shows that the Coast Guard’s overall repair and recapitalization expenditures for fixed and floating ATON increased during fiscal years 2014 through 2018. Specifically, our analysis of Coast Guard data shows that total ATON repair and recapitalization costs increased from about $12 million in fiscal year 2014 to about $20 million in fiscal year 2018. As shown in figure 7, the majority of the costs for fixed ATON were spent on repairs whereas the majority of the costs for floating ATON were spent on recapitalizations. According to Coast Guard documents, data, and officials, the Coast Guard has faced a variety of challenges in managing its fixed and floating ATON. The reported challenges include the availability of ATON vessels, difficulty in conducting routine ATON servicing in a timely manner, and capacity limits at ATON major repair and refurbishment facilities. Our analyses of Coast Guard data on maintenance required of ATON cutters and boats during fiscal years 2014 through 2018 show that ATON cutter and boat availability varied by type and across classes. As described below, our data analyses showed that 10 of the 12 ATON cutter classes consistently met availability targets, whereas 4 of the 7 classes of ATON boats consistently met availability targets. The Coast Guard determines the condition of its ATON cutters and boats using the following measures—planned and unplanned maintenance days, maintenance hours, and achieved material availability rate. Specifically, Planned maintenance days are the number of days that a vessel is not mission capable due to scheduled maintenance. This measure is applicable to cutters. Unplanned maintenance days are the number of days that a vessel is not mission capable due to unforeseen maintenance issues and associated repair efforts. This measure is applicable to cutters. Maintenance hours are the total number of hours that a vessel spent in maintenance, including both planned and unplanned maintenance. This measure is applicable to boats. Achieved material availability rate is calculated based on a vessel’s availability and performance. For cutters, the target availability rate range is between 53 percent and 65 percent. For boats, the target availability rate target is 80 percent. According to our analysis of Coast Guard data, the number of maintenance days for ATON cutters generally decreased during fiscal years 2014 through 2018, as shown in Figure 8. In addition, our analysis shows that the biggest decrease was with planned maintenance days. The Coast Guard has established a target range for the achieved material availability rate for ATON cutters that includes a minimum rate of 53 percent to a maximum rate of 65 percent. According to our analyses of Coast Guard data, the achieved material availability rate for the ATON cutters varied by cutter class during fiscal years 2016 through 2018, with 10 of the 12 cutter classes having met or exceeded the minimum target material availability rate for all 3 years and the remaining 2 ATON cutter classes having met or exceeded the minimum target material availability rate for 2 of the 3 years analyzed. While most of the ATON cutters met Coast Guard availability rate targets during fiscal years 2016 through 2018, officials in 7 of the 9 districts noted that maintaining some older ATON cutters can take longer to repair because of old and obsolete equipment and the lack of available parts, which decreases their availability to conduct missions. Figure 9 shows the achieved material availability rate for ATON cutters for fiscal years 2016 through 2018. According to our analysis of Coast Guard data, we found that the total number of maintenance hours for ATON boats generally decreased during fiscal years 2014 through 2018, although there was an increase from fiscal year 2017 to 2018. Figure 10 shows the total maintenance hours for ATON boats during fiscal years 2014 through 2018. In comparison to ATON cutters, which have a target range for the achieved material availability rate, ATON boats have a material availability threshold of 80 percent. According to our analyses of Coast Guard data, 4 of the 7 classes of ATON boats consistently achieved the 80 percent availability threshold during fiscal years 2014 through 2018. In particular, we found that the four smaller classes of ATON boats—those 16 to 26 feet in length—consistently achieved the 80 percent availability threshold during fiscal years 2014 through 2018, whereas the three larger classes of ATON boats—those 49-feet in length and longer—failed to consistently meet the 80 percent availability threshold during this 5-year period. In addition to the data on achieved material availability rates, Coast Guard officials from 3 of the 9 districts noted they experienced challenges with the availability of ATON boats. Figure 11 shows the achieved material availability rate for seven classes of ATON boats. The Coast Guard has taken positive steps to manage the ATON program, including issuing strategic plans and directions, creating a unit to provide a Coast Guard-wide perspective in managing ATON, and developing various initiatives to improve management of fixed and floating ATON. However, we found that some ATON-related initiatives to be implemented Coast Guard-wide, such as the foam buoy implementation initiative, do not contain certain elements that can provide better assurance that they will be effectively implemented, such as milestones and completion dates, and desired outcomes to be achieved. The Coast Guard has developed strategic plans and directions that provide guidance for addressing challenges faced in managing fixed and floating ATON. In June 2007, the Coast Guard issued the Maritime Short Range Aids to Navigation Strategic Plan to coordinate and standardize a number of ATON mission activities. According a Coast Guard official, at the time this strategic plan was issued, ATON units within the Coast Guard’s nine districts were operating largely independently in terms of planning and conducting ATON missions and activities. The 2007 plan changed this by developing a strategic approach to ATON management and it identified a number of initiatives to improve ATON program management, including reducing ATON lifecycle costs and maintenance needs, increasing efficiency and service intervals, and improving the performance and reliability of fixed and floating ATON. More recently, the Coast Guard issued the Navigation Systems Strategic Voyage Plan for Fiscal Years 2017-2022, which updates and expands on the 2007 strategic plan by identifying priorities that impact ATON program management broadly and the management of fixed and floating ATON in particular. The plan specifically identifies initiatives, including the use of non-steel floating ATON, development of year-round floating ice ATON, increased use of LED lighting, and the increased use of less expensive fixed ATON alternatives in lieu of lighthouses. In addition to the 2007 and 2017 strategic plans, the Coast Guard also issues annual Strategic Planning Directions. These annual directions outline the Coast Guard’s strategic commitments and are the primary mechanism for apportioning resources and providing guidance to field units on initiatives and actions to improve mission operations, including the ATON mission. For example, the Coast Guard has emphasized continuing to leverage electronic ATON technology where appropriate in an effort to reduce seasonal ATON workload, such as in districts with ATON in waters that are subject to freezing during a part of the year. In addition to developing a strategic approach to management of fixed and floating ATON through its strategic plans, the Coast Guard also created a new unit to provide a Coast Guard-wide, centralized perspective in managing fixed and floating ATON engineering and logistics. In particular, in 2016, the Coast Guard created the Waterways Operations Product Line (WOPL) to centrally manage the distribution, repair, and replacement of fixed and floating ATON and parts; as well as to formulate requests for ATON resources and funding. Since its creation, WOPL has coordinated and helped to implement various Coast Guard- wide initiatives to improve the management of fixed and floating ATON. These initiatives include centralized funding for ATON inspection and major repair services, changes in cost limits for floating ATON refurbishments, and expansion of commercial depot-level maintenance contracts to supplement the Coast Guard’s ATON major repair and refurbishment capacity. WOPL has also analyzed and recommended the transition from steel to foam buoys, where appropriate, to increase life cycle cost savings and reduce servicing times. In addition, WOPL has initiated changes to better manage and sustain the duration of floating ATON, including extending time in the water between major refurbishments from 6 to 9 years for some buoys and increasing the allowance for selected steel buoy hull repair weld hours. The Coast Guard has developed and is implementing a variety of initiatives to address specific ATON management challenges that were discussed earlier in this report. These initiatives include the following: Improving the Availability of ATON Cutters and Boats: The Coast Guard has ongoing initiatives to extend the service lives and to recapitalize certain ATON cutters and boats to improve their availability rates. For example, in fiscal year 2019, the Coast Guard continued the major maintenance availability efforts on the 225-foot Seagoing Buoy Tender fleet. In addition, from 2006 to 2016, a portion of the Coast Guard’s ATON fleet (River Tenders, Buoy Tenders, and Construction Tenders) underwent a limited maintenance program to act as a bridging strategy until replacement assets could be acquired. Our 2018 report on Coast Guard acquisitions noted that the designed service life for each of these tenders is 30 years, but as of the time of the report, their average age was 53 years. In 2018, we reported that the Department of Homeland Security approved the Waterways Commerce Cutter Program to replace aging River Tenders, Buoy Tenders, and Construction Tenders. While the acquisitions have been approved, it will likely be years before the new cutters are built and deployed. The Coast Guard has also had an ongoing initiative since 2007 that has recapitalized the boat fleet by 290 boats. Conducting Routine ATON Servicing in a Timely Manner: The Coast Guard has issued guidance to its districts to look for opportunities to reduce the number of ATON that do not significantly increase navigational risk and explore and leverage new technologies, such as the use of electronic ATON, where feasible. Collectively, these efforts should help to ease the servicing burden. In addition, the Coast Guard has also introduced initiatives focused on improving ATON servicing time. For example, officials in one district told us that they require their ATON units to send in monthly reports on ATON servicing due dates and plans. District officials review this information and may shift ATON servicing work to another unit when the primary servicing vessel or unit is not available to provide the needed service in a timely manner. Another ongoing initiative the Coast Guard is exploring is the use of year-round buoys for ice prone areas to reduce servicing requirements. The Coast Guard has received positive feedback in two of the three districts where such buoys have been in service. Improving Capacity Limits at ATON Major Repair and Refurbishment Facilities: According to a Coast Guard official, the Coast Guard has had commercial contracts in District 9 (the Great Lakes region) and District 13 (the Pacific Northwest) going back decades to provide floating ATON major repair and refurbishment services. Then, in March 2019, WOPL awarded four regional commercial contracts to provide increased capacity for ATON major repairs and refurbishments in an effort to help reduce the Coast Guard’s floating ATON major repair and refurbishment backlog. Specifically, the Coast Guard (1) renewed the contract in District 13; (2) awarded a contract covering California (part of District 11); (3) awarded a contract for a zone covering New England and the Mid-Atlantic (Districts 1 and 5); and (4) awarded a contract covering Guam (part of District 14). According to Coast Guard officials, the addition or renewal of these four regional contracts has resulted in greater capacity and flexibility to reduce the floating ATON major repair and refurbishment backlog. While the Coast Guard has developed various initiatives to improve management of fixed and floating ATON, these initiatives do not contain certain elements, which limit assurance that the initiatives will be effectively implemented. For example, we found that many initiatives we evaluated do not contain milestone and completion dates for Coast Guard-wide implementation, which are elements that can guide decisions on the success of the initiatives. Under the foam buoy implementation initiative, the Coast Guard evaluated the use of foam buoys in lieu of steel buoys (which are more expensive to overhaul) and found that it was feasible to replace steel buoys with foam buoys in some locations but not in others. For example, the Coast Guard’s evaluations showed that foam buoys cannot stand up to ice conditions. On the basis of its evaluations, the Coast Guard plans to continue replacing certain classes of steel buoys with foam buoys where operationally feasible. However, the initiative does not contain milestone dates or desired outcomes for Coast Guard-wide implementation. According to guidance from the Program Management Institute, programs or projects—like the ATON-related initiatives being implemented by the Coast Guard—are to include specific, desired outcomes, along with the appropriate steps and time frames needed to achieve the final outcomes and results to implement the enhanced capabilities across the organization. In addition, our leading practices in capital decision- making state that such initiatives should include milestones and completion dates. According to Coast Guard officials, WOPL is a relatively new unit and is still developing ATON guidance and procedures for ATON-related initiatives and responsibilities to be performed by the districts. By updating its ATON-related initiatives to include the specific outcomes desired and timeframes for completing them, the Coast Guard would have better assurance that its initiatives to address ATON management challenges will be effectively implemented. Available Coast Guard data indicate that despite some slight declines in the condition of fixed and floating ATON, and increasing repair and recapitalization costs for floating ATON, the Coast Guard’s ability to meet its ATON mission did not show a marked decline during fiscal years 2014 through 2018. However, the future of the fixed and floating ATON and the vessels used to service them bears close watching given the challenges the Coast Guard faces in managing its fixed and floating ATON. The fact that many of the ATON have reached, or will soon be reaching, the end of their designed service lives could lead to an increase in the number of ATON requiring major repairs or replacement in the near future. According to Coast Guard data, the Coast Guard’s ability to refurbish or replace its aging ATON is made more challenging by limited capacity for conducting major repairs and refurbishments of floating ATON. The Coast Guard has taken positive steps to develop strategic plans to guide the ATON program, and these plans have led to the development of various initiatives to improve management of fixed and floating ATON, but these initiatives would benefit from the inclusion of certain elements, such as desired outcomes to be achieved and associated milestone dates, to have better assurance that the initiatives will be effectively implemented. The Commandant of the Coast Guard should direct the Assistant Commandant for Engineering and Logistics and Assistant Commandant for Prevention Policy to update the Coast Guard’s ATON-related initiatives to include the specific outcomes to be achieved and associated time frames. (Recommendation 1) We provided a draft of this report to DHS for review and comment. In its comments, reproduced in appendix II, DHS concurred with our recommendation and stated that the Coast Guard plans to review and update ATON-related initiatives to include specific outcomes with associated implementation milestones by December 31, 2020. DHS also provided technical comments that we incorporated into the report, as appropriate. We are sending copies of this report to the appropriate congressional committee, the Secretary of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (206) 287-4804 or AndersonN@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Figure 14 provides information on the cutters and boats that comprise the Coast Guard’s fleet of aids to navigation (ATON) vessels. In addition to the contact name above, Christopher Conrad (Assistant Director), Hugh Paquette (Analyst in Charge), Chuck Bausell, Breanne Cave, Benjamin Crossley, Dorian Dunbar, Michele Fejfar, Tracey King, Joshua Lanier, and Adam Vogt made significant contributions to this report.", "summary": "One of the Coast Guard's statutory missions is the care and maintenance of ATON. Much like drivers need signs and universal driving rules, mariners need equivalent nautical “rules of the road.” As of November 2019, the Coast Guard managed 45,664 federal fixed and floating ATON that are designed to assist those operating in the U.S. Marine Transportation System, which includes about 25,000 miles of waterways, 1,000 harbor channels, 300 ports, and 3,700 terminals. According to the Coast Guard, as of July 2018, these ATON had a collective replacement value of about $1.6 billion. The Coast Guard has faced an array of challenges in managing its ATON, such as deteriorating buoys, and questions have been raised regarding the extent to which the Coast Guard is addressing these challenges. This report (1) describes what is known about the condition and costs of maintaining the Coast Guard's ATON, and (2) examines challenges the Coast Guard has experienced in managing its ATON and how it is addressing them. To address these issues, GAO reviewed ATON regulations and guidance, analyzed data on ATON condition and cost measures, collected input from all nine Coast Guard districts on ATON challenges, accompanied ATON units on mission activities, assessed agency initiatives using leading program management practices, and interviewed headquarters and field unit officials. The condition of the Coast Guard's aids to navigation (ATON), both fixed (e.g., lighthouses) and floating (e.g., buoys), have declined slightly while the overall costs for repairing or replacing them increased in recent years. According to Coast Guard data, its key metric for ATON condition—the Aid Availability Rate, or percentage of time that ATON are functioning correctly—declined from 98.0 to 97.1 percent during fiscal years 2014 through 2018, dipping slightly below the 97.5 percent target rate in fiscal years 2017 and 2018. During this time period, the overall costs to repair and replace ATON increased from about $12 million in fiscal year 2014 to about $20 million in fiscal year 2018. According to Coast Guard data, the majority of the costs for fixed ATON were spent on repairs whereas the majority of the costs for floating ATON were spent on replacements. The Coast Guard faces challenges in managing its fixed and floating ATON and has developed plans and initiatives to address them, but it has limited assurance that the plans and initiatives will be effectively implemented. According to Coast Guard officials, the challenges include decreased availability of vessels to service ATON, reduced ability to provide routine ATON servicing in a timely manner due to severe weather, among other factors, and limited capacity at ATON major repair and refurbishment facilities. The Coast Guard has developed plans to guide the ATON program, and these plans have led to the development and implementation of various initiatives at the headquarters and field unit levels to address these challenges. However, GAO found that the initiatives do not contain certain elements that help ensure effective implementation—such as desired outcomes and schedule milestones and completion dates—as recommended by leading program management practices. According to Coast Guard officials, they are still developing guidance and procedures for ATON-related initiatives that are to be implemented by the districts. By updating these initiatives to include certain elements, such as the specific outcomes desired and timeframes for completing them, the Coast Guard would have better assurance that its initiatives to address ATON management challenges will be effectively implemented. GAO recommends that the Coast Guard update its ATON initiatives by including the specific outcomes to be achieved and associated time frames. The Department of Homeland Security concurred with the recommendation.", "document_type": "gao"}
{"report": "GDUFA and GDUFA II both provided supplemental resources to FDA by giving it the authority to collect user fees from the generic drug industry, in addition to its regular appropriations, in order to make improvements to the generic drug application review process. GDUFA was enacted in July 2012, in part, to provide funding for more generic drug application reviewers at FDA in order to handle an increase in the volume of application submissions and speed up reviews. GDUFA II was enacted in August 2017 to reauthorize the generic drug user fee program from fiscal year 2018 through fiscal year 2022. In return, FDA provided Commitment Letters to Congress that detailed its plans to implement program enhancements and meet certain performance measures related to the review of generic drug applications. For example, in its Commitment Letter for GDUFA II, FDA stated that, for applications in the first review cycle, it would review and act on at least 90 percent of them within specified timeframes—8 months for certain priority applications and 10 months for all other applications. FDA stated that it met this performance measure for fiscal year 2018. The Commitment Letter also indicated that one of the goals of GDUFA II is to minimize the number of review cycles for applications to attain approval. FDA’s generic drug application review process includes a number of steps. The process begins when a generic drug application is submitted to FDA for review by the Office of Generic Drugs and the Office of Pharmaceutical Quality within FDA’s Center for Drug Evaluation and Research. The Office of Generic Drugs is responsible for providing regulatory oversight and strategic direction for FDA’s generic drug program to make safe, effective, and high-quality generic drugs available to the public. The Office of Generic Drugs’ Division of Filing Review determines whether the generic drug application is acceptable for review, meaning that the application is sufficiently complete for FDA to review the application, such as information about the amount of active ingredients present in the drug. If the generic drug application is not acceptable for review, FDA issues a Refuse to Receive letter to the applicant explaining what additional information is required before the application can be accepted for review. In response, the applicant can resubmit the generic drug application with additional information and officials within the Division of Filing Review will assess the information and determine if the resubmitted application is acceptable for review. Once the Division of Filing Review determines that a generic drug application is acceptable for review, the first review cycle begins, and FDA officials review the application across the following three review disciplines: Bioequivalence. Officials within the Office of Bioequivalence are responsible for examining whether the generic drug application is bioequivalent to the brand-name drug, meaning that the drug delivers the same amount of active ingredient in the same amount of time as the brand-name drug. Labeling. Officials within the Division of Labeling Review are responsible for ensuring that the proposed labeling language in the generic drug application, including the drug’s prescribing information, matches the language found in the labeling of the corresponding brand-name drug. Pharmaceutical quality. Officials within the Office of Pharmaceutical Quality have responsibility for ensuring the quality of drug products and assessing all drug manufacturing facilities, including both domestic and foreign. Officials within this office assess the risk of toxic substances or bacterial content in the drug, among other things. All generic drug applications are reviewed by primary reviewers and secondary reviewers. Primary reviewers assess the application to ensure that the documentation meets regulatory requirements in their specific disciplines. For example, primary reviewers within the Office of Pharmaceutical Quality evaluate documentation related to the proposed drug manufacturing process to ensure that the process produces quality drugs consistently. Secondary reviewers ensure the quality and consistency of primary reviewers’ assessments and the clarity of communication to applicants. During the first review cycle, FDA communicates with applicants when issues arise during its review that may prevent the agency from approving the generic drug application. This communication is typically made through the issuance of information requests or discipline review letters: Information requests. Information requests are letters sent to applicants to request clarification or additional information that is needed or would be helpful for FDA to complete its review. These letters may be sent by any review discipline and can be sent at any point after the Office of Generic Drugs accepts the generic drug application for review. Discipline review letters. Discipline review letters are letters issued by each review discipline at about the mid-point of the review cycle to identify possible deficiencies. FDA officials said they aim to issue these letters no later than 6 months into the first review cycle. In response to these letters, applicants can submit additional information for FDA to consider before the end of the review cycle. After FDA’s three review disciplines complete their review of an application and any additional information the applicant has submitted in response to information requests and discipline review letters, the agency issues an action letter that informs the applicant of whether the application is approved, marking the end of the first review cycle and the end of FDA’s review if the application is approved. There are three types of action letters: Approval letter. Issued when the agency has concluded its review of a generic drug application and the applicant is authorized to commercially market the drug. Tentative approval letter. Issued when the agency has completed its review of an application and has concluded that the generic drug application is sufficient, but patents or other exclusivities prevent approval. A tentative approval letter does not allow the applicant to market the generic drug. Complete response letter. Issued at the completion of the review of an application where deficiencies remain at the end of the review cycle. The complete response letter describes any deficiencies that must be corrected in order for the application to be approved. For a generic drug application that receives a complete response letter, the applicant can amend the application and seek another full review, which begins the second or subsequent review cycles. During these cycles, FDA officials review changes made to generic drug applications in response to deficiencies that FDA identified. Our analysis of FDA data shows that 12 percent of the 2,030 generic drug applications that FDA reviewed in fiscal years 2015 through 2017 received approval in the first review cycle. See figure 1. We identified several factors, including certain characteristics of generic drug applications, that may have contributed to whether an application received approval in the first review cycle, including the sufficiency of the application, deficiencies in drug quality, the type of drug reviewed, and the application’s priority status. Sufficiency of the application. We found that the sufficiency of the generic drug application, including the completeness of the application and the degree to which the applicants understood and fulfilled application requirements, affected its likelihood of receiving an approval in the first review cycle. According to FDA, one indication of the sufficiency of the generic drug application is whether FDA had previously refused to receive the application for review because it was not substantially complete upon its first submission. Our analysis of FDA data found that applications that had previously been refused were slightly less likely to be approved in the first review cycle compared with applications that had not previously been refused, and rates of approvals decreased for applications with two previously refused attempts. See table 1. According to stakeholders we interviewed, the sufficiency of a generic drug application may partially reflect the level of experience the applicant has in submitting applications, and we found some evidence to support this explanation. FDA managers and reviewers said that, in general, less experienced applicants are more likely to produce lower-quality generic drug applications compared to applicants with relatively more experience. Our analysis of FDA data found that applicants that submitted just one application during fiscal years 2015 through 2017 had a slightly lower rate of approval in the first review cycle (10 percent) compared to the rate across all applicants (12 percent). Additionally, in our review of 35 selected generic drug applications, we identified three applications that were from applicants that had no previously approved generic drug application submissions, an indication that they may have little or no experience with these applications. None of the three applications were approved in the first review cycle. One of these applications elicited reviewer comments that outlined basic application requirements, potentially reflecting the lack of experience of the applicant. Drug quality deficiencies. Our review of documentation from the first review cycle for 35 generic drug applications included 26 that were not approved in that cycle. Among those 26 applications, the most common deficiencies that remained at the end of the first cycle were related to the quality of the drug—356 out of 435 deficiencies. These deficiencies included issues related to the drug manufacturing facilities, which can affect the quality of the drug produced and the stability of the drug over time, among others. Officials from one large applicant told us that most of the comments they received from FDA reviewers are related to the quality of the drug. Three out of five applicants we interviewed also noted that the results from inspections of drug manufacturing facilities—which FDA includes as a component of its review of the quality of the drug—are a factor that may cause an application not to be approved in the first review cycle. Among the 26 applications we reviewed that were not approved, eight had an outstanding deficiency related to the manufacturing facility. Type of drug. We also found that the rate of approval in the first review cycle differed based on certain characteristics of the type of drug reviewed, including the route of administration, which may indicate the complexity of the drug. Complexity can also be influenced by other factors including, for example, the drug’s active ingredient or formulation. FDA officials noted that some complex drugs—including those that combine drug products with drug delivery devices, such as asthma inhalers—are less likely to be approved in the first review cycle. Officials we interviewed from one large applicant—which we identified based on the number of approved generic drug applications it had in fiscal year 2018—reported that their company had never submitted a generic drug application for a complex drug product that received approval in the first review cycle despite having significant experience with producing complex drugs. Officials we interviewed from another applicant said that very few of its dermatological products, which are considered complex, had received approval in the first review cycle. In our review of FDA data, we also found that applications for drugs with certain routes of administration—the method by which the drug is taken, such as oral, topical, or intravenous—had different rates of approval in the first review cycle. In particular, from fiscal years 2015 through 2017, FDA reviewed generic drug applications for 41 ophthalmic and 20 transdermal drugs—types of drugs that FDA considers complex—and none of these applications received approval in the first review cycle. In contrast, generic drug applications for topical drugs, which FDA also identifies as complex, had higher approval rates. Specifically, our analysis found that the rate of approvals in the first review cycle for generic drug applications for topical drugs was 25 percent—more than double the rate for all applications included in our analysis. FDA officials stated that in recent years FDA released several product-specific guidances for topical drugs—technical guidance intended to help applicants identify the most appropriate methodology for developing certain drugs and generating the evidence needed to gain approval. FDA officials told us these guidances may have contributed to the higher rates of approval in the first review cycle for topical drugs. See table 2. Generic drug application priority review designation. In addition, we found that a generic drug application’s priority review status may affect the rate of approval in the first review cycle. FDA may grant priority review status to applications under several circumstances, including for the first generics of brand-name drugs and other designations, such as for drugs that could help address public health emergencies. Our analysis of FDA data found that the rates of approval in the first cycle were lower for applications for first generics than for applications with no priority designation—6 percent and 14 percent, respectively. One potential explanation for the relatively low rate of approval is that for a first generic, applicants have a financial incentive to be the first to submit an application to FDA. Officials from one trade association stated that applications for first generics may be of lower quality because the applicants are rushing to submit their applications. In other cases, priority designations were associated with higher first-cycle approval rates. First-cycle approval rates for applications with other types of priority designations were higher than for applications with no priority designation—18 percent for applications that were marked as priority for other reasons, such as drug shortages or public health emergencies. See table 3. Our review of FDA guidance and regulations found that FDA has taken steps to enhance communication with applicants to increase the rate of generic drug application approvals in the first review cycle. Specifically, since the beginning of fiscal year 2013 in response to GDUFA, FDA increased communication with applicants prior to and during a generic drug application’s first review cycle consistent with its goal of helping applicants prepare approvable applications. These changes have included the following: Additional product-specific guidance. FDA has continued to release new and revised product-specific guidance to support a generic drug application’s approval within the first review cycle. Since GDUFA’s implementation and at the time of our review, FDA told us that it issued 993 new and revised product-specific guidance documents that describe acceptable methodologies for developing generic drugs and generating evidence needed to support a generic drug application’s approval. FDA officials indicated that product- specific guidance helps streamline both application development and review. Additional regulatory guidance. FDA has also issued regulatory guidance to communicate the agency’s expectations for the content and format of generic drug applications, which can facilitate approval in the first review cycle. In addition, in 2018, FDA issued draft guidance that described common application deficiencies and sought to promote approval during the first review cycle by providing recommendations on avoiding these recurring deficiencies. Presentations to industry. In presentations to applicants and others, FDA officials have presented information about generic drug application reviews and deficiencies frequently identified in generic drug applications. FDA has posted some of these presentation materials, including several video recordings, publicly on its website to share the information with industry. Communication during the review cycle. FDA has changed its review process to encourage reviewers to communicate with applicants at about the mid-point of the review cycle. FDA reviewers now aim to issue discipline review letters at about the mid-point of the first review cycle, rather than waiting until the end of the review cycle to communicate deficiencies. According to FDA, these earlier communications are intended to provide applicants with an opportunity to address issues before the end of the first review cycle and facilitate more approvals during that cycle. Assistance with applications for complex drugs. FDA has taken steps to assist applicants developing generics of complex drugs, such as drugs with complex active ingredients, formulations, or routes of administration. Beginning in fiscal year 2018 when GDUFA II was implemented, applicants developing complex drugs may request meetings with FDA prior to submitting a generic drug application and at the mid-point of the review cycle. In addition, FDA officials told us that since fiscal year 2013 when GDUFA was implemented, the agency has released 378 product-specific guidance documents focused on complex drugs. While all five applicants we interviewed generally described FDA’s efforts to increase communication as helpful, they offered opportunities for improvement. For example, four out of the five applicants said that product-specific guidance helps them understand exactly what the requirements are for product development, which can facilitate approval in the first review cycle. However, two of these applicants also said that FDA does not obtain sufficient input from the generic drug industry when developing product-specific guidance documents. To increase transparency, stakeholders said that FDA could solicit industry input to avoid proposing unrealistic guidance and one stakeholder suggested that FDA could create a workgroup to prevent unintended consequences following the implementation of new and revised guidance. However, FDA officials told us that draft guidance documents have a public comment period for stakeholders to provide comments on guidances before they are finalized. In December 2017, we similarly found that stakeholders indicated they would benefit from greater transparency in FDA’s process for developing guidance. We recommended that FDA publicly announce the agency’s plans for issuing new and revised product-specific guidance for nonbiological complex drugs within the next year. FDA agreed with this recommendation and published a website in April 2019 that provides information about upcoming product-specific guidance documents for complex generic drugs. In addition, all five applicants we interviewed said that FDA has improved communication with them, such as by increasing the frequency and timing of communications, and two applicants indicated that discipline review letters add predictability to the review process. (See app. I for more information on FDA’s changes to the generic drug application review process to improve communication with applicants.) While all five applicants we interviewed generally agreed that the increased communications would help increase rates of first-cycle approval, they also suggested that additional flexibility to communicate with FDA informally mid-cycle, such as by phone, could further facilitate the review process by helping applicants respond to questions or get clarity on questions included in the information requests or discipline review letters. FDA officials told us that applicants currently can request teleconferences with FDA, such as after receiving a complete response letter; however, they also noted that applicants generally prefer email communications and such teleconferences are not frequently utilized. FDA has also taken steps to improve consistency among reviewers. These steps could facilitate more approvals in the first review cycle because receiving consistent comments from FDA reviewers typically makes it easier for applicants to respond more quickly, which—according to some stakeholders—can result in approval in the first review cycle. These changes include the following: Creating review templates. Officials from FDA’s review disciplines have developed templates to guide reviewers through the generic drug application review process. According to FDA’s Manual of Policies and Procedures, these templates are intended to increase reviewers’ efficiency and improve assessment consistency. Developing common phrases. FDA also issued internal guidance on common phrases that reviewers may use to communicate generic drug application deficiencies in their comments to applicants during the first review cycle. For example, officials from the Division of Labeling Review said they maintain a database of common phrases and train reviewers on how to explain deficiencies to applicants. Officials explained that the Division of Labeling Review is also working toward pre-populating some parts of the review template to increase efficiency and consistency in the review process. Understanding the generic drug industry. FDA has taken steps to increase reviewers’ and applicants’ common understanding of industry practices and FDA review standards, such as through visits to manufacturing facilities, to improve the quality and consistency of reviewers’ comments in the first review cycle. However, all five applicants we interviewed noticed inconsistency among reviewers. FDA officials and two applicants suggested that this may be because FDA reviewers have different professional backgrounds. One applicant noted that some reviewers benefit from visiting manufacturing facilities if they do not have prior experience in the generic drug industry. Officials from a trade association said that such steps improve applicants’ understanding of what FDA reviewers are looking for in generic drug applications and may enhance their ability to submit applications that are approvable in the first review cycle. While stakeholders stated that the changes FDA made to improve reviewer consistency were positive, they noted that inconsistency among reviewers still remained, and we also found inconsistencies among reviewers. In addition, while stakeholders we interviewed raised concerns that the timing of brand-name labeling changes could affect whether applications were approved in the first review cycle, FDA has not taken steps to assess the validity of these concerns. Inconsistency among FDA reviewers. While FDA has taken steps to improve consistency among generic drug application reviewers, stakeholders noted that inconsistencies persist, and these inconsistencies may influence whether an application is approved during the first review cycle. For example, most stakeholders we interviewed (three out of five trade associations and four out of five applicants) indicated that they were aware of examples when different FDA reviewers within the same review discipline provided substantively different assessments of similar generic drug applications, specifically by requesting additional information from applicants for some applications and not others. For example, one applicant cited an example of two similar topical drugs whose applications relied on the same data set. The reviewer for one application required additional data, while the reviewer for the other application did not. To improve consistency, four applicants we interviewed suggested that FDA improve its reviewer training and one suggested that FDA create a workgroup to examine and address inconsistencies among reviewers. Four of the five applicants we interviewed also reported variation in the consistency of reviewers’ comments, including a lack of clarity in the information required for the applicant to address the comments. For example, one applicant said that they have received comments where reviewers did not specify what further information was required, and added that comments that suggest specific resolutions are extremely helpful, which would be consistent with FDA’s Manual of Policies and Procedures. This manual describes a standard process for FDA reviewers to use when assessing the completeness of generic drug applications, including clearly communicating with applicants about deficiencies that must be corrected for their applications to be approved in order to reduce the number of review cycles. According to the manual, primary reviewers are responsible for assessing whether applications meet the regulatory requirements for approval, while secondary reviewers are responsible for ensuring consistency among assessments and quality of communications to the applicant. Further, the manual advises primary and secondary reviewers to ensure that comments to applicants about deficiencies include similar content. The manual also indicates that comments should include the following four elements: (1) refer to a specific location within the generic drug application; (2) identify the omitted information or explain the problem with the information submitted; (3) explain the actions necessary to resolve the deficiency; and (4) explain why the information or revision is needed. Finally, FDA provides reviewers with plain language writing guidelines and other writing resources to support the development of clear messages for external communications. In our review of documentation from the first review cycle for 35 generic drug applications, we found variation in the clarity and specificity of some reviewer comments that may have influenced the outcome of the first review cycle. For example, some discipline review letters included clear descriptions of potential remedies for some deficiencies, while others did not clearly describe the deficiency or FDA’s expectations for an approvable generic drug application. Of the 35 generic drug applications, we conducted a more detailed review of four applications from fiscal year 2018—two that received approval in the first review cycle and two that did not. In the two applications that were not approved in the first review cycle, we identified 32 instances in which the comments did not fully meet FDA’s criteria. Two of the four applications we reviewed in more detail had similar numbers and types of deficiencies identified in the discipline review letters sent to the applicants mid-way through the first review cycle, such as quality deficiencies related to the drug substance and drug product, but the clarity and content of the comments included in the discipline review letters varied considerably. For one of the applications, the comments were written clearly and, consistent with FDA’s Manual of Policies and Procedures, identified options for addressing the deficiencies. For the other application, the comments were less clear and did not clearly identify ways to address the deficiencies. The applicant of the generic drug application with clearly written comments resolved all of the deficiencies raised in their discipline review letter and the generic drug application was approved in the first review cycle. In contrast, the applicant of the other application did not resolve all deficiencies raised in their discipline review letter and was not approved in the first cycle. See Table 4 for more detail on these two examples. Inconsistency among reviewers could affect the rate of approvals in the first review cycle if comments provided to applicants differ in content or are not clearly communicated. For example, if some reviewers provide unclear comments, it could be more difficult for the applicant to address deficiencies in a timely manner, while applicants that received clear comments could potentially address deficiencies within the first review cycle. This could delay some generic drugs from entering the market if applicants require more time, including potentially additional FDA review cycles, to understand and respond to unclear comments. This has potential impacts on patient access to generic drugs and on applicants’ abilities to effectively manage their expectations for when their generic drug applications will be approved. FDA officials explained that although secondary reviewers are experienced, they do not consistently receive additional training to ensure clarity and consistency among primary reviewers. They noted that FDA offers training in clear writing for FDA employees but it is not required for reviewers. FDA managers noted that some inconsistency among reviewers may persist due to various factors such as tenure with the agency and different professional backgrounds or interpretations of the generic drug application information. They also said that standardizing reviewers’ writing is challenging since each reviewer might have his or her own writing style and scientific expertise. Two applicants and one trade association we interviewed also said that the length of reviewers’ tenure with the agency could impact the substance of their comments in information requests and discipline review letters and the likelihood of the applicant attaining approval in the first review cycle. For example, one trade association we interviewed said that inexperienced reviewers typically request information from applicants that a more experienced reviewer would already know, such as information about drug manufacturing facilities. Unknown effects of labeling changes. Three applicants we interviewed noted that they believe FDA could improve the rate of approvals during the first review cycle if they took steps to mitigate delays that stakeholders said result from brand-name drug labeling changes that occur mid-cycle. Because generic drug labels generally must match brand-name labels, most applicants we interviewed said that changes made by brand-name drug companies to the labeling of drugs during the review process can delay or prevent approval of generic drugs in the first review cycle because the applicant of the generic drug would likely need to update the label before it is approved. In addition, five of the 10 stakeholders we interviewed said they believe such labeling changes negatively impact the rate of first-cycle approvals, and three said they believe that brand-name companies may strategically time updates to a brand-name drug’s labeling to occur right before the approval of a generic competitor in order to delay generic drug approvals. Although our review of 35 selected applications did not identify examples where labeling changes made during the first review cycle were the only factor that prevented approval, we identified two instances where labeling changes were among multiple factors that prevented approval. Specifically, we identified two applications for which the complete response letters noted recent changes in the brand-name drug’s labeling as one of multiple factors that contributed to the generic drug application’s failure to receive approval in the first review cycle. One of these applications had successfully addressed several labeling deficiencies during the first review cycle, but received a complete response letter that included new labeling deficiencies because of recent changes in the brand-name drug’s labeling. Three applicants and one trade association identified labeling changes as a concern during our interviews. Two of these applicants and the trade association suspected updates are strategically timed to delay generic drug approvals. However, FDA does not know whether there is validity to these concerns because it has not conducted analysis that would enable it to assess their validity. FDA officials noted that they were aware of these types of concerns, but thought it would be difficult for brand-name drug companies to successfully time changes in their drugs’ labeling to affect applications under review, and that labeling changes for brand- name drugs must be justified, for example, to note an adverse reaction identified after approval. However, FDA officials also acknowledged that there may be an incentive for brand-name drug companies to change the label on a drug frequently to make it more difficult for a generic drug application to be approved. FDA officials stated that the Office of Generic Drugs does not assess how often brand-name companies change their labeling or track how often such labeling changes occur because such changes are reviewed by the Office of New Drugs. Further, while FDA officials noted that the two offices have coordinated on the review of some labeling changes, they stated that they do not coordinate on the timing of approval of brand- name drug label changes. This is inconsistent with federal internal control standards, which state that agencies should identify risks that affect their defined objectives and use quality information to achieve these objectives, including by identifying the information required to achieve the objectives and address related risks. In addition, FDA can approve a generic drug application even though changes have been made in the brand-name drug labeling that the applicant has not incorporated into its proposed labeling, provided the applicant meets certain criteria; however, FDA officials told us that applications rarely meet the required criteria. Conducting an assessment of the extent to which the timing of such changes affect the approval of generic drugs in the first cycle of review would provide FDA with the necessary information to respond to stakeholder concerns and take action, as appropriate, such as by coordinating with the Office of New Drugs on this issue. To the extent that brand-name companies’ labeling changes are creating unnecessary delays in generic drug approval, such delays may impede generic drug entry into the market, which would be inconsistent with FDA’s stated goals of speeding up generic drug reviews. The timely approval of safe generic drugs in FDA’s first review cycle can provide substantial cost savings to patients and third-party payers. Since the enactment of GDUFA, FDA has taken steps to help applicants submit stronger generic drug applications and correct deficiencies within the first review cycle. However, according to FDA its most recent analysis found that the average generic drug application required three cycles of review before approval. Opportunities exist to enhance FDA’s efforts to increase the rates of approval for generic drugs in the first review cycle, including improving the consistency and clarity of reviewer comments and assessing the effects of the timing of brand-name companies’ changes to labeling. Taking such steps could help FDA meet the agency’s goals of minimizing the number of review cycles necessary for generic drug application approval and increasing the overall rate of approval, including within the first review cycle. Increasing the rate of approval in the first review cycle, while maintaining the efficacy and safety of generic drugs, can expand consumer access to relatively lower cost medications and has the potential to save patients and third-party payers billions of dollars. We are making the following two recommendations to FDA. The Commissioner of FDA should take additional steps to address inconsistency in its written comments to generic drug applicants— including the clarity of writing and the content of comments—among reviewers, such as requiring additional training for reviewers. (Recommendation 1) The Commissioner of FDA should assess the extent to which the timing of brand-name drug companies’ drug labeling changes affect the approval of generic drug applications in the first review cycle, and take steps, as appropriate, to limit the effect of brand-name drug labeling changes on pending generic drug applications. (Recommendation 2) We provided a draft of this report to HHS for review and comment. In its written comments, which are reproduced in appendix II, HHS concurred with our recommendations. HHS stated that it will take steps to improve the clarity and content of primary reviewers’ comments by, for example, providing training on written communication. Additionally, HHS stated that it will take steps to assess examples in which a brand-name drug company labeling change impacted the timeline of a generic drug approval and assess what actions could address this issue. In addition, HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the congressional addresses, the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at 202-512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. The Food and Drug Administration (FDA) made several changes to its review process for generic drug applications in its implementation of the Generic Drug User Fee Amendments of 2017 (GDUFA II) that are intended to improve communications with applicants, such as drug companies, and included the following: Additional communication with applicants. In its GDUFA II Commitment Letter to Congress, 1. FDA committed to notify applicants of potential deficiencies in an application that could prevent approval in the first review cycle through information requests or discipline review letters by about the mid-point of the review cycle; and 2. FDA committed to continue to issue additional information requests or discipline review letters late in the review cycle as needed, and, in certain circumstances, to work beyond the review timeframe to issue an approval. Pre-submission facility correspondence. In its GDUFA II Commitment Letter to Congress, FDA committed to communicating with applicants of priority generic drug applications before the application is submitted. For priority generic drug applications, FDA accepts information about facilities associated with an application, such as manufacturing facilities, at least 2 months before the application is submitted. If FDA finds the pre-submission facility correspondence includes complete and accurate information, the application may receive a review timeline of 8 months, rather than 10. Figure 2 provides an overview of the timeline for the first review cycle for generic drug applications under the Generic Drug User Fee Amendments of 2012 (GDUFA) and the revised review process under GDUFA II. In addition to the contact named above, Gerardine Brennan (Assistant Director), Rebecca Rust Williamson (Analyst-in-Charge), Caroline Hale, and Elizabeth Leibinger made key contributions to this report. Also contributing were Kaitlin Farquharson, Cathy Hamann, Dan Lee, Laurie Pachter, and Vikki Porter.", "summary": "Generic drugs—copies of brand-name drugs—lead to significant cost savings. Before a generic drug can be marketed, FDA must approve the generic drug application. According to FDA, applications go through an average of three cycles of review before being approved, which may take years. The FDA Reauthorization Act of 2017 included a provision for GAO to study issues regarding the approval of generic drug applications in the first review cycle. This report examines 1) the first review cycle approval rate of generic drug applications in recent years and factors that may have contributed to whether applications were approved; and 2) changes FDA has made to increase the first review cycle approval rate. GAO reviewed FDA data on all generic drug applications reviewed from fiscal years 2015 through 2017 and documentation from the first review cycle for a judgmental selection of 35 applications from fiscal years 2017 and 2018. GAO also interviewed a non-generalizable selection of stakeholders. Applications and stakeholders were chosen to ensure variation in experience with the approval process. GAO found that 12 percent of the 2,030 generic drug applications reviewed by the Food and Drug Administration (FDA) from fiscal years 2015 through 2017 were approved in the first review cycle. The first review cycle begins when FDA accepts a generic drug application for review and ends when FDA makes its first decision about whether the drug should be approved for marketing and sale. For applications that were not approved in that first cycle, the application must undergo one or more subsequent review cycles to obtain approval, delaying the generic drug's arrival to market. GAO identified several factors that may have contributed to whether a generic drug was approved during the first review cycle. For example, certain types of complex drugs were less likely to receive approval in the first review cycle, such as eye drops or other drugs administered through the eye. FDA has taken steps to increase the rate of generic drug approvals in the first review cycle. For example, FDA has increased communication with applicants and introduced templates for reviewers to improve the consistency and clarity of their comments. However, GAO's review of a judgmental selection of 35 applications found examples of variation in the clarity and content of FDA's comments to applicants. Such variation may have contributed to whether applicants could adequately address deficiencies within the first cycle, and therefore whether the applications were approved. In addition, stakeholders GAO interviewed expressed concern that changes to the brand-name drug's labeling mid-cycle could delay or prevent generic drugs' approval in the first review cycle, and some stakeholders said they believe that the labeling changes may be strategically timed to delay approvals. Although FDA officials noted that it would be difficult for brand-name companies to time labeling changes in this way, they said that the agency has not conducted analysis that would enable it to assess the validity of these concerns. Therefore, FDA lacks the information needed to respond to these concerns or address problems should they exist. GAO recommends that FDA 1) take additional steps to address inconsistency in its written comments to generic drug application sponsors and 2) assess the extent to which the timing of brand-name drug companies' drug labeling changes affects the approval of generic drugs and take steps, as appropriate, to limit the effect. HHS concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "The Controlled Substances Act (CSA) was enacted in 1970 to regulate and facilitate the use of controlled substances, including certain prescription drugs such as opioid pain relievers, for legitimate medical, scientific, research, and industrial purposes while preventing them from being diverted for illegal uses. According to DEA, the CSA requires DEA to maintain a “closed system” of distribution, which includes limiting the amount of certain controlled substances that are available in the marketplace by setting quotas. Various CSA provisions also require persons who handle controlled substances to register with the DEA. This includes businesses that import, export, manufacture, or distribute controlled substances; certain health care practitioners, such as physicians, licensed to dispense, administer, or prescribe them; and pharmacies authorized to fill prescriptions, referred to as “registrants.” The registration mechanism creates a “closed system” of distribution in which distribution may lawfully occur among the registrants. The closed system of distribution, along with registrant compliance with the CSA’s regulatory requirements, helps to ensure that a particular controlled substance is always accounted for by a DEA-registered entity, from its creation until it is dispensed to a patient or is destroyed. The CSA places controlled substances in one of five schedules based generally on findings related to the substance, including whether the substance has a currently accepted medical use in treatment in the United States, its relative potential for abuse, and the degree of dependence the drug or other substance may cause. For further information on this and other legal requirements, please see appendix II. The prescription drug supply chain is the means through which prescription drugs are ultimately delivered to patients with legitimate medical needs. Although there can be many variations in the flow of prescription drugs through the supply chain, in a common example, prescription drugs are produced by manufacturers; are purchased and stored by distributors, who take orders and deliver them to customers such as pharmacies; and ultimately are dispensed by pharmacies to patients who have a prescription from a practitioner, as shown in figure 1. Although prescription drugs are intended for legitimate medical uses, the prescription drug supply chain may present opportunities for the drugs to be diverted and abused as the drugs move through the various components of the supply chain. For example, an individual may visit multiple practitioners posing as a legitimate patient, referred to as a doctor shopper, to obtain prescriptions for drugs for themselves or others, or criminal enterprises may rob distributors and pharmacies of prescription drugs to sell to others. DEA, through its Diversion Control Division, is responsible for preventing, detecting and investigating the diversion of controlled substances from legitimate sources while ensuring an adequate and uninterrupted supply is available for legitimate medical, commercial, and scientific needs. The division is responsible for enforcing the CSA and its regulations pertaining to pharmaceutical controlled substances and listed chemicals. In doing so, it conducts domestic investigations, among other things, in DEA’s 23 field division offices. By law, generally, manufacturers, distributors, and reverse distributors are required to report to DEA every sale, delivery, or other disposal of any controlled substance. As we previously reported, manufacturers and distributors of schedules I and II drugs and schedule III narcotics must file reports with DEA through ARCOS, a drug reporting system that allows the agency to monitor the flow of DEA controlled substances from their point of manufacture through commercial distribution channels to point of sale or distribution at the dispensing/retail level. In addition, certain schedule III non-narcotics and some schedule IV narcotics are also covered by the ARCOS reporting requirements. DEA implemented the ARCOS database in 1997, and approximately 1,250 distributors, manufacturers, and reverse distributors report more than 72 million transactions into ARCOS each year, according to DEA. Generally, certain registrants must report certain data at least quarterly and they have the option to report voluntarily on a monthly basis. By law, each registrant, such as manufacturers and distributors of controlled substances, is required (1) to design and operate a system that is compliant with applicable federal and state privacy laws to identify suspicious orders of controlled substances, and (2) upon discovering a suspicious order or series of orders, notify the DEA Administrator and the special agent in charge of the appropriate DEA field division office. The SUPPORT Act, which amended the CSA in part, also includes requirements related to preventing drug diversion. The SUPPORT Act provisions require the DEA Administrator to establish a centralized database for collecting suspicious orders reports, which is discussed in more detail below. In addition, the SUPPORT Act requires the Attorney General to make certain data available to registered manufacturers and distributors through ARCOS. The SUPPORT Act also requires the Attorney General to submit to Congress a report that provides information about how the Attorney General is using ARCOS data to identify and stop suspicious activity no later than one year after the date of enactment of the SUPPORT Act. DEA operates and maintains various information systems containing registrant information, transaction data, and suspicious drug orders that support its efforts to prevent, detect, and investigate the diversion of pharmaceutical controlled substances. These include Controlled Substance Ordering System (CSOS). This system is used primarily by manufacturers and distributors, as well as pharmacies and hospitals to place orders for controlled substances in a secure electronic environment, and includes information such as the number of packages, size of packages and name of items ordered, according to DEA. ARCOS. As discussed above, ARCOS monitors the flow of transactions of schedule I, II, III and select schedule IV controlled substances from their point of manufacture to their point of sale or distribution at the dispensing or retail level (such as hospitals, retail pharmacies, practitioners, and teaching institutions). The data in ARCOS are used to, among other things, track regulatory compliance in the pharmaceutical drug industry and to detect abuse of legally manufactured pharmaceuticals that are diverted to illegal markets, according to DEA Diversion Control Division officials. Suspicious Order Reporting System (SORS). DEA developed SORS to receive and store suspicious order reports. To date, DEA has developed three versions of SORS as described below. SORS Online version. In late October 2019, DEA launched the Suspicious Orders Report System (SORS) Online, a centralized database required by the SUPPORT Act, for registrants that distribute controlled substances to report suspicious orders to DEA. Reporting a suspicious order to SORS Online constitutes compliance with the reporting requirement that registrants notify the Administrator of the Drug Enforcement Administration and the Special Agent in Charge of the Division Office of the Drug Enforcement Administration for the area in which the registrant is located or conducts business. SORS online is the third version of DEA’s SORS system that was originally developed in 2008. Unlike earlier SORS versions, SORS Online requires users to provide a reason an order is suspicious. At the time of our study, the use of SORS Online was voluntary. Registrants who are under active MOAs with DEA are reporting to the new SORS Online system, according to DEA. Follow-up version. Suspicious order reports reported by registrants since March 2017 operating under an active memorandum of agreement (MOA) with the DEA that required them to submit their reports electronically to DEA headquarters as opposed to their local DEA field division office using the SORS Follow-up version, according to DEA Diversion Control Division officials. Initial version. The initial version of SORS stores suspicious order reports for registrants with an expired MOA but who elected to voluntarily continue to report suspicious orders in the same way as under the MOA, according to DEA Diversion Control Division officials. The initial version of SORS was established in 2008. Figure 2 provides an overview of the information DEA obtains and uses to support its diversion control efforts. A PDMP is an electronic database that tracks controlled substance prescriptions, managed within and at the state level. State PDMPs can provide health care providers and authorities timely information about prescribing and patient behaviors that may indicate drug abuse or diversion and facilitate a response. Authorized users, such as practitioners and pharmacists, may access information submitted to PDMPs by dispensers. A state’s PDMP is housed by a specified statewide regulatory, administrative or law enforcement agency. The PDMP distributes data from the database to individuals who are authorized under state law to receive the data for purposes of their profession. PDMP data can assist law enforcement and health care providers such as practitioners and pharmacists in identifying patterns of prescribing, dispensing, or receiving controlled substances that may indicate abuse or diversion. PDMPs vary in numerous ways across states, including what data they collect; what drugs they cover; who has access to, or who is required to use, the prescription drug monitoring program; and which state agency oversees and administers the program. DEA may request state PDMP data through submitting requests or subpoenas to the state official operating the PDMP database, for example, to support diversion control investigations. The requirements on requesting and accessing state PDMP vary from state to state according to DEA Diversion Control Division officials. Officials noted that the different state-by-state requirements create difficulties for federal law enforcement during a multi- state or national case as law enforcements’ requests for data have to be addressed at the state level. Data-analytics activities can include a variety of techniques to prevent and detect diversion, including data matching and data mining. Data matching is the large scale comparison of records and files to detect errors or incorrect information. It can be used to verify information provided by recipients or detect unreported changes. Data mining is the use of automated computer algorithms to detect patterns, including those that are otherwise not obvious, correlations, or anomalies within large data sets indicative of potential diversion. Entities may identify many types of analytics techniques that can be used to address improper transactions, such as Rules based – Identify suspicious orders with rules, such as orders that go above a threshold; Anomaly – Detect individual and aggregated abnormal patterns versus peer group, for example, the orders from one pharmacy compared to other pharmacies in the same geographic area; and Predictive – Assess against known diversion. A provider that has characteristics similar to those of known bad actors. DEA uses industry-reported ARCOS data to help generate leads, support enforcement actions, and allocate resources. The agency uses these data in a number of ways, including supporting field diversion control activities and developing analytical products. Field-Based requests for data analysis. DEA’s Diversion Control Division’s ARCOS Unit responds to requests for data analysis from its field division offices in support of diversion control enforcement activities. According to DEA officials, this unit is responsible for the collection, maintenance, and analysis of ARCOS data. For example, DEA said this unit conducts analysis on controlled substances that are bought and sold in a particular timeframe between a seller and a buyer. The ARCOS Unit also obtains information on the quantity, dosage units, grams, and ingredients of the drugs in the sale and conducts analyses in response to specific requests from field-based investigators who send their requests to the unit. For example, DEA officials said that out of the 800 field division office requests for analysis sent to the DEA ARCOS Unit in calendar year 2018, about 60 percent of those were for “enhanced” validations. This process includes a controlled substance report which the unit provides to field investigators for their use during scheduled drug investigations, and contains a summary of, among other things, an ARCOS registrant’s reported sales and purchases compared against what other registrants report was sold to them. This process uses both automatic and manual checks. According to DEA officials, they received approximately 480 requests for enhanced validations from DEA field investigators in 2018. While DEA officials noted that DEA’s enhanced validation procedures are not documented, they acknowledged that the ARCOS Unit is in the process of developing standard operating procedures for ARCOS data quality control, including the enhanced validation process. All requests for validations submitted to the ARCOS Unit are analyzed and compiled, and sent to field-based investigators to support scheduled investigations. Although validations are primarily requested for scheduled investigations, field offices can request these reports pursuant to any scheduled or non-scheduled investigation. DEA Analytic Product - Drug Profiles. Using ARCOS data, DEA creates drug profiles for suspected bad actors at the retail level (such as certain pharmacies), who have irregular transactions–also known as outliers, according to DEA officials–in a specific area or zip code and provides this information to its field division offices. The ARCOS Unit compares this suspected “bad actor” with other area competitors. DEA Analytic Product – Annual Threat Assessments. DEA’s ARCOS Unit also uses ARCOS data to develop threat assessments annually to aid field investigators. The threat assessments use ARCOS data to provide drug-related transaction trends and patterns related to a given DEA field division office area of operations to help establish priorities and allocate resources. DEA officials noted that field division office staff use these assessments to develop work plans identifying which registrants will be subject to the office’s routine regulatory investigation that year. Field Querying of ARCOS Data. – Field division offices may also use ARCOS querying tools to analyze ARCOS data to proactively identify diversion targets, such as reviewing ARCOS data to identify information on top purchasers of controlled substances. In a written response to our questions, DEA officials told us that several ARCOS drug profiles they developed have contributed to state and federal administrative, criminal and regulatory investigations. DEA officials recently informed us that as part of a reorganization, it established the Targeting and Special Projects Section whose goal is to focus on leveraging DEA’s data capabilities. Specifically, this section is composed of two units, including the Reports Analytics Unit and the Targeting and Special Projects Unit, which was established in March 2019 and is responsible for conducting data analytics on ARCOS and other data, according to DEA Diversion Control Division officials. DEA is currently working to determine the types of analysis these units will conduct. We found that while DEA uses ARCOS data to support ongoing investigations and conducts analysis on this data to identify investigative leads for its field division offices, it could conduct more robust analysis using automated computer algorithms to help identify questionable patterns in the data. This analysis in turn could be used to identify registrants that need to be investigated. According to DEA officials, most of the analysis DEA currently conducts on ARCOS data is used by the field division offices. For example, upon receiving information on pharmacies that have a high frequency of reporting stolen or lost-in-transit drugs a field division office may contact DEA’s ARCOS Unit to request ARCOS information. DEA then analyzes the ARCOS data to produce the requested reports to support the field’s ongoing investigations. DEA also conducts routine analysis of ARCOS data to identify high volumes of drugs sold by a distributor to a single purchaser, high volumes of drugs purchased by a single purchaser, and trends in drugs sold or purchased in a given geographic area compared to similar nearby areas. DEA officials also identified one type of analysis it conducts using a computer algorithm. Specifically, DEA uses a computer algorithm when comparing large volumes of drugs purchased in a given geographic area to the area’s population data. According to DEA officials, DEA conducts this analysis quarterly. However, DEA did not report conducting active and recurring monitoring of transactions using algorithms to detect and flag transactions that indicate potential diversion, either on a real-time or near real-time basis. We identified several additional opportunities for DEA to proactively analyze ARCOS data using computer algorithms to identify unusual patterns of drug distribution on a more routine basis. Such analyses could be used to proactively support or generate leads for investigations of potential drug diversion. For example, DEA could Analyze ARCOS data to identify unusual volumes of drugs that were disposed of rather than sold. Conduct analysis of ARCOS data to identify unusual numbers of deleted transactions or deletions of transactions of high volumes of drugs. Analyze ARCOS data by comparing the amount of drugs being acquired by a registrant to the amount of drugs accounted for, through being sold or disposed of, among other things, by each registrant to determine any differences. Analyze ARCOS data to identify trends in distribution or purchases of drugs in a given geographic area. DEA could look for unusual patterns when comparing such activity in an area with that of other nearby areas; or analyze volumes of drugs purchased in a geographic area when adjusted by the area population. In addition to the analysis noted above using ARCOS data, we also identified further analysis that DEA could perform using ARCOS data and additional available data to help identify potentially suspicious purchase or distribution patterns. Specifically, in our review and analysis of ARCOS data and information about PDMP data, we identified an opportunity for DEA to analyze ARCOS and PDMP data together for a more complete picture of drug transactions from distribution to retailers through dispensation to patients. We determined this could help in assessing whether the amount of drugs being prescribed is consistent with the amount of drugs being purchased or distributed in a given geographic area. For example, in areas where the number of prescriptions increases, a subsequent increase in drug orders and distribution to that area could be considered understandable. However, where the number of prescriptions in an area remains the same, or decreases, a significant increase in drug orders and distribution to that area could be considered unusual, especially if this pattern persists over several reporting periods. DEA stated that it occasionally performs such analysis manually, noting however that its access to PDMP data is contingent upon each state’s requirements and willingness to share its PDMP data with federal law enforcement. In July 2019, DEA officials responsible for overseeing the use and analysis of ARCOS data expressed an interest in improving DEA’s ARCOS data analytic capabilities but stated that they needed more staff and resources. Specifically, they noted they would like to hire additional staff, such as data scientists, to conduct analysis on ARCOS data using, for example, additional computer algorithms. DEA also noted that it was considering automation of additional types of analyses, but did not provide a start date or estimate as to when it would move forward on that consideration. While DEA created the new Targeting and Special Projects Section in March 2019 to enhance DEA’s data analytics and set aside some positions for program analysts and subject matter experts, among other positions, as of October 2019, DEA officials did not have any details or documentation about the data analysis efforts the new division plans to undertake. We have previously reported that new approaches to combining and “making sense of” large amounts of varied data—methods referred to as advanced analytics—are helpful to uncover patterns, identify anomalies, and provide insights not suggested by assumed hypotheses. In addition, other federal entities responsible for detecting diversion and abuse of controlled substances utilize computer algorithms as part of their analysis of available data in order to flag and prioritize potential instances of diversion for further investigation. For example, the Centers for Medicare & Medicaid Services and its National Benefit Integrity Medicare Drug Integrity contractor use proactive data analysis to detect aberrant patterns and potential diversion in drug prescribing. As a result, the contractor is able to produce “prescriber risk assessments,” which provide a comparison of controlled substance prescribing patterns across peers. The Centers for Medicare & Medicaid Services also uses proactive data analysis to identify providers with potentially inappropriate prescribing patterns, especially as it concerns opioids. Similarly, some opioid drug distributors use computer algorithms to identify suspicious orders that are the basis for the suspicious order reports they are required to provide to DEA. The establishment of this new section within DEA focused on its data analytics capabilities presents an opportunity for DEA to more proactively use data analytics with regard to its ARCOS and other data. In doing so, DEA could more effectively identify possible diversion activities or unusual activity to aid its ongoing efforts to prevent, detect, and investigate diversion more quickly and assist it in reporting on how it is using ARCOS data to identify suspicious activities. In October 2019, DEA established the Suspicious Orders Report System (SORS) Online, a centralized database for collecting suspicious order reports, which is required by the SUPPORT Act to be established by October 24, 2019. The SORS Online data fields include a requirement for registrants to note their reasons for identifying an order as suspicious, drug quantity, and dosage strength. The successful implementation of the centralized database is important because it could address the fragmented way in which suspicious order reports are currently submitted. However, reporting to the centralized database is currently voluntary. Registrants may notify DEA of a suspicious order using other means, including email, facsimile, or telephone. The systems and reports are not currently integrated, and investigators must query each system or office separately in order to find, for example, information related to a lead they are investigating. Currently, registrants are required upon discovery of a suspicious order or series of orders, to notify the Administrator of the DEA and the Special Agent in Charge of the division office of the DEA for the area in which the registrant is located or conducts business. Prior to DEA establishing the SORS Online centralized database, registrants with an existing or a prior MOA also have reported suspicious orders into one of two SORS databases when reporting to headquarters. The new SORS Online is the only electronic mechanism for reporting suspicious orders now, according to DEA. Registrants who are under active MOAs with DEA are reporting to the new SORS Online system, according to DEA. Registrants that are not under an MOA may also use SORS Online, but are not required to do so. Registrants not under an MOA may also use a paper-based process, among others, when reporting to the field division offices and DEA headquarters. However, no integration exists across headquarters’ and field division offices’ various electronic- and paper-based systems. DEA officials we met with said that some of the suspicious order reports received at the field division office level are stored in hard copy in accordion file folders, instead of being digitized or entered into a searchable database. Reporting to SORS Online satisfies the requirement to report such orders to the Administrator of the DEA and the Special Agent in Charge of the Division Office of the DEA for the area in which the registrant is located or conducts business. Successfully managing the SORS Online database could lead to needed efficiency improvements and more effective use of the suspicious order report data. Although DEA has guidance, policies and procedures regarding the use of some of its information systems, it has not established a formalized data governance structure to manage its collection and use of data used to support the Diversion Control Division’s mission. DEA specifically has not institutionalized and clearly documented policies and procedures that describe division staff’s roles and responsibilities for collecting and analyzing data nor has it provided a structure that describes the agency’s approach to establishing and maintaining such a program. We have identified a number of issues with DEA’s management of data. For example, DEA does not have any documentation on their process for ensuring the quality of data registrants submit to its ARCOS database— the main system that enables DEA to monitor the flow of controlled substances. As a result, it is difficult to understand the controls they have over this important data. A data governance structure is defined as an institutionalized set of policies and procedures for providing data governance throughout the life cycle of developing and implementing data standards. A data governance structure also helps to ensure important data assets are formally managed and fully utilized, and can also provide consistent data management. We previously reported on key practices based on several data governance models, including developing and approving data standards, managing, controlling, monitoring, and enforcing consistent application of data standards, and delineating roles and responsibilities for decision making and accountability. Additionally, in June 2019, the Office of Management and Budget established a Federal Data Strategy (Strategy) as a framework of operational principles and practices to help agencies use and manage data. We found several areas where DEA’s current practices do not reflect select leading data governance practices. Agencies should identify data needs to answer key agency questions: We found that DEA does not have a governance structure to determine and prioritize its data requirements for either suspicious order reports it receives or data reported into its ARCOS systems. For example, DEA has not established standard requirements for the information required in a suspicious order report. As a result, distributors’ suspicious order reports vary and may contain inconsistent and insufficient data for DEA to make investigative decisions. In addition, DEA does not have a governance structure to identify agency and industry stakeholder data needs to help inform its opioid diversion control efforts. Agencies should provide resources explicitly to leverage data assets: Agencies should ensure that sufficient human and financial resources are available to support data driven agency decision- making, and accountability. As mentioned earlier, while DEA created the new Targeting and Special Projects Section in March 2019 to enhance DEA’s data analytics, as of October 2019, DEA officials did not have details or documentation about the data analysis efforts the new division plans to undertake or the resources they plan to provide for those efforts. As a result, DEA is unable to conduct the analysis that would enable it to more effectively use its existing data in making decisions about diversion related efforts. Agencies should prioritize data governance: Agencies should ensure there are sufficient authorities, roles, organizational structures, policies, and resources in place to transparently support the management, maintenance, and use of strategic data assets. Similarly, leading practices for data governance includes delineating roles and responsibilities for decision-making and accountability, including roles and responsibilities for stakeholder input on key decisions. As mentioned earlier, DEA established the Targeting and Special Projects Section in March 2019 whose goal is to focus on leveraging DEA’s data capabilities and conducting data analytics on ARCOS and other data, according to Diversion Control Division officials. While the new section appears to hold promise, DEA has not clearly defined and adopted the new section’s roles and responsibilities for managing and analyzing data across the DEA or how the new section will communicate and collaborate with other Diversion Control Division headquarters and field staff. As a result, the new division may not operate in a predictable, repeatable, and accountable way. Agencies should support non-federal stakeholders: Agencies should engage with industry, academic, and other nonfederal users of data to share expert knowledge of data assets, promote wider use, improve usability and quality, and advance innovation and commercialization. Later in this report, we identify an opportunity for DEA to collaborate with industry stakeholders and seek their input for an initiative that is supposed to assist industry stakeholders in their responsibilities to report suspicious orders to DEA. Although DEA has not incorporated these data governance practices, it is in the early stages of developing a data governance structure. As of September 2019, DEA officials told us that its Office of Information Systems’ Chief Data Officer just recently started to work with DOJ and other components to develop a data strategy in response to the recently released department wide strategy, and therefore does not have any additional documentation or information related to timelines and deliverables for formally implementing a DEA data governance or other data structure for the agency. Without additional details, such as a timeframe for developing the structure or more information about what it would entail, it is unclear how or if these efforts will incorporate leading practices for data governance and if they will be effective. Data governance processes are important for DEA given it works with an extensive and complex network of stakeholders to manage opioid diversion risks and uses industry-reported data to help it identify patterns that might indicate potential diversion. An effective data governance structure could help DEA ensure its important data assets are formally managed and fully utilized, and can also help ensure consistent data management. Industry and technology councils, domestic and international standards-setting organizations, and entities within the federal government endorse the establishment and use of a governance structure to oversee the development, management and implementation of data standards, digital content and other data assets. DEA does not have an existing mechanism or a comprehensive database of orders before they are filled that it can analyze, on a real-time basis, to identify potentially suspicious orders. However, most industry stakeholders we spoke with on the usefulness of real-time data noted that such a mechanism would not add extensive value to diversion detection. DEA’s current data systems either contain historical, not real-time, data or do not contain all drug order data that could be reported. ARCOS. The data in the ARCOS database is historical, rather than real- time, on orders that have been filled. Every registered manufacturer is required, at such time or times and in such form as required by the Attorney General, to make periodic reports to the Attorney General of every sale, delivery or other disposal of any controlled substance. Each distributor is required to make such reports with respect to narcotic controlled substances. For example, as part of the reporting to ARCOS, acquisition and distribution transaction reports are required, by regulation, to be filed every quarter, except that a registrant may be given permission to file more frequently, but not more frequently than monthly, depending on the number of transactions being reported each time by that registrant. In addition, manufacturing transaction reports are required to be filed annually, except that a registrant may be given permission to file more frequently, but not more frequently than quarterly. CSOS. DEA does not require registrants to use CSOS and thus it is not used by all registrants. As previously discussed, CSOS is an electronic ordering system which allows registrants to place orders for controlled substances. Shipments of all ARCOS-reportable controlled substances, ordered through CSOS, are included in registrant’s periodic ARCOS reporting. Suspicious Order Reports. Suspicious order reports are intended to identify orders of unusual size, orders deviating substantially from a normal pattern, and orders of unusual frequency before they are filled. By law, each registrant is required to design and operate a system to identify suspicious orders that it receives. Registrants identify, then report, to DEA using their own systems to determine suspicious orders. As discussed previously in this report, registrants can report these orders into DEA’s newly launched SORS Online database, but reporting into this database is voluntary and registrants have an option to report in other ways, so this database does not capture all suspicious orders and is therefore not comprehensive. Suspicious orders are likely identified in close to a “real-time” basis. Orders that have been identified and reported as suspicious by the registrants, are orders that have not yet been filled. While two individual drug distribution companies we interviewed said they saw some value in real-time reporting, most industry stakeholders we spoke with on the usefulness of real-time data, including a broad cross- section of associations representing pharmacies and drug distribution companies, said that real time dissemination of suspicious orders by DEA would not add extensive value to efforts to detect possible diversion. Instead, some industry stakeholders suggested that a focus on data that provide trends over time might be more useful. As discussed earlier in this report, we provide examples of data analysis DEA currently performs and could perform on its existing data that could potentially help DEA determine or identify possible patterns of aberrant behavior in drug order information. Others we spoke with raised concerns about the varying ways companies determine what is suspicious and that using real-time data reported from DEA on these orders could be like comparing apples to oranges. Most of the associations that represent pharmacies and drug distributors that we met with indicated they did not see much value in either reporting, or receiving reports, of suspicious orders in “real-time.” For example, a representative from an association representing pharmacists told us that rarely would there be a case where a single order was so egregious that stopping it would have a significant impact on public health. This representative also noted that it would be more important to focus on historical trends, given “trends don’t happen in a day.” Other stakeholders we spoke with said that while there may be utility in real-time reporting of suspicious orders, they also had concerns about its feasibility, given current available data. They noted it would be difficult to compare suspicious order data as reported by registrants because companies rely on their own methods to determine a suspicious order. For example, Officials from an association that represents a large number of drug distributors indicated that receiving more real-time data might allow their members to have an additional check on orders that a wholesale distributor receives, but this utility would largely be contingent on the distributors’ ability to compare suspicious order reports across one another. Distributors use different criteria for determining whether an order is suspicious; there is no continuity across them; and they experience varying order volumes and patterns across their customers and over time as patient needs change. Thus, such analyses would be difficult to conduct, if they could be done at all, and would not necessarily result in useful comparisons. A representative from one drug distribution company told us that having knowledge of other distribution companies’ suspicious orders is not helpful because the company would not know how the other distributor made a determination on the suspicious order. Another representative stated that distributors are operating proprietary systems that may or may not vary substantially from each other depending on a large number of varying circumstances, and may be operating “wildly different” systems for identifying suspicious orders and therefore the information would not be valuable. Representatives from two drug distribution companies identified additional challenges to real-time reporting of suspicious orders if the determination of whether an order was suspicious or not was made by DEA. First, they did not believe DEA had sufficient resources or knowledge to identify suspicious orders. One representative said DEA does not know the history and market dynamics in the pharmaceutical industry to help inform decisions it would need to make on an order. Second, identifying an order as suspicious before it is filled would add a tremendous burden on DEA. According to one of the representatives, their company typically ships orders on the same day the order is received, consistent with “just-in-time” inventory management practices. If DEA were expected to make suspicious order determinations without the risk of disrupting patient care needs, it would be imperative for DEA to act quickly to identify suspicious orders. These distribution companies did not believe DEA would be able to identify them rapidly as needed. As noted above, DEA’s current systems are not designed for real-time reporting, and it does not have an existing mechanism or a comprehensive and complete database of orders before they are filled that it can analyze, on a real- time basis, to identify potentially suspicious orders. Officials from the association that represents a large number of drug distributors were careful to point out, however, that systems sometimes differ intentionally due, for example, to varying customer bases, service requirements and patient care needs. Thus, a certain amount of variability in suspicious order systems, criteria and decisions may be warranted, and even desirable. While DEA has developed some performance measures to track and publicly report the progress and results of its efforts in reducing diversion, DEA has not developed objectives, outcome-oriented goals, or measurable performance targets to assess the effectiveness of its opioid diversion control data analysis efforts and the link between DEA’s use of data and progress toward its diversion goals and strategies. DEA does have performance measures including the number of civil penalties and administrative actions it has undertaken, planned or scheduled investigations completed, and community outreach events completed. While these measures are useful, they do not account for outcomes of these actions, such as their potential impact on the volume of opioids being improperly sold or purchased. DEA officials noted that it adheres to goals established through the Office of National Drug Control Policy’s National Drug Control Strategy, such as reducing the prescription opioid rate by one-third within three years, reducing overdose deaths, and within five years, ensuring all health care providers have adopted best practices for opioid prescribing. However, those goals involve a multitude of federal agencies, and are not directly related to DEA’s use of industry-reported data, nor linked specifically to DEA diversion control efforts. DEA also noted that they have a number of goals across strategies such as DEA’s 360 strategy in addition to the goals in DOJ’s strategic plan; a performance measure with a measurable target for its agency-wide objective related to dismantling drug trafficking organizations—maximizing the monetary value of currency, property, and drugs seized; and a measure for curbing opioid and other illicit drug use. GPRAMA directs agencies to develop and document goals, as well as performance measures to assess progress towards their goals. Agencies can use performance measurement to make various types of management decisions to improve programs and results, such as developing strategies and allocating resources, including identifying problems and taking corrective action when appropriate. Additionally, GPRA as amended by GPRAMA states that management should define outcome-oriented objectives in specific and measurable terms. Measurable targets help decision makers conduct assessments of whether program goals were achieved, and linkages between an organization’s goals and performance measures create a line of sight so that everyone understands how program activities contribute to the organization’s goals. DEA officials view their existing performance goals as sufficient overall. However, without defining objectives in specific measurable terms, DEA is likely not able to adequately assess whether its respective investments and efforts are helping it to limit the availability of and better respond to the opioid prescription diversion threat. Until program officials can review the effectiveness of these systems based on quantifiable benefits and measurable performance targets, they are not well-positioned to determine the extent to which suspicious order reports or ARCOS data and systems are enhancing the effectiveness of the agency’s opioid related regulatory and criminal diversion investigations, prosecutions and civil actions. Documenting program goals and developing measurable performance targets and linkage to program goals could provide DEA with the information it needs to assess progress and make informed decisions about current and future operations. DEA developed an ARCOS query option for registrants to use, called the ARCOS Enhanced Lookup Buyer Statistic Tool, in February 2019 to better support registrants’ efforts to identify and report suspicious orders. This tool allows registrants to query certain ARCOS data maintained by DEA. Although this tool was supposed to be an improvement upon a prior iteration of the lookup tool DEA had developed, distributors and an industry association representing distributors identified several limitations with the tool. Specifically: Single query challenges and no bulk downloads. The distributor can only query the tool one pharmacy at a time, even though some distributors supply thousands of pharmacies on a daily basis. Thus, if certain distributors were to query all of its pharmacies for possible suspicious order patterns, the process could be time-consuming or not feasible. DEA noted it was working on this limitation. Limited login credentials. DEA only provides each distributor with one set of login credentials, so only one employee can log in at a time to query the tool. DEA noted it was working on this limitation. Data provided in the tool are not detailed enough. The tool does not provide detailed enough information to be useful to facilitate the identification of suspicious orders. For example, when a distributor queries the tool, the search results will list the total dosage units for a particular opioid for the past six months at the pharmacy. Because some opioid drug dosages are more commonly abused than others, distributors told us that simply having the total number of dosage units is not as helpful as seeing the breakdown of the different dosage units. In another instance, the data provided to distributors does not include critical details about the number of suppliers. One distributor might have multiple warehouses and distribution centers that it uses to package and ship pharmaceutical products. In the ARCOS data that DEA provides to distributors, these individual warehouses are counted as distinct suppliers in the total supplier count data provided to the distributors. Therefore, the number of suppliers may appear inflated to the distributors, even though it is only a single company providing the products. According to DEA, the ARCOS lookup tool is meant to be a pointer and assist distributors in conducting due diligence so they can “know their customer.” Regardless if a distributor is shipping from multiple distribution centers and therefore showing as multiple suppliers in the lookup tool, these are all unique DEA registration numbers and are therefore unique suppliers to the customer. According to DEA, the important part here is that distributors can see quantity and gram totals per registrant (such as, a pharmacy customer) that they query. When evaluating whether an order is suspicious, a distributor uses its own internal transaction data to evaluate a buyer’s ordering patterns. However, purchasers of controlled substances, such as pharmacies and medical practices, may use multiple distributors for their purchases. Distributors have previously raised concerns that they did not have access to additional transaction data, such as whether the purchaser is also buying controlled substances from additional suppliers. They have noted that this additional data would be useful when making decisions about whether an order is suspicious, and specifically, that ARCOS data would be useful in helping them evaluate whether an order was suspicious. For example, in 2018, one distributor testified that, given DEA’s access to the controlled substance transaction data that distributors report, “nly DEA has visibility over the entire landscape and can track and analyze aggregate data on the distribution of controlled substances in particular jurisdictions.” In addition, an industry organization we met with provided comments to DOJ in 2017 that certain data could provide more context for them to identify problematic orders. Specifically, the organization noted that if DEA could provide ARCOS data in aggregate form without identifying individual distributors’ competitors, the distributor could consider a pharmacy’s orders in the context of the pharmacy’s overall ordering from all distributors. An industry association representing distributors, and two of the distributors that we interviewed stated that the Enhanced Lookup Buyer Statistic Tool is a step in the right direction. However, the industry association and four distributors that we interviewed stated that the tool remains limited in helping distributors improve how they identify suspicious orders, as noted above. DEA officials told us that distributors have brought some of these concerns about the ARCOS Enhanced Lookup Buyer Statistic Tool’s usability to their attention. For example, in May 2019, an industry association representing distributors sent a letter to DEA, outlining a consolidated list of industry’s concerns about the tool. In recent discussions in June 2019, DEA officials acknowledged some of these limitations and stated that some industry concerns would be easier to fix than others, but that they had not established a timeframe for when the changes would be implemented. For example, DEA officials noted it might be easier to provide additional login credentials to distributors and make the data available to be downloaded in a more functional way for distributors. For some of the other limitations industry stakeholders identified, such as providing more detailed ARCOS data to the distributors, DEA officials raised concerns. For example, DEA officials noted that distributors could use the additional detailed data as a market research tool in order for distributors to gain unfair market advantages or to learn more about their competitor’s business contracts with pharmacies. In September 2019, DEA officials told us that it was not currently addressing changes to the ARCOS Enhanced Lookup Buyer Statistic Tool, due to competing priorities within DEA. Specifically, DEA officials noted that it is focused on existing priorities related to meeting upcoming requirements mandated in the SUPPORT Act, including establishing a suspicious order centralized database, as discussed previously in this report. While we recognize that agencies need to determine and set priorities, it is important for DEA to continue to work with industry in ensuring that the tool it created to address the SUPPORT Act requirement will help industry in addressing its suspicious order reporting requirement under the CSA, as amended. The SUPPORT Act requires DEA to provide distributors with access to ARCOS data to help the distributors identify, report, and stop suspicious orders of opioids and reduce diversion rates. By identifying solutions – in consultation with industry stakeholders – to address the limitations of the ARCOS Enhanced Lookup Buyer Statistic Tool, DEA could better ensure registrants have more useful information at their disposal when evaluating whether an order is suspicious. DEA officials and DEA field division offices we interviewed identified a number of limitations with suspicious order reports they received, and, due to these limitations, they rarely use suspicious order reports to generate potential investigative leads. The issues DEA identified included: Threshold-based algorithm triggers. Several DEA headquarters and field division officials told us that some distributors used fixed thresholds to identify suspicious orders, which DEA officials stated are not helpful or useful because the information is often not actionable. Lack of documented rationale. During the course of our review, DEA officials told us that many suspicious order reports do not include the rationale for why the registrant decided the order was suspicious, making it difficult to determine which suspicious order reports might contain actionable intelligence. In September 2019, the DOJ Office of the Inspector General reported that the current regulatory language governing industry suspicious order reporting does not require manufacturers and distributors to state why they believe an order is suspicious. In October 2019, DEA launched a new centralized database of suspicious order reports, as required by the SUPPORT Act. DEA’s new reporting format of suspicious order reports includes a required field for “Reason,” for registrants to provide an explanation of why the order is suspicious. However, currently reporting to the new centralized database is voluntary. Differing methodologies. As discussed earlier in this report, the definition of a “suspicious order” may include, but is not limited to, an order of a controlled substance of unusual size, an order of a controlled substance deviating substantially from a normal pattern, and orders of controlled substances of unusual frequency. However, it is up to the individual distributor companies to decide the more specific metrics, according to DEA Diversion Control Division officials. Each distributor must design and operate a system to identify suspicious orders. Therefore, distributors utilize different methods to flag customer orders as suspicious. According to our analysis of DEA data, it has collected at least 1.5 million suspicious order reports since 2014, and these reports may contain data on attempted purchases that were denied, based on indicators of suspicious patterns. These data could help with DEA’s efforts to prevent, detect, and investigate diversion. Officials from six DEA field division offices we interviewed said they refer to suspicious order reports when conducting their routine regulatory investigations of registrants. DEA field division officials also stated that, while suspicious order reports are generally not used as the primary or sole impetus to initiate an investigation, officials will infrequently refer to related suspicious order reports when there is an ongoing criminal investigation that is initiated through other means. However, of the DEA field division offices we interviewed, officials from two offices told us that they had used a suspicious order report as the sole or primary impetus for initiating a criminal investigation in the past year – one stating that it happened once, and another estimating that it happened one to three times. Another field division told us that they had two convictions “in recent memory” that began with a suspicious order report. Three offices told us that they had not used suspicious order reports as the sole or primary impetus for a criminal investigation in the past year, and one told us they did not know if a suspicious order report had been used in that way. DEA field division offices we interviewed also identified reasons why suspicious order reports may not be as useful as they could be in helping to identify investigative leads. For example, one DEA field division office characterized the suspicious order reports they received from one particular registrant as being “spot on” and always warranting a DEA follow-up investigation, given the amount of detail and evidence of the registrant conducting its own on-site investigation into the customer. However, the same DEA office reported that other suspicious order reports were based on industry-developed thresholds that were not useful because the resultant reports did not indicate why the order was suspicious. Of the five DEA field division offices that we asked to characterize the quality of suspicious order reports, three of them reported that suspicious order reports were either “moderately” or “somewhat” useful. Officials from one field division office said that suspicious order reports are “very useful,” while officials from another DEA field division office reported that suspicious order reports are “not at all useful.” We have previously reported on these issues, including DEA communication with registrants, and in June 2015, we found that additional guidance from and additional communication with DEA was needed about registrants’ roles and responsibilities under the CSA, as amended. We recommended that DEA develop additional guidance for distributors for suspicious order monitoring and reporting. DEA did not expressly agree or disagree with our recommendation, but raised concerns about the recommendation, stating that “short of providing arbitrary thresholds to distributors, it cannot provide more specific suspicious orders guidance because the variables that indicate a suspicious order differ among distributors and their customers.” In responding to this recommendation, DEA officials told us that the agency had refocused its efforts on revising draft regulations in line with the SUPPORT Act, and that the revised draft was undergoing internal DEA and DOJ review. The agency noted that it expected the rule to codify existing legal obligations related to due diligence and suspicious order reporting and provide additional guidance regarding the nature and timing of the suspicious order reporting requirement, but also indicated that it was not possible to be certain of the precise nature of the draft rule. The 2015 recommendation remains relevant and important, and while DEA has reported taking some actions to address it, as noted above, DEA has not taken all the necessary steps to address the recommendation. We will continue to monitor DEA’s progress in addressing our recommendation. Given the extensive and complex network of stakeholders DEA works with to manage opioid diversion risks and the agency’s use of a large amount of industry-reported data, DEA could do more to use proactive, automated computer algorithms to analyze its data sources in detecting questionable patterns in industry-reported drug transaction data. It is missing opportunities to more effectively identify questionable ordering patterns and possible diversion activities than through its current analysis methods. Using more automated analyses, similar to other federal entities that use computer algorithms as part of their analysis of available data to help flag instances of diversion, DEA could enhance its ongoing efforts to prevent, detect, and investigate diversion more quickly and assist it in reporting on how it is using ARCOS data to identify suspicious activities. Furthermore, because DEA does not have a documented data governance structure in place to manage its data, it risks challenges related to quality, availability, and integrity of the data it uses to support opioid diversion. Although DEA has started to explore developing a data governance structure, it is important for DEA to document and define its process about what the structure would entail. This would help the agency determine the effectiveness of its structure, an important consideration given the large amounts of varied data DEA receives from industry stakeholders. Also, while DEA does have some performance goals related to opioid diversion, it lacks outcome-oriented goals and measurable performance targets to assess the extent to which the industry-reported data it obtains and uses support the agency’s diversion control activities. Defining these targets could help DEA adequately assess whether its respective investments and efforts are helping it to limit the availability of and better respond to the opioid prescription diversion threat. DEA’s efforts to provide registrants with additional information to facilitate the identification of suspicious orders is promising, but has limitations. Due to these limitations, registrants, such as distributors, might not have complete information when they are identifying suspicious orders. By identifying solutions – in consultation with industry stakeholders – to address the limitations of the ARCOS Enhanced Lookup Buyer Statistic Tool, such as the need for additional login credentials or the ability to bulk download data, DEA could better ensure registrants have more useful information at their disposal when evaluating whether an order is suspicious. Finally, we continue to monitor implementation of our 2015 recommendation that DEA provide additional guidance to distributors related to suspicious orders, and we believe that it remains relevant and important. We are making the following four recommendations: The DEA Administrator should develop and implement additional ways to use algorithms in analyzing ARCOS and other data to more proactively identify problematic drug transaction patterns. (Recommendation 1) The DEA Administrator, in coordination with the department-wide efforts on data strategy, should establish and document a data governance structure to ensure DEA is maximizing its management of industry-reported drug transaction data. (Recommendation 2) The DEA Administrator should establish outcome-oriented goals and associated measurable performance targets related to opioid diversion activities, using data it collects, to assess how the data it obtains and uses supports its diversion control activities. (Recommendation 3) The DEA Administrator, in consultation with industry stakeholders, should identify solutions to address the limitations of the ARCOS Enhanced Lookup Buyer Statistic Tool, to ensure registrants have the most useful information possible to assist them in identifying and reporting suspicious orders to DEA. (Recommendation 4) We provided a draft of this report to DOJ, including DEA, for review and comment. In its comments, reproduced in appendix III, DEA agreed with three of the four recommendations, and neither agreed or disagreed with the fourth. DEA also provided technical comments, which we incorporated as appropriate. In response to our first recommendation that DEA should develop and implement additional ways to use algorithms in analyzing ARCOS and other data to more proactively identify problematic drug transaction patterns, DEA concurred and stated it will continue to examine a variety of technologies to analyze ARCOS and other data and implement additional ways to use algorithms to more proactively identify problematic drug transaction patterns. If these and other actions to expand the agency’s analytic capabilities are effectively implemented, DEA would address the intent of our recommendation. DEA also concurred with our second recommendation that DEA, in coordination with the department-wide efforts on data strategy, should establish and document a data governance structure to ensure DEA is maximizing its management of industry-reported drug transaction data. In its response, DEA stated it is currently implementing this recommendation and will continue to mature its data governance structure. The intent of this recommendation is for DEA to establish a formalized data governance structure to manage its collection and use of data used to support the Diversion Control Division’s mission. By establishing such a structure, DEA could better ensure its important data assets are formally managed and fully utilized, and could also help ensure consistent data management across the Diversion Control Division. DEA neither agreed nor disagreed with our third recommendation that DEA should establish outcome-oriented goals and associated measurable performance targets related to opioid diversion activities, using data it collects, to assess how the data it obtains and uses supports its diversion control activities. In its response, DEA stated it recognizes that measurable performance targets related to opioid diversion activities can serve as leading practices at different organizational levels including the program, project, or activity level. However, DEA stated it needs additional clarification on the specific actions needed to fulfill this recommendation. Our recommendation is intended to ensure that DEA can demonstrate the usefulness of the data it collects and uses to support its opioid diversion control activities. We will continue to work with DEA to address the specific actions needed to assess how the data it obtains and uses support its diversion control activities to fully address the intent of this recommendation. Based on our review of DEA’s existing performance goals and targets for its opioid diversion efforts, as well as our previous work on performance measurement, we believe that further development of related performance goals and targets is warranted and could potentially improve the usefulness of the data DEA collects and uses in support of its diversion control program. DEA also stated in its comments that the limited timeframe did not allow GAO to meet with DEA officials responsible for performance metrics for opioid diversion. However, in our interviews with DEA regarding its performance metrics for opioid diversion, we submitted our questions in advance of meeting with DEA officials to allow time for the questions to be reviewed by relevant officials. DEA stated in its comments that it will ensure that GAO meets with the appropriate officials to address metrics. As stated earlier, we will continue to work with DEA to address the specific actions needed to meet the intent our recommendation. DEA concurred with our fourth recommendation that DEA, in consultation with industry stakeholders, should identify solutions to address the limitations of the ARCOS Enhanced Lookup Buyer Statistic Tool, to ensure registrants have the most useful information possible to assist them in identifying and reporting suspicious orders to DEA. DEA stated it has consulted with industry stakeholders and has identified solutions to address the limitations of the tool. We believe such consultation will be beneficial for DEA to understand its industry stakeholders’ needs and that identifying solutions for addressing these needs would help ensure registrants have the information necessary to help identify and report suspicious opioid orders. We are sending copies of this report to the appropriate congressional committees, the Attorney General, the DEA Administrator and the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6691 or McneilT@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. To understand the extent to which DEA obtains and uses industry- reported data, and the opportunities that exist to improve how that data are obtained and used, including the feasibility of real-time reporting, we reviewed applicable laws, regulations, court cases, and DEA internal documentation. We also conducted interviews with DEA headquarters offices, including the Diversion Control Division and DEA field division offices. To determine DEA registrant legal reporting requirements related to prescription drug orders and the meaning of suspicious orders, we reviewed applicable laws and regulations, including the CSA and its subsequent amendments and related DEA regulations and guidance. In addition, we reviewed the recently enacted SUPPORT Act. To identify policies and guidelines DEA uses to obtain and review registrant-reported data, we reviewed DEA procedures for conducting drug-related investigations, information system manuals for data and information systems used by DEA, and DEA written communications to registrants and DEA forms registrants use to report prescription drug transactions to DEA. As part of our work examining the information systems used to obtain and analyze data reported by registrants, we interviewed officials who oversee the management of DEA information systems, such as Automation of Reports and Consolidated Orders System (ARCOS), Controlled Substances Ordering System (CSOS), Registrant Information Consolidated System, and the Suspicious Orders Reporting System (SORS) systems used to obtain and store suspicious order reports at DEA headquarters. We interviewed DEA officials in headquarters and field division offices to determine how information that industry members report to DEA is obtained and used to detect and identify potential diversion activities. The perspectives we gathered from field division offices cannot be generalizable to the entire population of field division offices, but did provide us with insights into the agency’s diversion efforts and use of industry-reported data. To identify what opportunities exist, if any, for DEA to improve these efforts, such as using computer algorithms or real-time reporting, we also interviewed DEA officials responsible for developing analytical products based on industry-reported data. In addition, we interviewed DEA officials at eight field division offices to learn about how diversion investigators use industry-reported data and what, if any, improvements might be needed. To identify which of the 23 DEA field division offices to interview, we prioritized our selection based on four primary criteria: 1) the controlled prescription drug availability rate in their geographic area, according to a 2017 DEA threat assessment report, indicating whether the field division office had a “high” or “moderate” rate of availability; 2) whether the office was within the location of a DOJ Opioid Fraud and Detection Unit task force location; 3) whether the office was located within top ten state with high controlled prescription drug prescribing rate, as identified by the CDC; and 4) whether the office was located within a state that the CDC identified as having a high ER visit rate for opioid overdoses. We also ensured that the DEA field division offices we interviewed represented different geographic areas within the United States. We also conducted interviews with four pharmaceutical distributors and one trade organization whose membership includes wholesale distributors. We interviewed three organizations representing pharmacies, pharmacists, and drug diversion professionals to gather their perspectives and experiences with efforts to detect and report suspicious opioid orders. We based our initial interview selection of distributors based on DEA- provided ARCOS data of opioid-related transactions, which indicated the three largest distributors for opioids. To identify smaller distributors to gather their perspectives, we contacted an industry association representing distributors to facilitate our efforts to arrange for an interview, resulting in an interview with one additional distributor. In addition, we interviewed officials from a state prescription drug monitoring program (PDMP) that collects real-time data, a Bureau of Justice Assistance grant program that supports PDMPs, and a company that operates 44 of the 53 state PDMPs to gain insights on the data they collect. The views of these organizations cannot be generalized to the entire population, but provided important insights and perspectives about suspicious order detection and reporting. We reviewed the data DEA collects to identify possible types of analyses DEA could conduct using ARCOS data to identify unusual patterns. In addition, we reviewed key data governance practices used by organizations and identified through our past work to determine the extent to which DEA has a governance structure in place to manage how it collects and uses data to support diversion control efforts. Additionally, we reviewed the June 2019 Office of Management and Budget Federal Data Strategy which provides a framework of operational principles and practices to help agencies use and manage data. The key practices we identified to compare DEA’s data governance efforts against were: identify data needs to answer key agency questions; provide resources explicitly to leverage assets; prioritize data governance; and support non-federal stakeholders. We selected these practices because they are important to early development of a data governance structure. We also reviewed the February 2019 Data Strategy, released by DOJ, that is to serve as a roadmap for DOJ components to manage their data assets. To understand the extent to which DEA assesses the results of the data it obtains and uses from its ARCOS system and through suspicious order reporting, we reviewed DEA’s performance measures and applicable laws governing performance reporting in the federal government, including the Government Performance and Results Act of 1993 (GPRA), as updated and expanded by the GPRA Modernization Act of 2010 (GPRAMA). Although GPRA and GPRAMA requirements apply to those goals reported by departments (e.g., DOJ), we have previously reported that they can serve as leading practices at other organizational levels, such as component agencies for performance management. We also reviewed related national, DOJ, and DEA strategy documents that are used to communicate diversion control goals and performance. These documents included the 2018 National Drug Threat Assessment and 2019 National Drug Control Strategy, DOJ’s department-wide strategic plan, DOJ Annual Performance Report, DEA’s 360 strategy guide, and DEA congressional budget justification documents. In addition, we evaluated DEA’s performance measures against criteria in Standards for Internal Control in the Federal Government. Furthermore, we reviewed the extent to which DEA defined objectives and outcome-oriented goals, or established measurable performance targets to evaluate the effectiveness of how it obtains and uses data and compared them to GPRAMA requirements, which may serve as leading practices for DEA. To determine what opportunities exist, if any, for DEA to improve its use and collection of industry-reported data, such as using computer algorithms or real-time reporting, we interviewed DEA officials to determine what analytics, if any, DEA is using to detect and identify potential opioid diversion activities. In our interviews with field division offices, we requested information regarding how investigators received suspicious order reports from registrants and how the investigators requested and used ARCOS and other system analysis to conduct or support their investigative work. We also interviewed officials from other entities with opioid diversion prevention responsibilities, such as state level Prescription Drug Monitoring Programs, the Department of Health and Human Services, including the Centers for Medicare and Medicaid Services, and the Department of Justice (DOJ) U.S. Attorney’s Office Opioid Fraud and Abuse Detection Unit. To obtain perspectives of industry stakeholders on how data, such as suspicious orders may be better reported to DEA, we interviewed four industry associations whose memberships include industry stakeholders. We selected these associations based on their roles in representing various DEA registrant communities, such as pharmacists, pharmacies, and distributors. We also reviewed documentation describing the data available to DEA via its ARCOS database, as well as documentation that described examples of unusual patterns of orders. Based on such information, two GAO specialists identified methods that could be implemented using computer algorithms to analyze ARCOS data to identify patterns that might indicate unusual activity. Additionally, these specialists identified related opportunities that DEA could use to analyze ARCOS data combined with data from other sources, such as prescription rate information, to identify these patterns. To address the extent to which DEA collaborates with industry stakeholders to combat opioid diversion, we examined DEA policies and procedures, and interviewed relevant DEA officials, industry associations, and private sector industry members. Specifically, we examined DEA agency-wide directives and guidance, and component management policies and procedures for providing information to industry stakeholders related to industry’s suspicious order reporting requirements, including written communication DEA sent to industry stakeholders related to suspicious order reporting. In addition, DEA officials provided us with a demonstration of SORS, ARCOS, and the ARCOS Enhanced Lookup Buyer Statistic Tool – available to distributors to help them identify and report suspicious opioid orders. We interviewed DEA officials in eight field division offices who interact with industry stakeholders on, among other things, identifying and reporting suspicious orders. These officials provided their perspectives on the usefulness of suspicious order reports to their investigations as well as other industry self-reported data collected in DEA information systems. We interviewed opioid distributors of varying sizes, as noted above, including some of the largest opioid distributors, based on DEA-provided ARCOS data of opioid-related transactions, for their perspectives on the information and tools DEA provides to them, including the Lookup Buyer Statistics Tool and the ARCOS Enhanced Lookup Buyer Statistic Tool. The views of these distributors are not generalizable to the entire population, but provide insights and information on how industry detects and reports suspicious orders through use of ARCOS data and other tools. We conducted this performance audit from January 2019 through January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Timeline of Selected Events and Legislation Impacting or Related to Industry- Reported Data on Prescription Opioids Pub. L. No. 91-513, 84 Stat. 1242 (Title II of the Comprehensive Drug Abuse Prevention and Control Act of 1970, Pub. L. No. 91-513, 84 Stat. 1236). In addition to the contact above, Tonnye’ Conner-White (Assistant Director), Gary M. Malavenda (Analyst in Charge), David Bruno, Jill Center, Billy Commons, Peter DelToro, Kathleen Drennan, Melissa Hargy, Will Horowitz, Hayden Huang, Eric Hauswirth, Susan Hsu, Nicole Jarvis, Benjamin T. Licht, Amanda Miller, and Jan Montgomery all made key contributions to this report.", "summary": "Since 1999, more than 700,000 people have died of a drug overdose in the United States, with about 48,000 dying of an opioid overdose in 2017 alone. The DEA administers and enforces the Controlled Substances Act as it pertains to ensuring the availability of controlled substances, including certain prescription drugs, for legitimate use while limiting their availability for abuse and diversion. The Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment for Patients and Communities Act, enacted in 2018 included a provision for GAO to study the reporting of suspicious opioid orders on a real-time basis nationally using computer algorithms. This report examines, among other things, how DEA obtains and uses industry-reported data to identify and address suspicious opioid orders and opportunities for DEA to improve these efforts, such as using computer algorithms or real-time reporting. GAO analyzed program documentation and DEA data, and interviewed DEA and industry officials as well as officials from national associations representing distributors, investigators, state boards of pharmacy, and other federal and state agencies. The Drug Enforcement Administration (DEA) collects industry-reported data on the sale and purchase of controlled substances and prescription drugs, including opioids. It uses these data to support ongoing investigations into the diversion of such substances into the illegal market place and to identify investigative leads for its field division offices. GAO identified deficiencies associated with DEA's drug diversion efforts, including the following: Limited proactive and robust analysis of industry-reported data. While DEA's current data systems are not designed to conduct real-time analysis, and it conducts some analyses of industry-reported data, such as in response to requests from its field division offices, DEA could conduct more analyses using automated computer algorithms to help identify questionable patterns in the data. For example, DEA could analyze data to identify unusual volumes of deleted transactions or unusual volumes of drugs that were disposed of rather than sold. It could also analyze data to identify trends in distribution or drug purchases in a given geographic area. Other analysis DEA could perform is to look for unusual patterns when comparing drug orders in one geographic area with other nearby areas. These analyses could potentially help DEA proactively identify suspicious activities or registrants that may warrant investigation. No data governance structure to manage all drug transaction data. Although DEA has guidance, policies and procedures for the use of some information systems, it has not established a formal data governance structure to manage all data it collects and maintains, which are integral to its diversion control activities. A data governance structure is defined as an institutionalized set of policies and procedures for providing data governance throughout the life cycle of developing and implementing data standards. Industry and technology councils, domestic and international standards-setting organizations, and federal entities endorse the use of a governance structure to oversee the development, management, and implementation of data standards, digital content, and other data assets. While DEA began efforts to develop a governance structure, it is in the early stages of development and does not have additional details or documentation of its efforts. An effective data governance structure could help DEA ensure its important data assets are consistently and fully utilized. GAO is making four recommendations related to DEA's collection and use of industry-reported data. DEA agreed with three of the four recommendations, and neither agreed nor disagreed with the fourth.", "document_type": "gao"}
{"report": "According to FTA’s National Transit Database, about 1,500 rural transit providers, including tribal transit providers, supply vital mobility and connections to essential services for people living in rural communities. Rural transit providers generally have low budgets, few employees, and small vehicle fleets. Rural transit providers provide a variety of transit services, including: demand-response, which is scheduled in response to calls from passengers; fixed-routes, which are buses operating according to a set schedule; and deviated-fixed routes, which are fixed-routes that allow for minor route deviations in response to passenger calls. Service areas for rural providers may span dozens of square miles in remote areas—with long trips and only a few riders at any given time—or be located in smaller, more developed rural areas surrounding major cities. DOT primarily supports rural transportation through formula grants, some of which require states and rural transit providers to coordinate. Specifically, these rural transportation formula grants are apportioned to state departments of transportation based on various factors, and these state agencies then allocate funding to rural transit providers as sub- grantees. Sub-grantees can be regional or local governments, non-profit organizations, or federally recognized tribes, which provide public transit services in their communities. DOT also awards rural transit program funds directly to federally recognized Indian tribes through the Tribal Transit Program. See table 1 for a description of the DOT’s primary formula-grant programs that support rural transit. Within DOT, FTA and its 10 regional offices administer these programs; their responsibilities include: 1. grant funding, including targeted grants and contracts for coordination-related projects to enhance mobility and access nationwide; 2. oversight of state transportation agencies and tribal-transit program grantees through State Management Reviews and Tribal Transit Assessments; 3. training and technical assistance to states and rural transit providers; 4. policy interpretations and development to enhance mobility and access. DOT and FTA also lead the Coordinating Council, which is charged with improving coordination across federal programs that fund transportation services for transportation-disadvantaged persons. The Coordinating Council consists of 11 federal agency members, namely, the departments of Agriculture, Education, HHS, Housing and Urban Development, Interior, Justice, Labor, Transportation, and Veterans Affairs (VA); the National Council on Disability; and the Social Security Administration. Aside from DOT, transportation is not the primary mission of these federal agencies. However, each member agency has programs that provide funding for transportation to enable program beneficiaries to access the various health and human service programs within the agencies’ primary missions, such as job training, education, or medical care. For example, the HHS’s Medicaid program requires assurance from states that Medicaid beneficiaries have access to necessary medical services; this medical service includes arranging and providing funding for transportation to medical appointments and other health services when beneficiaries cannot transport themselves. In 2012, we found, among other things, that Coordinating Council member agencies were not effectively collaborating and recommended that the Coordinating Council strengthen its coordination efforts across federal programs. In 2014, we again identified the need to strengthen federal coordination efforts and recommended that the Coordinating Council develop both a strategic plan and a cost-sharing policy to promote and enhance federal, state, and local nonemergency medical transportation coordination activities. For a full description of our prior recommendations and their implementation status, see appendix II. State and local stakeholders—including state transportation agencies, regional planning organizations, rural and tribal transit providers—and health and human service providers, coordinate rural transportation services when they share resources and responsibilities and plan activities to achieve common goals and for the overall benefit of the community. Coordination of rural transportation services can occur across geographic jurisdictions, funding sources, and various local, state, and federal programs. Coordination of transportation services has the potential to reduce transportation program costs by clustering passengers, using fewer one-way trips, and sharing the use of personnel, equipment, and facilities; at the same time, people in need of transportation also often benefit from greater and higher quality services when transportation providers coordinate their operations. Various factors affect rural transit coordination, according to stakeholders we interviewed, participants from three discussion groups, and literature we reviewed. Factors that can affect coordination include availability of resources, alignment of different federal program requirements, availability of coordinating mechanisms, and the distances between transit providers. (See fig. 1.) As discussed below, we found that these factors are often interrelated and can serve as both a motivating factor and a barrier to coordination. The availability of resources was the most commonly cited factor affecting rural transit coordination in our literature review and interviews. Almost two-thirds of the stakeholders we spoke with (30 of 43) and participants in three discussion groups told us that it is difficult to coordinate transit services in rural communities with limited resources, such as funding, staff and time, and technology. For example, a rural transit provider told us that while it provides public transit to a neighboring national park for its visitors during the summer season, insufficient funding from the national park combined with very limited access to FTA’s rural transit funds limits the providers’ ability to effectively coordinate services. We also reported in 2014 that smaller budgets and fewer employees can influence rural transit providers’ ability to coordinate. A 2018 survey of state and local transit and health and human services providers conducted by the National Center for Mobility Management also noted that the availability of resources can be a key barrier to transportation coordination both in rural and non-rural areas. Resources specifically affecting rural transit coordination include: Availability of Matching Funds. The availability of matching state and local funds can affect coordination, as rural transit providers tend to rely on a variety of funding sources to provide transit services. Federal programs generally require a share of state or local funding to match federal funds. Approximately one-third of selected stakeholders (13 of 43) and participants in three discussion groups said that they face challenges identifying enough state or local funding to meet FTA’s matching fund requirements. Some rural transit providers (4 of 21) told us they have access to funds from different sources, but others (4 of 21) said that they are challenged with securing state or local matching funds. For example, local, regional, or state taxes provide some funding streams for public transit, including rural transit providers, in California, Georgia, New Mexico, and Washington. Although revenues from state or local taxes may be available as a funding source, rural transit providers still told us that identifying and coordinating state and local funding sources can be challenging. We previously reported that constrained state and local budgets can make securing these funds difficult as rural transit competes for funding with other needs within a community, such as public safety. Technology and Coordination: Greater Columbia Call-Center People For People, a rural transit provider in Yakima, Washington, uses technology to coordinate and operate the Greater Columbia 2-1-1 (GC211) call center. GC211 maintains a statewide database of community resources, including transportation options. It is one of the state’s seven regional 2-1-1 call centers that directs riders to social, health, and transportation resources. Staffing and time. Some stakeholders (12 of 43) said that rural transit providers do not have enough staff and time to pursue or engage in coordination efforts. For example, three rural transit providers told us that staff sometimes take on multiple duties, such as bus driver and dispatcher in addition to grant and program manager, duties that affect their time and ability to coordinate. Representatives from a national transit planning association also told us that staffing constraints are an issue, particularly with rural transit providers because they are usually more understaffed than urban transit agencies. Technology. Access to technology can help coordinate trips and schedules across rural transit services. About half of the rural transit providers (11 of 21) we interviewed stated that they use software and other technology to schedule trips and operate call centers to facilitate coordination efforts. For example, People For People, a rural transit provider in Yakima, Washington, uses technology to coordinate and operate the Greater Columbia call-center. (See sidebar). However, a handful of stakeholders (4 of 43) mentioned that access to broadband, which is needed to enable technology and scheduling software, can be limited in certain areas, especially on tribal lands. For example, an official from EBCI Transit, a tribal transit provider in North Carolina, said EBCI experienced poor cell phone service and other communication limitations, which affected its ability to schedule and coordinate trips. Our recent work on telecommunications found that tribal lands have significantly lower levels of broadband internet access relative to the country as a whole. Formal Coordinating Mechanisms: State- and Regional-Coordinating Bodies As one of the regional coordinating bodies, the Southwest Georgia Regional Commission has played a central role in coordinating rural transit services through much of its region; it currently provides public transit services in 13 counties to the general public as well as to riders with specific needs to access health and human services in 14 counties. The availability of coordinating mechanisms can facilitate information sharing and coordination. About half of the stakeholders (18 of 43) told us that they participate in some statewide, regional, or local coordinating bodies as part of a process to facilitate coordination. For example, the Georgia Department of Transportation works with regional commissions to coordinate rural transit throughout Georgia. (See sidebar). In contrast, officials from the North Carolina Department of Transportation told us that the state disbanded its coordinating council, which may be contributing to challenges in providing nonemergency medical transportation services. We previously reported that state and local transportation agencies and aging network organizations used a variety of different mechanisms, such as state-, regional-, and local-planning bodies to coordinate transportation services for older adults. Half of the states we selected (4 of 8) have statewide-coordinating bodies. For example, participants from one discussion group said that state requirements can facilitate coordination when the state statute requires rural transit providers applying for or receiving federal, state, or local assistance to coordinate with other state agencies, including the state’s health and human services department, for funding and services. Rural transit providers also told us that they participate in regional- and local-coordinating bodies. For example, all transit providers in Montana are required to coordinate through local Transportation Advisory Committees that plan and prioritize local transportation needs. About one-third of the stakeholders (13 of 43) and participants in three discussion groups also mentioned knowledge-sharing forums—such as conferences and training organized by state transportation agencies, transit industry associations, and FTA—as mechanisms to facilitate coordination. For example, officials from Pullman Transit told us that these forums, such as the Washington State Transit Association’s annual conference, presented opportunities to share and learn about various federal transportation programs, coordinating efforts, and information on best practices. We and others have reported that transit providers, as well as health and human service providers, may encounter substantial challenges trying to coordinate services across different programs when program requirements do not align. For our current work, about one-third of stakeholders (13 of 43) and participants in three discussion groups told us that they face a wide array of barriers coordinating across differing federal laws, regulations, and program requirements. The different federal program requirements can affect rural transit providers’ ability to coordinate transit services as some federal programs are dedicated to specific groups of riders (e.g., older adults, people with disabilities, and low-income riders) with specific needs; such specification of groups makes it difficult to coordinate trips for different riders. Three rural transit providers stated that it is sometimes difficult to coordinate transportation to medical appointments for “blended riders” (i.e., senior citizens, veterans, and the general public) in one trip. For example, VA’s Highly Rural Transportation Grants require rural transit providers to serve only veterans, while Medicaid’s nonemergency medical transportation funds require serving only Medicaid beneficiaries. Rural transit providers— which provide service to the general public within their service areas—are sometimes challenged with providing an efficient and coordinated transit service for VA or Medicaid beneficiaries to access their programs. FTA and the Coordinating Council’s 2018 Focus Group Report also identified federal program requirements, including trip purpose restrictions, as a barrier for coordination. As discussed later in the report, the Coordinating Council has been charged with addressing this barrier, among others, and is currently examining whether and how federal program requirements could be better aligned. Coordination in rural areas can be both essential and challenging because rural transit passengers often need to travel long distances (e.g., 30-100 miles) to reach critical services, such as doctor appointments or grocery shopping. About a quarter of stakeholders (11 of 43) and participants in two of the discussion groups said that the long distance between transit providers in remote rural communities sometimes makes it difficult to find entities or other providers interested in or able to coordinate. Two rural transit providers also told us they have no neighboring transit provider to coordinate with due to the extremely remote rural locations. For example, an official from Turtle Mountain Transit in North Dakota said it is challenging to coordinate with other neighboring tribal transit providers due to the long distance to the nearest tribal transit provider in Spirit Lake, which is about 100 miles away. Turtle Mountain Transit, like a number of other tribal transit providers, often serves large and fairly remote areas. We previously reported that tribal lands can vary in size, and range from the smallest at less than one square mile to the largest, the Navajo Nation, which is more than 24,000 square miles or the size of West Virginia, and extends into the states of Utah, Arizona, and New Mexico. Despite encountering some of the factors that can make coordination difficult, all rural transit providers we interviewed told us that they currently coordinate trips or schedules with other local or regional stakeholders. Such coordination efforts include establishing common drop-off points or common schedules (21 of 21), coordinating to provide access to health and human services (14 of 21) and using technologies, such as software, to facilitate coordination of transportation (11 of 21). Rural transit providers told us that they coordinate with others because coordinating may help them meet increasing rural-transit service demand and improve service. They mentioned that the benefits of their coordination efforts include: increased ridership or access, cost efficiency or reduced costs, and enhanced quality of services. Examples of coordination cited by our selected rural transit providers are summarized in table 2 below. All of the rural transit providers we interviewed also told us they coordinated across various funding sources or shared other resources with nearby transit providers. The most commonly cited coordination and resource-sharing activities included pursuing funding from several programs and raising local revenue for transit (18 of 21); participating in opportunities to share knowledge, such as training (11 of 21); sharing vehicles and related resources, such as maintenance capabilities (8 of 21); and sharing staff to achieve a common goal (5 of 21). Four of our selected rural transit providers also stated that full consolidation of their transit services across multiple jurisdictions or providers resulted in cost savings. Specific examples of these activities cited by our selected rural transit providers are summarized in table 3 below. As the lead agency of the Coordinating Council, FTA has taken a number of steps in recent years, including those summarized below, to work with other Coordinating Council member agencies to enhance federal interagency coordination. From January 2017 through June 2019, FTA and the Coordinating Council members were involved in more than 90 interagency- coordinating activities, according to the Coordinating Council’s summary of recent activities posted on its website. Coordinating activities included interagency meetings, trainings, and webinars to share information and coordinate interagency efforts that support rural communities and improve transportation access to health and human services. For example, in September 2018, staff from FTA and the Department of Agriculture held a webinar for federal, state, and local officials on the opioid crisis and increasing transportation in rural areas to improve access to treatment centers, the courts, and other services in rural West Virginia. In 2018, FTA and Coordinating Council members engaged in significant efforts to inform the strategic direction of the Coordinating Council. From March through June 2018, FTA and some Coordinating Council members convened a series of focus groups with state and local stakeholders, including transit and health and human services providers to be informed of the current state of transportation services and identify leading practices and barriers to transportation coordination. FTA also obtained input from state and local transit and health and human services stakeholders via a survey that the National Center for Mobility Management conducted from June through November 2018 to identify promising practices, barriers, and challenges around coordinated transportation. Working group efforts under way are addressing some of the challenges facing rural transit providers. For example, the Coordinating Council’s Program Analysis Work Group, which was convened in November 2018, is currently examining all federal programs with transportation funding available and conducting program analyses to determine whether and how federal program requirements could be better aligned. FTA officials stated that the Coordinating Council plans to submit a report to Congress with some proposed changes and recommendations for improved alignment of federal requirements by September 2020. While these coordinating activities are constructive and encouraging steps, the Coordinating Council’s progress has been slow in other key areas. In 2014, we recommended that the Coordinating Council develop a strategic plan and cost-sharing policy to promote and enhance federal, state, and local nonemergency medical transportation coordination activities. In addition, the 2015 Fixing America’s Surface Transportation Act (FAST Act) required the Council to publish a strategic plan by December 2016 that, among other things, identifies a strategy to strengthen interagency collaboration and that develops a cost-sharing policy in compliance with applicable federal laws. The FAST Act also required the Coordinating Council to submit a final report containing the Council’s final recommendations to Congress for enhancing interagency coordination. However, the Coordinating Council did not issue the required strategic plan until October 2019, about 3 years after the 2016 deadline. We are currently evaluating this plan as part of our follow-up on the implementation status of our 2014 recommendations. Regarding the final report to Congress on interagency coordination, FTA officials told us that they plan to submit the final report to Congress by September 2020. Additionally, we previously reported on the long-standing challenge of the Coordinating Council Executive Committee, which is tasked with providing top management direction for the Council, providing limited leadership and guidance that can have a broad effect on rural transportation. Specifically, we reported that the Council Executive Committee had provided limited leadership, had not met since 2007, and had not issued key guidance documents that could promote coordination. Accordingly, we recommended that the Council meet and issue guidance documents. According to FTA officials, the Executive Committee met for the first time since 2007 in October 2019 and issued the strategic plan noted above. As previously mentioned, we will continue following up on our prior recommendations (see app. II). FTA also facilitates coordination of rural transit services by engaging directly with state and local stakeholders, including transit and health and human services providers. FTA has, for example, taken the following actions: It created a website that provides resources and information on planning and coordinating rural transportation services. This website includes a self-assessment toolkit for state and local transportation agencies on “Building a Coordinated Transportation System” and a link to case studies on coordination of state and regional councils. FTA staff provides ongoing training, resources, and technical support to state transportation agencies and transit and human services providers through its three technical assistance centers—the National Rural Transit Assistance Program, the National Aging and Disability Transportation Center, and the National Center for Mobility Management. FTA and its three centers have been disseminating and sharing some coordination-focused information through their websites, training, and conferences. For example, FTA officials pointed us to the National Aging and Disability Transportation Center’s webpage on “Annual Trends Report and Spotlight Series” that posted best practices information on a non-profit agency that recruits and uses volunteers to transport older adults to social outings and medical appointments. FTA also annually awards competitive grants for innovative, coordinated health and transportation programs. For example, FTA awarded approximately $9.6 million in fiscal year 2019 to 37 projects that were selected as innovative projects for the transportation of disadvantaged populations that are designed to improve the coordination of transportation services and nonemergency medical transportation services. FTA has also bi-annually recognized rural transit providers with an FTA Administrator’s Award for outstanding rural-transit programs, selected in part based on coordination efforts. FTA officials told us that recipients of this award are expected to share their successful practices at the National Rural Transit Assistance Program conference, which is attended by many rural transit providers. Although FTA has a number of efforts under way to facilitate coordination, we identified limitations with FTA’s current communication and information sharing approach. More than a third of the stakeholders we spoke with (16 of 43) stated that communication and information sharing on coordination opportunities from FTA have been limited. FTA officials told us that they disseminate and share some coordination-focused information through its three technical assistance centers, training, conferences, and regular meetings with state transportation agencies as its direct grantees and transportation industry associations. However, about a quarter of the stakeholders (11 of 43) and participants in one discussion group told us that while they have attended FTA trainings and conferences and have used FTA’s technical centers, the focus has been on grant management issues, such as compliance with drug and alcohol policy and procurement, and not on coordination opportunities. Stakeholders stated that they wanted more information on: ways to coordinate with other providers, how providers addressed coordination challenges, technologies that were used to facilitate coordination, and any quantifiable data and results on coordination. Additional information on leading coordination practices that FTA can share with stakeholders include those that we previously identified, such as defining and articulating a common outcome that agencies can engage in to sustain coordination efforts. In December 2014, we recommended that FTA and the Coordinating Council collect data to track and measure progress in achieving results, including the extent of coordination efforts under way. FTA officials told us that the Council’s recent adoption of their strategic plan includes goals and objectives that represents progress toward measuring the extent of coordination efforts at the federal level. FTA officials also told us that the Council’s final report to Congress that will be submitted in September 2020 will report on the implementation status of the objectives in the strategic plan. We have previously reported on the importance of information sharing on coordination across federal, regional, state, and local government entities. Office of Management and Budget guidance on using information as a “strategic resource” notes that making federal information “discoverable, accessible and useable” can fuel innovation. Further, according to Standards for Internal Control in the Federal Government, agencies should communicate necessary and quality information externally so that external parties can achieve their objectives and periodically evaluate methods of communication, so that the agency has the appropriate tools to communicate quality information with external parties on a timely basis. FTA, however, has not clearly communicated and conveyed information on coordination opportunities and leading practices. For example, while FTA officials told us that they rely on their website to share information with stakeholders, more than a third of the stakeholders (17 of 43) told us that information on coordination opportunities and leading coordination practices are not clearly identifiable on FTA’s website or easily accessible. Two stakeholders, for example, said that while locating program requirement information, such as on procurement, was fairly easy, it was difficult to locate coordination-related information. An official from a transit industry association also commented that “stakeholders would benefit if FTA and the technical assistance centers make coordination resources and training more visible on their websites.” This visibility could include “having coordination as a standalone topic and/or creating a page(s) dedicated to coordination on their websites.” We also determined that coordination-related information was fragmented on FTA’s website and found it difficult to navigate FTA’s website to find leading practices information on coordination. For example, FTA officials referred us to its website on FTA’s Access and Mobility Partnership Grant (also known as the Innovative Coordinated Access and Mobility Grant) for information on leading practices for transportation coordination. In our review of this website, we found a description of projects that FTA selected for the grant, the grant amount, and how the funds will be used. We could not identify any information specifically on how these projects identified opportunities to coordinate or exhibited leading coordination practices. We also examined FTA’s website that provides a self- assessment toolkit for building a coordinated transportation system, as we previously mentioned. FTA officials also mentioned that they developed the Coordination Council’s webpage to present information targeted to coordination. FTA does not have a strategy for communicating and sharing information on coordination opportunities and leading coordination practices for its wide audience of rural and tribal providers, state transportation agencies, and other stakeholders. FTA officials told us that they develop individualized communication plans when they undertake any major activities and examine an approach to communicating and sharing information when they develop annual statements of work for their three technical centers and meet with stakeholders. However, FTA could not provide us with a documented strategy that outlines how it communicates and shares coordination-focused information with state and local stakeholders. In light of the multiple means by which FTA and the Coordinating Council are attempting to communicate information about coordinating rural and tribal transit services, a comprehensive plan or strategy that assesses what information state, local, and transit providers would benefit from receiving and how that information can be effectively communicated could help FTA’s information-sharing efforts have their intended effect. Without such a strategy, stakeholders are without valuable information that could aid them in identifying potential coordination opportunities, leading practices, and data to help inform and facilitate their coordination efforts. Coordination is important to help state transportation agencies, rural transit providers, and human health and service providers meet the increasing needs of those who rely on rural transit systems, particularly in light of limited resources. FTA has taken a number of steps to enhance and facilitate coordination, including having interagency meetings, trainings, and webinars to coordinate interagency efforts that support rural communities and improve transportation access to health and human services. Going forward, it will be critical for the Coordinating Council’s Executive Committee to implement our prior recommendations on key coordination issues. In addition, although FTA, along with its three technical centers, has developed resources to facilitate coordination, its communication efforts have fallen short. Without a communication strategy to effectively reach state and local stakeholders, FTA is missing opportunities to enhance communication and information sharing that can improve coordination among state transportation agencies and rural and tribal transit providers. The Administrator of FTA should develop a communication plan that will effectively share information with state transportation agencies and rural and tribal transit providers on coordination opportunities and leading coordination practices in an accessible and informative way. (Recommendation 1) We provided a draft of this report to the Department of Transportation (DOT) and Department of Health and Human Services (HHS) for review and comment. DOT provided written comments, which are reproduced in appendix III and summarized below. DOT and HHS also separately provided technical comments, which we incorporated as appropriate. In written comments, DOT partially concurred with our recommendation. DOT provided examples of its communication efforts with stakeholders on coordination opportunities and practices and highlighted two recent initiatives to further support the coordination of rural transportation services. For example, in October 2019, DOT established the Rural Opportunities to Use Transportation for Economic Success (ROUTES) initiative to enable better coordination among agencies to address underserved rural areas and to collect input from stakeholders on the benefits rural transportation offers for safety and economic outcomes. In partially concurring with our recommendation, DOT wrote that it plans to direct each of its technical assistance centers to reorganize its web pages to centralize coordination information and best practices. We acknowledge FTA’s efforts and highlighted the progress FTA has made in communicating and facilitating coordination in this report. We noted that FTA has provided ongoing training, support, and resources through its technical assistance centers. While DOT’s plans to have its technical assistance centers’ web pages reorganized may help in communicating coordination opportunities with stakeholders, they fall short of a comprehensive communication plan. Such a plan would define a strategy for effectively communicating and sharing information with stakeholders and ensuring that methods of communication are reaching all intended stakeholders. Among other things, FTA’s plans to increase access to coordination information does not include reorganizing and centralizing coordination-related information on FTA’s web pages, a strategy that is different from these technical centers’ web pages and one where many stakeholders can turn to and search for communication and information. We believe that a comprehensive communication plan that includes FTA’s strategy for ongoing communication on coordination opportunities would enable FTA to ensure that coordination information is reaching intended stakeholders to inform them of opportunities to enhance rural transit services. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions regarding this report, please contact at me (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. Appendix I: Rural Transit Stakeholders GAO Interviewed and Discussion Group Participants Industry groups Community Transportation Association of America National Association of Development Organizations National Association of Regional Councils National Center for Mobility Management National Rural Transit Assistance Program Small Urban, Rural and Tribal Center on Mobility Federal Transit Administration (FTA) Regional Office FTA Region IV ^ State transportation agencies Caltrans - California Department of Transportation New Mexico Department of Transportation North Carolina Department of Transportation North Dakota Department of Transportation Washington State Department of Transportation Rural transit providers (including tribes’ names, where appropriate) Carlsbad Municipal Transit System CSKT Transit (Confederated Salish and Kootenai Tribes of the Flathead Reservation) * EBCI Transit (Eastern Band of Cherokee Indians) *^ Missoula Ravalli Transportation Management Association ^ Morongo Transportation Department (Morongo Band of Mission Indians) * North Central Regional Transit District Pueblo of Santa Clara, New Mexico * Rocky Boy’s Transit (Chippewa Cree Indians of the Rocky Boy’s Reservation, Montana)* Southeast Tennessee Human Resource Agency ^ Turtle Mountain Transit (Turtle Mountain Band of Chippewa Indians of North Dakota) * Williston Council for the Aging Legend: * = Recipient of FTA’s Tribal Transit Program funding. ^ = Site visit to interview stakeholder. State transportation agencies Mississippi Department of Transportation VTrans - Vermont Agency of Transportation Rural transit providers (including tribe names, where appropriate) Big Woods Transit (Bois Forte Band (Nett Lake) component of Minnesota Chippewa Tribe, Minnesota) Center for Community the RIDE (Sitka Tribe of Alaska) * Choctaw Tribal Transit (The Choctaw Nation of Oklahoma) * Heart of Iowa Regional Transit Agency Hualapai Transit (Hualapai Indian Tribe of the Hualapai Indian Reservation, Arizona) * Oglala Sioux Transit (Oglala Sioux Tribe) * Salt River Transit (Salt River Pima-Maricopa Indian Community of the Salt River Reservation, Arizona) * Legend: * = Recipient of FTA’s Tribal Transit Program funding. In addition to the individuals named above, Heather MacLeod (Assistant Director); Jennifer Kim (Analyst-in-Charge); Matthew Bond; Delwen Jones; Rosa Leung; Theresa Lo; Anna Maria Ortiz; Cheryl Peterson; Malika Rice; Kelly Rubin; Pamela Snedden; Lisa Van Arsdale; and Sarah Veale made key contributions to this report.", "summary": "Public transportation in rural areas is critical to connecting people to medical services, jobs, education, and shopping. FTA allocated about $2.1 billion in formula grants over the last 3 years to support rural and tribal transit. In 2014, GAO reported that providing transit services in rural areas can be challenging and that coordination of transportation services among federal programs is limited. GAO was asked to examine ongoing efforts and challenges of coordinating rural transit systems. This report addresses (1) factors affecting rural transit coordination and selected rural and tribal transit providers' coordination efforts and (2) the extent to which FTA facilitates coordination of rural transit services. GAO reviewed program documentation and literature on rural transit coordination. GAO also interviewed federal officials from FTA and the Department of Health and Human Services, which also funds transportation services, and rural transit stakeholders, including state transportation agencies, rural and tribal transit providers, and public transit industry groups. GAO selected states and rural and tribal transit providers based on federal-funding levels and geographic representation, among other factors. Coordination of rural transportation services across geographic jurisdictions and federal- and state-funding sources has the potential to reduce costs and improve services. Such coordination by transit agencies in rural areas can lead to efficiencies. A variety of factors, however, adversely affect rural transit coordination, including the availability of resources, according to GAO's literature review and stakeholder interviews. About 70 percent of the selected stakeholders GAO interviewed, including rural and tribal transit providers, explained that it is difficult to coordinate transit services in rural communities with limited resources, such as funding, staff, and technology. For example, three rural transit providers said that program managers sometimes assume multiple duties, such as a driver and dispatcher, a practice that affects their time and ability to coordinate. Other cited factors included the extent to which different requirements of federal programs that fund rural transit are aligned to allow transit providers to coordinate trips for riders with specific needs (e.g., people with disabilities) and the availability of coordinating mechanisms, among other factors (see figure). Nonetheless, selected rural and tribal transit providers said they were engaged in various coordination efforts to improve rural transit services. The most commonly cited efforts under way included coordinating trips—for example, by establishing convenient drop-off points—and sharing resources. The Federal Transit Administration (FTA) has several efforts under way to facilitate coordination, but results are mixed. At the federal level, FTA and the federal interagency Coordinating Council on Access and Mobility issued a strategic plan in October 2019, outlining their strategic goals. However, they have yet to submit to Congress a final report containing recommendations for enhancing interagency coordination. FTA officials told us they plan to submit the report by September 2020. At the state and local level, FTA has provided technical support to stakeholders to faciliate coordination. GAO, however, found limitations with FTA's current information-sharing approach. These limitations make information on coordination-related issues difficult to identify and access. Stakeholders want additional information from FTA on leading coordination practices, such as ways to coordinate with other providers. Improving communication and sharing additional coordination-related information could help rural and tribal transit providers identify additional coordination practices they could pursue to improve rural transportation services. GAO recommends that FTA develop a communication plan that will effectively share information with state and local stakeholders on coordination opportunities in an accessible and informative way. FTA partially concurred with the recommendation. As discussed in the report, GAO continues to believe the recommendation is warranted and should be fully implemented.", "document_type": "gao"}
{"report": "The United States has more than 19,000 airports, which vary substantially in size and the type of aviation services they support. Of these, roughly 3,300 airports are designated by FAA as part of the national airport system and are therefore eligible for federal assistance for airport capital projects. The national airport system consists of two primary types of airports—”commercial service” airports—which are publicly owned, have scheduled service, and board at least 2,500 or more passengers per year—and “general aviation” airports—which have no scheduled service and board fewer than 2,500 passengers. Federal law divides commercial service airports into various categories of airports, based on the number of passenger boardings, ranging from large hub airports to commercial service non-primary airports (see fig. 1). Consistent with our prior work, we have grouped airports into two broader categories: larger airports, which includes large and medium hubs, and smaller airports, which includes small hubs, non-hubs (also referred to as “non-hub primary”), and non-primary commercial service airports as well as reliever airports, general aviation airports, and new airports. The majority of passenger traffic is at larger airports, which accounted for 88 percent of all commercial airport enplanements in 2017. From 2013 to 2017, enplanements have increased at airports of all hub sizes. Specifically, commercial airport enplanements at larger and smaller airports increased by 16 percent and 15 percent, respectively, during this time period. National system airports are eligible to receive federal funding from AIP grants for infrastructure development. AIP funds are first authorized in FAA reauthorization acts, and Congress then appropriates funds for AIP grants from the Airport and Airway Trust Fund, which is supported by a variety of aviation-related taxes, such as taxes on tickets, cargo, general aviation gasoline, and jet fuel. While AIP grants are an important source for airports’ infrastructure funding, the amount of funding authorized for the AIP grant program has not changed since 2012. In 2018, Congress passed the FAA Reauthorization Act of 2018, which authorized annual AIP grant levels at $3.35 billion annually through fiscal year 2023 and authorized additional amounts for supplemental discretionary funding each year from 2019 through 2023, starting at $1.02 billion and increasing each year thereafter. In addition, the Consolidated Appropriations Act of 2018 appropriated $1 billion in supplemental annual funding from the general fund for the AIP discretionary grant program. Subsequently, in February 2019, the Consolidated Appropriations Act of 2019 provided $500 million from the general fund to the AIP discretionary grant program. The distribution of federal AIP grants is complex. It is based on a combination of formula funds—also referred to as entitlement funds—that are available to national system airports, and discretionary funds that FAA awards for selected eligible projects. Entitlement funds are apportioned by formula to airports and may generally be used for any eligible airport improvement or planning project. Discretionary funds are approved by FAA based on FAA selection criteria and a priority system, which FAA uses to rank projects based on the extent to which they reflect FAA’s nationally identified priorities. AIP grants must be used for eligible and justified projects, which are planned and prioritized by airports, included in their capital improvement plans, and reviewed and approved by FAA staff and the Secretary of Transportation. Generally, most types of airfield improvements—such as runways, lighting, navigational aids, and land acquisition—are eligible for AIP funding. AIP-eligible projects for airport areas serving travelers and the general public—called “landside development”—include entrance roadways, pedestrian walkways and movers, and common space within terminal buildings, such as waiting areas. See figures 2 and 3 for more information about the types of projects eligible for AIP funding. For all AIP-funded projects, the airport must provide a share of matching funds. The federal share is from 75 to 95 percent depending on the size of the airport or type of project. Revenue from PFCs is another means of support for airport infrastructure projects. PFCs are federally authorized fees which were established in 1990 to help pay for infrastructure at commercial service airports. Although PFCs are local funds subject to the airport’s control, FAA oversees the PFC program and approves applications by airports to collect PFC revenues. PFCs are currently capped at $4.50 per flight segment with a maximum of two PFCs charged on a one-way trip or four PFCs on a round trip, for a maximum of $18 total. On behalf of the airports, airlines collect the PFC at the time of the ticket purchase and remit the PFC, minus an administrative fee, to the airport. To meet future planned infrastructure costs, airports have sought an increase in the cap on PFCs. However, airlines oppose a PFC increase because they believe airports already receive sufficient PFC revenues and that higher ticket prices could reduce passenger demand and airline revenues. We have previously reported that increasing the PFC cap would significantly increase PFC collections available to airports under three scenarios GAO modeled but could also marginally slow passenger growth and growth in revenues to the Airport and Airway Trust Fund (AATF). Airports have more flexibility in using PFCs to fund infrastructure projects as compared to AIP funding. Airport infrastructure projects eligible for PFC funding must meet one or more of the following: preserve or enhance safety, security, or capacity; reduce noise or mitigate noise impacts; or increase air carrier competition. Airports are able to fund projects with PFC revenues that might not be eligible for AIP funding, such as passenger terminal projects and development at gates, airline ticketing areas, and passenger check-in facilities at hub airports. In addition to being applied to FAA approved eligible projects, PFCs can be used as a match for AIP grants or to finance the debt on approved projects. FAA and ACI-NA each produce reports summarizing 5-year estimates of U.S. airports’ infrastructure project costs. More specifically, FAA is required to publish a 5-year estimate of AIP-eligible development every 2 years. FAA provides this information in its NPIAS report. FAA relies on airports, through their planning processes, to identify individual AIP- eligible projects for funding consideration and inclusion in the NPIAS. The ACI-NA also collects data on all proposed capital development projects at U.S. airports and every 2 years publishes a report of U.S. airports’ 5-year infrastructure cost estimates. From fiscal years 2013 through 2017, national system airports received an annual average of about $15 billion in funding from a variety of sources for infrastructure development projects, including: federal AIP grants (about $3.2 billion annually); airport revenue from passenger charges (about $3.1 billion annually), and airport-generated revenue (about $7.7 billion annually); and capital contributions (about $715 million annually). These figures, however, do not represent the full amount of funding that is available to airports for infrastructure development. For example, some airports also received funding from state grants and bond proceeds through debt financing to fund airport infrastructure investments. In addition, the proportion of funding that larger and smaller airports received from these sources varies. From fiscal years 2013 through 2017, the total amount of AIP grants that national system airports received has generally remained constant. As shown in figure 4 below, the amount of AIP grant funding that airports received ranged from $3.1 billion to $3.3 billion annually for fiscal years 2013 through 2017. Overall, airports received an average of $3.2 billion annually in AIP grants. The total amount of AIP grant funding that FAA allocates to airports may vary slightly year to year for many reasons. For example, according to FAA, each year a small amount of AIP funding is returned from prior-year grants and the FAA is permitted to re-obligate those funds on either existing or new grants. Collectively, smaller airports received more AIP grant funding compared to larger airports during this time period. As shown in figure 4, from fiscal years 2013 through 2017, smaller airports received the largest share of AIP grant funding, approximately 75 percent, (an annual average of $2.4 billion), compared to 25 percent received by larger airports (an annual average of $812 million). Larger airports are generally able to rely on other sources of revenue generated from airport-generated revenue and PFCs due to higher enplanements compared to smaller airports. In addition, the amount of AIP grants’ funding that smaller hub airports received increased by about 10 percent between fiscal years 2013 through 2017, while the amount of AIP’s funding for larger airports decreased by 3 percent in the same time period. However, smaller airports receive less funding per AIP grant compared to larger airports. For example, smaller airports received an average of $897,000 per grant, while larger airports received an average of $5 million per AIP grant. Some airports also received state funding, primarily in the form of grants used as matching funds for federal AIP grants. Data for fiscal years 2013 through 2017 on states’ grant funding are not available. However, in 2015, we conducted a survey of airports, in collaboration with NASAO, for fiscal years 2009 through 2013, and reported that states provide an annual average of $477 million to national system airports. According to NASAO officials we interviewed for our current work, states’ grant-funding levels have remained unchanged. From fiscal years 2013 through 2017, airports collected revenue from a variety of sources, including PFC charges and airport-generated revenue (both aeronautical and non-aeronautical), which have both increased during our 5-year time period. Some airports also received funding from capital contributions, but that amount has decreased from fiscal year 2013 through 2017. Airport revenue is the largest source of funding for larger airports. Specifically, larger airports generated an annual average of $10.4 billion in airport revenue (or 90 percent of all airport revenue) during our 5-year time period. Smaller airports generated less airport revenue, with an annual average of $1.2 billion (or 10 percent of all airport revenue), compared to larger airports. Larger airports’ ability to generate more airport revenue reflects that PFCs and airport-generated revenue could be driven by the higher levels of passenger enplanements and airline activity associated with current economic conditions. According to FAA officials, while total airport revenue has increased over this time frame, it does not necessarily mean that airports have more revenue available for new capital expenditures. For example, airport revenue is also used to pay for existing debt service and operating costs, which according to FAA officials, has also increased during this time period. Overall, from fiscal years 2013 through 2017, U.S. airports collected an annual average of $3.1 billion in PFC revenue. As shown in figure 5, during this period, the annual average for PFC collections for all airports increased by 9 percent from $3 billion to $3.3 billion. Because PFCs are generated by the number of enplaned passengers, this increase was mostly driven by a 16 percent increase in passenger enplanements during this period for both smaller and larger airports. As shown in figure 5, larger airports collected most (89 percent) of the PFC revenues in fiscal years 2013 through 2017. In addition, although both larger airports and smaller airports experienced an increase in passenger enplanements in fiscal years 2013 through 2017, larger airports experienced a 10 percent increase in PFC revenue while smaller airports experienced an overall decrease in PFC revenue during this period of about 3 percent. According to FAA officials, smaller airports may have experienced an overall decrease in PFC revenues because airports’ PFC collections may cease when they have fully collected the approved amount for a project. According to FAA, this cessation is particularly true for smaller airports that do not have multiple projects for which PFC collections have been approved for a long period of time. In addition, if an airport has approved collections but one or more airlines make significant reductions in activity levels, this factor can also slow the rate of collections at airports. Larger airports hold a larger market share of flights, representing 88 percent of enplanements. Ratings agency representatives said that larger airports rely more on PFCs and bonding to fund infrastructure projects. From fiscal years 2013 through 2017, U.S. airports generated an annual average of $7.7 billion in airport-generated revenue. During this period, airport-generated revenue increased 18 percent, from $7.1 billion to $8.4 billion for all airports. Overall, both larger and smaller airports generated more income over this time period, with larger airports generating substantially more revenue compared to smaller airports. Specifically, from fiscal years 2013 through 2017, larger airports generated an annual average of $7.1 billion in revenue, and smaller airports generated an annual average of $567 million in revenue. Airport-generated revenue consists of both “airside” aeronautical revenues derived from the operation and landing of aircraft, passengers, or freight, as well as “landside” non-aeronautical revenues derived from terminal concessions and parking fees. Of the $103 billion in airport- generated revenue over our 5-year time period, 54 percent came from aeronautical revenues and 46 percent came from non-aeronautical revenues (see fig. 6). Commercial service airline rates and charges— which include passenger airline’s landing fees and passenger arrival fees, rents, and utilities—made up 75 percent of the total $55.9 billion in aeronautical revenue. The remainder came from a variety of other fees and taxes paid by airlines, general aviation, the military, and other aeronautical sources. Of the non-aeronautical revenues, parking and ground transportation accounted for the greatest portion (41 percent), followed by rental cars operations revenue (19 percent). Aeronautical revenues increased by 11 percent and non-aeronautical revenues increased by 16 percent over the time period. Capital contributions for airport infrastructure projects make up a small amount of funding in comparison to other sources, such as airport- generated revenue and AIP funding. These contributions—made on an individual project basis—may be provided by an airport’s sponsor (often a state or municipality) or by other sources such as an airline. According to FAA data on commercial airports’ annual financial reports for fiscal years 2013 through 2017, commercial airports received an annual average of $715 million in capital contributions. Of this amount, $471 million, or 66 percent, went to larger airports, and $244 million, or 34 percent, went to smaller airports. The amount of capital contributions varies by year and by hub size. According to FAA officials, the sources of capital contributions funding (i.e., airport sponsor, state, air carriers, or other airport users) vary depending on the type of project and funds available. Airports can also obtain financing for airport infrastructure projects by issuing bonds. Airport bonds entail leveraging future funding to pay for projects. This financing mechanism enables airport authorities to borrow money up front to finance infrastructure projects; this money can then be paid back with interest over a longer time period. U.S. airports may qualify for tax-exempt bonds to support airport projects for federal tax purposes because the airports are owned by states, counties, cities, or public authorities. The tax-exempt status enables airports to issue bonds at lower interest rates than taxable bonds, thus reducing a project’s financing costs. FAA officials said that because airports use some PFCs and airport-generated revenue to pay off debt service, not all revenue generated from these two sources is available for additional infrastructure investment. FAA collects data in its financial reporting database of an airport’s total indebtedness. Based on our analysis of this data, from fiscal years 2013 through 2017, airports had averaged $84.6 billion in total bond debt per year. The total indebtedness measure provides an overall aggregate of the level of long-term bond debt held by airports for the year. FAA’s data do not differentiate indebtedness for each type of bond, nor do its data differentiate between existing, new, or refinanced bonds. As a result, we were not able to analyze how much airports obtained on average for new projects by issuing bonds from fiscal years 2013 through 2017. In addition, we were not able to determine whether U.S. airports borrowed increasing amounts of new bond proceeds from fiscal years 2013 through 2017 to meet infrastructure needs. Moreover, FAA does not collect data on the time frame that airports anticipate to pay back bonds, as FAA officials said that airports have the latitude to determine their own debt- payment schedules. During fiscal years 2013 through 2017, larger airports received the vast majority of bond proceeds, representing 95 percent of the total (see fig. 7). This amount includes debt from all long-term bonds. We previously reported that bond financing has traditionally been an option more commonly exercised by larger rather than smaller airports because they are more likely to have a greater and more certain revenue stream to support debt repayment. We have also reported that when smaller airports issue bonds, they make greater use of general obligation bonds that are backed by tax revenues of the airport sponsor, which is often a state or municipal government. FAA officials added that larger airports tend to issue airport revenue bonds, which are backed solely by airport revenue, while some smaller airports may be able to benefit from bond proceeds issued by the broader county or municipal government and backed by that entity's taxing authority. Based on our analysis, airports’ planned infrastructure costs are projected to average $22 billion annually for fiscal years 2019 through 2023. To arrive at this estimate, we combined FAA’s $7 billion estimate of AIP- eligible planned infrastructure costs and ACI-NA’s $15 billion estimate of planned infrastructure costs for projects that are not eligible for AIP grants. Our $22 billion estimate would represent an increase of 19 percent from FAA’s and ACI-NA’s fiscal years 2017 through 2021 infrastructure cost estimates. This increase is largely driven by an increase in ACI-NA’s estimate of AIP-ineligible planned projects. Specifically, ACI-NA’s annual average of about $15 billion in planned AIP- ineligible costs reflects an increase of $3.3 billion or 28 percent when compared to the annual average estimate of AIP-ineligible projects from ACI-NA’s fiscal year 2017–2021 estimates. Similarly, FAA’s annual average of $7 billion in planned AIP-eligible costs reflects an increase of $289 million or 4 percent from FAA’s fiscal year 2017–2021 estimates. A variety of factors may be contributing to the increase in FAA’s and ACI- NA’s cost estimates, factors that we will discuss later in the report. Overall, larger airports (large and medium hub) accounted for 75 percent of the $22 billion annual cost estimate and make up a greater percentage of the estimated increase in planned development costs when comparing the fiscal years 2017 through 2021 and fiscal years 2019 through 2023 estimates. For example: Among planned AIP-eligible projects, estimated annual planned- development costs increased from $1.4 to $1.7 billion (an 18 percent increase) for large hub airports and from $641 to $735 million (a 15 percent increase) for medium hub airports, according to FAA’s cost estimates. By comparison, estimated planned development costs for small hub and non-hub airports decreased by 3 and 2 percent respectively over the same time period. Among AIP-ineligible projects, ACI-NA estimates show that annual planned development costs increased more significantly for medium hub airports. Specifically, ACI-NA’s report shows that annual planned development costs for AIP-ineligible projects increased by 22 percent for large hub airports, 71 percent for medium hub airports, and 29 percent for small hub airports. ACI-NA representatives stated that the increase in medium hub airport’s planned development (for both AIP-eligible and AIP–ineligible projects) is due to the underinvestment at medium hub airports in prior years. Specifically, ACI-NA representatives stated that in response to the loss of air service immediately following the 2007–2009 recession, some medium hub airports scaled back their capital investments. ACI-NA representatives stated that as passenger traffic has recovered with economic growth, medium hub airports are now investing in previously deferred improvements. According to ACI-NA’s report on airports’ capital development needs for 2019–2023, medium hub airports–such as Austin- Bergstrom International Airport (Austin airport), Norman Y. Mineta San Jose International Airport, and Dallas Love Field Airport–are undertaking major infrastructure improvement programs. According to officials from Austin airport, the airport recently completed a 10-year plan for its capital development program, with an estimated cost of $3.5 billion, for a new terminal, concourse, airfield improvements, runway improvements, and improved landside access. Austin airport officials stated that the airport is 20 years old and nearing the end of its lifecycle, and airport officials are trying to manage aggressive growth while rebuilding the airport. The sources of funding and types of infrastructure projects that smaller and larger airports have planned also differ. For example, smaller airports have more AIP-eligible planned costs compared to larger airports, according to FAA cost estimates. Specifically, smaller airports accounted for about $4.6 billion (or 66 percent) of AIP-eligible project costs for all airports but, according to ACI-NA cost estimates, only $878 million (6 percent) of AIP-ineligible projects. In addition, among AIP-eligible projects, while the top four types of infrastructure projects that larger and smaller airports have planned are similar (see table 1), estimated costs are more concentrated among the top two project-type categories for smaller airports. Specifically, reconstruction projects, which are projects to replace or rehabilitate airport facilities such as runways, and projects to meet FAA standards for airport design represented about 79 percent of smaller airports’ AIP-eligible estimated project costs. ACI-NA’s data do not break out AIP-ineligible project costs by project type. As a result, we were not able to determine what types of projects constitute the largest shares for AIP-ineligible project costs. However, ACI-NA does provide information about project type across all the projects in its cost estimate. According to ACI-NA’s representatives, the types of projects that are generally not funded with AIP grants that airports need to fund include landside projects, such as terminal projects; rental car and parking facility projects; concession redesign projects; and airport access projects. The increase in planned infrastructure costs for fiscal years 2019 through 2023 can be attributed in part to an increase in planned terminal projects during this 5-year time period. Specifically, both FAA’s and ACI-NA’s cost estimates show an increase in planned terminal projects. For example, according to FAA’s estimates of planned projects funded by AIP grants, terminal projects now represent the third largest share of total estimated costs from fiscal years 2019 through 2023 and experienced the greatest percentage increase over the previous 5-year period. As shown in table 2, overall annual average cost estimates for terminal projects increased by 51 percent between the two periods. Environmental projects was the only other project category that had significant increases (about 38 percent), while estimated costs for many other types of projects decreased. According to FAA officials, the increase in environmental projects is due to increases in environmental-related NPIAS costs (such as mitigation of development impacts and costs for environmental studies) at large and medium hub airports and additional noise mitigation at hub airports. Similarly, according to ACI-NA’s analysis, for fiscal years 2019 through 2023, terminal projects represented 53 percent of the total infrastructure- development costs among both AIP-eligible and AIP-ineligible projects. Terminal projects included terminal building projects (37 percent) and projects to provide access to the terminal (16 percent). FAA and ACI-NA representatives stated that terminal projects can be more expensive than other types of projects because of the scale of these improvements. For example, terminal projects may involve complex vertical construction, an array of special systems such as baggage and passenger screening systems, and integration of security and access control systems, all of which can contribute to the overall higher cost of these projects. In contrast, runway and airfield infrastructure generally rely on common design standards and standard construction methods according to ACI- NA representatives. Additionally, officials from most (16 out of 19) of the airports that we spoke to stated that they are planning terminal improvement projects over the next 5 years. Officials from these airports told us they are focused on making terminal improvements because existing terminals are aging and in need of repairs and to accommodate an increase in passenger enplanements due in part to airlines using larger aircraft holding more passengers. Examples of planned terminal projects at selected airports and factors contributing to these investments are below. Large hub airport terminal project. Officials from a large hub airport that we spoke to stated that they have two ongoing major terminal projects. The first project will expand and renovate the airport’s north terminal. The 468,000-square-feet facility will include a new upper- level mezzanine, seismic upgrades, and an upgraded baggage- handling system, among other improvements. According to airport officials, capacity constraints and the age of the terminal were factors for renovating the terminal. Phase 1 of the project began in February 2017 and was completed in mid-2019. As of July 2019, nine gates are operational. The second phase of construction is expected to be completed in mid- 2021. The estimated cost of the project is projected at $658 million. The airport is also developing a new international arrivals facility at its airport. According to airport officials, this facility is intended to significantly enhance the international passenger experience, and improve the arrival process for international passengers without adding new gates. Airport officials stated that the current facility is not able accommodate the city’s growing demand for international travel. The facility is estimated to cost about $968 million and is expected to open in the fall of 2020. Medium hub airport terminal project. According to officials from a medium hub airport, growth in passenger traffic is driving the need for a new terminal at that airport. International traffic at the airport has tripled between 2012 and 2017, with airlines adding three new service destinations to Europe. According to airport officials, the existing terminal will soon reach its capacity to handle international arrivals, and the first phase of the terminal project was substantially completed in 2019 and cost about $350 million. Small hub airport terminal project. Officials from a small hub airport stated that airlines have started replacing existing aircraft with larger aircraft, and this process has placed capacity constraints at their terminal. The terminal was built in 1948, and the passenger waiting area was built in the 1960s when airlines providing service to the airport were using aircraft with 100 seats. Now, however, airlines are using larger aircraft, which can accommodate up to 180 seats. Airport officials stated that they are beginning construction of a new terminal, which will expand passenger capacity at the airport. The overall estimated costs of the terminal project are $513 million, and the project is expected to be completed in 2028, pending additional funding. FAA officials and ACI-NA representatives agreed that the increased focus on terminal projects is due in part to airlines changing their business models and aircraft fleets and an increase in passenger traffic. The officials stated that as part of the industry’s fleet rationalization efforts, airlines are eliminating some smaller aircraft and replacing them with larger aircraft to increase passenger-seating capacity. FAA officials added that passenger growth at large and medium hub airports is also contributing to the increase of AIP-eligible terminal costs, as airports need to expand terminals to add capacity. According to FAA, terminal projects at large and medium hubs are generally funded through PFCs and other funding sources rather than through AIP funding. For its 2019–2023 NPIAS report, however, FAA officials said they asked airports to provide information about AIP-eligible projects regardless of whether they were planning to apply for AIP funding for the projects. According to FAA officials, this factor may also have contributed to the apparent increase in AIP-eligible terminal costs. According to FAA, another factor driving the increase in terminal costs is that seven airports have planned major terminal projects over the next 5 years. The costs of these projects are reflected in FAA’s AIP-eligible cost estimate. In addition to an increased focus on terminal projects, FAA officials, ACI- NA representatives, and selected airports cited other factors that are contributing to an increase in infrastructure costs estimates, such as increased construction costs, an overall healthier economy, increased traffic, airline consolidation, and airlines’ strategic shift to focus on hub operations. For example, according to Nashville International Airport’s officials, a growing economy has resulted in more competition for construction materials and skilled workers, competition that has increased construction costs in the Nashville area and has resulted in higher airport development costs. According to ACI-NA representatives, other larger cities such as Salt Lake City, Los Angeles, and Seattle have also reported cost escalation in their construction markets. Officials from most (18 out of 19) selected airports we interviewed stated that the funding and revenue available to them from existing funding sources—such as AIP grants and PFC revenues—may not be sufficient to cover the costs of future and planned infrastructure projects. For example, 14 airport officials we spoke to stated that the amount of funding that they have received in the past and that they anticipate receiving in the future from AIP formula or discretionary grants will not be sufficient to cover the costs of their future planned AIP-eligible projects. Airports may use a variety of other funding sources to pay for AIP-eligible projects. As such, differences between available AIP funding and AIP-eligible cost estimates do not necessarily reflect a funding shortfall. In addition, the NPIAS estimates represent planned AIP-eligible project costs and do not reflect actual expenditures. Below are some examples of AIP-eligible projects that airport officials stated will be a challenge to complete without additional funding: Airfield safety projects. Officials from a small hub airport stated that they have two major airfield-safety projects planned that are intended to align their airport’s current runway and taxiway to FAA safety standards. According to airport officials, their airport has been on the FAA’s top-10 list of airports with highest “incursions” for 4 consecutive years, and officials stated these airfield improvements would help them mitigate runway incursions at their airport. According to airport officials, these projects are expected to cost about $230 million, which they stated is a significant cost for an airport of their size. Their primary source of funding is AIP funding and PFC revenues; however, their current AIP formula funding and PFC revenues are not sufficient to cover the cost of the projects. Without additional funding, officials said that they will need to complete the project in phases, which could lead to a multi-year project ranging from 4 to 12 years to complete. Airport officials stated that a multi-year project of this length would significantly affect their airport operations and increase overall costs. They also stated that ideally, it would be most efficient to execute the project in fewer phases to reduce costs and to benefit airport users, as construction may negatively affect airport operations. Runway rehabilitation project. Similarly, officials from another small hub airport said their airport receives about $5 million annually in AIP formula funding, which they said is not sufficient to cover the costs of their planned runway pavement rehabilitation and reconstruction project. The total cost of the project is about $20 million. According to airport officials, if they are unable to find alternate sources of funding for the project, they will have to postpone the runway project, and such a postponement would have a significant effect on their airport operations. Runway replacement project. Officials from a medium hub airport are planning to invest in a new runway project that is expected to cost about $350 million. The existing runway is nearing the end of its useful life and needs to be replaced. They anticipate receiving approximately $4.5 million annually in AIP formula funding and plan to apply for discretionary AIP funding as well. They stated that currently, this airport’s PFC revenues have been obligated until 2032 and that therefore, they are not able to use this funding source to pay for the runway. According to airport officials, without these funding sources the airport will be required to use their existing bonding capacity to pay for this critical infrastructure, a move that would reduce their future bonding capacity for future critical infrastructure improvements. Fourteen airport officials also stated that revenue generated from PFCs is also not sufficient to cover the costs of planned infrastructure. For example, officials from one large hub airport stated that they have been successful in being able to keep up with the pace of growth at their airport, but based on their forecasts, they anticipate that they would be unable to meet infrastructure demands without an increase in PFC funding. Officials from six airports stated that because the PFC cap has remained at $4.50 since 2000 and has not been adjusted for inflation, the value of the PFC has decreased. In 2015, we reported that an inflation adjusted PFC cap would be $6.46. Representatives from eight airlines that we spoke to, however, disagree that the PFC cap should be increased citing increases in passenger traffic, increases in PFC revenues, and availability of other adequate sources of funding. According to FAA officials, increases in passenger traffic and other changes have also increased the need for capital facility investments. Officials from about half of the airports (nine out of 19) that we spoke to— including a mix of smaller and larger airports—stated that that the revenue their airport generates from PFCs are already obligated toward current infrastructure projects, which they stated could affect their ability to use debt financing for future infrastructure projects. An additional three airports we spoke to stated that they plan to use PFC revenues to finance planned infrastructure projects and that they anticipate that these revenues will be obligated over a long term period—about 30 years— limiting their ability to use debt financing for other projects. FAA’s financial data show that airports committed a significant share of their PFCs to debt service during fiscal years 2013 through 2017. Specifically, of the $16 billion in PFC revenues (or an annual average of $3.1 billion) collected in fiscal years 2013 through 2017, airports paid a total of $12 billion for debt service (or an annual average of $2.5 billion)— which is about 78 percent of total PFC revenues generated during this time period. The debt service includes payments on new bonds, existing bonds, and refinanced bonds which, as previously noted, are collectively tracked in FAA’s database. As shown in figure 8, over our 5-year time period, larger airports accounted for the vast majority (over 90 percent) of the PFCs dedicated to debt service. According to ACI-NA’s report on airports’ capital development needs for 2019–2023 and some selected airport officials, because airports have already committed a significant portion of their current and future PFCs to servicing debt on current or completed projects, airports will have less PFC funding available for future projects. According to ACI-NA’s report on airports’ capital development needs for 2019–2023, the entire national airport system is carrying a combined debt of $91.6 billion from past projects and may be unable to pay for future needed projects unless the existing cap on PFCs is increased. Officials from three small hub airports stated that they are currently facing challenges obtaining financing for infrastructure projects, because they are already fully leveraged and have pledged their PFCs over the mid- to long-term. For example, officials from a small hub airport said that they obtained $120 million in financing, which will be carried until 2040, to build a parking garage and concourse. They said that because the airport is at capacity for debt issuance, they cannot take on any new debt for additional infrastructure projects. FAA data show that as of August 2019, 117 airports (about 30 percent) have obligated their PFCs past 2030 and that 30 airports (about 8 percent) have obligated their PFCs past 2040. One airport has obligated its PFCs through 2070. While some airports we spoke to raised concerns about being able to use debt financing for future airport-infrastructure projects, representatives from two rating agencies that we spoke to stated that for the airports they rate, the bond market is currently favorable, allowing for easier and economical access to financing. Rating representatives stated that currently, the outlook for domestic airports is either stable or positive due to the fact that airport passenger traffic growth has exceeded the gross domestic product’s growth, and airport ratings have remained consistent. For example, according to one rating agency, since 2012, its airport ratings have remained consistent and the annual airport outlook in those years has been “stable” or “stable to positive.” FAA officials added that while the perspective of rating agencies, bond insurers, and underwriters are important, a favorable credit rating does not mean that an airport should make the decision to take on additional debt. Moreover, according to FAA officials, for airports that need airline approval to issue debt, a favorable credit rating may not be sufficient to persuade the airlines of the need for the additional investment. Officials from 13 airports we spoke to stated that they are taking several actions to address funding challenges. These airport officials stated that they have deferred or delayed infrastructure investments, completed projects in phases in order to be able to fund projects in stages, or are looking for other ways to generate airport revenues from passenger services or leases. For example, officials from one airport we spoke to stated that their airport has developed a strategy of breaking up infrastructure projects into phases so as to utilize available FAA funding. According to these airport officials, this strategy lengthens the construction time and results in higher construction costs, but helps the airport to align its project needs with available FAA funding. Another airport official we spoke with said that the airport is introducing a dynamic- pricing parking program to generate additional parking revenue and that the program is expected to bring in an additional 5 to 15 percent in parking revenue. Officials from about half (11 out of 19) of our selected airports stated that AIP’s eligibility funding criteria are too narrow and do not allow airports to fund the infrastructure projects that they currently need, such as terminal projects. FAA’s AIP handbook provides guidance on the criteria to determine which components of a project are eligible for AIP funding. AIP-eligible projects, outlined in statute, include airport planning, airport development, noise compatibility planning, and noise compatibility projects. Certain airport projects, such as revenue-producing parking facilities, hangars, revenue portions of terminals, off-airport roads, and off-airport transit facilities are not eligible for AIP funding. Some terminal projects, however, are eligible for AIP funding, such as a terminal structure shell’s development and development of public use areas directly related to the movement of passengers and baggage in terminal facilities within an airport. This eligibility includes public use spaces that passengers may need to occupy as part of their air travel or utility support space needed to make the public space operational, including the mechanical and electrical rooms. Four airport officials we spoke to stated that they have infrastructure projects planned that are eligible for AIP discretionary funding, but that due to FAA’s criteria for AIP discretionary funding and FAA’s process for prioritizing projects, it is difficult for airports to receive discretionary funding for these projects. According to FAA officials, the eligibility criteria for AIP projects funded through entitlement and discretionary funding are the same. Discretionary funding, however, has some additional restrictions. For example, large, medium, and small hub airports are not eligible to use discretionary funding for terminal building projects. General aviation airports, however, may use discretionary funding for some airport terminal projects. In addition, unlike with entitlement funding, discretionary funding is not reimbursable and airports cannot apply for discretionary funding for projects that have already begun construction. In addition, unlike entitlement funding, not all airports receive discretionary funding. Airports must compete for the limited amount of discretionary funding available each year based on FAA’s AIP prioritization. According to FAA officials, while discretionary funding criteria do not change year to year, FAA may fund projects with discretionary funding one year, but a similar project may not receive discretionary funding a different year due to the project mix that year. FAA officials also stated that in September 2019, FAA updated its Formulation of the NPIAS and Airports Capital Improvement Plan order, which lays out the criteria and prioritization process for selecting projects for discretionary funding. According to FAA officials, projects with the highest priority include safety- and runway-related projects, such as runway signage or resolving complex geometry causing runway incursions. FAA officials stated that other projects have lower priority and ranking in the AIP discretionary-funding prioritization process. Below are examples from airport officials that stated they have certain projects planned that are eligible for AIP discretionary funding but that they believe will likely not rank high in FAA’s prioritization: Non-airfield projects. According to officials from a large hub airport we interviewed, the airport has made several investments in their airfield in the last few years and does not have any major airfield projects planned. These officials stated that they do have several non- airfield projects planned that are AIP-eligible, such as renovating gate holding areas in the terminal. However, airport officials stated that non-airfield projects do not compete well for AIP discretionary funding based on FAA’s prioritization process. As a result, they do not anticipate that they will receive AIP funding for these projects. Airfield projects. Similarly, airport officials from one medium hub airport explained that some of the airfield projects that they have planned, are eligible for AIP discretionary funding, but are not considered “high priority” projects according to FAA prioritization criteria. For example, they currently have a taxiway and apron upgrade project planned, but this project may not compete well against other projects when considering FAA’s AIP prioritization process. According to this airport official, runways are the highest priority and almost always get AIP funding. The official added, however, that the farther away you get from the runway, the less likely it is that you will be able to get AIP funding for the project. In addition, five airport officials noted that while overall AIP grant-funding levels have remained relatively constant in recent years, demand for discretionary AIP grant funding has increased, thereby increasing competition for this funding. According to FAA officials, the amount of funding that FAA has available for discretionary grants changes year-to- year. For example, the amount of discretionary funding allocated to airports annually can vary based on an airport’s decisions to carry entitlement funding over multiple years, as entitlement funding that is carried over becomes discretionary. According to FAA officials, because a very high percentage of discretionary funding comes from funding that has been carried over, it is difficult for airports to plan for or count on this funding being available in any given year. Between fiscal years 2013 through 2017, the amount of discretionary funding that was awarded averaged $1.6 billion annually. Of this amount, the amount representing “pure” discretionary funding averaged $56 million annually or about 4 percent of total AIP discretionary funding. Pure discretionary funding refers to the amount remaining after discretionary set-asides have been funded. FAA distributes pure discretionary funding to eligible projects at any airport on a competitive basis. As previously discussed, an additional $1 billion in supplemental discretionary AIP funding was appropriated in 2018, and an additional $500 million was appropriated in discretionary AIP funding in 2019. However, according to FAA officials, the number of applications they received for this funding exceeded the amount of funding that was available. Specifically, according to officials, FAA received more than 2,500 funding requests totaling more than $10 billion in 2018 for the $1 billion authorized as supplemental discretionary AIP grant funding. As of May 2019, FAA has awarded or anticipates awarding $985 million in supplemental discretionary AIP grant funding to 164 airports in 50 states, the District of Columbia, and Puerto Rico. The supplemental grants fund projects ranging from runway reconstruction and rehabilitation, to the construction of taxiways, aprons, and terminals. About half (12 out of 19) of the airport officials we spoke to stated that competing airport and airline priorities for capital infrastructure investments can pose challenges to funding infrastructure projects and can delay projects. For example, some of these officials stated that if an airline does not agree that there is a business case or that an infrastructure project is justified, then that lack of agreement can affect the airport’s ability to fund the project or delay the project altogether. The extent to which airlines are involved in the decision-making of airport infrastructure investments varies by airport and depends on the type of “use-and-lease” agreement between the airport and the airline. These agreements set forth the terms and conditions for establishing airline rates and charges and investing in capital improvements. Some agreements have a “majority-in-interest” (MII) provision, which requires airports to obtain airlines’ approval for certain infrastructure investments. One large hub airport stated that they have an MII agreement, requiring airlines’ approval of certain projects and project financing strategies. They further explained that debt financing would affect their airline rates and charges and would therefore require the airport to obtain approval from airlines before using general airport-revenue bonds on a project. While airport officials would like to add more gates to the airport and finance that project with general airport revenue bonds, these officials stated that some airlines may not support unassigned gate additions because it could increase competition. According to FAA officials we spoke with, some airports have been able to move toward shorter-term agreements with greater flexibility to adapt to changing needs; however, many agreements still include some form of MII provisions. According to officials from four smaller airports, airlines are less likely to support infrastructure-related increases in airline rates and charges at smaller airports than at larger airports. For example, a non-hub airport stated that smaller airports have a more difficult time negotiating higher rates and charges with airlines because of competition from other nearby airports. ACI-NA representatives also stated that medium hub airports that are not connecting hubs for the three large U.S. network airlines have less of an opportunity to receive capital investments from network airlines compared to larger airports. Representatives from all eight airlines that we spoke to stated that the types of airport infrastructure projects that they see a need for are demand-driven infrastructure development projects that expand airfield capacity, increase the number of gates at an airport, or address safety. Of these airlines, six also stated that they see a need for infrastructure development at larger airports in particular. For example, representatives from one airline stated that they want to collaborate with airports on capital development projects that are scalable and where passenger enplanements are increasing. In addition, representatives from five airlines that we spoke to said that they would like to have more input on airport infrastructure investment decisions. In addition, representatives from five airlines raised concerns that airlines do not have a role in decisions on how airports can invest PFC revenues. According to our prior work, PFCs provide airports a source of funding for airport development over which they have greater local control and airlines have more limited say in their use as compared to the use of airport terminal rents or landing fees. In addition, representatives from two airlines we spoke to said that FAA exercises limited oversight of infrastructure projects funded by PFCs, and that this limited oversight results in airports’ using PFC funding for projects that airlines do not see a need for. The representatives stated that FAA largely approves most PFC applications for projects and that they believe FAA should do more to ensure that airports are not using PFC revenues for unnecessary capital development not supported by airlines. For example, one airline objected to the use of over $1.5 billion of PFC funds for the multi-phase construction of the Phoenix Sky Train linking light rail, parking, and terminals, as representatives believed that there was not an adequate business case to justify the construction of the Sky Train. According to these airline officials, because the airport used PFC revenues for the project, other necessary terminal improvements have been largely debt funded. According to FAA officials, when reviewing PFC applications, they assess the extent to which the airport has demonstrated a need for the project. FAA officials stated that airports are familiar with FAA criteria and will generally not submit projects that will not meet the criteria and that could be denied. In addition, FAA officials stated that while it is unusual for FAA to deny an application, they have denied individual projects. We provided a draft of this report to the Department of Transportation (DOT) for review and comment. DOT provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, DOT, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-2834 or krauseh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Traditionally, airports around the world were primarily owned and managed by national governments, but that has changed over time. Beginning in the 1980s and through the 1990s, governments outside of the United States began shifting toward privatization and deregulation of airports. According to the 2016 Airports Council International - World’s (ACI-World) inventory of privatized airports worldwide, 614 commercial service airports (14 percent) have private sector participation. Although ACI-World estimates that a majority (86 percent) of the 4,300 airports with scheduled traffic around the world are publicly owned by a government or government entity, airports with private sector participation handle over 40 percent of all global air traffic. Today, there is a range of airports’ ownership and operating models. Through a literature review of ACI-World’s, the Airports Council International - EUROPE’s (ACI- EUROPE), the International Civil Aviation Organization’s, and the International Air Transport Association’s reports and other documents, we identified five general types of airport ownership structures outside of the United States: Government owned and operated: The airport is fully owned and operated by a public authority or by a mixture of public authorities at a local, regional, national, or transnational level. Government owned and privately operated: The airport is government owned but the airport operator—considered as the entity that is responsible for the day-to-day operation of airport services and facilities—is a private company. Partially privatized: The airport is partially privatized (e.g., mixed public-private ownership), meaning the airports’ shares are owned by a combination of private investor(s) and public authorities of the country where the airport is located. Fully privatized: The airport is fully owned and operated by a commercial company wholly owned by private individuals or enterprises. Not-for-profit, private corporation: The airport has been transferred to or leased by a not-for-profit corporation. The not-for-profit corporation is expected to be financially self-sufficient and fully responsible for funding all operating and infrastructure costs. While U.S. airports are predominantly publicly owned and operated, private participation, like private ownership or private operation contracts, is more common at airports in other countries. Airport Ownership in the United States In the United States, nearly all of the 3,330 commercial-service or general-aviation airports, designated as part of the national airport system, are publicly owned by local and state governments, regional airport authorities, or port authorities. Airport ownership in the United States has evolved under a public model since the 1920s as a way to promote the development of the U.S. aviation industry. In 1996, the Federal Aviation Reauthorization Act of 1996 established the Airport Privatization Pilot Program, which reduced some of the barriers to privatizing airports, and allowed for commercial service airports to be leased and for general aviation airports to be sold or leased. However, as we have previously reported, 18 years following the program’s inception, two airports have privatized, with one of these airports reverting to public control. While participation in the Airport Privatization Pilot Program has been very limited, some airports have entered into public-private partnerships with private entities through management contracts for terminals, which may be leased or outsourced to airlines or other contractors, or for food, rental car, and other concession agreements. For example, the Paine Field Snohomish County Airport in Washington, previously a general aviation airport, entered into a ground-lease agreement with a private airport developer—Propeller Airports—to build and operate a small passenger terminal for commercial service. The terminal was open for commercial service in March 2019, and is depicted in figure 9. Propeller Airports is responsible for the landside infrastructure investments and terminal maintenance. Snohomish County is responsible for maintaining and operating the airside infrastructure, which includes the runways and taxiways, but leases the aprons and the terminal land to Propeller Airports. Airport Ownership in Other Countries Privatized airports are more prevalent in foreign countries. According to a 2016 report by ACI-EUROPE, which examined ownership structures of airports across Europe, about 41 percent of European airports are fully privately owned or partially privatized. According to ACI-World, 75 percent of airports with passenger traffic in Europe have private sector involvement through fully privatized airports or public-private partnerships. Latin America-Caribbean airports (60 percent) and Asian airports (45 percent) have the second and third highest private sector involvement. Industry stakeholders we interviewed said that in some Asian countries, such as Japan and Singapore, airports that were previously government owned have already privatized or are transitioning to privatization. In addition, while ownership models can vary by country, they can also vary within a country. For example, according to ACI-EUROPE’s 2016 report, the United Kingdom’s airports are 53 percent fully private, 26 percent partially privatized, and 21 percent fully public. As we have previously reported, different airport ownership structures, motivations, and financing have driven airport privatization in other countries. For example, in several countries, the national government built, owned, and operated the country’s airports prior to privatization. We previously reported that national ownership enables a central government to direct the sale of its airports and can make for a more streamlined privatization transaction, reducing transaction costs for both the public- sector owner and private-sector bidders. Foreign governments may also be more motivated to privatize their airports than U.S. public-sector airport owners. According to the International Civil Aviation Organization, foreign governments’ reasons for privatizing their airports vary, including an identified need for private-sector capital investments in existing or new airports and a national move toward privatization of public assets or companies. We have previously reported that airports in other countries often have less access to public funds or tax-exempt bonds than publicly owned and operated U.S. airports, making them more reliant on private financing for airport improvements. Our prior work found that a key factor that can hinder U.S. airport privatization is the loss of some federal AIP funds and the loss of easy access to tax-exempt financing. Most of the five foreign airports we selected for our review do not receive government funding. We selected and reviewed five airports in other countries that represent each type of ownership structure previously discussed. Representatives from our five selected foreign airports all said that they rely on aeronautical revenue, which includes revenue from passenger charges and airline rates and charges as the primary source for capital development. Representatives from four of our selected foreign airports said that they rely on debt financing for infrastructure funding as well. Representatives from only one selected airport, Changi Airport in Singapore, said that they have received government funding for infrastructure projects. Table 3 below summarizes the main sources of infrastructure funding available to these selected airports. Representatives from the selected airports we interviewed said that they generate infrastructure funding from various sources of aeronautical revenue, including airline rates and charges. Some foreign airport representatives told us that revenue from airline rates and charges are not required to be used for aeronautical-related costs or infrastructure, or within the airport. Some airports, such as Helsinki Airport, may operate within a consortium network, where revenue is shared among all airports in the network to cover costs. Additionally, some airports have regulations for setting airline rates and charges. For example, the Civil Aviation Authority in the United Kingdom regulates Heathrow Airport’s airline rates and charges. Selected airport representatives we spoke with said that they consult airlines when adjusting airline rates and charges. For example, the Helsinki Airport official said that the airport updates its airline charges once a year and that airlines have an opportunity for the airlines to appeal the change. Representatives from the International Air Transport Association and the Steer Group Inc. said that some foreign airports may have higher airline rates and charges compared to some airports in the United States due to several factors, including the need to generate returns for private financing and flexibility in setting rates and charges, as outlined below. Generating returns for private financing. Foreign airports with private investment or financing may have higher rates because they need to generate returns to pay back private financing. Privately owned airports may also be under pressure to generate returns for investors and therefore need to further divert revenue from funding infrastructure. Flexibility in setting rates and charges. Foreign airports generally have greater flexibility to set airline charges to meet airport needs, a flexibility that may result in higher rates and charges. For example, Canadian airports are generally able to set and adjust airline and passenger charges as needed, and charges vary by airport. In Singapore, Changi Airport has a passenger charge and a pre-funding levy for its new terminal project. Airports in the United Kingdom, including Heathrow Airport, have a regulator that sets the airline and passenger charge cap, and adjusts it every 2 years. In addition, foreign airports have limited airline input on determining airport capital investments and fees charged to airlines. For example, according to ACI-WORLD, airports consult airlines on airport charges and on capital developments, but airport proposals can usually be implemented even if airlines do not support them, as long as a due and proper consultation process is held. An international airline stakeholder said that the extent of airline input on airport capital investment and fees charged to airlines is dependent on the country’s specific regulatory model and the willingness of the airport operator to consult with airlines, but that in some countries, airline consultation is limited. Representatives from our selected foreign airports said they generally keep airlines informed. For example, the Toronto Pearson International Airport has a consultative committee approach with airlines on larger projects costing over $50 million. If the airlines do not approve a project through the consultative committee, the project must be put on hold for one year before it can proceed. Other sources of aeronautical revenue include passenger charges. As of October 2019, for the foreign airports we reviewed, passenger charges ranged from the U.S. dollar equivalent of $9.65 to $58.58 per local traffic passenger (see table 4). Industry stakeholders and international airport association stakeholders said that U.S. airports have a unique ownership and funding model compared to foreign airports. U.S. airports have an element of public control of funding through the federal Airport Improvement Program (AIP) grants and passenger facility charges (PFC), as projects funded through these sources must receive approval from the Federal Aviation Administration. According to these stakeholders, U.S. airports are subject to different regulations related to setting passenger charges. As a result, we have determined that the comparability of these charges is limited. In addition, differences in ownership models, private investment, and funding between U.S. and foreign airports also limited the comparability of these charges. Table 4 provides an overview of passenger charges and levies at selected airports in other countries. Selected foreign airports adjust passengers’ charges based on the airport’s building and infrastructure needs and the cost imposed by passengers on the airport system. How and when these airports make adjustments varies. For example, one of our selected airports has a government entity that regulates passenger charges. More specifically, the Civil Aviation Authority in the United Kingdom regulates Heathrow Airport’s passenger charges. Every 5 years, the Civil Aviation Authority determines the maximum amount that the airport can charge based on the costs incurred by the airport. Other selected airports consider adjustments on an “as needed” basis, including the Toronto Pearson International Airport. Representatives from the Toronto Pearson International Airport said that they set and adjust passenger charges as needed to fund infrastructure investments. The airport assesses charges annually and only adjusts the passenger charges if there is a material imbalance between required cost recoveries against charges. Airport officials also stated the airport increases airline rates and passenger charges only when needed to generate sufficient revenue to cover the costs of planned infrastructure. Similar to airline rates and charges, selected foreign airport representatives told us that there generally are no restrictions on how the airports use revenue from passenger charges for infrastructure or operational costs. Industry stakeholders said that some airports, such as Heathrow Airport, do not have revenue diversion limitations, so revenue generated from passenger charges at the airport is not required to be reinvested back into the airport. Comparatively in the United States, airport revenue is regulated and generally speaking, revenue generated by the airport must go toward certain costs at the airport. Most of our selected foreign airport representatives (4 out of 5) also said that they rely on debt financing, through private bonds or commercial loans. Industry stakeholders said that airport debt financing internationally is similar to that in the United States, but foreign airports generally do not have access to the municipal bond market. Airports’ bonds are generally tax exempt in the United States. Representatives of our selected foreign airports said that they use various types of debt financing, including commercial loans from financial institutions; equity or debt financing, such as bonds in commercial capital markets; or loans from private investors. Most of our selected foreign airports (3 out of 5) do not receive government funding. International airport associations said that the extent to which an airport receives government funding may depend on whether the government owns the airport or has a role in operating the airport. For example, Changi Airport officials said that the Singaporean government is providing Changi’s government-owned, privately operated airport an unspecified amount of government funding for their new Terminal 5 project. In another example, Toronto Pearson International Airport does not receive government funding; however, in Canada, small or rural airports can receive some funding from the Canadian Airports Capital Assistance Program. Similarly, Finavia officials said that although the Helsinki Airport is publicly owned and operated, it does not receive any government funding. To provide information about how each of the five, selected foreign airports fund and finance infrastructure projects, we developed the following case studies. These airports were selected based on selection criteria of ownership models and passenger traffic. The case studies provide information on main sources of funding and financing for the airports’ infrastructure developments, factors considered when setting airline and passenger charges, coordination with airlines on capital development, and recent and planned infrastructure investments for each selected foreign airport. The majority of airports in Finland are owned and operated by the government-owned company Finavia Corporation (Finavia), a limited liability company wholly owned by the Finnish government. Specifically, Finavia operates a network of 21 Finnish airports, of which 19 offer commercial service and two are military airports. Of the remaining three airports in Finland, two are owned by local municipalities, and one is privately held. Background Helsinki Airport is owned and operated by a government-owned company, Finavia. Of Finavia’s airports, according to the Finavia representative, Helsinki Airport has the most connecting international flights and passenger boardings. For example, Helsinki Airport provides direct service to 162 international destinations, including 22 direct flights to Asia. Helsinki Airport has experienced strong passenger growth in recent years. In 2018, Helsinki Airport had 21-million passenger boardings, an increase from the prior year of about 10 percent. Most of this increase was attributable to international traffic. The Finavia representative said that Finavia anticipates passenger traffic to slow in 2020, due to an anticipated slowdown in Europe’s economic growth. Main Sources of Funding and Financing for Airport Infrastructure Investments According to the Finavia representative, Helsinki Airport’s main sources of funding for infrastructure improvements are (1) airline rates and charges, (2) passenger charges, (3) other airport-generated revenue, and (4) debt financing. Helsinki Airport collects aeronautical revenue from airline rates and charges and passenger charges directly from the airlines. Helsinki Airport does not receive any public or government funding, despite being government owned, and the airport does not have any public-private partnerships. In 2010, Finavia began operating as a limited liability company, rather than a government agency. The Finnish government corporatized Finavia to align with the European Union (EU) principles on EU services, movement of services, and competition. The Finavia representative said that the change in corporate structure helps ensure that the government is not subsidizing or promoting unfair competition practices. Airline rates and charges: Helsinki Airport generates revenue from air carrier and other aircraft operator rates and charges such as landing, aircraft parking, and electricity charges. In 2019, Finavia raised airline charges by 2.1 percent from 2018 levels, prompted by higher service costs resulting from airport investments. The Finavia representative said that airline rates and charges make up approximately 40 percent of the airport’s total aeronautical revenue. Under the airport network approach, Finavia can offset losses at one airport with revenue from a more successful airport. The Finavia representative said that some airports in the network are self- sustaining and generate sufficient revenue to cover the costs of airport operations; other network airports do not. According to the Finavia representative, Finavia applies uniform airport charges within the airport network to recover operational and infrastructure costs Passenger charges: Helsinki Airport collects a passenger charge from airlines in order to fund infrastructure used for servicing the passengers. As of January 2019, Helsinki Airport has a euro (€) 8.60 (U.S. dollar (USD) $9.65) fee per departing passenger and a across the airport network and to comply with EU directives on airport charges. €4.10 (USD $4.60) fee per transferring passenger. The Finavia representative said that passenger fees make up approximately 60 percent of the airport’s total aeronautical revenue, which include both airline and passenger fees. According to the Finavia representative, Helsinki Airport does not designate revenue from airline and passenger charges for a specific use. Revenue from airline and passenger charges has been used to cover costs from providing services and operations within the Finavia network. According to the Finavia representative, aeronautical charges, including airline rates and charges and passenger charges, are evaluated and updated once a year and Finavia sets the same charges for all airports in the Finavia airport network. Other airport-generated revenue: Helsinki Airport also generates non-aeronautical revenues from sources such as concessions, commercial services at terminals, parking services, security control, and rental income from real estate. Debt financing: Helsinki Airport uses debt financing from a variety sources, including private banks, financial institutions, and public sector sources such as the European Investment Bank, a financing institution financed by the European Union, and the Nordic Investment Bank. The financing that Helsinki Airport has obtained is similar to traditional debt financing. According to the Finavia representative, Helsinki Airport does not have any restrictions or legal requirements on the types of loans that the airport can take on, nor does Finavia pledge revenue from any specific source towards the repayment of loans. However, the Finavia representative stated that Finavia does not issue bonds. The representative said that generally, the airport has relied on traditional lending because it is easier to obtain and repay a bank loan as compared to other types of debt. Factors Considered when Setting Airline and Passenger Charges The representative said Finavia considers several factors when setting airline and passenger charges. The Finnish Act on the Airport and Network and Airport Charges requires that the pricing of airport charges within the airport network are uniform, common, and transparent, based on the service level offered, and are applied on non-discriminatory and equal grounds. Finavia therefore considers the Finavia airport network revenue; the cost of providing aeronautical services (including operational and electricity costs); and the costs of capital for infrastructure investment when setting the airport’s airline rates and charges. According to the Finavia representative, Helsinki Airport also considers the airport market to ensure that its airline and passenger fees are competitive with similar airports in other European countries. When Finavia makes changes to its airline or passenger charges, the Finavia representative said that airlines have an opportunity to appeal the change. The Finnish Transport and Communications Agency acts as an independent supervisory authority to process disagreements on airport charges. Coordination with Airlines on Capital Development As part of the capital development process, Finavia must consult with airlines to seek input on planned capital investments at the airport before the airport carries out any major new infrastructure projects. Finavia organizes these discussions to assist with negotiations, but the Finavia representative said these discussions are specific to the individual airport rather than the overall Finavia network. In addition, according to the Finavia representative, when setting airline and passenger charges, Finavia consults with airlines and provides information about how airport charges relate to the facilities and services at the airport. According to the Finavia representative, the Helsinki Airport development program, initiated in 2014 with a 2030 anticipated completion date, is the largest expansion project in the airport’s history. It will expand Helsinki Airport’s capacity and increase the number of gates. For example, the airport has planned a terminal building project that will expand the terminal by 45 percent and double the number of gates for wide-body aircraft from eight to 16 gates. In 2016, as seen in figure 10, Helsinki Airport opened one of the passenger terminal expansions, which added 12 new departure gates to the airport. On the airside, the airport will also renovate the apron area to accommodate large aircraft. Additionally, Helsinki Airport is working on a project to improve luggage and baggage handling capabilities to accommodate the anticipated increase in baggage volume expected from airlines’ use of larger aircraft. According to the Finavia representative, Helsinki Airport planned these capital improvements in response to expected passenger traffic growth. The representative anticipates that between 2025 and 2030, annual passenger boardings at Helsinki Airport will reach 30 million. A rendering of the entrance to Helsinki Airport’s completed terminal expansion is shown in figure 11, below. Finavia will use airport cash flows from passenger fees, aeronautical revenue, and non-aeronautical revenue to fund the infrastructure projects. Finavia estimates that the total cost of the Helsinki Airport infrastructure expansion will be €1 billion (USD $1.1 billion). Singapore has two airports that provide commercial service—Changi International Airport (Changi Airport) and Seletar Airport, which is a smaller airport that provides commercial and general aviation service. Background Changi Airport is the primary commercial airport in Singapore, located off the eastern coast of the country. Changi Airport was built in 1981, and according to ACI-World, was the world’s 19th busiest airport in terms of passenger boardings in 2018. While Changi Airport is government owned, the airport is operated by the Changi Airport Group—a private limited company. The Changi Airport Group is responsible for the airport’s operations and management, air hub development, commercial activities, and airport emergency services. It is also responsible for maintaining and investing in airport infrastructure and ensuring the airport is financially self-sustaining. Both airports are owned by the Singapore Ministry of Finance and operated by the Changi Airport Group. The Singapore Ministry of Finance does not have a role in the daily operations and management of the airports but reviews the types of planned airport infrastructure investments. The Changi Airport Group’s board of directors is made up of two representatives from the Singapore Ministry of Finance and other board members from the private sector. The board has discretion to design, budget, and build infrastructure projects. Changi Airport is a major hub for the region, and according to the Changi Airport Group representative, passenger boardings have been increasing steadily. For example, from 2005 to 2018, boardings increased by 30 percent. In 2018, the airport had 66.6-million boardings, an increase of about 5.5 percent from the prior year. The Changi Airport representative said that the airport is currently operating at 85 percent capacity for passenger boardings, but anticipates reaching 100 percent capacity by approximately 2026–2027. The airport has made significant investments to enhance the passenger experience at the airport. For example, the airport has enhanced terminal features for passengers, including a butterfly garden, indoor waterfalls, a four-story slide, 19 airport lounges, and luxury shopping (see fig. 12). The 2019 World Airport Awards named Changi Airport the World's Best Airport for the seventh consecutive year. From 1984 until 2009, Singapore’s airports were owned by the Singaporean government, and operated by the Civil Aviation Authority of Singapore, under the Ministry of Transport. In 2009, the airports were corporatized, and the Changi Airport Group took over airport operations and management. Through two companies, Temasek Holdings and GIC Private Limited, the Ministry of Transport owns and invests in companies that serve strategic national interests, such as infrastructure. For example, according to the Changi Airport representative, Temasek has a 50 percent stake in much of Singapore’s major infrastructure, including a 54 percent stake in Singapore Airlines, the country’s national carrier. The Civil Aviation Authority of Singapore continues to economically regulate the Changi Airport, promote the growth of the air hub and aviation industry in Singapore, oversee and promote aviation safety, and provide air navigation services. Frankfurt Airport’s Case Study Passenger traffic: 222 million Number of airports: 36 commercial airports which includes 16 international airports and 20 regional airports with scheduled passenger service. Background Frankfurt Airport began operations in 1936. According to Fraport AG’s 2018 annual report, in fiscal year 2018, Frankfurt Airport was the largest commercial service airport in Germany and the fourth largest commercial service airport in Europe. The airport is partially privatized and is owned and operated by Fraport AG. Frankfurt Airport was previously jointly owned by the federal government, the State of Hesse, and the City of Frankfurt. In June 2001, Frankfurt Airport was partially privatized, with private entities acquiring a minority ownership stake in the airport. Currently, the State of Hesse and City of Frankfurt own about 51 percent of the airport, with the remaining, about 49 percent, held by private entities. Until the 1980s, airports in Germany were traditionally owned and operated by the government. Following the 1982 creation of a federal program to privatize airports, several airports were partially privatized. According to an Airports Council International-EUROPE survey conducted in 2015, there are now two different airport ownership structures in Germany. Passenger traffic at Frankfurt Airport has increased over the last few years. According to Fraport AG’s 2018 annual report, Frankfurt Airport reached 69.5 million passengers in 2018—an increase of 5 million passenger or about 8 percent over the prior year. Partially privatized: about 47 percent of airports in Germany are partially owned by local, regional, or federal governments. Main Sources of Funding and Financing for Airport Infrastructure Investments Frankfurt Airport’s main source of funding for capital improvements are (1) airline rates and charges, (2) passenger charges, (3) other airport- generated revenue, and (4) debt financing. Airline rates and charges: Frankfurt Airport collects revenue from Fully government owned: about 53 percent of airports in Germany are owned by a public authority, or by a mixture of public authorities, at a local, regional, national, or transnational level. airline rates and charges paid by airlines servicing Frankfurt Airport. These charges include airline takeoff and landing, noise, parking, and other charges. Under German law, airports must obtain approval for certain airline rates and charges from the regional aviation authority, including airline takeoff and landing charges, noise charges, aircraft movement area charges, and parking charges. The only airport charges not subject to approval are charges for central ground-service infrastructure facilities and ground service charges. The regional aviation authority responsible for Frankfurt Airport is the Ministry of Economics, Energy, Transport and Regional Development, State of Hesse. In addition, the Airports Council International (ACI)- EUROPE’s representatives said that the majority of airports in Europe with commercial service, including Frankfurt airport, offer discount incentives to airlines in exchange for delivering higher volumes of passengers. Passenger charges: Frankfurt Airport has passenger charges that vary depending on the destination of the passenger’s flight. As with airline rates and charges, airports must also obtain approval for passenger charges from the regional aviation authority. For example, as of January 1, 2019, these charges range from euro (€)12,93 (U.S. dollar (USD) $14.51) for transfer flights to all destinations to €25,16 (USD $28.23) for international flights initiating from Frankfurt Airport. Other airport-generated revenue: Frankfurt Airport also generates revenue from airport concessions, real estate leases, parking, and other sources. Debt financing: Frankfurt Airport also relies on debt financing to fund infrastructure projects. However, we were unable to receive data from Fraport AG on how much debt financing Frankfurt airport used for capital development projects in 2018. We were not able to confirm financial information with Fraport AG about how much total revenue Frankfurt Airport generated from each of the individual sources described above. Therefore, we are not able to provide information on the total revenue generated by Frankfort Airport in 2018. However, information is available on the total revenue for all airports in the Fraport AG network. Specifically, according to Fraport AG’s 2018 annual report, the total revenue generated from approved airline rates and charges, passenger charges, and passenger services combined for the full Fraport AG group was €1,006 million (USD $1.2 million). In addition, the total revenue generated from other airport-generated revenue for the full Fraport AG group was €507 million (USD $599 million) in 2018. Fraport AG is in the process of building a new terminal—Terminal 3—at Frankfurt Airport to provide sufficient capacity and accommodate growing air traffic at Frankfurt airport. Construction for the project began in 2015 and is estimated to be completed in 2023. The first phase of the project involves construction of the main terminal building, which will include the arrival and departure levels, lounges, concession area, and a baggage handling system. This phase of the project is expected to provide capacity for about 14-million passengers a year. The second phase of the project will expand the airport facility and is expected to increase passenger capacity by up to 5-million additional passengers when completed in 2021. According to Fraport AG’s current plans, the new terminal is expected to increase capacity by up to 21 million more passengers. Fraport Ausbau Süd GmbH, a wholly owned subsidiary of Fraport AG, is responsible for managing, supervising, and monitoring the construction project. The project is being privately financed, and the estimated budget of the project is about €3.5 billion to €4 billion (USD $4.1 billion to $4.7 billion). According to Fraport AG, this project is Fraport’s largest single investment at Frankfurt Airport. We were unable to confirm information with Fraport AG representatives about factors they consider when setting airline and passenger fees or how they coordinate with airlines on the airport’s infrastructure development. Background Heathrow Airport is Europe’s busiest airport with the highest passenger boardings, and is the United Kingdom’s hub airport. Heathrow Airport has undergone transformation from a government-owned airport to a privately-owned airport. Heathrow Airport was privatized in 1987 as part of the privatization of the British Airports Authority. Currently, Heathrow Airport Holdings Limited owns and operates Heathrow Airport. In 1965, the Airports Authority Act established the British Airports Authority, an independent government agency, which assumed ownership and management of airports in the United Kingdom. Between 1966 and1987, the British Airports Authority acquired ownership and operation of seven of the 22 government airports— Heathrow, Stansted, Prestwick, Gatwick, Edinburgh, Aberdeen, and Glasgow airports. Although Heathrow is privatized, any airline and passenger charges the airport collects are subject to economic regulation by the U.K.’s Civil Aviation Authority. The Civil Aviation Authority—a government agency— regulates airport charges for U.K. airports with more than 5-million annual passengers. Airports Council International (ACI)-EUROPE representatives said that the Civil Aviation Authority regulates Heathrow on the basis that Heathrow is likely to possess significant market power for aeronautical services. In 1987, the United Kingdom privatized the British Airports Authority due to limited government funding and a need for significant capital development at large airports, according to Heathrow Airport representatives and industry stakeholders. All seven airports owned by the authority were privatized. The authority was subsequently acquired in 2006 by an international consortium led by Ferrovial Aeropuertos S.A. of Spain (Ferrovial S.A.) and named BAA Ltd. This entity was later renamed Heathrow Airport Holdings Limited. The United Kingdom became the first country to privatize its major airports. According to an Airports Council International-EUROPE survey conducted in 2015, airports in the United Kingdom have one of the following three ownership structures: Government owned: about 21 percent of airports in the United Kingdom are owned by local, regional, or national governments; experienced increased passenger numbers as a result of airlines’ use of larger aircraft that have more seats per aircraft. Main Sources of Funding and Financing for Airport Infrastructure Investments Heathrow Airport’s main sources of funding for capital improvements are (1) airline rates and charges, (2) passenger charges, (3) other airport- generated revenue, and (4) debt financing. Airline rates and charges: Heathrow Airport collects revenue from Fully privatized: about 53 percent of airports in the United Kingdom are owned by private entities. charges that it imposes on airlines that fly to and from Heathrow Airport. These charges include landing, parking, and emissions charges. Under the authority of the Civil Aviation Act of 2012, the Civil Aviation Authority establishes a pricing formula known as the “maximum revenue yield,” which sets limits on the airline and passenger charges on a per-passenger basis. In 2018, Heathrow Airport generated pounds (£) 549 million (U.S. dollar (USD) $734 million) in landing and parking charges, according to Heathrow Airport’s 2018 financial statements. Passenger charges: Heathrow Airport has several categories of passenger charges, which vary in rates depending upon the time of year of travel; whether the passenger is on a departing, transfer, or transit flight; or whether the flight destination is inside or outside of the European Union. For example, under the 2019 charges for Heathrow Airport, the passenger service charge would range from £19.84 to £46.02 (USD $25.25 to USD $58.58). In 2018, Heathrow Airport generated £1.2 billion (USD $1.6 billion) in revenue from passenger charges, according to Heathrow Airport’s 2018 financial statements. Other airport-generated revenue: Heathrow Airport also generates other revenue from retail airport concessions, parking, and other sources. Heathrow Airport generated £656 million (USD $876 million) from these sources in 2018, according to Heathrow Airport representatives. Debt financing: Heathrow airport also relies on debt financing to fund infrastructure projects. In 2018, Heathrow (SP) Limited raised approximately £2.3 billion (USD $3.1 billion) of debt financing to fund infrastructure projects. According to Heathrow Airport representatives, as of 2018, the airport has a total debt of £12 billion (USD $16 billion), which includes shareholders’ indebted equity. According to Heathrow Airport representatives, Heathrow Airport’s largest source of funding is from airline rates and charges and passenger charges, and in 2018 the airport generated £1.7 billion (USD $2.3 billion) from airline and passenger charges combined. Factors Considered when Setting Airline and Passenger Charges As previously discussed, the Civil Aviation Authority is responsible for economic regulation of Heathrow and other airports in the United Kingdom. Specifically, it regulates airline and passenger charges and determines the maximum amount in fees that Heathrow Airport can charge airlines and passengers on a 5-year basis, with adjustments every 2 years as needed. The level of airport charges that Heathrow levies each year is in accordance with the aviation authority’s pricing formula. Each year, Heathrow Airport publishes Conditions of Use that describes its airport charges. According to Heathrow Airport representatives, they have flexibility in how they categorize charges, but the charges must align with the European Union’s and United Kingdom’s non-discrimination principle standards and with the Civil Aviation Authority’s regulations. According to Heathrow Airport representatives, they consider several factors, such as the infrastructure needs at the airport and the real cost of providing services, when setting airport charges. They also set charges to influence and incentivize airline behavior. For example, to incentivize airlines to replace aircraft with newer, less polluting models, the airport charges airlines a higher fee per landing when they use older aircraft. In addition, Heathrow’s passenger fees vary depending on the passenger’s anticipated airport use and with the costs imposed on the airport system. For example, passengers on domestic flights have lower charges than passengers traveling on international flights. This differential is because domestic passengers do not use the same facilities or the same baggage facilities as an international passenger and the costs of those facilities are higher than for facilities serving domestic passengers. Coordination with Airlines on Capital Development Heathrow Airport coordinates with airlines on capital development. For example, the airport organized an Airport Consultative Committee structure to obtain input on its most recent capital development plan from the 93 airlines operating at the airport. According to representatives from the International Air Transport Association, which is an association that represents airlines, the airport used this committee to reach agreement with these airlines on a capital expenditure plan related to development at multiple terminals at the airport. Recent and Planned Infrastructure Investments at Heathrow Airport According to Heathrow Airport representatives, within the last 15 years, Heathrow Airport has completed two large capital-development projects, and the airport is currently in a planning phase. In 2008, Heathrow Airport opened Terminal 5, which had a total project cost of £4.3 billion (USD $8 billion). Subsequently, in 2014, Heathrow Airport renovated its passenger terminal—Terminal 2—which cost approximately £2.5 billion (USD $4.1 billion) to complete. Planning and design is now under way for the construction of a third lateral runway and an associated new terminal facility at Heathrow Airport, according to Heathrow Airport representatives (see fig. 13). The new runway is intended to alleviate constraints on the number of available slots for landing and takeoff. According to Heathrow Airport representatives, the new runway is expected to add capacity for at least an additional 260,000 flights per year, and the overall project will expand the airport’s surface space by 50 percent. Representatives said that according to current plans, construction of the runway and associated terminal is expected to begin in 2022 and operations are expected to start in 2027. The runway project is estimated to cost £14 billion (USD $18 billon) and will be funded through cash flows from operations, equity, and debt, according to Heathrow Airport representatives. Background The Greater Toronto Airports Authority manages and operates the Toronto Pearson International Airport (Toronto Pearson). According to Statistics Canada passenger traffic data, Toronto Pearson is Canada’s busiest airport in terms of total passenger traffic. In addition, it is North America’s second busiest airport in terms of international traffic, according to Toronto Pearson’s 2018 annual report. The Greater Toronto Airports Authority is a not-for-profit corporation without share capital, meaning it does not have any shareholders and any profits earned are invested back into the airport. Until the early 1990s, the Canadian federal government owned, operated, and maintained most airports and air navigation facilities in Canada. In 1994, the Canadian federal government issued the National Airports Policy, which created different ownership structures for NAS and non-NAS airports. The Greater Toronto Airports Authority assumed operations and management of Toronto Pearson in 1996 through a lease arrangement with the federal government. According to representatives from the airports authority, because Toronto Pearson generates the most revenues among Canadian airports, the authority pays the highest ground lease rate for Toronto Pearson among Canadian airports. For every Canadian dollar (CAD) $1 (U.S. dollar (USD) $0.75) that the airport authority earns in revenue over CAD $250 million (USD $188 million), it pays CAD $0.12 cents (USD $.09) for the ground lease. For NAS airports, the National Airports Policy devolved responsibility for the operations, management and expenditures of NAS airports from the federal government to Canadian Airport Authorities, which were set up as not-for-profit and non-share corporations. The Canadian government, however, still owns these airports. Under the law, Canadian Airport Authorities pay lease payments to the government under 60-year leases that include an option to renew for 20 years. These airport authorities are required to invest airport-generated revenues in airport operation and capital development. Passenger traffic at Toronto Pearson has increased in recent years and representatives from the Greater Toronto Airports Authority stated that according to their projections, passenger traffic is expected to continue to increase. In 2018, about 48-million passengers traveled through Toronto Pearson—an increase of 2.4 million, or 5 percent, over the prior year. According to these representatives, about 70 percent of this traffic is from origin and destination passengers and 30 percent from connecting passengers. According to the airports authority’s forecasts, passenger traffic at Toronto Pearson is expected to increase to 85 million in 2037. By contrast, for non-NAS airports, the National Airports Policy transferred ownership of these airports from the federal government to regional or local entities, such as local municipalities. The government continues to support remote and Arctic non-NAS airports that service isolated communities. Main Sources of Funding and Financing for Airport Infrastructure Investments Toronto Pearson’s main sources of funding for capital improvements are (1) airline rates and charges, (2) passenger charges, (3) other airport- generated revenues, and (4) debt financing. Toronto Pearson does not receive any government funding, although some limited government funding is available to smaller airports through Canada’s Airports Capital Assistance Program. Airline rates and charges: Toronto Pearson collects revenue from airline rates and charges, which include landing fees, terminal fees for general use of the terminal space, apron fees, deicing facility fees, and other airline charges. According to representatives from the Greater Toronto Airports Authority, airline rates and charges at Toronto Pearson have not been increased since 2012. Toronto Pearson generated about CAD $510 million (USD $393 million) in airline rates and charges in 2018 according to Toronto Pearson’s 2018 annual report. Passenger charges: Passenger charges, called Airport Improvement Fees, are fees charged at every major Canadian airport and currently range from CAD $5 to CAD $40 (USD $3.76 to USD $30.12) per passenger. Each airport authority sets its own passenger fees, and there is no cap on how much each airport can charge. According to an international industry stakeholder, airport authorities, such as the Greater Toronto Airports Authority, set their respective fees based on their analysis of what the market can bear. Toronto Pearson’s passenger fee is CAD $25 (USD $18.82) for departing passengers and CAD $4 (USD $3.01) for passengers connecting through the airport as of January 1, 2019. The airport can only use this revenue for aeronautical-related expenses, such as capital development. The Greater Toronto Airports Authority has an agreement with each air carrier that takes off from and lands at Toronto Pearson whereby air carriers agree to collect passenger fees from each of their enplaned passengers on behalf of the authority. The airports authority commits in these agreements to use passenger-fee revenues for capital programs, including associated debt service. According to representatives from the Greater Toronto Airports Authority, the airport has not increased its passenger fees since 2012, as the increased volume of passengers has generated sufficient revenue for the airport. In 2018, Toronto Pearson generated CAD $460 million (USD $355 million) from passenger fees, in the form of Airport Improvement Fees, according to Toronto Pearson’s 2018 annual report. Other airport-generated revenues: Toronto Pearson also generates revenue from other sources such as airport concessions, rental properties, car rentals, parking, and advertising. The Greater Toronto Airports Authority has more flexibility in how it can use this category of revenue, including for operating costs and for capital needs. According to the Greater Toronto Airports Authority’s 2018 annual report, the long-term objective is to increase the proportion of total revenues generated through commercial streams at the airport—from non-aeronautical sources such as parking, retail, and dining concessions—to over 40 percent. In recent years, commercial revenues have been the fastest growing component of the airport authority’s revenues. In 2018, Toronto Pearson generated about CAD $502 million (USD $387 million) in other airport-generated revenue, according to Toronto Pearson’s 2018 annual report. Debt financing: Canadian airports can generally use equity or raise debt in capital markets. In 2018, Toronto Pearson obtained CAD $500 million (USD $386 million) in bond financing. According to representatives from the Greater Toronto Airports Authority, the authority issues bonds to fund existing bond maturities and capital programs that exceed cash from operations. Revenue from passenger fees, in the form of Airport Improvement Fees, are used to service debt for infrastructure projects. Projects that cost less than CAD $400 million (USD $301 million) are funded with passenger-fee revenues, airline rates and charges, and other airport-generated revenues, according to these representatives. Factors Considered when Setting Airline and Passenger Charges Representatives from the Greater Toronto Airports Authority stated that the structure that Toronto Pearson has in place allows the airport to increase airline rates and passenger charges only when needed to generate sufficient revenue to cover the costs of planned infrastructure. According to these representatives, charges are assessed annually, but only change if there is a material imbalance between required cost recoveries against charges. To establish airline rates and charges and passenger fees, the Toronto Pearson Airport uses the “dual till” model whereby airline and passenger charges are set to recover aeronautical costs only. This contrasts with the “single till” model where all airport activities (including aeronautical and non-aeronautical) are taken into consideration when determining the level of airport charges. Representatives from the Greater Toronto Airports Authority stated that Toronto Pearson is unique among Canadian airports in doing so. Coordination with Airlines on Capital Development As part of Toronto Pearson’s passenger-fee agreements with airlines, the Greater Toronto Airports Authority must consult with airlines and obtain approval for certain capital projects in excess of CAD $50 million (USD $38 million). Approval is sought through an airline consultation committee that the airport authority established to include representatives from airlines that provide service at Toronto Pearson. If the consultative committee does not approve a project, the airport must put the project on hold for 1 year. After the 1-year hold, the project may be initiated. According to representatives from the Greater Toronto Airports Authority, if the airport has a major capital project planned, the authority keeps the airline community informed. In particular, the airport communicates regularly with the two major Canadian airlines, which make up 70 percent of the airport’s service volume, to keep them informed of planned infrastructure improvements. Recent and Planned Infrastructure Investments at Toronto Pearson Airport In 2018, the Greater Toronto Airports Authority completed several infrastructure improvements at Toronto Pearson, according to Toronto Pearson’s 2018 annual report (see fig. 14). Some of these improvements relate to ongoing projects that the airport initiated in prior years. For example, the airports authority is upgrading and expanding its capacity at Terminal 1 to accommodate narrow-body aircraft operations in response to increased passenger traffic. During 2018, the authority expended CAD $16 million (USD $12 million) for this project. In addition, the airport expended about CAD $13 million (USD $10 million) in 2018 to make improvements at Terminal 3, which is intended to enhance passenger experience and improve passenger flow. The Greater Toronto Airports Authority also expended about CAD $23 million (USD $18 million) on Phase 1 of its baggage-handling improvement project, which will add baggage-handling capacity and is intended to improve system reliability. According to representatives from the Greater Toronto Airports Authority, the authority has developed a 5-year capital plan that includes several projects intended to increase capacity and improve passenger flow at the airport. For example, the airports authority has begun the design phase for construction of a new concourse at Terminal 1 and an expansion project at that terminal. The airports authority is also in the design phase for constructing an integrated Regional Transit and Passenger Centre, and replacement of the baggage systems. The airport also plans to add more retail space and provide U.S. Customs and Border Protection space in the terminal to reduce international passengers’ connecting time by improving passenger flow. According to representatives from the Greater Toronto Airports Authority, the estimated cost of its 5-year capital plan is CAD $3.46 billion (USD $2.61 billion), which will allow the airport authority to handle 65 million passengers. This report discusses (1) levels of federal and other funding that U.S. airports received from fiscal years 2013 through 2017 for infrastructure investments, (2) projected costs of planned infrastructure investments at U.S. airports from fiscal years 2019 through 2023, and (3) any challenges selected airports face in obtaining airports’ infrastructure funding and financing. We also examined how selected airports in other countries fund and finance airport infrastructure investments. This information is presented in appendix I. To obtain information for all objectives, we reviewed relevant literature, including academic and industry literature on airport funding and financing in the United States and in other countries. We also reviewed laws, regulations, agency guidance, and prior GAO reports related to this topic. To determine what federal and other funding U.S. airports received from fiscal years 2013 through 2017 for infrastructure investments, we obtained and analyzed information on the main sources of airport funding which included: funding from federal Airport Improvement Program (AIP) grants and state grants, revenue from passenger facility charges (PFC), airport-generated revenue, capital contributions, and amounts of financing airports received from bond proceeds and other debt financing. Because comprehensive data on airport capital spending is not available, we framed our research objective to examine funding received rather than how much airports expended on infrastructure projects. We selected fiscal years 2013 through 2017 because it was the most recent 5-year period where complete data were available. For each funding source, we determined average annual-funding amounts for fiscal years 2013 through 2017 for all U.S. national system airports, as well as separately for larger airports and smaller airports. We defined larger airports to include large and medium hubs, and smaller airports to include small hubs, non-hubs, non-primary commercial service, reliever, and general aviation airports. We also analyzed how the amounts of funding received have changed from fiscal years 2013 through 2017. We presented all funding amounts in 2017 dollars. We obtained funding data from various sources, as follows: AIP funding: To determine how much funding airports received from federal AIP grants, we obtained and analyzed data from the Federal Aviation Administration’s (FAA) System of Airports Reporting (SOAR) database on AIP grants awarded by FAA during our study period. This database includes detailed information about AIP grants and PFC applications, approvals, and collections. We analyzed the AIP grant data to determine total annual funding by airport type for fiscal years 2013 through 2017, as well as average annual funding by airport type and project type over the same time period. State grants: Data on state funding for fiscal years 2013 through 2017 are available but are not complete, and we were not able to obtain additional information to verify the data’s reliability. As part of our 2015 review of airports’ infrastructure funding, we conducted a survey in 2014 with the assistance of the National Association of State Aviation Officials (NASAO), to determine how much funding airports received from state grants for fiscal years 2009 through 2013. Results from this survey were reported in our 2015 report and in NASAO’s August 2015 report, NASAO State Aviation Funding and Organizational Data Report. For this review, we interviewed NASAO officials and they confirmed that the level of state funding has largely remained unchanged since the 2015 study. Therefore, we incorporated information from the 2015 survey into our current report. PFCs: To determine how much funding airports received from PFCs, we obtained and analyzed data from the SOAR database on PFC collection amounts at all airports that collected PFCs during fiscal years 2013 through 2017. Because we were unable to obtain data on airports’ expenditures of PFC revenues by project type from fiscal years 2013 through 2017, we instead obtained data on airports’ FAA- approved applications from 1992 through February 2019 showing the types of projects on which airports intended to spend their PFC revenue. Airport-generated revenue: For airport-generated revenue, which we defined as revenue available for capital development, we obtained and analyzed airport financial data from FAA’s Certification Activity Tracking System (CATS). Examples of airport-generated revenue include aeronautical revenue (including revenue earned from leases with airlines and landing fees) and non-aeronautical revenue (such as earnings from airport terminal concessions and vehicle parking fees). We analyzed the financial data to determine the amount of airport- generated revenue that airports had available for infrastructure investments, as well as amounts by airport type, for each fiscal year 2013 through 2017. We calculated airport-generated revenue by using data for the total operating revenue of an airport, subtracted by the subtotal of operating expenses, prior to subtracting depreciation, which yields operating income plus interest income. For data precision, we used a different methodology to calculate airport- generated revenue than that of our 2015 report on airport finance by not subtracting an estimated amount of PFCs used to pay for interest expense. As a result, airport-generated revenue reported here is not comparable to airport-generated revenue in our 2015 report. Airport capital contributions: To determine how much funding airports received from capital contributions, we analyzed the same set of airport financial data from CATS that we used for airport-generated revenue, discussed above. We used the line item for capital contributions (8.5 Capital Contributions) in CATS for our analysis. Airport bonds: In addition to the sources of airport funding listed above, this report also separately discusses information on airport bonding—a common financing mechanism for some airports. We analyzed FAA financial data from the CATS database on the amounts of financing that airports received from bond proceeds (line item 14.1). We also interviewed representatives at two ratings agencies—Fitch Ratings and Moody’s Investors Service—and a representative from Piper Sandler (formerly Piper Jaffray) to obtain their perspectives on the availability of airport bond financing. We assessed the reliability of FAA’s CATS data on airport financial information and SOAR data by reviewing documentation about the data and the systems that produced these data. We also interviewed FAA officials knowledgeable about the collection, maintenance, and security of these data. We also reviewed documentation that also relied on the FAA’s CATS and SOAR data and that was collected for our prior review of airport infrastructure funding and financing for a similar purpose. We determined that these data were sufficiently reliable to report funding and financing that airports received from AIP, PFCs, airport-generated revenue, capital contributions, and bond revenue for fiscal years 2013 through 2017. To determine the projected cost of airports’ planned capital development from fiscal years 2019 through 2023, we combined (1) FAA’s most recent estimate for AIP-eligible development from its Report to Congress National Plan of Integrated Airport Systems (NPIAS) 2019-2023, released in September 2018, and (2) Airports Council International – North America’s (ACI-NA) most recent estimate for AIP-ineligible development for the same time period, as reported in its February 2019 report, Terminally Challenged: Addressing the Infrastructure Funding Shortfall of America’s Airports. We developed estimates of infrastructure development costs for all national system airports, as well as by airport type. We also presented estimates of AIP-eligible development costs by project type; these estimates were based on estimates in the NPIAS report. We did not, however, present estimates of AIP-ineligible data by project type because ACI-NA’s data do not readily support such a presentation. We presented all dollar amounts in 2017 dollars. To identify changes in airports’ project costs of planned infrastructure investments, we also reviewed FAA’s NPIAS report for fiscal years 2017– 2021 and ACI-NA’s report on airports’ capital development needs for fiscal years 2017–2021, and we compared the estimates in those reports to the fiscal years 2019–2023 estimates. ACI-NA’s estimates of U.S. airports’ infrastructure project costs differ from those of FAA’s due to scope, methodology, and other reasons. For example, the ACI-NA cost estimate includes estimates for AIP-eligible and AIP-ineligible projects, while FAA only includes AIP-eligible projects as required by statute. ACI-NA’s estimate also includes projects that have already identified funding sources as well as those that have not. By comparison, FAA only includes projects without identified funding. The methodology that FAA and ACI-NA use to develop their estimates also differs. For example, FAA developed its estimates for the fiscal year 2019 through 2023 time period by reviewing information from airport plans that were available through 2017. According to ACI-NA’s report on airports’ capital development needs for 2019–2023, its cost estimates for fiscal years 2019–2023 are based on a survey of 86 airports completed in 2018. This number represents the airports with 90 percent of all enplanements in 2017. ACI-NA survey respondents were asked to report all infrastructure costs, including interest, construction and management costs, architectural and engineering costs, and contingency costs. FAA’s estimate does not include interest and contingency costs. We reviewed FAA documentation describing the methodology for producing the NPIAS cost estimate from airport-planning documents, and interviewed FAA officials. We determined FAA’s estimate of AIP-eligible planned infrastructure costs to be reliable for the purposes of our report. Similarly, we reviewed ACI-NA’s methodology for developing its report on airports’ capital development needs for 2019–2023 and interviewed ACI-NA representatives about their methodology for developing this estimate. We determined that ACI-NA’s response rates, shares of enplanements represented by the airports that responded, and ACI-NA’s estimation methodology were sufficiently reliable for the purposes of presenting an estimate of planned infrastructure costs for AIP-ineligible projects. To obtain information about any challenges airports face in obtaining airport funding and financing, we reviewed documents from and conducted interviews with representatives from ACI-NA and airport officials from 19 selected U.S. airports. We also interviewed representatives from the American Association of Airport Executives. Through our document review and interviews, we obtained information about the sources of funding and financing that airports currently receive, planned infrastructure projects, and challenges to obtaining funding and financing for these projects. We selected airports representing different hub sizes, airports with the highest planned development costs as reported in FAA’s NPIAS fiscal years 2019–2023 report, airports with increasing and decreasing enplanements in calendar years 2013 through 2017, airports that were mentioned in our literature review and that were recommended by FAA and other stakeholders whom we interviewed, and we considered the geographic location of the airport. We also visited three locations from our selected airports to discuss and view examples of airports’ planned infrastructure projects. The airports we visited included Seattle-Tacoma International Airport, Spokane International Airport, and Paine Field Airport. See table 4 for a list of all the airports where we conducted interviews. We also interviewed representatives from Airlines for America (A4A)—the U.S. airline association—and representatives from eight selected U.S. airlines to obtain their views on airport infrastructure funding and financing issues. We selected airlines with the highest passenger traffic, as measured by revenue passenger miles. In addition, we selected airlines representing legacy and low cost carriers, and airlines that provide service outside the United States. Selected airlines that we interviewed were: Alaska Airlines, American Airlines, Delta Air Lines, Frontier Airlines, JetBlue Airways, Southwest Airlines, Spirit Airlines, and United Airlines. Collectively, the selected airlines transported about 90 percent of total U.S. passenger traffic in 2018. Because we used a nonprobability sample of airport and airlines to interview, our interviews are not generalizable. Last, to obtain information about how foreign airports fund and finance infrastructure development, we reviewed documents from and conducted interviews with international airport associations, international aviation- industry stakeholders, and representatives from four of the five foreign airports that we selected as case studies. These airports included: Toronto Pearson International Airport (Canada); Frankfurt Airport (Germany); Heathrow Airport (United Kingdom); Helsinki Airport (Finland); and Changi Airport (Singapore). Representatives from Frankfurt Airport provided us with written responses and documents for our review. See table 5 for a list of international organizations and foreign airports where we conducted interviews. For each of the five selected foreign airports, we collected information about airport infrastructure funding at the airports, including the sources of funding and financing the airports use, types of projects the airport has planned, and factors they consider when setting airport charges, among other topics. In addition, for each of our case studies, we presented financial information in the appropriate foreign currency as well as in U.S. Dollars (USD) in parentheses. We converted foreign currency information to U.S. Dollars using Federal Reserve data on foreign exchange rates. For 2018 data, we used the Federal Reserve 2018 annual rate. For 2019 data, we calculated a Federal Reserve 2019 annual rate. The primary criterion that we used to select foreign airports as case studies was the ownership model of the airport. To ensure our selection included a mix of ownership models, we selected airports that fit each of the following ownership models: Government owned and operated Government owned and privately operated Partially privatized Not-for-profit, private corporation As secondary criteria, we selected foreign airports with the highest passenger traffic among international airports, airports which had service by U.S. carriers, and airports located in regions where it would be feasible to obtain information and interview officials. Because we used a nonprobability sample of foreign airports to interview, our interviews are not generalizable. While our case studies of foreign airports and their experiences with funding and financing airport infrastructure are not generalizable to all foreign airports, they provide a range of examples of how foreign airports fund and finance airport infrastructure. We conducted this performance audit from September 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Jean Cook and Susan Zimmerman (Assistant Directors); Maria Mercado (Analyst-in-Charge); Pin-En Annie Chou; Jessica Du; Sharon Dyer; David Hooper; Delwen Jones; Grant Mallie; Josh Ormond; Pam Snedden; Kelly Rubin; and Rebecca Rygg made key contributions to this report.", "summary": "U.S. airports are important contributors to the U.S. economy, providing mobility for people and goods, both domestically and internationally. About 3,300 airports in the United States are part of the national airport system and eligible to receive federal AIP grants to fund infrastructure projects. To help fund these projects, certain categories of airports are also authorized by federal law to collect PFCs, which passengers pay when buying tickets. GAO was asked to examine airport- funding sources and planned infrastructure projects. This report examines, among other issues: (1) levels of federal and other funding that U.S. airports received from fiscal years 2013 through 2017 for infrastructure projects, (2) projected costs of planned infrastructure investments at U.S. airports from fiscal years 2019 through 2023, and (3) any challenges selected airports identified in obtaining projects' funding and financing. GAO analyzed airport-funding data for AIP grants, PFCs, airport-generated revenue, and other sources for fiscal years 2013–2017—the most recent years for which data were available—and FAA's and Airports Council – North America's cost estimates of airports' planned infrastructure projects for fiscal years 2019–2023. GAO also interviewed FAA officials; representatives from airline and airport associations, and bond-rating agencies; officials from 19 selected airports representing airports of different sizes and with the highest planned development costs, among other things; and representatives from eight selected airlines, selected based on factors such as passenger traffic. From fiscal years 2013 through 2017, U.S. airports received an average of over $14 billion annually for infrastructure projects. The three largest funding sources are below: Funding from federal Airport Improvement Program (AIP) grants has remained relatively constant, at an annual average of $3.2 billion. Smaller airports (small hub, non-hub, and general aviation) collectively received more AIP funding compared to larger airports (large and medium hub). Revenue from federally authorized passenger-facility charges (PFC), a per-passenger fee charged at the ticket's point of purchase, increased by 9 percent, with an annual average of $3.1 billion. Increases in passengers and PFC revenue at larger airports contributed to this increase. Airport-generated revenue (e.g., concessions and airline landing fees) increased by 18 percent, with an annual average of $7.7 billion. While both larger and smaller airports experienced increases in these revenues, the larger airports made up 92 percent ($7.1 billion) of these revenues. In addition to these sources, some airports obtained financing by issuing bonds, secured by airport revenue or PFCs. According to Federal Aviation Administration (FAA) data, larger airports were able to generate more bond proceeds than smaller airports in part because larger airports are more likely to have a greater, more certain revenue stream to repay debt. Airports' planned infrastructure costs for fiscal years 2019 through 2023 are estimated to average $22 billion annually (in 2017 dollars)—a 19 percent increase over prior estimates for fiscal years 2017 through 2021. These costs are expected to increase in part because airports are planning to invest in more terminal projects. For example, cost estimates for AIP-eligible terminal projects increased about 51 percent when compared to FAA's prior 5-year estimate. FAA and airport association representatives stated that terminal projects can be more expensive than other projects because of the scale of the improvements, which can include renovating terminals to repair aging facilities and accommodate larger aircraft and growth in passengers. Officials from GAO's 19 selected airports cited several challenges to funding infrastructure projects. For example, officials stated that the funding and revenue they receive from combined sources may not be sufficient to cover the costs of planned infrastructure projects. The officials also raised concerns about being able to finance future airport-infrastructure projects because they have already obligated their current and future PFCs to service debt on completed and ongoing infrastructure projects. According to FAA data, in fiscal years 2013 through 2017, airports paid a total of $12 billion—or 78 percent of total PFC revenues collected—for debt service. Bond-rating agencies, however, continue to give airports high or stable ratings, and rating agencies' representatives stated that airports' access to capital markets continues to remain favorable. Some airport officials stated that to address funding challenges, they have deferred some needed infrastructure investments or completed projects in phases, steps that increased construction times and costs.", "document_type": "gao"}
{"report": "DOD space systems support and provide a wide range of capabilities to a large number of users, including the military services, the intelligence community, civil agencies, and others. These capabilities include positioning, navigation, and timing; meteorology; missile warning; and secure communications, among others. Space systems can take a long time to develop and involve multiple segments, including space, ground control stations, terminals, user equipment, and launch, as figure 1 below shows. DOD satellite systems are also expensive to acquire. Unit costs for current DOD satellites can range from $500 million to over $3 billion. The associated ground systems can cost over $6 billion to develop and maintain and the cost to launch a satellite can climb to well over $100 million. Table 1 provides highlights of the current status of DOD’s major space programs. As the table shows, DOD is also in the beginning phases of acquiring several constellations of new satellites and ground processing capabilities—including for missile warning, protected communications, space-based environmental monitoring, and space command and control. We have work underway to assess the Air Force’s space command and control development efforts and examine DOD’s analysis of alternatives for wideband communication services. For a more complete description of these major space programs, see appendix I. In addition, DOD is exploring alternatives for acquiring wideband satellite communications as well as funding development of new launch vehicles as it pursues a new acquisition strategy for procuring launch services. Our prior work has shown that many major DOD space programs have experienced significant cost increases and schedule delays. For instance, the total program cost for the Advanced Extremely High Frequency (AEHF) satellite program, a protected satellite communications system, has grown 117 percent since the program’s original cost estimate and its first satellite was launched more than 3.5 years late. For the Space Based Infrared System (SBIRS), a missile warning satellite program, the program cost grew 265 percent from its original estimate and the launch of the first satellite was delayed roughly 9 years. Both programs moved to the production phase where fewer problems tend to surface, and where there is typically less risk of significant cost and schedule growth. A more recent major satellite program, Global Positioning System (GPS) III, has seen an almost 4-year delay due to technical issues and program cost growth of about 32 percent. Cost and schedule growth has also been a challenge for satellite ground systems and user equipment. Ground system delays have been so lengthy, that satellites sometimes spend years in orbit before key capabilities can be fully exploited. For example, The command and control system for GPS III satellites, known as the Next Generation Operational Control System, or OCX, is approximately 5 years behind schedule. As a result, the Air Force has had to start two separate back-up efforts to modify the current ground system to ensure the continuity of GPS capabilities and to make anti- jamming capabilities available via Military Code, or M-code, until OCX is delivered. Our ongoing review of GPS includes an assessment of OCX schedule risk and potential impacts on OCX delivery, acceptance, and operation. We expect to issue our report on GPS in spring 2019. Development of GPS user equipment that can utilize the M-Code signal has lagged behind the fielding of GPS M-code satellites for more than a decade, due to prolonged development challenges. In December 2017, we found that while DOD had made some progress on initial testing of the receiver cards needed to utilize the M-code signal, additional development was necessary to make M-code work with the over 700 weapon systems that require it. We also found that DOD had begun initial planning to transition some weapon systems to use M-code receivers, but significantly more work remained to understand the cost and schedule of transitioning to M-code receivers across DOD. Further, in December 2017, we found that multiple entities were separately maturing their own receiver cards. We recommended that DOD assign responsibility to a single organization to collect test data, lessons learned, and design solutions so that common design solutions are employed and DOD could avoid duplication of efforts. DOD concurred with the recommendation, but has not yet taken action on it. We have previously reported that over 90 percent of the capabilities to be provided by Mobile User Objective System communications satellites—currently, five satellites are in orbit, the first of which launched in 2012—are being underutilized because of difficulties with integrating the space, ground, and terminal segments and delays in fielding compatible user terminals. Largely because of technical and management challenges, the Joint Space Operations Center Mission System (JMS) Increment 2 program—intended to replace and improve upon an aging space situational awareness and command and control system—was almost 3 years behind schedule and 42 percent over budget before the Air Force stopped development work last year. Last month, we reported that operational testing in 2018 found that JMS Increment 2 was not operationally effective or suitable due, in part, to missing software requirements, urgent deficiencies that affected system performance, and negative user feedback. Cost and schedule growth in DOD’s space programs is sometimes driven by the inherent risks associated with developing complex space technology; however, over the past 10 years we have identified a number of other management and oversight problems that have worsened the situation. These include making overly optimistic cost and schedule estimates, pushing programs forward without sufficient knowledge about technology and design, and experiencing problems in overseeing and managing contractors, among others. We have also noted that some of DOD’s programs with operational satellites, such as SBIRS, were also exceedingly ambitious, which in turn increased technology, design, and engineering risks. While SBIRS and other satellite programs provide users with important and useful capabilities, their cost growth has significantly limited the department’s buying power at a time when more resources may be needed to protect space systems and recapitalize the space portfolio. DOD faces significant challenges as it replenishes its satellite constellations. First, DOD is confronted with growing threats in space, which may require very different satellite architectures and acquisition strategies. Second, DOD is in the midst of planning major changes to its leadership for space. While these changes are designed to streamline decision-making and bring together a dispersed space workforce, they could cause some disruption to space system acquisition programs. Third, in fiscal year 2016, Congress required DOD to establish guidance to speed up acquisition timeframes by streamlining acquisition processes and oversight for certain acquisitions. GAO is examining DOD’s application of streamlining to its weapons programs. For space, challenges with past streamlining efforts may offer some lessons learned. And fourth, DOD may face resource and capacity challenges in taking on multiple space acquisitions at one time. For example, our work and other reports point to potential gaps in the space acquisition workforce and ongoing difficulties managing software development. According to Air Force Space Command and others, U.S. space systems face intentional and unintentional threats that have increased rapidly over the past 20 years. These include radio frequency interference (including jamming), laser attacks, kinetic intercept vehicles, and ground system attacks. Additionally, the hazards of the already-harsh space environment (e.g., extreme temperature fluctuations and radiation) have increased, including numbers of active and inactive satellites, spent rocket bodies, and other fragments and debris. According to a February 2019 Defense Intelligence Agency report, China and Russia in particular are developing a variety of means to exploit perceived U.S. reliance on space-based systems and challenge the U.S. position in space. The report also states that Iran and North Korea have demonstrated some counterspace capabilities that could pose a threat to militaries using space-based services. In response, recent governmentwide and DOD strategic and policy guidance have stressed the need for U.S. space systems to be survivable or resilient against such threats and DOD has taken steps to be more resilient in some of its new programs. As we found in October 2014, one way to do this is to build more disaggregated systems, including dispersing sensors onto separate satellites; using multiple domains, including space, air, and ground to provide full mission capabilities; hosting payloads on other government or commercial spacecraft; or some combination of these. With capabilities distributed across multiple platforms, rather than centralized onto just a few satellites, it may be more difficult for an adversary to target all assets to attack full system capabilities, and if an attack does take place, the loss of one smaller satellite or payload could result in less capability loss than damage to, or loss of, a large multifunctional satellite. In addition to disaggregation, DOD could make satellites more maneuverable and build in defense capabilities to protect themselves as a means to increase survivability. We also found in October 2014 that some of these options could have beneficial impacts on acquisition. For example, acquiring smaller, less complex satellites may require less time and effort to develop and produce. This may be in part due to improved requirements discipline, as more frequent production rates may allow program managers to delay new requirements to the next production cycle instead of incorporating them into ongoing timelines midstream. Building more, less-complex satellites might also provide DOD the opportunity to use commercial products and systems that have already been tested in the market. At the same time, however, addressing the need to make satellites more resilient could introduce complications. For example, DOD may need to acquire higher quantities of satellites, which may make it more difficult to manage acquisition schedules. In addition, potentially more development and production contracts may result in more complexity for program offices to manage, requiring increased oversight of contractors. Adding more satellites and new technologies may also complicate efforts to synchronize satellite, terminal, and ground system schedules, limiting delivery of capabilities to end users. Our work has also found potential barriers to making satellites more resilient. For example, in October 2014, we found that disaggregation could require DOD to make significant cultural and process changes in how it acquires space systems—for instance, by relying on new contractors, relinquishing control to providers who host government payloads on commercial satellites, using different contracting methods, and executing smaller but more numerous and faster-paced acquisition programs. It will likely require DOD to be more flexible and agile when it comes to satellite acquisitions, especially with regard to coordinating satellite delivery with interdependent systems, such as user equipment. Yet, as we have previously found, DOD’s culture has generally been resistant to changes in space acquisition approaches, and fragmented responsibilities have made it very difficult to coordinate and deliver interdependent systems. Senior leaders have recognized the need to change the space acquisition culture, and as discussed below, changes are being made to space leadership and acquisition approaches. More recently, in July 2018, we found that two factors have contributed to DOD’s limited use of commercially hosted payloads. First, DOD officials identified logistical challenges to matching government payloads with any given commercial host satellite. For example, most of the offices we spoke with cited size, weight, and power constraints, among others, as barriers to using hosted payloads. Second, while individual DOD offices have realized cost and schedule benefits from using hosted payloads, DOD as a whole has limited information on costs and benefits of hosted payloads. Further, the knowledge DOD obtained is fragmented across the agency—with multiple offices collecting piecemeal information on the use of hosted payloads. The limited knowledge and data on hosted payloads that is fragmented across the agency has contributed to resistance among space acquisition officials to adopting this approach. We recommended, and DOD concurred, that the department bolster and centralize collection and analysis of cost, technical, and lessons learned data on its use of hosted payloads. Lastly, in October 2018, we found that DOD faced mounting challenges in protecting its weapon systems—satellites and their ground systems included—from increasingly sophisticated cyber threats. We reported that this was due to the computerized nature of weapon systems, DOD’s late start in prioritizing weapon system cybersecurity, and DOD’s nascent understanding of how to develop more secure weapon systems. In operational testing, DOD routinely found mission-critical cyber vulnerabilities in systems that were under development, yet program officials GAO met with believed their systems were secure and even discounted some test results as unrealistic. Using relatively simple tools and techniques, testers were able to take control of systems and operate largely undetected, due in part to basic issues such as poor password management and unencrypted communications. DOD has recently taken several steps to improve weapon system cybersecurity, including issuing and revising policies and guidance to better incorporate cybersecurity considerations. Further, in response to congressional direction, DOD has also begun initiatives to better understand and address cyber vulnerabilities. We and others have reported for over two decades that fragmentation and overlap in DOD space acquisition management and oversight have contributed to program delays and cancellations, cost increases, and inefficient operations. For example, in February 2012 we found that fragmented leadership contributed to a 10-year gap between the delivery of GPS satellites and associated user equipment. The cancellations of several large programs over the past 2 decades were in part because of disagreements and conflicts among stakeholders. In July 2016, in response to a provision of a Senate Report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2016, we issued a report that reviewed space leadership in more depth and concluded that DOD space leadership was fragmented. We identified approximately 60 stakeholder organizations across DOD, the Executive Office of the President, the Intelligence Community, and civilian agencies. Of these, eight organizations had space acquisition management responsibilities; eleven had oversight responsibilities; and six were involved in setting requirements for defense space programs. At the same time, many experts stated that no one seemed to be in charge of space acquisitions. Our report highlighted the pros and cons of various options to reorganize space functions recommended in prior congressionally-chartered studies. The issue has taken on more importance in recent years, as DOD has realized satellites are highly vulnerable to attacks and needs to make dramatic changes in space system architectures and operations. We have found that leadership has not been focused enough to overcome interagency rivalries and resistance to change, and it has not been able to get concurrence on future architectures. The President’s Administration and DOD have taken significant steps to change space leadership. Most recent is the President’s Space Policy Directive-4, issued on February 19, 2019, and DOD’s subsequent legislative proposal submitted on March 1, 2019, to establish a United States Space Force as a sixth branch of the United States Armed Forces within the Department of the Air Force. The Policy Directive states that this is an important step toward a future military department for space and that the Space Force will (1) consolidate existing forces and authorities for military space activities, as appropriate, to minimize duplication of effort and eliminate bureaucratic inefficiencies; and (2) not include the National Aeronautics and Space Administration, the National Oceanic and Atmospheric Administration, the National Reconnaissance Office, or other non-military space organizations or missions of the United States Government. According to the Policy Directive, the Space Force would include the uniformed and civilian personnel conducting and directly supporting space operations from all DOD Armed Forces, assume responsibilities for all major military space acquisition programs, and create the appropriate career tracks for military and civilian space personnel across all relevant specialties. Pertaining to organization and leadership, the Policy Directive states that there should be a civilian Under Secretary of the Air Force for Space, to be known as the Under Secretary for Space, appointed by the President, and establishes a Chief of Staff of the Space Force, who would serve as a member of the Joint Chiefs of Staff. Furthermore, the Policy Directive states that as the Space Force matures, and as national security requires, it will become necessary to create a separate military department, to be known as the Department of the Space Force. This department would take over some or all responsibilities for the Space Force from the Department of the Air Force. The Policy Directive requires the Secretary of Defense to conduct periodic reviews to determine when to recommend that the President seek legislation to establish such a department. Our past work has identified fragmentation in space leadership, but because implementation has not yet occurred, it remains to be seen whether this policy directive and proposed legislation would resolve these issues. In implementing these changes there are many complexities to consider. For example, because space capabilities are acquired and used across the military services and defense agencies, it will be important to address many details on how to implement a Space Force among these equities. Our past work suggests that without close attention to the consequences of the compromises that will inevitably have to be made to carve out a new force structure from existing space functions, there is risk of exacerbating the fragmentation and ineffective management and oversight the Space Force is intended to address. For instance, in March 2019, DOD established the Space Development Agency to unify and integrate efforts across DOD to define, develop, and field innovative solutions. But it is unclear how this new organization will mesh with the Air Force Space and Missile Systems Center, which acquires satellites, the Defense Advanced Research Projects Agency, which creates breakthrough technologies and capabilities, and similar organizations. Moreover, even if changes are implemented effectively, they are only a first step toward addressing space acquisition problems. As we discuss below, programs will still need to embrace acquisition best practices, such as using demonstrable knowledge to make decisions. Our prior work has found that they will also need to be open to flexible and innovative approaches, and work effectively with a very wide range of stakeholders, including those that will not be part of the Space Force, such as the intelligence agencies, civilian space agencies, the current military services, as well as entities within the Office of the Secretary of Defense who help oversee and manage acquisitions. Senior leaders have acknowledged that additional changes are needed and have taken steps to help bring them about, such as the restructuring of the Air Force’s Space and Missile Systems Center, which is designed to break down stovepipes and streamline acquisition processes. DOD is managing a number of new space acquisition programs using a new authority, established under Section 804 of the National Defense Authorization Act for Fiscal Year 2016, which is to provide a streamlined alternative to the traditional DOD acquisition process. Specifically, the programs—which include follow-on missile warning and protected communications satellites, among others—will be exempted from the acquisition and requirements processes defined by DOD Directive 5000.01 and the Joint Capabilities Integration and Development System. Instead, program managers are encouraged to use a tailored approach to documentation and oversight to enable them to demonstrate new technologies or field new or updated systems within 2 to 5 years. We have ongoing work looking across the military departments at how middle-tier acquisition authority is being implemented, including for the Air Force’s space acquisition programs, and plan to issue a report later this spring. GAO and others have highlighted lessons learned from past efforts to streamline, specifically with an approach adopted for space systems in the 1990s known as Total System Performance Responsibility (TSPR). TSPR was intended to facilitate acquisition reform and enable DOD to streamline its acquisition process and leverage innovation and management expertise from the private sector. Specifically, TSPR gave a contractor total responsibility for the integration of an entire weapon system and for meeting DOD’s requirements. We found in May 2009 that because this reform made the contractor responsible for day-to-day program management, DOD did not require formal deliverable documents—such as earned value management reports—to assess the status and performance of the contractor. As a result, DOD’s capability to lead and manage the space acquisition process diminished, which magnified problems related to unstable requirements and poor contractor performance. Further, the reduction in DOD oversight and involvement led to major reductions in various government capabilities, including cost- estimating and systems-engineering staff. This, in turn, led to a lack of technical data needed to develop sound cost estimates. Best practices that we identified in the aftermath of TSPR include retaining strong oversight and insight into programs; using quantifiable data and demonstrable knowledge to make decisions to proceed, not allowing development to proceed until certain thresholds are met, empowering program managers to make decisions on the direction of the program but also holding them accountable for their choices, and canceling unsuccessful programs. Similarly, in its study of TSPR programs, the Defense Science Board/Air Force Scientific Advisory Board Joint Task Force emphasized the importance of managing requirements, sufficiently funding programs, participating in trade-off studies, and assuring that proven engineering practices characterize program implementation, among other actions. See appendix II for a more complete list of the best practices we have identified for developing complex systems. DOD is simultaneously undertaking new major acquisition efforts to replenish its missile warning, protected communications, GPS, and weather satellites. At the same time, it is boosting efforts to increase space situational awareness and protect space assets. It is also helping to fund the development of new launch vehicles, and it is considering additional significant acquisitions in wideband satellite communications and in support of missile defense activities. While there is increased attention within DOD on funding for space and building the Space Force, such widespread acquisition activities could still pose resource challenges. For example: Funding requests for space system modernization have in the past 10 years represented a small percentage (3.9 to 5 percent) of total weapon system modernization funding DOD requested. Space is competing with ships, aircraft, and the nuclear triad, among other programs for funding. This can be challenging, because over the past 2 years, DOD has begun over 9 new space acquisition programs to recapitalize current space capabilities and enhance system resiliency. In the past, we have found that it has been difficult for DOD to fund multiple new space programs at one time, particularly when it was concurrently struggling with cost overruns and schedule delays from its legacy programs. For example, OCX system development challenges have resulted in a $2.5 billion cost increase and approximate 5-year delay to the system becoming operational— using more resources for a longer time—at a cost to other programs. It is unclear whether DOD has a sufficient workforce to manage multiple new space programs. We issued a report last month that found DOD did not routinely monitor the size, mix, and location of its space acquisition workforce. We collected and aggregated data from multiple DOD space acquisition organizations and found that at least 8,000 personnel in multiple locations nationwide were working on space acquisition activities at the end of 2017. Echoing concerns raised in our prior work, we also found that DOD had difficulty attracting and retaining candidates with the requisite technical expertise. Officials from the Air Force’s Space and Missile Systems Center were concerned that there are not enough experienced mid- level acquisition personnel and also expressed concern that the bulk of military personnel assigned to program management positions were more junior in rank than the Center was authorized to obtain. We recommended that DOD (1) identify the universe of its space acquisition programs and the organizations that support them, and (2) collect and maintain data on the workforce supporting these programs. DOD concurred with our first recommendation but not the second. Software is an increasingly important enabler of DOD space systems. However, DOD has struggled to deliver software-intensive space programs that meet operational requirements within expected time frames. Although user involvement is critical to the success of any software development effort, we found in our report issued last month on DOD software-intensive space programs that the programs we reviewed that experienced cost or schedule breaches often did not effectively engage users to understand requirements and obtain feedback. Program efforts to involve users and incorporate feedback frequently did not match plans. The lack of user engagement has contributed to systems that were later found to be operationally unsuitable. The programs we reviewed also faced challenges in delivering software in shorter time frames, and in using commercial software, applying outdated tools and metrics, as well as having limited knowledge and training in newer software development techniques. DOD acknowledged these challenges and is taking steps to address them, including identifying useful software development metrics and ways to include them in new contracts. We recommended, and DOD concurred, that the department ensure its guidance addressing software development provides specific, required direction on the timing, frequency, and documentation of user involvement and feedback. Moreover, it should be noted that software development has been a struggle for other non-space weapons programs as well. The Defense Innovation Board recently reported that the department’s current approach to software development is broken and is a leading source of risk to DOD—it takes too long, is too expensive, and exposes warfighters to unacceptable risk by delaying their access to the tools they need to assure mission success. Chairman Cooper, Ranking Member Turner, and Members of the Subcommittee, this concludes my statement. I am happy to answer any questions that you have. If you or your staff have any questions about this statement, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contacts for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this statement include Rich Horiuchi, Assistant Director; Erin Cohen (Analyst in Charge); Emily Bond; Claire Buck; Maricela Cherveny; Susan Ditto; Burns C. Eckert; Laura Hook; and Anne Louise Taylor. Key contributors for the previous work on which this statement is based are listed in the products cited. Our previous work on weapons acquisitions in general, and space programs in particular, identified best practices for developing complex systems. We summarize these best practices in table 3, below. DOD Space Acquisitions: Including Users Early and Often in Software Development Could Benefit Programs. GAO-19-136. Washington, D.C.: March 18, 2019. Defense Space Systems: DOD Should Collect and Maintain Data on Its Space Acquisition Workforce. GAO-19-240. Washington, D.C.: March 14, 2019. Weapon Systems Cybersecurity: DOD Just Beginning to Grapple with Scale of Vulnerabilitie. GAO-19-128. Washington, D.C.: October 9, 2018. Military Space Systems: DOD’s Use of Commercial Satellites to Host Defense Payloads Would Benefit from Centralizing Data. GAO-18-493. Washington, D.C.: July 30, 2018. Weapon Systems Annual Assessment: Knowledge Gaps Pose Risks to Sustaining Recent Positive Trends. GAO-18-360SP. Washington, D.C.: April 25, 2018. Global Positioning System: Better Planning and Coordination Needed to Improve Prospects for Fielding Modernized Capability. GAO-18-74. Washington, D.C.: December 12, 2017. Space Launch: Coordination Mechanisms Facilitate Interagency Information Sharing on Acquisitions GAO-17-646R. Washington D.C.: August 9, 2017 Satellite Acquisitions: Agencies May Recover a Limited Portion of Contract Value When Satellites Fail. GAO-17-490. Washington, D.C.: June 9, 2017 Space Acquisitions: DOD Continues to Face Challenges of Delayed Delivery of Critical Space Capabilities and Fragmented Leadership. GAO-17-619T. Washington, D.C.: May 17, 2017. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-17-333SP. Washington, D.C.: March 30, 2017. Global Positioning System: Observations on Quarterly Reports from the Air Force. GAO-17-162R. Washington, D.C.: October 17, 2016. Defense Space Acquisitions: Too Early to Determine if Recent Changes Will Resolve Persistent Fragmentation in Management and Oversight. GAO-16-592R. Washington, D.C.: July 27, 2016. Evolved Expendable Launch Vehicle: DOD Is Assessing Data on Worldwide Launch Market to Inform New Acquisition Strategy. GAO-16-661R. Washington, D.C.: July 22, 2016 Defense Weather Satellites: DOD Faces Acquisition Challenges for Addressing Capability Needs. GAO-16-769T, Washington, D.C.: July 7, 2016. Defense Weather Satellites: Analysis of Alternatives is Useful for Certain Capabilities, but Ineffective Coordination Limited Assessment of Two Critical Capabilities. GAO-16-252R. Washington, D.C.: March 10, 2016. Space Acquisitions: Challenges Facing DOD as it Changes Approaches to Space Acquisitions. GAO-16-471T. Washington, D.C.: March 9, 2016. Space Acquisitions: GAO Assessment of DOD Responsive Launch Report. GAO-16-156R. Washington, D.C.: October 29, 2015. Space Situational Awareness: Status of Efforts and Planned Budgets. GAO-16-6R. Washington, D.C.: October 8, 2015. GPS: Actions Needed to Address Ground System Development Problems and User Equipment Production Readiness. GAO-15-657. Washington, D.C.: September 9, 2015. Evolved Expendable Launch Vehicle: The Air Force Needs to Adopt an Incremental Approach to Future Acquisition Planning to Enable Incorporation of Lessons Learned. GAO-15-623. Washington, D.C.: August 11, 2015. Defense Satellite Communications: DOD Needs Additional Information to Improve Procurements. GAO-15-459. Washington, D.C.: July 17, 2015. Space Acquisitions: Some Programs Have Overcome Past Problems, but Challenges and Uncertainty Remain for the Future. GAO-15-492T. Washington, D.C.: April 29, 2015. Space Acquisitions: Space Based Infrared System Could Benefit from Technology Insertion Planning. GAO-15-366. Washington, D.C.: April 2, 2015. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-15-342SP. Washington, D.C.: March 12, 2015. Defense Major Automated Information Systems: Cost and Schedule Commitments Need to Be Established Earlier. GAO-15-282. Washington, D.C.: February 26, 2015. DOD Space Systems: Additional Knowledge Would Better Support Decisions about Disaggregating Large Satellites. GAO-15-7. Washington, D.C.: October 30, 2014. U.S. Launch Enterprise: Acquisition Best Practices Can Benefit Future Efforts. GAO-14-776T. Washington, D.C.: July 16, 2014. 2014 Annual Report: Additional Opportunities to Reduce Fragmentation, Overlap, and Duplication and Achieve Other Financial Benefits. GAO-14-343SP. Washington, D.C.: April 8, 2014. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-14-340SP. Washington, D.C.: March 31, 2014. Space Acquisitions: Acquisition Management Continues to Improve but Challenges Persist for Current and Future Programs. GAO-14-382T. Washington, D.C.: March 12, 2014. Evolved Expendable Launch Vehicle: Introducing Competition into National Security Space Launch Acquisitions. GAO-14-259T. Washington, D.C.: March 5, 2014. The Air Force’s Evolved Expendable Launch Vehicle Competitive Procurement. GAO-14-377R. Washington, D.C.: March 4, 2014. Space Acquisitions: Assessment of Overhead Persistent Infrared Technology Report. GAO-14-287R. Washington, D.C.: January 13, 2014. Space: Defense and Civilian Agencies Request Significant Funding for Launch-Related Activities. GAO-13-802R. Washington, D.C.: September 9, 2013. Global Positioning System: A Comprehensive Assessment of Potential Options and Related Costs is Needed. GAO-13-729, Washington, D.C.: September 9, 2013. Space Acquisitions: DOD Is Overcoming Long-Standing Problems, but Faces Challenges to Ensuring Its Investments are Optimized. GAO-13-508T. Washington, D.C.: April 24, 2013. Satellite Control: Long-Term Planning and Adoption of Commercial Practices Could Improve DOD’s Operations. GAO-13-315. Washington, D.C.: April 18, 2013. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-13-294SP. Washington, D.C.: March 28, 2013. Launch Services New Entrant Certification Guide. GAO-13-317R. Washington, D.C.: February 7, 2013. Evolved Expendable Launch Vehicle: DOD Is Addressing Knowledge Gaps in Its New Acquisition Strategy. GAO-12-822. Washington, D.C.: July 26, 2012. Space Acquisitions: DOD Faces Challenges in Fully Realizing Benefits of Satellite Acquisition Improvements. GAO-12-563T. Washington, D.C.: March 21, 2012. Space and Missile Defense Acquisitions: Periodic Assessment Needed to Correct Parts Quality Problems in Major Programs. GAO-11-404. Washington, D.C.: June 24, 2011. Space Acquisitions: Development and Oversight Challenges in Delivering Improved Space Situational Awareness Capabilities. GAO-11-545. Washington, D.C.: May 27, 2011. Space Acquisitions: DOD Delivering New Generations of Satellites, but Space System Acquisition Challenges Remain. GAO-11-590T. Washington, D.C.: May 11, 2011. Global Positioning System: Challenges in Sustaining and Upgrading Capabilities Persis., GAO-10-636. Washington, D.C.: September 15, 2010. Defense Acquisitions: Challenges in Aligning Space System Components. GAO-10-55. Washington D.C.: October 29, 2009. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "DOD space systems provide critical capabilities that support military and other government operations. They can also be expensive to acquire and field, costing billions of dollars each year. As DOD seeks to replenish its satellite constellations, it faces a number of challenges to ensuring funds are used effectively. Because space-based capabilities are fundamental to U.S. national security and civilian activities, it is essential that DOD manage its space system acquisitions carefully and avoid repeating past problems. This statement provides an update on DOD's space acquisitions, focusing on challenges facing acquisitions of new space systems. This statement is based on GAO reports issued over the past 10 years on DOD space programs. In addition it draws on recent work performed in support of GAO's 2019 annual reports on the progress of major defense acquisition programs as well as duplication, overlap, and fragmentation across the federal government, among other sources. DOD is simultaneously undertaking new major acquisitions to replenish its missile warning, protected communications, navigation, and weather satellites. At the same time, it is boosting efforts to increase space situational awareness and protect space assets. Such widespread acquisition acitivites could face a wide range of resource and management challenges that GAO has reported on, including: Growing threats to satellites. Threats to satellites from both adversaries—such as jamming and cyber attacks—and space debris are increasing. DOD is making changes to how it designs its space systems to increase the resilience and survivability of space capabilities. But it has been challenged in adopting new approaches, such as using commercial satellites to host payloads, and in prioritizing cybersecurity for all of its weapon systems. For hosted payloads, GAO recommended, and DOD concurred, that the department bolster and centralize collection and analysis of cost, technical, and lessons learned data. Implementing leadership changes . DOD is planning major changes to leadership for space. It recently proposed legislation to establish a United States Space Force—initially to be housed within the Department of the Air Force—that would, according to the President's Space Policy Directive, consolidate existing military space activities and minimize duplicative efforts across DOD. GAO found in July 2016 that changes are needed to reduce fragmentation that has negatively affected space programs for many years. But open questions remain about governance as new programs get underway and whether the changes themselves may result in further fragmentation. For example, it is unclear at this time how the new Space Development Agency will mesh with organizations currently involved in testing and acquiring new space technologies. Having the right resources and know-how. While there is increased attention on funding for space and building the Space Force, new programs can still face resource challenges. DOD has begun over 9 new space programs at a time when it is also seeking increased investments in ships, aircraft, and the nuclear triad, among other programs. Moreover, it is unclear whether DOD has a sufficient workforce to manage its new programs. GAO issued a report last month that found DOD does not routinely monitor the size, mix, and location of its space acquisition workforce. Further, DOD has difficulty attracting and retaining candidates with the requisite technical expertise. GAO recommended that DOD collect and maintain data on its space acquisition workforce. DOD did not concur, but GAO maintains that DOD should have better information on such personnel, especially in light of its proposal for establishing the Space Force. GAO also found in March 2019 that selected software-intensive space programs often did not effectively engage users to understand requirements and obtain feedback. GAO recommended, and DOD concurred, that the department ensure its guidance addressing software development provides specific, required direction on the timing, frequency, and documentation of user involvement and feedback. Past GAO reports have recommended that DOD adopt acquisition best practices to help ensure cost and schedule goals are met. DOD has generally agreed and taken some actions to address these recommendations.", "document_type": "gao"}
{"report": "Medicaid is a joint federal-state health care program that provides health care coverage to low-income and medically needy individuals. At the federal level, the Centers for Medicare & Medicaid Services (CMS), within HHS, is responsible for overseeing Medicaid, while states administer their respective Medicaid programs’ day-to-day operations. Each state’s Medicaid program, by law, must cover certain categories of individuals and provide a broad array of benefits. Within these requirements, however, states have significant flexibility to design and implement their programs, resulting in more than 50 distinct state-based programs. The federal government requires coverage for certain mandatory services under Medicaid, but states may decide to include other optional services as well. Some of the largest and most commonly included services include prescription drugs, nursing facilities, home and community-based care, and hospital inpatient care. Although pharmacy coverage is an optional service under Medicaid, all 50 states and the District of Columbia provide coverage for prescription drugs. State Medicaid programs that opt to cover prescription drugs are generally required to cover all of the outpatient drugs of any drug manufacturer participating in the Medicaid Drug Rebate Program, including all of the drugs’ formats. State Medicaid programs do not directly purchase prescription drugs, but instead reimburse pharmacies for covered prescription drugs dispensed to Medicaid beneficiaries. Providers (including physicians, nurse practitioners, and physician assistants) and pharmacies provide health care services, seek payment, and are reimbursed for services by state Medicaid agencies. States may directly pay health care providers for services rendered using a fee-for- service system or may delegate these responsibilities to MCOs. Under managed care, the state contracts with MCOs to provide comprehensive health care services through its network of providers. Buprenorphine, buprenorphine-naloxone, and naltrexone may be prescribed, administered, or dispensed for use in MAT. These medications come in a variety of formats, including oral, implantable, and injectable. Buprenorphine. Buprenorphine is a partial opioid agonist, meaning it binds to opioid receptors and activates them. It reduces or eliminates opioid withdrawal symptoms, including drug cravings, and blunts the euphoria or dangerous side effects of other opioids, such as heroin. It can be used for detoxification treatment and maintenance therapy. Buprenorphine is available as a MAT medication in two oral formats—(1) tablets for sublingual (under the tongue) administration, and (2) film for sublingual or buccal (inside the cheek) administration; as a subdermal (under the skin) implant; and in an injectable format. Oral formats are often used for beneficiaries that are in the beginning stages of treatment. The implantable format is generally used for beneficiaries who are already stable on a low or moderate dosage of oral buprenorphine. The oral formats are taken daily, while the injectable format is administered monthly, and the implantable format is administered every 6 months. The medication carries a risk of abuse, particularly in oral formats where it can be used inappropriately or illegally re-sold. The injectable and implantable formats of buprenorphine are intended to minimize this risk and to increase beneficiary compliance, because the medication is administered by a provider. Buprenorphine-naloxone. Naloxone is a medication added to some oral formats of buprenorphine to reduce the chances of misuse or abuse. Buprenorphine-naloxone is available in an oral format as either a film or a tablet. It discourages people from inappropriately injecting a crushed and dissolved tablet of buprenorphine by inducing symptoms of opioid withdrawal when injected by individuals physically dependent on opioids. Naltrexone. Naltrexone is an opioid antagonist, meaning it binds to opioid receptors, but does not activate them, thereby blocking the euphoria the user would normally feel from opioids. It also may result in withdrawal symptoms if recent opioid use has occurred. Therefore, it is used for relapse prevention following complete detoxification from opioids. It can be taken daily in an oral format or as a once-monthly injection; though due to low patient compliance, SAMHSA does not recommend using oral naltrexone for OUD treatment. Naltrexone carries no known risk of abuse. For the injectable format of naltrexone, beneficiaries have to be free from opioids for at least a week before they can begin the medication. Subject to certain requirements, state Medicaid agencies may use different strategies, such as prior authorization and preferred drug lists (PDL), to manage the cost of prescription medications and ensure that patients are taking medications that are clinically appropriate. Prior authorization requires that certain conditions are met before services can be provided to patients, in part, to control utilization and prevent improper payments. PDLs reflect state Medicaid agencies’ determinations on whether medications, including those used for MAT, will be covered and whether these medications will be categorized as preferred or non- preferred. A PDL indicates the first-choice or preferred medication for a beneficiary’s particular medical condition. PDLs are utilized by state Medicaid agencies to incentivize providers to prescribe certain types of medications. In addition, the preferred or non-preferred categorization of medication can vary between fee-for-service and managed care plans within a state. If a medication is not listed on a PDL, prior authorization may be required. However, medications listed on a PDL may still have prior authorization requirements, such as requirements to ensure patient safety. There are three distribution methods by which beneficiaries can obtain MAT medications. (See fig. 1.) 1. Retail pharmacy. After receiving a prescription from a health care provider, pharmacists at a retail pharmacy, such as CVS or Walgreens, prepare and dispense (or deliver) the medication directly to a beneficiary. The pharmacist is reimbursed for the cost of the medication by a payer, such as Medicaid or private insurance, and by any co-payment from the beneficiary. 2. External delivery from a specialty pharmacy. After receiving a prescription from a physician or other health care provider, a specialty pharmacy delivers the medication directly to the provider so the medication can be administered (injected or implanted by the provider) to the beneficiary for whom it was prescribed. The specialty pharmacy ensures that any specific requirements for a medication are maintained; for example, injectable naltrexone requires the use of refrigerated warehouses, insulated shipping containers, and temperature monitoring equipment. The specialty pharmacy is reimbursed by Medicaid or another payer. 3. Buy-and-bill. A health care provider purchases the medication from a manufacturer or distributor and stores the medication until it is dispensed or administered to the appropriate patient. After the medication is dispensed or administered, the provider bills Medicaid or another payer for the cost of the medication. Each of the distribution methods used for MAT medications has characteristics with implications for beneficiaries, providers, pharmacies, and payers. (See table 1.) Medications containing buprenorphine, including buprenorphine- naloxone, are considered controlled substances, which are governed at the federal level by the Controlled Substances Act (CSA), and may be subject to state laws as well. The CSA assigns controlled substances— including narcotics, stimulants, depressants, hallucinogens, and anabolic steroids—to one of five schedules based on the substance’s medical use, potential for abuse, and risk of dependence. In addition to the laws and regulations that apply to controlled substances generally, buprenorphine—when used in the treatment of OUD—is subject to additional requirements under the CSA and implementing regulations issued by the Drug Enforcement Administration (DEA) and SAMHSA. Buprenorphine can be administered or dispensed in a SAMHSA-certified and DEA-registered opioid treatment program when used for OUD treatment. In addition, eligible providers may obtain a Drug Addiction Treatment Act of 2000 (DATA 2000) waiver from SAMHSA in order to dispense or prescribe buprenorphine, including buprenorphine- naloxone, to a limited number of patients for OUD treatment in an office- based setting, such as a doctor’s office. Until 2016, only physicians were eligible to receive a DATA 2000 waiver. However, the Comprehensive Addiction and Recovery Act of 2016 amended the CSA to allow nurse practitioners and physician assistants to receive a DATA 2000 waiver through October 1, 2021. In 2018, the SUPPORT Act eliminated the time limit, thereby permanently allowing nurse practitioners and physician assistants to obtain DATA 2000 waivers. To qualify for a waiver, providers must be appropriately licensed under state law and meet applicable certification, training, or experience requirements. Providers who prescribe, dispense, or administer buprenorphine under a DATA 2000 waiver are also subject to the CSA’s inventory and recordkeeping requirement. The waiver requirements include the following: Physicians must complete an 8-hour training course or have certain certifications or experiences, while nurse practitioners and physician assistants must complete a 24-hour training course. Physicians that receive a DATA 2000 waiver can generally treat 30 patients in their first year and may apply to increase to 100 patients after a year. Physicians that meet certain criteria can treat 100 patients in the first year and up to 275 patients after one year of prescribing at the 100- patient limit. Nurse practitioners and physician assistants may treat 30 patients in their first year with the waiver and 100 patients thereafter. State Medicaid programs have policies related to the coverage and distribution of prescription drugs that can restrict beneficiary access to MAT medications. CMS has undertaken various coordination efforts aimed generally at addressing OUD. Federal requirements and state laws can also restrict beneficiaries’ access to the treatment medications. Our review of research and interviews with stakeholders found that several state Medicaid program policies related to prescription drug coverage and distribution can restrict beneficiaries’ access to MAT medications. These are policies governing coverage of MAT medications, prior authorization requirements, preferred drug lists, and limits placed on distribution methods. While some of these policies are generally used to manage utilization and costs related to a wide range of medications, the research we reviewed and stakeholders we interviewed said that these policies can also restrict beneficiaries’ access to the medications used in MAT. In what follows, we describe these policies, including selected states’ and CMS efforts to address the potential access barriers related to these policies. Recent research suggests that several state Medicaid programs may not cover all MAT medications in all formats. Specifically, in 2018, SAMHSA reported that while all 50 states’ and the District of Columbia’s Medicaid programs covered oral formats of MAT medications and extended-release injectable naltrexone, it found no indication that 21 states (41 percent) covered either implantable buprenorphine, extended-release injectable buprenorphine, or both. CMS officials said that evidence of coverage may be difficult to find if the medications are billed as part of a medical procedure rather than separately as a medication. However, according to the study’s methodology, SAMHSA took steps to check whether the MAT medications were covered as a medical procedure, and did not find any evidence of such coverage. According to CMS officials, all the manufacturers of MAT medications in our review participate in the Medicaid Drug Rebate Program, and as a result, state Medicaid programs are required to cover these medications and all their formats. CMS officials stated that the agency generally investigates complaints about lack of drug coverage, but had not received any complaints regarding MAT medications. In addition, the officials said they were unaware of the SAMHSA report and had not taken action based on the report’s findings. Therefore, CMS lacks the information to confirm whether or to what extent gaps may exist in state Medicaid programs’ coverage of MAT medications in all formats, as SAMHSA’s report indicates. As such, Medicaid beneficiaries undergoing medication-assisted treatment may not have access to the medications they need for treatment and that are required by law to be covered. In addition, the SUPPORT Act includes a new requirement for state Medicaid programs to cover medication-assisted treatment, including all Food and Drug Administration-approved MAT medications, from October 2020 through September 2025. CMS officials stated that the agency is drafting guidance related to this requirement and plans to communicate the guidance to state Medicaid programs through a State Medicaid Director Letter prior to October 2020. The officials told us that they have not determined when the guidance will be issued. When state Medicaid agencies cover a MAT medication, they may impose certain constraints, including requiring prior authorization from the MCO or the state Medicaid agency, before a beneficiary can receive the medication. However, these requirements can have unintended consequences, such as preventing timely access to MAT. According to SAMHSA’s 2018 report, several states use prior authorizations to ensure that patients receive behavioral therapy in addition to their MAT medications or to ensure that patients have abstained from opioids for a certain period of time, which is necessary before receiving a naltrexone injection. Further, when a patient switches from one medication to another (or another format of the same medication), prior authorization may be required for a variety of reasons, such as to ensure patient safety. Officials from a stakeholder organization representing providers and officials from a manufacturer said that prior authorization for injectable buprenorphine was particularly burdensome and that decisions on whether the state Medicaid agency will allow a prescription to be dispensed can take up to 14 days. Providers in our selected states and literature we reviewed noted that these delays could be life threatening, because patients may return to drug use and possibly overdose before receiving their medication. We were also told by officials from one manufacturer that small providers may not have the office staff to promptly process the prior authorization paperwork, creating additional delays. Potential Consequences of OUD Patients Missing Treatment A health care provider told us that patients receiving treatment for opioid use disorder (OUD) need consistent access to medication- assisted treatment (MAT) medications, just as diabetic patients need consistent access to insulin. According to a survey of providers, patients with OUD who experience delays in their MAT medications could lose motivation for their treatment, which could be life threatening. Literature we reviewed and stakeholders we interviewed described other ways that prior authorization requirements can delay access to MAT medications for Medicaid beneficiaries and other patients. The examples described include the following: Talking to patients before authorizing medications. According to literature we reviewed and officials we spoke with from a manufacturer and an organization representing providers, some insurance companies require that their staff or the pharmacist talk to the patient before approving a MAT medication when using external delivery from a specialty pharmacy. This is to affirm that the patient wants the medication and agrees that the pharmacy can bill the state Medicaid program. However, speaking directly to patients can be particularly challenging for this population. Officials representing a manufacturer of a MAT medication and officials representing a health care provider organization noted that patients undergoing residential treatment may not have access to a phone, and patients in outpatient treatment are often encouraged to change their phone numbers to reduce contact with people involved in their past drug use. Also, patients may not answer phone calls from unrecognized numbers. Medication reauthorization. Providers from the District of Columbia told us that they need to reauthorize MAT medication prescriptions every 6 months, but patients may not realize the authorization is about to expire so they run out of the medication, causing them to wait hours or days to get the new prescription filled. Transportation. Prior authorization requirements for MAT medications can result in multiple trips to the pharmacy, which is problematic for patients and beneficiaries without adequate transportation. Providers from the District of Columbia noted that sometimes prescriptions for MAT medications are not ready for patients when they arrive at the pharmacy, and repeated trips to the pharmacy can be problematic for those who lack adequate transportation. Nevertheless, the patient may need multiple trips to go back to the provider and then the pharmacy again, which can be especially challenging. Fail-first requirements. Literature we reviewed, officials we interviewed representing a provider organization, and state Medicaid officials and providers noted that some prior authorizations require that a provider cannot begin treatment with certain MAT medications until treatment with other MAT medications has failed. This literature indicated that this treatment failure can increase the risk of drug use, overdose, and death. Some states have taken steps to reduce these access barriers by removing prior authorizations through changes in state policies or laws. Officials from a nonprofit organization specializing in addressing addiction told us that, as of September 1, 2019, at least 12 states had laws that prohibited prior authorizations for substance use disorder medications, including MAT medications. States may also address prior authorizations through other means, such as policies or guidance. Among our selected states, all four have taken steps to remove prior authorization requirements. The District of Columbia began to generally allow providers to prescribe and dispense MAT medications without prior authorization in April 2019. Minnesota Medicaid officials told us that in August 2018 they removed prior authorization requirements for all MAT medications on their PDL. North Carolina Medicaid officials told us that in November 2017 they eliminated their prior authorization requirement for providers to submit a treatment plan before treating patients with any MAT medication. After the requirement was removed, the officials observed an increase in beneficiaries receiving MAT medications and an increase in the number of providers writing prescriptions for buprenorphine. The officials said North Carolina has never required prior authorization for injectable buprenorphine, but that the state does have some prior authorization requirements for certain forms of oral buprenorphine or buprenorphine-naloxone. Specifically, in order to prescribe an alternative oral medication, the provider needs to demonstrate the patient tried and failed with or is medically unable to use the buprenorphine-naloxone film. Ohio Medicaid officials told us they have no prior authorization requirements for injectable naltrexone, and they removed prior authorization requirements for oral buprenorphine in January 2019. According to the officials, the state has prior authorization requirements for implantable and injectable buprenorphine to ensure patients are initially stable on oral buprenorphine before beginning these other formats. According to stakeholders we interviewed, having multiple PDLs within a state or changing PDLs can create confusion for health care providers, because they need to keep track of and follow different requirements for the same MAT medication. Such confusion can result in reduced beneficiary access to MAT medications. For example, in the District of Columbia, four MCOs and the fee-for-service program have separate PDLs. Health care providers in the District of Columbia told us that the four MCOs have different dosage restrictions for the same MAT medications. A stakeholder group representing pharmacies told us that having a uniform PDL for the state makes it easier for pharmacists to comply with the relevant restrictions and minimize delays in accessing MAT medications. In addition, a PDL may change multiple times within a short time frame, which can create further problems for patients who had become comfortable with the medication they had been taking, according to officials from a provider organization. To address any possible confusion due to the use of multiple PDLs, some states have a uniform PDL for their Medicaid programs, which means that all PDLs used in the state cover the same MAT medications in the same way. Uniform PDLs can simplify the process for prescribers and eliminate some confusion for beneficiaries when they switch health plans. For example, Minnesota implemented a uniform PDL in July 2019 to ensure more consistent access for Medicaid beneficiaries and minimize disruptions if a beneficiary changes health plans. In addition, North Carolina plans to institute a uniform PDL when its Medicaid program moves to a managed care model in November 2019. Ohio also plans to institute a uniform PDL across the state in January 2020. Ohio Medicaid officials told us the uniform PDL will have both brand name and generic oral buprenorphine as preferred medications. According to stakeholders we interviewed, the characteristics of each distribution method, as well as states’ policies on distribution methods, have implications for beneficiary access to MAT medications. The following describe the different ways in which the distribution methods may restrict beneficiaries’ access to MAT medications. Retail pharmacies generally offer access to oral formulations of medications. However, retail pharmacies do not typically administer injectable or implantable buprenorphine, and some retail pharmacies may choose not to offer any MAT medications. One survey of physicians found that some pharmacies may either treat individuals prescribed buprenorphine poorly or refuse to carry the MAT medications. External delivery from specialty pharmacies is often used by providers for the injectable or implantable MAT medications, because the specialty pharmacy deals with the administrative responsibilities of the prescription; however, processing delays can impede access to MAT medications through this method, according to literature we reviewed and stakeholders we interviewed. These specialty pharmacies handle the administrative responsibilities of acquiring the medication, including purchasing the medication and sending it to the provider for administration, and receiving reimbursement from the payer, such as Medicaid. Health care providers who administer these medications may still encounter logistical challenges in their acquisition and storage. Other challenges identified by literature and stakeholders include the following: The patient must return to the provider for a follow-up appointment to receive the medication, because the medication is delivered to the provider. However, if the patient does not return, stakeholders—including those representing specialty pharmacies—told us that the unused medication must be disposed of. Providers may face challenges ensuring staff are available to receive medication deliveries—particularly in rural locations or in small practices with multiple office locations that are not always staffed, according to stakeholders representing specialty pharmacies and providers. Prescriptions are not always filled by the specialty pharmacy until they have confirmed that they will be reimbursed by the payer, according to officials from one manufacturer we interviewed. The officials stated that when the claim is processed manually, it can take over 20 days to fill the prescription. In contrast, the officials said that if a claim can be processed electronically, payment and delivery of the medication can be almost immediate. Buy-and-bill distribution allows patients to have immediate access to MAT medications, because their provider has the medications in stock; however, some providers prefer not to use this method, because it places them at financial risk. In particular, smaller health care practices may not have the infrastructure or resources to deal with the administrative responsibilities associated with buy-and-bill, and they may not have the financial ability to pay for medications up front and then wait for reimbursement, according to stakeholders we interviewed. For example, one stakeholder we interviewed said that the cost for just 2 to 3 doses of injectable medication obtained through buy-and-bill could take up a significant portion of the profit margin for a smaller medical practice. Furthermore, if patients do not use these medications before they expire or if the reimbursement from the payer does not equal the cost of the medication, the provider may face a financial loss. According to providers we interviewed in our four selected states, the high cost of some medications—as much as $1,200 per dose for injectable medications—makes the financial risk of buy-and-bill too high. Providers also told us that some providers choose not to store buprenorphine, because they are concerned that they could be subject to a DEA inspection. Surveys of health care providers have found provider concerns related to these inspections. And as with specialty pharmacies, the provider must have someone available to receive deliveries, which can be difficult for smaller practices, according to providers in our selected states. How Access Barriers Can Affect Opioid Use Disorder Treatment A health care provider we interviewed described how access barriers affected a Medicaid beneficiary’s opioid use disorder treatment. This beneficiary was initially prescribed oral buprenorphine, but the medication was repeatedly stolen by the patient’s partner. The provider and beneficiary agreed that injectable buprenorphine would allow treatment to continue without the risk of theft. Initially, the provider was not able to find a specialty pharmacy with an electronic prescription system compatible with the provider’s system, which was necessary to receive the prescriptions. The provider told us that after a compatible specialty pharmacy was identified and the order was completed, the delivery was further delayed, because two staff members—as required—were not available to sign for the delivery when it arrived. Three months after the decision was made to switch medications, the delivery was completed and the provider administered the medication. Furthermore, state Medicaid policies that require or prevent the use of certain distribution methods for MAT medications can restrict providers from using methods that may be best suited for their patients or practice, which may in turn affect beneficiaries’ access to the medications, according to stakeholders. Medicaid officials and providers we interviewed told us that some states require the use of certain distribution methods when a provider prescribes a MAT medication. For example, Minnesota’s fee-for-service plan (which covers about 25 percent of the state’s Medicaid population) requires that health care providers use buy- and-bill for all physician administered medications, including those that are injected or implanted. Minnesota providers we interviewed told us that they are reluctant to prescribe either the injectable or implantable versions of MAT medications, due to payment delays or other problems they experienced when they attempted to use buy-and-bill. Stakeholders told us that access to MAT medications would be maximized if providers and beneficiaries are not restricted when choosing among the three distribution methods—and some states have removed such restrictions. Officials from one manufacturer told us that since 2016, nine states that required use of buy-and-bill for their medication have eliminated those requirements. Medicaid officials in North Carolina told us that because smaller medical practices do not want the inventory costs associated with buy-and-bill, the state has moved to allow providers to obtain the injectable buprenorphine through either buy-and-bill or a specialty pharmacy. According to the officials, this has resulted in the increased use of the medication. Similarly, Medicaid officials in the District of Columbia told us that prior to 2017, injectable MAT medications were only available through buy-and-bill—despite Medicaid reimbursements being lower than providers’ costs. In 2017, these medications became available from specialty pharmacies. CMS has undertaken various coordination efforts aimed generally at addressing OUD. These efforts include the following: Opioid Steering Committee—composed of CMS senior leadership and staff, according to agency officials—helps coordinate opioid policy across the agency. CMS officials told us that the bi-weekly committee meetings have included discussions about reducing barriers related to prior authorization, other utilization management practices, and implementation of the SUPPORT Act, among other opioid related topics. Action Plan to Prevent Opioid Addiction and Enhance Access to MAT—an effort by an interagency task force—is intended to address OUD barriers in Medicaid, among other things, as required by the SUPPORT Act. In September 2019, CMS held a public meeting and requested public input to develop this action plan, which it plans to issue by January 2020, as mandated by the act. State Opioid Workshop, organized by CMS, brought together state officials to share innovative practices and discuss efforts to decrease barriers to accessing treatment for OUD, according to CMS officials. The second of such workshops was held in September 2018, and CMS documentation shows that the workshop included sessions focused on MAT, including a session on states’ approaches to improving the availability and use of MAT through benefit, payment, and system design. Informational bulletins have been used by CMS to communicate information states need to manage their Medicaid programs, including recommended actions. For example, in July 2014, CMS issued a bulletin to states providing background information on MAT, examples of state-based initiatives to increase access to MAT, and resources to help ensure proper delivery of MAT services. In January 2016, CMS issued another bulletin that focused on best practices for addressing prescription opioid overdoses, misuse, and addiction, and urged states to take action to reduce the potentially dangerous usage of opioids used for pain. While this bulletin was not focused on MAT, it suggested generally that states consider reviewing benefits coverage and service utilization to ensure beneficiaries have sufficient access to MAT services, and indicated that some benefit requirements, such as prior authorizations, can reduce the use of and access to MAT. Drug Utilization Review Survey, conducted annually by CMS, contains information that the agency publishes on states’ activities related to all prescription drugs in the Medicaid program, including some limited information about MAT medications. Other CMS efforts also addressed Medicaid beneficiaries’ access to MAT. In November 2017, CMS announced a new policy to increase flexibility for states seeking a section 1115 demonstration to improve access to and quality of OUD treatment for Medicaid beneficiaries. CMS has approved section 1115 demonstrations that included OUD- related provisions for 26 states and the District of Columbia between August 2015 and November 2019. States implementing these demonstrations are expected to take action to ensure access to MAT for Medicaid beneficiaries, including by establishing a requirement that inpatient and residential settings provide access to MAT. CMS has also examined access to OUD treatment through its Innovation Accelerator Program, which provides resources to states to introduce delivery system and payment reforms in a variety of areas, including OUD. Our review of literature and interviews with stakeholders show that in addition to Medicaid policies, other federal and state policies can limit Medicaid beneficiaries’ access to MAT medications. According to stakeholders we interviewed and literature we reviewed, requirements associated with DATA 2000 waivers may limit the number of providers willing to prescribe or administer buprenorphine for MAT. Stakeholders and the literature note that providers may be reluctant to obtain the DATA 2000 waiver, due to the hours of training associated with the waiver and the cost of registering with the DEA after obtaining the waiver, among other things. According to officials from a stakeholder organization representing providers and providers in one of our selected states, the requirements to obtain a DATA 2000 waiver, including the associated hours of required training—ranging from 8 hours for physicians to 24 hours for nurse practitioners and physician assistants—contributes to perceptions that prescribing buprenorphine for the treatment of OUD is dangerous, particularly since waivers are not required to prescribe buprenorphine for pain management. A 2019 National Academy of Sciences report also notes that treatment with buprenorphine is less risky than many other OUD treatments that do not require special training. Another study suggested that other opioids not used in the treatment of OUD—and not requiring special training—are more commonly misused, diverted, or responsible for overdoses, compared with buprenorphine. All providers who prescribe controlled substances are required to register with DEA. For providers with a DATA waiver who wish to administer injectable or implantable buprenorphine in multiple office locations, the requirement that each office location be registered with DEA may be an additional burden, as these fees are $731 for 3 years, according to DEA. DEA requires that the provider pay the registration fee for each location where controlled substances are stored, administered, or dispensed, which might not be recouped if only a small number of patients are treated at the various locations. Stakeholders we interviewed and literature we reviewed also noted a concern among some health care providers that having a waiver would subject them to increased oversight from DEA and other law enforcement agencies. Specifically, officials from an organization representing addiction providers and providers in our selected states told us that the possibility of interaction with law enforcement can intimidate some providers and can be anxiety-provoking and disruptive. Literature we reviewed reported that this can lead to providers not pursuing a waiver or ceasing to prescribe buprenorphine. Surveys of health care providers have found similar concerns. These factors can create a potential treatment barrier for patients and beneficiaries by limiting the number of available providers, according to officials from an organization representing addiction providers we interviewed and literature we reviewed. The waivers also limit how many patients, including Medicaid beneficiaries, providers may treat with MAT medications. These limits may create an additional barrier to OUD treatment, particularly for providers who specialize in addiction medicine. Studies have consistently found that providers who have waivers treat fewer OUD patients than their waiver allows—and some may not accept new patients. For example: A 2016 study of rural physicians found that more than half of providers with waivers were not accepting new patients; those with a 30-patient waiver limit were treating an average of fewer than nine patients; and more than half of the providers with waivers were not treating any patients. Providers with a 100-patient waiver limit treated an average of 57 patients, although more than one-quarter were at or approaching their patient limit. A survey of physicians, nurse practitioners, and physician assistants who obtained a waiver or increased their patient waiver limit in 2017 found that these providers were treating about one-third of their patient limit. Literature we reviewed noted that providers might not treat the maximum number of patients allowed by their waiver limit, because they are not specialists in addiction medicine, or they do not want to treat a larger number of patients with OUD. These providers may have obtained a waiver to respond to the needs of their existing patients who have OUD, rather than to add new patients. In contrast, one of these studies and officials from one organization representing health care providers in addiction medicine we interviewed noted that there are providers who are addiction medicine specialists that cannot work a full-time schedule if they are only allowed to treat 275 patients, which is the maximum allowed under the waiver rules. The study projected that a capacity range of 378 to 524 patients would be necessary for providers to practice addiction medicine full time. CMS officials told us they have taken some steps to increase the number of providers with DATA 2000 waivers through funding new planning grants in 15 states, as authorized by the SUPPORT Act. According to CMS officials, the grants cover training expenses to help providers obtain the waiver, among other things. Federal laws allow certain non-physicians—such as nurse practitioners and physician assistants—to obtain a DATA 2000 waiver to prescribe and administer buprenorphine to treat OUD; however, some states’ laws may restrict their ability to do so. These laws determine the type of health care services that can be provided by different types of providers. According to literature we reviewed and stakeholders we interviewed representing physician assistants and nurse practitioners, some state laws do not allow non-physicians to write prescriptions for any controlled substances and some specifically limit their ability to write prescriptions for buprenorphine for the treatment of OUD, while others may impose no restrictions for non-physicians beyond the federal training and patient limit requirement associated with the DATA 2000 waiver. For example, officials from an organization representing providers reported that physician assistants in some states, such as Kentucky and Tennessee, cannot prescribe buprenorphine for the treatment of OUD. Further, according to officials from another organization representing providers, most states require nurse practitioners to be supervised by or have a collaborative agreement with a physician. Thus, to prescribe buprenorphine for MAT, the nurse practitioners in these states must obtain a DATA 2000 waiver and have supervision from or a collaborative agreement with a physician. North Carolina Medicaid officials told us that physician assistants and nurse practitioners in the state that have a DATA 2000 waiver must consult with a physician, but do not need to have direct affiliation with a supervising physician. The supervision requirements can affect patients’ access to MAT, including for Medicaid beneficiaries, according to stakeholders. For example, officials from an organization representing providers told us that some nurse practitioners may find it difficult to identify a qualified physician with whom they can have a collaborative agreement. In the states where nurse practitioners are not required to collaborate with a physician, these officials also told us that they see higher percentages of nurse practitioners prescribing MAT medications. HHS has identified expanding access to medication-assisted treatment as a key component of its efforts to reduce opioid use disorder and opioid overdoses. Through our work, we identified state Medicaid policies and federal and state laws that may create barriers to treatment for Medicaid beneficiaries and other patients with OUD by restricting access to MAT medications. We also identified efforts by states and CMS to address these barriers. Under federal law, state Medicaid programs are required to cover all formats of MAT medications reviewed in our study, because all manufacturers of those medications participate in the Medicaid Drug Rebate Program. In addition, the SUPPORT Act will mandate broader coverage of MAT beginning in October 2020. However, a study by SAMHSA found that nearly half of all state Medicaid programs do not cover all formats of MAT medications in our review. Yet, CMS has not taken steps to determine whether state Medicaid programs do cover all of these MAT medications and their formats, as required. Until CMS determines the extent to which state Medicaid programs cover all MAT medications, as required—and address coverage gaps when found— Medicaid beneficiaries may not be able to obtain the most effective medications to treat their opioid use disorder. We are making the following recommendation to CMS: The Administrator of CMS should determine the extent to which state Medicaid programs are in compliance with federal requirements to cover MAT medications in all formats and take actions to ensure compliance, as appropriate. (Recommendation 1) We provided a draft of this report to HHS for review. HHS provided written comments which are reprinted in appendix I. HHS also provided technical comments, which we incorporated as appropriate. In its written comments, HHS concurred with our recommendation. Specifically, HHS stated that it will examine the extent to which state Medicaid programs are in compliance with the requirements of the Medicaid Drug Rebate Program as it relates to the coverage of MAT medications and take actions to ensure compliance, as appropriate. HHS also reiterated its plans to develop guidance for states on the SUPPORT Act’s new requirement for states to cover MAT medications. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or DeniganMacauleyM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Will Simerl (Assistant Director), Carolyn Feis Korman (Analyst-in-Charge), Rebecca Hendrickson, Shirin Hormozi, Virginia Lefever, Drew Long, Leslie McNamara, and Carla Miller made key contributions to this report. Also contributing were Leia Dickerson, Carolyn Garvey, Ethiene Salgado-Rodriguez, and Emily Wilson Schwark.", "summary": "Almost 70,000 people died from drug overdoses in 2018, an estimated 69 percent of which involved opioids. Medicaid, a joint federal-state health care program for low-income and medically needy individuals, is one of the largest sources of coverage for individuals undergoing treatment for opioid use disorder. Congress included a provision in statute for GAO to review access barriers to MAT medications, including the distribution methods. This report describes policies that can restrict Medicaid beneficiaries' access to MAT medications, including any related to the distribution methods. To do this work, GAO reviewed relevant laws, policies, and documents, as well as studies describing access barriers and the benefits and challenges of the distribution methods. GAO also interviewed federal officials; stakeholders representing state Medicaid directors, health care providers, patients, and pharmacies; and state officials and health care providers from Minnesota, North Carolina, and Ohio, as well as the District of Columbia. GAO selected these three states and the District of Columbia based on their Medicaid programs' coverage of the MAT medications, their programs' spending for the treatment of opioid use disorder, and other criteria. Medication-assisted treatment (MAT)—which combines behavioral therapy and the use of certain medications, such as buprenorphine—has been shown to be effective at reducing the misuse of or addiction to opioids and increasing treatment retention. The federal government has identified expanding access to MAT as important for reducing opioid use disorders and overdoses, and has taken action to increase access. However, GAO found that some state and federal policies can restrict Medicaid beneficiaries' access to MAT medications. Some of these policies, and three selected states' and the District of Columbia's efforts to address potential access barriers, include the following: MAT medication coverage. A 2018 study found that about 40 percent of states may not provide Medicaid coverage for some formats of MAT medications, such as injectable and implantable formats, as required by federal law; however, the Centers for Medicare & Medicaid Services (CMS), which oversees Medicaid, has not determined the extent to which states are in compliance with the federal requirements to cover MAT medications. Prior authorization requirements. Some MAT medications and formats are subject to prior authorization, which requires these medications to be pre-approved before being covered by Medicaid. While these requirements are generally used to reduce expenditures, unnecessary utilization, and improper payments, stakeholders told GAO the requirements may cause life-threatening delays in the case of MAT medications. Some states, including three states and the District of Columbia that GAO reviewed, have taken steps to remove prior authorization requirements for MAT medications. Distribution methods. States may mandate the ways MAT medications can be distributed. For example, Minnesota's fee-for-service plan requires the use of the buy-and-bill distribution method for all injectable and implantable medications. This method requires providers, such as physicians, to purchase and store these medications until administered to the patient, allowing immediate access to the MAT medication for Medicaid beneficiaries. However, for expensive injectable medications, which can cost $1,200 per treatment, this method places providers at financial risk if the medication is not used or the reimbursement is less than the providers' costs, requiring resources some providers may lack, according to providers in the selected states and District of Columbia. As a result, some states have removed such restrictions to maximize beneficiary access. Federal waiver for prescribing buprenorphine. According to stakeholders GAO interviewed, some providers are unwilling to obtain the federal waiver necessary to prescribe or administer buprenorphine for opioid use disorder—due to reasons such as the hours of training associated with the waiver—which can restrict beneficiary access to this MAT medication. In addition, while nurse practitioners and physician assistants are eligible for these waivers, some state laws require them to be supervised by a physician. Stakeholders told GAO that some nurse practitioners may find it difficult to identify a qualified physician, which may affect patient access to MAT. GAO recommends that CMS determine the extent to which states are in compliance with federal requirements to cover MAT medications, and take action as appropriate. HHS concurred with this recommendation.", "document_type": "gao"}
{"report": "Our previous work on board diversity describes some of the different roles and responsibilities of corporate and FHLBank boards and their directors. Generally, a public company’s board of directors is responsible for managing the business and affairs of the corporation, including representing shareholders and protecting their interests. Corporate boards vary in size. According to a 2018 report that includes board characteristics of large public companies, the average board has about 11 directors. Corporate boards are responsible for overseeing management performance and selecting and overseeing the company’s CEO, among other duties. Directors are compensated for their work. The board generally establishes committees to enhance the effectiveness of its oversight and focus on matters of particular concern, such as an audit committee and a nominating committee to recommend potential directors to the full board. Our previous reports on board diversity include a recent report on the FHLBank System. Each of its 11 federally chartered banks has a board of directors and is cooperatively owned by its member institutions, including commercial and community banks, thrifts, credit unions, and insurance companies. Each bank’s board of directors is made up of directors from member institutions and independent directors (who cannot be affiliated with the bank’s member institutions or recipients of loans). As of October 2018, each FHLBank board had 14-24 directors, for a total of 194 directors. The Federal Home Loan Bank Act, as amended by the Housing and Economic Recovery Act of 2008, and its regulations set forth a number of requirements for FHLBank directors, including skills, term length, and the percentage who are member and independent directors. Research we reviewed for our prior work cited several benefits associated with board diversity. For example, academic and business research has shown that the broader range of perspectives represented in diverse groups requires individuals to work harder to come to a consensus, which can lead to better decisions. In addition, research has shown that diverse boards make good business sense because they may better reflect a company’s employee and customer base, and can tap into the skills of a broader talent pool. Some research has found that diverse boards that include women may have a positive impact on a company’s financial performance, but other research has not. These mixed results depend, in part, on differences in how financial performance was defined and what methodologies were used. Our prior work found the number of women on corporate boards and the number of women and minorities on FHLBank boards had increased, but their representation generally continued to lag behind men and whites, respectively. While the data sources, methodologies, and time frames for our analyses were different for each report, the trends were fairly consistent. In our 2015 report, we analyzed companies in the S&P 1500 and found that women’s representation on corporate boards increased steadily from about 8 percent in 1997 to about 16 percent in 2014. However, despite the increase in women’s representation on boards, we estimated that it could still take decades for women to achieve balance with men. When we projected the representation of women on boards into the future assuming that women join boards in equal proportion to men—a proportion more than twice what we had observed—we estimated it could take about 10 years from 2014 for women to comprise 30 percent of board directors and more than 40 years for the number of women directors to match the number of men directors (see fig. 1). Similarly, in our 2019 report on FHLBank board diversity, we found that the share of women board directors increased from 2015 to October 2018 but that women still comprised less than 25 percent of FHLBank board directors as of 2018 (see fig. 2). Our 2019 FHLBank board report also showed an increase in FHLBank directors from 2015 to 2017 for some minority groups, including African- American, Hispanic, and Asian, but they still reflected a small portion of these boards. Further, the size of the increases in minority directors on FHLBank boards was less clear than for women directors due to incomplete data on directors’ race and ethnicity (see fig. 3). In 2015 and 2019, we identified similar factors that contributed to lower numbers of women and minorities on corporate and FHLBank boards. Notably, stakeholders, board members, and others we interviewed said three key factors generally limited greater board diversity: (1) not prioritizing diversity in recruitment efforts, (2) limitations of the traditional board candidate pipeline, and (3) low turnover of board seats. In our reports on corporate and FHLBank board diversity, we found that not prioritizing diversity in recruiting efforts was contributing to a lack of women and minority candidates represented on these boards. For example, stakeholders told us board directors frequently relied on their personal networks to identify potential board candidates. Some stakeholders said that given most current board members are men, and peoples’ professional networks often resemble themselves, relying on their own networks is not likely to identify as many women board candidates. In our 2019 report on FHLBank board diversity, stakeholders we interviewed raised similar challenges to prioritizing diversity in recruitment efforts. Some FHLBank representatives said that member institutions—which nominate and/or vote on director candidates—may prioritize other considerations over diversity, such as a candidate’s name recognition. Stakeholders we interviewed for our 2015 report suggested other recruitment challenges that may hinder women’s representation on corporate boards. For example, stakeholders said that boards need to prioritize diversity during the recruiting process because unconscious biases—attitudes and stereotypes that affect our actions and decisions in an unconscious manner—can limit diversity. One stakeholder observed that board directors may have a tendency to seek out individuals who look or sound like they do, further limiting board diversity. In addition, our 2015 report found some indication that board appointments of women slow down once one or two women are on a board. A few stakeholders expressed some concern over boards that might add a woman to appear as though they are interested in board diversity without really making diversity a priority, sometimes referred to as “tokenism.” Our reports on corporate and FHLBank board diversity also identified challenges related to relying on traditional career pipelines to identify potential board candidates—pipelines in which women and minorities are also underrepresented. Our 2015 report found that boards often appoint current or former CEOs to board positions, and that women held less than 5 percent of CEO positions in the S&P 1500 in 2014. One CEO we interviewed said that as long as boards limit their searches for directors to women executives in the traditional pipeline, boards will have a difficult time finding women. Expanding board searches beyond the traditional sources, such as CEOs, could increase qualified candidates to include those in other senior level positions such as chief financial officers, or chief human resources officers. In 2019 we reported that FHLBank board members said they also experienced challenges identifying diverse board candidates within the traditional CEO talent pipeline. Stakeholders we interviewed cited overall low levels of diversity in the financial services sector, for example, as a challenge to improving board diversity. Some bank representatives said the pipeline of eligible women and minority board candidates is small. Several FHLBank directors said the requirements to identify candidates from within corresponding geographic areas may exacerbate challenges to finding diverse, qualified board candidates in certain areas of the country. By statute, candidates for a given FHLBank board must come from member institutions in the geographic area represented by the vacant board seat. Similarly, in 2011 we reported on Federal Reserve Bank directors and found they tended to be senior executives, a subset of management that is also less diverse. Our report also found that diversity varied among Federal Reserve districts, and candidates for specific board vacancies must reside in specific districts. Recruiting board candidates from within specific professional backgrounds or geographic regions is further compounded by competition for talented women and minority board candidates, according to some stakeholders. In 2019, board directors from several FHLBanks described this kind of competition. For example, a director from one bank said his board encouraged a woman to run for a director seat, but the candidate felt she could not because of her existing responsibilities on the boards of two publicly traded companies. We heard of similar competition among Federal Reserve Bank officials in 2011, where organizations were looking to diversify their boards but were competing with private corporations for the same small pipeline of qualified individuals. The relatively small number of board seats that become available each year also contributes to the slow increase in women’s and minorities’ representation on boards. Several stakeholders we interviewed for our 2015 report on corporate boards cited low board turnover, in large part due to the long tenure of most board directors, as a barrier to increasing women’s representation. In addition, with respect to FHLBank board diversity, Federal Housing Finance Agency staff acknowledged that low turnover and term lengths were challenges. Several stakeholders we interviewed for our 2019 report on FHLBank boards said balancing the need for board diversity with retaining institutional knowledge creates some challenges to increasing diversity. One director said new board directors face a steep learning curve, so it can take some time for board members to be most effective. As a result, the directors at some banks will recruit new directors only after allowing incumbent directors to reach their maximum terms, which can be several years. Just as our 2015 and 2019 reports found similar challenges to increasing the number of women and minorities on corporate and FHLBank boards, they also describe similar strategies to increase board diversity. While the stakeholders, researchers, and officials from organizations knowledgeable about corporate governance and FHLBank board diversity we interviewed generally agreed on the importance of diverse boards and many of the strategies to achieve diversity, many noted that there is no one-size-fits-all solution to increasing diversity on boards, and in some cases highlighted advantages and disadvantages of various strategies. Based on the themes identified in our prior work, strategies for increasing board diversity generally fall into three main categories—making diversity a priority; enlarging the pipeline of potential candidates; and addressing the low rate of turnover (see fig. 4). Setting voluntary targets. Several strategies we identified in our 2015 report encouraged or incentivized boards to prioritize diversity. These strategies include setting voluntary targets for the number or proportion of women or minorities to have on the board. Many stakeholders we interviewed for our prior work supported boards setting voluntary targets for a specific number or percentage of women and minority candidates rather than externally imposed targets or quotas. Requiring a diverse slate of candidates. Many stakeholders we interviewed supported a requirement by corporate boards that a slate of candidates be diverse. A couple stakeholders specifically suggested that boards should aim for slates that are half women and half men; two other stakeholders said boards should include more than one woman on a slate of candidates so as to avoid tokenism. Tokenism was also a concern for a few of the stakeholders who were not supportive of defining the composition of slates. Filling interim board seats with women or minority candidates. Our 2019 report included strategies for making diversity a priority for FHLBank boards. For example, some FHLBank directors and Federal Housing Finance Agency staff said filling interim board seats with women and minority candidates could increase diversity. By regulation, when a FHLBank director leaves the board mid-term, the directors may elect a replacement for the remainder of his or her term. One director we interviewed said that when a woman or minority director fills an interim term, the likelihood increases that he or she will be elected by the member institutions for a subsequent full term. Emphasizing the importance of diversity and diverse candidates. Our 2015 report found that emphasizing the importance of diversity and diverse candidates was important for promoting board diversity. Almost all of the stakeholders we interviewed indicated that CEOs or investors and shareholders play an important role in promoting diversity on corporate boards. For example, one stakeholder said CEOs can “set the tone at the top” by encouraging boards to prioritize diversity efforts and acknowledging the benefits of diversity. As we reported in 2019, FHLBanks have taken several steps to emphasize the importance of board diversity. For example, all 11 FHLBanks included statements in their 2017 election announcements that encouraged voting member institutions to consider diversity during the board election process. Six of the 11 banks expressly addressed gender, racial, and ethnic diversity in their announcements. In addition, we found that FHLBanks had developed and implemented strategies that target board diversity in general and member directors specifically. For example, the banks created a task force to develop recommendations for advancing board diversity and to enhance collaboration and information sharing across FHLBank boards. Each bank is represented on the task force. Directors we interviewed from all 11 FHLBanks said their banks conducted or planned to conduct diversity training for board directors, which included topics such as unconscious bias. Mentoring women and minority board candidates. In addition, several stakeholders we interviewed about corporate and FHLBank boards noted the importance of CEOs serving as mentors for women and minority candidates and sponsoring them for board seats. For example, conducting mentoring and outreach was included as a strategy in our 2019 report for increasing diversity on FHLBank boards, including current directors pledging to identify and encourage potential women and minority candidates to run for the board. One director we interviewed said he personally contacted qualified diverse candidates and asked them to run. Another director emphasized the importance of outreach by member directors to member institutions to increase diversity on FHLBank boards. Member directors have the most interaction with the leadership of member institutions and can engage and educate them on the importance of nominating and electing diverse directors to FHLBank boards. Improving information on board diversity. As we reported in 2015, several large investors and many stakeholders we interviewed supported improving federal disclosure requirements on board diversity. In addition to increasing transparency, some organization officials and researchers we interviewed said disclosing information on board diversity could cause companies to think about diversity more. While the SEC aims to ensure that companies provide material information to investors that they need to make informed investment and voting decisions, we found information companies disclose on board diversity is not always useful to investors who value this information. SEC leaves it up to companies to define diversity. As a result, there is variation in how much and the type of information companies provide publicly. Some companies choose to define diversity as including characteristics such as relevant knowledge, skills, and experience. Others define diversity as including demographic characteristics such as gender, race, or ethnicity. (See fig. 5) In February 2019, SEC issued new guidance on its diversity disclosure requirements, which aims to clarify the agency’s expectations for what information companies include in their disclosures. Nearly all of the stakeholders we interviewed for our 2015 report said investors also play an important role in promoting diversity on corporate boards. For example, almost all of the board directors and CEOs we interviewed said investors or shareholders can influence board diversity by exerting pressure on the companies they invest in to prioritize diversity when recruiting new directors. One board director we interviewed said boards listen to investors more than anyone else. For example, there have been recent news reports of investor groups voting against all candidates for board positions when the slate of candidates is not diverse. In addition, in 2019 we recommended that the Federal Housing Finance Agency, which has regulatory authority over FHLBanks, review FHLBanks’ data collection processes for demographic information on their boards. By obtaining a better understanding of the different processes FHLBanks use to collect board demographic data, the Federal Housing Finance Agency and the banks could better determine which processes or practices contribute to more complete data. More complete data could ultimately help increase transparency on board diversity and would allow them to effectively analyze data trends over time and demonstrate the banks’ efforts to maintain or increase board diversity. The Federal Housing Finance Agency agreed with this recommendation and said it intends to engage with FHLBanks’ leadership to discuss board data collection issues. The agency also stated that it plans to request that the FHLBank Board Diversity Task Force explore the feasibility and practicability for FHLBanks to adopt processes that can lead to more complete data on board director demographics. Expanding board searches beyond CEOs. Expanding searches for potential board members is yet another strategy for increasing board diversity, as we reported in 2015 and 2019. Almost all the stakeholders we interviewed supported expanding board searches beyond the traditional pipeline of CEO candidates to increase representation of women. Several stakeholders suggested that boards recruit high performing women in other senior-level positions or look to candidates in academia or the nonprofit and government sectors. Our 2015 analysis found that if boards expanded their director searches beyond CEOs to include senior-level managers, more women might be included in the candidate pool. Our 2019 report on FHLBank board diversity also included looking beyond CEOs as a strategy for increasing diversity. For example, we reported that FHLBanks can search for women and minority candidates by looking beyond member bank CEOs. By regulation, member directors can be any officer or director of a member institution, but there is a tendency to favor CEOs for board positions, according to board directors, representatives of corporate governance organizations, and academic researchers we interviewed for the report. Similar to the findings from our 2015 report, the 2019 report found that the likelihood of identifying a woman or minority candidate increases when member institutions look beyond CEOs to other officers, such as chief human resources officers. Several directors of FHLBanks also reported hiring a search firm or consultant to help them identify women and minority candidates, which is a strategy that can be used to enlarge the typical pool of applicants. Adopting term limits or age limits. Several stakeholders discussed adopting term or age limits to address low turnover of board members. Most stakeholders we interviewed for our 2015 report were not in favor of adopting term limits or age limits, and several pointed out trade-offs. For example, one CEO we interviewed said directors with longer tenure often possess invaluable knowledge about a company that newer board directors do not have. Many of the stakeholders who opposed these strategies noted that term and age limits seem arbitrary and could result in the loss of high-performing directors. Expanding board size. Several stakeholders we interviewed supported expanding board size either permanently or temporarily so as to include more women. Some stakeholders noted that expanding board size might make sense when a board is smaller, but expressed concern about challenges associated with managing large boards. Evaluating board performance. Another strategy we identified in our 2015 report to potentially help address low board turnover and in turn increase board diversity was conducting board evaluations. Many stakeholders we interviewed generally agreed it is good practice to conduct evaluations of the full board or of individual directors, or to use a skills matrix to identify skills gaps. However, a few thought evaluation processes could be more robust. Others said that board dynamics and culture can make it difficult to use evaluations as a tool to increase turnover by removing under-performing directors from boards. Several stakeholders we interviewed discussed how it is important for boards to identify skills gaps and strategically address them when a board vacancy occurs, and one stakeholder said identifying such gaps could help boards think more proactively about finding diverse candidates. The National Association of Corporate Directors has encouraged boards to use evaluations not only as a tool for assessing board director performance, but also as a means to assess board composition and gaps in skill sets. Chairwoman Waters, Ranking Member McHenry, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Chelsa Gurkin, Acting Director of Education, Workforce, and Income Security, at (202) 512-7215 or GurkinC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony include Betty Ward-Zukerman (Assistant Director), Meredith Moore (Analyst-in-Charge), Ellie Klein, and Chris Woika. In addition, key support was provided by Susan Aschoff, James Bennett, Ben Bolitzer, Ted Burik, Michael Erb, Daniel Garcia-Diaz, Monika Gomez, Kay Kuhlman, Sheila McCoy, Anna Maria Ortiz, James Rebbe, Karen Tremba, and Walter Vance. Financial Services Industry: Representation of Minorities and Women in Management and Practices to Promote Diversity, 2007-2015. GAO-19-398T. Washington, D.C.: February 27, 2019. Federal Home Loan Banks: Steps Have Been Taken to Promote Board Diversity, but Challenges Remain. GAO-19-252. Washington, D.C.: February 14, 2019. Diversity in the Technology Sector: Federal Agencies Could Improve Oversight of Equal Employment Opportunity Requirements. GAO-18-69. Washington, D.C.: November 16, 2017. Financial Services Industry: Trends in Management Representation of Minorities and Women and Diversity Practices, 2007–2015. GAO-18-64. Washington, D.C.: November 8, 2017. Corporate Boards: Strategies to Address Representation of Women Include Federal Disclosure Requirements. GAO-16-30. Washington, D.C.: December 3, 2015. Federal Home Loan Banks: Information on Governance Changes, Board Diversity, and Community Lending. GAO-15-435. Washington, D.C.: May 12, 2015. Diversity Management: Trends and Practices in the Financial Services Industry and Agencies after the Recent Financial Crisis. GAO-13-238. Washington, D.C.: April 16, 2013. Federal Reserve Bank Governance: Opportunities Exist to Broaden Director Recruitment Efforts and Increase Transparency. GAO-12-18. Washington, D.C.: October 19, 2011. Women in Management: Female Managers’ Representation, Characteristics, and Pay. GAO-10-1064T. Washington, D.C.: September 28, 2010. Financial Services Industry: Overall Trends in Management-Level Diversity and Diversity Initiatives, 1993–2008. GAO-10-736T. Washington, D.C.: May 12, 2010. Financial Services Industry: Overall Trends in Management-Level Diversity and Diversity Initiatives, 1993–2004. GAO-06-617. Washington, D.C.: June 1, 2006. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Corporate boards take actions and make decisions that not only affect the lives of millions of employees and consumers, but also influence the policies and practices of the global marketplace. Many organizations and businesses have recognized the importance of recruiting and retaining women and minorities for key positions to improve performance and better meet the needs of a diverse customer base. Academic researchers and others have highlighted how diversity among board directors increases the range of perspectives for better decision making, among other benefits. Prior GAO reports have found challenges to increasing diversity on boards and underscored the need to identify strategies that can improve or accelerate efforts to boost representation of women and minorities. These include reports examining the diversity of publicly-traded company boards and the boards of federally chartered banks, such as the FHLBanks. This statement is based on two GAO reports, issued in December 2015 and February 2019, on the representation of women on corporate boards and the representation of women and minorities on the boards of FHLBanks, respectively. Information about the scope and methodologies used can be found in the original reports. This statement focuses on (1) the extent of diversity on such boards (2) factors that hinder diversity on these boards, and (3) strategies to promote board diversity on corporate and FHLBank boards. Prior GAO reports found limited diversity on both publicly-traded company boards (corporate boards) of directors and Federal Home Loan Bank (FHLBank) boards. For example, GAO's 2019 report on FHLBank boards found women's board representation was at 23 percent in 2018; in 2015 it had been 18 percent. In a 2015 report on corporate boards, GAO projected the representation of women into the future—assuming that women join boards in equal proportion to men—and estimated it could take more than 40 years for the number of women directors to match the number of men directors. GAO's report on FHLBank boards also showed an increase in FHLBank directors from some minority groups, including African-American, Hispanic, and Asian since 2015, but they still reflected a small portion of these boards. The size of the increases in minority directors on FHLBank boards was less clear than for women directors due to incomplete board member demographic data. Similar factors may limit corporate and FHLBank boards' efforts to increase diversity, according to stakeholders, board members, and others GAO interviewed. These factors include not prioritizing diversity in board recruitment efforts, limitations of the traditional board candidate pipeline, and low turnover of board seats. GAO identified a number of strategies for increasing the representation of women and minorities on corporate and FHLBank boards based on a review of relevant literature and discussions with researchers and corporate and government officials (see figure).", "document_type": "gao"}
{"report": "The INA provides for the Secretary of Homeland Security, after consultation with other agencies, to designate a foreign country for TPS if the conditions in that country fall into one or more of three statutory categories. These categories are generally described as consisting of (1) ongoing armed conflict, (2) environmental disaster, and (3) extraordinary and temporary conditions. The Secretary may designate a country for a period of at least 6 months but no more than 18 months. At least 60 days before the end of the designation period, the Secretary is required, after consulting with other appropriate agencies, to undertake a review of the conditions in the foreign country for which a designation is in effect and to determine whether the conditions for such designation continue to be met. The Secretary must subsequently take one of the following actions: Extend the country’s TPS designation for a period of 6, 12, or 18 months, if the Secretary determines that country conditions warrant an extension of TPS. This provides TPS beneficiaries with an extended period of protection from removal. Terminate the country’s TPS designation, if the Secretary determines that the country no longer meets the statutory criteria. This results in an expiration of the period of protection for foreign nationals who were granted TPS under a country’s designation. In addition, the Secretary may exercise his or her discretion, on the basis of this review, to redesignate the country for TPS. With a redesignation, the Secretary allows eligible nationals from the designated foreign country who have arrived in the United States since the initial designation, or another date established by the Secretary, to apply for TPS. TPS provides temporary humanitarian protection to eligible foreign nationals in the United States who, for various reasons, may not have otherwise lawful status and therefore, in the absence of TPS, would be subject to enforcement and removal under the INA. Foreign nationals may be present in the United States without valid status and potentially removable for various reasons, such as having entered without inspection and admission at a port of entry or having remained in the country beyond the expiration of previous temporary status (e.g., tourist, foreign student). Eligible foreign nationals may also seek TPS when they currently have another lawful status, according to USCIS officials. USCIS officials noted that this may occur, for example, when a foreign national has a temporary nonimmigrant status nearing its end date when TPS is designated for his or her country and applies for TPS before the existing status expires. Under the INA, applicants for TPS must apply during the registration period established by the Secretary of Homeland Security for a particular country designation. To be eligible for TPS, an applicant from a designated country must have been physically present in the United States continuously since the most recent designation’s effective date and must have resided in the United States continuously since the date established by the Secretary of Homeland Security. The INA also specifies that an individual is ineligible for TPS if he or she has been convicted of any felony or of two or more misdemeanors committed in the United States; if any of the statutory bars to asylum apply, such as involvement in persecution of others; or if he or she is reasonably regarded as a danger to the security of the United States, among other bases. In addition to protecting beneficiaries from removal, TPS authorizes them to work in the United States for the designation period. To receive evidence of work authorization, TPS beneficiaries generally apply to USCIS for an employment authorization document, Form I-766. USCIS provides this document as a plastic card that shows proof of the individual’s authorization to work in the United States and includes a photograph of the individual. Although USCIS does not require beneficiaries to apply for an employment authorization document, according to USCIS officials, beneficiaries typically apply to obtain these cards as evidence of their authorization to work in the United States. Figure 1 shows an example of an employment authorization document issued by USCIS. Several key DHS and State components may be involved in the TPS decision process, as table 1 shows. Additionally, other DHS offices and components, as well as agencies such as the Department of Defense or U.S. Agency for International Development, may provide information about country conditions to help inform the Secretary of Homeland Security’s decisions. Since TPS was established in 1990, foreign nationals in the United States from 22 countries have been granted TPS. Our review of Federal Register notices published in fiscal years 1990 through 2019 found varying bases for the 22 countries’ TPS designations. We also found that designations for 20 of these countries were subsequently extended or the countries were redesignated one or more times. Somalia, first designated for TPS in September 1991, had the longest overall designation period since TPS was established. As of the end of fiscal year 2019, Somalia’s designation had been extended 21 times and the country had been redesignated twice; its most recent extension was set to expire in March 2020. Designations for only two countries were terminated without any extensions or redesignations—Kuwait, designated in 1991, and Guinea-Bissau, designated in 1999. Figure 2 shows all effective dates of TPS designations and subsequent decisions, including extensions, terminations, and redesignations, as well as the bases for the designations for each of the 22 countries in fiscal years 1990 through 2019. As figure 2 shows, 26 TPS designations occurred in fiscal years 1990 through 2019, and 22 designations were extended at least once. As of September 30, 2019, the designations for all but four countries had been terminated and the termination of six countries’ designations since fiscal year 2018 had been temporarily halted because of ongoing litigation. Redesignations occurred 20 times. Designations. Of the 26 TPS designations, three were for one country, Liberia, and four were for two countries, El Salvador and Sierra Leone, that were each designated twice. Extensions. The majority of TPS designations (17 of 26 designations) were extended up to eight times. Designations for five countries—El Salvador, Honduras, Nicaragua, Somalia, and Sudan—were extended more than 10 times each. Three of the 22 countries’ designations were not extended before termination. Terminations. The TPS designations for all countries except Somalia, South Sudan, Syria, and Yemen had been terminated as of September 30, 2019. The termination of six countries’ designations since fiscal year 2018 had been temporarily halted because of ongoing litigation. Several lawsuits had been filed regarding the Secretary of Homeland Security’s decisions to terminate TPS for El Salvador, Haiti, Honduras, Nepal, Nicaragua, and Sudan. In October 2018, a U.S. district judge in California issued a preliminary injunction for one of the lawsuits, temporarily blocking DHS from enforcing the Secretary’s TPS termination decisions for El Salvador, Haiti, Nicaragua, and Sudan. The U.S. government filed an appeal in response to the preliminary injunction. According to USCIS officials, DHS has regularly published notices of its continued compliance with the court’s injunction and has stated that it will continue to publish such notices pending resolution of the case In April 2019, a district court judge in New York issued a second preliminary injunction covering Haiti, which the U.S. government appealed in June 2019. Additionally, under an agreement to stay the proceedings in response to a lawsuit filed in California in February 2019, the government stipulated that it would temporarily halt terminations for Honduras and Nepal until the appeal of the October 2018 injunction had been resolved. Redesignations. Of the 20 TPS redesignations, six were for countries that were redesignated once, two were for one country that was redesignated twice, and twelve were for four countries that each were redesignated thrice—the largest number of TPS redesignations. USCIS data show that applications for TPS were approved for a total of 431,848 foreign nationals in fiscal years 2000 through 2018 and that the number of TPS beneficiaries each year grew from about 70,000 in fiscal year 2000 to about 420,000 in fiscal year 2018. The number of TPS beneficiaries increased most rapidly in fiscal years 2000 through 2005, particularly after the designation of Honduras in 1999 and El Salvador in 2001. According to USCIS officials, because adjudicating all TPS applications can take years, depending on the number of applicants from a country, the number of TPS beneficiaries for a designated country may continue rising after the established registration period for the specific designation. For example, although Honduras was initially designated for TPS in 1999, with an applicant registration period that ended on July 5, 1999, USCIS data show that the number of beneficiaries from Honduras who were granted TPS peaked in 2007 at 85,759 foreign nationals. See appendix II for additional information on the numbers of TPS beneficiaries in fiscal years 2000 through 2018, by country. Data on the number of TPS beneficiaries for fiscal year 2018—the most recent available—show that the majority of TPS beneficiaries were from three countries (El Salvador, Honduras, and Haiti), as figure 3 shows. About 98 percent of beneficiaries from six countries (Sudan, Honduras, Nicaragua, El Salvador, Haiti, and Nepal) in fiscal year 2018—408,773 foreign nationals—held TPS because the termination of their country’s TPS designation was temporarily halted because of ongoing litigation. In addition, about 2 percent of beneficiaries from four countries (Somalia, South Sudan, Syria, and Yemen) in fiscal year 2018—9,019 foreign nationals—held TPS because their country’s designation was extended. See appendix II for additional information about beneficiary characteristics in fiscal year 2018, including age, gender, and location. Our review of documentation for selected TPS decisions in fiscal years 2014 through 2018 and our interviews with DHS, USCIS, and State officials indicated that DHS’s approach for initial or subsequent reviews of countries for TPS consists of three primary steps: 1. The Secretary of Homeland Security initiates a review of a country for TPS. For an initial TPS designation, the Secretary may initiate consideration of a country in response to various triggering factors. Such factors may include, for example, a request from a U.S. government entity or a foreign government for a TPS designation based on the statutory conditions for TPS (i.e., armed conflict, environmental disaster, or extraordinary and temporary conditions). For an existing designation approaching its end date, a statutory deadline requires the Secretary to undertake a review. 2. DHS collects information on country conditions and recommendations from USCIS and State and provides this information to the Secretary of Homeland Security to inform his or her decision regarding an initial or existing TPS designation. Other DHS components and non-DHS entities, including other agencies and nongovernmental organizations, may also provide information to the Secretary or USCIS. 3. The Secretary of Homeland Security receives the information and recommendations and makes a decision about TPS for the country. The Secretary exercises discretion in determining whether to initially designate a country for TPS. For an existing designation, under the INA, the Secretary is required to determine whether country conditions warrant an extension of TPS or whether the country no longer meets the statutory criteria and TPS must be terminated. Also, the Secretary exercises discretion in determining whether to redesignate a country that was previously designated for TPS. Figure 4 illustrates these three steps. Various factors may trigger consideration of a country for an initial TPS designation, according to USCIS officials. Officials stated that the Secretary of Homeland Security’s consideration of a country for an initial designation is discretionary. However, subsequent reviews of existing designations are required by statute. See figure 5. USCIS and State officials stated that for initial TPS designations, a request from DHS, State, the White House, members of Congress, or foreign governments may trigger consideration of whether to designate a country on the basis of one or more of the three statutory categories (i.e., armed conflict, environmental disaster, or extraordinary and temporary conditions). USCIS officials added that, under the INA, the Secretary of Homeland Security has the sole authority to determine whether and when to consider a country for an initial TPS designation. Further, they noted that a request does not automatically result in a formal review of a country for TPS even if the country has experienced country conditions specified in one or more of the statutory categories, such as an armed conflict or environmental disaster. For subsequent reviews of existing TPS designations, at least 60 days before the end of the designation period, the Secretary is required, after consulting with other appropriate agencies, to undertake a review of the conditions in the foreign country for which a designation is in effect. DHS collects similar information for each review of a country for TPS, according to DHS officials and our review of selected decisions. DHS officials identified four primary sources of information that the department collects to inform the Secretary of Homeland Security’s TPS decisions: country conditions reports compiled by USCIS and State and recommendations from USCIS and State leadership. According to DHS and State officials, DHS generally consults with State on TPS decisions, although it is not specifically required to do so under the statute. Our review of 26 TPS decisions for the eight selected countries found that DHS collected the following documents to inform each decision: 1. a country conditions report compiled by USCIS, 2. a memo with a recommendation from the USCIS Director to the Secretary of Homeland Security, 3. a country conditions report compiled by State, and 4. a letter with a recommendation from the Secretary of State to the Secretary of Homeland Security. USCIS manages and coordinates the TPS information-gathering process for the Secretary of Homeland Security. While State formally provides its input through the Secretary of State’s letter and recommendation to the Secretary of Homeland Security, USCIS officials said that USCIS generally incorporates the input from State into USCIS’s country conditions report and recommendation on TPS. DHS officials noted that other internal DHS components, government agencies, and other entities may also provide information about country conditions or other factors to inform the Secretary of Homeland Security’s decisions. Figure 6 shows the information collected to support the Secretary of Homeland Security’s TPS reviews. USCIS officials indicated that the time frames for conducting TPS reviews may vary. They noted that a review for an initial designation may have a shorter time frame than a review for an existing designation, depending on the situation. In addition, the officials noted that USCIS generally starts the review process for an existing TPS designation about 6 months to a year before the end date of the country’s current designation. They added that they generally start the review process within this timeframe, given the INA requirement that the Secretary of Homeland Security either undertake a review and make a determination regarding country conditions at least 60 days in advance of the prior designation’s end date or automatically extend the designation for 6 months. According to USCIS officials, at the start of a review for an initial or existing designation, USCIS’s Office of Policy & Strategy generally reaches out to USCIS’s Refugee, Asylum and International Operations Directorate (RAIO) to request input on country conditions. USCIS officials also said that the office coordinates with State’s Bureau of Population, Refugees, and Migration regarding the target time frame for receiving State’s input. In general, once USCIS receives the input from RAIO and State, USCIS finalizes its country conditions report and recommendation memo for the Secretary of Homeland Security. Our review of documentation for the eight countries in our nongeneralizable sample of 26 TPS decisions found variation in the time frames for USCIS’s recommendation memos and for State’s recommendation letters. For the 24 reviews of existing TPS designations, USCIS provided recommendation memos to the Secretary of Homeland Security about 2 to 7 months before the end date of the prior designations. Most of State’s 26 recommendation letters were dated about 2 days to 6 months before the USCIS recommendation memos. RAIO officials noted that they use an internal template as informal guidance for the draft country conditions reports that they compile for USCIS’s Office of Policy & Strategy for reviews for initial or existing TPS designations. We reviewed the RAIO template and found, for example, that for reporting on a country being considered for a TPS designation on the basis of an environmental disaster, the template includes sections (e.g., several paragraphs) about the population harmed, damage to infrastructure, disruption in services, and status of disaster response and reconstruction. Officials added that country conditions reports may deviate from the template, because its use is not required; instead, it serves as general, informal guidance. RAIO officials also noted that information in the country conditions reports they compile is generally based on publicly available information or data related to country conditions. According to the officials, sources for such information may include U.S. agencies, foreign governments, international organizations, nongovernmental organizations, and news articles. According to State officials, after State initiates its internal process for compiling information for the Secretary of Homeland Security’s TPS review, the Bureau of Population, Refugees, and Migration generally requests input internally from the relevant regional bureau and post before compiling information for the Secretary of State’s consideration. See the text box for more details of State’s internal process for developing country conditions reports and recommendation letters to inform the Secretary of Homeland Security’s TPS reviews. State Department’s Internal Process for Compiling Information for the Secretary of Homeland Security’s Temporary Protected Status Reviews The Department of State’s (State) internal process for developing input for the Secretary of Homeland Security’s Temporary Protected Status (TPS) reviews generally includes compiling information on country conditions as well as proposed recommendations from the relevant regional bureau and overseas post, according to documentation for selected TPS decisions in fiscal years 2014 through 2018 and our interviews with DHS, USCIS, and State officials. State’s Bureau of Population, Refugees and Migration (PRM) facilitates and coordinates State’s internal process for developing this input, according to informal guidance, which State officials said the bureau has used at the working level since 2012, as well as our interviews with State officials. After DHS initiates a TPS review, PRM generally directs the relevant regional bureau to reach out to overseas posts for information about country conditions, according to State officials. State officials noted that in some cases, the regional bureau’s country desk officer takes the lead in drafting the country conditions report, depending on the country context. Officials stated that the regional bureau generally uses a questionnaire on country conditions to request information from the post for a TPS review and that the post generally also provides a recommendation, in addition to the questionnaire responses, via cable or email to the regional bureau. For example, for a country that had an existing TPS designation based on ongoing armed conflict in the country, a country conditions cable provided, among other things, information about the status of the armed conflict, an assessment of whether the return of foreign nationals would pose a serious threat to their personal safety and whether the country was unable to handle the return of nationals, and information about the impact of the conflict on economic and humanitarian conditions. State and U.S. Agency for International Development (USAID) officials noted that other agencies represented at the overseas posts, such as USAID, may provide information for a post’s input on country conditions, including information gathered “on the ground” as well as from publicly available sources. Once the regional bureau receives any input from post, the bureau desk officer prepares a draft country conditions report and recommendation, and the regional bureau works with PRM to compile a joint action memo. PRM generally provides the joint action memo, which includes a country conditions report, to the Secretary of State, according to State officials. The memo may include a joint recommendation or varying recommendations (e.g., from PRM and the regional bureau) for the Secretary’s consideration. After the Secretary determines what the department will recommend, State provides a final country conditions report and recommendation letter to the Secretary of Homeland Security as well as to U.S. Citizenship and Immigration Services’ Office of Policy & Strategy. We found that the USCIS and State country conditions reports and recommendation memos or letters that DHS and State provided for our nongeneralizable sample of 26 TPS decisions included information such as background on the cause (or reason for consideration) of the initial TPS designation and a summary of the country’s recovery from, or the status of, the situation to date. In addition, documentation provided to us for some of the TPS decisions included other information, such as certain economic indicators or broader country context. Specifically: Cause and recovery or status. USCIS and State documentation for each of the 26 TPS decisions in our review generally included (1) information related to the cause (or reason for consideration) of the initial TPS designation and (2) a summary of the country’s recovery from, or the status of, the situation to date. For example, documentation for a country designated on the basis of armed conflict described the status of the conflict and ceasefire agreements; provided information about violence against civilians and recruitment of child soldiers; provided an update on civilian casualties since the prior review; and described humanitarian challenges stemming from the conflict, such as the risk of famine. For a country designated on the basis of environmental disaster, documentation described the status of investments in recovery and efforts to rebuild after the disaster, including the number of houses and schools that had been rebuilt or repaired. This documentation also included assessments of disruption in living conditions and the extent to which economic activity and basic services had been restored. Economic indicators. USCIS documentation for 16 TPS decisions and State documentation for 12 TPS decisions in our review included information about economic indicators. Examples of such information included an estimate of damages from an environmental disaster as a percentage of a country’s gross domestic product, a summary of growth in a country’s gross domestic product in recent years, and data on the increase in food prices as a result of armed conflict in a country. Broader country context. USCIS documentation for 23 TPS decisions in our review and State documentation for 20 TPS decisions provided information about broader country context. For example, documentation for a country designated on the basis of armed conflict included broader context regarding topics such as recent natural disasters and the country’s geography. As another example, documentation for a country designated on the basis of environmental disaster provided information about subsequent natural disasters as well as violence, criminal activity, and corruption in the country. In addition to USCIS and State, other DHS offices and components and non-DHS entities may provide information to inform the Secretary’s decision. DHS officials noted that such information varies, may be solicited or unsolicited, and may be provided directly to the Secretary of Homeland Security or to USCIS. We reviewed examples of such information for several of the TPS decisions in our nongeneralizable sample. This information included items such as immigration data or intelligence analyses from other DHS offices and components—for example, the Office of Immigration Statistics, U.S. Customs and Border Protection, and U.S. Immigration and Customs Enforcement; updates from the Department of Defense on the security situation in a technical input from the Centers for Disease Control and Prevention regarding the status of an epidemic; and input from other entities, including letters from members of Congress, foreign government officials, and nongovernmental organizations. In addition, DHS officials stated that the Secretary of Homeland Security may hold briefings or meetings on TPS reviews both internally and with external entities, such as White House officials, foreign government officials, and nongovernmental organizations or advocacy groups. According to DHS officials, after USCIS and State compile their country conditions reports and recommendations for the Secretary of Homeland Security’s consideration, other DHS components—including the Office of Strategy, Policy, and Plans; the Office of the General Counsel; and the Management Directorate—review the documents as part of the standard departmental clearance process before providing them to the Secretary. Officials from these DHS components noted that the purpose of their review is generally to provide relevant technical comments and ensure that complete information has been gathered for the Secretary’s review. According to USCIS officials, after receiving the information and recommendations from USCIS and State, as well as information from any other sources, the Secretary of Homeland Security makes a decision regarding a country’s initial or existing TPS designation. USCIS officials indicated that the Secretary’s decisions may not always follow the recommendations of the USCIS Director or the Secretary of State. For example, among the 26 TPS decisions from 2014 through 2018 that we reviewed, the Secretary of Homeland Security’s decision was the same as State’s recommendation in 21 cases and differed from State’s recommendation in five cases. Initial designation. USCIS officials stated that if the Secretary of Homeland Security determines a country meets the statutory criteria for designation, the Secretary may then exercise discretion in deciding whether to initially designate the country for TPS. Existing designation. According to USCIS officials, the Secretary of Homeland Security exercises discretion in determining whether the conditions in a country satisfy statutory conditions for retaining an existing designation. However, the officials indicated that if the Secretary determines that the conditions for TPS designation continue to be met, the Secretary is required under the INA to extend the designation. Additionally, USCIS officials stated that if the Secretary determines a country no longer meets conditions for TPS designation, the Secretary is required under the INA to terminate the designation. Finally, USCIS officials stated that the Secretary may exercise discretion in deciding to redesignate a country with an existing designation and that factors such as a significant deterioration in country conditions may weigh in favor of a redesignation. Once the Secretary of Homeland Security decides whether to designate a country or to extend or terminate TPS, the decision may be documented through a signed memorandum or communicated orally to USCIS, according to USCIS officials. DHS provided memorandums or notices documenting the Secretary’s TPS decisions for all 26 decisions in our nongeneralizable sample. After the Secretary makes a TPS decision, DHS typically communicates the decision to State before announcing it to the general public. Either DHS or State then communicates the decision to the foreign embassy in Washington, D.C., and State may communicate it to the foreign government overseas. Finally, under INA provisions related to TPS, the Secretary’s decision is published in the Federal Register (see fig. 7). Since 1990, all TPS decisions have been communicated to the public through statutorily required notices in the Federal Register. DHS has also used other mechanisms, including press releases and its website, to help disseminate TPS-related information to the public. We found that a Federal Register notice was published for all TPS decisions, as required under the INA, from November 1990 to September 2019. In addition, DHS frequently used Federal Register notices as a mechanism for communicating other related information, such as effective dates for TPS designation periods, applicant registration periods, TPS beneficiary eligibility requirements, and information about employment authorization for beneficiaries. For example, the Federal Register notice extending the TPS designation of El Salvador, published on July 8, 2016, included the following: summary information about the extension, such as the period of extension and the start and end date of the extension; procedures and eligibility information for beneficiaries to register or reregister for TPS and to apply for renewal of employment authorization documents, including required forms and fees to register or reregister; directions for obtaining additional information and help with questions by accessing the USCIS website or by contacting an identified USCIS official or a USCIS customer contact center; and general information about TPS as well as information about El Salvador’s initial TPS designation and about the Secretary’s authority and reason for extending TPS for El Salvador. For a Federal Register notice of a TPS decision, according to USCIS officials, USCIS generally takes about 2 weeks to draft the notice. DHS then completes an internal review before submitting the notice to the Office of Management and Budget (OMB) for interagency review, according to officials. OMB’s Office of Information and Regulatory Affairs coordinates the notice review process, including gathering comments or proposed revisions from relevant executive branch agencies. For example, we reviewed examples of technical comments from the Centers for Disease Control and Prevention regarding draft notices of TPS decisions for the Ebola-affected countries that included information and data on the status of the epidemic and an assessment of health care infrastructure. According to USCIS officials, OMB comments are returned to DHS without identifying the agency that made each comment, and additional interagency review and comment may occur before DHS publishes the notice in the Federal Register. USCIS officials also noted that, under regulation, OMB can take up to 90 days to complete the interagency review, although the officials added that OMB aims to complete the process in a timely manner for TPS notices and generally takes about a month. According to USCIS officials, to help raise awareness of TPS decisions, USCIS has generally also issued press releases announcing all TPS decisions and published them on its website in addition to publishing Federal Register notices. Table 2 summarizes information from DHS’s publication of a press release and Federal Register notice for a 2016 TPS decision. USCIS has also taken other steps to communicate TPS decisions and related information to the public. USCIS has updated its TPS country- specific webpages with alerts about the latest TPS decisions and registration periods, among other information. Further, according to USCIS officials, the Office of Public Affairs hosted periodic national TPS teleconferences for stakeholders and conducted outreach meetings to respond to questions and discuss TPS information in communities where there might be a large number of TPS beneficiaries. For example, a teleconference invitation from USCIS to stakeholders to discuss the extension of Haiti’s TPS designation in May 2017 indicated that USCIS officials would share information about the TPS reregistration period and procedures for eligible Haitian nationals and would respond to stakeholder questions. Officials from USCIS’s Office of Public Affairs also stated that the office has drafted guidance for communicating most TPS decisions. We reviewed examples of the guidance, which included planned time lines for publishing the press releases and information to USCIS’s website as well as for conducting outreach to Congress, stakeholder groups, and TPS beneficiaries. USCIS officials noted that once the Secretary of Homeland Security makes a TPS decision, time frames for publishing the Federal Register notice may vary. USCIS officials stated that, in an effort to ensure public awareness of the decisions as soon as possible, USCIS has in some cases published a press release before the Federal Register notice of a decision was finalized and published. In reviewing TPS decisions for existing designations (i.e., extensions, terminations, and redesignations) in fiscal years 1990 through 2019, we found the following: About two-thirds of Federal Register notices announcing TPS decisions for these existing designations were published at least 30 days before the end date of the previous designation period (100 of 158 total notices). In fiscal years 1990 through 2005, 21 Federal Register notices announcing TPS decisions for existing designations were published after the end of the previous designation period. In fiscal years 2006 through 2019, all 71 Federal Register notices announcing TPS decisions for existing designations were published 4 to 159 days before the end date of the previous designation period. See figure 8 for more details. Since 1990, two mechanisms—Federal Register notices and individually mailed notifications, which TPS beneficiaries may use as evidence of their eligibility for employment—have been used to communicate automatic extensions of employment authorization documents. However, USCIS’s published guidance has not consistently identified each of these as official mechanisms to verify eligibility, resulting in confusion among employers about TPS beneficiaries’ employment eligibility. The INA states that DHS shall provide TPS beneficiaries with “an ‘employment authorized’ endorsement or other appropriate work permit” but does not specify the mechanisms that DHS should use to communicate TPS employment authorization. To receive documentation of work authorization, TPS beneficiaries generally apply for an employment authorization document after an initial TPS designation and also after any subsequent extensions or redesignations of TPS. See the text box for a description of the process that TPS beneficiaries and employers must follow to verify beneficiaries’ employment eligibility. According to USCIS officials, USCIS aims to adjudicate both initial employment authorization applications and renewal applications within 90 days after receiving an application. When it is unable to process the adjudications in this time frame, USCIS issues automatic extensions of expiring employment authorization documents for TPS beneficiaries from a specific country, to allow time for USCIS to process the volume of applications associated with a TPS reregistration period. In some instances, USCIS may issue additional automatic extensions of employment authorization documents for specific countries if it has been unable to process all pending applications within the initial automatic extension period, according to USCIS officials. When employment authorization documents are automatically extended for eligible TPS beneficiaries, the documents may appear to have expired even though they remain valid. According to USCIS officials, DHS has used the Federal Register notices announcing TPS decisions to communicate most automatic extensions of TPS employment authorization documents. For example, on January 17, 2017, DHS published a Federal Register notice extending the TPS designation of Somalia for 18 months and, in the same notice, automatically extended for 6 months the validity of employment authorization documents issued under Somalia’s TPS designation. DHS has also communicated automatic extensions of TPS employment authorization documents through Federal Register notices independent of a TPS decision. Generally, Federal Register notices announcing automatic extensions of TPS employment authorization documents include instructions for employers for completing the Form I-9, among other things. Additionally, some notices state that, to reduce employer confusion regarding automatic extensions of TPS employment authorization documents, beneficiaries should explain the extension to their employer and may also provide their employer with a copy of the relevant Federal Register notice. In five cases, beginning in fiscal year 2018, USCIS mailed notifications of automatic extensions of employment authorization documents to thousands of TPS beneficiaries from Haiti, El Salvador, Syria, and Honduras as an alternative or a supplement to posting the information in Federal Register notices. USCIS officials told us that in these cases, they mailed individual notifications of the automatic extensions to ensure that the beneficiaries would not experience any gaps in employment authorization. According to the officials, they began this practice because of the large number of affected beneficiaries. Our examination of USCIS documents found that in four of these five cases, USCIS mailed individual notifications to the TPS beneficiaries without also posting a Federal Register notice communicating the automatic extension. In all five cases, USCIS published guidance on its website to inform TPS beneficiaries and employers about the use of individually mailed notifications to communicate employment authorization document extensions. USCIS’s website states that TPS beneficiaries may present the Federal Register notice or individually mailed notification to their employer along with their expired employment authorization documents to show proof of continued employment authorization. The individual notifications also state that beneficiaries may show the notifications, along with the expired employment authorization document, to any U.S. employer as proof of continued employment authorization. However, a USCIS handbook for employers and related guidance do not specifically identify the individually mailed notifications as an official means of communicating these extensions. USCIS’s Handbook for Employers: Guidance for Completing Form I-9 (M-274) provides guidance for employers on how to properly complete Form I-9, which helps employers verify that individuals are authorized to work in the United States. The handbook contains a section about automatic employment authorization document extensions for TPS beneficiaries that references USCIS’s use of Federal Register notices to inform the public of these extensions. However, the handbook for employers does not mention USCIS’s use of individually mailed notifications to communicate the automatic extensions. USCIS’s Instructions for Form I-9, Employment Eligibility Verification notes that certain employees, including TPS beneficiaries, may present an expired employment authorization document, which may be considered unexpired if the document has been extended by USCIS. The guidance also notes that employees should enter the expiration date of an automatic extension on Form I-9. However, the instructions for Form I-9 do not detail USCIS’s mechanisms for communicating these extensions, including its use of individually mailed notifications. Some employers have reportedly refused to accept expired employment authorization documents as proof of work authorization when the documents had been automatically extended. For example, the Department of Justice’s Civil Rights Division telephone interventions website indicates that on approximately 50 occasions from September 2017 through May 2019, the Immigrant and Employee Rights Section intervened to deter employers or medical licensing boards from rejecting valid work authorization documents and, in some cases, from terminating employment for TPS beneficiaries whose employment authorization documents had been automatically extended. Also, a letter to USCIS signed by 70 law professors and scholars states that some legal service providers have reported instances of employers’ terminating TPS beneficiaries’ employment because the employer did not understand or accept the individually mailed notifications. Further, USCIS has received feedback from certain stakeholders concerned that beneficiaries might not be receiving the individual notifications in time to avoid any potential gaps in work authorization, according to USCIS officials. USCIS officials told us that the Federal Register process may be beneficial for communicating employment authorization in some cases but that they may also continue to use the individually mailed notifications as a mechanism to communicate future extensions, depending on the circumstances. USCIS has acknowledged the potential benefits of updating external guidance regarding automatic extensions of TPS employment authorization documents. However, as of December 2019, USCIS had not taken action to do so. Replying to a letter of concern from an advocacy group, USCIS stated that it could consider updating the handbook for employers to add additional guidance regarding individually mailed notifications. Effective information and communication are vital for an entity to achieve its objectives. According to Standards for Internal Control in the Federal Government, management should document policies in the appropriate level of detail and externally communicate the necessary quality information to achieve an entity’s objectives. Updating external guidance, such as the employer handbook, to clearly identify each of the official mechanisms that USCIS may use to communicate automatic extensions of TPS employment authorization documents could help USCIS ensure that employers understand and accept each of its official mechanisms for communicating these automatic extensions. This, in turn, would help to reduce the risk of employers’ terminating beneficiaries from their jobs as a result of confusion caused by unclear or inconsistent guidance. The Secretary of Homeland Security has granted TPS, providing work authorization and protection from removal, to foreign nationals from 22 countries since TPS was established in 1990. DHS has generally communicated information about employment authorization for TPS beneficiaries in a Federal Register notice, although in some cases USCIS used individually mailed notifications to communicate automatic extensions of employment authorization documents. However, USCIS’s published guidance has not consistently identified individually mailed notifications as a mechanism that may be used, leading to confusion about beneficiaries’ employment eligibility and reportedly resulting in termination of some beneficiaries’ employment. Consistent published guidance that clearly identifies each of the mechanisms used to communicate automatic extensions of TPS employment authorization documents could help USCIS ensure that employers understand and accept the evidence USCIS provides for employment authorization, reducing the risk of erroneous termination of beneficiaries’ employment. The Director of USCIS should update published guidance, such as Handbook for Employers: Guidance for Completing Form I-9 (M-274), to consistently identify each of the official mechanisms that USCIS may use to communicate automatic extensions of TPS employment authorization documents. (Recommendation 1) We provided a draft of this report to DHS, State, the Department of Defense, the Department of Health and Human Services, and the U.S. Agency for International Development for review and comment. In its written comments, reproduced in appendix III, DHS agreed with our recommendation and noted planned actions to implement it, including updating guidance in DHS’s M-274 handbook. DHS’s planned actions will address the intent of our recommendation if they include updating guidance regarding each of the official mechanisms that USCIS may use to communicate automatic extensions of TPS employment authorization documents, including the use of individually mailed notifications. The U.S. Agency for International Development also provided written comments, which are reproduced in appendix IV. In addition, DHS and State provided technical comments that we incorporated as appropriate. The Department of Defense and the Department of Health and Human Services did not provide comments. We are sending copies of this report to the appropriate congressional committees, and the Acting Secretary of Homeland Security and Secretary of State, as well as the Secretary of Defense, the Secretary of Health and Human Services, the Director of the Centers for Disease Control and Prevention, and the Administrator of the U.S. Agency for International Development. If you or your staff have any questions about this report, please contact Chelsa Gurkin at (202) 512-2964 or GurkinC@gao.gov, or Rebecca Gambler at (202) 512-6912 or GamblerR@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this reports are listed in appendix V. Our objectives were to (1) describe Temporary Protected Status (TPS) determinations and numbers of beneficiaries since TPS was established in 1990; (2) describe the approach that the Department of Homeland Security (DHS), in consultation with the Department of State (State) and other relevant agencies, takes to inform the Secretary of Homeland Security’s TPS reviews; and (3) examine DHS’s public communication regarding TPS decisions and related information, including work authorization. To describe TPS determinations since TPS was established in 1990, we reviewed information and data in Federal Register notices for all TPS designations in fiscal years 1990 through 2019. Specifically, we reviewed the designation time frames and bases (i.e., ongoing armed conflict, environmental disaster, or extraordinary and temporary conditions) for each designation since TPS was established. We also analyzed U.S. Citizenship and Immigration Services (USCIS) data on numbers of TPS beneficiaries for fiscal years 1990 through 2018. In addition, we analyzed USCIS data on TPS beneficiaries’ characteristics, such as numbers, location, age, and gender of foreign nationals granted TPS, for fiscal year 2018. To assess the reliability of USCIS data on TPS beneficiaries, we reviewed documentation and interviewed USCIS officials to identify and rectify any missing or erroneous data. According to USCIS officials, USCIS removes from its data on TPS beneficiaries any who become U.S. citizens or whose status is withdrawn, either because they no longer meet eligibility requirements or because they requested that USCIS withdraw their status. However, according to officials, the data may include foreign nationals who have since died, moved out of the country, or have an additional immigration status. Additionally, because the data comprise information provided by TPS applicants, the data may include a small number of applicant errors, according to officials. We determined that the data for fiscal years 2000 through 2018 were sufficiently reliable to provide general information about the size and characteristics of TPS beneficiaries. USCIS was not able to provide reliable data on numbers of TPS beneficiaries before fiscal year 2000 because, according to USCIS officials, these data were not consistently entered electronically in USCIS information systems. To describe the approach that DHS, in consultation with State and other relevant agencies, takes to inform the Secretary of Homeland Security’s TPS reviews, we reviewed provisions in the Immigration and Nationality Act (INA) related to TPS as well as DHS and State documentation, such as informal guidance documents used since fiscal year 2014 or earlier regarding steps taken for a TPS review. We also conducted interviews with DHS and State officials related to the processes they have used to collect information for TPS reviews since fiscal year 2014. Specifically, we interviewed DHS officials from U.S. Customs and Border Protection; the U.S. Coast Guard; U.S. Immigration and Customs Enforcement; the Management Directorate; the Office of the Executive Secretary; the Office of Intelligence and Analysis; the Office of Legislative Affairs; the Office of Partnership and Engagement; the Office of Public Affairs; the Office of Strategy, Policy, and Plans, including the Office of Immigration Statistics; and USCIS—in particular, USCIS’s Office of Policy and Strategy and USCIS’s Refugees, Asylum, and International Operations Directorate. We interviewed State officials from the Bureau of Population, Refugees, and Migration and several regional bureaus, including desk officers from the Bureaus of African Affairs, Near Eastern Affairs, South and Central Asian Affairs, and Western Hemisphere Affairs. We also interviewed State officials from overseas posts for countries that we selected for our review, including El Salvador, Haiti, Honduras, Nepal, Sudan, and Yemen. We reviewed documentation that DHS and State provided for a judgmental, nongeneralizable sample of eight countries for which DHS rendered TPS decisions in fiscal years 2014 through 2018 (El Salvador, Haiti, Honduras, Nepal, Nicaragua, Sudan, Syria, and Yemen); the TPS decisions for these eight countries represented 26 of a total of 42 TPS decisions for 13 countries in that period. We selected this sample to represent a range of decision types and designation reasons, among other factors. While this sample cannot be generalized to the countries or decisions we did not review, it provided valuable information about the approach that DHS uses for TPS reviews. The primary documents that we reviewed for each decision included information about country conditions that USCIS and State had compiled and recommendations that USCIS and State leadership had provided to the Secretary of Homeland Security. Some of the documents that we received had been redacted because of ongoing litigation related to TPS. Table 3 provides additional details of the decisions in our judgmental sample. In addition, we reviewed examples of other information that may be provided for a TPS review, including examples of input from other DHS components, other U.S. agencies, the White House, members of Congress, foreign governments, and nongovernmental organizations. Specifically, we received examples of this type of information for each of the eight countries in our judgmental, nongeneralizable sample, representing 15 of the 26 TPS decisions. For example, this information included immigration data and internal intelligence analyses compiled by DHS’s Office of Immigration Statistics, U.S. Customs and Border Protection, U.S. Immigration and Customs Enforcement, and Office of Intelligence and Analysis. We also reviewed examples of updates provided by senior Department of Defense officials for the Secretary of Homeland Security regarding the security situation in a country; technical input from the Department of Health and Human Services Centers for Disease Control and Prevention about the status of an epidemic in a country; and information from the U.S. Agency for International Development about country conditions on the ground. In addition, we interviewed officials from these three agencies regarding the types of information that they may provide for TPS reviews. Further, we reviewed examples of letters from members of Congress, foreign government officials, and nongovernmental organizations related to TPS reviews. Moreover, we reviewed examples of briefing or meeting agendas and related materials for internal and external briefings, including external briefings with White House officials, foreign government officials, and nongovernmental organizations. To examine DHS’s public communication regarding TPS decisions and related information, including work authorization, we reviewed DHS’s public communications related to TPS, including Federal Register notices, press releases, and USCIS’s website, among other information. We analyzed information in Federal Register notices published from November 29, 1990, through October 1, 2019 (the most recent available at the time of our review), to determine the timing of notices for TPS decisions and the types of information included in the notices. We reviewed examples of USCIS’s Office of Public Affairs guidance for public communication of TPS decisions. We also interviewed USCIS officials regarding the mechanisms that DHS used to communicate TPS decisions and related information, including DHS’s process for drafting and publishing Federal Register notices. Further, we examined DHS’s guidance and procedures as of fiscal year 2019 for communicating TPS employment authorization, including automatic extensions of employment authorization. We reviewed USCIS’s public communications related to automatic extensions of TPS employment authorization for both beneficiaries and employers in Federal Register notices, individually mailed notifications, an employer handbook, and information published on USCIS’s website. We interviewed USCIS officials regarding USCIS’s approach to communicating TPS employment authorization, including automatic extensions. We also reviewed information from the Department of Justice Civil Rights Division’s website related to confusion over automatic extensions of employment authorization documents for TPS beneficiaries. Additionally, we reviewed a letter to USCIS signed by 70 law professors and scholars related to instances of employers terminating TPS beneficiaries. Finally, we compared DHS’s guidance and procedures with federal internal control standards related to documenting policies and externally communicating information. We conducted this performance audit from September 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 4 lists the numbers of TPS beneficiaries, by country of citizenship, in fiscal years 2000 through 2018. During this period, the country with the largest number of TPS beneficiaries in any given fiscal year was El Salvador, with 262,262 in fiscal year 2010; followed by Honduras, with 85,759 in fiscal year 2007; and Haiti, with 58,294 in fiscal year 2014. In contrast, during the same period, Montserrat had the smallest maximum number of TPS beneficiaries in any given fiscal year, with a maximum of 21 in fiscal year 2004; followed by Angola, with a maximum of 47 in fiscal year 2002; and Burundi, with a maximum of 50 in fiscal year 2007. In addition to the contacts named above, Miriam Carroll Fenton and Taylor Matheson (Assistant Directors), Elisabeth Helmer, Cristina Norland, Ben DeYoung, Martin De Alteriis, Neil Doherty, Jenny Grover, Reid Lowe, Mary Moutsos, Jan Montgomery, Jon Najmi, Nicole Willems, and Bailey Wong made key contributions to this report. Alana Miller and Danielle Rudstein provided technical assistance.", "summary": "The INA includes provisions for eligible foreign nationals residing in the United States to obtain temporary humanitarian protection from removal, as well as work authorization, when their country of origin is designated for TPS. Since 1990, nationals of 22 countries have received TPS. The Secretary of Homeland Security may designate a country for TPS after consulting with other agencies and determining that the country meets statutory criteria related to armed conflict, environmental disaster, or extraordinary or temporary conditions that prevent its nationals from returning in safety. The Secretary may designate a country for TPS for periods of 6 to 18 months and can extend a TPS designation if deemed appropriate. GAO was asked to review the TPS decision process. This report, among other things, (1) describes the approach DHS takes to inform the Secretary of Homeland Security's TPS reviews and (2) examines DHS's communication to the public regarding TPS decisions and related information, including employment authorization. GAO reviewed documentation and data related to TPS decisions, including a nongeneralizable sample of 26 decisions for eight countries in fiscal years 2014 through 2018. GAO selected the countries to reflect various types of TPS decisions, among other factors. GAO also interviewed agency officials. The Department of Homeland Security's (DHS) reviews of countries for Temporary Protected Status (TPS) include three main steps, according to DHS and other agencies' documents and officials. First, the Secretary of Homeland Security may initiate a review of a country for TPS designation in response to various triggering factors, such as a request from a foreign government, on the basis of one or more statutory conditions. The Immigration and Nationality Act (INA) requires subsequent reviews after an initial designation. Second, U.S. Citizenship and Immigration Services (USCIS)—which manages and coordinates the TPS review process for DHS—and the Department of State (State) compile country conditions reports and recommendations to inform the Secretary's decision. Although the INA does not prescribe the other agencies that must be consulted for a TPS review,State generally has a role in providing input for the Secretary of Homeland Security's consideration. GAO found DHS collected country conditions reports and recommendations from USCIS and State for all eight of the countries GAO selected for its review. Other DHS components and non-DHS entities may also provide information. Third, under the INA,the Secretary of Homeland Security exercises discretion in deciding whether to initially designate a country for TPS. For an existing designation, the Secretary determines whether country conditions warrant an extension or termination of TPS. DHS provides official notice of decisions in the Federal Register. DHS has communicated TPS decisions to the public through required Federal Register notices as well as other mechanisms. However, DHS has not provided consistent guidance regarding mechanisms it uses to communicate automatic extensions of TPS employment authorization documents. USCIS officials stated that the agency has typically communicated these extensions of documents for TPS beneficiaries through Federal Register notices. However, for five recent automatic extensions, USCIS instead mailed individual notifications to thousands of beneficiaries. USCIS guidance on its website identifies the individual notifications as a mechanism for communicating automatic extensions, but an employers' handbook and related guidance do not. As a result, some employers reportedly terminated TPS beneficiaries' employment because the employers did not understand or accept the notifications as proof of employment authorization. Consistent guidance about the mechanisms USCIS uses could help reduce the risk that TPS beneficiaries will lose their jobs because of confusion about their authorization to work in the United States. GAO recommends USCIS consistently identify in published guidance the mechanisms used to communicate automatic extensions of TPS employment authorization documents. DHS concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "FinCEN oversees the administration of the Bank Secrecy Act and related AML regulations, and has authority to enforce BSA, including through civil money penalties. FinCEN issues regulations and interpretive guidance, provides outreach to regulated industries, conducts examinations, supports select examinations performed by federal and state agencies, and pursues civil enforcement actions when warranted. FinCEN’s other responsibilities include collecting, analyzing, and disseminating information received from covered institutions, and identifying and communicating financial crime trends and methods. See figure 1 for federal supervisory agencies involved in the BSA/AML framework. FinCEN primarily relies on supervisory agencies and other entities to conduct examinations of U.S. financial institutions to determine compliance with BSA/AML requirements (see table 1). FinCEN delegated BSA/AML examination authority to these supervisory agencies, including the banking regulators, SEC, CFTC, and IRS. IRS has been delegated authority to examine certain financial institutions (such as money services businesses) not examined by the federal functional regulators for BSA compliance. The SROs that SEC and CFTC oversee—such as FINRA and NFA respectively—have BSA/AML compliance responsibilities for the activities of their members. Apart from their delegated examination authority under the BSA, the federal functional regulators and SROs have their own regulatory authority to examine institutions they supervise for compliance with BSA. FinCEN, the banking regulators, and SEC may assess civil money penalties for BSA violations and take enforcement actions for noncompliance. The SROs have established BSA-related rules or requirements for their members based on federal requirements and may take disciplinary actions against them for violations of these rules. IRS issues letters of noncompliance to institutions it oversees and generally relies on FinCEN for formal civil enforcement action, but IRS-CI has the authority to investigate criminal violations. Other law enforcement agencies (for example, DOJ Criminal Division, FBI, and ICE- HSI) also can conduct criminal investigations of BSA violations. More generally, law enforcement agencies and prosecutors may review and start investigations into a variety of criminal matters based on BSA reporting filed in their areas of jurisdiction. According to FinCEN, BSA recordkeeping and reporting requirements establish a financial trail for law enforcement investigators to follow as they track criminals, their activities, and their assets. Finally, DOJ prosecutes financial institutions and individuals for violations of federal criminal money laundering statutes. U.S. financial institutions can assist government agencies in the detection and prevention of money laundering and terrorist financing by complying with BSA/AML requirements such as maintaining effective internal controls and reporting suspicious financial activities. BSA regulations include recordkeeping and reporting requirements, such as to keep records of cash purchases of negotiable instruments, file CTRs on cash transactions exceeding $10,000, and file SARs when institutions suspect money laundering, tax evasion, or other criminal activities. Law enforcement agencies and prosecutors (through FinCEN) may utilize the 314(a) program to locate accounts and transaction information from U.S. financial institutions when terrorism or money laundering activity is reasonably suspected based on credible evidence. Most financial institutions must develop, administer, and maintain effective AML programs. At a minimum, those financial institutions must establish a system of internal controls to ensure ongoing compliance with the BSA and its implementing regulations; provide AML compliance training for appropriate personnel; provide for independent testing; and designate a person or persons responsible for coordinating and monitoring day-to-day compliance. In addition to these requirements, FinCEN issued a final rule in 2016 requiring banks, brokers or dealers in securities, mutual funds, futures commission merchants, and introducing brokers in commodities to establish risk-based procedures for conducting customer due diligence. More specifically, covered financial institutions are to establish and maintain written policies and procedures designed to (1) identify and verify the identity of customers; (2) identify and verify the identity of the beneficial owners of legal entity customers opening accounts; (3) understand the nature and purpose of customer relationships to develop customer risk profiles; and (4) conduct ongoing monitoring to identify and report suspicious transactions and, on a risk basis, maintain and update customer information. For example, covered financial institutions must collect from the customer the name, birthdate, address, and Social Security number or equivalent of any beneficial owners. The financial institutions covered by this rule—which do not include money services businesses, casinos, or insurance companies—had until May 11, 2018, to comply. Supervisory agencies and SROs oversee financial institutions’ compliance with BSA/AML requirements primarily through compliance examinations, which, for banking regulators, can be components of regularly scheduled safety and soundness examinations. All supervisory agencies and SROs we interviewed that examine financial institutions for BSA/AML compliance have established BSA/AML examination manuals or procedures. For example, to ensure consistency in the application of BSA requirements, in 2008 FinCEN issued a BSA examination manual for use in reviewing money services businesses, including for IRS and state regulators. According to FinCEN officials, FinCEN has been updating the entire manual and completed a draft of the update in the fourth quarter of fiscal year 2018, with the goal of finalizing the updated manual by the end of fiscal year 2019. Similarly, in 2005 the federal banking regulators collaborated with FinCEN on a BSA/AML examination manual issued by the Federal Financial Institutions Examination Council (FFIEC). The entire FFIEC manual has been revised several times since its release (most recently in 2014). In May 2018, FFIEC also issued new examination procedures to address the implementation of the 2016 customer due diligence and beneficial ownership rule, discussed earlier. These updated customer due diligence examination procedures replaced the existing chapter in the FFIEC BSA/AML examination manual and added a new section “Beneficial Ownership Requirements for Legal Entity Customers—Overview and Examination Procedures.” In addition, the FFIEC has been working on an update of the entire FFIEC manual, which is expected to be complete by the end of the calendar year 2019 or early 2020. SEC and FINRA, as well as CFTC’s respective SROs, have nonpublic procedures for conducting examinations of the institutions they oversee. SEC, FINRA, and NFA officials all stated that they have updated procedures to address the new customer due diligence regulations that were applicable beginning in May 2018. We discuss examination activities of the supervisory agencies in more detail later in this report. FinCEN and supervisory agencies consider risk when planning BSA/AML examinations and all utilized BSA data to some extent to scope and plan examinations (see table 2). As we reported in prior work, BSA/AML examinations are risk-based—examiners have the flexibility to apply the appropriate level of scrutiny to business lines that pose a higher level of risk to the institution. Covered financial institutions are expected to complete a BSA/AML risk assessment to identify specific products, services, and customers, which supervisory agencies can use to evaluate the compliance programs of financial institutions and scope their examinations. Most officials from supervisory agencies and SROs said they also consider asset size, among other factors, to determine examination frequency and scope. For example, the federal banking regulators implemented less frequent examination cycles for smaller, well-capitalized financial institutions. FinCEN is the administrator of BSA and delegated BSA/AML examination authority to the supervisory agencies. FinCEN officials told us they have been considering how regulators of financial institutions of different size and risk assess BSA/AML compliance and continue to work with federal regulators to identify better ways to supervise examinations. For example, in a February 2019 speech, the Director of FinCEN stated that one of FinCEN’s current regulatory reform initiatives was reviewing the risk- based approach to the examination process. Although supervisory agencies with delegated authority conducted the vast majority of BSA/AML compliance examinations, FinCEN has conducted a few of its own examinations in areas it considers a high priority. FinCEN officials told us it mostly considers risk (not size) when conducting its own examinations because even small institutions could pose money laundering risk. FinCEN states that it uses an intelligence- driven approach to target examinations in high-risk areas. For example, FinCEN officials told us they have conducted BSA/AML compliance examinations of financial institutions on issues such as virtual currencies and data breaches in domestic branches of foreign banks. In an August 2018 speech, the Director of FinCEN noted that FinCEN, working closely with BSA examiners at IRS, had examined more than 30 percent of identified registered virtual currency exchangers and administrators since 2014—totaling about 30 examinations, according to FinCEN officials. FinCEN officials said they conducted a total of five BSA/AML examinations with IRS in fiscal years 2017 and 2018. In addition, FinCEN conducted a BSA/AML examination in fiscal year 2018 of a branch of a foreign bank that had been previously examined by its banking regulator to review the effectiveness of the bank’s BSA compliance department. All of the banking regulators with which we spoke stated they considered risk and, to some extent, asset size to determine examination frequency and scope. The FFIEC BSA/AML examination manual establishes a risk- based approach for bank examinations, including incorporating a review of BSA/AML risk assessments of a financial institution in the scoping and planning of an examination. In considering asset size to determine the frequency of examinations, all of the banking regulators adopted rules to reduce the frequency of examinations for small, well-capitalized financial institutions—as seen in table 2. In addition, in their annual reports to FinCEN the banking regulators provide a description of the criteria used for determining the timing and scope of BSA/AML examinations, such as risk and asset size. For instance, FDIC and the Federal Reserve noted in their annual reports to FinCEN that the timing and scope of their BSA/AML examinations are primarily determined by an institution’s BSA/AML risk profile and factors such as its condition, overall rating, and asset size. OCC, in its annual report, said that examination scope included consideration of the bank’s BSA/AML risk assessment, quality of validated independent testing (internal and external audit), previous examination reports, BSA reports, and other relevant factors, including data from the OCC’s Money Laundering Risk System. OCC officials said the system identifies potential indicators of BSA/AML risk by measuring the extent to which various types of products, services, customers, and geographies are offered or served by supervised banks. For banks that report into that system, OCC officials said they factor information from the system into developing an examination strategy that helps determine resource allocation and expertise needs. According to NCUA, each credit union must receive a BSA examination each examination cycle—although the frequency and scope of these examinations may vary based on the credit union’s size and other risk factors. For example, small credit unions with assets under $50 million may be subject to a defined-scope examination (which includes a BSA examination) where the risk areas have already been identified and the scope is pre-determined. NCUA also provides a BSA questionnaire that is publicly accessible to assist its examiners in implementing BSA examinations (for example, to help examiners assess the BSA risk of the credit union and scope the examination). Factors considered in the questionnaire include prior violations, correspondence from law enforcement related to BSA compliance, whether or not the credit union conducted a risk-assessment, and high-risk accounts. While the FFIEC BSA/AML examination manual and other federal banking documentation discuss considering BSA/AML risk when determining the scope and frequency of examinations, officials from all four banking associations with whom we spoke said, in practice, examiners do not always use a risk-based approach when assessing BSA compliance. Nearly all said examiners may take a zero-tolerance approach when conducting examinations. For example, representatives from two industry associations said that although failure to file a single SAR or unintentional errors should be treated differently than egregious, intentional noncompliance, or a pattern of negligence (in terms of level of noncompliance), that sometimes has not been the case. Federal Reserve officials noted that each examination is specific to the facts and circumstances of that examination and that systemic deficiencies in a bank’s BSA/AML compliance program are generally treated differently than nonsystemic deficiencies. As discussed earlier, FFIEC has been working on updating its entire FFIEC BSA/AML examination manual, including updates to more clearly state the agencies’ approach to risk-based supervision, according to OCC officials. Representatives from two of the four banking associations with which we spoke with stated they were involved in providing input on recent updates to FFIEC’s examination manual and all four had provided input to the effort to implement the customer due diligence and beneficial ownership rule. For example, OCC officials said that the risk-based approach is most clearly discussed in the opening pages of the current FFIEC manual and could be more directly incorporated throughout the manual to provide enhanced guidance to examiners. These officials stated that the agencies have been drafting proposed edits for drafting group consideration. More generally, FFIEC undertook its Examination Modernization Project as a follow-up to reviews required under the Economic Growth and Regulatory Paperwork Reduction Act. One of the project’s efforts seeks feedback from selected supervised institutions and examiners on ways to improve the examination process. For example, the FFIEC examination modernization project reviewed, compared, and identified common principles and processes for risk-focusing examinations of community financial institutions. FFIEC members also committed to issue reinforcing and clarifying examiner guidance on these risk-focused examination principles. In addition, Treasury, FinCEN, and the banking regulators established a working group to identify ways to improve the efficiency and effectiveness of BSA/AML regulations and supervision. In October 2018, the working group issued a joint statement to address instances in which banks with less complex operations and lower-risk BSA/AML profiles may decide to enter into collaborative arrangements with other banks to share resources to manage their BSA/AML obligations in order to increase efficiency and reduce burden. In December 2018, the working group issued another joint statement that recognized that banks may use existing tools in new ways or adopt new technologies to more effectively and efficiently meet their BSA/AML obligations. SEC shares responsibility for broker-dealer examinations with SROs, but has sole responsibility for examinations of mutual fund companies and maintains supervisory authority over SROs. SEC’s Office of Compliance Inspections and Examinations conducts risk-based examinations of regulated entities including mutual funds (under the Investment Adviser/Investment Company Examination Program) and broker-dealers (under the Broker-Dealer Exchange Examination Program). According to SEC documentation, the scope of examinations is based on a risk assessment of various factors such as the type of business a firm engages in and its customer base. This includes consideration of whether the firm engages in high-risk activities. The Office of Compliance Inspections and Examinations assesses the risks from information sources such as tips, complaints and referrals, FinCEN BSA data, pre- examination due diligence, and previous examination history. During the period we reviewed, BSA/AML examinations of mutual funds accounted for less than 1 percent of all securities BSA/AML examinations and no mutual funds were cited for violations of BSA. SEC staff said investors primarily purchase shares of mutual funds through a distributor (such as a broker-dealer) and, in these cases, mutual funds do not know, and are not required to know, the identities of individual investors. In these cases, the broker-dealer distributor has more information about the individual investors and may be examined for BSA compliance as part of FINRA and SEC BSA examinations. FINRA conducts the majority of examinations of broker-dealer firms and imposes anti-money laundering rules on its members. FINRA officials told us that they use a risk-based approach for AML examinations, which considers the size, complexity, customer types, and risks posed by business activities in assessing potential BSA/AML risk. These risk factors affect the timing of their reviews (for example, if a broker-dealer is deemed to be higher-risk, it will be examined in the same year it was assessed). According to FINRA officials they have different expectations for firms’ AML programs, based on size (larger firms typically are expected to have more complex AML programs than smaller firms). FINRA publishes a template for small firms to help them fulfill their responsibilities to establish an AML compliance program. The template provides text examples, instructions, relevant rules, websites, and other resources useful for plan development. However, representatives from a securities industry association told us that BSA/AML rulemaking and examinations sometimes do not take into account the varying levels of risk of different types of business models and activities among firms. Furthermore, these representatives stated that sometimes compliance expectations are communicated through enforcement actions rather than through rulemaking or guidance. As noted previously, one of FinCEN’s has been reviewing the risk-based approach to the examination process. According to a February 2019 speech by the Director of FinCEN, FinCEN’s initiatives also included reviewing agencies’ approach to supervision and enforcement and identifying better ways to communicate priorities. Representatives from this securities industry association also identified certain training and tools on BSA/AML compliance and implementation that FINRA and SEC staff provide as helpful to the securities industry in identifying priorities and compliance deficiencies. For example, SEC’s Office of Compliance Inspections and Examinations and FINRA publish annual examination priorities, which identified both customer due diligence and suspicious activity monitoring as key areas for 2019. According to SEC staff, SEC and FINRA examination priorities have identified suspicious activity monitoring as a key area for the past several years and have identified customer due diligence as a priority since the implementation of the customer due diligence rule in 2018. FINRA published examination findings for the first time in 2017 and again in 2018, including selected findings related to BSA/AML compliance, which representatives from the industry association said have been very useful because they describe specific BSA/AML compliance deficiencies identified by FINRA across the industry and can assist firms in improving their compliance programs. Additionally, FINRA and SEC included an AML-topic in their 2017 National Compliance Outreach Program for broker-dealers. SEC also occasionally publishes risk alerts on its website and participates in industry outreach efforts. The SROs that conduct the majority of examinations of futures firms use a risk-based approach. CFTC has authority to examine futures commission merchants and futures and commodities introducing brokers, but does not routinely conduct examinations of the firms it supervises. Instead, CFTC oversees the examinations conducted by its SROs. CFTC delegated examination authority to two SROs—NFA and the CME Group. NFA conducts the majority of BSA examinations and is the only SRO that examines independent introducing brokers. During the period we reviewed, NFA was assigned the majority of futures firms and conducted a majority of AML examinations. NFA and CME Group stated in CFTC’s annual reports to FinCEN that they utilize a risk-based approach for AML examinations. CME Group reported that it determined both the frequency and the scope of examinations through an overall assessment of the financial and operational risks posed by a futures commission merchant. NFA is required to examine futures commission merchants annually, but reported that the timing and frequency of introducing broker examinations were based predominately on the risks present with a firm. NFA’s risk models measure the riskiness of each firm, and firms are prioritized for examination based on the output from the risk model. In an interpretative notice, NFA recognized that financial institutions vary in size and complexity, and that firms should consider size, among other factors (such as the nature of business and its risks to money laundering) in designing a program to implement requirements such as customer verification and suspicious activity reporting. Representatives from a futures industry association told us that there is a one-size-fits-all approach to BSA/AML compliance in that the rules are broadly applied to varying types of financial institutions. They noted that BSA/AML guidance tends to focus on banks and treat other types of financial institutions (money service business, casinos, and others) as one group, despite their diversity. In relation to the futures industry, the representatives stated that this makes it difficult for futures commission merchants to implement requirements because the rules or guidance do not necessarily take into consideration their unique business structure. CFTC staff told us that BSA requirements could be applied differently to different types of firms and were supportive of tailoring or reducing requirements where the obligations were duplicative, overly burdensome, and BSA-related risks were low. For example, CFTC staff recommended that FinCEN relieve (1) certain introducing brokers known as voice brokers and (2) futures commission merchants that are initial clearing firms from customer identification program requirements because they have limited interaction with the customer and do not have access to customer information that would allow them to perform customer due diligence. CFTC staff told us they have been working with FinCEN on implementing these recommendations. In July 2019, FinCEN issued additional guidance on the application of the customer identification program rule and the beneficial ownership rule to certain introducing brokers, which stated that an introducing broker that has neither customers nor accounts as defined under the customer identification program rule has no obligations under that rule or the beneficial ownership rule. IRS examination staff use a risk-based approach to examine for BSA/AML compliance. In 2008, FinCEN and IRS issued a manual for use by IRS (and state regulator) examiners who perform risk-based examinations of money services businesses, which are a category of nonbank financial institutions. The BSA/AML manual for money services businesses states that examiners should determine the appropriate depth and scope of the examination procedures based on their assessment of the risks of the businesses. Specifically, the manual also states examiners should scope their examinations based on their assessment of the risks, which they can assess by analyzing information including the business’ BSA/AML risk assessment and AML compliance program, and then conduct selective transaction testing to determine if the AML program is effective. The amount of transaction testing will vary based on the assessed level of risk—the amount of testing would be reduced if the examiner determined the risks were minimal. IRS officials said that IRS examiners do not perform scheduled examinations of all money services businesses every year; rather, they review a percentage of businesses each year based on risk-related factors such as a history of noncompliance, high-risk geographic areas, and financial institutions identified by referrals. Thus, there may be some money services businesses that are not examined for years and some that are examined much more frequently. As discussed earlier, FinCEN has been updating the BSA/AML Manual for money services businesses. According to the manual, IRS examiners should consider size, among other things, as a factor in their examination approach. IRS officials with whom we spoke said that smaller money transmitters may not have the resources or understand monitoring methods necessary to implementing BSA/AML compliance programs such as suspicious activity monitoring and reporting. IRS procedures state that it is the responsibility of BSA examiners to ensure the financial institution is informed of reporting, registration, recordkeeping, and compliance program requirements of the BSA. IRS officials further explained that they share methods of detecting suspicious activity with small money transmitters to help them meet their requirements. FinCEN enforcement actions can be based on sources that include referrals from examining authorities, information from financial institutions, interviews, and leads from law enforcement. Supervisory agencies, including the federal banking regulators, SEC, CFTC, and their respective SROs are to promptly notify FinCEN of any significant potential BSA violations. IRS also makes referrals to FinCEN for violations it identifies in its BSA examinations, such as willful violations of AML program requirements and recordkeeping and reporting regulations and structuring. Additionally, financial institutions can self-report violations, DOJ or other law enforcement agencies may provide leads, and FinCEN personnel can refer potential violations to FinCEN’s Enforcement Division to be investigated. According to FinCEN officials, after receiving a referral FinCEN’s Enforcement Division opens a case in the Financial Intelligence Repository System, and Enforcement Division staff and management evaluate the circumstances of the alleged violation and provide a written recommendation for action. FinCEN generally resolves its referrals through one of three ways: (1) closing the case without contacting the subject of the referral, (2) issuing a letter of warning or caution to the subject institution or individual, or (3) assessing a civil monetary penalty. According to FinCEN officials, management in the Enforcement Division approve which action will be taken to close the referral, and if the recommendation is to pursue some type of civil enforcement action—the Director of FinCEN and the Office of Chief Counsel would be involved in that determination. FinCEN officials said that factors the Enforcement Division considers when determining which action to recommend or take include: any impact or harm to FinCEN’s mission by identified violations; pervasiveness of the violations; the gravity and duration of the violations; the institution’s history of violations; continuation of the activity; possible obstruction or concealment; any remedial actions taken by institution; and whether the institution received financial gain or benefit from violation. According to FinCEN officials, the Enforcement Division maintains an administrative record for all cases that result in an enforcement action, and when the action is complete, the Financial Intelligence Repository System is updated to reflect that the referral is closed. From January 1, 2015, to September 25, 2018, FinCEN received 419 referrals directly from supervisory agencies (see table 3). Two reports have noted some issues associated with referrals to FinCEN, including delays in reporting by an agency and inconsistent status updates from FinCEN to agencies. A 2018 report by the Treasury Inspector General for Tax Administration found FinCEN had long delays in processing IRS referrals and assessed penalties on a small proportion of referrals. For example, 49 of 80 cases referred by IRS during fiscal years 2014–2016 remained open as of December 31, 2017, and FinCEN assessed penalties in six of the 80 referrals. In response, FinCEN management said the primary reason for not processing referrals was the “age” of the violations when the referral was made to FinCEN, which according to FinCEN officials impedes a thorough investigation of the violations due to an imminent expiration of the applicable statute of limitations. The report recommended that IRS consider having its FinCEN referral process reviewed by process experts to make it more efficient because delays in submitting cases to FinCEN could lead to FinCEN taking longer to process referrals or not considering cases for further civil penalty. In response to the recommendation, IRS stated that it completed a process improvement review of its FinCEN referral process, and had since updated its internal guidelines (in February 2019) to reflect the improved procedures. The Office of Inspector General of Treasury reported in 2016, among other findings, that several federal and state regulators told it that FinCEN did not routinely inform them of the status of their referred cases. The Office of Inspector General recommended that FinCEN implement a process to periodically notify federal and state regulators of the status of and actions taken on referred cases. In its response, FinCEN agreed with the recommendation, and stated that it follows its standard operating procedures for case processing. FinCEN’s response stated that its case processing procedures provide that in all FinCEN enforcement actions taken in coordination with other government partners (including other regulators), FinCEN’s Enforcement Division will provide regulators with a copy of FinCEN’s consent order that details the violations, factual findings, and proposed settlement terms. FinCEN also noted that its Enforcement Division holds standing and ad hoc meetings with each of its federal regulatory partners to discuss, among other matters, the status of top priority referrals. Treasury’s Office of Inspector General closed the recommendation based on FinCEN’s response and its review of FinCEN’s standard operating procedures—which it said included procedures to provide regulators with a copy of FinCEN’s approved consent order and proposed settlement terms in the case of formal enforcement actions. FinCEN officials also told us that FinCEN has been working to update and finalize its policies and procedures to further address the recommendation from Treasury’s Office of Inspector General, but did not have a time frame for completion. When FinCEN assesses a penalty for BSA violations, it may do so independently or concurrently with supervisory agencies. In a concurrent action, FinCEN will assess a penalty with the other regulator and has sometimes deemed the penalty (or a portion of its penalty) satisfied by a payment to the regulator. FinCEN took 26 enforcement actions over the period we reviewed (from fiscal year 2015 through the second quarter of fiscal year 2018), five of which were concurrent with supervisory agencies. Casinos, depository institutions, and money services businesses each had eight enforcement actions and a precious metals firm and a securities/futures firm had one each. In December 2018, FinCEN assessed a $14.5 million civil monetary penalty against UBS Financial Services, $5 million of which was paid to Treasury and the remainder satisfied by payment of penalties for similar or related conduct imposed by SEC and FINRA. Federal banking regulators identify and cite violations of BSA/AML requirements as part of the supervision process, including the examination process. The regulators employ progressive enforcement regimes to address supervisory concerns that arise during the examination cycle or through other supervisory activities. If the institution does not respond to the concern in a timely manner, the regulators may take informal or formal enforcement action, depending on the severity of the circumstances. Informal enforcement actions include obtaining an institution’s commitment to implement corrective measures under a memorandum of understanding. Formal enforcement actions include issuance of a cease-and-desist order or assessment of a monetary penalty, among others. Some factors that the banking regulators reported considering when determining whether to raise an informal enforcement action to a formal enforcement action include the severity of the weakness and the bank’s commitment to correct the identified deficiencies. See appendix II for recent data on enforcement actions taken by the banking regulators. All SEC enforcement actions and all SRO disciplinary actions are public. SEC has authority to enforce compliance with BSA for mutual funds and broker-dealers. If SEC examiners find significant deficiencies with a firm’s BSA program, the examiners may refer the finding to SEC’s Division of Enforcement or an SRO for enforcement. In addition, SEC’s BSA Review Group in the Division of Enforcement’s Office of Market Intelligence may refer matters identified through the review of BSA reports to staff in SEC’s Division of Enforcement and in the Office of Compliance Inspections and Examinations for further consideration and potential follow-up. SEC’s Division of Enforcement will assess whether to proceed with an investigation, determine whether a violation has occurred, and if so, whether an enforcement action should be recommended against the firm or any individuals. In certain cases, SEC’s Division of Enforcement may undertake an investigation where there has been a widespread or systemic failure to file SARs or systemic omission of material information from SARs. When making this assessment, SEC staff said SEC considers a number of factors including: the egregiousness of the conduct, the length of time over which the violations occurred, number of SARs that were not filed or that omitted material information, the disciplinary history of the firm, and adherence to any internal policies and procedures. FINRA has enforcement authority that includes the ability to fine, suspend, or bar brokers and firms from the industry and has two separate procedures (settlement and formal complaint) through which it applies enforcement actions. Through a settlement, a firm or broker in violation can offer to settle with FINRA through a Letter of Acceptance, Waiver, and Consent. A formal complaint is filed with and heard before FINRA’s Office of Hearing Officers. See appendix II for recent data on enforcement actions taken by SEC and FINRA. Although CFTC delegated examination authority to NFA and the CME Group, it retained authority to pursue enforcement actions against futures firms. While CFTC does not typically conduct BSA/AML examinations, it does have a BSA review team that reviews SARs to identify potential violations of futures laws, and CFTC has taken enforcement actions based on leads developed from SARs reviewed. SROs generally conduct BSA examinations of futures firms, and at the conclusion of an examination, the SROs will issue a report to the futures firm to notify the firm of any deficiencies in its AML program. If the deficiencies are not significant, NFA officials stated NFA will cite the deficiency in the examination report and close the examination with no disciplinary action but require corrective action before closing it. If examination findings are significant, then NFA may issue a warning letter or recommend that its Business Conduct Committee issue a formal complaint charging the firm with violating NFA’s AML requirements (which is an enforcement action). NFA officials told us it resolves most enforcement actions related to violations of NFA’s BSA/AML rules through settlement agreements that assess a fine. NFA may take other types of actions for violations of their rules, such as suspension of membership or expulsion. See appendix II for recent data on informal and formal actions SROs took. Although FinCEN has delegated authority to IRS to conduct civil BSA/AML examinations for a variety of nonbank financial institutions and individuals, IRS does not have authority to enforce most civil BSA violations identified. If IRS Small Business/Self-Employed Division examiners find BSA violations when examining an institution, the division can send a letter of noncompliance—a letter 1112—with a summary of examination findings and recommendations to the institution, which also includes an acceptance statement for the institution to sign. Additionally, if IRS Small Business/Self-Employed Division examiners identify significant civil violations during a BSA/AML examination, such as willful violations of BSA reporting and record-keeping requirements, they may refer civil violations to FinCEN or refer certain violations of potential criminal activity to IRS-CI. See appendix II for recent data, including the number of institutions issued a letter 1112. In recent years, Treasury and FinCEN have led efforts to identify BSA goals and priorities such as issuing a national strategy and risk assessments for combating illicit financing crimes. They also established key mechanisms for BSA/AML collaboration, such as interagency working groups, information-sharing agreements, and liaison positions that encompass multiple federal, state, and local agencies and private-sector participants. However, these key mechanisms have been less inclusive of the futures industry than other financial sectors. Treasury and FinCEN led collaborative efforts to identify BSA goals and priorities, including the following: National Strategy. In December 2018, Treasury issued the National Strategy for Combating Terrorist and Other Illicit Financing as required by 2017 legislation. The national strategy discussed various agencies’ BSA-related goals and objectives, including those of the supervisory agencies and law enforcement groups with which we spoke for our review. It also laid out key priorities, such as protecting the United States from terrorist attacks, simplifying the BSA regulatory framework to work more effectively and efficiently, and ensuring the stability of domestic and global markets by reducing fraud, money laundering, and other economic crimes. The strategy also discussed interagency coordination and information-sharing mechanisms (including public-private information sharing). For example, the national strategy states that FBI provided a classified briefing twice a year to selected personnel from the 20 largest financial institutions in the United States to share information on terrorist financing trends. In addition, the national strategy provided data on prosecutions related to money laundering. For example, in fiscal years 2015–2017, DOJ annually charged on average 2,257 defendants with money laundering. Risk assessments. Congress also directed Treasury and relevant agencies to evaluate the effectiveness of existing efforts that address the highest level of risks associated with illicit finance. In December 2018, Treasury issued three risk assessments that identified money laundering, terrorist financing, and proliferation financing risks and describe Treasury’s and relevant agencies’ efforts to address these risks. The three risk assessments underpin the 2018 National Strategy. Treasury involved multiple agencies in the development of the risk assessments, including supervisory agencies, SROs, and several law enforcement agencies. The terrorist financing and money laundering risk assessments built on previous Treasury-led risk assessments issued in 2015, but the 2018 proliferation financing risk assessment was the first ever issued. Treasury’s Strategic Plan (2018–2022) and other guidance. Prior to the publication of the National Strategy, Treasury issued a strategic plan in February 2018 that identified strategies, goals, measures, and indicators of success to meet its strategic goal for preventing terrorists and other illicit actors from using the U.S. and international financial systems. FinCEN also issued advisories or guidance that identify BSA and law enforcement priorities. For example, in February 2014 FinCEN issued guidance that clarified how financial institutions should align their BSA reports to meet federal and state law enforcement priorities if the institutions provide services to marijuana-related businesses. The related federal and state law enforcement priorities included preventing the proceeds of marijuana sales from going to criminal enterprises, gangs, and cartels. Two industry associations (with which we spoke before the issuance of the December 2018 national strategy and risk assessments) noted the importance of establishing BSA priorities to better inform industry. For example officials from one industry association said that Treasury’s risk assessments identified priorities and suggested that it produce these types of reports more frequently (for example annually). This may be addressed, in part, by Congress’ requirement that the national strategy— including a discussion on goals, objectives, and priorities—be updated in 2020 and 2022. In addition, Treasury has been conducting a broad review of BSA/AML laws, regulations, and supervision—focusing on how effectively current requirements and related activities achieve the underlying goals of the BSA. Interagency working groups, interagency memorandums of understanding, and liaison positions, as shown in table 4, are key BSA/AML collaborative mechanisms that were identified through our interviews with officials from FinCEN, supervisory agencies and law enforcement agencies and in agency documents. Bank Secrecy Act Advisory Group (BSAAG). Congress directed Treasury to create BSAAG in 1992. The group, led by FinCEN, is the primary and longest-established BSA/AML collaboration mechanism and is used to share information and receive feedback on BSA administration. The advisory group meets twice a year and includes working groups on BSA/AML-related issues that may meet more frequently. BSAAG recently has been focusing on improving the effectiveness and efficiency of the regulatory and supervisory regime. SEC and Federal Reserve officials told us that BSAAG is a helpful and effective collaborative mechanism to discuss BSA/AML issues. However, as we discuss later, representatives from CFTC, the primary futures SRO, and a futures industry association expressed concerns that the futures industry was not as well represented on BSAAG as other industries. FinCEN invites the public to nominate financial institutions and trade groups for 3-year membership terms on BSAAG. In making selections, the Director of FinCEN retains discretion on all membership decisions and seeks to complement current BSAAG members in terms of affiliations, industry, and geographic representation. Memorandums of understanding (MOU). FinCEN established interagency agreements—information-sharing and data-access MOUs— relating to BSA data. For example, FinCEN entered into an information- sharing MOU with the federal banking regulators in 2004 and has since established similar MOUs with other supervisory agencies, including many state supervisory agencies. FinCEN consolidates the data from the four federal banking regulators and told us that it shares the consolidated reports with banking regulators. In addition, FinCEN officials told us they use data from the information-sharing agreements to help in certain initiatives and training. For example, FinCEN officials told us that a recently funded initiative focused on nonbank financial institutions will use information from the MOUs to proactively identify risks and better inform related compliance efforts. All the supervisory agencies told us they informally update and monitor their information-sharing MOUs through frequent meetings and regular communication with FinCEN. For example, FinCEN officials told us they have been working to update how they collect information on violations related to the customer due diligence and beneficial ownership rule. In addition, FinCEN contracts an annual MOU satisfaction survey that FinCEN officials said helps them monitor the effectiveness of the MOUs. In the survey, respondents were asked about their satisfaction with their MOU and scored their satisfaction around 80 out of 100 in 2017. FinCEN also has more than 400 data-access MOUs with federal, state, and local law enforcement agencies as well as with federal and state regulatory agencies. FinCEN has data-access MOUs, or provides direct data access, with or to all the federal supervisory agencies and with FINRA, a securities SRO—but not with NFA, a futures SRO. As discussed previously, supervisory agencies use these data primarily to help scope and conduct their BSA/AML compliance examinations. In a later section, we discuss access issues in relation to supervision of the futures industry. Law enforcement agencies use BSA data to assist in ongoing investigations and when initiating new investigations. Liaison positions. FinCEN has used on-site liaison positions for more than a decade to help avoid overlap and duplication of efforts. According to FinCEN officials, as of April 2019, FinCEN had 18 law enforcement liaisons from 10 law enforcement agencies. Some law enforcement officials with which we spoke said the liaison position allowed feedback and information exchange between law enforcement and FinCEN. Supervisory agencies generally told us that the liaison program was for law enforcement agencies and that they did not participate. FinCEN officials said that while FinCEN does not have on-site liaisons from supervisory agencies that are comparable in scope to the law enforcement liaisons, they work closely with the supervisory agencies. For example, FinCEN currently has a part-time detailee from FDIC who collaborates on-site at FinCEN with FinCEN analysts. FinCEN officials said they hosted a temporary on-site detailee from NCUA in 2017. NCUA officials told us that they also expressed an interest to FinCEN to implement routine detailing of staff. SEC staff told us that in the past they had a FinCEN detailee onsite working with SEC’s Division of Enforcement, which allowed SEC to better understand FinCEN’s methodology and approaches, and assess their own approaches to BSA enforcement. SEC staff expressed interest in hosting another FinCEN detailee, and the agency has been considering a FinCEN request to send an SEC liaison to FinCEN. There are also other BSA/AML collaborative mechanisms among regulatory or law enforcement agencies, such as the FFIEC BSA/AML working group, SAR review teams, and geographic targeting orders (see table 4). We also obtained perspectives on collaboration from FinCEN and relevant key law enforcement and regulatory agencies on three selected BSA criminal cases, which are discussed in appendix III. The futures industry has been less represented in key mechanisms for BSA/AML collaboration (those related to BSAAG and data-access agreements) than other industries. Representatives from CFTC, the primary futures industry SRO, and a futures industry association expressed concerns that the futures industry was not as well represented on BSAAG as other industries. CFTC, as the delegated supervisory agency, always has been a member of BSAAG. However, the primary futures industry SRO—which has developed rules to implement AML requirements for its members and conducts a majority of AML examinations of futures firms—and futures industry associations have had less consistent participation. Officials from the primary futures SRO expressed concern that they were not a regular member of BSAAG. They noted that they were a BSAAG member in the mid-2000s but then not selected as a member of BSAAG for almost 5 years (from 2014) until they were invited to be a member again in March 2018, at which point, the futures industry association’s BSAAG membership was not renewed when its term expired. Representatives from all key federal supervisory agencies have been regular members of BSAAG. In particular, the securities industry, which also uses SROs to monitor BSA compliance, has had its primary SRO as a member of BSAAG since 2008. Representatives from the primary securities SRO said that their participation in BSAAG allowed them to coordinate BSA/AML efforts. Representatives from the primary futures SRO said that their role regarding oversight of the futures industry was similar to the primary securities SRO. These representatives stated that they adopted AML rules; were the only SRO with jurisdiction over all futures entities subject to AML requirements; and conducted a majority of AML examinations. Accordingly, representatives said that they were in the unique position of seeing first-hand how AML requirements are implemented in the futures industry and identifying issues, as well as potential gaps in implementation. CFTC staff said that all significant representative groups for the futures industry should participate in BSAAG—in particular, the primary futures SRO because it supervises all types of registered firms in the futures industry and the leading industry association for the futures, options, and centrally cleared derivatives markets. In addition, representatives from industry associations we spoke with from other industries have been regular members of BSAAG including banking associations and the primary securities industry association. The primary securities industry association has been a member since 2008, concurrent with the primary securities SRO (also a member since 2008). Representatives from this association said that BSAAG is a mechanism that FinCEN uses to solicit feedback from the industry. Officials from the futures industry association that had previously participated in BSAAG, told us that their current lack of participation may prevent FinCEN from obtaining an in-depth understanding of futures industry issues and may prevent the futures industry from obtaining information on BSA/AML goals and priorities and other key communications. CFTC staff said that in addition to the primary futures SRO, BSAAG also should include a primary industry association. FinCEN officials told us that there is a limit on the number of BSAAG representatives allowed and that they have had a futures representative that was not always an active participant. In addition, FinCEN officials said that when selecting BSAAG members they need to consider the top money laundering risk areas as well as the appropriate number of members to have productive discussions. They added that because membership rotates, additional futures representatives could be added based on needs and topic areas. Furthermore, FinCEN officials told us that although the most recent BSAAG (October 2018) did not include a futures industry association, it did include the primary futures industry SRO and six large diversified financial firms that are listed as members of the key futures industry association. However, these firms represent a small percentage of the association’s membership and are not smaller firms or clearing organizations, exchanges, and global and regional executing brokers. As noted in Treasury’s 2018 national strategy, BSAAG is the main AML information conduit and policy coordination mechanism among regulators, law enforcement, and industry and has been focusing on improving the effectiveness and efficiency of the regulatory and supervisory regime. Without regular participation by the primary futures SRO that has developed AML rules and conducts the majority of BSA examinations for the futures industry, FinCEN may be missing opportunities to better understand compliance in the futures industry and the SRO may not be fully up to date on BSA/AML compliance issues and related initiatives that may affect the AML rules it develops. Furthermore, without representation on BSAAG by the key futures industry association, the diverse array of futures industry participants may not be fully represented, informed, or updated on key BSA/AML information. Standards for Internal Control in the Federal Government state that management should externally communicate the necessary quality information to achieve the entity’s objective. In addition, the statutory purpose of BSAAG includes informing private-sector representatives of how BSA reports have been used and receiving advice on how reporting requirements should be modified. Additional futures industry representation on BSAAG could enhance both regulator and industry awareness of BSA/AML compliance issues and potential money laundering risks. In addition, NFA, the SRO conducting the majority of BSA examinations for the futures’ industry, does not have direct access to BSA data—unlike all key supervisory agencies and FINRA. In our 2009 report, we recommended that FinCEN expand data-access MOUs to SROs conducting BSA examinations that did not already have direct access to BSA data. In 2014, FinCEN completed a data access MOU with FINRA. But it did not pursue an MOU with NFA because, at that time, CFTC did not ask FinCEN to arrange an MOU with NFA. However, CFTC staff, as of April 2019, told us that access to BSA data would enhance the tools that NFA has to perform its functions, including its ability to scope and perform BSA/AML examinations, and to use BSA data more extensively and more frequently. Currently, when conducting its examinations, NFA must obtain SAR information from CFTC, as well as reviewing SARs provided by a firm while conducting an on-site examination. FinCEN officials told us that NFA has not requested direct access to BSA data. However, NFA representatives told us they welcomed a discussion with CFTC and FinCEN on the benefits and drawbacks of having direct access to BSA data. FinCEN officials said they would need to better understand any negative impacts of NFA not having direct access and NFA would need to meet the required criteria to obtain direct access. Standards for Internal Control in the Federal Government state that management should externally communicate the necessary quality information to achieve the entity’s objectives. Supervisory agencies with direct data access all have utilized BSA data to some extent to scope and plan examinations. Direct access to BSA data would enhance NFA’s ability to scope BSA examinations and generally conduct its oversight responsibilities for BSA in the futures industry. FinCEN and two law enforcement agencies with which we spoke generated metrics on the usefulness of BSA reporting—such as the number of BSA reports that led to new investigations. But FinCEN, whose role it is to collect and disseminate BSA data, has not consistently communicated these metrics—instead only communicating some available metrics on an ad-hoc basis through methods such as published speeches or congressional testimonies. FinCEN and nearly all the law enforcement agencies with which we spoke provided some feedback to financial institutions on how to make BSA reports more useful through formal mechanisms (such as conferences and training sessions) and informal relationships. However, institution-specific feedback, which all industry groups said their members preferred, has not been widely provided. Two of the six law enforcement agencies (IRS-CI and FBI) we interviewed produced metrics on the usefulness of BSA reporting (for example, percentage of investigations utilizing BSA data). However, FinCEN (which has statutory responsibilities for the central collection, analysis, and dissemination of BSA data) did not consistently communicate this information, but rather communicated on an ad hoc basis through published speeches or congressional testimony. IRS-CI annually publishes a report with data on investigations, including those generated by BSA reports. For example, in fiscal year 2018, IRS-CI reported that 515 BSA investigations were initiated (see table 5). FinCEN’s website generally did not refer to IRS-CI metrics, but in a November 2018 congressional testimony, the Director of FinCEN included information on the percentage of IRS-CI investigations that began with a BSA source— 24 percent in fiscal year 2017. In addition, IRS-CI also tracks the work of SAR review teams and has created some metrics on the usefulness of BSA reporting, including: the number of investigations initiated, indictments, convictions, sentenced, and total dollars seized based on the work of the SAR review teams (see table 6). While this information is not routinely reported publicly, IRS officials said they have shared information about results from SAR review teams’ during presentations to the public, law enforcement, and financial industries. FBI analyzes BSA filings to support existing cases and initiate new investigations, and FBI and FinCEN have reported related metrics to the public, but not routinely. FBI created a BSA Alert System that searches subjects’ names, dates of birth, Social Security numbers, telephone numbers, email addresses, and other identifying information across BSA filings, and automatically emails the results to agents. In a November 2018 congressional testimony, the FBI section chief of its Criminal Investigative Division stated that these searches produce an average of 2,000 alerts per month and provided statistics on the results of the agency’s use of BSA data. From January 2017 to June 2018, BSA reporting was directly linked to the main subject of approximately 25 percent of pending FBI investigations (up from 8.9 percent in 2012). The November 2018 FBI testimony also described FBI’s use of SARs data analysis to identify new cases. For example, FBI analysts run a series of search terms and criteria related to money laundering, terrorist financing, human trafficking, fraud, corruption, transnational organized crime, and other schemes against SAR filings. The persons identified through the searches are automatically searched against FBI case files and watchlist data, and the results incorporated into reports to appropriate field offices. FinCEN also communicated some of the FBI metrics in an August 2018 speech by the FinCEN director. For example, the director said more than 20 percent of FBI investigations utilized BSA data and for some types of crime, like organized crime, nearly 60 percent of FBI investigations used BSA data. The other four law enforcement agencies with which we spoke did not generate metrics on the usefulness of BSA reporting due to confidentiality or data reliability concerns, among other reasons, but some tracked other BSA-related efforts. DHS officials said that while they do not have any metrics on the usefulness of BSA reports, the agency provided data on the usefulness of ICE-HSI’s Cornerstone outreach program—in which ICE-HSI provided training to financial institutions on issues such as trends in how criminals earn, move, and store illicit proceeds. ICE-HSI reported that in fiscal year 2017, based on the Cornerstone outreach program, special agents initiated more than 72 financial investigations, made 55 criminal arrests, and seized almost $2 million in illicit proceeds. Secret Service officials said that they have been trying to develop an internal tracking system for their use of BSA reports, but were not tracking any metrics as of April 2019. They told us that they use BSA data for investigative purposes only and they do not discuss or report it, because they consider it confidential information—thus making it difficult for them to gather metrics on the use of BSA reports. An official from DOJ’s Criminal Division said that the division has not established any performance measures or collected any statistics that measure the effectiveness of BSA record-keeping and reporting requirements (for example, because the success of investigations depending on multiple factors not just BSA reporting, and other challenges described later in this report). However, the official said that the division recognizes the usefulness of BSA data in criminal investigations because the data help them with prosecutions of crimes. Officials from DOJ Executive Office for United States Attorneys said that they track the number of cases with statutory provisions relating to BSA in which the U.S. Attorney’s Offices prosecuted or enforced BSA violations. However, the officials said their case management system does not track if BSA filings were used to initiate or assist the case. Supervisory agencies we interviewed generally said FinCEN and law enforcement are better positioned to compile metrics on the usefulness of BSA reporting because FinCEN and law enforcement agencies are the primary users of BSA reports. However, two of the seven supervisory agencies in our review that also have law enforcement functions—SEC and CFTC—have their own BSA review teams, which analyze SARs to identify potential violations of federal laws, including BSA violations, and refer matters for further examination or investigation as appropriate. For example, on average, from fiscal years 2016 to 2018, SEC’s BSA review team reviewed about 27,000 SARs each year that related to current or potential investigative matters, or entities regulated by SEC. CFTC staff told us they review an estimated 7,500–8,000 SARs annually. On average, in about 100 instances a year, CFTC’s BSA review team refers SARs to investigative teams in support of new or existing investigations. As of December 2018, CFTC staff said they had taken 33 enforcement actions based on leads developed from SARs, with two of the actions related to BSA/AML violations. FinCEN collected some metrics on the usefulness of BSA data through annual surveys and other initiatives; however, the survey results are not public and other metrics are not regularly published. FinCEN contracts an annual survey that includes questions to BSA data users (such as federal and state law enforcement and regulators) about the usefulness of BSA data to, among other things, provide new information or supplement known information or identify new leads or investigations. BSA data users are asked to score the value and impact of BSA data and scored it at about 80 out of 100 for both 2016 and 2017. FinCEN contracts another survey that solicits feedback on the 314(a) program. The 2017 survey found the respondents that utilized the 314(a) program gave it high scores for its usefulness—close to 90 out of 100 for 2016, and 2017. The results from both surveys are not publicly available. In addition, FinCEN periodically publishes a 314(a) Fact Sheet that contains some data on the usefulness of the 314(a) program—such as the number of 314(a) requests and the percentage of requests that contributed to arrests or indictments. Based on information FinCEN collected from law enforcement, approximately 95 percent of 314(a) requests contributed to arrests or indictments. In addition, FinCEN reported the number of cases submitted and related subjects of interest identified in 314(a) requests for each 2-week period from January 5, 2016, to January 29, 2019. For example, for the 2-week period starting on January 29, 2019, 16 requests resulted in 162 subjects of interest. FinCEN contracted a study on the value of BSA reporting—which began in January 2019 and is to be completed by the end of 2019—with the goal of identifying common attributes of BSA value among stakeholders; assessing how to use available data to establish metrics for evaluating and calculating the value of BSA; identifying gaps in data and other information needed to measure the value of BSA reporting; and proposing actions to improve FinCEN’s ability to identify, track , and measure the value of BSA reporting. However, the performance work statement for FinCEN’s BSA value study, which outlines the objectives for the study, does not include actions related to communicating such metrics. As discussed above, FinCEN has not consistently communicated available metrics. FinCEN officials told us their current approach was to communicate metrics through mechanisms such as speeches and congressional testimonies. FinCEN officials told us that it has an ongoing initiative to create a new communication strategy incorporating the results of the BSA value study—but had no time frame for its completion. Our prior work found that agencies can implement a number of practices that can enhance or facilitate the use of performance information— including communicating performance information frequently and routinely. In addition, Standards for Internal Control in the Federal Government state that management should externally communicate the necessary quality information to achieve the entity’s objectives. Officials from some supervisory agencies and most industry associations also told us they would like FinCEN to provide them with more aggregated data on the usefulness of SARs filed by financial institutions. By consistently communicating currently available metrics on the usefulness of BSA reporting to industry, and any metrics later identified by FinCEN’s BSA value study, financial institutions may be able to more fully understand the importance and outcomes of their efforts. FinCEN and nearly all of the law enforcement agencies with which we spoke provided some feedback to financial institutions on how to make BSA reports more useful through formal mechanisms (such as conferences and training sessions) and informal relationships. However, institution-specific feedback, which all industry groups said their members preferred, has not been provided on a regular basis and only on a small scale. FinCEN’s feedback mechanisms include a new information exchange program, advisories, and BSAAG. For example: FinCEN Exchange. On December 4, 2017, FinCEN publicly launched the FinCEN Exchange, a public-private program that brings together law enforcement, FinCEN, and different types of financial institutions to share information to help identify vulnerabilities and disrupt money laundering, terrorist financing, and other financial crimes. As of December 2018, FinCEN convened more than a dozen briefings with law enforcement agencies across the country, involving more than 40 financial institutions. According to Treasury’s 2018 national strategy, the information provided by financial institutions through SARs after the briefings helped FinCEN map and target weapons proliferators, sophisticated global money laundering operations, human trafficking and smuggling rings, and corruption and trade-based money laundering networks, among others. FinCEN officials told us that these exchanges provide a forum in which law enforcement can request specific information and provide information on typologies to financial institutions that allows financial institutions to improve their BSA monitoring and reporting. FinCEN advisories. FinCEN issues public and nonpublic advisories to financial institutions to help financial institutions better detect and report suspicious activity related to a particular risk and related typology. For example, in October 2018 FinCEN posted an advisory on its website to alert U.S. financial institutions of the increasing risk that proceeds of political corruption from Nicaragua might enter the U.S. financial system. It also posted an advisory on the Iranian regime’s illicit activities and attempts to exploit the financial system. These advisories included specific instructions on how to file SARs related to this type of suspicious activity. Some of the industry associations with which we spoke had positive feedback on FinCEN advisories and said they would like to see more red flags and specific guidance to help improve their BSA monitoring programs. BSAAG. Among its functions, the advisory group serves as a forum for industry, supervisory agencies, and law enforcement to communicate about how law enforcement uses SARs and other BSA data. For example, sometimes law enforcement agencies present specific cases using BSA data or information on money laundering and terrorist financing threats. Many of the industry associations and supervisory agencies with which we spoke cited BSAAG as a useful feedback mechanism. As discussed previously, the advisory group is only open to those invited and not a public forum, so not all financial institutions receive or can provide feedback at these meetings. Law enforcement awards. FinCEN officials said that annual law enforcement awards ceremonies are one of the mechanisms they use to provide financial institutions with feedback on the usefulness or effectiveness of BSA/AML information. The award ceremonies highlight successful cases utilizing BSA data. FinCEN officials told us that FinCEN also sends thank you letters to the selected financial institutions that provided the underlying financial data used in the awarded cases, publishes overviews of the cases for which law enforcement agencies received awards, and documents nominated cases. FinCEN issues press releases about the winning cases as another way to share information with financial institutions. Outreach events. FinCEN representatives regularly have participated in outreach events about BSA/AML issues, such as by sharing information at BSA/AML conferences. According to FinCEN officials, the conferences allow FinCEN representatives to both formally (speeches, presentations) and informally (personal interactions) solicit and offer feedback on how financial institutions can improve BSA reporting. Additionally, Treasury reported that its Office of Terrorism and Financial Intelligence regularly engages public and private-sector practitioners and leaders, both domestic and international, on money laundering and terrorist financing issues. For example, the office convenes multilateral and bilateral public-private sector dialogues with key jurisdictions and regions to discuss mutual anti-money laundering and counter-terrorist financing issues of concern. Representatives from nearly all of the federal law enforcement agencies we interviewed said that they conducted outreach events and developed relationships with financial institutions to solicit and provide feedback on their BSA reports including providing feedback on ways to improve BSA reporting and to enhance BSA compliance by financial institutions. Conferences. Law enforcement agencies have presented at conferences on BSA/AML topics and host conferences for financial institutions. For example, for more than a decade ICE-HSI, FBI, Secret Service, IRS-CI, and the Drug Enforcement Administration jointly have hosted an annual conference that includes speakers from law enforcement, supervisory agencies, FinCEN, and financial institutions. According to an ICE-HSI official, the intent of the conference is to educate the private financial sector. FBI officials also said they conduct outreach, such as hosting and participating in conferences, and said that this type of outreach reached more than 6,000 people in the last year (as of August 2018). Briefings and financial institution-specific training. Some law enforcement agencies have their own outreach programs on BSA topics for financial institutions. For example, ICE-HSI has the Cornerstone Outreach Program that began to work with the private sector in 2003 to identify money laundering vulnerabilities in the financial system. The program is to encourage partnerships with the private sector by sharing distinguishing traits or forms of criminal behavior (either crime-centered or person-centered) and methods, and providing training to financial institutions. ICE-HSI officials said they conducted about 300 Cornerstone Outreach presentations in fiscal year 2018. FBI officials also told us they host a couple of meetings annually for financial institutions and sometimes conduct institution-specific training upon request, such as on SAR usefulness. FBI officials told us that for the institution-specific SAR trainings, they change the information on the SARs for training purposes and highlight how institutions can improve SAR filings. They also provide some summary-level statistics and work with the financial institution’s SAR teams to train them on trends. They estimated they conduct from about eight to 10 such sessions annually (as of April 2019). Informal relationships with financial institutions. Officials from nearly all the law enforcement agencies with whom we spoke said they have informal relationships with financial institutions to solicit and provide feedback on their BSA reports. Most supervisory agencies we interviewed said that they did not provide feedback to financial institutions on the usefulness of their BSA reporting due to factors such as law enforcement being better positioned to provide feedback and SAR confidentiality restrictions. However, CFTC staff noted that their BSA review team communicates the general usefulness of SARs filed by their institutions at conferences and through telephone contacts with the filer after the relevant case is filed. SEC staff told us they do not reach out directly to provide financial institutions specific feedback on the usefulness of SARs, but provide training on what makes a good or bad SAR through routine interaction with the primary securities industry association and presentations at BSAAG. As discussed earlier, some supervisory agencies regard FinCEN and law enforcement as the primary end users of BSA reports, and thus, in a better position to provide feedback to financial institutions on BSA reporting. Additionally, many supervisory agencies told us that it would be helpful if FinCEN and law enforcement could provide more frequent or systematic feedback on financial institutions’ SAR reporting. Some supervisory agencies, industry associations, and law enforcement agencies with which we spoke identified limitations with some of FinCEN’s feedback mechanisms, including FinCEN Exchange and law enforcement awards. Representatives from all the industry associations we spoke with indicated that financial institutions would like to see more institution-specific feedback on their SARs to improve their monitoring systems and reporting. FinCEN Exchange. Some industry associations appreciated FinCEN’s outreach, but noted that the new FinCEN Exchange program was on a small-scale and industry associations had not been invited to participate or provide feedback. An official from one industry association said that the association could help identify banks, such as community banks, that could be a good fit for the program. Supervisory agencies also generally said they were not involved in the FinCEN Exchange program. Officials from OCC said that they would like to be involved because they are the primary regulator for many of the financial institutions in the program and thought their participation would add value. Some law enforcement agencies had some concerns about the FinCEN Exchange program, such as private-sector representatives not being properly vetted or the risk of talking about ongoing investigations. For example, officials from ICE-HSI and FBI told us their institution-specific training included only vetted or trusted financial institutions. FinCEN officials said that they collaborated with regulators on the FinCEN Exchange and solicited feedback on the program from certain industry associations. In addition, FinCEN posts frequently asked questions about the FinCEN Exchange program on its website and encourages feedback from financial institutions on how they can support FinCEN priorities such as information sharing. FinCEN officials said that the FinCEN Exchange is an invitation-based program and that FinCEN vets information received from financial institutions and consults with law enforcement, as appropriate, to convene a briefing. Furthermore, FinCEN’s frequently asked questions about the program note that financial institutions that voluntarily participate in a FinCEN Exchange briefing must adhere to the terms noted in FinCEN’s invitation, including any requirement of confidentiality given the sensitivity of information provided. Awards. Representatives from CFTC, FBI, and three industry associations with whom we spoke made suggestions for expanding FinCEN’s law enforcement awards and related thank you letter initiatives. For example, CFTC suggested that FinCEN expand the awards program to include civil cases as well as criminal cases. FinCEN officials also told us in April 2019 that they were considering awards for civil cases. Industry associations generally said their member financial institutions appreciated receiving thank you letters, but some noted that there were limitations with these letters. For example, a representative from one industry association said that only a small percentage of financial institutions receive the awards, and representatives from another industry association said that the letters should provide more specific feedback. Two other industry associations said that the confidential nature of SARs makes it difficult to share the success of the financial institution that submitted the reporting. Many law enforcement agencies with which we spoke said that the law enforcement awards were a good idea, and FBI officials recommended creating awards for the financial institutions as well. FinCEN officials stated that due to SAR confidentiality rules, it cannot publicize awards to financial institutions. Institution-specific feedback. Representatives from all the industry associations with whom we spoke told us, or have publically stated that financial institutions would like to see more institution-specific feedback on their SARs to improve their monitoring systems and reporting. SAR reporting is labor-intensive for financial institutions because it requires researching and drafting narratives for a SAR filing and justifying cases where a SAR is not filed, according to many industry association representatives. However, many representatives said that financial institutions get little institution-specific feedback on their SAR reporting. We found that while law enforcement conducts some small group briefings that industry associations said were useful, these briefings cover a small number of financial institutions in relation to the size of the U.S. financial industry. ICE-HSI stated that it conducted 302 institution-specific trainings and briefings in fiscal year 2018, and FBI, as discussed previously, estimated it has conducted from about eight to 10 institution- specific SAR reporting trainings annually in relation to the more than 10,000 depository institutions, more than 26,000 money services businesses registered with FinCEN, and almost 4,000 active broker- dealers registered (as of January 2019). The American Bankers Association, Independent Community Bankers of America, and The Clearing House all have issued papers—recommending more institution- specific feedback on financial institution SAR reporting. Some industry associations and other stakeholders pointed to international efforts that provided feedback through public-private partnerships. For example, the United Kingdom’s Joint Money Laundering Intelligence Taskforce (joint task force), formally established in May 2016, includes regulators, law enforcement, and more than 40 major United Kingdom and international banks conducting a large proportion of financial activity in the United Kingdom (89 percent of the volume of personal accounts in the United Kingdom). The joint task force has a system in place to routinely convene these partners, included vetted banking representatives, to set AML priorities and share intelligence. According to the intergovernmental Financial Action Task Force’s (FATF) mutual evaluation report of the United Kingdom, financial institutions involved in the joint task force are required to file SARs for suspicious activity identified through the program, and these SARs are considered to be of high value. FATF’s report also noted that the joint task force is considered to be best practice in public-private information sharing. According to Treasury’s 2018 national strategy, FinCEN collaborated with the United Kingdom’s joint task force in implementing the FinCEN Exchange program. In prior work, we reported that FinCEN recognized that financial institutions do not generally see the beneficial impacts of their BSA/AML efforts. FinCEN, law enforcement, and some industry associations with which we spoke identified challenges in providing institution-specific feedback to financial institutions on the usefulness of their BSA reporting. In addition to the large number of financial institutions in the United States, officials from FinCEN and law enforcement agencies told us that law enforcement cases may be sensitive and time-consuming, and the unauthorized disclosure of SARs or sharing of certain information with financial institutions might compromise ongoing investigations. Two industry associations also identified the confidential nature of SARs as a challenge for FinCEN and law enforcement to provide institution-specific feedback to financial institutions. As we have discussed, FinCEN has been undertaking a study to better understand the value and effectiveness of BSA. In addition, FinCEN and some law enforcement agencies have made efforts to provide some institution-specific feedback through various methods on BSA reporting, but the feedback has been periodic, sometimes only at the request of financial institutions, and provided on a small scale. FATF standards on information sharing state that anti-money laundering authorities should provide feedback to financial institutions to assist them in complying with anti-money laundering requirements—these mechanisms can include feedback loops, whereby more consistent and more fully explained feedback is provided to the private sector on suspicious transaction reports. FinCEN’s statutory duties also include information sharing with financial institutions in the interest of detection, prevention, and prosecution of terrorism, organized crime, money laundering, and other financial crimes. As discussed, other countries have put in place mechanisms (such as the United Kingdom’s joint task force) to provide regular feedback on AML reporting (including SAR-like instruments) to financial institutions representing a large portion of the country’s financial activity. Additional and more regular institution-specific feedback, designed to cover different types of financial institutions and those with significant financial activity, may enhance the U.S. financial industry’s ability to effectively target its efforts to identify suspicious activity and provide quality BSA reporting. FinCEN, numerous supervisory agencies (covering various financial sectors), and law enforcement agencies are responsible for enforcing the BSA/AML regulatory framework with the end goal of detecting and preventing money laundering and other financial crimes. While these agencies have processes and mechanisms in place to collaborate on key BSA/AML issues, such collaboration and information sharing could be enhanced by additional and more regular involvement of representatives of the futures industry—a complex and unique financial markets sector. Unlike the other key federal supervisory agencies and securities SRO involved in BSA compliance, the primary futures SRO was not consistently included in BSAAG. Thus, FinCEN may be missing opportunities to better understand compliance in the futures industry and the SRO may not be updated on related BSAAG initiatives. The key futures industry association also has had less consistent participation in BSAAG, and although it has been a member of BSAAG in the past, it was not a member concurrently with the futures SRO—thereby, potentially missing opportunities to engage FinCEN and other agencies on BSA issues in futures markets. In addition, by providing NFA with direct access to BSA data (similar to the access the key securities SRO already has) FinCEN could facilitate NFA oversight and enable it to scope examinations proactively to address BSA risks. Some federal agencies have taken steps to provide metrics and institution-specific feedback on the usefulness of BSA reporting to industry; however, metrics were not provided regularly and feedback efforts were provided on a small scale. Additionally, challenges to expanding and enhancing metrics and feedback remain (such as those related to measuring the usefulness of BSA reporting, providing feedback to thousands of individual institutions, and the sensitive nature of ongoing law enforcement investigations). FinCEN has an ongoing effort to identify additional measures of the value and usefulness of BSA reporting, which is expected to be completed at the end of 2019. But opportunities exist to enhance feedback and reporting before that date and in general. For example, in the interim FinCEN routinely could communicate currently available metrics on usefulness to help financial institutions more fully understand the importance and value of their efforts to report BSA-related information. Furthermore, with today’s rapidly changing financial markets and potential changes to money laundering risks, it is important that FinCEN and federal agencies take steps to provide institution-specific feedback—while keeping in mind any confidentiality concerns—to cover different types of financial institutions and those with significant financial activity. Increasing the feedback on BSA reporting could help make the BSA reporting of financial institutions more targeted and effective and enhance collaboration among key stakeholders in U.S efforts to combat illicit financial crime. We are making the following four recommendations to FinCEN: The Director of FinCEN, after consulting with CFTC, should consider prioritizing the inclusion of the primary SRO conducting BSA examinations in the futures industry in the Bank Secrecy Act Advisory Group (BSAAG) on a more consistent basis and also making the primary futures industry association a concurrent member. (Recommendation 1) The Director of FinCEN, after consulting with CFTC, should take steps to explore providing direct BSA data access to NFA. (Recommendation 2) The Director of FinCEN should review options for FinCEN to more consistently and publicly provide summary data on the usefulness of BSA reporting. This review could either be concurrent with FinCEN’s BSA value study or through another method. (Recommendation 3) The Director of FinCEN should review options for establishing a mechanism through which law enforcement agencies may provide regular and institution-specific feedback on BSA reporting. Options should take into consideration providing such feedback to cover different types of financial institutions and those with significant financial activity. This review could either be part of FinCEN’s BSA value study or through another method. (Recommendation 4) We provided a draft of this report to Treasury/FinCEN, CFTC, NCUA, DHS, DOJ, the Federal Reserve, FDIC, IRS, OCC, and SEC for their review and comment. FinCEN, CFTC, and NCUA provided written comments, which are reproduced in appendixes IV, V, and VI. FinCEN, DHS, the Federal Reserve, FDIC, OCC, and SEC provided technical comments on the draft report, which we incorporated as appropriate. In emails, DOJ and IRS audit liaisons stated that the agencies did not have any formal or technical comments. In its written response, FinCEN concurred with one recommendation, disagreed with two, and agreed with the spirit of one recommendation but noted some concerns. Specifically, FinCEN concurred with the recommendation that FinCEN more consistently and publicly provide summary data on the usefulness of BSA reporting (Recommendation 3). FinCEN disagreed with the draft report’s recommendation that FinCEN, after consulting with CFTC, should ensure that the primary SRO conducting BSA examinations in the futures industry is a regular member of BSAAG and also should consider making the primary futures industry association a concurrent member (Recommendation 1). FinCEN’s written response stated that while the primary futures SRO presently is a BSAAG member, only federal agencies are considered permanent members, and FinCEN will not make future membership commitments to any specific SRO or any other nonfederal organization. As such, we modified the recommendation to give FinCEN more flexibility to address the issues that prompted our recommendation. We continue to believe that prioritizing futures representation in BSAAG to be consistent with securities industry representation would help FinCEN better understand BSA compliance in the futures industry and keep the futures industry updated on related BSAAG initiatives. As noted in the report, the primary securities SRO has been a member of BSAAG since 2008 and a key securities industry association has been a concurrent member. FinCEN disagreed with the recommendation that FinCEN, after consulting with CFTC, explore providing direct BSA data access to NFA (Recommendation 2) because FinCEN said it has not received a request from CFTC or NFA to engage on this matter. FinCEN also said it would review any future request for direct access in accordance with established procedures, stating it must ensure that proper controls are in place and that direct access to the BSA database is limited to those who truly need it. As discussed in our report, CFTC stated that NFA’s direct access to BSA data would enhance NFA’s ability to scope and perform BSA/AML examinations, and to use BSA data more extensively and more frequently to perform its functions, including conducting the majority of BSA examinations for the futures industry. NFA representatives also told us they welcomed a discussion with CFTC and FinCEN on the benefits and drawbacks of having direct access to BSA data. We continue to believe the recommendation is valid as it provides FinCEN flexibility to explore providing NFA data access and would not preclude FinCEN from ensuring that NFA had proper controls in place. In its written responses, FinCEN neither agreed nor disagreed with the recommendation that FinCEN review options for establishing a mechanism through which law enforcement agencies may provide regular and institution-specific feedback on BSA reporting (Recommendation 4). FinCEN said it agreed with the spirit of this recommendation—that law enforcement feedback on the value and usefulness of BSA information is important—and stated that FinCEN regularly takes necessary steps to review options for establishing additional mechanisms through which law enforcement agencies can provide regular feedback. FinCEN also stated that it provides a consolidated view of law enforcement feedback as well as feedback on the value and usefulness of institution-specific BSA information. However, as discussed in the report, we found that the current institution-specific feedback mechanisms were not occurring on a regular basis or were on a relatively small scale. In its response, FinCEN also noted that unless mandated by Congress, law enforcement feedback will be voluntary and that FinCEN cannot compel law enforcement compliance with feedback initiatives. We continue to believe the recommendation is valid as it allows FinCEN flexibility in reviewing options for establishing a mechanism through which law enforcement may choose to provide regular feedback to reach a larger number of financial institutions from diverse industries, without requiring FinCEN to compel law enforcement agencies to participate. In its written responses, CFTC agreed with all our recommendations. In particular, CFTC agreed that the primary futures SRO should be a regular member of BSAAG (Recommendation 1). CFTC added that FinCEN should consider making another futures SRO a concurrent member. In a later discussion, a CFTC Assistant General Counsel said that, in general, CFTC would like to see more futures participation in BSAAG, including SROs and industry associations. CFTC also agreed with our recommendation that the Director of FinCEN, after consulting with CFTC, explore providing NFA direct access to BSA data (Recommendation 2). In its written response, NCUA also agreed with all of our recommendations, which it stated would enhance coordination and collaboration and increase visibility about the value of BSA reporting requirements. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of the Treasury, the Attorney General, the Acting Secretary of Homeland Security, the Commissioner of IRS, the Chairman of CFTC, the Chairman of FDIC, the Chairman of the Federal Reserve, the Chairman of NCUA, the Comptroller of the Currency, the Chairman of SEC, and other interested parties. This report will also be available at no charge on our website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or ClementsM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. The objectives of this report were to:(1) describe how the Financial Crimes Enforcement Network (FinCEN) and supervisory agencies supervise, examine for, and enforce Bank Secrecy Act and related anti- money laundering requirements (collectively, BSA/AML) compliance; (2) discuss how FinCEN, supervisory agencies, and law enforcement collaborated on implementing and enforcing BSA/AML requirements; and (3) examine the extent to which FinCEN, supervisory agencies, and law enforcement established metrics and provided feedback to financial institutions on the usefulness of their BSA reporting. For this report, we identified the key agencies and entities, including FinCEN, a bureau in the Department of the Treasury (Treasury), which is responsible for the administration of BSA, and the supervisory agencies that oversee BSA compliance. The supervisory agencies include the federal banking regulators—Federal Deposit Insurance Corporation (FDIC), Board of Governors of the Federal Reserve System (Federal Reserve), National Credit Union Administration (NCUA), Office of the Comptroller of the Currency (OCC)—as well as the Internal Revenue Service (IRS), Commodity Futures Trading Commission (CFTC), and Securities and Exchange Commission (SEC). Self-regulatory organizations (SRO) for the securities and futures industries—including the Financial Industry Regulatory Authority (FINRA) and National Futures Association (NFA)—also have BSA/AML responsibilities and conduct BSA examinations of their members. The Department of Justice may pursue investigations and prosecutions of financial institutions and individuals for both civil and criminal violations of BSA/AML regulations. To address the first objective, we reviewed relevant laws—including the Bank Secrecy Act, its related statutes, and key provisions of the USA PATRIOT Act—regulations, and agency documentation. To better understand how supervisory agencies conduct their examinations, we reviewed the following BSA/AML examination manuals: the 2014 BSA/AML Examination Manual, developed by the Federal Financial Institutions Examination Council (FFIEC); the Bank Secrecy Act/Anti- Money Laundering Examination Manual for Money Services Business (developed by FinCEN and IRS); and SEC’s nonpublic manual and futures SROs nonpublic examination procedures. We reviewed and analyzed data from FinCEN summary reports on the examination and enforcement activities of supervisory agencies for fiscal years 2015 through 2018 (second quarter), which were the most recent data available at the time of our analysis. We also reviewed FinCEN’s enforcement actions for this time period as provided on its website, to identify the number and types of financial institutions, and the number of concurrent actions FinCEN brought jointly with a regulator. We also reviewed and analyzed FinCEN referral data from January 1, 2015, to September 25, 2018. Referrals are potential BSA violations or deficiencies referred by supervisory agencies, the Department of Justice, or state regulators. We assessed the reliability of the FinCEN summary report data and referral data by reviewing documentation related to these datasets, interviewing knowledgeable officials, and conducting manual data testing for missing data, outliers, and obvious errors. We determined the data to be sufficiently reliable for reporting on supervisory agency, SRO, and FinCEN BSA/AML compliance and enforcement activities. For this and our other objectives, we interviewed officials at Treasury’s Office of Terrorism and Financial Intelligence and FinCEN, the other supervisory agencies, and two SROs—FINRA and NFA. To address the second objective, we judgmentally selected six law enforcement agencies based on their (1) focus on financial crimes, (2) role in investigating or prosecuting recent large criminal cases we selected involving financial institutions with BSA violations, (3) participation in FinCEN’s liaison program, and (4) identification by FinCEN as a key user of BSA data. We selected the following law enforcement agencies: the Criminal Division (Money Laundering and Asset Recovery Section), the U.S. Attorney’s Offices (through the Executive Office for United States Attorneys), and the Federal Bureau of Investigation in the Department of Justice; IRS Criminal Investigation in the Department of Treasury; and U.S. Immigration and Customs Enforcement-Homeland Security Investigations and the U.S. Secret Service in the Department of Homeland Security. The views of selected law enforcement agencies are not generalizable. To identify key collaborative mechanisms used to implement BSA/AML responsibilities, we reviewed agency documentation (such as strategic plans, national strategies, and risk assessments) and prior GAO reports that contained discussions of collaborative mechanisms, and we interviewed agency officials from FinCEN, supervisory agencies, SROs, and selected law enforcement agencies. We obtained agency documentation and data related to the identified collaboration mechanisms and interviewed officials from FinCEN, supervisory agencies, and selected law enforcement agencies for their perspectives on these efforts. We compared agencies’ collaboration efforts to criteria in federal internal control standards on management communication. To gain further insight into the collaboration process, we also reviewed documentation on three criminal cases involving BSA/AML violations by financial institutions to illustrate how law enforcement investigates and prosecutes BSA violations and coordinates with FinCEN and other supervisory agencies. We selected the cases on the basis of recent occurrence (calendar year 2017 or 2018) and on their having involved criminal violations of BSA by financial institutions, required coordination on penalties among multiple supervisory agencies and law enforcement, and resulted in a large monetary penalty. While not generalizable, the cases helped provide additional context for our review. To obtain additional perspectives on the effectiveness of BSA/AML collaboration processes, we interviewed representatives of seven selected industry associations based on their published work and relevant experience and for coverage of key financial industries (banking, securities, futures, and the money services business). While not generalizable, these interviews helped provide context for how industry views the effectiveness of BSA/AML collaboration efforts. For the third objective, we reviewed agency documentation and data on metrics related to BSA reporting and feedback mechanisms that FinCEN, the supervisory agencies, or the six selected law enforcement agencies had established. Key documents we reviewed included Treasury’s most recent strategic plan, national strategy for combating illicit financing, and related risk assessments. For all agencies we interviewed, we requested any available metrics. We reviewed agency websites, annual reports, and recently published speeches and testimonies on BSA/AML-related topics to identify any metrics. We also requested and reviewed contract documentation from FinCEN, such as the performance work statement for a study that FinCEN commissioned on how to establish metrics for and identify the value of BSA data. We compared metrics on the usefulness of BSA and how they were communicated against key criteria for enhancing or facilitating the use of performance metrics that GAO previously identified and federal internal control standards on management communication. For feedback mechanisms, we obtained documentation on any steps FinCEN, supervisory agencies, or the selected law enforcement agencies took to provide feedback on BSA reporting to financial institutions and we interviewed agency representatives on these efforts. The documents we reviewed included those identified above related to metrics, as well as agency advisories, guidance, and rulemaking. We compared the feedback efforts against Treasury’s information-sharing statutory duties and strategic plan, and international anti-money laundering standards and guidance. To gain industry perspectives on the usefulness of BSA reporting and on feedback received from FinCEN, supervisory agencies, and law enforcement, we conducted seven interviews with the selected industry associations. While not generalizable, the interviews helped provide context for financial industry perspectives on BSA/AML reporting and feedback. We conducted this performance audit from February 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. As part of its oversight of supervisory agencies, the Financial Crimes Enforcement Network (FinCEN) routinely collects data from supervisory agencies as established in information-sharing memorandums of understanding (MOU). The MOUs establish that supervisory agencies should provide FinCEN with examination data such as the number of Bank Secrecy Act /anti-money laundering (BSA/AML) violations, informal actions, and formal enforcement actions (on a quarterly basis). Finally, the Internal Revenue Service (IRS) told us it has MOUs with some state regulators to obtain state examinations, which IRS officials said help to identify issues among and plan examinations of money services businesses and determine if the businesses had addressed prior deficiencies. The following sections provide more information on each supervisory agency’s (1) examinations, (2) violations, and (3) enforcement actions. Also see appendix I for more information on the types of data we collected for each agency and any data limitations. From fiscal year 2015 to the second quarter of fiscal year 2018, the most common BSA violations cited by the federal banking regulators were violations of requirements to report suspicious activities, 314(a) information-sharing requirements, rules for filing of reports, BSA training, and a system of internal controls. For example, regulators could cite a violation if a financial institution failed to file a required suspicious activity report (SAR), failed to file a SAR in a timely manner, or failed to maintain confidentiality of SARs. Violations of internal controls include a financial institution failing to establish a system of internal controls to ensure ongoing compliance, including staff adherence to the financial institution’s BSA/AML policies. From fiscal year 2015 to 2018 (second quarter), the federal banking regulators cited thousands of violations (11,752) and brought 116 formal enforcement actions (see table 7). The number of informal enforcement actions compared to the number of formal enforcement actions varied by banking regulator. For example, in fiscal year 2017 the National Credit Union Administration (NCUA) brought 1,077 informal enforcement actions and no formal enforcement actions. In the same period, the Office of the Comptroller of the Currency (OCC) brought two informal enforcement actions and six formal enforcement actions. SEC and its SROs took 71 formal enforcement actions against broker- dealers from fiscal year 2015 through the second quarter of fiscal year 2018 (see table 8). FINRA took the majority of enforcement actions against broker-dealers. From fiscal year 2015 to the second quarter of fiscal year 2018, SEC and the SROs for broker-dealers most frequently cited violations of FINRA AML program rules. They included violations of policies and procedures relating to reporting suspicious activity, internal controls, and annual independent testing, as well as BSA violations of AML program requirements for brokers or dealers and customer identification programs for brokers or dealers. From fiscal year 2015 to the second quarter of fiscal year 2018, the National Futures Association (NFA) cited all BSA/AML violations, and took all informal and formal enforcement actions for BSA/AML deficiencies for the futures industry (see table 9). The violations NFA most commonly cited were against introducing brokers and fell under its AML program rules that related to policies and procedures for internal controls, training, and annual independent testing, and BSA requirements for AML programs and customer identification programs. The CME Group did not cite any futures commission merchants for violations during this period. In response to violations, NFA brought almost 200 informal enforcement actions and a few (10) formal enforcement actions over the period of our review. For example, in 2017 NFA took 64 informal and four formal enforcement actions. IRS referred more than 100 cases to FinCEN from fiscal year 2015 through the second quarter of 2018 and issued letter 1112s to thousands of institutions, which contain a summary of examination findings and recommendations to the institution for corrective action (see table 10). From fiscal year 2015 to the second quarter of fiscal year 2018, the most common violations cited by IRS fell under general AML program requirements for money services businesses, which require such businesses to develop, implement, and maintain an effective AML program (one designed to prevent a business from being used to facilitate money laundering and the financing of terrorist activities). AML program requirements have several subcomponent violations. Among the most commonly cited subcomponent violations were those related to overall program deficiencies; policies, procedures, and internal controls; training of appropriate personnel to identify suspicious transactions; and providing for independent testing of the AML program. The Financial Crimes Enforcement Network (FinCEN) and supervisory agencies may be asked to provide information as part of law enforcement investigations and can take parallel, but separate, enforcement actions against the same institutions to address Bank Secrecy Act/anti-money laundering (BSA/AML) concerns. FinCEN and supervisory agencies may refer potential violations of a criminal nature an appropriate federal law enforcement agency or to the Department of Justice (DOJ)—and within DOJ, the U.S. Attorney’s Office—and may be asked to assist law enforcement investigations. For example, supervisory agencies may be asked to interpret financial institution documents or serve as expert witnesses and records custodians in a trial. FinCEN, supervisory agencies, and law enforcement agencies have conducted parallel civil and criminal investigations. Federal law enforcement and supervisory agency officials have told us that such investigations should remain separate and independent. We selected three recent cases in which FinCEN, supervisory agencies, and law enforcement collaborated to conduct parallel investigations and took concurrent but separate civil and criminal BSA enforcement actions. Officials with whom we spoke from agencies that were involved in these cases said the agencies coordinated with each other (for example, by establishing liaison positions, scheduling regular conference calls, and coordinating on global settlements). Rabobank National Association (Rabobank). On February 7, 2018, DOJ and the Office of the Comptroller of the Currency (OCC) both announced actions against Rabobank for deficiencies in its BSA/AML compliance program and obstruction of the primary regulator (OCC). DOJ announced that Rabobank pleaded guilty to a felony conspiracy charge for impairing, impeding, and obstructing its primary regulator OCC by concealing deficiencies in its AML program and for obstructing OCC’s examination of Rabobank. The bank agreed to forfeit $368,701,259 for allowing illicit funds to be processed through the bank without adequate BSA/AML review and OCC issued a $50 million civil money penalty against Rabobank for deficiencies in its BSA/AML compliance program. DOJ’s Money Laundering and Asset Recovery Section Bank Integrity Unit, the U.S. Attorney’s Office of the Southern District of California, U.S. Immigration and Customs Enforcement-Homeland Security Investigations (ICE-HSI) within the Department of Homeland Security, Internal Revenue Service Criminal Investigation (IRS-CI), and the Financial Investigations and Border Crimes Task Force conducted the criminal investigation. The investigation occurred in parallel with OCC’s regulatory investigation and the investigation by FinCEN’s Enforcement Division. OCC officials told us they collaborated extensively with other agencies over a 4-year period, participated in numerous calls and meetings, and provided law enforcement with examination information and access to OCC examiners for interviews. Officials from the U.S. Attorney’s Office of the Southern District of California said that a practice they found helpful in this case was establishing a liaison with the agencies involved. The liaisons allowed the different parties to share information effectively, provided access to data as needed, and responded to questions in a timely manner. U.S. Bancorp. On February 15, 2018, DOJ, OCC, and FinCEN announced actions against U.S Bancorp and its subsidiary U.S. Bank, N.A., for violations of several provisions of BSA, including an inadequate BSA/AML program and failure to file suspicious activity reports (SAR) and currency transaction reports (CTR). Under a deferred prosecution agreement with the U.S. Attorney’s Office of the Southern District of New York, U.S Bancorp and its subsidiary agreed to pay $528 million for BSA violations and agreed to continue to reform its AML program. Of the $528 million, $75 million was satisfied by a penalty paid to the Department of the Treasury as part of OCC’s civil money penalty assessment, which cited the bank in a 2015 consent order for failure to adopt and implement a program that covered required BSA/AML program elements. FinCEN also reached an agreement with U.S. Bank to resolve related regulatory actions, which required U.S. Bank to pay an additional $70 million for civil violations of the BSA. On the same day as the FinCEN agreement, the Board of Governors of the Federal Reserve System (Federal Reserve) imposed a $15 million penalty against U.S. Bancorp for deficiencies (including BSA violations) related to the bank under its supervision. According to officials from the U.S. Attorney’s Office of the Southern District of New York, their office, OCC, FinCEN and the Federal Reserve coordinated the terms of their respective resolutions to avoid the unnecessary imposition of duplicative penalties. OCC officials told us that the U.S. Attorney’s Office of the Southern District of New York contacted them to obtain additional information about its examination conclusions that supported OCC’s 2015 cease and desist order. OCC provided examination documents and information to the U.S. Attorney’s Office of the Southern District of New York for 2 years, including making OCC examiners available for interviews with the U.S. Attorney’s Office personnel and to answer follow-up inquiries. Federal Reserve officials said they coordinated in the U.S. Bancorp case through a global resolution with the firm. Banamex. In May 2017, Banamex admitted to criminal violations and entered into a non-prosecution agreement, which included an agreement to forfeit $97.44 million. The bank also admitted that it should have improved its monitoring of money services businesses’ remittances, but failed to do so. The investigation was conducted by the Bank Integrity Unit of DOJ’s Money Laundering and Asset Recovery Section, U.S. Attorney’s Office of the District of Massachusetts, IRS-CI, Drug Enforcement Administration, and the Federal Deposit Insurance Corporation’s (FDIC) Office of Inspector General. The agencies consulted on a general level, but the agencies’ investigations were at all times kept separate from the criminal investigation. In July 2015, FDIC and the California Department of Business Oversight assessed civil money penalties against Banamex requiring a total payment of $140 million to resolve separate BSA regulatory investigations. In February 2017, FDIC also announced enforcement actions against four former senior bank executives relating to BSA violations. IRS-CI officials stated that involvement by the Bank Integrity Unit of DOJ’s Money Laundering and Asset Recovery Section in financial institution investigations is extremely helpful as the unit bring a wealth of knowledge and resources. DOJ officials told us there was close collaboration between all agencies involved. DOJ officials said that all agencies had meetings frequently and created a liaison position to encourage interagency collaboration as the case progressed. In May 2018, DOJ issued a new policy to encourage coordination among DOJ and supervisory agencies during corporate investigations. In a May 2018 speech, the DOJ Deputy Attorney General identified the Securities and Exchange Commission, Commodity Futures Trading Commission, Federal Reserve, FDIC, OCC, and the Department of the Treasury’s Office of Foreign Assets Control as agencies with which DOJ works to be better able to detect sophisticated financial fraud schemes and deploy adequate penalties and remedies to ensure market integrity. He noted that many federal, state, local, and foreign authorities that work with DOJ were interested in further coordination with DOJ. DOJ’s new policy encourages coordination and consideration of the amount of fines, penalties, or forfeiture paid among DOJ components and other law enforcement or other federal, state, local, or foreign enforcement authorities seeking to resolve a case with a company for the same misconduct. Similarly, in June 2018, the Federal Reserve, FDIC, and OCC issued a joint statement on coordination among federal banking agencies during formal enforcement actions. In addition to the contact named above, Allison Abrams (Assistant Director), Verginie Tarpinian (Analyst in Charge), Peter Beck, Joseph Cruz, Brian James, Moira Lenox, Benjamin Licht, Robert Lowthian, Marc Molino, Ifunanya Nwokedi, Barbara Roesmann, Tyler Spunaugle, Farrah Stone, and Sarah Veale made key contributions to this report.", "summary": "Illicit finance activity, such as terrorist financing and money laundering, can pose threats to national security and the integrity of the U.S. financial system. FinCEN is responsible for administering BSA and has delegated examination responsibility to supervisory agencies. FinCEN also is to collect and disseminate BSA data. BSA requires that financial institutions submit reports, which may be used to assist law enforcement investigations. Industry perspectives on BSA reporting have included questions about its usefulness. This report examines, among other objectives, how FinCEN and supervisory and law enforcement agencies (1) collaborate and (2) provide metrics and feedback on the usefulness of BSA reporting. GAO reviewed related laws and regulations; agency documentation; examination and enforcement action data; and interviewed FinCEN, supervisory agencies, and a nongeneralizable selection of six law enforcement agencies and seven industry associations. The Financial Crimes Enforcement Network (FinCEN)—within the Department of Treasury—supervisory agencies (such as banking, securities, and futures regulators), and law enforcement agencies collaborate on implementing Bank Secrecy Act/anti-money laundering (BSA/AML) regulations, primarily through cross-agency working groups, data-sharing agreements, and liaison positions. FinCEN and law enforcement agencies provided some metrics and institution-specific feedback on the usefulness of BSA reporting (such as suspicious activity reports) to the financial industry but not regularly or broadly. FinCEN and some agencies have metrics on the usefulness of BSA reports. One law enforcement agency annually publishes aggregate metrics on BSA reports that led to investigations and indictments. But FinCEN did not consistently communicate available metrics; it generally did so on an ad-hoc basis such as through published speeches. In 2019, FinCEN began a study to identify measures on the value and usefulness of BSA reporting—to be completed by the end of 2019. By consistently communicating currently available metrics (summary data), and any later identified by the study, FinCEN may assist financial institutions in more fully understanding the importance of their efforts. Industry associations GAO interviewed noted financial institutions would like to receive more institution-specific feedback on the usefulness of their BSA reporting; they also identified suspicious activity reports as labor-intensive. In 2017, FinCEN began providing such feedback and some law enforcement agencies have ongoing efforts to provide institution-specific briefings. But these efforts have not been regularly made and involved relatively few institutions. Additional and more regular feedback, designed to cover different types of financial institutions and those with significant financial activity, may enhance the ability of the U.S. financial industry to effectively target efforts to identify suspicious activity and provide quality BSA reporting. GAO makes four recommendations, including that FinCEN review options to consistently communicate summary data and regularly provide institution-specific feedback on its BSA reporting. FinCEN concurred with the recommendation on summary data and agreed with the spirit of the recommendation on feedback. FinCEN raised concerns with the need for the two other recommendations. GAO continues to believe the recommendations have merit, as discussed in the report.", "document_type": "gao"}
{"report": "The issues VA experienced during the height of the COVID-19 pandemic were a result of global supply chain challenges, but longstanding problems that our work has previously identified posed additional challenges to VA’s response. In November 2017, we reported weaknesses in VA’s implementation of its MSPV-NG program—VA’s primary means for purchasing medical supplies. These included the lack of an effective medical supply procurement strategy, clinician involvement, and reliable data systems. We also found that several of VA’s medical supply management practices were not in line with those employed by private sector leading hospital networks. We recommended, among other things, that VA develop, document, and communicate to stakeholders an overarching strategy for the program. This strategy, originally planned for completion by December 2017, was delayed to March 2019, and then further delayed due to VA’s implementation of its new MSPV 2.0 program, which is also delayed. We also found that VA’s initial formulary consisted of around 6,000 items at launch, and, according to senior VA contracting officials, many items on the formulary were not those needed by medical centers. These factors resulted in an initial formulary that did not meet the needs of VA’s medical centers (VAMC). The MSPV-NG program office subsequently took steps to expand the formulary, growing it to over 22,000 items, and is developing the next iteration of the program, called MSPV 2.0. MSPV 2.0 is intended to address some of the shortfalls we previously identified in MSPV-NG, including more than doubling the number of items on the formulary, to a planned 49,000. VA’s MSPV 2.0 prime vendor procurement has been subject to multiple bid protests. After three protests challenged the terms of the solicitation, VA responded by voluntarily taking corrective action and revising the solicitation. The terms of the revised solicitation were challenged in a subsequent protest that was sustained, resulting in VA further revising the solicitation to address the matter. Because of these events, agency officials told us that VA has altered its MSPV 2.0 procurement plans several times and there has been significant delay in program implementation from the originally planned March 2020 date to as late as February 2021. Based on preliminary observations of our ongoing work, some of the current MSPV-NG challenges persist and may not be remedied by MSPV 2.0. Specifically, medical center staff we interviewed from May 2019 through October 2019 cited continued problems with consistently receiving the supplies they order through MSPV-NG, such as backorders on frequently ordered items. For example, preceding the COVID-19 pandemic, supply chain problems with one of VA’s prime vendors created supply shortages for infection control gowns, and staff at one VAMC we visited in June 2019 had to obtain gowns from its emergency cache as a temporary measure. Further, VA’s plans for MSPV 2.0 give no indication that they will update their practice of manually maintaining the formulary using spreadsheets, which, based on our discussions with several VAMC logistics officers, can lead to errors such as inadvertent omission of items from the formulary. We plan to issue a report on our review of the MSPV 2.0 program in fall 2020. According to senior VA procurement and logistics officials interviewed during our ongoing review of VA’s COVID-19 procurement for critical medical supplies, VA experienced difficulty obtaining several types of supplies needed to protect its front-line workforce during the COVID-19 response, ranging from N95 masks to isolation gowns. According to senior VA acquisition and logistics officials, beginning in late February to early March 2020, VA requested that medical centers provide daily updates via spreadsheets to try to obtain the most real time information possible on the levels of PPE on hand, usage, and gaps. These spreadsheets, which were reported manually on a daily basis from each of the VAMCs, were the primary means by which Veterans Health Administration (VHA) leadership obtained detailed information on the stock of critical supplies at its VAMCs in real-time. The insight provided by these spreadsheets was not something that VHA leadership had in any type of ongoing or systematic way, prior to the COVID-19 pandemic. In April 2020, VA developed an automated tool to manage this reporting process, but, according to officials, the information must still be gathered and manually reported by each of the 170 VAMCs on a daily basis. In May 2019, the VA Inspector General found that proper inventory monitoring and management was lacking at many VAMCs, noting that inventory management practices ranged from inaccurate to nonexistent. In 2013, we also reported on weaknesses in VA’s inventory management systems and made recommendations to VA to evaluate its efforts to improve in this area. However, our preliminary observations from our ongoing review of VA’s MSPV program indicate that VA will likely rely on its antiquated system for the foreseeable future. Specifically, VA plans to transition to the Defense Logistics Agency’s (DLA) inventory management system, called Defense Medical Logistics Standard Support (DMLSS). DMLSS serves as DLA’s primary MSPV ordering system and supports DLA’s inventory management, among other things. According to DLA officials, DMLSS produces data that VAMCs could use to analyze their order history and find recommendations for future purchases. VA’s implementation schedule shows that it will take seven years to roll out DMLSS and its successor at all VAMCs. In the near-term, VA had planned to implement DMLSS at three medical centers in mid-to-late 2019. However, due to technology integration issues between VA’s financial system and the DMLSS system, implementation at these three VAMCs is delayed. According to the Chief Supply Chain Officer at one of these VAMCs, the original DMLSS implementation date has changed several times from an initial start date of August 2019, which may be delayed to at least October 2020. VA uses a “just in time” inventory supply model—a practice employed by many hospital networks where only limited stock is maintained on-site. However, for this model to succeed, VA needs both visibility into current stock and consistent deliveries from the MSPV-NG program. Based on our preliminary observations, VA faces challenges with both visibility and delivery. VA acquisition leadership has recognized the shortcomings in its medical supply chain management, and has identified supply chain modernization as a priority. As part of our ongoing review of VA’s MSPV program, we reviewed VHA’s Modernization Campaign Plan, dated March 2019, and VHA’s Modernization Plan briefing slides, dated February 2020, which describe several modernization initiatives including MSPV 2.0 and DMLSS. VHA’s February 2020 update on its modernization effort identified both its DMLSS deployment and MSPV 2.0 program at critical risk of not meeting system modernization milestones. Based on our preliminary observations from our ongoing review of VA’s procurement of critical medical supplies, in response to COVID-19, VA is using various existing and new contracting organizations and mechanisms to try to meet its PPE needs. These include using national and regional contracting offices to procure supplies and services, and using existing contract vehicles and new sources. In response to the pandemic, VA’s Office of Acquisition and Logistics also issued a memorandum on March 15, 2020, to implement emergency flexibilities available under the Federal Acquisition Regulation, such as increasing the micro-purchase threshold to $20,000. Our analysis of contracting activity in the Federal Procurement Data System-Next Generation (FPDS-NG) indicates that VHA’s Network Contracting Offices—which support the various regions of VA’s hospital network—increased their supply purchases, mostly by entering into new contracts. Department-wide contracting organizations that would normally not make individual supply purchases—such as VHA’s Program Contracting Activity Central and VA’s Strategic Acquisition Center—also played a substantial role. In addition, logistics staff at VAMCs continued to use the MSPV-NG program to order supplies. VA had existing clauses in MSPV-NG contracts that established terms for the suppliers to maintain support to VA in the event of a catastrophe. But, according to senior VA acquisition officials, because those suppliers faced the same shortages in the broader market, they were not able to provide enough supplies to meet VA’s surging demand. Figure 1 shows the COVID-19-related contract obligations, from March 13, 2020 through June 3, 2020, made by the various VA contracting offices. These obligations include both supplies, such as PPE, and services, such as information technology systems to support telemedicine. Our analysis of preliminary data on orders placed directly by VAMC staff for COVID-19-related items found that, in April 2020, the value of VA’s reported COVID-19-related purchases through the MSPV-NG program began to decrease relative to the values reported in prior months. According to senior VA acquisition and logistics officials, in part, because MSPV-NG and other existing VA supply contracts and agreements did not meet VA’s needs, its acquisition workforce had to make purchases through other contracting mechanisms, such as micro-purchases using government purchase cards, to fill the gap. Between March 13, 2020 and June 3, 2020 VA obligated more than 51 percent ($687 million) of the $1.3 billion it spent on products and services for the COVID-19 response through purchases made outside the MSPV-NG program and other established VA contracting mechanisms. About 27 percent of this $1.3 billion ($364 million) was for veteran-owned small business set-aside purchases, under VA’s Veterans First program. On April 17, 2020, VA placed its first supply requests through the Federal Emergency Management Administration’s (FEMA) Strategic National Stockpile program, according to VA senior acquisition and logistics officials. As of June 5, 2020, according to information provided by the VA, it had received shipments of several different types of supplies through FEMA from these requests, as shown in Table 1. According to VA senior procurement and logistics officials, VA’s Emergency Management Center has an existing relationship with FEMA. However, these senior procurement and logistics officials noted that VA support services officials—who had primary responsibility for requesting medical items through FEMA—did not have an existing relationship with FEMA or a process in place prior to the COVID-19 pandemic for placing medical supply requests through FEMA. Officials said that this led to a brief, initial delay in processing VA’s first request. In summary, VA experienced many of the same challenges obtaining medical supplies as most private sector hospitals and other entities in responding to this devastating pandemic. This situation put stress on an already overburdened acquisition and logistics workforce—resulting in staff initially scrambling to address supply chain shortfalls while simultaneously working with VA’s antiquated inventory system, through manual, daily reports on PPE levels to VA leadership. While VA has made progress in addressing some of the issues that have led us to identify VA acquisition management as high risk, it will take many years for VA to put in place a modern supply chain management system that would position it to provide the most efficient and effective service to our nations veterans. Chairman Moran, Ranking Member Tester, and Members of the Committee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Shelby S. Oakley at 202-512-4841 or OakleyS@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Lisa Gardner, Assistant Director; Teague Lyons, Assistant Director; Daniel Singleton, Analyst-in-Charge; Jeff Hartnett, Nicolaus Heun, Kelsey M. Carpenter, Sara Younes, Matthew T. Crosby; Suellen Foth, Lorraine Ettaro, Rose Brister, Susan Ditto, Roxanna Sun, Carrie Rogers, and Helena Johnson. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "VA spends hundreds of millions of dollars annually to meet the health care needs of about 9 million veterans. In March 2019, GAO added VA Acquisition Management to its High Risk list due to longstanding problems such as ineffective purchasing of medical supplies and lack of reliable data systems. This statement summarizes findings from GAO's 2017 MSPV-NG report and 2019 High Risk report and preliminary observations from two ongoing GAO performance audits to discuss VA's progress in building a more resilient supply chain. For the ongoing work, GAO reviewed VA documentation and interviewed VA officials, and VA medical center staff. Finally, GAO met with senior VA officials on June 5, 2020, to obtain agency views on the new observations GAO discusses in this statement. The Department of Veterans Affairs (VA) has taken some steps in recent years to modernize its processes to acquire hundreds of millions of dollars-worth of medical supplies annually. However, implementation delays for key initiatives, including a new, enterprise-wide inventory management system, limit VA's ability to have an agile, responsive supply chain. Prior to the Coronavirus Disease 2019 (COVID-19) pandemic, in November 2017 and in GAO's High-Risk report in March 2019, GAO reported on weaknesses in VA's acquisition management. For example, GAO reported that VA's implementation of its Medical-Surgical Prime Vendor-Next Generation (MSPV-NG) program—VA's primary means for purchasing medical supplies—lacked an effective medical supply procurement strategy, clinician involvement, and reliable data systems. GAO also found that several of VA's medical supply management practices were not in line with those employed by private sector leading hospital networks. VA is developing another iteration of its MSPV program, called MSPV 2.0, which GAO's preliminary observations show is intended to address some of the shortfalls GAO has identified in its past and ongoing program reviews. In November 2017, GAO recommended that VA develop, document and communicate an overarching MSPV-NG strategy—to include how the program office will prioritize categories of supplies and increase clinician involvement in this process. Preliminary observations from GAO's ongoing work indicate that VA has taken some steps, as it implements MSPV 2.0, to address this priority recommendation. However, GAO's preliminary observations also indicate that the MSPV 2.0 program implementation is delayed and some of these existing program challenges may not be remedied. Based on preliminary observations from GAO's ongoing work, VA's implementation of a new supply and inventory management system is delayed. As a result, VA had to rely on an antiquated inventory management system, and initial, manual spreadsheets to oversee the stock of critical medical supplies at its medical centers. This limited the ability of VA management to have real-time information on its pandemic response supplies, ranging from N95 face masks to isolation gowns, to make key decisions. As of April 2020, VA has an automated tool to manage its reporting process, but the information must be gathered and manually reported by each of VA's 170 medical centers on a daily basis. GAO's preliminary observations also show that in response to COVID-19, VA is using various contracting organizations and mechanisms to meet its critical medical supply needs. These include using national and regional contracting offices to obtain supplies from existing contract vehicles, new contracts and agreements, and the Federal Emergency Management Administration's Strategic National Stockpile to respond to the pandemic. GAO has made 40 recommendations since 2015 to improve acquisition management at the VA. VA agreed with those recommendations and has implemented 22 of them. Further actions are needed to implement the remaining recommendations, such as GAO's recommendation that VA implement an overarching MSPV strategy, and demonstrate progress toward removing this area from GAO's High-Risk list.", "document_type": "gao"}
{"report": "DOD is up to 21 months late in fully addressing five remaining requirements of section 911 related to DOD’s organizational strategy and cross-functional teams, as shown in figure 1 and discussed below. Specifically, DOD has not fully addressed the following statutory requirements: 1. Issue an organizational strategy: DOD has not issued its organizational strategy, which as of June 2019 is 21 months past the statutorily required issuance date of September 1, 2017. In January 2019, we reported that OCMO officials had revised the draft organizational strategy, incorporating, among other things, the criteria that distinguish cross-functional teams established under section 911 from other types of cross-functional working groups, committees, integrated product teams, and task forces, as required by section 918(b) of the John S. McCain NDAA for Fiscal Year 2019. The revised draft of the organizational strategy also includes steps DOD plans to take to advance a collaborative culture. As we reported in our June 2018 report, these steps, as outlined in the draft strategy, align with our leading practices for mergers and organizational transformations, which we recommended that DOD incorporate into its strategy. Based on our review of OCMO’s current draft of the organizational strategy, we found that it addresses all required elements laid out in section 911 of the NDAA for Fiscal Year 2017. That January 2019 draft strategy, according to an official from OCMO’s Administration and Organizational Policy Directorate, was provided to OCMO leadership for review as early as August 2018, but has not been approved. A senior OCMO official stated that approval of the draft was delayed to ensure it aligned with the National Defense Strategy, issued in January 2018, and the National Defense Business Operations Plan, issued in May 2018, and to incorporate additional requirements of the John S. McCain NDAA for Fiscal Year 2019, which was enacted in August 2018. In addition, according to senior OCMO and Office of the Deputy Secretary of Defense officials, the Acting CMO and the Deputy Secretary of Defense informally discussed the draft organizational strategy, but those conversations did not lead to the Acting CMO formally approving the draft for department-wide coordination. In May 2019, a senior OCMO official told us that the Acting CMO was fully committed to completing department-wide coordination of the draft strategy in June 2019 and advancing it for issuance by the Secretary of Defense in July 2019. After providing a draft of this report to the department for comment, we learned that the organizational strategy was circulated for department-wide coordination on July 12, 2019, with components expected to provide input by August 5, 2019. 2. Issue guidance for cross-functional teams: DOD has not issued guidance for cross-functional teams, which, as of June 2019, is 20 months past the required date of September 30, 2017. In June 2018, we reported that OCMO officials had revised the draft guidance to fully address all section 911 requirements and incorporate leading practices for effective cross-functional teams in the guidance, consistent with our February 2018 recommendation. Based on our review of this draft, we found that it addresses all required elements from section 911 of the NDAA for Fiscal Year 2017, as well as all of the leading practices for effective cross-functional teams. That draft guidance, according to an official from OCMO’s Administration and Organizational Policy Directorate, was provided to OCMO leadership for review as early as August 2018, but has not been approved by the CMO. 3. Provide training to cross-functional team members and their supervisors: OCMO officials have provided some of the required training to members and leaders of a recently established cross- functional team described later in this report. The training included several required elements, including information on the characteristics of successful cross-functional teams, conflict resolution, and how to appropriately represent the views and expertise of functional components. However, OCMO officials have not provided training to supervisors in team members’ functional organizations as required. We reported in February 2018 that DOD had developed a draft curriculum for this training that addressed the section 911 requirements. An OCMO official told us it has not altered the curriculum since then, but that the department has still not provided the training to team members’ supervisors because the curriculum has not been approved by the Acting CMO or the Secretary of Defense. Such approval, though not required by statute, would demonstrate senior leadership support for cross-functional teams, a leading practice we have identified. Further, according to an OCMO official, department-wide coordination and approval would serve to strengthen the effectiveness of the training. However, the need for this training is evident. For example, when we observed one of the training sessions, a member of a cross-functional team stated that he did not believe his supervisors knew what cross-functional teams were. 4. Provide training to presidential appointees: OCMO has not provided the required training to individuals filling presidentially appointed, Senate-confirmed positions in the Office of the Secretary of Defense. Section 911 requires these individuals to complete the training within 3 months of their appointment or DOD to request waivers. As of June 2019, 24 of 36 such officials had been appointed and in their positions for more than 3 months, and, according to an OCMO official, none had received their training or been granted a training waiver. An OCMO official told us in October 2018 he had revised the draft training curriculum following our February 2018 report to include all the required elements in section 911. However, as of May 2019, OCMO officials had not provided a copy of the revised curriculum for our review. After the curriculum is approved, the officials stated that they plan to recommend to the Secretary of Defense that all presidential appointees in the Office of the Secretary of Defense receive the training, rather than request waivers. 5. Report on successes and failures of cross-functional teams: OCMO has not completed an analysis of the successes and failures of DOD’s cross-functional teams, which, as of June 2019, is 3 months past its required completion date. Section 911 requires that an analysis of the success and failures of the teams and how to apply lessons learned from that analysis is completed 18 months after the establishment of the first cross-functional team. With the establishment of the first cross-functional team on personnel vetting in August 2017, the required completion date for the report was February 25, 2019. An OCMO official stated that OCMO planned to conduct an analysis on the personnel vetting team, but had not yet begun and had not set a time frame for doing so. DOD has not addressed most of these remaining requirements of section 911 because, according to an OCMO official, the Acting CMO has not approved the draft documents prepared by OCMO staff to satisfy the requirements. Moreover, the Acting CMO has not coordinated most of the documents department-wide and provided them to the Secretary of Defense for review and issuance. Specifically, according to an OCMO official, the Acting CMO has not reviewed or approved the guidance on cross-functional teams or curricula for cross-functional team members, their supervisors, and presidential appointees. These delays occurred in part because the department has not established and communicated internal deadlines for reviewing, coordinating, and approving these documents. According to OCMO officials, the primary reason they have not met these other outstanding requirements, including the guidance and training for cross-functional teams, is that they would like to have the organizational strategy approved and issued first, so that it can be reflected in the accompanying materials. However, while the OCMO has set an internal time frame for the organizational strategy, it has not set similar time frames for completing the remaining requirements. Standards for Internal Control in the Federal Government emphasize the need to establish time frames to implement actions effectively. In addition, as we reported in June 2018, establishing time frames with key milestones and deliverables to track implementation progress are important for agency reform efforts. By not setting and following clear internal deadlines for meeting the outstanding section 911 requirements, DOD has continued to fall short in meeting statutory requirements and missed opportunities to effectively implement its cross-functional teams and advance a collaborative culture that could bolster broader efforts within the department, such as reforming its business operations. Sections 918 and 1053(c) of the John S. McCain NDAA for Fiscal Year 2019 required the Secretary of Defense to establish a cross-functional team pursuant to section 911 of the NDAA for Fiscal Year 2017 on electronic warfare to identify gaps in electronic warfare and joint electromagnetic spectrum operations, capabilities, and capacities within the department across personnel, procedural, and equipment areas. In addition, section 1053(d) of the act required the electronic warfare cross- functional team to, among other things, (1) update the department’s Electronic Warfare Strategy in coordination with the Electronic Warfare Executive Committee by February 9, 2019, and (2) provide assessments of the electronic warfare capabilities of the Russian Federation and the People’s Republic of China in consultation with the Director of the Defense Intelligence Agency by May 10, 2019. Section 918 of the John S. McCain NDAA for Fiscal Year 2019 required the team’s establishment by November 11, 2018; however, DOD did not establish an electromagnetic spectrum operations cross-functional team until February 2019, and the team did not begin its work until April 2019. An official from the team told us that the standup of the team was delayed due to the extensive department-wide review of the February 2019 memorandum that established the team. Because of the delayed establishment of the team, DOD officials estimated that the required update to DOD’s Electronic Warfare Strategy would be completed by the end of September 2019—7 months after the statutory deadline—and that the required assessments would be provided by fall 2019. According to the team’s establishment memorandum, the team will continue its work until at least fiscal year 2022. In addition to the requirements discussed above, section 911 of the NDAA for Fiscal Year 2017 includes specific requirements for cross- functional teams established under that section, including that each team’s objectives be clearly established in writing and that the team should establish a strategy to achieve those objectives. We found that DOD and the electromagnetic spectrum operations cross-functional team have addressed 10 of 11 of those requirements for cross-functional teams. We also found that the team demonstrates several of the leading practices for cross-functional teams. For example, we found that the team has a well-defined team structure and well-defined team goals. However, as previously discussed, DOD has not fully addressed the section 911 requirement for training for cross-functional team members’ supervisors. We were also told by team officials that DOD was delayed in providing administrative support and funding to support the team’s operations. According to the memorandum establishing the electromagnetic spectrum operations team, the CMO is responsible for providing administrative support to the new team, to include providing the team with office space, information technology equipment, contracting, human resources, security, cross-functional team training, and other services, as appropriate. The memorandum also requires the team to work with the CMO to develop resource requirements for team operations for fiscal years 2019 and 2020 to ensure adequate resources are immediately available. However, according to a team official, funding was not provided to the team until late May 2019—over 3 months after the team was established and over 1 month after most of the team members were provided by their home units to work on the team full time. According to a team official, this funding was to be used for several team requirements, including dedicated office space, computer systems, travel funds, and contractor support. This funding was delayed in part because of disagreements over responsibility for funding the team under the terms of the memorandum establishing the team. Specifically, according to a team official, OCMO officials believed that funding should be provided by another organization, such as the Joint Staff. Team and Joint Staff officials told us that they believed the OCMO was responsible for this funding based on the memorandum establishing the team. A team official further stated that funding was provided only when the Deputy Secretary of Defense directed that funding be provided to the team. OCMO officials told us that because the team was not a budgeted activity for fiscal year 2019, the team was added to DOD’s unfunded requirements list. The Under Secretary of Defense (Comptroller) identified funds for the team via the unfunded requirements process at the end of April 2019. However, a team official told us funding for the team for future fiscal years has not been identified and responsibility for providing that funding is still unclear. OCMO officials told us that the team will continue to rely on the unfunded requirements process for funding, since the team is not a budgeted activity for fiscal year 2020, and would need to compete for funding through DOD’s program budget review process for fiscal year 2021 and later fiscal years. Those officials also told us that the team has not yet signed a memorandum of agreement that is required to execute transfer of the funds to the team. A team official told us the team had not yet signed the memorandum because it believed the memorandum would transfer responsibility for funding the team from OCMO to the team. As noted previously, team officials believe the OCMO is responsible for this funding based on the memorandum establishing the team. According to a team official, this delay in funding hampered the team’s ability to achieve full operating capability. For example, until late May the team was working from the Pentagon Conference Center and OCMO conference rooms with only one secure laptop. A team official told us in June 2019 that though the team has moved into its own office space, that space does not have the level of security required for the team to work on a third of its initiatives. As a result, the team was also delayed in conducting mission analysis, work plan development, organizational design, and production of executive-level briefings. A team official told us the team expects to be at full operating capability in late July 2019. Leading practices for implementing effective cross-functional teams highlight the importance of senior management providing teams with access to resources. In addition, Standards for Internal Control in the Federal Government state that agencies’ management should assign responsibility to achieve the entity’s objectives. If DOD does not clarify roles and responsibilities for providing funding for the new cross- functional team, the Acting CMO and the electromagnetic spectrum operations team may continue to have delays in funding and those delays may negatively affect the team’s ability to conduct its work and to meet its objectives. Section 911 of the NDAA for Fiscal Year 2017 called for organizational and management reforms to assist DOD in addressing challenges that have hindered collaboration and integration across the department. The department has taken some steps to implement the section 911 requirements, but still has not met statutory time frames for implementing key requirements intended to support its cross-functional teams and to advance a more collaborative culture within the department. Setting specific internal deadlines would help ensure action on these outstanding statutory requirements. Moreover, DOD has established a new electromagnetic spectrum operations cross-functional team under section 911—one of the only requirements for which the department has made progress since our last report—but has not ensured that the team will have the funding it needs beyond fiscal year 2019 to maintain full operational capability and accomplish its assigned objectives. Senior leadership commitment to fully supporting this team and fulfilling all section 911 requirements could help the department make important advances in the type of collaboration necessary for the department to accomplish some of its most ambitious goals. We are making the following six recommendations to DOD: The Secretary of Defense should ensure that the CMO meets DOD’s August 2019 deadline for final submission of the organizational strategy to the Secretary of Defense for review and issuance. (Recommendation 1) The Secretary of Defense should ensure that the CMO meets DOD’s September 2019 deadline for review and approval of DOD’s guidance on cross-functional teams and final submission to the Secretary for review and issuance. (Recommendation 2) The Secretary of Defense should ensure that the CMO meets DOD’s September 2019 deadline for review and approval of DOD’s training curriculum for cross-functional team members and their supervisors. (Recommendation 3) The Secretary of Defense should ensure that the CMO meets DOD’s September 2019 deadline for review and approval of DOD’s training curriculum for presidential appointees. (Recommendation 4) The Secretary of Defense should ensure that the CMO meets DOD’s November 2019 deadline for drafting, review, and approval of DOD’s report on the success and failures of cross-functional teams and final submission to the Secretary for review and approval. (Recommendation 5) The Secretary of Defense should ensure that the CMO and the electromagnetic spectrum operations cross-functional team clarify roles and responsibilities for providing administrative support and funding for the team beyond fiscal year 2019 in accordance with the memorandum establishing the team. (Recommendation 6) We provided a draft of this report to DOD for review and comment. In written comments that are reproduced in appendix IV, DOD concurred with our recommendations. DOD officials provided separate oral technical comments, which we incorporated as appropriate. In its response, DOD provided new information on a timeline for completing the outstanding section 911 requirements. Specifically, DOD updated its internal deadline for submission of the organizational strategy to the Secretary of Defense from July 2019 to August 2019. DOD also stated that it plans to issue the guidance on cross-functional teams and training for cross-functional team members, their supervisors, and presidential appointees by September 2019 and complete its report on the successes and failures of cross-functional teams by November 2019. We updated our first five recommendations to reflect this information. Establishing these timelines is an important step forward in meeting the statutory requirements under section 911 as well as addressing our recommendations. As part of our next and final audit of DOD’s implementation of section 911 requirements, we will assess the extent to which the department has met these new internal deadlines and fully addressed our recommendations in this report. Fully addressing these outstanding requirements will strengthen DOD’s ability to effectively implement its cross-functional teams and advance a collaborative culture within the department. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and DOD’s Deputy Chief Management Officer. In addition, the report is available at no charge on our website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Section 911 of the NDAA for Fiscal Year 2017 included a provision for us—every 6 months after the date of enactment on December 23, 2016, through December 31, 2019—to submit to the congressional defense committees a report. Each report is to set forth a comprehensive assessment of the actions that DOD has taken pursuant to section 911 during each 6-month period and cumulatively since the NDAA’s enactment. Table 1 identifies our four prior reports on DOD’s implementation of section 911 and the status of the five recommendations from those reports. Section 911 of the National Defense Authorization Act for Fiscal Year 2017 requires the Secretary of Defense to take several actions. Table 2 summarizes these requirements, the due date, and the date completed, if applicable, as of June 2019. In February 2018, we reported on eight leading practices for implementing effective cross-functional teams. Table 3 identifies these leading practices and their related key characteristics. In addition to the contact named above, Margaret Best (Assistant Director), Tracy Barnes, Arkelga Braxton, Sierra Hicks, Michael Holland, Matthew Kienzle, Amie Lesser, Ned Malone, Judy McCloskey, Sheila Miller, Richard Powelson, Daniel Ramsey, Ron Schwenn, and Sarah Veale made key contributions to this report.", "summary": "DOD continues to confront organizational challenges that hinder collaboration. To address these challenges, section 911 of the NDAA for Fiscal Year 2017 directed the Secretary of Defense to, among other things, issue an organizational strategy that identifies critical objectives that span multiple functional boundaries; establish cross-functional teams to support this strategy; and provide related guidance and training. The NDAA for Fiscal Year 2017 also included a provision for GAO to assess DOD's actions in response to section 911. This report assesses the extent to which DOD has made progress in implementing the requirements of section 911, including establishing a new cross-functional team on electromagnetic spectrum operations. GAO reviewed documentation, interviewed cross-functional team members and other DOD officials, and compared DOD's actions to section 911 requirements and leading practices for cross-functional teams. The Department of Defense (DOD) is up to 21 months late in fully addressing five of seven requirements of section 911 of the National Defense Authorization Act (NDAA) for Fiscal Year 2017. These remaining five requirements are designed to strengthen collaboration within the department to foster effective and efficient achievement of objectives and outputs (see figure). DOD has not addressed most of these remaining requirements of section 911 largely because the Chief Management Officer (CMO) has not approved the documents drafted to meet the requirements or coordinated department-wide review of the documents and provided them for Secretary of Defense issuance. According to Office of the CMO (OCMO) officials, some of the draft documents were provided to the CMO for review and approval as early as August 2018. After providing a draft of this report to the department for comment, GAO learned that the organizational strategy was circulated for department coordination in July 2019, with components expected to provide input by August 2019. However, while the OCMO has set an internal time frame for the organizational strategy, it has not set similar time frames for completing the other four remaining requirements, such as delivering guidance and training on cross-functional teams. GAO previously reported that establishing internal deadlines with key milestones and deliverables is important for tracking progress and implementing actions effectively. DOD established a cross-functional team pursuant to section 911 on electromagnetic-spectrum operations (EMSO), but according to a team official, funding for the team was delayed. EMSO refers to those activities consisting of electronic warfare and joint electromagnetic-spectrum management operations used to exploit, attack, protect, and manage the electromagnetic operational environment to achieve the commander's objectives. According to the memorandum establishing the team, the CMO is required to provide administrative support to and coordinate with the team to ensure adequate resources are immediately available. However, team officials stated that this funding was delayed in part because of disagreements over responsibility for funding the team under the terms of this memorandum. Moreover, according to a team official, plans for funding in future fiscal years have not been developed. If DOD does not clarify roles and responsibilities for funding the team, the CMO and the EMSO team may face additional delays securing funding, which could negatively affect the team's ability to conduct its work and meet its objectives. GAO is making six recommendations, including that DOD set and ensure that it meets specific internal deadlines for review and approval of outstanding requirements of section 911, and that DOD clarify roles and responsibilities for providing funding for the EMSO cross-functional team. DOD concurred with GAO's recommendations and set deadlines for addressing the remaining requirements.", "document_type": "gao"}
{"report": "The DATA Act was enacted May 9, 2014, for purposes that include expanding on previous federal transparency legislation by requiring the disclosure of federal agency expenditures and linking agency spending data to federal program activities, so that both policymakers and the public can more effectively track federal spending. The act also holds agencies accountable for submitting complete and accurate data to Treasury and requires that agency-reported award and financial data comply with OMB and Treasury data standards. The DATA Act requires OMB and Treasury to establish government-wide data standards that to the extent reasonable and practicable provide consistent, reliable, and searchable spending data for any federal funds made available to or expended by federal agencies. These standards specify the data elements to be reported under the DATA Act and define and describe what is to be included in each data element, with the aim of ensuring that data will be consistent and comparable. The DATA Act requires OMB and Treasury to ensure that the standards are applied to the data made available on USAspending.gov and also requires agencies’ OIGs and GAO to review these data and report on their completeness, timeliness, accuracy, and quality. USAspending.gov has many sources of data, including data that agencies submitted to Treasury through their financial management systems and other data extracted from government-wide award systems that collect data from federal agencies and external award recipients. Treasury’s DATA Act Broker (Broker) is a key component of the data collection and reporting framework. The Broker enables agencies to upload, validate, and certify financial data and create linkages between the financial and award data for publication on the USAspending.gov website. Agencies are expected to submit three data files with specific details and data elements to the Broker from their financial management systems in accordance with Treasury guidance documents. File A: Appropriations account includes summary data such as the fiscal year cumulative federal appropriations account balances and includes data elements such as the agency identifier, main account code, budget authority appropriated amount, gross outlay amount, and unobligated balance. File B: Object class and program activity includes summary data such as the names of specific activities or projects as listed in the program and financing schedules of the annual budget of the U.S. government. File C: Award financial includes award transaction data such as the obligation amounts for each federal financial award made or modified during the reporting quarter (e.g., January 1, 2017, through March 31, 2017). The Broker also extracts data from four government-wide award reporting systems: the Federal Procurement Data System–Next Generation (FPDS-NG), System for Award Management (SAM), Financial Assistance Broker Submission (FABS), and the FFATA Subaward Reporting System (FSRS). These systems supply award and sub-award data (e.g., federal grants, loans, and contracts) to USAspending.gov. The systems compile data that agencies and external federal award recipients submit to report procurement and financial assistance award data required under the Federal Funding Accountability and Transparency Act of 2006 (FFATA). The four files produced with data that the Broker extracts from the four systems are as follows: File D1: Procurement includes data on the attributes of the award and the awardee and the recipient of the award (extracted from FPDS-NG on a daily basis) for procurement awards (contracts), if any, and contains elements such as the total dollars obligated, current total value of award, potential total value of award, period of performance start date, and other data to identify the procurement award. File D2: Financial assistance includes award and awardee attribute data (extracted from FABS nightly) on financial assistance awards (grants and loans) and contains data elements such as the federal award identification number, the total funding amount, the amount of principal to be repaid for the direct loan or loan guarantee, the funding agency name, and other data to identify the financial assistance award. File E: Additional awardee attributes includes additional data (extracted from SAM) on the award recipients and contains elements such as the awardee or recipient unique identifier; the awardee or recipient legal entity name; and data on the award recipient’s five most highly compensated officers, managing partners, or other employees in management positions. File F: Subaward attributes includes data (extracted from FSRS) on awards made to subrecipients under a prime award, if any, and contains elements such as the subaward number, the subcontract award amount, total funding amount, the award description, and other data to facilitate the tracking of subawards. According to Treasury guidance, after agencies submit Files A, B, and C, the Broker runs a series of validations and produces warnings and error reports for agencies to review. After passing validations for these three files, the Broker generates Files D1 and D2 containing details on procurement and assistance awards and performs a cross-file validation of linkages between File C and Files D1 and D2, which generates error and warning reports, as appropriate. The Broker also generates Files E and F containing data on highly compensated officers and subawards associated with the prime awards. There are no field-level or cross-file validations for Files E and F. With their quarterly submission, agency senior accountable officials (SAO) are required to certify the data submissions and to provide assurance over the alignment of Files A through F and that the data are valid and reliable in accordance with OMB guidance. According to Treasury officials, once the certification is submitted, a sequence of computer program instructions or scripts is issued to transfer the data from the Broker to tables set up in a database used as a source for the data on the website. Data are then displayed on USAspending.gov along with certain historical data from other sources, including Monthly Treasury Statements. OMB and Treasury implementation guidance called for customer agencies to consider how best to leverage their SSPs to capture data for their submissions, engage with their SSPs throughout the implementation process, and document the SSP role in agency DATA Act submissions. According to the SSPs, 60 non–Chief Financial Officers Act of 1990 (CFO Act) agencies use a federal SSP for all or part of the data submissions out of the 82 reporting data under the DATA Act as of the fourth quarter in fiscal year 2018. In 2014, Treasury designated four federal financial management SSPs to provide financial management services to other federal agencies. Although the four Treasury-designated federal financial management SSPs have changed over the years, the four federal financial management SSPs, which performed DATA Act services for external customers as of December 2018, are as follows: The Administrative Resource Center (ARC) is a Treasury SSP that provided financial management services to 42 customer agencies external to Treasury. According to ARC, 21 of those agencies received DATA Act services from ARC. The Enterprise Services Center (ESC) is a Department of Transportation financial management SSP that provided services to seven external customer agencies. According to ESC, six of those customer agencies received DATA Act services from ESC. Pegasys Financial Services (PFS) is a Department of Agriculture financial management SSP that provided services to 37 external customer agencies. According to PFS, 24 of those customer agencies received DATA Act services from PFS. The Interior Business Center (IBC) is a Department of the Interior financial management SSP that provided services to 18 external customer agencies. According to IBC, nine of those customer agencies received DATA Act services from IBC. The DATA Act requires agencies’ OIGs to issue reports assessing the quality of the agencies’ spending data submissions and compliance with the DATA Act. In the OIGs’ reports covering their agencies’ second quarter fiscal year 2017 submissions, nine OIGs reported issues with their agencies’ use of an SSP for DATA Act submissions. Five of the nine OIGs issued recommendations related to these issues, and four agencies concurred with the recommendations. For example, one OIG recommended that its agency work closely with its SSP to address timing and coding errors that the SSP caused for future DATA Act submissions. Another OIG recommended that its agency work with its SSP to identify OMB requirements that the SSP is to perform and insert them into the service-level agreement, in order to address errors caused by confusion as to whether the SSP or the agency should submit certain types of data. Although our prior reports on the DATA Act included recommendations, our recommendations were not related to SSPs’ implementation of the DATA Act. The DATA Act requires OIGs and GAO to issue their second reports on data quality in November 2019. The 27 customer agencies that responded to our survey reported that the four federal financial management SSPs provide a variety of services (see fig. 1). All 27 agencies reported using their federal financial management SSPs for DATA Act services, and almost all of the agencies used their SSPs for several other financial management services, such as general ledger accounting, financial reporting, and hosting the customer agencies’ financial systems. As such, the SSPs play a key role in helping to ensure that these customer agencies successfully carry out the requirements of the DATA Act and submit Files A, B, and C from their financial management systems. In addition, 17 agencies reported using their SSPs for payroll or budget execution services, while fewer reported using their SSPs for other financial management services, such as grant or loan processing. DATA Act services. All 27 customer agencies responding to our survey reported using an SSP for DATA Act services. As discussed in more detail below, these DATA Act services may include activities such as preparing DATA Act files from financial systems, consolidating DATA Act files from multiple agency component entities, reconciling DATA Act files to other source data, and uploading DATA Act files to the Broker for validation. General ledger accounting. Twenty-six agencies reported using SSPs for general ledger accounting, which may include activities such as general ledger setup and maintenance, posting transactions to the general ledger, accrual and liability processing, and period-end general ledger closing. Financial reporting. Twenty-six agencies also reported using SSPs for financial reporting, which may include activities such as Treasury reporting, financial statement preparation, cash forecasting and reporting, and financial performance and operational reporting. Financial system hosting. Twenty-five agencies reported using SSPs for financial system hosting, which may include services such as systems management and monitoring, disaster recovery, help desk administration, network security compliance and controls, and continuity of operations plans and testing. Invoice processing. Twenty-four agencies reported using SSPs for invoice processing, which may include services such as recording receiving and acceptance reports, recording invoices, matching invoices to receiving and acceptance reports, and routing invoices to obtain approval for payment. Budget execution. Seventeen agencies reported using their SSPs for financial management services related to budget execution, which may include activities such as budget setup and maintenance, fund allocation and control, and budgetary reporting. Payroll. Seventeen agencies reported using SSPs for payroll. SSP payroll services may include recording payroll and benefit payments; reconciling payroll service data with financial management data; and recording credits, payment adjustments, and employee receivable offsets. Procurements/contracts. Thirteen agencies reported using SSPs for procurement and contract services. SSP procurement and contract services may include recording credits and payment adjustments; auditing payments; processing payments for incurred expenses and payments in advance; and capturing award identifier data, such as the Procurement Instrument Identifier (PIID) and agency Unique Record Identifier (URI) to support DATA Act reporting. Grants processing. Seven agencies reported using SSPs for grants processing, which may include recording requests for grant payments, matching grant payment requests to obligating documents, routing grant payment requests for approval, and generating payment transactions. These processes also include payments for expenses and payments in advance as well as capturing award identifier data, such as Federal Award Identification Numbers (FAIN) and Catalog of Federal Domestic Assistance (CFDA) codes to support DATA Act reporting. Loans processing. Three agencies reported using SSPs for loan processing. Loan processing services may include recording requests for loan payments, matching loan payment requests to obligating documents, generating payment transactions, resolving payment issues, and recording credits and payment adjustments. All 27 agencies responding to our survey reported that their federal SSPs perform a variety of DATA Act services or activities, as shown in figure 2. Preparing data files A, B, or C and uploading them to the Broker are the most prevalent DATA Act services or activities that the federal SSPs perform, whereas fewer than half of the SSPs certify and publish the files for the agency after receiving agency approval. All 27 agencies reported that their SSPs prepare at least one of the Files A, B, or C using data from either SSP or customer agency financial systems. In addition, 15 of the 27 agencies reported that their SSPs consolidate DATA Act files from multiple agency components. Seventeen agencies reported that their SSPs reconcile Files A, B, or C to other source data. For example, a reconciliation of general ledger and subledgers may include verifying that (1) general ledger account balances can be traced to aggregated or discrete agency transactions and (2) aggregated or discrete agency transactions can be traced to the point of origination and source documents. Twenty-five of the 27 agencies reported that their SSPs upload Files A, B, or C to the Broker for validation. In turn, the Broker runs a series of data validations and produces warnings and error reports for agencies to review after the files are submitted. Twenty-one agencies reported that their SSPs address these warnings and errors on their behalf. After warnings have been reviewed and all errors have been addressed, Files A, B, and C have been uploaded and Files D1, D2, E, and F have been generated, the agency’s SAO is required to certify the validity and reliability of the data submissions in accordance with OMB guidance. Twenty-four agencies reported that their SSPs provide final Files A, B, or C for the customer agency to review and certify in the Broker, and 11 agencies reported that their SSPs finalize the files in the Broker and click the Certify and Publish button after receiving agency approval to certify. We asked customer agencies in our survey to specify the challenges associated with using an SSP that they experienced since their initial DATA Act submission; the SSP’s role in these challenges; and whether the challenges affected the timeliness, completeness, or accuracy of their submissions. Sixteen of the 27 customer agencies that responded to our survey identified one or more challenges associated with using an SSP (see fig. 3), many of which affected the timeliness, completeness, and accuracy of agency submissions. The survey questions and summarized results are shown in appendix II. In addition, officials from all four federal SSPs described various challenges they experienced in helping their agency customers with DATA Act submissions. The challenges reported by these 16 customer agencies and the federal SSPs are summarized below. Depending on the effectiveness of agencies’ and SSPs’ actions to address them (as discussed further below), these challenges may increase the risk that agencies will be unable to submit quality data in accordance with the DATA Act. Dependencies. Ten agencies reported that they have experienced challenges related to agency submission activities that depend on relationships with, or actions being taken by, the SSP before the agency can proceed. One agency reported that it must rely on its SSP to prepare, validate, and finalize all DATA Act files prior to agency certification and the files are often submitted close to the due date. Another agency reported that its SSP provided DATA Act submission files to the agency the day before or near the certification deadline. Relying on SSPs to prepare DATA Act files in a timely manner increases the risk that agencies may be unable to certify and publish their DATA Act submissions on time. Resources. Seven agencies said that they have experienced resource challenges related to a lack of funding or human resources at the customer agency or its SSP. One agency noted that its SSP has only a small group of people that assist with all of its SSP’s services, making it challenging in particularly busy seasons (such as the close of the fiscal year) for the SSP to meet internal deadlines and resolve data discrepancies affecting the timeliness, completeness, and accuracy of data submissions. Another agency reported challenges with funding resources, noting that the agency has been unable to use its SSP’s integrated financial and procurement system because of the costs associated with implementation, operation, and maintenance. The shortage of resources to bring on more staff or improve systems increases the risk that agencies may be unable to submit quality data and fully carry out DATA Act requirements. Competing priorities. Five agencies said that they have experienced challenges related to statutory, regulatory, policy, or other matters that have competing priorities or conflicting requirements that may affect an agency or its SSP’s DATA Act submission process. One agency reported that the fourth quarter DATA Act reporting deadline falls within fiscal year- end reporting time frames, requiring the agency to prioritize fiscal year- end reporting over some of the DATA Act reporting tasks. Similarly, officials from an SSP told us that they also experienced challenges with the short turnaround times required to incorporate system updates for DATA Act submissions. Data quality. Four agencies reported that they have experienced challenges related to meeting DATA Act requirements for data quality because they use an SSP, including completeness and accuracy of agency data to be reported as well as SAO certification and reporting of nonfinancial data elements. One agency reported that its SSP included extraneous transactions in its File C that were not required for DATA Act reporting. This created a high volume of warning messages when the Broker compared the data with file D2 during the validation process. Another agency reported that its SSP provided incomplete files for agency certification and that the data in the files did not reflect the data in the agency’s financial reports. These challenges not only increase the risk of lower-quality agency data submissions but may also require SSPs and customer agencies to expend additional resources to address warning messages that the Broker generated. Guidance. Four agencies reported experiencing challenges involving incomplete, unclear, missing, and evolving OMB and Treasury guidance related to SSP implementing requirements and Broker changes, including data elements, the technical schema, and other key policies. One agency noted that guidance on performing quarterly certifications is not readily available. Two agencies reported challenges involving a lack of guidance on how to communicate error corrections and desired changes to their SSPs. Lack of guidance could result in misunderstandings or miscommunications between the SSP and its customer agency, increasing the risk of delays or errors in the agency’s data submissions. Technology. Four agencies reported that they have experienced technology challenges with developing and submitting required files. These challenges include SSP infrastructure issues, such as integrating multiple existing and disparate management systems or their SSPs needing to modify existing systems to implement the DATA Act. Agencies reported that some systems are unable to include the required data elements for all reported transactions. Similarly, an SSP official told us prior to our survey of customer agencies that the SSP also experienced technical challenges with systems and data that have since been resolved. Another SSP experienced challenges with its financial systems not capturing award identification data elements, such as the PIID. Such limitations in customer agency and SSP technology may require the use of limited resources for error corrections and manual work-arounds, increasing the risk of reporting errors, and may hamper customer agencies’ and SSPs’ ability to submit quality data in accordance with the DATA Act. Project management. Two agencies reported that they have experienced challenges related to their SSPs’ project management, such as the lack of a designated project manager and inadequate documentation of progress made or key decisions. Specifically, both customer agencies said that their SSPs did not provide data in a timely manner for their review prior to submission. One agency reported that although this did not affect the timeliness of its submission, it affected data quality because the agency did not have sufficient time to test and implement sufficient internal controls and validation procedures prior to data being published on USAspending.gov. Additionally, the same agency reported that there is no senior project manager at the SSP who oversees the processes used to provide financial management services to the agency. This challenge may affect agencies’ ability to resolve errors, increasing the risk that they submit incorrect data. However, none of the four federal SSPs described any project management challenges when we asked them prior to our survey what challenges they faced in carrying out their roles and responsibilities for assisting their customer agencies with implementing the DATA Act. We asked customer agencies in our survey to describe the steps they and their SSP have taken to address reported challenges and help mitigate risks associated with them. Of the 16 agencies that identified challenges, 12 agencies reported that they had already taken steps to address them and five agencies said they were aware of steps their SSPs had taken. As discussed in more detail below, communication and coordination between the SSP and Treasury, as well as customer agency technological improvements and manual work-arounds were the steps most often reported by agencies to address identified challenges. Communication and coordination. Eight of the 12 agencies described communication efforts with their SSPs or Treasury to facilitate coordination and seek information needed to address their challenges associated with using an SSP. These efforts include requesting information from the SSP, Treasury, and other government resources to obtain additional knowledge regarding the DATA Act and to prepare internal guidance and procedures. One agency reported implementing a weekly meeting with its SSP on DATA Act reporting. Another agency reported that its SSP has been very proactive in sharing information (bulletins, updates, etc.) and assisting with submitting DATA Act information. According to the eight customer agencies, increased communication and coordination has helped to address several technology, dependency, and resource challenges. Technology improvements. To address technology challenges, four agencies discussed making improvements in technology at both the agency and the SSP. The improvements include implementing an integrated financial and procurement system platform and working with the software vendor to obtain access to FPDS-NG for anticipated 2019 procurement activity reporting. Another agency is currently implementing a technological solution to aid in consolidating and reconciling files. In addition to technological solutions, two agencies reported using manual work-arounds, such as developing and implementing internal manual processes to reconcile and correct data files. Some agencies also discussed actions their SSPs had taken to address technology issues, including two SSPs that are working with the developers to address software issues. One agency reported that in addition to addressing technology challenges, these improvements also provided substantial cost savings. We also asked the 27 customer agency survey respondents to specify the internal control processes and activities they use to assure the quality of data submitted to the Broker. Twenty-four of 27 agencies reported that they use various processes and activities to provide such assurance. Specifically, these 24 agencies reported that they reconcile data files to other agency data and sources (e.g., the Governmentwide Treasury Account Symbol Adjusted Trial Balance System). These reconciliations can help identify errors in data files and ensure that they are consistent with other agency data. In addition, 18 agencies reported that they review their SSPs’ Service Organization Control (SOC) reports to identify any internal control deficiencies, and nine agencies reported that they implemented controls to address control deficiencies identified in their SSPs’ SOC reports. Twenty agencies reported that they review or verify agency data displayed on USAspending.gov. By reviewing these data, agencies can confirm that the data that they uploaded to the Broker are presented accurately on the website. Nineteen agencies reported that they incorporate the results of OMB Circular No. A-123 reviews that affect their DATA Act submissions. OMB Circular No. A-123 provides a methodology for agency management’s reporting on internal controls over reporting, and it also establishes an assessment process based on our Standards for Internal Control in the Federal Government that management must implement in order to properly assess and improve internal controls over operations, reporting, and compliance. Twenty of the 27 agencies that responded to our survey described useful practices in working with an SSP on DATA Act submissions. These agencies reported most often that discussing issues with the SSP and performing data reconciliations or comparisons were helpful. For example, 12 agencies reported that working and communicating with the SSP was useful. They also reported that having a readily available point of contact for better communication and communicating early about the need to complete the DATA Act submission helps to resolve any concerns prior to the due date. One agency reported that it conducts weekly meetings with its SSP to discuss key topics, including implementation, data quality, and reporting processes and procedures. Eleven agencies reported that conducting data reconciliations or comparisons, and creating a standard operating procedure to ensure that their data are consistently reviewed, reconciled, corrected, and certified, was also useful. One agency noted that in addition to the quarterly files that require certification, the SSP also provides monthly files that the agency can review to provide additional time to correct any identified errors. These 20 agencies also suggested other practices for successfully working with an SSP on DATA Act submissions, such as automating reconciliations and other internal control processes to increase efficiency, implementing continuous training and monitoring, assigning an accountant as an agency contact, and conducting an analysis of agency risk as recommended in the Data Quality Playbook. One agency reported that it performs extensive comparisons of agency-generated data reports to SSP-prepared data files, and that it partially automated this process to help increase efficiency. Nine of the 16 agencies that identified challenges associated with using an SSP reported that their agencies need to take additional steps to address their identified challenges. Some of these agencies reported that they still need to address issues such as correcting data and improving communication with their SSPs. These agencies also reported the need to negotiate annual service-level agreements with their SSPs to address resource and competing priority concerns. One of these agencies reported that it continues to work with its SSP to understand what is lacking in the process of correcting PIID information for obligations. A few agencies reported the need to develop internal guidance on topics such as data quality plans per OMB guidance and a reconciliation process to address their data quality challenges. Finally, one agency reported that it is in the process of hiring additional personnel to address its challenges with competing priorities. While customer agencies are primarily responsible for the quality of their DATA Act submissions, six of the 16 agencies that reported challenges reported that their SSPs also need to take certain steps to address identified challenges, such as communicating with the customer agency and making technology improvements. For example, one agency reported that its SSP does not provide the customer agency with updated data submission files after the agency has requested changes. The customer agency suggested that the SSP provide updated files more often to help the agency ensure that the changes are included in the final file it submits to the Broker. Another agency reported that its SSP has been experimenting with different methods to eliminate cross-file warnings and errors in File C that need to be addressed. Five of the 27 customer agencies we surveyed reported that additional tools or guidance from OMB, Treasury, or other entities (such as the SSP) could assist agencies with using an SSP for DATA Act submission. Specifically, three agencies reported that they would like to have guidance, including standard operating procedures, for communicating and working with their SSPs. One agency suggested additional OMB or Treasury training on compliance with the DATA Act, and one agency suggested improvements by Treasury to prevent Broker errors that result from normal business scenarios and require manual work-arounds to the agencies’ system-generated files. OMB staff told us that they are involved with the DATA Act Executive Steering Committee, working closely with Treasury to oversee all aspects of both policies and implementation related to federal spending transparency efforts. According to OMB staff, neither the SSPs nor their customer agencies have reported any current challenges with DATA Act submissions to OMB. OMB staff stated that effective implementation of OMB Memorandum M-18-16 guidance to agencies and SSPs, which discusses establishing entity-level controls related to using SSPs, would help to ensure that the SSPs provide quality services. In April 2019, OMB issued Memorandum M-19-16 on shared services, which among other things described the process and desired outcomes for shared services and established a governance and accountability model for achieving them. For example, as it relates to the DATA Act, the memorandum calls for the Shared Services Governance Board to leverage the DATA Act Executive Steering Committee’s work on DATA Act standards. Treasury officials told us prior to our survey that they held two workshops for SSPs in the early stages of implementation to address specific concerns and questions on DATA Act implementation. Treasury officials stated that the only challenge reported by SSPs to the department related to linking and certifying award data using the Broker when the awarding agency and the file C reporting agency were different. To address this concern, Treasury added clarification in guidance on files A, B, and C submissions and a new Broker feature allowing agencies to specify whether the award data submitted in Files D1 and D2 comes from the funding agency or the awarding agency. We provided a draft of this report to OMB; Treasury, including ARC; ESC; PFS; and IBC for comment. OMB, ESC, and IBC told us that they had no comments on the draft report. Treasury and PFS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Director of the Office of Management and Budget, the Secretary of the Treasury, the four federal financial management shared service providers, and interested congressional committees and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-9816 or rasconap@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. The Digital Accountability and Transparency Act of 2014 (DATA Act) requires us to review Office of Inspector General (OIG) reports and issue reports of our own assessing and comparing the completeness, timeliness, accuracy, and quality of the data that federal agencies submit under the act and the implementation and use of data standards. We issued our first report on data quality in November 2017, as required. In July 2018, we issued a report on our review of OIG reports on agencies’ first DATA Act submissions and in the course of our review found that some OIGs reported challenges involving the use of federal shared service providers (SSP) that helped agencies implement the DATA Act. For this report, our objectives were to describe (1) the types and variations of services that the federal SSPs provide to their financial management customer agencies to assist them in implementing the DATA Act and meeting the act’s requirements and (2) any challenges that federal SSPs and their financial management customer agencies have encountered in their efforts to ensure the quality of data submissions consistent with the standards established under the DATA Act and the steps they have taken to address those challenges. To address our first objective, we interviewed four federal SSPs and surveyed their financial management customer agencies to identify the types and variations of services the SSPs provide related to DATA Act implementation and meeting the act’s requirements. We also obtained and reviewed selected service-level agreements executed between the four SSPs and their financial management customer agencies to determine the types and variations of DATA Act services that the SSPs provided to them. In December 2018, we emailed a survey questionnaire to 67 customer agencies that the four federal SSPs told us were external customers for DATA Act or other financial management services and that also submitted data under the DATA Act as of December 2018. During the survey, we determined that 60 of those 67 agencies actually received DATA Act services from a federal SSP and were eligible members of our study population. We received survey responses from 31 agencies by our January 2019 deadline and, based on our review, determined that 27 were eligible and sufficiently complete for our purposes. After excluding ineligible agencies from our population, the response rate was 45 percent. In developing, administering, and analyzing the survey, we took steps to minimize the five types of potential errors, described below, that may affect survey results. Because we surveyed all agencies in our population, there was no sampling error. To minimize the effects of coverage error—the exclusion of some eligible members of the population, or inclusion of ineligible members—we identified as ineligible and removed seven initially identified agencies because we determined that they did not use a federal SSP to provide DATA Act services. Measurement error may result from differences in how a question is interpreted and the sources of information available to respondents. To help prevent measurement error, we conducted pretests of the draft questionnaire with four customer agencies, each using a different SSP, and made revisions to improve the validity and minimize the burden of responding to our questions. Nonresponse error may result when a survey fails to capture information from all agencies selected in the survey, and it may introduce bias if those agencies that did not respond would have given materially different answers than those that did. To maximize survey response, we sent multiple email reminders to the surveyed agencies and extended the submission deadline. While we do not have evidence of material bias from those not responding, we limit our survey results in this report as representing only those 27 agencies responding. Finally, to limit the possibility of processing error, survey responses were checked for invalid or illogical answer patterns, and data edits were made as necessary to facilitate processing and analysis of the results. This analysis was verified by a separate data analyst. Table 1 lists the 27 customer agencies (by shared service provider) for which we obtained, reviewed, and included customer agency survey responses. The survey questions and summarized results are shown in appendix II. To meet our second objective, we interviewed the officials of the four federal SSPs prior to our survey to obtain information on the challenges the SSPs and their customer agencies encountered and steps taken to address them. We reviewed and analyzed the 27 customer agency survey responses to identify challenges responding agencies reported since their initial DATA Act submissions because they are working with an SSP and any steps SSPs and their financial management customers took to address challenges and to help ensure the quality of data submissions. We did not corroborate the customer agencies’ survey responses with the four federal SSPs, Office of Management and Budget (OMB), or Department of the Treasury (Treasury). To identify steps taken to address challenges, we also obtained and reviewed any reports related to DATA Act implementation that the SSPs or their respective OIGs produced. In addition, we interviewed OMB staff and Treasury officials about any guidance they have provided or actions they have taken to assist the four SSPs and their financial management customers with any challenges related to DATA Act compliance. We conducted this performance audit from July 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. United States Government Accountability Office Survey of Customer Agencies’ Use of Shared Service Providers for Digital Accountability and Transparency Act (DATA Act) Submissions This questionnaire asks about your agency’s use of and relationship with your federal shared service provider (SSP). It should be completed by officials knowledgeable about the DATA Act services your federal SSP provides. Please submit only one survey response per agency, but consult with other officials as needed; when answering, please consider any agency component activity and experiences together, and answer at the agency level to the best of your ability. If your agency uses more than one federal SSP, please be sure to include information about both SSPs in your answers. This is a fillable PDF form. You can click buttons and type into highlighted boxes throughout the form; the boxes will accommodate more text than is immediately visible. Save this file to a drive now, and save your answers periodically as you go. When completed, save this file and email it to DATAActImplementation@gao.gov. If a “Submit” button appears in the upper right corner of your screen, you may also use that to automatically email your completed questionnaire (some viewers will not see this button depending on your system’s Javascript settings). If you have any questions, or feel this questionnaire was sent to your office in error, please contact 1. What is the name of the person completing this questionnaire, title, agency name, and contact information? (Please submit only one survey response per agency) 2. What is the name and title of the individual who reviews and certifies your agency’s DATA Act submission in the Treasury broker as ready for publishing? a. Is this person also your agency’s DATA Act Senior Accountable Official (SAO)? 3. Which federal SSP(s) (if any) does your agency currently use for the financial management services listed below? (Select all that apply) 4. Which specific DATA Act services/activities does the federal SSP(s) identified in question 3 perform for your agency (in whole or in part)? (Select all that apply) Finalize the files in the Treasury broker by clicking the “Certify and Publish” button after receiving agency certification Other SSP services or activities (specify in the box below) 5. Which, if any, of the following activities does the federal SSP(s) you identified above initiate for your agency (in whole or in part) for the broker to perform? (Select one answer in each row) 6. What are the steps taken by your agency to certify the final DATA Act files before they are published (e.g., by whom and how are the data validated, reviewed, and comments (if any) provided on the files)? Summary included in report 7. Since your agency’s initial DATA Act submission, has your agency experienced any challenges in the following areas because it is working with an SSP, and did the challenge(s) in working with the SSP have an impact on the timeliness, completeness, and accuracy of any of your agency’s submissions? (Select all the area(s) with challenges and any impacts that apply) Technology issues Including challenges with developing and submitting required files, and SSP infrastructure issues such as integrating multiple existing and disparate financial and management systems, or the SSP needing to install new systems or modify existing systems to implement the DATA Act. Dependencies Agency submission activities depend on relationships with or actions being taken by the SSP before the agency can proceed. Guidance Incomplete, unclear, missing, and evolving guidance related to the SSP implementing requirements and broker changes, including data elements, the technical schema, and other key policies issued by OMB and Treasury. Resources Lack of funding or human resources by your agency or SSP. Project management Challenges related to the SSP’s project or program management, such as lack of a designated project manager and inadequate documentation of progress made or key decisions. Data quality Issues related to meeting DATA Act requirements for data quality because of the use of an SSP, including completeness and accuracy of agency data to be reported, as well as Senior Accountable Official certification and reporting of nonfinancial data elements. Competing priorities Statutory, regulatory, policy or other matters that have competing priorities or conflicting requirements that may affect an agency or their SSP’s DATA Act submission process. Challenge? Contract Management Challenges related to the management of the Service Level Agreement (SLA) and/or tasks and services that the SSP provides for the customer agency or the SSP should be providing but are not in the SLA. Other challenges (specify in box below) 8. What were the specific challenge(s) with your agency’s DATA Act submissions identified in question 7, and how did your SSP play a role? If your agency used multiple SSPs, please specify to which provider the challenge(s) was related. a. What steps, if any, has your agency taken to address these challenge(s)? b. What steps, if any, remain to be taken by your agency to address these challenge(s)? c. What steps, if any, are you aware of that your SSP has taken to address these challenge(s)? d. What steps, if any, remain to be taken by your SSP to address these challenge(s)? Summary included in report 9. What management or oversight practices has your agency found to be useful in working with your SSP on DATA Act submissions? 10. What internal control processes and activities does your agency use to provide assurance over the quality of data submitted to the Treasury broker and displayed on USAspending.gov? Internal control processes and activities: Reconcile data files to other agency data and sources (e.g., SF 133, GTAS) Review SSP’s Statement on Standards for Attestation Engagements No. 18/Service and Organization Controls (SOC) reports to identify any internal control deficiencies. (Describe internal control deficiencies related to DATA Act submissions, if any, in box below) Implement complementary controls to address SSP control deficiencies identified in the SOC report. (Describe controls implemented, if any, in box below) Review/verify agency data displayed on USAspending.gov Incorporate the results of A-123 reviews that have an impact on DATA Act Other internal control processes or activities (specify in box below) 11. What additional tools or guidance, if any, are needed from OMB, Treasury, or others to assist with your agency’s use of the SSP or DATA Act submission? 12. Please provide any additional comments or explanations not already discussed above. Please save and e-mail your responses to DATAActImplementation@gao.gov. Thank you for completing our questionnaire! In addition to the contact named above, Michael LaForge (Assistant Director), Laura Pacheco (Auditor in Charge), Umesh Basnet, Thomas Hackney, Roy Kilgore, and Diane Morris made major contributions to this report. Other key contributors include Dave Ballard, Jenny Chanley, Peter Del Toro, Patrick Frey, Ricky Harrison, Maxine Hattery, Jason Kelly, James Kernen, Christina Quattrociocchi, Carl Ramirez, Michelle Sager, and James Sweetman. DATA Act: OMB Needs to Formalize Data Governance for Reporting Federal Spending. GAO-19-284. Washington, D.C.: March 22, 2019. Streamlining Government: OMB and GSA Could Strengthen Their Approach to Implementing a New Shared Services Plan. GAO-19-94. Washington, D.C.: March 7, 2019. Open Data: Treasury Could Better Align USAspending.gov with Key Practices and Search Requirements. GAO-19-72. Washington, D.C.: December 13, 2018. DATA Act: Reported Quality of Agencies’ Spending Data Reviewed by OIGs Varied Because of Government-wide and Agency Issues. GAO-18-546. Washington, D.C.: July 23, 2018. DATA Act: OMB, Treasury, and Agencies Need to Improve Completeness and Accuracy of Spending Data and Disclose Limitations. GAO-18-138. Washington, D.C.: November 8, 2017. DATA Act: As Reporting Deadline Nears, Challenges Remain That Will Affect Data Quality. GAO-17-496. Washington, D.C.: April 28, 2017. DATA Act: Office of Inspector General Reports Help Identify Agencies’ Implementation Challenges. GAO-17-460. Washington, D.C.: April 26, 2017. DATA Act: Implementation Progresses but Challenges Remain. GAO-17-282T. Washington, D.C.: December 8, 2016. DATA Act: OMB and Treasury Have Issued Additional Guidance and Have Improved Pilot Design but Implementation Challenges Remain. GAO-17-156. Washington, D.C.: December 8, 2016. DATA Act: Initial Observations on Technical Implementation. GAO-16-824R. Washington, D.C.: August 3, 2016. DATA Act: Improvements Needed in Reviewing Agency Implementation Plans and Monitoring Progress. GAO-16-698. Washington, D.C.: July 29, 2016. DATA Act: Progress Made but Significant Challenges Must Be Addressed to Ensure Full and Effective Implementation. GAO-16-556T. Washington, D.C.: April 19, 2016. DATA Act: Data Standards Established, but More Complete and Timely Guidance Is Needed to Ensure Effective Implementation. GAO-16-261. Washington, D.C.: January 29, 2016. DATA Act: Progress Made in Initial Implementation but Challenges Must be Addressed as Efforts Proceed. GAO-15-752T. Washington, D.C.: July 29, 2015.", "summary": "Over the past 2 decades, the federal government has undertaken efforts to save money and increase efficiencies by encouraging agencies to use administrative and operational services and processes that other federal and external parties provide, commonly referred to as shared services. The DATA Act was enacted to increase accountability and transparency and, among other things, establish government-wide data standards. Certain agencies have used shared services of federal SSPs to implement the act. The act also requires a series of oversight reports by agencies' Offices of Inspector General (OIG) and GAO. OIGs for five agencies have made recommendations related to agencies' use of SSPs for DATA Act services, and four agencies concurred with the recommendations. The objectives of this report are to describe (1) the types and variations of services that federal SSPs provide to their financial management customer agencies to assist them with implementing the DATA Act and meeting the act's requirements and (2) the challenges federal SSPs and their financial management customer agencies have encountered in their efforts to ensure the quality of data submissions consistent with DATA Act standards and steps they have taken to address those challenges. To address these objectives, GAO interviewed staff at four federal SSPs, OMB, and Treasury; reviewed selected agreements between the SSPs and their customer agencies; conducted a survey of customer agencies from December 2018 to January 2019; and analyzed the survey responses. GAO found that the 27 agencies that responded to its survey use federal shared service providers (SSP) for a variety of services, including financial system hosting, general ledger accounting, financial reporting, and various Digital Accountability and Transparency Act of 2014 (DATA Act) services. Sixteen of the 27 SSP customer agencies reported that they experienced challenges associated with using an SSP, many of which affected the timeliness, completeness, or accuracy of agency DATA Act submissions. Ten of these agencies experienced challenges with depending on an SSP to take actions before the agency could proceed. Agencies responding to GAO's survey also reported other challenges, such as a lack of guidance from the Office of Management and Budget (OMB) and the Department of the Treasury (Treasury), limited customer agency and SSP resources, SSP errors affecting data quality, and inadequate SSP project management activities. Twelve of these 16 agencies stated that they are taking steps to address these challenges—such as increasing communication with their SSPs, making technology improvements, and performing manual work-arounds to reconcile and correct data files. Nine agencies reported remaining additional steps, for example, correcting data errors and developing a reconciliation process and internal guidance on topics such as data quality plans. While agencies are primarily responsible for the quality of DATA Act submissions, five agencies also reported that their SSPs had taken similar steps to address identified challenges. Twenty of the 27 agencies described useful practices for working with SSPs on DATA Act submissions, including the agency discussing issues with the SSP and obtaining data files from the SSP each month to provide additional time to correct any identified errors. Treasury officials stated prior to GAO's survey that they held workshops for SSPs in the early stages of DATA Act implementation and clarified guidance issued in June 2018 to specifically address their concerns and questions. After GAO's survey, in April 2019, OMB issued a memorandum on shared services that among other things described the process and desired outcomes for shared services and established a governance and accountability model for achieving them.", "document_type": "gao"}
{"report": "Alaska’s location makes the United States an Arctic nation. Alaska has over 6,000 miles of coastline, and is bordered by the Beaufort, Chukchi, and Bering Seas; the Arctic Ocean; and the Bering Strait, whose jurisdiction is divided between the United States and Russia (see fig. 1). According to the 2010 Census, the U.S. Arctic coastal regions are home to about 26,000 people, including the cities of Nome, located near the Bering Strait, and Utqiagvik (formerly Barrow), the northernmost city in the United States. The U.S. Arctic coastal region is sparsely populated even by the standards of Alaska, which has the lowest population density of any state in the nation. Specifically, this region accounted for about 4 percent of Alaska’s total population of approximately 710,000 according to the 2010 Census. Alaska is also the largest state in the nation, and— given its size, terrain, environment, and population distribution—its transportation system is unique. Much of Alaska’s rail and highway infrastructure is located in the south central part of the state, and many U.S. Arctic cities and villages are accessible only by air or water. As Arctic waterways become more accessible due to declining sea ice, opportunities have increased to use maritime transportation to bring natural resources to market. The U.S. Arctic remains a frontier economy; many of its products and much of the value of commercial activities derive from natural resources. According to an assessment of undiscovered but technically recoverable oil and gas resources by the Bureau of Ocean Energy Management, the outer continental shelf regions of the U.S. Arctic’s Chukchi and Beaufort Seas contain about 24 billion barrels of oil and about 105 trillion cubic feet of natural gas. The U.S. Arctic also contains $1 trillion worth of minerals, such as zinc, nickel, and lead. The extraction of these natural resources presents technical challenges and requires large financial investments given the Arctic environment. Although warming over the past decades has made trans-Arctic maritime routes more accessible, Arctic sea ice extent remains seasonal, with most shipping occurring during a narrow window extending from summer to early fall. Arctic sea ice typically reaches its maximum extent in March and its minimum in September each year; as a result, the shipping season is typically from June through October. The minimum sea ice extent in September 2019 was tied with 2007 and 2016 as the second lowest on record since satellite observations began in 1979; the 13 lowest extents in the satellite record have all occurred in the last 13 years. As shown in figure 2, the September (minimum) sea ice extent in 2019 had a much smaller coverage area than the median September extent from 1981 to 2010. This contraction of sea ice over time has increased accessibility to the two key trans-Arctic maritime routes: the Northwest Passage (NWP) through the Canadian archipelago, and the Northern Sea Route (NSR) along the northern border of Russia. These two routes enable shipments between non-Arctic destinations, such as between Asia and Europe. However, most traffic in the U.S. Arctic is destinational, meaning it transports goods to and from the U.S. Arctic. Such traffic includes transporting natural resources extracted from the U.S. Arctic to the global marketplace and shipping supplies to U.S. Arctic communities. Maritime shipping in the U.S. Arctic involves challenges, given that the region lacks many of the typical elements of a maritime transportation system. See table 1 for examples of the types of maritime infrastructure gaps that CMTS and federal agencies have reported in the U.S. Arctic. Many federal agencies are involved with, and have a role in, U.S. Arctic maritime shipping and infrastructure (see table 2). Although these agencies’ missions are not specifically tied to the U.S. Arctic, they extend to the U.S. Arctic like any other geographic region of the country. Other state, local, and international organizations also play a role. For example, the state of Alaska’s Department of Environmental Conservation is involved with oil spill response. In addition, the North Slope Borough, a municipal government that encompasses an area of nearly 95,000 square miles along Alaska’s northern coast, has a search and rescue department that provides airborne emergency response. Alaska Native organizations represent communities that have inhabited the Arctic region for thousands of years and have cultures that are particularly sensitive to environmental changes, since they rely on hunting animals such as whales, seals, and walruses. To represent local concerns, the Arctic Waterways Safety Committee, which is comprised of subsistence hunters and others, was created in October 2014 to develop best practices for safe and efficient use of Arctic waterways. Alaska Native Corporations are private entities that manage land and assets on behalf of Alaska Natives. Lastly, international forums such as the Arctic Council and international organizations such as the IMO also have a role in establishing Arctic maritime policies and regulations. For nearly 50 years the U.S. government has articulated its interest in the Arctic through a series of strategies. For example, in 1971 a then- classified memo from National Security Council (NSC) under the Nixon Administration called for the sound and rational development of the Arctic, guided by the principles of minimizing adverse environmental effects, promoting international cooperation, and protecting security interests, including the preservation of the freedom of the seas. These same priorities, along with promoting scientific research, were underscored by the Reagan Administration in 1983. In January 2009, the George W. Bush Administration issued an Arctic Region Policy, which outlined priorities for maritime transportation in the Arctic including to facilitate safe, secure, and reliable navigation and protect maritime commerce and the environment. More recently, the Obama Administration issued a National Strategy for the Arctic Region (National Strategy) in May 2013, which identified three goals for the region: to advance U.S. security interests, pursue responsible stewardship, and strengthen international cooperation. Subsequent implementation plans for the National Strategy indicated maritime shipping and infrastructure fell under all three of these stated goals. For example, “preparing for increased activity in the maritime domain” fell under advancing U.S. security interests, “charting the Arctic region” fell under pursuing responsible stewardship, and “promoting waterways management” fell under strengthening international cooperation. As federal strategies related to the Arctic region have evolved over the years, so have the interagency groups to implement and guide these efforts. Interagency activity in the U.S. Arctic has historically been coordinated through the NSC, including the 1971 and 1983 strategies. In 1984, legislation established the U.S. Arctic Research Commission as well as the Interagency Arctic Research Policy Committee (IARPC). More recently, to enhance coordination of national efforts in the Arctic, particularly those related to the 2013 National Strategy, a 2015 Executive Order established the interagency Arctic Executive Steering Committee (AESC). AESC is chaired by the Director of the Office of Science and Technology Policy (OSTP), which is an office within the White House that leads interagency science and technology policy coordination efforts. AESC also includes NSC as a member, along with 20 other federal departments and entities. The 2016 National Strategy Implementation Framework assigned portions of the strategy’s areas of focus to interagency groups; specifically, NSC was assigned responsibility for advancing national security interests, OSTP for pursuing responsible stewardship, and the Department of State for strengthening international cooperation. The U.S. Committee on the Marine Transportation System (CMTS), which was required in 2010 to coordinate the establishment of domestic transportation policies in the Arctic to ensure safe and secure maritime shipping, has issued several relevant reports, including a 10-year projection of maritime activity in 2015, and a 10-year prioritization of infrastructure needs in the U.S. Arctic in 2016—both of which were directed by the 2014 National Strategy implementation plan. More recently, CMTS issued a 2018 report revisiting its 2016 near-term recommendations for prioritizing infrastructure needs in the U.S. Arctic and a 2019 update to its projections of Arctic maritime shipping activity from 2020 to 2030. These and other reports addressing maritime infrastructure in the U.S. Arctic are listed in appendix I. U.S. Coast Guard data indicate the number of vessels in the U.S. Arctic increased from 2009 through 2019 (see fig. 3). The types of vessels the U.S. Coast Guard tracks in the U.S. Arctic includes vessels conducting marine scientific research; tugs that provide communities with supplies; and adventurer vessels such as private yachts. U.S. Coast Guard data also include bulk cargo vessels from the Red Dog mine, one of the largest zinc mines in the world. The mine trucks its zinc ore to a facility on the Chukchi Sea, where it is stored for maritime transport during the shipping season. The U.S. Coast Guard District responsible for the U.S. Arctic counts more types of vessels in its area of interest—such as research, tug, and adventurer—than are typically counted for the purposes of tracking commercial shipping. Even at its peak, maritime shipping in the U.S. Arctic remained limited compared to global commercial shipping, although CMTS recently reported that the number of flag states, or countries where vessels are registered, has increased. Specifically, the 307 vessels in the U.S. Arctic in 2019 represented a small portion of the total number of shipping vessels operating globally. For comparison, according to the United Nations Conference on Trade and Development, in 2015 the world fleet of commercial shipping vessels was approximately 89,000. However, in its 2019 traffic projections report, CMTS analyzed U.S. Coast Guard data and other data sources and found that between January 2015 and December 2017, the number of flag states in the U.S. Arctic increased. CMTS noted this indicates a shift away from regionally focused operators toward a more diverse and international set of operators. CMTS found that the majority of vessels were flagged to the United States (about 41 percent) or Russia (about 24 percent) over this time period, with the remaining 35 percent from 35 other flag states, each with a considerably smaller percentage than the United States or Russia. Given that a single vessel can make multiple trips per shipping season, U.S. Coast Guard also measures maritime activity by the number of transits that vessels make per year through the Bering Strait, a key convergence point for trans-Arctic routes that connects the NWP and NSR to the Pacific Ocean. According to U.S. Coast Guard data, the number of transits through the Bering Strait has ranged from as few as 280 in 2009, to as many as 514 in 2015 (see fig. 4). There were far fewer transits through the Bering Strait than through some other convergence points for established major maritime transportation routes that have more developed maritime infrastructure. For example, the number of transits through the Panama Canal, which like the NWP connects the Atlantic and Pacific Oceans, was almost 14,000 in 2018 and the number of vessels that transited the Suez Canal, which like the NSR enables shipping between Asia and Europe, was over 18,000. Stakeholders told us that along with factors such as demand that shape shipping trends worldwide, factors unique to the Arctic also play a role, such as potential cost savings due to shorter routes; additional operating costs incurred by Arctic-capable ships; environmental hazards like unpredictable weather and sea ice; and a lack of maritime infrastructure typically found along shipping routes. The 20 stakeholders we interviewed representing the shipping industry, research institutions, and state, local, and Alaska Native groups among others, described the following factors that affect U.S. Arctic maritime shipping. As mentioned earlier, diminished sea ice has presented opportunities for maritime shipping of natural resources extracted from the Arctic, such as oil, gas, and minerals. However, such activities decreased domestically after Royal Dutch Shell, PLC (Shell) discontinued its offshore oil and gas exploration of the Burger prospect in Alaska’s Chukchi Sea in 2015. As shown in figure 4, the number of transits in the Bering Strait steadily declined from 514 in 2015 to 369 in 2018. Specifically, CMTS reported that Shell demobilized its drill ship, anchor handling vessels, and anti- pollution ships from the study area prior to the start of the 2016 shipping season. One stakeholder said there was a reduction in the number of seasonal transits after Shell suspended exploration activities, since Shell had previously accounted for more than a hundred transits through the Bering Strait. Other traffic related to domestic natural resource extraction stayed at consistent levels. Specifically, representatives from the Red Dog zinc mine reported that from 1999 to 2019 they consistently shipped between 21 and 26 cargo vessels per year, averaging 24 vessels per year over the 20-year period. Meanwhile, several stakeholders said international activities related to natural resource development, particularly in the Russian Arctic, have recently increased, and that Russia has been investing heavily in Arctic infrastructure. The U.S. Coast Guard attributed increased cargo traffic levels in 2016 to construction projects in the Russian Arctic, particularly a liquefied natural gas (LNG) facility on the Yamal peninsula (see fig. 3 above). In 2017, a Russian LNG tanker, the Christophe de Margerie, became the first ship to transit the NSR without being accompanied by an icebreaker. Demand for tourism cruises in the U.S. Arctic has increased slightly recently. A representative from an Arctic cruise industry association told us that the overall cruise industry worldwide grows 5 to 10 percent a year and that there is growing demand for expedition cruises to farther-flung areas like the Arctic. In both 2016 and 2017, the cruise ship Crystal Serenity transited the NWP with over a thousand passengers on board. Stakeholders noted that cruise ship voyages in the U.S. Arctic, such as the Crystal Serenity voyages, raised concerns for passenger safety given the lack of infrastructure, particularly for search and rescue. However, according to an Arctic cruise industry association representative, the number of smaller ships purpose-built for Arctic conditions is growing; the association estimates 25 to 30 such vessels are under construction. Domestic and foreign research vessels have also increased in number in the U.S. Arctic due to greater interest in the region’s changing environment. For example, according to National Science Foundation officials, their polar-capable vessel Sikuliaq entered service in 2016. Internationally, China has increased its activity in the Arctic since gaining observer status on the Arctic Council in 2013 and now operates two icebreaking research vessels. One stakeholder said that such investments by countries such as China may be the first step towards achieving longer-term economic goals for those countries. Trans-Arctic routes can reduce travel time between certain destinations compared to traditional routes and may therefore reduce fuel and labor costs. For example, the route from Shanghai, China, to Northwestern Europe via the NSR is 27 percent shorter than via the Suez Canal. The operators of the Russian LNG tanker that transited the NSR in 2017, the Christophe de Margerie, reported they completed the journey in 19 days, 30 percent faster than the Suez Canal. For reasons such as these, according to news reports, Russia has announced plans to develop the NSR and ship 80 million tons of goods through the route by 2024. Similarly, an official from a Canadian ship owner and operator told us that, depending on the vessel’s origin and destination, using the NWP can be 10-15 days faster than using the Panama Canal, resulting in a cost savings of $100,000 to $150,000. Although trans-Arctic routes have the potential for cost savings due to shorter distances, they require additional investments not necessary for traditional routes that may offset those savings. For example, representatives of one carrier said Arctic-capable ships cost three to four times more than ordinary ships because they require more steel and higher power output to withstand ice conditions. The additional steel also limits the amount of cargo the vessel can carry; representatives from another carrier noted every ton of steel used to construct the ship is a ton of cargo that the ship cannot carry in order to recoup expenses. The size of vessels that can safely operate in the region is also constrained by draft limitations that specify the maximum weight and size at which ships can navigate the shallow waters of the Arctic. By contrast, the trend among ocean carriers over the past decades, which have capitalized on advances in fuel-efficient engine technology, is toward constructing increasingly larger vessels to capture economies of scale. In addition, stakeholders told us that shippers operating in the Arctic must invest in special onboard equipment and prepare for contingencies due to the lack of maritime infrastructure usually found in traditional routes, such as deep-draft ports, harbors of refuge, reliable communications, and search and rescue infrastructure. Stakeholders noted Arctic voyages also require additional training for crew members on navigating in ice conditions. Shippers must determine whether the cost savings obtained from shorter trans-Arctic routes outweigh the additional operating expenses. For example, although Maersk, one of the largest shipping companies in the world, successfully completed a trial passage of a container ship through the NSR in September 2018, the company emphasized at the time that the transit was a “one-off trial designed to gain operational experience in a new area and to test vessel systems” and that it did not view the route as a commercially viable alternative to existing routes. In a press release, Maersk noted that the NSR was only feasible for around 3 months a year and required the use of more costly ice-classed vessels. Despite this, news reports in June 2019 indicate Maersk is exploring sending more goods through the NSR in cooperation with a Russian icebreaker company in response to demand for the transport of goods from Asia to West Russia. Although diminished sea ice has prolonged the shipping season and opened up shipping routes, environmental changes have also resulted in less predictable conditions, such as more volatile weather and sea ice. One stakeholder involved with Arctic research noted that the conditions that have led to open waters can also lead to harsher conditions such as strong low pressure systems, gale force winds, and storms. Such conditions pose challenges for shipping—one shipper representative said that it is difficult to load barges in shallow waters and that typically loading and unloading activities have to be suspended with swells above 3 feet. In addition, stakeholders told us variation in ice conditions from year to year makes planning Arctic voyages difficult to do with reasonable accuracy. For example, while warming trends might suggest that overall sea ice diminishes further each year, one carrier representative noted its vessel encountered severe ice conditions in June 2018. This representative noted that diminished overall ice coverage can lead to localized conditions with more mobile and older ice migrating into shipping lanes. The unpredictable and harsh weather and ice conditions, combined with the vast distances and lack of maritime infrastructure, pose safety risks. For example, according to stakeholders, the “tyranny of distance” in the Arctic stretches the limited search and rescue capabilities, resulting in slow incidence response. Furthermore, a lack of a designated harbor of refuge means vessels do not have a place to moor safely in case of emergency. As a result, a representative from the International Union of Marine Insurance noted that in the Arctic even a minor incident, such as a small engine failure, can result in substantial damages and even loss of life. Some stakeholders we interviewed expressed concerns about impacts of shipping on wildlife, including the species that Alaska Natives rely on for food. One stakeholder noted that awareness has grown in the past 10 years of the environmental impact of shipping. Such impacts include emissions containing sulphur oxide and black carbon from ships’ engines that could damage the fragile Arctic ecosystem. As a result of such environmental concerns, the IMO is currently considering a ban on heavy fuel oil in the Arctic. In addition, in 2019 several major carriers, including CMA CGM, Hapag-Lloyd, and Mediterranean Shipping Company, announced they would not pursue trans-Arctic shipping. Furthermore, in 2019 Nike and Ocean Conservancy launched the Arctic Corporate Shipping Pledge, a voluntary commitment by consumer goods and shipping logistics companies to not send ships through the Arctic. The pledge also supports precautionary Arctic shipping practices to enhance the environment and human safety, which may include a heavy fuel oil ban and an evaluation of low impact shipping corridors that protect important ecological and cultural areas. A representative of one carrier we spoke with said a heavy fuel oil ban in the Arctic could increase the cost of transporting cargo and, as a result, severely impact shipping in the region. While agencies have taken actions to address maritime infrastructure gaps, federal efforts lack (1) a government-wide assessment of risks posed by gaps in maritime infrastructure, (2) a current government-wide strategy for addressing maritime infrastructure that includes goals, performance measures, and appropriate responses to prioritized risks, and (3) an interagency mechanism and consistent leadership to guide agency actions related to maritime infrastructure. Without these elements, federal agencies may lack information on which to base decisions and prioritize actions, assurance that their investments are directed to prioritized risks, and the ability to demonstrate progress in addressing maritime infrastructure. Furthermore, agencies may miss opportunities to work together and leverage resources towards achieving broader outcomes. Agencies have taken some actions since 2013, when CMTS first reported on gaps in U.S. Arctic infrastructure. For example, U.S. Coast Guard reported that it has taken a flexible approach to addressing infrastructure gaps by establishing seasonal, forward operating bases in the U.S. Arctic as needed to provide search and rescue support in areas where major shipping activity is occurring. See table 3 for selected examples of agency actions. Although federal agencies have taken some steps to address gaps in U.S. Arctic infrastructure, those efforts are not based on a government- wide assessment of the economic, environmental, and safety risks posed by maritime infrastructure gaps to inform investment decisions in the U.S. Arctic. Rather, agency officials said that they currently base Arctic infrastructure decisions on their agency-specific missions, strategies, and collaborative efforts. Specifically, agency officials said that securing the resources to address U.S. Arctic infrastructure is challenging because such projects must compete with other established agency mission areas. For example, officials told us that infrastructure investments may not compete as well against other agency-established priorities in other parts of the country, in part, because the Arctic is an emerging region and because of the considerable costs of developing infrastructure in the harsh Arctic environment. Leading management practices we reviewed note the importance of assessing risks in order to select and prioritize countermeasures to prevent or mitigate risks. A 2016 Office of Management and Budget (OMB) circular emphasized the importance of risk assessment and called for agencies to use a structured and systematic approach to identify risks and assess the causes, sources, probability of the risk occurring, and potential outcomes, and then prioritize the results of the analysis. Such an approach can be used by decision makers to evaluate the effectiveness of, and to prioritize, countermeasures relative to the associated costs. Risk management is a widely endorsed strategy for helping policymakers make decisions about allocating finite resources and taking actions in conditions of uncertainty. Such a framework is especially applicable to the U.S. Arctic given the uncertain conditions in the region and safety and environmental risks described above. Without a risk assessment, agencies lack assurance that their investments are addressing the highest-priority risks. In particular, we found that agencies’ actions to address maritime infrastructure gaps were not fully consistent with the areas that the stakeholders we interviewed identified as the most critical (see fig. 5). For example, 11 of the 20 stakeholders we interviewed identified charting Arctic waters as the highest priority to address, and in May 2019 NOAA reported that it had acquired nearly 1,500 square nautical miles of hydrographic survey data in the Arctic over the prior 3 years. This is, however, less than 1 percent of the over 200,000 square nautical miles of waters NOAA has identified as significant to navigation in the U.S. Arctic. In addition, nine stakeholders identified addressing gaps in communications in the U.S. Arctic as a key priority. However, CMTS reports indicate no change in the status of communications capabilities in the U.S. Arctic between 2013 and 2018. CMTS has in the past noted the importance of conducting a risk assessment to inform Arctic decision-making. Specifically, CMTS’s 2013 report noted that greater access to the U.S. Arctic and increased activity presents additional risks for people, vessels, and the environment in the fragile region and that managing that risk requires an in-depth understanding of the issues and trade-offs associated with key decisions. Although CMTS reported that developing an assessment tool that provides a quantifiable level of risk and that accounts for the unique risk elements in the Arctic was a challenge for the nation, it proposed a model for determining risk that considered the likelihood of adverse events actually occurring, vulnerability to damage, and potential consequences. CMTS further stated that, given the rate at which other nations are progressing with Arctic shipping and development, the United States should decide the acceptable degree of risk for Arctic operations. Although CMTS has provided useful information on maritime infrastructure gaps to decision makers and described possible risks to the U.S. Arctic, it has not systematically assessed the risks posed by these gaps. For example, in 2016, CMTS made near-, mid-, and long-term recommendations for addressing maritime infrastructure needs, but noted this ordering was not intended to create a hierarchy of infrastructure needs from most to least important. CMTS officials told us that they have not systematically assessed risks posed by maritime infrastructure gaps in the U.S. Arctic because CMTS’s priorities are established by its member agencies, and that CMTS has not been directed to conduct such an assessment by its members. However, CMTS is required by statute to, among other things, coordinate the establishment of domestic transportation policies in the Arctic to ensure safe and secure maritime shipping and make recommendations with regard to federal policies that impact the marine transportation system. Furthermore, according to CMTS officials, there is nothing in CMTS’s authority that would prevent it from doing a risk assessment. Given its previous reports and work in the U.S. Arctic and its coordinating role with its member agencies, CMTS is well suited to conduct a government-wide assessment of the risks posed by gaps in maritime infrastructure in the U.S. Arctic. For example, CMTS published a traffic projections report in September 2019 that aimed to provide decision makers with a wide-ranging portrait of potential changes in vessel activity in the U.S. Arctic over the next decade. To inform its risk assessment, CMTS can draw on the expertise of its member agencies, such as U.S. Coast Guard and NOAA. For example, U.S. Coast Guard officials told us that they have responded to the unpredictable economic changes in the U.S Arctic—including fluctuations in the level and type of maritime activity in the region—by investing in icebreakers and seasonal forward operating bases, rather than developing permanent infrastructure. In addition, CMTS can also draw on numerous reports discussing maritime infrastructure in the U.S. Arctic that have been published since 2013, as detailed in appendix I. For example, in 2019 the University of Alaska’s Arctic Domain Awareness Center held a series of workshops on the factors that impact the ability of the U.S. Coast Guard and other stakeholders to conduct safe, secure, and effective operations in the Arctic environment. A government-wide risk assessment could better enable agencies to evaluate potential U.S. Arctic infrastructure expenditures and assess the extent to which these expenditures will mitigate identified risks. For example, a report on the U.S. Coast Guard’s Arctic capabilities suggested that a systematic analysis of needs and risks in the U.S. Arctic could help the agency generate momentum for closing Arctic capability gaps. By conducting such a risk assessment, agencies would have better information on which to base decisions for agency expenditures in the U.S. Arctic and prioritize appropriate actions in response to risks. We found that the federal interagency efforts to address U.S. Arctic maritime infrastructure lack a current strategy and consistent interagency leadership to guide agencies’ actions. Although several agencies have developed strategies to guide their own agencies’ efforts, these do not provide government-wide direction or establish current government-wide goals, objectives, and performance measures as leading management practices call for. Moreover, the federal agencies lack clarity on which interagency entity is responsible for guiding federal efforts and providing consistent leadership to ensure government-wide objectives are met. The federal government lacks a current government-wide strategy for addressing U.S. Arctic maritime infrastructure gaps that includes key characteristics, such as goals, objectives, and performance measures, and appropriate responses to risks. Agency officials and stakeholders said the 2013 National Strategy is outdated because conditions in the U.S. Arctic have changed since 2013. In particular, agency officials said national security is a growing concern in the Arctic. OSTP officials agreed that conditions had changed in the Arctic, but could not state whether the 2013 National Strategy was still current. Our past work on interagency collaboration found that written agreements documenting how participating agencies collaborate, such as strategies, are most effective when they are regularly updated and monitored. The changing conditions in the Arctic described above make a current government-wide strategy for maritime infrastructure in the U.S. Arctic particularly important. In our past work, we have reported that complex interagency efforts— such as those to address maritime infrastructure in the U.S. Arctic—can benefit from developing a national strategy. Our prior work has identified key characteristics of national strategies, which we refer to in this report as a government-wide strategy, including: (1) problem definition and risk assessment which addresses the threats the strategy is directed towards; and (2) goals, objectives, and performance measures to gauge and monitor results. Furthermore, our prior enterprise risk management work has noted that risk assessment should include a ranking of risks based on priorities in relation to strategic objectives, and that senior leaders should determine if a risk requires treatment or not based on risk tolerance or appetite. Leaders then review the prioritized list of risks and select the most appropriate response to address the risk. These key characteristics help managers determine the extent of investment needed and facilitate effective targeting of federal resources; this is especially important when multiple agencies are involved, as is the case with maritime infrastructure in the U.S. Arctic. Although several federal agencies have recently updated their Arctic strategies, these agency-specific Arctic strategies are not linked to a current government-wide strategy for the Arctic region and are not specifically focused on addressing Arctic maritime infrastructure gaps. Specifically: U.S. Coast Guard. In April 2019, U.S. Coast Guard published its Arctic Strategic Outlook, which supersedes its 2013 Arctic Strategy. The updated strategy established three lines of effort: (1) enhance capability to operate effectively in a dynamic Arctic domain, (2) strengthen the rules-based order, and (3) innovate and adapt to promote resilience and prosperity. We recommended in 2016 that the U.S. Coast Guard develop measures for assessing how its actions have helped to mitigate Arctic capability gaps and design and implement a process to systematically assess its progress. However, as of February 2020, the U.S. Coast Guard has not implemented these recommendations. U.S. Coast Guard officials state that they are currently developing an implementation plan and Strategic Metrics Framework for the Arctic Strategic Outlook. U.S. Navy. In January 2019, the U.S. Navy published its Arctic strategy, which updated its previous strategy from 2014. The updated strategy included the following strategic objectives: defend U.S. sovereignty and the homeland from attack; ensure the Arctic remains a stable, conflict-free region; preserve freedom of the seas; and promote partnerships to achieve the above objectives. Department of Defense. In June 2019, Department of Defense updated its 2016 Arctic strategy which included the following as part of its approach: (1) building Arctic awareness, (2) enhancing Arctic operations, and (3) strengthening the rules-based order in the Arctic. NOAA. NOAA officials originally told us that they were working to complete an update to NOAA’s strategic plan for the Arctic in 2019. However, as of February 2020, officials told us that this update is currently on hold pending the completion of a new government-wide National Strategy. As mentioned previously, OSTP staff said they could not state whether the 2013 National Strategy was still current, and OSTP provided no additional information as to whether a new strategy was in development. NOAA officials told us that the agency’s current three priorities in the Arctic are (1) weather and water (including weather and water research, observations, and Arctic contributions to global weather monitoring); (2) blue economy (including ocean mapping, seafood competitiveness, tourism, and coastal resilience); and (3) innovative partnerships in Alaska and the Arctic. CMTS has taken some steps to monitor agencies’ progress in addressing maritime infrastructure, but the current lack of performance measures makes it difficult to monitor agencies’ progress over time. We reported in 2014 that CMTS was developing a process to monitor such progress and noted that such monitoring would help agencies develop a shared understanding of current priorities and actions needed. However, while CMTS did issue reports that described the status of maritime infrastructure in the U.S. Arctic in 2016 and 2018, the reports did not include goals or performance measures to assess agencies’ progress. According to officials, CMTS did not develop or include performance measures to monitor agencies’ progress because it does not have the authority to designate agencies’ priorities, and that agencies are best situated to identify priorities in pursuit of their individual missions. Priorities in the U.S. Arctic are currently based on each agency’s mission, which makes it difficult to take a government-wide approach to responding to risks. To improve unity of effort, the U.S. Coast Guard has expressed support for a national approach to Arctic planning in both its 2013 and 2019 Arctic strategies. Without a current government-wide strategy that includes goals and objectives, agencies lack assurance that their investments are directed to prioritized risks. Furthermore, without performance measures, agencies are not able to demonstrate, and decision makers are unable to monitor, the extent to which agency actions have addressed maritime infrastructure gaps. We have previously reported that federal agencies use a variety of mechanisms, including interagency groups, to implement interagency collaborative efforts and that such mechanisms benefit from key features such as sustained leadership and inclusion of all relevant participants, such as stakeholders. We also reported that leadership should be sustained over time to provide continuity for long-term efforts and that having top-level commitment from the President, Congress, or other high- level officials can strengthen the effectiveness of interagency collaborative groups. We also found that lack of continuity is a frequent issue with interagency mechanisms that are tied to the Executive Office of the President, particularly when administrations change, and that transitions and inconsistent leadership can weaken a collaborative mechanism’s effectiveness. In addition, our prior work has noted the importance of ensuring that all relevant participants are included in the collaborative effort, such as participants with the appropriate knowledge, skills, and abilities to contribute. There are many interagency groups involved in the U.S. Arctic, including: AESC was established by Executive Order in 2015 to shape national priorities and set strategic direction in the Arctic. NSC Arctic Policy Coordinating Committee (PCC) is the current interagency forum for executive-level Arctic collaboration, according to agency officials. CMTS is the main forum for collaboration regarding maritime infrastructure according to agency officials. These interagency groups vary in the extent to which they meet the key features noted above. Specifically: Sustained leadership: Both the NSC, which, as mentioned previously, has traditionally played a role in Arctic collaboration dating back to 1971, and the AESC, which was chaired by the OSTP within the White House, would have top-level commitment given their proximity to the White House. However, according to agency officials, the AESC has not met in the past 2 years and is now dormant. OSTP staff told us that they are not aware of any current AESC activities. Meanwhile, although CMTS has been active in the area of U.S. Arctic maritime infrastructure for the past decade, CMTS officials said CMTS does not sit within the Executive Office of the President. As a result, CMTS officials note their role is to facilitate an interagency partnership, share information among member agencies, and provide information to decision-makers to support agencies’ efforts. CMTS’s statutory authority addresses, among other things, the coordination of federal policies that impact the maritime transportation system, including in the U.S. Arctic, rather than the development and execution of government-wide policies. Inclusion of relevant stakeholders: The AESC, when it was active, included a wide range of over 20 federal departments and entities, including those less associated with maritime infrastructure such as the Department of Agriculture. For the NSC Arctic PCC, we were unable to verify the participants, so it is unclear whether relevant stakeholders are involved. However, agency officials noted the PCC’s focus is on national security rather than on maritime infrastructure, which may indicate not all maritime infrastructure stakeholders are included. Lastly, CMTS includes stakeholders involved directly with maritime transportation. For example, officials from the U.S. Army Corps of Engineers noted that they actively participate in CMTS, including its Arctic Integrated Action Team, but do not participate in other interagency groups, where they are often represented by higher- level Department of Defense officials. The Executive Office of the President has not designated an interagency group as responsible for developing or executing the administration’s strategy for maritime infrastructure in the U.S. Arctic. We have previously reported that interagency efforts can benefit from the leadership of a single entity to provide assurance that federal programs are well coordinated and based upon a coherent strategy. Agency officials said priorities in the U.S. Arctic have shifted to national security under the current administration, which may have led executive-level interagency collaboration efforts to move from AESC to the NSC Arctic PCC. However, it is unclear whether the NSC Arctic PCC includes the relevant stakeholders. Moreover, the shift in Arctic priorities to security issues does not diminish the importance of Arctic maritime infrastructure. As indicated in the 2013 National Strategy, maritime shipping and infrastructure are a key component of overarching goals in the region like advancing U.S. security interests, pursuing responsible stewardship, and strengthening international cooperation. Without an interagency mechanism with sustained leadership and inclusion of relevant stakeholders to direct federal efforts related to U.S. Arctic maritime infrastructure, agencies may miss opportunities to leverage resources toward achieving a broader outcome. For example, as noted earlier, stakeholders we spoke to identified communications as a key infrastructure gap. According to U.S. Coast Guard officials, communications is a whole-of-government effort, requiring partnerships across agencies including the Department of Defense. Without an interagency collaboration mechanism designated to lead these efforts, it is unclear who has responsibility for such whole-of-government efforts to address maritime infrastructure in the U.S. Arctic. The U.S. Arctic, including the Bering Strait, is an essential part of the increasingly navigable Arctic and a key convergence point for maritime transportation routes connecting the Pacific and Atlantic oceans. The risks inherent to Arctic shipping—including vast distances, extreme ice conditions, and unpredictable weather—are exacerbated by gaps in maritime infrastructure in the U.S. Arctic. While agencies have taken some actions to address these gaps, without a government-wide assessment of risks posed by maritime infrastructure gaps in the U.S. Arctic and a current strategy to address those risks, agencies lack assurance that their actions are effectively targeting priority areas. Without a strategy that includes goals, objectives, and performance measures, agencies cannot demonstrate the results of their efforts, and decision makers cannot gauge the extent of progress in addressing maritime infrastructure gaps. In addition, without a designated interagency group to provide sustained leadership, agencies lack the ability to leverage resources to address maritime infrastructure and achieve government-wide priorities in the complex and changing U.S. Arctic. The U.S. Committee on the Marine Transportation System should: Complete a government-wide assessment of the economic, environmental, and safety risks posed by gaps in maritime infrastructure in the U.S. Arctic to inform investment priorities and decisions. (Recommendation 1) The appropriate entities within the Executive Office of the President, including the Office of Science and Technology Policy should: Develop and publish a strategy for addressing U.S. Arctic maritime infrastructure that identifies goals and objectives, performance measures to monitor agencies’ progress over time, and the appropriate responses to address risks. (Recommendation 2) Designate the interagency group responsible for leading and coordinating federal efforts to address maritime infrastructure in the U.S. Arctic that includes all relevant stakeholders. (Recommendation 3) We provided a draft of this report to the Executive Office of the President’s Office of Science and Technology Policy (OSTP); the U.S. Committee on the Marine Transportation System (CMTS); and the Departments of Homeland Security, Commerce, Defense, Interior, State, and Transportation for comment. With the exception of the Department of Defense, all of these entities provided technical comments, which we incorporated as appropriate. Only CMTS provided written comments, which were transmitted via letter from the Department of Transportation, and are reprinted in appendix II. In its technical comments, the Department of Homeland Security’s U.S. Coast Guard provided Arctic traffic data for the 2019 shipping season. As stated in our report, we originally selected the decade of 2009 through 2018 for our analysis when designing our review, as 2018 was the most recent year for which data were available at that time. In response to the U.S. Coast Guard’s comments submitted in April 2020, we revised our report to include data from the 2019 shipping season on (1) the number of vessels in the U.S. Coast Guard District 17 Arctic area of interest and (2) the number of transits in the Bering Strait, to ensure the report contained the most current information available. In its written comments, CMTS partially concurred with our recommendation that CMTS complete a government-wide assessment of the economic, environmental, and safety risks posed by gaps in maritime infrastructure in the U.S. Arctic to inform investment priorities and decisions. However, CMTS also noted several areas of disagreement with our conclusions, which we address here: First, CMTS noted that GAO’s draft report contained dated information and that the 2019 data contradicts GAO statements that suggest a decrease in vessel activity since 2015. CMTS noted that the 2019 data shows that vessel traffic has increased steadily over the last decade, and that although growth slowed between 2015 and 2017, “it did not stall.” However, we dispute this characterization. The 2019 shipping data included in this report emphasizes the finding from our draft report that maritime shipping activity generally increased over the time period of our review. However, this trend does not reflect a steady increase throughout the entire timeframe or “slowed growth” between 2015 and 2017 as CMTS indicates. Specifically, the data show year-to-year decreases in the number of vessels from 2015 to 2017 in the U.S. Coast Guard District 17 Arctic area of interest (see fig. 3) and in the number of transits in the Bering Strait from 2015 to 2018 (see fig. 4). CMTS’s own 2019 report indicated that the number of vessels had decreased from a peak in 2015, after Shell’s decision in 2015 to not pursue further exploratory drilling efforts. As such, we stand by our description of the overall growth in maritime activity in the U.S. Arctic since 2009, as well as the pattern of declining traffic within that period from 2015 through 2018. Second, CMTS also noted in its written comments that our use of data from 2009 to 2018 in the draft report do not lead to the conclusions and recommendation to assess infrastructure risks and prioritize future investment in the Arctic. We dispute this characterization. Our decision to include the 2019 data further emphasizes the finding in our draft report of a general increase in maritime activity in the U.S. Arctic and the need for an assessment of risks posed by gaps in maritime infrastructure. As we note in the report, CMTS has reported that the U.S. Arctic does not have the typical elements of a maritime infrastructure system such as a deep- draft port or robust communications infrastructure. These infrastructure gaps exacerbate the inherent challenges of maritime activity in the Arctic—vast distances, dangerous weather, and extreme ice conditions— that can pose safety risks to mariners and environmental risks to the fragile Arctic ecosystem. While agencies have taken some steps to address infrastructure gaps, without a risk assessment, agencies lack assurance that their investments are addressing the highest-priority risks. As such, we stand by our conclusion and recommendation that increasing maritime traffic poses risks, and a government-wide assessment of those risks would inform federal decisions on investments to appropriately address risks. Third, CMTS disagreed with GAO’s statement that a government-wide risk assessment could better enable agencies to evaluate potential U.S. Arctic infrastructure expenditures. Although CMTS agreed that understanding infrastructure gaps is critical to improving the Arctic marine transportation system, CMTS contends that such risk assessments are the responsibility of each agency as directed by the Office of Management and Budget (OMB). As we note in the report, many agencies have a role in U.S. Arctic maritime shipping and infrastructure and although agencies and others have conducted many reviews of maritime infrastructure in the U.S. Arctic (see appendix I), agency-by- agency assessments do not reflect or analyze risks from a government- wide perspective. CMTS itself has previously noted the importance of evaluating risks on a government-wide basis. Specifically, in 2013 CMTS noted that increased activity in the U.S. Arctic presents additional risks for the people, vessels, and the environment and that managing that risk requires an in-depth understanding of the issues and trade-offs associated with key decisions, such as how to prioritize investments. As our report states, CMTS stated that developing a tool to assess the unique risk elements in the Arctic was a challenge for the nation, and it proposed a model for determining risk that considered the likelihood of adverse events actually occurring, vulnerability to damage, and potential consequences. This model is similar to the 2016 OMB circular, which called for agencies to, among other things, assess the causes, sources, probability of the risk occurring, and potential outcomes. As stated in our report, given its previous work in the U.S. Arctic and its coordinating role with its member agencies, CMTS is well suited to conduct a government-wide assessment of the risks posed by gaps in maritime infrastructure in the U.S. Arctic. As such, we stand by our recommendation. Based on these items, CMTS did not agree to perform and lead a government-wide risk assessment. Instead, as an “alternate action” to address GAO’s recommendation, CMTS noted it plans to update a table of information published in its past reports on infrastructure gaps in the U.S. Arctic and provide an inventory of existing risk assessments and their criteria, which agencies can then use to improve their own assessments to inform decisions. In our view, the proposed action described by CMTS would not provide the same level of information proposed by CMTS itself in 2013 and by OMB’s 2016 circular, which calls for, among other things, assessing the causes, sources, probability of the risk occurring, and potential outcomes. As stated in our report, CMTS is uniquely positioned as a federal interagency coordinating committee focused on the maritime transportation system to draw on the expertise of its member agencies, such as U.S. Coast Guard and the National Oceanic and Atmospheric Administration, to complete this risk assessment. Moreover, CMTS is required by statue to coordinate the establishment of domestic transportation policies in the Arctic to ensure safe and secure maritime shipping and make recommendations with regard to federal policies that impact the marine transportation system. Furthermore, according to CMTS officials, there is nothing in CMTS’s authority that would prevent it from doing a risk assessment. As such, we stand by our recommendation as written and do not believe CMTS’s alternate action is sufficient to address the recommendation. In comments provided via email, OSTP neither agreed nor disagreed with the report’s recommendations. OSTP acknowledged the Arctic is of critical national importance and noted interagency coordination can be implemented through the entities of the National Science and Technology Council, which is located within OSTP. OSTP noted the need for, and role of additional federal coordination, such as the Arctic Executive Steering Committee, is under consideration by OSTP. We continue to believe that the appropriate entities within the Executive Office of the President, including OSTP, should designate the interagency group responsible for leading and coordinating federal efforts to address maritime infrastructure in the U.S. Arctic that includes all relevant stakeholders. As we note in our report, without an interagency collaboration mechanism designated to lead these efforts, it is unclear who has responsibility for whole-of- government efforts to address U.S. Arctic maritime infrastructure. In addition, we stand by our other recommendation to OSTP and other entities within the Executive Office of the President to develop and publish a strategy for addressing U.S. Arctic maritime infrastructure that identifies goals and objectives, performance measures to monitor agencies’ progress over time, and the appropriate responses to address risks. As we note in the report, without such a strategy agencies lack assurance that their actions are effectively targeting priority areas and decision makers cannot gauge the extent of progress in addressing maritime infrastructure gaps. We are sending copies of this report to the appropriate congressional committees; the Executive Office of the President; the U.S. Committee on the Marine Transportation System; the Secretaries of Homeland Security, Commerce, Defense, Interior, State, and Transportation; and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. National Ocean Council. National Ocean Policy Implementation Plan. Washington, D.C.: April 2013. White House. National Strategy for the Arctic Region. Washington, D.C.: May 10, 2013. U.S. Coast Guard. United States Coast Guard Arctic Strategy. Washington, D.C.: May 2013. U.S. Committee on the Marine Transportation System. U.S. Arctic Marine Transportation System: Overview and Priorities for Action. Washington, D.C.: 2013. The White House. Implementation Plan for the National Strategy for the Arctic Region. Washington, D.C.: January 30, 2014. U.S. Navy. The United States Navy Arctic Roadmap for 2014 to 2030. Washington, D.C.: February 2014. GAO. Maritime Infrastructure: Key Issues Related to Commercial Activity in the U.S. Arctic over the Next Decade. GAO-14-299. Washington, D.C.: March 19, 2014. U.S. Department of Commerce. National Oceanic and Atmospheric Administration. NOAA’s Arctic Action Plan: Supporting the National Strategy for the Arctic Region. Silver Spring, M.D: April 2014. GAO. Arctic Issues: Better Direction and Management of Voluntary Recommendations Could Enhance U.S. Arctic Council Participation. GAO-14-435. Washington, D.C.: May 16, 2014. Brigham, L. W. Alaska and the New Maritime Arctic Executive Summary: Executive Summary of a Project Report to the State of Alaska Department of Commerce, Community and Economic Development. Fairbanks, A.K.: February 1, 2015. The International Council on Clean Transportation. A 10-Year Projection of Maritime Activity in the U.S. Arctic Region. Washington, D.C.: January 2015. This report was contracted and coordinated under the U.S. Committee on the Marine Transportation System. Executive Order No. 13689. Enhancing Coordination of National Efforts in the Arctic. 80 Fed. Reg. 4191. January 26, 2015. The White House. Arctic Executive Steering Committee. National Strategy for the Arctic Region Implementation Report. Washington, D.C.: January 30, 2015. Alaska Arctic Policy Commission. Final Report of the Alaska Arctic Policy Commission. Anchorage and Bethel, A.K.: January 30, 2015. U.S. Army Corps of Engineers. Alaska District. Alaska Deep-Draft Arctic Port System Study: Draft Integrated Feasibility Report, Draft Environmental Assessment (EA), and Draft Finding of No Significant Impact (FONSI). Nome, AK: February 2015. GAO. Arctic Planning: DOD Expects to Play a Supporting Role to Other Federal Agencies and Has Efforts Under Way to Address Capability Needs and Update Plans. GAO-15-566. Washington, D.C.: June 19, 2015. Arctic. Status on Implementation of the AMSA 2009 Report Recommendations. Tromsø, Norway: April 2015. World Economic Forum. Global Agenda Council on the Arctic. Arctic Investment Protocol: Guidelines for Responsible Investment in the Arctic. Geneva, Switzerland: November 2015. U.S. Coast Guard. Arctic Strategy Implementation Plan. Washington, D.C.: December 2015. Arctic Economic Council, Telecommunications Infrastructure Working Group. Arctic Broadband: Recommendations for an Interconnected Arctic. Tromsø, Norway: Winter 2016. Copenhagen Business School and Arctic Institute. Arctic Shipping: Commercial Opportunities and Challenges. CBS Maritime, January 2016. The White House. Arctic Executive Steering Committee. 2015 Year in Review: Progress Report on the Implementation of the National Strategy for the Arctic Region. Washington, D.C.: March 2016. The White House. Arctic Executive Steering Committee. 2015 Year in Review: Progress Report on the Implementation of the National Strategy for the Arctic Region; Appendix A, Implementation Framework for the National Strategy for the Arctic Region. Washington, D.C.: March 2016. U.S. Committee on the Marine Transportation System. Arctic Marine Transportation Integrated Action Team. A Ten-Year Prioritization of Infrastructure Needs in the U.S. Arctic. Washington, D.C.: April 15, 2016. GAO. Coast Guard: Arctic Strategy is Underway but Agency Could Better Assess How Its Actions Mitigate Known Arctic Capability Gaps. GAO-16-453. Washington, D.C.: June 15, 2016. Department of Defense. Report To Congress On Strategy To Protect United States National Security Interests In The Arctic Region. Washington, D.C.: December 2016. RAND Corporation. Maintaining Arctic Cooperation with Russia: Planning for Regional Change in the Far North. RR-1731-RC. Santa Monica, CA: 2017. U.S. Committee on the Marine Transportation System. Recommendations and Criteria for Using Federal Public-Private Partnerships to Support Critical U.S. Arctic Maritime Infrastructure. Washington, D.C.: January 2017. Arctic Council, Emergency Prevention, Preparedness and Response. Circumpolar Oil Spill Response Viability Analysis: Technical Report. March 7, 2017. Council of Foreign Relations. Arctic Imperatives: Reinforcing U.S. Strategy of America’s Fourth Coast, New York, N.Y.: March 2017. Ocean Conservancy. Navigating the North: An Assessment of the Environmental Risks of Arctic Vessel Traffic. Anchorage, A.K.: June 28, 2017. Center for Strategic and International Studies, Maritime Futures: The Arctic and the Bering Strait Region. Washington, D.C.: November 2017. RAND Corporation. Identifying Potential Gaps in US Coast Guard Arctic Capabilities.RR-2310-DHS. Santa Monica, CA: 2018. U.S. Committee on the Marine Transportation System. Revisiting Near- Term Recommendations to Prioritize Infrastructure Needs in the U.S. Arctic. Washington, D.C.: 2018. Department of Defense. Report to Congress on Assessment of Requirement for a Strategic Arctic Port. Washington, D.C.: January 2018. Department of Homeland Security, Office of the Chief Financial Officer. Arctic Search and Rescue: Fiscal Year 2017 Report to Congress. Washington, D.C.: March 13, 2018. International Union of Marine Insurance. IUMI Position Paper: Arctic Sailings. Hamburg, Germany: August 2018. U.S. Coast Guard Acquisition Directorate, Research & Development Center. Alaska AIS Transmit Prototype Test, Evaluation, and Transition Summary Report for the Near Shore Arctic Navigational Safety Information System (ANSIS). New London, C.T.: October 2018. GAO. Arctic Planning: Navy Report to Congress Aligns with Current Assessments of Arctic Threat Levels and Capabilities Required to Execute DOD’s Strategy. GAO-19-42. Washington, D.C.: November 8, 2018. Alaska Federation of Natives. Indigenous Engagement with Their Countries’ Military and Civilian Services/ Government on Maritime Arctic Issues. Anchorage, A.K.: December 2018. RAND Europe. The Future of Arctic Cooperation in a Changing Strategic Environment, PE-268-RC. Cambridge, United Kingdom: 2018. U.S. Navy. Strategic Outlook for the Arctic. January 2019. U.S. Coast Guard. Arctic Strategic Outlook. April 2019. Department of Defense. Office of the Secretary of Defense. Annual Report to Congress on Military and Security Developments Involving the People’s Republic of China 2019. Washington, D.C.: May 2, 2019. Pompeo, Michael, R. Secretary of State. U. S. Department of State. Looking North: Sharpening America’s Arctic Focus. Remarks. Rovaniemi, Finland, May 6, 2019. Pompeo, Michael R. Secretary of State. U.S. Department of State. Remarks. Arctic Council Ministerial Meeting. Rovaniemi, Finland, May 7, 2019. Department of Defense. Office of the Under Secretary of Defense for Policy. Report to Congress on Department of Defense Arctic Strategy. Washington, D.C.: June 2019. U.S. Committee on the Marine Transportation System. A Ten-Year Projection of Maritime Activity in the U.S. Arctic Region. 2020-2030: Washington, D.C.: September 2019. Congressional Research Service. Changes in the Arctic: Background and Issues for Congress. Washington, D.C.: November 27, 2019. U.S. Army Corps of Engineers, Alaska District. Port of Nome Modification Feasibility Study: Draft Integrated Feasibility Report and Supplemental Environmental Assessment. Nome, A.K.: December 2019. Congressional Research Service. Changes in the Arctic: Background and Issues for Congress. Washington, D.C.: January 23 2020. In addition to the contact named above, Susan Fleming, Director; Matt Barranca, Assistant Director; Emily Larson, Analyst in Charge; Chuck Bausell; Geoff Hamilton; Georgeann Higgins; Ned Malone; John Mingus; Jan Montgomery; Kaleb Mount; Fatima Sharif; Curtia Taylor; Sarah Veale; and Laurel Voloder made key contributions to this report.", "summary": "Arctic sea ice has diminished, lengthening the navigation season and increasing opportunities for maritime shipping. However, the U.S. Arctic lacks maritime infrastructure—such as a deep-draft port and comprehensive nautical charting—to support increased traffic. The lack of infrastructure exacerbates risks inherent to shipping in the Arctic such as vast distances and dangerous weather. This report examines (1) how U.S. Arctic shipping trends have changed since 2009 and factors that have shaped shipping in the region, and (2) the extent to which U.S. agencies' efforts to address Arctic maritime infrastructure gaps have aligned with leading management practices. GAO collected U.S. Coast Guard traffic data from 2009 through 2019 and interviewed 20 stakeholders selected to represent a range of views. GAO also analyzed Arctic strategies, interviewed selected agencies involved with maritime infrastructure, and compared efforts to leading management practices. Maritime shipping activity, as indicated by the number of vessels in the U.S. Arctic, generally increased from 2009 through 2019. Domestic maritime activity declined after the discontinuation of offshore oil and gas exploration activities in Alaska's Chukchi Sea in 2015. However, since 2015, international activities related to natural gas development, particularly in the Russian Arctic, have increased, according to stakeholders. Factors affecting decisions of ship operators about whether to operate in the U.S. Arctic include increased operating costs of Arctic-capable ships, environmental changes that have caused more volatile weather and ice conditions, and concerns over environmental impacts. Agencies have taken some steps to address Arctic maritime infrastructure gaps identified by federal agencies, such as a lack of nautical charting, but federal efforts lack a current strategy and interagency leadership. Examples of agency actions include the U.S. Coast Guard developing recommended shipping routes and the National Oceanic and Atmospheric Administration continuing to chart Arctic waters. To guide federal efforts, the White House developed a National Strategy for the Arctic Region in 2013 and established an interagency Arctic Executive Steering Committee (AESC) in 2015. However, agency officials and stakeholders noted the strategy is now outdated due to changing conditions in the Arctic. As a result, federal efforts lack a current government-wide strategy that aligns with key management practices such as identifying goals, objectives, and establishing performance measures. Moreover, U.S. Arctic interagency groups do not reflect leading collaboration practices, such as sustained leadership and inclusion of all relevant stakeholders, and the White House has not designated which entity is to lead U.S. Arctic maritime infrastructure efforts. For example, the AESC is now dormant according to agency officials and staff at the White House Office of Science and Technology Policy (OSTP), which chairs the AESC. Without a current strategy and a designated interagency entity with these collaboration practices in place, agencies may miss opportunities to leverage resources and target infrastructure improvements in areas that would best mitigate risks. GAO is making three recommendations, including that OSTP and other appropriate entities within the Executive Office of the President: develop and publish a strategy to address gaps and designate the interagency mechanism responsible for leading federal efforts. OSTP neither agreed nor disagreed but noted it is considering the need for and role of additional federal coordination. GAO stands by its recommendations.", "document_type": "gao"}
{"report": "The DRC is a vast, mineral-rich nation with an estimated population of more than 85 million people and an area that is roughly one-quarter the size of the United States, according to the UN. Since gaining its independence from Belgium in 1960, the DRC has undergone political upheaval and armed conflict. From 1998 to 2003, the DRC and eight other African countries were involved in what has become known as “Africa’s World War,” which resulted in a death toll of an estimated 5 million people in the DRC, according to State. During that period, in 1999, the UN deployed a peacekeeping mission to the DRC, and since then the United States and the international community have sought to improve security in the DRC. However, eastern DRC continues to be plagued by violence—including numerous cases of sexual violence reported by the UN—often perpetrated against civilians by nonstate armed groups and some members of the Congolese national military. More recently, presidential elections were originally scheduled for 2016, when the president’s final term in office expired, but the government delayed elections until December 2018. During this time, the UN reported an increase in human rights violations. In 2018 and 2019, the UN reported that serious violations of human rights remain widespread in the DRC, including continued acts of sexual violence by government security forces as well as nonstate armed groups. In addition, the UN noted that criminal networks and armed groups, including members of the Congolese national military and police, continued to derive illegal revenues from smuggling and illicit taxation of minerals from eastern Congolese mines. Various industries, particularly manufacturing industries, use the four conflict minerals—tin, tungsten, tantalum, and gold—in a wide variety of products. For example, tin is used to solder metal pieces and is also found in food packaging, steel coatings on automobile parts, and some plastics. Tungsten is used in automobile manufacturing, drill bits, and cutting tools, and other industrial manufacturing tools and is the primary component of filaments in light bulbs. Most tantalum is used to manufacture capacitors that enable energy storage in electronic products, such as cell phones and computers, or to produce alloy additives used in turbines in jet engines. Gold is used as reserves and in jewelry and is also used by the electronics industry, including, for example, in cell phones and laptops. In August 2012, SEC adopted its conflict minerals disclosure rule in response to Section 1502(b) of the Dodd-Frank Act. In the summary section of the adopting release for the rule, SEC noted that to accomplish the goal of helping to end the human rights abuses in the DRC caused by the conflict, Congress chose to use the Dodd-Frank Act’s disclosure requirements to bring greater public awareness of the sources of companies’ conflict minerals and to promote the exercise of due diligence on conflict mineral supply chains. The map in figure 1 shows the countries covered by the SEC disclosure rule, including the DRC and its 26 provinces. The SEC disclosure rule addresses the four conflict minerals named in the Dodd-Frank Act originating from the covered countries. The rule outlines a process for companies to follow, as applicable, to comply with the rule. (See app. II.) The process broadly requires a company to 1. determine whether it manufactures, or contracts to be manufactured, products with “necessary” conflict minerals; 2. conduct a reasonable country-of-origin inquiry concerning the origin of those conflict minerals; and 3. exercise due diligence, if appropriate, to determine the source and chain of custody of those conflict minerals, adhering to a nationally or internationally recognized due diligence framework, if such a framework is available for these necessary conflict minerals. If companies choose to disclose that their products are “DRC conflict free” in a conflict minerals report, the SEC disclosure rule requires companies to obtain an independent private-sector audit. Following an appellate court decision that a portion of the disclosure required by the SEC disclosure rule violated the First Amendment, SEC staff issued guidance on April 29, 2014, indicating that, pending further action by the SEC or a court, companies required to file a conflict minerals report would not have to identify their products as “DRC conflict undeterminable,” “not found to be ‘DRC conflict free,’” or “DRC conflict free.” In April 2017, following the entry of the final judgment in the case, the SEC’s Division of Corporation Finance issued revised guidance, indicating that, in light of the uncertainty regarding how the commission would resolve those issues and related issues raised by commenters, the Division of Corporation Finance had determined that it would not recommend enforcement action to the commission if companies did not report on specified due diligence disclosure requirements. However, the SEC staff told us that the guidance is not binding on the commission and that the commission could still initiate enforcement action if companies did not report on their due diligence in accordance with the rule. According to SEC staff, the 2017 guidance, while temporary, is still in effect, pending review of the rule by the commission. As of June 2019, the rule was on the SEC’s long-term regulatory agenda, which means— according to SEC staff—that any action would likely not take place until after March 2020. In 2018, 1,117 companies filed conflict minerals disclosures—slightly fewer than the number of companies that filed in 2017 and 2016 (1,165 and 1,230, respectively). Our analysis of a generalizable sample of the 1,117 filings found that an estimated 85 percent of the companies filed as domestic, while the remaining 15 percent filed as foreign. This domestic- to-foreign ratio is similar to the ratio in 2017 and 2016. Overall, when reporting on the conflict minerals used in their products, an estimated 62 percent reported using tantalum; 63 percent, tungsten; and 66 percent, gold—percentages similar to those reported in 2017 and 2016. An estimated 76 percent reported using tin, which was similar to the 69 percent reported in 2017 and significantly higher than the 61 percent in 2016. An estimated 24 percent did not specify the minerals they used. Our analysis of our generalizable sample found that, as in 2017 and 2016, almost all companies that filed conflict minerals disclosures indicated that they had conducted country-of-origin inquiries. Specifically, an estimated 100 percent of companies that filed reported that they had conducted such an inquiry, similar to the percentages that reported doing so in the prior 2 years. As a result of the inquiries they conducted, an estimated 56 percent of companies that filed reported whether the conflict minerals in their products came from covered countries—similar to the estimated 53 percent in 2017 and 49 percent in 2016. The percentage of companies able to make such a determination significantly increased between 2014 and 2015, and has since leveled off. (See figure 2.) As in past years, our review of our generalizable sample of filings found that some of the companies in our generalizable sample reported taking the same actions to improve supply chain data collection that they had taken in past years, including using standardized tools and conducting surveys. Those companies that conducted surveys reported doing further investigation into the source of minerals, for example, by following up with suppliers to improve the specificity and completeness of their survey responses. Other actions companies reported taking to improve supply chain data collection included educating suppliers about conflict-free sourcing and creating and publicizing conflict minerals policies. In interviews, representatives of selected companies and other industry participants also noted, as they had in prior years, that awareness among suppliers about the use of conflict minerals continued to increase. However, many companies reported difficulties in determining the country of origin of conflict minerals, in part as a result of lack of access to suppliers and complex supply chains involving many suppliers and processing facilities. Specifically, some companies reported that some suppliers did not respond to requests for information, or that supplier and smelter information was incomplete or contained errors. Some companies also reported, among other factors, confusion among suppliers about the requirements of the SEC disclosure rule, and gaps in supplier education and knowledge. Our review of our generalizable sample found that 94 percent of the companies that were required to conduct due diligence, as a result of their country-of-origin inquiries, reported conducting it. This percentage is similar to those in prior years: 96 percent in both 2017 and 2016. An estimated 89 percent of the companies that were required to conduct due diligence reported using a due diligence framework prescribed by the Organisation for Economic Co-operation and Development (OECD) guidance to conduct due diligence on the source and chain of custody of the conflict minerals in their products. This percentage is comparable to the 87 percent in 2017 and 92 percent in 2016. The remainder of the companies reported using non-OECD guidance or did not specify the guidance they used, if any. Of all the companies that conducted due diligence (a subset of the companies that conducted country-of-origin inquiries shown in figure 2 above), an estimated 35 percent reported that they were able to determine that their conflict minerals came from covered countries or from scrap or recycled sources, compared with 37 percent in 2017 and 39 percent in 2016. However, an estimated 61 percent of the companies reported in 2018 that they could not definitively confirm the source of the conflict minerals in their products, compared with 47 percent in 2017 and 55 percent in 2016. As in prior years, almost all of the companies that conducted due diligence reported that they could not determine whether the conflict minerals in their products had financed or benefited armed groups. Three companies in our generalizable sample determined that the minerals in at least some of their products had not financed or benefited armed groups in covered countries. None of these three companies declared their products “DRC conflict free,” which would trigger the requirement to file an independent private-sector audit report. However, one of the three companies did include one such audit report. Overall, SEC officials approximated that a total of 14 companies filed independent private-sector audit reports in 2018, compared with 16 in 2017 and 19 in 2016. Some companies and industry representatives told us—as they did last year—that even though the revised guidance and other statements made by SEC staff had raised some uncertainty about the filing process, companies generally planned to continue to report conflict minerals disclosure information. As noted earlier, the SEC’s Division of Corporation Finance issued revised guidance in April 2017 indicating that it would not recommend enforcement action to the commission if companies did not report on specified due diligence disclosure requirements. Some companies and industry participants told us that the SEC staff’s revised guidance had caused confusion among some suppliers and stakeholders about reporting requirements, sometimes leading suppliers to be reluctant or slow to share information required by companies for their due diligence reporting. In addition, some companies had changed their approach to filing as a result of the guidance. Specifically, one company in our generalizable sample of SEC filings for 2018 cited the SEC staff’s revised guidance recommending no enforcement action as the reason for its decision not to report on due diligence efforts, despite noting it had determined there was reason to believe that minerals in its products may have come from covered countries. Another company we interviewed cited the same SEC staff guidance as one of the reasons the company chose not to file an independent private-sector audit. However, representatives of other companies we interviewed told us that, generally, their companies planned to continue to report conflict minerals disclosure information, including information from their due diligence efforts. In addition, as noted above, our review of a generalizable sample of SEC filings from 2018 found that the filings were similar in number and content to those filed in 2017. Some companies told us that they would continue to file, and even expand their due diligence, in response to the conflict minerals disclosure rule and other incentives for filing—such as consumer pressure and European Union reporting requirements scheduled to take effect in 2021. Furthermore, State reported they had begun to take actions related to the revised guidance. Specifically, State officials told us that they had conducted public outreach, such as attending industry events to remind stakeholders that the conflict minerals disclosure rule was still in effect, provide an overview of the rules and requirements, and answer questions. In addition, as of June 2019, the SEC’s long-term regulatory agenda included an item indicating that the SEC Division of Corporation Finance is considering recommendations for the commission to address the effect of litigation over the conflict minerals rule. According to SEC staff, these recommendations may affect the 2017 guidance pertaining to the conflict minerals rule. We did not identify any new information on the rate of sexual violence in eastern DRC, Burundi, Rwanda, or Uganda since we last reported in June 2018; we did identify new case-file information and other information from UN reports for the DRC and Burundi. Since 2011, we have reported annually on rates of sexual violence derived from population-based surveys, as well as on case-file data as applicable, for eastern DRC (which consists of the provinces of Ituri, Maniema, North Kivu, and South Kivu) and three countries that adjoin that region: Burundi, Rwanda, and Uganda. See appendix III for population-based surveys containing sexual violence rates published since 2007. As explained in the sidebar, case-file information is unsuitable for estimating rates of sexual violence. international entities, law enforcement agencies, or medical service providers on sexual violence victims Data from population-based surveys provide a more appropriate basis for deriving a rate of sexual violence because such surveys are conducted using random sampling techniques and their results are generalizable to the target population from which a representative sample was surveyed. As we have previously reported, several factors make case-file information unsuitable for estimating rates of sexual violence. For example: Case-file data are not based on a random sample of a population, and therefore the results of analyzing these data are not generalizable. Case-file data are not aggregated across However, case-file data can provide indicators that sexual assaults are occurring in certain locations and can help service providers respond to the needs of victims. We did not identify any new population-based surveys providing rates of sexual violence in eastern DRC, Burundi, Rwanda, or Uganda published since our June 2018 report. The most recent information for eastern DRC and Rwanda dates from 2016, and for Burundi and Uganda, from 2018. UN entities, State, USAID, and a USAID-funded program have produced additional case-file information reported in 2018 and 2019 about instances of sexual violence in the DRC and Burundi that occurred in 2017 and 2018. While State’s annual country report on human rights practices for Uganda noted that rape remained a common problem in the country in 2018, we did not identify new case-file information for the country, nor did we find new case-file information regarding Rwanda. Periodic Reporting of Case-File Information on Sexual Violence in the DRC and Adjoining Countries United Nations (UN) entities and the U.S. Department of State (State) report periodically on case-file information, while the U.S. Agency for International Development (USAID) periodically receives such information from an implementing partner, as follows: Rights Office in the Democratic Republic of the Congo reports annually on human rights violations in the Democratic Republic of the Congo (DRC), including sexual violence. Representative of the Secretary- General on Sexual Violence in Conflict reports annually on cases of conflict- related sexual violence in several countries, including the DRC, using information from the United Nations Stabilization Mission in the Democratic Republic of the Congo and the United Nations Population Fund, among others. reports containing case-file information from a 5-year program that began in 2017 to counter gender-based violence in parts of eastern DRC’s North and South Kivu provinces. UN entities, State, USAID, and a USAID-funded 5-year program located in North and South Kivu provinces have produced new case-file information pertaining to sexual violence in the DRC. UN entities reported the following case-file information pertaining to sexual violence in the DRC for calendar year 2018: United Nations Joint Human Rights Office in the Democratic Republic of the Congo (UNJHRO) confirmed and documented at least 939 sexual violence victims (657 women, 279 children, and three men). According to UNJHRO, this sexual violence was perpetrated by DRC armed forces and police in many instances. Specifically, Armed Forces of the Democratic Republic of the Congo (FARDC) soldiers were responsible for 218 of these victims, 195 of whom were located in conflict-affected provinces of the DRC. Members of the Congolese National Police were responsible for 100 victims of sexual violence, 60 of whom were in conflict-affected provinces of the DRC. United Nations Stabilization Mission in the Democratic Republic of the Congo (MONUSCO) documented and verified 1,049 cases of conflict-related sexual violence against 605 women, 436 girls, four men, and four boys. According to MONUSCO, 741 of those cases were perpetrated by combatants of nonstate armed groups and armed militiamen, with the remaining 308 perpetrated by FARDC soldiers and Congolese National Police. United Nations Population Fund (UNFPA) reported 32,342 incidents of sexual violence in conflict-affected provinces between January 2018 and September 2018. UN agencies also reported in 2018 that they had provided medical assistance to over 5,200 survivors of sexual violence, and MONUSCO reported that it had supported legal clinics that provided counseling and referrals to 2,243 civilian survivors of sexual violence for calendar year 2017. State noted two instances of armed groups in eastern DRC perpetrating sexual violence reported by UN entities in calendar years 2017 and 2018. Specifically, the Bana Mura, an armed group with ties to local government, kidnapped 66 people (64 of them children) in Kasai province and used them as sexual slaves, and members of Raia Mutomboki, a rebel armed group, perpetrated sexual violence, including gang rape, against at least 66 women and girls in South Kivu province. In 2018, USAID reported that it had provided medical, legal, and other services to 7,755 survivors of sexual and gender-based violence, and had also worked with local organizations to strengthen their ability to respond to and prevent such violence, during calendar year 2017. USAID also reported that it had collaborated with the Ministry of Education to develop a curriculum focused on preventing such violence, and had worked with gender-based violence monitoring committees in 618 schools. One of USAID’s implementing partners addresses sexual and gender-based violence as part of a 5-year program. This implementing partner reported reaching 3,135 victims of gender-based violence (including 2,559 adults and 576 children) in North and South Kivu provinces, providing those victims with health, legal, and psychosocial support services during fiscal year 2018. The implementing partner also reported providing services to 1,150 victims (including 953 adults and 197 children) during the first quarter of fiscal year 2019. State’s annual human rights report for 2018, as well as UNFPA, provided some case-file information on sexual violence in Burundi. State’s annual human rights report for 2018 noted that the government-operated Humura Center had recorded 627 cases of sexual and gender-based violence in Burundi, including domestic violence, from January 2018 to early September 2018. This organization provides survivors of sexual and domestic violence with legal, medical, and psychosocial services. UNFPA reported in 2018 that it had recorded 10,592 cases of gender- based violence in 2017 and noted that the Burundian government had decided to close the local UN Office of the High Commissioner for Human Rights in December 2018, reducing the access of survivors of sexual violence to legal services. UN entities noted that the government of the DRC had taken steps to address sexual violence in the DRC since 2013, but identified an increase in the number of incidents reported beginning in 2017. The reports also noted continued difficulties providing services to victims of sexual violence and combating a climate in which perpetrators act with impunity. According to the 2018 annual UN report on conflict-related sexual violence and UN officials we interviewed in 2019, the government of the DRC has continued to take steps to address sexual violence by, for example, holding awareness-raising campaigns and establishing a nationwide victim helpline. The UN Special Representative of the Secretary-General on Sexual Violence in Conflict cited other examples, including the prosecution of military and police officials, as well as leaders of nonstate armed groups, for conflict-related sexual violence. Specifically, the UN reported in 2018 that 59 members of the Congolese National Police and the FARDC were convicted of rape in 2017. Among those convicted was a FARDC colonel sentenced for failing to prevent subordinates from committing rape. The UN also noted that the DRC had successfully prosecuted a commander of the armed group Democratic Forces for the Liberation of Rwanda for sexual violence as a war crime, and a South Kivu provincial lawmaker and his militia for crimes against humanity for the abduction and rape of 39 children. In 2019, an armed group leader—and former FARDC colonel—was convicted of war crimes, including rape. As mentioned earlier, armed conflict and political upheaval within the DRC and particularly in eastern DRC have long created an environment of persistent human rights abuses, including sexual violence, according to UN reports. The UN reported this environment worsened during the lead- up to the presidential elections between 2016 and 2018. Case-file information the UN collected on sexual violence for 2017 and 2018 indicated an upward trend in incidents in the DRC, according to UN reports. A UN report cited an increase in documented cases of sexual violence, linking it to two factors: (1) nonstate armed groups’ use of sexual violence to enforce control over illicit exploitation of natural resources, such as gold, and (2) FARDC military operations responding to the activities of these nonstate armed groups. In addition to these recent developments, UN officials we interviewed cited longstanding difficulties such as a significant shortage of response services in the DRC; common instances of retaliation against survivors who reported abuse; and, as mentioned above, a climate in which perpetrators act with impunity. The UN Commission of Inquiry on Burundi did not identify any steps taken by the government of Burundi to address the country’s human rights issues, including sexual violence, in 2017 or 2018. The Commission of Inquiry—which, according to State, was denied access to the country by the government of Burundi but conducted interviews with more than 400 witnesses living in exile—reported that serious human rights violations, including acts of sexual violence, persisted in 2017 and 2018. For example, the commission reported that the National Intelligence Service, police, and the youth wing of the ruling political party used sexual violence to target supporters of the political opposition or their relatives. The commission also recommended that the government of Burundi establish investigative bodies to look into human rights violations and take measures to ensure that victims of sexual violence have access to appropriate care, including sexual health services and psychological support. We provided a draft of this report to the SEC, State, and USAID for comment. USAID provided written comments describing some of their related activities in the DRC, which we have reprinted in appendix IV. All three agencies provided technical comments, which we have incorporated as appropriate. We are sending copies of this report to appropriate congressional committees and to the Chairman of the Securities and Exchange Commission, the Secretary of State, and the Administrator of the U.S. Agency for International Development. The report is also available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8612 or gianopoulosk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. In this report, we (1) examine how companies responded to the U.S. Securities and Exchange Commission (SEC) conflict minerals disclosure rule when filing in 2018 and (2) provide recent information on the rate of sexual violence in eastern Democratic Republic of the Congo (DRC) and adjoining countries that was published in 2018 and early 2019. To address our first objective, we downloaded the specialized disclosure reports (Form SD) from the SEC’s publically available Electronic Data Gathering, Analysis, and Retrieval (EDGAR) database in September 2018. We downloaded 1,117 Form SD filings and any associated conflict minerals reports included in EDGAR. Companies filed these Forms SD, along with related conflict minerals reports in some instances, to provide information in response to the SEC disclosure rule. To review the completeness and accuracy of the EDGAR database, we reviewed relevant documentation, interviewed knowledgeable SEC officials, and reviewed our prior reports on internal controls related to the SEC’s financial systems. We determined that the EDGAR database was sufficiently reliable for identifying the universe of Form SD filings. We reviewed the conflict minerals section of the 2010 Dodd-Frank Wall Street Reform and Consumer Protection Act (Dodd-Frank Act) and the requirements of the SEC disclosure rule to develop a data collection instrument that guided our analysis of a generalizable sample of Forms SD and conflict minerals reports. Our data collection instrument was not a compliance review of the Forms SD and conflict minerals reports. The questions were written in both yes–no and multiple-choice formats. An analyst reviewed the Forms SD and conflict minerals reports and recorded responses to the data collection instrument for all of the companies in the sample. A second analyst also reviewed the Forms SD and conflict minerals reports and verified the responses recorded by the first analyst. Analysts met to discuss and resolve any discrepancies. We randomly sampled 100 Forms SD from a population of 1,117 to create estimates generalizable to the population of all companies that filed. We selected this sample size to achieve a margin of error of no more than plus or minus 10 percentage points or less at the 95-percent confidence level, which applies to all our estimates except where noted. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have generated different estimates, we express our confidence in the precision of our particular sample’s results as a 95-percent confidence interval. This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. After using the data collection instrument to analyze the sample of filings submitted in 2018, we compared the resulting estimates with our estimates regarding filings submitted in prior years to determine whether there had been any statistically significant changes. We also attended an industry conference on conflict minerals and spoke with company representatives and industry representatives to gain additional context and perspectives. To address our second objective, we identified and assessed any information on sexual violence in eastern DRC and the three adjoining countries—Burundi, Rwanda, and Uganda—that had been published or otherwise had become available in 2018 and early 2019 and therefore would not have been included in our most recent report on the topic. We discussed the collection of sexual violence–related data in the DRC and adjoining countries, including population-based survey data and case-file data, with Department of State and U.S. Agency for International Development officials and with representatives of nongovernmental organizations and researchers. We also interviewed officials from the United Nations (UN) Children’s Fund, the UN Special Representative of the Secretary-General on Sexual Violence in Conflict, and the UN Statistics Division, and we obtained information from the UN Population Fund and UN Organization Stabilization Mission in the Democratic Republic of the Congo. In addition, we searched research databases, including MEDLINE and Scopus, to identify new academic articles containing any additional information on sexual violence published in 2018 and early 2019. Through these searches, we identified an initial list of 164 articles, which we then narrowed down to a priority list of studies by considering a variety of factors pertaining to the studies’ relevance to our second objective. These factors included (1) whether the study included rates, particularly related to the nation-wide rate of sexual violence in the DRC and region-wide rate in eastern DRC; (2) whether the study included case-file information; (3) whether the study contained data from 2011 or later; (4) whether the study focused on a subset of a broader population; (5) the geographic scope of the study; and (6) whether the study included original research. We reviewed the priority list of 16 articles and determined that none of them met our criteria for inclusion. We conducted this performance audit from September 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The U.S. Securities and Exchange Commission (SEC) conflict minerals disclosure rule requires certain companies to file a specialized disclosure report (Form SD), if the company manufactures, or contracts to have manufactured, a product or products containing conflict minerals that are necessary to the functionality or the production of those products. The rule also requires each company, as applicable, to conduct a Reasonable County of Origin Inquiry to determine whether it knows, or has reason to believe, that its conflict minerals may have originated in the covered countries or that the conflict minerals may not be from scrap or recycled sources. If the company’s inquiry shows both conditions to be true of its conflict minerals, the company must exercise due diligence and provide a description of the measures it took to exercise due diligence in determining the source and chain of custody of the conflict minerals, the facilities used to process the conflict minerals, their country of origin, and of the efforts it made to determine the mine or location of origin with the greatest possible specificity. The Form SD provides general instructions for filing conflict minerals disclosures and specifies the information that companies must provide. Companies were required to file under the rule for the first time by June 2, 2014, and annually thereafter on May 31. Figure 3 shows the flowchart included in the SEC’s adopting release for the rule, which summarized the conflict minerals disclosure rule at the time it was adopted. Since 2011, we have reported on population-based surveys containing sexual violence rates in eastern Democratic Republic of the Congo (DRC) and three adjoining countries: Burundi, Rwanda, and Uganda. Figure 4 shows the publication dates for these surveys, starting with surveys published in 2007. In addition to the individual named above, Godwin Agbara (Assistant Director), Katherine Forsyth (Analyst-in-Charge), Debbie Chung, Justin Fisher, Jieun Chang, Christopher Keblitis, Grace Lui, Nisha Rai, John Villecco, and Timothy Young made key contributions to this report. Diana Blumenfeld, Julia Jebo Grant, Farahnaaz Khakoo-Mausel, and Michael McAtee provided additional assistance. Conflict Minerals: Company Reports on Mineral Sources in 2017 Are Similar to Prior Years and New Data on Sexual Violence Are Available. GAO-18-457. Washington, D.C.: June 28, 2018. Conflict Minerals: Information on Artisanal Mined Gold and Efforts to Encourage Responsible Sourcing in the Democratic Republic of the Congo. GAO-17-733. Washington, D.C.: August 23, 2017. SEC Conflict Minerals Rule: 2017 Review of Company Disclosures in Response to the U.S. Securities and Exchange Commission Rule. GAO-17-517R. Washington, D.C.: April 26, 2017. Conflict Minerals: Insights from Company Disclosures and Agency Actions. GAO-17-544T. Washington, D.C.: April 5, 2017. SEC Conflict Minerals Rule: Companies Face Continuing Challenges in Determining Whether Their Conflict Minerals Benefit Armed Groups. GAO-16-805. Washington, D.C.: August 25, 2016. SEC Conflict Minerals Rule: Insights from Companies’ Initial Disclosures and State and USAID Actions in the Democratic Republic of the Congo Region. GAO-16-200T. Washington, D.C.: November 17, 2015. SEC Conflict Minerals Rule: Initial Disclosures Indicate Most Companies Were Unable to Determine the Source of Their Conflict Minerals. GAO-15-561. Washington, D.C.: August 18, 2015. Conflict Minerals: Stakeholder Options for Responsible Sourcing Are Expanding, but More Information on Smelters Is Needed. GAO-14-575. Washington, D.C.: June 26, 2014. SEC Conflict Minerals Rule: Information on Responsible Sourcing and Companies Affected. GAO-13-689. Washington D.C.: July 18, 2013. Conflict Minerals Disclosure Rule: SEC’s Actions and Stakeholder- Developed Initiatives. GAO-12-763. Washington, D.C.: July 16, 2012. The Democratic Republic of the Congo: Information on the Rate of Sexual Violence in War-Torn Eastern DRC and Adjoining Countries. GAO-11-702. Washington, D.C.: July 13, 2011. The Democratic Republic of the Congo: U.S. Agencies Should Take Further Actions to Contribute to the Effective Regulation and Control of the Minerals Trade in Eastern Democratic Republic of the Congo. GAO-10-1030. Washington, D.C.: September 30, 2010.", "summary": "Since the UN first deployed a peacekeeping mission to the DRC 2 decades ago, the United States and the international community have sought to improve security in the country. In eastern DRC, armed groups have committed severe human rights abuses, including sexual violence, and reportedly profit from the exploitation of “conflict minerals”—in particular, tin, tungsten, tantalum, and gold—according to the UN. Congress included a provision in the 2010 Dodd-Frank Wall Street Reform and Consumer Protection Act that, among other things, required the SEC to promulgate regulations regarding the use of conflict minerals from the DRC and adjoining countries. The SEC adopted these regulations in 2012. The act also included a provision for GAO to annually assess the SEC regulations' effectiveness in promoting peace and security and to report on the rate of sexual violence in the DRC and adjoining countries. In this report, GAO (1) examines how companies responded to the SEC conflict minerals disclosure rule when filing in 2018 and (2) provides recent information on the rate of sexual violence in eastern DRC and adjoining countries. GAO analyzed a generalizable random sample of SEC filings and interviewed relevant officials. GAO also reviewed U.S. government, UN, and international organization reports; interviewed DRC officials and other stakeholders; and conducted fieldwork in California at an industry conference. Companies' conflict minerals disclosures filed with the U.S. Securities and Exchange Commission (SEC) in 2018 were, in general, similar in number and content to disclosures filed in the prior 2 years. In 2018, 1,117 companies filed conflict minerals disclosures—about the same number as in 2017 and 2016. The percentage of companies that reported on their efforts to determine the source of minerals in their products through supply chain data collection (country-of-origin inquiries) was also similar to percentages in those 2 prior years. As a result of the inquiries they conducted, an estimated 56 percent of the companies reported whether the conflict minerals in their products came from the Democratic Republic of the Congo (DRC) or any of the countries adjoining it—similar to the estimated 53 and 49 percent in the prior 2 years. The percentage of companies able to make such a determination significantly increased between 2014 and 2015, and has since leveled off, as shown below. In their 2018 disclosures, some companies reported taking the same actions to improve supply chain data collection that they had taken in past years, and many noted difficulties in determining conflict minerals' country of origin. A subset of the companies in the figure had not determined their minerals' origin or had reason to believe their minerals were from covered countries (and not from scrap or recycled sources) and were, as a result of the inquiry, required to conduct additional research (due diligence). Of those that conducted due diligence, an estimated 61 percent reported they were unable to confirm the source of minerals in their products. An estimated 35 percent reported using conflict minerals from covered countries or from scrap or recycled sources. Although some companies noted that guidance the SEC staff revised in 2017 had caused uncertainty about the filing process, most filings were similar to those submitted in prior years. GAO found no new population-based surveys on the rate of sexual violence in eastern DRC and three countries adjoining that region—Burundi, Uganda, and Rwanda—but found other types of information on sexual violence.", "document_type": "gao"}
{"report": "Since the early 1990s, our high-risk program has focused attention on government operations with greater vulnerabilities to fraud, waste, abuse, and mismanagement, or that are in need of transformation to address economy, efficiency, or effectiveness challenges. To determine which federal government programs and functions should be designated high risk, we use our guidance document, Determining Performance and Accountability Challenges and High Risks. We consider qualitative factors, such as whether the risk (1) involves public health or safety, service delivery, national security, national defense, economic growth, or privacy or citizens’ rights; or (2) could result in significantly impaired service, program failure, injury or loss of life, or significantly reduced economy, efficiency, or effectiveness. We also consider the exposure to loss in monetary or other quantitative terms. At a minimum, $1 billion must be at risk, in areas such as the value of major assets being impaired; revenue sources not being realized; major agency assets being lost, stolen, damaged, wasted, or underutilized; potential for, or evidence of improper payments; and presence of contingencies or potential liabilities. Before making a high-risk designation, we also consider corrective measures that are planned or under way to resolve a material control weakness and the status and effectiveness of these actions. We release a High-Risk Series report every two years at the start of each new Congress. Our biennial reports detail progress made on previously designated high-risk issues. We designate any new issue areas we identify as high risk, based on the above criteria, in these reports or in separate products outside of the two-year cycle. We make out-of-cycle designations—as has been the case for seven other high-risk designations we have made—to highlight urgent issues, help ensure focused attention, and maximize the opportunity for the federal government to take action. The Office of National Drug Control Policy (ONDCP) was established by the Anti-Drug Abuse Act of 1988 as a component of the Executive Office of the President, and its Director is to assist the President in the establishment of the policies, goals, objectives, and priorities for the National Drug Control Program. In October 2018, the SUPPORT Act, among other things, reauthorized ONDCP and amended its authorities. ONDCP is responsible for (1) leading the national drug control effort, (2) coordinating and overseeing the implementation of national drug control policy, (3) assessing and certifying the adequacy of National Drug Control Programs and the budget for those programs, and (4) evaluating the effectiveness of national drug control policy efforts. As part of these efforts, ONDCP is to coordinate with more than a dozen federal agencies—known as National Drug Control Program agencies— that have responsibilities for activities including education and prevention, treatment, and law enforcement and drug interdiction (see fig. 1). Within these agencies, there may be components or offices that handle specific aspects of drug control. Some examples include SAMHSA and CDC within HHS, and the Drug Enforcement Administration (DEA) within the Department of Justice. Rates of drug misuse and drug overdose deaths have generally increased in the United States. Nationally representative data show that this increase in the estimated rate of drug misuse has occurred across several demographic categories such as sex and education levels. Nationally, the rate of drug overdose deaths decreased in 2018 after increasing almost every year since 2002. Drug overdose death rates vary by region and by different types of drugs. Drug misuse—the use of illicit drugs and the misuse of prescription drugs—has generally increased in the United States since 2002. According to SAMHSA, estimates of self-reported drug misuse among people aged 12 or older increased from 14.9 percent in 2002 to 16.7 percent in 2014, and then further increased from 17.8 percent in 2015 to 19.4 percent in 2018 (see fig. 2). The increase in estimated drug misuse from 2015 to 2018 by people aged 12 or older is evident in people across a broad range of demographic groups, including sex, race or ethnicity, military veterans, income and education levels, employment status, and geographic categories, with few exceptions (see figures 3 through 5). Additionally, the estimated percentage of drug misuse within certain demographic groups increased for some years and decreased for others, in every year more than 10 percent of the people in every demographic group reported misusing drugs. The rate of drug overdose deaths in the United States increased between 2002 and 2018 (see fig. 6). For context, in 2002, there were 23,518 drug overdose deaths, and in 2018, there were 67,367 drug overdose deaths, according to CDC data. Furthermore, the rate of drug overdose deaths increased more rapidly in recent years; the rate increased on average by 2 percent per year from 2006 through 2013, and by 14 percent per year from 2013 through 2016; however, the rate decreased by 4.6 percent between 2017 and 2018. Rates of drug overdose deaths varied in counties across the nation in 2003 and 2017, the most recent year that county-level data were available (see fig. 7). In 2017, there were some areas of the country with high rates of drug overdose deaths. For example, in 2017, 1,354 counties (43.2 percent of counties) had estimates of more than 20 drug overdose deaths per 100,000 people, including 448 counties with rates that were significantly higher than this amount. The rate of overdose deaths for different types of drugs increased between 2002 and 2018. Rates of drug overdose deaths involving synthetic opioids, natural and semi-synthetic opioids, methadone, heroin, cocaine, benzodiazepines, psychostimulants, and antidepressants generally increased between 2002 and 2018 (see fig. 8). It is important to note that drug overdose deaths may involve more than one drug, and the drugs most frequently involved in overdose deaths were often found in combination with each other. The most common drugs involved in overdose deaths vary in different parts of the United States, according to data for each of the 10 HHS public health regions (see fig. 9). Generally, in eastern regions, fentanyl was the most common drug involved in overdose deaths in 2017, the most recent year that data were available, whereas methamphetamine was the most common drug involved in overdose deaths in western regions. As previously discussed, many drug overdose deaths involve more than one drug. Past GAO work, as well as other selected government and academic studies, have found that drug misuse results in high costs for society and the economy. Such costs vary and include health care costs, criminal justice costs, workplace productivity costs, education costs, human services costs, and mortality costs. Figure 10 below includes examples of costs and other effects of drug misuse in these areas. These costs are born by federal, state, and local governments; private businesses and nonprofit organizations; employers; families, and individuals who misuse drugs. While selected studies we reviewed provided estimates for some of the costs of drug misuse, one study also indicated it is difficult to precisely quantify these costs. For example, concepts such as the quality of life or the pain and suffering of family members are difficult to fully capture or quantify. Our recent work on the topic of drug misuse and its effects has highlighted challenges the federal government faces that impede national efforts to address the drug crisis. We categorized these challenges as related to sustained leadership and strengthened coordination; capacity to address the crisis; and measurement, evaluation, and demonstration of progress. In the course of our work on the topic of drug misuse, we have identified many actions that if taken could help to address challenges in each of these areas, and have made specific recommendations to federal agencies about these actions. While over 25 of these recommendations have been implemented by National Drug Control Program agencies since fiscal year 2015, over 60 of our recommendations to at least 10 federal agencies—including recommendations that have received our highest priority designation— have not yet been implemented as of February 2020. The information below describes our findings and how agencies’ inaction on our recommendations has contributed to the federal government’s lack of progress in addressing the drug crisis. Sustained leadership and strengthened coordination. Making progress in high-risk areas requires demonstrated, strong, and sustained commitment and coordination, which we have found to be a challenge facing the federal government’s drug control efforts. Our work has identified the need for ONDCP to improve its efforts to lead and coordinate the national effort to address drug misuse and for agency leaders to engage in more effective coordination across the government and with stakeholders. ONDCP has a responsibility to coordinate and oversee the implementation of the national drug control policy across the federal government, and the National Drug Control Program agencies also have important roles and responsibilities that involve reducing drug misuse and mitigating its effects. ONDCP’s responsibility to develop the National Drug Control Strategy offers the office an important opportunity to help prioritize, coordinate, and measure key efforts to address the drug crisis. Our work has shown that ONDCP can improve its efforts to develop a National Drug Control Strategy that meets statutory requirements and effectively coordinates national efforts to address drug misuse. In 2017 and 2018, ONDCP lacked a statutorily required National Drug Control Strategy, and we recently reported that the 2019 National Drug Control Strategy did not fully comply with the law. In December 2019, we recommended that ONDCP develop and document key planning elements to help ONDCP structure its ongoing efforts and to better position the agency to meet these requirements for future iterations of the National Drug Control Strategy. ONDCP subsequently issued the 2020 National Drug Control Strategy on February 3, 2020. We reviewed this Strategy and found that it made progress in addressing several statutory requirements. For example: The 2020 National Drug Control Strategy includes 17 annual quantifiable and measurable objectives and specific targets, such as reducing overdose deaths by 15 percent by 2022, whereas we found that the 2019 National Drug Control Strategy did not contain such annual targets. The 2020 Strategy also includes a description of how each of the Strategy’s long-range goals was determined, including required consultations and data used to inform the determination, and a list of anticipated challenges to achieving the Strategy’s goals, such as limitations in existing data systems that provide little insight into emerging patterns of drug misuse, and planned actions to address them. However, the 2020 Strategy fell short in meeting other requirements. For example, the 2020 Strategy does not include a list of each National Drug Control Program agencies’ activities and the role of each activity in achieving the Strategy’s long-range goals, as required by law. The federal government invests billions of dollars each year in programs spanning over a dozen agencies, and therefore the development and implementation of a comprehensive Strategy is critical to guiding and ensuring the effectiveness of federal activities to address drug misuse. In December 2019, we recommended that ONDCP routinely implement an approach to meet the requirements for future Strategy iterations, and ONDCP agreed. ONDCP is uniquely situated to promote coordination across federal agencies. For example, the National Drug Control Strategy is required to include a description of how each of the Strategy’s long-range goals will be achieved, including a list of each existing or new coordinating mechanism to achieve each goal and a description of ONDCP’s role in facilitating achievement of each goal. The 2020 Strategy partially addressed these required elements. By including these descriptions in future iterations of the Strategy and effectively implementing them, ONDCP has the potential to strengthen coordination and provide sustained leadership. ONDCP has previously used its unique position to help implement some of our recommendations aimed at improving coordination across federal agencies in their efforts to prevent and respond to drug misuse. For example, ONDCP implemented our recommendation to assess the extent of overlap and potential for duplication across federal programs engaged in drug abuse prevention and treatment activities and to identify opportunities for increased coordination as well as developed performance metrics and reporting data regarding field-based coordination to prevent drug trafficking. We have also reported on the lack of available treatment programs for pregnant women and newborns with neonatal abstinence syndrome as well as gaps in research related to the treatment of prenatal opioid use. As of February 2020, ONDCP implemented our recommendation to document the process the agency uses to identify gaps and action items to track federal activities related to prenatal opioid use and neonatal abstinence syndrome. Sustaining and building on these coordination efforts will help maximize opportunities, leverage resources, and better position ONDCP to identify opportunities for increased efficiencies in preventing and treating drug misuse. National Drug Control Program agencies also have a responsibility to coordinate their efforts, and we have reported that gaps in agency coordination have hindered national drug control efforts. For example, the Department of Homeland Security (DHS), the U.S. Postal Service (USPS), and U.S. Customs and Border Protection (CBP) each have important roles in enforcing certain data-sharing and enforcement requirements of the Synthetics Trafficking and Overdose Prevention Act of 2018 (STOP Act). The STOP Act requires DHS to promulgate regulations detailing additional USPS responsibilities—beyond those included in the Act—related to sharing advance electronic data with CBP that can be used to identify shipments at high risk of transporting illegal drugs by October 24, 2019. However, as of November 2019, DHS had not drafted these regulations, and therefore USPS’s and CBP’s responsibilities for sharing advance electronic data—a key tool that could help stop the flow of illicit drugs into the United States—remain unclear. As we reported in December 2019, DHS does not have a plan for drafting these regulations, and therefore we recommended that DHS develop a timeline to do so; DHS agreed with this recommendation. It is also important for the federal government to coordinate among different levels of government and across issue areas, including with state, local, and tribal agencies, as well as with community groups and organizations in the private sector working to address the drug crisis. Our prior work has also found ways in which coordination between federal efforts to address drug misuse and those of local governments and other stakeholders could be more effective. In January 2018, we reported that states cited the need for additional guidance, training, and technical assistance from HHS to address the needs of infants born with prenatal drug exposure. HHS disagreed with our recommendation to provide such guidance regarding the safe care for substance-affected infants, and has not implemented the recommendation. HHS stated that it had already clarified guidance in this area and believed that states needed flexibility to meet the program requirements in the context of each state’s program. We found that states continued to report issues with the guidance, and that the clarifications did not address an ongoing challenge regarding the program requirements. We continue to believe our recommendation is warranted. As of February 2020, HHS continues to disagree with us and with the states. Without adequate supports and services to ensure their safety, these vulnerable infants may be at risk for child abuse and neglect. We have also recently recommended in January 2020 that DEA should, in consultation with industry stakeholders—such as drug distributors— identify solutions to address the limitations of the ARCOS Enhanced Lookup Buyer Statistic Tool, to ensure industry stakeholders have the most useful information possible to assist them in identifying and reporting suspicious opioid orders to DEA. DEA agreed with our recommendation, and is starting to assess how to address this recommendation. These limitations, including a lack of appropriately detailed data, may limit the usefulness of the tool in assisting distributors in determining whether an order is suspicious. In addition, we have previously reported in 2019 that coordination across private health plans, health-care prescribers, pharmacists, and at-risk beneficiaries could contribute to the success of Medicare drug monitoring programs, which are designed to identify beneficiaries at risk of opioid misuse. We also have ongoing work on how federal departments and agencies coordinate their drug prevention efforts in schools as well as on how effectively federal agencies coordinated their counter-drug activities with Mexico. Capacity to address the crisis. We have identified ongoing challenges related to the nation’s capacity to address the drug crisis. Sufficient capacity and efficient use of that capacity are key components for making progress in high-risk areas; they are necessary for federal, state, and local agencies to achieve strategic goals in addressing drug misuse, such as implementing the National Drug Control Strategy. In our work designating high-risk government programs and functions, we define capacity as having the people and resources sufficient to address the risk. Our prior work has found that the nation faces insufficient capacity to successfully address persistent, troubling trends in drug misuse, including the lack of treatment options. In addition, the nation’s existing capacity may be plagued by inefficiencies and gaps in information about what resources are most effective in addressing drug misuse. These capacity challenges permeate every level of government and affect the nation’s key social services and health care programs. As a result, effectively addressing the drug crisis requires harnessing capacity across agencies within the federal government as well as coordinating with state and local governments and community-based nongovernment organizations. The availability of treatment for substance use disorders has not kept pace with needs, and the federal government has faced barriers to increasing treatment capacity. For example, we have reported on barriers to increasing access to evidence-based treatment for opioid use disorder, and federal efforts to address these barriers. Such barriers to treatment include a lack of Medicaid coverage for treatment medications in some states, delays that can be caused by the need for prior authorizations for some treatment medications, and the unwillingness of some health care providers to obtain the federal waiver required to prescribe some treatment medications. We have also reported that, according to officials at the Veterans Health Administration (VHA), many veterans lack access to residential substance use treatment programs because of high demand relative to capacity. Developing and maintaining sufficient capacity to address the drug crisis also requires that federal agencies use existing resources—such as data—effectively. For example, we have recently reported in January 2020 that DEA should be more proactive in using the data it already collects from DEA registrants to identify problematic drug transaction patterns. According to DEA officials, one analysis that they conduct on a quarterly basis involves using a computer algorithm when comparing large volumes of drugs purchased in a given geographic area to the area’s population data. However, DEA did not report conducting active and recurring monitoring of transactions using algorithms to detect and flag transactions that indicate potential diversion, either on a real-time or near real-time basis, to help identify questionable patterns in the data or unusual patterns of drug distribution on a more routine basis. Such analyses could be used to proactively support or generate leads for investigations of potential drug diversion. Registrants already report data on controlled-substance transactions to the DEA. DEA could use these data to identify trends in distribution or purchases of drugs in a given geographic area. DEA could also look for and compare unusual patterns in drug order activity in different locations to identify potential issues that warrant further investigation. Further, DEA has not established a way to manage all of the data it collects and maintains. DEA agreed with three of our four recommendations to better manage and use the data it collects. DEA neither agreed nor disagreed with the fourth recommendation. However, DEA has not yet implemented any of the recommendations. By implementing these recommendations, DEA could ensure that important data assets are formally managed and fully utilized to inform investigations and prevent diversion of prescription opioids to be sold illegally. Overall, federal efforts to address the drug crisis could make better use of available data to assist in identifying emerging patterns of misuse, allowing the government to respond more quickly to evolving trends. Beyond specific capacity challenges that we have identified, in December 2019 we reported on challenges federal agencies face in assessing the resources they will need to achieve the goals of the National Drug Control Strategy. ONDCP is required to issue drug control funding guidance to the heads of departments and agencies with responsibilities under the National Drug Control Program by July 1 of each year, and such funding guidance must address funding priorities developed in the National Drug Control Strategy. Since ONDCP did not issue a Strategy in 2017 or 2018, ONDCP could neither provide funding guidance to National Drug Control Program agencies based on the Strategy, nor could it review and certify budget requests of these agencies to determine if they are adequate to meet the goals of the Strategy, in accordance with and as required by law. Without a National Drug Control Strategy in 2017 or 2018, ONDCP used other sources—such as policy priorities identified in the President’s Budget from fiscal year 2018—to identify drug policy priorities and develop funding guidance. ONDCP issued a National Drug Control Strategy in 2019 and 2020, but neither Strategy included a 5-year projection for program and budget priorities, as required by law. In December 2019, we recommended that ONDCP develop and document key planning elements to help ONDCP structure its ongoing efforts and to better position the agency to meet these requirements for future iterations of the National Drug Control Strategy. We also found that the 2020 National Drug Control Strategy does not include estimates of federal funding or other resources needed to achieve each of ONDCP’s long-range goals. The 2020 Strategy includes a plan to expand treatment of substance use disorders that identifies unmet needs for substance use disorder treatment and a strategy for closing the gap between available and needed treatment. The plan also describes the roles and responsibilities of relevant National Drug Control Program agencies for implementing the plan. However, the plan does not identify resources required to enable National Drug Control Program agencies to implement the plan or resources required to eliminate the treatment gap, as required by law. The National Drug Control Strategy is important for assessing the nation’s capacity to address drug misuse through both the development of federal funding estimates and the certification of agency budget requests that aim to meet the goals of the Strategy. Additionally, we have ongoing work on the federal government’s capacity to address the drug crisis. For example, we are studying gaps in the capacity of the health care system to treat substance use disorders, and examining how grantees use funding from selected SAMHSA grant programs to increase access to substance use disorder treatment. We are also studying school-based drug prevention programs and the effects of drug misuse on the workforce. This work will examine challenges that states and local educational entities face in serving the needs of communities affected by the drug crisis. We also have planned work examining the effectiveness of federal funding to combat the ongoing opioid crisis. Measurement, evaluation, and demonstration of progress. The federal government faces challenges related to measuring, evaluating, and demonstrating progress towards addressing the crisis. We have reported that key data needed to measure and evaluate progress towards strategic goals are not reliable or are not collected and reported. We have also found that some agencies lack plans or metrics to measure the effectiveness of specific programs to address the drug crisis and to demonstrate that these programs are making progress towards stated national goals, including reducing drug overdose deaths and expanding access to addiction treatment. Successfully addressing drug misuse requires ongoing measurement and evaluation of efforts towards stated goals and the ability to share and use performance information to make midcourse changes and corrections where needed. Regarding challenges related to data, we have identified gaps in the availability and reliability of data for measuring progress. ONDCP and other federal, state, and local government officials have identified challenges with the timeliness, accuracy, and accessibility of data from law enforcement and public health sources related to both fatal and non- fatal overdose cases. In March 2018, we recommended that ONDCP lead a review on ways to improve overdose data; ONDCP did not indicate whether it agreed with our recommendation. Additionally, in December 2019, we found that ONDCP’s Drug Control Data Dashboard did not include all of the data required by the SUPPORT Act, such as data sufficient to show the extent of the unmet need for substance use disorder treatment. We recommended that ONDCP establish the planning elements to ensure that these data were included in the Data Dashboard, and ONDCP disagreed with our recommendation. Having accessible and reliable data, including data on drug overdoses will help ONDCP and other agencies better measure the scope and nature of the drug crisis. We also found in 2019 that the State Department cannot ensure the reliability of its program monitoring data for its Caribbean Basin Security Initiative, which seeks to reduce illicit drug trafficking. The State Department agreed with the recommendation to ensure the development and implementation of a data management system for centrally collecting reliable program monitoring data for all Caribbean Basin Security Initiative activities, but has not yet implemented it. Without this action, there may be discrepancies in how Caribbean Basin Security Initiative program performance data is defined and collected, and the State Department cannot report comprehensively or accurately on the Initiative’s activities to reduce illicit drug trafficking or track data trends across countries. While ONDCP is responsible for evaluating the effectiveness of national drug control policy efforts across the government, we found that ONDCP has not developed performance evaluation plans for the goals in the 2020 National Drug Control Strategy. Some of the long-range goals listed in the 2020 Strategy include expanding access to evidence-based treatment, reducing the availability of illicit drugs in the United States, and decreasing the over-prescribing of opioid medications. However, the 2020 National Drug Control Strategy does not include performance evaluation plans to measure progress against each of the Strategy’s long-range goals, as required by law. These performance evaluation plans are required by statute to include (1) specific performance measures for each National Drug Control Program agency, (2) annual and—to the extent practicable—quarterly objectives and targets for each measure, and (3) an estimate of federal funding and other resources necessary to achieve each performance objective and target. Without effective long-term plans that clearly articulate goals and objectives and without specific measures to track performance, federal agencies cannot fully assess whether taxpayer dollars are invested in ways that will achieve desired outcomes such as reducing access to illicit drugs and expanding treatment for substance use disorders. Additionally, National Drug Control Program agencies are responsible for evaluating their progress toward achieving the goals of the National Drug Control Strategy, and in some cases have improved how to measure this progress. For example, although the federal government continues to face barriers to increasing access to treatment for substance use disorders, HHS has recently implemented our recommendation to establish performance measures with targets to expand access to medication-assisted treatment (MAT) for opioid use disorders. As of March 2020, HHS has established such performance measures with targets to increase the number of prescriptions for MAT medications and to increase treatment capacity, as measured by the number of providers authorized to treat patients using MAT. Monitoring progress against these targets will help HHS determine whether its efforts to expand treatment are successful or whether new approaches are needed. We have also identified challenges regarding how federal agencies demonstrate the progress of specific programs toward addressing the drug crisis. We reported in 2018 on DEA’s 360 Strategy—which aims to coordinate DEA enforcement, diversion control, and demand reduction efforts—as well as on ONDCP’s Heroin Response Strategy under its High Intensity Drug Trafficking Areas program. We found that neither DEA’s 360 Strategy nor ONDCP’s Heroin Response Strategy included outcome- oriented performance measures for its activities and goals, respectively. DEA disagreed with and has not yet implemented our recommendation to establish these types of performance measures for its activities. ONDCP neither agreed nor disagreed with our recommendation to establish outcome-oriented performance measures for the goals of the Heroin Response Strategy, and has not yet implemented the recommendation. Without these measures, it is unclear the extent to which DEA or ONDCP can accurately and fully gauge their efforts and their overall effectiveness in combatting heroin and opioid use and reducing overdose deaths. Additionally, we have found that DEA does not have outcome-oriented goals and performance targets for its use of data in opioid diversion activities, making DEA likely not able to adequately assess whether its investments and efforts are helping to limit the availability of and better respond to the opioid prescription diversion threat. DEA neither agreed nor disagreed with our recommendation to establish these outcome- oriented goals and related performance targets for its opioid diversion activities, and has not implemented this recommendation. We have also reported that the Department of State has not established performance indicators for its Caribbean Basin Security Initiative to facilitate performance evaluation across agencies, countries, and activities, inhibiting the assessment of the program’s progress to reduce illicit drug trafficking. The State Department agreed with our recommendation to develop and implement a data management system for centrally collecting reliable program monitoring data. The State Department has not yet implemented this recommendation. Without robust assessments of how specific programs help to achieve the goals of the National Drug Control Strategy, federal agencies may be unable to demonstrate progress in addressing the drug crisis, and may be unable to make any needed adjustments to their strategies. Illicit drug use and misuse of prescription drugs is a long-standing national problem that will continue to evolve. The terrible effects of drug misuse on families and communities have persisted over decades, despite ongoing federal, state, and local efforts. Federal agencies and Congress can and must work to ensure that available resources are coordinated effectively to mitigate and respond to the drug misuse crisis. Maintaining sustained attention on preventing, responding to, and recovering from drug misuse will be challenging in the coming months as many of the federal agencies responsible for addressing drug misuse are currently focused on addressing the COVID-19 pandemic. However, the severe public health and economic effects of the pandemic could fuel some of the contributing factors of drug misuse, such as unemployment— highlighting the need to sustain drug misuse prevention, response, and recovery efforts. Addressing these challenges will require sustained leadership and strengthened coordination; the necessary capacity to address the crisis; and the systems to measure, evaluate, and demonstrate progress. The more than 60 related GAO recommendations that have yet to be implemented are an indication of how federal agencies may begin addressing these challenges. For example: ONDCP should ensure future iterations of the National Drug Control Strategy include all statutorily required elements. Examples of statutorily required elements include a 5-year projection for the National Drug Control Program and budget priorities; a description of how each of the Strategy’s long-range goals will be achieved, including a list of each National Drug Control Program agency’s activities, and the role of each activity in achieving these goals, and estimates of federal funding or other resources needed to achieve these goals; performance evaluation plans for each year covered by the Strategy for each long-range goal for each National Drug Control Program agency; and resources required to enable National Drug Control Program agencies to implement the plan to expand treatment of substance use disorders and eliminate the treatment gap; ONDCP should take steps to ensure effective, sustained implementation of the 2020 National Drug Control Strategy and future strategies; HHS should provide guidance to states for the safe care for infants born with prenatal drug exposure, who may be at risk for child abuse and neglect; DEA should take steps to better analyze and use drug transaction data to identify suspicious opioid orders and prevent diversion of prescription opioids to be sold illegally; and the State Department should develop and implement a data management system for all Caribbean Basin Security Initiative activities to reduce illicit drug trafficking or track data trends across countries. Through our ongoing and planned work, we will continue to review the effects of drug misuse, the federal response, and opportunities for improvement. We provided draft report excerpts regarding our analysis of the 2020 National Drug Control Strategy to the Office of National Drug Control Policy for review and comment. ONDCP officials stated that they plan to address the statutory requirements that we identified as missing in additional documents, including the Fiscal Year 2021 Budget and Performance Summary. We will review and assess any additional materials that ONDCP publishes in response to the requirements for the 2020 National Drug Control Strategy. Findings regarding other programs and activities are drawn from past GAO work and our follow-up work on our recommendations; the related content was previously provided to the respective agencies for review as part of the original work. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of National Drug Control Policy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Triana McNeil at (202) 512-8777 or McNeilT@gao.gov, Mary Denigan- Macauley at (202) 512-7114 or DeniganMacauleyM@gao.gov, or Jacqueline M. Nowicki at (617) 788-0580 or NowickiJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. International Mail: Stakeholders' Views on Possible Changes to Inbound Mail Regarding Customs Fees and Opioid Detection Efforts. GAO-20- 340R. Washington, D.C.: February 27, 2020. Medicaid: States' Changes to Payment Rates for Substance Use Disorder Services. GAO-20-260. Washington, D.C.: January 30, 2020. Drug Control: Actions Needed to Ensure Usefulness of Data on Suspicious Opioid Orders. GAO-20-118. Washington, D.C.: January 29, 2020. Opioid Use Disorder: Barriers to Medicaid Beneficiaries’ Access to Treatment Medications. GAO-20-233. Washington, D.C.: January 24, 2020. Social Security Disability: Action Needed to Help Agency Staff Understand and Follow Policies Related to Prescription Opioid Misuse. GAO-20-120. Washington, D.C.: January 9, 2020. Countering Illicit Finance and Trade: U.S. Efforts to Combat Trade-Based Money Laundering. GAO-20-314R. Washington, D.C.: December 30, 2019. Drug Control: The Office of National Drug Control Policy Should Develop Key Planning Elements to Meet Statutory Requirements. GAO-20-124. Washington, D.C.: December 18, 2019. International Mail: Progress Made in Using Electronic Data to Detect Illegal Opioid Shipments, but Additional Steps Remain. GAO-20-229R. Washington, D.C.: December 18, 2019. Counternarcotics: Treasury Reports Some Results from Designating Drug Kingpins, but Should Improve Information on Agencies’ Expenditures. GAO-20-112. Washington, D.C.: December 16, 2019. Mental Health and Substance Use: State and Federal Oversight of Compliance with Parity Requirements Varies. GAO-20-150. Washington, D.C.: December 13, 2019. Veterans Health Care: Services for Substance Use Disorders, and Efforts to Address Access Issues in Rural Areas. GAO-20-35. Washington, D.C.: December 2, 2019. Coast Guard: Assessing Deployable Specialized Forces' Workforce Needs Could Improve Efficiency and Reduce Potential Overlap or Gaps in Capabilities. GAO-20-33. Washington, D.C.: November 21, 2019. Substance Use Disorder: Prevalence of Recovery Homes, and Selected States’ Investigations and Oversight. GAO-20-214T. Washington, D.C.: October 24, 2019. Medicaid: Opioid Use Disorder Services for Pregnant and Postpartum Women, and Children. GAO-20-40. Washington, D.C.: October 24, 2019. U.S. Assistance to Central America: Department of State Should Establish a Comprehensive Plan to Assess Progress Toward Prosperity, Governance, and Security. GAO-19-590. Washington, D.C.: September 26, 2019. Science & Tech Spotlight: Opioid Vaccines. GAO-19-706SP. Washington, D.C.: September 16, 2019. U.S. Assistance to Mexico: State and USAID Allocated over $700 Million to Support Criminal Justice, Border Security, and Related Efforts from Fiscal Year 2014 through 2018. GAO-19-647. Washington, D.C.: September 10, 2019. Prescription Opioids: Patient Options for Safe and Effective Disposal of Unused Opioids. GAO-19-650. Washington, D.C.: September 3, 2019. Land Ports of Entry: CBP Should Update Policies and Enhance Analysis of Inspections. GAO-19-658. Washington, D.C.: August 6, 2019. Drug Control: Certain DOD and DHS Joint Task Forces Should Enhance Their Performance Measures to Better Assess Counterdrug Activities. GAO-19-441. Washington, D.C.: July 9, 2019. VA Mental Health: VHA Improved Certain Prescribing Practices, but Needs to Strengthen Treatment Plan Oversight. GAO-19-465. Washington, D.C.: June 17, 2019. Health Centers: Trends in Revenue and Grants Supported by the Community Health Center Fund. GAO-19-496. Washington, D.C.: May 30, 2019. Prescription Opioids: Voluntary Medicare Drug Management Programs to Control Misuse. GAO-19-446. Washington, D.C.: May 20, 2019. Drug Policy: Assessing Treatment Expansion Efforts and Drug Control Strategies and Programs. GAO-19-535T. Washington, D.C.: May 9, 2019. Drug Policy: Preliminary Observations on the 2019 National Drug Control Strategy. GAO-19-370T. Washington, D.C.: March 7, 2019. Behavioral Health: Research on Health Care Costs of Untreated Conditions is Limited. GAO-19-274. Washington, D.C.: February 28, 2019. Security Assistance: U.S. Agencies Should Establish a Mechanism to Assess Caribbean Basin Security Initiative Progress. GAO-19-201. Washington, D.C.: February 27, 2019. Drug Control: DOD Should Improve Its Oversight of the National Guard Counterdrug Program.GAO-19-27. Washington, D.C.: January 17, 2019. Colombia: U.S. Counternarcotics Assistance Achieved Some Positive Results but State Needs to Review the Overall U.S. Approach. GAO-19-106. Washington, D.C.: December 12, 2018. Illegal Marijuana: Opportunities Exist to Improve Oversight of State and Local Eradication Efforts. GAO-19-9. Washington, D.C.: November 14, 2018. Opioid Crisis: Status of Public Health Emergency Authorities. GAO-18-685R. Washington, D.C.: September 26, 2018. Adolescent and Young Adult Substance Use: Federal Grants for Prevention, Treatment, and Recovery Services and for Research. GAO-18-606. Washington, D.C.: September 4, 2018. Foster Care: Additional Actions Could Help HHS Better Support States’ Use of Private Providers to Recruit and Retain Foster Families. GAO-18-376. Washington, D.C.: May 30, 2018. VA Health Care: Progress Made Towards Improving Opioid Safety, but Further Efforts to Assess Progress and Reduce Risk Are Needed. GAO-18-380. Washington, D.C.: May 29, 2018. Prescription Opioids: Medicare Needs Better Information to Reduce the Risk of Harm to Beneficiaries. GAO-18-585T. Washington, D.C.: May 29, 2018. Illicit Opioids: Office of National Drug Control Policy and Other Agencies Need to Better Assess Strategic Efforts. GAO-18-569T. Washington, D.C.: May 17, 2018. Substance Use Disorder: Information on Recovery Housing Prevalence, Selected States’ Oversight, and Funding. GAO-18-315. Washington, D.C.: March 22, 2018. Illicit Opioids: While Greater Attention Given to Combating Synthetic Opioids, Agencies Need to Better Assess their Efforts. GAO-18-205. Washington, D.C.: March 29, 2018. Substance-Affected Infants: Additional Guidance Would Help States Better Implement Protections for Children. GAO-18-196. Washington, D.C.: January 19, 2018. Prescription Opioids: Medicare Should Expand Oversight Efforts to Reduce the Risk of Harm. GAO-18-336T. Washington, D.C.: January 17, 2018. Preventing Drug Abuse: Low Participation by Pharmacies and Other Entities as Voluntary Collectors of Unused Prescription Drugs. GAO-18-25. Washington, D.C.: October 12, 2017. Border Patrol: Issues Related to Agent Deployment Strategy and Immigration Checkpoints. GAO-18-50. Washington, D.C.: November 8, 2017. Prescription Opioids: Medicare Needs to Expand Oversight Efforts to Reduce the Risk of Harm. GAO-18-15. Washington, D.C.: October 6, 2017. Opioid Use Disorders: HHS Needs Measures to Assess the Effectiveness of Efforts to Expand Access to Medication-Assisted Treatment. GAO-18-44. Washington, D.C.: October 31, 2017. Counternarcotics: Overview of U.S. Efforts in the Western Hemisphere. GAO-18-10. Washington, D.C.: October 13, 2017. Newborn Health: Federal Action Needed to Address Neonatal Abstinence Syndrome. GAO-18-32. Washington, D.C.: October 4, 2017. Anti-Money Laundering: U.S. Efforts to Combat Narcotics-Related Money Laundering in the Western Hemisphere. GAO-17-684. Washington, D.C.: August 22, 2017. Nonviolent Drug Convictions: Stakeholders’ Views on Potential Actions to Address Collateral Consequences. GAO-17-691. Washington, D.C.: September 7, 2017. Medicaid: States Fund Services for Adults in Institutions for Mental Disease Using a Variety of Strategies. GAO-17-652. Washington, D.C.: August 9, 2017. International Mail Security: Costs and Benefits of Using Electronic Data to Screen Mail Need to Be Assessed. GAO-17-606. Washington, D.C.: August 2, 2017. Drug Control Policy: Information on Status of Federal Efforts and Key Issues for Preventing Illicit Drug Use. GAO-17-766T. Washington, D.C.: July 26, 2017. Medicaid Expansion: Behavioral Health Treatment Use in Selected States in 2014. GAO-17-529. Washington, D.C.: June 22, 2017. Border Security: Additional Actions Could Strengthen DHS Efforts to Address Subterranean, Aerial, and Maritime Smuggling. GAO-17-474. Washington, D.C.: May 1, 2017. VA Health Care: Actions Needed to Ensure Medical Facilities’ Controlled Substance Programs Meet Requirements. GAO-17-442T. Washington, D.C.: February 27, 2017. VA Health Care: Actions Needed to Ensure Medical Facility Controlled Substance Inspection Programs Meet Agency Requirements. GAO-17-242. Washington, D.C.: February 15, 2017. Drug Free Communities Support Program: Agencies Have Strengthened Collaboration but Could Enhance Grantee Compliance and Performance Monitoring. GAO-17-120. Washington, D.C.: February 7, 2017. Highlights of a Forum: Preventing Illicit Drug Use. GAO-17-146SP. Washington, D.C.: November 14, 2016. Opioid Addiction: Laws, Regulations, and Other Factors Can Affect Medication-Assisted Treatment Access. GAO-16-833. Washington, D.C.: September 27, 2016. Drug Enforcement Administration: Additional Actions Needed to Address Prior GAO Recommendations. GAO-16-737T. Washington, D.C.: June 22, 2016. Controlled Substances: DEA Should Take Additional Actions to Reduce Risks in Monitoring the Continued Eligibility of Its Registrants. GAO-16-310. Washington, D.C.: May 26, 2016. Office of National Drug Control Policy: Progress toward Some National Drug Control Strategy Goals, but None Have Been Fully Achieved. GAO-16-660T. Washington, D.C.: May 17, 2016. Veterans Justice Outreach Program: VA Could Improve Management by Establishing Performance Measures and Fully Assessing Risks. GAO-16-393. Washington, D.C.: April 28, 2016. State Marijuana Legalization: DOJ Should Document Its Approach to Monitoring the Effects of Legalization. GAO-16-419T. Washington, D.C.: April 5, 2016. DOD and VA Health Care: Actions Needed to Help Ensure Appropriate Medication Continuation and Prescribing Practices. GAO-16-158. Washington, D.C.: January 5, 2016. State Marijuana Legalization: DOJ Should Document Its Approach to Monitoring the Effects of Legalization. GAO-16-1. Washington, D.C.: December 30, 2015. Office of National Drug Control Policy: Lack of Progress on Achieving National Strategy Goals. GAO-16-257T. Washington, D.C.: December 2, 2015. Drug Control: Additional Performance Information Is Needed to Oversee the National Guard’s State Counterdrug Program. GAO-16-133. Washington, D.C.: October 21, 2015. Medicaid: Additional Reporting May Help CMS Oversee Prescription-Drug Fraud Controls. GAO-15-390. Washington, D.C.: July 8, 2015. Prescription Drugs: More DEA Information about Registrants’ Controlled Substances Roles Could Improve Their Understanding and Help Ensure Access. GAO-15-471. Washington, D.C.: June 25, 2015. Behavioral Health: Options for Low-Income Adults to Receive Treatment in Selected States. GAO-15-449. Washington, D.C.: June 19, 2015. Drug-Impaired Driving: Additional Support Needed for Public Awareness Initiatives. GAO-15-293. Washington, D.C.: February 24, 2015. Prenatal Drug Use and Newborn Health: Federal Efforts Need Better Planning and Coordination. GAO-15-203. Washington, D.C.: February 10, 2015. Medicare Program Integrity: CMS Pursues Many Practices to Address Prescription Drug Fraud, Waste, and Abuse. GAO-15-66. Washington, D.C.: October 24, 2014. Office of National Drug Control Policy: Office Could Better Identify Opportunities to Increase Program Coordination. GAO-13-333. Washington, D.C.: March 26, 2013. Drug Control: Initial Review of the National Strategy and Drug Abuse Prevention and Treatment Programs. GAO-12-744R. Washington, D.C.: July 6, 2012. Prescription Pain Reliever Abuse: Agencies Have Begun Coordinating Education Efforts, but Need to Assess Effectiveness. GAO-12-115. Washington, D.C.: December 22, 2011. Adult Drug Courts: Studies Show Courts Reduce Recidivism, but DOJ Could Enhance Future Performance Measure Revision Efforts. GAO-12-53. Washington, D.C.: December 9, 2011. Drug Control: U.S. Assistance Has Helped Mexican Counternarcotics Efforts, but Tons of Illicit Drugs Continue to Flow into the United States. GAO-07-1018. Washington, D.C.: August 17, 2007. Adult Drug Courts: Evidence Indicates Recidivism Reductions and Mixed Results for Other Outcomes. GAO-05-219. Washington, D.C.: February 28, 2005. Prescription Drugs: OxyContin Abuse and Diversion and Efforts to Address the Problem. GAO-04-110. Washington, D.C.: December 19, 2003. Drug Courts: Better DOJ Data Collection and Evaluation Efforts Needed to Measure Impact of Drug Court Programs. GAO-02-434. Washington, D.C.: April 18, 2002. Drug Abuse: Efforts under Way to Determine Treatment Outcomes. T-HEHS-00-60. Washington, D.C.: February 17, 2000. Emerging Drug Problems: Despite Changes in Detection and Response Capability, Concerns Remain. HEHS-98-130. Washington, D.C.: July 20, 1998. Drug Courts: Overview of Growth, Characteristics, and Results. GGD-97- 106. Washington, D.C.: July 31, 1997. Drug Control: Reauthorization of the Office of National Drug Control Policy. T-GGD-97-97. Washington, D.C.: May 1, 1997. Confronting the Drug Problem: Debate Persists on Enforcement and Alternative Approaches. GGD-93-82. Washington, D.C.: July 1, 1993. War on Drugs: Federal Assistance to State and Local Drug Enforcement. GGD-93-86. Washington, D.C.: April 29, 1993. Drug Control: Coordination of Intelligence Activities.GGD-93-83BR. Washington, D.C.: April 2, 1993. Drug Abuse Prevention: Federal Efforts to Identify Exemplary Programs Need Stronger Design. PEMD-91-15. Washington, D.C.: August 22, 1991. VA Health Care: Inadequate Controls over Addictive Drugs. HRD-91-101. Washington, D.C.: June 6, 1991. The War on Drugs: Arrests Burdening Local Criminal Justice Systems. GGD-91-40. Washington, D.C.: April 3, 1991. Drug Treatment: Targeting Aid to States Using Urban Population as Indicator of Drug Use. HRD-91-17. Washington, D.C.: November 27, 1990. Controlling Drug Abuse: A Status Report. GGD-88-39. Washington, D.C.: March 1, 1988. Drug Abuse Prevention: Further Efforts Needed To Identify Programs That Work. HRD-88-26. Washington, D.C.: December 4, 1987. Comprehensive Approach Needed To Help Control Prescription Drug Abuse. GGD-83-2. Washington, D.C.: October 29, 1982. Action Needed To Improve Management and Effectiveness of Drug Abuse Treatment. HRD-80-32 Washington, D.C.: April 14, 1980. Identifying and Eliminating Sources of Dangerous Drugs: Efforts Being Made, but Not Enough. B-175425. Washington, D.C.: Jun 7, 1974. United States Efforts to Increase International Cooperation in Controlling Narcotics Trafficking. B-176625. Washington, D.C.: October 4, 1972. Efforts to Prevent Dangerous Drugs from Illicitly Reaching the Public. B- 175425. Washington, D.C.: April 17, 1972. Triana McNeil at (202) 512-8777 or McNeilT@gao.gov; Mary Denigan- Macauley at (202) 512-7114 or DeniganMacauleyM@gao.gov; or Jacqueline M. Nowicki at (617) 788-0580 or NowickiJ@gao.gov. In addition to the contacts named above, Alana Finley (Assistant Director), Bill Keller (Assistant Director), Will Simerl (Assistant Director), Meghan Squires (Analyst-in-Charge), James Bennett, Ben Bolitzer, Breanne Cave, Billy Commons, Holly Dye, Wendy Dye, Brian Egger, Kaitlin Farquharson, Sally Gilley, Sarah Gilliland, Mara McMillen, Amanda Miller, Sean Miskell, Jan Montgomery, Dae Park, Bill Reinsberg, Emily Wilson Schwark, Herbie Tinsley, and Sirin Yaemsiri made key contributions to this report. Key contributors to the prior work discussed in this report are listed in each respective product.", "summary": "Drug misuse—the use of illicit drugs and the misuse of prescription drugs—has been a persistent and long-standing public health issue in the United States. Ongoing drug control efforts seek to address drug misuse through education and prevention, addiction treatment, and law enforcement and drug interdiction, as well as programs that serve populations affected by drug misuse. These efforts involve federal, state, local, and tribal governments as well as community groups and the private sector. In recent years, the federal government has spent billions of dollars and has enlisted more than a dozen agencies to address drug misuse and its effects. This report provides information on (1) trends in drug misuse (2) costs and other effects of drug misuse on society and the economy, and (3) challenges the nation faces in addressing the drug crisis. GAO analyzed nationally representative federal data on drug misuse and deaths from overdoses for 2002–2018 (the most recent available); reviewed selected empirical studies published from 2014–2019; and compared GAO's High-Risk list criteria to findings and recommendations in over 75 GAO reports issued from fiscal year 2015 through March 2020. Nationally, since 2002, rates of drug misuse have increased, according to GAO's analysis of federal data. In 2018, the Substance Abuse and Mental Health Services Administration reported that an estimated 19 percent of the U.S. population (over 53 million people) misused or abused drugs, an increase from an estimated 14.7 percent in 2003. People across a broad range of demographic groups—including sex, race or ethnicity, education levels, employment status, and geographic categories—reported misusing drugs (see figure below). The rates of drug overdose deaths have also generally increased nationally since the early 2000s. Over 716,000 people have died of a drug overdose since 2002, and in 2018 alone, over 67,000 people died as a result of a drug overdose, according to the Centers for Disease Control and Prevention. Although the number of drug overdose deaths in 2018 decreased compared to 2017, drug misuse in the United States continued to rise. Rates of drug overdose deaths varied in counties across the nation in 2003 and 2017, the most recent year that county-level data were available (see figure below). In 2017, 43.2 percent of counties had estimates of more than 20 drug overdose deaths per 100,000 people, including 448 counties with rates that were significantly higher than this amount. Note: CDC's National Center for Health Statistics used a statistical model to estimate rates of drug overdose deaths to account for counties where data were sparse because of small population size. GAO work and other government and academic studies have found that the negative health and societal effects of drug misuse are widespread and costly—for example, the increased need for health care, human services, and special education; increased crime, childhood trauma, reduced workforce productivity; and loss of life. The federal government is making progress in some areas, but a strategic, coordinated, and effective national response—with key sustained leadership from federal agencies—is needed. This report identifies opportunities to strengthen the federal government's efforts to address this persistent and increasing problem. These opportunities include addressing challenges in providing sustained leadership and strengthened coordination; the necessary capacity to address the crisis; and systems to measure, evaluate, and demonstrate progress. For example: the Office of National Drug Control Policy should ensure future iterations of the National Drug Control Strategy include all statutorily required elements. Examples of statutorily required elements include a 5-year projection for the National Drug Control Program and budget priorities; a description of how each of the Strategy's long-range goals will be achieved, including estimates of needed federal resources; and performance evaluation plans for these goals, among other requirements; the Office of National Drug Control Policy should ensure effective, sustained implementation of the 2020 Strategy and future strategies; the Department of Health and Human Services should provide guidance to states for the safe care for infants born with prenatal drug exposure, who may be at risk for child abuse and neglect; the Drug Enforcement Administration should take steps to better analyze and use drug transaction data to prevent diversion of prescription opioids to be sold illegally; and the State Department should develop and implement a data management system for all Caribbean Basin Security Initiative activities to reduce illicit drug trafficking or track data trends across countries. In GAO's March 2019 High-Risk report, GAO named drug misuse as an emerging issue requiring close attention. Based on 25 GAO products issued since that time and this update, GAO has determined that this issue is high risk. Moreover, the severe public health and economic effects of the Coronavirus Disease 2019 (COVID-19) pandemic could fuel some of the contributing factors of drug misuse, such as unemployment—highlighting the need to sustain and build upon ongoing efforts. However, maintaining sustained attention on preventing, responding to, and recovering from drug misuse will be challenging in the coming months, as many of the federal agencies responsible for addressing drug misuse are focused on addressing the pandemic. Therefore, GAO will include this issue in the 2021 High-Risk Series update and make the high-risk designation effective at that time. Since fiscal year 2015, GAO has made over 80 recommendations to multiple agencies responsible for addressing the drug crisis; over 60 of these recommendations have yet to be implemented.", "document_type": "gao"}
{"report": "The primary purpose of accreditation is to help ensure that schools provide a quality education to students. Accrediting agencies, also known as accreditors, are generally nongovernmental, nonprofit entities that work with Education and states as part of the “triad” that oversees postsecondary schools participating in federal student aid programs. The Higher Education Act and Education’s regulations require accreditors to meet certain criteria and have certain operating procedures in place to be “recognized” by Education as reliable authorities on assessing academic quality (see fig. 1). Accreditors must have their recognition renewed by Education at least every 5 years. To recognize an accrediting agency, Education officials and the National Advisory Committee on Institutional Quality and Integrity (NACIQI), which advises the Secretary of Education on accreditation issues, review among other things whether the accreditor applies its own standards, policies, and procedures when they accredit schools. While Education is required to determine whether accrediting agencies have standards for schools in certain areas, such as student achievement and curricula, before recognizing them, the accrediting agencies are responsible for evaluating member schools to determine if they meet the accreditors’ standards. The specific standards that accreditors develop in these areas can differ, and accreditors may also establish additional standards in areas not required by law. When schools do not meet accreditor standards, accrediting agencies may impose sanctions, such as placing a school on probation or terminating the school’s accreditation. Education conducts annual reviews of the financial condition of all schools participating in federal student aid programs to determine if they are financially responsible, based on criteria and processes established in federal law and regulations. The specific financial responsibility standards that apply to each school depend on the school’s ownership type, and the bulk of Education’s financial oversight efforts focus on private nonprofit and for-profit schools. One key financial responsibility standard that Education uses to assess nonprofit and for-profit schools is a financial composite score that is calculated for each school based on items drawn from the school’s audited financial statements. The composite score—a metric for evaluating a school’s financial condition—uses a formula based on three financial ratios. A passing score is 1.5 to 3.0; a “zone” score is from 1.0 to 1.4, and a failing score is from -1.0 to 0.9. (See fig. 2) Schools that receive a zone or failing composite score, or do not meet one or more of the other financial responsibility standards, may continue to participate in federal student aid programs if they agree to additional oversight. Education may place these schools under heightened cash monitoring (increasing schools’ reporting requirements and postponing the timing for receiving federal student aid payments), or require schools to post a letter of credit (a financial commitment from a bank to protect Education against potential liabilities should the school close), or a combination of the two. Education may rescind a school’s ability to participate in federal student aid programs if a significant percentage of its borrowers—generally, 30 percent or more of borrowers for 3 consecutive years or more than 40 percent in 1 year—default on their federal student loans within the first 3 years of repayment. This calculation is called the cohort default rate. To compute a school’s cohort default rate, Education divides the number of student loan borrowers in a cohort—those entering repayment in the same fiscal year—who have defaulted on their loans in the initial 3 years of repayment by the total number of a school’s student loan borrowers in that cohort (see fig. 3). The cohort default rate does not hold schools accountable for borrowers who default after the initial 3 years. Borrowers in deferment and forbearance—options that allow borrowers to temporarily postpone monthly payments— are considered to be “in repayment” and current on their loans for the purpose of calculating a school’s cohort default rate, even though borrowers in these loan statuses are not expected to make any monthly payments. We have previously reported on a number of challenges with the accreditation system’s oversight of academic quality. Although Education is prohibited from specifying the specific content of accreditor standards, the agency is responsible for assessing whether accreditors are effectively overseeing schools’ academic quality as part of their criteria for recognizing accreditors. Our 2014 analysis found that schools with weaker student outcomes were, on average, no more likely to be sanctioned by accreditors than schools with stronger student outcomes, and that the proportion of their member schools that accreditors sanctioned varied. For example, our analysis of Education’s sanction data from October 2009 through March 2014 found that two accreditors sanctioned less than 2 percent of their member schools during this time frame, compared to 41 percent sanctioned by another accreditor. Our 2017 report also discussed challenges with the accreditation system’s oversight of academic quality. For example, some experts and literature stated that accreditors may be hesitant to terminate schools’ accreditation when they identify issues because such action would adversely affect schools’ eligibility for federal student aid programs. Despite inconsistencies in accreditors’ use of sanctions, our 2014 report found that Education did not systematically examine data on accreditor sanctions that could have helped it identify insufficient accreditor oversight and thereby reduce potential risk to students and federal funds. Accreditors provide Education with records of terminations and probations. However, Education officials told us that they had not used this sanction information for oversight of accreditors because Education’s regulations did not have specific criteria that require them to do so. While Education is not required to use sanction data or analyze accreditor sanctions as part of the accreditor recognition process, we found that it could be useful for Education to consider these data when evaluating whether accreditors meet prescribed criteria, such as whether they consistently apply and enforce standards. Federal internal control standards call for federal agencies to track data to help them make decisions, as well as conduct ongoing, consistent monitoring to identify weaknesses. Since accreditors are gatekeepers for tens of billions of dollars in federal student aid from Education, as well as the key oversight bodies for ensuring academic quality at schools, we found that failure on the part of Education to spot weaknesses in accreditors’ processes could result in poor quality schools gaining access to federal funds. To strengthen Education’s oversight of accreditors, we recommended in 2014 that Education draw upon accreditor data to determine whether accreditors are consistently applying and enforcing their standards to ensure that the education offered by schools is of sufficient quality. For example, Education could systematically use available information related to the frequency of accreditor sanctions or could do additional analyses, such as comparing accreditor sanction data with Education’s information on student outcomes, to inform its recognition reviews. Education agreed with this recommendation and initially started to track the number of accreditor sanctions issued by each accrediting agency. However, Education has since questioned the usefulness of this information and has not yet used this sanction data to inform its discussions of accreditor recognition and oversight. We continue to believe that implementing the recommendation could help inform Education’s reviews of accreditors and ultimately reduce potential risk to students and federal funds. For example, analyses of accreditor sanction data could help reveal patterns in individual accreditor behavior and the extent to which they are consistently enforcing standards. This recommendation remains open and we will continue to monitor Education’s efforts in this area. Holding schools accountable for their financial condition can help protect taxpayers and students against the risk of school closure, but the limitations of Education’s financial composite score hamper its effectiveness at identifying at-risk schools. Although a relatively small number of schools close each year, these closures can affect tens of thousands of students and result in hundreds of millions of dollars in financial losses for the federal government and taxpayers from unrepaid student loans. However, we reported in 2017 that Education’s composite score has been an imprecise predictor of school closures. Half the colleges that closed in school years 2010-11 through 2015-16 received passing financial composite scores on their last assessment before they closed. For example, 58 of the 96 schools that closed in school year 2015-16 had recently received passing scores. Closures can be difficult to predict in part because each school faces its own unique challenges, both financial and nonfinancial, that can eventually push it into financial trouble. Education’s composite score is not designed to account for nonfinancial risks; however, it is a primary means of securing financial protections in the form of a letter of credit from schools at risk of closure. The composite score’s inconsistent performance in identifying at-risk schools is due in part to limitations of the underlying formula and the fact that it has remained unchanged for more than 20 years. The composite score is based on common financial ratios that Education selected in 1997 after consulting with an accounting firm, school officials, and other experts. However, the composite score formula has not been updated since then and several experts and school officials we interviewed identified three key weaknesses: Accounting changes: The composite score has not kept pace with changes since 1997 in accounting practices and standards, creating ambiguity and making it more difficult to apply the formula in a uniform manner. Accounting practices and standards are periodically updated, for example, to improve the comparability and usefulness of financial reporting. When these updates diverge from the components and definitions in Education’s composite score, certain components of the composite score are no longer directly linked to items on schools’ audited financial statements. These accounting changes can also cause large shifts in schools’ composite scores. For example, administrators at one school we talked to said changes to state laws have affected how some schools categorize their endowment holdings in financial audits, and that this had the effect of reducing the school’s composite score from passing to not passing. However, Education has not updated the composite score formula to ensure the score is a reliable measure of financial health. Outdated financial measures: The composite score does not incorporate new financial metrics that would provide a broader indication of schools’ financial health. For more than 20 years, the composite score formula has remained unchanged as the field of financial analysis has continued to evolve with new measures becoming important as economic conditions change. For example, liquidity (i.e., access to cash) has become an important financial measure since the 2007-09 economic downturn, when some schools had trouble meeting payroll and fulfilling contractual obligations. More sophisticated methodologies used by credit rating agencies have sometimes resulted in assessments of a school’s financial condition that are strikingly different from the school’s composite score. For example, in 2016, two credit rating agencies assigned non-investment grade (i.e., junk bond) ratings to 30 schools that received passing composite scores from Education. Vulnerability to manipulation: We previously reported that the composite score can be manipulated by some schools that take on long-term debt (e.g., loans with terms in excess of 12 months) because these debts can increase a school’s composite score and help it avoid requirements to post a letter of credit. Long-term debt usually represents a long-term investment in a school’s campus and buildings, and the composite score formula treats this type of debt in a positive manner. An accountant for multiple schools told us that some schools have taken advantage of this provision and taken on a million dollars in debt in order to obtain a passing composite score. Corinthian Colleges, which closed in 2015, also exploited this vulnerability to boost its composite score and avoid having to post a letter of credit that could have been used by Education to cover some of the hundreds of millions in student loan discharges resulting from the school’s closure, according to company documents and Education documents and officials. These three weaknesses with the financial composite score hamper Education’s ability to effectively fulfill its statutory responsibility to determine whether schools participating in federal student aid programs are financially responsible. Identifying and responding to risks is a key component of federal internal control standards, but Education’s financial composite score formula has remained unchanged for over 20 years despite significant changes in the financial landscape of higher education. To address these limitations, we recommended in our 2017 report that Education update the composite score formula to better measure schools’ financial conditions and capture financial risks. Education generally disagreed with this recommendation and stated that the issues identified in our report did not necessarily mean that the composite score was an unreliable measure of schools’ financial strength. Since our report was issued, new regulations have gone into effect specifying that certain financially risky events, such as those related to litigation and certain accreditor actions, will generally trigger a recalculation of a school’s composite score. In addition, Education has also published proposed regulations that would update some of the definitions of terms used to calculate a school’s composite score to conform with changes in accounting standards and also make an adjustment to how the formula treats long-term debt, which according to Education would be intended to make the formula less susceptible to manipulation. However, Education has not finalized these regulations and has not released a timeline for when it plans to do so, nor has it indicated that it has any broader plans to update the composite score, as we recommended. Since the existing composite score calculation remains unchanged, we are leaving this recommendation open and will continue to monitor Education’s efforts in this area. The cohort default rate, which is specified in federal law, is a key measure for holding schools accountable for borrower outcomes and for protecting borrowers and the federal government from the costs associated with default. However, in 2018 we reported that this rate has limitations as an accountability tool. Some schools managed their 3- year cohort default rate by hiring consultants that encouraged borrowers with past-due payments to put their loans in forbearance, an option that allows borrowers to temporarily postpone payments and bring past-due loans current. At five of the nine default management consultants we selected (that served about 800 schools), we identified examples when forbearance was encouraged over other potentially more beneficial options for helping borrowers avoid default, such as repayment plans that base monthly payment amounts on income. Four of these consultants also provided inaccurate or incomplete information to borrowers about their repayment options in some instances. Although Education officials and student loan experts said that forbearance is intended to be a short-term option, our analysis of Education data found that 20 percent of borrowers who began repaying their loans in 2013 had loans in forbearance for 18 months or more during the 3-year cohort default rate period. Spending this much time in forbearance reduces the potential for borrowers to default within the 3- year period, thus helping improve a school’s cohort default rate. However, postponing loan payments through forbearance can increase borrowers’ loan costs in the long term. For example, a typical borrower with $30,000 in loans who spends the first 3 years of repayment in forbearance would pay an additional $6,742 in interest, a 17 percent increase, over the life of the loan. In addition, borrowers in forbearance for 18 months or longer defaulted more often in the fourth year of repayment, when schools are not accountable for defaults, than they did during the 3-year period. While forbearance can help borrowers avoid default in the short term, this finding suggests that forbearance may have delayed—not prevented— default, potentially resulting in increased costs to the federal government. Reducing the number of borrowers in long-term forbearance and directing them toward other options for avoiding default, such as repayment plans that base monthly payment amounts on income, could help reduce the number of borrowers that later default and may eventually save the federal government money. Specifically, for William D. Ford Federal Direct Loans issued in fiscal year 2018, Education estimates that it will not recover over 20 percent of defaulted loans. These unrecovered defaulted loan amounts total an estimated $4 billion, according to our analysis of Education’s budget data. Schools are seldom held accountable for their students’ defaults, in part because of the high rate of borrowers in long-term forbearance. To examine the impact of long-term forbearance on schools’ 3-year default rates, we recalculated schools’ cohort default rates by excluding borrowers who were in forbearance for 18 months or more and who did not default during the 3-year period. We found that over 260 additional schools—receiving a combined $2.7 billion in Direct Loans and Pell Grants in academic year 2016-2017—would potentially have had a default rate high enough to put them at risk of losing access to federal student aid programs. The reduced effectiveness of cohort default rates as a tool for holding schools accountable creates risks to the federal government and taxpayers, who are responsible for the costs associated with high rates of default. Since the way the cohort default rate is calculated is specified in federal law, any changes to its calculation would require legislation to be enacted amending the law. Our 2018 report suggested that Congress consider strengthening schools’ accountability for student loan defaults, for example, by revising the cohort default rate calculation or using other accountability measures to complement or replace the cohort default rate. In the 115th Congress, proposals were introduced to revise, supplement, or replace the cohort default rate, though none of the legislation was enacted. This matter for congressional consideration remains open. We continue to believe that strengthening the accountability measure for loan defaults could further protect borrowers and the billions of dollars of federal student aid the government distributes each year. In conclusion, the large federal investment in higher education makes it essential that the federal government maintain a robust system of accountability to protect students and taxpayers. My statement has highlighted three actions Education and Congress could take to strengthen the existing accountability tools for educational quality, financial sustainability, and student loan defaults. Students deserve to go to schools that provide a quality education and are financially stable. Taxpayers deserve an accountability system that protects federal student aid funds from going to schools that are financially irresponsible or push borrowers into forbearance for long periods in order to reduce the school’s cohort default rate. We believe that fully implementing the two recommendations and matter for congressional consideration discussed in this testimony would improve federal accountability, help students, and potentially lead to financial savings for taxpayers. Chairwoman Davis, Ranking Member Smucker, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Melissa Emrey-Arras, Director of Education, Workforce, and Income Security, at (617) 788-0534 or emreyarrasm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony include Debra Prescott (Assistant Director), Will Colvin (Analyst-in-Charge), and Brian Schwartz. In addition, key support was provided by Susan Aschoff, James Bennett, Deborah Bland, Marcia Carlsen, Alex Galuten, Sheila McCoy, Jessica Rider, and Walter Vance. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In fiscal year 2018, nearly 13 million students and their families received over $122 billion in federal assistance to help them pursue higher education through programs authorized under Title IV of the Higher Education Act of 1965, as amended. Education administers these programs, and is responsible, along with accreditors and states, for maintaining accountability and protecting the federal investment in student aid for higher education. This testimony summarizes the findings and recommendations from GAO's prior reports, issued between 2014 and 2018, examining Education's role in: (1) recognizing accrediting agencies, (2) overseeing the financial condition of schools, and (3) overseeing schools' student loan default rates. This statement also updates the status of selected recommendations and a matter for congressional consideration. GAO has identified opportunities to strengthen federal higher education accountability in three areas: educational quality, financial stability, and federal student loan defaults. Educational quality. Accreditors—independent agencies responsible for ensuring that schools provide a quality education—must be recognized by the Department of Education (Education) as reliable authorities on educational quality. The accreditors can issue sanctions, including terminations and probations, to schools that do not meet accreditor standards. However, GAO previously found that schools with weaker student outcomes were, on average, no more likely to be sanctioned by accreditors than schools with stronger student outcomes, and Education does not make consistent use of sanction data that could help it identify insufficient accreditor oversight. In 2014, GAO recommended that Education use accreditor data in its recognition review process to determine whether accreditors are consistently applying and enforcing their standards to ensure schools provide a quality education. Education agreed with the recommendation, but has yet to use this data in this manner. Financial stability. Education uses a financial composite score to measure the financial health of schools participating in federal student aid programs, and increases its oversight of schools when it identifies concerns to protect against the risk of school closures. School closures, although rare, can result in hundreds of millions of dollars in unrepaid federal student loans and displacement of thousands of students. However, the composite score has been an imprecise risk measure, predicting only half of closures from school years 2010-11 through 2015-16. This is due in part to the fact that the composite score does not reflect changes in accounting practices and standards, relies on outdated financial measures, and is vulnerable to manipulation. Despite these limitations, Education has not updated the composite score since it was first established more than 20 years ago. In 2017, GAO recommended that Education update its financial composite score. Education has proposed some revisions, but changes have not yet been implemented to protect students and taxpayers against financial risks. Student loan defaults. According to federal law, schools may lose their ability to participate in federal student aid programs if a significant percentage of their borrowers default on their student loans within the first 3 years of repayment. However, GAO previously found that some schools managed these default rates by hiring consultants that encouraged borrowers with past-due payments to put their loans in forbearance, an option that allows borrowers to temporarily postpone payments and bring past due loans current. Although Education officials and student loan experts said forbearance is intended to be a short-term option, GAO's analysis of Education data found that 20 percent of borrowers who began repaying their loans in 2013 had loans in forbearance for 18 months or more. These borrowers defaulted more often in the fourth year of repayment, when schools are not accountable for defaults, suggesting long term forbearance may have delayed—not prevented—default. In 2018, GAO suggested that Congress consider statutory changes to strengthen schools' accountability for student loan defaults. Legislation has not yet been enacted.", "document_type": "gao"}
{"report": "Black lung benefit payments include both cash assistance and medical care. Maximum cash assistance payments ranged from about $670 to $1,340 per month in 2019, depending on a beneficiary’s number of dependents. Miners receiving cash assistance are also eligible for medical treatment of their black lung-related conditions, which may include hospital and nursing care, rehabilitation services, and reimbursement for drug and equipment expenses, according to DOL documentation. DOL estimates that the average annual cost for medical care in fiscal year 2019 was approximately $8,225 per miner. During fiscal year 2019, about 25,700 beneficiaries received black lung benefits (see fig. 1). The number of beneficiaries has decreased from about 174,000 in 1982 as a result of declining coal mining employment and an aging beneficiary population, according to DOL. Black lung beneficiaries could increase in the near term due to the rise in the occurrence of the disease in its most severe form, progressive massive fibrosis, particularly among Appalachian coal miners, according to the National Institute for Occupational Safety and Health (NIOSH). NIOSH reported that coal miners in central Appalachia are disproportionately affected; as many as 1 in 5 show evidence of black lung, which is the highest level recorded in 25 years. NIOSH has attributed the rise in occurrence of black lung to multiple factors, including increased exposure to silica. Black lung claims are processed by the Office of Workers’ Compensation Programs within DOL. Contested claims are adjudicated by DOL’s Office of Administrative Law Judges, which issues decisions that can be appealed to DOL’s Benefits Review Board. Claimants and mine operators may further appeal these DOL decisions to the federal courts. If an award is contested, claimants can receive interim benefits until their case is resolved, which are generally paid from the Trust Fund, according to DOL. In fiscal year 2019, about 33 percent of black lung claims were approved, according to DOL data. Final awards are either funded by mine operators—who are identified as the responsible employers of claimants—or the Trust Fund, when responsible employers cannot be identified or do not pay. Of the approximately 25,700 beneficiaries receiving black lung benefits in 2019, 13,335 were paid from the Trust Fund; 7,985 were paid by responsible mine operators; and 4,380 were receiving interim benefits, according to DOL data. DOL officials told us that the more common reasons that beneficiary claims are paid from the Trust Fund include operator insolvency and unclear employment history of miners, among other reasons (see fig. 2). The operator responsible for the payment of benefits is generally the operator that most recently employed the miner. Federal law generally requires coal mine operators to secure their black lung benefit liability. A self-insured coal mine operator assumes the financial responsibility for providing black lung benefits to its eligible employees by paying claims as they are incurred. Operators are allowed to self-insure if they meet certain DOL conditions. For instance, operators applying to self-insure must obtain collateral in the form of an indemnity bond, deposit or trust, or letter of credit in an amount deemed necessary and sufficient by DOL to secure their liability. Operators that do not self-insure are generally required to purchase coverage from commercial insurance companies, state workers’ compensation insurance funds, or other entities authorized under state law to insure workers’ compensation. DOL regulations require commercial insurers to report each policy and federal black lung endorsement issued, canceled, or renewed in a form determined by DOL. DOL accepts electronic reporting of this information from insurers via their respective rating bureaus. DOL retains this information— insured company name, address, federal employer identification number, and policy and endorsement data—so that DOL staff can later research claims to determine which operator and insurer may be liable. As we have noted in prior reports, insurance companies are regulated primarily by the states with state law providing state regulators with the authority and funding to regulate insurance. State insurance regulation is designed to, among other things, help insurers remain solvent and able to pay claims when due. Effective insurer underwriting and risk management practices—such as reinsurance–serve a similar function. While insurer insolvency occurs infrequently, when it does state insurance commissioners are typically appointed as receiver and supervise the rehabilitation or liquidation of these insurers, and state guaranty funds may assume liability for paying covered claims of insolvent insurers that have liquidated. Of the eight coal mine operator bankruptcies we identified, three resulted in a transfer of estimated benefit liability from the coal operator to the Trust Fund and five did not, according to DOL. Using Bloomberg data, we identified coal mine operators that filed for bankruptcy from 2014 through 2016. Figure 3 shows how many operators were self-insured or commercially-insured at the time of bankruptcy, and if responsibility for benefits was shifted from the bankrupt operator to the Trust Fund. Three self-insured coal mine operator bankruptcies affected the Trust Fund. Specifically, the bankruptcies of Alpha Natural Resources (Alpha), James River Coal (James River), and Patriot Coal (Patriot) resulted in a transfer of benefit liability to the Trust Fund of an estimated $865 million, according to DOL. DOL officials said that the amount of collateral they required from these three operators to self-insure was inadequate to fully cover their estimated benefit liability. When this occurs, benefit liability in excess of the collateral can be transferred to the Trust Fund. For example, the collateral DOL required from Alpha was about $12 million and approximately $494 million of estimated benefit liability transferred to the Trust Fund, according to DOL’s estimate (see table 1). DOL estimates for how these three operator bankruptcies will affect the Trust Fund have more than doubled from what DOL had previously reported. In June 2019, we reported that DOL estimated that between $313 million to $325 million in benefit liabilities would transfer to the Trust Fund as a result of these bankruptcies. In January 2020, however, DOL provided updated estimates stating that $865 million in benefit liabilities would transfer to the Trust Fund as a result of these bankruptcies. According to DOL, their estimates increased to account for higher black lung benefit award rates that occurred from fiscal years 2016 through 2019; higher medical treatment cost inflation in recent years; and different discount rate assumptions. Additionally, DOL’s prior estimate for the Patriot bankruptcy did not account for future claims and the effect of those claims on the Trust Fund. The three other self-insured coal mine operator bankruptcies we identified did not affect the Trust Fund. Specifically, Arch Coal, Peabody Energy, and Walter Energy were also self-insured operators, but DOL officials said that their federal black lung benefit liabilities were assumed by a reorganized company or by a purchaser, and therefore did not transfer to the Trust Fund. DOL officials said that they take three key actions, as appropriate, to protect the financial interests of the Trust Fund during self-insured operator bankruptcies. 1. DOL officials said that they file a claim in every case with the bankruptcy court for the reimbursement of an operator’s full estimated federal black lung benefit liability. 2. If an operator plans to reorganize or if it is acquired by a purchaser, DOL officials said that they negotiate with the company or the purchaser, as appropriate, to help ensure benefit responsibility will be “passed through” to a reorganized operator or purchaser, rather than be discharged and become the responsibility of the Trust Fund. 3. If benefit liabilities are not “passed-through” to an operator, DOL officials said that they seek settlement agreements, whereby the Trust Fund receives an allowed general unsecured claim in an amount based on an operator’s estimated benefit liability. DOL officials said that during the bankruptcy of James River they negotiated a settlement agreement providing DOL with a general unsecured claim in an amount commensurate with its estimate of the operator’s benefit liability at the time of bankruptcy. However, these officials said that given the low priority under bankruptcy law for their general unsecured claim, the payout they received was only about $400,000, which was just a small portion of the estimated benefit liability that transferred to the Trust Fund. DOL officials said that during the bankruptcy of Alpha they negotiated both a “pass through” and a settlement agreement in which certain liabilities would be transferred to the Trust Fund, while other liabilities would be retained by Alpha. DOL officials said that they received a payout from Alpha of $7.4 million, although $494 million in estimated benefit liability transferred to the Trust Fund. Further, as a condition of the agreement, DOL officials said that they agreed to let Alpha self-insure after it emerged from bankruptcy. Since 2016, several other self-insured operators have also filed for bankruptcy, according to DOL officials, including Cambrian Coal, Cloud Peak Energy, Murray Energy, and Westmoreland Coal. DOL officials said that $17.4 million in estimated black lung benefit liability will transfer to the Trust Fund as a result of Westmoreland Coal’s bankruptcy. Given the uncertainty of the bankruptcy process in terms of whether liabilities will or will not transfer to the Trust Fund, however, DOL officials said that they could not speculate on how the other bankruptcies may affect the Trust Fund. Insurance contracts or policies to secure operators’ benefit liabilities are required by law to include a provision that insolvency or bankruptcy of an operator does not release the insurer from the obligation to make benefit payments. As previously discussed, state insurance regulation, insurer underwriting and risk management practices, and state guaranty funds also help to protect the Trust Fund from having to assume responsibility for paying black lung benefits on behalf of bankrupt coal operators. Thus, by being commercially insured, the two operator bankruptcies we identified that filed for bankruptcy between 2014 and 2016—Energy Future Holdings and Xinergy Ltd—did not affect the Trust Fund, according to DOL (see fig 3). State insurance commissioners monitor the financial health of insurers, including performing periodic examination of insurer financial statements. Further, rating agencies, such as Standard & Poor’s, Moody’s, and AM Best, issue insurer financial strength ratings, which represent the agencies’ opinions on insurers’ financial strength and ability to pay policy and contract obligations. Eight of the nine insurers that issued approximately 90 percent of the workers’ compensation policies with federal black lung coverage from 2016 through 2018, according to our review of DOL data, had at least an “A-” financial strength rating from AM Best (with the one remaining being a state insurer that was not rated). In deciding whether to provide federal black lung coverage, insurers we interviewed said they consider an operator’s historical black lung claim losses, financial condition, and mine location among other factors. However, insurance company officials identified various challenges in writing and pricing black lung coverage that produces an appropriate amount of premiums to cover expected losses. The challenges cited by these officials included the long latency period of black lung disease; changes in law regarding benefit eligibility and how the disease is defined; the ability of miners to refile claims indefinitely; and the inability of insurers and operators to settle claims. One official noted that there is much risk and little profit in black lung coverage. Insurance companies can use reinsurance to protect themselves from catastrophic losses that could threaten their solvency and ability to pay claims, and to reduce wide fluctuations in their annual losses. For example, workers’ compensation claims can take years to fully develop after premiums have been set, which in turn can adversely affect an insurer’s financial position if premiums have underestimated actual claims. Insurance company officials said that they reinsure their workers’ compensation coverage, but some said that their reinsurance policies either explicitly excluded occupational disease claims, including black lung, or cover black lung but have conditions and loss thresholds that would generally result in the exclusion of such claims. However, reinsurance, even if it does not explicitly cover federal black lung claims, can help manage the risk of workers’ compensation losses and losses in other lines of insurance that an insurer writes, thereby indirectly helping to ensure that the insurer can pay all types of claims, including federal black lung. If an insurer becomes insolvent, state guaranty funds reduce the potential for the Trust Fund to assume responsibility for paying claims. States have different rules for guaranty fund benefit coverage and limits. In the states we reviewed, state guaranty funds generally pay federal black lung benefits, although there may be certain limitations on the claims they will pay. For example, in West Virginia, there is no maximum claim limit that the state guaranty fund will pay on standard workers’ compensation claims; but in Kentucky, a state guaranty fund official told us that, in the guaranty fund’s opinion, state law limits federal black lung claims to $300,000. Also, a guaranty fund could reject a federal black lung claim, which could result in the Trust Fund having to assume responsibility for paying the claim. An official from one state guaranty fund that maintained data on rejected black lung claims said that the most common reason for rejection is that claims are filed after the date set by the bankruptcy court for receiving claims. DOL officials said it is very uncommon for the Trust Fund to assume responsibility for federal black lung claims of insolvent insurers. However, DOL does not maintain data to readily determine the extent to which this actually occurs, as discussed later in this report. In overseeing coal mine operator self-insurance in the past, DOL did not estimate future benefit liability when setting collateral; regularly review operators to monitor their changing financial conditions; or always use enforcement tools available to protect the financial interests of the Trust Fund, such as by revoking an operator’s ability to self-insure, if warranted. In July 2019, DOL began implementing a new self-insurance process that, if implemented effectively, should help to address some of these past deficiencies. Specifically, DOL plans to consider an operator’s future benefit liability when setting collateral and to review self-insured operators more frequently. However, the new process still lacks procedures for self- insurance renewals and coal operator appeals, which could hinder DOL from taking enforcement actions to protect the Trust Fund as needed. Additionally, DOL does not monitor whether operators that do not self- insure maintain adequate and continuous commercial insurance coverage as required by law. Agency regulations require DOL to obtain collateral from coal mine operators applying to self-insure in an amount deemed by DOL to be necessary and sufficient to secure the payment of the operators’ liability. To determine collateral amounts under the former process, agency procedures stated that DOL first assess an operator’s net worth by reviewing, among other factors, the operator’s audited financial statement and black lung claims information. DOL then determined the amount of collateral equal to 3, 5, or 10 years of the operator’s annual black lung benefit payments made at the time of the operator’s self-insurance application, depending on its net worth. Specifically, if net worth was $1 billion or greater, agency procedures stated that DOL set collateral equal to 3 years of benefit payments. If net worth ranged from $500 million to $1 billion, DOL set collateral equal to 5 years of benefit payments. If net worth ranged from $10 million to $500 million, DOL set collateral equal to 10 years of benefit payments. Agency procedures did not permit operators with net worth less than $10 million to self-insure. DOL’s former process for determining collateral did not routinely consider potential future claims for which an operator could be responsible. DOL had periodically reauthorized coal operators to self-insure, by reviewing an operator’s most recent audited financial statement and claims information, among other things. DOL prepared memos documenting these reviews and communicated with coal operators about whether their financial circumstances warranted increasing or decreasing their collateral. Estimating future costs based on sound actuarial practice is essential to the integrity of the insurance and the risk financing system and is key to fulfilling the promises embodied in insurance contracts, according to Actuarial Standards Board standards. Additionally, in three of the four states we contacted, state insurance officials said that they used actuarial methods to assess an operator’s future estimated benefit liability when considering how much collateral should be required to self- insure. The remaining state, Wyoming, did not allow coal mine operators to self-insure. Table 2 provides information on the 22 operators that were self-insured under DOL’s former process, including the date of each operator’s last DOL reauthorization; the amount of DOL-required collateral; and the operator’s estimated black lung benefit liability, if available. Agency regulations state that DOL may adjust the amount of collateral required from self-insured operators when experience or changed conditions so warrant, but DOL did not regularly monitor these operators to reauthorize their ability to self-insure. In reviewing DOL’s most recent reauthorization memos for each of the 22 self-insured operators, we found that while some of these operators had been reauthorized more recently, others had not been reauthorized by DOL in decades. One operator in particular had not been reauthorized by DOL since 1988. DOL officials stated that from 2009 to 2012, six employees handled coal operator reauthorizations and associated work actions. Due to attrition, however, this number dropped at times to three employees, according to DOL officials. Additionally, DOL had no written procedures that specified how often reauthorizations should occur after an operator’s initial 18- month reauthorization. In contrast, in two of the four states we contacted, state insurance officials were required to review self-insured employers at least annually. Revoking an operator’s ability to self-insure, fining mine operators for operating without insurance, and placing liens on operator assets are tools DOL has available to mitigate financial losses to the Trust Fund. Based on our review of DOL documentation, however, we found instances when DOL did not use these tools to protect the Trust Fund, or was hindered from doing so because of an operator’s ongoing appeal or bankruptcy. In September 2001, DOL required $5 million in additional collateral from James River, which would have increased its collateral from $0.4 million to $5.4 million. Although DOL did not receive the additional collateral, it did not revoke the operator’s authority to self-insure, which is a potential option under agency regulations. Further, DOL had not reauthorized James River at any point from August 2001 until it filed for bankruptcy in April 2014. If DOL had revoked James River’s ability to self-insure, it could have potentially prevented the Trust Fund from being responsible for claims based on a miner’s employment from 2001 through 2016, when James River liquidated. Additionally, if the operator had been unable to obtain commercial insurance, DOL could have potentially fined the operator for each day it operated without insurance. Instead, DOL took no action during these years and estimated benefit liability of $141 million was shifted to the Trust Fund, according to DOL. DOL officials stated that they do not have records explaining why James River did not provide the additional collateral or why they did not revoke its authority to self-insure. In August 2014, DOL required $65 million in collateral from Patriot, increasing its collateral from $15 million to $80 million. Patriot appealed this decision and, in the 8 months that followed before Patriot filed for bankruptcy in May 2015, DOL did not obtain additional collateral, or revoke Patriot’s ability to self-insure because the appeal was still pending. DOL officials said they would not typically revoke an operator’s authority to self-insure during an ongoing appeal. As a result, DOL was hindered from using this enforcement tool. Liens on operator assets can be an effective tool to protect the Trust Fund if an operator defaults on its benefit liabilities, but DOL officials said that they are hindered from using this tool if an operator files for bankruptcy. DOL can place a lien on a coal operator’s assets under federal law if they refuse the demand to pay the black lung benefit payments for which they are liable. In the event of bankruptcy or insolvency, federal law states that the lien imposed shall be treated in the same manner as a lien for taxes due and owing to the United States under certain laws. However, DOL officials said that operators rarely stop paying benefits until after they file for bankruptcy. Once a bankruptcy occurs, DOL officials said that they are generally prevented by the court from placing a lien and taking an operator’s assets in lieu of payment of current and future benefit liabilities. Under bankruptcy law, DOL officials said that they have no special status over other creditors with outstanding financial claims. Instead, DOL officials said that obtaining sufficient collateral is a better way to protect the Trust Fund. In July 2019, DOL began implementing a new process for coal mine operator self-insurance that should help to address some past deficiencies if implemented effectively. Specifically, DOL is to consider an operator’s future benefit liability when setting collateral and plans to more frequently review self-insured operators (see text boxes). Under the new process, DOL officials plan to assess the risk of operator bankruptcy using various financial metrics related to profitability and solvency. As a result, DOL officials said that the amount of collateral they will require from operators to self-insure going forward will be based on both an estimate of an operator’s current and future black lung liability and the risk of default due to insolvency. As of October 2019, DOL officials said that most self-insured operators had submitted their application and supporting documentation and that they were reviewing this information to decide whether these operators should continue to be self-insured. DOL’s New Self-Insurance Process Will Include Estimates of Future Benefit Liability Coal mine operators applying to DOL to self-insure will be required to submit: a completed application; a certified consolidated financial statement for each of the 3 years prior to its application; recent black lung claims information; and a certified actuarial report on the operator’s existing and future black lung benefit liabilities. DOL plans to use the information submitted by coal mine operators to assess the insolvency risk of each operator using various financial metrics related to profitability and solvency. Depending on the results of their analysis, DOL plans to categorize the risk-level of each applicant as low, medium, or high. DOL will then set the amount of collateral required to self-insure by linking the operator’s risk category to a corresponding percentage of the operator’s actuarial estimated benefit liability. DOL policies state that they would require a high-risk operator to secure with collateral 90 percent of estimated benefit liability, a medium-risk operator to secure 45 percent, and a low-risk operator to secure 15 percent. However, in February 2020, DOL officials said they plan to revise these percentages to 100 percent, 85 percent, and 70 percent for high-risk, medium-risk, and low-risk operators, respectively. DOL’s New Self-Insurance Process Will Require More Frequent Coal Mine Operator Reviews Coal mine operators that are already authorized to self-insure will be required to submit: DOL plans to use the information self-insured operators submit to update their insolvency risk analysis. If an operator’s risk category changes (e.g., from low-to medium-risk), DOL plans to send a form to the operator requiring an additional amount or type of collateral. Upon receiving the completed form, and proof that the collateral has been obtained, DOL stated that they will notify the operator that its a self-insurance renewal application (annually); a financial summary (quarterly); a certified consolidated financial statement (annually); black lung claims information (annually); and actuarial estimate of benefit liability (to be submitted every three years). authority to self-insure has been reauthorized. DOL’s new self-insurance process made important changes, but overlooked other key internal control improvements that are needed to protect the financial interests of the Trust Fund. DOL’s new requirements for setting collateral and for the annual and quarterly review of self- insured operators are key components of internal controls, which call for agency management to implement control activities through policy. However, DOL’s new self-insurance process lacks procedures that could help to prevent past oversight deficiencies from reoccurring. Among other things, DOL’s procedures do not specify (1) the duration of an operator’s self-insurance authority, (2) the time frames for submitting renewal applications and supporting documentation, and (3) the conditions under which an operator’s self-insurance authority would not be renewed. Without such procedures, DOL has no basis to take enforcement action should an operator not submit its self-insurance renewal application and supporting documentation. DOL staff are also hindered from taking enforcement action during an operator’s ongoing appeal, as previously mentioned. DOL policies state that an operator may request reconsideration if its self-insurance application has been denied or if it believes the collateral required by DOL is too high to secure its benefit liabilities. However, DOL lacks procedures that specify, among other things, the length of time that operators have to submit supporting information. Further, DOL does not specify a goal for how much time DOL appeals decisions should take. For example, in October 2015, DOL recommended revoking Murray Energy’s (Murray) authority to self-insure due to deteriorating financial conditions. Murray appealed this decision, and DOL officials said they postponed responding to the appeal until their new self-insurance process was implemented so that they could evaluate Murray under its new process along with the other self-insured operators. However, Murray filed for bankruptcy in October 2019 and DOL had not revoked its authority to self-insure or requested additional collateral because Murray’s appeal was still pending and DOL was still evaluating how much collateral it would require from the operator under its new self-insurance process. DOL does not monitor coal mine operators that do not self-insure and, thus, must commercially insure their federal black lung liabilities to make certain they maintain adequate and continuous coverage as required by law. DOL previously monitored operators' compliance with the program's insurance requirements by annually sending letters to a selection of operators seeking confirmation that they had maintained adequate coverage, but discontinued the process once the agency began receiving NCCI policy data. In order to use the policy data for the purpose of identifying operators that have not maintained coverage, DOL would, as a starting point, have to maintain a record of all employers that operate a coal mine. However, DOL officials explained that they do not currently maintain such a record. In the absence of effective DOL monitoring of operator compliance, we evaluated the potential risk that uninsured operators could pose to the Trust Fund. Specifically, in examining the 13 largest coal operators that were not approved to self-insure their federal black lung liabilities and, therefore, had to obtain commercial coverage, we found that some insurers erred in reporting endorsements and in one instance an operator did not have adequate coverage. We found six operators (parent or subsidiary) that were not insured for the entire 3 year period from 2016 through 2018, according to our review of DOL data. When we discussed our findings with DOL, agency officials had to research each operator individually and in some cases contact the operator or their insurer to find out whether or not they had been covered. DOL concluded that these entities were insured. However, the insurers had not properly reported the federal black lung endorsement on new policies or subsequent renewals, in addition to other reporting issues. One of these six operators also had, inadvertently, not maintained adequate commercial coverage for its mining operations in Texas, and had not self-insured those operations. In this instance, the operator obtained an excess loss policy that only pays claims once they exceed a high threshold and, therefore, is not sufficient by itself to secure the payment of the operator’s benefit liabilities. DOL data does not include information on excess loss policies and, while the data NCCI provides on standard workers’ compensation policies with federal black lung endorsements lists operators’ addresses, they do not provide the specific states for which endorsements apply. Designing processes to achieve agency objectives and respond to risks is a principle of effective internal controls. Without a process to monitor operator compliance with program insurance requirements, DOL risks not identifying a lapse or cancellation of operator coverage. This could result in the Trust Fund having to assume responsibility for paying benefits that would otherwise have been paid by an insurer. DOL officials said the Trust Fund infrequently pays claims on behalf of uninsured operators due to the civil penalties that it can impose on operators and certain company officers. These officials also said that operators that do not maintain insurance coverage typically employ few miners and are out of business by the time a claim is filed and, thus, cannot be held liable for benefit claims. However, DOL officials acknowledged that they do not track how often claims are paid by the Trust Fund on behalf of uninsured operators that should have been insured. We attempted to examine the extent to which claims were paid by the Trust Fund in fiscal year 2018 on behalf of uninsured operators that should have been insured. We found that DOL’s black lung claimant and payment system does not identify whether potentially responsible operators should have had commercial insurance coverage. The data on responsible operators and insurers, as well as the basis on which an operator was determined to be responsible, were not consistently recorded. DOL officials said that the data fields that identify responsible operators and their insurers should reflect the information collected from DOL’s initial determination. DOL officials said that in some cases, after an adjudication decision determined the Trust Fund was responsible for paying benefits, claim examiners may have deleted the previously recorded responsible operator and insurer data, creating potential inconsistencies in the data. DOL officials acknowledged that its processes and guidance for recording information on responsible operators and the basis for those decisions resulted in inconsistent and potentially inaccurate recording of claim and benefit data. As a result, DOL issued preliminary guidance in February 2019 to field supervisors and claims examiners. However, the revised guidance does not include how to identify potentially responsible operators that should have had commercial coverage but did not. Monitoring agency internal control systems and evaluating the results of those activities is a principle of effective internal control. Without complete and consistently recorded information on potentially responsible operators and insurers, and the basis for determination decisions, DOL is not able to effectively evaluate the financial impact claims paid on behalf of uninsured operators have on the Trust Fund. Determining the financial impact of these claims would be important to DOL’s evaluation of the effectiveness of a process for monitoring operator compliance with black lung program insurance requirements. The Black Lung Disability Trust Fund faces financial challenges, and DOL’s limited oversight of coal mine operator insurance has further strained Trust Fund finances by allowing operator liabilities to transfer to the federal government. DOL’s new self-insurance process may help to address past deficiencies in setting collateral and reviewing self-insured operators if implemented effectively. However, DOL still lacks procedures on self-insurance renewals and coal operator appeals that could help to ensure that DOL staff will take enforcement actions when needed. Establishing clear self-insurance renewal procedures could better position DOL to take action to protect the Trust Fund should an operator not submit its renewal application and supporting documentation, or comply with DOL’s collateral requirements. Procedures that identify time lines for self-insured operators to submit documentation supporting their appeals, and that identify a goal for how much time DOL should take to make appeals decisions could help to ensure that DOL is able to revoke an operator’s ability to self-insure, when warranted. Commercially-insured federal black lung liabilities can limit the Trust Fund’s exposure to financial risk, but only if operators maintain adequate and continuous coverage as required. Currently, DOL does not identify lapses or cancellations in coverage among commercially-insured operators until after a claim is filed. Establishing a process to identify lapses and cancellations in coverage before claims get filed could help prevent the Trust Fund from becoming responsible for these claims. We are making the following three recommendations to the Department of Labor: The Director of the Office of Workers’ Compensation Programs should develop and implement procedures for coal mine operator self- insurance renewal that clarifies how long an operator is authorized to self-insure; when an operator must submit its renewal application and supporting documentation; and the conditions under which an operator’s self-insurance authority would not be renewed. (Recommendation 1) The Director of the Office of Workers’ Compensation Programs should develop and implement procedures for self-insured coal mine operator appeals that identify time lines for self-insured operators to submit documentation supporting their appeals and that identify a goal for how much time DOL should take to make appeals decisions. (Recommendation 2) The Director of the Office of Workers’ Compensation Programs should develop and implement a process to monitor operator compliance with commercial insurance requirements and periodically evaluate the effectiveness of this process. This process should be designed to detect errors and omissions in reporting insurance coverage using complete, accurate, and consistently recorded data. (Recommendation 3) We provided a draft of this report to the Department of Labor (DOL) for review and comment. Their written comments are reproduced in appendix I. DOL also provided technical comments and clarifications, which we have incorporated, as appropriate. DOL agreed with our three recommendations and said it is acting to implement them to achieve further improvements in ensuring the effective oversight of coal mine operator insurance. DOL acknowledged the importance of improving oversight of coal mine operator insurance and commented that it made major oversight improvements in recent years. DOL commented that it began developing a new coal mine operator self-insurance process in 2015, before GAO began its review, and DOL formally approved this process in 2017. In July 2019, DOL stated that its new process was finalized when the Office of Management and Budget (OMB) approved the forms to collect financial and other information from coal mine operators. DOL stated that it is now reviewing information obtained from coal mine operators, and expects to set the amount of collateral required to self-insure under its new process in the first half of 2020. We commend DOL’s efforts to address the deficiencies of its past self-insurance process. However, we remain concerned about continuing coal operator bankruptcies and the looming unsecured black lung benefit liabilities that still threaten the Trust Fund. DOL commented that adopting GAO’s recommendations would further improve its oversight of coal mine operator insurance going forward. Specifically, DOL reported that it will (1) ensure letters granting or renewing self-insurance authority will inform operators that their authorization expires in one year and that they must submit renewal information three months in advance of the expiration date, (2) ensure letters denying self-insurance will inform operators that they have a 30- day appeal period (limited to one extension) and that DOL has set a goal of resolving all appeals within 90 days of the denial letter, and (3) modify existing computer systems to identify lapses or cancellations of commercial insurance coverage, and require operators identified as having lapsed or cancelled coverage to obtain or provide proof of coverage within 30 days. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Labor, and other interested parties. In addition, the report will be available at no charge on GAO’s web site at http://www.gao.gov. If you or your staffs should have any questions about this report, please contact Cindy Brown Barnes at (202) 512-7215 or brownbarnesc@gao.gov, or Alicia Puente Cackley at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contacts named above, Blake Ainsworth (Assistant Director), Patrick Ward (Assistant Director), Justin Dunleavy (Analyst-in- Charge), Monika Gomez, Courtney LaFountain, Rosemary Torres Lerma, and Scott McNulty made key contributions to this report. Also contributing to this report were James Bennett, Nancy Cosentino, Caitlin Cusati, John Forrester, Alex Galuten, Ellie Klein, Emei Li, Corinna Nicolaou, Almeta Spencer, Curtia Taylor, and Shana Wallace.", "summary": "In May 2018, GAO reported that the Trust Fund, which pays benefits to certain coal miners, faced financial challenges. The Trust Fund has borrowed from the U.S. Treasury's general fund almost every year since 1979 to make needed expenditures. GAO's June 2019 testimony included preliminary observations that coal operator bankruptcies were further straining Trust Fund finances because, in some cases, benefit responsibility was transferred to the Trust Fund. This report examines (1) how coal mine operator bankruptcies have affected the Trust Fund, and (2) how DOL managed coal mine operator insurance to limit financial risk to the Trust Fund. GAO identified coal operators that filed for bankruptcy from 2014 through 2016 using Bloomberg data. GAO selected these years, in part, because bankruptcies were more likely to be resolved so that their effects on the Trust Fund could be assessed. GAO analyzed information on commercially-insured and self-insured coal operators, and examined workers' compensation insurance practices in four of the nation's top five coal producing states. GAO also interviewed DOL officials, coal mine operators, and insurance company representatives, among others. Coal mine operator bankruptcies have led to the transfer of about $865 million in estimated benefit responsibility to the federal government's Black Lung Disability Trust Fund (Trust Fund), according to DOL estimates. The Trust Fund pays benefits when no responsible operator is identified, or when the liable operator does not pay. GAO previously testified in June 2019 that it had identified three bankrupt, self-insured operators for which benefit responsibility was transferred to the Trust Fund. Since that time, DOL's estimate of the transferred benefit responsibility has grown—from a prior range of $313 million to $325 million to the more recent $865 million estimate provided to GAO in January 2020. According to DOL, this escalation was due, in part, to recent increases in black lung benefit award rates and higher medical treatment costs, and to an underestimate of Patriot Coal's future benefit claims. DOL's limited oversight of coal mine operator insurance has exposed the Trust Fund to financial risk, though recent changes, if implemented effectively, can help address these risks. In overseeing self-insurance in the past, DOL did not estimate future benefit liability when setting the amount of collateral required to self-insure; regularly review operators to assess whether the required amount of collateral should change; or always take action to protect the Trust Fund by revoking an operator's ability to self-insure as appropriate. In July 2019, DOL began implementing a new self-insurance process that could help address past deficiencies in estimating collateral and regularly reviewing self-insured operators. However, DOL's new process still lacks procedures for its planned annual renewal of self-insured operators and for resolving coal operator appeals should operators dispute DOL collateral requirements. This could hinder DOL from revoking an operator's ability to self-insure should they not comply with DOL requirements. Further, for those operators that do not self-insure, DOL does not monitor them to ensure they maintain adequate and continuous commercial coverage as appropriate. As a result, the Trust Fund may in some instances assume responsibility for paying benefits that otherwise would have been paid by an insurer. GAO is making three recommendations to DOL to establish procedures for self-insurance renewals and coal operator appeals, and to develop a process to monitor whether commercially-insured operators maintain adequate and continuous coverage. DOL agreed with our recommendations.", "document_type": "gao"}
{"report": "Throughout the course of a construction project, small and large contract changes can be expected after the contract is awarded. These changes are made through modifications to a contract. There are two types of contract changes discussed in this report: bilateral and unilateral. Bilateral change. A bilateral change (also called a supplemental agreement) is a contract modification that is signed by the contractor and the contracting officer. In these cases, the contractor and contracting officer come to an agreement on the price of a contract change prior to the execution of work. Unilateral change. The contracting officer may direct a unilateral change, executed through a change order, without the contractor’s agreement on the terms and conditions of the change. A unilateral contract modification is signed only by the contracting officer. The contractor is generally required to perform the related work. When change orders do not include an agreed-upon price for the work, they may also be referred to as an unpriced change. If a contract change causes an increase or decrease to the cost of performing the work or the scheduled time for performing the work, the contractor will communicate these price and schedule changes to the contracting officer. For there to be an adjustment to the contract’s price, the contractor must submit a specific request or proposal seeking reimbursement for the change. If the contract change has been ordered unilaterally by the government, the contractor may submit a request for equitable adjustment (REA) that reflects these cost and schedule changes and requests reimbursement. In other circumstances, the contractor may submit a proposal in response to a request by the agency that similarly reflects the contractor’s estimate for that increased or decreased cost and the schedule changes. Bilateral and unilateral contract changes typically begin with a similar set of activities, but then the processes diverge once the bilateral or unilateral determination is made. Initial process steps include: identifying the need for a change; determining that the change is within the scope of the existing receiving a cost estimate; and verifying that funds are available for the change. It is generally after this point that the contracting officer determines the type of change—unilateral or bilateral—required. See figure 1 for a notional representation of a change process. Individual contract changes may involve circumstances and process steps that are not outlined below. Agency regulations and policies provide additional direction for managing the construction contract change process (see table 1). In prior work at the Department of Veterans Affairs (VA), we identified challenges and made several recommendations related to the time required for the construction contract modification process. In 2013, we found that VA had not developed guidance to ensure that change orders were approved in a prompt manner, and recommended that officials implement guidance on streamlining the change-order process. VA agreed with our recommendations and has implemented them. In 2017, we found that VA did not collect sufficient information to determine if new guidelines intended to ensure the timely processing of change orders were being followed. We also found that it did not have a mechanism in place to evaluate data on time frames to process change orders. Without such a mechanism, VA could not determine how processing time frames and design changes affect costs and schedules, and thus was at risk for unexpected cost increases and schedule delays. We recommended that VA establish a mechanism to monitor the extent that major facilities projects were following guidelines on change orders’ time frames and design changes. VA has also addressed this recommendation. In 2018, we found that the Veterans Health Administration, a component of the VA, had not established time frames for processing contract changes, and did not have a way to monitor the length of time or the reason contract changes occur. We recommended that officials collect information on contract modifications, establish target time frames that trigger a higher-level review of contract modifications, and centrally establish a mechanism to monitor and review certain contract modifications that were taking longer than the established target time frame. To date, the Veterans Health Administration has not yet fully implemented the recommendations. At a May 2017 congressional hearing before two subcommittees of the House Committee on Small Business, witnesses from the construction industry identified the contract change process as a challenge. They stated that the change process negatively affects cash flows, increases administrative and legal costs, and creates a risk of not receiving reimbursement for completed work. Industry representatives we spoke with reiterated these concerns. Industry representatives also explained that while contract changes were a challenge for businesses of all sizes, small business were likely to be more susceptible to challenges due to their having fewer financial and administrative resources. One resource for small businesses is an agency’s Office of Small and Disadvantaged Business Utilization or Office of Small Business Programs. These offices are responsible for working with agency officials to facilitate participation of small businesses in procurement. However, the small business advocates at GSA and USACE told us that their offices had a limited role in the construction contract change process. According to small business advocates at GSA, for example, their office may get involved in a limited manner when a small business contractor is having difficulty receiving payment by providing guidance on how to make a claim. Congress recently took action that will prompt agencies to gather information on the time it takes to make certain contract changes. Section 855 of the Fiscal Year 2019 National Defense Authorization Act includes a provision that requires agencies to make available information about the agency’s past performance in finalizing, or “definitizing,” REAs with certain construction solicitations. The provision also requires agencies to provide information about its policies and practices associated with how the agencies comply with Federal Acquisition Regulation requirements to definitize REAs in a timely manner. Agencies must start including this information no later than August 13, 2019. A variety of factors affect how long it takes to process a contract change. The factors include the time needed for making a change determination, creating a cost estimate, identifying funds, negotiating with the contractor, completing reviews, and processing the change. According to agency officials, some of these steps play a role in protecting the government’s best interests. For example, creating robust cost estimates helps provide the government with information to inform negotiations with the contractor. Unauthorized work—resulting from unauthorized direction or miscommunication—is another factor that can affect the change process timelines. When the contractor performs unauthorized work, the agency must then take additional steps, such as reviewing the work to determine if it should be reimbursed. Data we reviewed from USACE indicate that a majority of contract changes made from January 2013 through August 2018 were finalized in fewer than 60 days, and a little more than 3 percent took more than 1 year. Contractors and the government sometimes have different perceptions about when the contract change process begins—and therefore how long it takes—based on when the change begins to impact the work. The construction contract change process includes a number of steps that can factor into the time frames for finalizing a contract change, depending on the facts and circumstances surrounding an individual change. For example, USACE officials stated that obtaining a complete proposal from the contractor—with sufficient information on cost and schedule changes to begin negotiations—is a significant factor affecting contract change time frames. Figure 2 illustrates where these factors fall in a notional change process and describes how they may affect time frames. Agency contracting officials at both PBS and USACE note that some of these procedural steps are necessary to protect the government’s interests—which includes negotiating a fair and reasonable price for the work related to the change. According to USACE and PBS contracting officials, any unauthorized work undertaken by the contractor is another factor that can extend contract change process timelines. When unauthorized work is done, the government must take steps such as determining (1) if the work was required; (2) if the work constituted a change to the existing contract; and (3) if so, a fair and reasonable price for the work. Unauthorized work may occur, for example, when the contractor receives direction from a person who is not authorized to direct work, like a project manager. An authorized individual, such as the contracting officer, must provide such direction. Agency officials explained that unauthorized work can be the result of miscommunication between a government project official and the contractor. The contractor may interpret instructions from the unauthorized official as a formal direction to proceed with a change. In other cases, the contractor may begin work in anticipation of a contract change, before receiving any direction at all. One contractor representative told us that, at times, contractors feel pressured to start work without authorized direction to avoid disruption to the overall project that may result in negative performance reviews from the agency. According to USACE contracting officials, the agency compiles and reviews data on construction contract changes on an ad hoc basis to gain insight into time frames for the contract change process within that agency. The data and analysis show that the majority of changes from 2013 through 2018 at that agency were finalized within 60 days; however, a smaller percentage took substantially longer. Approximately 45 percent of the completed contract changes took more than 60 days to finalize, and a little more than 3 percent took more than 1 year. See figure 3 for information on USACE contract changes by the number of days taken to finalize the change. Contracting officials at USACE, as well as industry representatives, told us that government officials and contractors often have different perspectives on when the contract change process begins and, therefore, the time needed to complete it. For example, one industry representative said that the process begins for some contractors when the need for a contract change is identified. The representative explained that this is the point that the project work can change and the contractor begins to experience an impact on cost and schedule. Another industry representative said that some businesses think that the process begins when they submit their request for equitable adjustment, but that the government may not start measuring the process until a government official actively begins to address the request. Meanwhile, USACE contracting officials stated that process time should be measured from when they receive a complete proposal from the contractor, with no missing information. USACE officials told us that the data collected in its contract information system do not always reflect this metric, however. USACE contracting officials told us that, when recording the proposal receipt date that it uses as the start date for the contract change process, some contracting officers enter the date that the initial proposal was received, and others enter the date that a complete proposal was received. USACE contracting officials stated that they plan to address this issue in the future as part of a larger system upgrade. An industry representative explained that these varying viewpoints between government contracting officials and contractors are exacerbated by the contractors’ lack of understanding about the contract change process. The representative also stated that contractors find that the process is not transparent and implementation of the process varies by agency and even by district within the same agency, increasing confusion. While the amount of information on contract changes varies between USACE and PBS, neither agency regularly monitors contract change time frames. In addition to agency guidance that establishes time frames for certain contract change order actions, federal standards for internal control state that an organization should obtain quality information to achieve management objectives and establish monitoring activities. Neither GSA nor USACE has fully established such controls over the contract change process at the headquarters level, limiting management’s ability to identify and respond to problems. USACE information systems have data on contract changes for its more than 40 districts that are sufficient to calculate time frames for finalizing contract changes, but the agency does not regularly aggregate or monitor the information. Officials explained that this was in part due to the manual process required to compile the data centrally and perform calculations. A user must pull data for each USACE district from its contract information system and then manually manipulate the data to determine the time frames. As a result, the data are not reviewed by officials at headquarters on a routine basis. The contracting officials we spoke with said that contract change time frames are reviewed at the local level, specifically by project teams, typically on a weekly basis. Contracting officials also stated that contract change time frames are a factor in performance reviews for contracting personnel. There is currently no agency guidance or documentation for how often contract changes should be reviewed at either the project or district levels, the officials said. USACE contracting officials noted that they are in the early stages of planning for a system upgrade that they hope will automate the process of compiling and analyzing construction contract change data. However, these plans are preliminary. USACE has not yet determined which systems will be involved, nor has it documented these planning efforts to date. PBS contracting officials cannot track time frames for contract changes. While GSA’s contract information system does track and centrally compile data on all contract modifications, PBS contracting officials said there was no efficient way to separate the types of contract changes that we included in our review from other modifications, such as administrative changes or the exercise of options, preventing the calculation of time frames for contract changes. Our review of the GSA data confirmed that the data cannot be used to distinguish between the various types of contract changes. According to PBS contracting officials, to identify a contract change type, a reviewer would have to seek information at the local level by going into the individual contract file and reviewing the modification. Given these limitations, USACE and PBS cannot centrally identify emerging problems with contract change time frames or monitor compliance with existing Department of Defense (DOD) and GSA requirements. As noted above, DOD and GSA have established time frames for certain contract changes. USACE contracting officials said that they would likely establish additional, broad goals for finalizing contract changes in future policy revisions because more targeted goals were often not practical due to the unique circumstances that may affect process times. PBS contracting officials said that compliance with those time frames should be monitored by local staff, such as the contracting officer assigned to the project; however, there is no regular monitoring of that data or systematic way for contracting officers to track this information at the local level. There is currently no effort under way to develop a strategy to address data limitations at the local and headquarters level via information technology system upgrades, according to GSA officials. Further, USACE and GSA anticipate, and our analysis of available data confirms, that system limitations at both agencies are likely to make implementing section 855 of the Fiscal Year 2019 National Defense Authorization Act more difficult. This provision generally requires agencies to include information on recent time frames for definitizing REAs with any construction solicitations anticipated to be awarded to small businesses no later than August 2019. For example, GSA officials stated that to implement this provision would require substantial changes to their contract information system, which they must plan for 2 years in advance. USACE officials said that staff level discussions were ongoing on potential ways to comply with the requirement. They added, however, that in the absence of a system change making the data readily available, they would likely compile data manually, similar to what was provided to us, as an ad hoc substitute. In addition, both agencies said that they had questions about what information they would include in solicitations. Specifically, while section 855 refers to REAs, a USACE contracting official stated that REA could be interpreted differently by the government and industry. Similarly, GSA contracting officials said that the statutory language potentially covers a broad category of information, making it difficult to decide what data to capture and report. USACE officials stated that they will wait for DOD and the Department of the Army to provide direction before changing their system. GSA officials stated that they were not going to take action until further information is provided. One potential source of additional direction is Federal Acquisition Regulation (FAR) case 2018-020, which is developing a proposed FAR rule to implement section 855. The proposed rule is anticipated to be released in the first quarter of fiscal year 2020. Routine, central data collection on the construction contract change process can help agencies understand the scope of any problems encountered. While USACE can compile and review construction contract change information on an ad hoc basis, the agency does not conduct regular monitoring at the headquarters level and must manually manipulate data to review this information. GSA lacks information on the contract change process and its time frames at the headquarters, regional, and local levels. Without regular collection and review of information on the contract change process, contracting officials may be unable to spot potential problems—such as long process times that may affect project schedules—as they occur and respond accordingly. In addition to needing data for management purposes, agencies must also implement new legislative requirements when issuing certain construction solicitations starting in August 2019. While the proposed FAR rule, when issued, should provide agencies with more information on how to implement the new requirements, GSA and USACE could immediately begin to develop strategies to support routine collection and monitoring of time frames. Pursuing preliminary strategies on basic issues—such as what systems may need to be updated and what groups or individuals should be involved—would help these agencies better position themselves to comply with the requirement in a timely manner, and more quickly expand the data available for management purposes. We are making the following two recommendations: The Administrator of General Services should ensure that the Commissioner of the Public Buildings Service develops a strategy that outlines the steps needed to routinely collect information on and monitor the time frames for finalizing construction contract changes at the headquarters level. The strategy could address issues such as the types of construction contract changes that should be included, when the measurement of the contract change process should begin, and the information systems that will be affected. (Recommendation 1) The Secretary of the Army should direct the Chief of Engineers and Commanding General of the U.S. Army Corps of Engineers to develop a strategy to expand on existing data and systems to routinely collect information on and monitor the time frames for finalizing construction contract changes at the headquarters level. (Recommendation 2) We provided a draft of this product to DOD, GSA, and OMB for comment. DOD and GSA provided written comments, reproduced in appendixes II and III, respectively. DOD concurred with our recommendation and provided a technical comment, which we incorporated as appropriate. GSA also concurred with our recommendation, and noted that the agency is developing a plan to address it. OMB provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget, the Acting Secretary of Defense, and the Administrator of General Services. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or woodsw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report were Tatiana Winger (Assistant Director); Betsy Gregory-Hosler (Analyst-in-Charge); Michael Dworman; Gail-Lynn Michel; Peter Anderson; George Depaoli; Lorraine Ettaro; Lori Fields; Gina Hoover; Sam Portnow; Bill Shear; and Anne Louise Taylor. This report (1) identifies factors that affect the time it takes to finalize contract changes at selected agencies, and (2) assesses the extent to which selected agencies monitor time frames for finalizing contract changes. In this report we examined the process for managing unilateral and bilateral contract changes, but exclude certain types of contract modifications to focus on the issues of payments and cash flow challenges. Specifically, we excluded (1) administrative modifications because they do not entail changes to contract costs or time frames; (2) contract changes that go beyond the scope of the existing contract, referred to as cardinal changes; (3) contract options because exercising an existing priced option does not entail the same type of negotiations that unilateral and bilateral changes require; (4) contract disputes and claims because they follow a separate and distinct process; (5) the payment process after a contract change has been finalized because that process is directed by the Prompt Payment Act; and (6) any processes taking place between a prime contractor and its subcontractors because that is outside the focus of this review. To identify agencies for our review, we analyzed Federal Procurement Data System – Next Generation (FPDS-NG) data on construction contract obligations for fiscal year 2017, the most recent data available at the time. This allowed us to identify defense and civilian agencies that had large amounts of construction contract obligations and a relatively significant portion of those obligations going to small business. The data that we used assigned the contract obligations to the agency that managed the construction project rather than the funding agency. We found that the Department of the Army’s U.S. Army Corps of Engineers (USACE) obligated approximately $10.5 billion for construction contracts, with approximately $3.9 billion going to small business concerns. This obligated amount is more than any other federal agency or service within the Department of Defense. We found that the General Services Administration’s (GSA) Public Buildings Service (PBS) obligated approximately $1.9 billion for construction contracts, with approximately $870 million going to small business concerns. To assess the reliability of the FPDS-NG data we used, we (1) performed electronic testing of selected data elements, and (2) reviewed existing information about the FPDS-NG system and the data it produces. Specifically, we reviewed the data dictionary, data validation rules, and other documentation. Based on these steps, we determined the data were sufficiently reliable for the purposes of this report. To identify federal construction industry representatives for this engagement, we collected information on potential associations from various sources including previous congressional testimony and our prior work. From this list of options, we sought organizations that were focused on federal construction contracting, included a small business focus, represented a large number of contractors, and had performed previous advocacy work on the issues of under review in this engagement. Based on these criteria, we selected two organizations to interview: the Associated General Contractors of America and the National Association of Small Business Contractors. The Associated General Contractors of America, which sent a representative to a congressional hearing on the contract change process, represents 26,000 member firms and includes a division dedicated to federal construction as well as a small business committee. The National Association of Small Business Contractors specializes in small business contractors working with the federal government, and is affiliated with the American Small Business Chamber of Commerce. We interviewed representatives from these associations to confirm background information about how the change process impacts industry and further discuss the factors that affect process time frames. To identify the factors which affect the time it takes to finalize contract changes at selected agencies, we reviewed relevant legislation such as the John S. McCain National Defense Authorization Act for Fiscal Year 2019, regulations including the Federal Acquisition Regulation (FAR), the Defense Federal Acquisition Regulation Supplement, GSA Acquisition Regulation, and the GSA Acquisition Manual and relevant agency policies and guidance. We interviewed staff from the Office of Management and Budget’s Office of Federal Procurement Policy—the Administrator of which serves as the Chair of the FAR Council—and contracting officials from the PBS and USACE. In addition, we interviewed officials from GSA’s Office of Small Business Utilization and USACE’s Office of Small Business Programs to discuss their role in the change process and their perspective on possible impacts to small business concerns. To assess the extent to which selected agencies monitor time frames for finalizing contract changes, we collected and reviewed available GSA data on contract modifications. We also collected available data and analysis from USACE on construction contract changes from January 1, 2013 to August 17, 2018—representing more than 62,000 changes from the more than 40 USACE districts and one office that execute construction contracts—obtained from the USACE’s Resident Management System. We reviewed USACE analysis of those data that calculated time frames for the contract changes by measuring the time elapsed from the date a proposal is received to when the contract change is finalized by the signature of Standard Form 30, which officially modifies the contract. We also reviewed system documentation on the requirements for users to enter data into the systems. We interviewed PBS and USACE officials at the headquarters level to discuss the time frames for contract changes, including on how long officials believe the process takes, what data are available, and who reviews any data collected on the contract change process. We discussed the provided USACE data with knowledgeable USACE officials who performed the calculations to understand their process, assumptions, and methodology. We determined the data were sufficiently reliable for the purposes of describing what is known about the time frames for finalizing construction contract changes. We also interviewed an official in GSA’s Office of Government-wide policy, to discuss any GSA-wide plans for system changes. We conducted this performance audit from August 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings based on our audit objectives.", "summary": "In fiscal year 2018, federal agencies spent more than $36 billion on construction contracts, with more than 45 percent going to small business. Typically, construction projects involve some degree of change as the project progresses. Some federal construction contractors have raised concerns that delays in processing contract changes and making payments creates challenges, particularly for small businesses. Section 855 of the National Defense Authorization Act for Fiscal Year 2019 requires agencies to report information related to how quickly they finalize contract changes. GAO was asked to review federal construction contract change processes and timeframes. GAO (1) identified factors that affect the time it takes to finalize contract changes, and (2) assessed the extent to which selected agencies monitor time frames for finalizing contract changes. GAO reviewed relevant regulations and agency policies, analyzed available data, and interviewed officials from GSA's Public Buildings Service and USACE—two agencies with large amounts of obligations on construction—and two industry associations. Multiple factors affect the time it takes to finalize a construction contract change. For example, preparing cost estimates can be time consuming, particularly for complex changes. Yet the time may be used to help ensure the government has adequate cost data to inform negotiations. In addition, according to agency officials, miscommunication during the contract change process—which can lead to problems such as unauthorized work undertaken by the contractor—can result in additional reviews and longer time frames. According to U.S. Army Corps of Engineers (USACE) data, most of its construction contract changes are finalized within 60 days. Some take much longer, however (see figure). Agency officials and industry representatives agreed that perceptions differ about the length of the contract change process. For example, because a change can impact the contractor's cost and schedule immediately, the contractor typically perceives that the process starts earlier—and lasts longer—than the government does. Neither GSA nor USACE regularly monitors how long it takes to finalize construction contract changes, limiting management's ability to identify and respond to problems. Internal controls require agencies to collect and use quality data for management purposes such as monitoring agency activities. GSA systems do not collect data that permit analysis of contract change timeframes at the headquarters level. USACE systems produce contract change data for its districts, but data consolidation and calculations must be done manually and are not done regularly. Neither agency has a strategy in place to address these issues. Without regular review of these timeframes, USACE and GSA contracting officials may be unaware of any existing or potential problems, such as long process times that may affect project schedules. In addition, these data system limitations are likely to create difficulties for agencies when providing the information required by new legislation. GAO is making two recommendations: that GSA's Public Buildings Service and USACE each develop a strategy to routinely collect information on and monitor time frames for construction contract changes at the headquarters level. Both agencies concurred with our recommendation.", "document_type": "gao"}
{"report": "The federal government’s civilian real-property holdings include thousands of leased office buildings and warehouses across the country that cost billions of dollars annually to rent, operate, and maintain. GSA’s Public Building Service acquires space on behalf of the federal government through new construction and leasing and acts as a caretaker for federal properties across the country. The type and amount of space for each lease varies based on a particular agency’s need, and GSA categorizes leases by value depending on factors such as square- footage and location. As of fiscal year 2018, the Public Building Service held or leased space in 8,681 buildings or other assets and maintained an inventory of more than 370-million square feet of workspace for 1.1 million federal employees, plus support contractors. The federal-leasing process contains several stages, and brokers can be involved in many parts of this process (see fig. 1) as a way to supplement the work of GSA’s leasing staff. For example, in the “requirements development” phase, GSA can task brokers with drafting project milestones and working with federal agencies that are seeking building space to provide a complete requirements package. In the “lease acquisition” phase, brokers can conduct market research on rental rates, negotiate rates and terms of the lease, and prepare final contract forms. For such work, brokers can earn a commission based on a percentage of the aggregate lease value. However, pursuant to the Federal Acquisition Regulation, brokers are not allowed to complete some activities, as contractors cannot be used for the performance of inherently government functions. Brokers are not allowed to complete all required leasing tasks to execute a federal government lease. For example, according to GSA officials, a broker cannot sign a lease contract on behalf of the federal government with a property owner since that action is considered an inherently governmental function. The broker may prepare the final lease contract, but GSA’s contracting officials are responsible for signing the lease. Even when a broker is involved in the leasing process, GSA officials oversee and approve the broker activities. Prior to 2015, GSA had implemented various changes to how it used brokers to assist with its leasing program. Before 1997, GSA’s in-house staff completed all leasing acquisition work, but starting in the late 1990s, downsizing initiatives at GSA reduced the number of staff and therefore its in-house capacity to acquire leases. In 1997, GSA began to increase its use of brokers by signing regional contracts for broker services and paying brokers by using appropriated funds. By 2003, brokers were completing approximately 20 percent of GSA’s leasing work. In 2003, GSA analyzed the advantages, disadvantages, and costs of different types of contracting options for using the brokers, including having them negotiate leases on a nationwide basis, as compared to designated geographic zones or local areas. Based on that analysis, GSA concluded that contracting for brokers to negotiate leases nationally represented the best option available and formalized the program as the National Broker Contract program. In 2004, under this program, GSA awarded nationwide contracts to four commercial real- estate brokerage firms, moving from a regional to a national approach. In 2010, GSA established the second iteration of the broker program (called the National Broker Contract 2), which maintained a similar nationwide structure with four national contracts to brokers. We have previously found that GSA has been unable to demonstrate cost savings with its broker program. For example, in 2007, we found that GSA was unable to quantify savings from the program and recommended that GSA develop processes for doing so. In response to our recommendation, GSA conducted a comparative analysis of prior agency contracts and broker contracts; this analysis demonstrated program cost savings. However, GSA’s subsequent efforts to demonstrate continued cost savings were less conclusive. For example, in 2012, when GSA attempted to compare rental rates negotiated by brokers with those negotiated by in-house staff, the agency found little difference between the two and noted that the data were insufficient to conduct a meaningful comparison. In 2013, we found that GSA had not linked its goals and metrics for evaluating the broker program to the anticipated cost savings in rental rates. As a result, GSA had no means of evaluating and reporting on this aspect of the program, and the value of the broker program in terms of cost savings continued to be unclear. We recommended that GSA link program goals to anticipated cost savings and develop and implement a means of evaluating and reporting program results. In response, GSA developed a metric for measuring the efficacy of utilizing brokers to assist with lease workloads and a performance report that included information on financial savings and productivity, among other things. We found limitations; however, with these efforts, as discussed in the second section of this report. GSA has made changes to the broker program to allow more brokers to participate and to increase GSA’s flexibility in its use of brokers. In 2015, GSA changed the program’s name to the GSA Leasing Support Services program (GLS). Under this version of the program, GSA moved from using four brokers on a nationwide basis to designating brokers within four geographical zones. GSA awarded contracts to two or three brokers for each zone (see table 1). Thus, each GLS contract covers a zone rather than the entire country, as was previously done under the National Broker Contract. Currently, there are six GLS brokers, and each broker can serve up to two zones. Of the six GLS brokers, five participated in the National Broker Contract programs. According to GSA officials, modifying the program to operate by zone provided a greater opportunity to involve more brokers, increase competition and local market specialization, and strengthen relations among brokers and GSA regional offices. In addition, awarding contracts by zones rather than the entire country has allowed small businesses to participate as brokers, and GSA selected two small-business brokers as prime contractors: Carpenter Robbins Commercial Real Estate, Inc., and Public Properties LLC. Multiple GSA regional offices oversee and monitor brokers within each zone, except for the National Capital region, which is its own zone. In early 2020, GSA plans to announce the brokers that will be involved in the fourth iteration of the program. In this iteration, called GLS Plus, the zones and number of brokers within each zone will remain the same. In addition to establishing the zones, GSA has also allowed its regional staff to have more flexibility in deciding how to use brokers. During the past two iterations of the National Broker Contract, brokers had to be involved during the entire leasing process. In the GLS program, regional GSA officials choose broker services for specific parts of the leasing process based on the needs of the region. For example, several regional officials said they could now request brokers to perform market research or negotiate a lease, while GSA staff performs other tasks to complete a lease. Officials in three of the six regional offices we interviewed said this change provided additional flexibility in how GSA involves the brokers in the leasing process and helped balance the workload of GSA staff. In GLS Plus, GSA will request that brokers provide additional post-award services such as evaluating pricing for proposed renovations and monitoring on-site construction progress for the leased facility. In the GLS program, about 64 percent of the brokers’ workload were high- value leases. GSA officials told us they typically task brokers to negotiate these high-to-moderate value leases because brokers are paid through commissions as a percentage of the lease’s value. Since they earn more money with high value leases, they have a greater incentive to participate in the program. Consistent with what GSA officials said, the agency’s leasing data showed that leases involving brokers tended to have large square-footage and higher rents than the rents for leases that did not involve brokers, as shown in table 2. According to GSA’s leasing data from October 2005 to July 2019, the agency used brokers in about 37 percent of all leases. Available data did not clearly demonstrate the extent to which brokers negotiated lower lease rates than GSA’s in-house staff for similar types of properties. Although there are differences in rental rates and other outcomes of leases involving brokers compared to those that do not, it is difficult to determine whether these differences are due to having brokers involved in the process as opposed to the characteristics of the leases themselves. Various factors affect rental rates in federal leases, such as local market areas, type of facility, square footage, and unique requirements, among other issues. According to the Public Buildings Service’s Commissioner, brokers are more successful at negotiating lower lease rates relative to the market than GSA in-house staff and using brokers provides savings to the government. GSA officials said they believe this result is in part because brokers negotiate what are called “commission credits”—a percentage of the total commission that goes back to the federal tenant agency in the form of a reduction in rent— which can result in lower costs for federal tenant agencies. In contrast, several lessors (property owners) said that when GSA uses brokers to negotiate leases, broker commissions have to be paid by the lessor and that this cost is ultimately passed on to GSA’s federal-agency-tenant clients. Furthermore, three real estate economists we interviewed indicated that real-estate sale prices and rental rates are driven primarily by competitive market forces and thus would not be heavily influenced by broker negotiation. These economists were not aware of any research indicating that brokers could affect commercial real estate rental rates. As previously noted, GSA typically tasks brokers to negotiate high-to- moderate value leases. A broker-negotiated GSA lease includes a total commission negotiated between the lessor and the broker that represents a percentage of the aggregate lease value. This total commission is comprised of the standard commission paid to the broker and commission credits given back to the federal tenant agency. In the GLS program, the total commission sometimes includes a “best value” commission that a broker may earn on top of the standard commission. This total commission includes the following three components: Standard Commissions. The standard commission a broker earns is normally a percentage of the total lease value. Our analysis showed that brokers earned about $390 million in standard commissions since fiscal year 2006 (see table 3). For the GLS program, brokers had earned just over $35 million as of July 2019. At the time of our review, the program was ongoing, and brokers were still completing leases. Best Value Commission. Under the GLS, in addition to the standard commission that a broker always earns, the broker can be paid an additional commission, called the “best value commission,” by negotiating a lease rate below an established market rate target and earning high evaluation ratings from GSA. Specifically, the best value commission was expected to incentivize brokers to negotiate lower rental rates. This best value commission is paid out of the commission credit the tenant agency would otherwise receive and does not increase the total cost of the commission. As of July 2019, brokers had collected about $3.5 million in best value commissions during the GLS program. GSA plans to eliminate the best value commission in the new iteration of its broker program, GLS Plus. Officials said determining whether brokers met the best value criteria was burdensome for regional officials and that brokers prefer a steady volume of future government leases as an incentive. Similarly, two real estate economists we interviewed said that the best value commission was unnecessary to incentivize brokers to seek the best rates for their GSA client, and that the prospect of additional future business negotiating government leases was a sufficient incentive. Commission credits. The commission credit is a percentage of the total commission that goes back to the federal tenant agency in the form of a reduction in rent. As part of the total commission, brokers have negotiated over $340 million in commission credits. GSA estimates that its future GLS Plus program will generate $129 million in commission credits throughout the duration of the program. Lessors and real estate economists we interviewed highlighted various issues about GSA’s commission structure, including commissions paid to the broker and commission credits paid back to the tenant agency. The interviewees had different perspectives on whether GSA’s broker program and the current commission structure are beneficial to the federal government. Some questioned whether the use of brokers saves the federal government money. As previously noted, according to GSA officials, lessors, through the commission, pay the brokers, which is customary in commercial real estate. Although GSA does incur some costs from appropriated funds because GSA officials oversee the work of brokers, GSA officials noted that GSA does not currently use its own appropriated funds to compensate brokers for services performed as a part of the broker program. However, four lessors that we interviewed said that broker commission costs are passed through to federal tenants in their leases. These lessors questioned the benefits of using brokers for federal leases. In contrast, two real estate economists we interviewed said that GSA could potentially be missing cost-saving opportunities when brokers are not used because rental rates are generally set by competitive market forces, also GSA’s in-house staff may not negotiate commission credits. GSA officials, however, disagreed with this statement, saying in-house staff generally seek to receive credit or concessions for leases they negotiate since there is no commission to be paid to a broker. Another real estate economist we interviewed indicated that paying brokers based on a fixed price basis, versus a commission basis, could result in lower costs to the government because this type of payment structure could involve GSA brokers’ bidding for lease acquisition assignments in fixed-price terms only. This real estate economist also said that this approach could potentially address past concerns involving GSA’s commission structure. Over the years and with different iterations of the program, GSA has established various goals for the broker program; most of these goals pertain to cost savings. During our review, GSA officials also said that the main purpose of the program is to serve as a workforce multiplier for the regional offices—providing needed personnel to complete leases that GSA does not have enough staff to complete on its own. Our review of GSA documents and interviews with GSA staff identified various program goals as shown in table 4. For GLS Plus, the fourth iteration of the broker program, which GSA plans to start in mid-2020, the proposed goals include achieving taxpayer savings, improving the customer experience, and leveraging broker expertise. GSA officials also said that maximized productivity would be a goal of the program. As previously discussed, one of the main goals of the broker program is to avoid costs and save the taxpayer money. In November 2019, GSA headquarters officials said that they demonstrate cost savings of the broker program through its Lease Cost Avoidance Plan, which aggregates cost-savings from several efforts, including negotiating leases below market rates, reducing rented square footage, and leasing vacant space. A metric within the Lease Cost Avoidance Plan that seeks to show whether leases are negotiated below market rates is called Lease Cost Relative to Market, which is a comparison of the negotiated rental rate to the target market rate. According to this metric, as reported by GSA, over the last 3 years, brokers have negotiated 303 leasing deals, 60 percent of which were below the market rate (17.8 percent below the market rate, on average), which helped GSA avoid $676 million in costs. In addition, GSA found that brokers negotiated better rental rates than GSA in-house staff, on average. For example, GSA reported that in fiscal year 2018, 56 percent of leases negotiated by brokers were below the market rate compared to 38 percent of leases negotiated by GSA in-house staff. As discussed previously, however, there are various factors, including the type of lease that may account for these differences. This metric is calculated primarily using market lease rates that GSA determines using a tool it developed called the “Bullseye” report. To develop the report, GSA gathers available market data from commercial real estate databases and compiles these data to identify local information, analysis, and insight regarding the local real estate submarket. According to GSA guidance, the success of the GLS program is dependent on the brokers’ negotiating competitive lease rates through full utilization of the Bullseye report and standardized negotiation objectives. The guidance further states that the Bullseye report should be utilized by GSA regional offices as a tool to make informed leasing decisions on behalf of the U.S. government and can provide the necessary backup documentation to aid leasing personnel in their negotiation with an offeror. While GSA headquarters officials noted that this tool is adequate for this use, other GSA officials and brokers had concerns about whether the Bullseye report accurately reflects market rates and conditions. GSA regional officials we interviewed had mixed views on the accuracy of the Bullseye report. For example, several officials questioned the accuracy or noted limitations to the Bullseye report. In addition, four of the six brokers found the Bullseye report to be rarely or only sometimes accurate. As a result, brokers told us that they found it difficult to negotiate a rental rate at or below the target Bullseye rate. In addition, two lessors we interviewed agreed that the gap between the Bullseye report and local market rates potentially affected negotiations with GSA. Furthermore, brokers publicly questioned the accuracy of Bullseye reports in written responses to GSA’s draft solicitation for the 2020 GLS Plus program. They also suggested that the new broker program should include an adjudication process for revisiting Bullseye target rates. Selected GSA regional officials and brokers in our review identified several factors that may affect the accuracy of the Bullseye reports: Geography. According to GSA officials, the Bullseye report includes market rates from over 85 major markets in the U.S. However, GSA regional officials and brokers we interviewed said that the Bullseye report provides limited submarket rental rates for specific areas or neighborhoods within large metropolitan areas. This can be problematic because there can be significant rental differences among different areas within a given market. For example, in response to GSA’s draft solicitation for the new broker program, brokers stated that they found the Bullseye target rates to be an obstacle in the rapidly moving West Coast urban markets, and there can be significant discrepancies between Bullseye rates and actual market rates. One selected GSA regional office in our review provided examples of the Bullseye target rate being below the market rate in several instances. For example, the average asking rent for office space in San Diego, CA, was 36 percent higher than the Bullseye rate. Federal requirements. According to GSA regional officials and to brokers, the Bullseye report does not take into account the unique building requirements for federal leases. For example, GSA officials and brokers reported that the Bullseye report develops a target rental rate based on certain classes of buildings (A, B, and C). Although the government generally accepts A and B class buildings, C buildings are generally unacceptable for federal leases. Brokers we interviewed said including these C building rates could lower the market rates identified by the Bullseye market report for certain areas. GSA officials said they are not able to remove the C building rates from the Bullseye report because the data are purchased from a private-sector data source that includes various building rates from a local area. In addition, brokers said the Bullseye report does not take into account the unique requirements of federal buildings. For example, federal law enforcement agencies require certain security measures, such as special entrances. Brokers reported that landlords may increase their pricing to account for these factors. Brokers also identified these issues in the draft solicitation for the new broker contract, noting that the Bullseye does not use comparable buildings that take into account the uniqueness of a specific space requirement. Lag time. Several brokers and GSA officials told us that federal leases generally take significantly longer than commercial leases due to the federal leasing acquisition process. As a result, GSA officials and brokers found that by the time a lease was awarded, which could be years later, the initial target market rate provided by the Bullseye report was outdated. GSA headquarters officials told us if the Bullseye report is over a year old, then an updated report should be requested, although it’s not mentioned by the 2016 Bullseye guidance memo. Officials from selected GSA regional offices varied on whether those updates occurred or not. Furthermore, several brokers in our review told us that they found that the Bullseye report is not always updated after a year. One broker told us that there have been several instances when a lease is about to be awarded—which can be 1 to 2 years after the initial Bullseye report was generated—and the tenant agency is not willing to accept the rental rates negotiated in the lease. Or GSA’s leasing staff is hesitant to execute the lease due to differences between the Bullseye rate and the actual lease contract rate. This can cause significant delays or result in the project being canceled all together. Concerns about the reliability of the Bullseye report call into question whether the Lease Cost Relative to Market metric can accurately demonstrate how brokers’ efforts lead to cost savings, either through achieving rental rates below market or better rates than GSA in-house staff. Even though GSA provided us cost-savings data in November 2019 based on this metric, at other times during our review, GSA officials described limitations and questioned the efficacy of using this metric. Specifically, in April 2019, GSA headquarters officials told us that GSA had stopped using this metric because GSA found it unreliable. For example, GSA found the comparison was not indicative of broker effectiveness or ability to negotiate low rental rates. At that time, GSA officials cautioned against using the Lease Cost Relative to Market data for comparative purposes, such as comparing broker performance to in- house GSA staff performance. The officials said it is impossible to assess the financial information of a lease transaction and evaluate a specific procurement method—using brokers or not—without talking directly to the GSA in-house staff responsible for overseeing the procurement. Furthermore, GSA officials told us in April 2019, that leases negotiated by brokers were not comparable to leases negotiated by in-house staff because they work on different types of leases. In December 2019, however, GSA officials told us that GSA does still track this metric, uses it for GSA’s Lease Cost Avoidance Plan, and that the agency believes brokers can achieve better deals for the government than in-house staff. Nonetheless, GSA officials told us that they have not assessed the reliability of or made any changes to how they calculate the Bullseye report. According to Standards for Internal Control in the Federal Government, management should use quality information to achieve the entity’s objectives and to inform decision-making. Until GSA assesses the reliability of the information used to calculate reported cost savings for the broker program, it is hindered in its ability to fully assess the effectiveness of the program. As noted above, throughout the various iterations of the program, GSA has identified various goals for the broker program. During this review, a key goal consistently stated by GSA officials we interviewed was the use of the broker program as a workforce multiplier—providing additional people that enable GSA to complete leasing work it would otherwise be unable to complete. The effectiveness of the broker as a workforce multiplier is significant to GSA because leasing staff has decreased by over 50 percent since the late 1990s, from over 800 personnel to less than 400 in 2019. Consequently, GSA staff rely on brokers to deliver leased space to federal agencies. GSA officials told us that a broker may not accomplish a lease faster or cheaper than GSA staff but that the agency does not currently have the personnel to complete its leasing work. GSA’s Strategic Plan 2018-2022 also states that GSA will use brokers where appropriate to improve efficiency in awarding leases. Although GSA has set target goals for utilizing brokers and tracks the number of leases regional officials assign to brokers, we found that GSA had limited ability to track how using brokers to augment the GSA’s leasing workforce achieves results for GSA’s leasing efforts. For example, GSA has increased its broker utilization targets in recent years, as described in figure 2, requiring regional offices to award more lease projects to brokers. Moreover, GSA tracks performance relative to these targets, and regional officials in our review told us that they are evaluated based on the number of leases they task to brokers. Additionally, in April 2019, GSA developed a model to project, on average, the number of hours a broker saves the GSA’s lease-contracting officer and project manager. The project estimated the broker saved roughly 175 to 125 hours, respectively, per project over a 3-year period. GSA then multiplied the hourly salary of GSA leasing personnel by the potential number of hours saved to generate their reported personnel savings of $3 million per year. However, tracking these outputs alone does not provide GSA with a means to measure the effectiveness of the broker program as a workforce multiplier. An output measure tracks the direct product or activity delivered by a program, while an outcome measure tracks the progress the program is making toward achieving its goal. Tracking the number of hours a broker saves for GSA officials provides limited information to help GSA understand the overall benefits of the broker program as a workforce multiplier. For example, this goal does not demonstrate if brokers are more productive than in-house staff or if they are completing leases more efficiently, such as brokers completing an additional number of leases on an annual basis. According to GSA officials, the principal way they measure broker program outcomes is through its Lease Cost Avoidance Plan, which, as we previously discussed, aggregates cost savings from a number of GSA leasing efforts, including the broker program. The plan identifies realized cost avoidance through various metrics such as leases negotiated below market costs and reductions in rental square footage and vacant space. However, aside from the negotiated rental rates, GSA does not currently have specific metrics that allow it to distinguish the particular role brokers play in achieving those results. For example, GSA officials said that the more leases that can be replaced by using brokers, the more GSA can tackle its expiring lease inventory and right size leases with rental square foot reductions. Specifically, GSA officials said that brokers contributed to a 2.5 percentage square footage reduction in fiscal years 2018 and 2019. However, since this metric applies to the leasing program in general and is not specific to the brokers, GSA is unable to demonstrate the extent to which such reduction is attributable to the use of brokers. GSA officials also told us that using brokers allows GSA to replace more leases on time and thus avoid extending leases, which is more costly and can lead to agencies renting space under less favorable terms. GSA measures this through its lease replacement rate, which tracks the percentage of expiring leases that are replaced in a timely manner. For example, GSA reported that in fiscal year 2019, it replaced 61 percent of its lease inventory, which represented $481 million of its $791 million lease inventory. However, while GSA tracks the number of lease extensions brokers have worked on, GSA is unable to demonstrate the extent to which the use of brokers helps GSA avoid lease extensions and holdovers. Furthermore, similar to the Lease Cost Avoidance Plan, this metric applies to the leasing program in general and is not specific to the brokers. As a result, GSA has limited information on the extent to which brokers contributed to leasing program outcomes. GPRA, as amended, creates a framework for articulating unified goals and outcome measures that can provide federal agencies with a clear direction for successful implementation of program activities and improve the efficiency and accountability of agencies’ efforts. We have previously reported that the GPRA framework can serve as a leading practice at other organizational levels, such as component agencies, offices, programs, and projects. GPRA calls for outcome-based metrics that are linked to goals, which allow a program to track the progress an organization is making toward achieving its intended outcome. Because GSA lacks outcome-based metrics that demonstrate the broker’s role in achieving the program’s goal for being a workforce multiplier, GSA is hindered in its ability to distinguish the role brokers played in its reported program results. Furthermore, having such a metric could help GSA make better decisions about the balance of brokers versus in-house leasing staff since GSA received $34 million dollars for fiscal year 2020 to hire an additional 34 GSA lease-contracting officers and specialists. GSA officials said they plan to complete this hiring in 2020. GSA has developed a program that allows the agency to utilize expertise and personnel from leading commercial real-estate brokers to help it complete thousands of federal leases. GSA has stated cost savings and workforce goals for the broker program but lacks the information necessary to assess if the program is achieving its intended results. If GSA envisions that the use of brokers is to save money, then having quality, reliable data and information is critical to demonstrating this result. If using brokers to augment GSA’s workforce were also a goal, then having outcome-based metrics would allow GSA to show whether it is achieving that goal. This information is especially critical as the program has changed over time and could provide GSA insight on what has been successful in the past. This information would also inform GSA’s decision-making as it launches another version of its broker program and uses millions of dollars in appropriated funds to increase the agency’s leasing personnel. We are making the following two recommendations to the Administrator of GSA: GSA should assess and address the reliability of the information used to calculate reported cost savings for the broker program. (Recommendation 1) GSA should develop outcome-based metrics to evaluate the effectiveness of using brokers to supplement the GSA’s leasing workforce. (Recommendation 2) We provided a draft of this product to GSA for review and comment. GSA provided written comments, which are summarized below and reproduced in appendix I. GSA said it did not agree with our main conclusions and findings because it believed our report did not acknowledge brokers’ demonstrated benefits. We noted throughout the report that brokers play an important role in helping GSA achieve various leasing-related goals. Our position is that the lack of quality data and outcome-based metrics inhibit GSA’s ability to demonstrate the brokers’ specific effect in achieving GSA’s goals as compared to other factors. With regard to the first recommendation about data used to calculate reported cost savings from the broker program, GSA said it concurred with the recommendation and is making changes to its data systems that it believes will improve its data on brokers. GSA said it did not agree with the second recommendation as it was originally worded about having outcome-based measures to evaluate the effectiveness of using brokers. In providing technical comments on our report, GSA officials raised concerns that this recommendation gave the impression that GSA had no metrics to assess the brokers. The agency said that it has several outcome-based metrics in place that it believes can be correlated with the value of the brokers, including achieving cost savings, replacing leases on time, and reducing the need to hire more GSA staff. In GSA’s letter, it referenced these statistics, several of which we had included in our report as well. For example, our report discusses GSA’s Lease Cost Relative to Market measure, which is a comparison of the negotiated rental rate to a target market rate. We also noted, however, that this metric is calculated primarily using data, that GSA staff and other stakeholders we interviewed expressed concerns about as unreliable. These concerns resulted in our first recommendation. Further, other metrics, such as reducing square footage and replacing leases, that GSA pointed to relate to GSA’s leasing efforts in general and are not designed in a way to distinguish the brokers’ contributions specifically. Specifically, GSA officials said that brokers contributed to a 2.5 percentage square footage reduction in fiscal years 2018 and 2019. This metric, however, applies to the overall leasing program, and GSA is unable to demonstrate the extent to which such reduction is attributable to the use of brokers. In addition, GSA does not have a means to measure the effectiveness of the broker program in supplementing its workforce to achieve these goals, a result that GSA staff in headquarters and regional offices consistently told us was the primary reason GSA uses brokers. Tracking the number of hours a broker saves for GSA officials provides limited information to help GSA understand the overall benefits of the broker program. Such information does not demonstrate if brokers are more productive or efficient than in-house staff, such as whether brokers are completing an additional number of leases on an annual basis, for example. Additional metrics focused on evaluating the outcomes of GSA’s use of brokers would benefit the agency because it has lost over 50 percent of its leasing personnel since the 1990s. Furthermore, GSA received $34 million to hire additional agency lease-contracting officers and specialists in 2020. Consequently, it is imperative that it has information and data that could inform the right mix of brokers and GSA leasing personnel as the agency moves forward with its leasing work. In response to GSA’s concerns and to make our recommendation more specific, we clarified the recommendation. Specifically, we focused it more narrowly on the need to evaluate the effectiveness of using brokers to supplement the GSA leasing workforce. We also made some additional changes to the draft to include more information about the metrics GSA uses and that it believes can be correlated to the use of brokers. We are sending copies of this report to the appropriate congressional committees, the Administrator of the General Services Administration, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or RectanusL@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Lori Rectanus, (202) 512-2834 or RectanusL@gao.gov. In addition to the individual named above, other key contributors to this report were: Andrew Huddleston (Assistant Director), Nelsie Alcoser, (Analyst-in-Charge), Caitlin Cusati, Josh Ormond, Colleen Taylor, Jack Wang, Michelle Weathers, Crystal Wesco, and William Woods.", "summary": "As the leasing agent for the federal government, GSA acquires space for federal agencies and currently manages over 8,000 leases. To help negotiate leases, GSA contracts with commercial real-estate brokerage firms. In previous reviews, GAO reported that GSA was unable to demonstrate cost savings and results from its use of brokers, and GAO made related recommendations. A statute included a provision for GAO to review GSA's broker program. This report examines: (1) how GSA's broker program has changed over time and (2) GSA's goals for the broker program and how GSA measures the program's results. GAO reviewed documentation from GSA's broker program and GSA's available data on leases assigned to brokers from October 2005 to July 2019. GAO interviewed officials from GSA headquarters, selected GSA regional offices that work with brokers, as well as other stakeholders, including representatives from the six brokers currently participating in the program. The General Services Administration (GSA) contracts with commercial real estate brokers to perform a variety of services needed to acquire and complete leases. GSA uses brokers to negotiate leases meeting certain thresholds in urban areas (see figure). GSA has made several changes to its broker program since 2015, including: changing how brokers can be assigned to leases, i.e., using brokers for specific geographical zones rather than on a nationwide basis; allowing greater flexibility in when and how brokers can be used during the leasing process; and changing the name from the National Broker Contract program to the GSA Leasing Support Services program. Statistics for General Services Administration's (GSA) Leases That Involve Brokers Compared For the broker program, GSA's goals include saving money and supplementing its leasing workforce; however, potentially inaccurate data and limited outcome-based metrics could affect GSA's ability to assess whether it is meeting these goals. According to GSA, in the last 3 years, brokers have negotiated 303 leases, 60 percent of which were below the market rate (17.8 percent below the market rate, on average), an outcome that, GSA says helped it avoid $676 million in costs. However, selected GSA regional officials and brokers expressed concerns about the accuracy of the market reports used to calculate these cost savings. Additionally, while GSA has identified various outcome-based metrics related to its leasing program, these metrics do not indicate whether using brokers to supplement its leasing workforce has enabled GSA to complete leasing work it would have otherwise been unable to complete. For example, GSA sets targets for and tracks the number of leases assigned to brokers each year, but this measure is not an indicator of the effectiveness of using brokers. Quality information, along with additional reliable outcome-based measures, is important for GSA to define success for its 2020 broker program which creates new contracts and expands services performed by brokers. GAO is making two recommendations that GSA should: (1) assess and address the reliability of the information used to calculate reported cost savings and (2) develop outcome-based metrics to evaluate the effectiveness of using brokers to supplement the GSA leasing workforce. GSA concurred with the first recommendation but did not concur with the second. GAO continues to believe that GSA needs metrics to assess the brokers' role as a workforce supplement.", "document_type": "gao"}
{"report": "The Coast Guard is a multimission, maritime military service that is responsible for maritime safety and security, environmental protection, and national security, among other missions. Given the Arctic region’s expansive maritime domain, the Coast Guard plays a significant role in Arctic policy implementation and enforcement. Therefore, as we have reported, as more navigable ocean water has emerged in the Arctic and human activity increases, the Coast Guard has faced, and will continue to face, expanding responsibilities in the region. In June 2016, we found that the Coast Guard assessed its capability to perform its missions in the Arctic in fiscal year 2012 and identified various capability gaps, including the following: Communications: including the lack of communications architecture. Harsh weather conditions, high latitude disturbances, and geomagnetic storms combine to make communications in the Arctic difficult. Arctic maritime domain awareness: including limited nautical charting, inadequate navigation systems, and insufficient surveillance. Extremely limited operational assets and support infrastructure in the Arctic, as well as the harsh operating environment, make achieving maritime domain awareness a challenge. Infrastructure: including limited aircraft infrastructure on the North Slope in northern Alaska and limited logistical support. Facilities located below the Arctic Circle, and even those within Alaska, provide limited capability to support Arctic missions due to the long transits to the Arctic region. No deepwater ports currently exist on the North Slope or near the Bering Strait that are capable of refueling and re- provisioning polar capable cutters. This forces the Coast Guard’s polar capable cutters to expend significant time transiting long distances to and from replenishment ports. Development of infrastructure to support operations is challenging, in part, due to the high cost of transporting materials to the Arctic and short construction seasons. Training and exercise opportunities: including a limited pool of Arctic-trained and experienced Coast Guard personnel, and limited training, exercise, and educational opportunities to enhance Arctic skills among staff. According to Coast Guard officials, few opportunities exist to train in the Arctic, in part, because of limited Coast Guard icebreaking capacity. Icebreaking: including limited icebreaking capacity given the Coast Guard’s existing active inventory of one medium and one heavy polar icebreaker, as discussed later in this testimony. At the time of our June 2016 review, Coast Guard officials stated that the capability gaps were not the sole responsibility of the Coast Guard to mitigate and did not completely impair or eliminate their ability to perform operations. For example, while communications can be a challenge in remote regions, the risk of lost communications can be mitigated by using multiple assets working together to mitigate risk if lost communications is anticipated. Coast Guard officials also stated that given its activity levels at the time, the mobile and seasonal nature of its Arctic presence, and its ability to leverage partners’ resources, the Coast Guard has had sufficient resources to fulfill its Arctic responsibilities. However, Coast Guard officials stated they would reassess their approach as Arctic activity and resulting mission requirements change over time. As we reported in June 2016, if Arctic activity continues to increase, as anticipated, the Coast Guard may have insufficient resources to meet expanded Arctic requirements. In June 2016, we also found that the Coast Guard worked with its Arctic partners—such as other federal agencies—to carry out actions to help mitigate Arctic capability gaps. For example, the Coast Guard took steps to enhance Arctic maritime domain awareness by testing communications equipment belonging to DOD during a 2015 annual operation in the Arctic, extending communications capabilities further north than previously possible. However, we found that the Coast Guard did not systematically assess how its actions helped to mitigate these gaps. Such an assessment—which includes developing measures for gauging its progress, when feasible—is critical to the Coast Guard’s understanding of its progress towards addressing these gaps. By systematically assessing and measuring how its actions have helped to mitigate capability gaps, the Coast Guard will be better positioned to effectively plan its Arctic operations, including its allocation of resources and prioritization of activities to target the gaps. As a result, we recommended in June 2016 that the Coast Guard (1) develop measures for assessing how its actions have helped to mitigate Arctic capability gaps and (2) design and implement a process to systematically assess its progress on this. DHS concurred with our recommendations. As of January 2020, the Coast Guard had not yet taken action to implement these two recommendations, in part because the Coast Guard issued its Arctic strategic outlook in April 2019 and is currently updating its corresponding implementation plan for this strategy. The plan is expected to provide the foundation for systematically assessing efforts to address Arctic capability gaps. Coast Guard officials stated that they are also developing a strategic metrics framework for measuring progress in addressing the capability gaps. Coast Guard officials did not identify when they plan to complete the plan and framework, stating that these are longer-term efforts. The Coast Guard highlighted the Arctic capability gaps in its 2013 Arctic Strategy and again in its 2019 Arctic strategic outlook. The 2019 strategy highlighted the need to elevate the Arctic region’s prominence as a strategically competitive space due to (1) the resurgence of nation-state competition from the United States’ two nearest-peer powers, Russia and China, and (2) reduced ice conditions in the Arctic which have led to increased human and economic activity in the region. In addition, the 2019 Arctic strategy highlighted three overarching goals: enhance capability to operate effectively in a dynamic Arctic domain, strengthen the rules-based order, and innovate and adapt to promote resilience and prosperity. Further, the 2019 Arctic strategy noted that the Coast Guard is the sole provider and operator of the U.S. polar icebreaking fleet—a critical component in achieving the Coast Guard’s overarching goals in the strategy—but currently does not have the capability or capacity to ensure access in the Arctic region. The Coast Guard’s polar icebreaking fleet comprises two operational polar icebreakers—the Polar Star and Healy— of which only the Healy is currently active and operating in the Arctic. The Healy is a medium icebreaker that primarily supports Arctic research, and while it is capable of carrying out a wide range of activities, it cannot ensure timely access to some Arctic areas in the winter given that it does not have the icebreaking capabilities of a heavy polar icebreaker. See figure 1 for photographs of the Coast Guard’s active icebreakers. In November 2018, the Coast Guard Assistant Commandant for Acquisition testified that the Coast Guard’s current polar icebreaking fleet provides minimal capacity to carry out current icebreaking missions and that the nation must take swift action to rebuild and enhance this critical national capability. To this end, DHS approved the Coast Guard’s Polar Security Cutter acquisition program’s cost, schedule, and performance baselines in February 2018. In September 2018, we found that the Coast Guard did not have a sound business case when it established the acquisition baselines for the Polar Security Cutter program in March 2018 due to risks in four key areas: technology, design, cost, and schedule. Our prior work has found that successful acquisition programs start with solid, executable business cases before setting program baselines and committing resources. A sound business case requires balance between the concept selected to satisfy operator requirements and the resources—design knowledge, technologies, funding, and time—needed to transform the concept into a product, which in this case is a ship with polar icebreaking capabilities. Without a sound business case, acquisition programs are at risk of breaching the cost, schedule, and performance baselines set when the program was initiated—in other words, experiencing cost growth, schedule delays, and reduced capabilities. To address the key risks we identified and help establish a sound business case for the Polar Security Cutter program, we made six recommendations to DHS, Coast Guard, and the Navy in our September 2018 report. The agencies concurred with all six recommendations and have taken steps to address some of the risks, as noted below. Technology. The Coast Guard planned to use proven technologies for the program, but did not conduct a technology readiness assessment to determine the maturity of key technologies prior to setting baselines. As a result, the Coast Guard did not have full insight into whether these technologies were mature and was potentially underrepresenting the technical risk of the program. We recommended that the program conduct a technology readiness assessment, which DHS completed in June 2019. DHS determined that two of the three key technologies were mature and the remaining technology was approaching maturity. The Coast Guard now has plans in place to use testing results to increase the maturity and reduce risks for the remaining technology—the hull form. Design. The Coast Guard set program baselines before conducting a preliminary design review. This review is a systems engineering event intended to verify that the contractor’s design meets the requirement of the ship specifications and is producible. By not conducting this review before establishing program baselines, the program is at risk of having an unstable design, thereby increasing the program’s cost and schedule risks. We recommended that the program update its baselines prior to authorizing lead ship construction and after completion of the preliminary design review. DHS and the Coast Guard agreed and plan to take these steps by fiscal year 2022. Cost. The cost estimate that informed the program’s $9.8 billion cost baseline—which includes life cycle costs for the acquisition, operations, and maintenance of three polar icebreakers—substantially met our best practices for being comprehensive, well-documented, and accurate. But the estimate only partially met best practices for being credible. The cost estimate did not quantify the range of possible costs over the entire life of the program, such as the period of operations and support. As a result, the cost estimate was not fully reliable and may underestimate the total funding needed for the program. We recommended that the program update its cost estimate to include risk and uncertainty analysis on all phases of the program life cycle, among other things. Subsequently, in December 2019, we found that while the Coast Guard updated the cost estimate in June 2019 to inform the budget process, the estimate did not reflect cost changes resulting from the contract award two months prior. Coast Guard officials acknowledged these cost risks and plan to address them as part of the next update to the program’s cost estimate. Coast Guard officials told us that they plan to update the cost estimate by the end of February 2020. Schedule. The Coast Guard’s initial planned delivery dates of 2023, 2025, and 2026 for the three ships were not informed by a realistic assessment of shipbuilding activities. Rather, these dates were primarily driven by the potential gap in icebreaking capabilities once the Coast Guard’s only operating heavy polar icebreaker—the Polar Star—reaches the end of its service life. In addition, our analysis of selected lead ships for other Coast Guard and Navy shipbuilding programs found the icebreaker program’s estimated construction time of 3 years to be optimistic. An unrealistic schedule puts the Coast Guard at risk of not delivering the icebreakers when promised. As a result, the potential gap in icebreaking capabilities could widen. We recommended that the program develop a realistic schedule, including delivery dates, and determine schedule risks during the construction phase of the program. In response, the Coast Guard is now tracking additional schedule risks for the program and is in the process of updating its program schedule. Further, in December 2019, we found that the contract delivery date for the lead ship, May 2024, is 2 months after the delivery date in the program’s schedule baseline. Coast Guard officials said they plan to address this risk when they update the program’s schedule by the end of March 2020. In summary, the Arctic region has increased in strategic importance in recent years, and with the increase comes more responsibility for the Coast Guard. The Coast Guard has emphasized that as the Arctic continues to open and strategic competition drives more actors to look to the Arctic for economic and geopolitical advantages, the demand for Coast Guard leadership and presence will continue to grow. As the Coast Guard embarks on the acquisition of its new polar icebreakers, it faces a number of key acquisition risks. The Coast Guard has begun to take steps to address these risks and must remain committed to executing a sound business case for the program to mitigate capability gaps in the Arctic. To this end, we will continue to monitor the Coast Guard’s progress in addressing our recommendations. Chairman Correa, Ranking Member Lesko, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this statement, please contact Marie A. Mak, (202) 512-4841 or makm@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony include Rick Cederholm, Assistant Director; Claire Li, Analyst-in-Charge; Peter Anderson; Jay Berman; Tracey Cross; Laurier Fish; Miranda Riemer, and Roxanna Sun. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Coast Guard—a component of the Department of Homeland Security (DHS)—is a multimission, maritime military service that is responsible for maritime safety and national security, among other missions. Given the Arctic region's expansive maritime domain, the Coast Guard plays a significant role in Arctic policy implementation and enforcement. The Coast Guard is also the sole provider and operator of the U.S. polar icebreaking fleet—a critical component in ensuring year-round access to the Arctic. The Coast Guard is developing the first of three heavy polar icebreakers—the Polar Security Cutter—it has acquired in over 40 years. This statement addresses (1) the Coast Guard's assessment of capability gaps in the region, and (2) key risks facing the Polar Security Cutter acquisition. This statement is primarily based on GAO's June 2016 report examining capability gaps in the Arctic and its September 2018 report examining the Coast Guard's polar icebreaker acquisition. In fiscal year 2012, the Coast Guard—the primary federal maritime agency in the Arctic—assessed its capability to perform its missions in the region and identified a number of capability gaps. These gaps, which still exist today, include communications, infrastructure, maritime domain awareness, and icebreaking. The Coast Guard has worked to mitigate these gaps with its Arctic partners, such as other federal agencies. For example, during a 2015 annual operation in the Arctic, the Coast Guard took steps to enhance maritime domain awareness by testing the Department of Defense's communications equipment, extending communications capabilities further north than previously possible. However, in June 2016, GAO found that the Coast Guard did not systematically assess the extent to which its actions helped to mitigate these gaps. In response to GAO's recommendation, the Coast Guard is currently developing an implementation plan and corresponding metrics for its April 2019 Arctic Strategy. In September 2018, GAO found that the Coast Guard faced four key risks when it established the Polar Security Cutter program in March 2018: technology, design, cost, and schedule. For example, the Coast Guard's initial planned delivery dates of 2023, 2025, and 2026 for the three ships were not informed by a realistic assessment of shipbuilding activities. The schedule was driven, instead, by the potential gap in icebreaking capabilities once the Coast Guard's only operating heavy polar icebreaker—the Polar Star —reaches the end of its service life (see figure). GAO recommended in September 2018 that the program develop a realistic schedule and determine schedule risks for the program. In response, the Coast Guard is now tracking additional schedule risks for the program and is in the process of updating its program schedule. GAO will continue to monitor the Coast Guard's progress in addressing this recommendation and other recommendations GAO made to address key risks, such as design and cost, facing the Polar Security Cutter program. In June 2016, GAO recommended, among other things, that Coast Guard develop measures for assessing how its actions have helped to mitigate Arctic capability gaps. In September 2018, GAO recommended that the Polar Security Cutter program develop a program schedule according to best practices. DHS concurred with all of the recommendations, and the Coast Guard is in the process of addressing them.", "document_type": "gao"}
{"report": "DLA’s primary purpose is to meet the logistics requirements of the armed forces for food, clothing, fuel, spare parts, and other items. DLA’s major responsibilities are to buy and distribute about 5 million distinct consumable, expendable, and reparable items. In order to fulfill the logistics requirement of the armed forces, DLA provides more than $35 billion in goods and services annually. As part of its responsibility to provide spare parts, DLA officials stated that three of DLA’s major subordinate commands conduct reverse engineering—Aviation, Land and Maritime, and Troop Support. These commands are directly responsible for meeting the following military services’ needs. Specifically: Aviation provides aviation weapons systems spare parts, flight safety equipment, maps, environmental products, and industrial plant equipment. Land and Maritime provides for ground-based and maritime weapons systems spare parts, small arms parts, and fluid handling systems. Troop Support handles food, textiles, construction material, industrial hardware, and medical supplies and equipment, including pharmaceuticals. Reverse engineering is the process of replicating a design by physically examining and measuring an existing item to develop the technical data necessary to reproduce the item functionally and dimensionally. In other words, it is the process of extracting information about an item from the item itself. According to DLA’s procedure, the intent of reverse engineering is either to develop an approved technical data package—the details needed to duplicate the item such as drawings or specifications, among other things—or approve a new supplier. The technical data package could be used for future competitive procurements when the item is needed for sustainment purposes. DLA uses reverse engineering for several reasons, such as identifying potential new sources for obsolete parts or those supplied by only one source, increasing competition, and achieving savings. Parts are identified for reverse engineering generally because the government does not have necessary legal rights to the drawings or lacks data needed to facilitate competitive procurements. However, reverse engineering is generally considered the least attractive alternative for replenishing parts because it is expensive and time-consuming. Therefore, according to DLA guidance, before starting reverse engineering DLA personnel should try to buy data from the original equipment manufacturer, previous manufacturing sources, or other data rights holder. DLA personnel also need to consider the costs of acquiring the data rights before initiating reverse engineering. DLA guidance for its reverse engineering processes is the same regardless of the size of the businesses involved. In order to start a reverse engineering project, DLA has established criteria, such as documenting a business case and showing that the part meets certain yearly purchase thresholds. Specifically, according to DLA guidance, procurements of the part over the past 2 years generally should exceed $10,000 in each year. Additionally, the engineering support activity at a military department (Army, Air Force, or Navy) responsible for the part generally approves a project before reverse engineering can begin. Once reverse engineering is completed, the technical data package must be approved by the appropriate engineering support activity. Figure 1 shows the general process from identifying a part as a potential candidate for reverse engineering to approving a technical data package or a new source of supply, and indicates whether DLA or a contractor is responsible at each step. DLA has two reverse engineering programs—the Replenishment Parts Purchase or Borrow program, funded by contractors, and the DLA internally funded efforts. Through the Replenishment Parts Purchase or Borrow program, contractors reverse engineer a part at their own expense. DLA’s goals for this program are to increase competition and achieve savings. After successful reverse engineering and military approval of a technical data package, the contractor becomes a new source of supply for the part. Contractors may identify parts for reverse engineering—mostly small hardware items and electrical components, such as antennas and cables—from a candidate list on a DLA website or through their own research. To conduct reverse engineering, contractors may purchase or borrow parts under agreement with the government subject to certain conditions, such as the part is not classified or considered a critical part— one that is crucial enough that a failure of that part would result in serious injury or impact the success of a mission—among other things. The government incurs minimum cost, if any, in this program. The goal of DLA-funded reverse engineering is to develop a technical data package that will be used in future competitive procurements. Typically, DLA funds efforts for parts that are available from only one source, are obsolete, or have limited data rights. These efforts occur in several ways: DLA engineers conduct the reverse engineering. DLA funds the efforts through partnerships with other DOD entities. DLA awards contracts to companies to create drawings of parts that the government can then use in competitive procurements. The government obtains full use of these drawings. Intellectual property derives from the work of the mind or intellect and is an application, right, or registration relating to property—such as an idea, invention, or process. It includes patents and proprietary information: Patents—grants an inventor the right to exclude others from making, using, or selling an invention in the United States, typically for a period of 20 years. The holder of a valid patent is the only authorized supplier of the patented item unless another supplier has acquired a license to manufacture and sell the item. Proprietary information—includes technical data which represents trade secrets usually developed at private expense, such as design, material composition, or manufacturing processes. The owner of a specific item does not make the information available to others without obligations concerning its confidentiality. This confidentiality of proprietary information does not protect it from discovery by reverse engineering. In order to share technical data, for example, to issue a solicitation for competitive procurement of an item, DLA needs to have sufficient data rights. Data rights are the government’s contractual license rights for technical data—recorded physical and material characteristics, such as item specifications, engineering drawings, or operating and maintenance manuals. If the government is entitled to and acquires unlimited data rights, it is allowed to use, reproduce, or disclose that technical data. When the government acquires limited rights the government may only use the data internally, such as for the operation of equipment, but may not disclose technical data for the purpose of procuring an item from another contractor. DLA procedures indicate the government should consider the cost of acquiring the data rights before initiating reverse engineering. DLA’s Aviation, Land and Maritime, and Troop Support major subordinate commands initiated over 1,600 reverse engineering projects during fiscal years 2015 through 2018, according to DLA data. Table 1 describes the number of projects initiated in each fiscal year by DLA’s Aviation, Land and Maritime, and Troop Support for both the contractor- and DLA-funded efforts. Nearly two-thirds of all reverse engineer projects involved parts in the following five categories. 1. Hardware and abrasives. For example, screws, nuts, washers, and keys. 2. Vehicular equipment components. For example, floor mats, vehicle door hinges, and tailgates. 3. Electrical and electronic equipment components. For example, pressure switches, electrical assemblies, and antennae. 4. Electric wire, and power and distribution equipment. For example, batteries, wiring harnesses, and special purpose cable assemblies. 5. Aerospace craft components and accessories. For example, insulation blankets, filters, and door handles. Figure 2 shows examples of items in these categories. Some of DLA’s projects resulted in lower prices for the reverse engineered parts in subsequent procurements. According to DLA data, the agency saved at least $22 million from reverse engineering projects initiated from fiscal years 2015 through 2018 as a result of lower prices paid. For example, in one project we reviewed, a small business successfully reverse engineered a retaining ring, which assists in securing parts in an aircraft engine, purchased by Troop Support and became a new approved source of supply. A subsequent purchase of this part resulted in a unit price that was almost $70 lower per unit compared to the most recent purchase before reverse engineering. We found that Troop Support saved over $11,000 through this project. We found that 141—or less than 10 percent—of all projects initiated from fiscal years 2015 through 2018 were successfully completed. Table 2 describes the number of projects that were completed, in-process, and canceled as of December 2018, according to DLA. DLA officials told us that less than 10 percent of projects have been completed for several reasons. DLA officials stated that the engineering support activities at the military services sometimes take a long time to respond to requests for approval, which adds time to reverse engineering projects. For contractor-funded projects under the Replenishment Parts Purchase or Borrow program, reasons include the following: DLA officials explained that contractors sometimes decide not to complete reverse engineering because the contractors determine the effort would be more labor intensive than originally anticipated or their priorities shift. DLA officials also stated that DLA does not necessarily hear back from a contractor after it purchases a part to begin the reverse engineering process. Representatives from contractors we spoke with told us that they experience delays in obtaining responses from DLA and military service engineers regarding approvals for their projects. In addition, DLA officials explained that the completion rate for DLA- funded projects is low for the following reasons: More urgent priorities arise after a project begins. Personnel who work on reverse engineering typically have other duties in addition to these efforts and if more urgent priorities emerge after a reverse engineering project has started, the effort may need to be postponed or abandoned to attend to the more pressing priority. The need no longer exists for some parts that were identified as reverse engineering candidates. We found that DLA’s reverse engineering programs created opportunities for contractors—particularly small businesses—to become new suppliers. According to DLA data, 124 contractors participated—or had the opportunity to become new sources of supply for parts—in the contractor- funded reverse engineering program from fiscal years 2015 through 2018. Of these, we determined that 103 were small businesses. In addition, while DLA performed reverse engineering work for most of the DLA- funded projects, DLA awarded contracts to 6 companies to conduct reverse engineering, all of which were small businesses. Roughly one-third of the 124 contractors worked on only one spare part during the time frame we reviewed, while the others initiated reverse engineering for multiple parts. For example, one small business requested to reverse engineer more than 50 parts during fiscal years 2015 through 2018. A representative for this contractor told us they use a business analytics system that queries publicly available information in order to identify opportunities for reverse engineering. While most companies conducting reverse engineering were small businesses, almost half the contractors that supplied parts prior to reverse engineering were not small businesses. Specifically, for fiscal years 2015 through 2018, DLA identified 74 contractors whose parts had been successfully reverse engineered, of which we determined that 34 were not small businesses and 26 were. In general, representatives at the small businesses we spoke with stated that the reverse engineering programs help small businesses. For example, representatives of one contractor that requested to reverse engineer a part told us that DLA’s reverse engineering program as a whole benefits small businesses because most of the parts being reverse engineered are originally from other than small businesses, and the program is a path that allows small businesses to become approved suppliers. In addition, representatives from one contractor that reverse engineered multiple projects stated that working with DLA has allowed their business to establish past performance ratings, which will help in future government procurements. Officials from DLA competition advocate and small business offices stated that reverse engineering is generally beneficial for small businesses. They stated that small businesses have not registered complaints about DLA’s reverse engineering program. Rather, small businesses seek opportunities for additional business with DLA, which reverse engineering can help provide. In addition, industry associations we spoke with stated that reverse engineering is a way to involve small business. They stated that reverse engineering projects provide small businesses opportunities to become qualified suppliers and compete for future DLA contracts. We found that DLA has processes to protect intellectual property, such as patented designs and proprietary information during reverse engineering. Of the 10 small businesses we spoke with that were involved with our 19 selected projects, none of the representatives identified concerns with DLA’s practices for protecting intellectual property. Although DLA’s standard operating procedure applies to both DLA-funded efforts and contractor-funded efforts, the provisions that safeguard against patent infringement are specific to the contractor-funded program. According to these provisions, patented materials should not be approved or shared for reverse engineering. Aviation and Land and Maritime officials stated that they physically review parts for patent marks before reverse engineering can take place for contractor-funded efforts under the Replenishment Parts Purchase or Borrow program. However, Troop Support officials said they do not check for patents under this program because they are supplying parts for legacy systems—systems that are typically 20 years or older—which means any potential patents would have expired. They also stated that they conducted reverse engineering to a limited extent through the Replenishment Parts Purchase or Borrow program. We found Troop Support initiated 30 of the 617 Replenishment Parts Purchase or Borrow efforts from fiscal years 2015 through 2018. For DLA-funded efforts, Aviation and Land and Maritime officials stated that they physically review parts for patents, and the project is stopped if one is found. For example, officials from Land and Maritime canceled a project for an eye guard because officials found drawings marked with a patent number. The officials could not determine if the patent had expired so they canceled the project. Aviation officials also stated that they rarely encounter patent markings; they have seen four or five in the last 2 years. Troop Support officials stated they do not check for patent markings and they rely on the engineering support activities to check for patents. Figure 3 shows an example of a patent marking. DLA’s guidance for reverse engineering also does not allow the release of limited rights data for the contractor-funded projects under the Replenishment Parts Purchase or Borrow program. DLA officials at all three major subordinate commands told us that they do not release drawings that have limited data rights to contractors conducting reverse engineering under this program. DLA guidance does not cover whether data can be used under DLA-funded efforts. Officials at Aviation, Land and Maritime, and Troop Support all stated that proprietary or limited release drawings or technical data related to the part cannot be used by engineers who work on reverse engineering efforts. Aviation officials noted that the tracking system records those who have seen propriety drawings, and this information is used to ensure that these individuals do not work on reverse engineering projects related to those drawings. Land and Maritime officials stated that the tracking system used for reverse engineering projects also identifies drawings marked as proprietary and controls who has access to drawings. In general, the small businesses we met with did not express any concerns about how DLA handles intellectual property. We spoke with four small businesses that supplied parts that had been reverse engineered by other businesses. Representatives of these four small businesses confirmed that they did not hold patents on the parts that were reverse engineered. Further, the small businesses that conducted reverse engineering stated that DLA adequately protected intellectual property. In one case, a representative from a small business that participates in the Replenishment Parts Purchase or Borrow program stated that DLA has never released a part that had controlled data to them and DLA takes protection of proprietary data seriously. Another contractor who performs DLA funded reverse engineering efforts stated drawings are not released and sometimes the function of the part is not even shared with contractors. DLA command officials stated they had not heard of any concerns from small businesses about their intellectual property being used inappropriately. In addition, officials from the DLA small business office stated they had not heard concerns from any small businesses. Further, one official from the small business office noted that the parts DLA purchases are not new innovations and so do not necessarily have protected intellectual property. Competition advocates stated they received no complaints from small businesses. The industry associations we met with asked their small business membership if there were any specific concerns regarding DLA’s protection of intellectual property. Officials from the Aerospace Industries Association, the National Defense Industrial Association, and the National Association of Manufacturers stated there were no complaints from the small businesses represented by their groups about the businesses’ experiences working with DLA. We provided a draft of this report to DOD for review. DOD had no comments. We are sending copies of this report to the Secretary of Defense; the Director, Defense Logistics Agency; appropriate congressional committees; and other interested parties. This report will also be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by e-mail at makm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. In addition to the contact named above, Penny Berrier, Assistant Director; Brandon Booth, Alexandra Dew Silva, Stephanie Gustafson, Victoria Klepacz, Jean McSween, Ralph Roffo, Roxanna Sun, Anne Louise Taylor, and Alyssa Weir made key contributions to this report.", "summary": "The Department of Defense spends tens of billions of dollars annually to sustain military assets including aircraft, ships, and missiles. In support of this effort, DLA strives to maintain a competitive supplier base through reverse engineering—the process of examining an item, such as a spare part, with the intent of replicating its design. Contractors consider intellectual property, such as their technical data and patented material, essential to their success. DLA also takes steps to safeguard contractors' intellectual property during reverse engineering. The Senate Armed Services Committee report accompanying a bill for the fiscal year 2018 National Defense Authorization Act included a provision for GAO to review DLA's reverse engineering efforts, including the protection of small businesses' intellectual property. This report describes (1) DLA's reverse engineering programs and the extent to which small businesses participated in these programs from fiscal years 2015 through 2018; and (2) how DLA safeguards certain intellectual property within its reverse engineering efforts. GAO analyzed data from three DLA commands—Aviation, Land and Maritime, and Troop Support, those that conduct reverse engineering—from fiscal years 2015 through 2018. GAO reviewed a nongeneralizable sample of 19 reverse engineering projects involving 13 parts, selected to include a variety of characteristics, such as the size of the contractors involved. GAO reviewed DLA's guidance and interviewed DLA officials and representatives from small businesses about safeguarding intellectual property as part of reverse engineering. The Defense Logistics Agency (DLA) is responsible for providing logistics support to the warfighter, including spare parts for military assets. From fiscal years 2015 through 2018, DLA initiated over 1,600 reverse engineering projects for spare parts at three of its commands—Aviation, Land and Maritime, and Troop Support. DLA uses reverse engineering to identify potential new sources for spare parts that are available from only one source and to achieve savings. DLA funded about 1,000 of the reverse engineering projects, while contractors funded the remaining 600 projects. Nearly two-thirds of all reverse engineering projects involved parts in five categories, with examples of the three largest categories illustrated in the figure. GAO found that the majority of contractors conducting reverse engineering for DLA were small businesses. Specifically, DLA identified 124 contractors that conducted reverse engineering projects from fiscal year 2015 through 2018, 103 of which GAO determined were small businesses. According to small business representatives and DLA officials, reverse engineering is beneficial for small businesses and can help provide opportunities for additional business with DLA. GAO found that the three DLA commands had processes to safeguard certain intellectual property in their reverse engineering efforts. Specifically: Officials from all three commands stated they do not release drawings with limited data rights to contractors interested in reverse engineering parts. Aviation and Land and Maritime officials stated that they check for patent markings on parts to ensure patented parts are not reverse engineered. Troop Support officials stated they do not check for patent marks because the parts they supply are often too old to have valid patents. The small businesses GAO met with did not identify concerns with how DLA handles intellectual property. Further, DLA officials stated that they had not received any complaints from small businesses about their intellectual property being used inappropriately.", "document_type": "gao"}
{"report": "In passing the NVRA in 1993, Congress found that unfair registration laws and procedures can have a direct and damaging effect on voter participation in federal elections. The NVRA was intended, in part, to establish procedures to increase the number of eligible citizens who register to vote in federal elections, as well as to protect the integrity of the electoral process and ensure accurate and current voter registration rolls. As such, the NVRA includes provisions focusing on both increasing opportunities for voter registration and improving voter registration list maintenance. Table 1 below includes a summary of these provisions. The Help America Vote Act of 2002 (HAVA), which amended the NVRA, requires states to implement an interactive computerized statewide voter registration list and perform regular list maintenance. HAVA requires states to perform regular list maintenance by comparing their voter registration lists against state records on felons and deaths. HAVA also established the Election Assistance Commission to assist the states regarding HAVA compliance and to serve as a national clearinghouse of election administration information, among other purposes. In the United States, the authority to regulate elections is shared by federal, state, and local officials. DOJ is responsible for (1) civil investigations and enforcement under federal voting rights laws, such as the NVRA, and (2) criminal investigations and prosecutions under federal election crime statutes, such as those prohibiting double voting or voting by noncitizens. With regard to enforcement of NVRA provisions: the Civil Rights Division’s Voting Section (Voting Section), within DOJ, enforces the civil provisions of federal laws that protect the right to vote, including provisions of the NVRA, as well as HAVA, the Voting Rights Act of 1965, and the Uniformed and Overseas Citizens Absentee Voting Act, among others. In addition to DOJ’s role in enforcing the NVRA, the law also allows a private party (a person or organization) who is aggrieved by a violation of the NVRA to bring a civil action against the state or local agency responsible for voter registration. With regard to enforcement of federal election crime statutes: the Criminal Division’s Public Integrity Section supervises DOJ’s nationwide response to election crimes, such as voter fraud and campaign finance offenses, and reviews all major investigations and criminal charges proposed by U.S. Attorneys’ Offices relating to election crime. Public Integrity Section attorneys investigate and prosecute selected cases involving alleged corruption (including election crimes) by federal, state, or local government officials. U.S. Attorneys’ Offices investigate and prosecute a wide range of criminal activities, including federal election fraud, within their respective federal judicial districts. Each U.S. Attorney exercises wide discretion in the use of his or her resources to further the priorities of the local jurisdictions and needs of their communities. DOJ’s civil and criminal enforcement actions are recorded in case management systems which differentiate between matters and cases. A matter is defined as an activity, such as an investigation of an allegation, that has not yet resulted in the filing of a complaint, indictment, or information in court. A matter may eventually become a case, or may be closed without further action. A case is defined as an activity that has resulted in the filing of a complaint, indictment, or information in court. Cases typically start as matters. The process for initiating matters and filing cases varies across the three DOJ components we reviewed. For example, within the Criminal Division, staff are to open a matter when they have worked on an investigation for a minimum of 30 minutes. States are responsible for the administration of state and federal elections, and states regulate various aspects of elections including, for example, registration procedures, absentee and early voting requirements, and Election Day procedures. Within each state, responsibility for managing, planning, and conducting elections is largely a local process, residing with about 10,500 local election jurisdictions nationwide. Under the NVRA and HAVA, states are required to have a voter registration list maintenance program, and state and local election jurisdictions are responsible for ensuring that the registration lists are accurate, and that ineligible voters are lawfully removed. The NVRA specifies certain categories under which jurisdictions may remove registrants from voter registration lists, including: if a registrant has moved outside of a jurisdiction and either (1) confirmed the move in writing or (2) failed to respond to address confirmation mailings and failed to vote in two consecutive federal general elections subsequent to the mailing; death of the registrant; criminal conviction of the registrant, as provided for in state law; and mental incapacity of the registrant, as provided for in state law. State and local election officials can only remove registrants from the voter registration list after meeting certain requirements outlined in the act. Specifically, the NVRA stipulates that list maintenance activities must be uniform, non-discriminatory, and in compliance with the Voting Rights Act; and that programs to systematically remove ineligible voters must not be undertaken within 90 days of a federal election, except under certain circumstances. As noted above, election officials may remove a registrant from the voter registration list for change of residence if the registrant confirms the move in writing, or fails to respond to an address confirmation notice and fails to vote in two subsequent federal general elections following the mailing of the address confirmation notice. While state procedures differ, states generally designate registrants who are sent an address confirmation notice or fail to respond to the address confirmation notice in a timely manner as “inactive.” The “inactive” status generally indicates that the election officials may need to receive information from the registrant or other sources to confirm the registrant’s address. See figure 1 for an illustration of the NVRA confirmation and removal process for registrants who may have moved outside of the jurisdiction. States and local jurisdictions use different data sources and different processes and procedures to obtain information under the NVRA removal categories and to maintain accurate voter registration lists. For example, election offices in some states collaborate with their state’s motor vehicles agencies—such as a Department of Motor Vehicles—to acquire information on changes to registrants’ addresses or other identifying information. Some states also participate in interstate exchanges—such as the Electronic Registration Information Center (ERIC) and Crosscheck—to compare information from their voter registration lists and other state and local sources. States may also use national databases— such as the U.S. Postal Service’s NCOA database or the Social Security Administration’s public DMF—to identify registrants who have moved to another jurisdiction or state, or who have died. Multiple factors such as state laws, costs, the security of voter registration information, and related privacy considerations play a role in election officials’ list maintenance activities and procedures. In some states, the state maintains the responsibility for matching some data sources (such as data on deaths and moves) against the voter registration list and removing certain ineligible voters; whereas in other states, local jurisdictions have a larger role in the list maintenance process. Within DOJ, the Voting Section has the authority to initiate a matter or pursue a case under the NVRA, among the other voting laws for which it is responsible. According to Section officials, the Section identifies potential NVRA violations through several means, including reviewing publically available federal elections and other data, reviewing publically available federal and third party reports, receiving complaints, and conducting compliance investigations that may include visits to state and local offices. Officials stated that after initiating and conducting an investigation (or matter), the Section makes a recommendation to the head of the Civil Rights Division who then decides which action to take, such as pursuing litigation by filing a case against a state or local election jurisdiction. The Voting Section categorizes its NVRA-related matters and cases as related to providing registration opportunities for voters (registration opportunities), or related to the rules regarding maintenance of voter registration lists under specified conditions, which includes both wrongful removals of eligible voters and failure to remove ineligible voters (list maintenance). In addition to enforcing the NVRA through initiating matters and filing cases, the Voting Section participated in NVRA cases as an amicus curiae or “friend of the court,” entered into settlement agreements with states to address issues related to NVRA provisions, and engaged in other efforts to assess compliance with NVRA requirements. According to Civil Rights Division data we analyzed, the Voting Section initiated 1,295 matters from fiscal years 2001 through 2017 to investigate issues related to provisions of statutes such as the NVRA, HAVA, and the Voting Rights Act. Of these 1,295 matters, 99 involved allegations under the NVRA. As shown in figure 2, the Section initiated the largest number of NVRA matters during this period in fiscal years 2008 (15) and 2011 (25). In initiating matters under the NVRA, the Voting Section investigated issues related to state and local jurisdiction efforts to provide registration opportunities for voters and issues related to list maintenance. Specifically, of the 99 NVRA matters the Voting Section initiated, 58 matters involved registration opportunity issues, 17 involved list maintenance issues, and 5 involved both registration opportunity and list maintenance issues. As shown in figure 3, the Section initiated registration opportunity matters in each year except fiscal year 2007. The Section initiated the most registration opportunity matters in fiscal years 2008 (13), 2011 (10), and 2013 (7). The Section did not initiate any list maintenance matters in some years, and initiated between one and four in other years. From fiscal years 2001 through 2017, the Voting Section participated in 234 cases, including those with claims brought under statutes such as the NVRA, HAVA, and the Voting Rights Act. Of the 234 total cases, 23 involved claims brought under the NVRA. Figure 4 shows the total number of cases, and the number of NVRA related cases, in which the Section participated, by fiscal year. In contrast to matters, the Voting Section filed more cases related to list maintenance allegations under the NVRA than cases related to registration opportunities. Of the 23 cases where the Section took action to enforce the NVRA, the Section was the plaintiff or plaintiff intervenor in 14 cases. As shown in figure 5, eight of the 14 NVRA cases the Section filed as the plaintiff or plaintiff intervenor involved allegations under the law’s list maintenance provisions, and two involved allegations under both the list maintenance and registration opportunity provisions. The remaining four cases involved allegations under the law’s registration opportunity provisions. Of the 10 total cases involving list maintenance allegations, eight were filed between fiscal years 2002 and 2007. See appendix II for a summary of each NVRA related case the Section filed from fiscal years 2001 through 2017. With regard to list maintenance cases, as shown in figure 5, the Voting Section filed 10 such cases from fiscal years 2001 through 2017. NVRA list maintenance cases may involve two types of allegations: (1) in conducting a required program to remove ineligible voters from the voter registration list, a state or local jurisdiction did not incorporate certain safeguards, thus unlawfully removing eligible voters; and (2) a state or local jurisdiction did not have an adequate program to remove ineligible voters from the voter registration list. We reviewed the allegations in each of the 10 cases involving NVRA list maintenance claims and found that: Four of the 10 cases (filed in fiscal years 2002, 2007, 2012, and 2017) involved claims that the state or local jurisdiction unlawfully removed voters from registration lists. For example, in one case the Section alleged that the state systematically removed voters from its voter registration rolls within 90 days of a federal election, in violation of the NVRA, among other claims. Four of the 10 cases (filed in fiscal years 2006 and 2007) involved claims that the state or local jurisdiction did not have an adequate program to remove ineligible voters from registration lists. For example, in one case the Section alleged that a state failed to conduct a program that makes a reasonable effort to identify and remove ineligible voters from the state’s registration list, and that, as a result, the state had counties with excessively high registration totals compared to the voting age population. Two of the 10 cases (filed in fiscal years 2004 and 2006) involved both types of claims. For example, in one case the Section alleged that a number of local jurisdictions in one state did not regularly remove persons who died from their voter registration lists, resulting in ineligible voters remaining on the lists. The Section further alleged that local jurisdictions in the state did not always follow NVRA notice and timing requirements with respect to voters who may have moved, resulting in the unlawful removal of voters from voter registration lists. With regard to registration opportunities, the Voting Section filed six cases involving allegations under the NVRA’s registration opportunities provisions from fiscal years 2001 through 2017. We reviewed the allegations in each of these six cases and found that: Three of the six cases involved claims that the state failed to offer voter registration opportunities in public assistance offices and offices that provide state-funded programs primarily serving persons with disabilities. For example, in one case the Section alleged that employees in state offices that provide public assistance, and employees in state-funded programs serving persons with disabilities, failed to distribute voter registration applications. The Section also alleged that such offices failed to train and monitor their employees to ensure that they distribute voter registration applications to clients and transmit completed applications to the state and local election offices. One of the six cases involved claims that the state failed to offer voter registration opportunities in both motor vehicle and public assistance offices. Specifically, the Section alleged that the state did not provide a voter registration form with the state’s driver’s license application form. The Section further alleged that employees in state offices that provide public assistance, and employees in state-funded programs serving persons with disabilities, failed to distribute voter registration applications, among other claims. Two of the six cases involved claims that local election jurisdictions failed to process and register voter registration applicants. For example, in one case the Section alleged that a local election office did not process voter registration applications submitted by applicants at least 30 days before an election in a timely manner, which resulted in eligible applicants not being able to vote in their appropriate precincts in that election. DOJ officials have provided various perspectives on the department’s NVRA enforcement efforts. For example, in October 2009, we reported that the Assistant Attorney General for the Civil Rights Division prioritized NVRA list maintenance cases from fiscal years 2001 through 2007. Specifically, we reported that, according to Voting Section officials, the department focused during this period on both ensuring states had a list maintenance program and ensuring that such programs incorporated required safeguards. In a 2013 report, the DOJ Office of Inspector General reported that Civil Rights Division leadership initiated an effort to enforce the NVRA’s list maintenance provisions in late 2004. The report further noted that Civil Rights Division leadership placed a higher priority on the enforcement of the NVRA’s ballot access, or registration, provisions between 2009 and 2012. Section officials we interviewed for this review did not identify any overall Section-wide priorities between fiscal year 2010 and fiscal year 2017 that focused specifically on either list maintenance or registration. These officials explained that the Section cycles through the various NVRA provisions over time, but provided limited details and did not directly attribute any increase in matters or cases over time to Section initiatives or priorities. Officials further noted that the Section pursued fewer NVRA related cases after 2010 in part due to resource limitations and other priorities within the Section. For example, officials stated that the Section handled a number of Voting Rights Act cases during this time, which required a significant amount of staff time and resources. In addition to initiating matters, and filing NVRA cases as a plaintiff, the Voting Section engaged in efforts to enforce the NVRA’s registration opportunity and list maintenance provisions by participating as an amicus curiae or “friend of the court” in eight NVRA cases from fiscal year 2001 through fiscal year 2017. The Section participated in seven of these eight cases between fiscal years 2012 and 2017. Four of the eight cases involved registration opportunity complaints and four involved list maintenance complaints. According to Voting Section officials, amicus participation increased in these years in part because it was a way for the Section to participate in cases in a manner which did not require a significant amount of resources. Specifically, officials stated that filing an amicus brief takes considerably less time and fewer staff resources than litigating a case. The Voting Section entered into five out-of-court settlement agreements with states (in lieu of filing a case) to address allegations of NVRA non- compliance between fiscal years 2008 and 2017. All five of the agreements were related to the law’s registration opportunity provisions. For example, in one settlement agreement, a state agreed to make modifications to its internet site and the forms, procedures, and electronic system used at its motor vehicle offices in order to meet the requirements of section 5 of the NVRA, which stipulates that states offer voter registration opportunities at state motor vehicle agencies. The state further agreed to produce a compliance plan to meet these goals and to develop and implement a mandatory NVRA training program, among other things. The agreement included monitoring procedures, such as requiring the state to provide DOJ with quarterly reports of the number of in-person driver’s license applications received and completed voter registration forms accepted and transmitted to county boards of elections. According to Voting Section officials, the determination of the appropriate type of enforcement action in a matter, such as a settlement agreement or court order, can depend on a range of factors. For example, officials stated that relevant factors can include the nature, scope, and length of the violation, the level of cooperation by relevant actors regarding remedies, and the authority of relevant officials under state law to take remedial actions. The NVRA settlement agreements we reviewed are all multi-year agreements and Section officials noted that they try to collaborate with the state or jurisdiction regarding the appropriate steps (e.g., generating monthly, quarterly or biannual reports) for measuring and monitoring compliance during the period of the agreement. Section attorneys monitor settlement agreements by reviewing each required report and conferring with managers about progress towards compliance. According to Voting Section officials, the Section engaged in various efforts to assess state and local jurisdiction compliance with NVRA registration opportunity and list maintenance requirements, including conducting reviews of federal election administration and other data, and compliance investigations. Specifically, Section officials said that they conduct periodic reviews of the U S. Election Assistance Commission’s biennial Election Administration and Voting Survey (EAVS) to assess compliance with different NVRA provisions. For example, officials noted they may review EAVS data summarizing states’ motor vehicle agency driver license and voter registration transactions to help determine whether states are following NVRA section 5. In addition to using EAVS data, officials said they review publically available third party reports, which often include state specific registration data and other qualitative information about state processes. Section officials said this information can help them identify states that are potentially not in compliance with the NVRA. Officials also said that Section investigators have conducted observations at motor vehicle agencies and social services agencies as part of their efforts to assess and enforce NVRA compliance. Section officials noted that these efforts are not conducted on a regular schedule; rather, they are conducted periodically, on an intermittent, rolling basis. These officials said such efforts may lead them to request additional information from states, conduct compliance investigations, and initiate enforcement actions if necessary. For example: The DOJ Office of Inspector General reported that, in 2004, the Voting Section reviewed census and voter registration data for all 50 states to determine which states had more people registered to vote than the voting-age population. The Inspector General further reported that, based on the results of the research, the Section sent letters to 12 states requesting information on their efforts to remove ineligible voters from their registration lists, and ultimately filed two cases as a result of this enforcement initiative. In June 2017, the Voting Section sent letters to the 44 states subject to the NVRA requesting information related to states’ compliance with the law’s list maintenance provisions. Section officials stated that, as of March 2019, two actions have resulted from this effort: (1) the Section became a plaintiff-intervenor in a June 2018 case against Kentucky for having an inadequate list maintenance program; and (2) the Section entered into a February 2019 memorandum of understanding with the state of Connecticut regarding its efforts to identify registered voters who have died. Officials noted that the effort begun in 2017 does not have any specific time frames, goals, or objectives but that the Section is reviewing the data states provided and focusing detailed reviews on states whose data suggest possible non-compliance. Section officials said that in general, assessing compliance with NVRA section 8 (list maintenance) is more challenging than for the other sections, such as section 5 (voter registration opportunities at motor vehicles agencies). For registration opportunity provisions, they can send an investigator to the agency to observe whether the agency is offering people the opportunity to register as part of their standard transactions. However, officials noted there is no observation they can conduct to determine if list maintenance is occurring as required. As such, officials stated that DOJ is uniquely dependent on information and data from the states and local jurisdictions to indicate whether list maintenance efforts are taking place and what type. Officials further noted that they may have reduced time to analyze data or otherwise pursue more general enforcement efforts in time periods where the Section is overseeing a high number of defensive cases (ones in which the U.S. government is the defendant). Federal, state, and local authorities share responsibility for addressing allegations of election fraud. Within the federal government, DOJ has jurisdiction over election fraud investigations and prosecutions in elections where a federal candidate is on the ballot. In the absence of a federal candidate on the ballot, DOJ may have jurisdiction where facts exist to support the application of federal criminal laws that potentially apply to both federal and non-federal elections. According to DOJ officials, federal authorities would ordinarily defer to state and local authorities in deciding who would pursue an election fraud investigation or case because of states’ primary authority over the election process. DOJ’s Federal Prosecution of Election Offenses states that election fraud usually involves the corruption of one of three processes: the obtaining and marking of ballots, the counting and certification of election results, or the registration of voters. Within DOJ, the Public Integrity Section and U.S. Attorneys’ Offices maintain certain data on the election fraud matters and cases they initiate and prosecute. Within their respective databases, DOJ attorneys select a program category for each matter and case, which helps define the type of criminal act being investigated or prosecuted, for example, election fraud or health care fraud. U.S. Attorneys’ Offices use the program category “election fraud” for all election related charges; attorneys in the Public Integrity Section use either “election fraud” or “election crime other.” According to DOJ officials, categorization of matters and cases as election fraud (or any other category) is at the discretion of the investigating or prosecuting attorney based upon an examination of the facts. We refer to matters and cases that were either categorized as election fraud or election crime other, or included individual charges we identified as “election fraud related.” Election fraud related matters and cases in the DOJ databases we reviewed included charges brought under a wide variety of statutes, including those related to providing false information in registering or voting and vote buying (52 U.S.C. § 10307(c)) and voting by noncitizens (18 U.S.C. § 611), as well as more general charges such as the general federal conspiracy charge (18 U.S.C. § 371). From fiscal years 2001 through 2017, the Public Integrity Section initiated 1,408 matters, of which 33 were election fraud related, or about two percent of its overall matters. As shown in figure 6, the Section initiated 10 of the 33 election fraud related matters in fiscal year 2011, six in fiscal year 2013, four in fiscal year 2003, and four in fiscal year 2012. From fiscal years 2001 through 2017, the Public Integrity Section filed 695 cases; of which 19 were election fraud related, or about three percent of its overall caseload. As shown in figure 7, the Section filed election fraud related cases in five of those fiscal years, with seven of the 19 cases filed in fiscal year 2003 and five filed in fiscal year 2014. Public Integrity Section officials stated that the Section’s involvement in election fraud related matters and cases may vary over time depending on a variety of factors, including the number of complaints received and staffing levels within the Section. Officials stated that the Section allocates attorneys to work on election related matters and cases as needed, if resources allow. U.S. Attorneys’ Offices are required to consult with the Public Integrity Section with regard to all federal criminal matters that focus on corruption of the election process, in addition to federal patronage and campaign finance-related crimes. The Section reviews this information and consults with U.S. Attorneys’ Offices on their elections related work. U.S. Attorneys’ Offices may also request assistance with a case if they lack sufficient resources to prosecute a complex case, or if the office needs to recuse itself. If the Section does not have sufficient staff available, officials stated that they may not have the ability to offer assistance in investigating matters and prosecuting cases. In these circumstances, officials said that the U.S. Attorney’s Office would likely proceed with the case without the Section’s assistance, except in recusal cases. The Public Integrity Section initiated at least one election fraud related matter in 11 of 12 regional federal circuits as shown in figure 8. The Section initiated the most matters in the Sixth Circuit (10 of 33) and the Fifth Circuit (seven of 33). The Public Integrity Section filed election fraud related cases in four of the 94 federal judicial districts nationwide. These four districts are located in three states: Kentucky, Texas, and Massachusetts. Specifically, the Section filed 11 of its 19 cases in the Eastern District of Kentucky; five cases in the Southern District of Texas; two cases in the Western District of Kentucky; and one case in the District of Massachusetts. The Public Integrity Section prosecuted election fraud related cases with charges under six statutes. As shown in table 2, the Section most frequently brought charges under 52 U.S.C. § 10307(c) which was charged in 17 of the 19 cases the Section filed. This statutory provision prohibits giving false information for purposes of registering or voting, vote buying, and conspiring to vote illegally. Public Integrity Section officials stated the Section did not focus its efforts on particular types of election fraud, but vote buying (generally charged under 52 U.S.C. § 10307(c)) was the most frequent type of election fraud related crime the Section prosecuted during the period of our review. Officials said vote buying is the most common type of election fraud related crime that has come to their attention in recent decades and noted that it tends to occur in communities that are more insular and isolated and have higher levels of poverty. For example, officials observed that in rural communities with high levels of poverty, some residents may be more vulnerable to vote-buying efforts due to their difficult circumstances or the power of local officials who seek to buy votes to provide or cut off needed services. Officials stated that matters and cases tend to be geographically concentrated because, while the Section does not have any formal initiatives in particular circuits or districts, they are in close contact with U.S. Attorney’s Offices nationwide and can offer additional assistance in those areas that may be more vulnerable to recurring or frequent election fraud. Example of Public Integrity Section Election Fraud Prosecutions Seven cases filed in the Eastern District of Kentucky in fiscal year 2003, in which 10 defendants were charged, concerned the 1998 primary election for multiple Knott County government positions and candidates, including county judge executive (the county executive) and county clerk. The 1998 primary election also included a contest for federal office (U.S. Senator). The presence of a candidate for federal office on a ballot is sufficient to establish federal jurisdiction under most election fraud related statutes as the federal candidate’s election could be, or could appear to be, tainted by the fraud. From fiscal years 2001 through 2017, U.S. Attorneys’ Offices initiated more than 2.2 million criminal matters (i.e., investigations), of which 525 were election fraud related, or 0.02 percent of their overall matters. As shown in figure 9, U.S. Attorney’s Offices initiated between 11 and 65 election fraud related matters each year during this time period. U.S. Attorneys’ Offices initiated the most election fraud related matters in fiscal years 2003 (44), 2004 (53), 2005 (65), and 2011 (46). The percentage of election fraud related matters of all matters initiated ranged from 0.01 percent to 0.06 percent. From fiscal years 2001 through 2017, U.S. Attorneys’ Offices filed just over 1 million criminal cases. Of these, 185 cases were election fraud related, or 0.02 percent of their overall caseload. According to officials from EOUSA, which provides guidance, direction, and oversight to the U.S. Attorneys’ Offices, election fraud was one of the least frequent crimes addressed by U.S. Attorneys’ Offices. In fiscal year 2017, the most frequent felony cases filed by U.S. Attorneys’ Offices were for immigration, drugs, and violent crime offenses. Officials further noted that election fraud related cases were taken seriously and thoroughly investigated when facts supporting such charges were uncovered. As shown in figure 10, U.S. Attorneys’ Offices filed the most election fraud related cases in fiscal years 2003 through 2005, and in fiscal years 2007 and 2017, with 15 or more cases filed each fiscal year. U.S. Attorneys’ Offices filed fewer than five election fraud related cases during fiscal years 2001, 2002, and 2015. The percentage of election fraud related cases of all cases filed ranged from less than 0.01 percent to 0.03 percent. From fiscal years 2001 through 2017 U.S. Attorneys’ Offices initiated at least one election fraud related matter in 85 of the 94 federal judicial districts. As shown in figure 11, three districts cumulatively accounted for 145 out of 525 matters, or approximately 28 percent of all election fraud related matters initiated. Of these three, two judicial districts, the Southern District of Florida and the Eastern District of Kentucky accounted for nearly one quarter of all election fraud related matters U.S. Attorneys’ Offices initiated. About half of the 185 election fraud related cases filed by U.S. Attorneys’ Offices occurred in three of the 94 federal judicial districts. As shown in figure 12, the Southern District of Florida filed 42 cases (23 percent), the Eastern District of Kentucky filed 36 cases (19 percent), and the Eastern District of Wisconsin filed 15 cases (eight percent). U.S. Attorneys’ Offices filed the remaining cases (92 cases, or 50 percent) in 42 federal judicial districts; of these, 20 districts had only one election fraud related case during the time period. EOUSA officials said that there could be a number of reasons why cases occurred more frequently in some districts than others. These officials noted that individual U.S. Attorneys utilizing their prosecutorial discretion may have taken an interest in election fraud or encountered evidence of a series of election fraud related crimes that generated a number of matters or cases. For example, according to the respective U.S. Attorneys’ Offices: In the Southern District of Florida, a 2004 case involving allegations of noncitizen voting resulted in the U.S. Citizenship and Immigration Services referring a series of additional similar investigations to the U.S. Attorney’s Office; In the Eastern District of Kentucky, a drug investigation in 2003 revealed evidence of vote buying that led to a series of vote buying cases; and In the Eastern District of Wisconsin, 14 of the 15 cases filed were uncovered in a joint investigation regarding the results of the 2004 presidential election, which showed a discrepancy between the number of ballots counted and individuals voting in one Wisconsin county. That investigation ultimately determined the discrepancy was caused by clerical error, but also uncovered 10 individuals who voted despite being ineligible due to their felon status and four who voted more than once. U.S. Attorneys’ Offices utilized approximately 100 different statutes in bringing charges in election fraud related cases. Table 3 shows the statutes charged in 15 or more election fraud related cases filed by U.S. Attorneys’ Offices. The most frequently charged statute was 52 U.S.C. § 10307 (prohibited voting acts), charged in 52 cases, with subsection (c) (false information in registering or voting and vote buying) charged in 38 of those cases. The next three statutes of 18 U.S.C. § 371 (conspiracy), 18 U.S.C. § 1001 (false statements), and 18 U.S.C. § 611 (voting by noncitizens) were each charged in 38 or more cases. EOUSA officials explained that U.S. Attorneys’ Offices select charges based on the specific facts and circumstances of a case. These officials noted that the offices may use some statutes, such as 18 U.S.C. § 371 and 18 U.S.C. § 1001, more frequently in cases due to their generality, which makes them widely applicable to different types of criminal conduct. Each of the selected data sources we reviewed is one tool election officials may use to maintain their voter registration lists. These selected data sources are used to identify (1) registrants who move—U.S. Postal Service National Change of Address (NCOA), Interstate Voter Registration Crosscheck Program (Crosscheck), and returned mail; (2) deceased registrants—the public version of the Social Security Administration Death Master File (DMF) and state vital records; and (3) registrants with disqualifying felony convictions—U.S. Attorneys’ records on felony convictions. State and local election officials may use a variety of other databases or lists (data sources) to identify ineligible registrants who should be removed from voter registration lists, and state policies and procedures for using various data sources to identify and remove registrants from voter lists vary. Despite variations, election officials with whom we spoke stated that list maintenance—including the use of the selected data sources—provides benefits such as cost savings, smoother Election Day processes, reductions in administrative burden, and fewer opportunities for election fraud. Moreover, election officials told us that each of the selected data sources helps improve voter registration list accuracy, despite some limitations. For example, officials identified benefits from using these data sources, such as helping reduce the number of address errors on voter registration lists and helping identify and remove registrants who have moved outside of the election jurisdiction, are deceased, or have a disqualifying criminal conviction from voter registration lists. Officials also identified limitations with using these selected sources. In particular, three of the six selected data sources consist of administrative records collected for purposes other than voter registration, which can present some challenges when election officials use these sources to maintain their voter registration lists. For example, election officials noted that such data sources may inaccurately indicate that registrants moved unless election officials conduct additional work to verify the information. In addition, these data sources may not include the records for some registrants who are deceased and should be removed from the voter registration lists. Appendix III includes a description of a range of data sources states may use to maintain their voter registration lists. With regard to possible election fraud, state officials from all five selected states we visited noted that list maintenance activities in general help to identify or prevent election fraud because accurate and complete voter registration lists make it more difficult for individuals to commit fraud. Specifically, duplicate registrations—more than one registration for the same person across election jurisdictions—and ineligible registrations, such as those for deceased individuals, if present in voter registration lists, may provide opportunities for a person to vote more than once or vote using someone else’s identity. Thus, registration lists that contain one registration for each eligible registrant with accurate and current identifying information help to prevent election fraud from occurring. The majority of election officials we interviewed did not specify any one data source used to identify election fraud; however, state officials from Michigan and Oregon noted that the limited instances of election fraud in their states, in their view, is in part the result of their strong voter registration list maintenance efforts which have helped to reduce opportunities for fraud. In using data sources as a tool for maintaining voter registration lists, state and local election offices utilize data-matching procedures by which attributes of one registration record are compared to attributes of another record from another database or list to identify registrants who should be removed from voter registration lists under the NVRA’s removal categories. States are required to have computerized statewide voter registration lists, which allow election officials to conduct electronic data matching of their voter registration list to other databases or lists. These other databases or lists may include federal or state administrative records, interstate databases, and local lists or other information. Information on Data Matching Procedures Procedures for determining that a voter registration record is a “match” to another record may vary across states, local election offices, and interstate data matching programs. In general, a “match” should accurately identify the same individual across the two data sources being matched. However, data matching may result in improper indications of a match when a non-match should be indicated (false positives). False positive matches pose risks that election officials may improperly remove registrants from voter registration lists. Data matching can also result in improper indications of a non-match when a match should be indicated (false negatives), posing risks that election officials may fail to remove ineligible registrants from voter registration lists. According to a National Academy of Sciences report, the quality of the underlying data (from either the voter registration list or other data sources used for matching) may contribute to false positive or false negative matches. National Academy of Sciences, Improving State Voter Registration Databases, The National Academies Press, 2010. Further, matching procedures may differ with regards to how data in specific data fields are compared across databases to determine a match. For example, some procedures may require that the name from the voter registration list exactly match the name from the other data source (e.g. each letter, hyphen, space, or apostrophe must match). An exact match requirement would not accept as a match the name entries “Mary Jones-Smith” and “Mary Jones Smith”, even if all other data fields match across data sources and the entries represent the same individual, thus resulting in a false negative match. Below we discuss in detail the selected data sources and their benefits and challenges, as identified by literature we reviewed and election officials with whom we spoke. According to reports we reviewed, registrants who move from one election jurisdiction to another jurisdiction within the state or to another state account for the majority of ineligible registrants and duplicate registrations on voter registration lists. When individuals register to vote, their voter registrations are linked to their residential address. This connection between a voter’s registration and residence is intended to ensure reliable and accurate voter registration lists, and to ensure that voters only vote for races and ballot questions that affect the communities in which they live. According to the 2016 Election Administration and Voting Survey (EAVS), the most common reason for a registrant’s removal from the rolls was cross-jurisdiction change of address (31.1 percent of removals), followed by registrants failing to respond to a confirmation notice sent as part of the NVRA process and subsequently not voting in the following two federal elections (26.1 percent of removals). As previously discussed, under the NVRA, data that indicate a registrant’s change of address and a potential move can be used to start the address confirmation notice process, but cannot, on their own, result in the automatic removal of registrants from voter registration lists. The NCOA database comprises change-of-address records with the names and addresses of individuals, families, and businesses who filed a change of address with the U.S. Postal Service. Election officials can access the NCOA data by obtaining a license to directly receive the data from the U.S. Postal Service or having their voter registration list processed by a licensed third-party service provider. Election officials in the five states we visited compare selected records or the entire voter registration list against the NCOA database at the state or local level and at varying frequency to identify registrants who have potentially moved and to start the address confirmation and registrant removal process. For example Nebraska, Oregon, and Virginia state election officials said that they compare their statewide voter registration lists to NCOA on a bi-annual, monthly, and annual basis, respectively, to identify registrants who have potentially moved. In contrast, Florida and Michigan officials said they do not use NCOA data at the state level, though state laws provide local election officials the option of comparing their local jurisdiction’s voter registration list to NCOA when they conduct list maintenance activities related to changes in address. Although initial data comparisons of NCOA with the voter registration lists can be conducted at either the state or local level, in all of the states we visited when the results of the NCOA data-matching indicated a potential move, local election officials managed the results of the confirmation notices that were sent to registrants to confirm their address. Local election officials subsequently updated addresses on the voter registration lists with responses they received from the confirmation notices, or flagged registrants for potential removal if the registrants did not respond to the confirmation notice or the notice was returned undeliverable. State election officials from Nebraska, Oregon, and Virginia, and local officials from five of the jurisdictions we visited, reported that the primary benefit to using NCOA data is that it helps them to maintain accurate voter registration lists by (1) providing current and accurate addresses for their registrants, and (2) identifying registrants who have potentially moved and no longer reside in the voting jurisdiction. For example, local officials in one jurisdiction reported that they mailed approximately 60 percent fewer confirmation notices in 2017 compared to 2010 due to improvements in the accuracy of address information in their voter registration lists after using NCOA data during this period. Local officials in another jurisdiction reported they used the NCOA data as part of a one- time list maintenance effort, which generated over 100,000 confirmation mailings and resulted in the removal of a number of ineligible voters who no longer resided in the jurisdiction. Election officials also noted that using NCOA data to update voter registration lists may result in administrative efficiencies such as a more efficient election administration process and cost savings. For example, state officials from Oregon, a vote-by-mail state, said that NCOA data help them to maintain clean voter registration lists by providing current and accurate addresses for their registrants, which reduces mailing costs incurred from sending ballots to individuals who have moved out of the state. Further, local officials from one jurisdiction said that using NCOA data helped to reduce the number of address errors in the poll books and, as a result, decrease the number of registrants voting by provisional ballots on Election Day. A report we reviewed and election officials we interviewed cited a number of limitations to using NCOA data for voter registration list maintenance purposes. Specifically, in 2015 the U.S. Postal Service Office of the Inspector General reported that the NCOA data do not capture all change of address information because people do not always notify the U.S. Postal Service when they move. As a result, election officials may not be able to identify registrants who do not report changes of address to the U.S. Postal Service. Another limitation election officials cited is that an indication of a change in address in NCOA data does not necessarily reflect a change in residence, which is what determines the eligibility of a registrant to vote in a given election jurisdiction. According to U.S. Postal Service officials, the main purpose of the NCOA database is to maintain current and updated addresses for mail delivery and a change of address form may reflect a change in mailing address rather than a permanent change in residence. Nebraska, Oregon, and Virginia state officials and officials from three local jurisdictions reported that they have difficulty determining whether a registrant’s change in address as indicated in the NCOA data is a permanent change in residence or a change in mailing address due to a temporary move or other mailing needs. For example, military personnel may prefer to maintain their voter registration at their home of record. Upon assignment to another duty location they may file a change of address with postal authorities for mailing purposes, even if it is not a change of residence for voting purposes. Officials from two local jurisdictions reported similar issues for individuals who retain residency in the jurisdiction while attending college outside the jurisdiction. Further, registrants who had vacation homes outside the jurisdiction in the summer or winter months could be identified as registrants who potentially changed residences on a permanent basis using the NCOA data, according to Nebraska election officials. As a result of the potential difference between mailing and residential addresses, Virginia state election officials and election officials from two local jurisdictions reported that registrants may be inaccurately flagged for confirmation mailings. They told us that registrants would not be automatically removed after being flagged for confirmation mailings; however, they would be required to respond to the mailing or vote in one of the next two federal elections, as prescribed by the NVRA, to stay on the voter registration list. Officials also told us that they may have to take additional steps to use NCOA data to identify registrants who potentially moved and to update voter registration lists. For example, officials from one local jurisdiction that matches its county voter registration list to NCOA data noted that it can take a significant amount of time and resources to standardize their voter registration data to the NCOA format and to calibrate their data matching procedures to avoid false positive matches. Such false positive matches would inaccurately indicate an address change. These local officials said that they take steps to ensure that they do not get an indication of a change in address based on the standardization of an address (e.g. a “Street” to “ST” difference in address between the two data sources). Oregon state election officials and officials from one local jurisdiction further noted that they may have to do additional work to determine the appropriate election jurisdiction to which the address in the NCOA data should be assigned. Officials explained that some street addresses or buildings, like apartment complexes, cross election jurisdiction boundaries, which makes it difficult to determine within which election jurisdiction an address or a specific unit of an apartment complex falls. Oregon state officials said that local tax assessor data may help election officials reconcile these jurisdictional boundary issues. Crosscheck is an interstate data sharing program that compares participating states’ voter registration lists against one another to identify registrants who are registered in more than one state, which may indicate a move, and to identify individuals who may have voted in more than one state. The Crosscheck program began in 2005 with four participating states—Kansas, Iowa, Missouri, and Nebraska—and had grown to include 31 participating states by 2016. To participate in the Crosscheck program, each state signs a memorandum of understanding upon joining the program. Then, in January of each year, member states provide information such as full name, date of birth, and address for registered voters, as well as turnout data for the previous calendar year to Crosscheck program administrators—the Kansas Secretary of State’s office—in a prescribed format. Using the information provided by member states, Crosscheck program administrators return to each participating state a list of registrations in that state that share the same first name, last name, and date of birth, with a registration in another participating state. Crosscheck results also include other identifying information that varies depending on whether the member states provided the data. There are no membership or annual fees associated with joining or participating in Crosscheck. Of the states we visited, Michigan, Nebraska, and Virginia participated in the Crosscheck program for multiple years, while Oregon and Florida each participated once in 2012 and 2013, respectively. Oregon and Florida state officials explained that they did not use the Crosscheck data they received to conduct any voter registration list maintenance activities. Michigan, Nebraska, and Virginia state officials said that they received and processed Crosscheck data at the state level before sending a subset of results to the local jurisdictions to conduct additional verification and list maintenance activities. According to some state and local election officials we interviewed, Crosscheck data can be beneficial as one of the data sources used to identify registrants who may have moved out of state or whose moves are not captured by other data sources. Specifically, officials from four local jurisdictions told us that using Crosscheck data in conjunction with other data sources, such as the NCOA, helps keep voter registration lists accurate. Further, state election officials from Virginia and election officials from one jurisdiction reported that the fact that neighboring states participate in the Crosscheck program is particularly beneficial to them because their residents are more likely to move to neighboring states and the Crosscheck data may capture the change in residence if these residents also registered to vote in the neighboring states. Nebraska state officials also noted that Crosscheck data complement the NCOA change of address data. In particular, Crosscheck data can provide information on registrants who did not record change of address information under NCOA, who had not responded to a notice sent as a result of NCOA data and had moved a second time, or whose moves were not recent and may not be captured in the most recent change of address information provided by NCOA. Nebraska state officials noted that the Crosscheck data were particularly helpful in this manner the first year that Nebraska participated in Crosscheck and whenever a new state joined the program. In addition, election officials from Nebraska and state officials from Michigan identified Crosscheck data on possible instances of double voting as a source which could potentially help determine whether an individual might have voted in two or more states. For example, officials from two local jurisdictions said that they identified a few potential instances of double voting using Crosscheck data. They referred these instances of potential double voting to their Secretary of State. According to reports we reviewed and state officials we interviewed in all five states we visited, Crosscheck data contain numerous matches when a non-match should be indicated (false-positive matches) because the program uses matching criteria that rely on data elements, such as names and birth dates, that may be shared by more than one person. Specifically, the Crosscheck program matches participating states’ voter registration information by comparing registrants’ first name, last name, and date of birth. However, according to reports we reviewed, the odds are sufficiently high that two registrants could have the same name and birth date in groups as large as statewide (or multistate) voter registration lists. Nebraska state officials noted that when there were four participating Crosscheck states in 2005, a match indicating a duplicate registration was more likely to be a valid match (rather than a false positive); however as the number of participating states increased, the quality of the matched results has dropped substantially. Oregon state officials told us that they submitted data to the Crosscheck program in 2012 and that many of the resulting 20,000 potential duplicate registration matches were false-positive matches. Florida state officials also expressed concern about the reliability and quality of the matching criteria, in addition to the number of false positive matches in the data they received. In addition, a study on double voting found that Crosscheck data may not provide enough information for election officials to determine whether a match indicating potential duplicate registrations or double voting is valid. As previously discussed, Crosscheck results for potential duplicate registrations are based on a match of the first name, last name, and date of birth. Crosscheck results provided to participating states may also include additional information—such as registrants’ middle name, suffixes, registration address, and the last four digits of a registrant’s Social Security number, if available—which election officials can use to help determine whether a match is a valid indication of a duplicate registration. In particular, the last four digits of the Social Security number can help distinguish between two distinct individuals who happen to share the same first name, last name, and date of birth. Using Crosscheck data returned to Iowa in 2012 and 2014, the study found that two-thirds of potential duplicate registrations identified by Crosscheck data did not include the last four digits of the Social Security number associated with at least one of the registration records in the match. Thus, the study concluded that more often than not, an election administrator would not have enough information to distinguish which matches are valid indications of duplicate registrations. Further, Nebraska state officials noted that the reliability of the data provided by participating states can affect the reliability of Crosscheck information on double voting. For example, Nebraska state officials reported that one state incorrectly sent Crosscheck its 2014 voting history data the year participating states were to provide their 2016 data to the Crosscheck program. These officials noted that the incorrect voter history data made it appear as though many people had double voted. Nebraska officials said that once they identified this issue, they omitted any matched results involving the state that had provided the 2014 data from their review of registrants who potentially double voted. According to the Crosscheck 2014 Participation Guide, processing the duplicate registrations and researching possible double votes require a commitment of time from state and local officials. State election officials from Michigan, Nebraska, Oregon, and Virginia and officials from two local jurisdictions told us that they have spent a significant amount of time and staff resources to review the Crosscheck data and determine which matched records represent valid matches. State officials from the three states that participated in Crosscheck for multiple years (Michigan, Nebraska, and Virginia) said they implemented additional criteria to refine the Crosscheck data they received in order to identify valid matches of potentially duplicate registrations and send confirmation notices, according to the NVRA requirements. For example, Michigan state officials said that they further filter the Crosscheck results they receive to determine valid potential matches of duplicate registrations. Specifically, they filter Crosscheck results to include duplicate registrations where the registrants’ first names, middle initials, last names, dates of birth, and last four digits of Social Security numbers are an exact match. In addition, state election officials review the registration dates provided in the Crosscheck results to confirm that the registrant’s most recent voter registration activity occurred outside of Michigan before providing a refined list of valid potential matches to responsible local officials who conduct the address confirmation process. In its June 2017 Annual List Maintenance Report, Virginia state officials reported that they also review whether the last four digits of the Social Security number on Crosscheck results they receive match, to determine valid potential matches of duplicate registrations. While election officials from two jurisdictions we visited identified Crosscheck as a source which helped them identify potential instances of election fraud, such as instances of double voting, Nebraska state officials also noted the data were not generally reliable for these purposes without additional investigation. According to one study we reviewed, Crosscheck data on both double voting and duplicate registrations yield a high number of false-positive matches. Additionally, in another report, the New Hampshire Department of State found that of approximately 90,000 match records of duplicate registrations New Hampshire received from Crosscheck in 2017, only a small portion of the records were considered potential instances of double voting. Election officials can use the returned mail from targeted list maintenance mailing efforts and returned “undeliverable” mail from other mailings to registrants to send address confirmation notices to registrants who have potentially moved outside the election jurisdiction. These confirmation notices are subsequently used to update addresses on the voter registration lists with results of the confirmation mailing or flag registrants for potential removal. Specifically, targeted list maintenance mailing efforts may include sending a notice to all or a group of registrants in order to determine whether the registrant may have moved from the address on record. For example, Florida law states that local election officials can send notices to registrants who have not voted in the last 2 years and who have not made a written request that their registration be updated during the two year period. Targeted list maintenance mailing efforts may result in either a response from the registrants or returned undeliverable mail. Returned undeliverable mail occurs when the U.S. Postal Service cannot deliver mail to the address specified on the label, indicating a potential change in the registrant’s address and therefore residence. In addition to targeted list maintenance mailings, election offices may send other notices—such as sample ballots, or information about changes in polling locations—which may also generate returned undeliverable mail. See figure 13 for an example of other voter registration notices (not part of a targeted list maintenance effort) that may be returned to election officials as undeliverable and therefore indicate a potential move. Election officials from all five states we visited use returned mail from targeted list maintenance mailing efforts, or from other mailings to voters, to update registrants’ addresses or to send a notice to the registrants to confirm their address. According to Nebraska state election officials, returned undeliverable mail is a valuable tool for identifying registrants who may have moved. Local election officials we spoke with also said that returned undeliverable mail can provide them with a timely indication that a registrant has potentially moved. Furthermore, election officials told us that because mailings can be conducted on a periodic basis, processing returned mail at the time of receipt can help election officials distribute the list maintenance workload throughout the year. Specifically, election officials from four local jurisdictions said that returned undeliverable mail from voter notices sent to registrants periodically throughout the year is usually a more recent indicator of registrants’ changes in address compared to largescale list maintenance activities such as an annual mailing based on NCOA data. Further, officials from one local jurisdiction also noted that staying on top of returned undeliverable mail throughout the year helps reduce the workload during the state’s annual NCOA confirmation mailing, which would otherwise be too big to manage if the jurisdiction only processed address changes once a year. According to reports we reviewed as well as officials we interviewed, returned undeliverable mail may not be a reliable indicator that a person has moved, which can result in an inflation of the number of registrants who are flagged as inactive. For example, in 2015, the U.S. Postal Service Office of the Inspector General reported that approximately 60 percent of returned undeliverable mail is a result of the mail not getting delivered by postal service employees or insufficient address information on the mail, as opposed to the registrant having moved without notifying the U.S. Postal Service. Further, according to one report we reviewed, a registrant may not have received the mailing, or the mailing may be returned undeliverable for a number of reasons, including that the registrant may be temporarily away from his/her permanent residence; may not be listed on the mailbox of the residential address such as when the registrant shares an address with roommates or family members; or may live in a non-traditional residence such as homeless shelter or government building that will not accept mail for residents. In addition, Virginia state officials noted that using returned undeliverable mail can inflate the number of registrants who are flagged as inactive and can also result in additional costs. Specifically, these state election officials told us that they usually have a low response rate from registrants for mailings, including targeted mailings for list maintenance purposes or confirmation mailings. Registrants who are sent a confirmation notice or do not respond to confirmation mailings are then generally flagged as inactive. Nebraska state officials said that having inactive registrants on the registration lists has resulted in costs to local jurisdictions in the past because local officials were formerly required to mail a ballot to all registered voters, including those that were on the inactive list, when a special election was conducted by mail. Further, local election officials in one state said that inflated numbers of inactive registrants on voter registrations lists may result in fewer than needed voting precincts, to the extent that election officials determine the number of precincts based only on the number of active registrants on the lists. The NVRA provides for states to remove deceased registrants from registration lists by reason of death. This may be carried out by the state’s department of elections, local election jurisdictions, or a combination of the two, as provided by state law. According to the 2016 EAVS, states removed over 4 million registrants due to death from November 2014 through November 2016, which accounted for 24.6 percent of the total number or registrants removed from voter registration lists. According to a National Association of Secretaries of State 2017 report, in most states, information on deceased registrants is provided by a state office of vital statistics, the state department of health, or a similar state-level entity. Additionally, the report notes that a number of states permit election officials to remove a deceased registrant using information from sources such as obituary notices, copies of death certificates, and notification from close relatives. The public version of the DMF contains nearly 101 million records of deaths reported to the Social Security Administration from 1936 through March 1, 2019. It is a subset of the Social Security Administration’s full death file; it does not include state-reported death data, but includes other death data reported by family members, funeral directors, post offices, financial institutions, and other federal agencies such as the Department of Veterans Affairs and Centers for Medicare & Medicaid Services. The public DMF accounts for about 19 percent fewer death records than the full death file. The Social Security Act limits the sharing of the full death file to federal benefit-paying agencies, and other specifically enumerated purposes. Generally, DMF records include the Social Security number, full name, date of birth, and date of death of deceased individuals. Agencies or other entities, including election administrators, having a legitimate business purpose for the information can purchase the DMF from the National Technical Information Service of the U.S. Department of Commerce, which is authorized to distribute the DMF. Subscribers to the DMF are required to purchase monthly or weekly updates to the DMF to ensure that the records are up-to-date. Of the five states we visited, Florida, Michigan, Oregon, and Virginia compare their statewide voter registration list against DMF data on a regular basis, and Nebraska used the data once in 2014, to identify and remove registrants who had died. Specifically, Florida and Michigan directly receive the DMF data and conduct data-matching with their state’s voter registration list to identify deceased registrants on a weekly basis. Oregon and Virginia use DMF data through their participation in the Electronic Registration Information Center (ERIC) program. On a monthly basis, ERIC provides Oregon and Virginia state election officials a report on their deceased registrants based on matches of DMF data with these states’ voter registration lists. States’ procedures for removing deceased registrants from the voter registration list vary, depending on requirements outlined in state law. For example, Virginia state law provides that local election officials have the authority to determine the qualification of an applicant for registration. Further, the Virginia law requires election officials to send a cancellation notice once a voter registration record is cancelled due to death. As a result, Virginia state officials forward all valid matches of potentially deceased registrants to the responsible local official who reviews the match, marks the registrant as deceased in the voter registration database, cancels the registration, and sends a cancellation notice. In contrast, Michigan law allows either state or local officials to cancel a voter registration upon receipt of reliable information that the registrant is deceased. In addition, according to Michigan state election officials we met with, there is no legal requirement for officials to send notices of cancellation due to death. Michigan state election officials told us that they cancel voter registrations based on data-matching with DMF data at the state level. State election officials from all five selected states and officials from one local jurisdiction reported that they have found DMF data to be useful for identifying registrants who have died. Further, state election officials from four selected states stated that the DMF data are accurate and reliable. For example, officials said that they have experienced very few instances where they have had to reverse cancelled registrations because a registrant was incorrectly identified as deceased based on DMF data. Nebraska and Oregon state officials also noted that DMF data are particularly useful for identifying registrants who died out of state. Officials said that out-of-state death information would not be captured by other data sources they use, such as state vital records data. In addition, Michigan state officials noted that historically they would receive notification of a person’s death closer to the date of death when using DMF data than when using death data from the state vital records office. We previously reported that state-reported deaths, which the DMF does not include, are expected to account for a larger proportion of all Social Security Administration death records over time. As a result, we reported that agencies that purchase the DMF, including election offices, will likely continue to access fewer records over time as compared with those government agencies that obtain the Social Security Administration’s full death file. We also reported that because the deaths reported by states are generally more accurate than other death information reported to the Social Security Administration by post offices, financial institutions, and other government agencies, it is likely that agencies using the DMF could encounter more errors than agencies using the Social Security Administration’s full death file. According to Social Security Administration officials, Social Security death data are accurate when used to administer the Social Security Administration benefit programs, which includes removing deceased individuals from the beneficiary rolls and informing surviving spouses and children of their eligibility for benefits. Virginia state officials further noted that DMF death information can be less timely in identifying an individual as deceased when compared to state death records because state records are collected during the death certification process while the Social Security Administration relies on the transmission of information after the death certification from other entities, such as other government agencies, to identify an individual as deceased. Election officials can also use state vital records to identify and remove registrants who are deceased from their voter registration lists. Due to the federal requirement for state election officials to coordinate with the designated state agency responsible for compiling records of deaths, most states receive state level information on deceased registrants from their state office of vital statistics. State death records are collected electronically by most states, and maintained in each state’s Electronic Death Registration System. As of December 2018, 46 states, the District of Columbia, and Puerto Rico used an Electronic Death Registration System to collect and maintain death data within their jurisdiction. All five states we visited receive data on deceased individuals at varying intervals from their state vital records office and match these records to the statewide voter registration list to identify and remove deceased registrants. For example, on a daily basis, Florida state election officials receive state death data electronically from the Florida Bureau of Vital Statistics. They use the information to identify potentially deceased registrants and provide a list of these individuals to local election officials. Nebraska state election officials receive state death data from their state department of health on a weekly basis, Oregon and Virginia receive death information from their respective state departments of health on a monthly basis, and Michigan officials said they receive the information periodically, on either a weekly or bi-weekly basis. According to state election officials from Florida, Nebraska, Oregon, and Virginia and local election officials from four of the jurisdictions we visited, state vital records on deceased individuals are generally accurate and reliable, in part because state vital records data are reviewed and validated. Specifically, state vital records data on deceased individuals are linked to information on the death certificate which is validated by authorized persons, such as physicians and funeral directors, during the death certification process. Virginia state officials said that in comparison, other sources such as the Social Security DMF data may include reported deaths that are not directly linked to the death certification, from entities such as post offices and financial institutions. Additionally, officials from one jurisdiction told us that state death records are helpful in identifying people who died in another jurisdiction in the state. Further, officials from this jurisdiction noted that in the past they reviewed obituaries to identify deceased registrants, but that they have seen a decline in the use of obituaries to announce deaths and state death records help fill the information gap previously provided by obituaries. Nebraska state officials also noted that state death records can help prevent fraudulent registrations because state officials are able to check new registrations against death records received from the state health department. Nebraska state officials and officials from two local jurisdictions said that one limitation of state death records is that they generally only include information on deaths that have occurred in the state, and as a result election officials lack death records for residents who died out of state. From our interviews with the state vital records officials in the states we visited and information we reviewed on national death sources, we learned that, in some states, state death records may include information on deaths that occurred out of state, through the state’s participation in interstate data exchanges. Additionally, while some state officials found state death records timely for updating voter registration lists, Michigan state officials said their state death records were not as timely as DMF data. Specifically, Michigan state election officials said they used to receive notification approximately six months after a person’s death when using state death records, compared to within two weeks of death using DMF data. Officials explained that the lag in the death notification when using the state death records was due to low participation rates in the state’s Electronic Death Registration System when the system was first implemented. Michigan state election officials noted that state death records have improved and are timelier as the participation rate in the state Electronic Death Registration System has increased in recent years. Oregon state officials also noted that state death records may be less timely than the data counties receive from their local health departments, and thus local election officials may have received notice of an individual’s death from the county health department prior to receiving the state vital records data. State laws regarding the voting eligibility of individuals with a felony conviction vary. In some states, individuals who were previously convicted of a felony are not permitted to vote unless they are pardoned, or their voting rights are specifically restored by the government; in other states, the right to vote is reinstated automatically at the end of the individual’s sentence or after a designated period of time following the end of the sentence. Additionally, in some states, individuals with felony convictions may vote if they are on probation or have been granted parole; and, in two states, felons are allowed to vote even while incarcerated. Election officials are generally required to remove registrants with a felony conviction from voter registration lists, in accordance with state law. U.S. Attorneys are required by law to notify the states’ chief election officials of felony convictions in federal court. The notices must contain a person’s name, age, residence address, date of entry of the judgment, a description of the offenses of which the individual was convicted, and the sentence imposed by the court. Election officials from all five states we visited said that they receive records from U.S. Attorneys on residents who are convicted of a federal felony. Florida, Nebraska, and Virginia use this information to remove registrants from their voter registration lists given the nature of their state laws, which restrict voting eligibility after a felony conviction until rights are restored or for a period after completion of the sentence. Michigan and Oregon prohibit individuals from voting while serving their sentences after conviction, but voting rights are automatically reinstated once a person is released from prison. As such, state officials from Michigan told us that they do not use U.S. Attorneys’ records to remove voters from their voter registration lists. Oregon officials noted they use U.S. Attorneys’ records on federal felony convictions to change a registrant’s status to “inactive.” Election officials from three states in our review that use U.S. Attorneys’ felony conviction records to remove registrants from voter registration lists said that this information was valuable, as they would not be able to acquire information about federal convictions from state sources. While federal conviction information can be helpful to election officials, an official from one local jurisdiction said that it can be difficult to determine whether the individual identified by a U.S. Attorney’s Office as having a federal conviction is the same person as the registrant. This is because criminals may have used aliases or provided incorrect Social Security numbers when registering to vote, which results in a less confident match. In addition, the information state and local officials receive on federal convictions is not required to include an individual’s projected date of release or date of sentence completion, which state and local officials from Florida and Nebraska said could help them determine whether the registrant is ineligible to vote and thus should be removed from voter registration lists. This makes it difficult for election officials to determine if the registrant’s sentence was already completed by the time they receive the information. In Nebraska, where voting rights are reinstated two years after a sentence is completed, election officials said it is initially difficult to know whether the individual’s voter registration is valid without the date of release or sentence completion. To mitigate limitations related to the lack of a projected release date or sentence completion date, Florida election officials said that they review case judgments which provide the details of the case, including date of sentence completion, to determine if the registrant’s sentence was completed and then check if the registrant’s rights were restored. Nebraska state election officials said they review court records and also noted that they would contact the local U.S. Attorney’s Office to obtain the federal release date for a particular registrant. We provided a draft of this report to DOJ, the U.S. Postal Service, the Social Security Administration, the Election Assistance Commission, the Crosscheck program, and election offices in the five states and ten local jurisdictions we visited. DOJ, the U.S. Postal Service, the Election Assistance Commission, and the Crosscheck program did not provide written comments. The Social Security Administration submitted a letter noting that it did not have any substantive comments, which is reproduced in appendix IV. We incorporated technical comments from DOJ, the U.S. Postal Service, the Social Security Administration, Crosscheck, and state and local officials as appropriate. We are sending copies of this report to the Attorney General, the Postmaster General, the Social Security Administration, the Election Assistance Commission, election offices in the five selected states and ten local jurisdictions that participated in our research, appropriate congressional committees and members, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in appendix V. In addition to the Department of Justice’s (DOJ) role in enforcing the National Voter Registration Act of 1993 (NVRA), the law allows a private party (a person or organization) who is aggrieved by a violation of NVRA provisions to bring a civil action against a state or local agency responsible for voter registration. In some cases, DOJ may participate in these private party cases by intervening on behalf of the plaintiff (as a plaintiff intervenor) or defendant, or by filing an amicus brief. The NVRA includes provisions that focus on both increasing opportunities for voter registration and improving voter registration list maintenance. Table 4 includes a summary of these provisions. To identify cases filed by private parties that included a claim under the NVRA, we searched an online legal database (Lexis Advance) for U.S. Circuit Courts of Appeals decisions from fiscal years 2008 through 2018 that contained the term “National Voter Registration Act.” We reviewed the decisions and also obtained and reviewed related case documents, including district court decisions, dockets, and complaints, to determine whether a claim had been filed under the NVRA and the nature of the claim, among other case information. We focused on cases that reached the federal appellate level because decisions issued by the U.S. Circuit Courts of Appeals create binding precedent for all of the districts in that circuit, among other considerations. We identified 19 cases that were filed by private parties with claims under the NVRA that reached the U.S. Circuit Courts of Appeals (federal appellate level) from fiscal years 2008 through 2018. Eleven of the 19 cases included claims that were related to NVRA provisions that require states to provide registration opportunities. Six cases included claims related to the NVRA requirement to remove voters from registration lists under specified conditions (list maintenance). Private parties filed 11 cases involving claims under the NVRA’s registration opportunity provisions that reached the U.S. Circuit Courts of Appeals. We reviewed the claims in each of the 11 cases and found that: five of the 11 cases involved a claim under section 5 related to voter registration opportunities at motor vehicle offices; four of the 11 cases involved a claim under section 7 related to registration opportunities at public assistance offices; two of the 11 cases involved a claim under section 6 related to mail-in registration application forms; and one of the 11 cases involved claims under section 8 related to the requirement that states register voters whose applications are received at least 30 days before an election. Private parties filed six cases involving list maintenance claims under the NVRA that reached the U.S. Circuit Courts of Appeals. NVRA list maintenance cases may involve two types of allegations under section 8: (1) in conducting a required program to remove ineligible voters from the voter registration list, a state or local jurisdiction did not incorporate certain safeguards, with the potential effect of unlawfully removing eligible voters; and (2) a state or local jurisdiction did not have an adequate program to remove ineligible voters from the voter registration list. Five of the six cases included a claim under section 8 related to the potential unlawful removal of voters from voter registration lists. The sixth case included a claim under section 8 related to the inadequate removal of ineligible voters from voter registration lists. DOJ submitted an amicus brief or statement of interest in nine of the 19 NVRA cases filed by private parties that reached the U.S. Circuit Courts of Appeals between fiscal years 2008 through 2018. Five of the nine cases in which DOJ participated involved issues related to registration opportunities: DOJ participated in all four of the cases that included a claim under section 7 related to registration issues involving public assistance offices. For example, in one case, plaintiffs alleged that the state of New Mexico failed to provide voter registration forms to applicants for public assistance who did not decline, in writing, to register to vote. DOJ submitted a brief in support of the plaintiffs. DOJ participated in one case that included a claim under section 6 related to mail-in voter registration application forms. DOJ also participated in one case under section 8 that related to the public disclosure of records concerning voter registration list maintenance activities. The remaining three cases involved issues related to list maintenance, specifically allegations that an election jurisdiction’s list maintenance program did not have appropriate safeguards to protect against the unlawful removal of eligible voters. For example, in one case, plaintiffs alleged that the state of Ohio violated the NVRA by using failure to vote as the sole trigger to start the confirmation process for removing voters from registration rolls based on a change of residence. In 2016, DOJ filed an amicus brief in support of the plaintiffs. In 2017, the case was appealed to the U.S. Supreme Court and the department reversed its original position and filed a brief supporting the state’s list maintenance practices. In June 2018, the Supreme Court upheld Ohio’s process for removing voters on change-of-residence grounds and ruled that failure to vote could serve as evidence that a registrant had moved. Within the Department of Justice (DOJ), the Civil Rights Division’s Voting Section enforces the civil provisions of federal laws that protect the right to vote, including the National Voter Registration Act of 1993 (NVRA), the Help America Vote Act, the Voting Rights Act of 1965, and the Uniformed and Overseas Citizens Absentee Voting Act, among others. From fiscal years 2001 through 2017, the Voting Section participated in 234 cases, including 14 cases involving NVRA claims in which the Section was the plaintiff or the plaintiff intervenor. Cases with NVRA claims included allegations related to providing registration opportunities for voters, and allegations related to the requirement to remove voters from registration lists under specified conditions (list maintenance). Table 5 below provides a brief summary of the allegations in each case. To address how selected data sources are used at the state and local level and to obtain perspectives on how these sources help maintain voter registration lists, we selected and reviewed six commonly received data sources that may be used to remove ineligible voters who have moved, died, or committed a disqualifying criminal conviction. We also selected state and local election offices in five states and conducted interviews with election officials to obtain information on policies and procedures for using selected data sources, and perspectives on their benefits and limitations. This appendix describes our data source and site selection methodologies, and additional information on the data sources and sites we selected. To determine which data sources to include in our review, in June 2018 we sent a structured questionnaire to state election directors for each of the 49 states and the District of Columbia with voter registration requirements to identify commonly received data sources which states can potentially use to conduct voter registration list maintenance. The National Voter Registration Act of 1993 (NVRA) specifies certain categories under which election officials may remove registrants from voter registration lists including: 1. if a registrant has moved outside of a jurisdiction and either (a) confirmed the move in writing or (b) failed to respond to an address confirmation mailing and failed to vote in two consecutive federal general elections subsequent to the mailing; 2. death of the registrant; 3. criminal conviction of the registrant, as provided for in state law; and 4. mental incapacity of the registrant, as provided for in state law. We asked state election directors to identify the sources from which data were received at either the state or local level at any point between January 2017 and May 2018. We summarized responses from election directors in 35 states and the District of Columbia to identify commonly received data sources. Table 6 provides a summary of responses to the structured questionnaire, with the data sources organized according to the NVRA categories that may be used to remove registrants from voter lists. From the list of commonly received sources above, we then selected six data sources that can be used to address the following NVRA categories for removing registrants—move outside election jurisdiction, death, and, disqualifying criminal conviction. These categories each account for more than 1 percent of total removals from voter registration lists nationwide, based on the most recent data reported to the U.S. Election Assistance Commission. We did not select any data source that addresses the “disqualifying mental incapacity” NVRA removal category since it accounted for less than 1 percent of total removals nationwide for this time period. Specifically, we selected three sources that address moves, two sources that address deceased registrants, and one source that addresses disqualifying criminal convictions, to generally reflect recent data reported on the distribution of registrant removals, by removal category, from voter registration lists nationwide. We also selected (a) at least one nationwide source that captures data from all states; (b) at least one source that only includes data specific to the particular state or local jurisdiction that receives data from the source; and (c) one interstate data exchange that involves the sharing of data between multiple states. We selected sources from each of these categories in order to identify potential issues that may arise when election officials match their voter registration data with various other types of data sources. Table 7 presents the data sources we selected for further review. To obtain information on policies and procedures for using selected data sources for voter registration list maintenance, and election officials’ perceptions on the benefits and limitations of using them, we selected five states that indicated in their responses to our questionnaire that they have received data from at least five of the six selected data sources between January 2017 and May 2018. We also considered variation in states’ population size, when possible, and geographic diversity in order to capture possible regional differences in election administration practices. See table 8 for a list of the states we selected and a summary of the selected data sources received by each state. For each of the five selected states, we selected two local election jurisdictions (counties or cities/towns)–one with a larger population and one with a smaller population–based on the recommendation of the state election officials, population size, and other factors. See table 9 for demographic information on the states and local jurisdictions we visited. In addition to the contact named above, Tom Jessor (Assistant Director), David Alexander, Justine Augeri, Colleen Candrl, Jamarla Edwards, Jonathan Ferguson, Alana Finley, Eric Hauswirth, Richard Hung, Amanda Miller, Heidi Nielson, Kevin Reeves, Christine San, Janet Temko-Blinder, Jeff Tessin, and Sarah Turpin made key contributions to this report.", "summary": "The NVRA was intended to increase the number of eligible citizens who register to vote in federal elections, protect the integrity of the electoral process, and ensure that accurate and current voter registration rolls are maintained. GAO was asked to examine issues related to the NVRA's voter registration and voter registration list maintenance requirements, as well as issues related to election fraud. This report addresses (1) DOJ's efforts to ensure states and localities comply with NVRA requirements to offer registration opportunities and administer voter registration list maintenance programs, and address potential instances of election fraud; and (2) how selected data sources are used at the state and local level to help maintain voter registration lists, and perspectives on how these data sources help ensure list accuracy and address potential voter eligibility and fraud issues. GAO analyzed data on DOJ's efforts to ensure NVRA compliance and address election fraud–as measured by matters initiated and cases filed--for fiscal years 2001 through 2017 (the last full year of data available when requested from DOJ). This period covered eight federal elections. GAO also interviewed DOJ officials. GAO selected six commonly received data sources that may be used in list maintenance efforts. GAO reviewed literature and interviewed state and local election officials in five states for perspectives on how the data sources are used and any benefits and limitations. These states used at least five of the data sources and provided geographic diversity. The results from these five states are not generalizable, but provide insight into state and local perspectives on list maintenance. From fiscal years 2001 through 2017, the Department of Justice's (DOJ) Voting Section (which enforces the civil provisions of voting rights laws) initiated matters (e.g., investigations), filed cases against state or local governments in federal court, and engaged in other efforts to enforce provisions of the National Voter Registration Act of 1993 (NVRA). Specifically, the Voting Section: initiated 99 matters involving allegations of NVRA violations related to voter registration opportunities and list maintenance; filed 14 cases involving allegations of NVRA violations; eight included list maintenance allegations; four included registration opportunities allegations; and two included both types of allegations; and DOJ's Public Integrity Section (which supervises nationwide election law enforcement and prosecutes selected cases involving alleged corruption by government officials), and U.S. Attorneys' Offices (which enforce criminal laws within their districts) engaged in efforts to address election fraud from fiscal years 2001 through 2017, including filing cases against individuals in federal court. For example: The Section initiated 33 matters and filed 19 cases related to election fraud, accounting for about three percent of its overall caseload. Of these cases,17 involved vote buying and false information charges. U.S. Attorneys' Offices initiated 525 matters and filed 185 cases related to election fraud, accounting for about .02 percent of their overall caseload. Of these cases, 52 involved charges such as vote buying and voting more than once, and 49 involved conspiracy. GAO reviewed six data sources election officials may use to maintain voter registration lists and remove voters who become ineligible due to a move, death, or disqualifying criminal conviction: (1) the U.S. Postal Service's National Change of Address (NCOA), (2) the Interstate Voter Registration Crosscheck Program, (3) returned mail, (4) the public version of the Social Security Administration's Death Master File, (5) state vital records, and (6) U.S. Attorneys' records on felony convictions. Election officials GAO interviewed and literature reviewed reported benefits and limitations associated with each source. According to officials, each source helps improve list accuracy, despite some limitations, and list maintenance efforts in general help reduce opportunities for election fraud. For example, officials said that NCOA data helped them maintain accurate lists by identifying registrants who moved outside the election jurisdiction; however, they also noted that NCOA data may not capture all address changes because people do not always notify the U.S. Postal Service when they move. GAO incorporated technical comments provided by federal agencies and state and local election officials as appropriate.", "document_type": "gao"}
{"report": "As of November 2018, 43 states operated at least part of their Medicaid programs under demonstrations. State demonstrations can vary in size and scope, and many demonstrations are comprehensive in nature, affecting multiple aspects of states’ Medicaid programs. In fiscal year 2017, federal spending on demonstrations accounted for more than one- third of total federal Medicaid spending and in eight states accounted for 75 percent or more of Medicaid expenditures. CMS typically approves demonstrations for an initial 5-year period that can be extended in 3- to 5-year increments with CMS approval. Some states have operated portions of their Medicaid programs under a demonstration for decades. Each demonstration is governed by special terms and conditions, which reflect the agreement reached between CMS and the state, and describe the authorities granted to the state. For example, the special terms and conditions may define what demonstration funds can be spent on—including which populations and services—as well as specify reporting requirements, such as monitoring or evaluation reports states must submit to CMS. In January 2018, CMS announced a new policy to support states interested in using demonstrations to make participation in work or community engagement a requirement to maintain Medicaid eligibility or coverage. CMS’s guidance indicates that states have flexibility in designing demonstrations that test work requirements, but it also describes parameters around the populations that could be subject to work requirements and other expectations. CMS guidance addresses several areas, including the following: Populations. Work requirements should apply to working-age, non- pregnant adult beneficiaries who qualify for Medicaid on a basis other than a disability. Exemptions and qualifying activities. States must create exemptions for individuals who are medically frail or have acute medical conditions. States must also take steps to ensure eligible individuals with opioid addiction and other substance use disorders have access to coverage and treatment services and provide reasonable modifications for them, such as counting time spent in medical treatment toward work requirements. The guidance indicates that states can allow a range of qualifying activities that satisfy work requirements, such as job training, education programs, and community service. The guidance also encourages states to consider aligning Medicaid work requirements with work requirements in other federal assistance programs operating in their states. Beneficiary supports. States are expected to describe their strategies to assist beneficiaries in meeting work requirements and to link them to additional resources for job training, child care assistance, transportation, or other work supports. However, CMS’s guidance specifies that states are not authorized to use Medicaid funds to finance these beneficiary supports. About one-third of states have either received CMS approval or submitted applications to CMS to test work requirements in their demonstrations. Nine states have had work requirements approved as part of new demonstrations or extensions of or amendments to existing demonstrations as of May 2019. Also as of May 2019, seven more states had submitted demonstration applications with work requirements, which were pending CMS approval. (See fig. 1.) States with approved work requirements were in various stages of implementation as of August 2019, and three states faced legal challenges to implementation. The requirements were in effect in Arkansas for 9 months before a federal district court vacated the approval in March 2019. Work requirements became effective in Indiana in January 2019 and will be enforced beginning in January 2020. CMS’s approval of work requirements in Kentucky was vacated in March 2019— several days before the work requirements were set to become effective on April 1, 2019. As of August 2019, CMS was appealing the court decisions vacating demonstration approvals in Arkansas and Kentucky. Other states’ requirements are approved to take effect in fiscal years 2020 and 2021. (See fig. 2.) Implementing work requirements, as with other types of beneficiary requirements, can involve an array of administrative activities by states, including developing or adapting eligibility and enrollment systems, educating beneficiaries, and training staff. In general, CMS provides federal funds for 50 percent (referred to as a 50 percent matching rate) of state Medicaid administrative costs. These funds are for activities considered necessary for the proper and efficient administration of a state’s Medicaid program, including those parts operated under demonstrations. CMS provides higher matching rates for certain administrative costs, including those related to IT systems. For example, expenditures to design, develop, and install Medicaid eligibility and enrollment systems are matched at 90 percent, and maintenance and operations of these systems are matched at 75 percent. States may also receive federal funds for administrative activities delegated to MCOs. The amount of federal Medicaid funds states receive for payments to MCOs that bear financial risk for Medicaid expenditures is determined annually by a statutory formula based on the state’s per capita income, known as the Federal Medical Assistance Percentage (FMAP). The FMAP sets a specific federal matching rate for each state that, for fiscal year 2019, ranges from 50 percent to 76 percent. There are exceptions to this rate for certain populations, providers, and services. For example, states that chose to expand Medicaid under the Patient Protection and Affordable Care Act (PPACA) receive a higher FMAP for newly eligible adults, equal to 93 percent in 2019. (See fig. 3.) CMS has several different related processes under which the agency oversees Medicaid administrative costs, including those for demonstrations. Demonstration approval, monitoring, and evaluation. States seeking demonstration approvals must meet transparency requirements established by CMS. For example, states must include certain information about the expected changes in expenditures under the demonstration in public notices seeking comment at the state level and in the application to CMS, which is posted for public comment at the federal level. In addition, CMS policy requires that demonstrations be budget neutral—that is, that the federal government should spend no more under a demonstration than it would have without the demonstration. Prior to approval, states are required to submit an analysis of their projected costs with and without the demonstration. CMS uses this information to determine budget neutrality and set spending limits for demonstrations. During the demonstration, CMS is responsible for monitoring the state’s compliance with the terms and conditions of the demonstration, including those related to how Medicaid funds can be spent and the demonstration spending limit. States must also evaluate their demonstrations to assess the effects of the policies being tested, which could include impacts on cost. Review and approval of federal matching funds for IT projects. To request higher federal matching rates for changes to Medicaid IT systems, including eligibility and enrollment systems, states must submit planning documents to CMS for review and approval. States’ plans must include sufficient information to evaluate the state’s goals, procurement approach, and cost allocations within a specified budget. States may request funds for system development related to a proposed demonstration before the demonstration is approved. Funding can be approved and expended under the approved plan while the demonstration application is being reviewed. States submit updates to planning documents annually for CMS review, which can include requested changes to the approved budget. Quarterly expenditure reviews. In order to receive federal matching funds, states report their Medicaid expenditures quarterly to CMS, including those made under demonstrations. Expenditures associated with demonstrations, including administrative expenditures, are reported separately from other expenditures. CMS is responsible for ensuring that expenditures reported by states are supported and allowable, meaning that the state actually made and recorded the expenditure and that the expenditure is consistent with Medicaid requirements. With regard to consistency, this includes comparing reported expenditures to various approval documents. For example, CMS is responsible for comparing reported demonstration expenditures against the special terms and conditions that authorize payment for specified services or populations and establish spending limits. CMS is also responsible for reviewing states’ reported expenditures against budgets in states’ planning documents to ensure that states do not exceed approved amounts. A list of GAO reports related to these CMS oversight processes is included at the end of this report. States took different approaches to designing work requirements under their Medicaid demonstrations. These requirements varied in terms of the beneficiary groups subject to the requirements; the required activities, such as frequency of required reporting; and the consequences beneficiaries face if they do not meet requirements. In the nine states with approved work requirements as of May 2019, we found differences in the age and eligibility groups subject to work requirements, and, to a lesser extent, the number of hours of work required and frequency of required reporting to the state. For example: Age and eligibility groups subject to work requirements. Four of these states received approval to apply the requirements to adults under the age of 50, similar to how certain work requirements are applied under the Supplemental Nutrition Assistance Program (SNAP). Among the other five states, approved work requirements apply to adults up to the age of 59 (Indiana and Utah), 62 (Michigan), and 64 (Kentucky and New Hampshire). States generally planned to apply the requirements to adults newly eligible under PPACA or a previous coverage expansion, but some states received approval to apply the requirements to additional eligibility groups, such as parents and caretakers of dependents. Number of hours of work required and frequency of required reporting. Under approved demonstrations in seven states, Medicaid beneficiaries must complete 80 hours of work or other qualifying activities per month to comply with work requirements. Five states’ approved demonstrations require beneficiaries to report each month on their hours of work or other qualifying activities, using methods approved by the state, such as online or over the phone. (See table 1.) We saw similar variation under the seven state applications that were pending as of May 2019. All nine states with approved work requirements as of May 2019 exempted several categories of beneficiaries and counted a variety of activities as meeting the work requirements. For example, all nine states exempted from the work requirements people with disabilities, pregnant women, and those with certain health conditions, such as a serious mental illness. In addition, depending on the state, other groups were also exempted, such as beneficiaries who are homeless, survivors of domestic violence, and those enrolled in substance use treatment programs. States also counted activities other than work as meeting the work requirements, such as job training, volunteering, and caregiving for non-dependents. In addition to work requirements, eight of the nine states received approval under their demonstrations to implement other beneficiary requirements, such as requiring beneficiaries to have expenditure accounts. (See app. I for more information on these other beneficiary requirements.) The consequences Medicaid beneficiaries faced for non-compliance and the timing of the consequences varied across the nine states with approved work requirements. The consequences for non-compliance included coverage suspension and termination. For example, Arizona received approval to suspend beneficiaries’ coverage after 1 month of non-compliance. In contrast, Wisconsin will not take action until a beneficiary has been out of compliance for 4 years, at which time coverage will be terminated. Three states (Arkansas, Michigan, and Wisconsin) imposed or planned to impose a non-eligibility period after terminating a beneficiary’s enrollment. For example, under Arkansas’ demonstration, after 3 months of non-compliance, the beneficiary was not eligible to re-enroll until the next plan year, which began in January of each year. Thus, beneficiaries could be locked out of coverage for up to 9 months. (See table 2.) For states with pending applications, suspension or termination of coverage takes effect after 2 or 3 months of non- compliance. For states that suspend coverage for beneficiaries, there are different conditions for coming into compliance and lifting the suspension. For example: Arizona received approval to automatically reactivate an individual’s eligibility at the end of each 2-month suspension period. In other states, such as Indiana, beneficiaries must notify the state that they have completed 80 hours of work or other qualifying activities in a calendar month, after which the state will reactivate eligibility beginning the following month. (See text box.) Indiana’s Suspension Process for Non-Compliance with Medicaid Work Requirements At the end of each year, the state reviews beneficiaries’ activities related to work requirements. Beneficiaries must meet the required monthly hours 8 out of 12 months of the year to avoid a suspension of Medicaid coverage. If coverage is suspended for not meeting work requirements, the suspension will start January 1 and could last up to 12 months. During a suspension, beneficiaries will not be able to access Medicaid coverage to receive health care. Beneficiaries with suspended Medicaid coverage can reactivate coverage if they become medically frail; or employed, enrolled in school, or engaged in volunteering. Beneficiaries must contact the state to reactivate coverage. To prevent suspension from taking effect, two states (Kentucky and New Hampshire) require beneficiaries to make up required work hours that were not completed in order to maintain compliance with work requirements. For example, in Kentucky, if the beneficiary worked 60 hours in October (20 hours less than the required 80), the beneficiary must work 100 hours in November to avoid suspension of coverage in December. Available estimates of the costs to implement Medicaid work requirements varied considerably among the five selected states, and these estimates did not account for all costs. These states estimated that federal funding would cover the majority of these costs, particularly costs to modify IT systems. Selected states (Arkansas, Indiana, Kentucky, New Hampshire, and Wisconsin) reported estimates of the costs to implement work requirements that ranged from under $10 million in New Hampshire to over $250 million in Kentucky. These estimates—compiled by states and reported to us—did not include all planned costs. The estimates were based on information the states had readily available, such as the costs of contracted activities for IT systems and beneficiary outreach, and primarily reflect up-front costs. Four selected states (Arkansas, Indiana, Kentucky, and New Hampshire) had begun implementing work requirements and making expenditures by the end of 2018. Together, these states reported to us having spent more than $129 million in total for implementation activities from the time the states submitted their demonstration applications through the end of 2018. (See table 3.) Several factors may have contributed to the variation in the selected states’ estimated costs of administering work requirements, including planned IT system changes and the number of Medicaid beneficiaries subject to the work requirements. IT system changes. Selected states planned distinct approaches to modify their IT systems in order to administer work requirements. For example: Indiana, which implemented work requirements by expanding on an existing work referral program, planned to leverage existing IT systems, making modifications expected to result in IT costs of $14.4 million over 4 years. In contrast, Kentucky planned to develop new IT system capabilities to communicate, track, and verify information related to work requirements. Kentucky received approval to spend $220.9 million in fiscal years 2019 and 2020 to do that and make changes needed to implement other beneficiary requirements in its demonstration. Number of beneficiaries subject to requirements. The estimated cost of some activities to administer work requirements depended on the number of Medicaid beneficiaries subject to work requirements, which varied across selected states. For example: Kentucky estimated 620,000 beneficiaries would be subject to work requirements—including those who may qualify for exemptions—and estimated costs of $15 million for fiscal years 2019 and 2020 to conduct beneficiary education, outreach, and customer service. In contrast, Arkansas had fewer beneficiaries subject to work requirements (about 115,000 in February 2019, with about 100,000 of those eligible for exemptions) and estimated fewer outreach costs. The state estimated $2.9 million in costs from July 2018 through June 2019 to conduct education and outreach. As noted earlier, states’ available estimates did not include all expected Medicaid costs. For example, four of the five selected states planned to use MCOs or other health plans to help administer work requirements, but two of these four did not have estimates of the associated costs. Indiana and Kentucky estimated additional payments to MCOs—$20.7 million in Indiana to administer work requirements in 2019 and $50.7 million in Kentucky to administer its demonstration from July 2018 through June 2020. In contrast, officials in New Hampshire told us that no estimates were available. In Arkansas, where beneficiaries receive premium support to purchase coverage from qualified health plans on the state’s health insurance exchange, plans were instructed to include the costs of administering work requirements in the premiums, according to Arkansas officials. State officials and representatives from a qualified health plan we spoke with could not provide the amount that the state’s premium assistance costs increased as a result. States’ estimates also did not include all ongoing costs that they expect to incur after the up-front costs and initial expenditures related to implementation of the work requirements. States had limited information about ongoing costs, but we collected some examples. For instance, New Hampshire provided estimated costs of $1.6 million to design and implement the evaluation of its demonstration, which all states are required to perform. In addition, officials or documents in each selected state acknowledged new staffing costs that may be ongoing, such as Indiana’s costs for five full-time employees to assist beneficiaries with suspended coverage to meet requirements or obtain exemptions. Finally, states reported that administering Medicaid work requirements will increase certain non-Medicaid costs—costs that are not funded by federal Medicaid, but are borne by other federal and state agencies, stakeholders, or individuals. For instance, New Hampshire officials planned to use approximately $200,000 to $300,000 in non-Medicaid funds for six positions performing case management for workforce development. Similarly, in July 2017, Indiana estimated that providing beneficiaries with job skills training, job search assistance, and other services would cost $90 per month per beneficiary, although state officials said these costs were uncertain after learning they were not eligible for federal Medicaid funds. In addition, beneficiaries and entities other than states, such as community organizations, may incur costs related to the administration of work requirements that are not included in states’ estimates. All five selected states expected to receive federal funds for the majority of estimated costs and expenditures (described previously) for implementing work requirements. For example, the four selected states that provided data on expenditures to administer work requirements through 2018 (Arkansas, Indiana, Kentucky, and New Hampshire) expected the portion of those expenditures paid by the federal government to range from 82 percent in Indiana to 90 percent in New Hampshire and Kentucky. These effective matching rates exceed the 50 percent matching rate for general administrative costs, largely due to higher matching rates of 75 and 90 percent of applicable IT costs. For example, Kentucky received approval to spend $192.6 million in federal funds for its $220.9 million in expected IT costs over 2 years to implement work requirements and other beneficiary requirements, an effective match rate of 87 percent. In addition to higher federal matching rates for IT costs, the selected states receive federal funds for the majority of MCO capitation payments, which the states planned to increase to pay MCOs’ costs to administer work requirements. Each of the three states that planned to use MCOs to administer work requirements planned to increase capitation payments in order to do so. For example, Indiana planned to increase capitation payments to MCOs by approximately 1 percent (or $20.7 million in 2019) to pay for a variety of ongoing activities to administer work requirements, including requiring MCOs to help beneficiaries report compliance, reporting beneficiaries who qualify for exemptions, and helping the state verify the accuracy of beneficiary reporting, according to state officials. The federal government pays at least 90 percent of capitation payments to MCOs to provide covered services to beneficiaries who are newly eligible under PPACA, the primary population subject to work requirements among the five selected states. Indiana and Kentucky also received approval to apply work requirements to other populations, and capitation payments for these other populations receive federal matching rates of 66 percent in Indiana and 72 percent in Kentucky in fiscal year 2019. States’ approaches to implementing work requirements can affect the federal matching funds they receive. For example, Arkansas officials told us that the state decided to collect information on beneficiary compliance through an on-line portal—the initial cost of which received an effective federal matching rate of 87 percent, according to Arkansas. Officials told us that the state avoided having beneficiaries report compliance to staff— costs of which receive a 75 percent matching rate. However, after approximately 17,000 beneficiaries lost coverage due to non-compliance with work requirements, Arkansas revised its procedures to allow beneficiaries to report compliance to state staff over the phone. Three of the five selected states sought to leverage other programs funded by the federal government to help implement work requirements or provide beneficiary supports, such as employment services. Kentucky officials reported piloting elements of Medicaid work requirements using its SNAP Employment and Training program. Similarly, Arkansas officials sought a waiver to be able to use TANF funds to provide employment services to individuals without children in order to serve Medicaid beneficiaries subject to work requirements. New Hampshire also used TANF funds to provide employment services to Medicaid beneficiaries who were also enrolled in TANF. CMS does not consider administrative costs when approving any demonstrations—including those with work requirements—though these costs can be significant. The agency has recently taken steps to obtain more information about demonstration administrative costs. However, we identified various weaknesses in CMS’s oversight of administrative costs that could result in states receiving federal funds for costs to administer work requirements that are not allowable. CMS’s demonstration approval process does not take into account the extent to which demonstrations, including those establishing work requirements, will increase a state’s administrative costs. CMS policy does not require states to provide projections of administrative costs in their demonstration applications or include administrative costs in their demonstration cost projections used by CMS to assess budget neutrality. CMS officials explained that in the past demonstrations had generally not led to increases in administrative costs, and as such, the agency had not seen a need to separately consider these costs. However, the officials told us and have acknowledged in approval letters for demonstrations with work requirements, that demonstrations may increase administrative costs. Kentucky provides an example of this, reporting to us estimated administrative costs of approximately $270 million—including about $200 million in federal funds—to implement the demonstration over 2 years. However, neither Kentucky nor the other four selected states provided estimates of their administrative costs in their applications to CMS, and CMS officials confirmed that no additional information on administrative costs was provided by the states while their demonstration applications were being reviewed. By not considering administrative costs in its demonstration approval process, CMS’s actions are counter to two key objectives of the demonstration approval process: transparency and budget neutrality. Transparency. CMS’s transparency requirements are aimed at ensuring that demonstration proposals provide sufficient information to ensure meaningful public input. However, CMS officials told us that they do not require the information states provide on the expected changes in demonstration expenditures in their applications to account for administrative costs. This information would likely have been of interest in our selected states, because public commenters in each state expressed concerns about the potential administrative costs of these demonstrations. In prior work, we reported on weaknesses in CMS’s policies for ensuring transparency in demonstration approvals. Budget neutrality. The aim of CMS’s budget neutrality policy is to limit federal fiscal liability resulting from demonstrations, and CMS is responsible for determining that a demonstration will not increase federal Medicaid expenditures above what they would have been without the demonstration. However, CMS does not consider administrative costs when assessing budget neutrality. For three of our five selected states, the demonstration special terms and conditions specify that administrative costs will not be counted against the budget neutrality limit. Even though demonstrations’ administrative costs can be significant, CMS officials said the agency has no plans to revise its approval process—either to (1) require states to provide information on expected administrative costs to CMS or the public, or to (2) account for these costs when the agency assesses whether a demonstration is budget neutral. CMS officials explained that the agency needs more experience with policies that require administrative changes under a demonstration before making any revisions to its processes. Without requiring states to submit projections of administrative costs in their demonstration applications, and by not considering the implications of these costs for federal spending, CMS puts its goals of transparency and budget neutrality at risk. This is inconsistent with federal internal control standards that call for agencies to identify, analyze, and respond to risks related to achieving program objectives. CMS recently implemented procedures that may provide additional information on demonstrations’ administrative costs. These included implementing new procedures to identify costs specific to demonstrations when approving federal matching funds for states’ planned IT costs and issuing guidance on monitoring and evaluating demonstrations. However, it is unclear whether these efforts will result in data that improve CMS’s oversight. (See table 4.) In addition to these new initiatives, states’ quarterly expenditure reports provide CMS with some information on their demonstration administrative costs, but this information also has limitations. States are required to separately track and report administrative expenditures attributable to their demonstrations in their quarterly expenditure reports. However, CMS officials told us that states typically use the same resources, such as staff, to administer their demonstrations and their regular Medicaid program, which can affect the demonstration costs states report. We found that about a quarter of states with demonstration expenditures in fiscal year 2017 reported no administrative expenditures related to their demonstrations. CMS officials acknowledged that the data states submit in their quarterly expenditure reports may not provide a meaningful measure of states’ demonstration-related administrative costs. CMS’s recently implemented procedures may provide more information on the amounts states are spending on demonstration administrative costs, but they do not address weaknesses we found in CMS’s oversight of administrative costs. In four of the five selected states, we identified examples of states requesting federal matching funds for costs to administer work requirements that do not appear to be allowable, or at higher matching rates than appropriate under CMS guidance. In some cases, states received CMS approval for planned administrative costs while in others it was unclear whether CMS would have identified the issues through their oversight procedures. Areas of risk included funds for planned IT costs, funds for beneficiary supports, and funds provided under managed care contracts. Federal funds for planned IT costs that may not be allowable or eligible for higher matching rates. Three of our five selected states requested and received funding approval for planned IT costs to implement their demonstrations that did not appear to be allowable or at higher matching rates than appropriate under CMS guidance. Kentucky and Indiana requested and received funding approval for planned IT costs that do not appear to be allowable under CMS guidance. Kentucky requested and received CMS approval for funds (at the 90 percent federal matching rate) for a contract that included activities to assist Medicaid beneficiaries obtain employment. (See text box.) However, CMS’s 2018 guidance states that Medicaid funding is not available to finance beneficiary supports, such as job training or other employment services. CMS officials said that the agency did not review the contract and approved the request based on Kentucky’s assertion that these costs were specific to technology. Indiana received approval to receive IT funds to develop a website that provides beneficiaries access to information and tools to seek, acquire, and retain employment, costs that also appear related to beneficiary supports. Kentucky Received Approval of Information Technology Funding for Activities Aimed at Helping Beneficiaries Obtain Employment In 2018, in an update to its information technology budget request, Kentucky included costs for a contract with the state’s Department of Workforce Services to assist Medicaid beneficiaries in developing skills needed to obtain and retain employment. The contracted services included activities such as assessing beneficiaries’ eligibility for non-Medicaid programs, providing services to beneficiaries at career assistance centers, and making referrals to other agencies and programs. Kentucky budgeted $21 million for this contract at a 90 percent federal matching rate ($18.9 million in federal funds) for fiscal year 2019 and another $21 million at a 75 percent matching rate ($15.8 million in federal funds) for fiscal year 2020. CMS approved Kentucky’s budget request without reviewing the contract. Medicaid Services. | GAO-20-149. Indiana and New Hampshire received funding approval for federal IT funds at the 90 percent matching rate for costs that do not appear eligible for that rate. In 2018, CMS approved Indiana’s request for a 90 percent match rate to pay $500,000 in consulting fees to develop work requirement policies, despite CMS guidance indicating that policy research and development activities should be matched at 50 percent. New Hampshire requested and received CMS approval in 2018 for federal funds at a 90 percent matching rate for $180,000 in costs to educate beneficiaries about work requirements, including costs to place outreach calls through an existing contracted call center. CMS guidance indicates that these costs should receive funding at a lower matching rate. Federal funds for beneficiary supports that are not allowable. Wisconsin requested and planned to seek federal funds for beneficiary support costs that are not allowable until our work identified the issue for CMS. Wisconsin officials told us that it was their understanding during the planning phase of the demonstration that administrative costs incurred by state programs providing such services were eligible for federal matching funds. State officials said that CMS officials told them on multiple occasions that the state could receive a 50 percent federal match for these costs. Based on this, the state requested budget authority from its legislature for $51.2 million for employment and training services, of which it anticipated $23.1 million would come from federal Medicaid funds. CMS officials told us that such costs are not eligible for federal matching funds and maintained that the agency’s guidance—which indicates that beneficiary support costs are not eligible for federal matching funds—was clear. In response to our inquiries, the agency contacted the state in April 2019 and clarified this with officials. Federal funds for costs to administer work requirements provided through managed care contracts, which may not be allowable. As noted earlier, three of the five selected states (Indiana, Kentucky, and New Hampshire) required or planned to require MCOs to perform a number of activities to implement work requirements. These activities included, for example, providing information on options to satisfy work requirements, assisting beneficiaries with reporting compliance with work requirements, and providing referrals to state work requirement resources. To fund these activities, officials in these states said that they plan to increase their capitation payments. States will receive at least a 90 percent federal matching rate for most of these payments, because the payments are largely for beneficiaries who are newly eligible under PPACA. It is unclear, however, whether including these activities in capitation payments is allowable. CMS regulations provide that states may only include administrative costs that are related to the provision of covered health care services in their MCO capitation payments. In addition, CMS guidance notes that implementing work requirements will not change the types of expenditures that are allowable. We provided CMS with specific examples of activities states delegated or planned to delegate to MCOs and asked if these types of activities met CMS’s criteria to be included under capitation payments. CMS officials told us that federal review of the related managed care contracts in Indiana and New Hampshire had not been completed as of June 2019 and could not make a definitive statement. While CMS guidance requires states to carry out a range of activities to implement work requirements—some of which are not eligible for federal Medicaid funds—agency officials told us that CMS has not updated any procedures for the various reviewers of these costs. Further, CMS has not completed a risk assessment to determine whether current procedures for overseeing administrative costs are sufficient, and agency officials told us that there were no plans to do so. According to federal internal control standards, agencies should identify, analyze, and respond to risks related to achieving program objectives (in this case, ensuring that administrative expenditures under demonstrations are allowable and matched at the correct rate). Without identifying, assessing, and addressing the risks posed by demonstrations that may increase administrative costs, CMS may be providing federal funds for costs that are not allowed or at inappropriately high matching rates. A third of states have sought approval to implement work requirements in their Medicaid programs. CMS has acknowledged that demonstrations, including those with work requirements, may increase Medicaid administrative costs—and therefore overall Medicaid spending. Yet, CMS is not factoring these costs into its approval decisions, which is counter to the agency’s goals of transparency and budget neutrality. Further, the agency has not taken steps to assess and respond to risks of federal funds being spent for administrative costs that are not allowable or matched at rates higher than what is appropriate, risks we found in four of the five demonstrations we reviewed. While administrative costs are a relatively small portion of states’ Medicaid spending, the weaknesses in CMS’s oversight of these costs could take on increased importance as more states seek and receive approval to implement work requirements. We are making the following three recommendations to CMS: The Administrator of CMS should require states to submit and make public projections of administrative costs when seeking approval of demonstrations, including those with work requirements and all other demonstrations. (Recommendation 1) The Administrator of CMS should account for the administrative costs of demonstrations, including those with work requirements and all other demonstrations, when assessing whether demonstrations are budget neutral. (Recommendation 2) The Administrator of CMS should assess the risks of providing federal funds for costs to administer work requirements that are not allowable and should respond to risks by improving oversight procedures, as warranted. This assessment should consider risks related to costs for information systems, beneficiary supports, and managed care. (Recommendation 3) We provided a draft of this report to HHS for comments and its comments are reproduced in appendix II. HHS also provided us with technical comments, which we incorporated in the report as appropriate. HHS did not concur with our recommendations. In general, HHS commented that it expects administrative costs to represent a relatively small proportion of total Medicaid spending and that its current approach to overseeing administrative costs—including those incurred under Medicaid demonstrations—is appropriate given the level of financial risk. HHS commented that administrative costs were approximately 5 percent of Medicaid expenditures. While these cost may represent a relatively small share of total spending, CMS projected them to be $18 billion in federal funds in fiscal year 2019—and this does not include all administrative spending. In particular, it does not include amounts paid to MCOs for administrative costs, which are likely considerable given that managed care payments now represent about half of all Medicaid spending. Further, demonstrations may represent a heightened financial risk given our finding that they can result in additional administrative costs that would not otherwise occur. Regarding our first recommendation to require states to submit and make public projections of administrative costs, HHS commented that its experience suggests that demonstration administrative costs will be a relatively small portion of total costs and therefore HHS believes making information about these costs available would provide stakeholders little to no value. As noted, Medicaid is a significant component of federal and state budgets. In each of the five states we reviewed, public commenters expressed concerns about the potential administrative costs of Medicaid demonstrations with work requirements, suggesting stakeholders would value information about these costs. We maintain that requiring states to make public information about administrative costs would help to ensure that demonstration proposals provide sufficient information to ensure meaningful public input. Regarding our second recommendation to account for administrative costs when assessing whether demonstrations are budget neutral, HHS again commented that its experience suggests that demonstration administrative costs will be a relatively small portion of total costs and that it believed that its current approach is appropriate for the level of financial risk. However, we found that demonstration administrative costs could be significant and HHS’s current policy of not considering these costs in its assessments of budget neutrality could increase federal fiscal liability. For example, in Kentucky, we found estimated administrative costs for implementing the demonstration exceeded $270 million over about 2 years. We maintain that including administrative costs in its assessments will help HHS ensure that demonstrations are budget neutral. Regarding our third recommendation to assess and respond to risks of providing federal funds for costs to administer work requirements that are not allowable, HHS commented that (1) all states’ requests for federal Medicaid funding are subject to the same federal regulations and requirements; (2) the expenditures reported by states to GAO had not been reviewed against federal requirements or certified by states to be accurate and permissible; and (3) HHS believes its existing approach is appropriate for the low level of risk that administrative expenditures represent. Our findings indicate that CMS’s oversight procedures—which are designed to prevent state spending on costs that do not meet federal requirements—have vulnerabilities, particularly given the types of administrative activities associated with work requirements. Four of the five states we reviewed were planning to seek federal funds for costs (1) that did not appear allowable, or (2) at higher matching rates than appear appropriate, and three states succeeded in gaining CMS approval to do so. We agree with HHS that CMS may also identify inappropriate expenditures during its reviews of state-reported expenditures. However, our past work has identified weaknesses in that review process. In 2018, we reported that CMS officials indicated that resource constraints have limited the agency’s ability to target risk during such reviews, potentially allowing errors to go undetected. Finally, the basis for HHS’s conclusion that its current approach is appropriate for the risks posed by these administrative expenditures is unclear. As we note in our report, CMS officials told us that they had not assessed whether current procedures sufficiently address risks posed by administrative costs for work requirements and had no plans to do so. We maintain that assessing these risks of providing federal funds for costs that are not allowable and improving oversight, as warranted, would help HHS to ensure the integrity of the Medicaid program. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Health and Human Services, the appropriate congressional committees, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7144 or yocomc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix III. Eight of the nine states that received approval for work requirements, as of May 2019, also received approval under their demonstrations for other beneficiary requirements, such as requiring beneficiaries to have expenditure accounts. Some of these beneficiary requirements preceded work requirements, while others were newly introduced with the work requirements. For example, Kentucky was developing and implementing work requirements at the same time as other beneficiary requirements, such as the requirement for beneficiaries to have two expenditure accounts and make premium payments. (See table 5.) In addition to the contact named above, Susan Barnidge (Assistant Director), Russell Voth (Analyst in Charge), Linda McIver, and Matt Nattinger made key contributions to this report. Also contributing were Giselle Hicks, Drew Long, Ethiene Salgado-Rodriguez, and Emily Wilson Schwark. Medicaid Demonstrations: Approvals of Major Changes Need Increased Transparency. GAO-19-315. Washington, D.C.: April 17, 2019. Medicaid: CMS Needs to Better Target Risks to Improve Oversight of Expenditures. GAO-18-564. Washington, D.C.: August 6, 2018. Medicaid Demonstrations: Evaluations Yielded Limited Results, Underscoring Need for Changes to Federal Policies and Procedures. GAO-18-220. Washington, D.C.: January 19, 2018. Medicaid Demonstrations: Federal Action Needed to Improve Oversight of Spending. GAO-17-312. Washington, D.C.: April 3, 2017. Medicaid: Federal Funds Aid Eligibility IT System Changes, but Implementation Challenges Persist. GAO-15-169. Washington, D.C.: December 12, 2014. Medicaid Demonstration Waivers: Approval Process Raises Cost Concerns and Lacks Transparency. GAO-13-384. Washington, D.C.: June 25, 2013. Medicaid Demonstration Waivers: Recent HHS Approvals Continue to Raise Cost and Oversight Concerns. GAO-08-87. Washington, D.C.: January 31, 2008.", "summary": "Section 1115 demonstrations are a significant component of Medicaid spending and affect the care of millions of low-income and medically needy individuals. In 2018, CMS announced a new policy allowing states to test work requirements under demonstrations and soon after began approving such demonstrations. Implementing work requirements can involve various administrative activities, not all of which are eligible for federal funds. GAO was asked to examine the administrative costs of demonstrations with work requirements. Among other things, this report examines (1) states' estimates of costs of administering work requirements in selected states, and (2) CMS's oversight of these costs. GAO examined the costs of administering work requirements in the first five states with approved demonstrations. GAO also reviewed documentation for these states' demonstrations, and interviewed state and federal Medicaid officials. Additionally, GAO assessed CMS's policies and procedures against federal internal control standards. Medicaid demonstrations enable states to test new approaches to provide Medicaid coverage and services. Since January 2018, the Centers for Medicare & Medicaid Services (CMS) has approved nine states' demonstrations that require beneficiaries to work or participate in other activities, such as training, in order to maintain Medicaid eligibility. The first five states that received CMS approval for work requirements reported a range of administrative activities to implement these requirements. These five states provided GAO with estimates of their demonstrations' administrative costs, which varied, ranging from under $10 million to over $250 million. Factors such as differences in changes to information technology systems and numbers of beneficiaries subject to the requirements may have contributed to the variation. The estimates do not include all costs, such as ongoing costs states expect to incur throughout the demonstration. GAO found weaknesses in CMS's oversight of the administrative costs of demonstrations with work requirements. No consideration of administrative costs during approval. GAO found that CMS does not require states to provide projections of administrative costs when requesting demonstration approval. Thus, the cost of administering demonstrations, including those with work requirements, is not transparent to the public or included in CMS's assessments of whether a demonstration is budget neutral—that is, that federal spending will be no higher under the demonstration than it would have been without it. Current procedures may be insufficient to ensure that costs are allowable and matched at the correct rate. GAO found that three of the five states received CMS approval for federal funds—in one case, tens of millions of dollars—for administrative costs that did not appear allowable or at higher matching rates than appeared appropriate per CMS guidance. The agency has not assessed the sufficiency of its procedures for overseeing administrative costs since it began approving demonstrations with work requirements. GAO makes three recommendations, including that CMS (1) require states to submit projections of administrative costs with demonstration proposals, and (2) assess risks of providing federal funds that are not allowable to administer work requirements and improve oversight procedures, as warranted. CMS did not concur with the recommendations and stated that its procedures are sufficient given the level of risk. GAO maintains that the recommendations are warranted as discussed in this report.", "document_type": "gao"}
{"report": "We previously reported that the Army began its modernization efforts— defined as efforts to enhance its capabilities and upgrade its weapon systems—in the fall of 2017. As a part of this effort, the Army identified six modernization priorities. 1. Long-Range Precision Fires— focused on improving the targeting, range, and lethality of, among other things, artillery and rockets. 2. Next Generation Combat Vehicle—focused on developing manned and unmanned combat vehicles with updated firepower, protection, mobility, and power generation. 3. Future Vertical Lift— focused on developing manned and unmanned aircraft capable of attack, lift, and reconnaissance missions. 4. Army Network— focused on developing a mobile system of hardware, software, and infrastructure for reliable and secure communications. 5. Air and Missile Defense—focused on improving capabilities for protection against modern and advanced air and missile threats. 6. Soldier Lethality—focused on improving capabilities, equipment, and training for all fundamentals of combat including shooting, moving, communicating, protecting, and sustaining combat operations. We also reported that, to fund these priorities, in 2017 the Army realigned over $1 billion in science and technology funding away from efforts that it determined did not align with these priorities. The Army subsequently announced plans to spend an additional $7.5 billion on these priorities over the next 5 years. Army Futures Command was formed less than a year ago and has not finalized its structure. The Army established the Army Futures Command in June 2018 to consolidate its modernization efforts under one entity and it began initial operations in July 2018. Army Futures Command selected Austin, Texas, as its headquarters location and began to integrate and align resources and personnel. The new command headquarters includes a number of administrative and functional offices that report directly to it, not all of which are co-located with the command in Austin. Specifically: Administrative offices are responsible for providing contracting support, legal support, and small business engagement support to headquarters. These offices are located in Austin, Texas. Army Applications Laboratory is responsible for coordinating outreach to businesses, including small businesses, for headquarters. The Army Applications Laboratory is located in Austin, Texas. Cross-functional teams are the eight teams responsible for identifying capability needs and developing requirements associated with the Army’s six priorities. The teams are located in different parts of the country in areas relevant to their capability focus. Medical Research and Development Command is responsible for seeking and developing new medical technologies for use by the Army. This command is in the process of transferring from Army Medical Research and Materiel Command and is located at Fort Detrick, Maryland. In addition to these organizations, the command has three major subordinate components, comprised of several existing requirements and technology development organizations. Specifically: Futures and Concepts Center is responsible for identifying and prioritizing capability and development needs and opportunities. This organization subsumed the Army Capabilities Integration Center— formerly part of Army Training and Doctrine Command—on December 7, 2018 and is located at Fort Eustis, Virginia. Combat Capabilities Development Command is responsible for conceptualizing and developing solutions for identified needs and opportunities. This organization subsumed the Research, Development, and Engineering Command—formerly a part of Army Materiel Command—on February 3, 2019 and is located at Aberdeen, Maryland. Combat Systems Directorate is responsible for refining, engineering, and producing new capabilities. The directorate is to communicate with the program executive offices and program management offices reporting to the Assistant Secretary of the Army for Acquisition, Logistics, and Technology. The command is in the process of establishing Combat Systems Directorate in Austin, Texas. Army Futures Command is expected to become fully operational in July 2019, when its headquarters and its subordinate components are fully staffed. Locations for components of the new command are shown in figure 1. According to Army Futures Command officials, as part of their modernization efforts, they plan to coordinate with other existing Army organizations. These include the Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology—the civilian authority responsible for the overall supervision of acquisition and contracting for the Army. They also plan to coordinate with Army Contracting Command, which is the principle buying agent and provider of contracting support for the Army and operates within Army Materiel Command. As we previously stated, others have reported that small businesses are a vital part of the defense industrial base and engaging with them can produce innovative capabilities and emerging technologies to support the warfighter. For the purposes of this report, engagement with small business is defined as a range of activities including: initial outreach to small businesses to identify companies that may have useful information or ideas, information sharing on the Army’s capability needs, and formal engagement including processes to enter into business relationships, including contracts and other arrangements. The Small Business Act requires federal agencies to establish annual goals that provide small businesses with contracting opportunities to the maximum extent practicable. Pursuant to the Act, the Small Business Administration negotiates annual small business goals with federal agencies, including the Department of Defense. A portion of the overall goals for the Department of Defense is assigned to the various military components— including the Army—that have contracting authority. The Army Office of Small Business Programs, responsible for enhancing Army contracting opportunities for small businesses, then assigns portions of the Army’s goal to its four major commands with contracting authority: Army Materiel Command, Army Medical Command, Army Corps of Engineers, and the National Guard Bureau. Army Materiel Command is the primary command responsible for the execution and oversight of contracts for Army Futures Command. Historically, the Army has engaged with small businesses in a variety of ways, including awarding contracts for various goods and services that support the warfighter. Federal contracts, including those awarded by the Army, are tracked in the Federal Procurement Data System-Next Generation database. Using data provided by the Army from this database, we identified over 4,500 contracts awarded to small businesses for research and development efforts in the 5 years prior to the establishment of Army Futures Command—fiscal years 2013 through 2017. The number of contracts awarded during this time period is summarized in table 1. We identified almost $2.3 billion in obligations to small businesses for research and development from fiscal years 2013 through 2017, or about half of the total amount the Army obligated for all research and development contracts. The obligations for these Army contracts awarded to small businesses for research and development are summarized in table 2. These contract obligations for research and development went to 1,815 small businesses throughout the United States from fiscal years 2013 through 2017. Figure 2 shows this information for each state as well as the District of Columbia and Puerto Rico. About half of the Army contract awards and obligations to small businesses for research and development from fiscal years 2013 through 2017 supported two organizations—Research, Development, and Engineering Command and Medical Research and Materiel Command— which have transitioned, or are in the process of transitioning, to Army Futures Command. To support research and development efforts for these two organizations, the Army awarded 2,948 out of a total 4,514 small business contracts, and obligated about $1.3 billion out of $2.3 billion from fiscal years 2013 through 2017. In addition to the contracts discussed above, the Army can use other arrangements to engage with small businesses. These other arrangements include: agreements using other transaction authority for research and development activities and developing prototypes; financial assistance mechanisms including grants—which are used when the principal purpose of the relationship is to transfer a thing of value to the recipient to carry out a public purpose authorized by law, and substantial involvement by the agency is not expected—and cooperative agreements—which are also used to transfer a thing of value to carry out a public purpose, but where substantial involvement by the agency is expected; and cooperative research and development agreements under which the government and nonfederal partners may share resources and increase the commercialization of federally developed technology. Unlike contracts, the Federal Procurement Data System-Next Generation database cannot be used to quantify engagement with small businesses using these other arrangements. For example, the financial assistance mechanisms, as well as cooperative research and development agreements, are not generally tracked in the Federal Procurement Data System-Next Generation database. In addition, while it is the Department of Defense’s policy to report the use of other transaction authority for prototype projects in the Federal Procurement Data System- Next Generation, the data for this reporting does not distinguish business size. As a result, it cannot be used to quantify the Army’s engagement with small businesses under this arrangement. The Army conducted several analyses related to its modernization efforts, including those directly focused on the creation of Army Futures Command. We identified the following key analyses the Army used to support its modernization efforts: In October 2017, Army reviewed its science and technology portfolio and determined which investments contributed to the Army’s modernization priorities and which might be curtailed or eliminated to realign funding. According to Army officials, this review was focused on identifying solutions to known capability needs, not on how small businesses would be affected by the realignment of funds. In early 2018, Army analyzed several options for the roles, responsibilities, staffing, and organizational structure for the proposed Army Futures Command. This analysis did not include an assessment of how small business would be affected by its establishment. In April 2018, Army completed a report on its modernization strategy as mandated by the Congress. The report focused on warfighting challenges, risks, costs, and acquisition timelines for fielding future capabilities. It also included analyses of near-peer competitors, operational requirements, strategic portfolio analyses, and capability gaps. It did not include information on what role, if any, small businesses would have in developing or supplying the means to close capability gaps. Multiple Army officials explained that they did not specifically analyze the effect of modernization on small business as they anticipated continuing their current level of engagement with these entities and perhaps increasing it. Further, senior Army Futures Command officials stated that they consider engagement with small businesses to be critical to their modernization efforts as well as a key aspect of their mission. They also noted that the command’s headquarters location in Austin, Texas was chosen, in part, because of its close proximity to science, technology, and engineering talent and small business start-ups that can provide innovative solutions. Senior Army Futures Command officials told us they intend to continue the small business engagement efforts undertaken by components being integrated into the new command. Command officials stated that organizations transitioning to Army Futures Command will continue engaging with small businesses as they have in the past. For example, organizations transitioning to Army Futures Command awarded about $1.3 billion to hundreds of small businesses from fiscal years 2013 through 2017. In addition, prior to transitioning to the new command, the Combat Capabilities Development Command Army Research Laboratory and the Medical Research and Materiel Command participated in outreach events, such as industry days and conferences focused on small businesses, to network with and identify small businesses for potential future awards. According to officials from these commands, these efforts have historically led to business relationships using a variety of arrangements, including contracts, agreements using other transaction authority, grants, cooperative agreements, and cooperative research and development agreements. Officials from Army Futures Command stated that the past efforts of its components aimed at small business engagement would continue. The command also plans to continue utilizing the Small Business Innovation Research and Small Business Technology Transfer programs to award contracts, grants, and cooperative agreements to small businesses. Army Futures Command also intends to use their cross-functional teams to enhance small business engagement. These teams identify capability needs and requirements derived from the Army’s six modernization priorities. Officials told us that these cross-functional team efforts can serve as a way to focus small business engagement. For example, the cross-functional teams develop problem statements that describe the capabilities currently needed by the warfighter for a specific activity, such as a need for better communications and networking equipment. These problem statements can then be shared with small businesses as part of outreach efforts—such as challenge competitions or industry days—and lead to discussions about potential solutions. In addition, Army Futures Command officials told us the command intends to enhance its small business engagement through several initiatives—some of which are underway and some of which are in development. Officials told us they were not certain how many of these initiatives have led to specific contracts or awards, but noted that they had in some cases. Command officials told us that they have undertaken four initiatives to engage with small businesses for research and development: Army Research Laboratory Open Campus 2.0 is based on an existing Army Research Laboratory program to transition scientific research from universities to Army technology concepts. It will work with the research communities within universities to develop these concepts and potentially commercialize them. This program is currently directed by the office of the Deputy Commanding General, which is located at the command’s headquarters in Austin, Texas. Army Capability Accelerator is a new initiative that engages small businesses in developing and maturing concepts into prototypes and validating early-stage technologies. The accelerator is managed by the Army Applications Laboratory, which is located with the command’s headquarters in Austin, Texas. It also provides the support and infrastructure needed to accelerate small businesses’ concepts into solutions for warfighter capability gaps. Army Capability Accelerator has offices in Austin, Texas, and New York City, New York, and Army Futures Command intends to establish additional offices across the country. Army Capability Accelerator has hosted or co-hosted events allowing small businesses to demonstrate their capabilities and engage with the command. For example, the Austin office hosted a challenge competition in September 2018 to develop a solution for countering a drone threat. Similarly, according to officials, the New York City office hosted a challenge competition in December 2018 where the command funded awards to small businesses for positioning, navigation, and timing capabilities. Army Strategic Capital is a proposed restructuring of a prior initiative intended to leverage venture capital to offset Army development costs through co-investment with existing Army Small Business Innovation Research and Small Business Technology Transfer programs. According to Army Futures Command officials, this initiative will be managed by the office of the Deputy Commanding General in Austin, Texas, but is in the planning stages and could involve legislative or policy changes to clarify or augment the authorities of the command. Halo is a new initiative intended to accelerate the adaptation and transition of commercial and startup-derived products to Army applications and programs. This initiative involves more mature technologies and focuses on the acceleration and integration of prototypes. Army officials stated that Army Applications Laboratory will manage this initiative and that it is under development. These four initiatives are described further in Figure 3 below: Army officials noted that many of their new initiatives address concerns raised by small businesses in working with the government, including the Army, on research and development activities. According to a representative involved with the capability accelerator office in Austin— which involves a private company that works with small businesses to facilitate opportunities both across the private sector and, now, with the Army—small businesses have expressed concerns about working with the government. Specifically, these representatives identified concerns related to barriers to entry, length of time to reach an award, and the complexity of the government contracting process, among others. Similarly, representatives from the capability accelerator office in New York City stated that the Army needs a way to increase its visibility to small businesses in order to attract the interest of these companies. Army Futures Command officials acknowledged these concerns and said that they are developing efforts to alleviate or overcome them. For example, as part of its Halo initiative, the Army created a program intended to guide small businesses through the government contracting processes. In addition, Halo also plans to use business arrangements designed to decrease the time between initial contact with small businesses and the award of contracts or other agreements. In its initial efforts to enhance engagement with small businesses, Army Futures Command did not fully leverage the expertise of other Army organizations that previously facilitated small business engagement. Various Army officials have identified several early instances in which the command took steps to engage with small businesses without consulting other Army offices with relevant expertise. For example: Army Office of Small Business Programs—According to Army Office of Small Business Programs officials, the command did not consult with them (1) before engaging with small businesses in Texas for research and development efforts; (2) when establishing its small business office, which is still ongoing; and (3) before announcing hiring positions for that office. Army Office of Small Business Programs is positioned to provide direct support to various commands on small business activities. In particular, we previously reported that small business offices are responsible for assisting agencies in increasing small business participation and provide advice on acquisition strategies and market research. Subordinate Commands—According to Army officials, Army Futures Command has not fully engaged the organizations that transitioned, or are transitioning to, the command in terms of small business research and development efforts. Combat Capabilities Development Command and its subordinate command Army Research Laboratory, these organizations have years of experience working with small businesses on research and development efforts. Army Research Laboratory is the Army lead for the Small Business Technology Transfer program, and participates in the Small Business Innovation Research program along with other Combat Capabilities Development Command organizations; both of which are designed to stimulate technological innovation. Combat Capabilities Development Command officials stated they have had limited involvement with Army Futures Command headquarters on small business research and development issues. In addition, Medical Research and Development Command officials stated that Army Futures Command headquarters has not interacted with them on small business engagement beyond planning for the organization’s transfer to Army Futures Command. Historically, Medical Research and Materiel Command participated in the Small Business Technology Transfer and the Small Business Innovation Research programs and conducted outreach to small businesses through various events, such as industry days and conferences focused on small businesses. Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology—We reported in January 2019 that it was not yet clear how Army Futures Command will coordinate its responsibilities with the Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology. The office conducts outreach to small businesses, sponsors challenge competitions, and promotes small business participation in Army acquisitions. More recently, according to Army officials, the command is seeking to improve and formalize coordination roles and responsibilities related to research and development within and outside the command. For example, Although a formalized agreement between the command and Army Office of Small Business Programs does not yet exist, the command is now actively consulting with this office. According to Army small business officials, the command has been familiarizing small business staff with their office and its small business research and development efforts. The command has also been establishing its small business office with support from Army Office of Small Business Programs. In addition, Army small business officials stated that the office is assessing the command’s small business needs to determine how to allocate workforce resources. However, the effort has not been finalized. The command is also working to formalize small business relationships within and among its components. As part of this, the command established a Directorate of Operations at headquarters to facilitate integration of command activities across components, which would include those related to small business research and development. However, the command has not yet assigned a permanent director for the new directorate. According to Army Futures Command officials, as well as Army documents, the command will continue to develop coordination procedures related to research and development with the Assistant Secretary of the Army for Acquisition, Logistics, and Technology. The command is also working with the Assistant Secretary’s office on a challenge competition that aims to facilitate small business engagement with the Army and spur innovative technology. Army Futures Command does not have its own procurement authority, so the Army Contracting Command will provide it with contracting support. This support includes making awards to small businesses on behalf of Army Futures Command. Army Contracting Command officials told us they are also supporting the establishment of an Army Futures Command contracting office that would advise on contracting needs. For example, they sent temporary support staff to the headquarters of the new command and are helping with recruitment efforts for permanent personnel. Army Futures Command officials told us they had not prioritized coordinating with other Army organizations that have small business expertise because the command and its officials had other, more pressing priorities, such as establishing the command and engaging directly with small businesses as quickly as possible. Federal internal control standards state that during the establishment of an organizational structure management should consider how organizations across and outside of it interact in order to fulfill their overall responsibilities. This includes establishing reporting lines and roles and responsibilities within and outside the organization as they relate to small business engagement. With those coordination roles and responsibilities established, organizations are better able to communicate the quality information necessary to fulfill their overall small business engagement responsibilities. By taking actions to formally coordinate with and leverage other Army organizations’ expertise, such as coordinating outreach events, Army Futures Command could improve its opportunities to engage with small businesses and obtain access to the innovative research and development they could provide. Further, if the command does not formalize coordination roles and responsibilities, it risks potentially duplicating small business-related work and creating overlap and fragmentation. As previously noted, Army Futures Command stated it is continuing the efforts of its subordinate commands to engage with small businesses and is taking additional steps to enhance engagement. However, command officials told us they do not systematically track the number and timing of outreach events, the number of participants at these events, and the extent to which these outreach efforts result in business arrangements such as contracts. As a result, Army Futures Command officials were uncertain of how often the command—across all of its components—was engaging with small businesses for research and development efforts. For example, Army Applications Laboratory officials were not able to identify the number and timing of challenge competitions the command has hosted or is planning to host in the future. Some organizations that have transitioned to Army Futures Command, such as Combat Capabilities Development Command, continue to track small business engagement activities for their component. However, Combat Capabilities Development Command officials told us that they were unsure if this data will be tracked at Army Futures Command headquarters. According to Army Futures Command officials, the command has not prioritized tracking small business activities because it focused instead on establishing the command and engaging with small businesses as quickly as possible to identify innovative solutions. Officials did not provide a specific plan for tracking such engagement. According to Federal Internal Control Standards, management should establish monitoring activities for its internal control system and evaluate the results to remediate any identified challenge on a timely basis. Further, management should use quality information from reliable sources in a timely manner to achieve the objectives of the command. By tracking its small business engagement activities, Army Futures Command would have a more comprehensive understanding of the various efforts underway across the command. This would provide opportunities to examine its overall small business engagement efforts. Tracking such information would also allow the command to make adjustments to those efforts to ensure it obtains the innovative input from small businesses the command has stated it needs to achieve its modernization goals. Tracking small business engagement across the command components could also help reduce inefficiencies including overlap, fragmentation, and duplication of its small business engagement efforts. While Army Futures Command officials told us they consider small businesses to be critical to their success and they have taken steps to engage with small businesses, the command has not yet established measures for evaluating the effectiveness of that engagement across the command nor has it developed a plan to systematically assess these efforts. Command officials told us that they are in the process of considering various measures to do so, but they have not yet determined which specific measures, if any, they will use. There is also no time frame to establish these measures. According to Army Futures Command officials, they would consider small business engagement successful if, for example, a Small Business Innovation Research award resulted in an innovation or a technology that was later transitioned to a weapon systems program or a product that would further support an Army weapon systems program. Command officials told us they have not formalized and implemented these measures because the command and its officials have prioritized focusing on establishing the new command. Components subsumed by Army Futures Command have historically used performance measures to assess their small business engagement. For example, officials from Combat Capabilities Development Command told us that they previously used several outcome-based measures, including the number of Small Business Innovation Research products incorporated into fielded Army acquisition programs, contracts awarded to small businesses, and total dollars obligated to small businesses for research and development. This previously collected information was then provided to management in various small business offices in semiannual reports. Officials told us they have continued to monitor this information since the transition to Army Futures Command. Officials from Medical Research and Development Command also reported that they have performance measures and that they use these measures to assess the success of their small business engagement. For example, they said that they develop summary reports after outreach events with small businesses. These reports describe the event, outcomes, and how participation at the event enhanced utilization of small businesses for research and development efforts. The reports are also used internally as market research for future opportunities. Internal control standards call for management to use quality information to make informed decisions and to define objectives in specific and measurable terms so that performance toward achieving those objectives can be assessed. Management should also determine whether performance measures for the objectives are appropriate for evaluating performance. Once performance measures are defined, management should then establish and operate monitoring activities that allow them to evaluate the effectiveness of the internal control system. Establishing performance measures and developing a plan to capture and monitor information on its small business engagement would help ensure Army Futures Command is not missing opportunities to make informed management and investment decisions for its research and development efforts. Establishing these measures and a plan to monitor how the command assesses small business engagement would also help it to evaluate the overall effectiveness of its small business engagement in providing support to the warfighter and identifying which small business efforts have been most effective. The establishment of Army Futures Command represents a considerable change to how the Army develops new weapon systems and prepares for the future. While Army Futures Command is still finalizing how it will operate, it is already engaging with small businesses in various ways. However, the command could better manage these efforts. In particular, formalizing coordination roles and responsibilities with Army organizations that already have small business experience, such as the Army Office of Small Business Programs, would allow the command to leverage additional expertise as it pertains to small business engagement for research and development. In addition, Army Futures Command does not systematically track engagement across the command. By tracking this activity, the command could more effectively oversee and manage overall small business engagement. Finally, while Army Futures Command officials consider engaging with small businesses critical to the success of modernization, it has not yet developed performance measures to assess the effectiveness of its small business engagement nor has it developed a plan for systematically assessing its efforts. Establishing performance measures, and using them to assess small business engagement, would provide the command with information to evaluate, and potentially enhance, its engagement with small businesses to help accomplish its research and development efforts. We are making three recommendations to the Secretary of the Army. The Secretary of the Army should direct the Commanding General of Army Futures Command to formalize coordination roles and responsibilities for small business engagement in support of research and development with relevant Army entities. (Recommendation 1) The Secretary of the Army should direct the Commanding General of Army Futures Command to systematically track its small business engagement in support of research and development across its subordinate organizations. (Recommendation 2) The Secretary of the Army should direct the Commanding General of Army Futures Command, in coordination with relevant Army entities, to establish command-wide performance measures and develop a plan to use these measures to systematically assess the effectiveness of small business engagement in support of research and development. (Recommendation 3) We provided a draft of this report to the Army for review and comment. In its written comments, reproduced in appendix II, the Army concurred with all three of our recommendations. The Army also provided technical comments which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees; the Acting Secretary of Defense; and the Acting Secretary of the Army. In addition, the report is available at no charge on the GAO Website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or LudwigsonJ@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. You asked us to examine how small businesses that support research and development efforts could be affected by the establishment of Army Futures Command. This report (1) describes what analyses, if any, the Army conducted to determine the effect of its modernization initiatives on small businesses; (2) describes how Army Futures Command is engaging with small businesses to support research and development efforts and assesses how it is coordinating with other relevant Army organizations; and (3) assesses how Army Futures Command plans to track and measure the performance of its engagement with small businesses to support research and development efforts. We analyzed research and development contract awards and obligations made during fiscal years 2013 through 2017 for the Army. The data are presented in the background as it is prior to the establishment of Army Futures Command in 2018. For the number of contracts, we used the number of new base contract awards for research and development. For the obligations, we analyzed both newly awarded base contracts and associated orders under indefinite-delivery contracts since funds would be obligated at the order level. The obligations in this analysis include only those made during the fiscal year the contract was awarded. To identify and analyze contracts awarded during that time period, we requested data in the Federal Procurement Data System-Next Generation database from the Army. The Army used the product and service codes for research and development to extract the relevant data for fiscal years 2013 through 2017. The data also included contracts awarded through the Small Business Innovation Research and Small Business Technology Transfer programs for that time period and business size and registered location. We excluded foreign military sales obligations. We did not include subcontractor data. We obtained the funding codes for organizations that are transitioning to Army Futures Command, which includes the former Army Research, Development, and Engineering Command and the Army Medical Research and Materiel Command, portions of which are transitioning to the new command. To determine the proportion of contracts and associated obligations that supported these organizations, we used their funding codes to identify the number of contracts and associated obligations during our selected time period. To assess the reliability of the Federal Procurement Data System-Next Generation data, we electronically tested for missing data, outliers, and inconsistent coding. Based on these steps, we determined the data were sufficiently reliable for identifying and analyzing Army contracts awarded from fiscal years 2013 through 2017 for research and development efforts and their obligations. We obtained data on grants, cooperative agreements, and other types of agreements using the Defense Assistance Awards Data System. We conducted initial analysis on the data and discussed reliability and validity of the data with agency officials. As a result, we determined that the data were not sufficiently reliable for the purpose of this engagement and we excluded them from our review. To describe analyses the Army conducted on the potential effect modernization efforts could have on small businesses, we collected and reviewed available studies and analyses the Army conducted. We reviewed the Army’s science and technology portfolio analysis, studies related to the establishment and future organizational structure of Army Futures Command, and the Army’s modernization strategy to determine if the Army analyzed how small businesses could be affected. To describe how Army Futures Command is engaging with small businesses to support research and development efforts, we reviewed policies, procedures, and guidance from the Department of Defense, Department of the Army, Army Futures Command, and other relevant Army organizations on small business engagement. We also reviewed relevant sections of the Federal Acquisition Regulation, as well as Defense and Army supplements to the Federal Acquisition Regulation, to understand the framework for small business participation in support of research and development efforts. We also reviewed relevant statutes, regulations, and policies regarding research and development and small business programs. We collected and analyzed documentation on how Army Futures Command engages with small businesses, including its roles and responsibilities, outreach efforts, and award documentation as well as those of its subordinate components. To assess how Army Futures Command coordinates with other Army organizations, we reviewed policy documentation, such as a memorandum of understanding on coordinating contract support and for small business engagement, in addition to operational orders outlining roles and responsibilities. We assessed the information we collected against Federal Standards for Internal Control related to organizational structure, reporting lines, roles and responsibilities, and using quality information. To assess how Army Futures Command plans to track and measure its engagement with small businesses, we reviewed policies from the Department of Defense and Army on engagement with small businesses. To understand how Army Futures Command plans to track its small business engagement, we reviewed policy documentation from the command, operational orders, briefs and memoranda. We also reviewed documentation on how organizations tracked this data prior to transitioning to Army Futures Command. In order to assess any performance measures Army Futures Command plans to use to evaluate its small business engagement, we reviewed available documentation on the establishment of the command. We also reviewed documentation from organizations transitioning to Army Futures Command to determine how these organizations previously monitored and evaluated their small business engagement. In addition, we assessed the information we collected against Federal Standards for Internal Control related to establishing monitoring activities, using quality information, defining objectives, and evaluating results. To more completely understand the small business engagement efforts of the new command, we interviewed officials from various Army offices, including the Office of the Under Secretary of the Army, Army Futures Command, organizations transitioning to the new command, Army Office of Small Business Programs, members of the Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology, and Army Contracting Command. We also met with two private sector entities the Army has coordinated with for outreach to small businesses. These entities have experience in engaging small businesses both in the private sector and for government programs and discussed with us the concerns and challenges small businesses have in working with the government. These views are not generalizable but provide perspective on matters relevant to the Army’s efforts to engage with small businesses. We conducted this performance audit from September 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Jon Ludwigson at (202) 512-4841 or LudwigsonJ@gao.gov. In addition to the contact named above, J. Kristopher Keener (Assistant Director), Andrea C. Evans (Analyst-in-Charge), Hilary Benedict, Emily Bond, Frederick K. Childers, Matthew T. Crosby, Lori A. Fields, Julia Kennon, Jean McSween, Monique Nasrallah, Anh Nguyen, Kevin O’Neill, William Shear, and Anne Stevens made contributions to this report.", "summary": "The Army is modernizing its weapon systems to improve its ability to face near-peer adversaries. To consolidate and oversee these efforts, the Army established Army Futures Command. The command plans to work with small businesses to develop innovative capabilities through research and development activities. GAO was asked how the establishment of Army Futures Command could affect small businesses that support research and development efforts. This report examines, among other objectives, how the command (1) engages with small businesses and coordinates with other Army organizations and (2) plans to track and measure the effectiveness of that engagement. GAO reviewed the Army's internal analyses of its own modernization efforts; reviewed and analyzed policies and procedures on the command's small business engagement; and interviewed Army officials engaged in modernization efforts as well as two private companies selected because they facilitate Army's work with small businesses. Army Futures Command, established in June 2018 by combining several existing Army organizations and expected to be fully operational in July 2019, is engaging with small businesses. The command considers small business engagement critical to its success and officials reported it intends to continue the engagement activities of the organizations that are moving into it such as conducting outreach and awarding contracts. The Army recognizes the importance of small businesses and has awarded $2.3 billion to hundreds of small businesses from fiscal year 2013 through 2017. The command is also taking initial steps to enhance small business engagement (see figure). Army officials noted that these new efforts are intended to address concerns raised by small businesses in working with the government, such as delays between initial outreach and entering into contracts. However, the command has not fully leveraged other Army organizations that work with small businesses, such as the Army Office of Small Business Programs. According to command officials, they prioritized setting up the command structure and engaging with small businesses quickly, instead of focusing on coordination. The command has recently been working to improve coordination, but has not formally coordinated such as by establishing agreements with other Army organizations that have small business expertise. Doing so would help Army Futures Command leverage this past experience and avoid missing opportunities to engage with these companies and access innovative research and development. The command does not track how frequently or in what ways it engages with small businesses for research and development across all command components. Similarly, command officials stated they have considered performance measures to assess the effectiveness of their engagement efforts, but have not yet developed command-wide measures or a plan to assess effectiveness. Tracking and measuring engagement would help ensure the command obtains quality information that may help the Army evaluate, and potentially enhance, its small business engagement GAO is making three recommendations including that the Army Futures Command coordinate with relevant Army organizations on small business engagement efforts for research and development; systematically track its small business engagement; and develop command-wide performance measures and a plan to use them to assess the effectiveness of its small business engagement. The Army concurred with all three recommendations.", "document_type": "gao"}
{"report": "The national pipeline system consists of more than 2.7 million miles of networked pipelines transporting natural gas, oil, and other hazardous liquids. Natural gas and hazardous liquid pipelines—primarily buried underground in the continental United States—run under remote and open terrain, as well as densely-populated areas. There are three main types of pipelines based on the types of materials transported: Hazardous liquid: About 216,000 miles of hazardous liquid pipeline transport crude oil, diesel fuel, gasoline, jet fuel, anhydrous ammonia, and carbon dioxide. Natural gas transmission and storage: About 319,000 miles of pipeline—mostly interstate—transport natural gas from sources to communities. Natural gas distribution: About 2.2 million miles of pipeline—mostly intrastate—transport natural gas from transmission sites to consumers. Figure 1 depicts the network of hazardous liquid and natural gas transmission pipelines in the United States. More than 3,000 pipeline companies operate the nation’s pipeline systems, which can traverse multiple states and the U.S. borders with Canada and Mexico. Many pipeline systems are comprised of the pipelines themselves, as well as a variety of facilities, such as storage tanks, compressor stations, and control centers. Most pipeline systems are monitored through automated industrial control systems or Supervisory Control and Data Acquisition (SCADA) systems using remote sensors, signals, and preprogramed parameters to activate and deactivate valves and pumps to maintain flows within established tolerance levels. Pipeline accidents can occur from a variety of causes, including third- party excavation, corrosion, mechanical failure, control system failure, and operator error. Natural forces, such as floods and earthquakes, can also damage pipelines. Although pipeline releases have caused relatively few fatalities, a single pipeline accident can be catastrophic in terms of public safety and environmental damage. Figure 2 shows notable pipeline accidents since September 2010. According to TSA, pipelines are also vulnerable to physical attacks by crude or unsophisticated tactics, such as rudimentary explosives, arson, or equipment sabotage—largely due to their stationary nature, the volatility of transported products, and the dispersed nature of pipeline networks spanning urban and outlying areas. Threats to the nation’s pipeline systems include sabotage by activists, physical attack by terrorists, and cyber attack or intrusion by nations. In October 2016, environmental activists forced the shutdown of five crude oil pipelines in four states: Minnesota, North Dakota, Montana, and Washington State. Further, in January 2019, the Director of National Intelligence stated that China has the ability to launch cyber attacks that have caused localized, temporary disruptive effects on critical infrastructure—such as disruption of a natural gas pipeline for days to weeks—in the United States. Federal policy and public-private plans establish the roles and responsibilities for the protection of critical infrastructure, including pipelines. These policies and public private plans include Presidential Policy Directive /PPD-21 (PPD-21) and the National Infrastructure Protection Plan (NIPP). PPD-21, issued in February 2013, was developed to advance a national unity of effort to strengthen and maintain secure, functioning, and resilient critical infrastructure, which includes pipelines. PPD-21 reflects an all-hazards approach to protecting critical infrastructure, by accounting for the protection of critical infrastructure from natural or manmade threats or incidents. Examples of threats or incidents include natural disasters, cyber incidents, industrial accidents, pandemics, acts of terrorism, sabotage, and destructive criminal activity targeting critical infrastructure. PPD-21 also identifies the 16 critical infrastructure sectors and assigns roles and responsibilities for each sector among nine designated federal sector-specific agencies as shown in Figure 3. While PPD-21 identifies the critical infrastructure sectors and assigns responsibility for each sector’s sector-specific agency, the NIPP outlines critical infrastructure stakeholder roles and responsibilities regarding critical security and resilience. The NIPP describes a voluntary partnership model as the primary means of coordinating government and private sector efforts to protect critical infrastructure. As part of the partnership structure, the designated sector-specific agencies serve as the lead coordinators for security programs of their respective sector. For example, DHS and DOT are designated as co-sector-specific agencies for the transportation systems sector, which includes pipelines. Each sector also has a government coordinating council, consisting of representatives from various levels of government, and many have a sector coordinating council (SCC) consisting of owner-operators of these critical assets or members of their respective trade associations. For example, the Transportation Government Coordinating Council has been established, and the Pipeline Modal SCC has also been established to represent pipeline operators. Protecting the nation’s pipeline systems is a responsibility shared by both the federal government and private industry. As a result, several federal departments, agencies, and the private sector have significant roles in pipeline safety and security. The entities primarily responsible for pipeline safety and security are included below. Transportation Security Administration (TSA). TSA has primary oversight responsibility for the physical security and cybersecurity of transmission and distribution pipeline systems. Within TSA, the Policy, Plans, and Engagement’s Pipeline Security Branch is charged with overseeing its pipeline security program. Pursuant to the Implementing Recommendations of the 9/11 Commission Act of 2007 (9/11 Commission Act), TSA’s Pipeline Security Branch issued voluntary Pipeline Security Guidelines in 2011, and released revised guidelines in March 2018. Further, in accordance with the 9/11 Commission Act, TSA’s Pipeline Security Branch also identifies the top 100 critical pipeline systems in the nation. TSA also ranks the relative risk among these top 100 systems. Additionally, the Pipeline Security Branch is responsible for conducting voluntary security reviews, which assess the extent to which these 100 pipeline systems are following the intent of TSA’s Pipeline Security Guidelines. Pipeline and Hazardous Materials Safety Administration (PHMSA). PHMSA, within DOT, is responsible for regulating the safety of hazardous materials transportation and the safety of pipeline systems, some aspects of which may relate to pipeline security. PHMSA develops regulations for domestic interstate and intrastate natural gas and hazardous liquid pipelines. Its regulatory programs are focused on ensuring safety in the design, construction, operation, and maintenance of pipelines. Under PHMSA’s pipeline safety program, pipeline operators have primary responsibility for ensuring the integrity of their pipelines. PHMSA and some state pipeline safety offices are responsible for conducting inspections to oversee operators’ compliance with federal pipeline safety regulations and other federal requirements. Inspectors from PHMSA’s five regional offices and states are responsible for inspecting nearly 3,000 companies that operate 2.7 million miles of pipelines. Private sector. Although TSA has primary federal responsibility for overseeing interstate pipeline security, private sector and publicly-owned pipeline operators are responsible for implementing asset-specific protective security measures. As we previously reported, since the September 11th terrorists attacks, operators have increased their attention on security by incorporating security practices and programs into their overall business operations. Pipeline operators’ interests and concerns are primarily represented by five major trade associations—the Interstate Natural Gas Association of America, American Gas Association, American Public Gas Association, American Petroleum Institute (API), and Association of Oil Pipe Lines. According to TSA officials, pipeline operators, and association representatives, these associations have worked closely with the federal government on a variety of pipeline security-related issues, including collaborating on TSA’s voluntary standards and information sharing. The MOU Annex delineates TSA and PHMSA mutually agreed-upon pipeline security roles and responsibilities, consistent with their respective missions, and acknowledges that both agencies benefit by sharing each other’s expertise, among other things. Specifically, the MOU Annex identifies 11 program areas, where TSA and PHMSA agreed to coordinate their respective roles and responsibilities. The first program area for example, calls for both agencies to coordinate efforts to identify critical infrastructure, and to share relevant data and observations found during respective safety inspections and security assessments. Another program area addresses coordination in developing transportation security standards, regulations, guidelines, or directives. The MOU Annex further provides that TSA and PHMSA are to seek early and frequent coordination in developing such standards, regulations, guidelines, or directives. They are also to review the adequacy of existing standards in the private and public sector, and identify any gaps that should be addressed through rulemaking, guidelines, or directives, among other items. For a complete listing of the MOU Annex’s 11 program areas, including TSA and PHMSA roles and responsibilities and agreed-upon actions, see appendix I. TSA and PHMSA have both noted various developments that have occurred since 2006 that may affect their roles and responsibilities related to pipeline security. However, the MOU Annex has not been updated since its inception in 2006 to consider incorporating these changes which includes subsequently issued presidential directives, the establishment of the Cybersecurity Infrastructure and Security Agency (CISA), and distinctions between current TSA and PHMSA current inspection operations. As a result, the Annex is not current and may not fully reflect the agencies’ pipeline safety and security-related activities. For example, Homeland Security Presidential Directive/HSPD-7 (HSPD-7), which is cited as an underlying authority in both the 2004 MOU and 2006 MOU Annex was revoked and replaced by PPD-21 in 2013. According to PPD-21, the directive advances a national unity of effort to strengthen and maintain secure, functioning, and resilient critical infrastructure by, among other things, refining and clarifying critical infrastructure-related functions, roles, and responsibilities across the federal government. PPD- 21 further provides, however, that plans developed pursuant to HSPD-7 shall remain in effect until specifically revoked or superseded. According to TSA and PHMSA officials, statements of Executive Branch policy including presidential directives such as PPD-21 include changes that could impact their pipeline security and safety roles and should be considered in any future revisions to the MOU Annex. Further, PHMSA officials also told us that TSA and PHMSA’s roles and responsibilities in identifying critical infrastructure should be reviewed given the establishment of the CISA in November 2018. CISA, formerly the DHS National Protection and Programs Directorate, is responsible for, among other things, coordinating a national effort to secure and protect against critical infrastructure risks. These responsibilities include coordinating with sector-specific agencies to carry out its cybersecurity and critical infrastructure activities. TSA and PHMSA officials stated that they have closely coordinated in identifying critical infrastructure when responding to past national emergencies. For example, TSA identified and provided PHMSA with information on the pipelines that supplied fuel to specific airports during the hurricane seasons in 2017 and 2018. However, PHMSA officials stated that both TSA and PHMSA should consider reviewing how these types of efforts may need to be coordinated with CISA in the future and whether any adjustments to respective roles and responsibilities in the MOU Annex are needed. In addition, representatives from all of the industry associations that we interviewed stated that the agreement should be revised to consider how the establishment of CISA may impact current TSA and PHMSA pipeline security roles and responsibilities. TSA officials stated that they do not believe that the establishment of CISA impacts TSA’s roles and responsibilities for identifying pipeline critical infrastructure. While CISA may or may not have impacts on TSA and PHMSA’s pipeline security roles, reviewing the MOU Annex in light of new developments, such as the CISA, would allow the TSA and PHMSA to determine whether updates are necessary. TSA and PHMSA officials stated that distinctions in current inspections and enforcement operations necessitate a revision to the MOU Annex. The MOU Annex states that agencies are to explore opportunities for collaboration in inspection and enforcement activities. According to TSA and PHMSA officials, they have since explored the possibility for conducting joint activities and found that distinctions in their respective operating environments and roles and responsibilities do not allow for joint inspection and enforcement activities. For example, PHMSA conducts physical inspections of facilities to assess pipeline operators’ compliance with pipeline safety regulatory requirements and relies on a range of enforcement activities, such as civil penalties to ensure that pipeline operators correct safety violations and prevent safety problems. TSA, however, conducts voluntary security assessments of pipeline’s corporate security programs and critical facilities and relies on pipeline operators’ willingness to participate and implement recommended changes to improve pipeline security. As a result, TSA and PHMSA officials stated that pipeline operators are reluctant to participate in a voluntary assessment that might include PHMSA inspectors because they represent a regulatory agency. TSA, PHMSA and industry association representatives we interviewed agreed that the annex should be updated to accurately reflect current distinctions in the agencies’ roles and responsibilities and their respective operating environments. PHMSA officials stated that they had planned to review the MOU Annex in 2018 to assess current roles and responsibilities and determine whether any updates to the MOU Annex were needed, but efforts were delayed because of competing priorities such as addressing the aftermath of major hurricanes in 2017 and 2018. Specifically, TSA and PHMSA had agreed to an initial list of timeframes for reviewing the MOU Annex and these timeframes called for the agencies to complete the MOU Annex revision in 2018. However, as of March 2019, TSA and PHMSA have yet to complete the review and although both agencies stated that the review is ongoing, neither agency could provide updated timeframes for completion. Furthermore, while the Annex recognizes that TSA and PHMSA may propose agreed-upon amendments or modifications to the agreement, it does not call for regular or periodic reviews to identify whether any updates or revisions are needed and, as appropriate, implemented. TSA and PHMSA officials, as well as the industry association representatives we interviewed all reported that the MOU Annex helped to coordinate pipeline security and safety efforts because: (1) it is a signed written agreement that can be readily consulted; (2) it memorialized respective TSA and PHMSA roles and responsibilities for government leaders and staff at the time; and (3) it can be modified or amended as needed. Standards for Internal Control in the Federal Government states that periodic review of policies, procedures, and related control activities should occur to determine their continued relevance and effectiveness in achieving identified objectives or addressing related risks. In addition, documentation of any changes made as a result of such reviews, such as changes to an entity’s roles and responsibilities or in technology, should occur to ensure that such controls are clear over time as staff change within an organization. Standards for project management state that managing a project involves, among other things, developing a timeline with milestone dates to identify points throughout the project to reassess efforts under way to determine whether project changes are necessary. By developing and implementing mutually agreed upon time frames for reviewing the annex and updating it, as appropriate, TSA and PHMSA could better ensure that the roles and responsibilities for TSA and PHMSA remain current. Additionally, including a provision in the annex for periodically reviewing for needed updates would help ensure the agreement consistently reflects relevant and updated information on TSA and PHMSA’s roles and responsibilities. TSA and PHMSA have communicated their respective pipeline safety and security roles and responsibilities by issuing pipeline security guidance and safety regulations, issuing a joint advisory bulletin, and maintaining informal contacts with pipeline stakeholders when conducting outreach activities, pipeline security assessments, or safety inspections. TSA security guidelines. TSA’s Pipeline Security Branch first issued its voluntary Pipeline Security Guidelines in 2011, and revised them in March 2018. The guidelines include TSA’s recommendations for pipeline industry security practices, such as establishing a corporate security program, conducting security vulnerability assessments, and identifying critical facilities. The guidelines also recommend facility security and cybersecurity measures, which serve as the basis for the pipeline security assessments conducted by TSA’s Pipeline Security Branch. PHMSA regulations. PHMSA’s Office of Pipeline Safety issues and enforces intrastate and interstate regulations covering aspects of pipeline safety, including the design, construction, operation and maintenance, and spill response for hazardous liquid and gas pipeline facilities, including liquefied natural gas facilities. Advisory bulletins. PHMSA also issues advisory bulletins to communicate safety-related conditions to pipeline operators, and can issue advisory bulletins in coordination with TSA to notify pipeline operators of a security incident. Such bulletins may include identifying the affected operators, describing the threat, and providing information on federal resources for assistance. For example, in response to physical intrusions on pipelines and a coordinated campaign by domestic saboteurs, and to remind pipeline operators of the importance of safeguarding and securing their pipelines from physical and cyber intrusion or attack, PHMSA, in coordination with TSA, issued an advisory bulletin in 2016. The bulletin also included a brief discussion of TSA’s and PHMSA’s roles on pipeline safety and security. Forums and routine interactions with operators. TSA and PHMSA officials also reported that they communicate their agencies’ respective roles and responsibilities for pipeline safety and security to stakeholders when conducting general outreach, information sharing efforts, or inspections or assessments. TSA and PHMSA officials noted that these activities provide opportunities for agency officials and pipeline stakeholders to clarify their roles and responsibilities should pipeline operators have questions. Examples of such community outreach activities include attending meetings of the Oil and Natural Gas subsector SCC or the Pipeline Modal SCC, and TSA’s annual International Pipeline Security Forum. TSA officials also said that TSA’s monthly and quarterly unclassified threat briefings provided TSA officials and pipeline stakeholders the opportunity to discuss and clarify their roles and responsibilities. Additionally, TSA produces classified and unclassified threat assessments on physical and cyber threats to pipelines, which according to agency officials can help to clarify TSA’s security role. Finally, TSA and PHMSA officials said that pipeline security assessments and safety inspections and other enforcement activities that the agencies regularly conduct are also opportunities to communicate their roles and responsibilities. For example, TSA officials reported that should an operator ask for assistance regarding a safety issue while TSA staff was conducting a security review, TSA staff would be able to refer the operator to PHMSA to address the issue. Similarly, PHMSA officials stated that inspectors would refer an operator to TSA or its pipeline security guidelines should the operator have questions regarding, for example, what security measures to implement. The representatives of the four pipeline associations we interviewed reported that TSA and PHMSA had clearly communicated their respective roles and responsibilities to pipeline stakeholders. Specifically, all of the association representatives said that their membership understood that TSA is responsible for pipeline security matters and PHMSA is responsible for pipeline safety matters. For example, one industry association representative stated that they had contacted their members to determine whether they were unclear regarding TSA’s and PHMSA’s respective roles and responsibilities and that members reported the roles were clear to them. Further, another association representative reported that the initial security reviews and outreach efforts that TSA conducted after the pipeline security program was created helped pipeline operators to understand that its role was to oversee pipeline security. In addition, all of the association representatives we interviewed stated that the MOU Annex helped ensure that TSA and PHMSA understood and respected each other’s roles and responsibilities. As a result, according to the association representatives, their pipeline operator membership had not experienced challenges associated with overlapping or duplicative efforts on the part of TSA and PHMSA pipeline safety or security programs. In accordance with the 9/11 Commission Act, TSA issued its Pipeline Security and Incident Recovery Protocol Plan in March 2010. The plan’s stated intent is to establish a comprehensive interagency approach to counter risks, coordinate federal agencies’ actions, and minimize the consequences of incidents involving pipeline infrastructure as well as recovery time from them. The plan also defines the roles and responsibilities of federal agencies; tribal, state, and local governments; and the private sector during a pipeline incident. It also defines the measures they may take related to pipeline infrastructure security incidents. According to the plan TSA, PHMSA, the Department of Energy (DOE), and the Federal Bureau of Investigation (FBI) have principal roles in pipeline incident response, while other agencies such as the U.S. Coast Guard, the Federal Emergency Management Agency (FEMA), and the National Transportation Safety Board (NTSB) have supporting roles. The following are examples of agencies’ roles and responsibilities in each of the plan’s three response phases. Prevention/protection. TSA is responsible for monitoring pipeline owner and operators’ implementation of its pipeline security guidelines, and PHMSA is responsible for enforcing its pipeline safety regulations. TSA, in addition to the FBI, is responsible for assessing the credibility of any physical or cyber threat information it receives and sharing any intelligence related to pipeline security with pipeline owners and operators. Response. TSA is responsible for coordinating information sharing between federal agencies and pipeline stakeholders, and PHMSA is responsible for coordinating federal agency activities with the affected pipeline operator and state pipeline safety agency. The plan also states that the FBI is responsible for investigating attempted or successful attacks on pipeline infrastructure including those that are believed to have a nexus to terrorism. Recovery. PHMSA is primarily responsible for working with the pipeline operator, along with other supporting federal agencies, to facilitate service restoration. DOE is responsible for monitoring flows of throughput in the affected pipeline system or systems, assessing regional, national, and global impacts of an incident on energy infrastructure throughout all three phases. Appendix I provides more details on key federal agencies’ and pipeline operators’ roles and responsibilities, as well as the actions they may take in response to an incident as detailed in the plan. TSA’s plan states that it will be updated periodically to address changes in pipeline security threats, technology, and federal laws and policies. Further, Standards for Internal Control in the Federal Government states that periodic review of policies, procedures, and related control activities should occur to determine their continued relevance and effectiveness in achieving identified objectives or addressing related risks. In addition, internal control standards also states that changes in an entity’s programs or activities, organizational structure, personnel, or technology can affect the operating environment and management can respond by revising internal controls on a timely basis to ensure effectiveness. However, TSA has not reviewed or revised its 2010 plan to ensure it addresses changes in at least three key areas: cybersecurity-related laws and policies, incident management policies, and DHS’s terrorism alert system as described below. TSA’s 2010 plan includes some discussion of cyber threats and refers operators to guidance they may use to better secure their SCADA and control systems. However, the plan does not identify the cybersecurity roles and responsibilities of federal agencies that are identified in the plan, such as DOE, Federal Energy Regulatory Commission (FERC), or the FBI, or discuss the measures these agencies should take to prevent, respond to, or support pipeline operators following a cyber incident involving pipelines. TSA’s 2010 plan also has not been updated to reflect current cybersecurity incident response guidance. In December 2016, DHS issued its National Cyber Incident Response Plan (NCIRP). The NCIRP is to be the primary framework for stakeholders, including pipeline operators, to understand how federal departments and agencies provide resources to support response operations for a significant cyber incident. NCIRP identifies the FBI and the National Cyber Investigative Joint Task Force as responsible for investigating reported cyber incidents. NCIRP also identifies the National Cybersecurity and Communications Integration Center (NCCIC), an agency within DHS, as responsible for providing technical assistance to affected entities, such as pipelines, to mitigate vulnerabilities and reduce impacts of cyber incidents. NCCIC is also to share information across the public and private sectors to protect against similar incidents in the future. In addition, NCIRP provides guidance detailing when and to which federal agencies or entities the public should report a cyber incident. These include the FBI, the National Cyber Investigative Joint Task Force, U.S. Secret Service, and NCCIC. For example, NCIRP states that any cybercrime—including computer intrusions or attacks, theft of trade secrets, criminal hacking, terrorist activity, espionage, sabotage, or other foreign intelligence activity—is to be reported to FBI field offices’ cyber task forces. However, TSA’s plan does not include this information or describe what measures, if any, the agencies with pipeline-related roles and responsibilities listed in NCIRP are to take in response to a pipeline cyber incident. Moreover, the 2010 plan does not account for other agencies whose roles and responsibilities are related to critical infrastructure, such as pipelines and cybersecurity. Specifically, the plan does not account for the role of NCCIC, which was established in 2009. In addition, TSA’s 2010 plan does not account for CISA’s role in cyber threat response activities or how it may affect other agencies’ roles and responsibilities for pipeline incident response. TSA officials acknowledged that reviewing and, as appropriate, revising the plan would be beneficial to ensuring the plan addresses current pipeline security threats, technology, and federal laws and policies. They stated TSA had not updated the plan to include cybersecurity response protocols because an overarching cybersecurity response protocol for all critical infrastructure sectors—not just pipelines—should first be developed. According to TSA officials, developing a pipeline cybersecurity response protocol would require a whole-of-government approach, as well as coordination with private sector and input from many sectors because of the challenges and complexity of critical infrastructure cybersecurity in general. However, through NCIRP, DHS provided a cybersecurity response protocol across all critical infrastructure sectors in December 2016. Further, NCIRP states that public and private sector entities should consider creating an operational cyber incident response plan to further organize and coordinate their efforts in response to cyber incidents. Therefore, TSA could potentially provide such an operational cyber incident response plan for the pipeline sector in its plan. TSA could also better ensure that pipeline operators understand how federal agencies may provide support in response to a cyber incident by periodically reviewing and, as appropriate, revising the plan to include its cyber incident response plan. Representatives of the four pipeline associations we interviewed told us that their membership more clearly understood federal agencies’ roles and responsibilities related to physical incidents than to cybersecurity. For example, for physical incidents the representatives stated that their members clearly understood that they are to first notify local first responders (often through the emergency 911 system) and appropriate state or federal regulators, and are to contact either the National Response Center or TSA’s Transportation Security Operations Center (TSOC), depending on the nature of the incident. However, they stated that they did not believe all of their members clearly understand that they are to report any actual or suspected cyber incidents that could impact pipeline industrial control systems or other information technology-based systems to the NCCIC. All of the association representatives told us that the process for reporting a cyber incident is less clear because, in part, of the large number of federal agencies with a cybersecurity-related role. One of the representatives also attributed the lack of clarity to the reorganization of NCCIC, and the establishment of CISA. Further, all of the representatives we interviewed indicated that clarifying the cybersecurity roles and responsibilities of DOE, Federal Energy Regulatory Commission (FERC), and TSA would, among other things, improve operators’ ability to appropriately report and respond to a cyber incident. TSA also has not updated the plan to address changes in federal incident management and response policies that have occurred since the plan was developed in 2010. The plan states that it is to be consistent with the National Response Framework (NRF) and the National Incident Management System (NIMS) incident command system procedures. The NRF was first issued in 2008 and described the roles, responsibilities and coordinating structures for delivering core capabilities during incident response. According to FEMA, it revised the NRF in 2013 and 2016 to reflect lessons learned from real world events and other experiences since the framework was first developed. Likewise, NIMS was developed in 2004 as a comprehensive, national approach to incident management that was to be applicable at all jurisdictional levels and across functional disciplines, such as law enforcement, public health, or public works. According to FEMA, it revised NIMS in 2017 to reflect and incorporate policy updates and lessons learned from exercises and real-world incidents. The revision was also intended to clarify that NIMS applies to all stakeholders with incident management roles, and to enhance guidance on information management processes, data collection plans, social media integration, and the use of geographic information systems. TSA officials acknowledged the benefit of periodically reviewing, and if necessary, revising the plan to reflect FEMA’s revisions to NIMS or the NRF, but had not done so because of competing priorities. TSA has also not updated the plan to address changes DHS made to its terrorist alert system in 2011. Consistent with the 9/11 Commission Act, the plan describes actions that federal agencies can take at each color- coded level of the Homeland Security Advisory System to ensure the increased security of pipeline infrastructure. For example, under the protect/prevent phase, the plan states that when there is a high risk of a terrorist attack (i.e., red: severe condition) and threat is general and not specific to pipelines, TSA and PHMSA are to coordinate to identify the potential for any related or cascading events that may impact the pipeline sector. However, if there is a specific threat to pipelines, TSA, in collaboration with pipeline operators, is to identify any immediate protective measures that pipeline operators are to implement. TSA is also to ensure pipeline operators have the information necessary to implement these measures, and, if necessary, to issue security directives. In 2011, DHS replaced the four color-coded alert system of the Homeland Security Advisory System with the National Terrorism Advisory System, which has only two alert levels (elevated threat and imminent threat). TSA issued revised protective measures that pipeline operators are to take under either threat condition in April 2011 and March 2018. However, TSA has not updated the plan to communicate the actions federal agencies can take at either level of the National Terrorism Advisory System to ensure the increased security of pipeline infrastructure. TSA officials acknowledged that periodically reviewing and, as appropriate, revising the plan would help to clarify federal agencies’ roles and responsibilities for addressing pipeline security. TSA officials reported that they have not updated the plan since 2010 because they faced competing priorities. However, as described earlier, TSA’s incident response plan was developed to provide a comprehensive interagency approach to important activities such as countering risks, coordinating federal agencies’ actions and minimizing the consequences of incidents involving pipeline infrastructure. Further, the plan itself states that it will be updated periodically to address changes in pipeline security threats, technology, and federal laws and policies. By periodically reviewing and, as appropriate, revising its Pipeline Security and Incident Recovery Protocol Plan, TSA could better ensure that the plan addresses all possible and relevant threats to pipeline systems, such as cybersecurity, and fully incorporates relevant changes, such as those related to incident management and DHS’s terrorism alert system. By doing so, TSA could also provide greater assurance that federal agencies understand the actions they are to take to prevent, respond to, or recover from a physical or cyber incident. TSA and PHMSA share responsibility for safeguarding the nation’s pipeline systems from catastrophic events. While the 2006 MOU Annex delineates TSA’s and PHMSA’s mutually agreed-upon pipeline security roles and responsibilities, it has not been reviewed since its inception to consider pipeline security developments. By developing and implementing a mutually agreed upon timeline with timeframes for reviewing the annex and as appropriate, updating it, TSA and PHMSA could better ensure that their roles and responsibilities are properly documented and updated in a timely manner to remain current. Furthermore, by revising the MOU Annex to include a provision for periodically reviewing the annex for needed updates, TSA and PHMSA could better ensure the agreement consistently reflects relevant and updated information on their roles and responsibilities. Similarly, TSA’s Pipeline Security and Incident Recovery Protocol Plan— which defines the roles and responsibilities of federal agencies; tribal, state, and local governments; and the private sector for responding to a pipeline incident—also has not been updated to reflect changes in federal laws or policies since the plan was issued in 2010. By periodically reviewing and, when appropriate, updating its Pipeline Security and Incident Recovery Protocol Plan, TSA could better ensure that the plan addresses and fully incorporates changes relevant to cybersecurity, incident management and DHS’s terrorism alert system, among others. By doing so, TSA could also better ensure that federal agencies’ actions are well coordinated in response to a pipeline-related physical or cyber incident, and that pipeline stakeholders understand federal agencies’ roles and responsibilities in preparing for, responding to, or supporting pipeline operators to restore service after a pipeline-related physical or cyber incident. We are making a total of five recommendations including three to TSA and two to PHMSA: The TSA Administrator should work with the PHMSA Administrator to develop and implement a timeline with milestone dates for reviewing and, as appropriate, updating the 2006 MOU Annex. (Recommendation 1) The PHMSA Administrator should work with the TSA Administrator to develop and implement a timeline with milestone dates for reviewing and, as appropriate, updating, the 2006 MOU Annex. (Recommendation 2) The TSA Administrator, in consultation with the PHMSA Administrator should revise the 2006 MOU Annex to include a provision requiring periodic reviews of, and as appropriate, corresponding updates to the Annex.(Recommendation 3) The PHMSA Administrator, in consultation with the TSA Administrator should revise the 2006 MOU Annex to include a provision requiring periodic reviews of, and as appropriate, corresponding updates to the Annex.(Recommendation 4) The TSA Administrator should periodically review, and as appropriate, update the 2010 Pipeline Security and Incident Recovery Protocol Plan to ensure the plan reflects relevant changes in pipeline security threats, technology, federal law and policy, and any other factors relevant to the security of the nation’s pipeline systems. (Recommendation 5) We provided a draft of this report to DHS and DOT. DHS and DOT provided written comments which are reproduced in appendices III and IV respectively. We also provided draft excerpts of this product to the American Petroleum Institute (API), the American Gas Association, the Interstate Natural Gas Association of America, and the American Public Gas Association. For those who provided technical comments, we incorporated them as appropriate. With regard to our first recommendation, that TSA work with the PHMSA to develop and implement a timeline with milestone dates for reviewing and, as appropriate, updating the 2006 MOU Annex, DHS stated that TSA will work with PHMSA to develop and implement a timeline with milestone dates for reviewing and updating, as appropriate, the 2006 MOU Annex. DHS estimated that this effort would be completed by August 31, 2019. This action, if fully implemented, should address the intent of this recommendation. With regard to our second recommendation, that PHMSA work with TSA to develop and implement a timeline with milestone dates for reviewing and, as appropriate, updating the 2006 MOU Annex, DOT concurred and stated it would provide a detailed response within 180 days of the issuance of this report. With regard to our third recommendation, that TSA, in consultation with PHMSA, revise the 2006 MOU Annex to include a provision requiring periodic reviews of, and as appropriate, corresponding updates to the Annex, DHS stated that TSA will, in consultation with PHMSA, revise the 2006 MOU Annex to include a provision requiring periodic reviews of, and as appropriate, corresponding updates to the Annex. DHS estimated that this effort would be completed by March 31, 2020. This action, if fully implemented, should address the intent of this recommendation. With regard to our fourth recommendation, that PHMSA, in consultation with TSA, revise the 2006 MOU Annex to include a provision requiring periodic reviews of, and as appropriate, corresponding updates to the Annex, DOT concurred and stated it would provide a detailed response within 180 days of the issuance of this report. With regard to our fifth recommendation, that TSA periodically review, and as appropriate, update the 2010 Pipeline Security and Incident Recovery Protocol Plan to ensure the plan reflects relevant changes to pipeline security threats, technology, federal law and policy, and any other factors relevant to the security of the nation’s pipeline systems, DHS concurred and estimated that TSA will complete its first review by December 31, 2019. DHS further stated that it will establish a timeline for updating the plan should the review determine that an update is necessary. This action, if fully implemented, should address the intent of this recommendation. We are sending copies of this report to the appropriate congressional committees; the Secretary of Homeland Security, Secretary of Transportation; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact William Russell at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V. The Transportation Security Administration (TSA) and Pipeline and Hazardous Materials Safety Administration (PHMSA),”the parties”, recognize that the following program areas are important to the development and deployment of an enhanced security strategy for the transportation of hazardous materials by all modes, including pipeline. This appendix summarizes the roles and responsibilities of key federal agencies as well as the actions that they may take in response to an incident as detailed in Transportation Security Administration’s (TSA) 2010 Pipeline Security and Incident Recovery Protocol Plan. A summary of pipeline stakeholder’s roles, responsibilities, and examples of actions that may be taken during each incident response phase is presented below. Prevention/Protection. During the prevention/protection phase, pipeline operators are to use TSA’s pipeline security guidance and the Pipeline and Hazardous Materials Safety Administration’s (PHMSA) safety regulations as the framework to prepare and prevent against an incident. TSA is responsible for monitoring pipeline owners and operators’ implementation its security guidelines, and PHMSA is responsible for enforcing its safety regulations. The plan also states that during this phase TSA is to assume a primary role for ensuring federal agencies’ actions are coordinated through protective security advisors (PSAs). In addition, the Federal Bureau of Investigation (FBI) is responsible for assessing the credibility of a known threat, preparing and implementing a preliminary investigative plan, and, if necessary, disseminating public safety notifications. The Department of Energy (DOE) is responsible for assessing and monitoring pipeline systems for supply shortages. The prevention/protection section of the plan also describes how agencies are to share and assess threat information. For example, the plan states that TSA, PHMSA, or any federal agency that receives threat information regardless of the source, must immediately notify the FBI. It also states that if the FBI receives intelligence about a pipeline threat, it is to share this information with TSA. TSA is then to notify the pipeline operator and, if necessary, provide recommendations for additional protective measures. Finally, the prevention/protection section of the plan defines actions various agencies can implement during a heightened security threat level to increase protection from a potential attack. For example, when there is a high risk of a terrorist attack (i.e., red: severe condition) and threat is general and not specific to pipelines, TSA and PHMSA are to coordinate to identify the potential for any related or cascading events that may impact the pipeline sector. If there is a specific threat to pipelines, TSA is, in collaboration with pipeline operators, to identify any immediate protective measures that ought to be taken by pipeline operators, and ensure pipeline operators have the information necessary to implement them, and, if necessary issue security directives. Response. According to the plan, pipeline owners or operators are to notify local first responders and state regulators through the emergency 911 system. After the pipeline operator has notified local government, they are to contact the National Response Center (NRC) if the incident results in an unintentional release or causes significant damage. As we previously reported, pipeline operators are also requested to report any physical security incident that is indicative of a deliberate attempt to disrupt pipeline operations or activities that could be considered precursors to such an attempt to TSA’s Transportation Security Operations Center (TSOC). Once TSA has been notified of an incident by a pipeline operator, its Pipeline Security Branch is to monitor the incident, notify relevant federal agencies, and, if deemed appropriate, activate the Interagency Threat Coordination Committee (ITCC). PHMSA may also deploy on-scene pipeline inspectors and investigators which are to among other things, coordinate federal agencies’ activities with the affected pipeline operator and state pipeline safety agency, provide subject matter expertise to the incident command, and direct safe restoration of pipeline facilities and services. The plan also states that, during the response phase, responsibility for investigating the incident falls to NTSB or the FBI depending on whether the incident is determined to be the result of criminal activity. The FBI is solely responsible for investigating any pipeline security incident that appears to be an intentional criminal act. For example, if the incident were suspected to be the result of terrorist attack, the National Joint-Terrorism Task Force would conduct an investigation of the attack, and if appropriate, with assistance from other FBI assets. If, however, the incident resulted in fatalities, substantial property damage, or significant injury to the environment, NTSB would have responsibility for investigating the incident, and may issue safety recommendations to help prevent future accidents. Recovery. When response activities are complete, PHMSA is to have primary responsibility for overseeing pipeline operators’ safe restoration of service with TSA and other federal agencies serving primarily in support roles. PHMSA, for example, is to work with the owner/operator to facilitate restoration of service by, among other things, providing technical oversight, advice, and guidance to owner/operators; coordinating recovery activities with state pipeline safety agency, and evaluate whether to a special permit is necessary to facilitate an expedited restoration of services. Meanwhile, DOE is to continue to assess the impacts of an incident on energy infrastructure, and advise federal, state, tribal, and local authorities on priorities for energy restoration, assistance, and supply. In addition to the contact named above, Ben Atwater, Assistant Director and Michael C. Lenington, Analyst-in-Charge, managed this assignment. Nanette Barton, Eric Hauswirth, Susan Hsu, and Thomas Lombardi also made significant contributions to this report.", "summary": "More than 2.7 million miles of pipeline transport natural gas, oil, and other hazardous liquids needed to operate vehicles and heat homes, among other things, in the United States. Responsibility for safeguarding these pipelines is shared by TSA, within the Department of Homeland Security (DHS); PHMSA, within the Department of Transportation (DOT); and pipeline operators. TSA oversees the security of all transportation modes, including pipelines. PHMSA oversees pipeline safety. DHS and DOT signed a MOU on their roles across all transportation modes in 2004. In 2006, TSA and PHMSA signed an annex to the MOU (MOU Annex) to further delineate their pipeline security-related responsibilities. The TSA Modernization Act includes a provision for GAO to review DHS and DOT roles and responsibilities for pipeline security. This report addresses, among other things: (1) the extent the MOU Annex delineates TSA's and PHMSA's pipeline security roles and responsibilities; and (2) the extent TSA has communicated federal incident response procedures for pipeline breaches to stakeholders. GAO reviewed the MOU annex and related documents and TSA's Pipeline Security and Incident Recovery Protocol Plan, and interviewed officials from PHMSA, TSA, and four pipeline associations. The memorandum of understanding (MOU) Annex signed by the Transportation Security Administration (TSA) and Pipeline and Hazardous Materials Safety Administration (PHMSA) in 2006 delineates their mutually agreed-upon roles and responsibilities for pipeline security, but has not been reviewed to consider pipeline security developments since its inception. As a result, the annex may not fully reflect the agencies' pipeline security and safety-related activities. Efforts to update the annex were delayed by other priorities. As of June 2019, there are no timeframes for completion. By developing and implementing timeframes for reviewing the MOU Annex and updating it, as appropriate, TSA and PHMSA could better ensure any future changes to their respective roles and responsibilities are clearly delineated and updated on a regular basis. TSA's Pipeline Security and Incident Recovery Protocol Plan, issued in March 2010, defines the roles and responsibilities of federal agencies and the private sector, among others, related to pipeline security incidents. For example, in response to a pipeline incident, TSA coordinates information sharing between federal and pipeline stakeholders and PHMSA coordinates federal activities with an affected pipeline operator to restore service. However, TSA has not revised the plan to reflect changes in at least three key areas: pipeline security threats, such as those related to cybersecurity, incident management policies, and DHS's terrorism alert system. By periodically reviewing and, as appropriate, updating its plan, TSA could better ensure it addresses changes in pipeline security threats and federal law and policy related to cybersecurity, incident management and DHS's terrorism alert system, among other things. TSA could also provide greater assurance that pipeline stakeholders understand federal roles and responsibilities related to pipeline incidents, including cyber incidents, and that response efforts to such incidents are well-coordinated. GAO is making five recommendations, including that: (1) TSA and PHMSA develop and implement a timeline for reviewing and, as appropriate, updating the 2006 MOU Annex; and (2) TSA periodically review, and as appropriate, update its 2010 pipeline incident recovery plan. DHS and DOT concurred with these recommendations.", "document_type": "gao"}
{"report": "VHA began using the SAIL system in 2012 to measure, evaluate, and benchmark the quality, efficiency, and productivity of medical centers, and to highlight successful strategies of high-performing medical centers. SAIL includes 29 performance measures (27 quality measures and two measures of overall efficiency and capacity) in areas such as acute-care mortality, access to care, and employee satisfaction. (See appendix I for the full list of SAIL measures.) SAIL is a diagnostic tool that allows VHA to assess medical centers’ performance relative to their peers, and determine how much absolute improvement they have made in the past year based on relevant clinical data. VHA publishes SAIL results quarterly to provide information to network and medical center officials regarding improvement opportunities at each medical center. SAIL data are also available on VHA’s intranet site. VHA staff can view a wide range of detailed information about their medical center, compare performance to other medical centers, and (for those staff with medical-record-level access) view information on patients with a particular medical condition. VHA conducts annual performance appraisals for all network and medical center officials. The appraisal process begins when officials from VHA’s office of Workforce Management and Consulting transmit a performance plan template to the network directors. The template identifies performance priorities and expectations for the upcoming appraisal period and criteria to be used to measure performance outcomes and ratings for each performance element. Network directors use the template to develop performance plans that include targets and time frames—the schedule of when performance targets are to be achieved during the year—with each of the medical center directors in their network. According to VA policy, performance plans resulting from the template should be finalized within 30 days of the start of the appraisal period. After expectations have been set for a medical center director, the director, in turn, sets performance expectations for the department heads within the medical center. VHA officials told us they primarily use the SAIL system to assess the performance of medical centers. Specifically, VHA uses SAIL data to calculate and assign each medical center an annual star rating of 1 (lowest) to 5 (highest) stars as an assessment of overall quality. SAIL documentation states that the goal of the star ratings is for low-performing medical centers to learn from the best practices of high-performing ones, although all medical centers have the opportunity to improve. VHA applies a weighting and calculation methodology to each of SAIL’s 27 quality measures to determine a single composite score for each medical center. The scores are then ranked and grouped by percentile and the associated medical centers are assigned initial star ratings based on their relative ranking. For example, the lowest performing 10 percent of medical centers as determined by SAIL’s 27 quality measures are assigned a 1-star rating, and the next lowest performing 20 percent of medical centers are assigned a 2-star rating. (See fig. 1.) After the initial star rating is determined by SAIL measures each year, VHA officials can make changes to the rating if a medical center meets certain conditions. For example, SAIL documentation states that a medical center that initially received a 5-star rating will be reduced to a 4- star rating if it has a high mortality rate. In addition, VHA officials told us they can decide to increase a 1-star medical center’s rating to a 2-star rating if the medical center outperforms the bottom 10 percent of U.S. hospitals in certain criteria as measured by external systems such as the Centers for Medicare & Medicaid Services’ Hospital Compare website. We found that the percentage of medical centers that received a final 1- star rating ranged from 4 percent to 10 percent from fiscal years 2013 through 2018. VHA officials publish the final annual star ratings for each medical center both internally and externally. See figure 2 for the number of medical centers that received each final star rating for fiscal years 2013 through 2018. Although the specific medical centers within each star-rating category could change from year to year, we found that the fiscal year 2018 star ratings for 110 of the 127 medical centers (87 percent) that received star ratings in fiscal year 2013 did not differ by more than 1 star from their fiscal year 2013 star rating. For example, eight of the 10 1-star medical centers in fiscal year 2013 received either a 1- or 2-star rating in fiscal year 2018. (See fig. 3.) In addition, 44 of the 127 medical centers had the same rating in fiscal year 2018 as they did in fiscal year 2013. At the end of the 6-year period of our review, only one medical center differed by more than 2 stars from its fiscal year 2013 star rating, decreasing from 5 stars to 2. VHA officials told us they use SAIL tools on VHA’s intranet when conducting site visits to medical centers and for other performance management efforts. The SAIL system includes several performance management tools that present data in greater detail than SAIL’s quarterly data release and enable officials to identify areas for improvement. VHA, network, and medical center officials we interviewed mentioned three in particular: Opportunity matrix – This matrix shows how a medical center ranks compared to others on all SAIL performance measures based on quarterly data. Each performance measure is labeled by quintile, with the first quintile comprising the top 20 percent of medical centers and the fifth quintile comprising the bottom 20 percent. Officials told us they use this tool to focus improvement efforts by examining specific measures for which a medical center needs improvement. Geometric control charts – These charts, referred to as G-Charts, allow officials to monitor on a daily basis what VHA considers to be rare occurrences. For example, one G-Chart allows VHA to monitor patient safety indicators that contain information on occurrences of specific medical conditions, such as cardiac arrest, pneumonia, and sepsis. Medical center officials can use these charts to examine the occurrence of events over time, analyze patient-level data, and quickly detect changes in the frequency of these events. Other events that the charts allow VHA to monitor include inpatient complications and deaths. Symphony action triggers – Symphony is an online tool that tracks over 100 performance measures daily, related to medical center access, outcomes, and productivity, and includes an early warning system to notify network and medical center officials of results that require attention. Officials can use Symphony to view patient-level information to understand the details of particular events and determine solutions. VHA officials also told us that they use these tools to manage medical center performance as part of their ongoing support of lower performing medical centers. Specifically, officials who oversee SAIL identify lower performing medical centers using SAIL and conduct site visits as part of VHA’s Strategic Action for Transformation initiative. This initiative utilizes a four-tiered, escalating approach based on the severity of concern at a medical center. In order of increasing severity, the four levels are watch, high-risk, critical, and VA receivership. One-star medical centers are automatically placed on the high-risk list, along with some 2-star medical centers with decreasing performance. If performance continues to decrease, medical centers are considered critical, and can be escalated to VA “receivership,” at which point VHA officials may step in to correct ongoing problems and replace network or medical center leadership officials. As of January 2019, VHA officials told us no medical center had entered VA receivership since the initiative began. VHA officials told us that they may also conduct site visits or hold calls with medical center leadership by request, although their focus is on lower performing medical centers. In addition to the SAIL tools, which report data on performance measures across the entire medical center, VHA officials told us that they may also use other data sources as part of medical center performance management. For example, several program offices—such as primary care, mental health, and surgery—have dashboards that track performance and quality of care specific to those offices. In addition, VA’s Inpatient Evaluation Center focuses on mortality data, including estimates of expected patient mortality. We found that VHA relies heavily on medical center performance information to assess the performance of its network and medical center directors. VA’s Senior Executive Service Part V. Performance Appraisal System handbook states that directors are assessed using five appraisal elements established by the Office of Personnel Management: (1) Results Driven, (2) Leading People, (3) Leading Change, (4) Business Acumen, and (5) Building Coalitions. The five elements are included in VHA’s performance plan template, which forms the basis for network and medical center directors’ performance plans. The handbook designates a relative weight for each element used to calculate a director’s rating. (See fig. 4.) The handbook states that a director is rated in each element on a scale of level 1 to level 5, with 5 being the highest level. Each rating is then multiplied by the weight for its corresponding element, and the results are added to generate a summary score. According to the handbook, the summary score is used to identify potential recipients of pay increases and monetary awards. The most heavily weighted appraisal element in the handbook, Results Driven, represents 40 percent of a director’s overall performance and is based entirely on medical center performance information. Specifically, for fiscal year 2018, SAIL results comprised 25 percent of the overall rating and included measures such as patient mortality, length of stay, and readmissions. Other medical center performance information comprised the remaining 15 percent of the overall rating. (See fig. 5.) Medical center performance information is also used when assessing directors’ performance across other appraisal elements. For example, in VHA’s fiscal year 2018 performance plan template, the Leading Change appraisal element included the implementation of suicide prevention initiatives, using medical center performance in the SAIL mental health domain as criteria. In addition, the Leading People element included performance information from VA’s All Employees’ Survey, which included medical center staff. Although medical center performance information plays a prominent role in the performance assessment process, VHA officials told us that there are other considerations that may result in medical center directors receiving a rating that is higher than that indicated by the star rating of the medical center. For example, VHA officials told us that when calculating a medical center director’s rating for the Results Driven element, they consider whether the medical center’s overall performance improved or deteriorated compared to the previous year’s performance. These officials also stated that they take into consideration the length of a director’s tenure, such as cases where a director started at a low- performing medical center partway through the rating year and would not have had sufficient time to improve the medical center’s performance from the previous year. In our review, we also found that the release of VHA’s performance plan template is often delayed, which can limit its effectiveness as a means of assessing directors’ performance. Specifically, in fiscal years 2016, 2017, and 2018, VHA released the performance plan template to network directors in November or December, close to the end of the first quarter of the performance appraisal period. Directors at two of the medical centers in our review expressed frustration with the delay and not having a full year to meet performance expectations, but directors at the two other medical centers stated that they find the process clear and are able to anticipate performance expectations. Officials from VHA’s office of Workforce Management and Consulting, which sends out the template, told us that they have been working in recent years to shorten the template’s development and review process within VHA; however, the delays may continue because of late changes from VA or the Office of Personnel Management. In our December 2016 review of human resource management practices at VHA, we also reported on delays in the release of VHA’s performance plan template. We reported that the delay limited medical center officials’ ability to use the template as a tool to align expectations and performance, which is inconsistent with leading practices on employee performance management. We recommended that VHA accelerate its efforts to develop a modern, credible, and effective performance management system, including the timely release of the performance plan template. VA partially concurred with our recommendation and has made limited progress in implementing it. As of December 2018, this recommendation remains open and we reiterate the need for VHA to implement it. Although SAIL is used in the assessment of both medical centers’ and directors’ performance, VHA officials have not assessed and implemented as appropriate the recommendations from previous evaluations of the SAIL system to ensure its effectiveness. This is inconsistent with federal standards for internal control, which state that management should remediate identified internal control deficiencies on a timely basis. This remediation may include assessing the results of reviews to determine appropriate actions, and, once decisions are made, completing and documenting corrective actions on a timely basis. VHA officials told us that since it was established in 2012, there have been two evaluations of SAIL: The first evaluation was an internal review, which VHA officials told us was completed in February 2014 and submitted to the director of VHA’s Office of Analytics and Business Intelligence and reviewed by the then Under Secretary for Health and Principle Deputy Under Secretary for Health. The internal review, which had 22 recommendations, found issues related to the validity and reliability of SAIL as a tool for measuring performance and fostering accountability. For example, it included a recommendation that VHA no longer use aggregate star ratings for accountability, or for presenting medical center quality and efficiency information to stakeholders. Rather, the recommendation called for VHA to work to identify valid and reliable approaches for presenting this information. The second evaluation was an external review, which VHA officials told us was submitted to the Office of the Principal Deputy Under Secretary for Health in April 2015. The external review included 19 recommendations for short- and long-term improvements to SAIL, such as a recommendation to examine the potential for misclassifying medical centers—i.e., assigning star ratings that do not reflect medical centers’ pattern of performance on the underlying measures. The review noted two ways such misclassification could occur: (1) two medical centers with summary scores that are close together could receive different star ratings, or (2) two medical centers with widely different summary scores could receive the same star rating. The findings of the previous SAIL evaluations are similar to concerns that officials from the four networks and four medical centers in our review expressed about SAIL’s effectiveness, including whether the star ratings were an accurate reflection of medical center performance. For example, officials from one medical center told us that, because the mortality measure has a higher weight relative to other SAIL measures, it can amplify the importance of a small difference between medical centers. As a result, a 1-star medical center may appear to be performing much more poorly on this measure than it is in practice; and officials from two medical centers told us that the length-of-stay measure may not be an accurate reflection of quality of care, as there are valid clinical reasons why some veterans need a longer length of stay that may not be reflected in the diagnostic and procedure codes for that stay. Therefore, the difference in performance on the length of stay measure between two medical centers may be the result of how data were entered into the medical record and coded, rather than actual differences in quality of care. In addition, VHA officials also expressed concerns about SAIL and how it is currently being used to assess medical center performance. For example, VHA officials who oversee SAIL told us it was designed to be an internal performance improvement tool, but is now also being used as a performance accountability tool. The external review included a recommendation that VHA consider whether the primary purpose of SAIL is improvement or accountability, as SAIL would need to be redesigned to do both. One VHA official told us that SAIL is being used in punitive ways through the Strategic Action for Transformation initiative. For example, at one medical center, officials told us that they received a letter from VHA’s Executive in Charge about the medical center’s low performance only a few months after its star level increased from 1 to 2 stars. Officials said the letter warned them that medical center leadership could be removed if performance does not improve. Medical center officials described this as counterproductive for their improvement efforts, as it was demoralizing while not identifying any specific areas for improvement. VHA officials confirmed that, other than their routine reviews to determine the need for annual adjustments to SAIL measures and other minor adjustments to the system, they have not assessed or implemented as appropriate the recommendations from the internal and external SAIL evaluations. In addition, although the Under Secretary for Health received a response to the internal review’s recommendations from an individual program office, VHA officials told us no action was taken on the response or to formally assess the recommendations from the internal review. Officials noted that two reasons for the lack of action taken to assess recommendations for implementation were leadership turnover and attention diverted to other issues, such as concerns about extended wait times for medical appointments at VHA medical facilities. In addition, officials stated that the evaluations were not widely distributed within VHA. As a result, officials we spoke with from several VHA offices were unaware that SAIL had ever been evaluated. To address the federal internal control standard for timely remediation of identified deficiencies, federal agencies assign responsibility and authority for carrying out and documenting corrective actions. VHA officials told us they did not formally assign responsibility to an office to assess recommendations from previous evaluations of SAIL. As a result, when the officials who received both evaluations left VHA, there were no other individuals or offices responsible for ensuring that recommendations were acted on. VHA officials who oversee SAIL told us that they are planning to use the 2015 external review as part of their plans to make changes to SAIL and its measures. However, there is no documentation available describing the planned changes to SAIL or how those planned changes will incorporate the results of the external review. If changes made to SAIL run counter to the evidence, it could potentially diminish the integrity of the system to effectively evaluate performance. VHA primarily uses the SAIL system to assess and compare the performance of medical centers. Veterans can also view SAIL data to compare medical center performance when making health care decisions. Officials from the networks and medical centers in our review expressed concerns about how SAIL is being used and whether star ratings are an accurate reflection of medical center quality. SAIL has been evaluated twice, and both evaluations have found similar concerns with SAIL. However, VHA has yet to use the results of those evaluations to address identified concerns and make evidence-based improvements to the SAIL system. Specifically, VHA has not taken action to ensure that officials assess the recommendations from SAIL evaluations, document their decisions, and implement recommendations as appropriate. If changes to SAIL are implemented without this assessment of existing evaluations, VHA may make changes that run counter to the evidence, potentially diminishing the integrity of the system to effectively evaluate performance. We are making the following two recommendations to VA: The Under Secretary for Health should assess recommendations from two previous evaluations of SAIL. This assessment should include the documentation of decisions about which recommendations to implement and assignment of officials or offices as responsible for implementing them. (Recommendation 1) The Under Secretary for Health should implement, as appropriate, recommendations resulting from the assessment of the two previous SAIL evaluations. (Recommendation 2) We provided VA with a draft of this report for review and comment. VA provided written comments, which are reprinted in appendix II. In its written comments, VA concurred with both of the report’s recommendations, and identified actions it is taking to implement them. We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, the Under Secretary for Health, and other interested parties. In addition, the report is available at no charge on the GAO web site at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at draperd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. 2 Risk adjusted complication Index In‐Hospital risk adjusted mortality (SMR) Measure 30‐day risk adjusted mortality (SMR30) Desired Direction of Measure Lower 3 Severity adjusted average length of stay (ALOS) %Acute admission reviews met InterQual criteria %Acute continued stay reviews met InterQual criteria Inpatient core measures mean percentage (ORYX) HEDIS outpatient core measure mean percentage (chart abstract) HEDIS outpatient core measure mean percentage (population based) 5 HCAHPS score (patient rating of hospital) Rating of primary care provider Rating of specialty care provider Care Transition (inpatient) Stress discussed (PCMH) 2 Best Places to Work score Hospital-wide all conditions 30-day readmission rate 5 Timely Appointment, Care and Information – PCMH Timely Appointment, Care and Information – SC Same Day Appointment When Needed – PCMH Call center speed in picking up calls 3 Mental health population coverage Mental health continuity of care Mental health experience of care 2 Stochastic frontier analysis (= 1/SFA) The acronyms VHA used in the table are as follows: SMR=standard mortality ratio; HEDIS=Healthcare Effectiveness Data and Information Set; HCAHPS= Hospital Consumer Assessment of Healthcare Providers and Systems; PCMH=patient-centered medical home; ACSC=ambulatory care sensitive conditions; SC=specialty care. Debra A. Draper, (202) 512-7114 or draperd@gao.gov. In addition to the contact named above, Janina Austin, Assistant Director; Sarah Harvey and Malissa G. Winograd, Analysts-in-Charge; Jennie F. Apter; Frederick Caison; and Alexander Cattran made key contributions to this report. Also contributing were Vikki Porter and Jennifer Whitworth. VA Health Care: Actions Needed to Improve Oversight of Community- Based Outpatient Clinics. GAO-18-375. Washington, D.C.: Apr. 12, 2018. Veterans Health Care: Additional Actions Could Further Improve Policy Management. GAO-17-748. Washington, D.C.: Sept. 22, 2017. Veterans Affairs: Improved Management Processes Are Necessary for IT Systems That Better Support Health Care. GAO-17-384. Washington, D.C.: June 21, 2017. VA Health Care: Improvements Needed in Data and Monitoring of Clinical Productivity and Efficiency. GAO-17-480. Washington, D.C.: May 24, 2017. Veterans Health Administration: Management Attention Is Needed to Address Systemic, Long-standing Human Capital Challenges. GAO-17-30. Washington D.C.: Dec. 23, 2016. Veterans Health Care: Improvements Needed in Operationalizing Strategic Goals and Objectives. GAO-17-50. Washington D.C.: Oct. 21, 2016. VA Health Care: Processes to Evaluate, Implement, and Monitor Organizational Structure Changes Needed. GAO-16-803. Washington D.C.: Sept. 27, 2016.", "summary": "VHA anticipates that it will provide care to more than 7 million veterans in fiscal year 2019. The majority of veterans using VHA health care services receive care in one or more of the 172 medical centers or at associated outpatient facilities. VHA collects an extensive amount of data that can be used to assess and manage the performance of medical centers. Many measures are publicly reported on VA web pages, allowing veterans the ability to compare medical centers' quality of care. GAO was asked to assess VHA's management of medical center performance. This report examines (1) the tools VHA uses to assess medical center performance; (2) VHA's use of medical center performance information to assess medical center directors; and (3) the extent to which VHA has evaluated the effectiveness of the SAIL system. GAO reviewed VHA policies, guidance, and performance information for medical centers and their associated directors. GAO also interviewed officials from VHA as well as from four VA medical centers, selected for variation in performance and geographic location. Department of Veterans Affairs' (VA) Veterans Health Administration (VHA) officials told GAO they primarily use the Strategic Analytics for Improvement and Learning (SAIL) system to assess VA medical center performance. SAIL includes 27 quality measures in areas such as acute care mortality and access to care. VHA officials use SAIL to calculate and assign each medical center an annual star rating of 1 (lowest) to 5 (highest) stars as an assessment of overall quality. For the 146 medical centers that received star ratings in fiscal year 2018, the distribution of star ratings was as follows: 6 percent, 1 star; 24 percent, 2 stars; 38 percent, 3 stars; 19 percent, 4 stars; and 12 percent, 5 stars. Although the specific medical centers within each star-rating category could change from year to year, GAO found that the fiscal year 2018 star ratings for 110 of the 127 medical centers (87 percent) that received star ratings in fiscal year 2013 did not differ by more than 1 star from their fiscal year 2013 rating. Changes in VHA Strategic Analytics for Improvement and Learning Star Ratings, Fiscal Year 2013 Compared to Fiscal Year 2018 GAO found that VHA's appraisal process for assessing medical center director performance relies heavily on medical center performance information, including SAIL. For example, the most heavily weighted appraisal element (40 percent of the overall rating) is made up entirely of medical center performance information. SAIL was evaluated in 2014 and 2015, but VHA has not assessed the recommendations from those evaluations, or taken action on them. The evaluations, which found issues related to the validity and reliability of SAIL and its star ratings for measuring performance and fostering accountability, together included more than 40 recommendations for improving SAIL. The findings are similar to concerns expressed by officials GAO interviewed from VHA, networks, and medical centers about SAIL's effectiveness and how it is currently being used to assess medical center performance. VHA officials told GAO the findings and recommendations of the previous SAIL evaluations were not assessed because the evaluation reports were not widely distributed within VHA due to leadership turnover, as well as attention that was diverted to other concerns such as extensive wait times for medical appointments. Without ensuring that the recommendations resulting from these previous evaluations are assessed and implemented as appropriate, the identified deficiencies may not be adequately resolved, and VHA's ability to hold officials accountable for taking the necessary actions may be diminished. GAO recommends that the Under Secretary for Health: (1) assess recommendations from previous evaluations of SAIL for implementation; and (2) implement, as appropriate, recommendations resulting from the assessment. VA concurred with GAO's recommendations and identified actions it is taking to implement them.", "document_type": "gao"}
{"report": "According to OMB guidance, evidence can consist of quantitative or qualitative information and may be derived from a variety of sources. Those sources include foundational fact-finding (e.g., aggregate indicators, exploratory studies, descriptive statistics, and other research), performance measurement, policy analysis, and program evaluation. OMB recommends that agencies build a portfolio of high-quality, credible sources of evidence—rather than a single source—to support decision- making. Further, since different sources of evidence have varying degrees of credibility, the use of evidence in decision-making requires an understanding of what conclusions can—and cannot—be drawn from the information. Evidence-building can be viewed as a cycle of activities that can help decision makers obtain the evidence they need to address policy questions or identify the questions they should address. As illustrated in figure 1, the following four activities comprise the evidence-building cycle: assessing existing evidence to determine its sufficiency and if additional evidence is needed to further understand results and inform decision-making; prioritizing among the identified needs which new evidence to generate, when, and how; generating new evidence, by collecting, analyzing, and synthesizing sources of data and research results; and using that evidence to support learning and decision-making processes. Our prior work highlights long-standing challenges agencies continue to face in generating some sources of evidence—developing performance measures for federal programs and conducting evaluations of their programs. Our work also identified variations in the use of evidence for decision-making by agency leaders and managers. Fragmentation refers to those circumstances in which more than one federal agency (or organization within an agency) is involved in the same activity and opportunities exist to improve implementation of that activity. The Commission on Evidence-Based Policymaking found that evidence- building activities are fragmented in the federal government. For example, it found that within agencies, many organizations have evidence-building responsibilities, including statistical agencies and programs, evaluation and policy research offices, performance management offices, policy analysis offices, and program administrators. In addition, the commission highlighted challenges the federal government faces in fully addressing cross-cutting research and policy questions when evidence-building activities span multiple agencies. The commission’s final report noted that this fragmentation (see sidebar) can lead to duplication of effort or missed opportunities for collaboration. The commission’s report stated that when activities are fragmented within an agency or across the federal government, they should be coordinated to improve the capacity to fully address a specific research or policy question. Similarly, our past work highlights the importance of coordination and collaboration to reduce or better manage fragmentation, overlap, and duplication. We found that uncoordinated or poorly coordinated efforts can waste scarce funds and limit their effectiveness. Even when efforts are coordinated, enhancements to those efforts can lead to improvements in effectiveness. As noted earlier, our work also identified leading practices that can help agencies enhance and sustain their implementation of collaborative efforts. Congress and OMB have taken actions to strengthen federal evidence- building activities and improve coordination of those activities during the last decade. Figure 2 provides a timeline of selected actions. Appendix II provides additional detail regarding the selected actions. Tiered evidence grants seek to incorporate evidence of effectiveness into grant making. Federal agencies establish tiers of grant funding based on the level of evidence grantees provide on their approaches to deliver social, educational, health, or other services. Grantees generally are required to evaluate their service models as a condition for the receipt of grant funds. spent on home visiting models with sufficient evidence of their effectiveness. To support this requirement, the program incorporated activities across each element of the evidence-building cycle. For example, through its Home Visiting Evidence of Effectiveness review, HHS annually assesses existing evidence about the effectiveness of new and existing home visiting models to identify those that meet criteria for inclusion in the program. The most recent review, in October 2018, identified 20 models that met HHS’s criteria for an evidence-based early childhood home visiting model. Of those, HHS determined that 18 models were eligible for MIECHV grantees to select for implementation. In addition, based on statutory requirements, officials prioritized the generation of new evidence to assess the program’s results in certain areas, including child health and development, and child maltreatment. The program generated this evidence through program evaluations assessing both program implementation and results. For example, an impact evaluation of four home visiting models published in January 2019 found that these models may reduce household aggression. Because child abuse has been shown to be associated with negative long-term outcomes, reducing household aggression could benefit children as they grow older. In another example of the use of tiered evidence, the Department of Labor’s (DOL) Workforce Innovation Fund, established in 2011, intends to generate long-term improvements in the performance of the public workforce system. The fund established and funded projects in three different tiers: 1. those that proposed new and untested approaches, with little or no evidence of effectiveness; 2. those with promising approaches that were tested and existing evidence suggested could be effective; and 3. those that adapted proven approaches, supported by ample and robust evidence. To further build DOL’s base of evidence on the effectiveness of evidence- based approaches, it required grantees to plan for third-party evaluations of their programs. During the first grant round in 2012, the Workforce Innovation Fund awarded 26 grants, including one for approximately $1.4 million in tier one funding to the Pasco-Hernando Workforce Board in Florida. This grant supported making one-stop services, such as employment workshops and workforce program orientations, more accessible to job seekers by providing online access. In addition, the grant supported offering virtual case management and business services through a call-in Employment Support Center to individuals who found it difficult to access these services in person. According to a 2016 case study of this project conducted by DOL, users of the online one-stop accessed services nearly twice as much during this 3-year grant period when compared to the prior 3-year period. In addition, the case study found there was a 53 percent increase in job placements during this 3-year grant period. The selected agencies’ evidence-building activities also aligned with implementation actions outlined by OMB for selected cross-agency priority (CAP) goals. As required by the GPRA Modernization Act of 2010, at least every 4 years, OMB is to coordinate with other agencies to develop and implement CAP goals. Two current CAP goals, established in March 2018 in the President’s Management Agenda, place a particular focus on evidence-building activities. Leveraging data as a strategic asset. OMB and agency efforts to implement this goal included developing a long-term, enterprise-wide federal data strategy to better govern and leverage the federal government’s data. Published in June 2019, this strategy established 10 principles and 40 practices intended to leverage the value of federal data assets while protecting security, privacy, and confidentiality. Officials at each of the five selected agencies described actions taken by their agencies that aligned with the federal data strategy’s principles and practices. Federal Evidence Clearinghouses According to the Office of Management and Budget (OMB), evidence or “what works\" clearinghouses are repositories that synthesize evaluation findings in ways that make research more useful to decision makers, researchers, and service organizations. These repositories provide tools for understanding what service models are ready for replication or expansion and disseminating results. grade. Officials told us in September 2018 that preliminary evidence suggested the model could help close the literacy gap for the target population. In addition, officials told us they intended to disseminate the final results to stakeholders to help inform their decision-making about the approach. To do so, Education officials developed a communication plan to share this evidence via the OELA website, its Facebook account, the National Clearinghouse for English Language Acquisition (see sidebar), and a listserv of more than 10,000 recipients, among other means. As of September 2019, this study had not been completed. Therefore Education has not implemented its communication plan. Results-oriented accountability for grants. One of the four strategies for this CAP goal focuses on the achievement of grant program goals and objectives. In October 2019, OMB staff told us that the strategy aims to hold grant recipients accountable for promising performance practices that support the achievement of those goals and objectives while streamlining compliance requirements for those grant programs that demonstrate results. According to the September 2019 quarterly update for this goal, initial efforts for this strategy involved developing performance management processes to help grant-making entities improve their ability to monitor, and ultimately improve, the performance of grantees. The update stated that OMB and the Chief Financial Officers Council completed efforts in fiscal year 2019 that included soliciting information from agencies on their current grants performance management practices and identifying emerging and innovative performance practices. Subsequent efforts for this goal involved hosting monthly grants practitioner sessions (called Innovation Exchange Sessions) to share new ideas and approaches to grants management, which began in May 2019. The September 2019 session focused on data- driven decision-making for grants. We identified actions that each of the selected agencies took, aligned with the intent of this CAP goal, to better assess the performance of their grant programs. Officials at each agency told us that they took steps to further incorporate evidence-building requirements into their grant programs. They told us they did this based in part on their experiences in implementing the evidence-based approaches, such as the tiered evidence grants described earlier in this report. For example, officials at the Corporation for National and Community Service (CNCS) described their incorporation of evidence-building requirements into the agency’s AmeriCorps State and National program. Agency officials told us that grantees have been required to evaluate their programs since 2005. In recent years, CNCS embedded the evidence generated by these evaluations into their grant-making activities. For instance, its grant announcement for 2019 stated that AmeriCorps State and National applications would be scored, in part, based on the reported empirical evidence supporting the applicants’ proposed projects. In addition, the announcement required applicants proposing projects in the education focus area to choose one of 13 models that had previously demonstrated effectiveness. According to CNCS officials, this was based on evidence generated in previous projects supported by AmeriCorps State and National grants or CNCS’s Social Innovation Fund. Although the Evidence Act’s requirements apply to the agency-wide level, OMB’s guidance strongly encourages lower-level organizations within agencies to develop and implement their own learning agendas (see side bar). We found instances where officials developed learning agendas at lower organizational levels within several of the selected agencies prior to the issuance of the June 2019 OMB guidance. These learning agendas covered individual component agencies, bureaus, offices, and programs. Learning Agendas According to Office of Management and Budget (OMB) guidance for implementing the Evidence Act, a learning agenda is to define and prioritize relevant questions and identify strategies for building evidence to answer them. In developing a learning agenda, an agency should involve key leaders and stakeholders, to help (1) meet their evidence needs for decision-making and (2) coordinate evidence-building activities across the agency. For example, from September 2016 to June 2017, the U.S. Agency for International Development (USAID) conducted a landscape analysis of learning agendas, in which officials identified 15 documented, office-, bureau-, or initiative-wide learning agenda processes at different stages of development within USAID. This included an office-wide learning agenda developed by the Center of Excellence on Democracy, Human Rights, and Governance (DRG). According to USAID, DRG seeks to elevate and integrate democracy, human rights, and governance issues within USAID’s overall development portfolio. According to DRG’s 2017 learning agenda, its development was informed by ongoing DRG research and evaluation efforts, and consultations with a range of internal stakeholders, including USAID staff from other bureaus and missions. The learning agenda included a set of 11 questions across five thematic areas, as illustrated in figure 3. DRG outlined steps it planned to take throughout 2017 to address each question, such as assessing existing evidence, identifying any gaps, and conducting new research and evaluation activities to fill those gaps. For example, DRG commissioned a study to help answer a question about the effects of human rights awareness campaigns. The study, published in September 2017, synthesized the results of a literature review to identify (1) characteristics of effective campaigns, and (2) typical causes of unintended negative consequences of human rights awareness campaigns and ways to avoid them. We found that evidence-building activities are fragmented within each of the five selected agencies and occur at multiple levels and entities within and across the agencies. As illustrated in figure 4, this fragmented approach to evidence-building includes separate component agencies or offices with responsibilities for building specific sources of evidence, such as performance information, evaluations, and statistical data. For example, at the Department of Labor (DOL), different organizations at the department level are responsible for certain evidence-building activities. This includes the Bureau of Labor Statistics (collecting statistical data), Office of the Chief Evaluation Officer (conducting program evaluations) and Performance Management Center (developing performance information). In addition, some evidence-building activities are dispersed throughout agencies and occur at multiple organizational levels (see figure 5). For example, at the Department of Health and Human Services (HHS), evidence-building activities are generally managed at the component agency level (referred to as divisions). The divisions manage their own offices and programs, which include evidence-building responsibilities. For instance, within the Administration for Children and Families (ACF), an operating division within HHS—the Office of Planning, Research, and Evaluation—is responsible for ACF-related evidence-building activities. These activities include program evaluations, research syntheses, descriptive and exploratory studies, data analyses, and performance management activities. Officials at the selected agencies said that evidence-building activities are fragmented and occur at lower levels for a variety of reasons. First, this approach helps ensure that decision makers at different levels within the organization have the evidence they need to inform decisions. Second, officials stated that many times these evidence-building activities have been undertaken in response to direction from Congress—for example, through provisions in laws or related committee reports directed at a component agency or program. Third, agency officials said they have undertaken these activities based on OMB direction, such as memorandums or budget guidance. This has encouraged agencies to take actions at different organizational levels. However, each of the selected agencies had established processes for coordinating their evidence-building activities. For example, officials at each agency established one or more processes intended to regularly coordinate the assessment and prioritization of evidence needs across the agency, as described later in this report. Agency officials also described other efforts to coordinate evidence- building activities, but these efforts were either ad hoc (i.e., they did not occur regularly) or not comprehensive in nature (i.e., they did not focus broadly across different sources of evidence or did not cover the entire agency). For example, in August 2017, the Corporation for National and Community Service (CNCS) published the results of an assessment of existing evidence—results from research and evaluation activities conducted between fiscal years 2015 and 2016—in its State of the Evidence report. However, CNCS has not conducted a similar analysis or issued a similar report since that time. Moreover, the assessment did not cover all of the agency’s activities. While the report included evidence related to its programs, CNCS did not assess evidence related to other activities, such as internal management functions including information technology or human capital management. We identified instances in which effective coordination helped selected agencies better manage their fragmented evidence-building activities. For example, the U.S. Agency for International Development (USAID) developed an agency-wide Private Sector Engagement learning agenda, published in May 2019. This learning agenda is intended to guide and coordinate crosscutting efforts to develop evidence of effective approaches for engaging the private sector to help partner countries meet development goals and ultimately move beyond the need for foreign assistance. This learning agenda includes establishing performance measures to monitor progress on engagement with the private sector, and further evaluate the results of its activities. The coordinated evidence- building approach established by this learning agenda can help USAID better focus limited resources on building new evidence in this crosscutting area for use across the agency, thereby reducing any unwarranted overlap or duplication of effort. Effectively-coordinated processes can help agencies ensure they are comprehensively and systematically looking across their organizations to leverage their existing evidence and focus limited resources on building new evidence. They can also help agencies manage their fragmented evidence-building activities to improve effectiveness and reduce the potential for any unwarranted overlapping or duplicative efforts. Such processes can help ensure agencies are well positioned to meet forthcoming Evidence Act requirements related to assessing and prioritizing evidence across the entire agency. Each of the five selected agencies established a similar approach for assessing existing evidence and identifying gaps or other evidence needs across the agency. Agency officials said that these approaches responded to OMB guidance for agencies to conduct annual strategic reviews. Specifically, in its guidance for implementing the GPRA Modernization Act of 2010, OMB established an annual process in which each agency is to review progress in achieving strategic objectives— goals that reflect the outcome or impact the agency is seeking to achieve—established in its strategic plan. According to OMB’s guidance, as a part of those reviews, the assessment of existing evidence should inform agency decisions about where to focus limited available resources to build new evidence to fulfill any identified needs. OMB’s guidance encourages agencies to leverage existing decision- making processes, such as the budget development process, to implement these reviews. Each of the five selected agencies conducts strategic reviews and associated evidence assessments in similar ways, through a variety of existing decision-making processes: CNCS and HHS use their budget formulation processes; Education incorporates strategic objective reviews into existing quarterly reviews of progress in meeting goals; DOL uses a stand-alone strategic review process; and USAID leverages an existing review process conducted at lower levels (i.e., its missions). Officials at selected agencies identified instances in which they used their agency strategic reviews to (1) assess a variety of existing sources of evidence—a portfolio of evidence—to determine progress toward a strategic objective, and (2) identify the need for additional evidence, as illustrated by the following examples. Assessing a portfolio of evidence. DOL’s guidance for its strategic review process directs its component agencies to assess a variety of evidence sources to determine results and risks or challenges that may affect future outcomes. This includes performance information, program evaluations, risk assessments, and findings from reports by us and the department’s Office of Inspector General (OIG), among other sources. In its fiscal year 2018 Annual Performance Report, DOL identified different sources of evidence to demonstrate the effectiveness of some of its programs, and challenges related to others, for its strategic objective to create customer-focused workforce solutions for American workers. For example, it cited statistics and performance data to provide context and some quantitative results related to this objective. It also shared the results from several program evaluations, including a 2017 impact evaluation that suggested DOL’s Adult and Dislocated Worker programs were effective at increasing participants’ earnings and employment. DOL’s performance report also highlighted that its OIG identified aspects of several programs that support this objective as Top Management and Performance Challenges for Fiscal Year 2018. One of those challenges related to maintaining the integrity of Foreign Labor Certification Programs. DOL’s performance report stated that balancing the quality review of applications with employers’ needs for timely processing has been a challenge for years. Based on the totality of evidence, DOL identified this strategic objective as a focus area for improvement for fiscal year 2018. Identifying evidence needs. In its Strategic Plan for Fiscal Years 2018-22, Education established a strategic objective to increase high- quality education options and empower students and parents to choose an option that meets their needs. To implement this strategic objective, the strategic plan states that the department will encourage state and local education agencies to expand school choice by administering programs that increase education options, such as the Charter Schools Program (CSP). One of the performance measures Education uses to assess the program and progress on this strategic objective is the aggregate number of charter schools that are open, operating, and supported by CSP. Education officials told us that they identified limitations with this measure through the department’s strategic review process, and the need for additional evidence. As an aggregate count, the measure did not allow the department to accurately identify underlying changes in individual charter schools served by the program or the results and activities of CSP. For example, Education officials set a goal to increase the number of CSP-supported charter schools by 150 for the 2017-2018 school year. However, Education reported a decrease of four charter schools for this time period. To better understand CSP’s performance, Education officials told us they needed additional evidence to assess other aspects of the program’s performance. Education officials identified additional sources of evidence within the department that they could use to understand the program’s performance. These included statistics from Education’s National Center for Education Statistics (NCES) on the total number of charter schools that opened and closed over the same time period, and annual performance reports from grantees. According to information on Performance.gov, these additional sources of information showed that, in the 2017-2018 school year, 134 new charter schools supported by CSP opened, and 101 charter schools expanded under a CSP grant. These actions illustrate an instance of effective coordination of evidence- building activities to manage fragmentation and reduce the risk of duplication. Education officials looked across the agency and leveraged existing evidence generated by different organizational units—CSP and NCES—to better understand program performance. Had this not occurred, CSP might have collected data that duplicated what was already generated by NCES. Agencies’ assessments of the sufficiency of their existing evidence— conducted via processes for their strategic reviews—reflect the four leading collaboration practices. Although OMB’s guidance provides flexibility in how the reviews are conducted, it also sets specific expectations for who should lead the process, who should participate in the process, and the types of roles and responsibilities for these individuals. Table 1 provides illustrative examples of the selected agencies’ evidence assessment processes that reflect leading practices for collaboration. Unlike the similar processes they use for assessing existing evidence and identifying needs, the five selected agencies use a variety of processes to prioritize new evidence to generate. Agency officials told us that much of this prioritization takes place at lower organizational levels. For example, at HHS, the department’s component agencies—11 operating divisions and 14 staff divisions—generally lead their own evidence-building processes, through which they prioritize which evidence to generate. Officials from HHS’s Office of the Assistant Secretary for Planning and Evaluation told us that this decentralized model is due to the size and complexity of the department, and that it respects the unique needs of the divisions. According to these officials, a 2017 review by this office found variation in the processes that the components use for this purpose. HHS officials said that most components prioritize their evidence needs through their budget formulation processes. Officials at each of the selected agencies identified one or more processes intended to coordinate the prioritization of evidence needs across the entire organization. Table 2 describes these processes. We identified instances in which officials used these processes to more effectively focus limited resources to build new evidence through coordination across the agency. For example, CNCS officials described an instance in which agency leadership used the agency’s budget formulation process to prioritize evidence-building activities to address knowledge gaps about the AmeriCorps National Civilian Community Corps (NCCC) program. According to CNCS officials, through the agency’s evidence assessment processes, they found that the agency did not have evidence to fully assess the impact of NCCC programs on members and communities. Moreover, existing evidence showed that NCCC had experienced a decline in the number of qualified applicants and the retention of its members since 2014. To better understand the performance and results of this program, CNCS officials told us that agency leadership approved funding in fiscal years 2018 and 2019 for NCCC to undertake a multi-year impact evaluation. This evaluation, which is being conducted in conjunction with CNCS’s Office of Research and Evaluation and an independent contractor, is expected to examine the member retention, leadership development, and community impact of NCCC programming. Officials at each of the selected agencies told us that they were considering how best to meet Evidence Act requirements to take a systematic and coordinated approach to prioritizing evidence-building activities, such as through learning agendas. For example, as described in table 3, Education created a new body in March 2019—the Evidence Leadership Group—to coordinate its evidence-building activities. Education officials told us that in establishing this new group, they took into consideration our leading practices for collaboration. As described in table 3, all five selected agencies identified one or more leadership models for their evidence prioritization processes. We found that all five of the selected agencies involved at least some relevant participants in their evidence prioritization processes, as summarized in table 4. Our past work related to evidence-building activities identified a wide range of relevant participants to involve. Within agencies, these participants include agency leadership, program staff, and those with functional management responsibilities including budget, human capital, and information technology. External stakeholders include Congress, other federal agencies, state and local governments, grant recipients, and regulated entities. The five selected agencies include a range of relevant internal participants, although the evidence prioritization process at CNCS does not always include key internal stakeholders. CNCS’s budget hearings involve discussions about prioritizing evidence, but primarily focus on budget formulation decisions. Therefore, agency leaders and budget officials are consistently involved in the hearings, but others, such as the Director of the Office of Research and Evaluation, are not. Involving all key internal stakeholders helps ensure that those involved in a collaborative effort can commit resources, make decisions, and share their knowledge, skills, and abilities. This can also help ensure that the evidence that will be subsequently generated will be useful to decision makers across the organization. Education and USAID established expectations to seek input from external stakeholders in their evidence prioritization processes. Education’s charter for its recently-established Evidence Leadership Group states that the group is to engage a wide array of external stakeholders in its work. Similarly, for the evidence prioritization activities that occur through USAID’s program cycle and learning agendas, related guidance sets expectations to involve or obtain the perspectives of external stakeholders. As USAID developed its Self-Reliance learning agenda, it sought input from external stakeholders including officials from other federal agencies, organizations that implement USAID programs, and experts in international development, among others. Three of the selected agencies, however, do not always have mechanisms in place to involve, or consider the evidence needs of, a range of external stakeholders in their evidence prioritization processes. Officials at CNCS, HHS, and DOL told us that, because they consider their prioritization processes to cover internal management purposes and decisions, including external stakeholders is not appropriate. Officials at these three agencies described ways in which they sought input on evidence needs from some stakeholders, such as from interactions with grant recipients and external researchers. However, these agencies have not developed an approach to collect and consider input on evidence needs from all relevant stakeholders to inform their prioritization processes. Our past work highlights the importance of engaging key external stakeholders, especially Congress, to better understand and meet their evidence needs. Engaging external stakeholders can also create a shared understanding of competing demands facing the agency and ensure that their efforts and resources are targeted at the highest priorities across the agency. Moreover, through this engagement, agencies may find that external stakeholders have, or are aware of, existing evidence that helps the agency meet its needs or provide a fuller picture of performance. Involving a full range of relevant stakeholders in the process for prioritizing new evidence to generate would help each of the selected agencies ensure it is meeting the evidence needs of decision makers within and external to the agency. Four of the selected agencies—Education, HHS, DOL, and USAID—fully define roles and responsibilities for those involved in their evidence prioritization processes, while the process at CNCS partially reflects this practice, as described in table 5. CNCS officials said that the primary focus of the agency’s process is budget formulation. Therefore, roles and responsibilities are generally related to that purpose instead of the evidence prioritization activities that also take place during that process. Clearly defining roles and responsibilities can ensure all participants are aware of and agree upon (1) who will have what responsibilities, (2) how they will organize their joint and individual evidence-building efforts, and (3) how they will make decisions. As described in table 6, Education and USAID’s processes reflect this practice, while those at CNCS, DOL, and HHS reflect it in part. Officials at CNCS, HHS, and DOL gave different reasons for why their written guidance and agreements related to evidence prioritization processes do not fully reflect this leading practice. CNCS’s and HHS’s written guidance primarily focuses on their budget formulation processes, since this is where their evidence prioritization activities take place. Thus, these guidance documents contain information on leadership, participants, and roles and responsibilities related to budget formulation activities, but not all relevant details related to evidence prioritization. Officials at DOL stated that they do not want to take a “one-size-fits- all” approach to developing learning agendas within the department. They told us they had not developed specific written guidance for that process to provide flexibility to component agencies to develop processes that work best for them in developing their learning agendas. As we have previously found, documenting a clear and compelling rationale to work together—and how that work will be done and by whom—is a key factor in successful collaboration. By incorporating this leading practice into their existing guidance, CNCS, HHS, and DOL would have greater assurance that they are effectively collaborating to prioritize evidence needs. Decision makers need evidence to help them address pressing governance challenges faced by the federal government. Agencies undertake a range of efforts at different organizational levels to build evidence to meet their own decision-making needs, as well as those of others, such as Congress. However, these evidence-building activities are fragmented within agencies. Through a more comprehensive and coordinated framework, Evidence Act implementation provides opportunities to improve the effectiveness of federal evidence-building activities. The five selected agencies have taken steps to improve the coordination of evidence-building activities across their organizations, with Education’s and USAID’s evidence-building activities reflecting the leading practices for collaboration. CNCS, DOL, and HHS would have greater assurance that they are comprehensively considering evidence needs across their individual organizations by fully incorporating leading collaboration practices into their agency-wide efforts to prioritize new evidence to generate. These actions could also help ensure these agencies are better managing fragmented evidence-building activities and more effectively focusing their limited resources to generate evidence to meet decision makers’ needs. In addition, improved coordination could reduce the potential for any unwarranted overlap and duplication in their efforts, and better position the agencies to meet the Evidence Act’s requirements and related implementation actions outlined in OMB’s guidance. We are making a total of seven recommendations, including three to CNCS, two to HHS, and two to DOL. Specifically: The Chief Executive Officer of CNCS should develop an approach to ensure that all relevant participants are involved in the agency-wide process for prioritizing evidence needs. (Recommendation 1) The Chief Executive Officer of CNCS should define roles and responsibilities for all relevant participants involved in the agency-wide process for prioritizing evidence needs. (Recommendation 2) The Chief Executive Officer of CNCS should revise written guidance for the agency-wide process for prioritizing evidence needs to ensure it identifies all relevant participants and their respective roles and responsibilities. (Recommendation 3) The Secretary of Health and Human Services should develop an approach to ensure that all relevant participants are involved in the department-wide process for prioritizing evidence needs. (Recommendation 4) The Secretary of Health and Human Services should revise written guidance for the department-wide process for prioritizing evidence needs to ensure it identifies all relevant participants and their respective roles and responsibilities. (Recommendation 5) The Secretary of Labor should develop an approach to ensure that all relevant participants are involved in the department-wide process for prioritizing evidence needs. (Recommendation 6) The Secretary of Labor should revise written guidance for the department-wide process for prioritizing evidence needs to ensure it identifies all relevant participants and their respective roles and responsibilities. (Recommendation 7) We provided a draft of this product for comment to OMB and the five selected agencies—CNCS, Education, HHS, DOL, and USAID. CNCS, Education, HHS, DOL and USAID provided written comments, which are summarized below and reproduced in appendixes V, VI, VII, VIII, and IX, respectively. In addition, CNCS, Education, HHS, USAID, and OMB provided technical comments, which we incorporated as appropriate. In its written comments, CNCS neither agreed nor disagreed with the three recommendations we directed to it. The agency stated that it believes the planned actions included in its Strategic Evidence Plan, published in September 2019, address those recommendations. The plan includes a goal to strengthen how the agency prioritizes and uses evidence, and outlines various actions intended to achieve that goal. The plan does not include sufficient details to enable us to assess the extent to which its implementation would fully address the issues identified in our review and covered by our recommendations. Education stated in its written comments that the department is committed to maximizing the performance of its programs, and it views building, using, and disseminating evidence as critical to those efforts. Education also outlined planned and proposed actions that it believes would further its evidence-building activities. In its written comments, HHS did not concur with the two recommendations we directed to it. In response to both recommendations, HHS stated that the department had developed an approach for including all relevant participants in its process for prioritizing evidence needs. However, according to an HHS official in November 2019, HHS had not yet finalized the approach, and therefore was unable to provide any additional information about it. Thus we could not assess the extent to which HHS’s stated actions would address our recommendations. DOL agreed with the two recommendations we directed to it, and in its written comments described an action it plans to take to address them. We will monitor DOL’s action, which we believe would likely address our recommendations, if effectively implemented. USAID, in its written comments, reiterated the agency’s commitment to a comprehensive and integrated approach for its evidence-building activities. In the draft of this report we sent to USAID for its review in October 2019, we included a recommendation to USAID that it ensure that all relevant participants are involved in agency-wide processes for prioritizing evidence needs. USAID subsequently provided documentation that it had not provided previously that showed the agency had taken various steps to seek the input of a range of external stakeholders. We determined that these actions addressed our draft recommendation. Thus, we removed the draft recommendation from our report. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget, the Chief Executive Officer of the Corporation for National and Community Service, the Secretary of the Department of Education, the Secretary of the Department of Health and Human Services, the Secretary of the Department of Labor, the Administrator of the U.S. Agency for International Development, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or sagerm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix X. This report responds to a request that we review the coordination of federal evidence-building activities. This report (1) describes activities selected agencies have taken that align with congressional and Office of Management and Budget (OMB) direction to strengthen evidence- building, and (2) examines the extent to which selected agencies’ processes for assessing and prioritizing evidence needs reflect leading practices for collaboration. To address both objectives, we analyzed agency documents about federal evidence-building activities and interviewed relevant staff at OMB and officials at five selected agencies: the Departments of Education, Health and Human Services, and Labor; the Corporation for National and Community Service; and the U.S. Agency for International Development. We selected these five agencies based on their experiences incorporating evidence-building activities into program design and implementation. These experiences include evidence-based approaches such as pay for success projects, performance partnerships, and tiered evidence grants. At the time we made our selection, these five agencies had designed or implemented evidence-based approaches to a greater extent than other agencies we identified. The agencies we selected vary in size—as measured by budget authority and employees—and organizational structure (see table 7). For the first objective, we reviewed information from the five selected agencies and identified examples of evidence-building activities within each agency since 2010. We then determined if these examples illustrated actions that aligned with evidence-building statutory requirements and directions from OMB including guidance, memorandums, and activities outlined in the President’s Management Agenda. To do so, we reviewed relevant laws and OMB guidance. For the second objective, we evaluated processes each selected agency had established to take a coordinated approach to assessing and prioritizing evidence needs across the agency. We compared these processes to four selected leading practices for collaboration identified in our prior work (see table 8). We selected these four collaboration practices because our past work on evidence-building activities, such as analysis of performance information and program evaluations, has similarly identified them as key approaches related to evidence-building. Table 9 illustrates this alignment for selected past reports. We conducted this performance audit from April 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Office of Management and Budget (OMB) has issued several memorandums and other key policy documents that encourage agencies to take actions to strengthen their capacity to build evidence. For example, in a July 2013 memorandum, OMB encouraged agencies to identify proposals for building evidence in their budget requests. Such proposals could be used to improve existing programs or inform decisions about new programs. The OMB guidance highlighted several evidence- based approaches for agencies to consider, including pay for success, performance partnerships, and tiered evidence grants, described further in the text box below. Examples of Evidence-Based Program Approaches Identified in Office of Management and Budget (OMB) Guidance Pay for success. Pay for success is a contracting mechanism under which final payment is contingent upon achieving specific outcomes. The government specifies performance outcomes in pay for success contracts and generally includes a requirement that contractors assess program outcomes or impacts through an independent evaluation. The evaluators may also generate and analyze performance data to inform program management and improvement during implementation. Performance partnerships. Performance partnerships allow federal agencies to provide grant recipients flexibility in how they use funding across two or more programs along with additional flexibilities. In exchange, the recipient commits to improve and assess progress toward agreed-upon outcomes by developing and using evidence. Tiered evidence grants. Tiered evidence grants seek to incorporate evidence of effectiveness into grant making. Federal agencies establish tiers of grant funding based on the level of evidence grantees provide on their approaches to deliver social, educational, health, or other services. The grant generally requires grantees to evaluate their service models as a condition for the receipt of grant funds. In addition, Congress passed laws aimed at strengthening and better coordinating evidence-building activities, which OMB reinforced through related guidance to implement those laws. GPRA Modernization Act (GPRAMA). GPRAMA established a framework aimed at taking a more crosscutting and integrated approach to improve government performance. Requirements included in that framework, such as cross-agency priority (CAP) goals and strategic reviews, were intended to strengthen evidence-building activities and improve coordination. CAP goals. At least every 4 years, OMB is to coordinate with other agencies to develop and implement CAP goals. These goals are to address issues in a limited number of policy areas requiring action across multiple agencies, or management improvements that are needed across the government. The President’s Management Agenda, released in March 2018, established the third set of CAP goals since GPRAMA was enacted. Implementation of each CAP goal can involve evidence-building activities; however, two goals in particular are to focus on them, as described further in the text box. Cross-Agency Priority (CAP) Goals Focused on Evidence-Building Leveraging data as a strategic asset. The President’s Management Agenda highlights several root causes for the challenges the federal government faces. One root cause is that agencies do not consistently apply data-driven decision-making practices. This agenda states that agencies need to make smarter use of data and evidence to orient decisions and accountability around service and results. The administration established this CAP goal to improve the use of data in decision- making to increase the federal government’s effectiveness. Results-oriented accountability for grants. According to the June 2019 update for this goal, the federal government uses grants to invest approximately $700 billion each year in mission-critical needs. However, the report states that grant managers report spending 40 percent of their time using antiquated processes to monitor compliance instead of analyzing data to improve results. The administration established this CAP goal to maximize the value of grant funding by applying a risk- based, data-driven framework that balances compliance requirements with demonstrating successful results. A strategic objective is a type of goal that reflects the outcome or impact the agency is seeking to achieve. The agency is to identify the strategies—the portfolio of organizations, regulations, tax expenditures, programs, policies, and other activities—within and external to the agency that contribute to each strategic objective. As a set, the agency’s strategic objectives are to encompass all of its activities. Strategic reviews. In its guidance for implementing GPRAMA, OMB established an annual process in which each agency is to review progress in achieving the strategic objectives established in its strategic plans (see sidebar). To do so, OMB’s guidance directs agencies to assess existing sources of evidence to understand the progress made toward each strategic objective and identify where additional evidence is needed to determine effectiveness. In addition, OMB’s guidance states that another purpose of strategic reviews is to strengthen collaboration. It notes that the reviews can do so by identifying and addressing crosscutting challenges and fragmentation. The Foreign Aid Transparency and Accountability Act of 2016 (FATAA). Among other things, FATAA requires the President to establish guidelines for establishing measurable goals, performance metrics, and monitoring and evaluation plans for federal foreign assistance. In January 2018, OMB issued guidelines for federal agencies that administer foreign assistance—which includes the Departments of Labor and Health and Human Services, and the U.S. Agency for International Development. Among other things, the guidelines provide direction on strengthening evidence-building activities, such as establishing annual monitoring and evaluation plans, and disseminating findings and lessons learned. Agencies were directed to align their monitoring and evaluation policies with the guidelines by January 2019. The Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act). In June and July 2019, OMB released its initial guidance on implementing the Evidence Act. Among other things, this guidance provides direction to agencies on developing evidence-building plans, also known as learning agendas (see text box below). According to OMB, these plans will serve as the driving force for other evidence- building activities required by the Evidence Act. Prior to the enactment of the Foundations for Evidence-Based Policymaking Act (Evidence Act), both the Office of Management and Budget (OMB) and the Commission on Evidence-Based Policymaking highlighted and recommended the use of learning agendas by federal agencies to strengthen and coordinate their evidence- building activities. According to OMB’s guidance for implementing the Evidence Act, a learning agenda is to define and prioritize relevant questions and identify strategies for building evidence to answer them. A federal agency developing a learning agenda should involve key leaders and stakeholders to help (1) meet their evidence needs for decision-making, and (2) coordinate evidence-building activities across an agency. OMB’s guidance stated that the Evidence Act emphasizes the need for collaboration and coordination of agency staff and activities to achieve successful implementation. The guidance provides time frames for a phased approach to implement several Evidence Act requirements. For example, although learning agendas are not required to be published until February 2022, OMB’s guidance includes several interim milestones and deliverables to build toward the final published version. 5 U.S.C. §§ 306, 312. OMB embedded portions of Evidence Act implementation guidance in its 2019 update to Cir. No. A-11. In it, OMB noted that many of the Evidence Act’s provisions support the Federal Performance Framework for Improving Program and Service Delivery (Part 6 of the Circular), which provides guidance for implementing GPRAMA and other related laws and policies. OMB Cir. No. A-11, at § 200.2 (2019). We identified 20 examples of the five selected agencies’ incorporating evidence-based approaches in their program design and implementation. Table 10 describes each of these examples. OMB’s July 2013 memorandum stated that agencies’ use of evidence- based approaches could help strengthen agencies’ abilities to improve program performance by using experimentation and innovation to test new approaches for service delivery. In addition, it noted that these approaches can be used to (1) generate new knowledge, and (2) apply existing evidence about approaches found to be effective. Generate new knowledge. OMB guidance notes that new knowledge can be used to improve existing programs or inform decisions about new ones. For example, Education designed the First in the World program to generate evidence about effective strategies for improving college completion rates for underrepresented, underprepared, or low-income students. Program officials told us that, prior to the issuance of the 2014 grant solicitation for the program’s first year, Education had limited evidence of effective approaches. As noted in the solicitation, Education sought to expand its evidence base about effective approaches through the first round of grant awards. Using a tiered evidence approach, the program awarded grants to institutions of higher education to implement and evaluate the effectiveness of approaches, such as coaching or advisement services, intended to increase the number of these students who complete postsecondary education. The first round awarded grant funds to projects in a single evidence tier to test and evaluate the effectiveness of approaches. Education officials told us that after the program’s first year, they conducted a literature review to identify approaches that were supported by some evidence of their effectiveness. Using this evidence, Education created a second tier for the 2015 grant awards, for which grantees could receive increased funding by implementing one of the program designs identified in the literature review. Officials told us they intend to publish the final results of First in the World grant recipient evaluations in Education’s What Works Clearinghouse. Evaluation results will not be available until after the completion of the grant periods, the first of which ended in September 2019. However, Education officials told us that the evidence they have generated to date has improved their understanding of services that could potentially help at-risk students complete post-secondary education. Apply effective approaches. To meet increased demand for services in a constrained resource environment, OMB’s guidance encourages agencies to allocate resources to programs and approaches backed by strong evidence of effectiveness. In addition, OMB’s guidance encourages agencies to “scale up” effective program approaches by expanding them to a larger or different group of recipients. For example, USAID created the Development Innovation Ventures program in 2010 as a tiered evidence grant competition to create a portfolio of innovative approaches to reducing global poverty. This program provides funding in three tiers, with greater funding provided to those approaches with greater evidence of effectiveness. These three tiers (which USAID referred to as stages) were as follows: 1. Proof of concept. The program provided smaller grants to test the viability of an innovative approach; 2. Testing and positioning for scale. Grantees determined, through rigorous assessments, whether their approach could achieve greater results and also be implemented successfully at a larger scale; and 3. Scaling. The program funded the expanded implementation of an effective approach within one country or replicated that approach in another country. For example, from 2013 to 2015, the Development Innovation Ventures program awarded stage two funding to a nonprofit organization in India. The organization designed a methodology to help primary school students improve reading skills by grouping students according to skill level, instead of age or grade and tailoring lessons to their learning level. Evidence generated through randomized control trials showed that the approach was effective. Based on that evidence, in 2017, the program awarded stage three funding to replicate the approach in Zambia. Earlier in this report, we discussed agency-wide evidence assessment and prioritization processes established by the five selected agencies. In addition to those processes, officials described other actions they have taken to coordinate fragmented evidence-building activities across organizational levels (see table 11). Some of these actions were ad hoc (i.e., they did not occur regularly) or not comprehensive in nature (i.e., they did not focus broadly across different sources of evidence or did not cover the entire agency). In addition to the above contact, Benjamin T. Licht (Assistant Director), Daniel Webb (Analyst-in-Charge), Amanda Prichard, Kelly Turner, and Brian Wanlass made significant contributions to this report. Valerie Caracelli, Jacqueline Chapin, Ann Czapiewski, Steven Putansu, and Andrew J. Stephens also made key contributions.", "summary": "Congress and OMB have taken steps intended to strengthen federal evidence-building activities. In September 2017, a federal commission found that agencies had uneven capacity to support, or did not fully coordinate, a full range of evidence-building activities. GAO was asked to examine the coordination of federal evidence-building activities. This report (1) describes selected agencies' actions that align with direction from Congress and OMB to strengthen evidence-building activities and (2) examines the extent to which selected agencies' processes for coordinating those activities reflect leading practices for collaboration. To address these objectives, GAO reviewed documents and interviewed officials about federal evidence-building activities at five selected agencies. GAO selected these agencies based on the greater number of experiences they had in comparison to other agencies incorporating these activities into the design and implementation of certain programs. GAO assessed their coordination of these activities against four leading practices for collaboration identified in GAO's past work. Federal decision makers need evidence about whether federal programs and activities achieve intended results as they set priorities and consider how to make progress toward national objectives. The five agencies GAO reviewed took actions that align with direction from Congress and the Office of Management and Budget (OMB) to strengthen their evidence-building activities. The five agencies are: the Departments of Education, Health and Human Services (HHS), and Labor (DOL); the Corporation for National and Community Service (CNCS); and the U.S. Agency for International Development. For example, based on a statutory requirement, a majority of grant funding for HHS's Maternal, Infant, and Early Childhood Home Visiting program is to be used for home visiting models with sufficient evidence of their effectiveness. Consistent with this requirement, HHS annually assesses evidence, such as the results of program evaluations, to identify effective home visiting models that grantees can implement. Evidence-building can involve assessing existing evidence, identifying any new evidence needs, and prioritizing when to fulfill those needs. These efforts are fragmented within each of the five agencies—that is, each has multiple organizational units with responsibilities for evidence-building. For example, DOL has established separate units responsible for different sources of evidence—evaluations, performance information, and statistics. Effective collaboration can help agencies manage this fragmentation, and lead to improved results. ; (3) clarifying roles and responsibilities ; and (4) documenting that information in written guidance . However, agencies' processes for determining which new evidence to generate, when, and how (i.e., prioritizing new evidence) did not always reflect the leading practices (see figure). GAO is making a total of seven recommendations to DOL, CNCS, and HHS to better reflect leading collaboration practices in their evidence prioritization processes. DOL concurred, CNCS neither agreed nor disagreed, and HHS did not concur with the recommendations. CNCS and HHS stated, but did not provide information to support, that each had already taken relevant actions. GAO continues to believe the recommendations are valid, as discussed in the report.", "document_type": "gao"}
{"report": "Federal agencies’ fleets consist of many types of vehicles that support a variety of purposes. For example, federal vehicles may be used to carry staff and gear to remote, off-road locations to perform maintenance or other tasks; to transport and provide healthcare to veterans; or to support daily operations on military installations. Congress and several administrations have required federal agencies to take various steps to reduce federal fleets’ petroleum use and greenhouse gas emissions. During fiscal year 2017, agencies were: to meet requirements to acquire alternative fuel vehicles and low greenhouse-gas-emitting vehicles; to increase use of alternative fuel; and to decrease use of petroleum and per-mile greenhouse gas emissions (see table 1). According to DOE guidance for the 2015 Executive Order, acquiring such vehicles and increasing the use of alternative fuels can facilitate the goals of reducing both petroleum use and greenhouse gas emissions. For fiscal year 2017, in addition to meeting the above requirements, federal agencies were to meet other requirements related to overall fleet management. Federal regulations require agencies to complete a fleet management plan annually and conduct an assessment of their fleet at least every 5 years. In addition, an Executive Order issued by the prior administration in 2015 directed agencies to determine and plan for their optimum fleet inventory with emphasis placed on eliminating unnecessary or non-essential vehicles. Certain federal fleet energy directives in place in fiscal year 2017 were revoked by an Executive Order issued in May 2018. Specifically, directives related to acquiring zero emission (electric) vehicles and reducing per-mile greenhouse gas emissions, as well as the additional fleet management expectations, were revoked. The Trump administration issued a new Executive Order requiring that the Secretary of Energy, in collaboration with other federal agencies, review existing federal vehicle fleet requirements and report to the Council on Environmental Quality (CEQ) and the Office of Management and Budget (OMB) regarding opportunities to optimize federal fleet performance, reduce associated costs, and streamline reporting and compliance requirements. According to DOE officials, DOE submitted a report to CEQ and OMB as required. In April 2019, CEQ and OMB issued implementing instructions for the Executive Order. The implementing instructions emphasized that agencies should focus on the statutory requirements while increasing efficiency, optimizing performance, and reducing waste and costs. The guidance particularly emphasized agencies’ focus on reducing petroleum use and increasing alternative fuel consumption. The guidance did not mention the extent to which agencies should continue to acquire any specific type of alternative fuel vehicle. Annually, federal agencies are responsible for reporting vehicle inventory (including acquisitions and disposals), fuel consumption, mileage, and cost to the FAST database. Additionally, federal agencies are required to annually report on their fleets’ inventories, operating costs, and other fleet data. Costs submitted to the FAST database include acquisition costs, maintenance, fuel costs, indirect costs, commercial lease, GSA lease, and disposal proceeds. Prior to fiscal year 2017, agencies submitted this data at an aggregate, rather than the vehicular level, so that costs or other performance could not be analyzed at the vehicular level. For fiscal year 2017, as required by GSA and DOE, agencies began submitting vehicular level data to the FAST database, providing more detail about agency’s vehicles. The FAST database specifically tracks data to assess agencies’ performance relative to fleet energy requirements in federal statute and executive orders. A range of vehicles qualify as alternative fuel vehicles (see fig. 1). This range includes vehicles that run entirely on alternative fuel, such as electricity, and dual-fueled vehicles that can run on an alternative fuel as well as on gasoline, such as flex-fuel vehicles, which can run on gasoline or ethanol fuel blends (E85). In 2008, the definition of alternative fuel vehicles was amended to include hybrid electric vehicles, which run on gasoline with help from an electric battery, and, in certain circumstances, other vehicles that would achieve a significant reduction in petroleum consumption, such as highly fuel efficient gasoline vehicles that are also low greenhouse gas-emitting vehicles. Alternative fuel vehicles, including electric vehicles, can offer environmental benefits compared to similarly-sized conventional petroleum-fueled vehicles but also carry their own environmental costs. For example, flex-fuel vehicles, if fueled by E85, reduce petroleum use because E85 consists of up to about 85 percent ethanol, and according to DOE, using ethanol as a vehicle fuel reduces greenhouse gas emissions, along with emission of other harmful toxics. However, using ethanol increases other harmful emissions deemed carcinogenic and may also contribute to ozone formation. Furthermore, as we reported in May 2019, the production of biofuels, such as ethanol, just like the production of gasoline, results in greenhouse gas emissions throughout its life- cycle—including growing the corn feedstock, transporting it, converting it to ethanol, distributing the ethanol, and burning it in an engine. Other emissions are released indirectly through broad economic changes associated with increased biofuel use, including increased ethanol use, such as when changes in land use to grow corn cause the conversion of previously nonagricultural lands into agricultural lands. Nonetheless, recent studies have found the life-cycle emissions of corn ethanol to be lower than those of gasoline. Similarly, battery-electric, plug-in hybrid electric, and hybrid-electric vehicles rely on batteries for all or some of their power, reducing or eliminating petroleum use and associated tailpipe greenhouse gas emissions, but charging, producing, and disposing of these batteries can result in environmental effects. With respect to charging, the production of electricity to power these vehicles results in emissions, the amount of which is dependent on the source of the electricity, a factor we discuss in greater detail later in this report. With respect to production, GAO previously reported that extracting lithium and other minerals from locations where it is abundant, such as in South America, can pose environmental challenges that would damage the ecosystems in these areas. With respect to disposal, according to DOE’s alternative-fuels data center, the disposal of batteries used in electric and hybrid-electric vehicles can result in hazardous materials entering the waste stream— but work is under way to develop battery recycling processes that minimize the life-cycle effects of such batteries. According to DOE, as electric-drive vehicles become increasingly common, the battery-recycling market may expand. In addition, the climate in which battery-electric and plug-in electric vehicles are used can affect the life of the battery. However, federal agencies do not collect the data that would allow analysis of these effects specific to the use of vehicles in federal agencies’ fleets. Furthermore, emissions related to fuel production or battery production or disposability are not incorporated into the requirements placed on federal agencies with respect to their fleets. As we discuss in more detail later, the various types of alternative fuel vehicles vary in the extent to which they can help agencies meet existing requirements to reduce petroleum use and the subsequently revoked requirement in place for fiscal year 2017 to reduce tailpipe greenhouse gas emissions. According to DOE officials, DOE is responsible for overseeing energy goals and requirements and assists agencies in meeting these federal energy requirements. DOE tracks whether federal agencies are meeting the fleet energy requirements by analyzing the fleet inventory, fuel consumption, and fuel use data uploaded to the FAST database. DOE also oversees the Fleet Sustainability Dashboard (FleetDASH) database. FleetDASH tracks agencies’ fuel consumption through data produced when employees use fuel cards. This tool can track where vehicles are filling up and if there was an alternative fuel station nearby that could have been used. FleetDASH can also provide agency fleet managers with reports on alternative fuel use and when drivers missed opportunities to fuel with alternative fuels. DOE also issues guidance and conducts research into vehicle technologies that can support energy requirements, including electric vehicles. In prior work, we recommended that DOE develop guidance for agencies that specifies the elements that agencies should include in their plans for acquiring a mix of vehicles to meet federal requirements and goals. In June 2010, DOE issued the Comprehensive Federal Fleet Management Handbook, implementing this recommendation. DOE’s Fleet Management Handbook recommends to agencies how to develop greenhouse gas and petroleum reduction strategies and acquire vehicles in support of these strategies, among other issues. DOE also has developed online tools to help provide guidance to agencies and consumers on the fuel efficiency and environmental effects of vehicles. GSA is responsible for providing vehicles for federal agencies to purchase or lease. GSA is a mandatory source for purchase of new vehicles for executive agencies and other eligible users. Federal agencies can also use GSA to acquire leased vehicles. Under this arrangement, an agency informs GSA what kind of vehicle is necessary for its mission. Every year, GSA publishes an annual guide on vehicles available for purchase or lease that includes the vehicles’ fuel type, purchase and lease prices, size, and other specifications. In setting the lease prices, GSA is required by law to recover all costs it incurs in providing vehicles and services to federal customers. Agencies that lease vehicles from GSA generally pay a monthly rate and a mileage rate. These charges are designed to cover fixed costs such as: (1) the vehicle’s acquisition cost; (2) administrative costs (including staff and facilities); and (3) depreciation—as well as the variable costs of fueling (except electricity used) and vehicles’ maintenance. In the case of alternative fuel vehicles, if the cost of the vehicle is greater than that of an equivalent conventional vehicle, agencies must cover these higher costs. Pursuant to law, GSA distributes these higher costs for alternative fuel vehicles across the agency’s entire leased fleet via a flat per-vehicle monthly surcharge in the year the vehicle was acquired. Surcharges are set at the agency headquarters’ level. According to a GSA fact sheet, this approach allows GSA to offer a greater variety of alternative fuel vehicles without affecting lease rates of non-alternative fuel vehicles and spread the additional cost across all agencies. At times, GSA has conducted special pilot programs that have waived higher costs of alternative fuel vehicles in order to test new technology. For example, in 2011 and 2014, GSA ran two pilot programs that added over 300 electric vehicles and charging stations to the fleet. According to GSA officials, these pilots were designed to help GSA Fleet understand more about the performance, costs, and maintenance needs of electric vehicles to help them prepare for the potential increase in electric vehicles in the fleets in order to better advise other agencies on these vehicles’ use and operation. In these programs, GSA spent over $5.9 million covering the additional costs for the electric vehicles and spent another $1.2 million on purchasing electric-vehicle-charging stations. The majority of agencies subject to federal-fleet energy requirements reported meeting most requirements for fiscal year 2017 by changing the mix of vehicles acquired and improving fleet management. Specifically, agencies credited acquiring low greenhouse-gas-emitting and alternative fuel vehicles for helping to reduce petroleum use and per-mile greenhouse gas emissions. Agencies also described improving their fleet management in other ways, such as removing unnecessary vehicles and reducing miles traveled in order to reduce petroleum use and greenhouse gas emissions. Agencies’ fleets reflected increasing numbers of alternative fuel vehicles over the past 10 years, predominantly flex-fuel vehicles. DOE and other agency officials we spoke with from agencies that met the reduction targets for petroleum use and per-mile greenhouse gas emissions generally attributed their ability to meet these requirements to efforts in two areas: 1. acquiring low greenhouse-gas-emitting vehicles whenever they could (even if they did not meet the related requirement) as well as alternative fuel vehicles, and 2. improving fleet management in other ways, such as by eliminating unnecessary vehicles or driving fewer miles, in line with GSA’s fleet management guidance. In line with these efforts, a majority of agencies reported meeting most fleet energy requirements for fiscal year 2017 (see table 2). Fleet managers at two of the case study agencies said that acquiring low greenhouse-gas-emitting vehicles was key to their ability to meet the fiscal year 2017 targets for reducing petroleum use or greenhouse gas emissions. For example, although VA reported not meeting the low greenhouse-gas-emitting acquisitions requirement for fiscal year 2017, VA officials said that they did acquire low greenhouse gas vehicles when they could, and that to the extent they acquired such vehicles, it was the primary reason they were able to reduce their per-mile greenhouse gas emissions by 24 percent from fiscal year 2014 to fiscal year 2017. This reported reduction far exceeded the requirement for a 4 percent reduction in per-mile greenhouse gas emissions during this time frame. According to VA officials, VA’s acquisition process requires them to consider low greenhouse-gas-emitting vehicles for each acquisition and to select one whenever one is available that will meet the purpose for the vehicle. According to VA officials, the reason VA reported not meeting the low greenhouse-gas-emitting acquisitions requirement for fiscal year 2017 was that the agency did not consistently self-certify for exceptions to the requirement in cases where there was no low greenhouse-gas-emitting vehicle available that met their mission needs, an issue we also heard from GSA officials. (As shown in table 2, above, this was the one fleet- energy requirement that was reported as being met by less than a majority of the 29 agencies, with 8 reporting meeting this requirement for fiscal year 2017). Fleet managers at all of our case study agencies emphasized that they sought to acquire low greenhouse-gas-emitting vehicles whenever one was available that would serve their needs. GSA officials told us agencies are acquiring significant numbers of low greenhouse gas vehicles. By their count, of the sedans agencies acquired in fiscal year 2018, 92 percent were low greenhouse-gas-emitting vehicles; of the light-duty sport-utility vehicles and trucks agencies acquired, 45 percent were low greenhouse-gas-emitting vehicles. GSA officials stated that according to their analysis, it is likely that the low number of low greenhouse gas vehicles being reported is a result of how the vehicles are identified and reported, and that the number reported is lower than the number acquired. Vehicles considered to be low greenhouse-gas-emitting vehicles include selected makes and models of conventionally fueled vehicles that were identified by EPA as highly efficient, as well as different types of alternative fuel vehicles, such as selected makes and models of flex fuel vehicles, plug-in hybrid electric vehicles, and hybrid electric vehicles, and all battery electric vehicles. Thus, the costs of vehicles considered to be low greenhouse-gas-emitting vary widely. We discuss later in the report the costs of different types of alternative fuel vehicles. Along with the acquisition of low greenhouse- gas-emitting vehicles generally, fleet managers at some case study agencies stated that their acquisition and use of alternative fuel vehicles also helped them to meet the fiscal year 2017 targets for reducing petroleum and per-mile greenhouse gas emissions. Fleet managers at two agencies we spoke with stated or reported that their acquisitions of hybrid vehicles and, to a lesser extent, small numbers of plug-in hybrid and battery electric vehicles also helped managers to meet petroleum and greenhouse gas emissions reduction targets. According to Interior’s fiscal year 2015 Strategic Sustainability Performance Plan, over 1,300 hybrids helped the agency reduce petroleum consumption, increase fuel efficiency, and reduce greenhouse gas emissions. Within Interior, officials at the National Park Service told us that they replaced older, inefficient gas vehicles with more fuel efficient hybrids. EPA officials stated that acquiring hybrid vehicles and plug-in hybrid electric vehicles helped them exceed their per-mile greenhouse gas emission reduction target for fiscal year 2017 by just over 9 percent. Furthermore, of the 29 agencies we surveyed, 20 identified that a key benefit to acquiring battery-electric or plug-in hybrid electric vehicles was environmental, particularly in reducing greenhouse gas emissions. In addition, some fleet managers emphasized the role that flex-fuel vehicles fueled with E85 had played in their efforts to meet these targets. Some agencies told us that they acquired flex-fuel vehicles to meet alternative fuel vehicle acquisition requirements, and that using E85 in these vehicles contributed to reducing petroleum use and per-mile greenhouse gas emissions. For example, DOT’s fleet manager stated that DOT’s acquisition of flex-fuel vehicles and focus on using E85 to fuel those vehicles when available helped DOT to meet these targets for fiscal year 2017. Similarly, in the 2016 Strategic Sustainability Performance Plan, EPA emphasized that using alternative fuel in flex-fuel vehicles helped the agency reduce petroleum use. According to DOE officials, for agencies that met the fiscal year 2017 petroleum reduction target, about 11 percent of their petroleum reduction was due to using alternative fuel. According to DOE officials, the balance of petroleum reduction for these agencies was achieved through fuel efficiency improvements and behavioral changes, including reduction in vehicle miles traveled. In spite of the emphasis some agencies put on alternative fuel use as part of their strategy to reduce petroleum use and greenhouse gas emissions, alternative fuel use in federal fleets overall has dropped in recent years. According to data reported in FAST, while alternative fuel use increased from 4.9-million gasoline gallon equivalents in fiscal year 2005 to 16.2- million gasoline gallon equivalents in fiscal year 2013, since fiscal year 2013 it declined to 12.1-million gasoline gallon equivalents in fiscal year 2017 (see fig.2). The fleet energy requirement to increase use of alternative fuel by 10 percent is based on a fiscal year 2005 baseline, and most agencies reported continuing to meet this requirement. In fact, as a whole, the federal government could continue to decrease its alternative fuel use by as much as 6.7 million gasoline gallon equivalents and still meet the targeted 10 percent increase above the fiscal year 2005 baseline. While E85 was the primary alternative fuel used, according to DOE data, alternative fuel use per dual-fueled vehicle is also at comparatively low levels—decreasing between fiscal years 2012 and 2016 from 123 to 90 gasoline gallon equivalents. This decrease was despite DOE’s reporting that the number of dual-fueled alternative fuel vehicles with access to alternative fuel increased from about 80,000 vehicles to about 112,000 over the same period. DOE officials said agencies could be using more alternative fuel, but suggested the recent decline could be due to a general lack of available E85 stations, among other reasons. Fleet managers from all five case study agencies reported that their efforts to improve fleet management—even beyond those specifically related to acquiring alternative fuel vehicles—also helped them to reduce petroleum use and greenhouse gas emissions. Officials at several agencies reported in their Strategic Sustainability Performance Plans or told us that carrying out required fleet reviews helped them reduce the number of vehicles and change to more fuel-efficient vehicles, which directly helped them meet energy requirements. For example, EPA officials told us that through reviewing their vehicle usage, they identified which vehicles to either eliminate or replace with more efficient ones, moves that resulted in reducing petroleum use. Furthermore, in its 2017 Strategic Sustainability Performance Plan, EPA cited that it has reduced its fleet by 170 vehicles in the past 5 years and that its last study showed the potential to discontinue use of 80 to 100 vehicles in the next 5 years. Similarly, DOD reported in its fiscal year 2016 Strategic Sustainability Performance Plan that Army’s strategy to meet the requirement to reduce petroleum use was to reduce its fleet size and find the right mix of vehicles to meet its mission needs—in addition to acquiring fuel-efficient and alternative fuel vehicles. In this plan, Army reported that between fiscal year 2011 and fiscal year 2015, it reduced its fleet’s size by 16,400 vehicles. According to GSA officials, at times, an agency may reduce its petroleum use and greenhouse gas emissions more by replacing large, inefficient vehicles (such as older, large trucks) with more efficient vehicles (such as new small trucks or sedans) even if both are fueled by gasoline—than by replacing an already highly efficient conventionally fueled small sedan with an alternative fuel vehicle of the same size. Our review of FAST data suggests that agencies were more successful in reducing the number and size of their sedans and size of their sport utility vehicles than in reducing the number or size of their larger vehicles, such as vans and trucks (see fig. 3). For example, overall, the number of sedans in federal fleets fell by 4 percent from fiscal year 2013 to fiscal year 2017, with the number of larger sedans falling by 15 percent and the number of subcompact sedans increasing by 37 percent, suggesting that agencies moved to smaller, more efficient sedans. On the other hand, among passenger vans, there was an increase in heavier, medium-duty passenger vans, and an overall increase in trucks was fueled by an increase in medium- duty trucks, while the number of light-duty trucks fell. In addition to reviewing and changing fleets, fleet managers also reported that encouraging certain driver behavior helped them to meet energy goals. According to VA’s, Interior’s, and EPA’s fleet managers, agencies also reduced greenhouse gas emissions through educating or encouraging drivers to make behavioral changes such as reducing vehicle idling and overall miles traveled. For example, according to EPA fleet managers, certain regional offices have systems in place that facilitate their combining of motor pools and sharing trips to reduce petroleum use. As previously indicated, according to DOE officials, 11 percent of the reduction in petroleum use for agencies that met the petroleum reduction target was due to an increase in alternative fuel use. According to DOE officials, the balance of petroleum reduction for these agencies was achieved through fuel efficiency improvements and behavioral changes, including reduction in vehicle miles traveled. As a result of agencies’ efforts to meet federal fleet energy requirements, the number of alternative fuel vehicles in federal fleets has grown steadily over the past 10 years, largely due to an increase in flex-fuel vehicles. The number of alternative fuel vehicles in federal fleets increased by 65 percent from fiscal year 2008 through fiscal year 2017, according to FAST data (see fig. 4). During that same time, the number of conventional petroleum-fueled vehicles decreased by 19 percent. As a result, as of fiscal year 2017, alternative fuel vehicles made up about 38 percent of approximately 604,000 total domestic vehicles in the fleet. Most of the alternative fuel vehicles in the federal fleets—about 87 percent in fiscal year 2017—are flex-fuel vehicles. As previously mentioned, while flex-fuel vehicles can contribute to reducing petroleum consumption when E85 is used, data show that the usage of E85 continues to fall (see fig. 2), thus reducing the potential environmental benefits of acquiring these vehicles. While the majority of flex-fuel vehicles offered to federal agencies by GSA in fiscal year 2017 did not cost more for agencies to acquire than equivalent petroleum-fueled vehicles, some flex fuel vehicles did cost more for agencies to acquire, with, for example, a few sport-utility flex-fuel vehicles costing between $4,000 and $7,000 more than comparable vehicles. Within the past decade, the number of hybrid vehicles in federal fleets also increased significantly, from almost 1,800 in fiscal year 2008 to over 25,000 in fiscal year 2017. Hybrids accounted for about 11 percent of all alternative fuel vehicles in fiscal year 2017. Finally, while agencies have acquired some electric vehicles, the number of electric vehicles in federal fleets has remained very small—consisting of just over 1,000 plug-in hybrid electric and battery electric vehicles in fiscal year 2017. In spite of federal agencies’ reported general success in meeting fleet energy requirements, several challenges may hinder agencies’ further progress towards the goals of reducing federal fleets’ petroleum use and greenhouse gas emissions. First, although acquiring electric and hybrid vehicles could help agencies meet the current fleet energy goals to reduce petroleum use and per-mile greenhouse gas emissions in federal fleets, depending on where and how the vehicles are used, costs can be prohibitive. The costs of these vehicles and charging infrastructure make it challenging for agencies to acquire them on a large scale. Second, a lack of fuel and infrastructure availability limits agencies use of alternative fuel, specifically E85. Third, agency officials we interviewed stated that a continuing need for larger vehicles to perform certain tasks limits the number of low greenhouse gas vehicles agencies can acquire—and thus the potential to reduce petroleum use and greenhouse gas emissions. Acquiring electric and hybrid vehicles could help agencies meet fleet energy goals, but higher costs pose challenges. As described previously, prior to May 2018, federal agencies were under a directive to acquire zero-emission (electric) and plug-in hybrid electric vehicles for 20 percent of all new agency passenger vehicle acquisitions by December 31, 2020, and for 50 percent by December 31, 2025. Some of the discussions we had with agency officials about challenges related to acquiring electric vehicles took place while this directive was in effect. In part because guidance on the new Executive Order had not been issued at the time we spoke with them (although it was subsequently issued in April 2019), agency officials we spoke with after this directive was revoked said they were uncertain of the effect of the new Executive Order and would continue to try and meet fleet energy goals until new guidance was issued. Compared to other alternative fuel vehicles available from GSA, battery electric, plug-in hybrid electric, and hybrid electric vehicles can offer potential to further general federal goals to reduce petroleum use and tailpipe greenhouse gas emissions. Specifically, battery electric vehicles consume no petroleum and produce zero tailpipe greenhouse gas emissions, while plug-in hybrid electric vehicles have the potential to consume very little gasoline, with a correspondingly small amount of tailpipe greenhouse gas emissions from the gasoline used, and hybrid electric vehicles offer higher fuel economy than many other vehicles. According to DOE’s Fleet Management Handbook, replacing a petroleum- fueled vehicle with a battery electric vehicle provides a 100 percent reduction in that vehicle’s use of petroleum. In addition, according to DOE officials, for purposes of tracking agencies’ compliance with the now- revoked Executive Order’s fleet requirements, battery electric vehicles were considered emissions free, and plug-in hybrids were considered emissions free when run on electricity. The now-revoked fleet requirements did not consider emissions generated during the production of fuel or the manufacturing process. The Council on Environmental Quality guidance states that emissions generated from the production of electricity are not counted toward agencies’ fleet emissions because those emissions are assumed to be captured in each agency’s facility electricity reporting and their annual greenhouse gas data report. Counting them as fleet emissions would result in double counting. Nevertheless, to fully consider the potential environmental benefits of alternative fuel vehicles, these emissions would need to be considered and compared to the emissions generated by the production of fuel and manufacturing process of conventionally fueled vehicles. From a full life-cycle perspective, greenhouse gases emitted during the manufacturing of a vehicle affect a vehicle’s overall emissions. Accurately determining the amount of greenhouse gas emitted during the manufacturing of different types of vehicles is complicated, and we found no federal source that publishes this information. However, a study by the International Energy Agency found that manufacturing battery electric vehicles results in higher greenhouse gas emissions than manufacturing conventional internal combustion engine gasoline-fueled vehicles—but that over the typical life of an electric vehicle, the elimination of tailpipe emissions results in these vehicles having lower greenhouse gas emissions overall than conventional gasoline-fueled vehicles, with the amount of emissions savings depending on the carbon intensity of power generation used to charge the vehicles. Another study, by Argonne National Laboratory, considered mid-size light-duty vehicles. According to this study, on a life-cycle basis—including emissions related to the manufacture and disposal of the vehicles, the production of the fuel, and the use of fuel to operate the vehicle—hybrid electric vehicles produced about 25 percent fewer greenhouse gas emissions per mile than conventionally fueled gasoline vehicles, plug-in hybrid electric vehicles produced about 26 to 29 percent fewer greenhouse gas emissions per mile than conventionally fueled gasoline vehicles, and battery electric vehicles produced about 26 to 34 percent fewer greenhouse gas emissions per mile. The study also considered the life-cycle greenhouse gas emissions for flex fuel vehicles run on E85, finding them to produce about 20 percent fewer greenhouse gas emissions per mile than a conventionally fueled gasoline vehicle. This study also considered the costs of alternative fuel vehicles in light of their potential to reduce greenhouse gas emissions. It estimated that in 2013 dollars and, based on high volume production, a 15-year vehicle life-cycle, and a 5 percent discount rate, the greenhouse gas emissions avoided by using hybrid-electric vehicles compared to a conventional gasoline fueled vehicle cost $240 per metric ton. For plug-in hybrid electric vehicles, the cost is between $390 and $860 per metric ton of greenhouse gas emissions avoided, and for battery electric vehicles the cost is from $1,090 to $2,600 per metric ton of greenhouse gas emissions avoided. For flex fuel vehicles, the cost was estimated to be $170 per metric ton of greenhouse gas emissions saved. Based on these findings, when an agency replaces a petroleum fueled vehicle with a battery electric vehicle, a plug-in hybrid electric vehicle, or a hybrid electric vehicle, it can reduce its petroleum use and greenhouse gas emissions, though the extent of its reduction depends on the type of vehicle the agency acquires, and the type of vehicle it replaces, as well as many other factors. However, it may currently be paying more for such vehicles from a life-cycle perspective. In the time since this study was published, according to DOE, battery costs have continued to fall, and these vehicles may be cost competitive in the near future. For battery-electric vehicles and plug-in hybrid electric vehicles, which must be regularly charged from the electrical grid, one consideration included in the Argonne National Lab study’s analysis of how much greenhouse gasses are emitted through the vehicle’s operation is the level of greenhouse gas emissions associated with electricity generation. Such emissions affect the extent to which using electricity instead of gasoline to fuel vehicles reduces the amount of greenhouse gas emissions generated into the atmosphere—and this varies by location. While the Argonne National lab study described above based its analysis on the average mix of electrical generation in the U.S., the amount of greenhouse gas emissions associated with electricity generation in the U.S. actually varies widely depending on the sources used to generate the electricity. These sources vary depending on the region of the country where the electricity is produced. For example, the production of electricity from burning coal causes relatively high greenhouse gas emissions, while the production of electricity from solar or wind causes little to no greenhouse gas emissions. As a result, a battery electric vehicle charged in a region with low coal electricity generation, such as the Northeast—whose electricity generation mix includes about 2.6 percent coal—will result in greater greenhouse gas emissions reductions than those charged in regions where most electricity generation comes from coal, such as the upper Midwest, which uses about 62.3 percent coal (see fig. 5). These figures are meant to illustrate the differences in electricity generation, and they do not account for other factors that may affect vehicles’ efficiency and thus the extent to which they lead to reductions in emissions. For example, in extreme weather conditions, the range of battery-electric vehicles can be reduced, resulting in more frequent charging, and thus more electricity use. Further, the use of air conditioning or other components in the vehicle can also impact their fuel efficiency. We analyzed emissions data on vehicles operating in different parts of the country and found that when considering both tailpipe and fuel-production greenhouse gas emissions, electric and plug-in hybrid electric vehicles produce less greenhouse gas emissions than an equivalent gasoline-only vehicle in both higher-coal and lower-coal electricity generation regions. In higher-coal electricity generation regions, however, electric vehicles can offer less or about an equivalent reduction in greenhouse gas emissions to comparably-sized hybrid electric vehicles, whereas in lower- coal electricity generation regions, electric vehicles offer the opportunity to reduce greenhouse gas emissions by a greater extent than comparably-sized hybrid electric vehicles. In 2009, we recommended that DOE develop guidance to help agencies plan to acquire the right mix of vehicles that can meet requirements while also taking into account the energy sources used to generate the electricity used to fuel electric vehicles. In response, DOE issued guidance that recommended agencies consider, among other things, whether coal-based electricity is used in an area in order to evaluate the location and emissions-reduction potential of using such vehicles. However, of the five case study agencies we spoke to, no agency officials said that they specifically worked to locate electric vehicles where the production of electricity was likely to produce fewer greenhouse gases. Since greenhouse gas emissions due to the production of electricity were not considered in the now-revoked executive order’s requirements and, according to the case study agency officials, was not stressed by GSA in discussions about increasing electric vehicles, they stated that this had not been a focus of their efforts. Instead, they stated that they focused on locating electric vehicles where they were able to install electric charging stations and had a mission need that fit with the use of electric vehicles. According to some agency officials, the higher acquisition costs associated with electric vehicles and the costs of installing charging infrastructure have hindered the extent of their integration into federal fleets. (See app. III for a more detailed discussion of life-cycle costs of electric vehicles.) As part of an effort to further the overall goal of reducing greenhouse gas emissions, the now-revoked 2015 Executive Order called for agencies to increase their acquisition of zero-emission vehicles (battery-electric vehicles) or plug-in hybrid electric vehicles by 2020. While all five case study agencies had acquired small numbers of electric vehicles and associated charging infrastructure, two fleet managers said that the cost challenges would have made it difficult to acquire sufficient numbers of vehicles to meet the Executive Order’s requirements by 2020, had the Executive Order not been revoked. To meet the revoked electric-vehicle acquisition requirements, federal agencies would have had to acquire close to 3,000 battery electric or plug-in hybrid electric vehicles per year starting in 2020, according to GSA officials. According to data provided by GSA, in fiscal year 2017, agencies purchased 373 battery electric or plug-in hybrid electric vehicles. Just over half of these vehicles were plug-in hybrid electric minivans, with the rest being sedans. The purchase of these 373 battery electric or plug in hybrid electric vehicles, plus an additional 4,584 hybrid electric sedans, made up about 31 percent of the just over 16,000 sedans and minivans acquired that year—and increased the total amount agencies spent purchasing sedans and minivans by about $10.5 million (see table 3)—or about 3 percent of the total of approximately $314 million spent purchasing sedans and minivans overall. Among the hybrid electric, battery electric, and plug-in hybrid electric sedans and minivans, federal agencies purchased the largest numbers of hybrid- electric sedans, which had the smallest additional average per-vehicle costs as compared to comparably sized gasoline or flex-fueled vehicles. As a result, agencies spent an average amount of about $2,000 more per battery electric, plug-in hybrid electric, and hybrid electric vehicle acquired, although the average amount per vehicle varied widely by size and type of vehicle acquired. As described below, some of the higher acquisitions costs of these alternative fuel vehicles will be recovered due to lower maintenance and fuel costs of the vehicles over time. However, we were unable to get data on federal agencies’ actual lifecycle costs of these vehicles because, according to agency officials, agencies had not tracked these data consistently. Of the 29 agencies we surveyed, 11 identified acquisition costs as a challenge to acquiring and using electric vehicles. In addition, 20 of the 29 agencies identified charging infrastructure as a key challenge to acquiring electric vehicles, citing the costs of installation among other challenges. In discussions with case study agencies, federal officials did not cite the acquisition costs of flex-fuel vehicles as a challenge to acquiring these vehicles. Some officials stated that these vehicles’ relatively low costs compared to other alternative fuel vehicle options was one reason that agencies have largely met the alternative fuel vehicle acquisitions requirement through the acquisition of flex fuel vehicles. GSA’s purchasing data did not provide sufficient detail for us to analyze the extent to which agencies paid more to purchase flex fuel vehicles. According to GSA’s leasing data on GSA-leased vehicles, for fiscal year 2017, agencies acquired over 20,600 alternative fuel vehicles, of which over 14,700 were flex fuel vehicles leased at no additional cost. However, agencies also acquired 1,268 flex fuel vehicles that, on average, had an additional cost of about $2,300, with the result that agencies spent a total of about $2.9 million more to acquire these vehicles to lease than if they had acquired equivalent gasoline-fueled vehicles. When agencies choose to lease an alternative-fuel vehicle that is more expensive than a comparable conventionally fueled vehicle, under law, GSA must spread that difference in cost—sometimes referred to as the incremental cost—across the agency’s entire fleet during the year the alternative fuel vehicle is acquired. According to GSA officials, this requirement makes it easier for agencies to incorporate higher-priced alternative fuel vehicles, such as battery-electric or plug-in hybrid electric vehicles, into their fleets. The difference in cost between acquiring a plug- in hybrid electric or battery-electric vehicle compared to an equivalently sized conventionally fueled vehicle can vary depending on the amount GSA has negotiated with the dealer to pay for a particular vehicle. For example, GSA’s lease offerings showed that for fiscal year 2019, agencies would have to pay anywhere from about $5,300 to $19,400 more to acquire a plug-in hybrid electric vehicle than to acquire an equivalently sized conventionally fueled vehicle, and approximately $16,100 to $18,800 more to acquire a battery electric vehicle that is an equivalently sized conventionally-fueled vehicle. Officials from two case study agencies told us that because GSA spreads the additional costs over an agency’s entire leased fleet, the costs may not affect the agency’s budget much as long as the agency acquires only a small number of vehicles. For example, according to a local DOT official, the acquisition of two battery-electric Ford Focuses added an additional $15 per vehicle to each of its vehicles in its fleet. While electric vehicles have higher acquisition costs, they generally have lower fuel and maintenance costs than conventionally fueled vehicles, and as a result, GSA officials charge agencies lower mileage rates for these vehicles. GSA also charges agencies lower mileage rates for hybrid vehicles, based on their higher fuel efficiency. Of the agencies we surveyed, 14 of the 29 identified lower fuel and maintenance costs as a key benefit to acquiring battery electric or plug-in hybrid electric vehicles. Because of these lower mileage rates, the more miles an agency drives a leased electric vehicle, the more the overall cost difference to the agency between an electric vehicle and a conventionally fueled vehicle will shrink. However, our analysis of GSA’s leasing rates showed that over 5 years— the typical life of a lease of an electric vehicle—and with average mileage—these lower mileage costs would not make up for the higher acquisition costs of these vehicles (see fig. 6). GSA officials and several fleet managers also told us that in their experiences with leasing electric vehicles, lower utilization coupled with the lower mileage costs charged by GSA to agencies had not made up for the significantly higher acquisition cost over the life of the leases. The GSA lease costs consider the lifetime costs of the vehicles, including fueling and maintenance and eventual disposal of the vehicle through auction. The five case study agencies we spoke with did not use a life-cycle analysis to compare costs across various vehicle types when making vehicle procurement decisions. However, all five case study agencies told us they analyze life-cycle costs to inform their lease versus purchase decisions. See appendix III for more discussion on life-cycle costs. Fleet managers at three of the case study agencies we spoke with before the Executive Order was revoked told us that they had worked to increase the number of electric vehicles in their fleets, in spite of the higher costs. Officials at a few agencies stated that when the budget allowed, they would try to acquire electric vehicles. For example, VA officials told us that VA budgets for electric vehicles on the local level, and that local staff decide how much of their budget will go towards funding of electric vehicles. VA and Interior officials said their acquisitions of electric vehicles had thus far not greatly affected their budgets, but within Interior, the fleet managers for Fish and Wildlife Services and the Bureau of Indian Affairs said cost could become an issue if more electric vehicles were to be acquired. GSA Office of Governmentwide Policy officials told us that agencies could fit the higher costs of acquiring electric vehicles into their budget by reducing their fleet size and acquiring a few of these more expensive vehicles. Further, GSA has introduced several initiatives to help agencies finance alternative fuel vehicle acquisitions, including specific electric vehicle initiatives. For example, in fiscal year 2016, according to an Army fleet manager, Army acquired electric vehicles through GSA at a price GSA had negotiated that was equal to the price for comparably sized petroleum fueled vehicles. However, this pricing was only offered in 2016 as part of a one-time deal that GSA negotiated with the vehicle manufacturer. In addition to the costs of purchasing or leasing electric vehicles, agencies described challenges balancing the costs of purchasing and installing charging stations with other competing priorities. Agency officials told us they generally prefer charging stations, such as Level 2 stations, that can charge a vehicle in a few hours to allow vehicles to be used multiple times a day. These types of Level 2 charging stations can cost anywhere from about $400 to $8,000 depending on the model and its features and do not include installation costs. Generally, the less expensive models may not include features such as energy monitoring that tracks electricity use or communication capabilities that enables data communication that some fleet managers said they view as necessary to manage and track the performance and costs of electric vehicles. We were unable to determine the total amount that agencies had spent to acquire existing charging stations to date because data were not available at a sufficient level of detail. Installation costs also varied widely, depending, among other things, on the complexity of the installation, such as the need for trenching or upgrading the electrical service. For example, officials from VA told us that sometimes in order to install charging stations, they have had to trench an entire parking lot to ensure the units have the necessary power to charge its vehicles—which can be expensive. DOE estimates that to install a charging station it costs about $100 per foot to trench through concrete, lay conduit, and refill. As a result, it could cost up to $10,000 to trench 100 feet. Further, the Veterans Health Administration indicated that funding for purchasing and installing charging stations at their facilities had to compete with other priorities. Specifically, the costs for charging stations came out of the facilities’ capital-planning budget, which also includes funding for veterans’ care. Similar to determining what agencies have spent on charging stations, we were also unable to determine what total installation costs have been to date because of data limitations. Although many federal facilities are not equipped with fast charging infrastructure and the number of public charging stations remains limited, federal agencies had begun taking steps to install more charging stations. Prior to the 2015 Executive Order being revoked, agencies had recently begun to install more of these stations as part of their efforts to prepare for the requirement that 20 percent of light-duty vehicle acquisitions be zero emission (electric) vehicles or plug-in hybrid vehicles by 2020. We found 12 out of the 29 agencies we surveyed had installed more than 20 charging stations, while 14 others had installed at least one charging station, and only 3 agencies had not installed any charging stations. According to past Strategic Sustainability Performance Plans, agencies had started to implement strategies to increase their electric- vehicle infrastructure. For example, according to EPA’s fiscal year 2016 plan, it planned to conduct a survey of its parking facilities to develop a charging infrastructure policy and plan, including identifying potential locations for charging stations. Similarly, Army officials described taking additional steps, including sending specialized teams to several of its bases to determine the optimal and least costly placement of its charging stations. However, fleet managers also told us they were having difficulties installing electric vehicle infrastructure, in particular at leased facilities. Specifically, several agencies’ fleet managers told us that it was difficult or impossible to install charging stations at leased properties unless their installation was negotiated into the lease from the beginning. In part because guidance on the new Executive Order had not been issued at the time we last spoke with agency officials on this issue, the extent to which the revoking of the directive related to acquiring electric vehicles would affect agencies’ efforts to install charging infrastructure was unclear. Fleet managers told us that another challenge that may limit progress toward energy goals was a lack of fuel availability—in particular the availability of E85—which made it difficult to fuel flex-fuel vehicles with alternative fuel. Of the 29 agencies, 20 identified the availability of E85 as a challenge to using alternative fuel in flex-fuel vehicles. While some agencies still largely rely on flex-fuel vehicles to meet alternative fuel vehicle acquisition requirements, E85 can only be found at about 2 percent of all refueling stations, according to GSA. To help agencies locate alternative fuel stations, such as those with E85, DOE developed an Alternative Fuel Station Locator tool that maps nearby refueling stations. VA and Interior officials said they routinely use the tool to check for accessible alternative fuel stations prior to acquiring an alternative fuel vehicle. However, outside the rural Midwest and Texas, E85 may be difficult to find. In addition, when E85 is available, agency officials from two case study agencies said these locations may be mislabeled, out of service, or too far from the vehicle’s operating location. We reported similar concerns in 2011; specifically, that while agencies acquired primarily flex-fuel vehicles, the low availability of E85 resulted in a majority of flex-fuel vehicles receiving a waiver from the requirement to use alternative fuel, and as a result, agencies refueled their flex-fuel vehicles with petroleum. Another difficulty fleet managers face with regard to increasing the use of E85 is that, even when E85 is available and conveniently accessible, drivers still may refuel with gasoline—even though federal agencies have undertaken a number of efforts to encourage its use. As we mentioned previously, to help agencies track their fleet fuel purchases, DOE developed the FLEETDASH system that can identify opportunities where drivers could have refueled with E85 within 5 miles of their location but, instead, chose not to because they were unaware or unwilling. Some agency officials described using this system to try to increase alternative fuel use. For example, VA officials told us they use FLEETDASH to track and identify opportunities to increase their alternative fuel use. In another example, EPA officials told us that to increase their use of alternative fuels, drivers at one location started to print out maps that identified alternative fuel refueling locations near their routes. DOE recently estimated that if federal agencies refueled flex-fuel vehicles with E85 every time they refueled within 5 miles of an E85 station, the use of E85 would quadruple, and agencies could decrease their use of petroleum by 10 percent and reduce greenhouse gas emissions by a further 9 percent. Another challenge that may limit further progress towards energy goals is that agencies continue to need larger, less efficient vehicles for many of their mission needs, according to many agency officials. According to FAST data, about 85 percent of agencies’ fleets in fiscal year 2018 was comprised of sport-utility vehicles, passenger vans, and trucks (as illustrated previously in fig. 3). In response to our survey, 26 of 29 agencies indicated that mission or intended use was a very important factor when selecting a vehicle, and officials at some case study agencies told us that they had a significant need for larger vehicles to meet certain missions. For example, Interior operates on large rural Indian reservations where they need pick-up trucks or sport-utility vehicles to navigate the often rugged terrain. In another example, DOT officials stated that to support their national airspace facilities, their vehicles must drive off-road carrying bulky or sensitive tools to go to remote air strips. For these purposes, they look to acquire larger vehicles such as cargo vans and enclosed pickup trucks with 4-wheel drive capabilities or 2- wheel-drive sport-utility vehicles that have the ground clearance to meet their needs. GSA and agency officials told us that the vehicles designated as low greenhouse-gas-emitting vehicles are typically smaller vehicles and in some cases are not suitable for these mission needs. For example, GSA offered one 4x2 hybrid-electric sport-utility vehicle and one 4x4 plug-in hybrid-electric sport-utility vehicle in fiscal years 2017 and 2018. In fiscal year 2019, additional vehicles have been added. While these options are considered low greenhouse-gas-emitting vehicles, an agency official told us that they have a variety of other characteristics that may make them less desirable for certain missions—for example, they may cost significantly more than other options to acquire, or, in the case of the plug-in, rely on charging infrastructure that the agency may not have in the location where the vehicle is needed. According to VA staff, there are not enough low greenhouse gas vehicle options to ensure fleet managers can meet mission goals and low greenhouse-gas-emitting vehicle acquisition requirements. For example, VA relies on minivans to transport patients and deliver health care services; however, no gasoline or E85- fueled minivans offered by GSA in fiscal year 2017 were designated as low greenhouse-gas-emitting vehicles. Furthermore, in some cases, when an agency has determined it needs a larger vehicle, fleet managers told us they are likely to choose a flex-fuel vehicle because these vehicles are offered in larger, more rugged models. These vehicles are often not designated as low greenhouse-gas-emitting vehicles but count towards the alternative fuel vehicle acquisition requirements. In contrast, officials representing four case study agencies stated that when the mission need is suitable for a sedan, the agency seeks to acquire low greenhouse-gas-emitting vehicles. GSA offers a number of alternative fuel vehicle options for sedans, including hybrid, battery electric, and plug-in electric hybrid vehicles. Further, many GSA offered gasoline-fueled sedans are also designated as low greenhouse-gas- emitting vehicles. Officials at one agency told us, when possible, the agency acquires alternative fuel sedans such as flex-fuel vehicles, hybrid vehicles, or, in a few cases, electric vehicles. Furthermore, officials at this agency stated that when they are acquiring a vehicle where alternative fuel is not readily available, they will sometimes acquire a low greenhouse-gas-emitting vehicle that runs only on gasoline. We provided a draft of this report to Army, DOE, DOT, EPA, GSA, Interior, and the VA for their review and comment. In response, Army, DOE, EPA, GSA, Interior, and VA provided technical comments which were incorporated as appropriate. Army and DOT reviewed the report but did not provide any comments. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretaries of the Departments of Defense, Energy, Interior, and Veterans Affairs, and the Administrators of GSA and EPA. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff has any questions about this report, please contact me at 202-512-2834 or vonaha@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made major contributions to this report are listed in appendix IV. In April 2018, we initiated a survey of 29 federal agencies’ fleet managers. The questions we asked and the aggregate results of the responses to the closed-ended questions are shown below. Our survey was comprised of closed- and open-ended questions. We do not provide results for the open-ended questions. We received 29 completed survey responses—a response rate of 100 percent. 1. What is the process your agency follows when acquiring a new vehicle to replace a vehicle? Please list (in numerical order) the sequence of events from deciding to acquire a vehicle to actually acquiring it. To the extent that the process is different when adding an additional vehicle, please describe that as well. (Written responses not included) 2. At what point in the above process, does your agency consider whether to acquire an alternative fuel vehicle or a petroleum fuel vehicle when replacing a vehicle? To the extent that the process is different when adding an additional a vehicle, please describe that as well. 3. (Written responses not included)In the process to replace a vehicle described above, does your agency consider vehicle life-cycle cost information as part of a lease versus purchase analysis? 3a. If yes, does your agency consider the following factors in their vehicle life-cycle cost analysis? Please check one answer for each row. 4. In the process to add an additional vehicle, does your agency consider vehicle life-cycle cost information as part of a lease versus purchase analysis? 4a. If yes, please describe how, if at all, the above lease versus purchase analysis differs in the case of adding an additional vehicle, and in particular any differences in the type of life-cycle cost information considered in the case of adding a vehicle. (Written responses not included) 5. Excluding the lease versus purchase analysis, does your agency conduct any other vehicle life-cycle cost analysis at any other point in the vehicle replacement process described in Question 1? 5a. Does your agency compare the life-cycle costs of multiple vehicle types prior to selecting a type of vehicle to acquire? 5b. Does your agency perform a cost analysis comparing life-cycle costs of acquiring a non-electric vehicle to costs of acquiring an electric vehicle? 5c. If no, please describe how your agency considers the results of this life-cycle cost analysis—excluding the lease versus purchase analysis. (Written responses not included) 5d. What factors below does your agency consider in this life-cycle cost analysis? Please check one answer for each row. Useful life (number of years it is expected to be used) 6. In the process to add an additional vehicle, does your agency consider vehicle life-cycle cost information at any point outside the lease versus purchase analysis? 6a. If yes, please describe how, if at all, any life-cycle cost analysis described in question 5 differs in the case of adding an additional vehicle, and in particular any differences in the type of life-cycle cost information considered in the case of adding a vehicle. (Written responses not included) 7. Has your agency ever determined that an electric vehicle is the most appropriate vehicle to meet the agency’s needs? 7a. If yes, please provide some examples of those situations and how your agency determined the type of electric vehicle (i.e. electric vehicle, plug-in electric hybrid vehicle, hybrid electric, etc.). (Written responses not included) 8. How important are the following factors when determining whether the vehicles your agency acquires will be alternative fuel vehicles or petroleum fuel vehicles? Mission (The expected function or purpose of the vehicle) Availability of alternative fuel vehicles Other (specify in box below) For agencies that indicated there were other factor(s), we provided an open-ended question that requested a description of the factor(s) and 3 agencies provided descriptions of other factors not shown here. 9. What are the benefits, if any, (including any related to costs, maintenance, environment, safety, federal requirements, etc.) of acquiring and using each of the following types of alternative fuel vehicles relative to petroleum fuel vehicles? 9a. Electric vehicles (EVs) and plug-in hybrid electric vehicles (PHEVs) that use battery power (Written responses not included) 9b. Hybrid electric vehicles (HEVs) powered by an internal combustion engine (Written responses not included) 9c. Flex Fuel Vehicles (FFVs) designed to run on E85 (Written responses not included) 9d. Other alternative fuel vehicles (Written responses not included) 10. What are the challenges, if any, (including any related to costs, maintenance, environment, safety, federal requirements, etc.) of acquiring and using each of the following types of alternative fuel vehicles relative to petroleum fuel vehicles? 10a. Electric vehicles (EVs) and plug-in hybrid electric vehicles (PHEVs) that use battery power (Written responses not included) 10b. Hybrid electric vehicles (HEVs) powered by an internal combustion engine (Written responses not included) 10c. Flex Fuel Vehicles (FFVs) designed to run on E85 (Written responses not included) 10d. Other alternative fuel vehicles (Written responses not included) 11. How many electric charging stations has your agency installed? 12. Has your agency encountered any challenges while trying to site and install electric charging stations? 12a. If yes, what were those challenges and how, if at all, have you been able to overcome them? (Written responses not included) 13. Has your agency encountered any challenges related to acquiring and using alternative fuel vehicles and alternative fuel while trying to meet federal fleet energy requirements, including Executive Order 13693? 13a. If yes, what were those challenges and how, if at all, have you been able to overcome them? (Written responses not included) 14. Has your agency taken steps to prepare for Executive Order 13693’s requirement that 20 percent of all new passenger vehicles be zero emission vehicles or plug-in hybrids by 2020? 14a. If yes, please provide some examples of the steps you have taken. (Written responses not included). 15. Has the availability of alternative fuel vehicles from GSA’s inventory ever prevented your agency from acquiring an alternative fuel vehicle? 15a. If yes, please describe what vehicle you were interested in and why it was not available. (Written responses not included) You asked us to review the costs and challenges related to federal agencies’ meeting the different federal energy requirements for vehicle fleets. This report addresses: (1) how agencies meet fleet energy requirements and how their efforts changed agencies’ fleets and (2) challenges federal agencies faced related to furthering fleet energy goals. The report also includes information on the extent agencies consider life- cycle costs when selecting vehicles. To determine the extent to which federal agencies reported meeting fleet energy requirements and the composition of federal agencies’ fleets, we analyzed data from the Federal Automotive Statistical Tool’s (FAST) database on the composition and fuel use of federal agencies’ fleets from fiscal years 2008 through 2017, the most current data available at the time of our review. Annually federal agencies must submit data on all of their non-tactical vehicles to this database, which the General Services Administration (GSA) and the Department of Energy (DOE) established in 2000 and is used to satisfy statutory and regulatory reporting requirements. We reviewed the data relative to selected statutory requirements and directives that were in effect for fiscal year 2017. Specifically, we analyzed these data to identify the total numbers of alternative fuel vehicles by fuel type and vehicle size in federal fleets and the changes in alternative fuel use during this time period. DOE provided us fleet performance data on the extent to which each of the agencies subject to these federal requirements met requirements or directives to acquire alternative fuel vehicles, use alternative fuel, and reduce petroleum use and per-mile greenhouse gas emissions for fiscal year 2017. In addition, the Environmental Protection Agency (EPA) reported on the extent to which agencies were meeting the requirement to acquire low greenhouse-gas-emitting vehicles for fiscal year 2017, based on the same database. To assess the reliability of these data, we interviewed DOE officials on how the data were checked for accuracy and collected written responses from them on how the data were collected, maintained, analyzed and presented. This assessment included how DOE flags suspicious data, reviews the data, and validates final entries. Based on the information collected, we found the data sufficiently reliable for our purposes of identifying the number of vehicles by type of vehicle and size, and fuel consumed by federal fleets in order to describe how vehicle fleets changed over the past decade. In May 2018, a new Executive Order was issued that revoked a previous Executive Order. The previous Executive Order contained two directives, to acquire zero emission (electric) vehicles and reduce per-mile greenhouse gas emissions by specific targets and specific years. Thus, while the above statutory requirements for fiscal year 2017 remained in effect for fiscal year 2018, the directives related to acquisition of zero emission (electric) vehicles and per-mile greenhouse gas emissions reductions were no longer in effect after May 2018. To understand the different federal energy requirements for vehicles fleets and guidance for agencies to implement them, we reviewed federal statutes, agency rules, and executive orders, and examined DOE and GSA guidance on the various statutory and regulatory requirements and executive orders. For example, we reviewed DOE’s federal fleet management handbook intended for agencies to select and implement strategies to reduce fleet greenhouse gas emissions and use of petroleum, and EPA guidance on how to meet the requirement to acquire low greenhouse-gas-emitting vehicles, among other documents. In April 2019, CEQ and OMB issued implementing instructions for the Executive Order. The implementing instructions emphasized that agencies should follow the statutory requirements that are still in place and annually identify targets for petroleum reduction and increases in alternative fuel use as part of agencies’ Strategic Sustainability Plans. To broaden our understanding of agencies efforts to meet requirements, we also identified five case study agencies—Department of the Interior (Interior), Department of Veterans Affairs (VA), Department of Transportation (DOT), the Army, and the EPA. We selected these case study agencies based on data from the FAST database and their planning documents to represent different sized fleets, a mix of alternative fuel vehicle types, including electric vehicles, and missions with varying vehicle needs. Interior, VA, and Army represented larger fleets, whereas DOT represented medium and EPA small sized fleets. In part, we also chose DOT and EPA to learn about their unique vehicle acquisition processes and plans for acquiring electric vehicles, based on their responses to the survey we conducted, which is described below. From these case study agencies and their sub-agencies, we interviewed agency officials, including fleet managers, to learn their efforts to meet requirements, how they acquired vehicles, and how they managed their fleets. We spoke with these agencies before and after the Executive Order was revoked in May 2018. We also reviewed documents reporting on the extent to which these agencies met fleet energy requirements. The results from the case studies cannot be generalized to make inferences about all agencies. However, we determined that our selection methodology was appropriate for our design and objectives and that this methodology would generate valid and reliable evidence to support our work. To determine any challenges agencies face related to further meeting fleet energy goals, we surveyed 29 federal agencies, and asked them to describe their vehicle acquisition processes, the type of cost analysis done when acquiring an alternative fuel vehicle, and the benefits and challenges of using alternative fuel vehicles. We identified and surveyed agencies that were required to comply with fleet energy requirements and conducted the survey beginning in April 2018. Overall, 31 federal agencies were subject to these requirements in fiscal year 2017; however, as part of our review of Department of Defense (DOD) documentation, we found that its various military departments operate independently and decided to survey Air Force, Army, Marine Corps, and Navy separately. We also excluded the Court Services and Offender Supervision Agency because of the decentralized nature of its fleet and the Defense Agencies within DOD because it was small relative to other DOD agencies. To increase the validity and reliability of our survey, we conducted pretests of the survey with fleet management officials from three federal agencies: VA, Interior, and the Government Accountability Office. We received a 100 percent response rate to our survey. (See app. I for survey results.) To further learn about the challenges of alternative fuel vehicles as well as strategies agencies were using to acquire these vehicles, we interviewed agency officials, including fleet managers, from our five case study agencies, GSA and DOE. In addition, to understand agencies’ efforts to further fleet energy goals and the challenges they faced, we reviewed the Fleet Management Plans and Strategic Sustainability Performance Plans of each agency we surveyed. The strategic sustainability plan is to prioritize agency actions to support the reduction of greenhouse gas emission and other agency wide targets. The fleet management plan is to specifically address how an agency’s fleet will meet its greenhouse gas reduction targets, petroleum reduction targets, and other relevant fleet requirements. We also focused our analysis only on selected types of alternative fuel vehicles. Specifically, we included flex-fuel vehicles, hybrid-electric vehicles, plug-in hybrid electric vehicles, and battery electric vehicles because these represent the most numerous in federal fleets or those with specific acquisition requirements. We obtained vehicle cost information from GSA’s Alternative Fuel Vehicle Guide that lists the costs and specifications of each alternative fuel vehicle GSA offers, and analyzed cost differences based on fuel type. For the purposes of our analysis, we focused on lease costs, not the costs of purchasing a vehicle from GSA, because in fiscal year 2017, 70 percent of agencies battery electric and plug-in hybrid electric vehicles were leased. To analyze and compare petroleum consumption and greenhouse gas emissions, we judgmentally selected a sample of vehicles from GSA’s Alternative Fuel Vehicle Guide and first estimated their annual fuel using DOE’s Vehicle Cost Calculator. We then entered their estimated fuel use into Argonne National Laboratory’s Alternative Fuel Life-Cycle Environmental and Economic Transportation (AFLEET) tool to estimate well to wheel greenhouse gas emissions. To assess the reliability of these tools, we interviewed and collected written responses from DOE officials regarding the source of the data and the values and assumptions used in its calculations. Based on the information collected, we found that they were sufficiently reliable to estimate petroleum consumption and greenhouse gas emissions. We conducted this performance audit from November 2017 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Until May 2018—during the time when the previous administration’s Executive Order was in effect—our case study agencies acquired limited numbers of battery electric and plug-in hybrid electric vehicles with a general understanding that, when the mission need was compatible, acquiring such vehicles was supported by the Executive Order’s requirements in spite of their higher costs compared to a conventional vehicle. As of February 2019, the last time we spoke with agency officials on this issue, agency officials stated that they were uncertain of the effect of the new executive order and would continue to try and meet fleet energy goals until new guidance was issued. This guidance was subsequently issued in April 2019, and emphasized that agencies should focus on the statutory requirements while increasing efficiency, optimizing performance, and reducing waste and costs. Until May 2018, when the previous Executive Order was revoked, agencies were expected to increase their acquisition of battery electric or plug-in hybrid electric vehicles. Specifically, agencies were to acquire “zero-emission” or plug-in hybrid electric vehicles for 20 percent of all new agency passenger vehicle acquisitions by December 31, 2020—and for 50 percent of all new agency passenger vehicle acquisitions by December 31, 2025—in addition to meeting the other various federal fleet requirements. According to Department of Energy guidance on this Executive Order, the targets phased in over time to account for the expected future market availability and cost competitiveness of these vehicles. However, as of fiscal year 2017, GSA officials and several fleet managers also told us that in their experiences leasing electric vehicles, the lower mileage costs of these vehicles had not made up for the significantly higher acquisition cost over the life of the leases, a situation that they described as a challenge to significantly increasing the numbers of such vehicles in their fleets. Three case study agencies described acquiring battery electric and plug-in hybrid electric vehicles—despite the higher costs—largely because of the Executive Order’s requirement. Similarly, 10 of the 29 agencies we surveyed identified “meeting federal requirements” as a key benefit to acquiring electric vehicles. All five case study agencies had acquired small numbers of electric vehicles in light of the Executive Order’s requirements. Agency officials described acquiring these vehicles when their mission and budgets allowed for it. For example, a case study agency with a larger fleet told us that mission needs drove its vehicle acquisitions, and there were limited instances in which an electric sedan would have met the agency’s mission needs. However, when the agency acquired a vehicle for a mission that could be met with an electric vehicle— such as to ferry officials to and from different offices in an area where charging stations were easily accessible—it would have been likely to select an electric vehicle, in part, to help the agency take steps towards meeting the Executive Order’s acquisition goals. Agency officials at four of the five case study agencies said once they had identified an opportunity to acquire an electric vehicle—generally at a location where the mission aligned with the capabilities of an electric vehicle, recharging infrastructure was available, and there were sufficient funds in the budget—they would conduct a lease versus purchase analysis to determine whether leasing or purchasing the vehicle would be most the cost effective option, a key aspect of a life-cycle cost analysis. We have previously reported that a life-cycle cost analysis, which considers vehicle costs from the beginning to the end of vehicle ownership, can help agencies make cost-effective decisions. Officials at the fifth case study agency, Army, stated that the agency had conducted an agency-wide analysis that had determined that leasing was always a better option than purchasing for non-tactical vehicles, and so it no longer conducted this analysis on a vehicle-by-vehicle basis. Officials at our case study agencies stated they did not conduct life- cycle cost analysis to compare and contrast different types of vehicles during the acquisitions process because they considered mission and federal fleet energy requirements to be the key drivers of which type of vehicle to select. However, about half of the agencies that responded to our survey stated that they did do so. Specifically, 14 of 29 agencies indicated they conduct a life-cycle costs analysis outside of a lease-versus-buy analysis when replacing a vehicle, and 13 of these agencies responded that they did such an analysis to compare the costs of an electric vehicle to a non-electric vehicle. Almost all of these agencies responded that they considered initial acquisition cost, fuel cost, electricity consumption, useful life, maintenance costs, and annual miles, with fewer agencies checking that they considered other costs, such as depreciation and disposal costs. As of February 2019, the last time we spoke with agency officials on this issue, agency officials stated that they were unsure of how the revoking of the previous Executive Order and implementation of the new Executive Order would affect the extent to which they acquired electric vehicles in the future. Officials at one case agency stated that with the uncertainty surrounding the requirement to acquire more of these vehicles in the future, it was likely that they would not acquire electric vehicles due to their higher costs. Another case study agency said that although the Executive Order had been revoked, the agency may continue to acquire a limited number of these vehicles in locations where it had already invested funds for electric vehicle infrastructure. In addition to the individual named above, Alwynne Wilbur (Assistant Director); Eric Hudson (Analyst-in-Charge); Ross Gauthier; Bonnie Ho; Malika Rice; Amy Rosewarne; Kelly Rubin; Andrew Stavisky; and Crystal Wesco made key contributions to this report.", "summary": "Since 1988, a series of laws have been enacted and executive orders issued related to federal goals of reducing federal fleets' petroleum use and greenhouse gas emissions. For fiscal year 2017, federal agencies were required to: (1) to acquire certain types of vehicles, (2) to use more alternative fuel, and (3) to meet targets for reducing petroleum and per-mile greenhouse gas emissions. Federal agencies were also under a directive to increase acquisitions of zero emission (electric) vehicles. GAO was asked to review federal agencies' efforts related to these fiscal year 2017 requirements. This report addresses: (1) how agencies reported meeting fleet energy requirements and how agencies efforts changed their fleets and (2) challenges agencies face related to further meeting fleet energy goals. To conduct this review, GAO surveyed 29 federal agencies subject to fleet energy requirements and selected 5 agencies—of a variety of sizes and missions—for case studies. The case studies results are not generalizable to all agencies. GAO also: (1) reported on DOE's and GSA's data on federal fleets for fiscal years 2008 through 2017, including GSA's acquisition and cost data for fiscal year 2017, the most current data available; (2) reviewed DOE's and EPA's information on agencies' performance related to fiscal year 2017 requirements; and (3) interviewed federal officials. The directives to reduce per-mile greenhouse gas emissions and increase acquisitions of electric vehicles were revoked by an Executive Order issued in May 2018. In responding to fleet management requirements over the past 10 years, agencies have incorporated an increasing number of alternative fuel vehicles into their fleets. These have been predominantly flex-fuel vehicles, as hybrid and battery electric vehicles continue to make up a small percentage of agencies' fleets (see figure). The Department of Energy (DOE) is responsible for overseeing agencies' compliance by analyzing fleet data. Most agencies reported meeting the fiscal year 2017 requirements to reduce petroleum use and per-mile greenhouse gas emissions. DOE and other agency officials attributed agencies' success in meeting these requirements to (1) acquiring low greenhouse-gas-emitting and alternative fuel vehicles, and (2) improving general fleet management such as by reducing miles traveled. According to agency officials, three challenges have continued to hinder agencies' efforts to further the goals of reducing federal fleets' petroleum use and greenhouse gas emissions. First, while hybrid and electric vehicles can offer reductions in petroleum use and greenhouse gas emissions, the costs of these vehicles and their charging infrastructure make it challenging for agencies to acquire them on a large scale. According to GSA data, agencies purchased 373 electric vehicles (sedans and minivans) in fiscal year 2017—along with about 4,500 hybrid electric sedans—out of a total of over 16,000 sedans and minivans acquired. In total, agencies spent about $10.5 million more to purchase hybrid or electric vehicles than they would have to purchase comparably sized conventionally fueled vehicles. However, agencies did not consistently track the life-cycle costs of these vehicles. Second, agencies also stated that a lack of fuel and infrastructure availability limits agencies' use of alternative fuel. Third, agency officials stated that a continuing need for larger vehicles limits the number of low greenhouse-gas-emitting vehicles agencies can acquire.", "document_type": "gao"}
{"report": "Confucius Institutes are entities that seek to promote Chinese language and culture in foreign countries. Their establishment is guided by Hanban, which is headquartered in Beijing, China, and, according to various sources, is affiliated with the Chinese government’s Ministry of Education. The first Confucius Institute in the United States was established in 2004, and there were approximately 525 institutes worldwide as of September 2018, according to Hanban’s website. Most Confucius Institutes in the United States are based at colleges and universities. We identified 96 Confucius Institutes in operation at U.S. colleges and universities in 44 states and the District of Columbia as of January 2019. See our February 2019 report on Confucius Institutes for a full list of the schools and their locations. Figure 1 shows U.S. states with one or more Confucius Institute on college or university campuses. Additionally, in recent years, some U.S. universities have partnered with Chinese universities to establish degree-granting institutions in China approved by the country’s government. The Chinese government requires that U.S. universities seeking to establish such an education arrangement in China partner with a Chinese university, and establish written agreements with the Chinese university defining the academics, governance, operations, finances, and other aspects of the arrangement. At the time of our review in August 2016, the 12 institutions we reviewed ranged from fewer than 40 to more than 3,000 students. More than 90 percent of the students across the 12 institutions were Chinese, and less than 6 percent were U.S. citizens. In February 2019, we reported that Confucius Institutes in the United States that we reviewed were established as a partnership between a U.S. school and a Chinese college or university, funded and arranged in part by Hanban. Management of the institutes varies by school. Some Confucius Institutes at U.S. schools are part of an academic department or an administrative office, while others report directly to the school president or other school leadership. Confucius Institute personnel generally consist of a Confucius Institute director or directors, Confucius Institute teachers, and a board of directors. At the 10 case study schools that were part of our review, the Confucius Institute director was a U.S. school employee—either a school administrator, faculty member, or professional hired to manage the Confucius Institute. In addition, several case study schools had a Chinese assistant director, who reported to the Confucius Institute director from the United States, and often was an employee at the Chinese partner university. In February 2019, we reported that officials we interviewed from case study schools stated that Confucius Institutes’ benefits include opportunities for schools to forge international connections and receive funding and other resources for China-related programs. These officials noted that because Hanban pays the salaries of Confucius Institute teachers who teach language and assist with Chinese programs at schools, sparing the schools these costs, these schools could offer Chinese language courses even when enrollment was low. Case study school officials also stated that Confucius Institutes provide valuable resources and opportunities to increase knowledge of and exposure to China and Chinese culture within the school and in the broader community. Case study school officials, researchers, and others we interviewed also offered various perspectives on whether having Confucius Institutes on campuses could bring about undue Chinese influence. These parties discussed the potential for or absence of Chinese interference in events and activities at the institute and on campus. They also expressed views on Confucius Institute teacher hiring, and quality of those teachers. Several school officials, researchers, and others we interviewed expressed concerns that hosting a Confucius Institute could limit events or activities critical of China—including events at the institute and elsewhere on campus. Two officials who expressed these concerns were faculty members at one case study school who have not applied for Confucius Institute funding for a research project because they believed Hanban would not approve of the topic. In contrast, officials at multiple case study schools noted that U.S. school faculty members make all decisions regarding conference themes, guest speakers, and topics for events at their institute. Officials at some schools offered examples of events and activities their Confucius Institute had sponsored that addressed topics that could be considered critical of China. Specifically, they reported hosting a conference discussing intellectual property in relation to China and events on Tibet, territorial disputes in the South China Sea, and religion in China. In addition, multiple researchers and others we spoke with expressed concerns with the Confucius Institute teacher selection process whereby Hanban or the Chinese partner school accepts initial applications from potential Confucius Institute teachers and proposes candidates to the U.S. school. These individuals noted that the Chinese entities could use such a process to effectively screen out candidates based on inappropriate criteria, such as political or religious affiliation. Officials we interviewed at multiple case study schools that had Confucius Institute teachers, however, expressed no concerns about the process for hiring teachers. School officials stated that they believed their school generally controlled the hiring process and were thus satisfied with it. Most officials emphasized that while institute teachers often come from the Chinese partner university, and are referred by the partner or Hanban, the U.S. school makes the final hiring selection. Case study school officials, researchers, and others also suggested ways to improve the institutes, including changing the language in agreements governing Confucius Institutes and policies for sharing these agreements. These parties stated that schools should remove the confidentiality section of their agreements and make the agreements publicly available online. Several researchers and others also emphasized that making the agreements publicly available would dispel questions and concerns over their contents. Several representatives of higher education institutions told us that they believed the confidentiality language in agreements was unnecessary and schools should consider removing it from their agreements. A few case study school officials, researchers, and others we interviewed stated that schools should include stronger language in the agreements to make it clearer that the U.S. school has executive decision-making authority. School officials and others we interviewed suggested other steps that schools could take to ensure they protect against undue Chinese influence. Several school officials stated that the schools should clearly delineate between the Confucius Institutes’ programs and their own Chinese language programs, such as by locating the institute apart from these departments within the school’s organizational structure. A few school officials and others noted that Confucius Institute teachers should not teach credit-bearing courses, even if those courses use curriculum developed by the school’s language department. One school administrator, who stated that his school’s Confucius Institute would never have a Chinese assistant director because the position suggests an excessive degree of Chinese influence, recommended that other schools remove the Chinese assistant director position from their institutes. Officials from two case study schools and others we interviewed stated that schools should organize events through the institute specifically intended to address what some might perceive as a topic sensitive to Chinese interests to demonstrate the school and institute were not subject to undue Chinese influence. In August 2016, we reported that the 12 U.S. universities we reviewed generally reported receiving support for their institutions in China from their Chinese partner universities and from Chinese government entities, with limited funding from U.S. government agencies and private donors. Most universities reported being granted land, resources for construction of buildings, and the use of the Chinese university’s campus facilities. The amount of support reported by the universities varied widely and was in some cases substantial. For example, one university reported receiving nearly 500 acres of land and a commitment from the Chinese provincial and local governments to spend about $240 million for construction and development of facilities. Five of the 12 universities reported receiving federal funding, which in most cases consisted of federal financial aid to U.S. students. At the time of our review, most universities we reviewed included language in their written agreements or other policies that either embodied a protection of academic freedom or indicated that the institution in China will adhere to academic standards commensurate with those at their U.S. campus. Six universities in our review included language in either their written agreements or other university policies that indicated a protection of academic freedom, such as permitting students to pursue research in relevant topics and allowing students to freely ask questions in the classroom. For example, one university’s agreement stated that all members of and visitors to the institution in China will have unlimited freedoms of expression and inquiry and will not be restricted in the selection of research, lecture, or presentation topics. Another three universities’ written agreements included language indicating that the institution in China will adhere to academic standards commensurate with either the U.S. campus or the university’s accrediting agency or other authoritative bodies. Fewer agreements addressed other types of protections at the time of our review. About half of the universities GAO reviewed addressed access to information, such as providing faculty and students with access to physical or online libraries, though a few universities’ agreements and policies include language protecting internet access. Written agreements and policies for about half of the universities we reviewed included language that suggested a protection of at least one of the freedoms of speech, assembly, and religion or worship, though the number of universities addressing each freedom varies. For example, regarding freedom of speech, student and faculty handbooks at a few of these universities contained language indicating that students have the ability to discuss sensitive topics. Regarding freedom of religion or worship, several of the universities included language in their policy documents indicating that religious practices will be protected. The more than 130 faculty and students we interviewed from universities’ institutions in China during our 2016 review generally reported that academic freedom had not been restricted. Faculty told us they did not face academic restrictions and could teach or study whatever they chose. For example, several faculty members asserted that neither they nor their colleagues would tolerate any academic restrictions, and one faculty member told us he and his colleagues intentionally introduced class discussions on politically sensitive topics to test whether this would trigger any complaints or attempted censorship. Students also generally indicated that they experienced academic freedom and could study or discuss any topic. Some students who had also studied or knew others who studied at Chinese universities contrasted their experiences at a U.S. institution in China, noting that they could have interactive dialogue with faculty, discuss sensitive topics, and freely access information at the U.S. institution but not at a Chinese university. Through interviews and responses to our questionnaire, university administrators reported that academic freedom was integral to their institutions in China. Administrators at several universities told us that academic freedom was nonnegotiable, while others noted that the same curriculum used in the United States also applied to their institution in China. However, fewer than half of the universities we reviewed had uncensored internet access at the time of our review. We visited universities with and without uncensored internet access, and observed university members accessing search engines, newspapers, and social media sites that have been blocked in China—such as the New York Times, Google, and Facebook—at some universities but not others. At several universities that lacked uncensored internet access, students and faculty told us that, as a result, they sometimes faced challenges teaching, conducting research, and completing coursework. For example, one faculty member told us that she sometimes asked others outside of mainland China to conduct internet research for her because they can access information she could not. Several students at another university told us their ability to conduct academic research was constrained by the internet limitations. We also reported in August 2016 that additional factors that could create obstacles to learning at U.S. universities in China, including self- censorship and constraints specific to Chinese students. Administrators, faculty, and students representing more than half of the universities we reviewed gave examples of self-censorship, including some cases where individuals were advised by teachers or others in positions of authority to avoid certain topics. For example, an administrator at one university noted that he believed it was advisable, as a guest of China, to refrain from insulting China, while an administrator at another university noted that the university advised teachers to avoid discussing sensitive subjects in class. In addition, we found that some conditions specific to Chinese students may constrain their academic experience. For example, some noted that Chinese students may know or suspect that their Chinese classmates are government or Communist Party monitors and will report on whatever the students say. An administrator at one university told us that he assumed there were Chinese students and faculty in the institution who reported to the government or the Communist Party about the activities of other Chinese students. Faculty members at several universities told us that they understood there were Chinese students in class who intended to report on the speech of faculty or Chinese students. Finally, we also observed that three of the 12 universities we reviewed that were approved by the Chinese Ministry of Education as having independent legal status shared characteristics that may be correlated with greater academic and other freedoms on campus. We found that these three universities had campuses built specifically for the joint institution that were located relatively far away from their Chinese university partner’s campus, generally controlled their own day-to-day operations, had uncensored internet access, and offered extensive campus and student life programs. In contrast, the other nine universities we reviewed did not consistently share these characteristics at the time of our review. Chairman Portman, Ranking Member Carper, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Jason Bair, Acting Director, International Affairs and Trade at (202) 512-6881 or bairj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Joseph Carney (Assistant Director), Caitlin Mitchell (Analyst in Charge), Joyce Kang, Neil Doherty, Melissa Emrey-Arras, Meeta Engle, Elizabeth Repko, Aldo Salerno, Michael Silver, and Nicole Willems. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Numerous U.S. universities and colleges have partnered with Chinese entities to establish (1) Confucius Institutes in the United States and (2) degree-granting institutions in China. Confucius Institutes are partnerships between Chinese entities and schools in other countries, arranged and funded in part by Hanban, which seek to promote Chinese language and culture. There were 96 institutes located at colleges and universities in the United States as of January 2019. U.S. universities have also partnered with Chinese universities to establish degree-granting institutions in China approved by the Chinese government. School officials have noted these types of partnerships provide valuable educational, cultural, and other benefits. Some researchers, government officials, and others, however, have raised concerns about them, including about the contents of written agreements and the role of the Chinese government, which, according to the Department of State, has made efforts to restrict academic freedom and impose censorship at Chinese universities and other institutions. Some have expressed concern that U.S. universities partnering with the Chinese government may face similar restrictions. This testimony discusses funding, agreements, and operations of (1) Confucius Institutes in the United States and (2) U.S. universities in China. This testimony is based on GAO's February 2019 report on Confucius Institutes in the United States and GAO's August 2016 report on U.S. universities in China. GAO reviewed 90 agreements establishing Confucius Institutes and spoke to officials about benefits and concerns related to the institutes. Agreements between Hanban—an affiliate of the Chinese Ministry of Education—and U.S. colleges and universities generally describe similar activities, funding, and management, though institute operations vary in practice. Confucius Institutes receive funding from Hanban and U.S. schools, and do not receive direct federal funding. While 42 of 90 agreements contained language about the document being confidential, some were available online or upon request, and one-third of the 90 agreements explicitly addressed how U.S. school policies apply to the institutes. Officials GAO interviewed at 10 case study schools noted U.S. school policies apply to institutes at their schools. GAO also interviewed some researchers and others who expressed concern that the presence of Confucius Institutes could constrain campus activities and classroom content. For example, several suggested schools with institutes might avoid hosting events on topics that could include criticism of China, such as Taiwan or Tibet, so as to not offend Chinese partners. School officials offered examples to illustrate that these concerns did not apply to their institute, noting institutes had sponsored events on such topics. Nonetheless, school officials and others suggested ways schools could improve institute management, such as renegotiating agreements to clarify U.S. schools' authority and making agreements publicly available. In August 2016, GAO reported that U.S. universities that have partnered with Chinese universities to establish degree-granting institutions in China emphasize academic freedom, but face internet censorship and other challenges. The 12 U.S. universities GAO reviewed generally reported receiving support for their institutions in China from Chinese government entities and universities, and 5 reported receiving U.S. government funding, mostly federal financial aid to U.S. students. Universities' agreements with Chinese partners or other policies GAO reviewed generally included language protecting academic freedom or indicating their institution in China would adhere to U.S. standards. University members generally indicated that they experienced academic freedom, but also stated that internet censorship, self-censorship, and other factors presented constraints. At several universities that lacked uncensored internet access, faculty and students noted that, as a result, they faced challenges teaching, conducting research, and completing coursework at that time.", "document_type": "gao"}
{"report": "There were 445 State/INL and USAID Mérida Initiative projects active from fiscal year 2014 through fiscal year 2018, which includes some projects that started before this period and some that continued after this period. State/INL funded 388 of the projects, and USAID funded 57. USAID’s projects tended to be larger with higher funding amounts than State/INL projects. State/INL projects generally focused on providing training and assistance to Mexican officials from the justice sector, border security, military, and law enforcement, as well as equipment, including for forensic laboratories, drug detection, and border surveillance. USAID projects were intended to engage with Mexican government institutions, civil society organizations, and the private sector to address corruption, promote trust in government, or prevent crime and violence, such as through skill building for youth, efforts to advance human rights, or technical support for judicial system development. State/INL allocated about $542 million and USAID allocated about $182 million for assistance to Mexico under the Mérida Initiative from fiscal year 2014 through fiscal year 2018. State/INL and USAID are the lead U.S. agencies for developing the Mérida Initiative’s programming. In these roles, State/INL and USAID work with Government of Mexico officials to outline plans, goals, and objectives for Mérida Initiative projects. State/INL and USAID both manage and fund the Mérida Initiative with the support of a wide range of project implementers, including DOJ, DHS, and DOD, as well as private contractors, nongovernmental organizations, and international organizations. State/INL and USAID implement Mérida Initiative projects primarily through contracts, grants, and agreements with international organizations. State/INL also implements some Mérida Initiative projects through interagency agreements with other U.S. agencies (e.g., DOJ, DHS, and DOD). State/INL and USAID contracting, grant, and agreement officers, are responsible for administering and overseeing contracts, grants, and other agreements that the agencies award, including for Mérida Initiative projects. They delegate the day-to-day monitoring responsibilities to agency officials located in Mexico City, particularly State/INL and USAID Contracting Officer Representatives (COR) for contracts, State/INL Grant Officer Representatives (GOR) for grants, State/INL Agreement Officer Representatives (AOR) for interagency agreements or letters of agreement with international organizations, and USAID AORs for grants and cooperative agreements, according to agency officials. Key monitoring responsibilities of the CORs, GORs, and AORs typically include reviewing quarterly, annual, and other progress reports submitted by project implementers; ensuring other required documents are submitted; communicating with the implementers on the status of assistance activities; and conducting site visits, among other things. In 2019, we reported on 14 leading practices for monitoring foreign assistance that agencies should incorporate in their monitoring policies to help ensure that they effectively manage foreign assistance, address impediments, and meet their assistance goals. From these leading practices we derived eight key practices that can help agencies monitor the implementation and performance at the project level. To facilitate discussing these key monitoring practices, we grouped them into three areas: (1) assigning monitoring duties to qualified staff, (2) planning monitoring approach, and (3) monitoring project implementation. (See table 1.) These practices are generally consistent with the Office of Management and Budget’s guidelines for Federal departments and agencies that administer United States foreign assistance and related guidance, as well as State’s and USAID’s monitoring policies. We reviewed 15 of State/INL’s high–dollar value Mérida Initiative projects to assess the extent to which State/INL followed key practices for monitoring foreign assistance projects in the areas of assigning monitoring duties to qualified staff, planning a monitoring approach, and monitoring project implementation. For these projects, the agency generally followed the key practices about half of the time, as shown in figure 1, and for a subset of four selected projects, it did not consistently track performance data or compare them to established performance measures. State/INL does not have procedures in place for monitoring staff to consistently follow all the key practices. Instead, officials said they focused on tracking implementation of the projects’ activities. Consistently following key monitoring practices would allow State/INL to stay well informed of projects performance, take corrective action when necessary, and help ensure that projects achieve their intended results. State/INL generally followed key practices for assigning monitoring duties to qualified staff almost always. Assigning staff with the appropriate certification helps ensure that they have the necessary knowledge and skills to perform those duties. Establishing roles and responsibilities helps ensure that the assigned monitoring staff are aware of their monitoring duties. State/INL requires that staff responsible for monitoring Mérida Initiative projects be certified as a COR, GOR, or AOR. State/INL also assigns roles and responsibilities to monitoring staff through a designation letter in which a contract or grant officer designates a COR, GOR, or AOR to oversee each project. However, of the 15 projects we reviewed, one had a gap in the documentation for staff certifications, and four had gaps in the documentation of designation letters. For example, in one case State/INL could not provide documentation to demonstrate that the official responsible for monitoring a project on police training had been officially designated or that the official had a valid certification during the full implementation period of the project. According to State/INL staff, the monitoring staff roles and responsibilities are also outlined in other documents such as the State Department’s Foreign Affairs Manual and the AOR Handbook, of which staff are expected to be aware. Figure 2 illustrates the extent to which State/INL followed each related key practice for assigning monitoring duties. State/INL generally followed key practices for planning a monitoring approach a third of the time. Two projects—one for helicopter pilot training and the other for aviation maintenance training—did not have monitoring plans and thus did not meet any of the three key practices for planning a monitoring approach. According to a State/INL manager, State/INL is no longer working with this implementer due to long-standing difficulties in obtaining documentation needed to monitor the projects. Most of the other 13 projects partially met the key practices for planning a monitoring approach. For example, goals and objectives were included in planning documents other than the monitoring plan. Furthermore, while only three of the projects had a monitoring plan that addressed risk, we determined that 10 of the projects partially addressed this key practice, because risks were assessed or considered, but the identified risks were not addressed in the monitoring plan. In addition, almost all of the projects had relevant project-level performance measures. Developing a monitoring plan that identifies project objectives helps focus monitoring efforts on assessing projects outcomes. In addition, identifying and addressing risks in that plan helps focus monitoring efforts on those aspects of project implementation that are most likely to threaten the success of the project in meeting its goals. We did not see evidence that State/INL had procedures in place to ensure that monitoring officials consistently follow key practices in the area of planning monitoring approach. Figure 3 illustrates the extent to which State/INL followed each related key practice to planning a monitoring approach. State/INL provided documentation to demonstrate that monitoring managers generally followed key practices for monitoring project implementation about half of the time. Monitoring project implementation helps ensure that projects are meeting their objectives, so that any necessary adjustments or corrective actions can be taken in a timely manner. We found that State/INL did not generally collect all expected progress reports from implementers for seven projects, and of those seven, it did not collect any reports for three projects. Furthermore, State/INL did not provide documentation for eight projects demonstrating that monitoring staff had generally assessed and approved implementers’ periodic progress reports. We also found that for seven projects, State/INL did not provide documentation demonstrating that monitoring staff had generally conducted site or field monitoring visits or taken other steps to validate the partner’s performance implementing the project. For example, for one project that provided training to Mexican immigration officers on the southern border, State/INL only provided one quarterly progress report of the four we requested for the period of our review. For this project, State/INL also did not provide documentation that monitoring staff had taken steps to review and approve the report or that they had conducted any monitoring site visits. A State/INL official explained that they requested the quarterly reports, but at times implementers did not submit them. Without implementing procedures to consistently collect, assess, and approve performance reports from implementers, monitoring staff may not have sufficient information to assess implementers’ performance and determine whether corrective actions are needed. We did not see evidence that State/INL had procedures in place to ensure that monitoring officials consistently follow key practices in the area of monitoring project implementation. Figure 4 illustrates the extent to which State/INL followed each related key practice for monitoring project implementation. State/INL monitoring officials did not consistently track performance data against established measures for four Mérida Initiative projects we reviewed; these four projects were a subset of the 15 State/INL projects discussed above. Tracking performance data—a key practice for monitoring project implementation—can provide meaningful information on projects’ progress in achieving intended results. The four projects we reviewed included two grants focused on police professionalization; one interagency agreement focused on assistance to forensic laboratories; and one agreement with an international organization focused on conducting a survey on police standards, training, and professionalization. We reviewed how State/INL tracked performance data for these selected projects as part of its efforts to assess and approve implementing partners’ periodic performance reports and data as outlined in the key monitoring practices. Specifically, we analyzed the extent to which State/INL tracked data contained in quarterly progress reports and compared these data to established performance measures. State/INL and the project implementers outlined these performance measures in monitoring documents that these implementers developed and State/INL approved. Some of these projects’ monitoring documents also included data sources, data collection frequency, and performance targets. State/INL did not track performance data for two of the four selected projects and tracked such data inconsistently for the other two selected projects. As a result, State/INL cannot ensure that it has accurate and reliable performance data for its Mérida Initiative projects. Such data could help State/INL determine whether projects are achieving intended results and take necessary corrective actions to improve project performance over time. For the two police professionalization projects we reviewed, State/INL did not track performance data against established performance measures outlined in the project narrative at the start of the projects. Some of these projects’ performance measures reflected outputs—such as the number of participants completing at least 25 hours of police training and the number of citizen surveys conducted on public trust of law enforcement. Other performance measures reflected outcomes—such as the percentage of law enforcement officials who feel ready for promotion after completing training and results of citizen surveys on perceived security where law enforcement trainings were conducted. (See examples in table 2.) However, State/INL did not clearly track or reference such performance measures in these two projects’ quarterly progress reports. Instead, State/INL provided details in these reports on project activities and training that did not clearly link to the projects’ performance measures. For example, State/INL noted the number of participants who took a specific training course on a certain date, but did not provide the total number of participants’ training hours to compare them to the performance measure on the total number of participants who completed at least 25 hours of training. State/INL monitoring officials said they had not systematically tracked data on the performance measures of these projects over time, but instead focused on ensuring the trainings were conducted and the number of training participants were tracked. These officials acknowledged the need to improve their tracking of these projects’ progress against their performance measures. We also identified information in quarterly progress reports for two projects suggesting that the reports did not accurately reflect project activities in those quarters. For example, for one project, State/INL included identical information in two separate quarterly reports even though the implementer conducted different project activities in those two quarters. Thus, at a minimum, the information in one of the quarterly reports did not accurately reflect the project’s activities conducted in that quarter. We found the same issue with another project’s reports. State/INL officials said they were not aware that the project information in these reports were identical. For the two other State/INL projects we reviewed (one forensics laboratory accreditation project and one police survey project), State/INL tracked some performance data but did so inconsistently. These projects’ performance measures reflected outputs, such as the number of survey pollsters hired and trained and the number of accredited forensic laboratories that maintain their accreditation. Other performance measures reflected outcomes, such as the percentage of forensic laboratories trainees reporting improved knowledge of subject matter and satisfaction rates for training courses for the forensics laboratory project. (See examples in table 3.) In one of these two projects’ quarterly reports, the project implementers inconsistently described and numbered some of the performance measures, and they did not explain the discrepancies. Also, the implementers mentioned different performance measures in different quarterly progress reports—with some measures dropping off in some quarters and new measures appearing in others—without providing a rationale in the reports. As a result, State/INL could not consistently track progress of some of the performance measures over time. State/INL officials stated that these two implementers only included activities in the quarterly reports that they conducted in that quarter, which would result in different and inconsistent performance measures in each report. In addition, some of the reported project activities did not consistently and clearly align with the performance measures to allow State/INL to track the project’s progress against these measures. For example, some performance measures reflected percentages (e.g., 90 percent of authorities responsible for forensic laboratories routinely attend regional and national conferences), but the report listed the names of conference participants, dates, and locations in a table next to that performance measure. When asked about these discrepancies, State/ INL officials said that they did not ensure that implementers provided complete information to clearly track the project’s progress against performance measures. However, they said that they also conduct monitoring through informal methods not documented in the progress reports, such as through communication via phone calls and emails with the implementers. Such informal methods do not provide State/INL with the necessary data to assess a project’s performance against its goals. For the four State/INL projects we reviewed, State/INL monitoring managers did not establish procedures to collect and review project performance data, such as the number of people who completed a targeted number of hours of training, or the results of training surveys. These managers said they did not prioritize establishing performance tracking procedures and instead focused on the implementation of the projects’ activities, such as counting the number of participants who attended one training course for a particular month. For example, while some monitoring staff sent monthly emails to their managers describing project activities, State/INL monitoring managers did not establish procedures—such as holding regular meetings with or requiring reporting from monitoring staff—that focused on tracking the projects’ progress against established performance measures. State/INL receives activity data from project implementers that it considers useful in helping the agency monitor the projects’ implementation and activities. State/INL officials told us that project activity data in the quarterly progress reports—such as when trainings were conducted and how many people attended—help keep them informed of and monitor the projects’ implementation. In addition, since 2015, State/INL Mexico has collected detailed data and information in tracking databases on (1) training events and related surveys on that training, and (2) forensic laboratory accreditations and correctional facility accreditations. The training tracking database contains data on over 6,000 training events, 100,000 trainee records, and over 20,000 survey responses from training event participants. This database can generate numerous reports covering the number of people who completed a specific trained course, which training courses a specific person completed, training survey results, and which implementer conducted the training, among other information. State/INL databases also collect information on the status of forensics laboratories and correctional facilities across Mexico that are being accredited through Mérida Initiative projects. The forensics database includes pages for each laboratory with detailed information about the level of accreditation received, and types of trainings conducted, among other things. The correctional facilities database is structured similarly to the laboratories database with pages for each facility with detailed information on accreditation status and timeline, among other things. According to State/INL officials, like the training tracking system, the forensics and correctional facilities databases can generate reports, such as monthly progress reports. Finally, State/INL Mexico is implementing a new cloud-based monitoring database—called DevResults—that will consolidate and track data on activity, output, and outcome indicators for all Mérida Initiative projects. According to State/INL officials, they implemented DevResults so that State/INL could track a project’s progress and trends in real time against its performance goals. According to State/INL officials, DevResults included data for 84 projects as of February 2020. They also noted that agency officials and implementers have completed training on DevResults, and additional training will be provided as needed. State/INL officials said they plan to continue adding data for past and present Mérida Initiative projects in 2020. We reviewed five of USAID’s Mérida Initiative projects to assess the extent to which USAID followed key monitoring practices in the areas of assigning monitoring duties to qualified staff, planning a monitoring approach, and monitoring project implementation. For these projects, USAID almost always followed key practices—as shown in figure 5—and for a subset of two selected projects, it consistently tracked project performance. According to USAID officials, USAID management conducted periodic portfolio reviews to ensure that monitoring staff adequately monitored Mérida Initiative projects and followed key practices. However, for all five USAID projects we reviewed, monitoring plans did not address identified risks, which could help the agency allocate monitoring resources to those aspects of the projects that warrant closer scrutiny. USAID generally established roles and responsibilities for technical staff responsible for monitoring projects, but for two of the five projects we reviewed it did not maintain documentation showing that it assigned staff with appropriate certifications. Like State/INL, USAID requires that staff responsible for monitoring Mérida Initiative projects be certified as CORs or AORs, which typically includes periodic training in monitoring projects. USAID assigns roles and responsibilities to these staff through a designation letter in which a contract or agreement officer designates a COR or AOR, respectively, to conduct technical oversight of each project. For the five projects we reviewed, USAID properly designated monitoring roles and responsibilities to technical staff, however there were gaps in staff certification documentation for technical staff for two projects. For example, we found that the person responsible for monitoring a project promoting justice reform and rule of law in Mexico did not have a valid certificate for 9 months of the project’s 4-year period of performance. Maintaining complete documentation of monitoring-related activities helps USAID management ensure adequate, continuous monitoring of projects. According to USAID, the gaps in documentation were caused by staff turnover and trouble accessing the government-wide system for recording the certification of staff, which was difficult to access or down from December 2017 to March 2018. Officials said that once the system to record certificates was brought back online, they were able to track certifications. Figure 6 illustrates the extent to which USAID followed each related key practice for assigning monitoring duties. USAID generally developed monitoring plans that included program goals and objectives and project-level performance measures, but the monitoring plans did not address project risks. All five projects generally had a monitoring plan that identified project goals and objectives, and relevant project-level performance measures. However, none of the monitoring plans generally addressed identified risks related to achieving project objectives. While USAID provided documentation showing that the agency had conducted various assessments considering risk for each project, the results of these assessments were not addressed in the projects’ monitoring plans. For example, for a project to promote justice and rule of law in Mexico, USAID assessed risks relating to terrorism, environmental effects, sustainability, and gender equity in carrying out the project. However, the project’s monitoring plan did not address identified risk levels and related monitoring actions designed to mitigate risks identified in these assessments. USAID explained that they address ongoing monitoring of risk through several other processes, such as project design, procurement actions, financial management, award management and administration, semi-annual project portfolio reviews, and annual risk-based assessments of the USAID’s portfolio in Mexico, among others. However, identifying and addressing risks in the monitoring plan can help ensure that monitoring staff are aware of potential impediments to project success about which they need to be vigilant or take steps to mitigate as they monitor the projects. Additionally, determining which activities warrant greater oversight can also help agencies manage monitoring resources cost effectively. Figure 7 illustrates the extent to which USAID followed each related key practice for planning a monitoring approach. USAID generally followed key practices for monitoring project implementation about two-thirds of the time. We found that USAID collected all progress reports for four of the five projects we reviewed. For two projects, USAID did not provide documentation demonstrating that monitoring staff had generally assessed and approved implementers’ periodic progress reports. For all five projects, USAID provided documentation demonstrating that monitoring staff had generally validated implementing partners’ performance through site visits. Figure 8 illustrates the extent to which USAID followed each related key practice for monitoring project implementation. USAID monitoring officials consistently tracked performance data and compared them to established performance measures for the two projects we reviewed; these two projects were a subset of the five USAID projects discussed above. To review the extent to which USAID assessed and approved implementing partners’ periodic reports and data—one of the eight key monitoring practices—we determined whether USAID tracked performance data contained in quarterly or annual progress reports. USAID funds one of the two projects through a cooperative agreement focused on strengthening human rights, and the other project through a contract focused on improving the criminal justice sector. USAID and project implementers outlined these projects’ performance measures in project-specific monitoring plans that both parties developed at the start of the project or revised after the project was in place. Project implementers developed these plans, and USAID approved them. The plans included details related to the performance measures, such as data sources, data collection frequency, and targets. In accordance with these plans, USAID and project implementers tracked performance measures in annual progress reports, while they primarily tracked detailed project activity in quarterly progress reports. The two USAID projects’ progress reports included tables that tracked project performance. Some of the projects’ performance measures reflected outcomes, such as prosecution rates of Mexican government prosecution units that received technical support and the number of improved measures to address serious human rights violations. Some performance measures reflected outputs, such as the number of Mexican officials trained in human rights advocacy areas. See table 4 for examples of performance measures and information in the progress reports we reviewed. When the implementer and USAID changed performance measures, they also revised project-specific monitoring plans to document these changes. For example, for one project we reviewed, the established measures were no longer effective in measuring progress toward the project’s objectives, according to USAID officials. As a result, the implementer and USAID modified the project’s monitoring plan at least twice, revising the performance measures to better align with the project’s objectives. The subsequent progress reports we reviewed for these projects included data on the revised performance measures. USAID has procedures to help ensure that monitoring staff track performance data. According to USAID officials, USAID began sending out a standard spreadsheet to all Mérida Initiative implementing partners in 2018 that requires them to report performance data on a quarterly or annual basis. USAID uses these spreadsheets to track Mérida Initiative project performance data. Since May 2017, USAID has also conducted 6- month portfolio reviews in which monitoring managers and their staff review project activities and performance data collected for their projects and discuss project successes and challenges. USAID managers told us that they implemented these reviews to help ensure that their staff monitor project performance. According to State/INL, the Government of Mexico provides data to State/INL that help the agency monitor its Mérida Initiative assistance efforts and provides insights into the implementation of the initiative overall. State/INL also noted that, in 2014, the agency hired a contractor to work with both the U.S. and Mexican governments to develop a comprehensive set of indicators to evaluate the progress and results of the Mérida Initiative. In 2015, Mexico agreed that it would provide data to State/INL on this set of indicators to demonstrate the effects of the Mérida Initiative, according to State/INL officials. These officials told us that they try to obtain the data on an annual basis. They also noted that the purpose of collecting the data from Mexico was to establish a mechanism to share information on the Mérida Initiative’s effects and to improve U.S.- Mexico cooperation on the initiative. According to State/INL officials, various Mexican agencies collect the data, such as the Army, Air Force, Navy, Tax Administration Service/Customs, Attorney General’s Office, and National Institute of Statistics and Geography. The Mexico data comprise about 170 indicators (data points) related to the overall goals and program areas of the Mérida Initiative: Counternarcotics/Special Investigations, Criminal Prosecutions, Border Security and Ports of Entry, and Security and Law Enforcement. Some data are closely linked to Mérida Initiative–funded projects, such as the accreditation status of Mexican correctional facilities. Other data provide broader context, such as Mexican civil society’s perception of Mexican agencies. In addition, data, such as the number of accredited forensic laboratories and correctional facilities, may reflect progress in institution building. Other data, such as the number of accounts blocked by the Mexican Financial Intelligence Unit, may reflect operational capacity development. See table 5 below for examples of the indicators, as reported by Mexico to State/INL. State/INL officials said they use the indicator data in discussions with Mexican officials to help monitor the implementation and activities of the Mérida Initiative, including which best practices can be replicated across Mexico. State/INL officials said the data also inform the agency’s internal decision making on which Mérida Initiative programs are effective and which programs it should modify. For example, according to State/INL officials, the indicator data help track the use of equipment donated to Mexico through the Mérida Initiative. If the data show extensive use of equipment, State/INL can use the data to justify a request for additional equipment or to approve maintenance of the equipment, according to agency officials. For over a decade, the Mérida Initiative has funded programs intended to address serious challenges to security and the rule of law. As the United States continues to support hundreds of Mérida Initiative projects in Mexico, it is important that State/INL monitor these projects carefully and stay well informed of the projects’ performance to ensure that they are as effective as possible. USAID has established procedures that help ensure that it follows most key monitoring practices, including those related to assigning monitoring duties to qualified staff and monitoring project implementation. State/INL management has not established such procedures for the projects we reviewed, limiting its ability to stay well informed of project performance and make course corrections to improve performance when necessary. While State/INL and USAID often conducted assessments to identify risks that may affect the achievement of project objectives, they generally did not address the results of the risk assessments in projects’ monitoring plans. Developing monitoring plans to address risks would help establish the appropriate level of oversight needed for each project, which in turn could lead to more cost-effective management of these projects. We are making the following two recommendations, one to State and one to USAID: The Secretary of State should ensure that State/INL establishes procedures that verify that monitoring officials for Mérida Initiative projects follow the key practices. (Recommendation 1) The USAID Administrator should establish procedures to ensure that monitoring officials for Mérida Initiative projects develop monitoring plans that address risks. (Recommendation 2) We provided a draft of this report to State, DOD, DHS, DOJ, and USAID for review and comment. Some of the agencies provided technical comments, which we incorporated as appropriate. State and USAID also provided formal comments, which are reproduced in appendixes III and IV. State agreed with our recommendation to establish procedures for staff monitoring Mérida Initiative projects to follow key practices. State indicated that it is working to create new monitoring and evaluation guidance consolidated across State/INL, based in part on GAO’s leading practices. According to State, the new guidance will address the areas highlighted in this report related to monitoring Mérida Initiative projects. State/INL plans to institute annual program reviews in which monitoring staff will assess project performance, effects, and alignment with current and planned priorities. State indicated that annually reviewing State/INL programming will help identify underperforming projects, give relevant staff a forum to discuss any issues or challenges to implementation and monitoring, and ensure the bureau follows the key monitoring practices outlined in this report. USAID also agreed with our recommendation to establish procedures to ensure that staff monitoring Merida Initiative projects develop monitoring plans that address risk. USAID indicated that USAID/Mexico is revising its Project and Activity Design Mission Order to incorporate recently issued USAID guidance and address our recommendation. According to USAID, the mission order will provide a framework and guidance to ensure that USAID/Mexico systematically addresses project risks and incorporates them into the respective monitoring plan. We are sending copies of this report to the appropriate congressional committees, the Secretary of State, and the USAID Administrator. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2964 or GurkinC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made contributions to this report are listed in appendix V. This report (1) examines the extent to which the Department of State (State), Bureau of International Narcotics and Law Enforcement Affairs (State/INL), follows key practices in monitoring Mérida Initiative projects and tracks project performance data against established measures; (2) examines the extent to which the United States Agency for International Development (USAID) follows key practices in monitoring Mérida Initiative projects and tracks project performance data against established measures; and (3) describes how State/INL uses data from the Government of Mexico to help monitor the implementation of Mérida Initiative projects. To address these objectives, we reviewed relevant State and USAID agency documents and interviewed agency officials from the Departments of State (State), Homeland Security (DHS), Defense (DOD), and Justice (DOD), and USAID in Washington, D.C., and officials from State and USAID in Mexico City. In 2019, we reported on 14 leading practices for monitoring foreign assistance that agencies should incorporate in their monitoring policies to help ensure that they effectively manage foreign assistance, address impediments, and meet their assistance goals. From these leading practices, which are focused on a high-level assessment of agency monitoring policies, we derived eight key practices that can help agencies monitor the implementation and performance at the project level, such as those implemented under the Mérida Initiative. These eight key practices include those that in our judgment directly relate to monitoring project- level performance activities. We did not address monitoring of financial activities, because our review focused on performance monitoring. We made minor modifications to the key practices selected to reflect the focus of our review. We also grouped the selected key monitoring practices into three areas: (1) assigning monitoring duties to qualified staff, (2) planning a monitoring approach, and (3) monitoring project implementation. To determine the extent to which State/INL and USAID followed key practices in monitoring Mérida Initiative projects, we selected a nongeneralizable sample of 15 high–dollar value State/INL projects and five high–dollar value USAID projects that started between January 1, 2014, and December 31, 2016. (See app. II for details on these 20 projects). Some of these projects were ongoing after fiscal year 2019. We selected the projects from a list provided by State/INL and USAID. State’s list included 388 projects, and USAID’s list included 57 projects for a total of 445 projects under the Mérida Initiative. We selected projects implemented through a variety of mechanisms. For State/INL, we selected two letters of agreement with international organizations, four grants, three contracts, and two interagency agreements implemented by DOD, two interagency agreements implemented by DHS, and two interagency agreements implemented by DOJ. For USAID, we selected two contracts and three grants. The value of the 15 State projects in our sample is about $88 million, and the value of the five USAID projects in our sample is about $107 million. These 15 State/INL projects represent about 25 percent of the total value of the State/INL projects that started during this period. These five USAID projects were the highest value contracts and grants cooperative agreements and represent about 70 percent of the total value of USAID projects that started during this period. Because State/INL implements about 90 percent of all Mérida Initiative projects, we chose a larger State/INL sample than USAID sample. We assessed the agencies’ monitoring of the 20 selected Mérida Initiative projects against eight key monitoring practices largely derived from GAO’s Leading Practices for Monitoring Foreign Assistance. We reviewed documents to determine the extent to which State/INL and USAID followed the eight key monitoring practices for each of the selected Mérida Initiative projects. Specifically, for each selected project, we requested monitoring plans; work plans; risk assessments; Contract, Grant, or Agreement Officer Representative Certificates; Contract, Grant, or Agreement Officers Representatives Designation Letters; implementer progress reports for the latest year of activity of each project (at the time of our review); samples of field or site visit reports; and samples of monitoring emails between monitoring staff and the implementers. We reviewed available documents as they related to each key practice to determine the extent to which the agency had taken steps to follow and document the key practice for each project. On the basis of our review, we assessed whether the key practices were “generally followed,” “partially followed,” or “not followed.” We rated the extent to which the agency followed each key practice as “generally followed” if we received evidence that all critical elements of the key practice were conducted and documented to a large or full extent, “partially followed” if we received evidence that some but not all critical elements of the key practice were conducted and documented, and “not followed” if we did not receive evidence that any of the critical elements of the key practice were conducted and documented. To perform these analyses, two analysts reviewed the documents to rate the extent to which each key practice was met. The analysts worked iteratively, comparing notes and reconciling differences at each stage of the analysis. In addition, GAO staff independent of the two analysts reviewed the final analysis, and modified it as appropriate. To determine the extent State/INL and USAID track project performance, we chose a nongeneralizable subset of the 20 projects listed above. Specifically, we chose six projects—four State/INL projects and two USAID projects—primarily based on their high–dollar values. (See app. II for details on these six projects.) We chose a small subset of State/INL and USAID projects to conduct a detailed analysis of data in the projects’ annual and quarterly reports. Specifically, for the four State/INL projects, we chose high–dollar value projects for each of the following implementing mechanisms: grants, interagency agreements, and agreements with international organizations. We excluded contracts from the State/INL subset sample, because the high–dollar value contracts generally did not have the project-level performance measures needed to assess State’s tracking of performance data. We included a second grant in our sample in place of a contract, because more Mérida Initiative State/INL projects are grants than interagency agreements or agreements with international organizations. As a result, our State/INL sample consisted of two grants, one interagency agreement, and one agreement with an international organization. For the USAID sample, we chose one grant or cooperative agreement and one contract. We did not choose other types of implementing agreements because grants/cooperative agreements and contracts comprise over 98 percent of USAID projects for the timeframe of our review. For both the State/INL and USAID selected projects, we reviewed project monitoring documents—such as project narratives, workplans, and monitoring plans—and identified the performance measures outlined in these documents for each project. We then reviewed these projects’ latest year of implementer quarterly and annual progress reports (at the time of our review), and assessed the extent to which State/INL and USAID assessed and approved implementing partners’ periodic performance reports and data in accordance with the key monitoring practice of assessing and approving performance information. We also met with State/INL and USAID monitoring officials in Washington, D.C., and Mexico to understand the process for how these officials track the performance of these selected projects, including in the projects’ quarterly and annual reports. We also reviewed the reports to identify any discrepancies or errors. To describe the type of Government of Mexico data that State/INL uses to monitor Mérida Initiative implementation, we reviewed data from fiscal years 2015-2018 related to Mérida Initiative projects collected by the Government of Mexico and shared with State/INL. We also met with State/INL officials in Washington, D.C., and Mexico City to discuss the data, including how it is used and its reliability. After our discussions with State/INL officials, State/INL selected some unclassified examples of the indicators, which we included in our report. The purpose of this component of our review was to describe the nature and use of the Mexico data. We conducted this performance audit from November 2018 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix provides a list of the 15 Department of State (State), Bureau of International Narcotics and Law Enforcement Affairs (State/INL) Mérida Initiative projects, and five United States Agency for International Development (USAID) Mérida Initiative projects selected for our review. We assessed State/INL and USAID monitoring of these projects against key monitoring practices as described in appendix I. The subset of these projects (four State/INL and two USAID) selected for our analysis of the agencies’ tracking of performance data is noted below. State/INL provided the details in table 6, and USAID provided the details in table 7. In addition to the contact named above, James Michels (Assistant Director), Francisco Enriquez (Analyst-in-Charge), Terry Allen, Ashley Alley, Lilia Chaidez, Martin De Alteriis, Neil Doherty, Teresa Heger, John Hussey, and Andrew Kincare made key contributions to this report.", "summary": "The Mérida Initiative is a bilateral U.S.-Mexico partnership to address crime and violence and enhance the rule of law in Mexico. Through this initiative, managed by State/INL and USAID, the United States has provided a wide range of assistance, including training and equipment. Since fiscal year 2008, U.S. funding for the Mérida Initiative has totaled about $3 billion. GAO has identified key practices for monitoring foreign assistance programs that agencies should implement to address impediments, effectively manage foreign assistance, and meet assistance goals. These practices are generally consistent with policies of State, USAID, and the Office of Management and Budget. GAO was asked to review issues related to Mérida Initiative implementation and objectives. This report examines the extent to which State/INL and USAID follow key practices in monitoring Mérida Initiative projects and track project performance against established measures. GAO reviewed State and USAID documents and data for a nongeneralizable sample of 20 high-dollar value projects, and interviewed officials from State; USAID; and other U.S. agencies in Washington, D.C., and Mexico City. For the 15 Department of State (State) Bureau of International Narcotics and Law Enforcement Affairs (State/INL) projects GAO reviewed, State/INL generally followed key monitoring practices about half of the time. (See figure.) For example, State/INL almost always assigned staff with appropriate qualifications to monitor Mérida Initiative projects. However, for most projects, State/INL did not generally follow the key practices for developing monitoring plans that identify project goals and objectives and address risks to achieving them. Furthermore, State/INL did not consistently track project performance data. By establishing procedures for following key monitoring practices, State/INL would be better positioned to stay well informed of its projects' performance, take corrective action when necessary, and help ensure that projects achieve intended results. For the five United States Agency for International Development (USAID) projects GAO reviewed, USAID almost always followed key monitoring practices and tracked performance data. USAID established procedures, such as periodic portfolio reviews, to ensure its staff consistently monitored projects. While USAID identified risks to implementing projects, it did not address those risks in its monitoring plans. (See figure.) Developing monitoring plans to address risks could help USAID determine the appropriate level of oversight for each Mérida Initiative project and manage monitoring resources more cost effectively. GAO is making two recommendations, including that State establish procedures to verify monitoring staff follow key practices, and that USAID ensure that monitoring plans address risks. State and USAID concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "The United States has maintained a cooperative relationship with North Macedonia across a broad range of political, economic, cultural, military, and social issues since North Macedonia gained its independence from Yugoslavia in 1991. The United States formally recognized North Macedonia in 1994, and the countries established full diplomatic relations in 1995. Following a civil conflict between the country’s ethnic Albanian minority and the Macedonian majority in 2001, the United States and the EU mediated a resolution and supported efforts to agree to a peaceful, political solution to the crisis, known as the Ohrid Framework Agreement. Figure 1 shows Macedonia’s location in southeastern Europe. corridor from Western and Central Europe to the Aegean Sea 2,118,945 (146th largest in the world, 2018 $31.03 billion gross domestic product in 2017 22.4 percent unemployment rate (2017 est.) Macedonian, 64.2 percent; Albanian, 25.2 percent; Turkish, 3.9 percent; Romani, 2.7 percent; Serb, 1.8 percent; other, 2.2 percent (2002 est.) In 2011, USAID and State assessed that North Macedonia’s conservative party, the Internal Macedonian Revolutionary Organization–Democratic Party for Macedonian National Unity (known as VMRO-DPMNE, or VMRO) was consolidating political power when it became the ruling party in 2006. USAID and State found that government control over North Macedonia’s judiciary, Parliament, media, civil society, and local government was increasing. In December 2012, security personnel ejected members of the Social Democratic Union of Macedonia (SDSM), the main opposition party, from the Parliament building, along with journalists who had been observing the session, after SDSM members protested VMRO’s proposed budget. SDSM boycotted Parliament for approximately 2 months after this incident but returned in March 2013, when the parties reached an agreement. In May 2014, SDSM boycotted Parliament again, accusing VMRO of having violated the country’s electoral code in April 2014 elections, in which VMRO retained its parliamentary majority. In December 2014, USAID concluded that inadequate mechanisms for competition and political accountability represented the primary democracy and governance problems in North Macedonia. USAID noted, among other things, that the ruling party had deployed public resources and control of the media to limit competition; captured executive, legislative, and judicial institutions; and put pressure on, and excluded, civil society. In February 2015, the leader of SDSM began releasing phone conversations allegedly recorded by the government’s counterintelligence service that revealed widespread corruption and state capture by the ruling party, VMRO, triggering a political crisis. (See fig. 2 for a timeline of the crisis.) Street protests followed these leaks. The four main political parties invited the United States and EU to facilitate negotiations to broker a peaceful resolution to the crisis, known as the Przino Agreement, in June 2015. The parties agreed to, among other things, hold free and fair elections by the end of April 2016. After two failed attempts to hold elections in early 2016, the United States and EU convened North Macedonia’s political parties for another round of negotiations in the summer of 2016. The parties reached agreement on a number of key reforms and set the conditions for parliamentary elections by the end of 2016. These elections took place on December 11, 2016, without a clear majority winner. Although SDSM leader Zoran Zaev formed a majority coalition in February 2017, then-President Ivanov refused to give Zaev the mandate to form a new government until May 2017, following a violent storming of Parliament by hundreds of protesters in April. In May 2017, President Ivanov authorized SDSM to form a government with a coalition of ethnic Albanian parties. The new coalition government expressed support for North Macedonia’s accession to the EU and membership in the North Atlantic Treaty Organization (NATO). On February 12, 2019, the Republic of Macedonia formally changed its name to the Republic of North Macedonia, ending a longstanding dispute over its name with Greece, which had for years exercised its veto power in NATO to block North Macedonia’s membership (see the text box for details of North Macedonia’s NATO aspirations and name dispute with Greece). On February 6, 2019, NATO members signed an accession protocol with North Macedonia, paving the way for North Macedonia to become the 30th member of NATO. The EU states also opened the path to potential EU accession negotiations with North Macedonia in June 2019, contingent on the country’s full implementation of its agreement with Greece and its demonstrated progress in implementing EU-recommended reforms. However, the EU postponed the decision until no later than October 2019. On February 15, 2019, the U.S. government recognized North Macedonia’s name change. North Macedonia’s NATO Aspirations and Name Dispute with Greece In 2008, having determined that North Macedonia met North Atlantic Treaty Organization (NATO) membership criteria, NATO allies decided that North Macedonia would be invited to join NATO as soon as North Macedonia and Greece, a NATO member, resolved a dispute regarding North Macedonia’s name. A brief timeline of this dispute follows. 1991: The “Republic of Macedonia” declared its independence from the former Yugoslavia. Greece objected to this name, viewing “Macedonia” as representing territorial claims against Greece, which has a northern province by the same name. Because Greece has veto power in NATO, it was able to prevent the Republic of Macedonia from joining the organization. 1995: Greece and the Republic of Macedonia reached an interim accord in which Greece agreed not to block applications by the Republic of Macedonia to international organizations if made under the name “Former Yugoslav Republic of Macedonia.” 2008: At a NATO Summit in Bucharest, Greece blocked the Republic of Macedonia’s bid to join NATO. Dec. 2011: The International Court of Justice ruled that Greece had been wrong to block the Republic of Macedonia’s bid to enter NATO in 2008, but the decision did not affect NATO’s consensus-based decision-making process. June 12, 2018: The foreign ministers of Greece and the Republic of Macedonia signed the Prespa agreement, whereby the Republic of Macedonia would change its name to the Republic of North Macedonia, Greece would no longer object to North Macedonia’s Euro- Atlantic integration, and both countries would promise to respect existing borders. Sept. 30, 2018: The Republic of Macedonia held a referendum on changing its name to the Republic of North Macedonia, with nearly 92 percent of votes in favor of the change. Overall turnout for the referendum was about 37 percent, as opponents of the name change boycotted the referendum. Oct. 19, 2018: A two-thirds majority in North Macedonia’s Parliament voted in favor of the name change. Jan. 11, 2019: North Macedonia’s Parliament approved a constitutional amendment that renamed the country to the Republic of North Macedonia. Jan. 25, 2019: The Greek Parliament voted to approve the deal outlined in the Prespa agreement. Feb. 6, 2019: NATO’s 29 members signed an accession protocol with North Macedonia, paving the way for the country to become the 30th member of the alliance. Feb. 8, 2019: Greece became the first NATO member to ratify the accession protocol. Feb. 12, 2019: The Republic of Macedonia formally changed its name to the Republic of North Macedonia. Feb. 15, 2019: The U.S. government recognized the Prespa Agreement’s entry into force and North Macedonia’s name change. According to State, democracy assistance seeks to advance freedom and dignity by assisting governments and citizens to establish, consolidate, and protect democratic institutions, processes, and values. These components include participatory and accountable governance, rule of law, authentic political competition, civil society, human rights, and the free flow of information. Democracy assistance falls into six program areas—Rule of Law, Good Governance, Political Competition and Consensus-Building, Civil Society, Independent Media and Free Flow of Information, and Human Rights—each with different program elements. See appendix V for descriptions of democracy program areas and program elements. The U.S. government provides democracy assistance through multiple bureaus and offices in USAID, State, and NED. For a list of these agencies’ roles and responsibilities related to democracy assistance overseas, see table 1. Federal laws governing agencies’ use of contracts and grants seek to promote discipline in the selection and use of procurement contracts, grant agreements, and cooperative agreements; maximize competition in making procurement contracts; and encourage competition in making grants and cooperative agreements. USAID’s operational policy, the Automated Directives System, incorporates these requirements into agency guidance. Thus, in selecting recipients of democracy assistance, agency staff are required to guarantee the integrity of the competitive award process by ensuring overall fairness and considering all eligible applications for an award. Since North Macedonia’s separation from Yugoslavia in 1991, the United States has provided democracy assistance to support North Macedonia’s Euro-Atlantic integration and the development of prosperous and democratic institutions. This assistance has focused on promoting rule of law, political processes, citizen engagement, and free media. In light of North Macedonia’s 2015 political crisis, as well as democratic backsliding observed in the years before the crisis, USAID narrowed its assistance goals for the country to focus on more inclusive citizen engagement in civic life, political processes, and the free flow of information to support better functioning checks on executive authority. The USAID mission in North Macedonia’s strategic plan for 2011 through 2015 identified three primary objectives of U.S. democracy assistance in North Macedonia: Promote greater checks and balances in democratic processes by empowering local governments, promoting greater equilibrium among the branches of government at the national level, and promoting political accountability. Develop a basic education system that prepares youth for a modern economy and stable democracy by improving students’ basic skills, expanding workforce skills, and enhancing ethnic integration in the education sector. Increase job-creating private-sector growth in targeted sectors by improving the country’s business environment in critical areas and strengthening key private-sector capacities. Additionally, USAID and State relied on a broader strategic framework, the integrated country strategy, when developing democracy projects in North Macedonia. This interagency, multiyear, overarching strategy outlines U.S. policy priorities and objectives for North Macedonia. Its objectives include improving North Macedonia’s democratic and civil society environment to improve the country’s prospects for joining NATO and for completing accession negotiations with the EU. U.S. government agencies obligated more than $45 million in democracy assistance funding for North Macedonia in fiscal years 2012 through 2017, according to agency award documents and data (see table 2). This assistance was provided to support U.S. strategic objectives for North Macedonia, including promoting the rule of law, political processes, citizen engagement, and free media. USAID obligated approximately $38 million, and NED obligated approximately $4.2 million. Additionally, the Public Affairs Section of the U.S. Embassy in Skopje provided about $3.7 million in assistance. However, we are unable to report total State obligations for democracy assistance for North Macedonia because of uncertainty about the reliability of award data from State’s Bureau of International Narcotics and Law Enforcement Affairs (INL). In addition, State’s Bureau of Democracy, Human Rights, and Labor (DRL) provided democracy assistance in North Macedonia solely through regional grants and did not specify which obligated funds were provided for democracy assistance in North Macedonia. See appendixes II through IV for a full list of USAID, NED, and State awards for democracy assistance in North Macedonia in fiscal years 2012 through 2017. USAID provided about $38 million in democracy assistance for North Macedonia in fiscal years 2012 through 2017. As table 3 shows, the majority of USAID funding—approximately $17 million—supported projects in the civil society program area, while more than $7 million supported political competition and consensus building. Several USAID bureaus and offices provided democracy assistance in North Macedonia during that period. The Bureau for Democracy, Conflict, and Humanitarian Assistance and the Bureau for Europe and Eurasia provided such assistance through contracts, grants, and cooperative agreements. According to agency documents, USAID supported U.S. foreign policy in North Macedonia by promoting democracy and respect for the rule of law and human rights, through activities such as supporting civil society organizations and developing the capacity of independent media outlets in the country. USAID also promoted political competition and accountability by working with political parties and state institutions to enable an environment for free and fair elections. In addition, USAID’s Office of Transition Initiatives (OTI) provided short- term assistance to groups in the country. OTI established an office in North Macedonia in September 2015 to support reform processes outlined in the Przino Agreement. According to OTI documents, OTI supports U.S. foreign policy objectives by promoting stability, peace, and democracy through fast, flexible, short-term assistance targeted to key political transition and stabilization needs. The office works with civil society organizations, media groups, and government institutions to increase access to reliable information, promote free and open civic discourse, and support democratic reforms. In North Macedonia, OTI funded initiatives such as a televised debate series that presented civil dialogue and diverse viewpoints on issues affecting citizens of North Macedonia. OTI grants have also supported digital media initiatives and civic engagement projects. USAID assistance supported initiatives in a range of democracy program areas. Table 4 shows examples of USAID projects across different program areas, some of which are related to democracy assistance. NED awarded 72 grants totaling nearly $4.2 million in North Macedonia in fiscal years 2012 through 2017. Of these, six grants, totaling almost $1.7 million, were awarded to two of NED’s core institutes—the National Democratic Institute and the Center for International Private Enterprise— while 66 grants, totaling about $2.6 million, were awarded to other organizations. In addition, NED awarded 61 grants totaling more than $17.1 million for regional programs that included North Macedonia. NED does not disaggregate cost data by individual country due to the nature of the Balkan regional programs NED supports. Thus, we are unable to report the amounts NED provided in North Macedonia through regional programs during the period of our review. After the onset of the political crisis in 2015, NED focused its democracy assistance in North Macedonia on three program areas: promoting good governance, supporting independent media, and fostering positive interethnic relations. NED grants supported a range of initiatives, including projects to improve investigative reporting on democratic reforms and rule-of-law matters, and to encourage youth leadership and activism. NED’s funding to the National Democratic Institute and the Center for International Private Enterprise supported a range of activities in North Macedonia. The institute worked with the country’s Parliament to improve its management and organization of the legislative process by, among other things, assisting Parliament in reviewing its legislative and oversight procedures. Other National Democratic Institute initiatives included encouraging participation by various groups in the democratic process, including the Roma population, women, and civil society organizations. The Center for International Private Enterprise received funding for one grant devoted to developing youth leadership. Several State offices—U.S. Embassy Skopje, INL, and DRL—provided funding for democracy assistance in North Macedonia, but only the funding provided by the embassy can be reliably reported. The embassy’s Public Affairs Section provided at least $3.7 million in democracy assistance in North Macedonia in fiscal years 2012 through 2017. INL was unable to provide reliable data on obligations on its awards in North Macedonia. DRL obligated more than $2 million to support democracy assistance activities at the regional level but due to the regional nature of its projects, was unable to provide country-level breakdowns of obligations. In fiscal years 2012 through 2017, Embassy Skopje’s Public Affairs Section obligated approximately $3.7 million in democracy assistance grants to organizations in North Macedonia. According to State officials, the embassy works with the Coordinator of U.S. Assistance for Europe and Eurasia to allocate democracy assistance and helps align assistance activities with the U.S. strategic goals for North Macedonia. The embassy’s Public Affairs Section also provides democracy assistance through other means, including media training programs, youth engagement projects, speaker programs, and the Democracy Commission Small Grants Program. The embassy granted $1.8 million for 91 grants through the Democracy Commission Small Grants Program in fiscal years 2012 through 2017. According to the embassy, grants through this program, which cannot exceed $24,000, support nongovernmental organizations’ efforts to promote the rule of law, independent media, interethnic community building, the empowerment of women and youth, human rights, and the institutionalization of open and pluralistic democratic political processes. Examples of awards for Democracy Commission grant–funded activities include the following: Women’s Rights Center ($22,900). This award funded a program to strengthen the capacities of organizations that are working with women victims of domestic violence. Civil Lyceum Project ($17,830). This project aimed to mobilize youth in Skopje to become more involved in the civil society sector and to help create young leaders who understand the value of civic engagement and advance democratic values. Way Out ($7,858). This award funded the maintenance and development of the online version of a student magazine. The remainder of the embassy’s Public Affairs Section awards for assistance in North Macedonia supported activities such as youth engagement projects, speakers, and media training programs, which included short-term trips for journalists from North Macedonia to receive training in the United States. INL provided democracy assistance to organizations in North Macedonia in fiscal years 2012 through 2017. INL was unable to provide reliable data on project-level obligations; however, it reported bulk obligations for democracy assistance projects that supported efforts to reform North Macedonia’s criminal justice system to meet rule-of-law benchmarks for Euro-Atlantic integration. INL’s assistance in North Macedonia focused on three primary areas: developing the country’s criminal justice system, developing legal professionals’ skills, and professionalizing the police. According to agency officials, this assistance is intended to strengthen North Macedonia’s justice sector and independent institutions. Specific INL activities included assisting with revisions to the criminal procedure code to promote a more adversarial justice system, providing technical advisors and equipment to the Special Prosecutor’s Office, and promoting accountable policing efforts by providing training to local police on crime scene management. In December 2017, we reported that INL funding data for democracy assistance projects were unreliable and we recommended that State identify and address factors that affect the reliability of its democracy assistance data. State concurred with this recommendation. As of July 2019, INL reported continued efforts to improve data quality and reliability, including ensuring that current and future transactions would maintain coding integrity. However, officials stated that, because of missing codes or miscoded items, they were unable to provide reliable data on obligations for INL awards for democracy assistance projects in North Macedonia for fiscal years 2012 through 2017. Although we determined that data for specific INL democracy awards were unreliable, INL reported providing bilateral assistance of approximately $14.2 million in North Macedonia in fiscal years 2012 through 2017, including $6.9 million for democracy assistance. However, we did not independently verify that INL provided this amount of bilateral assistance. DRL funded four awards that benefited North Macedonia in fiscal years 2012 through 2017. However, DRL awarded this assistance at the regional level and does not track country-level obligations for North Macedonia. One regional award with obligations of roughly $300,000 supported a project focusing on Roma populations in Bulgaria, North Macedonia, Romania, and Serbia. A second regional award provided more than $2 million for a project promoting the rule of law in the Balkans. The two remaining DRL awards provided $25,000 to organizations supporting local civil society organizations working to promote human rights. Our review of 13 USAID grants and cooperative agreements for democracy assistance—representing roughly half of USAID obligations in North Macedonia in fiscal years 2012 through 2017—found that in selecting recipients, the agency generally followed operational policies intended to ensure a fair and transparent selection process. (See table 5 for a list of the awards in our sample.) We found that staff at the USAID mission in North Macedonia generally evaluated applicants against the merit review criteria stated in public notices. We also found that USAID considered and recorded the strengths and weaknesses of applicants in selection committee memorandums for 10 of the 13 awards in our sample. For three awards originating from the same public notice, we were unable to determine, on the basis of available documentation, whether USAID considered the strengths and weaknesses of all applicants. Finally, we found that USAID documented the review procedures it used to assess applicants in selection committee memorandums. USAID’s selection committee considered merit review criteria that were consistent with those included in the agency’s public notices for 10 of the 13 awards for democracy assistance in North Macedonia that we reviewed. USAID’s process for selecting recipients of assistance for competitive awards requires announcing opportunities, reviewing applications, and making award decisions on the basis of published merit review criteria. USAID announces a grant opportunity by developing a notice of funding opportunity. Merit review criteria are developed by the USAID staff and reflect the agency’s strategic priorities for democracy assistance. After interested parties have submitted applications, a selection committee, also known as a technical evaluation committee, is appointed to review applications. All 13 awards in our sample included merit review criteria in public notices during the concept paper phase of awards, while 10 of the awards included merit review criteria for the full application phase. Many of the awards required selection committees to consider some of the same merit review criteria in assessing applicants. Examples of commonly applied criteria include the following: Technical approach. Reviewers are to assess the extent to which an applicant’s proposed activity is clear, logical, and technically sound and meets the objectives of the funding outlined in the public notice. Management plan and key personnel. Reviewers are to assess the extent to which an applicant considered staffing, roles and responsibilities, and other management issues for their proposed activity. Organizational capacity and past performance. Reviewers are to assess the extent to which the applicant demonstrated the technical and managerial resources and expertise to achieve their program objectives. Reviewers are also to assess the extent to which the applicant demonstrated technical and managerial resources and expertise in past programs and performed satisfactorily in similar programs executed in recent years. We found that in reviewing the 13 awards in our sample, USAID generally applied the criteria published for each award. Six of the 13 awards in our sample were two-phased awards, for which the mission required potential applicants to first submit an executive summary or concept paper for their proposed activity. For these awards, the mission published separate merit review criteria for concept papers and full applications, and selection committees assessed each type of submission against the relevant set of criteria. The selection committee memorandums for three awards showed that these merit review criteria were consistent with the criteria outlined in the public notices for each award. Specifically, in the first phase of the award process, staff at the USAID mission in North Macedonia applied the published criteria for concept papers in reviewing the submitted papers and selected those that best met the criteria. In the second phase for three awards, USAID solicited applications from the selected applicants and applied the published criteria for full applications in reviewing the submitted applications. In the case of three awards that originated from the same public notice, the notice lacked merit review criteria for the full application phase. The public notice for these three awards did not include the merit review criteria the selection committee would use to evaluate full applications. For the remaining seven one-phased awards in our sample, the selection committee memorandums showed that USAID applied the criteria published in the award solicitations in reviewing the applications that were submitted, consistent with USAID’s operational policies. We found that USAID officials generally assessed applicants’ strengths and weaknesses when reviewing applications for awards for democracy assistance in North Macedonia. USAID operational policy requires selection committees to evaluate the strengths and weaknesses of each applicant for an award relative to the merit review criteria. The committee then prepares a written selection memorandum recording its assessments, which is then sent to the agreement officer. For the 13 awards in our sample, selection committee memorandums show that officials generally considered and recorded their assessments of applicants’ strengths and weaknesses against the criteria outlined in the public notices. For example, in considering the applicants for one award in our sample, the selection committee assessed the strengths and weaknesses of applicants’ technical approaches by looking at the logical connection between their activities and stated objectives, their plans for community outreach, and their awareness of potential problems that might arise over the course of their projects. The committee also assessed applicants’ strengths and weaknesses with regard to management plans and key personnel by considering, among other things, applicants’ plans to train staff, their knowledge of the stakeholders they planned to engage, and the relevant experience of the organizations’ leaders. In addition, the committee assessed applicants’ strengths and weaknesses with regard to organizational capacity and past performance, primarily by examining whether applicants had successfully managed projects of similar magnitude, scope, and sensitivity in recent years. For this award, the selection committee provided an overall score for each criterion based on the numerical scoring outlined in the award’s public notice and ultimately recommended the top-scoring applicant to the agreement officer. For three of the six two-phased awards we reviewed, selection committee officials considered and recorded their assessments of applicants’ concept papers as well as the full applications they received. For three two-phased awards that originated from the same public notice, we could not determine, on the basis of available documentation, whether the selection committee assessed the strengths and weaknesses of applicants relative to the merit review criteria. We found that USAID documented its review procedures, consistent with USAID policy. USAID operational policy requires that the selection committee include in its review documentation a discussion of its procedures for reviewing awards. For all 13 awards, the selection committee memorandums included a discussion of the review procedures that the committee used to assess applicants. These review procedures included actions such as the following: The establishment of the selection committee, including its purpose and composition A requirement for selection committee members to sign a certificate regarding nondisclosure, conflict of interest, or rules of conduct Individual reviews of the applications by each selection committee member A review of the rating system the committee used to assess A joint meeting to discuss individual reviews and ratings of applications, resulting in consensus among selection committee members about the strengths and weaknesses of each application For the two-phased awards in our sample, the selection committee memorandums include documentation of review procedures for both the concept paper and full application phases of awards. The selection committee memorandum for the full application phase of these awards included other actions that the selection committee took, such as the following: A summary of the committee’s procedures and results in the concept An evaluation of the proposals from applicants who were invited to A discussion of the programmatic weaknesses that USAID asked applicants to address before submitting their full applications We are sending copies of this report to the USAID Administrator, the Secretary of State, and the President of NED. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. Our objectives were to examine (1) U.S. funding for democracy assistance in North Macedonia in fiscal years 2012 through 2017 and (2) the extent to which the U.S. Agency for International Development (USAID) adhered to relevant operational policies in selecting recipients of democracy assistance in North Macedonia. To identify the United States’ strategic objectives and goals for providing democracy assistance in North Macedonia, we reviewed USAID and Department of State (State) strategic documents and interviewed cognizant USAID and State officials in Washington, D.C. To examine U.S. funding for democracy assistance in North Macedonia, we analyzed award data from USAID, State, and the National Endowment for Democracy (NED) for fiscal years 2012 through 2017, the most recent 5-year period for which these data were available. To determine the data’s reliability, we interviewed agency officials and reviewed relevant documentation. We determined that USAID’s and NED’s data were sufficiently reliable for the purposes of our reporting objectives. We further determined that State’s data on the U.S. Embassy in Skopje’s Public Affairs Section awards were reliable for these purposes. However, on the basis of interviews with State officials, our review of their data, and our prior work, we determined that the data maintained by State’s Bureau of International Narcotics and Law Enforcement Affairs (INL) could not be reliably reported. We determined that data provided by State’s Bureau of Democracy, Human Rights, and Labor Affairs (DRL) were reliable; however, we could not determine what portion of DRL funding went only to North Macedonia, because DRL made regional awards during this period that benefited several Balkan countries. Therefore, we report State obligations as approximations for awards for which we had more reliable data. To identify the recipients of democracy assistance in North Macedonia and describe the process through which the U.S. government grants such assistance, we reviewed award data, relevant award documents, and bilateral agreements and other communications between the United States and North Macedonia regarding this assistance. We interviewed USAID, State, and NED officials in Washington, D.C., who oversee democracy assistance in North Macedonia regarding U.S. funding for such assistance. We also interviewed representatives of organizations that implement this assistance that have offices in Washington, D.C. In addition, during audit work in Skopje, North Macedonia, we interviewed USAID and State officials who manage democracy assistance. We also met with officials from the government of North Macedonia, including the Minister of Defense and members of Parliament, the State Election Commission, and the Agency for Audio and Audiovisual Services, to determine the types of activities the U.S. government supported during the period of our review. In addition, we conducted individual and group interviews with representatives of 41 implementing partners of USAID, State, and NED in Skopje who received funding during the period of our review. To assess the extent to which USAID officials followed operational policies in selecting recipients of democracy assistance, we analyzed award data and documentation for a sample of awards made between fiscal years 2012 through 2017. We excluded from our sample any contracts and other awards for which no public notice was issued, because these awards were not openly competed. We further excluded grants under contract arrangements that USAID entered into with local partners in North Macedonia, because these awards also were not openly competed. Such awards include those made by USAID’s Office of Transition Initiatives and under the Consortium for Elections and Political Process Strengthening process. Our sample comprised the 13 largest- value grants and cooperative agreements that USAID made for North Macedonia in fiscal years 2012 through 2017, constituting 46 percent of all USAID obligations in North Macedonia during this period. We analyzed USAID operational policies contained in the Automated Directives System (ADS) and other USAID policy documents outlining the agency’s strategic plan and assistance priorities for North Macedonia. We analyzed relevant documents for the awards in our sample, including the notices of funding opportunity and selection committee memorandums, and we assessed the extent to which these documents showed that USAID had met the requirements of its operational policy outlined in the ADS. In particular, for each award, we examined the extent to which the merit review criteria published in the notice of funding opportunity matched the criteria the selection committee used, the selection committee assessed the strengths and weaknesses of the submitted applications and recorded these assessments, and the selection committee included a discussion of its review procedures in its review documentation. Finally, we interviewed USAID officials in Washington and Skopje regarding USAID’s operational policies in fiscal years 2012 through 2017 as well as its process for selecting recipients of democracy assistance. We conducted this performance audit from May 2017 to October 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 6 lists the U.S. Agency for International Development’s (USAID) awards for democracy assistance in North Macedonia in fiscal years 2012 through 2017. Table 7 lists the National Endowment for Democracy’s (NED) democracy assistance awards in North Macedonia in fiscal years 2012 through 2017. Tables 8 and 9 list the Department of State’s (State) awards for democracy assistance to North Macedonia in fiscal years 2012 through 2017. These awards were provided by U.S. Embassy Skopje through its Public Affairs Section. Table 8 shows the grants that the embassy’s Public Affairs Section awarded through the Democracy Commission Small Grants Program, and table 9 shows other, non–Democracy Commission grants awarded by the Public Affairs Section. Table 10 provides an overview of the program areas and program elements that fall into democracy, human rights, and governance assistance according to the Department of State (State). U.S. foreign assistance is categorized through a system called the Standardized Program Structure and Definitions, which comprises broadly agreed-on definitions for foreign assistance programs and provides a common language to describe programs. According to this system, democracy assistance includes the following six program areas. In addition to the contact named above, Rob Ball (Assistant Director), Cheryl Goodman (Assistant Director), Rachel Dunsmoor (Analyst-in- Charge), Parul Aggarwal, R. Gifford Howland, Ashley Alley, Justin Fisher, Christopher Keblitis, and Reid Lowe made key contributions to this report.", "summary": "Since fiscal year 1991, the United States has provided over a billion dollars in assistance to North Macedonia. In recent years, USAID and State have expressed concern about an erosion of democracy in the country. These concerns were heightened by the onset of a political crisis in February 2015, when the then-opposition party released phone conversations revealing alleged corruption in the ruling party. This crisis prompted the four major political parties to invite the United States and the European Union to help broker an agreement. The parties later agreed to hold early parliamentary elections in December 2016. Though the opposition party formed a majority coalition, the President refused to give the opposition leader a mandate to form a new government until May 2017, after protesters violently attacked North Macedonia's Parliament. This report examines (1) U.S. government funding for democracy assistance in North Macedonia and (2) the extent to which USAID adhered to relevant policies in selecting recipients of democracy assistance in North Macedonia. GAO analyzed U.S. government data and documents and interviewed U.S. officials in Washington, D.C., and in Skopje, North Macedonia. The U.S. government provided more than $45 million for democracy assistance in North Macedonia through the U.S. Agency for International Development (USAID), National Endowment for Democracy (NED), and U.S. Department of State (State) in fiscal years 2012 through 2017. During this 5 year period—the most recent for which funding data were available—USAID obligated about $38 million to support rule of law and human rights, governance, political competition and consensus building, civil society, and an independent media and free flow of information. NED—a nongovernmental organization funded largely through appropriated funds—provided $4.2 million for activities such as training in investigative reporting and rule of law. The U.S. embassy in Skopje obligated at least $3.7 million for rule of law and human rights, governance, and civil society. State's Bureau of International Narcotics and Law Enforcement Affairs (INL) and Bureau of Democracy, Human Rights, and Labor (DRL) also provided funding for democracy initiatives. However, GAO is unable to report State's total obligations, because INL's data were unreliable and because DRL, due to the regional nature of its projects, does not track country-level obligations for North Macedonia. Legend: USAID = U.S. Agency for International Development, NED = National Endowment for Democracy, State = U.S. Department of State. Note: Only obligations from the Public Affairs Section of the U.S. Embassy in Skopje are shown for State. State's other funding data were either unreliable or not tracked at the country level. GAO's review of 13 USAID democracy assistance awards, representing roughly half of USAID obligations in fiscal years 2012 through 2017, found that the agency generally complied with operational policy intended to ensure a fair and transparent selection process. USAID policy requires officials to consider merit review criteria specified in public notices and to assess applicants against these criteria. GAO found that the merit review criteria USAID included in public notices were generally consistent with the criteria that selection committees used to evaluate applicants. GAO also found that selection committees generally discussed the relative strengths and weaknesses of award applications and recorded these discussions in selection memorandums, consistent with USAID policy. In prior work, GAO recommended that State identify and address factors affecting the reliability of INL's democracy assistance data. State concurred and, in July 2019, reported that INL was continuing efforts to improve data reliability. GAO will continue to monitor State's efforts to ensure this recommendation is fully implemented.", "document_type": "gao"}
{"report": "Opioids—such as hydrocodone, oxycodone, morphine, and methadone— can be prescribed to treat both acute and chronic pain. Many opioids have a high potential for abuse and may lead to severe psychological or physical dependence. OUD, which is a type of substance use disorder, is generally characterized by a loss of control of opioid use, risky opioid use, impaired social functioning, tolerance, and withdrawal. According to SAMHSA and the National Institute on Drug Abuse, OUD is a chronic, treatable illness. SAMHSA states that treatment for OUD should be individualized and can include a range of treatment options that include medication and behavioral health services. Specifically, services related to the treatment of OUD include the following: Screening can identify individuals who have OUD, are at risk for developing OUD, or have medical problems related to opioid use. Screening, brief intervention, and referral to treatment (SBIRT) is a specific type of screening that involves a health care provider educating individuals with a positive screen for opioid use and referring them to specialized treatment, as needed. Outpatient counseling and therapy includes counseling and treatment services individually or in a group. Inpatient hospital services include those that occur in a hospital, such as services for detoxification. Inpatient residential services include care in a 24-hour residential setting. Inpatient residential providers offer medical care in combination with housing, typically lasting from a week to several weeks or more. Medication-assisted treatment (MAT) combines the use of certain prescription medications (such as methadone, buprenorphine, and naltrexone) and behavioral therapy. Methadone and buprenorphine suppress withdrawal symptoms and control the craving for opioids, while naltrexone suppresses the euphoric effects of opioids. Research has shown that MAT for OUD reduces opioid use and increases the chance that patients will continue OUD treatment compared to abstinence-based treatment (where patients are treated without medication). Case management services include providing coordination and management of care across multiple health care providers. Crisis intervention includes immediate care intended to prevent harm. Peer recovery coaching includes recovery support through a certified peer specialist with experience of recovery from addiction. States administer their Medicaid programs within broad federal requirements and according to a state plan approved by CMS. The Medicaid program allows states to design and implement their programs within certain federal parameters, resulting in more than 50 distinct programs. A state’s approved Medicaid plan outlines the services provided and the groups of individuals covered. States also have the option of using waivers to expand services under the Medicaid program. As such, the types of services covered by Medicaid can vary across states. Historically, Medicaid eligibility has been limited to certain categories of low-income individuals, such as children, parents, pregnant women, persons with disabilities, and individuals aged 65 and older. The Patient Protection and Affordable Care Act (PPACA), enacted in 2010, allowed states to expand Medicaid coverage to nearly all individuals with incomes up to 138 percent of the FPL, regardless of eligibility category. As of October 2019, 33 states and the District of Columbia expanded Medicaid eligibility, and 17 states had not. Under federal law, state Medicaid programs must provide coverage for health care services for certain pregnant women, including low-income pregnant women with incomes at or below 138 percent FPL. Most states opt to extend coverage to pregnant women with incomes above this threshold. By statute, states are permitted to limit the services covered for certain pregnant women, including low-income pregnant women, to services related to the pregnancy. Such coverage is referred to as “pregnancy-related coverage.” CMS defines pregnancy-related services as those services necessary for the health of the pregnant woman and fetus, or that have become necessary as a result of the woman having been pregnant, which includes prenatal, delivery, postpartum, and family planning services, as well as services for other conditions that may complicate the pregnancy. In contrast, states are required by statute to provide pregnant women who qualify for Medicaid on another basis, such as a disability, full Medicaid benefits. At a minimum, states must provide Medicaid coverage for pregnant enrollees through 60 days postpartum, though states may extend coverage further. Some women may qualify for continued Medicaid coverage after the 60-day postpartum period if they meet the requirements for another eligibility pathway, such as for parents, while others may transition to other programs or become uninsured. Medicaid’s Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) benefit is the primary mechanism to help ensure the provision of appropriate health care services to children under Medicaid. Under the EPSDT benefit, states are required to cover comprehensive health screenings and preventive health services, such as those related to vision and oral health, and all other Medicaid coverable services that are necessary to correct or ameliorate any conditions discovered through screenings. States are required to follow a schedule of screenings, known as a periodicity schedule, that are recommended for children at specific ages and frequencies. States can develop their own schedule within federal parameters or follow an established schedule, such as from the American Academy of Pediatrics (AAP). States are required to report annually on the provision of certain EPSDT services to CMS. States must report information on the number of children provided health screening services, the total number of health screenings services provided, the number of children referred for corrective treatment, the number of children receiving dental services, and the state’s results in attaining EPSDT participation goals. States may also voluntarily report annually on the quality of health care services provided under EPSDT using a set of quality measures known as the Child Core Set. CMS plans to increasingly use the Child Core Set in the future and state reporting will be mandatory beginning with the state reports for fiscal year 2024. States also have flexibility in determining where EPSDT services can be delivered. As a means of improving access—particularly in underserved communities, such as rural areas—Medicaid programs may cover certain services delivered by health care providers in schools. There were approximately 130,000 schools across the United States as of 2016, according to data from the Department of Education. Some of these schools have health clinics. Additionally, there were approximately 2,600 school-based health centers as of 2017, some of which served children in underserved communities, according to the School-Based Health Alliance. Some state Medicaid programs allow for services to be delivered via telehealth, including in schools. Telehealth can be used to provide clinical care remotely, such as for screening, counseling, and therapy. Health care providers offer care to patients through remote technology, such as a live, two-way video call. Such services could be provided, for example, via a video conference on a desktop computer or laptop that connects a student in school with a provider in another location. States have flexibility to choose whether to cover services delivered via telehealth. Because the federal Medicaid statute does not recognize telehealth as a distinct service, CMS views telehealth as a service delivery mechanism. According to CMS, services delivered via telehealth are subject to the same Medicaid requirements as those services provided in person. The Medicaid programs in our six selected states provided coverage of most OUD services during pregnancy and the first 60 days postpartum, as of January 1, 2019. Specifically, the selected states’ Medicaid manuals indicate these states provide coverage for at least seven of the eight OUD services we identified in our review, such as screenings, inpatient and outpatient services, and MAT. (See fig. 1.) In the six states we reviewed, we did not identify differences between the types of covered OUD services for pregnant women in expansion and non-expansion states. For example, the three expansion states— Arkansas, Colorado, and Massachusetts—and the three non-expansion states—Alabama, South Dakota, and Texas—each covered at least seven of the eight OUD services for pregnant women. Similarly, we did not identify differences in Medicaid coverage for OUD services between the selected states that limited coverage to pregnancy- related services and those that provided full benefits. Arkansas and South Dakota—the two states providing only pregnancy-related coverage— covered seven of the eight OUD services for pregnant women. According to Medicaid officials in these two states, the programs cover these OUD services because they are considered medically necessary. In contrast, neither state generally provides Medicaid coverage for peer recovery coaching for beneficiaries, including pregnant women. Three of the selected states—Alabama, Colorado, and Texas—covered certain OUD services for pregnant women that they do not cover for other beneficiaries under Medicaid. For example, in Alabama, screening services and inpatient residential services were covered only for pregnant women, but not other eligible, low-income women. In Colorado, pregnant women were the only group in the state for whom residential OUD services were covered under Medicaid. In Texas, pregnant women were the only group in the state for whom case management is a covered service under Medicaid. In all six selected states, once Medicaid coverage furnished on the basis of pregnancy ends after 60 days postpartum, women can continue to receive most OUD services under Medicaid if they qualify for Medicaid on another basis. For example, these women could qualify if their income is equal to or lower than the maximum allowable income for parents. However, in the six selected states, the maximum allowable income for eligible parents was generally lower than that for pregnant women, as of January 2019. (See fig. 2.) Women in the six selected states who are eligible to maintain Medicaid coverage after 60 days postpartum can continue most of the same OUD services that were covered during pregnancy. However, officials in two states said that the OUD services covered specifically for pregnant women under Medicaid would generally not be covered after the postpartum period ends. Four of the selected states provided estimates of the number of women who maintain Medicaid eligibility after the postpartum period ends. For example, officials in Massachusetts, an expansion state, estimated that in 2017 and 2018, approximately 99 percent of women with Medicaid coverage while pregnant maintained Medicaid coverage after the postpartum period ended. State officials in Colorado, also an expansion state, estimated that in 2015, 75 percent of women maintained coverage after the postpartum period ended. Additionally, in Arkansas, another expansion state, officials estimated that about 60 percent of women in 2017 and 2018 maintained Medicaid coverage after the postpartum period ended. Officials in Alabama, a non-expansion state, estimated that in 2017, about 43 percent of women maintained Medicaid coverage after the postpartum period ended. States may also obtain approval from CMS, such as under a waiver, to extend Medicaid coverage for women with OUD beyond 60 days postpartum, according to CMS officials. However, CMS officials were not aware of the number of states that have done so. In our review, we found that one of the six selected states, Colorado, used a section 1915(b) waiver to extend Medicaid eligibility for substance use services, including OUD services, for women beyond 60 days postpartum. Under the state’s “Special Connections” program, which was approved under the waiver, women who are eligible for Medicaid during their pregnancy can continue Medicaid coverage for OUD services for up to 12 months postpartum, including inpatient residential services, which would not otherwise be covered under Medicaid. A state official told us that the program began in 1991 to provide substance use disorder services to pregnant women and up to 60 days postpartum. In 2006, the state extended coverage under the program to provide substance use disorder services up to 12 months postpartum. The program aids in early identification and intervention for pregnant women with substance use disorders who are at risk of delivering low birth weight babies with health complications. Officials said the goal of the program is to improve the likelihood that the mother remains free from substance abuse. According to Colorado officials, 227 women participated in the program in 2018. We also found that the six selected states use other funding sources to provide coverage of OUD services for women with incomes that exceed the state’s Medicaid eligibility thresholds. Officials from the six selected states reported that they received SAMHSA grants so each state could provide OUD services for pregnant and postpartum women that extend beyond 60 days. According to the SAMHSA officials we interviewed, grants have been used to increase access to MAT, expand recovery support for pregnant women, and provide enhanced services for women to access OUD treatment. Furthermore, SAMHSA officials said that pregnant and postpartum women are specifically identified as target populations for grants, such as the agency’s State Targeted Response to the Opioid Crisis grant and its State Opioid Response grant. For example, officials in Arkansas told us that by using SAMHSA grants, they are able to allow uninsured or underinsured women who are seeking treatment for OUD to continue MAT treatment after 60 days postpartum. In addition, officials in the six selected states told us that women beyond 60 days postpartum generally would not experience gaps in treatment for OUD when transitioning from Medicaid to SAMHSA grant-funded programs, as women can generally continue receiving the same services and seeing the same providers. For example, state officials in Alabama, South Dakota, and Texas told us the state Medicaid agency contracts with providers that agree to participate in both the state’s Medicaid program and SAMHSA’s grant programs to allow for continuity of eligible services. Officials in Texas also told us they used state funds to implement a program to provide OUD services for up to 18 months postpartum. State officials told us that in this program—called the Neonatal Abstinence Syndrome-Opioid Treatment Services program—when a woman’s Medicaid coverage ends, she transitions to state-funded treatment to continue the same OUD services with the same provider. According to state officials, this program, funded since 2015, expands treatment services to postpartum women who would typically lose Medicaid coverage and become unable to pay for MAT services, which officials say help reduce relapse, overdose, and maternal mortality risk. State officials added that there is flexibility to extend services for postpartum women participating in the program for up to 2 years, if needed. State officials told us that since 2016, 296 women have participated in the program. In addition to the efforts in the selected states, the federal government has planned efforts to help states combat the opioid crisis, specifically for pregnant and postpartum women with OUD. For example, CMS plans to offer up to 12 cooperative agreements to states under the Maternal Opioid Misuse model as a way to improve access to services under Medicaid to pregnant and postpartum women with OUD. The model will have a 5-year performance period, 2020 through 2024, to allow states to implement strategies to improve the quality of care for pregnant and postpartum women with OUD. According to CMS officials, the model does not require that states extend coverage beyond 60 days postpartum, but states could choose to do so. CMS published the funding opportunity for the model in February 2019, and plans to select states to participate by the fall of 2019. In July 2019, CMS issued guidance to states regarding Medicaid coverage of services such as counseling for postpartum women while their infant is receiving treatment for Neonatal Abstinence Syndrome. The Centers for Disease Control and Prevention also issued a publication in September 2019 summarizing an initiative on state strategies to address OUD among pregnant and postpartum women and infants prenatally exposed to substances. The initiative identified five focus areas, including access to and coordination of quality services, provider awareness and training, and financing and coverage. In addition, the SUPPORT Act includes a provision for HHS to issue guidance to improve care for postpartum women with substance use disorder by the fall of 2019. The six selected states provide Medicaid coverage for annual screenings of eligible children for substance use, including opioids, as well as any medically necessary treatment for conditions identified through these screenings, as of 2019. This coverage is provided through the Medicaid EPSDT benefit. Based on our review of Medicaid state plans and EPSDT policies and periodicity schedules, we found that the six selected states established the following screening schedules at the time of our review: Arkansas’ Medicaid state plan and Colorado’s and South Dakota’s EPSDT policies specify that these states follow AAP’s screening schedule. AAP recommends annual substance use screening for all children beginning at age 11 until they reach the age of 21. Alabama’s EPSDT policy requires annual screening for all children ages 6 to 13. Massachusetts’ EPSDT policy requires providers to conduct an annual assessment of every child’s risk for substance use as part of a health history assessment during a child’s annual visit. This assessment can be conducted at any age. Texas’ EPSDT periodicity schedule recommends annual screening for all children ages 12 to 18. Additionally, Massachusetts and Texas Medicaid programs require behavioral health screening for all eligible children. The Medicaid programs in these states provide separate payment for behavioral health screening if the screening is conducted using an approved screening tool, some of which also screen for substance use, such as opioids. Texas officials reported that substance use screenings are considered part of the required overall mental health screening component of annual checkups. Similarly, Arkansas officials said that as part of a new EPSDT policy they are drafting, the state Medicaid program will require behavioral health screening for all children, which can include substance use screening if determined medically necessary by the provider. The six selected state Medicaid programs report data on the total number of screenings provided under EPSDT for children’s health care needs. However, officials from five of the selected states said that it is difficult or impossible to separate and thus track the number of the substance use screenings as distinct from other types of EPSDT screenings or visits that are recorded in Medicaid data. States are required to track and report the total number of EPSDT screenings provided, but not the number of substance use disorder screenings. Officials from all six selected states said that they conduct outreach and education to providers and parents to ensure awareness of the EPSDT benefits, as required. We found that the extent of information the states provide on these services varied among the six states. For example, outreach materials from three of the six selected states included information about the availability of substance use screening, and one of these three states, Alabama, also included information about services for opioid use. For all six selected states, officials emphasized that Medicaid’s EPSDT benefit requires states to cover any medically necessary treatment or service to address health conditions for a child, including opioid use. Officials from the six selected states also described a variety of initiatives to increase access to substance use disorder, including OUD, services for children. For example: Officials from Alabama said they recently began a program that offers more substance use disorder services in schools in a face-to-face capacity to help increase convenience and reduce stigma around these services for both the children and the parents. They explained that the Alabama Department of Mental Health added modifiers to ensure that their systems can capture data appropriately and analyze trends in providing school-based services, which are currently offered in over 40 individual schools. Officials added that Medicaid pays for covered services that are provided to Medicaid-eligible children in schools under this program. Officials from two states—Arkansas and Massachusetts—said they recently expanded the types of substance use disorder services covered for all Medicaid beneficiaries. Officials from Arkansas added that they are working to expand the number of providers who can offer substance use disorder treatment under Medicaid. Officials from two states—Massachusetts and Texas—said they had recently developed programs specifically aimed at serving children with substance use disorder, including OUD, using federal authority. Massachusetts received approval from CMS to conduct a Medicaid demonstration to establish OUD programs for children. Texas is using SAMHSA grant funding to support eight youth recovery centers that are intended to improve services and recovery supports for youth with substance use disorder, including OUD. We conducted outreach to Medicaid officials from all 50 states and the District of Columbia between February and July 2019 to inquire about whether the state provided Medicaid coverage of OUD services delivered via telehealth in schools as a means of increasing access to these services for children. Officials from 31 states and the District of Columbia reported that they provide Medicaid coverage of OUD services delivered in schools via telehealth. Medicaid officials from some states reported that their Medicaid policies explicitly allow for coverage of OUD services provided in schools via telehealth, while others reported that they allow for Medicaid coverage of these services, but their policies do not explicitly address the issue. (See app. I for the state responses regarding Medicaid coverage of OUD services provided in schools via telehealth.) Officials from the remaining 18 states reported that their Medicaid policies do not allow for payment for OUD services delivered in schools via telehealth. Some of these state officials reported that they did not allow schools to serve as a location for patient services during a telehealth visit for Medicaid payment purposes. Other state officials reported that they allowed Medicaid payment for certain services provided in schools via telehealth, but OUD services were not among them. For example, officials from Texas said the state established a school-based telehealth program for behavioral health services. However, this program does not include services for the treatment of substance use disorder, including OUD. Officials added that the state has a requirement that substance use disorder services can only be delivered in certain approved facilities. While Medicaid officials from 31 states and the District of Columbia reported that they provide Medicaid coverage of OUD services in schools via telehealth, they also said they were not aware of any instances of these services being utilized. Medicaid officials from the 31 states and the District of Columbia reported that either these services were not being provided based on data or other information they had, or they were unaware if the services were being provided. Officials from seven states responded that they either reviewed Medicaid utilization data or asked school-based staff and determined that there was no utilization of these services. For example, officials from one state—Ohio—conducted a data query and found that in 2018 there had been two instances of substance use disorder services billed to Medicaid that were delivered via telehealth in a school. However, officials noted that these instances involved treatment for substances other than opioids. While not for OUD, these two instances were the only instances of Medicaid payment for substance use disorder services delivered via telehealth in schools that we identified in our review. As part of our outreach to states and background research, we did find that some states or localities have taken steps to facilitate the use of telehealth for delivery of substance use disorder services, including OUD, in schools. For example: Officials from one county in Maryland said they recently began using a telehealth smart device application to screen students in schools for substance use disorder, including OUD, and refer them to treatment. However, county officials said that the program was locally funded, and they had not considered seeking Medicaid payment. A South Dakota tribal reservation recently implemented a new school- based telehealth program for behavioral health. According to officials, this program could include OUD services delivered via telehealth in schools, and these services could be billed to Medicaid if the provider was already licensed to bill Medicaid; however, officials said that none of these services had been provided to date. Massachusetts recently expanded its Medicaid telehealth policy to allow for the payment of mental health and substance use disorder services provided in many locations, including schools. However, officials said the state was still building the telehealth infrastructure, and services had not yet been provided at the time of our review. Officials were unsure whether OUD services would be delivered via telehealth in schools under the new policy once implementation began. State officials and subject matter experts that we spoke with also reported a range of potential benefits and challenges associated with providing substance use disorder services, including OUD services, in schools via telehealth. (See table 1.) There have also been federal efforts to emphasize the use of telehealth to improve access to OUD services for children. For example, these efforts include the following: In June 2018, CMS issued guidance emphasizing the use of telehealth as a means of improving access to OUD services and noted that states need not necessarily submit a change to their state plan to begin delivering covered Medicaid services through telehealth. Similar to what we heard from experts, this guidance suggests that leveraging technology to provide such services might help with addressing provider shortages, particularly in rural areas. In July 2019, CMS and SAMHSA jointly issued guidance on addressing mental health and substance issues in schools. The guidance states that telehealth for mental health services in schools has been found to be effective. This guidance also emphasizes that telehealth can be helpful for ensuring that Medicaid services are provided to Medicaid beneficiaries who are in rural areas or in areas where qualified practitioners are scarce. HRSA officials we spoke with also described several different HRSA programs from which funds could be used to facilitate or deliver substance use disorder services via telehealth, including in some school-based health centers; however, the officials were not able to determine whether telehealth is being utilized to deliver OUD services in school-based health centers, specifically. The SUPPORT Act also includes a provision for HHS to issue guidance to states on Medicaid coverage of substance use disorder services delivered via telehealth, including in school-based health centers, by fall of 2019. We provided a draft of this report to HHS for review. HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Health and Human Services, the Administrator of the Centers for Medicare & Medicaid Services, the Health Research and Services Administration, the Substance Abuse and Mental Health Services Administration, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Jessica Farb at (202) 512-7114 or farbj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix II. In addition to the contact named above, Rashmi Agarwal (Assistant Director), Kaitlin McConnell (Analyst-in-Charge), Arushi Kumar, Kimberly Lloyd Perrault, Jennifer Rudisill, and Emily J. Weisenberger made key contributions to this report. Also contributing were Drew Long and Ethiene Salgado-Rodriguez. Medicaid: Additional CMS Data and Oversight Needed to Help Ensure Children Receive Recommended Screenings. GAO-19-481. Washington, D.C.: August 16, 2019. Opioid Crisis: Status of Public Health Emergency Authorities. GAO-18- 685R. Washington, D.C.: September 28, 2018. Adolescent and Young Adult Substance Use: Federal Grants for Prevention, Treatment, and Recovery Services and for Research. GAO- 18-606. Washington, D.C.: September 4, 2018. Newborn Health: Federal Action Needed to Address Neonatal Abstinence Syndrome. GAO-18-32. Washington, D.C.: October 4, 2017. Drug Control Policy: Information on Status of Federal Efforts and Key Issues for Preventing Illicit Drug Use. GAO-17-766T. Washington, D.C.: July 26, 2017. Medicaid Expansion: Behavioral Health Treatment Use in Selected States in 2014. GAO-17-529. Washington, D.C.: June 22, 2017. Health Care: Telehealth and Remote Patient Monitoring Use in Medicare and Selected Federal Programs. GAO-17-365. Washington, D.C.: April 14, 2017. School Based Health Centers: Available Information on Federal Funding. GAO-11-18R. Washington, D.C.: October 8, 2010.", "summary": "The misuse of prescription opioid pain relievers and illicit opioids, such as heroin, has contributed to increases in OUD and overdose deaths. Pregnant women with OUD have an increased risk of overdose during the postpartum period. Opioids also caused over half of drug overdose deaths among youth in 2017. Medicaid plays a key role in covering services to treat OUD for low-income women and children. The SUPPORT Act includes a provision for GAO to study Medicaid coverage for pregnant and postpartum women with a substance use disorder, including OUD. The act also includes a provision for GAO to examine children's access to these services, such as through telehealth. This report describes Medicaid coverage of OUD services for (1) pregnant and postpartum women in selected states; (2) children in selected states; and (3) children delivered via telehealth in schools across all states, and utilization of these services. GAO reviewed documentation and interviewed officials from federal agencies within HHS to understand Medicaid coverage of OUD services for pregnant and postpartum women, as well as children. GAO also interviewed officials and reviewed documentation from six states selected for variation in opioid use rates, status of Medicaid expansion, and geographic variation, among other things. GAO also conducted outreach and received responses from 49 of 50 states and the District of Columbia about Medicaid coverage and use of OUD services delivered via telehealth in schools. HHS provided technical comments, which were incorporated as appropriate. All state Medicaid programs are required to provide coverage of health care services to pregnant women with incomes at or below 138 percent of the federal poverty level through 60 days postpartum. With regard to opioid use disorder (OUD), GAO found that six selected state Medicaid programs provide coverage of a range of services for eligible pregnant women with OUD. Specifically, the six states—Alabama, Arkansas, Colorado, Massachusetts, South Dakota, and Texas—covered OUD services, such as screening for opioid use, counseling, and medication-assisted treatment, which combines the use of medications with counseling. In the six selected states, women who are eligible for Medicaid coverage after 60 days postpartum can receive most of the same OUD services that were covered during pregnancy. Furthermore, GAO found that the six selected states also use other sources of funding, such as federal grants, to provide coverage of OUD services for postpartum women who are not eligible for Medicaid. GAO did not review how frequently the OUD services were actually provided to pregnant and postpartum women. GAO found that the state Medicaid programs in all six selected states cover annual screenings for substance use, which includes opioid use, for eligible children. This coverage is provided as part of Medicaid's Early and Periodic Screening, Diagnostic, and Treatment benefit, under which all states are required to cover certain screenings for eligible children under age 21. GAO also found that Medicaid programs in 31 states and the District of Columbia covered OUD services, including screenings, delivered through telehealth in schools. However, state Medicaid officials said they were not aware of any instances of these services being utilized through telehealth in schools. Telehealth can be used to provide clinical care remotely, such as for screening, counseling, and therapy. Such services could be provided, for example, via a video conference on a desktop computer or laptop that connects a student in school with a provider in another location. State officials and experts cited both benefits and challenges with providing OUD services through telehealth in schools. For example, benefits included addressing provider shortages, particularly in rural areas, as well as reducing the amount of time students spend outside of the classroom accessing services. Challenges included lack of needed infrastructure and provider discomfort with using telehealth. Agencies within the Department of Health and Human Services (HHS) have recently issued guidance emphasizing the use of telehealth for OUD services, particularly in schools.", "document_type": "gao"}
{"report": "BOP is a component of DOJ and is responsible for housing male and female federal inmates in a controlled, safe, and humane prison environment while also providing a safe workplace for employees. BOP operates 122 prisons across the United States. These prisons are characterized by five security levels: high, medium, low, minimum, and administrative. Table 1 below provides a description of each of these security levels and the number of prisons at each. According to BOP data, in fiscal year 2019, BOP housed 149,701 inmates in its prisons. During this same time, the BOP employed 32,525 employees, of which 15,664 were correctional officers with responsibility for the day-to-day supervision of the inmates. According to a July 2012 BOP memorandum, BOP was approved to conduct a pilot study on pepper spray. The goals of the pilot were to increase the safety of staff and inmates when responding to incidents involving violence and to prevent injury to staff and inmates due to an assault or serious resistance to staff control. BOP began issuing pepper spray at high security prisons in August 2012 as part of its pilot study. In February 2015, BOP issued a program memorandum requiring employees in high, medium, and administrative security prisons to carry pepper spray. Further, in September 2018, BOP issued a program statement that expanded pepper spray to employees in low security prisons. Figure 1 provides a more detailed time line of events on the use of pepper spray in BOP prisons, including requirements under the Eric Williams Correctional Officer Protection Act of 2015. Pepper spray is a natural inflammatory agent that can cause coughing, tearing, and discharge of excessive mucous when deployed in the facial region. According to BOP training guidance and policy, pepper spray is to be used in incidents that require an immediate use of force (for example, an unplanned use of force because of an attack on staff or an inmate) or a calculated use of force in which employees have time to coordinate their response (for example, when an inmate refuses to vacate his or her cell). For calculated uses of force, employees are to consult medical personnel to determine if an inmate has a medical condition that will exempt the inmate from being pepper sprayed. BOP policy states that employees should receive initial training on pepper spray and annual refresher training. In training, employees are taught effective tactical communication for using pepper spray; use of force policy; how to use pepper spray; and the decontamination process, among other topics. According to BOP’s Use of Force and Application of Restraints policy, a prison’s warden may authorize the use of chemical agents, such as pepper spray, only under the following situations: (1) the inmate is armed or barricaded; or, (2) the inmate cannot be approached without danger to self or others; and (3) it is determined that a delay in bringing the situation under control would constitute a serious hazard to the inmate or others or would result in a major disturbance or serious property damage. Pepper spray, moreover, should only be used when all other reasonable efforts to resolve a situation have failed. This policy further states that staff shall appropriately document incidents involving the use of pepper spray using BOP’s Form 583—Use of Force Report. Form 583 contains fields to enter the date and time of the incident; inmates and staff involved; injuries; medical reports; a description of the incident; and other information, such as the existence of video of the incident. The form is to be completed by the lieutenant on duty at the time of the incident and sequentially forwarded to the captain, assistant warden, warden, and regional office for review. After a Form 583 is completed, the warden, associate warden, health services administrator, and captain at the prison, collectively, conduct an after-action review of the incident to determine if the pepper spray was used in accordance with policy. Results of the after-action review are documented on BOP’s Form 586—After Action Report. According to BOP headquarters officials, in addition to documenting the results of the after- action review, a completed Form 586 often includes recommendations on how to improve the response to such incidents in the future. Incident data captured on Forms 583 and 586 are maintained in BOP’s TRUINTEL database. To enhance BOP employee safety, BOP provides its employees with a variety of protective equipment. BOP generally requires employees working within the secure prison perimeter to carry a radio, body alarm, pepper spray (as appropriate), and keys while on duty. These items are usually checked out from the control center using a chit—a small, brass, circular token inscribed with the BOP employee’s first initial and last name. As of March 2020, some employees also wear stab-resistant vests to help enhance their safety. Although BOP employees are furnished with protective equipment, their first line of defense to protect themselves against an inmate is expected to be their verbal communication with the inmate. BOP policy, training documents, and officials state that effective communication with inmates is essential to officer safety. Figure 2 depicts some of the protective equipment worn by BOP employees operating within the secure prison perimeter of prisons. BOP conducted a pilot study on the issuance of pepper spray from August 2012 through December 2013 at selected high-security prisons. To conduct its study, BOP compared injury sustained by staff and inmates data from immediate use of force incidents in which pepper spray was used to similar incidents in which pepper spray was not used. BOP found that pepper spray was effective in helping to reduce containment time—the amount of time it takes to bring an incident under control—and injury rates. Specifically, containment time of incidents decreased from an average of 4.3 minutes when pepper spray was not used to 2.7 minutes when it was used. This is a reduction of 1.6 minutes in containment time; pepper spray was used mostly in incidents involving two or more inmates, such as fights and assaults. When pepper spray was used, the rate at which staff received no injury increased by 9 percent compared to when pepper spray was not used. Further, the rate at which staff received minor and moderate injury declined by 60 and 76 percent, respectively, compared to when pepper spray was not used; and the inmate injury rate rose slightly, by 2.6 percent, primarily in minor injuries when pepper spray was used; however, BOP concluded this change was not statistically significant. All 90 of the BOP employees we spoke with from United States Penitentiary Atlanta, Federal Correctional Complex Coleman, and Federal Medical Center Devens indicated that pepper spray has been effective in enhancing safety as well as deterring incidents. Generally, these employees noted that pepper spray (1) reduces staff injuries because staff do not have to physically engage with inmates as often to break up incidents, (2) strongly deters incidents from occurring, and (3) allows employees to break up incidents more quickly than if they did not have pepper spray. Pepper spray is not as effective for a small percentage of inmates, such as those with mental illness or those who are under the influence of drugs or alcohol, according to some BOP employees. According to BOP data, in 2018, pepper spray was used in 1,680 incidents as follows: 993 incidents in high security prisons; 557 incidents in medium security prisons; 22 incidents in low security prisons; and 108 incidents in administrative security prisons. Officials from BOP’s Office for Internal Affairs stated that 179 allegations of inappropriate use of force incidents that involved pepper spray were reported from August 2012 through September 2018. Among these cases, BOP’s Office for Internal Affairs has investigated and closed 86. Among these 86 closed cases, investigators found that 21 involved an inappropriate use of pepper spray and were adjudicated in various ways (see table 2). The remaining 93 allegations were still being investigated as of January 2020. According to BOP data, the total cost for pepper spray–specifically the cost to purchase pepper spray canisters and train employees in its use— was approximately $300,000 in fiscal year 2018, which was relatively small compared to BOP’s overall budget. BOP headquarters officials told us that because pepper spray cost information is maintained at the prison level, it would be overly burdensome for them to independently validate the data. Nonetheless, the cost information we received provides a general sense on the extent of costs. Canisters. Officials estimated that a canister of pepper spray costs $7 to $14. Canisters of pepper spray have a shelf-life of approximately 5 years and, according to a BOP headquarters official, are purchased in bulk. As a result, pepper spray does not necessarily need to be purchased on an annual basis. According to BOP officials, each BOP prison contracts with its own supplier rather than using a national contract across all of BOP. BOP headquarters officials told us that pepper spray costs vary across vendors and locations, among other factors. Each BOP prison is responsible for recording and tracking its own budget data on the cost of procuring, training, and issuing BOP employees pepper spray. According to BOP officials, this approach is intended to lower the costs of pepper spray, based on the premise that each prison is able to secure the best market price for pepper spray for its location and for the volume of canisters needed from the vendor. Training. Prison officials told us that pepper spray refresher training is combined with other employee training, making it difficult for them to provide us with specific cost for pepper spray training. All BOP staff are required to take initial and annual refresher training on the use of pepper spray. The initial training lasts about 4 hours, while the annual refresher training lasts about 2 hours. BOP issued a program statement in September 2018, which states that pepper spray is not to be issued to employees working at minimum security prisons. However, the senior BOP officials we interviewed—none of whom said they were involved directly in the policy decision—told us they do not believe the explanatory documentation of the decision to not issue pepper spray to minimum security prisons exists. Officials stated that the decision was likely made for several reasons: inmates at minimum security prisons are usually nonviolent offenders, incidents at minimum security prisons are usually very minor and do not require the use of pepper spray, the concern that public perception of using pepper spray on inmates at minimum security prisons would not be positive, and canisters of pepper spray would expire before they would be used at minimum security prisons. BOP officials we spoke with also stated that inmates at minimum security prisons are less likely than inmates at other security level prisons to become involved in incidents because they do not want to be reassigned to a higher security prison. We found, nonetheless, that BOP’s TRUINTEL database shows that incidents do occur at these prisons— some of which have led to assaults, minor injuries and death. Based on our analysis of BOP incident data from TRUINTEL, we found that in 2018 there were 47 reported incidents in the seven BOP minimum security prisons. These incidents included assaults on staff and other inmates; sexual harassment; and fighting, among others. Five of the incidents resulted in minor injuries to 10 BOP employees, and 18 incidents resulted in minor injuries to inmates. Further, one incident led to an inmate fatality. Additionally, during our site visits, 56 out of 73 officials across various security levels stated that deployment of pepper spray should be expanded to minimum security prisons because it would give employees an additional tool to protect their safety. BOP headquarters officials told us they believe the agency’s decision to not issue pepper spray to minimum security prisons remains appropriate. Regarding the 47 incidents that occurred at minimum security prisons in 2018, officials stated that many of the confrontational incidents occurring at these prisons can be handled using verbal commands. While a decision to not issue pepper spray at minimum security prisons may be justified based on an analysis of relevant information, BOP officials could not provide documentation of such analysis to support its decision. This analysis could include assessing available incident data at minimum security prisons and determining whether any of the incidents could have been prevented or handled more effectively if the officer on duty was carrying pepper spray. Additionally, BOP employee perspectives on issuing pepper spray at minimum security prisons is another possible source of relevant information that could be included in an analysis to inform BOP’s decision. BOP issued policies in 2015 and 2018 that stated that while the preferred method of resolving issues with inmates is through a verbal intervention, there are instances where other means will be required to restore order. In addition, the policies state that the safety of staff, inmate(s), or others in any dangerous encounter is paramount and that the use of force— including use of pepper spray—may be needed to ensure safety. According to Standards for Internal Control in the Federal Government, management should use quality information to make informed decisions and to evaluate the entity’s performance in achieving key objectives and addressing risks—in this case, the possible safety risks to BOP employees and inmates. By conducting an analysis on available BOP data on incidents that have occurred at minimum security prisons, employee perspectives on the value of having pepper spray at such prisons, and other relevant data, such as cost data, as appropriate, BOP would have useful data with which to inform its decision on whether or not to authorize pepper spray for employees at minimum security prisons. Four BOP headquarters officials, 18 wardens and their executive staff, and 10 union officials rated the potential impact of 15 selected factors (see app. I) on the safety of BOP employees in prisons. BOP officials rated the following five factors as having the most significant impact on BOP employee safety in prisons: (1) corrections officer understaffing, (2) disruptive inmate behavior due to illegal drugs, (3) inmate use of unauthorized communication devices, (4) inmate gangs, and (5) insufficient corrections training. See figure 3 for a diagram of the top five factors identified across the different groups of BOP officials who responded to the structured questions. Across all three groups, corrections officer understaffing was rated among the top five factors. No other factor was equally represented. For at least two groups, inmate use of unauthorized communication devices, disruptive inmate behavior due to illegal drugs, and insufficient information-sharing among managers and staff were rated among the top five factors. When asked to identify any additional challenges beyond the selected factors we included, BOP officials we interviewed stated they were not aware of other challenges. BOP officials told us that they are taking steps to mitigate some of the challenges officials we interviewed indicated are impacting employee safety in prisons. Officials identified the following: Corrections officer understaffing. Corrections officer understaffing refers to the staffing level—usually measured by the inmate-to-staff ratio—being too low to adequately prevent violence and maintain a safe prison. Among the BOP headquarters officials, wardens and their executive staff, and union officials we interviewed, two underlying reasons generally cited for understaffing conditions were hiring freezes and difficulty recruiting new correctional officers due to low starting salaries. According to the BOP Director’s testimony before the Senate Judiciary Committee in November 2019, building adequate staffing at BOP prisons is one of her highest priorities. The Director stated that BOP established 10-percent recruitment, relocation, and retention incentives for hard-to-fill positions; established a higher entry pay scale for experienced new correctional established a 5-percent nationwide retention incentive for retirement- used 3,000 temporary positions to help ensure seamless succession planning by avoiding the lag to hire someone to fill a position. We issued a report in December 2017 on BOP’s use of retention incentives. At that time, we found that BOP had taken steps to determine workforce needs and how to fill those needs but had not strategically planned for and evaluated its use of retention incentives. We recommended that BOP include in its strategic human capital operating plan (1) human capital goals; and (2) strategies on how human capital flexibilities, including retention incentives, will be used to achieve these goals. We also recommended that BOP evaluate the effectiveness of its use of retention incentives to determine whether the incentives have helped achieve BOP’s human capital goals or if adjustments in retention incentives are needed. DOJ concurred, and BOP implemented our first recommendation by drafting a human capital plan with goals and strategies for how retention incentives could be used to meet those goals. To implement our second recommendation, BOP conducted an analysis of its use of retention incentives and their effect on retaining BOP employees. Disruptive inmate behavior due to illegal drugs. According to BOP officials, some inmates obtain illegal synthetic drugs by mail. These drugs are sprayed onto inmate mail and other documents before being sent to the inmate in prison. Inmates burn the mail to get high off of the synthetic drug. In addition to the threat to the inmate population posed by inmates who are behaving under the influence of the drugs, entry of these drugs can expose staff—including those handling the mail—-to hazardous chemicals. In an effort to stop illegal drugs from entering prisons by this method, according to BOP officials we spoke with and the BOP Director in her November 2019 testimony, some prisons are photocopying mail before it is delivered to inmates. For example, officials at one prison we visited told us they photocopy inmates’ mail. Further, a BOP headquarters official stated that BOP is piloting various mail-scanning technologies aimed at reducing the number of drugs entering prisons. Inmates’ use of unauthorized communication devices. According to BOP officials and the BOP Director’s testimony, inmates’ possession of cell phones is a major problem. BOP officials stated that, in an effort to stop the unauthorized use of cell phones, some prison officials are using specialized equipment to detect cell phone usage and are exploring options to use cell phone jammers. We reported in September 2011 that BOP and selected state officials told us that cell phones were a major security concern because they allow inmates to hold unmonitored conversations, for example, to sell drugs or harass individuals. We recommended that BOP’s Director formulate evaluation plans for cell phone detection technology to aid decision-making, require BOP staff to use these plans, and enhance regional collaboration with states. DOJ concurred with our recommendations, and BOP addressed them by developing policy and testing procedures to improve their ability to evaluate new technology. BOP also established plans to enhance collaborative information-sharing with state and local agencies on combating cell phone smuggling and use. Working in a federal prison presents inherent risks. Since 2018, BOP has authorized the use of pepper spray at all prison security levels with the exception of minimum security prisons. BOP’s issuance of pepper spray was supported by evidentiary information—that is, its pilot study indicated that pepper spray was an effective tool for enhancing staff safety. Notably, BOP’s current policy on pepper spray allowance does not extend to minimum security prisons. While BOP was not able to provide us with a documented analysis behind the nonissuance to minimum security prisons, the officials we interviewed made several arguments in support of the decision. While their arguments may hold merit, we found evidence based on our limited analysis that appears to question their underlying decision. To the extent that officials are operating under assumptions not fully examined, BOP is missing a potential opportunity to enhance the safety of its correctional officers. We believe that our concerns are amplified by our finding that a majority of BOP frontline employees want pepper spray expanded to minimum security prisons. Similar to the decision to issue pepper spray to other levels was based on pilot information, BOP has an opportunity to bring—either for or against issuance—a better case forward. Analyzing available data on incidents that have occurred at minimum security prisons, such as determining whether any of them could have been prevented or handled more effectively with pepper spray, and considering BOP employees’ perspectives, BOP could inform its decision whether to authorize pepper spray for employees at these prisons. We are making the following recommendation to BOP: The Director of BOP should conduct an analysis, using available incident and cost data, and other information as appropriate, to determine if the current decision to not issue pepper spray to minimum security prisons should remain in effect. (Recommendation 1) We provided a draft of this product to DOJ, including BOP, for review and comment. DOJ concurred with our recommendation and told us they had no comments on the draft report. DOJ did provide technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Attorney General, the BOP Director, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or goodwing@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Throughout our audit work, we asked Bureau of Prisons (BOP) officials with whom we interviewed at the headquarters and selected prisons about factors that impact the safety of BOP employees, as well as efforts, if any, they had made to mitigate those factors. We specifically targeted three groups of BOP personnel—BOP headquarters, wardens and their executive staff, and union officials—to rate the impact of 15 selected factors on employee safety at the groups and by prison security level. We then analyzed their responses and identified the top five factors that these BOP officials identified as having an impact on employee safety. We received responses from four BOP headquarters officials, 18 wardens and their executive staff, and 10 union officials. Officials were provided the structured questions (see below) in advance of the site visit, and the team recorded their responses during the interview. We held one interview with four Bureau of Prisons (BOP) headquarters officials, nine interviews with 18 wardens and their executive staff, and seven interviews with 10 union officials about 15 selected factors that impact the safety of BOP employees, using a structured questions set (see app. I). These officials’ responses, which are broken down by group and security level, are presented in the figures below. In addition to the contact named above, Brett Fallavollita (Assistant Director), Sonja S. Ware (Analyst-in-Charge), Anthony DeFrank, and Emily Martin made key contributions to this report. Willie Commons III, Elizabeth Dretsch, Eric Hauswirth, and Susan Hsu also contributed to this work.", "summary": "Within the Department of Justice, BOP is responsible for housing male and female federal inmates at 122 prisons in a safe environment for staff and inmates. Pepper spray is one of the methods BOP employees use to enhance their safety. The Eric Williams Correctional Officer Protection Act of 2015 includes a provision for GAO to examine certain matters related to the issuance of pepper spray to officers and employees in BOP prisons. This report addresses (1) what is known about the effectiveness and cost of issuing pepper spray in BOP's high, medium, low, and administrative security prisons; (2) BOP's position on expanding the issuance of pepper spray to minimum security prisons and the support used to make this decision; and (3) the challenges, if any, BOP officials identified as affecting the safety of BOP employees and the steps, if any, BOP has taken to address them. To address these objectives, GAO reviewed BOP policies, guidance, incident reports, and cost data on pepper spray use and interviewed knowledgeable officials at BOP headquarters and nine prisons at three locations, selected to represent varying security levels and other characteristics. Pepper spray is an effective tool for reducing the time needed to control incidents involving inmates and for reducing any related injury to Bureau of Prisons (BOP) employees, according to a 2012 BOP pilot study and BOP officials interviewed by GAO. BOP first issued pepper spray to employees in high security prisons in August 2012 and to medium, low, and administrative security prisons in subsequent years. Officials estimated that a canister of pepper spray costs $7 to $14. However, the total cost to purchase pepper spray and train employees on its use is not readily available because purchases are tracked at the prison level, and pepper spray training costs are commingled with other training costs. BOP determined that it would not issue pepper spray to minimum security prisons. BOP headquarters officials stated that this decision was made because inmates at such prisons are usually nonviolent offenders, among other reasons. However, GAO's analysis of BOP data found 47 reported incidents that included assaults on staff and other inmates across BOP's seven minimum security prisons in 2018. In addition, 56 of 73 officials GAO interviewed said pepper spray should be expanded to minimum security prisons. BOP officials stated they were not aware of an analysis of incident data or other information to support its decision but said that the decision remains appropriate. However, by analyzing available data on incidents that have occurred at minimum security prisons, BOP could better inform its decision on whether to issue pepper spray to employees at minimum security prisons. BOP officials rated the following factors as having the most significant impact on BOP employee safety, as shown in the figure below. BOP officials stated that they are taking steps to mitigate factors impacting safety. GAO recommends that BOP conduct an analysis to determine if its decision to not issue pepper spray to minimum security prisons should remain in effect. The Department of Justice concurred with the recommendation.", "document_type": "gao"}
{"report": "Navy shipbuilding is a costly and complex endeavor that requires billions of dollars to develop, design, and construct ships. However, the acquisition phase of a ship’s life cycle only accounts for approximately 30 percent of a ship program’s total life-cycle cost. Notionally, the remaining 70 percent of the life-cycle cost of a ship program is incurred after the Navy delivers new ships to the fleet during the phase known as O&S. DOD guidance states that these long-term sustainment costs are determined in large part by decisions made early in the acquisition process. Approximately 80 percent of a program’s O&S costs fixed at the time the shipbuilding program’s requirements are set and the ship is designed. Additionally, we have found that once these decisions are made, it can be very difficult and costly to make changes if sustainment improvements are needed. According to DOD, operational support is a function of several related factors—reliability, availability, maintainability, and cost—that are determined in large part by decisions made before the start of construction. Reliability is the probability that an item, such as a system, can perform a required function under stated conditions for a specified period of time. Availability is a measure of the degree to which an item is in an operable state and can be called upon to work at the start of a mission and at an unknown (random) point in time. In other words, the degree to which a system is operable and available for mission tasking when needed. Maintainability is the ability of an item, such as a system, to be retained in or restored to a specified condition when maintenance is performed by skilled personnel using prescribed procedures and resources, at each prescribed level of maintenance and repair. Cost refers to the O&S costs associated with sustaining the ship. When planning for and executing ship sustainment, DOD guidance states that the program manager’s goal is generally to find a solution that maximizes reliability, availability, and maintainability within cost constraints. As the Navy acquires its ships, it makes a series of decisions that have implications for how a ship class can be affordably sustained, including decisions about engineering, ship design, equipment selection, and planned maintenance approaches. As such, DOD guidance advises acquisition programs, including Navy shipbuilding programs, to plan for and design reliability, availability, and maintainability into the weapon system early in the acquisition effort. For the purposes of this review, we define early in the acquisition process as the time period between the beginning of the program and the start of construction on the lead ship. Giving attention to these sustainment issues early in the acquisition process is intended to help programs ensure that their ships will be sustainable and affordable over their entire life cycle. Conversely, if reliability, availability, and maintainability are not adequately designed into the ship, there is a risk the ship will cost more to own and operate than expected and will not be available for use when needed by the fleet. Since Navy ships are comprised of numerous systems that need to work together, planning for sustainment and designing reliability, availability, and maintainability into a ship is a complicated task. Most Navy ships can accomplish several different missions, and accomplishing these missions usually requires a set of mechanical, electrical, combat, information technology, and other systems to work together. Each of these systems individually needs to be reliable, available, and maintainable in order for the ship as a whole to be sustainable. As such, addressing sustainment during the acquisition process is an effort that requires coordination and input from a variety of officials associated with the program, including the program manager, requirements officials, ship design managers, engineers, PSMs, and others. DOD acquires new weapons, including ships, for its warfighters through a process described in two key acquisition policies: Department of Defense Directive 5000.01, which establishes the overarching framework for the Defense Acquisition System, and Department of Defense Instruction 5000.02, which implements the Defense Acquisition System. These policies provide management principles and detailed procedures, respectively, for the execution and management of defense acquisition programs. Specifically, these policies establish the phases of the acquisition process, key milestone decision points, required acquisition documentation, and roles and responsibilities for acquisition officials, among other things. Under this framework, shipbuilding programs move through several acquisition phases, including requirements setting, material solution analysis, technology development, ship design, ship construction, deployment, and sustainment. In order to proceed through the acquisition process, shipbuilding programs must be reviewed periodically at key decision points, called milestones, at which a Milestone Decision Authority assesses the program’s progress in achieving its goals. These milestones typically coincide with significant increases in the resources devoted to the program. To ensure senior leadership is well- informed at these decision points, shipbuilding programs are generally required to create or update key acquisition documents for milestone reviews that contain information on the program’s requirements, costs, and schedule, among other things. The Navy has also established its own acquisition policy and processes to supplement the DOD-wide acquisition policies and to oversee acquisitions managed internally to the Navy. The Navy’s acquisition policy, Secretary of the Navy Instruction 5000.2, provides instructions for implementing the Defense Acquisition System within the Navy, as well as additional Navy-specific acquisition procedures. In particular, Navy acquisition policy establishes a series of seven Navy decision points throughout the acquisition process, called Gate reviews, which complement the DOD milestones. These Gate reviews are split into two phases that the Navy calls passes: the first is led by the CNO and focuses on requirements setting and the second is led by the Assistant Secretary of the Navy for Research, Development, and Acquisition (ASN (RD&A)) and focuses on acquisition. As programs move through the acquisition process, Navy leadership—comprised of officials from the acquisition, requirements, resources, and warfighting communities— convenes Gate reviews to conduct oversight and ensure programs are on track to achieve their acquisition and sustainment goals. Each Gate review has a different objective and list of topics that need to be included in the Gate briefing. Lastly, DOD and Navy policy both allow for the Milestone Decision Authority to tailor the acquisition process outlined in these policies. Figure 1 depicts the acquisition process for Navy shipbuilding programs, as established by DOD and Navy acquisition policies. The Navy’s acquisition policy also details the acquisition responsibilities of key Navy officials, including the ASN (RD&A), the CNO, and program managers. The CNO and the ASN (RD&A) are key Navy leaders who chair the Gate review process and approve acquisition documents. For most shipbuilding programs, the ASN (RD&A) also serves as the decision authority to approve the advancement of these programs through the acquisition process at the appropriate milestones. Further, the policy enclosures delineate various elements of acquisition programs, such as systems engineering, testing, and sustainment planning. DOD and Navy acquisition policies both include requirements for shipbuilding programs to consider sustainment throughout the acquisition process. For instance, prior to Milestone A, DOD policy states that sustainment planning and considerations should inform the development of program requirements and early ship design decisions. As programs move into the design and construction phases, programs are to develop a comprehensive product support package and evaluate it through engineering reviews and other tests to ensure it is sufficient to meet sustainment requirements and affordability targets. The planning documents that comprise these support packages, such as life-cycle sustainment plans (LCSPs), are intended to set the foundation for how the fleet will sustain a class of ships. In addition to the requirements set in DOD and Navy policies, Congress has passed laws related to increasing DOD and Navy attention on sustainment throughout the acquisition process. Chief among these is the creation of the role of the PSM. A PSM should develop and implement a comprehensive product support strategy for weapon systems, among other things. More recently, Congress has directed organizational changes related to DOD and Navy acquisition leaderships’ attention to sustainment. In response, the Navy has added a sustainment function to the ASN (RD&A)’s portfolio. The Navy implemented this direction in fiscal year 2020 with the appointment of a Deputy Assistant Secretary for Sustainment that reports to the ASN (RD&A). Congress has also established several requirements related to DOD and Navy management of acquisition programs’ O&S costs, sustainment planning, and sustainment reporting. For example, statute requires weapon system programs to consider, where appropriate, sustainment in key acquisition documents, such as acquisition strategies, designs, contracting, and cost estimates. Additionally, statute requires DOD to provide Congress with annual Selected Acquisition Reports that have sustainment and life-cycle cost information. Shipbuilding programs are required to develop a suite of acquisition documents that provide information about the goals of the program and how the program office is developing and executing to these goals, pursuant to DOD and Navy acquisition policies. Many of these key acquisition documents contain information about the program’s sustainment requirements and plans, as discussed below. The Capability Development Document should define the program’s operational requirements, including the program’s key performance parameters. Key performance parameters are the most critical requirements a system must demonstrate to deliver an effective military capability. In 2007, DOD updated its requirements setting policy, called the Joint Capabilities Integration and Development System, to require all programs to establish key performance parameters for sustainment in response to concerns that acquisition programs were not adequately planning for sustainment. This requirement helps ensure that acquisition programs provide a weapon system to the warfighter with optimal availability and reliability at an affordable price. The sustainment key performance parameter is comprised of two measures—operational availability and materiel availability—which addresses the availability of the ship while in operations and under maintenance, respectively: Operational availability measures the probability that a system will be ready for use when expected. This requirement helps programs determine how reliable, maintainable, and supportable a system needs to be. Operational availability is also understood as the percentage of time a ship can perform its primary mission. Materiel availability measures the percentage of the total inventory of a system that is operationally capable based on materiel condition, which for ship platforms, is the percentage of a ship class available for deployment. This metric helps programs determine how many ships to buy in order to meet planned deployment schedules. This requirement should inform decisions that could increase or decrease planned maintenance time for a shipbuilding program. According to DOD guidance, the operational and materiel availability requirements should be considered in tandem to produce ships that work as expected and are available when needed, as shown in figure 2 below. During the acquisition process, the operational availability requirement should inform decisions about how to best increase reliability for systems needed to meet the key performance parameter. To do this, engineers can, among other things: (1) design systems that require less frequent maintenance, (2) add redundancy to key systems, or (3) ensure that systems can be fixed quickly and cheaply. At the same time, the materiel availability requirement should inform how many ships are purchased based on the quantity needed to accomplish missions at any one time. It also informs acquisition decisions that could affect the length of maintenance availabilities, such as maintenance time needed to repair or replace key components. The Life-Cycle Cost Estimate should provide information on the total estimated cost to develop, build, deploy, sustain, and dispose of a ship class over its life cycle, regardless of funding source. The life- cycle cost estimate is based on program objectives and operational requirements for the ship class. It should reflect a realistic appraisal of the program’s risks and the level of cost most likely to be realized. The life-cycle cost estimate includes O&S costs, which provide information on the estimated costs for crewing, operations, maintenance, sustaining support, continuing system improvements, and indirect support. The Acquisition Program Baseline (APB) is an overarching acquisition document that describes the shipbuilding program and presents the program’s approved cost, schedule, and performance goals. The APB is a formal, binding agreement between the Milestone Decision Authority, Program Manager, and their acquisition chain of command to be used for tracking and reporting on the program. The Life-Cycle Sustainment Plan (LCSP) should document the program’s product support strategy and governs planning for sustainment during the acquisition process, as well as the execution of sustainment activities after ships are delivered to the fleet. The LCSP describes the efforts necessary to develop and integrate sustainment requirements into the ship’s development, design, procurement, testing, fielding, and operation. It also lists the activities necessary for the shipbuilding program to develop, implement, and deliver a product support package that maintains affordable operational effectiveness over the ship’s life cycle. For example, the LCSP should contain information on sustainment engineering, O&S cost estimates and affordability constraints, reliability analysis, and sustainment contracts, among other things. The Independent Logistics Assessment (ILA) should be an impartial analysis of a program’s sustainment planning and execution to determine its ability to meet established performance and sustainment requirements. The ILA is intended to assess the adequacy of the program’s product support strategy, product support risks that are likely to drive future O&S costs, changes to system design that could reduce O&S costs, and effective strategies for managing O&S costs. According to DOD guidance, programs should use the results of the ILA to improve sustainment outcomes. There are a large number of Navy stakeholders involved in the effort to design, build, and support a ship class over its life cycle. In general, the acquisition community is led by the ASN (RD&A), while the operations and sustainment community is led by the CNO. Naval Sea Systems Command (NAVSEA) provides support to both the acquisition and sustainment communities and is comprised of experts across multiple disciplines. Figure 3 provides more information on the various acquisition and sustainment stakeholders that support the Navy’s ship classes. The ASN (RD&A) acts as the Navy Service Acquisition Executive and oversees the Navy’s shipbuilding program offices. Program Executive Offices are responsible for the life cycle management of their assigned programs. The program executive office is led by a program executive officer who, according to DOD’s updated acquition policy, should balance the risk, cost, schedule, performance, interoperability, sustainability, and affordability of a portfolio of acquisition programs and deliver an integrated suite of mission capability to users. For ships, there is a shipbuilding program office that is responsible for acquiring ships and an in-service program office that supports ships in sustainment. In some cases, these program offices are located within the same program executive office while, in other cases, these offices are split between different Navy organizations (typically the program executive office and NAVSEA). As such, the Navy’s shipbuilding programs and some program executive offices do not have responsibility for ship programs throughout their life cycle. The shipbuilding program offices manage their assigned shipbuilding programs through program initiation, technology development, ship design, construction, testing, and delivery. Acquisition program managers lead shipbuilding program offices and are responsible for the management of a program. Acquisition policies delineate a number of sustainment-related responsibilities for acquisition program managers, such as: developing and implementing an LCSP to inform acquisition and sustainment phases of the program; developing strategies for managing intellectual property; using systems engineering to identify tradeoffs between life-cycle costs and performance requirements during design and construction; implementing a comprehensive reliability and maintainability engineering program; developing an obsolescence management plan; monitoring the program’s performance against its sustainment requirements and developing strategies to improve operational availability, O&S affordability, maintainability, and reliability, as necessary; and working with a PSM, among other things. Product Support Managers (PSMs) generally work with the acquisition program manager and are tasked with developing and implementing a comprehensive product support strategy for their assigned programs. PSMs are supposed to ensure that a comprehensive product support strategy is developed and implemented. The CNO is the senior military officer of the Department of the Navy, overseeing the Navy’s fleet and NAVSEA, among other organizations. The CNO also has acquisition responsibilities, such as approving a shipbuilding program’s requirements and determining whether to accept delivery of ships from the shipbuilders. The Office of the Chief of Naval Operations (OPNAV) is a collection of offices under the purview of the CNO responsible for various functions necessary for the operation of the Navy. For example, there are divisions within OPNAV that manage the Navy’s budget, logistics, and requirements setting process, among other things. The operational fleet forces (fleet) of the Navy, including operational units and type commands, assume full financial responsibility for operating and maintaining ships. Naval Supply Systems Command provides supply and services support to the Navy by managing supply chains and inventory for Navy aircraft, surface ships, submarines, associated weapons systems, and non-nuclear ordinance stockpiles. NAVSEA is responsible for providing expertise in designing, engineering, building, buying, and maintaining ships, submarines, and combat systems to meet the fleet’s operational requirements. NAVSEA is comprised of directorates and warfare centers that specialize in these areas of expertise. NAVSEA reports to the CNO, but also supports the shipbuilding program offices, and is organized by the following specialties, among others: Naval Sea Systems Command Engineering Directorate (NAVSEA 05) is an engineering command that is comprised of cost estimators, ship designers, systems engineers, and other technical experts. Among other things, this office is responsible for the development of life-cycle cost estimates and systems engineering for ships. Naval Sea Systems Command Acquisition and Commonality Directorate (NAVSEA 06) is a command that brings together personnel dedicated to bridging communication gaps between government and industry, in order to enable cost reductions and commonality throughout the acquisition life cycle. Among other things, this office leads the Navy’s ILA process. Naval Warfare Centers are a group of centers that offer services on a fee-for-service basis, including: obsolescence mitigation, in-service engineering, and data analysis, among many other tasks. Shipbuilding program officials did not identify or mitigate sustainment risks during the acquisition process that subsequently resulted in significant and costly problems for the fleet. During the course of our review, the fleet identified 150 problems that affected multiple ships in a class. These problems resulted in more effort and cost for the fleet in sustainment than expected. In particular, we estimated that the Navy’s fleet has spent or is planning to spend at least $4.2 billion to mitigate and correct approximately 30 percent of these problems beyond what was planned for during the acquisition process. We could not quantify the cost impact of the remaining 70 percent of problems because the Navy was unable to provide data on the cost to correct them. Examples from the SSN 774, LPD 17 Flight I, and LHD 8/LHA 6 ship classes illustrate how shipbuilding program officials did not identify and mitigate sustainment risk during the acquisition process, which resulted in significant and costly maintenance paid for by the fleet once realized. The fleet identified 150 sustainment problems affecting multiple ships in a class that required more sustainment effort than planned for during acquisition, which we verified through Navy data and documentation. Officials in the fleet, such as operators, maintainers, and engineers, reported these problems to us as major class-wide problems requiring more sustainment effort than planned. These problems manifested after ships were delivered and most of these problems have yet to be resolved. Where data were available on the cost to correct the problems, we estimated that the fleet paid at least $4.2 billion and had to perform more onerous maintenance than planned. These problems stemmed from shipbuilding program officials not identifying or mitigating sustainment risks in sustainment planning during the acquisition process, before ships were delivered to the fleet. Figure 4 summarizes the number of problems among multiple ships in the same class that required more sustainment effort than the shipbuilding programs’ had planned. It also reflects the costs associated with fixing these problems for the 30 percent of the problems where we could identify these costs based on available data. According to fleet leadership, these problems contribute to the fleet’s inability to maintain ships at planned cost and schedule, which we have previously found is a significant Navy-wide issue. In part to accommodate this extra effort, the Navy has experienced maintenance delays and has had to defer planned maintenance for ships in operations that the fleet determined was not as urgent as other maintenance needs. For instance these problems have contributed to: nearly 5,300 total days of delays to planned maintenance availabilities since 2012 on ships built during the last 10 years, new ships deferring planned maintenance, and insufficient funding to meet maintenance needs. To generate the list of 150 problems, we interviewed operators and maintainers for the shipbuilding programs in our review and asked them to discuss problems that occurred across multiple ships in the same class. We then verified these problems with available Navy data on system reliability and equipment failures. The list of problems only includes those that stemmed from risks that were not identified, evaluated, or mitigated by the shipbuilding program offices in their sustainment planning during the acquisition process. The list does not include problems that can be attributed to normal wear and tear or problems caused by sailor error. The estimate of $4.2 billion in additional costs to address these problems includes the fleet’s cost to correct or mitigate problems, but excludes costs associated with day-to-day maintenance that the fleet must perform. If the Navy’s fleet chooses to correct a problem, it typically requires the Navy to replace systems on ships that have already been delivered to the fleet or are under contract, which can be a costly undertaking. According to fleet maintenance officials, if a permanent correction is not implemented, the Navy’s operators and maintainers typically have to incorporate a more onerous maintenance approach than expected. The effects of more onerous day-to-day maintenance costs are hard to quantify using available Navy data. For example, the Navy used a brand new toilet and sewage system on the CVN 77 and 78, similar to what is on a commercial aircraft, but increased in scale for a crew of over 4,000 people. To address unexpected and frequent clogging of the system, the Navy has determined that it needs to acid flush the CVN 77 and 78’s sewage system on a regular basis, which is an unplanned maintenance action for the entire service life of the ship. According to fleet maintenance officials, while each acid flush costs about $400,000, the Navy has yet to determine how often and for how many ships this action will need to be repeated, making the full cost impact difficult to quantify. We generally did not include these types of ongoing costs in our calculation. In our cost calculation, we also excluded costs associated with adding sailors to ships to address maintenance gaps because sailors have been added for many reasons, making it difficult to isolate the money spent on sailors to address equipment problems. For instance, we omitted the $225 million that the fleet plans to spend to add sailors to LCS class ships, even though the Navy is taking this action in part, to ensure that the ship’s crew can perform necessary maintenance. We determined that the 150 problems identified by the fleet generally fall into three categories: (1) problems maintaining commercial equipment on ships, (2) ship design that did not effectively consider maintainability, and (3) untested sustainment assumptions that turned out to be incorrect after ships were delivered to the fleet. We found that nearly all Navy shipbuilding programs we reviewed experienced problems in each of these three categories, as shown in figure 5. The following examples illustrate each of the three categories of problems: Problems maintaining commercial equipment on ships. Dozens of primarily commercial systems on multiple SSN 774 class submarines are experiencing unexpected failures. During the acquisition process, the Navy based sustainment planning decisions on the assumption that these parts would last for the life of the submarine without the need for any maintenance. According to officials, the Navy did not verify these assumptions and now at least 16 of these systems require scheduled maintenance and several more systems need periodic updates that were not previously planned. As a result, as we have previously found, operators and maintainers have had difficulty obtaining the spare parts, accomplishing this planned maintenance within resource constraints. Ship designs that did not effectively consider maintainability. The Navy used a new design for CVN 77’s stores elevators, which are used to move provisions between decks. However, among other issues, the elevators are too small to fit a standard sized pallet jack. Thus, provisions cannot be loaded or unloaded with a pallet jack or a forklift and must be manually unpacked and stacked by hand on to the elevator. Unloading is further complicated, according to the ship’s crew, because the elevator doors are so small that the average sailor cannot stand up as they enter and exit the elevator. The fleet has mitigated a few of these problems, but a redesign of the elevator would be necessary to fit standard pallets and fully resolve the other problems. Untested sustainment assumptions that turned out to be incorrect after ships were delivered to the fleet. The Navy had originally planned to use a contractor to conduct the majority of LCS maintenance. However, the fleet determined that a heavy reliance on contracted support is inefficient for maintaining and sustaining the LCS and is in the process of establishing maintenance teams comprised of Navy personnel. Since it planned to use contractor support, the LCS shipbuilding program officials stated that they did not purchase the technical documentation necessary to maintain the commercial equipment used on the ship. As a result, fleet engineers told us that they are now attempting to buy and develop the necessary maintenance data, which adds cost and complexity to the maintenance process. The following section highlights four of the 150 problems identified by the fleet. Other examples from among these 150 issues are discussed throughout the report when appropriate to illustrate how the acquisition process contributed to sustainment problems. A full listing of the 150 problems is in a version of this report that is for official use only. In an effort to improve sustainment of the LPD 17 class ships, the Navy decided to install titanium piping to carry seawater for firefighting and to cool machinery instead of copper-nickel piping because of its lighter weight and increased durability. However, instead of saving effort in sustainment, these pipes required more maintenance effort than planned and, in many cases, eventually had to be replaced. Early in the acquisition process, the Navy studied this decision and discovered that unlike copper-nickel piping, titanium piping carrying seawater is susceptible to “biofouling”—meaning sea life such as shellfish grow inside the pipes—as shown in figure 6. To prevent biofouling, Navy engineers determined that a chlorination system—which adds chlorine to seawater entering the ship in order to kill biological material in the water—and a dechlorination system—which removes the chlorine before the seawater is dumped from the ship— would be needed and included specifications for the shipbuilder to install these systems. Then, according to shipbuilding program officials, after the ship was on contract, the shipbuilder reported to the Navy that it could not find suitable chlorination and dechlorination systems. The program office decided to proceed with ship construction absent these systems and evaluate the extent of the biofouling problem after ship delivery. We reviewed the LPD 17 program’s sustainment planning documents and found that a discussion of this sustainment risk was not included in any of the maintenance planning documents, and, according to the fleet, this risk was not communicated to the Navy’s maintenance organizations. In July 2009, about one year after the lead ship was provided to the fleet, Navy operators and maintainers began to notice biofouling in the piping system. Biofouling degraded the functionality of a number of other systems on the ship that depend on the water delivered by the piping system, resulting in overheating of main and ship service engines and loss of electric power generation, among other problems. To address these and related issues across the LPD 17 class, the Navy’s fleet spent at least $250 million to: (1) buy and install new copper-nickel piping that is now costlier, heavier, and not as durable as titanium; (2) install chlorination systems that were later found to be unreliable, requiring significant maintenance; and (3) conduct unplanned maintenance and replace systems that broke due to shellfish contamination, among other interventions. The Navy’s attack submarines utilize a special covering on the hull. However, as shown in figure 7, portions of the hull-covering have de- bonded from the hull resulting in additional maintenance requirements during scheduled availabilities. Shipbuilding program officials told us that, during the acquisition process, they did not analyze how long the special hull treatment would last even though it is a critical technology. According to the program office, they now have identified the root cause and have continuously conducted engineering analysis to monitor and improve the material and construction processes. Due to the 5 to 6 year process of building a submarine, the time from identification to proven success can be 8 to 10 years, which is a long time to wait to know if a potential solution works in operations. However, in the meantime, the shipbuilding program has continued to deliver submarines to the fleet without knowing how long the special hull treatment will adhere to the vessel. As a result, maintainers cannot effectively plan for special hull treatment replacements in advance and, instead, are replacing material as needed. Performing timely and necessary maintenance is further complicated because it takes up to two years to receive this material after the Navy orders it from the manufacturer. Currently, Navy maintainers are budgeting $735 million to address the missing hull treatment on 11 of the 14 submarines constructed prior to implementing the potential solution. To enable reduced crew sizes and sustainment costs, the Navy chose to use an automated machinery control system on LHD 8 and LHA 6. Sailors describe the machinery control system as a vital software-based system that controls the operation of 92 percent of shipboard systems. The Navy initially sought to purchase a highly-automated commercial system that would perform tasks previously completed by the ship’s crew. However, according to the shipbuilding program, during the acquisition process, it verified reliability testing conducted by the manufacturer of the system. At the end of the shipbuilding process, the Navy discovered that this system required more maintenance and sustainment effort than planned. Specifically, the Navy’s Board of Inspection and Survey—the organization that inspects ships prior to delivery—discovered problems with this system on LHD 8 in March 2009. The Board of Inspection and Survey identified false alarms and a lack of technical documentation as a serious defect. Specifically, the test report found that the system’s spurious and numerous alarms created an environment whereby the ship’s sailors would be conditioned to ignore alarms and that more sailors would be needed to monitor the ship’s systems. Nevertheless, the CNO decided to take delivery of the ship and the shipbuilding program did not correct these problems prior to providing the ship to the fleet. Additional problems emerged on LHD 8’s first deployment, such as overheating that led to failure of the electrical distribution system resulting in loss of power on multiple occasions. However, the technical data provided by the manufacturer, according to Navy engineers, was insufficient for the sailors to operate, troubleshoot, and repair the system. Further, according to ship engineers and the shipbuilding program, 9 of 28 critical components within the machinery control system hardware were obsolete when LHA 6 was delivered to the Navy. As a result, fleet officials told us that it has been difficult to obtain replacement parts. The Navy has spent over $90 million to repair the software and replace key components of the system on LHD 8, LHA 6, and LHA 7. The LPD 17 Flight I knuckleboom crane carries boats and cargo (such as ammunition) from the ship to the water and back again, and is pictured below in figure 8. However, according to Navy reliability data, this system only works 30 percent of the time it is supposed to and has been difficult for sailors to use and maintain since the lead ship was delivered in 2005, nearly 15 years ago. There are a number of challenges in sustaining this crane that the Navy did not identify or sufficiently mitigate during the acquisition process. For example, the fleet does not have the necessary technical data to operate and fix the system, spare parts can be difficult to find or take many months to obtain, and pieces of the system are obsolete. According to fleet officials who use the data, the shipbuilding program office did not acquire sufficient technical data nor conduct sustainment planning for this large and complicated crane primarily because they planned to contract for the maintenance of the entire ship, including this system. The Navy subsequently discovered that contracting for the maintenance of the whole ship was cost prohibitive and maintenance responsibility was transferred back to the Navy. However, because there had not been adequate sustainment planning, the fleet did not have necessary resources, such as technical data, to effectively maintain the system. Additionally, as the fleet has been developing the capacity to maintain this crane, the shipbuilding program office continues to accept cranes with unmitigated risks leading to unplanned fleet effort. For example, across the eleven LPD 17 Flight I ships that have been delivered, there are four different versions of the crane, which further complicates maintainability because it increases the types of spare parts needed and the knowledge required of the sailors to fix the system. Specifically, officials stated that sailors who learned to maintain a crane on one ship cannot transfer all of their knowledge to other ships in the class. Due to the numerous sustainment challenges the fleet has experienced with this crane on LPD 17 Flight I, LPD 17 program officials told us that the Navy has since revised its new construction crane requirements for LPD 17 Flight II. According to the shipbuilding program office, these requirements allow the shipbuilder to use a more standard crane, which will be easier to sustain. While we could not calculate the added costs of maintaining this crane, we found that the Navy has spent over $10 million on the following actions: (1) contracting with the original equipment manufacturer for repairs, (2) replacing key components of the system, and (3) making changes to improve the system. DOD policy that the Navy uses to set sustainment requirements does not capture factors that affect whether ships are reliable and maintainable. This results in shipbuilding programs having ineffective sustainment requirements that do not support sound acquisition decisions. When sustainment requirements are used to inform acquisition decisions, they can help ensure that shipbuilding programs design and build reliable ships that can be effectively sustained within planned costs. The effectiveness of a shipbuilding program’s sustainment requirements depends on how the requirements are set, used, and reported. Setting the sustainment requirements. We found that weaknesses with specific portions of DOD’s requirements policy resulted in the Navy setting sustainment requirements that are poorly defined and not representative of the availability of the ship during operations and sustainment. Using the sustainment requirements. To achieve the requirements, shipbuilding programs need to incorporate the requirements into decisions made throughout the acquisition process, such as developing the ship design. Due to problems setting the requirements, shipbuilding programs cannot incorporate the sustainment requirements into acquisition decisions. Reporting on the sustainment requirements. Statute requires that programs report on the status of these requirements on a regular basis. However, the Navy’s reporting on these requirements is misleading because it is based on the Navy’s deficient sustainment requirements and it does not reflect the fleet’s experience. The Navy sets sustainment requirements based on definitions for ships established by DOD policy, called the Joint Capabilities Integration and Development System, but the shipbuilding programs’ requirements are not robust even when they follow DOD policy. This is because the definitions for ship sustainment requirements in DOD requirements setting policy do not capture all factors that reduce the ability of ships to achieve their missions. For example, the definitions of operational and materiel availability in this policy exclude key factors and failures that reduce ship availability, such as catastrophic failures of mission-critical systems and unplanned maintenance. DOD policy states that the purpose of sustainment requirements is to ensure ships work when expected and are available when needed. But because the definitions of these requirements for ships do not capture all factors that can influence operational or materiel availability, the specific definitions for setting sustainment requirements for ships do not support the achievement of this goal. DOD’s requirements setting policy has designated these metrics to be key performance requirements since 2007, which means that they are one of a small number of mandated critical requirements that a weapon system must demonstrate. Without a definition for ship sustainment requirements in DOD policy that accounts for all factors that make Navy ships unavailable for operations, Navy shipbuilding programs cannot reasonably ensure that they are setting sustainment requirements that will result in reliable, maintainable, and available ships. In 2015, DOD added guidance to its policy that instructed shipbuilding programs to establish operational and materiel availability requirements based on the extent to which ships are expected to experience major failures, referred to as category 4 casualty reports. The fleet writes casualty reports when there are significant equipment failures that contribute to the ship’s inability to perform its missions. There are three categories of casualty reports (2, 3, and 4), with category 4 being the most severe. According to Navy guidance, category 3 and 4 casualty reports indicate degradation to critical mission capability that needs immediate repair, while category 2 reports contain failures that are important to the fleet but do not affect the ship’s core missions. In particular, DOD policy was updated to define operational and materiel availability for ships as follows: Operational availability (work when expected) is the percentage of time an operationally deployed ship is not in a category 4 casualty report state over a given operating period. The Navy typically sets this requirement at approximately 80 percent for shipbuilding programs. Materiel availability (ready when needed) is the portion of a ship class available for tasking. Ships are typically not available for tasking when in a planned maintenance availability or have an open category 4 casualty report. The Navy followed DOD requirements setting policy by establishing these key performance parameters for the four shipbuilding programs we reviewed that established requirements since fiscal year 2015—SSBN 826, FFG(X), DDG 51 Flight III, and LPD 17 Flight II. Prior to 2015, there were no ship-specific definitions in DOD requirements setting policy. Shipbuilding programs that set requirements prior to 2015 have generally adapted the definitions in JCIDS for calculating and reporting operational and materiel availability, which is why we include examples from these programs as appropriate The following two sections discuss shortfalls with DOD’s policy for setting sustainment requirements for Navy shipbuilding programs. DOD’s definition of operational availability for ships in its policy is problematic because it defines operational availability: (1) using category 4 casualty reports and (2) for the entire ship with a single metric. As a result, the operational availability requirement does not capture all critical failures that reduce a ship’s ability to perform mission-critical tasks. Category 4 casualty reports. DOD’s operational availability definition for ships counts only the most severe casualty reports—category 4. The definition excludes category 3 casualty reports, which also represent a severe degradation to the Navy’s primary missions. According to several fleet officials, category 4 casualty reports are typically used only in rare instances when the entire ship is out of commission. Fleet officials added that category 3 casualty reports can also represent severe mission-critical casualties that affect the ability of the ship to perform primary missions. In addition, the Navy’s categorization of casualty reports tends to be subjective or based on other factors than the severity of the defect, such as, according to maintenance officials, communicating a maintenance priority. In other words, there are additional deficiencies that could be mission-critical that may not be captured by category 3 or 4 casualty reports. Of the 11 ship classes in our review, six have delivered ships and have casualty report data available. We reviewed Navy casualty report data for 18 ships from these six ship classes and found that all of these ships had near-perfect operational availability when using only category 4 casualty reports. However, when we calculated operational availability using category 3 casualty reports, we found that 14 of these 18 ships fell short of their operational availability targets. Table 2 summarizes the category 3 and 4 casualty reports during two LCS missions as an example of how major failures are captured as category 3, and not category 4, equipment casualties. Therefore, by using category 4 casualty reports to define operational availability, the Navy is developing a requirement that does not accurately account for all ship failures that affect whether or not a ship works as expected. Setting operational availability at a whole ship level. DOD requirements setting policy specifies that shipbuilding programs should establish a single metric for the entire ship. However, when set at the ship level, the operational availability requirement is not effective at capturing the probability of whether or not a ship and its systems will work as expected. This is because ships are comprised of hundreds of systems that are of varying importance to achieving missions. For example, a ship may have an air-defense mission that requires a select group of systems—such as an air-search radar and a missile system—to work together to achieve the mission. However, a ship-level requirement is set using a single metric for the entire ship, which does not account for the fact that some systems are critical to achieving a ship’s primary missions while some systems are not as critical. Further, a ship level requirement is difficult to calculate. According to a Naval Sea Systems Command operational availability manual, it is improbable that the operational availability of hundreds of complex systems within a ship can be accurately calculated and represented in a ship level requirement. Figure 9 below illustrates how setting requirements pursuant to DOD requirements setting policy resulted in an operational availability requirement for the FFG(X) program that the fleet considers unacceptable. According to Navy handbooks and manuals on using operational availability during ship design, the operational availability requirement is a more effective input for acquisition decisions when it is set at the mission level. Since ships have multiple missions, this would result in multiple operational availability requirements instead of a single ship-level requirement. The Navy’s operational availability handbooks and manuals endorse this approach because a mission-level requirement is focused on a smaller group of systems that support the mission and, therefore, allows the Navy to prioritize availability for these key systems. Setting operational availability requirements by mission area would provide shipbuilding programs with information about how to identify and prioritize key systems for additional reliability analysis or sustainment planning to ensure that they will be sufficiently available to meet mission needs. Also, even though this would likely result in several operational availability requirements for each ship class, it would simplify the calculation of these requirements, which could make them more helpful inputs for acquisition decisions. We found that DOD’s definition of materiel availability for Navy ship classes in its requirements setting policy does not ensure that ships will be ready when needed—the purpose of the materiel availability requirement. This is because DOD requirements setting policy for ships does not specifically account for other factors that affect materiel availability—such as unplanned maintenance, unplanned losses, and training—during which times ships may not be available for operations. Unplanned maintenance. Unplanned maintenance can occur when planned ship maintenance lasts longer than expected or a mission- critical failure occurs during deployment that needs immediate attention. As our prior work has found, Navy ships experience significant levels of unplanned maintenance. For example, from fiscal year 2012 through fiscal year 2018, the Navy has reported over 3,900 days of unplanned maintenance across the ships we reviewed. Unplanned losses. Unplanned losses are instances when a ship is out of commission for an extended length of time due to severe damage or when a vessel was not prioritized for maintenance. For example, we have previously reported that due to heavy shipyard workload, some submarines are waiting significantly longer than planned—in some cases several months or years—to enter maintenance periods. Training. The Navy also conducts several training periods, and the DOD requirements setting policy does not address whether or not a ship is considered available or unavailable during these training periods. Six of the 11 shipbuilding programs we reviewed developed their program requirements since DOD made sustainment requirements mandatory in 2007. One of these six programs—LHA 6—did not established a materiel availability requirement as required by DOD requirements setting policy. LHA program officials told us that materiel availability does not apply to ships, which is not reflected in DOD requirements setting policy. Four shipbuilding programs—DDG 51 Flight III, LPD 17 Flight II, FFG(X), and LCS—developed materiel availability requirements that generally align with DOD’s requirements setting policy and, as such, do not specifically account for unplanned maintenance, unplanned losses, and training. The remaining shipbuilding program—SSBN 826—went above and beyond DOD requirements setting policy by incorporating these additional areas that could affect materiel availability. Program officials stated that sustainment requirements are more critical to achieving the SSBN 826’s missions than other shipbuilding programs. However, DOD and Navy guidance clearly state that materiel availability is a mandatory critical requirement for all programs. Since DOD’s definition for materiel availability does not include all factors that could result in a ship being unavailable for operations, shipbuilding programs cannot ensure that ships will be ready when needed. Because of how DOD policy defines sustainment requirements for ships, these requirements do not provide the information needed to support acquisition decisions. In particular, the Navy’s sustainment requirements developed according to DOD policy rarely provide adequate information about how reliable, available, and maintainable ships need to be, which is necessary to support well-informed decisions pertaining to ship concept development, design, and construction. For example, during the acquisition process, shipbuilding program offices make decisions that transform top-level requirements—like operational and materiel availability—into detailed, low-level requirements that can be achieved with available resources. We found that ongoing and new shipbuilding programs continue to make acquisition decisions that influence sustainment without the information that could be provided by better- defined sustainment requirements. Since shipbuilding programs cannot use these requirements to inform acquisition decisions, they cannot ensure that ships will be sufficiently reliable and available. The following two sections discuss the Navy’s issues with using sustainment requirements when making acquisition decisions for its shipbuilding programs. The Navy’s operational availability requirements for ships—which follow the DOD policy discussed above—do not provide adequate information to support acquisition decisions that affect whether or not ships are reliable enough to meet their missions. For example, in January 2020, we found that engineers can use a variety of activities when designing weapon systems to increase reliability to meet requirements, such as conducting failure analysis and adding redundant systems. In order for these engineering decisions to be successful, the requirements that inform the process must be firm, well-defined, feasible, and affordable. However, when the operational availability requirements do not adequately describe the needed reliability and maintainability for key systems—as is the case for most of the shipbuilding programs we reviewed—Navy engineers cannot ensure that the ship’s design supports the program’s top-level operational availability requirement. Further, they cannot identify aspects of the design that could put the requirement at risk. Instead of using the operational availability requirement to inform decisions across all key ship systems, Navy ship engineers told us that they interpret the requirement to only apply to catastrophic failures that put the entire ship out of commission. Therefore, in practice, shipbuilding program officials told us that they only apply this requirement to systems that the ship needs to get underway, such as the main engines and propellers. As such, shipbuilding programs are making engineering decisions during the acquisition process for many mission-critical systems, such as radars, weapons, and systems necessary for launching and recovering aircraft, without understanding how often these systems need to work to achieve key missions. This means the operational availability requirement only applies to the bare minimum of ship systems needed to get underway rather than the full complement of systems needed to meet the ship’s missions. For instance, LPD 17 Flight I ships can often sail away and are considered operationally available even as key systems—such as the knuckleboom crane, davit, air conditioning, and potable water systems among others—work less than 75 percent of the time the ship is at sea, according to fleet databases that track system failures. By interpreting the requirement to only focus on systems needed to move the ship and not accounting for other mission critical systems, shipbuilding programs cannot ensure that all critical systems needed to meet missions will work as expected. In addition, since shipbuilding programs have a ship-level operational availability requirement and interpret this requirement to focus on systems needed to get ships underway, they have not consistently leveraged available data on various ship systems when making engineering and ship design decisions. Navy sustainment experts told us that shipbuilding programs rarely use data on the actual availability of ship systems. If the requirement was set at the mission-level and focused on key systems, the data could show whether or not planned systems, already operating in the fleet, are available enough to meet requirements. Then, if the data shows that these systems are not sufficiently available, shipbuilding programs could make investments in improving the availability, such as improving supply support, making the system more reliable, or adding redundancy. Since shipbuilding programs cannot use operational availability requirements to make informed acquisition decisions, they are at risk of continuing to deliver ships to the fleet that are not as reliable and sustainable as needed. Of the five shipbuilding programs we reviewed that had established materiel availability requirements, we found that only one program has a requirement that provides adequate information for acquisition decisions. In particular, the SSBN 826 program’s materiel availability requirement has been a key input in establishing the submarine class’ planned maintenance schedules and procedures. Shipbuilding program officials told us they are using the maintenance period length determined by the materiel availability requirement to inform acquisition decisions—such as adjusting the submarine’s design to facilitate timely maintenance. For instance, the SSBN 826 shipbuilding program assessed the potential effect of new technology on the amount of maintenance that the submarine is planned to undergo. In doing so, the shipbuilding program officials believe that, if the new technology works as planned, the SSBN 826 class will meet the same presence requirement as its antecedent class with two fewer submarines. While this concept is a good example of how materiel availability can be used during the acquisition process, it is too early to know if the Navy’s plan will work for this class of submarines and fleet officials told us that they have doubts that the Navy can achieve this goal as planned. Officials from other shipbuilding program offices told us that they are not using the materiel availability requirement to inform maintenance decisions. Further, according to these shipbuilding program offices, the materiel availability requirements do not connect with the ship class’ planned maintenance schedules and, therefore, they do not make decisions to ensure that planned maintenance can be achieved within specific time frames. Program officials from several of these programs stated that the materiel availability requirement is not critical to performance goals, and, as such, it is not a priority to achieve this requirement. Without improving how the Navy defines and uses materiel availability requirements, shipbuilding programs are missing opportunities to make informed acquisition decisions about how ships are maintained and, therefore, cannot ensure that ships are available for operations when needed. The Navy’s reports to Congress are misleading because they do not reflect all of the failures and factors that reduce ship operational and materiel availability once ships are in the fleet. Shipbuilding programs report all key requirements in their Selected Acquisition Reports to Congress, including operational and materiel availability. According to DOD guidance for executing Selected Acquisition Reports, DOD program offices should provide accurate information to Congress to aid in determining if the program is meeting its key requirements. We reviewed the December 2018 Selected Acquisition Reports for the five shipbuilding programs that reported one or both of these sustainment requirements to Congress. We found that the Navy reported that these shipbuilding programs were meeting or surpassing their sustainment requirements. However, based on our analysis of data on mission-critical failures after ships were delivered, we found failures that would prevent these ships from conducting critical missions. Hence, the Navy’s reports to Congress do not reflect the actual availability of ships in the fleet. As a result, Congress does not have full insight into whether shipbuilding programs are on track to meet their operational and materiel availability requirements. The following two sections further discuss the Navy’s issues with reporting sustainment requirements for its shipbuilding programs. We found three out of seven shipbuilding programs report on operational availability in their Selected Acquisition Reports. These three programs all stated that they were meeting or exceeding their requirements, but these reports often did not match the fleet’s experience. For example: For one vessel class, the Navy reported that it was exceeding its operational availability goal by over 10 percent. At the same time, however, several mission critical systems are unreliable. Officials from the fleet stated that critical ship equipment is consistently failing. The Navy is reporting that another ship class—that has yet to finish construction —is exceeding its operational availability target by 5 percent. This ship class has already experienced several catastrophic failures that limit its ability to conduct primary missions during its limited at-sea periods. These examples demonstrate how reporting based on a ship-level operational availability requirement does not provide insight into reliability and maintainability problems that the fleet is experiencing and that prevent ships from meeting missions. Consequently, Congress is not receiving accurate information on the results of its investments and the sustainment problems the fleet is experiencing. We found that two of the Navy’s shipbuilding programs we reviewed currently report materiel availability in Selected Acquisition Reports to Congress. One other shipbuilding program that has materiel availability as a key requirement in its approved baseline does not report this requirement, contrary to DOD guidance. For example, the LCS shipbuilding program indicates that it is meeting the requirement despite evidence of issues with materiel availability. The Navy’s Selected Acquisition Report for the LCS states that the program is meeting its materiel availability requirement even though internal DOD reports state that the LCS’ materiel availability is significantly below its requirement. Further, fleet officials stated they are worried the maintenance workload required for the LCS class ships may result in additional unplanned maintenance delays that further reduce materiel availability. The Navy has also chosen to take steps that will reduce the materiel availability of the ship class throughout the ship class’ service life, such as assigning the first four ships as test ships, making one of every four LCS a training ship on a rotating basis, and increasing planned maintenance days, among other things. Since several of the Navy’s shipbuilding programs do not report information to Congress on this critical requirement, Congress does not have insight into whether or not ships are as available as intended. The shipbuilding programs included in our review did not consistently conduct effective sustainment planning when developing three key acquisition documents: life-cycle cost estimates, life-cycle sustainment plans (LCSPs), and independent logistics assessments (ILAs). According to DOD and Navy acquisition policy, these documents, along with other documents, help programs ensure the ships they are acquiring can be sustained affordably and adequately over their life cycle. However, for the shipbuilding programs in our review, we found that these documents did not provide a thorough assessment of the sustainment implications and risks for many of the programs’ acquisition decisions. Specifically, we found that: 1) O&S costs in shipbuilding programs’ life-cycle cost estimates did not account for major sustainment risks and grew significantly; 2) LCSPs rarely included information needed to demonstrate ships could reliably meet sustainment requirements at an affordable cost; and 3) ILAs did not consistently identify major sustainment risks that were subsequently realized by the fleet. Because shipbuilding programs are not effectively using these acquisition documents to plan for sustainment, they are passing unmitigated sustainment risks on to the fleet. We found that shipbuilding programs’ current estimates of O&S cost are significantly higher than initial estimates. This is largely because the Navy cost estimators based their initial estimates for the shipbuilding programs in our review on unproven sustainment assumptions without assessing the potential cost risk of the assumptions. According to shipbuilding program officials, O&S cost estimates grew after shipbuilding programs revised their sustainment assumptions, such as by increasing the number of crew required to operate and maintain the ships or by changing the level of maintenance needed for various ship systems. We compared programs’ initial life-cycle cost estimates for the six shipbuilding programs in our scope that had available estimates to current cost estimates that were updated after programs delivered ships to the fleet. As shown in table 3, we found that the shipbuilding programs’ estimates of O&S costs increased by over $130 billion from the initial estimate to the most recent estimate. Navy cost estimators stated that up to 20 percent, $26 billion, of the cost estimate growth could be accounted for by process changes that resulted in including more indirect costs, such as health and child care for sailors, into O&S estimates. Further, we adjusted our analysis to account for any program quantity changes over time. Even accounting for these changes, the Navy still experienced over $100 billion in O&S cost growth. The O&S cost growth for these six shipbuilding programs is likely higher than the $130 billion that we calculated in table 3. This is because the Navy has not updated these estimates to reflect actual O&S costs for several of the ship classes. For example, the LCS program, in its initial O&S cost estimate, projected $7.1 million (in fiscal year 2019 dollars) per year per hull for maintenance. However, thus far, the average LCS seaframe currently costs $21 million (in fiscal year 2019 dollars) per hull per year to maintain—an increase of over $13 billion if these higher than planned maintenance costs continue over the life of the ship class. We found that the shipbuilding programs we reviewed underestimated initial O&S costs, largely because cost estimators used unproven O&S assumptions without assessing the sensitivity of those assumptions on potential cost growth, as discussed below. The O&S costs estimates we reviewed had grown primarily because initial unproven assumptions turned out to be optimistic. O&S cost estimates for four of the six shipbuilding programs we reviewed were based on a Navy-wide effort that began in the early 2000s to reduce crew sizes on Navy ships and lower O&S costs by, among other things, replacing some sailors with automated systems. We found that cost estimators used the shipbuilding program offices’ unverified assumptions regarding crew size to develop the initial O&S estimate for four of these six programs. Over time, the Navy found that the automated systems were not as reliable as planned and, therefore, reduced crewing levels were not realistic. To address this and other issues, the Navy added sailors back on to ships—resulting in increases in O&S cost estimates. For example, cost estimators for the CVN 78 class initially estimated a 15 to 23 percent decrease in crewing levels compared to the previous class of carriers in order to create O&S savings. However, the Navy is now in the process of adding crew back on to the ship, even before its initial deployment, thereby contributing to increased O&S cost estimates, as shown in Table 4. Similarly, DDG 1000, LCS, and LPD 17 program officials also reported that increasing crew sizes was a major contributor to higher sustainment costs for these programs. Further, the shipbuilding programs we reviewed made assumptions based on unproven initiatives, in conjunction with reducing crew sizes that ended up having a greater effect on the cost of maintaining ships than initially estimated. For example, for four ship classes—SSN 774, DDG 1000, LPD 17 Flight I, and LCS—the Navy originally planned to use a maintenance initiative called performance-based logistics, which called for the use of contractors to conduct maintenance instead of sailors on board the ships. In 2001, DOD policy recommended that all weapon systems use performance-based logistics and Navy shipbuilding programs subsequently anticipated that this strategy would reduce maintenance costs. Based on our review of shipbuilding program cost estimates, we found that Navy cost estimators included cost savings from these new and unproven approaches—assuming that they would work as expected. Shipbuilding program officials stated that the Navy has now largely abandoned this approach after attempting to contract for performance-based logistics and discovering that it was much more costly than planned. Another initiative that began in the early 2000s involved the Navy using more shipbuilder-provided commercially-bought systems on ships rather than systems the Navy developed and provided to the ship. However, maintaining commercial systems has been more expensive than anticipated for a variety of reasons, such as systems becoming obsolete and challenges acquiring manufacturer support. For example, the SSN 774 shipbuilding program made an effort to use commercial equipment that it assumed would never need repair or replacement—meaning that these parts would last the life of the submarine—without evaluating whether these parts actually had no repair needs. Further, SSN 774 program officials told us that the program office did not plan for the Navy to support many of the submarine’s commercial components because they initially planned to contract for logistics support. In all, the SSN 774 program asserted that over 4,000 parts on the submarine class would not need maintenance for the duration of the submarine’s life. However, since the submarines have been operating, many of these parts are failing, which has created unanticipated expenses. For example, Navy maintenance officials stated that they are planning to pay $360 million over the next 12 years to maintain a part of the propulsion system that it wrongly assumed would not need any maintenance at the time O&S costs were established. A key reason that shipbuilding programs underestimated O&S costs is that the Navy’s cost estimators did not test the sensitivity of key O&S cost assumptions to quantify risks. According to DOD and Navy guidance and GAO-identified cost estimating best practices, cost estimates should include risk and sensitivity analyses to understand how changing program assumptions can affect cost—including O&S costs. However, for the six cost estimates that we reviewed, the Navy did not conduct risk and sensitivity analysis on key sustainment assumptions, such as unproven crewing and maintenance assumptions. The Navy’s cost estimators told us that they typically only conduct sensitivity analysis on the acquisition portion of a life-cycle cost estimate and not the O&S portion of the estimate. Instead, cost estimators told us that they use shipbuilding program office assumptions about the crew and how the ship class will operate as defined requirements that will not change. However, as discussed throughout this report, we found numerous instances in which incorrect maintenance assumptions resulted in billions of dollars of O&S cost growth. As a result, Navy’s cost estimators had reduced estimated O&S costs to reflect the programs’ presumed sustainment efficiencies without accounting for and quantifying the corresponding risk inherent in these assumptions. As such, in several cases, shipbuilding programs had optimistic estimates of O&S cost that later grew when unproven assumptions did not pan out as anticipated. According to shipbuilding program officials, their programs experienced significant O&S cost growth because the initial cost estimate did not sufficiently account for the risk of major changes to the program, such as revisions to the shipbuilding program’s assumptions about sustainment, that were realized once ships were provided to the fleet. For example, on the shipbuilding programs that adopted reduced crewing initiatives, Navy cost estimators reduced O&S costs due to fewer planned sailors on board, but did not determine how the O&S costs would be affected if automation did not achieve its intended efficiencies and the Navy had to add additional sailors to the crew. If the Navy’s cost estimators had conducted risk and sensitivity analyses of the O&S costs early in the acquisition process, shipbuilding programs could have had better insight into how much their O&S costs might increase if the key sustainment assumptions were not correct. Such insight into the potential sustainment cost impact could help shipbuilding programs identify the assumptions most likely to drive O&S cost growth. In turn, this information could help shipbuilding programs justify allocating additional resources during the acquisition process to ensure these sustainment assumptions are achieved, such as investing in additional testing to ensure the reliability of automated systems needed to reduce crewing levels. See figure 10 for an example of how unproven assumptions that were not evaluated using risk and sensitivity analyses led to optimistic O&S cost estimates for the DDG 1000 program. Navy officials told us that they are considering several pilot programs to improve cost estimators’ ability to conduct sensitivity analyses of maintenance costs, but have yet to provide details on these programs or the time frame for implementing them. While it is not possible for shipbuilding programs to predict future O&S costs with complete certainty, risk and sensitivity analyses could help shipbuilding programs’ better identify potential drivers of cost growth. In the absence of this cost analysis, shipbuilding programs will lack a clear assessment of the range of O&S costs their ships may require after they are delivered to the fleet. Additionally, without this O&S cost information, shipbuilding programs cannot provide Navy leadership with full insight into the range of resources that will potentially be required to sustain new ship classes over their lifetime or support recommendations for additional resources during acquisition to achieve sustainment assumptions. Five of the eleven shipbuilding programs we reviewed do not have LCSPs, and we found that the six programs that have LCPS do not use them to inform acquisition decisions that could help ensure ships are sustainable at an affordable cost. As of a September 2011 policy memorandum, DOD guidance requires every acquisition program we reviewed to have a LCSP. Shipbuilding programs, according to DOD acquisition policy, should develop and maintain LCSPs beginning at Milestone A, which is early in the acquisition process. According to DOD guidance, these plans should be the basis for all of the programs’ sustainment efforts. In particular, shipbuilding programs’ LCSPs should include information that demonstrates how a ship class can be affordably operated and maintained while meeting its sustainment requirements. To do so, DOD guidance describes that shipbuilding programs should use LCSPs to establish connections between life-cycle costs, reliability requirements, and crew size estimates, and identify and address sustainment issues, among other things. With nearly half of its shipbuilding programs not having completed LCSPs, the Navy is making acquisitions decisions without the context of a comprehensive sustainment planning document to help identify and mitigate the sustainment effect of its decisions. Figure 11 provides an example of a sustainment issue with the CVN 78 advanced arresting gear, which was identified during testing but not addressed in a LCSP. Officials from two of the five shipbuilding programs that do not have LCSPs stated that they had drafts of the plan, in some cases for several years, which leadership has yet to approve. In another case, shipbuilding program officials stated that they were not required to complete an LCSP even though DOD’s 2011 guidance directed them to create these plans immediately. For the six shipbuilding programs that had LCSPs, we found several challenges with how the programs develop and use these documents. Specifically, we found that the LCSPs: (1) rarely included a business case analysis, as required, that analyzed the relationship between life-cycle costs, reliability requirements, and crew size estimates; and (2) rarely identified and addressed sustainment issues in line with guidance. We found that none of the six LCSPs we reviewed contained business case analyses as required by DOD acquisition policy and guidance. According to DOD’s acquisition policy, an acquisition program’s LCSP should include a business case analysis annex, which should contain relevant assumptions, constraints, and analyses used to develop the product support strategy to the LCSP. According to DOD’s guidance for PSMs, who are responsible for developing and maintaining LCSPs, acquisition programs should use a product support business case analysis to help establish a product support package that balances sustainment costs against required sustainment outcomes. As such, the LCSP’s business case analysis is a tool to help programs assess the costs, benefits, and risks of key acquisition decisions from a sustainment perspective. Additionally, the LCSP should contain information on the activities needed to achieve the sustainment key performance parameters and a discussion of how much funding is required for those efforts. For example, Navy leadership approved the LCSP for FFG(X) in March 2019 even though the plan lacked the required sustainment business case analysis. Instead, the FFG(X) LCSP contains ship-level sustainment requirements and O&S cost information from the program’s life-cycle cost estimate, but no accompanying business case analysis demonstrating how the desired sustainment requirements (operational and materiel availability) can be achieved within these costs. As another example, several ship classes were designed with highly automated systems to enable reduced crew sizes and lower O&S costs, such as the LHD 8/LHA 6 machinery control system discussed earlier in this report. However, the LCSPs for these programs did not analyze the extent to which meeting O&S estimates and sustainment requirements were reliant on the reliability of these automated systems and the risks associated with using automation. Without connecting life-cycle costs to key sustainment factors such as reliability and crew size estimates, the Navy will not know if its sustainment planning is achievable within cost constraints until ships are provided to the fleet and have been operated for a significant period of time. We have previously found that it is often too expensive or time- consuming to make meaningful changes to the ship at this point in the shipbuilding process. LCSPs we reviewed rarely identified and proposed a plan to address programs’ sustainment issues, as described by guidance. According to DOD’s LCSP guidance, acquisition programs should assess their progress, challenges, and corrective actions when developing a plan to sustain a ship class. Two shipbuilding programs identified some sustainment risks and only one of the six LCSPs included plans for mitigating or correcting these risks. In the absence of proactively identifying and mitigating sustainment risks in the LCSP during the acquisition process, as described by guidance, we found that the Navy discovered and mitigated many of its sustainment challenges only after ships were delivered to the fleet. Without creating LCSPs that identify sustainment risks and proposing a plan to mitigate these risks, the Navy cannot ensure that it is making acquisition decisions that support ship sustainment. Two examples of significant sustainment risks that were experienced by nearly all of the programs we reviewed, but not identified or mitigated in LCSPs are: (1) insufficient technical data and (2) the use of performance- based logistics. Technical data. The LCSPs we reviewed that included an intellectual property strategy, as required by DOD acquisition policy during the operations and support phase, did not consistently address the full spectrum of potential intellectual property related issues, such as attaining intellectual property needed to repair and replace ship systems. According to DOD’s acquisition policy, shipbuilding programs should document the intellectual property strategy initially in the acquisition strategy and later in the LCSP to assess technical data needs and determine what intellectual property deliverables and license rights the program needs to acquire from contractors. Nearly all of the LCSPs we reviewed stated, in general terms, that the Navy would obtain the technical data to which it had rights. However, in these LCSPs, the Navy did not address how this strategy met the Navy’s needs for competitive and affordable acquisition and sustainment over the life cycle of a ship class, such as to ensure maintenance could be carried out as planned by a ship’s crew. Without ship programs fully planning for acquiring needed intellectual property to maintain ship systems in the LCSP, we found that the fleet was often not aware that certain ship systems were considered proprietary and only discovered what intellectual property was unavailable after ship systems were broken and Navy maintainers could not repair them. At this point, fleet maintainers stated that it is often too late to implement proactive strategies, such as working on an agreement with the manufacturer. Instead, after ships are delivered, fleet maintainers told us that they have several options, all of which are expensive and time- consuming. Fleet maintainers can (1) purchase these data on an expensive sole-source basis from the original equipment manufacturer; (2) spend significant time and effort reverse-engineering the system to be able to repair it; or (3) pay the manufacturer to conduct maintenance. Performance-based logistics. For three shipbuilding programs that planned to use performance-based logistics, the shipbuilding programs assumed it would work as expected and did not identify the risks associated with this maintenance approach or develop any mitigation plans. For example, as stated earlier in figure 10, the DDG 1000 program adopted a performance-based logistics approach during the acquisition process in an attempt to reduce sustainment costs. As such, the program’s LCSP stated that a contractor would be responsible for maintaining the ships in the class, including a number of new and unique systems installed on the ships. However, the LCSP also noted that the DDG 1000 program had not been able to determine how much the performance-based logistics approach was likely to cost or what sustainment outcomes the Navy could expect from this approach, in large part due to the number of new systems installed on the ships. After the shipbuilding program delivered the first ship in the class from the shipyard, DDG 1000 program officials determined that the fleet and other Navy maintenance organizations would instead be responsible for the maintenance that the shipbuilding program previously planned to execute by hiring a contractor. According to fleet officials, since taking over maintenance responsibility, the Navy has also determined that these systems are difficult to sustain, citing lack of commonality, missing technical data, and other challenges. In some cases, the fleet is now replacing DDG 1000’s unique systems after delivery with systems common to other Navy ships in an effort to mitigate sustainment cost growth and readiness effects. Despite these critical changes in the sustainment approach, the DDG 1000 program has not updated its LCSP since 2009. While the Navy has conducted an ILA on nearly every shipbuilding program we reviewed, we found that many of these assessments did not identify key sustainment issues or make recommendations to mitigate them. ILAs are conducted by assessment teams comprised of officials from across the Navy. The Navy ILA teams often validated program office sustainment assumptions contained in the LCSPs and other sustainment planning documents without evaluating those assumptions and identifying key areas of risk—even when programs introduced new sustainment concepts. DOD acquisition policy establishes that ILAs should provide an independent assessment of the shipbuilding program’s sustainment planning, including the identification and evaluation of issues that are likely to drive future O&S costs, design changes that could reduce O&S costs, and the adequacy of the product support strategy, among other things. ILAs are also supposed to make recommendations for mitigating the issues identified in the report, according to DOD and Navy guidance. Statutory requirements similarly emphasize the role of ILAs in identifying and mitigating sustainment risks that could increase O&S costs, and require DOD to establish guidance that requires the Navy to conduct ILAs prior to key acquisition decision points, including milestone decision events. ILAs for the shipbuilding programs included in our review did not sufficiently identify and evaluate the program offices’ sustainment assumptions and risks during the acquisition process. This was the case even when Navy testers had identified sustainment risks in early assessments conducted prior to the development of the LCSPs and ILAs. The following examples discuss instances in which Navy testers or maintainers identified sustainment risk before the ILA was conducted that have since caused sustainment challenges for the fleet, but the ILA team did not identify or make recommendations to address these problems. The SSN 774 shipbuilding program. As early as 2014, supply officials identified delays in over 1,000 supply orders for spare parts—many of these orders were in excess of 5 months old. However, in 2016, the ILA team rated this area as low risk and found that the supply support planning and execution was “outstanding.” Since supply support was rated as “low risk,” the ILA team did not make any recommendations to improve this planning. Subsequently, the SSN 774 class has experienced significant supply support issues. For example, the Navy’s maintainers routinely cannibalize hundreds of parts in 2017 and 2018 from SSN 774 class submarines to prepare other submarines for deployment, at an estimated rework cost of $2-3 million per year. The CVN 78 shipbuilding program. In 2013, testers stated that the number of berthing spaces on CVN 78 class carriers may not be sufficient to accommodate the planned crew size, particularly for the life of the carrier. When conducting its ILA in 2016, the ILA team rated crewing as low risk and the assessment noted extensive analysis had been conducted to validate the platform crewing profile. However, the ILA team did not document validation of the assumptions underpinning this analysis, such as whether or not automated systems needed to reduce crew levels would work as intended. The crewing concerns identified in 2013, but for which the ILA team did not make recommendations, are now a problem for the Navy’s fleet. For example, the Navy has already increased the size of the planned crew to the maximum allowed by the ship’s design. Nonetheless, additional crewing concerns persist for key systems—including the weapons elevators, advanced arresting gears, the machinery control system, among others—that are not yet well understood and may require additional sailor support to operate and maintain. The LCS shipbuilding program. In 2005 and 2006, Navy testers expressed significant concerns about the validity of the assumptions necessary to execute the program’s logistics support plan, specifically that the design of the new logistics system failed to include needed features to enable this logistics approach. In 2012, the Navy ILA team rated this area as low risk, specifically noting that the LCS program had developed a wide-range of well-written, informative, and comprehensive logistics planning documents. However, in part, since the ILA team did not recognize that the underlying issues previously identified by the testers had not been mitigated, the program provided ships to the fleet that had logistics issues. Specifically, the CNO conducted a study in 2016 that found the shipbuilding program’s logistics approach to be unstable and overly complex. As a result, the Navy is undertaking an overhaul of the LCS logistics approach, by taking actions such as creating Navy-led maintenance teams. The DDG 1000 shipbuilding program. The Navy requires significant volumes of technical data to manage the systems on the DDG 1000. In 2005, Navy testers noted that there were many details absent from the technical data management plan, including multiple sections that were left blank. In 2011, the Navy’s ILA team found that technical data management was low risk and stated that the requirements for technical data were well-written and clearly identified. According to fleet engineers and maintainers, as of September 2019—more than 3 years after lead ship delivery—all of the manuals remain in draft and are accurate enough for the sailors to acquaint themselves with systems, but not sufficient for supporting these systems. For example, fleet maintenance officials stated that several key documents for operating and maintaining critical ship systems, which were identified in the ILA as sufficiently complete, are not suitable for crew use. Several Navy officials across NAVSEA and shipbuilding program offices told us that ILAs are largely a document compliance check and vary significantly depending on the competency of the lead assessor. Therefore, in practice, according to Navy officials responsible for conducting these assessments, ILAs are not a thorough assessment of a ship classes’ logistics planning. This falls short of the purpose of ILAs, stated in Navy guidance, which is to provide acquisition programs with an effective measure of the program’s product support planning and execution. Officials from the NAVSEA organization responsible for ILA guidance also told us that they are in the process of improving how the Navy conducts ILAs for ships, such as by developing a new handbook and refocusing ILAs to better assess the quality of the sustainment planning. Specifically, these officials discussed the following five improvements: 1) starting ILAs as early as preliminary design; 2) tying the ILAs more closely to programs’ systems engineering efforts; 3) increasing focus on analytics, modeling, and simulation; 4) giving the Navy’s fleet and maintainers approval authority over the assessment; and 5) making investments to ensure that assessments are always led by officials with appropriate skills and expertise. If the Navy makes changes such as these or others, it would be a positive step toward making ILAs a more thorough and effective assessment of shipbuilding programs’ sustainment planning early in the acquisition process. However, these officials also stated that there is pushback from Navy program offices regarding these improvements because a more robust ILA requires more time and money from shipbuilding programs. Navy officials also noted that implementing the planned improvements is predicated on finding evaluators to conduct ILAs with appropriate skill sets, which has been a challenge. Until the Navy evaluates and implements proposed changes or other changes to improve the ILA process, the Navy will continue to be at risk of not identifying and resolving shipbuilding programs’ sustainment challenges during the acquisition process, before ships are provided to the fleet. We found that the senior leaders responsible for shipbuilding program oversight—the ASN (RD&A) and the CNO—have generally prioritized acquisition outcomes during Gate reviews, without considering how acquisition decisions affect sustainment outcomes. Navy acquisition policy states, however, that programs should be managed from a life- cycle perspective, with attention to both acquisition and sustainment outcomes. In an effort to increase senior leaders’ and shipbuilding programs’ attention on sustainment outcomes and to be responsive to Congressional efforts to improve weapon system sustainment, the Navy recently began pursuing two new initiatives—a Gate 7 for sustainment and the sustainment program baseline. These are promising steps that could help increase leadership insight into shipbuilding programs’ sustainment outcomes once ships are delivered to the fleet. However, we found that some of these efforts will likely not address the underlying need for Navy leadership to improve its consideration of shipbuilding programs’ sustainment goals early in the acquisition process as programs are making the decisions that have a long-term effect on ship sustainment. In addition, Congressional decision makers do not have full insight into sustainment cost growth. Navy leadership has not consistently reviewed shipbuilding programs’ sustainment planning at acquisition Gate reviews. According to senior Navy policy officials, in an effort to increase leadership attention on program sustainment, the Navy recently updated its acquisition policy to add a Gate for sustainment, called Gate 7. However, this recent change will not address the need for leadership to more consistently assess sustainment during earlier Gates. In addition, the Navy established a Deputy Assistant Secretary for Sustainment within the ASN (RD&A)’s office who will be responsible for managing the Navy’s sustainment funding and life-cycle management policies. However, it is too soon to assess the role that this official may have in the acquisition process. The Navy’s acquisition policy states that participants in Gate reviews should review program health and discuss and resolve areas of concern. Additionally, shipbuilding programs should be overseen and executed from a life-cycle perspective—in other words, with attention paid to balancing near-term acquisition outcomes and long-term sustainability. In support of this goal, the policy establishes required sustainment-related briefing content or actions for each Gate. While Gate 7 will function as the dedicated Gate for sustainment, all of the earlier Gates have sustainment- related requirements as well, as shown in Table 5 below. These Gate reviews offer Navy leadership opportunities to conduct oversight of shipbuilding programs’ sustainment planning during early phases of the acquisition process when key program decisions about requirements, design, and contracts are being made. Navy acquisition policy establishes that leadership should be briefed on a number of sustainment factors at Gate reviews, with a program’s life- cycle sustainment strategy/plan and O&S cost drivers being the minimum amount of sustainment information required for nearly all Gate reviews, as presented in table 5. We analyzed briefings and meeting minutes prepared for the 22 Gate reviews held for the shipbuilding program in our review between fiscal year 2014 and 2018. We found that Navy leadership had not assessed shipbuilding programs’ life-cycle sustainment strategies/plans in approximately 86 percent of Gate reviews and had not assessed O&S cost drivers in approximately 64 percent of Gate reviews, as shown in figure 12. According to Navy acquisition policy, this sustainment information should have been evaluated during all 22 of the Gate reviews held between fiscal year 2014 and 2018 for the shipbuilding programs included in our review. Instead, we found that the Gate reviews most often discussed acquisition updates. While a focus on acquisition updates during Gate reviews is appropriate, by infrequently devoting attention to how acquisition decisions affect sustainment, Navy leadership is missing an opportunity to assess the comprehensiveness and validity of shipbuilding programs’ sustainment plans and cost estimates, among other sustainment factors. As we previously discussed, shipbuilding programs’ LCSPs and O&S cost estimates were incomplete or insufficient, and, therefore, did not provide a thorough assessment of the programs’ sustainment risks. Additionally, Navy leadership is not consistently using Gate reviews to communicate to shipbuilding programs that achieving sustainment goals is a high priority. For pre-construction Gate reviews (Gates 1-5), Navy leadership evaluated three of the programs included in our report—SSBN 826, FFG(X), and DDG 51—in the 5-year period between fiscal year 2014 and 2018. These Gate review briefings included some discussion of program sustainment but did not meet all of the objectives and goals described by Navy acquisition policy for sustainment briefing content, as presented in table 5. As such, the Gate reviews did not provide a complete assessment of whether the programs’ acquisition decisions about sustainment would support the delivery of ships that could meet sustainment requirements at an affordable cost. Officials from the majority of programs included in our review told us that these early phases of the program are critical because it is at this point in the program where decisions are made that can have a long-term effect on ship sustainment, and it is difficult to make significant changes to sustainment outcomes after these key decisions are made. For example, when Navy leadership reviewed the SSBN 826 program at a Gate 4 review in November 2015 and a Gate 5 review in September 2016, the briefing discussed the SSBN 826 program’s sustainment costs in detail, including O&S cost goals, cost drivers, and contract incentives for O&S affordability. However, among other things, the Gate 4 briefing did not include a review of the program’s life-cycle sustainment strategy, and the Gate 5 briefing did not verify that all critical technical data and intellectual property issues had been addressed, which fleet and engineering officials stated are known sustainment issues for the Virginia class of submarines. Officials from the SSBN 826 program stated that some sustainment information that was not discussed in the Gate reviews was addressed in other forums. For example, leadership approved the program’s LCSP in August 2016, between the Gate 4 and Gate 5 reviews. In another example, when Navy leadership reviewed Flight III of the DDG 51 program at a combined Gates 4 and 5 review in March 2014, none of the required sustainment topics were included in the briefing. By not thoroughly assessing and resolving the sustainment effect of early acquisition decisions during its Gate reviews, Navy leadership is missing opportunities to ensure that shipbuilding programs are adequately considering sustainment goals and is at risk of allowing programs to proceed through the acquisition process without verifying that there is adequate planning for sustainment. For Gate 6 reviews held between fiscal year 2014 and 2018, we similarly found that Navy leadership did not consistently discuss sustainment, even as programs began ship construction and delivering ships to the fleet. Our analysis of Gate 6 documentation showed that the primary focus of most Gate 6 briefings and meeting minutes was acquisition outcomes, such as construction progress or follow-on ship contract awards. In particular, we found that 16 of the 18 Gate 6 reviews we assessed for eight shipbuilding programs did not include information about both the program’s life-cycle sustainment plan and O&S cost drivers, which are part of the required briefing content for every Gate 6 review. Officials from most of the programs in our review confirmed that leadership placed greater emphasis on acquisition updates than sustainment during Gate 6 reviews. For example, the SSN 774 program is pursuing a reduction in total ownership costs initiative for its Block IV submarines, but the program’s recent Gate 6 briefings included only limited details on design changes that the program was pursuing to improve sustainment and no information on the anticipated O&S cost savings from the effort. Officials from this program confirmed that leadership has historically focused only on acquisition issues during the Gate 6 reviews. Additionally, we found that Navy leadership issued sustainment-related action items to only three of the eight programs in the Gate 6 reviews we assessed, even though all of these programs had ongoing sustainment challenges, as discussed earlier in this report. Although nearly 90 percent of the Gate 6 reviews we assessed did not include briefing content on the program’s life-cycle sustainment plan and O&S costs, as required, nearly all of the Gate 6 reviews included a discussion of at least one ongoing sustainment challenge affecting the ship class. In these cases, the discussion centered on mitigating realized sustainment issues already being experienced by the fleet after ship delivery. For example, all of the LPD 17 Gate 6 reviews over the past 5 years included updates on the activities of the LPD 17 Strike Team and its progress in resolving class-wide design and construction issues that negatively affected the ships’ operational availability and reliability after they began fleet operations. While Gate 6 can be used as a venue to discuss sustainment issues that are already being experienced by the fleet, until Navy leadership more consistently reviews programs’ sustainment planning and expected outcomes during earlier Gates, programs will continue to be at risk of delivering ships to the fleet that have unmitigated sustainment risks or are unaffordable. The Navy recently updated its acquisition policy to expand the scope of its Gate process and add a new Gate 7 for sustainment. Effective March 2019, the Gate 7 reviews will begin 5 years after shipbuilding programs achieve initial operational capability and recur every 5 years thereafter. As such, Navy officials told us that the scope of the Gate 7 review will be oversight of programs that are well into production and delivering ships to the fleet. According to the Navy’s acquisition policy, Gate 7 will evaluate the effectiveness of a program’s product support strategy, compare actual sustainment costs to estimates, discuss fleet-identified sustainment issues, and assess sustainment risks and mitigation measures, among other things. Senior officials told us that the Navy developed a Gate 7 for sustainment for two reasons. First, similar to our findings, officials stated that the Navy recognized sustainment was generally not being discussed during existing Gate reviews, particularly during Gate 6 reviews as ships were starting to be delivered to the fleet, even though this was required briefing content for Gate 6 in the Navy’s acquisition policy. Second, in the National Defense Authorization Act for Fiscal Year 2017, Congress directed the military services to conduct sustainment reviews on major weapons systems—such as the shipbuilding programs included in our review—within 5 years of the weapon system achieving initial operational capability and then periodically throughout their life cycles. Such sustainment reviews are to assess the weapon system’s product support strategy, performance, and O&S costs. Based on our analysis of the Navy’s revised acquisition policy, the new Gate 7 appears responsive to the Congressional requirement for sustainment reviews and, if implemented as planned, will provide an oversight forum for addressing realized sustainment challenges. However, we found that adding a new Gate to the end of the acquisition process is too late to drive meaningful improvements to sustainment outcomes and is not sufficient to address current shortfalls in how the Navy’s acquisition process addresses sustainment concerns. Senior Navy officials we spoke to who had knowledge of this change expressed doubt that a Gate 7 for sustainment would be an effective means of holding programs accountable for addressing acquisition-related sustainment issues, since it occurs late in the acquisition process. Whereas the Gate 7 for sustainment will occur at the end of the acquisition process, the decisions that influence sustainment outcomes, such as decisions about ship design and the planned sustainment strategy, are made much earlier in the process, normally between Gates 1 and 5. Thus, while Gate 7 will provide leadership with insight into the execution of ship sustainment and any challenges being experienced by the fleet, it does not address the need for Navy leadership to evaluate shipbuilding programs’ efforts to design and plan for sustainable ships during earlier Gates, when key long-term decisions are being made. According to a senior Naval Sea Systems Command official, Gate 7 is timed well for being able to “sit back and admire the problem” as opposed to preventing the issue. Until Navy leadership brings attention to sustainment during earlier Gate reviews, it will continue to miss opportunities to proactively ensure shipbuilding programs are acquiring sustainable ships before they are provided to the fleet. We found that acquisition program baselines (APB)—which are intended to be binding agreements between leadership and the program manager and document the program’s goals—currently include limited information about sustainment. While the Navy is developing a new initiative to create a dedicated baseline for sustainment, it does not have a mechanism for holding shipbuilding programs accountable for sustainment goals during the acquisition process. Like all major weapon systems, shipbuilding programs have APBs that summarize the programs’ cost, schedule, and performance goals and set the baseline from which programs must, as appropriate, obtain approval from agency leadership to deviate and must report certain changes to Congressional defense committees. Statute requires that baselines will contain information on the program’s cost estimate, schedule, performance, and supportability, among other factors. In practice, for shipbuilding programs in our review, we found that the program goals established in the APB are largely focused on acquisition cost, acquisition schedule, and performance requirements, with limited information provided on sustainment. In particular, the sustainment information provided is generally limited to a high-level O&S cost estimate and the sustainment key performance parameters, if the program has them. A Congressionally established panel, called the Section 809 panel, charged with making recommendations to improve the efficiency and effectiveness of DOD’s acquisition process, among other things, recently studied challenges with the sustainment of major weapon systems. It similarly found that the APB does not provide sufficient governance of the sustainment phase of an acquisition program since it is focused on acquisition cost, schedule, and performance goals. The panel further noted that program success has been measured against the achievement of the APB’s acquisition goals, so program managers have generally prioritized the achievement of acquisition outcomes and deemphasized sustainment. As a result, the panel recommended the creation of an additional program baseline, called the sustainment program baseline (SPB), to help ensure programs are held accountable for sustainment- related outcomes and establish balance between acquisition and sustainment priorities. In March 2019, the Navy initiated an effort to begin developing an SPB framework. Senior officials stated that the Navy intends to pilot the SPB with a few aviation programs in fiscal year 2020 before expanding the initiative to ship classes that are already in sustainment, and then finally to programs that are still in the acquisition process. According to Navy officials involved with this initiative, the SPB is intended to complement the APB, and Navy leadership will use the two program baselines to review and approve the acquisition and sustainment aspects of a program throughout the acquisition process. The shipbuilding program should draft the initial SPB early in the acquisition process to support Milestone A and update it as the program matures. Officials in the office of the ASN (RD&A) told us that the Navy plans for the SPB to be grounded in a program’s sustainment key performance parameters for operational and materiel availability and include targets for various other sustainment metrics, such as sparing, equipment failure rates, mission capable time, and logistics time, among others. The SPB should also provide detailed information about all of the costs and funding sources that will support sustainment. Navy officials identified a number of potential improvements the SPB could offer for how shipbuilding programs consider sustainment, such as devoting additional time and resources to the development of sustainment metrics early in a shipbuilding program, assessing the sustainment effect of acquisition decisions, creating a common understanding of a program’s sustainment goals across disparate stakeholders, and providing a more accurate accounting of sustainment funding. If the Navy implements the SPB as described, it will likely be a positive step toward ensuring shipbuilding programs are increasing their focus on sustainment planning during the acquisition process. While the SPB could potentially provide increased attention on program sustainment, we found that developing this new baseline may not fully address the underlying challenge of shipbuilding programs managing to the APB’s acquisition goals and the lack of consideration of sustainment in acquisition decision-making. This is because, according to current proposals, programs will continue to be measured against the APB during the acquisition process, with the SPB not serving as the governing baseline until later in the program life cycle during the sustainment phase. Instead, during the acquisition process, the Navy’s efforts related to the SPB will be limited to initially developing the SPB and updating it as the program matures. While updates to the SPB during the acquisition process could provide more transparency into the sustainment effect of various acquisition decisions within the program and to leadership, this approach primarily documents the sustainment effect of a decision. Because the APB will remain the governing baseline during the acquisition process and the program will not be measured against the SPB until the sustainment phase, shipbuilding programs will continue to have an incentive to prioritize acquisition outcomes over sustainment when making acquisition decisions. DOD does not provide Congress with detailed information on the extent and causes of shipbuilding programs’ O&S cost growth during the acquisition process. For example, a mechanism for Congressional oversight of major defense acquisition programs’ unit cost growth, called the Nunn-McCurdy statute, is focused on acquisition costs and not sustainment cost estimates. A Nunn-McCurdy breach is triggered by increases in a program’s unit cost estimates against the acquisition unit cost goals established in the program’s APB. The Nunn-McCurdy statute provides Congress greater visibility into major defense acquisition programs’ estimated acquisition cost growth and encourages DOD to manage costs by requiring programs in a breach to include acquisition cost estimates in Selected Acquisition Reports and notify Congress of a breach. While the APB also includes O&S cost estimates, the Nunn- McCurdy statute does not require reporting of O&S cost growth to Congress. The Nunn-McCurdy statute also requires DOD to take a series of actions whenever a program experiences critical acquisition cost growth, which is growth in the program acquisition unit cost estimate of at least 25 percent over the current baseline estimate documented in the APB or of at least 50 percent over the initial baseline estimate. Among other things, these actions include (1) conducting a root cause analysis of the cost growth, (2) reassessing program costs, and (3) terminating the program or taking other steps that include restructuring the program. If DOD decides not to terminate a program that has critical cost growth, the Secretary of Defense must restructure the program in a manner that addresses the root cause of the cost growth, rescind the program’s most recent Milestone decision, and review the program regularly, among other tasks. As stated earlier, we found that leadership oversight during Gate reviews and program execution is primarily focused on acquisition outcomes. Additionally, as the Section 809 panel noted, the Nunn-McCurdy breach provided a strong incentive for major defense acquisition programs to control acquisition cost, but that there was not an equivalent incentive for controlling sustainment costs. As such, the shipbuilding programs’ acquisition decisions and Congress’ oversight mechanisms have focused on acquisition cost outcomes, without a comparable focus on sustainment cost outcomes during the acquisition process. For example, when the DDG 1000 program experienced a critical acquisition cost growth breach, the Nunn-McCurdy statute required DOD to reassess and certify to Congress the need for the program at the increased cost levels. DOD was also required to identify and address the cause of the acquisition cost growth when reassessing the program and conduct additional program oversight, among other things. According to DDG 1000 program officials, DOD and the Navy recognized that the acquisition decisions leading up to and following the breach would have a sustainment effect. For example, the decision to reduce the number of ships in the class to manage acquisition cost growth has contributed to higher per ship O&S costs, as the investment needed to sustain this new class is now spread across fewer ships than initially planned. However, the focus of their restructuring efforts was on addressing the acquisition cost growth. By contrast, there was not a similar effort to manage growth in the program’s O&S cost estimates, which have increased by more than 50 percent on a per ship per year basis. For the six shipbuilding programs with O&S cost estimates we were able to assess, we found that four experienced cost growth greater than 50 percent for their average annual O&S per hull cost, as compared to the programs’ original estimates. Table 6 shows the extent of these shipbuilding programs’ O&S cost estimate growth over time. This level of cost growth for acquisition costs would have constituted a Nunn-McCurdy breach. While the Selected Acquisition Reports for these programs include some information on shipbuilding programs’ O&S costs, this reporting does not provide Congress with detailed information about the causes of the cost growth and potential program changes to address it and, therefore, does not facilitate the same level of oversight as is given to acquisition unit cost growth. In particular, DOD was not required to notify Congress that the programs had experienced high levels of O&S cost growth above a certain threshold, and DOD was not required to identify the root cause of the O&S cost growth and restructure the programs to address the cost growth. As a case in point, for the programs we reviewed, Navy leadership only directed one of the programs—LPD 17—to identify opportunities to reduce O&S costs following a Gate 6 review. For other programs that had extensive O&S cost growth, the programs were not required to take additional steps during the shipbuilding process to manage these costs and mitigate the long-term sustainment effect of their acquisition decisions. The LCS program, for example, has seen the highest rate of per ship O&S cost growth among the shipbuilding programs included in our O&S cost analysis, but Congress and agency leadership have not required the shipbuilding program to take action to address these issues. Instead, the shipbuilding program continues to deliver ships to the fleet that are significantly more expensive to maintain than initially planned and which have significant maintenance and logistics challenges, according to sustainment officials. The fleet is now undertaking its own efforts to improve sustainment outcomes for LCS, such as changing its crewing and maintenance approaches, which are further adding to the O&S cost growth for the program. According to DOD and Navy acquisition policy, program managers should be the single point of accountability for the full life cycle of ship programs. However, without a mechanism to provide Congress with more detailed information about shipbuilding programs’ O&S cost growth and the drivers of such cost growth, Congress cannot know if shipbuilding programs are accounting for the full life-cycle implications of their acquisition decisions. In particular, without such a mechanism, Congress will continue to lack full insight into the extent to which shipbuilding programs’ O&S cost estimates have grown over time and what steps DOD and the Navy could take to better control O&S cost growth during the acquisition process. Congress directed DOD to establish PSMs as key sustainment managers for weapons systems, such as shipbuilding programs. However, we found that PSMs in the shipbuilding program offices have limited influence on decisions made during the acquisition process that affect ship sustainment. In 2009, Congress passed legislation that required DOD to appoint PSMs to support each major weapon system. According to DOD guidance, PSMs are senior sustainment officials in program offices who are tasked with ensuring that DOD weapon systems, including Navy ships, are reliable and can be maintained effectively at an affordable cost. The guidance states that PSMs should be involved in the acquisition decision-making process to ensure the weapon system—in this case a ship—can be supported throughout its life cycle. All but one of the shipbuilding programs included in our review have a dedicated PSM. However, we found that these sustainment experts have generally had limited involvement in key acquisition decisions, such as developing sustainment requirements and estimating O&S costs, because: (1) Navy acquisition policy does not ensure that PSMs are involved early in the acquisition process when key decisions that affect sustainment are made, and (2) their responsibilities to support sustainment outcomes during the acquisition process are often at odds with the program office’s overarching focus on acquisition cost and schedule outcomes. Navy acquisition policy does not ensure that PSMs are appointed early enough to inform key acquisition documentation and initiate sustainment planning early in the acquisition process. Until recently, Navy acquisition policy did not specify when PSMs should be involved in the acquisition process. However, a March 2019 update to the Navy’s acquisition policy established that Navy leadership should assign PSMs by initiation (normally Milestone B). We found that this timing is too late in the acquisition process, as critical acquisition decisions that have significant repercussions for sustainment are made before Milestone B, such as developing the program’s requirements and initial sustainment strategy. For example, according to DOD’s PSM guidance, PSMs need to be involved prior to initiation of the program. Among other things, the PSM should provide a sustainment perspective into key decisions such as developing the acquisition strategy and setting requirements. This guidance also states that the PSM is responsible for authoring or providing input on key program documents, such as the LCSP, which are required by Milestone A. The Navy policy, therefore, does not facilitate the early contributions of PSMs to key documents as described by DOD guidance, and it does not help ensure PSMs are appointed to shipbuilding program offices early enough to influence key decisions about the program’s sustainment. For its two most recent shipbuilding programs, which began after the enactment of the PSM legislation in 2009, Navy acquisition policy has not ensured that PSMs are involved early enough in the acquisition process to influence decisions that affect sustainment. As a result, the programs have appointed PSMs at different points in the acquisition process and their ability to influence key decisions has varied, with the PSM appointed earlier able to affect more decisions related to sustainment. For example, the SSBN 826 program’s PSM was appointed before the program reached Milestone A. This is in line with DOD guidance but before Navy acquisition policy requires the PSM to be appointed. As a result, the SSBN 826 PSM stated that he was involved in the setting of the program’s sustainment requirements and has subsequently used those requirements to ensure sustainment is being considered in the acquisition process, including during the development of the submarine’s design. By contrast, the FFG(X) program, which began in 2017, does not yet have a dedicated PSM as the program approaches the Milestone B review. While this is permitted by the Navy’s acquisition policy, the program has now made critical sustainment decisions, such as developing the sustainment strategy, the maintenance and training schedule, and the sustainment key performance parameters, without a PSM. For the nine shipbuilding programs in our review that started prior to 2009, key acquisition decisions were made without the input of a senior sustainment official who has the responsibility and authority of a PSM. Nearly all of the PSMs for these nine programs stated they that they were not involved in or did not have insight into key acquisition decisions that took place early in the acquisition process, such as ship design. Instead, PSMs told us that their job has been to implement decisions that were already made. For example, one PSM said that “the die has been cast” once major decisions about automation, crew size, and service life are made, and after that all the PSM can do is “try to undo the sustainment harm that has been caused.” Given these results, officials from nearly all of the shipbuilding programs we spoke with stated that shipbuilding programs should assign PSMs at the very beginning of the program when key decisions are being made about how and what to acquire. In particular, program officials stated that the PSM should be appointed at the start of the program to ensure early decisions consider sustainment. Such decisions include establishing the sustainment requirements, developing the acquisition strategy, and designing the ship. We previously found that Navy PSMs considered early appointment of the PSM critical to ensuring they can influence their programs’ sustainment considerations. If shipbuilding programs do not appoint PSMs early in the acquisition process, the programs will continue to make critical decisions that affect sustainment without the input of the programs’ senior sustainment official. Without revising its acquisition policy to establish that PSMs should be appointed to shipbuilding programs at the beginning of the acquisition process, the Navy cannot ensure PSMs are involved early enough to influence key decisions that affect sustainment, such as requirements setting and the drafting of the LCSP. Since PSMs focus on sustainment and the shipbuilding programs focus on managing acquisition outcomes, the PSMs’ roles and responsibilities are at times at odds with the goals and priorities of the program office in which they work. A Navy working group recently found that the effectiveness of PSMs is limited because the PSM’s goals do not always align with the shipbuilding program’s acquisition cost and schedule goals. The Navy issued a strategic plan for fiscal years 2018 to 2023 that was focused on strengthening the life-cycle logistics workforce that supports acquisition programs, including PSMs. The strategic plan established a working group on product support authority, which found that program manager and PSM roles and responsibilities are often in conflict and misaligned, reducing the authority and effectiveness of PSMs. As a result, the working group is assessing possible changes to improve the effectiveness of PSMs, such as revising Navy policy to better reflect the PSMs’ statutory authority or increasing PSMs’ independence by creating an additional reporting chain of command outside of their acquisition program. We similarly found that the ability of PSMs to influence key acquisition decisions may be limited because their focus on improving sustainment outcomes can be at odds with the shipbuilding programs’ emphasis on achieving acquisition goals, such as acquisition cost and schedule. As discussed above, Navy leadership has generally only focused on shipbuilding programs’ acquisition outcomes during the Gate process, without considering how acquisition decisions affect sustainment. In turn, program officials from all of the shipbuilding programs we reviewed reported that Navy leadership had directed them to prioritize the achievement of acquisition outcomes, such as acquisition cost goals, during the execution of their programs, and none had been directed to devote additional attention to sustainment. Additionally, officials in many of the shipbuilding programs we reviewed told us that a key ASN (RD&A) memorandum on managing acquisition costs framed their decision- making, including decisions about program changes to improve sustainment. This focus on managing acquisition costs can run counter to PSMs’ efforts to improve sustainment outcomes, such as increasing system reliability or providing adequate technical documentation, as these efforts frequently require investment of additional shipbuilding funds. Rather than investing acquisition funds to improve sustainment outcomes, we found that shipbuilding programs instead have an incentive to delay sustainment improvements until after ships are delivered to the fleet and funding sources other than those managed by the shipbuilding program can be used for these purposes. According to officials from 16 different acquisition, engineering, and sustainment offices, because shipbuilding programs are only responsible for ships until they are provided to the fleet, the Navy’s shipbuilding programs have an incentive to delay sustainment improvements until after ships are delivered to the fleet, when other parts of the Navy take over responsibility for funding them. As one fleet official explained, shipbuilding programs are not incentivized to address sustainment issues because the shipbuilding programs are held responsible only for the achievement of acquisition cost goals and not for sustainment cost goals. Some Navy officials characterized this dynamic as throwing sustainment concerns “over the fence” once ships are provided to the fleet. Further, we found that Navy leadership made decisions, in some cases, even though PSMs expressed concerns about the feasibility of implementing the decision from a sustainment perspective. Figure 13 provides an example of when LCS sustainment officials in the shipbuilding program expressed concern about the feasibility of the LCS crew size. While it is important for shipbuilding programs to manage acquisition cost and schedule, focusing only on these acquisition outcomes reduces the effectiveness of the PSM and increases the risk that ships will have long- term sustainment challenges. The quantity and breadth of issues identified in this report—resulting in billions of dollars in unexpected costs, maintenance delays, and unreliable ships—suggest that existing policies and guidance have not ensured that new ships are reliable and can be sustained as planned. Recently, due to some of these problems, DOD and the Navy have recognized the importance of considering the requirements and costs of sustainment during the acquisition process, and Congress has passed legislation related to sustainment planning. This report, along with other DOD initiatives discussed in this review, demonstrate that the Navy needs to take many steps to infuse its acquisition decision-making with a greater focus on sustainment outcomes. Systemic changes are needed to improve shipbuilding programs’ sustainment outcomes, including: setting clear sustainment requirements that are useful for acquisition decision-making and reporting the results to Congress, improving O&S cost estimates, sustainment planning, and logistics assessments, and involving the PSM early in the acquisition process. However, these changes will only be successful if Navy leadership commits more time, attention, and resources to ensuring that sustainment is thoroughly considered throughout the acquisition process. Until the Navy resolves these issues, its shipbuilding programs will continue to pass costly sustainment risk to the fleet that results in ships and submarines that experience major sustainment problems. Congress should consider developing an oversight mechanism for evaluating shipbuilding programs’ sustainment cost estimate growth during the acquisition process, with requirements for the Navy to: (1) report sustainment cost estimate growth information to Congress and (2) reassess shipbuilding programs that are experiencing a high level of sustainment cost estimate growth. We are making the following 11 recommendations to DOD: The Secretary of Defense should change its definition for setting operational availability for ships in its Joint Capabilities Integration and Development System policy by adding information that defines the operational availability requirement by mission area in addition to the ship level and includes all equipment failures that affect the ability of a ship to perform primary missions. (Recommendation 1) The Secretary of Defense should change its definition for setting materiel availability for ships in its Joint Capabilities Integration and Development System requirements policy to include all factors that could result in a ship being unavailable for operations, such as unplanned maintenance, unplanned losses, and training. (Recommendation 2) The Secretary of the Navy should direct the ASN (RD&A) and the CNO, once DOD requirements setting policy is revised, to update existing operational availability requirements for ongoing shipbuilding programs. When revising these requirements, the Navy should set operational availability requirements that: (1) are based on failures that affect the ability of a ship to perform primary missions and (2) are set at the mission level instead of ship level. (Recommendation 3) The Secretary of the Navy should direct the ASN (RD&A) and the CNO, once DOD requirements setting policy is revised, to update the materiel availability requirements for ongoing shipbuilding programs. When developing or revising these requirements, the Navy should set materiel availability requirements that fully capture all factors that could preclude a ship from being ready when needed. (Recommendation 4) The Secretary of the Navy should direct the ASN (RD&A) and the CNO, once the Navy revises its sustainment requirements, to ensure that shipbuilding programs report operational availability and materiel availability requirements in Selected Acquisition Reports, and alternatives to the Selected Acquisition Reports, for Congress. (Recommendation 5) The Secretary of the Navy should direct the Commander of Naval Sea Systems Command to ensure that cost estimators follow current guidance and GAO-identified best practices and conduct sensitivity analyses and other analyses to improve their assessment of cost risk in the O&S costs in shipbuilding programs’ life-cycle cost estimates. (Recommendation 6) The Secretary of the Navy should direct the ASN (RD&A) to ensure all shipbuilding programs develop and update LCSPs, in accordance with DOD policy, that demonstrate how a ship class can be affordably operated and maintained while meeting sustainment requirements, including associated business case analyses and identifying sustainment risk. (Recommendation 7) The Secretary of the Navy should direct the Commander of Naval Sea Systems Command to evaluate and implement changes to the ILA in order to position the ILA to effectively identify key sustainment risks and make recommendations for risk mitigation, which may include existing Navy proposals to change the ILA process. (Recommendation 8) The Secretary of the Navy should direct the ASN (RD&A) and the CNO to ensure sustainment-related briefing topics prescribed by the Navy’s acquisition policy are consistently discussed at Gate reviews. (Recommendation 9) The Secretary of the Navy should direct the ASN (RD&A) and the CNO to implement the sustainment program baseline initiative for shipbuilding programs and, in so doing, develop a mechanism that ensures that sustainment outcomes are a factor in shipbuilding programs’ decision- making during the acquisition process. (Recommendation 10) The Secretary of the Navy should revise SECNAVINST 5000.2 and other associated guidance to ensure PSMs are assigned to shipbuilding program offices in time to inform early acquisition decisions, including development of the program’s sustainment requirements and LCSPs. (Recommendation 11) We provided a draft of our report to DOD for comment. DOD’s written comments are reprinted in appendix III of this report. DOD concurred with 8 recommendations and partially concurred with 3 recommendations. However, for at least 5 of the recommendations in which DOD partially concurred and concurred, DOD did not describe the specific actions it is planning to take to address our recommendations. These are discussed below. In response to our first and second recommendations on operational and materiel availability requirements, DOD stated that the Navy and Joint Staff would revisit requirements definitions for shipbuilding programs to better ensure that they are traceable to a ship’s mission and can be used across ship development and fielding. DOD also agreed that it will align the sustainment definitions with how the Navy defines critical failures for ship programs. While these are important steps, they do not fully address our recommendations. Specifically, DOD officials told us that the department plans to continue defining operational availability with a single metric for an entire ship or ship class. While this approach is appropriate for materiel availability, as we state in the report, it is misaligned with Navy guidance for operational availability, which states that such an approach is not mathematically feasible for ships. Until DOD ensures that its sustainment requirements for ships are well-defined and usable during acquisition and sustainment, shipbuilding programs will continue to implement requirements that do not result in reliable and available ships. In response to our third and fourth recommendations, DOD agreed to incorporate changes to its requirements-setting policy into new shipbuilding programs. However, DOD and the Navy may miss key opportunities to improve the Navy’s sustainment requirements for existing programs, including at least four ship classes that have plans for a new flight, block, and/or major modification. This approach also excludes existing programs that have established requirements but have yet to start design or construction. Changing these requirements, in line with our recommendation, would help ensure that more rigorous sustainment requirements inform Navy ship designs. For example, as we discuss in the report, the current FFG(X) operational availability requirement would allow the ships to be out of service for extraordinary lengths of time. Until the FFG(X) requirement and those for other existing ships (such as DDG 51 Flight III) are remedied, the sustainment requirements will continue to be poorly defined and unable to influence design decisions in a manner that results in more reliable ships. In response to our fifth recommendation, DOD concurred with the recommendation because it stated that it already reports the status of both sustainment requirements in its Selected Acquisition Reports. However, as we state in our report, implementing this recommendation is dependent on the Navy changing the definition of its sustainment requirements to improve the accuracy of its reporting to Congress. Since DOD only agreed to modify material availability requirements for existing ship programs as it deemed appropriate, its Selected Acquisition Reports could continue to be misleading for many of its ship programs because they may not reflect all of the failures and factors that reduce operational and materiel availability once ships are in the fleet. In addition to DOD’s response, the Navy’s ASN (RD&A) also submitted a letter stating that he generally agreed with the recommendations and indicated that his office has already started making some changes over the last 10 years to improve consideration of sustainment while acquiring ships. The Navy also sought to add context to some of our report findings. We respond to the ASN RD&A’s letter in appendix III. DOD and the Navy also provided technical comments that we incorporated as appropriate. We are sending copies of this report to the Secretary of Defense, Secretary of the Navy, interested congressional committees, and other interested parties. This report will also be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by e-mail at oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. This review assesses the extent to which: (1) the Navy’s shipbuilding programs deliver ships to the fleet that can be sustained as planned; (2) the Navy develops and uses effective key sustainment requirements during the acquisition process; (3) shipbuilding programs effectively identify and evaluate sustainment costs and risks in key acquisition planning documents; (4) Navy and Congressional leadership have insight into and effectively consider programs’ sustainment planning and outcomes; and (5) the shipbuilding programs leverage Product Support Managers (PSMs) during the acquisition process. The scope of our review included all shipbuilding programs for warships that had ships under construction or in development in the last 10 years, from fiscal years 2009 through 2019. We defined a shipbuilding program as under construction if any ship in the class was under construction in the last 10 years. We defined a shipbuilding program as in development if the Navy had awarded a development or design contract for the class in the last 10 years. We excluded military sealift command vessels and other Navy vessels with logistics missions from this review to help ensure that our resources matched the scope of our review. We assessed LHD 8 and CVN 77 as their own classes for the purposes of our review because the Navy considers them to be transitional designs between antecedent classes. These parameters resulted in 11 ship classes for inclusion in our review. We also selected several ships within these classes to serve as case studies for additional analysis. To select these ship-specific case studies, we reviewed all warships delivered from fiscal years 2007-2018 and selected up to four of the most recent hulls within this time frame from each class as case studies. We selected these ships for additional analysis because they are still relatively new, but the fleet has had experience maintaining them and could discuss sustainment challenges for those ships, if any. All ship classes and case study hulls in scope are listed in table 7. Over the course of this audit, we interviewed officials from over 100 Navy organizations involved in designing, building, inspecting, testing, sustaining, and operating Navy ships to gain an understanding of the extent to which they are involved in the acquisition process and how they consider and manage sustainment risk during the acquisition process. These interviews also provided information on the nature and magnitude of sustainment issues being experienced by the fleet on recently delivered ships. These included approximately 30 interviews with organizations reporting to the ASN (RD&A), 31 interviews with organizations reporting to the Chief of Naval Operations, 29 interviews with organizations within Naval Sea Systems Command (NAVSEA), interviews with shipbuilders that have been awarded multiple Navy shipbuilding contracts, and three interviews with other Department of Defense (DOD) entities. We conducted these interviews in several locations throughout the United States, including Washington, D.C.; San Diego, CA; Norfolk, VA; Philadelphia, PA; and Mechanicsburg, PA. During visits to naval bases, we toured DDG 111, DDG 1000, LHD 8, LPD 22, LCS 3, LCS 4, CVN 77, and CVN 78. To identify the extent to which ships can be sustained as planned, we interviewed shipbuilding program officials, in-service program officials, engineers, and fleet organizations, as well as analyzed ship and system performance data from many Navy organizations. Through this assessment, we identified and analyzed 150 significant class-wide issues across the shipbuilding programs in our scope that required more sustainment resources than planned. Such issues include systems or parts that exhibited poor design, construction, reliability, or planning; systems that were obsolete before or soon after ship delivery; and systems that could not be maintained by the fleet due to vendor or manufacturer proprietary information. We counted only issues that were class-wide, meaning they were related to ship design, equipment used across the class, or construction procedures, rather than hull-specific issues that could be caused by a unique accident or sailor error. We also did not assess issues related to fleet preference. For example, one ship’s crew told us they did not prefer the location in which consoles for operating a certain system were installed, as they are typically installed in a different location on other ship classes. However, because the consoles were installed in the location specified in the design, we eliminated this issue from our analysis. We also eliminated issues if maintenance and other work on the affected system were accounted for during the acquisition process in the program’s initial Operating and Support (O&S) cost estimate, rather than being an unexpected expense. For example, program offices can address expected obsolescence by budgeting for future system modernizations or purchasing quantities of spare parts that will last for the ship’s entire life cycle. To identify the costs associated with fixing problems that are the result of not being able to sustain ships as planned, we reviewed documentation from Navy organizations, budget justifications, and estimates provided by Navy officials. We were able to collect cost information for 30 percent of the problems reported to us by the fleet. To assess the extent to which maintenance schedules are executed as planned, we analyzed Navy data on regularly scheduled, depot-level maintenance periods for surface ships—including those maintained at overseas homeports and in the United States. NAVSEA collects and manages data on these maintenance periods—known as Chief of Naval Operations maintenance availabilities—for surface ships, submarines, and aircraft carriers. We obtained the data on surface ship depot-level maintenance periods used by NAVSEA’s Surface Maintenance Engineering Planning Program and the Commander, Navy Regional Maintenance Center. We reviewed the data we obtained for inconsistency errors and, when possible, obtained multiple documents that discussed the same problem for validation. We then discussed these problems with multiple officials across the Navy, including officials involved in ship maintenance and operation. From these efforts, we determined that these data are sufficiently reliable for the purposes of this report. To assess the extent to which shipbuilding programs develop and use effective sustainment requirements during the acquisition process, we reviewed DOD requirements setting policy and determined the extent to which shipbuilding programs set requirements in accordance with this policy. In doing so, we assessed the extent to which DOD policy aligned with fleet experience and captured all factors that influence ship availability and analyzed any discrepancies. We then assessed the extent to which the Navy set sustainment requirements that contributed to well- informed decision-making throughout the acquisition process and in accordance with DOD policy and Navy guidance. To assess how accurately the Navy measures operational availability and materiel availability outcomes, we reviewed the Navy’s operational availability measurements as reported in Selected Acquisition Reports to Congress, and compared these values to fleet reliability data and casualty reports, as well as information about the ships’ performance obtained in interviews with Chief of Naval Operations and NAVSEA officials. To assess the extent to which shipbuilding programs effectively identify and evaluate sustainment costs and risks in key acquisition planning documents, we evaluated the Navy’s development and use of life-cycle cost estimates, Life-Cycle Sustainment Plans and Independent Logistics Assessments. To evaluate the Navy’s development of O&S cost estimates, we reviewed the life-cycle cost estimates created when programs were in development and compared them to updated estimates of O&S costs reported in Selected Acquisitions Reports and Navy provided data. We adjusted program estimates for quantity to more accurately capture cost growth between initial and current O&S estimates. Further, we adjusted the estimates for inflation to compare the O&S estimates as accurately as possible. For programs that experienced O&S cost growth, we interviewed program officials and Navy cost estimators to determine the process that the Navy’s cost estimators used to build O&S cost estimates for shipbuilding programs and to discuss the reasons for cost growth. We also reviewed DOD cost estimation guidance to determine whether the cost estimators and programs complied with its requirements. While we have previously found issues with the reliability of the Navy’s cost estimates, we believe that the cost estimates we reviewed are sufficiently reliable for the purposes of this report. To evaluate the Navy’s use of key sustainment planning documents, we reviewed LCSPs and ILAs for programs in our scope. We interviewed program, NAVSEA, fleet, and maintenance officials to determine the extent to which the LCSPs and ILAs for those programs were used to plan for sustainment, including whether these documents identified and mitigated sustainment risks. We compared the results of the ILAs to realized ship sustainment problems that we identified through interviews shipbuilding program officials, in-service program officials, engineers, and fleet organizations, as well as to analyses of ship and system performance data from many Navy organizations. To evaluate the extent to which Navy and Congressional leadership has insight into and considers sustainment planning and outcomes, we examined the Navy’s Gate review process and Congress’ Nunn-McCurdy breach process. To assess the Navy’s use of the Gate review process, we reviewed Navy acquisition policy governing the reviews, as well as the briefings and meeting minutes from reviews for programs in our scope from fiscal years 2014 through 2018. We compared the content of the briefings and meeting minutes to the acquisition policy to determine the extent to which required sustainment topics were briefed and discussed at each review and identified other mentions of sustainment issues that were outside the scope of the policy requirements. We also reviewed a recent revision to Navy acquisition policy that creates a Gate 7 review for sustainment and interviewed senior Navy officials to obtain their perspectives on how Gate 7 will affect ship sustainment. To assess Navy leadership’s effectiveness in holding shipbuilding programs’ accountable for achieving sustainment outcomes using Acquisition Program Baselines (APB), we reviewed statute that established the APB as well as the findings of the Section 809 Panel, which recommended the creation of the SPB to supplement the APB. We also interviewed Navy officials involved in developing the SPB framework in accordance with the Panel’s recommendations to obtain information on their work. To determine what information Navy shipbuilding programs are required to provide to Congress about sustainment cost issues during the acquisition process, we reviewed the statutory requirements found in Nunn-McCurdy, a key Congressional oversight tool requiring information about baselines and cost estimate growth. We also assessed how the Nunn-McCurdy breach influenced programs’ management of acquisition and sustainment costs by interviewing Navy officials in the shipbuilding program offices, Office of the Chief of Naval Operations, and ASN (RD&A) offices, among others. Additionally, we reviewed O&S cost growth for programs in our scope and compared the percent increase to the 50 percent cost growth threshold used for Nunn-McCurdy acquisition cost breaches to determine if the sustainment cost growth was of a magnitude the Congress considers critical for acquisition costs. To assess how shipbuilding programs leverage PSMs during the acquisition process, we reviewed DOD and Navy acquisition guidance governing the roles and responsibilities of program offices, program managers, and PSMs. We interviewed officials from shipbuilding programs in our scope about their priorities and responsibilities throughout the life cycle of a ship class. Further, we reviewed legislation creating the PSM role, DOD and Navy acquisition guidance regarding PSMs, prior GAO reporting on PSMs, and interviewed PSMs from programs in our scope. We compared the key acquisition activities that legislation requires PSMs to participate in with the activities the PSMs reported they had participated in. We also compared DOD and Navy guidance on assigning PSMs to a program office to when program officials told us the PSMs needed to be assigned to be effective. We also reviewed findings that NAVSEA logistics officials reached about the authority and effectiveness of PSMs. We conducted this performance audit from April 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to responding to our recommendations, the Assistant Secretary of the Navy for Research, Development, and Acquisition (ASN (RD&A)) provided observations on a number of issues related to the findings in our report. In his letter, the ASN (RD&A) agreed with our recommendations but sought to add context to our report’s conclusion that the Navy can save billions by improving its consideration of sustainment throughout the acquisition process. Our response to the ASN (RD&A)’s letter is as follows. In his letter, the ASN (RD&A) highlighted a number of changes that the Navy has instituted over the last 10 years to improve sustainment planning, including policies pertaining to life cycle sustainment plans and independent logistics assessments, strengthening the role of the Product Support Managers, and establishing a new Gate 7 review focused on sustainment. We agree that the Navy’s framework for including sustainment planning in the acquisition process offers promise and we discuss these policies and processes in depth in this report. However, we found considerable weaknesses in the Navy’s application of its own policies. Specifically, we found that the Navy did not provid a thorough assessment of the sustainment implications and risks in its LCSPs and ILAs and Product Support Managers aften are not assigned until well into a shipbuilding program thereby limiting their influence on early acquisition decisions. While adding a Gate 7 offers benefits, it is not a substitute for discussions about sustainment concepts during earlier Gates, when key long-term decisions are being made. Our findings and recommendations demonstrate that DOD and the Navy should better use the policies and processes it currently has, including the Gate reviews, as well as Product Support Managers, LCSPs and ILAs, to improve their understanding of how their acquisition decisions will affect sustainment. In his letter, the ASN (RD&A) stated that many of the Navy’s ship programs were designed with sustainment initiatives early in the acquisition process and, further, acknowledged that these initiatives did not achieve efficiencies as initially planned. We agree with both of these points, as we discuss in depth in this report. Whereas the ASN (RD&A) indicated in his letter that leadership, philosophical, and technology changes can lead to outcomes that were not originally envisioned, we found that these initiatives largely failed because, early in the acquisition process, the Navy did not sufficiently assess the costs or evaluate the risks associated with pursuing these initiatives. Absent such analysis, the Navy did not mitigate the risks that threatened their success. The ASN (RD&A) highlighted several examples of sustainment initiatives considered early in the acquisition process for several ships. We believe that these examples (many of which we discuss in our report) serve to further highlight our findings. For example: The ASN (RD&A) discussed the use of a “full service contractor,” meaning performance-based logistics for LPD 17 class ships. According to the ASN (RD&A), while this approach had been successfully used for aircraft, the Navy had never applied it to ships. As we state in our report, in attempting to use performance-based logistics for several shipbuilding programs including LPD 17 class ships, the Navy did not consider the challenges in implementing this radical departure from traditional ship maintenance and did not consult the fleet on this change until after ships were delivered. The Navy’s life-cycle sustainment plans and cost estimates for several shipbuilding programs did not articulate how much the performance-based logistics approach was likely to cost or what sustainment outcomes the Navy expected. For instance, for three out of the four programs that pursued performance-based logistics, the Navy learned that this approach was cost-prohibitive once it began seeking contractors to sustain its ships. The ASN (RD&A) stated that the Navy’s focus on Ford class sustainment has reduced sustainment costs and labor by an estimated $4 billion across the Ford class carriers compared to the previous class of carriers. However, it is too early to tell how much the Navy will save compared to the cost of its previous class of carriers because the Navy’s fleet has yet to operate the new carrier. Further, while the O&S estimated for the Ford class may currently be lower than the previous carrier class, our report notes that the O&S costs for the Ford class carrier program are nearly $46 billion more than initially estimated. Finally, in his letter, the ASN (RD&A) stated that the Navy plans to correct the vast majority of CVN 78 sustainment problems (including those we identified in this report) with ship construction funding—and these cost will not be passed on to the fleet. The $4.2 billion to address the 150 problems that we identified in this report already excludes all ship construction funding and also excludes corrections on CVN 78. Our calculation of $4.2 billion only includes the costs to correct the problems that are not funded using ship construction funding. We agree with the ASN (RD&A)’s assertion that external factors can take place over the lengthy time needed to design and build a new ship that can lead to changes that were not initially envisioned. While the Navy cannot prepare for all of the unknowns, it can critically evaluate sustainment assumptions that form the basis of its shipbuilding programs early in the acquisition process. Such analysis could significantly improve the Navy’s ability to response to changes over time and increase the likelihood of success. Further, critical analysis could also help decision makers determine when an initiative is too risky before implementing it on an entire shipbuilding program. In its letter, the ASN (RD&A) also states that a careful reading of the early program documentation demonstrates that sustainment stakeholders were involved in the acquisition process. We reviewed available acquisition documents for 11 shipbuilding programs in the last 20 years and found that sustainment leadership, specifically the CNO and other in OPNAV, attended meetings and approved sustainment planning documents. However, we found that sustainment was rarely discussed during early acquisition meetings—even when the planned shipbuilding programs sought new sustainment initiatives. Further, we reviewed thousands of Navy documents and met with over 100 Navy organizations and found that sustainment organizations across the Navy that are responsible for ship sustainment have a limited role in the acquisition process, even when having such a role could have likely prevented many of the problems we discuss in the report. As we state in our report, the quantity and breadth of the 150 problems we found— resulting in billions of dollars in unexpected costs, maintenance delays, and unreliable ships—suggest that existing policies and guidance have not ensured that new ships are reliable and can be sustained as planned. We are concerned that the ASN (RD&A)’s letter is an indication that the Navy’s shipbuilding program offices will not take the necessary action to improve sustainment planning during the acquisition process. The ASN (RD&A)’s letter did not mention the recent establishment of a new Deputy Assistant Secretary for Sustainment that we discuss in our report. We believe that this office has the opportunity to contribute to improved outcomes byp providing leadership to ensure that sustainment considerations are critically evaluated during the acquisition process. Absent such leadership, the Navy is at risk of continuing to provide ships to the fleet that are incomplete, unreliable, and cost more than expected to maintain. In addition the contact name above, the following staff members made key contributions to this report: Diana Moldafsky, Assistant Director; Laurier Fish, Analyst-in-Charge; Jillian Schofield; Sarah Evans; Lori Fields; Ann Halbert-Brooks; Joshua Garties; Laura Greifner; Tara Kumar; Shivarthn Maniam; Alexis Olson; Kya Palomaki; Anne Louise Taylor; and Tonya Woodbury. Carl Barden; Brian Bothwell; Anna Irvine; and Jean McSween also made contributions to this report. Defense Acquisitions: Senior Leaders Should Emphasize Key Practices to Improve Weapon System Reliability. GAO-20-151. Washington, D.C.: January 14, 2020. Guided Missile Frigate: Navy Has Taken Steps to Reduce Acquisition Risk, but Opportunities Exist to Improve Knowledge for Decision Makers. GAO-19-512. Washington, D.C.: August 9, 2019. DOD Acquisition Reform: Leadership Attention Needed to Effectively Implement Changes to Acquisition Oversight. GAO-19-439. Washington, D.C.: June 5, 2019. Columbia Class Submarine: Overly Optimistic Cost Estimate Will Likely Lead to Budget Increases. GAO-19-497. Washington, D.C.: April 8, 2019. DOD Depot Workforce: Services Need to Assess the Effectiveness of Their Initiatives to Maintain Critical Skills. GAO-19-51. Washington, D.C.: December 14, 2018. Navy and Marine Corps: Rebuilding Ship, Submarine, and Aviation Readiness Will Require Time and Sustained Management Attention. GAO-19-225T. Washington, D.C.: December 12, 2018. Navy Readiness: Actions Needed to Address Costly Maintenance Delays Facing the Attack Submarine Fleet. GAO-19-229. Washington, D.C.: November 19, 2018. Navy Readiness: Actions Needed to Address Costly Maintenance Delays Affecting the Attack Submarine Fleet. GAO-19-192C. Washington, D.C.: October 31, 2018. Navy Shipbuilding: Past Performance Provides Valuable Lessons for Future Investments. GAO-18-238SP. Washington, D.C.: June 6, 2018. Columbia Class Submarine: Immature Technologies Present Risks to Achieving Cost, Schedule, and Performance Goals. GAO-18-158. Washington, D.C.: December 21, 2017. Navy Readiness: Actions Needed to Address Persistent Maintenance, Training, and Other Challenges Affecting the Fleet. GAO-17-809T. Washington, D.C.: September 19, 2017. Naval Shipyards: Actions Needed to Improve Poor Conditions That Affect Operations. GAO-17-548. Washington, D.C.: September 12, 2017. Weapon Systems Management: Product Support Managers’ Perspectives on Factors Critical to Influencing Sustainment-Related Decisions. GAO-17-744R. Washington, D.C.: September 12, 2017. Navy Readiness: Actions Needed to Address Persistent Maintenance, Training, and Other Challenges Facing the Fleet. GAO-17-798T. Washington, D.C.: September 7, 2017. Navy Shipbuilding: Policy Changes Needed to Improve the Post-Delivery Process and Ship Quality. GAO-17-418. Washington, D.C.: July 13, 2017. Navy Force Structure: Actions Needed to Ensure Proper Size and Composition of Ship Crews. GAO-17-413. Washington, D.C.: May 18, 2017. Navy Ship Maintenance: Action Needed to Maximize New Contracting Strategy’s Potential Benefits. GAO-17-54. Washington, D.C.: November 21, 2016. Littoral Combat Ship: Need to Address Fundamental Weaknesses in LCS and Frigate Acquisition Strategies. GAO-16-356. Washington, D.C.: June 9, 2016. Defense Inventory: Further Analysis and Enhanced Metrics Could Improve Service Supply and Depot Operations. GAO-16-450. Washington, D.C.: June 9, 2016. Military Readiness: Progress and Challenges in Implementing the Navy’s Optimized Fleet Response Plan. GAO-16-466R. Washington, D.C.: May 2, 2016. Navy and Coast Guard Shipbuilding: Navy Should Reconsider Approach to Warranties for Correcting Construction Defects. GAO-16-71. Washington, D.C.: March 3, 2016. Acquisition Reform: DOD Should Streamline Its Decision-Making Process for Weapon Systems to Reduce Inefficiencies. GAO-15-192. Washington, D.C.: February 24, 2015. Ford-Class Aircraft Carrier: Congress Should Consider Revising Cost Cap Legislation to Include All Construction Costs. GAO-15-22. Washington, D.C.: November 20, 2014. Weapon Systems Management: DOD Has Taken Steps to Implement Product Support Managers but Needs to Evaluate Their Effects. GAO-14-326. Washington, D.C.: April 29, 2014. Navy Shipbuilding: Opportunities Exist to Improve Practices Affecting Quality. GAO-14-122. Washington, D.C.: November 19, 2013. Trends in Nunn-McCurdy Cost Breaches for Major Defense Acquisition Programs. GAO-11-295R. Washington, D.C.: March 9, 2011. Defense Management: DOD Needs Better Information and Guidance to More Effectively Manage and Reduce Operating and Support Costs of Major Weapons Systems. GAO-10-717. Washington, D.C.: July 20, 2010. Best Practices: High Levels of Knowledge at Key Points Differentiate Commercial Shipbuilding from Navy Shipbuilding. GAO-09-322. Washington, D.C.: May 13, 2009. Defense Logistics: Improved Analysis and Cost Data Needed to Evaluate the Cost-effectiveness of Performance Based Logistics. GAO-09-41. Washington, D.C.: December 19, 2008. Defense Acquisitions: Cost to Deliver Zumwalt-Class Destroyers Likely to Exceed Budget. GAO-08-804. Washington, D.C.: July 31, 2008. Defense Acquisitions: Realistic Business Cases Needed to Execute Navy Shipbuilding Programs. GAO-07-943T. Washington, D.C.: July 24, 2007.", "summary": "The U.S. Navy requested over $40 billion each of the last 3 years to build, operate, and sustain its fleet. Acquisition decisions made as ships are developed and built can have a long-term effect on sustainment costs and ship quality. GAO was asked to assess the extent to which DOD considers and plans for sustainment when acquiring weapons. Among other objectives, this report assesses the extent to which: (1) Navy ship programs deliver ships to the fleet that can be sustained as planned; (2) the Navy develops and uses effective sustainment requirements during acquisition; (3) ship programs are effectively identifying and evaluating sustainment risks in planning documents; and (4) leadership considers programs' sustainment planning and outcomes. GAO reviewed DOD and Navy acquisition policy and guidance, evaluated acquisition plans, collected sustainment metrics, and conducted interviews with more than 100 organizations, including program office and fleet units. GAO assessed 11 classes of shipbuilding programs (all nine that delivered warships during the last 10 years, as well as two newer classes of ships). The Navy has delivered warships—such as aircraft carriers, destroyers, and submarines—to its fleet over the past 10 years that require more effort to sustain than initially planned. In assessing how these classes of ships are sustained, GAO found 150 examples of class-wide problems, such as unreliable ship systems. These problems stemmed from shipbuilding programs not identifying, evaluating, or mitigating sustainment risks during the acquisition process. GAO found that it would cost the Navy $4.2 billion to correct just the 30 percent of these problems for which the Navy had data on estimated repair costs. GAO found that shipbuilding programs' requirements for sustainment reflect weaknesses with how Department of Defense (DOD) policy defines these requirements for ships. Sustainment requirements should influence acquisition decisions that determine the sustainability of a ship class, such as the ship's design. However, the Navy's sustainment requirements do not provide key information on how reliable and maintainable mission-critical systems should be and, therefore, cannot adequately inform acquisition decisions. GAO also found that shipbuilding programs did not consistently address sustainment risks in acquisition planning documents. For example, the operating and support costs included in cost estimates did not capture all sustainment risks that could affect costs or evaluate sensitivity to changing sustainment assumptions, contrary to DOD and Navy cost estimating guidance. As a result, for six shipbuilding programs whose costs GAO could assess, the Navy had underestimated sustainment costs by $130 billion, as shown below. The Navy has begun making some changes to its acquisition oversight process, such as developing sustainment program baselines and adding a sustainment oversight review. While positive, these changes focus on considering sustainment after key decisions are made early in the acquisition process. GAO also found that DOD is not required to provide detailed information about shipbuilding programs' sustainment cost growth to Congress. As such, Congress does not have full insight into the extent of shipbuilding programs' cost growth and why such growth occurred. GAO is making one matter for Congressional consideration to enhance oversight and 11 recommendations to help DOD and Navy improve ship sustainment. DOD concurred with 8 and partially concurred with 3 recommendations but did not describe specific actions, which GAO believes are necessary to improve sustainment outcomes.", "document_type": "gao"}
{"report": "In September 2017, Hurricane Irma struck the islands of St. Thomas and St. John, and two weeks later, Hurricane Maria struck the island of St. Croix in the USVI, causing catastrophic damage across the entire territory and neighboring Caribbean islands (see fig. 2). The storms severely damaged the USVI’s critical infrastructure, leaving many of the territory’s 107,000 residents without electricity, phone service, food, or running water. According to a September 2018 report from the USVI Hurricane Recovery and Resilience Task Force (USVI Task Force Report), the hurricanes devastated the territory’s electricity grid and telecommunications systems, shutting down both for months. Further, the storms damaged more than half of the territory’s housing units, as well as its hospitals, government buildings, schools, water and wastewater facilities, and more (see figs. 3 and 4). Overall, the report estimated that the hurricanes caused approximately $10.7 billion in total damages across the USVI. In response to the request of the Governor of the USVI, the President declared a major disaster the day after each hurricane struck the territory. Major disaster declarations can trigger a variety of federal response and recovery programs for government and nongovernmental entities and households and individuals, including assistance through the Public Assistance program and Hazard Mitigation Grant Program. Under the National Response Framework and National Disaster Recovery Framework, DHS is the federal department with primary responsibility for coordinating disaster response and recovery, and within DHS, FEMA has lead responsibility. The Administrator of FEMA serves as the principal adviser to the President and the Secretary of Homeland Security regarding emergency management. Once the President has declared a major disaster, FEMA, the state or territorial government (the recipient), and local or territorial entities (the subrecipient) work together to, among other things, identify and develop projects through the Public Assistance program and Hazard Mitigation Grant Program. After a project has completed FEMA’s review process and is approved, FEMA obligates funding for the project by placing money into an account where the recipient has the authority to draw down—or expend—funding to pay for eligible work upon completion. Further, when a project has been completed, FEMA conducts a close-out process to certify that all eligible work has been completed and reconciles the actual cost incurred. If the actual cost of the completed work is less than the amount of money FEMA obligated for the project, FEMA will deobligate funding. However, if the actual cost of the completed work is greater than the amount of money FEMA obligated for the project, FEMA may reimburse the subrecipient for these additional costs. A state or territorial governor may designate a governor’s authorized representative to oversee all aspects of disaster assistance—including Public Assistance program and Hazard Mitigation Grant Program funding—to ensure the USVI’s compliance with federal regulations and FEMA requirements. Among other responsibilities, the governor’s authorized representative is to confirm that subrecipients submit complete documentation demonstrating that all eligible work completed is in accordance with program requirements. FEMA’s Public Assistance program provides grant funding to state, territorial, local, and tribal governments, as well as certain types of private nonprofit organizations, to assist with the repair or replacement of disaster-damaged public infrastructure. To develop projects under the Public Assistance program, FEMA and USVI officials collaborate to identify and document the damage caused by a disaster to a particular system or facility. These officials then use this damage assessment to formulate the scope of work—or activities required to fix the identified damage—as well as the estimated cost of these activities. As shown in figure 5, Public Assistance grant funds are organized broadly as “emergency work” or “permanent work.” Within these areas are separate categories of work. In addition to emergency work and permanent work, the program includes category Z, which represents indirect costs, direct administrative costs, and any other administrative expenses associated with a specific project. Under the Public Assistance program’s permanent work categories, FEMA also provides grant funding for cost-effective hazard mitigation measures to reduce or eliminate the long-term risk to people and property from future natural and man-made disasters and their effects. FEMA provides this funding in conjunction with the repair of disaster-damaged facilities to enhance their resilience during future disasters. For example, this funding could be used to replace damaged wooden utility poles with composite fiberglass ones to increase the resilience of an electricity distribution system and mitigate the potential for future damage from hurricane-force winds. FEMA’s Hazard Mitigation Grant Program provides grant funding for long- term mitigation solutions to reduce the risk of loss of life and property from future disasters. Unlike mitigation measures funded through the Public Assistance program to further protect disaster-damaged infrastructure, the Hazard Mitigation Grant Program may fund measures for systems, facilities, or properties that were not damaged in the disaster. For example, program funding can be used to construct floodwater control measures—such as berms and rock linings—that did not exist prior to the disaster, or to update existing hazard mitigation plans to accurately reflect current mitigation goals. In July 2018, FEMA approved the use of the Public Assistance alternative procedures pilot program for permanent work projects in the territory. Unlike the standard Public Assistance program wherein FEMA will fund the actual cost of a project, the alternative procedures require awards for permanent work projects to be made on the basis of fixed-cost estimates. As a result, the recipient or subrecipient is ultimately responsible for any project costs that exceed the agreed-upon fixed-cost estimate at the time of the close-out process. However, the alternative procedures program also provides the USVI with financial incentives for the timely and cost-effective completion of work and additional flexibilities that are not available through the standard Public Assistance program. For example, the USVI may use excess grant funding for cost-effective hazard mitigation measures and, in certain circumstances, consolidate permanent work projects approved under the alternative procedures and share obligated funding across these projects. Further, section 20601 of the Bipartisan Budget Act of 2018 authorizes FEMA, when using the alternative procedures, to provide assistance to fund the replacement or restoration of disaster-damaged infrastructure that provide critical services without regard to pre-disaster condition (see fig. 6). For example, through the Act, FEMA may fund the restoration of a disaster-damaged school building—which provides a critical service—to accepted industry standards applicable to the construction of education facilities. Therefore, according to FEMA policy, if components of the school building were not up to industry standards or in poor condition prior to the 2017 hurricanes, the Act allows FEMA to fund the restoration of this building to a better condition than it was in prior to the storms. The Sheltering and Temporary Essential Power (STEP) pilot program is an emergency sheltering program implemented under FEMA’s emergency work authority and funded through the Public Assistance program’s category B emergency work. The program—which was created following Hurricane Sandy in 2012—allows FEMA to fund emergency, temporary repairs to make damaged homes habitable by, for example, restoring electricity to a private home and applying temporary patches to roofs and windows to protect the interior. In funding these types of repairs, FEMA’s goal is to quickly make damaged homes habitable in the short term until the homeowner could complete more permanent repairs independently through other FEMA programs or private insurance payments. Since 2012, FEMA has implemented the program as a tool in addressing the unique circumstances and challenges associated with providing safe sheltering options for disaster survivors. As of June 30, 2019, FEMA obligated more than $1.9 billion in grant funding for 640 projects through the (1) Public Assistance program and (2) Hazard Mitigation Grant Program in the USVI. First, FEMA obligated more than $1.8 billion in Public Assistance grant funding for 618 projects across the USVI (see fig. 7). Specifically, FEMA obligated more than $1.1 billion for emergency work projects (categories A and B), about $588.5 million for permanent work projects (categories C through G), and about $141.2 million for management costs (category Z). Of the approximately $1.8 billion FEMA obligated in Public Assistance grant funding as of June 30, 2019, the USVI had expended nearly $1.1 billion (59 percent) to reimburse subrecipients for completed work. Appendix I provides more detailed information on the status of Public Assistance grant funding in the USVI. Second, FEMA obligated about $60.6 million for 22 Hazard Mitigation Grant Program projects in the territory as of June 30, 2019. These projects are designed to fund mitigation measures to increase the longer- term resilience of the USVI’s infrastructure during future disasters. Of the $60.6 million FEMA obligated as of June 30, 2019, the USVI expended about $1.7 million (3 percent) across 5 projects. Appendix II provides more detailed information on the status of Hazard Mitigation Grant Program funding in the USVI. While these data represent the status of grant funding as of June 30, 2019, the amount of FEMA obligations and USVI expenditures for both programs will likely increase over time as additional projects are finalized and approved. FEMA, USVI officials, and contractor personnel identified challenges across three areas that affected the implementation of the Public Assistance program and the Hazard Mitigation Grant Program in the USVI. Specifically, they cited: (1) the limited availability of local staff in the USVI to implement and oversee recovery programs, (2) the inability of local construction crews to undertake the large number of recovery projects, and (3) the impact of the USVI’s difficult fiscal situation on recovery efforts. Limited availability of local staff. USVI and FEMA officials cited the limited number of local USVI personnel with the knowledge and expertise necessary to staff recovery-related positions in key USVI agencies as a significant challenge following the 2017 hurricanes. For example, USVI officials told us that the Virgin Islands Territorial Emergency Management Agency—the agency initially responsible for overseeing all aspects of both the Public Assistance program and Hazard Mitigation Grant Program in the territory—did not have enough employees on staff to effectively implement and manage these programs. Further, a senior FEMA official noted that after the storms, the USVI had only one individual responsible for managing all aspects of the Hazard Mitigation Grant Program across the territory. In addition, the limited availability of local staff in the USVI was exacerbated by the departure of qualified individuals following the hurricanes as well as competition among recovery agencies to hire qualified staff that remained in the territory, according to USVI officials. To address these challenges, the USVI hired two contractors to augment the territory’s capacity in the shorter term and established a new Office of Disaster Recovery to oversee recovery efforts in the longer term. First, in December 2017, the USVI hired two contractors to assist the territory in planning, developing, implementing, and overseeing recovery projects, among other responsibilities. Second, in February 2019, the USVI established the Office of Disaster Recovery as the primary territorial agency responsible for coordinating and overseeing all disaster recovery efforts in the USVI, including the Public Assistance program and Hazard Mitigation Grant Program. The office’s Director told us that while contractor personnel had been valuable in augmenting the USVI’s capacity, the territory was prioritizing the hiring and training of qualified local hires to replace these contractors for the longer term. Shortage of local construction crews. Due to the territory’s relatively small population, FEMA and USVI officials stated there were not enough local construction crews to address the large amount of construction work required to repair and rebuild damaged infrastructure following the 2017 hurricanes. These officials told us this construction crew shortage affected the USVI’s ability to keep Public Assistance program and Hazard Mitigation Grant Program projects proceeding on time. FEMA and contractor personnel stated that unlike in the contiguous United States, the USVI does not have neighboring states that can easily send construction crews to affected areas to augment local crews. In addition, historically, the USVI relied on Puerto Rico to supplement the territory’s capacity, but this was not an option as Puerto Rico was undergoing its own massive recovery effort as a result of Hurricane Maria. The USVI’s fiscal situation. USVI officials and contractor personnel stated that the challenging fiscal situation in the territory directly affected its ability to effectively implement recovery programs. Specifically, USVI officials told us that the territory’s financial condition made it difficult to provide initial funding to reimburse subrecipients for completed work prior to drawing down funding from the account holding FEMA-obligated money. These officials stated this process was problematic because instead of funding all eligible projects as quickly as possible to move the recovery forward, the USVI had to prioritize certain recovery projects over others based on the availability of funding. Further, USVI contractor personnel told us that the territory often does not have the cash on hand necessary to provide these reimbursements to subrecipients, which can result in delays in paying subrecipients and contractors. According to USVI officials, pursuing projects under the Public Assistance alternative procedures program may help to address these issues by providing more flexibility regarding when and how projects are funded. The Public Assistance alternative procedures program provides the USVI with financial incentives and new flexibilities in implementing recovery projects that are not available through the standard Public Assistance program. However, FEMA and USVI officials stated that implementing the alternative procedures program in the USVI presented challenges that affected recovery efforts and delayed the obligation of funding for permanent work projects. Specifically, they cited challenges in (1) developing accurate fixed-cost estimates for program projects and (2) implementing the new flexibilities authorized by section 20601 of the Bipartisan Budget Act of 2018. Senior USVI officials told us that due to these challenges and the financial risk associated with the use of fixed- cost estimates, the USVI is planning to take a cautious approach in pursuing alternative procedures projects. As established in FEMA guidance, USVI officials have a deadline of March 2020 to finalize the fixed-cost estimates for such projects for inclusion in the alternative procedures program. Fixed-cost estimates. As the USVI is financially responsible for any actual costs that exceed the fixed-cost estimate for any given alternative procedures project, ensuring the accuracy of these estimates is critical due to the USVI’s already difficult fiscal situation. However, USVI officials told us that developing fixed-cost estimates that accurately forecast the future costs of completing large, complex permanent work projects in the remote island territory is difficult given the unique circumstances that influence construction costs in the USVI, such as the limited availability of local resources and the need to import construction materials and labor. To address this challenge, in October 2018, FEMA asked an independent contractor to analyze whether a USVI-specific “cost factor” should be incorporated into FEMA’s process for developing fixed-cost estimates to ensure the actual costs of implementing permanent work projects in the territory were captured. According to FEMA officials, the independent contractor determined that a cost factor was appropriate for use in the USVI and the contractor proposed several options. However, territorial officials contended that these proposals did not sufficiently or accurately capture the unique circumstances that influence construction costs in the territory. Further, USVI officials stated that ensuring the accuracy of the cost factor was critical given the significant financial risk using fixed-cost estimates posed to the USVI. Since incorporating a cost factor into the process for developing fixed- cost estimates increases the base cost for any given permanent work project—and therefore the amount of funding FEMA obligates—FEMA officials explained the USVI had an incentive to delay the obligation of projects until FEMA finalized this factor. As a result, FEMA officials told us in May 2019 that obligations for permanent work projects had been mostly on hold since October 2018 while the contractor worked to develop the USVI-specific cost factor. As the USVI is reliant on federal recovery funding to reimburse subrecipients for completed work, this delay in obligations directly affected the USVI’s ability to move recovery projects forward. In May 2019, the contractor proposed a new cost factor, which FEMA approved on an interim basis pending further analysis. In July 2019, FEMA officials told us that while additional analyses are required to ensure its final process for developing fixed-cost estimates in the USVI accurately captures construction costs, using this interim cost factor in the meantime allows FEMA and USVI officials to move forward with the development and final approval of alternative procedures projects. In August 2019, a senior USVI official told us the territory plans to begin using the interim cost factor, where appropriate, to keep projects progressing forward. However, she stated that the USVI questioned whether the interim cost factor did, in fact, sufficiently capture the actual costs of construction in the USVI. Given the uncertainty around these fixed-cost estimates, USVI officials told us the territory will need to balance the potential flexibilities provided by the alternative procedures program with the financial risk posed by cost overruns when deciding whether to use the alternative procedures or the standard Public Assistance program for any given permanent work project. We are currently assessing FEMA’s process for developing cost estimates for projects under both the standard and alternative procedures programs, and plan to report our results in early 2020. The Bipartisan Budget Act of 2018. While FEMA and USVI officials told us that section 20601 of the Bipartisan Budget Act presented a valuable opportunity to advance the USVI’s recovery, they also reported challenges with implementing the new flexibilities authorized by the Act, which made developing eligible permanent work projects difficult. For example, USVI officials stated that, at times, they were unclear about the implementation process for key components of the Act and thus ensuring subrecipients understood the process was difficult. Further, FEMA officials in the USVI told us that initially, they had difficulty obtaining clarification from FEMA headquarters regarding how to implement key provisions of the Act, such as the process for identifying and incorporating relevant industry standards for specific alternative procedures projects. As a result, permanent work projects that were eligible to use the flexibilities provided by the Act remained on hold until FEMA could clarify the process for implementing the Act and pertinent industry standards could be approved. In addition, the Bipartisan Budget Act was signed into law in February 2018 and applies exclusively to federal disaster assistance to the USVI and Puerto Rico. As a result, FEMA officials faced the challenge of interpreting the Act’s language and appropriately implementing its provisions for the first time. For example, the Act allows for a new process for determining whether a disaster-damaged facility is eligible to receive funding to (1) repair the existing facility or (2) replace the facility with a new structure. Under the standard Public Assistance program, this determination is calculated using the “50 percent rule”—if the cost of repairing the disaster-related damage sustained by the facility exceeds 50 percent of the cost of replacing it, FEMA may fund the replacement of the facility. In contrast, the Act does not provide a similar cost estimating process for use in developing fixed-cost estimates through the alternative procedures program. In September 2018, FEMA issued guidance for implementing section 20601 of the Bipartisan Budget Act through the Public Assistance alternative procedures program, which provides that critical services infrastructure—such as medical and educational facilities—is eligible for replacement “if repair is feasible, but replacement is more prudent.” FEMA officials in the USVI told us that since the agency did not have further guidance or criteria on the appropriate process for evaluating repair or replacement under this new standard, they were responsible for developing the agency’s first justification to support the replacement of a hospital in St. Croix based on their interpretation of the new standard. These officials also stated that since their rationale justifying the facility’s eligibility for replacement was the first of its kind and would set a precedent for future projects, they submitted it to FEMA headquarters for review. In May 2019, FEMA officially approved the replacement of this hospital through the alternative procedures program. For more information on how the Public Assistance program and the Bipartisan Budget Act are affecting recovery efforts at this facility, see appendix III. The Additional Supplemental Appropriations for Disaster Relief Act of 2019, which was signed into law in June 2019, provides additional direction to FEMA regarding the implementation of section 20601 of the Act. Among other things, this legislation includes a provision directing FEMA to change its process for determining whether a disaster-damaged facility is eligible for repair or replacement. FEMA evaluated this and other provisions of the Act and, in September 2019, issued an updated policy to provide clear guidance moving forward, according to agency officials. The USVI Governor and senior territorial officials stated that due to the challenges outlined above and the financial risk posed by exceeding fixed-cost estimates, the USVI plans to take a cautious approach in implementing the Public Assistance alternative procedures program. Specifically, the Governor told us the territory will most likely pursue alternative procedures projects that are simple, have clear scopes of work, and do not include high levels of uncertainty to reduce the financial risk of potential cost overruns. USVI officials added that if they are not comfortable with the fixed-cost estimate for any given alternative procedures project, the territory has the option to pursue the project under the standard Public Assistance program. Under the standard program, the USVI cannot take advantage of the flexibilities and financial incentives provided by the alternative procedures and the Bipartisan Budget Act, but FEMA would reimburse the USVI for the actual cost—including any cost overruns—of all work completed in accordance with a project’s approved scope of work, thereby mitigating the territory’s financial risk. The USVI is ultimately responsible for deciding whether the benefits provided through the alternative procedures program and the Bipartisan Budget Act outweigh the financial risk associated with agreeing to fixed-cost estimates for permanent work projects. Since the territory has until March 2020 to finalize these fixed-cost estimates, it remains too early to determine the extent to which the alternative procedures program will play a role in the USVI’s long-term recovery strategy. In October 2017, FEMA authorized the STEP pilot program in the USVI in response to the widespread damage to homes that displaced residents and overwhelmed sheltering and temporary housing resources in the territory. Through the program, FEMA funded minimal, temporary protective repairs (or “Phase I” repairs) to private homes to allow residents a safe place to shelter. For example, Phase I emergency repairs included applying temporary patches to roofs and windows to protect the interior from outside weather conditions and ensuring a functional kitchen and bathroom and safe sleeping area. According to FEMA documentation, the intent of these minimal temporary repairs was to quickly make damaged homes habitable in the short term until homeowners could complete more permanent repairs independently through other FEMA programs or using private insurance payments. In August 2018, FEMA expanded the STEP pilot program to include the “permanent” repair or replacement of damaged roofs (or “Phase II” work)—the first time in its history that FEMA authorized such work through this pilot program. Phase II work funded more permanent work on USVI residents’ damaged roofs—either by repairing damages to the existing roof or replacing it with a new one. In addition, Phase II work included incorporating roof hardening measures, such as installing hurricane clips to the roof berms, to increase the resiliency of the roofs against hurricane-force winds. Figure 8 provides two examples of USVI homes that participated in Phase II of the STEP pilot program. FEMA expanded the STEP pilot program to address the USVI’s unique, longer-term sheltering needs. Specifically, as the 2018 hurricane season arrived, FEMA was faced with the challenge of ensuring adequate sheltering options were available to USVI residents in the event that another hurricane struck the territory. The following factors contributed to FEMA’s decision to expand the STEP pilot program: Infeasibility of other sheltering programs. Alternate sheltering options were not viable in the USVI due to the unique circumstances in the territory. For example, the Transitional Sheltering Assistance program—where FEMA funds non-congregate sheltering (typically in hotels or motels) for displaced residents who cannot safely return to their homes—was not a feasible option as there was only one operating hotel in the USVI capable of sheltering disaster survivors. Further, FEMA officials told us that temporary housing units—such as manufactured housing units or recreational vehicles—could not be deployed to supplement the territory’s available housing stock due to logistical challenges, including the prohibitive costs of shipping these units to the remote territory and the limited availability of space to install them. Operation Blue Roof caused additional damage to homes. According to FEMA officials, FEMA’s decision to allow homes that had received temporary blue tarps as an emergency roofing measure through Operation Blue Roof to be eligible for the STEP pilot program led to expanding the scope of allowable work funded by the program. FEMA and USVI officials told us this change was implemented to address several issues with the blue tarps installed on homes, including the temporary nature of the tarps—which had a post-installation lifespan of only 30 days—and the need to fix the damage caused by installing the blue tarps on undamaged sections of roofs. FEMA officials stated that expanding the STEP pilot program to conduct more permanent roof repairs on these homes helped to ensure homeowners were able to safely shelter in the event of another hurricane. Shortage of construction crews. As previously discussed, FEMA and USVI officials cited the limited number of construction crews available to implement recovery work as a challenge, including for the STEP pilot program. Specifically, this challenge made it difficult for private homeowners to independently hire qualified contractors to conduct permanent repairs to their homes, according to FEMA officials. Therefore, these officials explained that using Phase II of the STEP pilot program to manage contractors in an official capacity made it more likely that necessary permanent repairs would be completed in a timely manner. Evacuation was not an option. When requested, FEMA is responsible for providing safe sheltering options following a disaster and, in the absence of feasible local sheltering options, FEMA is responsible for evacuating residents to a safe location outside the potentially affected area. However, according to FEMA documentation, developing and executing a plan to evacuate the USVI’s more than 100,000 residents in the event of another hurricane was impractical. Given these factors and the risk of another hurricane, FEMA officials determined that authorizing Phase II roof repairs or replacements of a permanent nature represented an appropriate solution to ensure eligible program participants could safely shelter in their homes. The STEP pilot program in the USVI officially ended on April 15, 2019. FEMA reported that 7,381 homes ultimately received repairs through the program. Specifically, 6,372 homes received Phase I temporary repairs and 1,631 homes received Phase II roof repairs or replacements of a permanent nature. In addition, 622 homes received both Phase I and Phase II repairs. According to FEMA officials, the agency is now conducting the close-out process for the STEP pilot program in the USVI, which includes reviewing the paperwork for each participating home to ensure all work was completed in accordance with both the home’s approved scope of work and overall programmatic requirements. In May 2019, FEMA’s Chief Counsel stated that FEMA had decided to discontinue the STEP pilot program due to significant challenges and lessons learned from prior experiences implementing the program. Specifically, FEMA stated that while FEMA had implemented the STEP pilot program within its authority pursuant to Section 403 of the Stafford Act, the agency was no longer “comfortable from a legal, policy, or pragmatic perspective” with implementing the STEP pilot program following future disasters. FEMA cited two main challenges in implementing the program in the USVI and elsewhere: (1) limiting the program’s scope to provide only minimal, emergency repairs, as intended, and (2) completing these emergency repairs in a timely manner. First, FEMA stated that in multiple iterations of the STEP pilot program—including in the USVI—FEMA officials had “succumbed to the pressure” from state and territorial leaders to expand the scope of allowable repairs under the program to conduct more extensive repairs. For example, in the USVI specifically, although expanding the program to authorize permanent roof repairs was legally supportable and represented an earnest effort to meet the territory’s needs, the expansion did “push the boundaries of appropriateness” and increased FEMA’s risk of interfering with the agency’s authority to provide assistance through other FEMA programs. Second, FEMA stated that the lengthy process for delivering the STEP pilot program—including in the USVI—undercut the program’s stated intent of providing emergency sheltering within 3 to 4 months following a disaster. For example, while FEMA authorized the program in the USVI in October 2017, initial repairs did not begin until March 2018 and eligible work was not completed until April 2019—18 months after the program’s authorization. According to FEMA, completing STEP pilot program repairs took longer than intended across most instances of the program’s implementation due to the amount of time required to develop disaster- specific program guidance, hire a large number of construction crews to undertake the repair work, obtain the necessary permissions from homeowners, and ultimately complete the repairs. Given the STEP pilot program’s protracted period of implementation, FEMA stated the agency had not been successful in ensuring that program repairs provided disaster survivors with emergency shelter in a timely manner. FEMA’s decision to discontinue the STEP pilot program following future disasters raises questions about how the agency plans to address the emergency sheltering needs of disaster survivors in the future—especially in communities that face challenges and circumstances similar to those the program was specifically designed to address. Since implementing it in 2012, FEMA used the STEP pilot program to supplement other FEMA sheltering programs and provide necessary additional capacity to help address the emergency sheltering needs of disaster-affected communities, as described below. In certain cases, the program provided assistance to more disaster survivors than other relevant FEMA programs, including the Transitional Sheltering Assistance program and the provision of temporary housing units. FEMA implemented the STEP pilot program in the following locations: Louisiana: FEMA authorized the STEP pilot program to supplement other federal programs implemented in Louisiana following severe storms and flooding in 2016. Specifically, the Transitional Sheltering Assistance program was not a viable option for most survivors, partially due to the limited availability of hotels and motels in the affected area, and FEMA ultimately used this program for approximately 4,300 households. In addition, FEMA deployed temporary housing units for approximately 4,600 households. FEMA also funded repairs through the STEP pilot program for nearly 11,000 homes. Texas: FEMA authorized the STEP pilot program to supplement other federal programs in Texas following Hurricane Harvey in 2017. Specifically, FEMA determined that implementing the STEP pilot program provided a useful option since the number of displaced survivors significantly exceeded the available capacity for sheltering survivors in local hotels and motels. FEMA used the Transitional Sheltering Assistance program for approximately 55,000 households and deployed temporary housing units for more than 3,500 households. FEMA supplemented these programs by funding repairs through the STEP pilot program for approximately 15,700 homes. Puerto Rico: FEMA authorized the STEP pilot program to address the unique emergency sheltering needs in Puerto Rico following Hurricanes Irma and Maria in 2017. Specifically, FEMA determined that approximately 80 percent of the island did not have power and would not have it restored for an extended period of time. As a result, FEMA implemented the STEP pilot program to, among other repairs, reconnect homes to a functioning electricity grid or, as necessary, fund the installation of generators if the grid could not be restored in a timely manner. Further, similar to what occurred in the USVI, FEMA authorized the STEP pilot program to repair homes that participated in Operation Blue Roof to address the roof damage caused by the installation of the blue tarps. FEMA funded repairs through the STEP pilot program for nearly 108,500 homes in Puerto Rico. In addition, due to the lack of available hotels and motels in Puerto Rico, participation in FEMA’s Transitional Sheltering Assistance program was limited to approximately 7,000 households, most of which were relocated to hotels and motels in the contiguous United States. Further, as detailed below, FEMA implemented its new Voluntary Agencies Leading and Organizing Repair program for the first time in Puerto Rico to conduct repairs to approximately 4,600 homes. North Carolina: FEMA authorized the STEP pilot program to address the particular emergency sheltering needs in North Carolina following Hurricane Florence in 2018. FEMA amended the STEP pilot program to allow both contracted construction crews and voluntary organizations to conduct the repairs, and ultimately funded repairs through the program for approximately 2,200 homes. In addition, FEMA used the Transitional Sheltering Assistance program for more than 870 households and deployed temporary housing units for approximately 650 households. Overall, FEMA authorized the STEP pilot program following 8 declared disasters since 2012 and obligated approximately $2.6 billion to fund repairs to more than 167,000 disaster survivors’ homes, according to FEMA documentation. While the program may not have provided repairs as rapidly as FEMA intended, these repairs nonetheless played a significant role in ensuring these disaster survivors could safely shelter in their homes. Since discontinuing the program, FEMA has not evaluated its options for addressing the emergency sheltering needs of disaster survivors. FEMA stated that it continues to support the use of congregate sheltering and will consider authorizing the Transitional Sheltering Assistance program, among other options, to address disaster survivors’ needs following future disasters. However, as detailed above, FEMA used the STEP pilot program for the specific purpose of providing necessary additional capacity to supplement these and other federal programs. Further, in certain cases, the STEP pilot program was used when implementing these other programs was unfeasible, such as in the USVI and New York where the particular circumstances on the ground made using the Transitional Sheltering Assistance program or deploying temporary housing units impractical. FEMA officials also told us the agency could utilize voluntary organizations to a greater extent than in the past to conduct the same types of repairs provided through the STEP pilot program. Specifically, FEMA officials stated that the Voluntary Agencies Leading and Organizing Repair program—which was implemented in Puerto Rico in response to the 2017 hurricanes and used to repair about 4,600 homes— could be used for this purpose following future disasters. However, given the program’s limited implementation, it is too early to determine the extent to which it represents a feasible solution for addressing emergency sheltering needs, or is capable of providing assistance on as large a scale as other federal programs, such as the STEP pilot program that funded repairs on more than 100,000 homes in Puerto Rico alone. Further, in North Carolina—where FEMA amended the STEP pilot program to allow both hired construction crews and voluntary organizations to conduct the repairs—the number of homes repaired by the construction crews—about 2,000—far exceeded the number of homes repaired by voluntary organizations—about 150—which raises questions about the ability of voluntary organizations to undertake the large volume of repairs necessary following disasters. Standards for Internal Control in the Federal Government states that management should identify, on a timely basis, significant changes to internal conditions that have already occurred, including changes to the entity’s programs or activities. Further, management should identify, analyze, and respond to risks related to achieving the entity’s defined objectives. FEMA has not assessed how its decision to discontinue the STEP pilot program will affect its ability to provide emergency sheltering assistance following future disasters. While FEMA officials told us they plan to use other sheltering programs when the next disaster strikes, these programs may not be sufficient in addressing the emergency sheltering needs of disaster survivors, especially in communities where implementing such programs is not feasible. FEMA officials also stated that given the agency’s decision to discontinue the STEP pilot program, conducting a broad evaluation of FEMA’s emergency sheltering programs and the agency’s options for addressing emergency sheltering needs would be useful to ensure that FEMA is prepared to respond effectively to future disasters. Conducting such an evaluation would help FEMA understand its ability to provide sheltering options and to properly plan for the provision of effective emergency sheltering assistance to disaster- affected communities. USVI. As the recipient of federal disaster funding, the USVI is responsible for providing oversight over the Public Assistance program and Hazard Mitigation Grant Program to ensure they are implemented in compliance with applicable laws and regulations, as well as FEMA policies and guidance. Following the 2017 hurricanes, the USVI took steps to address its responsibilities for receiving grant funding through these programs, including by: (1) developing administrative plans, (2) designating two territorial entities to manage the administration of disaster recovery funding, and (3) establishing the new Office of Disaster Recovery. First, as required by FEMA, the USVI developed administrative plans for the Public Assistance program and Hazard Mitigation Grant Program to ensure that subrecipients are in compliance with the conditions of these grant programs. These plans outline programmatic and project monitoring activities as well as the financial and administrative procedures for both programs. The plans require, among other things, the USVI to submit quarterly progress and financial reports on the status of projects to FEMA and describe the USVI’s specific roles and responsibilities for implementing and overseeing these two programs. Second, the Governor of the USVI designated two territorial entities to manage and oversee the implementation of recovery programs—the Virgin Islands Territorial Emergency Management Agency as the programmatic manager and the governor’s authorized representative as the grant administrator for all federal recovery funding in the USVI. Among other responsibilities, these entities are responsible for ensuring that Public Assistance program and Hazard Mitigation Grant Program participants are in compliance with all programmatic and administrative requirements. For example, the Virgin Islands Territorial Emergency Management Agency is responsible for, among other things, preparing and submitting quarterly progress and financial reports to FEMA for certain Public Assistance program projects and for all Hazard Mitigation Grant Program projects. In addition, the governor’s authorized representative is responsible for ensuring the territory’s compliance with all requirements outlined in the FEMA-approved administrative plans. Third, in February 2019, the USVI’s new Office of Disaster Recovery assumed responsibility for overseeing the Public Assistance program and Hazard Mitigation Grant Program, including tracking and reporting on the progress of individual projects and overseeing the submission of reimbursement requests for completed work. The office is also responsible for monitoring and publicly reporting the status of federal recovery funding at http://www.usviodr.com/. In addition, according to the office’s Director, her team also meets with FEMA officials on a bi- weekly basis to discuss recovery activities more generally and raise any potential issues for discussion, such as challenges with submitting quarterly progress and financial reports to FEMA in a timely manner. FEMA. FEMA officials at the USVI, regional, and headquarters level are responsible for overseeing the USVI’s implementation of federal recovery programs. Specifically, once FEMA obligates grant funding for a project, FEMA officials on the ground in the USVI are responsible for the day-to- day monitoring of individual projects using a variety of tools. For example, FEMA officials stated they use the information included in quarterly progress and financial reports to help ensure that subrecipients are in compliance with applicable federal requirements and that potential problems are identified and addressed in a timely manner. Further, FEMA officials are to conduct quarterly meetings with USVI officials to ensure regular communication and coordination and to discuss program implementation, raise potential challenges, and identify solutions. In addition to monitoring ongoing projects, FEMA officials in the USVI are responsible for managing the process for closing out Public Assistance program and Hazard Mitigation Grant Program projects that have been completed. To facilitate this close-out process, the USVI is to compile all required documentation for an individual project and submit this paperwork to FEMA. FEMA is to review the documentation to ensure that all work has been completed in accordance with the project’s scope of work as well as relevant laws, federal regulations, and program requirements. FEMA Region II officials and FEMA officials in the Grant Programs Directorate in headquarters also provide higher-level oversight of Public Assistance program and Hazard Mitigation Grant Program implementation. For example, FEMA Region II officials stated they analyze the USVI’s quarterly financial reports to identify patterns that may indicate financial challenges, such as irregularities in the amount of funding the USVI has drawn down for projects or an excess or lack of reimbursements to subrecipients for completed work. At FEMA headquarters, officials in the Grant Programs Directorate stated they assess the financial condition of projects annually to identify potential challenges in administering grant funding and may enhance monitoring efforts as needed. These officials stated they also review the timeliness, completeness, and accuracy of information submitted in quarterly reports to help monitor project milestones and identify potential challenges that require FEMA’s attention. FEMA has issued numerous documents that provide useful information and guidance for implementing and monitoring the Public Assistance program and the Hazard Mitigation Grant Program, such as program and policy documents, fact sheets, job aids, and operational manuals. For example, one document for monitoring the Public Assistance program is the Public Assistance Program Management and Grant Closeout Standard Operating Procedure. This standard operating procedures document provides FEMA officials across all disasters nationwide with a common understanding of the expectations and requirements for managing projects. It also clearly and concisely outlines the roles and responsibilities, requirements, key tasks and milestones, and performance measures associated with monitoring and closing out Public Assistance program projects. However, FEMA has not developed a similar consolidated standard operating procedures document to provide a clear and concise roadmap for monitoring and closing out projects under the Hazard Mitigation Grant Program. In July 2019, FEMA officials in headquarters and FEMA Region II told us that guidance on key policies and procedures for managing FEMA’s varied hazard mitigation efforts can be found across multiple sources. However, these officials stated that FEMA has not consolidated this guidance into a single document for FEMA-wide use that focuses on the oversight of Hazard Mitigation Grant Program projects specifically. Further, FEMA’s existing guidance documents do not provide a concise roadmap that outlines roles and responsibilities, key tasks and milestones, and performance measures for FEMA officials to use when monitoring and closing out individual program projects for any given disaster. For instance, the Hazard Mitigation Assistance Guidance includes information on many topics, such as program eligibility requirements; roles and responsibilities for recipient and subrecipient personnel; and oversight requirements spanning three separate FEMA mitigation programs. Likewise, FEMA’s 250-page Hazard Mitigation Field Operations Guide includes operating procedures, descriptions of major tasks for mitigation positions, and job aids, including samples and templates of tools to help hazard mitigation staff implement defined tasks. While these and other FEMA documentation collectively provide important information regarding the agency’s broader hazard mitigation efforts, they are not focused specifically on the Hazard Mitigation Grant Program, or the detailed processes FEMA officials on the ground should use to conduct the day-to-day monitoring of individual projects. Further, these documents do not detail the use of performance measures in effectively monitoring Hazard Mitigation Grant Program projects— information that is included in FEMA’s standard operating procedures document for the Public Assistance program. When we asked FEMA officials responsible for implementing the program in the USVI why the agency had not developed a consolidated standard operating procedures document specific to the Hazard Mitigation Grant Program, they stated that developing this document had not been a priority for the agency—which could be due to the relatively small size of this program compared to larger and more complex recovery programs, such as the Public Assistance program. Although FEMA obligates more funding through the Public Assistance program, the Hazard Mitigation Grant Program nonetheless plays a critical role in ensuring that disaster- affected communities can undertake mitigation measures specifically designed to enhance the resilience of their infrastructure during future disasters. Standards for Internal Control in the Federal Government states that management should document in policies and procedures each unit’s responsibility for an operational process’s objectives in the appropriate level of detail to allow management to effectively monitor the control activity. In addition, these standards state that management should define objectives in specific and measurable terms so they are understood at all levels of the organization. This includes clearly defining what is to be achieved, who is to achieve it, how it will be achieved, and the timeframes for achievement. Further, leading practices identified in the Program Management Institute’s Standard for Program Management call for agencies to develop a program roadmap outlining information on the program’s intended direction and providing a set of documented success criteria for each key milestone and decision point. This roadmap can be a valuable tool for managing the execution of the program and for assessing the program’s progress toward achieving its goals. FEMA’s consolidated standard operating procedures document for the Public Assistance program provides such a roadmap, stating that FEMA’s main goal is to provide effective assistance and excellent customer service necessary to assist disaster-affected communities to recover while also ensuring the responsible stewardship of public funds. FEMA does have guidance on key policies and procedures for managing its hazard mitigation efforts, including through the Hazard Mitigation Grant Program. However, FEMA could further strengthen its existing guidance by consolidating this information into a single document for FEMA-wide use, similar to the one FEMA uses for the Public Assistance program. FEMA officials told us that having a detailed standard operating procedures document for the Hazard Mitigation Grant Program that clearly and concisely outlined roles and responsibilities, key objectives and tasks, and milestones for conducting monitoring and close-out activities would be helpful in effectively overseeing program projects. In addition to FEMA officials, the governor’s authorized representative in the USVI also told us that such a document would help to ensure that both FEMA and USVI officials are following the necessary procedures and guidance when conducting program management and close-out activities. Assessing the need for such a consolidated standard operating procedures document for the Hazard Mitigation Grant Program would help FEMA determine whether existing guidance should be strengthened to ensure that agency officials across all disasters are using a consistent approach in carrying out their responsibilities under the program. As of June 30, 2019, FEMA had obligated more than $1.9 billion in grant funding through the Public Assistance program and Hazard Mitigation Grant Program to help the USVI recover from the catastrophic 2017 hurricane season. As part of the Public Assistance program, FEMA authorized the STEP pilot program in the USVI—and in other locations— to supplement other FEMA programs and provide necessary additional capacity to help address the emergency sheltering needs of disaster survivors. While FEMA decided to discontinue the STEP pilot program, the agency has not evaluated how this decision will affect its ability to provide emergency sheltering assistance in the future. FEMA has a responsibility to provide assistance, when requested, to address the emergency sheltering needs of disaster survivors. Given that FEMA will no longer use the STEP pilot program, taking steps to evaluate its options for addressing these needs will help FEMA to assess its capacity for providing effective emergency sheltering assistance in the future and to properly plan for when the next disaster inevitably strikes. In addition, FEMA has issued numerous policy documents, guides, and other useful documents to assist FEMA officials to effectively monitor and oversee Hazard Mitigation Grant Program projects. However, FEMA has not developed a consolidated standard operating procedures document specific to the Hazard Mitigation Grant Program that provides a clear and concise roadmap for FEMA officials’ use in monitoring individual projects. Assessing the need for a consolidated roadmap for agency-wide use would help FEMA determine whether existing guidance for effectively monitoring Hazard Mitigation Grant Program projects should be strengthened. We are making the following two recommendations to FEMA: The FEMA Administrator should evaluate the agency’s options for providing future emergency sheltering assistance. (Recommendation 1) The FEMA Administrator should assess the need for an agency-wide consolidated standard operating procedures document for the Hazard Mitigation Grant Program that provides detailed information on the roles and responsibilities, requirements, and key tasks and milestones for monitoring and closing out program projects. (Recommendation 2) We provided a draft of this product to DHS and the USVI government for review and comment. DHS provided written comments, which are reprinted in appendix IV and summarized below. DHS and the USVI government provided technical comments, which we incorporated as appropriate. DHS concurred with both our recommendations and described the actions it plans to take in response. With regard to our first recommendation, DHS stated that it will evaluate FEMA’s options for providing emergency sheltering assistance through its Individual Assistance Division and provide any recommendations for action, as appropriate, to FEMA’s Assistant Administrator for Recovery. DHS anticipates completing this evaluation by February 2020. This action, if fully implemented, should address the intent of the recommendation. With regard to our second recommendation, DHS stated that FEMA will assess the need for an agency-wide consolidated standard operating procedures document for the Hazard Mitigation Grant Program and, if deemed necessary, FEMA will develop this document. DHS anticipates this effort will be completed by August 2020. This action, if fully implemented, should address the intent of the recommendation. In the meantime, DHS noted that FEMA will consider updating its website to include a single portal providing access to all existing guidance documents relevant to monitoring and overseeing Hazard Mitigation Grant Program projects. We will monitor DHS’s and FEMA’s efforts to address these two recommendations. We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the FEMA Administrator, the USVI government, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you and your staff have any questions, please contact me at (202) 512- 8777 or curriec@gao.gov. GAO staff who made key contributions to this report are listed in appendix V. The Federal Emergency Management Agency (FEMA) obligated more than $1.8 billion in Public Assistance grant funding for 618 projects across the U.S. Virgin Islands (USVI) as of June 30, 2019. Specifically, FEMA obligated more than $1.1 billion for emergency work projects (categories A and B), about $588.5 million for permanent work projects (categories C through G), and about $141.2 million for management costs (category Z). As of that date, the USVI expended nearly $1.1 billion—about 59 percent of total Public Assistance obligations to the USVI—to reimburse subrecipients for completed work. Of this nearly $1.1 billion, the USVI expended about $857.5 million (78 percent) for emergency work projects, $211.3 million (19 percent) for permanent work projects, and $29.9 million (3 percent) for management costs. The majority of FEMA’s obligations and the funding the USVI expended as of June 30, 2019, are for emergency work because these projects began soon after the disasters struck and focused on debris removal and providing assistance to address immediate threats to life and property. In contrast, permanent work projects take time to identify, develop, and ultimately complete as they represent the longer-term repair and restoration of public infrastructure. While the data below represent the status of Public Assistance funding as of June 30, 2019, the amount of grant funding FEMA obligates and the USVI expends will likely increase over time as additional projects are finalized and approved. Emergency work. Of the more than $1.8 billion FEMA obligated as of June 30, 2019, more than 1.1 billion (61 percent) was obligated for 410 emergency work projects in Public Assistance program categories A and B. Category A: Debris removal. FEMA obligated about $139.9 million for 88 projects focused on debris removal activities across the territory. For example, FEMA obligated $81.8 million to the USVI Water and Power Authority for territory-wide debris removal efforts (see fig. 9). Of the $139.9 million FEMA obligated for debris removal, the USVI expended about $76.9 million (55 percent) as of June 30, 2019. Category B: Emergency protective measures. FEMA obligated about $985.6 million for 322 projects focused on emergency measures. For example, FEMA obligated about $278.2 million for the Sheltering and Temporary Essential Power (STEP) pilot program as of June 30, 2019, to fund certain types of temporary repairs to private homes. In addition, FEMA obligated approximately $111.9 million for the purchase and installation of modular units to be used as temporary classrooms and other facilities while permanent school buildings are repaired or replaced (see fig. 10). Of the $985.6 million FEMA obligated for emergency protective measures, the USVI expended about $780.7 million (79 percent) as of June 30, 2019. Permanent work. Of the more than $1.8 billion in Public Assistance grant funding FEMA obligated as of June 30, 2019, about $588.5 million (32 percent) was obligated for 200 permanent work projects across categories C through G. These permanent work projects included more than $383.1 million for cost-effective hazard mitigation measures to reduce the future risk of disaster damage to infrastructure. Category C: Roads and bridges. FEMA obligated about $5.9 million for 40 projects focused on repairing roads and bridges in the territory, 9 of which included hazard mitigation measures totaling about $1.6 million. For example, FEMA obligated about $233,000 for one project to repair a road on St. Croix damaged by floodwaters. This project included approximately $61,000 for hazard mitigation measures to reduce the likelihood of erosion during future flooding events. Of the $5.9 million FEMA obligated for category C, the USVI expended about $86,000 (1.5 percent) as of June 30, 2019. Category D: Water control facilities. As of June 30, 2019, FEMA did not have any projects in this category. According to FEMA officials, the USVI does not have water control infrastructure—such as berms or levees—that would fall under category D. Category E: Buildings and equipment. FEMA obligated about $68.3 million for 101 projects focused on repairing damaged structures in the territory, 49 of which included hazard mitigation measures totaling about $3.1 million. For example, FEMA obligated about $59.7 million for one project to replace 5 heavily damaged buildings in a public housing facility in St. Thomas (see fig. 11). While FEMA obligated this project through the standard Public Assistance program, FEMA and the USVI plan to work to develop a fixed-cost estimate with the intention of transitioning this project to the alternative procedures program, according to FEMA documentation. Further, this documentation states that hazard mitigation measures will be incorporated into the new structures by implementing internationally adopted building codes and standards for wall and window replacements. Of the $68.3 million FEMA obligated for Category E, the USVI expended about $533,000 (0.8 percent) as of June 30, 2019. Category F: Utilities. Of the $588.5 million FEMA obligated for permanent work projects, $505.6 million (86 percent) was obligated for 23 projects focused on repairing utilities, 13 of which included hazard mitigation measures totaling about $378.2 million. Specifically, FEMA obligated $481.8 million—or 95 percent of the $505.6 million—through the standard Public Assistance program for projects focused on territory-wide permanent electrical distribution system repairs. This includes replacing damaged wooden utility poles with more resilient composite fiberglass poles that can withstand 200 mile per hour winds as well as power transmission lines and transformers (see fig. 12). Of the $505.6 million FEMA obligated for category F, the USVI expended about $210.4 million (42 percent) as of June 30, 2019. Category G: Parks, recreational, and other facilities. As of June 30, 2019, FEMA obligated about $8.8 million for 36 projects focused on repairing parks, playgrounds, and other facilities, 5 of which included hazard mitigation measures totaling about $214,000. For example, FEMA obligated about $1.5 million in March 2019 for two projects to repair the USVI’s Tsunami Early Warning System, which comprises a network of warning stations that alert residents of a potential tsunami event (see fig. 13). As of September 2019, these were the only projects FEMA had obligated under the Public Assistance alternative procedures program in the territory, according to FEMA officials. These projects included about $185,000 for hazard mitigation measures to replace wooden poles with higher-rated steel poles that are able to withstand high winds and impacts from flying debris during a storm. Of the $8.8 million FEMA obligated for category G, the USVI expended about $246,000 (3 percent) as of June 30, 2019. As of June 30, 2019, the Federal Emergency Management Agency (FEMA) obligated about $60.6 million for 22 Hazard Mitigation Grant Program projects in the U.S. Virgin Islands (USVI) and the USVI expended about $1.7 million (3 percent) across 5 projects. Unlike Public Assistance program projects that, in many cases, are focused on rapidly providing emergency services or repairing critical disaster-damaged infrastructure and systems, Hazard Mitigation Grant Program projects are designed to fund a variety of measures to increase the longer-term resilience of the USVI’s infrastructure during future disasters. Information on selected Hazard Mitigation Grant Program projects in the USVI that received obligations as of June 30, 2019, is detailed below. Virgin Islands Territorial Emergency Management Agency and Bureau of Information Technology Emergency Operations Center and Safe Room Retrofit. FEMA obligated about $22.5 million to fund the retrofit of the USVI Territorial Emergency Management Agency’s Emergency Operations Center. The new facility will serve as the headquarters for both the USVI Territorial Emergency Management Agency and the USVI Bureau of Information Technology and house a 911 Emergency Call Center. According to FEMA documentation, the facility will include a safe room to allow emergency personnel to shelter in place during disasters and will contain sufficient space to house FEMA and other federal personnel, as necessary. Further, the facility will include a hardened communications system to ensure emergency responders are able to effectively communicate during emergency events, among other improvements. The Comprehensive Territorial Hazard Mitigation and Resilience Plan Project. FEMA obligated nearly $5.0 million to fund the development of an in-depth, comprehensive hazard mitigation and resilience plan for territory-wide use. FEMA officials stated that unlike in the contiguous United States, the USVI does not have any entities responsible for formally developing similar plans to guide operations and mitigation activities across various sectors, such as protecting the potable water supply and assessing economically feasible options for development. As a result, FEMA officials told us that this project represents an important effort to develop a holistic, territory-wide hazard mitigation plan that would cover all relevant sectors. The Spring Gut Watershed Green Space Acquisition and Stormwater Management Project. FEMA obligated nearly $1.0 million to fund the first phase of a $2.0 million project to purchase 50 acres of undeveloped land and develop storm water retention measures—such as berms and rock linings—to reduce downstream flooding and the associated damages to roads, homes, and infrastructure (see fig. 14). FEMA officials told us that the first phase of the project included an environmental and historical preservation review of the target locations to confirm program eligibility before actual construction activities can begin. Fortuna/Bordeaux Fire Station Retrofit. FEMA obligated more than $470,000 to fund the first phase of a nearly $5.0 million project to retrofit a fire station in St. Thomas. Specifically, the project will upgrade the facility’s structure to applicable codes and standards to mitigate the risks posed by hurricane-force winds, including the dangers posed by flying debris. Further, the retrofit will include the installation of a steel-reinforced concrete safe room and a back-up emergency power generator to ensure the safety and protection of emergency personnel and the continuity of emergency response activities during a disaster, according to FEMA documentation. The Governor Juan F. Luis Hospital and Medical Center (JFL hospital) in St. Croix provides an illustrative example of the processes and challenges associated with developing and implementing Public Assistance program projects in the U.S. Virgin Islands (USVI). In September 2017, Hurricane Maria’s strong winds and torrential rains caused severe damage to the facility’s roof; heating, ventilation, and air conditioning system; and electrical, water, and sewage systems, according to Federal Emergency Management Agency (FEMA) documentation. Further, the infusion of water—both during and after the storm—saturated the interior of the hospital, destroyed medical equipment and hospital furnishings, and facilitated the growth of hazardous mold. Figure 15 details selected hurricane damage to the facility. Due to the extensive damage, the JFL hospital has been operating at reduced capacity since the hurricane and certain functions have been relocated to undamaged areas, according to FEMA documentation. This documentation states that while the hospital continues to provide limited medical services to St. Croix residents, it is no longer capable of providing critical care services. Since alternate options were either limited or unavailable on the island following the storms, St. Croix residents in need of life sustaining medical treatments such as chemotherapy infusions or who are experiencing life-threatening health events such as cardiac failure or trauma must be transported out of the territory to receive life- saving care, according to FEMA documentation. Following Hurricane Maria, FEMA obligated grant funding for several Public Assistance emergency work projects to help keep the JFL hospital functioning and capable of providing limited medical services to St. Croix residents. For example, in August 2018, FEMA obligated $119,000 in grant funding to reimburse the JFL hospital for the use of an emergency backup generator through Public Assistance program category B, which provides funding for emergency protective measures. According to JFL personnel, this funding covered the cost of using the backup generator until the facility’s primary electrical system could be restored. In another example, FEMA obligated about $2.4 million in August 2018—also through Public Assistance program category B—to fund the rental of mobile dialysis trailers (see fig. 16). According to JFL personnel, these trailers were acquired to replace all 14 of the facility’s dialysis units that were destroyed during the storm. In addition to Public Assistance projects focused on maintaining the existing facility, FEMA obligated $43.2 million in January 2018 to fund the purchase and installation of modular units to serve as a temporary medical facility through category B emergency work. According to FEMA documentation, this temporary facility is intended to provide critical medical services to St. Croix residents—including an emergency room, pediatric care, a labor and delivery ward, and an intensive care unit, among other services—until a permanent facility is completed (see fig. 17). JFL personnel told us that completing this project is a key priority as it will enable them to transition all medical services from the main facility, which continues to deteriorate over time. However, they stated that implementing this project has been challenging. For example, JFL personnel told us that when former JFL administrators were developing the project, they incorporated the cost of procuring and installing the modular units, but omitted the costs associated with acquiring, installing, and certifying new medical equipment for use in the interim facility. These personnel stated that these costs should have been incorporated into the original project paperwork and clarified that the acquisition and certification of new equipment was critical in ensuring the hospital’s provision of medical services would not be disrupted. Specifically, they explained that relocating the facility’s existing medical equipment was not a feasible option as it would result in an unacceptable lapse in medical services during the time-consuming process of deconstructing, transferring, reinstalling, and recertifying this equipment in the modular facility. JFL personnel stated they worked closely with FEMA officials and contractor personnel to update the project’s paperwork and request additional program funding for this new medical equipment. In August 2019, FEMA approved the updated paperwork and obligated additional funding for this project, according to FEMA documentation. JFL personnel also stated they have limited capacity to effectively manage and oversee the construction of the temporary facility due to competing responsibilities. In addition to managing this project, they stated they were occupied with the continuous maintenance challenges associated with keeping the deteriorating main facility functioning while also working to develop options for a permanent facility through the Public Assistance alternative procedures program, as discussed below. They told us FEMA officials and contractor personnel had been helpful in providing assistance, but stated they would benefit from a larger hospital management team that could focus specifically on planning and implementing the facility’s numerous recovery efforts. As of July 2019, JFL personnel stated their aim is to officially open the temporary facility in the spring of 2020. In conjunction with the temporary facility’s construction, FEMA officials and JFL personnel are working to develop a permanent work project under the Public Assistance alternative procedures program to replace the damaged hospital. Further, in providing a critical service, the JFL hospital is eligible to use the new flexibilities provided by the Bipartisan Budget Act of 2018. This Act allows FEMA—when using the alternative procedures—to fund the repair or full replacement of the hospital to accepted industry standards regardless of any pre-disaster damage or wear and tear the facility may have sustained prior to the 2017 hurricanes. FEMA officials and JFL personnel stated the Act therefore provides a valuable opportunity to restore the facility to a better condition than it was in prior to the storms. However, JFL personnel told us that pursuing this permanent work project included challenges. For example, JFL personnel told us that maintaining the damaged facility while FEMA determined whether the hospital was eligible for repairs to the existing structure or a complete replacement under the Bipartisan Budget Act was challenging. Specifically, they explained that while FEMA worked to finalize this determination, management was in the difficult position of deciding where and how to invest its finite resources to keep the constantly deteriorating facility functioning. For example, these personnel explained that if FEMA determined that the facility is ineligible for replacement under the Act, they would immediately invest money into the existing facility to address critical components that require urgent attention, such as the water and wastewater systems. In contrast, if FEMA determined that the facility is indeed eligible for replacement, management would strategically invest the minimum amount of resources required to keep the facility functioning with the full knowledge that it would eventually be demolished. Figure 18 details selected examples of temporary fixes JFL personnel implemented to keep the facility functioning. In May 2019, FEMA officially determined that the JFL hospital was eligible under the Bipartisan Budget Act of 2018 for a complete replacement through the Public Assistance alternative procedures program. JFL personnel told us they are working with FEMA officials and medical industry experts to ensure that they take advantage of the flexibilities provided by the Bipartisan Budget Act when developing the project. As of July 2019, these personnel explained they are in the early stages of working with FEMA officials and territorial stakeholders to assess options for the replacement facility and are designing a strategy to ensure the future hospital is able to sufficiently address the healthcare needs of USVI residents. Chris Currie, (202) 512-8777 or curriec@gao.gov. In addition to the contact named above, Joel Aldape (Assistant Director), Bryan Bourgault (Analyst in Charge), Aaron Gluck, Eric Hauswirth, Brian Lipman, Amanda Miller, Heidi Nielson, and Kevin Reeves made key contributions to this report.", "summary": "In September 2017, two major hurricanes—Irma and Maria—struck the USVI, causing billions of dollars in damage. FEMA is the lead federal agency responsible for assisting the USVI to recover from natural disasters. FEMA administers the Public Assistance program and Hazard Mitigation Grant Program in partnership with the USVI government, providing grant funding for response and recovery activities, including life-saving emergency protective measures, the repair or replacement of public infrastructure, and measures to increase the territory's resilience during future disasters. GAO was asked to review the federal government's response and recovery efforts in the USVI. This report examines (1) the status of Public Assistance program and Hazard Mitigation Grant Program funding and challenges, if any, with implementation, (2) the STEP pilot program, and (3) the oversight of these programs. GAO reviewed documentation and data on the Public Assistance program and Hazard Mitigation Grant Program in the USVI as of June 30, 2019. GAO interviewed FEMA and USVI officials regarding the status of recovery efforts and associated challenges, and conducted site visits to the USVI islands of St. Croix, St. Thomas, and St. John. As of June 30, 2019, FEMA obligated more than $1.9 billion in grant funding for 640 projects in the U.S. Virgin Islands (USVI) through the Public Assistance program and Hazard Mitigation Grant Program in response to the 2017 hurricanes. However, the limited availability of local USVI personnel to staff key recovery positions and the territory's difficult fiscal situation presented challenges in implementing these programs. Further, FEMA and USVI officials stated they faced challenges with implementing the Public Assistance alternative procedures program, which provides the USVI with flexibility in determining when and how to fund projects. Specifically, these officials stated that developing accurate fixed-cost estimates and using new flexibilities authorized by law delayed longer-term recovery projects. USVI officials told GAO they plan to take a cautious approach when deciding whether to pursue projects using the alternative procedures. FEMA expanded its Sheltering and Temporary Essential Power (STEP) pilot program in the USVI to address the lack of other sheltering options for survivors, such as hotels. The program aimed to provide minimal, temporary repairs to damaged homes to quickly make them habitable. In May 2019, FEMA decided it would not use the STEP pilot program in the future since it did not provide assistance as rapidly as intended. Historically, the program was used to address survivors' emergency sheltering needs. However, since ending it, FEMA has not evaluated options for providing future emergency sheltering assistance. Doing so could help FEMA plan for when the next disaster inevitably strikes. The USVI and FEMA established structures for overseeing recovery efforts. For example, the USVI established a new office to oversee federal recovery programs and FEMA has processes in place to oversee recovery projects at the local, regional, and headquarters levels. However, GAO found that FEMA does not have a consolidated standard operating procedures document for monitoring Hazard Mitigation Grant Program projects. Assessing the need for a consolidated document would help FEMA determine whether its existing guidance should be strengthened. GAO recommends that FEMA (1) evaluate its options for providing emergency sheltering and (2) assess the need for a consolidated standard operating procedures document for the Hazard Mitigation Grant Program. The Department of Homeland Security concurred with these recommendations.", "document_type": "gao"}
{"report": "This section describes (1) U.S. climate risks and related impacts, (2) enhancing climate resilience using a risk management strategy, (3) GAO’s Disaster Resilience Framework, and (4) benefits and costs of climate resilience projects. Climate change poses risks to many U.S. environmental and economic systems, according to USGCRP’s Fourth National Climate Assessment. For example, high temperature extremes, heavy precipitation events, high-tide flooding events along the U.S. coastline, ocean acidification and warming, and forest fires in the western United States and Alaska have been observed and are all projected to continue to increase. In contrast, land and sea ice cover, snowpack, and surface soil moisture have been declining and are expected to continue to decline in the coming decades. Climate change is also altering the characteristics of many extreme weather and climate-related events, according to the Fourth National Climate Assessment. Some of these events have already become more frequent, intense, widespread, or of longer duration, and many are expected to continue to increase or worsen. Furthermore, according to the assessment, many places are subject to more than one climate-related impact. Examples include extreme rainfall combined with coastal flooding, or drought coupled with extreme heat. The compounding effects of these impacts result in increased risks to people, infrastructure, and interconnected economic sectors. According to the Fourth National Climate Assessment, without significant reductions in global greenhouse gas emissions and regional efforts to pursue climate resilience, climate change is expected to cause substantial losses to infrastructure and property and impede the rate of economic growth over this century. The potential for losses in some economic sectors could reach hundreds of billions of dollars per year by the end of this century, according to the assessment. Future climate risks are subject to several sources of uncertainty, as identified by USGCRP’s Fourth National Climate Assessment. According to the assessment, climate scientists find varying ranges of uncertainty in many areas, including observations of climate variables, the analysis and interpretation of those measurements, the development of new observational instruments, and the use of computer-based models of the processes governing Earth’s climate system. According to the assessment, the largest uncertainty in projecting future climate risks is the level of greenhouse gas emissions going forward, because the level of emissions depends on economic, political, and demographic factors that can be difficult to predict with confidence far into the future. According to the Fourth National Climate Assessment, enhancing climate resilience entails a continuing risk management process through which individuals and organizations become aware of and assess risks and vulnerabilities from climate and other drivers of change, take actions to reduce those risks, and learn over time. In December 2016, we reported on a risk management strategy that may help guide federal climate resilience efforts. Enterprise risk management can help federal agencies identify, assess, and manage risks, such as preparing for and responding to natural disasters. In our report, we identified six essential elements of enterprise risk management: (1) aligning the enterprise risk management process to goals and objectives, (2) identifying risks, (3) assessing risk, (4) selecting a risk response based on risk appetite, (5) monitoring risks to see if responses are successful, and (6) communicating and reporting on risks. For example, we reported that assessing risks involves considering both the likelihood of the risk and the impact of the risk on the mission to help prioritize risk response. We also reported that selecting a risk treatment response involves leaders reviewing the prioritized list of risks and selecting the most appropriate treatment strategy to manage the risk. In October 2019, we issued the Disaster Resilience Framework to serve as a guide for analysis of federal action to facilitate and promote resilience to natural disasters. The principles in this framework can help identify opportunities to enhance federal efforts to promote disaster resilience, including building resilience to climate change. According to the framework, strategic resilience goals integrated across relevant national strategies can help decision makers work toward a common vision and help ensure focus on a wide variety of opportunities to reduce disaster risk. Federal efforts can focus attention on disaster risk reduction by creating resilience goals in all relevant national strategies and linking those goals to an overarching strategic vision. Federal efforts can also facilitate coordination and promote governance approaches that mitigate fragmentation by requiring or funding mechanisms to enhance the continuity of different efforts across jurisdictions. In addition, because much of the nation’s infrastructure is not owned and operated by the federal government, many resilience-related decisions ultimately are made by nonfederal actors, such as the states, and those decision makers face competing priorities. Incentives—in the form of federal regulatory requirements or as conditions of federal grant programs and cooperative agreements—can help promote investment in disaster risk reduction. As shown in figure 1, the framework is organized around three broad overlapping principles and a series of questions to guide analysis that can help users consider opportunities to enhance federal efforts to promote disaster resilience. Each of the principles includes more specific sets of actions that those who oversee or manage federal efforts can consider when analyzing opportunities to enhance national disaster resilience. For example, according to the framework, bringing together disparate agency missions and resources that support disaster risk reduction can help to build a national culture of resilience. Accordingly, federal efforts can (1) facilitate coordination across programs, (2) facilitate the combination of federal funding streams, and (3) leverage the expertise of nonfederal partners. Information on the benefits and costs of climate resilience projects suggests that such projects can convey benefits, such as protecting life and property from climate hazards, according to the Fourth National Climate Assessment and other reports we reviewed. According to the Fourth National Climate Assessment, information on benefits is lacking in many sectors, though some information exists on the benefits and costs of resilience efforts in certain sectors, such as resilience efforts in coastal areas, resilience efforts designed to protect against riverine flooding (i.e., flooding that occurs when river flows exceed the capacity of the river channel), and resilience efforts related to agriculture at the farm level. According to this assessment, some of the actions in these sectors, at least in some locations, appear to have large benefit-cost ratios—both in addressing current variability and in preparing for future change. However, benefits may not exceed costs in some instances. According to the Fourth National Climate Assessment, more research is needed to comprehensively assess the benefits of specific strategies that individuals and organizations are considering. Similarly, several other reports we reviewed also suggest that projects can convey benefits such as protecting life and property from climate hazards. For example, a 2018 interim report by the National Institute of Building Sciences estimated that benefits to society (i.e., homeowners and communities) would exceed costs for several types of resilience projects by protecting lives and property and preventing other losses, though precise benefits are uncertain. Specifically, this interim report examined a sample of hazard mitigation grants awarded by FEMA, the Economic Development Administration, and the Department of Housing and Urban Development (HUD) from 1993 through 2016 to address various hazards. These hazards included fires in the wildland-urban interface (i.e., fires in areas where homes are built near or among lands prone to wildland fire), hurricane- and tornado-force winds, and riverine floods. According to the interim report, for every grant dollar the federal government spent across the projects examined in the report, over time, society is estimated to accrue benefits amounting to the following: About $3 on average from projects addressing the effects of fire in the wildland-urban interface, with most benefits (approximately 70 percent) coming from the protection of property (i.e., avoiding property losses). About $5 on average from projects to address hurricane- and tornado- force winds, with most benefits (approximately 90 percent) coming from the protection of lives. This includes avoiding deaths, nonfatal injuries, and cases of post-traumatic stress. About $7 on average from projects that buy out buildings prone to riverine flooding, with most benefits (approximately 65 percent) coming from the protection of property. The interim report also projected that society could accrue benefits amounting to about $11 on average for every dollar invested in designing new buildings to meet the 2018 International Building Code and the 2018 International Residential Code—the model building codes developed by the International Code Council—with most benefits (about 45 percent) coming from the protection of property. The interim report has been cited by the Congressional Budget Office, in congressional hearings, and in other arenas to describe the benefits of investing in resilience. However, the benefit-cost ratios provided in the interim report are based on a relatively narrow set of disaster-loss data, and the report is not comprehensive. In addition to conveying climate resilience benefits, such as protecting lives and property, climate resilience projects can also convey co- benefits—benefits beyond the primary protective function of resilience projects—according to the Fourth National Climate Assessment and several reports we reviewed. For example, according to a report by the National Academies, restoring coastal wetlands—a type of nature-based resilience project—may reduce an area’s vulnerability to coastal storms but could also provide co-benefits such as increasing biodiversity by creating new breeding grounds for fish and improving recreation and tourism amenities, thereby expanding the total potential benefits of a project. USGCRP officials we interviewed also told us that projects can convey a broad range of other co-benefits, including improvements in economic opportunity, human health, equity, and national security. However, according to the Fourth National Climate Assessment, quantifying these co-benefits can be difficult because different people value benefits differently. Several factors can influence the likelihood that the benefits from resilience projects exceed the cost of implementing and maintaining the projects. For example, benefits from climate resilience projects implemented in high-risk locations, such as areas more exposed to hurricanes, are likely to be higher and therefore exceed project costs than projects implemented in other, lower-risk areas, according to one report we reviewed. Similarly, projects that protect high-value assets may also be more likely to have benefits that exceed costs, according to this report. Several factors that affect the extent to which project benefits exceed costs remain uncertain, according to several reports. For example, according to the Fourth National Climate Assessment, benefit–cost ratios can have large uncertainties associated with estimates of costs, the projection of benefits, and the economic valuation of benefits. Furthermore, according to the assessment, the benefits and costs of resilience projects are larger in scenarios with high emissions, but the level of future emissions remains uncertain. Individual federal agencies have provided ad hoc funding for projects that may convey some climate resilience benefits, but our past work demonstrates an absence of government-wide strategic planning for climate change, and the federal government has not implemented key recommendations to improve strategic planning for climate resilience. In addition, the federal government does not have a strategic federal approach for investing in the highest priority climate resilience projects that includes periodically identifying and prioritizing projects as supported by enterprise risk management practices and our Disaster Resilience Framework. Federal Mainstreaming Efforts Some agencies have made efforts to manage climate change risk within existing programs and operations—a concept known as mainstreaming—and these efforts may convey climate resilience benefits. For example, an agency planning to build a seawall to protect a coastal facility might build it higher to account for rising sea level projections. Alternatively, the U.S. military may consider climate change as part of existing construction plans on coastal installations by, for example, raising a building to include a “sacrificial” first floor and protecting critical assets—such as computer servers—from potential flooding by locating them on the building’s higher floors. The agency may use the sacrificial floor for parking. According to the U.S. Global Change Research Program’s Fourth National Climate Assessment, a significant portion of climate risk can be addressed by mainstreaming, which can provide many climate resilience benefits. However, according to the assessment, the practice may prove insufficient to address the full range of climate risks. Additional, strategic federal investments in large-scale projects—such as those discussed in our report—may also be needed to manage some of the nation’s most significant climate risks, since climate change cuts across agency missions and poses fiscal exposures larger than any one agency can manage. aim to reduce flooding and storm damage. These and other projects have the potential to convey climate resilience benefits by protecting communities from damage from flooding, storms, and other extreme weather events that may be exacerbated by climate change. The Corps of Engineers’ policy is to integrate climate change preparedness and resilience in all activities—a concept known as mainstreaming. However, the Corps’ civil works program balances several diverse missions related to navigation, ecosystems management, and flood control, among others. As a result, while projects may individually incorporate consideration of climate change risk and resilience, they may not be prioritized to address the most severe expected future climate change risks. Even with ad hoc agency efforts, federal investment in projects specifically designed to enhance climate resilience to date has been limited. As stated in our Disaster Resilience Framework, most of the federal government’s efforts to reduce disaster risk are reactive, and many revolve around disaster recovery. To a lesser extent, the federal government also invests in activities to reduce risks not associated with a specific, recent disaster. As we reported in April 2018, since 1993 OMB has reported more than $154 billion spread across the government for federal activities to understand and address climate change. However, over that time frame, OMB reported only minimal funding directed specifically at climate resilience projects. We have issued multiple reports that review the federal government’s approach to addressing climate change, and these reports demonstrate an absence of government-wide strategic planning for climate change. Specifically, our past work identifies limitations related to strategic planning for climate change that include a lack of coordination, prioritization, and consolidation of strategic priorities. For example, we reported in October 2009 that the federal government’s emerging climate resilience activities were carried out in an ad hoc manner and were not well coordinated across federal agencies. In May 2011, we reported that federal officials did not have a shared understanding of strategic government-wide priorities related to climate change. In the same report, we found that there was not a consolidated set of strategic priorities integrating climate change programs and activities across the federal government. In our March 2019 high-risk update, we reported that one area of government-wide action needed to reduce federal fiscal exposure is in the federal government’s role as the leader of a strategic plan that coordinates federal efforts and informs state, local, and private-sector action. For this 2019 high-risk update, we assessed the federal government’s progress since 2017 related to climate change strategic planning against five criteria and found that the federal government had not met any of the criteria for removal from the high-risk list. Specifically, since GAO’s 2017 high-risk update, four ratings regressed to “not met” and one remained unchanged as “not met.” (See fig. 2). We have made 62 recommendations related to the climate change high-risk area, 17 of which address improving federal climate change strategic planning. As of August 2019, no action had been taken toward 14 of those 17 recommendations—one dating back to 2003. Executive Order 13783, Promoting Energy Independence and Economic Growth (Mar. 28, 2017). Executive Order 13653, Preparing the United States for the Impacts of Climate Change (revoked) (Nov. 6, 2013). Executive Order 13834, Efficient Federal Operations (May 17, 2018). Executive Order 13693, Planning for Federal Sustainability in the Next Decade (revoked) (Mar. 19, 2015). The Mitigation Framework Leadership Group, an intergovernmental coordinating body, finalized the National Mitigation Investment Strategy in August 2019. However, as noted, our review of the strategy indicates that it does not include a detailed strategic approach to prioritize investments for disaster risk reduction that explicitly accounts for future climate change risks. According to FEMA officials, the strategy sets goals and recommendations that set the stage for developing approaches to address changing conditions. GAO, Climate Change: Improvements Needed to Clarify National Priorities and Better Align Them with Federal Funding Decisions, GAO-11-317 (Washington, D.C.: May 20, 2011); Climate Change: Information on Potential Economic Effects Could Help Guide Federal Efforts to Reduce Fiscal Exposure, GAO-17-720 (Washington, D.C.: Sept. 28, 2017); and Climate Change: Analysis of Reported Federal Funding, GAO-18-223 (Washington, D.C.: Apr. 30, 2018). GAO, High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas, GAO-19-157SP (Washington, D.C.: Mar. 6, 2019). The federal government does not have a strategic approach for investing in climate resilience projects—that is, an intentional, cross-cutting approach in which the federal government identifies and prioritizes projects for the purpose of enhancing climate resilience. Federal agencies may take actions to invest in projects with potential climate resilience benefits related to their own mission areas using funds from federal programs designed for other purposes. In addition, the National Climate Assessment provides high-level information on what is known about observed and projected climate risks in the United States. However, no federal entity looks holistically at the federal government’s investments to strategically prioritize projects to ensure they address the nation’s most significant climate risks and provide the highest net benefits relative to other potential projects. Several stakeholders told us that the federal government’s emphasis has been on funding post-disaster efforts instead of funding resilience projects before a disaster occurs. This is consistent with findings from our July 2015 report that most federal funding for hazard mitigation is only available after a disaster. In addition, according to FEMA officials, some of the agency’s hazard mitigation programs are designed to empower state and local governments to determine their mitigation funding priorities, and these state and local priorities may or may not align with the federal interest. Although we did not identify a government-wide strategic approach specifically for investing in climate resilience projects, the National Mitigation Investment Strategy—a national effort under way to plan for pre-disaster resilience investments—represents a potential cross-agency vehicle for climate resilience planning. However, the strategy does not specifically address climate change or identify and prioritize specific climate resilience projects. In July 2015, we recommended that the Mitigation Framework Leadership Group—a multi-agency group led by FEMA to promote coordination of hazard mitigation efforts across the federal government—establish an investment strategy to identify, prioritize, and guide federal investments in disaster resilience and hazard mitigation-related activities and make recommendations to the President and Congress on how the nation should prioritize future disaster resilience investments. In response, in August 2019, the Mitigation Framework Leadership Group released a national strategy for advancing mitigation investment in the United States and increasing the nation’s resilience to natural hazards. The strategy acknowledges our 2015 recommendation and articulates several high-level recommendations that relate generally to climate resilience, including aligning program requirements and incentives. Specifically, the strategy states that successful risk mitigation requires shared priorities, consistent approaches, aligned funding, expanded incentives, and coordination between the federal government and nonfederal partners (i.e., state, local, tribal, and territorial governments and nonfederal organizations). However, the strategy does not explicitly address future climate change risks or include a strategic approach to identify and prioritize specific climate resilience projects for federal investment. According to FEMA officials, the strategy provides an overarching framework that can accommodate strategic investment related to changing conditions that impact disaster resilience. FEMA officials also told us that specific implementation strategies will be addressed in a later phase of the high- level strategy. While current federal climate resilience investments are ad hoc and not aligned with the nation’s most significant climate risks, our past work and other sources show that an iterative and strategic risk-informed approach for identifying and prioritizing climate resilience projects could better target federal investment. In particular, in December 2016, we reported that enterprise risk management—which involves identifying and assessing risks, as well as preparing appropriate risk responses—can help federal agencies manage risks, such as preparing for and responding to natural disasters. Elements of enterprise risk management call for reviewing a prioritized list of risks and selecting the most appropriate strategy to manage those risks. Furthermore, according to our 2019 Disaster Resilience Framework, the integration of strategic resilience goals across relevant national strategies can help decision makers work toward a common vision and help ensure focus on a wide variety of opportunities to reduce disaster risk. For example, our framework states that in some cases federal efforts have been hindered by multiple agencies pursuing individual efforts without overarching strategies. In addition, the National Academies highlights the importance of an iterative approach to prioritizing climate resilience actions. According to the National Academies, many current and future climate change impacts require immediate actions to improve the nation’s ability to adapt, and possible options need to be prioritized based on where and when urgent action is needed. In addition, because knowledge about future impacts and effectiveness of response options will evolve, policy decisions to manage climate change risks can be improved if they are made in an iterative fashion, according to the National Academies. However, no federal entity has been established to implement a strategic investment approach for climate resilience that includes identifying and prioritizing projects for federal investment in an iterative fashion. According to FEMA officials, without Congressional direction, no federal entity will identify and prioritize climate resilience projects for federal investment because existing federal programs are not designed to serve this purpose. Furthermore, investments by federal agencies are made according to their missions and operations within the federal investment guidelines put forth by OMB, according to officials from the Mitigation Framework Leadership Group. These officials explained that by law, agencies cannot make other investments, which hinders a more formalized climate resilience investment strategy at the agency level. Several stakeholders told us that a strategic approach would allow for a more purposeful, coordinated, and comprehensive federal response to climate risks. Such an approach could help target federal resources toward high-priority projects—namely, those that address the nation’s most significant climate risks and provide the greatest expected net benefits relative to other potential projects—that are not already addressed through existing federal programs. In particular, a strategic and iterative risk-informed approach for identifying and prioritizing climate resilience projects for federal investment could supplement the agency- specific approaches to climate resilience investment currently carried out by individual agencies with different statutes, goals, constituencies, and funding streams. Such an approach presents an opportunity to enhance the nation’s resilience to climate change and reduce federal fiscal exposure. Six key steps provide an opportunity for the federal government to strategically identify and prioritize climate resilience projects, based on our review of reports (including a National Academies report and the Fourth National Climate Assessment) that discuss adaptation as a risk management process, international standards, our past work (including our enterprise risk management criteria), and interviews with stakeholders. The six key steps are (1) defining the strategic goals of the climate resilience investment effort and how the effort will be carried out, (2) identifying and assessing high-risk areas for targeted resilience investment, (3) identifying potential project ideas, (4) prioritizing projects, (5) implementing high-priority projects, and (6) monitoring projects and climate risks. See Figure 3. We use domestic and international examples—the Louisiana coastal master planning effort and the Canadian Disaster Mitigation and Adaptation Fund (DMAF), respectively—and the aforementioned sources to illustrate the six key steps for identifying and prioritizing climate resilience projects (see text box). Domestic and International Examples of Approaches for Identifying and Prioritizing Climate Resilience Projects Two efforts—the Louisiana coastal master planning effort and the Canadian Disaster Mitigation and Adaptation Fund—illustrate approaches for identifying and prioritizing resilience projects. The scale and purpose of each of these approaches is distinct, but both seek to identify projects that help enhance community resilience to several emerging risks, including risks associated with climate change. Louisiana coastal master planning process: In 2005, the state of Louisiana consolidated coastal planning efforts previously carried out by multiple state and local entities into a single effort carried out by the Coastal Protection and Restoration Authority (CPRA). In this effort, CPRA periodically identifies high-priority coastal resilience projects designed to reduce flood risk and coastal land loss. With involvement from stakeholders from private industry and local communities, CPRA has published three coastal master plans in which it identified and evaluated potential projects. In Louisiana’s 2017 Comprehensive Master Plan for a Sustainable Coast, CPRA identified $50 billion in high-priority projects to be implemented as funding becomes available. Canadian Disaster Mitigation and Adaptation Fund: In 2018, the federal government of Canada launched the Disaster Mitigation and Adaptation Fund (DMAF), which seeks to enhance resilience by addressing the potential impacts of climate change in Canada. Canada’s DMAF is a financial assistance program that provides funds to other entities (e.g., Canadian provinces and territories, not-for-profit and for-profit organizations, local governments, and indigenous communities) for implementation. This US$1.5 billion fund will provide contributions over 10 years for large-scale, nationally significant projects that address a myriad of risks triggered by natural hazards such as floods, wildfires, and droughts. The DMAF also encourages partnerships between eligible recipients, according to a DMAF official. Canada’s DMAF effort is under way. Reports, our past work, stakeholders, and our examples from Louisiana and Canada illustrate the importance of several steps to define the climate resilience investment effort, including defining the efforts’ strategic goals, designating an entity and providing authority for it to lead the effort, identifying participants and defining responsibilities, and determining how the effort will be funded. Clear strategic goals can yield more effective decisions about which projects to prioritize and increase the likelihood that projects are strategically aligned around a common purpose. In October 2011, we reported that strategic goals explain the purpose of agency programs and the results that they intend to achieve. Our domestic and international examples also demonstrate the importance of having defined strategic goals. Specifically, Louisiana’s Coastal Protection and Restoration Authority (CPRA) defined five goals to guide its coastal master planning effort: reducing economic losses to homes and business from storm surge-based flooding, promoting sustainable coastal ecosystems, providing habitats for a variety of commercial and recreational activities across the coast, sustaining coastal Louisiana’s cultural heritage, and maintaining a viable working coast to support businesses and industry. The goal of Canada’s DMAF is to strengthen the resilience of Canadian communities through investments in large-scale infrastructure projects of national importance—including natural infrastructure projects—enabling these communities to better manage the risk associated with current and future natural hazards such as floods, wildfires, and droughts. This includes natural hazards that may be exacerbated by climate change. Several stakeholders we interviewed identified potential strategic goals for a federal climate resilience investment effort, including increasing the resilience of communities to climate hazards and reducing federal fiscal exposure to climate change. Furthermore, several stakeholders explained that a goal of federal resilience investment should include helping communities that do not have the capacity to implement climate resilience projects on their own for various reasons such as limited funds to plan and implement such projects. According to one stakeholder we interviewed, because the federal role in investing in climate resilience projects could be broad, it will be necessary to precisely define the nature and scope of the funding effort in a way that is manageable, potentially restricting funding to resilience projects that would not occur without federal intervention. For example, federal resilience investment could focus on large-scale, long-term climate resilience projects that are otherwise too big, expensive, or cross-jurisdictional for local, state, or private-sector actors to address, according to several stakeholders. Based on our review of several reports and past GAO work and discussions with several stakeholders, various types of entities could lead a federal climate resilience investment effort. This could include various organizational arrangements such as a federal entity or interagency collaborative effort—task forces, special councils, interagency offices, or interagency working groups led by agency and department heads or program-level staff. According to one stakeholder we interviewed, a federal climate resilience investment effort would need a high level of political support to be effective. Several other stakeholders explained that clear authority for the entity to conduct its work would be important to provide legitimacy for the effort and create buy-in among participants and the public. Authority for conducting a resilience effort could be provided via a legislative mandate or executive order. For example, in the case of Louisiana, the state legislature passed a law establishing CPRA, a state agency, in 2005 and providing it with a mandate to develop, implement, and enforce a comprehensive coastal protection and restoration master plan. Identifying participants and defining responsibilities could involve identifying an interdisciplinary team of experts to help evaluate climate risk, generate project ideas, and evaluate projects. According to several stakeholders, experts should have a breadth of expertise in disciplines such as climate science, resilience, social sciences (e.g., economics), engineering, finance, urban planning, infrastructure, and knowledge of affected systems (e.g., transportation systems, public health, and ecosystems). Several reports and several stakeholders also identified the importance of involving representatives from the communities and groups impacted by potential projects, explaining that doing so can increase support for the process and help ensure projects meet communities’ needs. For example, a CPRA official told us that building trust and communicating projects’ necessity with external stakeholders is extremely important when prioritizing projects because some stakeholders will be directly impacted by certain projects. For this reason, according to CPRA officials, CPRA conducted extensive outreach with community groups and other stakeholders to understand their perspectives on projects under consideration and their potential impacts. In addition, past GAO work identifies agreement on roles and responsibilities as one of several practices to enhance and sustain collaborative efforts. According to our September 2012 report, this includes considering clarity of roles and responsibilities and articulating and agreeing to a process for making and enforcing decisions. Determining how the effort will be funded includes identifying potential funding options (discussed later in this report) and establishing a budget for investments in resilience projects. Based on the domestic and international examples we reviewed, there are different ways to identify a budget for resilience projects. The budget for Canada’s DMAF—the equivalent of about US$1.5 billion over 10 years—was established through the Canadian budget process. In contrast, Louisiana’s CPRA used economic analysis to identify the optimal budget for the coastal master planning effort—$50 billion—with funds for specific projects to be solicited from various federal and nonfederal sources. High-risk areas for targeted resilience investment could include regions of the country at high risk for climate hazards, economic sectors at high risk (e.g., agriculture, health, or energy), or severe or costly expected climate hazards (e.g., sea level rise), based on our review of several reports, illustrative examples, and interviews with several stakeholders. According to the National Academies and several stakeholders we interviewed, climate resilience actions should address climate hazards that are acute (e.g., the risk of more frequent or intense extreme weather) and chronic (e.g., sea level rise). In Louisiana, CPRA identified two climate risks—flooding risk and loss of coastal land—for targeted resilience investment. The U.S. Climate Resilience Toolkit, a website designed to help people find and use tools, information, and subject matter expertise to build climate resilience, and several reports we reviewed identified several factors that influence a community’s level of climate risk. This information can help decision makers identify high-risk areas for targeted resilience investment. First, a community’s exposure is influenced by the population or assets exposed to a potential climate hazard (e.g., sea level rise, wildfire). For example, according to the Fourth National Climate Assessment, the expansion of human activity into forests and other wildland areas has been observed over the past few decades and is expected to further increase the exposure of people and property to fire risk. Second, the level of expected impact a community faces from a given climate hazard is influenced by the probability of a given climate hazard and its expected magnitude. Third, a community’s vulnerability to these hazards is influenced by its sensitivity to a given climate risk and its adaptive capacity—the ability to cope with stress or adjust to new situations. An area with high exposure but low sensitivity to a given climate hazard may have lower overall risk than an area with lower exposure to the same hazard but higher sensitivity. The degree of adaptive capacity can also serve to increase or decrease risks. For example, according to the Fourth National Climate Assessment, tribal nations are especially vulnerable to climate change because of their reliance on threatened natural resources for their cultural, subsistence, and economic needs. We reported in September 2017 that while estimates of the economic effects of climate change are imprecise due to modeling and information limitations, they can convey useful insight into broad themes about potential damages in different U.S. sectors or regions. This information could help decision makers identify significant climate risks as an initial step toward managing them and provide insight into high-risk areas for targeted investment. For example, we reported in September 2017 that the two national-scale studies available at the time that examined the economic effects of climate change across U.S. sectors suggested that the potential economic effects of climate change could be significant and unevenly distributed across sectors and regions. According to one of the studies, the Southeast, Midwest, and Great Plains regions likely will experience greater combined economic effects than other regions, largely because of coastal property damage in the Southeast and changes in crop yields in the Midwest and Great Plains. (See fig. 4). In addition, several stakeholders told us that USGCRP’s National Climate Assessment, which describes potential climate change risks to the United States, could help inform decisions about which regions of the country or climate risks to target for resilience investment. In addition, the Notre Dame Global Adaptation Initiative has developed an interactive database that provides information on the level of climate risk U.S. cities face and these cities’ readiness to enhance resilience. Nevertheless, one official from the Mitigation Framework Leadership Group noted that identifying climate risks is challenging, in part, because opinions about which risks are most urgent will vary according to the perspective of the observer. According to the National Academies, even though there are still uncertainties about the nature, timing, and magnitude of climate change impacts, mobilizing now to increase the nation’s resilience can be viewed as an insurance policy against climate change risks. Identifying potential project ideas that align with high-risk areas for targeted resilience investment is the third step in the process for identifying and prioritizing climate resilience projects for federal investment. Potential projects may differ in purpose and location and could include constructing hard infrastructure (e.g., flood defenses such as seawalls) and natural infrastructure (e.g., wetlands in coastal areas) to protect against climate hazards, relocating a community out of harm’s way, or developing a suite of projects designed to collectively address a climate hazard (e.g., wildfire risk or drought) in a particular region of the country, according to several stakeholders we interviewed and based on our review of several reports. From our interviews with several stakeholders and our review of our examples from Canada and Louisiana, we noted two methods for identifying ideas for resilience projects—”bottom up” and “top down”—that can be used individually or in combination. Several stakeholders told us that project ideas could come from a “bottom-up” method in which the federal government seeks proposals from tribal, state, and local governments; regional groups; or other stakeholders for projects. For example, Infrastructure Canada, the federal department that administers the DMAF, sought project ideas from provinces, territories, municipal and regional governments, indigenous groups, and others. Under the DMAF, these entities applied directly to Infrastructure Canada for funding. Likewise, in Louisiana, CPRA also used a “bottom-up” method to identify projects by allowing citizens, state agencies, nongovernmental organizations, academics, and others to submit project ideas. Where necessary, staff at CPRA developed the more detailed plans needed to evaluate and operationalize the projects. CPRA officials said that involving the communities where climate resilience projects will be located in the project identification process helped create support for these projects. Two stakeholders explained that the process for identifying potential project ideas must be sensitive to the fact that some communities do not have the administrative capacity to develop proposals. Otherwise, project ideas will primarily come from communities with ample institutional capacity, and locations with less administrative capacity—and the climate risks associated with these locations—will be missed. According to a 2014 report by the President’s State, Local, and Tribal Leaders Task Force on Climate Preparedness and Resilience, the federal government can drive more resilient community choices by, among other things, providing technical assistance to states, territories, tribes, and communities that lack capacity to adapt to climate change. In 2014, HUD launched the National Disaster Resilience Competition to fund disaster recovery and long-term community resilience in parts of the country that had recently been affected by major disasters. During the first phase of the competition, eligible states and communities impacted by a disaster from 2011 through 2013 could obtain technical assistance through resilience workshops. According to HUD, these workshops provided information and expertise to help communities understand resilience and identify various threats, hazards, economic stresses, and other potential shocks that could impact each community. The workshops also offered eligible applicants tools and concepts to better identify and assess their risk, engage with their communities, choose resilience- building opportunities, and develop strong applications. Several stakeholders told us that projects could be identified through a “top-down” method, in which potential projects would be identified by an interdisciplinary group of federal officials and other experts. According to one stakeholder, a “top-down” method could facilitate consideration of cross-cutting projects that address multiple climate risks and regions of the country. In addition, according to two stakeholders, such a top-down method could help identify projects unlikely to be suggested by local stakeholders for various reasons, such as the local communities not having the administrative capacity to develop and submit such proposals or a local community’s interest being at odds with the national interest (e.g., relocation of a high-risk community when relocation would result in the loss of local tax revenue). However, officials from the Mitigation Framework Leadership Group explained that without the involvement of communities and prioritization of local needs, a top-down approach could be viewed as disconnected from community needs. In Louisiana, CPRA supplemented its “bottom-up” method with “top-down” identification of additional potential projects by, among other things, reconsidering past project proposals that were not selected and working with stakeholders to design potential projects. Prioritizing projects is the fourth key step in the process for identifying high-priority projects for federal investment. Based on our review of several reports and interviews with several stakeholders, prioritizing projects for federal investment should involve evaluating individual projects using scientific and data-based processes. For example, according to a 2010 report by the National Academies, managing risk in the context of enhancing resilience to climate change involves using the best available social and physical science to understand the likelihood of climate impacts and their associated consequences and then selecting and implementing the response options that seem most effective. Stakeholders we interviewed, the Louisiana example, and our past work indicate the need to solicit feedback from communities on the potential impacts of proposed projects. Furthermore, according to several stakeholders we interviewed, projects should be prioritized by an independent, interdisciplinary group of experts capable of assessing projects against measurable criteria. For example, according to Canadian officials, Infrastructure Canada seeks considerations on potential projects from two committees of experts: the first one is comprised of a panel of experts from other federal departments, and the other is comprised of nonfederal experts, including urban planners, sustainability professionals, and individuals with various regional expertise. We identified several potential criteria and tools that could be used to evaluate projects and identify those that are high priority, as described below. We identified various potential criteria for evaluating projects and assigning priority for federal investment, based on our review of reports, interviews with stakeholders, and the Louisiana and Canadian examples. Potential criteria fell into three general categories: goal-oriented criteria (i.e., criteria that measure the extent to which a project enhances resilience and meets other goals), efficiency criteria (i.e., criteria that measure a project’s ability to maximize efficiency, including by maximizing benefits and minimizing costs), and administrative criteria (i.e., other criteria that program administrators may want to consider). See table 1 for more details. The federal government can select a limited number of criteria for evaluation that align with the overall strategic goals of the climate resilience investment effort, based on our discussions with stakeholders. Goal-oriented criteria. We identified several goal-oriented criteria— criteria that measure the extent to which a project enhances resilience to climate change and meets other goals—that decision makers may want to consider when evaluating which projects to prioritize, based on several reports we reviewed and stakeholders we interviewed. Several reports and several stakeholders suggested prioritizing projects that, among other things, focus on severe or costly climate hazards as well as climate hazards about which there is the most scientific certainty. Several stakeholders we interviewed explained that when prioritizing projects for implementation, it is important to consider a project’s potential to enhance resilience by protecting human lives, health, and safety, and assets that are critical, high- value, or culturally significant. In addition, several stakeholders told us that decision makers should not place too much emphasis on the monetary value of avoided property losses from a project because doing so can overemphasize projects that protect high-value assets and leave socially vulnerable populations with limited economic resources less protected. According to one report, the loss of assets is more difficult for a poor household to absorb than a wealthy household that has more assets to begin with and more access to insurance and credit. Similarly, the Fourth National Climate Assessment notes that poor or marginalized populations often face a higher risk from climate change because they live in areas with higher exposure, are more sensitive to climate impacts, or lack the capacity to respond to climate hazards. Several stakeholders told us that to account for a lack of social equity, it is important to prioritize projects in communities that have limited capacity to enhance resilience without federal financial assistance, including communities with limited financial means. In addition to these factors, several reports and several stakeholders discussed the importance of considering a project’s impacts on the environment, including its ability to protect unique or sensitive environmental habitats or species. Finally, several reports discussed the importance of considering the potential system-wide impacts of a project, including a project’s potential to provide benefits as well as the potential that risk may be transferred to neighboring communities. The DMAF applicant’s guide provides an example of potential risk transfer, explaining that the construction of new dikes along a river to protect a segment of the floodplain may confine the river, raising water levels upstream and increasing the velocity of the river downstream. This may reduce the hazard in the segment of river immediately adjacent to the structure but will transfer risk to upstream and downstream communities. Efficiency criteria. We identified several efficiency criteria—criteria that measure a project’s ability to maximize net benefits—that decision makers may want to consider when evaluating which projects to prioritize. Several reports we reviewed identified the importance of considering how a project’s expected benefits compare to its costs to help ensure a project represents an efficient use of federal dollars. With respect to costs, one stakeholder identified the importance of considering the current costs of implementing a project as well as how costs might change in the future if a project’s implementation is delayed to a later date. With respect to benefits, several stakeholders indicated that while it can be difficult to estimate the monetary value of some benefits, it is important to consider all expected benefits— including co-benefits—as fully as possible to draw accurate conclusions about how a project’s benefits compare to its costs. For example, several stakeholders discussed the need to account for future benefits because much of the value of a climate resilience project may be realized far in the future as climate risks become more pronounced. In addition, several reports identified ways to account for uncertainty about the specific nature of future climate risks when making decisions about which projects to prioritize. This includes, for example, prioritizing projects that provide benefits under a wide range of future climate scenarios or prioritizing projects that can be modified if future climate conditions are different than expected. In addition to these considerations, several stakeholders also suggested considering the long-term viability of communities being helped by a project. These stakeholders explained that some communities may face climate risks that are so severe over the long term that they preclude cost-effective investments in resilience. They explained that rather than make costly resilience investments in these communities, a more efficient use of federal funds might involve making investments in projects that help transition a community to a safer location. Similarly, according to a 2015 study by the U.S. Army Corps of Engineers, given current and projected sea level and climate change trends, some of the built environment will become unsustainable for communities presently located there, which may mean that communities may have to relocate in a responsible manner to sustain their economic viability and social resilience. Another stakeholder suggested prioritizing resilience projects that are unlikely to be funded without federal investment, such as projects for the public good that do not generate revenue and likely would not attract private investors. Administrative criteria. We identified several additional criteria that federal decision makers investing in climate resilience projects may want to consider when evaluating which projects to prioritize, including whether the project is feasible and timely. One stakeholder identified the importance of using federal dollars to invest in projects with novel resilience techniques since these projects otherwise might be unlikely to receive investment from other sources. For example, the Canadian DMAF awards merit to projects that offer effective solutions through unique innovative ideas. One stakeholder suggested that the federal government may want to consider the overall distribution of projects across hazards and regions to ensure that all hazards and regions of the country are getting at least some investment in resilience. Based on our review of several reports and illustrative examples, various tools used individually or in combination could help decision makers evaluate projects in order to identify high-priority ones and visualize project trade-offs. For example, using multi-criteria analysis involves decision makers identifying potential criteria, assigning weights to the criteria, ranking proposed projects against the weighted criteria, and using the results to compare projects and inform decisions about which projects to implement. In Canada, officials with the DMAF use multi-criteria analysis to rank potential resilience projects against multiple criteria including the extent to which projects strengthen community resilience and reduce the impacts of natural disasters. Quantitative modeling is another tool that can help decision makers visualize the potential benefits and costs of proposed projects under multiple future climate change scenarios, and thus facilitate identification of high-priority projects. For example, in Louisiana, CPRA used computer modeling tools to evaluate how projects could reduce future land loss and flooding risk, among other effects. To account for uncertainty about future climate and economic conditions, the modeling tools estimated project outcomes under multiple future scenarios representing varied climate conditions (e.g., sea level rise and the frequency and intensity of storms), economic growth conditions, and other factors. According to the Comprehensive Master Plan for a Sustainable Coast, information from the modeling tools helped support deliberations between CPRA and coastal stakeholders that helped identify high-priority projects for implementation. High-priority resilience projects can be implemented as funds become available, while decision makers consider the optimal timing of project implementation. For example, in Louisiana’s coastal master planning effort, CPRA identified $50 billion in projects to be implemented as various federal and nonfederal funding sources become available. CPRA sequences project implementation based on project effectiveness and benefits in the near term or the long term. See figure 5 for completed, ongoing, and planned projects. Project implementation may be influenced by the presence of “windows of opportunity”—periods of time when outside factors make it advantageous or cheaper to implement a project, based on our review of several reports. For example, according to the Fourth National Climate Assessment, many jurisdictions and businesses have significant stocks of aging transportation, water, energy, housing, and other infrastructure, and new infrastructure investments and capital stock turnover provides one particularly favorable opportunity for low-cost, proactive climate resilience investment. In addition to the availability of funding and windows of opportunity, projects may also need final approval from a decision-making entity—the Minister of Infrastructure, in the case of Canada’s DMAF— before implementation. In the case of Louisiana, the state legislature must approve the overall master plan, although, according to a CPRA official, the legislature does not approve the inclusion of individual projects or project concepts. Monitoring the projects being implemented and the state of climate risks can provide information to inform future decisions about high-priority climate resilience projects for federal investment. According to the 2010 report by the National Academies, policy decisions to manage risk can be improved if they incorporate the concept of “adaptive management”— monitoring progress in real time and changing management practices based on learning about and recognizing changing conditions. As an example, Louisiana’s CPRA monitors the performance of projects and the condition of the Louisiana coast using the results from these activities to adjust project management actions and inform future coastal master planning efforts. We identified two options for focusing federal funding on high-priority climate resilience projects—coordinating funding provided through multiple existing federal programs with various purposes and creating a new federal funding source specifically for high-priority climate resilience projects—and these options have strengths and limitations. In addition, our analysis of these sources identified opportunities to increase the climate resilience impact of these two funding options. Options for focusing federal funding on high-priority climate resilience projects—coordinating funding provided through multiple existing federal programs with varied purposes and creating a new federal funding source specifically for high-priority climate resilience projects—have strengths and limitations, based on our review of our prior work, relevant reports, and the Louisiana and Canadian examples, as well as interviews with stakeholders. See table 2. One option for focusing funding on high-priority climate resilience projects involves coordinating funds from multiple existing federal programs with varied purposes that were not designed specifically for climate resilience but whose purpose may be compatible with these projects. For example, the state of Louisiana’s coastal master planning effort uses multi-program coordination to fund projects. Specifically, funding for high-priority resilience projects identified in the master plan is provided via several federal and nonfederal programs designed for wetlands restoration, hurricane risk reduction, oil spill recovery, and community development, among other purposes, when the program’s purpose aligns with the project’s purpose. For example, the National Fish and Wildlife Foundation Gulf Environmental Benefit Fund—established in early 2013 as an outcome of plea agreements for the Deepwater Horizon explosion and oil spill—has been used to fund some projects consistent with the master plan that restore barrier islands and implement river diversions. Administrators of these federal and nonfederal funding programs, rather than CPRA, make decisions about how funds are to be spent, but they coordinate with CPRA to ensure decisions are consistent with the master plan. As with the Louisiana example, high-priority climate resilience projects could be funded via one or more federal programs compatible with the project’s purpose. We identified federal programs related to flood control and hazard mitigation that could be used to fund individual projects that may convey climate resilience benefits, including FEMA’s hazard mitigation assistance programs (i.e., Building Resilient Infrastructure and Communities, Pre-Disaster Mitigation, Flood Mitigation Assistance, and Hazard Mitigation Grant programs), HUD’s Community Development Block Grant Disaster Recovery program, and the U.S. Army Corps of Engineers’ civil works program. These programs are managed individually within their agencies and operate under different statutory authorities. However, no federal entity oversees funding for high-priority climate resilience projects, for example, by identifying which existing federal programs could be used to fund particular high-priority projects and coordinating the use of these programs to fund particular projects. Based on our review of the Louisiana example, interviews with stakeholders, and a report we reviewed, we identified several strengths of coordinating multiple existing federal programs with varied purposes to fund high-priority climate resilience projects: Leveraging existing programs. This option leverages an existing architecture of related federal programs and could encourage consideration of climate change in routine agency decisions, based on our interviews with several stakeholders and review of a related report. The federal government already has programs that address natural resources (e.g., coastlines, water resources, and forests) and human systems (e.g., public health, housing, and infrastructure) that will be affected by climate change, according to a 2010 report we reviewed and two stakeholders we interviewed. According to this report and stakeholders, rather than create an additional program to address climate change, it would be better to incorporate consideration of climate change into existing federal decision-making processes. Providing funding for high-priority climate resilience projects via existing federal programs could encourage agencies to think more intentionally about climate change on a regular basis when implementing their programs, according to several stakeholders we interviewed. Providing access to specialists and expertise. Federal officials who have specialized, sector-specific knowledge (e.g., infrastructure, agriculture, or ecosystems) that can be useful when evaluating which projects to fund may have a greater opportunity to provide input if funding decisions are made within existing federal programs, according to several stakeholders. According to one stakeholder, specialized knowledge that resides within federal agencies is necessary when evaluating the trade-offs of potential projects that address diverse systems and assets. This stakeholder explained that, for example, evaluating a project to strengthen a shipping port against hurricanes requires different expertise than evaluating a project to protect the surrounding community against these hurricanes, and agency officials’ specialized knowledge would be useful in evaluating the value of such distinct projects. Providing access to multiple funding sources. Using multiple existing federal programs means that multiple potential funding streams are available for projects. For example, one stakeholder whose community previously used federal funding to implement large- scale resilience projects said that when funding from one program is not available—for example, because the project does not match that program’s purpose or because of insufficient funds—having multiple existing programs from which to seek funding is advantageous. Similarly, Louisiana makes use of multiple federal and nonfederal funding sources to implement projects identified through its master planning effort. On the basis of our review of the Louisiana example, relevant reports, and interviews with stakeholders, as well as our past work—including the Disaster Resilience Framework—we identified several limitations of using existing programs to fund high-priority climate resilience projects: Administratively challenging to coordinate. Several stakeholders and a 2016 report we reviewed identified potential administrative challenges associated with using multiple existing programs with varied purposes to fund high-priority projects. For example, CPRA officials told us that the process of coordinating funding from multiple programs for coastal projects is complicated and requires dedicated staff to identify programs, assess whether projects meet program funding criteria, apply for funds, and ensure that program requirements are met. Several stakeholders told us that the budgets of existing programs may be too limited to fund large-scale climate resilience projects and that acquiring funding for a single project through multiple federal programs can be difficult. For example, FEMA officials told us that a potentially relevant FEMA program—the Pre- Disaster Mitigation Grant Program—has limited overall funding and restricts the financial size of a project, making it challenging to fund large-scale projects such as community relocation. Furthermore, according to a 2016 report about lessons learned from the HUD Rebuild by Design competition, grantees faced challenges combining funds from multiple programs to support comprehensive rebuilding visions because each program had its own procedural and administrative requirements, including different timelines for how and when the funds were made available. Similarly, according to our Disaster Resilience Framework, when multiple programs and activities and multiple funding streams are involved, there is a risk that the array of requirements will increase administrative complexity. As we reported in July 2015, jurisdictional officials engaged in disaster recovery have encountered complex review processes, conflicting federal guidance, and competing federal priorities that diminished the desire of localities to participate in resilience programs. Programs may be siloed. Existing federal programs may be “siloed,” according to several stakeholders and two reports we reviewed, meaning that agencies may have limited visibility over how their projects affect other agencies’ mission areas or a limited ability to consider those effects. The two reports we reviewed identified challenges with siloed agency programs, including that they can discourage more holistic resilience projects with benefits in multiple sectors. For example, according to the 2016 report about lessons learned from the HUD Rebuild by Design competition, program rules may restrict the use of federal funds to certain activities (e.g., flood control), which can make it difficult to justify the additional cost of a more holistic resilience project with benefits in other sectors (e.g., a larger-scale flood control project with water quality co-benefits). According to the National Academies, climate resilience activities have the potential to be redundant or to work at cross purposes if they are not coordinated across sectors, actors, scale, and time frames. For example, the National Academies identified potential tradeoffs between resilience activities in the agricultural, water, and ecosystem sectors, such as increased irrigation in response to drought competing with natural ecosystem flows and domestic water needs. Climate resilience is not the primary focus. Though it may be possible to use some existing federal programs to fund high-priority climate resilience projects, the primary purpose of these programs is not enhancing resilience to climate change, and they are not coordinated toward a common climate resilience goal, according to our work for this report. As a result, relying on existing programs for funding could result in inadvertent, ad hoc funding rather than intentional, coordinated, and strategic funding of high-priority projects, based on our past work and interviews with several stakeholders. In particular, according to FEMA officials, statutory and regulatory limitations could make it challenging to incorporate consideration of climate resilience into existing programs. Furthermore, according to several stakeholders, program funding criteria may not relate directly to climate resilience—this can lower the chance that climate resilience projects will receive funding. In our May 2014 report about DOD’s consideration of climate change in infrastructure planning, we reported that military installation officials rarely proposed climate resilience projects because the services’ criteria for ranking and funding potential military construction projects did not include climate change adaptation. In addition, a 2018 report about federal resilience policy we reviewed and several stakeholders we interviewed identified challenges with how cost-benefit formulas account for future climate risk when evaluating the costs and benefits of a project under consideration. Two stakeholders we interviewed told us that the discount rate—the interest rate used to convert benefits and costs occurring in different time periods to a common present value—used in federal cost benefit formulas may too heavily discount future benefits. They explained that when benefits accrue over long time horizons, this can result in future climate benefits appearing small relative to the current cost of project implementation and thus result in some climate resilience projects not being funded. Existing programs may be reactive, not proactive. Some existing programs—for example, HUD’s Community Development Block Grant Disaster Recovery program and FEMA’s Hazard Mitigation Grant Program—are limited to funding resilience projects after a disaster occurs, which may result in reactive instead of proactive funding, based on our review of our past work and discussions with several stakeholders. We concluded in July 2015 that funding hazard mitigation efforts in a post-disaster environment can create a reactive and fragmented approach in which disasters determine when and for what purpose the federal government invests in disaster resilience. Furthermore, tying climate resilience funding to a disaster can result in projects going unfunded in communities where there has not yet been a disaster but where there are legitimate risks from future climate change impacts—including chronic climate hazards such as sea level rise—according to several stakeholders we interviewed. For example, our past work and several stakeholders identified challenges in accessing funding from existing federal programs to relocate communities threatened by climate hazards, such as Alaskan native villages threatened by flooding and erosion caused by sea level rise. According to our June 2009 report, since many Alaskan native villages facing gradual erosion problems had not received a declared disaster designation, they did not qualify for some FEMA disaster recovery and hazard mitigation programs. In addition, according to a 2016 report we reviewed, disaster recovery programs tend to be reactive and backward looking, focusing on areas immediately affected by a disaster. This can limit the ability of grantees to fund projects that could more holistically reduce the full suite of future risks that a region or community face. Another option for focusing federal funding on high-priority climate resilience projects involves creating a new funding source specifically for such projects. We identified two main ways a new funding source could be designed in the United States: (1) a federal financial assistance program that could provide grants, loans, or loan guarantees to nonfederal entities implementing high-priority climate resilience projects, or (2) a climate resilience infrastructure bank that could combine federal funds with funds from other sources to provide funding to nonfederal entities for implementing high-priority climate resilience projects. The government of Canada employs both of these methods. Specifically, Canada created the DMAF as a one-time, centralized fund of about US$1.5 billion dollars for climate resilience projects over a 10-year period. Applications not eligible for or not selected to receive DMAF funding could be eligible under other infrastructure programs. Projects that could generate revenue are shared with Canada’s Infrastructure Bank for consideration. Based on our review of the DMAF and interviews with stakeholders, we identified several strengths of creating a new funding source for high- priority climate resilience projects: Administrative simplicity. Several stakeholders said that a new funding source avoids the administrative challenge of coordinating multiple funding sources to implement a large project or portfolio of projects. According to two stakeholders, such an option would avoid the challenge of having to utilize multiple programs with varying program rules, solicitation periods, and funding terms. Another stakeholder suggested that a single source would make it easier to track spending on climate resilience projects. Focusing on high-priority climate resilience projects. Several stakeholders said that an advantage of a new funding source is that it would provide dedicated funding for projects undertaken for the explicit purpose of climate resilience. For example, Canadian officials said that with the DMAF, climate resilience projects do not have to compete with other infrastructure projects for funding as they do within other programs administered in Canada that include multiple eligible project categories (e.g., water, wastewater, public transit). Canadian officials told us that this increases the likelihood that large-scale, nationally significant climate resilience projects will be funded. According to another stakeholder, a new funding source for high- priority climate resilience projects would allow for a proactive focus on the most pressing climate resilience needs instead of reactive project funding through post-disaster spending. In addition, another stakeholder told us this option could encourage communities to think “intentionally” about developing resilience, rather than climate resilience being an afterthought. Furthermore, several stakeholders said that such a funding source could be used for projects that otherwise would not receive funding through existing programs. For example, some projects may not receive funding because they are not compatible with current programs or because current programs have limited funding. Encouraging cross-sector projects. Several stakeholders told us that a new funding source for high-priority climate resilience projects could encourage cross-sector projects designed to achieve benefits in multiple sectors. According to one of these stakeholders, a dedicated fund for climate resilience could allow experts from multiple sectors— such as infrastructure, housing, transportation, and health—to collaborate on projects, leading to more creative, comprehensive approaches to enhance community resilience than would occur when funding projects through individual, existing federal programs. According to the Fourth National Climate Assessment, exploring the climate resilience nexus between sectors can identify co-benefits of resilience solutions and inform cost-effective resilience strategies. For example, the assessment describes co-benefits that resilience actions related to water consumption can have on the electricity sector. According to the assessment, California’s mandate to reduce urban water consumption to address drought conditions in 2015 resulted in significant reductions in both water use and use of electricity to treat and convey water and wastewater. Based on interviews with stakeholders, we identified some limitations of creating a new funding source for high-priority climate resilience projects: Practical challenges. Several stakeholders identified practical challenges with a funding source specifically for high-priority climate resilience projects. For example, such a funding source in the United States does not exist and would have to be created, which would require Congressional authorization. Furthermore, several stakeholders identified decisions that would have to be made about how to design such a funding source, including which agencies would be responsible for administering the fund. Two stakeholders identified additional challenges to success, such as designing effective programmatic rules and eliminating duplication with existing programs. For instance, if the funding source had overly restrictive or poorly designed rules, it might be challenging to use and provide only limited benefits relative to existing programs, according to one of these stakeholders. Discouraging mainstreaming in existing federal programs. Several stakeholders raised concerns that a new funding source for high-priority climate resilience projects could discourage mainstreaming climate change considerations into existing federal programs or lead to the elimination of other sources of funding for climate resilience projects. Several stakeholders explained that mainstreaming is a fundamental way the federal government will enhance resilience to climate risks. In particular, several stakeholders raised concerns that if federal agencies viewed a single funding source specifically for climate resilience projects as sufficient for addressing climate resilience, federal agencies might be less likely to consider climate change impacts when making routine agency decisions or place a lower value on climate resilience project attributes when making funding decisions. Opportunities exist to increase the climate resilience impact of options for focusing federal funding on high-priority climate resilience projects, based on our review of our past work, related reports, an international standard, and the Louisiana and Canadian examples, as well as interviews with stakeholders: Using both existing and new funding options. Several stakeholders told us that using both funding options—multiple, existing federal programs with varied purposes and a new funding source for high-priority climate resilience projects—in a strategic, coordinated way could help increase the impact of federal investment. Several stakeholders told us that directing both funding options at high-priority projects could result in a more effective approach that makes it less likely that high-priority projects fall through the cracks and more likely that these projects will help agencies work toward a common strategic goal. Two stakeholders told us that in practice, multiple, existing federal funding sources that are not specific to climate resilience could be coordinated to fund projects when their purposes and rules align and adequate funding is available. A funding source specifically for climate resilience could be used to fund proposed projects when no related program exists or when existing programs do not have sufficient funding available, according to these and other stakeholders. Helping ensure adequate and consistent funding. Several stakeholders we interviewed identified the need for adequate and consistent funding to implement high-priority climate resilience projects. For example, according to one stakeholder we interviewed, inconsistent, inadequate funding makes it difficult to complete large- scale projects and can lead to additional costs if significant delays occur during which existing work deteriorates. In addition, according to some international officials we interviewed for a May 2016 report, long-term consistency in budgeting provides predictable, reliable resources for climate resilience projects. According to USGCRP’s Fourth National Climate Assessment, adequate funding is a factor that contributes to the successful adoption and implementation of climate resilience by public-sector organizations. Furthermore, an industry standard identified the need to ensure that resources—including financial, human, and technical resources—needed for climate resilience actions are available. In addition to adequate and consistent funding, funding options should be designed to accommodate long-term projects since high-priority climate resilience projects can take multiple years to design and implement, according to two stakeholders we interviewed. Encouraging nonfederal investment. Several stakeholders we interviewed told us that the federal government could use a federal climate resilience investment effort to encourage nonfederal investment in high-priority climate resilience projects, thereby increasing the impact of federal investment. For example, several stakeholders identified the importance of a cost-share component so that funding recipients are invested in a project’s success. Canada’s DMAF encourages nonfederal investment by partially funding projects of national significance and requiring different levels of cost-share from funding recipients, ranging from 25 percent for Indigenous recipients to 75 percent for private-sector and other for-profit recipients. Several stakeholders also identified potential funding mechanisms—for example, public-private partnerships and loan guarantees—that could leverage federal dollars to encourage additional investment in climate resilience projects by nonfederal entities, including the private sector. According to the 2014 President’s State, Local, and Tribal Leaders Task Force report, one way the federal government can drive more resilient community choices is by encouraging innovative approaches that leverage private capital. Encouraging complementary resilience activities. To increase the impact of federal investment, a federal resilience investment effort presents an opportunity to encourage complementary resilience activities by nonfederal actors such as states, localities, and private- sector partners, based on interviews with several stakeholders, the Canadian example, and reports we reviewed. Several stakeholders suggested establishing conditions that funding recipients must meet in exchange for receiving federal funding. Alternatively, according to the 2014 President’s State, Local, and Tribal Leaders Task Force report and two stakeholders we interviewed, the federal government could use incentives (e.g., providing greater federal cost-share or giving additional preference in the project prioritization process) to encourage complementary resilience activities by nonfederal actors. Furthermore, our Disaster Resilience Framework states that incentives can make long-term, forward-looking risk reduction investments more viable and attractive among competing priorities. Specifically, incentives can lower the costs or increase the benefits of risk-reduction measures, which can help stimulate investment by state and local governments, individuals, and the private sector. The federal government could use a federal resilience investment effort to encourage several types of complementary resilience activities by nonfederal actors. For example, the federal government could encourage the use and enforcement of building codes that require stronger risk-reduction measures, according to two reports we reviewed and several stakeholders we interviewed. In the case of the DMAF, to be eligible for federal funding, all projects under the DMAF must meet or exceed building code requirements for their jurisdiction. In addition, several stakeholders suggested using a federal investment effort to encourage communities to limit or prohibit development in high-risk areas to minimize risks to people and assets exposed to future climate hazards. One example of this would be through zoning regulations. Another stakeholder suggested that communities receiving federal funding for resilience projects should be adequately insured against future climate risks so they have a potential source of funding for rebuilding in the event of a disaster. Allowing funds to be used at various stages of project development. Several stakeholders suggested that federal funds be allowed for use at multiple stages of project development—such as project design, implementation, or monitoring—to increase the impact of federal funds. For example, two stakeholders we interviewed told us that resilience projects can require significant amounts of design work to develop an implementable and effective project concept and that making funds available for project design could improve the quality of project proposals, thereby maximizing the impact of federal funds. Similarly, a CPRA official explained that many project proposals for Louisiana’s Comprehensive Master Plan for a Sustainable Coast are in the “concept stage” when they are received so funds are needed to refine the concept and craft an implementable project design. In addition to providing federal funds for project design, one stakeholder suggested making federal funding available to measure project outcomes (e.g., how effectively projects increased resilience) to improve future decisions by both the federal government and others making resilience investments. Individual federal agencies have provided ad hoc funding for projects that may convey some climate resilience benefits using existing federal programs. However, the federal government does not have a strategic approach for investing in climate resilience projects that targets federal resources toward projects that address the nation’s most significant climate risks. USGCRP projects that disaster costs will likely increase as certain extreme weather events become more frequent and intense due to climate change. The rising number of natural disasters and increasing reliance on the federal government for assistance is a key source of federal fiscal exposure. Investment in climate resilience projects can help prepare the country for the effects of climate change. We found that to strategically identify and prioritize climate resilience projects for federal investment, the federal government could take six key steps, based on reports we reviewed, past GAO work, international standards, and stakeholders we interviewed. In addition, opportunities exist to increase the climate resilience impact of funding options, such as by encouraging the use of climate-resilient building codes. However, no federal agency, government-wide coordinating body, or other organizational arrangement has been established to periodically identify and prioritize climate resilience projects for federal investment. Our past work and other sources highlight the importance of a strategic and iterative risk-informed approach to climate change and the need to reduce the federal government’s fiscal exposure. However, the federal government has made little measurable progress since 2017 to reduce its fiscal exposure to climate change. Although we have made 17 recommendations that address improving federal climate change strategic planning, as of August 2019, no action had been taken toward implementing 14 of those recommendations—one dating back to 2003. A strategic and iterative risk-informed approach for identifying and prioritizing climate resilience projects for federal investment, with an appropriate organizational arrangement, could help target federal resources toward climate resilience projects that have the greatest expected net benefit and that address the nation’s most significant climate risks. Congress should consider establishing a federal organizational arrangement to periodically identify and prioritize climate resilience projects for federal investment. Such an arrangement could be designed for success by considering the six key steps for prioritizing climate resilience investments and the opportunities to increase the climate resilience impact of federal funding options identified in our report. (Matter for Consideration 1) We provided a draft of this report to the U.S. Global Change Research Program, the Federal Emergency Management Agency, and the Mitigation Framework Leadership Group for review and comment. These entities provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Executive Director of the U.S. Global Change Research Program, the Acting Secretary of the Department of Homeland Security, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. In this report, we examine (1) the extent to which the federal government has a strategic approach for investing in climate resilience projects; (2) key steps that provide an opportunity for the federal government to strategically prioritize climate resilience projects for federal investment; and (3) strengths and limitations of options for focusing federal funding on high-priority climate resilience projects. To address all three audit objectives, we conducted semi-structured interviews with 35 stakeholders with relevant expertise, including federal officials, researchers, and consultants. We used a snowball approach to identify stakeholders with expertise on the topics addressed by our report. This involved identifying an initial list of stakeholders with expertise in climate resilience and hazard mitigation by reviewing related reports and based on stakeholder involvement in related present or past federal efforts—for example, work conducted by the U.S. Global Change Research Program (USGCRP)—the federal program responsible for coordinating climate change research and preparing the Fourth National Climate Assessment. We identified additional stakeholders with expertise in these and other relevant areas through interviews with this initial group of stakeholders and review of additional reports. We considered several factors when selecting stakeholders: the relevance of their expertise, the number of times they were recommended to us by other stakeholders as having relevant expertise, and their current or previous federal experience. We sought a balanced set of stakeholders with expertise in a variety of fields that could inform climate resilience decisions: climate resilience, decision sciences, hazard mitigation, economics and finance, insurance, engineering and project design, economic and community development, potentially related federal programs (e.g., Federal Emergency Management Agency hazard mitigation programs), and several affected resources (e.g., coasts, infrastructure, water resources, and ecosystems). We use the term “several” to represent three or more stakeholders or reports expressing a particular viewpoint. In other cases, we provide the exact number of stakeholders expressing a particular viewpoint. Because this is a nonprobability sample, our findings cannot be generalized to other stakeholders we did not interview. Rather, these interviews provided us with illustrative examples of (1) the extent to which the federal government has a strategic approach for investing in climate resilience projects, (2) key steps that provide an opportunity for the federal government to strategically prioritize climate resilience projects for federal investment, and (3) strengths and limitations of options for focusing federal funding on high-priority climate resilience projects. In addition, the specific areas of expertise varied among the stakeholders we interviewed, so not all of the stakeholders commented on all of the interview questions we asked. To determine the extent to which the federal government has a strategic approach for investing in climate resilience projects, we reviewed past GAO work on federal efforts related to climate resilience and climate change funding as well as reports from the Congressional Research Service, Congressional Budget Office, the Council on Climate Preparedness and Resilience, USGCRP, and other sources. We also reviewed federal documents, including the National Mitigation Investment Strategy—a national strategy for mitigating natural hazards. We interviewed officials from USGCRP and FEMA, the federal agency responsible for leading the Mitigation Framework Leadership Group, the interagency group that developed the National Mitigation Investment Strategy under Presidential Policy Directive 8. We also interviewed several other stakeholders on the extent to which the federal government has a strategic approach for investing in climate resilience projects and the nature and scope of the Mitigation Framework Leadership Group’s activities. We reviewed federal documents and websites to identify examples of instances in which federal programs and funding sources designed for other purposes, such as disaster funding, have been used to invest in climate resilience projects. To identify key steps that provide an opportunity for the federal government to strategically prioritize climate resilience projects for federal investment, we reviewed our prior work related to risk management, climate change, climate resilience, and hazard mitigation, including our Disaster Resilience Framework and enterprise risk management report. We also reviewed approximately 50 reports and other sources to identify steps that provide an opportunity for the federal government to strategically identify high-priority climate resilience projects, several of which contained examples of potential criteria the federal government could consider when prioritizing these projects. We identified these reports and other sources through our review of other reports and related news, discussions with stakeholders, and searches of databases such as Scopus and ProQuest. The reports we reviewed included climate resilience planning guidebooks that outline steps communities can follow to design a resilience plan to address climate risks. We also interviewed stakeholders with relevant expertise to gather information on key steps the federal government could take and criteria it could consider to strategically prioritize climate resilience projects for federal investment. During the course of this work, we identified domestic and international examples of governments that invest in climate resilience and related projects. We selected two of these examples for more in-depth review and presentation in the report: the state of Louisiana’s coastal master planning effort and the country of Canada’s Disaster Mitigation and Adaptation Fund. These examples represent distinct approaches for investing in high-priority projects that help communities adapt to emerging risks such as those associated with climate change. We selected these examples for further review because they focus on projects that are large in scale; are of national or statewide significance; address multiple risks; represent well-defined, current processes for identifying and prioritizing projects; and had sufficient information available to understand their approach. To examine the strengths and limitations of options for focusing federal funding on high-priority climate resilience projects, we identified relevant examples of the strengths and limitations of federal funding options in several of the reports we mentioned above. Where appropriate, we supplemented this review with a review of additional reports that discussed specific financial mechanisms that the federal government could use to fund large-scale climate resilience projects. We also interviewed stakeholders to discuss the strengths and limitations of options the federal government could use to fund climate resilience projects. When available, we gathered their views on specific funding sources that the federal government could use to fund large-scale climate resilience projects and additional steps that the federal government could take to enable more targeted federal resilience investment. We conducted this performance audit from January 2018 to October 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the individual named above, Joe Thompson (Assistant Director), Paige Gilbreath (Analyst in Charge), Taiyshawna Battle and Celia Rosario Mendive made key contributions to this report. Also contributing to this report were Colleen M. Candrl, Alicia P. Cackley, Kendall Childers, Steven Cohen, Christopher Curry, Cindy Gilbert, Kathryn Godfrey, Holly Halifax, Carol Henn, Susan Irving, Richard Johnson, Gwendolyn Kirby, Caroline N. Prado, Joseph Maher, Gregory Marchand, Diana Maurer, Kirk Menard, Tim Persons, William Reinsberg, Oliver Richard, Danny Royer, Jeanette Soares, Kiki Theodoropoulos, Sarah Veale, Patrick Ward, Jarrod West, Kristy Williams, Eugene Wisnoski, and Melissa Wolf.", "summary": "Federal funding for disaster assistance since 2005 has totaled at least $450 billion, including a 2019 supplemental appropriation of $19.1 billion for recent disasters. In 2018 alone, 14 separate billion-dollar weather and climate disaster events occurred across the United States, with total costs of at least $91 billion including the loss of public and private property, according to the National Oceanic and Atmospheric Administration. Disaster costs will likely increase as certain extreme weather events become more frequent and intense due to climate change, according to the U.S. Global Change Research Program, a global change research coordinating body that spans 13 federal agencies. In 2013, GAO included “Limiting the Federal Government's Fiscal Exposure by Better Managing Climate Change Risks” on its list of federal program areas at high risk of fraud, waste, abuse, mismanagement, or most in need of transformation. The cost of recent weather disasters has illustrated the need to plan for climate change risks and invest in climate resilience. Investing in climate resilience can reduce the need for far more costly steps in the decades to come. The Disaster Recovery Reform Act of 2018 provides one potential source of funding for climate resilience projects. In particular, it allows the President to set aside up to 6 percent of the estimated aggregate amount of grants from certain programs under a major disaster declaration to implement pre-disaster hazard mitigation activities. Officials estimate funds for the related program will average $300 million to $500 million annually. GAO was asked to review the federal approach to prioritizing and funding climate resilience projects that address the nation's most significant climate risks. This report examines (1) the extent to which the federal government has a strategic approach for investing in climate resilience projects; (2) key steps that provide an opportunity to strategically prioritize projects for investment; and (3) the strengths and limitations of options for focusing federal funding on these projects. GAO reviewed relevant reports and interviewed 35 stakeholders with relevant expertise, including federal officials, researchers, and consultants. In addition, during the course of this work, GAO identified domestic and international examples of governments that invest in climate resilience and related projects. GAO selected two of these examples for in-depth review and presentation in the report: the state of Louisiana's coastal master planning effort and Canada's Disaster Mitigation and Adaptation Fund. The federal government has invested in projects that may enhance climate resilience, but it does not have a strategic approach to guide its investments in high-priority climate resilience projects. Enhancing climate resilience means taking actions to reduce potential future losses by planning and preparing for potential climate hazards such as extreme rainfall, sea level rise, and drought. Some federal agencies have made efforts to manage climate change risk within existing programs and operations, and these efforts may convey climate resilience benefits. For example, the U.S. Army Corps of Engineers' civil works program constructs flood control projects, such as sea walls, that may enhance climate resilience. However, additional strategic federal investments may be needed to manage some of the nation's most significant climate risks because climate change cuts across agency missions and presents fiscal exposures larger than any one agency can manage. GAO's analysis shows the federal government does not strategically identify and prioritize projects to ensure they address the nation's most significant climate risks. Likewise, GAO's past work shows an absence of government-wide climate change strategic planning. As of August 2019, no action had been taken to implement 14 of GAO's 17 recommendations to improve federal strategic planning for climate resilience. GAO's enterprise risk management framework calls for reviewing risks and selecting the most appropriate strategy to manage them. However, no federal agency, interagency collaborative effort, or other organizational arrangement has been established to implement a strategic approach to climate resilience investment that includes periodically identifying and prioritizing projects. Such an approach could supplement individual agency climate resilience efforts and help target federal resources toward high-priority projects. Six key steps provide an opportunity for the federal government to strategically identify and prioritize climate resilience projects for investment, as GAO found based on its review of prior GAO work, relevant reports, and stakeholder interviews (see figure). GAO identified one domestic and one international example to illustrate these key steps: Louisiana's Coastal Protection and Restoration Authority (CPRA) coastal master planning effort and Canada's Disaster Mitigation and Adaptation Fund (DMAF). In the domestic example, in 2005 the Louisiana legislature consolidated coastal planning efforts previously carried out by multiple state entities into a single effort led by CPRA to address the lack of strategic coordination. CPRA periodically identifies high-priority coastal resilience projects designed to address two primary risks: flooding and coastal land loss. To identify potential projects, CPRA sought project proposals from citizens, nongovernmental organizations, and others. To prioritize projects, CPRA used quantitative modeling to estimate project outcomes under multiple future scenarios of varied climate and other conditions and coordinated with stakeholders to understand potential project impacts. In 2017, CPRA identified $50 billion in high-priority projects to be implemented as funds become available. In the international example, in 2018, the Canadian government launched the DMAF, a financial assistance program to provide US$1.5 billion over 10 years for large-scale, nationally significant projects to manage natural hazard risks, including those triggered by climate change. Infrastructure Canada, the entity responsible for administering the DMAF, seeks project ideas from provinces and territories, municipal and regional governments, indigenous groups, and others. These entities apply directly to Infrastructure Canada for funding. According to Canadian officials, two committees of experts—one composed of experts from other federal departments and the other composed of nonfederal experts (e.g., urban planners and individuals with regional expertise)—provide feedback on potential projects. These projects are prioritized based on multiple criteria such as the extent to which they reduce the impacts of natural disasters. On the basis of GAO's review of relevant reports and past GAO work, interviews with stakeholders, and illustrative examples, GAO identified two options—each with strengths and limitations—for focusing federal funding on high-priority climate resilience projects. The options are (1) coordinating funding provided through multiple existing programs with varied purposes and (2) creating a new federal funding source specifically for investment in climate resilience. A strength of coordinating funding from existing sources is access to multiple funding sources for a project. For example, one stakeholder GAO interviewed—whose community used federal funding to implement large-scale resilience projects—said that having multiple programs is advantageous because when funding from one program is not available—such as when the project does not match that program's purpose or when there are insufficient funds—funds could be sought from another program. A limitation of that option, according to CPRA officials, is that coordinating funding from multiple sources could be administratively challenging and could require dedicated staff to identify programs, assess whether projects meet program funding criteria, apply for funds, and ensure program requirements are met. Alternatively, one strength of a new federal funding source is that it could encourage cross-sector projects designed to achieve benefits in multiple sectors. For example, according to one stakeholder, such a funding source could allow experts from multiple sectors—such as infrastructure, housing, transportation, and health—to collaborate on projects, leading to more creative, comprehensive approaches to enhance community resilience. However, such a new funding source would have to be created, which would require Congressional authorization. In addition, GAO identified opportunities to increase the climate resilience impact of federal funding options. For example, a federal resilience investment effort presents an opportunity to encourage several types of complementary resilience activities by nonfederal actors such as states, localities, and private-sector partners. In this example, the federal government could require or provide incentives for communities to use and enforce climate-resilient building codes or limit development in high-risk areas through zoning regulations. Congress should consider establishing a federal organizational arrangement to periodically identify and prioritize climate resilience projects for federal investment. Such an arrangement could be designed using the six key steps for prioritizing climate resilience investments and the opportunities to increase the climate resilience impact of federal funding options that are identified in this report. The Federal Emergency Management Agency and two federal coordinating bodies reviewed a draft of this report and provided technical comments, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "The F-35 Lighting II program is a joint, multinational acquisition program intended to develop and field a family of next-generation strike fighter aircraft for the U.S. Air Force, Navy, and Marine Corps (hereinafter referred to as the services), eight international partners, and foreign military sales customers (collectively hereinafter referred to as program participants). There are three F-35 variants, and each will be a multi-role, stealthy strike aircraft replacement for or complement to the services’ legacy fighter aircraft. DOD initiated the F-35 program in October 2001, and began operational testing of the aircraft in December 2018. DOD has also, concurrently, been fielding and operating a growing fleet of aircraft as part of low-rate initial production. As of February 2019, more than 350 aircraft had been fielded and were operating from 16 bases worldwide. By 2023, the global F-35 fleet is expected to expand to more than 1,100 aircraft across 43 operational sites. In total, the program participants plan to purchase more than 3,300 F-35 aircraft, with the U.S. services planning to purchase nearly 2,500 of those aircraft. See figure 1 for a timeline of anticipated worldwide fleet growth and site activations in the F-35 program. Sustainment for the growing fleet of F-35 aircraft is a large and complex undertaking with several key stakeholders. The F-35 Joint Program Office, through its Product Support Manager, is responsible for managing and overseeing the support functions required to field and maintain the readiness and operational capability of the F-35 aircraft across the enterprise. As such, it establishes sustainment requirements, manages funding, develops contracts, and provides direction for and oversees the execution of F-35 sustainment strategy and policy. Additionally, in 2016, DOD established a Hybrid Product Support Integrator organization within the Joint Program Office, and it expects to fully implement this organization by the end of 2019. Once fully implemented, DOD intends for the Hybrid Product Support Integrator to bring together all government and commercial capabilities necessary to execute the F-35 sustainment strategy. The organization is led by a general officer, who is responsible for providing government oversight of all support providers to ensure that they deliver the required levels of performance. In particular, the F-35 program relies heavily on contractors to provide support for its F-35 aircraft. DOD has two primary contractors for the F-35 program: Lockheed Martin for the overall aircraft system and Pratt & Whitney for the engine. As the prime contractor for the overall aircraft system, Lockheed Martin (hereinafter referred to as the prime contractor) is responsible for managing the F-35 supply chain, depot maintenance, and pilot and maintainer training, as well as for providing engineering and technical support. Currently, DOD is contracting for this support with the prime contractor largely through annual contracts, and it plans to transition to multiple-year, fixed-price, performance-based sustainment contracts when the program achieves certain condition- based criteria, including the establishment of critical sustainment capabilities and the government’s ability to collect and more fully assess performance and cost data. In addition, the U.S. Air Force, Navy, and Marine Corps have each established an F-35 integration office or similar construct focused on how the services will operate and afford the F-35, among other things. Figure 2 below depicts how these key stakeholders provide support to the F-35 program participants across the three aircraft variants. DOD is planning to meet the sustainment requirements of its F-35 customers by providing a common, global support solution. As part of this common solution, participants share most sources of support, such as spare parts, depot maintenance, and training. At the core of the F-35 global support solution is the F-35 supply chain. At maturity, the F-35 supply chain is intended to be a network of manufacturers, commercial and government part repair depots, and base and regional part warehouses that will be located around the world to provide parts to support the operational and training requirements of all F-35 program participants. As a part of the F-35 supply chain, all F-35 program participants— including the U.S. military services, international partners, and foreign military sales customers—share a global pool of F-35 spare parts (formally called the Joint Spares Pool), which we refer to in this report as the F-35 global spares pool. These pooled assets comprise only parts used for F-35 aircraft, such as consumable and repairable spare parts for the airframe, engine system, support equipment, pilot flight equipment, and training devices. The F-35 global spares pool consists of four different packages of parts—the base spares package, the global spares package, the deployment spares package, and the afloat spares package—as described below and in figure 3. Base spares package: A base spares package is a retail-level supply of parts inventory that is positioned at each F-35 main operating base to support the F-35 aircraft operating from that location. Each base spares package is intended to have a sufficient amount of parts to support the number of aircraft and planned flying hours at the base. While inventory within each base spares package is sized to meet the projected needs of the aircraft at that particular location, parts within the base spare packages are intended to be available for sharing among all global participants, as needed. Global spares package: A global spares package is a wholesale- level supply of parts inventory that is positioned at regional warehouses, original equipment manufacturers, and depot repair facilities. The prime contractor manages this inventory to replenish the stocks of parts in base spares packages and the other packages below, and to meet participants’ requirements for parts that are not in their base inventories. If a part is needed for the repair of an aircraft, and the unit does not have the part in its base inventory, the prime contractor sends a part from the global spares package to meet the unit’s requirement. Parts within the global spares package are intended to be available for sharing among all global participants. Deployment spares package: A deployment spares package is a retail-level supply of parts inventory that is purchased by a program participant to support its wartime or contingency operations. This package is intended to have a sufficient amount of parts to support a program participant’s contracted operational requirements for a defined period of time, until the F-35 supply chain is able to ship replenishment parts to the participant’s deployed location. For example, a deployment spares package could be sized to provide parts for 12 aircraft to fly a specified number of flight hours over a 20- day period and be fully mission capable 70 percent of the time. The parts in this package are generally reserved for use only by the participant who purchased the package. Afloat spares package: An afloat spares package is a retail-level supply of parts inventory that is purchased by a program participant to support its F-35 operations aboard a naval vessel. This package is intended to have a sufficient amount of parts to support a program participant’s contracted operational requirements for a defined period of time until the F-35 supply chain is able to ship replenishment parts to the participant aboard the ship. For example, an afloat spares package could be sized to provide parts for six aircraft stationed on a ship to fly a specified number of flight hours over a 20-day period, and be fully mission capable 70 percent of the time. The parts in this package are generally reserved for use only by the participant who purchased the package. All of the parts within the global spares pool are owned by the U.S. government when not installed on a participant’s aircraft. The U.S. military services and international participants do not purchase parts directly, but rather purchase access to parts in the shared pool based on how many F- 35 aircraft they own and the number of flight hours they plan to fly, among other factors. Accordingly, the F-35 program has developed a series of business rules that are intended to govern how parts within the F-35 global spares pool will be managed and shared, and how the costs of the parts will be allocated across participants. The prime contractor manages the F-35 supply chain and is responsible for allocating parts to F-35 sites and participants based on contracted requirements, such as numbers of aircraft and planned flying hours, and program business rules. The effective management of the F-35 supply chain requires significant technical data about the F-35 aircraft and parts, such as engineering data, maintenance instructions, and information related to how often the aircraft experiences failures and how much time it takes to repair those failures. Technical data constitute an important part of a weapon system program, such as the F-35. We have previously reported that identifying technical data needs, costs, and ownership are essential for DOD to effectively consider and maximize competition for future product support of F-35 sustainment, including supply chain management. F-35 aircraft performance is not meeting warfighter requirements. While DOD is taking various actions to improve F-35 spare parts availability so that aircraft can fly and perform their missions, it will likely continue to struggle to meet warfighter requirements—due to how it is planning for and allocating spare parts. The performance of the F-35 fleet is hindered by lower-than-required aircraft availability and capability rates. Air vehicle availability, or the percentage of total time during which the aircraft can fly and perform at least one mission, was 45.8 percent across the F-35 fleet from May through November 2018, as compared with the warfighter minimum target of 65 percent. Full mission capability, or the percentage of time during which the aircraft can perform all of its tasked missions, was 26.8 percent from May through November 2018, as compared with the warfighter minimum target of 60 percent. However, parts availability and aircraft performance varied by aircraft variant and the age of the aircraft. For instance, fleet-wide rates of full mission capability for the F-35A aircraft were higher than those for the F-35B. Figure 4 below shows aircraft performance data by variant across key program metrics relative to stated U.S. warfighter requirements, referred to within the F-35 program as objective and minimum performance targets. From May through November 2018, fleet-wide F-35 aircraft performance did not meet any of the U.S. warfighter’s requirements. Lower-than-required F-35 aircraft performance is attributable in part to spare parts shortages. Specifically, the F-35 supply chain does not have enough spare parts available to keep aircraft flying enough of the time necessary to meet warfighter requirements. According to prime contractor data, from May through November 2018, F-35 aircraft across the fleet were unable to fly 29.7 percent of the time due to spare parts shortages (this metric is hereinafter referred to as the S-rate). Figure 5 below shows the percentage of aircraft that were unable to fly from May through November 2018 due to shortages of parts relative to the program’s target. According to prime contractor data, to keep aircraft flying despite parts shortages, from May through November 2018 F-35 squadrons cannibalized (that is, took) parts from other aircraft at rates that were more than six times greater than the services’ objective. These high rates of cannibalization mask even greater parts shortages, because personnel at F-35 squadrons are pulling parts off of other aircraft that are already unable to fly instead of waiting for new parts to be delivered through the supply chain. The F-35 program is taking a number of actions to try to increase the availability of spare parts, including steps to increase the capacity of suppliers to produce parts to meet sustainment requirements, improve the timing of spare parts deliveries, and address the reliability of certain parts that are failing more frequently than expected. DOD has identified specific parts shortages that are causing the greatest aircraft capability degradation, and it is developing short-term and long-term mitigation strategies to increase the quantity and reliability of these parts. For instance, DOD found that the special coating on the F-35 canopy that enables the aircraft to maintain its stealth failed more frequently than expected, and that the manufacturer could not produce enough canopies to meet demands. To address these challenges, the program is looking for additional manufacturing sources for the canopy and is considering design changes. A key contributor to spare parts shortages is the F-35 program’s limited capacity to repair broken parts. The average time to repair an F-35 part was more than 6 months, or about 188 days for repairs completed between September and November 2018—more than twice that of the program’s objective of 60—90 days. Also, there was a backlog of about 4,300 spare parts awaiting repair at depots or manufacturers (see figure 6). This backlog of parts awaiting repair is largely attributable to delays in the establishment of part repair capabilities at the military depots. The capabilities to repair all parts at the military depots were originally intended to be in place by 2016, but the F-35 program’s current plan now projects that the military depots will not have the capability to repair all parts at the expected repair demand rates until 2024. According to program officials and documentation, the plan includes the required material and technical instructions to repair parts, and DOD has allocated funding for these efforts in its budget planning. However, as of February 2019, funding decisions had not been finalized. In the meantime, to address the gap in part repair capabilities at the military depots, the prime contractor has begun incentivizing manufacturers to increase their capacity to repair spare parts by establishing performance-based repair agreements. As of October 2018, according to program documentation, the prime contractor had established seven such agreements, with six more planned by May 2019. In October 2017, we reported that DOD was experiencing supply chain challenges, largely as the result of sustainment plans that did not fully include key requirements or aligned funding. DOD concurred with our recommendation that it revise its sustainment plans to ensure that they include the key requirements and funding needed to fully implement its sustainment strategy. In January 2019, DOD issued an updated Life- Cycle Sustainment Plan for the F-35. The plan includes eight elements that DOD has identified as critical to enabling the program to achieve its aircraft capability and affordability targets by fiscal year 2024, including accelerating supply chain and depot repair capabilities. The F-35 program is a highly concurrent program wherein aircraft, spare parts, and mission software continue to be developed and redesigned while fielded aircraft must be sustained. As a result, there are at least 39 different part combinations across the fleet. Additionally, DOD’s training and operational squadrons are flying F-35 aircraft with three different blocks of mission software—2B, 3i, and 3F—with Block 3F software having the full warfighting capability. According to the program office, DOD spent more than $15 billion to purchase F-35 aircraft from the earliest lots of production, specifically lots 2 through 5 (hereinafter referred to as “early production aircraft”), but it faces challenges in providing enough spare parts for these aircraft. Early production F-35 aircraft have parts configurations and software that differ from those of later production aircraft, and they have faced more parts reliability issues and parts shortages than later-production aircraft. Figure 7 shows the differences in aircraft performance between early production aircraft and aircraft produced in production lot 6 or later (hereinafter referred to as “later production aircraft”). According to program documentation, DOD plans to upgrade all of its early production aircraft to Block 3F software capability. These upgrades were initially scheduled to be completed by the end of 2021, but DOD is taking actions to accelerate these modifications with the plan to complete the upgrades in September 2020. That upgrade is expected to address some of the reliability challenges the older aircraft have experienced. However, program and contractor officials said that these upgrades are not a comprehensive solution, as there will still be many parts that are used on these early production aircraft that are not reliable and are in short supply. Accordingly, DOD is taking action to retrofit some other parts that are not addressed by the modifications. These challenges disproportionately affect the U.S. services’ training fleets, as the majority of U.S. early production aircraft are currently being used for that mission. For example, the training units at Eglin Air Force Base were unable to fly due to parts shortages about 56 percent of the time from May 2018 through November 2018. The Autonomic Logistics Information System is an information technology system that is central to the F-35 sustainment strategy. It is intended to provide the necessary logistics tools to F-35 program participants as they operate and sustain the F-35 aircraft. ALIS consists of multiple software applications designed to support different squadron activities, including supply chain management, maintenance, training management, and mission planning. Specifically, for supply chain management, ALIS was intended to automate a range of supply functions—including updating the status of parts, generating supply work orders, and communicating critical data about parts. However, these capabilities are immature, resulting in numerous challenges and the need for maintainers and supply personnel at military installations to perform time-consuming, manual workarounds in order to manage and track parts. One Air Force unit estimated that it is spending the equivalent of more than 45,000 hours per year performing additional tasks and manual workarounds, including for supply-related functions, because ALIS is not functioning as intended. Supply and maintenance personnel we spoke with at various military installations cited challenges associated with ALIS, including the following: missing or corrupted electronic spare parts data that are required to install a part on an aircraft, necessitating extensive research and troubleshooting to resolve; maintenance and supply systems within ALIS not communicating with each other, resulting in difficulty in electronically tracking aircraft parts as they are physically moved between maintenance and supply locations at the same base; and limited automated capabilities, requiring manual and sometimes duplicative steps for receiving, tracking, and managing parts. We have previously reported on challenges related to ALIS. In April 2016, we reported that DOD did not have a plan to ensure that ALIS was fully functional as key program milestones approached. In October 2017, we reported that DOD faced delays in the development of required ALIS sustainment capabilities and uncertain funding for this development. We are currently conducting a separate review of ALIS, assessing how DOD is managing current and future issues related to the system. We plan to complete this review by the end of 2019. In September 2018, the Secretary of Defense directed the services to achieve and maintain 80 percent mission capability for the F-35 fleet by the end of fiscal year 2019, which program and Office of the Secretary of Defense officials have told us will be difficult to accomplish, given the supply and maintenance challenges facing the fleet. DOD is pursuing a phased approach to achieving this requirement for the F-35 aircraft. DOD’s first priority is to increase the capability of its operational fleet to achieve the 80 percent mission capability target by the end of fiscal year 2019, with the intent to increase the capabilities of its entire F-35 fleet to achieve the target by the end of fiscal year 2020. While DOD has ongoing efforts to increase the availability of spare parts as described above, it is likely to face additional challenges in meeting this requirement as well as the other warfighter aircraft performance requirements, because of the ways in which the program is planning for and allocating parts. The F-35 program is not planning for the quantity of parts necessary in its spare parts projections to meet warfighter performance requirements. The program’s S-rate requirement is used along with a number of other factors in an analytical model to determine the quantity of spare parts to be purchased. Based on this model, DOD is planning to purchase the quantity of parts necessary to achieve a fleet-wide S-rate of 20 percent— meaning the program is buying only enough parts to enable about 80 percent of its aircraft to be mission-capable based on the availability of parts. According to program documentation, the maximum fleet-wide mission capability rates that can be consistently expected when modeling for a 20 percent S-rate is about 70 percent—far lower than the warfighter’s requirements. This is the case because the time during which aircraft are unable to fly due to maintenance is also a factor, which the program projects will be about 10 percent. Figure 8 shows the difficulty that DOD will face in meeting the Secretary of Defense’s 80 percent mission capability target when planning for an S-rate of 20 percent given the time that is also required for maintenance. According to program and prime contractor documentation, DOD would need to model and fund the spare parts pool to achieve an S-rate of no higher than 10 percent in order to achieve requirements for aircraft performance, such as the mission capability target set by the Secretary of Defense and the services’ goals for air vehicle availability. Doing so would significantly increase the costs for spare parts. According to the prime contractor, in order to achieve a fleet-wide S-rate of 10 percent, the U.S. government would need to initially pay hundreds of millions of dollars to buy more parts for already-fielded aircraft. Costs would also increase on an annual basis—above the nearly $1 billion the U.S. services collectively paid in fiscal year 2018—to buy more parts each year. The current projected costs of F-35 sustainment are not affordable for the services. In 2018, DOD established constraints based on the military services’ future budget projections that indicate that DOD needs to reduce F-35 sustainment costs per aircraft per year by 43 percent for the F-35A, 24 percent for the F-35B, and 5 percent for the F-35C in order for the aircraft to be affordable for the services. DOD will be challenged to support this increase in annual costs for spare parts given its need to make significant cost reductions. Furthermore, as part of DOD’s fiscal year 2020 program budget review, DOD conducted modeling and analysis to project how various courses of action—such as increasing purchases of spare parts to compensate for how long it actually takes to repair parts or reducing aircraft production– would affect F-35 fleet performance. DOD’s analysis projected that if no additional actions were taken beyond what the U.S. services had already planned for and funded, F-35 aircraft performance would increase for a period of time. However, it would then worsen significantly with the growth of the fleet. Officials from the Office of the Secretary of Defense said that, as a result of this analysis, DOD is considering some additional investments to increase the availability of parts that would result in increased funding requirements for the U.S. services, but that as of January 2019, decisions were not finalized. They further said that their recent modeling and analysis efforts for the fiscal year 2020 program budget review did not formally consider additional investments to lower the planned S-rate to 10 percent as a course of action, but that this misalignment between the quantity of parts that DOD is planning to purchase and what is needed will hinder DOD’s ability to meet warfighter performance requirements. Supporting Recent F-35 Shipboard Deployments The F-35 program was not able to fill the Marine Corps’ afloat spares packages (packages of spare parts designed for aircraft stationed on ships) for the first F-35 deployments aboard the U.S.S. Essex and U.S.S. Wasp in 2018 in time to support those deployments. As a result, the F-35 program pulled spare parts from inventories at Marine Corps Stations Yuma, Arizona, and Iwakuni, Japan. Marine Corps officials stated that these actions reduced F-35 readiness in Iwakuni. Moreover, DOD may have limited options to increase spare parts availability for its operational fleet because of the way in which the program is currently structured to allocate parts. Within the F-35 program, the U.S. services do not have control over how F-35 parts are allocated, but rather share access to the parts along with the rest of the global fleet. The prime contractor is responsible for allocating parts to meet the requirements of all participants who share in the global spares pool. In response to parts shortages to date, Air Force and Marine Corps officials have said that the program has generally supported big events, such as the 2018 operational deployments of the U.S. services, by shifting parts to those units from the broader global spares pool (see sidebar). According to service officials, decisions to shift parts to different locations to support operational priorities could potentially be made by either a military service that owns those parts or DOD leadership within a legacy program. However, Office of the Secretary of Defense and program officials said that there is no mechanism within the current construct of the F-35’s global support strategy for program participants to optimize readiness for certain units by increasing the allocation of parts to those locations, short of deviating from existing program rules or contractual arrangements. As the size of the fleet and number of operational squadrons grow, the F-35 program will face increasing demands on its supply chain and competing operational priorities across participants that will likely make it more difficult for the program and the U.S. services to mitigate fleet-wide shortages of F-35 parts. GAO’s Standards for Internal Control in the Federal Government states that agencies should define objectives clearly to identify risk, including considering external requirements and internal expectations, and to design and implement activities to respond to those risks. DOD guidance on performance-based arrangements also states that performance-based logistics arrangements should be structured to deliver outcomes that are tied to warfighter requirements. Taken together, the current supply chain challenges and the issues related to how the program is planning for and allocating parts expose a significant gap between the F-35 aircraft performance targets the U.S. services need to achieve and what the F-35 supply chain is positioned to deliver within affordability constraints. DOD’s updated F-35 Life-Cycle Sustainment Plan identifies a number of actions needed to improve aircraft performance, such as those related to spare parts availability and repair capability. While the identification of such actions is a positive step, the plan also states that those actions do not take into account policy, program structure, or resource constraints, which could make them difficult to implement. Furthermore, DOD’s recent modeling efforts have already identified the need for some initial additional investments that could further strain the services’ budgets. Without a comprehensive review to determine what additional actions are needed to close the gap between warfighter requirements for aircraft performance and what the F- 35 supply chain is capable of delivering, taking into account also the need to reduce the sustainment costs of the F-35, DOD risks that its F-35 fleet may fall short of the capability needed to support its critical national defense missions in the future. DOD’s F-35 supply chain has provided spare parts to support the few F- 35 deployments that have occurred to date, including the following: U.S. Air Force deployment of 12 F-35A aircraft to Japan, November U.S. Marine Corps deployment of six F-35B aircraft aboard the U.S.S. Wasp, March—April 2018 (see figure 9); and U.S. Marine Corps deployment of six F-35B aircraft aboard the U.S.S. Essex, July 2018—February 2019. These units deployed with packages of parts to support the first 20 days of their deployment (that is, deployment and afloat spares packages), and then received replenishment parts from the broader global spares pool once their packages of parts were depleted. DOD officials generally characterized these deployments as operational successes and significant milestones for the F-35 program. In addition to these early deployments, the F-35 supply chain is also providing parts to activated U.S. and international F-35 bases in six different countries outside of the United States. DOD faces challenges in managing and moving parts to support a deploying and expanding global F-35 fleet. While the initial operational deployments have been successful and the program has established overseas F-35 bases in six different countries, these events have also highlighted several key risks that could hinder future F-35 fleet readiness. These risks are related to (1) the make-up of the afloat and deployment F-35 parts packages, (2) the prioritization process for distributing scarce parts among global F-35 participants, and (3) the F-35 program’s global networks for moving parts. DOD faces challenges in ensuring that the parts in its purchased afloat and deployment spares packages match the needs of deploying operational aircraft. According to Air Force and Marine Corps officials, ensuring that these parts packages are appropriately configured is of significant operational concern because units may be completely reliant on them while deployed to locations that the F-35 supply chain cannot yet readily support. The afloat and deployment spare parts packages are purchased according to a list of parts planned and paid for by an F-35 program participant at least 2 to 3 years in advance, aligning with the aircraft being purchased at that time and the best projections of what the demand for the parts will be. However, given the immaturity of the F-35 program, continued modifications to parts and aircraft can make such packages out-of-date by the time F-35 units are preparing to deploy. For example, Air Force officials told us that the spare parts packages for its November 2017—May 2018 operational F-35 deployment in Japan included parts that were not compatible with the aircraft with which they intended to deploy. Thus, the Air Force had to change its plans and deploy with older aircraft with less advanced capabilities that matched the parts in the package instead of the aircraft that best met their operational requirements. The Marine Corps faced similar challenges with its first shipboard deployments in 2018. Table 1 shows the number of parts and examples of parts in the Marine Corps’ afloat spares packages for the U.S.S. Wasp and U.S.S. Essex deployments that were not initially configured to be compatible with the Marine Corps’ deploying aircraft. Air Force and Marine Corps officials also said the quantity of parts within their parts packages were not fully reflective of the actual demands for certain parts, based on updated information about the reliability of certain parts and how frequently they needed to be replaced. In other words, the initially built packages did not have enough of the right parts to meet mission requirements. For example, Marine Corps officials said they were able to identify more than a dozen different parts in one of their afloat spares packages prior to deploying that were not provided in sufficient quantities because the program did not account for the actual fleet demand for these parts in its modeling for the afloat spares package. Air Force officials expressed similar concerns and said that they have had difficulty in getting information from the program that would enable the Air Force to assess whether there are enough of the right parts in its deployment spares packages relative to the actual demands for these parts. This is a concern for the Air Force as it prepares for its next F-35 deployment, because officials said that they cannot be sure that the package of parts with which they will deploy will have sufficient parts to support the deployment. The F-35 program does not have a process in place for changing out the parts within the afloat and deployment spares packages that are put on contract years before a deployment. Such a process is needed to ensure that the packages reflect the actual configurations of the deploying aircraft or updated demand projections for parts. Service and program officials said that such a process would need to include a review of the parts within the packages to ensure that they match deploying aircraft and aligning the funds to pay for any necessary updates or modifications to the parts, which could potentially cost tens of millions of dollars. F-35 program policy recognizes that the program may need to adjust the configurations or quantity of parts in the packages based on updated information, noting that such actions may necessitate contractual changes, but it does not specify the process for these adjustments. In our discussions with the prime contractor, program office, and military services, officials have lacked clarity regarding who is responsible for reviewing the parts in the package to ensure that they are appropriately configured and for determining whether additional contract actions or funding are needed to update the packages. In lieu of an established process to refresh these parts, service and contractor officials described an ad hoc and manual effort to review the packages prior to deployment. To address non-matching parts, contractor officials said that the program had to pull parts from the global and base spares packages to make exchanges. Officials said that this cuts into the parts that are available for the other F-35 units that rely on those packages, because the global and base packages are not stocked with the parts to support the deployments. For example, the program used 187 parts from the inventory at Marine Corps Air Station Iwakuni to backfill parts for the U.S.S. Wasp. The Marine Corps’ squadron in Iwakuni stated that this had a measurable effect on the squadron’s readiness to support its operational requirements, as reflected by lower availability of parts within their inventory to support broken aircraft. Specifically, during the time of the U.S.S. Wasp deployment, only about 46 percent of the critical parts (that is, parts needed to fix aircraft that cannot fly) that the squadron at Iwakuni needed were available in its inventory, and the squadron had to wait an average of about 12 days to receive these parts from off-base. As the F-35 fleet continues to expand and the number of operational deployments increases, military service officials said that these manual workarounds and the singular focus on ensuring that one unit has the appropriate parts to deploy will not be tenable. Program officials said that they have started a working group to look at options for addressing this issue, but they could not provide a timeframe or details about this effort. DOD guidance for risk management in acquisition programs states that defense programs must anticipate and address risks on a continuing basis, and suggests that programs implement processes that include risk identification, analysis, mitigation, monitoring, and planning. Further, the services have recognized that, to meet operational readiness objectives in a deployed environment, it is critical to have mechanisms ensuring that spare parts packages with which units plan to deploy are built to support the configurations and expected missions of the deploying aircraft, and have established guidance and processes to that effect. DOD also has a separate, ongoing initiative to determine whether using risk-based assumptions can produce a more efficient and effective mix of parts within deployment parts packages across a range of weapon systems, including the F-35. While this effort is nascent, it could potentially offer insights for the F-35 program to consider when reviewing the make-up of the F-35 deployment and afloat spares packages. Without a process for DOD to modify the F-35 afloat and deployment spares packages, to include reviewing the parts within the packages to ensure that they match deploying aircraft and accounting for updated parts demand, and without aligning any necessary funding for needed updates, the military services face risk that the parts that they have specifically purchased to meet their operational requirements will not be sufficient to do so. Uncertainty exists about how the program will prioritize scarce F-35 spare parts among global participants. The program has developed a set of business rules to govern the prioritization of scarce F-35 parts. The business rules are to differentiate between the relative significance of competing needs and create a structure to be responsive to customer requirements during both peacetime and war. These rules are critical to ensuring fair and transparent allocation of parts to all program participants, particularly given the significant shortages of spare parts throughout the F-35 program. Under these rules, F-35 units are assigned numerical designations based on the importance of their mission (that is, force activity designators), and their part requests are similarly assigned designations based on how important the part is to aircraft functionality (that is, urgency of need). Under these rules, the force activity designators of each unit and the urgency of need for each part request are combined to create an analysis that is applied to requests for scarce parts to determine which unit should receive the part. For example, according to such an analysis, a deployed F-35 unit that orders a part for an aircraft that cannot fly without that part would have priority over all other units. Conversely, an F-35 training unit that needs a part to replenish the inventory of parts on its shelves would have very low priority for the part relative to that of other units. See figure 10 for a general depiction of the prioritization scheme for F-35 parts. According to program and contractor officials, the prime contractor has been allocating parts according to these business rules, but these rules are not comprehensive. Officials from the Joint Staff, Office of the Secretary of Defense, program office, and military services cited a number of areas where the rules lack clarity and detail. For example, there is a lack of clarity around how force activity designations will be assigned and by whom. The business rules state that each unit’s force activity designation will be assigned by the participant’s national command authority, but they do not specify the process for doing so; provide for a clear role for the U.S. combatant commanders in the process; or specify the level of U.S. and international leadership required in order to make changes to this designation. In addition, stakeholders with whom we spoke said that the existing force activity designations do not provide for enough differentiation between types of activities or account for the unit’s unique mission requirements when determining how important a part is to aircraft functionality. For example, military units that are engaged in combat operations are assigned the same force activity designations as units that are forward-based to react to potential threats. These officials expressed concern that as the global fleet expands and more units are engaged in operations, this practice could lead to a situation in which too many units are a “priority” at any one time. Stakeholders have also raised questions about whether and how F-35 participants should be charged for increases in their force activity designations, as this matter is not addressed within the current business rules. Furthermore, the F-35 Product Support Manager has at times waived these business rules to support deployments and other activities, such as aircraft operational tests. For example, the Air Force unit that deployed to Japan in 2017 experienced significant readiness challenges because the business rules had established the replenishment of its spare parts package as a low priority relative to other competing demands for scarce parts. Air Force officials said that this contributed to its aircraft being unable to fly due to shortages of parts more than 30 percent of the time (cumulative over a month). According to Air Force and contractor officials, Air Force leadership then made a number of calls to the program office to request that its replenishment requirements be given higher priority. Subsequently, the F-35 Product Support Manager directed that the contractor deviate from the business rules to place a higher priority on the replenishment of the deployed unit’s parts package so that it could get parts faster. Service and program officials said that such deviations may be necessary to meet operational requirements, and that program leadership needs some flexibility in the business rules to make those decisions. According to program officials, the F-35 Product Support Manager has the authority to issue waivers to the business rules, but the business rules do not clearly grant this waiver authority to the Product Support Manager, or address how and when such waivers should occur. Stakeholders have been raising some of these concerns for several years. For example, the Office of the Secretary Defense and the Joint Staff developed related position papers that identified gaps in the business rules. Officials from these offices said that the papers were sent to the program office in 2014 and early 2017, respectively. In response, the F-35 program established a working group in May 2018 to begin revising the business rules. As of January 2019, program officials said that the revised business rules were undergoing internal review, but the date for completion was not yet determined due to potentially lengthy timeframes associated with obtaining formal approval through the F-35 governance process. This ongoing effort is promising, but the specific action items that the working group was tasked with incorporating into the business rules do not clearly address some of the areas of concern raised by stakeholders. For example, these action items do not include the issue of deviations from the business rules. DOD directs its components to comply with DOD’s established materiel management guidance, which outlines DOD policy, assigns responsibilities and specifically provides procedures for how parts and materiel should be prioritized for responding to customer supply chain demands for all DOD components, including outlining the application of force activity designators and the role of the combatant commanders. The F-35 program’s existing business rules incorporate many aspects of this standard DOD prioritization guidance, but they are not fully aligned with this guidance. For example, DOD’s standard process outlines the use of five potential force activity designators, while the F-35 program provides for only three different designations. Additionally, Standards for Internal Control in the Federal Government states that agencies should design control activities to achieve objectives and respond to risks, including implementing control activities through policies. U.S. service and international officials said that, as the fleet and competition for spare parts increases, they are concerned that participants may try to manipulate the system due to the lack of clarity within the existing rules. Without ensuring that the revisions to its business rules for the prioritization of scarce F-35 parts across all program participants define stakeholder roles and responsibilities, the process for assigning and arbitrating force activity designations, and the manner in which deviations from the business rules will be conducted, the F-35 program may face challenges allocating parts to support competing U.S. and international warfighter requirements. Further, F-35 program participants may lack confidence in the equity of decisions regarding scarce parts that affect their operational requirements. DOD is now moving F-35 parts around the world, but its global networks for doing so are immature and there is risk that they will not be fully capable to support an expanding fleet. The F-35 program has a growing number of U.S. and international participant bases outside of the United States and is providing supply support from its global spares pool for an increasing number of operational deployments. For its supply chain construct to work as intended, F-35 parts must be able to move freely and efficiently among U.S. and international program participants, suppliers, and repair facilities, regardless of the country or company of origin. The program has projected that F-35 parts could potentially be moved on 132 different paths between participating countries (for example, Italy to United Kingdom, Italy to Norway) and 2,162 paths between F-35 sites (for example, a warehouse in the Netherlands to a base in Norway). This will require the program to establish strategically located warehouses, synchronize global distribution networks, and navigate a complex web of import and export activities and international weapon control laws. However, the envisioned global network is not yet in place. For instance, regional warehouses planned for the Netherlands and Australia are not expected to reach initial operational capability until, at the earliest, late 2019 and 2020, respectively. Furthermore, the program is still working to establish functional shipping networks and locations at which to receive parts. It also does not have mechanisms in place to support the range of required import and export activities. Spare parts are instead being moved under a less efficient system, with the parts originating from and returning to the United States before being delivered to an international program participant. Figure 11 compares a depiction of the program’s intent for the future global network for moving F-35 parts with the existing “hub-and-spoke” network. The immaturity of the global network has contributed to long wait times for parts for the U.S. and international F-35 squadrons that are deployed or permanently based overseas. The 2018 F-35 sustainment contract establishes minimum and objective targets for customer wait times across the F-35 fleet. The targets are the same regardless of whether the aircraft are located inside or outside of the United States, thus reflecting the intended global nature of the network. However, customer wait times for parts for units located outside of the continental United States have been significantly higher than those for units located inside of the continental United States, as shown in figure 12. Unless otherwise noted, the data are inclusive of customer wait times for both U.S. and international participants. Officials from Marine Corps, Air Force, and international F-35 squadrons that were based or deployed overseas in Japan and the United Kingdom described long wait times of up to 17 days—well outside of the 6-and 10- day customer wait-time metric ranges for critical parts—to receive available parts overseas that have degraded their readiness. They cited several reasons for these delays, such as export and import licenses not being in place, delays in customs, inefficient routing or processing of parts, and ineffective commercial freight forwarders. For example, Air Force and contractor officials said that it was initially taking parts up to 14 to 16 days to reach the deployed Air Force unit in Japan using a commercial shipper, which was hurting the unit’s readiness. According to DOD and contractor officials, these concerns drove the prime contractor to start shipping parts via military air, which subsequently decreased customer wait times significantly. However, these officials said that the program did not have the appropriate contracting and funding mechanisms in place to utilize military air and had to return to using a different commercial shipper. The F-35 program’s plan for full establishment of the global networks for moving parts is not complete. Program officials and contractor officials told us that planning for this network is 3 to 4 years behind the need because the program was more focused on producing the aircraft than on sustainment. Prime contractor officials also said that they did not realize the complexity of setting up the network, which will require them to establish export and import authorizations in every country and to work through the Department of State to establish export licenses. In addition, the construct necessitates that each of the international participants takes actions within its own government to ensure that the appropriate arrangements are in place, such as obtaining waivers for taxes, tariffs, and duties, or pursuing any necessary changes to its own government’s laws. The F-35 program initiated its focused planning for this network in 2018, with the establishment of a working group tasked to develop plans for implementing the network. In January 2019, the F-35 program issued a high-level strategy that provided some limited information on the program’s objective and key principles for the network. It also indicated that a forthcoming F-35 program instruction would provide a framework for executing the strategy, but it did not have a timeline or details for the completion of this instruction. Also in January 2019, DOD selected the U.S. Transportation Command and the Defense Logistics Agency as the entities responsible for the global transportation and distribution networks for F-35 parts—a transition that is expected to occur over the next 12 to 24 months. According to Department of Defense documentation, existing U.S. Transportation Command and Defense Logistics Agency networks are already in place to support much of the required F-35 global parts movements, particularly for U.S. units and foreign military sales customers. However, these organizations will still be reliant on the F-35 program to establish the necessary licenses and legal frameworks for the movement of parts between partner countries. The program has established a target date of September 1, 2021 for full operational capability of the network, at which point spare parts are intended to be able to be moved freely throughout the F-35 enterprise. However, the program does not yet have a detailed plan with clear requirements and milestones or an integrated schedule to move the network from initial operational capability to full operational capability. Program officials stated that they believe this date is achievable, due to the increased emphasis on developing the network among all program participants. However, there are risks to the program’s planning effort. Beyond the complexity of the network, the F-35 program office and contractors do not control all elements needed to support the successful implementation of the network. Specifically, each international partner is responsible for establishing the necessary legal framework in its own country to support the network, which can be a lengthy process. Program officials further noted that other international participants have national laws or have made decisions that are not conducive to the free flow of parts throughout the global network. F-35 program policy provides some provisions to address non-conformance by partners—for example, stating that partners will be responsible for any taxes or tariffs charged to the program by their own countries. However, program officials said that the mechanisms to manage any such deviations will be complex to implement and are still being developed. Our prior work on acquisition management has identified a number of key program management practices that can improve program outcomes if implemented, such as clearly establishing well-defined requirements and developing realistic schedules that include risk analysis. DOD guidance related to managing risk in acquisition programs also states the importance of program managers taking actions to identify, manage, and mitigate programmatic risk, which can either be intrinsic to the program or arise from inadequate planning. The F-35 program’s recent focused efforts in this area are positive steps, but its planning efforts still lack detail about how the network will be fully implemented. Furthermore, the schedule, planning, and risks associated with this delayed global network are not addressed in DOD’s recently updated F-35 Life Cycle Sustainment Plan. Without completing a detailed plan for the establishment of the F-35 program’s global network for moving parts that outlines clear requirements and milestones to get the network to full operational capability, and includes mechanisms to identify and mitigate risks of delays or gaps in the global network, the program cannot ensure that its supply chain will support U.S. and international program participants as intended. Furthermore, delays or gaps in in the establishment of the envisioned global network will likely result in increased costs associated with additional travel segments and delays to the warfighter in receiving spare parts that could hurt the operational readiness of the global F-35 fleet. DOD cannot fully account for F-35 spare parts within the supply chain and their associated costs. Specifically, the department does not have records indicating how many F-35 spare parts it has purchased, or where they are all located. In addition, DOD does not have comprehensive cost information for individual F-35 spare parts, and the military services cannot track the funds that they have spent on F-35 spare parts to the actual parts purchased by the program office on their financial statements and supporting documentation. Accountability of government property, such as F-35 spare parts, facilitates financial audits by providing the necessary documentation to ensure the accuracy of transactions for government property and contracted services. Congress required the Secretary of Defense to ensure that an external audit be performed on DOD’s financial statements for fiscal year 2018, and to submit such audit to Congress no later than March 31, 2019. Congress directed this audit, in part, to help improve the accuracy and reliability of management information on DOD’s mission- critical assets—such as F-35 spare parts—and services for which they contract. Subsequently, DOD completed its first consolidated, department-wide, full financial statement audit in November 2018. The DOD Office of the Inspector General reviewed the department-wide financial statements and identified 20 material weaknesses—that is, serious problems with DOD’s internal processes that hamper its ability to reasonably assure that its financial reporting is reliable—including processes related to accountability for government property in the possession of contractors and the accuracy and completeness of financial statements. DOD cannot fully account for its spare parts within the F-35 supply chain, including the quantity of all the spare parts it owns and where they are located. The prime contractor manages the F-35 supply chain and the movement of all F-35 parts across the F-35 enterprise to meet warfighter needs. DOD initially did not intend to own the F-35 parts, but in 2012 the F-35 program’s executive steering board issued a decision memorandum declaring the F-35 parts in the global spares pool to be titled to the U.S. government when they are not installed on an aircraft. However, program officials told us that DOD did not develop a corresponding plan to maintain accountability over the parts that it already owned or would purchase in the future. According to program officials, this is due in part to property accountability not being a priority for the program in its effort to field aircraft. This is evidenced by the number of staff within the program office dedicated to this mission; program officials said that until recently there was only one government official at the program office overseeing property accountability for the F-35 system. In order to maintain accountability for government property, such as the spare parts within the F-35 supply chain, DOD guidance requires that DOD components establish and maintain a physical inventory control program for assets within the DOD supply chain to serve as a key internal control for providing information to inform inventory financial statements. Defense Contract Management Agency officials also told us that in order to improve F-35 readiness and decrease costs, DOD must have an understanding of the F-35 spare parts it owns, where those parts are located, and how those parts are being used to support the weapon system. However, the F-35 program has not consistently followed DOD guidance for property accountability. For example: As of December 2018, the program office had not populated an accountable property system of record with data for its F-35 parts. DOD components are required to establish and maintain accountable property systems of record for property that DOD components own and manage. An accountable property system of record is required to contain information such as cost, location, and custodial ownership data for property, including individual parts, that meet certain criteria, and to provide a comprehensive log of transactions that can be audited. Such a system would allow the F-35 program office to have asset visibility for spare parts within the F-35 supply chain. The program office has identified a database to use as its accountable property system of record, but DOD officials stated that the program office does not have the data necessary to populate it. According to program officials, the prime contractor keeps some of the required data in proprietary databases to which the program office does not have access. In addition, DOD officials told us that the program office is working through some limitations that need to be addressed with the system the program office has chosen to be its accountable property system of record in order to properly maintain data records. The program office has not fully identified which spare parts the prime contractor is required to enter into DOD’s Item Unique Identification registry (hereinafter referred to as DOD’s central registry for government property). In addition to component-specific accountable property systems of record, DOD’s central registry for government property is DOD’s primary data source for government furnished property, and it is intended to provide department-wide asset visibility for all government property and links with financial and accountability systems in order to maintain accountability over the assets DOD owns. DOD guidance states that agencies are to require contractors to report government furnished property in DOD’s central registry for government property. DOD guidance also states that DOD agencies are to identify which assets require unique item-level traceability. However, the program office has not clearly defined for the prime contractor all F-35 spare parts that should be entered into DOD’s central registry. As a result, DOD officials said the prime contractor is not entering in information about all required parts. Moreover, a property accountability official said that the prime contractor is not consistently entering F-35 parts into DOD’s central registry when the parts are delivered, because the prime contractor may delay entering information into DOD’s central registry until all items associated with a specific contract line item have been delivered to DOD. This official also said that there are some contract line items dating back to the first production lot, which delivered aircraft in 2011, that remain open, and thus there are potentially thousands of F-35 parts that are being used within the global spares pool that have not been entered into the registry, thereby impeding DOD’s visibility over these parts. DOD has not established a program policy that explicitly defines how it will maintain accountability of F-35 spare parts in accordance with DOD guidance. According to program officials, DOD has made some recent progress to address accountability issues, such as taking steps to bring contracts into compliance with property accountability regulations and increasing the number of staff focused on property accountability within the F-35 program office. However, DOD faces continued challenges in accounting for F-35 assets. In the absence of a program policy, the program lacks clarity on how to categorize assets and which property data the contractor is required to provide for those assets, how to implement policies and regulations, and how to define prime contractor roles and responsibilities. For example, F-35 contracts contain Federal Acquisition Regulation clauses that convey requirements for the prime contractor related to the accountability of government furnished property, including specifying the data that the contractor must maintain and provide to DOD. However, DOD officials said that the F-35 program office has not contractually established which items—including spare parts—are government furnished property, which has made it difficult for the program office to hold the contractor accountable for those required functions. As a result, the contractor has disputed which items should be considered as government furnished property, which has implications for how the prime contractor maintains accountability and provides data for F-35 spare parts it manages. Property accountability officials at the F-35 program office have developed a draft directive that seeks to address the factors currently impeding the program from being compliant with property accountability guidance by clarifying roles and responsibilities within the program office for maintaining accountability of all government furnished property and pooled assets, including the F-35 spare parts in the supply chain, and defining prime contractor responsibilities for managing these items and providing data to the program office for them. Officials told us, however, that the draft directive is undergoing internal review, and that its timeline for approval and implementation has not been established. Program officials said they are also in the process of developing a program instruction that may provide general procedures for implementing the policies that will be established in the directive. Furthermore, while the draft program directive defines property accountability goals for the F-35 program, it does not detail the actions the program office will take to achieve these goals. The program office will face challenges that may impede its ability to achieve the goals of the draft directive, both retroactively and prospectively, for the billions of dollars in F-35 spare parts for which it currently cannot fully account. For example, DOD officials said that the costs for the prime contractor to obtain the data required to meet DOD’s requirements for property accountability will likely be high, as the prime contractor does not centrally maintain all the data, nor do they maintain the data in a readily usable format for property accountability purposes. The contractor has estimated that more than 450,000 hours of labor could be necessary to provide the data. Program officials also acknowledged that the successful implementation of the draft directive is dependent upon support from program office leadership to ensure that its guidance is followed by both program officials and the prime contractor. However, according to these officials, the program has not historically prioritized property accountability in negotiations with the prime contractor because the program office has been focused on the production and fielding of aircraft and developing contracts to which the prime contractor will agree. Standards for Internal Control in the Federal Government states that agencies should define objectives to identify risk, and to design and implement control activities to respond to those risks. These standards also state that without a strong tone at the top to support an internal control system, the entity’s risk identification may be incomplete, risk responses may be inappropriate, control activities may not be appropriately designed or implemented, information and communication may falter, and results of monitoring may not be understood or acted upon to remediate deficiencies. DOD’s recent efforts related to property accountability are positive, but DOD stakeholders have raised concerns about issues related to property accountability within the F-35 program dating back to 2012 that have not been resolved, such as the program’s lack of a populated property system of record. As the fleet expands and the number of spare parts in the supply chain continues to grow, the program office will only continue to face increasing difficulty in obtaining accountability over its F-35 assets if it does not address these challenges. To address the scope of these challenges, DOD will need to establish a unified approach that provides clarity on how to categorize these assets, implement policies and regulations, and define prime contractor roles and responsibilities. Without developing a policy that clearly resolves these issues and defines how the F-35 program will maintain accountability for spare parts within the supply chain that is consistent with DOD guidance—and identifying the steps that it will take to implement it retrospectively and prospectively, such as how the program will obtain the necessary data from the contractor— DOD cannot ensure that it will be able to obtain and maintain comprehensive accountability and visibility over spare parts within the F- 35 supply chain. Moreover, without an understanding of the assets it owns and how those assets are being managed by the prime contractor, DOD cannot ensure that the prime contractor is providing sufficient readiness for its most expensive weapon system at a reasonable cost. DOD cannot identify individual costs for each F-35 spare part, nor can the military services track the funds that they have spent for the use of F-35 spare parts to the actual parts purchased on their financial statements and related documentation. According to contract administration officials, the ability to track costs and assets is also critical to understanding and improving F-35 fleet performance. DOD does not have comprehensive cost information for individual F-35 spare parts. DOD purchases a high volume of spare parts across several contracts each year. According to program documentation, DOD was appropriated more than $960 million for F-35 spare parts in fiscal year 2018 alone (see sidebar). DOD does not have a consistent, methodical process to identify and track the costs of individual F-35 spare parts, which would typically be done through the purchase contracts for the parts. However, the F-35 contracts do not identify the individual parts or their costs. Instead, these costs are aggregated under broad contract line items, such that individual pricing for spare parts cannot be determined. For example, the annual sustainment contract for fiscal year 2018 aggregates the costs to repair and replace spare parts for F-35A aircraft under one contract line item totaling $276 million. The contracts and related documentation do not specify how the money will be distributed among costs for repair or replacement, nor do they specify how many spare parts the contractor will purchase and at what cost. Program officials said that their system for contract management has limitations that make it difficult to separate individual F-35 parts into their own line items. Since those costs are not being specifically provided in the contracts, program officials said that DOD has relied upon several ad hoc, manual workarounds in an attempt to obtain such data for the thousands of F-35 spare parts it owns, but these efforts are not comprehensive. For example, a program official said that they are obtaining cost information from the inspection and receiving forms accompanying deliveries of F-35 spare parts and then manually entering these cost data into attachments to the sustainment contracts. However, DOD officials said that the inspection and receiving forms for deliveries of F-35 spare parts are often not being entered into the registry until years after the parts are delivered, because such forms are not required until the delivery of all parts purchased under the same contract line item are complete. Furthermore, DOD officials said that this process is not an effective long-term solution for maintaining cost data of the billions of dollars in F-35 spare parts that DOD owns, because data entered in the program’s contract management system through manual workarounds do not automatically link to the program office’s other data systems. Program officials said that such linkages are necessary to maintain proper accounting of F-35 spare parts, as cost data constitute one of the required data elements for an accountable property system of record. Similar to the challenges that DOD faces with property accountability, program officials said that DOD faces significant hurdles in obtaining cost data from the prime contractor for individual F-35 spare parts because the contracts have not been written to require those data from the outset of the program. According to program officials, the program office has attempted to negotiate for cost data for F-35 spare parts, but the attempts have not been successful because of the high price the prime contractor would have charged the government for these data. DOD guidance states that understanding program costs, such as those for F-35 spare parts, is critical to both achieving desired performance and supporting financial audits. Specifically, DOD guidance states that the government should clearly understand program costs in order to have effective performance-based arrangements. Along these lines, we have previously reported that DOD’s limited understanding of the actual sustainment costs of the F-35 system will hinder its ability to accurately determine how much fleet performance should cost under performance- based contracts, thus putting DOD at risk of overpaying the prime contractor while not receiving the expected level of sustainment support. Additionally, DOD guidance requires that DOD agencies assign dollar values for spare parts in financial accounting systems. Without a methodical process for consistently obtaining comprehensive cost information from the prime contractor for individual F-35 spare parts, the program office will not be able to maintain financial or property accountability over these parts in accordance with DOD guidance. Furthermore, DOD will continue to face challenges in developing a complete understanding of the costs for the F-35 system, which will impede its ability to effectively negotiate with the prime contractor for sustainment support and to improve readiness of the expanding F-35 fleet. The military services cannot track the funds that they have spent for the purchase of F-35 spare parts to the actual parts on their financial statements and related documentation due to the lack of an established accounting methodology for the parts within the global spares pool. Under this global spares pool construct, the military services and international partners each pay for access to the common pool of spare parts instead of owning the physical parts themselves. However, there is no established accounting methodology for defining how to track funding to the spare parts such that the military services can properly report assets on financial statements. DOD’s Financial Management Regulation requires that DOD agencies—such as the military services—account for all spare parts they purchase for accountability and financial reporting purposes. According to DOD officials, the F-35 program and the DOD Comptroller have been working to develop a policy that provides such guidance since 2015, but it has not yet been finalized and the timeline for completion is unclear. Specifically, program officials said that they are waiting for the DOD Comptroller to finalize a memorandum that would identify the DOD component responsible for maintaining financial accountability of the F-35 spare parts in the global spares pool. According to DOD officials, the memorandum would include an attachment that defines a methodology for tracking funding contributed by the military services and international partners to F-35 spare parts. A draft of this memorandum has laid out a possible methodology to maintain financial accountability for the spare parts within the global spares pool that includes identifying the program office as the DOD component responsible for financial reporting for F-35 parts, but a program official said that the DOD Comptroller has not yet completed this memorandum because the DOD Comptroller is reconsidering the proposed approach. DOD Comptroller officials said that they are reconsidering the proposed approach based on input received from independent public accountants who performed the services’ financial statement audits, to consider having the Department of the Navy or the Air Force, rather than the program office, be the reporting entity for F-35 parts. Without a DOD Comptroller-approved methodology for the services to account for the funds they have spent on F-35 parts within the global spares pool on their financial statements, DOD will be hindered in its efforts to comply with financial improvement and audit readiness requirements, provide supporting details for its financial statement transactions, and render accurate cost information for DOD management, Congress, and others stakeholders to use in assessing and managing program costs and other financial activities associated with the F-35 program. We previously reported that F-35 sustainment costs are not fully transparent to the military services and recommended that DOD should take steps to improve communication with the military services about how the F-35 sustainment costs they are being charged relate to the capabilities received. Furthermore, discrete cost information and an ability to account for funds spent would help DOD in its efforts to decrease costs and make one of its most expensive weapon systems more affordable. Challenges related to readiness and costs—including those we have discussed in this report—are driving the Office of the Secretary of Defense and the services to take actions that diverge from the established F-35 sustainment strategy. These actions indicate a potential shift in DOD’s intent for F-35 supply chain management and a growing desire for more direct involvement by the military services and access to program information from the prime contractor. reliant on the program office for information about system performance and costs. Furthermore, according to Office of the Secretary of Defense and service officials, many of the military services’ sustainment organizations that provide supply and maintenance support to other platforms have had almost no role in the planning for and establishment of sustainment capabilities or ongoing sustainment support for the F-35. Of these common items, more than 6,000 100,000 demands for these common items, 435 of which had impacts on fleet readiness. In April 2018, in a departure from the strategy and structure of the program and at the direction of the Assistant Secretary of Defense (Logistics and Materiel Readiness), the Defense Logistics Agency and the military services’ supply and sustainment organizations initiated planning efforts to develop an option for organic—that is, DOD-managed—supply chain management support that would include increased roles for the services’ supply organizations and the Defense Logistics Agency in assuming responsibility for F-35 supply chain management. In support of this effort, these organizations have begun to develop notional plans to provision an organic supply chain for F-35 aircraft, which includes determining how many parts are required to support the system and how they can be procured. In addition, the Defense Logistics Agency has begun to catalogue a limited portion of F-35 consumable parts from production lots 6 and 7 into DOD’s supply system (see sidebar). However, officials from the Office of the Secretary of Defense and the Defense Logistics Agency said that this initial cataloguing effort only includes the level of detail necessary to support disposal of the parts, and that more comprehensive cataloguing would require DOD to have access to significantly more technical data than are currently available. Prior to this effort, parts used on F-35 aircraft were not tracked by DOD in its logistics information systems. Officials from the Office of the Secretary of Defense said that there are multiple reasons behind DOD’s recent effort to develop an option for DOD-led, organic supply chain management, including DOD’s need to significantly reduce sustainment costs and improve readiness. For example, according to DOD officials, DOD’s early cataloguing efforts have identified more than 7,300 F-35 consumable items that are common to other DOD platforms. Defense Logistics Agency officials said that they are actively working with the program office and prime contractor to identify opportunities for the program to leverage the parts that are already on DOD’s shelves. In the longer term, identifying common parts could potentially allow DOD to directly procure them at a lower cost rather than through the prime contractor, and thereby provide economies of scale across other aviation platforms. Furthermore, the prime contractor and F-35 Joint Program Office have not been able to deliver the supply chain performance that the services need under the current sustainment strategy and structure, as discussed earlier in this report. According to an official from the Office of the Secretary of Defense, DOD is supposed to have a viable back-up plan for contractor logistics support under performance-based logistics contracts, in case the contractor cannot meet the government’s performance requirements. Prior to the ongoing effort, DOD did not have such a plan. Similarly, DOD guidance on performance-based agreements states that robust performance-based logistics solutions include appropriate criteria to cease the arrangement if necessary in order to manage risk. DOD officials involved in the cataloguing and provisioning efforts described a long-term (5 to 10 years) and phased approach to the potential development of DOD-led supply chain management capabilities for the F-35 that would require major changes to the F-35 program structure and contracts. It would also require DOD to obtain significant amounts of technical data on F-35 parts from the manufacturers of those parts (see sidebar). DOD has submitted a request to the prime contractor for a proposal regarding supplying the data necessary to provision an organic supply chain and to catalogue all F-35 parts into DOD’s supply inventory, but as of October 2018, DOD officials said that the prime contractor had not yet provided the costs of these data. Officials from the Office of the Secretary of Defense told us that DOD had initially planned to negotiate for these data as part of the annual sustainment contract for fiscal year 2019, but that the prime contractor had cautioned that this could delay the awarding of the sustainment contract because of the complexity around the data negotiations. Officials said that there were also questions about the type of funds that should be used for the acquisition of these data (that is, procurement or operations and maintenance), and whether some data would need to be directly procured by DOD from the original equipment manufacturers. The lack of data from the contractor to support competition in the F-35 supply chain and DOD’s understanding of the costs and performance of the system has long been a challenge, as we have previously reported. In September 2014, we recommended that DOD develop an Intellectual Property Strategy, to include identification of all critical technical data needs and associated costs. Further, in October 2017, we recommended that prior to entering into multi-year, fixed-price, performance-based contracts, DOD should ensure that it has sufficient knowledge of the actual costs of sustainment and technical characteristics of the aircraft after baseline development is complete and the system reaches maturity. DOD concurred with both recommendations but has not yet implemented them. In addition, ongoing dialogue among stakeholders within the Department of Defense demonstrates a growing desire for more direct military service influence and access to information within the F-35 program. In 2018, the Secretary of Defense directed the U.S. military service chiefs to correct the F-35 parts shortages and to be agents of change in pursuing 80 percent mission capability for the F-35 aircraft. In a September 2018 memorandum responding to the Secretary of Defense’s direction to address F-35 parts shortages, the Air Force Chief of Staff, the Chief of Naval Operations, and the Commandant of the Marine Corps raised concerns about the program’s inadequate supply chain and repair networks and reported on the funding that the services, as customers, provided to the Joint Program office to improve delivery of spare parts and accelerate depot maintenance capability. Furthermore, officials whom we interviewed from each of the military service headquarters expressed frustration with the current sustainment construct of the F-35 program in which they pay large sums of money for less-than-required readiness outcomes but have minimal influence on actions being taken to improve readiness and limited visibility into supply chain modeling and data to support their operational decisions. DOD has not yet determined the actions and investments needed to support the F-35 supply chain in the future, because the department has not charted a clear strategy for F-35 supply chain management. There is a tension between two distinct sustainment concepts—the official contractor logistics support construct and DOD’s current effort to have greater involvement in supply chain management—and F-35 program officials said that the program is caught between the two. In October 2018 DOD issued an updated F-35 Acquisition Strategy, but it did not clearly outline a shift in supply chain management. The new strategy includes references to the potential for increased organic support of the supply chain in the future—but does not provide details about the actions or timelines necessary to support this—while also reaffirming the current sustainment strategy of contractor logistics support for supply chain management. In addition, while the new strategy states the intent to support supply chain cataloguing and provisioning efforts, it does not provide detailed information regarding the investments in technical data necessary to support these efforts. In January 2019, DOD issued an updated F-35 Life-Cycle Sustainment Plan, which highlighted the absence of the technical data to support provisioning and cataloguing as a gap. The plan stated the intent to have all cataloguing and provisioning data available to the services by the end of fiscal year 2024. However, the plan did not provide details regarding how the data were to be procured or address DOD’s future strategy for supply chain management. According to F-35 program, Office of the Secretary of Defense, and Air Force officials, DOD has to provide clear and consistent direction regarding its intent for F-35 supply chain management in order to guide investments in technical data, negotiations with industry, and program actions. In particular, F-35 program officials said that DOD’s mixed messages about supply chain management have led to inefficiency as the F-35 program tries to support both the formal, current strategy and initiatives driven by the informal shift toward more DOD involvement in F- 35 supply chain management. According to program officials, the Product Support Manager organization at the F-35 Joint Program Office was structured for management of a program in which the primary contractors would be providing comprehensive contractor logistics support for the life of the program, and it has not grown in size as the fleet has grown. Furthermore, many of the positions at the program office that are critical to establishing and managing sustainment and supply chain capabilities are unfilled, even as the program office is taking on new responsibilities as Hybrid Product Support Integrator. For example, as of September 2018, Of the 16 positions on the product support maintenance team, which includes depot planning, three were vacant. Of the seven positions on the product support supply chain management team, two were vacant. As of January 2019 program officials said that the number of vacancies had grown to four of seven positions. Of the 42 positions in the directorate of sustainment strategy, 11 were vacant, including the lead roles for strategic planning and risk management and scheduling for the global support solution. In other cases, the numbers of staff dedicated to complex planning efforts are limited or have experienced frequent turnover. For example, officials said that there are only two officials within the program office dedicated to planning for the establishment of the program’s delayed global networks for moving parts, and the lead role had changed four times in a year. Moreover, program officials said that they are inundated with requests for data and information from the Office of the Secretary of Defense and the U.S. military services, which they partially attributed to the informal shift in the program’s strategic intent for sustainment, and to scrutiny related to sustainment performance failures. Officials said that the time spent in responding to requests for data is hindering their ability to focus on long- term actions to improve sustainment performance. The lack of clarity about the future F-35 sustainment strategy could also increase the risk perceived by industry, thus driving up tensions and potential costs in contract negotiations. Program officials said that the increasing technical data requests sent to the prime contractors to support DOD’s provisioning and cataloguing efforts signal to industry a potential change from the acquisition strategy of contractor logistics support for supply chain management. According to Hybrid Product Support Integrator officials, mixed messages about the F-35 program’s future supply chain strategy could make manufacturers reluctant to invest in increasing their capacity to produce new parts and to repair parts, if they do not have confidence in the scope of future business to warrant such investments. Many options for F-35 supply chain management are available to DOD on a spectrum ranging from full contractor logistics support to DOD-led supply chain management or a blend thereof, depending on the aircraft system or subsystem. DOD guidance for program managers states that a sound program strategy requires understanding and clarity of the program’s desired outcomes, and the plans and resources necessary to achieve those outcomes. Furthermore, federal internal control standards demonstrate the necessity of programs defining a clear strategy in order to support program actions. Specifically, the standards state that management should define objectives clearly so that they are understood at all levels of the organization, to include defining what is to be achieved, who is to achieve it, how it will be achieved, and timeframes for achievement. Without clearly defining its strategy for how it will manage the F-35 supply chain in the future and updating key strategy documents accordingly, DOD will continue to face uncertainty about how F-35 sustainment support will be provided over the system’s life cycle and the actions and investments needed to ensure that support. Such uncertainty could further hinder the program’s efforts to improve supply chain performance and reduce costs. The F-35 aircraft, with its advanced warfighting capabilities, is a critical component of the National Defense Strategy. However, DOD will need to overcome substantial supply chain challenges for the aircraft to perform its expected role. Current F-35 performance continues to fall short of warfighter requirements, largely due to spare parts shortages and delays in the development of key repair capabilities. Simply purchasing more F- 35 parts without other trade-offs may not be a viable long-term solution for DOD, given the steep reductions in sustainment costs that the military services have recognized are needed to make the aircraft affordable. These complex problems necessitate a comprehensive review by DOD to determine what actions should be taken to close the gap between warfighter requirements and the capabilities that the F-35 supply chain can deliver. Absent such actions, DOD risks that the F-35 will not be able to conduct the full range of intended missions. The military services are integrating the F-35 into their operations with recent deployments and the establishment of F-35 bases overseas, but these events have also highlighted key risks for DOD in how it is managing and moving aircraft parts around the world. If not addressed, these risks could hinder the readiness of the global fleet. To date, DOD has been able to mitigate some of these risks by placing singular focus on ensuring the success of early F-35 deployments, but this will not be possible with the rapid expansion of the fleet in the next few years. Specifically, without a process and funding to make changes to the spare parts within their afloat and deployment spares packages to ensure that these match their needs, the military services risk not meeting operational requirements during future deployments. Fleet-wide spare parts shortages are also putting the F-35 program’s process for prioritizing scarce F-35 parts to the test. Absent comprehensive business rules, the F-35 program could face challenges in transparently allocating parts to support competing U.S. and international requirements. Further, because the F-35 program did not fully recognize the complexity of establishing a global network for moving F-35 parts, this network is now several years behind schedule. Without a detailed plan that includes clear requirements and milestones to fully establish the network, as well as mechanisms to identify and mitigate the risk posed by any gaps or delays, DOD cannot ensure that it will be able to take the network from concept to reality so that F-35 participants do not experience long wait-times for parts in order to fly their aircraft. Moreover, in its rush to field aircraft and its heavy reliance on the prime contractor, DOD has not focused on property and financial accountability of F-35 spare parts. Simply put, DOD does not have records of all the F- 35 spare parts it has purchased; where those parts are located; and how much the military services paid for them. Until DOD establishes a policy that clearly defines how the F-35 program will maintain accountability for spare parts within the supply chain and lays out the steps that it will take to implement that policy, DOD will continue to lack critical visibility of F-35 assets, which is necessary to hold the prime contractor accountable for providing sufficient readiness at a reasonable cost. Additionally, without a process to consistently obtain comprehensive cost information from the prime contractor for F-35 spare parts, DOD will not have a full picture of F-35 costs, which could impede its ability to effectively negotiate with the prime contractor for sustainment support and to improve readiness of the expanding F-35 fleet. Further, absent a DOD Comptroller-approved methodology for the military services to record on their financial statements the funds spent on F-35 parts, DOD will be hindered in its efforts to comply with financial improvement and audit readiness requirements. As a result, DOD will not be able to assure the taxpayer that it fully understands how funds have been spent on this costly weapon system. Finally, from the start of the F-35 program, the U.S. military services have been largely reliant on the prime contractor to manage the F-35 supply chain and to support their operations, with oversight from the program office. However, the Office of the Secretary of Defense and the services have grown dissatisfied with the program’s inability to meet their readiness requirements and reduce costs, and they have begun to take actions that indicate the potential for a significant shift in DOD’s F-35 sustainment strategy that would have far-reaching implications for the program. This shift, if fully implemented, would give more control of the supply chain to the federal government, but it also would run counter to the way in which agreements with industry and international participants have been constructed. Until DOD clearly defines its strategy for managing the F-35 supply chain in the future—to include any additional actions and investments necessary to support that strategy—the F-35 program will lack the certainty and unity of effort necessary to meaningfully improve supply chain performance and reduce costs. We are making the following eight recommendations to DOD. The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment, together with the F-35 Program Executive Officer, the Secretaries of the Air Force and Navy, and the Commandant of the Marine Corps, conducts a comprehensive review of the F-35 supply chain to determine what additional actions are needed to close the gap between warfighter requirements for aircraft performance and the capabilities that the F-35 supply chain can deliver, in light of the U.S. services’ affordability constraints. Potential actions could include adjustments to the quantities of parts DOD is planning to procure, or developing a mechanism for providing increased availability of parts to operational units, as a means to mitigate fleet-wide shortages. (Recommendation 1) The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment, together with the F-35 Program Executive Officer, the Secretaries of the Air Force and Navy, and the Commandant of the Marine Corps, develops a process to modify the afloat and deployment spares packages, to include reviewing the parts within the packages to ensure that they match deploying aircraft and account for updated parts demand, and aligning any necessary funding needed for the parts updates. (Recommendation 2) The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment, together with the F-35 Program Executive Officer, the Secretaries of the Air Force and Navy, and the Commandant of the Marine Corps, revises the business rules for the prioritization of scarce F-35 parts across all program participants so as to clearly define the roles and responsibilities of all stakeholders, the process for assigning force activity designations, and the way in which deviations from the business rules will be conducted. (Recommendation 3) The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment, together with the F-35 Program Executive Officer, completes a detailed plan for the establishment of the global network for moving F-35 parts that outlines clear requirements and milestones to reach full operational capability, and that includes mechanisms to identify and mitigate risks to the F-35 global spares pool. (Recommendation 4) The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment, together with the F-35 Program Executive Officer, issues a policy consistent with DOD guidance that clearly establishes how DOD will maintain accountability for F-35 parts within the supply chain, and identify the steps needed to implement the policy retrospectively and prospectively—for example, how DOD will obtain the necessary data from the contractor. This policy should provide clarity on how F-35 parts will be categorized, specify how the program will implement DOD regulations, and define prime contractor roles and responsibilities. (Recommendation 5) The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment, together with the F-35 Program Executive Officer, develops a methodical approach to consistently obtain comprehensive cost information from the prime contractor for F-35 spare parts within the supply chain. (Recommendation 6) The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment, together with the Department of Defense Comptroller, the Secretaries of the Air Force and Navy, and the F-35 Program Executive Officer, completes and formalizes a methodology for the U.S. services to use in recording on their financial statements the funds spent on F-35 parts within the global spares pool. (Recommendation 7) The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment, together with the F-35 Program Executive Officer, the Secretaries of the Air Force and Navy, and the Commandant of the Marine Corps, clearly defines the strategy by which DOD will manage the F-35 supply chain in the future and update key strategy documents accordingly, to include any additional actions and investments necessary to support that strategy. (Recommendation 8) We provided a draft of this report to DOD for review and comment. In its written comments, reproduced in appendix II, DOD concurred with our recommendations and identified actions that it was taking or planned in response. We are providing copies of this report to appropriate congressional defense committees; the Acting Secretary of Defense; the Under Secretary of Defense for Acquisition and Sustainment; the F-35 Program Executive Officer; the Secretaries of the Air Force and Navy; and the Commandant of the Marine Corps. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix III. For each of our objectives, we reviewed relevant F-35 sustainment and supply chain plans, program briefs, guidance, and other documentation and collected information by interviewing officials from the Office of the Secretary of Defense for Acquisition and Sustainment, the F-35 Joint Program Office, the U.S. Air Force, the U.S. Navy, the U.S. Marine Corps, and the prime contractor, Lockheed Martin. To interview officials and observe F-35 supply and maintenance operations, we conducted site visits to two F-35 operational locations—Hill Air Force Base, Utah, and Marine Corps Air Station Yuma, Arizona; and one training location—Luke Air Force Base, Arizona. We selected these locations to obtain perspectives from both operational and training units from multiple U.S. military services using different variants of the aircraft, and to gather insights of international partners co-located at these bases, among other factors. Additionally, we interviewed officials from the only overseas- based U.S. F-35 operational squadron at Marine Corps Air Station Iwakuni, Japan, by phone. A complete listing of organizations we contacted for this review is provided later in this appendix. In support of our objectives, we gathered various data related to the F-35 supply chain, such as parts availability, repair, aircraft performance, and customer wait time data. We gathered data for fiscal year 2018 (October 2017 – September 2018) and available data from the F-35 program’s 2018 sustainment contract period (May – November 2018) in order to provide the most recent information for F-35 fleet performance and overall supply chain management available during our audit timeframes. To determine the reliability of these data, we collected information on how the data were collected, managed, and used through a questionnaire and interviews with relevant DOD officials and the prime contractor. Although we identified some limitations in the way that certain data are being collected and reported— such as data related to aircraft performance, aircraft that are not mission capable due to a lack of parts, and parts cannibalization that could potentially result in inaccuracies—we determined that they are sufficiently reliable for the way in which we reported them and our purposes of providing information on the progress and challenges within the program. Specifically, the parts cannibalization rates that we discuss are sufficiently reliable to discuss generally in comparison to program objectives. All other supply chain and performance data presented in our report are sufficiently reliable to present as specific data points. To assess the extent to which F-35 performance is meeting warfighter requirements and any challenges with spare parts availability, we reviewed DOD and contractor sustainment and supply chain plans, briefings, and reports, and interviewed Office of the Secretary of Defense, U.S. service, program office, and prime contractor officials to determine the degree to which the supply chain is currently able to provide parts to meet the U.S. services’ requirements. In addition, we obtained data related to F-35 parts availability and aircraft performance data for May through November 2018 and compared these to the program’s target and the U.S. services’ requirements for these metrics to identify any gaps between requirements and actual performance. We also obtained data related to 3-month average part repair times and part repair backlogs as of November 2018—the most currently available data at the time of our review. In order to assess the extent to which the supply chain is positioned to meet future warfighter requirements, we examined program plans, briefs, and other related documentation, and we interviewed Office of the Secretary of Defense, U.S. service, program office, and prime contractor officials to identify the actions that DOD is taking to increase the availability of F-35 spare parts, DOD’s projections for when these actions will result in improvements in F-35 aircraft performance, and ongoing areas of challenge that could create risk for the program in meeting future warfighter requirements. Finally, we used principles from the Standards for Internal Control in the Federal Government and DOD guidance for performance-based arrangements related to how programs should be structured to meet requirements and respond to risk as a basis to determine whether DOD needs to take further actions to ensure that the F-35 supply chain is positioned to meet future warfighter requirements. To assess the extent to which DOD can effectively manage and move F- 35 parts to support aircraft around the world, we reviewed military service and program briefings and data related to DOD’s fiscal year 2018 F-35 operational deployments, and we interviewed service, program office, and contractor officials about how the F-35 supply chain and its global spares pool were able to support these deployments, including the extent to which the packages of parts that the military services purchased to support these deployments were built to meet their requirements. We reviewed DOD guidance related to managing risk in acquisition programs and the Navy’s process and guidance for ensuring that the packages of parts for legacy aircraft are built to meet the requirements of deploying aircraft, and we assessed the F-35 program’s processes for identifying and addressing risks related to the sufficiency of its deployment parts packages against these criteria. We also reviewed the F-35 program’s business rules for allocating and prioritizing scarce F-35 assets and related documentation, and we interviewed officials from the Joint Staff, Office of the Secretary of Defense, the services, the program office, and the prime contractor to understand how the business rules are being applied and to identify any related F-35 program participant perspectives about or gaps in the rules. We also reviewed DOD guidance related to prioritizing materiel and parts to identify standard DOD policies for legacy aircraft, and Standards for Internal Control in the Federal Government, and we used these as a basis to assess whether the F-35 program’s business rules for allocating scarce F-35 parts are sufficiently clear and comprehensive. In addition, we reviewed available plans, briefs, and other documentation to understand the F-35 program’s envisioned global network for moving F-35 parts, the current state of the network, and the program’s projections for full implementation of the network. Further, we obtained data from December 2017 through November 2018 related to customer wait times for parts to determine whether program participants located outside of the continental United States are waiting longer for parts than those located inside of the continental United States. We also interviewed officials from the program office, prime contractor, Office of the Secretary of Defense, the services, and U.S. Transportation Command to discuss the progress being made and challenges the program faces in developing the global network to move F-35 spare parts. Finally, we assessed DOD’s plans for establishing its global network for moving parts against key acquisition program management practices that can improve program outcomes if implemented and DOD guidance related to managing risk in acquisition programs. To assess the extent to which DOD can account for F-35 spare parts within the supply chain and their associated costs, we reviewed program briefs, DOD guidance and the Federal Acquisition Regulation, and sustainment contracts and related documentation, and we interviewed program and contractor officials to determine how the program office is maintaining accountability for F-35 spare parts, to include roles and responsibilities for property accountability and any associated challenges. In addition, we reviewed draft guidance and program briefs and documentation, as well as interviewed officials from the program office, to identify the actions the program is taking to improve its ability to maintain accountability of parts in the F-35 program. We compared these efforts against criteria in DOD guidance for property accountability and Standards for Internal Control in the Federal Government to assess whether the program’s current efforts to obtain and maintain accountability for F-35 spare parts are sufficient to bring the program into alignment with DOD guidance, and whether any additional actions are needed. To assess the extent to which DOD is maintaining accountability over costs associated with F-35 spare parts, we reviewed program plans and documentation related to the construct of the global spares pool. We also reviewed sustainment contracts and supplemental contract documentation, and we interviewed officials from the program office, Office of the Secretary of Defense, and Defense Contract Management Agency to determine what information DOD has been able to obtain about the quantity and cost of F-35 spare parts and the approaches that DOD uses to collect such information. Additionally, we identified criteria within DOD guidance for performance-based arrangements and the DOD Financial Management Regulation to serve as a basis to assess whether the program office’s approach for obtaining cost information is sufficient to support program and financial management requirements. We also reviewed DOD and program office documentation and spoke with officials from the program office and the Office of the Under Secretary of Defense (Comptroller) to determine the extent to which the program office has developed a methodology to track the funds paid by the U.S. military services for F-35 parts to the actual parts within the global spares pool. Finally, we used the DOD Financial Management Regulation as a basis to assess whether the program has the ability to adequately track funds paid by the U.S. military services for F-35 spare parts to the actual parts within the global pool to support financial audits. To assess the extent to which actions DOD is taking to address supply chain challenges are consistent with the established F-35 program sustainment strategy, we reviewed key F-35 program strategy, planning, and structure documents—such as the 2016 and 2018 F-35 Acquisition Strategies, the Life-Cycle Sustainment Plan, and program office organizational structures—and F-35 sustainment contracts to determine the program’s formal strategy and structure for F-35 supply chain management. We also reviewed documentation related to DOD’s efforts to develop an option for DOD-management of the F-35 supply chain, such as data requests and a memorandum, and we interviewed officials from the Office of the Secretary of Defense for Acquisition and Sustainment, Defense Logistics Agency, military service sustainment commands, the program office, and the prime contractor to understand the extent to which DOD is pursuing a DOD-managed F-35 supply chain, whether these efforts are aligned with the established F-35 program strategy, and the effects of such actions on the program office’s ability to execute F-35 sustainment with the prime contractor, Lockheed Martin. In addition, we assessed DOD’s efforts to establish a DOD-managed option for supply chain management against principles from DOD planning guidance and Standards for Internal Control in the Federal Government for defining objectives and clearly aligning actions and resources to meet those objectives. In support of our work, we interviewed officials from the following DOD organizations and other organizations during our review. We selected these organizations based on their oversight, planning, and execution roles related to F-35 sustainment, supply chain management, and operations. United Kingdom Ministry of Defence We conducted this performance audit from January 2018 to April 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Alissa Czyz (Assistant Director), Vincent Buquicchio, Kasea Hamar, Amie Lesser, Sean Manzano, Michael Silver, Tristan T. To, and Cheryl Weissman made key contributions to this report. F-35 Joint Strike Fighter: Development Is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved. GAO-18-321. Washington, D.C.: June 5, 2018. Warfighter Support: DOD Needs to Share F-35 Operational Lessons Across the Military Services. GAO-18-464R. Washington, D.C.: April 25, 2018. Military Aircraft: F-35 Brings Increased Capabilities, but the Marine Corps Needs to Assess Challenges Associated with Operating in the Pacific. GAO-18-79C. Washington, D.C.: March 28, 2018. F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency. GAO-18-75. Washington, D.C.: October 26, 2017. F-35 Joint Strike Fighter: DOD’s Proposed Follow-on Modernization Acquisition Strategy Reflects an Incremental Approach Although Plans Are Not Yet Finalized. GAO-17-690R. Washington, D.C.: August 8, 2017. F-35 Joint Strike Fighter: DOD Needs to Complete Developmental Testing Before Making Significant New Investments. GAO-17-351. Washington, D.C.: April 24, 2017. F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities. GAO-16-390. Washington, D.C.: April 14, 2016. F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016. F-35 Joint Strike Fighter: Preliminary Observations on Program Progress. GAO-16-489T. Washington, D.C.: March 23, 2016. F-35 Joint Strike Fighter: Assessment Needed to Address Affordability Challenges. GAO-15-364. Washington, D.C.: April 14, 2015. F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.: September 23, 2014. F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26, 2014. F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014. F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013. F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but Affordability Challenges and Other Risks Remain. GAO-13-500T. Washington, D.C.: April 17, 2013. F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013. Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks. GAO-12-437. Washington, D.C.: June 14, 2012. Joint Strike Fighter: Restructuring Added Resources and Reduced Risk, but Concurrency Is Still a Major Concern. GAO-12-525T. Washington, D.C.: March 20, 2012. Joint Strike Fighter: Implications of Program Restructuring and Other Recent Developments on Key Aspects of DOD’s Prior Alternate Engine Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19, 2011. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011. Joint Strike Fighter: Restructuring Should Improve Outcomes, but Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.: March 15, 2011.", "summary": "DOD's F-35 fighter jet provides key aviation capabilities to support the U.S. National Defense Strategy. The F-35 is also DOD's most costly weapon system, with sustainment costs estimated at more than $1 trillion over a 60-year life cycle. The F-35's supply chain has a unique design. Rather than owning the spare parts for their aircraft, the Air Force, Navy, and Marine Corps—along with eight international partners and other foreign military sales customers—share a common, global pool of F-35 parts that are managed by the prime contractor. You asked us to review the F-35 supply chain. This report assesses, among other things, the extent to which (1) F-35 performance is meeting warfighter requirements and any challenges related to the availability of spare parts; (2) DOD can effectively manage and move F-35 spare parts to support aircraft around the world; and (3) DOD can account for F-35 spare parts and their costs within the supply chain. GAO reviewed DOD and contractor documentation, analyzed performance data, and interviewed relevant officials. F-35 aircraft performance is falling short of warfighter requirements—that is, aircraft cannot perform as many missions or fly as often as required. Figure: F-35 Fleet Aircraft Performance, May 2018 — November 2018 This lower-than-desired aircraft performance is due largely to F-35 spare parts shortages and difficulty in managing and moving parts around the world: Spare parts shortages and limited repair capabilities. F-35 aircraft were unable to fly nearly 30 percent of the May—November 2018 time period due to spare parts shortages. Also, the Department of Defense (DOD) had a repair backlog of about 4,300 F-35 parts. DOD is taking steps to fix these issues, such as improving the reliability of parts. However, it has not fully determined actions needed to close the gap between warfighter requirements and the performance the F-35 supply chain can deliver. Mismatched parts for deploying aircraft. DOD purchases certain sets of F-35 parts years ahead of time to support aircraft on deployments, including on ships. But the parts do not fully match the military services' needs because F-35 aircraft have been modified over time. For example, 44 percent of purchased parts were incompatible with aircraft the Marine Corps took on a recent deployment. Without a process to modify the sets of parts for deployments, DOD may be unable to meet the services' operational needs. An immature global network to move F-35 parts. DOD's networks for moving F-35 parts around the world are immature, and overseas F-35 customers have experienced long wait times for parts needed to repair aircraft. Without a detailed plan for the network, DOD may not be ready to support an expanding fleet. In addressing these challenges, DOD must grapple with affordability. The Air Force and Marine Corps recently identified the need to reduce their sustainment costs per aircraft per year by 43 and 24 percent, respectively. DOD has spent billions of dollars on F-35 spare parts but does not have records for all the parts it has purchased, where they are, or how much they cost. For example, DOD is not maintaining a database with information on F-35 parts the U.S. owns, and it lacks the necessary data to be able to do so. Without a policy that clearly defines how it will keep track of purchased F-35 parts, DOD will continue to operate with a limited understanding of the F-35 spare parts it owns and how they are being managed. If left unaddressed, these accountability issues will impede DOD's ability to obtain sufficient readiness within affordability constraints. GAO is making eight recommendations, including that DOD determine actions to close the gap between warfighter requirements and F-35 supply chain performance; and address challenges with deployments, global parts movement, and spare parts accountability. DOD concurred with all of GAO's recommendations.", "document_type": "gao"}
{"report": "The Cannon Building, completed in 1908, is the oldest congressional office building and occupied by Members and their staffs. (See fig. 1.) The Cannon Building houses 142 office suites, five conference rooms, four hearing rooms, and the Caucus Room, which can accommodate large meetings. The building also includes a library, food servery, and a health unit. AOC began developing the scope for the Cannon project in approximately 2004 when its consultant conducted a facility condition assessment that identified the building’s deficiencies. This condition assessment identified, for example, that the hot water heating and air-handling systems had components dating back to the 1930s that are in need of replacement. In addition, the assessment identified deficiencies such as an outdated fire alarm system for which repair parts were difficult to obtain, worn and damaged marble tile in corridors, and original windows that were damaged and often nonfunctional. AOC continued its planning and design work through 2014 to establish the final scope of the Cannon project, which entailed correcting most of the identified deficiencies and addressing current requirements such as for energy conservation, physical security, hazardous materials abatement, and historic preservation. Key components of the project, among other things, include: substantial reconfiguration of member suites and the reconstruction of the building’s top floor to convert storage space into new suites, refurbishment of windows and installation of a new roof, preservation of the building’s stone exterior, replacement of all plumbing, heating and cooling, fire protection, electrical, and alarm systems, and refurbishment of restrooms to make them more accessible to people with disabilities. As part of the development process for the Cannon project, AOC established a budget of approximately $753 million. Key components of the budget include costs for the construction contract; architect and engineering (A/E) design services; construction management support; security; furniture and fixtures; swing space design and construction; contractor incentive bonuses; and contingency. AOC is using the Construction Manager as Constructor (CMc) delivery method to implement the Cannon project. Under this approach, AOC: contracted with a construction contractor that consulted on the project’s design, and negotiated with the construction contractor to set a “guaranteed maximum price” for the construction work based on the completed design. AOC also contracted with an A/E firm, which produced the design for the project and is providing consultation during construction, and with a Construction Manager as Agent (CMa), that provides administrative and technical support to AOC in managing the construction work. AOC scheduled the Cannon project’s construction in five sequential phases with an initial phase (Phase 0) for utility work and four subsequent phases (Phases 1 through 4) to renovate the north-, south-, east-, and west-facing sides of the building. Each phase is scheduled around a 2- year congressional session. As the project progresses, tenants displaced during construction (Phases 1 through 4) are to move to temporary offices while other occupants are to remain in the building sections not affected by construction. Currently, AOC has substantially completed Phase 0 and Phase 1 of the five phases planned for the Cannon project and is progressing with work on Phase 2, which it expects to complete in November 2020. (See fig. 2.) AOC completed Phase 0, as planned and under its budget estimate, from January 2015 through December 2016. This work primarily included the construction contractor’s replacement of the utility infrastructure and distribution systems in the basement, garage, and courtyard. During this time, AOC also managed the work of its Construction Division to build 31 additional Member Suites to offset the suites that would be inaccessible when sections of the building were under construction. From January 2017 through December 2018, AOC managed the renovation of the first of four building sections, consisting of the building’s west side (facing New Jersey Avenue) and Rotunda (Phase 1). AOC substantially completed Phase 1 to enable occupancy of the building section, as planned, on January 3, 2019, at the start of the 116th Congress. However, it is continuing to address “punch-list” items of incomplete or corrective work from Phase 1. AOC expects to complete the punch-list items by December 2019. Further, AOC encountered several issues during the Phase 1 renovation that have prevented it from settling the costs for this phase and that will affect the cost of the project’s later phases. According to AOC’s most current (July 2019) Executive Summary, unforeseen conditions, design issues, and scope changes have increased both the estimated cost for Phase 1 and the project’s three remaining phases. For example, AOC found that more extensive exterior stone restoration was needed than planned and encountered some unforeseen asbestos-containing materials in the roof that it needed to mitigate. Further, AOC needed to provide additional security features to address U.S. Capitol Police requests. Collectively, these issues are creating cost pressures that have caused AOC to reassess the cost to complete the project. We discuss the project’s costs in greater detail later in this testimony. AOC is currently progressing, as planned, in renovating the north side of the building (facing Independence Avenue), which is the second of the four building sections to be renovated (Phase 2). Because the work in this phase and the Cannon project’s remaining phases is similar to work completed in Phase 1, AOC expects to benefit from its application of lessons learned. For example, AOC reported that its construction contractor experienced challenges installing the temporary roof enclosure that it used in Phase 1. Based on this experience, AOC officials told us that the contractor developed a new design for the temporary roof enclosure that the contractor expects to install more rapidly in the project’s remaining phases than in Phase 1. Further, because the materials in Phases 2 through 4 are the same as in Phase 1, AOC officials expect that the process of approving the construction contractor’s use of these materials should proceed faster in these later phases and enable construction to progress more efficiently. In 2009, we reported that AOC expected to request approximately $753 million for the Cannon project. At the time, AOC expected the project to be in five phases over 5 years. Because the project was in an early development stage at that time, we said: that AOC’s estimate should not be considered sufficiently accurate for funding purposes, that the cost and scope were likely to change, and that it would be important for AOC to continue to refine the project’s scope and cost estimate to provide Congress with the information it needed to make decisions about the project. When we next reported on the Cannon project in 2014, AOC had completed most of the planning and design and was preparing to award the contract for construction, which was to begin in January 2015. As part of our 2014 review of AOC’s cost estimating policies and guidance, we compared AOC’s cost estimate for the Cannon project—still $753 million—to our leading practices for developing high-quality, reliable cost estimates. We found the AOC’s cost estimate reflected several, but not all, of our leading practices. In particular, we found that AOC’s estimate included ground rules and assumptions; provided a reasonable explanation of the basic estimation methodologies; and integrated separately produced estimates from AOC’s architect, construction manager, and construction contractor to enable a reasonably accurate assessment of estimated costs. Further, we found AOC had conducted a cost risk and uncertainty analysis in accordance with a key leading practice. This analysis concluded that based on AOC’s inputs and assumptions, there was a high probability (over 90 percent) that actual costs would be equal to or less than AOC’s $753 million estimate. This estimate included contingency factors to account for risks and uncertainties. However, our review of AOC’s guidance for developing cost estimates found that the guidance did not provide documented reasons explaining how the actual contingency amounts were developed. In addition, we found that the method AOC used to model the project’s risks in its cost risk and uncertainty analysis (1) resulted in an unusually narrow range of estimated costs and (2) provided managers limited ability to understand the effects of individual risks. We recommended that AOC improve its cost-estimating process, such as by incorporating leading practices we identified as lacking for cost estimating into its cost- estimating guidance and policies. AOC has since implemented our recommendations. In January 2018, while Phase 1 of the Cannon project was in progress, AOC updated its analysis of risks by undertaking a study (termed an integrated cost-schedule risk analysis) to determine the potential effects of these risks on the project’s cost and schedule. Updating risk analyses and their effect on project cost estimates is consistent with leading practices for developing both a high-quality, reliable cost estimate and schedule. AOC’s 2018 analysis arrived at the same conclusion as its 2014 analysis—that the estimated $753 million total project cost was adequate and that there was a high probability (over 80 percent) that actual costs would be equal to or less than the $753 million estimate. However, this analysis was qualified on the assumption that AOC and project stakeholders are able to adequately mitigate risks identified through the analysis. Additionally, the analysis indicated that inaccurate estimates of costs for risk mitigations, currently unknown risks, and optimistic assumptions about the impact of risk mitigations on the project’s cost and schedule could affect the project’s total cost. As noted previously, the project is experiencing cost pressures from the greater-than-anticipated risks and ineffective mitigations stemming from unforeseen conditions, design issues, and scope changes. In June 2019, AOC reported that it expects that the cost to complete the Cannon project will increase by 10 to 15 percent over its initial estimate of $753 million, resulting in a final cost between approximately $828 million and $866 million. AOC reported that the following key factors affect the project’s cost: Phase 1 completion costs. While Phase 1 work has been substantially completed, AOC has yet to settle all outstanding change proposals. AOC reported that the cost to complete Phase 1 is greater than it initially planned and that it will not know the final cost for this phase until it completes negotiations of the cost of unsettled change proposals. Phase 2 modifications. While Phase 2 work has begun, AOC is awaiting the contractor’s proposal on the costs to address the requirements outlined in four “design bulletins” issued by AOC that, in part, describe changes to the project’s scope based on lessons learned in Phase 1. AOC estimates that the contract modifications described by the design bulletins will increase the cost of Phase 2. Phase 3 and 4 modifications. AOC expects that it will award these future phases of the project at higher amounts than it initially planned based, in part, on the estimated cost of incorporating the additional work described in the design bulletins. In August 2019, AOC began updating its integrated cost-schedule risk analysis, with the aim of more accurately determining the extent to which the project’s costs are increasing and its estimated cost at completion. By updating the analysis, AOC should be better able to make informed decisions as construction progresses. Further, updating the analysis should enable AOC to more precisely estimate the Cannon project’s cost at completion and better position AOC to make a more accurate budget request to Congress for remaining costs. Chairperson Lofgren, Ranking Member Davis, and Members of the Committee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff has any questions concerning this testimony, please contact Terrell Dorn at (202) 512-6923 or dornt@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contacts named above: Michael Armes (Assistant Director); George Depaoli (Analyst-in-Charge); Geoffrey Hamilton; Malika Rice; Kelly Rubin; Steve Schluth; and Amelia Michelle Weathers made key contributions to the testimony. Other staff who made contributions to the reports cited in the testimony are identified in the source products. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Cannon project intends to preserve the historic character while improving the functionality of the 111 year-old Cannon Building—the oldest congressional office building—as well as address deterioration to the building and its components. The project—nearing the mid-point of its planned 10-year duration—is being implemented in five sequential phases with an initial phase (Phase 0) for utility work and four subsequent phases (Phases 1 through 4) to renovate the north-, south-, east-, and west-facing sides of the building. Each phase is scheduled around a 2-year congressional session. This statement describes: (1) the status of the Cannon project and (2) changes to the project's estimated cost at completion. This statement is based on GAO's prior reports in 2009 and 2014 and ongoing monitoring of the project. To monitor the project, GAO has been observing the ongoing construction, attending project meetings, and analyzing AOC documents. The Architect of the Capitol (AOC) has substantially completed two of five planned phases to renovate the Cannon House Office Building (Cannon project). AOC completed Phase 0 utility work; has almost finished the Phase 1 work to renovate the building's west side, as planned; and is progressing with Phase 2 work to renovate the building's north side. From 2009 to 2018, AOC consistently estimated the project cost at $753 million, but AOC reported in June 2019 that it expects costs to increase by 10 to 15 percent, resulting in a total cost of approximately $828 million to $866 million. In 2014, GAO found that AOC's cost estimate of $753 million reflected several, but not all, of GAO's leading practices for high-quality, reliable cost estimates, including that AOC had conducted a risk and uncertainty analysis. In January 2018, AOC updated its analysis of risks by undertaking an integrated cost-schedule risk analysis. AOC's 2018 analysis arrived at the same conclusion as its earlier analysis—that the project's estimated $753 million total cost was adequate to complete the project. However, AOC's 2018 analysis indicated that inaccurate estimates of costs for risk mitigations, unknown risks, and optimistic assumptions about the effect of risk mitigations on the project's cost and schedule could affect its total cost. In June 2019, AOC reported that greater-than-expected risks, such as from unforeseen conditions that led to more extensive exterior stone restoration than anticipated and the unplanned mitigation of asbestos in roof materials, would increase the project's cost. AOC is currently determining the effect of these and other changes on Phase 1, where work has been substantially completed, but costs have not been settled. AOC is also determining how the costs of the project's remaining phases will be affected by scope changes stemming from lessons learned in Phase 1. Toward this end, in August 2019, AOC began updating its integrated cost-schedule risk analysis, with the aim of more accurately determining the extent to which the project's costs are increasing and its estimated cost at completion. In 2014, GAO made recommendations pertaining to AOC's cost-estimating guidance and policies. AOC has implemented these recommendations.", "document_type": "gao"}
{"report": "The purpose of federal banking supervision is to help ensure that depository institutions throughout the financial system operate in a safe and sound manner and comply with federal laws and regulations for the provision of banking services. In addition, federal banking supervision looks beyond the safety and soundness of individual institutions to promote the stability of the financial system as a whole. Each depository institution in the United States is primarily supervised by one of the following three federal banking regulators: The Federal Reserve supervises state-chartered banks that are members of the Federal Reserve System, bank and savings and loan holding companies, Edge Act and agreement corporations, and the U.S. operations of foreign banks. FDIC supervises insured state-chartered banks that are not members of the Federal Reserve System, state-chartered savings associations, and insured state-chartered branches of foreign banks. OCC supervises federally-chartered national banks and savings associations and federally-chartered branches and agencies of foreign banks. These federal banking regulators have broad authority to examine depository institutions subject to their jurisdiction. Federal banking regulators carry out a number of supervisory activities in overseeing management of large depository institutions (see table 1 for a summary of supervision programs for large depository institutions). The supervisory activities are conducted both off- and on-site. Generally, federal banking regulators use off-site systems to monitor the financial condition of an individual bank; groups of banks with common product, portfolio, or risk characteristics; and the banking system as a whole between on-site examinations. Federal banking regulators generally conduct on-site supervision by stationing examiners at specific institutions. This practice allows examiners to continuously analyze information provided by the financial institution, such as board meeting minutes, institution risk reports or management information system reports. This type of supervision is intended to allow for timely adjustments to the supervisory strategy of the examiners as conditions change within the institutions. FDIC, the Federal Reserve, and OCC are required to conduct a full- scope, on-site examination of each insured depository institution they supervise at least once during each 12-month period. The regulators may extend the examination interval to 18 months, generally for institutions that have less than $3 billion in total assets and that meet certain conditions, based on ratings, capitalization, and status of formal enforcement actions, among others. For large institutions, federal banking regulators do not conduct an annual point-in-time examination of the institution. Rather, they conduct ongoing examination activities that are generally intended to evaluate an institution’s operating condition, management practices and policies, and compliance with applicable laws and regulations. In particular, examiners review an institution’s condition using the Uniform Financial Institutions Rating System, also known as CAMELS (capital adequacy, asset quality, management, earnings, liquidity, and sensitivity to market risk). Evaluations of CAMELS components consider an institution’s size and sophistication, the nature and complexity of its activities, and its risk profile. Throughout the examination cycle, each target examination will result in a letter that is transmitted to the institution (where applicable). At the end of the supervisory cycle, a report of examination is issued to the institution. The target examination letter and report of examination may include supervisory concerns that examiners found and that an institution is expected to address within specific time frames. The regulators also issue supervisory guidance, which they describe as including interagency statements, advisories, bulletins, policy statements, questions and answers, and frequently asked questions issued to their respective supervised institutions. Supervisory guidance outlines the regulators’ supervisory expectations or priorities and articulates general views regarding appropriate practices for a given subject area. The guidance often provides examples of practices that the regulators generally consider consistent with safety and soundness standards or other applicable laws and regulations. According to the regulators, supervisory guidance is not legally binding. For instance, FDIC financial institution letters generally announce matters of interest to those responsible for operating an institution. Federal Reserve supervision and regulation letters address significant policy and procedural matters. OCC bulletins generally accomplish the same goals as FDIC and Federal Reserve letters. The letters and bulletins are published on each regulator’s website. Often, the contents of these documents are incorporated into broader examination manuals. Moreover, the federal banking regulators have developed internal control functions within the supervision programs for large depository institutions, which consist of several layers of review following examinations. Each regulator has a review process at the conclusion of examinations, and examiners prepare written products documenting their findings and meet with regional and headquarters officials to finalize decisions. Also, each regulator maintains an internal review function to ensure that examiners properly applied examination guidance. We and others previously found that regulators identified underlying risks at depository institutions that failed during the 2007–2009 financial crisis well before their failure, but did not always take timely supervisory action. As stated by the regulators, the strength or weakness of bank management can reflect an institution’s underlying risk. For example, according to FDIC, the quality of management, including the board of directors and executives, is probably the single most important element in the successful operation of an institution. The Federal Reserve noted that the culture, expectations, and incentives established by the highest levels of corporate leadership set the tone for the entire organization and are essential determinants of whether an organization is capable of maintaining fully effective risk-management and internal control processes. Also, according to OCC, an effective corporate and risk governance framework is essential to ensuring the safe and sound operation of the institution and helping to promote public confidence in the financial system. In our past work, regulators told us they recognized bank supervision needed to be more forward-looking and had incorporated more forward- looking elements into examinations. Forward-looking supervision seeks to mitigate emerging risks before they affect the financial condition of an institution. Regulators can respond to emerging risks in the banking sector with a variety of supervisory tools. These include micro-prudential tools, which traditionally have focused on the safety and soundness of individual financial institutions, and macro-prudential tools, which can be used to address vulnerabilities across the banking system and broader financial system. Supervisory concerns are an important micro-prudential tool to support forward-looking supervision by ensuring that a depository institution takes early action to correct deficiencies. Also, trends in examination data and enforcement activity can provide information on regulators’ identification of and response to concerns of institution safety and soundness and emerging risks. Since 2009, federal banking regulators have revised policies and procedures to address management weaknesses at large depository institutions, including by differentiating levels of severity for supervisory concerns and specifying when to communicate them to management at the institutions. Based on our review of selected examination documents, the regulators’ policies and procedures often took different approaches for overseeing management of large depository institutions but each generally addressed leading risk-management practices. Since 2009, federal banking regulators have revised policies and procedures to better address management weaknesses at large depository institutions identified in the aftermath of the financial crisis. Regulatory staff with whom we spoke noted that most important risk- management concepts had been included in their policies for some time. The post-crisis updates were intended to provide better definitions of certain risk categories and enable examiners to consider individual risks within the context of all risks facing the institution. For instance, in June 2009, FDIC re-emphasized the forward-looking approach, which FDIC states encourages examiners to consider the likelihood that identified weaknesses will cause material problems in the future, and consider the severity of damage to an institution if conditions deteriorate. FDIC further noted that this assessment reflects both the board of directors’ and management’s ability to identify, measure, monitor, and control the risks of the institution’s activities, ensure its safe and sound operations, and ensure compliance with applicable laws and regulations. FDIC policy provides that an assessment of management is not solely dependent on the current financial condition of the institution. Also, in 2015 FDIC updated policies and procedures for identifying and assessing the influence of dominant bank officials or policymakers on an institution, and stated the policy was intended to limit the influence of dominant officials when internal controls are inadequate and ensure independence of the risk-management function. In 2012, the Federal Reserve updated procedures for supervision of large financial institutions, which were intended to strengthen traditional firm- level supervision while also incorporating systemic considerations to reduce potential threats to the stability of the financial system and provide insights into financial market trends. In 2013, the Federal Reserve updated expectations for the assessment of an institution’s internal audit function and provided guidance about the degree to which examiners may rely on the work of an institution’s internal audit function. In 2015, OCC updated its Risk Assessment System to help examiners draw conclusions about the quantity of risk, quality of risk management, aggregate risk, and direction of risk for institutions under eight different risk categories. Also, in 2016, OCC published the Corporate and Risk Governance booklet of the Comptroller’s Handbook to incorporate heightened standards requirements for depository institutions with average total consolidated assets of $50 billion or more. The booklet provides guidance to examiners on board and management responsibilities, risk management assessment factors, and measurement and assessment of risk consistent with the heightened standards. Regulators also took steps to enhance their ability to resolve supervisory concerns in a timely manner through improvements to policies and procedures on identifying and communicating concerns. The regulators employ progressive enforcement regimes to address supervisory concerns that arise during the examination cycle (see table 2). If the institution does not respond to the concern in a timely manner, the regulators may take informal or formal enforcement action, depending on the severity of the circumstances. Informal enforcement actions include obtaining an institution’s commitment to implement corrective measures under a memorandum of understanding. Formal enforcement actions include issuance of a cease-and-desist order or assessment of a monetary penalty, among others. The regulators have continued to update these regimes to clarify the distinction between each level of concern and to improve communication of concerns to the boards of directors of depository institutions. For instance, in 2016, the board of directors of FDIC issued a statement setting forth basic principles to guide the identification and communication of supervisory recommendations. The board stated that a supervisory recommendation refers to FDIC communications with a depository institution that are intended to inform it of FDIC’s views about changes needed to its practices, operations, or financial condition. FDIC’s updated policies and procedures state that supervisory recommendations must be presented in writing and most are generally correctable in the normal course of business. When developing and communicating these recommendations, FDIC examiners are required to (1) address meaningful concerns, (2) communicate concerns clearly and in writing, and (3) discuss corrective action. Supervisory recommendations involving an issue or risk of significant importance and that typically would require more effort to address than those correctable in the normal course, would need to be brought to the attention of the board and senior management through matters requiring board attention (MRBA) comments. The Federal Reserve updated its policies and procedures on identification and communication of supervisory concerns in 2013. The supervision and regulation letter defined matters requiring immediate attention (MRIA) to include (1) matters that have the potential to pose significant risk to the safety and soundness of the banking organization; (2) matters that represent significant noncompliance with applicable laws or regulations; (3) repeat criticisms that have escalated in importance due to insufficient attention or inaction by the banking organization; and (4) in the case of consumer compliance examinations, matters that have the potential to cause significant consumer harm. The letter defines matters requiring attention (MRA) as deficiencies that are important and should be addressed over a reasonable period of time, but where the institution’s response need not be immediate. Therefore, the distinction between MRIAs and MRAs is the nature of and severity of the matter and the timing by which the institution must respond. No matter how serious the concern, it is addressed to the institution’s board of directors. According to the Federal Reserve’s policies and procedures, the communication of supervisory findings must be (1) written in clear and concise language, (2) prioritized based upon degree of importance, and (3) focused on any significant matters that require attention. The Federal Reserve proposed new supervisory concern policies and procedures in 2017, which provided that examiners and supervisory staff should direct most MRIAs and MRAs to senior management of institutions for corrective action. MRIAs or MRAs only would be directed to the board for corrective action when the board needed to address its corporate governance responsibilities or when senior management failed to take appropriate remedial action. The proposed policies would not change the definitions of MRAs and MRIAs or the content of communications to institutions. As of April 2019, the proposed policies and procedures had not been finalized. OCC updated its policies and procedures for examiners to identify and communicate MRAs in 2014 and further enhanced them in 2017. OCC’s policy states that MRAs describe practices that an institution must implement or correct, ideally before those deficient practices affect the bank’s condition. Specifically, MRAs describe practices that (1) deviate from sound governance, internal control, or risk-management principles, and have the potential to adversely affect the bank’s condition, including its financial performance or risk profile, if not addressed; or (2) result in substantive noncompliance with laws or regulations, enforcement actions, or conditions imposed in writing in connection with the approval of any application or other request by the bank. OCC refers to such practices as deficient practices. Such practices also may be unsafe or unsound— generally, any action, or lack of action that is contrary to generally accepted standards of prudent operation and the possible consequences of which, if continued, would be abnormal risk or loss or damage to an institution, its shareholders, or the Deposit Insurance Fund. OCC supervisory concerns are to be communicated in writing to the institution’s management and board of directors to ensure timely and effective correction. Written communications must incorporate the “five c’s” format: Describe the concern. Identify the root cause(s) of the deficient practice and contributing factors. Describe potential consequence(s) or effects on the bank from inaction. Describe supervisory expectations for corrective action(s). Document management’s commitment(s) to corrective action and include the time frame(s) and the person(s) responsible for corrective action. If the root cause of the deficient practice is not apparent, OCC’s procedures instruct examiners to direct management to perform a root- cause analysis as part of the corrective action. The regulators’ revised policies and procedures that relate to oversight of risk management at large depository institutions and to supervisory concerns generally were consistent with leading risk-management practices. We reviewed leading standards and practices (such as federal internal control standards) and then developed criteria with which to assess the regulators’ policies and procedures. Criteria we used included that guidance be clear and actionable and that examiners review risk- management and control functions, identify existing and emerging risks, and review compliance with laws and regulations. (See table 3 for the specific criteria we applied, appendix I for more information on our methodology, and appendix II for the list of policy and procedure documents we reviewed). While individual policies or procedures may not have satisfied all of our criteria, when viewed collectively the policies and procedures generally addressed leading risk-management practices. For example, the policies and procedures almost always provided examiners with clear and actionable objectives for risk-management governance; enabled examiners to identify whether an institution had established a clear governance framework; assisted examiners in identifying, reporting, and recommending changes to address existing and emerging risks; and required review of institutions’ compliance with applicable laws and regulations. More specifically, we found FDIC risk-management policies and procedures for examining large insured depository institutions generally provide clear, actionable risk-management objectives with a few exceptions that did not materially affect our overall assessment. For instance, we identified that a policy document contains clear parameters for examiners to assess identified risks, which is consistent with our criteria, but the parameters did not include instructions for when examiners should consider changing a bank’s rating based on identified risk levels. However, related guidance for examiners in considering the impact of risk on the institution can be found in the definitions and descriptions of CAMELS ratings. We also found that FDIC developed adequate policies and procedures to evaluate corporate governance. In particular, consistent with leading practices, the guidance requires separation of board and management; identification and response to dominant officials; and encourages detailed review of the control environment. FDIC also has processes on risk assessment, and tracking and monitoring risk to address existing and emerging risks. For example, examiners are required to review updates to the institution’s risk- management processes for new lines of business. Similarly, we found that Federal Reserve policies and procedures for large depository institutions generally identify clear, actionable risk- management objectives and explain activities that might be riskier at some institutions compared to others, but a few policies and procedures were not fully consistent with our criteria. For instance, while corporate governance policies and procedures provide detailed materials for examiners to use during examination, and there is extensive guidance on risk identification, assessment, and communication, we noted relatively limited written procedures regarding escalation of concerns to enforcement actions. We discuss this issue in more detail later in this report. We also found that the Federal Reserve included forward-looking risk assessment procedures within risk-identification processes, including preliminary risk assessment to address existing and emerging risks. Finally, we found that OCC policies and procedures for large depository institutions generally provide clear requirements for examiner evaluation of the supervised institution’s quantity of risk, quality of risk management, and direction of risk. But the methods of measurement and specific tolerances for risk in these policies and procedures are not as clear as suggested by the leading practices. However, guidance to evaluate the potential impact of risk is separately available to examiners in OCC’s MRA and enforcement action policies and procedures. We found that consistent with our criteria, policies and procedures are detailed to provide examiners a clear framework to review banks’ corporate governance and risk-management systems. In particular, appropriate attention is paid to board oversight and effective management practice, including clear outlines for board and management responsibilities and independence. To address existing and emerging risks, OCC requires examiners to assess a specific set of risks within its risk-based supervision approach using the Risk Assessment System. OCC uses the Risk Assessment System in conjunction with CAMELS and other regulatory ratings during the supervisory process to evaluate an institution’s financial condition and resilience. Our review of examination documents of nine depository institutions found that examiners from the three banking regulators generally applied their policies and procedures and identified and communicated management weaknesses to those institutions. Practices for communicating concerns varied among regulators and some practices led to communications that often lacked complete information that would help institutions’ boards of directors ensure that senior management respond to emerging risks in a timely manner. Lastly, examiners generally followed up on prior supervisory concerns consistent with their policies and procedures. For the examinations we reviewed, we found that examiners generally applied policies and procedures to assess management oversight of risk at large depository institutions, including those relating to corporate governance, internal controls, and internal audit. We compared selected elements of examiner policies and procedures (focusing on the management component of CAMELS) with selected 2014–2016 examination documents to determine how examiners applied policies and procedures. (See appendix III for the questions we used to make these determinations). Our non-generalizable review of examination documents of nine institutions found that examiners reviewed areas relating to corporate governance, internal controls, and internal audit, which are key components of risk-management frameworks for institutional management and governance. For instance, to assess the adequacy of an institution’s overall corporate governance, FDIC, Federal Reserve, and OCC examiners of the selected institutions generally conducted reviews of areas such as board and management oversight and internal audit. For example: In examination documents for one of the institutions, we found that FDIC examiners examined materials regarding independence and qualifications of directors and policies and procedures related to risk assessments. We noted for another institution that Federal Reserve examiners reviewed materials regarding directors’ fulfillment of duties and responsibilities and policies and procedures relating to corporate compliance. Also, we observed that for one institution, in describing the leadership of the board and management, OCC examiners described aspects of the control environment, risk assessment, control activities, accounting, information, and communication as well as self- assessment and monitoring. At eight of the nine institutions we reviewed, we also found that regulators took steps that were designed to communicate deficiencies they identified before the weaknesses affected an institution’s financial condition. More specifically, examiners identified concerns related to board oversight; risk monitoring; policies, procedures, and limits; and internal controls. Also, for at least four of the nine institutions we reviewed, examiners reported they downgraded the management component rating based on weaknesses identified in management of risks independent of the institutions’ financial condition. For example, at one institution, we observed examiners reporting that weaknesses in an institution’s risk management contributed to a less-than-satisfactory or “3” rating for the management component. Additionally, examiners downgraded the management component rating for two institutions with satisfactorily-rated financial positions because of significant weaknesses in the risk- management program. In another instance, we observed examiners reporting that management’s need to complete remediation of previously identified weaknesses contributed to a “fair” or “3” rating for the management component of CAMELS. As previously discussed, in the past regulators did not always take timely supervisory action on the management weaknesses they identified. In all the reports of examinations we reviewed, examiners generally explained the basis for the rating they assigned to the management component of CAMELS, such as management’s responsiveness to addressing weaknesses and compliance with laws and regulations. Practices for communicating supervisory concerns to institutions varied among regulators and some communications do not provide complete information that could help boards of directors monitor whether deficiencies are fully addressed by management. As discussed previously, the regulators require staff to communicate supervisory concerns to institutions through formal written communications. The written communications are generally directed to senior management and boards of directors, which have oversight responsibilities over senior management. According to the Federal Reserve, boards are inherently disadvantaged given their dependence on senior management for the quality and availability of information. One industry representative told us that supervisory concerns were not always clearly communicated, noting that communications of supervisory concerns sometimes can be difficult to interpret and correct. An official from one of the regulators stated that former examiners working as industry consultants sometimes may be hired to help interpret supervisory letters and assist depository institutions in responding to supervisory concerns. Federal internal control standards state that management should communicate quality information externally to help the entity achieve its objectives and address related risks. Quality information is defined as appropriate, current, complete, accurate, accessible, and provided on a timely basis. Other authoritative internal control sources, including Circular A-123 and the framework of the Committee of Sponsoring Organizations of the Treadway Commission (COSO) require cause analysis—that is, an identification of the cause of the deficiencies that have been found. Generally accepted government auditing standards require that auditors plan and perform procedures to develop all four elements of a finding (criteria, condition, cause, and effect) necessary to address audit objectives. Although these authoritative sources do not apply to federal banking regulators, the standards identify principles consistent with the goal of FDIC, Federal Reserve, and OCC guidance in ensuring clear and complete communication of supervisory recommendations. OCC. For two of the three OCC-supervised institutions whose examination documents we reviewed, OCC examiners generally communicated to boards of directors the information they would need to monitor to determine whether deficiencies were fully addressed by management. OCC’s policies and procedures on MRAs require examiners to identify and communicate in writing to depository institutions the concern, cause, consequences of inaction, required corrective action, and management’s commitment for corrective action. If the cause of the deficient condition is not apparent, examiners must direct the institution’s management to perform a root-cause analysis as part of the corrective action. According to OCC staff, they implemented the MRA requirements agency-wide in 2014 after having a positive experience applying them at the community bank level. OCC staff told us that it is necessary for examiners and institutions to understand the cause of a deficiency for examiners to make appropriate recommendations and institutions to address the concern and help ensure the deficiency does not reoccur. Failure of examiners to identify and communicate the root causes of inappropriate practices was among the key findings of an internal OCC review of supervision of sales practices at Wells Fargo. In September 2016, OCC took enforcement action against Wells Fargo for improper sales practices. In April 2017, OCC’s Office of Enterprise Governance and the Ombudsman published an independent review of OCC’s supervisory record for Wells Fargo, which identified gaps in OCC’s supervision and lessons learned. Review findings included that the OCC team responsible for supervising Wells Fargo did not ensure that examiners evaluated root causes of the improper sales practices. In addition, they found that the first MRA that identified the sales practices issue in 2010 did not list the issue as an unsafe or unsound practice and did not identify a root cause or responsible parties. Among the lessons learned was ensuring analysis of root causes and compliance with OCC MRA guidance. In our review, we also observed how OCC’s written communications of concerns changed as its requirements were implemented. For example, in documents from 2014 for two institutions, OCC examiners generally only communicated the concern or the required corrective action and management’s commitment to corrective actions. By 2016, examiners documented each of the required elements for MRAs in their written communication (for two institutions). FDIC. For the three FDIC-supervised institutions whose examination documents we reviewed, FDIC examiners did not communicate to boards of directors the information they would need to monitor whether deficiencies were fully addressed by management. For these three institutions, FDIC examiners stated the concern (deficiency) and required corrective action in their internal communications of supervisory recommendations and also externally with depository institutions. They sometimes stated the potential effect of the deficient condition on the safety and soundness of the institution. These practices were consistent with FDIC policies and procedures in place at the time. For example, in the written communication to one FDIC institution selected for our review, examiners conveyed specific information about the supervisory concerns, the effect of the deficiencies on the institution, and the required corrective action for the MRBAs related to an examination. In another instance, the communication of the supervisory concerns appeared less specific. In that case, examiners reported that the institution management’s actions did not fully address a deficient condition identified in the prior examination. We found that the prior written communication of concerns to the institution did not identify the cause of the deficient condition or propose specific action to be taken. FDIC staff told us they believed that updates to their policies and procedures in 2016 already require examiners to identify the cause for the deficient condition and communicate it to the depository institutions. Specifically, FDIC requires examiners to “describe the deficient practices, operations, or financial condition and how it deviates from sound governance, internal controls, or risk management or consumer protection principles, or legal requirements.” This requirement is similar to OCC’s requirement to “describe the concern.” Specifically, OCC examiners are required to “describe the deficient practice and how it deviates from sound governance, internal control or risk management principles.” FDIC’s policies and procedures do not require examiners to identify the factor(s) responsible for the deficient condition (the “why”) or communicate it to the institutions. Based on the examination documents we reviewed, we did not observe that FDIC examiners communicated the cause of the deficiency. Including the cause facilitates a better understanding of why an institution’s condition is not consistent with law or regulations and, ultimately, can help an institution determine how it could remedy the condition. Federal Reserve. In our review of examination documents for three institutions, Federal Reserve examiners did not include information that boards of directors would need to monitor whether deficiencies were fully addressed by management. Reserve Bank examiners stated the condition and required corrective action in their internal and external communications of supervisory recommendations to depository institutions, consistent with Federal Reserve policies and procedures. Furthermore, the condition and required corrective action were generally closely linked to the criteria examiners applied during the examination, which often consisted of Federal Reserve supervisory guidance. We found that the written communications to depository institutions did not always provide information that would convey the reason the deficient condition occurred (cause) or the potential consequences of the deficient condition (effect). As a result, the information conveyed in the written communications of supervisory concerns was limited. The Federal Reserve Board has broad criteria for Federal Reserve Bank examiners requiring them to communicate only the condition and required corrective action. Federal Reserve Board staff told us that they do not require examiners to identify the cause of a deficient practice or condition. Instead, they leave that responsibility to institutions. Staff stated that they believe the institution is in the best position to identify the cause. They noted that this also could reduce the amount of time examiners otherwise would spend searching for the cause. However, we noted that at least one Reserve Bank builds on the Board’s criteria for communicating supervisory concerns and developed policies and procedures that require examiners to identify condition, criteria, cause, and effect to support supervisory findings in review sessions with Reserve Bank management. As discussed previously, authoritative internal control sources require cause analysis. As an example applicable to banking regulators, OCC requires its staff to identify and communicate the cause of the deficiency that led to the supervisory concern, or, if the root cause is not apparent, to instruct institution management to identify root cause as part of its corrective action. OCC staff noted that identifying root cause in examinations does not require additional resources. Also, if the root cause is not apparent, examiners instruct the institution to identify root cause as part of the corrective action, per OCC’s MRA policy. Furthermore, a September 2018 interagency statement clarifying the role of supervisory guidance instructed examiners to not criticize institutions for a “violation” of supervisory guidance. Identification and communication of the potential effect of a deficiency could enable the Federal Reserve to move away from its practice of closely linking supervisory concerns to failure to comply with guidance and better explain why an institution’s condition is not consistent with law or regulations. FDIC and the Federal Reserve are missing an opportunity to communicate complete information, in writing, to the boards of institutions regarding the cause of the identified deficiency that led to the supervisory concern, which would facilitate a better understanding of why the institution’s condition deviates from safety and soundness standards. Additionally, without communicating the potential effect of a deficiency, the Federal Reserve is missing an opportunity to convey to boards of directors how the concern could undermine the institution’s safety and soundness. In the examination documents of nine institutions we reviewed, federal banking regulators generally followed up on supervisory concerns to determine an institution’s progress in correcting previously identified weaknesses. The regulators require that examiners follow up on corrective actions taken by depository institutions in response to supervisory concerns. Examiners used various methods to follow up on supervisory concerns, such as by conducting limited-scope targeted reviews of one or more issues or incorporating follow-up as part of their regularly scheduled examination of a functional area. In addition, we observed that at four institutions examiners performed follow-up as part of their ongoing supervisory activities. While there are time frame targets for completion of corrective action, concerns can remain open until examiners are satisfied with the effectiveness of the remedial actions taken to address the supervisory concern. For instance, at three institutions we found that examiners closed concerns in targeted follow-up examinations once they validated the completion of remedial action by reviewing documents and activities that verified the implemented action was effective. We also observed instances for at least three institutions in which examiners refrained from closing supervisory concerns because they determined that the institutions’ management had not yet adequately addressed the concerns and further attention was warranted to ensure the corrective action was sustainable. In performing regularly scheduled target examinations of specific functions or risk areas examined during a previous examination cycle, examiners assessed management’s progress in addressing prior supervisory concerns at eight of the nine institutions we selected for examination documentation review. They examined documents, and reviewed processes and other related actions taken by management to address weaknesses in the institution’s management of risk. Lastly, at four institutions, examiners reviewed management’s progress and reported updated information on the institutions’ actions to address supervisory concerns that were escalated to enforcement actions. For example, at one institution OCC examiners documented substantive discussion on the work they performed in conducting follow-up on a consent order, which included reviewing revised documents and reports as well as validation efforts by a third-party consultant. Federal banking regulators collect and analyze supervisory concern data but do so to different degrees, and FDIC collects supervisory concern data in a manner that challenges management’s ability to fully monitor its supervision activities. We reviewed supervisory concern data for all institutions supervised by FDIC, OCC, and the Federal Reserve. The data we reviewed indicate that management weaknesses have been a consistent concern since 2012. In general, the amount of time supervisory concerns remain open generally has been reduced. The Federal Reserve and OCC track escalation of supervisory concerns to enforcement actions, but the Federal Reserve lacks specific, measurable guidelines for examiners to consider when supervisory concerns are not addressed in a timely manner. Federal banking regulators analyze supervisory concern data to inform examination strategy and forward-looking supervision to varying degrees. FDIC staff uses the data to track the duration of open MRBAs. FDIC’s Risk Management Supervision Division has staff responsible for categorizing and analyzing MRBA summary comments quarterly and providing an analysis memorandum to the division’s management to assist with forward-looking risk identification. FDIC staff stated that these analyses supplement other data used to conduct supervisory follow-up. Federal Reserve Board staff told us that they use the data to track MRA and MRIA information over time within portfolios of depository institutions of different sizes. Staff noted that the data are used to inform supervisory strategy development for upcoming examination cycles. According to staff with whom we spoke, the data are useful for conducting horizontal reviews across a single portfolio and determining issues that crop up across institutions in that portfolio. Staff said that the data can be used to identify common issues as they relate to Board guidance. Staff said that the data also are used to determine whether MRAs and MRIAs are closed in a timely manner, both across portfolios and at a granular level—tracking the progress of individual firms. The data are aggregated across all supervision portfolios. OCC staff told us that they use MRA data to track the number of MRA concerns issued, amount of time open, the types of supervisory concerns for which an MRA was issued, and other information useful to OCC supervisory offices and the National Risk Committee. OCC conducts analysis of supervisory concern data in aggregate. Quarterly reports aggregate trends (including number of concerns, whether concerns are increasing or decreasing, and the number of banks with these concerns). For example, OCC analyzes the data by lines of business, examination areas, categories, and primary risk, which helps track existing risks and growing risks and whether MRA concerns have been escalated to enforcement actions. OCC staff said that data regarding aging of MRAs, which can raise visibility of longstanding concerns, are of particular interest to the National Risk Committee, which we observed in internal reports summarizing supervisory concern data. The regulators have internal tracking systems and policies and procedures to record and track examination data but FDIC does not collect certain data in a manner that provides management with comprehensive information to fully monitor the effectiveness of supervision activities. The Federal Reserve System has two systems for recording and tracking supervised institution data: the “C-SCAPE” platform for institutions with assets greater than $50 billion and all foreign banks, and the “INSite” platform for smaller community banks. Each Reserve Bank has issued guidance on recording MRAs and MRIAs specific to the examiners at those Reserve Banks. The MRA and MRIA data are recorded under a broad area of supervisory focus (for C-SCAPE) or MRA and MRIA category (for INSite), with subcategories for the name and description of the issue for greater detail. OCC’s supervisory information system is Examiner View, in which examiners record, update, and view MRAs. The baseline for the required fields is documented in OCC’s policy and procedures manuals on MRAs and Examiner View, as well as in a supplemental memorandum for large bank supervision. Since March 2017, the data have been recorded in a four-level concern framework (examination area, category, concern type, and topic), as determined by a cross- agency working group under OCC’s National Risk Committee. FDIC supervisory data are collected and retained in various systems. Supervisory recommendations are maintained (by institution) in text format in a separate system that is not readily searchable. FDIC maintains information on MRBAs that are not included in an enforcement action in the Supervisory Tracking and Reporting module of the ViSION system. Supervisory recommendations and MRBAs issued to large institutions supervised by FDIC are also tracked in spreadsheets by examination teams. Supervisory recommendations contained in an enforcement action are collected and tracked in the Formal and Informal Actions Tracking system. In 2017, FDIC updated its MRBA policies and procedures to require that examiners enter summary information into ViSION about individual MRBA events, rather than an overall summary of all MRBA events during an examination. But the summary approach means that MRBA data are not categorized at different levels (from a broad level such as examination area to more specific levels, including risk or concern type). Federal internal control standards state that management should use quality information to achieve objectives. Quality information is defined as appropriate, current, complete, accurate, accessible, and provided on a timely basis. Federal internal control standards also stress the importance of management conducting ongoing monitoring of the internal control system, which includes regular management and supervisory activities, comparisons, reconciliations, and other routine actions. As noted above, FDIC policies and procedures do not require examiners to record MRBAs under different categories in the MRBA reporting and tracking system. Instead, FDIC Risk Management Supervision staff is responsible for analyzing summary MRBA data entered by examiners and then categorizing the data for FDIC management reports. These categories are based on staff expertise rather than the experience of examiners in the field who developed the MRBAs. A structure that examiners could use to record more granular details about MRBAs directly after examinations would help ensure that reports prepared for FDIC management are not missing important details about FDIC MRBAs. Currently, FDIC management lacks complete information to better monitor the effectiveness of supervision activities in remediating emerging risks in a timely manner. Our analysis of supervisory concern data and federal banking regulators’ internal reporting based on the data indicate that management weaknesses at depository institutions of all sizes continued to exist through 2017. The number of supervisory concerns issued for all concern categories decreased each year during 2012–2016. All the regulators frequently cited management as a primary risk area in the supervisory concerns issued during the period. For instance, management and board and loan and credit administration were the largest of 14 categories of MRBAs issued by FDIC in 2012–2016, each constituting about 22 percent of all MRBAs. Corporate governance was the largest of 26 categories of MRAs issued by the Federal Reserve in that period, constituting approximately 19 percent of all MRAs. The next largest category of MRAs issued was credit risk management at 13 percent. Enterprise governance and operations was the third-largest of 16 examination areas of MRA concerns issued and closed by OCC in 2012–2016, constituting about 11 percent of all MRA concerns. The largest examination area of MRA concerns issued was credit at about 37 percent, followed by bank information technology at 13 percent. Similarly, internal reports from the regulators for late 2016 through 2017 indicated that supervisory concerns about management’s ability to control and mitigate risk at depository institutions continued. Our review of the reports showed that corporate governance issues were among the most common categories for issued supervisory concerns. In addition, the Federal Reserve reported in November 2018 that governance and controls issues constituted about 70 percent of outstanding supervisory concerns for the Large and Foreign Banking Organizations portfolio. Our review of supervisory concern data from the Federal Reserve and OCC from 2012 through 2016 generally showed that the amount of time concerns remained open was reduced (for example, see figure 2 for data on the supervisory concerns issued most frequently by the Federal Reserve and OCC during the period). Federal banking regulators told us that they have made efforts in recent years to have institutions remediate the deficiencies that cause supervisory concerns. FDIC data regarding MRBAs were limited and we were not able to determine how long MRBAs remained open by type of concern. Federal Reserve data indicated that the average amount of time needed to close corporate governance MRAs changed from 568 days in 2012 to 155 days in 2016. The time to closure for corporate governance MRAs ranged from 3 to 1,605 days for 2012-2016. Time to closure for credit risk-management concerns, the second-largest MRA category for the Federal Reserve, saw a similar decrease (from 431 days on average in 2012 to 246 days on average in 2016). For OCC, the average time to closure for enterprise governance and operations MRAs decreased from 517 days in 2012 to 245 days in 2016. The time to closure for enterprise governance and operations MRA concerns ranged from 7 to 1,724 days in 2012-2016. Time to closure for OCC’s largest MRA examination area (credit concerns) decreased from 445 days on average in 2012 to 241 days on average in 2016. Federal banking regulators vary in the nature and extent of data they collect on escalation of supervisory concerns to enforcement actions. As noted above, under their progressive enforcement regimes, the regulators may take informal or formal enforcement action against an institution if it does not respond to a supervisory concern in a timely manner. OCC collects data on escalation of supervisory concerns to enforcement actions. These data show that about 2,300 MRA concerns, or about 10 percent of all MRA concerns, were escalated to enforcement actions from 2012 through 2016. Of this amount, 18 percent related to enterprise governance and operations concerns, the second-largest number of escalated MRA concerns behind credit concerns at 41 percent. Federal Reserve data for escalation of MRAs to MRIAs and enforcement actions were collected in a manner that made it difficult for us to reliably determine the extent to which escalation occurred. Therefore, we did not use the Federal Reserve’s escalation data. FDIC does not track escalation of supervisory concerns in a manner that allowed us to determine the extent to which escalation occurred. FDIC and OCC have relatively detailed policies and procedures for escalation of supervisory concerns to enforcement actions, while the Federal Reserve has broad guidelines. Although the Federal Reserve tracks escalation of supervisory concerns, as noted above, Federal Reserve policies and procedures do not delineate specific factors for examiners to follow in deciding whether to identify a concern as warranting possible enforcement action. Instead, the Federal Reserve provides broad guidelines; for instance, stating only that informal enforcement actions are tools used when circumstances warrant a less severe form of action than formal enforcement actions. Federal Reserve staff told us that in practice the facts and circumstances of the case dictate when escalation is appropriate. They said that they take into account the institution’s response to prior safety and soundness actions against the institution and determine whether the institution’s conduct meets enforcement action standards. However, the Federal Reserve has not defined specific and measurable guidelines for when a supervisory concern would require escalation to a more formal regulatory action (such as an enforcement action). In contrast, FDIC and OCC have relatively detailed guidelines for escalating concerns. For example, FDIC guidelines published in 2016 instruct examiners to consider several factors, including management’s attitude towards complying with laws and regulations and correcting undesirable or objectionable practices; management’s history of instituting timely remedial or corrective actions; and whether management established procedures to prevent future deficiencies or violations. Similarly, OCC guidelines published in 2017 instruct examiners to consider several factors, including the board and management’s ability and willingness to correct deficiencies within an appropriate time frame; the nature, extent, and severity of previously identified but uncorrected deficiencies; and the bank’s progress in achieving compliance with any existing enforcement actions. Federal internal control standards provide that management conducts risk assessment to develop appropriate risk responses. Key attributes of effective risk assessment are definitions of objectives and risk tolerances, and management defines risk tolerances in specific and measurable terms so they are clearly stated and can be measured. In assessing risks that might necessitate an enforcement action, the Federal Reserve’s guidelines do not provide its examiners with guidance as to the acceptable level of variation in an institution’s performance relative to the achievement of supervision objectives. Without formalized, specific, and measurable guidelines for escalation of supervisory concerns, the Federal Reserve relies on the experience and judgment of examiners, Reserve Bank management, and Federal Reserve staff to determine when escalation is appropriate. Reliance on a single mechanism or tool can be risky. For instance, institutional knowledge can disappear in times of turnover, such as occurred after the 2007–2009 financial crisis. In addition, reliance on judgement alone can produce inconsistent escalation practices across Reserve Banks and supervision teams. Federal banking regulators have strengthened their approach to oversight of management at large depository institutions since 2009. This stronger approach is important as management weaknesses can reflect an institution’s underlying risk. However, we identified areas where written communication of supervisory concerns to institutions and monitoring of supervisory data at FDIC and the Federal Reserve could be strengthened. The communications of supervisory concerns from FDIC and the Federal Reserve did not fully convey why a practice at a depository institution was deficient and, for the Federal Reserve, the effect of the deficient practice on safety and soundness. Complete information about deficiencies is essential to ensuring timely corrective action by senior bank management before the deficiencies negatively affect safety and soundness at the institution. Furthermore, we identified data gaps in FDIC’s recording of MRBAs that resulted in incomplete information for FDIC management on supervisory concerns. Complete supervisory concern information would allow FDIC management to fully monitor the effectiveness of supervision activities (that is, to remediate risks in a timely manner). Finally, the Federal Reserve lacks specific, measurable guidelines for escalating supervisory concerns. Although escalation of a supervisory concern can depend on the facts and circumstances of the case, a lack of formalized, specific, and measurable guidelines for escalation of supervisory concerns could result in inconsistent escalation practices across Reserve Banks and examination teams. We are making a total of four recommendations: two to FDIC and two to the Federal Reserve. The Director of the Division of Risk Management Supervision of FDIC should update policies and procedures on communications of supervisory recommendations to institutions to provide more complete information about the recommendation, such as the likely cause of the problem or deficient condition, when practicable. (Recommendation 1) The Director of the Division of Supervision and Regulation of the Board of Governors of the Federal Reserve System should update policies and procedures on communications of supervisory concerns to institutions to provide more complete information about the concerns, such as the likely cause (when practicable) and potential effect of the problem or deficient condition. (Recommendation 2) The Director of the Division of Risk Management Supervision of FDIC should take steps to improve the completeness of MRBA data in its tracking system, in particular, by developing a structure that allows examiners to record MRBAs at progressively more granular levels (from a broad level such as examination area to more specific levels, including risk or concern type). (Recommendation 3) The Director of the Division of Supervision and Regulation of the Board of Governors of the Federal Reserve System should update policies and procedures to incorporate specific factors for escalating supervisory concerns. (Recommendation 4) We provided a draft of this report to FDIC, the Federal Reserve, and OCC for review and comment. During their review of the draft report, FDIC and the Federal Reserve provided oral comments about Recommendations 1 and 2 (to update policies and procedures for communication of supervisory concerns to provide more complete information, such as the likely cause and, for the Federal Reserve, potential effect). We modified the respective recommendations to address technical issues raised by their comments. FDIC provided written comments that are summarized below and reprinted in appendix IV. FDIC disagreed with Recommendation 1 and agreed with Recommendation 3. More specifically, FDIC stated that its current instructions to examiners meet the intent of Recommendation 1 (to update policies and procedures for communicating supervisory recommendations to provide more complete information). In particular, FDIC cited its policies and procedures on drafting supervisory recommendations in the report of examination, which include a section entitled, “Explain the Basis for any Supervisory Recommendations or Concerns.” FDIC stated this instruction requires examiners to communicate why there is a concern within the supervisory recommendation. Furthermore, FDIC issued an internal memorandum in October 2018 that reminds examiners to take prompt action to address root causes of deficiencies in complex and changing situations. FDIC stated that it began training in 2018 on developing strong enforcement action provisions to address root causes of deficiencies at problem banks, which continues in 2019. We describe FDIC’s policies and procedures in our report and agree that examiners are instructed to communicate why they are concerned about a deficient condition. However, examiners are not instructed to communicate what they believe to be the root cause of the deficient condition. We are encouraged that FDIC agrees it is important to identify root causes when addressing deficiencies in problem bank corrective actions. Nevertheless, the emphasis on identifying root cause is not found in examination policies and procedures. If, as FDIC indicated, examiners already identify the root causes of deficiencies during bank examinations, then FDIC can address our recommendation by formalizing that process in its policies and procedures. For Recommendation 3 (to improve MRBA data in its supervisory recommendations tracking system, by developing a structure that allows recording of MRBAs at more granular levels), FDIC agreed that a structure should be enhanced to allow staff to further categorize MRBAs at the point of entry into the system. FDIC further agreed that input of more granular information about MRBAs directly after examinations should provide the functionality to track an MRBA from a broad level such as examination to more specific levels, including concern type. The Federal Reserve provided written comments summarized below and reprinted in appendix V. The Federal Reserve did not state whether it agreed or disagreed with Recommendations 2 and 4 but responded that it would take our recommendations into consideration. For Recommendation 2 (to update policies and procedures for communicating supervisory concerns to provide more complete information, such as likely cause (when practicable) and potential effect), the Federal Reserve stated it recognizes that more effectively communicating supervisory concerns may achieve faster resolution of identified deficiencies and ultimately promote a more resilient banking system. The Federal Reserve noted it issued proposed guidance in August 2017 (which we discuss in the report) that would, in part, clarify expectations for communications of supervisory concerns, and that it continues to evaluate commenters’ suggestions. The Federal Reserve stated that it will consider ways to update its policies and procedures consistent with our recommendation. For Recommendation 4 (to update policies and procedures to incorporate specific factors for escalating supervisory concerns), the Federal Reserve stated it appreciated our recognition that the decision to escalate a supervisory concern ordinarily depends on the particular facts and circumstances of each case. The Federal Reserve stated that it will consider whether there are specific factors that staff should consider when escalating supervisory concerns. The Federal Reserve and OCC also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Chairman of the Board of Governors of the Federal Reserve System, the Chairman of the Board of Directors of FDIC, and the Comptroller of the Currency. This report will also be available at no charge on our website at http://www.gao.gov. Should you or your staff have questions concerning this report, please contact me at (202) 512-8678 or clementsm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI. This report examines (1) the extent to which federal banking regulators’— the Federal Deposit Insurance (FDIC), Board of Governors of the Federal Reserve System (Federal Reserve), and Office of the Comptroller of the Currency (OCC)—revised policies and procedures for supervision of management at large depository institutions were consistent with leading risk-management practices; (2) how examiners applied agency policies and procedures for supervision of management at large depository institutions they oversee; and (3) trends in regulators’ supervisory concern data for all depository institutions since 2012 and how regulators tracked and used such data. To address all our objectives, we focused on risk-management issues, such as those related to corporate governance, internal controls, and internal audit because management weaknesses in these areas could threaten the safe and sound operation of a depository institution. We selected this approach because recent GAO reports have addressed risk- management issues related to financial conditions such as capital and liquidity requirements, stress testing, and commercial real estate risk. We reviewed relevant federal laws and regulations, including sections of the Federal Deposit Insurance Act, Federal Reserve Act, National Bank Act, and interagency regulations on safety and soundness. We reviewed prior GAO reports, including reports on quantitative risk-management issues as they relate to financial condition, supervision of compliance with laws and regulations, and regulatory capture in bank supervision. We reviewed reports from the Offices of Inspector General for the federal banking regulators. We also drew on prior and on-going work related to regulatory capture in bank supervision. In addition, we reviewed the 2013 OCC-commissioned assessment of OCC’s supervision of large and mid-size institutions. We interviewed staff at FDIC, Federal Reserve, and OCC about examination policies and procedures for large depository institutions, processes related to supervision of management at such large institutions, and use of supervisory concerns to address weaknesses they identified. We interviewed staff in the Office of the Inspector General at each banking regulator. We also interviewed three industry representatives with prior experience in bank supervision to obtain their perspectives on bank examinations and supervisory concerns. For this objective, we took steps to identify relevant changes to examination approaches and processes (focusing on oversight of qualitative risk-management activities and communication of supervisory concerns). First we obtained confirmation from the regulators of the list of policies and procedures and other guidance documents we identified for review and solicited suggestions for additional documents to review. We then reviewed and analyzed guidance the agencies issued to examiners and depository institutions, relevant to (1) assessment of board and senior management’s management of risks, (2) metrics used to measure risk, and (3) assessment of depository institutions’ internal controls and audit procedures. Specifically, we reviewed and described regulators’ policy and procedural manuals, supervisory statements, and other supervisory guidance issued since 2009 to identify changes to the agency’s approach and process subsequent to the financial crisis. We focused primarily on changes to address oversight of risk management. We then reviewed documents from several standard-setting organizations to identify criteria for assessing risks and risk management. More specifically, we reviewed federal internal control standards; Internal Control - Integrated Framework of the Committee of Sponsoring Organizations of the Treadway Commission (COSO); safety and soundness standards developed by the federal banking regulators; Core Principles for Effective Banking Supervision of the Basel Committee on Banking Supervision; Federal Reserve’s enhanced prudential standards regulation, which applies to bank holding companies with assets greater than $10 billion and thus applies to the bank holding companies that own the depository institutions within the scope of our review; and GAO reports developing risk-management frameworks for government entities. Based on these documents, we selected a list of criteria to use in assessing the regulators’ risk-management guidance for examining large depository institutions (see table 3). We made connections between the principles listed in each of the documents to highlight the key elements of risk assessment, risk measurement, corporate governance, internal controls, and internal audit requirements. Additionally, we factored in regulators’ consideration of compliance with laws and regulations in their evaluation of the management component of CAMELS (capital adequacy, asset quality, management, earnings, liquidity, and sensitivity to market risk). Specifically for the first three criteria, we considered principles from GAO Standards for Internal Control, COSO’s Integrated Framework, the federal banking regulators’ safety and soundness standards, and the Federal Reserve’s risk management regulation. Additionally, for the second criterion we considered the Basel Committee on Banking Supervision Core Principles for Effective Banking Supervision. For the fourth criterion we considered the regulators’ safety and soundness standards. We also identified sub-criteria to help determine the extent to which the regulators’ guidance to address past supervisory weaknesses aligned with the criteria. Our baseline for the sub-criteria related to the first criterion was that the guidance communicates the need for clear lines of authority and responsibility for monitoring internal controls. The baseline for the sub-criteria related to the second criterion was that the guidance require independence of the risk management function. For the sub- criteria related to the third criterion, the baseline was that the guidance provide for identification of and timely action to address existing and emerging risks. Finally, for the sub-criteria related to the fourth criterion we looked for guidance to require compliance with laws and regulations, which regulators considered in the evaluation of management performance. For this objective, we undertook a multistep process to select institutions from which to obtain examination documents for review. First, we obtained the lists of institutions subject to examination by the regulators’ large bank examination programs in recent years. For FDIC, these were institutions with total assets of $10 billion or more; for the Federal Reserve and OCC, generally, these were institutions with assets greater than $50 billion. More specifically, we obtained a listing of all FDIC-supervised institutions in its Large Insured Depository Institution program that were subject to examination from June 2013 through March 2017, all Federal Reserve member banks in its Large Banking Organization portfolio as of December 2016, and all OCC-supervised institutions in its Large Bank Supervision portfolio from 2012 to 2016. Next, we selected a non-generalizable sample of three depository institutions from each of the regulators (nine in total) for which to request 2014-2016 examination documents for review. To assemble the sample, we determined the asset size of each institution supervised by the regulators’ large bank examination program as of December 2016, and selected institutions with a range of asset amounts. If these institutions were from the same geographic area (supervised by the same regional office or Reserve Bank), we selected other institutions with comparable asset amounts in order to have geographic dispersion in our sample. The purpose of this selection approach was to assess whether material differences existed in examinations conducted by the different regional offices in our sample. Also, if the selected institutions were headquartered in a foreign country, we selected other institutions with comparable asset amounts. The purpose of this selection approach was to omit institutions with only a branch office in the United States, which would allow the regulator to only examine a portion of the institution’s operations. In addition, if the selected institutions were not primarily engaged in traditional banking activities, we selected other institutions with comparable asset amounts. To make this determination, we conducted a separate analysis to determine if (1) the institutions engaged in traditional banking activities (accepting deposits and making consumer loans), (2) traditional banking activities made up a majority of the bank’s activities as recorded on the balance sheet, and (3) the bank’s loan activities were primarily domestic. The purpose of this selection approach was to omit companies that primarily conduct “non-traditional” banking activities such as investment banking and credit cards but have a regulated depository institution to support those activities. We conducted a separate analysis of OCC-supervised institutions in its Large Bank Supervision portfolio because a number of entities were nationally chartered banks under a foreign holding company or were not primarily depository institutions. In our analysis, we first determined if (1) an institution engaged in traditional banking activities, (2) traditional banking activities made up a majority of its activities as recorded on the balance sheet, and (3) the institution’s loan activities were primarily domestic. We included three federal savings banks in our universe of OCC-supervised institutions because we determined they were subject to many of the same supervision policies and procedures as national banks. We then determined that the geographic location of the examiners-in- charge for the institutions in the Large Bank Supervision portfolio determined the regional office to which the examiner-in-charge reported. To obtain geographic dispersion, we based our selection on the location of the examiners-in-charge to ensure that each examiner was associated with a different regional office. Using these criteria and considerations, we selected small, moderate, and large OCC-supervised institutions. To determine how regulators applied agency policies and procedures for supervision of management during examinations of large depository institutions, we requested selected examination documents from the regulators for the nine institutions we selected. For FDIC, initially we requested 2016 examination documents for the three selected large institutions subject to the Large Insured Depository Institution program. For the Federal Reserve, we initially requested 2016 examination documents for the three selected large institutions subject to the Large Banking Organization program. For OCC, we initially requested 2016 examination documents for the three selected large national banks subject to the Large Bank Supervision program. We reviewed these examination documents to learn how examiners reviewed qualitative risk-management issues, such as those relating to the management component of CAMELS. Based on our initial review, we submitted another document request to the regulators. FDIC. Through our initial review of FDIC documents, we identified the risk categories for which FDIC examined corporate-wide risk-management functions. We then requested relevant examination documents for each of the three FDIC-supervised institutions, such as scope, summary, and conclusion memorandums and supervisory letters related to corporate-wide risk-management functions and the Bank Secrecy Act; examination documentation for supervisory recommendation (remediation) follow-up reviews that were reviewed during the 2014, 2015, and 2016 supervisory cycles; summary examination documents related to ongoing monitoring work; explanation of planned target review areas that appeared to cover review of corporate-wide risk-management functions for the same supervisory cycles that had not been completed; and supervisory plans and reports of examination for 2014 and 2015 examination cycles. In total, we reviewed 94 FDIC examination documents. We took as criteria the examination procedures from the examination documentation modules referenced in FDIC’s Basic Examination Concepts and Guidelines and the Management portion of the agency’s examination policy manual. We also incorporated elements of other FDIC policies and procedures, such as those relating to internal routine and controls, dominant officials, and incentive compensation. Our criteria also included FDIC memorandums to assess communication and follow- up on supervisory recommendations, including matters requiring board attention (MRBA). Finally, we used information on enforcement policies and procedures in the agency’s Report of Examination Instructions manual. Federal Reserve. Based on our initial review, we requested conclusion memorandums and supervisory letters (letters of findings) pertaining to several targeted and enhanced continuous monitoring examinations the Federal Reserve conducted during the 2014, 2015, and 2016 supervisory cycles at the three institutions we selected. In total, we reviewed 83 Federal Reserve examination documents. To assess how examiners applied agency policies and procedures, we used examination procedures contained in the Commercial Bank Examination Manual for most of our criteria. In particular, the Commercial Bank Examination Manual includes a section on “Assessment of the Bank” with detailed examination procedures for review of boards of directors, management, internal controls, and audit. In addition, we used guidance from supervision and regulation letters to the extent the information was not incorporated in the manuals. OCC. Based on our initial review, we requested examination documents for targeted and ongoing examination work related to enterprise risk management, operational risk, and other safety and soundness (management) for the 2014, 2015, and 2016 examinations cycles. Specifically, we requested ongoing supervision memorandums, conclusion memorandums, supervisory letters, and risk assessments. We also requested the supervisory strategy and report of examination for the 2014 and 2015 examination cycles. In total, we reviewed 268 OCC examination documents. As criteria, we applied examination procedures from the Large Bank Supervision booklet for certain risk elements related to bank governance and management. We also applied examination procedures for internal control and audit as criteria. In addition, we included agency guidance on follow-up for matters requiring attention (MRA) and enforcement action. We then developed questions to assess the examination documents based on the criteria we selected. See appendix III for our list of questions. Using a data collection instrument populated with the selected questions, we assessed each of the regulators’ examination documents. To demonstrate how examiners applied each criterion, we either took language from the examination document or included explanatory language of what the examiner did during the examination to assess risk management. We also tracked the examiner’s findings on each individual risk area we reviewed to the annual report of examination to ensure that the risk was considered in the context of the entire institution. The results of our review of depository institution examination reports and examination documents are not generalizable to all of the regulators’ examination reports and documents. Each individual review serves as an independent assessment of the examiners’ application of relevant agency guidance. To evaluate the extent to which the federal banking regulators ensured that large depository institutions addressed risk management-related supervisory concerns, such as MRA, and addressed supervisory concerns since 2012, we (1) analyzed the regulators’ policies and procedures for escalating supervisory concerns to enforcement actions, and (2) analyzed aggregate supervisory concern data from 2012 to 2016 for all institutions supervised by FDIC, the Federal Reserve, and OCC. We did not collect data on all the different types of supervisory concerns issued. In particular, we did not collect data on supervisory recommendations by FDIC and matters requiring immediate attention (MRIA) by the Federal Reserve. Therefore, our analysis of the data does not provide a complete representation of the status of supervisory concerns issued by the regulators. To examine trends, we requested that each regulator provide the data by risk category so that we could analyze whether certain risk areas generated more timely resolution of risk management-related supervisory concerns and whether supervisory concerns were elevated to enforcement actions. FDIC. Because of the current structure of FDIC’s data collection and storage systems, FDIC could not provide data on MRBA in a format that would have been easily analyzable for our purposes. Specifically, FDIC examiners enter summary information about MRBAs into the system with no categorization by examination or risk area. FDIC provided us two data sets—raw data downloaded from its ViSION system; and a data set sorted by topics, which was prepared by the FDIC Emerging Risks section and used for publication in FDIC’s Supervisory Insights newsletter. For large institutions, FDIC informed us that the data were not complete because MRBAs reflected in ViSION were those that remained open at the end of the year when the annual report of examination was issued and that MRBAs opened and closed during the examination cycle were not recorded in the system. Due to the limitations with the data and the inability to combine the data sets, some analyses were completed with the raw data set and others with the data set divided by topics. As a result, the analysis provides a general understanding of trends in FDIC supervisory concerns, rather than a rigorous trend analysis. Federal Reserve. We obtained data on MRAs issued to all Federal Reserve-supervised institutions from 2012 through 2016. The Federal Reserve has two systems for recording and tracking supervised institution data: the “C-SCAPE” platform for institutions with assets greater than $50 billion and all foreign banks, and the “INSite” platform for smaller community banks. Some of the MRA data were not categorized by supervisory concern and were assigned a “null” value. According to Federal Reserve staff, in 2012 the Federal Reserve migrated from a legacy tracking system to the current C-SCAPE platform. The MRA data contain both broad MRA categories and sub-categories for greater detail. For ease of explanation and analysis, the data under the sub-categories were consolidated under their larger categories. The number of MRAs uncategorized by supervisory concern did not present a significant obstacle to our analysis. The data on escalation of MRAs to MRIAs and enforcement actions were collected in a manner that made it difficult for us to determine the extent of escalation. Specifically, the glossary that was provided with the data stated that issues closed through the “transformation process” are marked “closed,” and are distinguished from other closed issues by indicating how they were closed (for example, transformed to MRA, transformed to MRIA, or transformed to provision). We determined that any results we produced regarding escalation would be unreliable given the lack of clarity around data collection methods. OCC. We obtained MRA data from OCC that included records opened from January 2012 through December 2016. OCC’s supervisory information system is Examiner View, in which examiners record, update, and view MRAs (among other things). For our purposes, OCC staff stated that we could use the data to count the number of concerns; however, analyzing the concerns by categories could have been problematic because of changes to the classification method that occurred in October 2014 and March 2017. As a result of the 2017 changes, OCC supervisory concern data are recorded in a four-level framework (examination area, category of concern, type, and topic) that allows for tracking of supervisory concerns at the MRA level and at the “concern” level. Before 2017, the information was classified differently. The newer data allow for enhanced trend analysis and risk identification. We were able to analyze OCC data to show the MRAs issued in 2012– 2016 by exam area. We also could show trends in risk management- specific exam areas, as well as the average time it took to close risk- management specific concerns. Furthermore, we obtained and analyzed data on MRAs that were escalated to enforcement actions. For all the regulators, we assessed the reliability of the data. First, we interviewed staff at each of the regulators who were knowledgeable about the data. We asked for the source of the data, how frequently it was updated, and about the controls in place to ensure the data were accurate and complete. Additionally, in assessing the reliability of the data, we reviewed internal reports and other documents prepared by the regulators. Specifically, for FDIC we reviewed management reports for each quarter of fiscal year 2017. For the Federal Reserve, we analyzed draft 2017 annual assessment letters, feedback from the Operating Committee of the Large Institution Supervision Coordinating Committee to dedicated supervisory teams, and other organizing documents. For OCC, we analyzed management reports to different oversight committees for calendar year 2017. While the data did not allow all of the analysis we had planned to complete, overall, we determined that the FDIC, Federal Reserve, and OCC data were reliable for purposes of showing general trends in the number of supervisory concerns, the time frames for closing supervisory concerns, and—additionally for OCC—the number of supervisory concerns escalated to enforcement actions. We conducted this performance audit from March 2017 to April 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix lists the federal banking regulators’ policy and procedure documents included in our review. Division of Risk Management Supervision Manual of Examination Policies – Basic Examination Concepts and Guidelines section (section 1.1), including relevant Financial Institution Letters and internal memorandums. Provides overview of the Federal Deposit Insurance Corporation (FDIC) bank examination process, including rationale for examinations; the Uniform Financial Institutions Rating System, also known as CAMELS (capital adequacy, asset quality, management, earnings, liquidity, and sensitivity to market risk); examination types; scheduling guidelines; and communication with management. Division of Supervision and Consumer Protection Risk Management Manual of Examination Policies – Management section (section 4.1), including relevant internal memorandums. Focuses on the management component of CAMELS ratings, with the main emphasis on the powers, responsibilities, and duties vested in bank directors. It also includes policies and procedures for identifying and assessing the influence of dominant bank officials. Division of Risk Management Supervision Manual of Examination Policies – Internal and Routine Controls section (section 4.2), including relevant internal memorandums. Discusses internal controls, internal control programs, management’s responsibilities, internal control and fraud review examination instructions, and includes a reference tool for examiners. Division of Risk Management Supervision Manual of Examination Policies – Informal Actions section (section 13.1) Identifies procedures for memorandums of understanding to address weak operating practices, deteriorating financial conditions, apparent violations of laws or regulations, or weak risk-management practices. Division of Risk Management Supervision Manual of Examination Policies – Formal Administrative Actions section (section 15.1) Identifies the statute and regulations that authorize the use of formal enforcement actions when necessary to reduce risks and address deficiencies, particularly when an insured state nonmember bank is rated 4 or 5 and evidence of unsafe or unsound practices is present. Division of Risk Management Supervision Manual of Examination Policies – Report of Examination Instructions section (section 16.1), including relevant Financial Institution Letters. Includes procedures for examiners to communicate supervisory recommendations (including matters requiring board attention and deviations from safety and soundness principles underlying policy statements) and identifies schedules for inclusion in reports of examination. Large Bank Supervision Procedures (internal manual), including relevant internal memorandum Describes procedures and processes (in three broad categories: planning, examination activities, and communication) for conducting continuous examination programs at state nonmember banks with total assets exceeding $10 billion. Supervisory Recommendations, Including Matters Requiring Board Attention (internal memorandum) Describes policies and procedures for scheduling supervisory recommendations (including matters requiring board attention) in reports of examination and for tracking bank management’s actions in response to these items after examinations. Pocket Guide for Directors and Statement Concerning the Responsibilities of Bank Directors and Officers The pocket guide describes FDIC’s expectations for boards of directors of institutions to carry out their duties. A second document, the statement, responds to concerns expressed by representatives of the banking industry and others regarding civil damage litigation risks to directors and officers of federally insured banks. Consolidated Supervision Framework for Large Financial Institutions (SR 12-17) Framework for consolidated supervision of large financial institutions with more than $10 billion in total assets. Bank Holding Company Supervision Manual Provides guidance to examiners as they conduct on-site inspections of bank holding companies and their nonbank subsidiaries. Provides guidance to examiners as they assess risk-management practices of state member banks, bank holding companies, and savings and loan holding companies (including insurance and commercial savings and loan holding companies) with less than $50 billion in total consolidated assets, and foreign banking organizations. Supervisory Considerations for the Communication of Supervisory Findings (SR 13-13/CA 13-10) Discusses the standard language the Federal Reserve uses to enhance focus on matters requiring attention and highlights supervisory expectations for corrective actions, Reserve Bank follow-up, and other supervisory considerations. Also defines matters requiring attention and matters requiring immediate attention and outlines procedures that safety- and-soundness and consumer compliance examiners will follow in presenting and communicating their supervisory findings. Framework for Risk-Focused Supervision of Large Complex Institutions, including relevant supervision and regulation letter (SR 97-24) Describes aspects of the Federal Reserve’s program to enhance the effectiveness of its supervisory processes for state member banks, bank holding companies, and the U.S. operations of foreign banking organizations. Rating the Adequacy of Risk Management Processes and Internal Controls at State Member Banks and Bank Holding Companies (SR 95-51) Directs examiners to assign separate rating for risk management to state member banks and bank holding companies with $50 billion or more in total assets, and highlights the importance of risk management as a facet of the supervisory process. Comptroller’s Handbook – Bank Supervision Process Includes explicatory materials on types of banks, supervision responsibilities, regulatory ratings, supervisory process, functional regulation, rating systems, and disclosure. Comptroller’s Handbook – Large Bank Supervision Outlines the supervisory process for large banks: the core assessment, risk assessment system, evaluation of bank internal control, and audits. Comptroller’s Handbook – Corporate and Risk Governance Focuses on management of a variety of risks and the roles and responsibilities of the board of directors and senior management, and provides relevant examination procedures. Comptroller’s Handbook – Internal and External Audits Addresses risks inherent in the audit function (which compromises both internal and external audit functions) and the audit function’s role in managing risks. Also addresses internal and external audit functions’ effect on risk-management supervisory expectations and the regulatory requirements for prudent risk management. Includes guidance and examination procedures to assist examiners in completing bank core assessments affected by audit functions. Comptroller’s Handbook – Internal Controls Discusses the characteristics of effective controls to assist examiners and bankers to assess the quality and effectiveness of internal control. Describes OCC’s supervisory process for internal control reviews and the roles and responsibilities of boards of directors and management. Enforcement Action Policy (Policies and Procedures Manual 5310-3), internal memorandum Describes policy for taking appropriate enforcement action in response to violations of law, rules, regulations, final agency orders, and unsafe or unsound practices and conditions. Violations of Laws and Regulations (Bulletin 2017-18) Describes updated policies and procedures on violations of laws and regulations and provides the agency with consistent terminology for communication, format, follow-up, analysis, documentation, and reporting of violations. Articulates the level and type of risk the agency will accept while conducting its mission. Matters Requiring Attention (Policies and Procedures Manual 5400- 11), internal memorandum Describes procedures for examiners to identify and aggregate supervisory concerns into matters requiring attention including criteria, communication, and follow-up of concerns. Also describes the relationship between matters requiring attention and interagency ratings, OCC’s risk-assessment system and enforcement actions. Includes examiner tools in the appendixes. Risk Management of New, Expanded, or Modified Bank Products and Services (Bulletin 2004-20, replaced by Bulletin 2017-43) Outlines the expectations for national banks’ management and boards to implement an effective risk-management process to manage risks associated with new, expanded, or modified bank products and services. Guidance on Sound Incentive Compensation Policies 75 Fed. Reg. 36395 (June 25, 2010) Interagency statement on sound incentive compensation practices to banking organizations supervised by FDIC, the Board of Governors of the Federal Reserve System (Federal Reserve), and the Office of the Comptroller of the Currency (OCC). It is intended to assist banking organizations in designing and implementing incentive compensation arrangements and related policies and procedures that effectively consider potential risks and risk outcomes. This appendix lists the questions we used to determine how federal bank examiners applied their policies and procedures to assess management oversight of risk at large depository institutions. We found that each federal banking regulator has slight variation in its policies and procedures for oversight of management at large depository institutions. Therefore, we did not apply generally applicable criteria in our assessment; instead, we applied the specific policies and procedures used by each federal banking regulator. Federal Deposit Insurance Corporation: 1. To what extent did examiners assess board and management oversight? 2. To what extent did examiners assess the bank’s control environment, including whether management takes appropriate and timely action to address recommendations by auditors and regulatory authorities? 3. To what extent did examiners assess the bank’s risk assessment? 4. To what extent did examiners assess the bank’s control activities, to include determining if policies, procedures, and practices were adequate for the size, complexity, and risk profile of the bank and if management took appropriate steps to comply with laws and regulations? 5. To what extent did examiners assess the bank’s information and communication, to include adequacy of information systems to identify, capture, and report relevant internal and external information? 6. To what extent did examiners assess the bank’s systems in place to monitor risk arising from all major activities the bank is engaged in with respect to b. legal risk, and c. reputation risk? 7. In identifying matters requiring attention, did examiners consistently explain the rationale for the concern (whether the matter deviates from sound governance or internal controls and how it could adversely impact the condition of the institution)? 8. In communicating matters requiring attention, did examiners a. write in clear and concise language b. describe the deficient practices, operations, or financial condition, c. recommend actions the board should take to address the deficiency? 9. What steps did examiners take to follow up on matters requiring attention and verify completion? 10. To what extent did the examiner comment on how the bank accomplished compliance with enforcement actions or the reason why the bank is not in compliance with enforcement actions? Conclusions: To what extent did examiners follow agency risk- management guidance for this examination? To what extent do the conclusion memorandums link to the supervisory letter and report of examination? Board of Governors of the Federal Reserve System: 1. Within the context of the consolidated financial entity, to what extent did examiners assess the bank’s implementation of its corporate governance framework? 2. Within the context of the consolidated financial entity, to what extent did examiners assess management of the bank’s core business lines? 3. To what extent did the examiners assess the bank’s board and management for active oversight of the bank, to include the extent to which examiners a. assessed the adequacy of the bank directors’ fulfillment of their duties and responsibilities; and b. assessed bank management’s fulfillment of their duties and responsibilities? 4. To what extent did examiners assess the adequacy of the bank’s policies, procedures, and limits? 5. To what extent did examiners assess the adequacy of the bank’s risk monitoring and management information systems? 6. To what extent did examiners assess the adequacy of the bank’s internal controls? 7. To what extent did examiners assess the adequacy of the bank’s audit function, to include a. internal audit staff, c. internal audit function adequacy and effectiveness, d. external audit staff, and e. regulatory examinations? 8. How did examiners assess the Management rating for CAMELS? 9. In identifying matters requiring attention, did examiners consistently explain the rationale for the concern? 10. In communicating matters requiring attention, did examiners a. write in clear and concise language, b. prioritize based upon degree of importance, and c. focus on any significant matters that require attention? 11. To what extent did examiners follow-up on matters requiring attention and verify completion? 12. To what extent did the examiner comment on how the bank accomplished compliance with enforcement actions or the reason why the bank was not in compliance with enforcement actions? Conclusions: To what extent did examiners follow agency risk- management guidance for this examination? To what extent do the conclusion memorandums link to the supervisory letter and report of examination? Office of the Comptroller of the Currency: 1. To what extent did the examiners assess the quantity and quality of b. reputation risk, c. operational risk, and d. compliance risk? 2. To what extent did the examiners assess the bank’s internal controls, d. accounting information, communication, and e. self-assessment and monitoring? 3. To what extent did the examiners assess the bank’s audit function, b. audit management and processes, c. audit reporting, and d. internal audit staff? 4. How did examiners assess the Management rating for CAMELS? 5. In identifying matters requiring attention, did examiners consistently a. deviates from sound governance, internal control, or risk management principles, and has the potential to adversely affect the bank’s condition, including its financial performance or risk profile, if not addressed; b. results in substantive noncompliance with laws and regulations, enforcement actions, supervisory guidance, or conditions imposed in writing in connection with the approval of any application or other request by the bank; or c. describes an unsafe or unsound practice. An unsafe or unsound practice is generally any action, or lack of action, which is contrary to generally accepted standards of prudent operation, the possible consequences of which, if continued, would be abnormal risk or loss or damage to an institution, its shareholders, or the Deposit Insurance Fund? 6. In communicating matters requiring attention, did examiners a. describe the concern(s); b. identify the root cause(s) of the concern and contributing factors; c. describe potential consequence(s) or effects on the bank from d. describe supervisory expectations for corrective action(s); and e. document management’s commitment(s) to corrective action and include the time frame(s) and the person(s) responsible for corrective action? 7. In follow-up on matters requiring attention, did examiners consistently a. monitor the board and management’s progress implementing b. verify and validate the effectiveness of the board and management’s corrective actions; c. perform timely verification after receipt of the documentation or communication from the bank that the documentation is ready for review; d. meet, as necessary, with the bank’s board or management to discuss progress assessments and verification results; and e. deliver written interim communications to the board summarizing the findings of validation activity? 8. To what extent did examiners verify and validate bank actions to comply with enforcement actions? Conclusions: To what extent did examiners follow agency risk- management guidance for this examination? To what extent do the conclusion memorandums link to the supervisory letter and report of examination? In addition to the contact named above, Karen Tremba (Assistant Director), Philip Curtin (Analyst in Charge), Enyinnaya David Aja, Bethany Benitez, Rachel DeMarcus, M’Baye Diagne, Risto Laboski, Yola Lewis, Christine McGinty, Kirsten Noethen, David Payne, Amanda Prichard, Barbara Roesmann, Jena Sinkfield, and Farrah Stone, made key contributions to the report.", "summary": "Weaknesses identified after the 2007–2009 financial crisis included management weaknesses at large depository institutions and the need for federal regulators (FDIC, Federal Reserve, and OCC) to address the deficiencies in a timely manner. Concerns remain that positive economic results of recent years could mask underlying risk-management deficiencies. This report examined (1) how consistent regulators' revised policies and procedures are with leading risk-management practices, (2) how they applied examination policies and procedures, and (3) trends in supervisory concern data since 2012 and how regulators tracked such data. GAO compared regulators' policies and procedures for oversight against leading practices; compared documents from selected bank examinations for 2014–2016 against regulator's risk-management examination procedures; reviewed aggregate supervisory concern data for 2012–2016; and interviewed regulators and industry representatives. Since 2009, federal banking regulators have revised policies and procedures for use by examiners in supervising depository institutions' management activities (such as those related to corporate governance and internal controls) and for identifying and communicating supervisory concerns. For example, regulators differentiated levels of severity for supervisory concerns and specified when to communicate them to boards of directors at the depository institutions. GAO found that the updated policies and procedures generally were consistent with leading risk-management practices, including federal internal control standards. Examination documents that GAO reviewed showed that examiners generally applied the regulators' updated policies and procedures to assess management oversight at large depository institutions. In particular, for the institutions GAO reviewed, the regulators communicated deficiencies before an institution's financial condition was affected, and followed up on supervisory concerns to determine progress in correcting weaknesses. However, practices for communicating supervisory concerns to institutions varied among regulators and some communications do not provide complete information that could help boards of directors monitor whether deficiencies are fully addressed by management. Written communications of supervisory concerns from the Federal Deposit Insurance Corporation (FDIC) and the Board of Governors of the Federal Reserve System (Federal Reserve) that GAO reviewed often lacked complete information about the cause of the concern and, for the Federal Reserve, also lacked information on the potential consequences of the concern, which in one instance led to an incomplete response by an institution. Communicating more complete information to boards of directors of institutions, such as the reason for a deficient activity or practice and its potential effect on the safety and soundness of operations, could help ensure more timely corrective actions. While supervisory concern data indicated continuing management weaknesses, regulators vary in how they track and use the data. Data on supervisory concerns, and regulators' internal reports based on the data, indicated that regulators frequently cited concerns about the ability of depository institution management to control and mitigate risk. However, FDIC examiners only record summary information about certain supervisory concerns and not detailed characteristics of concerns that would allow for more complete information. With more detailed information, FDIC management could better monitor whether emerging risks are resolved in a timely manner. In addition, the regulators vary in the nature and extent of data they collect on the escalation of supervisory concerns to enforcement actions. FDIC and the Office of the Comptroller of the Currency (OCC) have relatively detailed policies and procedures for escalation of supervisory concerns to enforcement actions, but the Federal Reserve does not. According to Federal Reserve staff, in practice they consider factors such as the institution's response to prior safety and soundness actions. But the Federal Reserve lacks specific and measurable guidelines for escalation of supervisory concerns, relying solely on the judgment or experience of examiners, their management, and Federal Reserve staff, which can result in inconsistent escalation practices. GAO recommends that FDIC and the Federal Reserve improve information in written communication of supervisory concerns; FDIC improve recording of supervisory concern data; and the Federal Reserve update guidelines for escalating supervisory concerns. FDIC disagreed with the first recommendation, stating its policies address the issue, but GAO found clarification is needed. FDIC agreed with the second recommendation. The Federal Reserve neither agreed nor disagreed with the recommendations.", "document_type": "gao"}
{"report": "The overall objective of the Army Facilities Standardization Program is to achieve savings and benefits in the programing, design, and construction of Army facilities of excellence. To meet AFSP’s objectives in a timely, efficient, and cost-effective manner, the Army established the nine Centers in 2006 to support the AFSP, as shown in figure 1. The AFSP operates under the direction of the Army Facilities Standardization Committee (Committee). As shown in figure 2 below, the Committee is chaired by the Assistant Chief of Staff for Installation Management (ACSIM) and composed of members from USACE and the U.S. Army Installation Management Command (IMCOM). Each of these offices has representatives who are either full-fledged or advisory members of the Centers of Standardization Management Board (the Board). The Board members directly oversee the activities of the Centers and are responsible for developing performance measures and reporting them to the Committee. The Centers have primary responsibility for developing and managing Army standard design packages for designated facility types. The Centers, among other things, ensure that these standard designs and construction of projects comply with two other sets of facility guidelines: DOD’s Unified Facilities Criteria (Facilities Criteria) and general Army standards. As we previously reported, the Facilities Criteria are overarching, DOD-wide technical manuals and standards used for planning, design, construction, restoration, and maintenance of DOD facility projects. These criteria must be used to the greatest extent possible by all DOD components. They are developed through the joint efforts of the U.S. Army Corps of Engineers, the Naval Facilities Engineering Command, and the Air Force Civil Engineer Center, and they are approved by the Engineer Senior Executive Panel of the Unified Facilities Criteria Program. According to Army Regulation 420-1, Army standards are the immutable, unchanging, required facility elements and criteria that define the fundamental purpose and function of a facility’s design and construction. These Army standards are authorized by the Committee. Army standard designs define the facility key components, features, and characteristics that must be included in the design and construction or major renovation of all facilities of the same type regardless of location, available funding, command preferences, or installation mission. Essentially, Army standard designs may consist of architectural and engineering drawings as well as written design specifications that a construction team can easily adapt or modify for site-specific requirements. Figure 3 below compares Army standard designs with Facilities Criteria and general Army standards. In addition to developing and managing Army standard design packages, the Centers’ staff function principally as engineering and architectural consultants within larger project teams as they monitor and oversee the appropriate use of Army standard designs (as well as any incorporated Army standards or Facilities Criteria). According to Centers officials, 12 full-time and 21 part-time staff are currently dedicated to the Centers. Staff are located in USACE headquarters in Washington, D.C., as well as in eight USACE districts and one Engineering and Support Center. Each Center specializes in and is responsible for specific facility types and their designs. While the Centers support the Army’s overall efforts for standardization, not every Army facility is built according to a standard design. Appropriate Centers staff are required to review every proposed Army construction project at its outset and, if an installation has requested a waiver from an existing Army standard or standard design, all voting members of the Committee may authorize waivers in accordance with certain procedures. According to Centers officials, Army standard designs have been developed for about 70 regularly constructed facility types out of the Army’s nearly 900 facility types. For example, the Army has standard designs for fire stations, chapels, dining facilities, and weapons storage. (See appendix II for a listing of the 70 facility types that currently have standard designs or for which standard designs are under development.) According to Centers officials, the Centers’ 70 facility types account for approximately 60 percent of Army Military Construction (MILCON) projects and represent an estimated 55 percent to 70 percent of the overall Army MILCON budget for any given year. (See appendix III for information on overall DOD standardization program, including the Navy and Air Force standard design programs.) In fiscal year 2019, the Centers reported a combined annual budget of about $6.2 million for their operations and personnel. The Centers identified and engaged in a number of activities designed to support the key objectives found in their charter and these activities are consistent with key principles and concepts in OMB guidance for a disciplined capital programming process. The Centers’ charter includes the following three objectives: (1) developing and refining Centers’ policies and processes; (2) assuring consistent application of standards of the Centers program; and (3) monitoring the Centers’ execution to meet the overarching objectives and priorities of the AFSP and standardization process. To meet the three objectives, the Centers engage in different activities throughout the military construction process. Figure 4 below shows the various points at which the Centers are involved in the life- cycle of a military construction project and examples of the activities in which the Centers engage. For example, Engineer Regulation 1110-3- 113 states that during the design phase of projects, the Centers maintain a lead role and will be the technical lead for coordination, review, and acceptance of design deliverables, including providing field technical assistance, identifying and advising when a waiver is required and coordinating with appropriate authorities in this matter, and reviewing and editing requests for proposal documents—activities that according to our analysis support the Centers’ second objective. Based on our review of supporting documentation from five projects that used standard designs, we found that the Centers were undertaking the activities mentioned above. In addition, activities in which the Centers engaged during the design, construction, and post-construction phases of these projects were consistent with key principles and concepts in OMB guidance. Specifically, we found evidence that, for these five projects, Centers’ staff participated as integrated members of the project delivery teams in planning meetings, design reviews, assessments of the need for standard design waivers, value engineering studies, and life-cycle cost analyses during the projects’ design and construction phases. These activities were consistent with key principles and concepts in OMB guidance for a disciplined capital planning process, including that agencies should use integrated project teams, as appropriate, to manage the various capital programming phases or major acquisition programs within the agency. In addition, we found that other Centers’ activities—performing post- occupancy evaluations (POE) and updating standard designs when applicable—were also consistent with key principles and concepts in OMB guidance for a disciplined capital planning process. For instance, we found that a POE was completed for one project, a post-occupancy questionnaire was completed for another project, a POE was planned during fiscal year 2020 for a third project, and a fourth project was still under construction. According to OMB capital programming guidance, POEs are tools to evaluate the overall effectiveness of an agency’s capital acquisition process. The primary objectives of a POE include (1) identifying how accurately a project meets its objectives, expected benefits, and strategic goals of the agency and (2) ensuring the continual improvement of an agency’s capital-programming process based on lessons learned. The guidance identifies factors to be considered for evaluation in conducting a POE, such as standards and compliance, customer/user satisfaction, and cost savings. The guidance also notes that a POE should generally be conducted 12 months after the project has been occupied, to allow time for the tenant to evaluate the building’s performance and relevant aspects of project delivery. However, the guidance allows agencies some flexibility in the timing of a POE to meet their unique needs if 12 months is not the optimal timing to conduct the evaluation. Our review of Centers guidance and project documents also found that the Centers’ activities supported the Centers’ objectives as well as AFSP objectives and priorities. In addition, Centers officials emphasized that the Centers participate in all Army standard design construction projects to ensure that the facility designs support the objectives of the AFSP, specifically improving the programing, design, and construction processes for Army facilities. As shown in table 1 and further outlined below, we assessed whether the Centers’ activities undertaken on standard design construction projects were applicable to the Centers’ objectives. Then, for those that were applicable, we determined whether those activities supported the Centers’ objectives. (See appendix IV for a detailed analysis of how the Centers activities support the program’s objectives.) Centers use POEs to evaluate standard designs: We found, for example, that the POEs led by the Centers are designed to evaluate whether the project met fundamental Army functional and mission requirements, whether the project implemented Army standard design, and whether improvements to the design could be made. These reviews support Centers objectives 1, 2, and 3— developing and refining Centers’ policies and processes, consistently applying Army standard designs, and supporting AFSP objectives and priorities—by identifying areas of the design needing improvement, evaluating whether a facility was constructed in accordance with the approved project design, and eliciting customer feedback concerning whether the finished facility meets mission requirements. Centers review standard design waivers: The Centers review an installation’s waiver request and advise whether a waiver to Army standards or standard designs is required for that specific project. This process supports Centers objectives 1, 2, and 3—developing and refining Centers’ policies and processes, consistently applying Army standard designs, and supporting AFSP objectives and priorities. Specifically, part of the waiver review and approval process is the Centers’ assessing whether a waiver request represents a unique need of a specific end user or a possible permanent change to the Army standard design or Unified Facilities Criteria. In addition, if the Centers waive the use of or approve deviations from standard design prior to the beginning of the construction phase, it may reduce the number of change orders that occur during construction. The Army, through its Centers of Standardization Management Board, is responsible for oversight of the Centers and has performance measures to track their progress in achieving one of their three key objectives. However, the Army does not have performance measures for assessing progress for their other two objectives. The Board provides oversight to the Centers in support of the AFSP. The Board members are responsible for developing, implementing, and reporting on program metrics. The Centers’ Charter of 2006 broadly identifies the mission and objectives of the Board, while more recent program guidance and regulations describe its functions in more detail. The Charter states that the mission of the Board is to provide corporate oversight and consistent Centers execution in support of the AFSP. In overseeing the Centers, it is key that the Board has performance measures that provide it with evaluative information to help make decisions about the program—information that tells them whether, and why, a program is working well or not. Performance measurement is the ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals. It is typically conducted by program or agency management and is critical for providing information concerning whether a program is working well or not. Performance measures may address the type or level of program activities conducted (processes), the direct products and services delivered by a program (outputs), or the results of those products and services (outcomes). The Army has a performance measure to support its first key objective. Each fiscal year, the nine Centers develop budget execution plans that outline how they will support the design standards for the specific facility types for which they have responsibility. In these plans, the Centers establish goals for updating specific existing standard designs and developing new standard designs (that is, the output from the Centers’ efforts). The Board’s primary oversight process consists of monitoring program execution of the nine Centers. According to Center officials, the Board reviews these execution plans at the semi-annual board meetings to determine whether the Centers are executing as planned, that is have the Centers met their goals for updating and developing standard designs. We found that this oversight process enables the Board to assess the progress each of the Centers has made toward achieving its goals for updating existing standard designs and developing new ones. For example, in fiscal year 2017 the Fort Worth Center completed all four of its planned standard design updates, and the Honolulu Center completed three of its four planned updates. We also found that the Board does not evaluate progress toward ensuring that the Centers consistently apply standard designs across the Centers of Standardization program (second objective of the Centers). Specifically, as shown in table 1 above, the Centers engage in a number of activities that support the consistent application of Centers standards on a project-by-project basis. However, the Board does not maintain, consolidate, or analyze information about how frequently the Centers engage in such activities, or how the Centers’ activities affect the program. That is because, according to Army and Centers officials, neither the Board nor the Centers have developed and implemented performance measures to assess the progress the Centers are making in ensuring that standard designs are consistently used. Absent such measures, the Army lacks assurance that standard designs are being applied, when appropriate, and that standard designs are being applied consistently across the service. In fact, to provide the project-specific documentation that we reviewed, the Centers needed to request documents from the USACE district office responsible for the projects. According to Centers officials, this was necessary because the Centers currently do not have a document management system in which project documentation is stored. Instead, as the USACE organization responsible for specific projects, each district maintains its own project records. The officials stated that USACE recently moved to a cloud-based system for storing project documents and is exploring whether this system could provide a more central document storage system. We note that having access to such information, along with creating appropriate performance measures, could enable the Board to measure whether progress has been made in ensuring that standard designs are applied consistently. In addition, we found that the Board does not evaluate whether the Centers are making progress in supporting the objectives and priorities of the AFSP (third objective of the Centers). One of the objectives of the AFSP is to reduce design costs and time, construction costs and time, and the number of change orders issued during construction. Although Army and Centers officials told us that the use of standard designs reduces project costs, time, and change orders, they could not provide supporting data. That is because, according to Army and Centers officials, neither the Board nor the Centers have developed and implemented performance measures to assess the effects of the use of standard designs. Creating such measures could enable the Army to assess the extent to which the Centers are reducing design costs and time, construction costs and time, and the number of change orders issued. DOD’s Fiscal Year 2020 Annual Performance Plan and Fiscal Year 2018 Annual Performance Report established a goal of simplifying, delivering faster, and reducing costs of product and service procurement. One of the performance measures associated with this goal was to reduce cost overruns and schedule delays by up to 50 percent for military construction projects. Developing and implementing performance measures related to reducing design costs and time, construction costs and time, and the number of change orders issued would enable the Centers to demonstrate the extent to which they are supporting DOD’s annual performance goals. We found that the use of the standard design does not introduce increased liability for the Centers if issues arise during a construction project. Centers officials stated that a contractor could file a claim against the government if the contractor felt there was a flaw in the Army’s standard design or that using the standard design resulted in unanticipated costs during the design or construction phase. However, Centers officials stated that there have been no instances in which any of the Centers was a party to legal action related to the use of a standard design. According to Centers officials, the design for a facility project is typically developed by one of the USACE district offices or an architect-engineer contractor. Further, these officials stated that while the pertinent Army standard design guides the development of Army project designs, the final project design, certified by the USACE district office or an architecture/engineering contractor, represents the plan for a specific project. In addition, according to the Federal Acquisition Regulation (FAR), the architect-engineer contractor is responsible for the professional quality, technical accuracy, and coordination of all designs, drawings, specifications, and other services furnished by the contractor under its contract. Furthermore, the FAR states that the contractor shall, without additional compensation, correct or revise any errors or deficiencies in its designs, drawings, specifications, and other services. The FAR also stipulates that the contractor may be liable for government costs resulting from errors or deficiencies in designs furnished under the contract. Consequently, according to USACE officials, because the Centers are not responsible for the design of a specific project, they would not have increased liability in the event that changes were required during construction. The Centers of Standardization develop and update Army standards and Army standard designs within the Army Facilities Standardization Program. In addition, the Centers are responsible for ensuring that the design and construction of Army military construction projects comply with approved Army standards and Unified Facilities Criteria. While the Army tracks the Centers’ program execution related to the Centers’ efforts to develop new and update existing standard designs (first objective of the Centers), it does not have performance measures for assessing progress toward the Centers’ other two objectives. Specifically, the Army does not have performance measures in place to assess the progress the Centers have made toward assuring consistent application of standards from the Centers’ program (second objective of the Centers) or monitoring the Centers’ execution to meet the overarching objectives and priorities of the AFSP and standardization process (third objective of the Centers) including, among other things, reducing design costs and time, construction costs and time, and change orders during construction. This hinders the Centers’ ability to determine how well they are supporting the objectives of both the Army Facility Standardization Program and DOD’s annual performance plans, as well as the Centers’ ability to demonstrate the extent to which they are achieving their objectives. We are making two recommendations to the Secretary of the Army. The Secretary of the Army should ensure that the Assistant Chief of Staff for Installation Management, in conjunction with the Centers of Standardization and the U.S. Army Corps of Engineers, establish and implement performance measures to assess the progress the Centers are making in ensuring that standard designs are used consistently. (Recommendation 1) The Secretary of the Army should ensure that the Assistant Chief of Staff for Installation Management, in conjunction with the Centers of Standardization and the U.S. Army Corps of Engineers, establish and implement performance measures to assess the effects of the use of standard designs, specifically the progress the Centers are making in reducing design costs and time, construction costs and time, and change orders. (Recommendation 2) We provided a draft of this report to the Department of the Army for review and comment. In its written comments, the Army concurred with both of our recommendations, and stated it would take actions to implement them. The Army’s comments are printed in their entirety in appendix V. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense and the Secretaries of the Army, Navy, and Air Force. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact Diana Maurer at (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI. According to Centers officials, a total of 12 full-time and 21 part-time staff are assigned to the Centers of Standardization. Each Center specializes in and is responsible for specific facility types and their designs. Table 3 below lists the current staffing levels and the facility types supported by each of the Centers. The Department of Defense’s (DOD) department-wide standardization program has the goals of improving military operational readiness, reducing total ownership costs, and reducing cycle time. Overseen by the Office of the Under Secretary of Defense for Research and Engineering (OUSD(R&E)), the Department of Defense Standardization Program is described in DOD Manual 4120.24, which outlines its governing council, definitions, and procedures that apply to all components within the department. Under the Defense Standardization Program, DOD component heads ensure that materiel standardization, including information technology and facilities, is addressed throughout the acquisition process. The three overarching goals of the Defense Standardization Program are to (1) improve military operational readiness, (2) reduce total ownership costs of the department, and (3) reduce cycle times. The manual also defines the following terms: Standard. A document that establishes uniform engineering or technical criteria, methods, processes, and practices. Standardization. The process of developing and agreeing on (by consensus or decision) uniform engineering criteria for products, processes, practices, and methods for achieving compatibility, interoperability, interchangeability, or commonality of materiel. Defense standard. A document that establishes uniform engineering and technical requirements for military-unique or substantially modified commercial processes, procedures, practices, and methods. There are five types of defense standards: interface standards, design criteria standards, manufacturing process standards, standard practices, and test method standards. DOD’s Unified Facilities Criteria (Facilities Criteria) and Unified Facilities Guide Specifications (UFGS) provide facility planning, design, construction, operation and maintenance, sustainment, restoration, and modernization criteria for facility owned by the DOD. The Facilities Criteria contain technical guidance; introduce new and innovative technology; or provide mandatory requirements to implement laws, regulations, executive orders, and policies prescribed by higher authority documents. The Facilities Criteria also define performance and quality requirements for facilities to support their mission throughout their life cycle. According to DOD guidance, the Facilities Criteria provide the most current operationally effective, cost-efficient, and safe criteria at the time of publication. Both the Facilities Criteria and UFGS are developed through the joint efforts of the U.S. Army Corps of Engineers, the Naval Facilities Engineering Command, and the Air Force Civil Engineer Center, and are approved by the Engineer Senior Executive Panel of the Unified Facilities Criteria Program. The Facilities Criteria and UFGS systems were designed not only to establish uniformity among defense facilities, but to standardize and streamline the process for developing, maintaining, and disseminating construction criteria. The procedures for the development and maintenance of the Unified Criteria and Unified Specifications are outlined in Military Standard 3007G, which is updated by the Engineering Senior Executive Panel. Each military department (Army, Navy, and Air Force) has its own facilities standardization program that implements the Unified Criteria and Unified Specifications as well as service-specific facilities criteria, standards, and guides. The Army’s program, known as the Army Facilities Standardization Program (AFSP), is the oldest among the three departments, having been initiated in 1993. Due largely to the unique construction needs of the Army, the AFSP is the most complex and comprehensive of the facility standardization programs. It utilizes two levels of guidance for standardized facility types: a broad standard, called “Army Standards,” and a specific standard, called “Standard Design.” The Department of the Navy program began in 2014 and provides policy and standards for the design development, and revision of Navy project documents in Navy and Marine Corps Design and Facilities Criteria, while the Air Force program was started in 2016 and provides criteria in an Air Force Instruction for design and construction of Air Force facilities. The Centers of Standardization (Centers) undertake a number of activities designed to support the key objectives found in their charter, which includes supporting the objectives of the Army Facilities Standardization Program (AFSP). Table 4 identifies each of these activities along with the specific objectives that we determined the activities support. developing and refining Centers of Standardization policies and processes, assuring consistent application of standards of the Centers’ program, and monitoring the Centers’ execution to meet the overarching objectives of the AFSP and standardization process. increased credibility with the Congress through more consistent construction program development, increased consistency in facility types with equal treatment among Army Commands, installations, and users, improved master planning and site development activities, improved design quality, and the promotion of design excellence, simplified design and construction project management, reduced design costs and times, reduced construction costs and time, and reduced change orders during construction, and increased customer satisfaction through improved responsiveness to users’ functional and operational requirements.", "summary": "In 2006, the U.S. Army Corps of Engineers began its Centers of Standardization program to develop design standards for facility types that the Army constructs on a regular basis. The Centers support broader Army efforts under the AFSP to standardize facility types with objectives such as improving design quality, reducing design and construction costs and time, and reducing change orders. Senate Report 115-262 accompanying the John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to evaluate the Centers' effectiveness. This report assesses, among other things, the extent to which (1) the Centers have identified activities that support their objectives, and (2) the Army tracks the Centers' progress toward their objectives. GAO reviewed and analyzed applicable regulations and program and project documentation; compared Center activities to program objectives; and interviewed cognizant agency officials to gain an understanding of the Centers' operations and potential financial liabilities. The nine Centers of Standardization (Centers) within the U.S. Army Corps of Engineers undertake a number of activities designed to support each of their program objectives. Their charter includes three objectives: (1) developing and refining Centers' policies and processes; (2) assuring consistent application of the Centers' standards; and (3) monitoring execution to meet the overarching objectives and priorities of the Army Facilities Standardization Program (AFSP) and standardization process. We found that the Centers' various activitiessuch as conducting value engineering and life-cycle cost studies to identify possible cost savings and analyze long-term costs of new facilitiesare consistent with key principles and concepts in Office of Management and Budget guidance for a disciplined capital planning process. Additionally, the post-occupancy evaluations led by the Centers are designed to evaluate whether the Army functional requirements have been met, Army standard design has been implemented, and there are any areas where the design could be improved. These evaluations support all three of the Centers' objectives by evaluating whether a design needs improvement, a facility was constructed in accordance with the approved project design, and customer needs were met. The Army has limited performance measures to track the Centers' progress in achieving program objectives. Semi-annual meetings of the Army's Centers of Standardization Management Board (Board) enable the Army to track the Centers' progress toward their goal of developing and updating Center policies and processes (first objective of the Centers). However, GAO found that the Army lacks performance measures to assess the Centers' progress in ensuring the consistent application of Army standard designs (second objective of the Centers) and in monitoring how well the Centers meet the objectives and priorities of the AFSP and standardization process (third objective of the Centers). Specifically, the Board does not maintain, consolidate, or analyze information about how frequently the Centers participate in construction projects, or how this activity affects the program and supports AFSP objectives, such as reducing project costs, times, and change orders. Taking steps to develop and implement appropriate performance measures would enhance the Army's efforts to ensure that the Centers are meeting their program objectives. GAO is recommending that the Army establish performance measures to assess the Centers' progress to (1) ensure the consistent use of standard designs and (2) reduce construction costs and time and reduce the occurrence of change orders. The Army concurred with our recommendations.", "document_type": "gao"}
{"report": "With top leadership support and commitment, DHS has made important progress in strengthening its management functions, but considerable work remains. As shown in figure 1, as of March 2019, DHS had met three out of five criteria for removal from our High Risk List—leadership commitment, action planning, and monitoring progress. DHS has partially met the remaining two criteria: capacity (i.e., people and other resources) and demonstrated, sustained progress. To address the criteria for capacity, DHS needs to make additional progress in identifying and allocating resources in certain areas—namely, acquisition, information technology, and financial management—to fully demonstrate its capacity. For the criteria for demonstrated, sustained progress, we reported in March 2019 that DHS had fully addressed 17 out of the 30 outcomes that are the basis for gauging DHS’s progress across management areas, as shown in table 1. To fully meet the criteria for demonstrated, sustained progress, DHS needs to continue implementing its Integrated Strategy for High-Risk Management and maintain engagement with us to show measurable, sustainable progress in implementing corrective actions and achieving outcomes. DHS can accomplish this by, among other things, maintaining a high level of top leadership support and sustained commitment to ensure continued progress in executing its corrective actions through completion, and increasing employee engagement and morale. Examples of important programs and remaining work in the key management functions include: In the key management function of human capital management, DHS leadership is needed to address skills gaps that have had a significant role in the DHS management high-risk area. For example, we have found that DHS lacks guidance on how to identify critical cybersecurity and acquisition skills needed to support its new information technology delivery model. We have also found that DHS has insufficient technical skills to support its biometric identification services program. Addressing these skill gaps could help DHS fully demonstrate its capacity to strengthen and integrate its management functions. Additionally, within human capital management, DHS has struggled with low employee morale scores since it began operations in 2003. DHS’s 2018 score ranked 20th among 20 large and very large federal agencies. Increasing employee engagement and morale is critical to strengthening DHS’s mission and management functions. DHS has continued to strengthen its employee engagement efforts by implementing our 2012 recommendation to establish metrics of success within components’ action plans for addressing its employee satisfaction problems. Further, DHS has conducted audits to better ensure components are basing hiring decisions and promotions on human capital competencies. In addition, OPM’s 2018 Federal Employee Viewpoint Survey data showed that in the past 2 years, DHS’s score on the Employee Engagement Index (EEI) increased by 4 points—from 56 in 2016 to 60 in 2018—which was 1 point more than the government-wide increase over the same period. While this improvement is notable, DHS’s current EEI score is 1 point below its EEI baseline score in 2010, suggesting that DHS is still working to regain lost ground after an 8 point drop between 2010 and 2015. In the key management function of financial management, DHS officials have faced challenges modernizing DHS components’ financial management systems and business processes that affect the department’s ability to have ready access to timely and reliable information for informed decision-making. Effectively modernizing financial management systems for the Coast Guard, FEMA, and ICE would help improve the reliability of their financial reporting. As we have reported, perhaps the single most important element of successful management improvement and transformation initiatives is the demonstrated commitment of top leaders, as shown by their personal involvement in reform efforts. With regard to leadership commitment, DHS’s top leadership, including leaders at the Secretary and Deputy Secretary level, has demonstrated exemplary commitment and support for addressing the department’s management challenges. They have also taken actions to institutionalize this commitment to help ensure the long- term success of the department’s efforts. One such effort is the Under Secretary for Management’s Integrated Priorities initiative to strengthen the integration of DHS’s business operations across the department. During monthly leadership meetings with the Under Secretary for Management, the department’s Chief Executive Officers have been providing status updates on their respective actions to address this high- risk designation. Furthermore, top DHS leaders, such as the Under Secretary for Management and the department’s Chief Executive Officers, routinely meet with our management to discuss progress on high-risk areas. In April 2019, we sent a letter to the Acting Secretary of Homeland Security detailing 26 open recommendations that we deem highest priority for implementation. Priority recommendations are those that we believe warrant priority personal attention from heads of key departments or agencies. These 26 recommendations fall into six major areas— emergency preparedness and response, border security, transportation security, infrastructure and management, cybersecurity, and chemical and nuclear security. Many of these recommendations cut across DHS’s mission areas that are critical for national security. Given that these recommendations are often the most complex and difficult to implement, top DHS leadership will play a critical role in addressing them. Fourteen of the 26 priority open recommendations we identified in the April 2019 letter are directed to acting officials serving in vacant positions. We have issued 12 recommendations to the Secretary of Homeland Security who is currently an acting official. We have also issued two recommendations to FEMA which is currently operating under acting leadership. Committed and consistent leadership at the department and component levels will be critical for addressing our priority recommendations. For example: In September 2014, we recommended that the Secretary of Homeland Security work jointly with the Administrator of the General Services Administration to strengthen management of the ongoing acquisition project to develop the multi-billion dollar headquarters facilities at the St. Elizabeth’s campus in Washington, D.C. Leadership is critical in this effort, given the magnitude of the project and the impact of headquarters consolidation on DHS operations. In October 2008, we recommended actions that FEMA should take to improve its administration of the National Flood Insurance Program high-risk area. We also recommended in September 2012 that FEMA develop a methodology to better assess a jurisdiction's capability to respond to and recover from a disaster without federal assistance. In July 2015, we further recommended that the Mitigation Framework Leadership Group establish an investment strategy to identify, prioritize, and guide federal investments in disaster resilience. Implementing these actions could limit the federal government’s fiscal exposure and increase the nation’s resilience to extreme weather events as the costs and impacts of weather disasters resulting from floods, drought, and other events are expected to increase in significance as previously “rare” events become more common and intense. In July 2018, we recommended that U.S. Customs and Border Protection (CBP) analyze the costs associated with future barrier segments along the southwest border and include cost as a factor in the Impedance and Denial Prioritization Strategy. Obtaining this key information could help CBP evaluate designs and prioritize locations for future border barrier segments to deter cross-border illegal activity. In February 2017, we recommended that DHS establish metrics and methods by which to evaluate the performance of DHS’s National Cybersecurity and Communications Integration Center in relation to its statutorily-required cybersecurity functions. Until it develops metrics and methods to evaluate its performance, the center cannot ensure that it is effectively meeting its statutory requirements, while cyber- based intrusions and attacks on federal systems and systems supporting our nation’s critical infrastructure are becoming more numerous, damaging, and disruptive. We also recommended in February 2018 that DHS take steps to better manage and assess its cybersecurity workforce gaps and areas of critical need. Given its important role in the nation’s cybersecurity, taking steps to address these issues will be critical. We will continue to monitor DHS’s progress in strengthening management functions and addressing priority recommendations. We also plan to continue to meet quarterly with DHS management to gauge leadership commitment, discuss progress, and review DHS’s goals and corrective action plans in its Integrated Strategy for High-Risk Management, which DHS issues twice per year. Thank you, Chairman Thompson, Ranking Member Rogers, and Members of the Committee. This concludes my testimony. I would be pleased to answer any questions. For further information on this testimony, please contact Christopher P. Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals making key contributions to this work include Alana Finley, Assistant Director; Luis E. Rodriguez, Analyst-in-Charge; Karin Fangman; Andrew Howard; and Thomas Lombardi. Key contributors for the previous work that this is based on are listed in each product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In 2003, GAO designated Implementing and Transforming DHS as a high-risk area to the federal government. DHS has made considerable progress in transforming its original component agencies into a single cabinet-level department, and as a result, in 2013, GAO narrowed the scope of the high-risk area to focus on Strengthening DHS Management Functions . In addition, DHS leadership is responsible for implementing numerous recommendations that GAO has made to the department and its component agencies. Current vacancies in top leadership positions could pose a challenge to addressing high-risk areas and priority recommendations that span DHS's diverse missions, which include preventing terrorism and enhancing security, managing our borders, administering immigration laws, securing cyberspace, and responding to disasters. This testimony discusses the need for DHS leadership commitment to strengthen its management functions and address GAO's priority recommendations. This testimony is based on GAO's 2019 high-risk update and other reports issued from March 2006 through April 2019. With the support and commitment of top leadership, the Department of Homeland Security (DHS) has made important progress in strengthening its management functions; however, considerable work remains. As of March 2019, DHS had fully addressed 17 of the 30 outcomes related to its management functions (see table). DHS needs to continue to show sustained leadership commitment in implementing its Integrated Strategy for High-Risk Management to achieve the remaining outcomes. Leadership commitment is also pivotal in addressing other GAO high-risk areas where DHS has a role, such as ensuring the cybersecurity of the nation, the National Flood Insurance Program, and limiting the federal government's fiscal exposure by better managing climate change risks. Currently, DHS has acting officials serving in eight positions requiring Senate confirmation, including positions with responsibilities for implementing high-risk outcomes, such as the Secretary, Deputy Secretary, and Under Secretary for Management. a “Mostly addressed”: Progress is significant and a small amount of work remains. b “Partially addressed”: Progress is measurable, but significant work remains. c “Initiated”: Activities have been initiated to address the outcome, but it is too early to report progress. In April 2019, GAO sent a letter to the Acting Secretary of Homeland Security detailing 26 open recommendations that GAO believes warrant the highest priority personal attention from the department and its components. These 26 recommendations fall into six major areas—emergency preparedness and response, border security, transportation security, infrastructure and management, cybersecurity, and chemical and nuclear security. For example, GAO has recommended that DHS take steps to strengthen human capital management, such as better managing and assessing its cybersecurity workforce gaps and areas of critical need. Fourteen of the 26 recommendations have been issued to acting officials serving in vacant positions, including 12 to the Secretary of Homeland Security, and two to the Federal Emergency Management Agency which is currently operating under acting leadership. Since the creation of DHS, GAO has made approximately 2,800 recommendations to the department, and DHS has implemented more than 75 percent of them, strengthening program management and performance measurement, among other things. GAO will continue to monitor DHS's progress in strengthening management functions and addressing priority recommendations.", "document_type": "gao"}
{"report": "GSA serves as the federal government’s primary civilian real property agent. When GSA does not have available federally owned space that can meet the needs of federal agency tenants, it leases space for these agencies in privately owned buildings. The Administrator of GSA delegates leasing authority to GSA regional commissioners, who further delegate authority to lease contracting officers. For leases that GSA procures for tenant agencies, GSA serves as the lessee and pays rent to the building owner, who serves as the lessor. The tenant agency pays monthly rent to GSA, which includes a fee for GSA’s services, and uses the leased space subject to the terms of an occupancy agreement with GSA. This agreement typically specifies not only the rent fee but also the amount the tenant agency must reimburse the lessor for improvements to the leased space—such as changes to walls, electrical outlets, telephone lines, and secure rooms—these are known as “tenant Improvements.” GSA leasing process. GSA uses different processes to carry out the leasing process depending on the size, cost, and type of the lease. For all of these processes, the leasing-planning process begins when GSA receives a request for space from a tenant agency and determines that fulfilling the request will require leasing space. According to the typical process outlined in the GSA Public Buildings Service (PBS) PBS Desk Leasing Guide, officials work with the tenant agency to define the requirements for the leased space, including the geographic area in which GSA will search for available properties. After this initial stage, GSA takes additional steps to acquire a new lease, see figure 1. For certain office space leases larger than 500 square feet, which represent more than 90 percent of GSA’s leases as of the end of fiscal year 2019, GSA can deviate from its typical leasing process and instead use what it calls the Automated Advanced Acquisition Program (AAAP). GSA began using a predecessor to this program in 1991 in the National Capital Region only and rolled out the current version to all national markets in 2015. In this program, instead of GSA’s first proposing requirements to potential lessors, the lessors first submit offers to GSA for pre-existing available space, including the space’s size, location, and features, and the rent amounts the lessor is offering for different lease durations. Once GSA develops a set of requirements with a tenant agency, it evaluates these standing offers to eliminate those that would not meet the space requirements, ranks the bids by present value, and selects the lowest cost option, see figure 2. GSA is required to take further steps for high value leases with a net annual rent above the statutory “prospectus” threshold—adjusted by GSA to $3.1 million in fiscal year 2019. For these leases, GSA must submit a prospectus, or proposal, to the House and Senate authorizing committees for their review and approval. As of the end of fiscal year 2019, GSA managed 8,045 leases, of which 291, or about 4 percent, had current annual rents above the 2019 prospectus level. These leases, however, accounted for about 41 percent of GSA’s total annual rent obligations. GSA leases. GSA leases differ substantially from typical commercial leases. For example, in a GSA lease, GSA—as the lessee—proposes the lease requirements. In a typical commercial office space lease, however, the lessor drafts the lease requirements and proposes them to the prospective tenant. For additional examples of the differences between GSA and typical commercial leases, see table 1. GSA’s lease reform efforts. In 2011, GSA issued a lease-reform implementation plan in response to comments from lessors and tenant agencies. In this plan GSA recommended changes including developing new lease models to better tailor its lease requirements to specific circumstances, and improving elements of its leasing process. As part of this and other initiatives since then, GSA developed leasing products and tools that it can use in various situations. These include: Simplified lease model: GSA developed this lease model for lower value leases with a facility security level of I or II, and a net annual rent—total rent minus operating expenses—of up to $150,000. GSA designed this model as a faster and more efficient method of processing lower value leases. As compared to GSA’s standard and global lease models—which can be used on leases of any size—this model contains fewer requirements and may not have certain more complex elements such as annual operating-cost adjustments, real estate tax adjustments, or an allowance for tenant substitution. In addition, the model requires GSA and the tenant agency to finalize the complete set of space requirements prior to GSA’s advertising the lease, a requirement that eliminates negotiations on the tenant improvements after GSA awards the lease. Net-of-utilities leases: As discussed in table 1, in most GSA leases the lessor is responsible for paying the utilities, and must estimate future utility costs as part of its bid for the lease. In a net-of-utilities lease, the tenant pays the utility costs for tenant space directly. A 2016 GSA study indicated that GSA could achieve savings through net-of-utilities leases for a small number of leases with certain characteristics including: the lease being over 50,000 square feet, a single tenant agency occupying the entire space, the tenant agency consuming large amounts of energy, and several other factors. GSA estimates that around 360 of its more than 8,000 leases meet these criteria. Succeeding and superseding leases: In most cases, GSA is required to conduct a full and open competition for leases. However, in certain circumstances GSA instead pursues succeeding or superseding leases. In circumstances where relocating to a new leased property would result in substantial relocation or duplication costs that GSA could not reasonably expect to recover through competition, GSA is allowed to pursue a succeeding lease, and when market conditions warrant renegotiation of an existing lease or when the tenant agency needs to make substantial modifications to a space before the expiration of a lease, GSA is allowed to pursue superseding leases. The GSA leasing stakeholders we spoke with identified some aspects of GSA leasing that are attractive to potential lessors such as the government’s good credit and GSA’s long average occupancy. They also identified a number of aspects of these leases that they said can affect their costs and the number of lessors who are willing and able to bid on a GSA lease. These areas were: Structure: overall characteristics of a lease including the volume and complexity of requirements, and how GSA structures rent payments, reimbursements for tenant improvements, and provision of services; Requirements: specific provisions in the lease such as early termination, janitorial and maintenance, tenant substitution, and real estate taxes; and Process: the steps lessors must follow to complete a GSA lease, such as the length of time and GSA’s ability to remain in a space after the end of the lease. The stakeholders we spoke with identified a number of benefits of GSA leasing that are attractive to potential lessors, including the government’s credit worthiness, long average tenancy in a space, and positive relationships with GSA and tenant agencies. Eighteen of the 20 lessors we spoke with identified the government’s credit worthiness as a benefit. This credit, lessors said, is better than many private sector tenants and presents lower risks, and some of the more experienced lessors said that GSA leases are an important part of their overall lease portfolios. For example lessors said that GSA leases represent a reliable revenue stream and that they are confident they will be paid on time for the full term of the lease, while for commercial leases—even for large companies—there is an increased risk of a rent default. Eight of the 20 lessors said that GSA and tenant agencies are relatively easy tenants to work with once the lease is in place. For example, lessors said the tenant agencies are very professional, and some of them said that they generally do not receive many requests for service from the occupying staff. In addition, seven lessors mentioned GSA’s long average tenancy in a space, which they said helps with a lessor’s long-term financial stability. One lessor said that commercial tenants stay on average three to five years, while their GSA tenants have lease lengths of 10 or 15 years. According to GSA, agencies occupy spaces leased through GSA for an average of around 22 years. Lessor Perspective on GSA Leases “The government is a Grade A tenant.” The lessors and real estate brokers we spoke with told us that the way GSA structures aspects of its leases can affect cost and competition. These aspects include the volume and complexity of requirements in the leases, the way GSA structures rent payments, how GSA defines and reimburses costs for tenant improvements, and the full service nature of GSA leases. Many lessors told us that they increase their bid prices in response to these aspects of GSA leases. GSA officials said that each of these aspects reflects GSA’s contracting policy, and it is not required to structure its leases this way by law, regulation, or executive order; however, they use these requirements to provide additional flexibility in managing their lease portfolio and reduce risk to the government. About three-fourths of lessors we interviewed said the volume and complexity of GSA lease requirements make these leases less attractive to potential bidders and can result in fewer bidders competing for a lease. These lessors further stated that GSA’s leases, in contrast to many private sector leases, can be quite lengthy—up to 85 pages—and contain many references to other documents that are not included in the lease text such as a seismic certification, a small business subcontracting plan, a Department of Labor wage determination, and a foreign ownership and financing certification. Lessor Perspective on GSA Leases “GSA’s lease is three-fourths of an inch thick, has many cross- references, takes weeks to read, and requires an attorney to understand.” Lessors must look up these other documents to fully understand the lease requirements, and some of the lessors we spoke to said that it can be difficult for them to quickly find the most important information. Lessors also noted that—in response to the volume and complexity of requirements—they may increase their bid prices. To account for risks inherent in these complex contracts lessors may also use the services of legal counsel or other experts, which could also increase costs. GSA officials told us that in the past several years they have made efforts to streamline their leases, including by eliminating duplicative or unnecessary provisions. One lessor told us that GSA has improved its leases by making them more intuitive and easier to read, a development that could be helpful for new potential lessors. About half of the stakeholders we spoke with, including 10 of the 12 more experienced lessors, said the way GSA structures its rent payments makes it difficult for these lessors to predict what actual operating costs will be in the future. Lessors said that because the shell rent (i.e. the building structure and systems) portion is typically flat over the firm term of a lease, and the operating expenses only increase at the consumer price index’s rate, the rental payments they receive are generally not sufficient to cover their actual increases in expenses. In addition, these lessors said that in a GSA lease, the lessor is typically responsible for providing utility services and that lessors pass these costs through to GSA as part of the operating cost portion of the rent. In a private sector lease, these costs are typically the tenant’s responsibility. To account for these issues, 11 lessors told us that they increase their bid prices to ensure that they will cover their costs, and two lessors told us that they would not bid on another GSA lease unless there were additional cost increases built into the lease. GSA officials told us that structuring rent payments this way provides GSA with a standardized method for addressing inflation and budgeting for future rental costs. Lessor Perspective on GSA Leases “The way GSA accounts for base rent and operating expenses is different than in a private sector lease. In our leases, the base rent is frozen throughout the term of the lease and only the operating expenses are allowed to increase based on inflation. Because of this, when preparing a bid we have to estimate operating expenses years into the future, which can be difficult, and if we guess too low we can end up losing money on the lease.” About one-third of the stakeholders we spoke with said the way GSA structures reimbursement for tenant improvements is a challenge, and three lessors said GSA’s requirements for construction standards and space designs can be difficult to meet. Stakeholders said that GSA’s requirement that lessors fund construction costs for tenant improvements upfront can put financial stress on lessors. For example, stakeholders said that lessors often must take on substantial debt in order to finance the construction of the tenant improvements. GSA reimburses lessors for tenant improvement costs over the firm term of the lease, but lessors told us that these payments do not begin until after the space is occupied, which can be delayed by the tenant agency’s changing its requirements. In prior work we found that this process of paying tenant improvements over the firm term of a lease increases the overall cost to the federal government of leasing space, primarily due to interest costs passed through by the lessors. In addition, half of the lessors we spoke with identified challenges with the process of developing and finalizing agency requirements for leased space, including frequent changes to space requirements and the need to develop detailed construction information before bidding on a lease. Lessor Perspective on GSA Leases “At the beginning I had to agree to a certain dollar amount for the tenant improvements, even though I did not know when the construction would happen, or how I would get paid back. You can get paid back in a lump sum, or the tenant improvements can be amortized over the lease term, but you do not know which it will be at the start of the process. This makes financing difficult.” Six lessors told us that they increase the cost of their bids in part due to GSA often over-estimating the cost of tenant improvements. This situation can require a lessor to take out a larger loan than necessary, which adds financing costs to the project. Lessors said that this situation can also prevent some potential lessors from bidding if they cannot obtain the amount of financing GSA requires. Additionally, lessors cited some tenant agencies’ space requirements which can call for expensive materials or difficult to construct items. For example, they described leases where they had to construct multiple restrooms or heating and cooling systems for small offices that typically house fewer than five employees. GSA officials told us that they structure the tenant improvements requirements in this way in order to establish expectations for the lessor. They said that they rely on tenant agencies to develop initial requirements for leased spaces, and they work with those agencies on the final designs and construction standards. We examined space requirements of the five federal agencies we reviewed that lease large amounts of space through GSA, and each of these agencies uses standardized guidance such as a handbook or design guide. These documents included information on developing specific requirements for leased space such as identifying the size of space needed, the types of workspaces used, and sample layouts for different types of facilities. Officials from these agencies told us that they use these handbooks as their primary reference when setting requirements for leased spaces and approving the final designs, and to develop these handbooks they use agency mission needs, government- wide security standards, and requirements from laws, regulations, and executive orders. They said that they generally rely on GSA to provide them with local market information such as the availability of suitable existing buildings, market rents, and other factors. About one-third of stakeholders we spoke with identified the full service nature of GSA’s leases as difficult, time consuming, and expensive— requiring them to estimate highly variable costs far into the future. For example, one lessor spoke of being required to provide all services— janitorial, maintenance and utilities—which can include simple things like replacing light bulbs. Further, the lessor has to work around the tenant agency’s operating hours to provide these services. Five lessors told us that they raise their bid prices to cover the costs of full service leases because they are cost and labor intensive. One lessor said that lessors estimate on the high end of the range to make sure they make a profit. Lessor Perspective on GSA Leases “The biggest issue for a potential lessor to consider is how hands-on they want to be—GSA leases are full service leases requiring lots of attention.” GSA officials told us that they structure leases this way because full service leases are generally less expensive to the government—avoiding the administrative burden of having to establish and maintain a contract for each service and avoiding the risk of higher than expected utility costs. In 2017, GSA issued guidance to its lease contracting officers on using net-of-utilities leases—those structured so that the tenant agency pays the utilities. GSA officials and stakeholders we spoke with told us that having a tenant agency pay utilities directly gives agencies an incentive to cut down on energy use and could result in lower costs. According to GSA, structuring leases as net-of-utilities leases requires substantial resources to manage and monitor. Therefore, GSA’s current policy is to use this structure for only certain large, energy-intensive leases. GSA officials told us they plan to continue using net-of-utilities leases but do not have plans to expand the program further. Stakeholders identified a number of specific GSA lease requirements that they said can affect cost and competition. These requirements include early termination options, GSA’s unilateral ability to substitute the tenant agency, provisions for reimbursing real estate taxes, and ongoing janitorial and maintenance requirements. Most of these requirements are GSA contracting policy, but the janitorial and tenant substitution requirements have some elements that GSA says it uses in response to either a law, a regulation, an executive order, or a combination of these and other sources. About two-thirds of stakeholders, including all 12 more experienced lessors, identified GSA’s including early termination options in leases as affecting the cost of the leases. GSA leases typically have a date after which GSA can terminate the lease with as little as 90 days’ notice, and since many GSA leases require significant initial capital for construction of the tenant improvements, some lessors told us they need to take out a loan using GSA’s future rent payments as the source of repayment. However, stakeholders and other experts told us that many loan underwriters will not consider any payments after GSA’s termination right date due to the risk that the GSA will leave the space, a factor that they said makes the loans more expensive and difficult to obtain. Nine of the lessors and two of the other experts we spoke with also said that it was unlikely GSA would ever exercise its termination options. Four lessors told us that they increase their bid prices to reflect the increased risk and expense that the early termination clauses provide, and four lessors and one broker told us that lessors may not bid on a lease at all if GSA includes an early termination option. Lessor Perspective on GSA Leases “The market, and lenders, look at the firm term as the length of the lease, and don’t take the soft term into account as GSA does… soft terms are the biggest structural obstacle in GSA lease requirements. If GSA included soft terms in leases just for emergencies, rather than as a matter of practice, the soft terms would not be as much of a problem.” GSA officials told us that these options allow them to maintain flexibility and use space efficiently despite changing tenant agency missions and space needs. In response to data GSA has collected from AAAP bids showing that GSA receives lower bids for longer firm-term leases, GSA has begun lengthening the firm term of its new leases. Specifically, GSA’s analysis of AAAP bids data showed that for lease offers in fiscal years 2017 and 2018, lessors bid a lower rent amount for a 10-year firm term as opposed to a 5-year term 96 percent of the time with an average savings of around 10 percent. GSA officials told us that they have been using more 10- and 15-year firm terms as opposed to the previous standard practice of five years. For example, according to GSA, in fiscal year 2014, 19 percent of GSA’s leased inventory had a firm term of 10 years or more, and in fiscal year 2017, this figure had risen to 26 percent. In addition, GSA has implemented a lease-term-setting tool, which officials said will help them lengthen the firm terms of leases where appropriate. About one-third of the stakeholders we spoke with identified janitorial and maintenance services as a challenge, and two lessors said that costs for janitorial and maintenance services can be difficult to estimate. For example, one lessor told us that it is difficult to estimate these costs two years into the future, let alone for the 10 or more years of a GSA lease, because of changes to local job market conditions and labor laws. In addition, stakeholders said that GSA leases require more frequent or comprehensive janitorial and maintenance services than do private sector leases. For example, lessors said that some cleaning and paint and carpet replacement intervals were more frequent than the industry standard. Four lessors told us that they include the additional costs for these services into the cost of their bids, and some lessors told us that they include up to 125 percent of their estimated true costs in their bids. According to GSA, it developed some of these requirements, particularly those related to specific cleaning products that lessors must use, in response to a combination of several laws, executive orders, and agency initiatives or recommendations. Some of the other requirements, such as the intervals for carpet and paint replacement, are GSA’s contracting policy, and officials told us that they have remained relatively static since the 1990’s. Lessor Perspective on GSA Leases “In one lease, we found that janitorial services for GSA cost approximately twice as much as normal cost for a non-GSA lease.” About one-third of the stakeholders we spoke with said that lessors— particularly lessors with multi-tenant buildings—are concerned about GSA’s ability to substitute one tenant agency for another, a requirement that can affect competition for leases. One concern cited was the possibility of substituting a law enforcement agency (e.g., ICE or FBI) that may have armed officers into a building previously occupied by an administrative tenant agency. Another was that increased traffic may result from substituting a busy public-facing agency (e.g., SSA or IRS) into a formerly quiet building environment. Stakeholders and other experts we spoke with said that scenarios like these can affect other tenants’ willingness to renew leases in a building; however, as we found in 2016, they also told us that GSA rarely exercises this option. Two stakeholders and another expert told us that lessors take specific actions in response to this requirement, including negotiating with GSA over modifying this clause, which one said they have been successful in doing. Federal regulation requires GSA to include this clause in leases with annual rents above the simplified acquisition threshold unless the lease contracting officer determines that it would not be appropriate. This regulation, however, stems from a general GSA statutory authority regarding federal property. GSA’s leasing regulations do not require GSA to use this requirement in leases with net annual rents under the simplified lease acquisition threshold, but GSA officials told us that as a matter of practice they also include it in these smaller leases. GSA officials told us that GSA, as the lessee, is ultimately responsible for a lease’s financial obligation, and the ability to substitute tenant agencies helps GSA mitigate the costs of vacant leased space in the event a tenant agency chooses to leave a leased property. Lessor Perspective on GSA Leases “The substitution of tenant requirement is especially an issue in multi- tenant buildings, and lenders can have trouble with it as well, but GSA almost never uses it. Our organization tries to get GSA to modify these clauses, and we are successful about 50 percent of the time, but this varies by GSA region.” About one-third of the stakeholders we spoke with said GSA’s requirements for real estate tax reimbursement may lead lessors to increase their bid prices to account for real estate tax uncertainty. GSA reimburses lessors for increases in real estate taxes above a base year— the first full year after GSA certifies the leased space as fit for occupancy. Lessors told us that since the date of occupancy is dependent on the completion of the design and construction process, the duration of which is difficult to estimate, when setting bids they have to estimate taxes without knowing the base year. Two lessors told us that when bidding on a lease they estimate on the high side to make sure they cover their costs, and another other lessor said that their organization might not bid on a GSA lease because of issues with the real estate tax requirements. GSA officials told us that they use these requirements because they allow GSA to establish the real estate tax base and the portion that GSA will reimburse. Officials also told us that lessors have told them that their current approach to tax adjustment places a risk on lessors that may ultimately get passed on to GSA in the form of higher rent, and at a May 2018 GSA industry event, lessors discussed difficulties with setting the base year. GSA officials told us that they are developing new requirements for lease construction that would allow for real estate taxes to be directly passed through by the lessor to GSA. Lessor Perspective on GSA Leases “The base year is often not clearly stated in the lease and is sometimes mentioned informally (e.g., in emails)…the lessor has no recourse to negotiate over the tax base year with GSA. It poses one of the biggest risks and has caused us to walk away from some bids after not being able to get a clear lease amendment specifying the tax base year.” The lessors and real estate brokers we spoke with also identified a number of general areas of GSA’s leasing process that they said can increase costs or reduce the number of bidders. These areas included the length of time it can take to finalize a GSA lease, GSA’s ability to occupy a space after lease expiration generally without penalty or the payment of damages beyond continuing rent payments—referred to as a “holdover”— and GSA’s propensity for entering into short-term extensions for current leases while negotiating new leases. About two-thirds of the lessors we spoke with mentioned frustration with the length of time it takes to finalize a GSA lease. Lessors told us that after GSA awards a lease, it can take more than a year of additional negotiations with the lessor, GSA, and the federal tenant agency to finalize the design requirements and construct the space. In 2016 we reported that the total length of GSA’s leasing process could be up to six to eight years. Because GSA does not generally begin to pay rent until after the space is fit for occupancy, lessors said that the length of time it takes to complete the lease award, design and construction processes can create financial stress on a lessor. For example, one lessor said that GSA’s overall leasing process was challenging, and the largest issue, rather than any particular requirement, was agreeing on the design after lease award. This length of time was because the tenant agency was slow to make decisions about the space design, and while GSA tried to coordinate by setting up weekly meetings about this design among GSA, the tenant agency and the lessor, there were also several layers of time- consuming GSA review. About one-third of the lessors we spoke with also identified challenges communicating with GSA and the tenant agency during the lease negotiation process, including challenges identifying points of contact and resolving disputes. Three lessors said that they or other lessors might not bid on additional GSA leases specifically because of the lengthy and complex process to finalize a lease. GSA officials told us that they rely on space requirements from the tenant agency and that the faster they receive those requirements, the faster the bid award can be completed and design process finalized. Lessor Perspective on GSA Leases “If it were up to me, I wouldn’t bid on any more GSA leases because they are too time intensive not only for management at our organization, but also for our accounting, engineering, construction and property management teams.” GSA officials told us that they have been using a number of initiatives to speed up their leasing process. For example, they said that in response to these time pressures they have begun requesting requirements as much as 48 months in advance of when a new lease will be needed. Officials from three of the five tenant agencies we spoke with told us that it can be difficult to estimate their space needs so far in advance because their missions and space needs can change. In addition, since 2015 GSA has been using the AAAP—in which potential lessors submit standing bids for vacant space that GSA then matches to requirements for new and continuing leases—in all of its national real estate markets. Four of the more experienced lessors we spoke with told us that they had noticed positive changes as a result of the AAAP. These changes included faster lease processing times and an overall simpler leasing process with less negotiating. One lessor told us that they only bid on new GSA leases that are part of this program. One-quarter of the lessors we spoke with identified drawbacks associated with GSA lease holdovers and short-term extensions, and at least three of the lessors we spoke with had experienced a holdover for one of their leases. Lessors said that the possibility of GSA’s holding over in a space or signing a short-term extension can affect their ability to finance a building and their time frame for finding a new tenant if GSA exits a property. Lessors also noted communications difficulties with GSA, for example some said that they had reached out to GSA to negotiate a lease well in advance of an incumbent lease’s going into holdover, but this action did not help them get a new lease on time. Lessors told us that they bid much higher rates for short-term extensions than they do for leases awarded through the normal process. In 2015 we reported that a significant number of GSA leases experience a holdover or short-term extension and that these can cause uncertainty for tenant agencies and lessors, increase GSA’s workload, and delay the completion of building maintenance and other tenant improvements. Lessor Perspective on GSA Leases “Holdovers and short-term extensions diminish lessors’ opinions of GSA.” Reducing holdovers and short-term extensions is one of the key tenets of GSA’s 2018–2023 Lease Cost Avoidance Plan. One method GSA uses to more quickly process leases for tenant agencies remaining in their current space is the superseding and/or succeeding lease. In 2018 GSA developed a revised tool to help its officials more quickly estimate whether GSA would likely achieve lower costs using a succeeding lease as opposed to performing a full and open competition for a new lease. Lease contracting officers can use this tool to identify leases that would be likely candidates for a succeeding or superseding lease earlier in the process. We analyzed the leases GSA entered into during fiscal years 2016 through 2018 and found about 29 percent of them were succeeding or superseding leases. GSA officials told us that they have tried to increase awareness of the new tool and appropriate use of succeeding and superseding leases through training programs. GSA began reform efforts in 2011 by conducting outreach, introducing new lease models, and adjusting some leasing provisions in response to stakeholder concerns. While GSA has continued its industry outreach, its more recent outreach efforts have not gathered information from a representative group of lessors. Further, GSA has not analyzed the information it does collect and therefore does not know if its reform efforts are adequately addressing stakeholder concerns. Also, GSA has not assessed whether one of its reform efforts—the simplified lease model— is achieving its intended benefits or how it could affect risk. Since fiscal year 2018, GSA has conducted informal industry outreach to certain lessors and other stakeholders about the leasing process. These efforts have included attending and making presentations at industry conferences, facilitating industry meetings with regional commissioners, and hosting feedback sessions. For example, in May 2019 GSA gave a presentation to a large industry organization on the current status of its efforts to reduce lease costs, and in May 2018 staff participated in a training event organized by GSA’s Office of Government-wide Policy where officials from industry shared their experiences with the leasing process. GSA officials told us that they gather information primarily from two industry groups, both of which have reached out to GSA, have a large number of members that are GSA lessors, and have a significant amount of knowledge of the GSA leasing process. GSA officials told us that they have used information mainly from these two groups to inform reform efforts, including creating net-of-utilities leases and longer firm-term leases. However, these two groups are focused primarily on organizations such as real estate brokers and investment trusts that are experts in the GSA leasing process. These organizations are not representative of GSA’s total population of lessors, which also includes many smaller organizations that have less experience with the GSA leasing process. By focusing its efforts on these larger groups, GSA is missing the perspective of smaller lessors, whose representatives may not attend industry meetings. These smaller lessors may have different types of concerns that GSA is not capturing. For example, in our sample of 20 lessors we identified areas where the perspectives of organizations with varying levels of experience with GSA leases differed. More than half of the less experienced organizations identified experiencing communication challenges with GSA and the tenant agency, while only two of the more experienced organizations identified this concern. Concerns about early termination clauses in GSA leases were cited by less than half of the less experienced organizations, but all of the more experienced organizations mentioned this clause as affecting their willingness to do business with GSA. Also, one of the brokers we spoke with said that smaller lessors tend to have different concerns about leasing requirements than larger lessors, but also have less ability to react to those concerns by, for example, raising their bid prices. In addition to limiting outreach to two groups that do not represent all types of GSA lessors, GSA has not maintained official records of the information it receives from these efforts. Further, it has not analyzed the information that it collects from lessors and other stakeholders for use in revising the leasing process. These omissions hinder GSA’s ability to identify the full range of lessor concerns. GSA’s recent approach to outreach differs from earlier approaches where GSA conducted more formal outreach to lessors. For example, in 2011 GSA performed formal outreach in order to inform decisions about significant changes to its leasing process. Officials told us that they selected a wide variety of lessors and held formal outreach sessions where GSA took minutes and maintained a record of all of the comments. GSA then analyzed the comments and used the results of its analysis to inform the initiatives it was conducting at that time, including the development of the simplified lease model. In addition, in 2017 GSA established the Office of Leasing Industry Outreach Program, which was a formal program to allow industry representatives to discuss various leasing issues with GSA officials through conference calls, webinars, and in-person sessions. GSA conducted nine monthly sessions with this program in 2017 and kept a formal record of only the first four sessions. Officials told us that they have since shifted their approach to conduct outreach more like that conducted by the Office of Government-wide Policy discussed above. Federal internal control standards call for agencies to communicate with, and obtain quality information from, external parties such as stakeholders that can help the agency achieve its objectives. While GSA has in the past collected and analyzed information from a wide variety of stakeholders to the leasing process, the real estate market is constantly changing. By obtaining current information from a broad spectrum of stakeholders and documenting and analyzing the information collected, GSA would be better positioned to know whether its lease reforms are addressing stakeholder concerns and how its lease requirements affect cost and competition. As previously noted, GSA developed its simplified lease model in 2011 to simplify the acquisition of smaller value leases with the intent of making the leasing process more efficient and cost-effective. GSA officials told us that using this model is also intended to help them achieve other lease reform goals including reducing holdovers and short-term extensions by speeding up the leasing process and making GSA leases more attractive to a wider spectrum of potential lessors. In addition, officials said that they believe greater use of the simplified lease model would increase competition for leases, particularly in real estate markets with high demand for office space. Since initial implementation, GSA has undertaken initiatives to increase the use of this model, including by raising the eligibility threshold from $150,000 to $250,000, and GSA officials told us that they have proposed raising the threshold to $500,000, a move that would cover more than 70 percent of GSA’s leases. However, GSA has not performed any analysis on the number of leases that were eligible for, but did not use, this model. Using available data, we analyzed the leases GSA entered into during fiscal years 2016 through 2018 that were potentially eligible for the simplified lease model and compared those that used the model to those that used GSA’s global and standard lease models. We found that the group of leases where GSA had used the simplified lease model had achieved lower rents both overall and per square foot than the group of potentially eligible leases where GSA had used its standard or global models (see table 2). These leases had lower average costs even though they had shorter average total terms and firm terms. This finding is notable because, according to GSA, longer leases typically have lower costs than shorter ones. However, our analysis of available data also found that GSA only used the simplified lease model on 124 of the 406 leases that were potentially eligible, or about 31 percent (see table 2). GSA officials told us that they face two primary challenges in increasing adoption of the simplified lease model. First, lease contracting officers must choose to use the simplified model as opposed to GSA’s standard lease model. While GSA’s leasing policy states that lease contracting officers should use the simplified lease model to the maximum practical extent, the lease contracting officers generally have wide discretion in selecting the type of lease to use for a particular acquisition. GSA officials told us that they believe some lease contracting officers may be hesitant to use the model because it is less familiar to them. GSA officials also told us that they have provided training for lease contracting officers on the appropriate use of the simplified lease model and have encouraged them to use it. Second, in order for GSA to use the simplified lease model, tenant agencies must provide a complete set of space requirements that GSA can use in a lease solicitation—what GSA calls biddable requirements— prior to GSA’s advertising the lease. According to GSA officials, tenant agencies do not always provide these requirements on time. By having biddable requirements in place before receiving bids, GSA can avoid negotiating these requirements after the lease is awarded. GSA officials and lessors told us that not having these requirements in place is a major source of project delays. GSA tracks both when it receives initial requirements from the tenant agencies and when the more fully developed requirements that GSA uses in its standard lease model solicitations are in place. In order to use the simplified lease model, GSA and the tenant agency then develop biddable requirements that need additional detail. An Example of challenges agencies face in providing lease requirements to the General Services Administration (GSA): Officials from three of the five tenant agencies we spoke with told us that it can be difficult for them to provide GSA with requirements two or more years in advance because agency missions and space needs change. For example, Internal Revenue Service officials told us that providing requirements 36 months in advance of a lease’s expiring is difficult for them because they may not know what their agency budget and personnel will be that far in advance. Officials from the Federal Bureau of Investigation said that lead times greater than three years are challenging because their agency missions change frequently, which leads to changing space needs. GSA has taken some steps to increase use of the simplified lease model. For example, several GSA regions have begun to work with SSA on a pilot program to reduce the time it takes for GSA to complete leases with that agency, including by increasing the availability of the simplified lease model. This program is in the early stages and, according to the charter, developed in August 2019, its objectives are to reduce the total time it takes to complete leases, increase up-front knowledge of project costs, and minimize the number of changes needed to leases all while maintaining or reducing the average costs for these projects. GSA and SSA plan to accomplish these objectives by identifying the areas of the leasing process most prone to delays, developing strategies for more quickly finalizing the complete requirements needed to use the simplified lease model, and testing the improvements in both large and small real estate markets. GSA plans to begin testing the changes developed by this program during the first half of 2020. SSA officials told us that they typically begin planning approximately 42 months prior to lease expiration with the goal of providing initial requirements to GSA by 36 months prior. GSA lacks comprehensive information on the benefits and challenges of using the simplified lease model because it has not evaluated the results it has obtained from using it. For example, officials told us that they have not analyzed the lease processing times or rental rates they have achieved using the model. Officials also said that they already collect the data they would need to study the model and they have used this data to analyze related issues such as lease holdovers and short-term extensions. Officials also told us that they do not consider use of the simplified lease model to pose any financial risks provided that lease contracting officers follow GSA’s existing policies. However, they told us that GSA has not reviewed financial and other risks that may arise from using the model. These factors include risks due to the model’s not containing certain provisions that may protect GSA, such as tenant substitution. We have reported that agencies can use information about the performance of programs to identify problems or weaknesses, to try to identify factors causing the problems, and to modify programs to address them. Program assessment helps to establish a program’s effectiveness. Without conducting such an assessment, GSA does not have the information needed to determine whether the simplified lease model is achieving intended results, whether to make improvements, or how to mitigate any risks. The federal government spends nearly $6 billion annually on leasing space from private entities, and GSA has taken steps to encourage private sector competition for government leases. GSA’s efforts to address stakeholder concerns with lease requirements have had some success. Specifically, GSA’s 2011 formal stakeholder outreach and subsequent development of new lease models and other process changes have given GSA some options to reduce leases’ complexity and better tailor leases to the needs of individual projects. However, because GSA’s recent outreach has not included a representative group of its lessors, and it has not documented and analyzed the information collected from this outreach, GSA may not have the information it needs to fully address lessors’ concerns. Further, the simplified lease model—which GSA developed to address some of these stakeholder concerns and more effectively use its resources—has been in use for several years. Given that GSA has proposed further expanding the use of the model to higher value leases, it is important to know the results GSA has obtained from using the model, such as the characteristics of leases for which it achieves the greatest savings in costs and time, and the extent to which it bears financial or other risks from its use. Such information would help inform GSA’s future decision-making on the use of the simplified lease model. We are making the following three recommendations to GSA: The Administrator of the General Services Administration should expand its outreach as appropriate to obtain feedback from lessors that are representative of its entire lease portfolio. (Recommendation 1) The Administrator of the General Services Administration should, for future outreach efforts, document and assess lessors’ feedback about the leasing process. (Recommendation 2) The Administrator of the General Services Administration should evaluate whether the simplified lease model is achieving its intended results. (Recommendation 3) We provided a draft of this report for review to the General Services Administration, the Social Security Administration, and the Departments of Homeland Security, the Interior, Justice, and the Treasury. The General Services Administration concurred with our recommendations in its written comments, which are reproduced in appendix II. The General Services Administration and the Department of the Interior provided technical comments, which we incorporated as appropriate. The Departments of Homeland Security, Justice, and the Treasury, and the Social Security Administration had no comments on the draft report. As agreed with your offices, unless you publically announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees; the Administrator of the General Services Administration; the Secretaries of the Departments of Homeland Security, the Interior, and the Treasury; the Commissioner of the Social Security Administration; the Attorney General; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines (1) lease requirements selected stakeholders identified as affecting cost and competition and steps GSA has taken to address their concerns, and (2) how GSA has identified stakeholder concerns and evaluated its simplified lease model. To obtain information for both objectives, we reviewed laws, regulations, and executive orders covering GSA leases and GSA’s leasing process. We also obtained data from GSA on each of the 1,618 leases it entered into between the beginning of fiscal year 2016 and the end of fiscal year 2018, the most recent data available. This data included fields for the current annual rent, the size of the lease in rentable square feet, the lease model GSA used, the facility security level, the occupying agency, and the lease’s effective and expiration dates, among others. We assessed the reliability of this data by reviewing documentation; interviewing GSA officials; electronically testing the data by, for example, examining missing values and outliers; and verifying the accuracy of potentially erroneous data with GSA officials. We concluded that the data were reliable for the purposes of selecting a sample of GSA lessors and reporting on GSA’s portfolio of leases and the general characteristics of the groups of leases that used different lease models. In addition, to address both objectives, we collected information from and interviewed a non-generalizable sample of 20 GSA lessors to obtain their perspectives on GSA leases and GSA’s leasing process. To select these lessors, we used the fiscal year 2016–2018 lease data that GSA provided and selected leases using the annual rent amount as the primary selection criteria. We excluded leases that used models designed for specific lease products, such as leases for parking structures or leases on airport properties, and we also excluded leases that were successions or supersessions of leases that had already been established under different models. To make the selections, we first split the data into three groups based on annual rent, the first group of leases with annual rents under $150,000; the second group with annual rents between $150,000 and below $500,000; and the last group with annual rents above $500,000. We then randomly ordered the leases within each of the three groups, and selected 53 total leases in that order from the three groups. We checked this grouping to ensure that the selected leases had similar characteristics to GSA’s general population in other important lease characteristics such as lease model used and GSA region. We then randomly ordered the selected leases and contacted the lessors for those leases in that order. We interviewed the first 20 lessors from our selected leases who agreed to be interviewed. When contacting the lessors we found that in most cases the lessor named in GSA’s data was a subsidiary to another organization. In those cases, we interviewed the organization that self- identified as being responsible for the selected lease, or their representative. We conducted these interviews between March 2019 and June 2019 and used a semi-structured interview format with open-ended questions for those interviews. During these interviews, we asked for lessors views on the requirements in GSA’s leases that can affect their willingness to bid on GSA leases and the prices they can offer, actions they take in response to those requirements, other areas of GSA’s leasing process that can be difficult for them, the benefits to leasing to GSA, and their perspectives on GSA’s recent lease reform efforts. To obtain a broader perspective on GSA’s leasing process, we also conducted semi-structured interviews on the same topics with six real estate brokers who are participating in the GSA Leasing Support Services contract. We asked the brokers to provide their experiences on which areas of GSA leases result in the greatest number of cost and competition issues from lessors, and what the lessors do about those areas. We also interviewed four other experts on GSA leasing including professional organizations and attorneys who represent building owners, and former GSA officials. Although the results of these stakeholder interviews are not generalizable to the entire population of GSA lessors, they provide illustrative examples of lessors’ experiences with GSA leases and the leasing process. After conducting these semi-structured interviews with lessors and brokers, we conducted a content analysis of the interview data. To conduct this analysis, we organized the responses by topic area, and then one GAO analyst reviewed all of the interview responses and identified recurring themes. Using the identified themes, the analyst then developed categories for coding the interview responses and independently coded the responses for each question. To ensure accuracy, a second GAO analyst reviewed the first analyst’s coding of the interview responses, and then the two analysts reconciled any discrepancies. To identify the lease requirements that stakeholders we spoke with identified as affecting cost and competition, we synthesized information from our content analysis of interview responses to identify the most commonly mentioned requirements. We selected the eight most commonly mentioned requirements by summing the total number of responses from both the lessors and the brokers. As part of this analysis we also selected the four areas stakeholders most often mentioned as challenges that were related to GSA’s leasing process, as opposed to a specific requirement, but that stakeholders nonetheless identified as having effects on cost and competition. To assess how the responses from lessors may have differed based on how much experience a lessor has with GSA, we grouped the lessors we spoke with into two categories. The first category was those lessors who had told us that they had experience with three or more GSA leases, we referred to these lessors as “more experienced,” and the second category was those lessors who had experience with one or two GSA leases, we referred to those lessors as “less experienced.” To identify the source of the GSA requirements stakeholders identified, we reviewed GSA documents and interviewed officials to learn about each of the requirements. In addition, we reviewed laws, regulations and executive orders that governed GSA’s use of these requirements. To determine how GSA and tenant agencies develop requirements for leased space—one of the requirements stakeholders identified—we selected five bureau-level and independent agencies to review how they develop initial requirements for leased space and how they work with GSA and the lessor to finalize those requirements. We selected these agencies by the number of GSA leases they had entered into during fiscal years 2016-2018, using the lease data for that time period provided by GSA. We selected the agencies that had entered into the greatest number of leases, and in order to ensure that we had a diversity of experiences from across the federal government, and we limited our selection to executive branch independent agencies and one-bureau-level entity from each cabinet department. Based on these factors, we selected (1) Department of the Interior Fish and Wildlife Service (FWS); (2) Department of the Treasury Internal Revenue Service (IRS); (3) Department of Justice Federal Bureau of Investigation (FBI); (4) Social Security Administration (SSA); and (5) Department of Homeland Security Immigration and Customs Enforcement (ICE). While the views of these agencies are not representative of all executive branch agencies, they provide a range of examples and experiences with leasing space through GSA. We reviewed documents and interviewed officials from each of these five agencies to learn about how they develop requirements for leased space, how they work with GSA to identify feasible properties, how they participate in the development of the final space design and construction, and how they plan for their future leased space needs. To identify the steps GSA has taken to identify stakeholder concerns and evaluate its simplified lease model, we reviewed pertinent GSA documents and interviewed GSA officials on recent lease reform efforts, including how GSA has defined them, what information GSA used to develop them, how GSA has implemented them, and how GSA has assessed their performance. In addition, we obtained information from our interviews with lessors and real estate brokers about their impressions of GSA’s lease reform efforts, including whether they were aware of the efforts, and what effects they had observed. We compared GSA’s efforts to identify and address stakeholder concerns to Federal Standards for Internal Control related to external communication. To identify how often GSA has used its simplified lease model and the characteristics of the leases for which GSA used the model, we used the GSA fiscal year 2016–2018 lease data described previously. We analyzed the data to obtain information about the number of leases that had used each of GSA’s lease models, and the average rent amounts, size, and terms. Even though the facility security level is an additional eligibility requirement for the model, we could not include it in this analysis because GSA does not have security level information for many of the leases in this dataset. However, we determined that omitting this data field did not substantially change the results of this analysis because only a small number of leases with costs below $150,000 also had a facility security level of III or above. We were not able to assess the extent to which the lower rental costs might be attributable to the use of the simplified lease model because there are other factors that that contribute to its use that are not included in GSA’s data. For example, in order for GSA to use the simplified lease model, tenant agencies must provide fully developments prior to GSA advertising the lease. The data do not include the date GSA received these requirements. We compared GSA’s efforts to evaluate its simplified lease model to criteria from our prior work on the use of performance information for decision-making. We conducted this performance audit from October 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Amelia Bates Shachoy, Assistant Director; Alex Fedell, Analyst-in-Charge; James Duke; Cynthia Grant; Geoffrey Hamilton; Gina Hoover; Terence Lam; Malika Rice; Kelly Rubin; Jim Russell; Patrick Tierney; and Amelia Michelle Weathers made key contributions to this report.", "summary": "As the federal government's landlord, GSA works with lessors and real estate brokers to identify space for other federal agencies to use. As part of this process, GSA uses leases that include requirements not commonly used in the private sector. These requirements and GSA's lengthy and complex leasing process can affect federal leasing costs and competition for leases. GAO was asked to review issues related to cost and competition for GSA leases with private sector lessors. This report examines: (1) lease requirements selected stakeholders identified as affecting cost and competition and steps GSA has taken to address stakeholders' concerns, and (2) how GSA has identified stakeholders' concerns and evaluated its simplified lease model. GAO reviewed pertinent federal statutes and regulations and GSA's contracting policy and leasing data from fiscal years 2016–2018. GAO conducted interviews with 20 GSA lessors selected from GSA's data to represent a range of location, and cost of the leases and the six real estate brokers that work with GSA. Stakeholders, including 20 lessors (e.g., building owners) and the six real-estate brokers that negotiate federal government leases, identified several aspects of the General Services Administration's (GSA) leases that can affect cost and competition. For example, specific lease requirements such as early termination (see table) can lead lessors to increase their rent rates or decide not to bid on a lease—thereby increasing federal leasing costs or decreasing competition. According to GSA officials, many of these lease aspects reflect contracting policy rather than being required by law, regulation, or executive order. GSA has made some changes, such as lengthening the term of some leases, to address stakeholder concerns. Stakeholders also identified the time it takes to complete a lease and GSA's propensity for staying in a space beyond the term of a lease as increasing costs and making GSA leases less attractive to potential bidders. Source: GAO analysis of stakeholder information. | GAO-20-181 GSA has undertaken initiatives to identify stakeholders' concerns to inform its reform efforts, but it lacks complete information to address concerns or evaluate its efforts. Specifically, GSA has not gathered information from a representative group of lessors because its recent outreach has involved two industry groups that focus primarily on organizations such as real estate brokers and investment trusts that are experts in GSA leasing. These organizations may not have the same concerns as smaller, less experienced, organizations. By obtaining information from a broad spectrum of stakeholders, GSA would be better positioned to know whether its leasing reforms are addressing stakeholders' concerns. Additionally, to expedite processing of lower-value leases, GSA developed a simplified lease model that excludes some requirements that stakeholders identified as challenging but may protect GSA, such as tenant substitution. GAO found that for fiscal years 2016 to 2018, GSA used the model for only about one-third of potentially eligible leases. GSA has proposed increasing use of the model, but it does not know whether the model as currently used is achieving the anticipated benefits, including reduced lease processing times, or the impact of financial or other risks from this model because GSA has not evaluated its use. Without such an assessment, GSA does not have the information needed to determine whether the simplified lease model is achieving its intended results, whether to make improvements, or how to mitigate any risks. GAO is making three recommendations, including that GSA: (1) expand its outreach as appropriate to obtain feedback from lessors that are representative of its entire lease portfolio, and (2) evaluate whether the simplified lease model is achieving its intended results. GSA agreed with the recommendations and said it believes there are additional opportunities to expand its outreach efforts and evaluate the simplified lease model.", "document_type": "gao"}
{"report": "The NASA Authorization Act of 2010 directed NASA to develop a SLS, to continue development of a crew vehicle, and to prepare infrastructure at Kennedy Space Center to enable processing and launch of the launch system. To fulfill this direction, NASA formally established the SLS launch vehicle program in 2011. Then, in 2012, NASA aligned the requirements for the Orion program with those of the newly created SLS vehicle and the associated ground systems programs. The Exploration Systems Development (ESD) organization reports to NASA’s Associate Administrator for Human Exploration and Operations Mission Directorate and is responsible for managing and integrating the human space exploration programs. Figure 1 provides details about each SLS hardware element and its source as well as identifies the major portions of the Orion spacecraft. NASA established the EGS program to modernize the Kennedy Space Center to prepare for integrating hardware, as well as processing and launching SLS and Orion, and recovery of the Orion crew capsule. The EGS program consists of a number of components and processing centers including the Vehicle Assembly Building, Mobile Launcher, and Crawler-Transporter. The Mobile Launcher consists of (1) a two-story base that is the platform for the rocket and (2) a tower equipped with a number of connection lines, called umbilicals, and launch accessories that will provide SLS and Orion with power, communications, coolant, fuel, and stabilization prior to launch. During preparations for launch, the Crawler-Transporter will pick up and move the Mobile Launcher into the Vehicle Assembly Building. Inside the Vehicle Assembly Building, NASA will stack the SLS and Orion vehicle on the Mobile Launcher and complete integration for launch. Before launch, the Crawler-Transporter will carry the Mobile Launcher with SLS and Orion to the launch pad where engineers will lower the Mobile Launcher on to the pad and remove the Crawler-Transporter. During launch, each umbilical and launch accessory will release from its connection point, allowing the rocket and spacecraft to lift off from the launch pad. Figure 2 is a picture of the Mobile Launcher positioned on top of the Crawler-Transporter outside of the Vehicle Assembly Building. During Exploration Mission 1 (EM-1), the SLS vehicle is to launch an uncrewed Orion to a distant orbit some 70,000 kilometers beyond the Moon. All three programs—SLS, Orion, and EGS—must be ready on or before the EM-1 launch readiness date to support this integrated test flight. Exploration Mission 2 (EM-2) will be a 10- to 14-day crewed flight with up to four astronauts that will orbit the moon and return to Earth to demonstrate the baseline Orion vehicle capability. NASA establishes an agency baseline commitment—the cost and schedule baselines against which the program may be measured—for all projects that have a total life cycle cost of $250 million or more. A rebaseline is a process initiated if the NASA Administrator determines the development cost growth is more than 30 percent of the estimate provided in the baseline of the report, or if other events make a rebaseline appropriate. A replan is a process generally driven by changes in program or project cost parameters, such as if development cost growth is 15 percent or more of the estimate in the baseline report or a major milestone is delayed by 6 months or more from the baseline date. A replan does not require a new project baseline to be established. When the NASA Administrator determines that development cost growth is likely to exceed the development cost estimate by 15 percent or more, or a program milestone is likely to be delayed from the baseline’s date by 6 months or more, NASA must submit a report to the Committee on Science, Space, and Technology of the House of Representatives and the Committee on Commerce, Science, and Transportation of the Senate. Should a program exceed its development cost baseline by more than 30 percent, the program must be reauthorized by the Congress and rebaselined in order for the contractor to continue work beyond a specified time frame. NASA tied the SLS and EGS program cost and schedule baselines to the uncrewed EM-1 mission and the Orion program’s cost and schedule baselines to EM-2. Over the past 5 years, we have issued several reports assessing the progress of NASA’s human space exploration programs relative to their agency baseline commitments. In April 2017, we found that given the combined effects of ongoing technical challenges in conjunction with limited cost and schedule reserves, it was unlikely that these programs would achieve the committed November 2018 launch readiness date. We recommended that NASA confirm whether this launch readiness date was achievable and, if warranted, propose a new, more realistic EM-1 date and report to Congress on the results of its schedule analysis. NASA agreed with both recommendations and stated that it was no longer in its best interest to pursue the November 2018 launch readiness date. Subsequently, NASA approved a new EM-1 schedule of December 2019, with 6 months of schedule reserve available to extend the date to June 2020, and revised costs (see table 1). Because NASA delayed the EM-1 schedule by up to 19 months, the SLS and EGS programs—that are both baselined to EM-1—reported a replan to the Congress. The EGS program also reported its development costs increased by about 23 percent over the baseline. At the same time, NASA reported that the SLS program development costs would only increase by about 2 percent. Under the Federal Acquisition Regulation (FAR), a variety of contract types are available including those that incentivize a contractor in areas that may include performance, cost, or delivery. The type of contract used for any given acquisition inherently determines how risk is allocated between the government and the contractor. According to the FAR, since the contract type and the contract price are interrelated, the government must consider them together. The government can choose a contract type and negotiate price (or estimated cost and fee) that will result in reasonable contractor risk and provide the contractor with the greatest incentive for efficient and economical performance. For example, under firm-fixed-price contracts, the contractor assumes full responsibility for performance costs. Under cost-reimbursement contracts, the government provides for the payment of allowable incurred costs, to the extent prescribed in the contract. The government uses cost-reimbursement contracts when, for example, there are uncertainties involved in contract performance. Incentive contracts can be either fixed-price or cost-reimbursement type contracts. The contractor’s responsibility for the performance costs and the profit or fee incentives in incentive contracts are tailored to the uncertainties involved in contract performance. Incentive contracts— including award fee and predetermined, formula-type incentive fee contracts—are designed to attain specific acquisition objectives by, in part, including appropriate incentive arrangements that (1) motivate contractor efforts that might not otherwise be emphasized, and (2) discourage contractor inefficiency and waste. Award fees generally emphasize multiple aspects of contractor performance in areas that the government assesses more subjectively. In contrast, predetermined formula-type incentives are generally associated with a cost incentive, but can also emphasize performance in areas that the government assesses more objectively. The FAR indicates that award fee contracts are suitable when it is neither feasible nor effective to devise predetermined objective incentive targets, the likelihood of meeting acquisition objectives will be enhanced by using a contract that provides the government with the flexibility to evaluate both actual performance and the conditions under which it was achieved, and the administrative effort and cost are justified. Table 2 provides an overview of cost-plus-incentive-fee and cost-plus- award-fee contracts because these are the type used in the Orion and SLS programs. Multiple-incentive contracts contain more than one incentive. For example, these contracts may include both subjective award fee criteria and predetermined, formula-type incentives. Agencies can use incentive contracts to promote certain acquisition outcomes, such as keeping costs low, delivering a product on time, and achieving technical performance of the product. NASA awarded incentive contracts to both Boeing and Lockheed Martin—a cost-plus-incentive-fee/award-fee contract to Boeing for the SLS stages effort and a cost-plus-award-fee contract to Lockheed Martin for the Orion crew spacecraft effort. For the SLS stages incentive contract with Boeing, the contract includes both incentive and award fees, broken into these three components: Milestone-incentive fees. These fees are paid for successful completion of each program milestone event. Cost-incentive fees. These fees are initially negotiated and later adjusted by a formula and are paid based on the relationship of total allowable costs to total target costs. Award fees. These fees are determined through subjective evaluations relative to factors in the contract’s award fee plan. For the Orion crew spacecraft incentive contract with Lockheed Martin, the contract includes fee broken into three components. The government typically uses award fees when it is not feasible or effective to use predetermined objective criteria. Therefore, as noted above, award fees are typically determined against subjective criteria. However, this contract includes award fee with both subjective and objective criteria: Milestone award fees. These fees are paid for completing critical criteria and dates associated with each milestone. Performance incentive fee. These fees are paid for completing criteria and dates associated with each performance incentive. Period of performance award fee. These fees are determined through subjective evaluations relative to factors in the contract’s award fee plan. For purposes of discussion within this report, we group each of the fees for each contract into one of four categories—milestone fee, performance incentive fee, cost incentive fee, and award fee. When award fees are used that require a subjective assessment by the government, NASA generally defines award fee periods of at least 6 months for the duration of the contract and establishes performance evaluation boards to assess the contractor’s performance relative to the performance evaluation plan. For the contracts we reviewed, NASA evaluates contractor performance based on weighted evaluation factors to determine the award fee. Table 3 includes a description of the evaluation factors and the weighted percentages for each factor assigned to the SLS stages and Orion crew vehicle contracts. When developing a contractor’s evaluation for a period of performance, the members of the performance evaluation boards for each contract use descriptive ratings in their evaluations. Performance monitors for different areas within the programs compile a list of the contractor’s strengths and weaknesses relative to specific criteria and defined activities for each of the evaluation factors. The performance monitors then consider other factors, such as government-directed changes and obstacles that arose that may have affected the contractor’s performance, and prepare performance reports. Members of the performance evaluation boards consider the performance monitor’s reports and assign the scores and descriptive ratings for the specific evaluation period. Table 4 below outlines award fee adjectival ratings, award fee pool available to be earned, and descriptions of the award fee adjectival ratings from the Federal Acquisition Regulation. In November 2018—within 1 year of announcing a delay for the first mission—senior NASA officials acknowledged that the revised EM-1 launch date of December 2019 is unachievable and the June 2020 launch date (which takes into account schedule reserves) is unlikely. These officials estimate that there are 6 to 12 months of schedule risk associated with this later date, which means the first launch may occur as late as June 2021 if all risks are realized. This would be a 31-month delay from the schedule originally established in the programs’ baselines. Officials attribute the additional schedule delay to continued production challenges with the SLS core stage and the Orion crew and service modules. NASA officials also stated that the 6 to 12 months of risk to the launch date accounts for the possibilities that SLS and Orion testing and final cross-program integration and testing at Kennedy Space Center may result in further delays. These 6 to 12 months of schedule risk do not include the effects, if any, of the federal government shutdown that occurred in December 2018 and January 2019. In addition, NASA’s reporting of cost data for the SLS and Orion programs is not fully transparent. NASA’s estimates for the SLS program indicate 14.7 percent cost growth as of fourth quarter fiscal year 2018, but our analysis shows that number increases to 29.0 percent when accounting for costs that NASA shifted to future missions. Further, in summer 2018, NASA reported a 5.6 percent cost growth for the Orion program. However, this reported cost growth is associated with a program target launch date that is 7 months earlier than its agency baseline commitment launch date. If the Orion program executes to the launch date established in its agency baseline commitment, costs will increase further. The SLS program will not meet the June 2020 launch date for the first mission due, in part, to ongoing development issues with the core stage. For this mission, the SLS launch vehicle includes solid rocket boosters, an upper stage, and a core stage—which includes four main engines and the software necessary to command and control the vehicle. As of fall 2018, the program reported that the boosters, engines, and upper stage all had schedule reserves—time allocated to specific activities to address delays or unforeseen risks— to support a June 2020 launch. The core stage, however, did not have schedule reserves remaining as the program continues to work through development issues. According to the SLS program schedule, core stage development culminates with “green run” testing. For this test, NASA will fuel the completed core stage with liquid hydrogen and liquid oxygen and fire the integrated four main engines for about 500 seconds. The green run test carries risks because it is the first time that several things are being done beyond just this initial fueling. For example, it is also the first time NASA will fire the four main engines together, test the integrated engine and core stage auxiliary power units in flight-like conditions, and use the SLS software in an integrated flight vehicle. In addition, NASA will conduct the test on the EM-1 flight vehicle hardware, which means the program would have to repair any damage from the test before flight. The program has no schedule margin between the end of core stage production and the start of the green run test, and is tracking risks that may delay the test schedule. For example, as the NASA Office of Inspector General (OIG) found in its October 2018 report, the Stage Controller—the core stage’s command and control hardware and software needed to conduct the green run test—is 18 months behind schedule and may slip further. Any additional delays with the development of the core stage and stage controller will further delay the start of the green run test. In addition, the SLS program has no schedule margin between the green run test and delivery of the core stage to Kennedy Space Center for integration to address any issues that may arise during testing. In November 2018, senior NASA officials stated that they have accounted for the potential of continued core stage development delays—along with risks to the Orion and EGS programs—and stated that there are an additional 6 to 12 months of risk to the EM-1 launch date. We found that a delay of this length would push the launch date for EM-1 out as far as June 2021 should all of the risks be realized. This would represent a 31- month delay from the original schedule baseline. Further, these 6 to 12 months of schedule risk do not include the effects, if any, of the federal government shutdown that occurred in December 2018 and January 2019. Figure 3 below compares schedules of key events for the core stage shortly after NASA established the program baseline in August 2014, the December 2017 replan, and the program’s schedule as of November 2018. Officials from the SLS program and Boeing, the contractor responsible for building the core stage, indicated that an issue driving core stage delays was underestimation of the complexity of manufacturing and assembling the core stage engine section—where the four RS-25 engines are mated to the core stage—and those activities have taken far longer than expected. For example, around the time of the December 2017 replan, the SLS program schedule indicated that it would take 4 months to complete the remaining work. By late 2018, the estimate for the same work had increased to 11 months. Part of that delay included time required to resolve residue and debris discovered in the fuel lines, which was present because Boeing had not verified the processes that its vendors were using to clean the fuel lines. Further, installation of the fuel lines overlapped with other work in the engine section, making work in the limited space more difficult and complex than it otherwise would have been. NASA officials indicated that there have been additional issues behind core stage delays, including the following: Boeing underestimated the staffing levels required to build the core stage in the time available. According to a NASA official, as core stage production began, Boeing was focused on minimizing the number of technicians, in part to keep costs low, and hired about 100 technicians. The official stated that Boeing now has about 250 technicians on staff in order to address ongoing delays, however, because a number of the additional staff came from non-spaceflight projects, some time was lost getting those staff up to speed on SLS. In addition, the official noted that technicians were spending time performing work away from the vehicle, such as collecting tools and parts for the work they were completing. According to the official, Boeing has since hired additional support staff to perform off-vehicle tasks such as pre-packaging tools in order to allow technicians to spend their time working on the vehicle. The build plans for the core stage were not adequately mature when the contractor began work on the hardware itself, which led to additional delays. For example, according to NASA officials, they expected the work instructions—detailed directions on how the vehicle should be built—to be largely complete by the program’s critical design review, which precedes the production decision. In this case, however, the build plans were not complete by the start of production. Officials stated that the lack of build plans slowed progress, as technicians can only perform work that they have instructions to carry out. In addition, the time to perform some work activities needed to build the designed vehicle was not included in the schedule. For example, more than 900 engine section brackets that were in the design were not on the schedule and, according to NASA officials, Boeing had to install the brackets later, adding complexity to the work schedule. Boeing officials provided three additional perspectives regarding the delays. Boeing officials explained that they did not anticipate any changes from NASA for the loads—impacts and stresses of mass, pressure, temperature, and vibration that the vehicle will experience—following the program’s critical design review, but instead NASA provided three significant updates to those loads. In some cases, the changes were significant enough that they invalidated legacy systems Boeing had planned to use, which required rework. However, SLS program officials stated that they continued to update loads data as the environments anticipated during launch became clearer. Boeing officials also stated that they alerted NASA in September 2014 that a decision to decrease funding in fiscal year 2015 would require the contractor to delay the core stage delivery date. In October 2018, however, the NASA OIG reported that while Boeing anticipated receiving $150 million less than planned in fiscal year 2015, the company received only $53 million less; that a funding increase was received in fiscal year 2016; and that the value of Boeing’s contract increased by nearly $1 billion in May 2016. Finally, Boeing officials stated that it has been challenging to execute NASA’s development approach that called for the first set of hardware built to be used for the initial launch. Boeing officials stated that they are more used to an approach in which they use the first hardware built to qualify the design and that hardware is never flown. The challenge with the current approach, according to Boeing officials, is that all the learning associated with a first build is occurring on the flight unit, which requires extra scrutiny and slows down the process. SLS program officials stated that this approach has been part of the development plan since the initial contract with Boeing was signed. One area in which the program has benefited from the core stage delay is that development of SLS test and flight software, which has been a schedule concern for the program, now has additional time to complete development. Delays to date have been due to late hardware model deliveries and requirements changes according to program officials. The SLS program completed the qualification test—a verification that the software meets documented requirements—for the green run software in March 2018. Program officials stated that the verified test software release will be complete by April 2019, and the EM-1 flight software release will be complete by October 2019. The earlier they are able to complete the software before launch, the more time they will have to complete testing, fix any defects they find, and work with EGS to integrate with the ground software. Measuring to a June 2020 launch date, flight software development has about 6 months of additional time to address issues should they arise. However, the program has a number of test cycles remaining and the program continues to assess a risk regarding the potential impact that late requirements changes could have on software completion. The SLS program has been underreporting its development cost growth since the December 2017 replan because of a decision to shift some costs to future missions while not adjusting the baseline downward to reflect this shift. The SLS development cost baseline established in August 2014 for EM-1 includes cost estimates for the main vehicle elements—stages, liquid engines, boosters—and other areas. According to program officials, because of the December 2017 replan process, NASA decided that costs included as part of the SLS EM-1 baseline cost estimate would be more appropriately accounted for as costs for future flights. Thus, NASA decided not to include those costs, approximately $782 million, as part of the revised SLS EM-1 cost estimate. However, NASA did not lower the $7 billion SLS development cost baseline to account for this significant change in assumptions and shifting of costs to future flights, and NASA officials told us that they were not sure what the benefit to NASA would be in adjusting the baseline. This decision presents challenges in accurately reporting SLS cost growth over time. NASA’s decision not to adjust the cost baseline downward to reflect the reduced mission scope obscures cost growth for EM-1. NASA’s cost estimate as of fourth quarter fiscal year 2018 for the SLS program indicated development cost growth had increased by $1 billion, or 14.7 percent. However, our analysis shows that development cost growth actually increased by $1.8 billion or 29.0 percent, when the development baseline is lowered to account for the reduced mission scope. Essentially, NASA is holding the baseline costs steady, while reducing the scope of work included in current cost estimates (see figure 4). NASA’s current approach for reporting cost growth misrepresents the cost performance of the program and thus undermines the usefulness of a baseline as an oversight tool. NASA’s space flight program and project management requirements state that the agency baseline commitment for a program is the basis for the agency’s commitment to the Office of Management and Budget (OMB) and the Congress based on program requirements, cost, schedule, technical content, and an agreed-to joint cost and schedule confidence level. Removing effort that amounts to more than a tenth of a program’s development cost baseline is a change in the commitment to OMB and the Congress and results in a baseline that does not reflect actual effort. Further, the baseline is a key tool against which to measure the cost and schedule performance of a program. A program must be rebaselined and reauthorized by the Congress if the Administrator determines that development costs will increase by more than 30 percent. Accounting for shifted costs, our analysis indicates that NASA has reached 29.0 percent development cost growth for the SLS program. In addition, as we previously reported in May 2014, NASA does not have a cost and schedule baseline for SLS beyond the first flight. As a result, NASA cannot monitor or track costs shifted beyond EM-1 against a baseline. We recommended that NASA establish cost and schedule baselines that address the life cycle of each SLS increment, as well as for any evolved Orion or ground systems capability. NASA partially concurred with the recommendation, but has not taken any action to date. By not adjusting the SLS baseline to account for the reduced scope, NASA will continue to report costs against an inflated baseline, hence underreporting the extent of cost growth. NASA’s Associate Administrator and Chief Financial Officer stated that they understood our rationale for removing these costs from the EM-1 baseline and agreed that not doing so could result in underreporting of cost growth. Further, the Associate Administrator told us that the agency will be relooking at the SLS program’s schedule, baseline, and calculation of cost growth. The Orion program is not on schedule to meet the June 2020 launch date for the first mission due to delays with the European Service Module and ongoing component issues with the avionics systems for the crew module, including issues discovered during testing. European Service Module (ESM). Through a barter agreement, the European Space Agency developed and produced the ESM, which provides propulsion, air, water, and power to the crew module while in space. The European Space Agency delivered the ESM to NASA in November 2018, following several delays with its development. According to program officials, the most recent set of delays prior to delivery were due to issues and failures during ESM propulsion system testing as well as the need to redesign power system components. Orion and EGS officials explained that a total of 20 months is required from receipt of the ESM to prepare it for launch. This time frame includes 14 months for the Orion program to finalize testing of each module and complete program-level integration and testing and 6 months for the EGS program to complete integrated test and checkout with SLS and EGS. As a result, the earliest the Orion program could be ready to support a first mission based on the service module schedule alone is July 2020, 20 months after NASA accepted delivery in November 2018. ESD officials told us that the 6 to 12 months of risk that could push EM-1 to June 2021 includes ESM-related delays. These 6 to 12 months of schedule risk do not include the effects, if any, of the federal government shutdown that occurred in December 2018 and January 2019. Figure 5 compares schedules of key events for the Orion program, including delays with the ESM, from shortly after NASA established the program’s baseline in September 2015, the December 2017 replan, and as of November 2018. Crew Module. While the ESM remains the critical path—the path of longest duration through the sequence of activities that determines the earliest completion date—for the Orion program, the crew module is nearly the critical path due in part to component failures within the avionics systems during testing. Figure 6 is a picture of a crew module test article. In May 2018, we reported that the Orion program was addressing component issues in its avionics systems after they failed during vibration testing. For example, components throughout the crew and service module relied on computer cards used to regulate power. When those cards cracked during testing, the program needed to redesign the cards, retest them, and reinstall them for system tests. Since then, additional avionics failures have surfaced. In one instance, one of the vehicle’s global positioning system receivers failed to power up. In another, a part failed on one of the inertial measurement units, which provide navigation information like vehicle rotation and acceleration. In March 2019, program officials told us that they have addressed these issues in the avionics systems and all flight hardware is installed. Testing. The ability for Orion, SLS, and EGS to complete testing in the integrated test laboratory facility—where software and hardware or hardware simulators are tested together—remains an ongoing risk for both the first mission and then the timing of the second mission. The lab has limited time and test resources to complete the testing necessary for EM-1, and NASA officials indicated that at times it has more demand than it can support. In addition, some testing is taking longer than planned, delaying later tests. The risk associated with these delays is that the later the program discovers an issue, the less time there is to address the issue prior to launch. At the same time that the Orion program is completing EM-1 work in the integrated test lab, the program will also need to modify the lab’s configuration in order to support EM-2 efforts because of hardware and software differences between missions. The schedule currently includes periods of time during EM-1 testing where EM-1 efforts will be shut down in order to work on lab modifications for EM-2. Although program officials indicated that test lab delays for EM-1 will not adversely affect lab efforts for EM-2, resources directed to EM-2 will mean less resources will be available during those times to support EM-1. The Orion program has reported development cost growth but is not measuring that growth using a complete cost estimate. In summer 2018, the Orion program reported development cost growth of $379 million, or 5.6 percent above its $6.768 billion development cost estimate. The program explained that the major drivers of this cost growth were the slip of the EM-1 launch date, which reflected delays in the delivery of the service module; Orion contractor underperformance; and NASA-directed scope increase. However, during our review, Orion program officials stated that this cost estimate assumes an EM-2 launch date of September 2022, which is 7 months earlier than the program’s agency baseline commitment date of April 2023 that forms the basis for commitments between NASA, the Congress, and OMB. As a result, NASA’s current cost estimate for the Orion program is not complete because it does not account for costs that NASA would incur between September 2022 and April 2023. Subsequently, program officials told us that its cost projections fund one of those seven months. See figure 7. NASA officials originally told us that they do not have an Orion cost estimate through the EM-2 agency baseline commitment launch date of April 2023 because they plan to launch by September 2022, if not earlier. According to scheduling best practices, performance is measured against the program’s baseline even if a program is working to an earlier date. By not estimating costs through its baseline launch date, the Orion program is limiting the NASA Associate Administrator’s insight into how the program is performing against the baseline. According to federal law, the Administrator must be immediately notified any time that a designated official has reasonable cause to believe that either the program’s development cost is likely to exceed the estimate in the agency baseline commitment by 15 percent or more or a program milestone will slip 6 months or more beyond its schedule agency baseline commitment date. If the Administrator confirms the cost growth or schedule delay exceeds the given threshold, the Administrator must submit a report to the Committee on Science and Technology of the House of Representatives and the Committee on Commerce, Science, and Transportation of the Senate. Given that the program is already reporting cost growth to a date earlier than its baseline schedule, updating the cost estimate relative to the EM-2 baseline schedule would provide NASA management and Congress with more complete cost data and increased awareness of whether additional oversight is merited. Since the December 2017 replan, the EGS program has had to address several technical challenges that consumed schedule reserves. Nevertheless, officials expect to have EGS facilities and software ready by June 2020, the planned launch date. The program has completed many of its projects, including the renovation of the Vehicle Assembly Building and the launch pad. Since the replan, however, the project has had to address technical challenges with the Mobile Launcher. Figure 8 below compares the EGS schedule—including timeframes for the Mobile Launcher and software completion—shortly after NASA established the program’s baseline in September 2014, the December 2017 replan, and as of November 2018. It also shows the potential launch window reflecting the 6-12 months of risk NASA is tracking that could push EM-1 to June 2021. Mobile Launcher. The Mobile Launcher schedule deteriorated since the December 2017 replan due to problems with finalizing construction work prior to moving it to the Vehicle Assembly Building. Moving the Mobile Launcher into the Vehicle Assembly Building was intended to allow the program to begin multi-element verification and validation, a process that checks that the various launch and processing systems at Kennedy Space Center meet requirements and specifications and can operate together to fulfill their intended purpose. Challenges the program experienced with the Mobile Launcher included having to add structural supports after determining that the design was not adequate to carry the load of the SLS vehicle and fuel. In addition, program officials stated that construction work overall did not progress to the point desired to move the Mobile Launcher to the Vehicle Assembly Building. As a result, the program did not move the Mobile Launcher into the Vehicle Assembly Building until September 2018, 5 months later than in the schedule established after the December 2017 replan. Moving forward, the program has to complete the multi-element verification and validation process for the Mobile Launcher and Vehicle Assembly Building. We have reported on a number of issues related to the EGS program’s management of the Mobile Launcher, as well as the now-completed Vehicle Assembly Building project. For example, in 2016, we found that the program did not mature requirements and designs for the Mobile Launcher before beginning construction. In addition, the EGS program completed all major structural changes to the Mobile Launcher prior to completing the design and installation of the ground support equipment and the nine umbilicals that connect the Mobile Launcher directly to the SLS and Orion. There have also been ground support equipment and umbilical design changes both during and after the Mobile Launcher’s design phase because of vehicle requirement changes from SLS and Orion. Officials indicated this approach was problematic because the concurrency increased program risk. Further, according to officials, the decision to have separate contracts for design and construction exacerbated these challenges. Officials indicated that this contracting strategy meant that design changes required multiple levels of review and approval from NASA and each of the program’s contractors, which in turn led to numerous contract modifications. According to EGS officials, the program plans to incorporate lessons learned from developing the first Mobile Launcher into the acquisition approach for a second Mobile Launcher that NASA is building to allow for future configurations of the SLS vehicle. Specific lessons officials plan to carry forward to the second Mobile Launcher include: implementing an integrated design process, including establishing a process to better handle requirement changes during design and construction; developing and maintaining a three-dimensional (3D) model to facilitate integrated design; and enabling builder involvement during the design process to avoid pitfalls during construction. However, these lessons learned do not address metrics to assess design stability before starting construction. Our work on acquisition best practices show that good processes that mature designs early in development and ensure that the design meets requirements can position a program for future success and lead to more predictable cost and schedule outcomes. Traditionally, we have used the number of releasable engineering drawings as a metric to assess design stability. Specifically, our work has found that achieving design stability at the product critical design review, usually held midway through product development, is a best practice. Completion of at least 90 percent of engineering drawings at this point provides tangible evidence that the product’s design is stable. We have also found that the U.S. Navy and the commercial shipbuilding industry use 3D product models as tools to document design stability. We found that there are aspects of shipbuilding that are analogous to building a Mobile Launcher in that both involve designing and building a large metal structure and installing multiple complex integrated systems to support complex functions such as launching spacecraft, or in the case of the Navy, launching aircraft and/or missile systems. NASA officials agreed that developing a Mobile Launcher is analogous to shipbuilding. Best practices for commercial shipbuilding indicate that 3D product models documenting 100 percent of the system’s basic and functional designs should be complete before construction begins. Basic design includes fixing the ship steel structure; routing all major distributive systems, including electricity, water, and other utilities; and ensuring the ship will meet the performance specifications. Functional design includes providing further iteration of the basic design, providing information on the exact position of piping and other outfitting in each block, and completing a 3D product model. The combined basic and functional designs in conjunction with the 3D product model provide the shipbuilder a clear understanding of the ship structure as well as how every system is set up and routed throughout the ship. This detailed knowledge allows commercial shipbuilders to design, build, and deliver complex ships such as floating production storage and offloading vessels, which are able to collect, process, and store oil from undersea oil fields, within schedule estimates. The improved design processes the EGS program is pursuing in the development of the second Mobile Launcher, including the development of a 3D model to facilitate integrated design, have the potential to improve program outcomes. Further, achieving design stability before beginning construction would also improve this potential. Software. The program’s two software development efforts represent the EGS critical path, and program officials stated that recent changes have begun to address previous challenges with the software development. For example, officials explained that the program has implemented iterative integration testing and has identified lead engineers for each software development area. The iterative integration testing involves conducting tests on smaller segments of software throughout the development process instead of waiting to conduct testing when a software release is fully complete. According to officials, these efforts allow the program to identify and correct errors prior to completing a full software drop. These changes have also resulted in lower numbers of issues found in some software releases. Further, the 6-month delay to the SLS and Orion programs has provided additional flexibility to EGS’s software development schedule. Finally, with respect to EGS’s performance against its cost baseline, EGS updated its cost estimate as part of the December 2017 replan. The EGS program continues to operate within costs established for the June 2020 launch date, $3.2 billion, but any delays beyond June 2020 will result in additional cost growth. NASA’s award fee plans for the SLS stages and Orion crew spacecraft contracts provide for hundreds of millions of dollars to incentivize contractor performance, but the programs continue to fall behind schedule and incur cost overruns. Our past work shows that when incentive contracts are properly structured, the contractor has profit motive to keep costs low, deliver a product on time, and make decisions that help ensure the quality of the product. Our prior work also shows, however, that incentives are not always effective tools for achieving desired acquisition outcomes. We have found that, in some cases, there are significant disconnects between contractor performance for which the contractor was awarded the majority of award fees possible without achieving desired program results. Additionally, we have found that some agencies did not have methods to evaluate the effectiveness of award fees. The incentive strategies for both the SLS stages and the Orion crew spacecraft contracts include multiple incentives—milestone fees, performance incentive fees, cost incentive fees, and award fees—aimed at incentivizing different aspects of contractor performance. These contracts’ milestone fees, performance incentive fees, and cost incentive fees are generally determined against objective criteria, such as meeting a date and application of predetermined formulas. For example, NASA will pay a milestone fee to Boeing under the SLS contract when it meets a specific program milestone such as transferring the core stage to the government for the green run test. Under this contract, Boeing receives additional milestone fee when it beats a milestone date and reduced fee when it misses a milestone date. Likewise, pre-determined formula-type incentives—such as these contracts’ performance incentive fees and cost incentive fees—are typically determined based on objective criteria, such as meeting technical metrics or predetermined cost targets. Award fees on these types of contracts are generally determined at 6 to 12-month periodic evaluations of the contractor’s performance against criteria outlined in the award fee plan. For example, according to officials, NASA may evaluate the contractor against technical performance and criteria, such as the ability to avoid and predict cost overruns, manage risk, or accomplish small business goals. Upon the completion of a formal review, performance evaluation board officials make recommendations to the fee determination official on the amount of fee to be paid. Figures 9 and 10 provide overviews of the total incentive fee available on the current contracts for the SLS stages contract and the Orion crew spacecraft contract, by type and percentage. Under the terms of the current contracts, Boeing has earned about $271 million in award fee and Lockheed Martin has earned about $294 million in award fee. Since each program held its confirmation review, the point in time when a program established its cost and schedule baselines, NASA has paid the majority of available award fee to both contractors. Specifically, NASA has paid Boeing about 81 percent of available award fee—or about $146 million—and Lockheed Martin about 93 percent—or about $88 million—since their respective program confirmation reviews. During the annual award fee periods, the descriptive ratings both contractors received ranged from good to excellent. In the subjective appraisals supporting these ratings, NASA identified both strengths that indicate areas of good contractor performance and weaknesses that indicate areas of poor contractor performance. Table 5 includes the results of award fee determinations since the respective program confirmations. The numerical score for each evaluation period represents the percentage of fee paid to the contractor from the available fee pool. Examples of strengths and weaknesses NASA identified in the award fee letters include the following: For the Boeing award fee period ending February 2015, NASA identified several strengths, including effective and timely communication, but stated that its subcontractor management for the vertical assembly center was inadequate. In particular, the program discovered during this time that the as-built design of the vertical assembly center tool was not capable of serving its purpose, which is to build core stage hardware. The design issue resulted in several months of schedule delays. NASA also raised concerns about Boeing’s ability to manage to the baseline schedule in a subsequent award period. For the Lockheed Martin award fee period ending April 2017, NASA identified several strengths, including addressing top program development risks such as establishing a robust mitigation plan to address risks related to the heatshield block architecture. At the same time, NASA noted that Lockheed Martin was not able to maintain its schedule for the crew service module and that the contractor’s schedule performance had decreased significantly over the previous year. While both the SLS and Orion contractors have received the majority of available award fee in each award fee period, the programs have not always achieved overall desired outcomes. For example, in its December 2018 award fee letter to Boeing—representing the good assessment for the September 2017 through October 2018 period of performance—the fee determination official noted that the significant schedule delays on this contract have caused NASA to restructure the flight manifest for SLS. As previously discussed, within 1 year of announcing a delay for the first mission, senior NASA officials acknowledged that the SLS and Orion programs will not meet the new EM-1 schedule of December 2019, and the 6 months of schedule reserve available to extend the date to at least June 2020 has been consumed. In addition, the officials identified 6 to12 months of risk to that date, which could increase the delay up to 31 months. These 6 to 12 months of schedule risk do not include the effects, if any, of the federal government shutdown that occurred in December 2018 and January 2019 due to a lapse in appropriations for fiscal year 2019. Both the contractors and government bear responsibilities for these delays. We have previously found that NASA has made programmatic decisions—including establishing low cost and schedule reserves, managing to aggressive schedules, and not following best practices for earned value management—that have compounded technical challenges that are expected for inherently complex and difficult large-scale acquisitions. Further, we previously reported that NASA did not follow best practices for establishing cost and schedule baselines for these programs nor update cost and schedule analyses to reflect new risks. As a result, NASA overpromised what it could deliver from a cost and schedule perspective. At the same time, both contractors have had challenges that contributed to past delays. For example, in 2015, Boeing was unable to manufacture an intertank panel—which resides between the liquid oxygen and liquid hydrogen tanks—without significant cracking. At the time, NASA estimated that resolving this issue could result in a 6-month slip to the production schedule. Further, as previously discussed, NASA discovered during installation that fuel lines used in the engine section were contaminated with residue and other debris. According to a program official, Boeing had not verified the processes that its vendors were using to clean the fuel lines, resulting in about 2 months’ delay to resolve residue and debris issues. SLS officials indicated that the engine section has a very complex design with many parts in a relatively small, cramped area, so any time problems are found with parts that have already been installed, removing, repairing or replacing them often requires that other parts be removed. Furthermore, as some of the tubing sections had already been installed, resolving this issue, including inspecting, shipping, and cleaning the tubing, affected the overall program schedule. In addition, NASA determined in 2017 that Lockheed Martin would not meet the delivery date for the crew module—even if the European Service Module were on schedule—when numerous problems including design issues, damage during testing, and manufacturing process changes resulted in major schedule impacts to the program. Lockheed Martin also had a number of issues with subcontractor-supplied avionics system components failing during testing that have required time to address. NASA has highlighted concerns over Lockheed Martin’s ability to manage subcontractors in award fee evaluation periods from 2016 to 2018, and the resulting significant cost, schedule, and technical risk impacts to the program. In an attempt to resolve these issues and to improve subcontractor oversight moving forward, Lockheed Martin officials told us that they have placed staff in the subcontractor facilities. Because of these cost increases and delays, the agency plans to renegotiate the Boeing contract for SLS. NASA officials stated that Boeing expects its costs to exceed the cost-reimbursement contract’s not- to-exceed estimated total cost, which will lead to contract renegotiation. Consequently, the contractor has been executing work under an undefinitized contract action since September 2018. Contract actions such as these authorize contractors to begin work before reaching a final agreement with the government on contract terms and conditions. Orion program officials stated that NASA is modifying the cost and period of performance aspects of its contract with Lockheed Martin for Orion development and negotiating a new contract with Lockheed Martin for Orion operations and production. Officials told us the following: NASA is modifying the Orion development contract with Lockheed Martin because the contractor will exceed the cost reimbursement contract’s not-to-exceed estimated total cost. Orion program officials indicated that poor performance on the part of the contractor resulted in the contractor exceeding the costs allowed under the contract without completing the full scope of work. Consequently, NASA is modifying the contract to allow increased costs. Orion officials indicated that since the cost growth is contractor caused, the contractor will not have the ability to earn any fees on this increased cost. NASA is also modifying the Orion development contract to extend the contract period of performance. The current contract’s period of performance ends in December 2020, which is earlier than NASA’s planned EM-2 launch date of June 2022. Orion program officials stated that this extension is largely driven by delays in receipt of the European Service Module. According to officials, NASA is negotiating the terms of the Orion production and operations contract with Lockheed Martin. This contract is expected to support future production of the Orion spacecraft from Exploration Mission-3 potentially through 2029. In addition to production, this effort will include sustaining engineering and flight operations support, with limited development to allow mission kits to be built to specifications as mission objective are defined. Orion program officials indicated that NASA plans to eventually transition the contract to a fixed-price type contract for production, but that the development of mission kits will remain under a cost-reimbursement type contract with some type of incentive fee. In November 2018, senior leaders within the ESD organization told us that it was not clear whether NASA would renegotiate how incentive fees are distributed among milestone incentive fee, or cost incentive fee, and award fee as part of the upcoming Boeing contract renegotiations. NASA, however, has made these types of changes in the past. For instance, the Orion program redistributed fees in 2014 to include an incentive fee component when the contract transitioned from the Constellation program to the Orion program. The Federal Acquisition Regulation and NASA contracting guidance indicate that award fee is appropriate when the work to be performed is such that it is neither feasible nor effective to devise predetermined objective incentive targets applicable to cost, schedule, and technical performance. However, now that the SLS and Orion programs are further into the acquisition life cycle, the programs are at the point in development wherein it may be possible to determine more objective targets for cost, schedule, and technical performance, especially for the first mission. Further, a principle of federal internal controls is that management should design control activities to achieve objectives and respond to risks. This includes management conducting reviews to compare actual performance to planned or expected results, and taking corrective actions to achieve objectives. Without reevaluating its strategy for incentivizing contractors, NASA will miss an opportunity to consider whether changes to the incentive structure could better achieve expected results, such as motivating the contractor to meet upcoming milestone events within cost and schedule targets. NASA’s SLS, Orion, and EGS programs are a multi-billion dollar effort to transport humans beyond low-Earth orbit, but the agency has been unable to achieve agreed-to cost and schedule performance. NASA acknowledges that future delays to the June 2020 launch date are likely, but the agency’s approach in estimating cost growth for the SLS and Orion programs is misleading. And it does not provide decision makers, including the Administrator, complete cost data with which to assess whether Congress needs to be notified of a cost increase, pursuant to law. By not using a similar set of assumptions regarding what costs are included in the SLS baseline and updated SLS cost estimates, NASA is underreporting the magnitude of the program’s cost growth. Similarly, NASA is underreporting the Orion program’s cost performance by measuring cost growth to an earlier-than-agreed-to schedule date. As a result, Congress and the public continue to accept further delays to the launch of the first mission without a clear understanding of the costs associated with those delays. Further, NASA is now turning its attention to new projects to support future missions, including building a second Mobile Launcher. Ensuring design stability before construction start would better position NASA to improve its acquisition outcomes for this next Mobile Launcher. Finally, contractor performance to date has not produced desirable program cost and schedule outcomes. Ongoing and planned contract negotiations present an opportunity to restructure the government’s approach to incentives. Such steps may better position the agency to obtain better outcomes going forward. We are making the following 4 recommendations to NASA: We recommend the NASA Administrator ensure that the NASA Associate Administrator for Human Exploration and Operations direct the SLS program to calculate its development cost growth using a baseline that is appropriately adjusted for scope and costs NASA has determined are not associated with the first flight, and determine if the development cost growth has increased by 30 percent or more. (Recommendation 1) We recommend the NASA Administrator ensure that the NASA Associate Administrator for Human Exploration and Operations direct the Orion program to update its cost estimate to reflect its committed EM-2 baseline date of April 2023. (Recommendation 2) We recommend the NASA Administrator ensure that the NASA Associate Administrator for Human Exploration and Operations direct the EGS program to demonstrate design maturity by completing 3D product modeling of the basic and functional design of the second Mobile Launcher prior to construction start. (Recommendation 3) We recommend the NASA Administrator ensure that the NASA Associate Administrator for Human Exploration and Operations direct the SLS and Orion programs to reevaluate their strategies for incentivizing contractors and determine whether they could more effectively incentivize contractors to achieve the outcomes intended as part of ongoing and planned contract negotiations. (Recommendation 4) NASA provided written comments on a draft of this report. These comments, and our assessment of them, are included in appendix II. NASA also provided technical comments, which were incorporated as appropriate. In responding to a draft of this report, NASA concurred with three recommendations and partially concurred with a fourth recommendation, and identified actions that they plan to take. NASA partially concurred with our recommendation to direct the Orion program to update its cost estimate to reflect its committed EM-2 baseline date of April 2023. In its response, NASA stated providing the estimate to the forecasted launch date—September 2022—rather than to the committed baseline date of April 2023 is the most appropriate approach. Further, NASA stated that any additional slips to the program involve considerable uncertainty associated with “unknown-unknowns” which are, by their very definition, impossible to predict or forecast and that attempting to forecast these at this point is neither practical nor useful to help manage the program. If the schedule projections go beyond September 2022, NASA stated that the Orion program will follow standard Agency processes and update its cost estimate to reflect the updated schedule projections. NASA established Orion’s EM-2 launch date of April 2023 as part of the agency’s program confirmation process in 2015. According to federal law, NASA is required to track and report progress relative to the cost and schedule baselines established at the program’s confirmation review. While programs often pursue goals trying to beat these dates and/or cost estimates, the primary purpose of a cost and schedule baseline is to provide a consistent basis for measuring program progress over time. By developing cost estimates only to the program’s goals and not relative to the established baseline, the Orion program is not providing the Agency or the Congress the means of measuring progress relative to the baseline. We agree that it is difficult to forecast the potential impacts of unexpected problems. NASA guidance, however, provides instructions to programs on the percentage/relative level of cost reserves that should be maintained to deal with potential unknown-unknowns that are likely to come up late in development. We continue to believe that NASA should fully implement this recommendation. We are sending copies of this report to the NASA Administrator and interested congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. To assess the performance of the human space exploration programs, including any technical challenges, relative to their cost and schedule commitments, we obtained and analyzed cost and schedule estimates for the Space Launch System (SLS), Orion Multi-Purpose Crew Vehicle (Orion), and Exploration Ground Systems (EGS) programs through November 2018. We then compared these estimates against program baselines to determine cost growth and schedule delays. We also interviewed SLS program officials and reviewed cost data to determine how the program phases costs for future flights outside the current baseline. We then analyzed the SLS program’s current cost estimate to determine how the scope of the current estimate had changed relative to the scope of the SLS baseline cost estimate. Moreover, we obtained and reviewed quarterly reports and the programs’ risk registers, which list the top program risks and their potential cost and schedule impacts, including mitigation efforts to-date. We then discussed risks with program officials. We also compared program schedules across three points in time— schedules from when NASA first established baselines for each program, schedules established for each program following the replan in December 2017, and schedules as of November 2018—to assess whether program components and software were progressing as expected Furthermore, for the EGS program, we reviewed program-level lessons learned regarding the acquisition of the Mobile Launcher against acquisition best practices to determine the extent to which the program plans to incorporate these best practices as part of its acquisition planning for the second Mobile Launcher. To determine the extent to which NASA’s use of contract award fees are achieving desired outcomes, we analyzed contract modifications, award fee plans, and fee determination records for the Orion crew spacecraft and SLS stages—or stages—contracts. We selected these contracts because they represent the largest development efforts for each program. We analyzed contract documentation to determine the amount of award fee available on these contracts compared to other incentives, such as milestone incentives, and calculated fees paid to date. Specifically, for award fee on both contracts, we reviewed fee determination records for evaluation periods after the SLS program’s confirmation review in 2014 and the Orion program’s confirmation review in 2015 to determine fees paid, numeric and descriptive ratings awarded for each period and contractor strengths and weaknesses identified by the program. Moreover, we reviewed award fee documentation to identify broader program challenges and compared fee determination results to overall program outcomes since program confirmation. For the Orion contract, the scope of our incentive fee analysis included the full scope of incentive fees available for developing and manufacturing the Orion spacecraft from the beginning of the contract. For the SLS contract the scope of our incentive fee analysis included the incentive fees available for 1) contract line item number 9 of the contract which includes the full scope of stages work supporting SLS’s EM-1 effort, and 2) contract line item number 12 indefinite-delivery, indefinite-quantity support task activities for contract line item number 9. We performed our work at Johnson Space Center in Houston, Texas; the Boeing Company in Huntsville, Alabama; Marshall Space Flight Center in Huntsville, Alabama; Kennedy Space Center in Kennedy Space Center, Florida; Lockheed Martin Space Systems Company in Houston, Texas; and NASA headquarters in Washington, DC. We based our assessment on data collected prior to the federal government shutdown that occurred in December 2018 and January 2019 due to a lapse in appropriations for fiscal year 2019. This assessment does not reflect the effect, if any, of the shutdown on the programs’ costs and schedules or a March 2019 announcement that NASA is studying how to accelerate the SLS schedule. We assessed the reliability of program data we used to support this engagement using GAO reliability standards as appropriate, including reviewing related documentation, interviewing knowledgeable agency officials, and performing selected testing of data. We determined the data was sufficiently reliable for the purposes of this engagement. We conducted this performance audit from March 2018 to June 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. 1. This report acknowledges the complexity of NASA’s deep space exploration systems. The introduction section of this report acknowledges that NASA is developing systems planned to transport humans beyond low-Earth orbit, including the Moon and eventually Mars, and that each of these programs represents a large, complex technical and programmatic endeavor. The introduction also notes that these programs are in the integration and test phase of development, which our prior work has shown often reveals unforeseen challenges leading to cost growth and schedule delays. 2. Senior NASA officials told us that the revised EM-1 launch date of December 2019 is unachievable and the June 2020 launch date (which takes into account schedule reserves) is unlikely. These officials then estimated that there are 6 to 12 months of schedule risk associated with the June 2020 date. It would be misleading for us to continue to report the June 2020 launch date when we were told there was substantive risk to that date. Without a new approved schedule, Figure 3, Figure 5, and Figure 8 all present a notional launch window including the acknowledged schedule risks. We then used the information NASA provided us to report that the first launch may occur as late as June 2021, if all risks are realized. Further, this substantial delay to the first mission was acknowledged by senior officials less than one year after NASA announced up to a 19 month delay. We maintain that continued underperformance contributed to these additional schedule delays and associated cost increases. For example, for SLS, NASA discovered during installation that fuel lines used in the engine section were contaminated with residue and other debris. According to a program official, Boeing had not verified the processes that its vendors were using to clean the fuel lines, resulting in about 2 months’ delay to resolve residue and debris issues. For the Orion program, NASA determined in 2017 that Lockheed Martin would not meet the delivery date for the crew module—even if the European Service Module were on schedule—when numerous problems including design issues, damage during testing, and manufacturing process changes resulted in major schedule impacts to the program. As a result, we also maintain that these delays and cost growth reinforce concerns over the management of the programs. In addition to the underperformance, NASA’s management decisions on how to report cost growth is not fully transparent and, in particular, obscures the difficulties the SLS program has faced controlling costs. 3. We agree that that these are long-term, “multi-decadal” programs and that content is subject to change. As a result, we maintain that arbitrarily focusing on a single mission and not looking at long- term costs may have negative impacts to this human spaceflight system. We previously reported in May 2014, that NASA does not have a cost and schedule baseline for SLS beyond the first flight. As a result, NASA cannot monitor or track costs shifted beyond EM-1 against a baseline. We recommended that NASA establish cost and schedule baselines that address the life cycle of each SLS increment, as well as for any evolved Orion or ground systems capability. NASA partially concurred with the recommendation, but has not taken any action to date. Until action is taken to do so, as noted above, NASA’s decision to shift some SLS costs to future missions while not adjusting the baseline downward not only underestimates cost growth for the first mission, but also results in there being no mechanism to track these costs that NASA shifted to future missions. 4. Through the course of this review, NASA was transparent in its discussions with us of how it calculated costs for each of the programs. The findings of this report are not meant to convey that NASA is withholding information, but rather, that decisions NASA has made about how to calculate costs do not provide sufficient transparency into cost growth or cost estimates. Further, we have previously reported that without transparency into costs for future flights, NASA does not have the data to assess long-term affordability and Congress cannot make informed budgetary decisions. Cristina T. Chaplain, (202) 512-4841 or chaplainc@gao.gov. In addition to the contact named above, Molly Traci, Assistant Director; Andrea Bivens; Sylvia Schatz; Ryan Stott; Tanya Waller; John Warren; Alyssa Weir; and Robin Wilson made significant contributions to this report.", "summary": "NASA is undertaking a trio of closely related programs to continue human space exploration beyond low-Earth orbit. All three programs (SLS, Orion, and supporting ground systems) are working toward a launch readiness date of June 2020 for the first mission. The House Committee on Appropriations included a provision in its 2017 report for GAO to continue to review NASA's human space exploration programs. This is the latest in a series of reports addressing the mandate. This report assesses (1) how NASA's human space exploration programs are performing relative to cost and schedule commitments, and (2) the extent to which NASA's use of contract award fees is achieving desired program outcomes. To do this work, GAO examined program cost and schedule reports and contractor data, and interviewed officials. This report does not assess the effect, if any, of the government shutdown that ended in January 2019. Due to continued production and testing challenges, the National Aeronautics and Space Administration's (NASA) three related human spaceflight programs have encountered additional launch delays and cost growth. In November 2018, within one year of announcing an up to 19-month delay for the three programs—the Space Launch System (SLS) vehicle, the Orion spacecraft, and supporting ground systems—NASA senior leaders acknowledged the revised date of June 2020 is unlikely. Any issues uncovered during planned integration and testing may push the launch date as late as June 2021. Moreover, while NASA acknowledges about $1 billion in cost growth for the SLS program, it is understated. This is because NASA shifted some planned SLS scope to future missions but did not reduce the program's cost baseline accordingly. When GAO reduced the baseline to account for the reduced scope, the cost growth is about $1.8 billion. In addition, NASA's updated cost estimate for the Orion program reflects 5.6 percent cost growth. The estimate is not complete, however, as it assumes a launch date that is 7 months earlier than Orion's baseline launch date. If the program does not meet the earlier launch date, costs will increase further. Updating baselines to reflect current mission scope and providing complete cost estimates would provide NASA management and Congress with a more transparent assessment of where NASA is having difficulty controlling costs. NASA paid over $200 million in award fees from 2014-2018 related to contractor performance on the SLS stages and Orion spacecraft contracts. But the programs continue to fall behind schedule and overrun costs. Ongoing contract renegotiations with Boeing for the SLS and Lockheed Martin for the Orion program provide NASA an opportunity to reevaluate its strategy to incentivize contractors to obtain better outcomes. GAO is making four recommendations to NASA, including that the SLS program should calculate cost growth based on costs that are currently included in the first mission and the Orion program should update its cost estimate to reflect the schedule agreed to in its baseline. In addition, the SLS and Orion programs should reevaluate their strategy for incentivizing contractors. NASA concurred with three recommendations, and partially concurred with the recommendation related to the Orion program's cost estimate. GAO believes the recommendation remains valid, as discussed in the report.", "document_type": "gao"}
{"report": "In 2017, nearly one in three Medicare Part D beneficiaries received an opioid prescription, and Medicare spending for prescription opioids was almost $3.4 billion. Medicare data for 2017 showed that the beneficiaries potentially at risk for opioid misuse and abuse were more likely to be under 65 years of age, female, and dually eligible for Medicare and Medicaid. For years, to limit Medicaid at-risk beneficiaries’ access to controlled substances, state Medicaid programs have used lock-in programs, which restrict certain beneficiaries’ access to a single prescriber (such as a physician or other health-care provider), a single pharmacy, or both. States have broad discretion in how and whether to implement lock-ins, including how to identify at-risk beneficiaries. Medicaid lock-in programs have been associated with advantages including declines in inappropriate prescribing, decreases in abuse of controlled substances, and increased treatment options for beneficiaries. Research has identified an unintended consequence of the Medicaid lock-in programs; some individuals increased the amount of out-of-pocket payments they made for controlled substances, thus avoiding lock-in programs as data on these purchases are not collected by Medicaid. State Prescription Drug Monitoring Programs (PDMP) are state-run electronic databases that track the prescribing, dispensing, and purchasing of controlled substances by individuals, whether purchased out-of-pocket, through private insurance, or under insurance programs such as Medicare and Medicaid. Data are available to individuals or organizations as authorized under state law. For example, health-care prescribers can check PDMPs for a specific individual prior to prescribing controlled substances. State PDMPs have documentation of controlled substances obtained by individuals. As of February 2018, all 50 states, the District of Columbia, Guam, and Puerto Rico had operational PDMPs within their borders. Medicare DMPs perform case management to identify at-risk beneficiaries and attempt to mitigate risk by increasing communication and coordination across plan sponsors, health-care prescribers, pharmacists, and at-risk beneficiaries. CMS established a framework for plan sponsors that volunteer to establish Medicare DMPs. CMS’s framework includes two steps for identifying at-risk beneficiaries: 1) CMS identifies potentially at-risk beneficiaries based on key clinical factors and 2) plan sponsors use case management to identify the subset of potentially at-risk beneficiaries who are actually at risk. Step One: Identifying potentially at-risk beneficiaries. CMS identifies potentially at-risk beneficiaries based on the minimum Overutilization Monitoring System’s criteria and develops a quarterly list of these beneficiaries, which it sends to plan sponsors. The Overutilization Monitoring System was originally designed and implemented by CMS in July 2013 to oversee plan sponsors’ compliance with CMS’s opioid overutilization policy and is based on beneficiary claims data. It was enhanced in 2019 to support the implementation of DMPs. The system encompasses a medication safety approach with the goal of reducing beneficiary overutilization of opioids while maintaining beneficiary access to needed medication. The Overutilization Monitoring System creates and sends a list of beneficiaries meeting minimum criteria quarterly to plan sponsors with DMPs. These patients are considered potentially at- risk beneficiaries. Additional supplemental criteria can also be used by plan sponsors to identify other potential at-risk beneficiaries, such as those who use seven or more pharmacies for opioid prescriptions. Use of higher numbers of prescribers and pharmacies may put the beneficiary at more risk. (See table 1.) Step Two: Using case management to identify actually at-risk beneficiaries and mitigate risk of misuse or abuse. After identification of potentially at-risk beneficiaries by CMS, the second step in the framework is for plan sponsors to coordinate the provision of care, referred to as case management. Plan sponsors use case management to determine which potentially at-risk beneficiaries are deemed to be actually at risk. Plan sponsors’ clinicians must begin case management for each potentially at-risk beneficiary with the beneficiary’s prescribers of frequently abused drugs, with the purpose of determining if current treatment is appropriate, examining specifically if the beneficiary is being appropriately treated with frequently abused drugs. This step allows for health-care providers to bring the beneficiary’s clinical information into the discussion. Two groups of beneficiaries identified as potentially at risk are excluded from Medicare DMPs during the case management process: First, CMS will automatically exempt from Medicare DMP consideration any beneficiary who (a) has elected to receive hospice, palliative, or end-of- life care; (b) is a resident of a long-term-care facility or of another facility for which frequently abused drugs are dispensed for residents through a contract with a single pharmacy; or (c) has active cancer-related pain. Second, through a discussion of beneficiary-level clinical information as part of case management, beneficiaries found to be receiving appropriate treatment, or who have had cases resolved with adjustments to their treatment will also be excluded from Medicare DMP consideration. CMS officials told us that the beneficiaries’ health-care providers often adjust their prescribing after talking with the health plans’ clinicians and learning information about other health-care providers also treating the beneficiary. After eliminating the excluded beneficiaries, the plan sponsor sends an initial notification letter to the remaining beneficiaries who continue to be identified as potentially at risk, and thus, may require limiting the coverage of a prescription drug in some manner (coverage limitation). The notification letter notifies each beneficiary that he or she has been identified as potentially at risk; details which coverage limitations the sponsor intends to implement and for how long; explains how the beneficiary can submit preferences for the selected frequently abused drugs prescriber(s) and the selected dispensing pharmacy or pharmacies for frequently abused drugs in case a lock-in tool is used; provides information about resources and plan benefits that address prescription abuse; explains how additional information can be provided to the plan sponsor; and informs the beneficiary of the right to appeal, among other things. Medicare plan sponsors have three coverage limitation tools that can be used either concurrently or sequentially: Frequently abused drugs prescriber lock-in. The at-risk beneficiary will receive prescriptions from only one or more selected prescriber. Prescribers in a group practice will count as a single prescriber. Frequently abused drugs-dispensing pharmacy lock-in. The at- risk beneficiary will receive all prescriptions from one or more selected dispensing pharmacy. The pharmacy must be in their plan sponsor’s network. When a pharmacy has different locations that share real-time electronic data, such as chain pharmacies, all locations of the pharmacy will be treated as one pharmacy. Beneficiary specific point-of-sale claim edit. The at-risk beneficiary will be restricted to certain frequently abused drugs and amounts through the point-of-sale claim edit. This means that a plan sponsor must not cover a prescription for the frequently abused drug for an at- risk beneficiary that is in excess of the limit in the edit. Pharmacists will receive a message when a beneficiary attempts to fill a prescription if the prescription exceeds the limit in place for that beneficiary. After a 30-day period from the date of the initial notification letter, there are two possible outcomes: either the plan sponsor will determine that the beneficiary is at risk for misuse or abuse of frequently abused drugs and will proceed with the coverage limitation under its DMP, or the sponsor will determine that the beneficiary is not at risk. If the potentially at-risk beneficiary is found to be at risk, a second notification letter is sent to the beneficiary as soon as possible after the 30-day period but no later than 60 days from the date of the initial notification letter. The second letter includes the beneficiary’s identification as at risk for misuse or abuse of frequently abused drugs, the right to appeal the decision, coverage limitations to be employed, and the expiration date of the coverage limitations, among other things. The second notification letter also explains how the beneficiary can submit preferences for the selected frequently abused drugs prescriber(s) and the selected dispensing pharmacy or pharmacies for frequently abused drugs in case a lock-in tool is used. The selected frequently abused drugs prescriber(s) must agree to serve as the beneficiary’s only frequently abused drugs prescriber(s) and must be determined by the plan sponsor as not contributing to the beneficiary’s opioid misuse. The at-risk beneficiary generally has 60 days from the date of the second notification letter to request an appeal of an at-risk determination. A new at-risk determination is made by the plan sponsor as a result of continued case management (that is, continued coordination of care and clinical discussions) within a standard time frame of 7 days for redetermination, or an expedited time frame of 72 hours for redetermination. Alternatively, if it is determined that the potentially at-risk beneficiary is being treated appropriately for their medical condition and is therefore not at risk, an alternate second notification letter is sent to the beneficiary stating that the beneficiary was deemed not to be at risk and that access to frequently abused drugs will not be limited under a DMP. This alternate second notification letter must be sent no later than 60 days after the date on the initial notification letter. If a beneficiary switches to a different plan sponsor after being identified as potentially at risk or actually at risk, the determination does not automatically transfer to the new plan sponsor. The new plan sponsor will be able to see the previous plan sponsor’s determination, but the new plan sponsor may have to make its own determination through case management unless the prior plan’s case management information is up to date. According to CMS’s framework, the termination date of an at-risk determination is the earlier of two possible dates: 12 months from the effective date of the coverage limitation, unless the limitation is extended, or when the beneficiary demonstrates that he or she is no longer likely to be at risk before the end of the 12 months. At the end of the 12-month period, a clinical assessment of the at-risk beneficiary will determine if the DMP should continue for another year or be ended. The maximum coverage limitation period in a DMP is 24 months. After 24 months, the coverage limitation is halted, but according to the framework, CMS continues to monitor the beneficiary quarterly through the Overutilization Monitoring System and can re-identify the beneficiary as potentially at risk again if he or she meets the minimum Overutilization Monitoring System criteria. Reporting requirements for Medicare DMPs. Although the Medicare DMPs are currently voluntary, the framework places reporting requirements on plan sponsors that choose to establish DMPs. Medicare DMPs are integrated with the Opioid Drug Utilization Review Policy and Overutilization Monitoring System to improve medication safety. Plan sponsors must report to CMS, through the Overutilization Monitoring System, the results of case management for each potential at-risk beneficiary identified. For each beneficiary deemed to be at risk, the coverage limitation tools – frequently abused drugs prescriber lock-in, frequently abused drugs-dispensing pharmacy lock-in, and beneficiary specific point-of-sale claims edit – that will be used to limit the beneficiary’s access to frequently abused drugs must also be reported. Other required information to be reported to CMS includes dates of the initial and second notification letters, and the date that the plan sponsor decides to terminate a potential at-risk or at-risk status. This data allows CMS to track beneficiary-level data for those beneficiaries placed in DMPs. Input about Medicare DMP framework from plan sponsors and experts. Although the at-risk determination is the responsibility of the plan sponsor, four of the five plan sponsors we interviewed stated they would rely on results from case management, including the beneficiary’s primary health-care provider’s input, to make the determination. One plan sponsor told us it has a panel of physicians and pharmacists that makes the at-risk determination for beneficiaries. The panel takes into consideration the input of the beneficiary’s prescribers of frequently abused drugs, but the panel makes the final decision. None of the five plan sponsors we interviewed expressed concerns about beneficiaries not receiving clinically appropriate doses of opioids under the Medicare DMPs, given that the case management process includes clinician input, including the beneficiary’s primary health-care provider and plan sponsors’ clinicians. According to one medical expert, case management is beneficial because it allows for input from the beneficiary’s prescriber rather than relying solely on the Overutilization Monitoring System metric criteria to determine who is at risk. Three of the five plan sponsors we interviewed expressed prior positive experiences with lock-ins. All plan sponsors had experience with the lock- in tools from their experience participating in Medicaid programs or offering private insurance plans. Three of the five plan sponsors reported better coordinated oversight and management of patients and their conditions in those experiences. One sponsor stated that lock-in programs are perceived as an additional safety mechanism for at-risk beneficiaries by helping with the coordination of the beneficiaries’ care because the plan sponsor’s clinicians, prescribers, and pharmacies are aware of the lock-in programs. Officials with five plan sponsors and six stakeholder organizations representing physicians, pharmacy benefit managers, and patients we interviewed told us that several factors beyond the clinical input incorporated in the case management process could contribute to the success of DMPs. These factors include communication, collaboration, and flexibility for plan sponsors to manage and incorporate best practices. Communication among plan sponsors, opioid prescribers, and pharmacies dispensing opioids could reduce any potential resistance to DMPs by stakeholders, and contribute to DMP success, officials told us. Specifically, several officials with plan sponsors and stakeholders noted that sponsors should communicate with, and educate stakeholders to ensure that DMPs, especially their coverage limitation tools, are not viewed as punitive tools by beneficiaries, but rather as tools for keeping them safe. Some of these officials stated that through education and other means, plan sponsors should also encourage health-care providers and their enrollees to view opioid addiction by at-risk beneficiaries as a disease rather than a choice. Effective collaboration among prescribers and patients in making individualized treatment decisions for pain management is another key to successful DMPs, according to some stakeholders we interviewed. Specifically, some plan sponsors and a stakeholder noted that during and after case management plan sponsors should support collaboration between prescribers and patients to ensure access to the clinically appropriate level of opioids for each beneficiary. Flexibility to incorporate best practices from DMPs and flexibility in varying coverage limitation features to fit regional differences are other keys to success, some plan sponsors and a stakeholder told us, due to regional and other differences in population groups. Data from CDC show that there is some variation in death rates from opioids in the overall population by state, and a CDC study also shows that opioid prescribing patterns also vary by location—the average per capita prescribed amounts varied widely by county, and demographic and other factors are sometimes associated with higher prescribing patterns. In addition, some officials with plan sponsors and stakeholders stated that the criteria for DMPs such as how at-risk beneficiaries are identified, and other parameters of the DMP (such as which drugs are designated as frequently abused drugs) should be periodically reassessed by CMS with feedback by plan sponsors and adjusted, where appropriate, to incorporate evidence from the outcomes of the DMP. Officials with plan sponsors and stakeholders we interviewed also discussed several factors that could limit the success of the DMPs: An official with a plan sponsor and a stakeholder told us that purchases of opioids made by beneficiaries using cash are not captured in the available prescription drug claims data maintained by plan sponsors. These officials suggested therefore that sponsors should encourage the selected prescribers to review this information by using state PDMPs, which track controlled substance prescriptions in a state and include cash sales of opioids. One plan sponsor commented that coverage limitations could have a negative effect on beneficiary health satisfaction survey scores and therefore could affect how a physician prescribes opioids. Beneficiaries who are placed in a DMP or subject to one of the DMP tools may report decreased satisfaction in the “Consumer Assessment of Healthcare Providers and Systems” surveys. Under Medicare Advantage, patient experience about the ease of receiving care is one measure of quality and is a factor in plan sponsor reimbursement. Finally, some officials with plan sponsors and stakeholders told us that in localities where prescribers feel a stigma associated with dispensing opioids, the coverage limitation feature could have the unintended consequence of creating a disincentive for prescribers to take care of patients who require opioid treatment. According to an official, to combat the opioid crisis, some states send letters to the top opioid prescribers in their state as a way to create awareness about trends in opioid prescribing. Officials with a plan sponsor and a stakeholder said that providers who want to avoid being on the list of top prescribers refer patients to pain specialists for management of pain. Officials told us that the demand for these pain specialists is increasing and there are long waiting lists to access care by these pain specialists. CMS is taking several steps to assess the DMPs and gather information to make periodic changes to the program. For example, CMS officials told us they plan to monitor and analyze beneficiary complaints, appeals, prescription drug event data, and other data submitted by plan sponsors related to DMPs. According to CMS officials, the agency also will obtain feedback from plan sponsors about how the DMPs are working, and this feedback will be the basis for making periodic changes to the DMPs. CMS is also updating its Medicare Part C and D audit protocol so it can audit DMP related beneficiary notices. Finally, CMS tracks the utilization of opioids by Medicare Part D enrollees using Part D data and the Overutilization Monitoring System. For example in the 2020 Medicare Advantage and Part D draft call letter, CMS reported that between 2012 and 2017, there was a 33 percent decrease in the number of Part D enrollees meeting or exceeding 90 MME for at least one day, with the largest decrease (14 percent) in 2017. We provided a draft of this report to HHS for review. HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, the Administrator of CMS, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at CosgroveJ@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix I. In addition to the contact named above, Martin T. Gahart (Assistant Director), N. Rotimi Adebonojo (Analyst in Charge), Jennie Apter, Deborah J. Miller, Emily Wilson, Rick Lipinski, Todd Anderson, and Vikki Porter made key contributions to this report.", "summary": "Misuse and abuse of prescription opioids can lead to overdose and death. According to the Centers for Disease Control and Prevention (CDC), 47,600 overdose deaths in the United States in 2017 involved an opioid. GAO and other federal entities have raised concerns about opioid misuse and abuse in Medicare. The Comprehensive Addiction and Recovery Act of 2016 (CARA) authorized CMS and Medicare plan sponsors to establish voluntary DMPs that may limit access to frequently abused prescription drugs, such as opioids, for Medicare beneficiaries who are identified as being at risk for prescription drug abuse. DMPs will become mandatory in Medicare starting in January 2022. CARA included a provision for GAO to review DMPs under Medicare. This report: 1) describes how Medicare identifies beneficiaries at risk of opioid misuse and abuse and how it attempts to mitigate that risk; and 2) identifies the factors likely to affect the success of Medicare DMPs. GAO reviewed CDC's Guideline for Prescribing Opioids for Chronic Pain , CMS regulations, and other relevant CMS guidance. GAO also interviewed officials from CMS, the five largest Medicare Part D prescription drug plan sponsors, and officials from six other stakeholder organizations representing Medicare plan sponsors, physicians (including pain specialists), pharmacy benefit managers, state Medicaid programs, and patients. Medicare's drug management programs (DMP) identify beneficiaries at risk of opioid misuse or abuse, and attempt to mitigate that risk through the use of case management and coverage limitations. DMPs are overseen by the Centers for Medicare & Medicaid Services (CMS) and voluntarily implemented by Medicare Part D prescription drug plan sponsors (private health plans). CMS established a two-step framework for identifying at-risk beneficiaries under DMPs. First, CMS identifies potentially at-risk beneficiaries based on key factors, such as beneficiaries' daily dosage of opioids and the number of prescribers and pharmacies from which they receive opioids, with higher numbers possibly putting the beneficiary at more risk. Second, Medicare Part D prescription drug plan sponsors' clinicians coordinate the provision of care among prescribers and pharmacists (referred to as case management) to determine if those potentially at-risk beneficiaries are actually at risk. If a patient is deemed to be at risk, coverage limitation tools—such as limiting a beneficiary to a selected prescriber or pharmacy, and implementing point-of-sale restrictions on certain drugs or amounts—can be used to limit the at-risk beneficiary's access to opioids. Beneficiaries have an opportunity to appeal an at-risk designation. None of the five plan sponsors GAO interviewed expressed concerns about beneficiaries not receiving clinically appropriate doses of opioids under the Medicare DMPs. Medicare Part D prescription drug plan sponsors and other stakeholders GAO interviewed reported several factors beyond the case management process that could contribute to the success of DMPs. These factors included communication among sponsors, opioid prescribers, and pharmacies dispensing opioids to reduce potential resistance to participating in DMPs by opioid prescribers or beneficiaries. According to plan sponsors and stakeholders, plan sponsors could communicate with stakeholders to ensure that DMPs are not viewed as a punitive tool by beneficiaries, but rather as tools for keeping them safe. Plan sponsors and stakeholders noted that it is important for plan sponsors to have flexibility in varying coverage limitation features to fit regional and other differences in population groups. They noted that CMS should periodically reassess and adjust the elements of the DMP program where appropriate, to incorporate evidence from the outcomes of the DMP—such as how at-risk beneficiaries are identified, or which drugs are selected as frequently abused drugs. Finally, CMS officials told GAO that they are taking steps to assess the DMPs and gather the information required to make periodic changes to the DMP program. For example, CMS officials plan to analyze data for at-risk beneficiaries that DMPs are required to report to CMS, update their Medicare Part D audit protocol, and obtain feedback from plan sponsors about how the DMPs are working. The Department of Health and Human Services provided technical comments on a draft of this report, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "The Air Force has identified ABMS as its solution to support broad Department of Defense (DOD) efforts to develop Joint All-Domain Command and Control (JADC2) capabilities. These capabilities will eventually allow U.S. forces from all of the military services, as well as allies, to conduct military operations across all warfighting domains. Command and control is the collection and sharing of information to enable military commanders to make timely, strategic decisions; take tactical actions to meet mission goals; and counter threats to U.S. assets. Figure 1 shows the concept of DOD operations within a joint all-domain environment. When the Air Force began planning for ABMS in 2017, officials stated the intent was to replace and modernize the capabilities of the Airborne Warning and Control System (AWACS), which provides the warfighter with the capability to detect, identify, and track airborne threats, among other capabilities. According to officials, the Air Force currently plans to operate AWACS aircraft through 2035. In July 2018, the DOD Joint Requirements Oversight Council approved an ABMS Initial Capabilities Document that describes which capabilities would need to be developed and which associated gaps in current capabilities the Air Force would need to address. According to Air Force officials, after the Initial Capabilities Document was approved, the Air Force determined that its planned approach to ABMS was no longer compatible with the most recent National Defense Strategy, released in January 2018. The 2018 National Defense Strategy outlines DOD’s strategy for maintaining the defense of the United States based on new and reemerging threats from competitors, such as Russia and China. It also defines expectations for how DOD and its military departments should be prepared to engage those threats during future conflicts: forces would be expected to strike a diverse range of targets inside adversarial air and missile defense networks; forces would need capabilities to enhance close combat lethality; and DOD would prioritize investments that enabled ground, air, sea, and space forces to deploy, operate, and survive in all domains while under attack. Air Force officials stated that these expectations led the department to reassess requirements for ABMS and assess new options for developing more robust and survivable systems that could operate within contested environments. For example, the Air Force officially canceled a recapitalization program for the Joint Surveillance Target Attack Radar System (JSTARS)—an aircraft that provides surveillance and information on moving ground targets—in December 2018. The cancellation was linked to the 2018 National Defense Strategy, which calls for a more survivable and networked solution, among other things. A June 2018 Air Force report to Congress identified concerns regarding the survivability of the JSTARS aircraft in a contested environment and stated that the Air Force was instead planning for ABMS to eventually provide JSTARS’s capabilities. The Air Force determined that it could continue using some of its JSTARS aircraft into the 2030s. Officials stated the Air Force subsequently changed the scope and intent of ABMS to align with the 2018 National Defense Strategy and broader requirements for JADC2. According to senior Air Force officials, they concluded that, to align with the new defense strategy, ABMS needed to do far more than replace AWACS and JSTARS. They also concluded that no single platform, such as an aircraft, would be the right solution to providing command and control capabilities across multiple domains. In an April 2019 congressional testimony, the Air Force announced a new vision for ABMS as a multidomain command and control family of systems enabling operations in air, land, sea, space, and cyber domains. In that testimony, Air Force leadership explained the need to move away from a platform-centric approach (such as JSTARS) to a network-centric approach, one that connects every sensor to every shooter. The Air Force, however, did not formally document its decision to change the scope of ABMS. In November 2019, according to Air Force officials, ABMS was determined to be the Air Force solution for JADC2 in response to a July 2019 Joint Requirements Oversight Council memo outlining DOD requirements for command and control systems requirements across all domains. In May 2019, we reported that Air Force leadership determined that it would not designate ABMS as a major defense acquisition program because it would be a family of systems. The Air Force also determined that ABMS would be directed by a Chief Architect working across PEOs, rather than a traditional acquisition program manager. According to Air Force officials, the Chief Architect role will be instrumental in integrating the various programs and technologies into an overall system and is the first of its kind within the Air Force. Additionally, Air Force officials stated that they intend to use a flexible acquisition approach to develop ABMS, one that is outside of traditional pathways such as a major defense acquisition program or middle tier acquisition. According to the Chief Architect, this approach will allow ABMS to develop and rapidly field capabilities. Specifically, the Air Force intends to break up technology development into many short-term efforts, generally lasting 4 to 6 months each. The Chief Architect stated that the goal of breaking up development into smaller increments is to increase innovation by requiring multiple contractors—including those that may not usually engage with DOD—to compete for contracts more frequently. These short-term efforts will include prototyping and demonstrations to prove that the capabilities work. Those that are proven will be delivered to the warfighter. By using this approach, the Air Force intends to field capabilities sequentially and more quickly than if all were developed and delivered at one time as is typically done for traditional acquisitions. Additionally, Air Force officials indicated that this approach will not lock the Air Force into long-term development efforts with just one contractor and will allow the Air Force to more easily move on from unsuccessful development efforts. The Air Force has not established a plan or business case for ABMS that identifies its requirements, a plan to attain mature technologies when needed, a cost estimate, and an affordability analysis. As a result of recent ABMS management and scope changes, the Air Force remains early in the planning process and has not yet determined how to meet the capabilities or identify systems that will comprise ABMS. In December 2019, Air Force officials stated an overall plan for ABMS did not exist and would be difficult for the Air Force to develop in the near term due to the unclear scope of ABMS requirements. To date, the Air Force has not identified a development schedule for ABMS, and it has not formally documented requirements. As previously stated, ABMS will be managed as a family of systems and not as a traditional acquisition program typically governed by DOD Instruction 5000.02, nor as a middle tier acquisition. As a result, Air Force officials initially told us that they did not intend to develop most of the typical acquisition documentation, such as a cost estimate, that is generally required of major defense acquisition programs before entering the development phase. In March 2020, after we sent a copy of this report to DOD for comment, the Air Force provided us a draft tailored acquisition plan for ABMS in lieu of an acquisition strategy. Based on our initial review, this document includes some elements of a traditional acquisition strategy, such as contract and test strategies. However, this tailored acquisition plan does not include key information such as the overall planned capabilities and estimated cost and schedule for ABMS. We will continue to monitor the Air Force’s planning efforts as the program progresses. The Air Force also began preparing an analysis of alternatives in January 2019 to assess options for delivering capabilities such as surveilling moving targets and battle management command and control. The Air Force expects to complete the analysis in 2020, but Air Force officials expect it will inform only some aspects of ABMS planning. The Air Force has not defined what additional planning documentation it will develop to help it establish a business case for ABMS. For example, major defense acquisition programs are generally required to develop acquisition planning documents, such as a cost estimate. We have previously reported on the importance of establishing a solid, executable business case before committing resources to a new development effort. A business case demonstrates that (1) the warfighter’s needs are valid and that they can best be met with the chosen concept and (2) the chosen concept can be developed and produced within existing resources. In addition to an acquisition strategy, other basic elements of a sound acquisition business case include firm requirements, a plan for attaining mature technologies, and a reliable cost estimate and affordability analysis, further described below. 1. Firm requirements are the requisite technological, software, engineering, and production capabilities needed by the user. Acquisition leading practices state that requirements should be clearly defined, affordable, and informed. Deciding how best to address requirements involves a process of assessing trade-offs before making decisions. Unstable or ill-defined requirements can lead to cost, schedule, and performance shortfalls. 2. A plan to attain mature technologies when needed is critical in establishing that technologies can work as intended before integration into a weapon system. The principle is not to avoid technical risk but rather address risk early and resolve it ahead of the start of product development. Identifying technologies and defining a plan to ensure mature technologies can be attained when needed help guide development activities and enable organizations to track development and inform decisions on next steps. 3. A reliable cost estimate and affordability analysis are critical to the successful acquisition of weapon systems. GAO’s Cost Estimating and Assessment Guide states that a reliable cost estimate is comprehensive, well-documented, accurate, and credible. Leading practices have shown that realistic cost estimates allow program management to obtain the knowledge needed to make investment decisions and match requirements with resources. A cost estimate is the basis of an affordability analysis, which validates whether a program’s budget is adequate for the planned acquisition strategy. The process of developing and documenting a business case builds knowledge needed to match customer needs with available resources, including technologies, timing, and funding. The fact that the Air Force does not plan to establish such a business case for ABMS increases the risk of cost and schedule overruns and may impact Congress’s ability to exercise its oversight responsibilities. The status of key elements for the ABMS business case follows: Status of requirements. The Air Force has not established well-defined, firm requirements for ABMS, but Congress required that the Air Force start defining requirements for the networked data architecture necessary for ABMS to provide multidomain command and control and battle management capabilities by June 2020. The Air Force has not defined the changes in ABMS’s requirements, such as the need to provide multidomain command and control capabilities in support of joint operations. As a result, the only existing documentation of ABMS’s requirements resides in the ABMS Initial Capabilities Document from 2018, which generally focuses on the capabilities needed to replace AWACS. That document does not address the expanded JADC2 requirements and capabilities ABMS is expected to eventually fulfill. Air Force officials stated that ABMS requirements and the family of systems, or programs, that compose ABMS will be defined over time as they gain more knowledge. Given the lack of specificity regarding ABMS, Congress has kept a close eye on the effort and has implemented several reporting requirements. Since 2018, the Air Force has been required to provide quarterly updates to the defense committees on the status of ABMS development and associated technologies. In addition, the National Defense Authorization Act for Fiscal Year 2020 required the Air Force to provide ABMS-related documentation that describes certain requirements, a development schedule, and the current programs that will support ABMS, among other things, by June 2020. While the Air Force has not established firm requirements for ABMS to date, it has informally identified some broad requirements. For example, the Air Force anticipates that ABMS will provide interoperability between systems, present real-time information to military decision makers, and fully utilize the range of sensor data and capabilities across DOD to create a common battlespace operational picture. In addition, Air Force officials stated that ABMS would be developed as a government-owned open architecture family of systems, which would allow any system to be integrated into ABMS. The Air Force has identified seven different development categories that it plans to simultaneously address to meet its broad ABMS requirements. According to the Air Force, the categories are not intended to be comprehensive and may change as development progresses. These development categories include: Apps Although the Air Force has not defined these seven development categories, it has identified 28 development areas that fit within the categories. For example, one of these development areas, which falls under the “secure processing” category, is called cloudONE. It is intended to store and process data using a cloud infrastructure for multiple levels of classified and unclassified data. These development areas will eventually compose the architecture and technologies that make up ABMS. In January 2020, the Air Force provided us with a draft version of high-level descriptions of the 28 development areas; however, the document did not fully define the requirements or capabilities for the development areas nor identify which organizations would lead each effort. For example, the cloudONE description does not indicate specific technical requirements that must be met, such as amount of storage, the number of users, or data transmission rate. Although ABMS requirements are not fully defined, the Air Force awarded several short-term development contracts for ABMS. According to Air Force officials, these efforts are intended to show that its nontraditional development approach is feasible rather than to develop specific capabilities that will be integrated into ABMS. For example, the Air Force awarded several development contracts totaling approximately $8 million for gatewayONE, one of the 28 development areas that is intended to enable communication between platforms. As part of this effort, the Air Force conducted a demonstration in a joint military exercise in December 2019. While the exercise demonstrated some data transfer capability, it did not directly address the intent of gatewayONE to enable communication between multiple platforms using government-owned systems. According to Air Force officials, ongoing and future efforts will allow the Air Force to better define ABMS requirements and determine what existing and emerging technologies can fulfill those capabilities. The Air Force has not determined what development efforts will follow these early demonstration efforts, in part because it has not fully defined its requirements. Status of plan to attain mature technologies when needed. The Air Force has started development activities without first identifying what technologies are needed for the 28 development areas for ABMS. According to Air Force officials, they do not plan to identify all technologies needed while pursuing development activities. Therefore, the Air Force cannot assess whether technologies required for ABMS are mature or determine the necessary steps to ensure those technologies are mature when needed. Air Force officials stated that as ABMS development progresses, they plan to select commercially available or other mature technologies for integration. However, without first identifying the technologies it needs, the Air Force cannot develop a plan, or technology roadmap, with detailed actions to ensure those technologies will be mature when needed. For example, the Air Force plans for ABMS to assume the capabilities of AWACS and JSTARS aircraft, which are set to retire in the 2030s. However, the Air Force has not defined the technologies ABMS will need or established a roadmap to ensure those technologies are mature before the retirement of legacy aircraft. This increases the risk that the requisite technologies will not be mature before the Air Force must integrate them into ABMS, which increases the likelihood that those capabilities will not be developed when needed. The Chief Architect and other Air Force senior leaders stated that the ABMS development effort is an ambitious undertaking for the Air Force. Our prior work has found that some DOD programs related to ABMS development have posed challenges in the past, in part because technologies were not sufficiently mature when needed, as shown in table 1. Additionally, the Office of Cost Assessment and Program Evaluation assessed previous DOD programs that were similar to ABMS development and noted that the scope of ABMS will be larger than any of those individual programs. Officials from that office concluded that ABMS is a high-risk effort and the Air Force has not provided sufficient programmatic detail. As a result, they could not conclude that the Air Force would be able to overcome the cost, schedule, and performance challenges of these past programs. Air Force officials stated that the Air Force’s approach to ABMS development will avoid these past challenges because only mature technologies will be integrated into ABMS and the Air Force is expected to frequently evaluate development progress. However, since the Air Force has not identified what the technology needs for ABMS are, it cannot yet determine if those technologies are mature or will be mature when needed. We have previously found that starting development without first identifying and assessing the maturity of technologies increases the likelihood that those technologies are not mature when needed, which often results in cost overruns and schedule delays. Status of cost estimate and affordability. The Air Force has not developed a cost estimate for ABMS or an affordability analysis. According to the GAO Cost Estimating and Assessment Guide, even in cases where limited information is available, cost estimates should still be developed to inform budget requests. To date, the Air Force has requested nearly $500 million for ABMS efforts through fiscal year 2021. The Air Force, however, currently has no plans to develop a life-cycle cost estimate, which would provide a comprehensive account of ABMS costs, or an independent cost estimate, which would confirm the credibility of estimated costs. Officials stated that the Air Force has not developed a cost estimate because the capabilities, technologies, and systems that will compose ABMS are still to be determined and will change over time. Officials stated they intend to develop cost estimates for each of the 28 development areas in the future but did not identify a timeline. The GAO Cost Estimating and Assessment Guide acknowledges that cost estimating is more difficult when requirements—and the technologies and capabilities to meet them—are changing and the final product design is not known while the system is being built. In these cases, leading practices call for developing cost estimates that should be updated more frequently to reflect changes in requirements. Without a realistic and current cost estimate for ABMS efforts, the Air Force will be unable to effectively allocate resources and conduct informed long-range investment planning. The Air Force has also not determined if it can afford ABMS. Affordability is the degree to which the funding requirements for an acquisition effort fit within the service’s overall portfolio plan. Whether an acquisition effort is affordable depends a great deal on the quality of its cost estimate and other planned outlays. To conduct an affordability analysis, the budget requirements for the entire portfolio are identified for future years. This can help determine whether the funding needs are relatively stable or if the portfolio will require a funding increase in the future. The GAO Cost Estimating and Assessment Guide states that, as part of the cost estimating process, management should review and approve an affordability analysis to identify any funding shortfalls. Air Force officials stated that the Air Force does not plan to conduct a comprehensive affordability analysis for ABMS because it is managing it as a family of systems. They stated that any costs to the Air Force will be determined in the future by the various organizations that manage the systems that will eventually support ABMS. However, without an affordability analysis, the Air Force will be unable to determine whether it can commit sufficient resources for ABMS in future years. While the Air Force has taken some steps to establish an ABMS management structure, the authorities of Air Force offices to plan and execute ABMS efforts are unclear. Internal controls, which provide standards on effective management of programs, state that management should establish the organizational structure and authority necessary to enable the entity to plan, execute, control, and assess the organization in achieving its objectives. The Air Force, however, has not fully defined or communicated ABMS decision-making authorities to Air Force offices, and documentation to date regarding ABMS management has been limited. Several Air Force offices are involved in ABMS management, as shown in figure 2. Air Force Acquisition. This office is headed by the Assistant Secretary of the Air Force for Acquisition, Technology and Logistics, who is generally responsible for all acquisition functions within the Air Force. In an October 2018 memorandum, Air Force Acquisition established the position of the Chief Architect and stated that any unresolved ABMS issues between the Chief Architect and PEOs are to be brought to Air Force Acquisition for resolution. Chief Architect. The Air Force established this position in October 2018 to execute the overarching vision and strategy for ABMS. According to the Air Force, the Chief Architect will determine the overall design of ABMS, coordinate with the service-level commands and the acquisition programs involved to ensure their efforts are aligned with the overall design and development of ABMS, and identify the enabling technologies that will compose the ABMS family of systems. An October 2018 memorandum stated that individual PEOs and program managers that oversee programs supporting ABMS will retain all authority and responsibility for executing their respective programs. In November 2019, Air Force Acquisition issued additional ABMS management guidance that stated that the Chief Architect would select and fund ABMS development projects for PEOs to execute. However, the guidance did not address whether the Chief Architect has authority to direct the execution of efforts initiated and originally funded by the PEOs, which may support ABMS. Specifically, there is no documentation to clarify whether the Chief Architect would have the authority to realign PEO priorities or funding for ABMS projects. For example, the PEO for Space is currently executing a data integration project, which aligns with the cloudONE development area. Although some ABMS funds have been obligated for this project, there is no documentation to support that the Chief Architect will be able to direct the PEO to change the project objectives or timeline to align with ABMS requirements once they are defined. Air Force Warfighting Integration Capability (AFWIC). In October 2017, the Air Force established AFWIC. According to Air Force officials, AFWIC will ensure forces are operationally ready to perform JADC2 missions using ABMS technologies. According to an AFWIC senior official, in April 2019 AFWIC began leading multidomain command and control efforts for the Air Force. An October 2018 memorandum directed the Chief Architect to coordinate with AFWIC regarding the development of ABMS. Other documentation on ABMS execution indicates that AFWIC will also coordinate with major commands on Air Force doctrine and operations in support of ABMS. However, the documentation did not further define this coordination or indicate whether AFWIC would have any authority in directing ABMS activities. Chief Architect Integration Office. In December 2019, the Air Force established the Chief Architect Integration Office at Wright-Patterson Air Force Base to coordinate and integrate ABMS development efforts across PEOs and other organizations. Air Force officials stated that this office is in the process of being staffed and the roles and responsibilities still need to be formalized. However, as currently envisioned, this office would lead technology development risk reduction efforts by working with the PEOs and other organizations, such as federally funded research and development centers, to conduct ABMS demonstrations and prototypes. Air Force officials told us the Chief Architect Integration Office is expected to resolve issues across Air Force organizations, such as sharing of resources and personnel. An Air Force Life Cycle Management Center- led task force is currently developing an overall strategy for the office, to include resource and organizational requirements. Air Force officials stated that a proposed strategy will be completed in March 2020. Until the Chief Architect Integration Office has been fully established, it is unclear whether the office will have the required authorities to execute the mission of integrating ABMS development efforts across the Air Force. Air Force officials stated that the decision-making authorities across these offices will be developed over time. According to officials, details on these authorities have not been developed or communicated to the offices supporting ABMS and the Air Force has not established a timeline for doing so. The Air Force expects that multiple organizations within the Air Force will be responsible for executing ABMS development efforts. Internal controls, which provides standards for effective management of programs, states that organizational structure and authority is necessary to plan, execute, and assess progress. The absence of fully defined and documented decision-making authorities, which are communicated to all those involved, increases the risk to the Air Force’s ability to successfully plan, execute, and assess ABMS development efforts. The Air Force started ABMS development activities without a business case that defines ABMS requirements, a plan to ensure technologies are mature when needed, a cost estimate, and an affordability analysis. Developing these key elements of a business case helps to build a solid foundation for any successful technology and product development effort, even one using a nontraditional acquisition approach. Congress has already required the Air Force to define and report on certain ABMS requirements, among other aspects of ABMS planning, by June 2020. However, the Air Force does not intend to develop the other elements of a business case, even though it is requesting over $300 million for ABMS development activities in fiscal year 2021. Given the criticality of the battle management command and control mission and the planned retirement of legacy programs, the lack of an ABMS business case introduces uncertainty regarding whether the needed capabilities will be developed within required time frames. For example, without a plan to mature technologies needed to field ABMS capabilities, the Air Force cannot be certain those technologies will be ready when needed. While it may be difficult for the Air Force to formulate a complete ABMS business case at this time, due to the recent changes in ABMS’s scope, the Air Force is not precluded from beginning the process of defining and formalizing a business case. As ABMS continues to evolve, so too can the Air Force’s business case. For example, the Air Force does not yet know the total life cycle costs of ABMS, but it could provide Congress with a cost estimate based on its knowledge today and update the cost estimate over time. This would allow the Air Force to assess whether ABMS is affordable. Furthermore, the Air Force is already required to provide quarterly briefs to congressional defense committees on the status of ABMS, which affords the Air Force the opportunity to present Congress with information on its ABMS business case and explain any changes over time. Specifically, including updates on the scope of the Air Force’s plans to ensure ABMS will have mature technologies when needed, an overall cost estimate, and an affordability assessment would provide important information to Congress. Finally, the Air Force has started to execute ABMS development efforts without clearly defined decision-making authorities that have been communicated to the offices supporting those efforts. The absence of these defined authorities may hinder management’s ability to execute and assess ABMS development across multiple organizations within the Air Force. We are making the following four recommendations to the Secretary of the Air Force to direct the Assistant Secretary of the Air Force for Acquisition, Technology and Logistics: The Assistant Secretary of the Air Force for Acquisition, Technology and Logistics should direct the Chief Architect to develop a plan to attain mature technologies when needed for each ABMS development area, which includes an initial list of technologies and an assessment of their maturity that is updated to reflect changes, and update Congress quarterly. (Recommendation 1) The Assistant Secretary of the Air Force for Acquisition, Technology and Logistics should direct the Chief Architect to prepare a cost estimate that is developed in accordance with cost estimating leading practices, to include regularly updating the estimate to reflect ABMS changes and actual costs, and update Congress quarterly. (Recommendation 2) The Assistant Secretary of the Air Force for Acquisition, Technology and Logistics should direct the Chief Architect to prepare an affordability analysis that should be regularly updated, and update Congress quarterly. (Recommendation 3) The Assistant Secretary of the Air Force for Acquisition, Technology and Logistics should formalize and document acquisition authority and decision-making responsibilities of the Air Force offices involved in the planning and execution of ABMS, to include the Chief Architect. This document should be included as part of the submission to Congress in June 2020 and communicated to the Air Force offices that support ABMS. (Recommendation 4) We provided a draft of this product to the Department of Defense for comment. In its comments, reproduced in appendix I, the Department of Defense concurred with our recommendations. We will continue to monitor the Air Force’s actions to respond to these recommendations. We are sending copies of this report to the appropriate congressional committees. We are also sending a copy to the Secretary of Defense, the Secretary of the Air Force, and other interested parties. In addition, this report is available at no charge on GAO’s website at http://www.gao.gov. Should you or your staff have questions, please contact me at (202) 512- 4841 or MakM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact above, the following staff members made key contributions to this report: Justin Jaynes, Assistant Director; Jessica Karnis, Analyst-in-Charge; and Lauren Wright. Other contributions were made by Brian Bothwell, Rose Brister, Brian Fersch, Miranda Riemer, Megan Setser, Hai Tran, and Robin Wilson.", "summary": "The Air Force's ABMS is a family of systems intended to replace the command and control capabilities of aging legacy programs and develop a network of intelligence, surveillance, and reconnaissance sensors. Air Force officials stated ABMS has received $172 million in funding through fiscal year 2020 for efforts related to ABMS. The Air Force is not designating ABMS as a major defense acquisition program or a middle tier acquisition program. Congress included a provision in statute for GAO to review the status of ABMS. This report examines the extent to which the Air Force has (1) established a plan for ABMS development and (2) defined management and decision-making authorities for ABMS efforts. To conduct this assessment, GAO reviewed ABMS program documentation and interviewed Air Force officials. The Air Force's Advanced Battle Management System (ABMS) is intended to establish a network to connect sensors on aircraft, drones, ships, and other weapon systems to provide a real-time operational picture on threats across all domains, as depicted below. According to Air Force officials, the department will take a nontraditional approach to develop ABMS through short-term efforts that will enable it to rapidly field capabilities. As a result of this approach, ABMS requirements will change over time as development progresses. The Air Force started ABMS development without key elements of a business case, including: firm requirements to inform the technological, software, engineering, and production capabilities needed; a plan to attain mature technologies when needed to track development and ensure that technologies work as intended; a cost estimate to inform budget requests and determine whether development efforts are cost effective; and an affordability analysis to ensure sufficient funding is available. GAO's previous work has shown that weapon systems without a sound business case are at greater risk for schedule delays, cost growth, and integration issues. Congress has kept a close eye on the effort and required quarterly briefings on its status, as well as a list of certain ABMS requirements by June 2020. However, given the lack of specificity that remains regarding the Air Force's ABMS plans, Congress would benefit from future briefings that address the missing business case elements. While the Air Force has taken some steps to establish an ABMS management structure, the authorities of Air Force offices to plan and execute ABMS efforts are not fully defined. Unless addressed, the unclear decision-making authorities will hinder the Air Force's ability to effectively execute and assess ABMS development across multiple organizations. GAO is making four recommendations, including that the Air Force should develop and brief the Congress quarterly on a plan to mature technologies, a cost estimate, and an affordability analysis. In addition, the Air Force should formalize the ABMS management structure and decision-making authorities. The Air Force concurred with the four recommendations. GAO will continue to monitor the Air Force's actions to address these recommendations.", "document_type": "gao"}
{"report": "Education administers federal student aid programs, including the William D. Ford Federal Direct Loan (Direct Loan) program, through the Office of Federal Student Aid. Only Direct Loans are eligible for the PSLF program and the temporary expanded process. Under the Direct Loan program, Education issues and oversees federal loans provided to students, and contractors service these loans. Education currently contracts with nine loan servicers that each handle the billing and other services for a share of the over $1 trillion in outstanding student loans provided through the Direct Loan program. Borrowers interested in pursuing loan forgiveness under PSLF, or the temporary expanded process, must have their loans eventually transferred to Education’s sole PSLF loan servicer in order to proceed. This designated PSLF servicer handles day-to-day activities associated with the PSLF program and the temporary expanded process, which include responding to borrower inquiries, making preliminary determinations about whether borrowers’ employment and loans qualify for loan forgiveness, and processing loan forgiveness applications. The PSLF program and the temporary expanded process provide eligible borrowers with forgiveness on the remaining balance of their Direct Loans after they have met all program requirements. To receive forgiveness for a loan, borrowers are required to be employed in a qualifying public service job for 10 years when making 120 qualifying payments. Borrowers must also be employed in a qualifying public service job at the time they apply for forgiveness, and at the time they receive forgiveness for their loans. Although there are some differences in the eligibility requirements for PSLF and the temporary expanded process, borrowers are generally required to: Work full-time for at least 10 years at a public service organization, a government organization, agency, or entity at any level (federal, state, local, or Tribal); a nonprofit, tax exempt organization (under section 501(c)(3) of the Internal Revenue Code); or another private nonprofit organization that provides certain public services. Not be in default on their loans. Make 120 on-time monthly loan payments for the full amount due on their bill. These monthly payments do not need to be consecutive. Repay their loans through a qualifying repayment plan. The PSLF program generally requires borrowers to repay their loans through one of the eligible income-driven repayment plans or the Standard repayment plan. The temporary expanded process allows borrowers to qualify for loan forgiveness if they are participating in several additional types of repayment plans, including the Graduated repayment plan, Extended repayment plan, Consolidated Standard repayment plan, and Consolidated Graduated repayment plan. In addition, for the temporary expanded process, the following two payments generally must be at least as much as the borrower would have paid under an income-driven repayment plan: (1) the payment made immediately prior to applying for the temporary expanded process, and (2) the payment made 12 months prior to applying for the temporary expanded process. There are key differences in the processes for applying for loan forgiveness under the PSLF program versus the temporary expanded process (see table 1). Despite broad borrower interest in the PSLF program and the temporary expanded process, very few borrowers have been granted loan forgiveness. A large number of borrowers are pursuing the PSLF program, but our 2018 analysis found that Education had denied about 99 percent of borrowers that applied for loan forgiveness through the PSLF program during the first 8 months that Education was accepting applications (September 2017 through April 2018), according to data from the PSLF servicer. According to Education’s most recent publically released PSLF program data through March 2019, PSLF program denial rates have continued to hover around 99 percent since our 2018 review. Of the 76,002 loan forgiveness applications that had been processed, the PSLF servicer had denied 75,138 (99 percent), as of March 2019. According to data as of March 2019, close to one-half of the PSLF program loan forgiveness applications the PSLF servicer had processed were denied because the borrower had not yet made 120 qualifying payments. The other most common reasons PSLF program applications were denied were because of missing information on the application or because the borrower did not have qualifying federal loans. For borrowers that have been approved, Education had forgiven almost $31 million in outstanding student loans, an average of more than $59,000 per approved borrower. Denial rates are also very high for the temporary expanded process. We recently reported that from May 2018 through May 2019, Education had denied 99 percent of the completed requests from about 40,000 borrowers (see fig. 1). The majority of requests borrowers submitted for the temporary expanded process were ineligible for consideration and were therefore denied because the borrower had not previously submitted an application for the PSLF program, according to data from the loan servicer. For the 1 percent of applications that were approved from May 2018 through May 2019, Education had granted almost $27 million in loan forgiveness under the temporary expanded process, totaling about 4 percent of the $700 million appropriated funds, according to our 2019 report. Borrowers received an average of about $41,000 in loan forgiveness. The high denial rates for the PSLF program and temporary expanded process suggest that many borrowers are confused by the requirements. In our 2018 report, we noted that officials from the PSLF servicer said that borrowers were frequently confused by the PSLF program requirements related to qualifying loans, employment, repayment plans, and payments. PSLF servicer officials also said that borrowers were sometimes unaware that they were not on a PSLF-qualifying repayment plan or that forbearance, deferment, and loan consolidation would affect their qualifying payments. For example, the Consumer Financial Protection Bureau has reported that borrowers have complained of spending years making payments, believing they were making progress towards PSLF loan forgiveness, and then learning that they were not eligible. Similarly, in our 2019 report on the temporary expanded process, we noted that officials from Education, the PSLF loan servicer, and representatives from selected organizations representing student borrowers all said that the requirement to submit an application for the PSLF program to be eligible for the temporary expanded process can confuse borrowers. We have previously reported on how shortcomings in the information Education provides to the PSLF servicer has resulted in uncertainty about PSLF program requirements and increased the risk of potential errors in borrower eligibility determinations. To address these issues, we have made three recommendations to Education to provide the servicer with comprehensive guidance and instructions, additional information on qualifying employers, and standardized prior payment information (see table 2). Education agreed with these recommendations and has taken some actions, but has not yet fully implemented them. In our 2018 report, we found that Education does not have a comprehensive document or manual to provide the PSLF servicer with guidance and instructions. This made it difficult to effectively administer the PSLF program and provide consistent service to borrowers, according to PSLF servicer officials. We reported that Education’s guidance and instructions to the PSLF servicer are dispersed in a piecemeal manner across multiple documents, including Education’s original contract with the servicer, multiple updates to the contract, and hundreds of emails. As a result, PSLF servicer officials said that their staff were sometimes unaware of relevant PSLF program guidance and instructions in emails provided by Education, which creates a risk that some policy updates will be overlooked and not consistently implemented. The absence of a central, authoritative source of PSLF guidance and instructions creates a risk of differing interpretations and inconsistent implementation. It also makes it difficult to maintain program continuity in the event of staff turnover or if Education decides to contract with a new servicer to administer the PSLF program. Federal internal control standards state that agencies should communicate information to those who need it, in a form that enables them to carry out their responsibilities. Around the time our 2018 report was issued, Education officials told us they planned to develop a comprehensive PSLF servicing manual, but they did not have a timeline for completing it. In response, we recommended that Education develop a timeline for issuing a comprehensive guidance and instructions document for PSLF servicing. Education agreed with this recommendation and reported in September 2019 that it was continuing its efforts to improve and streamline guidance for the PSLF servicer. While Education said it is working on developing its comprehensive PSLF servicing manual, it does not yet have a timeline for how it will complete this manual and has pushed back the estimated implementation date for this recommendation to 2020. To help ensure that program requirements are applied consistently by the PSLF servicer, we continue to believe that Education should fully implement this recommendation. In 2018, we reported that Education had not provided the PSLF servicer with a definitive source of information for determining which employers qualify a borrower for PSLF loan forgiveness. Instead, Education had identified some data sources the PSLF servicer can use to determine whether borrowers are working for qualifying employers. However, we found that these sources were not comprehensive, and that PSLF servicer officials said they sometimes had to consult other sources that have significant limitations. For example, PSLF servicer officials told us they used an online directory of nursing home facilities to help determine if certain nursing homes were nonprofit employers. However, this website explicitly stated that it did not guarantee that the information it provided was accurate or current. Federal internal control standards state that agencies should communicate the necessary quality information to those who need it, and PSLF servicer officials said that having additional information would help them assess employers more quickly and minimize the risk of inaccurate decisions. Borrowers would also benefit from additional information about qualifying employers, according to PSLF servicer officials, in part because it would help them make better informed employment decisions. Our 2018 report recommended that Education provide additional information to the PSLF servicer and borrowers to enhance their ability to determine which employers qualify for PSLF. Education agreed with this recommendation, and said it planned to incorporate qualifying employer information into an online PSLF Help Tool. As of September 2019, Education reported that it had incorporated a feature into its online PSLF Help Tool to help borrowers determine if their employer fits within general eligibility criteria. However, Education said more specific information to help the PSLF servicer make employer eligibility determinations and an employer database will not be available until 2020. We believe that if Education fully implements this recommendation to provide the servicer with more definitive employer information, it would help reduce the risk of errors in assessing employer eligibility for PSLF. In our 2018 report, we found that Education does not ensure that the agency’s other loan servicers provide the PSLF servicer consistent information on borrowers’ prior loan payments, which could increase the risk of qualifying payments being miscounted for the PSLF program. PSLF servicer officials said inconsistencies in the information provided by other loan servicers make it challenging to determine whether borrowers are on qualifying repayment plans or making qualifying payments. Officials with Education and the PSLF servicer said that these inconsistencies increase the risk of miscounting qualifying payments. This is contrary to federal internal control standards, which state that agencies should use quality information. Our 2018 report recommended that Education standardize the payment information that the PSLF servicer receives from other loan servicers to ensure the PSLF servicer obtains more consistent and accurate payment information. Education agreed with this recommendation and stated that efforts were underway to improve the consistency of payment information exchanged between servicers. As of September 2019, Education reported that it is planning to standardize this loan payment data by spring 2020. If Education implements this recommendation, we believe it would reduce the potential risk of qualifying PSLF payment count errors moving forward. We have previously reported on how unclear processes and a lack of information about the PSLF program and the temporary expanded process could contribute to borrower confusion and high denial rates. We have also reported that borrowers can face challenges detecting any errors in payment counts for the PSLF program and with contesting eligibility determinations for the temporary expanded process. To address these issues, we have made five recommendations to Education to improve service to borrowers by expanding outreach, streamlining processes, and providing information to help borrowers catch and resolve errors (see table 3). Education agreed with these recommendations, but has not yet taken sufficient actions to fully implement them. Education uses several methods to inform borrowers about the PSLF program and temporary expanded process requirements, including through its website and webinars. Congress also appropriated $4.6 million in 2018 for Education to conduct outreach to borrowers about PSLF, including the temporary expanded process. However, our recent work has found several areas in which the agency’s outreach activities related to the temporary expanded process are limited. While Education and PSLF loan servicer officials told us that they primarily direct borrowers to online sources to inform them about requirements for the temporary expanded process, we found that the agency does not include information about the temporary expanded process in key online sources. For example, according to agency officials, one of Education’s primary PSLF outreach mechanisms—the online PSLF Help Tool, which the agency launched in December 2018—does not include any information about the temporary expanded process. Officials from Education and the PSLF servicer stated that integrating information about the temporary expanded process into the online PSLF Help Tool would be beneficial for borrowers and would reduce confusion about the temporary expanded process. In addition, our 2019 report found that while all nine of Education’s loan servicers’ websites contain some information on the PSLF program, none of them (other than the PSLF loan servicer) included information about the temporary expanded process on their websites or provided a link to Education’s website specific to the temporary expanded process. Education officials told us that only the PSLF servicer is required to have information about the temporary expanded process on its website; however, other loan servicers may also serve borrowers who are potentially eligible but may be unaware of the temporary expanded process. This limited outreach to borrowers about the temporary expanded process reduces the likelihood that borrowers are able to take advantage of this opportunity. Further, federal internal control standards state that management should externally communicate the necessary quality information to achieve the entity’s objectives. To improve Education’s borrower outreach about the temporary expanded process, our September 2019 report recommended that Education include information about the temporary expanded process in its online PSLF Help Tool and that Education require all loan servicers to provide information about the temporary expanded process on their websites. Education agreed with both of these recommendations, and stated that it would take steps to address them. If Education implements these two recommendations, we believe it would help the department provide better service to borrowers by raising awareness of the temporary expanded process and requirements. In September 2019, we reported that Education’s process for requesting loan forgiveness through the temporary expanded process is not clear to borrowers and may contribute to high denial rates. In particular, the requirement that borrowers must have already submitted a separate PSLF application in order to be eligible for loan forgiveness through the temporary expanded process can confuse borrowers. Borrowers currently must submit a separate PSLF application, even if they know it will be denied, before Education will consider their request for forgiveness through the temporary expanded process. Education officials acknowledged that the majority of requests for the temporary expanded process come from borrowers who have not first submitted a PSLF application. Similarly, our September 2019 report found that 71 percent of the denied requests were denied because the borrower had not submitted a PSLF application. Officials from the PSLF loan servicer said that borrowers who called were frequently confused when they received a denial for the temporary expanded process based on the fact that they had not first submitted an application for the PSLF program. This lack of a borrower-friendly process complicates the path towards loan forgiveness and does not align with Education’s strategic plan objective to improve the quality of service to customers. To address this issue, our 2019 report recommended that Education streamline the process for borrowers to request loan forgiveness through the temporary expanded process by integrating the request for temporary expanded process consideration into the PSLF application, eliminating the need for borrowers to submit a separate PSLF application prior to consideration. Education agreed with this recommendation and stated that it will integrate requests for the temporary expanded process into the PSLF application as part of its ongoing initiative to overhaul its online portal for student loan borrowers. Implementation of this recommendation would improve service to borrowers by making the process easier and less confusing. In 2018, we reported that although Education and PSLF servicer officials acknowledged the risk of miscounting qualifying payments, the PSLF servicer did not provide borrowers with sufficient information to easily identify PSLF program errors. Officials with the PSLF servicer said they rely on borrowers to catch any payment counting errors resulting from issues with information provided by other loan servicers. As we reported, the PSLF servicer provided borrowers with aggregate counts of qualifying payments, which are useful for helping borrowers track their progress, but did not provide borrowers with enough detail to check the servicer’s counts and identify prior payments that the servicer may have missed. This is also contrary to federal internal control standards which call for communicating necessary information to external parties. Our 2018 report recommended that Education ensure that borrowers receive sufficiently detailed payment information from the PSLF servicer to be able to identify any errors in the servicer’s counts of qualifying payments. Education agreed with this recommendation and stated that efforts were underway to standardize the payment count information that is provided to borrowers. As of September 2019, Education reported that it is reviewing communications from the PSLF servicer to ensure that borrowers receive sufficiently detailed information regarding payment counts and payment history and that this review will be completed by September of 2020. To help borrowers detect potential payment counting errors that could ultimately affect their eligibility for the PSLF program, we believe Education should implement this recommendation and provide borrowers with more detailed qualifying payment information. Further, our 2019 report on the temporary expanded process found that Education does not provide complete information to borrowers about options they have to contest payment counts or other aspects of the eligibility determination process. An Education official told us that while there is no formal process for borrowers who are dissatisfied with their temporary expanded process determinations to contest the determination, borrowers do have additional options for addressing concerns, such as an additional review by the PSLF servicer, or a complaint to Education’s Federal Student Aid Feedback System or Ombudsman. Education officials told us that the agency does not provide information about these options in its denial letters or on its website for the temporary expanded process, noting that borrowers could find this information at the bottom of Education’s Federal Student Aid main website. However, borrowers may not know where to find this information should they choose to contest their temporary expanded process determination, because this information is not effectively communicated to them in accordance with federal internal control standards. To address this, our 2019 report recommended that Education provide borrowers with more information on the website for the temporary expanded process and in the servicer’s denial letters about options available to borrowers should they wish to contest the servicer’s decision. Education agreed with this recommendation and stated that it would add information about the options borrowers have to contest temporary expanded process decisions to relevant websites and denial letters. Implementing this recommendation will increase the likelihood that borrowers with valid concerns about the temporary expanded process will have them appropriately resolved. In conclusion, my statement has highlighted several actions Education could take to strengthen the PSLF program and the temporary expanded process to deliver on the promise the federal government has made to borrowers pursuing careers in public service. Education is responsible for establishing an administrative structure for the loan servicer, but more than 10 years after the PSLF program was first established, Education has not provided the loan servicer with a comprehensive source of guidance and instructions on how to operate the PSLF program, and could provide additional information to help ensure that eligibility determinations are being made correctly. Education is responsible for ensuring that borrowers are aware of and understand programmatic requirements. However, the high denial rates for the PSLF program and its temporary expanded process suggest that borrowers are still confused. It is also important for Education to maintain borrower confidence, but the department has not provided critical information to borrowers to help them remedy potential errors. Large numbers of borrowers have pursued careers in public service, sometimes at lower pay than in the private sector, with the hope of one day achieving loan forgiveness through the PSLF program. They have often had to navigate the PSLF program requirements with a lack of sufficient information from Education only to be denied 10 years later when they applied for loan forgiveness because their prior years of employment or loan payments did not qualify. In addition, some borrowers who were denied may not be aware that they might be eligible for loan forgiveness through the temporary expanded process, potentially missing out on this temporary opportunity. Education needs to take action to better serve these borrowers and help smooth their long road towards loan forgiveness. Education has not yet taken action to fully implement the eight recommendations discussed in this testimony. We continue to believe that implementing these eight recommendations would strengthen program administration, improve service to borrowers, and help to fulfill the original goal of encouraging individuals to enter and continue in public service employment. We will continue to monitor Education’s efforts in these areas. Chairwoman Davis, Senior Republican Smucker, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Melissa Emrey-Arras, Director of Education, Workforce, and Income Security, at (617) 788-0534 or emreyarrasm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony include William Colvin (Assistant Director), Nora Boretti (Analyst-in-Charge), Linda Collins, and Aaron Karty. Additional support was provided by James Bennett, Deborah Bland, Alex Galuten, Lara Laufer, Sheila R. McCoy, and Jessica Orr. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The PSLF program was established in 2007 and forgives borrowers' remaining federal student loan balances after they have made at least 10 years of qualifying loan payments while working in public service. Starting in September 2017, the first borrowers potentially became eligible for the PSLF program and began applying to have their loans forgiven. In 2018, Congress appropriated $700 million to temporarily expand the PSLF program for certain borrowers who initially did not qualify for the program. This statement—based on GAO's reports issued in September 2018 ( GAO-18-547 ) and September 2019 (GAO-19-595 )—discusses (1) the extent to which borrowers' requests for loan forgiveness through PSLF and the temporary expanded process have been approved or denied, (2) the extent to which Education provides the PSLF servicer with sufficient information to administer the program, and (3) opportunities for improving service to borrowers. A large number of borrowers are pursuing the Public Service Loan Forgiveness (PSLF) program, but the Department of Education (Education) has denied about 99 percent of loan forgiveness applications as of March 2019. Close to one-half of these applications were denied because the borrowers had not yet made the required 120 qualifying monthly loan payments. As of May 2019, Education has also denied 99 percent of loan forgiveness requests made through the temporary expanded process, which is intended for borrowers who did not initially qualify for the PSLF program. In its 2018 report, GAO found that shortcomings in the information Education provided to the loan servicer that administers the PSLF program increased the risk of administrative errors. For example, Education had not provided the PSLF servicer with a definitive source of information for determining which employers qualify. GAO made three recommendations to Education to address these issues (see table below). Education agreed with these recommendations and has taken some actions, but has not yet fully implemented them. In its 2018 and 2019 reports, GAO found that Education can provide better service to borrowers by expanding outreach, streamlining processes, and sharing critical information with borrowers. For example, GAO found that Education does not include information for borrowers about the temporary expanded process in key online sources. GAO made five recommendations to Education to address these issues with the PSLF program and the temporary expanded process (see table below). Education agreed with these recommendations, but has not yet fully implemented them. GAO has made eight recommendations to Education to improve its implementation of the PSLF program and its temporary expanded process. Education agreed with GAO's recommendations. As of September 2019, GAO continues to believe that actions are necessary to fully implement all of the recommendations discussed in this statement.", "document_type": "gao"}
{"report": "At VA, and indeed at all federal agencies, strategic human capital management plays a critical role in maximizing the government’s performance and assuring its accountability to Congress and the nation as a whole. As we have long reported, there is a direct link between the effectiveness of an agency’s personnel management efforts and its ability to carry out its mission. Addressing challenges in areas such as disaster response, homeland security, economic stability, and numerous other complex and evolving issues requires a skilled federal workforce able to work seamlessly with other agencies, levels of government, and nongovernmental entities. In our March 2019 report, we identified key trends in agency operations and attitudes toward work that are affecting how federal work is done and, consequently the skills and competencies that workers will need to accomplish agency missions. Agencies will need to apply appropriate talent management strategies that are adapted to these trends to recruit, develop, and retain a high-performing workforce and better meet their missions (see fig. 1). Over the past two decades, we and others have expressed concern about certain human capital practices at VA and its components. For example, in November 2018, VA’s Office of Inspector General identified leadership and workforce investment as a major management challenge. The Inspector General noted that the root cause for many of the issues it identified at VA was poor and unstable leadership as well as staffing shortages. Similarly, in May 2019, we reported that leadership turnover impeded VA’s ability to address a number of management challenges we identified such as managing acquisitions, managing risk, and improving veterans’ health care. At VHA, we found that serious human capital shortfalls are undermining its ability to meet the health care needs of veterans. Key examples from our prior work include the following: In March 2019, we reported that VHA’s 172 medical centers have large staffing shortages, including physicians, registered nurses, physician assistants, psychologists, and physical therapists, as well as HR specialists and assistants. As of December 2018, VA reported an overall vacancy rate of 11 percent at VHA medical facilities, including vacancies of over 24,000 medical and dental positions and around 900 HR positions. In July 2016, we found that losses in VHA’s five clinical occupations with the largest staffing shortages, including physicians, registered nurses, and psychologists, increased from about 5,900 employees in fiscal year 2011 to about 7,700 in fiscal year 2015. Voluntary resignations and retirements were the primary drivers. VHA’s exit survey indicated that advancement issues or dissatisfaction with certain aspects of the work, such as concerns about management and obstacles to getting the work done, were commonly cited as the primary reasons people left. In December 2016, we found that several problems combined to impede VHA’s ability to improve delivery of health care services to veterans. These problems include high attrition (often involving transfers to other federal agencies), increased workload, and burnout among VHA’s HR staff. Another issue is a lack of effective internal control practices to support HR operations such as information systems that meet operational needs (see fig. 2). In our preliminary findings in a forthcoming report on the extent to which succession planning policies and procedures at VA and its components are consistent with key leading practices, we have identified several concerns. For example, according to VA officials, the agency has not produced a department-wide succession plan since 2009 due to leadership turnover. Department-wide, around 30 percent of VA employees on board as of September 30, 2017, will become eligible to retire in the next 5 years. Effective succession planning can help VA ensure it has a pipeline of talent to meet current and future mission requirements. In our prior work, we noted that effective succession planning is more than filling existing vacancies with people that have the same occupational skills and competencies. Rather, succession planning focuses on current and future needs and develops pools of high-potential staff to meet the organization’s mission over the long term. We have designated 40 of our prior recommendations to VA as priority recommendations because, upon implementation, they may have an especially significant impact on VA’s operations. Twelve of these priority recommendations are aimed at strengthening VA’s human capital management efforts and will help address VA’s challenges in such areas as recruiting and retaining doctors and nurses, performance management, and employee misconduct. To date, VA has implemented six of these priority recommendations, but needs to take additional action on the other six. While VA agreed or partially agreed with and is taking steps to implement five of these remaining priority recommendations, it disagreed with one related to developing a process to accurately count all physicians at each VA medical center because it does not believe this affects its ability to assess workload. Nevertheless, we continue to believe that VHA needs a systematic process to identify all physicians working at VA medical centers as part of the agency’s efforts to monitor and assess workload. The six unimplemented priority recommendations are for VA to 1. develop a process to accurately count all physicians providing care at each VA medical center (recommended in 2017), 2. develop a modern and effective performance management system in which VA managers make meaningful distinctions in employees’ performance ratings (recommended in 2016), 3. ensure that ratings-based performance awards are administered in a manner that is consistent with leading practices (recommended in 2016), 4. develop a plan to implement a modern information technology system to support employee performance management processes (recommended in 2016), 5. collect complete and reliable misconduct and associated disciplinary action data (recommended in 2018), and 6. ensure that employees who report wrongdoing are treated fairly and protected against retaliation (recommended in 2018). We will continue to monitor VA’s progress in implementing these and our other open recommendations. Beyond these specific recommendations, VA and other agencies can use talent management strategies to better compete for critical positions in a tight labor market and to help meet agency missions. In our prior work we noted that while these strategies are not an exhaustive list, collectively they suggest basic steps that agencies can take within existing authorities to address the demographic and technological trends affecting work that are discussed earlier in this statement. These strategies include: Align human capital strategy with current and future mission requirements. With shifting attitudes toward work, technological advances, and increased reliance on nonfederal partners, agencies need to identify the knowledge and skills necessary to respond to current and future demands. Key practices include identifying and assessing existing skills, competencies, and skills gaps. Acquire and assign talent. To ensure agencies have the talent capacity to address evolving mission requirements and negative perceptions of federal work (e.g., that it is too bureaucratic), agencies can cultivate a diverse talent pipeline, highlight their respective missions, recruit early in the school year, support rotations, and assign talent where needed. Incentivize and compensate employees. While federal agencies may struggle to offer competitive pay in certain labor markets, they can leverage existing incentives that appeal to workers’ desire for schedules and locations that provide work-life balance. Engage employees. Engaged employees are more productive and less likely to leave. Agencies can better ensure their workforces are engaged by managing employee performance, involving employees in decisions, and developing employees. A number of the staffing challenges facing VA are actually part of a broader set of human capital issues affecting government as a whole. As we noted in our March 2019 update of government high-risk areas, the federal government faces long-standing challenges in strategically managing its workforce. We first added strategic human capital management to our list of high-risk government programs and operations in 2001. Although Congress, OPM, and individual agencies have made improvements since then, strategic human capital management remains a high-risk area because mission-critical skills gaps within the federal workforce pose a high risk to the nation. Of the 34 other high-risk areas on our 2019 High-Risk List, skills gaps played a significant role in 16 of the areas, including information technology management and acquisitions, strengthening management functions at the Department of Homeland Security, and, as noted above, veterans’ health care at VA. While causes for these skills gaps related to high-risk areas vary, they often occur because of a shortfall in talent management activities such as robust workforce planning or training. Additionally, the changing nature of federal work and the high percentage of employees eligible for retirement have the potential to produce gaps in leadership and institutional knowledge and could threaten to aggravate the problems created from existing skills gaps. For example, 31.6 percent of permanent federal employees who were on board as of September 30, 2017 will be eligible to retire in the next 5 years, with some agencies having particularly high levels of employees eligible to retire. High-performing organizations have found that the full life cycle of human capital management activities needs to be fully aligned and focused on the cost-effective achievement of an organization’s mission. These activities include workforce planning, recruitment, on-boarding, compensation, engagement, succession planning, and retirement programs. Further, adding to agencies’ staffing challenges is the fact that much has changed since the Civil Service Reform Act of 1978 and the Classification Act of 1949 laid the foundation of much of today’s federal personnel system. Agencies’ missions have evolved and employees’ expectations of work and the workplace are changing. As a result, the extent to which the current and future workforce finds the government’s employment policies and practices relevant is an open question. We and others have identified several structural challenges within the federal human capital system that impede the ability of agencies to recruit, retain, and develop workers, both today and in the future. For example: Classification system. The General Schedule classification system—which defines and organizes federal positions primarily to assign rates of pay—has not kept pace with the government’s evolving requirements. Recruiting and hiring. Federal agencies need a hiring process that is applicant friendly and flexible while also meeting policy requirements. Pay system. Employees are compensated through an outmoded system that (1) rewards length of service rather than individual performance and contributions, and (2) automatically provides across- the-board annual pay increases, even for poor performance. Performance management. Federal agencies have faced long- standing challenges developing modern, credible, and effective employee performance management systems and dealing with poor performers. Going forward, to help agencies effectively carry out their missions, OPM and federal agencies must take some important steps to address ongoing human capital problems. These actions include continuing to develop the capacity to measure and address existing mission-critical skills gaps and using workforce analytics to predict and mitigate future gaps. Chairman Takano, Ranking Member Roe, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff have any questions about this testimony, please contact Robert Goldenkoff, Director, Strategic Issues, at (202) 512-2757 or GoldenkoffR@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Shirley Hwang (Assistant Director), Alexander Ray (Analyst-In-Charge), Sarah Green, Allison Gunn, and Shelby Kain. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "VA operates one of the largest health care delivery systems in the nation and provides billions of dollars in benefits and services to veterans and their families. However, VA faces serious and long-standing problems with management challenges and veterans' access to health care and disability benefits. These issues contributed to GAO's decision to list several areas involving VA on GAO's High-Risk List, including managing acquisitions, managing risk and improving veterans' health care, and improving and modernizing VA's disability programs. This testimony discusses (1) human capital challenges facing VA and its components, (2) GAO recommendations addressing some of those challenges, and (3) how those challenges are related to a broader set of government-wide human capital problems. This testimony is based on GAO's work on VA issued since 2017, as well as GAO's work on government-wide strategic human capital management issued since July 2014. To conduct these studies, GAO reviewed key agency documents and government-wide employment data and interviewed knowledgeable agency officials and managers, as well as subject matter specialists. Serious human capital shortfalls are undermining the Department of Veterans Affairs' (VA) ability to provide veterans with quality and timely services. Over the past two decades, GAO has identified major challenges with VA human capital practices. For example, in March 2019, GAO found large staffing shortages, including physicians and registered nurses, at the Veterans Health Administration's (VHA) 172 medical centers. In December 2016, GAO found that high attrition, increased workload, and burnout among VHA's human resources (HR) staff, along with ineffective internal controls to support its HR operations, have impeded VHA's ability to serve the nation's veterans (see figure). Continued leadership attention to addressing GAO's recommendations could help VA better execute its mission. GAO has made numerous recommendations to VA, 40 of which were designated as priorities because they could significantly improve VA's operations. Twelve of the 40 were aimed at strengthening VA's human capital management efforts. Of these, six have been addressed. However, VA still needs to take additional actions on the other six, such as developing a modern and effective performance management system. Beyond these priority recommendations, VA can use key talent management strategies that GAO has identified for acquiring, incentivizing, and engaging employees and thus be more competitive for a high-performing workforce in a tight labor market. Some of the challenges facing VA are part of a larger set of human capital issues affecting government as a whole. Although Congress, the Office of Personnel Management, and individual agencies have made improvements in recent years, human capital management in general remains a high-risk area because of mission-critical skills gaps within the federal workforce. Structural issues impede the ability of agencies to recruit, retain, and develop workers, including outmoded position classification and pay systems, ineffective recruiting and hiring processes, and challenges in dealing with poor performers. GAO has designated 40 of its prior recommendations to VA as priorities for implementation. Twelve of these priority recommendations are aimed at strengthening VA's human capital management efforts. To date, VA has implemented six of these priority recommendations, but needs to take additional action on the other six. GAO will continue to monitor VA's progress in implementing these recommendations.", "document_type": "gao"}
{"report": "DOD started the F-35 program in 2001 to develop a fifth-generation fighter aircraft intended to replace a range of aging aircraft in the U.S. military services’ inventories and to provide enhanced capabilities to warfighters that capitalized on technological innovations. Among other capabilities, the program designed the F-35 aircraft to be difficult to observe using radar and include sensors that can provide insights into potential targets and other warfighting information. The program is producing and delivering three variants of the F-35 aircraft: the F-35A conventional takeoff and landing variant for the Air Force, the F-35B short takeoff and vertical landing variant for the Marine Corps, and the F-35C carrier-suitable variant for the Marine Corps and the Navy. The characteristics of the services’ variants are similar, but each variant also has unique operating requirements. For example, the Marine Corps requires that the F-35B be capable of operating from aircraft carriers, amphibious ships, and main and austere operating bases alike, requiring the capability to conduct short takeoffs and vertical landings. Figure 1 shows an F-35B exercising this capability. While DOD plans to purchase 2,470 aircraft for the U.S. services, the F- 35 program is acquiring more than just aircraft. The complete F-35 air system has eight elements, including training and maintenance systems. Figure 2 shows the eight elements that make up the entire F-35 Air System and how they each support the aircraft. For example, the program intends for the Automated Logistics Information System (ALIS) to provide the necessary logistics tools to F-35 program participants as they operate and sustain the F-35 aircraft. To do this, ALIS consists of multiple software applications designed to support different squadron activities, such as supply chain management, maintenance, training management, and mission planning. For the F-35 aircraft to have full capability, each element of the air system has to be developed and fielded in sync with the aircraft. However, we found in March 2020 that problems with ALIS still pose significant challenges to day-to-day F-35 operations. According to DOD, it plans to replace ALIS with a new system named the Operational Data Integrated Network (ODIN). Furthermore, DOD reports that it is currently developing a strategy for ODIN, which will include key tasks, milestones and schedule, risks and opportunities, governance structure, and cost estimates. We concluded that, as DOD proceeds with replacing ALIS with ODIN, it will be important for the department to carefully consider and assess the key technical and programmatic uncertainties that we reported in March 2020. These include how much of ALIS will be incorporated in ODIN and the extent to which DOD has access to the data it needs to play a more active role in the management of the system. These issues are complex, and will require significant direction and leadership to resolve. Further, we reported in March 2020 that the F-35 program office was not able to provide us with historic costs showing how much the department had spent on ALIS over the years. Also, because DOD had not answered key questions about the future of the system, such as the extent to which the re-design will incorporate current ALIS software, DOD has not been able to develop accurate cost estimates for the ALIS re-design. We recommended that DOD develop and implement a strategy for the re- design of ALIS. The strategy should be detailed enough to clearly identify and assess the goals, key risks or uncertainties, and costs of re- designing the system. DOD concurred with the recommendation. DOD began development of the F-35 aircraft in 2001 without adequate knowledge of its critical technologies or a solid design, as we reported in March 2005. DOD’s acquisition strategy also called for high levels of concurrency between development and production—building aircraft while continuing to refine the designs of key components—which runs counter to GAO’s leading practices for major defense acquisition programs. In our prior work, we identified the F-35 program’s lack of adequate knowledge and high levels of concurrency as the major drivers of the program’s eventual significant cost and schedule growth, among other performance shortfalls. Since 2001, the program has been rebaselined with new cost and schedule estimates three times. DOD initiated the most recent restructuring in 2010 when the program’s cost estimates for each aircraft exceeded critical thresholds established by statute—a condition known as a Nunn-McCurdy breach. DOD then established a new acquisition program baseline that increased the program’s cost estimates by $162.7 billion and extended delivery schedules 5-6 years into the future. This last revision is the current program baseline, reflecting the cost and schedule estimates to deliver the aircraft and systems and to meet the original program requirements. From 2018 to 2019, the total cost estimate of the F-35 acquisition program increased by $22 billion, from $406 billion to over $428 billion. This increase was partially due to the addition of the estimated Block 4 modernization costs. Block 4 includes efforts to enhance and add capabilities—beyond the F-35 baseline program—through hardware and software upgrades. In April 2019, the F-35 program estimated that Block 4 development and procurement costs would add $13.9 billion to the program’s total baseline cost. Beyond this Block 4 increase, the F-35 program baseline costs also increased by $8 billion over the program’s 2018 estimate. Table 1 outlines the program’s baseline costs, the Block 4 modernization costs, and the sum total of the baseline and Block 4 cost estimates since 2001. In addition to the acquisition costs above, the program estimates that the sustainment costs to operate and maintain the F-35 fleet for its planned 66-year life cycle are $1.2 trillion, bringing the total cost of the F-35 program to over $1.6 trillion. The F-35 program office, in coordination with the Director of Operational Test and Evaluation (DOT&E), received approval to conduct some preliminary operational testing in January 2018. This included weapons, cybersecurity, and cold weather testing, among other things. The program’s formal operational testing (conducted by DOT&E) started in December 2018 and was ongoing in 2019. The purpose of operational testing is to assess the effectiveness, suitability, survivability, lethality, and mission capability of the F-35, including the information systems and the air vehicle, in an operationally representative environment. Operational testing includes cybersecurity assessments, some of which the program has conducted. The program plans for the remaining testing to take place through at least September 2020, while the program continues to produce and deliver aircraft. Through 2019, F-35 program test officials had identified over 3,200 deficiencies. Deficiencies represent specific instances where the weapon system either does not meet requirements or where the safety, suitability, or effectiveness of the weapon system could be affected. The test officials categorize deficiencies according to their potential impact on the aircraft’s performance. Category 1 deficiencies are critical and could jeopardize safety, security, or another requirement. Category 2 deficiencies are those that could impede or constrain successful mission accomplishment. In June 2018, we recommended that the program resolve all critical deficiencies before making a full-rate production decision, in part, to reduce the potential for additional concurrency costs stemming from continuing to produce aircraft while testing was ongoing. DOD concurred with our recommendation and stated that it would resolve critical deficiencies before full-rate production, currently planned to occur between September 2020 and March 2021. Production of the aircraft began one year after testing started in 2007, while development was in its early stages. Due to the concurrency of testing and production, according to an F-35 program official, as many as 550 aircraft delivered through 2020 will need retrofits to fix deficiencies and design issues found during testing. The program refers to the cost of these fixes as its concurrency cost, which the program estimates at $1.4 billion; this estimate did not change with the program’s last update in 2019. Until operational testing is complete, there is a risk that the program may identify additional deficiencies. As a result, as we have previously reported, the concurrency costs of retrofitting delivered aircraft could increase. In our June 2018 report, we found that the program was not on track to meet its reliability and maintainability (R&M) performance targets. R&M targets indicate how much time the aircraft will be in maintenance rather than operations. We concluded that the program was missing a prime opportunity to infuse affordability into the aircraft’s future with better R&M performance. As a result, we recommended that the F-35 program office identify what steps it needed to take to ensure the F-35 meets R&M requirements and update the R&M Improvement Program with these steps. DOD concurred with the recommendations, noting that the F-35 program office would update the R&M Improvement Program with the steps needed to ensure continued progress towards its goals. In April 2019, we found that F-35 R&M performance had shown some small improvements but that the program could take more actions to meet the R&M targets. We made additional recommendations to the Secretary of Defense, with which DOD concurred and has taken some actions to implement. Currently, the Office of the Under Secretary of Defense for Acquisition and Sustainment (OUSD (A&S)) is the acquisition decision authority for the F-35 program, and would direct the F-35 program office to take any further actions. In 2019, the program’s R&M performance generally remained unchanged. However, measurable improvements in R&M can take time to manifest. For example, fielded aircraft must be modified and flown for many hours before the program can measure improvements. For details about the R&M performance, see appendix IV. As we have previously reported, even though operational testing of the baseline program remains ongoing, the F-35 program office has turned its attention to Block 4 modernization activities using a different development approach. DOD refers to this approach as Continuous Capability Development and Delivery (C2D2). This method is loosely based on the Agile software development process. With this approach, the program plans to deliver capabilities to the warfighter faster than it did during the baseline development program. For example, rather than take years to develop and deliver all the required capabilities to the warfighter, the program intends to incrementally develop, test, and deliver small groups of capabilities every 6 months. In January 2018, to transition from the baseline development program to its Block 4 activities, the F-35 program started using the C2D2 approach to develop and test software updates to address deficiencies identified during testing. The planned $13.9 billion Block 4 effort exceeds the statutory and regulatory thresholds for what constitutes a major defense acquisition program, and Block 4 is more expensive than many of the other major weapon acquisitions already in DOD’s portfolio. To provide better oversight into Block 4 activities, in 2016, we recommended that the Secretary of Defense hold a milestone B review—a critical point in an acquisition program leading to the engineering and manufacturing development phase—and manage it as a separate major defense acquisition program. DOD did not concur with our recommendation, and it continues to manage Block 4 within the larger F-35 program. We maintain that DOD should manage the Block 4 activities as a separate program. In 2019, the F-35 program conducted a majority of its planned operational testing but added 9 months to the schedule to complete the remaining tests. Specifically, as of February 2020, according to test officials, the program completed 156 flight tests. The program must still conduct four open-air flight tests, the remaining cybersecurity tests of the air vehicle and mission systems, and 64 simulated flight tests. The 9-month delay needed to complete testing, however, also provides additional time for the program to address our June 2018 recommendation that it resolve critical deficiencies before making its full-rate production decision, currently planned to occur between September 2020 and March 2021. Figure 3 shows the test schedule as of 2019, the delay to the schedule into 2020, and the remaining tests events planned. The completion of operational testing hinges on three main tasks: (1) the final four open-air flight tests; (2) cybersecurity testing; and (3) the final development, integration, verification and validation of its simulator and 64 simulated flight tests. First, the program expects to complete the four remaining open-air tests between March and April 2020. To conduct these tests, the program must finish moving the Radar Signal Emulators—test assets that simulate long- range threat radars—from the Nevada Test and Training Range to the Point Mugu Sea Range in California. According to test officials, there is some risk with this move, such as damage to the sensitive test equipment. The test facilities will have to integrate the equipment into the testing infrastructure at Point Mugu. Second, while the program has conducted cybersecurity testing on several aspects of the F-35 aircraft and support systems, three air vehicle subsystems tests and two enterprise-level ALIS tests remain. The program expects to complete these by August 2020.The tests completed to date have identified multiple cybersecurity vulnerabilities. The F-35 program office has taken steps to address some identified vulnerabilities and is working to address the remainder. Test officials stated that some of the delays to cybersecurity testing of the aircraft are due to safety concerns and the risk of losing the use of a test aircraft before testing is complete. According to DOD policy, cybersecurity testing should be conducted as early in the operational test cycle as possible. Leaving this critical testing to the end of operational testing adds risk to the program because the program will not know the extent to which the aircraft may have cybersecurity vulnerabilities until near the expected decision to proceed to full-rate production. If the program cannot finish these tests by September 2020, officials stated that DOT&E could require that the cybersecurity testing be completed in follow-on testing and not hold up the full-rate production decision. Any additional cybersecurity vulnerabilities may require more time to develop and implement plans to address vulnerabilities in aircraft that have already been produced and those slated for production. Lastly, the program has not been able to complete the F-35 Joint Simulation Environment, which we refer to as the aircraft simulator, on time. The simulator runs the F-35’s mission systems software along with other software models (such as other weapons and modern threat systems) to provide complex test scenarios that the program cannot replicate in a real-world environment. We reported in April 2019 that the simulator’s development was behind schedule and was a risk to operational testing. Since then, the program has struggled to develop the complex software and functionality needed to complete the simulator. The difficulties stem, in part, from the program office’s original plan to have the contractor, Lockheed Martin, develop the simulator. However, in August 2017, program office officials decided that the contractor’s proposal was considered to be too expensive. To mitigate concerns over the cost of the proposal, the program decided to have the Navy complete the work. The program originally expected the Navy’s simulator to be ready for testing in 2017, but it is now 3 years behind schedule. According to program office officials, the simulator’s development effort has taken longer than expected to integrate F-35 aircraft and sensor data, in part because the contractor claimed the data as its own intellectual property. These issues were resolved by 2019 when the contractor provided the necessary data. Because of these delays, the program now expects that the simulator will be ready by August 2020, with the planned simulator testing expected to take about 3 weeks. According to test officials, there is increased risk that the completion of the simulator may face additional delays to correct deficiencies and add needed capabilities, but also stated that they can complete the tests by August 2020. Due to these delays to completing operational testing, the program has delayed its full-rate production decision by at least an additional 9 months. Though the program is working toward September 2020, the program has acknowledged this decision could be made as late as March 2021. Any additional delays due to challenges with moving the emulators, completing the simulator, or cybersecurity testing could further delay the end of operational testing and the program’s decision to enter into full-rate production. This delay, however, gives the program more time to complete two key steps consistent with statute and DOD policy. Complete operational testing, which is intended to demonstrate that the aircraft are operationally suitable. Resolve all deficiencies, which should be done prior to full-rate production, and is discussed below. Even with these delays, the program plans to have produced and delivered over 550 aircraft before operational testing is complete, adding to the risk of finding more deficiencies that will require retrofits—at additional cost—for the delivered aircraft. Statute and DOD policy states that the preliminary low-rate production quantities will be set at the development request for proposal decision point. If, at that time, low-rate initial production quantities are determined to be above 10 percent of the total quantity planned, the Secretary of Defense must explain the reasons for the increase in a report to Congress. When a program reaches the planned low-rate initial production quantity, and requires to exceed the quantity, the program may seek approval to produce quantities above that amount. The F-35 program will have delivered more than 10 percent of the total planned production quantities—due to the necessity to prevent a break in production—before operational testing and the full-rate production decision are complete. As noted above, this approach has contributed to the $1.4 billion in concurrency costs already incurred by the program. The program reports that none of the category 1 deficiencies is a safety of flight concern and all of them have operational workarounds. In 2019, the program split the category 1 deficiencies into two groups. Group A are deficiencies that may cause death, severe injury, severe occupational illness, or major loss or damage to equipment and has no workaround. The program has none of these deficiencies currently. Group B are deficiencies that may critically restrict the combat readiness capabilities or may result in adequate performance but not be able to accomplish the primary or alternate missions. All of the 9 category 1 deficiencies are in group B. In 2019, the F-35 program resolved nearly 300 of the deficiencies it had identified in developmental and operational testing, but discovered even more over the same period. Specifically, 331 new deficiencies were identified in operational testing during 2019. As of December 2019, the F- 35 program had 870 open deficiencies. Of the 870 open deficiencies, the program characterizes nine as category 1 and 861 as category 2. The program reports that none of the category 1 deficiencies is a safety of flight concern and all of them have operational workarounds. In 2019, the program split the category 1 deficiencies into two groups. Group A are deficiencies that may cause death, severe injury, severe occupational illness, or major loss or damage to equipment and has no workaround. The program has none of these deficiencies currently. Group B are deficiencies that may critically restrict the combat readiness capabilities or may result in adequate performance but not be able to accomplish the primary or alternate missions. All of the 9 category 1 deficiencies are in group B. Of the 9 open category 1 deficiencies, the program reports all have operational workarounds—procedures that avoid encountering the deficiency. This represents four fewer open category 1 deficiencies than we reported in April 2019, reflecting the resolution of previously identified deficiencies and the addition of new ones, some of which were resolved. For example, the program fielded a software fix to a category 1 deficiency, which showed that the F-35’s cockpit display could falsely indicate its AIM-9X weapon—an air-to-air missile—selection status as “selected” though the weapon’s status is not selected. Figure 5 shows the F-35 firing an AIM-9X missile. The program office plans to continue to address the open deficiencies, but officials report that some will not be fully resolved for several years. Further, some deficiencies may not be resolved ever and some may be resolved well after the program has completed testing, and after it expects to have made a full-rate production decision. According to DOT&E, there are many significant deficiencies the program should address to ensure the F-35 baseline aircraft configuration is stable prior to adding all of the new capabilities planned in Block 4. As of December 2019, the program office and the contractor have resolved over 2,300 deficiencies and program office officials stated that they have a process in place to address the high priority ones. In 2019, the program reported continuing to negotiate lower unit prices across all F-35 aircraft variants and delivered more aircraft on time. However, officials also reported that the airframe and engine contractors demonstrated some declines in production performance, such as the number of labor hours to produce each aircraft, as production rates increased. We also identified other risk indicators that could affect the contractors’ future production performance. Specifically, the airframe contractor’s manufacturing processes do not meet all manufacturing leading practices that programs should meet before full-rate production. Additionally, parts shortages increased significantly in 2019 and Turkey’s suspension from the program will likely further complicate existing supply chain challenges. According to the program office, the negotiated prices for all F-35 variants have generally been decreasing with each production lot and as more aircraft are being procured in each lot. In April 2019, we reported that the program set a goal of reducing the negotiated unit price of an F-35A to less than $80 million by lot 13. According to a program official, in October 2019, the program finalized the contract action for lots 12-14 that met this goal. Specifically, with the most recent contract, the program agreed to purchase 351 F-35As, with unit costs declining to $73 million in lot 14. Figure 6 shows how the negotiated price for an F-35A has decreased since production began, as reported by the program office. According to the program office, it negotiated lower unit prices by working with the airframe contractor to leverage economic order quantity purchases and invest in cost reduction initiatives. Economic order quantities involve the contractor making large purchases of components that it will use across multiple procurement lots of aircraft to reduce production costs by achieving economies of scale. The program office estimates that the economic order quantity purchases for lots 12-14 will save the program about $225.5 million. In addition, the program office and prime contractors have continued to invest in various initiatives to lower production costs. Specifically, the program office spent $320 million in efforts to improve manufacturing processes that it estimates could result in up to $10.5 billion in savings over the life of the program. The airframe contractor told us that it has invested $170 million as of January 2019 to further lower its production costs. The engine contractor also told us that it spent $33 million to potentially realize over $4 billion of cost savings. The airframe contractor—Lockheed Martin—delivered 43 more aircraft in 2019 than in 2018, and as of October 2019, there were 229 aircraft in various stages of assembly worldwide. The contractor also delivered more aircraft on time in 2019. According to contractor officials, the improved rates of on-time delivery are partially a result of the contractor’s efforts to obtain a performance incentive fee that was added to the lot 11 production contract. The program intended the incentive fee to focus the contractor on improving its performance in the final assembly phase of production, which was expected to improve its on-time deliveries. To earn the incentive fee, contractor representatives told us they took several steps to improve production rates. For example, because the F-35Cs were taking longer to produce and all variants had to move through the same final assembly area, the contractor made a separate final assembly line for the F-35Cs so work could proceed without delaying the other variants. This step, according to program office officials, allowed the contractor to improve on-time deliveries for F-35As and F-35Bs. Figure 7 highlights progress in the contractor’s aircraft deliveries since 2016. Other production metrics associated with the airframe, however, demonstrated varied performance over the last two years as production increased. For example, the average number of hours needed to build an aircraft decreased slightly for the F-35A but increased for the B and C variants. Defense Contract Management Agency officials told us the increase was partly attributable to new personnel. In particular, since January 2017, the contractor has hired and trained nearly 1,700 new personnel to accommodate increased production rates—nearly doubling its workforce. New personnel take time to train and gain experience on the production line. According to contractor representatives, as these new employees become more experienced and produce more aircraft, they expect the metric to improve. The contractor’s amount of rework needed was also mixed. During the course of production, the contractor may identify issues with a part or a process, which, in turn, may lead to scrap, rework, and repair to replace or fix the issue. Between 2016 and 2017, most F-35 variants realized improvements in the amount of scrap, rework, and repair needed. In 2018 and 2019, however, only the F-35A continued to show improvements. Figure 8 shows the average total hours for scrap, rework, and repair for each variant since 2016. According to the program office, the increased production rate posed a challenge, and because the contractor has not built as many F-35Cs, this has added to the increase in scrap, rework and repair. To improve performance in this regard, the contractor put teams in place to focus on addressing the main drivers of scrap, rework, and repair. Similarly, the engine contractor—Pratt & Whitney—increased its production rate by roughly 51 percent in 2019. However, engine on-time delivery performance has continued to decline which officials attribute to production quality issues and parts delays. Specifically, in 2019, 91 percent of engines delivered were late. In 2019, the airframe contractor was able to work around the late engine deliveries to deliver the entire aircraft on time. Figure 9 shows the engine contractor’s on-time and late deliveries since 2016. In addition, the average number of quality notifications per engine— production defects indicating a quality issue—has increased by 16 percent in 2019. Figure 10 highlights the engine contractor’s quality notifications per engine over the last 4 years. According to the Defense Contract Management Agency’s performance reports, engine test failures, among other quality issues, have affected engine deliveries. According to an official from this agency, there have been 18 engine test failures in 2019, which is eight more than in 2018, each requiring disassembly and rework. The engine contractor stopped deliveries due to the test failures, which has slowed engine acceptance and reduced on-time deliveries. These issues are affecting engines built at the engine contractor’s production facility in West Palm Beach, Florida, which opened in 2014. To address this issue, the engine contractor has developed new tooling for the assembly line and has established a team to identify characteristics leading to the test failures. Plans are also in place for additional training for employees. While F-35 aircraft have been in production since 2007 and have reached a high level of manufacturing readiness per DOD guidance, the program is not meeting two of eight manufacturing leading practices GAO has identified as indicators of a program’s readiness for full-rate production, or milestone C review. To date, the program is meeting or plans to meet six leading practices for this milestone: Demonstrating processes on a pilot production line. Building and testing production-representative prototypes to demonstrate product in intended environment. Collecting statistical process control data. Conducting an independent cost estimate. Conducting an independent program assessment. Conducting major milestone decision review to begin production. However, we also found that the production processes are not in control according to the Process Capability Index. This index is a tool to measure how closely the production steps result in a part or subsystem that meets predefined standards. According to the leading practices, meeting these standards provides greater confidence that the contractor can produce a high quality product consistently, to minimize variation which results in fewer defects or the need for rework. Additionally, the F- 35 aircraft have not achieved their reliability goals through testing of production representative prototypes. These two leading practices focus on gathering sufficient knowledge to determine the relative ease of manufacturing and whether the product is of high quality and sufficiently mature to move forward into full-rate production. Our analysis of contractor data shows that the airframe contractor’s production processes are in flux. The contractor continues to change some of its production processes, and in other cases, is not following its own established processes well, which has led to several quality issues over the years. For example, in 2018, we reported that the contractor had halted deliveries of aircraft after the Air Force identified corrosion between the aircraft’s surface panels and the airframe because the contractor did not apply a primer when it attached the panels. We reported in 2019 that the program office, the contractor, and the F-35 Program Executive Officer reached a mutual agreement on the cost to resolve this issue, the details of which they did not disclose publicly. In November 2019, a mechanic identified titanium fasteners installed in an area of the aircraft where the design calls for a fastener stronger than titanium. According to the program office, the incorrect fasteners were installed on most already- fielded F-35 aircraft. That same month, the contractor started implementing its corrective action plan. As of March 2020, the F-35 program office had reviewed and approved the contractors’ analysis as well as its durability and damage reports on the use of these fasteners. We describe other key F-35 technical risks in appendix V. Over the years, the airframe contractor has continued to change and refine production processes, aiming to improve efficiency amidst concurrent development and production. For example, the airframe contractor identified a particular process that installs wiring harnesses into the aircraft wings as a driver of one of its production quality issues. To address this issue, the prime contractor developed a new tool that helps the installer route the wires more consistently. While process changes like these can improve the quality of the product, they also indicate that the overall production process are not in control less than a year before the program’s planned full-rate production decision, or milestone C review. In 2019, according to our analysis, the total number of key F-35 manufacturing processes identified in the final assembly phase increased 70 percent, to a total of over 10,000 critical processes. Furthermore, of these critical processes, only 30 percent are currently able to produce a product within predefined design standards. According to manufacturing leading practices, critical processes should be repeatable, sustainable, and consistent in producing parts within quality standards. Meeting these practices provides confidence that the contractor can produce the product within cost, schedule, and quality targets. Without processes in control, the program could face continued quality issues that will add to the overall cost of the program. Figure 11 shows the F-35 aircraft in the final assembly phase of production where some of these processes take place. Another leading practice that should be met before making a full-rate production, or milestone C decision, is to demonstrate that a production representative prototype can meet the program’s R&M goals. The R&M goals lay out specific quantitative goals aimed at ensuring that an aircraft will be available for operations as opposed to out of service for maintenance. We reported in April 2019 that the F-35 aircraft in service around the world were still not meeting all of their R&M goals and recommended the program take actions to ensure that the aircraft would meet those goals. Despite some improvement in 2019, the program is not meeting half of its R&M goals. Until the program does so, the warfighter will continue to accept aircraft for delivery that are less reliable and more costly to maintain than originally planned. For details on the F- 35’s R&M performance, see appendix IV. The program has not met these two leading manufacturing practices, in part, due to the changes the airframe contractor made and continues to make to the production line and the program’s concurrent approach to acquisition. We have repeatedly found that DOD programs that moved into full-rate production carrying manufacturing risks experienced billions of dollars in cost growth in production, and nearly two-thirds reported increases in average procurement unit costs. With the risks the F-35 program still faces, it may realize additional cost and schedule growth if these production risks are not evaluated. Despite these risks, the program has continued to push forward with increased production rates and has not taken actions to determine the potential impact of not meeting these leading practices may have on future production and overall life-cycle costs. Furthermore, according to a program official, the F-35 program has not completed a comprehensive assessment of production risks and does not plan to ahead of its full-rate production decision. However, according to DOD officials, the F-35 program office and prime contractor convene a monthly Joint Risk Management Board, which identifies and manages overall program risk, and has completed an independent technical risk assessment to support the full-rate production decision, which identified production risks. Title 10 section 2366c of the U.S. Code requires the milestone decision authority for a major defense acquisition program to provide Congress with a report that includes, among other things, a summary of any manufacturing risks associated with the program; however, this summary is not required until 15 days after the authority grants approval for the program to enter the production and deployment phase. The program currently plans to obtain this approval between September 2020 and March 2021. In this case, however, the F-35 program has not met all of the manufacturing leading practices that should be met before the full-rate production decision. Furthermore, the underlying risks, such as not meeting R&M goals, have persisted for years and the program has yet to take steps to fully address these risks. If an evaluation of these risks is not provided ahead of the full-rate production decision, Congress will not be fully aware of the risks the program is taking by committing to increased production rates. According to program officials, some suppliers for the F-35 struggled to meet increased production demands in 2019 and, as a result, the program witnessed increased rates of late deliveries or parts shortages. In particular, the number of parts delivered late to the airframe contractor, as well as parts shortages, have grown steadily over the past 2 years. According to the Defense Contract Management Agency: Between August 2017 and July 2019, the number of parts delivered late increased from under 2,000 to more than 10,000. Between July 2018 and July 2019, the parts shortages per month increased from 875 to over 8,000. According to contractor representatives, roughly 60 percent of parts shortages are attributable to 20 suppliers. To mitigate late deliveries and parts shortages—and deliver more aircraft on time—the airframe contractor has utilized methods such as reconfiguring the assembly line and moving planned work between different stations along the assembly line. According to the program office, such steps can cause production to be less efficient, which, in turn, can increase the number of labor hours necessary to build each aircraft. Airframe contractor representatives and a program office official cited measures they are taking to improve supplier performance in light of the upcoming full-rate production decision. For example, the contractor instituted action plans to help problematic suppliers, sent task teams to struggling suppliers to help resolve issues, and, in some cases, is seeking alternative sources. Additionally, the program office has established joint meetings with the prime contractor to monitor progress on a weekly basis and holds a semiannual review to achieve executive-level coordination. While prime contractor representatives told us that they have been actively managing underperforming suppliers for several years, some of their efforts are new and will need time before results materialize. These supply chain risks may compound as the program continues to produce, deliver, operate, and maintain more aircraft each year. For example, in April 2019, we found that fielded, operational F-35 aircraft were not meeting warfighter requirements, largely due to spare parts shortages and difficulty in managing and moving parts around the world. We recommended that the program assess what actions it should take to meet warfighter requirements, which could include adjusting the amount of spare parts acquired. DOD concurred and is working toward addressing the recommendation to identify warfighter gaps with regard to the supply chain. However, with the aircraft in production also facing significant shortages, this problem could get worse as the program prepares to further increase the production rate from 141 aircraft in 2019 to 169 in 2022. We found that Turkey’s recent suspension from the F-35 program is likely to compound these existing supply chain issues. In July 2019, Turkey was suspended from the F-35 program. In particular, the Under Secretary of Defense for Acquisition and Sustainment directed that the F-35 program establish alternative sources and to stop placing orders from Turkish suppliers after March 2020. According to an official with that office, Turkish suppliers will provide parts through the end of lot 14 deliveries (scheduled to take place through 2022), in part, to avoid disruptions to aircraft deliveries and additional cost growth from standing up new suppliers. The F-35 program office identified that Turkish companies supplied 1,005 parts for the F-35 airframe and engine and some of these parts have been provided by only one supplier. As of December 2019, the program has identified new suppliers for all of these parts, but it still needs to bring roughly 15 parts currently produced in Turkey up to the current production rate. During our review, the program reported that production through lot 14 should not be adversely affected if it continues to accept parts from Turkey until lot 14 aircraft are delivered, but risk remains with the transition to alternate sources. However, lots 12-14 still face some risk receiving parts from Turkey. According to program officials, some of these new parts suppliers will not be producing at the rate required until next year, as roughly 10 percent are new to the F-35 program. Airframe contractor representatives stated it would take over a year to stand up these new suppliers, with lead times dependent on several factors, such as part complexity, quantity, and the supplier’s production maturity. In addition, these new suppliers are required to go through qualification and testing to ensure the design integrity for their parts. According to an official with the Under Secretary of Defense for Acquisition and Sustainment, by accepting parts from Turkish suppliers through lot 14, the program will have additional time to ensure new suppliers can meet demands for parts. Additionally, the program reported that it intends to utilize alternative sources for parts currently made in Turkey for aircraft delivered under lots 13 and 14 contracts. Furthermore, according to a program office official, it is also not clear how the prices for parts that will be obtained from new suppliers after Lot 14 will compare with the prices under the contracts with the suppliers from Turkey, but the official noted that alternative sources could be more costly. In its May 2019 report to Congress, DOD outlined its plans for Block 4 with a development cost estimate of $10.6 billion for activities through fiscal year 2024. Since the 2019 report, we found the program office has increased its estimate by about 14 percent, to $12.1 billion, primarily due to schedule delays. The program now expects to extend the delivery of Block 4 capabilities by 2 additional years, through 2026. In the meantime, DOD’s Block 4 annual reporting requirement to Congress is scheduled to end in 2023, 3 years before development is complete. Additionally, most of the capabilities the F-35 program planned to deliver in 2019 were delayed. Furthermore, we found that the program’s cost estimate used to support its report to Congress does not fully meet cost estimating leading practices. The Block 4 development cost and schedule have grown considerably since DOD’s last report to Congress. In 2016, GAO recommended that DOD manage Block 4 as a major defense acquisition program with its own reporting requirements, separate from the original F-35 development program. DOD did not concur with our recommendation, citing the F-35 as DOD’s most closely managed system and its existing F-35 program oversight. The NDAA for Fiscal Year 2017 required DOD to report annually on elements of a Block 4 baseline, such as development and retrofit cost estimates, beginning no later than one year after the award of the development contract for follow on modernization, until March 31, 2023. At that time, we reported that DOD had requested funding for the development and delivery of Block 4 through the end of 2022. However, over the last year, the program has revised its Block 4 schedule and now expects to field Block 4 capabilities into fiscal year 2026. As a result, there is no requirement for DOD to report on Block 4 progress for at least 3 years even though those efforts will be ongoing. In its May 2019 Block 4 report to Congress, DOD reported that the total cost to develop 66 Block 4 capabilities—both hardware and software— would be $10.6 billion for activities planned from fiscal years 2018 to 2024. The report also included the F-35 program office estimate of an additional $6.4 billion in fiscal year 2018 through 2024 funding to retrofit aircraft from the baseline F-35 configuration to a full Block 4 configuration. The F-35 program based the costs in this report on its Block 4 development cost estimate from July 2018. However, we found that reported Block 4 costs did not include all Block 4 costs. In particular, the report did not include Block 4 costs the program incurred prior to 2018 or costs that the effort will incur after 2024. Because the F-35 program office is not managing the Block 4 effort as a separate program, it has chosen to exclude the past and future costs in the Block 4 cost estimate it reported to Congress. Instead, the program reported on Block 4 costs for the future years defense program—which is DOD’s projected spending for the current budget year and the next four years. By excluding any costs prior to 2018 and those that would be incurred after 2024, the program did not report on the total costs of Block 4. In May 2019, the program also updated its Block 4 development cost estimate, increasing both the time and cost to complete the work, but this updated estimate was not included in its May 2019 report to Congress. The updated cost estimate reflects that the program office will be fielding Block 4 capabilities into fiscal year 2026. This new schedule adds 2 years to the costs DOD reported to Congress in May 2019. Additionally, our analysis of DOD’s updated cost estimate indicates the total cost of Block 4 development grew by $1.5 billion to a total of $12.1 billion for activities in fiscal years 2018 through 2026. Furthermore, in addition to the Block 4 development costs, the program also estimates it will need another $2.9 billion to develop other capabilities, such as upgrades to ALIS. Program officials attributed this schedule and cost growth to having better insight into the scope of work to develop and test Block 4 capabilities and noted that they would continue to refine and update these costs annually as modernization efforts progress further into development. Once the existing statutory reporting requirement expires in 2023, DOD will no longer be required to provide Congress key information that would be useful in making informed decisions regarding the Block 4 effort— which now extends until 2026. Furthermore, without a complete cost estimate for Block 4, inclusive of costs already incurred and those not yet incurred but estimated through completion, Congress is left without a complete picture of what DOD intends to spend on the total Block 4 effort. Without a complete picture of these costs, the Congress’s ability to assess the program’s cost and schedule performance in the future will be hindered. The airframe contractor did not deliver the Block 4 capabilities it planned to deliver in 2019. Specifically, according to the plan outlined in its May 2019 report to Congress, the F-35 program was going to deliver eight Block 4 capabilities in 2019. However, the program delivered only one—a software capability called the auto ground-collision avoidance system. This capability enables the aircraft to perform an automatic recovery when it predicts that the aircraft will strike the ground. This was ahead of schedule as the program had originally planned to deliver this capability after 2019. According to program officials, the development of the other capabilities is taking longer than planned and, as a result, the program pushed their delivery schedule into 2020. Development and delivery of the capabilities within the Block 4 effort are complex, and the program does not consider development complete until the products for all elements of the F-35 air system are ready. In particular, full capability delivery occurs when the contractor delivers all of the software and hardware needed for all of the F-35 air system elements to support the planned capability. Program officials stated they are still working to put the processes in place to synchronize the delivery of the late capabilities for all of the F-35 air system elements. For example, the airframe contractor had planned to deliver a capability called the interim full motion video for the Marine Corps in 2019. The contractor developed the software needed, but it is late in developing the hardware needed for the software to operate and, as a result, the contractor did not deliver the capability in 2019 as planned. DOD test officials we met with at Edwards Air Force Base stated that in 2019, using the C2D2 approach, the contractor delivered other, partial Block 4 capabilities to be tested. However, test officials told us those capabilities were delivered later than expected. Since the program could not fully test those capabilities on the aircraft, the program office deferred them to the next incremental update scheduled for 2020. Changes such as these have contributed to the Block 4 cost and schedule growth. The program is also discovering issues during Block 4 testing, causing the testing to take longer than anticipated. According to a DOT&E official, Block 4 software changes caused issues with functionality of F-35 baseline aircraft capabilities that worked before the program installed new Block 4 software onto the aircraft. The program discovered issues with each new software version during flight testing and has been working to fix these issues in subsequent software updates. Testing and DOD officials stated that the contractor had not performed adequate testing of the software before delivering it to the test fleet as the reason for these issues. Contractor representatives acknowledged these issues and stated that they will conduct additional lab testing for future software releases to avoid such problems going forward. We found that the F-35 program office’s Block 4 cost estimate did not fully meet the four key characteristics of GAO’s cost estimating leading practices when projecting Block 4 development costs. Table 2 presents key points from our assessment, and appendix II provides additional detail on our rationale. As reflected in table 2, our assessment of the F-35 Block 4 development cost estimate identified a number of missing elements. Specifically, the estimate does not rely on a product-oriented work breakdown structure (WBS), it does not address cost risk and uncertainty, it does not take into account risk related to technology maturity, and it does not have an independent cost estimate, as leading practices reflect. While the program office updates its cost estimate regularly, officials told us that they do not intend to address some of these missing elements in future updates. Work breakdown structure. According to cost estimating leading practices, the program should base its cost estimate on a program- level, product-oriented WBS that allows a program to track cost and schedule by defined deliverables, such as hardware or software components. The WBS ensures that the program does not leave out any portions of the work and makes it easier to compare it to similar systems and programs. According to program officials, the Block 4 cost estimate does not rely on a single WBS; rather, multiple, contractor-derived WBSs exist for the program. Without its own, program-office-level WBS, the program lacks a framework to develop a schedule and cost plan that it can use to track progress and accomplishments. Risk and uncertainty analyses. The program did not perform cost risk and uncertainty analyses. Program officials said they do not plan to conduct a formal risk analysis. The program office works jointly with the contractor to identify and manage risks for the F-35 program. For example, there are monthly Joint Risk Management Boards attended by both program office and contractor leadership. However, overall program risk management is different from quantitative cost risk and uncertainty analyses in that program risk management is not specific to costs and it is not used to assess the cost variance of the cost estimate itself. When planning for funding decisions for a program of this scale, analyzing program-level risks alone is inadequate. Without a risk analysis, the cost estimate will not be fully accurate or credible because it will not account for the effects of potential schedule slips or other risks that the program could realize. Technology maturity. A program office official stated that in developing the cost estimate they did not consider that technologies would not be mature, but rather assumed that most technologies needed to deliver each Block 4 capability would be mature before the program begins development for that capability. The official stated that the complexity of design, development, and testing based on the baseline program experience was reflected in the estimate, but the cost estimate did not identify if there were specific costs associated with maturing these technologies. The official further noted that Block 4 costs would increase if a capability takes longer than planned to design, integrate, and test due to its immaturity. In 2019, we recommended that the Secretary of Defense ensure that the F-35 program office completes an independent technology readiness assessment, as part of its business case for the initial Block 4 capabilities, before initiating additional development work. DOD did not concur with our recommendation. According to a program official, as of December 2019, the program office had not completed any technology readiness assessments even though the contractor has started development of over half of the capabilities within Block 4. Going forward, the program is considering holding incremental technology readiness assessments as it plans for and develops a new set of capabilities, in accordance with the C2D2 schedule. Program officials told us that, going forward, as they update the Block 4 cost estimate, they will consider the results of future technology readiness assessments. Until the program office does so, management cannot determine a reasonable level of additional resources that might be necessary to cover increased costs resulting from unexpected design complexity, incomplete requirements, technology uncertainty, and other uncertainties. Independent cost estimate. In 2019, we also recommended the F- 35 program office include an independent cost estimate as part of its business case for Block 4. As noted in table 2, the Block 4 effort still lacks an independent cost estimate. The program is planning for the Office of the Secretary of Defense, Cost Assessment and Program Evaluation to have a draft independent cost estimate for an interim program review scheduled for March 2020 and to have a complete independent cost estimate in June 2020. This estimate will evaluate the entire F-35 program, including Block 4. With these pieces currently missing, the Block 4 development cost estimate does not present a full picture of Block 4’s cost. Ultimately, without a complete understanding of Block 4 costs, the program could face additional cost growth, which will be hard to track without a complete cost baseline. The lack of a complete cost baseline hinders insight and oversight into the program’s costs, plans, and progress to date and going forward. Moreover, if a cost estimate does not fully or substantially meet all four characteristics of cost estimating leading practices, it cannot be considered reliable. DOD plans for the F-35 to be central to the warfighter prevailing in future conflicts. However, the program has been behind schedule and over cost almost since its inception. DOD is slated to move into full-rate production despite several key challenges in the production of aircraft. We acknowledge that the current F-35 program’s production rates are more commonly associated with programs already in full-rate production. However, the F-35 aircraft in the field have not met standards for reliability and maintainability, indicating that the program is not delivering aircraft at the level of quality expected. Additionally, the program’s concurrent approach and the contractor’s continual changes to the production line indicate that the production line processes are not in control. Leading practices indicate that mature production lines—production lines ready for full-rate production—should meet metrics for consistency. Furthermore, to minimize production risk and potential cost growth, suppliers should routinely meet quality and delivery schedules, although this is not yet true of the F-35 program. Not meeting these leading practices poses risks that DOD and the international partners will not routinely receive the F-35’s they specified and need. The long-standing challenges with receiving parts on time and efforts underway to replace Turkish suppliers of parts for the F-35 compound these production challenges and may raise additional risks. Unless the program office assesses and reports on these manufacturing risks ahead of the milestone C review, Congress may not have key insights into the risks that remain with the program and to the overall effort to deliver F-35s to the warfighter. Since the F-35 program is not managing the Block 4 effort as a separate program with traditional oversight tools, we are particularly concerned as Block 4 efforts proceed through development and testing. Specifically, because of the delays to the program, after 2023, DOD will not be required to provide Congress information on Block 4’s development efforts as the current reporting requirements will end. Furthermore, the program’s cost estimate, as presented in its report to Congress, does not fully present all incurred and future costs for Block 4. Without this information, Congress may not have the insight it needs to assess Block 4 cost and schedule progress as well as to make informed oversight and budgeting decisions. In addition, the Block 4 development cost estimate does not fully meet leading practices, lacking a full reflection of all costs. Specifically, the cost estimate does not have a program office level work breakdown structure, a risk and uncertainty analysis, and consideration of technology readiness. Without a comprehensive and credible cost estimate, DOD and Congress lack a sound basis for informed investment decision making, realistic budget formulation, meaningful progress measurement, proactive course correction when warranted, and program and contractor accountability for results. Congress should consider revising Section 224(d) of the National Defense Authorization Act for Fiscal Year 2017, Pub. L. No. 114-328, to extend DOD’s Block 4 reporting requirement until all Block 4 capabilities are fielded to ensure that Congress is aware of cost and schedule growth beyond 2023. (Matter for Consideration 1) We are making the following five recommendations to the Secretary of Defense to direct the Undersecretary of Defense for Acquisition and Sustainment (OUSD (A&S)). The OUSD (A&S) should direct the F-35 program office to provide information that is similar to that which is statutorily required after the milestone C review to Congress ahead of the milestone C review (full-rate production decision). This submission should include an evaluation of the production risks associated with critical production processes that are not in control, reliability and maintainability (R&M) targets that are not met, and supplier readiness—particularly for those replacing Turkish suppliers, along with the steps it is taking to address those risks. (Recommendation 1) The OUSD (A&S) should direct the F-35 program office to establish a Block 4 cost estimate baseline that includes all Block 4 costs, including incurred costs and future costs in its reports to Congress as required by the NDAA for Fiscal Year 2017, so that Congress has a complete understanding of all Block 4 costs and can compare this baseline to future cost estimates and performance. (Recommendation 2) The OUSD (A&S) should direct the F-35 program office to complete a program office level, product-oriented work breakdown structure for the next update to its Block 4 cost estimate to ensure that the estimate meets the comprehensive leading practices. (Recommendation 3) The OUSD (A&S) should direct the F-35 program office to conduct risk and uncertainty analyses for the next update to its Block 4 cost estimate to ensure that the estimate meets the credible leading practices. (Recommendation 4) The OUSD (A&S) should direct the F-35 program office to consider the results of its future technology readiness assessment of all Block 4 technologies and incorporate the cost and schedule risks of developing those technologies in the next update to its Block 4 cost estimate to ensure that the estimate meets the comprehensive leading practices. (Recommendation 5) We provided a draft of this report to DOD for review and comment. DOD provided written comments, which we have reproduced in appendix VI. DOD concurred with three of the recommendations related to the Block 4 modernization effort (recommendations 2, 4, and 5 above). While DOD did not concur with the other two recommendations, it outlined planned actions that we believe, if implemented, would meet the intent of our recommendations. DOD also provided technical comments, which we incorporated as appropriate. We will continue to monitor the program and evaluate implementation of these recommendations. DOD officials did not concur with the first recommendation, which, in the draft report, was to evaluate production risks and provide a statutorily required report to Congress ahead of the program’s full-rate production decision. While DOD did not concur with the draft recommendation, it agreed to keep the Congress apprised of these matters in its quarterly briefings to the defense committees. To clarify the actions we intended DOD to take to address our findings, we revised the recommendation to indicate that DOD should provide information to Congress on the production risks we identified in our report, ahead of the milestone C review. If the DOD provides a substantive assessment highlighting these production risks, as well as the steps it will take to mitigate them, during its quarterly briefing to Congress ahead of the milestone C review, it would address the intent of our recommendation. DOD also did not concur with our third recommendation for the F-35 program office to complete a program-level, product-oriented work breakdown structure (WBS) for the next update to its Block 4 cost estimate. DOD noted that its next scheduled update was due in April 2020, after we provided our report for comment. While DOD noted it would be unable to complete a program-level WBS by the April 2020 update, it agreed to evaluate moving to a program-level, product-oriented WBS in 2021. If the F-35 program office utilizes a program-level, product- oriented WBS for this cost estimate update, it would meet the intent of our recommendation. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; and the Under Secretary of Defense for Acquisition and Sustainment, the Secretary of the Air Force, the Acting Secretary of the Navy, and the Commandant of the Marine Corps. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or ludwigsonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. To assess the reliability of the F-35 Block 4 development cost estimate, we obtained and reviewed cost estimate documentation such as the Joint Program Office briefing on its May 2019 estimate, the Air System Procurement Playbook—a planning document for Block 4—and its cost estimate models. Additionally, we met with relevant staff in the F-35 program office and the Department of Defense’s Office of Cost Assessment and Program Evaluation. We analyzed this information and determined the extent to which the program office’s practices for developing the F-35 Block 4 development cost estimate were consistent with the leading practices identified in the GAO Cost Estimating and Assessment Guide. These practices have been found to be the basis for reliable cost estimates. We assessed each practice as being one of the following: Met—the agency provided data and documentation that satisfies the entire leading practice criterion. Substantially met—the agency provided data and documentation that satisfies a large portion of the leading practice criterion. Partially met—the agency provided data and documentation that satisfies about half of the leading practice criterion. Minimally met—the agency provided data and documentation that satisfies a small portion of the leading practice criterion. Not met—the agency provided data and documentation that does not satisfy any portion of the leading practice criterion. For our reporting needs, we collapsed GAO’s 18 leading practices into four general characteristics: comprehensive, well-documented, accurate, and credible. The assessment of each characteristic was based on an average of the F-35 program office’s scores for the leading practices included in that category. A second analyst verified the assessment and management reviewed the results. We determined the overall assessment rating by assigning each individual rating a number: Not met = 1, Minimally met = 2, Partially met = 3, Substantially met = 4, and Met = 5. Then, we took the average of the individual assessment ratings to determine the overall rating for each of the four characteristics. The resulting average becomes the Overall Assessment as follows: Not met = 1.0 to 1.4, Minimally met = 1.5 to 2.4, Partially met = 2.5 to 3.4, Substantially met = 3.5 to 4.4, and Met = 4.5 to 5.0. A cost estimate is considered reliable if the overall assessment ratings for each of the four characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and cannot be considered reliable. See table 4 for a high level summary of each leading practice and the reasons for the overall scoring. This report fulfills two mandates. First, the National Defense Authorization Act for fiscal year 2015 included a provision for GAO to review the F-35 acquisition program annually until the program reaches full-rate production. This is the fifth report under that provision. Second, the National Defense Authorization Act for Fiscal Year 2020 includes a provision for GAO to review the program’s production and Block 4 progress annually through 2025. In this report, we (1) provide information on the program’s progress toward completing operational testing and resolving deficiencies found in testing; (2) assess the program’s production performance and manufacturing efficiency initiatives; and (3) assess the program’s modernization cost estimate and progress with Block 4 development efforts. To provide information on the program’s progress in operational testing and the resolution of deficiencies, we first reviewed the baseline program’s costs, schedule, and performance plans and compared the actual progress in each area with the goals established in its 2012 baseline to identify any significant trends. We reviewed progress on test events completed versus those that remain, test schedules, program briefings, and DOD briefings. We traveled to Edwards Air Force base to interview DOD test authorities and met with officials from the program office, DOD test authorities, and the contractor Lockheed Martin (the prime aircraft contractor), to discuss key aspects of F-35 development progress, including flight testing, future test plans, and recent findings from test events. Specifically, we obtained updates on key events that are required to complete testing according to the program office’s current schedule. We also interviewed the Director, Operational Test and Evaluation office and F-35 program developmental and operational test pilots. To provide information on the program’s progress resolving deficiencies, we interviewed the same officials mentioned above and discussed how the number of open and closed deficiencies changed in 2019. We reviewed program and contractor information on deficiency reports, mitigations, resolutions, and the deficiency resolution process. To assess the program’s production performance and manufacturing efficiency initiatives, we obtained and analyzed the production metrics from Lockheed Martin and Pratt & Whitney (the prime engine contractor) and their aircraft and engine delivery rates from 2012 through 2019. We reviewed metrics and briefings provided by the program office, Lockheed Martin, Pratt & Whitney, and the Defense Contract Management Agency to identify progress in improving manufacturing processes. We analyzed delivery dates for lot 11 aircraft delivered in 2019. We traveled to the production facility in Fort Worth, Texas to discuss reasons for any delivery delays and plans for improvements with officials from Lockheed Martin. We obtained cost investment and savings estimates and discussed cost and manufacturing efficiency initiatives, such as the economic order quantity purchases, with the contractors and program office officials to understand potential cost savings and plans. We collected and analyzed the extent to which the program has met leading practices identified by GAO for full-rate production. We also obtained and analyzed metrics on parts and aircraft quality through December 2019 and discussed steps taken to improve quality and deliveries with Lockheed Martin and Pratt & Whitney officials. We interviewed officials from Lockheed Martin, Pratt & Whitney, and Northrop Grumman (a key subcontractor) regarding the administration’s decision to suspend Turkey from the program and the implications of the suspension for the contractors. We determined that the contractors’ production metrics and delivery dates were sufficiently reliable for our purposes of determining production efficiency and deliveries. We collected and analyzed production and supply chain performance data from the program office, Lockheed Martin, and Pratt & Whitney. To assess the reliability of the May 2019 Block 4 development cost estimate, we evaluated documentation supporting the estimate, such as the cost estimating models, the F-35’s Air System Procurement Playbook, its updated acquisition strategy, the Decision Memorandum requirements document, and briefings provided to the DOD decision authority. We assessed the cost estimating methodologies, assumptions, and results against leading practices for developing a comprehensive, accurate, well- documented, and credible cost estimate, identified in GAO’s Cost Estimating and Assessment Guide. We also interviewed program officials responsible for developing and reviewing the cost estimate to understand their methodology, data, and approach for developing the estimate. We found that the cost estimate was not reliable. To assess progress with Block 4 development efforts, we interviewed DOD and program office officials, and contractor representatives regarding the program’s Block 4 planning, development, testing, and production activities to date. We reviewed other program documentation, such as the F-35’s fiscal year 2020 budget request, to identify costs associated with the Block 4 effort. We compared the program’s accomplishments in 2019 to its plans and identified what capabilities the program office delivered to the fleet. We reviewed the program office’s plans to develop and deliver additional Block 4 capabilities from 2020 through 2025. We conducted this performance audit from June 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Joint Strike Fighter Operational Requirements Document, which outlines the requirements Department of Defense and the military services agreed the F-35 should meet, defines all eight reliability and maintainability (R&M) metrics. Table 5 shows each F-35 variants’ performance against these metrics’ targets, as of August 2019. F-35B Thrust Cutback: An F-35B aircraft can experience an unanticipated cutback in thrust during vertical landings (hover). The contractor put hover weight restrictions in place to mitigate the effect and has identified the root cause. The contractor is developing software and hardware fixes. F-35C Nose Landing Gear: During shipboard landings, the F-35C can experience bending stress, which causes cracking of the coating on a part in the nose landing gear. In the short term, this part will be inspected for damage every 400 flying hours. The contractor is also redesigning the part that is cracking and expects to test it between early 2020 to June 2021. F-35B Three Bearing Swivel Module: The module is mounted at the back of the aircraft and allows the thrust from the engine to be vectored from straight aft for conventional flight to straight down for short takeoff and vertical landing operations. In June 2019, an F-35B experienced a warning indicator in its short takeoff mode due to a hardware component. However, according to the contractor, this component should not cause a warning indicator or loss of functionality for the aircraft. The contractor has identified the root cause of the hardware issue and a gap in the software’s logic that led to the warning. As a result, the contractor is making manufacturing changes to the hardware and implementing software changes to address the issue. Canopy Coating Delaminations: The F-35 fleet has experienced over 50 incidents of the canopy transparencies delaminating after less than 100 flight hours since August 2017. This is over 30 more than we reported in 2019. The contractor tested solutions for the delaminations in 2019 and implemented a solution of adding a vent to the canopy’s frame. Since October 2019, the contractor has added a vent to 146 canopies with one subsequent delamination. Helmet Mounted Display: During low-light flights, the Helmet Mounted Display’s technology cannot display pure black images, instead presenting a green glow on the screen, which makes it difficult to see the full resolution of the night vision video feed. The contractor developed a new display to avoid this effect. According to F-35 program officials, they placed an initial order of 62 displays with 35 delivered by December 2019 to support U.S. Marine Corps and Navy F-35C fleet operations. Three F- 35C pilots completed initial day and night testing using the new display in July 2019 on a carrier. The contractor expects to have a fully qualified redesign by August 2021 and will incorporate it into the production of lot 12 aircraft. In addition to the contact named above, the following staff members made key contributions to this report: Justin Jaynes and Alissa Czyz (Assistant Directors); Diana Maurer, Desirée E. Cunningham (Analyst-in- Charge), Jillena Roberts, Tim Moss, Rose Brister, Juaná Collymore, Emile Ettedgui, Jennifer Leotta, and Jeff Hubbard, Other staff who contributed include Leslie Ashton, Priyanka Sethi Bansal, Vinayak Balasubramanian, Julia DiPonio, Christine Pecora, Ralph Roffo, Roxanna Sun, Jessica Waselkow, and Alyssa Weir. Weapon System Sustainment: DOD Needs a Strategy for Re-Designing the F-35’s Central Logistics System. GAO-20-316. Washington, D.C.: March 6, 2020. F-35 Aircraft Sustainment: DOD Faces Challenges in Sustaining a Growing Fleet. GAO-20-234T. Washington, D.C.: November 13, 2019. Defense Acquisitions: Observations on the F-35 and Air Force’s Advanced Battle Management System. GAO-19-456T: Washington, D.C.: May 2, 2019. F-35 Joint Strike Fighter: Action Needed to Improve Reliability and Prepare for Modernization Efforts. GAO-19-341. Washington, D.C.: April 29, 2019. F-35 Aircraft Sustainment: DOD Needs to Address Substantial Supply Chain Challenges. GAO-19-321. Washington, D.C.: April 25, 2019. F-35 Joint Strike Fighter: Development is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved. GAO-18-321. Washington, D.C.: June 5, 2018. Warfighter Support: DOD Needs to Share F-35 Operational Lessons Across the Military Services. GAO-18-464R. Washington, D.C.: April 25, 2018. F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency. GAO-18-75. Washington, D.C.: October 26, 2017. F-35 Joint Strike Fighter: DOD’s Proposed Follow-on Modernization Acquisition Strategy Reflects an Incremental Approach Although Plans Are Not Yet Finalized. GAO-17-690R. Washington, D.C.: August 8, 2017. F-35 Joint Strike Fighter: DOD Needs to Complete Developmental Testing Before Making Significant New Investments. GAO-17-351. Washington, D.C.: April 24, 2017. F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities. GAO-16-390. Washington, D.C.: April 14, 2016. F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016. F-35 Joint Strike Fighter: Preliminary Observations on Program Progress. GAO-16-489T. Washington, D.C.: March 23, 2016. F-35 Joint Strike Fighter: Assessment Needed to Address Affordability Challenges. GAO-15-364. Washington, D.C.: April 14, 2015. F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.: September 23, 2014. F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26, 2014. F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014. F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013. F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013. Fighter Aircraft: Better Cost Estimates Needed for Extending the Service Life of Selected F-16s and F/A-18s. GAO-13-51. Washington, D.C.: November 15, 2012.", "summary": "The acquisition cost for the F-35 program increased substantially in 2019, partially due to the program's addition of estimated costs for modernization of hardware and software systems, referred to as its Block 4 efforts. This is the fifth report under the provision that Congress included in statute for GAO to review the F-35 program annually until the program reaches full-rate production. This is also the first report under another provision in statute to review the program's production and Block 4 progress annually through 2024. Among other objectives, this report assesses (1) the program's production performance and (2) the program's modernization cost estimate and development progress. GAO reviewed Department of Defense (DOD) and contractor documentation and interviewed DOD officials and contractor representatives. The F-35 program is at risk of missing its test schedule and not meeting manufacturing leading practices. In 2019, the F-35 program conducted much of its planned operational testing but extended the schedule by 9 months, which delays the program's full-rate production decision to between September 2020 and March 2021. Over that time, the program will continue to deliver aircraft. In addition, while the F-35 program has increased the production rate and negotiated lower aircraft prices, it is not meeting manufacturing leading practices identified by GAO. Specifically, only about 3,000 of the over 10,000 airframe contractor's manufacturing key processes meet predefined design standards for ensuring product quality. Also, the fielded aircraft, over 500 so far, do not meet the program's reliability and maintainability goals. Although the contractor is changing manufacturing processes to address problems and improve efficiency, more remains to be done. Unless the program office evaluates the risks of not meeting these leading practices, the military services and international partners are at risk of not receiving the quality aircraft they purchased. In addition, the July 2019 suspension of Turkey from the F-35 program—due to security concerns after its acquisition of Russian defense equipment—is likely to compound production risks. The program has identified new sources for 1,005 parts produced by Turkish suppliers, but the program is assessing the effect of 15 key parts not currently being produced at the needed production rate. In 2019, estimated development costs to modernize the F-35's hardware and software systems—known as Block 4—increased by over $1.5 billion. The cost increase puts estimated Block 4 development costs at $12.1 billion. However, the cost estimate did not fully adhere to leading practices, such as including all life cycle costs. In addition, while development will continue through 2026, reports on Block 4 that the program submits to Congress are slated to end in 2023. Without continued Block 4 reporting through the development phase, Congress will lack important oversight information. Congress should consider extending DOD's reporting requirement for Block 4 modernization beyond 2023. GAO is also making five recommendations to DOD. While DOD did not concur with two of these recommendations—including to evaluate production risks and update its Block 4 cost estimate with a program-level plan, it identified actions that, if implemented, will meet the intent of these recommendations. DOD concurred with GAO's three other recommendations.", "document_type": "gao"}
{"report": "The Secret Service pursues two areas of responsibility simultaneously— protection and criminal investigations. The Secret Service’s Office of Protective Operations oversees the agency’s protective divisions, including the Presidential Protective, Vice Presidential Protective, and Uniformed Divisions. These divisions carry out permanent protective details and other protection-related assignments. Permanent protectees, such as the President and Vice President, have special agents permanently assigned to them from the Presidential Protective Division or Vice Presidential Protective Division. The Secret Service provides protection for the President, Vice President, and their families at all times. In fiscal year 2017, the Presidential and Vice Presidential Protective Divisions provided protection for 30 presidential and vice- presidential foreign trips in addition to providing protection for members of the President’s and Vice President’s families. The Uniformed Division protects certain facilities, including the White House and the Treasury Building, among others. Figure 1 illustrates an organizational chart of offices within the Secret Service. The Office of Investigations oversees the agency’s field activities, including investigations into crimes targeting the nation’s financial systems; surveys of locations a protectee may visit; investigations of threats to protected persons and facilities; and temporary support for protection. Figure 2 provides information about the components in the Office of Investigations. The Office of Investigations oversees the agency’s 21 international field offices and 141 domestic offices, consisting of 42 field offices, 60 resident offices, 13 resident agencies, and 26 domiciles. Special agents in these offices conduct investigations to identify, locate, and apprehend criminal organizations and individuals targeting the nation’s critical financial infrastructure and payment systems. Figure 3 shows the locations of Secret Service’s domestic field offices, resident offices, and resident agencies. Although the Secret Service was originally founded to investigate the counterfeiting of U.S. currency, the agency’s investigations now span a number of financial and computer-based crimes. Pursuant to 18 U.S.C. § 3056(b)(2), under the direction of the Secretary of Homeland Security, the Secret Service is authorized to detect and arrest any person who violates any of the laws of the United States relating to coins, obligations, and securities of the United States, including the investigation of the counterfeiting of U.S. currency. In addition, the Secret Service is authorized to identify, locate, and apprehend criminal organizations and individuals that target the nation’s critical financial infrastructure and payment systems. Secret Service special agents investigate financial crimes such as access device fraud (including credit and debit-card fraud); identity crimes and theft; business email compromise; bank fraud; and illicit financing operations. In addition, the agency investigates cybercrimes, including network intrusions, ransomware, and cryptocurrency, among other criminal offenses. The Secret Service also provides forensic and investigative assistance in support of investigations involving missing and exploited children. Finally, Secret Service special agents may investigate and make arrests for any offense against the United States committed in their presence, or any felony cognizable under the laws of the United States if they have reasonable grounds to believe that the person to be arrested has committed or is committing such felony. For more information on the evolution of the Secret Service’s statutory authorities, see appendix III. The Secret Service has established three phases for a special agent’s career, in which the special agent contributes to both investigative and protective operations—Phase 1: Career Entry/Field Office Assignment; Phase 2: Protective Assignment; and Phase 3: Post-Protective Field, Protection, or Headquarters Assignment. During Phase 1, after being hired and receiving 7 months of training, the special agent is assigned to a field office for at least 3 years, where the special agent performs investigations and participates in temporary protective assignments locally and away from the special agent’s home office. In Phase 2, the special agent is assigned for up to 8 years to a permanent protective detail or to one of the Secret Service’s specialty divisions, such as the Office of Strategic Intelligence and Information. In Phase 3, the special agent may return to a field office, serve in headquarters-based specialized roles, or continue permanent protection duty. Figure 4 illustrates the Secret Service’s special agent career progression model. Secret Service special agents are paid in accordance with the Office of Personnel Management’s general schedule, which determines the pay structure for the majority of civilian white-collar Federal employees. In addition to standard pay under the general schedule, special agents are eligible for law enforcement availability pay (LEAP). The Law Enforcement Availability Pay Act of 1994, as amended, established a uniform compensation system for federal criminal investigators who, by the nature of their duties, are often required to work excessive and unusual hours. The purpose of LEAP is to provide premium pay to criminal investigators to ensure their availability for unscheduled work in excess of a 40-hour workweek based on the needs of the employing agency. The LEAP Act authorized a 25 percent increase in base salary (LEAP premium pay) as long as specific requirements of the LEAP Act are met. Among these requirements is a condition that criminal investigators maintain an annual average of 2 or more unscheduled duty hours per workday. Federal employees under the general schedule are subject to caps on pay equal to the highest pay level in the general schedule. In recent years, legislation has been enacted to raise this pay cap for Secret Service special agents who, due to the high number of hours they worked, were not otherwise compensated for all hours worked. In 2016, the Overtime Pay for Protective Services Act of 2016 authorized any officer, employee, or agent employed by the Secret Service who performs protective services for an individual or event protected by the Secret Service during 2016 to receive an exception to the limitation on certain premium pay within certain limits. The Secret Service Recruitment and Retention Act of 2018 extended the Secret Service-specific waiver of the pay cap for basic and premium overtime pay through 2018 and included agents within the Secret Service Uniformed Division. Subsequently, the Secret Service Overtime Pay Extension Act extended the Secret Service- specific waiver through 2020. The Secret Service’s Office of Investigations supports protective operations in a variety of ways. According to our analysis of Secret Service data, special agents assigned to the Office of Investigations expended 11.2 million hours supporting protective operations during fiscal years 2014 through 2018. These 11.2 million hours accounted for 41 percent of all protection hours recorded by Secret Service law enforcement personnel during that period. Figure 5 shows the number of hours Secret Service law enforcement personnel expended on protection, including the percentage expended by special agents in the Office of Investigations. The Office of Investigations conducts numerous tasks in support of protective operations, including temporary protective assignments, protective intelligence investigations, and critical systems protection. Temporary protective assignments. When a Secret Service protectee travels, special agents in the Office of Investigations carry out numerous tasks, on a temporary basis, to assist the agency’s protective operations. These special agents facilitate preparations for a protectee visit and safeguard locations. For example, special agents may review the vulnerabilities of a site, conduct motorcade route planning, and coordinate with special agents on the permanent protective detail and with state and local law enforcement. In addition, these special agents provide physical protection when the protectee arrives. Special agents assigned to the Office of Investigations also travel to provide temporary protection and assist during presidential campaigns and National Special Security Events. During presidential campaigns, these special agents may accompany certain presidential candidates and their family members to provide 24/7 protection, and may also work on advance teams that provide site security for campaign events. Protective intelligence investigations. The Office of Investigations assists with the agency’s protective intelligence efforts by investigating threats against protected persons, including the President, and protected facilities, such as protectee residences. According to a Senior Secret Service official, special agents in the Office of Investigations locate, interview, and monitor individuals that make threats to a protectee. In fiscal year 2018, the Secret Service opened 2,011 protective intelligence investigations. Critical systems protection. The Critical Systems Protection program identifies, assesses, and mitigates risk posed by information systems to persons and facilities protected by the Secret Service. The program is coordinated by special agents in the Office of Investigations, and according to a Senior Secret Service official, the program draws on the investigative experience that special agents have developed in the Office of Investigations. For example, the official told us that, through the Critical Systems Protection program, the agency may monitor electronic systems that could be compromised in a hotel where a protectee is staying. The Office of Investigations can provide other benefits to protective operations, such as providing support during periods of increased protection demand and, according to special agents we interviewed, developing relationships with local law enforcement that assist with protective operations. Below are examples of these potential benefits. Support during periods of increased protection demand. The Office of Investigations can shift the focus of its special agents from investigations to protection during periods of increased protection demand. For example, according to Secret Service officials, in fiscal year 2016, the Office of Investigations shifted special agents from criminal investigations to help meet the additional protection demands of the 2016 Presidential Campaign. As shown in figure 6, in fiscal year 2014 special agents assigned to the Office of Investigations spent 52 percent of their time on investigations and 39 percent on protection. These percentages shifted to 31 percent on investigations and 58 percent on protection in fiscal year 2016. Secret Service officials told us that the percentage of hours that special agents spent on protection remained elevated after fiscal year 2016 due to protection demands associated with the President and his family. Pre-established state and local relationships. Resources and support from local law enforcement are needed for the Secret Service to carry out its protective operations, according to senior Secret Service officials. In our interviews with 40 current and former special agents, 38 reported that Secret Service personnel develop relationships with state and local law enforcement while conducting investigations, and that these relationships can benefit protective operations. Twenty-two special agents noted that contacts with state and local law enforcement are pre-established as a result of the agency’s investigative operations. Twenty special agents reported that assets or resources are more readily provided by their state and local partners because of the relationships they have built. In addition, special agents said that relationships developed with state and local law enforcement are either necessary for (11 special agents) or improves (8 special agents) the Secret Service’s protective activities. This is consistent with our prior reporting on the topic. Specifically, in our February 2016 review of Secret Service field offices, we reported that special agents in each of the 12 domestic offices we interviewed emphasized that it would not be possible to protect visiting dignitaries without extensive assistance from state and local law enforcement partners. For example, state and local law enforcement partners may provide equipment such as helicopters, vehicles, and communication equipment during dignitary visits. Supports employee retention and work-life balance. Secret Service officials told us that special agents generally cannot work protective assignments for their entire career, and that investigations help support a more reasonable work-life balance for special agents. A senior Secret Service official described that protective assignments require a high level of readiness and threat consciousness, which can lead to significant psychological stress that cannot be sustained for a 25-year career. Another Secret Service official told us that some special agents can spend 100 or 200 nights away from home per year on protective assignments, and that some special agents do not want to work on protection full-time. Seventy-five percent (30 of 40) of the special agents we interviewed reported that their work-life balance is better while working on an investigation versus a protective assignment. For example, eighteen special agents reported that investigative operations have more normal working hours than protective operations. Special agents also reported that working protective operations requires that they spend more time away from home than investigations (12 special agents) and requires a work schedule dictated by someone else’s (i.e., the protectee’s) schedule (14 special agents). Most special agents we interviewed did not report any instances where they were unable to fulfill a protective assignment due to investigative demands. Of the 40 special agents we interviewed, 35 said there had never been an instance in which they were unable to fully execute a protection-related assignment as a result of their investigative responsibilities. The five special agents who said there were instances in which they could not personally serve in an assignment reported an issue related to staffing. For example, a special agent would have been assigned to a temporary protective activity, but they already had an investigative commitment (e.g., serving as a trial witness). According to Secret Service officials, in these instances special agents are replaced before the protective assignment begins, and thus, there is no negative effect on protective operations. During the course of our interviews, 23 special agents said that during the last two years they frequently or sometimes were required to work on investigations while they were assigned to temporary protective operations. Examples provided by these special agents included working on investigations during protective shifts, before and after protective shifts, and during breaks to pursue investigative leads and respond to U.S. Attorneys. Additional examples associated with this topic are sensitive and have been omitted from this report. These statements are consistent with those expressed in an August 2016 report assessing quality-of-life issues at the Secret Service. Senior Secret Service officials told us that investigations can help prepare Phase 1 special agents for the protective responsibilities required in Phase 2 of their career, which includes an assignment to a permanent protective detail or a specialty division (e.g., counter-assault team). However, the agency has not identified which types of investigations and related activities best prepare special agents for Phase 2, or established a framework to help ensure Phase 1 special agents work on such cases and activities to the extent possible. As described earlier, special agents typically start their careers as Phase 1 special agents in a field office, and work on criminal investigations. Twenty-six of the 40 current and former special agents we interviewed reported that investigations are important in developing the skills necessary for protective assignments. Special agents we interviewed offered examples of skills developed, such as communication, interviewing, and operational planning skills; greater attention to detail; and experience working with law enforcement partners. Special agents further stated that certain types of investigations can offer more skill development opportunities than other types of investigations. For example, 18 special agents we interviewed reported that working on protective intelligence cases can help prepare special agents for protective operations. A senior official in the Office of Protective Operations agreed, and told us that experience with protective intelligence investigations allows special agents to gain insight into both the protectees and the threats against them. In addition, six special agents identified cyber investigations as helping prepare special agents for protective operations. However, 15 special agents reported a type of Secret Service investigation that does not help them develop protection skills. For example, nine special agents said financial crime investigations (e.g., credit card fraud) are not helpful in preparing special agents for protection. As one special agent described, the skills developed from financial investigations do not translate to protection. Similarly, five special agents said that investigations into counterfeiting are not helpful in preparing special agents for protection. The Secret Service’s December 2017 Office of Investigations Priorities and Roadmap states that the office must continually look to identify areas where the expertise it has developed for investigative purposes can be leveraged to advance the Secret Service’s ability to perform its protective responsibilities. In addition, consistent with Standards for Internal Control in the Federal Government, effective management of the Secret Service’s workforce is essential to achieving results, as is continually assessing knowledge, skill, and ability needs of the organization, and establishing training aimed at developing and retaining employee knowledge, skills, and abilities to meet changing organizational needs. Further, according to leading management practices related to training and development efforts, adequate planning allows agencies to establish priorities and determine the best ways to leverage investments to improve performance. However, Secret Service officials told us the agency has not identified which of its current types of criminal investigations and related activities best prepare special agents for protective responsibilities, nor has it established a framework to help ensure that Phase 1 special agents gain experience in those areas to the extent possible. According to Secret Service officials, a list of investigative experiences beneficial to protective assignments existed in the past; however, the list is no longer used in practice and a copy of the list no longer exists. Special agents we interviewed reported that certain types of investigations (e.g., protective intelligence investigations) are more helpful than others in preparing them for protective assignments. Secret Service officials agreed that identifying the types of investigations and activities that best prepare special agents for protective responsibilities, as well as developing a framework to help ensure Phase 1 special agents have the opportunity to work on such cases to the extent possible, could help better prepare their special agents for the protective responsibilities required in Phase 2 of their careers. In addition, a framework could better support the Secret Service’s protective operations by focusing Phase 1 training on building skills needed for successfully executing protective responsibilities. It could also help make Phase 1 special agents more readily available to assist the agency when faced with a surge in protective responsibilities. Types of financial crimes most often prosecuted by U.S. Attorneys based on Secret Service referrals during fiscal years 2014 through 2018 were similarly investigated by four additional federal law enforcement agencies, including the FBI, Homeland Security Investigation, IRS Criminal Investigation, and the U.S. Postal Inspections Service. As shown in figure 7 below, the selected agencies served as lead investigators in a total of 14,669 prosecuted cases across six financial crimes offense types during fiscal years 2014 through 2018, with Secret Service serving as the lead on 31 percent (4,620) of the cases. The Secret Service served as the lead investigating agency on more counterfeiting and forgery, identity theft, and aggravated identity theft cases prosecuted by U.S. Attorneys than any of the other selected law enforcement agencies during fiscal years 2014 through 2018. For example, the Secret Service served as the lead investigative agency on 1,368 counterfeiting and forgery cases that were prosecuted during this time period, while the FBI led 66 cases and IRS Criminal Investigations led six cases that were prosecuted (see figure 7). Although Secret Service was the lead investigative agency on the vast majority of counterfeiting and forgery prosecutions compared to the selected agencies, some types of cases were more evenly divided among the selected agencies. For example, between 2014 and 2018, U.S. Attorney’s Offices prosecuted 608 aggravated identity theft cases for which the Secret Service was the lead investigating agency, while the FBI led 484 prosecuted cases, U.S. Postal Inspections Service led 454 prosecuted cases, and IRS Criminal Investigations led 383 prosecuted cases. All 12 of the federal prosecutors we interviewed told us that the benefits of the Secret Service and selected agencies investigating similar crimes outweigh the drawbacks. These prosecutors highlighted the following three benefits: (1) additional staff resources; (2) agency-specific expertise; and (3) value added by having agencies work together on cases. For instance, three federal prosecutors we interviewed said that the occurrence of financial and cybercrimes in their district was pervasive, and that the number of criminal complaints they received far exceeded the number of federal agents available to investigate. With regard to agency-specific expertise, one federal prosecutor noted that although multiple agencies may conduct counterfeiting investigations, the Secret Service has expertise in this area that is appreciated by local businesses, such as casinos. Finally, agency collaboration can benefit criminal investigations, as in a June 2018 case in which the Department of Justice announced a coordinated effort to disrupt schemes designed to intercept and hijack wire transfers from businesses and individuals. The effort included an investigation by Secret Service and the FBI in which 23 individuals were charged in the Southern District of Florida with laundering at least $10 million. In addition, although the Secret Service and selected federal agencies can investigate similar crimes, federal prosecutors told us that federal agencies prioritize different types of crimes or cases. For example, eleven federal prosecutors told us that the Secret Service was the only agency that referred counterfeiting cases to their district, and 6 federal prosecutors said the Secret Service was the only agency that referred protective intelligence or threat cases. Further, according to senior FBI officials, they generally investigate large-scale financial crimes. On the other hand, the Secret Service may be willing to investigate financial crimes with smaller losses than the FBI, according to senior FBI officials and two federal prosecutors we spoke with. Table 1 below includes the mission and investigative priorities of the Secret Service and selected federal agencies. Although nine of 12 federal prosecutors we interviewed stated that there are no drawbacks to the Secret Service investigating crimes similar to those investigated by selected federal agencies, two of 12 federal prosecutors and one federal agency official identified drawbacks related to deconfliction and case assignment. Specifically, one prosecutor told us that, in the past, there was a greater need for deconfliction between the Secret Service and the FBI, but that deconfliction had not been an issue in the last 18 months. In addition, FBI officials in one field office told us that although the Secret Service and the FBI generally coordinated and worked well together, sometimes there were instances in which they could have deconflicted earlier in an investigation. Another federal prosecutor told us that it may be difficult to know what federal law enforcement agency would be best to assign an investigation since in the early stages of an investigation, the federal prosecutor’s office may lack adequate case information to know what law enforcement agency would be best positioned to conduct an investigation. In December 2017, the Secret Service released the Office of Investigations Priorities and Roadmap (Roadmap). The Roadmap states that fiscal constraints require that the agency prioritize its efforts and take steps to ensure that resources are aligned with its criminal investigative priorities. It further states that the Secret Service will align enterprise-wide investigative activities from independent or uncoordinated cases into a systematic, well-prioritized, and targeted operation to counter the networks of transnational criminals that present risks to financial and payment systems. Towards this effort, the Roadmap states that the Office of Investigations will “counter the most significant criminal threats to the financial and payment systems of the United States through criminal investigations,” and that these investigations will focus on three priority criminal threats: Criminal activity with significant economic and financial impacts to the United States. Criminal activity, such as cybersecurity threats, that operate at scale and present emergent or systemic risks to financial and payments systems. Transnational criminal activity involving corruption, illicit finance, fraud, money laundering, and other financial crimes. To implement the Roadmap, the Office of Investigations was to identify investigative targets, such as specific criminal networks or activities, and develop campaign plans for each investigative target. As described in the Roadmap, the campaign plans were to synchronize the efforts of the Secret Service to counter the targets. They were also to identify government and non-government partners for countering investigative targets. In addition, the campaign plans to counter the most significant criminal threats to the financial and payment systems of the United States were to be reviewed, updated, discontinued, or newly developed on an annual basis. The Secret Service has not, however, employed the practices as identified in the Roadmap because, according to Office of Investigations officials, the approach outlined in the Roadmap is not beneficial given the dynamic nature of the crimes they investigate. Instead, rather than identifying investigative targets based on the most significant threats on a yearly basis and developing campaign plans for each target as originally planned, Secret Service officials report that their Global Investigations Operations Center helps identify individual cases with national significance and coordinate resources necessary to investigate these cases throughout the year. In addition, every two weeks Office of Investigations leadership meets with field office management to discuss their significant cases, including discussions about resource demands for these cases. However, available documentation of efforts taken does not consistently demonstrate synchronized efforts across the agency to counter investigative targets, as envisioned in the Roadmap. This is in part because the process for identifying cases with national significance and coordinating related resources is not documented. The Office of Investigations provided us with campaign plans it developed since the Roadmap was released, and based on our review, there were inconsistencies in the type of information provided. For example, one campaign plan identified gas station pumps that may have been compromised by skimming devices—that is, devices that steal credit card related information. The plan also identified field offices responsible for executing investigations of the gas pumps, timeframes for the investigations, and potential partners. A different campaign plan was an informational alert regarding business email compromises, including details about how the attacks are executed and examples of information the attacker is attempting to steal. However, this plan did not identify offices responsible for combatting the attacks, timeframes, or potential partners. The plan also does not specify what resources would be necessary to combat the identified threat. The Roadmap states that fiscal constraints require the Secret Service to prioritize its efforts and take steps to ensure that resources are aligned with its priorities. This is consistent with the recommendation of an independent panel established by the Secretary of Homeland Security to assess the Secret Service’s operations, which in 2014 recommended that the Secret Service “clearly communicate agency priorities, give effect to those priorities through its actions, and align its operations with its priorities.” Further, Standards for Internal Control in the Federal Government require that management should implement control activities through policies and define objectives clearly. This involves clearly defining what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for realizing the achievement. Documenting a process to ensure the Office of Investigations dedicates resources to priority criminal threats can assist the Secret Service in combatting these threats and ensuring that resources align with its priorities. In addition, the documented process can help ensure that plans for addressing priority criminal threats consistently include key information, such as offices responsible for combatting specific priority criminal threats, timeframes for actions to be taken, potential partners, and resources necessary to combat the identified threat. The Roadmap identifies three priority criminal threats to the U.S. financial and payment systems. However, according to Secret Service officials, the agency does not have a process for identifying cases that address priority criminal threats. In addition, the agency does not collect data on the related expended resources, according to Secret Service officials. Secret Service officials told us they maintain a significant case database, which holds information about individual cases that field office management determine to be significant. However, Secret Service officials told us the significant case database does not currently have the capability to identify whether a case addresses one of the three priority criminal threats, and acknowledged that the criteria of a significant case differ from the criteria of a priority threat outlined in the Roadmap. For example, as stated in the significant case database guidance, “significant cases are those that represent a significant economic or community impact, as well as those that involve multi-jurisdictional districts or schemes that employ emerging technologies.” However, as described earlier in this report, the Roadmap identifies three priority criminal threats, one of which is described as “criminal activity, such as cybersecurity threats, that operate at scale and present emergent or systemic risks to financial and payments systems.” Standards for Internal Control in the Federal Government states that relevant, reliable, and timely information is needed throughout an agency in order to achieve its objectives. However, the Secret Service does not have a systematic process for identifying cases that address priority criminal threats or the related expended resources, according to agency officials. As a result, Office of Investigations management and senior Secret Service officials lack complete information on the number of criminal investigations and amount of resources expended agencywide to investigate the agency’s priority criminal threats. Until the agency identifies investigations that address each priority criminal threat and the related resources, Office of Investigations management and senior-level Secret Service officials will not know the extent to which its operations are aligned with the stated priorities. Capturing and analyzing this data could help inform future decisions on how to allocate resources for addressing priority criminal threats. Since 2017, the Office of Investigations has employed a staffing model to determine how many special agents are necessary to sustain protective and investigative operations in its field offices. The staffing model takes into account the number of hours special agents are expected to work under LEAP and standard overtime, but does not consider annual caps on federal employee salaries. According to the Secret Service’s Human Capital Strategic Plan for Fiscal Years 2018 through 2025, the special agent staffing model is used to analyze the protective workload of the field offices. In addition, the plan stated that the model is used to determine the appropriate levels of investigative and intelligence output while keeping travel and overtime at “tolerable levels.” To fulfill the requirements to qualify for LEAP, Secret Service special agents regularly work a 10-hour day, inclusive of 2 hours of LEAP premium pay, for an annual total of 520 hours beyond the standard work year of 2,080 hours. The Office of Investigations staffing model also assumes special agents will work an estimated standard overtime of 200 hours, among other hours. As a result, the staffing model assumes that each special agent will work an estimated 2,600 hours per year. See Figure 8. However, if certain special agents work the hours projected under the staffing model, they may not be compensated for all of their work time because they may exceed the annual caps on federal employee salaries. For example, in calendar year 2018, using the Secret Service’s pay scale for the Washington, D.C. metro area, the standard pay cap was $164,200. Special agents at pay grade GS 13 Step 9 would have lost compensation if, in addition to their regular hours, they worked 520 hours of LEAP and 200 hours of standard overtime (see table 2). Special agents at pay grade GS 14 Step 6 would have lost compensation if, in addition to their regular hours, they worked 520 hours of LEAP alone. Although legislation was enacted in recent years to address compensation for Secret Service special agents by temporarily raising the pay cap, special agents at higher pay levels may still exceed the temporary pay cap under the current staffing model. For instance, under the temporary cap implemented for fiscal years 2017 and 2018, special agents at the GS 15 Step 5 pay grade would have been uncompensated for some hours if they worked the hours projected under the staffing model. See table 2 for additional details. According to data received from the Secret Service, some special agents did work time that was uncompensated despite the pay cap waivers. In calendar years 2016 through 2018, between 8 and 80 special agents assigned to the Office of Investigations worked some hours without being compensated for their time each year. This resulted in more than $1 million in lost wages (see table 4). Without the pay cap waiver, between 426 and 819 special agents would have worked some hours without being compensated for their time, which would have resulted in a total of $15.4 million in lost wages. See Table 3 for more details. Due to the limits on special agent compensation, the Office of Investigation’s special agent staffing model currently plans for individuals to work hours for which they cannot be compensated. Without adjusting its staffing model to ensure compensation limits are accounted for when estimating staffing needs, certain Secret Service special agents will continue to be under-compensated for their work. Additionally, the Secret Service-specific waiver does not apply after 2020, at which point special agents in the Office of Investigations may further exceed the pay caps and work some hours without compensation. Standards for Internal Control in the Federal Government states that management should design control activities to achieve objectives and respond to risks, such as those related to the management of human capital and the entity’s workforce. Internal control standards also call for the consideration of excessive pressures, noting that excessive pressure can result in personnel “cutting corners” to meet the established goals, and that management can adjust excessive pressures using tools such as rebalancing workloads. The standards further state that management should recruit, develop, and retain competent personnel to achieve the entity’s objectives. Retention can be pursued by, among other things, providing incentives to motivate and reinforce expected levels of performance and desired conduct among staff. Working long hours without being fully compensated may cause special agents to be less focused when providing protection or to seek employment elsewhere. Because the Secret Service’s staffing model does not consider maximum pay cap allowances, the Secret Service will continue to overestimate the number of hours each special agent should work and underestimate the number of staff needed to meet its workload demands. In addition, maximum pay cap allowances are subject to change if legislation does not continue to increase them on an annual basis. As a result, absent developing an updated staffing model that accounts for compensation limits and using that model to estimate staffing needs, the Secret Service risks special agents continuing to work some hours without compensation, and continuing to underestimate staffing needs. The Secret Service plays a critical role in safeguarding both the leadership of the United States and its financial resources. The Secret Service’s Office of Investigations provides valuable support to its protective operations, such as by conducting protective intelligence investigations, building special agents’ protection skills, and allowing the agency the flexibility to shift special agents from investigations to protection in campaign years and other protection-heavy periods. However, the Secret Service could better leverage its investigative responsibilities for supporting protective operations by identifying the types of investigative activities that best prepare special agents for protection, and developing a framework to help ensure special agents participate in those activities to the extent possible. In addition, selected federal prosecutors reported that the Secret Service’s financial investigations are helpful to the law enforcement community as a whole, bringing specialized expertise to investigations and complementing investigations performed by other federal law enforcement agencies. However, although the Secret Service has identified priority criminal threats in its Roadmap, it has not employed the actions identified in its Roadmap to pursue these threats. Rather, the agency relies on its Global Investigations Operations Center to identify individual cases with national significance and coordinate resources because, according to current Office of Investigations officials, the approach outlined in the Roadmap is not beneficial given the dynamic nature of the crimes they investigate. Documenting the process of identifying priority criminal threats and developing campaign plans would help the agency better direct investigative resources towards priority criminal threats. In addition, until the Secret Service identifies cases that address priority criminal threats and captures data on resources used, agency management will not be able to determine the extent to which resources and operations are aligned with priority criminal threats. Finally, special agents can work long hours in carrying out their investigative and protective duties. Unless the Secret Service updates its staffing model to account for compensation limits, the agency risks continuing to underestimate staffing needs and having special agents work some hours without compensation. This could affect retention, potentially weakening the agency’s ability to provide the highest level of quality protection. We are making the following six recommendations to the Secret Service: The Director of the Secret Service should identify which types of investigations and activities best prepare special agents for protective responsibilities. (Recommendation 1) The Director of the Secret Service should develop a framework to help ensure special agents have an opportunity to work, to the extent possible, investigations and activities that best prepare them for protection. (Recommendation 2) The Director of the Secret Service should establish a documented process to ensure that Office of Investigations resources are aligned with priority criminal threats. The process should outline key information to be included in plans for addressing priority threats. (Recommendation 3) The Director of the Secret Service should identify investigations that address priority criminal threats agencywide and collect data on the resources expended to investigate the threats. (Recommendation 4) The Director of the Secret Service should revise its special agent staffing model to ensure compensation limits are accounted for when estimating staffing needs. (Recommendation 5) The Director of the Secret Service should, after revising the special agent staffing model, use the revised model to recalculate and estimate staffing needs. (Recommendation 6) We provided a draft of this report to DHS for review and comment. DHS provided written comments, which are reprinted in appendix IV, and technical comments, which we incorporated as appropriate. In its comments, Secret Service, through DHS, concurred with the six recommendations. In addition, in its written comments the Secret Service outlined steps to address the recommendations. With regard to identifying which types of investigations and activities best prepare special agents for protective responsibilities and establishing a framework to help ensure they have an opportunity to work on them, the Secret Service has established a pilot program to revise guidance on preparing special agents for protection. Upon completion of the pilot program in March 2020, the agency plans to revise a directive to give field office supervisors a framework for identifying key training and experiences to prepare special agents for protection. The agency anticipates the new directive being implemented by June 2020. The stated actions are an appropriate response to our recommendation that the Secret Service develop and implement a framework for preparing special agents for protective responsibilities. These actions, if implemented effectively, should address the intent of our first two recommendations. Regarding the establishment of a documented process to ensure that Office of Investigations resources are aligned with priority criminal threats, the Secret Service plans to replace its current guidance, the INV Priorities and Roadmap, with a new strategic document with the goal of better aligning resources to address priority threats by March 2020. Developing an effective strategic plan that sets goals and objectives and outlines effective and efficient operations necessary to fulfill those objectives is consistent with best practices. Likewise, making clear what information should be included in investigative plans for addressing these priority criminal threats will help the Secret Service ensure that its resources use will be aligned with the criminal threats the agency has identified as priorities. We will continue to monitor the Secret Service’s efforts in this area. To identify investigations that address priority criminal threats across the agency, the Office of Investigations intends to revise its internal policy to further define the role of the Global Investigative Operations Center (GIOC), including how the GIOC will identify and track investigations into priority criminal threats. The agency anticipates that these revisions will be published by March 2020. To collect data on the resources expended to address priority criminal threats, the Office of Investigations plans to consider new and additional data collection methodologies. The agency intends to have developed an analysis of the validity of its revised data aggregation methodology by September 2020. Finally, the Office of Investigations plans to address our recommendations related to its staffing model by working with the Office of Strategic Planning and Policy and the Office of Human Resources to revise the staffing model to ensure compensation limits are accounted for when estimating staffing needs. The Office of Investigations then intends to work with these offices and the Chief Financial Officer to use the revised model to recalculate staffing needs. As the Secret Service notes, this recalculation is likely to result in an increase to the number of special agents required for the agency to maintain its current level of investigative engagement. The agency intends to complete the revision of the staffing model by March 2020 and update staffing estimates by June 2020. We also provided the report to the Department of Justice (DOJ). The Executive Office of U.S. Attorneys (EOUSA), a component of the Department of Justice, provided written comments, which are reprinted in appendix IV. In its response, EOUSA, noted that it agreed with our statements that Secret Service is a valuable law enforcement partner in criminal investigations, particularly those related to counterfeit currency, cyber fraud, and identity theft. EOUSA further emphasized that Secret Service’s investigative mission is intrinsically valuable to federal law enforcement efforts. DOJ also provided technical comments, which we incorporated as appropriate. Finally, we provided the report to the Internal Revenue Service, which did not provide comments on the report. The U.S. Postal Service declined to review the public version of the report. We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the Attorney General of the United States, the Postmaster General of the United States, and the Commissioner of the Internal Revenue Service, as well as other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or GoodwinG@gao.gov. GAO staff who made key contributions to this report are listed in appendix V. This report addresses the following objectives: (1) how, if at all, do the U.S. Secret Service’s (Secret Service) investigative operations support or negatively affect its protective operations; (2) to what extent do the Secret Service and selected federal entities investigate similar financial crimes, and to what extent do selected federal prosecutors find this to be beneficial; (3) to what extent has the Secret Service developed a plan to combat its priority criminal threats; and (4) to what extent does the Office of Investigations’ staffing model ensures compensation limits are accounted for when estimating staffing needs. This is a public version of a sensitive GAO report that we issued in September 2019. Secret Service deemed some of the information in our September report as sensitive, which must be protected from public disclosure. Therefore, this report omits sensitive information on whether Secret Service’s investigative operations negatively affect its protective operations. Although the information provided in this report is more limited, the report addresses the same objectives as the sensitive report and uses the same methodology. To determine how the Secret Service’s investigative operations potentially support or negatively affect protective operations, we reviewed Secret Service policies and guidance, including those related to Office of Investigations roles and responsibilities, time and attendance, and training. For example, we reviewed the Secret Service’s December 2017 Office of Investigations Priorities and Roadmap (Roadmap) to assess whether the agency is leveraging the expertise it has developed for investigative purposes to advance special agents’ ability to perform protective responsibilities. We also analyzed Secret Service data for fiscal years 2014 through 2018. For example, we analyzed Secret Service time and attendance data to determine the number of hours special agents spent on investigation and protection activities. We focused on special agents in the Office of Investigations, as these personnel are responsible for conducting criminal investigations and temporary protective assignments. Further, the data we analyzed focused on special agents in a field location (e.g., field office or resident office), and thus did not include special agents at headquarters. We focused on field staff because that is how the agency captures and reports the hour-related data in its annual reporting. In addition, we analyzed data on the number of investigative cases opened and closed. We focused on fiscal years 2014 through 2018 as it was the most recent data available at the time of our review; included a fiscal year in which the Secret Service experienced the operational tempo of a presidential campaign (i.e., fiscal year 2016); and included data from two administrations. To assess the reliability of the data, we discussed with Secret Service officials how the data are entered and maintained in their Manhours Reporting System, which tracks special agent workload and tasks, and their Field Investigative Reporting System, which maintains data on field office staffing and investigations. In addition, we compared the data to recent Secret Service annual reports and congressional budget justifications, and inquired about any differences. We also reviewed the data for any obvious errors and anomalies. Based on our review of the data and related controls, we determined that the data were sufficiently reliable for the purposes of reporting the number of hours that special agents in the Office of Investigations expended on different activities and the number of cases opened and closed during fiscal years 2014 through 2018. We also interviewed Secret Service officials at headquarters and selected field offices. We selected office locations using the following criteria: highest number of criminal investigation and protection hours, diversity in types of offices, geographic diversity, and presence of other federal law enforcement agencies. In addition, we conducted semi-structured interviews with 40 current and former Secret Service special agents. Specifically, we randomly selected and interviewed 10 special agents from each of the Secret Service’s three career phases (30 special agents in total). We also interviewed 10 former special agents, including those that retired from the Secret Service and others that left the agency for other reasons. To select these 10 special agents, we asked special agents that we interviewed to recommend former special agents to participate in our study (i.e., snowball sampling) and contacted an association for former Secret Service personnel to help identify recently retired special agents. The information obtained from our interviews cannot be generalized across all current and former special agents; however, the information provided examples and perspectives on how investigative operations can support and negatively affect protective operations. To determine the extent to which the Secret Service and selected federal agencies conduct similar investigations, we analyzed federal prosecutor data from the Legal Information Office Network System (LIONS)—a system maintained by the Department of Justice’s Executive Office for United States Attorneys. We analyzed the data to determine the number and types of cases referred by the Secret Service during fiscal years 2013 through 2017, the latest years for which data was available when making the determination. Specifically, based on our data analyses, we identified the six LIONS categories wherein Secret Service (1) was identified as the lead investigative agency by the US Attorney’s Office and (2) referred the highest number of financial crime cases to federal prosecutors during fiscal years 2013 through 2017. The categories were counterfeiting and forgery, other white collar crime/fraud, financial institution fraud, identity theft, aggravated identity theft, and other fraud against businesses. Next, we identified federal law enforcement agencies that referred the highest number of cases in these categories. Based on our data analyses, we selected the following four law enforcement agencies: the Federal Bureau of Investigation (FBI), the U.S. Postal Inspection Service (USPIS), Homeland Security Investigations (HSI), and Internal Revenue Service – Criminal Investigation (IRS-CI). In the course of our investigation, data from fiscal year 2018 became available, and we analyzed data from fiscal years 2014 through 2018 to determine the extent to which our selected federal law enforcement agencies referred similar types of cases to U.S. Attorney’s Offices as those referred by Secret Service. The information obtained from selected federal agencies cannot be generalized across all federal agencies. However, the information provides examples of how federal law enforcement agencies can conduct similar types of investigations. In addition, the data may not account for all financial crimes cases each agency contributed investigative resources to. This is because the data only includes cases referred by each investigative agency wherein the agency was identified as the lead investigative agency as determined by the U.S. Attorneys who entered the data into LIONS. To assess the reliability of the LIONS data, we discussed with Department of Justice officials how the data are entered and maintained in the system. We also reviewed the data for any obvious errors and anomalies. Based on our reviews and discussions, we determined that the data were sufficiently reliable for the purposes of describing the extent that selected federal law enforcement agencies referred financial crimes cases to federal prosecutors similar to those referred by the Secret Service during fiscal years 2014 through 2018. To help identify potential benefits and drawbacks of the Secret Service and selected federal agencies conducting similar types of investigations, we conducted interviews with officials from the selected federal agencies. Specifically, we interviewed officials at the headquarters and the Miami and New York field office locations for each selected agency in conjunction with site visits to Secret Service field offices in those areas. In addition, we conducted semi-structured interviews with one representative with a high-level understanding of the office’s activities (e.g., criminal chief) at 12 U.S. Attorney Offices (USAO). To select U.S. attorney districts, we established the following criteria to help ensure that we gathered a range of perspectives and interviewed USAOs that were likely to have experience working with Secret Service: highest number of ongoing cases of the types Secret Service investigates the most during fiscal years 2013 through 2017, size of USAO district (as designated by the Department of Justice), geographic diversity, and USAOs located in a state with a Secret Service field office. The information obtained from selected USAOs cannot be generalized across all federal prosecutors; however, the information provided examples of the benefits and drawbacks of selected federal agencies and the Secret Service conducting similar types of investigations. To determine the extent to which the Secret Service has developed a plan to combat its priority criminal threats, we reviewed Office of Investigations policies and guidance. For example, we reviewed the December 2017 Roadmap and guidance related to the Secret Service’s Significant Case Database. In addition, as discussed earlier, we interviewed officials from the Office of Investigations at Secret Service’s headquarters and selected field offices. We held discussions with agency officials to better understand whether the agency had a plan to address priority criminal threats and whether it maintained data on the number of cases that addressed priority criminal threats in fiscal years 2014 through 2018. We also reviewed Standards for Internal Control in the Federal Government to assess whether the Secret Service has the necessary control activities and information to combat its priority criminal threats and carry out its responsibilities. Finally, to understand how the Office of Investigations develops and uses its staffing model, we reviewed agency guidance documents including guidance governing personnel utilization; the Secret Service human resources manual; and the fiscal years 2018-2025 human capital strategic plan. We also received a briefing on the development and use of the Office of Investigations staffing model and the assumptions and statistical methods used in the staffing model from officials in the Office of Investigations. To describe the ways in which federal law affects special agent pay, we reviewed federal laws, such as the Law Enforcement Availability Pay Act of 1994, the Overtime Pay for Protective Services Act of 2016, and the Secret Service Recruitment and Retention Act of 2018. Finally, we reviewed data provided by the Office of Human Resources to determine the number of special agents assigned to the Office of Investigations in calendar years 2016 through 2018 that were not compensated for all the time worked in each calendar year and the total sum unpaid. We determined the data were reliable for the purposes of this report through interviews with officials and evaluations of the system from which the data was pulled. We also reviewed Standards for Internal Controls and previous GAO products to assess the potential effects of some special agents working without compensation. We conducted this performance audit from November 2017 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with Secret Service from October 2019 to January 2020 to prepare this version of the original sensitive report for public release. This public version was also prepared in accordance with these standards. From fiscal years 2014 through 2018, the U.S. Secret Service (Secret Service) expended $9.2 billion, with an average of $1.8 billion per fiscal year. Secret Service officials told us that in fiscal years 2017 and 2018, the Secret Service changed the way it collected and reported expenditure data. Specifically, Department of Homeland Security management directed all agency components to use the Common Appropriations Structure (CAS). As a result, the Secret Service implemented CAS in fiscal year 2017. In addition, the officials told us the Secret Service updated its accounting software in fiscal year 2018, resulting in additional changes to the accounting structure. Secret Service officials told us that because of these changes, it is not possible to accurately compare expenditure data across fiscal years 2014 through 2018. However, Secret Service officials noted that in the future they will be able compare year- over-year fiscal data starting with fiscal year 2018 and beyond using a tool within the new accounting system. A description of the expenditure data for fiscal years 2014 through 2018 is provided below. Secret Service officials told us that in fiscal years 2014 through 2016, expenditure data was collected and reported according to the task being performed. For example, a special agent’s salary was reported under the investigation category if the special agent was performing investigation- related tasks, and it was reported under the protection category if the special agent was performing protection-related tasks. See table 4. According to Secret Service officials, in fiscal year 2017, the agency implemented CAS and began to collect and report expenditure data according to location. For example, a special agent’s salary was reported under the investigation category if the special agent was assigned to an Office of Investigations field office even if the special agent was performing a protection-related task. See table 5. In fiscal year 2018, Secret Service transferred its financial reporting to the Oracle R12 system, which tracks data according to both location and task. In addition, officials noted that other accounting structure changes were made in 2018, such as changes to what activities were classified as protection. As a result, expenditures data from fiscal year 2018 is not comparable to fiscal years 2014 through 2017. See table 6. In 1865, the Secret Service was established by the Secretary of the Treasury for the purpose of investigating the counterfeiting of U.S. currency. Over the course of the next 50 years, the Secret Service’s role within the department continued to evolve as additional duties, such as Presidential protection, were assigned to it. During this time, the authorities exercised by the Secret Service were those delegated to it within the Department of the Treasury and, on occasion, authorities enacted through annual appropriations, which expired at the end of the applicable fiscal year. In 1916, the Secret Service received its first grant of authority enacted by permanent legislation— the Federal Farm Loan Act—which authorized the Secret Service to investigate counterfeiting, embezzlement, fraud, and certain other offenses in the federal farm loan system. Ten years later, the Secret Service received another grant of authority to investigate the counterfeiting of government requests for transportation by common carrier. Later, the Banking Act of 1933 and its 1935 amendments charged the Secret Service with investigating offenses similar to those under the Federal Farm Loan Act, but as applied to the Federal Deposit Insurance Corporation (FDIC). In 1948, the Secret Service’s investigative authorities under the above statutes were consolidated into a single provision of law, 18 U.S.C. § 3056 (“the Secret Service Statute”). However, the 1948 codification effort did not account for the investigative or protective activities that the Secret Service was authorized to perform under a delegation of authority or annual appropriations acts. The authorizing legislation for these activities came three year later, with the 1951 revision of the Secret Service Statute. As originally enacted, the Secret Service’s protective duties extended to the President and his immediate family, the President- elect, and, upon request, the Vice President. On the investigative side, the 1951 statute authorized the Secret Service to investigate any federal offense related to U.S. or foreign coins, obligations and securities, thereby expanding its jurisdiction beyond the enumerated offenses enacted in 1948. Over the next three decades, a series of amendments to the Secret Service Statute added new investigative and protective duties. In 1984, a revised version of the Secret Service Statute was enacted, which incorporated all prior amendments while adding a new investigative responsibility. Although there has not been another wholesale revision of the Secret Service Statute since 1984, subsequent amendments have further increased the Secret Service’s protective and investigative responsibilities. Under the current codification of its primary protective authorities, 18 U.S.C. § 3056(a), the Secret Service protects the President, the Vice President, the President-elect, and the Vice President-elect. The Secret Service may also provide protection, unless declined, to the immediate families of the President, the Vice President, the President-elect, and the Vice President-elect; former Presidents and their spouses for their lifetimes (unless the spouse remarries); children of a former President who are under 16 years of age; visiting heads of foreign states or foreign governments; other distinguished foreign visitors to the United States and official representatives of the United States performing special missions abroad when the President directs that such protection be provided; major Presidential and Vice Presidential candidates and, within 120 days of the general Presidential election, the spouses of such candidates; and, finally, former Vice Presidents, their spouses, and their children who are under 16 years of age, for a period of not more than six months after the date the former Vice President leaves office. Under the current codification of its primary investigative authorities, 18 U.S.C. § 3056(b), the Secret Service conducts criminal investigations in areas such as financial crimes, identity theft, counterfeiting of U.S. currency, computer fraud, computer-based attacks on banking, financial, and telecommunications infrastructure, and a wide range of financial and cybercrimes. In addition to investigating financial and electronic crimes, special agents conduct protective intelligence—investigating threats against protected persons, including the President, and protected facilities, such as protected residences. Table 7 provides a chronology of key statutes enacting protective and investigative authorities under the Secret Service Statute, 18 U.S.C. § 3056. Table 8 provides a cross-reference to enumerated offenses within the Secret Service’s investigative jurisdiction under 18 U.S.C. § 3056(b)(1) of the Secret Service Statute. “the Secret Service is authorized to detect and arrest any person who violates . . . section 508, 509, 510, 871, or 879 of this title or, with respect to the Federal Deposit Insurance Corporation, Federal land banks, and Federal land bank associations, section 213, 216, 433, 493, 657, 709, 1006, 1007, 1011, 1013, 1014, 1907, or 1909 of this title.” The enumerated offenses generally involve fraud, counterfeiting, embezzlement, and certain other misconduct in connection with government transportation requests, federal farm loans, and the Federal Deposit Insurance Corporation. Table 8 provides a brief description of each of the cited offenses. In addition to the contact named above, Joseph P. Cruz (Assistant Director), Jeffrey Fiore, Miriam Hill, Lerone Reid, and Leslie Stubbs made key contributions to this report. Also contributing to this report were Willie Commons III, Christine Davis, Eric Hauswirth, Susan Hsu, Grant Mallie, Claire Peachey, Farrah Stone, Eric Warren, and Sonya Vartivarian.", "summary": "Commonly known for protecting the President, the Secret Service also investigates financial and electronic crimes (e.g., counterfeit currency and identity theft). In recent years, Congress and a panel of experts established by the Secretary of Homeland Security have raised concerns that the Secret Service's investigative operations may negatively affect its protective operations. GAO was asked to review the Secret Service's investigative operations. This report examines, among other things, the extent to which the Secret Service's (1) investigative operations support or negatively affect its protective operations; (2) Office of Investigations has developed a plan to combat its priority criminal threats; and (3) staffing model accounts for federal employee compensation limits. GAO analyzed Secret Service data related to investigation and protection activities from 2014 through 2018; conducted semi-structured interviews with current and former special agents and federal prosecutors; and reviewed Secret Service policies and guidance. This is a public version of a sensitive report that GAO issued in September 2019. Information that the Secret Service deemed sensitive has been omitted. The operations of the U.S. Secret Service (Secret Service) Office of Investigations, which conducts criminal investigations into financial and electronic crimes, generally support Secret Service protective operations in a variety of ways. For example, special agents in the Office of Investigations perform temporary protective assignments, such as during presidential campaigns or augment protective operations by securing a site in advance of a visit by a protectee. GAO found that personnel in the Office of Investigations spent 11.2 million hours supporting protective operations from fiscal years 2014 through 2018. Most of the 40 current and former special agents GAO interviewed said that their investigative duties did not negatively affect protection. However, over half identified that they were frequently or sometimes required to work on investigations while assigned to temporary protective operations. Details associated with this topic are sensitive and have been omitted from this report. In December 2017, the Secret Service developed a plan to align its resources to combat what it identified as priority criminal threats (e.g., criminal activity with significant economic and financial impacts). However, available documentation of efforts taken does not consistently demonstrate synchronized efforts across the agency to counter the priority criminal threats, as envisioned in the plan. Further, the Secret Service does not have a systematic approach for identifying cases that address priority criminal threats. Absent a documented process for aligning resources and identifying cases, Secret Service will continue to lack assurance that its resources are aligned to combat its priority threats. The Office of Investigations employs a staffing model to determine how many special agents are needed in its field offices. The staffing model takes into account the number of law enforcement premium pay and standard overtime hours special agents are expected to work. However, it does not consider annual caps on federal employee salaries. As a result, the agency may be underestimating the number of staff needed to meet its workload demands. GAO is making six recommendations, including that the Secret Service establish a documented process to ensure that resources are dedicated to priority criminal threats, identify investigations that address these threats, and ensure compensation limits are accounted for when estimating staffing needs. The Department of Homeland Security concurred with each of GAO's recommendations.", "document_type": "gao"}
{"report": "Generally, agencies dispose of their excess property through GSA’s government-wide property disposal process. See figure 1. Disposal is facilitated by GSA’s disposal system, known as GSAXcess. Once an agency has determined that it no longer has an internal agency need for its property during agency internal screening, it generally declares and reports the property as excess. Subsequently, the agency places information on the property in GSAXcess and then other federal agencies can screen, request, and, if approved by GSA, obtain the excess property for their own use or can then provide it to an authorized non-federal recipient free of charge, minus transportation costs. If no federal agency (for its own use or use by its eligible non-federal recipient) requests the excess property from GSAXcess, it then becomes surplus to the federal government, and a State Agency for Surplus Property can request it and provide it to eligible non-federal entities in their state, such as local governments and non-profits. Property not claimed by a State Agency for Surplus Property can then be sold to the general public typically through a GSA auction or an approved sales center. Finally, unsold property may be abandoned and destroyed by the reporting agency. Agencies can provide property to non-federal recipients in various ways. Some agencies, such as USDA and DOE, have been granted their own independent authorities that allow them to provide their unneeded or excess property to eligible non-federal recipients, such as public entities and colleges or universities. Eligible recipients are determined by program requirements. Other agencies, such as DOL, predominately provide excess property to non-federal recipients through a grant, contract, or cooperative agreement. For our three selected agencies, we focused on how USDA and DOE provided property to non-federal recipients using their independent authorities and how DOL provided property to non-federal recipients through contracts. Table 1 describes these agencies’ programs that provide property to eligible non-federal recipients. More detail on the independent authorities used by USDA and DOE can be found in appendix II. Additional information about excess property previously provided by DOL through cooperative agreements to apprenticeship programs can be found in appendix III. Agencies with independent authorities and programs differ in when they are able to provide property to non-federal recipients. Such agencies can allow eligible non-federal recipients, as determined by their agency’s independent authority, to screen for and request unneeded property during agency internal screening; in other words, before the property is declared excess and available to other agencies and entities. For example, USDA is authorized to provide certain equipment to its contractors or recipients when doing so would further agricultural research or teaching objectives. Currently, USDA allows eligible non- federal recipients to screen for all USDA unneeded property through an USDA internal module in GSAXcess at the same time as its sub- agencies, and before the property is made available to federal agencies in GSAXcess. If there is no demand for the property by an eligible non- federal recipient during internal screening, it is then made available in GSAXcess, where other agencies screen for and request property for their own use or for use by associated non-federal recipients. Regardless of how agencies provide property to non-federal recipients, they are required to annually report to GSA on property they provided. GSA provides agencies with guidance to assist with their reporting responsibilities, including: GSA Bulletin Federal Management Regulation B-27: defines terms and provides agencies guidance on using GSA’s Personal Property reporting tool. GSA Personal Property Reporting Tool (reporting tool): a template used by federal agencies to report excess property provided to non-federal recipients. The reporting tool has pre-determined drop- down menu items for agencies to select from when reporting property provided to non-federal recipients, such as the authority used. Technical Assistance and Guidance: GSA officials told us that they provide training, technical assistance, and guidance through webinars, email, and phone when agencies seek additional information on reporting requirements. GSA publishes the information reported by agencies in its annual Non- Federal Recipient Report, which includes information such as the agency, non-federal recipient, authority used, and the original acquisition cost of the property. USDA, DOE, and DOL established agency regulations or guidance for managing the disposition of property during the internal-screening process and once it has been declared excess, including providing property to non-federal recipients, as described below. USDA has three separate Federal Excess Personal Property Program handbooks specific to each sub-agency within USDA that manages property provided to non-federal recipients under the Federal Agriculture Improvement and Reform (FAIR) Act, the Forest Service, and the National Institute of Food and Agriculture‘s (NIFA) Federal Excess Personal Property Programs. These handbooks describe the process through which eligible non-federal recipients can screen (i.e., search for and select) for unneeded and excess property. Specifically, USDA makes property available to non-federal recipients for each of these programs during internal screening at the same time that other USDA sub-agencies can screen the property. DOE officials and guidance explained how its offices should dispose of federal excess personal property, including when eligible non- federal recipients can screen for unneeded and excess property. DOE makes property available to non-federal recipients after internal agency screening once it is determined the property is not needed within DOE. For DOE’s Economic Development Property Program, DOE makes property available to the eligible Community Reuse Organization by word of mouth or through a DOE excess email listing. For the Math and Science Equipment Gift Program, the recipient is made aware of property by word of mouth or as a result of a subcontract that has ended with a university. For the Laboratory Equipment Donation Program, DOE extracts energy-related property from within the Energy Asset Disposal System and allows eligible non-federal recipients to screen for that property on an external website. During this screening period by non-federal recipients, if property is requested and the request is approved by DOE, DOE then transfers the property directly to the non-federal recipient. DOL policy explains how Job Corps contractors may directly access GSAXcess to obtain excess property. Specifically, it explains how contractors can screen and obtain excess property when it is made available to all federal agencies and other eligible non-federal recipients, generally on a first-come, first-served basis. Additionally, for USDA and DOE, if there is no demand for unneeded property among eligible non-federal recipients, the property is then declared as excess property and reported to GSA and becomes available in GSAXcess where it is made available to all other federal agencies and eligible non-federal entities. Through the various programs at our selected agencies, officials reported to us that they provided property with an original acquisition value of between $0.4 and $33 million to non-federal recipients through their agency-specific programs in fiscal year 2017, most of it through the Forest Service’s Federal Excess Personal Property Program. (See table 2). The three agencies we reviewed assigned various offices the responsibility for monitoring property provided to non-federal recipients. The program officials in charge of monitoring are to ensure, among other things, that non-federal recipients use the property within a reasonable period of time and for the purpose it was intended, according to agency regulations and program requirements. See figure 2. Once property is provided to a non-federal recipient some agencies retain title, or ownership, of the property, while others pass ownership to the recipient. For agencies disposing of property using the GSA-regulated disposal process, GSA regulations require agencies to, among other things: (1) ensure the use of excess personal property acquired for use by the non- federal recipient is authorized and complies with applicable federal regulations and agency guidelines, (2) review and approve transfer documents once property is requested by the non-federal recipient, and (3) ensure the non-federal recipient does not place the property into storage (i.e., stockpile) property and uses the property within a reasonable time frame. Requirements in the authorizing legislation govern USDA and DOE disposal when these agencies use their independent authorities. While monitoring responsibilities were assigned, these selected agencies reported and we found that property provided to non-federal recipients was sometimes disposed of prematurely, not used at all, or not used within the required time frames. For example: According to Office of Property and Fleet Management officials responsible for property provided under the FAIR Act, they conducted an unscheduled property compliance check at a non-federal recipient location that revealed that a non-federal recipient (i.e., a school) improperly sold property before USDA’s 1-year requirement to use the property was met. As a result, the school was put on probation and was required to send inventory reports to USDA on a regular basis. An official from a state forestry department we spoke to reported having obtained a large vehicle that was not used. Furthermore, this official told us that due to a lack of indoor storage space the vehicle was stored outside exposed to the elements and its condition deteriorated over time. According to Agricultural Research Service officials responsible for property provided under the NIFA Federal Excess Personal Property Program, they revoked the participation of a non-federal recipient (i.e., a college) that was unable to provide information on how or whether property was being used. Several Laboratory Equipment Donation Program recipients reported instances where they did not report required information at the end of the first year of use, according to program requirements. One recipient told us that it never used several pieces of equipment it received because they were in poor condition and put them in storage, rather than disposing of the property. Whether these instances are widespread or uncommon is unknown, due to a lack of consistent monitoring at USDA, DOE, and DOL to determine how and whether the property provided to non-federal recipients was used. USDA’s guidance from the Federal Excess Personal Property Program handbook for the FAIR Act specifies that regular audits and reviews of participating institutions are required to ensure property is being used in support of research, educational, technical, and scientific activities for related programs. Specifically, USDA requires property that is obtained by an institution to be placed into use for the purpose it was acquired within 1-year of receipt and to be used for 1- year thereafter. However, USDA Office of Property and Fleet Management officials told us that due to a limited travel budget and staff to conduct monitoring they relied on informal “spot checks” to monitor property provided to non-federal recipients under the FAIR Act. DOE’s Office of Asset Management said it had discontinued monitoring any excess property provided by the Economic Development Property program to non-federal recipients. According to DOE Office of Asset Management officials, they mistakenly believed the Economic Development Property authority had expired, and thus believed they were relieved of their monitoring responsibilities of the property provided to non-federal recipients. According to officials, they determined during the course of our review that the authority had not expired, but stated DOE regulations currently do not reference Economic Development Property. Officials stated that they did not know when they had last monitored the program and were not informed of its activities, even though between fiscal years 2013 and 2017, DOE reported to GSA’s Non-Federal Recipient Report that the program provided over $154 million in property to non-federal recipients. According to DOE Office of Asset Management officials, they were unaware of the DOE sites that reported this data to GSA. In addition, DOE has previously acknowledged monitoring concerns with the program. Office of Asset Management officials told us they are determining how use of this authority will continue in the future. As of December 2019, DOE’s Office of Asset Management had not issued any new guidance or clarifications on the program, or a time frame for when such guidance or clarification might be issued. DOE’s Office of Science told us it had not consistently monitored property provided to Laboratory Equipment Donation Program recipients to ensure that required information was reported at the end of the first year, which is a requirement of the program. According to three Laboratory Donation Equipment Program recipients we spoke with, they had never provided information to DOE, and DOE had not requested information on property they received. According to Office of Science officials, they had not regularly contacted Laboratory Donation Equipment Program recipients because the process for doing so had been manual, and therefore was unsustainable and led to poor record-keeping. In March 2019, Office of Science officials established a new platform that will generate automatic email notifications to non-federal recipients of Laboratory Equipment Donation Program property within 11 months of receipt. DOE officials told us that the new system started receiving applications in June 2019, and thus DOE will begin the automated notifications no later than May 2020. Within DOL, the National Property Officer for DOL’s Job Corps Program retired in December 2018 and the position has not been officially filled. In September 2019, DOL officials told us that the National Property Officer’s responsibilities—which include periodically reviewing policies, procedures, and excess property provided to Job Corps centers—are temporarily being filled by another employee, in addition to that employee’s other responsibilities. They do not expect to hire a full-time National Property Officer before the end of calendar year 2019. It is unclear to what extent monitoring activities have been conducted within the National Office in the absence of a full-time National Property Officer. We identified discrepancies between the data provided to us by Job Corps Program officials on the excess property provided to Job Corps centers, and the data maintained by the Job Corps centers we visited. For example, we identified items that had been provided to Job Corps centers that were not tracked in DOL’s internal property-management system. According to DOL officials, property under a certain dollar threshold is not tracked internally, a practice that might account for the discrepancies. However, we identified several items that were obtained by Job Corps centers that were over the dollar threshold set by DOL. For example, one Job Corps center we visited obtained two walk-through metal detectors that exceeded the dollar threshold but are missing from DOL’s Job Corps Program data. Offices within our selected agencies also did not fully carry out their oversight responsibilities. According to federal standards for internal control, management should evaluate performance and hold individuals accountable for their internal control responsibilities as well as internally communicate the necessary quality information to achieve the entity’s objectives. Specifically, effective oversight and communication with key stakeholders are essential in ensuring that management is held accountable for carrying out their internal control responsibilities and meeting agency objectives. However, we found that the selected agencies did not take steps, such as communicating information, to ensure that the non-federal recipient programs were carried out in accordance with the agency’s property management regulations or program requirements, for various reasons: At USDA, Office of Property and Fleet Management officials acknowledged they have not consistently provided oversight of personal property across USDA because it was not considered a priority within the agency to do so. For example, until USDA established an inventory-compliance metric, sub-agencies did not regularly conduct required property inventories, and Office of Property and Fleet Management officials lacked the ability to require them to do so. As another example, officials said they requested that an office within USDA reconcile its non-federal recipient reporting data and make changes to the report to be provided to GSA. However, the office did not respond to their request, and the Office of Property and Fleet Management did not have the ability to enforce any corrective actions not taken. These experiences signaled to the Office of Property and Fleet Management that this area was not an agency priority and limited the ability to conduct oversight. However, USDA’s Office of Property and Fleet Management officials conceded that more consistent and robust agency-wide oversight of property provided to non-federal recipients would provide them with a better understanding of the effectiveness of their property-management controls. At DOE, communication problems have interfered with oversight. The Office of Asset Management is responsible for communicating information and providing guidance on the agency’s property management regulations to ensure that program offices are carrying out their property programs in accordance with those regulations. However, according to Office of Science officials, they were unaware that the Laboratory Equipment Donation Program was included in DOE’s property management regulations, though they had seen manuals about the program referenced in other DOE guidance. In addition, according to Office of Science officials, in the absence of information about the Economic Development Program in DOE regulations, they were using DOE guidance that reflected DOE policy to provide property to non-federal recipients. However, the guidance used by Office of Science was discontinued in 2011 and, as mentioned above, is currently under review, according to Office of Asset Management officials. Office of Asset Management officials stated that not having official guidance that can be communicated to the sites about the use of this program is problematic and said they recognized the need for improved guidance and communication between the offices going forward. In addition to these issues, we have reported in the past that managing property in general has been a low priority for federal agencies. Consistent with this report, officials from our three selected agencies stated that it was not always cost-effective to prioritize the monitoring and oversight of property programs for various reasons. Some also reported that, given limited resources, they prioritized high-risk or high-dollar value property that was still in the federal government’s possession rather than low-risk or low-dollar valued property within or divested from federal agency possession. We recognize that higher value property still being used may require more robust monitoring. However, as described above, there are good reasons to pay attention to whether the property provided to non-federal recipients, such as schools and state foresters, is being used according to regulations and guidance—not the least of which, it collectively represents millions of dollars in federal resources. As we described above, our three selected agencies alone provided about $76 million in property to non-federal recipients in fiscal year 2017. Furthermore, agencies may consider the property low value, because they are no longer using it, but if that property, for example an old fire truck, keeps a federal or non-federal entity from purchasing expensive new parts, then it is not as clear that the value of the property is actually low. Finally, no matter the value of the property, agencies without effective oversight of the authorities and programs they are responsible for cannot be assured that they are adhering to federal regulations and meeting program requirements, including whether property is being used as intended or to its fullest extent. Officials’ at the three agencies we reviewed told us that providing unneeded or excess property to non-federal recipients was cost-effective for them or the federal government. For example, DOE officials reported that being able to dispose of property during internal screening helped them dispose of property more quickly than they would be able to do through GSAXcess and also reduced warehousing costs. USDA officials told us that being able to provide property to non-federal recipients potentially saves USDA on warehouse costs, but there are also likely additional savings since many of their non-federal recipients also obtain excess property from other federal agencies. DOL officials told us they save on contracting costs, as the Job Corps centers are able to obtain federal property for free, versus having to purchase similar property, whose costs could be built into contracts with federal agencies and paid for with federal funds. Officials at our selected agencies told us that distributing unneeded and excess property to non-federal recipients also enhances their mission. For example, a USDA official told us a goal of the Federal Excess Property Program under NIFA—as managed by the Agricultural Research Service—was to provide property to non-federal recipients to establish relationships between USDA and state agricultural schools and programs. The official told us there is also increased value to USDA from the partnerships in the program, including an increase in agricultural experimental work and cooperative educational programs that assist USDA. DOE Office of Science officials told us that providing the scientific equipment through the Laboratory Equipment Donation Program encourages colleges and universities to develop energy-related programs. In addition, officials told us the program encourages future scientists to potentially work for DOE in the future. DOL officials told us that providing property to Job Corps center contractors helps DOL provide job training for at-risk youth. All 17 non-federal recipients we spoke with told us that federal property received from the selected agencies was beneficial for their program or department as well. For example, one DOE Laboratory Equipment Donation Program recipient told us that the equipment received was used to furnish a teaching laboratory, which the recipient would not have otherwise been able to purchase due to a limited budget. A state forester told us that property received from the Forest Service’s Federal Excess Property Program (such as fire trucks, gloves, and electronics) has had a real positive effect on rural fire departments because they would otherwise have been unable to purchase these items due to limited budgets. Officials from a DOL Job Corps center told us that the property they obtained as excess from GSAXcess is a lifeline for their operations, as they were able to obtain a lot of dorm and kitchen equipment to assist with their operations. See figure 3 below for examples of equipment obtained by non-federal recipients. While the selected agencies and non-federal recipients report benefits, the agency-specific disposal programs and agreements used at our selected agencies and other agencies may not benefit all federal agencies or even non-federal recipients. As we describe in more detail later in this report, GSA does not have reliable data on the scope of property provided to non-federal recipients across the federal government. However, based on our discussions with GSA officials and other stakeholders, as well as our review of 2003 property utilization and donation study, when agencies use their independent authority, in some instances, other stakeholders may not be eligible to acquire the property. First, non-federal recipients can obtain property at multiple points in the disposal process, a factor that could mean potential recipients get several chances to obtain property. For example, when agencies, such as USDA and DOE, provide unneeded property to non-federal recipients, the property does not enter GSAXcess. Additionally, other federal agencies and State Agencies for Surplus Property may not be eligible recipients to obtain unneeded property. According to the GSA property utilization and donation study, the increase in laws providing agencies with independent authority to give property to non-federal recipients has reduced the remaining pool of assets that would have otherwise entered the government-wide property disposal cycle. Additionally, when property does enter GSAXcess, an agency may obtain the property and provide it to a non-federal recipient. While GSA officials said they prioritize giving the property to the federal agency that plans to use it for its own needs over a federal agency that plans to provide it to a non-federal recipient, GSA officials said they are not always aware of how federal agencies plan to use the property. In this respect, a federal agency may acquire the excess property for use by a non-federal recipient instead of a federal agency acquiring the property for its own use. GSA officials also told us that they did not have data on the amount of property that is provided to non-federal recipients at the various points of the disposal process. Thus, it is unknown how often non-federal recipients obtain excess property from a federal agency, and whether or how often other recipients that may want excess and surplus property are missing out on property. Figure 4 illustrates the reduction in property that can occur when non- federal recipients obtain property at various points in the disposal cycle. Second, because of the decentralized nature of disposal, some non- federal recipients could benefit more than others. For example, a rural fire department eligible to receive property under the USDA Forest Service’s Federal Excess Property Program could potentially obtain property: (1) during USDA internal screening, (2) from USDA as excess, or (3) through their State Agency for Surplus Property once the property is deemed surplus to federal government. Officials from four out of five State Agencies for Surplus Property told us that they have some recipients that are eligible to receive property through multiple points in the disposal process. In contrast, other non-federal entities, such as non-profit groups, may only be able to obtain property through their State Agency for Surplus Property because they are not eligible to receive property under a federal agency-specific program. As a result, these non-federal entities may have less property available to them and would have to pay a fee to the State Agency for Surplus Property to obtain the property. In addition, DOE and USDA officials said they do not advertise their agency- specific property programs, so a smaller pool of eligible recipients may be competing for and benefiting from the property over those that are unaware of those programs. For example, one Laboratory Equipment Donation Program recipient told us he became aware of the program through a previous mentor and would have not otherwise known about the program because it is not advertised. GSA’s reporting tool and accompanying bulletin are unclear, a lack of clarity that resulted in inconsistent data on the number of non-federal recipients obtaining property. As the reporting tool and bulletin serve as the primary means for ensuring consistent information is collected on non-federal recipients that are provided property, it is important that they accurately convey the information agencies should report. However, we found the following three issues made the data unreliable for reporting the amount of property provided to non-federal recipients through authorities and agency specific programs. We found that agencies incorrectly reported the authorities and programs used to provide excess property to non-federal recipients, making it difficult to understand how many agencies are providing property to non- federal recipients or what authority they are using to do it. Our analysis of the non-federal recipient reports found that during fiscal years 2013 to 2017, 16 agencies reported providing property to a non-federal recipient through various types of authorities, including agency-specific authorities. However, one of our selected agencies reported using another agency’s independent authority or program to provide excess property to a non-federal recipient. Specifically, in fiscal year 2016, we found five instances where DOL reported using a DOD independent authority. DOL and GSA officials told us these instances were likely the result of data entry errors. We also found that agencies reported information incorrectly under their own programs. The full extent of such errors in unclear due to the inconsistency and incompleteness of the data; however, we found clear examples of reporting errors that agency officials confirmed. As previously discussed, DOE reported providing $154 million in unneeded property to non-federal recipients through the Economic Development Property program, but DOE officials stated that they do not know if the data were accurate or complete, in part, because the officials were not aware the Economic Development Program existed and thus were not conducting any oversight at the time. DOE officials told us that they are taking steps to clarify when the Economic Development Property program should be used in reporting, and anticipate that the correct reporting will take place in fiscal year 2020 once clarification is complete. These errors occur because GSA’s reporting tool is limited. Specifically, the tool allows those who are inputting the information to select authorities and programs that are not specific to their agencies, rather than limiting options to the drop down menu of selections that actually are appropriate. GSA Office of Government-wide Policy officials told us that they provide a definition sheet, and offer training to each agency on how to enter data, but they are not sure if agencies are using their guidance. Even if those inputting data did refer to the sheet, GSA officials told us that since these are agency-specific programs, they are not aware of all the ways in which agencies are able to provide property to non-federal recipients and that the reporting tool may not reflect all the current authorities and programs used. A DOE official told us that the categories are not mutually exclusive, a situation that is confusing and can lead to inconsistent reporting even among offices within DOE. Because of this data input issue, it makes it difficult to understand how many agencies are providing property to non-federal recipients under these independent authorities. We found that agencies inconsistently reported loaned property provided to non-federal recipients, resulting in inaccurate government-wide data on the amount of loaned property. Our analysis of the data found that only DOE, among all reporting agencies, reported providing loaned property ($104 million) to non-federal recipients between fiscal year 2013 and 2017 to GSA’s Non-Federal Recipient Report. According to DOE’s property guidance, all excess property, including loaned property furnished to non-federal recipients should be reported. Conversely, USDA and DOL officials told us that they did not believe loaned property had to be reported, because title or government ownership of that property remained with the federal government. It is unclear based on GSA’s guidance and interviews with GSA officials whether loaned property should be reported by agencies. GSA’s guidance states that excess property furnished in any manner whatsoever, including loaned property, should be reported. The reporting tool seems to support the guidance, as it included loaned property in the drop-down menu from which agencies could select the mechanism used to provide property. However, GSA’s guidance does not specify the circumstances in which loaned property should be reported and how it may differ from property loaned under an agency-specific program. For example, we found that USDA reported providing property to non-federal recipients under its agency-specific Federal Excess Property Programs when the title or ownership remained with the federal government, but did not report providing any loaned property outside of its agency-specific programs. GSA officials stated that there might be confusion among some agencies about whether excess property loaned to non-federal recipients needs to be reported when ownership remains with the federal government. Because only one agency reported loaned property outside of agency-specific programs, GSA guidance may not clearly specify whether and how loaned property should be reported. We found inconsistencies in how property obtained by agencies in GSAXcess on behalf of non-federal recipients was reported, leading to underreporting of property provided to non-federal recipients. For example, we found that DOL was not reporting property obtained in GSAXcess for its Job Corps centers, because it believed that since this property was obtained from GSAXcess, GSA should be reporting these transactions. GSA officials told us DOL is responsible for reporting this information. According to GSA’s bulletin, agencies are required to report all of their transactions involving excess property provided to non-federal recipients, but do not need to report items sold, transferred, or donated by GSA on their behalf as part of the disposal process. Thus, there may be confusion among agencies on whether property obtained in GSAXcess should be reported by the agency or GSA. As a result, there could be undercounting of property provided to non-federal recipients, as neither DOL nor GSA is reporting the property. GSA Office of Government-wide Policy officials told us that they realize data reporting can be improved but do not have concrete plans in place to do so. For example, GSA officials told us they have identified changes to the reporting tool to make it more user-friendly and to address some of the features that lead to reporting errors. GSA officials provided us with documentation listing some changes they would like to make to the reporting tool, including incorporating a range of data checks that will trigger caution or error messages for inappropriate data entries, and to generate agency system reminders to ensure data are turned in by each agency. GSA officials told us they made some of these changes to the fiscal year 2019 reporting tool. These changes represent a potential step in the right direction. However, GSA has not established a plan with time frames to implement further changes. Moreover, based on the documentation provided to us, it is unclear whether the proposed changes will address some of the limitations we identified including (1) agencies’ reporting property under another agency’s program in the reporting tool, (2) whether loaned property should be reported by agencies, and (3) clarifying what property GSA is reporting on behalf of agencies. According to federal standards for internal control, it is important for management to periodically review policies, procedures, and related control activities for continued relevance and effectiveness in achieving the entity’s objectives or related risks. As we have shown, each of these limitations obscure data that would be helpful in understanding whether and to what extent property provided to non-federal recipients is done so at a cost to the federal government. Without addressing the limitations of the reporting tool and bulletin, it is not clear that the non-federal recipients’ report data will be consistent moving forward. Moreover, due to limited data, the implications of providing property to non-federal recipients ahead of other recipients, such as federal agencies and State Agencies for Surplus Property are unknown. Without taking action to update the reporting tool and bulletin to identify issues we found, it is unclear the extent to which GSA will be able to improve the data collected in the Non-Federal Recipient Report. By using GSA’s government-wide disposal process as well as independent agency authorities, agencies have an opportunity to be good stewards of government property by allowing others to reuse federal property in lieu of purchasing new property. While there are benefits to allowing agencies to provide property to non-federal recipients before others receive it, there are also potential implications. In the past, we have observed there is a government-wide lack of attention to management of property other than real property, and we continue to find that lack in this review. A full assessment of whether these efforts are achieving the intended effects are impeded due to a lack of oversight, monitoring, and accurate data about what types and amounts of property are provided to non-federal recipients. Until USDA, DOE, and DOL direct their offices to fulfill their oversight responsibilities, there may be an ongoing lack of accountability for managing such programs. Furthermore, lack of effective monitoring will continue to undermine any assurances to agencies and Congress that this property is being used in a timely manner, as intended, or to its fullest extent. Finally, given the large amount of property managed and disposed of by the federal government each year, the lack of reliable data makes it difficult to understand the overall scope of property provided to non-federal recipients and the implications for the government-wide disposal process. We are making seven recommendations: two recommendations to USDA, two recommendations to DOE, one recommendation to DOL, and two recommendations to GSA. The Secretary of Agriculture should direct the Office of Property and Fleet Management to consistently monitor property provided to non-federal recipients within 1 year of receipt, and to ensure property is being used for its intended purpose 1 year after initial monitoring. (Recommendation 1) The Secretary of Energy should direct the Office of Asset Management to resume monitoring the Economic Development Property program, including property provided to non-federal recipients. (Recommendation 2) The Secretary of Labor should direct the Employment and Training Administration to take steps, such as reconciling data between Job Corps centers and the Job Corps National Office, to ensure that the entities responsible for overseeing and monitoring the Job Corps Program have accurate data on the excess property provided to non-federal recipients. (Recommendation 3) The Secretary of Agriculture should direct the Office of Property and Fleet Management to establish clear processes to oversee property programs, including excess property provided to non-federal recipients across the agency. (Recommendation 4) The Secretary of Energy should direct the Office of Asset Management to update its regulations and guidance on programs that provide property to non-federal recipients to ensure regulations are current and establish a process to regularly communicate information about non-federal recipient programs to DOE program offices. (Recommendation 5) The GSA Administrator should direct the Office of Government-wide Policy to revise the Personal Property Reporting Tool by updating the authorities agencies can select. (Recommendation 6) The GSA Administrator should direct the Office of Government-wide Policy to document in what circumstances excess property loaned to non- federal recipients should be reported and what property GSA is reporting on behalf of agencies, for example, by updating GSA guidance. (Recommendation 7) We provided a draft of this report to USDA, DOE, DOL, and GSA for comment. Three agencies provided comments, which are reprinted in appendixes IV through VI and summarized below. USDA informed us by email that it had no comments and concurred with the recommendations. DOE also provided technical comments, which we incorporated, as appropriate. In its written comments, DOE agreed with our recommendations and stated that the Office of Asset Management will update the annual property reporting requirements for Economic Development Property and will also update DOE’s internal policies and provide property information on DOE’s internal informational website. In its written comments, DOL’s Employment and Training Administration agreed with our recommendation and stated that it will take steps to improve the accuracy of data on excess property provided to Job Corps contractors and has recently taken actions to improve the monitoring and oversight of Job Corps property. For example, the Employment and Training Administration stated it is working closely with DOL’s Office of the Assistant Secretary for Administration and Management to develop a new process for GSAXcess review and will formalize property reporting requirements, processes, and roles and responsibilities in the next update to its property management guidance. In its written comments, GSA agreed with our recommendations and stated that it already added relevant authorities to the Personal Property Reporting Tool in July 2019. In addition, GSA stated it will continue to contact agencies to ensure that all relevant authorities are included in the reporting tool and will evaluate technical updates to the reporting tool to ensure that agencies select an appropriate authority when reporting. Also, GSA stated it will communicate with agencies to clarify any confusion regarding reporting requirements for loaned property and is committed to reviewing and updating relevant regulations and guidance, particularly in terms of reporting property that agencies obtain via GSAXcess. We are sending copies of this report to the appropriate congressional committees, the GSA Administrator, Secretary of Agriculture, Secretary of Energy, Secretary of Labor, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions regarding this report, please contact Lori Rectanus at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VII. Our review focused on how federal agencies provide, manage, and report on property provided to non-federal recipients. Our objectives were to examine (1) how selected agencies manage unneeded and excess property provided to non-federal recipients, and (2) what is known about the benefits, effects, and reported data of providing property to non- federal recipients. To address both objectives, we reviewed applicable federal statutes and regulations pertaining to property disposal, including General Services Administration (GSA) property management regulations, and agencies’ independent authorities for providing property to non-federal recipients. We also reviewed GSA bulletins, briefings, and a 2003 GSA property utilization and donation study to understand the effects and requirements for providing and reporting property to non- federal recipients. To assess how selected agencies manage unneeded and excess property provided to non-federal recipients, we selected three agencies and reviewed documentation and interviewed officials from the three agencies—the United States Department of Agriculture (USDA), the Department of Energy (DOE), and the Department of Labor (DOL). We selected these agencies using information from GSA’s government-wide Non-Federal Recipient Report that provides data on excess property provided to non-federal recipients by agency, and reports from GSA’s centralized property database (GSAXcess) on overall property disposed of and obtained by federal agencies from fiscal year 2013 to 2017. After reviewing those reports, we selected agencies based on: (1) the amount of property provided to non-federal recipients in terms of original acquisition cost, (2) the amount of property obtained through GSAXcess in terms of original acquisition cost, (3) the number of independent authorities reported being used by the agency to provide property to a non-federal recipient, and (4) the amount of property provided to non- federal recipients through a grant, contract, or cooperative agreement. We selected these agencies based on these factors because we were looking for agencies that provided a large amount of property to non- federal recipients through their independent authorities and programs, as well as an agency that provided less property through the independent authorities and programs, and more through grants, contracts, or cooperative agreements. We reviewed each selected agency’s policies and program guidance describing disposal processes, including processes for providing unneeded and excess property to non-federal recipients, and compared the processes to relevant federal internal control standards on oversight and monitoring. We interviewed agency property management officials as well as agency program officials responsible for managing property provided to non-federal recipients through agency programs, including DOE’s Laboratory Equipment Donation Program, Economic Development Property program, and Math and Science Equipment Gift Program and three USDA Federal Excess Personal Property programs, including the Federal Agriculture Improvement and Reform (FAIR) Act program, the Forest Service Federal Excess Property Program, and the National Institute of Food and Agriculture Federal Excess Property Program to gain a high-level understanding of the impetus of the agency-specific disposal programs, and how those programs were managed. For DOL, officials told us that they currently provided property to non- federal recipients through contracts with DOL Job Corps centers and had previously provided property through cooperative agreements and memorandums of understanding with apprenticeship programs, but these agreements were canceled in 2016. Thus, we interviewed agency officials knowledgeable about excess property obtained through GSAXcess and provided through contracts to Job Corps Centers to understand how DOL provided property to non-federal recipients. More detail on the independent authorities used by agencies can be found in appendix II and additional information about excess property DOL previously provided to apprenticeship programs can be found in appendix III. To assess what is known about the benefits, effects, and reported data on providing property to non-federal recipients, we interviewed officials from State Agencies for Surplus Property in Arizona, California, Georgia, Illinois, and Texas to obtain their views on the GSA property disposal process. We selected these states because their State Agency for Surplus Property was a top 20 recipient of surplus property in terms of original acquisition value during a given year from fiscal year 2014 to fiscal year 2017, according to data provided by GSA on surplus property donation. We also interviewed and obtained documentation from 17 non- federal recipients in those five states to understand how they used unneeded and excess property provided by the USDA’s Forest Service Federal Excess Property Program, the DOE’s Laboratory Equipment Donation Program, and DOL’s Job Corps Program and how monitoring of federal property occurred. We selected these non-federal recipients because they obtained property from these three agencies through their independent authorities or agency programs. Information we obtained from these non-federal recipients is not generalizable to all non-federal recipients of excess property. In addition, we interviewed knowledgeable officials from GSA’s Office of Government-Wide Policy and Office of Personal Property Management. See table 1 for a list of federal agencies, non-federal recipients, and other stakeholders interviewed. We also analyzed and summarized Non-Federal Recipient Report data from fiscal year 2013 to 2017 to understand the scope of excess property that agencies provided to non-federal recipients through various programs and agreements. We used these years because this was the most current data available to us at the time we started our review. To assess the reliability of the Non-Federal Recipient Report data, we (1) performed electronic testing for obvious errors in accuracy and completeness; (2) reviewed GSA’s agency guidance on reporting requirements; and (3) interviewed officials at our selected agencies to discuss identified data errors. We found that information in the database was not sufficiently reliable for reporting the amount of property provided to non-federal recipients through independent authorities and programs. As discussed in the report, we used some of the data to provide illustrative examples of reporting errors and to develop recommendations for improving or establishing management controls to help ensure data quality. We conducted this performance audit from June 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings, and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. For several decades, the U.S. Department of Labor’s (DOL) Employment and Training Administration (ETA) has provided excess property to support apprenticeship training programs, according to DOL officials. For about 15 years, the Office of Apprenticeship within ETA had agreements with two apprenticeship programs—the International Union of Operating Engineers (IUOE) and the International Training Institute for the Sheet Metal Workers and Air Conditioning Industry (ITI) to support the training of apprentices in the fields of heavy equipment operation and maintenance and sheet metal fabrication and installation, respectively. According to the most recent agreements, DOL’s objective was to increase the number of women and minorities in apprenticeships. According to IUOE staff, these agreements supported equipment needs and hands-on training hours at 63 of 64 apprenticeship programs that provide training for construction-industry jobs, and according to ITI staff, property was obtained by its 150 training centers. Under GSA regulations, federal agencies, including DOL, can provide excess property to their grantees, contractors, and cooperatives. DOL executed cooperative agreements and memorandums of understanding with IUOE and ITI to provide excess property to support their apprenticeship training programs. According to DOL officials, the cooperative agreements and memorandums of understanding served as the legal instrument that laid out the relationship between ETA and the apprenticeship programs and the terms and conditions for obtaining excess property. The most recent memorandums of understanding between ETA and the apprenticeship programs were signed in August 2015 and were set to expire on December 31, 2020. IUOE and ITI representatives were provided access to view and request federal excess property in GSAXcess, the General Services Administration’s (GSA) government-wide, web-based system for facilitating the disposal of excess property. As authorized by DOL, IUOE and ITI representatives could screen property at the same time as other federal agencies. Once property was requested, the request would be reviewed and approved by DOL officials, certifying that the property fulfilled a mission-need for the particular site requesting the property. If GSA allocated the property to DOL, the federal agency disposing of the property would transfer the property directly to the training program or school that requested it; the particular training program or school was required to pay any associated transportation costs. Once the property was transferred, the training program or school was responsible for maintaining the property, which remained under the ownership of DOL, and IUOE and ITI were responsible for annually inventorying and certifying the property in their possession. When the property was no longer needed, it could be transferred to another site that needed the equipment or was disposed of by DOL’s listing the property in GSAXcess. There are no available data on the types or number of property that has been historically provided for apprenticeship program use. There are, however, data on what property is currently held by IUOE and ITI. According to DOL, as of September 2019, IUOE had over 2500 pieces of construction equipment and vehicles they obtained from GSAXcess between 1979 and 2017, while ITI had over 2000 pieces of property acquired from GSAXcess between 1999 and 2013. According to IUOE and ITI staff, this property was useful to the sites that received it because it provided training hours to apprentices and lead to cost savings, but challenges were cited in disposing of property when it was no longer needed. Training hours: according to IUOE staff, the equipment they obtained, while often dated, provided invaluable opportunities for apprentices to receive training hours on equipment they might not otherwise obtain. For example, according to IUOE staff, a training center in Michigan obtained a used crane that could cost $1 million to purchase new, and uses it at a dedicated area onsite to support various types of disaster response training activities. According to ITI staff, the property obtained by their schools included hand saws, drills, computers, and furniture. Cost savings: DOL officials and apprenticeship program staff said that the ability to obtain equipment in this fashion lead to cost savings. For example, according to IUOE staff, the property that was obtained through GSAXcess was a key element to fulfilling equipment needs for their programs, particularly for smaller programs that did not have as many resources. However, these sites have other options to obtain equipment, such as from the original equipment manufacturer or on the market. In addition, according to IUOE staff at the Casa Grande Training Center in Arizona, equipment obtained by the site was primarily heavy equipment and rolling stock used to train apprentices and saved the center money because they did not have to purchase new equipment. See figure 5 for an example of excess equipment obtained. ITI staff stated that the property they obtained to support the training of apprentices in their schools allowed the schools to spend funds on other program areas, rather than equipment. Out-of-date equipment: Many IUOE sites continue to use the equipment they obtained, but it is not all in working condition. For example, Casa Grande has some equipment that is no longer in working order and the site does not want to invest money to repair the equipment, if it can no longer use it. According to ITI staff, they have not obtained excess property from GSAXcess since 2013 and have been unable to dispose of property received under prior agreements with DOL that is no longer needed. For example, staff estimated that about 90 percent of the equipment they obtained is now obsolete (over 2,000 items) and they would like to dispose of it. At a school in Miami, Florida, ITI had to purchase additional storage to store obsolete property and classrooms were filled with obsolete computers. ITI schools currently fulfill their equipment needs through loans from ITI headquarters or through purchasing their own equipment. In 2016, DOL made the determination that it would no longer provide equipment to apprenticeship programs due to legal and policy concerns, and according to DOL officials, they dissolved the agreements with IUOE and ITI in October 2016. In August 2017, DOL sent letters to IUOE and ITI stating that DOL would no longer continue to furnish excess property to non-federal entities. In cancelling these agreements, the department said it no longer wanted to retain ownership of the equipment, nor did it have a mechanism to allow IUOE and ITI to retain the property. However, recently DOL has received independent authority to provide property to the apprenticeship programs. Specifically, in its fiscal year 2018 appropriations, DOL received independent statutory authority to provide up to $2 million in excess property to apprenticeship programs for purposes of training apprentices in those programs through grants, cooperative agreements, contracts, or other arrangements. DOL did not provide excess property to these programs during fiscal year 2018. In its fiscal year 2019 appropriations, DOL was again authorized to provide up to $2 million in excess property. According to DOL officials, they planned to use the authority to transfer ownership of property already in IUOE’s and ITI’s possession that the programs would like to keep in support of its apprenticeship training programs. In April and May 2019, DOL officials sent letters to IUOE and ITI requesting that the apprenticeship programs take steps to verify property currently in their possession. In addition, IUOE and ITI were required to identify property for which they would like to obtain ownership from DOL and provided instructions for applying the fair market value to this property. In September 2019, DOL approved the transfer of ownership of 96 items at a fair market value of about $1.7 million IUOE wished to retain and 75 items with a fair market value of about $216,000 ITI wished to retain, for a total of $1.9 million in the aggregate. For property that IUOE and ITI did not want to keep, including obsolete items discussed above, DOL is in the process of disposing of it using GSAXcess, according to DOL officials. DOL officials told us that DOL does not plan to transfer any additional property to apprenticeship training programs in the future because the authority provided in the fiscal year 2019 appropriations expired at the end of the fiscal year. In addition to the contact above, the following staff made key contributions in this report: Aisha Cabrer; Lacey Coppage; Nancy Lueke (Assistant Director); Joshua Ormond; Nitin Rao (Analyst-in-Charge); Amy Rosewarne; Kelly Rubin; Atiya Siddiqi; and Crystal Wesco.", "summary": "The federal government owns and manages over a trillion of dollars of property that is not real property, such as vehicles, computers, and office furniture. Federal agencies generally get rid of excess property through GSA's disposal process, which then allows entities such as other federal agencies, to obtain that property if they want. Some agencies have independent authorities that allow them to provide property to non-federal recipients, such as universities, before or during the GSA disposal process. GAO was asked to review how federal agencies provide property to non-federal recipients. This report examines (1) how selected agencies manage unneeded and excess property provided to non-federal recipients and (2) what is known about benefits, effects, and data on property provided to these recipients. GAO analyzed GSA non-federal recipients' reports from fiscal years 2013 to 2017, the most current available at the start of our review, and selected three agencies—USDA, DOE, and DOL—to obtain variety on the methods used to provide property to non-federal recipients. GAO reviewed relevant processes and interviewed officials from GSA, selected agencies, and non-federal recipients. GAO found the U.S. Department of Agriculture (USDA), Department of Energy (DOE), and Department of Labor (DOL) established a process for providing property to non-federal recipients but had limited insight into how these recipients used this property. Officials told GAO that some of the property was disposed of prematurely or not used at all. Such outcomes are inconsistent with agency policy. Whether these instances are widespread or uncommon is unknown due to a lack of consistent monitoring and oversight. For example, DOE officials said they were not monitoring property provided by one of their programs, because they thought the authorization had expired. Without consistent monitoring or oversight, agencies cannot be assured that property is being used as required or achieving intended objectives. Selected agencies identified benefits of providing unneeded and excess property to non-federal recipients, but the larger effect of these efforts is unclear due to a lack of reported reliable data. Agency officials said providing property to these recipients saves costs and enhances their mission. However, other sources, including a General Services Administration (GSA) study, reported that using these authorities has reduced the amount of property that would otherwise be available to federal agencies or other recipients. While data on property provided to non-federal recipients are key to understanding the effects of the program, GAO found the government-wide data on property provided to non-federal recipients were unreliable. For example, GAO found that agencies reported incorrect authorities for transactions and underreported excess property provided to such recipients. GSA's current reporting tool and guidance are unclear on how agencies should report these items, and GSA does not have definite plans on what changes it will make to address these government-wide data issues. Until these changes are made, it will be hard to understand the scope of property provided to non-federal recipients and assess the effects on the federal government's disposal process, such as whether federal agencies and other recipients may be missing opportunities to obtain property. GAO is making seven recommendations, including one to DOL and two apiece to USDA, DOE, and GSA concerning improving oversight, monitoring, and data quality for property provided to non-federal recipients. All four agencies agreed with the recommendations.", "document_type": "gao"}
{"report": "We reported in June 2018 that CBP increased its emphasis on recruitment by establishing a central recruitment office and increasing its participation in recruitment events. Specifically, CBP’s recruitment budget allocated by the centralized recruting office almost doubled, from approximately $6.4 million in fiscal year 2015 to more than $12.7 million in fiscal year 2017. CBP also more than tripled the total number of recruitment events it participated in, from 905 events in fiscal year 2015 to roughly 3,000 in both fiscal years 2016 and 2017. In addition, we reported that CBP had increased its use of recruitment incentives for OFO specifically from fiscal years 2015 through 2017 to help staff hard-to-fill locations. A recruitment incentive may be paid to a newly-appointed employee if an agency determines that a position is likely to be difficult to fill in the absence of such an incentive. From fiscal years 2015 through 2017, OFO increased the number of recruitment incentives it paid to CBP officers from nine incentives in two locations at a total cost of about $77,600 to 446 incentives across 18 locations at a cost of approximately $4.3 million. AMO and Border Patrol did not use recruitment incentives from fiscal years 2015 through 2017. As a result of its efforts, CBP also experienced an increase in the number of applications it received for law enforcement officer positions across all three operational components from fiscal years 2013 through 2017. For example, with the exception of fiscal year 2014, applications for Border Patrol agent positions increased every year, from roughly 27,000 applications in fiscal year 2013 to more than 91,000 applications in fiscal year 2017. Further, during the same period, applications for CBP officer positions increased from approximately 22,500 to more than 85,000, and applications for AMO’s law enforcement officer positions increased from about 2,000 to more than 5,800. As we reported in June 2018, CBP’s law enforcement applicants undergo a lengthy and rigorous hiring process that includes nearly a dozen steps, including a background investigation, medical examination, physical fitness test, and polygraph examination. Several of these steps can be done concurrently—for example, CBP can begin the background investigation while the candidate completes the physical fitness test and medical examination process steps. Figure 1 depicts the hiring process for Border Patrol agent and CBP officer positions. From fiscal years 2015 through 2017, CBP generally improved its performance in two key metrics to assess the efficiency and effectiveness of its hiring process for law enforcement officer positions. Specifically, CBP reduced its time-to-hire (the average number of days that elapsed between the closing date of a job announcement and an applicant’s entry- on-duty date) and increased the percentage of applicants that are hired. With regard to the time-to-hire metric, as shown in table 1, CBP’s time-to- hire decreased from fiscal years 2015 through 2017. With regard to the percentage of applicants that are hired, CBP’s overall applicant pass rate metric calculates the estimated percentage of applicants who successfully complete the hiring process and enter on duty. CBP data indicate that overall applicant pass rates more than doubled for CBP officer and Border Patrol agent positions from fiscal years 2016 through 2017. CBP officials told us that higher overall applicant pass rates paired with recent increases in the number of applications received by the agency are starting to result in an increase in the number of law enforcement officers hired, as applicants complete CBP’s hiring process and officially enter on duty. As we reported in June 2018, CBP data indicated that more law enforcement officers entered on duty in the first half of fiscal year 2018 than entered on duty in the first half of fiscal year 2017. Specifically, the total number of CBP officers and Border Patrol agents that entered on duty in the first half of fiscal year 2018 increased by roughly 50 percent and 83 percent, respectively, when compared to the same period of the prior fiscal year. Further, the total number of AMO law enforcement officers that entered on duty in the first half of fiscal year 2018 more than doubled from the same period of fiscal year 2017. As we reported in June 2018, CBP has made efforts to improve its hiring process by revising certain aspects of the process, among other things. According to agency officials, these efforts to streamline and improve CBP’s overall hiring process have collectively resulted in the decreased time-to-hire and increased overall applicant pass rates discussed above. For example, in March 2017, CBP was granted the authority to waive the polygraph examination for veterans who meet certain criteria, including those who hold a current, active Top-Secret/Sensitive-Compartmented- Information clearance. Also, in April 2017, CBP received approval from the Office of Personnel Management to use direct-hire authority for law enforcement positions, which allows CBP to expedite the typical hiring process by eliminating competitive rating and ranking procedures and veterans’ preference. As of March 31, 2018, 77 CBP officers and 107 Border Patrol agents had entered on duty through this authority. CBP has also made revisions to specific steps in its hiring process, including the application, entrance examination, and polygraph examination, among others. For example, in fiscal year 2016, CBP reordered its hiring process to place the entrance examination as the first step directly after an applicant submitted an application. Prior to this change, CBP conducted qualification reviews on applicants to ensure they met position requirements before inviting them to take the entrance exam. According to CBP officials, this updated process provided applicants with the opportunity to obtain a realistic preview of the job they were applying for earlier in the hiring process. These officials explained that this helps to ensure that only those applicants who are committed to completing the hiring process and entering on duty at CBP continue through the hiring pipeline, which may help to address high applicant discontinue rates (e.g., roughly half of all eligible applicants in fiscal year 2015 did not take the exam). According to CBP officials, this revision also created efficiencies as the agency no longer has to spend time and resources on completing qualification reviews for applicants who either did not show up to take the exam or failed the exam itself. CBP has also made several changes to its polygraph examination process step, which has consistently had the lowest pass rate of any step in its hiring process. For example, among other things, CBP has increased the number of polygraph examiners available to administer the test, according to agency officials, and was piloting a new type of polygraph exam. According to CBP officials, the new examination focuses on identifying serious crimes and is sufficiently rigorous to ensure that only qualified applicants are able to pass. Preliminary data from CBP’s pilot show that this new exam has demonstrated higher pass rates when compared with CBP’s traditional polygraph exam while also taking less time, on average, per test to complete. At the time of our review, it was too early to tell if these efforts will result in improvements to the polygraph examination step. Available CBP data indicate mixed results. Specifically, while the average duration to complete this step decreased for all law enforcement officer positions from fiscal years 2015 through 2017, pass rates also declined slightly over this same period. For example, for Border Patrol agents, the pass rate declined from 28 to 26 percent, while for CBP officers, it declined from 32 to 25 percent. While CBP had reduced its time-to-hire and made efforts to improve its hiring process for law enforcement officers, CBP officials noted that the hiring process remained lengthy, which directly affected the agency’s ability to recruit and hire for law enforcement positions. CBP officials also stated that their ability to further improve CBP’s time-to-hire and increase law enforcement hires was affected by hiring process steps that can be challenging and time-consuming for applicants to complete, as well as CBP’s reliance on applicants to promptly complete certain aspects of the process. In fiscal year 2017, it took an average of 274 days for Border Patrol agent applicants and 318 days for CBP officer applicants to complete all hiring steps and enter on duty. According to a leading practice in hiring we identified for such positions, agencies should ensure that the hiring process is not protracted or onerous for applicants. According to CBP officials, the agency’s multi-step hiring process for its law enforcement officer positions was intentionally rigorous and involves extensive applicant screening to ensure that only qualified candidates meet the technical, physical, and suitability requirements for employment at CBP. Even so, CBP officials across several components told us that the agency’s time-to-hire was too long and directly affected the component’s ability to recruit and hire for law enforcement positions. For example, OFO officials told us that the longer the hiring process takes to complete, the more likely it was that an applicant will drop out. Further, qualified applicants may also decide to apply for employment at a competing law enforcement agency that may have a less rigorous process than CBP’s, according to CBP officials. One factor that affects CBP’s ability to efficiently process and onboard law enforcement officers are specific hiring process steps that are time- consuming and challenging for candidates to complete. For example, CBP officials cited the polygraph examination as a significant bottleneck within CBP’s hiring process. In addition to having the lowest pass rate of any step in CBP’s process, the polygraph examination also took CBP officer and Border Patrol agent applicants, on average, the longest amount of time to complete in fiscal year 2017—74 days and 94 days, respectively. Further, CBP officials told us that these already lengthy time frames may increase further because of the growing number of applicants for CBP’s law enforcement positions. In addition, on average, it took CBP law enforcement officer applicants across all three components 55 days or more to complete the medical examination and more than 60 days to complete the background investigation. In November 2017, CBP hired a contractor—Accenture Federal Services, LLC—to help the agency recruit and hire the 5,000 Border Patrol agents called for in Executive Order 13767, as well as an additional 2,000 CBP officers and 500 AMO personnel. Specifically, at the time of our June 2018 report, the contract had a total potential period of 5 years at a not- to-exceed value of $297 million. The contract included a base year and four 1-year option periods, which CBP may exercise at its discretion for a total potential period of 5 years. Under this performance-based contract, Accenture is responsible for enhancing CBP’s recruitment efforts and managing the hiring process for those applicants it recruits. We reported that the Accenture contract is intended to enhance CBP’s recruitment efforts by improving its marketing strategy and utilizing new ways to capture and analyze data to better inform recruitment efforts, according to CBP officials. To meet target staffing levels, CBP expected that the contractor would augment CBP’s current hiring infrastructure while pursuing new and innovative hiring initiatives. Specifically, the contractor is responsible for implementing the same hiring process steps and ensuring that all applicants recruited by Accenture meet CBP’s standards. CBP officials also told us that Accenture has the flexibility to pursue novel hiring tactics and pilot initiatives that CBP may not have considered or been able to undertake. For example, Accenture plans to pilot innovative ways to reduce the time-to-hire, including by streamlining steps in the hiring process, which could help to improve CBP’s overall process and generate increased hires for law enforcement positions. At the time of our June 2018 report, some key issues were still being negotiated between CBP and the contractor. For example, while CBP officials told us that the main metric used to assess Accenture’s effectiveness will be the total number of hires the contractor produces, they were still working to finalize other key metrics for evaluating the contractor’s effectiveness as well as an oversight plan to ensure the contractor operates according to agency requirements. As a result, we reported that it was too early to determine whether these initiatives would help increase the number and quality of applicants for CBP’s law enforcement officer positions. We also reported that it was too early to evaluate whether the contractor would be able to efficiently and effectively provide the surge hiring capacity CBP needs to achieve its staffing goals. In June 2018, we reported that CBP’s annual rates of attrition were relatively low, but CBP faced challenges retaining law enforcement officers in hard-to-fill locations. From fiscal years 2013 through 2017, OFO’s annual attrition rates for the CBP officer position were consistent at about 3 percent, while rates for Border Patrol agent and AMO’s Marine Interdiction Agent positions were below 5 percent in 4 out of the 5 fiscal years we reviewed. When we compared CBP’s annual attrition rates for these positions to those of other selected law enforcement agencies, we found that CBP’s attrition rates were similar to U.S. Immigration and Customs Enforcement’s (ICE) annual attrition rates for its law enforcement positions and generally lower than those of the Secret Service and the Federal Bureau of Prisons. Annual attrition rates for AMO’s aviation positions were higher, ranging from 5.0 percent to 9.2 percent for the Air Interdiction Agent position and 7.8 percent to 11.1 percent for the Aviation Enforcement Agent position. Even so, fiscal years 2015 through 2017, attrition rates for these positions have generally remained lower than those of the Secret Service and the Bureau of Prisons. In addition, from fiscal years 2013 through 2017, CBP’s ability to hire more law enforcement officers than it lost varied across positions. Specifically, CBP consistently hired more CBP officers and Aviation Enforcement Agents than it lost. Further, while CBP generally maintained its staffing levels for Marine Interdiction Agents, the agency consistently lost more Border Patrol agents and Air Interdiction Agents than it hired. Even so, onboard staffing levels for all five of CBP’s law enforcement officer positions have consistently remained below authorized staffing levels. CBP has acknowledged that improving its retention of qualified law enforcement personnel is critical in addressing staffing shortfalls, but CBP officials identified difficulties in retaining key law enforcement staff as a result of geographically-remote and hard-to-fill duty locations. CBP officials across all three operational components cited location—and specifically employees’ inability to relocate to posts in more desirable locations—as a primary challenge facing the agency in retaining qualified personnel. Border Patrol officials explained that duty stations in certain remote locations present retention challenges due to quality-of-life factors. For example, the officials told us that agents may not want to live with their families in an area without a hospital, with low-performing schools, or with relatively long commutes from their homes to their duty station. Border Patrol’s difficulty in retaining law enforcement staff in such locations is exacerbated by competition with other federal, state, and local law enforcement organizations for qualified personnel. According to Border Patrol officials, other agencies are often able to offer more desirable duty locations—such as major cities—and, in some cases, higher compensation. CBP data indicate that Border Patrol agents consistently leave the component for employment with other law enforcement agencies, including OFO as well as other DHS components such as ICE. For example, while retirements accounted for more than half of annual CBP officer losses from fiscal years 2013 through 2017, they accounted for less than a quarter of annual Border Patrol agent losses, indicating that the majority of these agents are not retiring but are generally leaving to pursue other employment. Further, according to CBP data, the number of Border Patrol agents departing for employment at other federal agencies increased steadily, from 75 agents in fiscal year 2013 to 348 agents in fiscal year 2017—or nearly 40 percent of all Border Patrol agent losses in that fiscal year. Border Patrol officials told us, for example, that working a standard day shift at ICE in a controlled indoor environment located in a major metropolitan area for similar or even lower salaries presents an attractive career alternative for Border Patrol agents who often work night shifts in extreme weather in geographically remote locations. The President of the National Border Patrol Council also cited this challenge, stating that unless Border Patrol agents have a strong incentive to remain in remote, undesirable locations—such as higher compensation when compared with other law enforcement agencies—they are likely to leave the agency for similar positions located in more desirable locations. While OFO officials told us the component did not face an across-the- board challenge in retaining CBP officers, they have had difficulty retaining officers in certain hard-to-fill locations that may be geographically remote or unattractive for families, such as Nogales, Arizona, and San Ysidro, California. As a result, CBP officer staffing levels in these locations have consistently remained below authorized targets. AMO has also had difficulty retaining its law enforcement personnel—and particularly its Air Interdiction Agent staff—in hard-to-fill locations, such as Aguadilla, Puerto Rico, and Laredo, Texas. However, given the unique qualifications and competencies required for the Air Interdiction Agent position, AMO does not compete with other law enforcement organizations. Instead, AMO officials told us they compete with the commercial airline industry for qualified pilots. Specifically, they stated that this competition is exacerbated by a nationwide shortage of pilots. In addition, AMO officials explained that there is a perception among applicants that commercial airlines are able to offer pilots more desirable locations and higher compensation. However, they told us that AMO generally provided pilots with higher starting salaries than many regional airlines as well as most career options available to helicopter pilots. All three CBP operational components have taken steps to retain qualified law enforcement personnel by offering opportunities for employees to relocate to more desirable locations and pursuing the use of financial incentives, special salary rates, and other payments and allowances. Relocation opportunities. Border Patrol, OFO, and AMO have formal programs that provide law enforcement officers with opportunities to relocate. For example, in fiscal year 2017, Border Patrol implemented its Operational Mobility Program and received initial funding to relocate about 500 Border Patrol agents to new locations based on the component’s staffing needs. According to Border Patrol officials, retaining current employees is a top focus for leadership at the component and this program provides Border Patrol agents with opportunities for a paid relocation to a more desirable location at a lower cost to CBP than an official permanent change of station transfer. As of April 2018, Border Patrol officials told us that 322 Border Patrol agents had accepted reassignment opportunities through the program and the component hoped to continue receiving funding to provide these opportunities. Financial Incentives and Other Payments and Allowances. CBP’s three operational components have also taken steps to supplement employees’ salaries through the use of human capital flexibilities—such as retention and relocation incentives and special salary rates—as well as other payments and allowances. CBP’s goal in pursuing these human capital flexibilities is to retain current employees—especially in remote or hard-to-fill locations—who are likely to internally relocate within CBP to more desirable duty locations or depart the agency for similar positions at other law enforcement organizations or commercial airlines. However, we found that from fiscal years 2013 through 2017, CBP’s use of such financial incentives and other payments was limited, as the agency paid a total of four retention incentives and 13 relocation incentives, and implemented one special salary rate for all positions during this 5-year period. From fiscal years 2013 through 2017, Border Patrol did not offer retention incentives to agents and paid two relocation incentives to transfer Border Patrol agents to Artesia, New Mexico, and Washington, D.C., at a cost of roughly $78,000. However, in fiscal year 2018, Border Patrol increased its use of relocation incentives to facilitate the transfer of agents to duty stations along the southwest border that are less desirable due to the remoteness of the location and lack of basic amenities and infrastructure. Specifically, as of April 2018, 67 Border Patrol agents had received such incentives to relocate to duty stations in Ajo, Arizona; Calexico, California; and Big Bend, Texas; among others. While Border Patrol did not offer retention incentives during our review period, it submitted a formal request to CBP leadership in February 2018 for a 10 percent across-the-board retention incentive for all Border Patrol agents at the GS-13 level and below, which represents the majority of the component’s frontline workforce. According to Border Patrol documentation, these incentives, if implemented, could help reduce Border Patrol’s attrition rate—which has consistently outpaced its hiring rate—by helping retain agents who may have otherwise left Border Patrol for similar positions in OFO, ICE, or other law enforcement agencies. According to CBP officials, as of April 2018, CBP leadership was evaluating Border Patrol’s group retention incentive request, including the costs associated with implementing this 10 percent across-the-board incentive. In addition, as the incentive would benefit Border Patrol agents in all of the component’s duty locations, the extent to which this effort would be effective in targeting agent attrition in the remote locations that represent CBP’s largest staffing challenges remains to be seen. Border Patrol approved the 10 percent retention incentive and is awaiting funding for implementation, according to officials. From fiscal years 2013 through 2017, OFO paid a total of four retention incentives at a cost of $149,000 to retain CBP officers in Tucson, Arizona; Detroit, Michigan; Carbury, North Dakota; and Laredo, Texas. Further, OFO paid seven relocation incentives at a cost of approximately $160,000 to relocate personnel to the hard-to-fill ports of Alcan and Nome, Alaska; Coburn Grove, Maine; and Detroit, Michigan. One OFO official told us OFO did not regularly use these incentives because its relatively low annual attrition rates make it difficult to propose a persuasive business case to CBP leadership that such incentives are necessary. Further, another OFO official explained that OFO’s strategy is focused on using recruitment incentives to staff hard-to-fill locations with new employees. From fiscal years 2013 through 2017, AMO did not offer retention incentives to law enforcement personnel and paid a total of four relocation incentives to transfer three Air Interdiction Agents and one Marine Interdiction Agent to Puerto Rico at a cost of approximately $84,000. However, AMO has taken steps to pursue additional human capital flexibilities to address its difficulty in retaining Air Interdiction Agents, including a group retention incentive and a special salary rate. In June 2018, we reported that CBP does not have a systematic process for capturing and analyzing information on law enforcement officers who are leaving, such as an exit interview or survey. As a result, the agency does not have important information it could use to help inform future retention efforts. Standards for Internal Control in the Federal Government states that management should obtain relevant data from reliable sources and process these data into quality information to make informed decisions in achieving key objectives. Taking steps to ensure that the agency’s operational components are systematically collecting and analyzing complete and accurate information on all departing law enforcement officers—including the factors that influenced their decision to separate—would better position CBP to understand its retention challenges and take appropriate action to address them. We recommended that CBP should ensure that its operational components systematically collect and analyze data on departing law enforcement officers and use this information to inform retention efforts. CBP agreed with the recommendation. CBP officials reported in February 2019 that they developed and implemented a CBP-wide exit survey in August 2018 and have taken steps to promote the survey and encourage exiting CBP employees to fill it out. The officials also noted that they plan to analyze the survey results on a quarterly basis starting in April 2019. These actions, if fully implemented, should address the intent of our recommendation. Chairwoman Torres Small, Ranking Member Crenshaw, and Members of the Subcommittee, this completes my prepared statement. I would be happy to respond to any questions you or the members of the committee may have. If you or your staff have any questions about this statement, please contact Rebecca Gambler at (202) 512-8777 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Adam Hoffman (Assistant Director), Bryan Bourgault, Sasan J. “Jon” Najmi, and Michelle Serfass. This is a w ork of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety w ithout further permission from GAO. How ever, because this w ork may contain copyrighted images or other material, permission from the copyright holder may be necessary if you w ish to reproduce this material separately.", "summary": "CBP is responsible for securing U.S. borders and employs nearly 45,000 law enforcement officers across its three operational components at and between U.S. ports of entry, in the air and maritime environment, and at certain overseas locations. In recent years, CBP has not attained target staffing levels for its law enforcement positions, citing high attrition rates in some locations, a protracted hiring process, and competition from other law enforcement agencies. This statement addresses CBP's efforts to (1) recruit and more efficiently hire law enforcement applicants, and (2) retain law enforcement officers. This statement is based on a GAO report issued in June 2018 on CBP's recruiting, hiring, and retention efforts along with updates as of February 2019 on actions CBP has taken to address GAO's prior recommendation. For the previous report, GAO analyzed CBP data on recruitment efforts, hiring process steps, and retention rates; examined strategies related to these activities; and interviewed CBP officials and union groups. GAO also reviewed information on CBP actions to implement GAO's prior recommendation. In June 2018, GAO reported that U.S. Customs and Border Protection (CBP) increased its emphasis on recruitment by establishing a central recruitment office in 2016 and increasing its participation in recruitment events, among other things. As a result, the number of applications it received for law enforcement positions across its operational components—the Office of Field Operations, U.S. Border Patrol, and Air and Marine Operations—more than tripled from fiscal years (FY) 2013 through 2017. Also, in November 2017, CBP hired a contractor to more effectively target potential applicants and better utilize data to enhance CBP's recruitment and hiring efforts. However, at the time of GAO's June 2018 report, it was too early to gauge whether the contractor would be effective in helping CBP to achieve its goal to recruit and hire more law enforcement officers. CBP improved its hiring process as demonstrated by two key metrics—reducing its time-to-hire and increasing the percentage of applicants that are hired. As shown in the table, CBP's time-to-hire decreased from FY 2015 through 2017. CBP officials stated that these improvements, paired with increases in applications, have resulted in more hires. However, the hiring process remains lengthy. For example, in FY 2017, CBP officer applications took more than 300 days, on average, to process. Certain factors contributed to the lengthy time-to-hire, including process steps that can be challenging and time-consuming for applicants to complete—such as the polygraph exam—as well as CBP's reliance on applicants to promptly complete certain aspects of the process—such as submitting their background investigation form. CBP enhanced its efforts to address retention challenges. However, staffing levels for law enforcement positions consistently remained below target levels. For example, CBP ended FY 2017 more than 1,100 CBP officers below its target staffing level. CBP officials cited employees' inability to relocate to more desirable locations as the primary retention challenge. CBP offered some relocation opportunities to law enforcement personnel and has pursued the use of financial incentives and other payments to supplement salaries, especially for those staffed to remote or hard-to-fill locations. However, retaining law enforcement officers in hard-to-fill locations continues to be challenging for CBP. GAO reported that CBP could be better positioned to understand its retention challenges and take appropriate action to address them by implementing a formal process for capturing information on all departing employees. In response, CBP officials reported taking steps to implement a CBP-wide exit survey and plan to analyze the results of the survey quarterly, beginning April 2019. GAO recommended in its June 2018 report that CBP systematically collect and analyze data on departing law enforcement officers and use this information to inform retention efforts. DHS concurred, and CBP has actions planned or underway to address this recommendation.", "document_type": "gao"}
{"report": "For many veterans, long-term care is provided directly or purchased by VA. VA provides or pays for long-term care for eligible veterans enrolled in VA’s health care through a variety of programs, including institutional- based care like nursing homes and noninstitutional programs like home health care, which provides care to veterans in their own homes. VA provides or pays for long-term care—ranging from assistance with dressing and bathing to clinical care for spinal injuries or dementia— through a range of three institutional and 11 noninstitutional programs. Institutional programs, such as nursing homes, typically provide more acute skilled nursing care in a residential facility; noninstitutional programs, such as the Home-Based Primary Care program, provide care to veterans in their homes or communities. (See fig. 1 for a list of VA’s institutional and noninstitutional long-term care programs and app. I for brief descriptions of these programs.) Institutional Programs. VA provides or pays for eligible veterans to receive long-term care in three institutional programs that primarily provide skilled nursing care, such as for rehabilitation after surgery or for health issues or disabilities that require 24-hour care in a residential facility. These three programs include: VA Community Living Centers (VA-owned and -operated), Community Nursing Homes (publicly or privately owned and under contract with VA), and State Veterans Homes (state-owned and -operated homes approved and supported by VA). Noninstitutional Programs. VA provides or pays for eligible veterans to receive noninstitutional long-term care through 11 home or community- based programs, where most veterans receive long-term care. Several of VA’s noninstitutional programs provide personal care assistance to help veterans with activities of daily living—e.g., dressing, eating, bathing— that enable veterans to remain living at home, including the Homemaker Home Health Aide, Community Adult Day Health Care, and Respite Care programs. VAMCs evaluate veterans to determine the extent to which they can perform activities of daily living and to identify the available programs that would best meet their needs. In addition, VA’s noninstitutional programs include the Community Residential Care program where caregivers—in settings such as Medical Foster Homes where no more than three residents receive care—provide 24 hour care for veterans who cannot live alone because of medical or mental health conditions. Several of VA’s long-term care programs serve veterans with special needs. For example, some of these programs, such as certain Community Nursing Homes, Adult Day Health Care, and Hospice and Respite Care programs, have specially trained staff to serve veterans with dementia. The Spinal Cord Injury and Disability Home Care program and certain VA Community Living Centers are equipped to serve veterans needing ventilator care. In addition, some programs offer specific services for younger veterans, such as certain Adult Day Health Care programs. All veterans enrolled in the VA health care system are eligible for VA’s basic medical benefits package, which includes certain institutional and noninstitutional long-term care services. A veteran’s eligibility for fully or partially covered nursing home care is determined by the veteran’s priority for care, which is generally based on the veteran’s service- connected disability status. Specifically, VA must cover the full cost of nursing home care for veterans who need this care for a service- connected disability and for veterans with service-connected disabilities rated at 70 percent or more. To the extent resources allow, VA may cover this nursing home care for certain other veterans, such as former prisoners of war and those awarded the Purple Heart. For all other veterans, VA may generally cover nursing home care to the extent resources and capacity allow and with the veteran’s agreement to share certain costs. Veterans’ placement in any particular institutional or noninstitutional long- term care program may depend on their clinical needs, disability ratings, preferences, and the availability of VA programs. When funds are limited, the agency may prioritize program placement based on veterans’ service- connected disability ratings. Decisions about which long-term care programs may be the best fit are made at the VAMC level between VA providers, veterans, and their families. VA providers may discuss a range of factors when making decisions about this care, such as health needs, the type of care provided in different programs, space availability, eligibility, and the veteran’s geographic preference. For facility-based programs, VAMC staff may also encourage veterans to take a tour of the prospective home. VA’s stated goal is to honor veterans’ preferences for care, including finding ways for veterans to age in their homes and communities instead of nursing homes. A diverse set of veterans receive care in VA’s long-term care programs. According to VA data for fiscal year 2018, 70 percent (370,821) of the veterans who received VA long-term care during the fiscal year were aged 65 or older. (See fig. 2.) In addition, 91 percent (480,299) of those who received this care had served in the military prior to September 11, 2001. Lastly, according to VA data for fiscal year 2018, 55 percent (291,197) of veterans receiving long-term care had some level of service- connected disabilities. VA’s planning for veterans’ long-term care is informed by broader strategic planning by VA and the VHA and then operationalized by GEC at the program level. Veterans Integrated Service Networks (VISN) then implement GEC strategies for their regions and VAMCs implement and manage the various programs. VA, through the Assistant Secretary for Enterprise Integration’s office, sets a strategic plan that identifies agency-wide goals. For example, VA’s fiscal year 2018 through 2024 strategic plan identifies a goal that veterans “choose VA for easy access, greater choices, and clear information to make informed decisions,” and the plan notes that VA should “understand veterans’ needs throughout their lives to enhance their choices and improve customer experiences.” VA develops its agency-wide strategic plan every four years. VHA, through its Office of Policy and Planning, identifies strategies within VA’s health care system to address VA’s agency-wide goals. For example, VHA’s fiscal year 2018 through 2019 strategy, operationalizing VA’s goal for veteran choice, is to “honor veterans’ preferences by offering home and community based care to prevent unwanted nursing home care.” VHA strategic planning occurs every two years according to VA officials. VHA’s Office of Enrollment and Forecasting uses the EHCPM to project the utilization of and cost for care across most of VA’s health care programs 20 years into the future, including most long-term care programs. GEC’s strategic planning operationalizes VA and VHA goals and strategies for long-term care at the program level. For example, to achieve VA’s goal of veteran choice and VHA’s strategy of honoring veteran preferences, GEC developed a model to identify veterans at the highest risk of needing nursing home care. According to GEC officials, the GEC strategic planning process generally occurs annually. VISNs are responsible for managing and overseeing VAMCs within their regions where long-term care is delivered, with a GEC point of contact at each VISN who can address GEC issues as they arise, according to VA officials. VAMCs within each VISN are, according to VA officials, responsible for the management of individual long-term care programs, including oversight of long-term care programs’ quality of care. As previously noted, VAMCs also have a role in guiding decisions about individual veterans’ long-term care placement. Other health care systems nationwide are also planning to meet the growing demand for long-term care and have developed strategies to address future long-term care challenges. For example, some state agencies, which provide long-term care through Medicaid, have developed strategies to help aging citizens live in their communities by enhancing community-based services and developing the workforce to provide care. VA has a federal Geriatrics and Gerontology Advisory Group to share knowledge with other long-term care providers and to advise the Secretary and Under Secretary for Health on all matters related to geriatrics and gerontology for the care of veterans. Our analysis of VA data shows that the number of veterans receiving care in one or more of the VA long-term care programs increased 14 percent from fiscal years 2014 through 2018, from 464,071 to 530,327 veterans. The data also show that utilization increased more for noninstitutional programs than for institutional programs. Specifically by program type, VA data show that the number of veterans receiving institutional long-term care increased 8 percent during these years, from 97,124 to 105,151, while the number receiving noninstitutional care increased 16 percent, from 395,736 to 459,783. VA officials told us that the agency is continuing to expand veterans’ access to noninstitutional care programs because institutional care is more costly than home- or community-based care, and because veterans prefer to delay or reduce the amount of nursing home care they receive. Our analysis showed that utilization of long-term care—in terms of various VA workload units—also generally increased from fiscal years 2014 through 2018. The average daily census increased for two of VA’s three institutional programs—Community Nursing Homes increased by 26 percent from 7,771 to 9,808 and State Veterans Homes increased by 1 percent from 23,176 to 23,423. Five of the 11 noninstitutional programs experienced increases in their workload over this period, ranging from 8 percent to 48 percent. For example, the number of VA clinic stops (one type of VA workload unit) in the Homemaker Home Health Aid program—which served approximately 23 percent of the veterans receiving noninstitutional long-term care in fiscal year 2018—increased 48 percent from 8.3 million to 12.3 million clinic stops. (See app. II for more information on veterans’ utilization of institutional and noninstitutional long-term care by program.) According to VA, veterans’ use of VA long-term care programs increased during fiscal years 2014 through 2018 for several reasons, including that a large number of Vietnam veterans are aging and that more veterans are receiving higher service-connected disability ratings. We found the number of veterans who served on or after 9/11 and received VA long- term care to have increased at a faster rate than the overall number of veterans who received this care, from fiscal year 2014 through 2018. Our analysis of VA data shows that VA’s spending for long-term care— which VA reports as obligations—increased 33 percent, from $6.8 billion in fiscal year 2014 to $9.1 billion in fiscal year 2018. Furthermore, over this time period institutional program obligations declined as a proportion of total obligations, from 74 percent to 67 percent, while the proportion of noninstitutional program obligations rose from 26 percent to 33 percent. (See fig. 3.) Looking at VA’s three institutional programs, our analysis shows VA’s obligations for these programs increased 21 percent from fiscal years 2014 through 2018, from $5.0 billion to $6.1 billion. The highest share of obligations for institutional care over this time period was for the VA Community Living Centers program, which increased 11 percent, from $3.3 billion to $3.7 billion. This percentage increase was less than the increases for the Community Nursing Homes program (49 percent) and the State Veterans Homes program (33 percent); however, costs for these last two programs are significantly lower than for the other institutional program. VA obligations for its 11 noninstitutional long-term care programs increased 66 percent, from $1.8 to $2.9 billion, between fiscal years 2014 and 2018. Noninstitutional programs with the highest share of obligations during that period included the Homemaker Home Health Aide, Home-Based Primary Care, Purchased Skilled Home Care, and Home Telehealth programs. Noninstitutional programs with the highest obligation increases included the Homemaker Home Health Aide (109 percent) and Purchased Skilled Home Care (164 percent) programs. However, two noninstitutional programs saw obligations decline during these years, including the State Home Adult Day Health Care program with a 59 percent decrease, and the Community Residential Care program with a 10 percent decrease. (See app. II for more information on VA’s obligations for institutional and noninstitutional long-term care by program.) VA projects utilization of long-term care will increase for most of the programs included in VA’s EHCPM from fiscal years 2017 through 2037. For the two institutional programs included in the EHCPM, VA projects that utilization based on workload units (average daily census) will increase by 80 percent for the Community Nursing Homes program but will decrease by 10 percent for the Community Living Centers program. For the 10 noninstitutional programs included in the EHCPM, VA projects that utilization based on workload units (which differ by program) will increase for nine of the 10 programs—with increases ranging from 1 percent to 95 percent. For example, the number of VA clinic stops for the Homemaker Home Health Aide program is projected to increase 84 percent. (See app. III for more information on projected utilization for institutional and noninstitutional long-term care by program.) VA reports that these projections are based on expected increases in the number of veterans who will rely on VA for their long-term care needs through fiscal year 2037. According to VA officials, these projected increases are due to a variety of factors, including that VA plans to continue expanding the availability of home- and community- based care, and plans to provide care to an increasing number of aging veterans and veterans rated in the highest service-connected disability groups. For example, VA data show that the proportion of long-term care provided to veterans with service-connected disabilities is projected to increase from 60 percent to 78 percent of utilization from fiscal year 2017 to 2037, and the proportion of this care provided to post-9/11 deployed combat veterans is projected to increase from 1 percent to 6 percent of all long- term care utilization during these years. Further, VA officials told us that the agency has planned to expand veterans’ access to noninstitutional care when appropriate, and they have integrated these assumptions into the EHCPM. VA projects that increases in overall demand for long-term care for veterans will result in future expenditure increases for the programs included in VA’s EHCPM. Specifically, VA’s model projects expenditures will more than double from fiscal years 2017 through 2037, increasing from $6.9 billion to $14.3 billion (107 percent). VA projects that its expenditures for its institutional programs will be higher than for its noninstitutional programs, reaching $7.5 billion and $6.8 billion, respectively, by fiscal year 2037. However, VA also projects that the proportion of expenditures for institutional long-term care will decrease from 63 percent to 53 percent, as the share for noninstitutional programs increases. (See fig. 4.) While VA expenditures are projected to increase for all long-term care programs included in the EHCPM from fiscal years 2017 through 2037, the size of these projected increases vary by program. For example, VA projects its expenditures for institutional programs to increase 71 percent overall over this time period, with the VA Community Living Centers program projected to increase 50 percent and the Community Nursing Homes program to increase 149 percent. VA projects that its expenditures for noninstitutional programs will increase 168 percent over this time, with the largest projected increases including the Community Adult Day Health Care (240 percent), Home Respite Care (231 percent), and the Homemaker Home Health Aide (212 percent) programs. (See app. III for more information on projected expenditures for institutional and noninstitutional long-term care by program.) The projected expenditures for care provided to veterans with service- connected disabilities are projected to represent a growing percent of VA’s long-term care expenditures, increasing from 64 percent to 79 percent of expenditures for this care from fiscal years 2017 through 2037. VA projects that its expenditures for care provided to veterans with service-connected disabilities will increase 156 percent during this period, from $4.4 billion to $11.3 billion, while expenditures for care provided to veterans without service-connected disabilities will increase only 19 percent, from $2.5 billion to $3.0 billion. In addition, VA projects that the proportion of spending for long-term care provided to post-9/11 deployed combat veterans will rise from 1 percent to 7 percent during these years, from $89 million to $981 million, as that cohort of veterans ages. As VA works to meet veterans’ growing demand for long-term care, it faces a number of key challenges: workforce shortages, geographic alignment of care, and difficulty meeting veterans’ needs for specialty care. (See table 1.) These challenges, which VA has identified, are similar to challenges faced by other health care systems. However, while VA’s GEC—the office that manages VA long-term care programs—is aware of these challenges, as of November 2019 GEC’s strategic planning has not identified measurable goals for addressing them. Addressing workforce shortages. According to VA, the agency faces challenges hiring the staff needed to meet veterans’ demand for long-term care, a challenge that is likely to grow as demand for care is projected to increase in coming years. We have previously reported on workforce shortages in key positions—such as nursing assistants and home health aides—that are critical for supporting long-term care programs and affect health care systems beyond VA. Within VA, the Healthcare Analysis and Information Group (HAIG) report found that 80 percent of VA community living centers had, at the time of the report, current vacancies for nursing assistant or health technician positions. These workforce challenges have led to waitlists for some long-term care programs. For example, VA officials told us staffing challenges were the key factor creating a waitlist of 1,780 veterans for the Home-Based Primary Care program. (The HAIG report found 65 percent of VA facilities cited staffing as a barrier to expanding Home- Based Primary Care.) GEC officials recognize these workforce challenges and told us they have developed some workforce strategies such as offering geriatrics training to rural primary care providers through GEC’s Geriatric Scholars Program. Aligning care geographically. According to VA, the agency faces challenges aligning its provided or purchased long-term care with where veterans live. VA data show that 2.8 million VA-enrolled veterans lived in rural areas as of 2018, and that veteran populations have shifted to different geographic regions. Providing long-term care in rural areas is a challenge experienced by other health care systems; for example, a report from the Rural Policy Research Institute identified challenges with providing long-term care in rural areas, including “more limited access to services and support” and the “absence of an adequate workforce and infrastructure.” VA officials also told us that veterans moving from one region to another presents demand and capacity challenges. For example, officials told us that veterans have moved away from the Northeast and to the South, and that VA now has too many long-term care beds in the Northeast and too few in the South. VA officials acknowledged the challenge of aligning care with where veterans live and pointed to telehealth, where veterans can receive care remotely, and to Veteran Directed Care program, which provides veterans with a budget to manage their own care, as approaches that could provide care to veterans in rural areas with limited access to VA provided or purchased care. GEC officials have also identified potential strategies to address the issue; for example, GEC’s strategic planning includes a proposal to expand telehealth geriatrics services to reach more veterans, although officials told us this effort is currently unfunded. Further, VA officials from the Office of Policy and Planning said an ongoing market assessment project will provide information that will help VA align its provided and purchased care with where veterans live to better meet veteran needs. Meeting needs for specialty care. According to VA, the agency faces challenges meeting some specialty care needs for veterans in long-term care. Specifically, it can be difficult to find appropriate long- term care settings for veterans with dementia, behavioral issues, and for veterans requiring a ventilator. Meeting specialty care needs is also a challenge for other health care systems; for example, a 2017 study from the RAND Corporation found that the U.S. health system does not have sufficient capacity to care for a growing number of people with Alzheimer’s disease. Challenges in providing this type of care are not new for VA. For example, in 2013 we reported that VA officials told us that while “in certain geographic areas [community living centers] provide certain services that are not available in the community, such as dementia care, behavioral health services, and care for ventilator-dependent residents,” in other areas “these specialized services might not be available in a [community living center] and instead might be available at a community nursing home.” As previously mentioned, VA has developed some programs to provide specialty care (e.g. VA’s Spinal Cord Injury and Disability Care program and the agency’s efforts to educate home caregivers on how to better serve veterans with dementia). While GEC recognizes and has taken some steps to address the challenges it faces in meeting the demand for long-term care, our review of GEC’s most recently approved strategic planning document from March 2019 shows that GEC has not established measurable goals for its efforts to address these three key challenges. GEC has not established measurable goals for its efforts to address workforce shortages, such as specific staffing targets necessary to address the waitlist for the Home-Based Primary Care program, or defining the number of rural providers it expects to train through the Geriatrics Scholar program. GEC has not established measurable goals for its efforts to address the geographic alignment of care, such as specific targets for providing long-term care within the Home Telehealth and Veteran Directed Care programs. GEC has not established measurable goals for its efforts to address difficulties meeting veterans’ needs for specialty care, such as specific targets for the number of available ventilators or the number of caregivers educated to help veterans with dementia. According to GAO’s body of work on effectively managing performance under the Government Performance and Results Act of 1993 (GPRA), as enhanced by the GPRA Modernization Act of 2010, federal agencies should clarify and clearly define measurable outcomes for each strategic objective and assess progress towards those goals. VA officials told us that competing priorities, including implementation of the VA MISSION Act of 2018, have affected GEC’s ability to effectively address challenges to meeting veterans’ long-term care needs. Without measurable goals, however, VA is limited in its ability to better plan for and understand progress towards addressing the challenges it faces meeting veterans’ long-term care needs. As VA works to address these challenges, it does so along with other health care systems, and VA has opportunities for leveraging outside experience through VA’s Geriatrics and Gerontology Advisory Group. For example, the Advisory Group recently acknowledged workforce challenges and recommended that VA “devise strategies to create incentives and identify and remove barriers” for the recruiting and retaining the health care workforce needed to care for VA’s growing geriatric veteran population. In addition to the key challenges that VA and many other health care systems face, VA has identified, but has not planned to take steps to fully address, challenges at the VAMC level that affect its ability to meet veterans’ long-term care needs. Specifically, VA has identified issues with inconsistency in the management of the 14 long-term care programs at the VAMC level that could lead to inefficient and inequitable decisions about long-term care across VA. While VA has identified the steps it can take to address these issues, it has not implemented these steps. First, VA identified that VAMCs do not have a consistent approach to managing VA’s 14 long-term care programs. GEC officials told us that fragmentation of the long-term care programs within the VAMCs—that is, where programs could be run by one or more departments within the VAMC, for example the Nursing department or the Social Work department at VAMCs where there are not GEC staff—hinders standardization and the ability to get veterans the right care. Similarly, the HAIG report found that VAMCs organize their long-term care programs differently and recommended that to “efficiently, reliably, and equitably serve veterans” VA align GEC programs “at all VISNs and eventually VAMCs nationwide.” GEC strategic planning documents outline a goal of alignment within the VISNs, and officials said alignment has been established within the VISNs. However, VA officials told us that, as of October 2019, they had not taken action to pursue VAMC-level alignment with a GEC point of contact at each VAMC that could provide consistency across long-term care programs at the VAMC level. Second, GEC has developed a tool to improve the consistency with which VAMCs determine the amount of services needed for veterans based on their specific health issues. However, as of October 2019, VA has not required the tool be used in all VAMCs. VA has identified that VAMCs do not have a consistent approach to determining the amount of noninstitutional long-term care services veterans need. VA officials told us that, as of October 2019, VAMCs used different methods to assess the amount of noninstitutional long-term care services veterans need—for example, how many hours of in-home care veterans need. As a result, decisions about the amount of services veterans receive may vary by VAMC. The HAIG report recommended that VA use a standardized approach to ensure the “balance of noninstitutional care programs, program reliability, and equity of resource distribution.” GEC officials said the tool they developed is currently being used by some VAMCs, and they expect VA will require the tool to be used by all VAMCs sometime in the next year. However, VA has not set time frames for this requirement. One of VA’s performance goals is to provide highly reliable and integrated care and support and excellent customer service. Furthermore, federal internal controls dictate that federal agencies should exercise oversight responsibility, for example by overseeing the remediation of deficiencies as appropriate and providing direction to management on appropriate time frames for correcting these deficiencies. Although VA has identified steps it can take to improve consistency in long-term care programs, according to officials, it has not prioritized their implementation. Without a reliably consistent approach to administering long-term care programs across its VAMCs, VA may not consistently and equitably meet veteran preferences and needs. VA currently faces difficult challenges meeting the demand for long-term care. These challenges—such as addressing workforce shortages, aligning care geographically, and meeting specialty care needs—are likely to intensify as veterans’ demand for long-term care grows. However, a lack of measurable goals in the strategic planning efforts of VA’s GEC, which has the lead responsibility for managing VA’s 14 long- term care programs, affects VA’s ability to appropriately plan for and understand its progress towards addressing long-term care challenges. In addition to these key challenges, VA has identified, but not yet fully addressed, inconsistencies in the management of the 14 long-term care programs at the VAMC level. These inconsistencies in determining both the best program for veterans and the amount of noninstitutional care veterans need can lead to inefficient and inequitable experiences with VA’s long-term care programs. We are making the following three recommendations to VA: The Secretary of VA should direct GEC leadership to develop measurable goals for its efforts to address key long-term care challenges: workforce shortages, geographic alignment of care, and difficulty meeting veterans’ needs for specialty care. (Recommendation 1) The Secretary of VA should direct GEC leadership to set time frames for and implement a consistent GEC structure at the VAMC level. (Recommendation 2) The Secretary of VA should direct GEC leadership to set time frames for and implement a VAMC-wide standardization of the tool for assessing the noninstitutional program needs of veterans. (Recommendation 3) We provided a draft of this report to VA for review and comment. In its comments, reproduced in appendix IV, VA concurred with our three recommendations and identified actions it is taking to implement them. Specifically, VA said that it will: (1) take steps to incorporate measurable goals and defined timelines into its strategies to meet the long-term care challenges; (2) work to establish a time frame for the execution of a uniform GEC structure at the VAMC level; and (3) work to establish a time frame for the execution of a VAMC-wide standardized tool for evaluating non-institutional care needs for veterans. VA also provided technical comments that we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Veterans Affairs, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at silass@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. VA provides or pays for long-term services and supports, or long-term care, for eligible veterans through a range of three institutional and 11 noninstitutional programs. VA covers the full or partial cost of nursing home care for eligible veterans who require skilled nursing home care in an institutional program. Specifically, VA covers the full cost of nursing home care for veterans who need this care for a service-connected disability—which is an injury or disease that was incurred or aggravated while on active duty—and for veterans with service-connected disabilities rated at 70 percent or more. To the extent resources allow, VA may cover this care for certain other veterans, such as former prisoners of war and those awarded the Purple Heart. For all other veterans, VA may cover nursing home care to the extent resources and capacity allow and with the veteran’s agreement to share certain costs. (See table 2 for more information about these programs.) In addition, all veterans enrolled in the health care system are eligible for VA’s basic medical benefits package, which covers, among other things, a comprehensive array of medically necessary home- and community- based health services. While a veteran’s priority for care generally determines whether these services are provided at full or partial cost, the VA may not charge a copay for home hospice care and may waive copays for home telehealth services. (See table 3 for more information about these programs.) A veteran’s placement in a particular program may depend on their clinical needs, preferences, and the availability of VA funding and programs. payment for a Home Hospice Care program visit from a community provider. The units for each program may differ. VA officials told us that these data do not include non-veterans and may differ from data included in VA’s congressional budget justification for a variety of reasons, including the timing of when they looked at the data, the inclusion of additional data, and that VA used a standard definition of services for all years. In addition to these programs, VA may provide stipends or other services to caregivers for veterans who were seriously injured in the line of duty through the Caregiver Support program. Disabled veterans may also be eligible for increased compensation benefits from the Veterans Benefits Administration. Appendix III: Projected Utilization and Expenditures for Department of Veterans Affairs’ (VA) Long-Term Care Programs, Fiscal Years 2017 through 2037 these data do not include non-veterans and may differ from data included in VA’s budget request for a variety of reasons, including the timing of when they looked at the data, the inclusion of additional data, and that VA used a standard definition of services for all years. Sharon M. Silas, (202) 512-7114 or silass@gao.gov In addition to the contact named above, Karin Wallestad (Assistant Director), Luke Baron (Analyst-In-Charge), Kye Briesath and Corinne Quinones made key contributions to this report. Also contributing were Laurie Pachter, Vikki Porter, Jennifer Rudisill, and Selah Myers. Veterans Affairs: Sustained Leadership Attention Needed to Address Long-Standing Workforce Problems, GAO-19-720T (Washington, D.C.: Sept. 18, 2019). Veterans Health Care: VA Needs to Improve Its Allocation and Monitoring of Funding, GAO-19-670 (Washington, D.C.: Sept. 23, 2019). VA Health Care: Actions Needed to Improve Family Caregiver Program, GAO-19-618 (Washington, D.C.: Sept. 16, 2019). Veterans Health Care: Opportunities Remain to Improve Appointment Scheduling within VA and through Community Care, GAO-19-687T (Washington, D.C.: July 24, 2019). VA Health Care: Estimating Resources Needed to Provide Community Care, GAO-19-478 (Washington, D.C.: June 12, 2019). VA Real Property: Improvements in Facility Planning Needed to Ensure VA Meets Changes in Veterans’ Needs and Expectations, GAO-19-440 (Washington, D.C.: June 13, 2019). VA Nursing Home Care: VA Has Opportunities to Enhance Its Oversight and Provide More Comprehensive Information on Its Website, GAO-19-428 (Washington, D.C.: July 3, 2019). Long-Term Care Workforce: Better Information Needed on Nursing Assistants, Home Health Aides, and Other Direct Care Workers, GAO-16-718 (Washington, D.C.: Aug. 16, 2016). VA Mental Health: Clearer Guidance on Access Policies and Wait-Time Data Needed, GAO-16-24 (Washington, D.C.: Oct. 28, 2015). VA Nursing Homes: Reporting More Complete Data on Workload and Expenditures Could Enhance Oversight, GAO-14-89 (Washington, D.C.: Dec. 20, 2013). Older Americans: Continuing Care Retirement Communities Can Provide Benefits, but Not Without Some Risk, GAO-10-611 (Washington, D.C.: June 21, 2010). VA Health Care: Long-term Care Strategic Planning and Budgeting Need Improvement, GAO-09-145 (Washington, D.C.: Jan. 23, 2009).", "summary": "Veterans rely on long-term care to address a broad spectrum of needs, from providing occasional help around the house to daily assistance with eating or bathing to round-the-clock clinical care. Veterans' eligibility for this care is primarily based on their service-connected disability status, among other factors. Congress included a provision in statute for GAO to review VA's long-term care programs. This report (1) describes the use of and spending for VA long-term care and (2) discusses the challenges VA faces in meeting veterans' demand for long-term care and examines VA's plans to address those challenges. GAO reviewed VA documents, such as strategic planning documents for long-term care programs and analyzed VA utilization and expenditure data for fiscal years 2014 through 2018 (the latest available at the time of the review) and projected data through 2037. GAO also interviewed officials from VA, including officials from VA's GEC, which is responsible for overseeing long-term care programs; and from Veterans Service Organizations. The Department of Veterans Affairs (VA) provides or purchases long-term care for eligible veterans through 14 long-term care programs in institutional settings like nursing homes and noninstitutional settings like veterans' homes. From fiscal years 2014 through 2018, VA data show that the number of veterans receiving long-term care in these programs increased 14 percent (from 464,071 to 530,327 veterans), and obligations for the programs increased 33 percent (from $6.8 to $9.1 billion). VA projects demand for long-term care will continue to increase, driven in part by growing numbers of aging veterans and veterans with service-connected disabilities. Expenditures for long-term care are projected to double by 2037, as shown below. According to VA officials, VA plans to expand veterans' access to noninstitutional programs, when appropriate, to prevent or delay nursing home care and to reduce costs. VA currently faces three key challenges meeting the growing demand for long-term care: workforce shortages, geographic alignment of care (particularly for veterans in rural areas), and difficulty meeting veterans' needs for specialty care. VA's Geriatrics and Extended Care office (GEC) recognizes these challenges and has developed some plans to address them. However, GEC has not established measurable goals for these efforts, such as specific staffing targets for programs with waitlists or specific targets for providing telehealth to veterans in rural areas. Without measurable goals, VA is limited in its ability to address the challenges it faces meeting veterans' long-term care needs. GAO is making three recommendations, including that VA develop measurable goals for its efforts to address key challenges in meeting the demand for long-term care. VA concurred with GAO's recommendations and identified actions it will take to implement them.", "document_type": "gao"}
{"report": "Collectively, the ongoing GPS acquisition efforts aim to (1) modernize and sustain the existing GPS capability and (2) enhance the current GPS system by adding a more cybersecure ground system that enables M- code. M-code is a stronger, encrypted, military-specific GPS signal designed to meet military positioning, navigation, and timing needs. It will help military users overcome GPS signal jamming by using a more powerful signal and protect against false GPS signals, known as spoofing, by encrypting the signal. Figure 1 below shows how GPS satellites, ground control, and user equipment—in the form of receiver cards embedded in systems—function together as an operational system. The Air Force’s OCX program is primarily a software development effort to replace the current ground system, the operational control system (OCS), with a modernized and more cybersecure system. OCS lacks modern cybersecurity protections and cannot currently control—or enable—modernized features of the three latest generations of GPS satellites now in orbit, including M-code and three new civilian signals. Because existing military receivers were not designed to work with the new M-code signal, military users will have to make investments in new receiver development and procurement timed to when the new signal will be available before they can use it. Raytheon is the prime contractor working to deliver OCX in a series of blocks that enable additional capabilities. Block 0, which is a subset of block 1 broken out after development started, was delivered in September 2017. It helped to successfully enable the launch and initial testing of the first GPS III satellite, which was launched in December 2018, and will continue to support subsequent GPS III satellite launches. Blocks 1 and 2, originally planned as separate deliveries, have been combined into a single delivery and are currently in development. This combined delivery enables OCX to command and control each satellite and begin using the full M-code signal, as well as control new civilian signals, among other capabilities. Because of significant delays to OCX, the Air Force initiated two additional programs to modify OCS to deliver some of the planned capabilities before OCX is operational. The first program is Contingency Operations (COps)—which will enable the control of GPS III satellites to operate with the same capabilities as current GPS satellites without the additional military and civilian signals. The second program is M-code Early Use (MCEU)—which will permit some functions of M-code to be used before OCX is delivered. Neither COps nor MCEU will enable the additional civilian signals or the full M-code functionality that is expected with OCX. DOD is required by statute to establish and approve cost and schedule baselines for major defense acquisition programs before those programs enter system development, also known as the engineering and manufacturing development phase. As part of program planning, including for major defense acquisition programs, DOD policy requires program managers to establish program goals for cost, schedule, and performance parameters. Approved program baseline parameters are reported in the program’s acquisition program baseline as objective and threshold values. The objective values represent goals in terms of what the user—in the case of GPS, the Air Force—desires and expects. The threshold values represent the limit of what is acceptable—meaning cost or schedule growth above threshold values are outside of the approved cost or schedule limits. For OCX, the cost and schedule objective and threshold dates in the baseline are tied to an event called “ready to transition to operations,” which will be the completion of the OCX acquisition program schedule. For the OCX program, this is a decision within the Air Force to switch control of the GPS constellation from the current GPS ground system, OCS—at this future point with COps and MCEU modifications already added—to OCX. The delivery date of the system by Raytheon and acceptance date by the Air Force will both come before the ready to transition to operations decision. These two dates are important because their timing may influence when OCX operates. What is a critical Nunn-McCurdy unit cost breach? For major defense acquisition programs, a critical Nunn-McCurdy unit cost breach of a unit cost threshold is triggered by cost increases of at least 25 percent or more of a program’s current cost baseline or at least 50 percent or more of a program’s original cost baseline. As an acquisition program works to achieve its objective and threshold values, the original baseline goals may become unachievable. When this occurs, a revised baseline, or rebaseline, is created so the program’s cost and schedule goals are updated to more realistically reflect the program’s current status. If the increase from the cost baseline meets certain thresholds, DOD is required to notify Congress in writing. This is known as a Nunn-McCurdy breach. This notification assists Congress with monitoring program progress, especially on troubled programs. A critical Nunn-McCurdy unit cost breach is the most serious type of breach and requires a program to be terminated unless the Secretary of Defense submits a written certification to Congress that certain criteria have been met, including that the new estimate of the program’s cost has been determined to be reasonable by the Director of DOD’s Office of Cost Assessment and Program Evaluation, and takes other actions, including restructuring the program. As we have previously reported, the Air Force has had significant difficulties developing OCX. The program’s cost and schedule baselines have been unstable and unexecutable since the first baseline was established in 2012. In total, there have been three OCX program baselines: 1. November 2012 original baseline at development start, 2. October 2015 rebaseline due to a schedule breach, and 3. September 2018 rebaseline prompted by a critical Nunn-McCurdy unit cost breach. Since 2012, reflecting the newest baseline and additional cost and schedule growth since the Nunn-McCurdy breach, the schedule has more than doubled and the costs have grown by approximately 68 percent. Figure 2 shows the three OCX baselines with their schedule and cost growth since the start of development. The National Defense Authorization Act for Fiscal Year 2017 required an independent assessment of OCX. The Act required an assessment of the Air Force’s ability to complete blocks 0 through 2 on a schedule necessary to transition OCX to full operation and an estimate of the cost, among other issues. The MITRE Corporation conducted the study and DOD provided it to congressional defense committees in December 2017. As a result of the 2016 Nunn-McCurdy unit cost breach, the program repeated the milestone associated with system development start and established new cost and schedule objectives and thresholds, conducted a baseline review of the schedule to verify the work necessary to complete the program, and received approval of the acquisition program baseline by the milestone decision authority—the Under Secretary of Defense for Acquisition and Sustainment. To support certification of OCX’s new baseline, in May 2017 the Air Force produced an $8.7 billion OCX service cost position for development, sustainment, and disposal. The Air Force service cost position was subsequently reaffirmed in 2018 by the Air Force and supported by an additional independent cost estimate from DOD’s Office of Cost Assessment and Program Evaluation in June 2018, which was approximately 3 percent higher in cost for the development portion. The Under Secretary of Defense for Acquisition and Sustainment selected the Air Force service cost position for the OCX baseline. In 2014, the Air Force identified root causes for OCX cost and schedule growth and concluded that the problems were driven by (1) incomplete systems engineering, (2) inadequate process discipline, and (3) difficulties implementing cybersecurity due to its complexity. We reported in 2015 that the program office paused development in late 2013 to fix what it believed were the root causes of development issues, and significantly increased the program’s cost and schedule estimates. Despite the pause to address root causes, problems persisted and in the same report we questioned whether all root causes—such as a persistently high software development defect rate—had been adequately identified, let alone addressed, and whether realistic cost and schedule estimates had been developed. We also found that the program was not following various acquisition best practices, such as the completion of a preliminary design review prior to development start. In 2015, we recommended that DOD assemble a task force to assess the OCX program and provide concrete guidance for addressing program problems, to determine root causes for OCX defects, and to establish a high confidence schedule and cost estimate, among other recommendations. DOD concurred with our four OCX-related recommendations and has taken some steps to implement some of them. However, to date, none have been fully implemented and they remain open. In 2016, DOD’s Director of Performance Assessments and Root Cause Analyses concluded that the root causes for OCX’s Nunn-McCurdy unit cost breach were (1) an unrealistic schedule driven by the need to sustain the GPS constellation, (2) an underestimation of the cost to fully implement information assurance, or cybersecurity, and (3) poor performance by both the government, caused by a lack of requisite software expertise, and Raytheon, caused by poor systems engineering that led to significant rework. We found and DOD’s 2016 root cause analysis has shown a significant and recurring cause of delays on the OCX program has been a lack of mutual understanding of the work between the Air Force and Raytheon. In December 2017, we found risks to the latest proposed (but not yet then approved) OCX schedule, noting that the schedule to which the program was working at that time (1) was built on certain unproven assumptions regarding planned coding and testing improvements, (2) had not yet undergone a baseline review to verify that the schedule incorporated all of the work required for program completion, and (3) did not yet include a number of changes that the Air Force needed to incorporate into the contract with Raytheon as modifications, which may lead to additional schedule slips. In 2017, we did not make additional recommendations for OCX because the Air Force had undertaken the COps and MCEU programs to provide interim capabilities to mitigate OCX delays. In 2016, Defense Digital Service—a DOD office established by the Secretary of Defense—engaged with the OCX program to suggest improvements to Raytheon’s software development practices. The office recommended that Raytheon change its software development approach to use an incremental development approach. This approach uses a continuous integration and testing process, where the software code is frequently integrated and tested so that defects are detected and addressed sooner. This is done through automation of the software development process, version control tools, and coordination between different teams building software. Traditional software development methods entail a more linear approach whereby each process is completed before proceeding to the next process in the sequence. By such an approach, the software development processes are completed prior to the testing of a full product before the product’s release to the end user. In 2016, DOD told the Air Force and Raytheon to utilize the new approach, which Raytheon began implementing in a series of seven phases. The first phase began in late 2016 and the last phase is scheduled to be in place by the end of 2019. According to the Air Force and Raytheon, through this new approach, the program aims to introduce efficiencies building software in several ways: 1. discovering defects in software code earlier; 2. reducing the number of defects; 3. reducing the amount of time it takes to repair defects; and 4. reducing the overall time to code, integrate, and test OCX software through automation for some aspects of the software development. OCX delivery, acceptance, and the ready to transition to operations decision will likely be delayed, potentially exceeding the April 2023 threshold date for completing the program. Actual development progress has been mixed, with some improvement to the pace of software development. However, the majority of the schedule reserve has been consumed and defect repairs are taking longer than assumed with significant work remaining. In addition, a number of new cost and schedule risks to OCX delivery have arisen since the program schedule was established. GAO’s schedule and cost estimating best practices recommend that the schedule assessment be periodically updated to reflect actual progress and new risks. To mitigate program optimism, GAO’s cost estimating best practices also state it is important to have an independent view of cost estimates and schedules. While the Air Force and the contractor periodically update their schedule estimates, no plans currently exist for further independent analysis of the full program schedule within DOD, and there is no requirement to do so. The OCX program has significant work remaining before OCX is operational, including years of integration and testing. Achieving the full program schedule requires two interrelated steps. First, in order to meet the program schedule there must be timely delivery by Raytheon and acceptance of the system by the Air Force. Second, there must be timely completion of government-run post-acceptance developmental testing. Once the Air Force determines that the developmental testing is completed, OCX will be ready to transition to operations, which ends the full program schedule. GPS operations will then be transferred from OCS to OCX. Figure 3 shows the major activities until the ready to transition to operations decision. OCX development is expected to continue for approximately 2 more years, after which Raytheon will submit a Material Inspection and Receiving Report (Form DD 250) at delivery. The Air Force will then evaluate OCX for acceptance. Air Force acceptance will be informed by numerous contractor-run developmental tests conducted to help the Air Force understand the maturity of the system. Air Force officials will use information from these contractor tests to inform their approval and complete acceptance. For example, the Air Force will review data and demonstrated system capabilities from the tests to determine whether OCX is ready for integration into the overall GPS. These tests have formal entrance criteria to demonstrate the system is ready for testing and exit criteria to ensure tests are successful before proceeding to the next activity. At the conclusion of contractor testing and delivery to the Air Force, the Air Force will inspect OCX over approximately 2 months before OCX is officially accepted. The Air Force will indicate acceptance by signing the Form DD 250. Currently, the period of performance under the contract ends June 30, 2021. Consequently, acceptance of the delivered OCX would need to occur prior to that date. After acceptance, Air Force program officials said OCX will go through government-run developmental testing—currently scheduled to last 7 months—that includes operator transition exercises and rehearsals of the system. According to OCX program officials, Raytheon will provide interim contractor support to address any defects or incomplete work as well as address any additional issues found during the planned 7 month post- acceptance developmental testing. According to program officials, the ground control operators—who have already been working and providing feedback—and training and readiness oversight personnel will continue to work with the new ground system to assess the system’s readiness through hands-on engagement with the installed system. At the end of this 7 month period, the Air Force will determine whether the system is ready to transition to operations. To make the ready to transition to operations decision, Air Force officials said the system must receive approval from different groups, including senior leadership within the Air Force. Once the decision has been made, the Air Force will transition ground control of the GPS satellite constellation from OCS to OCX. Additionally, after this transition, which completes the program schedule, OCX will undergo an operational test and evaluation period, which will support the Air Force’s separate operational acceptance decision for OCX. The OCX contractor’s delivery date is optimistic and much earlier than Air Force and independent projections. All government and independent analyses project OCX delivery will exceed June 2021 by at least 6 months, but still deliver in time to support the April 2023 threshold (latest acceptable) date for the full program schedule. However, meeting the ready to transition to operations threshold date depends on acceptance of OCX by September 2022, at the latest. This will allow for a planned 7 months of government-run developmental testing that must occur before April 2023. Numerous OCX schedule estimates were produced between December 2017 and January 2019. Table 1 indicates the estimator, date of the estimate, and the reason the estimate was completed. Figure 4 shows the results of the varying estimates for the start of OCX operations in months as measured from the beginning of calendar year 2019. The most recent independent OCX assessment of the delivery date is from the Defense Contract Management Agency in January 2019. That assessment estimates that Raytheon’s projected delivery and the cost at completion are both unrealistic based on staffing profiles, task movement, completion rates, baseline execution, and schedule performance metrics. The Defense Contract Management Agency projects that there are not enough cost and schedule reserves left to cover its own estimate to complete the work plus all of the identified risks. In fact, the Defense Contract Management Agency estimates Raytheon will need over $400 million more in cost reserves and that OCX will likely be delivered 11 months after June 2021. Actual development progress has been mixed. While the pace of software development has improved, implementing the new software development approach has been slower than expected. As a result, Raytheon has used the majority of its schedule reserve and delayed planned staff reductions, indicating that work is not being completed as quickly as planned. In addition, the schedule assumes improvements to software defect discovery have not all come to fruition and repair rates have not been achieved. Under its new software development approach, Raytheon is building and testing OCX software more quickly than under its previous approach. In 2018, the software development rate to build and test software was reduced in duration from 4 weeks or more to less than 7 hours on average—better than planned. Defense Contract Management Agency officials said that software development has improved compared to block 0 by having a better software development process in place. These officials cited in particular the improvement that has occurred with the introduction of software testing automation in some areas. The pace of software development is one area of many that is necessary to improve overall performance and achieve the delivery schedule. OCX program officials told us that the full implementation of the new software approach is foundational to the success of the program; failure to successfully implement the new approach on time would lead to cost growth and schedule delays. However, implementation of the new software approach has taken longer than planned, using a majority of the available schedule reserve. Defense Contract Management Agency officials found that since the current baseline was established, Raytheon consistently takes 5 months to perform 4 months of planned work. This has not yet delayed the delivery schedule because the program has been able to use cost and schedule reserves to cover the delays. Between April 2017, when the current schedule baseline was established, and January 2019, Raytheon used 4 of the 6 months of total schedule reserve. As of April 2019, Raytheon had approximately 26 months of work remaining until June 2021, but only 2 months of schedule reserve. As a result, there will not be enough time to complete OCX development and have it accepted by June 2021 unless the contractor significantly reduces its use of schedule reserve. Raytheon started using the new software approach on April 1, 2018 to improve software development, but implementation took longer than planned for six of the seven initial adoption phases, with two completing more than a year late. Some of the subsequent expansion phases are also experiencing delays. For example, phase 3 expansion was completed more than a year behind the planned schedule. Three other expansion phases are still in progress and scheduled to complete in mid- to late-2019. Raytheon’s divergence from the baseline staffing plan indicates that work is not being completed as quickly as planned, and more staff have been needed to prevent additional delivery delays. Raytheon had planned to reduce the number of staff working on the program from approximately 1,000 to 700 between the autumn of 2017 and the end of 2018. However, to maintain schedule, Raytheon delayed those reduction plans and increased staff by approximately 10 percent from January to August 2018. Figure 5 shows the difference between the staffing baseline and actuals for OCX between January 2018 and December 2018. Our analysis shows a gap between the January 2018 baseline planned staffing reduction and actual contractor staffing levels in each month from January to December 2018, collectively indicating an increase of approximately 29 percent above the plan. According to DOD’s Office of Cost Assessment and Program Evaluation officials, this increase is likely to continue through mid-2019. These officials estimated in their June 2018 independent cost estimate that contractor staffing levels will be higher than planned through May 2019 so that Raytheon can complete key software coding events. OCX program officials told us that the program has been able to afford the additional contractor staff as there are cost risks to support higher than anticipated staffing levels. They said that continued increases too far into 2019, however, will result in a breach of the cost threshold. Further, they said the increased contractor staffing is consistent with their priority on achieving the delivery schedule. The new software approach implementation will remain a cost and schedule risk until at least late 2019. At this time, the final expansion phase for the new software development approach is planned to be completed in order to support final testing of the entire system. Progress finding software defects sooner in development is also mixed. Raytheon officials told us that cost reductions are possible if they are able to find defects earlier, as this approach would lead to earlier defect resolution and reduce any backlog of defects. Further, they said there is efficiency in having the same developers repair software code that they created instead of different developers repairing the code later. In March 2018, Raytheon reported increasing the percentage of defects found in the phase of development where the defect was created from 27 percent in block 0 to 66 percent in blocks 1 and 2. However, an independent DOD assessment contradicted this improvement. DOD’s Office of Cost Assessment and Program Evaluation analyzed Raytheon’s defect discovery progress a few months into 2018 and found that after showing some initial improvement, the defect discovery rate dropped from approximately 53 percent to 24 percent. In addition, assessing progress discovering defects is now more difficult to compare with earlier development since Raytheon changed how it tracks and counts defects in 2018. According to OCX program officials, Raytheon now only counts a defect if it is repaired in a later phase. Therefore, if a defect is found and repaired in the same phase, it is not counted. As of November 2018, Raytheon officials said the predictive measure they are now using estimates the total number of defects expected while measuring the actual defects discovered. From this data, Raytheon found fewer total defects than it predicted, which Raytheon officials said will result in fewer defects likely to be discovered later in subsequent phases. Further, the defect repair-rate—or how many hours it takes to find and repair a defect—is currently projected to be higher than planned, placing additional pressure on the delivery schedule. According to Defense Contract Management Agency officials, the delivery schedule included defect repair assumptions that were unrealistic. That schedule assumed 30 hours to repair each defect. However, as of November 2018, Raytheon projects it will need 52 hours to repair each defect on average. For example, in one area of the program, defects required 61 hours to repair on average as of December 2018. Defense Contract Management Agency officials told us that they had concerns that the complexity of the defects was driving the time needed to repair them. They said that the more mature software created under the new software approach may be creating much more complex defects to repair than originally planned. This may lead to additional schedule delays as the time to repair these more complex defects may continue to be significantly higher than the delivery schedule assumed. More complete data on defects and defect repair rates will likely be available by the end of 2019, when the final expansion phases of the new software approach and more software development are completed. How do programs track risk as progress is made and risks evolve? A risk is an uncertain event that could Programs track risks to help manage and mitigate their effect on cost and schedule. completion requires knowing potential risks and identifying ways to respond to them before they happen—using risk management to identify, mitigate, and assign resources to manage risks so that their effects can be minimized. Raytheon’s estimate that OCX will be accepted by the end of June 2021 is further challenged because of significant identified risks that remain in the schedule and changeover in those risks in 2018. As of January 2019, Raytheon was tracking 48 risks it has identified with cost effects—26 with a moderate likelihood of occurring. For example, a moderate risk includes the possibility of finding more defects than planned, which could have both cost and schedule consequences. Other moderate risks include the possibility of software development taking longer to complete or needing to create more software code than planned. If realized, both of these risks have cost effects to pay for additional work and schedule effects to allow additional time to complete work. As of January 2019, Raytheon has no high risks that it tracks. There was also a significant amount of change in the risks themselves in 2018, as Raytheon added 27 new risks while closing 30. The majority of the risks that are currently tracked will not be realized or retired until late 2019, with at least one key risk of concern to the Air Force not realized or retired until 2020. As the program progresses, risks can (1) According to OCX program officials, approval to transition OCX to operations assumes a 7-month developmental test schedule after acceptance. As currently formulated, this period will be used to prepare for the transition from OCS to OCX via (1) transition exercises to train operators, (2) transition rehearsals to practice the actual handover from OCS to OCX, and (3) a 156-day integrated system test to verify OCX’s requirements, operational suitability, and readiness to enter operational testing. However, that 7-month duration may not be sufficient to conduct all of the activities that are necessary to verify OCX is ready to transition to operations. First, the head of the GPS Directorate’s Lead Development Test Organization, which plans and executes the 7-month developmental testing, said that there is some schedule risk because of concurrent activities that need to be accomplished, including crew rehearsals and other test events. Second, the content of the test period has not yet been finalized. The planned testing events will be reviewed and refined about 6 months before beginning the test as it becomes clearer what can be tested and what data will be available from the system. The test director and the OCX program manager are considering combining some test events and, if possible, starting some testing prior to acceptance. Third, the test director and the OCX program manager described a number of risks that could delay completion of developmental testing including (1) the late identification of issues requiring significant new software coding and retesting and (2) identification of new requirements that are not in the scope of the current effort. In addition, OCX program officials stated that neither they nor senior Air Force leadership would transition OCX to operations if the operators are not ready or requirements have not been verified. They also stated that there are numerous levels of review within the Air Force, and any of these decision makers can refuse to approve the transition of OCX to operations. As a result, according to OCX program officials it could take 5 to 7 months longer than planned, or potentially 14 months total, to complete developmental testing. In addition, experience with prior upgrades to the current GPS ground system indicates that the completion of developmental testing may require more time than the 7 months assumed in the schedule. Air Force Cost Analysis Agency officials provided us with data for two upgrades that were made to OCS, the existing operational ground system. Those upgrades took 11 and 8 months, respectively. The 11-month upgrade to OCS from 2006 to 2007 was for an effort that was significantly smaller in software size in comparison to the size of OCX, but similarly brought new capabilities to OCS related to the command and control of satellites. The 8-month upgrade to OCS from 2009 to 2010 also provided command and control of a new type of GPS satellite and enhanced security for the current GPS receiver cards. Figure 6 shows the different forecasts with 7- and 14-month developmental test periods as measured from the beginning of calendar year 2019. If the time doubles for the completion of post-acceptance government-run developmental testing, most OCX schedule estimates would exceed the program schedule threshold. GAO’s Cost Estimating and Assessment Guide (Cost Guide) and Schedule Assessment Guide (Schedule Guide) identify best practices for managing a program’s cost and schedule. According to these best practices, a well-planned schedule is a fundamental management tool that can help government programs use public funds effectively by specifying when work will be performed and measuring program performance against an approved plan. Typically, schedule delays are followed by cost growth. When this occurs management tends to respond to schedule delays by adding more resources or authorizing overtime. Therefore, a reliable schedule can contribute to an understanding of the cost effect if the program does not finish on time. Moreover, an integrated and reliable schedule can show when major events are expected, as well as the completion dates for all activities leading up to them, which can help determine whether a program’s parameters are realistic and achievable. Further, the Cost Guide states that, too often, programs overrun costs and schedule because estimates fail to account for the full technical definition, unexpected changes, and risks. The Cost Guide states that one of many challenges program managers face is too much optimism in the original estimate. The Cost Guide also states that because optimism is often prevalent, organizations will encourage goals that are unattainable by accentuating the positive. Because over-optimism potentially affects both cost estimates and schedules, an independent view and analysis is important to properly overcome this bias. An independent view also allows decision makers to react sooner and take steps to minimize any identified risks, like schedule delays. The following best practices recommend that the schedule estimate should be periodically updated to reflect (1) actual progress and (2) newly identified risks. Periodic Updates and Actual Progress: GAO’s Schedule Guide states that updating a schedule to reflect actual progress is important when assessing the realism of the initial schedule duration assumptions. Programs should make adjustments, if necessary, to the forecast of the remaining effort. Periodic Updates and Risk: GAO’s Schedule Guide states that prudent organizations recognize that uncertainties and risks can become better defined as the program advances and should conduct periodic reevaluations of risks. GAO’s Cost Guide states that program managers often do not sufficiently account for risks because they tend to be optimistic and because they believe in the original estimates for the plan without allowing for additional changes in scope, schedule delays, or other elements of risk. Since the current schedule was approved in September 2018, Raytheon has updated its delivery schedule estimate quarterly or as needed to reflect changes, and modifies the delivery and acceptance dates accordingly. Raytheon does not update the full program schedule because the government-run developmental testing is not included in its schedule estimate. OCX program officials said they are currently updating their program schedule estimate by incorporating Raytheon’s data through the end of 2018. No plans currently exist to conduct another OCX independent cost estimate—which would include a full, independent program schedule assessment—at the DOD-level, and currently there is no requirement to do so. An independent assessment of the schedule would normally be produced in conjunction with the statutory requirement to conduct another independent cost estimate at the next major program milestone. However, in September 2018 the milestone decision authority waived the requirement to hold the next major program milestone. DOD’s Office of Cost Assessment and Program Evaluation conducts independent cost estimates which account for a full program schedule when statutorily required. In addition, according to an official in that office, they also conduct only schedule assessments, without completing a full independent cost estimate, when requested by a program’s milestone decision authority. In June 2018, the Office of Cost Assessment and Program Evaluation provided the last full, independent cost estimate with a schedule assessment to the Under Secretary of Defense for Acquisition and Sustainment, the milestone decision authority, to support the decision to approve the OCX baseline. Officials from the Office of Cost Assessment and Program Evaluation said that they have not been asked by the OCX milestone decision authority to conduct another independent assessment. Without an independent schedule assessment, decision makers may lack updated information when determining whether to take new steps to avoid or mitigate additional delays. It is still unknown when OCX will be ready to support the command and control of the next generation of GPS satellites. While Raytheon has improved the pace of building and testing software, the majority of schedule reserve has already been consumed and work is not being completed as quickly and efficiently as the delivery schedule predicted. Once software development is complete, it must go through developmental testing. The schedule for this phase may also be optimistic as risks associated with competing activities have the potential to double the amount of time needed for testing. DOD will be in a better position to assess OCX’s progress and the potential for additional delays when the majority of its changes to its software development approach are completed at the end of 2019. At this time, however, while the program plans to continue assessing schedule progress, there are no plans in place for an independent schedule assessment. The program’s history has consistently shown program and contractor estimates to be optimistic and that independent assessments have provided useful insights about risks as well as past experience with similar activities. Our best practice guidance also emphasize that independent assessments are a necessary step to counter balance schedule optimism. Decision makers in DOD and Congress could use realistic knowledge about the schedule to either request or provide additional funds to complete the acquisition of OCX or develop contingency plans for delays. We are making the following recommendation to DOD: The Secretary of Defense should direct the Director, Office of Cost Assessment and Program Evaluation to conduct an independent schedule assessment of the full program schedule for the Global Positioning System’s next generation operational control system based on progress made through the end of calendar year 2019. (Recommendation 1) We provided a draft of this report to DOD for review and comment. In its written comments (reproduced in appendix II), DOD did not concur with our recommendation to conduct an independent assessment of the full OCX program schedule based on progress made through the end of calendar year 2019. DOD said that the Office of Cost Assessment and Program Evaluation conducted an independent cost and schedule estimate supporting the OCX program’s September 2018 system development milestone and that DOD subsequently funded OCX consistent with that estimate. Further, DOD said that the Office of Cost Assessment and Program Evaluation as well as the Defense Contract Management Agency continually assess the program’s ability to meet cost, schedule, and performance objectives. DOD also said the OCX forecast is currently holding to the government schedule, which is ahead of the Office of Cost Assessment and Program Evaluation’s independent cost estimate. Finally, DOD said senior executive reviews continue on a bi-annual basis to monitor the program’s progress. We continue to believe the recommendation is necessary. As stated in our report, DOD has not conducted an assessment of the full schedule since June 2018, since which time program risks have evolved. In addition, the other potential sources for schedule oversight suggested by DOD are limited in scope. The Defense Contract Management Agency does not look at the full OCX program schedule, as it examines the schedule only until contractor delivery. Officials from the Office of Cost Assessment and Program Evaluation said they do some programmatic monitoring of OCX, including on selected metrics, to inform DOD’s annual program and budget submission. But those metrics do not examine the full schedule that includes the developmental test period after delivery. The Office of Cost Assessment and Program Evaluation is in a position to independently assess the full OCX program schedule, as it has previously done, but only if DOD requests that it do so. We maintain that for complex programs, such as OCX, best practices state an independent view is necessary and that a periodic schedule assessment should be performed as progress is made and risks change. Given the mixed progress developing software, the number of new contractor risks discovered in 2018, the limited remaining schedule reserve held by the contractor (with at least two years of significant work remaining), and the potential for doubling the time frame for the planned 7-month post-acceptance government-run developmental testing period, we determined that the recommendation remains a prudent step. Such an assessment would help inform congressional and DOD decision makers as they consider what steps may be taken to address delays to the start of OCX operations and ensure the investments in needed new receivers are properly aligned. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Defense, the Secretary of the Air Force, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions concerns this report, please contact me at (202) 512-4841 or by email at chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. To determine the extent to which schedule risks may delay the delivery, acceptance, and approval for the operation of the Global Positioning System (GPS) next generation operational control system (OCX), we reviewed information relevant to OCX from Air Force GPS quarterly reports, senior management reviews, the program acquisition baseline, software development plans, monthly program management reviews that included schedule risks and progress updates, Air Force monthly acquisition reports, Air Force service cost position documentation, independent cost estimate documentation and analysis, earned value management data, Defense Contract Management Agency performance assessment reports, and slides and information provided by Raytheon Company (Raytheon), the prime contractor, on topics of our request. We reviewed the Air Force’s 2018 integrated baseline review results of the period until government acceptance and assessed the full program schedule—which includes the contractor’s schedule, government acceptance, and post-acceptance government-run developmental testing—until OCX is ready to transition to operations. We reviewed GAO’s best practice guides for cost estimating and assessment and schedule assessment to identify best practices for assessing a program’s cost and schedule and applied selected best practices. We also reviewed relevant reports and assessments focused on OCX completed by the government or required by Congress. We interviewed officials from the OCX program office and GPS Directorate, Defense Contract Management Agency, DOD’s Office of Cost Assessment and Program Evaluation, Air Force Cost Analysis Agency, the Lead Development Test Organization for the GPS Directorate, Defense Digital Services, Office of the Director, Operational Test and Evaluation, the MITRE Corporation, and Raytheon. We conducted this performance audit from November 2017 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, the following staff members made key contributions to this report: David Best (Assistant Director), Patrick Breiding (Analyst-in-Charge), Marie P. Ahearn, Pete Anderson, Brian Bothwell, Jonathan Mulcare, Andrew Redd, Karen Richey, Roxanna Sun, and Robin Wilson.", "summary": "The U.S. military and the public depend daily on GPS data. OCX, the ground system that will command and control next generation GPS satellites, is one of several interdependent systems the Air Force is developing to modernize GPS. OCX has been hampered by delays and $2.5 billion in cost growth since the program started in 2012. The Air Force set a new baseline for cost and schedule in 2018 after OCX breached its cost threshold in 2016. The National Defense Authorization Act for Fiscal Year 2016 contained a provision that the Air Force provide quarterly reports to GAO on the next generation GPS acquisition programs, and a provision that GAO brief the defense committees as needed. GAO provided numerous briefings from 2016 through 2018 and issued reports in 2016 and 2017. Continuing this body of work, this report focuses on the extent to which schedule risks may affect OCX delivery, acceptance, and approval for operation. GAO reviewed the Air Force's baseline review results, schedule risks, and progress, and applied selected best practices for cost and schedule management. GAO also reviewed OCX monthly management briefings and quarterly assessments, and interviewed officials from the OCX program office and Raytheon (the prime contractor), among others. The Global Positioning System's (GPS) next generation operational control system's (OCX) program schedule continues to be optimistic and, with significant development remaining, more delays are likely for delivery, acceptance, and operation. See the figure below for previous delays, cost growth, and the current baseline. Completing the full OCX program schedule requires (1) timely delivery by the contractor and acceptance by the Air Force and (2) an efficient completion of a planned 7-month government-run post-acceptance developmental testing. GAO found that there is potential for significant delays on both fronts. While there has been some improvement to the pace of software development, the rollout of the new development methodology has been delayed to a point where most of the contractor's schedule reserve has been used. Assumed improvements in how long it takes to repair software defects has not occurred as planned, placing additional pressure on the contractor's delivery date. Additionally, Air Force officials have acknowledged that the government developmental test period after acceptance could double in duration and delay operations further because of concurrency, test plan uncertainty, and risks of late discovery of problems. With approximately 2 years of work remaining before delivery, there is no plan to have the full schedule independently assessed. For complex programs, such as OCX, best practices state an independent view is necessary and that a periodic schedule assessment should be performed as progress is made and risks change. Such an assessment would help inform congressional and DOD decision makers as they consider what steps may be taken to address delays to the start of OCX operations and ensure the investments in needed new receivers are properly aligned. GAO recommends that DOD conduct an independent schedule assessment of the full program schedule at the end of 2019. DOD did not concur with the recommendation. GAO believes the recommendation remains valid.", "document_type": "gao"}
{"report": "Signed into law on May 9, 2014, the DATA Act required OMB, or an agency it designated, to establish a pilot program to facilitate the development of recommendations for (1) standardized reporting elements across the federal government, (2) elimination of unnecessary duplication in financial reporting, and (3) reduction of compliance costs for recipients of federal awards. To meet these requirements, OMB established a pilot program with two components—one that focused on federal grants and another on federal contracts (procurement). OMB designated HHS as the executing agency of the grants portion of the Section 5 Pilot with oversight from OFFM. OFPP was responsible for designing and leading the procurement portion of the pilot focusing on reporting of Federal Acquisition Regulation (FAR) procurement requirements. OFPP collaborated with the Chief Acquisitions Officers’ Council and GSA on specific aspects of implementation including the development of the Central Reporting Portal, a reporting tool which is intended to centralize FAR reporting. See figure 1 for a timeline of the activities undertaken by the grants and procurement portions of the pilot as well as deadlines required by the act. As part of our ongoing oversight of the DATA Act’s implementation, we have monitored OMB’s efforts to meet its statutory requirements related to the Section 5 Pilot. In April 2016, we reported on the design plans for the Section 5 Pilot. We found that HHS’s design for the grants portion of the pilot was generally on track to meet statutory requirements and partially adhered to leading pilot design practices. However, we also reported that the procurement portion was not on track to meet requirements, and that its plans did not follow leading pilot design practices. In response to a recommendation in our report, OMB revised its plan for the procurement portion to better reflect leading practices for pilot design identified in our April 2016 report. These changes included more fully documenting its data collection plans and including a sampling plan to meet diversity requirements for pilot participants. According to OMB staff, the ongoing work and related grants guidance resulting from the Section 5 Pilot reflects a broader strategy for reducing federal recipient reporting burden that is outlined in the President’s Management Agenda (PMA). Released in March of 2018, and led by the Executive Office of the President and the President’s Management Council, PMA is a strategy to modernize how federal agencies deliver mission outcomes and provide services in three key areas: (1) modern information technology; (2) data, accountability, and transparency; and (3) the workforce for the 21st Century. Several Cross-Agency Priority (CAP) goals include PMA’s milestones and activities. These CAP goals identify opportunities for multiple agencies to collaborate on government-wide efforts and report on goal progress quarterly. Two of these, CAP Goals 5 and 8, include strategies for reducing federal award recipient reporting burden. OMB staff told us that some of the findings from the Section 5 Pilot and recommendations from their subsequent report to Congress informed the focus of these CAP goals. For example, according to OMB staff, the grants portion of the Section 5 Pilot focused on identifying how changes in grants data collection and grant management may reduce federal recipient reporting burden. PMA CAP Goal 8 is described as building on these efforts by shifting the focus toward the life cycle of grants management and standardizing grants management activities using agile technology. We determined that the Section 5 Pilot fully met three of the DATA Act’s statutory requirements, substantively met one, and partially met two others. The Section 5 Pilot fully met the following statutory requirements: (1) that pilot data collection cover a 12-month reporting cycle; (2) timely issuance of OMB’s report to Congress in August of 2017 to select congressional committees; and (3) that the report to Congress contain a discussion of any needed legislative actions as well as recommendations related to automating and streamlining aspects of federal financial reporting to reduce the reporting burden of federal award recipients. We found that the pilot also substantively met the requirement that the pilot program include a combination of federal award recipients and an aggregate value of awards of not less than $1 billion but not more than $2 billion. Although the $122 billion in grants included in the pilot greatly exceeded the upper bound, this was principally a result of the decisions by OFFM and HHS to pilot different test models for reducing reporting burden, and to include a wide range of different types of grants. The total value of grant awards exceeded the amount envisioned by the act. OMB’s August 2017 report stated that the decision to go beyond the minimum requirement of testing one approach was made in the interest of achieving the DATA Act’s objective to identify ways to reduce reporting burden as well as the effect this decision would have on the aggregate value of grants sampled. We believe that the pilot substantively met this requirement and did not identify any negative effects related to the larger aggregate value of grants, contracts, and subawards included in the grants portion of the pilot. We found that the approach followed by OMB and HHS furthered the broader objective identified by this section of the act. In addition, we determined that the pilot partially met two of the act’s requirements. The first of these requirements concerns the act’s requirement that OMB’s report to Congress include a description of the data collected, the usefulness of the data provided, and the cost to collect pilot data from participants. The report that OMB issued to Congress in August 2017 included information on the first two of these but only partly addressed the third. Specifically, it contained cost information for only the grants portion of the pilot, stating that the cost associated with executing this portion during fiscal years 2015 through 2017 was more than $5.5 million. The report did not contain any cost information on the procurement portion of the pilot. The DATA Act also required that OMB issue guidance to agencies for reducing reporting burden for federal award recipients—including both grantees and contractors—but the guidance subsequently issued only pertained to the grants community. We determined that OMB only partially met this requirement. On September 5, 2018, OMB issued M-18- 24: Strategies to Reduce Grant Recipient Reporting Burden. Among other things, this memorandum contained guidance to federal agencies making the SF-424B form optional based on findings from the grants portion of the pilot. Form SF-424B is used by grantees to document assurances regarding their compliance with a wide range of rules and regulations. Figure 2 summarizes our assessment. As the agency designated by OMB to execute the grants portion of the Section 5 Pilot, HHS developed and analyzed six “test models” to determine if adopting the proposed changes would contribute to the pilot program’s objectives of reducing reporting burden and duplication. These test models examined a variety of grant reporting issues that HHS had identified as presenting challenges. All but one of the test models, the Common Data Element Repository (CDER) Library 2, based their findings on data collected from grantees. The text box below provides high-level summaries of each of the six models. Additional details on the approach followed for each model, as well as reported results, can be found in appendix II. OMB’s August 2017 report to Congress on the findings of the Section 5 Pilot contained three broad recommendations and stated that OMB plans to take action on these recommendations. These recommendations covered (1) standardizing core data elements, (2) eliminating duplication through auto-population of data, and (3) leveraging information technology open data standards to develop new tools across the federal government. We found that evidence from the grant test models supported all three recommendations for streamlining federal reporting discussed in the report. For example, OMB recommended that its staff standardize core data elements used for managing federal financial assistance awards based on reductions in administrative burden experienced in the CDER Library 1 test model. In another example, four test models supported OMB’s recommendation for increased use of data auto-population from existing federal data sources as a way to reduce duplication in reporting. Findings from the grants portion of the Section 5 Pilot also provided support for government-wide efforts to streamline reporting and reduce recipient reporting burden. These include OMB’s memorandum M-18-24: Strategies to Reduce Grant Recipient Reporting Burden, which discusses efforts to automate and centralize grant management processes. Among other things, M-18-24 requires that federal agencies evaluate the systems and methods currently used to collect information from grant recipients to eliminate duplicative data requests. OMB staff confirmed that M-18-24 incorporates findings from some of the test models of the grants portion of the pilot such as the Single Audit test model, which examined reducing duplicative reporting of grant recipients’ data. The efforts to reduce duplicative reporting in M-18-24 also align with OMB’s recommendation in its August 2017 report to Congress to eliminate unnecessary duplication in reporting by leveraging information technology that can auto-populate from existing data sources. In addition, OMB staff told us that findings from the grants portion of the pilot contributed to broader, government-wide initiatives related to federal reporting. For example, according to OMB staff, the three recommendations from the August 2017 report to Congress are reflected in CAP Goal 8 of the President’s Management Agenda, which focuses on results-oriented accountability for grants. These OMB staff also told us that findings from the grants portion of the pilot informed two CAP Goal 8 strategies. For example, the CAP Goal 8 grants management strategy focuses on standardizing grants management business processes and data. OMB developed a comprehensive taxonomy for core grants management data standards that is currently available for public comment. In addition, a second strategy focuses on incorporating a risk- based performance management approach to metrics in grant award operations to determine low-risk and high-value federal awards. CAP Goal 8 also states plans to streamline the 2019 Single Audit Compliance Supplement to focus on requirements that inform grant award performance. Unlike the grants portion of the pilot, the procurement portion did not result in data collection that could be used for an evidence-based assessment of ways to reduce reporting burden. OMB’s Office of Federal Procurement Policy (OFPP) sought to assess five test models that, according to the report to Congress, were essential to centralized procurement reporting. However, the pilot did not fully test any of the hypotheses associated with those test models. The reasons for not testing the hypotheses included a lack of contractor participation and a lack of iterative and ongoing stakeholder participation and engagement throughout the course of the pilot. See appendix III for additional information regarding the various procurement test models, associated hypotheses, and additional details regarding our assessment. The procurement portion of the pilot focused entirely on the development and testing of a central reporting portal to consolidate FAR reporting requirements. According to OFPP staff, the pilot intended to eventually identify ways to centralize a wide range of reporting requirements that contractors currently meet through decentralized methods. Contractors must report many types of information depending on the contract. Toward that end, OFPP, with the assistance of GSA, created a procurement reporting website called the Central Reporting Portal. To test the efficacy of this portal for reducing burden, OFPP initially decided to examine how well it handled a specific FAR reporting requirement—the reporting of payroll data in accordance with the Davis-Bacon Act. According to pilot plans, Davis-Bacon reporting requirements were selected because they were identified by contractors as “pain points” during initial stakeholder outreach conducted in 2014 and 2015. OFPP planned to collect and analyze 1 year of weekly Davis-Bacon wage reporting data from at least 180 contractors through the Central Reporting Portal to identify how centralized reporting might reduce contractor reporting burden. However, during the 12-month procurement data collection period, no contractors agreed to submit their Davis-Bacon data as part of the pilot. Consequently, OFPP did not collect any wage data. Despite OFPP stating in its plans and reiterating to us as late as September 2017 that it expected to be able to secure at least 180 pilot participants, only one contractor expressed interest in reporting its Davis-Bacon information using the portal. This contractor withdrew from the pilot before submitting any data through the Central Reporting Portal. OFPP staff told us they were aware of the potential for low pilot participation for Davis- Bacon reporting when pilot testing began in February 2017 because contractors already had established processes for fulfilling the highly complex Davis-Bacon reporting requirements, and pilot participation was optional. According to GSA contracting staff, the one contractor who initially expressed interest ultimately decided not to participate because the format in which the contractor tracked and reported payroll data was incompatible with that used by the pilot portal, resulting in additional burden. However, it was not until August 2017—approximately 7 months into its year-long data collection period—that specific steps were taken to address the fact that the procurement portion of the pilot had not collected any data from Davis Bacon contractors. During this period OFPP did not conduct pilot outreach activities with the contractors, who were key to successful implementation of the pilot. OFPP staff told us that at the time of the pilot launch they learned that contractors were interested in having the Central Reporting Portal be able to communicate with third-party payroll reporting systems to automate reporting. OFPP staff said that although they are exploring this possibility, it was not a capability that was included as part of the pilot. Had this type of feedback on stakeholder needs been obtained sooner, OMB could have explored the feasibility of adding this capability to the portal or engaged in communication with stakeholders to develop alternate approaches that might have persuaded more contractors to participate. The usefulness of iterative and ongoing communication is recognized by the Standards for Internal Control in the Federal Government. Those standards state that management should use quality information to achieve its objectives, and that management should collect quality information by engaging with stakeholders through iterative and ongoing processes and in a timely manner. In this case, key stakeholders include relevant agencies, contracting officials, and contractors using the system. OFPP’s plan for the procurement portion of the pilot recognized the importance of stakeholder engagement stating that, to include a diverse group of recipients in the pilot, they should identify eligible participants for the pilot, conduct outreach to identify participants, and repeat this process as necessary until they achieved the sample necessary to test the Central Reporting Portal. However, as previously stated, no contractors agreed to submit their Davis-Bacon data as part of the pilot. Therefore, OFPP did not repeat this process until the pilot obtained the necessary sample size. Such interactions could have provided important information on contractors’ needs and concerns that OFPP could have used to inform their decisions regarding the pilot’s implementation. In November 2017, OFPP expanded the type of data accepted by the pilot to include hydrofluorocarbon (HFC) reporting, a new FAR reporting requirement. However, this choice had limitations in its suitability for providing useful data for testing the hypotheses of the five procurement test models. Unlike Davis-Bacon reporting, where contractors submit weekly reports, HFC is an annual reporting requirement for contractors that emit HFC gases over a certain threshold. The Central Reporting Portal is the only location where contractors can submit HFC reporting. For the purposes of the pilot, the Central Reporting Portal accepted HFC submissions from November 2017 through February 2018. During the pilot, 11 HFC annual reports were submitted to the portal (see figure 3). As a result of the small number of reports collected, OMB collected much less data than it had initially expected to receive to test the capabilities of the Central Reporting Portal. If the procurement portion of the pilot had been executed as planned, it could have theoretically resulted in 9,360 Davis-Bacon submissions for analysis. A larger data set of contractors’ experiences using the Central Reporting Portal could have informed OMB’s decision-making process through analysis of more, and potentially more varied data. In addition to the small number of submitted HFC annual reports, the decision to switch to using HFC data had another limitation. These data could not be used to examine changes in reporting burden as a result of using the Central Reporting Portal. This is because HFC reporting was a new reporting requirement, and as such, it did not have an established reporting process to use as a point of comparison to assess changes in reporting burden. The objective of the procurement pilot was to assess how centralized reporting can reduce reporting burden. This objective could not be achieved without data on the existing reporting burden. Evidence from the procurement portion of the pilot did not support OMB’s government-wide recommendations for reducing reporting burden in its August 2017 report to Congress. As previously stated, OMB’s report to Congress included three recommendations that focused on (1) standardizing core data elements, (2) eliminating duplication by using data auto-population, and (3) leveraging information technology open standards to develop new tools. As support for the first recommendation, the report stated that results from the procurement pilot test models demonstrated that standard data elements—coupled with uniform data adoption—and the ability to centrally collect and share information reduces administrative burden. Since the procurement portion of the pilot did not gather or analyze any pilot data from the Davis-Bacon participants, OMB did not assess the extent to which the ability to centrally collect data actually reduces burden. Recommendation two stated that support from the procurement test model demonstrated that recipient burden is reduced when identical data can be entered once in one place and reused. However, the HFC data collection process did not reuse data when capturing information and did not have the ability to auto-populate data. HFC data collection was the only part of the procurement portion of the pilot that collected information that could have been used to inform this recommendation. According to OFPP staff, the Davis-Bacon portion of the portal had the capability to auto-populate data. However, no Davis-Bacon data were collected that would have allowed quantification of the effects of reusing data on reporting burden. OMB stated that support for the third recommendation included data and information collected from the pilot. Although there was some consultation with stakeholders during initial planning and design of the procurement portion of the pilot and the early development of the portal, the pilot did not actually collect any data from either Davis-Bacon contractors or through the HFC portion of the pilot in the data gathering and analysis portion of the pilot related to this recommendation. In August 2018, OMB announced plans to expand the use of the Central Reporting Portal for FAR reporting, stating that the portal allows contractors to report data to one central location. OFPP staff told us that they are considering centralizing a third FAR requirement using the portal in the future but have not yet determined what that will be. As discussed above, the procurement portion of the pilot did not collect sufficient data to test the effect of the portal on reporting burden. In addition, the plan for the procurement portion states that OFPP intended to analyze feedback on pilot data collection and, depending on that feedback, decide whether to expand the pilot to other FAR reporting requirements. However, the pilot did not collect any such feedback to inform its determination to expand the Central Reporting Portal in the future. As a result, OFPP has limited information regarding issues that could affect expanded use of the Centralized Reporting Portal. In the absence of such information, it is difficult for OFPP to determine whether continued or expanded use of the Central Reporting Portal will reduce reporting burden, and which additional FAR requirements, if any, to include. To reduce the burden and cost of reporting for recipients of federal funds, Congress included specific provisions in the DATA Act to encourage OMB to take a deliberate and evidence-based approach toward developing guidance for federal agencies in this area. The Section 5 Pilot offered OMB a valuable opportunity—namely, to test a variety of methods and techniques at a small scale before applying them more widely. Such a process may enhance the quality, credibility, and usefulness of evaluations in addition to helping to ensure that time and resources are used more effectively. Similar to what we found when we analyzed the design of the Section 5 Pilot in 2016, our review of its implementation and the results it produced found differences between the grant and procurement portions. OMB and HHS designed and executed a robust grants portion of the pilot that tested several different approaches for reducing the reporting burden experienced by federal grant recipients. The resulting findings were used to develop OMB’s government-wide recommendations, and to inform two subsequent goals in the 2018 President’s Management Agenda related to reducing recipient reporting burden. In contrast, OMB did not fully implement the procurement portion of the pilot consistent with its plans. The procurement portion did not collect data to test the hypotheses associated with any of its five test models, and therefore could not provide empirical support for either OMB’s government-wide recommendations or guidance related to reducing reporting burden. Among the factors responsible for this were the lack of Davis-Bacon contractor participation and OMB’s inability to find a suitable alternative. OMB has announced its intention to expand centralized reporting for FAR requirements across government. In the absence of timely information regarding the needs and concerns of stakeholders, OMB faces the risk of experiencing implementation challenges similar to those it experienced during the pilot. Although the use of a centralized reporting portal could ultimately prove useful for reducing burden, the lack of information from stakeholders—including the contractors who would use it—raises concerns about the future success of plans for expanding the Central Reporting Portal. The Director of OMB should ensure that information is collected regarding how centralized reporting of procurement requirements might reduce recipient reporting burden—including input from stakeholders such as contractors through an iterative and ongoing process—to inform OMB’s planned expansion of the Central Reporting Portal. We provided a draft of this report to OMB, HHS, and GSA for review and comment. HHS and GSA informed us that they had no comments. OMB provided technical comments, which we incorporated as appropriate. OMB neither agreed nor disagreed with our recommendation. We are sending copies of this report to the appropriate congressional committees, The Secretary of Health and Human Services, The Acting Director of OMB, the Administrator of GSA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at 202-512-6806 or sagerm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. GAO staff who made key contributions to this report are listed in appendix IV. This report assesses the extent to which (1) the Section 5 Pilot met the statutory requirements of the act, (2) the grants portion of the Section 5 Pilot demonstrated changes in federal award recipients’ reporting burden, and (3) the procurement portion of the Section 5 Pilot demonstrated changes in federal award recipients’ reporting burden. To assess the extent to which the pilot met statutory requirements we reviewed section 5 of the Federal Funding Accountability and Transparency Act of 2006, as amended by the Digital Accountability and Transparency Act of 2014, to determine the legal requirements set forth in the act pertaining to establishing, designing, and executing the Section 5 Pilot. We compared these requirements to documents from the Office of Management and Budget (OMB) and designated agencies. These documents included pilot plans for the grants and procurement portions of the pilot, OMB’s August 2017 report to Congress, M-18-23: Shifting from Low-Value to High-Value Work and M-18-24: Strategies to Reduce Grant Recipient Reporting Burden. We also interviewed staff from agencies involved in administering and executing the pilot on how they carried out their responsibilities. These agencies included the Department of Health and Human Services (HHS), OMB’s Offices of Federal Financial Management (OFFM) and Federal Procurement Policy (OFPP), and the General Services Administration (GSA). To assess the extent to which the grants portion of the Section 5 Pilot demonstrated changes in federal award recipients’ reporting burden, we reviewed HHS’ plans. We analyzed the plans compared to information collected from the various test models throughout the pilot. The data we assessed included survey data and analyses. We also assessed whether statements on changes in grantees’ reporting burden made in OMB’s August 2017 report to Congress were supported by documentation. We did this by verifying the statements against supporting information. We determined that the pilot data we reviewed were reliable for the purposes of our work by reviewing the data, tracing them back to underlying agency source documents, and interviewing relevant agency staff. We also interviewed OFFM staff and HHS officials on how the grants portion of the pilot was executed. To assess the extent to which the procurement portion of the pilot demonstrated changes in reporting burden, we reviewed OMB’s plans and compared them to actions OMB took to execute the pilot. We compared OMB’s actions to execute the procurement portion of the pilot against criteria identified in Standards for Internal Control in the Federal Government. We viewed a demonstration of the Central Reporting Portal tool for reporting Davis-Bacon and hydrofluorocarbon (HFC) submissions. GSA developed the portal and OFPP provided oversight for the portal’s development. We also reviewed documentation including HFC reporting submissions made through the portal. In addition, we interviewed OFPP staff, GSA officials responsible for administering the portal, and three contracting officials from GSA who were assigned to participate in the Davis-Bacon component of the procurement portion of the pilot regarding their actions related to implementing the procurement portion of the pilot. We conducted this performance audit from November 2017 to April 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix provides detailed information regarding the test models from the grants portion of the Section 5 Pilot. The Common Data Element Repository (CDER) Library is an online repository for federal grants-related data standards, definitions, and context. The library is intended to be an authorized source for data elements and definitions for use by the federal government and for recipients reporting grant information. Hypothesis: If grant recipients are provided with definitions of data elements through the CDER Library, then they will be able to accurately complete forms in a timely manner. Methodology: The Department of Health and Human Services (HHS) divided test model participants into two groups to read a scenario based on the grants lifecycle and complete a data collection tool. The first group used the CDER Library to complete the data collection tool while the second group used all other available sources to complete the data collection tool. After completion of the data collection tool, test model participants filled out a survey about their experiences using the CDER Library. Test Model Metrics: Accuracy and completeness of captured data within a period of time and survey results. Example of Test Model Results: On average, test model participants that completed a data collection tool using the CDER Library scored 11 percent higher in the accuracy of information requested and, on average, spent 6 fewer minutes when completing the tool. Number of Test Model Participants: Fifty-nine. The CDER Library 2 Test Model focused on identifying duplication in grant forms and data elements across the federal government based on the data standards, definitions, and context within the CDER Library 1. Hypothesis: If duplication across forms can be identified using the CDER Library, then agencies can update or reduce forms to reduce grant recipient burden. Methodology: HHS conducted an internal analysis of SF-424 form families, using the CDER Library, to identify duplication in data elements to determine which forms could be consolidated. Test Model Metrics: Number of duplicative fields within form families and across forms for selected federal entities Example of Test Model Results: The internal analysis conducted by HHS identified 371 instances of data element duplication across 10 agency grant funding applications when using standardized data elements from the CDER Library 1. Number of Test Model Participants: Not Applicable; the CDER 2 Library Test model did not collect information from test model participants because the test model was an internal document review. The CDER Library 2 test model tested the utility of the data element definitions within the CDER Library 1. The Consolidated Federal Financial Report Test Model focused on examining the potential early validation of consolidated CFFR data and potential future streamlining of the close-out process by allowing the submission of Federal Financial Report (FFR) data in one system, rather than in multiple entry systems. Hypothesis: If grant recipients can enter complete FFR information systematically through one entry point instead of multiple different avenues and that information could be shared electronically from that point forward, then grant recipient burden will be reduced and data accuracy will be improved. Methodology: HHS surveyed Administration for Children and Families grant recipients on their experience submitting a consolidated FFR via HHS’s Payment Management System, and grantees on their perceptions of the process for using a consolidated FFR through facilitated discussions. Test Model Metrics: Survey results. Example of Test Model Results: Sixty-four percent of the CFFR test model participants reported that submitting their FFR through a single system would result in reduced reporting time. In addition, 65 percent of the CFFR test model participants believed using the payment management system for submitting FFR data would improve the accuracy of the information they submitted. Number of Test Model Participants: One-hundred fifteen tested the pilot environment and 30 participated in the facilitated discussions. The Single Audit Test Model consisted of (1) an audit and opinions on the fair presentation of the financial statements and the Schedule of Expenditures of Federal Awards; (2) gaining an understanding of and testing internal control over financial reporting and the entity’s compliance with laws, regulations, and contract or grant provisions that have a direct and material effect on certain federal programs (i.e., the program requirements); and (3) an audit and an opinion on compliance with applicable program requirements for certain federal programs. The Single Audit Test Model focused on reducing reporting of data on duplicative forms. Hypothesis: If grant recipients do not have to report the same information on duplicative forms—for example, the SEFA compared to the Single Audit Report Package and Data Collection Form—then grant recipients’ burden will be reduced. Methodology: HHS collaborated with the Office of Management and Budget’s Office of Federal Financial Management and the Department of Commerce Federal Audit Clearinghouse (FAC) to create a pilot environment for test model participants to submit key portions of a modified Standard Form—Single Audit Collection. HHS conducted two focus groups with test model participants subject to the Single Audit. The first focus group discussed and completed a survey on the new form. The second group, a sample of test model participants who are subject to perform a Single Audit submitted the existing form in the FAC pilot environment, completed a separate data collection form similar to the new form, and completed a survey on the effectiveness and burden of the new form. Test Model Metrics: Focus group feedback and survey results. Example of Test Model Results: All test model participants with access to the Single Audit’s pilot environment believed the upload feature for reporting requirements could decrease duplication in required grant reporting. Number of Test Model Participants: Thirteen tested the pilot environment and 123 participated in facilitated discussions. This model focused on the feasibility of developing a standardized Notice of Award (NOA) to reduce reporting burden and facilitate access to standardized data needed to populate Single Audit information collection. Hypothesis: If grant recipients have a standardized NOA for federal awards, then grant-reporting burden may be reduced for recipients by standardizing access to data needed to populate information collections. Methodology: HHS divided test model participants into two groups and completed a data collection tool. The first group completed the data collection tool using three standardized NOAs, while the second group completed the data collection tool using three non-standardized NOAs. After completion of the data collection tool, test model participants self-reported their respective times to complete the data collection tool. They also filled out a survey about the standardized NOA’s impact on reporting burden and provided input on elements to include in a standardized NOA. Test Model Metrics: Self-reported form completion time, accuracy, and survey results. Example of Test Model Results: Test model participants with access to the standardized NOA coversheets spent an average of 3 minutes less when completing the test model’s data collection tool. Number of Test Model Participants: One-hundred four. The Learn Grants Test Model is a website on Grants.gov that summarizes and provides links to new and important grants information such as policies, processes, funding, and other information needed throughout the grants life cycle. The website intended to make it easier for stakeholders to find, learn about, and apply for federal grants and promote the standardization of grants terminology and data. Hypothesis: If grant recipients are supplied with grants life cycle information in one website, then they will have increased access to grants resources and knowledge of the grants life cycle process. Methodology: HHS developed a grants knowledge quiz from information on the Learn Grants website. HHS administered the knowledge quiz to test model participants in two phases. First, test model participants completed the knowledge quiz using existing knowledge and without the Learn Grants website. Next, test model participants completed the knowledge quiz with access to the Learn Grants website. HHS compared the results from both knowledge quizzes. After completion of the knowledge quiz, test model participants completed a survey on the usefulness of the Learn Grants website and its impact on increasing knowledge quiz scores. Test Model Metrics: Knowledge quiz accuracy and survey results on the usefulness of Learn Grants website. Example of Test Model Results: Test model participants experienced an average 10 percent (one quiz point) increase in their grant knowledge quiz scores when using the Learn Grants website. New grantees who participated in the test model also reported that the Learn Grants website provided useful grants information. Number of Test Model Participants: Fifty-seven. Appendix III: Assessment of Test Models in the Procurement Portion of the Section 5 Pilot Hypothesis not tested. Hypothesis: Verification of FAR standards for post award reporting will confirm the value of existing data standards and reduce variations that will, in turn, reduce contractor burden and cost. Original plan (Davis-Bacon): OFPP planned to execute this test model through focus groups. According to OFPP, no focus groups were conducted. Revised Strategy (HFC): This hypothesis could not be tested through HFC reporting because it was a reporting requirement without an existing reporting method through which to compare reporting burden. Hypothesis not tested. Original Strategy (Davis-Bacon): OFPP planned to test this hypothesis by gathering data on the time it takes to submit reporting data through the Central Reporting Portal and outside of the portal, with self-reported data from contractors. According to OFPP, data were not collected due to a lack of participation in the Davis-Bacon portion of pilot. Revised Strategy (HFC): This hypothesis could not be tested through HFC reporting because it was a reporting requirement without an existing reporting method through which to compare reporting burden. access (proof of concept) Hypothesis: If contractors can enter FAR-required reporting data systematically through one entry point instead of multiple different avenues, and that information can be shared electronically with appropriate individuals, then contractor burden will be reduced and data access improved. Assessment Rationale Original plan (Davis-Bacon): OFPP planned to test this hypothesis by gathering data on the time it takes to submit reporting data through the Central Reporting Portal and outside of the portal, with self-reported data from contractors. OMB also planned to conduct guided discussions. According to OFPP, data were not collected due to a lack of participation in the Davis-Bacon portion of pilot. Revised Strategy (HFC): This hypothesis could not be tested through HFC reporting because it was a reporting requirement without an existing reporting method with which to compare reporting burden. Hypothesis not tested, but metric associated with test model was met. Hypothesis: If interfaces can be built to support access to other reporting systems, contractor burden will be reduced. Original plan (Davis-Bacon): According to OFPP staff, the Davis-Bacon part of the Central Reporting Portal was able to provide prepopulating of data by interfacing with other reporting systems or drop down menus for all reporting fields. However, it could not demonstrate that such prepopulation resulted in a reduction of contractor burden. Revised Strategy (HFC): This is not applicable for HFC reporting which is reported through open fields. Although OFPP did not actually test the hypothesis associated with this test model, it did meet the metric that it had associated with the test model in its pilot plan. That metric is to develop prepopulating capabilities in the Central Reporting Portal by interfacing with other reporting systems. In addition to the contact named above, Peter Del Toro, Assistant Director; Silvia Porres-Hernandez, Analyst-in-Charge; Jazzmin Cooper; and Jimmy Nunnally made major contributions to this report. Also contributing to this report in their areas of expertise were Michael Bechetti, Jenny Chanley, Mike LaForge, Carl Ramirez, Stewart Small, Andrew J. Stephens, James Sweetman Jr., and Tatiana Winger.", "summary": "The DATA Act required OMB or a designated federal agency to establish a pilot program to develop recommendations for reducing recipient reporting burden for federal grantees and contractors. The grants portion of the pilot tested six ways to reduce recipient reporting burden while the procurement portion focused on testing a centralized reporting portal for submitting reporting requirements. This report follows a 2016 GAO review on the design of the pilot. This report assesses the extent to which (1) the pilot met the statutory requirements set out in the DATA Act, (2) the grants portion of the pilot demonstrated changes in reporting burden, and (3) the procurement portion demonstrated changes in reporting burden. GAO reviewed statutory requirements, pilot plans, agency data and reports and interviewed OMB staff and officials from HHS and GSA. In response to requirements of the Digital Accountability and Transparency Act of 2014 (DATA Act), the Office of Management and Budget (OMB) led implementation of a pilot program, known as the Section 5 Pilot, aimed at developing recommendations for reducing recipient reporting burden for federal grantees and contractors. The pilot program met many, but not all, of its statutory requirements. For example, the act required OMB to issue guidance to agencies for reducing reporting burden for federal award recipients (including both grantees and contractors) based on the pilot's findings. OMB partially met this requirement because the guidance it issued only applied to grants. The pilot program consisted of two parts, which differed considerably in both design and results: The grants portion, administered by the Department of Health and Human Services (HHS), examined six approaches for reducing grantee reporting burden and found positive results related to reductions in reporting time as well as reduced duplication. HHS incorporated ongoing stakeholder input during the pilot, and its findings contributed to government-wide initiatives related to federal reporting and reducing grantee-reporting burden. The procurement (contracts) portion of the pilot, led by OMB with assistance from the General Services Administration (GSA), did not collect sufficient evidence to determine whether centralizing procurement reporting through a single web-based portal would reduce contractor reporting burden—a key objective of the pilot. The pilot planned to test the portal by collecting weekly Davis-Bacon wage data from a minimum of 180 contractors, potentially resulting in thousands of submissions over a year. However, in the end, the pilot did not result in any Davis-Bacon data due to lack of contractor participation and the absence of iterative and ongoing stakeholder engagement. Subsequently, OMB expanded the pilot to include hydrofluorocarbon (HFC) reporting but received only 11 HFC submissions. (See figure.) In addition, HFC reporting was not suited for assessing changes in reporting burden because it was a new requirement and thus no comparative data existed. OMB plans to expand its use of the portal for additional procurement reporting requirements but still does not have information from stakeholders that could help inform the expansion. GAO recommends that the Director of OMB ensure that information is collected regarding how centralized reporting of procurement requirements might reduce recipient reporting burden—including input from stakeholders such as contractors through an iterative and ongoing process—to inform OMB's planned expansion of the Central Reporting Portal. OMB neither agreed nor disagreed with the recommendation but provided technical comments, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "In our April 2019 report, we found that all nine of the selected agencies have policies that are generally consistent with OSTP’s guidance for the principles of scientific integrity that we reviewed: foundations of scientific integrity in government and professional development of government scientists and engineers. OSTP’s guidance describes several components for each of these principles, which the selected agencies addressed either (1) through their scientific integrity policies, (2) in related policies, or (3) through related actions. For example, when addressing the components of foundations of scientific integrity in government, NOAA’s scientific integrity policy states that the agency will ensure the free flow of scientific information online and in other formats, consistent with privacy and classification standards, and in keeping with other Commerce and NOAA policies. In another example, NASA’s scientific integrity policy states that NASA facilitates the free flow of scientific and technological information among scientists and engineers, between NASA staff and the scientific and technical community, and between NASA employees and the public. The policy goes on to cite additional NASA policies on dissemination of information and public access to data. Similarly, we found that all nine selected agencies addressed all of the components of the principle professional development of government scientists and engineers. For example, EPA’s policy states that the agency encourages publication and presentation of research findings in peer-reviewed, professional, or scholarly journals and at professional meetings. NIST’s scientific integrity policy states that the agency supports scientists’ full participation in professional or scholarly societies, committees, task forces, and other specialized bodies of professional societies, with proper legal review and approval. The policy goes on to cite separate NIST guidance for staff on how to seek approval for memberships and participation in professional organizations. We found in our April 2019 report that the nine selected agencies have taken some actions to help achieve the objectives of their scientific integrity policies in the three areas we reviewed—communicating information to staff, providing oversight, and monitoring and evaluating performance. First, according to our analysis, seven of the nine selected agencies have taken some actions to educate and communicate to staff about their scientific integrity policies, and two have not. Specifically, FE and NIST have not provided scientific integrity training for staff, according to officials, or taken other actions to promote their scientific integrity policies with staff. Under the 2007 America COMPETES Act, civilian agencies that conduct scientific research are, among other things, required to widely communicate and readily make accessible to all employees their scientific integrity policies and procedures. According to FE and NIST officials, the agencies made their policies available to staff on their websites and believed no additional actions were needed. By taking action to educate and communicate their scientific integrity policies to staff through, for example, regular training, these agencies would have better assurance that employees have the information, skills, and competencies they need to help achieve agency scientific integrity objectives. We recommended the Secretary of Energy and Director of NIST take action to educate and communicate the agencies’ polices to staff through, for example, regular training. In DOE’s written comments on a draft of our report, reproduced in our final report, the department explained that it will designate a scientific integrity official to be responsible for leading and coordinating with other offices across DOE to develop measures to educate and communicate to staff about scientific integrity policies. In Commerce’s written comments, reproduced in in our final report, NIST identified ways it plans to provide training to its staff. Second, we found that eight of the nine selected agencies have designated scientific integrity officials, or the equivalent, who are responsible for overseeing the agencies’ implementation of their scientific integrity policies. FE, which follows DOE’s policy, does not have a scientific integrity official or the equivalent. DOE’s scientific integrity policy states that the Secretary of Energy will designate a scientific integrity official for the department. DOE officials explained that the scientific integrity official has not been designated because the scientific integrity policy was implemented in January 2017, as the administration was changing, and that the current Secretary has not yet designated a scientific integrity official. We recommended the Secretary of Energy should establish steps and a time frame for designating a scientific integrity official to oversee the department’s scientific integrity activities. In DOE’s written comments on a draft of our report, reproduced in our final report, the department concurred with our recommendation and estimated that it would address the recommendation by the end of 2019. Third, we found in our April 2019 report that four of the nine selected agencies—ARS, EPA, NASA, and NIH—monitor and evaluate the performance of their activities under their scientific integrity policies, or have plans to do so. The remaining five agencies—FAA, FE, NIST, NOAA, and USGS—have, for different reasons, not done so. Standards for Internal Control in the Federal Government states that management should design control activities to achieve objectives and respond to risks, which may include establishing activities to monitor performance measures and indicators. By establishing mechanisms to effectively monitor the implementation of their scientific integrity policies, agencies may be better positioned to evaluate and measure whether their scientific integrity policies are achieving their objectives and, where necessary, improve their implementation. We recommended in our April 2019 report that the five agencies develop mechanisms to regularly monitor and evaluate implementation of their scientific integrity policies, including mechanisms to remediate identified deficiencies and make improvements where necessary. All five agencies agreed with our recommendation and responded as follows: In a May 2019 letter from DOT, the department identified several mechanisms it plans to implement by the end of March 2020. In DOE’s written comments on a draft of our report, the department said that its scientific integrity official will have the responsibility to lead in developing procedures to monitor and evaluate implementation of DOE’s policy. In Commerce’s written comments, NIST stated that, beginning in fiscal year 2019, the agency will review implementation of its policy at least annually and make recommendations to the Director of NIST as to whether any improvements are needed. In Commerce’s written comments, NOAA stated that it will identify additional metrics for monitoring and evaluating its policy. The Department of the Interior’s written comments stated that the department plans to implement a biennial scientific integrity survey of USGS employees, beginning in 2020, to gauge scientific integrity policy awareness and effectiveness at USGS, among other things. Seven of the nine selected agencies—ARS, EPA, FAA, NIH, NIST, NOAA, and USGS—have specific, documented procedures for identifying and addressing alleged violations of their scientific integrity policies. Although the details of agencies’ procedures may vary, the procedures generally include five basic steps: (1) report allegation, (2) screen allegation, (3) investigate allegation, (4) respond to violation, and (5) appeal decision (see fig. 1). In contrast, two of the nine selected agencies—FE and NASA—do not have specific, documented procedures for identifying and addressing alleged violations of their scientific integrity policies. In March 2009, the President issued a memorandum on scientific integrity that states that each agency should have in place procedures to identify and address instances in which the scientific process or the integrity of scientific and technological information may be compromised. FE, which follows DOE’s scientific integrity policy, does not have specific procedures because DOE has not established any. DOE and FE officials said staff can report allegations to a supervisor, the whistleblower ombudsperson, or the U.S. Office of Special Counsel (OSC). Similarly, NASA officials said employees can report allegations through their chain of command, such as to a supervisor, for investigation on a case-by-case basis. However, without documented procedures for identifying and addressing alleged violations of their scientific integrity policies, DOE and NASA do not have assurance that all staff have a clear understanding of how to report allegations and that investigations will be conducted consistently. We recommended the Secretary of Energy and Administrator of NASA develop documented procedures for identifying and addressing alleged violations of their scientific integrity policies. In DOE’s written comments on a draft of our report, the department stated that it will be the responsibility of the scientific integrity official to lead, and coordinate with other elements of the department, in developing procedures for identifying and addressing alleged violations of its scientific integrity policy and estimated completing actions in June 2020. In written comments from NASA, the agency stated that it will develop documented procedures for identifying and addressing alleged violations of its policy and estimated completion by October 2020. Chairwoman Stevens and Chairwoman Sherrill, Ranking Member Baird and Ranking Member Norman, and Members of the Subcommittees, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this statement, please contact John Neumann, Managing Director, Science, Technology Assessment, and Analytics, at (202) 512-6888 or neumannj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Rob Marek (Assistant Director), Wyatt R. Hundrup (Analyst in Charge), Cheryl Harris, and Douglas G. Hunker. Also contributing to this testimony were Eric Charles and Ben Shouse. Additional staff who made contributions to our April 2019 report are identified in that report. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "This testimony summarizes the information contained in GAO's April 2019 report, entitled Scientific Integrity Policies: Additional Actions Could Strengthen Integrity of Federal Research ( GAO-19-265 ). The nine selected agencies GAO reviewed have taken various actions to help achieve the objectives of their scientific integrity policies in three areas: Educating staff. Seven of the nine agencies have taken some actions to educate and communicate to staff about their policies, consistent with the 2007 America COMPETES Act. However, the Office of Fossil Energy (FE), which follows the Department of Energy's (DOE) policy, and the National Institute of Standards and Technology (NIST) have not taken action. Providing oversight. Eight of the nine agencies have a designated official, or the equivalent, to oversee implementation of their scientific integrity policies. However, FE does not have such an official because DOE has not appointed one and currently has no plans or timeframe to do so, although DOE policy states that DOE will appoint an official for oversight. Monitoring and evaluating implementation. Four of the nine agencies have monitored and evaluated implementation of their scientific integrity policies, consistent with federal standards that call for such control activities. However, FE, the Federal Aviation Administration (FAA), NIST, the National Oceanic and Atmospheric Administration (NOAA), and the U.S. Geological Survey (USGS) have not undertaken such activities. Seven of the nine agencies have specific, documented procedures for identifying and addressing alleged violations of their scientific integrity policies. Although the details of agencies' procedures vary, they generally include the steps shown below. However, two agencies—FE, following DOE's policy, and the National Aeronautics and Space Administration (NASA)—do not have documented procedures for identifying and addressing alleged violations. A 2009 presidential memo on scientific integrity states that agencies should have procedures to identify and address instances in which the scientific process or the integrity of scientific and technological information may be compromised. Without procedures, FE and NASA do not have assurance that their staff understand how to report allegations and that investigations are conducted consistently.", "document_type": "gao"}
{"report": "In response to Executive Order 13781, USAID established the Transformation Task Team (T3) in June 2017 to plan and lead the agency’s reform efforts. As noted in a previous GAO report, USAID launched several internal reform efforts and participated in a joint State- USAID redesign process during mid-2017, which resulted in a joint reform plan. USAID also developed a supplemental reform plan that focused on issues internal to USAID. State and USAID submitted these plans to OMB in September 2017. In January 2018, USAID suspended its participation in the joint State–USAID redesign process and continued to plan and implement its own internal reforms. According to USAID, its reform efforts are intended to support its bilateral partners to become more self-reliant and capable of leading their own development, with the ultimate goal of ending the need for foreign assistance. To achieve this goal, USAID identified five objectives, referred to as “desired outcomes,” as the basis for its reform efforts. The five objectives are: (1) establish metrics and approaches to help host country recipients of assistance become more self-reliant; (2) restructure bureaus and offices to strengthen the organization’s core capabilities; (3) advance national security interests; (4) improve human capital processes; and (5) maximize taxpayer investments in foreign assistance. According to USAID officials, OMB generally approved the USAID reform plans and associated projects by March 2018. Figure 1 shows the key events in the initial phases of USAID’s reform efforts up to the point OMB provided this approval. In developing our June 2018 report to assist Congress, OMB, and agencies in assessing agency reform plans, we reviewed our prior work on key practices for organizational transformations; collaboration; government streamlining and efficiency; fragmentation, overlap, and duplication; and high risk and other long-standing agency management challenges. The resulting report includes 58 key questions to aid in assessing reform efforts. (See app. II for a complete list of the 58 key questions.) The questions are organized into four broad categories and 12 subcategories, as shown in table 1. These subcategories encompass the key practices that we used to assess USAID’s reform efforts. For the purposes of this report, we determined that the subcategory of Workforce Reduction Strategies was not applicable to our assessment because USAID is not undertaking workforce reductions as part of its reform effort. USAID’s reform efforts consist of a total of 32 reform projects—31 projects being implemented by USAID’s Transformation Task Team (T3) and an additional Human Resources Transformation project that predates USAID’s other reform efforts. As shown in table 2, as of July 2019, USAID has completed 19 projects and is implementing 12 others, all of which USAID intends to complete by 2021. The task team also has one project still in the planning phase. In order to develop and implement the 32 reform projects, USAID has identified approximately $33 million in estimated costs associated with its reforms up through April 2019. According to USAID, this total includes about $3 million to develop the T3 reform efforts in fiscal year 2018 and approximately $6 million to implement its reform efforts over a period of 2 years, which USAID assumes will cover fiscal years 2019 and 2020. In addition, USAID estimated that, as of April 2019, it has expended about $24 million in fiscal year 2017–2019 funds for human resource efforts that are associated with its ongoing Human Resources Transformation project. As shown in table 3, USAID’s reform efforts generally addressed nine of the key practices that we previously identified as critical to the success of agency reforms, and its reform efforts partially addressed two others. USAID determined the appropriate role of the federal government by considering the private sector and governments’ ability to manage responsibility for and invest their own resources into foreign development and humanitarian assistance programs. Our prior work shows it is important for agencies engaged in reforms to reexamine the role of the federal government in carrying out specific missions and programs, policies, and activities by reviewing their continued relevance and determining whether the federal government is best suited to provide that service or if it can be provided by some other level of government or sector more efficiently or effectively. In line with the USAID Administrator’s vision of ending the need for foreign assistance, USAID has developed several projects under its “Journey to Self-Reliance” objective to increase bilateral partner countries’ ability to plan, finance, and implement solutions to solve their own development challenges. Beginning in mid-2017, USAID launched a process to identify a set of third-party metrics for assessing a country’s level of self-reliance. In June 2018, USAID announced the identification of 17 metrics to capture a country’s overall commitment and capacity for self-reliance. The publicly available metrics cover areas such as open and accountable governance; inclusive development; economic policy; and the relative capacities of the government. Starting in fiscal year 2019, USAID produced 136 “country roadmaps,” or tools for measuring each low- and middle-income country’s overall level of self-reliance through its performance on the 17 metrics. USAID is using the country roadmaps as a tool to inform strategic decision-making and resource allocation processes, better focus USAID’s investments, and indicate when a recipient country should be considered for a “strategic transition” to a new partnership model with the U.S. government. For example, USAID identified Albania as a country to pilot this concept, which envisions a new partnership model for a country exhibiting an advanced level of self-reliance and the development of a strategy and plan for how to shift to this new model over time. In addition, USAID’s “Journey to Self-Reliance” efforts include a project to expand its engagement with the private sector. According to a USAID document, donor agencies are unable to fulfill their goals for sustainable development on their own; in contrast, the private sector has the scale and resources to address the complexity of challenges that developing countries face in becoming self-reliant. In December 2018, USAID released a new “Private Sector Engagement Policy” intended to increase and deepen the collaboration of USAID staff and its partners with the private sector across all areas of the agency’s work. USAID involved its employees and key stakeholders in its internal reform efforts. Our prior work has shown that it is important for agencies to directly and continuously involve not only their employees but also key stakeholders in the development of major reforms. USAID has involved its employees in its reform efforts through a variety of means. For example, since 2017, USAID reform leaders have conducted town-hall style meetings with employees in Washington, D.C., and in the field. USAID reform leaders have also briefed senior management, bureau- and office-level leadership, and mission directors about reform efforts. In addition, they have communicated reform updates in the agency’s internal newsletter and have informed employees of reform projects through multiple venues, such as web-based seminars and agency notices. USAID has also involved key stakeholders, including Congress and State, in its reform efforts. The Administrator has testified before Congress, and USAID officials have briefed Congress about the status of the reform efforts. USAID also submitted reorganization proposals to congressional committees for review and approval. Moreover, USAID engaged with State officials at the senior and working levels on several of its reform projects, including its self-reliance metrics, strategic transitions, and workforce flexibility and mobility projects. However, T3 officials noted that its engagement with State has been hindered by leadership challenges at State, including the lack of a single official or entity at State with responsibility for coordinating with USAID on reform efforts. In our prior work, we found a lapse in State’s leadership focus on reform efforts, and we recommended that State establish a dedicated team to manage the implementation of all reform projects that the Secretary of State decides to pursue. USAID’s T3 used various sources of evidence and data to design its reform plans, including recommendations made by external organizations and employee feedback. Our prior work has shown that agencies are better equipped to address management and performance challenges when managers effectively use data and evidence, such as from program evaluations and performance data that provide information on how well a program or agency is achieving its goals. USAID developed its reform projects based on research and recommendations from various sources, including GAO, the USAID Office of Inspector General, USAID’s Advisory Committee on Voluntary Foreign Aid, think tanks, and coalitions of organizations focused on international development. For example, USAID’s reform proposal to merge and restructure its Offices of U.S. Foreign Disaster Assistance and Food for Peace into the Bureau for Humanitarian Assistance stems, in part, from the results of an in-depth, external study that USAID commissioned in 2016, which entailed significant consultations with internal and external stakeholders as well as data analysis. As another example, USAID’s “Explore Delivery of Human Resources Operations” project was based, in part, on two GAO reports recommending steps to improve the collection of contract data. In May 2017, State launched a “listening tour” intended to gather ideas and feedback from State and USAID employees on the joint State-USAID redesign process. As a key component of this outreach effort, State hired a contractor to design and administer a confidential, online listening survey, which was sent to State and USAID employees. The listening survey identified pain points, recommendations, and themes that informed USAID’s reform plans. For example, USAID’s projects aimed at reorganizing its structure address a listening tour theme regarding the need to better align its bureau and office functions with USAID’s core mission. In another example, some of USAID’s human resource reform projects address another listening tour theme related to the need to support USAID employees in focusing more of their attention on achieving strategic priorities and less time on inefficient and burdensome administrative tasks. According to USAID, it sought to reduce or better manage fragmentation, overlap, and duplication through multiple reform efforts, including its restructuring projects, its consolidated framework for private sector engagement, and efforts aimed at redefining and rationalizing roles and responsibilities in areas such as countering violent extremism and civilian- military coordination. In our prior work, we have identified actions that agencies could take to achieve greater efficiency or effectiveness by reducing or better managing programmatic fragmentation, overlap, and duplication. In July and August 2018, USAID sent to various congressional committees for approval a series of initiatives to restructure its bureaus and offices to streamline operations and gain efficiencies. USAID included a proposal to restructure the Office of the Administrator by adding two associate administrators. According to a USAID document, this change would allow the administrator to more effectively manage the complexity of USAID’s work and reduce the number of entities directly reporting to the administrator from 27 to 11. One of the new associate administrators would manage USAID’s relief, response and resilience functions, and the other would manage the agency’s strategy, management, and operations. The congressional committees had not approved all of these proposals as of June 2019, according to USAID. As of June 2019, according to USAID, the congressional committees had approved five of the seven reorganized bureaus proposed by USAID: the Bureau for Humanitarian Assistance; the Bureau for Resilience and Food Security; the Bureau for Conflict Prevention and Stabilization; the Bureau for Development, Democracy, and Innovation; and the Bureau for Asia. Two other proposed bureaus had not yet received approval from all of the committees: the Bureau for Management and the Bureau for Policy, Resources, and Performance. Figure 2 shows USAID’s proposed changes to its headquarters organizational structure. According to USAID documents, reorganizing these bureaus is in part intended to reduce fragmentation, overlap, and duplication, as well as to make the agency more functionally aligned and field-focused. For example, USAID states that the Bureau for Humanitarian Assistance will reduce duplication and fragmentation by unifying humanitarian assistance and eliminating the distinction between food and non-food emergency response, eliminating confusion in the field, and providing beneficiaries and partners with one cohesive USAID platform and voice on humanitarian assistance. As another example, USAID states that the Bureau for Policy, Resources, and Performance would consolidate USAID’s policy, budget, and performance functions, which are currently divided among five bureaus and offices. USAID’s reform efforts address several high risk and long-standing management challenges, including a project to specifically address external audit findings and implement auditors’ recommendations. Our prior work noted that reforms improving the effectiveness and responsiveness of the federal government often require addressing long- standing weaknesses in how some federal programs and agencies operate. For example, agency reforms provide an opportunity to address the high risk areas and government-wide challenges that we have called attention to and that are vulnerable to fraud, waste, abuse, and mismanagement, or are in need of transformation. USAID has undertaken multiple projects to address high risk areas and long-standing challenges. USAID T3’s “Addressing the Audit Backlog” project was specifically designed to review, enhance, and revise USAID’s management of audit engagements and recommendations by eliminating the agency’s backlog of unresolved audit recommendations, developing and implementing practices that would strengthen current programs, and reducing the potential for a future backlog. In this way, USAID intends to save taxpayer dollars by preventing and responding to fraud, mismanagement, wasteful practices, and other challenges identified in the audits. USAID reported that it had eliminated the backlog of unresolved audit recommendations as of May 2018. As of early April 2019, USAID had implemented 75 of GAO’s 86 recommendations from fiscal years 2015 through 2018. In addition, several other reform projects address high risk areas and long-standing management challenges identified by the USAID Office of Inspector General (OIG). For example, USAID’s “Working in Non- Permissive Environments” project addresses challenges USAID faces working in insecure, inaccessible, or unstable environments. USAID OIG identified developing strategies to work effectively in non-permissive and contingency environments, as one of the five top management challenges for USAID in fiscal year 2017. USAID’s leadership has demonstrated focus on and attention to the planning and conduct of USAID’s reform efforts. Our prior work shows that a dedicated team of high-performing leaders within the agency should lead organizational transformations, such as agency reforms. USAID has demonstrated leadership at various levels to manage and guide the agency’s reform efforts. For example, USAID’s Administrator first outlined his vision of USAID’s mission as being focused on ending the need for foreign assistance in August 2017, and USAID’s reform efforts are aimed at operationalizing the Administrator’s vision to end the need for foreign assistance. USAID’s Administrator has had visible and continuous involvement in USAID’s reform efforts, including through informing various congressional committees, on multiple occasions, of ongoing developments with USAID’s reform process. USAID has designated leaders who are responsible for the day-to-day management of USAID’s reform efforts. In June 2017, USAID’s Acting Administrator established the Transformation Task Team (T3) to lead the agency’s response to Executive Order 13781 and the subsequent guidance from OMB. T3 is led by a Coordinator who concurrently serves as the Assistant to the Administrator in USAID’s Bureau for Policy, Planning, and Learning. The Coordinator told us that he meets with the USAID Administrator on a regular basis to report the status of USAID’s projects. T3 also includes seven deputy coordinators who are accountable for the progress of all of the projects within a desired outcome as well as 24 project managers who lead project implementation. The T3 Coordinator indicated that the size of his team will decrease over time as it hands over management of USAID’s reform projects to bureau-level leaders. USAID also assigned Senior Leader Champions to each of its reform projects. The champions provide strategic guidance and act as the representational “face and voice” of the project to Congress and the agency. Further, USAID also established a Transformation Advisory Council made up of senior leaders of USAID who have provided strategic guidance to USAID’s reform efforts since October 2017. The council is chaired by the T3 Coordinator and made up of Senior Leader Champions, mission director liaisons, T3 leadership, and other standing members. The Transformation Advisory Council meets to discuss the progress of reform projects, ensure cross-project coordination, and to resolve any duplication or dependencies. USAID has developed and maintained a system for managing and monitoring its reform process. We have previously reported that organizational transformations must be carefully and closely managed by developing an implementation plan with key milestones and deliverables to track and communicate implementation progress, among other actions. In May 2018, USAID T3 issued a task order for a contractor to help ensure that USAID has the capacity to manage the planning and implementation of USAID’s reform efforts. The contractor is responsible for providing project and performance management support. Such support included tracking USAID’s reform projects, providing summaries and executive reports on the progress of USAID reform projects, and also knowledge management, including the retention of key documents and information related to project and performance management. The contractor established a data tracking system that contains project end dates and deliverables to track the progress of reform implementation. The system notes which projects are on schedule, delayed, or complete. The contractor has also generated periodic executive reports that outline next steps for implementation reform and provide updates organized by USAID’s five reform objectives. USAID T3 has developed guidance for transferring responsibility for project implementation to the appropriate bureaus and offices. The guidance details who in the bureau will be responsible and accountable for the project, resources that will be needed to initiate and complete handover of the project, and the future end state of the projects, among other items. As of July 2019, USAID had completed bureau handover plans for 24 T3 reform projects. USAID has demonstrated transparency over its reform efforts through publicizing reform-related information on its website, including fact sheets on its projects. USAID has also publicly released several of its reform deliverables. For example, USAID made its “Journey to Self-Reliance” portal available on its external website. Through the portal, viewers have access to USAID’s Fiscal Year 2019 Country Roadmaps and can download a wide range of supporting resources on the “Journey to Self- Reliance” effort and the methodology that underpins this effort. USAID’s reform efforts generally addressed two interrelated subcategories of strategic workforce planning by instituting policies to manage employee engagement and to improve employee performance management. These policy initiatives were part of USAID’s broader effort to create a human resource services system that, according to USAID documents, will support a modern workforce in carrying out USAID’s mission. Our prior work has found that increased levels of employee engagement—generally defined as the sense of purpose and commitment employees feel toward their employer and its mission—can lead to better organizational performance and can sustain or increase levels of employee engagement and morale, even as employees weather reorganizations and other difficult external circumstances. Our prior work also found that performance management systems—which are used to plan work and set individual employee performance expectations, monitor performance, develop capacities to perform and to rate and incentivize individual performance—can help the organization manage employees on a daily basis and provide supervisors and employees with the tools they need to improve performance. USAID developed and began implementing its Human Resources Transformation project prior to the start of the current reform effort led by T3. This project includes objectives and initiatives to both promote employee engagement issues and establish a performance management system during the 5-year transformation. USAID created a project management office to plan and carry out between three and five initiatives associated with each of the Human Resources Transformation project’s objectives and a performance monitoring plan to track the progress of each initiative. As noted in figure 3, the three Human Resources Transformation objectives and the associated intermediate results called for by the project address both employee engagement and employee performance management issues. For example, Transformation Objective 3, “Agency Culture and Workplace Enhanced,” promotes employee engagement by calling for an agency workplace enhanced by a stronger focus on the culture of accountability with a workforce reflecting the diversity of America’s population. The project is also using Federal Employee Viewpoint Survey (FEVS) data to periodically gauge employees’ feedback and level of engagement on the reform efforts. Moreover, USAID noted in its April 2019 Human Resources Transformation performance monitoring plan that USAID intends to measure the effectiveness of its efforts to improve employee engagement by assessing the extent to which those efforts increase employees’ positive response rates to human resources service- and delivery-related questions over the generally low baseline rates set by the FEVS 2016 survey response (ranging from 10 percent to 26 percent positive response rates). The monitoring plan noted that USAID expects to increase the positive response rates to these questions on the FEVS to upwards of 74 percent by 2021. Furthermore, one of the intermediate results associated with Transformation Objective 2, “Agency Workforce Prepared for Today and the Future,” includes an effort to establish and uphold a performance management system in areas such as provision of feedback, professional development, and career advancement. T3 also initiated six projects associated with its “Empower People to Lead” objective that incorporate some of the Human Resources Transformation project efforts to improve employee engagement and implement a performance management system. For example, T3’s project on “Managing Human Capital Talent” is developing new automated tools to transition the paper-based Foreign Service and Civil Service performance management and evaluation processes into online evaluation systems administered electronically. As of July 2019, these tools include an automated Foreign Service assignment tool and a Civil Service performance management system and automated tool. However, USAID delayed its expected completion date for these Foreign Service and Civil Service tools from the end of December 2018 to March 2019 and August 2019, respectively. Further, T3’s “Leveraging Foreign Service National Talent” project expects changes in job satisfaction- related survey scores, over time, will help USAID measure the success of a reform project aimed at empowering the agency’s Foreign Service Nationals workforce. Our prior work indicates that agency reforms should clearly identify what an agency is trying to achieve by establishing outcome-oriented performance measures that enable the agency to assess the extent to which projects are achieving progress toward reform goals. Moreover, T3 guidance states that, as responsibilities for project implementation are transferred to bureau- and office-level units, project-level managers should develop performance indicators to measure progress. While USAID has established high-level goals associated with its reform efforts, such as ending the need for foreign assistance, it has established outcome-oriented performance measures for only four of its reform efforts. Table 4 below provides examples of outcome-oriented performance measures for those four reform projects. USAID has not established outcome-oriented performance measures that would enable it to gauge the effectiveness of the remaining reform efforts. For example, USAID’s five reform objectives—(1) Journey to Self-Reliance, (2) Strengthen Core Capabilities, (3) Advance National Security, (4) Empower People to Lead, and (5) Respect Taxpayer Investments—are not tied to outcome-oriented performance measures. In explaining why they had not developed outcome-oriented performance measures for all projects, USAID T3 officials indicated that thus far they have focused their efforts on establishing outputs (e.g., products and services) for the reform projects. Establishing outcome-oriented performance measures for its reform projects would enhance USAID’s ability to assess the effectiveness of its reform efforts. USAID documents and officials demonstrate that the agency is developing an agency-wide strategic workforce plan in support of its ongoing reform efforts, but the plan and its associated workforce planning tools were not ready to implement as of July 2019. Strategic workforce planning is an essential activity that an agency needs to conduct to ensure that its human capital program aligns with its current and emerging mission and programmatic goals, and that the agency is able to meet its future needs. Our prior work also indicates the importance of preceding any staff realignments or downsizing with strategic workforce planning so that changed staff levels do not inadvertently result in skills gaps or other adverse effects that could increase use of overtime and contracting. USAID has taken a number of steps since 2017 to develop an agency- wide strategic workforce plan both prior to and during the current reform effort, including developing staff realignment plans as part of its process for standing up the proposed new bureau structures. However, USAID has not yet developed or implemented the data collection and measurement tools that it has identified as necessary to gauge current workforce capabilities, assess staffing needs arising from the proposed reorganization, and identify ways to close gaps arising from changes in workforce requirements. USAID documents note that such tools could allow USAID to achieve its goal of hiring the right talent, at the right time, for the right duration. USAID is using both the Human Resource Transformation project and two of T3’s projects to develop a strategic workforce plan and associated tools: USAID developed and began implementing the Human Resources Transformation project prior to the start of the current reform effort with the expectation that by 2020 the agency would have the organizational structure and workforce characteristics that support achievement of USAID’s mission. This new structure would include an optimally sized workforce with an effective mix of all USAID employee types created through the use of a new workforce planning model. Project documents note, however, that developing this planning model in turn would require developing a Workforce Planning Tool to define workforce baselines and existing assets, identify future workforce needs, assess gaps, and build capacity where needed. In June 2016, USAID’s 2016–2021 Human Resource Transformation Strategy and Action Plan stated that developing this model would be difficult but nevertheless estimated that implementing this effort would require no more than 2 years. However, USAID officials noted that the Human Resources Transformation efforts did not “fully begin” until 2018. T3 is implementing two projects associated with its objective titled “Empower People to Lead.” First, T3’s Manage Human Capital Talent project instituted an Employee Portal to provide all direct-hire employees access to their human resources data in one centralized online location. According to USAID documents, this project is also developing for management an automated assignment, performance management, and workforce planning tools, including separate automated planning, performance, and assignment tools for its Civil Service and Foreign Service personnel. The agency originally intended to implement these tools by the end of calendar year 2018. USAID’s April 2019 performance monitoring plan indicates that the tools—particularly the workforce planning model that USAID describes as a human-capital data analytics system to automate various standardized and ad hoc reports and access previously unconnected personnel data sources—will not be available before the end of fiscal year 2019. Second, T3’s “Workforce Flexibility and Mobility” project is focused on implementing a demonstration project, the “Adaptive Personnel Project,” to replace non-career, program- funded positions with an excepted-service management system. The “Adaptive Personnel Project” is to be launched as a pilot project in two USAID bureaus in fiscal year 2020. As of April 2019, USAID documents and USAID and employee union officials noted that the strategic workforce plan has not yet been completed. Moreover, the April 2019 Human Resources Performance Monitoring Plan notes that the workforce planning tool needed to gauge current capabilities and close gaps is not yet deployed and in use due to competing programmatic and budgetary priorities. In addition, USAID’s T3 project data tracking system indicates that the agency has delayed the implementation of the projects needed to establish baselines and create pilot projects until late 2019 or later in order to focus on broader strategic workforce planning objectives, such as the Strategic Workforce plan and “Adaptive Personnel Project.” The lack of a strategic workforce plan may limit USAID’s efforts to estimate how its proposed reorganization will affect future staffing needs. For example, USAID officials indicated in 2018 that the proposed reorganization of its headquarters bureaus was intended to be “staff neutral.” Its congressional notification pertaining to this reorganization projected no net increase in its total combined headquarters workforce level of 3,262 employees. Nevertheless, in its Fiscal Year 2020 Congressional Budget Justification, USAID identified a need for 40 additional Civil Service positions to “refocus Washington bureaus and offices toward being effective service providers to the field consistent with the vision of ending the need of foreign assistance.” USAID requested $7.2 million to fund those positions in the restructured bureaus. Without a strategic workforce plan, USAID cannot determine whether its current or planned workforce requirements align with its reform and reorganization objectives. USAID is entrusted with managing billions of dollars in foreign assistance funding, and USAID leadership recognizes that reforming its internal operations and programming is integral to achieving its mission. In developing and implementing its reform efforts, USAID addressed many key practices that are critical to ensuring a successful agency reform or reorganization, such as using data and evidence and providing leadership focus and attention. Specifically, USAID’s reform efforts generally addressed nine of the 11 key practices we assessed. However, taking additional steps in two areas could further improve its reform efforts. First, while it established goals and desired outcomes for its reform efforts, it has not yet generally established outcome-oriented performance measures necessary to assess the effectiveness and success of these efforts. Second, while USAID has been developing a strategic workforce plan since 2017, it has yet to complete this plan, which includes developing the associated workforce planning tools to identify the staff needed to meet existing and emergent program demands associated with its transformation goals. Addressing these gaps could help USAID better position itself to make long-term and sustainable improvements in its efficiency and effectiveness. We are making the following two recommendations to USAID: The Administrator of USAID should establish outcome-oriented performance measures to assess the effectiveness of USAID’s reform projects. (Recommendation 1) The Administrator of USAID should ensure that the agency completes a strategic workforce plan necessary to support its reform efforts. (Recommendation 2) We provided a draft of this report to USAID, State, and OMB for review and comment. We received comments from USAID, which are reprinted in appendix IV. USAID concurred with our recommendations. We also received technical comments from USAID and State, which we incorporated in our report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Administrator of USAID, the Secretary of State, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6881 or BairJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. We performed our work under the authority of the Comptroller General to conduct work to assist Congress with its oversight responsibilities. This report (1) examines the status of the U.S. Agency for International Development’s (USAID) reform efforts and (2) assesses the extent to which USAID has addressed key practices and considerations critical to the successful planning and implementation of agency reform efforts. The scope of our review was limited to USAID’s internal reform efforts and did not include government-wide or interagency reform proposals, such as those referenced in the Office of Management and Budget’s Delivering Government Solutions in the 21st Century report. For both objectives, we reviewed USAID’s reform plans, proposals, and related documents and interviewed officials involved in USAID’s reform efforts. We interviewed USAID officials on the USAID Transformation Task Team, including the task team Coordinator and Deputy Coordinators. We also interviewed USAID representatives from two USAID employee unions: the American Federation of Government Employees and the American Foreign Service Association. In addition, we interviewed officials from the Department of State and the Office of Management and Budget. To determine the status of USAID’s reform efforts, we also reviewed USAID reform plans, reports, briefings, and project factsheets. We also interviewed USAID officials responsible for the planning and implementation of the agency’s reform projects. To determine the total number of USAID reform projects, we included all USAID reform projects identified by USAID as of July 2019. To provide the estimated costs associated with USAID’s reform efforts for contextual purposes, we obtained data from USAID on the costs of: 1) developing T3 reform efforts, including T3’s operational costs, 2) implementing T3 reform efforts, and 3) its Human Resource Transformation project contract data. We reviewed supporting documentation, and interviewed cognizant USAID officials about the completeness and accuracy of the data. We did not independently assess the data used to estimate the costs associated with its reform efforts. We determined it was beyond the scope of this review to perform a full cost-benefit analysis to assess the potential financial impact of USAID’s reform efforts using the cost estimates provided by USAID. To determine the extent to which USAID has addressed key practices for planning and implementing its reform efforts, we assessed USAID’s reform efforts against key practices identified in our June 2018 report, which are organized by 12 subcategories of change management practices. The subcategories are based on 58 key questions for consideration in assessing reform efforts. We did not apply criteria from the “Workforce Reduction Strategies” subcategory of our June 2018 report. We deemed those criteria not applicable to USAID’s reform efforts because USAID officials stated their proposals regarding workforce reductions were overtaken by events when congressional appropriations for fiscal years 2018 and 2019 maintained USAID staffing at the levels associated with its workforce as of December 2017. For the other 11 subcategories included in our assessment, we determined which key questions of each subcategory were most relevant USAID’s reform efforts and applied those key questions to our assessment. We categorized USAID reform-related actions into two separate categories: (1) those that generally addressed the subcategory and (2) actions that partially addressed the subcategory. We determined that USAID’s reform efforts had generally addressed a practice if we did not identify significant gaps in its coverage of the actions associated with this subcategory. We determined that USAID’s reform efforts had partially addressed a practice if we identified significant gaps in its coverage of the actions associated with this subcategory. We would have determined that USAID had not addressed a practice if it had not substantively addressed any of the key elements in the subcategory. However, we found that USAID at least partially addressed all of the practices. We defined “significant gaps” as the areas we identified, based on our analysis of the key questions of each subcategory, that were both relevant to USAID as an agency and important for the success of the reform efforts. Each of two analysts made an independent qualitative judgment as to whether or not USAID had generally, partially, or had not addressed those criteria. The two analysts then reviewed and reconciled any differences in the data used to reach each determination, and their results were subject to supervisory review. The analysts’ determinations were then reviewed by other GAO stakeholders with experience in this topic, and any concerns raised were resolved through discussion to reach the final determinations. We conducted this performance audit from February 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We developed key questions based on our prior work on key practices that can help assess agency reform efforts. The 58 questions are organized into four broad categories and 12 subcategories, as shown in table 5. As of June 2019, the U.S. Agency for International Development (USAID) headquarters was organized as shown in figure 4. In addition to the contact named above, Thomas Costa (Assistant Director), B. Patrick Hickey (Analyst in Charge), Joshua Akery, Peter Beck, David Dayton, Martin de Alteriis, Emily Gupta, Christopher Keblitis, Steven Putansu, Sarah Veale, and Alexander Welsh made key contributions to this report.", "summary": "In March 2017, the President issued an executive order to federal agencies intended to improve the efficiency, effectiveness, and accountability of the executive branch. The order required the Director of the Office of Management and Budget (OMB) to develop a plan to reorganize and streamline the government. In April 2017, OMB issued additional guidance to agencies on implementing the order. In response, USAID launched several efforts to reform its organizational structure, workforce, programs, and processes with the ultimate goal of ending the need for foreign assistance by helping partner countries become more self-reliant. GAO's prior work has shown that successful agency reforms depend on following key practices for organizational transformation, such as establishing goals and outcomes and involving key stakeholders. This report examines (1) the status of USAID's reform efforts and (2) the extent to which USAID has addressed key practices in planning and implementing those efforts. GAO reviewed USAID reform plans, proposals, and related documents and met with officials involved in its reform efforts. GAO also assessed USAID's planning and implementation of its reform efforts against 11 key practices identified in GAO's June 2018 report, Government Reorganization: Key Questions to Assess Agency Reform Efforts (GAO-18-427). The reform efforts of the U.S. Agency for International Development (USAID) consist of a total of 32 reform projects—31 projects being implemented by USAID's Transformation Task Team and an additional Human Resources Transformation project that predates the 31 projects. As of July 2019, USAID has completed 19 reform projects and is implementing 12 additional projects, which it intends to complete by mid-2021. The task team has one additional project in the planning phase. In planning and implementing these efforts, USAID has generally addressed nine of 11 key practices for organizational transformation and partially addressed two. For example, USAID generally addressed the key practice of involving employees and key stakeholders such as the Department of State and Congress through a variety of mechanisms, such as briefings and town halls. USAID also used data and evidence to guide its reform efforts by integrating employee and external input into its reform plans. Morever, USAID addressed fragmentation, overlap, and duplication by planning a restructuring effort to streamline operations and achieve efficiencies. Further, it generally addressed leadership focus and attention by designating a reform coordinator and establishing a dedicated team responsible for managing and planning USAID's reform efforts. However, while USAID established goals for its reform efforts, it established outcome-oriented performance measures for only four of its 32 projects. Establishing such measures would improve its ability to assess the results of the changes it is making. In addition, while USAID is developing a strategic workforce plan, it has yet to develop the tools needed to identify and meet staffing needs arising from the reforms in order to fully assess its workforce. Completing a strategic workforce plan with these tools could help USAID ensure it has the workforce needed to meet existing and emergent program demands. Addressing these gaps could help USAID make long-term improvements in its efficiency and effectiveness. USAID should (1) establish outcome-oriented performance measures to assess the effectiveness of its reform efforts and (2) complete a strategic workforce plan necessary to support its reform efforts. USAID concurred with the recommendations.", "document_type": "gao"}
{"report": "On September 6, 2017, the eye of Hurricane Irma traveled about 50 nautical miles to the north of the northern shore of Puerto Rico as a category 5 hurricane. Less than two weeks later, Hurricane Maria made landfall as a category 4 hurricane on the main island of Puerto Rico on the morning of September 20, 2017 with wind speeds up to 155 miles per hour. The center of the hurricane moved through southeastern Puerto Rico to the northwest part of the island, as shown in figure 1 below. In response to the request of the Governor of Puerto Rico, the President declared a major disaster the day after each hurricane impacted Puerto Rico. Major disaster declarations can trigger a variety of federal response and recovery programs for government and nongovernmental entities, households, and individuals, including assistance through the Public Assistance program. Under the National Response Framework, DHS is the federal department with primary responsibility for coordinating disaster response, and within DHS, FEMA has lead responsibility. The Administrator of FEMA serves as the principal adviser to the President and the Secretary of Homeland Security regarding emergency management. FEMA’s Public Assistance program provides funding to state, territorial, local, and tribal governments to assist them in responding to and recovering from major disasters or emergencies. As shown in figure 2, Public Assistance program funds are categorized broadly as either “emergency work” or “permanent work.” Within those two broad categories are separate sub-categories. In addition to the emergency work and permanent work categories, FEMA’s Public Assistance program includes category Z, which represents indirect costs, administrative expenses, and other expenses a recipient or subrecipient incurs in administering and managing projects. Puerto Rico’s agencies, such as the Department of Housing; public corporations, such the Puerto Rico Electric Power Authority and the Puerto Rico Aqueduct and Sewer Authority; and Puerto Rico’s 78 municipalities are eligible to apply for the Public Assistance program. FEMA’s Public Assistance program also provides funding for cost- effective hazard mitigation measures to reduce or eliminate the long-term risk to people and property from future natural and man-made disasters and their effects. Specifically, FEMA provides funding for hazard mitigation measures in conjunction with the repair of disaster-damaged facilities to enhance their resilience during future disasters. For example, a community that had a fire station damaged by a disaster could use Public Assistance funding to repair the facility and incorporate additional measures such as installing hurricane shutters over the windows to mitigate the potential for future damage. In Puerto Rico, the Public Assistance program is administered through a partnership between FEMA and the recipient (Puerto Rico), which provides funding to eligible subrecipients (local or territory-level entities). Under the standard Public Assistance program process, once the President has declared a disaster, Public Assistance staff work with the recipient or subrecipients to help them document damages, identify eligible costs and work, and prepare requests for Public Assistance grant funds by developing project proposals. Officials then review and obtain approval of projects prior to FEMA obligating funds to reimburse recipients or subrecipients for eligible work. The Sandy Recovery Improvement Act of 2013 authorized the use of alternative procedures in administering the Public Assistance program, thereby providing new flexibilities to FEMA, states, territories, and local governments for debris removal, infrastructure repair, and rebuilding projects using funds from this program. The stated goals of the alternative procedures are to reduce the costs to the federal government, increase flexibility in the administration of the Public Assistance program, expedite the provision of assistance under the program, and provide financial incentives for recipients of the program for the timely and cost- effective completion of projects. Unlike the standard Public Assistance program where FEMA will fund the actual cost of a project, the Public Assistance alternative procedures allow awards for permanent work projects to be made on the basis of fixed cost estimates to provide financial incentives for the timely and cost- effective completion of work. Under these procedures, if the actual cost of the project exceeds the fixed cost estimate agreed upon by FEMA and the recipient, the recipient or subrecipient is responsible for the additional costs. However, if the actual cost of completing eligible work for a project is below the estimate, the recipient or subrecipient may use the remaining funds for other eligible purposes, such as for additional cost-effective hazard mitigation measures to increase the resiliency of public infrastructure. These funds may also be used for activities that improve the recipient’s or subrecipient’s future Public Assistance operations or planning. Although FEMA had approved alternative procedure grants in 30 states as of April, 2018, in these cases, alternative procedures were used on a project-by-project basis. Puerto Rico’s recovery from the 2017 hurricanes is the first recovery to use alternative procedures for all large permanent work projects. On October 30, 2017, Puerto Rico requested to use the alternative procedures process for all large-project funding for Public Assistance permanent work, categories C through G. According to FEMA guidance, as part of the alternative procedures process in Puerto Rico, FEMA and Puerto Rico must agree on a group of personnel with cost estimation expertise who will serve as part of a center of excellence. This center of excellence will assist FEMA and Puerto Rico in developing cost estimating methodologies to be used for determining fixed cost estimates for Public Assistance permanent work projects. FEMA officials stated that they are in the process of conducting inspections for Public Assistance projects for permanent work and, as of August 2018, had a total list of 10,000 site inspections to complete. FEMA officials stated that October 2019 is their target date for completing all alternative procedures fixed cost estimates for Public Assistance permanent work. However, pursuant to 428 guidance published in April 2018, this time frame may be adjusted on a project-by-project basis, based on extenuating circumstances. Amendment 5 to the President’s disaster declaration imposed a number of grant conditions, including that Puerto Rico establish an oversight authority supported by third-party experts. This authority is to act as the grant recipient for all Public Assistance and hazard mitigation funding to ensure sound project management and enhanced, centralized oversight over FEMA grant distributions. In October 2017, the Governor of Puerto Rico established COR3, a Puerto Rico government office, to plan, guide, and oversee recovery efforts, including administering and overseeing the Public Assistance program. According to FEMA and COR3 officials, COR3 will fulfill the oversight requirements outlined in Amendment 5. According to COR 3 officials, COR3 was also established to ensure coordination with FEMA. The Executive Director of COR3 will act as the Governor’s Authorized Representative, which is the designated individual responsible for administering federal disaster assistance programs on behalf of Puerto Rico. Among other things, COR3 will: Identify, procure, and administer all federal, territorial, and private resources available to Puerto Rico related to recovery; Provide oversight of subrecipients using risk-based monitoring; and Provide technical assistance and advise Puerto Rico’s governmental agencies and municipalities regarding any matter related to recovery. According to COR3 officials, they will also implement internal controls, policies, and procedures to appropriately manage recovery funds. COR3 has also launched an online transparency portal intended to provide a breakdown of FEMA Public Assistance and other federal funding made available for disaster recovery in Puerto Rico. The Bipartisan Budget Act of 2018 (Bipartisan Budget Act) required that Puerto Rico submit an economic and disaster recovery plan to Congress by August 9, 2018, that defines the priorities, goals, and expected outcomes of Puerto Rico’s recovery related to a number of sectors, including, among other things, infrastructure, housing, electric power systems and grid restoration. The Bipartisan Budget Act also directs the Governor of Puerto Rico to develop the disaster recovery plan in coordination with FEMA, with support and contributions from other federal agencies having designated responsibilities in the National Disaster Recovery Framework. As of June 2015, Puerto Rico had roughly $66.9 billion in outstanding debt. According to the recovery plan, economic contraction in the years prior to the hurricanes contributed to a severe fiscal crisis and Puerto Rico’s credit rating dropped below investment grade in early 2014, followed by a series of defaults on debt payments. In response to Puerto Rico’s financial crisis, Congress passed and the President signed the Puerto Rico Oversight, Management, and Economic Stability Act in June 2016, which established the FOMB with broad budgetary and financial control over Puerto Rico. The Bipartisan Budget Act also permits the FOMB to review any federal funds over $10 million that are designated for Puerto Rico’s response to or recovery from Hurricanes Irma or Maria. FEMA obligated nearly $4 billion in Public Assistance funds for Puerto Rico’s emergency work projects, as well as the repair and restoration of its public infrastructure, among other things. In order to provide financial oversight of these funds, Puerto Rico is developing an internal controls plan as well as management policies and procedures that will, in part, help provide financial monitoring. In the interim, FEMA has instituted a manual reimbursement process to mitigate risk and ensure fiscal accountability. As shown in figure 3, FEMA has obligated $3.63 billion (93 percent) for emergency work (categories A and B), and $151 million (4 percent) for permanent work (categories C through G) in Puerto Rico as of September 30, 2018. An additional $136 million (3 percent) was obligated for management and administrative costs. As of the end of fiscal year 2018, Puerto Rico expended about $1.7 billion (about 43 percent) of the almost $4 billion Public Assistance funds obligated by FEMA. Ninety-eight percent of this amount went toward emergency work projects in categories A and B. For example, the Puerto Rico Aqueduct and Sewer Authority expended almost $91 million to cover the costs of generator usage. Aside from generators, one category B project by the Puerto Rico Emergency Management Agency repaired the emergency warning system for about $9.4 million. A third project put a temporary roof on a Puerto Rico Institute of Culture facility in Vieques for $4,000. As shown in table 1, the majority of FEMA’s obligations in Puerto Rico as of September 30, 2018, have been for emergency work categories because these projects began soon after the disasters struck and focused on removing debris and providing assistance to address immediate threats to life and property. In contrast, permanent work projects take time to identify, develop, and ultimately complete as they represent the longer-term repair and restoration of public infrastructure. Funds expended by Puerto Rico for permanent work have been mostly limited to roads and bridges (category C) because impassable roads like the one shown in figure 4 below impede the provision of critical services to citizens. They can also get in the way of other disaster recovery efforts. Expenditures for roads and bridges (category C) amount to approximately $32 million, while expenditures for other permanent work categories (D- G) total approximately $1 million. According to FEMA officials, Public Assistance projects in categories D-G are still pending prioritization and formulation. For example, figure 5 below shows a recreational public space along the edge of a river in Maricao. The dashed line indicates where the iron railing and concrete paving used to continue, overlooking the river, before Hurricane Maria. As of September 2018, the municipality was awaiting FEMA assistance to begin restoration FEMA categorizes Puerto Rico’s subrecipients as commonwealth public corporations, commonwealth agencies, municipalities, and all other entities. As shown in table 2 below, 89 percent of obligations, as of the end of September 2018, for Puerto Rico were awarded to commonwealth public corporations and commonwealth agencies, with 47 percent awarded to the Puerto Rico Electric Power Authority. Overall, about 43 percent of obligated funds have been expended. As previously discussed, Puerto Rico designated COR3 to administer and manage the Public Assistance program in coordination with FEMA. As part of COR3’s recovery oversight role, COR3 officials stated that they are developing an internal controls plan and recovery management policies and procedures with FEMA. According to COR3, these oversight documents will provide detailed guidance on grant application, procurement, payment and cash management, and financial monitoring and reporting, among other things. According to COR3 officials, they have held numerous meetings to coordinate with FEMA and have submitted drafts of the internal controls plan as well as management policies and procedures for FEMA’s consideration. In addition, according to COR3 officials, COR3 plans to provide direct technical assistance related to federal grants management to Puerto Rico’s cabinet-level agencies, public corporations, municipalities and other eligible subrecipients. As part of COR3’s advisory role, COR3 is expected to help Puerto Rico’s agencies, public corporations, municipalities, and some nonprofit entities formulate projects, draft cost estimates, and make funding requests, among other things. Federal grant award regulations allow FEMA to impose additional specific grant award conditions in specific circumstances, such as to mitigate risk and ensure fiscal accountability of the recipient or subrecipient. According to FEMA, once FEMA obligates funds, the recipient is able to expend funds as necessary. However, in November 2017, according to FEMA officials, the agency instituted a manual reimbursement process for subrecipients in Puerto Rico for federal funds, including Public Assistance funds, to mitigate fiduciary risk and decrease the risk of misuse of funds. Specifically, FEMA officials stated that they decided to institute this process because the government of Puerto Rico had expended funds prior to submitting complete documentation of work performed. According to FEMA officials, they also decided to institute the manual reimbursement process due to Puerto Rico’s financial situation, weaknesses in internal controls, and the large amount of recovery funds, among other things. This manual reimbursement process requires that COR3 fill out the Office of Management and Budget’s Standard Form 270 and submit supporting documentation before obligated funds can be withdrawn by Puerto Rico through COR3 and reimbursed to subrecipients. Subsequently, FEMA must review the submitted Standard Form 270 and all project documentation for completeness, compliance, and accuracy before disbursing funds to the recipient. In cases where FEMA requires additional documentation to process a Standard Form 270 request, FEMA will submit requests for information asking COR3 to supply the information needed for FEMA to complete the review. FEMA officials said that they aim to complete the entire process described above within ten calendar days, or 15-20 calendar days if FEMA needs to request additional information from COR3. Additionally, FEMA officials stated that the manual reimbursement process is intended as a temporary measure. They will cease the process once FEMA has reviewed the operational effectiveness of COR3’s internal controls and approved the final internal controls plan, which are under review. FEMA, COR3, and Puerto Rico municipal government officials from ten municipalities we interviewed reported initial challenges with the recovery process, including with Public Assistance alternative procedures. These concerns included (1) workforce capacity constraints, (2) a need for additional guidance, (3) delays related to choosing cost estimators, and (4) reimbursement for emergency work. Workforce capacity constraints. FEMA and municipality officials cited concerns about FEMA staff turnover and lack of knowledge about how the Public Assistance alternative procedures are to be applied in Puerto Rico. While several municipal officials we spoke to remarked positively on consistent communication with FEMA officials, municipal officials in six municipalities we visited cited high levels of turnover among FEMA staff as a challenge. For example, officials in three municipalities said that discontinuity in FEMA personnel has caused them to have duplicative conversations with FEMA. An official from one municipality described the disruption that had been caused by repeated changes in FEMA personnel, especially when their point of contact at FEMA changed at least six times since the hurricanes. FEMA officials acknowledged that more personnel with expertise in the alternative procedures process are needed to administer the Public Assistance program and assist subrecipients. According to FEMA officials, FEMA has leveraged existing expertise from personnel in the Federal Coordinating Officer Advisory Group to train new employees to increase workforce capacity. FEMA personnel from this group are rotating experts assigned to recovery issues to increase institutional understanding of alternative procedures and train local hires. According to FEMA officials, these local hires can serve as FEMA staff for up to one year before they become reservists. In addition, FEMA officials stated that they have identified contractors with previous experience regarding alternative procedures to provide additional assistance to subrecipients. Need for additional guidance. Municipal officials cited concerns about a lack of comprehensive guidance for the alternative procedures process. Specifically, officials in eight municipalities we interviewed cited problems with missing, incomplete, or conflicting guidance from FEMA. In addition, officials in four municipalities stated that they are waiting on additional written instructions to establish more clear and consistent guidance. Officials from one municipality told us that the lack of written guidance has meant that the municipality has had to re-submit documents to FEMA multiple times to respond to changing guidance that they have received verbally. Additionally, four municipalities cited missing, incomplete, or conflicting guidance from COR3 as a challenge. However, one municipality noted that the quality of communication with COR3 has improved over time as COR3 has become more established. According to FEMA officials, they are drafting supplemental guidance for the alternative procedures process with the goal of incorporating lessons learned from prior iterations of the alternative procedures. Similarly, according to COR3 officials, COR3 is currently developing additional guidance and standard operating procedures to help subrecipients, including municipalities and Puerto Rico government agencies, better understand FEMA Public Assistance grant requirements. Delays related to choosing cost estimators for Puerto Rico. As mentioned previously, FEMA’s guidance for alternative procedures requires that FEMA and Puerto Rico, through COR3, choose personnel with expertise in cost estimation to serve as a center of excellence, which will develop a cost estimating methodology. FEMA has chosen personnel to staff the center of excellence. However, in August 2018, FEMA officials told us that COR3 had not yet finalized their choice of personnel, which had delayed the cost estimation process. Subsequently, COR3 officials told us that personnel have been identified to serve on the center of excellence and the final contracting process for these personnel is now in progress. Reimbursement for emergency work. Officials in nine municipalities we spoke to said that they had not been fully reimbursed for emergency work they completed. Further, officials in five municipalities we interviewed stated that the lack of full reimbursement has caused financial hardships. For example, officials in three municipalities said that the lack of full reimbursement has meant that the municipalities have had to pause or delay recovery work due to lack of financial resources. A mayor in one municipality stated that they have scaled back some essential services, such as the frequency of garbage pick-up, while waiting for full reimbursement. According to FEMA officials, delays in providing reimbursement were due to several factors including a loss in FEMA personnel to process reimbursement requests and a significant increase in the volume of reimbursement requests submitted by COR3 to FEMA. FEMA officials also stated an increasing need to make requests for information to COR3 due to a lack of documentation submitted at the time of the reimbursement request. In response to these factors, FEMA officials told us that they have undertaken new procedures with COR3. For example, according to FEMA officials, COR3 adopted procedures to review the completeness of documentation prior to submitting a reimbursement request to FEMA. FEMA officials stated that the agency is also holding weekly meetings with COR3 to increase coordination, and that FEMA increased the number of personnel devoted to reimbursement reviews. According to officials from FEMA and COR3, these steps have contributed to reduced delays. In response to the Bipartisan Budget Act, Puerto Rico submitted an economic and disaster recovery plan (recovery plan) to Congress on August 8, 2018. The recovery plan defines the priorities, goals, and expected outcomes of Puerto Rico’s recovery related to building government capacity for the recovery and strengthening of Puerto Rico’s infrastructure, among other things. The recovery plan estimates infrastructure repair and recovery costs of $132 billion and total recovery costs of $139 billion for a time period starting in 2018 and ending in 2028. According to the recovery plan, COR3 will guide recovery investment and policy in the months and years ahead and is intended to serve as a focal point for strategic thought and management of Puerto Rico’s recovery. The recovery plan is generally responsive to the directives outlined in the Bipartisan Budget Act. For example, Puerto Rico submitted the plan to Congress within 180 days of enactment of the Bipartisan Budget Act. The recovery plan defines priorities, goals, and expected outcomes for Puerto Rico’s recovery effort based on damage assessments conducted by sector. As mentioned earlier, Puerto Rico developed the recovery plan in coordination with FEMA and with support of the U.S. Department of Energy, the U.S. Department of Health and Human Services, and other federal agencies with responsibilities outlined in the National Disaster Recovery Framework. Additionally, the FOMB of Puerto Rico certified the recovery plan on August 28, 2018, as directed in the Bipartisan Budget Act, but provided two caveats to its certification. First, FOMB expressed concern that the recovery plan lacks sufficient detail of funding sources and estimates a much greater amount of federal funding than the certified fiscal plan for Puerto Rico projects. The recovery plan states that at the time of its release, Puerto Rico had not undergone eligibility reviews in various federal funding programs, and therefore the ability to identify accurate funding sources was limited. COR3 officials confirmed that full recovery funding needs will not be known until all damage assessments are complete, and they will continue to identify and leverage all funding resources as they are made available. Second, FOMB indicated that the recovery plan does not address oversight of federal funds and the recovery process. While the Bipartisan Budget Act does not require specific mechanisms for oversight of federal funding as part of the recovery plan, according to COR3 officials, they plan to implement internal controls, policies, and procedures to provide oversight. Puerto Rico’s recovery plan outlines 276 “courses of action” (actions)— defined by the plan as “a collection of potential activities, policies, and other actions that could contribute to recovery”—selected by Puerto Rico to align with its future recovery vision. As shown in table 3 below, the actions reflect Puerto Rico’s short-term and long-term recovery vision, organized into three areas. First, the recovery plan proposes “precursor” actions—those that serve as a foundation for all future actions—that will be prioritized for implementation. For example, the recovery plan includes actions to build capacity of municipalities to secure and manage recovery funds, and to improve the quality and volume of public data available to decision makers. Second, the recovery plan proposes a set of actions that aim to build the infrastructure and systems that support Puerto Rico’s economy, society, and disaster resiliency, such as addressing vulnerabilities in Puerto Rico’s electric grid. Finally, the recovery plan proposes a set of actions that address Puerto Rico’s long-term recovery goals, such as developing and enhancing Puerto Rico’s visitor economy. Most individual actions in the recovery plan include initial and recurring cost estimates for the time period from 2018 through 2028. The recovery plan describes all cost estimates as preliminary, and says that more specific cost estimates require completion of damage assessments and more details about the implementation of actions. To develop Puerto Rico’s disaster recovery plan, FEMA assisted COR3 in retaining the Homeland Security Operational Analysis Center (HSOAC), a federally-funded research and development center operated by the RAND Corporation under contract with DHS. According to FEMA officials, FEMA provided funding and technical assistance, through contractor support, for Puerto Rico to develop the recovery plan, but COR3 and Puerto Rico will be responsible for its implementation. These officials also stated that Puerto Rico received input and technical assistance from other federal departments, such as those in the Recovery Support Function Leadership Group led by FEMA. HSOAC developed the recovery plan in consultation with Puerto Rico by developing a preliminary sector-by-sector assessment of damages and needs caused by Hurricanes Irma and Maria across Puerto Rico. In conjunction with Puerto Rico’s stated vision for the recovery process, HSOAC’s damage assessment report provided the baseline needed to define, compare, and prioritize actions. HSOAC worked in teams of sector-specific experts to develop and refine the actions by reviewing reports, proposals, best practices, and other literature. For example, in June 2018, the U.S. Department of Energy released a report on energy resilience for Puerto Rico’s electric grid, containing recommendations for Puerto Rico to consider when developing the Recovery Plan. HSOAC sought feedback from various subject matter experts and stakeholders while developing the recovery plan. According to FEMA officials, FEMA’s joint recovery office delivered interim drafts of the recovery plan to federal agency partners and Puerto Rico for feedback. HSOAC also sought input from local-level stakeholders, including Puerto Rico’s municipal governments. For example, HSOAC commissioned a survey of officials from municipalities to gauge the challenges they faced in the aftermath of the 2017 hurricanes. According to FEMA and HSOAC officials, the survey, along with other input provided by mayors led to the development of actions focused on building the capacity of municipal governments to support recovery efforts. HSOAC officials noted that while the final recovery plan was submitted to Congress, they will continue to produce products that will assist Puerto Rico and their stakeholders in recovery implementation. HSOAC intends to release updated versions of the recovery plan, including updated damage and needs assessments. Other expected products include detailed descriptions and cost estimates for each action and a lessons learned report. The Bipartisan Budget Act also directs Puerto Rico to develop a public report on the progress made in achieving the recovery plan’s goals every 180 days after submission. FEMA officials explained that the recovery plan serves as a strategic, direction-setting plan for recovery, and does not provide step-by-step or site-by-site guidance on the recovery process. FEMA officials also acknowledged that there may be some overlap between some of the actions in the recovery plan and some of the permanent work funded through FEMA’s Public Assistance program, but that it is COR3’s responsibility to merge and coordinate such recovery efforts. We provided a draft of this report to DHS and the government of Puerto Rico for review and comment. In its comments, reproduced in appendix I, DHS summarized the amount of Public Assistance funding provided to Puerto Rico through fiscal year 2018. DHS also described FEMA’s temporary manual reimbursement process instituted to mitigate risk and ensure fiscal accountability of taxpayer dollars, and stated that FEMA is committed to supporting Puerto Rico as it finalizes internal controls, management policies and procedures to oversee disaster recovery funds. DHS also provided technical comments, which we incorporated as appropriate. The government of Puerto Rico provided comments that we reproduced in appendix II. In its comments, the government of Puerto Rico stated that in addition to what was discussed in this report, COR3 achieved progress and faced additional challenges. This report is a part of an ongoing review of disaster recovery efforts in Puerto Rico. The remainder of our ongoing work will continue to examine Puerto Rico’s recovery process, including implementation of the Public Assistance alternative procedures process and efforts by FEMA and Puerto Rico to oversee disaster recovery funds, including the manual reimbursement process. If you and your staff have any questions, please contact me at (404) 679- 1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Chris Currie, (404) 679-1875 or CurrieC@gao.gov. In addition to the contact named above, Joel Aldape (Assistant Director), Pedro Almoguera, Aditi Archer, Michelle Bacon, Sylvia Bascope, Lilia Chaidez, Taylor Hadfield, Danielle Pakdaman, Lorraine Ettaro, Eric Hauswirth, Heidi Nielson, and Kevin Reeves made key contributions to this report.", "summary": "In 2017 two major hurricanes – Irma and Maria – caused extensive damage throughout Puerto Rico. Hurricane Maria, a Category 4 hurricane, was the most intense hurricane to make landfall in Puerto Rico since 1928, destroying roads, buildings, and cutting power and communication lines, among other things. Puerto Rico estimates that $132 billion will be needed to repair and reconstruct infrastructure and services. FEMA—a component of the Department of Homeland Security (DHS)—is the lead federal agency responsible for assisting Puerto Rico as it recovers. FEMA administers the Public Assistance program in partnership with Puerto Rico to provide funds to rebuild damaged infrastructure and restore critically-needed services. GAO was asked to review the federal government's recovery efforts related to the 2017 hurricanes. This report, among other objectives, describes (1) FEMA's Public Assistance spending in Puerto Rico and oversight efforts of federal recovery funds, and (2) initial challenges with the recovery process. GAO reviewed Public Assistance program documents; analyzed grant funding data; and interviewed officials from Puerto Rico and DHS about the Public Assistance program and recovery efforts, as well as officials from ten municipalities selected on the basis of population and Public Assistance spending. GAO is not making recommendations at this time, but will continue monitoring the recovery as part of its ongoing work. The Federal Emergency Management Agency (FEMA) obligated almost $4 billion in Public Assistance grant funding to Puerto Rico as of September 30, 2018 in response to the 2017 hurricanes. FEMA obligated about $3.63 billion for emergency work—emergency measures such as debris removal and generators—and about $151 million for permanent work to repair and replace public infrastructure such as roads (see figure). Puerto Rico established a central recovery office to oversee federal recovery funds and is developing an internal controls plan to help ensure better management and accountability of the funds. In the interim, FEMA instituted a manual reimbursement process—requiring FEMA to review each reimbursement request before providing Public Assistance funds—to mitigate risk and help ensure financial accountability. FEMA officials stated that they will remove this manual process once the agency approves Puerto Rico's internal controls plan. Officials from FEMA and Puerto Rico's central recovery office and municipalities that GAO interviewed reported initial challenges with the recovery process, including with Public Assistance alternative procedures. Unlike in the standard Public Assistance program where FEMA will fund the actual cost of a project, the Public Assistance alternative procedures allow awards for permanent work projects to be made on the basis of fixed cost estimates to provide financial incentives for the timely and cost-effective completion of work. Challenges identified included concerns about lack of experience and knowledge of the alternative procedures being applied in Puerto Rico; concerns about missing, incomplete, or conflicting guidance on the alternative procedures; and concerns that municipalities have not been fully reimbursed for work already completed in the immediate aftermath of the hurricanes, causing financial hardships in some municipalities. FEMA officials stated that the agency is taking actions to address reported recovery challenges, such as leveraging existing expertise to train personnel and developing supplemental guidance on alternative procedures and reducing delays in reimbursements. GAO will continue to monitor these issues and plans to report additional findings and recommendations as appropriate later this year.", "document_type": "gao"}
{"report": "The National Defense Strategy is DOD’s primary strategy document, providing a foundation for all other strategic guidance in the department. The National Defense Authorization Act for Fiscal Year 2017 required DOD to develop a national defense strategy and update it at least once every 4 years and, during the years without an update, to assess the implementation of the strategy and whether any revision is necessary. The National Defense Strategy replaces the Quadrennial Defense Review, which the Armed Services Committees concluded had become too slow and ineffective to provide relevant strategic direction to the department. For each new strategy, DOD is required to identify, among other things: DOD’s strategic priority missions; the force structure, readiness, posture, and capabilities needed to support the strategy; and major investments required by the strategy. A separate provision in the act also established a Commission to assess the 2018 National Defense Strategy. The provision required the Commission to review the assumptions, missions, force posture and structure, and risks associated with the strategy. Congress expressed continued interest in DOD’s strategy implementation and assessment in the John S. McCain National Defense Authorization Act for Fiscal Year 2019, which included several provisions related to these matters. The National Defense Strategy falls under the President’s National Security Strategy, which outlines the overarching security strategy for the federal government. The National Defense Strategy is above the National Military Strategy, which provides more detailed military direction. Figure 1 provides the hierarchy and description of key U.S. strategic guidance documents. Organizations across DOD play a role in providing analytic support to senior leaders as they make force structure decisions to support the National Defense Strategy. Table 1 provides a summary of the organizations with key roles and responsibilities for providing analytic support to senior leaders making force structure decisions. DOD established its approach, Support for Strategic Analysis (SSA), in 2002 to provide analytic support to DOD senior leaders as they deliberate strategy and budget matters and to support evaluations of force structure needs across the joint force. SSA is structured to do this by providing a common set of assumptions for various military threats that form the basis for further analysis across the department. DOD guidance states that SSA is intended to provide a common starting point for the exploration of various approaches to address the threats. DOD guidance further states that analyses should provide senior leaders with insights on the relative risks of various operational approaches and force structures. Senior leaders would then have a basis to weigh options, examine tradeoffs across the joint force, and drive any force structure changes necessary to meet the strategy. For more information on the origin of SSA, see the sidebar below. Origin of Support for Strategic Analysis DOD officials told us that the department developed what became SSA because then Secretary of Defense Donald Rumsfeld was frustrated by the lack of objective measures to compare competing force structure proposals. During the 1990s, each service developed its own analytic process and assumptions for assessing force structure needs to develop requirements for budget submissions. Each service’s analytic process tended to favor its preferred force structure and operational approach. DOD officials stated that the lack of a common analytic starting point for all of the services also meant that senior leaders had difficulty getting beyond debates about the services’ respective assumptions during discussions on force structure priorities. As a result, the Secretary of Defense had no objective basis by which to decide whether, for example, a Navy proposal to buy more ships or an Air Force proposal to buy more fighter aircraft was the best way for the department to use its limited resources to support strategic priorities. SSA is led by OUSD (Policy), the Joint Staff, and CAPE—collectively referred to as the Tri-Chairs. DOD guidance assigns each Tri-Chair responsibility for creating one of three increasingly detailed products for a variety of military threats that, taken together, comprise the common starting point for additional analysis of that threat. The resultant SSA product library is then available to the services and other DOD organizations for further analysis. DOD guidance notes that the threats SSA products address are examples of the types of threats U.S. joint forces are expected to be able to address with acceptable risk. However, the guidance states that the forces described in the products are not intended to constitute DOD’s force structure requirements. Instead, analysis using these products is intended to help senior leaders establish force structure requirements that balance risk across a range of threats, within fiscal constraints. Table 2 identifies the three SSA products that are intended to form the common starting point for analysis for a given plausible threat, along with the lead Tri-Chair for each product type. According to DOD guidance, the military services are to support the Tri- Chairs in developing the SSA products and, according to DOD officials, are the primary users of these products. The guidance requires that the services use SSA products as common starting points for studies evaluating their force structure needs for implementing the defense strategy and supporting their budget development, among other things. Although the starting points are common across the department, each service uses its own analytic process to evaluate its specific force structure needs for implementing the strategy and supporting its budget development (see app. I for further details on each service’s analytic process). The services may examine any plausible threat in the SSA library that they believe may help them understand their force structure needs. However, the 2018 National Defense Strategy identifies several key threats and the principal priorities for the department that the services must prioritize when developing their force structures. Specifically, the unclassified summary of the strategy calls for the department to increase and sustain investments towards the long-term strategic competitions with China and Russia, and to concurrently sustain its efforts to deter and counter rogue regimes such as North Korea and Iran, defeat terrorist threats to the United States, and consolidate gains in Iraq and Afghanistan with a more resource-sustainable approach. Further, budget guidance—in particular the Defense Planning Guidance—directs each service on which threats it must focus as part of its budget development process. Figure 2 provides a generalized overview of how the SSA process was designed to operate. SSA has not provided senior leaders with the analytic support they need to evaluate and make fully informed decisions regarding the force structure needed to implement the National Defense Strategy. DOD has recognized this and attempted to reform SSA for several years, including exploring alternative options for providing senior leaders with better decision-making support. However, DOD has not fully developed these approaches and it is unclear whether they will provide the analytic support needed. To date, SSA has not provided the analytic support senior leaders need to evaluate and determine the force structure required to implement the defense strategy. DOD senior leaders have documented concerns with SSA in relevant guidance. For example, DOD’s 2016 Defense Analytic Guidance stated explicitly that there were cracks in the department’s analytic foundation, many of which originate within SSA. Further, CAPE and the Joint Staff had disengaged from the SSA process by this time but, as of September 2018, the services were still using SSA products for their force structure analyses and budget development. Based on our analysis, we believe that SSA has not yielded the analytic support that it was intended to provide owing to three interrelated and persistent challenges: (1) cumbersome and inflexible products, (2) limited analysis that tends not to deviate from the services’ programmed force structures and has not tested key assumptions, and (3) an absence of joint analysis evaluating competing force structure options and cross- service tradeoffs. DOD has not kept the SSA products complete and up to date because they are cumbersome and inflexible. DOD guidance states that SSA products are to be common starting points for analyses, including key threats identified in strategic guidance. DOD guidance also states that SSA products should retain consistency with DOD strategy and current intelligence and should incorporate operational approaches effective at mitigating future threats. Credible independent analysis of an issue requires a detailed, well-understood, up-to-date common basis for that analysis. As of September 2018, DOD’s library of products was incomplete and outdated. Specifically, the Detailed View was not available for any of the threats, and Joint Staff officials told us they stopped producing joint CONOPS through SSA in 2015. Moreover, the Joint Staff retired all of the existing SSA CONOPS in March 2018 because they were outdated and/or not aligned with the 2018 National Defense Strategy—though they were still available for the department to access. Service officials also told us that many of the approved Defense Planning Scenarios and CONOPS for the key threats identified in the 2018 National Defense Strategy do not reflect up-to-date military objectives and adversary capabilities. Additionally, the 2018 National Defense Strategy outlines a new force posture and employment model that could have major implications for future CONOPS. However, DOD is still developing these concepts and, as such, they are not yet reflected in any SSA products. Specific details on the status of key SSA products were omitted because the information is classified. One of the key reasons DOD did not keep the products complete and up to date was that developing and approving highly detailed and complex SSA products was cumbersome, taking a significant level of effort and time. Tri-Chair officials told us that developing the CONOPS and Detailed View, in particular, was difficult because there was a desire to gain consensus with all of the stakeholders and because the services wanted these products to have high fidelity detail in order to run their campaign models. For example, CAPE and Joint Staff officials told us that it took between 1 and 2 years to build and approve the Detailed View for one threat scenario. The officials added that the level of detail included made the product inflexible and difficult to vary. CAPE and Joint Staff officials agreed that this product became far too detailed and time-consuming and used a substantial amount of the department’s analytic capacity. As a result, the officials told us that CAPE abandoned building additional Detailed Views in 2012. The lack of agreed-upon details about the forces required has had other effects. For example, OUSD (Policy) and Joint Staff officials told us that the services still wanted the comprehensive information that the Detailed View was supposed to provide for use in their campaign models. Without CAPE producing Detailed Views, the officials noted that some of the detailed information migrated into the higher level CONOPS, making developing and analyzing that product more difficult and time-consuming as well. However, all four military services told us that they need and continue to use the SSA products—specifically, the Defense Planning Scenarios and CONOPS—to support program and budget formulation. Service officials also told us they have adapted CONOPS, as individual services or with other services, to better reflect the operational environment (e.g., updating intelligence estimates on adversary capabilities). However, CAPE and OUSD (Policy) officials told us that this results in the services’ analyses no longer being common and comparable across the department. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 reiterates that OUSD (Policy) must, in coordination with the other Tri-Chairs, develop planning scenarios by which to assess joint force capabilities, among other things. Until the Tri-Chairs determine the analytic products needed and the level of detail that is sufficient to serve as a common starting point but also flexible enough to allow for variation of analysis, and ensure these products are updated, the military services will likely continue to generate budget requests based on analysis that is not comparable. As DOD’s 2016 Defense Analytic Guidance noted about the fiscal year 2017 budget review, the lack of a common basis for their analysis hampers the department’s ability to understand the relationship between future warfighting risks identified in analysis and the services’ programmatic decisions. Although DOD’s guidance stated that SSA will facilitate a broad range of analysis exploring innovative force structure approaches for mitigating future threats identified in the strategy, SSA has not done so. Innovative force structure approaches could include, for example, alternative CONOPS and deviations from programmed forces. The 2018 National Defense Strategy stated that DOD’s operational approach largely dates from the immediate post-Cold War era when U.S. military advantage was unchallenged and the threats were rogue regimes, which is no longer the case. OUSD (Policy) officials told us that SSA CONOPS also reflect this outdated approach that depends on overwhelming force for success, which is unrealistic against advanced adversaries. Similarly, DOD’s 2016 Defense Analytic Guidance called for SSA to emphasize analyzing and assessing risk against key threats rather than on defending predetermined force levels or capabilities. Rather, the 2018 strategy stated that the department must relentlessly pursue innovative solutions and devise insurmountable dilemmas for future adversaries and that incrementalism or evolutionary progress is inadequate. However, Tri-Chair and service officials told us the services have been reluctant to conduct or share these types of boundary-pushing analyses through SSA for fear that they will jeopardize their forces or limit their options. Tri-Chair officials also told us that the services have leveraged their participation in developing SSA products to ensure their favored major force structure elements are included in the common starting point. Joint Staff officials noted that they were able to do this because SSA did not constrain what force structure the services could use for their analysis. That is, if the force structure was programmed, they could use it because the goal was to overwhelm the adversary. However, by not significantly deviating from the starting points, the services were able to ensure that their analytic outcomes support the need for the already- programmed force. Additionally, several questionable assumptions underpin the analysis. Sensitivity analysis examines the effects that changes to key assumptions have on the analytic outcome and are helpful to understand risk. It can therefore provide insight to decision makers of how risk levels would change if conditions did not match the assumptions. However, Tri-Chair officials told us that the services, using SSA products as a starting point, generally have not conducted sensitivity analyses on key operational assumptions or on factors that may not be static (or at least have some uncertainty) and, if varied, may raise or lower the risk of completing assigned tasks or missions. According to these officials, as well as our past work, certain questionable assumptions have not been analyzed through sensitivity analysis as part of SSA. For example, all four services tend to assume that their readiness for a conflict will be high, consistent with the level directed in guidance. However, we reported in 2018 that at the individual service level, the military services continue to report readiness challenges and readiness rebuilding is anticipated to take 4 years or more. Specific details of service-specific assumptions that are problematic were omitted because the information is classified. The services have been reluctant to independently examine a broad range of innovative force structure options and conduct sensitivity analysis on key operational assumptions through SSA because, according to service officials, due to competing priorities they believe they can generally only affect marginal changes in their budgets from year to year and have limited analytic capacity. Service officials noted how the majority of their service’s budget each year is constrained by must pay bills, including personnel costs, supporting existing force structure, established contracts, sustaining the industrial base, and statutory mandates. As such, unless directed to by senior leaders, service officials told us that they typically do not use their limited analytic resources to conduct sensitivity analysis or explore alternative approaches. The sensitivity analyses they have been directed to conduct have generally been focused on smaller force structure changes, but have provided useful insights. For example, the Air Force conducted an analysis for its fiscal year 2019 budget request of how risk would be affected with various F-35 buy-rates and investments in munitions and base defense. The Air Force found that it could reduce risk by keeping its F-35 buy-rate steady instead of increasing it and could use the resulting savings to bolster its munitions stocks. DOD stated in its 2016 Defense Analytic Guidance that SSA is not adequately exploring innovative approaches to meet future challenges, and called for OUSD (Policy) to identify key operational assumptions for the services to use to conduct sensitivity analyses. However, the direction provided by the department has thus far been limited and has generally not provided specific guidance requiring the services to explore a range of innovative force structure approaches or identified key assumptions on which the services must conduct sensitivity analyses. For example, the three Defense Planning Scenarios updated in 2018 for the purposes of analysis in support of the fiscal years 2020 and 2021 budget requests included a number of parameters for further analytic exploration. However, the guidance encourages, but does not require, the services to conduct these analyses. As previously discussed, officials said the services are reluctant to conduct or share this analysis and are unlikely to do so without specific direction. As a result, SSA analysis largely reflects the services’ programmed force structures and has not driven any significant changes to force structure or resource allocation within DOD and lacks credibility with senior leaders, as documented in DOD guidance. Until DOD provides specific guidance requiring the services to explore a range of innovative force structure approaches relevant to the threats identified in the 2018 National Defense Strategy, including identifying key assumptions for sensitivity analyses, DOD senior leaders may not have full visibility into the risks in the joint force’s ability to execute the missions set out in the National Defense Strategy. A key stated goal of SSA was to create a common analytic foundation so that the services’ force structures could be evaluated as a joint force—as it would fight. However, SSA has not resulted in this type of joint analysis. Specifically, DOD guidance states that SSA is intended to facilitate the comparison and evaluation of competing force structure options and cross-service tradeoffs. DOD guidance also states that assessments of the aggregate capacity of the joint force can provide an analytic foundation to identify risk and understand tradeoffs across competing demands for the force. According to the services, SSA products provide a valuable resource and are critical to informing programmatic decisions. However, DOD’s 2016 Defense Analytic Guidance noted that there was a dearth of joint analysis at the operational and strategic levels; the department lacks a body or process to conduct or review joint force analysis; and the department’s SSA efforts were focused on developing, versus analyzing, the common starting points. Accordingly, it reiterated the need for SSA to free up time and resources to conduct joint analysis and review competing analyses. Tri-Chair officials told us that DOD currently compares and makes decisions on force structure options primarily through the budget process; however, such budget reviews are typically limited to specific areas of interest. The officials added that program and budget review is not the best place to evaluate joint force structure tradeoffs because the kinds of issues examined in the budget process are more limited in scope and generally do not include comprehensive cross-service comparisons. Lacking joint analytic capability to assess force structure needs could be problematic as the department moves forward to implement the 2018 National Defense Strategy. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 directed OUSD (Policy), in coordination with the other Tri-Chairs, to conduct assessments of the capabilities of the joint force to achieve required objectives. However, Tri-Chair officials also told us that, as of 2018, there was not a mechanism in place for DOD to routinely assess joint force needs and force structure tradeoffs across the military services. As previously discussed, in 2016 this was identified as an issue, and limited progress has been made since then to ensure adequate joint analysis to support senior leader decision-making. Further, OUSD (Policy) officials told us that SSA has not been responsive to senior leaders because it has not provided timely and comprehensive answers to important questions that only joint analysis can provide, such as the extent to which the joint force can successfully meet a campaign’s overall objectives (e.g., win the war) or the extent to which cross-service tradeoffs would affect a specific campaign. As a result, force structure decisions in the department based on SSA have remained largely relegated to marginal changes through program and budget review, according to DOD. The department’s gap in a joint analytic capability is particularly problematic in light of the National Defense Strategy’s call for urgent change at a significant scale and recent proposals by the services to greatly expand their force structure—including the Navy’s plan to grow the fleet by as much as 25 percent and the Air Force’s plan to grow squadrons by 24 percent. Based on our discussions with officials and our analysis, there are a number of different options the department has for conducting such joint analyses, including establishing a separate body with these capabilities or specifying the organizational responsibilities and processes for conducting these comparisons and analyses. Until the department has an approach for conducting joint analyses or comparing competing analyses, DOD senior leaders will not have a robust joint analytic foundation to rely on to evaluate competing force structure options and cross-service tradeoffs. The department has recognized that SSA has shortcomings and made repeated efforts to address them, including specific intervention and supplemental guidance promulgated in 2014 and 2016. However, Tri- Chair officials told us that these prior efforts fell short, and the department’s struggles with SSA led to two of the three Tri-Chairs disengaging from the process—CAPE in 2012 and the Joint Staff in 2015. The Tri-Chairs agree that DOD continues to need a process and products that are current, more responsive to senior leader needs, and able to provide insights on alternative approaches and force structures that span the joint force. In addition, Joint Staff officials noted that SSA was too focused on force sizing, which is not consistent with the 2018 National Defense Strategy’s focus on innovation, modernization, and readiness. In order to address this, the Joint Staff is pursuing an alternative approach to SSA that would largely eliminate a separate formal analytic process. Instead, the Joint Staff believes that the Tri-Chairs and the services can address senior leader needs more efficiently by continuing to execute their existing statutory roles and responsibilities within their own individual organizations in lieu of SSA. Since 2016, the Joint Staff has reinvigorated its own analytic capability to support the Chairman of the Joint Chiefs of Staff and other senior DOD leaders, according to Joint Staff officials. Although officials from other DOD organizations have supported the Joint Staff’s reinvigoration of its analytic support, they told us that this approach is focused on the Chairman’s responsibility rather than on wider departmental needs and does not address key shortfalls in providing analytic support to senior leaders, including the need for a common, flexible starting point. Further, the Joint Staff’s alternative approach would rely on CAPE’s analysis in the budget process as the culminating point for final DOD force structure decisions. CAPE officials told us that the program review can assist DOD leadership in optimizing relatively limited changes to DOD’s force structure by evaluating service budget submissions and identifying alternatives for consideration. However, budget cycle time constraints mean that little analysis occurs within program review and, as a result, program review relies on the foundational analysis SSA was intended to provide. As such, CAPE’s annual program review is inadequate for comprehensively examining needs and making major tradeoffs across the joint force, according to the officials. Finally, the department originally created SSA as a separate analytic process to address a shortfall not addressed by key DOD entities pursuing their statutory responsibilities. The Tri-Chairs have also undertaken an effort to identify an alternative approach to SSA. Specifically, shortly after the new strategy was released in 2018, CAPE initiated a Tri-Chair “blank slate” review of DOD’s analytic process in order to thoroughly review—without preconceived solutions— how to best provide analytic support to senior leaders. According to Tri- Chair officials, this effort is in the early stages of development and has not yet identified solutions to the challenges that hampered SSA or documented any aspects of a new approach. While the department’s recognition of the challenges confronting SSA is promising, the two efforts underway to identify alternatives to SSA are not complete and it is unclear the degree to which these efforts will address the challenges that have been long-standing with SSA. Addressing these challenges is critical to being able to provide needed information for senior leaders to make decisions on how best to implement and execute the National Defense Strategy. The 2018 National Defense Strategy calls for the department to make difficult choices to prioritize what is most important to field a lethal, resilient, and rapidly adapting joint force needed to address the growing threats to U.S. security. It also emphasizes that this environment demands analysis that accepts uncertainty and complexity and can drive innovation among rapidly changing threats. To prepare the joint force for the threats identified in the strategy, the department’s leadership needs to be supported by timely and comprehensive analyses. However, SSA—DOD’s current approach for providing such analytic support—has not provided the timely and comprehensive analyses that senior leaders need to make informed decisions about the joint force structure needed to implement the National Defense Strategy. Senior leaders have documented in relevant DOD guidance that there are cracks in the department’s analytic foundation, many of which originate with SSA. This is due in part to highly detailed and complex products that are difficult to produce and lack flexibility to analyze, insufficient guidance to overcome the interests of the services to protect their force structure equities, and the lack of a joint analytic capability. Congress, in the John S. McCain National Defense Authorization Act for Fiscal Year 2019, required OUSD (Policy), in coordination with the other Tri-Chairs, to develop joint force objectives and conduct assessments of the joint force’s capability to meet those objectives. The department has demonstrated a desire to fix SSA’s deficiencies but has thus far been unable to overcome these challenges. Without determining the analytic products needed and updating them, issuing specific guidance requiring alternatives and key assumptions to be fully analyzed, and developing an approach for conducting joint analysis, DOD may not be providing its leaders with the analytic support they need to prioritize force structure investments that would best manage risk and address the threats outlined in the National Defense Strategy. We are making three recommendations to DOD as it reevaluates its analytic approach. The Secretary of Defense should ensure that OUSD (Policy), the Joint Staff, and CAPE—in consultation with the services—determine the analytic products needed and the level of detail that is sufficient to serve as a common starting point but flexible to allow for variation of analysis to support senior leader decisions, and update these products to reflect current strategy and intelligence estimates, as well as the anticipated operational approaches needed to address future threats. (Recommendation 1) The Secretary of Defense should ensure that OUSD (Policy) provide specific guidance requiring the services to explore a range of innovative force structure approaches relevant to the key threats identified in the National Defense Strategy, including identifying key assumptions on which the services must conduct sensitivity analyses. (Recommendation 2) The Secretary of Defense should establish an approach for comparing competing analyses and conducting joint analyses for force structure to support senior leaders as they seek to implement the National Defense Strategy. This could include establishing a separate body with these capabilities and/or specifying the organizational responsibilities and processes for conducting these comparisons and analyses. (Recommendation 3) We provided a draft of the classified version of this report for review and comment to DOD. That draft contained the same recommendations as this unclassified version. In its written comments (reproduced in app. II), DOD concurred with our three recommendations and noted that the department has begun to address the recommendations with its new Defense Planning and Analysis Community initiative. We also received technical comments from DOD, which we incorporated as appropriate. DOD provided comments on its concurrence with the three recommendations. In its comments on the first recommendation, DOD suggested that we revise the recommendation to include that the Tri- Chairs consult with the services as they implement the recommendation. Throughout our report, we identified the important role the services play in providing analytic support to senior leaders, including supporting the development and use of the analytic products that provide the foundation of analysis in the department. As such, we agree with DOD’s proposed revision and have incorporated it to further clarify the services’ important role. In its comments on the second and third recommendations, DOD advised that we replace the term “force structure” with “force planning” to ensure that different audiences understand that we are referring to force sizing, shaping, capability, and concept development. DOD correctly stated that we were using the term “force structure” in a broad sense. However, the term force planning is not interchangeable with force structure because force planning is the act of analyzing and determining force structure needs. In order to provide further clarification, we added a note in the body of the report stating that when we refer to force structure analysis, it includes the force planning elements identified by DOD (i.e., force sizing, shaping, capability, and concept development). The department also provided some general comments on our report. Specifically, DOD noted that it has reservations about some of the report’s content because at times it seems to reflect statements based on particular organizational perspectives. DOD therefore requested that we acknowledge that Support for Strategic Analysis (SSA) suffered from poor implementation rather than being fundamentally unsound. However, DOD also stated that our report outlined that SSA failed due to overall suboptimal management and unwieldy stakeholder execution, and that the resulting failure to present analysis in a timely and responsive fashion impeded the flow of quality information to senior leaders. We believe that the three interrelated challenges we identified in our report adequately reflect that SSA faced significant challenges in being implemented as intended. Further, we identified that there are a broad range of views within the department on what the challenges have been and how to best address them. We continue to believe that it is important that these views be presented in the report and have attributed them as appropriate. DOD also commented that we reference a desire within the department to gain “consensus” amongst SSA stakeholders, but thought that “coordinated” was a more appropriate word than consensus, since consensus was not required to produce SSA products. In the report, we did not state that consensus was required, but noted that DOD officials told us that the desire for consensus amongst SSA stakeholders was a contributing factor in making SSA products cumbersome and inflexible. Further, DOD’s 2016 Defense Analytic Guidance similarly identifies the “degree of consensus” as an area requiring SSA process reform. DOD’s final comment noted that the military services used SSA products and routinely conducted sensitivity analysis for their internal use. We recognize in the report that the services conduct a variety of analyses, including some sensitivity analyses. However, we also identify important assumptions that remain untested. As we reported, service officials told us that they have limited analytic capacity and so tend not to do sensitivity analyses on topics unless specifically directed to do so. Further, we noted that the services have been reluctant to conduct or share boundary- pushing analyses through SSA for fear that they will jeopardize their forces or limit their options. As a result of this and the other challenges we identified in this report, the quality of SSA products and analysis and the information provided to senior leaders to inform decision-making has been limited. As DOD moves forward with implementing our recommendations, it will be important that it take the necessary steps to ensure that any future analytic processes thoroughly examine and test key assumptions and look across the joint force. Doing so would help ensure any new process can overcome the constraints that limited the effectiveness of SSA. We are sending copies of this report to congressional committees; the Acting Secretary of Defense; the Acting Under Secretary of Defense for Personnel and Readiness; the Under Secretary of Defense for Policy; the Chairman of the Joint Chiefs of Staff; the Director, Cost Assessment and Program Evaluation; the Secretaries of the Army, the Navy, and the Air Force; and the Commandant of the Marine Corps. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3489 or pendletonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in app. III. Each military service has its own process for determining its force structure requirements using national strategies, defense planning guidance, and Support for Strategic Analysis (SSA) products. Below is a description of each service’s process as of September 2018. Army. The process the Army uses for identifying its force structure needs has two phases: (1) “Capability Demand Analysis” where the Army uses SSA-approved Defense Planning Scenarios to determine how large a force is needed to support the National Defense Strategy and with what mix of units and (2) “Resourcing and Approval” where senior Army leaders assess each capability within the Army to determine where reductions and growth need to occur given available resources. The Secretary of the Army approves changes to force structure through the end of the Future Years Defense Program in a decision memorandum, and these decisions are documented in an Army Structure Memorandum. Navy. The process the Navy uses for identifying its force structure needs begins with the identification of the Navy’s steady-state, peacetime operations requirements. The Navy then conducts campaign and warfighting risk analyses to determine the force’s ability to fight and win SSA-approved Defense Planning Scenarios. Specifically, the Navy tests each force element against the most stressing Defense Planning Scenario, which provides the Navy with its battle force warfighting—to include surge—requirements. These warfighting requirements are compared with steady-state requirements and the more stressing forms the basis of the Force Structure Assessment, which establishes the long-term force structure goals of the Navy’s 30-year shipbuilding plan and aviation plan, and informs the programming and budget processes, among other things. Air Force. The Air Force has a largely decentralized process for identifying its force structure needs that is part of the Air Force’s annual budget development process. The Air Force manages its activities and budgets primarily across 12 Core Functions—the broad capabilities the Air Force provides to the combatant commanders. Much of the force structure analysis that informs budget decisions is also conducted at the Core Function level. The Air Force also conducts occasional leadership-directed studies on future capability needs in certain mission areas (e.g., air superiority needs beyond 2030) as well as a unified risk analysis of its entire force structure that is intended to inform senior leader budget decisions. The Air Force is currently revising its approach to better integrate its capability development and analysis earlier in the process. Marine Corps. The Marine Corps conducts service-level reviews of its force structure at the discretion of the Marine Corps Commandant. A Force Structure Review is typically directed as a result of major service-level issues, such as end strength or capability changes. Marine Corps Force 2025 is the most recent comprehensive assessment of the Marine Corps’ force structure and organization. This was a three-phased effort that relied on one Defense Planning Scenario to develop alternative force structures and evaluate them against a near-peer adversary. The Commandant directed this review to emphasize growing information warfare capabilities. The Marine Corps also conducts Force Optimization Reviews, which are biennial reviews designed to optimize the current and planned future force, taking into consideration new and emerging requirements. Table 3 shows some of the comparable elements of the individual service force structure development processes. In addition to the contact name above, Patricia Lentini, Assistant Director; Nicolaas Cornelisse; Martin De Alteriis; Carly Gerbig; Mae Jones; Amie Lesser; Shahrzad Nikoo; Carol Petersen; and Alex Winograd made key contributions to this report.", "summary": "DOD's 2018 National Defense Strategy continues the department's shift toward focusing on the challenges posed by major powers—China and Russia. The strategy concludes that DOD must pursue urgent change at a significant scale and starkly warns that failure to properly implement the strategy will rapidly result in a force that is irrelevant to the threats it will face. To implement the change DOD envisions, senior leaders must have quality information. Senate Report 115-125 includes a provision for GAO to review DOD's analytic approach for informing force structure decisions to implement the National Defense Strategy. This report assesses, among other things, whether DOD's analytic approach has provided senior leaders with the support needed. GAO reviewed DOD guidance, assessed whether DOD was meeting the objectives identified in its guidance, and interviewed agency officials. This is an unclassified version of a classified report issued in February 2019. Information that DOD deemed classified has been omitted. The Department of Defense's (DOD) analytic approach has not provided senior leaders with the support they need to evaluate and determine the force structure necessary to implement the National Defense Strategy. DOD's analytic approach—Support for Strategic Analysis (SSA)—is used by the services to evaluate their force structure needs and develop their budgets. However, GAO found that SSA has been hindered by three interrelated challenges: Products are cumbersome and inflexible. Although DOD guidance states that SSA products are to be common starting points for analysis on plausible threats, including threats identified in strategic guidance, DOD has not kept the products complete and up to date in part because they were highly detailed and complex and therefore cumbersome to develop and analyze. Analysis does not significantly deviate from services' programmed force structures or test key assumptions. Although DOD's guidance states that SSA should facilitate a broad range of analysis exploring innovative approaches to mitigate threats identified in the strategy, the services generally have not conducted this type of analysis because guidance has not specifically required the services to do so. DOD lacks joint analytic capabilities to assess force structure. Although DOD guidance states that SSA is intended to facilitate the comparison and evaluation of competing force structure options and cross-service tradeoffs, the department has not conducted this type of analysis because it lacks a body or process to do so. DOD efforts to revise its analytic approach are in the early stages and have not yet identified solutions to these challenges. Moreover, DOD has attempted reforms in the past without success. Without a functioning analytic process that addresses the above challenges, senior leaders do not have the analytic support they need to prioritize force structure investments that would best manage risk and address the threats outlined in the National Defense Strategy. GAO recommends that DOD (1) determine the analytic products needed and update them, (2) provide specific guidance requiring the services to explore a range of alternative approaches and force structures, and (3) establish an approach for conducting joint force structure analysis across the department. DOD concurred with the recommendations and noted the department has begun addressing them.", "document_type": "gao"}
{"report": "In the U.S. commercial airline industry, passengers travel on network, low-cost, and regional airlines. With thousands of employees and hundreds of aircraft, network airlines support large, complex hub-and- spoke operations, which provide service at various fare levels to many destinations. Low-cost airlines generally operate under a low-cost business model, which typically includes providing point-to-point service using fewer types of aircraft. Regional airlines typically operate small aircraft—turboprops or regional jets with up to 100 seats—and generally provide service to smaller communities on behalf of network airlines. Airlines rely on a wide variety of IT systems to schedule and transport passengers; some of these IT systems interface with networks operated by travel-booking sites, other airlines, and the FAA. These IT systems touch all phases of a passenger’s travel experience, including booking, check-in, boarding, and baggage, as well as airline operations behind the scene, including flight planning, crew scheduling, and flight dispatch, according to FAA. In addition, aviation stakeholders explained that airline IT systems operate in a dynamic, data-intensive environment that demands around-the-clock availability and real-time information. In recent years, the introduction of new mobile applications and telecommunications infrastructure has added to the myriad systems and network connections now critical to an airline’s operations. Airlines face challenges in maintaining or enhancing their IT systems. For example, some airlines operate a web of IT systems that were developed over many years as manual systems transitioned to electronic and computer-processed functions. Replacing software and upgrading these older systems, such as reservations and crew scheduling, can be complicated undertakings as airlines serve millions of travelers and need to keep data flowing across their networks. For example, in its financial filings, Southwest pointed to the significant challenges and costs involved in introducing new IT capabilities while managing existing systems. Increasingly dependent on the use of IT systems to run its ongoing operations, the company recently completed a multi-year initiative to transition to a new third-party reservation system through Amadeus, among other investments. In addition, a wave of industry consolidation stemming from airline bankruptcies in the late 2000s has affected airline IT systems, requiring significant sustained focus among airlines on merging different IT infrastructures necessary to support worldwide flight operations without interruption. For instance, we previously found that United struggled to integrate computer and reservation systems following its merger with Continental in 2010, although the airline has subsequently completed this transition, according to airline representatives. Likewise, in 2015 American pointed to its reliance on technology when discussing principal risks posed by the integration of its computer, communications, and other technology systems with those of US Airways following the merger of the two airlines. Additionally, some airlines rely on regional partners or third-party IT providers to help manage certain IT systems, such as reservations, crew scheduling, and flight dispatch, further adding to the variety of systems that airlines depend on to run their operations. Moreover, the airline industry is going through a transformation as it shifts to digital merchandizing and retailing to better serve consumers, a process which requires access to real-time information, according to an industry stakeholder. Finally, the speed of technology evolution has accelerated, making it a constant and iterative process to keep systems refreshed and operating in sync, a situation that poses additional challenges, according to a stakeholder. Passengers may be affected by an airline IT outage in different ways depending, in part, on the type and severity of the outage—for example, whether the outage stems from a software glitch or a hardware failure— and the system affected. (See fig. 1.) Effects can range from standing in line to be checked in by a ticket agent instead of using a mobile application to delayed and canceled flights if a hardware failure forces the airline to ground all of its flights until the system is back online. System failures may have cascading effects across other airline IT systems or operations, as well. For example, an outage in a flight dispatch system could cause hours-long delays for subsequent flights. Likewise, aviation stakeholders noted that crew positioning can hinder recovery from an outage as delayed flight crews “time out,” further extending the effects of an outage. In addition to these effects, passengers and airlines can also face higher costs from delayed or canceled travel, including increased operational expenses facing airlines as crews and aircraft sit idle, as well as indirect costs, such as those faced by travelers as their itineraries are delayed or canceled. FAA plays a key, but limited, operational role in responding to airline IT outages. As previously noted, FAA is responsible for ensuring the safe, efficient operation of the NAS. Agency officials we interviewed emphasized that airline IT outages have a limited effect on FAA’s management of the NAS because such outages tend to affect the demand for airspace, not its capacity. As a result, FAA officials explained that if flights are delayed or canceled because of an airline IT outage, the NAS is often less congested for those that remain flying. However, in managing the air-traffic control system, FAA is responsible for initiating and administering traffic management initiatives (such as a ground stop) if requested by an airline experiencing an IT outage. For example, an airline might request that FAA initiate a ground stop if the airline is unable to report flight dispatch information to the FAA, such as the weight and balance of aircraft. FAA works with airlines to accommodate flights back into the NAS when the outage is over. Once an airline recovers from an outage, FAA may also need to initiate traffic management initiatives if demand exceeds capacity in the system— potentially causing delays both for the airline that experienced the outage, as well as others. FAA does not routinely collect data about airline IT outages—which fall outside of its management of the NAS, according to agency officials— although it does collect data on NAS operations, which could include some information about these events. Specifically: The National Traffic Management Log (NTML)—the real-time narrative log of NAS traffic management initiatives kept by air traffic controllers—includes information about ground stops or other initiatives such as time the stop was put in place, affected airports, and when the initiative was lifted. Log entries may also include additional information about the outage, if such information is provided to air traffic control by the airline experiencing it. The Operations Network (OPSNET) system, among others, collects operational data, including air traffic operations and delay data to analyze the performance of the FAA’s air traffic control facilities. However, according to agency officials, data on the effects of airline IT outages (including delay and cancellation data related to airline IT outages) are discarded because information about airline-caused flight disruptions do not provide instructive information to FAA about whether the agency is efficiently operating the NAS. FAA does not directly oversee airline IT systems related to reservations, check-in, baggage, and boarding or their use, according to agency officials. These systems are managed by the airlines themselves. For airline IT systems that interface with FAA’s operational systems, such as automated systems used in air traffic control, FAA works with airlines to ensure that any output (i.e., data feeds) interfaces correctly with the agency’s systems. FAA may provide observations to the airline if its IT systems are not providing accurate information, such as if crews are not being correctly scheduled and tracked, fuel plans are not accurate, or flight plans are not correctly calculated and observable. DOT’s Office of the Assistant General Counsel for Aviation Enforcement and Proceedings and its Aviation Consumer Protection Division are responsible for helping ensure airlines’ compliance with passenger protection requirements and educating passengers on their rights. Airline IT outages are not specifically addressed by any of DOT’s consumer protection regulations. Rather, when these outages occur, they may trigger broader consumer protections afforded passengers. For example, airlines are required by DOT’s interpretation of the statutory prohibition on unfair and deceptive practices to provide refunds for flights that are canceled or significantly delayed if a passenger declines any rerouting that the airline may offer. In the case of delay, however, what amounts to a significant delay is not defined in this policy, and as discussed below, individual airlines may or may not set their own thresholds. According to agency officials, DOT is currently conducting a review of air carriers’ handling of involuntary changes to passengers’ travel itineraries. DOT also regulates compliance through its tarmac delay rule, which requires airlines to mitigate or avoid consumer harm in the event of a lengthy tarmac delay. In addition to these consumer protection regulations and policies, DOT oversees airlines’ compliance with obligations included in airline contracts of carriage or customer service plans. These contracts and plans must be publicly posted by airlines on their websites. As we have previously reported, DOT helps ensure airlines’ compliance with its passenger protection requirements by educating airlines on new regulations or clarifying existing regulations, responding to airlines’ questions, and reviewing airlines’ consumer service policies. According to DOT officials, the agency encourages proactive reporting of incidents by airlines, such as airline IT outages, including a brief description of the incident and any steps taken by the airline to provide accommodation to affected consumers. DOT also receives and investigates complaints from passengers and uses complaint data to identify which airlines to inspect and whether to begin investigations that may result in fines or enforcement actions. According to agency officials, DOT received 126 complaints that explicitly mentioned a domestic airline IT outage from 2015 through 2017. These complaints involved five such outages. For comparison, in all, the agency received between 17,000 and 21,000 complaints per calendar year during that timeframe, according to DOT’s Air Travel Consumer Report. According to DOT officials, complaints that explicitly mentioned an airline IT outage largely mirror in substance those received for other causes of flight disruptions. (These complaints are discussed in more detail below.) According to DOT officials, no investigations have been carried out focusing solely on airline IT outages, but DOT investigations have included airline IT outages that contributed to violations of DOT’s consumer protection regulations. For example, DOT found that an IT outage affecting Delta’s operational systems, including gate management and flight dispatch systems, caused significant surface congestion and resulted in a violation of tarmac delay regulations. This violation was among those included in enforcement proceedings resulting in a civil penalty and consent order to the airline. Finally, to monitor airline on-time performance and baggage handling and to provide information to consumers, DOT requires certain airlines to report data to BTS monthly, including the causes of flight delays and cancellations. However, the causes are grouped into broad categories and do not specify IT outages as a cause. BTS, which is an independent statistical agency within DOT, publishes summary data from reporting air carriers on the number of domestic on-time, delayed, canceled, and diverted flights on its website. DOT’s Office of Aviation Enforcement and Proceedings also publishes a monthly Air Travel Consumer Report with this information. We discuss these data in greater detail below. Using a variety of information sources, we identified 34 airline IT outages from 2015 through 2017 affecting 11 of the 12 airlines in our review. No government data, academic literature, or other information source could be used to determine a comprehensive count of airline IT outages, and information is also limited regarding the types, causes, and effects of these incidents. Additionally, airlines do not regularly share detailed data about their IT outages publicly, such as the number of flights or passengers affected or the technical cause of the outage, although general information about these incidents is sometimes provided on their websites and social media accounts or to the press. To identify airline IT outages in the absence of other sources of information, we validated a preliminary list of outages developed through a review of open source information, including media coverage. This preliminary list was validated through a combination of interviews with the airlines and third-party IT providers and a review of publicly available airline information, FAA NTML log entries, and DOT consumer complaints. Through our validation process, airline representatives and others identified additional airline IT outages that had not been reported or acknowledged publicly by airlines or third-party IT providers, reflecting the variation in quantity or quality of information available regarding these events. For example, we found more information about IT outages that had nationwide or multi-day consumer or operational effects because these incidents garnered more coverage—and often an official airline response—as compared to those that were of shorter duration or affected a regional carrier or smaller number of flights, passengers, or airports. Additionally, we found less or incomplete information on outages at third- party IT providers and regional carriers because their effects were dispersed across multiple airlines. We found that the number and severity of flight disruptions associated with the airline IT outages we identified varied widely. About 85 percent (29 of 34) of our identified outages resulted in some flight disruptions, including 5 outages we identified that caused over 800 delays or cancellations. However, we were unable to verify the exact number of disrupted flights caused by each outage. At least 14 outages resulted in a ground stop, some of which lasted for several hours, according to a review of FAA’s NTML logs. We identified seven outages that had no associated flight disruptions, although they inconvenienced customers in other ways. For example, during these incidents customers experienced problems buying tickets online, checking into flights on an airline’s website, or using frequent flier benefits. Because no comprehensive data are available on airline IT outages and their related effects, we could not compare these incidents with the effects on flights caused by other disruptive events, such as severe weather like hurricanes or snowstorms. However, FAA analysis of two of the IT outages that caused over 800 flight disruptions found that the number of delays or cancellations resulting from these outages was on par with or worse than those caused by severe weather in the same months the outages occurred. Likewise, representatives from one airline stated that operational effects from airline IT outages are comparable to severe weather events, although outages occur much less frequently. An aviation industry representative noted that these events are typically unexpected, hindering the ability of airlines to react and recover. By contrast, disruptions from weather may be forecast ahead of time, allowing airlines to prepare for predicted disruptions, including accommodating customers, adjusting flight crews and schedules, and pre-positioning aircraft, according to the same representative. The airline IT outages we identified were caused by a range of IT and infrastructure issues, according to airline representatives we interviewed and official press statements. These issues included hardware failures, software outages or slowdowns, power or telecommunications failures, and network connectivity issues, among others. In several instances, an IT issue in one airline system had cascading effects across other systems not affected by the initial outage. For example, a large volume of online traffic shut down an airline’s website and subsequently disrupted the airline’s reservations and check-in systems. Representatives from six airlines, an IT expert, and four other aviation industry stakeholders pointed to a variety of factors that could contribute to an outage or magnify the effect of an IT disruption. These factors ranged from underinvestment in IT systems after years of poor airline profitability, increasing requirements on aging systems or systems not designed to work together, and the introduction of new customer-oriented platforms and services. Representatives from airlines we interviewed also described some of their IT system investments and risk mitigation efforts undertaken in response to an outage or to address potential disruptions, such as investing in new backup systems or technologies. For example, five airlines have sought to reduce vulnerability by expanding IT operations beyond a single data center or moving them to the cloud, which allows for the delivery of computing services through the Internet. Likewise, two airlines described efforts to ensure connectivity and reduce the effects of IT disruptions by using multiple telecommunications network providers. Several airline representatives and an IT expert said that these airline IT investments are aimed at enhancing overall system functionality as well as revenue. However, the IT risk expert we spoke with noted that carrying out major upgrades to their IT systems can be challenging because these systems are always in use. Additionally, according to stakeholders we interviewed, airlines employ a variety of contingency planning and recovery strategies to respond to unforeseen technical issues, including IT outages. For example, one airline described incorporating routine system testing, artificial intelligence, and outage drills into planning for system disruptions to avoid outages or speed recovery. Airline efforts to increase the resiliency of their IT systems, such as those described above, could prevent or lessen the impact of such outages. BTS data capture the causes of flight delays and cancellations in several broad categories, which do not isolate flight disruptions resulting from airline IT outages and do not reflect the root cause of flight disruptions. As previously mentioned, BTS collects on-time performance data from the airlines, including the causes of flight delays and cancellations. On a monthly basis, certain airlines are required to report at least one cause of delay (in minutes) for each flight delayed 15 minutes or more from the following five categories: air carrier, extreme weather, NAS, security, and late arriving aircraft. Similarly, for each flight that was canceled, airlines are required to report the cause from one of four categories: air carrier, extreme weather, NAS, and security. BTS guidance instructs airlines to report flight delays that are within the control of the airlines in the air- carrier category. Also included in the air-carrier category, according to the guidance, are more than 40 other potential causes of delays or cancellations, such as aircraft maintenance, baggage, terminal operations, and crew matters. As a result, flight disruptions from IT outages are indistinguishable from other airline-caused issues within this category. Additionally, delays caused by airline IT outages may be captured in a category other than air carrier because of how airlines can report the causes of flight delays based on BTS guidance. For example: Multiple causes for a delay. Airlines have the option to report either just the main cause or all the causes for a flight delay as long as the airline consistently applies the same method in its monthly report to BTS. Also, if there is more than one cause for a flight delay that starts at the same time, airlines are required to report the cause that lasted the longest. As a result, delays caused by an airline IT outage may be attributed to other categories if they happen at the same time as other issues affecting an airline’s operations, such as poor weather or airport conditions. Late arriving aircraft delays. Airlines can report a flight delay in the late arriving aircraft category if the previous flight arrived late and caused the next flight (on the same aircraft) to depart late. Airlines are not required to provide additional information on the cause of the delay for the previous flight (air carrier, NAS, security, or extreme weather). As a result, delays from incidents that can cause ripple effects on an airline’s operations, such as an IT outage or severe thunderstorms, may be attributed to the late arriving aircraft category. NAS delays. Airlines can report delays in the control of the FAA, airport operators, or state and local officials in the NAS category, which includes ground stops, flight volume delays, and air traffic control issues, among others. However, BTS guidance does not specify how airlines should report delays caused by ground stops requested by the airlines, including after an IT outage. As a result, these delays may be captured in the NAS category. BTS data are collected to provide general information on the quality of airline performance to consumers and to improve airline scheduling, rather than detailed information about specific flights or events. Consequently, these data provide limited insight into the effects of individual events, including airline IT outages, both because flight disruptions may be captured in more than one category and because the data do not allow for the isolation of effects for affected flights. We reviewed BTS data for most of the airline IT outages we identified and found, for example, that for 3 outages, airlines reported the largest total number of flight delays in the NAS causal category on the day that the airline requested a ground stop because of the outage—rather than in the air-carrier category. In addition, we reviewed BTS data for the 5 outages we identified where the airline involved delayed or canceled at least 800 total flights and found that airlines spread the causes of flight delays and cancellations across several categories, primarily air carrier, late arriving aircraft, and NAS for the first day of these outages. For example, we found that airlines attributed 44 percent of all reported flight delays to late arriving aircraft for these days. (See fig. 4). DOT officials did not see a need for additional reporting requirements on flight delays and cancellations caused by airline IT outages given the effects of such events are not unique when compared to other causes of flight delay and because these incidents involve a small portion of consumer complaints received by DOT. Aviation stakeholders we spoke to told us that airlines track flight disruptions for internal purposes such as managing operations and scheduling. For example, representatives from one airline said that the airline tracks delays and cancellations associated with IT outages and other issues internally to identify patterns and reoccurring issues that need improvement, such as scheduling, staffing, and maintenance. DOT officials noted that obtaining more detailed information on the causes of flight delays and cancellations would require a cost and benefit analysis to determine whether the benefit from collecting the data would exceed the airlines’ cost to report the data. Officials also noted that the agency has undertaken efforts to provide additional information to consumers. Notably, to provide more insight into the underlying causes of delay attributed to late arriving aircraft, BTS began calculating the original causes of delays in the late arriving aircraft category and providing these data on its website in response to a recommendation made by the DOT Inspector General in 2013. No data are publicly available to quantify with any degree of precision the number of passengers affected by airline IT outages, and only one airline provided this type of information to us. Airline contracts of carriage set the minimum accommodations passengers are entitled to when their flights are delayed or canceled, which could include refunds, rebooking, or other amenities, such as food or meals. However, there is no comprehensive information about the accommodations that were actually received by passengers, and available information is largely anecdotal. Even with respect to the same IT outage, different people may be affected differently. For example, passengers may be affected by the complexity of the NAS and their individual circumstances. According to an airline representative we spoke with, an airline may be able to quickly rebook affected passengers on a different airline for one destination, for example, but may have difficulty rebooking passengers for another destination if other flights are full. Further, while network airlines have hub-and-spoke networks that include a number of route options or frequent service between cities, others—particularly point-to-point or low- cost carriers—may have more limited service, further constraining the ability to rebook individual passengers. Finally, passengers travel for different reasons and their tolerance for disruption can differ, as well, according to DOT officials. Thus, someone flying to visit a friend may have a different tolerance for delay than someone traveling for a job interview, they noted. Airlines are required by DOT to provide refunds for canceled—and significantly delayed—flights if a passenger chooses to cancel his or her trip. Beyond these requirements, however, airlines are not obligated to provide accommodations for flight disruptions such as cancellations and delays unless specified in an airline’s contract of carriage, according to DOT. These contracts govern what, if anything, a passenger is entitled to, although airlines may offer additional accommodations to inconvenienced passengers. Generally accommodations received by inconvenienced passengers could include rebooking on the same airline or alternate travel; refunds or compensation in the form of money or other benefits (e.g., credit for later travel); and amenities such as hotel stays and food, according to their contracts of carriage. Airlines can—and in some cases do—go above and beyond the obligations set forth in their contracts of carriage, as illustrated by some examples below. To better understand the accommodations that passengers may have received as the result of airline IT outages, we reviewed airlines’ contracts of carriage for the airlines in our scope with applicable contracts. None of these contracts addressed IT outages directly, but flight disruptions caused by outages would be covered under the broader contract terms addressing cancellations and delays. We found that the contracts vary in terms of what accommodations are provided for, as well as the extent to which airlines have discretion in providing them. For example, while several airline contracts include provisions to provide hotel vouchers, transportation to the hotel, or meals, other airlines—notably several low- cost carriers—do not. Likewise, some airlines establish set time thresholds for when they are obligated to provide a certain accommodation (e.g., after a delay of at least 4 hours), while others do not. Specific accommodations we identified in our review of airline contracts of carriage are discussed below, and table 1 further details some of the variation that we found. Alternate transportation. All nine airlines in our analysis provide for rebooking on their own airline in the event of a flight delay or cancellation such as might be caused by an airline IT outage, although Frontier includes certain airports near a passenger’s original destination as acceptable alternatives in its contract of carriage. Under this exception, for example, Frontier could rebook a passenger on a flight to Tampa if he or she had originally planned to travel to Orlando, or vice versa, in the event of a flight disruption. Three of the airline contracts of carriage we reviewed provide for travel on a different airline—or the use of alternate ground transportation— typically at their discretion, and a fourth airline provides for alternate transportation if a passenger’s flight has been diverted to a different airport. Airline representatives with two low-cost carriers described their unsuccessful efforts to develop agreements with network airlines to facilitate the rebooking of passengers on another airline. Refunds for cancellations. If a flight is canceled and no alternative is available—or if available flights are not acceptable to the passenger— all nine airlines in our analysis provide for refunds, although three airlines may instead reroute passengers to nearby cities. Under their contracts of carriage, airlines typically provide refunds for the unused portion of a ticket in the event of flight disruptions. If, for example, passengers have already completed the outbound portion of a roundtrip ticket, they would receive a partial refund for the unused, return portion, rather than the entire ticket. Finally, three airlines (Hawaiian, Southwest, and United) offer passengers the option of travel credits in lieu of a refund in their contracts of carriage. Refunds for delays. The majority of airlines in our review provides refunds or flight credit for flight delays, although refunds in some cases could be contingent on the absence of an acceptable alternative, such as being rebooked on a subsequent flight or to an alternate airport. As mentioned above, DOT requires airlines to provide refunds for flights that are “significantly delayed” but does not define how long such a delay is and instead relies on a case-by-case determination. Four of the contracts we reviewed establish a specific timeframe for the delay after which a passenger is entitled to a refund, while the others do not establish such a threshold. For example, a passenger flying on Alaska Airlines could request and receive a refund for a flight disruption lasting at least 2 hours, and passengers on Delta are entitled to a refund, if requested, after a 90 minute delay. By contrast, airlines without a defined threshold for a delayed flight have discretion for when passengers would be eligible for refunds, particularly with regard to nonrefundable tickets. Hotel stay. The majority of airlines in our review provide for hotel stays in their contracts of carriage (and ground transportation to the hotel), to varying degrees, although two low-cost carriers (Frontier and Southwest) do not. The contracts of carriage for seven airlines include a hotel stay for passengers inconvenienced by flight disruptions, and of these four stipulate that passengers have to be away from home or from their points of origin or destination; five require that the flight disruption span certain hours (e.g., 10pm to 6am); and one includes credit for a long-distance phone call. Four of the contracts we reviewed include additional provisions for hotel stays (or other accommodations) to passengers with disabilities or other needs. For example, under its contract of carriage, American will provide amenities to maintain the safety and welfare of certain passengers if they are delayed (e.g., customers with disabilities, unaccompanied children, the elderly, or others with special needs or circumstances). Food. Three airlines in our review provide for meals for passengers inconvenienced by flight disruptions in their contracts of carriage. For example, JetBlue’s contract of carriage provides for meal vouchers or pizza for flight delays of 6 or more hours. In addition, airlines may deliver meals or offer other amenities to passengers waiting for delayed or canceled flights, even in the absence of the promise of food in the contract of carriage. In these cases, additional accommodations may be publicly announced on airline websites, by social media accounts, or through statements to the press, or they may be provided directly to individual flights or passengers at the airport. For example, in response to severe thunderstorms in 2017, Delta had pizza delivered to passengers waiting in airports across the Southeast. Monetary compensation or travel credit. Inconvenienced passengers are not entitled to monetary compensation in the case of a flight delay or cancellation in the United States, and none of the airlines in our review includes such compensation in their contracts of carriage. Nevertheless, two airline contracts of carriage include provisions for travel credit—above and beyond a refund—for flight disruptions. JetBlue’s contract of carriage provides for travel credit for canceled or delayed flights with several tiers, depending on the timing of the cancellation or length of the delay. For example, passengers delayed over 6 hours are entitled to $250 credit for future travel on JetBlue. Likewise, Alaska’s contract provides for a discount code for future travel (and a letter of apology) for passengers delayed longer than 2 hours. Although not included in Delta’s contract of carriage, the airline provided $200 in travel vouchers to all customers with flight disruptions lasting at least 3 hours for two of the IT outages we identified, according to airline representatives. As mentioned above, collecting and analyzing passenger complaints is one way DOT helps ensure that an airline fulfills its obligations included in its contract of carriage and customer service plan, as well as any additional accommodations that may be publicly offered. Our review of passenger complaints filed with DOT stemming from airline IT outages found that they included complaints related to the lack of monetary compensation for delayed or canceled flights and refusals to refund other expenses, such as rental cars or missed hotel or cruise reservations, among other concerns. For example, complaints related to a Southwest outage in 2016 included several related to lack of compensation or other amenities, such as food or hotel stays offered by the airline. As noted above, Southwest’s contract of carriage does not provide for compensation, food, or hotel stays in the event of a delay or cancellation. Complaints filed after the Delta outage of 2016 acknowledged receipt of a $200 travel voucher in compensation or a hotel voucher, but pointed to other non-refunded expenses incurred or difficulties in redeeming these vouchers. The three consumer or passenger advocacy groups with whom we spoke raised several concerns with regard to passengers inconvenienced by airline IT outages. Stakeholders we spoke with responded to these concerns and addressed how airlines respond to IT outages. Passengers may not receive the same accommodations. In the absence of requirements for accommodations or compensation, passengers are dependent on whether or not the affected airline chooses to be generous, according to the consumer advocates we interviewed. They also noted that mileage plan or first class passengers may receive more accommodations than others, even when passengers are affected by the same underlying outage, as may be true in other circumstances, as well. Representatives from one airline told us that they attempt to promptly address the needs of all of their passengers but acknowledged that accommodations may vary depending on passenger circumstances, including passenger status (e.g., frequent-flyer program members or VIP travelers). Airline obligations toward affected passengers may be confusing for passengers. According to consumer advocates we spoke with, even if a passenger understands that an airline’s contract of carriage lays out its obligations to passengers affected by an IT outage, these contracts are often lengthy and difficult to understand. As noted above, our review of DOT complaints stemming from airline IT outages found that many passengers expected to receive compensation or other accommodations in response to these events, although such accommodations were not included in contracts of carriage. We reported in 2017 that airlines committed to reviewing their contracts of carriage to see if they could be simplified. Contracts of carriage may not clearly exclude IT outages from force majeure events, according to consumer advocates. Flight disruptions caused by extreme weather, terrorism, and other events that are seen as being beyond the control of the airline are typically treated as special situations in airline contracts of carriage, and as a result, inconvenienced passengers may not receive accommodations that they otherwise might. Consumer advocates voiced concerns that airline IT outages might be treated as events outside the airline’s control (i.e., Acts of God or force majeure events) given ambiguity in how these exceptions are defined. We found that IT outages were not explicitly included among the force majeure events identified in the contracts of carriage we reviewed. In interviews and written statements, representatives with four of the airlines in our review varied in the extent to which they characterized airline IT outages as incidents in the control of the airline, but generally indicated that passengers would be accommodated as if the outages were. We provided the Department of Transportation (DOT) with a draft of this report for review and comment. DOT responded by email and provided technical clarifications, which we incorporated into the report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or KrauseH@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Our objectives for this report were to: identify (1) the Department of Transportation’s (DOT) and Federal Aviation Administration’s (FAA) roles, if any, in relation to airline IT outages and their effects and (2) what is known about these outages, including the number of flights and passengers affected. The scope of this report focuses on those airline IT systems that affect passenger experiences, including systems related to reservations and check-in, as well as those used by airlines for flight planning and dispatch. Our scope excluded IT systems involved in avionics (such as aircraft navigation systems); in-flight operations (such as passenger WiFi networks); and internal operations (such as company email systems). Our analysis included the 12 airlines that were required to report on-time performance information to DOT’s Bureau of Transportation Statistics (BTS) from 2015 through 2017, including network carriers (Alaska, American, Delta, and United); low-cost carriers (Frontier, JetBlue, Spirit, Southwest, and Virgin America); regional carriers that provide service for partner airlines (ExpressJet and SkyWest); and Hawaiian, which provides a niche service. Given the role of third-party IT providers, we also included Amadeus and Sabre in our scope. To identify relevant DOT and FAA authorities and responsibilities vis-à-vis airline IT outages in several areas, including operations, oversight, and data-collection, we reviewed relevant laws, regulations, policies, and guidance, as well as prior GAO work addressing agency roles. We interviewed DOT officials with BTS, which collects data on airline on-time performance, and the Office of the Assistant General Counsel for Aviation Enforcement and Proceedings and its Aviation Consumer Protection Division, which oversee consumer protections and receive consumer complaints. We also interviewed FAA officials with the Office of the Chief Information Security Officer, which advises the agency on matters relating to IT management and security. Within FAA’s Air Traffic Organization, we interviewed officials with Systems Operations Services, which administers traffic management initiatives including ground stops, and its National Airspace System (NAS) Operations and Office of Performance Analysis. These two offices are responsible for programs related to air traffic control systems and assessing the performance of the NAS, respectively. Through our review of relevant plans and an interview with officials in DOT’s Office of the Secretary, we determined that airline IT systems are not included in federal plans for critical infrastructure protection. According to DOT officials, outages in these systems do not have the potential to reach established thresholds for potential casualties or damages. By contrast, air traffic control systems and airports are included in sector-specific plans addressing critical infrastructure protection in the case of a terrorist attack or other natural or manmade disaster. To determine what is known about airline IT outages, we reviewed DOT data sources, including BTS and FAA performance and operations data, as well as passenger complaints received by DOT in response to airline IT outages from 2015 through August 2018. We also conducted interviews with or received written responses from 11 (of 12) airlines in our scope, and interviewed other stakeholders, including third-party IT system providers Amadeus and Sabre; an IT risk expert (Robert Charette); industry associations, including Airlines for American (A4A), the Regional Airline Association (RAA), and Airports Council International (ACI); and employee union representatives with the Air Line Pilots Association (ALPA). We determined that DOT and FAA data were not designed, and could not be used, to comprehensively identify airline IT outages. To identify airline IT outages in the absence of detailed DOT or FAA data, academic literature, or internal (proprietary) airline data on these incidents, we validated a preliminary list of such outages developed using open source material that included media coverage and publicly available airline sources for outages from 2015 through 2017. Specifically, we searched GAO subscription databases (e.g., ProQuest, Nexis, and EBSCO) to create a preliminary list of 37 airline IT outages from media coverage; performed additional searches of articles and official airline websites to collect more information on and corroborate incidents identified; provided our list of identified IT outages to the 12 airlines in our scope and two third-party IT providers (Amadeus and Sabre) for confirmation; and corroborated 20 of the identified IT outages with FAA’s National Traffic Management Log’s (NTML) log entries and DOT’s consumer complaint data. Through this process, we were able to corroborate 34 airline IT outages from 2015 through 2017, and we are confident that our list of outages includes all of the outages large enough to garner national-level, multi- day media coverage and an official response from an airline executive. While accurate, our list is not comprehensive because three airlines and a third-party IT provider identified additional outages that we did not find in our preliminary search, including one airline that shared information on more than 20 additional outages. We did not include these additional outages in our count to ensure that our methodology was consistent. To account for outages that may have occurred subsequent to our review, we identified an online listing of airline IT outages and validated 9 of 11 of the outages included from 2018 through January 2019 using publicly available airline or airport information or coverage in at least 3 media sources. This list and our validation process provides evidence that airline IT outages continued to occur during this timeframe, but does not match the rigor applied to the identification of outages we identified from 2015 through 2017. As a result, we are not confident that this list identified all of the outages large enough to garner national-level, multi- day media coverage and an official response from an airline executive. Once we had identified airline IT outages through other sources and could look at data for specific dates, we were able to use DOT and FAA data to provide additional insight into flight disruptions (i.e., flight delays or cancellations) and ground stops caused by outages. For example, we requested that FAA conduct analysis on 3 of the 34 outages we had identified to determine what FAA operational data could reveal about the effects of these outages. We selected these 3 outages to reflect a range of flight disruptions for comparative analysis, including variations in size and cause of the outage. We also assessed the extent to which the effects on passengers could be seen in the BTS on-time performance data reported by airlines. For these data, we sought to determine the cause and magnitude of delays and cancellations for each outage. We also reviewed NTML log entries for the dates of known outages to further identify potential information, including incidents of ground stops. Finally, to obtain more information about the potential effects on passengers resulting from these events, we reviewed consumer complaints to DOT stemming from airline IT outages. These complaints were provided to us by DOT’s Aviation Consumer Protection Division and include reference to the associated outage. To understand how airlines accommodate inconvenienced passengers, we reviewed airline contracts of carriage for 9 of the 12 the airlines in our scope. These contracts are the legally binding contracts between carriers and passengers and may include specific provisions such as refund procedures and responsibility for delayed flights, among other things. We excluded two regional airlines (ExpressJet and SkyWest) that operate under the contracts of carriage of their mainline partners and Virgin America, which merged with Alaska in 2018 and no longer has a separate contract of carriage. In addition to the stakeholders mentioned above, we also interviewed consumer or passenger advocacy groups, including representatives with the Consumers Union, the National Consumers League, and Travelers United to identify any concerns regarding consumers affected by airline IT outages. We conducted this performance audit from February 2018 to June 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Heather Krause, (202) 512-2834 or KrauseH@gao.gov. In addition to the individual named above, other key contributors to this report were Jonathan Carver, Assistant Director; Molly Laster, Analyst-in- Charge; Neha Bhatt; David Hooper; Rich Hung; Delwen Jones; SaraAnn Moessbauer; Emily Mussey; Josh Ormond; Corinne Quinones; Pamela Snedden; James Sweetman, Jr.; and Elizabeth Wood.", "summary": "In recent years, the airline industry experienced several well-publicized IT system outages to reservation, check-in, flight planning, and other systems. Such outages can result in widespread disruption to air travel, inconveniencing passengers, who may be delayed or face out-of-pocket costs, and can also affect airlines' revenue and operations. Airlines are responsible for operating and maintaining their IT systems. GAO was asked to review airline IT outages. GAO examined: (1) DOT's and FAA's roles related to airline IT outages and (2) what is known about these outages and their effects on passengers. GAO identified relevant federal laws and responsibilities and interviewed DOT and FAA officials. In the absence of DOT and FAA data to identify airline IT outages, GAO identified outages using open source documents for the 12 airlines reporting to BTS from 2015 through 2017 and validated these outages using a multi-step process with publicly available airline information, interviews with airline representatives, and FAA and DOT data. GAO also reviewed airlines' contracts of carriage, which are legally binding contracts between airlines and passengers, to understand how airlines accommodate passengers inconvenienced by IT outages, as well as 140 consumer complaints related to airline IT outages received by DOT from 2015 through June 2018. The Department of Transportation (DOT) and, within it, the Federal Aviation Administration (FAA) have limited roles overseeing or addressing the effects of outages from information technology (IT) systems that airlines rely on to schedule and transport passengers (e.g., reservation or flight planning systems). FAA's operations and oversight. At an airline's request, FAA may halt the operation of all or part of that airline's flights during an outage and work with the airline to reintegrate flights upon recovery. FAA does not directly oversee airline IT systems but works with airlines to ensure that airline data interfaces correctly with FAA's operational systems. DOT's consumer protection. Airline IT outages are not specifically addressed in DOT's consumer protections for passengers, although other protections may apply, such as restrictions on tarmac delays if a passenger is held on a flight during an outage. DOT oversees airlines' adherence to their contracts with passengers. These may include specific provisions such as refund procedures and responsibility for delayed flights, among other things. DOT also receives consumer complaints and uses complaint data to initiate investigations that may result in fines or enforcement actions. DOT's data collection. DOT requires large airlines to report information about on-time performance to the Bureau of Transportation Statistics (BTS), including the causes of flight delays and cancellations in several broad categories (e.g., airline caused, weather, and late-arriving aircraft). Using multiple sources, GAO identified 34 IT outages from 2015 through 2017, affecting 11 of 12 selected airlines. No government data were available to identify IT outages or determine how many flights or passengers were affected by such outages. BTS data provide information to consumers about airline performance broadly but are not designed to identify the effects of individual events, such as the number of flight delays and cancellations resulting from IT outages. According to GAO's validation of multiple sources, however, about 85 percent of the identified outages resulted in some flight delays or cancellations. Because of limited data, information about how passengers have been inconvenienced from outages is largely anecdotal (see figure for examples of inconveniences). Further, airlines vary in what they provide to these passengers (e.g., food, hotel, or rebooking on another airline) when IT outages occur. Consumer complaints stemming from IT outages accounted for less than one percent of all complaints received by DOT from 2015 through June 2018, and according to agency officials, these complaints raised concerns similar to complaints resulting from other causes of flight disruption. Complaints reviewed by GAO included the lack of food, a hotel, or compensation, among other things.", "document_type": "gao"}
{"report": "Federal trust funds are an accounting mechanism used to link dedicated collections with their expenditure for a specific purpose or program (see textbox). Earmarked or Dedicated Collections Our budget glossary (GAO-05-734SP) includes two definitions of earmarking: 1. Dedicating collections by law for a specific amount for particular purposes by means of legislative language. Our 2001 report on trust funds (GAO-01-199SP) used the term “earmarked receipts” in accordance with the first definition. We use the term “dedicated collections” instead to avoid confusion between the two definitions. One of the earliest trust funds established was the Civil Service Retirement and Disability Fund, set up in 1920. In the federal budget, the meaning of the term “trust fund” differs significantly from its private sector usage. In the case of federal trust funds, the federal government owns the assets of federal trust funds, does not have a fiduciary responsibility to trust beneficiaries, and can raise or lower future trust fund collections and payments or change the purposes for which the collections are used by changing existing laws. Designation as a trust fund does not in and of itself impose a greater commitment on the government to carry out program activities than it has to carry out other government activities. It can, however, indicate the government’s intent to restrict the use of those funds to the specified purpose and—especially for a program funded in whole or in part by its beneficiaries—may influence debates about program changes. OMB and Treasury determine budgetary designation as a trust fund when a law both dedicates collections to a program and identifies the account as a “trust fund.” Trust funds, however, are not the only way dedicated collections are accounted for in the federal budget. Special funds and public enterprise funds also link dedicated collections with their expenditure for a specific purpose or program and are analogous to non- revolving and revolving trust funds, respectively (see figure 1). For the purpose of this report, we examine budget accounts designated as “trust funds” by OMB and Treasury and those that link dedicated collections with their expenditure. There are two other fund types in the federal budget that we did not include: general fund accounts, which hold all federal money not allocated by law to any other fund account, and intragovernmental fund accounts, which facilitate financing transactions primarily within and between federal agencies. The four fund types included in our definition of trust funds and other dedicated funds are: Non-revolving Trust Fund. An account designated as a “trust fund” by law that is credited with dedicated collections, which can often, but not always, be used without further appropriation action. For example, the Federal Hospital Insurance (HI) Trust Fund, also known as Part A of Medicare, is financed primarily through payroll taxes levied on workers and their employers and finances health care services related to stays in hospitals, skilled nursing facilities, and hospices for eligible beneficiaries. Special Fund. Analogous to a non-revolving trust fund but not classified as a trust fund in name. For example, the Universal Service Fund subsidizes telecommunication carriers that provide telecommunications services to all consumers, including low-income consumers, schools and libraries, and those who live in rural or high- cost areas. Revolving Trust Fund. An account designated as a “trust fund” by law that is credited with collections that are generally available for obligation without further appropriation action to carry out a cycle of businesslike operations in accordance with statute. For example, the Employees Health Benefits Fund collects health insurance premiums from federal employees, annuitants, and their employing agencies and disburses payments to private insurers who participate in the Federal Employees Health Benefits program. Public Enterprise Fund. Analogous to a revolving trust fund but not classified as a trust fund in name. A public enterprise fund is a type of revolving fund that carries out a cycle of businesslike operations, mainly with the public, in which it charges for the sale of products or services and uses the proceeds to finance its spending, usually without requirement for annual appropriations. The Postal Service Fund of the United States Postal Service is an example of this type of fund. Trust funds and dedicated funds have their own dedicated collections and the ability to retain accumulated balances. From the perspective of the trust fund or other dedicated fund, the accumulated balances represent the value of past taxes, fees, and the other income received by the fund in excess of past spending by the fund. The accumulated balances are not cash. Most money collected and disbursed by the federal government is held in the General Fund of the U.S. Government (General Fund). The dedicated taxes and fees collected from the public are deposited in the General Fund and the General Fund disburses the fund’s benefit and other payments to the public. When the General Fund receives the cash, the trust fund or other dedicated fund records an asset for these collections and the General Fund records a liability to the fund, which essentially means the trust fund has “lent” money to the General Fund. As cash is disbursed, these asset and liability accounts are reduced. From the government-wide perspective, the trust fund or dedicated fund asset and General Fund liability accounts eliminate with each other in consolidation. Some trust funds and other dedicated funds have the legal authority to invest their balances, most of which are held in U.S. Treasury securities. The value of the securities held is recorded as “debt held by government accounts” and represents debt owed by one part of the government to another (i.e., intragovernmental debt). In many ways, the special U.S. Treasury securities held by government accounts are indistinguishable from the marketable government debt sold to the public. A maturity date is set, interest is accrued at established market rates, and the securities count as part of the total federal debt. Generally, these securities are not traded in the financial markets and are able to be redeemed on demand by the government account. The interest they earn is credited to the fund accounts in the form of additional Treasury securities or is used to pay current expenses or benefits. Interest earned by government accounts on their Treasury securities is an internal transaction, made between two accounts within the federal government, and constitutes an expense for Treasury. Treasury must pay back the debt held by government accounts when these accounts need to redeem their securities to be able to make their expenditures. When this happens, Treasury must obtain cash to finance the government’s spending either through increasing taxes, cutting spending, or increasing borrowing from the public. Entitlement authority is another way to classify budget authority, but OMB’s budget data do not include that classification. Discretionary spending refers to budget authority that is provided in and controlled by appropriations acts. Mandatory spending, also known as direct spending, refers to budget authority provided in laws other than appropriations acts and the outlays that result from such budget authority. Entitlement authority is the authority to make payments to any person or government if, under the provisions of the law, the U.S. government is legally required to make the payments to persons or governments that meet the requirements. Generally, entitlement authority is a type of mandatory spending. The classification of the budget authority within a trust fund or other dedicated fund as mandatory or discretionary determines how budget control mechanisms apply. By itself, designation as a trust fund does not determine whether spending is controlled through the annual appropriations process or what limitations apply. Trust funds and dedicated funds are subject to various enforcement mechanisms intended to control revenues, spending, and deficits. The Balanced Budget and Emergency Deficit Control Act of 1985 (BBEDCA) first established sequestration, which is the cancellation of budgetary resources under a presidential order. The act set deficit reduction targets for the federal government and established sequestration procedures to enforce those targets. The Budget Control Act of 2011 amended BBEDCA and revived this budgetary enforcement mechanism by reinstating budget limits (also known as “caps”) to encourage agreement on deficit reduction legislation or, in the event that such agreement was not reached, to automatically reduce spending so that an equivalent budgetary goal would be achieved. Appropriations from trust funds and other dedicated funds designated as discretionary count toward these limits. The Statutory Pay-As-You-Go Act of 2010 (PAYGO) specifies a second type of sequestration triggered under certain conditions. The act establishes a permanent budget enforcement mechanism intended to prevent enactment of mandatory spending and revenue legislation that would increase the federal deficit. The act requires OMB to track costs and savings associated with enacted legislation and to determine at the end of each congressional session if net total costs exceed net total savings. If the costs exceed the savings, a separate sequestration will be triggered. Consequently, the same mandatory accounts that are subject to sequestration under BBEDCA could incur further reductions if a secondary PAYGO sequestration is triggered. PAYGO does not control the growth in spending that results from previously enacted laws, nor does it control discretionary spending. Hundreds of programs across the federal government are supported in whole or in part by a trust fund or other dedicated fund. Our analysis of OMB’s budget data shows 398 active federal trust funds and other dedicated funds in fiscal year 2018. Non-revolving trust funds and special funds make up the greatest number of these types of accounts and also hold the greatest total balances. See table 2. Our analysis of another government-wide source, Treasury’s Combined Statement, records 647 trust and other dedicated fund accounts in fiscal year 2018. This count is higher because Treasury includes accounts with smaller balances and does not combine groups of related accounts. Of the accounts in Treasury’s Combined Statement, 150 have balances that are below $500,000 and would fall below OMB’s rounding threshold of $1 million. The trust funds and other dedicated funds in Treasury’s Combined Statement are spread across all 29 major departments that are reported separately in the statement (see figure 2). Each department has at least two such accounts. The distribution of the number of trust fund or other dedicated fund accounts across federal agencies does not correspond with the balances held by these accounts. For example, the Social Security Administration has only four such accounts, but those four funds together held $2.9 trillion—more than double the balances of any other agency at the end of fiscal year 2018 (see figure 3). In contrast, the Department of the Interior had the greatest number of trust funds and other dedicated funds, but these 118 funds together held $14.9 billion, which is less than 1 percent of the total balances held in these types of accounts at the end of fiscal year 2018. The total balance in federal trust funds and other dedicated funds grew about 13 percent in nominal terms from fiscal year 2014 to fiscal year 2018. The five accounts that contributed the most to this overall growth are listed in table 3. Fund balances are affected by complex interactions of various economic, demographic, and programmatic factors, but these changes are reflective of some overarching trends. For example, the balances of civilian and military pension and benefit programs increased, in part reflecting agency and employee contributions to fund the ongoing accrual of benefits by civilian and military personnel. Treasury has also contributed to these accounts to help fund some of the benefits accrued in the past. Some of the other increases were a result of economic changes experienced during this time period such as declines in the unemployment rate, among other things. For example, both Social Security’s Federal Old-Age and Survivors Insurance Trust Fund (OASI) and the Unemployment Trust Fund are funded primarily by payroll taxes, which tend to gather more revenue during periods when employment goes up and wage growth increases. While the net change in total trust fund and other dedicated fund balances was positive from fiscal year 2014 to 2018, not all trust fund and other dedicated fund balances grew over the time period. The five accounts that experienced the largest balance decreases are listed in table 4. From fiscal year 2014 to 2018, the average trust fund and other dedicated fund balance decrease was less than the average balance increase, and a greater number of accounts increased than decreased over the period. About 28 percent of the 398 accounts in our scope had individual balances that changed less than $5 million over the time period (see table 5). The higher total balance in trust funds and other dedicated funds indicates an overall surplus—income exceeding outgo—from fiscal year 2014 to 2018, which could suggest that the federal government intends to dedicate more resources to these specified purposes. Neither the increased total balance nor an individual fund’s balance increase is a signal that any individual fund is on sound financial footing. Similarly, a decreasing balance does not necessarily signal that any individual fund is not on sound financial footing. Assessing the future outlook for some of these funds and programs requires actuarial or other projections and can be subject to various degrees of inherent uncertainty. Of our 13 case study accounts, 11 received income from general revenues in addition to their dedicated collections, either through a permanent appropriation or in an annual appropriation. The form, size, and purpose of income from general revenues that our case study accounts received varied greatly based on the design of the program. These accounts fall in to three basic types: those that received regular income from general revenues as a part of their program design, those that received intermittent general revenue income, and those that received income solely from their own dedicated collections. See appendix II for more detailed information about the income, outgo, investments, and current issues for each of these accounts. Eight of the case study accounts we examined regularly receive income from general revenues in addition to their dedicated collections. These general revenues are often for specific purposes that have been deemed public goods and are provided annually as a part of the program’s design. The Medicare Supplementary Medical Insurance trust fund sets medical insurance premium rates for Medicare Part B to cover 25 percent of expected costs for the year. The roughly 75 percent remaining expected program cost is funded through general revenue. The Medicare HI Trust Fund also regularly receives general revenues to reimburse the fund for the cost for certain uninsured beneficiaries, program management activities, and other miscellaneous activities. In fiscal year 2018, $1.6 billion in general revenue was transferred into the trust fund. Both the Civil Service Retirement and Disability Fund (CSRDF) program and the Federal Employees Health Benefits Fund receive contributions from both current employees and their employing agencies as their primary sources of income, but these accounts also receive some general revenue in addition to these dedicated collections. Treasury is required by law to transfer an amount annually to the CSRDF from the General Fund to subsidize in part the under- funding of the Civil Service Retirement System. The Civil Service Retirement System is closed to new participants but covers most federal employees who first entered a covered position prior to 1984. According to OPM officials, the Federal Employees Health Benefits program is funded about 30 percent by contributions from participants and about 70 percent by contributions from their employing agencies. OPM contributes the employer share of the premiums for most annuitants via an appropriation from general revenues. The U.S. Postal Service (USPS) receives annual appropriations from general revenues to fund mail for the blind and overseas absentee voting. These appropriations account for less than 0.1 percent of the total cash outlays of the Postal Service Fund. USPS received $58 million in appropriations for these activities in fiscal year 2018, when total outlays were $69 billion. The Social Security Trust Funds, both OASI and DI, receive reimbursements from general revenue for several distinct purposes, such as employee union expenses and the payroll tax holiday, among other things. The total appropriations for these two activities were about $23 million in fiscal year 2018. While the Airport and Airway Trust Fund primarily receives dedicated collections, it has received some appropriations from general revenue in recent years, and some of the programs it funds also receive regular appropriations from general revenue. The most prominent example is the operations and management account within the Federal Aviation Administration. While this account is funded mostly by transfers from the Airport and Airway Trust Fund, it also typically receives an annual appropriation from general revenues. In fiscal year 2018, the appropriation to the operations account was $1.36 billion, which was about 13 percent of the total budget authority in the account. Three of the case study accounts we examined were supported in part by general revenue income on an intermittent basis in recent years. These general revenues helped temporarily restore solvency to programs that were not designed to be fiscally sustainable. The Highway Trust Fund has received appropriations from general revenues as a part of its reauthorization process in recent years. The most recent reauthorization, provided $70 billion in general revenue to the Highway Trust Fund from fiscal year 2016 through fiscal year 2020. The appropriations have allowed outlays to exceed dedicated collections in most years without exhausting assets in the fund. The National Flood Insurance Fund had $16 billion of its debt canceled by the Additional Supplemental Appropriations for Disaster Relief Requirements Act, 2017. This cancellation converted a $16 billion liability of the fund to a cost borne by general revenues. However, the National Flood Insurance Program (NFIP) still owes $20.5 billion to Treasury. As we recently reported, NFIP likely will not generate sufficient revenues to cover its expenses and repay its outstanding debt because its premium rates do not reflect the full risk of loss. The Flood Insurance Reserve Fund did not directly benefit from the debt cancellation, but it did receive an indirect benefit since it was established as a reserve fund to help meet expected future obligations and repay debt. Two of our case study accounts did not receive income from general revenue in recent years. For both of these accounts, the agencies have some authority to adjust their dedicated collections to cover their projected costs. The flexibility to adjust income levels based on projections can help contribute to the sustainability of the funds. Although the Tennessee Valley Authority (TVA) was originally funded primarily by appropriations from Congress when it was established in 1933, TVA fulfilled its requirement to repay this investment in 2014 and currently collects enough revenue to cover its operating expenses. The TVA Board has the authority to determine rates for its electric power and the Tennessee Valley Authority Act of 1933 mandates that TVA keep rates as low as feasible while still collecting sufficient revenue. The Universal Service Fund (USF) does not receive income from general revenue. The Federal Communications Commission (FCC) has some flexibility to set the contribution factor, which determines the payments telecommunications carriers are required to make into the fund. FCC officials told us that they must set the rates at levels so that they collect enough in dedicated collections to cover the projected demand for the programs they have adopted. FCC sets the contribution factor quarterly to cover the projected cost of the USF programs for the upcoming quarter, up to the authorized level of spending for each program. Even funds that rely primarily on their dedicated collections may not be fiscally sustainable. For example, the Social Security OASI and DI, and Medicare HI trust funds do not receive income from general revenues to support benefit payments. However, projections show that their dedicated collections are expected to be insufficient to fully cover scheduled outlays in the next 7 to 33 years. Conversely, some accounts supported by the Airport and Airway Trust Fund received appropriations from general revenue in recent years. However, the Airport and Airway Trust Fund has received more in dedicated collections than are made available to outlay through appropriations. As such, the fund carries a balance that is unavailable without further appropriations action. At the end of fiscal year 2018, the total cash balance in the Airport and Airway Trust Fund was about $17 billion. CBO projects this balance to grow more than threefold over the next 10 years. Although overall federal trust and other dedicated fund balances grew over the past 5 fiscal years, this trend is not projected to continue. In CBO’s most recent trust fund projections, overall federal trust fund and special fund balances are projected to start declining in fiscal year 2022. CBO does not estimate projected balances for public enterprise funds. As shown in figure 4, the projected decline is largely explained by declines in the Social Security and Medicare fund balances. We have previously reported that demographic factors, such as an aging population and slower labor force growth, are contributing to a gap between Social Security program costs and revenues. According to the most recent Social Security Trustees Report, Social Security’s costs, on a combined OASI and DI basis, have exceeded its non-interest income since 2010 and are projected to exceed total income, including interest, starting in 2020. The Medicare and Social Security Trustees and CBO projections show that several major trust funds will deplete their assets in the next 3 to 33 years (see figure 5). If no action is taken, these trust funds are projected to be unable to fully support paying their projected obligations. Projected trust fund balances can provide a vital signaling function for policymakers about underlying fiscal imbalances in covered programs. However, program sustainability is ultimately determined by whether the government as a whole has the economic capacity to finance the claims on the trust funds at the cost of other competing priorities. The economic flexibility of the federal government may be limited as debt held by the public grows as a percentage of gross domestic product (GDP). Debt held by the public was $15.8 trillion—or 78 percent of GDP—at the end of fiscal year 2018. It is projected to surpass its historical high of 106 percent of GDP within 13 to 20 years, and climb to between about 250 to 500 percent by 2092. Further, neither the long-term projections of federal debt nor CBO’s trust fund balance projections include certain fiscal risks that could affect the federal government’s financial condition in the future. Fiscal risks, or fiscal exposures, are responsibilities, programs, and activities that may legally commit or create expectations for future federal spending. Many of the largest trust funds and other dedicated funds face fiscal risks that are highlighted in our High-Risk List due to the financial uncertainty they face. For example, USPS—USPS financial viability continues to be high-risk because USPS cannot fund its current level of services and financial obligations from its revenues. Pension Benefit Guaranty Corporation (PBGC)—PBGC’s liabilities exceeded its assets by about $51 billion as of the end of fiscal year 2018. PBGC’s financial future remains uncertain, due in part to a long- term decline in the number of traditional defined benefit plans and the collective financial risk of the many underfunded pension plans PBGC insures. NFIP—Emphasizing affordability has led to premium rates that in many cases do not reflect the full risk of loss and produce insufficient premiums to pay for claims. Highway Trust Fund (HTF)—The nation’s surface transportation system is under growing strain and the cost to repair and upgrade the system to meet current and future demand is estimated in the hundreds of billions of dollars. A sustainable solution would balance revenues to and spending from the HTF. Ultimately, major changes in transportation spending or in revenues, or in both, will be needed to bring the two into balance. The Medicare Program—Medicare continues to challenge the federal government because of its outsized impact on the federal budget and the health care sector as a whole, the large number of beneficiaries it serves, and the complexity of its administration. Federal spending for Medicare programs is expected to significantly increase in the coming years. As overall trust and special fund balances are projected to decrease, our projections and those from the Fiscal Year 2018 Financial Report of the United States Government, and CBO show that the federal government will have to borrow more from the public to offset the decrease in intragovernmental debt. We have reported that existing federal debt held by the public is already large by historical norms, and CBO has noted that large and growing amounts of federal debt held by the public would have negative long-term consequences for the economy and constrain future budget policy. To change the long-term fiscal path, policymakers will likely need to consider policy changes to the entire range of federal activities, both revenue and spending. During fiscal year 2018, almost 98 percent of outgo (i.e., outlays and transfers to another government account) from trust funds and other dedicated funds was mandatory budget authority. This is greater than the proportion of total federal spending that is mandatory. According to OMB, during fiscal year 2018, mandatory spending made up 69.3 percent of all federal outlays while discretionary spending accounted for the remaining 30.7 percent. Seventy-six percent of trust funds and other dedicated funds had some mandatory budget authority (see table 6). Some funds have a mix of mandatory and discretionary budget authority. In general, the collections and balances of accounts with mandatory spending authority are available for obligation. Mandatory authority provides some flexibility for agencies because they do not have to await congressional action to incur obligations and make payments. For example, the Social Security Trust Funds have mandatory budget authority, which authorizes the program to continue to make payments to beneficiaries during lapses in appropriations. Although programs with mandatory authority need not go through the annual appropriations process, they are still subject to congressional oversight. In some cases Congress has set obligation limits in annual appropriations acts. For example, although the Crime Victims Fund has mandatory budget authority to obligate funds from its available balances, limits in annual appropriations acts have often capped the amount that may be obligated in each fiscal year. As a result, annual income has exceeded outgo and the balance of the fund had grown to $16.6 billion at the end of fiscal year 2018. Designation as mandatory or discretionary budget authority determines how budget control mechanisms are applied to the funds. Sequestration applies annually to mandatory spending, but certain budget authority is exempt or subject to special rules. Of the 13 case studies we reviewed, nine are exempt from cancellation under budget enforcement sequestration procedures and four—Medicare Supplementary Medical Insurance, Medicare Hospital Insurance, the HTF, and the Airport and Airway Trust Fund—are partially sequestrable (i.e., certain budgetary resources specified by law within the accounts are not subject to cancellation under budget enforcement sequestration procedures). For example, Social Security, Medicaid, and veterans’ compensation are completely exempt, and Medicare reductions are limited to 2 percent. Exemptions and special rules lead sequestration to affect some areas of the federal government more than others. For example, programs without exempt status, such as the Commodity Credit Corporation Fund, bear a greater reduction than they would if cuts were applied evenly to all programs. Outgo from those trust funds and other dedicated funds that do not have mandatory budget authority are controlled in the annual appropriations process and count toward the annual discretionary spending limits laid out in the Budget Control Act of 2011 (BCA). For example, outlays from the Airport and Airway Trust Fund are discretionary. This means that the outlays for capital improvements and operations of the nation’s airport and airway system, except for airport grants, count toward government- wide discretionary spending limits. Some trust funds and other dedicated funds have a combination of budget authorities, which can affect balances. For example, the Harbor Maintenance Trust Fund (HMTF), which is supported through collections of the harbor maintenance fee, has mandatory income and discretionary outlays. Historically, HMTF income has exceeded outgo and by the end of fiscal year 2018, the balance in the fund had grown to $9.3 billion. Any proposed legislation to lower the fee revenues would require an offset so as not to increase the deficit. Conversely, since the spending is subject to the discretionary caps, any increase in spending to align with program revenues would count toward the discretionary spending limits. Most spending from trust funds and other dedicated funds is mandatory and not controlled by the annual appropriations process. We have previously reported that the increase in mandatory spending has long- term implications for the nation’s fiscal outlook overall, including the growing federal debt. The federal government has previously enacted fiscal rules in the form of laws that constrain fiscal policy decisions, including BCA and PAYGO. These fiscal rules apply the same way regardless of status as a trust fund or other dedicated fund. However, in practice, fiscal rules that apply to mandatory budget authority are more relevant to these types of accounts, because mandatory budget authority is more concentrated in trust funds and other dedicated funds than it is in the federal budget as a whole. Of the 23 largest trust funds and other dedicated funds we reviewed, 13 have entitlement authority, which legally requires payments to individuals or governments that meet the requirements of the programs (see table 7). For example, OASI beneficiaries are legally entitled to benefits based on a formula that takes into account the time they spent working and their earnings, among other factors. Some trust funds have mandatory budget authority, but not entitlement authority. For example, the USF, the National Flood Insurance Reserve Fund, and the Tennessee Valley Authority Fund all have mandatory budget authority, but have no entitlement authority. These programs have the most flexibility because their income is available without further appropriations action and their outgo is not driven by legal requirements to individuals or governments. For example, Federal Communication Commission officials told us that they manage the size of each program funded by the USF, to stay within an approved budget. Although entitlements represent a current legal commitment and trust funds and other dedicated funds demonstrate the government’s intent to restrict the use of those funds to a specific purpose, the government can change the terms of entitlement programs, including those financed through trust funds or other dedicated funds, by changing the substantive law. Congress and the President can raise or lower future trust fund collections or payments or change the purposes for which the collections can be used. For example, in 1983 a number of changes were made to the Social Security program, including an increase in the full retirement age and a new tax on a portion of Social Security benefits, which increased collections and lowered future outgo. We provided a draft of this report and the online dataset to the Director of OMB and the Secretary of the Treasury for review and comment. We also provided a draft of this report and the online dataset to our case study agencies: the Centers for Medicare & Medicaid Services, the Federal Communications Commission, the Federal Emergency Management Agency, the Department of Transportation (for the Federal Aviation Administration and the Federal Highway Administration), the Office of Personnel Management, the Social Security Administration, the Tennessee Valley Authority, and the U.S. Postal Service for review and comment. The Social Security Administration and the U.S. Postal Service provided written responses thanking us for providing the opportunity to review the report, which are published in appendixes III and IV. The Centers for Medicare & Medicaid Services, the Federal Communications Commission, the Department of Transportation, the Office of Personnel Management, the Tennessee Valley Authority, and the U.S. Postal Service provided technical comments, which we incorporated as appropriate. OMB, Treasury, and the Federal Emergency Management Agency reviewed our draft report and had no comments. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 7 days from the report date. At that time, we will send copies to interested congressional committees, the Director of the Office of Management and Budget, the secretaries and agency heads of the departments and agencies in our review, and other interested parties. In addition, the report is available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Tranchau (Kris) T. Nguyen at (202) 512-6806 or nguyentt@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report examines: (1) how the size and scope of federal trust funds and other dedicated funds in the federal budget have changed over time, (2) the extent to which federal trust funds and other dedicated funds are supported by their dedicated collections, and (3) the extent to which federal trust funds and other dedicated funds support mandatory programs, including major entitlement programs. To examine trends in the size and scope of federal trust funds and other dedicated funds, we used Office of Management and Budget (OMB) budget data to identify the income, outgo (i.e., outlays and transfers to another government account), and end of year balances for all revolving trust funds, special funds, non-revolving trust funds, and public enterprise funds reported in OMB’s budget database, OMB MAX, for fiscal years 2014 to 2018 in nominal terms. We excluded financing and credit accounts because they are non-budgetary. For the majority of these data we used the amounts reported in OMB MAX schedule J, which is used to produce the Status of Funds tables in the President’s Budget Appendix. While the list of accounts that report Status of Funds tables publicly in the budget is limited to 21 accounts, a schedule J is created in OMB MAX for all non-revolving trust funds and special funds and, for the years in our review, for all revolving trust funds. Schedule J data are not available for public enterprise funds, so we used guidance from OMB Circular No. A- 11 to approximate similar income, outgo, and balance data using data fields that are reported in the Program and Financing table. The public enterprise fund data are slightly different than the other fund types because borrowing authority as it is reported in OMB MAX only includes information on repayable advances and excludes information on outstanding debt and borrowing. We asked OMB staff to review our methodology to calculate these numbers and they agreed our approach was methodologically sound. To assess the reliability of OMB MAX data related to the income, outgo, and balances of trust fund and other dedicated fund accounts, we reviewed related documentation, interviewed knowledgeable OMB staff, and conducted electronic data testing. We found these data reliable for our purposes. OMB budget data are rounded to the nearest million and do not show funds with amounts less than $500,000. Accordingly, OMB instructs agencies to consolidate small trust fund accounts with larger general fund accounts so the total government-wide amounts will be complete. In addition, OMB sometimes reports trust fund groups under a single account rather than each individual trust fund account. Groups may include two or more trust funds with similar purposes. The Department of the Treasury (Treasury), on the other hand, tracks monies for each discrete account to the penny in order to fulfill its government wide accounting and cash management responsibilities. As such, we used data from the Treasury Fiscal Year 2018 Combined Statement of Receipts, Outlays, and Balances of the United States Government to provide a complete count of these funds, including accounts with small balances and accounts that are a part of groups. We interviewed Treasury officials, reviewed relevant documentation, and conducted electronic and manual testing to ensure the data were reliable for our purposes and concluded that they were. To examine the extent to which federal trust funds and other dedicated funds are supported by their dedicated collections, in addition to the data described above, we examined thirteen case study accounts in nine agencies. We selected a set of accounts to include the largest of each of the four types of trust funds and other dedicated funds and a variety of program designs (see table 8). We used gross outlays from fiscal year 2017 to identify the largest accounts, since that was the most recently available data at the time of the account selection. Overall, our selected accounts covered 88 percent of the total gross outlays among these types of accounts in fiscal year 2017. We also ensured that our set of case study accounts included: at least one account from each of the four fund types in our scope, a range of programs from different goals (e.g., infrastructure, insurance, federal employee benefits), and budget authority with different characteristics. The budget authority included in our case study selection represented examples of both mandatory and discretionary budget authority. We also ensured that budget authority from appropriations, borrowing authority, contract authority, and offsetting collections were represented in at least one case study. While the case studies were selected to capture the largest funds and a diversity of programs and funding characteristics, findings from the case studies cannot be generalized to all trust funds and other dedicated funds. We also reviewed agency financial, budget, and performance reports, Congressional Budget Office trust fund projections, the 2019 Annual Report of the Board of Trustees of the Federal Old-Age and Survivors Insurance and Federal Disability Insurance Trust Funds (Social Security Trustees), the 2019 Annual Report of the Board of Trustees of the Federal Hospital Insurance and Federal Supplementary Medical Insurance Trust Funds (Medicare Trustees), and our prior reports, and interviewed officials from each of the case study agencies. To examine the extent to which federal trust funds and other dedicated funds support mandatory programs, including major entitlement programs, we used OMB budget data to calculate the prevalence of discretionary budget authority, which is controlled through appropriations acts, and mandatory budget authority, which generally refers to budget authority provided through laws other than appropriations acts, in federal trust funds and other dedicated funds. OMB budget data do not systematically identify entitlement authority. To determine which of the largest trust funds and other dedicated funds have entitlement authority, we analyzed the authorizing statutes for our case study accounts and 10 additional accounts with the next largest gross outlays. While the entitlement analysis was designed to cover nearly all of the total outlays from these types of accounts, the findings from this analysis are not representative of all trust funds and other dedicated funds and cannot be generalized to the other 375 accounts in our scope. We also reviewed budget enforcement mechanisms, such as sequestration, that apply to these types of budget authority through review of relevant laws, our prior work, and OMB documents. We conducted this performance audit from October 2018 to January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. To illustrate the variety of federal trust funds and other dedicated funds, and examine the extent to which they are supported by their dedicated collections, we examined 13 case study accounts in nine agencies. We selected accounts listed in table 9 to include the largest of each of the four types of trust funds and other dedicated funds and a variety of program designs. Each case study profile in this appendix includes income, outgo, investments, and current issues related to the account, as well as the following account information: Fund types. OMB and Treasury designate budget accounts as “trust funds” and other fund types that link dedicated collections with their expenditure based on legislation. The fund types in this appendix include: Non-revolving trust fund. An account designated as a “trust fund” by law that is credited with dedicated collections, which can often, but not always, be used without further appropriation action. Special fund. Analogous to a non-revolving trust fund but not classified as a trust fund in name. Revolving trust fund. An account designated as a “trust fund” by law that is credited with collections that are generally available for obligation without further appropriation action, to carry out a cycle of businesslike operations in accordance with statute. Public enterprise fund. Analogous to a revolving trust fund but not classified as a trust fund in name. A public enterprise fund is a type of revolving fund that carries out a cycle of businesslike operations, mainly with the public, in which it charges for the sale of products or services and uses the proceeds to finance its spending, usually without requirement for annual appropriations. Entitlement authority. Whether or not outgo from the fund is controlled by an entitlement authority, which is the authority to make payments to any person or government if the U.S. government is legally required to make the payments to persons or governments that meet the requirements established by law. The Budget Enforcement Act category. OMB’s designation as to whether the funds in the account are classified as discretionary or mandatory depending on the nature of the substantive legislation creating the fund. Discretionary. Budget authority provided in and controlled through appropriations acts. Mandatory. Budget authority provided through laws other than appropriations acts, and the outlays that result from such budget authority. Sequestration status. OMB’s designation of the authority for purposes of sequestration, which is the cancellation of budgetary resources under a presidential order. We defined the status categories as follows: Exempt. Accounts for which budgetary resources are exempt from cancellation under budget enforcement sequestration procedures. Sequestrable. Accounts for which budgetary resources are subject to cancellation under budget enforcement sequestration procedures. Partially Sequestrable. Accounts for which certain budgetary resources specified by law within the account are not subject to cancellation under budget enforcement sequestration procedures. In addition to the contact named above, Susan E. Murphy (Assistant Director), Katherine D. Morris (Analyst in Charge), Alicia Cackley, Janice Ceperich, Jacqueline Chapin, Steven Cohen, Michael Collins, James Cosgrove, Robert Dacey, Karin Fangman, Paul Foderaro, Carol Henn, James A. Howard, Susan J. Irving, Charles Jeszeck, Kenneth John, Heather Krause, Natalie Logan, Scott McNulty, John Mingus, Sally Moino, Tracie Sanchez, Lori Rectanus, Frank Rusco, Dawn Simpson, Frank Todisco, Peter Verchinski, and Alicia White made key contributions to this report.", "summary": "Some of the largest federal programs, including Medicare, Social Security, and postal services, are funded through trust funds and other dedicated funds, which link collections that have been dedicated to a specific purpose with the expenditures of those collections. While these funds have the ability to retain accumulated balances, these collections do not necessarily fund the full current or future cost of the government's commitments to the designated beneficiaries. GAO was asked to review issues related to federal trust funds and other dedicated funds. This report examines (1) how the size and scope of federal trust funds and other dedicated funds in the federal budget have changed over time, (2) the extent to which these funds are supported by their dedicated collections, and (3) the extent to which these funds support mandatory programs, including major entitlement programs. GAO analyzed OMB data on trust funds and other dedicated funds for fiscal year 2014 through 2018 and the Department of the Treasury's (Treasury) Fiscal Year 2018 Combined Statement of Receipts, Outlays, and Balances . GAO also examined 13 case study accounts in nine agencies, selected to include the largest of each type of these funds and a variety of program designs. GAO reviewed agency reports, CBO trust fund estimates for 2018 and projections for 2019 to 2029, and prior GAO reports, and interviewed OMB staff and officials from Treasury and each of the case study agencies. GAO also is providing an online dataset of these funds at https://www.gao.gov/products/GAO-20-156 . Every major federal department has at least two trust funds or other dedicated funds. According to GAO analysis of Office of Management and Budget (OMB) data, balances in these funds, which can be used to support covered programs, grew 13 percent in nominal terms from fiscal year 2014 through 2018. Fund balances are affected by complex interactions of factors, but the total increase was driven largely by military and civilian retirement fund balances. The Congressional Budget Office (CBO) projects the total balance to start declining in fiscal year 2022 as decreases in Medicare and Social Security will exceed increases in military and civilian retirement balances. To offset the overall decrease, the federal government is projected to borrow more from the public. GAO found that 11 of 13 case studies recently received general revenue—collections that are not dedicated by law for a specific purpose. For example, medical insurance premiums for Medicare Part B are set to cover 25 percent of expected costs; the remaining 75 percent are covered by general revenues. Even funds that rely primarily on their dedicated collections may not be fiscally sustainable. For example, the Social Security Old-Age and Survivors Insurance Trust Fund only uses dedicated collections for benefit payments, but its balances are projected to be depleted by 2034. Nearly 98 percent of outlays and transfers from trust funds and other dedicated funds was through mandatory authority, which allows agencies to make payments without further congressional action. Most of the 23 largest funds also have entitlement authority, which generally requires payments to eligible parties based on legal requirements. Status as a trust fund, mandatory program, or entitlement does not prevent Congress and the President from changing related laws to alter future collections or payments.", "document_type": "gao"}
{"report": "In our December 2018 report, we found that TSA provides pipeline operators with voluntary security guidelines that operators can implement to enhance the security of their pipeline facilities. TSA also evaluates the vulnerability of pipeline systems through security assessments. Pipeline operators and industry association representatives who we interviewed also reported exchanging risk-related security information and coordinating with federal and nonfederal entities, including TSA. However, we also identified weaknesses in several areas of TSA’s pipeline security program management, including: (1) updating and clarifying pipeline security guidelines; (2) planning for workforce needs; (3) assessing pipeline risks; and (4) monitoring program performance. We found in our December 2018 report that all of the pipeline operators and industry association representatives that we interviewed reported receiving security information from federal and nonfederal entities. For example, DHS components including TSA’s Intelligence and Analysis and NCCIC share security-related information on physical and cyber threats and incidents. Nonfederal entities included Information Sharing and Analysis Centers, fusion centers, industry associations, and subsector coordinating councils. Pipeline operators also reported that they share security-related information with TSA and the NCCIC. For example, TSA’s Pipeline Security Guidelines requests that pipeline operators report physical security incidents to the Transportation Security Operations Center (TSOC) and any actual or suspected cyberattacks to the NCCIC. According to TSA officials, TSOC staff analyzes incident information for national trends and common threats, and then shares their observations with pipeline operators during monthly and quarterly conference calls. In our December 2018 report, we found that the pipeline operators we interviewed reported using a range of guidelines and standards to address their physical and cybersecurity risks. For example, all 10 of the pipeline operators we interviewed stated they had implemented the voluntary 2011 TSA Pipeline Security Guidelines that the operators determined to be applicable to their operations. Five of the 10 pipeline operators characterized the guidelines as generally or somewhat effective in helping to secure their operations, 1 was neutral on their effectiveness, and 4 did not provide an assessment of the guidelines’ effectiveness. Pipeline operators and industry association representatives reported that their members also use the Interstate Natural Gas Association of America’s Control Systems Cyber Security Guidelines for the Natural Gas Pipeline Industry, the American Petroleum Institute’s Pipeline SCADA Security standard, and the National Institute of Standards and Technology’s (NIST) Cybersecurity Framework as sources of cybersecurity standards, guidelines, and practices that may be scaled and applied to address a pipeline operator’s cybersecurity risks. We found that TSA’s Pipeline Security Branch had issued revised Pipeline Security Guidelines in March 2018, but TSA had not established a documented process to ensure that revisions occur and fully capture updates to supporting standards and guidance. The guidelines were revised to, among other things, reflect the dynamic threat environment and to incorporate cybersecurity principles and practices from the NIST Cybersecurity Framework, which was initially issued in February 2014. However, because NIST released version 1.1 of the Cybersecurity Framework in April 2018, the guidelines that TSA released in March 2018 did not incorporate cybersecurity elements that NIST added to the latest Cybersecurity Framework, such as the Supply Chain Risk Management category. Without a documented process defining how frequently TSA is to review and, if deemed necessary, revise its guidelines, TSA cannot ensure that the guidelines reflect the latest known standards and best practices of physical security and cybersecurity. We recommended that TSA implement a documented process for reviewing, and if deemed necessary, revising TSA’s Pipeline Security Guidelines at regular defined intervals. DHS agreed and estimated that this effort would be completed by April 30, 2019. In April 2019, TSA provided us with documentation outlining procedures for reviewing these guidelines. We are currently assessing this information to determine if it sufficiently addresses this recommendation. We also found that TSA’s Pipeline Security Guidelines lacked clarity in the definition of key terms used to determine critical facilities. TSA initially identifies the 100 highest risk pipeline systems based on the amount of material transported through the system. Subsequently, pipeline operators are to use criteria in the Guidelines to self-identify the critical facilities within those higher risk systems and report them to TSA. TSA’s Pipeline Security Branch then conducts CFSRs at the critical facilities identified by pipeline operators. However, our analysis of TSA’s data found that at least 34 of the top 100 critical pipeline systems TSA deemed highest risk indicated that they had no critical facilities. Three of the 10 operators we interviewed stated that some companies that reported to TSA that they had no critical facilities may possibly be taking advantage of the guidelines’ lack of clarity. For example, one of TSA’s criteria for determining pipeline facility criticality states that if a facility or combination of facilities were damaged or destroyed, it would have the potential to “cause mass casualties or significant health effects.” Two operators told us that individual operators may interpret TSA’s criterion, “cause mass casualties or significant health effect,” differently. For example, one of the operators that we interviewed stated that this criterion could be interpreted either as a specific number of people affected or a sufficient volume to overwhelm a local health department, which could vary depending on the locality. Without clearly defined criteria for determining pipeline facilities’ criticality, TSA cannot ensure that pipeline operators are applying guidance uniformly, that all of the critical facilities across the pipeline sector have been identified, or that their vulnerabilities have been identified and addressed. We recommended that TSA’s Security Policy and Industry Engagement’s Surface Division clarify TSA’s Pipeline Security Guidelines by defining key terms within its criteria for determining critical facilities. DHS agreed and estimated that this effort would be completed by June 30, 2019. TSA conducts pipeline security reviews—CSRs and CFSRs—to assess pipeline vulnerabilities and industry implementation of TSA’s Pipeline Security Guidelines. However, the number of reviews conducted has varied widely from fiscal years 2014 through 2018. These reviews are intended to develop TSA’s knowledge of security planning and execution at critical pipeline systems and lead to recommendations for pipeline operators to help them enhance pipeline security. For an overview of the CSR and CFSR processes, see Figure 1 below. We found that the number of CSRs and CFSRs completed by TSA has varied during the last five fiscal years, ranging from zero CSRs conducted in fiscal year 2014 to 23 CSRs conducted in fiscal year 2018, as of July 31, 2018 (see Figure 2 below). TSA officials reported that staffing limitations had prevented TSA from conducting more reviews. TSA Pipeline Security Branch staffing levels (excluding contractor support) also varied significantly over the past 9 years ranging from 14 full-time equivalents in fiscal years 2012 and 2013 to one in fiscal year 2014 (see Table 1 below). TSA officials stated that, while contractor support has assisted with conducting CFSRs, there were no contractor personnel providing CSR support from fiscal years 2010 through 2017, but that contractors increased to two personnel in fiscal year 2018. TSA officials stated that they expected to complete 20 CSRs and 60 CFSRs per fiscal year with Pipeline Security Branch employees and contract support, and had completed 23 CSRs through July 2018 for fiscal year 2018. In addition, pipeline operators that we interviewed emphasized the importance of cybersecurity skills among TSA staff. Specifically, 6 of the 10 pipeline operators and 3 of the 5 industry representatives we interviewed reported that the level of cybersecurity expertise among TSA staff and contractors may challenge the Pipeline Security Branch’s ability to fully assess the cybersecurity portions of its security reviews. We found that TSA had not established a workforce plan for its Security Policy and Industry Engagement or its Pipeline Security Branch that identified staffing needs and skill sets such as the required level of cybersecurity expertise among TSA staff and contractors. We therefore recommended that TSA develop a strategic workforce plan for its Security Policy and Industry Engagement Surface Division, which could include determining the number of personnel necessary to meet the goals set for its Pipeline Security Branch, as well as the knowledge, skills, and abilities, including cybersecurity, that are needed to effectively conduct CSRs and CFSRs. DHS agreed and estimated that this effort would be completed by July 31, 2019. The Pipeline Security Branch has developed a risk assessment model that combines all three elements of risk—threat, vulnerability, and consequence—to generate a risk score for pipeline systems. The Pipeline Security Branch developed the Pipeline Relative Risk Ranking Tool in 2007 for use in assessing various security risks to the top 100 critical pipeline systems based on volume of material transported through the system (throughput). The risk ranking tool calculates threat, vulnerability, and consequence for each pipeline system on variables such as the amount of throughput in the pipeline system and the number of critical facilities using data collected from pipeline operators, as well as other federal agencies such as the Departments of Transportation and Defense. The ranking tool then generates a risk score for each of the 100 most critical pipeline systems and ranks them according to risk, which was information used by TSA to prioritize pipeline security assessments. However, in our December 2018 report we found that the last time the Pipeline Security Branch calculated relative risk among the top 100 critical pipeline systems using the ranking tool was in 2014. Since the risk assessment had not changed since 2014, information on threat may be outdated and may limit the usefulness of the ranking tool in allowing the Pipeline Security Branch to effectively prioritize reviews of pipeline systems. We recommended that the Security Policy and Industry Engagement’s Surface Division update the Pipeline Relative Risk Ranking Tool to include up-to-date data to ensure it reflects industry conditions, including throughput and threat data. DHS agreed and in March 2019 TSA officials reported taking steps to update the data in the Pipeline Risk Ranking Tool to reflect current pipeline industry data. We are currently reviewing those actions to determine if they sufficiently address our recommendation. We also found that some of the sources of data and vulnerability assessment inputs to the ranking tool were not fully documented. For example, threats to cybersecurity were not specifically accounted for in the description of the risk assessment methodology, making it unclear if cybersecurity threats were part of the assessment’s threat factor. We recommended that the Security Policy and Industry Engagement’s Surface Division fully document the data sources, underlying assumptions, and judgments that form the basis of the Pipeline Relative Risk Ranking Tool, including sources of uncertainty and any implications for interpreting the results from the assessment. In March 2019, TSA officials stated that they had taken steps to document this information. We are currently reviewing those steps to determine if they sufficiently address our recommendation. In our December 2018 report, we also found that TSA developed three databases to track CSR and CFSR recommendations and their implementation status by pipeline facility, system, operator, and product type. TSA officials stated that the primary means for assessing the effectiveness of the agency’s efforts to reduce pipeline security risks was through conducting pipeline security reviews—CSRs and CFSRs. However, while TSA does track CFSR recommendations, we found that TSA had not tracked the status of CSR recommendations for security improvements in over 5 years—information necessary for TSA to effectively monitor pipeline operators’ progress in improving their security posture. We recommended that TSA take steps to enter information on CSR recommendations and monitor and record their status. DHS agreed and estimated that this effort would be completed by November 30, 2019. Chairman Rush, Ranking Member Upton, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff members have any questions about this testimony, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals making key contributions to this work include Ben Atwater, Assistant Director; Steve Komadina, Analyst-in-Charge; Nick Marinos, Michael Gilmore, Tom Lombardi, Chuck Bausell and Susan Hsu. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "More than 2.7 million miles of pipeline transport and distribute natural gas, oil, and other hazardous products throughout the United States. Interstate pipelines run through remote areas and highly populated urban areas, and are vulnerable to accidents, operating errors, and malicious physical and cyber-based attack or intrusion. Pipeline system disruptions could result in commodity price increases or widespread energy shortages. Several federal and private entities have roles in pipeline security. TSA is primarily responsible for the federal oversight of pipeline physical security and cybersecurity. This statement summarizes previous GAO findings related to TSA's management of its pipeline security program. It is based on a prior GAO product issued in December 2018, along with updates as of April 2019 on actions TSA has taken to address GAO's recommendations from the report. To conduct the prior work, GAO analyzed TSA documents, such as its Pipeline Security Guidelines ; evaluated TSA pipeline risk assessment efforts; and interviewed TSA officials, 10 U.S. pipeline operators—a non-generalizable sample selected based on volume, geography, and material transported—and representatives from five pipeline industry associations. GAO also reviewed information on TSA's actions to implement its prior recommendations. The Department of Homeland Security's (DHS) Transportation Security Administration (TSA) has developed and provided pipeline operators with voluntary security guidelines, and also evaluates the vulnerability of pipeline systems through security assessments. However, GAO's prior work, reported in December 2018, identified some weaknesses and made recommendations to strengthen TSA's management of key aspects of its pipeline security program. Pipeline security guidelines . GAO reported that TSA revised its voluntary pipeline security guidelines in March 2018 to reflect changes in the threat environment and incorporate most of the principles and practices from the National Institute of Standards and Technology's (NIST) Framework for Improving Critical Infrastructure Cybersecurity. However, TSA's revisions do not include all elements of the current NIST framework and TSA does not have a documented process for reviewing and revising its guidelines on a regular basis. GAO recommended that TSA implement a documented process for reviewing and revising TSA's Pipeline Security Guidelines at defined intervals. TSA has since outlined procedures for reviewing its guidelines, which GAO is reviewing to determine if they sufficiently address the recommendation. Workforce planning . GAO reported that the number of TSA security reviews of pipeline systems has varied considerably over time. TSA officials stated that staffing limitations within its Pipeline Security Branch have prevented TSA from conducting more reviews. Staffing levels for the branch have varied significantly, ranging from 1 full-time equivalent in 2014 to 6 from fiscal years 2015 through 2018. Further, TSA does not have a strategic workforce plan to help ensure it identifies the skills and competencies—such as the required level of cybersecurity expertise—necessary to carry out its pipeline security responsibilities. GAO recommended that TSA develop a strategic workforce plan, which TSA plans to complete by July 2019. Pipeline risk assessments . GAO identified factors that likely limit the usefulness of TSA's risk assessment methodology for prioritizing pipeline security reviews. For example, TSA has not updated its risk assessment methodology since 2014 to reflect current threats to the pipeline industry. Further, its sources of data and underlying assumptions and judgments regarding certain threat and vulnerability inputs are not fully documented. GAO recommended that TSA update its risk ranking tool to include up-to-date data to ensure it reflects industry conditions and fully document the data sources, assumptions and judgments that form the basis of the tool. As of April 2019, TSA reported taking steps to address these recommendations. GAO is reviewing documentation of these steps to determine if they sufficiently address the recommendations. Monitoring performance . GAO reported that conducting security reviews was the primary means for TSA to assess the effectiveness of its efforts to reduce pipeline security risks. However, TSA has not tracked the status of key security review recommendations for the past 5 years. GAO recommended that TSA take steps to update information on security review recommendations and monitor and record their status, which TSA plans to address by November 2019 GAO made 10 recommendations in its December 2018 report to strengthen TSA's management of its pipeline security program. DHS agreed and has described planned actions or timeframes for addressing these recommendations.", "document_type": "gao"}
{"report": "HUD’s WCF was established in 2016 to provide a mechanism for the department to centralize and fund federal shared services used across HUD offices and agencies. According to its Committee Charter, the three goals of the WCF are to: align incentives for efficient enterprise operations through users paying for goods and services; establish accurate and timely cost estimates for goods and services; improve planning, increase visibility and transparency, and support the efficient and effective delivery of goods and services. To begin WCF operations in fiscal year 2016, HUD transferred approximately $44 million in funding from the salaries and expenses accounts of OCFO and OCHCO to the newly established WCF. In fiscal year 2017, the WCF began to bill its customers—17 HUD offices that purchase services financed through the fund—directly for their estimated use of services. HUD’s WCF is different from other intragovernmental revolving funds that we have previously reviewed in that it does not fund internally provided services at this time. The WCF is currently used as a centralized funding method to pay for the costs of four established shared services agreements—or interagency agreements—between HUD and three external shared service providers: USDA’s National Finance Center (NFC) and Treasury’s Administrative Resource Center (ARC) and Shared Services Programs. See table 1 for more information about the agencies providing shared services to HUD. According to WCF Division officials, HUD plans to expand the WCF in the future to finance both internal and additional external goods and services. For example, the WCF’s fiscal years 2019 and 2020 budget justifications requested funding to centralize and support activities such as a Data Management Initiative and the Real Estate Assessment Center’s (REAC) physical and financial assessment services, respectively. However to date, HUD did not receive budgetary authority to proceed with including either activity in the WCF. Several offices within HUD share responsibility for the management and operations of the WCF, including the WCF Division and business line offices. See figure 1 for information on the WCF’s financial operations and entities involved. HUD defines most of the roles and responsibilities for management and oversight of the WCF. According to HUD policy and guidance documents: The WCF Committee provides financial and operational oversight of the WCF, including advising and supporting the WCF’s strategic direction and providing annual approval of the WCF financial plan and budget, among other responsibilities. The Committee includes representation from OCFO leadership and all customer offices. The WCF Division within OCFO oversees the financial management of the fund, including managing day-to-day operations and establishing cost accounting for all shared services and customers that use the fund. In addition, the WCF Division supports customers with WCF-specific services, such as billing and service usage reports. OCFO and OCHCO manage the provision of the services financed through the WCF to customer offices. As the designated business line offices, OCFO and OCHCO oversee the quality and timely delivery of services, including monitoring service provider performance and serving as the liaisons between HUD customer offices and service providers concerning any issues with service quality. WCF Customers place orders with the WCF Division to receive services from the external service providers. In addition, customers reimburse the WCF for their estimated use of those services. However, HUD also performs additional actions to support the efficient and effective delivery of goods and services consistent with the goals of the WCF. Specifically, the WCF Division conducts business process analyses to identify opportunities for efficiencies across the department. Yet, there is no mention in guidance of the roles and responsibilities of the WCF Division, business line offices, or other stakeholders in identifying, monitoring, and implementing the actions recommended because of these analyses. In support of the WCF’s goal to support efficient and effective delivery of goods and services, the WCF Division provides quarterly usage reports to customers and business line offices and assists them with monitoring their consumption and associated costs of shared services. WCF Division officials told us they conduct a more detailed review of the data when they find anomalies, such as unusually high volumes of transactions. WCF Division officials told us that they will collaborate with the responsible business line office to conduct a business process analysis, which is used to identify actionable ways to address the cause of the high service volume and costs in specific circumstances. A business process analysis is generally conducted when there is an availability of resources, support from the business line office, and potential for cost savings or operational efficiencies. For example, in 2018, in response to an increase in the volume of two service areas overseen by OCFO—help desk calls and commercial purchase order accruals—the WCF Division examined data and determined that HUD could reduce its service volume and costs. See text box below. Working Capital Fund (WCF) Business Process Analysis Help desk calls: The WCF Division found that an unnecessarily high number of customer calls to help desks for password resets were contributing to higher costs to the department. For example, more than 20 percent of customer calls to Treasury ARC’s financial management help desk were from customers requesting password resets, which can be manually resolved without calling the help desk and incurring a transaction fee. In fiscal year 2019, the cost to HUD per financial management help desk call was about $128. According to the WCF Division Director, the WCF Division presented its findings to OCFO leadership and the WCF Committee, including five recommendations targeted at reducing system password reset call volume and future costs to the department. Commercial purchase order transactions: Commercial purchase order accruals are more costly because they are manually processed. Among other findings, the WCF Division’s analysis determined that changing OCFO’s current business process for obligations below a certain threshold could reduce the volume of transactions processed. The OCFO official told us that OCFO plans to implement one of the recommendations with a new process for recording those accruals in the first quarter of fiscal year 2020 to reduce the volume of transactions. According to the WCF Division’s analyses, the implementation of this recommendation could achieve potential annual costs savings of nearly $600,000. However, OCFO has not taken actions to address seven remaining recommendations, which the WCF Division found could produce additional benefits, including potential cost savings of more than $400,000 annually. The OCFO official told us that OCFO plans to examine HUD’s fiscal year 2019 service usage to determine the effectiveness of the actions it has already taken to reduce the help desk call and commercial purchase order transaction volume. The estimated help desk call volume and associated cost to HUD for a given year, as agreed upon in HUD’s agreement with Treasury ARC, is generally based on the average of the previous 2 years’ call volume. As such, changes in usage in 1 year will not necessarily result in lower service costs in the next year, but HUD may realize cost savings over time if usage is consistently lower. While they have a process for identifying opportunities for efficiencies through the business process analyses, WCF Division officials acknowledged that they have not defined and documented the WCF Division’s own roles and responsibilities with regard to the analyses. WCF Division officials told us they are focused on other priorities, such as new business line proposals. However, they told us that they are open to defining these roles in the future. There are additional reasons why the WCF Division has not defined and documented roles and responsibilities for these activities. For example, Division officials told us that the Division faces organizational challenges which may limit its own ability to monitor and implement actions. First, as previously discussed, the business line offices are responsible for managing and overseeing the service lines. According to WCF Division officials, the business line offices are primarily responsible for identifying opportunities to achieve efficiencies with service usage, such as through conducting business process analyses. As such, WCF Division officials told us that they can support those offices by monitoring their usage and helping to identify actions to reduce high service volume and costs, but it is ultimately the business line offices’ responsibility to implement any changes to their own processes to improve service usage. In addition, given its location within OCFO, Division officials stated that the WCF Division has more leverage with OCFO to work with those officials to identify business process improvements. WCF Division officials told us that they have not collaborated with OCHCO or made recommendations for actions OCHCO could take to promote efficient and effective usage of the service lines it oversees. While the Division hopes to work with OCHCO to perform the same types of analyses, the WCF Division Director told us that making recommendations to OCHCO would be viewed as outside of its area of authority. According to Division officials, it is the role of the WCF Committee to provide oversight over the business line offices and ensure that such actions are implemented. However, officials acknowledged that these roles and responsibilities related to the business process analyses should be more clearly delineated in the WCF Handbook. Key operating principles for effective management of WCFs state that agencies should clearly delineate roles and responsibilities by defining key areas of authority and responsibility. In addition, federal standards for internal control state that management should establish an organizational structure, assign responsibility, and delegate authority to achieve the agency’s objectives. As part of this, management should develop an organizational structure with an understanding of the overall responsibilities and assign those responsibilities to discrete units to enable the organization to operate in an efficient and effective manner. Without clearly defining and documenting these roles and responsibilities, it is unclear who is responsible for identifying, monitoring, and implementing actions through the business process analyses to address inefficiencies with service usage across HUD. As a result, opportunities to more efficiently and effectively deliver goods and services may not be fully and consistently implemented across the department. HUD established eight total performance metrics which, according to WCF Division officials, are intended to align with one or more of the WCF’s three goals (see table 2). In fiscal year 2018, the WCF Division developed a draft performance scorecard to measure and track WCF performance in areas such as data and analysis, financial management, and stakeholder engagement. Division officials told us that they use 2019 data as their performance baseline for the scorecard and will continue to review and further develop the fund’s metrics and targets. Part of one of the WCF’s goals is to support the efficient delivery of goods and services. Some of the WCF’s metrics, such as those targeting timeliness, will help the WCF Division improve its efficiency with respect to managing the fund. For example, usage report timeliness measures the number of weeks it takes for the WCF Division to share usage reports with customers. As previously discussed, the WCF Division conducts other activities, such as its business process analyses, that are also intended to support efficient service delivery. However, HUD does not assess the results of the WCF Division’s business process analyses to better understand how they contribute to the WCF’s goal. We previously reported that high- performing agencies continuously assess their efforts to improve performance. As part of this, agencies use fact-based understandings of how their activities contribute to accomplishing the mission and broader results. The WCF Division Director told us they have considered metrics to assess broader results of WCF Division activities such as efficiencies, but noted that it is difficult to quantify cost savings attributable to the WCF. This is due, in part, to the fact that HUD’s service agreements are firm- fixed price contracts, meaning that a change in the volume of services HUD consumes in a given year will not result in direct cost savings that same year. However, HUD could assess the results of the WCF Division’s business process analyses, which identified measurable operational and cost efficiencies that HUD could achieve through implementing the division’s recommendations. For example, as previously discussed, in its analysis of help desk calls, the WCF Division identified potential efficiencies that it could track that would contribute to cost savings over time. While some of the recommendations may not directly result in cost savings, the Division identified other efficiencies such as process improvements that could improve the quality of services that it is capable of tracking. For example, the WCF Division determined that changes to HUD’s processes could improve the accuracy of purchase order accrual estimates. Assessing the results of the WCF Division’s business process analyses would help HUD better understand how the Division’s efforts contribute to its goal of supporting the efficient delivery of goods and services. Without doing so, HUD risks not fully realizing more than $1 million in total potential annual savings identified by the WCF Division’s analyses and freeing up resources that could be realigned for other departmental priorities. In addition to tracking progress towards its own goal, assessing these results would allow HUD to demonstrate how the WCF Division contributes to a 2018 cross-agency priority goal of improving the use, quality, and availability of administrative shared services, as well as the department’s related strategic objective to organize and deliver services more efficiently. In response to our review, HUD updated the WCF Handbook—the primary reference guide for customers and stakeholders on WCF operations—to include more current and complete information on WCF policies and procedures. For example, prior to February 2020, we found that the Handbook was not reconciled with more recently developed draft WCF procedures for contract and budget execution, and invoicing and payments. The WCF Handbook now includes these procedures, which contain detailed information about administrative and funds control responsibilities. For example, the procedures describe the WCF Division Director’s cash management responsibilities and designation as the WCF’s Funds Control Officer, as well as roles of WCF customer program and budget officers. In addition, during the course of our review, the WCF Division updated its Handbook to include current information on other policies and procedures. For example, the Handbook now reflects the WCF’s performance metrics, which we previously discussed were initially developed by the WCF Division in 2018, and changes to other key policies, such as the implementation of the WCF’s full cost recovery model in 2019. HUD now has reasonable assurance that its primary reference guide, the WCF Handbook, provides a current and complete understanding of existing WCF policies, consistent with federal standards for internal controls. The WCF’s price and cost allocation methodology is designed to equitably and transparently recover HUD’s annual costs for externally provided shared services financed through the fund. According to HUD officials, the WCF has roughly recovered its costs of financing HUD’s annual shared service agreements since its establishment in 2016. To recover its costs, the WCF Division has a process to divide HUD’s total cost of shared services among the 17 customer offices based on their estimated service usage. For fiscal years 2016 through 2018, the WCF reported a negative accumulated operating result of $400,372, meaning that it reported it recovered nearly all of its costs since its inception. During this time period, the WCF reported years of positive and negative net operating results. Revolving funds such as the HUD WCF are designed to break even over the long term; therefore, year-to-year fluctuations are to be expected. Table 3 provides a detailed breakdown of HUD’s reported cost recovery. According to WCF Division officials, its shared service providers set annual prices for each service line at the outset of the fiscal year using their own pricing methodologies. The service providers then bill HUD in aggregate for an agreed-upon price under annual interagency agreements at firm-fixed prices. As illustrated in figure 2, the WCF Division determines how much each customer office will pay into the WCF for its respective share of HUD’s total shared service costs using internally developed cost drivers and customers’ expected service usage. The cost drivers were selected by the WCF Committee, and are subject to annual review. According to WCF Division officials, the cost drivers are generally similar to those established by the external providers to maintain a clear connection between customer usage and provider charges. In some cases, however, the provider uses a nonunit based cost driver, such as “level of effort.” In those instances the WCF Division uses cost drivers which vary from the providers. According to HUD documentation, “employee count” is a common alternative driver used to fairly and equitably distribute costs among customers. In addition to the direct costs of HUD’s shared services, HUD officials told us that the WCF received authority to collect reimbursement from HUD customers for the WCF Division’s overhead costs in fiscal year 2019. The WCF’s overhead covers operational expenses, including: WCF Division staff salaries and benefits, travel, support contracts, supplies and materials, and training. Customers are billed for a percentage of the overhead based on their share of HUD’s total shared service costs. This charge is included as an individual line item in customers’ WCF billing statements. The WCF Division shares information on pricing and its cost allocation process with customers in several ways. The WCF Handbook includes the billing process, which describes the method for allocating costs among customers. The WCF Division provides customer offices with a billing model which illustrates how costs are allocated across customers by service line. Customer invoices are broken out to show how customers are charged for each service. In addition, the WCF Division provides quarterly usage reports to customers to help them understand their service consumption. According to WCF Division officials, the WCF Division holds meetings and meets with customer offices one-on-one to explain the information provided. Participants in two of our three focus groups said that the WCF cost allocation model increases accountability and is a more equitable and fair distribution of service costs. Participants in all three focus groups said the WCF improved transparency over the old service model because they can see and consider the costs of their shared service usage. For example, one participant told us that, before the WCF, customers did not directly pay for their shared services and, as a result, did not think about costs. The WCF Division has processes to estimate and manage the WCF’s unexpended balance, including establishing an operating reserve requirement. Properly managing unexpended balances is essential for ensuring self-sufficiency of the fund. Part of the unobligated balance includes an operating reserve which, according to WCF Division officials, is needed to finance ongoing revolving activities, facilitate payments, cover discrepancies between actual and projected shared service costs, and ensure continuity in case of funding disruptions. Evaluating Unexpended Balances: A Framework for Understanding In 2013, we identified the following questions for agencies and decision makers to consider when evaluating unexpended balances in federal budget accounts. Findings based on these questions can provide managers with important information about financial challenges and opportunities which may exist; in turn, this information may help guide more effective account and program management. In fiscal year 2017, the size of the WCF’s unexpended balance increased by 60 percent from $10 million to $16 million, and it was relatively stable from fiscal years 2017 to 2018, as shown in table 4. While the WCF Division does not actually provide the shared services that it finances, nor manage dispute resolution between customers and service providers, it does communicate with customers on fund-related issues—such as shared service billing and usage reports. Key operating principles for effective management of working capital funds state that to be flexible to customer input and needs, agencies should communicate with customers regularly and in a timely manner, and develop a process to assess whether customer demands are met. The WCF Division communicates and interacts directly with customers through a variety of channels. For example, WCF Division officials told us that they: organize quarterly WCF Committee meetings, hold meetings to provide information and answer questions about interpreting usage reports, and use an email inbox for communication between Division staff and customers. The WCF Division will also contact customers directly when issues, such as anomalies in shared service usage, are identified. Customers in all three focus groups reported that they turn to the WCF Division when issues or questions about WCF-related issues arise, and are generally satisfied with the Division’s communication and responsiveness. HUD’s business line offices—those offices that oversee HUD’s agreements for externally provided shared services—have mechanisms to communicate with customers and obtain feedback on shared service quality. For example, an official from OCFO—the office that oversees financial management, procurement, and travel services—told us that OCFO has an email inbox dedicated to questions and concerns regarding services. Officials from OCHCO—which oversees human resource (HR)- related services—told us that OCHCO holds recurring meetings with customer offices and reviews feedback from government-wide employee surveys. That feedback is then used to inform HUD’s annual negotiations for HR-related shared services and improve service delivery. According to the WCF Division Director, the WCF Committee quarterly meetings provide an additional opportunity for customer offices to provide feedback to business line offices on shared services. Business line office officials also told us they monitor data on service provider performance and go directly to the provider when discrepancies between the provider’s actual performance and agreed-upon performance metrics are identified. However, while participants in all three of our focus groups acknowledged that OCFO and OCHCO are the designated points of contact for day-to- day issues, participants in two of our three focus groups mentioned that they have not been given opportunities to provide feedback on overall shared service quality. In addition, all three customer focus groups expressed some level of dissatisfaction with the quality of HR services, particularly with hiring. For example, participants in at least one of our focus groups identified the following issues with HR services: complications and excessive time consumption associated with resolving inquiries; HR service providers operating without specialized skills and knowledge relevant to HUD offices and programs; and inadequate adaption to spikes in service demand. The WCF Division Director told us that the WCF Committee has not conducted periodic reviews of WCF business lines since HUD transitioned to shared services. According to the WCF Committee Charter, the WCF Committee is responsible for conducting and overseeing periodic reviews of WCF business lines, as appropriate, to ensure effective management, strong performance, and customer satisfaction. In addition, federal standards for internal control call for periodic reviews of policies, procedures, and related control activities for continued relevance and effectiveness in achieving the entity’s objectives or addressing related risks. According to the WCF Division Director, at this time the committee does not have plans to conduct such reviews. OCHCO officials told us that they are aware of customer complaints with the quality of HR services. According to officials, HUD has a new Chief Human Capital Officer as of May of 2019 who is taking action to obtain feedback on services by engaging directly with HUD customers through listening sessions. OCHCO officials told us they will introduce action plans in fiscal year 2020 to address recurring issues and customer complaints. In addition to these plans and the feedback OCHCO already obtains, OCHCO officials acknowledged that periodic reviews of the service line, as called for in the committee charter, would be valuable. Without conducting periodic reviews of shared services, HUD may not have a comprehensive understanding of whether customer needs are being met and could be missing out on opportunities to identify potential areas for improvement with the performance and management of services for which it is paying. Given the concerns customers in our focus groups told us about HR service lines, HUD should consider making it the first service line that is subject to a review. WCFs provide agencies with an opportunity to operate more efficiently by consolidating services and creating incentives for customers to exercise cost control. HUD could maximize the potential of these opportunities by ensuring that it has a solid framework in place for managing the WCF before it expands to include additional shared services. During the course of our review, HUD took important steps to ensure that the WCF Handbook—the primary reference guide for WCF operations— includes up-to-date and complete information on WCF policies and procedures. Providing access to current and complete information on the management of the WCF promotes an understanding of who should be held accountable, and helps ensure that funds are effectively managed. HUD also took steps to fully document its processes to effectively manage the operating reserves. This will be particularly important as HUD continues to consider expanding the services provided through the WCF. By documenting its existing operating reserve policies, HUD is better positioned to address potential risk and to identify opportunities to achieve budgetary savings or redirect resources to other priorities. However, there are additional opportunities for improvement. Defining roles and responsibilities promotes a clear understanding of who will be held accountable for specific tasks or duties. Most of HUD’s WCF roles and responsibilities are defined in guidance. However, while the WCF Division performs important business process analyses that identity opportunities to improve the efficiency of services, consistent with the goals of the WCF, HUD has not defined roles and responsibilities for the business process analyses, including who is responsible for identifying, monitoring, and implementing actions to achieve the efficiencies. This makes it difficult to hold offices accountable. By clearly defining the responsibilities of the WCF Division, business line offices, and other stakeholders, such as the WCF Committee, HUD could better ensure the business process improvements are being implemented fully and consistently across the department. Moreover, assessing the results of the WCF Division’s business process analyses would help HUD better understand how the Division’s efforts contribute to its goal of supporting the efficient delivery of goods and services. This would better position HUD to achieve the more than $1 million in potential annual savings identified by the WCF Division’s analyses. Finally, opportunities for customers to provide input about services in a timely manner enables agencies to regularly assess whether customer needs are being met. WCF customers have several ways that they can communicate day-to-day concerns about shared services to the business line offices. However, they raised larger concerns during our focus groups, particularly about the quality of the externally provided human resource related services that deserve attention. Periodic assessments of WCF business lines would provide a more comprehensive understanding of customers’ overall satisfaction and would help HUD identify potential areas for improvement with the services for which they pay. We are making a total of three recommendations to HUD. The Secretary of HUD should define and document roles and responsibilities for identifying opportunities to promote more efficient shared service usage through business process analyses, including defining roles for monitoring and implementing actions recommended because of these analyses. (Recommendation 1) The Secretary of HUD, in conjunction with OCFO, should ensure that the results of the WCF Division’s business process analyses are assessed to better understand how these analyses contribute to the WCF’s established goal to support the efficient delivery of enterprise goods and services. (Recommendation 2) The Secretary of HUD should ensure that the WCF Committee conducts periodic reviews of WCF business lines, as authorized in the WCF Committee Charter, to ensure effective management, strong performance, and customer satisfaction. (Recommendation 3) We provided a draft of this report for comment to the Departments of Agriculture (USDA), Housing and Urban Development (HUD), and the Treasury. In our draft report, we made five recommendations to HUD. HUD provided written comments, which are reproduced in appendix II. HUD officials agreed with four of the recommendations and described some steps they have taken or plan to take to address them. HUD sought additional clarification on one of the recommendations. One draft recommendation was that HUD ensure that existing WCF policies and procedures are current and complete, consolidated in the WCF Handbook, and made easily accessible to customers and stakeholders. HUD officials agreed with this recommendation, and during their review of the draft report, they provided documentation to show that they had updated the WCF Handbook in line with our draft recommendation. Another draft recommendation was that HUD fully document all existing processes related to the management of the WCF’s unexpended balances and operating reserve. HUD officials also agreed with this recommendation, and provided documentation to show that they had established written processes in line with our draft recommendation. As such, we revised our final report to include both actions taken by HUD in February 2020 and to remove these two recommendations. In its written comments, HUD sought clarification on recommendation 1. On February 26, 2020, we spoke with HUD officials and clarified that the recommendation is more specifically targeted to the roles and responsibilities for identifying, monitoring, and implementing actions related to the business process analysis and efficiency efforts than the general guidance that HUD identified in its written comments. We added additional clarification to the report where appropriate. In addition to the written comments we received, USDA, HUD, and Treasury provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees; the Secretaries of USDA, HUD, and Treasury; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have questions about this report, please contact Tranchau (Kris) T. Nguyen at (202) 512-6806 or nguyentt@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In addition to the above contact, Thomas J. McCabe (Assistant Director), Mackenzie D. Verniero (Analyst-in-Charge), Michael Alleyne, Jacqueline Chapin, Andrew J. Howard, Jason Marshall, Steven Putansu, and Alicia White made major contributions to this report. Ronald La Due Lake also contributed to this report.", "summary": "Moving to shared services is one way agencies can operate more efficiently. WCFs provide a way to centralize and simplify the funding of shared services. HUD's WCF was established in 2016 to provide HUD offices services on a cost-reimbursable basis. The fund currently finances services from external federal shared service providers—the Departments of the Treasury (Treasury) and Agriculture (USDA). Congress included a provision for GAO to evaluate HUD's WCF. This report examines the extent to which HUD (1) delineated WCF roles and responsibilities and established performance measures, (2) established a transparent and equitable process to recover WCF costs, and (3) developed processes to obtain WCF customer feedback. GAO analyzed agency documentation of WCF management and financial and budget data, using its work on effective WCF management and unexpended balances as criteria. GAO interviewed HUD, Treasury, and USDA officials and conducted three focus groups with WCF customer offices. The Department of Housing and Urban Development's (HUD) Working Capital Fund (WCF) is a self-sustaining fund that collects fees from HUD customers to pay for services needed across the department. HUD's WCF finances human resource (HR) and financial management related services provided by external federal shared service providers. HUD defines most roles and responsibilities in its WCF handbook—the primary reference guide for WCF operations—and has established performance metrics. In addition, in response to GAO's review, HUD updated its handbook in February 2020 to include more current and complete information on existing WCF policies and procedures. However: HUD has not defined who is responsible for identifying and implementing opportunities for achieving efficiencies with service usage, including roles for the business process analyses it periodically conducts. HUD has not assessed the results of the business process analyses, or how those results could contribute to supporting efficient service delivery. Clearly defining WCF roles and assessing the results of its analyses can help HUD better manage the WCF and improve its ability to identify, monitor, and potentially realize cost savings and other efficiencies. GAO found that HUD has a process designed to equitably and transparently recover the WCF's costs for externally provided federal shared services. Prior to February 2020, it had not fully documented existing policies for managing the WCF's unexpended balances and operating reserves. However, HUD has since established its operating reserve policy that reflects all of the ways that the operating reserve can be used, such as to provide pricing stability to customers and ensure continuity of WCF activities in case of funding disruptions. Written documentation of such policies is essential to ensure that funds are managed appropriately and consistently over time. Finally, the WCF Committee has not conducted periodic reviews of shared services to help ensure effective management, strong performance, and customer satisfaction. Officials from both business line offices—the Office of the Chief Human Capital Officer (OCHCO) and Office of the Chief Financial Officer —stated that they use a variety of mechanisms to obtain customer feedback on services. However, WCF customers in two of three focus groups GAO held said that they have not been given opportunities to provide feedback on the overall quality of services they receive, and some participants shared specific concerns with HR services. Officials from OCHCO—the office that oversees HR services—told GAO they are aware of customer concerns, plan to take additional actions to obtain customer feedback, and acknowledged the need for periodic reviews called for in the WCF Committee Charter. Until such reviews are conducted to regularly assess customer satisfaction, HUD will likely lack a comprehensive understanding of the extent to which customer needs are being met and could be missing out on opportunities to improve the performance and management of services for which it pays. GAO is making three recommendations to HUD on its WCF: define roles for achieving efficiencies; assess results of its analyses; and conduct periodic reviews of business lines. HUD agreed with two and sought additional clarification on one. GAO clarified the recommendation based on further discussion with HUD.", "document_type": "gao"}
{"report": "About 2 million of the more than 6 million veterans who received VHA services in fiscal year 2018 had at least one diagnosed mental health condition, with MDD being the most prevalent diagnosis. About half of these approximate 2 million veterans had a single mental health condition while the remaining half had multiple mental health conditions (see fig. 1). In fiscal year 2018, the three most prevalent mental health conditions among veterans using VHA services were MDD (15 percent), PTSD (12 percent), and GAD (3 percent): MDD. This condition is the most prevalent and disabling form of depression. In addition to the immediate depression symptoms (such as persistently feeling sad or anxious, loss of interest in activities, and difficulty sleeping or oversleeping), MDD can result in poor quality of life overall and decreased productivity, and increased risk of suicide. PTSD. Those with PTSD have experienced symptoms that have persisted for more than 1 month after exposure to a traumatic event, although the onset of symptoms may be delayed for much longer, and cause significant distress or impairment in social, occupational, or other important areas of functioning. Symptoms may include recurrent, involuntary memories of the traumatic event and flashbacks in which the veteran feels or acts as if the traumatic event were recurring. PTSD is strongly associated with reduced quality of life and adverse physical health outcomes. GAD. Those with GAD feel continually worried or anxious about a range of events or activities in their daily lives and have difficulty controlling or stopping this worry. Along with feeling worried, those with GAD experience symptoms of tension such as restlessness, feeling on edge, being easily tired, difficulty concentrating, and sleep difficulties. Veterans with mental health conditions may be offered a variety of treatment options. Of the approximate 2 million veterans with at least one diagnosed mental health condition in fiscal year 2018, 45 percent received non-pharmacologic therapy, 27 percent received a combination of non-pharmacologic therapy and psychotropic medication, and 10 percent received psychotropic medication only. Non-pharmacologic therapy. Non-pharmacologic therapy, or psychotherapy, involves treating mental health conditions using psychological rather than medical means. There are many different types of therapy options, although not all may be available at every VAMC. Examples of non-pharmacologic therapies include cognitive behavioral therapy and prolonged exposure therapy. Some therapy options may be provided to individual veterans, while others are offered to groups of veterans. Psychotropic medications. Psychotropic medications are used to affect one’s mood, thought, or behaviors. Veterans can be prescribed one or more psychotropic medications, from one or more classes, to treat their diagnosed mental health conditions. For example, sertraline—a psychotropic medication commonly known by its brand name Zoloft®—is used by VHA providers to treat both depression and anxiety. Combining treatment. Providers may decide to offer both psychotropic medications and non-pharmacologic therapy, rather than prescribing or offering either option alone. Decisions to offer any of these treatment options are made by providers in various VAMC outpatient care settings. Primary care setting. In addition to addressing other health care needs, PCPs may order non-pharmacologic treatment, prescribe psychotropic medications, or combine both treatment options to address a veteran’s mental health conditions. Through the primary care-mental health integration (PC-MHI) model, which VAMCs began implementing in 2007, PCPs may also collaborate with mental health providers (e.g., psychologists, social workers, nurses) who are collocated within the primary care clinic before making treatment decisions. These collocated mental health providers can also offer non-pharmacologic therapy to veterans without requiring a separate visit outside of primary care. Specialty care setting. Mental health providers in a specialty care setting, such as psychiatrists, decide whether to provide any type of treatment for veterans who have been referred to them by providers in primary care for services specific to their mental health conditions. Veterans may also seek services from a mental health provider in specialty care without first obtaining a referral from primary care. VHA has established certain requirements for providers’ documentation of specialty mental health care treatment plans, and the Joint Commission periodically reviews the documentation of such plans to ensure that they align with the Commission’s standards. VHA. To ensure that providers develop appropriate approaches to treating veterans with mental health conditions and reevaluate such treatment approaches over time, VHA has established certain policies to govern the documentation of mental health treatment decisions by mental health providers in specialty care. For example, in 2008, VHA issued its mental health services handbook to define minimum clinical requirements for mental health services at VAMCs, requiring that providers in specialty care document mental health treatment plans in veterans’ electronic medical records. The mental health services handbook specifies that plans should include certain components such as documentation that different evidence-based treatment options were considered by mental health providers and that approaches to monitor the outcomes of care were developed. The Joint Commission. The Joint Commission is an independent, not-for-profit organization responsible for accrediting and certifying health care organizations and programs in the United States (including VAMCs) at least once every 3 years, and it has developed standards to use as the basis of its evaluative process. These standards focus on specific patient and organization functions that are essential to providing safe and high-quality care, including plans for treatment provided in mental health care settings. VAMC officials we interviewed reported various factors as contributing to providers’ decisions to prescribe psychotropic medications and offer non- pharmacologic therapy to veterans. Specifically, officials from multiple VAMCs cited each of the following factors as contributing to treatment decisions: VAMC resources, complexity of veterans’ mental health conditions, comfort level of providers with treating conditions or prescribing medications, veterans’ preferences, and logistics of receiving mental health treatment. See table 2 for the factors and supporting examples offered by VAMC officials during our site visits. In our review of documentation VAMC providers may use when making treatment decisions, we identified some additional factors. For example, providers’ use of clinical practice guidelines (CPG) established by VA and the Department of Defense may contribute to providers’ treatment decisions. Specifically, the CPG for mental health conditions that are highly prevalent among veterans, including MDD and PTSD, are a resource that all VAMC providers may use when making treatment decisions. For example, the CPG for the management of MDD recommends that providers offer either psychotropic medications or non- pharmacologic therapies (such as behavioral therapy) for the primary treatment of uncomplicated MDD. In contrast, the CPG for the management of PTSD recommends initial treatment for this condition to be a specific type of non-pharmacologic therapy (individual trauma- focused therapy), and when this therapy is not readily available or preferred, then treatments include prescribing psychotropic medications or offering another form of non-pharmacologic therapy. Though it is not mandatory for providers to follow the recommendations of the CPGs, which are based on the strength of evidence and also the potential benefits and harms of treatment options, every provider is responsible for evaluating the appropriateness of applying CPG recommendations in any particular clinical situation. Another factor we identified in our review of documentation was service agreements that VAMCs have in place to help coordinate mental health services across outpatient settings. All five of the VAMCs in our review have formal agreements to help coordinate mental health services across outpatient settings to help manage VAMC resources. These agreements indicate that, for example, providers in primary care can provide treatment for certain mental health conditions, such as uncomplicated depression, without referring veterans to mental health providers in specialty care (see text box). Service Agreements between Primary and Specialty Care for the Treatment of Mental Health Conditions in Selected VA Medical Centers (VAMC) All five of the VAMCs in our review have formal service agreements to help coordinate treatment across primary and specialty care settings for certain mental health conditions, such as uncomplicated depression: All service agreements from the VAMCs in our review indicated that providers in primary care can treat uncomplicated depression without referring veterans to a mental health provider in a specialty care setting. All service agreements indicated at what point mental health providers in specialty care should be involved to help treat veterans with uncomplicated depression—for example, if veterans failed to respond to treatment after trying two different psychotropic medications, or if symptoms worsen over time. In addition to uncomplicated depression, other mental health conditions (including anxiety, PTSD, schizophrenia, and bipolar disorder) were addressed in four of the five service agreements we reviewed. For example, the four service agreements indicated that veterans with bipolar disorder should be treated by mental health providers in specialty care. In light of these factors, providers we interviewed reported on the extent to which each of the most prevalent mental health conditions resulted in PCPs prescribing medication to veterans prior to or without being referred to specialty care. Specifically, more providers reported that psychotropic medications are commonly prescribed to veterans with MDD, PTSD, or GAD prior to referring them to specialty care compared to providers who reported that it is common to prescribe without referring veterans to specialty care at all. See figure 2 for the percentages of providers reporting that psychotropic medications are commonly prescribed to veterans with these three conditions prior to referring them to specialty care. Providers also reported on the extent to which it was common for any provider to offer non-pharmacologic therapy to veterans with these three conditions in lieu of, or in addition to, prescribing psychotropic medications. More providers reported that non-pharmacologic therapy is commonly offered in addition to psychotropic medications compared to providers who reported that it is common to offer therapy instead of medication. See figure 3 for the percentages of providers reporting that non-pharmacologic therapy is commonly offered in addition to psychotropic medications. See appendix I for additional information about mental health treatment practices, including the prescribing of psychotropic medications and offering non-pharmacologic therapy to veterans in a random, nongeneralizable selection of medical records from the VAMCs in our review. See appendix II for information on the use of psychotropic medications or non-pharmacologic therapy by VHA providers to treat veterans with certain mental health conditions, nationally, in fiscal year 2018. VHA has not developed and disseminated guidance that specifies its expectation that mental health providers in specialty care document treatment plans in an easily identifiable way within veterans’ medical records. According to VHA officials responsible for overseeing mental health services, mental health providers should be documenting treatment plans in notes that are easily identifiable and separate from other health information, rather than embedding the plans in progress notes where they may combined with other information related to veterans’ medical histories and current health conditions. In our nongeneralizable review of 80 medical records for veterans who were seen by providers in specialty care and prescribed a psychotropic medication, we found that a majority (50) had a mental health treatment plan recorded in a progress note. We viewed several examples where the treatment plan was not the only information recorded within the progress note, making it difficult to readily identify the mental health treatment plan itself. A VHA official responsible for overseeing mental health services told us it is important for a mental health provider in specialty care to document each veteran’s treatment plan in such a manner so that the provider, or any other providers who may become involved in the veteran’s treatment, can readily refer to the plan as they evaluate progress. This may be particularly important during transitions between inpatient and outpatient care settings, or when adding providers to a veteran’s care team. Providers need to be able to readily access veterans’ mental health treatment plans to ensure that treatment is being provided as ordered, understand why certain treatments were decided against, and assess whether treatment changes are needed. The same VHA official told us that he encourages this practice to support VAMC compliance with the Joint Commission’s standards for mental health treatment plans. However, relevant VHA guidance documents for mental health providers do not specify this expectation: VHA mental health services handbook. The VHA mental health services handbook, published in 2008, requires that mental health providers in specialty care document treatment plans that include certain components. However, it does not specify where providers should document such plans within veterans’ medical records. VHA memo. A 2012 VHA memo promotes the use of a software program by mental health providers in specialty care that, according to VHA officials, facilitates the documentation of treatment plans in notes that are easily identifiable and separate from other information. However, the memo did not specifically state that documenting treatment plans in easily identifiable and separate locations from other information is the goal of using the software program, nor does the memo require providers to use the software. VHA health records handbook. This handbook, published in 2015, provides basic health information procedures for managing veterans’ health records and specifies that all outpatient providers must include treatment plans in progress notes. It does not explicitly reflect VHA’s expectation for mental health providers in specialty care to document mental health treatment plans in an easily identifiable way. Further, the health records handbook specifies that progress notes must also include other types of information, including the history of the veteran’s medical problem, the provider’s assessment of the problem, any tests or consults ordered, and instructions given to the veteran. VHA officials did not provide a rationale as to why they have not developed guidance that clearly directs mental health providers in specialty care to document treatment plans in an easily identifiable way within veterans’ medical records. They noted that VHA has relied upon the VAMCs to develop local processes for documenting specialty mental health treatment plans in an easily identifiable way when providers decide not to use the software program that VHA promoted in its 2012 memo. According to VHA officials, VHA is developing a new memo to communicate its expectation that mental health providers in specialty care document treatment plans in an easily identifiable way within veterans’ medical records. However, as of March 2019, VHA officials had not finalized this memo or indicated when the memo will be disseminated. Standards for internal control in the federal government require that agencies document responsibilities through policies and define objectives in terms that are understood at all levels. These standards also require that agencies communicate necessary information throughout all agency reporting lines to achieve the agencies’ objectives. Absent VHA guidance that clearly identifies its expectation for documenting specialty mental health treatment plans, providers may incorrectly record treatment plans in veteran’s electronic medical records such that they are not easily identifiable. As a result, there is a risk that a provider may be unable to readily access important information about a veteran’s mental health treatment, including the use of psychotropic medication or non- pharmacologic therapy, during changes in a veteran’s care. VHA may learn of the extent of this risk through efforts to collect information resulting from the Joint Commission’s accreditation survey process. Specifically, VHA uses various conference calls to discuss the Joint Commission accreditation survey process and results: According to a VHA official, VHA has weekly and quarterly conference calls with VISNs to, in part, help them prepare their VAMCs for future surveys and, as a result, VHA may learn about different types of citations that apply to multiple VAMCs. This, in turn, may allow VHA to identify concerns that may need to be addressed system-wide, including those related to mental health treatment planning. The Joint Commission provides VHA with an annual summary of data on common citations issued to VAMCs. According to the Joint Commission officials, the Commission provides VHA with this information through a conference call, which may also include a discussion of the underlying causes for any trends in system-wide citations. VHA officials may be able to use this information to address any systemic problems related to the documentation of specialty mental health treatment plans in an easily identifiable way within veterans’ medical records. According to a VHA official, VHA has not identified the documentation of specialty mental health treatment plans as an area for improvement across VAMCs. This issue was not included in the November and December 2018 conference calls with the VISNs, nor was it included in the 2018 annual summary of data that the Joint Commission provided to VHA. VHA has not developed or implemented an approach for monitoring whether mental health providers in specialty care are documenting their consideration of different evidence-based treatment options in mental health treatment plans as required by VHA’s mental health services handbook. In our review of a nongeneralizable sample of 80 medical records for veterans who were seen by such providers and prescribed a psychotropic medication, we found that none of the veterans had treatment plans that documented consideration of different evidence- based treatment options for the veterans’ mental health conditions. VHA relies on the Joint Commission to assess mental health treatment plans as part of the organization’s accreditation process for each VAMC, according to VHA officials. However, VHA does not obtain information resulting from the Joint Commission’s accreditation process that specifically relates to whether mental health providers are documenting consideration of different treatment options in their mental health treatment plans as required. The Joint Commission’s accreditation standards related to mental health treatment plans align with some, but not all, of VHA’s mental health services handbook’s required treatment plan components. For example, the standards align with VHA’s requirement that mental health providers in specialty care must document how they plan to track outcomes and re-evaluate treatment when needed. However, they do not call for the Joint Commission’s accreditation survey to assess whether specialty mental health treatment plans include providers’ consideration of different treatment options, and, according to organization officials, this is not something they look for when conducting their reviews. VHA’s mental health services handbook calls for monitoring through the use of metrics to ensure implementation of the handbook’s requirements, including those related to the documentation of the mental health treatment plan components by mental health providers in specialty care. Additionally, standards for internal control in the federal government require that agencies establish appropriate performance measures for defined objectives, perform ongoing monitoring activities, and remediate identified deficiencies on a timely basis. VHA’s lack of monitoring may contribute to inadequate documentation of the treatment options considered by mental health providers in specialty care in accordance with the mental health services handbook’s requirements. As a result, VHA cannot ensure that mental health providers in specialty care are appropriately considering all available evidence-based treatment options to provide the best care for veterans. This monitoring may be accomplished, for example, by establishing metrics and monitoring performance against such metrics, as called for by VHA’s mental health services handbook. Without metrics or other approaches to monitoring, VHA officials may not be identifying and addressing any systemic problems related to consideration of different evidence-based treatment options. VHA has reported improvement in the safe and effective prescribing of certain psychotropic medications used to treat veterans with mental health conditions since the 2013 start of its Psychotropic Drug Safety Initiative (PDSI). To date, PDSI has consisted of three phases, with each phase focusing on different classes or types of psychotropic medications, age groups, or mental health conditions and substance use disorders. PDSI is currently in phase 3 and VHA is in the process of planning for a new phase 4, scheduled to begin in July 2019. For each phase, VHA developed a set of performance metrics from which each VAMC was required to select a designated number as a focus for implementing prescribing-related quality improvement efforts (referred to as the VAMC’s priority metrics). See table 3. VHA reported improvements in the majority of the performance metrics from the past PDSI phases. Specifically, VHA reported nationwide improvements in 16 of the 20 metrics that it developed for phase 1, and all 14 of the metrics that it developed for phase 2. For example, upon the completion of phase 1, VHA found that there was a nationwide 5.4 percentage point decrease (indicating improvement on this metric) in the percentage of veterans with PTSD who received one or more outpatient prescriptions for a benzodiazepine (a type of antianxiety medication). VHA reported that the change in benzodiazepine prescribing, among other improvements in treating veterans with PTSD, was particularly noteworthy given that the number of veterans diagnosed with this mental health condition increased over the duration of phase 1. Further, upon the completion of phase 2, VHA found that there was a nationwide 1.7 percentage point decrease (indicating improvement on this metric) in the percentage of veterans 75 or older with an outpatient prescription for a benzodiazepine or sedative hypnotic medication. During each PDSI phase, VHA works with VISNs and VAMCs to support their quality improvement efforts related to their priority metrics. For example, VHA provides feedback and technical assistance to VISNs and VAMCs for developing and implementing quality improvement strategies for their priority metrics, which must be updated and submitted to VHA semiannually; convenes a bi-monthly PDSI conference call for VISN and VAMC staff and providers involved in PDSI, which serves as a forum for providing training to participants, discussing best practices, and facilitating collaboration among VAMCs that may have chosen the same priority metrics; develops a semi-annual feedback report for each VISN that includes, among other content, the most recent quarterly score on the priority performance metrics for each VAMC within the network, according to a VHA official; and provides VISNs and VAMCs access to a PDSI clinical management dashboard to use to identify veterans who may benefit from changes to their psychotropic medication prescriptions. These lists can be filtered by the care setting in which the patient is seen, such as the primary or specialty care settings. Although VISNs and VAMCs are not always required (but are encouraged) to continue quality improvement efforts related to VAMCs’ priority metrics from past PDSI phases, VHA continues to monitor VAMC performance on all metrics from each PDSI phase. Specifically, a VHA official told us that VHA monitors performance by calculating quarterly VAMC scores on all performance metrics, which are published on the PDSI clinical management dashboard. VHA also disseminates these scores to the VISNs in the semiannual feedback reports. In these feedback reports, VHA highlights any metric—from the current or a past phase—for which a VAMC within that VISN has regressed. A VHA official stated that if a VAMC regresses significantly in any area, VHA would work with that medical center to determine the cause and take action to reverse the trend as needed. See appendix III for information on PDSI’s planned focus on reducing the co-prescribing of benzodiazepines and opioids as well as the initiative’s collaboration with VHA’s Academic Detailing program, which has developed its own campaign related to stimulant prescribing. Since 2012, VHA has included psychotropic medications in multiple efforts to examine suicide risk among veterans, including two programs and three research studies. These efforts include: Recovery Engagement and Coordination for Health – Veterans Enhanced Treatment (REACH VET) Program. VHA includes psychotropic medications as part of its effort to examine veterans who may be at risk of suicide through its REACH VET program. Specifically, REACH VET uses prior research findings to conduct predictive modeling on data collected from VHA’s electronic medical records to identify veterans who are within the top tier (0.1 percent) of predicted suicide risk. These veterans may also be at increased risk of other adverse outcomes, such as overdoses, violence, and mental health hospitalization. Of note, five of the 61 variables used in REACH VET’s predictive model relate to the prescription of specific psychotropic medications (e.g., alprazolam), and three relate to the prescription of specific psychotropic classes (e.g., antidepressants). Other variables used in the model include demographic characteristics, past suicide attempts, measures of VHA care utilization, and certain diagnoses such as substance use disorder, MDD, and chronic pain. REACH VET coordinators staffed at VAMCs are responsible for notifying the appropriate mental health provider or PCP that a veteran has been identified as being at high risk for suicide, based on a high-risk list of veterans generated monthly by REACH VET’s predictive model. As shown in Figure 4, veterans identified as being at high risk for suicide may then receive targeted outreach from their mental health providers or PCPs if those providers conclude that outreach is warranted based on their review of the veterans’ medical records, according to VHA officials. This outreach may result in changes to the veteran’s treatment as agreed upon by the provider and veteran. VHA reported that within the first year of nationwide implementation, February 2017 through February 2018, the program identified close to 30,000 veterans at high risk for suicide. Behavioral Health Autopsy Program. VHA also includes psychotropic medications in its Behavioral Health Autopsy Program. This program examines information about veteran deaths by suicide that are reported to VAMC providers and suicide prevention coordinators. When informed that a veteran has died by suicide, suicide prevention coordinators are to electronically report, among other things, whether the veteran had (1) been prescribed psychotropic and other medications, for the treatment of a mental health condition within the previous year, and (2) adhered to the medications. Other sources of information collected through the program may include coroners’ and medical examiners’ reports, death certificates, and information provided by family members and significant others. Data are reported to and analyzed by VHA’s VISN 2 Center of Excellence for Suicide Prevention. One recommendation in the program’s 2017 annual report called for more efforts to study issues related to medication management, such as veterans’ medication adherence, overmedication, and frequent and abrupt medication changes. In the past, recommendations from the program have been used to inform VHA suicide prevention policies, programs, and educational efforts, according to VHA officials. For example, officials shared that the program informed the development of a tool kit for providers to use to help address veterans’ sleep issues after analyses found that sleep patterns were often altered for veterans prior to their death by suicide. Lithium for Suicidal Behaviors in Mood Disorders study. As of March 2019, VHA is in the process of conducting a randomized clinical trial that examines the effect of a specific psychotropic medication (lithium) on reducing suicide risk for veterans with MDD or bipolar disorder who either survived a recent suicide attempt or were hospitalized to prevent one. VHA plans to enroll 1,600 veterans in the study from 28 VAMCs and provide them with the appropriate treatment options as determined by their respective providers, as well as provide some additional care coordination. Additionally, half of the participants will receive lithium, and half of the participants will receive a placebo. The study’s investigators told us that, to their knowledge, this study is the first effort to test lithium’s efficacy for reducing suicide risk in a randomized clinical trial setting. Investigators also told us that, because all participants will receive medications already proven safe and effective for the treatment of their conditions, it is not considered unethical to withhold lithium, a yet untested medication for treating suicide risk, from half of participants. VHA investigators told us they hope to use the results of this clinical trial to inform future treatment options for patients with MDD or bipolar disorder and who are at risk of suicide. Drugs and Suicide Risk study. Between January 2017 and October 2018, VHA officials and collaborators at the University of Chicago and Columbia University analyzed 513 medications, which included psychotropic medications, prescribed between 2003 and 2014 for association with increased or decreased risk of suicidal events in VHA patients. According to the study’s investigators, they expect to be able to identify specific psychotropic medications that are found to be associated with the largest increases and decreases in suicide risk. VHA officials told us that as of March 2019, the research manuscript was under review for publication in a peer-reviewed journal. Using Big Data and Precision Medicine to Assess and Manage Suicide Risk in U.S. Veterans study. As of March 2019, VHA officials, in collaboration with the Department of Energy, were in the process of developing a new model to predict suicidal behavior among veterans by combining data on genetic and non-genetic risk factors, such as demographics, medical conditions, and stressful life events; psychotropic medications are also included as a risk factor, according to VHA officials. The researchers are expected to combine data from VHA electronic medical records with data from a VHA Office of Research and Development program that collects genetic information from veterans to develop the new algorithm. VHA officials we interviewed noted some broad challenges not exclusive to VHA that may affect efforts for any researcher in examining suicide risk and the use of psychotropic medications: Multiple risk factors. All VHA officials that we spoke with discussed the need for efforts examining psychotropic medications and suicide risk to account for other suicide risk factors beyond the use of these types of medication. Such factors may include having a substance use disorder or other mental health diagnoses; homelessness; chronic (non-mental health) medical conditions; age; and psychosocial factors, such as recent loss of a significant other or a history of abuse or violence. Methodological considerations. Most VHA officials that we spoke with mentioned some methodological considerations that must be considered when designing a research study to examine this relationship. For example, an official told us that measuring veterans’ medication adherence is important to track, but is difficult to do as VHA generally only has data on whether medications were dispensed to veterans, not whether medications were actually taken. Ethics. Some VHA officials that we spoke with noted that certain ethical considerations may limit the methodological options available to researchers, such as randomized clinical trials. For example, it would be unethical to withhold medications that have been proven as safe and effective from veterans who may clinically benefit from receiving such treatments, such as from veterans in a control group. In the face of these challenges, VHA officials also noted some advantages VHA researchers, in particular, may experience in examining the use of psychotropic medications and suicide risk: a large patient population with more than 2 million veterans who have at least one mental health condition, and a corresponding electronic medical records database, providing sufficient data and sample sizes needed to test hypotheses; internal funding streams dedicated to research activities examining issues related to suicide prevention, such as funds available through three of VA’s Office of Research and Development’s four central research services; and research centers with researchers who have specific expertise about issues related to suicide prevention and the treatment of serious mental health conditions. Veterans diagnosed with mental health conditions rely on providers in VAMCs across the country to make treatment decisions that are safe and effective, including whether to treat highly prevalent and serious conditions such as MDD and PTSD with psychotropic medications, non- pharmacologic therapy, or a combination of both. In recent years, VHA has taken steps aimed at improving the safety and effectiveness of prescribing decisions for certain psychotropic medications and noted important improvements resulting from these efforts. However, VHA’s oversight related to treatment planning needs improvement. VHA has yet to disseminate guidance that clearly reflects its expectation that mental health providers in specialty care document mental health treatment plans in a readily identifiable manner in veterans’ medical records. Additionally, VHA does not monitor whether mental health providers are considering evidence-based treatment options in treatment plans, as VHA requires in its mental health services handbook. As a result, VHA cannot ensure that providers are considering and documenting all appropriate treatment options, adequately evaluating patient care, and making treatment modifications as necessary, among other issues. Furthermore, the lack of monitoring may impede VHA’s ability to identify important factors that contribute to providers’ treatment decisions, which could in turn allow VHA to identify more systemic barriers to safe and effective treatment, such as needed training. In addition, being able to readily identify how veterans are being treated for mental health conditions may allow VHA to enhance its research efforts related to suicide risk. VHA has noted several advantages it has in conducting research involving the role of psychotropic medications in suicide risk among veterans, including that VHA researchers have access to a large patient population with at least one mental health condition. Monitoring veterans’ use of psychotropic medications and non- pharmacologic therapies and related outcomes may further enhance this capacity for research on suicide risk. We are making the following two recommendations to VA: The Veterans Health Administration should disseminate guidance for VISNs and VAMCs that more clearly reflects its expectation that mental health providers in specialty care should record mental health treatment plans within veterans’ medical records in an easily identifiable way. (Recommendation 1) The Veterans Health Administration should develop and implement an approach for monitoring treatment plans for veterans with mental health conditions to ensure that such plans include documentation that different evidence-based treatment options were considered. (Recommendation 2) We provided a draft of this report to VA for review and comment. In its written comments, which are reproduced in Appendix IV, VA concurred with our recommendations. VA agreed that the recommendations would promote adherence to mental health treatment planning requirements. VA stated that it is developing guidance to help ensure that mental health providers in specialty care record mental health treatment plans in separate, easily identifiable documents within veterans’ medical records. VA also stated that it will develop and implement a process for monitoring whether such plans include documentation that providers considered different evidence-based treatment options. We will monitor VA’s efforts to address our recommendations. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committee and the Secretary of Veterans Affairs. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at DraperD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. We reviewed a nongeneralizable, randomly selected sample of 75 veterans’ medical records from five Department of Veterans Affairs (VA) medical centers (VAMC)—25 for each of the three most prevalent conditions diagnosed among veterans—who had at least one primary care visit in fiscal year 2017 and were prescribed a new psychotropic medication. About half (37) of the 75 medical records we reviewed indicated the veteran was prescribed a new psychotropic medication prior to or without being referred to a mental health provider in specialty care. See table 4. We reviewed a separate, nongeneralizable, randomly selected sample of an additional 75 veterans’ medical records from the five VAMCs—25 for each of the three most prevalent conditions diagnosed among veterans— who were newly diagnosed within that year. Over half (44) of the 75 medical records we reviewed indicated whether the veteran was offered non-pharmacologic therapy in lieu of or in addition to being prescribed a psychotropic medication. See table 5. We analyzed national data obtained from VHA on the types of treatments received by veterans with a diagnosis of a single mental health condition who had encounters with VHA providers for that diagnosis in fiscal year 2018, including the three most prevalent mental health conditions diagnosed among veterans. See Figure 5 for the percentages of veterans with these three conditions or another mental health condition who received (1) non-pharmacologic therapy (psychotherapy), (2) at least one medication from a psychotropic medication class, (3) a combination of the two, or (4) neither psychotropic medication nor non-pharmacologic therapy in fiscal year 2018. We also analyzed national encounter data obtained from VHA for veterans with one of the three most prevalent mental health conditions and who received psychotropic medications in a VA medical center (VAMC) in fiscal year 2018. We found that for all three conditions, the largest percentage of veterans who received at least one psychotropic medication from one class were seen in the primary care setting only. The percentages of veterans with medications from two or three classes— typically veterans who had more complex mental health conditions, according to a VHA official—were larger for veterans seen by specialty care providers, compared to the percentages of veterans with medications from multiple classes seen in primary care only. See Figure 6. The Veterans Health Administration (VHA) has taken steps to improve the safe and effective prescribing of certain psychotropic medications used to treat veterans with mental health conditions through the Psychotropic Drug Safety Initiative (PDSI). PDSI has consisted of three phases since 2013, when the initiative began. Each phase has focused on making improvements related to the prescribing of different classes or types of psychotropic medications, or treating different age groups or mental health conditions and substance use disorders. PDSI is currently in phase 3, and VHA is in the process of planning for phase 4. According to a VHA official, PDSI phase 4 (expected to begin in July 2019) will, in part, increase the role of mental health providers in the monitoring and management of the co-prescribing of benzodiazepines (a type of antianxiety medication) and opioids. This includes tapering the use of these medications among this high-risk veteran population to a reduced dose or discontinuation entirely when the harms associated with their concurrent use outweigh the benefits. The same official told us that, to date, VHA has primarily focused on monitoring the concurrent use of these medications—which the Department of Veterans Affairs’ and the Department of Defense’s clinical practice guideline (CPG) for opioid therapy strongly recommends against—through the Opioid Safety Initiative and in the primary care setting (see text box). Veterans Health Administration’s (VHA) Efforts to Taper Veterans Co-Prescribed Benzodiazepines and Opioids Efforts focused on the establishment of safe and effective tapering programs in the primary care setting: VHA launched the Opioid Safety Initiative in 2013 to ensure that veterans are prescribed and use opioid pain medications in a safe and effective manner. This initiative seeks to establish safe and effective tapering programs for veterans who are co-prescribed opioids and benzodiazepines, among other goals. A VHA official told us that the initiative primarily focuses on monitoring and managing the concurrent use of these medications in the primary care setting. Efforts conducted on an individualized, gradual basis: The Department of Veterans Affairs’ and Department of Defense’s clinical practice guideline (CPG) for opioid therapy strongly recommends that tapering of opioids be done on an individualized basis, weighing the benefits and risks to each veteran as well as the veteran’s characteristics and needs. The CPG also notes that the sudden stopping of benzodiazepines should be avoided, since doing so can lead to seizures or death. Department of Veterans Affairs and Department of Defense, Clinical Practice Guideline for Opioid Therapy. To help achieve PDSI’s goal of improving the prescribing of certain psychotropic medications, VHA officials leading PDSI collaborate with VHA’s Academic Detailing program. Academic detailers, who are Veterans Integrated Service Networks (VISN) or Department of Veterans Affairs (VA) medical center (VAMC) clinical pharmacy specialists, disseminate resources and provide one-on-one educational outreach to providers to help them improve their psychotropic medication prescribing practices. Pharmacy staff, including staff involved in academic detailing, from four VISNs told us that they provide education related to PDSI. The Academic Detailing program has also implemented a campaign to improve the appropriate prescribing and monitoring of stimulants (see text box). Veterans Health Administration’s (VHA) Academic Detailing Program Prescription Stimulants Campaign According to a VHA official, in February 2018, the Academic Detailing program implemented a campaign to improve the treatment of patients receiving prescription stimulant therapy for adult attention-deficit / hyperactivity disorder. A VHA official told us that Veterans Integrated Service Networks (VISNs) or Department of Veterans Affairs (VA) medical centers (VAMCs) may choose, but are not required, to participate in this campaign. The stimulant campaign dashboard includes VAMC scores on 13 quality indicators related to (1) prescribing stimulants for off-label use, (2) assessing co-morbidities, (3) monitoring patients, and (4) managing medication. For example One quality indicator measures the percentage of veterans co-prescribed a stimulant and a benzodiazepine, and Another quality indicator measures the percentage of veterans co-prescribed a stimulant and an opioid. Academic detailers (VISN or VAMC clinical pharmacy specialists) may use the dashboard to identify providers who may benefit from changes to their stimulant prescribing practices. A VHA official reported that between February 2018 and the end of fiscal year 2018, academic detailers made 37 staff interactions with providers related to the national academic detailing program’s stimulant campaign. As of October 2018 (the most recent data available), 37,223 veterans with at least one diagnosed mental health condition had an active prescription for at least one stimulant, according to a VHA official. Among these veterans, 2,360 had a co-occurring cardiac condition.Sudden death, stroke, or other cardiac events have been reported with stimulants. The U.S. Food and Drug Administration has stated that stimulants should generally not be used in patients with serious heart problems or for whom an increase in blood pressure or heart rate would be problematic. See U.S. Food and Drug Administration, FDA Drug Safety Review Communication: Safety Review Update of Medications Used to Treat Attention-Deficit / Hyperactivity Disorder (ADHD) in Adults, accessed March 1, 2019, https://www.fda.gov/Drugs/DrugSafety/ucm279858.htm. In addition to the contact named above, Hernán Bozzolo (Assistant Director), Kaitlin Asaly (Analyst-in-Charge), Jennie F. Apter, Karen Belli, Topher Hoffmann, and Rebecca Rust Williamson made key contributions to this report. Also contributing were Rich Lipinski, Diona Martyn, Vikki Porter, and Jennifer Whitworth.", "summary": "In fiscal year 2018, of the roughly 6 million veterans who received services from VHA, approximately 2 million had a diagnosis for at least one mental health condition. Treatments for such mental health conditions can include psychotropic medications or non-pharmacologic therapies, which can be prescribed or offered by VA providers in outpatient settings including primary and specialty care. GAO was asked to review how mental health treatment decisions are made by providers in VAMCs and monitored by VHA. This report examines, among other things, (1) factors that contribute to providers' treatment decisions for veterans with mental health conditions, (2) VHA's guidance for documenting mental health treatment plans, (3) VHA's monitoring of whether providers document their consideration of different treatment options, and (4) VHA's efforts to improve the treatment of veterans prescribed psychotropic medications. GAO reviewed VHA documents and a nongeneralizable sample of veterans' medical records from five VAMCs (selected for variety in facility complexity and location); analyzed data on psychotropic medication prescribing; and interviewed VHA and VAMC officials. Officials from the five selected Department of Veterans Affairs (VA) medical centers (VAMC) GAO spoke with reported various factors that contribute to providers' mental health treatment decisions, including decisions regarding the prescribing of psychotropic medications and the offering of non-pharmacologic therapy. Examples of reported factors include: VAMC resources, such as the availability of appointments with mental health providers in specialty care, and the complexity of veterans' mental health conditions, such as the veterans' diagnoses and treatment history. Officials with VA's Veterans Health Administration (VHA) told GAO that specialty mental health care providers are expected to document mental health treatment plans in an easily identifiable way in veterans' medical records, but VHA has not developed guidance explicitly addressing this expectation. For example, VHA's mental health services handbook requires that treatment plans include certain components, but does not specify where to document the plan within a veteran's medical record. As a result, there is a risk that a provider may be unable to readily access information about a veteran's mental health treatment, including the use of medication or therapy, during changes in a veteran's care. VHA has not monitored whether mental health providers in specialty care document the required consideration of different treatment options—such as psychotropic medications or non-pharmacologic therapy—within mental health treatment plans. VHA officials told GAO that VHA relies on the Joint Commission (an independent, not-for-profit organization that accredits and certifies health care organizations) to assess specialty mental health treatment plans as part of the organization's accreditation process for each VAMC. However, the Joint Commission's standards do not specifically assess whether providers consider different treatment options. As a result, VHA cannot ensure that providers are considering all available treatment options and providing the most appropriate treatments to each veteran. VHA has taken steps to improve veterans' mental health treatment through the Psychotropic Drug Safety Initiative (PDSI)—an initiative focused on the safe and effective prescribing of certain psychotropic medications. For example, the first phase included a performance metric aimed at decreasing the percentage of veterans with post-traumatic stress disorder receiving one or more outpatient prescriptions for a benzodiazepine (a medication used to treat anxiety) because of risks associated with the medication. VHA reported a nationwide 5.4 percentage point decrease in the prescribing of this medication for these patients, as well as improvements in the majority of the initiative's other performance metrics. VHA should (1) disseminate guidance reflecting its expectation that providers document mental health treatment plans in an easily identifiable way, and (2) implement an approach for monitoring whether these treatment plans include consideration of treatment options. VHA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "The FMS program is intended to strengthen the security of the United States and partner countries. To accomplish this mission, DOD sells a variety of types of items and services to foreign partners. These sales can range from fighter jets and integrated air and missile defense systems to combat helmets and training on the use of items. (See figure 1.) According to DOD and State officials, FMS provides multiple benefits to foreign governments and the U.S. government. Foreign governments that choose to use FMS rather than direct commercial sales receive greater assurances of a reliable product, benefit from DOD’s economies of scale, improve interoperability with the U.S. military, and build a stronger relationship with the U.S. government. From the U.S. perspective, FMS expands the market for U.S. businesses and contributes to foreign policy and national security objectives. While State reviews and approves FMS purchases, DOD is responsible for program implementation. The responsibilities of DOD components vary: DSCA: DSCA is responsible for administering the FMS program for DOD, including overseeing the FMS transportation accounts’ operations and balances. DSCA also sets policies for the FMS process, including for how FMS-purchased items can be transported and how DOD will calculate the fees purchasers will pay to reimburse DOD for any costs of transporting the items. DFAS: DFAS provides DSCA’s accounting services for FMS and is responsible for accounting, billing, disbursing, and collecting funds for the FMS program. Military departments: The Departments of the Air Force, Army, and Navy are the primary DOD agencies that coordinate with purchasers to prepare and execute FMS agreements, including planning transportation, if necessary. U.S. Transportation Command (TRANSCOM): TRANSCOM supports transportation planned by the military departments to be conducted through the Defense Transportation System (DTS), which consists of military and commercial resources. Although FMS shipments may receive transportation support through TRANSCOM headquarters, the primary TRANSCOM components providing FMS transportation are the Military Surface Deployment and Distribution Command, which provides defense transportation by sea, rail, or highway, and the Air Mobility Command, which provides defense transportation by air. Contracts between TRANSCOM and private transportation service companies can provide additional commercial resources through DTS. DFAS processes bills to reimburse the TRANSCOM components and private transportation service companies for the costs of performing these transportation services. Foreign partners who purchase items and services through the FMS program may use their own funds or, if provided, U.S. funds, such as grants or loans provided through Foreign Military Financing. In addition, some FMS purchases are made using funds appropriated to DOD, State, or other U.S. government agencies for Building Partner Capacity (BPC) programs. These programs purchase items or services for foreign partners through FMS. Foreign partners and BPC programs have different options available to them for transporting items they purchase through FMS. With the exception of certain hazardous or sensitive items that must be transported via DTS, foreign partners have the option to arrange for their own transportation of FMS items they purchase, such as using a freight forwarder, for all or part of the transportation needed to reach the final destination. On the other hand, BPC programs use DTS to move all their FMS purchases. There are two ways DOD calculates the fees it charges FMS purchasers to use DTS that lead to collections into the FMS transportation accounts. Percentage of price. DOD most commonly calculates the FMS transportation fee using a percentage rate that is applied to the price of the item. The percentage rate varies depending on the extent of the U.S. government’s responsibility for transporting the items purchased, such as whether the U.S. government will transport the items to their final destination or to an intermediate destination. As seen in table 1, since fiscal year 2007, DSCA changed the rates in fiscal years 2009 and 2018. Over the full period, the transportation fee has been as high as 22.25 percent of purchase price, or as low as 2.75 percent, depending on where purchasers want to take custody of their items. Price per item. DOD may instead charge the FMS purchaser an estimated transportation price per item for certain types of items, such as those containing sensitive or hazardous materials. Eight transportation accounts within the FMS trust fund are used to hold transportation fees collected from FMS purchasers and to pay FMS transportation bills. In aggregate, we refer to these as the combined FMS transportation accounts: Main account. One main account holds transportation funds for all foreign partner purchasers and smaller BPC programs. BPC accounts. Seven segregated accounts hold transportation funds for certain larger BPC programs, such as the Afghan Security Forces Fund and the Iraq Security Forces Fund. DSCA created the first four BPC accounts in fiscal year 2012, one in fiscal year 2015, and two more in fiscal year 2018. Individual shipments trigger collections into and expenditures from the FMS transportation accounts. As shown in figure 2, after DOD ships an item and DFAS is notified of that shipment, DFAS moves the amount of the related transportation fee from the country account or BPC program account into the related transportation account and records the amount as a collection. Once DFAS collects funds into a FMS transportation account, funds are generally no longer segregated or tracked by their originating country or BPC program account. DFAS receives monthly bills from TRANSCOM that include the costs for FMS transportation, which DFAS pays out of the main transportation account, recording the amount paid as an expenditure. For FMS shipments associated with the seven larger BPC programs, the main account is then reimbursed from the appropriate BPC transportation account. Although aggregate FMS transportation fees are expected to approximate costs over time, we found that the combined FMS transportation account balance grew by over 1,300 percent from fiscal years 2007 to 2018. The ending balance for fiscal year 2018 was $680 million. Collections and expenditures for the account fluctuated from year to year, but collections have outpaced expenditures since 2014, particularly for the main transportation account, which has grown more quickly than the combined seven BPC accounts. The combined balance of the eight FMS transportation accounts grew substantially from the beginning of fiscal year 2007 through the end of fiscal year 2018—from $46 million to $680 million, or by 1,378 percent. As shown in figure 3, much of that growth occurred from the end of fiscal year 2011 through fiscal year 2018, during which time the account grew by approximately $630 million. This substantial recent balance growth was in contrast to balance activity from fiscal years 2007 to 2011, when the collections into the account more closely approximated the expenditures from the account. In fact, the FMS transportation account was at risk of insolvency starting in fiscal year 2009. In response, DSCA redistributed $80 million in fiscal year 2009 and $50 million in fiscal year 2011 from the FMS administrative fee account to the main FMS transportation account to ensure it contained sufficient funding to pay transportation bills. If not for the redistributions between accounts, the transportation account may have been unable to disburse payments from the account, for at least some parts of fiscal years 2009, 2010, and 2011. Collections and expenditures both fluctuated from year to year, as shown in figure 4. Year-to-year changes in collections ranged from decreases of 54 percent to increases of 121 percent, while year-to-year changes in expenditures ranged from decreases of 52 percent to increases of 133 percent. According to DSCA officials, demand for transportation of FMS purchases through DTS is unpredictable, and the accounts’ balances may experience volatile swings due to inconsistencies involved in billing the accounts. For example, delays in billing or reporting a particular shipment can result in DOD collecting the fee into the transportation accounts and reimbursing the transportation cost from the accounts at different times. Further, the fees collected and the costs expended for an individual shipment may differ because DOD uses different factors to calculate the transportation fee to charge the purchaser (e.g., the item’s value) than it uses to calculate the cost to bill the FMS transportation accounts (e.g., the shipment’s origin, destination, and weight, among other factors). Despite this volatility over time, from fiscal years 2014 to 2018, collections consistently exceeded expenditures, which drove the substantial balance growth. In figure 4, we show this relationship in a collections-to- expenditures ratio, for which a value of 1.0 would indicate collections equaled expenditures for the fiscal year. A ratio greater than 1.0 indicates an increasing account balance that fiscal year. The average collections- to-expenditures ratio for fiscal years 2007 to 2018 was 1.26; from fiscal year 2014 to 2018, this ratio ranged from 1.46 to 4.97. At the end of each fiscal year, any collections that exceed expenditures remain in the account and are carried over to the next fiscal year’s beginning balance, which contributes to balance growth from year to year. Much of the recent combined balance growth has been driven by growth in the main account’s balance, as shown in figure 5. The main account grew more quickly than the combined balance of the BPC accounts from fiscal year 2013–the first full year of operation for the BPC accounts–to fiscal year 2018. The main account grew by 316 percent, from $140 million at the beginning of fiscal year 2013 to $582 million at the end of fiscal year 2018, while the combined BPC accounts grew by 88 percent, from $52 million to $98 million, during the same time period. As seen in figure 6, our analysis shows that, for fiscal years 2013 to 2018, collections exceeded expenditures more frequently and by a greater extent in the main account than in the BPC accounts, which has driven balance growth. On average during this period, collections exceeded expenditures for the main account by $74 million per year, as compared to $7 million per year for the BPC accounts. DSCA officials speculated that BPC programs may use more air transportation for shipments to areas without regular TRANSCOM shipment routes, which may result in higher expenditures. DSCA officials could not provide any further explanation for why the main account’s balance has grown more quickly than the balances of the BPC accounts. DSCA has limited management oversight guidance for the FMS transportation accounts, which has contributed to their substantial balance growth. DSCA has established internal guidance for its two main management oversight processes to monitor for significant changes in the FMS transportation account balance—a daily review and annual review— but this guidance is unclear and lacks key details. As a result, DSCA’s implementation of these processes lacks rigor and DSCA’s reporting to its management has not included complete information about the causes for recent balance growth. In addition, DSCA has no internal guidance to ensure that funds remaining in BPC-specific transportation accounts after the related programs close are transferred to the miscellaneous receipts of the Treasury, which risks these funds not being transferred as DOD officials told us DOD intends to do. In fiscal year 2016, DSCA established a Managers’ Internal Control Program (MICP) for overseeing the FMS transportation accounts, according to DSCA officials. These procedures formalized two management oversight processes for the FMS transportation accounts that DSCA officials had performed previously: daily and annual reviews. These reviews both serve the purpose of ensuring the accounts have sufficient funds to pay expenses. MICP documentation to help guide these processes includes flow charts that explain certain steps that should be included in each of these reviews, a risk assessment that explains how each of the MICP processes mitigates risks for the FMS transportation accounts, and test procedures that lay out expectations for how each MICP process should be conducted so that DSCA can periodically test to ensure the processes were carried out as intended. Daily review. MICP procedures indicate that DSCA staff should review a report from DFAS daily that includes the previous day’s balances for each of the transportation accounts to ensure that the FMS transportation accounts do not drop below a “healthy level.” If DSCA staff identify a large decrease or “significant” level of change in the accounts, the procedures direct them to ask DFAS to explain what caused the change and to take corrective action, such as to ask for billing corrections, if necessary. According to the MICP risk assessment, the FMS transportation accounts experience volatile swings due to inconsistencies involved in billing the account, and reviewing the account balances on a daily basis helps to address this risk. The MICP procedures state that, if DSCA allows the FMS transportation accounts to drop below this “healthy level,” the accounts could become insolvent and be delinquent in disbursing transportation expenses. Annual review. MICP procedures indicate that DSCA should annually assess the financial health of the transportation accounts, which DSCA staff have stated they implement by preparing an annual report for DSCA leadership. To test whether the annual review has occurred, certain DSCA staff are to examine the annual report to confirm that DSCA assessed the FMS transportation account with the purpose of ensuring that the overall financial health of the accounts is strong and collections are sufficient to pay expenditures. DSCA has inconsistently implemented its daily reviews due to unclear internal guidance on these reviews. Specifically, the guidance does not specify the level of change that warrants further examination or what DSCA staff should consider as a healthy level, or target range, for the accounts. DSCA’s daily review procedures are meant to monitor for significant changes in the FMS transportation accounts so that such changes can be further examined and, if needed, corrected; however, MICP internal guidance does not establish criteria for determining what constitutes a significant change in these accounts’ balances. According to federal internal control standards, management should define the acceptable level of variation in performance, or risk tolerance, in specific and measurable terms. However, the MICP procedures use different and undefined terms when referencing the types of balance changes DSCA should look for in their daily review procedure. These terms include: “change,” “significant change,” and “significant reduction.” Although some of these terms could be interpreted as DSCA needing to monitor for any significant changes—whether increases or decreases in the accounts— DSCA staff have chosen to focus these reviews on decreases. As a result of DSCA’s unclear internal guidance, DSCA staff have inconsistently determined which changes warrant examination and should trigger them to contact DFAS to examine the reasons for the change. This makes it less likely that DSCA will be alerted to and take corrective action to address significant changes in the account balances. From fiscal year 2018, DFAS was able to provide one documented instance of DSCA staff contacting DFAS as a result of the daily review. The contact was regarding an 11 percent balance decrease of approximately $6 million in the Afghan Security Forces Fund’s transportation account that occurred on July 5, 2018. However, we identified a total of 30 instances of balance changes greater than 11 percent (12 decreases and 18 increases) in fiscal year 2018 across the eight FMS transportation accounts. For example, figure 7 shows the fiscal year 2018 daily balance changes for the Afghan Security Forces Fund. This figure includes the July 2018 balance decrease that resulted in DSCA contacting DFAS to examine the change, as well as nine other instances of balance changes greater than 11 percent that did not result in any documented contact between DSCA and DFAS. By inconsistently conducting daily reviews, DSCA weakens the effectiveness of this oversight mechanism to identify potential errors, which risks allowing either insufficient or excessive funds in the accounts. In particular, in recent years, the lack of clarity on what these reviews should monitor for has weakened DSCA’s oversight and contributed to the substantial balance growth. DSCA has not defined what it considers an acceptable target range for these accounts despite the unpredictability of transportation account balances and the MICP daily review procedures requiring DSCA officials to monitor account balances to ensure they remain at or above a “healthy level.” According to DSCA officials, DSCA has not determined an acceptable target range for the transportation accounts because future collections and expenditures are difficult to predict, making it difficult to know how much money DSCA needs in the accounts. However, this unpredictability makes it all the more important for DSCA officials to establish a target range for what is “healthy” account activity to enhance their oversight of the accounts. As we previously reported, to ensure the accountability of fee-funded programs and the ability to manage a program with sufficient reserves, federal agencies are advised to use a risk-based strategy to establish desired upper and lower bounds for account balances. DSCA has already established upper and lower bounds for two other FMS overhead fee accounts, the FMS administrative fee and contract administration services fee accounts. DSCA calculates these bounds based on the amounts of planned expenses from the accounts, which automatically adjusts the bounds over time to reflect the size and needs of the FMS program. DSCA’s internal guidance states that setting upper and lower bounds of acceptable levels provides the agency with a “control box” to alert it to a dramatic change in the FMS operating environment that may require an agency response such as a fee rate review. Similarly, establishing a target range, with an upper and lower bound, for the FMS transportation account balances could strengthen DSCA’s ability to use its daily reviews to manage the accounts’ volatility by identifying when the account balances are growing excessively high or falling excessively low. Such an upper bound could better inform DSCA leadership and help prevent excessive growth in the transportation accounts while a lower bound could help to ensure that the accounts have sufficient funds to pay for transportation bills. DSCA has no internal guidance for its staff to follow when preparing annual reports on the health of the FMS transportation accounts, which has led the reports DSCA produced for fiscal years 2015 to 2018 to contain incomplete information on the underlying causes for the trends in the accounts and for the reports to lack key details about the source of some of the funds in the main FMS transportation account. For fiscal years 2015 to 2018, DSCA produced annual reports assessing the financial health of FMS transportation accounts that contained incomplete information because DSCA did not use rigorous methods to determine the underlying causes for trends in the accounts. As a result, DSCA had a limited ability to make informed decisions about the accounts at a time when the balances were experiencing substantial growth. According to the DSCA staff who produce the annual reports, they distribute the reports within DSCA up to the agency’s Director to provide information about the health of the FMS transportation accounts. DSCA’s annual reports on the FMS transportation accounts for fiscal years 2015 to 2018 followed a consistent format. These reports contained information on the net change in balances for each of the transportation accounts during the fiscal year. The reports also included a summary of any major activity in each of the accounts. For example, the fiscal year 2018 assessment stated that the main FMS transportation account grew by $77.8 million during that fiscal year due to several large collections significantly greater than billings. All of the reports end with a conclusion regarding the health of the accounts, which for fiscal years 2015 to 2018, was that the accounts were healthy and should remain financially solvent. All of these annual reports also include statements regarding the underlying causes of account trends, which we found to be incomplete and unsupported by rigorous data analysis. When discussing reasons for year-to-year account balance increases, DSCA’s reports stated they were mainly due to a decline in oil prices and a legal change that DOD implemented in July 2014 that allowed TRANSCOM to charge lower DOD rates for FMS air shipments, both of which could likely affect expenditures from the account. However, DSCA officials said that they conducted no specific analysis to support the extent to which these two factors affected the account balance increases. As seen in figure 8, our analysis shows that these reasons could not fully explain the account balance increases in each of the annual reports from fiscal year 2015 to 2018. In particular, while FMS transportation expenditures began to decrease in fiscal year 2012, the price of oil did not begin to significantly decline and the legal change did not come into effect until 2014. Further, the annual reports did not discuss underlying reasons for trends in collection activity, which also affect the account balance. DSCA’s analysis for its annual reports is limited by the lack of internal guidance for completing these reports. Specifically, the MICP guidance for the annual review process does not specify how to prepare the annual report. Without such guidance, according to DSCA officials, DSCA’s analysis for the annual reports has involved re-reviewing the documentation related to the daily reviews as well as monthly reviews that DSCA performs for financial oversight purposes. DSCA officials completed no additional analysis to inform the annual reports, such as any quantitative analysis to understand annual changes or trends over time. Federal internal control standards state that effective internal guidance communicates the who, what, when, where, and why of what needs to be accomplished, and that management should obtain relevant data from reliable sources and process that data into quality information to aid decision making. Without clear internal guidance, the annual account reviews lack the rigor necessary to ensure DSCA management is provided reliable information for decision making. According to DSCA officials, DSCA’s annual review process should also involve an assessment of whether funds should be redistributed between the FMS overhead fee accounts; however, DSCA does not have specific internal guidance on when and how to perform such assessments or on what to include about this portion of the annual review in its resulting annual reports. This lack of guidance has led DSCA to produce annual reports without information related to redistributed funds and to not conduct assessments related to redistributed funds. According to DOD’s financial management regulations, DSCA and DFAS should periodically review activity in the FMS overhead fee accounts to serve as a basis for decisions by DSCA management to, among other purposes, redistribute account balances between these accounts. According to DSCA officials, if they were to perform these periodic assessments, they would perform them as part of their annual account reviews. However, the MICP guidance for the annual reviews does not describe how to assess whether or how much to redistribute funds between the fee accounts, or how or when to assess returning previously redistributed funds. The annual FMS transportation account and administrative account assessments for fiscal years 2015 to 2018 do not report that $130 million in the main FMS transportation account came from funds redistributed from the FMS administrative account between fiscal years 2009 and 2011 that have not been returned. According to DSCA officials, they only report redistributions in the year that they occur. In addition to not including this information in its annual reports, DSCA has not assessed the need for other redistributions of funds between the FMS fee accounts since it last redistributed funds from the FMS administrative account to the main FMS transportation account in fiscal year 2011. DSCA officials indicated they intend to return the funds to the administrative account but have not done so because they have no urgency, given that the FMS administrative account balance has been consistently above its lower bound in recent years. As of the end of fiscal year 2018, the FMS administrative account balance was approximately $4.7 billion, which was approximately $3.1 billion more than the account’s lower bound that DSCA determined was necessary to support FMS operations. The lack of specific internal guidance on how to assess and report redistributions has resulted in incomplete reports to DSCA management, which inhibits DSCA management’s ability to make informed decisions in overseeing the FMS fees. In particular, without reports that clearly state the amount of redistributed funds and their source(s), and assess their continued need, DSCA management is less informed when determining whether and when to redistribute funds, including whether to return previously redistributed funds. According to our User Fee Design Guide, assigning costs to identifiable users can promote equity and more informed rate-setting; however, redistributing fees from the FMS administrative account to the main FMS transportation account has intermingled funds that have different sources. DOD charges the FMS administrative fee to all FMS purchasers while DOD charges the FMS transportation fee to only certain purchasers for the portion of the transportation of their FMS items that uses DTS. Distributing funds from the FMS administrative account to the main FMS transportation account intermingled these fees, which has two main effects. First, not returning redistributed funds if the transportation account no longer needs them raises concerns regarding the fees’ equity in ensuring only the beneficiaries of a service pay for the cost of providing it. Second, the appropriateness of DSCA management’s rate-setting decisions for both fees is limited by incomplete information about the full expected balance of the fee accounts from which future expenditures could be paid. DSCA has no internal guidance to ensure proper disposition of any funds remaining in the BPC-specific transportation accounts after the related programs close and those remaining funds are no longer needed. In fiscal year 2020, DSCA expects the first BPC-specific transportation account to close, which had a balance of approximately $42 million at the end of fiscal year 2018. DSCA officials have said that funds remaining in the BPC-specific transportation accounts after the related programs close should be transferred to the miscellaneous receipts of the U.S. Treasury. According to DSCA officials, this process was agreed to with DOD’s Office of the Under Secretary of Defense (Comptroller) in November 2011 when DSCA met with that office to discuss how DSCA would handle creating the BPC-specific transportation accounts. DSCA officials also said that following this process would be in line with a requirement in DOD’s financial management regulations for any collections that are authorized or required to be credited to an account after that account’s closure to be deposited in the Treasury as miscellaneous receipts. However, DOD officials could not provide a documented agreement from the November 2011 meeting, and we do not consider the referenced regulation specific enough to this circumstance to alone serve as internal guidance that would ensure the funds are transferred. In particular, this regulation applies broadly to DOD collections received after an account’s closure, and does not specifically address the disposition of funds that had already been collected into an account upon the closure of that account . Officials from relevant DOD components have different understandings of how this process should occur, which could risk the process not being completed as intended without related specific internal guidance. According to DSCA officials, DFAS will be responsible for moving any remaining funds in these transportation accounts to the miscellaneous receipts of the Treasury, but the pertinent DFAS officials have stated they are unaware of what should be done in such circumstances. According to DSCA officials, they intend to write a memo to DFAS related to each instance of a BPC-specific transportation account closure instead of providing DFAS written guidance to follow in any such instance because DSCA officials prefer providing specific directions to DFAS regarding moving such funds. DSCA officials said they do not need specific internal guidance to ensure they direct DFAS to complete such fund transfers because DOD’s Office of the Under Secretary of Defense (Comptroller) would ensure that DSCA does so when that office reviews all DOD accounts. However, Comptroller’s Office officials stated that, as part of DSCA’s program oversight responsibilities for FMS, DSCA is responsible for ensuring any funds are identified and transferred to the miscellaneous receipts of the Treasury. Without clear internal guidance, DOD may not have accurate information on or sufficient oversight of its budgetary resources and account balances, and funds that could be put to other uses may remain in the BPC transportation accounts. Federal internal control standards state that effective internal guidance communicates the who, what, when, where, and why of what needs to be accomplished, thereby providing a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel. According to DSCA officials, the first BPC-specific transportation account likely to close is dedicated to the Iraq Security Forces Fund, which had a balance of approximately $42 million at the end of fiscal year 2018. According to DSCA records, this program’s appropriations were canceled at the end of fiscal year 2017 and, according to DSCA officials, by sometime in fiscal year 2020, the program’s FMS cases should go through their final reconciliation process. Through this process, DOD may pay outstanding bills or correct accounting errors and the related cases will close. According to DSCA officials, the BPC-specific transportation account would then be ready for closure. DSCA’s processes for setting the FMS transportation fee do not ensure that aggregate fees DOD collects approximate aggregate transportation costs over time, thus contributing to recent growth in the FMS transportation account balances. DSCA’s ability to set appropriate transportation fee rates is undermined by DSCA’s unclear guidance to the military departments on what data they should provide DSCA to analyze in its transportation fee rate reviews, leading DSCA to review data that is not timely or systematically sampled. Further, the lack of clarity in its internal guidance for these reviews has led DSCA to complete these reviews infrequently, perform limited analysis, and burden the military departments with compiling data DSCA did not use. In addition, our analysis raises concerns about negative effects of the current transportation fee rate structure, including that the structure makes it more difficult for DSCA to determine appropriate transportation fee rates. Finally, DSCA’s internal guidance to the military departments for estimating transportation prices, instead of rates, for certain items lacks key specific details. As a result, the military departments follow varying procedures for estimating these prices, and are unsure of the prices’ accuracy. DSCA’s ability to set appropriate transportation fee rates is undermined by unclear guidance for its reviews of these rates. The lack of clear guidance has led the military departments to provide DSCA data that is not suitable for rate-setting decisions because, while the individual data points DSCA analyzed were accurate, they may not accurately predict future rates because they were not timely or systematically sampled. Unclear guidance also led DSCA to perform infrequent and limited analysis of these data. DSCA’s ability to determine the appropriate FMS transportation fee rates is limited by the data analyzed in its rate reviews that are not timely or systematically sampled. According to its MICP documentation, DSCA is to review its FMS transportation fee rates to ensure the resulting transportation fees collected from FMS purchasers in aggregate cover the amount needed to pay for transportation expenses. DSCA requests the military departments provide historical data on transportation fees charged and transportation costs paid so that DSCA can analyze these data to determine appropriate fee rates. However, DSCA’s data requests to the military departments are unclear in multiple key respects, which leads the military departments to provide data to DSCA that—though it contains accurate cost and fee data—are unsuitable to use for DSCA’s resulting rate-setting decisions because it is not timely or systematically sampled. The combined effects of these deficiencies could skew DSCA’s rate review process. When DSCA requests data from the military departments for the rate reviews, DSCA does not specify key elements about which data to provide or which information sources to use to obtain each data element. As a result, the departments have followed different processes and provided data that was not timely. As shown in table 2, the data submitted by the military departments varied significantly. Because DSCA’s data requests did not specify where the data should be sourced, the military departments have had difficulty responding to these requests and the amount of data they have produced has been limited. Military department officials explained difficulties finding the necessary data in other DOD agencies’ systems, understanding those data’s reliability, and accurately matching the data across multiple systems. In particular, transportation cost data is stored in multiple TRANSCOM billing systems, which military department officials responsible for responding to DSCA’s data requests said they do not regularly access. In addition, DFAS has copies of transportation cost data in the monthly bills that it pays from the FMS transportation accounts. The bills include the individual costs of each shipment made during that month, but are stored in individual documents and are not accessible to the military departments. Transportation fee data is available in a DFAS system used to process the FMS transportation fee, but, according to DFAS documents and officials, this system is not built to easily extract such data and therefore neither DFAS nor the military departments can reliably pull fee data from this system specific to particular shipments or cases. According to DSCA officials, ultimately DSCA only used Army data for setting rates in 2018 because Navy and Air Force provided relatively small samples. Navy. Navy officials reported having particular difficulty finding data on transportation costs for the most recent rate review. After unsuccessfully requesting more specific guidance or assistance from DSCA and DFAS, according to Navy officials, Navy found a spreadsheet DSCA had provided Navy for an unrelated purpose that contained the costs for Navy FMS shipments moved by TRANSCOM’s Air Mobility Command. According to Navy officials, because researching the individual transportation fees for each FMS shipment was time-consuming and they lacked clear guidance about how much data DSCA needed for its rate review, they decided to provide related fee data on 103, or 3 percent, of the 3,536 air shipments for which Navy had cost data. Air Force. The U.S. Air Force Security Assistance and Cooperation Directorate has developed a detailed process, described in a 280- page internal guidance document, to respond to DSCA’s requests, but following this process does not yield much data. For the most recent review, Air Force provided DSCA with data for 639, or 2 percent, of 28,886 shipment orders for which they reviewed data because of the difficulty of finding relevant matching cost and fee data across the different systems used, as shown in table 3. Not only were the data DSCA reviewed not indicative of all FMS shipments since they included no Navy or Air Force data, the data were also not indicative of Army’s shipments and included older data because the DSCA data requests were unclear. In particular, DSCA’s data requests stated that each military department should provide at least 20 cost and fee comparisons for each fee rate for each of the FMS transportation accounts, and requested that these data include as many different foreign partners or FMS cases as possible. As a result, according to Army officials, the data Army provided to DSCA included a mix of different partners and cases of different dollar values; however, no systematic sampling methods were used that would have ensured that the resulting data were indicative of overall Army shipments during the time period covered. Also, DSCA’s request did not specify a time period the data should cover. Army provided data for cases that likely were at least 5 to 7 years old. According to DSCA officials, if the rate review is to analyze case-level data, such as Army provided, it is necessary to analyze data on cases for which the FMS agreements were signed multiple years prior, because shipments may not take place until multiple years into cases. However, the Army officials we spoke to about the data Army provided were unaware how long ago the shipments occurred for the related cases, and stated that some may have occurred years before. TRANSCOM pricing changes annually, so cost information that is multiple years old and not adjusted to reflect such changes would be unlikely to predict future costs. As a result, DSCA set rates to cover future costs based on a sample of cases that was not systematically sampled and may have included shipments over the past 5 or more years. DSCA officials stated that their data requests are not more specific because they thought the military departments had direct access to these data and that more specificity would hinder the military departments’ ability to respond to the requests. However, related data are available in TRANSCOM and DFAS, instead of military departments’, systems. Further, the current processes produce data that are not timely or systematically sampled, making it unsuitable to use to determine future costs and rates. In setting user fees, agencies should analyze timely and reliable data, consistent with applicable accounting standards, to avoid the risk of making skewed fee-setting decisions. DSCA’s use of data that are not timely or systematically sampled for its rate reviews could skew its rate-setting decisions, ultimately affecting transportation account balances. DSCA’s internal guidance for its rate reviews is unclear regarding the timing of the reviews and lacks key details, which has limited DSCA’s ability to use the rate review to set appropriate rates. Timing. DSCA’s internal guidance for overseeing the FMS transportation accounts is unclear. In one part the guidance indicates that DSCA should conduct a rate review every 5 years, which is in line with the expectations explained by DSCA officials who oversee these accounts. However, other parts of DSCA’s internal guidance indicate that DSCA should conduct such a review annually. How reviews should be conducted. DSCA’s internal guidance states that the rate reviews should allow DSCA to determine whether current transportation fee rates are sufficient, based on predetermined criteria, to cover the related costs. However, this internal guidance does not specify how these criteria should be determined or contain any procedures regarding how DSCA should analyze the data collected for its rate review. DSCA has not completed its transportation fee rate reviews in a timely manner, which allowed the FMS transportation account balances to grow over recent years as collections consistently exceeded expenditures but fee rates remained constant. Since fiscal year 2007, DSCA has completed two reviews more than 9 years apart: in March 2009 and May 2018. For these reviews, DSCA officials did not predetermine criteria for the level of alignment between cost and fee that each review should achieve and DSCA’s analysis considered few factors and involved a limited analysis of only Army data, which hindered DSCA’s ability to set appropriate fee rates. In particular: Fiscal year 2009: For this review, DSCA compared the transportation cost to the transportation fee charged across seven transportation fee rates for 144 of the thousands of Army’s FMS cases. In this sample, the transportation costs exceeded the fees paid by 19 percent overall. When briefing DSCA management on the review, DSCA officials reported a concentration of undercharges in two of the rates. As a result, DSCA decided to increase these two rates such that, if the new rates had applied to the full sample DSCA analyzed, fees on the cases in the full sample would have exceeded costs by 14 percent. Our analysis of the sample showed that while these two rates had the largest difference in value between the costs and fees, other rates also had large differences within this sample. Specifically, one other rate had a larger percentage of undercharges and three of the other rates had percentages of overcharges exceeding 1,000 percent. However, DSCA made no changes to these other rates. Fiscal year 2018: For this review, DSCA compared the transportation cost to the transportation fee charged across the seven transportation fee rates for a sample that contained data on 993 Army cases. For this sample, on average transportation fees charged to purchasers exceeded transportation costs by 158 percent, with all rates except one overcharging on average. However, when briefing DSCA management on the review, DSCA officials reported incorrect data to serve as the basis for decision making. In particular, according to the DSCA official responsible for the analysis, likely due to an oversight, DSCA included data on only 878 of these cases in the briefing to DSCA management. Total fees for this portion of the sample were 90 percent higher than the related total costs. Based on this limited data, DSCA decided to decrease all of its transportation fee rates such that, if the new rates had applied to the full sample DSCA analyzed, fees would still have exceeded costs by 77 percent, with five of the seven fee rates still exceeding the cost by more than 100 percent for that sample. DSCA officials stated that their intent in this rate review was to lower the rates modestly to see their effect on the account balances; however, their ability to accurately meet this goal is reduced by its lack of specificity and the limited analysis DSCA performed. Given that the data DSCA analyzed for both these reviews was not generalizable to all shipments, the above percentages do not indicate that the rates overall would have affected fees in these exact ways. Instead, DSCA’s decision making may have been further skewed by its method of analysis. In addition to completing these two reviews, DSCA also initiated rate reviews by sending requests to the military departments three additional times for data DSCA did not use, thereby placing an unnecessary burden on the military departments. Specifically, DSCA requested data from the military departments in November 2011, September 2013, and November 2014. After obtaining the data from the military departments, DSCA officials said that management decided DSCA would not analyze the data due to competing priorities, and DSCA did not use these data for any other purpose. Air Force officials said that the months of work put into responding to each of DSCA’s rate review requests seemed like a waste of resources because their data has consistently shown that the transportation fees collected were drastically higher than the related costs and yet the fee remained unchanged for years. To respond to DSCA’s request for data for the fiscal year 2018 rate review, each military department spent between 2 to 4 months of staff time to collect and prepare the data, according to military department officials. Asking for and then not using such data put an unnecessary burden on the military departments and wasted DOD staff resources. Without clearer internal guidance for its rate reviews regarding their timing and the analysis needed, it will be difficult for DSCA management to make appropriate fee-setting decisions based on future rate reviews. Federal internal control standards state that effective internal guidance communicates the who, what, when, where, and why of what needs to be accomplished. According to DSCA officials, DSCA is considering conducting its next transportation fee rate review in fiscal year 2020, with a goal of lowering the FMS transportation account balances. DSCA officials’ ability to meet this goal could be hindered without more clarity about the timing of the reviews and more rigorous analysis that involves explicit goals, such as for the level of alignment between cost and fee or of the account balances. The structure of the FMS transportation fee rate further hinders DSCA’s ability to set appropriate rates. According to DSCA officials, the current rate structure was developed to use data that are easily available, which limits DOD’s administrative burden in calculating the fee. However, our analysis raises concerns about the extent to which the current rate structure may have negative implications for the transportation fee’s equity, efficiency, and revenue adequacy. We have previously reported that fee design should balance ways to encourage greater efficiency, equity, and revenue adequacy while reducing administrative burden on the agency and payers of the fees, as shown in Table 4. These factors interact and often conflict with each other so that tradeoffs among these factors should be considered when designing a fee’s structure. The current transportation fee rate structure limits DSCA’s administrative burden because it relies on only a few factors, which involve easily accessible data, but these factors vary considerably from those TRANSCOM uses to price its transportation. The FMS transportation fee amount charged to purchasers is generally based on three factors, which should be identified in FMS agreements: (1) the price of the item; (2) the foreign destination rate area; and (3) the extent of U.S. government responsibility for transporting the item (e.g., to an inland destination in the continental United States or to a foreign inland or port destination). At the time of the FMS agreement, DSCA and the military departments lack information about other factors that would make it easier for DOD to set fee rates such that fees would approximate the actual cost of the transportation. For example: Mode. DOD may not know how it will move the items at the time of the FMS agreement, and costs vary depending on the mode of transportation, such as by air or a surface vessel. Route. Although DOD should be aware of the final destination for items, DOD may be unaware of where the shipment will originate or the specific route the items will take, and transportation costs can vary depending on the specific route. For example, to transport goods in a 20-foot container on a surface vessel door-to-door from a location on the East Coast of the United States to Afghanistan in fiscal year 2018, TRANSCOM rates ranged from $548.85 to $1,077.03 per measurement ton shipped, depending on the specific route, whereas DSCA’s fee rates would be constant and applied to the price of the items. Also, even if DOD knew the exact mode and route, approximating the exact cost for each shipment would be difficult because TRANSCOM updates its rates annually, and shipments often occur years after signing the FMS agreement. The distinct factors used to determine the fee and cost for FMS transportation make it difficult for the cost and fee to align, which has potential implications for the fee’s equity. Although the data DSCA obtained from the military departments for its fiscal year 2018 rate review was unsuitable for that purpose because it was not timely or statistically sampled, we performed extensive data reliability procedures to determine that the individual cost and fee data points are reliable and as a result analyzed these data to obtain insights into the extent to which the cost and fee were aligned within that sample. As shown in figure 9, we found extreme differences between the transportation cost billed to the FMS transportation accounts and the fee the purchaser paid. Within this nongeneralizable sample, costs and fees were within 10 percent of each other for only 30 of the 1,152 cases or shipments (3 percent), whereas the difference was more than 1,000 percent higher or lower for 492 of the cases or shipments (43 percent). In addition, we identified five instances of the difference between the cost and the fee exceeding 1,000,000 percent. Although these data were not systematically sampled to ensure they would be indicative of the full population of shipments, the high incidence of such large differences is concerning. Within this sample, we also found that certain countries were either always over-charged or always under-charged. Since the rate review data are not generalizable, this pattern may or may not be consistent across FMS shipments. However, such a pattern could plausibly occur due to the differences between TRANSCOM’s and DSCA’s rate areas. Potential concerns about the fee structure’s efficiency and revenue adequacy also stem from the difficulty in aligning the current fee structure with related costs. Efficiency. The large disparities between cost and fee in the current FMS transportation fee rate structure may be leading some FMS purchasers to choose not to use DTS. According to Army officials, some FMS purchasers choose to use their own freight forwarders instead of DTS because of a perception that the FMS transportation fee is too high. These decisions could have broader effects on DTS. According to TRANSCOM, the additional demand from FMS purchasers allows TRANSCOM to better leverage DTS, such as by filling excess capacity with paying cargo and supporting training needs to maintain combat readiness. Revenue adequacy and stability. The potentially large differences between the transportation cost and fee resulting from the current FMS transportation fee rate structure has led to large fluctuations in collections and expenditures over time. For example, in fiscal years 2009 and 2011, DSCA had to redistribute a combined $130 million into the main FMS transportation account from the FMS administrative fee account to cover costs and avoid insolvency. Around the time of the fiscal year 2009 rate review, DSCA began reviewing the fee rate’s structure as part of an overall attempt to address issues related to the transportation account nearing insolvency. As part of that review, DSCA worked with the military departments and TRANSCOM to assess factors such as administrative burden, data availability, and ability to more accurately charge transportation costs to FMS purchasers, which would have enhanced the fee’s equity and efficiency. Specifically, they considered the benefits and costs of six alternative rate structures: Three of the six options would have involved replacing the rate-based fee for some or all shipments, by charging actual transportation costs or estimating likely actual costs per type of item. According to documentation from this review, the DOD agencies said these three options would have placed high administrative burdens on the military departments and required changes to military department or TRANSCOM information systems. The other three options the DOD agencies considered would have modified the structure of the current rate-based fee to take into account additional factors, such as transportation method (e.g., air) and item weight, or creating additional rate areas to target specific locations where costs of transportation were higher. The agencies determined that some of these options would have a lower administrative burden than the first three options. However, DSCA decided to maintain its current fee rate structure and address the potential insolvency through other approaches such as by redistributing funds from the FMS administrative fee account to the transportation account. According to DSCA officials, DSCA made this decision because it could not obtain agreement with the military departments and TRANSCOM on any of the other options. DSCA has not since reviewed the rate structure. DSCA provides internal guidance to the military departments on how to estimate the transportation prices to be charged for certain items, but the internal guidance does not specify key details about how to calculate the estimates. As a result, the military departments follow different procedures for estimating these prices, and individual military department procedures may differ over time depending on staff turnover. Federal internal control standards state that management should use quality financial information that is complete and reasonably free from error, and that effective internal guidance informs users of the who, what, when, where, and why of what needs to be accomplished, thereby helping to retain organizational knowledge. Estimated Transportation Prices for Certain Items For certain items that need to be shipped via the Defense Transportation System, such as goods with sensitive or hazardous materials, and for which charging the transportation fee rate would significantly differ from transportation costs, DOD may instead charge a set transportation price per item. The fees collected from these estimated prices and the costs to transport these items are paid in and out of the FMS transportation accounts. These prices are not location-specific because DOD charges each purchaser of this item the same estimated price. According to DOD officials, such items are often low- weight, high-cost items, such as missiles, for which the usual transportation fee rate could greatly overcharge the FMS purchaser. DSCA’s internal guidance for how to estimate these transportation prices includes limited information and does not take into account key information for accurately estimating transportation costs. Specifically, the guidance lists certain types of transportation cost elements to include and not to include in these price estimates. For example, estimated port handling costs should be included while security costs should be charged to the FMS purchaser separately. The guidance also indicates the estimates should be on a per-item basis with two potential prices to transport each item, one for any transportation within the United States and one for transportation to any foreign destination. Other key factors in transportation costs, such as the transportation mode or specific origin or destination, are not considered. Also, DOD charges these prices per item, although economies of scale can be gained by transporting batches of the same item together. The lack of specificity in DSCA’s internal guidance has led the military departments to adopt inconsistent estimation processes that may not lead prices to approximate actual costs. These inconsistent processes could lead DOD to charge FMS purchasers more or less than DSCA intends and ultimately affect account balances. For example: Origin and destination. The three military departments take different approaches to compensate for having to estimate the cost of transporting an item without knowing its specific origin and destination. Although all military departments follow the same general process of estimating potential transportation costs for commonly used origin and destination ports and averaging these to attempt to estimate these prices, they all use different locations to create their estimates, which leads to different pricing. For example, one command within Army uses a central location within the United States as the origin for its estimates to simulate an average of potential costs for transportation from any continental United States location. However, according to Army officials, another command within Army attempts to ensure that the transportation price estimated will cover costs by simulating a “worst case scenario” by basing its estimates on locations distant from each other. Batch shipments. The military departments also vary in terms of how they estimate per item costs for items that could often be transported in batches. Air Force and Navy calculate how many of an item can fit in a container, and then divide the average price estimated to transport such a container by this batch size to determine final pricing, but Army does not. When Air Force and Navy estimate prices this way, they do not require shipments to be transported in a container of this size or for purchasers to buy or receive these items only in batches of this size, which could lead the price charged to vary greatly from the actual costs. For example, for one type of missile, Air Force determined that 20 of them could fit in a container and therefore divided the average price it had estimated to transport a container by 20 before submitting the price to DSCA. Therefore, if only one of the item were purchased, instead of the 20 built into the estimate, the transportation cost could be about 20 times the fee. The lack of specificity of DSCA’s guidance has also led to large changes in one of the military departments’ estimated prices after staff turnover. According to the Air Force official who prepared Air Force’s 2018 updates to these prices, that was the first year that official estimated these prices after another Air Force official had done so through 2015. The new Air Force official said that Air Force had not updated its prices during the previous 3 years because it lacked rates to estimate the costs of transporting explosive materials by ocean vessel. After receiving guidance from DSCA to exclude these rates from their estimates, the new Air Force official updated the prices for 2018. When doing so, this Air Force official found that some of the updated price estimates were much higher than the prior prices due to increased port handling rates, whereas the prices to transport items to foreign destinations were at times lower due to lower air rates used in the estimates. For example, the price to transport a certain item within the continental U.S. had been set at $278.00 per item for 2015 through 2017, and the 2018 price estimate was $8,447.00. DSCA initially accepted the updated prices, but Air Force later rescinded them after foreign partner countries voiced concerns about the increased prices affecting existing contracts and Air Force was unable to prove that the new estimates better approximated actual costs without the ability to compare actual bills with the price estimates. According to the responsible Air Force official, the calculation process from 2015 was used to recalculate the 2018 prices and was again used for 2019, albeit with current fiscal year rate information, due to continued uncertainty regarding this process. Since late 2016, the military departments have voiced concerns to DSCA regarding the difficulty of following DSCA’s internal guidance to estimate these transportation prices. In particular, in late 2016, Army officials developed a white paper for DSCA that described challenges developing these estimated prices posed by updates to how TRANSCOM calculates its transportation pricing. In September 2018, Air Force officials also raised various concerns regarding the accuracy of the prices, such as concerns about how the batch size of a shipment affects per item costs and the lack of key details affecting transportation costs. Military department officials said they would prefer more specific guidance from DSCA that could help them to more uniformly calculate these prices. In January 2019, DSCA officials stated they were at an early stage of exploring possible changes to the information required to calculate these types of transportation prices. In May 2019, DSCA officials stated that they were still working to define the problem and how it could be addressed. Further research into the military departments’ difficulties in establishing these price estimates and the costs and benefits of the methodologies they use would better inform DSCA on what pricing process could most accurately reflect costs moving forward. FMS is one of the primary ways the U.S. government engages in security cooperation with its foreign partners, by annually selling them billions of dollars in defense items and services. When transporting FMS items on their behalf, DOD charges purchasers a transportation fee such that, according to DOD, it should involve “no profit, no loss”–foreign partners should not be charged excessive fees and fee revenue should cover the program’s operating costs. However, from fiscal year 2007 to 2018, the FMS transportation accounts experienced substantial balance growth of over 1300 percent. To address risks such as the historical unpredictability of collections and expenditures prior to recent dramatic account growth, DSCA implemented processes to conduct daily and annual management oversight of the accounts. However, the effectiveness of these processes is limited by a lack of specific internal guidance. In particular, although the daily reviews are meant to keep DSCA aware of significant changes in the accounts and ensure that they maintain healthy balances, DSCA has not specified what should be considered as significant changes or how to calculate healthy target levels for the accounts. Lack of rigorous annual review processes has also led the annual reports provided to DSCA management to be missing key details. In particular, they have contained incomplete information on the causes for account trends and have omitted information on the source of $130 million that had been redistributed into this fee account from the FMS administrative fee account in fiscal years 2009 to 2011 to address a danger of insolvency that the FMS transportation accounts no longer face. The resulting reports inhibit DSCA management’s ability to oversee the accounts at a time when they have grown so quickly. In addition, a lack of clear internal guidance explaining how to assess when redistributions are needed and when to return unused BPC-specific transportation funds may lead to a surplus of funds in the FMS transportation accounts that could be used for other purposes. Similarly, DSCA has established a process to review FMS transportation fee rates but this process has several weaknesses that may skew DSCA’s rate setting decisions. DSCA’s rate review process involves analysis of historical cost and fee data provided by the military departments, but due to unclear requests to the military departments, the process is burdensome and leads to data that are untimely and unsystematically sampled. Although DSCA requested such data from the military departments five times between fiscal years 2007 to 2018, DSCA only conducted rate reviews using these data twice because DSCA did not prioritize use of its resources for the other reviews. In addition, for the two reviews it did conduct, DSCA never used Air Force or Navy data because unclear guidance from DSCA and difficulties finding sufficient data across disparate DOD information systems limited the data Air Force and Navy could provide. Further, DSCA based their reviews on minimal internal guidance and used limited analysis and unclear criteria upon which to set new rates. The current rate review process and the overall fee rate structure reduce DSCA’s administrative burden, but raise various concerns regarding the fee’s equity, efficiency, and revenue stability. DSCA also has similarly unclear internal guidance for the military departments for situations when the FMS purchaser is charged a set transportation price per item instead of a transportation fee rate. By strengthening these rate setting processes, DSCA would enhance its ability to manage account balances and to make timely decisions to ensure the FMS transportation fee rate is set to cover related transportation costs but not overcharge FMS purchasers. We are making the following 10 recommendations to DOD: The Secretary of Defense should ensure that the Director of DSCA clarify internal guidance for daily account reviews by specifying criteria for the level (such as percentage or dollar amount) of change in transportation account balances that would require DSCA to contact DFAS for further examination. (Recommendation 1) The Secretary of Defense should ensure that the Director of DSCA establish a methodology to calculate a target range, with desired upper and lower bounds, for FMS transportation account balances that could be used to better inform DSCA’s account reviews. (Recommendation 2) The Secretary of Defense should ensure that the Director of DSCA modify the internal guidance for the annual review process to include the specific steps DSCA officials should take in preparing the annual report, including ensuring that they incorporate rigorous analysis into the annual reports. (Recommendation 3) The Secretary of Defense should ensure that the Director of DSCA develop internal guidance related to the redistribution of funds between the FMS trust fund fee accounts. Such internal guidance could include criteria for when to consider redistributing funds between accounts and for when to return those funds, how to analyze the amount of any redistributions needed, and how to clearly report any redistributions to DSCA management. (Recommendation 4) The Secretary of Defense should ensure that the Director of DSCA assess whether funds redistributed from the administrative account to the transportation account should be moved back to the FMS administrative account and document this decision. If the Director of DSCA determines that the funds should be moved back to the FMS administrative account, the Director should ensure the movement of funds in accordance with this decision. (Recommendation 5) The Secretary of Defense should ensure that the Director of DSCA develop internal guidance for the steps that DSCA, in combination with DFAS, should undertake when a BPC-specific transportation account closes to help ensure that any remaining unused funds are transferred to the miscellaneous receipts of the U.S. Treasury in accordance with DOD officials’ stated intention to do so. (Recommendation 6) The Secretary of Defense should ensure that the Director of DSCA create specific internal guidance for how and from where data should be obtained to be used for its transportation fee rate reviews and the timeframes the data should cover to ensure DSCA has a systematic sample upon which to base its rate setting decisions. This updated internal guidance should be based on consultations with the military departments, DFAS, and TRANSCOM on which sources of transportation cost and fee data are the most reliable and comparable for use in its FMS transportation fee rate reviews. (Recommendation 7) The Secretary of Defense should ensure that the Director of DSCA develop specific internal guidance to follow when performing transportation fee rate reviews. Such internal guidance could specify when these reviews should occur; a process to obtain management commitment to complete a review before DSCA requests that the military departments compile data for it; and a process for performing the reviews that includes developing clear, documented goals and an appropriate level of analysis to best ensure that DSCA’s analysis meets those goals. (Recommendation 8) The Secretary of Defense should ensure that the Director of DSCA conduct a review of the current structure of the FMS transportation fee rate, in consultation with other relevant DOD agencies, to determine if other rate structures could better balance considerations related to administrative burden, equity, efficiency, and revenue adequacy. (Recommendation 9) The Secretary of Defense should ensure that the Director of DSCA clarify internal guidance for the military departments on how to calculate the estimated actual transportation prices to charge FMS purchasers for certain items, such as by specifying a calculation methodology. This updated internal guidance should be based on consultations with the military departments, TRANSCOM, and any other relevant DOD components on which sources of data and which calculation methodologies would be most accurate. (Recommendation 10) We provided a draft of this report to DOD and State for review and comment. DSCA provided written comments on behalf of DOD, which are reprinted in appendix II. DSCA concurred with all of our recommendations, and identified actions it plans to take to address them and initial steps it has begun to take toward addressing some of them. We also received technical comments from DOD, which we incorporated in our report as appropriate. State did not provide any written or technical comments. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of State, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6881 or BairJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines (1) the balances maintained in the Foreign Military Sales (FMS) transportation accounts for fiscal years 2007 through 2018, (2) the extent to which the Defense Security Cooperation Agency (DSCA) established and implemented policies and procedures to help ensure management oversight of the transportation accounts, and (3) the extent to which DSCA processes for setting transportation fee rates ensure that these rates are set appropriately. To examine the balances of the FMS transportation accounts, we analyzed fiscal year 2007 to 2018 overall collections, expenditures, and balance data for each of the individual FMS transportation accounts maintained by the Defense Finance and Accounting Service (DFAS) in the Defense Integrated Financial System (DIFS). We chose to review data from these fiscal years based on data availability. To determine the reliability of these data, we reviewed the data for internal consistency by reviewing for duplicate entries, gaps, and obvious errors, and we compared the data to similar data obtained for a prior review of two other FMS fees. We also reviewed relevant documentation, including annual account assessments conducted by DSCA and the internal control procedures for conducting such reviews. Lastly, we interviewed DFAS and DSCA officials to clarify questions about how to interpret the data. We did not conduct any independent testing of the data obtained from DFAS to determine whether the amounts reflected correct payments made toward accurate billings. As such, when presenting collections and expenditures, we note that they reflect the amount of funds in the aggregate moved into and out of the FMS transportation accounts. We determined the collections, expenditures, and balance data to be reliable for the purpose of showing the movement of funds in and out of the FMS transportation accounts and the accounts’ balances over time. To analyze trends in collections into and expenditures from the FMS transportation accounts, such as in figures 4 and 6, we adjusted the data to remove the effects of two redistributions from the FMS administrative fee account that took place in fiscal years 2009 and 2011, as well as amounts that were moved into certain new Building Partner Capacity (BPC) transportation accounts to initially fund them in fiscal years 2012 and 2015. We reviewed documentation related to the two redistributions of funds from the FMS administrative fee account to the transportation account and the initial funding amounts allocated to new BPC transportation accounts, and interviewed DFAS and DSCA officials to understand how they accounted for these fund movements. To assess the extent to which DSCA established and implemented policies and procedures to help ensure management oversight of the FMS transportation accounts, we reviewed DSCA internal guidance included in DSCA’s Managers’ Internal Control Program (MICP) procedures for daily and annual FMS transportation account reviews, federal internal control standards, our prior report on federal user fees, and documentation showing how DSCA officials implemented those procedures. We also interviewed DSCA officials responsible for these reviews. Daily reviews. We reviewed a DSCA spreadsheet in which DSCA officials documented the daily reviews they conducted in fiscal year 2018. We chose to review this one fiscal year of data because it was the most recent complete fiscal year and would thereby be most relevant to current implementation. We also analyzed these data against the related MICP procedures, interviewed relevant DSCA and DFAS officials, and requested documentation of related correspondence to determine the extent to which DSCA consistently took any actions in response to these reviews. Because the data in these daily reviews is sourced from the same balance data in DIFS as we analyzed for our first objective, we compared the data between the two sources to ensure its consistency, and interviewed DFAS and DSCA officials about how these data were pulled for the daily reports. Based on these steps, we determined these data to be sufficiently reliable for assessing DSCA’s implementation of the daily review process. Annual reviews. We reviewed the annual reports DSCA created for fiscal years 2015 to 2018—all of the years for which DSCA created such reports—and interviewed DSCA officials about their process for creating these reports and other aspects of the MICP procedures for the annual review. To determine the extent to which the annual reports accurately convey information about the causes of trends in the accounts, we compared account expenditures data to oil price data for fiscal years 2007 to 2018. We performed this analysis because DSCA’s annual reports cite declining oil prices as a factor contributing to the increasing account balances in the FMS transportation accounts. For data on oil prices, we analyzed data from the U.S. Energy Information Agency on Cushing, Oklahoma, West Texas Intermediate oil prices by month, which is an established source for these data that is used commonly as a global benchmark for oil prices. As such, we determined these data to be reliable to use for this purpose. We also reviewed legislation that changed the rates the Department of Defense (DOD) can charge for FMS air shipments, and interviewed DSCA and U.S. Transportation Command officials about the effect and timing of this legislative change. We also reviewed the fiscal year 2015 to 2018 annual reports for the FMS transportation and administrative accounts to determine whether the redistributions that had been made from the FMS administrative account to the FMS transportation accounts were clearly reported, and reviewed related internal guidance in DOD’s Financial Management Regulations. For BPC-specific transportation accounts, we reviewed DOD’s Financial Management Regulations and related DSCA documentation against federal internal control standards regarding the clarity of internal control guidance. We also interviewed DSCA officials and received written responses to questions from DOD’s Office of the Under Secretary of Defense (Comptroller) regarding the process DSCA should follow when any of the BPC-specific transportation accounts close. To review the extent to which DSCA processes ensure that transportation fee rates are set appropriately, we reviewed DSCA guidance and interviewed DSCA and military department officials about the different processes DSCA uses to set transportation fees. For the transportation fee rate review, we reviewed DSCA’s MICP procedures and the requests DSCA sent to the military departments for data to analyze in its rate reviews against the Statement of Federal Financial Accounting Standards No. 4, our prior report on federal user fees, and federal internal control standards. To understand the reliability of the data the military departments submitted to DSCA and what these data showed in terms of the alignment between transportation costs and fees, we reviewed the data, including by performing internal consistency checks on the data, such as by reviewing it for duplicate entries, gaps, or obvious errors. We also reviewed any military department procedures for compiling these data and interviewed or received written responses from military department officials responsible for compiling the data. Based on these steps, we determined that these data were reliable for our purposes of making some comparisons between costs and fees for the sample provided. However, as noted earlier in this report, the departments could only provide partial data, which they did not select using systematic sampling techniques to ensure the data were indicative of the full population of shipments. Therefore, we determined that these data were unsuitable for DSCA’s purpose of making fee-setting decisions. We also reviewed DSCA documentation of the analysis it performed for its 2009 and 2018 transportation fee rate reviews, including analysis spreadsheets and briefings to DSCA management on the reviews’ results. Regarding instances when DOD charges FMS purchasers estimated transportation prices instead of a transportation fee rate, we reviewed DSCA guidance on this process in the Security Assistance Management Manual against related federal internal control standards. We also reviewed any internal guidance the military departments have developed to further guide these estimation processes, examples of the military department estimation processes, and other documents that showed concerns regarding these processes that the military departments had previously raised to DSCA. We interviewed and sent questions for written responses to DSCA and military department officials regarding these processes and the military departments’ concerns. We conducted this performance audit from May 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Cheryl Goodman (Assistant Director), Heather Latta (analyst in charge), Adam Peterson, Benjamin L. Sponholtz, John (Ryan) Bolt, Ming Chen, John Hussey, and Brandon Voss made key contributions to this report. Martin de Alteriis, Christopher Keblitis, Grace Lui, Susan E. Murphy, Laurel Plume, Heather Rasmussen, and Chanetta Reed also contributed to this report.", "summary": "The FMS program is one of the primary ways the U.S. government supports its foreign partners, by annually selling them billions of dollars of items and services. According to DOD, the FMS program is intended to operate on a “no profit, no loss” basis, with purchasers not charged excessive fees and fee revenue covering operating costs. Foreign partners can arrange for their own transportation of FMS items or pay DOD a transportation fee to cover the costs of DOD transporting them. The fees are collected into transportation accounts in the FMS Trust Fund. House Report 114-537 and Senate Report 114-255 included provisions that GAO review DSCA's management of FMS fees. This report examines (1) the balances of the FMS transportation accounts for fiscal years 2007 through 2018, (2) DSCA's management oversight of the accounts, and (3) DSCA's processes for setting transportation fees. GAO analyzed DOD data and documents, and interviewed DOD officials. Fees charged by the Department of Defense (DOD) for the transportation of defense items sold through the Foreign Military Sales (FMS) program are intended to approximate DOD's transportation costs over time. However, GAO found that the FMS transportation accounts accrued a combined balance of $680 million by the end of fiscal year 2018. Much of the growth occurred from the end of fiscal year 2011 through fiscal year 2018, when the account grew by approximately $630 million. The Defense Security Cooperation Agency (DSCA) has developed limited management oversight guidance for the FMS transportation accounts, which has contributed to the substantial balance growth. DSCA internal guidance requires daily and annual reviews of the accounts to monitor for significant changes in account balances and to ensure the accounts maintain a “healthy” level. However, internal guidance does not define a significant change or “healthy” level, such as a target range for the account balances. This has led to inconsistent reviews and limited oversight of the recent balance growth. DSCA also has no internal guidance on how to perform certain aspects of its annual reviews or what information to include in the resulting reports. As a result, DSCA officials have produced reports with incomplete information, such as on the causes for trends in the account balances, undermining DSCA management's ability to make informed decisions about the accounts. DSCA's processes for setting the FMS transportation fee do not ensure that aggregate fees approximate aggregate costs. For its transportation fee rate reviews, DSCA sends requests to the military departments for historical cost and fee data that lack specificity, such as on timeframes, sampling methodology, and data sources. As a result, DSCA has analyzed data that are not timely or systematically sampled. In addition, military department officials reported difficulty providing the requested data in part because DSCA's guidance did not specify data sources. Consequently, for the most recent review, Air Force and Navy were unable to find sufficient matching cost and fee data for DSCA to consider them usable. Further, DSCA has established no goals for rate reviews and has no written procedures to follow in performing them. These factors together contributed to recent growth in the FMS transportation account balances and will continue to hinder DSCA's ability to make appropriate rate-setting decisions moving forward. GAO is making 10 recommendations to DOD, including six recommendations to strengthen DSCA's oversight of the transportation accounts—such as by clarifying internal guidance—and four recommendations to improve its transportation fee setting processes. DOD concurred with all of the recommendations and identified actions it plans to take to address them.", "document_type": "gao"}
{"report": "Executive Order 13548 committed the federal government to similar goals stated in an executive order issued a decade earlier and required federal agencies to take additional actions. Specifically, the prior Executive Order 13163 called for an increase in the hiring of individuals with disabilities across the federal government and for agencies to develop plans for increasing employment opportunities for individuals with disabilities. The additional actions stated in Executive Order 13548 specified that federal agencies were to implement strategies for retaining federal workers with disabilities in federal employment, to make increased use of Schedule A excepted hiring authority for persons with disabilities, and to designate a senior-level official to be accountable for meeting the goals of the order and developing and implementing the agency’s plan. In January 2017, EEOC issued a final rule amending the regulations requiring federal agencies to engage in affirmative action for individuals with disabilities. The rule codified many of the requirements placed on agencies by management directives and past executive orders, among other things. Agencies were to begin following the rule in January 2018. The revised regulation requires that agencies take specific steps until they meet specific employment goals set by EEOC for individuals with disabilities and targeted disabilities, provide personal assistance services to certain employees who need them because of a targeted disability, and meet a number of other requirements designed to improve employment opportunities for individuals with disabilities in the federal workforce. OPM, EEOC, and Labor each have roles in advancing the hiring and retention of persons with disabilities in the federal government. OPM is responsible for executing, administering, and enforcing the civil service laws, rules, and regulations. This includes ensuring compliance with merit system principles that prohibit discrimination—including on the basis of disability—in all aspects of personnel management, among other things. Additionally, OPM is responsible for monitoring federal agencies’ implementation of affirmative action programs for disabled veterans, including providing technical assistance and reporting on progress made by agencies. EEOC, in the federal sector, is responsible for enforcing the employment discrimination prohibitions of anti-discrimination laws, including the Rehabilitation Act, which prohibits discrimination on the basis of disability. EEOC is responsible for the annual review and approval of agencies’ affirmative action program plans for the hiring, placement, and advancement of individuals with disabilities. It is also responsible for establishing procedures for handling federal employees’ allegations of discrimination and for providing for the adjudication of complaints and hearing of appeals. Labor’s Office of Disability Employment Policy (ODEP) is to provide national leadership in developing policy to eliminate barriers to the employment of individuals with disabilities in the public and private sectors. ODEP works in collaboration with federal, state, and local agencies, private sector employers, and employer associations to develop and disseminate evidence-based policy strategies and effective practices. The office also assists agencies and employers with adopting such policies and practices. Additionally, Labor administers the Federal Employees’ Compensation Act, which provides workers’ compensation coverage to federal employees for employment-related injuries and occupational diseases. Under Executive Order 13548, each of these agencies were assigned roles and responsibilities often in consultation with each other. For example, OPM, in consultation with Labor and EEOC, was tasked to identify and assist agencies in implementing strategies for retaining federal employees with disabilities. Additionally, OPM was also to consult with Labor, EEOC, and OMB in designing model recruitment and hiring strategies for agencies and developing mandatory training on employment of the disabled. Labor was to consult with OPM in pursuing innovative re-employment strategies and develop policies that foster improved return-to-work of employees who were injured on the job. OMB’s initial role was to convene federal agencies and assist their start- up efforts to implement the Executive Order, according to staff in OMB’s Office of Performance and Personnel Management. OMB staff told us the agency helped to establish a framework for coordination and collaboration among the key leadership agencies focused on making the federal government a model employer for persons with disabilities and to provide support for regulatory and policy initiatives related to disability employment. In 2015, in furtherance of an executive order focused on improving diversity and inclusion in the federal workforce, OMB joined OPM and EEOC and issued a memorandum to all heads of executive departments and agencies announcing the establishment of the Diversity and Inclusion in Government Council. The council initially operated under the direction of OPM, OMB, and EEOC and was formed to provide a forum for improving senior leadership engagement and collaboration on strategic and operational diversity and inclusion priorities. OMB’s role has since diminished as it delegated much of the leadership responsibilities to the other key leadership agencies. For reporting purposes, the federal government distinguishes between two major categories of disabilities: targeted and other disabilities. Targeted disabilities, generally considered to be more severe, include traumatic brain injuries, deafness, blindness, partial or complete paralysis, significant mobility impairments, and psychiatric disabilities, among others. Other disabilities include such conditions as gastrointestinal disorders, cardiovascular or heart disease, autoimmune disorders, pulmonary or respiratory conditions, and learning disabilities. Federal statutes and regulations provide special hiring authorities for people with disabilities. These include Schedule A excepted service hiring authority—which permits the noncompetitive appointment of qualified individuals with intellectual, severe physical, or psychiatric disabilities and appointments and noncompetitive conversion for veterans who are 30 percent or more disabled. To qualify for a Schedule A appointment, an applicant must generally provide proof of disability. Proof of disability can come from a number of sources, including a licensed medical professional, or a state agency that issues or provides disability benefits. The federal government gathers data on the number of individuals with disabilities in the workforce through OPM’s Standard Form 256, Self- Identification of Disability (SF-256). Federal employees voluntarily complete this form to disclose their disability status, as defined by the Rehabilitation Act. Our past work highlighted concerns about the accuracy of data captured in the SF-256. For example, we reported that agency officials and advocates for people with disabilities believe there is an undercount of employees with disabilities because some individuals may not disclose their disability status out of concern they will be discriminated against or precluded from advancement. In addition, employees may develop a disability during federal employment and may not know how to or why they should update their status. Disability status information is confidential and cannot be used to affect an employee in any way. Given our previously reported concerns, we recommended that OPM assess the extent to which the SF-256 accurately measures progress toward the goal of Executive Order 13548 and to explore options for improving the accuracy of SF-256 reporting. To address our recommendation, OPM updated its 2012 Employee Feedback Survey to allow federal employees to confidentially self-disclose a disability and serve as a source of comparison through which OPM could assess the accuracy of the SF-256. Federal agencies exceeded the government-wide goal to hire an additional 100,000 persons with disabilities in the federal government by 2015, according to our analysis of OPM’s EHRI data across the 24 CFO Act agencies. During fiscal years 2011 through 2015, a total of approximately 143,600 persons with disabilities were hired across all positions, which includes full-time permanent positions and part-time or temporary positions. Of those hires, approximately 87,000—61 percent— were hired into full-time permanent positions. Similar hiring continued to increase in 2016 and 2017 as the federal government hired approximately an additional 79,600 persons with disabilities during those 2 years across all positions, of which approximately 49,200—62 percent—were full-time permanent positions. Figures 1 and 2 show the total government-wide number of persons with disabilities and targeted disabilities hired in fiscal years 2011 through 2017. Our determinations of the number of new hires each year were consistently lower than the numbers OPM included in its executive branch reporting. The discrepancy between our numbers and OPM’s reported counts is largely attributed to our exclusion of agency-to-agency transfers in our analysis. For the purpose of our analysis of government-wide hiring, we excluded transfers because we did not consider those to be new hires since those individuals remained employed in the federal government. Figure 3 shows the total government-wide number of persons without disabilities hired during the same time period. According to our analysis, a total of approximately 903,000 persons without disabilities were hired across all positions between 2011 through 2015. Of those hires, approximately 403,900—45 percent—were hired into full-time permanent positions. Hiring continued to increase with an additional 377,150 in 2016 and 2017 combined across all positions, of which approximately 189,200—50 percent—were full-time permanent positions. The data shown in figures 1 and 3, and summarized in Table 1, show that from 2011 through 2017, the percent of hires with disabilities steadily increased from 11 percent to almost 20 percent. Our analysis at the agency level shown in table 2 shows that all agencies increased the percentage of persons with disabilities hired from 2011 through 2017 and almost all agencies increased the percentage of persons with targeted disabilities hired over the same period. Table 2 shows this information by agency for fiscal years 2011, 2015, and 2017. We chose to present these years of data to mark the first and last years of the 5-year period specified in Executive Order 13548 and to also show the most recent data available at the time of our review. As part of our analyses of individuals hired during the 2011 through 2017 time period, we analyzed employee retention in terms of the number of years an individual hired during that time period remained employed. Across the federal government, of the more than 223,000 persons with disabilities hired during the 2011 through 2017 time period, approximately 39 percent of them stayed in the federal government for less than 1 year and approximately 60 percent stayed for less than 2 years, as shown in figure 4. These percentages are slightly better than the percentages of employees without disabilities who left within the same amount of time as shown in figure 5. Across the federal government, of the more than 1.28 million persons without disabilities hired during the 2011 through 2017 time period, approximately 43 percent of them stayed in the federal government for less than 1 year and approximately 60 percent of them stayed for less than 2 years. The data shown in figures 4 and 5 taken in context together provide an aggregate overview of government-wide hiring and retention trends of individuals with disabilities in comparison to hiring and retention trends of individuals without disabilities. We found the trends to be generally consistent between the employee groups during this time period, with the largest percentage of hires staying less than 1 year. These departures may be explained, in part, by the proportion of employees hired into temporary positions who therefore were not necessarily expected to stay on the job for a longer duration, or by employees who did not meet performance standards. To pinpoint the root causes behind these departure rates and to determine where appropriate improvements and potential solutions may be warranted, targeted data collection, tracking, and analysis is needed. Moreover, the loss of such a substantial percentage of new hires within their first 2 years of employment provides an opportunity for the federal government to examine why this occurs, identify any lessons learned, and better target its retention efforts as appropriate to potentially reduce such early departures. Further, these retention trends have implications related to agencies’ ability to meet and sustain progress toward the federal goals of ensuring that at least 12 percent of their workforce is comprised of employees with disabilities including 2 percent comprised of employees with targeted disabilities. In addition, we analyzed the number of persons with disabilities hired into each occupational category as identified in OPM’s EHRI database for fiscal years 2011 through 2015. The categories are administrative, blue collar, clerical, professional, technical, and other. Within each category, we identified the number of employees who remained in those positions for at least 2 years. Our analysis summarized in table 3 shows the highest retention rates for employees with disabilities and employees with targeted disabilities occurred in three categories: administrative, blue collar, and professional. For example, in the professional occupational category, the retention rates were approximately 48 and 43 percent for employees with disabilities and targeted disabilities, respectively—which were the highest levels of retention for persons with disabilities and targeted disabilities in any occupational category. However, the number of persons with disabilities hired into this category is considerably lower than that of non- disabled hires into the same category. Specifically, approximately 13 percent of persons with disabilities and approximately 11 percent of persons with targeted disabilities were hired into the professional occupational category. In contrast, as shown in table 3, 23 percent of persons with no disability were hired into this same occupational category and retained at a similar rate. Our analysis by GS level in table 4 shows that retention rates increase with GS level, regardless of disability, with retention rates being slighlty higher for persons without disabilities for the top three GS levels. Moreover, persons with disabilities and targeted disabilities were more likely to be hired at the lowest three GS levels, with one exception. Persons with disabilities fared equally or relatively well in GS-11 and above categories compared to persons without disabilities or with targeted disabilities. OPM does not routinely track or report retention data on employees with disabilities, which could help inform both agency-specific and government-wide assessments of how the federal government is performing with retaining the employees it hires. OPM officials said OPM has the ability to track the retention of all employees in the federal government and can do so for any specific category of employees on an as needed basis or upon request. For example, in 2015, OPM started reporting new hire retention data on employees who are veterans by including this information in its annual report on the employment of veterans in the federal government. This report also includes hiring data on disabled veterans. However, there is no similar OPM tracking or reporting of retention data for all individuals with disabilities including targeted disabilities. The federal regulations, executive order and management directive discussed earlier in this report all include statements about the importance of retaining individuals with disabilities in the federal government. For example, Executive Order 13548 stated that agencies must improve their efforts to employ workers with disabilities through increased recruitment, hiring, and retention of these individuals. Further, it stated that OPM, in consultation with Labor and EEOC, shall identify and assist agencies in implementing strategies for retaining federal workers with disabilities in federal employment. Federal regulations state that agencies shall give full consideration to the retention of qualified individuals with disabilities in the federal workforce. EEOC’s MD 715 requires agencies to conduct an internal review and analysis of the effects of their current and proposed policies, practices, procedures and conditions that relate to the employment—including retention—of individuals with disabilities. Making use of the agency-specific data OPM already gathers in its EHRI database complemented with the retention information agencies report in their annual MD 715 submissions would help to facilitate more comprehensive analyses of the retention of employees with disabilities across the federal government. Such analyses could provide a fuller picture of how the federal government is performing with retaining the employees it hires, help identify common agency experiences—both successes and challenges—and assist in pinpointing the root causes that contribute to retention rates of employees with disabilities in the federal workforce. Making retention data available to federal agencies for such use is also consistent with a federal internal control standard that states that management is to obtain relevant data from reliable internal and external sources in a timely manner so that they can be used for effective monitoring. Without routinely tracking and analyzing data on how long employees with disabilities remain employed in their agencies, federal managers are limited in their ability to assess the performance and effectiveness of the hiring and retention efforts put in place at their agencies. In addition, agencies are missing opportunities to leverage such information to help inform their own internal reviews and analysis of progress in meeting the goals included in federal regulations that at least 12 percent of their workforce be comprised of employees with disabilities including 2 percent comprised of employees with targeted disabilities. The three agencies we selected as case illustrations generally experienced increases in the percentage of employees hired with disabilities and targeted disabilities. Table 5 shows the percentage of employees hired by each agency in fiscal years 2011, 2015, and 2017. We chose to present these years of data to mark the first and last years of the 5-year period specified in Executive Order 13548 and to also show the most recent data available at the time of our review. For our analysis of individual agency-level hiring data, we included transfers in cases where employees transferred into an agency because we considered that to be a new hire at the individual agency level. Similar to the government-wide retention analysis described earlier, we also examined retention data at DOJ, SBA, and SSA. Of the employees with disabilities hired at DOJ and SSA from 2011 through 2017, approximately 31 percent and 33 percent, respectively, stayed in the federal government for less than 1 year. Approximately 53 percent and 51 percent, respectively, stayed for less than 2 years. These retention rates were slightly better than government-wide rates. In contrast, approximately 65 percent of employees with disabilities hired at SBA during that time period stayed for less than 1 year and approximately an additional 9 percent stayed for less than 2 years of employment. These departures may be explained, in part, by the proportion of employees hired into temporary positions who therefore were not necessarily expected to stay on the job for a longer duration. For example, SBA staff said that, on average, 45 percent of SBA’s workforce is comprised of temporary employees hired by the agency’s Office of Disaster Assistance during a disaster. As such, SBA expects turnover among those hires, including employees with disabilities. Similar to our analysis of government-wide retention rates by GS level and by occupational category, we identified the number of individuals hired at each of the three selected agencies during fiscal years 2011 through 2015 who stayed for at least 2 years. We found that generally across the three agencies, employees with disabilities were retained longer at the higher GS levels. As the GS levels increased, individuals without disabilities retained their jobs at a slightly higher rate than individuals with disabilities. Our analysis of occupational categories found that, in general, the three agencies each retained people with disabilities at lower rates than people without disabilities. More detailed hiring and retention data for each of the three agencies are included in appendix I. To aid recruitment and employment opportunities for individuals with disabilities, the three agencies we interviewed reported using (1) collaboration with other federal agencies for knowledge and information sharing and (2) coordination with employee resource and advisory groups. The following examples are illustrations of practices that selected agencies implemented. We did not assess the effectiveness or attempt to quantify the costs or benefits of the practices. Two agencies provided examples of their collaboration with other federal agencies for knowledge and information sharing. For example, DOJ officials told us that staff from their agency’s Criminal Division participated in an OPM effort using a “Resume Mining” feature in the USAJOBS Agency Talent Portal, in which the division’s human resources specialists searched through active resumes and filtered the searches based upon candidates who were eligible to be hired non-competitively under the Schedule A hiring authority. According to SBA officials, they used the Workforce Recruitment Program—a resource managed through Labor to help federal hiring managers connect with qualified candidates with disabilities for all jobs. SBA also retains a repository of resumes for individuals with disabilities to share with hiring managers. In 2015, to assist hearing impaired candidates and in a joint effort with the Federal Communications Commission, SBA hired staff fluent in American Sign Language (ASL) to provide video relay services directly to the deaf and hard-of-hearing communities. As a result, SBA officials told us SBA’s ASL customer support staff is able to communicate with and assist hearing-impaired job candidates. SBA also developed a National Strategic Recruitment Plan, which highlights Labor’s Workforce Recruitment Program for College Students with Disabilities. SBA officials said this plan has served as a successful tool for recruitment and hiring managers within their agency. Two of the three selected agencies we reviewed, DOJ and SSA, have disability employee resource or advisory groups made up of employees and management. These groups are generally made up of a variety of representatives from across the agency, including human resources professionals, hiring managers, recruitment coordinators, and employees with disabilities. The purpose of these groups includes helping to identify policies and procedures that support a positive work environment for people with disabilities. For example, DOJ’s Attorney General’s Advisory Committee for People with Disabilities (AGCPD) meets quarterly and works with DOJ management on disability employment issues. AGCPD advisory members told us one of their most significant contributions has been assisting with developing an agency-wide policy to help increase the use of the Schedule A hiring authority between 2010 and 2012. As a result, the number of individuals with disabilities hired at DOJ increased, according to AGCPD members. However, they said the agency has been unable to sustain those numbers in recent years. DOJ staff said this may also be attributed, in part, to a hiring freeze across DOJ at the time that affected all hires. AGCPD members also told us they routinely review DOJ’s disability hiring and retention percentages to monitor agency progress on this issue. According to SSA officials, SSA’s employee advisory group, the National Advisory Council of Employees with Disabilities (NACED), advises the agency regarding reasonable accommodations, recruiting, and creating pathways for promotions and retention of employees with disabilities. SSA’s management was involved in establishing guidelines for the advisory group to operate within the agency. NACED has a senior executive service member who serves as the council’s liaison with SSA senior management. NACED assisted in the creation of mandatory agency training for managers and employees at SSA on disability awareness and sensitivity. The group also assisted the agency in producing a video that features SSA employees with disabilities and is available on SSA’s intranet website. In addition, the advisory group assisted the agency to ensure SSA’s systems are compliant with assistive technology. In addition, according to SSA officials, the agency has placed designated Selective Placement Program Coordinator (SPPC) points of contact in each of its regional offices to support disability recruitment and hiring efforts. SSA officials told us the role of their SPPC has been instrumental in building coalitions and networks with their internal and external stakeholders, including connecting SSA’s human resources, equal employment opportunity (EEO), and employee affinity groups. SSA officials said these essential connections enable their agency to acquire the information needed to make informed disability employment and general EEO program and policy decisions. As noted earlier, federal statutes and regulations provide special hiring authorities for people with disabilities, which includes Schedule A hiring authority. Agencies are not required to use Schedule A authority and can choose to use the traditional competitive process to fill job vacancies. However, Executive Order 13548 called for increased utilization of the federal government’s Schedule A excepted service hiring authority for persons with disabilities, as appropriate. Consistent with federal emphasis on the use of Schedule A, all three selected agencies reported to us that they provide training on Schedule A hiring authority to their hiring managers and human resources professionals. For example: According to SBA officials, the agency provides supervisory training to all hiring managers and supervisors to emphasize Schedule A hiring authority, among other hiring flexibilities. SSA officials told us their agency holds annual mandatory training for managers and human resource specialists on special hiring authorities that apply to individuals with disabilities, including Schedule A, and reasonable accommodations. SSA also provides a manual to its managers focused specifically on recruitment, interviewing, and hiring related to Schedule A authority. DOJ officials told us their agency participated in ongoing training and other initiatives designed to increase the use and understanding of Schedule A. Nevertheless, the agencies we spoke with reported that some hiring managers and human resources staff are unfamiliar with or unsure of how to use the Schedule A hiring authority. Consequently, the agencies have found that there is a continual need to increase hiring managers’ awareness of Schedule A and to educate both managers and human resource personnel on the use of the hiring authority. For example: SBA officials said their managers often have questions about what Schedule A is and how to use it in the hiring process. SSA officials said they continue to receive questions about the hiring authority from their newer managers, which they address on a case- by-case basis. Similarly, the key leadership agencies underscored this as an issue they have seen government-wide in their experience. For example, EEOC staff said because hiring managers change frequently, information and the use of the Schedule A hiring authority may be a topic that was not part of their previous work experiences or portfolios. EEOC officials said that all managers could benefit from more training to understand how and when it is permissible to use the special authority to hire individuals with disabilities. To help address issues around the use of Schedule A, officials from the key leadership agencies emphasized the importance of federal agencies having designated staff familiar with disability issues, such as an SPPC, in which a part of his or her job responsibilities is to help educate and train the workplace on disability issues such as the use and benefits of the Schedule A hiring authority. Consistent with this guidance, two of the three agencies use SPPCs to provide guidance and, in one case, provide training. For example: SBA’s SPPCs frequently provide guidance on the option to utilize the Schedule A hiring authority prior to opening a competitive job announcement on USAJOBS. SSA has designated SPPCs in each of its regional offices. The SPPCs provided guidance and training to managers on the appointment of individuals with disabilities using the Schedule A appointment authority. As a result, in fiscal year 2019, SSA officials said these efforts contributed to their agency filling more than 250 positions using the Schedule A hiring authority. Additional opportunities exist to further address issues around the use of Schedule A. We have previously reported that training at all staff levels, in particular training on hiring, reasonable accommodations, and diversity awareness can help disseminate leading practices throughout an agency and communicate expectations for implementation of policies and procedures related to improving employment of people with disabilities. In addition, our past work has underscored the importance of assessing and measuring the real impact of training to determine how it contributes to the accomplishment of agency goals and objectives. Moreover, a leading training investment practice is to evaluate the benefits achieved through training, such as having a formal process for evaluating improvement in performance and tracking the impact of training on the agency’s performance goals. While assessing training is important, the three selected agencies said they do not assess the impact of their training related to Schedule A. For example, according to SBA officials, their training covers a range of hiring flexibilities beyond Schedule A. As such, SBA officials said they are unable to evaluate the effect of the training to specifically measure an increased level of hiring managers’ and human resources professionals’ understanding of how and when to use Schedule A authority. SSA officials told us that while their agency does not evaluate their training, the agency is currently developing an evaluation module to allow employees and managers to provide feedback on the effectiveness of their Schedule A training. However, SSA did not provide a committed timeframe for completion of such a module. DOJ staff said training is provided by its various component agencies and is updated when appropriate. However, DOJ did not provide any further details to explain the frequency, content, or results of such evaluations. Without evaluating the impacts of training to ensure that hiring managers understand how and when to use the Schedule A hiring authority, agencies may be missing opportunities to enhance awareness of and sensitivity to disability issues and opportunities to increase the number of employees with disabilities across the federal workforce. Federal agencies are required to provide reasonable accommodation to qualified employees or applicants with disabilities, unless to do so would cause undue hardship. In general, a reasonable accommodation is a change in the work environment or in the way things are customarily done that would enable an individual with a disability to apply for a job, perform the duties of a job, or enjoy the benefits and privileges of employment. Officials from the three selected agencies indicated that many reasonable accommodation provisions are low- to no cost to their agencies, often involving minor changes to an employee’s workspace or work schedule, or modifications to work-related technologies. For example, the most common reasonable accommodation requests cited by each of the agencies included: providing ergonomic adjustments or modifications to the layout of workspaces; adjusting work schedules to allow employees with chronic medical conditions to attend medical appointments and complete their work at alternate times or locations; providing sign language interpreters or closed captioning at meetings making materials available in braille or large print. In addition, according to information posted on the Office of Disability Employment Policy website within Labor, examples of other job accommodations that are low cost and often involve minor changes to a person’s work environment include: physical changes, such as installing a ramp or restroom modifications; accessible and assistive technologies such as providing screen reader software or using videophones to communicate with employees who have impaired hearing; and policy enhancements, such as allowing service animals in the workplace. Federal agencies are required to post on their websites, and make available to all applicants and employees in written and accessible formats, procedures for reasonable accommodation. Agencies are also required to collect specific information about each reasonable accommodation, including whether the accommodation was granted and the basis for any denial. All three of the selected agencies indicated in their 2018 MD 715 reports to EEOC that their agencies have these established procedures in place and are in compliance with EEOC regulations and guidance. While the three selected agencies reported they have processes in place for receiving reasonable accommodations requests, only SSA has procedures for obtaining employee feedback from employees after an accommodation is provided. According to agency officials, the agency offers employees who have requested job accommodations various opportunities to provide feedback to agency management about their reasonable accommodation experience. For example, SSA officials said their agency uses a dedicated email inbox and telephone number to receive inquiries and feedback from reasonable accommodations customers and stakeholders. Both of these are monitored daily by the agency’s Center for Accommodations and Disability Services (CADS) to ensure emails and calls are logged and tracked. Additionally, according to agency officials, if an employee prefers to contact the reasonable accommodations office anonymously, employees can complete the anonymous Process Improvement Comments Survey to submit concerns, comments, or recommendations for reasonable accommodations process improvement. To address issues and concerns received through any of these means, CADS staff reach out to the relevant managers, as appropriate, and only share information on a need-to-know basis, or as otherwise required by applicable law. According to SSA officials, SSA’s policy also requires that managers or CADS staff confirm with the employee that a job accommodation was received and is effective prior to closing the request in the agency database. Finally, SSA’s policy requires supervisors to continually engage in this interactive process to ensure the continued effectiveness of job accommodations. In contrast, DOJ and SBA officials reported that their agencies do not have any specific procedures in place to solicit ongoing employee feedback from employees who request reasonable accommodations. Staff from both agencies said that communication between the supervisor and individual needing a reasonable accommodation is encouraged. In general, if an afforded accommodation is ineffective or needs modification, the employee and supervisor are responsible for contacting the appropriate disability employment program manager to address the issue. Federal agencies are not explicitly required to obtain feedback from employees about the effectiveness of their job accommodations experience. However, EEOC policy guidance states that agencies should keep cumulative records for at least 3 years to track their performance with regard to providing reasonable accommodations to employees. Tracking performance over a 3-year period is critical to an agency’s ability to assess whether it has adequately processed and provided reasonable accommodations, according to EEOC guidance. Agencies are encouraged to use this tracking information to evaluate whether and where they need to improve their handling of reasonable accommodation requests. In addition, this type of monitoring is consistent with federal internal control standards. Specifically, the standard calls for ongoing monitoring to be built into the entity’s operations, performed continually, and responsive to change. Without periodically soliciting, obtaining, and documenting employee feedback on agencies’ reasonable accommodations efforts, management is missing opportunities to evaluate the effectiveness of their programs, identify potential risks, and identify any improvements that may be warranted. For example, such information could provide valuable insights about the timeliness of processing and fulfilling employees’ requests and the ongoing effectiveness of an accommodation. In some cases, an accommodation may no longer be effective for an employee for various reasons such as if the employee’s limitations change, workplace equipment changes, job responsibilities change, or the accommodation involves equipment or software that requires maintenance or updates. EEOC, OPM, and Labor took various actions during the course of the 5- year period specified under the executive order for meeting the government-wide hiring goal and have continued their efforts. For example, the agencies began to meet quarterly immediately after the executive order was signed to establish collaborative actions they could take to increase disability hiring and retention measures and to discuss best practices focused on hiring and retaining individuals with disabilities. Officials from OPM, EEOC, and Labor continue to meet quarterly as participants in an interagency working group called the Federal Exchange on Employment and Disability (FEED). FEED meetings cover a broad range of federal disability topics, including sharing best practices and establishing collaborative partnerships designed to make the federal government a model employer of people with disabilities. For example, at one FEED meeting, OPM announced a new resource to help address some common questions OPM receives about Schedule A. At another FEED meeting, OPM and EEOC officials discussed possible strategies agencies can consider when they are planning to re-survey their agencies through the Standard Form 256, Self-Identification of Disability (SF-256), such as initiating the re-survey campaign during Disability Awareness Month when there is increased attention on disability issues. OPM assisted agencies with disability hiring plans and authorities and compiled government-wide data. Under EO 13548, OPM was required to implement a system for reporting regularly to the President, heads of agencies, and the public on agencies’ progress in implementing their disability hiring plans and meeting the objectives of the executive order. In May 2012, we reported on OPM’s progress in reviewing agencies’ hiring plans and found that many plans had deficiencies that needed to be addressed. For example, not all plans identified a senior- level official responsible for development and implementation of the plan. We recommended that OPM incorporate information about such deficiencies in its external reporting. OPM did so, and also worked with agencies to correct any plan deficiencies by November 2012. In 2016, OPM issued its capping report announcing the success of the government’s effort, which included a summary of the initiatives taken to improve agency coordination, education, and training accompanied by a series of tables showing the composition of disability hires across the federal workforce. OPM also continues to collect government-wide disability data, which is available to agencies through the MAX.gov web portal, and provides assistance to agencies upon request. In October 2018, the Director of OPM issued a joint memorandum with the Chair of EEOC to the Chief Human Capital Officers Council regarding updates to the SF-256 to reflect changes to terms used to describe targeted disabilities, serious health conditions, and other disabilities. As discussed in an earlier section of this report, individuals use this form to voluntarily self- identify a disability, and OPM uses the information provided through this form for data collection purposes only. The revised form includes simplified condition descriptions and provides respondents with the option of identifying if they have a targeted disability, disability, or serious condition without specifying a diagnosis. SF-256 continues to be the primary tool for measuring the workforce participation of persons with disabilities in the federal government. The joint memorandum reminded agencies that OPM and EEOC are available to assist agencies in their efforts to help employees self-identify as people with disabilities and people with targeted disabilities, as appropriate. EEOC collects information through MD 715, issued regulations, and provides technical assistance. EEOC’s ongoing data and information collection efforts under MD 715 require agencies to report annually on the status of their equal employment opportunity programs. This includes agency-specific self-assessments of the extent to which they are meeting their responsibilities to provide employment opportunities for qualified applicants and employees with disabilities and targeted disabilities. If agencies identify any barriers to the equal employment of persons with disabilities, they must work to eliminate the barrier. EEOC’s MD 715 annual reporting requirement included under Part J captures agencies’ descriptions of how their affirmative action efforts improve the recruitment, hiring, advancement, and retention of applicants and employees with disabilities. According to EEOC’s guidance to agencies, Part J is to assist agencies in meeting the requirements for an affirmative action plan. Specifically, Part J requires agencies to examine employment trends and participation rates of persons with reported and targeted disabilities in agency programs. In 2017, Part J was revised and now solicits agency information about voluntary and involuntary separations of employees with disabilities. For example, agencies are to confirm whether voluntary and involuntary separations occurred at a rate exceeding that of employees without disabilities. Agencies are required to complete Part J and, for transparency purposes, post their affirmative action plans on their external websites. The importance of this type of information is underscored by the analysis summarized in an earlier section of this report showing that approximately 60 percent of persons with disabilities hired into the federal government during 2011 through 2017 stayed for less than 2 years of service. Also as noted earlier, opportunities exist to enhance collection and analysis of retention data and learn about what factors contribute to retention rates of employees with disabilities in the federal government. EEOC provides various types of support to agencies to help them implement requirements of the revised regulations on affirmative action for individuals with disabilities. For example, EEOC officials said they visited all agencies to provide guidance and technical assistance with their hiring plans. EEOC continues to provide ongoing feedback to agencies, both formally and informally, and visits agencies on a 3-year rotation cycle. As part of EEOC’s outreach, agency representatives provide presentations to, and participate in meetings with, federal employees and employers. The agency’s website also includes a list of outreach coordinator contacts for each of its field offices. EEOC’s Training Institute provides a variety of training programs specialized for the federal sector, including courses on disability issues and MD 715 barrier analysis, as well as customized training throughout the year to meet particular agencies’ needs. EEOC’s federal training courses can be delivered on site or virtually. Labor provides tools, resources, education, and training to agency managers. Labor has implemented and supported a number of initiatives aimed at enhancing the federal sector’s performance on disability employment. For example, Labor’s Office of Disability Employment Policy supports the Employer Assistance and Resource Network on Disability Inclusion (EARN), which is a federal resource that provides education, training, tools, and resources for managers on the hiring, retention, and advancement of persons with disabilities. In 2018, EARN issued a federal framework—in partnership with EEOC and OPM—which outlined various employment strategies and practices for agencies to consider and incorporate into their own efforts related to disability inclusion in the workforce. In addition, Labor leads an interagency working group known as the Federal Exchange on Employment and Disability, which is comprised of federal staff across government with roles in developing, implementing and managing disability employment programs to foster cross-agency collaboration and share best practices. The agency also developed a toolkit for Federal Agencies on Hiring People with Disabilities outlining a five-step process and related resources to assist federal agencies in their efforts to increase the employment of people with disabilities. Another effort supported by Labor provides more targeted technical assistance and free consulting services on workplace accommodations through the Job Accommodations Network. To increase the recruitment of persons with disabilities, Labor also plays a lead role in the Workforce Recruitment Program for College Students with Disabilities, which is a recruitment, and referral program that connects federal and private sector employers nationwide with college students and recent graduates with disabilities for summer or permanent employment. Labor has also developed and provided assistance on various trainings for federal hiring managers and human resources professionals, including an OPM course titled, “A Roadmap to Success: Hiring, Retaining and Including People with Disabilities.” In its effort to become a model employer, the federal government increased employment opportunities for persons with disabilities; provided specific direction and guidance to agencies through various executive orders, management directives, and regulations; and exceeded its goal to hire an additional 100,000 individuals with disabilities. However, OPM does not routinely track or report retention data, which could help pinpoint the root causes behind disabled employee departure rates. Making use of the agency-specific data OPM already gathers in its EHRI database complemented with the retention information agencies report to EEOC would allow for more comprehensive retention analyses of employees with disabilities across the federal government. Such analyses would provide a fuller picture of how the federal government is performing with retaining the employees it hires and help to identify common agency experiences, both success and challenges. Without comprehensive analyses of retention data, the federal government is limited in its ability to assess the performance and results of the hiring and retention efforts for this segment of the workforce. Selected agencies implemented a number of practices that helped bolster their recruitment and hiring of persons with disabilities, including collaborating with other federal agencies for knowledge and information sharing, coordinating efforts with employee resource or advisory groups, and providing additional training for hiring managers and human resources staff on using Schedule A hiring authority—one of the commonly used hiring flexibilities available to agencies to onboard qualified individuals with disabilities. However, the selected agencies do not assess or measure the impact of their Schedule A training to determine how it contributes to the accomplishment of federal goals to increase the number of employees with disabilities across the federal workforce. In addition, opportunities exist to enhance the effectiveness of selected agencies’ reasonable accommodations programs by obtaining employee feedback from employees about their job accommodations experience. OPM, EEOC, and Labor have worked collaboratively to assist agencies with enhancing their recruitment and hiring efforts. They compiled government-wide data, issued guidance and regulations to clarify agencies’ responsibilities and obligations to strengthening employment opportunities for disabled persons, and provided various resources, education, and training. We are making the following recommendation to OPM: The Director of OPM should routinely track and report retention data for employees with disabilities and make such data available to federal agencies, including EEOC, through a centralized web portal—such as MAX.gov. For example, OPM could track and report such data by General Schedule level pay groupings, which could help pinpoint root causes that contribute to retention rates, inform assessments of government-wide progress on employee retention, and identify needed improvements. (Recommendation 1) We are making the following recommendations to DOJ: The Attorney General of the United States should develop and implement policies and procedures for assessing the impact of training provided to agency hiring managers and human resources staff on Schedule A hiring authority. This includes assessing the impact of its training on agency performance goals related to increased hiring of individuals with disabilities and targeted disabilities. (Recommendation 2) The Attorney General of the United States should develop and implement policies and procedures for obtaining employee feedback about the agency’s reasonable accommodations efforts and use such information to evaluate the ongoing effectiveness of the program. This may include identifying any effects on employee retention, identifying potential risks, and determining any improvements that may be warranted. (Recommendation 3) We are making the following recommendations to SBA: The Administrator of SBA should develop and implement policies and procedures for assessing and tracking the impact of training provided to agency hiring managers and human resources staff on Schedule A hiring authority. This includes assessing the impact of its training on agency performance goals related to increased hiring of individuals with disabilities and targeted disabilities. (Recommendation 4) The Administrator of SBA should develop and implement policies and procedures for obtaining employee feedback about the agency’s reasonable accommodations efforts and use such information to evaluate the ongoing effectiveness of the program. This may include identifying any effects on employee retention, identifying potential risks, and determining any improvements that may be warranted. (Recommendation 5) We are making the following recommendation to SSA: The Commissioner of SSA should develop and implement policies and procedures for assessing and tracking the impact of training provided to agency hiring managers and human resources staff on Schedule A hiring authority. This includes assessing the impact of its training on agency performance goals related to increased hiring of individuals with disabilities and targeted disabilities. (Recommendation 6) We provided a draft of the report to OPM, EEOC, Labor, OMB, DOJ, SBA, and SSA for review and comment. We received written comments from 3 agencies—OPM, SBA, and SSA—that are reprinted in appendices II through IV and summarized below. EEOC informed us that they had no comments. Labor and DOJ provided technical comments, which we incorporated as appropriate. OMB did not provide comments on the draft. OPM concurred with our recommendation to routinely track and report retention data for employees with disabilities and make such data available to federal agencies. OPM stated that it already routinely tracks retention data for persons with disabilities by agency. In addition, OPM responded that retention data for employees with disabilities by agency and GS level pay groupings for fiscal years 2017 and 2018 can be obtained by federal agencies through the MAX.gov website. However, OPM did not provide any supporting documentary evidence or further details to explain its tracking efforts or which data are available to federal agencies. SBA disagreed with the retention data we present in figure 8, showing that approximately 65 percent of employees with disabilities hired at SBA between 2011 through 2017 stayed less than one year. In its written comments, SBA stated that under hiring authorities it uses in responding to disasters, appointments are generally not to exceed one year. As indicated in our report, we acknowledge that each of our retention analyses include full-time permanent hires and part-time or temporary hires. We also include a specific statement regarding temporary hires at SBA’s Office of Disaster Assistance. SBA concurred with our recommendation to assess and track the impact of training provided to agency hiring managers and human resources staff on Schedule A hiring authority. SBA responded that it will formally evaluate the impact of training to ensure hiring managers understand the use of Schedule A hiring authority and assess hiring trends and retention. SBA partially concurred with our recommendation to obtain employee feedback about its reasonable accommodation efforts. SBA stated that its procedures require supervisors to contact the Disability Employment Program Manager with concerns about the effectiveness of a provided accommodation and work together to make any necessary adjustment. SBA further stated that the procedures have been revised and will include a requirement for completing a feedback survey aimed to determine the effectiveness of the reasonable accommodation program and make any adjustments required. SBA stated that it also established an internal mailbox for reasonable accommodation communications that is monitored daily. Effective implementation of SBA’s plans, including administering a survey, would meet the intent of the recommendation. SSA concurred with our recommendation to assess and track the impact of training provided to agency hiring managers and human resources staff on Schedule A hiring authority. SSA stated that it is revising its framework to include outcome-based evaluations for training related to the employment and support of individuals with disabilities, including Schedule A hiring. DOJ did not agree or disagree with the recommendations. We are sending copies of this report to the appropriate congressional committees, the Director of OPM, the Chair of EEOC, the Secretary of Labor, the Director of OMB, the Attorney General of DOJ, the Administrator of SBA, and the Commissioner of SSA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or JonesY@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. As part of our review, we selected three agencies as case illustrations to examine practices they have adopted to increase hiring and retention of individuals with disabilities. The three selected agencies are the Department of Justice (DOJ), the Social Security Administration (SSA), and the Small Business Administration (SBA). Our selection was based on various factors including the agency’s size in terms of total full-time employees and average percentage of total employees with reported disabilities or targeted disabilities during 2011 through 2017. For each of the three agencies, we analyzed personnel data captured in the Office of Personnel Management’s (OPM) Enterprise Human Resources Integration (EHRI) database including the General Schedule (GS) levels in which individuals with disabilities were placed and their position classifications. The following figures and tables summarize our analyses of hiring and retention rates of individuals with and without disabilities in the three selected agencies during fiscal years 2011 through 2017. These analyses provide an aggregate overview of hiring and retention trends of individuals with disabilities at the three selected agencies as compared to hiring and retention trends of individuals without disabilities at these agencies. We found the trends to be generally consistent between the employee groups. During the 2011 through 2017 time period we examined, 31 percent of the total number of persons with disabilities hired at DOJ during that time stayed in the federal government for less than 1 year and nearly 54 percent of them stayed for less than 2 years, as shown in figure 6. During that same time period, approximately 24 percent of the total number of persons without disabilities who were hired stayed for less than 1 year of service while approximately 46 percent of hires stayed for less than 2 years of service, as shown in figure 7. The data shown in figures 6 and 7 taken in context together indicate that retention at DOJ during this time period was generally consistent for persons both with and without disabilities. These departures may be explained, in part, by the proportion of employees hired into temporary positions who therefore were not necessarily expected to stay on the job for a longer duration, or by employees who did not meet performance standards. Tables 6 and 7 show the results of our analysis of employee retention at DOJ by occupational category and GS level for individuals hired in fiscal years 2011 through 2015 and stayed for at least 2 years. During the 2011 through 2017 time period we examined, approximately 65 percent of the total number of persons with disabilities hired at SBA during that time stayed in the federal government for less than 1 year, as shown in figure 8. During that same time period, approximately 55 percent of the total number of persons without disabilities that were hired at SBA stayed for less than 1 year of service, as shown in figure 9. The data shown in Figures 8 and 9 taken in context together indicate that retention at SBA during this time period was generally consistent for persons both with and without disabilities. These departures may be explained, in part, by the proportion of employees hired into temporary positions who therefore were not necessarily expected to stay on the job for a longer duration, or by employees who did not meet performance standards. For example, SBA staff said that on average, 45 percent of SBA’s workforce is comprised of temporary employees hired by the agency’s Office of Disaster Assistance during a disaster. As such, SBA expects turnover among those hires, including employees with disabilities. Tables 8 and 9 show the results of our analysis of employee retention at SBA by occupational category and GS level for individuals hired in fiscal years 2011 through 2015 and stayed for at least 2 years. During the 2011 through 2017 time period we examined, approximately 33 percent of the total number of persons with disabilities hired at SSA during that time stayed in the federal government for less than 1 year, as shown in figure 10. During that same time period, approximately 25 percent of the total number of persons without disabilities that were hired at SSA stayed for less than 1 year of service, as shown in figure 11. The data shown in figures 10 and 11 taken in context together indicate that retention at SSA during this time period was generally consistent for persons both with and without disabilities. These departures may be explained, in part, by the proportion of employees hired into temporary positions who therefore were not necessarily expected to stay on the job for a longer duration, or by employees who did not meet performance standards. Tables 10 and 11 show the results of our analysis of employee retention at SSA by occupational category and GS level for individuals hired in fiscal years 2011 through 2015 and stayed for at least 2 years. Yvonne D. Jones at (202) 512-6806 or JonesY@gao.gov. In addition to the contact named above, Leah Querimit Nash (Assistant Director), Arpita Chattopadhyay, Anthony Patterson, and Erik Shive made key contributions to this report. In addition, Michael Bechetti, Elizabeth Curda, Karin Fangman, Rob Gebhart, Michele Grgich, Amalia Konstas, Serena Lo, Art Merriam, and Sharon Miller made contributions to this report.", "summary": "Federal agencies are required to provide equal opportunity to qualified individuals with disabilities in all aspects of federal employment. GAO was asked to examine agencies' efforts to increase the employment of individuals with disabilities. Among other objectives, this report examines: (1) the extent to which agencies met the 2010 federal goal to hire an additional 100,000 individuals with disabilities by 2015, and the retention rates of those employees between 2011 and 2017; and (2) practices selected agencies used to increase hiring and retention of individuals with disabilities. GAO analyzed data and documents from OPM and interviewed agency officials. GAO interviewed officials from DOJ, SBA, and SSA about their efforts to enhance employment opportunities for disabled persons. GAO selected these three agencies because they represent a range of agency size and relatively high or low percentages of total employees with disabilities. Approximately 143,600 persons with disabilities were hired during 2011 through 2015—plus an additional 79,600 hires in 2016 and 2017—across the 24 Chief Financial Officers Act agencies, exceeding the stated goal of 100,000 by 2015. About 39 percent of individuals with disabilities hired during 2011 through 2017 stayed less than 1 year and approximately 60 percent stayed less than 2 years. Of the total individuals without disabilities hired during that same time period, approximately 43 percent stayed less than 1 year and approximately 60 percent stayed less than 2 years. Although targeted data tracking and analyses could help pinpoint root causes contributing to departure rates, the Office of Personnel Management (OPM) does not track or report retention data on disabled employees. Doing so, and making such data available to agencies would facilitate more comprehensive analyses of the retention of employees with disabilities and identify needed improvements. Officials at three agencies GAO examined—Department of Justice (DOJ), Small Business Administration (SBA), and Social Security Administration (SSA)—used various practices to increase hiring, such as training staff on Schedule A—a commonly used hiring authority to employ individuals with disabilities. However, the agencies neither assess the impact of training nor how it relates to contributing to performance goals of increasing the number of disabled hires. Agencies are expected to track performance related to providing reasonable accommodations. The selected agencies reported having processes in place for receiving reasonable accommodations requests, but only SSA has procedures for obtaining feedback from employees after an accommodation is provided. Without such feedback, DOJ and SBA are limited in their ability to assess the continued effectiveness of reasonable accommodations provided to employees. GAO is making 6 recommendations: OPM should track and report retention data; DOJ, SBA, and SSA should assess training impacts; and DOJ and SBA should obtain employee feedback on reasonable accommodations. OPM and SSA concurred with GAO's recommendations; SBA concurred with one and partially concurred with one recommendation; DOJ did not agree or disagree with the recommendations. GAO continues to believe all recommendations are warranted.", "document_type": "gao"}
{"report": "Oil and gas exploration and production involves disturbing lands in several ways. For example, when operators drill oil and gas wells, they typically remove topsoil and construct a well pad, where the drilling rig will be located. Other equipment on-site can include generators and fuel tanks. In addition, reserve pits are often constructed to store or dispose of water, mud, and other materials that are generated during drilling and production, and roads and access ways are often built to move equipment to and from the wells. Once wells cease production, which may occur many decades after they are drilled, they can become inactive. Inactive wells have the potential to create physical and environmental hazards if operators do not properly reclaim them, a process that may involve plugging the well, removing structures, and reshaping and revegetating the land around the wells. For example, inactive wells that are not properly plugged can leak methane into the air or contaminate surface water and groundwater. Well sites that are not properly reclaimed can contribute to habitat fragmentation and soil erosion, and equipment left on-site can interfere with agricultural land use and diminish wildlife habitat. Costs for well reclamation vary widely and are affected by factors such as the depth of the well. Although BLM does not estimate reclamation costs for all wells, it has estimated reclamation costs for thousands of wells whose operators have filed for bankruptcy. Based on our analysis of these estimates, we identified two cost scenarios: low-cost wells typically cost about $20,000 to reclaim, and high-cost wells typically cost about $145,000 to reclaim. As shown in figure 1, BLM regulations or policies outline how BLM is to initially collect bonds from operators, review bonds, and ultimately return the bond to the operator or use it to cover costs of reclamation. Bonds collected from operator. BLM regulations require operators to submit a bond to ensure compliance with all of the terms and conditions of the lease, including, but not limited to, paying royalties and reclaiming wells. BLM regulations generally require operators to have one of the following types of bond coverage: individual lease bonds, which cover all of an operator’s wells under one lease; statewide bonds, which cover all of an operator’s leases and operations in one state; or nationwide bonds, which cover all of an operator’s leases and operations nationwide. (See figure 2.) BLM can accept two types of bonds: surety bonds and personal bonds. A surety bond is a third-party guarantee that an operator purchases from a private insurance company approved by the Department of the Treasury. The operator pays a premium to the surety company that can vary depending on various factors, including the amount of the bond and the assets and financial resources of the operator. If operators fail to reclaim their wells, the surety company is responsible for paying BLM up to the amount of the bond to help offset reclamation costs. A personal bond must be accompanied by one of the following financial instruments: certificates of deposit issued by a financial institution whose deposits are federally insured, granting the Secretary of the Interior full authority to redeem it in case of default in the performance of the terms and conditions of the lease; cashier’s checks; negotiable Treasury securities, including U.S. Treasury notes or bonds, with conveyance to the Secretary of the Interior of full authority to sell the security in case of default in the performance of the lease’s terms and conditions; or irrevocable letters of credit that are issued for a specific term by a financial institution whose deposits are federally insured and meet certain conditions and that identify the Secretary of the Interior as sole payee with full authority to demand immediate payment in case of default in the performance of the lease’s terms and conditions. BLM bond reviews. BLM regulations provide flexibility to increase bonds above minimums and require increases above minimum amounts if operators meet certain criteria. Specifically, BLM regulations require BLM to increase the bond amount when an operator who applies for a new drilling permit had previously failed to reclaim a well in a timely manner. For such an operator, BLM must require a bond in an amount equal to its cost estimate for reclaiming the new well if BLM’s cost estimate is higher than the regulatory minimum amount. BLM regulations also authorize increases in the bond amount—not to exceed the estimated cost of reclamation and any royalties or penalties owed—whenever the authorized officer determines that the operator poses a risk due to factors such as that the expected reclamation costs exceed the present bond. In response to our previous recommendation in 2011 that BLM develop a comprehensive strategy to revise its bond adequacy review policy to more clearly define terms and conditions that warrant a bond increase, BLM issued a bond adequacy review policy in July 2013, Instruction Memorandum 2013-151. The policy contained directives for conducting reviews when bonds meet certain criteria. Specifically, the 2013 bond adequacy review policy called for field offices to, among other things, review each bond at least every 5 years to determine whether the bond value appropriately reflected the level of potential risk posed by the operator. If it did not, authorized officers were to propose an increase (or decrease) in the bond value. In November 2018, BLM issued a revised bond adequacy review policy, Instruction Memorandum 2019-014, which supersedes the 2013 policy. The 2018 policy continues to call for field offices to review each bond at least every 5 years, but it revised the point system worksheet that field offices are to use when determining whether a bond increase (or decrease) is warranted. Also, in response to our 2018 recommendation that BLM ensure that the reviews of nationwide and statewide bonds reflect the overall risk presented by operators, the 2018 policy calls for additional coordination between BLM headquarters, state offices, and field offices when reviewing nationwide and statewide bonds. BLM returns or uses bond. If operators reclaim their wells, BLM returns the bond to the operator. Many decades may pass between when BLM collects a bond and when it is returned. If operators do not reclaim their wells, BLM may redeem the certificate of deposit, cash the check, sell the security, or make a demand on the letter of credit to pay the reclamation costs. Liability for reclaiming a well on onshore federal lands can fall to either the lease holder or the operator, and BLM may also hold past owners or operators liable. The liability for past owners or operators extends only to reclamation obligations that accrued before BLM approved the transfer of their lease to a subsequent lessee. They are not liable for reclamation and lease obligations incurred after that transfer is approved. Based on our review of BLM data, the value of bonds held by BLM for oil and gas operations on a per-well basis were slightly lower in 2018 as compared to 2008. Although the total value of bonds held by BLM for oil and gas operations was higher in 2018 than in 2008 (about $204 million compared to about $188 million, in 2018 dollars), the average bond value per well was slightly lower because the number of wells on federal land was also higher in 2018 than in 2008 (96,199 wells compared to 85,330). Specifically, in 2008, BLM held bonds worth an average of $2,207 per well in 2018 dollars.23, BLM held bonds worth an average of $2,122 per well in 2018, a decrease of 3.9 percent as compared to 2008 (see table 1). BLM bonds do not typically cover an individual well; however, we calculated the average bond value on a per-well basis (bond amount divided by the number of wells covered by the bond) to compare the value over time adjusted for the increased number of wells. When reporting on all wells, we calculated the average bond value per well as the aggregate value of all BLM bonds divided by the total number of producible well bores. Appendix I provides additional information on our scope and methodology. category for bonds that were linked to wells in the data. We found that, on average, as of 2018 an individual lease bond covered about 10 wells, a statewide bond covered about 49 wells, and a nationwide bond covered 374 wells. However, some bonds cover more than the typical number of wells and some fewer. As of 2018, individual lease bonds had the highest average bond value per well at $2,691, and nationwide bonds had the lowest average bond per well value at $890. Statewide bonds had an average bond value per well of $1,592. The share of the total value of bonds held by BLM that are individual lease, statewide, or nationwide bonds differed in 2018 from 2008 (see Figure 3). The share of individual lease bonds was slightly higher in 2018 as compared to 2008 (about 8 percent in 2008 and about 9 percent in 2018). In 2008, statewide bonds represented about 80 percent (approximately $130 million) of the total value of bonds. In 2018, statewide bonds represented about 59 percent of total bond value (approximately $120 million), but this category still represented the largest share of total bond value. In contrast, nationwide bonds were a lower share of total bond value in 2008 (about 6 percent, approximately $10.2 million) than in 2018 (30 percent, approximately $61.8 million). BLM officials told us that changes in the composition of the oil and gas industry may have contributed to these changes in the composition of bonds. In particular, officials said some larger companies may have expanded their operations in recent years, sometimes acquiring smaller companies. Large companies with expansive operations are more likely than small companies to have nationwide bonds because such bonds can cover operations in multiple states, which statewide and individual lease bonds do not. Therefore, an industry shift to larger companies would tend to increase the share of nationwide bonds. Bonds do not provide sufficient financial assurance to prevent orphaned wells for several reasons. First, BLM has identified new orphaned wells— wells whose bonds were not sufficient to pay for needed reclamation when operators or other parties failed to reclaim them. As we reported in May 2018, BLM does not track the number of orphaned wells over time and so cannot identify how many wells became orphaned over specific time frames. However, our analyses of BLM’s orphaned well lists from different years have shown that BLM has continued to identify new orphaned wells since 2009. We reported in January 2010 that BLM identified 144 orphaned wells in 2009. Then, in May 2018, we reported that BLM identified 219 orphaned wells in July 2017—an increase of 75 orphaned wells. In April 2019, BLM provided a list of 296 orphaned wells that included 89 new wells that were not identified on the July 2017 list. Bonds are not sufficient to prevent orphaned wells in part because they do not reflect full reclamation costs for the wells they cover. Bonds that are high enough to cover all reclamation costs provide complete financial assurance to prevent orphaned wells because, in the event that an operator does not reclaim its wells, BLM can use the bond to pay for reclamation. On the other hand, bonds that are less than reclamation costs may not create an incentive for operators to promptly reclaim wells after operations cease because it costs more to reclaim the wells than the operator could collect from its bond. We analyzed bonds that are linked to wells in BLM’s data, and found that most of these bonds would not cover reclamation costs for their wells. Specifically, we compared the average bond coverage available for these wells to the two cost scenarios we described above. About 84 percent of these bonds—covering 99.5 percent of these wells—would not fully cover reclamation costs under a low-cost scenario (these bonds have an average value per well of less than $20,000). Less than 1 percent of bonds—covering less than 0.01 percent of these wells—would be sufficient to reclaim all the wells they cover if they were high cost (these bonds have an average value per well of $145,000 or more). The remaining bonds—about 16 percent—have average bond values per well of between $20,000 and less than $145,000. The majority of bond values do not reflect reclamation costs in large part because most bonds—82 percent—remain at their regulatory minimum values. These regulatory minimums are not reflective of reclamation costs for a number of reasons: Regulatory bond minimums have not been adjusted since the 1950s and 1960s to account for inflation. As shown in figure 4, when adjusted to 2018 dollars, the $10,000 individual lease bond minimum would be about $66,000, the $25,000 statewide bond minimum would be about $198,000, and the $150,000 nationwide bond minimum would be about $1,187,000. Bond minimums are based on the bond category and do not adjust with the number of wells they cover, which can vary greatly. According to BLM’s data, in 2018 the number of wells covered by a single bond ranged from one well to 6,654 wells. On average, a single bond covered about 68 wells. As wells are added to a bond, the total associated reclamation cost increases even if the bond value does not. A bond that increases with each additional well it covers and then decreases as wells are reclaimed could increase the financial incentive for operators to reclaim their wells in a timely manner. This is because operators would have to contribute additional bond value or would recover some bond value when they add or reclaim a well, respectively. Currently, bond minimums do not automatically adjust in this manner and therefore provide limited financial incentives for an operator to reclaim wells in a timely manner. Bond minimums do not reflect characteristics of individual wells such as depth or location, but such characteristics can affect reclamation costs, according to BLM officials. Wells are being drilled deeper than in the past; in 1950, well depth averaged about 3,700 feet, and in 2008, it averaged about 6,000 feet. Newer wells may be drilled 10,000 feet vertically. Officials from one BLM field office told us they assume a cost of $10 per foot of well depth to plug a well, so as wells are drilled deeper, plugging costs typically increase proportionally. Additionally, the location of some wells makes them more expensive to reclaim. For example, BLM officials told us about several wells that may cost three times more to reclaim than other nearby wells because they are located in the middle of a river, making them hard to reach. In addition to BLM having identified orphaned wells over the last decade, we identified inactive wells at increased risk of becoming orphaned and found their bonds are often not sufficient to reclaim the wells. Our analysis of BLM bond value data and Office of Natural Resources Revenue production data showed a significant number of inactive wells remain unplugged and could be at increased risk of becoming orphaned. Specifically, we identified 2,294 wells that may be at increased risk of becoming orphaned because they have not produced since June 2008 and have not been reclaimed. Further, for a majority of these at-risk wells, their bonds are too low to cover typical reclamation costs for just these at-risk wells. Our analysis of oil and gas production data showed these wells have not produced oil or gas or been used in other ways, such as serving as injection wells, since at least June 2008, when oil and gas prices were at or near record highs. Given that the Energy Information Administration projects oil and natural gas prices will remain at levels significantly below the 2008 highs through 2050, it is unlikely price will motivate operators to reopen these wells. Some of these wells have been inactive for far longer. Since these at-risk wells are unlikely to produce again, an operator bankruptcy could lead to orphaned wells unless bonds are adequate to reclaim them. If the number of at-risk wells is multiplied by our low-cost reclamation scenario of $20,000, it implies a cost of about $46 million to reclaim these wells. If the number of these wells is multiplied by our high-cost reclamation scenario of $145,000, it implies a cost of about $333 million. When we further analyzed the available bonds for these at-risk wells, we found that most of these wells (about 77 percent) had bonds that would be too low to fully reclaim the at- risk wells under our low-cost scenario. More than 97 percent of these at- risk wells have bonds that would not fully reclaim the wells under our high-cost scenario. BLM has a policy for reviewing the adequacy of bonds but has not been able to consistently secure bond increases when needed, and this policy has not resulted in bonds that would be adequate to reclaim most wells. BLM’s bond adequacy review policy calls for field office staff to review oil and gas bonds at least every 5 years to determine whether the bond amount appropriately reflects the level of potential risk posed by the operator. However, according to BLM documentation, its offices did not secure about 84 percent of the proposed bond increases in fiscal years 2016 and 2017. BLM officials at one field office and one state office noted it is difficult to secure increases from bond reviews when firms are already in difficult financial situations. In November 2018, BLM updated its bond adequacy review policy and called for the agency to focus on securing bond increases from operators that show the highest risk factors. BLM’s updated policy more explicitly lays out steps to secure bond increases, including that BLM should not approve new applications to drill from an operator while waiting for a bond increase. The new policy also gives BLM officials discretion to not pursue a bond increase after considering other priorities demanding staff time and workload. It is unclear whether the update will improve BLM’s ability to secure bond increases, as it may not address the underlying challenge of attempting to increase bonds from operators who are already in a difficult financial position. While BLM’s federal oil and gas bond minimums do not sufficiently reflect the costs of well reclamation, requirements for bond amounts for other federal mining and energy development activities account for potential reclamation costs to some extent. For example, for bonds for surface coal mining and hardrock mining on federal lands, the Department of the Interior requires bond amounts based on the full estimated cost of reclamation. For grants of federal rights-of-way for wind and solar energy development in designated leasing areas, BLM requires bonds based on a minimum amount per wind turbine or per acre of solar. For such grants in all other areas, the bonds are based on the estimated cost of reclamation but cannot be less than the per-turbine or per-acre amounts previously mentioned. Additionally, some states have minimum bond requirements for oil and gas wells on lands in the state that, unlike federal bond minimums, adjust with the number of wells they cover or the characteristics of the wells, or both. For example, Texas and Louisiana offer operators with wells on lands in those states the choice of a bond based on total well depth or based on the number of wells. Specifically, the Texas Railroad Commission lets operators choose bonds based on either the total depth of all wells on lands in the state multiplied by $2 per foot, or minimums based on the number of wells covered. If operators choose the latter, the bond for 0 to 10 wells is $25,000; the bond for 11 to 99 wells is $50,000; and the bond for 100 or more wells is $250,000. In Louisiana, the Office of Conservation offers operators with wells on lands in the state the choice of a bond based on total well depth or based on the number of wells. Louisiana further specifies a multiplier that varies depending on the total depth of the well. For example, the bond calculation is $2 per foot for wells less than 3,000 feet deep, $5 per foot for wells from 3,001 to 10,000 feet deep, and $4 per foot for wells 10,001 feet deep or deeper. Operators in Louisiana can alternatively choose to follow a system based on number of wells, with a minimum bond for 10 or fewer wells set at $50,000, a minimum bond for 11 to 99 wells set at $250,000, and a minimum bond for 100 or more wells set at $500,000. Pennsylvania’s Department of Environmental Protection requires bonds for unconventional wells that vary based on the number of wells and well bore length. The Mineral Leasing Act of 1920, as amended, requires federal regulations to ensure that an adequate bond is established before operators begin surface-disturbing activities on any lease, to ensure complete and timely reclamation of the lease tract as well as land and surface waters adversely affected by lease operations. The Mineral Leasing Act of 1920 does not require that BLM set bonds at full reclamation costs. However, the gap between expected reclamation costs and minimum bond amounts has grown over time because the minimums have not been adjusted since they were established in the 1950s and 1960s, whereas reclamation costs have increased due to inflation and the changing characteristics of wells being drilled. In the absence of bond levels that more closely reflect expected reclamation costs, such as by increasing regulatory minimums and incorporating consideration of the number of wells on each bond and their characteristics, BLM will continue to face risks that its bonds will not provide sufficient financial assurance to prevent orphaned wells. In particular, adjusting bond minimums so that bonds more closely reflect expected reclamation costs up front could help decrease the need for bond increases later when companies are potentially in financial distress. In addition to fulfilling its responsibility to prevent new orphaned wells, it falls to BLM to reclaim wells that are currently orphaned, and BLM has encountered challenges in doing so. We reported in May 2018 that 13 BLM field offices identified about $46.2 million in estimated potential reclamation costs associated with orphaned wells and with inactive wells that officials deemed to be at risk of becoming orphaned. There is also a risk more wells will become orphaned in coming years, as we described above. Based on the most recent orphaned well lists we received from BLM, 51 wells that BLM identified in 2009 as orphaned had not been reclaimed as of April 2019. The Energy Policy Act of 2005 (EPAct 2005) directs Interior to establish a program that, among other things, provides for the identification and recovery of reclamation costs from persons or other entities currently providing a bond or other financial assurance for an oil or gas well that is orphaned, abandoned, or idled. One way in which BLM may be able to accomplish this is through the imposition of user fees. In 2008, we found that well-designed user fees can reduce the burden on taxpayers to finance those portions of activities that provide benefits to identifiable users. Further, according to Office of Management and Budget guidance, it may be appropriate for an agency to request authority to retain the fee revenue if the user fees offset the expenses of a service that is intended to be self-sustaining. The volume of drilling applications and inactive wells provide an opportunity to fund reclamation costs. According to BLM data, the agency processes more than 3,500 applications to drill each year, on average, and has over 14,000 inactive wells. Based on our calculations, a separate fee of about $1,300 charged at the time a drilling application is submitted (in addition to the current drilling application filing fee, which is $10,050), or an annual fee of less than $350 for inactive wells could generate enough revenue to cover, in a little over a decade, the entire $46 million potential reclamation costs field offices identified to us. In commenting on a draft of this report, BLM stated that it does not have the authority to seek or collect fees from lease operators to reclaim orphaned wells. Developing a mechanism to obtain funds from operators to cover the costs of reclamation, consistent with EPAct 2005, could help ensure that BLM can completely and timely reclaim wells without using taxpayer dollars. Other federal programs, including other BLM programs, collect fees from users to fund reclamation activities. For example, the federal government collects fees from mining companies to reclaim abandoned mines. Specifically, the federal abandoned mine reclamation program is funded in part by fees on coal production. We reported in March 2018 that the program had spent about $3.9 billion to reclaim abandoned mine lands since the program’s creation in 1977. Additionally, some states with oil and gas development have dedicated funds for reclaiming orphaned wells. In Wyoming, the state’s Oil and Gas Conservation Commission’s Orphan Well Program reclaims orphaned wells on state or private lands for which bonds and operator liability are unavailable or insufficient to fund reclamation. The program is funded through a conservation tax assessed on the sale of oil and natural gas produced in Wyoming. Through this program, the Wyoming Oil and Gas Conservation Commission has reclaimed approximately 2,215 wells since 2014, according to a Commission official. Similarly, in Arkansas, operators make annual payments to its abandoned well plugging fund based on the number of wells and permits they have, on a sliding scale. For example, at the low end, operators with one to five wells or permits pay $100 per well, and at the high end, operators with over 300 wells or permits pay $4,000 per operator. The Arkansas fund was used to reclaim 136 wells in fiscal years 2016 through 2018, according to an official with the state’s Oil and Gas Commission. Virginia’s Orphaned Well Fund is funded through a $200 surcharge on each permit application. The fund is administered by the Virginia Division of Gas and Oil, which prioritizes wells to reclaim according to their condition and potential threat to public safety and the environment. BLM oversees private entities operating thousands of oil and gas wells on leased federal lands and has taken steps over the years to strengthen its management of the potential liability that oil and gas operations represent should operators not fully reclaim wells and return lands to their original condition when production ceases. For example, the agency’s 2013 bond adequacy review policy outlined how bonds were to be reviewed every 5 years and bond amounts adjusted depending on risks presented by operators. However, we found average bond values were slightly lower in 2018 as compared to 2008 and BLM has not obtained bond increases for the majority of instances in which its reviews identify that increases are needed. Instead, most bonds are at their regulatory minimum values, which are not sufficient to cover reclamation costs incurred by BLM. Without adjusting bond levels to more closely reflect expected reclamation costs—such as by considering the effects of inflation, the number of wells covered by a single bond, and the characteristics of those wells—BLM faces ongoing risks that not all wells will be completely and timely reclaimed, resulting in additional orphaned wells. Further, BLM faces a backlog of orphaned wells to reclaim—with 51 dating back at least 10 years. Unlike some other federal and state programs that obtain funds from industry through fees or dedicated funds, BLM does not do so for reclaiming orphaned wells. According to BLM, it does not have the authority to seek or collect fees from lease operators to reclaim orphaned wells. Authorizing and requiring the implementation of a mechanism to obtain funds from oil and gas operators to cover the costs of reclamation could help ensure BLM can completely and timely reclaim wells. Congress should consider giving BLM the authority to obtain funds from operators to reclaim orphaned wells, and requiring BLM to implement a mechanism to obtain sufficient funds from operators for reclaiming orphaned wells. (Matter for Consideration 1) The Director of BLM should take steps to adjust bond levels to more closely reflect expected reclamation costs, such as by increasing regulatory minimums to reflect inflation and incorporating consideration of the number of wells on each bond and their characteristics. (Recommendation 1) We provided a draft of this product to BLM for comment. In its written comments, reproduced in appendix II, BLM concurred with the recommendation. BLM stated that it is committed to ensuring that its field offices continue to review oil and gas bonds at least every 5 years, or earlier when warranted, and noted its November 2018 Instruction Memorandum 2019-014 updated its bond review policy. BLM further stated that, while the adjustment of bond values may not reflect the inflation index, the policy is intended to increase bond amounts while fostering an environment conducive to BLM’s leasing operations. As we point out in this report, BLM has historically had difficulties securing bond increases through bond reviews, and so additional steps may be needed to adjust bond levels to more closely reflect expected reclamation costs. In the draft we provided to BLM for comment, we included a recommendation that the Director of BLM should take steps to obtain funds from operators for reclaiming orphaned wells. BLM did not concur with this recommendation, saying it does not have the authority to seek or collect fees from lease operators to reclaim orphaned wells. We continue to believe a mechanism for BLM to obtain funds from oil and gas operators to cover the costs of reclamation for orphaned wells could help ensure BLM can completely and timely reclaim these wells, some of which have been orphaned for at least 10 years. We have therefore instead made a matter for Congressional consideration. BLM also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Interior, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report (1) describes the value of bonds for oil and gas wells in 2018 compared to 2008, and (2) examines the extent to which the Bureau of Land Management’s (BLM) bonds ensure complete and timely reclamation and thus prevent orphaned wells. To describe the value of bonds for oil and gas wells in 2018 compared to 2008, we analyzed oil and gas well data from BLM’s Automated Fluid Minerals Support System (AFMSS) as of May 2018 and data from BLM’s Legacy Rehost 2000 (LR2000) system on bonds as of May 2018. Bond data we reviewed included the bond category (e.g., individual lease or nationwide) and bond value. We compared these data to data obtained from the same systems for 2008 and reported by GAO in 2010. We matched the May 2018 data from the two systems based on the bond number—a variable in both systems—to identify how many wells were covered by each bond and to determine the average bond value per well for each bond category. To assess the reliability of AFMSS and LR2000 data elements, we reviewed agency documents, met with relevant agency officials, and performed electronic testing. We found these data to be sufficiently reliable for our purposes. We also interviewed BLM headquarters officials to understand why bond composition may have changed over time. To report on the number of bonded wells held by BLM, we used a published BLM value for producible well bores—wells capable of production—which should represent a lower bound on the number of bonded wells in September 2018 because some wells may be plugged or temporarily incapable of production but would still require a bond if the surrounding site had not been fully reclaimed. To determine the average value of bonds per well in 2018, we divided the total value of all bonds held by BLM by the total number of producible well bores. To examine the extent to which BLM’s bonds ensure complete and timely reclamation and prevent orphaned wells, we conducted the following analyses: Reclamation cost scenarios: To determine whether bonds are sufficient to cover potential reclamation costs for the wells they cover, we identified typical high- and low-cost scenarios for well reclamation (including plugging the well and reclaiming the surrounding well site) and compared those scenarios to the average bond value available per well. To determine high- and low-cost reclamation scenarios, we analyzed BLM’s well reclamation cost estimates on proofs of claim submitted to the Department of Justice from calendar year 2016 through May 2018. These 59 proofs of claim listed estimated reclamation costs for 8,664 well sites. We calculated the average reclamation cost per well for each individual proof of claim by dividing the total dollar value claimed for reclamation liability (actual liability plus potential liability) by the total number of wells listed in each proof of claim document. We found the average reclamation cost estimates for each proof of claim have a bimodal distribution, meaning that data are clustered around two distinct cost levels, rather than clustered around a single average cost. As a result, we determined that using two separate measures that indicate typical values for separate groups of low-cost and high-cost wells would provide more meaningful statistics about cost. We therefore selected reclamation costs of $20,000 for the low-cost reclamation scenario and $145,000 for the high-cost scenario based on the 25th and 75th percentiles of the distribution of average estimated reclamation cost per proof of claim, weighted by the number of wells on each proof of claim. Bond value per well: To determine the average bond value available per well, we analyzed bonds listed in LR2000 that were tied to wells listed in AFMSS using the bond number—a variable in both systems. We found that 1,547 out of the 3,357 unique bond numbers in LR2000 had wells tied to them in AFMSS. These 1,547 bonds covered about 80 percent of the wells in AFMSS. The other 20 percent of wells in AFMSS either did not list a bond number, or the bond number listed was not in LR2000. For each bond in LR2000 covering wells in AFMSS, we calculated the bond available per well as the bond value divided by the number of wells it covers. We then compared the bond values per well against both high ($145,000 per well) and low ($20,000 per well) reclamation cost scenarios to identify which bonds would be adequate to reclaim all the wells they covered under different cost scenarios. If AFMSS bond information was incomplete, it is possible that there are more wells covered by bonds than we were able to identify—and therefore the bond value per well would be lower than we found. At-risk wells: To identify wells that may be at greater risk of becoming orphaned and determine whether their bonds are sufficient to cover potential reclamation costs, we used well production data from the Office of Natural Resources Revenue’s Oil and Gas Operations Report (OGOR) as of June 2017 and bond values from LR2000. First, we defined wells as “at risk of becoming orphaned” if they met several criteria. Specifically, we identified wells that (1) had recent OGOR reports (on or after March 2017); (2) had not been used productively from at least June 2008 through the most recent record (meaning the well did not report producing any volume of oil or gas during this timeframe, nor were any volume of water or materials injected into the well during this timeframe); (3) were not being used as a monitoring well in the most recent record, which we considered a productive use; and (4) had not been plugged and abandoned. We selected June 2008 as the cutoff date for productivity because in June and July of 2008, oil and gas prices hit peaks that have not since been reached again, and which the Energy Information Administration does not expect prices to reach again through at least 2050. We believe our analysis is a conservative estimate of wells at greater risk, in part because we did not include wells that produced when prices were at their peaks and stopped producing soon afterward and may be unlikely to produce in the future unless prices reach the same peaks again. In addition, our lower-bound estimate does not include some coalbed methane wells that have been inactive for less than 9 years but are unlikely to produce at current prices because of the relatively higher cost of coalbed methane production. We also excluded wells that reported any volume of oil or gas production or water injection since June 2008, although some very low-producing wells may also be at risk of becoming orphaned. Bond value for at-risk wells: To calculate the average bond value per at-risk well, we identified bonds listed in LR2000 that were tied to at- risk wells in AFMSS to determine the value of bonds available to reclaim these at-risk wells if needed. We identified 2,041 of the 2,294 at-risk wells were linked to bonds. For each bond, we divided the bond value by the number of at-risk wells it covered to determine the bond amount per at-risk well. In cases in which an at-risk well was linked to more than one bond, we additionally calculated the average of the bond value per at-risk well for each bond linked to the well. To determine the sufficiency of bonds for at-risk wells, we identified the number of wells with an average bond value per at-risk well equal to or greater than $20,000 (low cost reclamation scenario) or $145,000 (high cost reclamation scenario). Orphaned wells: We compared three lists of orphaned wells based on data provided by BLM in 2009, July 2017, and April 2019. The 2009 data are from our January 2010 report, which used Orphaned Well Scoring Checklists that list information such as the well’s name and location. The July 2017 data are from our May 2018 report, which used an orphaned well list generated through a query of AFMSS by BLM. The April 2019 list was generated through a query of an updated version of AFMSS known as AFMSS 2. We compared the lists to identify how many wells that were on the 2009 list remained on the 2019 list, and how many wells that were on the 2017 list were on the 2019 list. To assess the reliability of the AFMSS, LR2000, and OGOR data elements we used, we reviewed agency documents, met with relevant agency officials, and performed electronic testing. We found these data elements to be sufficiently reliable for our purposes. Similarly, to assess the reliability of the 2019 orphaned well list, we reviewed agency documents and met with relevant agency officials. Though we identified shortcomings with data on orphaned wells, we nevertheless found these data to be sufficiently reliable for the purpose of describing the orphaned wells BLM has identified. To assess the reasonableness of proofs of claim data, we interviewed relevant agency officials and reviewed agency documents. To understand how BLM manages bonds, we reviewed BLM’s policies and interviewed officials from four BLM state offices and four BLM field offices. We selected these state and field offices because, according to AFMSS data, they were responsible for managing the largest numbers of wells on federal land. These BLM state offices were California, New Mexico, Utah, and Wyoming. These BLM field offices were Bakersfield, Buffalo, Carlsbad, and Farmington. We also interviewed officials from BLM’s headquarters office in Washington, D.C. Findings from the selected BLM offices cannot be generalized to officials we did not interview but provide a range of views. To understand how some states with oil and gas development on state lands set minimum bonds and fund orphaned well reclamation, we contacted officials from oil and gas oversight agencies in Arkansas, Louisiana, Pennsylvania, Texas, Virginia, and Wyoming. We conducted this performance audit from January 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Quindi Franco (Assistant Director), Marietta Mayfield Revesz (Analyst-in-Charge), Marie Bancroft, William Gerard, Cindy Gilbert, Gwen Kirby, Joe Maher, Shaundra Patterson, Dan Royer, and Jerry Sandau made key contributions to this report.", "summary": "The oil and natural gas produced from wells on federal lands are important to the U.S. energy supply and bring in billions in federal revenue each year. However, when wells are not properly managed, the federal government may end up paying to clean up the wells when they stop producing. Specifically, wells on federal lands that an operator does not reclaim and for which there are no other liable parties fall to BLM to reclaim (restore lands to as close to their original natural states as possible). These wells become orphaned if the operator's bond held by BLM is not sufficient to cover reclamation costs. BLM regulations set minimum bond values at $10,000 for all of an operator's wells on an individual lease, $25,000 for all of an operator's wells in a state, and $150,000 for all of an operator's wells nationwide. GAO was asked to review the status of oil and gas bonding for federal lands. This report (1) describes the value of bonds for oil and gas wells in 2018 compared to 2008, and (2) examines the extent to which BLM's bonds ensure complete and timely reclamation and thus prevent orphaned wells. GAO analyzed agency data on bonds and wells and interviewed BLM officials. The average value of bonds held by the Bureau of Land Management (BLM) for oil and gas wells was slightly lower on a per-well basis in 2018 ($2,122) as compared to 2008 ($2,207), according to GAO's analysis of BLM data. The total value of bonds held by BLM for oil and gas operations increased between these years, as did the number of wells on federal land. Bonds held by BLM have not provided sufficient financial assurance to prevent orphaned oil and gas wells (wells that are not reclaimed by their operators and, among other things, whose bonds were not sufficient to cover remaining reclamation costs, leaving BLM to pay for reclamation). Specifically, BLM identified 89 new orphaned wells between July 2017 and April 2019, and BLM offices identified to GAO about $46 million in estimated potential reclamation costs associated with orphaned wells and with inactive wells that officials deemed to be at risk of becoming orphaned in 2018. In part, bonds have not prevented orphaned wells because bond values may not be high enough to cover the potential reclamation costs for all wells under a bond, as may be needed if they become orphaned. GAO's analysis indicates that most bonds (84 percent) that are linked to wells in BLM data are likely too low to reclaim all the wells they cover. Bonds generally do not reflect reclamation costs because most bonds are set at their regulatory minimum values, and these minimums have not been adjusted since the 1950s and 1960s to account for inflation (see figure). Additionally, these minimums do not account for variables such as number of wells they cover or other characteristics that affect reclamation costs, such as well depth. Without taking steps to adjust bond levels to more closely reflect expected reclamation costs, BLM faces ongoing risks that not all wells will be completely and timely reclaimed, as required by law. It falls to BLM to reclaim orphaned wells, but the bureau does not assess user fees to cover reclamation costs, in part because it believes it does not have authority to do so. Providing such authority and developing a mechanism to obtain funds from operators for such costs could help ensure that BLM can completely and timely reclaim wells. Congress should consider giving BLM the authority to obtain funds from operators to reclaim orphaned wells, and requiring BLM to implement a mechanism to do so. GAO also recommends that BLM take steps to adjust bond levels to more closely reflect expected reclamation costs. BLM concurred. BLM did not concur with a proposed recommendation to develop a mechanism to obtain funds, citing lack of authority. GAO changed it to a matter for Congressional consideration.", "document_type": "gao"}
{"report": "HUD created REAC in 1997 to obtain consistent information on, among other things, the physical condition of its public and multifamily properties. REAC generally inspects properties every 1 to 3 years, using a risk-based schedule (discussed in detail below). REAC developed a standardized protocol to inspect properties, referred to as the Uniform Physical Condition Standards. As part of the protocol, REAC also inspects properties to identify health and safety deficiencies, including exigent health and safety deficiencies, which are life-threatening and require immediate action or remedy (such as exposed electrical wires or blocked access to windows or doors in case of a fire). REAC’s data system automatically generates an overall inspection score for the property from 0 to 100 based on the information an inspector records. At the end of each day of an inspection, an inspector is required to inform a property manager or other representative if the inspection identified exigent health and safety issues. Before releasing the inspection score, REAC reviews the inspection through a quality assurance process to ensure it is accurate. Following verification of the inspection score, REAC releases an inspection report to the property owner or PHA and the relevant HUD program office. The inspection report contains the overall inspection score, as well as more detailed information on physical deficiencies identified during the inspection. REAC primarily uses contractors—who are trained and certified in REAC’s Uniform Physical Condition Standards protocol—to conduct inspections of multifamily and public housing properties. In addition to these contract inspectors, REAC uses quality assurance inspectors, who are HUD employees, to oversee and monitor contract inspectors, as well as to ensure that REAC provides accurate and reliable inspections. Both contract and quality assurance inspectors complete several phases of training on the inspection protocol, including online, classroom, and field- based training. To procure inspections of HUD-assisted properties, REAC primarily uses an auction process to award contracts either to eligible contract inspectors or to companies that employ contract inspectors. This process, called a reverse auction program, occurs at least once a quarter. Contract inspectors or companies bid to inspect properties across the United States and its territories in a web-based auction. At the close of the auction, REAC awards the inspection to whoever bids the lowest price and is eligible to conduct inspections. The contract inspector then schedules and performs the property inspections in accordance with Uniform Physical Condition Standards protocol. According to REAC officials, this process is designed to increase cost savings and small business participation. REAC is situated within PIH. Several departments within REAC are involved in facilitating the physical inspection process: Physical Assessment Subsystem (PASS): PASS has three primary divisions that are responsible for different aspects of the inspection process. The PASS Physical Inspection Operations division coordinates the procurement of inspections. The PASS Quality Assurance division evaluates and monitors REAC’s inspection program to ensure reliable, replicable, and reasonable inspections; trains contract and quality assurance inspectors; and provides technical assistance to HUD-assisted properties and other relevant stakeholders. The PASS Inspector Administration division monitors the performance of inspectors and takes administrative actions, such as decertifying inspectors who do not meet REAC’s standards for inspectors. Research and Development: REAC’s Research and Development division produces data analysis and statistical reports on REAC’s information products (e.g., physical inspection reports and Public Housing Assessment System scores) and assesses these products to ensure they are accurate and valid. REAC is also responsible for evaluating additional conditions, beyond physical conditions, of multifamily and public housing properties. Specifically, REAC evaluates the financial conditions of multifamily properties and assesses the financial and management performance of public housing properties. This performance assessment is conducted through the Public Housing Assessment System. REAC uses several data systems to collect, score, and report on the financial and management conditions of public housing properties, along with evaluating the utilization of property modernization and development funds (capital funds). We describe this process in more detail later in the report. HUD’s PIH, Multifamily Housing, and Departmental Enforcement Center are responsible for ensuring that the owners of REAC-inspected properties (including PHAs) correct the identified physical deficiencies. PIH: This office helps low-income families by providing rental assistance through three programs; our review focuses on physical inspections of the public housing program. In 2018, HUD’s public housing program provided low-rent housing units to over 1 million eligible households. Public housing consists of reduced-rent developments owned and operated by local PHAs and subsidized by the federal government. About 3,300 PHAs own and manage public housing properties. These properties can include high-rise and low-rise buildings and scattered single-family properties, or they can be part of mixed-income housing developments, and they can range in size from fewer than 100 units to more than 30,000 units. PHAs typically have an executive director to manage their operations, as well as a governing board—called a Board of Commissioners—to approve policy, clarify goals, and ensure compliance with federal regulations. PHAs have contracts, called Annual Contributions Contracts, with the federal government. Under the terms of their contracts, PHAs agree to administer their properties according to federal regulations, in exchange for federal funding in the form of operating and capital grants. PIH is organized into six geographic networks, each with several field offices. Multifamily Housing: This office manages HUD’s portfolio of multifamily properties and provides rental assistance through several programs, including Section 8 project-based rental assistance, in which HUD contracts with private property owners to rent housing units to eligible low-income tenants for an income-based rent. Multifamily Housing also oversees the Federal Housing Administration’s multifamily mortgage insurance on loan originations and administers supportive housing for the elderly and programs for persons with disabilities. Collectively, the properties that Multifamily Housing oversees provided affordable rental housing to more than 1.2 million low-income households in 2017. Property owners or management agents of multifamily properties sign business agreements with HUD. Under these agreements, owners or agents agree to administer their properties according to federal rules and regulations, and in exchange, among other benefits, they receive federal assistance through mortgage insurance or housing assistance payments. Multifamily Housing has 12 field offices across five geographic regions. Departmental Enforcement Center: The Departmental Enforcement Center is located within HUD’s Office of General Counsel and works with several of HUD’s program offices, including PIH and Multifamily Housing, to ensure that program funds are used according to federal regulations. These program offices make referrals for the Departmental Enforcement Center to review the financial and other conditions of properties receiving rental assistance from HUD. Based on these reviews, the Departmental Enforcement Center can take various enforcement actions, such as imposing administrative sanctions to bar individuals from participating in HUD programs or civil money penalties for violations. REAC conducts inspections on multifamily and public housing properties using a risk-based schedule defined in federal regulations. According to our analysis of REAC inspection data, REAC conducted 44,486 inspections of multifamily properties and 15,156 inspections of public housing developments from fiscal years 2013 through 2017. For multifamily properties, REAC inspects properties every 1 to 3 years. Generally, properties that receive an inspection score below 80 are inspected within 1 year of the previous inspection; between 80 to 89 within 2 years; and 90 to 100 every 3 years. The inspection frequency for public housing developments varies depending on the overall size of the PHA (that is, the number of units and properties that they manage), an individual housing development’s inspection score, and the PHA’s overall performance on the Public Housing Assessment System. For PHAs with 250 housing units or more, REAC inspects developments every 1 to 3 years, using the same risk- based thresholds as Multifamily Housing. For small PHAs with fewer than 250 units, their score on the Public Housing Assessment System determines the inspection frequency, with higher scores associated with less frequent inspections. However, all developments—regardless of the number of units—that receive an overall performance assessment score (as part of the Public Housing Assessment System) of less than 60 out of 100 are designated to have a physical inspection every year. REAC’s Uniform Physical Condition Standards inspection protocol is designed to help provide assurance that physical deficiencies will be identified at HUD-assisted properties. Under the protocol, contract inspectors inspect five areas of a property using a handheld data collection device to help identify and record deficiencies (see fig. 1). The devices have embedded software that provides step-by-step instructions on conducting the inspection. The software helps to ensure consistency between inspectors and consistency with the protocol, according to REAC staff. The software includes a decision-tree model to guide the inspectors on recording and classifying the severity of deficiencies they identify. For example, if an inspector identifies a deficiency with a door in a dwelling unit, the software will ask the inspector to identify which door has the deficiency and the nature of the deficiency (e.g., door lock does not work). The software then assigns a severity level to the deficiency and, if it is severe enough, requires the inspector to take a photo (see fig. 2). REAC has a number of quality assurance processes intended to ensure that contract inspectors identify deficiencies and conduct quality inspections: Collaborative quality assurance (CQA) review. In CQA reviews, REAC quality assurance inspectors observe contract inspectors to help ensure their inspections are accurate and consistent with protocol. REAC uses CQA reviews to coach contract inspectors to help improve their performance. Post-inspection review process. Completed inspections receive two levels of review by REAC quality assurance staff, who use software that compares certain aspects of the current and previous inspections—such as inspection scores, property profiles (for example, number of units), site measurements, and time taken to complete the inspection—and highlights large variances. Quality control inspection (QCI). If REAC reviewers find large variances in current and previous inspection scores and other aspects, they may reject the inspection and schedule a QCI. The QCI is a review of a previously inspected property to evaluate an inspector’s performance and identify potential weaknesses in the quality of the inspection. This review process requires a REAC quality assurance inspector to conduct a second inspection of the same property, including selecting the same sample of buildings and units of the original inspection. Once the QCI is completed, REAC’s reviewers, in collaboration with REAC’s Research and Development division, identify any deficiencies missed and determine whether the contract inspector was complying with REAC’s physical inspection standards. Property owners may appeal deficiencies REAC has identified during the physical inspection. For example, an owner might appeal a deficiency resulting from a window air conditioner blocking egress by providing evidence that this is permitted by local building code. If the appeal is successful, REAC removes the deficiency and the inspection software updates the score. Contract inspectors, REAC quality assurance inspectors, and representatives of property owner associations with whom we spoke had mixed views on REAC’s inspection process. Participants in three of the five discussion groups we held with contract and quality assurance inspectors said that the inspection process provides a comprehensive review of a property and that the inspection software helps promote consistency in inspections. Likewise, representatives from one property owner association we met with said that the inspection process was more standardized and less subjective than in the past. Representatives from another association said that the inspection process effectively identified deficiencies. However, participants in three of the same five discussion groups with contract and quality assurance inspectors noted inconsistent application of protocols and standards, noting that some cases were unclear and required judgment in identifying deficiencies. REAC’s inspection process has features similar to those of home inspection organizations such as the American Society of Home Inspectors (ASHI) and the International Association of Certified Home Inspectors (InterNACHI). For example, ASHI and InterNACHI have developed standards of practice that guide their inspectors on conducting inspections, similar to the role of REAC’s Uniform Physical Condition Standards inspection protocol. In addition, ASHI and InterNACHI require their inspectors to inspect the same five areas of a property that REAC does. Finally, ASHI and InterNACHI have codes of conduct that specify what constitutes ethical conduct for their inspectors; similarly, REAC has developed business rules that define ethical conduct for contract inspectors. REAC has made two major changes to the inspection process over the past 6 years. First, in 2012, REAC updated its inspection software to include the decision-tree model previously discussed and established a point-loss cap to limit the amount by which a single deficiency in an inspectable area could reduce the overall property score. For example, if an inspector found numerous tripping hazards within the same inspectable area, the inspector would record all instances of this hazard, but the software would only deduct from the inspection score once rather than multiple times, according to REAC staff. Second, in 2017, REAC updated its compilation bulletin to address concerns that property owners were making cheap, non-industry-standard repairs to disguise deficiencies during a REAC physical inspection. REAC now requires its inspectors to determine if deficiencies have been corrected consistent with industry standards. For example, property owners cannot use materials such as asphalt, caulking, spray foam, or screws to cover or fill a crack or opening in an electrical panel because that repair would not be consistent with industry standards (see fig. 3). As shown in table 1, from fiscal years 2013 through 2017, the median inspection scores for multifamily and public housing properties were in the mid- to high-80s, with scores trending downward toward the end of that time frame. (See apps. II and III for additional data on REAC scores.) However, a small percentage of multifamily properties scored below 60, which for multifamily properties is defined as a failure and triggers enforcement actions that Multifamily Housing or the Departmental Enforcement Center can take to require the correction of physical deficiencies. Of 27,486 multifamily properties that were inspected during fiscal years 2013 through 2017, 1,760 properties (6 percent) failed at least one inspection, and 272 properties (1 percent of the total) failed two or more inspections. Staff in Multifamily Housing field offices said multiple failed inspections are a sign of serious owner noncompliance, such as an owner who plans to sell and thus lacks motivation to make needed repairs. Multifamily Housing staff said that they take enforcement action in these cases. A higher percentage of public housing properties scored below 60 during this same period. Of the 7,699 public housing properties that were inspected during this period, 887 (11 percent) scored below 60 for at least one inspection, and 291 (4 percent) scored below 60 for two or more inspections. REAC has not conducted a comprehensive review of its inspection process since 2001, even though new risks to its process have emerged since then. A concern of REAC staff is that some property owners have taken advantage of the scoring system and others have misrepresented the conditions of their properties. Specifically, because more points are deducted for deficiencies on the property site than for deficiencies in a dwelling unit, some property owners prioritize site repairs over unit repairs. Additionally, some property owners attempt to cover up, rather than address, deficiencies—such as by using mulch on a building exterior to hide erosion. REAC staff have also raised concerns about property owners employing current or former REAC contract inspectors to help prepare for an inspection, sometimes by guiding owners to repair just enough to pass inspection rather than comprehensively addressing deficiencies. REAC also continues to find that some contract inspectors are conducting inspections that do not meet REAC’s quality standards (discussed later in the report). Property owner associations we met with also raised concerns about the fairness of the inspection process. Specifically, representatives of two property owner associations said that REAC’s inspection process penalizes properties for items that do not affect the livability of a unit (e.g., property receives severe deficiency for chips on exterior bricks even though the dwelling units are in good condition). Representatives from one property owner association said that some properties’ scores have fluctuated even though the condition of the property has not changed. HUD’s Office of Inspector General (OIG) also identified some weaknesses in the inspection process. Specifically, the OIG found that REAC did not verify the accuracy of sampled units for public housing agencies. Further, REAC fundamentally changed the entities that conduct inspections. In 1998, REAC employed a few large inspection companies to conduct the inspections. However, in 2005, REAC introduced the reverse auction program and opened up the inspection process to a larger number of small businesses, which resulted in a change in the composition of inspectors conducting the inspections. One of the subgoals of REAC’s strategic plan for 2011–2015 was for REAC to produce inspections of HUD-assisted properties that are reliable, replicable, and reasonable. To meet this subgoal, the plan states that REAC should assess its inspection process and apply lessons learned over the last 10 years in order to improve the process. The plan also states that REAC should conduct independent, internal audits and reviews of the inspection process to identify strengths and weaknesses and develop recommendations for improvement. Further, federal internal control standards state that management should implement control activities through policies, such as by periodically reviewing policies, procedures, and related control activities for continued relevance and effectiveness in achieving the entity’s objectives or addressing related risks. REAC officials stated that they understand the importance of conducting a comprehensive review of the inspection process similar to what they did in 2001 but that they have focused their staff and resources on other priorities—for example, upgrading their technology and quality assurance processes, hiring and training quality assurance inspectors, and conducting targeted assessments of their inspection process in reaction to specific events or risks. For example, REAC staff worked on an intra- agency team to develop recommendations to address weaknesses in the inspection process that were identified as part of the assessment of Eureka Gardens. (We describe this effort later in the report.) REAC staff also said that they updated the compilation bulletin in reaction to property owners who were making cheap, non-industry-standard repairs to disguise deficiencies during a REAC physical inspection. In addition, REAC staff noted that they meet biweekly to address certain parts of the inspection process, such as the appeals and quality assurance processes. However, these efforts help identify weaknesses in the inspection process related to specific risks and were not comprehensive enough to identify and address broader risks. For example, REAC has not assessed how changes to one part of its inspection process (for example, changing how many points are deducted for a particular inspectable area) can affect other parts of the process or result in unintended consequences. Without a comprehensive review to assess its inspection process, REAC cannot determine if it is meeting the goal of producing inspections that are reliable, replicable, and reasonable. REAC may not be identifying all properties that need more frequent inspections or enforcement actions because it does not consider sampling errors of the inspection scores. REAC’s inspection process does not require the inspection of all units and buildings within large properties due to REAC’s limited inspection resources. For these properties, the inspection process provides for inspecting statistical samples of units and buildings. The results for the sample are then used to estimate a score that represents the condition of the entire property. Sampling introduces a degree of uncertainty, called sampling error, which statisticians commonly express as a range associated with numerical results. For example, for a property that scored 62 on its physical inspection, REAC would consider this a passing score that requires an annual inspection and no enforcement action. However, due to sampling error, the range associated with this score could be between 56 on the lower bound and 68 on the upper bound. HUD takes enforcement action for multifamily properties with a score below 60. Federal internal control standards state that management should use quality information to achieve the entity’s objectives. In particular, internal control standards note the importance of using the entity’s objectives and related risks to identify the information requirements needed to achieve the objectives and address the risks. REAC’s property inspection scores are currently presented as numerical results without any information on the range associated with the score. REAC’s prior version of its scoring software automatically calculated the sampling errors in the inspection scores, and this information was available for inspection scores from fiscal years 2002 through 2013. However, according to REAC staff, the current version of its scoring software does not automatically calculate the sampling errors, in part because of a lack of resources and also because they believe there is no need to calculate them. Yet, in a review we conducted of REAC in 2000, officials told us that they planned to adjust the score downward and take appropriate actions for inspection scores with a lower bound that fell under an administrative cutoff, such as 60 points. During our current review, REAC staff told us that they did not implement this plan because they would need to coordinate with other HUD offices, such as the Office of Housing, and issue a notice in the Federal Register for public comment. Based on our analysis of REAC inspection data, HUD potentially could have taken enforcement actions against more properties if REAC had taken sampling errors in inspection scores into account. For example, from fiscal years 2002 through 2013, about 4.3 percent of inspections of multifamily and public housing properties had an inspection score of 60 or slightly above 60 but had a lower bound score under 60. In addition, some multifamily and public housing properties might have been inspected more frequently if the sampling errors were taken into account. For example, federal regulations require inspections of multifamily properties scoring 90 or greater once every 3 years; scoring 80 to 89 once every 2 years; and scoring less than 80 every year. Taking sampling errors into account, about 7.1 percent of multifamily properties inspected from fiscal years 2002 through 2013 might have been inspected 1 year after the most recent inspection rather than 2 years. Likewise, about 7.2 percent of inspections of multifamily properties might have occurred 2 years after the most recent inspection, rather than 3 years. Without reporting on sampling errors and considering the results, REAC will not identify some properties which could require more frequent inspections or enforcement actions. REAC lacks comprehensive or organized documentation of the sampling methodology it uses to make generalizable estimates about the condition of properties with its scoring system. REAC’s documentation supporting its sampling methodology is contained in five documents, none of which provides a comprehensive description of the methodology with all changes to the methodology incorporated. The main document that describes the sampling methodology is a paper presented to the American Statistical Association in 2002. This document provides a very short summary of the sampling methodology, but some key assumptions, calculations, and details are not included. For example, this document does not show how REAC derived one of the variables used to calculate the number of units to sample. When we asked REAC staff to provide us with documentation on how they derived this variable, they could only provide us with an email from 2005 from a former REAC statistician that discussed some of the statistical considerations that went into the derivation of the sample-size formula. The other four documents related to the sampling methodology are dated prior to 2002 and include the initial methodology developed and subsequent changes, but these also do not provide complete information on why key assumptions were used, or the documents lack certain formulas. Further, REAC has not updated any of its documents related to the sampling methodology since 2002 to reflect current practices. Federal internal control standards state that management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. In particular, the standards note the importance of developing and maintaining documentation of the internal control system. This documentation provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel, as well as a means to communicate that knowledge as needed to external parties, such as external auditors. Further, this documentation of controls, including changes to controls, is evidence that controls are identified, capable of being communicated to those responsible for their performance, and capable of being monitored and evaluated by the entity. However, REAC does not have a process to ensure comprehensive and organized documentation of the sampling methodology of its inspection process. Instead, REAC relies on the institutional knowledge of individual staff members. For example, when we requested documentation of its sampling methodology, REAC relied on a statistician who had been with the organization for many years to locate and provide us with the documents we requested. In addition, we had to interview this individual to better understand the methodology because key pieces of information were missing from these documents. REAC staff told us that since the inspection process has remained relatively consistent over time, they have not seen the need to ensure that documentation of the sampling methodology is comprehensive and organized. By interviewing multiple individuals, reviewing multiple documents, and conducting our own calculations, we were able to determine that REAC’s sampling methodology is suitable for making generalizable estimates about the condition of a property with the scoring system. However, the lack of comprehensive and organized documentation could affect REAC’s ability to preserve institutional knowledge and make changes or improvements to its inspection process if key staff leave the agency. REAC schedules inspections of multifamily properties based on the prior REAC inspection score, but it did not meet its schedule for about 20 percent of inspections from calendar years 2013 through 2017. As discussed earlier, federal regulations require inspections of multifamily properties scoring 90 or greater once every 3 years, those scoring 80 to 89 once every 2 years, and those scoring less than 80 every year. Our analysis of REAC inspection data showed that about 20 percent of the properties were not inspected within 3 months before or after what HUD has identified as the ideal date to conduct the inspection, called an ideal future date. On average, REAC conducted inspections for these properties about 6 months past the ideal future date. REAC staff told us that there may be legitimate reasons for not conducting an inspection according to the ideal future date. For example, Multifamily Housing can delay an inspection because of natural disasters or major rehabilitations to the property, among other reasons. However, REAC maintains limited data on the reasons why inspections have been rescheduled or cancelled. In addition, these data are not readily available to understand retrospectively why an inspection did not occur on schedule. REAC also does not track its progress toward meeting its requirement for inspecting multifamily properties within prescribed time frames. Federal internal control standards state that management should use quality information to achieve the entity’s objectives. In particular, the standards note the importance of designing a process that uses the entity’s objectives and related risks to identify the information requirements needed to achieve the objectives and address the risks. Further, management should obtain relevant data from reliable internal and external sources in a timely manner and process these data into quality information that supports the internal control system. Multifamily Housing depends on REAC inspections to provide assessments of the physical condition of properties under its jurisdiction (discussed in greater detail later in this report). REAC’s inability to adhere to the inspection schedule for multifamily properties could hinder Multifamily Housing’s ability to monitor the physical condition of properties on a timely basis and take enforcement actions when warranted. The lack of a mechanism to track REAC’s progress toward meeting its requirement for inspecting multifamily properties also hinders REAC’s ability to determine what factors are contributing to delays in conducting the inspections. As a result, REAC lacks the information needed to determine the scope of the problem and what actions it can take to ensure multifamily properties are inspected on a timely basis. REAC has started a pilot program to staff inspections that contractors typically do not bid on, but it has not developed a formal plan to evaluate the results of this pilot. Since 2005, REAC has used the reverse auction program to save money on inspections and increase small business participation. However, under the reverse auction program, REAC has faced challenges in obtaining bids for inspections in some urban areas, such as Chicago, and some remote areas. To address this challenge, for a select number of properties, REAC has implemented a pilot program as an alternative to the reverse auction program. Under this alternative process, REAC has awarded multiple Indefinite Delivery/Indefinite Quantity (IDIQ) contracts to four companies to conduct these inspections. The IDIQ contracts are intended to ensure that REAC obtains physical inspections of all HUD-assisted properties on one task order rather than allowing contractors to selectively choose properties under the current program. The “all or none” approach, a key feature of these IDIQ contracts, eliminates the need to re-auction the same properties multiple times at higher prices to incentivize contractors to bid on the property. The pilot differs from REAC’s current physical inspection process in a number of ways. The pilot requires the companies that have been awarded the IDIQ contract to inspect all properties in a geographic region rather than to select which individual properties they want to bid on. Another difference is that the companies conduct quality assurance functions normally conducted by REAC staff, such as ensuring that inspectors are certified and identifying and addressing any gaps in inspectors’ performance. As of November 2018, REAC had focused its efforts on implementing the pilot but had not developed a formal plan to evaluate its results. GAO’s guide for designing evaluations states that a program evaluation is a systematic study using research methods to collect and analyze data to assess how well a program is working and why. Some key attributes of effective program evaluation design include the following: identification of data sources and collection procedures to obtain relevant, credible information; clear criteria for making comparisons that will lead to strong, defensible evaluation conclusions; and an established evaluation scope that will ensure the evaluation is tied to research questions. Federal internal control standards also state that management should use quality information to achieve the entity’s objectives. In particular, the standards note the importance of management designing a process that uses the entity’s objectives and related risks to identify information requirements needed to achieve the objectives and address the risks. Further, the standards stress the importance of management obtaining relevant data from reliable internal and external sources in a timely manner based on the identified information requirements. REAC staff told us that they plan to measure the success of the pilot program by determining whether companies are completing quality inspections in a timely manner. However, REAC staff did not provide details about how the results of the pilot would be compared to the existing process and how the quality of inspections and the performance of inspectors would be measured and assessed. Absent a formal process that incorporates key attributes for effectively evaluating the results of the pilot program, REAC may lack the information needed to determine if the pilot is a success or whether changes are needed before moving from a pilot to a permanent process. HUD has made limited progress in implementing recommendations from an internal review of REAC that was conducted in 2016. HUD created the Rapid Response and Resolution team—which consisted of staff from REAC and other units within HUD, including Multifamily Housing—in response to, among other things, problems associated with Eureka Gardens, a multifamily property in Jacksonville, Florida. In 2015, REAC conducted a physical inspection of Eureka Gardens, and the contract inspector gave the property a score of 85. However, REAC later declared that the inspection was out of standard when it learned that the contract inspector had only inspected one of the two properties associated with Eureka Gardens (the better of the two properties). REAC officials told us that property management engaged in some deceptive practices (such as making quick, cheap repairs) in an attempt to influence the inspection score. According to these officials, the inspector did not conduct the inspection consistent with REAC’s standards and was subsequently decertified. REAC then reinspected the entire Eureka Gardens property with its own quality assurance staff and found numerous deficiencies, which resulted in the property receiving an inspection score of 62. The Rapid Response and Resolution team developed 31 recommendations, 8 of which were specific to REAC, in January 2017. As of December 2018, nearly 2 years after the recommendations were developed and 3 years after the initial inspection of Eureka Gardens, REAC had reached concurrence with Multifamily Housing on 3 of these recommendations and asked for Multifamily Housing’s consideration of the funding and rulemaking requirements for the remaining 5. HUD had also not yet implemented the 3 recommendations on which it reached concurrence. Some of these recommendations address REAC’s management of the inspection process. They include the following: Weighting of dwelling units in inspection score. The review team recommended that REAC consider increasing the weight of dwelling- unit deficiencies in the physical condition score. This recommendation attempts to address the issue, discussed earlier, of property owners who focus their repairs on common areas of the property over dwelling units. Notice provided to property owners of impending inspection. This recommendation reduces the time that REAC can take to notify property owners of an upcoming inspection from 15 days to 3 days for properties that have failed their previous REAC inspection. REAC staff said that this change would provide a more accurate picture of the condition of properties since property owners generally address the maintenance of the property just before an inspection. In addition, this recommendation could address the concern discussed earlier of property owners hiring current or former REAC contract inspectors to help them prepare for an inspection. This recommendation should also encourage property owners to maintain properties in good condition at all times, according to REAC staff. Exigent health and safety risks. Another recommendation was that REAC work with Multifamily Housing and PIH to implement a risk- based exigent health and safety abatement verification policy. According to REAC staff, some properties certify that they have corrected exigent health and safety deficiencies when they have not done so. We found that many inspections conducted from fiscal years 2013 through 2017 had at least one exigent health and safety deficiency, and the percentage has been higher in recent years (see table 2). Field office staff from PIH and Multifamily Housing may check to ensure that these repairs have been made when they are onsite. However, neither of these offices has a formal program to ensure that property owners are actually addressing the exigent health and safety issues. As a result, property owners may choose to correct only those deficiencies that they believe will be checked by HUD field office staff, according to REAC staff. Federal internal control standards state that management should identify, analyze, and respond to risks related to achieving the defined objectives. By establishing the Rapid Response and Resolution team, HUD took the steps of identifying the risks to its inspection process and designing responses to these risks. However, the standards also call for remediating identified internal control deficiencies on a timely basis. HUD officials we met with attributed the delay in implementing the recommendations to prior vacancies in some senior leadership positions, including positions in Multifamily Housing. HUD’s delay in implementing most of the recommendations from the Rapid Response and Resolution team affects REAC’s ability to respond to weaknesses it has identified in the inspection process in a timely manner. Contract inspector candidates certify through an application that they meet REAC’s qualification requirements, but REAC does not currently verify that candidates have met these requirements before REAC selects them for training and determines them to be eligible to inspect HUD- assisted properties. Before inviting candidates to participate in inspector training, REAC requires them to certify that they meet three main qualifications: Inspections. Candidates must have conducted a minimum of 250 residential or commercial inspections. Building trades knowledge. Candidates must have building trades knowledge, such as knowledge of construction methods or electrical systems. Computer literacy. Candidates must be able to use email, the internet, and Microsoft Windows. However, REAC does not require documentation from contract inspector candidates demonstrating that they successfully conducted 250 inspections. REAC officials told us that they intend to verify a sample of the 250 inspections for each inspector, but as of November 2018 they had not yet developed a process for doing so, such as by developing a methodology for sampling and a timeline for contacting references. In contrast, one of the home inspection associations we met with, ASHI, requires certified inspector candidates to submit a list of 250 fee-paid home inspections that meet or exceed the ASHI standards and to provide a notarized affidavit validating those inspections. In addition, REAC staff told us that some contract inspector candidates have inspection experience based on inspections that are not as rigorous as those conducted using the Uniform Physical Condition Standards protocol. Participants in three of the four discussion groups we held with REAC quality assurance inspectors and supervisors told us that they had trained candidates who had included information on their applications about previous inspection experience that was not well matched to REAC’s inspection process. For instance, some inspector candidates submitted Federal Emergency Management Agency inspections and U.S. Army Office of Housing inspections as evidence of having completed 250 inspections, but REAC officials said these inspections are not as comprehensive as REAC inspections because they do not assess building systems, such as electrical or heating, ventilation, and air conditioning systems. Federal internal control standards call for management to recruit competent individuals so that they are able to accomplish their assigned responsibilities. In addition, key principles for workforce planning state that agencies need to determine the critical skills and competencies necessary to achieve their goals. REAC officials told us that the inspector training program should weed out inspector candidates that may not have the appropriate qualifications. However, although REAC officials told us that inspector candidates have been removed from training for not having the requisite skills, the officials were not able to determine how many candidates had misrepresented their qualifications on their application or had failed training for other reasons. REAC does not verify the inspections submitted by inspector candidates—relying instead on training to screen out unqualified candidates—and does not determine the type of inspection that may count as a qualifying inspection. As a result, REAC may be allowing candidates with insufficient experience to proceed in the training process, which may waste resources by training candidates who are unlikely to become successful inspectors. Contract inspector candidates must complete several phases of REAC training—online, in-class, and field—and pass associated examinations, as well as a background check. Online training. Inspector candidates first complete a 6-week online training that includes web-based modules on the Uniform Physical Condition Standards protocol and the use of the software system for the handheld data collection device. Candidates must pass a pre- certification examination to progress to the next phase of training. In-class training. After passing a background check, inspector candidates then begin in-class training. This phase consists of 3 to 4 days of in-class training led by REAC quality assurance inspectors and covers the compilation bulletin, Uniform Physical Condition Standards protocol, best practices, simulations of the inspection software, and hands-on practice exercises using the software. To proceed to field training, inspector candidates must pass a certification examination with a minimum score of 75 percent that covers material from both the compilation bulletin and the Uniform Physical Condition Standards protocol. Field training. The last phase is a 5-day field training course that culminates in a field examination. REAC quality assurance inspectors lead and provide instruction for the first 4 days of field training. Inspector candidates independently conduct a mock inspection using the Uniform Physical Condition Standards protocol on the fifth day, and a quality assurance inspector evaluates the candidate’s performance. REAC has made changes to training in recent years. For example, REAC began using actual HUD-assisted properties, rather than simulated properties, for the mock inspection. Some quality assurance staff and property owner associations told us they regarded the changes made in recent years to be beneficial. Participants in three of the four discussion groups we held with quality assurance supervisors and inspectors, as well as two representatives of property owner advocacy organizations, said that, in addition to classroom training, field training on a physical property helped to assess the competency of inspector candidates. In addition, stakeholders—including property managers, contract inspectors, and REAC staff—told us the mock inspections have been effective at providing training to new inspectors, and that the professionalism of contract inspectors has improved. REAC contracts with a private vendor to provide the contract inspector online training, and the vendor provides data and reports that REAC staff use to track inspector candidates’ progress through the online training modules. REAC officials told us they use this information to identify areas of the training where candidates struggle and to help revise the training material. In addition, REAC solicits feedback from contract inspector candidates on the online training. REAC also uses key performance indicators to track the number of inspector candidates who enroll and whether they pass or fail training. However, REAC does not currently have formal metrics or use data to track the effectiveness of its three phases of training. For instance, REAC does not track key measures of performance that could provide management with information to improve the training process, such as how individuals score in each section of the in-class training examination and their field examinations. REAC also does not track the resources spent on training, either in terms of funds spent or number of quality assurance inspectors who participate. According to key practices we have identified for training and development, agencies should have processes to systematically track the cost and delivery of training and measure the effectiveness of those efforts. REAC officials said that they would like to have such mechanisms and have developed a proposal to consolidate training functions and better align training to REAC’s strategic goals. However, the proposal does not include performance measures for evaluating the effectiveness and efficiency of training. Use of cost-tracking and performance measures tied to its strategic goal of improving the inspection process could improve REAC’s ability to manage scarce resources, evaluate the effectiveness of its training program, and plan for future training. REAC’s quality assurance inspectors—who train and oversee contract inspectors—must be able to conduct physical inspections of properties as well as assess contract inspectors’ performance. To assess contract inspector performance, quality assurance inspectors oversee and mentor contract inspectors during CQA reviews and provide them feedback in a collaborative manner, an approach REAC management implemented in 2017. Some senior quality assurance inspectors are also responsible for leading classroom and field training for contract inspectors. According to REAC officials, REAC’s quality assurance inspectors receive the same training as contract inspectors on the Uniform Physical Condition Standards inspection protocol. However, REAC’s training for quality assurance inspectors does not include formal instruction on how to coach or provide feedback during CQA reviews. Instead, new quality assurance inspectors are provided with on-the-job training, and they can only conduct CQA reviews independently when quality assurance supervisors are satisfied that they are sufficiently competent. Beyond the on-the-job training, quality assurance inspectors are encouraged to undergo additional online training on coaching, but there are no specific training requirements related to conducting CQA reviews. Participants in two of our three discussion groups with quality assurance inspectors told us they were not sure how to provide the collaborative coaching and mentorship REAC officials said they wanted. REAC also does not specifically train quality assurance inspectors in how to provide classroom and field training to contract inspectors, and participants in all three discussion groups with quality assurance inspectors told us that instructors do not seem to take a consistent approach to classroom and field training. In addition, REAC’s training requirements for quality assurance inspectors are not documented in the quality assurance standard operating procedures or other documents we reviewed. For example, REAC’s new, more collaborative approach to CQA reviews was communicated to quality assurance inspectors during staff meetings, but REAC staff have not documented the approach or developed any specific training. Some contract inspectors told us that quality assurance inspectors often have less experience conducting inspections than they do. They suggested that this gap may affect quality assurance inspectors’ ability to competently oversee CQA reviews and conduct QCIs. REAC officials told us they are considering changing quality assurance inspector training requirements to be more rigorous than contract inspector training. For example, staff from REAC’s Quality Control group, created in 2017, told us that they are considering expanding training on the five inspectable areas and assessing quality assurance inspectors to see if they need additional support in any of these areas. They would also like to require quality assurance inspectors to pass the training examinations with minimum scores of 90 percent, instead of the score of 75 percent that currently applies to both contract and quality assurance inspectors. However, the Quality Control group has not implemented these changes, officials said, because its staff resources are limited, and staff have been reallocated to support other projects within REAC. In comparison, one of the home inspector associations we met with, InterNACHI, has specific requirements for its instructors. According to the association, its instructors are certified master inspectors, have completed a minimum of 1,000 paid inspections or hours of education or some combination thereof, and have conducted inspections for a minimum of 3 years. The instructors also assist in developing the educational material for training courses. Federal internal control standards state that management should demonstrate a commitment to recruit, develop, and retain competent individuals. In particular, the standards note that agency personnel need to possess and maintain a level of competence that allows them to accomplish their assigned responsibilities. The standards also note the importance of management enabling individuals to develop competencies appropriate for key roles and tailoring training based on the needs of the role. Without assessing whether training for quality assurance inspectors is sufficient and requiring additional training as needed, REAC may not have reasonable assurance that these inspectors have the skills required to oversee contract inspectors. Federal internal control standards also state that management should design control activities to achieve objectives and respond to risks, and they note the importance of documenting internal control—for example, in management directives, administrative policies, or operating manuals. These standards also state that management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. For example, the standards note that effective documentation assists in management’s design of internal controls by establishing and communicating the who, what, when, where, and why of internal control execution to personnel. Without documenting training requirements that encompass all job responsibilities, REAC may not have reasonable assurance that the required skills and competencies are clearly communicated to and understood by quality assurance inspectors and aligned with job duties. REAC has ongoing requirements for contract inspectors to maintain their eligibility, but these do not include continuing education requirements. Contract inspectors must conduct at least 25 successful inspections per year—that is, inspections found to be within REAC’s inspection standards—and pass a background check every 5 years to remain certified. REAC also offers optional training through online refresher modules. However, REAC does not know how many contract inspectors use these resources or how effective they are. In comparison, ASHI and InterNACHI have continuing education requirements for their certified inspectors. ASHI requires inspectors to earn 20 continuing education credits annually. The qualifying training courses must be ASHI-approved, the inspector must submit a signed affidavit attesting to having attended the training, and ASHI spot checks to monitor compliance. InterNACHI requires inspectors to earn 24 continuing education credits annually and to pass the InterNACHI Online Inspector Examination with a score of 80 percent or better every 3 years. REAC encourages quality assurance inspectors to take additional training but does not require continuing education. REAC offers optional “dine- and-learn events” to update both contract and quality assurance inspectors on policy and procedure changes and point out errors they commonly observe. In 2017 REAC also began offering limited coaching to contract inspectors by quality assurance inspection reviewers, a process separate from CQA reviews. The reviewers compare physical defects identified by contract inspectors to those that were identified by quality assurance inspectors during QCIs of the same property. Reviewers then provide one-on-one feedback to contract inspectors to address any discrepancies in inspection scores. REAC officials said that for continuing education they prefer self-paced learning to formal instruction because it more appropriately matches the varying education needs of inspectors. REAC’s strategic plan proposes developing REAC-wide policies for staff training and skill development, but it does not include any requirements for continuing education. Key practices we have previously identified for training and development suggest that agencies should encourage employees to take an active role in their professional development, which can include requiring employees to complete a specific level of continuing education. Ongoing training requirements for contract and quality assurance inspectors could help REAC ensure that inspectors are up-to- date on REAC policies and industry standards. Such continuing education could also refresh existing knowledge, helping contract and quality assurance inspectors conduct high-quality inspections consistently. Continuing education could also help quality assurance inspectors to develop their mentoring and coaching skills, which would better enable them to develop and oversee contract inspectors. REAC’s mechanisms to monitor and evaluate its contract inspectors include collaborative quality assurance reviews, quality control inspections, and various other monitoring tools. REAC uses CQA reviews to monitor, evaluate, coach, and provide feedback to contract inspectors. REAC has documented its processes for conducting and reporting on CQA reviews in field training guidance, standard operating procedures, and the compilation bulletin. To determine which inspections will receive a CQA review, REAC combines a risk-based approach—targeting low-performing inspectors—with scheduling needs, based on the timing and geographic location of the inspection. REAC conducted almost 3,000 CQA reviews from fiscal years 2013 through 2017. As shown in table 3, the percentage of inspections each year that received a CQA review ranged from about 3 to 8 percent. Our analysis of CQA review data shows that some contract inspectors are not conducting inspections in accordance with REAC standards. From fiscal years 2013 through 2017, an average of 17 percent of CQA reviews resulted in contract inspectors receiving a rating that was outside of REAC’s physical inspection standards, referred to as an outside standard rating. This rating is based on the contract inspector committing at least 1 of 18 types of performance- or scheduling-related REAC protocol violations, with 8 of the performance violations resulting in an automatic outside standard rating. The percentage of outside standard ratings was significantly higher in fiscal years 2015 through 2017, as compared to 2013 and 2014 (see fig. 4). According to REAC officials, this increase was likely due, in part, to an increase in the number of less experienced contract inspectors. Specifically, from September 2013 through December 2015, REAC attempted to expand the pool of contract inspector candidates by lowering the required number of inspections from 250 to 50. REAC officials confirmed that these inspectors were less experienced and more likely to violate protocols. As previously discussed, REAC has taken steps to make its CQA reviews more collaborative, but some stakeholders said that challenges remain. REAC’s 2015 standard operating procedures stated that the goal for CQA reviews should not be to designate a contract inspector as outside standard, but rather to ensure inspection accuracy and to improve the knowledge of the contract inspector. However, it was not until more recently that REAC management emphasized in several staff meetings the need for quality assurance inspectors to take a collaborative approach, according to REAC staff. Despite this new emphasis, participants in a discussion group we held with contract inspectors told us that they believe a punitive approach persists with some quality assurance inspectors, that repeated pairings of contract and quality assurance inspectors could lead to bias, and that contract inspectors who receive a high number of CQA reviews may feel like REAC is targeting them. REAC uses QCIs to further ensure the accuracy of inspections conducted by contract inspectors. As previously noted, a combination of factors can lead REAC to reject an inspection and trigger a QCI, including significant differences between the current inspection and previous inspections and other contextual factors, such as the inspector’s past CQA review performance. In a QCI, the quality assurance inspector reviews an inspection report and then conducts a new inspection to identify potential weaknesses and evaluate the inspector’s performance. The QCI results in a new inspection score. REAC has standard operating procedures that document requirements for scheduling, conducting, and reporting QCIs. REAC completed 226 QCIs from March 2017 through June 2018. Our review found that more than 50 percent of QCIs resulted in an outside standard rating. On average, contract inspectors gave properties a score that was 16 points higher than the score given subsequently by quality assurance inspectors, indicating that those contract inspectors missed deficiencies. Of these inspections, about 15 percent that had initially received a passing score from the contract inspector failed the subsequent QCI. REAC also uses ratings and reports to oversee contract inspectors. REAC assigns each contract inspector a rating based on a combination of factors, including CQA results and percentage of inspections rejected, and these ratings help target which inspectors should receive CQA reviews and CQIs. REAC also produces two reports for all contract inspectors to assist in its oversight: Defect Comparison Reports compare the specific deficiencies reported by an inspector to the frequency with which other contract inspectors reported that same deficiency across properties. REAC primarily uses the results to target areas to coach contract inspectors who have been flagged for a QCI. Defect Delta Reports compare deficiencies in a contract inspector report to deficiencies a quality assurance inspector reported in a follow-up inspection (usually a QCI). REAC primarily uses this information to identify the types of deficiencies the contract inspector is missing. REAC did not meet management targets for the number of CQA reviews to be conducted in any quarter from fiscal years 2013 through 2017 (see fig. 5). REAC officials told us their management target is to conduct three CQA reviews for each high-risk contract inspector and two CQA reviews for lower-risk contract inspectors each quarter. REAC officials told us that in developing the targets, they attempted to balance risks to the quality of inspections and resources available. In addition, as of June 2018, REAC had not met management targets for timeliness of QCIs in any quarter. REAC has no documented timeliness goals for QCIs, but REAC officials told us that QCIs are supposed to take place within 30 days of the original inspection date because the condition of the property can change over time (see fig. 6). REAC officials told us they did not meet these management targets for CQAs and QCIs because they did not have enough quality assurance inspectors. In addition, when quality assurance inspectors are pulled onto other projects, such as supporting HUD’s efforts related to natural disasters, REAC’s ability to conduct CQAs and QCIs is reduced. For example, in fiscal year 2018, 28 quality assurance inspectors were pulled offline to assist HUD in areas affected by hurricanes. REAC officials told us these temporary reassignments have affected their ability to implement the quality assurance process. REAC staff told us that the quality of inspections may have deteriorated because some contract inspectors were aware that quality assurance inspectors would not be conducting CQA reviews or QCIs during those post-disaster periods. REAC has recently hired more quality assurance inspectors to help address staffing shortages. In addition, REAC officials told us that they intend to take into account the likely effects of natural disasters on their ability to conduct quality assurance reviews when planning for these reviews in the future, but REAC has not yet developed a plan to meet its targets that includes, for example, mechanisms to mitigate resource constraints and unforeseen demands on staff. According to REAC’s strategic plan, to produce physical inspections that are reliable, replicable, and reasonable, REAC is to look for patterns and trends in inspection results, such as inconsistencies between inspectors, regional and area differences, and patterns in different inspection criteria. In addition, the strategic plan calls for REAC to assess and improve the quality of contract inspectors. However, if REAC is unable to meet its management targets for CQA reviews, it may not be able to consistently produce high-quality inspections because it is not providing routine opportunities for contract inspectors to receive coaching from quality assurance inspectors, which could include addressing deficiencies that the contract inspectors did not initially identify and record. In addition, if QCIs are not conducted shortly after the original inspections, REAC may not be able to verify the quality of the inspection because the condition of the property could change over time. REAC’s inability to meet management targets for CQA reviews and QCIs could also affect its ability to monitor patterns in inspection results because, for example, the quality of QCI data would be less reliable due to the lapse in time. As a result, REAC may not be using quality assurance inspector resources as effectively as possible. REAC’s Inspector Administration division administers a variety of administrative actions and disciplinary sanctions against contract inspectors in response to complaints or CQA reviews and QCIs. Inspector Administration officials said that they acknowledge and follow up on all complaints received from property representatives and residents, among others. Complaints about a contract inspector can relate to conduct, inspection protocol violations, scheduling, and conflicts of interest, among other issues. Inspector Administration uses the Code of Conduct, Uniform Physical Condition Standards inspection protocol, and compilation bulletin as standards for evaluating contract inspector conduct. In order of increasing severity, Inspector Administration can issue a letter of warning, issue a performance deficiency, suspend or decertify the inspector, or refer the inspector to HUD’s OIG, among others. While Inspector Administration can use professional judgement in adjudicating complaints, some actions are automatic. For example, decertification is automatic for inspectors with three or more performance deficiencies, inspectors found to have engaged in egregious misconduct, or inspectors who conduct fewer than 25 inspections annually. Inspector Administration took more than 700 administrative enforcement actions against contract inspectors from fiscal years 2013 through 2017 (see fig. 7). As part of its effort to reform its contract inspector pool, REAC decertified 127 inspectors from fiscal years 2013 through 2017 due to inactivity, conduct, or performance issues. For example, REAC decertified the contract inspector who gave Eureka Gardens a passing inspection score even though it was in poor physical condition. Two advocacy organizations told us they noticed that REAC was decertifying more inspectors than in the past, and one said that the quality of contract inspectors had improved as a result. In response to concerns from contract inspectors, Inspector Administration is proposing changes to, among other things, provide contract inspectors who are subject to potential enforcement actions with more opportunities to present their perspective. For example, Inspector Administration would allow contract inspectors to appeal performance deficiencies earlier in the process. Other proposed changes would make performance deficiencies based on outside standard ratings discretionary instead of automatic and would remove a performance deficiency from a contract inspector’s record after 25 consecutive inspections without a new performance deficiency instead of 30. Inspector Administration officials said the new rules would also adjust decertification sanction periods, which specify the amount of time a decertified contact inspector must wait before reapplying to REAC, to account more appropriately for the reason the contact inspector left REAC (e.g., resignation, performance, or conduct). REAC created the Quality Control group to standardize quality assurance inspector reviews by conducting more frequent oversight and looking for trends across all quality assurance inspectors, according to a Quality Control official. This official said that one type of oversight involves a quality control staff member conducting an identical inspection 1 day after that of a quality assurance inspector to determine how well the inspector recorded deficiencies. Inspections are then rated as either “acceptable” or “unacceptable” based on whether the inspector followed established protocols and observed and accurately recorded 90 percent or greater of the existing deficiencies. According to the official, inspection reviews are expected to be shared with quality assurance management and individual supervisors to support quality assurance inspector development. This official also told us Quality Control plans to conduct reviews of quality assurance inspectors at least once a year, or more frequently as needed. In November 2018, Quality Control developed a mission statement which says that the primary goal of the group is to improve the consistency of inspections. Also in November 2018, Quality Control developed procedures for reviewing quality assurance inspectors, which include processes for conducting field reviews of completed inspections, criteria for acceptable inspections, and processes for providing feedback. An official from Quality Control said that the group worked with other divisions within REAC, such as PASS Quality Assurance and Research and Development, to develop the procedures and criteria for evaluating quality assurance inspectors. The official told us both its mission statement and procedures have not been implemented, in part because Quality Control staff have been repeatedly pulled onto other special projects. The official told us that these documents have been approved by REAC management and that Quality Control intends to implement the procedures in 2019. According to federal internal control standards, management should implement control activities through policies. For example, the standards call for documenting in policies each unit’s responsibility for an operational process’s objectives and related risks. Without finalizing and implementing its policies and procedures for reviewing quality assurance inspectors, Quality Control may not be able to provide consistent reviews of quality assurance inspectors, which could affect the quality of inspections as well as the feedback and coaching quality assurance inspectors provide to contract inspectors. Prioritizing the implementation of Quality Control’s review procedures could help ensure that Quality Control achieves its objectives and provides consistent reviews of quality assurance inspectors. The standards REAC uses to measure quality assurance inspectors’ performance do not fully align with their job duties. Quality assurance supervisors are primarily responsible for evaluating quality assurance inspector performance using five performance elements, which REAC’s performance appraisal system describes as follows: Collaboration: Provide customer service communication both verbally and in writing to internal and external HUD stakeholders, customers, or anyone who comes in contact with quality assurance services. Individual training: Develop competencies and perform individual training associated with job duties. Personal investment: Improve processes, such as through special projects or self-initiated projects that improve the quality assurance division’s standard operating procedures, the Uniform Physical Condition Standards inspection protocol, the compilation bulletin, or others. Risk management: Maximize scarce resources and be cost efficient to the government in all aspects of job duties and assignment. Meeting the need for quality affordable rental housing: Perform in accordance with all protocols and standard operating procedures, and complete CQA reviews and Uniform Physical Condition Standards inspections. REAC’s performance appraisal system includes descriptions of the standards for each of the five performance elements, as well as supporting behaviors. For example, to be rated fully successful for “meeting the need for quality affordable rental housing,” quality assurance inspectors should independently complete Uniform Physical Condition Standards inspections with no more than two inspections being rejected by REAC within the rating period. Based on the standards, quality assurance inspectors are also expected to conduct CQA reviews and field trainings for contract inspector candidates. However, the performance appraisal system for quality assurance inspectors does not include performance elements with competencies that relate to all of their job duties. For example, the performance appraisal system does not define expectations for performing CQA reviews or QCIs. In addition, it does not include criteria for evaluating the training, coaching, and mentoring that quality assurance inspectors are expected to provide to contract inspectors. Quality assurance supervisors can incorporate information from reviews of quality assurance inspectors by Quality Control in their performance evaluations. However, REAC officials told us that Quality Control does not evaluate inspectors based on the performance elements and standards. Instead, Quality Control’s reviews only evaluate an inspector’s performance as it relates to the QCI being reviewed, such as following established protocols and observing and accurately recording 90 percent or greater of the existing deficiencies. In addition, Quality Control’s reviews do not include evaluations of a quality assurance inspector’s performance for other key job duties, such as training and mentoring contract inspectors. According to key practices we have identified for effective performance management, agencies should use competencies to define the skills and supporting behaviors that individuals need to effectively contribute to organizational results. REAC staff told us they do not know when the performance elements and standards for quality assurance inspectors were last revisited, and new job duties such as conducting QCIs have been added that are not part of the performance elements. Better alignment between the performance competencies and the job responsibilities of quality assurance inspectors would help ensure that inspectors are assessed on all their key duties—including training and mentoring contract inspectors—which could improve the quality of inspections and reviews. PIH and Multifamily Housing each have separate processes to monitor the conditions of HUD-assisted properties, including physical conditions, and take enforcement actions if properties are not decent, safe, sanitary, and in good repair. PIH assesses the performance of PHAs on key indicators through a federal regulatory process—the Public Housing Assessment System. PIH also monitors PHAs through a Risk Assessment Protocol, which incorporates qualitative data and determines actions to address identified risks. The Risk Assessment Protocol is intended to be a proactive approach to address risk at PHAs and use resources efficiently. Separately, Multifamily Housing monitors properties that score below 60 on the REAC physical inspection. To account for properties scoring 60 or above on the REAC inspection, as well as to monitor property characteristics other than physical conditions, Multifamily Housing assesses properties through its risk rating system. The Public Housing Assessment System uses the REAC physical inspection score for each public housing development to determine the physical performance of the PHA. The physical performance of PHAs is one of four indicators within the Public Housing Assessment System, which assesses the performance of PHAs and determines a performance designation. The four indicators are (1) the physical condition of the PHA’s housing developments, (2) the financial condition of the agency, (3) the management operations of the agency, and (4) utilization of property modernization and development funds (capital fund). REAC inspection scores are adjusted to reflect the size of each housing development, and their weighted average is the physical performance indicator for a PHA. The physical indicator score, worth a maximum of 40 points (out of 100 points total) toward the overall Public Housing Assessment System score, has the highest value of the four indicators. To determine the financial, management, and capital fund indicators, PHAs upload information electronically to REAC, and REAC’s data systems generate a score for each indicator and overall. Figure 8 shows the maximum value for each indicator and overall score. Table 4 explains how the indicator and overall assessment scores lead to a performance designation. PHAs are assessed and receive a performance designation every 1 to 3 years, according to their size and prior performance designation. PHAs with at least 250 units receive an assessment annually. PHAs with fewer than 250 units receive an assessment every 3 years if designated as a high performer, 2 years if designated as a standard or substandard performer, and annually if designated as a troubled or capital fund troubled performer. In years that smaller PHAs do not receive an assessment, they must provide financial data to REAC but do not receive a published assessment score or designation. Following REAC’s release of the performance designations, PHAs and PIH each have a role in ensuring that physical deficiencies are corrected. REAC is not responsible for ensuring that PHAs correct physical deficiencies. According to federal regulations, PHAs must take certain actions depending on their performance designation. PHAs designated as troubled must enter into a recovery agreement with PIH to improve their performance within 2 years. PHAs designated as standard or substandard performers must correct deficiencies identified in the assessment within 90 days, or they may develop a plan to correct the deficiency within a specified time frame. PIH officials told us they monitor whether PHAs designated as standard or substandard performers have developed a plan or if field offices are assisting the PHA. PHAs designated as high performers are not required to correct deficiencies. Table 5 shows the number of PHAs in each designation for fiscal years 2013 through 2017, including some PHAs exempt from receiving a performance designation (e.g., small PHA deregulation). If PHAs do not correct deficiencies or improve their performance, PIH officials told us they can initiate a series of actions. First, PIH field offices are to remind PHAs of their obligation to provide housing that is decent, safe, sanitary, and in good condition. If those conversations are not effective, PIH can take administrative or enforcement actions. For example, PIH can refer PHAs to the Departmental Enforcement Center, which can exclude PHA leadership from participating in HUD programs. However, we previously found that PIH refers PHAs to the Departmental Enforcement Center infrequently, making 12 referrals in 2017 and 25 referrals in 2016. In rare instances, PIH also can place the PHA into administrative receivership and take control of the PHA’s operations. These actions also could be part of a recovery agreement for troubled performer PHAs. PIH officials told us they initiate actions specified in the recovery agreement for troubled performer PHAs that do not improve their performance within 2 years. To inform its monitoring efforts, PIH uses the Risk Assessment Protocol to assess PHAs in four risk categories: physical, governance, financial, and management. PIH collects quantitative data from various HUD data systems and qualitative data from a survey administered by PIH field offices. The physical risk category uses a PHA’s Public Housing Assessment System physical indicator score—which is determined using the REAC inspection score—as one factor in determining physical risk. PIH also assesses physical risk using the qualitative survey and location of the PHA. Additionally, the performance designation from the Public Housing Assessment System—which incorporates the REAC inspection score—is included as part of assessing governance risk. The financial and management categories do not incorporate the REAC physical inspection score. For each risk category, PIH assigns points and designates a risk level for each PHA, as shown in figure 9. A higher number of points is associated with higher risk. For example, PIH assigns 25 points to PHAs with a physical indicator score of 25 or below and zero points to PHAs with a physical indicator score of 28 or higher. After assigning points, PIH designates a risk level for each risk category, as well as overall, based on the average PHA score for each category and overall. These risk designations are very high, high, moderate, and low. PHAs furthest from and above the average score are designated as very high risk, and PHAs closest to the average score are designated as low risk. PIH designates a risk level to PHAs every quarter, although some information used to determine the risk level is not updated every quarter. For example, the qualitative survey is updated every other quarter. PIH determines actions—called risk treatments—to address each risk category based on a PHA’s risk level. PIH determines actions each quarter for every PHA newly designated as very high or high risk, and it determines actions every other quarter for all other very high, high, or moderate risk PHAs. To address physical risks, field office staff may provide training or technical assistance to PHAs. For example, field office staff told us they provided technical assistance by explaining the physical inspection standards and policies related to using operating funds to make physical repairs. Risk treatments have a completion date, and PIH field office staff are to monitor whether the treatment is effective. If the risk treatments do not result in improvements, PIH officials told us they can seek technical assistance from subject matter experts within PIH or can elevate the risk treatment, among other actions. For example, PIH can provide on-site assistance rather than remote assistance. Multifamily Housing is required to direct property owners to correct physical deficiencies based on the REAC inspection score. For properties that score below 60 on the REAC physical inspection, Multifamily Housing issues property owners a notice to take the following actions: (1) provide a copy of the notice to residents; (2) survey 100 percent of the property to identify all physical deficiencies; (3) correct all deficiencies identified during the survey and the REAC inspection; (4) certify that they have corrected all deficiencies; and (5) submit a 100- percent survey of the property and certification of corrected deficiencies to HUD. Property owners should complete these actions within 60 days of receiving the notice but may request an extension if correcting the deficiencies will take longer than 60 days. For example, Multifamily Housing may issue extensions for notices received in winter months because seasonal conditions may make certain repair work, such as pouring concrete, more difficult to complete within 60 days. Multifamily Housing schedules a follow-up inspection depending on whether property owners submit a certification, as well as if the property scores 30 or below on the inspection. For property owners who certify that deficiencies have been corrected, Multifamily Housing schedules the property’s next inspection to take place within 1 year after the date of the last inspection. For property owners who do not submit the certification or for properties that score 30 or below on the REAC inspection, Multifamily Housing or the Departmental Enforcement Center schedules a follow-up inspection as soon as possible. Multifamily Housing uses the REAC score from that next inspection to determine whether the owner corrected deficiencies. After issuing a notice, Multifamily Housing can take various actions when properties’ scores on the REAC inspection remain below 60, if owners do not certify or correct physical deficiencies. Table 6 summarizes the actions Multifamily Housing took in fiscal years 2016 and 2017. For example, Multifamily Housing officials initially can place a flag in a data system to indicate that an owner has not met requirements for properties to be decent, safe, sanitary, and in good repair. This flag may prevent the owner from further participation in HUD programs. Another action Multifamily Housing officials can take is to change the property’s management agent. Multifamily Housing officials told us this action has been successful in improving the physical conditions of properties when properties do not require significant repair work. Multifamily Housing also can take more significant actions, such as terminating a rental assistance contract or foreclosing on a loan and relocating tenants from these properties. In addition to taking these actions, REAC and Multifamily Housing refer properties to the Departmental Enforcement Center when they score below a determined threshold. Upon publishing the inspection score, REAC refers properties that score 30 or below on a REAC inspection to the Departmental Enforcement Center automatically. Multifamily Housing officials told us they coordinate with relevant stakeholders to discuss these properties. Multifamily Housing also can refer properties electively to the Departmental Enforcement Center when they score between 31 and 59 on the REAC inspection. Further, Multifamily Housing can recommend specific actions for the Departmental Enforcement Center to take regardless of a property’s inspection score. The Departmental Enforcement Center can impose civil money penalties to encourage compliance with HUD’s regulations or limit a property owner from participating in HUD programs. Our analysis of referral data for physical conditions from fiscal years 2012 through 2017 shows that for 12 referrals, the Departmental Enforcement Center imposed money penalties through a settlement, and that no referrals resulted in a suspension or debarment. However, according to our previous work on the Departmental Enforcement Center, most referrals result from financial reviews rather than physical inspections. The Office of Multifamily Housing’s current practice of issuing notices to property owners when the REAC score is 59 or below is inconsistent with the legal requirement. As previously discussed, for properties that score 59 or below on the REAC inspection, HUD issues notices for property owners to certify that deficiencies have been identified and corrected within 60 days. However, the 2017 and 2018 Consolidated Appropriations Acts state that HUD must provide a notice to owners of properties that score 60 or below on the REAC physical inspection. Multifamily Housing officials told us that they believe language in the appropriations acts is not clear regarding the threshold to issue notices to property owners. Specifically, the appropriations acts state that HUD should issue a notice for properties that score 60 or below, and also that HUD may withdraw the notice to property owners when they successfully appeal their inspection score to 60 or above. Additionally, Multifamily Housing officials told us that HUD’s long-standing and current practice is to issue notices when a property receives a score of 59 or below. According to our analysis of inspection data, 30 properties received a score of 60 from May 2017 to December 2017 and would not have received a notice to correct physical deficiencies under HUD’s approach. Unless Congress changes the threshold identified in appropriations acts from 60 to 59 or HUD changes its practice to issue notices to properties that score 60 or below, HUD’s actions will continue to be inconsistent with the legal requirement. Multifamily Housing also uses other processes to monitor the physical condition of properties, including properties that score 60 or above on the REAC inspection. These other processes incorporate additional aspects of properties beyond physical conditions. Multifamily Housing’s risk rating system uses information on properties’ physical, financial, and management conditions to assign one of three risk ratings—troubled, potentially troubled, or not troubled—to each property. The REAC inspection score, along with actions taken to correct deficiencies, is one factor that determines the risk rating. Specifically, properties that score between 30 and 70 on the REAC inspection are rated as potentially troubled if the property owner is addressing physical deficiencies. Properties that score between 30 and 59 are rated as troubled if the owner has not certified that deficiencies have been corrected. Properties that score below 30 are rated as troubled and maintain that rating until the next REAC inspection. Multifamily Housing field office and headquarters staff told us they provide greater monitoring and oversight to properties rated as troubled and potentially troubled. Properties rated as troubled are required to develop an action plan to identify and document steps to address their risk, including physical risk. For example, a plan to improve the physical condition of a property may direct property owners to rehabilitate units. Properties rated as potentially troubled may develop such a plan but are not required to do so. Additionally, Multifamily Housing headquarters staff conduct a monthly call with field office staff to discuss properties rated as troubled. Multifamily Housing officials told us they review properties every 3 to 12 months based on the risk rating and can take actions if properties are not correcting issues. If property owners do not correct issues outlined in their plan, Multifamily Housing can take many of the actions listed previously, such as changing the management agent. Multifamily Housing can also monitor properties through other processes, such as site visits or other reviews. According to Multifamily Housing officials, field office staff conduct site visits of properties if they receive multiple complaints from tenants or notice a particular concern, or if the property receives media attention. Multifamily Housing also can conduct site visits of properties through a Management and Occupancy Review. Multifamily Housing officials told us they are moving toward a risk-based approach, using results from prior reviews and a property’s risk rating to determine how often to conduct these Management and Occupancy Reviews. However, Multifamily Housing officials told us that budget and staffing constraints continue to limit the number of reviews completed annually, with less than half of project-based rental assistance properties reviewed in 2017. To complete Management and Occupancy Reviews, HUD staff or contractors review documentation to monitor whether properties are adhering to requirements for receiving HUD funding and to target potential issues. This review gathers information on seven factors of property management, including management of a property’s physical condition. As part of gathering information, HUD staff or contract administrators interview the property owner or agent and may visit a sample of housing units to verify that deficiencies identified in the REAC inspection have been corrected. The Management and Occupancy Review specifies—in a summary report for owners and agents— corrective actions to take within targeted completion dates, not to exceed 30 days, based on the documentation review and on-site visit. Properties that perform poorly on the review also must provide proof of taking these actions. If properties do not provide proof of taking these corrective actions, Multifamily Housing can take some of the previously listed actions, such as changing the agent of a property. REAC’s inspection process annually identifies properties that are in poor physical condition and contain life threatening health and safety issues. With over 2 million moderate- and low-income households living in public housing or multifamily properties assisted or insured by HUD, it is imperative that these properties are decent, safe, sanitary, and in good repair. Our review of REAC found areas for improvement in its inspection process: Review of inspection process. A comprehensive review of the inspection process could help REAC identify risks and ensure it is meeting the goal specified in its strategic plan that inspections be reliable, replicable, and reasonable. Sampling errors in inspection scores. If REAC were to resume reporting on sampling errors and develop a process to address properties that fall below certain cutoff scores when the sampling error is taken into account, it would have the information it needs to identify properties that may require more frequent inspections or enforcement actions. Sampling methodology documentation. Comprehensive and organized documentation of the sampling methodology could help REAC preserve the institutional knowledge of important features of its inspection process, particularly when key staff leave the agency. Timing of housing inspections. Improvements in REAC’s on-time performance of multifamily property inspections could provide HUD with more timely information on the physical condition of these properties and the information it needs to take any enforcement actions. Further, by developing mechanisms to track its progress on meeting the schedule for inspections and improving its collection of data on why inspections are delayed, REAC could better determine what factors are contributing to delays in conducting inspections. Staffing inspections. A formal evaluation plan could help REAC determine if its pilot program for staffing inspections in difficult geographic areas is a success or whether changes are needed before moving from a pilot to a permanent process. Implementation of open recommendations. Taking timely actions on internal-review recommendations could help HUD to improve REAC’s inspection process and the safety of HUD-assisted properties. We also found areas for improvement in REAC’s processes for selecting, training, and overseeing contract and quality assurance inspectors: Inspector candidates’ qualifications. A more robust process for verifying contract inspectors’ qualifications could reduce the number of candidates with insufficient experience who participate in REAC’s training program, which could help REAC to expend fewer resources on training candidates who are unlikely to become successful inspectors. Contract inspector training. Evaluating the effectiveness of its training program for contract inspectors could help REAC better assess the quality of the program and plan for future training. Quality assurance inspector training. By developing and documenting training for quality assurance inspectors that encompasses all of their job responsibilities, REAC can better ensure that inspectors have the skills required to oversee contract inspectors. Continuing education requirements. Continuing education requirements for contract and quality assurance inspectors could help REAC ensure that inspectors are up-to-date on REAC policies and industry standards. Targets for reviews of contract inspectors. Improving its ability to meet management targets for CQA reviews and QCIs could help REAC better ensure that contract inspectors are receiving the feedback needed to improve their performance, thereby improving the quality of inspections. Formal policies for Quality Control group. By implementing policies and procedures for the Quality Control group, REAC can help ensure that the group achieves its objective of providing consistent reviews of quality assurance inspectors that will enable these inspectors to improve their oversight roles. Performance standards for quality assurance inspectors. Reviewing and updating REAC’s performance standards for quality assurance inspectors so that they align with their job duties can help REAC ensure that staff understand how their duties are prioritized within REAC’s mission and improve the quality of performance reviews. Finally, Multifamily Housing’s current practice of taking actions against property owners when the REAC score is 59 or below is inconsistent with the legal requirement to take action when the score is 60 or below. While in practice this affects very few properties, without either Congress changing the threshold identified in appropriations acts from 60 to 59 or HUD changing its practice to issue notices to properties that score 60 or below, HUD’s actions will continue to be inconsistent with the legal requirement. We are making the following 14 recommendations to HUD: The Deputy Assistant Secretary for the Real Estate Assessment Center should conduct a comprehensive review of the physical inspection process. (Recommendation 1) The Deputy Assistant Secretary for the Real Estate Assessment Center should resume calculating the sampling error associated with the physical inspection score for each property, identify what changes may be needed for HUD to use sampling error results, and consider those results when determining whether more frequent inspections or enforcement actions are needed. (Recommendation 2) The Deputy Assistant Secretary for the Real Estate Assessment Center should develop comprehensive and organized documentation of REAC’s sampling methodology and develop a process to ensure that documentation is maintained going forward. (Recommendation 3) The Deputy Assistant Secretary for the Real Estate Assessment Center should track on a routine basis whether REAC is conducting inspections of multifamily housing properties in accordance with federal guidelines for scheduling and coordinate with the Deputy Assistant Secretary for Multifamily Housing to minimize the number of properties that can cancel or reschedule their physical inspections. (Recommendation 4) The Deputy Assistant Secretary for the Real Estate Assessment Center should design and implement an evaluation plan to assess the effectiveness of the Indefinite Delivery/Indefinite Quantity pilot in ensuring timely and quality inspections for properties in hard-to-staff geographic areas. (Recommendation 5) The Deputy Assistant Secretary for Multifamily Housing and the Deputy Assistant Secretary for the Real Estate Assessment Center should expedite implementation of the recommendations from the Rapid Response and Resolution Team. (Recommendation 6) The Deputy Assistant Secretary for the Real Estate Assessment Center should follow through on REAC’s plan to create a process to verify candidate qualifications for contract inspectors—for example, by calling references and requesting documentation from candidates that supports their completion of 250 residential or commercial inspections. The plan should also consider whether certain types of inspections—such as Federal Emergency Management Agency inspections and U.S. Army Office of Housing inspections—satisfy REAC’s requirements. (Recommendation 7) The Deputy Assistant Secretary for the Real Estate Assessment Center should develop a process to evaluate the effectiveness of REAC’s training program—for example, by reviewing the results of tests or soliciting participant feedback. (Recommendation 8) The Deputy Assistant Secretary for the Real Estate Assessment Center should revise training for quality assurance inspectors to better reflect their job duties. Revised training should be documented, include expanded subject matter training, and address skills that may not be included in training for contract inspectors—for example, instructing contract inspector candidate trainings and coaching and providing feedback. (Recommendation 9) The Deputy Assistant Secretary for the Real Estate Assessment Center should develop continuing education requirements for contract and quality assurance inspectors. (Recommendation 10) The Deputy Assistant Secretary for the Real Estate Assessment Center should develop and implement a plan for meeting REAC’s management targets for the timeliness and frequency of CQA reviews and QCIs. The plan should include consideration of resources of and demands on quality assurance inspectors, including the effect of natural disasters and other special assignments. (Recommendation 11) The Deputy Assistant Secretary for the Real Estate Assessment Center should ensure that Quality Control’s policies and procedures for overseeing quality assurance inspectors are implemented. (Recommendation 12) The Deputy Assistant Secretary for the Real Estate Assessment Center should review quality assurance inspector performance standards and revise them to better reflect the skills and supporting behaviors that quality assurance inspectors need to effectively contribute to REAC’s mission. (Recommendation 13) The Deputy Assistant Secretary for Multifamily Housing should report to Congress on why the agency has not complied with the 2017 and 2018 Consolidated Appropriations Acts requirement to issue notices to properties when the REAC score is 60 or below, including seeking any statutory flexibilities or exceptions believed appropriate. (Recommendation 14) We provided a draft of this report to HUD for review and comment. In written comments, reproduced in appendix V, HUD agreed with 11 recommendations, partially agreed with 2, and neither agreed nor disagreed with 1. In its written comments, HUD noted that it largely agreed with the findings and has been examining how it can develop, pilot, and evaluate an alternate approach to its inspection model that will address the issues raised in our report. Consistent with our report, HUD recognized that after 20 years, its physical inspection process has become susceptible to manipulation. HUD said it plans to pilot a new physical inspection process in one of HUD’s administrative regions later this year. HUD stated that given its limited resources, it will be unable to simultaneously develop the new process and implement all of the recommendations to its current process. We maintain that implementing the recommendations will help REAC to ensure that properties are decent, safe, sanitary, and in good repair. HUD agreed with 11 recommendations and provided specific information about planned steps to implement them. For example, for our first recommendation on conducting a comprehensive review of REAC’s physical inspection process, HUD noted in its written comments that it plans to develop new standards, protocols, scoring approaches, and software to be validated through a demonstration. In addition, if resources are available, HUD plans to contract with an external vendor to assess the accuracy and effectiveness of the new inspection process and the statistical validity of scoring. For our eighth and ninth recommendations on evaluating and revising training for contract and quality assurance inspectors, HUD noted that it would evaluate its internal training program for contract inspectors as it pilots its new inspection process and compare the results with its evaluation of an outsourced training approach. In addition, HUD noted that it would identify the subject matter expertise needed for quality assurance inspectors and provide training to address any skills gaps among these inspectors. HUD partially agreed with our fourth and sixth recommendations and noted some considerations for addressing them. HUD partially agreed with our fourth recommendation regarding tracking its progress on conducting inspections of multifamily properties in accordance with federal guidelines, but did not identify the reason for its partial agreement. In written comments, HUD described actions it plans to take that we consider consistent with the intent of the recommendation. We maintain that this recommendation should be implemented to achieve benefits, including better understanding of the factors that contribute to inspection delays. HUD also partially agreed with our sixth recommendation regarding expedited implementation of recommendations from the Rapid Response and Resolution Team. In written comments, HUD noted that in order to balance resources invested in the current approach with those needed to design future operations, it would consider whether the remaining recommendations from the Rapid Response and Resolution Team fit with the new inspection model that it plans to pilot. Whether in the current inspection model or a future one, we maintain that expediting implementation of the recommendations from the Rapid Response and Resolution Team will support that team’s intention to address conditions at troubled multifamily properties. HUD neither agreed nor disagreed with our second recommendation to resume calculating the sampling error associated with the physical inspection for each property, identify the changes that may be needed for HUD to use sampling error results, and consider those results when determining whether more frequent inspections or enforcement actions are needed. In response to this recommendation, HUD noted in its written comments that it is examining resource implications, regulations and policies that would need to be changed, and the viability and effectiveness of making the changes included in our recommendations. We maintain that implementing this recommendation would improve REAC’s inspection process by identifying properties that may require more frequent inspections or enforcement actions. We are sending copies of this report to the appropriate congressional committees and the Secretary of Housing and Urban Development. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or GarciaDiazD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. This report examines (1) the Department of Housing and Urban Development’s (HUD) Real Estate Assessment Center’s (REAC) process for identifying physical deficiencies; (2) REAC’s processes for selecting, training, and developing contract and quality assurance inspectors; (3) REAC’s processes for monitoring contract and quality assurance inspectors; and (4) HUD’s monitoring and enforcement processes for addressing physical deficiencies and how REAC’s information is used to support these processes. To address the first objective, we reviewed regulations and policies and procedures related to REAC’s physical inspection process. Specifically, we reviewed the final notice on REAC’s physical inspection scores and the 2017 update to REAC’s compilation bulletin, which is the guidance document for inspectors conducting physical inspections. We also reviewed REAC’s user guide, which explains how REAC’s inspection software and handheld data collection devices are used to conduct the inspection and record deficiencies. To describe the quality assurance processes for physical inspections, we reviewed REAC’s quality assurance standard operating procedures, which provide instructions to REAC’s quality assurance inspectors on how they are to conduct various monitoring activities over contract inspectors to assess the quality of inspections. We also reviewed REAC’s standard operating procedures for post-inspection reviews. As part of our assessment of the physical inspection process, we reviewed the statistical methodology used by REAC to determine the sample size for dwelling units and buildings. We reviewed REAC’s documentation describing the sample-size calculations for units and buildings and interviewed a REAC statistician to obtain information on the statistical approach and assumptions used in the sample size calculations. With this information, we were able to conduct our own calculations on the sample-size and compare our results to REAC’s. To report on the number of physical inspections conducted from fiscal years 2013 through 2017, as well as other data on inspections over this period, we accessed REAC’s Record and Process Inspection Data database. This database contains information related to physical inspections, such as the types and locations of properties inspected, dates of inspection, and inspection scores. To assess the reliability of the database, we first identified the various tables in the database that held the relevant data we needed for our analysis. We also identified the common identifier in each of these tables to construct records of inspections with the relevant data. We met with REAC’s staff to confirm that our selection of the tables and our construction of records was correct. We then performed our analysis and developed various descriptive statistics, such as the number of inspections per year by property type from fiscal years 2013 through 2017, the number of multifamily properties that failed their REAC inspection (scored below 60) for fiscal years 2013 through 2017, the percentage of multifamily property inspections that occurred on time given their inspection score, and various inspection score ranges by state. We compared our statistics on the number of inspections per year with comparable statistics developed by REAC. In cases where we had differences, we obtained explanations from REAC for these differences and revised our analysis where appropriate. Based on our overall assessment of the REAC data we used, we found them to be sufficiently reliable for analyzing the number and timing of inspections and trends in scoring. To obtain the views of various stakeholders on the inspection process, we held discussion groups with contract inspectors and REAC’s quality assurance inspectors and supervisors. Each discussion group had between 6 and 13 participants and was facilitated by a GAO staff member. We covered a number of topics in these discussion groups, including the inspection and quality assurance processes. We held one discussion group with contract inspectors, three with REAC quality assurance inspectors, and one with REAC quality assurance supervisors: Contract inspectors. For the discussion group with the contract inspectors, we invited all of the contract inspectors who were attending a conference at REAC’s headquarters in Washington, D.C. Thirteen contract inspectors attended the discussion group. Quality assurance inspectors. For the discussion groups with the quality assurance inspectors, we reached out to all quality assurance staff and coordinated with REAC to arrange specific meeting times to maximize the number of participants. We held two separate discussion groups with experienced inspectors. The first of these groups had 11 participants, and the second group had 6. We also held a separate discussion group with 11 newly hired quality assurance inspectors. Quality assurance supervisors. For the last discussion group, we reached out to all quality assurance supervisors and met with 6 of them. We recorded all of the discussion groups to help transcribe the conversations. In order to analyze the discussion group transcripts, we identified phrases that represented key themes across the groups. One GAO analyst reviewed one of the transcripts to identify any additional phrases we should add to our analysis. Once we arrived at our final set of key themes, one GAO analyst reviewed all of the transcripts and matched responses in the transcripts to the key themes. A second GAO analyst then checked the work to determine if he agreed with the coding of the first analyst. If there were any disagreements on the coding, the two analysts met to reach consensus on the appropriate coding. Finally, to obtain the perspectives of property owners on REAC’s inspection process, we met with four organizations representing multifamily or public housing property owners. These organizations were the Council for Large Public Housing Authorities, the National Affordable Housing Management Association, the National Leased Housing Association, and the Public Housing Authorities Directors Association. Also, to understand how private home inspection associations developed their inspection processes, we interviewed staff from the American Society of Home Inspectors and the International Association of Certified Home Inspectors. To address the second and third objectives, we reviewed REAC’s policies and procedures for selecting, training, developing, and monitoring contract and quality assurance inspectors. We reviewed the contract inspector candidate assessment questionnaire and construction analyst job announcement, which describe the requirements to become a contract and quality assurance inspector, respectively. We also reviewed documents describing the online (Phase Ia), classroom (Phase Ib), and field (Phase II) training courses. We also reviewed an assessment that Deloitte, a management consultant firm, conducted of REAC’s training, quality assurance, and inspector oversight processes. We compared REAC’s training processes for inspectors with key attributes of effective training and development programs. In our discussion groups with contract and quality assurance inspectors, we also asked their views on REAC’s selection, training, and monitoring processes. In addition, we interviewed REAC management officials to discuss their processes for the selection, training, monitoring, and oversight of contract and quality assurance inspectors. We spoke with staff from the American Society of Home Inspectors and the International Association of Certified Home Inspectors to understand how their selection and training requirements for inspectors who are members of home inspection associations compared with REAC’s. To examine REAC’s processes for monitoring contract inspector performance, we reviewed REAC’s quality assurance standard operating procedures, REAC’s strategic plan, and various tools REAC has developed to assess how contract and quality assurance inspectors perform relative to their peers. We obtained data on collaborative quality assurance reviews for fiscal years 2013 through 2017 and data on quality control inspections for January 2017 through June 2018. We analyzed the data to determine, for example, how often contract inspectors were conducting inspections in accordance with REAC’s Uniform Physical Conditions Standards protocol and its quality assurance standard operating procedures, and how often REAC was meeting its goals for timeliness and frequency of reviews. We assessed the reliability of the data by interviewing knowledgeable officials and conducting manual testing on relevant data fields for obvious errors. Based on these steps, we found the data to be sufficiently reliable for the purposes of our analyses. To examine REAC’s processes for monitoring and overseeing quality assurance inspector performance, we reviewed the performance standards and performance elements REAC uses to evaluate quality assurance inspectors. We interviewed staff from REAC’s Quality Control department, which conducts inspection reviews on quality assurance inspectors. We compared REAC’s performance management processes to key practices we have identified for effective performance management. We also compared REAC’s policies for oversight and monitoring of quality assurance inspectors to criteria in Standards for Internal Control in the Federal Government. To address the fourth objective, we reviewed documentation related to monitoring and enforcement processes for HUD’s Office of Multifamily Housing (Multifamily Housing), Office of Public and Indian Housing (PIH), and the Departmental Enforcement Center. For example, we reviewed relevant protocols and guidance documents on PIH’s and Multifamily Housing’s processes to address physical risk, among other risks. We also reviewed the relevant legal authorities in the 2014 through 2018 Consolidated Appropriations Acts and federal regulations for these HUD program offices to take enforcement actions for properties with physical deficiencies. We interviewed officials from Multifamily Housing, PIH, and the Departmental Enforcement Center on their processes to monitor the physical condition of properties and take enforcement actions. We further selected two Multifamily Housing and four PIH field offices throughout the United States to understand actions they take to monitor properties and ensure that physical deficiencies are corrected. We developed a two- stage process to select field offices with a higher percentage of inspections with scores 70 and below. We first selected HUD regions based on our score criteria and then selected specific field offices within those regions using similar score criteria. Because we selected a nonprobability sample of field offices, the information we obtained cannot be generalized more broadly to all field offices. However, the information provides important context and insight into how the enforcement process for physical deficiencies works across the country. In addition, we obtained data on performance designations for public housing agencies within PIH, actions taken by Multifamily Housing for properties scoring below 60 on the REAC inspection, and actions taken by the Departmental Enforcement Center for Multifamily Housing properties. We assessed the reliability of the data by reviewing relevant HUD guidance and obtaining written responses from agency officials on how the data were collected, maintained, analyzed, and presented. Based on these steps, we found the data to be sufficiently reliable for the purposes of our analyses. Finally, we reviewed prior reports from GAO and from the HUD Office of Inspector General that discussed efforts to monitor the physical condition of properties, among other conditions. We conducted this performance audit from July 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Real Estate Assessment Center (REAC) conducted 44,486 inspections of Office of Multifamily Housing (Multifamily Housing) properties from fiscal years 2013 through 2017, according to our analysis of REAC’s inspection data. Properties received a score from 0 to 100, with a score below 60 considered as failing. Table 7 shows the percentage of inspections conducted in each state across three score ranges. States varied in the percentage of inspections that fell within the score range considered failing (0 to 59), from a low of 1 percent to a high of 10 percent. REAC inspects properties with lower scores more frequently than properties with higher scores. For example, properties that scored below 80 would have been inspected annually over this period, while properties that scored 90 or above would have been inspected every 3 years. The Real Estate Assessment Center (REAC) conducted 15,156 inspections of public housing properties from fiscal years 2013 through 2017, according to our analysis of REAC’s inspection data. Properties received a score from 0 to 100, with a score below 60 considered as failing. Table 8 shows the percentage of inspections conducted in states or U.S. territories across three score ranges. States varied in the percentage of inspections that fell within the score range considered failing (scores 0 to 59), from a low of 1 percent to a high of 34 percent. REAC generally inspects properties with lower scores more frequently than properties with higher scores. According to our analysis, REAC conducted fewer than 100 inspections of public housing properties in 18 states or territories. Table 9 shows the number of inspections conducted within three score ranges for these 18 states or territories. The Rapid Response and Resolution Team was created by the Department of Housing and Urban Development (HUD) in May 2016 to address troubled multifamily properties by improving HUD’s internal processes for assessing properties and analyzing risk so that properties do not become troubled; improving HUD’s processes for inspecting properties so that troubled ones are identified earlier and more reliably and communicating the results to stakeholders; and improving HUD’s processes for enforcing corrective actions and resolving troubled properties and working with owners so that HUD resources are used only on safe and healthy housing. The team consisted of staff from the Real Estate Assessment Center (REAC) and other units within HUD, including the Office of Multifamily Housing (Multifamily Housing). In January 2017, the team presented 31 recommendations, 8 of which were specific to REAC. As of December 2018, REAC had not yet implemented any of these recommendations. REAC had reached concurrence with Multifamily Housing on 3 of these recommendations and asked for Multifamily Housing’s consideration of the funding and rulemaking requirements for the remaining 5 recommendations. The 8 recommendations that were specific to REAC are as follows: 1. Implement a risk-based exigent health and safety abatement verification policy. 2. Inspect properties that have a REAC physical inspection score of less than 60 after a 3-day notice. 3. Increase the scoring weights of units and reexamine point deduction caps. 4. Expand photo capability in the inspection process to level 1 and level 2 deficiencies and a panoramic photo of the property. 5. Inspect carbon monoxide detectors in the inspection process. 6. Develop health and safety abatement requirements, including focusing on water ponding and missing lead-based paint disclosure forms and inspection reports. 7. Take enforcement action to protect tenants before the 45-day appeal period is over for properties that score under 30 points and that have exigent health and safety deficiencies. 8. Require electronic exigent health and safety certifications and abatements within 24 hours of the inspection. In addition to the contact named above, Andy Pauline (Assistant Director), José R. Peña (Analyst in Charge), Carl Barden, Chloe Brown, Hannah Dodd, Juan Garcia, Jeff Harner, Emily Hutz, Jill Lacey, Jerry Sandau, Jessica Sandler, Jennifer Schwartz, and Jena Sinkfield made key contributions to this report.", "summary": "Over 2 million low- and moderate-income households live in HUD-assisted (subsidized) or -insured multifamily housing. HUD's REAC uses contractors to inspect the physical condition of these properties to determine that they are decent, safe, sanitary, and in good repair. The 2017 Consolidated Appropriations Act, Joint Explanatory Statement, included a provision for GAO to review REAC's policies and processes. This report discusses, among other things, (1) REAC's process for identifying physical deficiencies and (2) REAC's selection, training, and monitoring of contract inspectors and its own quality assurance inspectors. GAO reviewed HUD documents and data related to REAC's physical inspection process, use of contract and quality assurance inspectors, and enforcement processes. GAO also interviewed HUD officials and housing industry stakeholder groups and conducted discussion groups with contract and quality assurance inspectors. The Department of Housing and Urban Development's (HUD) Real Estate Assessment Center's (REAC) standardized process to identify physical deficiencies at HUD multifamily properties (including public housing) has some weaknesses. For example, REAC has not conducted a comprehensive review of its inspection process since 2001, even though new risks to its process have emerged, such as property owners misrepresenting the conditions of their properties. A comprehensive review could help REAC identify risks and ensure it is meeting the goal of producing inspections that are reliable, replicable, and reasonable. In addition, REAC does not track its progress toward meeting its inspection schedule for certain properties, which could hinder HUD's ability to take enforcement actions. Finally, in the wake of concerns that inspections were not always identifying troubled properties, REAC and other HUD units, including the Office of Multifamily Housing, made eight recommendations in January 2017 to enhance the inspection process, but HUD had only approved three of these recommendations and had not implemented any of them as of December 2018. REAC uses contractors to inspect properties; these contract inspectors are trained and overseen by quality assurance inspectors hired directly by REAC. However, REAC's processes to select, train, and monitor both contract inspectors and quality assurance inspectors have weaknesses. Selection. REAC does not verify the qualifications of contract inspector candidates before they are selected to begin training to become certified inspectors. Formal processes to verify qualifications may help REAC identify unqualified candidates before they begin training and avoid expending resources on training these candidates. Training. REAC lacks formal mechanisms to assess the effectiveness of its training program for contract and quality assurance inspectors. In addition, unlike other professional inspection organizations, REAC does not have continuing education requirements. Formal mechanisms to assess the effectiveness of its training program could help REAC ensure that its program supports the development needs of inspectors. Further, requiring continuing education could help REAC ensure that inspectors are current on any changes in REAC's policies or industry standards. Monitoring. REAC has not met management targets for the number and timeliness of its inspection oversight reviews of contract inspectors. For example, REAC has not met its target of conducting three quality assurance reviews of poor-performing contractors per quarter. As a result, if deficiencies are not identified and recorded by contract inspectors, they may not be addressed in a timely manner. In addition, REAC's performance standards for its quality assurance inspectors have not been updated to reflect their broader job duties, such as conducting inspector oversight reviews and coaching and mentoring contract inspectors. Performance standards that are directly linked to these job duties would help ensure that inspectors are assessed on all of their key responsibilities. GAO makes 14 recommendations to HUD to improve REAC's physical inspection process and its selection, training, and monitoring of contract and quality assurance inspectors, among other things. HUD agreed with 11 recommendations, partially agreed with 2, and neither agreed nor disagreed with 1. GAO maintains that its recommendations should be fully addressed to improve the inspection process.", "document_type": "gao"}
{"report": "DOD has completed three additional statutory requirements of section 911 since our August 2019 report, but has not completed three remaining requirements, as shown in table 2. We previously reported that DOD had completed four of the statutory requirements, specifically awarding a contract for a study to determine how to best implement cross-functional teams, providing the results of the study to Congress, establishing any cross-functional teams to address critical department objectives and outputs, and reporting to Congress on the establishment of the cross- functional teams. Thus, in total, DOD has completed seven of the 10 statutory requirements. For more detail on all 10 statutory requirements, see appendix II. On October 29, 2019, the Secretary of Defense approved DOD’s organizational strategy. In preparing the strategy for review and approval, OCMO obtained input on the draft organizational strategy from other DOD and OSD components, consistent with a recommendation from our February 2018 report that OCMO obtain stakeholder input on the development of the organizational strategy. We found that the strategy addresses key requirements of section 911, including identifying critical objectives that would benefit from the use of cross-functional teams and providing for the appropriate use of these teams. As part of the organizational strategy, DOD also identified the actions it has taken to streamline the organizational structure and processes of the Office of the Secretary of Defense, another requirement of section 911. For example, the strategy states that DOD has delegated authority to approve certain global force management actions to the Chairman of the Joint Chiefs of Staff and delegated certain acquisition oversight functions to the military departments. Further, consistent with our recommendation from our February 2018 report that the CMO address how the department will promote and achieve a collaborative culture, the organizational strategy includes a short reference to our leading practices for mergers and organizational transformations. However, while the approved organizational strategy cites the leading practices, it does not include specific implementation steps that explain how DOD will follow these practices. Earlier drafts of the organizational strategy that we had reviewed included more specific implementation steps, but those steps were removed during the internal DOD review process. As we reported in August 2019, a January 2019 draft of the organizational strategy included practical implementation steps DOD planned to take to advance a collaborative culture, each of which were shown to align with our leading practices for mergers and organizational transformations. For example, consistent with the leading practice for establishing a coherent mission and integrated strategic goals to guide the transformation, the January 2019 draft proposed that the CMO develop an implementation plan with goals and milestones for its efforts to implement the organizational strategy, communicate those goals and milestones, and report periodically on the achievement of the goals. However, in place of these specific steps, the approved organizational strategy simply lists these leading practices and makes a broad statement that DOD is committed to further incorporating and institutionalizing these practices at every opportunity. An OCMO official told us these implementation steps were removed as the OCMO prepared the draft for department-wide coordination and submission to the Secretary of Defense for review and approval. According to that official, OCMO officials made this change because the Secretary and Deputy Secretary of Defense were newly confirmed, and OCMO officials did not want to commit them to a specific course of action. That official also told us that DOD might use its senior leadership forums, such as the Deputy’s Management Action Group (DMAG), to monitor implementation of the organizational strategy and identify opportunities to improve collaboration, including implementation of our leading practices. However, the official acknowledged that any plan to use such forums for monitoring implementation has not been finalized. As we stated in making our February 2018 recommendation that the department address how it would promote and achieve a collaborative culture, section 911 identified several outcomes that DOD should achieve to advance such a culture. We also noted that DOD could use our leading practices for mergers and organizational transformations to address how the department will advance a culture that is collaborative, team-oriented, results-driven, and innovative. We further stated that DOD would be better positioned to transform and meet its mission if it incorporated these leading practices in its organizational strategy as a way to better articulate how the department will achieve the outcomes that advance a collaborative culture across DOD and address the requirements of section 911. Specific implementation steps like those included in earlier drafts of the organizational strategy offered the department a clear path forward for pursuing the goals of section 911 and promoting a collaborative culture. Absent these steps, such as developing an implementation plan with goals and milestones, it is less clear how DOD intends to implement the organizational strategy and assess progress toward its goals. Identifying and documenting specific implementation steps to advance a collaborative culture—such as those OCMO included in earlier drafts of the organizational strategy—is necessary to fully address the requirements of section 911. On December 12, 2019, the Secretary of Defense approved DOD’s guidance on cross-functional teams. We found that this two-page guidance addresses most, but not all, of the 911 requirements and leading practices for cross-functional teams. Specifically, it addresses in whole or in part six of the seven section 911 requirements and six of the eight leading practices. The Secretary-approved guidance also directs the CMO to develop more detailed implementing guidance. It will be important for the CMO to develop and issue this detailed implementing guidance to fully address section 911 requirements and our leading practices for effective cross- functional teams, consistent with a recommendation in our February 2018 report. According to an OCMO official, OCMO plans to use previously drafted terms of reference as the basis for the CMO’s more detailed implementing guidance. Based on our review, when the Secretary of Defense approved guidance is considered along with the draft terms of reference expected to serve as detailed implementing guidance, both documents will fully address all section 911 requirements and leading practices for effective cross-functional teams. We will monitor the department’s progress in issuing this guidance as part of our normal process of assessing DOD’s efforts to implement our recommendations. DOD has not approved its curriculum for training for cross-functional team members and their supervisors. In February 2018, we reported that DOD’s draft curriculum for cross-functional team members and their supervisors addressed the section 911 requirements for that training. We reported in August 2019 that DOD had provided required training using its draft curriculum to members of the EMSO team—DOD’s only established section 911 team at the time—but had not provided training to their supervisors. According to DOD’s comments on our August 2019 report, DOD expected the draft curricula for training for cross-functional team members and their supervisors to have been approved simultaneously with the issuance of the Secretary’s guidance on cross-functional teams. According to OCMO officials, however, DOD has contracted for the delivery of the required training for cross-functional team members and their supervisors. One of those officials also told us OCMO now expects that training to be completed in 2020. Another OCMO official told us that the OCMO has been further refining the draft curriculum based on feedback from the members of the EMSO team and external experts before submitting the curriculum for review and approval. DOD has not provided required training on cross-functional teams and related subjects to presidential appointees and the curriculum has not been approved. Section 911 required presidentially appointed, Senate- confirmed officials to receive training on leadership, modern organizational practice, collaboration, and the operation of cross- functional teams within 3 months of their appointment or to receive a waiver from the President of the United States. As of October 2019, 23 of 36 such positions had been filled and the officials had been in their positions for more than 3 months; none had received the statutorily mandated training. According to DOD’s comments on our August 2019 report, DOD expected the draft training curricula for presidential appointees to have been approved simultaneously with the issuance of the Secretary’s guidance on cross-functional teams. According to OCMO officials, however, DOD has contracted for the delivery of the required training for presidential appointees. One of those officials also told us they now expect that training to be provided in 2020. An OCMO official told us that OCMO has been further refining the draft curriculum and discussing possible venues for providing this training for presidential appointees, including one of the weekly meetings that the Deputy Secretary of Defense has with all Office of the Secretary of Defense presidential appointees. DOD has not completed the required analysis of the successes and failures of its cross-functional teams. Section 911 requires DOD’s analysis to be completed with support from external experts in organizational and management sciences within 18 months of the establishment of the first cross-functional team under section 911. Because the first cross-functional team was established in August 2017, this analysis was due in February 2019. According to OCMO officials, DOD has contracted with an organization to help develop the analysis. One of the officials also told us DOD expects the analysis to be completed in 2020. Another OCMO official told us in December 2019 that work on the assessment, including a survey and structured interviews, was underway, and that an initial draft report was expected by the end of December 2019. DOD’s EMSO team is continuing to work toward its mission to develop requirements and specific plans to improve EMSO capabilities across the department and to achieve operational superiority. The team is developing 13 initiatives in four areas—governance, organization, capabilities and gaps, and training and readiness. In addition, the team issued a report required by section 1053 of the John S. McCain National Defense Authorization Act for Fiscal Year 2019, which included the mandated assessments of the electronic warfare capabilities of the Russian Federation and the People’s Republic of China in consultation with the Director of the Defense Intelligence Agency. Section 1053 also required the team to, among other things, update the department’s Electronic Warfare Strategy in coordination with the Electronic Warfare Executive Committee. According to an EMSO official, the team is developing a new strategy, which is scheduled for completion in June 2020. The EMSO team is continuing to demonstrate leading practices for effective cross-functional teams, similar to what we reported in August 2019. Specifically, at that time, we reported that the EMSO team was demonstrating leading practices for effective cross-functional teams, such as a well-defined team structure and well-defined team goals. Based on the results of our recent survey of individual team members, most team members believe the team is demonstrating open and regular communication, an inclusive team environment, has an empowered cross-functional team leader, and has well-defined team goals. However, our survey results show that team members responded less favorably to questions about senior management support than to questions about the other leading practices. Specifically, less than half of the respondents agreed with the statements that DOD’s senior leadership provides the team with sufficient resources for its work, supported the team as a priority, and supported the team’s goals and objectives. In their survey responses and during interviews, team members expressed their concerns about the lack of resources, such as funding and sufficient office space to perform their work. According to team officials, they are continuing to work with the OCMO to resolve the team’s resource issues. In our August 2019 report, we stated that the team’s progress was negatively affected by funding delays resulting from disagreements among senior leadership over the responsibility for funding the team. The disagreement had been resolved for fiscal year 2019, but had not yet been resolved for future fiscal years. We recommended, and DOD concurred, that the CMO and EMSO cross- functional team clarify roles and responsibilities for providing administrative support and funding for the team beyond fiscal year 2019 in accordance with the memorandum establishing the team. According to EMSO team officials, however, funding for future years has not been identified. The team has discussed its funding needs with the OCMO and staff from the Office of the Deputy Secretary of Defense, but there is still no clarity regarding responsibility for funding the team. The team’s budget submission as part of DOD’s fiscal year 2021 budget process was withdrawn because the amount requested was smaller than the amounts typically reviewed in the process. According to an EMSO team official, the OCMO is providing funding for the team incrementally on a quarterly basis, and is facing challenges with funding the team’s request for a contract due to the department operating under a continuing resolution. According to another EMSO team official, the team is maintaining the status quo with its current funding and is not considering any additional initiatives. We also reported in our August 2019 report that, according to a team official, while the team had its own office space, the space did not have the level of security required to allow the team to work on a third of its initiatives. Team officials have since told us the team plans to move to an appropriately secure space in early 2020. We will continue to monitor DOD’s progress to secure resources and office space for the team as part of our normal process of assessing DOD’s efforts to implement our recommendations. In its approved organizational strategy, DOD identified two existing task forces to expand the number of cross-functional teams. First, the Secretary of Defense established the Close Combat Lethality Task Force in February 2018 to develop, evaluate, recommend, and implement improvements to U.S. squad-level infantry combat formations and strengthen the combat, lethality, survivability, resiliency, and readiness of infantry squads. Second, the Secretary of Defense established the Protecting Critical Technology Task Force in October 2018 to address concerns over the security of the department’s critical technology and the loss of classified information and controlled unclassified information that puts DOD’s investments at risk and erodes the lethality and survivability of U.S. forces. According to an OCMO official, the OCMO updated the organizational strategy and designated these two task forces as cross- functional teams as a result of input from DOD senior leadership during their review of the draft organizational strategy. Based on our review of the documents used to establish the two task forces, we found that they would meet only some of the requirements we reviewed for cross-functional teams as mandated by section 911. For example, we found that the documentation for the Close Combat Lethality Task Force, as it was established, shows that the task force has a clearly established objective; is directed to develop recommendations such as policy changes and investment decisions; and is directed to make decisions on cross-functional issues—some of the key section 911 requirements. Similarly, we found that the documentation for the Protecting Critical Technology Task Force, as established, shows that the director of the task force has the authority to select the membership from across the department, another key requirement. However, based on the documents we reviewed, we found that the teams would not meet other requirements. For example, we found that the documentation for both teams did not ensure that those team members and leaders who are supervisors receive training in elements of successful cross-functional teams. According to an OCMO official, the OCMO will ensure that these task forces identified as cross-functional teams meet the requirements of section 911. For example, the OCMO will provide the required training. DOD’s newly issued guidance on cross-functional teams could help ensure that existing and any new cross-functional teams meet section 911 requirements. In addition, it could help provide these existing and any new teams with the information, direction, and authority they need to comply with mandated requirements for cross-functional teams. For example, section 911 permits the Secretary to delegate to cross- functional teams any decision-making authority that the Secretary considers appropriate to achieve the objectives of the teams; DOD’s guidance delineates the decision-making authority of cross-functional teams. More than 3 years after the passage of the National Defense Authorization Act for Fiscal Year 2017, DOD has begun to take key steps to address the requirements of section 911 and to promote a more collaborative culture in the department, including issuing its organizational strategy and making greater use of cross-functional teams under the act. Even as it has taken these steps, challenges for the departments’ ongoing implementation of section 911 remain. The department has still not addressed key requirements to help promote a collaborative culture and, according to officials, still has not identified responsibility for funding one of its cross-functional teams established under section 911. Further, specific implementation steps that would have offered the department a clear path forward for pursuing the goals of section 911 and promoting a collaborative culture at DOD were removed from DOD’s approved organizational strategy—a disappointing development. Identifying and documenting specific implementation steps to encourage a collaborative culture is necessary to fully address the requirements of section 911 and encourage such a culture. The Secretary of Defense should ensure that the Chief Management Officer identify and document specific implementation steps to advance a collaborative culture, consistent with our leading practices for mergers and organizational transformations. We provided a draft of this report to DOD for review and comment. In its written comments, which are reproduced in Appendix IV, DOD concurred with our recommendation. DOD also provided additional information on the steps that DOD has taken or plans to take to advance a collaborative culture, such as the Secretary and Deputy Secretary of Defense’s use of DOD’s senior governance forums to encourage collaboration across the department. DOD also stated that it plans to incorporate policies based on best practices for cultivating a collaborative organizational climate into the CMO’s guidance on the implementation of cross-functional teams as well as future National Defense Strategies and National Defense Business Operations Plans. We are sending copies of this report to the appropriate congressional committees and to the Secretary of Defense and Chief Management Officer. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this are listed in appendix V. Section 911 of the NDAA for Fiscal Year 2017 included a provision for us—every 6 months after the date of enactment on December 23, 2016, through December 31, 2019—to submit to the congressional defense committees a report. Each report is to set forth a comprehensive assessment of the actions that DOD has taken pursuant to section 911 during each 6-month period and cumulatively since the NDAA’s enactment. Table 3 identifies our five prior reports on DOD’s implementation of section 911 and the status of the 11 recommendations from those reports. Section 911 of the National Defense Authorization Act for Fiscal Year 2017 requires the Secretary of Defense to take several actions. Table 4 summarizes these requirements, the due date, and the date completed, if applicable, as of December 2019. In February 2018, we reported on eight leading practices for implementing effective cross-functional teams. Table 5 identifies these leading practices and their related key characteristics. In addition to the contact named above, Margaret Best (Assistant Director), Daniel Ramsey (Analyst-in-Charge), Sierra Hicks, Alexa Kelly, Richard Powelson, and Paulina Reaves made key contributions to this report. Other contributors included Tracy Barnes, Arkelga Braxton, Michael Holland, Ned Malone, Judy McCloskey, Jeremy Rogers, Ron Schwenn, and Sarah Veale.", "summary": "DOD has had longstanding organizational and management challenges that hinder collaboration. Section 911 of the NDAA for Fiscal Year 2017 directed the Secretary of Defense to, among other things, issue an organizational strategy that identifies critical objectives that span multiple functional boundaries, establish cross-functional teams to support this strategy, and provide related guidance and training. The NDAA for Fiscal Year 2017 also included a provision for GAO periodically to assess DOD's actions in response to section 911. GAO has issued a series of reports since June 2017 and made a number of recommendations to DOD. This report assesses the extent to which DOD has made progress in (1) implementing the requirements of section 911 and (2) establishing cross-functional teams. GAO reviewed documentation, interviewed cross-functional team members and other DOD officials, and compared DOD's actions to section 911 requirements and leading practices for cross-functional teams. Since GAO's August 2019 report, the Department of Defense (DOD) has taken actions to complete three additional statutory requirements of section 911 of the National Defense Authorization Act (NDAA) for Fiscal Year 2017, but has not completed three remaining requirements. These requirements are intended to support cross-functional teams and to promote department-wide collaboration (see table). Cross-functional teams rely on individuals with different types of expertise to work toward a common, well-defined goal, and are thought to deliver better and faster solutions to complex and fast-moving problems. DOD's approved organizational strategy addresses key requirements of section 911, including identifying critical objectives that would benefit from the use of cross-functional teams and providing for the appropriate use of these teams. However, the strategy did not include practical, specific implementation steps to guide DOD's efforts to advance a collaborative culture, which had been included in earlier draft versions of the strategy. These steps had aligned with GAO's leading practices for mergers and organizational transformations. Specific implementation steps like those included in earlier drafts of the organizational strategy offered DOD a clear path forward for pursuing the goals of section 911 and for promoting a collaborative culture. Absent identifying and documenting specific implementation steps, it is less clear how DOD intends to implement the organizational strategy and assess progress toward its goals. DOD's existing cross-functional team charged with improving electromagnetic spectrum operations and defending its communication systems from attacks is continuing its work by issuing a statutorily mandated report, among other efforts, but DOD has not clarified responsibility for funding the team. GAO will continue to monitor DOD's progress toward providing such support to the team as GAO recommended in August 2019. In addition, DOD has designated the Close Combat Lethality Task Force and the Protecting Critical Technology Task Force as new cross-functional teams, although they meet only some of the section 911 requirements. DOD officials said they will ensure that the newly designated teams meet these requirements, including providing required training. In this report, GAO recommends that DOD identify and document specific implementation steps to advance a collaborative culture, consistent with GAO's leading practices. GAO also reiterates the importance of addressing its prior recommendations. DOD concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "Sanctions provide a range of tools that Congress and the President may use to seek to alter or deter the behavior of a foreign government, individual, or entity in furtherance of U.S. national security or foreign policy objectives. For example, sanctions may be used in response to human rights abuses, weapons proliferation, or occupation of a foreign country; ultimately seeking to change the behavior of those perpetrating these offenses. Sanctions may include actions such as limiting trade; blocking assets and interest in assets subject to U.S. jurisdiction; limiting access to the U.S. financial system, including limiting or prohibiting transactions involving U.S. individuals and businesses; restricting private and government loans, investments, insurance, and underwriting; and denying foreign assistance and government procurement contracts. Sanctions can be comprehensive or targeted. Comprehensive sanctions. Generally, comprehensive sanctions include broad-based trade restrictions and prohibit commercial activity with an entire country. Examples of comprehensive sanctions include U.S. sanctions against Iran and Cuba. Targeted sanctions. Targeted sanctions restrict transactions of and with specific persons or entities. For example, the U.S. sanctions program related to Somalia targets persons engaging in acts threatening the peace, security, or stability of Somalia. Sectoral sanctions are a form of targeted sanctions directed at a specified sector, or sectors, of a target’s economy. For instance, Executive Order 13662 authorized sanctions targeting certain sectors of the Russian economy as might later be determined by the Secretary of the Treasury in consultation with the Secretary of State, such as the financial services, energy, mining, and defense and related materiel sectors. The United States also uses supplementary sanctions, known as secondary sanctions, which target third-party actors doing business with, supporting, or facilitating targeted regimes, persons, and organizations. For example, in February 2017, Treasury imposed sanctions against 13 individuals and 12 entities, including persons outside Iran, for their involvement in or support for Iran’s ballistic missile program, as well as for acting for or on behalf of, or providing support to, Iran’s Islamic Revolutionary Guard Corps-Qods Force. There are currently 20 country-based or country-related sanctions programs, according to lists of sanctions programs published by Treasury and State. The sanctions may target the governments of these countries or individuals and entities. Figure 1 shows country-based and country- related U.S. sanctions programs as of July 2019. Treasury, State, and Commerce, as well as various other U.S. agencies, play roles in implementing sanctions. Treasury implements sanctions by taking actions such as designating entities for the application of sanctions. These sanctions may include blocking entities’ access to U.S.-based assets, prohibiting them from engaging in financial transactions in the United States, and restricting access to U.S. financial services. Treasury’s Office of Foreign Assets Control (OFAC), which is part of the Office of Terrorism and Financial Intelligence (TFI), has primary responsibility for Treasury’s sanctions implementation, according to Treasury. TFI is charged with safeguarding the U.S. financial system against illicit use and combating rogue nations, terrorist facilitators, weapons of mass destruction proliferators, money launderers, drug kingpins, and other national security threats. As part of its implementation of sanctions, OFAC publishes a list, known as the Specially Designated Nationals List, of individuals, groups, and entities whose assets in the United States are blocked and with which U.S. persons are prohibited from dealing. The addition of an individual, group, or entity to this list is referred to as a sanctions designation. Entities or groups listed include those owned or controlled by, or acting for or on behalf of, targeted country governments. OFAC also lists individuals, groups, and entities, such as terrorists and narcotics traffickers, designated under targeted sanctions programs that are not country specific. OFAC may also issue licenses, general or specific, to permit activities that would otherwise be prohibited under a sanction. For example, OFAC has issued a general license to allow nongovernmental organizations to engage in not-for-profit activities in Syria in support of humanitarian projects, democracy-building, education, and noncommercial development projects directly benefitting the Syrian people. According to Treasury, OFAC participates in all aspects of sanctions implementation, including, targeting, outreach to the public, and compliance. OFAC also enforces sanctions by conducting civil investigations of sanctions violators and working with law enforcement agencies. State implements economic and other sanctions through a variety of actions, such as implementing sanctions-related controls on defense exports, restricting foreign aid, implementing arms embargoes pursuant to United Nations Security Council resolutions, and restricting visas. State’s primary sanctions coordination office is the Office of Economic Sanctions Policy and Implementation (SPI), which is part of the Division for Counter Threat Finance and Sanctions in State’s Bureau of Economic and Business Affairs. According to State, SPI is responsible for developing and implementing foreign policy–related sanctions adopted to counter threats to national security posed by particular activities and countries. In addition, according to State, SPI builds international support for implementing sanctions, provides foreign policy guidance to Treasury’s OFAC and Commerce’s Bureau of Industry and Security on sanctions implementation, and works with Congress to draft legislation that advances U.S. foreign policy goals in these areas. Further, according to State, SPI works to remove sanctions when appropriate to reward and incentivize improved behavior or demonstrate U.S. support for newly established democratic governments. Although SPI is State’s primary sanctions coordinating office, other State bureaus, offices, and overseas posts may have significant roles in sanctions implementation, depending on the sanctions program. Some functional bureaus interact with OFAC within their areas of expertise. For example, according to State, the Bureau of International Security and Nonproliferation has expertise on missile, chemical, and biological proliferation as well as how to counter proliferation. The bureau assists in developing sanctions programs and designating sanctions targets under nonproliferation law, according to State. Also, the Bureau of Counterterrorism and Countering Violent Extremism takes part in developing and evaluating sanctions policy as well as helping target entities for sanctions under various authorities, including an executive order targeting those that commit or support terrorism and the Foreign Terrorist Organization section of the Immigration and Nationality Act, according to State. Additionally, the Bureau of International Narcotics and Law Enforcement Affairs uses its expertise in drug trafficking, corruption, and crime to assist in selecting targets for counternarcotics sanctions, transnational criminal organization sanctions, and corruption-related sanctions under human rights law, according to State. SPI also works with State’s regional bureaus, such as the Bureau of African Affairs; country offices; and overseas posts to develop potential targets for given sanctions programs, such as those in Somalia and Burundi. Both Treasury and State also have intelligence offices that provide the sanctions-implementing offices with information to facilitate sanctions targeting and enforcement efforts and developing new sanctions policy. Treasury’s Office of Intelligence and Analysis (OIA). TFI’s OIA is responsible for TFI’s intelligence functions as well as for integrating the Treasury Department into the larger Intelligence Community and providing support to both Treasury leadership and the Intelligence Community. State’s Bureau of Intelligence and Research (INR). INR’s primary mission is to provide all-source intelligence and analysis to serve U.S. diplomacy. INR provides independent analysis of events to State policymakers as well as officials throughout the U.S. government and coordinates with other intelligence agencies to obtain relevant information to inform State policymakers. For example, INR’s analytical offices and its Sanctions Support Team, when requested, gather and provide information—both classified and open sourced— on sanctions targets to policy officials at State and Treasury. In addition to OIA and INR, other U.S. intelligence agencies provide support to the sanctions-implementing agencies. Commerce implements sanctions by restricting licenses for exports, reexports, and transfers (in-country) involving U.S.-origin items— commodities, software, and technology—subject to its jurisdiction and destined for sanctioned persons, entities, and destinations. Through its export licensing process, Commerce’s Bureau of Industry and Security (BIS) restricts sanctioned countries’ and persons’ access to U.S. items. BIS also enforces export controls through its Office of Export Enforcement, which conducts criminal and administrative investigations of potential violations of export regulations. Other U.S. agencies with roles in sanctions implementation include the Departments of Defense, Energy, Homeland Security, and Justice. The agencies involved and the extent of their involvement depend largely on their area of expertise. The following are a few examples of how other agencies are involved with sanctions: The Department of Defense restricts arm sales and other forms of military cooperation and is involved in decisions regarding export licenses. The Department of Energy assists in implementing nonproliferation sanctions. The Department of Homeland Security’s Customs and Border Protection helps assure that shipments to and from sanctioned countries and entities do not leave or enter the United States. The Department of Justice investigates and prosecutes violations of sanctions and export laws and provides legal reviews of sanctions’ designations. The roles of Treasury, State, and Commerce in implementing sanctions are assigned either directly by the statute or executive order authorizing the sanctions or through an interagency process and agreement. Some statutes and executive orders designate an agency to serve as the primary agency for sanctions implementation and also designate one or more agencies to support the primary agency through consultation. For example, Executive Order 13570, Prohibiting Certain Transactions With Respect to North Korea, authorizes Treasury, in consultation with State, to carry out actions to employ all powers granted to the President by specified laws to carry out the purposes of the order. While some statutory authorities may designate specific agencies for implementation, most do not make such designations but rather delegate the authority to do so to the Office of the President, according to State officials. Agency officials also noted that they are often involved in drafting new sanctions legislation and, if the statute will designate specific agency roles, are able to advise lawmakers regarding the selection of the primary agency for implementing sanctions. When a statute or executive order authorizing sanctions delegates authority to the Office of the President, specific agency roles are assigned through an interagency process at the National Security Council (NSC). According to State officials, the NSC’s Principals Coordinating Committee discusses and assigns agency roles in a sanctions program. According to State officials, most of the committee’s decisions about agency roles are made at the staff level, and the actual principals become involved only if there is a disagreement among the agencies’ staffs. State officials told us that each agency’s area of expertise and its available resources factor into the selection of an agency to lead implementation of a particular sanctions authority. For example, according to a State official, Treasury is often the lead for country-based sanctions, because these programs often focus on international financial transactions, while State usually serves as the lead for sanctions requiring more specialized knowledge, such as those relating to weapons of mass destruction and nuclear nonproliferation. State officials added that there is usually very little, if any, disagreement among the agencies regarding whether they should have primary or consultative roles. Once a decision is made, the President typically issues a delegation memo assigning responsibility for implementation of elements of the sanctions authority to each agency involved, according to Treasury officials. Treasury, State, and Commerce each provide publicly available information about the sanctions they implement and the authorities underlying those sanctions. Treasury. OFAC maintains a publicly available list of all sanctions laws and executive orders that Treasury has a role in implementing. The list is organized by sanctioned country and functional program. For each country-based, country-related, or functional program, the entry in the list includes a discussion of statutory authorities, executive orders, and regulations under which the program is implemented. According to Treasury officials, OFAC staff track and update changes in U.S. sanctions policy as needed and post new sanctions information to the agency’s website as soon as a sanction is approved. State. SPI also maintains publicly available lists of the major sanctions laws and executive orders that State has a role in implementing. These lists are organized by sanctioned country and by functional program. According to State officials, SPI typically updates these lists when authorities are established or rescinded and periodically reviews and updates the web pages where it posts the lists. According to State officials, the lists are not intended to be comprehensive and are meant only to give the reader a general understanding of some of State’s high-profile sanctions programs and to provide an initial resource for information and recent actions. Commerce. BIS produces a compilation of legal authorities pertaining to the administration of export controls under the Export Administration Regulations. Unlike Treasury and State’s lists, Commerce’s compilation comprises all of Commerce’s legal authorities to control exports, reexports, and transfers (in-country). These include executive orders, laws, and presidential declarations authorizing controls related to national security, chemical and biological weapons, and nuclear nonproliferation reasons, as well as controls for foreign policy–related sanctions. According to Commerce officials, the compilation is updated annually to reflect additions to, or deletions of legal authorities. BIS also issues rules amending the Export Administration Regulations to implement new executive orders and statutory and other legal authorities on a frequent basis, at times within a few days of the announcement or enactment of the underlying authority. According to Commerce officials, publishing rules amending the Export Administration Regulations provides the public with timely notice of changes to Commerce’s sanctions authorities and actions taken pursuant to these authorities. Treasury, State, and Commerce assess potential and observed impacts of specific sanctions, but officials stated they do not conduct agency assessments of the effectiveness of sanctions in achieving broader U.S. policy goals and cited various difficulties in doing so. Each agency’s sanctions implementation offices rely mainly on assessments performed by the Intelligence Community, including Treasury’s OIA and State’s INR. These assessments analyze the impacts of specific sanctions on a particular aspect of the sanction’s target—for example, the sanctions’ impact on the target country’s economy or trade, according to agency officials. However, these assessments do not analyze sanctions’ overall effectiveness in achieving broader U.S. policy goals or objectives, such as whether the sanctions are advancing the national security and policy priorities of the United States, according to Treasury officials. Treasury, State, and Commerce have not conducted such broader assessments on their own, and agency officials cited a variety of difficulties related to doing so. However, according to Treasury, State, and Commerce, agency assessments of sanctions’ impacts often contribute to broader interagency discussions, typically coordinated through the NSC, that examine the effectiveness of sanctions in achieving policy goals. According to agency officials, an NSC-led process allows the U.S. government to draw upon multiple agencies’ inputs and perspectives, and to consider these issues in the larger policy context, because sanctions are often only one element of broader government-wide strategies to achieve U.S. policy goals. Treasury has assessed both the observed and potential impacts of specific sanctions designations on various aspects of targets, such as a target country’s economy. Treasury’s intelligence component, OIA, conducts the majority of these impact assessments and produces analytic papers on sanctions’ impacts, according to officials. OIA officials stated that the type of analysis varies depending on the purpose or nature of the assessment. For example, some analytic papers focus on the overall economic impact of the sanction on the target country, while others examine the impact on a specific target, such as an entity or group of entities. According to Treasury officials, the office has conducted both short-range and long-range analyses of sanctions’ impacts at both the country-specific and the authority-specific level. Treasury officials said that the frequency of assessments conducted for a particular country or authority varies according to the sanctions program’s size and relative importance to current U.S. policy goals. OIA officials reported that the Under Secretary for TFI requires that impact assessments be conducted prior to an action as part of the targeting process and retrospectively after a designation takes place. According to Treasury officials, TFI, including OIA, considers conducting such impact assessments to be part of OIA’s mission. OIA officials noted that OIA, as well as TFI more broadly, considers understanding sanctions impact to be integral to developing sanctions policy and making targeting decisions. OIA officials stated that their impact assessments are circulated within Treasury and their broader analytic papers are circulated within the Intelligence Community and interagency. In addition, OFAC officials reported that they request impact assessments from OIA whenever new sanctions targets are being considered. OFAC officials stated that OIA’s impact assessments are an integral part of any targeting matrix prepared by OFAC’s Office of Global Targeting. According to OFAC officials, the type of assessment requested depends on the issue, program, and target. The requested assessments may include, for example, determining whether a target has assets in the United States to an extent that sanctions would be impactful, identifying the holdings of a given target globally and its interactions with the United States, or analyzing the second- and third-order effects of a potential sanctions designation. OFAC officials said that these assessments are also used in risk- mitigation planning. For example, if an assessment revealed that a particular sanction would lead to an undesirable consequence, such as blocking important medical supplies or other humanitarian items, OFAC might take preemptive measures to mitigate that undesirable consequence through a general license or other available tools. Treasury’s Office of International Affairs also prepares some assessments of sanctions’ impacts. According to Treasury officials, the Office of International Affairs occasionally conducts macroeconomic assessments of the impact of specific sanctions to inform TFI policymaking. A senior Office of International Affairs official reported that most of the office’s macroeconomic analyses of sanctions’ impacts are focused on the potential impact on economic growth and financial stability in the target country. In addition, OFAC officials stated that the Office of International Affairs often participates in agency discussions and may provide verbal or written assessments of sanctions’ impact on foreign partners’ industries and markets as well as on U.S. companies. State conducts some assessments of the impact of sanctions on their intended targets. INR produces most reports on sanctions impact, which are based on all sources of information (i.e., classified and open source). According to INR officials, these reports are often produced at the request of State policymakers, and occasionally coordinated with the broader Intelligence Community. INR facilitates the review of sanctions’ impacts on particular governments or other areas of interest at the request of, or in partnership with, State’s regional and functional bureaus. According to INR, most of INR’s intelligence support responds to specific questions and requests, such as whether a particular company is still operating in a sanctioned country. According to State officials, INR provides responses to requests in written products, such as formal INR or Intelligence Community assessments, or more informally through channels such as oral briefings or email responses. INR officials noted that written products often inform interagency discussions on sanctions at the NSC, since questions asked at State often become relevant to broader policy discussions. Other State entities have also examined the impact of sanctions. In 2016, State’s Office of the Chief Economist, responding to a request from SPI, analyzed the economic impact of targeted sanctions on Russian firms. According to SPI officials, they commissioned the study because they wanted to understand the specific impact of sanctions on a country that was already facing economic challenges, given that sanctions were among several foreign policy tools used to address Russian behavior. According to SPI officials, this was the only analysis of sanctions impact that SPI had requested of the Office of the Chief Economist in the past 5 years. In addition, some embassies have used cables to State headquarters to report on the impact of sanctions. According to State, most such information on a sanction’s impact is captured in a sentence or two as part of a cable focused on other issues. However, embassies in countries where sanctions are imposed on the host government (or nearby governments) often dedicate significant time to reporting on the impact of sanctions and how they affect broader foreign policy, according to State officials. For example, the U.S. embassy in Seoul produced a series of cables in 2017 and 2018 detailing observed impacts of sanctions on the North Korean economy. Commerce has conducted some assessments of the prospective impacts of sanctions, according to Commerce officials. According to Commerce officials, the Under Secretary or Deputy Under Secretary communicates requests for analyses of sanctions that originate with the NSC’s Principals Coordinating Committee. According to Commerce officials, these requests are infrequent, with very few received in recent years, and generally related only to Russia and Iran. According to Commerce, the results of these assessments may include two components: (1) a simulation of potential economic impact and (2) background data on trade flows and vulnerabilities. The first component may include a projection of sanctions’ impact on gross domestic product (GDP), consumer prices, production in specific industries, jobs, and trade flows. The second component may include background on the amount and nature of any U.S. trade with countries that might be sanctioned. For example, in March 2015, Commerce produced an analysis to determine the areas of greatest interdependence among the United States, Russia, and U.S. partners that were at risk of being affected by prospective sanctions against Russia. Treasury and State officials reported using assessments of sanctions’ impacts provided by intelligence agencies outside Treasury or State. Assessments used by Treasury. OFAC officials reported requesting assessments from other intelligence agencies, in addition to OIA’s assessments. According to OFAC, the type of assessment requested—for example, gauging the reaction of a target or government leadership to sanctions or examining a target’s assets globally—depends on the issue and the program. OFAC also reported requesting analysis of sanctions’ impact on strategic targets and their associates. OFAC officials reported that these assessments are taken into account as Treasury considers developing additional sanctions policies, targets, or both. Assessments used by State. INR and SPI officials stated that they use assessments of sanctions’ impact conducted by intelligence agencies outside State. According to an INR official, the INR sanctions team will obtain Intelligence Community assessments relevant to State policymakers concerns. SPI officials stated that most of the assessments they use are focused on the potential impact of proposed sanctions. According to the officials, the assessments help them design sanctions tools and develop targets to maximize impact. For example, according to SPI officials, the Intelligence Community will assess where the largest impact might be by assessing whether actors are likely to cease particular activities if targeted or will identify points where targets interface with the U.S. financial system. SPI officials stated that the number of assessments conducted depends on multiple variables, including current events in the targeted country and the degree of senior policymaker interest. An INR official stated that most Intelligence Community resources (i.e., intelligence collection and analysis) are focused on just a few sanctions regimes, such as North Korea, Iran, and Russia. Moreover, according to State officials, routine, finished analysis—assessing the impact of sanctions either before or after their imposition—is not always available from the Intelligence Community or is slow in delivery. State officials stated that this type of regular intelligence reporting and analysis is critical to informing sanctions policymaking at all stages (e.g., planning, targeting, implementing, enforcing, and revising). Treasury, State, and Commerce officials identified a range of analytic issues that make it difficult to assess the effectiveness of a sanctions program in meeting broad U.S. foreign policy goals. The difficulties they cited included the following: Isolating sanctions’ effects from other factors is difficult. Agency officials cited the difficulty—or, in some cases, the impossibility—of identifying sanctions as the sole or most significant cause of a target’s action relative to U.S. policy goals. For example, a sanctioned country may decide to cease certain behavior for any number of reasons that may be unrelated to the sanctions or other U.S. policy measures. OFAC officials also stated that behavioral change can be subtle, incremental, and lacking clear correlations with specific causes. In addition, Treasury officials noted that sanctions are often used in conjunction with other policy tools, such as diplomatic engagement with the target, export controls, and visa bans. Distinguishing the impact of each policy tool used is exceedingly difficult due to the limited information available via intelligence and law enforcement channels, according to Treasury officials. Policy goals and objectives often shift. Treasury officials stated that U.S. policy goals and objectives underpinning the sanctions can change over the course of a sanctions program, making it difficult to measure sanctions’ effectiveness in achieving any ultimate policy objective. According to OFAC officials, because sanctions programs are ongoing, any assessments of a sanctions program’s effectiveness would necessarily be interim, not final, and the metrics used to measure effectiveness might change over the program’s duration. Reliable data are sometimes lacking. Agency officials stated that a lack of reliable data on certain targets or countries can also make it difficult to assess the effectiveness of sanctions. According to Treasury, State, and Commerce officials, given these difficulties and limited resources, they do not conduct their own assessments of the overall effectiveness of existing sanctions programs in achieving broad policy goals. Instead, they have directed resources toward the assessments of sanctions’ impacts on targets, such as the impact on a target country’s economy or trade. Agency officials also noted that there is no policy or requirement for agencies to assess the effectiveness of sanctions programs in achieving broad policy goals. However, Treasury and State officials stated that sanctions policy is continuously evaluated informally by those implementing the sanctions, as new information comes in and as new targets are developed. Moreover, Treasury, State, and Commerce stated that agency assessments of sanctions’ impacts often contribute to broader interagency discussions, typically coordinated through the NSC, that examine the effectiveness of sanctions in achieving broad policy goals. According to agency officials, an NSC-led process allows the U.S. government to draw on multiple agencies’ inputs and perspectives, and to consider these issues in the larger policy context, given that sanctions are often only one element of broader government-wide strategies to achieve U.S. policy goals. We found strong evidence—based on studies examining factors that contributed to the effectiveness of sanctions in changing targeted countries’ behavior—that sanctions have been more effective when implemented through an international organization, or when targeted countries had some existing dependency on or relationship with the United States. We also found strong evidence—based on studies examining factors that increased the economic impact of sanctions on targeted countries—that sanctions imposed through an international organization were associated with greater impact. In addition, we found strong evidence that the economic impact of sanctions has generally been greater when they were more comprehensive in scope or severity. Sanctions may also have unintended consequences for targeted countries, such as negative impacts on human rights or public health. In some studies, larger economic impacts were associated with more unintended consequences, suggesting an important policy trade-off. Some aspects of U.S. sanctions policy, such as targeted sanctions, were generally not analyzed separately in the studies we reviewed, which could reduce the studies’ applicability to contemporary policymaking. We found strong evidence, based on studies examining factors that contributed to the effectiveness of sanctions in changing behavior, that sanctions have been more effective when they were implemented through an international organization (e.g., the United Nations) or when the target had some existing dependency on or relationship with the United States (e.g., U.S. foreign aid, military support or alliance, or relatively large bilateral trade relationship). Studies using different methods, datasets, and time periods consistently found that the United States was more likely to achieve its sanctions goals when an international organization was involved or when the target had some existing dependency on or relationship with the United States. We found some evidence, based on a smaller number of studies, that sanctions have been more effective when the target state had low per- capita income, when a country’s threat of imposing sanctions was assessed to be credible, or when sanctions imposed relatively high costs on the target state. For example, one study found that the likelihood of the target’s acquiescing to all of the sanctioning country’s demands increased when sanctions were imposed on a target with low per-capita income. Another study found that targets were more likely to acquiesce in response to threatened sanctions when the United States had not backed down against a resisting target recently. A third study found that more-severe sanctions increased the likelihood that the sanctioning country achieved more of its goals, suggesting that sanctions imposing relatively high costs have been more effective. Our review also suggests that in some circumstances, the risk of sanctions has deterred states from undertaking activities that would likely have resulted in the imposition of sanctions. Factors that have increased the measured effectiveness of sanctions may also increase their deterrent effect. For example, two studies found that the greater the trade flows between the target state and the sanctioning country, the greater the likelihood of sanctions’ success. A separate study demonstrated that this same dependency—greater trade between the target and the United States—led to greater deterrence of nuclear proliferation. More generally, states are likely to consider the risks associated with undertaking activities that could lead to the imposition of economic sanctions, among other factors. These risks include the likelihood of sanctions being imposed or removed, the states’ vulnerabilities to the different types and amounts of pressure that could result from sanctions, and the consequences that the states would experience if sanctions were imposed. See the text box for more detail on the potential risks that states that could be the target of sanctions might consider. (The text box is intended to provide a more general framework for understanding how states may anticipate and respond to sanctions; it reflects, but is not limited to, the specific factors included in the studies we reviewed.) Risk Framework for States That May Be Targets of Economic Sanctions Likelihood of sanctions’ being imposed or removed. States that may be targets of sanctions may assess the credibility of any explicit threats to impose or maintain sanctions and the credibility of any assurances that sanctions will be removed when the activity that motivated the imposition of sanctions ceases. Vulnerabilities to potential pressure from sanctions. States that may be targets of sanctions may assess whether the benefits of withstanding pressure that could result from the sanctions exceed the costs. For example, states may be concerned that higher economic costs from sanctions could be associated with greater impact on the material wellbeing of individuals and firms. Higher economic costs could also make it more difficult to compensate those affected by the sanctions—and those costs could be especially burdensome in states with low per-capita income. However, states likely consider not only the costs from sanctions but also the extent to which they might over time avoid or adapt to these costs. For example, if potential sanctions are likely to disrupt trade and investments from major commercial partners, states that are potential targets may examine whether developing or expanding relationships with third parties could mitigate the loss of these economic relationships. Sanctions imposed via an international organization (e.g., a multilateral approach associated with the United Nations) may make it more difficult for targets to avoid or adapt to sanctions—for example, by finding alternative commercial partners—and may signal a more robust international consensus regarding the objectives of the sanctions. Consequences if sanctions are imposed. States that may be targets of sanctions may assess the direct financial impact as well as future diplomatic, political, or security implications of the potential sanctions. That is, before engaging in activities that could trigger sanctions, states that depend on the United States may consider the possible impact of their actions on their future relationships with the United States in other areas, including military cooperation or the provision of aid. Conversely, states that are less dependent on the United States might anticipate fewer ongoing benefits from acquiescing to U.S. demands. Two important types of U.S. sanctions—targeted sanctions and secondary sanctions—were present during the time periods covered by the studies we reviewed. However, the studies generally did not account differently for these two sanctions types than for non-targeted and primary sanctions, respectively. As a result, the studies generally did not reflect differences between the effectiveness of these types of sanctions. This limitation of the available studies could reduce the applicability of this research to contemporary policymaking. Targeted sanctions. Targeted sanctions restrict transactions of and with specific entities and individuals, such as those who may have influence with a state’s government. In response to such sanctions, the targeted actors may in turn influence their government to change its behavior. Targeted sanctions seek to minimize impact on society at large and maintain most trade relationships with non-targeted actors in the country. However, our interpretation of studies of sanctions suggests that the targeted actors may use their influence with their government to extract concessions that compensate them for the impact of sanctions, which could limit the effectiveness of certain targeted sanctions. Secondary sanctions. Secondary sanctions, also known as supplementary sanctions, target third-party actors doing business with, supporting, or facilitating targeted regimes, persons, and organizations. From the perspective of a third-party actor, secondary sanctions likely increase the risk involved in commercially partnering with primary sanctions targets. Thus, secondary sanctions, especially those implemented by a country as large and interconnected as the United States, may make it more difficult for primary targets to avoid or adapt to sanctions. Our interpretation of studies of sanctions suggests that the effects of secondary sanctions imposed by the United States could be similar to the effects of sanctions imposed with a large or multilateral coalition through an international organization, since sanctions imposed through an international organization also increase the difficulty of finding alternative commercial partners. However, our interpretation of the studies suggests that if secondary sanctions were imposed without an international organization they would be unlikely to signal a robust international consensus regarding the sanctions’ objectives, and thus may not as effectively deter their targets, or third parties, from developing alternative commercial arrangements. While the studies we reviewed generally did not separately analyze targeted or secondary sanctions, states remain likely to consider the risks associated with undertaking activities that could lead to the imposition of these sanctions and sanctions in general. With respect to targeted and secondary sanctions, states—both primary targets and third-country actors—are likely to consider, among other things, the risks associated with undertaking activities that could result in targeted or secondary sanctions and the consequences they would experience if targeted or secondary sanctions were imposed. We found strong evidence, based on studies examining factors that increased the economic impact of sanctions, that sanctions’ economic impacts on targets have generally been greater when the sanctions were more comprehensive or were imposed through an international organization. For example, one study found that UN sanctions had an adverse impact on target countries’ economic growth and that this impact increased with more-comprehensive sanctions. Another study found that imposing sanctions along with other countries led to reductions in both U.S. and other Group of Seven countries’ bilateral trade with targeted countries. Some other studies suggest that sanctions may also have unintended consequences. For example, some studies suggest that sanctions have had a negative impact on human rights, the status of women, public health, or democratic freedoms in target countries. In addition, more frequent and comprehensive use of sanctions could encourage sanctions targets, potential targets, and their commercial partners to develop trade and financial ties that are less dependent on the United States. The extent of these unintended consequences can be proportionate to the comprehensiveness or economic impact of sanctions. As a result, the cost or comprehensiveness of sanctions could entail important policy trade-offs—that is, higher economic costs may be more coercive but may also yield greater unintended consequences. For example, two academic studies, based on data from sanctions implemented between 1972 and 2000, found that the negative impact of sanctions on democratic and press freedoms was generally greater with more comprehensive sanctions. Two other studies found that the public health effect of sanctions depended on the costliness or economic impact of the sanctions. Targeted sanctions could, in principle, reduce the unintended consequences of sanctions by reducing economic impacts on society at large. We provided a draft of this report to Treasury, State, and Commerce for review and comment. We received technical comments from all three agencies, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Treasury, the Secretary of State, the Secretary of Commerce, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8612, or GianopoulosK@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Our objectives were to (1) describe how the roles of the Departments of the Treasury (Treasury), State (State), and Commerce (Commerce) in implementing U.S. sanctions authorities are identified; (2) examine the extent to which U.S. agencies assess the effectiveness of sanctions; and (3) identify factors that have been shown by publicly available studies to contribute to the effectiveness of economic sanctions. To describe how Treasury’s, State’s, and Commerce’s roles in implementing U.S. sanctions authorities are identified, we reviewed legal authorities, including statutes and executive orders, that authorize various sanctions programs and interviewed relevant agency officials. We also discussed with Treasury, State, and Commerce officials the interagency process used in determining sanctions roles. To examine the extent to which U.S. agencies assess the effectiveness of sanctions, we interviewed officials and reviewed documents from Treasury, State, Commerce, and the Office of the Director of National Intelligence. We also obtained and reviewed agency assessments for sanctions programs related to Burundi, North Korea, Russia, and Somalia. We selected these country-based sanctions programs to obtain at least one country program with more than 200 current sanctions designations and at least one country program with fewer than 200 but more than 10 current sanctions designations as of September 2018. In addition, we included a mixture of different-size economies, based on annual gross domestic product (GDP). We used the agencies’ assessments of the selected programs to gain insight into the types of analysis conducted. To identify factors that have been shown by publicly available studies to contribute to the effectiveness of economic sanctions, we conducted a literature search for studies that examined: factors that contributed to the effectiveness of economic sanctions in changing behavior, and factors that increased the economic impact of sanctions. To identify existing studies, we used three methods. First, we conducted searches of various databases, which produced 280 studies. Second, we conducted snowball sampling, by identifying additional studies cited in papers we had already identified. Third, we asked several academic experts to validate our list of studies and recommend any additional studies that they felt met our criteria. To focus on recent research on the factors that contributed to the effectiveness or economic impact of economic sanctions and to target articles for detailed review, we included studies that met the following criteria: The study evaluated the factors that contributed to the effectiveness or economic impact of sanctions. The study included quantitative analysis of research data, which aggregated and identified patterns across many sanctions episodes. The study was published in a peer-reviewed journal or was an academic working paper. The study included data on U.S.-imposed bilateral or multilateral sanctions but may also have included sanctions imposed by other countries. The study was in English. The study was published from 2004 through October 2018. As an additional date restriction, we only included studies with at least some data from 2000 through October 2018, though the study could have included earlier data as well, in order to improve the likely relevance of the research. The publication date restriction made it more likely that included studies would be cognizant of an important source of bias in earlier sanctions research. Prior to 2004, researchers tended to examine the impact of implemented sanctions and generally excluded cases where the threat of sanctions might have led a target to change their behavior prior to implementation. More generally, observed outcomes of implemented economic sanctions are not representative of the range of circumstances in which sanctions could be imposed, threatened, or useful for deterrence, and as a result these observed outcomes tend to understate the effectiveness of economic sanctions. Finally, to select the studies to be included in our in-depth review, we evaluated them to determine whether they met additional criteria for methodological soundness. We assessed whether the studies used and clearly described appropriate statistical methods to adjust, or control, for factors that could influence the effectiveness or economic impact of sanctions. Additionally, we included only papers that ascribed statistical precision to modeled estimates. To validate the studies we selected for in-depth review, we requested suggestions regarding our list of studies from the following academic experts: Daniel W. Drezner, Bryan R. Early, and T. Clifton Morgan. We identified these researchers on the basis of the relevance of their publications to our objectives, the methodological impact of their contributions to the literature, and the number of citations of any relevant publications since 2009. Applying the selection criteria and the criteria for methodological soundness and incorporating the academic experts’ suggestions resulted in a list of 17 sufficiently rigorous studies, all of which had appeared in peer-reviewed journals. Ten studies were relevant to the factors that contributed to the effectiveness of economic sanctions and seven studies were relevant to the factors that increased the economic impact of sanctions. To obtain relevant context and background, we also examined additional studies related to the factors that contributed to the effectiveness of economic sanctions. These studies did not meet our criteria for inclusion in our in-depth review but provided insight into issues related to the analysis of effectiveness of sanctions and potential unintended consequences of sanctions. All of the studies that met the criteria for our in-depth review, as well as others we cited, are included in appendix II. To review the 17 studies we selected, we used a data collection instrument (DCI) designed to record each study’s research methodology, including its data, outcome measures, control variables, limitations, and analytic techniques and to summarize its major findings. Analysts then independently reviewed the studies and the information captured in the DCIs, reconciling any differences in their assessments through discussion. Next, we summarized the findings and categorized and aggregated the factors relevant to the effectiveness or economic impact of sanctions. We also shared a summary of our initial findings with the academic experts, who generally concurred with our findings. We characterized factors as being supported by “strong evidence” for the purposes of our review only if at least four studies—including more than half of studies that included this factor—found it to have a statistically significant effect and no studies found a statistically significant effect with the opposite sign. We characterized factors as being supported by “some evidence” for the purposes of our review only if at least two studies— including at least half of studies that included this factor—found it to have a statistically significant effect and no studies found a statistically significant effect with the opposite sign. The studies we examined varied in the quality of their methodologies, and as a result, we could not confidently report on precise estimates of the impact of different factors on the effectiveness or economic impact of sanctions. While the statistical models used in the studies we reviewed controlled for factors that could influence the success or failure of sanctions in different circumstances, these models are also subject to some biases and imperfections. For example, studies may not have accounted for all factors that might influence the success of sanctions or may not have recognized or controlled for selection biases that influenced when and how sanctions were imposed. Finally, sanctions datasets include variables for which researchers exercised varying degrees of judgment to code accurately and consistently and which therefore may be measured with imprecision or error. We conducted this performance audit from May 2018 to October 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Bapat, Navin A., Tobias Heinrich, Yoshiharu Kobayashi, and T. Clifton Morgan. “Determinants of Sanctions Effectiveness: Sensitivity Analysis Using New Data.” International Interactions, vol. 39, no. 1 (2013): pp. 79- 98. Bapat, Navin A., and T. Cliff Morgan. “Multilateral Versus Unilateral Sanctions Reconsidered: A Test Using New Data.” International Studies Quarterly, vol. 53, no. 4 (2009): pp. 1075-1094. Biglaiser, Glen, and David Lektzian. “The Effect of Sanctions on U.S. Foreign Direct Investment.” International Organization, vol. 65, no. 3 (2011): pp. 531-551. Caruso, Raul. “The Impact of International Economic Sanctions on Trade: Empirical Evidence over the Period 1960-2000.” Rivista Internazionale di Scienze Sociali, vol. 113, no. 1 (2005): pp. 41-66. Early, Bryan R. “Unmasking the Black Knights: Sanctions Busters and Their Effects on the Success of Economic Sanctions.” Foreign Policy Analysis, vol. 7, no. 4 (2011): pp. 381-402. Early, Brian R., and Robert Spice. “Economic Sanctions, International Institutions, and Sanctions Busters: When Does Institutionalized Cooperation Help Sanctioning Efforts?” Foreign Policy Analysis, vol. 11, no. 3 (2015) pp. 339-360. Hatipoglu, Emre, and Dursun Peksen. “Economic Sanctions and Banking Crises in Target Economies.” Defence and Peace Economics, vol. 29, no. 2 (2018): pp. 171-189. Krustev, Valentin L., and T. Clifton Morgan. “Ending Economic Coercion: Domestic Politics and International Bargaining.” Conflict Management and Peace Science, vol. 28, no. 4 (2011): pp. 351-376. Lektzian, David, and Dennis Patterson. “Political Cleavages and Economic Sanctions: The Economic and Political Winners and Losers of Sanctions.” International Studies Quarterly, vol. 59, no. 1 (2015): 46-58. Major, Solomon. “Timing Is Everything: Economic Sanctions, Regime Type, and Domestic Instability.” International Interactions, vol. 38, no. 1 (2012): pp. 79-110 Miller, Nicholas L. “The Secret Success of Nonproliferation Sanctions.” International Organization, vol. 68, no. 4 (2014): pp. 913-944. Neuenkirch, Matthias, and Neumeier, Florian. “The Impact of UN and US Economic Sanctions on GDP Growth.” European Journal of Political Economy, vol. 40, part A, (2015): pp. 110-125. Nooruddin, Irfan, and Autumn Lockwood Payton. “Dynamics of Influence in International Politics: The ICC, BIAs, and Economic Sanctions.” Journal of Peace Research, vol. 47, no. 6 (2010): pp. 711–721. Peksen, Dursun. “Autocracies and Economic Sanctions: The Divergent Impact of Authoritarian Regime Type on Sanctions Success.” Defence and Peace Economics, Vol. 30, No. 3 (2017): pp. 253-268. Peksen, Dursun and Byunghwan Son. “Economic Coercion and Currency Crises in Target Countries.” Journal of Peace Research, vol. 52, no. 4 (2015): pp. 448-462. Peterson, Timothy M. “Sending a Message: The Reputation Effect of US Sanction Threat Behavior.” International Studies Quarterly, vol. 57, no. 4 (2013): pp. 672-682. Shin, Geiguen, Seung-Whan Choi, and Shali Luo. “Do Economic Sanctions Impair Target Economies?” International Political Science Review, vol. 37, no. 4 (2016): pp. 485-499. Allen, Susan Hannah, and David J. Lekztian. “Economic Sanctions: A Blunt Instrument?” Journal of Peace Research, vol. 50, no. 1 (2013): pp. 121-135. Drezner, Daniel W. “Sanctions Sometimes Smart: Targeted Sanctions in Theory and Practice.” International Studies Review, vol. 13, no. 1 (2011): pp. 96-108. Drezner, Daniel W. “The Hidden Hand of Economic Coercion.” International Organization, vol. 57, no. 3 (2003): pp. 643-659. Drury, A. Cooper, and Dursun Peksen. “Women and Economic Statecraft: The Negative Impact Economic Sanctions Visit on Women.” European Journal of International Relations, vol. 20, no. 2 (2014): pp. 463-490. Forrer, John. “Economic Sanctions: Sharpening a Vital Foreign Policy Tool.” Atlantic Council Issue Brief. Washington, D.C.: Atlantic Council, June 2017. Harrell, Peter. “Is the U.S. Using Sanctions Too Aggressively? The Steps Washington Can Take to Guard Against Overuse.” Foreign Affairs. September 11, 2018. Licht, Amanda A. “Hazards or Hassles: The Effect of Sanctions on Leader Survival.” Political Science Research and Methods, vol. 5, no.1 (2017): pp. 143-161. Marinov, Nikolay. “Do Economic Sanctions Destabilize Country Leaders?” American Journal of Political Science, vol. 49, no. 3 (2005): pp. 564-576. Peksen, Dursun. “Coercive Diplomacy and Press Freedom: An Empirical Assessment of the Impact of Economic Sanctions on Media Openness.” International Political Science Review, vol. 31, no. 4 (2010): pp. 449-469. Peksen, Dursun. “Economic Sanctions and Human Security: The Public Health Effect of Economic Sanctions.” Foreign Policy Analysis, vol. 7, no. 3 (2011): pp. 237-251. Peksen, Dursun, and A. Cooper Drury. “Coercive or Corrosive: The Negative Impact of Economic Sanctions on Democracy.” International Interactions, vol.36, no. 3 (2010): pp. 240-264. Wood, Reed M. “‘A Hand upon the Throat of the Nation’: Economic Sanctions and State Repression, 1976–2001.” International Studies Quarterly, vol. 52, no. 3 (2008): pp. 489-513. In addition to the contact named above, Drew Lindsey (Assistant Director), Michael Maslowski (Analyst in Charge), Eugene Beye, Nisha Rai, Michael Hoffman, Reid Lowe, Christopher Keblitis, Grace Lui, Justin Fisher, Leia Dickerson, Michael Simon, and Julia Robertson made key contributions to this report.", "summary": "The United States maintains dozens of economic sanctions programs to counteract activities that threaten U.S. national interests. There are currently 20 country-based or country-related sanctions programs, according to lists of sanctions programs published by Treasury and State (see map). Additional countries may also be affected by sanctions programs that target entities regardless of their geographic location, such as counter-narcotics sanctions. Treasury, State, and Commerce, among other agencies, coordinate to implement these programs. Sanctions may place restrictions on a country's entire economy, targeted sectors of the economy, or individuals or corporate entities. Reasons for sanctions range widely, including support for terrorism, narcotics trafficking, weapons proliferation, and human rights abuses. Economic restrictions can include, for example, denying a designated entity access to the U.S. financial system, freezing an entity's assets under U.S. jurisdiction, or prohibiting the export of restricted items. GAO was asked to review issues related to the implementation and effectiveness of economic sanctions. Among other things, this report (1) examines the extent to which U.S. agencies assess the effectiveness of sanctions, and (2) identifies factors that have been shown by publicly available studies to contribute to the effectiveness of economic sanctions. GAO reviewed documents and interviewed officials at Treasury, State, and Commerce and in the U.S. Intelligence Community. GAO also reviewed academic studies that used rigorous statistical methods to analyze the impact and effectiveness of economic sanctions across many sanctions programs. The Departments of the Treasury (Treasury), State (State), and Commerce (Commerce) each undertake efforts to assess the impacts of specific sanctions on the targets of those sanctions. For example, Treasury and State both analyze or compile information on sanctions programs' impacts, such as on a target country's economy. In addition, Commerce assesses prospective impacts of some sanctions on targeted countries and others. According to Treasury and State officials, the agencies also use Intelligence Community assessments to gauge sanctions' impacts. However, agency officials cited several difficulties in assessing sanctions' effectiveness in meeting broader U.S. policy goals, including challenges in isolating the effect of sanctions from other factors as well as evolving foreign policy goals. According to Treasury, State, and Commerce officials, their agencies have not conducted such assessments on their own. However, they stated that agency assessments of sanctions' impacts often contribute to broader interagency discussions that examine the effectiveness of sanctions in achieving policy goals. The academic studies GAO reviewed suggest that several factors have contributed to more-effective sanctions. Studies examining factors that contribute to the effectiveness of sanctions in changing targeted countries' behavior provided evidence that sanctions have been more effective when (1) they were implemented through an international organization (e.g., the United Nations) or (2) the targeted countries had some existing dependency on, or relationship with, the United States, such as a trade or military relationship. In addition, studies examining factors that increased sanctions' economic impact provided evidence that the impact has generally been higher when the sanctions were more comprehensive in scope or severity, or—similar to the findings on effectiveness in changing behavior—were imposed through an international organization. Sanctions may also have unintended consequences for targeted countries, such as negative impacts on human rights or public health. In some studies, larger economic impacts were associated with more unintended consequences.", "document_type": "gao"}
{"report": "Medicare and Medicaid FFS are federal health care programs, though there are certain distinctions between the programs’ coverage and financing. Medicare coverage policies are generally established at the national level, and the program directly pays providers for services rendered. Medicaid is a federal-state program, and states are provided flexibility to design their coverage policies. State Medicaid agencies pay providers for services rendered, and the federal government and states share in the financing of the program, with the federal government matching most state expenditures. The Improper Payments Information Act of 2002 (IPIA), as amended, requires federal executive branch agencies to report a statistically valid estimate of the annual amount of improper payments for programs identified as susceptible to significant improper payments. To accomplish this, agencies follow guidance for estimating improper payments issued by OMB. According to the HHS-OIG, which conducts annual compliance reviews and regularly reviews the estimation methodology for both the Medicare FFS and Medicaid improper payment measurement programs, the methodology for both programs’ estimates comply with federal improper payment requirements. To estimate improper payments in Medicare and Medicaid FFS, respectively, CMS’s CERT and PERM contractors randomly sample and manually review medical record documentation associated with FFS claims for payment from providers, also known as medical reviews. The CERT and PERM programs project the improper payments identified in the sample to all FFS claims to estimate improper payment amounts and rates for the programs nationally for a given fiscal year. For Medicare, the CERT contractor conducted medical reviews on about 50,000 Medicare claims in fiscal year 2017. For Medicaid, the PERM contractor conducted medical reviews on nearly 31,000 Medicaid claims across fiscal years 2015, 2016, and 2017 to estimate fiscal year 2017 improper payments. Although IPIA, as amended, only requires agencies to develop one improper payment estimate for each identified program, both the CERT and PERM programs also estimate national service-specific improper payment amounts and rates to identify services at high risk for improper payment. Additionally, the PERM program estimates state-level improper payment rates based on the amounts of improper payments identified through medical reviews in each state. The CERT and PERM contractors conduct medical reviews to determine whether claims were paid or denied properly in accordance with program coverage policies—including coverage policies based on statutes, regulations, other CMS coverage rules, and each state’s coverage policies in the case of Medicaid. To perform medical reviews, trained clinicians review documentation—such as progress notes, plans of care, certificates of medical necessity, and physician orders for services—to ensure that claims meet program coverage policies. In general, Medicare and Medicaid documentation requirements define the documentation needed to ensure that services are medically necessary and demonstrate compliance with program coverage policies. For example, Medicare home health services must be supported by documentation demonstrating compliance with the coverage policy that beneficiaries be homebound, among other requirements. Certain coverage policies and documentation requirements were implemented to help reduce the potential for fraud, waste, and abuse. For example, Medicare implemented a requirement that DME providers maintain documentation demonstrating proof of item delivery, to better ensure program integrity. (Figure 1 presents an example of a progress note to support the medical necessity of Medicare home health services. See App. III for additional examples of provider documentation). The CERT and PERM contractors classify improper payments identified through medical review by the type of payment error. Two types of errors are related to documentation—no documentation and insufficient documentation. No documentation: Improper payments in which providers fail to submit requested documentation or respond that they do not have the requested documentation. Insufficient documentation: Improper payments in which providers submit documentation that is insufficient to determine whether a claim was proper, such as when there is insufficient documentation to determine if services were medically necessary, or when a specific, required documentation element, such as a signature, is missing. In fiscal year 2017, insufficient documentation comprised the majority of estimated FFS improper payments in both Medicare and Medicaid, with 64 percent of Medicare and 57 percent of Medicaid medical review improper payments. Improper payments stemming from insufficient documentation in Medicare FFS increased substantially starting in 2009, while insufficient documentation in Medicaid has remained relatively stable since 2011 (see Fig. 2). CMS has attributed the increase in Medicare insufficient documentation since 2009 in part to changes made in CERT review criteria. Prior to 2009, CERT medical reviewers used “clinical inference” to determine that claims were proper even when specific documentation was missing if, based on other documentation and beneficiary claim histories, the reviewers could reasonably infer that the services were provided and medically necessary. Beginning with CMS’s fiscal year 2009 CERT report, in response to 2008 HHS-OIG recommendations, CMS revised the criteria for CERT medical reviews to no longer allow clinical inference and the use of claim histories as a source of review information. More recent policy changes that added to Medicare documentation requirements may have also contributed to the increase in insufficient documentation in Medicare FFS. Medicare’s CERT and Medicaid’s PERM contractors make multiple attempts to contact providers to request medical record documentation for medical reviews, and review all documentation until they must finalize the FFS improper payment estimate. The CERT and PERM contractors allow providers 75 days to submit documentation, though providers can generally submit late documentation up to the date each program must finalize its improper payment estimate, known as the cut-off date (See Fig. 3.). Both programs also contact providers to subsequently request additional documentation if the initial documentation submitted by the providers does not meet program requirements. Initial documentation request: The CERT and PERM contractors make initial requests for documentation by sending a letter and calling the provider. After the initial provider request, if there is no response, the contractors contact the provider at least three additional times to remind them to submit the required documentation. If there is no response, the claim is determined to be improper due to no documentation. Claims are also classified as improper due to no documentation when the provider responds but cannot produce the documentation, such as providers that do not have the beneficiary’s documentation or records for the date of service, among other reasons (see Table 1). For referred services, such as home health, DME, and laboratory services, the CERT contractor also conducts outreach to referring physicians to request documentation. For example, for a laboratory claim, the CERT contractor may contact the physician who ordered the laboratory test to request associated documentation, such as progress notes. Conversely, the PERM contractor told us they generally do not contact referring physicians to request documentation. Subsequent documentation request: If a provider initially submits documentation that is insufficient to support a claim, then the CERT and PERM contractors subsequently request additional documentation. In fiscal year 2017, of the 50,000 claims in the CERT sample, the contractor requested additional documentation from 22,815 providers. Providers did not submit additional documentation to sufficiently support 56 percent of the associated claims. For the 3 years that comprise the 2017 Medicaid improper payment rate, of the nearly 31,000 claims in the PERM sample, the contractor requested additional documentation for 5,448, and providers did not submit additional documentation to sufficiently support about 8 percent of the 5,448 claims. In addition to having similar outreach to providers for obtaining documentation, the CERT and PERM contractors also have processes to refer suspected fraud to the appropriate program integrity entity, to ensure the accuracy of medical reviews, and to allow providers to dispute improper payment determinations. Suspected fraud: When CERT and PERM contractors identify claims with evidence of suspected fraud, they are required to refer the claims to other program integrity entities that are responsible for investigating suspected fraud. CERT and PERM contractor officials said that in 2017, the CERT contractor referred 35 claims, and the PERM contractor did not make any referrals. Interrater reliability (IRR) reviews: As a part of their medical review processes, both the CERT and PERM contractors conduct IRR reviews, where two reviewers conduct medical reviews on the same claim and compare their medical review determinations. These IRR reviews ensure the consistency of medical review determinations and processes for resolving differences identified through the IRR reviews. CMS staff said that they also review a sample of the CERT and PERM contractors’ payment determinations to ensure their accuracy. CERT: The contractor performs IRR reviews for at least 300 claims each month, including claims with and without improper payment determinations. PERM: The contractor conducts IRR reviews of all improper payment determinations, except improper payments due to no documentation, and 10 percent of all correctly paid claims in the sample, which combined was about 3,600 claims for the fiscal year 2017 national improper payment rate. Disputing improper payment determinations: Both CERT and PERM contractors have processes in place for disputing the CERT or PERM contractor’s improper payment determinations. These processes involve reviewing the claim, including any newly submitted documentation, and may result in upholding or overturning the initial improper payment determination. Improper payment determinations that are overturned prior to the CERT and PERM contractors’ cut-off dates are no longer considered improper, and estimated improper payment amounts and rates are adjusted appropriately. CERT: Medicare Administrative Contractors, which process and pay claims, may dispute the CERT contractor’s improper payment determinations first with the CERT contractors and then, if desired, with CMS. Additionally, Medicare providers can appeal the CERT contractor’s improper payment determinations through the Medicare appeals process. PERM: State Medicaid officials may dispute the PERM contractor’s improper payment determinations first with the PERM contractor and then, if desired, with CMS. Providers are not directly involved in this process; instead, providers can contact the state to appeal the improper payment determination. We found that Medicare, relative to Medicaid, had a higher estimated FFS improper payment rate primarily due to insufficient documentation in fiscal year 2017. According to CMS data, across all services in fiscal year 2017, the rate of insufficient documentation was 6.1 percent for Medicare and 1.3 percent in Medicaid, substantially greater than the difference in rates for all other types of errors, which were 3.4 and 1.0 percent, respectively. For home health, DME, and laboratory services, the insufficient documentation rate was at least 27 percentage points greater for Medicare than for Medicaid, and for hospice services, the rate was 9 percentage points greater (see Fig. 4). Differences between Medicare and Medicaid coverage policies and documentation requirements likely contributed to the substantial variation in the programs’ insufficient documentation rates for the services we examined. Among the services we examined, there are four notable differences in coverage policy and documentation requirements that likely affected how the programs conducted medical reviews: face-to-face examinations; prior authorization; signature requirements; and documentation from referring physicians for referred services, as discussed below. Face-to-face examinations. In part to better ensure program integrity, the Patient Protection and Affordable Care Act established a requirement for referring physicians to conduct a face-to-face examination of beneficiaries as a condition of payment for certain Medicare and Medicaid services. States were still in the process of implementing the policies for Medicaid in fiscal year 2017. include narrative information that sufficiently supported that the beneficiary had a life expectancy of less than 6 months. include the certification date span. documentation supporting that the referring physician conducted an examination when certifying the medical necessity of the service. Hospice providers must submit documentation of a face-to-face examination when recertifying the medical necessity of hospice services for beneficiaries who receive care beyond 6 months after their date of admission. (See sidebar for examples of insufficient documentation in Medicare hospice services.) CMS officials told us that documentation requirements for the face-to-face examination policy for home health services in particular led to an increase in insufficient documentation. When initially implemented in April 2011, home health providers had to submit separate documentation from the referring physician detailing the examination and the need for home health services. Beginning January 2015, CMS changed the requirement to allow home health providers to instead use documentation from the referring physician, such as progress notes, to support the examinations. CMS and several stakeholders attributed recent decreases in the home health improper payment rate to the amended documentation requirement (see Fig. 5). agency did not apply to the sampled day of care associated with the claim. health and DME services in Medicaid in 2016; however, the requirement likely did not apply to many claims subject to fiscal year 2017 PERM medical reviews. Medicaid does not have a face-to-face policy for hospice services, and most states we interviewed did not have such policies. (See sidebar for examples of insufficient documentation in Medicaid.) Prior authorization. Medicare does not have the same broad authority as state Medicaid agencies to implement prior authorization, which can be used to review documentation and verify the need for coverage prior to services being rendered. State Medicaid agencies we spoke with credit prior authorization with preventing improper payments from being paid in the first place. CMS has used prior authorization in Medicare for certain services through temporary demonstration projects and models, as well as one permanent program. In April 2018, we found that savings from a series of Medicare temporary demonstrations and models that began in 2012 could be as high as about $1.1 to $1.9 billion as of March 2017. We recommended that CMS take steps, based on its evaluations of the demonstrations, to continue prior authorization. All six of our selected states use prior authorization in Medicaid for at least one of the four services we examined. In particular, all six selected states require prior authorization for DME, and five require prior authorization for home health. Officials from several states noted that they often apply prior authorization to services at high risk for improper payments, and most told us that prior authorization screens potential improper payments before services are rendered. We did not evaluate the effectiveness of states’ use of prior authorization, or review the documentation required by states for prior authorization. (See Fig 6 for an example state Medicaid prior authorization form.) Physician signatures: While both Medicare and state Medicaid agencies require signatures on provider documents to ensure their validity, Medicare has detailed standards for what constitutes a valid signature. physician did not support the medical necessity for the specific type of catheter ordered. variety of situations. For example, illegible signatures and initials on their own are generally invalid, though they are valid when over a printed name. Examples of insufficient documentation in Medicare laboratory Documentation from the referring physician did not support the order or an intent to order the billed laboratory tests. In Medicaid, PERM contractor staff told us that state agencies generally have not set detailed standards for valid signatures, and that reviewers generally rely on their judgment to assess signature validity. Documentation from the referring physician did not support that the beneficiary’s currently has diabetes for a billed laboratory test for the management and control of diabetes. Documentation for referred services. Medicare requires documentation from referring physicians to support the medical necessity of the referred services that we examined—home health, DME, and laboratory services—but Medicaid generally does not require such documentation. Medicare generally requires documentation from the referring physician, such as progress notes, to support the medical necessity of referred services. CMS officials told us that Medicare requires such documentation from referring physicians to ensure that medical necessity determinations are independent of the financial incentive to provide the referred service, particularly as certain referred services are high risk for fraud, waste, and abuse. (See sidebar for examples of insufficient documentation in Medicare home health, DME, and laboratory services.) In Medicaid, documentation requirements to support the medical necessity of referred services are primarily established by states, and states generally do not require documentation, such as progress notes from referring physicians, to support medical necessity. Further, PERM contractor staff told us that they generally do not review such documentation when conducting medical reviews of claims for referred services. Officials from CMS, the CERT contractor, and provider associations told us that Medicare’s documentation requirements for referred services present challenges for providers of referred services to submit sufficient documentation since they are dependent on referring physician documentation to support medical necessity. Some officials further stated that referring physicians may lack incentive to ensure the sufficiency of such documentation, as they do not experience financial repercussions when payments for referred services are determined to be improper. Officials told us that: It is generally not standard administrative practice for laboratories or DME providers to obtain referring physician documentation, and referring physicians may not submit them when the referred services are subject to medical review. For example, laboratories generally render services based solely on physician orders for specific tests, and generally do not obtain associated physician medical records. Referring physicians may not document their medical records in a way that meets Medicare documentation requirements to support the medical necessity of referred services. Officials from a physician organization told us that physicians refer beneficiaries for a broad array of services, and face challenges documenting their medical records to comply with Medicare documentation requirements for various referred services. We previously reported on CMS provider education efforts and recommended that CMS take steps to focus education on services at high risk for improper payments and to better educate referring physicians on documentation requirements for DME and home health services. CMS agreed with and has fully addressed our recommendation. Medicare and Medicaid pay for many of the same services, to some of the same providers, and likely face many of the same underlying program risks. However, because of differences in documentation requirements between the two programs, the same documentation for the same service can be sufficient in one program but not the other. The substantial variation in the programs’ improper payment rates raise questions about how well their documentation requirements help in determining whether services comply with program coverage policies, and accordingly help identify causes of program risks. This is inconsistent with federal internal control standards, which require agencies to identify, analyze, and respond to program risks. CMS officials attributed any differences in the two programs’ documentation requirements to the role played by the states in establishing such requirements under Medicaid, and told us that they have not assessed the implications of how differing requirements between the programs may lead to differing assessments of the programs’ risks. CMS relies on improper payment estimates to help develop strategies to reduce improper payments, such as informing Medicare’s use of routine medical reviews, educational outreach to providers, and efforts to address fraud. Without a better understanding of how documentation requirements affect estimates of improper payments, CMS may not have the information it needs to effectively identify and analyze program risks, and develop strategies to protect the integrity of the Medicare and Medicaid programs. CMS’s Patients over Paperwork initiative is an ongoing effort to simplify provider processes for complying with Medicare FFS requirements, including documentation requirements. Although CMS officials said this initiative is intended to help providers meet documentation requirements in both Medicare and Medicaid, current efforts only address Medicare documentation requirements. As part of the initiative, CMS solicited comments from stakeholders through proposed rulemaking on documentation requirements that often lead to insufficient documentation, and CMS officials stated that they have met with provider associations to obtain feedback. The initiative is generally focused on reviewing documentation requirements the agency has the authority to easily update, namely requirements that are based on CMS coverage rules, as opposed to requirements based on statute. Through this initiative, CMS has clarified and amended several Medicare documentation requirements. For example, CMS clarified Medicare documentation requirements for DME providers to support proof of item delivery. As part of another initiative to examine insufficient documentation in Medicare, CMS found that 3 percent of improper payments due to insufficient documentation were clerical in nature in fiscal year 2018. For the CERT’s fiscal year 2018 medical reviews, the CERT contractor classified whether improper payments due to insufficient documentation were clerical in nature—meaning the documentation supported that the service was covered and necessary, had been rendered, and was paid correctly, but did not comply with all Medicare documentation requirements. Such errors would not result in an improper payment determination if the documentation had been corrected. For example, such clerical errors may involve missing documentation elements that may be found elsewhere within the medical records. According to CMS officials, the information gathered on clerical errors may inform efforts to simplify documentation requirements. Specifically, CMS plans to use this information to help identify requirements that may not be needed to demonstrate medical necessity or compliance with coverage policies. CMS said that it does not plan to engage in similar efforts to examine insufficient documentation errors in Medicaid because of challenges associated with variations in state Medicaid documentation requirements and the additional burden it would place on states. On a national basis, CMS’s PERM program generates statistically valid improper payment estimates for the Medicaid FFS program. At the state level, however, CMS officials told us that the PERM contractor’s medical reviews do not generate statistically generalizable information about improper payments by service type and, as a result, they do not provide robust state-specific information on the corrective actions needed to address the underlying causes of improper payments. According to CMS, the number of improper payments identified through medical reviews is too small to generate robust state-specific results. In fiscal year 2017, the PERM contractor identified 918 improper payments nationwide out of nearly 31,000 claims subjected to medical reviews. More than half of all states had 10 or fewer improper payments identified through medical reviews in fiscal year 2017, and these made up about 7 percent of total sample improper payments identified through medical reviews (see Table 2). According to CMS officials, estimating improper payments for specific service types within each state with the same precision as the national estimate would involve substantially expanding the number of medical reviews conducted and commensurately increasing PERM program costs. CMS officials also estimated federal spending on PERM Medicaid FFS medical reviews at about $8 million each year, which does not include state costs, the federal share of the state costs, or providers’ costs. Of our six selected states, officials from one state said that data on service-specific improper payment rates at the state level would be useful, though officials had reservations about increasing sample sizes because of the resources involved in doing so. CMS requires state Medicaid agencies to develop corrective actions to rectify each improper payment identified. However, since the Medicaid review sample in a state typically is not large enough to be statistically generalizable by service type, the identified improper payments may not be representative of the prevalence of improper payments associated with different services within the state. Accordingly, corrective actions designed to rectify specific individual improper payments may not address the most prevalent underlying causes of improper payments. For example, state Medicaid officials in four of our six states said that most improper payments identified through PERM medical reviews are unique one-time events. Federal internal control standards require agencies to identify and analyze program risks so they can effectively respond to such risks, and OMB expects agencies to implement corrective actions that address underlying causes of improper payments. Without estimates that provide information on the most prevalent underlying causes of improper payments within a state, particularly by service type, a state Medicaid agency may not be able to develop appropriate corrective actions or prioritize activities to effectively address program risks. Corrective actions that do not address the underlying causes of improper payments are unlikely to be an effective use of state resources. Increasing sample sizes of the PERM is one approach that could improve the usefulness of the medical reviews for states—but other options also exist. For example, PERM findings could be augmented with data from other sources—such as findings from other CMS program integrity efforts, state auditors, and HHS-OIG reports. States conduct their own program integrity efforts, including medical reviews, to identify improper payments and state Medicaid officials we spoke with in four of our six selected states said that they largely rely on such efforts to identify program risks. One state’s Medicaid officials said that state-led audits allow them to more effectively identify—and subsequently monitor—services that are at risk for improper payments in the state. CMS also could use data from other sources on state-specific program risks to help design states’ PERM samples. These options could help CMS and the states better identify the most prevalent causes of improper payments and more effectively focus corrective actions and program integrity strategies to address program risks. State Medicaid agencies may, but are not required to, determine whether providers included in the PERM sample are under fraud investigation and notify the PERM contractor. Under CMS policy, when a state notifies the PERM contractor of a provider under investigation, the contractor will end all contact with the provider to avoid compromising the fraud investigation, and the claim will be determined to be improper, due to no documentation. In fiscal year 2017, of the 328 Medicaid improper payments due to no documentation, 27 (8 percent) from five states, according to CMS, were because the provider was under fraud investigation. If a state Medicaid agency does not notify the PERM contractor about providers under fraud investigation, the PERM contractor will conduct its medical review, which involves contacting the provider to obtain documentation as a part of its normal process, and communicate about improper payment determinations. Contacting providers that are under fraud investigation as part of PERM reviews could interfere with an ongoing investigation, such as in the following ways we identified based on information from the Association of Certified Fraud Examiners and others. The contact by the PERM contractor to request documentation, although unrelated to the fraud investigation, may give the impression that the provider is under heightened scrutiny. This could prompt the provider to change its behavior, or to destroy, falsify, or create evidence. These actions could in turn disrupt or complicate law enforcement efforts to build a criminal or civil case. The PERM contractor’s communication about improper payment determinations may prompt states to conduct educational outreach to the provider about proper billing procedures. This may inadvertently change the billing practices of a fraudulent provider for whom law enforcement is trying to establish a pattern of behavior. We found that states may not have processes to determine whether providers included in the PERM sample are under fraud investigation. Of the six states we spoke with, officials from two states said they did not have a mechanism in place to identify providers under fraud investigation. However, it is a best practice for investigative and review entities to communicate and coordinate with one another to determine if multiple entities are reviewing the same provider and for investigators to work discreetly without disrupting the normal course of business, based on our analysis of information from the Association of Certified Fraud Examiners and others. Accordingly, investigators should be aware of other government entities that are in contact with providers under investigation, such as the PERM contractor, who may contact providers multiple times to request documentation, and refer identified improper payments for recovery. If multiple entities are reviewing the same provider, one entity may be directed to pause or cease its activities, such as a PERM medical review, to reduce the risk of compromising an active fraud investigation. CMS has stated that it is not the agency’s intention to negatively impact states’ provider fraud investigations and, therefore, it has provided states the option to notify the PERM contractor of any providers under investigation to avoid compromising investigations. However, CMS does not require states to determine whether providers under PERM medical reviews are also under fraud investigation, which creates the potential that PERM reviews could interfere with ongoing investigations. State Medicaid agencies may not have incentives to notify the PERM contractor of providers under fraud investigation, as doing so will automatically result in a no documentation error, which increases states’ improper payment rates. Medicaid officials from one state we spoke with said that while they check whether providers subject to PERM reviews are under investigation for fraud, they do not report these instances to the PERM contractor because the PERM contractor would find a no documentation error and the claim would be cited as improper, increasing the state’s improper payment rate. Officials from another state said this policy penalizes states, in the form of higher state-level improper payment rates that may reflect poorly on states. Additionally, officials from this state were reluctant to develop corrective actions for improper payments stemming from such no documentation errors. CMS and states need information about the underlying causes of improper payments to develop corrective actions that will effectively prevent or reduce future improper payments in Medicare and Medicaid FFS. The substantial variation in Medicare and Medicaid estimated improper payment rates for the services we examined raise questions about how well the programs’ documentation requirements ensure that services were rendered in accordance with program coverage policies. While our study focused on certain services with high rates of insufficient documentation, differences in documentation requirements between the programs may apply to other services as well. Without examining how the programs’ differing documentation requirements affect their improper payment rates, CMS’s ability to better identify and address FFS program risks and design strategies to assist providers with meeting requirements may be hindered. At the state level, PERM medical reviews do not provide robust information to individual states. CMS’s requirements to address individual improper payments may lead states to take corrective actions that may not fully address underlying causes of improper payments identified through medical review, and may misdirect state efforts to reduce improper payments. Absent a more comprehensive review of existing sources of information on the underlying causes of Medicaid improper payments, CMS and states are missing an opportunity to improve their ability to address program risks. In addition, the lack of a requirement for state Medicaid agencies to determine whether providers whose claims are selected for PERM medical reviews are also under fraud investigation risks compromising ongoing investigations. Further, citing such claims as improper payments in states’ estimated improper payment rates may discourage state Medicaid agencies from notifying the PERM contractor that a provider is under investigation. We are making the following four recommendations to CMS: The Administrator of CMS should institute a process to routinely assess, and take steps to ensure, as appropriate, that Medicare and Medicaid documentation requirements are necessary and effective at demonstrating compliance with coverage policies while appropriately addressing program risks. (Recommendation 1) The Administrator of CMS should take steps to ensure that Medicaid medical reviews provide robust information about and result in corrective actions that effectively address the underlying causes of improper payments. Such steps could include adjusting the sampling approach to reflect state-specific program risks, and working with state Medicaid agencies to leverage other sources of information, such as state auditor and HHS-OIG findings. (Recommendation 2) The Administrator of CMS should take steps to minimize the potential for PERM medical reviews to compromise fraud investigations, such as by directing states to determine whether providers selected for PERM medical reviews are also under fraud investigation and to assess whether such reviews could compromise investigations. (Recommendation 3) The Administrator of CMS should address disincentives for state Medicaid agencies to notify the PERM contractor of providers under fraud investigation. This could include educating state officials about the benefits of reporting providers under fraud investigation, and taking actions such as revising how claims from providers under fraud investigation are accounted for in state-specific FFS improper payment rates, or the need for corrective actions in such cases. (Recommendation 4) We provided a draft of this report to HHS for comment, and its comments are reprinted in appendix I. HHS also provided us with technical comments, which we incorporated in the report as appropriate. HHS concurred with our first recommendation that CMS institute a process to routinely assess and ensure that Medicare and Medicaid documentation requirements are necessary and effective. HHS stated that CMS’s Patients over Paperwork initiative is focused on simplifying Medicare documentation requirements and noted that for the Medicaid program, CMS will identify and share documentation best practices with state Medicaid agencies. CMS’s Patients over Paperwork initiative may help CMS streamline Medicare documentation requirements. However, we believe CMS should take steps to assess documentation requirements in both programs to better understand the variation in the programs’ requirements and their effect on estimated improper payment rates. Without an assessment of how the programs’ documentation requirements affect estimates of improper payments, CMS may not have the information it needs to ensure that Medicare and Medicaid documentation requirements are effective at demonstrating compliance and appropriately address program risks. HHS did not concur with our second recommendation that CMS ensure that Medicaid medical reviews provide robust information about and result in corrective actions that effectively address the underlying causes of improper payments. HHS noted that increasing the PERM sample size would involve increasing costs and state Medicaid agencies’ burden, and that incorporating other sources of information into the PERM sample design could jeopardize the sample’s statistical validity. HHS also commented that it already uses a variety of sources to identify and take corrective actions to address underlying causes of improper Medicaid payments. We acknowledge that increasing the sample size would increase the costs of the PERM medical review program, though the level of improper payments warrants continued action. Further, under the current approach, we found that CMS and state Medicaid agencies are expending time and resources developing and implementing corrective actions that may not be representative of the underlying causes of improper payments in their states. It is important that corrective actions effectively and efficiently address the most prevalent causes of improper payments, and our report presents options that could improve the usefulness of the PERM’s medical reviews—such as augmenting medical reviews with other sources of information during the development of corrective actions. We continue to believe that corrective actions based on more robust information would help CMS and state Medicaid agencies more effectively address Medicaid program risks. HHS concurred with our third and fourth recommendations that CMS minimize the potential for PERM medical reviews to compromise fraud investigations and address disincentives for state Medicaid agencies to notify the PERM contractor of providers under fraud investigation. In its comments HHS described the actions it has taken and is considering taking to implement these recommendations. We are sending copies of this report to appropriate congressional committees, to the Secretary of Health and Human Services, the Administrator of CMS, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114, or cosgrovej@gao.gov or yocomc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. During the period of our review, fiscal year 2017 data represented the most recent, complete data for both Medicare and Medicaid fee-for- service (FFS) estimated improper payment amounts and rates. As of March 2019, the Centers for Medicare & Medicaid Services published the fiscal year 2018 Medicare FFS Supplemental Improper Payment Data report, but had not published the 2018 Medicaid FFS Supplemental Improper Payment Data report. The Centers for Medicare & Medicaid Services estimated Medicare FFS spending of $389 billion, and $32 billion in improper payments. Table 3 below presents updated fiscal year 2018 data for the Medicare improper payment data by the services examined in our report. Medicare and state Medicaid agencies have released template medical record documentation, such as certificates of medical necessity and plans of care that providers may use to document information necessary to ensure compliance with coverage policies. This appendix presents examples of such templates. Figure 7 presents a Medicare template that referring physicians can use to certify beneficiary need for home health services. Figure 8 presents a Medicare template that referring physicians can use to certify beneficiary need for home oxygen supplies. Figure 9 presents a template from the Indiana Medicaid program that hospices may use to document beneficiary plans of care. In addition to the contact named above, Leslie V. Gordon (Assistant Director), Michael Erhardt (Analyst-in-Charge), Arushi Kumar, and Dawn Nelson made key contributions to this report. Also contributing were Sam Amrhein, Vikki Porter, and Jennifer Rudisill.", "summary": "In fiscal year 2017, Medicare FFS had an estimated $23.2 billion in improper payments due to insufficient documentation, while Medicaid FFS had $4.3 billion—accounting for most of the programs' estimated FFS medical review improper payments. Medicare FFS coverage policies are generally national, and the program directly pays providers, while Medicaid provides states flexibility to design coverage policies, and the federal government and states share in program financing. Among other things, GAO examined: (1) Medicare and Medicaid documentation requirements and factors that contribute to improper payments due to insufficient documentation; and (2) the extent to which Medicaid reviews provide states with actionable information. GAO reviewed Medicare and Medicaid documentation requirements and improper payment data for fiscal years 2005 through 2017, and interviewed officials from CMS, CMS contractors, and six state Medicaid programs. GAO selected the states based on, among other criteria, variation in estimated state improper payment rates, and FFS spending and enrollment. The Centers for Medicare & Medicaid Services (CMS) uses estimates of improper payments to help identify the causes and extent of Medicare and Medicaid program risks and develop strategies to protect the integrity of the programs. CMS estimates Medicare and Medicaid fee-for-service (FFS) improper payments, in part, by conducting medical reviews—reviews of provider-submitted medical record documentation to determine whether the services were medically necessary and complied with coverage policies. Payments for services not sufficiently documented are considered improper payments. In recent years, CMS estimated substantially more improper payments in Medicare, relative to Medicaid, primarily due to insufficient documentation (see figure). For certain services, Medicare generally has more extensive documentation requirements than Medicaid. For example, Medicare requires additional documentation for services that involve physician referrals, while Medicaid requirements vary by state and may rely on other mechanisms—such as requiring approval before services are provided—to ensure compliance with coverage policies. Although Medicare and Medicaid pay for similar services, the same documentation for the same service can be sufficient in one program but not the other. The substantial variation in the programs' improper payments raises questions about how well the programs' documentation requirements help identify causes of program risks. As a result, CMS may not have the information it needs to effectively address program risks and direct program integrity efforts. CMS's Medicaid medical reviews may not provide the robust state-specific information needed to identify causes of improper payments and address program risks. In fiscal year 2017, CMS medical reviews identified fewer than 10 improper payments in more than half of all states. CMS directs states to develop corrective actions specific to each identified improper payment. However, because individual improper payments may not be representative of the causes of improper payments in a state, the resulting corrective actions may not effectively address program risks and may misdirect state program integrity efforts. Augmenting medical reviews with other sources of information, such as state auditor findings, is one option to better ensure that corrective actions address program risks. GAO is making four recommendations to CMS, including that CMS assess and ensure the effectiveness of Medicare and Medicaid documentation requirements, and that CMS take steps to ensure Medicaid's medical reviews effectively address causes of improper payments and result in appropriate corrective actions. CMS concurred with three recommendations, but did not concur with the recommendation on Medicaid medical reviews. GAO maintains that this recommendation is valid as discussed in this report.", "document_type": "gao"}
{"report": "Priority Enforcement Program. Under PEP, which was in effect from January 5, 2015 until February 20, 2017, DHS personnel were directed to, among other things, prioritize the apprehension, detention, and removal from the United States of aliens who pose a threat to national security, border security, and public safety, including convicted felons. It further directed DHS personnel to prioritize for removal new immigration violators and those who had been issued a final order of removal on or after January 1, 2014 and to exercise prosecutorial discretion, as appropriate, in accordance with these priorities and existing guidance. A 2011 ICE memorandum identified factors to consider when exercising prosecutorial discretion, such as the length of the individual’s presence in the United States, whether the person or person’s immediate relative has served in the U.S. military, on the basis of humanitarian reasons such as personal or family illness, among other factors. Executive Order 13768. Executive order 13768, issued on January 25, 2017, focuses on immigration enforcement within the United States. Among other things, the executive order lays out the administration’s immigration enforcement priorities for removable aliens. Specifically, the executive order prioritizes for the removal from the United States aliens who are removable based on certain criminal and security grounds in the Immigration and Nationality Act; as well as removable aliens who have been convicted of, charged with, or committed acts that constitute a criminal offense; have engaged in fraud or otherwise abused any government program; or who are determined to otherwise pose a risk to public safety or national security. In addition, it calls for the termination of the PEP and reinstitution of Secure Communities. See table 1 for a description of enforcement priorities for the removal of aliens from the United States under PEP and Executive Order 13768. The Secretary of Homeland Security issued the 2017 DHS memo to implement Executive Order 13768. According to the 2017 DHS memo, in addition to the priorities outlined in the executive order, the Director of ICE, Commissioner of CBP, and Director of U.S. Citizenship and Immigration Services may allocate resources to prioritize enforcement activities as they deem appropriate, such as by prioritizing enforcement actions against convicted felons or gang members. ICE issued a memo further directing efforts to implement the executive order and apply the guidance from the 2017 DHS memo. The ICE memo stated that ICE was to review all existing policies and guidance documents and revise or rescind relevant policies in order to ensure consistency with the executive order. In addition, ICE’s Office of the Principal Legal Advisor (OPLA) issued additional guidance to OPLA attorneys to implement the 2017 DHS memo. OPLA is responsible for providing legal advice, training, and services to support the ICE mission, and for defending the interests of the United States in the administrative and federal courts including immigration court proceedings. See figure 1 for a timeline of DHS memoranda and Executive Order establishing immigration enforcement priorities from 2015 to 2018. Prosecutorial Discretion. Prosecutorial discretion is the longstanding authority of an agency charged with enforcing a law to decide where to focus its resources and whether or how to enforce, or not to enforce, the law against an individual. Due to limited resources, ICE cannot respond to all immigration violations or remove all persons who are determined to be in the United States without legal status, and therefore, must exercise prosecutorial discretion in the enforcement of the law. In accordance with the DHS, ICE, and OPLA memos, agents and officers are to exercise prosecutorial discretion on a case-by-case basis based on the individual facts presented in consultation with the head of the field office, and prosecutorial discretion is not to be exercised in a manner that exempts or excludes a specified class or category of foreign nationals from enforcement of the immigration laws. ICE’s ERO conducts civil immigration enforcement actions, which includes administrative arrests, detentions, and removals. Arrests. ERO arrests aliens for civil violations of U.S. immigration laws. Through the Criminal Alien Program, ICE identifies and arrests potentially removable aliens who are incarcerated within federal, state, and local prisons and jails. The National Fugitive Operations Program identifies and arrests removable aliens who are at-large. ICE does not detain all aliens it arrests, due to lack of bed space, among other factors. To inform custody decisions for aliens who are arrested and not subject to mandatory detention, ICE guidance requires officers to consider certain factors, including risk of flight, risk of harm to public safety, and special vulnerabilities. For example, individuals with a physical or mental illness or disability, or individuals who fear being harmed in detention based on their sexual orientation or gender identity may be considered for release or alternatives to detention (ATD) based on these special vulnerabilities. The ATD program requires that, among other things, aliens released into the community agree to appear at all hearings and report to ICE periodically. Non-detained Unit. ERO is also responsible for supervising and ensuring that aliens who are not held in detention facilities comply with requirements to appear in immigration court for their administrative removal proceedings. ICE uses one or more release options when it determines that an alien can be released from ICE custody—including bond, order of recognizance, order of supervision, parole, and on condition of participation in the ATD program. Total ATD enrollment numbers ranged from about 29,000 in calendar year 2015 to over 78,000 in calendar year 2018. ICE does not track specific characteristics of individuals enrolled in ATD programs, including aliens who are pregnant, nursing, disabled, elderly, primary caregivers of minor children, among others. ICE may also release aliens on bond or an order of recognizance who do not pose a threat to public safety, present a low risk of flight, and who are not required to be detained. In addition, in rare instances, ICE may release an alien on an order of supervision when there is no significant likelihood of removal in the reasonably foreseeable future. For example, ICE may not be able to coordinate travel arrangements for certain aliens with final orders of removal who are from countries with which the United States does not have repatriation agreements. An alien subject to a final order of deportation or removal may also request a stay of deportation or removal. ICE may also release certain aliens on parole for urgent humanitarian reasons or significant public benefit, or for a medical emergency or legitimate law enforcement objective, on a case-by-case basis. Detentions. ICE is responsible for providing safe, secure, and humane confinement for detained aliens in the United States who may be subject to removal while they await the resolution of their immigration cases or who have been ordered removed from the United States. This includes aliens transferred to ICE from CBP who were apprehended at or between ports of entry. In fiscal year 2019, ERO oversaw the detention of aliens in 147 facilities authorized to house detainees for over 72 hours. ICE manages these facilities in conjunction with private contractors, state and local governments, and through contract with another federal agency. Within ERO, ICE Health Service Corps (IHSC) is responsible for providing direct medical, dental, mental health care, and public health services to detainees in 20 facilities authorized to house detainees for over 72 hours. Facilities serviced by IHSC include service processing centers, contract detention facilities, dedicated intergovernmental service agreement facilities, and family residential centers. IHSC medical staff are to monitor and implement policy provisions related to pregnant and mentally ill detainees. At detention facilities that are not staffed with IHSC personnel, similar services are provided by local government staff or private contractors and overseen by ICE. Removals. ICE removes aliens who have been determined to be removable and not eligible for any requested relief or protection pursuant to an administrative final order of removal. A removal is defined as the compulsory and confirmed movement of an inadmissible or deportable alien out of the United States. ICE removals include both aliens arrested by ICE and aliens who were apprehended by CBP and transferred to ICE. ERO operates across 24 areas of responsibility nationwide and each area of responsibility is led by a field office director. Each ERO field office director is required by ICE policy to designate supervisory level employees to serve, as a collateral duty, as field liaisons for their area of responsibility tasked with monitoring and implementing the provisions of policies for certain selected populations. These field liaison roles include the LGBTI Field Liaison, Child Welfare Field Point of Contact, Supporting Disability Access Coordinator, and Juvenile Coordinator. In addition to ERO and OPLA, ICE Homeland Security Investigations (HSI) conducts worksite enforcement operations among other law enforcement operations such as oversight of the Student and Exchange Visitor program. This includes arresting undocumented workers and employers who knowingly hire them. We did not include HSI worksite enforcement arrests in our analysis of ICE arrest data because we were unable to identify the number of unique arrests in these data for the purpose of depicting general arrest trends. ERO arrests, detentions, and removals varied during calendar years 2015 through 2018, and increased overall for the period, as shown in figure 2. Specifically, males, aliens from four countries—Mexico, Guatemala, El Salvador, and Honduras—and convicted criminals accounted for the majority of ICE arrests and removals. The majority of ICE detentions were made up of males, aliens from the same four countries, and non- criminals. See appendix II for additional information on ERO arrests, detentions, and removals by gender, country of citizenship, arresting agency, and criminality. ERO Arrests. The number of ERO arrests varied from calendar years 2015 through 2018 but increased overall from 112,870 in 2015 to 151,497 in 2018, see figure 2 above. Male aliens, citizens of four countries— Mexico, Guatemala, El Salvador and Honduras—and arrests of aliens from state and local jails, through the Criminal Alien Program, accounted for the majority of these arrests each year from 2015 through 2018. Further, ERO arrests increased in all ERO areas of responsibility from calendar years 2015 and 2016, when PEP was in effect, to calendar years 2017 and 2018, following implementation of the 2017 DHS memo. Arrests of convicted criminals accounted for the majority of arrests in all areas of responsibility during both periods. However, as shown in figure 3, the proportion of arrests of convicted criminals decreased in each area of responsibility due to an increased number of arrests of non-criminals following the implementation of the 2017 DHS memo. See appendix II for additional information on ERO arrests by gender, country of citizenship, arresting agency, and criminality. ERO Detentions. The number of ERO detentions varied from calendar years 2015 through 2018 but increased overall from 324,320 in 2015 to 438,258 in 2018. Male aliens and citizens of four countries—Mexico, Guatemala, El Salvador and Honduras—collectively accounted for most ERO detentions. The majority of detentions resulted from CBP arrests at or between ports of entry. While the number of ERO detentions of convicted criminals stayed relatively stable from 2015 to 2018, the number of detentions of non-criminals increased from 171,856 in 2015 to 279,469 in 2018 and accounted for the majority of ERO detentions each year, as shown in figure 4. See appendix II for additional information on detentions by gender, country of citizenship, arresting agency, and criminality. For the purposes of this report and our presentation of ICE data, we refer to potentially removable aliens without criminal convictions known to ICE as “non-criminals” and aliens with criminal convictions known to ICE as “convicted criminals.” According to ICE officials, administrative arrests of non-criminals include individuals who have been charged but not convicted of a crime as well as those with no prior criminal history. According to ICE, ICE officers electronically request and retrieve criminal history information about an alien from the FBI’s National Crime Information Center database, which maintains a repository of federal and state criminal history information, and other sources. We used ICE’s determination of criminality for our analysis. ERO Removals. The number of ERO removals varied from calendar years 2015 through 2018 but increased overall from 231,559 in 2015 to 261,523 in 2018. Male aliens and citizens of four countries—Mexico, Guatemala, El Salvador and Honduras—collectively accounted for most ERO removals. The majority of removals resulted from CBP arrests at or between ports of entry. While removals of both convicted criminals and non-criminals increased overall, removals of convicted criminals accounted for the majority of removals each year, see figure 5. See appendix II for additional information on removals by gender, country of citizenship, arresting agency and criminality. According to ICE officials, in early 2018, ERO conducted a review of all existing policies and related documents to help ensure alignment with the 2017 DHS memo, resulting in operational policies related to six of the eight selected populations discussed in this report. The six policies in effect as of July 2019 for the selected populations provide direction and guidance to ERO officers on the identification, detention, care, and removal of aliens who are: individuals with mental disorders, transgender, individuals with disabilities, parents of minors, pregnant, and juveniles. Of the six policies in effect, three were not impacted by the 2017 DHS memo and ERO did not make changes to these policies; two were impacted by the 2017 DHS memo and were revised to remove language ERO determined to be inconsistent with the memo; and guidance on managing juveniles was first issued after the 2017 DHS memo. For the remaining two populations, ERO does not have a separate policy on care provided to detainees who are nursing and as a result of the policy review, rescinded a prior policy related to exercising prosecutorial discretion for elderly individuals, as shown in figure 6. Individuals with Mental Disorders. In May 2014, ICE issued a memo titled Identification of Detainees with Serious Mental Disorders or Conditions, which sets forth procedures to assist ICE and detention facility personnel in identifying detainees with serious mental disorders or conditions in order to assess appropriate facility placement and treatment. To identify individuals with mental disorders, ICE’s national detention standards require facilities to conduct an initial medical screening for all detainees, including a documented mental health screening, a 14-day full medical assessment, with mental health components, and timely referral for follow-up mental evaluations, diagnosis, and treatment. ICE’s policy also requires detention facilities to notify ICE field office directors of detainees with specified serious mental disorders. In addition, the policy requires that relevant personnel meet regularly to monitor the cases of detainees with serious mental disorders until their removal or release. ERO officials in all six areas of responsibility we visited said that these meetings are conducted weekly or biweekly with attorneys, medical staff, and ERO management staff to discuss and evaluate the needs of each detainee’s medical care and security needs. According to ICE, this memo did not need to be revised to align with the 2017 DHS Memo. Our analysis of ICE data shows that the number of detentions of individuals with mental disorders at IHSC-staffed facilities varied from calendar years 2015 through 2018 but increased overall from 8513 to 8796 individuals. Transgender Individuals. In June 2015, ICE issued a memo titled Further Guidance Regarding the Care of Transgender Detainees, which provides guidance regarding the placement and care of transgender adult detainees in ERO custody. This memo provides guidance for initial processing of transgender detainees who voluntarily disclose their gender identity to detention officers. Further, when a detainee self-identifies as transgender, the memo directs ERO officers to make individualized placement determinations to ensure the detainee’s safety, and to ensure the facility chosen for placement is able to provide appropriate care for the individual, and to the extent practicable to consider the availability of medical personnel who have experience providing care and treatment to transgender detainees, including the delivery of hormone therapy. This memo also directs ERO to designate a National LGBTI Coordinator to serve as the primary point of contact and subject matter expert for ERO regarding the care and treatment of detainees in ERO custody who self- identify as transgender. Specifically, the National LGBTI Coordinator is to evaluate and report information from all relevant ICE data systems regarding the demographics, care, and custody of transgender detainees and ensure field compliance with the provisions of this memo, among other things. Further, each field office is required to have a LGBTI Field Liaison, appointed by the Field Office Director. The memo directs LGBTI Field Liaisons to provide regular updates to the national ERO LGBTI Coordinator and ERO Headquarters on the progress of implementing and maintaining the provisions of this memo, which includes determining the appropriateness of facilities to house transgender detainees. In addition, the memo requires certain detention facilities to convene a meeting no later than 72 hours after a transgender detainee’s arrival to the facility to assess medical, psychological, and housing needs. During our site visits, officers in three of the six areas of responsibility we visited said that they conduct these meetings with relevant ERO management staff and medical officials in accordance with the memo. According to ICE officials, the transgender care memo did not need to be revised to align with the 2017 DHS Memo. The transgender care memo states that field office directors may exercise prosecutorial discretion for transgender individuals who are not subject to mandatory detention. Field ERO officers in five of the six areas of responsibility we visited explained that ERO generally does not detain transgender individuals unless their criminal histories warrant detention, in accordance with the memo. Specifically, officers in three of these five areas of responsibility reported that transgender individuals are likely to be released on bond or under an order of supervision. However, in the sixth area of responsibility, one ERO officer observed an increase in the detention of transgender individuals beginning in early 2017, which the official attributed to the revised priorities described in the 2017 DHS memo. In addition, attorneys from three NGOs we met with also observed an increase in the detention of transgender individuals or described ongoing challenges related to a decrease in the availability of dedicated transgender housing facilities. They also provided anecdotes of transgender clients who had been detained or who experienced challenges obtaining access to appropriate medical care while in detention. Our analysis of ICE data shows that the number of detentions of transgender individuals increased from 237 in calendar year 2016 to 284 in calendar year 2018. While ICE does not have separate policies for aliens who are lesbian, gay, bisexual, or intersex, the national LGBTI coordinator and LGBTI field liaisons also serve as subject matter experts for the care and treatment of these detainees. In addition, the transgender care memo prohibits discrimination or harassment of any kind based on a detainee’s sexual orientation or gender identity. As such, ERO officers may take steps to protect a detainee who expresses safety concerns based on their sexual orientation, according to ERO officials. According to ERO officers in five of the six areas of responsibility we visited, they do not ask detainees about sexual orientation unless the individual voluntarily discloses this information. Additionally, ERO officers in the same five areas of responsibility stated that they do not take sexual orientation into consideration for detention or housing decisions, unless an individual specifically requests protective custody due to safety concerns or harassment. Individuals with Disabilities. In December 2016, ERO issued a directive titled Assessment and Accommodations for Detainees with Disabilities, which establishes policy and procedures for ERO to oversee and communicate with detention facilities on the identification, assessment, and accommodation of detainees with disabilities. According to this directive, ERO field leadership is to notify detention facilities in each area of responsibility of their existing obligations under federal law to accommodate detainees with disabilities. These obligations include maintaining a process to identify these detainees through observation, assessments, screenings, and self-identification; notifying detainees of their right to request accommodations; and establishing a process to inform a detainee of the final decision on the request for accommodations, including whether the facility will provide alternative accommodations that are equally effective as those requested; among other things. In addition, this directive requires ERO to designate an ERO disability access coordinator who is to serve as the primary point of contact and subject matter expert for ERO headquarters and the field regarding the accommodation of, and communication with, detainees with disabilities in ERO custody. Among other duties, the ERO disability access coordinator is responsible for evaluating information from all relevant ICE data systems regarding the identification, care, approved accommodations and custody of detainees with disabilities; as well as maintaining records of detainees with communication and mobility impairments, including records of denials of detainee requests for accommodations by facilities. According to the directive, detainees with communication impairments include detainees with hearing, visual, and speech impairments (e.g., detainees who are deaf or hard of hearing, blind, or nonverbal). Detainees with mobility impairments include detainees with physical impairments who require a wheelchair, crutches, prosthesis, cane, other mobility device, or other assistance. Accommodations for these impairments may include accessible showers, Braille material, or note takers for persons with physical and sensory impairments, among other things. The ERO disability access coordinator is also responsible for helping to ensure compliance with the provisions of this directive. Field office directors are required to appoint at least one supervisory-level employee to serve as the supporting disability access coordinator for each area of responsibility. Supporting disability access coordinators are responsible for serving as the main point of contact for their field office regarding compliance with federal law and DHS, ICE, and ERO regulations, detention standards, policies, and procedures related to detainees with disabilities. Supporting disability access coordinators are also responsible for collaborating and communicating with ERO headquarters, field office, detention facility, and health care personnel to monitor the care and treatment of detainees with disabilities, among other things. In all six areas of responsibility we visited, supporting disability access coordinators and medical staff told us that they track detainees who receive accommodations for communication and mobility impairments by recording the accommodation on a form that they submit to ERO headquarters. According to ICE, the Assessment and Accommodations for Detainees with Disabilities directive did not need to be revised to align with the 2017 DHS Memo. This directive states that it is meant to implement and complement the requirements of Section 504 of the Rehabilitation Act of 1973 and states that detainees with disabilities will be provided an equal opportunity to access, participate in, or benefit from in-custody programs, services, and activities, and that detainees with disabilities will be provided with auxiliary aids and services as necessary to allow for effective communication. Further, the directive states that a field office director may consider releasing from ICE custody a detainee with an impairment or disability who is not subject to mandatory detention. ERO officers in five areas of responsibility we visited reported that they consult with the supporting disability access coordinator, medical staff, or a supervisor to determine whether local detention facilities are able to provide appropriate accommodations. Our analysis of ICE data shows that the number of detentions of individuals with communication and mobility impairments increased from 434 to 530 in calendar years 2017 to 2018. Parents or Legal Guardians of Minors. In August 2017, ICE issued a policy titled Detention and Removal of Alien Parents or Legal Guardians, which provides guidance regarding the detention and removal of alien parents and legal guardians, including those with children who are U.S. citizens and legal permanent residents and parents with ongoing cases in family court or child welfare proceedings in the United States. This policy directs ERO to designate a child welfare coordinator to serve as the primary point of contact and subject matter expert for all ICE personnel regarding child welfare issues related to detained alien parents. The child welfare coordinator is also responsible for evaluating information from all relevant ICE data systems regarding detained alien parents or legal guardians of U.S. citizen and legal permanent resident minors and sharing appropriate information with field points of contact, among other things. Specifically, this policy directs field office directors to make appropriate arrangements for detained parents to attend child welfare proceedings. ERO officers in three of the six areas of responsibility we visited stated that they coordinate visits to family courts for the detained parent to appear at these hearings. The field office director in each area of responsibility is to designate a field point of contact to communicate with the child welfare coordinator and address public inquires related to detained parents or legal guardians in ERO custody. The August 2017 policy superseded an August 2013 policy titled Facilitating Parental Interests in the Course of Civil Immigration Enforcement Activities, which ERO revised to align with the 2017 DHS memo. In the revised policy, ERO removed language indicating that field office directors should weigh whether an exercise of prosecutorial discretion may be warranted for an alien who is a parent or legal guardian of a U.S. citizen or legal permanent resident minor or is a primary caretaker of a minor, and to exercise such discretion as early as possible. ERO officers in five of the six areas of responsibility we visited stated that they typically do not detain parents of minors, unless criminal history warrants detention. Attorneys we met with from a NGO that provides services to immigrant families and refugees stated that they have observed an increase in the number and length of detentions of parents or legal guardians of minors since January 2017. We were not able to identify trends in detention of detained parents because ERO does not collect or maintain data on this population in a readily available format. Pregnant Women. In December 2017, ICE issued a directive titled Identification and Monitoring of Pregnant Detainees, which sets forth policy and procedures to ensure pregnant detainees in ICE custody for immigration violations are identified, monitored, tracked, and housed in an appropriate facility to manage their care. According to ICE policy on women’s health, pregnant women are identified upon arrival to a detention facility because all women of childbearing age undergo a pregnancy test during intake processing. According to the December 2017 directive, IHSC personnel are responsible for notifying the field office director and IHSC headquarters, as soon as practical, when a pregnant detainee is identified; monitoring the condition of pregnant detainees, including the general health of the pregnant detainee and medical condition of the fetus; and communicating with the field office director about any specific risk factors or concerns. In addition, IHSC personnel are to provide oversight and review of facility capabilities to determine if the needs of a pregnant detainee can be accommodated and recommend to the field office director when a transfer to another facility is necessary for appropriate medical care. Further, IHSC personnel are to develop and maintain a system for tracking and monitoring all pregnant detainees. This policy superseded an August 2016 version with the same title, which ERO revised to align with the 2017 DHS memo, according to ICE officials. In the revised version, ERO removed language stating that absent extraordinary circumstances pregnant women will generally not be detained by ICE. In five of the six areas of responsibility we visited, ERO officers stated that unless mandatory detention is required, they still generally avoid detaining pregnant women. In addition, ERO officers in all six areas of responsibility we visited stated that they are less likely to detain and may release a woman who is having a high risk pregnancy or in the third trimester of her pregnancy. However, an official in the sixth area of responsibility noted that under the revised policy, pregnant women may be detained during the third trimester, if she is likely to be removed quickly and has medical clearance to fly. Officers in two of the six areas of responsibility we visited noted that pregnant women may also be released on bond, under an order of supervision, or other non- detention options, after assessing the facts of the case. Attorneys and policy advocates we met with from three NGOs that represent a range of immigrant populations stated that they have observed increases in the detention of pregnant women since January 2017. Attorneys from another NGO we met with provided anecdotes of cases of pregnant detainees who experienced medical challenges, including miscarriages while in custody. Our analysis of ICE data shows that the number of detentions of pregnant women varied, but increased overall, from 1380 in calendar year 2016 to 2098 in calendar year 2018. Juveniles. In April 2018, ICE issued the Field Office Juvenile Coordinator Handbook to guide ERO staff in processing, transporting, managing, and removing juveniles—persons encountered by ERO who have not reached 18 years of age. Field office juvenile coordinators, who serve as local subject-matter experts on juvenile matters for each area of responsibility, provide policy guidance to ERO personnel within their areas of responsibility, and assist with case review and custody redeterminations. Field office juvenile coordinators are also required to coordinate with other federal agencies including the Office of Refugee Resettlement, where juveniles designated as unaccompanied alien children are typically transferred. According to ERO policy, unaccompanied alien children apprehended by ERO or transferred into ERO custody by CBP are to be placed in the care of the Office of Refugee Resettlement within 72 hours of identification, if they are not repatriated at the border. The Field Office Juvenile Coordinator Handbook was released after the 2017 DHS memo and aligns with the 2017 DHS Memo. According to officers in four of the six areas of responsibility we visited, ERO does not target juveniles for arrests, unless they have criminal records. For example, officers we met with in one area of responsibility stated that ERO typically does not target juveniles in that location, unless they are affiliated with gangs, because they are unlikely to pose a public safety threat. Our analysis of ICE data shows that the number of arrests of juveniles varied, but increased overall, from calendar years 2015 through 2018. We excluded juveniles from our analysis of individual ICE detention data because ICE is generally not responsible for detaining juveniles, as discussed above. Nursing Women. While ICE does not have a separate policy on the care, detention, or removal of women who are nursing, the 2017 Directive on Women’s Health Services provides guidance to IHSC staff on the delivery and administration of health services to this population. According to this directive, women who are nursing are identified during initial processing before being placed into custody at a detention facility because ERO officials and medical personnel are required to ask women if they are breastfeeding. Medical personnel make recommendations pertaining to the detention of women who are nursing, and in most cases, these detainees are placed in IHSC-staffed facilities. IHSC personnel record and use this information to monitor the care and needs of women who are nursing, according to IHSC officials. In five of the six areas of responsibility we visited, officers stated that they typically do not detain women who are nursing, unless their criminal histories warrant detention. Specifically, health officials in one of the five areas of responsibility explained that if a nursing mother is detained, she is typically released within a few hours or placed on bond or order of supervision. Our analysis of ICE data shows that the number of detentions of nursing women at IHSC-staffed facilities varied from calendar years 2015 through 2018 but increased overall from 157 in 2015 to 381 in 2018. Elderly Individuals. ICE no longer has a policy guiding the detention or care of elderly detainees. According to ICE guidance on assessing individuals with special vulnerabilities during the intake process, ICE generally considers someone to be elderly starting at age 65. However, the guidance instructs agents and officers to assess whether these individuals have physical indicators of infirmity or fragility caused by old age when making decisions regarding detaining or releasing them. In February 2018, as part of its effort to align internal policies with the 2017 DHS memo, ERO rescinded a 2009 policy directing officers to administratively close cases of non-criminal fugitives who are 70 years old or older for humanitarian/health reasons. ERO officers in five of the six areas of responsibility we visited reported that they do not target individuals who are elderly and continue to consider criminal history and medical condition when deciding whether to detain them. For example, officials in one of these five areas of responsibility explained that someone who committed an aggravated felony would be subject to mandatory detention regardless of age, but if the individual has a serious medical condition, such as advanced cancer, ERO may decide to release them from custody because the agency would be responsible for the cost of their medical treatments while they are in custody. Officers in the sixth area of responsibility said they have started to detain individuals who are elderly following the issuance of the 2017 DHS memo, but noted that they coordinate with the courts to expedite these hearings before an immigration judge who may order the release of an elderly detainee. Attorneys we met with from a NGO that provides services to immigrant families and refugees stated that they have observed an increase in detentions of individuals who are elderly, and only those with serious medical issues were considered for release. Our analysis of ICE data shows that the number of detentions of individuals who were elderly varied, increasing overall, from 882 in calendar year 2015 to 1159 in calendar year 2018. Available ICE data show that detentions of most of the selected populations in our review varied between calendar years 2015 and 2018. Specifically, detentions of transgender individuals and pregnant women increased from calendar years 2016 to 2018, after ICE began collecting data for these populations. Similarly, detentions of individuals with disabilities increased from 2017 to 2018, after ICE began collecting data for this population. Detentions of individuals with mental disorders and nursing women at IHSC-staffed facilities varied from calendar years 2015 to 2018. Finally, detentions of individuals who were elderly varied, increasing overall during the same timeframe. We were unable to obtain data on parents or legal guardians of minors in ICE custody because ICE does not collect or maintain data on this population in a readily available format. ICE began collecting and maintaining data on transgender individuals who voluntarily disclose their gender identity to ICE officers in November 2015, as previously discussed. ERO officials said they use these data to monitor the placement and care of transgender individuals in ICE custody, in accordance to ICE’s memo on Further Guidance Regarding the Care of Transgender Detainees. These data show that the number of detentions of transgender individuals increased from calendar years 2016 through 2018, as shown in table 2. Detentions resulting from CBP arrests accounted for about half of the total detentions of transgender individuals in 2016 and 2017, increasing to 69 percent in 2018. Also shown in table 2, detentions of non-criminal transgender individuals increased from calendar years 2016 through 2018, increasing from 46 percent of total detentions of transgender individuals in 2016 to 71 percent in 2018. Detentions of non-criminal transgender individuals include both detentions of individuals with pending criminal charges (ranging from 12 to 24 percent) and individuals with no recorded criminal history (ranging from 76 to 88 percent). Detentions resulting from CBP arrests comprised most of these detentions (ranging from 77 to 91 percent). Detentions of transgender individuals with criminal convictions decreased over the same period, and most resulted from ICE arrests (ranging from 71 to 84 percent). For the purposes of this report and our presentation of ICE data, we refer to potentially removable aliens without criminal convictions known to ICE as “non-criminals” and aliens with criminal convictions known to ICE as “convicted criminals.” According to ICE officials, administrative arrests of non-criminals include individuals who have been charged with, but not convicted of a crime, (we refer to these as “aliens with pending criminal charges”), as well as those with no prior criminal history, (we refer to these as “aliens with no recorded criminal history”). According to ICE, ICE officers electronically request and retrieve criminal history information about an alien from the FBI’s National Crime Information Center database, which maintains a repository of federal and state criminal history information, and other sources. We used ICE’s determination of criminality for our analysis. ICE began collecting and maintaining data on certain detainees with disabilities–i.e., those with communication and mobility impairments— who disclosed their impairment or who were identified by facility staff as having an impairment in January 2017, in accordance with its directive, titled Assessment and Accommodations for Detainees with Disabilities. These data show that detentions of individuals with disabilities increased from calendar years 2017 to 2018, as shown in table 3. Detentions resulting from ICE arrests accounted for the majority of these detentions (70 percent in 2017 and over 50 percent in 2018). Also shown in table 3, detentions of convicted criminals with disabilities decreased from calendar years 2017 to 2018, and accounted for the majority of total detentions of this population (67 percent in 2017 and 53 percent in 2018). Most of these detentions resulted from ICE arrests (89 percent in 2017 and 72 percent in 2018). Detentions of non-criminals in this population increased from calendar years 2017 to 2018. Detentions of individuals with no recorded criminal history accounted for most detentions of non-criminals in this population (71 percent in 2017 and 79 in 2018 percent), and the majority resulted from CBP arrests (68 percent in 2017 and 74 percent in 2018). ICE began collecting and maintaining data on pregnant women in ICE’s custody in June 2015. IHSC officials said they use these data to monitor the condition of pregnant women in ICE custody, including the term of the pregnancy, general health of the pregnant detainee, and medical conditions of the fetus, in accordance to ICE’s directive on Identification and Monitoring of Pregnant Detainees. These data show that the number of detentions of pregnant women varied, but increased overall from calendar years 2016 through 2018, as shown in table 4. Detentions resulting from CBP arrests accounted for most of the total detentions of pregnant women each year (ranging from 90 to 96 percent). Also shown in table 4, detentions of non-criminal pregnant women varied from calendar years 2016 through 2018, but increased overall. Detentions of non-criminal pregnant women accounted for most of the total detentions of pregnant women each year (ranging from 91 to 97 percent), and detentions of women with no recorded criminal history accounted for almost all of these detentions (ranging from 96 to 99 percent). Detentions of convicted criminal pregnant women also increased overall for the period. ICE began collecting and maintaining data needed to identify individuals with mental disorders at IHSC-staffed facilities in August 2013. According to IHSC officials, ICE does not collect these data for non-IHSC staffed facilities, in part because many of these facilities do not have electronic health records. However, IHSC personnel are notified of detainees with mental disorders at non-IHSC staffed facilities and these individuals may be transferred to another facility if the current facility is unable to provide appropriate care. While we were not able to present the overall number of detentions of individuals with mental disorders in ICE custody, we reviewed available ICE data to indicate the number and characteristics of detentions of individuals with mental disorders at IHSC- staffed facilities. These data show that the number of detentions of individuals with mental disorders at IHSC-staffed facilities varied from calendar years 2015 through 2018, as shown in table 5. Detentions resulting from CBP arrests accounted for the majority of these detentions (ranging from 53 to 67 percent) in 2015, 2016, and 2018. In 2017, detentions resulting from ICE arrests accounted for the majority (51 percent) of these detentions. Also shown in table 5, detentions of non-criminals with mental disorders varied from calendar years 2015 through 2018. These detentions accounted for the majority of total detentions of individuals with mental disorders in 2015, 2016, and 2018 (ranging from about 53 to 58 percent). Detentions of individuals with no recorded criminal history accounted for most detentions of non-criminals for this population (ranging from 79 to 92 percent), and most resulted from CBP arrests (ranging for 77 to 97 percent). Detentions of convicted criminals with mental disorders varied over the period and the majority resulted from ICE arrests (ranging from 71 to 79 percent). IHSC began collecting and maintaining data needed to identify women who are nursing at IHSC-staffed facilities, which is where ICE typically detains women who are nursing, in August 2013. These data are used to monitor the care and needs of women who are nursing, according to IHSC officials. While we were not able to present the overall number of detentions of nursing women in ICE custody, we reviewed available ICE data to indicate the number and characteristics of detentions of nursing women at IHSC-staffed facilities. These data show that the number of detentions of nursing women at IHSC-staffed facilities varied from calendar years 2015 through 2018, as shown in table 6. Detentions resulting from CBP arrests accounted for most of the detentions of women who were nursing each year (ranging from 98 to 99 percent). Also shown in table 6, detentions of both non-criminal and convicted criminal nursing women at IHSC-staffed facilities varied from calendar years 2015 through 2018. Detentions of non-criminal women who were nursing accounted for most of the total detentions of nursing women at IHSC-staffed facilities each year (ranging from 98 to 99 percent), and detentions of women who were nursing with no recorded criminal history accounted for almost all of these detentions (ranging from 99 to 100 percent), and resulted from CBP arrests (ranging from 98 to 100 percent). From calendar year 2015 through 2018, ICE collected and maintained data on a detainee’s date of birth and is able to identify whether an individual is elderly, defined as someone who is over 65 years old, by calculating the individual’s age at the time they are detained. ICE does not collect or maintain specific data on whether an individual is elderly because it does not have a separate policy for elderly detainees. Rather, ICE considers an individual’s health, criminal history, and other factors when making detention determinations, according to officials. ICE data show that the number of detentions of individuals who were elderly varied, but increased overall from calendar years 2015 through 2018, as shown in table 7. Detentions resulting from ICE arrests accounted for the majority of detentions of individuals who were elderly each year (ranging from 64 to 71 percent). Also shown in table 7, detentions of both non-criminal and criminal individuals who were elderly varied from calendar years 2015 through 2018, and increased overall. Detentions of convicted criminals accounted for the majority of detentions of individuals who were elderly each year (ranging from 65 to 74 percent) and most of these detentions resulted from ICE arrests (ranging from 82 to 85 percent). Detentions of individuals who were elderly with no recorded criminal history accounted for most detentions of non-criminal individuals who were elderly (ranging from 80 to 91 percent), and the majority resulted from CBP arrests (ranging from 70 to 74 percent). While ICE collects information on detained parents or legal guardians, including those of U.S. citizens and legal permanent resident minors, this information is not maintained in a readily available format that would allow ICE to systematically identify such detained parents and ensure officers are collecting information on this population as required by policy. According to ICE officials, before making custody determinations, ICE officers are instructed to inquire whether arrested aliens are parents or legal guardians of minors, including parents of U.S. citizen and legal permanent resident minors. ICE officers are to enter this information in a separate tab in the ENFORCE Alien Detention Module, a subsystem within ICE’s data system for recording information about individuals in its custody. This information on detained parents, however, cannot be readily searched to identify all detained parents or legal guardians in custody. Therefore, ICE does not know how many detained parents or legal guardians are in custody, including parents of U.S. citizen and legal permanent resident minors, during any given time. In accordance with a currently recurring Congressional reporting requirement, ICE generates a semi-annual report on removals of parents of U.S.-born citizen children. However, officials explained that they must review this information manually to generate the report and added that ICE is not required to report in an aggregate way on detained parents of U.S. citizen or legal permanent residents. ICE also tracks individual cases requiring specific actions, such as arranging transportation for parents to attend child welfare proceedings or accommodating visitation for parents with mandated child visitation schedules. However, according to ICE officials, these parents represent a small proportion of all parents in ICE custody. ICE’s policy on Detention and Removal of Alien Parents or Legal Guardians requires ICE personnel to enter information into ENFORCE once a detained alien has been determined to be a parent or legal guardians of a U.S. citizen or legal permanent resident minor. As previously mentioned, this policy also requires the Child Welfare Coordinator to evaluate information from all relevant ICE data systems regarding detained parents or legal guardians of minors, including parents of U.S. citizen and legal permanent resident minors, and share appropriate information with the ERO field points of contact. ICE’s policy further states that in pursuing the enforcement of U.S. immigration laws against parents of minors, ICE personnel should remain cognizant of the impact enforcement actions may have on U.S. citizen or legal permanent resident minors. Standards for Internal Control in the Federal Government call for design of any data collection to collect quality information, and for management to use quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks. Because information entered into ICE’s data system on detained parents or legal guardians, including those of U.S. citizen or legal permanent resident minors, is not maintained in a readily available format, ICE headquarters officials cannot ensure that ICE officers are collecting and entering this information into the system as required by policy. According to ICE officials, the agency had previously considered implementing a system update to readily identify certain detained parents of minors, but as of October 2019 is no longer considering this update. Collecting and maintaining information in a readily available format on detained parents of U.S. citizen or legal permanent resident minors could help ensure that ICE personnel can identify, evaluate, and share information on this population, as required by ICE policy. In addition, collecting and evaluating this information would provide greater transparency regarding the impacts of ICE’s enforcement actions on U.S. citizen or legal permanent resident minors. In 2015, DHS reported that about 12 million aliens were residing in the United States without lawful status or presence, which includes parents of U.S. citizen, legal permanent resident, and alien minors. Through its policies, ICE has established the importance of collecting and maintaining information on detained parents and legal guardians of U.S. citizen and legal permanent resident minors. However, because ICE has not implemented a process to collect or maintain this information in a readily available format, it does not have reasonable assurance that it can identify all detained parents and legal guardians of U.S. citizen and legal permanent resident minors. Therefore, ICE cannot evaluate and share this information and ensure its officers are collecting information on this population in accordance with its policy. Implementing a process to collect and maintain this information in a readily available format would allow ICE to better assess the impacts of its enforcement actions on U.S. citizen and legal permanent resident minors and help improve ICE oversight efforts. The Director of ICE should implement a process to collect and maintain data in a readily available format on detained parents or legal guardians of U.S. citizen and legal permanent resident minors to ensure that information on this population is entered into ICE’s data system as required by policy. (Recommendation 1) We provided a draft of this report for review and comment to DHS. DHS provided comments, which are reproduced in appendix XI. DHS also provided technical comments, which we incorporated, as appropriate. DHS did not concur with our recommendation. Specifically, in its comments, DHS stated that data on detained parents or legal guardians of U.S. citizens and legal permanent residents are available to approved EARM users and that we did not identify any problems with the quality of the data. However, as we noted in our report, these data are not readily available because ICE’s data on family relationships, including parents or legal guardians of U.S. citizens and legal permanent resident minors, can only be accessed by manually reviewing each separate case file in EARM. To that end, we or anyone else wishing to do so are unable to determine whether there are problems with the data as ICE is not able to provide aggregate data that would allow us to assess the quality or to report on these data. In its comments, DHS states that ICE does not have any requirement or need to aggregate data on this particular group and doing so would not better inform ICE’s decision making processes. However, as noted in the report, ICE’s policy states that in pursuing the enforcement of U.S. immigration laws against parents of minors, ICE personnel should remain cognizant of the impact enforcement actions may have on U.S. citizen or legal permanent resident minors. Without making these data readily available, ICE is not able to account for the overall impact of its enforcement actions on U.S. citizen or legal permanent resident minors whose parents or legal guardians have been detained. Additionally, headquarters and field officials we met with during the course of this review agreed that having this information readily available would be useful. They also explained that ICE was developing a method to better track and report on primary caregivers of children. However, in October 2019, ICE officials stated that the agency is no longer considering this improvement. We continue to believe that collecting and maintaining information in a readily available format on detained parents or legal guardians of U.S. citizen or legal permanent resident minors could help ensure that ICE personnel can identify, evaluate, and share information on this population, as required by ICE policy. Without such data, ICE headquarters officials cannot ensure that ICE officers are collecting and entering this information into the system as required. In addition, collecting and evaluating this information would provide greater transparency regarding the impacts of ICE’s enforcement actions on U.S. citizen or legal permanent resident minors. We are sending copies of this report to the appropriate congressional committees, and the Acting Secretary of the Department of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or goodwing@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix XII. This appendix provides additional information on our objectives, scope, and methodology. Specifically, our objectives were to examine the following questions: 1. What does ICE data show about ICE arrests, detentions, and removals from calendar years 2015 through 2018? 2. What policies are in effect for selected populations and what changes did ICE make to align these policies with the 2017 DHS memo? 3. To what extent does ICE collect data on selected populations in detention and what do these data show? To address our first question, we analyzed individual-level data from the U.S. Immigration and Customs Enforcement (ICE) Integrated Decision Support (IIDS) database, to determine the total number of ICE Enforcement and Removal Operations (ERO) administrative arrests (arrests), detentions, and removals from January 2015 (the start of the Priority Enforcement Program) through December 2018 (to include the first two years for the 2017 DHS Memo). ERO conducts civil immigration enforcement actions, which includes arrests for civil violations of U.S. immigration laws, detentions, and removals. Arrests. We analyzed individual-level arrest data from IIDS to determine the total number of ERO arrests for each calendar year 2015 through 2018. We examined multiple data fields from the individual-level arrest data, including alien file number, family name, given name, gender, country of citizenship, arrest date, area of responsibility, and criminality, among other variables. Because aliens may have multiple arrests, we used alien number and arrest date to identify the unique number of arrests rather than the number of unique aliens who were arrested. We excluded from our analysis arrest records that had a missing alien number, an invalid alien number— i.e., that included all zeroes or had duplicate alien number and arrest date combinations—or records that indicated test in the name fields. We analyzed these data to determine total numbers of arrests by gender, country of citizenship, criminality, arresting program, and area of responsibility. To determine the number of arrests by gender, we analyzed IIDS individual-level arrest data. We also analyzed these data to determine the number of arrests by criminality for each gender, using ICE’s determination of criminality for our analysis, as discussed below. To determine the number of arrests by country of citizenship, we analyzed IIDS individual-level arrest data. ICE obtains country of citizenship data from arrest reports, which may be based on documentation or self-reported. To determine the number of arrests by criminality, we analyzed IIDS individual-level arrest data. For the purposes of this report and our presentation of ICE data, we refer to potentially removable aliens without criminal convictions known to ICE as “non-criminals” and aliens with criminal convictions known to ICE as “convicted criminals.” According to ERO officials, arrests of non-criminals include individuals who have been charged but not convicted of a crime as well as those with no prior criminal history. According to ICE, ICE officers electronically request and retrieve criminal history information about an alien from the FBI’s National Crime Information Center (NCIC) database, which maintains a repository of federal and state criminal history information. ICE officers are also able to manually enter criminal history information in ICE’s data system if they discover additional criminal history information that was not available in NCIC. ICE officers may also check for criminal convictions committed outside the United States, on a case-by-case basis. Most of the ICE data we reviewed indicated criminal or non-criminal history, where criminal included convictions, and non-criminal included both pending criminal charges and other immigration violations. Therefore, wherever we referred to criminality, we used ICE’s determination of criminality—criminal or non-criminal—for our analysis. To determine the number of arrests by arresting program, we analyzed IIDS individual-level arrests data to determine the number of arrests at-large in the communities by ICE’s fugitive operations teams and those resulting from an incarceration in federal, state, and local prisons and jails through the Criminal Alien Program. To determine the number of arrests by ERO area of responsibility, we analyzed IIDS individual-level arrests data for calendar years 2015 through 2018. We also used these data to calculate the proportion of arrests of convicted criminals by ERO area of responsibility. We compared the number of arrests across the 24 ERO areas of responsibility to examine the differences in enforcement actions between the years the Priority Enforcement Program were in effect (2015-2016) and the years immediately following implementation of the DHS memo (2017-2018). We excluded from our analysis arrest records that had a missing or unknown area of responsibility. We also analyzed IIDS individual-level arrest data to determine the total number of arrests of juveniles during calendar years 2015 through 2018. Because aliens may have multiple arrests, we used alien number and arrest date to identify the unique number of arrests rather than the number of unique aliens who were arrested. We excluded from our analysis arrest records that had a missing alien number, an invalid alien number—i.e., that included all zeroes or had duplicate alien number and arrest date combinations. We used these data to determine the total number of arrests of juveniles by age and gender. Detentions. We analyzed individual-level detention data from IIDS to determine the total number of ERO detentions during calendar years 2015 through 2018. We examined multiple data fields from the individual-level detention data, including alien file number, person id, family name, given name, gender, country of citizenship, arresting agency, criminality, detention facility, book-in date, book-out date, release reason, and length of stay, among other variables. Because aliens may have multiple detentions, we used alien number and initial book-in date fields—i.e., the first date the individual is taken into ICE custody—to identify the unique number of detentions rather than the number of unique aliens who were detained. We excluded from our analysis arrest records that had a missing alien number or had an invalid alien number—i.e., that included all zeroes. We analyzed these data to determine total numbers of detentions by gender, country of citizenship, arresting agency, and criminality. To determine the number of detentions by gender, we analyzed IIDS individual-level detention data. We also analyzed these data to determine the number of detentions by arresting agency—ICE or U.S. Customs and Border Protection (CBP)—and criminality for each gender. We included all detentions resulting from both ICE and CBP arrests because ICE is responsible for detaining certain aliens apprehended by CBP at or between ports of entry. To conduct our analysis, we used ICE’s determination of criminality— criminal or non-criminal—which ICE determines by conducting electronic criminal history checks, as previously discussed. To determine the number of detentions by country of citizenship, we analyzed IIDS individual-level detention data. ICE obtains country of citizenship data from arrest reports, which may be based on documentation or self-reported. To determine the number of detentions by arresting agency, we analyzed IIDS individual-level detention data for detentions resulting from ICE arrests and those resulting from CBP arrests at or between ports of entry. To determine the number of detentions by criminality, we analyzed IIDS individual-level detention data. We also examined the extent to which detentions varied by criminality and arresting agency. To conduct our analysis, we used ICE’s determination of criminality— criminal or non-criminal—which ICE determines by conducting electronic criminal history checks, as previously discussed. Removals. We analyzed individual-level removal data from IIDS to determine the total number of ERO removals during calendar years 2015 through 2018. We examined multiple data fields from the individual-level removal data, including alien file number, family name, given name, gender, country of citizenship, criminality, arresting agency, and removal date, among other variables. Because aliens may have multiple removals, we used alien number and removal date to identify the unique number of removals rather than the number of unique aliens. We excluded from our analysis removal records that had a missing alien number, an invalid alien number—i.e., that included all zeroes, or had duplicate alien number and removal date combinations, or records that indicated test in the name fields. We analyzed these data to determine total numbers of removals by gender, country of citizenship, arresting agency, and criminality. To determine the number of removals by gender, we analyzed IIDS individual-level removal data. We also analyzed these data to determine the number of removals by arresting agency and criminality for each gender. To conduct our analysis, we used ICE’s determination of criminality—criminal or non-criminal—which ICE determines by conducting electronic criminal history checks, as previously discussed. To determine the number of removals by country of citizenship, we analyzed IIDS individual-level data. ERO obtains country of citizenship data from arrest reports, which may be based on documentation or self-reported. To determine the number of removals by arresting agency, we analyzed IIDS individual-level removal data for removals resulting from ERO arrests and those resulting from CBP arrests at or between ports of entry. To determine the number of removals by criminality, we analyzed IIDS individual-level removal data. To conduct our analysis, we used ICE’s determination of criminality—criminal or non-criminal— which ICE determines by conducting electronic criminal history checks, as previously discussed. We determined that the data used in each of our analyses were sufficiently reliable for the purposes of this report by analyzing available documentation, such as related data dictionaries; interviewing ICE officials knowledgeable about the data; conducting electronic tests to identify missing data, anomalies, or erroneous values; and following up with officials, as appropriate. We also analyzed arrest data from Homeland Security Investigations (HSI) worksite enforcement to determine the total number of criminal and administrative arrests conducted by HSI worksite enforcement between January 2015 and December 2018. We were unable to use these data for the purposes of reporting the total number of arrests by HSI worksite enforcement for each calendar year. Specifically, we identified combined arrest, charge, and conviction dates in the same field, among other issues, which limited our ability to identify the number of aliens arrested by HSI as a result of worksite enforcement operations each year. To address our second question, we reviewed a master list of ICE policies and interviewed policy officials to identify policies related to individuals with special vulnerabilities. Based on this review as well as input from nongovernmental organizations (NGOs) that serve or represent various populations, we selected eight populations including aliens who are: lesbian, gay, bisexual, transgender, and intersex (LGBTI), individuals with disabilities, juveniles, parents or legal guardians of minors, pregnant, individuals with mental disorders, women who are nursing, or individuals who are elderly. To identify the changes ICE made to align these policies with the 2017 DHS memo, we reviewed specific provisions in the executive order and implementing memoranda. We then analyzed existing policies as well as policies that ICE revised or rescinded to align with the 2017 DHS memo, including policies related to prosecutorial discretion and selected populations. We conducted interviews with officials from ICE headquarters offices, including the Office of the Principal Legal Advisor, Office of Policy, Homeland Security Investigations, as well as program officials within ERO, including Domestic Operations, Fugitive Operations, and Custody Management Divisions. We met with six national organizations that serve or represent immigrants as well as six state or regional organizations that serve or represent immigrants in the locations we visited to obtain their perspectives on how the policies affected the individuals they represent. The perspectives of NGOs are not generalizable and my not be indicative of care provided at all detention facilities. We selected these NGOs to reflect a range of types of populations served or represented as well as based on their proximity to ICE areas of responsibility we visited, see table 8 for more information on the organizations we interviewed. We conducted site visits to six selected ICE ERO areas of responsibility (Atlanta, Dallas, Los Angeles, San Diego, St. Paul, and Washington, D.C.) and interviewed ICE officials to obtain their perspectives on the policy revisions. We selected these locations based on the prevalence of arrests in fiscal year 2017, percent changes in arrests from fiscal year 2016 to 2017, and geographical dispersion. Specifically, we identified locations that had the highest arrest numbers in fiscal year 2017 or the largest percentage increases in arrests from fiscal years 2016 to 2017, and then selected locations that provided wide geographical representation. In each location we met with ERO liaisons and officers responsible for monitoring and implementing the provisions of policies for certain selected populations, as well as detention and deportation officers and supervisors who oversee the detention and removal of aliens, including those with special vulnerabilities. We also met with ICE medical staff in areas of responsibility with this position. In one area of responsibility, we limited our visit to a detention facility and met with the staff at that facility due to its proximity to another area of responsibility we visited. The information obtained from these site visits is not generalizable and may not be indicative of care provided to all populations at all detention facilities, but provided insights into how selected ICE areas of responsibility conduct enforcement activities and implement immigration enforcement policies. To address our third question, we reviewed multiple data sources that ICE uses to track information on certain aliens with special vulnerabilities in detention and matched these data with IIDS individual-level detention data to determine what ICE data show about detentions of selected populations between January 2015 and December 2018. To conduct our analysis, we first excluded records that contained missing alien numbers or alien numbers that were all zeroes. Then, we matched each data source to the IIDS detention data using alien number and excluded additional records we were unable to match. Because aliens may have multiple detentions, we compared the admission or book-in date from each data source with the book-in dates from the IIDS detention data, and excluded additional records with dates beyond 30 days apart. We analyzed this information to determine the total number of detentions for six of the eight selected populations (aliens who are: transgender, individuals with disabilities, pregnant, individuals with mental disorders, nursing, and elderly); and the number of detentions resulting from ICE versus CBP arrests; as well as detentions by criminality and the length of detention for each of these six populations. We excluded juveniles from our analysis because ERO is generally not responsible for detaining juveniles. To determine the extent to which ICE maintains data on detained parents or legal guardians of minors, we reviewed ICE policies pertaining to detained parents, including those that set forth requirements for tracking detained parents or legal guardians of U.S. citizens and legal permanent resident minors. We also interviewed ERO officials about ICE’s data collection processes and any limitations with the data it collects and maintains. We assessed ICE’s efforts to track this population against agency policy. To conduct our analysis of criminality for each population, we used ICE’s determination of criminality—criminal or non-criminal—which ICE determines by conducting electronic criminal history checks, as previously discussed. We also analyzed IIDS data on criminal charges for detentions of aliens that resulted from ICE arrests to determine the type of charges (e.g., immigration-related or other criminal charges) associated with these detentions. To conduct our analysis on length of detention, we compared initial book-in date with the most recent book-out date to calculate the total days in detention for each of our selected populations. Transgender Individuals: We matched ERO records for transgender detainees from calendar years 2016 through 2018 with IIDS individual-level detention data to determine the total number of detentions of transgender individuals, as well as the number of detentions by arresting agency, criminality, and length of detention. We excluded 4 of the unique transgender detainee records for 2016, 33 for 2017 and 27 for 2018. These records were excluded because we were unable to match these records to the IIDS individual level- detention data using alien number and book-in date combinations. According to ICE officials, this may be due to data entry errors. Our analysis is based on those records we were able to match: 228 for 2016, 241 for 2017, and 277 for 2018. ICE also recorded 55 transgender detainees in 2015; however, we excluded these records from our analysis since ICE did not collect complete data on this population in 2015. For the LGBTI population, ICE only collects and maintains data on transgender individuals in detention. Therefore, we were only able to analyze data for this subset of the LGBTI population. Individuals with Disabilities: We matched ERO records for individuals with communication and mobility impairments in ERO custody during calendar years 2017 and 2018 with IIDS individual- level detention data to determine the total number of detentions of these individuals, as well as the number of detentions by arresting agency, criminality, and length of detention. We excluded 5 of the unique detainee records for 2017, and 1 for 2018 because we were unable to match these records to the IIDS individual level-detention data using alien number and book-in date combinations. According to ICE officials, this may be due to data entry errors. Our analysis is based on those records we were able to match: 424 for 2017, and 516 for 2018. When ICE began collecting these data, it included aliens who were placed in detention prior to January 2017. We excluded 99 records for this reason from our analysis since ICE did not collect complete data on this population prior to January 2017. Pregnant Women: We matched ICE Health Service Corps (IHSC) records for pregnant women in ERO custody during calendar years 2016 through 2018 with IIDS individual-level detention data to determine the total number of detentions of pregnant women, as well as the number of detentions by arresting agency, criminality, and length of detention. We excluded 60 of the unique pregnant detainee records for 2016, 20 for 2017 and 32 for 2018 because we were unable to match these records to the IIDS individual-level detention data using alien number and book-in date combinations. According to ICE officials, this may be due to data entry errors. Our analysis is based on those records we were able to match: 1,377 for 2016, 1,150 for 2017, and 2,094 for 2018. ICE also recorded 675 pregnant detainees in 2015; however, we excluded these records from our analysis since ICE did not collect complete data on this population in 2015. Elderly Individuals: We analyzed data records in IIDS for elderly individuals (those 65 years or older at the time of initial book-in) in ERO custody during calendar years 2015 through 2018 to determine the total number of detentions of elderly individuals, as well as the number of detentions by arresting agency, criminality, and length of detention. According to ERO, the agency does not maintain separate data records for elderly individuals in ERO custody; however, ERO officials were able to identify these detainees by calculating their age at the time they were detained. We excluded 4 of the unique elderly detainee records for 2015, 3 for 2016 and 4 for 2018 because we were unable to match these records to the IIDS individual-level detention data using alien number and book-in date combinations. According to ICE officials, this may be due to data entry errors. Our analysis is based on those records we were able to match: 863 for 2015, 736 for 2016, 763 for 2017, and 1,132 for 2018. Individuals with Mental Disorders and Nursing Women: We matched IHSC records for individuals with mental disorders and nursing women detained at IHSC-staffed facilities during calendar years 2015 through 2018 with IIDS individual-level detention data to determine the total number of detentions of each of these populations, as well as the number of detentions by arresting agency, criminality, and length of detention. Because ICE did not maintain data on individuals with mental disorders or nursing women detained at the over 200 non-IHSC staffed facilities, our findings for these two populations are not generalizable, but provided valuable insights into these detentions. We excluded 207 of the unique detainee with mental disorders records for 2016, 850 for 2017, and 1,233 for 2018 because we were unable to match these records with the IIDS individual-level detention data using alien number and book-in date combinations. Our analysis is based on the unique detainee with mental disorders records we were able to match: 8,138 for 2015, 9,466 for 2016, 8,643 for 2017, and 8,501 for 2018. Similarly, we excluded 2 of the unique nursing detainee records for 2015, 3 for 2017 and 5 for 2018 for the same reason. Our analysis is based on the unique nursing detainee records we were able to match: 157 for 2015, 399 for 2016, 564 for 2017, and 381 for 2018. According to ICE officials, this may be due to data entry errors. We assessed the reliability of the data used in each of our analyses by analyzing available documentation, such as related data dictionaries; interviewing ERO officials knowledgeable about the data; conducting electronic tests to identify missing data, anomalies, or erroneous values; and following up with officials, as appropriate. We determined the data were sufficiently reliable for depicting general trends in detentions for the selected populations. We conducted this performance audit from November 2017 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The number of Enforcement and Removal Operations (ERO) administrative arrests (arrests) by gender, country of citizenship, ICE enforcement program, criminality, and area of responsibility from calendar years 2015 through 2018. The number of detentions by gender, country of citizenship, arresting agency, and criminality from calendar years 2015 through 2018. The number of removals by gender, country of citizenship, arresting agency, and criminality from calendar years 2015 through 2018. We analyzed individual-level Immigration and Customs Enforcement (ICE) data to identify ERO arrests, detentions, and removals during calendar years 2015 through 2018. The Number of Arrests Varied during the Period, Increasing Overall. The number of ERO arrests varied from calendar years 2015 through 2018, and increased more than 30 percent overall for the 4-year period (from 112,870 arrests in 2015 to 151,497 arrests in 2018). During the two years Priority Enforcement Program (PEP) was in effect, the number of ERO arrests varied little, decreasing 5 percent from 2015 to 2016. Following issuance of the 2017 DHS memo, ERO arrests increased 41 percent from 2016 to 2017, and stayed relatively the same from 2017 to 2018. Arrests by Gender. Each year from calendar years 2015 through 2018, arrests of males accounted for the majority of ERO arrests (ranging from 92 to 93 percent), as shown in figure 7. Arrests by Country of Citizenship. Each year from 2015 through 2018, ERO arrests of citizens of Mexico, Guatemala, El Salvador, and Honduras collectively accounted for about 86 percent of all ERO arrests, with individuals from Mexico accounting for the majority (ranging from 59 to 65 percent), as shown in figure 8. All other individual countries collectively accounted for about 14 to 15 percent of total arrests each year. Arrests by ICE Enforcement Program. Arrests of individuals from federal, state and local prisons and jails, through the Criminal Alien Program, accounted for the majority (ranging from 72 to 76 percent) of ERO arrests each calendar year from 2015 through 2018, as shown in figure 9. Arrests of individuals at-large through Fugitive Operations (ranging from 17 to 19 percent) and other programs accounted for the balance of the arrests each year. Criminal Alien Program arrests also accounted for most of the increase in ERO arrests in calendar years 2017 and 2018 (see figure 9). Arrests by Criminality. As shown in figure 10, the number and proportion of ERO arrests of non-criminals aliens increased each year from calendar years 2015 through 2018. For the purposes of this report and our presentation of ICE data, we refer to potentially removable aliens without criminal convictions known to ICE as “non-criminals” and aliens with criminal convictions known to ICE as “convicted criminals.” Specifically, the arrests of non-criminals increased from 13,494 (12 percent of total arrests) in 2015 to 51,513 (34 percent of total arrests) in 2018. According to ERO officials, arrests of non-criminals include individuals who have been charged with but not convicted of a crime as well as those with no prior criminal history. The number of ERO arrests of convicted criminals stayed relatively stable from calendar years 2015 to 2018, ranging between about 91,000 and 107,000. Each of these years, arrests of convicted criminals comprised the majority of total arrests, but decreased from 88 percent in 2015 to 66 percent in 2018. Most arrests of convicted criminals resulted from the Criminal Alien Program (ranging from 76 to 80 percent), followed by Fugitive Operations (ranging from 15 to 19 percent). Arrests by Areas of Responsibility. The number of ERO arrests increased in all ERO areas of responsibility when comparing calendar years 2015 and 2016, when PEP was in effect, to calendar years 2017 and 2018, following implementation of the 2017 DHS memo. These increases ranged from less than 1 percent increase in the Los Angeles area of responsibility to a 99 percent increase in the Miami area of responsibility. Arrests of convicted criminals accounted for the majority of total arrests in all areas of responsibility. However, the proportion of arrests of convicted criminals to total arrests decreased in all areas of responsibility from 2015 and 2016 to 2017 and 2018. This decrease is partially due to the increase in the number of ERO arrests of non- criminals in all areas of responsibility during these years. Table 9 presents total numbers of ERO arrests for each of ERO’s 24 areas responsibility nationwide. It also presents the percentage of arrests of convicted criminals by area of responsibility for calendar years 2015 and 2016 combined and calendar years 2017 and 2018 combined. The Number of Detentions Varied, Increasing Overall. The number of ERO detentions varied from calendar years 2015 through 2018, and increased more than 30 percent overall for the 4-year period (from 324,320 detentions in 2015 to 438,258 detentions in 2018). ERO detention data include detentions resulting from both ICE and CBP arrests. During the two years PEP was in effect, the number of ERO detentions increased 13 percent, from 324,320 in 2015 to 366,740 in 2016. Following issuance of the 2017 DHS memo, ERO detentions decreased 15 percent from 2016 to 2017 (from 366,740 to 310,309 detentions), and increased 41 percent from 2017 to 2018 (to 438,258 detentions). Detentions by Gender. Each year from calendar years 2015 through 2018, detentions of males accounted for the majority of ERO detentions (ranging from 74 to 81 percent), as shown in figure 11. Detentions by Country of Citizenship. Each year from 2015 through 2018, ERO detentions of citizens of Mexico, Guatemala, El Salvador, and Honduras collectively accounted for the most detentions (ranging from 84 to 89 percent). All other individual countries collectively accounted for 11 to 16 percent of total detentions each year, as shown in Figure 12. Detentions by Arresting Agency. Detentions resulting from CBP arrests at or between ports of entry accounted for the majority of ERO detentions each year from calendar years 2015 through 2018 (ranging from 52 to 71 percent). Detentions resulting from CBP arrests also accounted for most of the variation in detentions from year to year, as shown in figure 13. Detentions resulting from ICE arrests varied little from 2015 to 2016, increased in 2017, and then varied little from 2017 to 2018. Detentions by Criminality. As shown in figure 14, the number of ERO detentions of non-criminals varied, but increased overall from calendar years 2015 to 2018. These detentions accounted for the majority of total ERO detentions each year (ranging from 53 to 64 percent). The variation in the number of detentions of non-criminals was partially due to fluctuations in detentions that resulted from CBP arrests. The number of ERO detentions of convicted criminals stayed relatively stable from 2015 to 2018, and accounted for the minority of total ERO detentions (ranging from 36 to 47 percent). The majority of these detentions resulted from ICE arrests (ranging from 64 to 76 percent) rather than CBP arrests. The Number of Removals Varied, Increasing Overall. The number of ERO removals varied from calendar years 2015 through 2018, and increased 13 percent overall for the 4-year period (from 231,559 removals in 2015 to 261,523 removals in 2018). ERO removal data include removals resulting from both ICE and CBP arrests. During the two years PEP was in effect, the number of ERO removals varied little, increasing 6 percent from 2015 to 2016. Following issuance of the 2017 DHS memo, ERO removals decreased 12 percent in 2017, and increased 21 percent from 2017 to 2018. Removals by Gender. Removals of male aliens accounted for most of ERO removals (about 90 percent) each year from calendar years 2015 through 2018, as shown in figure 15. Removals by Country of Citizenship. In addition, from calendar years 2015 through 2018, ERO removals of citizens of Mexico, Guatemala, El Salvador, and Honduras collectively accounted for most of the removals each year (ranging from 90 to 94 percent). Citizens of all other countries collectively accounted for 6 to 10 percent of total removals each year, as shown in figure 16. Removals by Arresting Agency. Each year, removals resulting from CBP arrests at or between ports of entry accounted for the majority of total ERO removals (ranging from 60 to 74 percent). ERO removals resulting from CBP arrests also accounted for most of the variation in total removals from year to year, as shown in figure 17. Removals by Criminality. The number and proportion of ERO removals of non-criminals varied, but increased overall, from calendar years 2015 through 2018, as shown in figure 18. Specifically, removals of non- criminals increased from 40 percent of total removals in 2015 to 43 percent of total removals in 2018. Most removals of non-criminals resulted from CBP arrests (ranging from 80 to 95 percent), rather than ICE arrests. ERO removals of convicted criminals varied, increasing overall, from calendar years 2015 to 2018, and accounted for the majority of total ERO removals each year (ranging from 55 to 60 percent). Removals of convicted criminals resulted from CBP and ICE arrests at approximately equal levels. This appendix presents the overall number of Enforcement and Removal Operations (ERO) administrative arrests (arrests), detentions, and removals of males from calendar years 2015 through 2018, including the number of arrests by criminality and the number of detentions and removal by criminality and arresting agency. We analyzed individual- level Immigration and Customs Enforcement (ICE) data to identify ERO arrests, detentions, and removals of males during calendar years 2015 through 2018. The Number of Arrests of Males Generally Increased. The number of ERO arrests of males varied from calendar years 2015 through 2018 but generally increased by 32 percent across the period, as shown in figure 19. During the two years the Priority Enforcement Program (PEP) was in effect, between calendar years 2015 and 2016, the number of ERO arrests remained stable, decreasing by about 5 percent in that period. The following year, after the issuance of the 2017 DHS memo in February 2017, ERO arrests increased by about 40 percent from calendar years 2016 to 2017, and decreased by less than 1 percent in calendar year 2018. Arrests of Males by Criminality. During the same time, the proportion of ERO arrests of convicted criminal males decreased each year from 90 percent of total arrests of males in calendar year 2015 to 69 percent in calendar year 2018, as shown in figure 19. For the purposes of this report and our presentation of ICE data, we refer to potentially removable aliens without criminal convictions known to ICE as “non-criminals” and aliens with criminal convictions known to ICE as “convicted criminals.” Conversely, the proportion of ERO arrests of non-criminal males increased each year, from 10 percent of total arrests of males in calendar year 2015 to 31 percent of total arrests in calendar year 2018. According to officials, arrests of non-criminals include individuals who have been charged with but not convicted of a crime as well as those with no prior criminal history. For the purposes of this report and our presentation of ICE data, we refer to potentially removable aliens without criminal convictions known to ICE as “non-criminals” and aliens with criminal convictions known to ICE as “convicted criminals.” According to ICE officials, administrative arrests of non-criminals include individuals who have been charged with but not convicted of a crime as well as those with no prior criminal history. According to ICE, ICE officers electronically request and retrieve criminal history information about an alien from the FBI’s National Crime Information Center database, which maintains a repository of federal and state criminal history information, and other sources. We used ICE’s determination of criminality for our analysis. Detentions of Males Increased Overall. The number of ERO detentions varied from calendar years 2015 through 2018, but increased overall by 32 percent over the period, as shown in figure 20. ERO detention data include detentions resulting from both ICE and U.S. Customs and Border Protection (CBP) arrests. During the two years PEP was in effect, the number of ERO detentions of males increased by more than 8 percent from calendar years 2015 to 2016. Following the issuance of the 2017 DHS memo, the number of male detentions decreased by more than 8 percent in calendar year 2017, and increased again in calendar year 2018, by over 32 percent. Detentions of Males by Arresting Agency. Detention of males resulted from both ICE and CBP arrests from calendar years 2015 through 2018, as shown in figure 20. For all the years in this period, except calendar year 2017, detentions resulting from a CBP arrest at or between ports of entry account for the majority of the detentions of males (ranging from about 58 to 63 percent). In calendar year 2017, detentions resulting from ICE arrests accounted for about 56 percent of all male detentions. Detentions of Males by Criminality. During the same time, the number and proportion of ERO detentions of convicted criminal males varied, ranging from 45 to 57 percent of all detentions of males, as shown in figure 21. The majority of these detentions resulted from ICE arrests, ranging from 66 to 77 percent of all convicted criminal male detentions. The number of ERO detentions of non-criminal males also varied, ranging from 43 to 55 percent of all detentions of males. Detentions of non- criminal males primarily resulted from CBP arrests, which ranged from 69 to 93 percent of detentions of non-criminal males between calendar years 2015 and 2018. Removals of Males Increased Overall. The number of ERO removals of males varied from calendar years 2015 through 2018, but increased overall by 14 percent over the period, as shown in figure 22. ERO removal data include removals resulting from both ICE and CBP arrests. During PEP, which was in effect from calendar years 2015 and 2016, the number of ERO removals of males increased by about 6 percent. From calendar years 2016 to 2017, following the issuance of the 2017 DHS memo, the number of these removals decreased by more than 11 percent, then increased by more than 20 percent in calendar year 2018. Removals of Males by Arresting Agency. From calendar years 2015 to 2018, the majority of ERO removals of males resulted from CBP arrests at or in between ports of entry (ranging from 58 to 72 percent), as shown in figure 22. Removals of Males by Criminality. From calendar years 2015 through 2018, ERO removals of convicted criminal males accounted for the majority of removals each year, ranging from 58 to 63 percent of the total removal of males, as shown in figure 23. The removals of convicted criminal males were the result of both CBP and ICE arrests. For all the years in this period, except calendar year 2017, removals resulting from a CBP arrest account for the majority of the removals of convicted criminal males (ranging from about 52 to 56 percent). In calendar year 2017, removals resulting from ICE arrests accounted for about 56 percent of all removals of convicted criminal males. ERO removals of non-criminal males varied, increasing overall, from calendar years 2015 to 2018, and accounted for the minority of ERO removals of males each year (ranging from 37 to 42 percent). Most of the removals of non-criminal males were as a result of CBP arrests, ranging from 79 to 95 percent of all removals of non-criminal males. This appendix presents the overall number of Enforcement and Removal Operations (ERO) administrative arrests (arrests), detentions, and removals of females from calendar years 2015 through 2018, including the number of arrests by criminality and the number of detentions and removals by criminality and arresting agency. We analyzed individual- level Immigration and Customs Enforcement (ICE) data to identify ERO arrests, detentions, and removals of females during calendar years 2015 through 2018. The Number of Arrests of Females Generally Increased. The number of ERO arrests of females generally increased more than 70 percent from calendar years 2015 through 2018, as shown in figure 24. Between 2015 and 2016, the two years the Priority Enforcement Program (PEP) was in effect, the number of ERO arrests remained stable, decreasing by less than 1 percent in that period. Following the issuance of the 2017 DHS memo, ERO arrests increased by 65 percent from calendar years 2016 to 2017, and increased by less than 5 percent in calendar year 2018. Arrests of Females by Criminality. During the same time, the proportion of arrests of non-criminal females increased each year from 43 percent in calendar year 2015 to 63 percent of total arrests of females in calendar year 2018. For the purposes of this report and our presentation of ICE data, we refer to potentially removable aliens without criminal convictions known to ICE as “non-criminals” and aliens with criminal convictions known to ICE as “convicted criminals.” According to officials, arrests of non-criminals include individuals who have been charged with but not convicted of a crime as well as those with no prior criminal history. Conversely, the proportion of ERO arrests of convicted criminal females decreased each year from 57 percent in calendar year 2015 to 37 percent in calendar year 2018, as shown in figure 24. Detentions of Females Increased Overall. The number of ERO detentions varied from calendar years 2015 through 2018, and increased more than 45 percent over the period, as shown in figure 25. ERO detention data include detentions resulting from both ICE and U.S. Customs and Border Protection (CBP) arrests. During the two years PEP was in effect, the number of ERO detentions of females increased by more than 28 percent from calendar years 2015 through 2016. Following the issuance of the DHS memo, the number of detentions decreased by about 36 percent in 2017, then increased by over 77 percent in calendar year 2018. Detentions of Females by Arresting Agency. Detentions of females resulting from CBP arrests at or between ports of entry accounted for most of the detentions of females each year from calendar years 2015 through 2018 (ranging from 84 to 94 percent), as shown in figure 25. Detentions of Females by Criminality. As shown in figure 26, the number of ERO detentions of non-criminal females varied, but increased overall from calendar years 2015 to 2018. These detentions accounted for most of the total ERO detentions of females each year (ranging from 87 to 92 percent). Most of the detention of non-criminal females resulted from CBP arrests (ranging from 91 to 98 percent) rather than ICE arrests. The number of ERO detentions of convicted criminal females stayed relatively stable from calendar years 2015 through 2018, and accounted for the minority of total ERO detentions (ranging from 8 to 13 percent). CBP and ICE arrests accounted for approximately the same number of detentions of convicted criminal females. Removals of Females Increased Overall. The number of ERO removals of females remained relatively stable from calendar years 2015 through 2018, but increased overall by 6 percent over the period, as shown in figure 27. ERO removal data include removals resulting from both ICE and CBP arrests. During the PEP, which lasted from calendar years 2015 and 2016, the number of ERO removals increased by more that 2 percent. From calendar years 2016 to 2017, following the issuance of the 2017 DHS memo, the number of ERO removals decreased by more than 14 percent, then increased by more than 20 percent in 2018. Removals of Females by Arresting Agency. Each calendar year, removals resulting from CBP arrests at or between ports of entry accounted for most of the ERO removals of females (ranging from 80 to 90 percent), as shown in figure 27. Removals of Females by Criminality. From calendar years 2015 through 2018, the majority of ERO removals were of non-criminal females (ranging from 66 to 72 percent), as shown in figure 28. Most removals of non-criminal females resulted from CBP arrests (ranging from 88 to 97 percent), rather than ICE arrests. ERO removals of convicted criminal females varied, increasing overall, from calendar years 2015 to 2018, and accounted for the minority of ERO removals of females each year (ranging from 28 to 34 percent). The majority removals of convicted criminal females also resulted from CBP arrests (ranging from 56 to 71 percent). This appendix presents the overall number of Enforcement and Removal Operations (ERO) administrative arrests (arrests) of juveniles—persons encountered by ERO who have not reached 18 years of age—as well as the number of juvenile arrests by age and gender. We analyzed individual-level Immigration and Customs Enforcement (ICE) data to identify the number of ERO arrests of juveniles during calendar years 2015 through 2018. The Number of Arrests of Juveniles Increased Overall. The number of ERO arrests of juveniles increased overall by 53 percent from calendar years 2015 through 2018, as shown in figure 29. During the two years the Priority Enforcement Program was in effect, ERO arrests of juveniles increased 47 percent (from 887 arrests in 2015 to 1,307 arrests in 2016). Following issuance of the 2017 DHS memo, ERO arrests of juveniles increased 76 percent in calendar year 2017 (2,294 arrests), and decreased 41 percent in calendar year 2018 (1,361 arrests). Arrests of Juveniles by Age. The proportion of arrests for juveniles of all age groups—ages 0 to 6, 7 to 12, and 13 to 17—varied between calendar years 2015 and 2018, as shown in figure 30. For instance, the proportion of arrests of juveniles ages 0 to 6 between calendar years 2015 and 2018, ranged from 31 to 43 percent of the total number of arrests of juveniles. The proportion of arrests of juveniles ages 7 to 12 ranged from 16 percent to 23 percent of total arrests of juveniles during this same period while arrests of juveniles ages 13 to 17, during the same period ranged from 34 percent to 50 percent of total arrests of juveniles. Arrests of Juveniles by Gender. Each calendar year from 2015 through 2018, arrests of male juveniles accounted for the majority of ERO arrests of juveniles (ranging from 57 to 66 percent), as shown in figure 31. This appendix presents the number of U.S. Immigration and Customs Enforcement (ICE) Enforcement and Removal Operations (ERO) administrative arrests by country of citizenship for calendar years 2015 through 2018. Each year from 2015 through 2018, ERO administratively arrested aliens from over 200 countries. This appendix presents the number of U.S. Immigration and Customs Enforcement (ICE) Enforcement and Removal Operations (ERO) detentions by country of citizenship for calendar years 2015 through 2018. Each year from 2015 through 2018, ERO detained aliens from over 200 countries. This appendix presents the number of U.S. Immigration and Customs Enforcement (ICE) Enforcement and Removal Operations (ERO) removals by country of citizenship for calendar years 2015 through 2018. Each year from 2015 through 2018, ERO removed aliens from almost 200 countries. This appendix presents the number and type of criminal charges of U.S. Immigration and Customs Enforcement (ICE) Enforcement and Removal Operations (ERO) detentions of selected populations (aliens who are: transgender, individuals with disabilities, pregnant, individuals with mental disorders, women who are nursing, or individuals who are elderly) resulting from ICE arrests. ICE administrative arrests of aliens for civil violations of U.S. immigration laws include arrests of both aliens with prior criminal convictions and those without prior criminal convictions. According to ICE, ICE officers electronically request and retrieve criminal history information about an alien from the FBI’s National Crime Information Center (NCIC) database, which maintains a repository of federal and state criminal history information. ICE officers are also able to manually enter criminal history information in ICE’s data system if they discover additional criminal history information that was not available in NCIC. ICE officers may also check for criminal convictions committed outside the United States, on a case by case basis. To identify which convictions or charges were immigration-related for these selected populations, we reviewed the criminal history information recorded in ICE’s data system by ICE officers. m ICE collected data to identify each of these populations beginning at different timeframes or subsets within the population, as shown below. For information on the number of detentions of selected populations resulting from ICE arrests by criminal charge type, see tables 13 through 18. This appendix presents the length of U.S. Immigrations and Customs Enforcement (ICE) Enforcement and Removal Operations detentions of selected populations—aliens who are: transgender, individuals with disabilities, pregnant, individuals with mental disorders, women who are nursing, or individuals who are elderly. Available ICE data varied for each of these populations because ICE began collecting these data at different time periods. In addition, the length of some detentions from a particular year may be undetermined because they were still ongoing at the time of our review (as of May 15, 2019). We present available data for each of the populations. Detentions of Transgender Individuals. Based on available records each year from 2016 through 2018, the majority of detentions of transgender individuals were 90 days or less (ranging from 62 to 70 percent), as shown in table 19. Detentions of Individuals with Disabilities. Based on available records in calendar years 2017 and 2018, the majority of detentions of individuals with disabilities were 90 days or less (56 and 65 percent, respectively), as shown in table 20. Detentions of Pregnant Women. From calendar years 2016 through 2018, the majority of detentions of pregnant women were 15 days or less (ranging from 71 to 93 percent), as shown in table 21. Detentions of Individuals with Mental Disorders at ICE Health Service Corps-staffed facilities. Based on available records each year from calendar years 2015 through 2018, the majority of detentions of individuals with mental disorders at ICE Health Service Corps (IHSC)- staffed facilities were 90 days or less (ranging from 59 to 71 percent), as shown in table 22. Detentions of Nursing Women at IHSC-staffed facilities. From calendar years 2015 through 2018, most detentions of nursing women at IHSC-staffed facilities were 30 days or less (ranging from 77 to 97 percent), as shown in table 23. Detentions of Elderly Individuals. Based on available records each year from calendar years 2015 through 2018, most of the detentions of elderly individuals were 90 days or less (ranging from 80 to 84 percent), with the majority being of 30 days or less, as shown in table 24. In addition to the contact name above, Meg Ullengren (Assistant Director), Carissa Bryant (Analyst-in-Charge), Hiwotte Amare, Michele Fejfar, Eric Hauswirth, Dainia Lawes, Marycella Mierez, Heidi Nielson, and Claire Peachey made key contributions to this report.", "summary": "In January 2017, the President issued Executive Order 13768 that instructs the Department of Homeland Security (DHS) to enforce U.S. immigration law against all removable individuals. In February 2017, the Secretary of DHS issued a memorandum (2017 DHS memo) establishing policy and providing guidance related to the Executive Order. Within DHS, ICE is responsible for providing safe confinement for detained aliens, including certain vulnerable populations. GAO was asked to review ICE immigration enforcement priorities, including those for vulnerable populations. This report examines (1) ICE data on arrests, detentions, and removals from calendar years 2015 through 2018; (2) the policies in effect for selected populations and any changes ICE made to align these policies with the 2017 DHS memo; and (3) the extent to which ICE collects data on selected populations and what those data show. GAO analyzed ICE data on arrests, detentions, and removals from calendars years 2015 through 2018; reviewed policies and documents on eight populations GAO selected based on ICE policies and input from organizations that represent various vulnerable populations; and interviewed agency officials. The numbers of administrative arrests (arrests), detentions, and removals of aliens (people who are not citizens or nationals of the United States) by U.S. Immigration and Customs Enforcement (ICE) varied during calendar years 2015 through 2018, and increased overall for the period. Males, aliens from four countries—Mexico, Guatemala, El Salvador, and Honduras—and convicted criminals accounted for the majority of ICE arrests and removals. The majority of detentions were made up of males, aliens from the same four countries, and non-criminals. ICE has policies related to six of the selected populations GAO examined, including aliens who are: transgender, individuals with disabilities, individuals with mental disorders, juveniles, parents of minors, and pregnant. These policies provide guidance on identifying, detaining, caring for, and removing aliens in these populations. After issuance of the 2017 DHS memo, ICE removed language from its existing policies for individuals who are pregnant and parents of minors that it determined to be inconsistent with 2017 DHS memo. Available ICE detention data show that detentions of transgender and pregnant individuals increased from calendar years 2016 to 2018 and detentions of individuals with disabilities increased from 2017 to 2018. Detentions at facilities staffed by ICE medical personnel of individuals with mental disorders and women who are nursing varied from calendar years 2015 to 2018. We found that ICE does not collect or maintain readily available data on detained parents or legal guardians of U.S. citizen or legal permanent resident minors, as required by ICE policy. Without such information, ICE headquarters officials cannot ensure that ICE officers are collecting and entering this information into the system as required by policy. ICE officials said they have considered actions to identify this population, but are no longer considering these actions as of October 2019. Maintaining these data in a readily available format could help ensure that ICE personnel identify, evaluate, and share information on this population. GAO is recommending that ICE collect readily available data on detained parents or guardians of U.S. citizen and legal permanent resident minors. DHS did not concur with the recommendation. GAO continues to believe this recommendation is valid as discussed in the report.", "document_type": "gao"}
{"report": "According to EPA, risk assessments provide information on potential health or ecological risks. Information from risk assessments, in combination with other information, provides the basis for risk management actions, as illustrated in the risk assessment model in figure 1. EPA may also consider scientific and economic factors; court decisions; and social, technological, and political factors during the risk management process. A number of program and regional offices at EPA prepare chemical risk assessments. These risk assessments in turn provide the foundation for EPA’s risk management decisions, such as whether EPA should establish air and water quality standards to protect the public from exposure to toxic chemicals. To prepare these risk assessments, some EPA program and regional offices often rely in part on chemical assessments that the IRIS Program, as part of ORD, prepares. IRIS assessments generally include the first two steps of the risk assessment process seen in green in figure 1: (1) hazard identification and (2) dose-response assessment. Hazard identification identifies credible health hazards associated with exposures to a chemical; dose-response assessment characterizes the quantitative relationship between chemical exposure and each credible health hazard. The program derives toxicity values through this quantitative relationship. These toxicity values are combined with exposure assessments (produced by other offices within EPA) to produce a risk assessment. OCSPP, which oversees TSCA implementation, also prepares chemical risk assessments, though it does not generally rely on IRIS toxicity values. OCSPP’s risk evaluations provide the foundation for a risk management action under TSCA if a use is found to present unreasonable risk of injury to human health or the environment. Risk management actions under TSCA can include but are not limited to restrictions or bans on a chemical or a condition of use, limitations on processing or manufacture, or changes to product labeling. Figure 2 shows EPA’s organizational structure, including the program and regional offices that prepare chemical risk assessments. EPA created the IRIS Program in 1985 to help develop consensus opinions within EPA about the health effects from lifetime exposure to chemicals. The IRIS database of chemical assessments contains EPA’s scientific positions on the potential human health effects that may result from exposure to various chemicals in the environment, and as of November 2018, it included information on 510 chemicals. Based on our body of work on the IRIS Program, the program’s importance has increased over time as EPA program offices and regions have increasingly relied on IRIS chemical assessments in making environmental protection and risk management decisions. In addition, state and local environmental programs, as well as some international regulatory bodies, rely on IRIS chemical assessments in managing their environmental protection programs. The IRIS Program uses a seven-step process to produce chemical assessments, as shown in figure 3. The first step in the assessment development process is developing a draft assessment. This begins with IRIS Program staff determining the scope and initial problem formulation of an assessment in consultation with EPA program and regional offices. This information is documented in an IRIS Assessment Plan and released for agency and public comment. After obtaining feedback on the IRIS Assessment Plan, IRIS Program staff prepare an assessment protocol for public comment that describes the methods that IRIS will use to conduct the assessment. During Step 1 (Scoping and Problem Formulation) IRIS Program staff conduct preliminary searches of scientific literature and screen relevant studies to understand the extent and nature of the available evidence. This informs the level of effort, identifies areas of scientific complexity, and helps the IRIS Program estimate time frames for conducting the assessment. The program staff select and extract relevant data and analyze and integrate the evidence into the draft assessment. The final step in preparing the draft assessment is deriving chemical toxicity values. After these draft development steps (step 1 in fig. 3), the draft assessment goes through internal agency and interagency review, public comment, and peer review, as shown in steps 2 through 4 in figure 3. After making revisions to address comments received (step 5), the assessment goes through another round of internal and interagency review (steps 6a and 6b), and then the program finalizes and posts the assessment to the IRIS website. According to IRIS officials, in order to prepare IRIS assessments, a group of staff with specialized skills are required. On any given assessment, approximately a dozen staff drawn from several different backgrounds (e.g., toxicologists and epidemiologists) work on each assessment. While some of the assessment preparation—that is, setting up database searches and performing initial search screenings—can be performed by any staff, other parts of assessment development require that the staff have specific expertise. The IRIS assessment development process—and the associated implementation of systematic review processes—has continued to evolve since 2011, primarily as a result of NAS recommendations made in two reports issued in 2011 and 2014. The 2011 report was a NAS peer review of the IRIS assessment of formaldehyde. In that report, NAS recommended several changes to the formaldehyde assessment and also offered recommendations more generally about the IRIS assessment development process. For example, NAS recommended methods for identifying evidence to be included in IRIS assessments; assessing and weighting that evidence in preparing the assessment; selecting studies that are used for calculating toxicity; and documenting how those toxicity calculations are carried out. A House appropriations committee report for fiscal year 2015 directed EPA to implement the 2011 report’s recommendations and NAS to review the changes that EPA was making (or proposing to make). In its review, NAS made additional recommendations to the program. In April 2018, NAS released a report on the IRIS Program’s responses to the 2014 recommendations. IRIS assessments are one potential source of information for risk assessors in OCSPP who conduct risk evaluations informing risk management activities under TSCA. The purpose of risk evaluation is to determine whether a chemical substance presents an unreasonable risk to human health or the environment. TSCA authorizes EPA to evaluate and, if appropriate, regulate existing chemicals and new chemicals. TSCA generally covers chemicals manufactured, imported, processed, distributed in commerce, used, or disposed of in the United States. If EPA finds that any of these activities with respect to a specific chemical presents an unreasonable risk of injury to health or the environment, EPA must issue regulations that can, among other things, restrict or prohibit these activities. TSCA also specifies the information obtained from chemical companies that EPA must publicly disclose and the circumstances under which chemical companies can claim certain information, such as data about chemical processes, as confidential business information. EPA’s OPPT within the Office of Chemical Safety and Pollution Prevention manages risk assessment and risk management strategies for chemicals under TSCA. According to EPA officials, OPPT’s Risk Assessment Division uses a number of different streams of information—including IRIS assessments—to prepare chemical risk assessments in order to make determinations about the safety of chemicals, and the Chemical Control Division uses those risk assessments to prepare risk management plans for chemicals. Prior to 2016, environmental and industry stakeholder organizations expressed concern that public confidence was decreasing regarding the safe use of chemicals in commerce and that federal oversight should be strengthened. For example, according to an American Bar Association new TSCA guide, the desire for reform was driven by a proliferation of state-based chemical initiatives threatening to disturb interstate commercial transactions and by a continuing erosion of public confidence in TSCA’s ability to protect human health and the environment from unreasonable risks presented by chemicals. In addition, according to a statement from the Environmental Defense Fund, federal oversight could not keep pace with science or rapidly expanding production and use of chemicals. In June 2016, Congress passed the Lautenberg Act, which amended TSCA in several ways. Table 1 summarizes some of the major changes in the act, along with the purpose and application of TSCA’s major sections. Since passage of the Lautenberg Act, several areas of disagreement have arisen among stakeholders regarding the implementation of various aspects of the act. One of the main points of ongoing discussion centers on what conditions of use EPA must consider in a chemical risk evaluation under TSCA. EPA and some stakeholders also disagree on other areas such as the methodologies EPA uses in its systematic review approach, the extent to which companies’ data are exempt from disclosure, and the extent to which the fees rule accurately reflects EPA’s costs for implementing TSCA. Some of these issues have resulted in litigation. The IRIS Program has addressed many process challenges, such as by making changes to address the length of time it takes to develop chemical assessments and to increase transparency, but EPA has not made progress toward producing chemical assessments. However, the release of documents related to IRIS assessments was delayed for nearly 6 months because EPA leadership instructed the IRIS Program not to release any assessment documentation pending the outcome of EPA leadership deliberations concerning IRIS Program priorities. The IRIS Program in 2011 began making changes to address identified challenges, particularly the length of time the program took to produce assessments and the level of transparency in how the program prepared assessments. The program has made some progress since the beginning of 2017 toward producing assessments and is ready to release assessment-related documents. These changes were made in response to program implementation challenges identified by governmental, industry, academic, and non-governmental stakeholders in recent years. For example, in its 2011 report, NAS identified timeliness and transparency as issues. In our review of the 2011 and 2014 NAS reports and other documentation as well as our interviews with IRIS officials and leadership and officials in program and regional offices that use IRIS assessments, we identified the key actions the IRIS Program has taken to address lack of timeliness in producing assessments and lack of transparency in how it produces assessments. Developing IRIS assessments has historically been a lengthy process. Because of the rigor of the IRIS process and the amount of literature that program staff must search and consider, producing an assessment typically takes several years, as we found in December 2011. Program and regional offices that use IRIS assessments understand this, and officials from several program and regional offices told us that despite the length of time it takes for the IRIS Program to complete its assessments, they prefer these assessments as sources of information over other agencies’ toxicity assessments. To address the length of time it takes to produce assessments, the IRIS Program is (1) employing project management principles and specialized software that enable the program to better plan assessment schedules and utilize staff to make the systematic review process more efficient; (2) focusing on better scoping assessments to create timely, fit-for-purpose products that address specific agency needs; and (3) streamlining the peer review process as much as possible. The Program Has Adopted Project Management Principles and New Software The first way in which the IRIS Program is addressing the length of time it takes to produce assessments is by utilizing project management principles and new software that enable the program to better plan assessment schedules and utilize staff. IRIS officials said that by using these tools, IRIS staff are able to view project tasks, timelines, and milestones to manage their individual tasks and assessment work. For example, IRIS officials said that as part of an EPA-wide initiative, they began incorporating lean management techniques, which aim to improve efficiency and effectiveness by reducing unnecessary process steps and waiting time. Additionally, IRIS officials said that they have begun using a staffing model that trains staff to be proficient in all phases of the systematic review process (i.e., screening, data extraction, study evaluation, and evidence synthesis). This modularity will make it easier for staff to work across teams and on multiple projects, assisting with systematic review needs while also contributing in their areas of expertise, according to IRIS Program officials. In addition, the IRIS Program began using both project management software and business intelligence and visualization software in 2017. IRIS Program leadership is using this software to generate resource allocation reports showing staff assignments, enabling leadership to better manage staff workloads. According to IRIS officials, the recent adoption of specialized systematic review software also enables program staff to perform more literature searches faster, and the ability to filter search results allows staff to find more quickly the most relevant information for an assessment. Use of software tools with machine-learning capabilities facilitate program staff’s ability to screen studies for relevance more quickly compared to approaches used before 2017. Prior to the adoption of these specialized software tools, much of the development of an assessment was manual (i.e., using a spreadsheet). For example, for one assessment developed manually, contactors working on an IRIS assessment took over 200 hours to screen and catalog 1,200 epidemiological studies, including carrying out quality assurance checks. By comparison, using machine-learning tools, EPA staff were able to screen almost 5,500 articles in about 30 hours. With the new tools, quality assurance was embedded into the workflow by having two independent reviewers and a software-facilitated process track and resolve screening conflicts. Additionally, an official from EPA’s National Health and Environmental Effects Research Laboratory said that the laboratory uses a similar screening process. The laboratory worked with the IRIS Program to identify similar constructs in their processes and used each other’s results to make changes and validate tools used by both. According to IRIS officials, as a result, the use of these tools has created more efficient workflow processes, leading to considerable cost and time savings. The incorporation of systematic review software tools has greatly helped the program more efficiently carry out tasks like screening literature, evaluating study quality, extracting data, and developing visualizations, according to IRIS Program officials we interviewed. Most importantly, the software tools allow multiple staff members to work on tasks simultaneously, rather than one at a time, facilitating concurrent completion of key assessment pieces. The Program Tailors Assessments to Program and Regional Office Needs The second way in which the IRIS Program is reducing the length of time it takes to produce assessments is by tailoring them to program and regional office needs, called fit-for-purpose assessments. According to IRIS officials, part of the reason assessments historically were time- consuming was because the program tried to synthesize and present all possible information on the human health effects of a particular chemical, including multiple exposure pathways (e.g., inhalation, ingestion, or dermal) and reference doses, reference concentrations, and cancerous and non-cancerous effects. This required large amounts of data extraction and was very time intensive. Beginning in early 2017, the program began implementing the fit-for-purpose approach to producing assessments. IRIS officials said the idea is that instead of producing a wide-ranging assessment, the program can produce assessments that are more limited in scope and targeted to specific program and regional office needs, reducing the amount of time IRIS staff needed to search for information, synthesize it and draft, review, and issue an assessment. For example, if the Office of Air and Radiation needed a chemical assessment that examined only inhalation exposures, the IRIS Program could limit its assessment to a single exposure pathway, which would reduce the amount of data that staff review and extract and, with less text to draft and less complex peer reviews, allow the assessment to more quickly move through the process. IRIS officials said that if offices make subsequent requests for other effects or exposure pathways, the IRIS Program can update the original assessment. IRIS officials said that they expect time savings as a result of moving to the fit-for-purpose model. As of November 1, 2018, the IRIS Program had produced two fit-for-purpose assessments: a request for correction on chloroprene and an update of the assessment on acrolein. An assessment on perfluorobutane sulfonic acid (PFBS) was also released for public comment following peer review. PFBS are a member of a class of man-made chemicals known as per- and polyfluoroalkyl substances (PFAS)—a groups that also includes perfluorooctane sulfonate acid (PFOS), perfluorooctanoic acid (PFOA), GenX, and many others. In addition, since 2017, the IRIS Program released scoping and problem formulation materials for six IRIS chemical assessments (nitrates/nitrites, chloroform, ethylbenzene, uranium, ammonia, and naphthalene). Additionally, the program is examining ways to assist program and regional offices with information that may not necessitate developing a full assessment. For example, the Office of Air and Radiation was doing work using a toxicity value for acrolein that the California Environmental Protection Agency prepared in 2008, because that value was more recent than the value in the IRIS database. However, a large number of studies on acrolein had been released since 2008, so the IRIS Program searched approximately 10,000 new studies and concluded that the study used by California Environmental Protection Agency in 2008 was still the most appropriate study for chronic toxicity value derivation. In addition, IRIS staff developed an updated draft reference concentration for acrolein based on this study. The screening and update process took approximately 4 months, demonstrating how the IRIS Program’s use of new tools and a targeted scope resulted in more timely attention to program office needs. The Program Is Streamlining the Peer Review Process The third way the IRIS Program is addressing the length of time it takes to produce assessments is by streamlining the peer review process as much as possible without compromising the quality of the review. EPA guidelines require peer review of all IRIS assessments. Smaller, less complex assessments may be peer reviewed through a contractor-led letter review or panel; more complex assessments are usually reviewed by a full Scientific Advisory Board (SAB) or a NAS panel, though IRIS leadership determines the most appropriate method of peer review based on Office of Management and Budget and EPA Peer Review Handbook guidelines. While the contractor-led letter or panel reviews are no less robust than full SAB or NAS panel reviews, the contractor-led reviews are usually smaller and completed in less time because they are reviewing smaller, less complex IRIS assessments. The time savings occur because the reviewers do not typically meet in person, or may meet only once, typically taking a few months to complete their reviews. In contrast, SAB and NAS panels involve larger numbers of people who meet multiple times, review longer and more complex assessments, and must reach consensus on their reviews. As a result, SAB and NAS peer reviews can take more than a year to complete. IRIS officials said that as they try to produce more fit-for-purpose assessments that are smaller in scope, they plan to utilize letter reviews as appropriate, to streamline the peer review process. IRIS Program officials said they also hope that other changes they recently implemented—primarily, increased transparency and systematic review—will help speed up the peer review process by producing a higher-quality overall draft. Another major category of NAS recommendations that the IRIS Program has addressed is the need for greater transparency in how the program conducts assessments. For example, one industry representative expressed concern in August 2018 about transparency before the program began making changes, describing the IRIS Program as a “black box” because “no one knew how the program created its methodologies, weighted evidence, or produced assessments.” In response, the IRIS Program has in the past several years (1) implemented systematic review, which provides a structured and transparent process for identifying relevant studies, reviewing their methodological strengths and weaknesses, and integrating these studies as part of a weight of evidence analysis, and (2) increased outreach efforts with stakeholders and the public, both in terms of the frequency and the depth of content about assessment preparation. The Program Began Implementing Systematic Review as a Basis of Its Assessments The IRIS Program began addressing the need for greater transparency by implementing systematic review as a basis for every assessment and has been doing so for several years. A systematic review is a structured and documented process for transparent literature review. It is a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. The goal of systematic review methods is to ensure that the review is complete, unbiased, reproducible, and transparent. By using systematic review, the IRIS Program can demonstrate that it considered all available literature in forming conclusions and deriving toxicity values. Utilizing the new software tools described above allows program staff to search more widely than before and to identify the most relevant results faster and more accurately. The IRIS Program is working with technical experts to increase the applications of machine learning for carrying out systematic review. Additionally, new software allows the IRIS Program to save and publish its search strings and to indicate why it selected certain studies over others for review and inclusion. The software also allows multiple staff to check searches and concur or not-concur with the initial assessment about including a scientific article in the draft assessment. IRIS officials told us that the transparency associated with systematic review and clearer explanation of methodologies in assessments (as well as releasing subsidiary documents, such as IRIS Assessment Plans and Assessment Protocols) will improve stakeholders’ understanding of how the program arrives at its conclusions. The Program Has Made Changes to Communication Frequency and Type The IRIS Program also furthered transparency by increasing the frequency, structure, and content of communications with EPA program and regional offices about overall program priorities and individual assessments. This allows EPA program and regional offices to know when to expect assessments, as well as what those assessments will cover. To prepare the 2015 Multi-Year Agenda, the IRIS Program solicited requests from EPA program and regional offices about which chemical assessments they needed; these requests were released in December 2015. When new leadership joined the IRIS Program in early 2017, the new officials began reaching out to individual program and regional offices to re-confirm their needs and priorities. IRIS officials said this effort was in part to ensure that the IRIS Program was delivering what the program offices needed, as well as to help the IRIS Program keep its priorities up to date and ensure that resources (primarily staff) were aligned with EPA-wide priorities. Based on these conversations with program and regional office staff, the IRIS Program made some chemical assessments higher priority and removed others from the program’s workflow, consistent with stated needs. In May 2018, the IRIS Program prepared a statement for posting on the IRIS website outlining these changes to the program’s workflow and an updated list of assessments that were being developed with anticipated completion time frames. However, EPA leadership in ORD—the office that oversees the IRIS Program—did not approve this statement for release because current EPA leadership in program and regional offices had not formally requested these assessments. Nevertheless, officials from program and regional offices that use IRIS assessments told us that they received clear communication from the IRIS Program about priorities and timelines for individual assessments. According to these officials, some of this communication took place when IRIS Program leadership reached out to program and regional office officials to confirm their needs, and some took place during monthly telephone calls the IRIS Program held to update stakeholders on assessment development timelines. Program and regional office officials told us that they appreciated the IRIS Program’s recent efforts to understand program and regional office needs and timelines; communicate the status of assessments more frequently; and find ways to assist program offices that may not require developing a full assessment, such as assessment updates or literature reviews. Since 2013, the IRIS Program has released preliminary assessment materials—including IRIS Assessment Plans and assessment protocols— so that EPA and interagency stakeholders and the public could be aware of scoping and problem formulation for each assessment. Since 2017, according to EPA, these documents had a new structure and better demonstrate the application of systematic review, and they continue to convey EPA’s need for each assessment and frame questions specific to each assessment. Officials in several program and regional offices that use IRIS assessments told us that the release of IRIS Assessment Plans and protocols was very helpful because it allowed them to offer early input to the IRIS Program about the scope of an assessment, when it could affect the direction of the assessment. IRIS officials also said that they created templates for several parts of the assessment process, including the IRIS Assessment Plans and assessment protocols, which help maintain consistency throughout assessment development and from one assessment to the next. During calendar year 2018, the IRIS Program planned to release documents or hold meetings for 15 of the 23 ongoing chemical assessments in development, as well as for the IRIS Handbook and a template for assessment protocols. From January through May 2018, the IRIS Program met each of its internal deadlines for work on 9 different chemical assessments and released the template for assessment protocols for agency review. The IRIS Program also produced a report to Congress on the program’s work in January 2018 and took part in a NAS review of the program in February 2018. The NAS review, which offered a third-party assessment of the program’s efforts, provided a supportive assessment of ongoing transformations aimed at ensuring data quality, new systematic approaches for data analysis and expanded stakeholder engagement efforts, and increased the efficiency of assessments. According to the report, NAS reviewers were impressed with the changes being instituted in the IRIS Program since 2014, including substantive reforms by new IRIS Program leadership, such as the development, implementation, and use of systematic review methods to conduct IRIS assessments. In addition, as of August 2018, the final IRIS assessment of hexahydro-1,3,5-trinitro-1,3,5-triazine (RDX) was issued. In early November 2018, IRIS officials told us that the agency had almost completed internal review of the handbook, which was being prepared for public release. In December 2018, the IRIS Program and OPPT participated in a NAS workshop that informed the systematic review of mechanistic evidence. The IRIS Program has made important changes aimed at producing more timely and transparent assessments, but IRIS officials told us that proposed budget cuts have caused them concern about whether they will have sufficient resources to expand assessment work in the future. The human health risk assessment area, of which IRIS’s budget makes up approximately half, has been funded at about $38 million annually since fiscal year 2013 based on our review of EPA budget documents. However, the President’s budget request for human health risk assessment work in fiscal years 2018 and 2019 was $22.5 million and $22.2 million, respectively. This represents a cut of approximately $17 million from previous budget levels dating back to fiscal year 2013. The IRIS Program budget would drop approximately 40 percent from $20.8 million to approximately $12 million if these cuts were enacted. Congress did not support these reductions. Specifically, according to the joint explanatory statements accompanying the Consolidated Appropriations Act, 2018, and Consolidated Appropriations Act, 2019, Congress had agreed to continue providing funding at fiscal year 2017 enacted levels. Cuts to the program could impact EPA’s regulatory work: Officials in almost all of the program and regional offices that use IRIS assessments told us that they rely on IRIS assessments to do their work—it is the first place they look for chemical toxicity values, and if the IRIS Program is unable to produce assessments, their offices would be challenged to meet statutory deadlines and there would be a generally negative effect on public health. The IRIS Program made progress developing assessments and producing assessment documentation (e.g., IRIS Assessment Plans and protocols) in early 2018. However, EPA leadership deliberations about the program’s priorities that took place from June through December 2018 delayed the program’s assessment production. IRIS officials told us that in early June 2018 EPA leadership in ORD informed them that the IRIS Program could not release an assessment without a formal request for that assessment from the current leadership of a program office. At the request of the Administrator, IRIS officials prepared a survey of program and regional offices, asking them to re- confirm their needs for 20 assessments that were in development. This survey was sent by memorandum in August 2018. Program office responses were to be signed by the Assistant Administrator of each program office to ensure that the re-confirmations were consistent with the priorities of EPA program office leadership. While survey responses were being compiled, EPA leadership in ORD instructed the IRIS Program not to publically release any assessment documentation. As a result, any assessment or subsidiary assessment document (e.g., an IRIS Assessment Plan or protocol) that was ready for agency review, public comment, or peer review was unable to proceed through the IRIS assessment development process. In late October 2018, prior to releasing results of the initial program and regional office survey, EPA leadership in ORD made a second request of program offices for a prioritized list of assessments. According to officials from the IRIS office, who were queried for advice by officials from some program offices, ORD’s second request was made verbally at a meeting and included direction to the program offices to limit their requests to no more than three to four chemicals. ORD’s request did not provide information on the basis for selecting priorities or the reason for the limit of three or four chemical assessments from the original survey submissions. The calls for advice from program office officials represented the first time the IRIS Program heard about the requests for a prioritized list, according to IRIS program officials. And since neither the program and regional offices nor the IRIS Program had information from the Administrator’s office about what the prioritization was meant to achieve, the IRIS Program was unable to provide guidance about what chemicals might be considered a priority, or how many they might be able to continue work on. When EPA leadership’s deliberations about the program’s priorities were completed, a memorandum was issued on December 4, 2018, that listed 11 chemical assessments that the IRIS Program would develop. This was a reduction of the program’s workflow from 22 assessments, but the memorandum announcing the reduced workflow gave no reason for the reduction. The memorandum accompanying the list of 11 chemicals gave no indication of when more assessments could be requested or if IRIS’s workflow would remain at 11 chemicals for the foreseeable future. According to the memorandum, the 11 chemicals were requested by two EPA program offices (the Office of Water and the Office of Land and Emergency Management). We received this memorandum at the end of our review and did not have the opportunity to review the prioritization process that led to its drafting. Two weeks after the issuance of the memorandum, the IRIS program publicly issued an outlook of program activities, which included two additional assessments that were not included in the memorandum. These two assessments, ethyl tertiary butyl ether (ETBE) and tert-butyl alcohol (TBA), were not included in the memorandum because they were out for public comment and external peer review. Furthermore, four assessments that were in the later stages of development and had not been issued were not included in the December 2018 Outlook. The four assessments were: acrylonitrile, n-Butyl alcohol, formaldehyde, and polycyclic aromatic hydrocarbon (PAH). The assessment of formaldehyde was, according to the “IRIS Assessments in Development” website, at Step 4 of the IRIS process (an assessment is drafted and was ready to be released for public comment and external peer review). The absence of these four assessments from the December 2018 Outlook could create confusion for stakeholders interested in them. EPA provided no information on the status of these four assessments or whether it planned to discontinue working on them or restart them at another time. As we have previously reported, an overarching factor that affects EPA’s ability to complete IRIS assessments in a timely manner is that once a delay in the assessment process occurs, work that has been completed can become outdated, necessitating rework throughout some or all of the assessment process. Thus, it remains to be seen when these assessments can be expected to move to the next step in the IRIS process or be completed. As of December 19, 2018, the status of the 13 assessments in the December 2018 Outlook was: External peer review: ETBE and TBA. Draft Development: arsenic, inorganic; chromium VI; polychlorinated biphenyls (PCBs; noncancer); perfluorononanoic acid (PFNA); perfluorobutanoic acid (PFBA); perfluorohexanoic acid (PFHxA); perfluorohexane sulfonate (PFHxS); and perfluorodecanoic acid (PFDA). Scoping and Problem Formulation: mercury salts; methylmercury; vanadium and compounds. According to IRIS officials, the IRIS Program was unable to release any work since June 2018, while it was waiting for feedback from the Administrator’s office regarding whether its assessment workflow was consistent with agency priorities. IRIS officials told us that staff continued whatever draft development work that they could do internally, but several IRIS staff have been working increasingly for OPPT to support its work preparing risk evaluations under TSCA. ORD reported to us that in September 2018—3 months after IRIS assessments were stopped from being released because of ongoing EPA leadership deliberations—5 of approximately 30 IRIS staff were supporting OPPT with 25 to 50 percent of their time. In October 2018—4 months after IRIS assessments were stopped from being released—28 of approximately 30 IRIS staff were supporting OPPT with 25 to 50 percent of their time. According to IRIS officials, this was occurring primarily because OPPT has a significant amount of work to do to meet its statutory deadlines, and OPPT needed IRIS staff expertise to help meet those deadlines. As noted above, TSCA establishes a regulatory standard that generally differs from those under other environmental laws, so the TSCA assessments will not necessarily be relevant to other EPA programs that have relied on IRIS endpoint values in making their regulatory decisions. EPA has demonstrated progress implementing TSCA by responding to TSCA’s statutory deadlines through the end of fiscal year 2018, including promulgating rules, developing guidance, and releasing reports. However, EPA faces key challenges to its ability to implement TSCA, such as managing the risks posed by ongoing litigation, ensuring appropriate resources, developing guidance to ensure consistency, and ensuring that the new chemicals review process is efficient and predictable. EPA has responded to initial statutory deadlines under TSCA, as amended by the Lautenberg Act, including requirements to promulgate new rules, develop guidance, and release reports. For example, EPA began 10 risk evaluations drawn from the 2014 update of the TSCA Work Plan within 180 days of enactment of the Lautenberg Act (§ 6(b)(2)(A)); submitted an initial report to Congress estimating capacity for and resources needed to complete required risk evaluations within 6 months of enactment (§ 26(m)(1)); carried out and published in the Federal Register an inventory of mercury supply, use, and trade in the United States by April 1, 2017. (§ 8(b)(10)(B)); developed guidance to assist interested persons in developing and submitting draft risk evaluations within 1 year of enactment (§ 26(l)(5)); and developed a plan for using alternative test methods to reduce use of vertebrate animal testing within 2 years of enactment (§ 4(h)(2)(A)). In addition, in four areas in which Congress required EPA to establish processes and structures for TSCA, EPA finalized four rules detailing the general processes for prioritizing and evaluating chemicals under TSCA, known together as the Framework Rules. EPA responded to the 1-year deadlines to establish three of the four Framework Rules. These three rules are the risk prioritization rule, which explains EPA’s process for prioritizing existing chemicals for risk evaluation; the risk evaluation process rule, which explains EPA’s process for conducting risk evaluations on existing chemicals; and the inventory notification rule, which requires manufacturers and processors of chemical substances to report which chemicals are currently in commerce. The fourth Framework Rule EPA issued, which had no issuance deadline, implements a Lautenberg Act provision authorizing EPA to collect fees for carrying out a number of different activities under TSCA, including collecting fees from manufacturers and processors that submit new chemicals or submit chemicals for significant new uses to EPA for review. Though EPA responded to all of the statutory deadlines, some environmental and industry stakeholder organizations we interviewed told us that they do not believe this is a complete measure of how well EPA is implementing TSCA. Representatives from one environmental stakeholder organization told us in July 2018 that it is still too early to assess how well EPA is implementing TSCA because none of the existing chemical risk evaluations ongoing under the new process have been released; the wording in the new rules and documentation is unclear; and the risk prioritization rule, the risk evaluation rule, and the inventory reset rule have been challenged in court. However, in January 2019 they told us that they were too optimistic in their assessment of TSCA implementation and believe EPA is falling behind in its progress. As of December 2018, representatives from another environmental stakeholder group told us that, while EPA has met a number of major statutory deadlines, the agency’s rules and other actions do not reflect the best available science and are contrary to both the letter and intent of the new TSCA Act. However, in January 2019 an industry stakeholder organization noted that the 2016 amendments to TSCA are generally being implemented effectively and efficiently as Congress envisioned, and the agency continues to meet important deadlines required by the law. In addition, they also told us that EPA’s TSCA program is also utilizing the best available science and a weight of the evidence approach to make high quality chemical management decisions. Representatives from industry stakeholder organizations we interviewed told us they believe the rules are consistent with TSCA, but that EPA is not consistently meeting the 90-day deadline to make determinations on new chemicals or the 30-day deadline to make determinations on low-volume exemptions. EPA faces challenges with its ability to implement TSCA, such as managing the risk posed by ongoing litigation, ensuring appropriate resources, developing guidance documents to ensure consistency, and ensuring that the new chemicals review process is efficient and predictable. Three of the four Framework Rules that EPA issued to implement TSCA have been challenged in court: the risk prioritization rule, the risk evaluation rule, and the inventory notification rule. Procedures for Prioritization of Chemicals for Risk Evaluation under the Toxic Substance Control Act (risk prioritization rule). In Safer Chemicals, Healthy Families v. U.S. Environmental Protection Agency, a collection of environmental and public health organizations challenged several aspects of EPA’s TSCA implementation, including the risk prioritization rule. Specifically, the environmental organizations argue, among other things, that the plain language of TSCA requires EPA to consider all conditions of use in prioritizing chemicals for review under TSCA, rather than excluding, for example, uses that EPA believes are “legacy uses” for which a chemical is no longer marketed. EPA and chemical industry intervenors respond by arguing that TSCA grants EPA discretion to determine what conditions constitute a chemical’s conditions of use and to generally exclude legacy activities—primarily historical activities that do not involve ongoing or prospective manufacturing, processing, or distribution in commerce of a chemical substance as a product. Procedures for Chemical Risk Evaluation under the Amended Toxic Substances Control Act (risk evaluation rule). In Safer Chemicals, Healthy Families v. U.S. Environmental Protection Agency, the environmental organizations also contend that EPA’s risk evaluation rule is contrary to TSCA, in part because, as noted above, the rule “impermissibly” excludes uses that the law requires EPA to include in its risk evaluations. EPA and industry intervenors responded by arguing that TSCA grants EPA discretion to determine what conditions constitute a chemical’s conditions of use. The organizations also argued that the risk evaluation rule would deter public participation in the risk evaluation process by imposing criminal penalties on a member of the public who submits incomplete information to EPA but does not impose similar penalties on manufacturers. In August 2018, the government moved to vacate the penalty regulation, and the environmental organizations consented to this motion. Inactive) Requirements (inventory notification rule). In Environmental Defense Fund v. U.S. Environmental Protection Agency, an environmental organization challenged EPA’s inventory notification rule, which EPA issued in response to a TSCA requirement that EPA identify which chemicals in the TSCA inventory are still in use and require substantiation of claims that chemical identities constitute confidential business information that can be withheld from public disclosure. The environmental organization argued, among other things, that the rule impermissibly allows any persons to assert confidentiality claims for any chemical they manufacture or process, rather than just the original claimant. EPA and industry intervenors responded in part by arguing that TSCA specifically allows any affected manufacturers to maintain an existing confidentiality claim for a specific chemical identity, which the industry intervenors assert constitutes critically important intellectual property. OPPT officials told us they are trying to not anticipate the results of the litigation and, instead, address the outcome of each case as it is decided. They stated that they are staying aware of developments in ongoing litigation and are constantly considering potential outcomes but believe it would not be reasonable to prepare explicit resource plans for unknown future scenarios. If EPA loses any of these lawsuits, it may need to devote additional resources to implement the relevant provisions of TSCA. For example, if the suit involving the risk evaluation rule is successful, EPA may be forced to redo parts of its risk evaluations close to the December 2019 deadline to finalize these evaluations. EPA is required to complete its first 10 existing chemical evaluations not later than 3 years after the date on which it initiated the risk evaluations, which was December 2016. TSCA also allows for an extension of the risk evaluation deadlines for up to 6 months if the agency deems it necessary. The Lautenberg Act greatly increased OPPT’s workload. Prior to the enactment of the Lautenberg Act, EPA did not have deadlines for completing existing chemical evaluations. Under the Lautenberg Act, EPA must finalize 10 ongoing risk evaluations by December 2019, which represents a tight deadline, according to EPA officials. Furthermore, the law requires EPA to ensure that 20 risk evaluations are ongoing for high- priority substances 3-1/2 years after enactment and that at least 20 chemical substances have been designated as low-priority substances. In addition, under TSCA prior to the Lautenberg Act, a new chemical could enter commerce after 90 days unless EPA took action to the contrary. Under the Lautenberg Act, EPA is required to make a determination on a new chemical before it can be manufactured—another source of increased workload. Partially because of the increased workload, some OPPT officials told us that they have concerns about staff capacity within OPPT. Officials in both the Chemical Control Division (responsible for risk management) and the Risk Assessment Division (responsible for risk assessment) said that they do not have sufficient resources to do their work. This included staff from all five technical teams we interviewed in the Risk Assessment Division. Technical teams are working groups organized by discipline that bring together experts from across OPPT branches. The Risk Assessment Division is particularly affected by the heavy workload, according to OPPT officials and representatives from an industry stakeholder organization. The division must review all of the premanufacture notices for new chemicals and contribute to the first 10 existing chemical evaluations. Officials from the Chemical Control Division told us that the Risk Assessment Division is struggling more because its work requires more technical employees. The officials said that EPA is hiring additional full-time equivalents (FTE), but it takes time to train new people, and this will initially increase workload. Officials told us that in July 2018, OPPT had about 300 FTEs and was authorized to hire 40 additional FTEs. As of October 2018, OPPT officials told us that they had hired or extended offers to 20 to 25 of that 40 and continued to hire more employees. OPPT officials told us that reaching an appropriate level of FTEs—including recruiting and retaining staff—is challenging. OPPT officials said they expect that the recently announced initiative to implement direct hiring authority for scientific and technical positions will have a positive impact on these efforts. To address the staffing challenge, staff have also been reassigned from other parts of EPA to OPPT. For example, staff in the Safer Choice Program—an EPA program that helps consumers, businesses, and purchasers find products that perform and are safer for human health and the environment—were redeployed to the Chemical Control and the Risk Assessment Divisions. Representatives from both industry stakeholder organizations we interviewed told us that it can be difficult to work with recently reassigned staff who are not familiar with the chemicals they are working on. Representatives from an industry stakeholder organization told us that, in some cases, OPPT staff are ill-prepared to make decisions about a premanufacture notice. OPPT senior officials said there is always a learning curve for reassigned employees, but they do not put new people in positions to make decisions on premanufacture notices. They said that these decisions are never made by one person in a vacuum. OPPT officials and staff told us that they are generally optimistic about an upcoming reorganization of OPPT that will separate assessment and management of new and existing chemicals programs and better align the structure of OPPT with the focus of TSCA’s provisions. For example, the Chemical Control Division and the Risk Assessment Division currently each handle both new and existing chemicals, and the planned reorganization will divide the divisions into new and existing chemical divisions. However, staff told us that they have concerns about whether the new divisions will be adequately staffed, the timing of the reorganization, and their future placements. Staff from multiple technical teams we interviewed in the Risk Assessment Division said that they are not sure if, after the reorganization, the new divisions will be adequately staffed. Staff from one technical team said there has been increased attrition in recent years, partially because of concerns about the upcoming reorganization. Staff from another technical team said that a large number of management positions are unfilled. Staff from multiple technical teams told us that it will take time after the reorganization to redistribute work and train staff. Staff from one team said the reorganization is ill-timed because there are currently too many other ongoing high-priority projects. Staff from multiple technical teams also told us that they are experiencing anxiety about their future placements and with whom they will work. In commenting on a draft of this report, EPA stated that the concerns raised by staff are likely common to any program undergoing change. OPPT officials said they submitted the reorganization proposal to EPA’s Office of Mission Support—formerly the Office of Administration and Resources Management—in October 2018 and that it could take several more months as EPA management works out details with labor unions and addresses other issues. Officials said that they anticipate implementing the reorganization in early 2019. OPPT senior officials said that now that OPPT has many new responsibilities and a heavier workload, they are taking steps to improve capacity by implementing the reorganization and hiring new staff. The officials said that though there will inevitably be growing pains, the changes are part of a larger plan specifically designed to better position OPPT to implement TSCA. Senior officials also told us that they have spent considerable time setting expectations for new and existing staff. In tandem with the major changes that increased EPA’s workload, the 2016 amendments to TSCA authorize EPA to establish fees to defray a portion of the costs of administering TSCA sections 4, 5, and 6 and collecting, processing, reviewing, providing access to, and protecting information about chemical substances from disclosure, as appropriate, under TSCA section 14. Affected businesses began incurring fees under the new rule as of October 1, 2018, but it is unclear whether the fees collected will be sufficient to support relevant parts of the program. OPPT officials told us that while they are uncertain how much the fees rule will generate the first year, they believe that over the course of a few years, the amount of money generated should stabilize. The first year is where officials are not sure how much they may receive. Officials expect to collect an average of $20 million per year over the next 3 fiscal years. In fiscal year 2019, however, they expect to collect approximately $7 million to $8 million. According to EPA, the agency will be tracking its costs and use that information to adjust future fees, if appropriate. As required by law, EPA will evaluate and readjust, if necessary, the fees every 3 years. EPA estimates the average yearly cost of TSCA implementation for fiscal years 2019 through 2021 to be $80,178,000. EPA’s fiscal year 2019 budget justification shows $57,973,700 allocated to TSCA implementation. However, EPA does not expect a budget shortfall in fiscal year 2019 because, according to officials, they (1) have funds available from 2018 to support fiscal year 2019 needs, (2) receive support from other EPA offices like the Office of General Counsel and the Office of Research and Development, (3) expect fiscal year 2019 costs to be lower than the 3-year average described in the fees rule, and (4) expect some indirect costs to be covered by non-TSCA budget categories. EPA also faces challenges in developing guidance to ensure consistency in implementing the law. OPPT officials said that, given the tight timelines that TSCA requires, they have not yet created all the necessary guidance for staff implementing the law. Officials likened it to building an airplane as they fly it, as they must create guidance and processes, while simultaneously applying them to chemical evaluations. Staff from four of five technical teams we interviewed are either currently updating their guidance, still developing their guidance, or have never developed guidance before. Staff from two teams told us that they are developing the guidance as they apply it to their work. OPPT officials told us that they are using some guidance that was in place before the Lautenberg Act was enacted, though they are working on updates. Representatives we interviewed from industry stakeholder organizations said they want EPA to be clear about its standards for the new chemicals program and how they are defining terms in TSCA. Representatives from one industry stakeholder organization suggested that EPA should establish some definitions and develop guidance on how to apply those definitions, in order to help both chemical manufacturers and reviewers within OPPT. In June 2018, EPA released “Points to Consider When Preparing TSCA New Chemical Notifications,” guidance that representatives from industry stakeholder organizations said is helpful, but they are still not sure how EPA is using information like the Points to Consider guidance in its evaluations and against what standard EPA’s reviewers are reviewing and assessing a chemical. Representatives we interviewed from industry stakeholder organizations said that decisions on new chemical reviews depend on individual reviewers because EPA has not provided the reviewers with guidance that ensures consistency. OPPT officials also said consistency is a challenge in conducting risk assessments. Representatives we interviewed from environmental stakeholder organizations did not mention consistency as an area of challenge. Representatives from both industry stakeholder organization we interviewed also told us that the new chemicals program is too slow and unpredictable, which can negatively affect innovation. For example, representatives from one company told us in comments they provided through an industry stakeholder organization we interviewed that it submitted a premanufacture notice for a substance that would decrease the potential for worker and environmental exposure while providing improved product performance. The approval process extended to nearly 550 days compared to the 90 days it typically took to obtain approval prior to TSCA’s amendment. EPA can request extensions, and submitters can voluntarily suspend the review process; therefore, the overall process can extend beyond the 90-day requirement. For example, in the new chemical review process, EPA first makes an initial determination. If a company does not like this initial determination, it can request more time to provide additional data or develop new data in an effort to get a positive final determination. A company withdraws its submission prior to a final EPA determination if it is clear the determination will not be favorable and the chemical will be regulated. EPA officials said the agency does not violate the mandated timelines because submitters agree to voluntarily suspend the review process. However, representatives from one industry stakeholder organization told us that as of December 2018, with the passage of time and greater familiarity with Lautenberg, OPPT’s decision making process has improved and is more predictable. EPA officials said that historically, even among new chemicals for which EPA completed review, 57 percent actually entered commerce. Officials said that in the past companies submitted new chemicals just to see what determinations EPA would make. Going forward, as of October 2018, officials said they expect larger fees will result in some companies choosing to be more selective in the chemicals they submit to the program. In addition, EPA officials told us that after OPPT’s reorganization, a more devoted team will focus on pre-notice meetings with companies. Officials said this should reduce some of the back and forth with submitters, thereby improving timelines. Representatives we interviewed from industry stakeholder organizations also told us that delays motivate companies to introduce chemicals first in foreign markets. For example, one company told us through comments it provided through an industry stakeholder organization we interviewed that it developed a new technology in the United States, but because of the lengthy delays experienced with new chemicals reviewed under TSCA, they will neither register nor commercialize the product in the United States at this time. Rather, the company has decided to pursue commercialization in Europe, which will enable the company to deliver the benefits of this new technology to their customers in the European market sooner than is possible in the United States. We provided a draft of this report to EPA for its review and comment. We received written comments from EPA that are reproduced in appendix I and summarized below. In its written comments, EPA stated that while the draft comprehensively describes the challenges facing the TSCA and IRIS programs, it does not appropriately address EPA’s extensive progress in implementing TSCA, and EPA recommended that our final report include information regarding its accomplishments under the new law. Specifically, we report on the steps EPA has taken to respond to the requirements of the law because in many instances, whether EPA’s response is legally sufficient is in litigation, and GAO does not typically express a view on legal or factual matters in dispute before a court. We have updated our report with additional examples, which the agency provided in its comments, of steps it has taken to implement TSCA. In addition, EPA requested that we consider its progress made in addressing and controlling toxic chemicals with respect to the five criteria for removal from our high-risk list. The application of the high-risk criteria was not within the scope of this report. Our forthcoming 2019 high-risk update will address actions taken by agencies on the list, including EPA, since the last update in 2017. EPA said that to monitor progress, it had put into place a rigorous program; as a regular practice, EPA stated that Deputy Assistant Administrators from the Office of Chemical Safety and Pollution Prevention conduct monthly Business Review meetings with the Office Directors, Deputy Office Directors, lead region representatives, and other key staff. EPA stated that during these meetings they discuss their organizations’ operations and performance, including TSCA implementation status, using performance charts to track progress on mission measures, identify and update countermeasures, and resolve problems. However, over the year that we conducted our review, EPA officials did not mention conducting such meetings and did not provide documentation that such meetings took place. Further, in its written comments, EPA provided technical comments on the draft report, which we address as appropriate. In one comment, EPA stated that instead of noting that the agency has successfully implemented many statutory requirements, the draft report stated that EPA responded to deadlines. We believe the report correctly characterizes steps EPA has taken to implement TSCA, and, as noted above, whether EPA’s response is legally sufficient is in litigation, and GAO does not typically express a view on legal or factual matters in dispute before a court. In another case, the technical comments contradicted facts that we gathered during our review. For instance, while EPA stated that the draft report incorrectly noted that most of the IRIS staff had been working on TSCA activities, we provide further information to support our original statement; we replaced the term ‘most’ with specific data on the number of IRIS staff and the percentage of their time that was devoted to TSCA activities. Also in its technical comments, EPA stated that our analysis highlighted uncertainty resulting from the agency’s recent activities to ensure IRIS Program efforts were aligned with the highest priorities of the agency. EPA acknowledged that this action did result in a delay but that in the long term, it would ensure that EPA’s program and regional office priorities are being addressed and that each office is fully engaged in the development of IRIS assessments that will strengthen the agency’s ability to address its mission for protecting human health and the environment. However, as we state in our report, prior to releasing results from the initial program and regional office survey, EPA leadership in ORD made a second request for a prioritized list of chemical assessments. According to officials from the IRIS office, who were queried for advice, the second request was made verbally at a meeting and did not provide the offices with information on the basis for selecting priorities or the reason for limiting the number of assessments to three or four chemicals. In addition, the ultimate priority list EPA issued in December 2018 reflected the priorities of two program offices and did not provide evidence that other EPA program offices had no interest in IRIS assessments. Because EPA did not identify the basis for program offices to select priorities or the reason for limiting the number of chemicals to assess, the process was not transparent, leaving room for uncertainty. EPA also provided additional technical comments, which we have incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Administrator of the Environmental Protection Agency, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. In addition to the contact named above, Diane Raynes (Assistant Director), Summer Lingard-Smith (Analyst in Charge), Alisa Carrigan, Tara Congdon, Richard P. Johnson, Amber Sinclair, and William Tedrick made key contributions to this report. In addition Karen Howard, Dennis Mayo, Dan Royer, and Sara Sullivan made important contributions. Chemical Innovation: Technologies to Make Processes and Products More Sustainable. GAO-18-307. Washington, D.C.: February 8, 2018. Chemicals Management: Observations on Human Health Risk Assessment and Management by Selected Foreign Programs. GAO-16-111R. Washington, D.C.: October 9, 2015. Chemical Assessments: Agencies Coordinate Activities, but Additional Action Could Enhance Efforts. GAO-14-763. Washington, D.C.: September 29, 2014. Chemical Regulation: Observations on the Toxic Substances Control Act and EPA Implementation. GAO-13-696T. Washington, D.C.: June 13, 2013. Chemical Assessments: An Agencywide Strategy May Help EPA Address Unmet Needs for Integrated Risk Information System Assessments. GAO-13-369. Washington, D.C.: May 10, 2013. Toxic Substances: EPA Has Increased Efforts to Assess and Control Chemicals but Could Strengthen Its Approach. GAO-13-249. Washington, D.C.: March 22, 2013. Chemical Assessments: Challenges Remain with EPA’s Integrated Risk Information System Program. GAO-12-42. Washington, D.C.: December 9, 2011. Chemical Assessments: Low Productivity and New Interagency Review Process Limit the Usefulness and Credibility of EPA’s Integrated Risk Information System. GAO-08-440. Washington D.C: March 7, 2008. High-Risk Series: An Update. GAO-09-271. Washington D.C: January 2009. High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others. GAO-17-317. Washington D.C: February 15, 2017.", "summary": "EPA is responsible for reviewing chemicals in commerce and those entering the marketplace. Currently there are more than 40,000 active chemical substances in commerce, with more submitted to EPA for review annually. EPA's IRIS database contains the agency's scientific position on the potential human health effects that may result from exposure to various chemicals in the environment. EPA's IRIS Program, which produces toxicity assessments, has been criticized in the past for timeliness and transparency issues. In response, the IRIS Program committed to making program improvements starting in 2011, which the National Academy of Sciences (NAS) recently commended. TSCA as amended in 2016 provides EPA with additional authority to review both existing and new chemicals and to regulate those that EPA determines pose unreasonable risks to human health or the environment. This report describes (1) the extent to which the IRIS Program has addressed identified challenges and made progress toward producing chemical assessments; and (2) the extent to which EPA has demonstrated progress implementing TSCA. GAO reviewed NAS and EPA documents and interviewed officials from EPA and representatives from two environmental and two industry stakeholder organizations. The Environmental Protection Agency's (EPA) Integrated Risk Information System (IRIS) Program, which prepares human health toxicity assessments of chemicals, has made progress addressing historical timeliness and transparency challenges in the assessment process. Efforts to address timeliness include employing project management principles and specialized software to better plan assessments and utilize staff. To address the need for greater transparency in how the program conducts assessments, IRIS officials and the IRIS Program have implemented systematic review, which provides a structured and transparent process for identifying relevant studies, reviewing their methodological strengths and weaknesses, and integrating these studies as part of a weight of evidence analysis. Since the process improvements were implemented, the program made progress toward producing chemical assessments through May 2018. In June 2018, the EPA Administrator's office told IRIS officials that they could not release any IRIS-associated documentation without a formal request from EPA program office leadership. In August 2018, according to IRIS officials, program office leadership was asked to reconfirm which ongoing chemical assessments their offices needed. In late October 2018, these offices were asked to limit their chemical requests further, to the top three or four assessments. At the same time—4 months after IRIS assessments were stopped from being released—28 of approximately 30 IRIS staff were directed to support implementation of the Toxic Substances Control Act of 1976 (TSCA), as amended, with 25 to 50 percent of their time, according to officials. Then on December 19, 2018, the Office of Research and Development released its IRIS Program Outlook, which provided an updated list of 13 assessments. Eleven of the 13 chemicals on the IRIS Program Outlook were requested by two EPA program offices. A memorandum issued earlier in December, gave no indication of when additional assessments could be requested or what the IRIS Program's workflow would be in the near term. EPA has demonstrated progress implementing TSCA, which was amended in June 2016, by responding to statutory deadlines. For example, EPA finalized rules detailing the general processes for prioritizing and evaluating chemicals, known as the Framework Rules, but three of the four rules have been challenged in court. Environmental organizations have argued, among other things, that TSCA requires EPA to consider all conditions of use in prioritizing and evaluating chemicals, rather than excluding, for example, uses that EPA believes are \"legacy uses,\" for which a chemical is no longer marketed. EPA argued that TSCA grants it discretion to determine what constitutes a chemical's conditions of use. Amendments to TSCA in 2016 increased EPA's responsibility for regulating chemicals and in turn, its workload. As such, EPA is required to prioritize and evaluate existing chemicals by various deadlines over an extended period and to make a regulatory determination on all new chemicals. Senior management told GAO that they were confident that ongoing hiring and reorganization would better position the office that implements TSCA. GAO made recommendations previously to improve the IRIS Program and TSCA implementation. EPA provided comments, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "Of TCJA’s 119 provisions, 86 relate to business and international tax law, ranging in scope from tax treatment of commuter benefits to significant modifications to international aspects of U.S. income tax. See table 1 for select examples of tax law changes resulting from TCJA. While Congress amends the tax code routinely, the time constraints and magnitude of changes within TCJA are less common. According to IRS officials, the last time IRS implemented major tax law changes was in 1986. For TCJA, IRS had a relatively short time frame to implement because the law included many time-sensitive provisions that were either retroactively effective, or immediately effective upon the law’s enactment. To implement TCJA, IRS established working groups to provide project management oversight, coordinate the implementation of TCJA provisions across IRS, and identify priorities, challenges, and risks of the new tax law changes. For instance, TRIO was established in January 2018 as a short-term centralized authority to prioritize, oversee, and coordinate implementation of TCJA, as shown in figure 1. TRIO’s oversight model was based on the working groups convened during IRS’s implementation of Patient Protection and Affordable Care Act. Made up of officials with expertise across IRS, including IRS’s business operating divisions (BOD)—the offices responsible for implementation, oversight, and compliance of tax laws—TRIO was established to temporarily oversee TCJA implementation through the 2019 filing season. TRIO’s objectives for the 2019 filing season included ensuring that taxpayers understood their tax obligations and that IRS could process tax returns, payments, and refunds. In March 2019, TRIO dissolved and transitioned oversight and operations to the BODs. In addition to the working groups that were established to implement TCJA, IRS also relied on its usual practices for implementation of tax law changes, including developing guidance, training employees, and updating technology systems: Determining appropriate guidance for release. IRS interprets the law and develops guidance using a variety of documents and services to communicate its interpretation to help taxpayers understand their tax obligations. IRS guidance includes Treasury Decisions (the formal name for final or temporary tax regulations), which are considered the legally binding interpretation of the statute and IRS’s official position on federal tax law. Treasury generally has 18 months after tax law changes to issue final regulations for them to be retroactively effective to the date of enactment, though there is an exception to prevent abuse. In some cases, Treasury will issue temporary regulations—to provide immediate guidance—prior to issuing final tax regulations. Other forms of guidance include proposed regulations—a step in the regulation development process—revenue rulings, revenue procedures, and notices, among other documents, to provide additional official guidance to taxpayers. IRS also provides taxpayers with a range of other information sources, including frequently asked questions, webinars, YouTube videos, and news releases. Developing guidance. IRS’s Chief Counsel, in coordination with Treasury’s Office of Tax Policy, drafts tax regulations and also works with IRS BODs and the public. Draft tax regulations are circulated throughout IRS and Treasury for review and approval before being published as a proposed tax regulation in the Federal Register. The public is given the opportunity to provide comments, which are analyzed and incorporated as appropriate into another draft of the regulation. The draft tax regulation is again circulated through IRS and Treasury for review and approval, before being published as a final tax regulation in the Federal Register. In some instances, once the draft proposed and final regulations have gone through the IRS and Treasury approval process, regulations may be subject to review by OIRA. OIRA reviews tax regulations that may create serious inconsistencies or otherwise interfere with another agency’s actions, raise novel legal or policy issues, or have an annual non-revenue effect on the economy of $100 million or more measured against a no-action baseline. Conducting stakeholder outreach. In addition to the development of regulations, IRS works with the public to gather feedback, educate taxpayers on published guidance, and inform it of upcoming efforts to provide additional guidance on key areas where IRS and stakeholders require additional clarity. Developing internal policies and procedures. IRS updates its Internal Revenue Manual (IRM)—the official compilation of instructions to staff on the administration and operation of the IRS—with procedures that inform staff of the steps they should take to correctly complete work and administer new tax law changes. Training employees. IRS trains employees to understand revisions in the tax code and ensure they have the tools necessary to manage key priorities such as using data and analysis to improve customer service and enforcement efforts. Modifying Information Technology (IT) Systems. IRS’s Information Technology organization updates the application programs of the tax return intake systems to allow IRS to accept and process tax returns. Generally, IRS captures data from electronically filed (e-filed) tax returns through its Modernized e-File application in a format that can be used for compliance and enforcement purposes. Given the magnitude and short timeline for TCJA implementation, IRS reassessed priorities to implement the law. While some IRS officials said they were largely able to balance TCJA implementation with their other work, other officials from Chief Counsel told us they decreased some field services to taxpayers, and scaled back non-TCJA guidance development. Additionally as a result of TCJA, IRS temporarily postponed some planned work, including some IT work and publication of previously planned taxpayer guidance on health savings accounts, the work opportunity tax credit, and other areas. To meet statutory requirements and best meet taxpayer needs, IRS prioritized 33 TCJA provisions for initial implementation, including 12 business and international provisions as the highest priorities, as shown in table 2. IRS officials said their highest priorities were to implement retroactive provisions because they affected the tax year beginning prior to January 1, 2018—and entirely new provisions. According to IRS planning documentation, in making these decisions, the agency considered the anticipated amount of public scrutiny, as well as the necessary amount of internal collaboration, external stakeholder coordination, and the extent of IT system modifications required to implement. To help taxpayers understand the new tax law and meet their tax obligations, IRS released various types of guidance. Officials told us they aimed to address the most significant questions through early guidance, before answering secondary questions in subsequent guidance. IRS developed comparisons of TCJA with the previous law to help taxpayers understand changes by topic and conducted public information campaigns targeting specific audiences, such as small businesses, to help taxpayers identify the right information and resources to meet their tax obligations. According to IRS officials, one challenge with taxpayers needing to rely on guidance when guidance has not been finalized is that unresolved questions can create uncertainty and guesswork for some taxpayers. While some tax practitioners we spoke with said the release of shorter and earlier information was helpful to provide insight into initial IRS positions on provisions that required immediate instruction, other tax practitioners said that the absence of complete information meant that taxpayers had to file their taxes without certainty. For instance, IRS worked to provide early information to taxpayers on the immediately effective repatriation tax. Repatriation tax payments were due in 2018; however, under the law, taxpayers had the option to pay in installments over 8 years. IRS did not have time to release comprehensive guidance in advance. To provide some early information to taxpayers, IRS instead released three notices (in January, February, and April 2018) and a revenue procedure (February 2018) to help taxpayers understand topics such as whether they were subject to the tax and their tax liability. In May 2019, the Treasury Inspector General for Tax Administration (TIGTA) reported that the short implementation time frame did not leave taxpayers sufficient time to understand the guidance and comply with their resulting tax liability. TIGTA reported that while IRS made a reasonable effort to inform taxpayers of the requirements under the repatriation tax, some taxpayers overpaid their first-year repatriation tax installment without the knowledge that IRS would not be refunding excess remittances of installments. The initial repatriation tax information issued mid-filing season instructed taxpayers to make two separate payments— one for their income tax liability and one for their repatriation tax liability, language that was later clarified. Subsequently, IRS announced that excess payments would be applied to the unpaid portion of the taxpayer’s liability and that IRS was legally precluded from issuing a refund of any excess remittances. TIGTA reported that for the 2017 tax year, 115 taxpayers filed repatriation tax refund claims—amounting to $2.8 billion—which, according to TIGTA, indicated that these were unintended overpayments. TIGTA recommended that IRS take steps to inform taxpayers when the next payment is due and how their excess payment was applied to their repatriation tax balance. According to TIGTA, IRS agreed to these recommendations and is taking steps to implement them. Because of the magnitude of the changes and the immediate effective dates of many TCJA provisions, Chief Counsel collaborated earlier and more frequently with IRS BODs to implement TCJA. IRS officials said these enhanced collaborative efforts were best practices and critical to timely TCJA guidance development. Although there are some guidelines in the IRM for intra-agency coordination, IRS has not identified instances when this enhanced collaboration would benefit guidance development or taken steps to document these parameters to assure consistency and accountability. Based on our analysis, we found that IRS officials leveraged several key practices for implementing collaborative mechanisms to support TCJA implementation, including identifying leadership roles and responsibilities, identifying relevant participants, and using resources to facilitate collaboration, which we identified in prior work. For instance, IRS formed TRIO to manage TCJA implementation and centralize accountability and decision-making. Additionally, Chief Counsel’s earlier and more frequent work with the BODs allowed for participants with appropriate skills and expertise to contribute to guidance development and highlight potential enforcement concerns. This collaboration included weekly, and in some instances daily, meetings for participants to provide implementation status updates. Further, IRS developed joint project documents and leveraged collaborative technologies to track and manage TCJA implementation and facilitate sharing across the agency. IRS officials stated that there were several benefits to this enhanced collaborative approach, including: more efficient and effective development of comprehensive regulations, for instance guidance for the QBI provision; faster decision-making on time-sensitive regulations; earlier identification of tax administration and enforcement concerns; mitigation of potential enforcement challenges, such as narrowing the definition of specified trades and businesses on the QBI deduction; and ability to begin compliance planning earlier. Our prior work on interagency collaboration mechanisms and the Standards for Internal Control in the Federal Government identify areas for agencies to improve and sustain collaboration. For instance, we identified that frequent communication can help facilitate working across agency boundaries and that articulating agreement in formal documents that are regularly updated and monitored can strengthen commitment to working collaboratively. In addition, federal internal control standards state that agencies should ensure stakeholders from different parts of an organization communicate to help the agency fulfill its mission. Chief Counsel officials acknowledged the value of this enhanced collaboration but as of December 2019 had not identified or documented criteria for when this collaborative approach would benefit guidance development and help achieve agency goals. Chief Counsel officials said that the guidelines as written provided flexibility in determining when to collaborate early with other offices during TCJA implementation. In addition, officials said that the value of collaboration depends on the scope and complexity of a tax law change and the decision to use earlier and more frequent collaboration would need to consider tradeoffs and other considerations such as other IRS priorities and the effects of pulling employees away from other activities. However, IRS officials described the enhanced collaboration used throughout TCJA implementation as unprecedented and key to successful implementation, indicating that identifying the situations when this earlier and more frequent collaboration would make sense and updating relevant documentation to reflect this could benefit IRS guidance development. Documenting the parameters and procedures for enhanced collaboration practices would better position IRS to be prepared to use enhanced collaboration during implementation of complex or time- sensitive changes to the tax code. For example, enhanced collaboration may help to identify and mitigate potential administrative effects of regulatory design decisions, potentially helping IRS identify more cost- effective alternatives within the limits of available resources. These potential benefits are also supported by our past work on regulatory design during the rulemaking process. Specifically, we found that it is important for agencies to consider enforcement and compliance issues during regulation development because different design choices have implications for future enforcement and compliance efforts. While developing regulations to help implement and administer TCJA provisions, Treasury and IRS made discretionary decisions in the regulatory development process that have meaningful effects on taxpayers’ tax liability and government revenue collection that were not included in their analysis. Changes to tax liability have distributional consequences, as taxes transfer money from taxpayers to the government, but do not directly affect the total resources available to the country. These distributional effects are one element that should be recognized during the regulatory development process, along with costs and benefits of the regulations. While we found that, among the provisions we looked at more deeply, Treasury’s analyses did recognize some costs and benefits related to factors such as administrability, compliance costs, and economic distortions, Treasury’s analyses did not generally assess the distributional effects, including effects on tax revenue collection, the regulations had as a result of changes in tax liability. As part of the regulatory development process, Treasury and IRS must adhere to Executive Order (E. O.) 12866, which establishes standards for regulatory planning and review. E.O. 12866 instructs agencies to select regulatory approaches that maximize net benefits, including economic, distributive, and equity effects, unless a statute requires another regulatory approach. Any regulation that is determined to be significant must be submitted to OIRA for review, along with an analysis of the costs and benefits of that regulation. However, until 2018, Treasury’s and IRS’s tax regulations were not regularly subjected to analysis and review under E.O. 12866. In many cases, tax regulations were deemed not significant under E.O. 12866, and as a result, Treasury and IRS did not perform regulatory analyses and they were not reviewed by OIRA. Some tax regulations were also exempt from OIRA review, which was otherwise required under E.O. 12866 based on an agreement between OMB and Treasury. However, E.O. 13789, signed in 2017, instructed the Secretary of the Treasury to reconsider the scope of that exemption, and in April 2018, Treasury and OMB signed a Memorandum of Agreement (MOA) subjecting certain tax regulations to OIRA review. In accordance with the MOA and the requirements in E.O. 12866, for regulations deemed significant, Treasury is responsible for conducting and producing an analysis of the impact of the regulations, including an assessment of costs and benefits. Tax regulations with an anticipated annual non-revenue effect of $100 million or more are deemed economically significant, and are subjected to this additional analysis. Under the MOA, Treasury was allowed a 12-month transition period to obtain reasonably sufficient resources to meet the additional requirements for economically significant regulations. The transition period expired in April 2019, and any new regulations will be subjected to these additional analyses where applicable. E.O. 12866 and OMB Circular No. A-4, a guide developed by OMB for agencies to perform regulatory analyses required by E.O. 12866, emphasize that agencies should assess the costs and benefits of proposed regulations. In some cases, regulations may transfer money from one group to another, creating no net costs or benefits to society as a whole, but nonetheless affecting those who have been affected by the transfers. When regulations have this effect, they are said to have a distributive impact on society, and both E.O.12866 and OMB Circular A-4 instruct agencies to consider distributive effects. Because revenues raised through taxation are transfers and are not costs or benefits to society, OMB Circular A-4 instructs agencies to develop a description of the distributional effects of a regulation that is separate from the costs and benefits. Such an analysis should recognize the effects of the regulation across the population and the economy, divided up in various ways, such as income groups, race, sex, industrial sector, or geography. Treasury’s and IRS’s significant proposed and final rules used to implement TCJA included a section analyzing the impact of the regulations; however, we found these analyses generally overlooked the distributional effects of the regulations arising from changes in tax liability and revenue collection. The illustrative examples below from TCJA regulations highlight the potential effects of Treasury’s and IRS’s regulatory decisions on tax liability and how those were reflected in Treasury’s analysis. Eligibility for QBI deduction for real estate and insurance brokers. The QBI deduction provides a deduction of up to 20 percent of QBI, but depending on a taxpayer’s taxable income, a specified service trade or business (SSTB) may not be a qualified trade or business and therefore may not produce QBI. The statute defines SSTBs as trades or businesses within a list of broadly-identified fields. Treasury and IRS determined that guidance clarifying the types of trades or businesses that would be considered to be within the listed fields was needed. As one example, the statute specified that “brokerage services” are considered an SSTB, but Treasury and IRS regulations further specified that “brokerage services” was limited to securities brokers, while other brokerage services, including real estate brokers and insurance brokers, were explicitly excluded from the definition of “brokerage services.” The choices Treasury and IRS made when providing additional guidance on SSTBs will significantly affect the tax burden and revenue collected from certain businesses. In its analysis of its decisions regarding the definitions of SSTBs, Treasury stated that articulating which business activities were or were not considered SSTBs would provide clarity to taxpayers and prevent similarly-situated taxpayers from behaving differently, which could potentially create economic inefficiencies. Treasury did not address the fact that decisions about which business activities would be considered SSTBs would affect eligibility for a 20 percent deduction, and would affect the distribution of resources between certain taxpayers and the federal government. According to data from IRS’s Statistics of Income on sole proprietorships—one of several business structures that can earn a QBI deduction—categories representing insurance agencies and brokerages, and offices of real estate agents, brokers, property managers, and appraisers recorded more than $35 billion in net income in 2016. The precise effect of not being categorized as an SSTB depends on the specific circumstances of the individual businesses, but given the magnitude of their annual net revenue, excluding real estate and insurance brokers from the definition of SSTB could lower their collective tax burden by billions of dollars annually. This could result in a reduction in federal tax revenues compared to the regulatory alternative of considering these sectors to be SSTBs. End date for opportunity zones. An investor who invests capital gains in a Qualified Opportunity Fund, and maintains that investment for at least 10 years, is eligible to make an election at the time of sale that would render such gains no longer taxable. TCJA’s statutory language did not specify an end date for investors to make this election, or a point at which taxpayers must dispose of investments in opportunity funds and recognize future capital gains to be taxed. IRS’s October 2018 proposed regulations for opportunity zones stated that investors will have until December 31, 2047, to dispose of investments and make this election. The decision to set an end date of December 31, 2047, was one of four approaches discussed in the proposed regulations. The other options considered were to offer no further guidance on this issue, to specify no end date to elect the gain exclusion, and to allow the election until December 31, 2047, but without disposition of the assets. In its analysis of this decision, Treasury considered how providing clarity would help taxpayers make more efficient investments in opportunity zones. Treasury also considered how forced dispositions could lead to economic inefficiencies, while a longer time horizon could lead to greater investment, but more administrative costs. Treasury did not, however, assess how the different decisions would influence the ultimate tax liability of investors. The determination of a disposition date can have a potentially large effect on tax liability. For example, if a taxpayer invested $1 million into an opportunity fund in 2019, and that grew at a 7 percent rate, it would be worth approximately $2 million after 10 years, $3.9 million after 20 years, and $6.6 million in 2047. Under Treasury’s and IRS’s regulations, such capital gains—$5.6 million in this example—would be exempted from taxation. We found that in the course of developing regulatory impact analyses for TCJA regulations, Treasury generally excluded any analysis of distributional effects due to changes in tax revenue collection. In the examples above, Treasury’s decisions would significantly affect tax liability for certain taxpayers, which were not reflected in Treasury’s analyses of the regulations. Treasury officials did not conduct distributional analyses related to revenue effects because in their view, the MOA instructed them to focus only on non-revenue effects and superseded E.O. 12866. This view is reflected in Treasury’s guidance to staff on how to conduct regulatory analyses. Specifically, Treasury’s internal guidance instructs staff to conduct distributional analyses, describing how benefits, costs, and transfers are distributed among subpopulations. This guidance further states that staff should not include transfers of revenue to the government, and Treasury officials told us that they did not think they should include any analysis of these effects in their regulatory impact analyses. However, Treasury’s understanding that revenue effects should be excluded from its analyses is inconsistent with the MOA and OIRA’s position. While Section 1 of the MOA between Treasury and OMB excludes revenue effects for the purposes of determining whether or not a regulation is economically significant, and thus subject to OIRA review, that limitation does not appear elsewhere in the MOA, and the MOA does not state that revenue should be excluded from all analysis. OIRA officials told us that that all agencies, including Treasury, are subject to the same requirements of E.O. 12866, and that outside of the MOA, OIRA had no agreements with Treasury that would otherwise modify the requirements. OIRA officials we spoke with reiterated that all agencies, including Treasury, should generally analyze the distributional impact of their regulations, and OMB’s guidance identifies changes in tax revenue as an example of a transfer that would have a distributional impact. OIRA officials stated that they recognize conducting these analyses was a new procedure for Treasury, and that Treasury officials were still learning how to apply the analytical framework in Circular A-4. Treasury’s internal guidance for conducting regulatory impact analyses is inconsistent with the standards in E.O. 12866 and OMB Circular A-4 that all agencies are expected to follow. Considering distributional effects related to tax revenue in the analyses would improve transparency surrounding how decisions made by Treasury and IRS affect various groups across the population. Robust analysis ensures that regulatory choices are made after appropriate consideration of the likely consequences, and provides transparency to the public and policymakers. Our prior work emphasizes the importance of transparency in the rulemaking process, and specifically that a regulatory impact analysis consistent with E.O. 12866 and OMB Circular A-4 provides a systematic framework for identifying and assessing the economic tradeoffs associated with alternative regulatory choices. By excluding analyses of distributional effects due to changes in tax liability, including effects on tax revenue collection, Treasury and IRS risk making regulatory decisions that have significant economic effects without fully understanding the consequences of their decisions. Further, the consequences of Treasury and IRS decisions and the tradeoffs they considered are not transparent to the public without an acknowledgement of the distributional effects of tax revenue changes. A lack of full information may also inhibit OIRA’s ability to effectively review the regulations and limit decision makers’ understanding of the effects of a law. Data-reliability issues in IRS’s documents for tracking implementation of TCJA’s business and international provisions made it challenging to characterize both the scope and status of implementation activities. However, based on IRS data we corroborated with publicly-available information (e.g., published guidance), we determined that IRS has made considerable progress in implementing many of TCJA’s business and international provisions through issuing guidance, updating IT systems, and training IRS staff. Given the magnitude of changes and near immediate effective dates, tax professionals we interviewed generally spoke favorably about IRS’s pace in developing TCJA guidance and the quality of the guidance developed. We found errors and inconsistencies in IRS’s documentation used to track TCJA implementation. While we did not find errors and inconsistencies in the majority of IRS’s TCJA implementation tasks, we did identify multiple instances of inaccurate recording of the task status, conflicting information in separate tracking documents, and several other miscellaneous errors. Examples include: Seven TCJA provisions and six updates to the IRM were inaccurately identified as complete in the tracking document for the responsible BODs, potentially delaying work on implementation. IRS had cancelled IRM updates for five IRM sections for one provision, but tracking documents across multiple BODs did not accurately capture this fact, which could result in a misallocation of staff and resources. IRS officials could not verify whether all tasks included in TRIO’s Enterprise Integrated Project Plan (EIPP) tracking document had been carried over to the new tracking documents following the dissolution of TRIO, increasing the risk of previously planned tasks mistakenly being left incomplete. At least 22 unique identifiers used to track tasks across iterations of TRIO’s EIPP tracking document were inconsistent between updates, limiting IRS’s ability to accurately track changes in guidance planning over time. The lack of consistency and accuracy across IRS’s tracking documentation is not in accordance with Standards for Internal Control in the Federal Government. These standards direct management to use quality information to make informed decisions and evaluate the entity’s performance or efficiency in achieving key objectives and addressing risks. Changes in IRS’s method of monitoring TCJA implementation status contributed to the data-reliability issues we identified. Early TCJA implementation efforts involved close coordination among multiple internal organizations. When TRIO was responsible for coordinating implementation efforts, it maintained a unified tracking system as part of its coordination management. However, when TRIO was disbanded, BODs and other IRS organizations used several different methods of tracking implementation status. According to IRS officials, in some cases, these new methods were not compared with TRIO’s documentation to ensure all necessary tasks were carried over. Additionally, the implementation tracking tools used by these organizations were not uniform in data included, format, or the frequency with which they were updated. These issues may impede the ability to coordinate internally and to monitor overall implementation status. IRS officials stated that these inconsistencies did not pose obstacles to implementation, and that the IRS organization with overall responsibility for a given task was accurately tracking implementation status. While IRS officials said the inconsistencies did not impact implementation, developing a process or modifying the existing process to accurately and consistently track the implementation status of provisions could improve IRS’s ability to prioritize resources and coordinate implementation efforts. For example, such tracking could help prevent misunderstandings regarding the implementation status of a provision that could lead management to reallocate resources away from ongoing implementation tasks. Further, it could help ensure IRS’s implementation efforts are efficient, as each BOD would have the same information to help coordinate prioritization efforts. While TCJA implementation is a one-time effort, IRS officials stated that efforts will extend beyond a decade into the future, as some provisions (such as opportunity zones) may require further guidance as key deadlines are reached. Additionally, IRS has identified the need for further guidance or implementation tasks as implementation has progressed, and the timeline for full implementation may be extended as IRS receives new information or observes changes in taxpayer behavior. Further, IRS is implementing provisions of a new law reforming aspects of the agency and may face similarly extensive implementation projects in the future. The Taxpayer First Act, signed into law on July 1, 2019, calls for several IRS reforms, including changes to rules related to enforcement as well as modernizing IRS structure and technology, among other things. Management may be able to identify issues with, or improvements to, the implementation process using quality information on implementation status. By improving the ability to monitor and evaluate implementation progress, IRS will be better equipped to evaluate existing implementation processes. IRS also will be better positioned to effectively implement significant tax law or organizational changes in the future. IRS has attempted to determine the amount of guidance required for TCJA implementation throughout the implementation process, but the amount of guidance has fluctuated for several reasons. For instance, in July 2018 IRS planned to issue 40 proposed regulations and 35 final regulations by December 2021 to implement the 86 business and international provisions. But by the end of the 2019 fiscal year, IRS planned to issue 53 proposed regulations and 51 final regulations by February 2022. According to IRS officials, they initially expected to issue less guidance than now planned, but as work progressed, they discovered they would need to issue more guidance or issue some guidance through multiple regulations to address taxpayer comments and inquiries. Conversely, in some cases IRS determined that some guidance initially planned was no longer necessary after further consideration. As of the end of fiscal year 2019, IRS Chief Counsel reported that it had issued 90 pieces of guidance and was developing another 43 to implement the 86 business and international provisions of TCJA. Overall, as of the end of fiscal year 2019, IRS publicly issued approximately half of planned official guidance. As shown in figure 2, for the 12 provisions that IRS identified as high-priority, the agency issued 13 of 19 planned proposed regulations and three of 18 planned final regulations. IRS missed internal target dates for issuing 10 guidance documents initially targeted for publication by the end of the fiscal year, including three final regulations. According to IRS officials, several factors affected IRS’s ability to issue guidance within planned time frames: Ambitious project planning. Scheduled completion dates for some tasks were “aspirational” and developed early in the implementation process. Officials stated that they understood from the beginning of implementation that their planned dates might change, and that they did not expect there to be any impact on taxpayers. Revised regulatory review process. As discussed earlier in this report, beginning in April 2018, OIRA began subjecting more tax regulations to further review as agreed to in the MOA between OMB and Treasury. Based on our analysis, from July 2018 to September 30, 2019, OIRA took an average of about 38 calendar days to review 25 TCJA business and international regulations. See appendix III for a table of all TCJA regulations relating to business and international provisions reviewed by OIRA and associated review times. Partial lapse in appropriations. According to IRS officials, a partial lapse in appropriations from December 22, 2018, through January 25, 2019, contributed to implementation delays. For example, IRS officials estimated that the issuance of final regulations for the qualified business income deduction and repatriation tax was delayed 1 to 2 weeks. While IRS was generally able to continue working on TCJA implementation tasks, it had to allocate some resources towards unplanned administrative tasks during this period. During a lapse of appropriations, the Antideficiency Act generally restricts agencies from continuing operations funded by annual appropriations. However, Congress passed a separate 2-year appropriation for IRS to perform TCJA implementation activities. Some IRS personnel that would otherwise have been furloughed were instead able to continue TCJA implementation work through the use of this special appropriation. Additionally, IRS had to develop justifications for the Federal Register to publish TCJA regulations during the lapse in appropriations that it would not have had to do in the absence of a partial lapse in appropriations, which reduced available resources for implementation tasks. IRS officials also stated that they faced issues working with partners at Treasury and OIRA. Treasury’s Lapse in Appropriations Plan states that Office of Tax Policy staff could work on policies to restore appropriations and developing revenue estimates for pending appropriations negotiations, but does not include work on TCJA. Further, while OIRA continued regulatory review in certain circumstances, approximately 67 percent of OMB’s staff was furloughed. Of the remaining guidance, IRS plans to issue 13 of the remaining final TCJA regulations related to business or international provisions by December 31, 2019. IRS plans to issue 12 final regulations in 2020, three in 2021, and one in 2022. It has not determined publication dates for 14 final regulations. To implement TCJA, IRS has provided a substantial amount of written guidance. Between TCJA’s enactment and the end of fiscal year 2019, IRS published 1,383 pages of guidance related to TCJA’s business and international provisions in the Internal Revenue Bulletin, out of a total of 4,064 pages published during that period. By comparison, from 2013 to 2015, IRS published approximately 2,000 pages of guidance annually. IRS also issued more than 115 pieces of business- and international- related products, including news releases, frequently asked questions, virtual webinars, YouTube videos, and targeted publications, such as the example in figure 3. Tax practitioners we spoke with were generally favorable about IRS’s pace in developing TCJA guidance and the quality of the guidance developed. For example, they generally stated that IRS’s multi-pronged approach to providing both official guidance and other information sources was helpful and allowed practitioners to understand of the likely impacts of tax reform prior to the release of final regulations better. Additionally, Tax Notes—a well-regarded publisher of a collection of professional tax products—named the IRS’s tax reform regulatory team as Person of the Year for 2018 for issuing many TCJA regulations in less than a year. According to IRS officials, IRS’s Information Technology organization completed all TCJA tasks that the organization agreed to complete prior to the opening of the 2019 filing season, including updates to electronic forms and the underlying technology IRS uses to receive returns. According to IRS officials, they completed these tasks by prioritizing TCJA work over other tasks and modifying its routine processes for implementing IT changes. IRS’s Information Technology organization also worked with the BODs to determine which data were most important to have in Extensible Markup Language (XML) format, which is more accessible than data in Portable Document Format (PDF) format. While BOD officials requested programming for TCJA-related requirements that would necessitate that the Information Technology organization enable forms in XML format, IRS’s Information Technology organization ultimately determined that it could not deliver updates for all TCJA affected forms in advance of the 2019 filing season, and forms where IT could not deliver updates in XML format would be implemented in PDF format. For example, according to IRS officials, they prioritized having tax year 2018 XML data for the repatriation tax because this tax was immediately effective for tax year 2017, had a short-lived time frame, and presented challenges for monitoring. BODs requested that all affected forms be converted for the 2020 filing season. To further facilitate the implementation of TCJA-related IT tasks, IRS officials told us that they designed a framework to streamline communication between the Information Technology organization, subject matter experts, and the IRS BODs. These sessions enabled staff to work through and identify IT requirements in real time, rather than requiring Information Technology organization staff to wait until the BODs submitted a work request to begin work. As of October 2019, IRS’s Information Technology organization had identified an additional 124 TCJA-related tasks for the 2020 filing season. Officials expected to complete these tasks prior to the filing season. According to IRS documentation and officials, these tasks include updating underlying programming of IT systems to capture tax return information in a way that can be more easily used for compliance purposes, updating critical IT systems, and implementing error resolution codes to correct some mistakes on submitted returns. While the Information Technology organization had not yet approved all work and some TCJA requested work was pending analysis or approval in its work tracking spreadsheet, according to Information Technology organization officials, they are aware of the work and proceeding with implementation for the Modernized e-File application, the system used to file returns electronically. According to IRS documentation, the agency has begun training staff on several TCJA provisions, including high-priority provisions, and plans to deliver additional training in 2020. According to IRS, workforce training is a critical component of tax law implementation to ensure that the workforce is equipped to identify and address potential audit issues associated with the new tax law provisions as well as to provide the appropriate level of taxpayer service. According to IRS documentation and officials as of the end of fiscal year 2019, the agency delivered training for business and international TCJA changes in multiple formats, including virtual and in-person training. These sessions have addressed at least 28 of the 69 business and international provisions identified as requiring training. IRS began larger scale in-person training in August 2019 and is developing content for further training in fiscal year 2020. The in-person training primarily addresses high-priority TCJA provisions such as QBI deduction, opportunity zones, the repatriation tax, the limitation on the interest deduction, the tax on global intangible low-taxed income, and the base erosion and anti-abuse tax. IRS officials said that their training efforts have been a major undertaking and that they focused their training efforts on high priority provisions and provisions that affected a large number of taxpayers. Some of these training sessions will culminate in an interactive risk assessment exercise. IRS planned to train about 8,500 employees in these sessions. IRS plans to continue TCJA training in 2020 as IRS finalizes regulations. According to SB/SE’s implementation tracking documentation, it plans to complete training by the end of 2020. According to LB&I documentation, it plans to hold virtual training in March, May, and June 2020 addressing, among other things, some high priority provisions, including the repatriation tax and base erosion and anti-abuse tax. Treasury did not issue all planned final regulations within the 18 months the agency generally has to issue regulations retroactive to the date of a law’s enactment or before taxpayers were required to file tax returns, which has the potential to be significant for both taxpayers and IRS. Specifically, of the 51 planned final regulations to implement TCJA business and international provisions, Treasury issued five within the 18- month time frame. Treasury also issued one temporary regulation within this time frame. Treasury did not release any final regulations for eight of its 12 priority provisions. As discussed earlier in this report, taxpayers and other stakeholders appreciated the supplemental information Treasury provided in the absence of final regulations. According to IRS Chief Counsel officials, however, a significant effect of relying on proposed regulations rather than final regulations is uncertainty. In instances where Treasury has yet to issue regulations or any other guidance, taxpayers must rely on the statutory language to understand the law. For example, LB&I officials said that taxpayers may not be able to correctly calculate tax for foreign branch losses because IRS included limited information on related forms as final guidance had not yet been issued. Similarly, tax practitioners we interviewed cited several provisions in need of additional guidance and identified challenges associated with those provisions that have the potential to affect taxpayers’ ability to comply with the law. Challenges identified by tax practitioners we interviewed included confusion regarding and challenges related to the definitions of “related party” and “interest” in the proposed regulations for the limitation on the deduction for interest and difficulty for individuals and corporations to understand and comply with international changes given the interdependence of several of the international provisions. A September 2019 Treasury Inspector General for Tax Administration (TIGTA) report also raised concerns related to taxpayers’ ability to comply with the international provisions. Further, proposed regulations are subject to change when Treasury finalizes them, which could create additional burdens for taxpayers. For example, Treasury’s proposed rule—issued in August 2018—for determining whether a foreign corporation’s earnings are subject to the repatriation tax was modified from a 5-percent threshold for application of the special attribution rules relating to partnerships and trusts to a 10- percent threshold under the final regulations—issued in February 2019. Because the repatriation tax was immediately effective, taxpayers needed to pay their tax liability, or make installment payments towards that liability, before IRS was able to finalize its regulations. Some taxpayers who would have been subject to the tax had the proposed regulations been finalized without change may not be subject to this tax because of changes between the proposed and final regulations, and any payments towards repatriation tax liability would no longer be needed. According to IRS officials, taxpayers who initially made repatriation tax payments but are not subject to the tax under the final regulations will need to file an amended return to receive a refund of their repatriation tax payments. The lack of finalized guidance can also create challenges for IRS in the agency’s efforts to ensure compliance with the new law. For example, LB&I officials told us they have identified form changes needed related to at least one TCJA provision for which Treasury had yet to issue final regulations, but they need to be mindful when proposing form changes because final regulations could require additional form changes and could require rework. Further, in September 2019, TIGTA reported that the lack of final of final guidance delayed training for LB&I staff, which could hinder LB&I’s ability to respond to emerging compliance risks. According to IRS Chief Counsel officials, if IRS believes that a rule articulated in proposed regulations under a statutory provision is correct, it may proceed to enforce that interpretation of the statute in the absence of final regulations. However, in the event of litigation, the interpretation set forth in the proposed regulations would not carry the same weight as final regulations. IRS may also face additional challenges administering the law in instances where the agency has yet to issue proposed regulations. Treasury can issue final regulations that are retroactively effective to the proposed regulations. As of the end of fiscal year 2019, Treasury had not issued 27 planned proposed regulations for business and international provisions. Generally final regulations not issued by the end of calendar year 2019 would not be effective until 2020. According to TIGTA, if IRS makes substantial changes to the proposed regulations, Treasury and IRS may decide not to apply those revisions retroactively to the date of the proposed regulations. While Treasury was unable to issue all final regulations within the 18- month time frame and before taxpayers needed to begin filing tax returns affected by TCJA changes, IRS took actions to mitigate the potential impact of the lack of final guidance. According to IRS officials, they prioritized which regulations needed to be issued to be retroactively applicable to the date of the law’s enactment. For example, Treasury’s QBI deduction regulations included anti-abuse rules to prevent taxpayers from being able to engage in transactions that will artificially increase their deduction. Treasury’s repatriation tax regulations also included rules preventing taxpayers from being able to take actions to reduce their repatriation tax liability. Further, in one instance, Treasury issued a temporary regulation in a situation where Treasury did not have time to issue proposed and final regulations to prevent abuse of TCJA changes related to a deduction for dividends received from certain foreign corporations. We identified 11 business and international provisions where TCJA’s statutory language either required or authorized additional information reporting to administer and enforce them. These include the QBI deduction, repatriation tax, and base erosion and anti-abuse tax. TCJA changes also enabled IRS to address a prior reporting gap related to foreign branch activity that will help with compliance and enforcement efforts, according to LB&I officials. As shown in the examples below, in some instances, the statute did not include an information reporting framework to enforce provisions, and IRS has taken some steps to mitigate information reporting gaps. Limitation on interest deduction. Tax practitioners we interviewed told us that they doubted that IRS would be able to verify information related to controlled foreign corporations that are subject to the limitation of business interest expense because there are limitations on information reporting from other countries. According to IRS officials, the statute made substantial changes to this code section and did not correspondingly include a framework for IRS to require information reporting. IRS is taking mitigation actions to help ensure compliance despite the lack of information reporting framework. For example, according to officials, IRS has the authority to require information from taxpayers and developed a new form to collect information needed to ensure taxpayer compliance with this change. In addition, IRS is planning to make changes to another form to help with compliance efforts. Opportunity zones. While the statute did not grant IRS specific authority to require information reporting for opportunity zones—a tax expenditure that is intended to spur economic growth in low-income areas—IRS has general authority to require information reporting and plans to require and use information reporting to ensure compliance with this provision. As shown in table 3, IRS plans to use information reported on four forms. Taxpayers who invest in qualified opportunity funds may qualify for potentially large benefits that are time dependent. When taxpayers initially invest eligible capital gains in qualified opportunity funds, they can defer the tax due on those gains until the earlier of 2026 or when taxpayers dispose, in whole or in part, of (e.g., sell or exchange) those investments. Specifically, taxpayers receive an increase in the basis of their investment in the qualified opportunity fund if they hold the investment at least 5 years and an additional increase in their basis if they hold their investments an additional 2 years. Taxpayers who hold investments at least 10 years can elect to have their investments valued at the fair market value when they dispose of the investments, and thus would not need to pay taxes on any gains on their initial investments. IRS plans to use taxpayer-reported information and possibly some fund-reported information on the forms listed above in table 3 to identify taxpayers who have invested in qualified opportunity funds to confirm eligibility for tax benefits for investing in and holding those investments in qualified opportunity funds. In other instances, third-party information is not available for IRS to corroborate taxpayer-related information. For example, above certain income thresholds only businesses engaged in an eligible trade or business qualify for the QBI deduction and this information is self- reported. Our past work has found that one of the important factors contributing to the tax gap is the extent to which information is reported to IRS by third parties. For example, according to 2011–2013 IRS data, for income types where there is little or no third-party information reporting (e.g., business income), taxpayers misreported more than half of this income. Without reliable information reporting, IRS will likely need to conduct labor-intensive audits, such as correspondence or face-to-face audits, to ensure compliance with certain TCJA provisions. The potential need to conduct labor-intensive audits could create challenges for IRS given recent trends in audit rates and staffing reductions. Specifically, IRS audit rates of large corporations with assets of $10 million or greater declined from 17.7 percent in fiscal year 2011 to 7.9 percent in fiscal year 2017. We previously reported that IRS’s staffing has declined each year since 2011, and has significantly reduced enforcement activities. In September 2019, TIGTA reported LB&I had difficulty hiring personnel with the skills needed for TCJA implementation. This could limit IRS’s ability to conduct correspondence or face-to-face audits to ensure taxpayer compliance, including TCJA provisions. LB&I and SB/SE officials expressed their confidence in IRS’s ability to audit TCJA provisions sufficiently. SB/SE has developed compliance plans for TCJA provisions identified as having the potential for fraud. SB/SE officials said TCJA work will be prioritized and SB/SE can use some filtering to help identify noncompliance. For example, regarding the QBI deduction, they said IRS may be able to identify returns that need further review based on tax return data. According to LB&I officials, they planned to hire an additional 600 staff, including about 300 revenue agents by the end of fiscal year 2019, and as of the end of the fiscal year, LB&I had selected 430 applicants to hire to help with compliance and enforcement efforts. Revenue agents are of particular importance to IRS’s enforcement efforts as they conduct audits of tax returns. In March 2019, we reported that IRS has skills gaps within its revenue agent workforce, and the agency was taking action to address those gaps. For example, the agency established communications with revenue agents to increase awareness about detail and developmental opportunities, and was developing a plan for more effectively including revenue agents in management training. We recommended that IRS take actions to reduce skills gaps among revenue agents, including developing schedules for skills assessments and reporting on agency efforts to close those gaps. IRS agreed with our recommendation and, as of December 2019, IRS plans to report on efforts to close skills gaps among revenue agents by December 2021. Because IRS had not yet updated all systems prior to accepting tax year 2018 (filing season 2019) returns, IRS was not able to capture all return information in XML format—a format that allows for greater accessibility and analysis. According to IRS documentation and officials, the agency was unable to obtain Extensible Markup Language (XML) data for 11 provisions that LB&I and SB/SE had requested for tax year 2018, including certain high-priority provisions. Instead, according to IRS officials, the agency captured this information in PDF, which is challenging for officials to use for data analytics and trend analysis. According to IRS officials, examiners will be able to view the PDFs and use that information if the return is selected for audit. Officials also told us they have other ways to select returns for audit in the absence of XML data. While the agency does not have any agency-wide plans to retroactively convert PDF data to XML data, which could help with compliance analytics and planning, IRS is capable of conducting this work. For example, IRS staff could transcribe, or manually enter, selected information from returns filed on paper into IRS’s IT systems to process these returns. Additionally, Information Technology organization officials told us they could develop a program to convert PDF forms to an XML format, if the effort is deemed a high priority. Converting data into usable formats for compliance purposes would be consistent with IRS’s strategic plan and Standards for Internal Control in the Federal Government. IRS’s strategic plan includes a strategic goal to advance data access, usability, and analytics to inform decision-making and improve operational outcomes. Specifically, IRS is to use analytics to improve enforcement efforts and maximize learning from tests and data. According to Standards for Internal Control in the Federal Government, agencies should use quality information to achieve their objectives. As part of this, agencies should obtain data and process these data into quality information. LB&I officials said they are taking steps to convert their PDF data into useable data for compliance purposes. According to LB&I officials, they identified which provisions’ data would be useful to retroactively transcribe and they are coordinating with other parts of IRS to complete the transcription. They identified the data on forms related to certain new TCJA provisions as a higher priority for transcribed data. According to officials, they then coordinated with various IRS offices, including the Office of Research Applied Analytics and Statistics, that have the capability to use optical character recognition technology to convert certain forms for these TCJA sections into a more useable format. Statistics of Income, a division within the Wage and Investment Division, is providing clerical staff to perform data validation on the converted data. According to LB&I officials, LB&I plans to use this information to help develop filters and compliance models and it will enable them to conduct analysis earlier than planned because they had not expected to have access to this data. Unlike LB&I, SB/SE had not reviewed the costs and benefits of converting PDF forms for their provisions to determine which PDF forms, if any, would be a good use of IRS resources to convert to XML format to help with compliance planning. According to SB/SE officials, they did not know IRS had the capacity to retroactively convert PDF data to XML format and were unaware of LB&I’s efforts to convert select TCJA PDF forms to useable data. Assessing the costs and benefits of converting PDF data to a more useable format, such as XML format, would be consistent with OMB guidance on using cost-benefit analysis to support agency planning efforts. OMB provides guidance to agencies for conducting economic cost-benefit and cost-effectiveness assessments that promote efficient resource allocation through well-informed decision-making. These assessments should consider different alternatives to meet program objectives along with a discussion of costs and benefits. For provisions where IRS does not have XML data, IRS may not be able to adequately identify both intentional and unintentional compliance risks and may be missing opportunities to better ensure compliance with and enforce TCJA provisions. For example, we previously reported that without comprehensive transcribed data, examiners cannot immediately access and review all data reported on tax returns, which burdened taxpayers as well as made examiners less efficient in doing their jobs. According to IRS officials, retroactively transcribed data would be helpful to SB/SE for compliance planning and enforcement efforts, especially for at least one TCJA provision. Further, taxpayers may think they are in compliance and may not be alerted to their errors until IRS has data stored in a format that can be analyzed more easily. Similarly, in October 2011 we reported that IRS said that having more tax return information available electronically, such as through transcription, would reduce burdensome examinations for compliant taxpayers, as well as facilitate enforcement efforts, make case resolution faster, and increase compliance revenue. However, without an analysis of the costs and benefits of retroactively converting PDF data to XML data, SB/SE cannot determine which PDF forms would likely yield benefits that would outweigh the costs of this effort. Management also cannot make an informed decision as to which PDF data would benefit SB/SE if converted to XML format without this information. While IRS may not have complete data on the potential benefits of converting PDF data to XML data, high-level analysis could show whether the potential benefits outweigh the costs. In instances where IRS finds that potential benefits outweigh the costs, SB/SE and IT could provide this information to management to inform its decision as to whether the work is cost effective. Using this information, management could determine if the work should be conducted, and if it should be a high priority for SB/SE and the Information Technology organization. As of the end of fiscal year 2019, IRS made considerable progress implementing TCJA, however, much work remains, and IRS has publicly issued approximately half of planned official guidance. Given the magnitude and immediate effective dates for many TCJA provisions, Chief Counsel collaborated earlier and more closely with IRS BODs which enabled the agency to more efficiently and effectively develop guidance that accounts for tax administration and enforcement concerns. Moving forward, IRS can leverage the lessons learned from this enhanced collaboration. By identifying situations when this earlier and more frequent collaboration would benefit IRS’s guidance development process and by updating any relevant policies or procedures to document beneficial collaboration practices, IRS will be better prepared to implement the next set of complex or time-sensitive changes to the tax code. In developing regulations for TCJA provisions, Treasury and IRS made decisions that could potentially affect tax liability by billions of dollars per year, which would have distributional effects on the economy, but these effects were not included in their regulatory analyses. The distributional effects of tax liability changes from regulations can be significant; updating Treasury’s internal guidance to include analysis of these effects in the rulemaking process would provide greater transparency to the public, and would better inform decision makers who must determine which regulatory alternative is the best to adopt. Addressing data reliability issues in IRS’s tracking documentation could better ensure that further TCJA implementation work is performed in an efficient and timely manner and better enables IRS to identify opportunities for improvements to their implementation process. Additionally, this could enable IRS to better complete and evaluate existing TCJA implementation processes, as well as be better equipped to improve those processes for future application. SB/SE’s ability to analyze tax return data and efficiently plan compliance efforts is impeded by the lack of easily accessible and useable data for certain TCJA changes. Taking steps to obtain these data in instances where the potential benefits outweigh the costs would help the agency identify return filing trends and potential noncompliance to help the agency improve audit selection. It would also help SB/SE fulfill IRS’s goals of improving operations using data analytics and would also help the agency be able to effectively ensure compliance with and enforce TCJA provisions. We are making a total of five recommendations, including four to IRS and one to Treasury. Specifically: The Chief Counsel of the Internal Revenue Service, in coordination with appropriate offices, should identify and document parameters and procedures for applying enhanced collaborative approaches to regulation and other guidance development with IRS Business Operating Divisions. (Recommendation 1) The Commissioner of Internal Revenue should develop a process to accurately and thoroughly capture implementation status of ongoing projects in accordance with Standards for Internal Control in the Federal Government. (Recommendation 2) The Commissioner of Small Business/Self Employed should coordinate with appropriate IRS divisions or offices to identify the costs and benefits of retroactively transcribing taxpayer data resulting from TCJA. (Recommendation 3) Based on the costs and benefits identified in recommendation 3, the Commissioner of Small Business/Self Employed should determine which TCJA provisions’ data should be converted into a more useful electronic format for compliance and enforcement purposes and work with the appropriate offices to obtain the transcribed data, as appropriate. (Recommendation 4) The Assistant Secretary of Tax Policy should update Treasury’s internal guidance to ensure that Treasury’s regulatory impact analyses include examination of the distributional effects of revenue changes when regulations influence tax liability. (Recommendation 5) We provided a draft of this report to the Commissioner of Internal Revenue, the Secretary of the Treasury, and the Director of the Office of Management and Budget for review and comment. In its written comments, which are summarized below and reproduced in appendix IV, IRS disagreed with the four recommendations addressed to that agency. The Director of Treasury’s Office of Tax Analysis did not comment on the merits of the recommendation directed to Treasury and provided other comments by email, which are summarized below. In addition, IRS, Treasury, OMB also provided technical comments, which we incorporated as appropriate. IRS disagreed with our recommendation to identify and document parameters and procedures for applying enhanced collaborative approaches to regulation and other guidance development (Recommendation 1). IRS stated it believes that its Internal Revenue Manual provides sufficient guidance and flexibility on when such enhanced collaboration is appropriate and that establishing specific criteria is likely to reduce the flexibility and independent judgement that presently exists. Additionally, IRS said that this type of collaboration is not needed for more routine tax law changes. We are recommending that IRS document the collaboration procedures that were cited as critical for implementing TCJA for use in specific instances—such as during complex or time-sensitive tax law changes. As discussed in the report and acknowledged in IRS’s letter, this collaboration was particularly helpful for TCJA implementation and had many benefits, such as faster decision-making and identifying enforcement concerns earlier in the guidance development process. We believe that by implementing this recommendation, IRS can help ensure that institutional knowledge and beneficial practices from TCJA implementation will be documented and effectively leveraged to support implementation of future time-sensitive or complex tax law changes without restricting IRS’s flexibility. Documenting procedures would ensure IRS can retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel. For our recommendation that IRS develop a process to accurately and thoroughly capture implementation status of ongoing projects in accordance with federal internal control standards (Recommendation 2), IRS disagreed that a new process is needed and said that inaccurate reporting of implementation status did not harm IRS implementation of any TCJA provision. As we acknowledge in this report, IRS officials told us implementation was not impeded by data inconsistencies. However, accurately and thoroughly capturing implementation status on ongoing projects would provide accurate information to decision makers and could prevent potential misreporting, mismanagement, or inefficient resource investment in the future. For example, our ability to use these data to inform Congress of TCJA implementation status was impeded because we deemed the data unreliable for this purpose. Our recommendation does not require IRS to develop a new process for capturing and tracking implementation status. If deemed appropriate, IRS could, instead, update or modify existing processes in ways designed to ensure data reliability. IRS disagreed with our recommendations to identify the costs and benefits of retroactively transcribing certain taxpayer data and then to implement transcription based on this determination (Recommendations 3 and 4). IRS stated that retroactively transcribing data is a resource- intensive, manual process. We disagree with this assertion. LB&I is using optical character recognition to convert PDF data into a more useable format, which is a semi-automated process. Further, as also stated in this report, IRS IT officials we interviewed told us they had the capability to develop a program that would convert PDF data to a more useable format if IRS management deemed it a priority. In its response, IRS also states that the benefits of converting data to a more useable format are unknown. We do not expect IRS to conduct a complex and detailed cost-benefit analysis. Rather, as acknowledged in this report, a high-level analysis of costs and benefits could help IRS management determine what, if any, data would benefit compliance and enforcement efforts. IRS could use readily available existing information (such as the number of returns affected by a certain provision, LB&I and IT cost data on conversion efforts already implemented, or the usefulness of past compliance analytics in similar areas) to inform the analysis. IRS also states that the potential noncompliance costs are unknown until the agency completes audits of TCJA provisions. As we reported, conducting audits is labor-intensive and IRS’s audit rate and enforcement efforts have declined since 2011. Further, senior IRS officials we interviewed stated that a limitation of taxpayer information in the PDF format is that it is not easily analyzed. Therefore, we believe that converting data in instances where the benefits outweigh the costs would better position IRS to more effectively and efficiently pursue its mission of ensuring taxpayer compliance. In an email, the Director of Tax Analysis indicated that Treasury generally did not agree with the report’s findings regarding its economic analyses. The Director did not specifically comment on the merits of our recommendation that Treasury update its guidance for conducting regulatory impact analyses (Recommendation 5), but stated that the analyses underlying Treasury’s tax regulations have fully complied with the MOA established with OMB, which in Treasury’s view focuses on non- revenue effects. We maintain that decisions Treasury and IRS made when developing regulations to implement TCJA could potentially impact tax liability by billions of dollars per year; however, Treasury’s internal guidance dictates that these revenue effects should not be included in its economic analyses of the regulations. Amending Treasury’s guidance to ensure that impacts on tax revenue and liability are included would make the guidance consistent with E.O. 12866 and OMB Circular A-4, which underlie the MOA and instruct agencies to analyze the distributional consequences of regulations. Including these effects of tax regulations, as we recommended, is necessary in order to provide greater transparency to the public and better inform decision makers, who must determine which regulatory alternative is the best to adopt. We are sending copies to the appropriate congressional committees. We are also sending copies of the report to the Commissioner of Internal Revenue, the Secretary of the Treasury, the Director of the Office of Management and Budget, and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9110 or lucasjudyj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report (1) examines the Internal Revenue Service’s (IRS) processes that it has in place to provide guidance to taxpayers on Public Law 115- 97, commonly referred to as the Tax Cuts and Jobs Act of 2017 (TCJA) business and international provisions; (2) assesses the economic analyses Department of the Treasury (Treasury) conducted as part of the regulatory development process; (3) evaluates IRS monitoring of implementation of these provisions and describes implementation status; and (4) examines any challenges that could affect IRS’s ability to effectively administer these provisions. We defined business and international provisions as provisions assigned to IRS’s Large Business & International (LB&I) Division or Small Business/Self-Employed (SB/SE) Division. To examine IRS’s processes to provide guidance to taxpayers, we analyzed IRS documentation, such as prioritization records, guidance development records, and actual regulations and other guidance documents (e.g., notices and news releases) and interviewed IRS officials. Specifically, we reviewed IRS’s documentation of prioritization of TCJA provisions and interviewed IRS officials in the Tax Reform Implementation Office (TRIO) and the Office of Chief Counsel (Chief Counsel) to examine the criteria IRS used to prioritize TCJA provisions for implementation. We also reviewed IRS documentation on internal coordination and interviewed IRS TRIO and Chief Counsel officials to examine IRS’s strategy for and process of guidance development and IRS’s plan to provide taxpayers with timely information. We used criteria from Standards for Internal Control in the Federal Government and our key practices for collaboration to determine the extent to which IRS’s process for providing guidance to taxpayers was consistent with these standards and best practices. To assess the economic analyses Treasury conducted as part of the regulatory development process, we analyzed IRS, Treasury Office of Tax Policy (OTP), and Office of Management and Budget (OMB) documentation detailing the regulatory development and decision-making processes. We also interviewed officials from IRS TRIO, Chief Counsel, OTP, and OMB Office of Information and Regulatory Affairs (OIRA). Specifically, to identify the factors Treasury and IRS considered when analyzing trade-offs presented by different regulatory options to decide which regulatory options to select, we analyzed underlying regulatory development documentation and interviewed relevant officials. For example, we examined issues lists, internal memorandums, emails discussing regulatory alternatives and their tradeoffs, and early drafts regulations with internal comments. We also analyzed TCJA published regulations and interviewed OIRA and Treasury OTP officials to determine the extent to which Treasury OTP and IRS included discussions of regulatory alternatives and cost-benefit and economic analyses of these alternatives in the published regulations. We used criteria from OMB regulatory guidance for executive branch agencies to examine Treasury’s development and analyses of regulatory alternatives. This guidance includes the Memorandum of Agreement (MOA) between Treasury and OMB prescribing OMB review of tax regulations under Executive Order 12866; Executive Order 12866, Regulatory Planning and Review; and OMB Circular A-4, Regulatory Analyses, to determine the extent to which Treasury’s analyses met OMB guidance for developing regulations. To describe the implementation status of business and international TCJA provisions, we analyzed IRS project management documentation, such as IRS’s Enterprise Integrated Project Plan (EIPP) for TCJA implementation and publicly issued guidance and met with IRS officials. Specifically, we analyzed EIPP to determine which tasks were guidance or training related based on description, and developed keywords to limit our dataset to only relevant tasks. We interviewed IRS TRIO officials to ensure we accurately interpreted the description and status of the identified implementation tasks. We monitored for progress on guidance tasks by regularly reviewing IRS’s tax reform website and the Federal Register, as well as SB/SE and LB&I’s implementation trackers and Chief Counsel’s guidance planning documentation. We reviewed Chief Counsel, LB&I, and SB/SE documentation (e.g., implementation trackers) and met with those officials to understand their internal tracking mechanisms for TCJA tasks and implementation status. To monitor training tasks, we used the EIPP to establish which provisions would require training and reviewed training documentation (e.g., training schedules and materials) from LB&I and SBSE. To describe and monitor information technology (IT) implementation status, we analyzed IRS’s Information Technology organization’s TCJA implementation documentation and met with IRS Information Technology organization officials. We reviewed IRS’s IRM website to determine whether IRS had updated its IRM sections as planned in its EIPP and other planning documents. While we identified potential data reliability issues with the EIPP, LB&I’s implementation tracking documentation, and SB/SE’s implementation tracking documentation (including inaccurate recording of the completion status of multiple categories of tasks, inconsistent use of unique task identifiers across tracking documentation, and potential errors introduced in the transition from the EIPP to the subsequent tracking documentation), we determined that the data were sufficiently reliable for the purpose of reporting the status of guidance releases, training, and overall TCJA IT tasks. We did not find the data sufficiently reliable to report on the status of IRM updates. We were also unable to report on the number of IT tasks specific to business and international provisions as a subset of overall TCJA IT implementation because IRS’s IT organization did not track work by TCJA section. To assess the reliability of the EIPP, we met with TRIO officials to understand how the EIPP was created and updated, as well as verified information from outside sources, including the Federal Register and IRS’s tax reform website. After identifying potential discrepancies in LB&I’s and SB/SE’s TCJA tracking documentation, we followed up with SB/SE and LB&I to determine whether the status of our selected tasks was accurate and complete. SB/SE and LB&I provided responses and statements indicating that the status of some tasks was not accurately recorded. For example, we identified an instance where LB&I’s tracking documentation had a provision’s final regulations listed as issued in July 2019, when IRS had yet to issue the guidance. Based on these discrepancies and inconsistencies, we used criteria from the Standards for Internal Control in the Federal Government to evaluate IRS’s project management activities. To identify the impact of OIRA’s effect on the status of TCJA implementation, we analyzed information available on the agency’s public website to determine the length of time of OIRA review of regulations. We compared the length of time of OIRA’s review to agreed-upon time frames for OIRA review of tax regulations in the Memorandum of Agreement, Review of Tax Regulations under Executive Order 12866 (MOA) between Treasury and OMB to determine the extent to which OIRA met the MOA’s 10- and 45-day time frames. To examine challenges that could affect IRS’s ability to effectively administer these provisions, we analyzed TCJA and IRS documentation. Further, we interviewed TRIO, LB&I, SB/SE, IT, Chief Counsel officials, and outside tax practitioners. We analyzed TCJA’s statutory language to identify instances where the law included compliance safeguards, such as anti-abuse provisions or information reporting requirements. We reviewed IRS documentation (e.g., SB/SE compliance plans) and interviewed IRS officials to understand IRS’s views on the opportunities, challenges, and risks to administering and ensuring compliance with the new law. We also interviewed and subsequently analyzed statements from seven randomly selected tax practitioners who had submitted public comments on IRS’s proposed regulations for the qualified business income (QBI) deduction, opportunity zones, and the repatriation tax (see below for discussion of provisions we further analyzed) to identify outside perspectives on challenges for IRS administration and enforcement. We downloaded the public comments on these proposed regulations on April 9, 2019. For the QBI deduction, the open comment period was from August 16, 2018, to October 1, 2018, and as of the time we downloaded comments, there were 340 comments. For the repatriation tax, the open comment period was from August 9, 2019, to October 9, 2019, and as of the time we downloaded comments, there were 188 comments. For opportunity zones, the open comment period was from October 29, 2018, to February 8, 2019, and as of the time we downloaded comments, there were 185 comments. We also interviewed tax practitioners from two of the four “Big Four” tax firms and one outside tax practitioner to which we were referred to describe some outside opinions’ on challenges for IRS administration and enforcement. The views expressed in these interviews are not necessarily representative of those of other tax practitioners, or tax practitioners as a whole, and the views of the tax practitioners we interviewed are being used as illustrative examples throughout our report. We examined these challenges and risks and subsequently followed up with IRS to examine the extent to which IRS was aware of them and planning mitigating actions to address them. We used IRS’s strategic plan and to Standards for Internal Control in the Federal Government as criteria for identifying any gaps between mitigation efforts and overall agency-wide goals and priorities. As part of our work, we further analyzed three provisions—the QBI deduction, opportunity zones, and the repatriation tax—to gain specific insights into the decision-making process for prioritizing and developing guidance and regulations and factors that may affect IRS’ ability to effectively administer these provisions. We selected these three provisions for closer examination based on a number of factors, including (1) IRS designating them higher priority for implementation and identification, and (2) IRS, the National Taxpayer Advocate, and other knowledgeable stakeholders identifying them as especially challenging or complex to implement, administer, or enforce. Further, these three selections ensured we were able to examine at least one provision impacting domestic taxpayers managed by SB/SE division and at least one provision impacting foreign, or multinational, taxpayers managed by LB&I. We conducted this performance audit from August 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We found that, as of September 30, 2019, on average, the Office of Management and Budget’s (OMB) Office of Information and Regulatory Affairs (OIRA) completed its review of 25 Tax Cuts and Jobs Act of 2017 (TCJA) regulations in about 38 calendar days, as shown in tables 5 and 6. While according to the Memorandum of Agreement between OMB and the Department of Treasury, OIRA has 45 calendar days to review tax regulations, OIRA agreed to consider an expedited review of 10 business days for TCJA regulations. As of September 30, 2019, OIRA agreed to review three regulations in an expedited fashion and OIRA completed two of these reviews in 10 business days and the third in 12 business days. In addition to the contact named above, Brian James (Assistant Director), Dawn Bidne (Analyst-in-Charge), Michael Bechetti, Justin Bolivar, Tara Carter, Jacqueline Chapin, Nina Crocker, Robert Gebhart, Thomas Gilbert, Travis Hill, Naomi Joswiak, Mark Kehoe, Shelbe Klebs, Daniel Mahoney, Regina Morrison, Benjamin Moser, Sabine Paul, Bradley Roach, Erin Saunders-Rath, Jerome Sandau, Andrew J. Stephens, Rachel Stoiko, Jennifer Stratton, Peter Verchinski, and Sarah Wilson made key contributions to this report. 2019 Tax Filing: IRS Successfully Implemented Tax Law Changes but Needs to Improve Service for Taxpayers with Limited-English Proficiency. GAO-20-55. Washington, D.C.: January 15, 2020. Tax Gap: Multiple Strategies Are Needed to Reduce Noncompliance, GAO-19-558T. Washington, D.C.: May 9, 2019. Internal Revenue Service: Strategic Human Capital Management is Needed to Address Serious Risks to IRS’s Mission. GAO-19-176. Washington, D.C.: March 26, 2019. 2018 Tax Filing: IRS Managed Processing Challenges and Enhanced Its Management of Tax Law Changes. GAO-18-471. Washington, D.C.: September 10, 2018. Federal Regulations: Key Considerations for Agency Design and Enforcement Decisions. GAO-18-22. Washington, D.C.: October 19, 2017. Regulatory Guidance Processes: Treasury and OMB Need to Reevaluate Long-standing Exemptions of Tax Regulations and Guidance. GAO-16-720. Washington, D.C.: September 6, 2016. Federal Rulemaking: Agencies Included Key Elements of Cost-Benefit Analysis, but Explanations of Regulations’ Significance Could Be More Transparent . GAO-14-714. Washington, D.C.: September 11, 2014.", "summary": "According to IRS, TCJA was the most sweeping tax law change in more than three decades, with 86 provisions that modified, added to, or repealed business and international taxes, such as the qualified business income deduction. IRS determined it would take significant effort to implement the law given the limited time-frame and magnitude of the provisions. GAO was asked to review IRS's implementation of TCJA business and international provisions. Among other reporting objectives, this report examines IRS's (1) progress implementing the provisions, (2) processes to provide guidance, and (3) challenges for effectively administering these provisions. To address these objectives, GAO analyzed IRS documentation on project management, compliance planning, and regulation development. Additionally, GAO interviewed IRS officials and tax practitioners. The Internal Revenue Service (IRS) has made considerable progress issuing guidance to taxpayers for Public Law 115-97—commonly known as the Tax Cuts and Jobs Act of 2017 (TCJA)—but has additional work remaining to issue all planned guidance, as shown in the figure. To improve efficiency of TCJA guidance development, IRS internally collaborated earlier and more frequently than during more routine tax law changes. IRS officials said the benefits of this enhanced collaboration included faster decision-making on time-sensitive guidance, including regulations. IRS officials agreed enhanced collaboration had value but as of December 2019 had not identified the parameters for when this collaborative approach would be warranted. IRS may face challenges ensuring compliance with certain TCJA provisions because third-party information reporting is not always available. GAO's past work has found that one of the important factors contributing to the tax gap is the extent to which information is reported to IRS by third parties. Without third-party reporting, IRS will have to rely on resource-intensive audits to enforce certain TCJA provisions, which could be challenging given recent trends of declining audit rates and enforcement staff. GAO has recommendations from March 2019 for IRS to take actions to mitigate hiring risks and reduce skill gaps. IRS was also unable to update all information technology systems prior to the start of the 2019 tax season due to the magnitude of TCJA changes. As a result, IRS was not able to capture certain tax return information in a format that can be easily analyzed to help with compliance planning activities. One IRS division took steps to convert certain tax return data to a more useable format, but efforts to identify other viable opportunities have not been taken. Without appropriate data for analyses, IRS could face challenges enforcing certain TCJA provisions. GAO is making five recommendations, including that IRS develop and document procedures for continued enhanced collaboration and convert tax return data to a more useable format for compliance purposes. IRS disagreed; however, GAO believes that these recommendations will benefit guidance development and tax administration. In prior work, GAO recommended that IRS measure which activities are producing desired hiring outcomes and take steps to reduce skill gaps among revenue agents. IRS agreed with these recommendations and, as of December 2019, plans to report on efforts to close skill gaps by December 2021.", "document_type": "gao"}
{"report": "The Freedom of Information Act establishes a legal right of access to government information on the basis of the principles of openness and accountability in government. Before FOIA’s enactment in 1966, an individual seeking access to federal records faced the burden of establishing a “need to know” before being granted the right to examine a federal record. FOIA established a “right to know” standard, under which an organization or person could receive access to information held by a federal agency without demonstrating a need or reason. The “right to know” standard shifted the burden of proof from the individual to a government agency and required the agency to provide proper justification when denying a request for access to a record. Any person, defined broadly to include attorneys filing on behalf of an individual, corporations, or organizations, can file a FOIA request. For example, an attorney can request labor-related workers’ compensation files on behalf of his or her client, and a commercial requester, such as a data broker who files a request on behalf of another person, may request a copy of a government contract. In response, an agency is required to provide the relevant record(s) in any readily producible form or format specified by the requester, unless the record falls within a permitted exemption that provides limitations on the disclosure of information. Various amendments have been enacted and guidance issued to help improve agencies’ processing of FOIA requests. For example: The Electronic Freedom of Information Act Amendments of 1996 (1996 FOIA amendment) strengthened the requirement that federal agencies respond to a request in a timely manner and reduce their backlogged requests. Executive Order 13392, issued by the President in 2005, directed each agency to designate a senior official as its chief FOIA officer. This official was to be responsible for ensuring agency-wide compliance with the act. The chief FOIA officer was directed to review and report on the agency’s performance in chief FOIA officer reports. The OPEN Government Act, which was enacted in 2007 (2007 FOIA amendment), made the 2005 executive order’s requirement for agencies to have a chief FOIA officer a statutory requirement. It also required agencies to include additional statistics, such as more details on processing times, in their annual FOIA reports. The FOIA Improvement Act of 2016 (2016 FOIA amendment) addressed procedural issues, including requiring that agencies (1) make records available in an electronic format if they have been requested three or more times; (2) notify requesters that they have not less than 90 days to file an administrative appeal, and (3) provide dispute resolution services at various times throughout the FOIA process. Further, the act required OMB, in consultation with the Department of Justice, to create a consolidated online FOIA request portal that allows the public to submit a request to any agency through a single website. The 1996 FOIA amendment required agencies, including DHS, to generally respond to a FOIA request within 20 working days. Once received, the request is to be processed through multiple phases, which include assigning a tracking number, searching for responsive records, and releasing the records to the requester. In responding to requests, FOIA authorizes agencies to use nine exemptions to withhold portions of records, or the entire record. These nine exemptions can be applied by agencies to withhold various types of information, such as information concerning foreign relations, trade secrets, and matters of personal privacy. FOIA allows a requester to challenge an agency’s final decision on a request through an administrative appeal or a lawsuit. Agencies generally have 20 working days to respond to an administrative appeal. Created in 2003, DHS assumed control of about 209,000 civilian and military positions from 22 agencies and offices that specialize in one or more aspects of homeland security. By the nature of its mission and operations, the department creates and has responsibility for vast and varied amounts of information covering, for example, immigration, border crossings, law enforcement, natural disasters, maritime accidents, and agency management. According to its 2018 Chief FOIA Officer Report, DHS’s organizational structure consists of 24 offices, directorates, and components. FOIA requests are split between the department’s Privacy Office, which acts as its central FOIA office, and FOIA offices in the department’s component agencies. Three of the major operational components of DHS are: U.S. Citizenship and Immigration Services (USCIS) promotes an awareness and understanding of citizenship, and ensures the integrity of the nation’s immigration system. Its records include asylum application files and other immigration-related documents. Customs and Border Protection (CBP) secures the border against transnational threats and facilitates trade and travel through the enforcement of federal laws and regulations relating to immigration, drug enforcement, and other matters. The agency maintains records related to agency operations, activities, and interactions. Immigration and Customs Enforcement (ICE) promotes homeland security and public safety through the criminal and civil enforcement of federal laws governing border control, customs, trade, and immigration. It maintains information related to the law enforcement records of immigrants and detainees, as well as information pertaining to human trafficking/smuggling, gangs, and arrest reports. According to its 2018 Chief FOIA Officer Report, DHS and its component agencies reported that they processed 374,945 FOIA requests in fiscal year 2018—the most of any federal government agency. As of its 2018 report, the department had a backlog of 53,971 unprocessed requests— the largest backlog of any federal agency. Amendments and guidance relating to FOIA call for agencies, including DHS, to implement key requirements aimed at improving the processing of requests. Among others, these requirements call for agencies to (1) update response letters, (2) implement tracking systems, (3) provide FOIA training, (4) provide records online, (5) designate chief FOIA officers, and (6) update and publish timely and comprehensive regulations. As we noted in our June 2018 report, DHS had implemented these six FOIA requirements. Update response letters: The FOIA amendments require that certain information be included in agency response letters. For example, if part of a FOIA request is denied, agencies are required to inform requesters that they may seek assistance from the FOIA public liaison of the agency or the National Archives and Records Administration’s Office of Government Information Services (OGIS); file an appeal to an adverse determination within a period of time that is not less than 90 days after the date of such adverse determination; and seek dispute resolution services from the FOIA public liaison of the agency or OGIS. DHS had updated its FOIA response letters to include this specific information, as required per the amendments. Implement tracking systems: DHS used commercial automated systems, as called for by various FOIA amendments and guidance, and had established telephone or internet services to assist requesters in tracking the status of a request. The department used modern technology (e.g., mobile applications) to inform citizens about FOIA. The commercial systems allowed requesters to submit a request and track the status of that request online. In addition, DHS developed a mobile application that allowed FOIA requesters to submit a request and check its status. The department’s FOIA tracking systems were compliant with requirements of Section 508 of the Rehabilitation Act of 1973 (as amended), which required federal agencies to make their electronic information accessible to people with disabilities. Provide FOIA training: DHS’ chief FOIA officer offered FOIA training opportunities to staff in fiscal years 2016 and 2017, as required by the 2016 FOIA amendments. Specifically, the department provided training in responding to, handling, and processing FOIA requests. Provide records online: DHS posted records online for three categories of information, agency final opinions and orders, statements of policy, and frequently requested orders as required by 2009 memorandums from both the President and the Attorney General. Designate chief FOIA officers: DHS designated its Chief Privacy Officer as its Chief FOIA Officer. This position was a senior official at the assistant secretary or equivalent level, as required by a 2005 executive order and the 2007 FOIA amendments. Update and publish timely and comprehensive regulations: Guidance from the Department of Justice Office of Information Policy (OIP) encourages agencies to, among other things, describe their dispute resolution process; describe their administrative appeals process; notify requesters that they have a minimum of 90 days to file an administrative appeal; include a description of unusual circumstances and restrictions on an agency’s ability to charge certain fees when FOIA’s times limits are not met; and update agency regulations in a timely manner (i.e., update regulations by 180 days after the enactment of the 2016 FOIA amendment). DHS had addressed these five requirements in updating its regulations, as called for in the 2016 FOIA amendment and in related OIP guidance. The Attorney General’s March 2009 memorandum called on agency chief FOIA officers to review all aspects of their agencies’ FOIA administration and report to Justice on steps that have been taken to improve FOIA operations and disclosure. Subsequent Justice guidance directed agencies that had more than 1,000 backlogged requests in a given year to describe their plans to reduce their backlogs. Beginning in calendar year 2015, these agencies were to describe how they had implemented their plans from the previous year and whether that had resulted in a backlog reduction. In June 2018, we reported that DHS received about 191,000 to about 326,000 requests per year—the most requests of any agency—for a total of 1,320,283 FOIA requests in fiscal years 2012 through 2016. Further, the department had a backlog ranging from 28,553 in fiscal year 2012 to 53,971 in fiscal year 2018. The total numbers of these requests and backlogs are shown in table 1. We also reported that DHS, in its chief FOIA officer reports from fiscal years 2012 to 2016, stated that it had implemented several methods to reduce backlogs. According to the reports, the DHS Privacy Office, which is responsible for oversight of the department’s FOIA program, worked with components to help address the backlogs. The reports noted that the Privacy Office sent monthly emails to component FOIA officers on FOIA backlog statistics, convened management meetings, conducted oversight, and reviewed workloads. Leadership met weekly to discuss the oldest pending requests, appeals, and consultations, and determined steps needed to process those requests. In addition, in 2018, we noted that several other DHS components reported implementing actions to reduce backlogs. CBP hired and trained additional staff, encouraged requesters to file requests online, established productivity goals, updated guidance, and used better technology. USCIS, the National Protection and Programs Directorate, and ICE increased staffing or developed methods to better forecast future workloads to ensure adequate staffing. ICE also implemented a commercial off-the-shelf web application, awarded a multimillion-dollar contract for backlog reduction, and detailed employees from various other offices to assist in the backlog reduction effort. Due to these efforts by the Privacy Office and other components, the backlog dropped 66 percent in fiscal year 2015, decreasing to 35,374 requests. Yet, despite the continued efforts, the backlog numbers increased again. According the 2018 Chief FOIA Officer’s report, the department ended 2018 with a backlog of 53,971 requests. DHS attributed these increases to several factors, including an increase in the number of requests received, the increased complexity and volume of responsive records for those requests, and the loss of staff needed to process the requests. In June 2018, we reported that one reason DHS was struggling to consistently reduce its backlogs is that it lacked documented, comprehensive plans that would provide a more reliable, sustainable approach to addressing backlogs. In particular, it did not have documented plans that described how it intended to implement best practices for reducing backlogs over time. These best practices, as identified by Justice’s OIP, included specifying how DHS would use metrics to assess the effectiveness of backlog reduction efforts and ensuring that senior leadership supports backlog reduction efforts. In our June 2018 report, we recommended that the department take steps to develop and document a plan that fully addresses best practices with regard to the reduction of backlogged FOIA requests. In response, DHS reported that it had initiated a department-wide compliance assessment and stated that it planned to use the results of the assessment to help guide it in identifying best practices and areas of improvement. As of this month (October 2019), the department stated that the draft plan is currently with the components for review and is pending clearance. Until it has a final plan that fully addresses best practices, DHS will likely continue to struggle to reduce its backlogs to a manageable level. This is particularly important, as the number and complexity of requests will likely increase over time. Among the most frequent FOIA requests made to DHS are those for immigration files. These files usually contain various types of information pertaining to immigrants, including asylum applications, law enforcement records, and border crossing documents. As such, they may contain information and records that are generated by various DHS components or other agencies. In 2014, we reported that within DHS, three components—USCIS, CBP, and ICE—created most of the documents included in immigration files. USCIS was the custodian of the files, and all FOIA requests for such files were either initiated with, or referred to, USCIS for processing. Specifically, to process a FOIA request for an immigration file, the USCIS staff to whom the request was assigned first manually entered the requester’s data, such as a name and address, into USCIS’s FOIA system to establish a record of the request. Next, the staff retrieved and scanned the documents in the requested file and reviewed the documents. If all of the documents were generated by USCIS, the staff made redactions as needed, sent the documents to the requester, and closed out the request. Further, if the FOIA request covered files containing documents generated by CBP, then USCIS was able to process the request on the basis of an agreement to that effect with CBP. By having USCIS process such requests for CBP documents, the two components avoided duplication in their response to a FOIA request. In November 2014, however, we reported that USCIS and ICE did not have such an agreement for documents generated by ICE. Thus, the USCIS staff was to identify any such documents and make them available to ICE’s FOIA staff for their separate processing. In doing so, we noted that USCIS and ICE engaged in duplicative processing of FOIA requests for those immigration files containing documents related to law enforcement activities that were generated by ICE. Specifically, to facilitate ICE’s review of such files, USCIS staff transferred copies of the ICE-generated documents to a temporary electronic storage drive maintained by USCIS. ICE retrieved the documents, and the ICE staff then re-entered the data to create a new FOIA request in ICE’s FOIA processing system. The staff then proceeded with processing the requested documents, and released them to the requester—in essence, undertaking a new, and duplicate, effort to respond to the FOIA request. Figure 1 depicts the duplication that occurred in USCIS’s and ICE’s downloading and re-entering of data to respond to FOIA requests for immigration files. We noted that, up until April 2012, USCIS and ICE had an agreement whereby USCIS processed ICE’s documents contained in an immigration file. However, the components’ officials stated that, since that agreement ended, the components had not made plans to enter into another such agreement. According to ICE’s FOIA Officer, USCIS’s processing of ICE’s documents in immigration files was viewed as being too costly. Nonetheless, while there would be costs associated with USCIS processing ICE’s documents in immigration files, the potential existed for additional costs to be incurred in the continued duplicate processing of such files. Our work has noted that duplication exists when two or more agencies or programs are engaged in the same activities or provide the same services to the same beneficiaries. We concluded that the duplicate processing of a single FOIA request by USCIS and ICE staff contributed to an increase in the time needed to respond to a FOIA request for immigration files. Because USCIS did not send the immigration file to ICE until it had completed its own processing of the relevant documents— which, according to USCIS, took on average 20 working days—ICE usually did not receive the file to begin its own processing until the 20-day time frame for responding to a request had passed. We pointed out that re-establishing an agreement that allows USCIS to process ICE-generated documents included in requests for immigration files, to the extent that the benefits of doing so would exceed the cost, could enable the two components to eliminate duplication in their processes for responding to such a request. Further, it could help reduce the time needed by these components in responding to a request. Therefore, in November 2014, we recommended that DHS direct the Chief FOIA Officer to determine the viability of re-establishing the service- level agreement between USCIS and ICE to eliminate duplication in the processing of immigration files. We stressed that, if the benefits of doing so would exceed the costs, DHS should re-establish the agreement. We also reported on our finding and recommendation regarding duplicate processing in our reports and updates on fragmentation, overlap, and duplication, issued in 2015 through 2019. In response, DHS indicated that it was working on a system intended to address the duplication. Specifically, in August 2018, DHS’s Privacy Office Director of Correspondence/Executive Secretary stated that the Privacy Office was leading a working group in collaboration with the Office of the Chief Information Officer to develop requirements for a single information technology solution for processing incoming FOIA requests. The director added that DHS used three disparate systems to track, manage, and process FOIA requests and that moving USCIS and ICE to one processing solution should result in processing benefits and lower overall administrative costs. We continue to track DHS’s progress in implementing this recommendation. However, as of October 2019, DHS’s Privacy Office stated that these actions were still in progress. In conclusion, DHS has implemented a number of key FOIA practices. However, it does not have a comprehensive plan to address its FOIA backlog, nor has it yet addressed duplication in its FOIA process. Addressing both of these issues is important, as the number and complexity of requests will likely increase over time and DHS may be challenged in effectively responding to the needs of requesters and the public. Chairwoman Torres Small, Ranking Member Crenshaw, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staffs have any questions about this testimony, please contact Vijay A. D’Souza, Director, Information Technology and Cybersecurity, at (202) 512-6240 or dsouzav@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony statement. GAO staff who made key contributions to this testimony include Neela Lakhmani and Anjalique Lawrence (assistant directors), Kara Epperson, Christopher Businsky, Nancy Glover, and Scott Pettis. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "FOIA requires federal agencies to provide the public with access to government records and information based on the principles of openness and accountability in government. Each year, individuals and entities file hundreds of thousands of FOIA requests. DHS continues to receive and process the largest number of FOIA requests of any federal department or agency. For fiscal year 2018, over 40 percent of federal FOIA requests (about 396,000) belonged to DHS. GAO was asked to summarize its November 2014 and June 2018 reports which addressed, among other things, (1) DHS's methods to reduce backlogged FOIA requests and (2) duplication in DHS's processing of FOIA requests. In conducting this prior work, GAO evaluated the department's and components' FOIA policies, procedures, reports, and other documentation; and interviewed agency officials. GAO also followed up on its recommendations to determine their implementation status. The Department of Homeland Security's (DHS) responsibilities for processing Freedom of Information Act (FOIA) requests are split between the department's Privacy Office, which acts as its central FOIA office, and FOIA offices in the department's component agencies, such as U.S. Citizenship and Immigration Services and Immigration and Customs Enforcement. In 2018, GAO reported that DHS had implemented several methods to reduce backlogged FOIA requests, including sending monthly emails to its components on backlog statistics and conducting oversight. In addition, several DHS components, implemented actions to reduce their backlogs. Due to efforts by the department, the backlog dropped 66 percent in fiscal year 2015, decreasing to 35,374 requests. Although there was initial progress by the end of fiscal year 2015, the number of backlogged requests increased in fiscal years 2016 and 2018 (see figure). One reason DHS was struggling to consistently reduce its backlogs is that it lacked documented, comprehensive plans that would provide a more reliable, sustainable approach to addressing backlogs and describe how it will implement best practices for reducing backlogs over time. DHS attributed the increase in its FOIA backlogs to several factors, including the increased numbers and complexity of requests received and the volume of responsive records for those requests. Until it develops a plan to implement best practices to reduce its backlogs, DHS will likely continue to struggle to reduce the backlogs to a manageable level. In addition, in 2014 GAO reported that certain immigration-related requests were processed twice by two different DHS components. The duplicate processing of such requests by the two components contributed to an increase in the time needed to respond to the requests. GAO continued to report this issue in its 2019 annual product on opportunities to reduce fragmentation, overlap, and duplication. In its prior reports, GAO made five recommendations to DHS. These included, among other things, that DHS (1) take steps to develop and document a plan that fully addressed best practices with regard to reducing the number of backlogged FOIA requests and (2) eliminate duplicative processing of immigration-related requests. The department agreed with the recommendations. However, as of October 2019, DHS had not fully implemented all of them.", "document_type": "gao"}
{"report": "DOD uses military and commercial satellite communications (SATCOM) to meet its global communications requirements. DOD acquires wideband capacity through two methods: DOD purpose-built: DOD obtains some of its SATCOM through its purpose-built systems, which include Wideband Global SATCOM (WGS) satellites. While DOD awards contracts to commercial companies to build these systems, the department is responsible for the systems’ procurement, operations and sustainment; therefore, they are considered purpose-built. Commercial contracts: DOD also purchases commercial SATCOM services to supplement its purpose-built systems, such as for satisfying users who have needs beyond available military satellite resources, supporting training on ground systems, or meeting the needs of unique users. In these cases, DOD acquires commercial SATCOM bandwidth through several competitively selected vendors, who are responsible for operating and sustaining their own systems. Military SATCOM architectures fall into three types: protected, which provides secure, assured communications; wideband, which supports worldwide capacity for high data rate communications, including high-quality voice and imagery; and narrowband, which provides reliable and secure communications less vulnerable to adverse weather conditions or other physical limitations, such as distance, dense foliage, and terrain. DOD’s primary wideband satellite communications system, WGS, currently provides a portion of DOD’s required SATCOM bandwidth, but the Air Force estimates its satellite constellation’s capabilities will begin to degrade in the late 2020s. The Air Force is adding at least one more satellite to the WGS constellation and plans for an enhanced WGS-11 to provide the capacity of two satellites. During the Wideband AOA, DOD estimated that adding this satellite to the constellation would extend the availability of wideband communications to 2031. According to the Air Force, there is potential for adding a 12th WGS satellite to the constellation. Like other types of space systems, DOD’s wideband SATCOM systems generally involve four types of interrelated segments that make a space capability fully functional. As illustrated in figure 1, they include (1) the space segment—namely the satellites; (2) the ground segment, with network services and also including satellite and payload control systems and data processing subsystems and facilities; (3) user equipment, such as radios, terminals, and routers needed by the warfighter to use the capability; and (4) launch vehicles and facilities. Within the space segment, satellites operate in several different types of orbits to meet different communication and mission needs, as shown in figure 2. The orbital location of a satellite can affect its capacity to transmit data, or what parts of the Earth can receive its signal. For example, highly elliptical orbits are necessary for providing long dwell times over northern latitudes due to the curvature of the Earth, while other orbits cover remaining latitudes. Wideband satellites operate in different radio frequency spectrum bands. DOD typically relies on C, X, Ku, and Ka-bands to provide wideband connectivity, determined by where and how users are operating. Each of these frequency bands has advantages and disadvantages for various applications. Satellite transponders operating at the lower C-band frequencies are less susceptible to degradation from rain than other bands. In the United States, the X-band is specifically designated for use by the U.S. government and the North Atlantic Treaty Organization. The Ku-band operates at higher frequencies and can communicate with smaller antennas and offer more flexibility. The still-higher-frequency Ka- band satellites can transmit more data than C, X, and Ku-band satellites, but their signals are more susceptible to degradation from water vapor and rain than satellites in lower frequency bands. Commercial satellite communication providers have historically operated primarily in the Ku- band but are now expanding services in the Ka-band to offer higher data rates. An AOA is a key first step in DOD’s acquisition process and assesses alternative solutions for addressing future needs. DOD acquisition guidance provides the purpose and procedures associated with conducting an AOA to support decision making. DOD experts in areas such as cost estimating, technological analysis, and acquisitions, along with military and commercial stakeholders, comprise the AOA study team. The study team is involved in the day-to-day work of the AOA process and conducts the analyses that form the foundation of the assessment. During the AOA study period, the study team develops alternatives to satisfy capability gaps that they assess against pre-established performance requirements. We have identified 22 best practices for an AOA process. Of these, 6 best practices are associated with a “comprehensive” AOA. Comprehensive means that the AOA process ensures that the mission need is defined in a way to allow for a robust set of alternatives, that no alternatives are omitted, and that each alternative is examined thoroughly for the project’s entire life cycle. Without a clearly defined mission need and comprehensive list of alternatives, the AOA process could overlook the alternative that best meets the mission need. Furthermore, without considering the complete life cycle of each alternative, decision makers will not have a comprehensive picture of the alternatives analyzed. DOD completed its analysis of wideband SATCOM alternatives in June 2018 and identified 11 alternatives that represent several possible approaches to SATCOM acquisitions. We found the Wideband AOA to be a comprehensive assessment. The Office of the Secretary for Defense for Acquisition and Sustainment completed the Wideband AOA in June 2018 to support decision making for future wideband architectures. Several subsystems comprise a SATCOM architecture and can include the number, type, orbital location, and capacity of satellites and associated ground or user segments. WGS constellation satellites will begin reaching their end of life in the early 2030s, which means DOD will need to begin launching replacement system satellites in the late 2020s. DOD satellite systems take, on average, over 7 years to develop and launch the first satellite of a purpose-built system. Given these time frames, the Wideband AOA study team focused on possible alternatives DOD could begin developing as early as 2019. In October 2016, the Office of the Secretary of Defense- Cost Assessment and Program Evaluation developed the Wideband Communications Services Analysis of Alternatives Study Plan. This Study Plan provided the schedule and tasks to be conducted for the Wideband AOA. These tasks included identifying study questions to be addressed and listing measures of performance and effectiveness. The Study Plan also described the organizational structure and methodology for executing the Wideband AOA. The Wideband AOA study team developed 11 alternatives that broadly represented three different acquisition approaches: legacy DOD SATCOM procurement focused on purpose-built systems with some commercially-contracted services; commercial-focused SATCOM procurement; and a strategy that would transition from a mainly purpose- built system to a more commercial SATCOM-oriented model. Historically, DOD has bought purpose-built SATCOM assets, including satellites and supporting ground systems, while contracting for supplemental commercial bandwidth. Table 1 summarizes the architectures and these approaches. Our assessment of the Wideband AOA found that it met our criteria for a comprehensive AOA process. Table 2 shows our determinations of how fully the Wideband AOA met each of our six best practices. Appendix I provides more detail on our AOA best practices. Based on our analysis, we found that the Wideband AOA study team thoroughly addressed a wide range of possible satellite system alternatives. Moreover, the Wideband AOA study examined the ground segment systems—including user terminals—which will communicate with the satellite system DOD chooses to replace WGS. Although user terminals were not the primary focus of this AOA, DOD officials told us this effort was the first time DOD has studied and consolidated department-wide costs for these terminals, which they said provided valuable context to decision-makers. We discuss this new information on terminals in further detail later in this report. As set forth in the AOA Study Plan, the Wideband AOA study team solicited and incorporated input from across DOD stakeholders, such as the military services, operational users, and SATCOM partner nations. The study team also solicited and incorporated information from commercial SATCOM vendors to inform its alternatives. Additionally, the Wideband AOA study team incorporated information from interrelated studies, referred to as pilots and pathfinders, that the Air Force and Defense Information Systems Agency conducted. These studies recommended ongoing experimentation and adaptation to identify, incorporate, and guide future commercial SATCOM development, as well as changes to DOD’s approach to SATCOM acquisitions. As set forth in its Study Plan, the Wideband AOA study team obtained military input from across DOD and information from commercial SATCOM vendors to inform its alternatives. AOA working groups were one of several mechanisms DOD used to obtain stakeholder input. The AOA study plan directed the establishment of eight working groups to consolidate subject matter experts for relevant SATCOM topics, as shown in table 3. Each working group, task force, and team conducted its analysis and wrote an appendix to the AOA report summarizing its methodology, inputs, and results. Each team also provided its own conclusions or recommendations, which contributed to the overall findings and recommendations of the AOA report. Military service representatives who participated in the Wideband AOA described to us how their personnel were involved in many or all of the working groups. AOA study leaders also emphasized the quality of the input from the working groups and were confident the AOA successfully captured the perspectives of acquisition, operational, and user communities—personnel responsible for buying, controlling, and using wideband SATCOM. In addition to the working groups, the Wideband AOA study team developed functional requirements for the alternatives by requesting SATCOM user demand data from the services, and invited SATCOM partner nations to participate in the AOA—a portion of which accepted. These efforts provided additional information from user communities. Wideband AOA study team leaders described how they relied on a formal Joint Chiefs of Staff process to obtain inputs from the military services on their current and projected bandwidth demands. Through this process, the department obtained SATCOM user demand data from combatant commands, military services, and their sub-commands. The AOA study team then used these results to develop an aggregate user demand projection that was foundational to the AOA. Any viable alternative had to provide sufficient bandwidth to meet future user demand. DOD requested inputs from commercial SATCOM vendors and the Commercial Working Group used these to identify the space system subcomponents, namely technical characteristics, including frequency bands, orbit, and satellite mass that the Technologies and Alternatives Working Group eventually combined into the 11 final alternatives. The Commercial Working Group’s intent in identifying these subcomponents was to represent capabilities the SATCOM industry will have on-orbit by 2023, without depicting any single vendor’s potential system. The Commercial Working Group also incorporated results from DOD pilot and pathfinder efforts (discussed below) to develop a roadmap for DOD to implement an enterprise management approach to SATCOM procurement and operations. The Air Force and Defense Information Systems Agency conducted interrelated pilot and pathfinder studies before and during the Wideband AOA that provided information on SATCOM business arrangements, user terminal prototyping, and acquisition efficiencies. In 2014 and 2015, Congress authorized, and then directed, DOD to carry out a pilot program on the acquisition of commercial satellite communication services. As part of this pilot, DOD initiated pathfinder projects to test the feasibility of these new business arrangements. The Air Force and Defense Information Systems Agency studied and prototyped methods to improve commercial SATCOM acquisition and provide more flexible satellite connections for mobile SATCOM users. The agencies did so by contracting with commercial SATCOM providers for the following: Air Force Pilot – define and demonstrate prototyping to improve access to commercial SATCOM. The Air Force completed phases 1 and 2 of this 3-phase pilot program, studying preferential purchasing approaches that incentivize industry and the types of SATCOM architectures that enable such purchasing, such as a managed services approach that consolidates commercial SATCOM procurement for DOD users. Phase 1 studied commercial satellite communication architecture and business structures. The Wideband AOA’s Commercial Working Group used the phase 1 results in its modeling of SATCOM enterprise management. Phase 2 demonstrated a flexible modem-to-terminal interface to allow a terminal to “roam” or switch between different manufacturers’ satellite constellations. Phase 3 is ongoing and focuses on network integration risk reduction efforts. Air Force Pathfinders – prove that innovative business arrangements can meet DOD requirements and reduce costs. Through the pathfinder research efforts, the Air Force purchased an on-orbit transponder as well as pre-launch transponder to demonstrate different strategies for buying SATCOM. The final pathfinder effort is ongoing and is to demonstrate how access to shared bandwidth and more flexible ground systems can improve SATCOM access for warfighters. These types of capabilities help users to move more quickly and easily, with a reliable SATCOM connection. Defense Information Systems Agency Pathfinders – examine how acquisition efficiencies improve SATCOM services. The pathfinders’ findings provided observations on market trends for SATCOM contracting, namely that pricing will continue to decrease. The pathfinders also showed that DOD’s typical SATCOM requirements are not stable from year to year, meaning DOD cannot accurately predict when or where it will need surge SATCOM capacity. The pathfinders also identified management challenges to aggregating SATCOM requirements. The pilot and pathfinder efforts recommended ongoing experimentation and adaptation to identify, incorporate, and guide developing commercial SATCOM capabilities, as well as changes to DOD’s traditional approach to SATCOM acquisitions. In particular, both the Air Force and Defense Information Systems Agency recommended that DOD adapt to changing business models, especially for managed services in commercial SATCOM, in which DOD would purchase SATCOM services but would not own or manage the systems and data rates. Changing business models could also include greater coordination with the SATCOM industry, so DOD can better incorporate commercial technology into future systems. The Defense Information Systems Agency also recommended that DOD pursue an alignment of common types of user terminals and SATCOM architectures. For example, many programs use a different approach to procuring terminals and SATCOM architectures, which prevents DOD from taking advantage of commonalities that could save resources. Such commonalities include users in the same geographic area. These Air Force and Defense Information Systems Agency recommendations overlap with half of the findings and recommendations of the Wideband AOA. DOD concluded in the Wideband AOA that integrating purpose-built satellite systems and commercially-provided systems into a hybrid architecture would be more cost effective and capable than any single purpose-built or commercial system alone. The AOA study team recommended actions to obtain more information on transitioning to a more integrated architecture of purpose-built and commercial systems and reducing risk. However, DOD does not have a plan to implement these recommendations and inform timely decision-making. During the AOA, DOD found that integrating purpose-built satellites and commercially-provided systems into a hybrid architecture would save costs and provide more capability than any single purpose-built or commercial system alone. The department currently uses a mix of purpose-built and commercial SATCOM contracts, but DOD has not historically managed these systems in coordination, or with an enterprise approach. DOD considered 11 architectures in its final analysis and all were to some extent hybrids of purpose-built and commercial systems because the AOA study team found that DOD requires a combination of military and commercial system capabilities. The Wideband AOA report identified three of the 11 potential architectures that would best meet DOD’s wideband SATCOM needs: Legacy Purpose-Built and Commercial Contracting Architecture - Procure and field a new purpose-built constellation for X and Ka-band capabilities with anti-jam technologies and upgraded antennas. DOD would continue to contract for commercial SATCOM as needed. Commercial-Oriented Architecture - Pursue advanced commercial high capacity satellites with steerable beams over the Ka-band. Also procure 10 purpose-built satellites to meet the military’s requirement for X-band communications. Transitional Step to Commercial Architecture - Transition to commercially-managed services architecture in low-Earth orbit for approximately 5,000 users over the long term. DOD would procure and field the modernized, purpose-built legacy architecture described above, then modify its suite of user terminals to align with the new low-Earth orbit satellites, emphasizing a cost-effective strategy to do so. For users who do not transition to the new commercial satellites, the purpose-built constellation provides continued X and Ka-band capability. During the Wideband AOA, DOD found that any post-WGS solution must continue to provide purpose-built SATCOM capabilities. For example, some users require X-band communications and identified this as the single most important capability to maintain. However, commercial constellations provide limited X-band communications due to this band’s historical use for military communications. The companies and international partners that do offer X-band communications provide fragmented coverage that does not fully meet DOD’s needs. In addition, commercial satellite constellations do not offer services in all of the areas DOD operates, such as over oceans and in polar regions. At the same time, because purpose-built systems alone cannot meet all military requirements, DOD found it will need to rely on commercial capabilities as part of a future architecture. Consequently, the AOA study team assessed alternatives that would expand DOD’s use of emerging commercial technologies. For example, DOD expects certain operations, like aerial vehicle flights that rely on wideband SATCOM, to increase and drive demand for commercial SATCOM capabilities. Moreover, the AOA study team found that emerging commercial capabilities could meet routine military needs, such as training, at a competitive cost. The AOA study team concluded integrating these capabilities into a future architecture would be beneficial. In its Wideband AOA report, the AOA study team made a series of recommendations focused on maintaining current wideband capabilities and overcoming near-term information gaps in transitioning to new SATCOM acquisition and management approaches. All of the recommendations focused on gaining information needed to transition to a hybrid architecture of purpose-built and commercial systems in the long term. Table 4 provides examples of DOD’s recommendations and the additional knowledge DOD needs to obtain as it pursues a post-WGS solution. The Wideband AOA recommendations also addressed risks associated with any new SATCOM architecture, which the study team found include: (1) the uncertain stability and maturity of emergent commercial SATCOM systems and (2) the magnitude of replacing or modifying SATCOM user terminals. Commercial Technology Stability and Maturity: DOD found in the Wideband AOA that the commercial SATCOM market needs time to grow and stabilize as industry seeks to build a consumer base, especially for low-Earth-orbit-based internet services. The AOA study team found that if commercial companies cannot close their businesses cases around proposed solutions, DOD investments or programs that rely on those proposed solutions may fail. Further, many commercial systems, especially those based in low-Earth orbit, are still maturing. SATCOM providers have not yet worked closely with DOD to see how they would need to modify such constellations to operate with future DOD systems, including ground systems. Wideband AOA stakeholders—military and commercial—also described their struggle to share information on technical requirements, new capabilities, and pricing. For example, military stakeholders wanted more detailed engineering data on emerging commercial capabilities while commercial stakeholders wanted additional information on proposed alternatives for providing cost data. Commercial stakeholders also sought to protect their proprietary information. DOD’s recommendation to invest in and shape commercial SATCOM development is aimed at reducing this risk and improving information sharing between DOD and the SATCOM industry. Replacing or Modifying User Terminals: Managing user terminal development and upgrades is complex and, according to DOD officials, is one of the largest challenges the department faces in selecting a post-WGS architecture. In its analysis, DOD found that managing upgrades or replacement costs and schedules for over 17,000 terminals of approximately 135 different designs was a major challenge. The AOA’s analysis showed that out-of-cycle terminal replacement would drive significant costs and affect DOD operations. For example, vehicles like Humvees or ships have maintenance periods that are scheduled years in advance. Changing terminals could require unscheduled maintenance, potentially disrupt personnel planning, and cost more than if the terminals were upgraded on their planned refresh cycles. Certain users also cannot transition to commercial SATCOM and still meet operational requirements. For example, Navy stakeholders told us their terminals were not considered for transition to commercial systems during the Wideband AOA due to a number of issues, including Ku-band radio frequency interference, all-weather availability, open ocean coverage, and network constraints. Both our past work and the Wideband AOA found that DOD faces ongoing risks in aligning its satellite and ground control systems. We have reported that these risks have arisen, in part, because user terminal development programs are typically managed by different military acquisition organizations than those managing the satellites and ground control systems. The AOA recommendation to develop an enterprise SATCOM terminal strategy is aimed at reducing the risk user terminals present to DOD’s post-WGS SATCOM architecture. DOD’s recommendations that focus on gaining additional knowledge align with GAO’s acquisition best practices for knowledge-based decision- making and risk reduction, but DOD lacks a formal plan to implement these recommendations. More specifically, DOD’s recommendation to gain knowledge about the viability and maturity of commercial SATCOM system technologies corresponds with our best practices that outline the importance of ensuring needed technologies are proven to work as intended before programs begin. According to officials we spoke with from various DOD organizations involved in the Wideband AOA and SATCOM acquisitions, they have work ongoing that provides relevant information, including Air Force pathfinders and a study of ground infrastructure supporting WGS. However, these officials told us that there is no formal plan to guide post-AOA efforts including coordinating and providing the knowledge DOD needs to mitigate risks and inform timely decisions on DOD’s next wideband communications architecture. If DOD does not develop and implement a plan—including roles, responsibilities, and time frames—for building knowledge, then DOD risks not having enough information to make timely, knowledge-based decisions on systems that provide critical communications for military operations. For example, the Wideband AOA recommended developing an enterprise terminal strategy to centralize user terminal procurement. Without a plan to guide such an effort, it is unclear what organization within DOD would begin working with the military services to develop this strategy and potentially adjust the services’ acquisition approach to terminals. At the same time, it is important to note that DOD space acquisition is facing a changing leadership environment, and developing and implementing a plan for post-AOA efforts would need to take place in the midst of such changes. In 2016, we reported that for over 2 decades, fragmentation and overlap in DOD space acquisition management and oversight had led to ineffective and untimely decision-making, leading to delays in space system development and increasing the risk of capability gaps across critical weapons systems. DOD and Congress are taking steps designed to ultimately streamline decision-making and clarify authorities for space; however, it will likely take several years to implement such changes. Moreover, it is unclear the extent to which these changes will affect acquisition of user terminals—a long-standing challenge for DOD because the organizations responsible for buying terminals are not the same organizations that buy satellites. The changes being instituted include: Re-established United States Space Command. In August 2019, the President re-established the U.S. Space Command as a unified combatant command. DOD will form today’s Space Command with some offices from Strategic Command responsible for space operations, with the mission to protect and defend space assets. Although U.S. Space Command does not conduct space acquisitions, it is responsible for the satellite operators who help systems like WGS function—stakeholders in a post-WGS decision. Transferred commercial SATCOM procurement to Air Force Space Command. At the direction of the National Defense Authorization Act for Fiscal Year 2018, Air Force Space Command assumed responsibility for procuring commercial satellite communications for DOD in December 2018. The Defense Information Systems Agency previously managed most commercial SATCOM acquisitions and is still responsible for other types of ground segment systems. Proposed Establishment of a United States Space Force. Early in 2019, the President and DOD proposed the establishment of a U.S. Space Force as a sixth branch of the U.S. Armed Forces within the Department of the Air Force. The Space Force would include the uniformed and civilian personnel conducting and directly supporting space operations from all DOD armed forces, assume responsibilities for all major military space acquisition programs—including those for SATCOM, and create the appropriate career tracks for military and civilian space personnel. Congress is deliberating the final composition of the proposed Space Force. Established the Space Development Agency. In March 2019, DOD established the Space Development Agency to unify and integrate efforts across DOD to define, develop, and field innovative satellite solutions, including communications. The Space Development Agency is focused on a low-Earth-orbit constellation to provide communications and other satellite-based operational support for DOD, which could also provide information for selecting a post-WGS architecture. As of this time, DOD has not determined how this new organization will mesh with the Air Force Space and Missile Systems Center that acquires satellite systems; the Defense Advanced Research Projects Agency, which creates breakthrough technologies and capabilities; and similar organizations within the department. The Wideband AOA’s recommendations for gathering additional information to reduce risk and inform DOD’s decision-making are good first steps to ensure any post-WGS architecture will effectively and efficiently meet DOD’s needs. The addition of one or two more WGS satellites provides some extra time for DOD to field new satellites, avoid capability gaps, and implement the AOA recommendations. However, given the typical 7-year development timelines for space systems, DOD will need to decide on a way forward within the next several years so that new satellites will be available when needed. Attempting to implement the Wideband AOA recommendations without developing a plan for guiding multiple knowledge-building efforts across DOD raises risk that information gaps will not be closed in time to be useful or not closed at all. Consequently, it is important for DOD to coordinate these efforts and focus on how best to obtain a future wideband architecture that provides critical communications for military operations. The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment develop and implement a plan to guide and coordinate efforts to implement the Wideband AOA recommendations to support timely, informed decisions on its next wideband satellite communications architecture. (Recommendation 1) We provided a draft of this report to DOD for review and comment. DOD provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or by email at chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. The analysis of alternatives (AOA) process is an analytical study that is intended to compare the operational effectiveness, cost, and risks of a number of potential alternatives to address valid needs and shortfalls in operational capability. This process helps ensure that the best alternative that satisfies the mission need is chosen on the basis of the selection criteria, such as safety, cost, or schedule. GAO has identified 22 best practices for an AOA process by (1) compiling and reviewing commonly mentioned AOA policies and guidance used by different government and private-sector entities and (2) incorporating experts’ comments on a draft set of practices to develop a final set of practices. These practices can be applied to a wide range of activities and situations in which a preferred alternative must be selected from a set of possible options, as well as to a broad range of capability areas, projects, and programs. These practices can also provide a framework to help ensure that entities consistently and reliably select the project alternative that best meets the mission need. The guidance below is meant as an overview of the key principles that lead to a successful AOA process and not as a “how to” guide with detailed instructions for each best practice identified because each entity may have its own process in place. The 22 best practices that GAO identified are grouped into the following five phases: Initialize the AOA Process: includes best practices that are applied before starting the process of identifying, analyzing, and selecting alternatives. This includes determining the mission need and functional requirements, developing the study time frame, creating a study plan, and determining who conducts the analysis. Identify Alternatives: includes best practices that help ensure the alternatives that will be analyzed are sufficient, diverse, and viable. Analyze Alternatives: includes best practices that compare the alternatives selected for analysis in terms of costs, benefits, and risks. The best practices in this category help ensure that the team conducting the analysis uses a standard, quantitative process to analyze the alternatives. Document and Review the AOA Process: includes best practices that are applied throughout the AOA process, such as documenting in a single document all steps taken to initialize, identify, and analyze alternatives, selecting a preferred alternative, and independently reviewing the AOA. Select a Preferred Alternative: includes the final step of comparing alternatives and selecting a preferred alternative that best meets the mission need. The five phases address different themes of analysis necessary to complete the AOA process and comprise the beginning of the AOA process (defining the mission need and functional requirements) through the final step of the AOA process (select a preferred alternative). There are three key entities who are directly involved in the AOA process: the customer, the decision maker, and the AOA team. The customer refers to the group that implements the final decision (i.e. the program office, agency, and the like). A complex AOA process that impacts multiple agencies can have multiple customers. The decision maker is the person or entity who signs off on the final decision and analysis documented by the AOA report, and who will select the preferred alternative based on the established selection criteria. The decision maker should remain informed throughout the AOA process. For example, the decision maker could form a committee that consists of management and other groups independent of the AOA process who possess the required technical expertise or broad organizational knowledge to keep the decision maker apprised of and to inform the AOA process. The AOA team is the group involved in the day-to-day work of the AOA process and who conducts the identification and assessment of alternatives that is the foundation of the AOA process. We assessed the Department of Defense’s (DOD) Wideband Communication Services AOA against the “comprehensive” characteristic. Overall, the AOA met the six best practices we identified. Table 5 shows the relevant AOA best practices for the “comprehensive” characteristic. The Department of Defense (DOD) made the following recommendations in its Wideband Communications Services Analysis of Alternatives (AOA) report: 1. Immediately conduct a business case analysis that examines incorporating anti-jam and cybersecurity features that improve upon legacy capability into the Wideband Global SATCOM (WGS) Space Vehicle (SV) 11/12 procurement. 2. Investigate the impacts of WGS SV 11/12 to ground infrastructure, mission management, and user terminals to understand necessary modifications. 3. Develop and implement a DOD Enterprise Satellite Communications (SATCOM) Terminal Strategy that targets an approved Joint Information Environment architecture, reduces complexity of terminal diversity and programmatic governance, facilitates rapid modernization, and drives innovating business reforms, optimizing cost, schedule, and performance and interoperability. 4. Fund a purpose-built capability post-WGS SV 11/12 meeting user demands, including all weather capabilities, with a recommended start in fiscal year 2020, including consideration of alternate orbital regimes and approaches to cost-effectively meet needs while addressing proliferation, protection, and resiliency. The purpose is to ensure availability of DOD SATCOM resources to meet requirements where anticipated commercial offerings fail to materialize or are insufficient. 5. Continue efforts to invest in and shape commercial capabilities to support future DOD needs, including protection features, resilience, contested and all-weather capabilities, and polar coverage. Additionally, invest in and shape commercial industry development and risk reduction efforts focused on cybersecurity, terminal militarization/weapon system integration, management and control, technology assessment and development, and spectrum access. 6. Continue to fund existing and new SATCOM risk reduction efforts, evaluate blended commercial/military constellations, and expand the scope of pilots to include development of architectural standards and interface controls for enterprise management and control, terminal recapitalization plans, and means for terminals and/or weapon system platforms to transition satellite constellations and any DOD managed services. 7. Fund the design and implementation of a prototype wideband enterprise SATCOM management and control capability based on an approved Joint Information Environment architecture that integrates the management of Military, Commercial, and International Partner- provided SATCOM services and networks and supports the Enterprise Operational Management requirement in the Joint Space Communications Layer Initial Capabilities Document Change 1. 8. Plan for investment in Protected Tactical Waveform capabilities to commercial and military band terminals to align with the Protected Anti-Jam Tactical SATCOM planned ground and space milestones. 9. Fund pilot efforts to identify risks and opportunities to use commercially-managed services for Army’s Combat Support Logistics Very Small Aperture Terminals and ways to mitigate that risk. 10. Pursue partnership opportunities with Norway and Canada to achieve earlier Arctic coverage capability. Cristina T. Chaplain, (202) 512-4841 or chaplainc@gao.gov. In addition to the contact named above, Rich Horiuchi, Assistant Director; Burns C. Eckert (Analyst in Charge); Erin Cohen; Emile Ettedgui; Jon Felbinger; Kurt Gurka; Stephanie Gustafson; Jennifer Leotta; Roxanna Sun; and Jay Tallon made key contributions to this report.", "summary": "DOD officials estimate spending an average of $4 billion each year to acquire and sustain wideband satellite communications that provide fast and reliable voice, video, and data transmissions critical to military operations. DOD is considering how to meet its future wideband needs across many different operating environments and scenarios. The National Defense Authorization Act for Fiscal Year 2016 required DOD to conduct a Wideband Communications Services AOA to identify ways to replace current systems as the satellites reach the end of their service lives. The National Defense Authorization Act for Fiscal Year 2017 contained a provision for GAO to assess DOD's analysis. This report addresses (1) whether the Wideband AOA was comprehensive, (2) how DOD solicited input from stakeholders, and (3) the conclusions DOD reached through the Wideband AOA. GAO reviewed the Wideband AOA along with DOD policies, documentation, and analyses; interviewed DOD officials and commercial stakeholders; and assessed the AOA against best practices for a comprehensive AOA process. The Department of Defense (DOD) conducted a comprehensive analysis of alternatives (AOA) process for wideband satellite communications, as determined through an assessment of the AOA against relevant GAO best practices. A comprehensive analysis of alternatives process indicates that the analysis team thoroughly addressed a wide range of possible satellite system alternatives. DOD used multiple methods to obtain stakeholder input, in accordance with its Wideband AOA study plan. For example, the study team incorporated input from across the military services and operational users, among others. Moreover, the Air Force and Defense Information Systems Agency conducted interrelated studies to provide additional information to the Wideband study team. DOD's analysis concluded that integrating military and commercial systems into a hybrid architecture would be more cost effective and capable than either acquisition approach alone. However, DOD also found that it needs more information to select its next satellite communications architecture and made recommendations for further study. Examples of these recommendations include: Develop an enterprise satellite communications terminal strategy – DOD found the magnitude of replacing user terminals to work with new systems was challenging and that more information on emerging technology and possible changes to terminal acquisition approaches would help DOD address this challenge. Invest in commercial technologies – DOD found that it lacked detailed technical information on commercial systems' cyber protections and that additional information on such protections would help DOD determine the extent to which they would meet DOD's needs. Such recommendations align with GAO's acquisition best practices for knowledge-based decision-making and have the potential to improve the department's satellite communications acquisitions. However, DOD stakeholders said there is no formal plan to guide and coordinate implementation of the AOA recommendations. Without such a plan, DOD is at increased risk of not having the information it needs to make timely, knowledge-based decisions on future systems to provide critical communications for military operations. GAO is recommending that DOD develop a plan to guide implementation of the Wideband AOA recommendations. DOD provided technical comments on a draft of this report, which GAO incorporated as appropriate.", "document_type": "gao"}
{"report": "DOD’s travel pay program is comprised of payments made by the department to active, reserve, and National Guard service members and civilian employees for temporary and permanent travel expenses. DOD travel is generally documented using authorizations and vouchers. Travel authorizations direct an individual or group of individuals to travel and provide information regarding what travel expenses are authorized to be paid. Travelers submit travel vouchers after the travel is completed to claim reimbursement for the official travel expenses they have incurred. There are a number of DOD entities involved in creating, reviewing and approving, paying, and reporting on DOD travel payments: Travelers are the service-members and civilian employees engaging in travel who create, amend, and digitally sign travel authorizations and vouchers and are legally liable for submitting false or fraudulent claims for payment. Authorizing officials are responsible for authorizing travel and controlling the use of travel funds. The DTS Regulations state that authorizing officials must review, verify, and approve authorizations prior to travel. Certifying officers certify vouchers for payment. According to the DOD guidance on DTS, known as the DTS Regulations, certifying officers must implement, maintain, and enforce internal procedures and controls to minimize erroneous payments; they are presumed negligent and may be pecuniarily liable for all improper payments that they certify. Authorizing officials who are also certifying officers review and certify travel vouchers and verify all required supporting documentation before the vouchers are paid. The Defense Travel Management Office (DTMO) oversees and facilitates DTS, including any necessary changes or enhancements to the system. It establishes and maintains the DTS Regulations, which define the responsibilities of users by role and the minimum required training for each user role, among other things. DTMO also maintains DTS travel payment data that are used for estimating and reporting on improper payments. The Defense Finance and Accounting Service (DFAS), as part of DOD’s efforts to reduce improper travel payments, is responsible for reviewing a sample of paid DTS travel vouchers to estimate and report improper travel payments. DFAS also provides data on improper travel payments to DOD components on a quarterly basis. The Office of the Under Secretary of Defense (Comptroller) compiles DOD-wide data on improper payments annually as part of DOD’s Agency Financial Report. It also oversees and facilitates DOD efforts to reduce improper travel payments. The Improper Payments Information Act of 2002, which was later amended by the Improper Payments Elimination and Recovery Act of 2010 (IPERA) and the Improper Payments Elimination and Recovery Improvement Act of 2012 (IPERIA), defines an improper payment as any payment that should not have been made or that was made in an incorrect amount (including overpayments and underpayments) under statutory, contractual, administrative, or other legally applicable requirements. In accordance with OMB guidance, DOD has identified travel pay as susceptible to improper payments based on the large volume of transactions and high dollar amount of the program. As a program considered susceptible to significant improper payments, DOD travel pay is subject to certain IPIA requirements. Specifically, IPIA, as amended, requires federal executive branch agencies to (1) develop a statistically valid estimate of the annual amount of improper payments for programs identified as susceptible to significant improper payments, (2) implement corrective actions to reduce improper payments and set reduction targets, and (3) report on the results of addressing these requirements. IPERA also requires executive agencies’ Offices of Inspector General to annually determine and report on whether their agencies complied with certain IPERA-related criteria. These criteria include the requirements to publish a report for the most recent fiscal year that meets OMB reporting requirements, publish statistically valid improper payment estimates, publish and meet reduction targets for improper payment rates, and publish corrective action plans. If an agency does not meet one or more of the six IPERA criteria for any of its programs or activities, the agency is considered noncompliant with IPERA. The DODIG reported that in fiscal year 2018, DOD travel pay was not in compliance with IPIA, as amended, for the seventh consecutive year. Specifically, DOD met three of the six IPERA-related criteria for its travel pay program, by publishing all required information in the Payment Integrity section of its Agency Financial Report; conducting program-specific risk assessments; and reporting an improper payment rate of less than 10 percent for each of the eight programs that included an improper payment estimate in the fiscal year 2018 Agency Financial Report. However, the DODIG reported that DOD did not publish reliable improper payment estimates, include all required elements for the descriptions of corrective action plans, or meet its targets for reducing improper payments. To meet IPIA requirements, agencies follow guidance issued by OMB for estimating improper payments. OMB Circular No. A-123, Appendix C instructs agencies to obtain the input of a statistician to prepare a statistical sampling and estimation method that produces statistically valid estimates of improper payments. Agencies are required to meet a number of requirements on the content of the sampling plans, including providing clear and concise descriptions of the methods used that also address the assumptions used, sample sizes, and precision, among other aspects. The guidance also says that agencies should incorporate refinements to their methods based on recommendations from agency staff or auditors, such as their agency Inspector General or GAO, whenever possible. OMB guidance also includes requirements for annual reporting on improper payment estimates. According to the guidance, when calculating a program’s annual improper payment amount, agencies should use only the amount paid improperly. For example, if a $100 payment was due, but a $110 payment was made erroneously, then the amount applied to the annual estimated improper payment amount should be $10. In addition, when an agency’s review is unable to discern whether a payment was proper as a result of insufficient or lack of documentation, this payment must also be considered an improper payment. OMB also requires agencies to identify and report on the root causes of the improper payments and implement corrective actions to prevent and reduce these causes for programs that have been identified as susceptible to significant improper payments, including DOD’s travel pay program. OMB emphasizes that, in identifying root cause, it is important to distinguish between what constitutes a root cause that created an error and an internal control problem that failed to catch an error. The guidance instructs agencies to implement corrective actions that are responsive to root causes, are proportional to the severity of the associated amount and rate of the root cause, and are measurable. It also instructs agencies to annually review their existing corrective actions to determine whether any action can be intensified or expanded to achieve its intended result. To comply with IPIA and OMB requirements, and in response to our prior recommendations, DFAS updated its statistical sampling plan in fiscal year 2017 to develop and report improper payment estimates for DTS. The plan is designed to estimate the dollar amount of improper payments, which includes both travel payments that were made in excess of the correct amount (overpayments) and those that were made for less than the correct amount (underpayments). When DOD is unable to discern whether a travel payment is proper because there is insufficient or no documentation to support it, that payment is also included in the improper payment estimate. On a monthly basis, DFAS statistically samples paid travel vouchers, stratified first by component and then by dollar amount. DFAS officials then conduct a review of the sampled post-payment vouchers to identify erroneous travel vouchers and the types of errors that were made. Based on the errors found during the review, DFAS calculates an estimate of the improper payments for each component. The military services process a small portion of their travel payments through other disbursing systems and are responsible for conducting their own post-payment reviews to estimate the improper payments for those systems. The DOD improper payment rate is the estimated total of improper payments from all post- payment reviews divided by the total number of payments. For example, in fiscal year 2018, DOD reported an improper payment rate of 4.59 percent, or $365.32 million of the $7.96 billion total travel payments. Using DTS data, we calculated that DOD had spent an average of $6.1 billion annually on DTS travel payments in fiscal years 2016 through 2018—a total of about $18.3 billion in travel payments for those years. Travel for active duty servicemembers accounted for the largest portion of those travel payments. We calculated that DOD components reported over $9.5 billion in DTS travel payments for active duty servicemembers in fiscal years 2016 through 2018, accounting for approximately 52 percent of the total travel payments. For the same time period, DTS travel payments for DOD civilian employees totaled about $5.3 billion (29 percent of the total), and travel payments for Reserve and Guard members totaled about $3.5 billion (19 percent of the total) (see fig. 1). DOD data on DTS travel payments show that out of 10 different categories used to identify the purpose of travel, the category representing “training” accounted for the largest percentage of the travel payments. Payments for “training attendance” accounted for about $6.6 billion (36 percent) of the $18.3 billion in total travel payments for fiscal years 2016 through 2018 (see table 1). Payments for the trip purpose “other travel” accounted for about $3.1 billion (17 percent) of the total travel payments for that time period. “Other travel” is any travel for reasons not covered by the other trip purpose categories; the purpose must be further specified in the travel authorization. Based on our analysis, most travel categorized as “other travel” was further specified with the trip type “routine TDY,” which refers to a travel assignment to a location other than the employee’s permanent duty station. The two other trip purposes that accounted for the highest percentage of travel payments, based on our analysis of the DTS data, are “special mission” and “site visit,” which each accounted for about $2.9 billion (16 percent) of the total travel payments for fiscal years 2016 through 2018. Using DTS data, we also calculated that DOD’s reported total travel payments increased from fiscal years 2016 through 2018, for a total increase of approximately $1 billion (16 percent) in nominal dollars and $0.68 billion (11 percent) in constant dollars during fiscal years 2016 through 2018 (see fig. 2). The DOD officials we interviewed were unable to explain why travel payments increased during fiscal years 2016 through 2018 but speculated that overall increases in DOD’s budget likely corresponded with additional travel expenses. Officials also stated that travel expenses are tied to DOD’s mission requirements. For instance, DOD military and civilian personnel provided support to civil authorities in areas such as humanitarian assistance and disaster recovery during the period of our review, according to these officials. Travel by DOD personnel to locations for these missions would contribute to DOD’s travel expenses. According to data provided by DFAS, the annual average of DOD improper travel payments was about $322 million for fiscal years 2016 through 2018, totaling $965.5 million (or 5.3 percent of total DTS travel payments) for those years. For fiscal year 2016, DFAS calculated that an estimated $416.6 million in travel payments (7.3 percent of total fiscal year 2016 DTS travel payments) were improper. For fiscal year 2017, DFAS’s estimate of improper payments was $252.4 million (4.2 percent of total fiscal year 2017 DTS travel payments). However, data availability issues limited the scope of that year’s post-payment review, which is used to estimate the improper payment rate. For fiscal year 2018, DFAS’s estimate of improper payments was $296.6 million (4.5 percent of total fiscal year 2018 DTS travel payments). These improper payment amounts include both overpayments and underpayments and do not necessarily indicate a monetary loss to the government. According to DOD’s Agency Financial Report, payments identified as improper do not always represent a monetary loss. For instance, an otherwise legitimate payment that lacks sufficient supporting documentation or approval is reported as improper but is not considered a monetary loss if documentation or approval is subsequently provided. Monetary loss is an amount that should not have been paid and could be recovered. With respect to monetary loss, DFAS calculated that of the DTS improper payments, the department incurred an estimated $205 million (1.6 percent of total DTS travel payments) loss to the government for fiscal years 2017 and 2018 (see fig. 3). Specifically, for fiscal year 2017, DFAS calculated an estimated monetary loss of $97.7 million (1.6 percent of total DTS travel payments), and for fiscal year 2018, it calculated an estimated monetary loss of $107.3 million (1.6 percent of total DTS travel payments). According to DFAS officials, the monetary losses estimated by DFAS were a result of travel voucher errors such as claiming an expense that is automatically generated by DTS during the booking process, rather than updating the travel voucher with the amount actually paid. Other errors that DFAS considers to indicate a monetary loss to the government include duplicate paid vouchers, mileage paid incorrectly, lodging expenses paid twice, and expenses that do not match the receipts (e.g., lodging). DOD established and has taken steps to implement a Remediation Plan aimed at reducing improper travel payments that includes specific requirements for all DOD components as well as a committee to monitor the efforts of 10 components that DOD identified as key to addressing improper travel payments. However, DOD did not consider available data on improper travel payment rates in its selection of these 10 components to implement its risk-based approach. Further, the 10 components have not fully implemented the Remediation Plan requirements, and other components were generally unaware of the requirements in the Remediation Plan and DOD’s broader efforts to resolve and mitigate improper travel payments. In October 2016, DOD established a Remediation Plan for improper payments in its travel pay program. The memorandum establishing the plan specified that it applied to the Military Departments, Defense Agencies, Joint Staff, and Combatant Commands. The Under Secretary of Defense (Comptroller) noted in the memorandum that the rate of improper travel payments had reached an unacceptable level, causing the department’s program for preventing improper payments to be non- compliant with IPERA. Accordingly, the Remediation Plan specified steps that DOD components were required to take to reverse the department’s poor performance. Specifically, it stated that the military services, defense agencies, DOD field activities, Joint Staff, and combatant commands must each designate in writing a Senior Accountable Official (SAO) responsible for implementing the plan’s requirements for that component, train travelers and approving officials, issue guidance on holding approving officials pecuniarily liable for improper travel payments, and prepare component-specific remediation plans and identify corrective actions, among other things. DOD specified that certain steps were to be completed by November 1, 2016. The requirements specified in DOD’s Remediation Plan are listed in table 2. DOD officials informed us that they also established a Senior Accountable Official Committee (SAO committee) consisting of the SAOs from the 10 components. The committee provided a mechanism for DOD’s Deputy Chief Financial Officer to monitor the implementation of the Remediation Plan’s requirements by those components. The SAO committee included the four military services and six additional components: the U.S. Special Operations Command, the Defense Logistics Agency, the Defense Contract Management Agency, the Defense Information Systems Agency, the Missile Defense Agency, and the Defense Contract Audit Agency. An Office of the Under Secretary of Defense (OUSD) (Comptroller) official told us that DOD did not monitor the implementation of other components’ efforts to implement the Remediation Plan’s requirements. The SAO committee met four times from January 2017 through September 2017, with a fifth meeting in May 2018. At these meetings, components represented on the committee discussed approaches they had taken to prevent improper travel payments and highlighted examples of best practices to educate travelers and approving officials about how to avoid improper travel payments. In addition, DFAS officials presented the results of monthly post-payment reviews to identify the most common errors associated with improper travel payments. In June 2018, DOD broadened the scope of the SAO committee and chartered the DOD Improper Payments Senior Accountable Officials Steering Committee, which was established to address all programs included in DOD’s improper payments reporting—not just travel pay. As of May 2019, this steering committee had met twice, in December 2018 and again in March 2019. DOD identified components to include on the SAO committee based on fiscal year 2016 DTS travel payments but did not consider components’ improper payment rates as selection criteria. According to OUSD (Comptroller) officials, DOD used a risk-based approach to select the 10 components to include in the SAO committee, because these components accounted for the significant majority of the department’s DTS travel payments. However, as a result of the way in which DOD reports its estimated rates of improper travel payments, it is unclear whether there is an association between the volume of DTS travel payments and improper travel payment rates. DOD officials told us that they did not use estimated improper travel rates as a selection criterion because DFAS does not report estimated improper payment rates for all DOD components in its annual agency financial report. Instead, DFAS uses a stratified sampling method for the post-payment review of travel vouchers, which means that the sample sizes for certain individual components—such as smaller defense agencies—may be too small to be statistically reliable. As a result, DFAS reports improper payment rates for the individual military services and U.S. Special Operations Command, but it reports an aggregate rate for the defense agencies that DFAS officials told us also includes “joint commands.” Notwithstanding DOD’s current sampling approach for determining improper payment rates, DOD has previously reported discrete improper payment rates for components that are not represented on the SAO committee, and there may be additional data sources on component- specific improper payment rates. First, a 2016 DODIG report on improper travel payments presented the results of a DFAS review of DTS vouchers for 58 DOD components for July through December, 2014, including 48 components not represented on DOD’s SAO committee. Second, DOD has reported improper payment rates for specific components other than the military services as part of the Remediation Plan effort. Specifically, DFAS has reported an improper payment rate for U.S. Special Operations Command in the quarterly reports it provided to the SAO committee separately from the aggregate rate it reports for other “joint commands.” Third, we found that other sources of data on estimated improper travel payment rates may be available to the department. For example, of the non-SAO components that responded to our survey, 7 of 28 indicated that they track their rate of improper travel payments. Because DOD’s approach to monitoring specific components’ implementation of the Remediation Plan was based solely on the amount of DTS travel payments, DOD lacks assurance that the components it selected for greater scrutiny were the ones most at risk for improper travel payments. Standards for Internal Control in the Federal Government notes that management can define risk tolerances for defined objectives, specifically the acceptable level of variation in performance relative to the achievement of objectives. Federal internal control standards also state that agencies should evaluate whether a risk-based approach is appropriately designed by considering whether it is consistent with expectations for the defined objectives. If the approach is not consistent with expectations, agencies should revise the approach to achieve consistency. In this case, DOD decided to accept the risk associated with targeting its Remediation Plan efforts to only those components that accounted for most of the department’s total travel payments in fiscal year 2016. However, without including improper payment rates in its analysis, DOD may have excluded components with lower overall travel payments that had significant improper payment rates. As a result, DOD cannot be assured that it has implemented the Remediation Plan in a way that is both efficient and effective in reducing improper travel payments. The 10 components that make up the SAO committee and were identified as key to the effort to reduce improper payments took some steps to address the Remediation Plan requirements but did not complete all of the requirements outlined in the Plan. For example, 7 of the 9 components that responded to our survey reported that they had designated an SAO. Further, these components indicated that their SAOs had completed some required steps, such as issuing guidance to ensure that front-end internal controls were in place to prevent improper travel payments; reviewing training plans to determine their effectiveness in preventing improper travel payments; and providing initial or refresher training to all travelers and approving officials, among other actions. However, none of the components that responded to our survey had completed all of the requirements by the due date of November 1, 2016. As of March, 2019, when we surveyed the 10 DOD components, only four of the 9 components that responded to our survey had completed all of the requirements (see table 3). For instance, 1 component (the Defense Information Systems Agency) had not developed a component-level remediation plan, and 6 of the 10 components had not developed corrective action plans to address the improper travel payments they identified, as required by the Remediation Plan. OUSD (Comptroller) officials told us that they required only the military services to complete corrective action plans, because these components accounted for about 92 percent of DTS travel payments. We found that, while DOD established specific milestones for certain actions in the Remediation Plan, it did not establish milestones for completing most of the actions. Specifically, as shown in table 2 earlier in this report, only 5 of the 11 requirements in the Remediation Plan had an associated due date. Further, while DOD established a mechanism to monitor whether the components had implemented the Remediation Plan requirements through the SAO committee, this mechanism was not effective in holding them accountable for doing so. For example, at the first SAO committee meeting (January 18, 2017), the SAOs were told to complete the Remediation Plan requirements by March 1, 2017, and to be prepared to discuss them at the next SAO committee meeting. However, at the next meeting (March 29, 2017), only 3 components—the Navy, the Defense Information Systems Agency, and the Defense Logistics Agency—were prepared to present their component-level remediation plans to the committee. At the meeting, the DOD Deputy Chief Financial Officer, serving as the chair of the committee, emphasized that components needed to document progress in order to demonstrate that the department was working toward identifying root causes and implementing corrective action plans to prevent and reduce improper travel payments. At the May 24, 2018 SAO committee meeting, 3 additional components—the Air Force, the Defense Contract Management Agency, and the Missile Defense Agency— presented their plans to the committee. However, as of March 2019, the U.S. Special Operations Command and the Defense Contract Audit Agency had still not presented their plans to the committee. Standards for Internal Control in the Federal Government states that management should evaluate performance and hold individuals accountable for their internal control responsibilities. Accountability is driven by the tone at the top of an organization and supported by the commitment to integrity and ethical values, organizational structure, and expectations of competence, which influence the control culture of the entity. In addition, the standards state that management should establish and operate monitoring activities to monitor the internal control system and evaluate the results. As we stated earlier in this report, DOD has been challenged by inaccurate and inconsistent estimates of improper payment rates, which do not allow for reliably tracking the rate of improper travel payments over time. By establishing milestones, monitoring progress, and holding component leadership accountable for the implementation of the requirements of the Remediation Plan, DOD would have greater assurance that it has taken sufficient actions to reduce improper travel payments. As we noted above, the department memorandum outlining the Remediation Plan was addressed to all components, and DOD officials confirmed that, although they monitored implementation of the Remediation Plan for the 10 components represented on the SAO committee, the 42 components not represented on the SAO committee (non-SAO committee components) were still required to complete the actions specified in the Plan. However, we found that, based on our survey results, half of the components that responded to our survey were unaware of the requirements established in the Remediation Plan. Of the 28 non-SAO committee components that completed our survey, 14 (50 percent) responded that they were either not at all familiar with the Remediation Plan requirements or only slightly familiar with the requirements. Our survey results and review of DOD documentation also indicate that many of the 42 non-SAO committee components had taken some steps to reduce improper payments, consistent with the Remediation Plan requirements, but had not completed all of the Plan’s requirements. For example, of the 28 non-SAO committee components that completed our survey, 10 (36 percent) responded that they had not designated an SAO or other lead entity in writing, and 8 (29 percent) did not know whether their component had designated a SAO. Our survey results also indicate that most of the components not represented on the SAO committee who responded to our survey were unaware of department efforts to prevent and reduce improper travel payments. Specifically, many of the non-SAO committee components had not been made aware of efforts to implement the Remediation Plan across the department through mechanisms such as the SAO committee meeting minutes or quarterly DFAS reports. Sixteen of the 28 non-SAO committee components who responded to our survey reported that no one from their organization had ever attended an SAO committee meeting, and 11 responded that they did not know if anyone from their component had attended. Further, 15 of the 28 components who responded to our survey reported that they had never received a copy of the official SAO committee meeting minutes, and 13 responded that they did not know whether they had. Nine of the 28 components responded that they did not receive copies of the DFAS quarterly reports on improper payments, which are used to track the types of errors that occur in travel payments and help components to target actions to address them. Standards for Internal Control in the Federal Government states that management should internally communicate the necessary quality information to achieve the entity’s objectives. Communicating quality information down, across, up, and around reporting lines to all levels of an entity contributes to the design, implementation, and operating effectiveness of this principle. An OUSD (Comptroller) official confirmed that DOD did not take action to share information on the Remediation Plan requirements or implementation efforts with components not represented on the SAO committee. When DOD made the decision to focus the SAO committee on 10 components, it did not establish a mechanism or document how information on Remediation Plan efforts would be communicated to the non-SAO committee components, which are also required to implement the Plan. As a result, the components that are not represented on the SAO committee have not benefited from information on the Plan’s requirements or lessons learned and best practices that were identified during the SAO committee effort—which may have helped them to reduce their improper payments. Providing opportunities for all components to benefit from the Remediation Plan efforts would give DOD greater assurance that it has taken steps to reduce its overall improper payment rate. DOD has established mechanisms to identify and address the errors that most frequently lead to improper travel payments, but we found some limitations with these mechanisms because they did not consistently identify the root causes of the errors. DTMO Compliance Tool. In response to a requirement in the National Defense Authorization Act for Fiscal Year 2012, DTMO developed a compliance tool that uses a set of digital queries to automatically review vouchers submitted for payment through DTS to determine whether they meet criteria that indicate the potential for improper payment. According to DTMO, as of fiscal year 2018, the tool had recovered $25 million over 5 years. If a voucher is flagged by this tool, an email is automatically generated to the traveler and approving official associated with that voucher with instructions for correcting the error. For example, the compliance tool flags vouchers with duplicate expenses, such as expenses for lodging or rental cars. However, the tool does not flag all potential improper payments, because it does not identify all types of voucher errors. For instance, according to DTMO officials, the tool cannot identify vouchers that have been submitted without required receipts. For fiscal year 2018, the average rate for DTS vouchers identified as erroneous by the DTMO compliance tool was 0.044 percent. In contrast, DOD reported an improper payment rate of 4.5 percent for DTS vouchers in fiscal year 2018. In addition, the tool does not identify the root causes leading to those errors. Rather, the tool simply notifies the traveler and approving official associated with a specific voucher with characteristics indicative of a potential improper payment and requests that they amend the voucher to remove any errors. DFAS Sampling. Each month, DFAS selects a sample of vouchers that have been processed in DTS and assigns staff to review those vouchers to determine whether any resulted in an improper payment. According to DFAS officials, DFAS provides the results of these reviews to the components represented on the SAO committee. DFAS also prepares quarterly reports that summarize the most frequent errors that lead to improper travel payments and presents these reports for discussion at SAO committee meetings. DFAS reports the frequency of voucher errors for each military service and U.S. Special Operations Command and an aggregate rate for defense agencies and joint commands. The DFAS reports also suggest corrective actions to address the identified errors. For example, in November 2018, DFAS reported that the voucher error leading to the third largest amount of improper payments was “Lodging—Paid Without a Receipt,” which accounted for a total of $21,810 in improper payments in that month. The corrective action DFAS suggested was for reviewers or approving officials to verify that receipts were uploaded to DTS and that any uploaded receipts met the criteria for valid receipts. If either of these conditions was not met, the reviewer was to return the voucher to the traveler to correct and resubmit. However, these corrective actions did not address the root causes of those errors. Specifically, neither DFAS nor the SAO committee determined why travelers were not uploading receipts for lodging expenses or why officials were approving vouchers without receipts. According to DFAS reports, errors related to missing lodging receipts were among the top 5 errors from October 2016 through June 2017. By December 2018, these were was the most common errors DFAS identified—accounting for a total of $53,125 in improper payments in that month—yet DOD did not develop corrective actions to address the root cause (i.e., why travelers were continuing to submit vouchers without lodging receipts). SAO Committee Effort. As we discussed earlier in this report, beginning in January 2017, OUSD (Comptroller) convened five meetings of the SAOs from 10 components that, according to officials, accounted for the majority of DOD travel payments in fiscal year 2016. At these meetings, representatives from the components discussed approaches they were using to reduce improper travel payments. In addition, representatives from DTMO and DFAS presented trends resulting from their efforts to identify improper travel payments using the DTMO Compliance Tool and DFAS post-pay sampling. These presentations conveyed information about the types of voucher errors that were leading to improper travel payments, and SAOs in attendance discussed how to mitigate those errors. However, our review of SAO Committee meeting minutes and the remediation plans prepared by those components represented on the committee found that the components did not identify the root causes of errors leading to improper travel payments. Military Services’ Corrective Action Plans. The military services, in coordination with OUSD (Comptroller), developed corrective action plans to address improper travel payments. OUSD (Comptroller) provided the military services with guidance on developing the corrective action plans that states that corrective action plans are required to reduce improper payments, as well as to address specific audit recommendations and issues of IPERA non-compliance. OUSD (Comptroller) also provided the military services with a corrective action plan template that instructs them to describe what the plan is intended to address, i.e., improper payments, a specific audit recommendation, or noncompliance issues. The template also defines root causes as “underlying issues that are reasonably identifiable, can be controlled by management, and require implementing corrective actions to mitigate.” As of May 2019, the military services had prepared 12 corrective action plans for the travel pay area. However, we found that only 4 of them included specific corrective actions addressing the root causes of improper travel payments. We also found that the plans varied in terms of their sophistication in discussing and identifying root causes. For example, none of the corrective action plans prepared by the Air Force targeted the root causes of improper travel payments. By contrast, one of the Navy’s corrective action plans clearly identified the root cause of an error (vouchers being approved without the required forms) and specified 10 milestones and associated corrective actions to address the root cause. Of the Army’s two corrective action plans, one addressed weaknesses in the Army’s sampling plan for determining improper payments at overseas offices but did not discuss identifying the root causes of improper travel payments, and the other required Army travel management officials at overseas offices to improve their reporting of improper travel payments to more clearly link corrective actions with root causes. While DOD has taken some positive steps to identify the errors that most frequently lead to improper travel payments, our review found that component officials do not have a clear understanding of what constitutes the “root cause” of an improper travel payment. For example, component officials who responded to our survey consistently mischaracterized root causes as the specific errors leading to improper payments (e.g., missing receipts) rather than the underlying reasons for those errors. Our survey asked respondents if their component had taken steps to identify the root causes of voucher errors that led to improper travel payments in fiscal year 2018 and, if so, to provide examples of root causes they had identified. While 31 of the 37 (84 percent) components that responded to the question indicated that they had taken steps to identify root causes, and 28 (76 percent) indicated that they had taken steps to address those identified root causes, open-ended survey responses indicated that the components did not understand the term “root cause.” Specifically, 24 of the 31 (77 percent) components that provided open-ended responses with examples of the root causes they identified cited voucher errors—such as missing receipts—rather than identifying the root causes for why those errors occurred. This indicates that the 31 components that responded to this question did not understand the term “root cause”. It also suggests that the number of components that actually took actions to address root causes is likely significantly lower than the numbers reported by the survey respondents. OMB guidance specifies that agencies should ensure they have identified a true root cause of an improper payment, because it is critical to do so in order to formulate effective corrective actions. DOD’s Financial Management Regulation (FMR) states that root causes of improper payments must be identified and corrective plans developed and monitored on a regular basis to ensure that future improper payments will be reduced and eliminated. However, neither DOD’s FMR nor the June 2018 charter for the DOD Improper Payments SAO Steering Committee defines the term “root cause.” And while DOD has established some mechanisms to try to help components identify root causes, our survey demonstrates that many travel management officials at DOD components do not clearly understand the meaning of root cause. Specifically, of the 31 components that provided examples of what they believed to be the root causes of voucher errors, only 7 provided examples of actual root causes. Until DOD defines the term “root cause” to ensure a common understanding of the term across the department, DOD travel management officials will likely miss opportunities to make changes that could help to address the underlying causes of improper travel payments. All of the corrective action plans prepared by the military services that are intended to identify root causes of improper travel payments specified the costs associated with implementing the corrective actions. While many of the actions do not fully address root causes, as previously discussed, it is important that the department weighs the cost-effectiveness of its actions. However, we found that the services had not incorporated a consideration of cost-effectiveness into their decisions on whether to implement those actions, at least in part because OUSD (Comptroller) had not provided guidance on how they should assess the cost-effectiveness of potential corrective actions. Specifically, the template OUSD (Comptroller) provided to the military services for preparing corrective action plans neither asked for information on costs nor specified how to determine the cost-effectiveness of specific corrective actions. In May 2019, an OUSD (Comptroller) official told us that DOD is considering formulating guidance on how components should determine cost-effectiveness. OMB guidance states that agencies should be able to measure the effectiveness and progress of each individual corrective action on an annual basis. The guidance further states that agencies should annually review their existing corrective actions to determine if any existing action can be intensified or expanded so that it results in a high return on investment in terms of reduced or prevented improper payments. Addressing the root causes of improper travel payments can be costly, requiring investments in technology changes, among others. For example, component officials whom we interviewed and who responded to our survey indicated that several of the root causes for improper travel payments were related to design flaws in DTS. According to DOD officials, a feature of DTS called “Trip Workbook” is used by travelers to upload and attach receipts to vouchers. However, “Trip Workbook” is not visible to approving officials when they process the voucher for approval and payment. As a result, vouchers are being approved without the required receipts, because approving officials cannot determine whether or not the receipts have been attached. Officials stated that changes to DTS are often costly and can take a long time, and in some instances they can be more costly than the improper payment amounts they are intended to reduce. Without clear guidance to assist components in determining whether proposed corrective actions are cost-effective to implement, DOD travel management officials will be hampered in making informed decisions about which actions to implement and which to leave unfunded. DOD spent about $6 billion annually in DTS travel payments from fiscal years 2016 through 2018 for its personnel to travel in support of its mission, but since 2012 the DODIG has consistently found the DOD travel program to be non-compliant with statutory requirements to mitigate improper payments. In 2016, DOD began implementing a Remediation Plan to address weaknesses in its management of improper travel payments. However, DOD did not consider component-specific improper payment rates in addition to overall travel payments when developing its risk-based approach to monitoring the implementation of the Plan. Thus, DOD lacks assurance that the components it selected for greater scrutiny were the ones most at risk for improper travel payments. Further, even the components that DOD determined were critical to implementing the Remediation Plan did not fully implement the Plan’s requirements, because DOD had not established milestones for completing all of the requirements, monitored whether the components had completed them on time, or held them accountable for completing the requirements. In addition, DOD did not establish a mechanism to share the results of the SAO committee’s initiatives to reduce improper payments with travel management officials across the department, limiting opportunities for the components that were not represented on the SAO committee to benefit from Remediation Plan efforts. DOD has taken some positive steps to identify the errors associated with improper travel payments but can do more to effectively and efficiently address the underlying root causes. First, DOD has not established a common definition of root cause so that travel management officials across the department can clearly identify actions needed to address improper travel payments. In the absence of such a definition, the department is limited in its ability to address the underlying reasons for improver travel payments. Second, DOD components lack guidance to assist them in determining the cost-effectiveness of addressing root causes of improper travel payments. Such guidance would help to provide assurance that investments are targeted to actions that are cost effective to implement. We are making five recommendations to DOD. The Secretary of Defense should ensure that the Under Secretary of Defense (Comptroller) revises the approach for selecting components to implement the DOD Travel Pay Improper Payments Remediation Plan to consider available improper payment rate data in addition to data on the components’ amount of travel payments. (Recommendation 1) The Secretary of Defense should ensure that the Under Secretary of Defense (Comptroller) expedites completion of the remaining Travel Pay Improper Payments Remediation Plan requirements by establishing milestones for the requirements, monitoring whether the components have completed them on time, and holding components accountable for completing the requirements. (Recommendation 2) The Secretary of Defense should ensure that the Under Secretary of Defense (Comptroller) establishes a mechanism to share the results of the SAO committee’s initiatives to reduce improper travel payments with all appropriate travel management officials across the department. (Recommendation 3) The Secretary of Defense should ensure that the Under Secretary of Defense (Comptroller) takes action to ensure a common understanding of the concept of root cause across the department. This could be done by, among other actions, revising the Financial Management Regulation or the charter for the DOD Improper Payments SAO Steering Committee to include a definition of the term and including a definition of the term in the mechanism used to share the results of the SAO committee’s initiatives to reduce improper travel payments with travel management officials across the department. (Recommendation 4) The Secretary of Defense should ensure that the DOD Deputy Chief Financial Officer directs the chairs of the SAO Committee, with the input of OUSD (Comptroller), DTMO and DFAS, to provide guidance to the components on how to determine whether actions that would address root causes are cost effective to implement. (Recommendation 5) We provided a draft of this report to DOD for review and comment. In its written comments, reproduced in appendix III, DOD did not concur with our first recommendation, partially concurred with our second and fifth recommendations, and concurred with our third and fourth recommendations and outlined its plan to address them. DOD also provided technical comments, which we incorporated in the report where appropriate. In non-concurring with our first recommendation that OUSD (Comptroller) revise the approach for selecting components to implement the DOD Travel Pay Improper Payments Remediation Plan (Remediation Plan) to consider available improper payment data in addition to the amount of travel payments of DOD components, DOD stated that OUSD (Comptroller) had focused implementation of its remediation efforts on the 10 components that accounted for approximately 95 percent of the department’s travel pay disbursements in DTS. DOD added that this approach achieved maximum coverage of travel payments, given its time and resource limitations. DOD also stated that improper payment metrics reported by DFAS supported this approach, as these data show that the military services accounted for 92 percent of DTS travel payments and a majority of improper travel payments. We acknowledge in our report that DOD identified the 10 components to include on the SAO committee because these components accounted for the significant majority of the department’s fiscal year 2016 DTS travel payments. However, our report also states that it is unclear whether there is an association between the volume of DTS travel payments and improper travel payment rates (measured in terms of the percentage of DTS travel payments made improperly), because DOD does not routinely collect data on improper travel payment rates for all components even though—as we also note in our report—such data are available. As a result, DOD may have excluded components with relatively lower travel payments but higher rates of improper payments. DOD’s approach can serve to reduce DOD’s total improper travel payment amounts, but it may not fully support a key goal of DOD’s Remediation Plan—to reduce the risk of improper travel payments. Thus, we continue to believe that DOD should incorporate improper payment rates into its approach to oversee the implementation of its remediation efforts. In partially concurring with our second recommendation that OUSD (Comptroller) expedite completion of the remaining Remediation Plan requirements by establishing milestones for the requirements, monitoring whether the components have completed them on time, and holding components accountable to completing the requirements, DOD stated that OUSD (Comptroller) will expedite completion of the Remediation Plan requirements for the six components that have not yet completed them. DOD specified that OUSD (Comptroller) will establish milestones for the remaining requirements, monitor their progress, and hold components accountable for their completion. DOD stated that it would complete these actions by January 31, 2020. DOD also reiterated that it does not believe detailed oversight beyond the largest components is cost-effective, but noted that it would continue to monitor the non-SAO components and their impact on improper travel payments. The intent of our recommendation is to ensure that DOD expedites completion of the Remediation Plan requirements for, at a minimum, the 10 components that accounted for a significant majority of DOD’s DTS travel payments. We believe the planned actions that DOD outlined in its response will meet the intent of our recommendation. Further, as discussed in our report, requiring additional components to complete the Remediation Plan requirements may be warranted if those components have relatively high improper payment rates. Therefore, DOD’s stated plan to monitor other components and their impact on improper travel payments would be responsive to our recommendation, provided the department holds non- SAO committee components accountable for addressing high improper payment rates. In partially concurring with our fifth recommendation that the DOD Deputy Chief Financial Officer direct the chairs of the SAO Committee, with the input of OUSD (Comptroller), DTMO and DFAS, to provide guidance to the components on how to determine if actions that would address root causes are cost-effective to implement, DOD stated that OUSD (Comptroller) will revise the improper payments corrective action plan template to require reporting components to perform a cost-benefit analysis to determine the best or most cost-effective solution, resulting in savings to the department. DOD added that OUSD (Comptroller) will not provide specific steps to the components on how to determine whether their actions are, in fact, cost-effective to implement. DOD further stated that it believes that the criteria and/or appropriate steps to determine whether corrective actions are cost-effective for a component must be identified and agreed upon internally within the component. DOD stated that it would complete these actions by October 31, 2019. The intent of our recommendation is to ensure that DOD components determine the cost-effectiveness of actions to address the root causes of improper travel payments. DOD’s stated plan to require the reporting components to perform a cost-benefit analysis will meet the intent of our recommendation, provided that the department ensures that the components are evaluating the cost-effectiveness of planned corrective actions that address the root causes of improper travel payments. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the DOD Chief Management Officer, the Under Secretary of Defense (Comptroller), the Secretary of the Army, the Secretary of the Air Force, the Secretary of the Navy, the Commandant of the Marine Corps, the Chairman of the Joint Chiefs of Staff, the Director of the Defense Finance and Accounting Service, and the Director of the Defense Travel Management Office. In addition, the report is available at no charge on our website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2775 or FieldE1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Defense Media Activity (DMA) Missile Defense Agency (MDA) Defense Acquisition University (DAU) Defense Advanced Research Projects Agency (DARPA) Defense Commissary Agency (DECA) Defense Contract Audit Agency (DCAA) Defense Finance and Accounting Service (DFAS) Defense Intelligence Agency (DIA) Defense Logistics Agency (DLA) Defense Security Service (DSS) Defense Technical Information Center (DTIC) Defense Technology Security Administration (DTSA) Defense Threat Reduction Agency (DTRA) Department of Defense Education Activity (DODEA) National Defense University (NDU) National Geospatial-Intelligence Agency (NGA) Defense POW/MIA Accounting Agency (DPAA) Defense Health Agency (DHA) Court of Appeals for the Armed Forces (CAAF) Uniformed Services University of Health Sciences (USU) DOD Inspector General (DOD IG) Defense Contract Management Agency (DCMA) Defense Security Cooperation Agency (DSCA) White House Military Office (WHMO) Defense Microelectronics Activity (DMEA) Test Resource Management Center (TRMC) Office of the Secretary of Defense (OSD) Office of Economic Adjustment (OEA) Office of General Counsel (OGC) Defense Human Resources Activity (DHRA) Component Name Washington Headquarters Service (WHS) Pentagon Force Protection Agency (PFPA) Joint Chiefs of Staff (JCS) U.S. Africa Command (AFRICOM) U.S. Central Command (CENTCOM) U.S. European Command (EUCOM) U.S. Northern Command (NORTHCOM) U.S. Indo-Pacific Command (INDOPACOM) U.S. Special Operations Command (SOCOM) U.S. Strategic Command (STRATCOM) U.S. Transportation Command (TRANSCOM) Inter American Defense Board (IADB) Joint Interagency Task Force – West (JIATF-W) North Atlantic Treaty Organization (NATO) United Nations Command/US Forces Korea (USFK) U.S. Military Entrance Processing Command (USMEPCOM) Components represented on the Senior Accountable Official Committee (SAO committee) since establishment of the committee. The SAO committee had a total of 13 member components, but DOD officials told us that 3 components (the Office of the Under Secretary of Defense (Comptroller), the Defense Finance and Accounting Service, and the Defense Travel Management Office) served in support roles and were not held accountable for completing the Remediation Plan requirements. Our objectives were to examine (1) the amount the Department of Defense (DOD) spent on Defense Travel System (DTS) travel payments for fiscal years 2016 through 2018 and how much of those payments DOD estimated to be improper; (2) the extent to which DOD implemented its Remediation Plan; and (3) the extent to which DOD established mechanisms to identify errors leading to improper travel payments, the root causes of those errors, and the cost effectiveness of addressing root causes. To address our first objective, we collected DTS data on travel payments for fiscal years 2016 through 2018, by DOD component and trip purpose, from the Defense Travel Management Office (DTMO). We used this time period because DOD issued its plan to remediate improper payments in 2016. We calculated the total payments for that time period, as well as the average annual payments and subtotals for various categories—such as the military services and the trip purposes—that represented the top three highest percentages of payments. We discussed with DTMO officials how the data were generated and what the data points represented. We chose to focus on DTS because it is the primary system for processing travel vouchers for DOD, and the vouchers it processes account for the majority of DOD travel. We also collected data from the Defense Finance and Accounting Service (DFAS) on travel payments made in DTS that were identified as improper, as well as data on the dollar amount of those improper payments that were estimated to result in a monetary loss to the government. We discussed with DFAS officials the methodology that they used to estimate both the improper payment amounts and the portions of those amounts that were estimated to be monetary losses to the government. To assess the reliability of the data we obtained, we reviewed corroborating documentation, analyzed the data for inconsistencies, and interviewed service officials about the reliability of the data. We determined that the data were sufficiently reliable for our reporting purposes, which were to determine the amount of DOD’s DTS travel payments and to provide insight into the estimated improper travel payment amounts that the department reported for fiscal years 2016 through 2018. However, we also determined that, based on persistent problems with DOD’s improper payment estimates that we and the DOD Inspector General have reported since 2013, these data were not sufficiently reliable for other purposes, such as determining the specific progress DOD has made in reducing its rates of improper travel payments. To address our second objective, we reviewed documents and met with officials to discuss DOD’s implementation of its Remediation Plan. We also conducted a web-based survey of officials at DOD components. We administered the survey from February 4 through March 29, 2019, soliciting information on the extent to which components had implemented the Remediation Plan, steps the components had taken to address improper travel payments, the types of issues that frequently lead to improper travel payments, and challenges associated with reducing improper travel payments. We sent this survey to 52 components, 37 (71 percent) of whom responded. More specifically, 9 of 10 (90 percent) components represented on the Senior Accountable Official (SAO) committee (SAO components) responded and 28 of 42 (67 percent) components not represented on the SAO committee (non- SAO components) responded. The survey results represent the views of only those components that responded and may not be generalizable to all components. The results of our survey provide measures of component officials’ views at the time they completed the survey in February and March 2019. Please see appendix I for a list of the 52 components we contacted. How familiar are you, in responding to this survey on behalf of the #COMPONENT, with DOD’s Travel Pay Improper Payments Remediation Plan (dated October 1, 2016), if at all? (Response options provided: Checkboxes labeled “Very familiar,” “Moderately familiar,” “Slightly familiar,” “Not at all familiar,” and “No opinion/no response.”) Has a lead entity in the #COMPONENT been designated for implementing DOD’s Travel Pay Improper Payments Remediation Plan (dated October 1, 2016) requirements? (Response options provided: Checkboxes labeled “Yes, an office has been designated the lead for this effort,” “Yes, a person has been designated the lead for this effort,” “No entity has been designated to lead implementation requirements,” and “Don’t know”) Has the #COMPONENT designated in writing a Senior Accountable Official (SAO)? (An SAO is a Senior Executive Service member, general officer, or flag officer designated by a component as responsible for reducing improper payments.) (Response options provided: Checkboxes labeled “Yes,” “No, but my component is represented by an SAO in another component or organization,” “No,” and “Don’t know.”) Has the #COMPONENT completed this? (Response options provided: Checkboxes labeled “Yes,” “No,” and “Don’t know.”) If yes, what was the month the #COMPONENT completed the action? (Response option provided: one text box.) If yes, what was the year the #COMPONENT completed the action? (Response option provided: one text box.) Has the #COMPONENT completed any of the following actions? Review Defense Finance and Accounting Service (DFAS) reports on improper travel payments. (Response options provided: “Yes,” “No,” “Not applicable (do not receive DFAS reports),” and “Don’t know.”) Have representatives of the #COMPONENT attended the quarterly Senior Accountable Official (SAO) meetings since they were first held in January 2017? An SAO is a Senior Executive Service member, general officer, or flag officer designated by a component as responsible for reducing improper payments. (Response options provided: “Yes, a representative of our component attended all of the meetings,” “Yes, a representative of our component attended some, but not all, of the meetings,” “No, a representative of our component has never attended an SAO meeting,” and “Don’t know.”) Has the #COMPONENT received a copy of the official minutes of the quarterly Senior Accountable Official (SAO) meetings since they were first held in January 2017? (Response options provided: “Yes, our component received a copy of the minutes for all of the meetings,” “Yes, our component received a copy of the minutes for some, but not all, of the meetings,” “No, our component has not received a copy of the minutes for any of the SAO meetings,” and “Don’t know.”) Has the #COMPONENT taken steps to identify the root causes of voucher errors that led to improper travel payments in fiscal year 2018? Note, for the purpose of this question we define root causes as “the reasons personnel made errors preparing or approving vouchers,” including but not limited to: travelers were insufficiently trained on voucher preparation, approvers did not have sufficient time to review vouchers, and/or Defense Travel System was not effectively designed to process vouchers. (Response options provided: “Yes,” “No,” and “Don’t know.”) What are some examples of root causes of voucher errors that the #COMPONENT identified in fiscal year 2018? (Response option provided: one text box.) Has the #COMPONENT taken steps to address any identified root causes of voucher errors that led to improper travel payments in fiscal year 2018? (Response options provided: “Yes,” “No,” and “Don’t know.”) What steps have been taken by the #COMPONENT to address the root causes of voucher errors that led to improper travel payments in fiscal year 2018? (Response option provided: one text box.) Because the majority of survey respondents did not provide open-ended responses to each question, we did not conduct a formal content analysis of the responses. We determined that the open-ended responses would not be representative of all components that responded to our survey, and we therefore present them only as illustrative examples. To analyze open-ended comments provided by those responding to the survey, GAO analysts read the comments, jointly developed categories for the responses, and flagged relevant responses for inclusion in this report. To address our third objective, we reviewed DOD’s Remediation Plan, documents related to DOD’s implementation of the Remediation Plan, such as the minutes of SAO committee meetings, and the June 2018 DOD Improper Payments Senior Accountable Officials Steering Committee Charter. In addition, we met with DOD and component officials to discuss efforts to identify and address root causes of improper travel payments and conducted a web-based survey of travel administrators in 52 DOD components (summarized above) to obtain information on their efforts to identify and address the root causes of improper travel payments. We compared the information we obtained with OMB guidance on how agencies are to identify and address the root causes of improper payments, as well as the definition of root cause contained in the template DOD uses for corrective action plans intended to address improper travel payments. We conducted this performance audit from April 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Matthew Ullengren (Assistant Director), Vincent Buquicchio, Christopher Gezon, Foster Kerrison, Jill Lacey, Joanne Landesman, Rob Letzler, Kelly Liptan, and Michael Silver made key contributions to this report.", "summary": "Improper payments—including payments that should not have been made or were made in an incorrect amount—are a long-standing, significant challenge in the federal government. Both GAO and the DOD Inspector General have reported on problems related to improper payments in DOD's travel pay program. This report examines (1) the amount DOD spent on DTS travel payments for fiscal years 2016 through 2018 and how much of those payments DOD estimated to be improper and the extent to which DOD has (2) implemented its Remediation Plan and (3) identified travel payment errors, the root causes of those errors, and the cost-effectiveness of addressing root causes. GAO analyzed fiscal years 2016 through 2018 data on DTS payments, reviewed DOD's Plan and documentation, interviewed officials about implementation efforts, and surveyed 52 DOD components about steps taken to address improper travel payments. The Department of Defense's (DOD) Defense Travel System (DTS)—the primary system DOD uses to process travel payments—accounts for most of DOD's travel payments. DOD spent $18.3 billion on DTS travel payments from fiscal years 2016 through 2018, while incurring a reported $965.5 million in improper travel payments. In that period, DOD averaged $6.1 billion in DTS travel payments and $322 million in improper travel payments annually. Not all improper travel payments—such as legitimate payments that initially lacked supporting documentation―represented a monetary loss to the government. Officials said DOD first estimated a monetary loss from improper travel payments in fiscal year 2017. For fiscal years 2017 and 2018 it estimated a total monetary loss of $205 million out of $549 million in improper DTS payments (see fig.). In October 2016, DOD established a Remediation Plan to reduce improper travel payments and a committee to monitor implementation of the plan at 10 DOD components. DOD selected these 10 components because they accounted for a significant percentage of total travel payments. However, DOD did not take into account the components' own estimates of their improper payment rates. As of March 2019, only 4 of the 9 components that responded to GAO's survey had completed all of the plan's requirements, in part because of a lack of milestones in the plan and ineffective monitoring for required actions. As a result, DOD does not have reasonable assurance that its actions have been sufficient. DOD has mechanisms to identify errors leading to improper travel payments, and some components have developed specific corrective plans to address the errors. However, GAO found that these efforts did not clearly identify the root causes of the errors, in part because there is no common understanding of what constitutes the root cause of improper travel payments. DOD components also have not incorporated considerations of cost-effectiveness into decisions about whether to take actions that could reduce improper payments. Without addressing these issues, DOD will likely miss opportunities to implement the changes necessary to address the root causes of improper travel payments. GAO made 5 recommendations, including that DOD consider data on improper payment rates in its remediation approach; define the term “root cause”, and consider cost effectiveness in deciding how to address improper payments. DOD generally concurred with 4 recommendations, but did not concur with revising its approach for selecting components to implement its Remediation Plan, stating that it has already taken actions that address this issue. GAO believes the recommendation remains valid.", "document_type": "gao"}
{"report": "The Coast Guard owns or leases more than 20,000 facilities consisting of various types of buildings and structures. The Coast Guard’s shore infrastructure is organized into five product lines and 13 asset types, known as asset lines. For example, within its shore operations asset line, the Coast Guard maintains over 200 stations along U.S. coasts and inland waterways to carry out its search and rescue operations, as well as other missions such as maritime security. Much of the Coast Guard’s infrastructure is vulnerable to the effects of extreme weather and can be costly to repair or replace after major storms. From December 2005 through June 2019, the Coast Guard received about $2 billion in supplemental appropriation funds to, among other things, rebuild or relocate 15 facilities damaged by hurricanes. During this time, the Coast Guard relocated facilities further inland or to higher ground, upgraded facilities to be more resilient, and designed new facilities with features to protect them from natural disasters. For example, after being damaged by Hurricane Ike in 2008, the Coast Guard relocated a regional facility in Houston, Texas further inland to help protect the new facility from extreme weather. The facility was also designed to withstand wind speeds of up to 115 miles per hour. In February 2017, the Coast Guard’s Civil Engineering program also issued guidance intended to increase the likelihood that new or recapitalized buildings would be designed to withstand natural disasters, and to enable the Coast Guard to better manage risks to its operations and personnel, among other things. We found in February 2019 that the condition of the Coast Guard’s shore infrastructure was deteriorating and almost half of it was past its service life—resulting in (1) recapitalization and new construction and (2) deferred maintenance backlogs of at least $2.6 billion as of 2018. In 2018, the Coast Guard graded its overall shore infrastructure condition as a C minus based on criteria it derived from standards developed by the American Society of Civil Engineers. Table 1 shows information about the number of assets, replacement value, service life of, and condition grades assigned by the Coast Guard for each of its asset lines for fiscal year 2018. The aging and deteriorating condition of the Coast Guard’s shore infrastructure has led to at least $2.6 billion in deferred construction projects and maintenance backlogs. With almost half of its infrastructure past its service life, and given recent Coast Guard funding requests for its shore infrastructure, it will take many years for the agency to address these backlogs. For example, in 2018 the Coast Guard estimated that it would take almost 400 years to address just the $1.774 billion recapitalization and new construction backlog—assuming an overall 65- year service life and that funding would continue at the fiscal year 2017 appropriations level. This time frame estimate excludes the Coast Guard’s $900 million deferred depot-level maintenance backlog. Table 2 provides information on the Coast Guard’s two shore infrastructure backlogs as of August 2018. Nevertheless, the size and estimated costs of the Coast Guard’s backlogs may be understated. We found in February 2019 that the Coast Guard’s estimated costs did not include hundreds—or the majority—of the projects on the recapitalization and new construction backlog. For example, we reported that there were 205 projects on the backlog without cost estimates. Officials explained that they had not prepared cost estimates for these projects because they were in the preliminary stages of development. Our previous reports have identified various steps the Coast Guard has taken to begin to improve how it manages its shore infrastructure. Some of the steps the Coast Guard has taken align with leading practices for managing public sector backlogs and key practices for managing risks to critical infrastructure, including identifying risks posed by the lack of timely investment, identifying mission-critical facilities, disposing of unneeded assets, and beginning an assessment of shore infrastructure vulnerabilities. Specifically, the Coast Guard has: Identified risks posed by lack of timely investment. In February 2019, we found that the Coast Guard had a process to identify, document, and report risks to its shore infrastructure in its annual shore infrastructure reports for fiscal years 2015 through 2018. These reports identified the types of risks the Coast Guard faces in not investing in its facilities, including financial risk, capability risk, and operational readiness risk. The Coast Guard met this leading practice to identify risk in general terms—for example, in terms of increased lifecycle costs, or risk to operations. Identified mission-critical and mission-supportive shore infrastructure. In February 2019, we found that since at least 2012, the Coast Guard had documented its process to classify all of its real property under a tier system and established minimum investment targets by tier as part of its central depot level maintenance expenditure decisions. These tiers—which range from mission- critical to mission-supportive assets—were incorporated into guidance that Coast Guard decision makers are to follow in their deliberations about project funding, and to help them determine how to target funding more effectively. For example, Coast Guard guidance for fiscal years 2019 through 2023 prioritized expenditures on shore infrastructure supporting front line operations, such as piers or runways, over shore infrastructure providing indirect support to front line operations, such as administrative buildings. Assessed selected buildings for vulnerabilities. We issued a report today that discusses the Coast Guard Civil Engineering program’s efforts to conduct a vulnerability assessment of its owned and occupied buildings, which the Coast Guard initiated in 2015 and aims to complete in 2025. The Coast Guard calls this infrastructure review the Shore Infrastructure Vulnerability Assessment. The focus of Phase I of this assessment, completed in 2019, was to determine the vulnerability of certain occupied buildings to 10 natural disasters. Further, the assessment results are intended to assist with contingency planning by identifying which Coast Guard facilities are likely to remain operational after a natural disaster. During Phase I of this assessment, completed in 2019, the Coast Guard analyzed 3,214 buildings, almost 16 percent of its infrastructure, for vulnerabilities to disasters such as floods, earthquakes, and hurricanes. The analysis identified Coast Guard- wide infrastructure vulnerabilities to coastal risks such as shoreline loss, coastal erosion and earthquakes, as well as tsunami risks on the West Coast of the United States, Alaska, Guam, and Hawaii, and immediate and serious flood risks in Puerto Rico and the Gulf and East Coasts. The Phase I report recommended that Coast Guard units and contingency planners consider these vulnerabilities when preparing contingency plans or making capital investments. The Coast Guard has also initiated a follow up effort involving structural analyses for buildings it believes to be more susceptible to damage from earthquakes and wind. Officials involved said their aim is to complete this effort in 2025. The Coast Guard has taken actions to begin to improve its shore infrastructure management. However, as we previously reported, the Coast Guard has not fully applied leading practices and key risk management steps to improve its shore infrastructure management. Specifically, we found, among other things, that the following actions could help improve the Coast Guard’s shore infrastructure management efforts: Employ models for predicting the outcome of investments and analyzing tradeoffs. In February 2019, we found that a 2017 Coast Guard Aviation Pavement Study employed a model that found that the Coast Guard could more efficiently prioritize investment in aviation pavement. A subsequent Coast Guard aviation pavement plan recommended actions to use the study results and potentially save $13.8 million. However, we found that the Coast Guard had not fully implemented its own recommended actions to achieve the cost savings. Additionally, we found that while a similar analytical approach to efficiently prioritizing investments in aviation pavement could be applied to all of the shore infrastructure asset lines, the Coast Guard had not applied the approach to other asset lines. By not employing similar models across its asset lines for predicting the outcome of investments, analyzing tradeoffs, and optimizing decisions among competing investments, the Coast Guard is missing opportunities to potentially identify and achieve cost savings across other asset lines. We recommended that the Coast Guard employ models for its asset lines that would predict the investment outcomes, analyze tradeoffs, and optimize decisions among competing investments. The Coast Guard agreed with our recommendation but as of August 2019 had not addressed it. The Coast Guard stated that it plans to assess the use of modeling tools used by the Department of Defense as well as other alternatives to enhance its real property asset management capability. We will continue to monitor its actions. Dispose of unneeded assets. In October 2017, we found that disposing of unneeded assets, such as closing unnecessarily duplicative boat stations, based on a sound analytical process, could potentially generate $290 million in cost savings over 20 years. Specifically, the Coast Guard identified 18 unnecessarily duplicative boat stations with overlapping coverage that could be permanently closed without negatively affecting the Coast Guard’s ability to meet its mission requirements, including its 2-hour search and rescue response standard. In 2017, the Coast Guard affirmed that its leadership believes the study remains valid, but as of September 2019 it has not closed any stations. Figure 1 depicts the extent of the Coast Guard’s overlapping boat and air station search and rescue coverage, as identified by the Coast Guard, some of which the Coast Guard determined to be unnecessarily duplicative. In February 2019, we found that 5 of the 18 boat stations recommended for closure had projects listed on the Coast Guard’s current project backlog. For example, Station Shark River, in New Jersey, was recommended for recapitalization in fiscal year 2017, despite Coast Guard recommendations to close the station in 1988, 1996, 2007, and 2013. Notably, the Coast Guard has made multiple attempts in previous years to close such stations but was unable to due to congressional intervention, and subsequent legislation prohibiting closures. In October 2017, we recommended that the Coast Guard establish and implement a plan with target dates and milestones for closing boat stations that it has determined provide overlapping search and rescue coverage and are unnecessarily duplicative. In February 2019, we further recommended disposing of unneeded assets to more efficiently manage resources and better position the Coast Guard and Congress to address shore infrastructure challenges. The Coast Guard agreed with our recommendations. As of September 2019, the Coast Guard reported that it was considering changes in the operational status of several stations, such as closing the stations during the winter months when they conduct few, if any, search and rescue cases. The Coast Guard estimated that it will continue to consider changes until March 2020. These are positive steps, but we continue to believe that it is important for the Coast Guard to dispose of unneeded assets. Given the Coast Guard’s competing acquisition, operational, and maintenance needs, and its existing $1.774 billion project backlog of recapitalization and new construction projects, these actions may help to mitigate some of its resource challenges. We will continue to monitor the Coast Guard’s efforts to implement these recommendations. Report shore infrastructure project backlogs accurately. In February 2019, we found areas in which the Coast Guard could increase budget transparency for shore infrastructure by accurately reporting project backlogs and costs in Congressionally-required plans. Specifically, we found that the Coast Guard had not provided accurate information to Congress necessary to inform decision- makers of the risks posed by untimely investments in maintenance and repair backlogs. For example, the Coast Guard had not provided complete information to Congress in its Unfunded Priorities Lists of shore infrastructure projects, including information about tradeoffs among competing project alternatives, as well as the impacts on missions conducted from shore facilities in disrepair. We also found that Coast Guard budget requests related to shore infrastructure for fiscal years 2012 through 2019 generally did not identify funding to address any backlogs of deferred maintenance or recapitalization, except for one fiscal year—2012—when the Coast Guard requested $93 million to recapitalize deteriorated/obsolete facilities. We also found that the Coast Guard had not provided accurate information about its requirements-based budget targets for shore infrastructure in its budget requests. According to Coast Guard officials, a requirements-based budget is an estimate of the cost to operate and sustain its shore infrastructure portfolio of assets over the lifecycle of the asset, from initial construction or capital investment through divestiture or demolition. Further, we found that Coast Guard recapitalization targets showed a far greater need than was reflected in the appropriations it requested from fiscal years 2012 through 2019. Specifically, Coast Guard targets for recapitalization of shore assets indicated the Coast Guard needs $290 to $390 million annually for its recapitalization efforts. However, its budget requests for fiscal years 2012 through 2018 have ranged from about $5 million to about $99 million annually. We recommended that the Coast Guard include supporting details about competing project alternatives and report tradeoffs in Congressional budget requests and related reports. Without such information about the Coast Guard’s budgetary requirements, the Congress will lack critical information that could help to prioritize funding to address the Coast Guard’s shore infrastructure backlogs. While the Coast Guard agreed with our recommendation, in August 2019 officials reported that they will continue to develop budgets as the agency has done but will include additional information in future required reports to Congress. We will continue to monitor these actions. Fully implement DHS’s Critical Infrastructure Risk Management Framework. In September 2019, we found that the Coast Guard has taken some steps to improve the resilience of its shore infrastructure by rebuilding storm-damaged facilities and initiating a vulnerability assessment, but its processes to improve shore infrastructure resilience are not fully aligned with the five steps DHS has identified for critical infrastructure risk management (DHS Critical Infrastructure Risk Management Framework). The five steps include: (1) setting goals and objectives, (2) identifying critical infrastructure, (3) assessing and analyzing risks and costs, (4) implementing risk management activities, and (5) measuring the effectiveness of actions taken. We found that the Coast Guard is not positioned to provide decision makers with complete details of which infrastructure facilities are critical, and the type of information the DHS Critical Infrastructure Risk Management Framework recommends for making cost effective risk management decisions. The Coast Guard identified occupied buildings that may be important to operations and assessed their vulnerability through its Shore Infrastructure Vulnerability Assessment process, but this process did not identify all shore infrastructure assets that are critical to its missions—such as aircraft runways—or screen them for all vulnerabilities, such as flooding. Similarly, we found that while the Coast Guard identified almost 800 buildings that may be vulnerable to tornadoes and another 1,000 buildings vulnerable to hurricanes, it has not analyzed the potential consequences, such as economic losses, costs for rebuilding, and impact on mission, should this infrastructure suffer damage from those vulnerabilities. Without a complete understanding of both the vulnerabilities of its infrastructure and the consequences to its mission operations if its infrastructure is damaged, the Coast Guard risks questionable recapitalization investments for improving resilience when selecting projects to fund. Such an understanding is especially important given its existing project backlogs of at least $2.6 billion. The five steps of the DHS Critical Infrastructure Risk Management Framework are intended to guide decision making and prioritize actions to more effectively achieve desired outcomes. Therefore, in September 2019 we recommended that the Coast Guard implement risk management processes that more fully align with the five key steps outlined in DHS’s Critical Infrastructure Risk Management Framework to better guide its shore infrastructure investment decisions. The Coast Guard agreed with our recommendation. It stated that it plans to make progress towards implementing the recommendation while developing and implementing its Component Resilience Plan, in accordance with the recently mandated DHS Resilience Framework. It intends to complete these efforts by the end of 2021. The Coast Guard also intends to develop, by July 2020, goals and objectives for measuring the effectiveness of actions taken to identify resilience readiness gaps and resource needs. We will continue to monitor these efforts. Chairman Maloney, Ranking Member Gibbs, and Members of the Subcommittee, this completes my prepared statement. I would be happy to respond to any questions you may have at this time. If you or your staff members have any questions about this testimony, please contact me at (202) 512-3841 or andersonn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contact above, Dawn Hoff, Assistant Director; Andrew Curry, Analyst-in-Charge; Peter Haderlein; Landis Lindsey; Calaera Powroznik, and Molly Ryan made key contributions to this testimony. Other staff who made key contributions to the reports cited in the testimony are identified in the source products. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Coast Guard, within the Department of Homeland Security (DHS), owns or leases more than 20,000 shore facilities—such as piers, boat stations, air stations, runways, and housing units—at more than 2,700 locations, from which it carries out its missions. This shore infrastructure is often positioned along the nation's coastlines where it can be vulnerable to damage from extreme weather. This statement summarizes GAO findings related to the condition of Coast Guard shore infrastructure, actions the Coast Guard has taken to improve its management of its shore infrastructure, and additional actions it needs to take. This statement is based on three GAO products issued from October 2017 through September 2019, along with selected updates on actions the Coast Guard has taken to address GAO's recommendations from these reports. GAO analyzed relevant Coast Guard documents, management processes and decisions, and interviewed Coast Guard officials. To conduct updates, GAO also reviewed information on the Coast Guard's actions to implement its prior recommendations. In February 2019, GAO reported that the Coast Guard's $18 billion portfolio of shore infrastructure was deteriorating, and almost half of it was past its service life as of 2018. Coast Guard data showed that it would cost at least $2.6 billion to address its maintenance and recapitalization (major renovation) project backlogs at recent funding levels. Coast Guard data also showed that hundreds of projects had not been factored into the backlog costs. GAO's prior work has shown that the Coast Guard has taken initial steps toward improving how it manages its shore infrastructure, including conducting an initial assessment of shore infrastructure vulnerabilities. However, GAO also found that the Coast Guard had not fully applied leading practices and key risk management steps in managing its shore infrastructure, and needs to take the following actions: Employ models for predicting the outcome of investments and analyzing tradeoffs . In February 2019, GAO found that the Coast Guard had used a model to determine that it could more efficiently prioritize its investment in aviation pavement—one segment of an almost $3 billion portfolio of aviation shore infrastructure—and save about $13.8 million. However, as of February 2019, the agency had not implemented the aviation pavement study results. Moreover, according to Coast Guard officials, the agency could employ models to its entire portfolio of shore infrastructure. By not implementing the results of its aviation pavement model or employing similar models across its shore infrastructure assets, the Coast Guard is missing opportunities to potentially identify and achieve cost savings across other assets. Dispose of unneeded assets. In October 2017, GAO found that closing boat stations that the Coast Guard had found to be unnecessarily duplicative could potentially generate $290 million in cost savings over 20 years. However, in February 2019, GAO found that instead of closures, the Coast Guard was planning recapitalization projects at 5 of the 18 stations it had recommended for closure. Given the Coast Guard's competing shore infrastructure priorities and existing project backlogs, GAO recommended disposing of unneeded assets to more efficiently manage resources and better position the Coast Guard and Congress to address shore infrastructure challenges. Implement DHS's Critical Infrastructure Risk Management Framework. In September 2019, GAO found that DHS has recognized the importance of protecting critical infrastructure from extreme weather and other risks. However, the Coast Guard has not fully aligned its processes for improving shore infrastructure resilience with DHS's five key steps for critical infrastructure risk management. For example, when identifying and then assessing risks to its infrastructure—two of the steps in the DHS process—the Coast Guard did not identify all assets that are critical to its missions, such as aircraft runways, or screen them for all vulnerabilities, such as flooding. Aligning its processes with the DHS steps would provide greater assurance that the Coast Guard is investing its resources to minimize potential damage and expenses caused by future extreme weather events. In the three reports, GAO made 10 recommendations to improve the Coast Guard's asset management efforts, including employing models for predicting investment outcomes, disposing of unneeded assets, and implementing DHS's critical infrastructure risk management framework to guide shore infrastructure resilience decisions. DHS concurred and generally described planned actions to address these recommendations, but has not yet fully implemented them.", "document_type": "gao"}
{"report": "BLS currently produces a number of different price indexes to estimate price inflation (see table 1). In line with its strategic plan, BLS aims to make these estimates as accurate as possible, meaning that they reflect the average level of price inflation for a selected group of consumers. The accuracy of a price index can be assessed in multiple ways, such as the extent to which the index applies appropriate formulas to data that are complete and drawn from sufficiently large samples covering the relevant group of people. BLS bases its collection of these data on the population covered by the Consumer Price Index for All Urban Consumers (CPI-U). BLS then uses data collected for the CPI-U to produce three other price indexes. After introducing the CPI-U as its primary, or headline, index, BLS maintained a separate data collection for the CPI-W from 1978 through 1980 but found little difference between data for CPI-W and CPI- U. According to BLS, as a result of this and budgetary issues, BLS stopped collecting separate data for the CPI-W in 1981 and began using CPI-U data to derive the CPI-W. To create the CPI-U, BLS chooses a sample of outlets (e.g., stores or internet sites) at which the CPI-U population shops (see fig. 1 for more information on how BLS creates price indexes). BLS then collects price data at these outlets for goods and services the CPI-U population buys and uses the data to develop basic, or elementary, indexes for each good and service. BLS combines the elementary indexes into a single, aggregated index by applying a set of expenditure weights—factors that determine, for example, whether a change in the price of apples or mobile phone service has a larger effect on total inflation (see fig. 2). These expenditure weights reflect the proportion of spending consumers direct to each good or service. To develop expenditure weights, BLS directs the Census Bureau to gather data about the relative importance of each purchase within the target population’s “market basket” of consumer goods and services. The Census Bureau collects these data in the Consumer Expenditure Survey, a nationwide household survey conducted by BLS to determine how consumers spend their money that also contains demographic data about the households surveyed. BLS uses 2 years of Consumer Expenditure Survey data to calculate the expenditure weights, in part so the sample sizes are large enough to produce accurate weights. From data collected to produce the CPI-U, BLS derives two subpopulation indexes—indexes that focus on the spending patterns of a portion of the population of all urban consumers: the CPI-W and the CPI- E. To produce these subpopulation indexes, BLS adjusts the relative importance of price changes in each good and service through a process sometimes referred to as “reweighting,” meaning BLS develops alternate sets of expenditure weights that reflect the spending patterns of the subpopulation. For example, since medical care comprises more of the CPI-E subpopulation’s total expenditures (about 12 percent) than of the CPI-U population’s total expenditures (about 9 percent), the CPI-E gives more weight to medical care than the CPI-U. BLS also creates a “chained” index using the same data for the entire CPI-U population but changing the formula used to combine indexes for each good and service, known as elementary or basic indexes, into a single aggregated index. This formula captures how consumers shift spending among different types of goods and services as prices change (see text box). In contrast, the other indexes assume that consumers keep purchasing various categories of goods and services in the same proportions over a 2-year period regardless of price changes. What is a chained price index? A chained price index uses a formula that is believed by some economists to better approximate a cost-of-living index by more accurately accounting for changes in consumption patterns in response to relative price changes. They contend that such a formula reduces the potential for overstating inflation relative to the other indexes BLS produces, which assume consumers keep buying goods and services in the same proportions no matter their price. Like the other three indexes BLS produces, the Chained CPI-U reflects consumers’ ability to adapt to changing prices by choosing among closely related goods and services as prices change, for example purchasing a different type of apple because it is on sale. However, unlike the other three indexes, the Chained CPI-U further reflects consumers’ ability to choose among all available goods and services as prices change, such as taking a train to work instead of driving when the price of gasoline rises, and purchasing headphones to listen to music during the commute. We previously reported that, were federal retirement benefits to be indexed to the Chained CPI-U, SSA and other agencies would need to determine whether to base retirement COLAs on final data that may be outdated or preliminary data that may be inaccurate. This is because the data needed to use a superlative index formula only become available after a significant time lag. This lag delays issuance of final monthly estimates for the Chained CPI-U by up to 1 year. Additionally, the chair of a panel convened at the request of BLS to examine issues in measuring the cost of living cautioned that chained indexes may not accurately reflect the way people with varying incomes substitute goods and services. For example, retirees with lower incomes might not have the same ability as retirees with higher incomes to substitute other goods and services when the prices of needed medical care or prescription drugs rise. BLS receives input on its processes from several sources. For example, BLS receives advice and recommendations from several advisory committees that variously focus on technical issues and the needs of users of BLS statistics. BLS also periodically receives input on its price indexes through external commissions and panels. For example, in May 1995, the U.S. Senate created the Advisory Commission to Study the Consumer Price Index, commonly referred to as the “Boskin Commission,” after its chairman, Michael J. Boskin. In December 1996, the Boskin Commission released its final report identifying sources of bias in the production of CPIs that the commission concluded were causing the indexes to overstate inflation. BLS also receives input on its price indexes through public comment. For example, in May 2019, the Office of Management and Budget issued a request for public comments on the various price indexes produced by BLS and BEA. While there are a number of federal retirement benefit programs, Social Security is by far the largest provider of indexed retirement and disability benefits in the United States, paying out over $1,047 billion in retirement and disability benefits in 2019. Social Security was established in 1935 to provide for the general welfare of older Americans by, among other things, establishing a system of federal old-age benefits, including a retirement program. To determine a worker’s initial retirement benefit, Social Security indexes the worker’s earnings to an average wage index. According to SSA, this ensures that a worker’s future benefit reflects the general rise in the standard of living that occurred during his or her working lifetime. Since 1975, Social Security has also indexed retirement benefits after the initial benefit level has been set to a CPI. According to SSA, this ensures that benefits are not eroded by inflation over time. When SSA began indexing benefits, CPI-W was the only national CPI available, and SSA continues to use the CPI-W to determine COLAs. As we have previously reported, the Social Security program faces financial difficulties that, if not addressed, will affect its long-term stability. In April 2020, SSA projected that Social Security’s retirement program trust fund will be unable to pay full benefits in 2034. We have also reported that, according to projections by SSA and the Congressional Budget Office, use of an alternate index to determine COLAs would have less effect on Social Security’s long-range finances than some other options for addressing the program’s finances, such as changing the taxation of earnings or raising the retirement age. That said, we found that, according to SSA projections, using an alternate CPI to calculate COLAs would affect Social Security’s finances in different ways. Specifically, using the CPI-E would increase expected COLAs and thus program costs and using the Chained CPI-U would decrease expected COLAs and thus program costs, while using the CPI-U would result in little change to either. Produced by BEA, the National Accounts are a set of statistics on U.S. production, income, consumption, investment, and saving. Among these are Gross Domestic Product, a measure of the goods, services, and structures produced across the economy, and the Personal Consumption Expenditures index, a measure of consumer inflation similar to CPIs, but constructed using different methods and data sources and covering different populations and transactions. Data collected by BEA to produce the National Accounts differ in a number of ways from those collected by BLS to produce CPIs. For example, while CPIs focus on the expenditures of households in urban areas, the National Accounts also include expenditures on institutional populations, such as individuals living in nursing homes. Further, while CPI expenditure data are based on the recollection of consumers, National Accounts expenditure data primarily reflect the records of the businesses that serve consumers. In other words, to collect data on the quantity of goods and services consumed, BLS surveys consumers about how much they bought, whereas BEA surveys companies about how much they sold. The National Accounts are produced primarily from data collected by federal government agencies. These data include both “statistical” data collected from federal statistical agencies, such as the Census Bureau, as well as “administrative” data collected by federal agencies as a byproduct of administering their programs. For example, BEA uses sample data generated by the Internal Revenue Service in processing tax returns to estimate corporate profits. BEA supplements these statistical and administrative data collected by federal agencies with data obtained from trade associations, businesses, international organizations, and other sources. BLS faces a number of challenges related to the accuracy and timeliness of CPIs, as well as challenges related to measuring inflation for older Americans. Some of these challenges may have implications for federal retirement benefit adjustments. According to BLS officials and documentation, BLS is unsure if the data sources it uses to produce the CPI-U are adequate to produce accurate subpopulation estimates—specifically, the CPI-E and CPI-W. For the CPI-E, BLS has not evaluated the adequacy of the CPI-U data it uses to measure inflation for the 62-and-older subpopulation. Specifically, BLS has not evaluated the extent to which CPI-U data represent the outlets where members of this older subpopulation shop, the prices they pay, or the mix of goods and services they purchase. BLS considers the CPI-E an experimental index, in part, because of the relatively small sample size within the Consumer Expenditure Survey used to create the expenditure weights for this subpopulation, which account for the mix of goods and services the subpopulation purchases. According to BLS documentation, the expenditure weights for the CPI-U rely on about 65,000 household interviews, which are collected quarterly over 2 years. In contrast, the expenditure weights for subpopulation indexes use about one-third or less of that: 21,000 interviews for the CPI-E and 16,000 for the CPI-W. For the CPI-W, BLS has not evaluated the adequacy of using CPI-U data since 1980, but the relative sample size used to calculate the expenditure weights for the CPI-W subpopulation has been shrinking in part because of declining response rates and demographic shifts away from the occupations included in the CPI-W. For example, occupations in the CPI- W include blue-collar jobs such as clerical, sales, laborer, and construction jobs. BLS officials and documentation indicate that as a result of these demographic shifts and the subsequent shrinking sample size within the Consumer Expenditure Survey, the accuracy of the CPI-W expenditure weights may be deteriorating. A core element of BLS’s mission is to provide accurate products. Moreover, standards of internal control call for agencies to obtain relevant data from reliable internal and external sources to meet information requirements for meeting their objectives. For BLS, this could include obtaining relevant data from reliable sources for producing CPIs. BLS officials said they have not evaluated the adequacy of the existing data because it is costly to undertake a full evaluation, but there may be cost- efficient ways to do so. BLS also has not evaluated different methods to conduct a cost-efficient analysis. Without taking actions to understand available options for a cost-efficient solution, BLS lacks reasonable assurance that adjustments to Social Security and other retirement benefits are based on indexes that reflect what they are intended to reflect. Specifically, benefits could be subject to adjustment based on potentially inaccurate information. Most experts we interviewed identified potentially cost-efficient methods to evaluate the adequacy of existing data for subpopulation indexes. For example, five experts we interviewed, including some on BLS advisory groups, suggested that BLS may be able to use existing data to examine the adequacy of using Consumer Expenditure Survey data for the CPI-E. Specifically, one expert suggested that BLS could compare expenditure patterns for the older subpopulation in the Consumer Expenditure Survey to those in third-party data. Another expert added that the overall prices older Americans pay may not be significantly different than the prices the general population pays. For example, gas stations generally charge the same price to each customer regardless of age, so this expert said that it may not be worthwhile for BLS to collect separate price data for older Americans. Another expert indicated that, while it might not be possible to link expenditures and demographics (such as age) for all CPI categories using third-party data, it may be possible for certain categories such as groceries, which are a sizeable portion of the older population’s expenditures. Another suggested that to improve subpopulation indexes, BLS could shift resources from cost savings realized from other ongoing projects. BLS officials acknowledged some potentially cost-efficient methods could exist to evaluate the adequacy of existing data for subpopulation indexes. For example, they said that a recent change in survey methodology will enable them to connect demographic information with information on where people shop beginning in 2019. The ability to make this connection should allow them to determine whether certain subpopulations shop at the same or different outlets and could help them determine the adequacy of their outlet sample selection. According to agency officials, BLS advisory groups could weigh in on such issues, but BLS has not asked the advisory groups to do so nor do the advisory groups have any recent or ongoing research on indexes for subpopulations such as older Americans. BLS officials added that obtaining transaction and demographic data from credit card companies could help, but cautioned that companies may be unwilling to share these data. BLS is currently undertaking a project to improve how it estimates its subpopulation indexes, CPI-E and CPI-W, in part by examining changes to the formulas used to apply expenditure weights. As part of its justification for the project, BLS expressed concerns about the decrease in the relative sample size for the CPI-W population in the Consumer Expenditure Survey and reiterated the importance of the CPI-W in adjusting federal retirement benefits. This project is a step in the right direction but does not fully address the question of whether the CPI-U data are adequate to produce CPI-W and CPI-E. In 2009, BLS began another project to address measurement error in and households’ willingness to respond to the Consumer Expenditure Survey, which is primarily conducted to create expenditure weights for CPIs. According to agency documents, the survey faces increasing costs and declining response rates. One particular goal of the project is to reduce error due to underreporting. For example, BLS is currently testing replacing a paper record of household expenditures with an online form with the goal of more accurately capturing expenditures and maintaining response rates. The project is ongoing and BLS expects to implement changes in stages through and beyond 2022. According to agency officials, the project was not designed to address subpopulation indexes, but instead was designed to address broader issues with the accuracy of the Consumer Expenditure Survey. BLS also faces challenges regarding the timeliness and relevance of CPIs. In particular, most CPIs are published using expenditure data that can be up to 4 years old, and, in this dynamic economy, as expenditure data age, they become less relevant to present-day expenditure patterns. Most of BLS’s price indexes, including the CPI-U, CPI-E, and CPI-W, rely on 2 years of expenditure data and the data require additional time to be collected and processed for use, referred to as a lag. For example, the CPIs produced from January 2014 to December 2015 used expenditure data from 2011 through 2012. BLS officials said reducing the lag could enable more timely use of expenditure data for CPIs but would not be possible without a significant change to the use or design of the Consumer Expenditure Survey. Another of BLS’s indexes, the Chained CPI-U, aims to incorporate current-period expenditure data, which may be most relevant for current- period price changes, but as we reported in 2019, the data are subject to revision and BLS produces the final, revised Chained CPI-U with a 10 to 12 month delay. BLS officials told us they do not currently have timely enough expenditure data to produce the Chained CPI-U without this delay. We found in our 2019 report that if the Chained CPI-U were to be used to calculate Social Security or other federal retirement benefit COLAs, it could result in permanent differentials stemming from measurement error that would have a larger effect on people who receive benefits longest or have lower incomes. BLS also faces several other challenges measuring inflation for older Americans, several of which BLS is examining in the subpopulation project discussed above. Large purchases. BLS is examining how to treat large purchases that are acquired in one time period but used throughout many time periods, such as owner-occupied housing and durable goods. BLS’s current approach to owner-occupied housing is to calculate what it would cost to rent a similar home. In part, because many seniors own their homes, BLS is considering instead calculating how much it costs to own and occupy the home (e.g., by including mortgage interest payments but not the purchase price of the home). Definition of average. BLS is also examining whether a subpopulation index should represent the average expenditures of all households (as its CPIs currently do) or the expenditures of an average household. The current approach of representing the average expenditures of all households is simpler because the index can be constructed from information on average expenditures. The alternate approach of representing expenditures of an average household is more complicated because it gives each household equal weight, and requires first constructing a price index for each household, then an averaging of those indexes. According to BLS, the current approach tends to give more relative weight to the purchasing behavior of higher-income households, whereas the alternate approach may be more appropriate for a subpopulation index, such as the one used to adjust Social Security benefits. For example, taking the average of all expenditures tends to reflect the more expensive purchases typically made by higher-income households. In contrast, measuring the average household’s expenditures may better represent expenditures made by a particular subpopulation, such as recipients of federal benefits programs like Social Security. User needs. BLS is also examining how to define the subpopulation of interest to meet the needs of its users, such as the Social Security Administration. Specifically, CPI-E is based on households headed by someone age 62 or older and the CPI-W is based on households with particular occupations, and BLS is examining whether other definitions could meet user needs. For example, BLS said it plans to contact stakeholders to ask about whether expanding the CPI-W to include all labor force participants (thereby increasing sample size) would meet user needs. Quality change vs. inflation. A further challenge for all price indexes is determining what portion of the price change is due to changes in quality as opposed to inflation, according to eight of the nine experts we interviewed. BLS has several methods to adjust for quality changes. For example, if an older television is replaced with a new model with an increased price, BLS analysts collect information on the characteristics of those televisions and conduct an analysis to determine how much of the price change is due to a change in quality (e.g., the new television has additional features). The remainder of the price change is attributed to inflation. While accounting for quality change is a challenge for all price indexes, four of the nine experts we interviewed said it may be particularly difficult when measuring inflation for older populations. According to these experts, this is because older populations tend to consume more medical care goods and services, for which quality changes are particularly difficult to measure. Alternative data. To improve its price indexes, BLS is exploring the use of alternative data sources, such as “big data” obtained directly from companies, from third parties, or from the internet (see text box below). For example, BLS recently purchased a large private dataset to use in an experimental index for new vehicles. According to BLS, big data may lead to methodological improvements and cost savings in the CPIs. Notably, some big data may provide “real-time” expenditure data that could potentially be used to capture consumer behavior in response to relative price changes, thereby addressing substitution bias. According to agency officials and most experts we spoke with, big data may be promising but incorporating them in the CPIs requires additional considerations and adjustments to the processes BLS currently has in place. For example, the data may not be consistently available with the information needed to produce CPIs. Additionally, big data are not always free and some companies may be reluctant to share these data. What is “big data?” Big data encompass a number of very large data sets that can be structured or unstructured and have the potential to be mined for information. Web-scraped data and scanner data are two prominent types of big data relevant for consumer price indexes. Web- scraped data are price data collected on goods sold online. Scanner data include price and quantity data on sales of goods obtained by scanning bar codes for goods, such as at electronic points of sale in retail outlets. Advances in technology have allowed large amounts of data to be collected and stored easily and could be used in consumer price index construction. In addition to big data, BLS currently uses some administrative data collected by the federal government to improve inflation estimates for certain goods and services. For example, BLS obtains information from the Department of Energy on household consumption averages for electricity and piped gas service. It also uses administrative data from the Centers for Medicare & Medicaid Services about which facilities provide adult home care. According to BLS officials, they are unable to use some administrative data (e.g., certain federal tax data) because of current law. Other data collected by the federal government (National Accounts data). While BLS is exploring numerous alternative data sources, BLS has not fully explored the potential to update expenditure weights on a more frequent basis using supplementary data from the National Accounts in years when the most current biennial weights using Consumer Expenditure Survey data are not available. As discussed earlier, BLS typically requires 2 years of data from the Consumer Expenditure Survey to produce expenditure weights, which have a lag. In contrast, National Accounts data comprise administrative and statistical data representing the whole economy, many of which have a large sample size and are available on an annual basis. Standards of internal control call for agencies to obtain relevant data from reliable internal and external sources in a timely manner to meet information requirements for meeting their objectives. For BLS, this could include obtaining relevant data from reliable sources for producing CPIs. As part of its strategic plan, BLS maintains goals to improve the accuracy and timeliness of BLS data and to ensure relevance in an ever-changing economy. Without adequately exploring the potential of using National Accounts data to supplement Consumer Expenditure Survey data, BLS may be missing an opportunity to move closer towards those goals. Over time, expenditure survey data lose their accuracy and relevance to the present-day expenditure patterns of consumers, which can introduce bias in measures of inflation used to adjust federal retirement benefits. For example, the longer the time period between expenditure weight updates, the longer the delay to include new products in the expenditure patterns reflected in the CPIs. This delay could become increasingly important because of the rapid development in new technology, such as smart phones. Of the 15 publications we reviewed, six discussed ways to improve the CPI and four of these suggested more timely expenditure weight updates could make the CPIs more accurate and relevant. For example, a 2009 working paper by BLS staff found that more frequent weighting may offer better representation of current price change, as well as a closer approximation to a cost-of-living index. In particular, the authors simulated updating expenditure weights annually, which resulted in slower inflation increases that the authors posited are a closer approximation to a cost-of-living index. While these improvements may not be currently possible given the lag in Consumer Expenditure Survey data, the authors conclude that further examination of the weighting issue is a potentially fruitful avenue of research. The three other studies similarly indicated that more timely weight updates would result in more relevant CPIs, for example by better reflecting changes in consumer spending patterns. BLS officials acknowledged that updating the weights more frequently would make the index more relevant, though they did not believe using the Consumer Expenditure Survey to do so was practical in part because they said it would require additional costs to increase the sample size. In 2002, BLS increased the frequency of its weight updates from every 10 years to every 2 years, which they said was an improvement but required a sample size increase in Consumer Expenditure Survey. As previously described, the Consumer Expenditure Survey faces increasing costs and declining response rates and, according to agency officials, obtaining a large enough sample to update weights annually would require a 50 to 100 percent increase in sample size, for example, to avoid an increase in sampling error. Indeed, three studies we reviewed suggested that it can be challenging to obtain enough responses for household surveys such as the Consumer Expenditure Survey, indicating that alternate data sources may become more important. In contrast, BLS officials acknowledged that National Accounts data could provide useful supplementary information if the expenditure survey is not providing timely enough data. However, BLS officials said they have not explored using National Accounts data, in part because they have not examined the effects of altering the expenditure weights in about 10 years. BLS officials expressed concern that National Accounts data can be subject to revision. According to the Bureau of Economic Analysis (BEA), the revisions do not reflect errors but are driven by the incorporation of more complete source data. BLS officials also noted that some National Accounts data are adjusted by the CPI, so BLS would have to remove the CPI’s effect in order to use National Accounts data in the CPI. Moreover, the supplementary use of National Accounts data could also help address some of the concerns with measurement error in household surveys, according to some literature we reviewed. Specifically, National Accounts data could be used to address underreporting due to recall bias, the difficulty some survey respondents have recalling infrequent purchases, or underreporting of certain goods that may be seen as socially undesirable, such as tobacco and alcohol. For example, according to a recent Brookings Institution report, the National Accounts data used for the BEA’s Personal Consumption Expenditure index weights are mostly based on business surveys and administrative data and thereby avoid the reporting biases inherent in the Consumer Expenditure Survey. BLS’s Technical Advisory Committee recommended using administrative data to address such underreporting in fiscal year 2016, as did a National Academy of Sciences report in 2013. While BLS has taken steps toward increased use of administrative data, BLS has not fully implemented the Technical Advisory Committee recommendation as of March 2020. Our review of Organisation for Economic Co-operation and Development (OECD) countries’ national pension systems revealed that it is relatively uncommon to use a retiree-specific index (i.e., a CPI for the older subpopulation) for the purpose of adjusting national pension benefits. Of the 36 OECD countries, 27 have national pension programs in which indexation is based, at least in part, on prices after initial benefits have been set, similar to Social Security in the United States (see app. I). Most OECD countries use their primary measures of inflation to adjust national pension benefits, according to reports and documents about the retirement systems in these countries. Of the 27 countries using prices to adjust national pension benefits, we found evidence in 10 that the national statistical agency produces an index for the older subpopulation. Each of these 10 countries generally uses the same price information for the older subpopulation index as the main CPI but reweights the price information based on the expenditures for that subpopulation, rather than gathering new information that is unique to that group (see text box). A similar approach is used for the CPI-E in the United States. However, of these 10 countries, only four countries use the index for the older subpopulation to adjust their national pension benefits (Australia, Czech Republic, Hungary, and the Slovak Republic). The others produce the subpopulation index for research or other purposes, but do not use it for pension benefit adjustments. Agency officials in all three of our case study countries (Australia, New Zealand, and the United Kingdom) said they generally saw a value in having a primary index for macroeconomic purposes, such as inflation targeting, and a subpopulation index that could be used for other purposes, such as indexation of benefits. Methods for Validating Use of Existing CPI Data in Subpopulation CPIs In the three case study countries we selected for review, each national statistical agency relied upon different approaches to validate the use of existing data from the primary (main) CPI in the subpopulation CPI. Agency officials indicated that some of the methods for validating the use of existing CPI data for the subpopulation CPIs were cost efficient. Australia agency officials said they validated the use of existing data in the index for the older subpopulation in part by both researching whether pensioners pay different prices or shop at different outlets and cross-checking some data from industry sources. Officials said they expected that pensioners and the general population generally pay the same prices for most items and included different prices in the index for the older subpopulation for those items known to be discounted for pensioners. To get a better sense of the older population’s expenditures, they also increased the sample size of the expenditure survey from about 7,000 households to about 10,000 households to include more pensioners. New Zealand agency officials said they validated the use of existing data in part by using existing expenditure data to confirm that goods and services most important to the older subpopulation were adequately represented in the data. They also said they consider the coverage of the subpopulation group when determining the make-up of the CPI basket. Since older people may shop at different stores than the general population, New Zealand’s statistical agency also developed separate outlet weights for the older subpopulation, which more accurately reflect the different mix of outlets, or stores, frequented by this group, according to agency officials. Overall, officials said they found that using subpopulation-specific outlet data instead of general CPI outlet data had very little impact on the index for the older subpopulation. United Kingdom agency officials said they validated the use of existing data by organizing expenditure data from the household survey into categories that align with national expenditure data, which allowed them to generate bigger samples than exist in the household survey data. As a result of the larger sample, their statistical agency said they were able to achieve more precise estimates for the index for the older subpopulation. It is also relatively uncommon for a country to produce a chained index for the purpose of adjusting national pension benefits. Of those 27 OECD countries that are using price indexation, five of them produce a chained index (Australia, Canada, the United States, the United Kingdom, and New Zealand). However, none of the OECD countries use the chained index to adjust their national pension benefits. In our three case study countries, the statistical agencies used the chained index as an analytical tool to measure bias in the CPI or for comparative purposes. Officials we spoke with said that the delay required to produce a chained index made it impractical to use the index to adjust benefits. While some of the stakeholders in selected case study countries indicated it could be theoretically possible to create a chained CPI for the older subpopulation, we did not identify any countries with such an index during this review. While government-collected data are often collected for reasons other than the production of the CPIs, the three selected case study countries are using government-collected data to help fill the gaps in data they collect expressly for the CPI (see table 2). According to agency officials in the three selected countries, use of this government-collected data improves accuracy of the CPIs and can be a relatively affordable way to supplement data collected for the CPI. National Accounts, key sources of government-collected data, are typically used for national summary measures like the Gross Domestic Product. However, all three of the selected countries are also using relevant consumption data from National Accounts to supplement their CPI data, which agency officials in Australia said is in-line with recommendations from the International Labour Organization (see text box). Australia, New Zealand, and the United Kingdom are all using their National Accounts data to supplement expenditure survey data in their CPIs, while New Zealand is also using another form of government- collected administrative data to improve its CPIs. International Guidance for Calculating CPIs and Subpopulation Indexes The International Labour Organization produces a manual that provides an overview of issues that national statistical offices can consider when making decisions on how to deal with the various problems in the compilation of Consumer Price Indexes. Researchers from many countries’ national statistical agencies, universities, and international organizations (such as the World Bank, International Monetary Fund, and Organisation for Economic Co-operation and Development) are involved in creating the manual. The manual also establishes international conventions, such as a suggestion that countries regularly evaluate the use of average wages as opposed to price indexes (and vice versa). Last published in 2004, an update to the manual is scheduled to be released in 2020. The upcoming revised manual is expected to elaborate on the use of National Accounts data and alternative data sources to develop expenditure weights. Australia. Australia’s statistical agency uses consumption data from their National Accounts to update the CPI expenditure weights more frequently than officials said was previously possible. Using this data has helped reduce substitution bias, meaning that the data better reflect changes in consumer purchases in response to price changes. Previously, Australia updated its expenditure weights every 6 years, when its household expenditure survey was released. In other words, the CPI was previously calculated assuming that consumers’ expenditure patterns did not change for 6 years. As a result, the CPI did not account for substitution patterns to different goods and services over significant periods of time, leading to bias in the CPI. In 2018, the Australian statistical agency incorporated National Accounts data in the CPI in those years when the expenditure survey was not conducted, allowing the expenditure weights to be updated annually to reflect what statistical agency officials described as more timely and relevant consumption patterns and to improve the accuracy of the data. According to Australian statistical agency officials, they did not have the budget to increase the frequency of their household expenditure survey, which they said is very costly. Instead, officials said they researched alternative ways that would allow for more frequent reweighting and settled on using the National Accounts data in between survey years to update the weights annually. This approach does not require a budget increase because the National Accounts data are already produced. Australian officials said more frequent weighting helped reduce substitution bias in their CPIs by about 0.2 percentage points per year, which can have a large impact on benefits over time. By incorporating consumption data from the National Accounts, Australian statistical agency officials said they can generate more timely and relevant CPI measures, including the subpopulation indexes. Australia’s index for the older subpopulation, called the Pensioner and Beneficiary Living Cost Index, also benefits from more frequent updates of the expenditure weights and subsequent reduction in substitution bias in the CPI, according to agency officials. Agency officials said that despite not having demographic information in the National Accounts, their methods have made use of this consumption data fit for purpose for the subpopulation indexes, and the subpopulation indexes are as methodologically sound as the primary CPI. New Zealand. New Zealand’s statistical agency also uses National Accounts data to estimate expenditure weights for insurance services, which are relatively difficult to measure in survey data, according to agency officials. Specifically, the expenditure weights for health and life insurance are based on data from the National Accounts. United Kingdom. In the United Kingdom, annual spending data from the National Accounts are the main source for CPI expenditure weights, as stakeholders noted that the National Accounts spending data are more precise and timely than their household expenditure survey. According to statistical agency officials, household expenditure data are ultimately obtained by organizing the United Kingdom’s expenditure survey data into categories that align with the National Accounts and scaling up these data to the National Accounts data. Officials said this method allows the United Kingdom’s statistical agency to achieve larger sample sizes, and thus smaller variances and more precision in estimates for subgroup indexes. United Kingdom officials said that their National Accounts estimates are more accurate and comprehensive than their household expenditure survey, which has a smaller sample size of nearly 6,000 households. Having more accurate expenditure data and weights leads to a more accurate and relevant primary index for pension benefits, as well as a more accurate subpopulation index, according to agency officials. The National Accounts data also help the United Kingdom adjust for any potential underreporting of particular goods in the household expenditure survey, such as alcohol, further increasing the accuracy and relevance of the dataset, according to officials. Collecting prices directly from the source is more accurate than relying on someone to recall how much they spent on items, according to one stakeholder. Government agencies from selected countries also produce other administrative data that can be useful in measuring the CPI. For example, New Zealand’s statistical agency partnered with the Ministry for Business, Innovation, and Employment to use its tenancy bond database, which covers approximately 85 percent of all rental housing units in the country. These data facilitated a new way to measure rent in their CPI. Moreover, this partnership enabled New Zealand’s statistical agency to create an index of rent prices monthly, instead of quarterly, which resulted in a more accurate and timely depiction of what people are spending on rent and a more accurate indexation of benefits overall. According to agency officials, the transition to these administrative data replaced the CPI survey of landlords, and in doing so it lowered respondent burden, increased the timeliness of the rental component of New Zealand’s CPI, and improved population coverage. In all of our case study countries, various data are used to measure housing prices (see text box). Housing and the Consumer Price Index Measuring the change in housing prices for CPI is widely acknowledged by experts to pose methodological and data challenges. In response, national statistical agencies have developed a variety of approaches to address the measurement of owner-occupied housing costs, both in the primary CPI and subpopulation indexes. Officials in the national statistical offices of the case study countries said that one of the factors underlying the approach to housing is whether the measure should reflect inflation in the economy overall or inflation as experienced by households. In Australia and New Zealand, the primary CPI includes price changes stemming from the purchase of a new home but not via mortgage interest payments (known as the acquisitions approach), while the subpopulation index excludes the purchase of a new home but includes mortgage interest (referred to as outlays or payment approach). In the United Kingdom, there are two versions of the primary CPI: one that uses “rental equivalence” (a calculation of what the owner would pay in rent for an equivalent house) and one that excludes owner-occupied housing costs. In addition, the United Kingdom’s subpopulation index uses a payments approach. Officials in our selected case study countries said they are using alternative big data sources, such as web-scraping data and transactional (scanner) data to help them more accurately index their national pension benefits (see table 3). These officials said that these alternative data sources allow countries to obtain a higher volume of data and more accurate data to incorporate into their CPIs, subsequently making the indexation of benefits more accurate. Electronic price data obtained from a retailer, whether through the retailer’s website or through scanner data the retailer shares with the national statistical agency, reflects accurate and timely data on the price and quantity of goods and services sold. Electronic price data can be an improvement over data collected in household expenditure surveys, for example, as several experts and agency officials in one case study country noted that household expenditure surveys suffer from recall bias, resulting in less accurate spending data. The three selected countries are at different stages of incorporating scanner data into their CPI. Officials at the national statistical agencies in all three of our case-study countries stated that they are primarily focused on incorporating scanner data from grocery stores into their CPI. Using grocery store data is possible, in part, because these countries contain a relatively small number of stores that dominate grocery sales, according to agency officials, which is a difference from the United States. Australia. According to stakeholders, the Australian statistical agency developed a formula that incorporates a chained formula into a portion of the CPI using high- frequency scanner data from the country’s dominant grocery stores, which provides timely price and expenditure data on food items for their indexes. Integrating this type of high- frequency data is not easy, they said, since the traditional CPI formulas are not built to handle the volume of data that scanner data produce. However, in consultation with academics and statistical agencies from around the world, Australia was able to develop a chained formula that uses an innovative statistical method, known as a multilateral approach, to incorporate the scanner data. As a result, the portion of the CPI for which Australia has scanner data (about one-sixth of the CPI, comprised mostly of food and other grocery data) is based on a chained formula. Incorporating these data allows the country to include all of the products available in the datasets, rather than a small sample of products, leading to a more accurate calculation of food prices and a more accurate index overall, for both the general population and the older subpopulation, according to agency officials. Stakeholders in Australia noted that the international price statistics community has since reached a consensus that multilateral methods are the most effective way to capitalize the full amount of information provided in scanner data, and they said that the forthcoming update of the International Labour Organization’s CPI Manual is expected to recommend this method as well. New Zealand. New Zealand’s statistical agency is working towards incorporating more scanner data, primarily from its two large supermarket chains, in the production of the country’s CPIs, which will help achieve a more accurate index for both the general population and the older subpopulation, according to agency officials. New Zealand started using retail scanner data to supplement its expenditure data in its CPI in 2006, and in 2014 New Zealand incorporated direct measurement from scanner data for consumer electronics products into its CPI. Officials from the national statistical agency said they hope to expand their use of this type of big data in the near future. They have already received the data from supermarkets, whose goods account for roughly 20 percent of the goods and services in the CPI, but they have not yet integrated the data into their CPIs. Agency officials said they expect to integrate this in the next year. New Zealand’s statistical agency officials said they have a goal to obtain scanner data for other CPI components soon as well, such as fuel. United Kingdom. In the United Kingdom, agency officials said improvements in technologies have resulted in new alternative sources for price data that could be used in the compilation of their price indexes in the near future. The United Kingdom’s statistical agency is currently exploring both scanner data and online price data. The agency currently has several streams of research looking into the expanded use of alternative data, including research studying the feasibility of moving away from collecting prices manually towards using electronic means wherever feasible and efficient. The agency is now receiving web-scraped data from an online source that captures prices from online sales of goods like clothing. The United Kingdom’s statistical agency is also continuing to engage with retailers on receiving scanner data covering areas such as clothing and groceries, targeting some of the largest retailers from which the agency currently manually collects prices. These data sources may provide a more efficient way to capture the increase in online expenditures that has occurred over the last decade, and will likely continue to occur. These new data are initially being used for research work, but over time the web-scraped online prices and scanner data will be used when calculating primary inflation indexes, according to agency officials. The research done by the United Kingdom’s statistical agency into grocery store items has also enabled officials there to explore different methods of collecting web scraped prices in-house. The officials said this has led to wider benefits for the agency in general, with an increase in knowledge and experience that has contributed to the success of other big data projects. Our selected case-study countries use committees with stakeholders and advisory panels, including academic researchers with subject matter expertise, to implement innovative changes to their CPIs (see table 4). The statistical agencies in these three countries have shown a willingness to act on recommendations that came out of these collaborative efforts. These countries are also seeking input from the international statistical community, which country officials said has led to positive developments in their CPIs. Australia. Australia’s statistical agency has taken a variety of approaches to collaborate with external stakeholders, which agency officials said has led to positive changes to their CPIs, and thus indexation of benefits over the years. According to agency officials, Australia’s collaborative efforts include: conducting regular reviews and seeking stakeholder input every 6 years with the release of the expenditure survey; convening workshops with stakeholders including both academics and users (e.g., the agencies that distribute benefits); participating in international conferences to receive feedback on changes to the country’s CPI and subpopulation indexes; partnering with methodology experts in other agencies such as the Treasury and central bank, occasionally by obtaining staff on detail; and commissioning reports that research and review measures to strengthen the financial security of seniors. These reviews and associated collaborative efforts have helped the agency learn more about the issues it faces and have helped trigger changes that will improve the accuracy of the nation’s CPI, according to agency officials. For example, as discussed above, agency officials said that a 2011 CPI review revealed concern by the Reserve Bank of Australia and others that the infrequent reweighting was resulting in bias in the CPI that affected inflation targeting by the central bank, as well as benefit expenditures. This review helped spur innovations, such as including the incorporation of scanner data into the nation’s CPI, which delivered positive results with respect to more timely and relevant data being used to estimate inflation. Australia’s statistical agency officials said they sought extensive input from key governmental stakeholders, a number of academic experts, as well as international experts to research how to best incorporate scanner data into their CPI, which agency officials noted was necessary to facilitate the integration of high-frequency scanner data into the CPI. They also conducted numerous bilateral and multilateral consultations with key stakeholders in the government that use CPI data, including the Reserve Bank of Australia, the Treasury, Department of Finance, Department of Social Services, and State Treasuries. Australian statistical agency officials suggested that consulting with users of the data frequently was an important part of implementing changes to the measurement of the CPI and subpopulation indexes. New Zealand. New Zealand’s statistical agency has also used CPI advisory committees composed primarily of external stakeholders who make use of the agency’s CPIs. For example, in 2013 New Zealand’s statistical agency convened a committee to independently review the methods and practices used to compile the CPI and make recommendations, for example, about how additional indexes should be measured. The committee also incorporated public submissions on the scope and uses of the CPI, for example, from nongovernmental organizations and interest groups such as retiree advocacy groups. The committee then released a report recommending the creation of additional CPIs that are designed for microeconomic purposes, such as the indexation of retirement benefits, to better reflect changes in the purchasing power of the incomes of particular subgroups of the population, like the older subpopulation. The committee also recommended that New Zealand’s statistical agency review the sample size and collection methods of their expenditure survey to improve the reliability of expenditure estimates of the required population subgroups so that the estimates could eventually be of high enough quality to be published, which they subsequently were. According to officials, the committee’s report helped lead to the creation of New Zealand’s subpopulation indexes. Moreover, the committee recommended that the statistical agency try to use retail scanner data to measure price change and stated that the method aligns with international best practices. New Zealand’s statistical agency recognized these best practices and the international consensus that multilateral methods are the preferred way to incorporate big data. Indeed, it has started to use these methods in the rental prices data and it plans to continue to research implementing these methods further. United Kingdom. The United Kingdom has also developed advisory panels on consumer prices to provide independent advice to the National Statistician, which officials said has allowed the United Kingdom’s statistical agency to learn more about challenges with the nation’s CPIs and to find possible solutions. Similar to the United States, the United Kingdom has advisory groups on technical issues, as well as on the uses of price indexes. The reports published by various advisory groups have raised technical issues with the Retail Price Index (RPI), which is the United Kingdom’s longest running measure of inflation. These technical issues resulted in the RPI being higher than the CPI. Ultimately, agency officials said consultations and advisory panel input helped lead to the RPI being decertified as a national statistic (see text box). The United Kingdom’s statistical agency also hosted numerous meetings and a collaborative workshop about the conceptual foundations of its subpopulation indexes, which are currently being developed. According to agency officials, obtaining input from internal and external stakeholders has been critical to developing solutions to indexation challenges. The United Kingdom’s Experience Changing Price Index Used for Pension Adjustments Changing the index used for benefit adjustments can be difficult, as switching price indexes can involve tradeoffs. For example, public and private pension benefits in the United Kingdom have traditionally been indexed by the Retail Price Index (RPI), the oldest index in the United Kingdom. The United Kingdom recently switched indexation of certain government benefits, including pension benefits, from the RPI to the slower-growing CPI. This is expected to result in lower payouts from the government. In contrast, the government continued using the faster- growing RPI for some provisions, such as student loan interest rates, that resulted in higher payments to the government. Stakeholders suggested that having multiple measures of inflation can create incentives for the government to use different indexes for its own budgetary advantage, with pensioners receiving lower benefit adjustments and students facing relatively higher loan payment adjustments. The United Kingdom’s experience highlights that changing the index for benefits may result in advantages and disadvantages for different groups and thus may be politically difficult, according to agency officials. Federal retirement programs like Social Security have relied upon a subpopulation price index to adjust benefits since automatic cost-of-living adjustments were first enacted almost 45 years ago. This index estimates changes in purchasing power for wage earners as opposed to changes in the standard of living or some other type of measurement. In recent years, numerous legislative proposals have been suggested to change this index from one that measures the purchasing power of wage earners to one that targets some different population, for example one solely focused on the elderly. Much of the debate over using a different index has centered on the ability (i.e., the accuracy) of the indexes to capture changes in the cost of living for a particular group in society. BLS is unsure whether the data sources it currently uses are adequate to produce accurate CPI-E and CPI-W subpopulation indexes on a timely basis, according to BLS officials and documentation. While the CPI-E is experimental and not used by federal programs, the CPI-W is used to adjust billions of dollars of Social Security and other federal retirement program benefits. It is therefore critical that the measurement be as accurate as possible. However, ensuring the measurement’s accuracy may require a reexamination of the underlying data used to produce the subpopulation indexes. BLS has not evaluated the adequacy of existing data because it is costly to undertake a full evaluation, according to agency officials. But experts we interviewed, including some on BLS advisory groups, indicate there may be cost-efficient ways to conduct such a review. Although the experiences of other countries may not be directly applicable, other countries have found ways to evaluate the use of existing data for their subpopulation indexes, and officials in all three of our case study countries expressed the view that some of these methods were cost efficient. Absent BLS evaluating the adequacy of the existing data it uses to produce its subpopulation indexes, BLS will continue to be uncertain if its subpopulation indexes are accurate and it may not learn of potential areas for improvement. In addition, BLS currently relies on the Consumer Expenditure Survey to produce expenditure weights that measure the mix of goods and services consumers purchase and, because of survey shortcomings and processing lags, the weights reflect spending patterns that can be up to 4 years out of date. Although BLS has taken other steps to improve the accuracy, timeliness, and relevance of data used in the CPIs, BLS has not fully explored the potential to update expenditure weights on a more frequent basis using annual data from the National Accounts, which are currently collected in part to measure Gross Domestic Product. While not specifically designed for use in CPIs, the National Accounts data may provide BLS an opportunity to supplement Consumer Expenditure Survey data in the intervening years. Moreover, some literature we reviewed indicated that the use of National Accounts data has the potential to mitigate measurement error in the Consumer Expenditure Survey, thereby increasing accuracy. Without adequately exploring the potential of such an option, BLS may be missing an opportunity to improve its CPIs. We are making the following two recommendations to the Department of Labor: The Secretary of Labor should ensure that BLS explores cost-efficient ways to evaluate the data sources currently used to produce subpopulation indexes, such as by engaging more directly with other stakeholders or seeking input from its advisory groups and other knowledgeable entities about approaches to expand data collection in a cost-efficient manner. (Recommendation 1) The Secretary of Labor should ensure that BLS explores the use of already collected National Accounts data to produce more accurate, timely, and relevant CPIs. (Recommendation 2) We provided a draft of the report to the Department of Labor, the Social Security Administration, and the Department of State for their review and comment. We also sent an informational copy to the Bureau of Economic Analysis. The Department of Labor and the Social Security Administration provided technical comments, which we have incorporated where appropriate. In an email, the Department of State said it had no comments on the report. The Department of Labor also provided written comments, which are reproduced in appendix III and discussed below. In its written comments, the Department of Labor stated that BLS continually improves its measures according to a guiding principle to provide accurate, objective, relevant, timely, and accessible information. The Department of Labor agreed with the first recommendation to explore cost-efficient ways to evaluate the data sources currently used to produce subpopulation indexes and stated that it would continue to investigate improvements to subpopulation indexes. The Department of Labor disagreed with the second recommendation to explore the use of National Accounts data in the construction of its indexes, stating that the National Accounts data are not a replacement for Consumer Expenditure Survey data. While we agree that the National Accounts data are not a wholesale replacement for the Consumer Expenditure Survey data, we believe that it would be useful to examine National Accounts data as an augmenting, alternative source of data that could supplement or enrich the Consumer Expenditure Survey. Such an effort could potentially lead to more accurate, timely, and relevant CPIs. Although the Department of Labor stated that the Consumer Expenditure Survey is a continuous survey and that data are received quarterly, most CPIs still rely on expenditure weights based on Consumer Expenditure Survey data that are up to 4 years out-of-date. In addition, the Consumer Expenditure Survey faces increasing costs and declining response rates. The Department of Labor stated in its comments that it is exploring ways to accelerate the data collection and processing time and that it periodically investigates the frequency of updating expenditure weights. We commend the Department of Labor for considering these efforts, and we maintain that they could take further action to explore additional opportunities for improvement. For example, the Department of Labor could research the extent to which there are instances or categories for which the National Accounts data could be used to produce more up-to- date expenditure weights than the Consumer Expenditure Survey. As we noted in our report, Department of Labor officials told us they periodically examine National Accounts expenditure data to explore differences with the Consumer Expenditure Survey data, not to explore supplementary use of alternative data. While it cannot be ensured that every expenditure data point in the National Accounts will be of use for producing CPIs, we maintain that further exploring the National Accounts expenditure data as a complement to the Consumer Expenditure Survey data may provide opportunities for BLS to improve the accuracy, timeliness, and relevance of its CPIs. We are sending copies of this report to the Secretary of Labor, the Commissioner of Social Security, and the Secretary of State. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or jeszeckc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Appendix II: Additional Information about Selected Case Study Countries technical corrections as necessary. We note also that the fact that a legal feature was successful in one or more of the countries we visited, which may have significantly different cultures, histories, and legal systems than the United States, does not necessarily indicate that it would be successful in the United States. Charles A. Jeszeck, (202) 512-7215 or jeszeckc@gao.gov In addition to the contact named above, Michael Collins (Assistant Director), Laura Hoffrey (Analyst in Charge), Emilio Fonseca, Kathleen McQueeney, Tom Moscovitch, and Julie Miller made key contributions to this report. Also contributing to this report were Deborah Bland, Alicia Cackley, Charles Ford, Sarah Gilliland, Susan Irving, Kelsey Kreider, Sheila McCoy, Jessica Orr, Oliver Richard, Joseph Silvestri, Almeta Spencer, Curtia Taylor, Frank Todisco, Walter Vance, Adam Wendel, and Sirin Yaemsiri.", "summary": "In the United States, federal retirement programs typically include cost-of-living adjustments based on a CPI that measures inflation for a subpopulation of workers. This includes Social Security, which provides benefits for more than 60 million older Americans, workers with disabilities, and their families. As the life expectancy of Americans continues to increase, more Americans will be subject to these adjustments, so it is critical for them to be accurate. GAO was asked to review U.S. and international efforts to measure the cost of living for older populations. This report examines (1) key issues that BLS faces in measuring the cost of living for older Americans; and (2) the experiences of other countries that developed alternate methods of adjusting retirement benefits. GAO reviewed pertinent literature; assessed BLS efforts to measure inflation; conducted case studies in three countries—Australia, New Zealand, and the U.K.—with a variety of CPIs, which GAO selected based on expert referral and document review; and interviewed agency officials and experts. The U.S. Bureau of Labor Statistics (BLS) faces accuracy, timeliness, and relevancy challenges developing consumer price indexes (CPI) for subpopulations of blue-collar workers and older Americans. For example, the CPI for these workers is used to adjust federal retirement benefits for inflation, including Social Security. BLS has not evaluated the extent to which its existing data are adequate to produce CPIs that reflect what these subpopulations pay, where they shop, and what they purchase. Officials cite budgetary reasons for not having done this, but there may be cost-efficient methods for evaluating the adequacy of these data. Without an evaluation, federal retirement benefits could be subject to adjustment based on potentially inaccurate information. Additionally, BLS has made limited use of certain data already collected by the federal government—such as National Accounts data on U.S. production and consumption—that could be used to increase the accuracy, timeliness, and relevancy of CPI calculations that reflect the mix of goods and services consumers purchase. Without adequately exploring the potential of using these data, BLS may be missing an opportunity to improve its CPIs. Reports about the retirement systems in the 36 Organisation for Economic Co-operation and Development countries indicate that most use their primary measures of inflation to adjust government retirement benefits. In addition, all three of GAO's case study countries (Australia, New Zealand, and the United Kingdom, or U.K.) have a variety of CPIs, including for subpopulations, and they filled information gaps in their CPIs with National Accounts and other data. For example, Australia and the U.K. use National Accounts data annually to update their calculations of the mix of goods and services consumers buy, thereby making the CPIs more relevant and accurate. All three countries also collaborated with stakeholders—such as other agencies—to implement changes, for example by gathering input on the design of subpopulation CPIs. GAO recommends that BLS explore cost-efficient ways to evaluate the data currently used to produce subpopulation indexes, and explore the use of National Accounts data to produce more accurate, timely, and relevant CPIs. BLS agreed with the first recommendation but disagreed with the other. GAO continues to believe both recommendations are warranted, as discussed in the report.", "document_type": "gao"}
{"report": "Sanctions are imposed pursuant to statute, executive order, or other authorities. For example, the President may use authorities granted in the International Emergency Economic Powers Act (IEEPA) and the National Emergencies Act (NEA) to issue executive orders authorizing sanctions. The United Nations Participation Act of 1945 provides the basis for the U.S.’s implementation of United Nations Security Council sanctions mandated under Article 41 of the United Nations Charter. Sanctions provide a range of tools that Congress and the President may use to attempt to alter or deter the behavior of a foreign government, an individual, or an entity in furtherance of U.S. national security or foreign policy objectives. For example, sanctions may be imposed in response to human rights abuses, weapons proliferation, or occupation of a foreign country. Sanctions may include actions such as limiting trade; blocking assets and interests in assets subject to U.S. jurisdiction; limiting access to the U.S. financial system, including limiting or prohibiting transactions involving U.S. individuals and businesses; restricting private and government loans, investments, insurance, and underwriting; and denying foreign assistance and government procurement contracts. The United States imposes comprehensive sanctions and targeted sanctions. Comprehensive sanctions generally include broad-based trade restrictions and prohibit commercial activity with an entire country. Examples of comprehensive sanctions include U.S. sanctions against Iran and Cuba. Targeted sanctions restrict transactions of, and with, specific persons or entities. For example, the U.S. sanctions program related to Somalia targets persons engaging in acts threatening the peace, security, or stability of that country. Sectoral sanctions are a form of targeted sanctions directed at a specified sector, or sectors, of a target’s economy. For instance, Executive Order 13662 authorized sanctions targeting persons operating in certain sectors of the Russian economy as might later be determined by the Secretary of the Treasury in consultation with the Secretary of State, such as the financial services, energy, mining, and defense and related materiel sectors. Supplementary sanctions, also known as secondary sanctions, target third-party actors doing business with, supporting, or facilitating targeted regimes, persons, and organizations. For example, in February 2017, Treasury imposed sanctions against 13 individuals and 12 entities for their involvement in, or support for, Iran’s ballistic missile program as well as for acting for or on behalf of, or providing support to, Iran’s Islamic Revolutionary Guard Corps–Qods Force. OFAC’s implementation of sanctions includes publishing the Specially Designated Nationals and Blocked Persons List of individuals, groups, and entities whose assets in the United States are blocked and with whom U.S. persons are prohibited from dealing. The addition of an individual, group, or entity to this list is referred to as a sanctions designation. Agencies may issue licenses to authorize transactions with sanctioned entities that otherwise would be prohibited by existing sanctions. According to OFAC, many of its licensing determinations are guided by U.S. foreign policy and national security concerns. In making these determinations, OFAC must often coordinate with State and other government agencies, such as Commerce. OFAC issues two types of licenses: (1) general licenses, which authorize a particular type of transaction for a class of persons without the need to apply for a specific license, and (2) specific licenses, which OFAC issues to a particular person or entity to authorize a particular transaction. Commerce’s Bureau of Industry and Security (BIS) issues two forms of authorization: (1) an individual validated license requiring an application and (2) a license exception allowing an export or reexport, under stated conditions, for which no application is required. Laws and executive orders establishing sanctions may designate agency implementation roles. Some sanctions-related executive orders designate both primary and consultative agencies. For example, Executive Order 13818 establishes sanctions that include blocking the U.S. assets of persons whom the Secretary of the Treasury, in consultation with the Secretary of State and the Attorney General, determines to be responsible for, or complicit in, serious human rights abuse, among other measures. Executive orders may also broadly direct U.S. government agencies to take appropriate measures within their authorities to perform specified functions and duties. When roles are not assigned by the law or executive order authorizing the sanctions, agency roles are typically assigned through an interagency process. The IEEPA and the NEA mandate that the President report to Congress when using authorities granted under those laws. The IEEPA requires the President to report, among other things, actions taken in the exercise of IEEPA authorities to Congress at least once during each succeeding 6- month period following the administration’s initial reporting of the authorities’ use. The NEA requires the President to transmit a report to Congress within 90 days after the end of each 6-month period following a declaration of a national emergency, providing the total U.S. government expenditures that are directly attributable to the exercise of powers and authorities conferred by declaration of the emergency. The President has delegated responsibility for many of these reports to the Secretary of the Treasury. However, the President delegated responsibility for the report on the National Emergency With Respect to Proliferation of Weapons of Mass Destruction, Executive Order 12938, to the Secretary of State. The Foreign Narcotics Kingpin Designation Act (Kingpin Act), enacted in 1999, mandates that the President prepare classified reports by July 1 of each year that include the number of new Kingpin Act designations and the personnel and resources directed toward the imposition of Kingpin sanctions. The Trade Sanctions Reform and Export Enhancement Act of 2000 (TSRA) mandates that the applicable department or agency submit quarterly and biennial reports on activity under the act regarding the department or agency’s determinations and processing of license applications for export of agricultural commodities, medicines, and medical devices to specified entities and destinations, including state sponsors of terrorism. OFAC and Commerce’s BIS submit reports in response to the TSRA. To implement sanctions, agencies need to identify the human resources needed for the work. Strategic workforce planning focuses on developing long-term strategies for acquiring, developing, and retaining an organization’s total workforce to meet the needs of the future. Agency approaches to such planning can vary with each agency’s particular needs and mission. We have previously identified five principles that a strategic workforce planning process should address: 1) Involve top management, employees, and other stakeholders. 2) Determine the critical skills and competencies that will be needed. 3) Develop strategies that are tailored to address gaps in number, deployment, and alignment of human capital approaches. 4) Build the capability needed to address administrative, educational, and other requirements important to support workforce strategies. 5) Monitor and evaluate progress toward human capital goals and the contribution that human capital results have made toward achieving programmatic goals. Treasury, State and Commerce have units dedicated primarily to sanctions implementation and also have units with roles in sanctions implementation in addition to other responsibilities. Other agencies, including the Departments of Defense, Energy, Homeland Security, and Justice and federal financial regulatory agencies, play specific roles in sanctions implementation based on their expertise or broader duties. Agencies’ roles in sanctions implementation may be assigned to them in legislation, by executive order, in presidential memorandums, or through the interagency process. Table 1 shows the roles that agencies may have in sanctions implementation and examples of agency actions associated with each role. Treasury, State, and Commerce each have units that focus primarily on sanctions implementation and that act in all five of the roles we identified. Treasury. Treasury’s OFAC, part of the department’s Office of Terrorism and Financial Intelligence (TFI), administers and enforces economic sanctions based on U.S. foreign policy and national security through consultation with the Secretary of State. OFAC acts under presidential national emergency powers, as well as authority granted by specific legislation, to impose controls on transactions and freeze assets under U.S. jurisdiction. OFAC consists of four offices: The Office of Sanctions Policy and Implementation leads OFAC’s design, implementation, and evaluation of sanctions programs and develops OFAC’s public guidance, licenses, and regulations. The Office of Compliance and Enforcement works to promote compliance with OFAC’s sanctions programs and investigates apparent violations. The Office of Global Targeting works with other units within TFI, other U.S. agencies, and foreign partners to identify and investigate targets for sanctions designation. The Office of Sanctions Support and Operations supports all sanctions-related functions at OFAC, including human capital and budgetary functions. State. State’s Office of Economic Sanctions Policy and Implementation (SPI)—housed in the Bureau of Economic and Business Affairs, Division for Counter Threat Finance and Sanctions—is responsible for providing foreign policy guidance for the vast majority of sanctions programs and obtaining international cooperation with U.S. agencies enforcing sanctions. According to SPI, it acts as State’s central coordinating office for 25 of the 30 sanctions programs that were active as of April 2019. SPI also implements sanctions under authorities delegated to the Secretary of State, including sanctions on Iran and Syria. Commerce. In Commerce’s BIS, the Foreign Policy Division (FPD) of the Office of Nonproliferation and Treaty Compliance is one of the components that implements sanctions through U.S. export controls. The division is responsible for developing, analyzing, evaluating, and coordinating export controls related to sanctions policy. In addition to having units that primarily focus on sanctions, Treasury, State, and Commerce have units that carry out roles in sanctions implementation in addition to other responsibilities. Treasury. Treasury has several other units that support sanctions implementation. For example, in TFI, the Office of Intelligence and Analysis examines classified and unclassified reporting, financial transactions, and open-source databases for evidence of sanctions violations. The Financial Crimes Enforcement Network monitors and analyzes financial information on threats, producing intelligence reports that may identify targets for designation and sanctions violators. In addition to TFI units, the Internal Revenue Service, the Office of International Affairs, and the Office of the General Counsel also have roles in sanctions implementation. For example, the Office of International Affairs helps to assess the likely impact of sanctions and conducts outreach to foreign counterparts regarding sanctions implementation. State. Units at State have sanctions implementation roles related to their expertise. Some of these units take actions in all five of the sanctions roles shown in table 1 and are responsible for specific sanctions authorities within State, according to State officials. For example, the Bureau of International Narcotics and Law Enforcement Affairs is responsible for coordinating and communicating State’s position on existing or proposed new sanctions in relation to the Kingpin Act and transnational criminal organizations. According to State officials, the Bureau of Counterterrorism and Countering Violent Extremism leads State in designating Specially Designated Global Terrorists under Executive Order 13224 and Foreign Terrorist Organizations under Section 219 of the Immigration and Nationality Act. The Bureau of Economic and Business Affairs’ Office of Threat Finance Countermeasures has a primary role in implementing sanctions under Executive Order 13224, which targets terrorist financiers and others who provide material support to terrorists. Commerce. Commerce has several other units that support sanctions implementation. For example, the Office of Export Enforcement provides input regarding sanctions proposals and feedback regarding any adverse impact to existing investigations. The Office of National Security and Technology Transfer Controls implements primarily sectoral sanctions by providing technical analyses of items and recommendations during sanctions development. The Office of Exporter Services provides a range of resources, including electronic resources and educational seminars, which provide exporters with guidance on export compliance processes and procedures. Table 2 provides an overview of the various roles that Treasury, State, and Commerce units play in sanctions implementation. See appendix II for additional details. Several other agencies have more-specific roles in sanctions implementation, with the extent of their involvement dependent largely on their area of expertise. These agencies carry out their sanctions-related roles in addition to other responsibilities. Department of Defense. The Office of the Under Secretary of Defense for Policy contributes to sanctions implementation, participating in all roles except targeting. The office coordinates department units’ reviews of sanctions proposals, provides the department’s recommendation to interagency partners during sanctions development, and represents the department during interagency discussions regarding sanctions enforcement. Department of Energy. The National Nuclear Security Administration supports sanctions by providing technical analyses of weapons of mass destruction and conventional arms transactions that may be subject to sanctions and by providing recommendations during sanctions development. The National Nuclear Security Administration also reviews export licenses for munitions and items with both military and commercial applications, known as dual-use items, which may include parties subject to sanctions. Department of Homeland Security. Units of the Department of Homeland Security also have varied roles in sanctions implementation. For example, the Human Rights Violators and War Crimes Unit in U.S. Immigration and Customs Enforcement’s Homeland Security Investigations includes a Global Magnitsky investigative support team, which targets serious human rights abusers and corrupt foreign officials through OFAC sanctions and visa denials. Units in U.S. Customs and Border Protection maintain a list of sanctioned countries and couriers for which shipment applications are rejected and use an automated targeting system to identify high- risk shipments and coordinate appropriate enforcement actions. Department of Justice. Multiple Department of Justice units contribute to sanctions implementation, participating in all roles except licensing. For example, the National Security Division works with law enforcement partners to facilitate the investigation and prosecution of sanctions violators. Financial regulatory agencies. Financial regulatory agencies with roles in sanctions implementation may review the compliance programs of the institutions they oversee with respect to OFAC requirements. Some of these agencies can also enforce penalties for significant deficiencies in institutions’ OFAC compliance programs. Financial regulatory agencies generally examine institutions’ compliance with OFAC policies concurrently with examinations for compliance with the Bank Secrecy Act (BSA) and anti–money laundering (AML) statutes. Table 3 provides an overview of the various roles of these agencies in sanctions implementation. Also see appendix II for additional details of agency units’ sanctions implementation roles. See appendix III for information about agency units’ number of personnel with sanctions implementation responsibilities. All three of the sanctions implementation units we reviewed have generally received steady or increasing resources since fiscal year 2015 but have faced challenges in filling some positions. OFAC has received increasing inflation-adjusted budgetary and authorized human resources each fiscal year since 2015 but has consistently experienced a gap between the number of authorized and actual full-time equivalents (FTEs). OFAC officials attributed the gap to challenges in hiring due to competition from other agencies and the private sector and the time needed for new hires to obtain security clearances. State SPI has also generally received additional authorized inflation-adjusted budgetary and human resources but has not been fully staffed in recent years. Commerce’s FPD has received relatively steady inflation-adjusted budgetary resources but, according to Commerce officials, lacks funding to fill one of its 10 positions. OFAC received increasing budgetary resources in each of the last 5 fiscal years. In inflation-adjusted terms, OFAC’s budgetary resources increased by a total of 58 percent, from approximately $29.7 million in fiscal year 2014 to approximately $46.8 million in fiscal year 2019. (See fig. 1.) OFAC has also received authority to hire additional FTEs since fiscal year 2014, yet a number of the additional authorized positions have remained unfilled. According to OFAC officials, OFAC allocated most of its additional authorized FTEs to the Office of Global Targeting, which is responsible for conducting investigations of sanctions targets. At the start of fiscal year 2014, 10 of OFAC’s 173 authorized positions (6 percent) were unfilled. By the start of fiscal year 2020, 55 of OFAC’s 259 authorized positions (21 percent) were unfilled. In the intervening period, the gap between authorized and actual FTEs at the start of each fiscal year ranged from 34 to 58 positions (14 to 26 percent of authorized FTEs). (See fig. 2.) Despite the increase in authorized FTEs, OFAC has faced challenges in filling the additional positions. At the start of fiscal year 2020, 21 percent of OFAC’s authorized sanctions investigator positions (13 of 62) were not filled. Also unfilled were nine of 25 OFAC sanctions licensing officer positions, three of 18 enforcement officer positions, two of 15 sanctions policy analyst positions, and six of 14 sanctions compliance officer positions. Officials of both OFAC and Treasury’s Office of the Assistant Secretary for Management cited three primary challenges in hiring candidates with the necessary qualifications: Competition with other agencies, including those in the intelligence community, which can use direct-hire authority to expedite the hiring process Competition with the private sector, which offers higher salaries The time required for security clearance processing, which delays hiring for positions, such as sanctions investigators, who need a special sensitive investigation that must be adjudicated at the top secret/sensitive compartmented information levelTreasury does not currently have direct-hire authority for OFAC but can use other authorities to address hiring challenges. OFAC can use TFI’s agency-specific schedule A authority, which excepts up to 100 positions at TFI from competitive selection requirements; schedule A authority is not specific to OFAC. In August 2019, officials of Treasury’s Office of the Assistant Secretary for Management stated that the office was not seeking direct-hire authority through the Office of Personnel Management. Additionally, the officials noted that Treasury has used flexibilities such as veterans’ hiring preferences to fill positions. However, in December 2019, Treasury officials stated that they had determined to seek direct-hire authority and would support the passage of legislation providing such authority. SPI received annual budgetary resource increases in fiscal years 2015 through 2018, before a slight decline in fiscal year 2019. In inflation- adjusted terms, SPI budgetary resources increased overall by 42 percent, from $2.3 million in fiscal year 2014 to $3.2 million in fiscal year 2019. (See fig. 3.) SPI has received authority to hire six additional FTEs for fiscal year 2020, but more than half of its authorized positions were vacant at the start of the year. SPI’s authorized FTEs ranged from 13 to 16 in fiscal years 2014 to 2019 and increased to 21 FTEs for fiscal year 2020. At the start of each fiscal year from 2014 to 2019, SPI had one to three fewer actual FTEs than authorized. However, the increase in authorized FTEs for fiscal year 2020 followed a decline in the number of filled positions during fiscal year 2019, when SPI lost more than a third of its staff. As a result, as of the beginning of fiscal year 2020, more than half of SPI’s 21 authorized FTEs were unfilled. (See fig. 4.) According to SPI officials, the departures during fiscal year 2019 were for the most part unscheduled and resulted from staff promotions, moves to elsewhere in State, or resignations to accept positions in other agencies or the private sector. SPI officials added that a department-wide backlog in hiring constrained SPI’s ability to fill these gaps and that the office would have to pay for the additional six FTEs without an increased budget. As of December 2019, State was recruiting to fill some of these positions, according to SPI officials. SPI expected one staff member to start in early January, had extended an offer to another, and was advertising to fill four additional positions. While SPI has generally received increased budgetary resources and authorized FTEs in recent years, State discontinued the Office of the Coordinator for Sanctions Policy, formerly housed in the Office of the Secretary. The office was responsible for, among other things, coordinating sanctions strategies, integrating sanctions into foreign policy plans, and analyzing the effects of sanctions. According to data that State provided, the office had an authorized staff of seven FTEs at the start of each fiscal year from 2014 through 2018, with the exception of fiscal year 2016, when eight FTEs were authorized. State also reported that the office had one to four unfilled positions at the start of each fiscal year during this period. FPD received an overall increase in budgetary resources from fiscal year 2014 to fiscal year 2019, but most of the increase occurred from fiscal year 2014 to fiscal year 2015. Overall, FPD’s budgetary resources increased by 28 percent, adjusted for inflation, from fiscal year 2014 to fiscal year 2019. However, after a 24 percent increase in fiscal year 2015, resources remained steady through fiscal year 2019 at approximately $1.4 million per year, adjusted for inflation. (See fig. 5.) FPD has had the same number of authorized FTEs since fiscal year 2014, maintaining an authorized level of 10 FTEs from fiscal year 2014 to fiscal year 2020. FPD generally had one fewer actual FTE than authorized as of the beginning of each fiscal year. (See fig. 6). At the beginning of fiscal year 2020, according to Commerce officials, the Foreign Policy Division lacked funding to advertise and hire for the vacant position. According to Commerce officials, FPD receives a funding amount for personnel and the funding they have received is sufficient for nine FTEs. Officials at sanctions-focused units at Treasury, State, and Commerce all described their use of the annual budget process to assess their resource needs, and Treasury and Commerce have undertaken broader planning efforts. Treasury’s OFAC has begun an internal workforce planning process that, if implemented as described, would satisfy principles for strategic workforce planning that we have previously identified. According to State SPI officials, SPI assesses its resources in the annual budget formulation process and has been able to add temporary positions in response to workforce needs. Commerce BIS officials stated that they shift resources in response to needs, and BIS has previously prepared a budget strategy that included its office primarily responsible for sanctions implementation. Treasury, State, and Commerce all face challenges in measuring changes in their sanctions workload over time. Treasury’s OFAC reviews and requests resources as part of the annual TFI budget development process, which considers OFAC’s requests along with those of other TFI components. According to OFAC officials, OFAC submits its funding and resource needs to TFI for consideration. The OFAC budget justification for TFI includes the number of positions requested for all OFAC components as well as a description of each request. According to OFAC, once TFI has considered all of its component submissions, TFI submits its budget request to the Assistant Secretary for Management, who considers it as part of Treasury’s larger budget request. OFAC also stated that it has also used quarterly meetings and discussions as part of Treasury’s quarterly performance reviews to review resource needs and challenges. In addition to undertaking reviews as part of the budget process, OFAC launched a workforce planning effort in fiscal year 2019 and stated that it would be led by OFAC’s Office of Sanctions Support and Operations. As part of this effort, the Office of Sanctions Support and Operations stated that it plans to use Treasury’s department-wide workforce planning model and tools to gather information from OFAC’s component offices as a basis for, among other things, analyzing risks to OFAC’s mission, identifying resource gaps, and developing an action plan to address them. OFAC further stated that it plans to use its ongoing workforce planning model to assess the effectiveness of its current hiring authorities. In October 2019, OFAC officials stated that they expected to submit preliminary recommendations for each OFAC component to OFAC leadership by the end of December 2019. However, OFAC officials later stated that, because of the departure of the Assistant Director of Management Programs—the OFAC senior leader responsible for implementing the workforce planning initiative—on October 1, 2019, the planned date to submit preliminary recommendations to OFAC leadership was rescheduled to March 31, 2020. We analyzed the model and tools that OFAC is using for its ongoing resource analysis, to determine whether the process they set out would address five principles for strategic workforce planning that we had previously identified. We concluded that, if it were implemented as OFAC documents describe, the process would satisfy these principles. For example, the process calls for involving management and employees during its development and implementation and calls on managers to consider critical skills and competencies in their workforce analysis. State SPI requests resources as part of its annual budget process. State does not request a separate budget for SPI but instead combines SPI with the Office of Threat Finance Countermeasures (TFC) in its annual budget request. According to SPI officials, State sends the combined request for TFC and SPI to the Office of Management and Budget (OMB) every year, although the resources obtained may not reflect SPI’s original request. For example, SPI officials stated that SPI requested a greater increase in authorized positions for fiscal year 2020 than it ultimately received. SPI officials described ways that they assess staff workloads and seek to add or adjust resources on a continual basis. According to SPI officials, they have worked to fill positions on a temporary basis in response to rising needs. For example, SPI was authorized to add three temporary positions to cover the additional workload from Iran and Venezuela sanctions in early 2019. According to SPI officials, in justifying the request for additional temporary positions, SPI noted a significant increase in officer workload during the reimposition of sanctions against Iran, as well as maximum-pressure campaigns against Iran and Venezuela and increased activity related to existing and new sanctions authorities. As of October 2019, State planned to convert the three positions to permanent positions. Similarly, SPI officials stated that SPI justified its request for an increase in positions for fiscal year 2020 by noting an increasing use of sanctions as part of U.S. maximum economic pressure campaigns across multiple regions. Agency approaches to workforce planning can vary depending on each agency’s particular needs and mission. For subunits such as SPI, using the budget process, identifying changing priorities, and responding flexibly to those changes can address workforce planning needs. SPI officials further stated that SPI expects to review its workforce needs and structure if new executive orders delegate additional sanctions authorities to the Secretary of State. Commerce BIS units such as FPD assess and communicate their resource needs as part of the annual budget formulation process, according to BIS officials. BIS officials described budget formulation at Commerce as a “bottom-up” process, with BIS units providing information that is folded into Commerce’s overall budget. During this process, BIS budget office staff meet with program staff, review budget guidance provided by OMB as well as BIS’s own guidance, and ask program officials to identify any new initiatives or any new requirements for resources. According to BIS officials, each program office prepares a summary description of the request and needed resources for approval by the Assistant and Deputy Assistant Secretary for that office, the Deputy Under Secretary, and ultimately the BIS Under Secretary. According to BIS officials, BIS’s budget office then requests additional information about the approved activities. BIS’s Budget Office in turn submits the materials to the Commerce Departmental Budget Officer, who takes into account any known OMB and congressional viewpoints and department priorities. According to BIS officials, because of competing priorities, BIS funding priorities are not always carried over into the department’s overall request. BIS officials noted that, absent additional resources, they have some flexibility to shift personnel within the bureau to address periods of increased sanctions-related demand. For smaller units such as FPD, using the budget process, identifying changing priorities, and responding flexibly to those changes can address their workforce planning needs. Commerce previously prepared a multiyear budget strategy that assessed workforce needs throughout BIS, including FPD. In 2016, a contractor that Commerce hired prepared a Five-Year Budget Strategy Plan, which included workforce planning and projections. As part of the assessment, the plan analyzed BIS license volume and estimated the amount of time that staff in the BIS Export Administration’s Office of Nonproliferation and Treaty Compliance (which includes FPD) spent on particular tasks, such as conducting license application reviews, making license determinations, and developing regulations related to sanctioned countries. The plan projected future BIS license volume, external factors that would affect BIS workload, and the future FTEs that BIS would need to perform its mission. The plan examined the workload projection and the effect of attrition and concluded that FPD would need 0.5 additional FTEs by 2020 and 1.25 additional FTEs by 2022. BIS officials stated that they initially used the budget strategy plan to help with budgeting. However, according to the officials, the plan and its assumptions quickly became obsolete and they did not use it in subsequent years. In addition, BIS officials stated that the plan did not recognize BIS’s ability to shift resources or request appropriations as needed. Treasury, State, and Commerce units that focus primarily on sanctions implementation have information that can measure changes in agency workload over time; however, agency officials cited challenges in using this information as accurate measures of workload for the purpose of informing resource needs. For example, counting the number of individual actions taken to implement sanctions (e.g., designations, licenses, or the imposition of a penalty) does not capture the actions’ varying complexity or the time spent on developing potential actions that are ultimately not taken. Agency officials noted that, in general, the drivers of their workloads are global events and U.S. foreign policy priorities that may lead to more or less sanctions activity. Table 4 shows (1) selected information that can be used to measure changes in agency workload over time and (2) the potential weaknesses of these measures. OFAC and State each prepare and submit reports in response to the requirements of the IEEPA and the NEA. Both OFAC and State report sanctions implementation actions in response to the requirements of the IEEPA. OFAC’s NEA-mandated reports generally include information on expenditures reported by Treasury and State and by any other agencies identified in the relevant executive order. However, according to State’s most recent NEA reports, no specific State expenditures were directly attributable to the exercise of authorities conferred by the declaration of a national emergency under the NEA during the reporting period. In previous reviews, we and Treasury’s Office of Inspector General have found weaknesses in the consistency and timeliness of OFAC reports mandated by the Kingpin Act and the TSRA, respectively. Both OFAC and State include information on actions taken to implement sanctions programs in response to the requirements of the IEEPA. OFAC’s reports on sanctions programs under the IEEPA include data on the number of designations and the type of entity designated, the number of licensing actions, and the number and value of blocked transactions for sanctions programs authorized by the IEEPA. State’s IEEPA-mandated report for a weapons of mass destruction sanctions program (Executive Order 12938), prepared by State’s Bureau of International Security and Nonproliferation (ISN), summarizes the actions State has taken to address nonproliferation through bilateral and multilateral channels, including actions taken against Russia, North Korea, Syria, and the reimposition of nuclear-related sanctions on entities in Iran. Both OFAC and State included the reports responding to IEEPA requirements as part of the same document submitted in response to the NEA report requirements. OFAC’s reports on sanctions programs under the NEA include a summary total of expenditures reported by various agencies to implement those programs, as well as a listing of the agencies whose expenditures are included in the reports. The reports state that the expenditures included are predominantly personnel wage and salary costs. OFAC contacts multiple agencies to compile estimates of total expenditures for its NEA reports. According to OFAC officials, OFAC contacts an agency about its expenditures if the relevant executive orders have delegated sanctions implementation authority to the agency or tasked it with certain duties. Using a standardized request message, OFAC asks such agencies to estimate their expenditures for the national emergency by, for example, estimating the hours spent by staff members on activities related to the emergency and multiplying that number by appropriate hourly compensation rates. OFAC stated that it always asks State to provide estimated expenditure information and contacts other agencies to seek their expenditures on a program-by-program basis. OFAC’s NEA reports include Treasury and other agencies. All 25 of the NEA reports from mid- to late 2018 that we reviewed included Treasury expenditures, which were in many cases limited to OFAC and the Treasury Office of General Counsel. All but one report included State expenditures. Three reports included Commerce expenditures, five included Department of Homeland Security expenditures, and 12 included Department of Justice expenditures. While the reports did not include other agencies’ expenditures, some of the reports explicitly acknowledged that they did not reflect certain operating costs incurred by the intelligence and law enforcement communities. State ISN’s May 2019 NEA-mandated report for Executive Order 12938 stated that there were no specific expenditures directly attributable to the exercise of authorities conferred by the declaration of a national emergency under the NEA during the 6-month reporting period. The prior two reports also stated that there were no specific expenditures directly attributable to the sanctions program. The reports included no other information about the program expenditures. In response to our requests, State officials provided additional information about the NEA reporting of expenditures. According to the officials, State reported no expenditures for implementation activities for Executive Order 12938 because those activities have been subsumed into expenditures for normal, daily work—similar to overhead expenses. Expenditures for the implementation activities are mixed with, and indivisible from, the ongoing programming activities of the relevant offices and agencies. State officials indicated that State would report an amount other than zero if funds were reprogrammed, additional staff were required, or staff engaged in activities in addition to daily, normal work to implement the executive order. State officials also told us that they consulted State’s Bureau of Arms Control, Verification and Compliance, regional bureaus, and offices in the Departments of Commerce, Defense, and Energy in preparing the report. However, State’s reports have not included any of these additional statements about the information that State considered in concluding there were no specific expenditures attributable to the sanctions program. Standards for Internal Control in the Federal Government states that management should externally communicate the necessary quality information to achieve the entity’s objectives so that external parties can help the entity achieve its objectives and address related risks. Because State’s reports do not include the additional information that State considered, Congress lacks complete information regarding sanctions implementation expenditures. We have previously found that agencies do not report expenditures in response to OFAC’s Kingpin Act data requests in a consistent fashion. The Kingpin Act mandates that the President prepare a classified report to the Permanent Select Committee on Intelligence of the House of Representatives and the Select Committee on Intelligence of the Senate by July 1 of each year that, among other things, includes the status of sanctions imposed under the Kingpin Act and the personnel and resources directed to the imposition of Kingpin sanctions. OFAC compiles and submits these reports. OFAC’s Kingpin reports include previous year and cumulative data on the number of asset-blocking actions and Kingpin designations. The reports also include Treasury, State, DOD, and Justice expenditures, which the reports indicate are mostly personnel salary costs. However, we recently found that the agencies did not use consistent methods, across agencies and time, in providing their expenditures to OFAC for Kingpin Act program activities. We recommended that the Secretary of the Treasury (1) ensure that OFAC provide its partner agencies more specific guidance regarding Kingpin Act–related expenditure data to improve the consistency of data submitted by these agencies and (2) disclose information about limitations in the consistency and reliability of the agency expenditure data in its annual reports to Congress. Treasury OFAC and Commerce BIS each submit reports to Congress mandated by the TSRA. Treasury’s Inspector General found that OFAC had not submitted its reports in a timely fashion and recommended OFAC take steps to improve the timeliness of its submissions. OFAC. OFAC’s TSRA-mandated reports include information about its determinations regarding applications for licenses as well as the time it spent processing the applications. In April 2018, Treasury’s Office of Inspector General found that OFAC had not issued these reports in a timely manner and recommended that OFAC provide guidance to ensure that future TSRA-mandated reports are timely. According to the Treasury Office of Inspector General, Treasury’s actions in response—bringing its submission of the TSRA-mandated reports up to date and revising its TSRA report procedures—satisfied the intent of the office’s recommendation, but the Inspector General would continue to follow up. However, OFAC’s submission of the TSRA- mandated reports has continued to lag. OFAC released the TSRA- mandated reports for the second, third, and fourth quarters of fiscal year 2018 (i.e., January through September 2018) in November 2019; released the report for the first quarter of fiscal year 2019 in December 2019; and released the report for the second quarter of fiscal year 2019 in February 2020. OFAC’s most recent biennial report, for October 2014 through September 2016, was issued in August 2019. BIS. BIS’s TSRA-mandated reports include information about the licensing actions taken by BIS in relation to exports of agricultural commodities to Cuba, as well as processing times for those actions. BIS submitted its most recent report on January 17, 2020, covering the period from October 1 to December 31, 2019. BIS’s most recent biennial report, for October 2016 through September 2018, was issued in November 2018. The United States has increasingly relied on sanctions as a means to achieve important foreign policy goals. Implementing these sanctions involves multiple government agencies, some of which have multiple units with roles in sanctions implementation. Key agencies that implement sanctions have generally received steady or growing resources in recent years, but Treasury and State have staffing gaps and face challenges in securing the staff needed to fill their authorized positions. Treasury OFAC has an ongoing effort to assess its workforce needs, and Treasury, State, and Commerce all assess workforce needs through the budget process. The IEEPA and NEA each include requirements for reports to Congress that Congress can use to review the activities and expenditures that have been used for implementing these sanctions. However, State’s reports for Executive Order 12938 have not explained the information that State considered in reporting no expenditures. As a result, Congress does not have complete information about the data that State considers in calculating its sanctions implementation resources, which Congress could use to inform its review of agency resource requests. The Secretary of State should direct the Assistant Secretary for International Security and Nonproliferation to include additional information about the expenditures it considers in its NEA-mandated reporting for Executive Order 12938. We provided a draft of this report to the Departments of Commerce, Defense, Energy, Homeland Security, Justice, State, and the Treasury, as well as the Commodity Futures Trading Commission, Federal Deposit Insurance Corporation, Federal Reserve System, Internal Revenue Service, National Credit Union Administration, Office of the Comptroller of the Currency, and Securities and Exchange Commission for review and comment. State provided official comments, which are reproduced in appendix IV. State concurred with our recommendation and indicated that it will provide additional clarity on its procedures in future NEA-mandated reporting for Executive Order 12938. The Departments of Commerce, Homeland Security, Justice, State, and the Treasury, as well as the Internal Revenue Service, Office of the Comptroller of the Currency, and Securities and Exchange Commission also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and to the Secretaries of Commerce, Defense, Energy, Homeland Security, Justice, State, and the Treasury, as well as the Chairman and Chief Executive of the Commodity Futures Trading Commission, Chairman of the Federal Deposit Insurance Corporation, Chair of the Board of Governors of the Federal Reserve System, Commissioner of the Internal Revenue Service, Chairman of the National Credit Union Administration, the Comptroller of the Currency, and the Chairman of the Securities and Exchange Commission. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8612, or GianopoulosK@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Our objectives were to examine (1) agencies’ roles in sanctions implementation, (2) the resources available to agency units that focus primarily on sanctions implementation, (3) the extent to which agency units that primarily focus on sanctions implementation have assessed their resource needs, and (4) agencies’ reporting to Congress on sanctions implementation expenses and activities. To examine agencies’ roles in sanctions implementation, we identified agencies involved in sanctions implementation by reviewing sanctions authorities, including statutes and executive orders, and agency documents and websites and interviewing agency officials. We used these documents and interviews to summarize agencies’ principal roles in sanctions implementation, and we vetted this summary with the Departments of the Treasury (Treasury), State (State), and Commerce (Commerce), which we had identified through our initial interviews and review of background materials as having units that focus primarily on sanctions implementation. We then prepared a data collection instrument to obtain information on sanctions implementation from agencies across the government. Using this instrument, we requested information about the specific actions these agencies performed for each of the roles we identified, the number of staff they devoted to sanctions implementation, and the estimated percentage of time these staff spent on sanctions implementation in fiscal year 2019. We also requested information about the sources and methods that agencies or agency units used to produce these estimates. We pretested the instrument with the Office of the Comptroller of the Currency and the Department of Homeland Security’s U.S. Customs and Border Protection and made changes based on the results of the pretest before sending the instrument to all agencies or agency units that we had identified as having a role in sanctions implementation. To estimate in full-time equivalents (FTE) the staff resources that agencies devoted to sanctions implementation, we multiplied agencies’ estimates of the number of staff devoted to sanctions implementation by the agencies’ estimates of the percentage of time those staff spent on sanctions-related duties. To examine the resources available to agency units that focus primarily on sanctions implementation, we reviewed congressional budget justifications and used a data collection instrument to obtain information on (1) funding for units that focused primarily on sanctions implementation at Treasury, State, and Commerce in fiscal years 2014 through 2019 and (2) personnel in these units as of the beginning of fiscal years 2014 through 2020. We compared the information that agencies provided with data in their congressional budget justifications and determined that these data were sufficiently reliable for reporting on trends in funding, authorized FTEs, and filled positions at these agency units. We then examined challenges associated with hiring for, and filling, positions at these agency units by interviewing agency officials and reviewing agencies’ responses to our written questions. To examine the extent to which agency units that primarily focus on sanctions implementation have assessed their resource needs, we interviewed agency officials and reviewed their written responses to our questions about their budget development processes and any relevant workforce analyses and plans they had prepared. We reviewed documentation of Treasury’s ongoing workforce planning process against criteria for strategic workforce planning that we had previously identified, to assess whether the process, if completed according to plan, would address principles of strategic workforce planning that we had previously identified. We reviewed agency performance reports and annual reports and interviewed agency officials representing Treasury, State, and Commerce units that focus primarily on sanctions implementation, to identify any additional information the agencies had that could measure changes in agency workload over time. We then reviewed that information and interviewed agency officials to assess how accurately the measures reflected each agency’s sanctions workload. To examine agency reporting to Congress on sanctions implementation expenses and activities, we reviewed background information on sanctions implementation to identify mandated reports that included information on sanctions expenses and activities. We confirmed our list of the mandated reports that included sanctions expenses and activities with Treasury’s Office of Foreign Assets Control. We also reviewed sanctions legislation such as the International Emergency Economic Powers Act, the National Emergencies Act, the Foreign Narcotics Kingpin Designation Act and the Trade Sanctions Reform and Export Enhancement Act of 2000 to identify the specific requirements for those mandated reports on agency expenses and activities. We then requested from agency officials copies of the agencies’ most recently submitted mandated reports as of January 2019 and analyzed the agencies and types of expenses the reports identified. We requested information from agency officials and reviewed supporting documentation in order to describe how agencies estimated their expenses for sanctions implementation. We conducted this performance audit from October 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. To determine agencies’ roles in sanctions implementation, we sent a data collection instrument to all agency units that we had identified as having a role in sanctions implementation, requesting information on the specific actions the agency units perform for each role. Tables 5 through 12 summarize the information provided in the agency units’ responses to the data collection instrument. We identified units of 13 agencies that have a role in sanctions implementation, and we requested that each unit report the number of personnel with sanctions-related duties and the estimated percentage of time these personnel spent on such duties in fiscal year 2019. The agency units used various methods to generate their estimates. Several of the units were unable to estimate numbers of personnel with sanctions- related duties or the percentage of time these personnel spent on sanctions-related duties. In many cases, agency units were unable to disaggregate the relatively minimal resources devoted to sanctions implementation from the resources for wider duties related to their mission. The following provides information about each agency or agency unit. Department of State (State). All nine units that State identified as having a role in sanctions implementation were able to estimate the number of personnel with sanctions implementation duties in fiscal year 2019. The units used sources such as position descriptions and management surveys of staff to generate the estimates. Department of the Treasury (Treasury). Of the seven Treasury units from which we received information, five were able to estimate the number of personnel with sanctions implementation duties in fiscal year 2019. Officials of the sixth unit stated that they could not provide such an estimate. The seventh unit, the Office of Intelligence and Analysis of the Office of Terrorism and Financial Intelligence (TFI), provided an estimate of the percentage of time its analytic staff devoted to sanctions but, because of sensitivity concerns, did not provide estimates of the number of personnel with sanctions implementation duties. Department of Commerce (Commerce). Of the six Commerce units from which we received information, five were able to estimate the number of personnel with sanctions implementation duties in fiscal year 2019. However, Export Enforcement, a much larger BIS unit with over 170 employees, was not able to disaggregate the time its personnel spent on sanctions implementation from its broader export control enforcement activities. According to Export Enforcement officials, its investigative management system does not record whether its activities respond to potential violations of Office of Foreign Assets Control (OFAC) sanctions or violations of the Export Administration Regulations. Many of the cases that the office investigates include potential violations of both sanctions and the regulations. Department of Defense. The Department of Defense’s Office of the Under Secretary of Defense for Policy, which includes the Defense Technology Security Administration, was able to estimate the number of personnel with sanctions implementation duties. To generate the estimates, the office used sources including position descriptions and management judgement of time spent by individual action officers on sanctions. Department of Energy. The Department of Energy’s National Nuclear Security Administration relied on management judgment to estimate the number of personnel with sanctions duties. Department of Homeland Security. At the Department of Homeland Security, units in U.S. Immigration and Customs Enforcement were able to estimate the number of personnel with sanctions implementation duties in fiscal year 2019 by analyzing their investigative case management database. However, other department units were unable to provide such estimates. For example, Coast Guard officials reported that it would be difficult to estimate the number of personnel with sanctions implementation duties because these personnel are located throughout the United States and the world and do not record the time they spend on sanctions. Department of Justice. At the Department of Justice, the National Security Division and most sections of the Criminal Division were able to estimate the number of personnel with sanctions implementation duties in fiscal year 2019. However, other department units were unable to provide such estimates. For example, Drug Enforcement Administration (DEA) officials stated that it is difficult to quantify the time that DEA’s special agents dedicate specifically to sanctions. According to agency officials, investigations and operations to secure evidence for indictments can also be used to support sanctions designations. As a result, according to DEA, the agents spend minimal time on sanctions implementation that they would not have spent on their work in any case. The Federal Bureau of Investigation noted the same justification for why the bureau was unable to provide estimates of the number of personnel with sanctions-related duties. Financial regulatory agencies. The six financial regulatory agencies identified as having a role in sanctions implementation were unable to estimate numbers of personnel with sanctions implementation duties. Financial regulators were generally unable to disaggregate the time that personnel spent on OFAC compliance examinations because these are often performed concurrently with broader Bank Secrecy Act/Anti–Money Laundering examinations. See table 13 for additional information about each agency unit. In addition to the contact named above, Drew Lindsey (Assistant Director), Michael Simon (Analyst-in-Charge), Neil Doherty, Justin Fisher, Reid Lowe, Grace Lui, Christina Pineda, Julia Robertson, and Paul Sturm made key contributions to this report.", "summary": "The United States has implemented dozens of sanctions programs to counteract activities that threaten U.S. national interests. Sanctions may place restrictions on entire countries, sectors of countries' economies, or specific corporations or individuals. Examples of restrictions include limiting access to the U.S. financial system, freezing assets under U.S. jurisdiction, and restricting trade. The United States has implemented an increasing number of sanctions in recent years, including sanctions on countries that conduct a significant amount of international trade, such as Russia, Venezuela, and Iran. GAO was asked to examine the resources U.S. agencies have devoted to sanctions implementation. This report examines (1) agencies' roles in sanctions implementation, (2) resources available to agency units that focus primarily on sanctions implementation, (3) the extent to which agency units that focus primarily on sanctions implementation have assessed their resource needs, and (4) agencies' reporting to Congress on sanctions implementation expenses and activities. GAO gathered data from 13 agencies and their sub-units to identify their roles and the personnel they used for sanctions implementation. GAO also reviewed agency reporting, planning, and budget documents and interviewed agency officials. Agencies may have one or more roles in sanctions implementation—for example, developing policy and investigating, enforcing, and prosecuting violations. The Departments of the Treasury, State, and Commerce each have a unit focused primarily on sanctions—Treasury's Office of Foreign Assets Control (OFAC), State's Office of Economic Sanctions Policy and Implementation (SPI), and Commerce's Bureau of Industry and Security's (BIS) Foreign Policy Division (FPD). GAO identified 10 other agencies with roles in sanctions implementation. OFAC, SPI, and FPD generally received steady or growing resources in recent years, but OFAC and SPI face hiring challenges. In fiscal years 2014 to 2019, OFAC received a 58 percent budget increase and additional hiring authority, but vacancies ranged from 6 to 26 percent of its authorized full time equivalents (FTEs). OFAC attributed its hiring challenges to competition from other agencies and the private sector and the time needed for security clearances. State SPI received authority to hire six additional FTEs in fiscal year 2020, for a total of 21, but more than half of its authorized positions were vacant at the start of the fiscal year. FPD lacks funding to fill one of its 10 authorized positions. OFAC, SPI, and FPD all consider resource needs as part of annual budget processes, and OFAC has an ongoing process to assess its workforce needs. OFAC began its workforce planning process in fiscal year 2019 and expects to make preliminary recommendations in March 2020. According to SPI officials, SPI cited the increasing use of sanctions across multiple regions in justifying its request for additional fiscal year 2020 positions. BIS prepared a 2016 plan that assessed its workforce, including FPD, but stated that it no longer uses the plan. Agencies provide information on selected sanctions expenses and activities in mandated reports. Treasury's reports on 25 sanctions programs include expenses for Treasury, State, and other agencies if relevant executive orders identify them. State reported activities for a weapons of mass destruction sanctions program but also reported no specific expenditures for the program. State reviewed program information to prepare the reports, but the reports do not describe what it considered, limiting information available to Congress. GAO recommends that State include additional information about the expenditures it considers in its reporting for the Proliferation of Weapons of Mass Destruction sanctions program. State concurred with the recommendation.", "document_type": "gao"}
{"report": "To help manage its multi-billion dollar acquisition investments, DHS has established policies and processes for acquisition management, requirements development, test and evaluation, and resource allocation. The department uses these policies and processes to deliver systems that are intended to close critical capability gaps, helping enable DHS to execute its missions and achieve its goals. DHS policies and processes for managing its major acquisition programs are primarily set forth in its Acquisition Management Directive 102-01 and Acquisition Management Instruction 102-01-001. DHS issued the initial version of this directive in November 2008 in an effort to establish an acquisition management system that effectively provides required capability to operators in support of the department’s missions. DHS has issued multiple updates to its acquisition management directive and instruction, in part to be responsive to GAO’s recommendations. DHS issued the current version of the directive in February 2019 and the current version of the instruction in May 2019; however, we did not assess programs against these updates because the programs in our review established initial baselines prior to the approval of the directive and instruction. DHS’s Under Secretary for Management is currently designated as the department’s Chief Acquisition Officer and, as such, is responsible for managing the implementation of the department’s acquisition policies. DHS’s Under Secretary for Management serves as the acquisition decision authority for the department’s largest acquisition programs, those with LCCEs of $1 billion or greater. Component Acquisition Executives—the most senior acquisition management officials within each of DHS’s components—may be delegated acquisition decision authority for programs with cost estimates between $300 million and less than $1 billion. Table 1 identifies how DHS has categorized the 29 major acquisition programs we reviewed in this report, and table 8 in appendix II specifically identifies the programs within each level. DHS acquisition management policy establishes that a major acquisition program’s decision authority shall review the program at a series of predetermined acquisition decision events (ADE) to assess whether the major program is ready to proceed through the acquisition life cycle phases. Depending on the program, these events can occur within months of each other or be spread over several years. Figure 1 depicts the acquisition life cycle in the March 2016 version of DHS acquisition management policy. An important aspect of an ADE is the decision authority’s review and approval of key acquisition documents. See table 2 for a description of the type of key acquisition documents identified in the March 2016 acquisition management directive and instruction that required department-level approval before a program moves to the next acquisition phase. DHS acquisition management policy establishes that the APB is the agreement between program, component, and department-level officials establishing how systems being acquired will perform, when they will be delivered, and what they will cost. Specifically, the APB establishes a program’s schedule, costs, and key performance parameters. DHS defines key performance parameters as a program’s most important and non-negotiable requirements that a system must meet to fulfill its fundamental purpose. For example, a key performance parameter for an aircraft may be airspeed and a key performance parameter for a surveillance system may be detection range. The APB establishes objective (target) and threshold (maximum acceptable for cost, latest acceptable for schedule, and minimum acceptable for performance) baselines. According to DHS policy, if a program fails to meet any schedule, cost, or performance threshold approved in the APB, it is considered to be in breach. Programs in breach are required to notify their acquisition decision authority and develop a remediation plan that outlines a timeframe for the program to return to its APB parameters, re-baseline—that is, establish new schedule, cost, or performance goals—or have a DHS-led program review that results in recommendations for a revised baseline. In addition to the acquisition decision authority, other bodies and senior officials support DHS’s acquisition management function: The Acquisition Review Board reviews major acquisition programs for proper management, oversight, accountability, and alignment with the department’s strategic functions at ADEs and other meetings as needed. The board is chaired by the acquisition decision authority or a designee and consists of individuals who manage DHS’s mission objectives, resources, and contracts. The Office of Program Accountability and Risk Management (PARM) is responsible for DHS’s overall acquisition governance process, supports the Acquisition Review Board, and reports directly to the Under Secretary for Management. PARM develops and updates program management policies and practices, reviews major programs, provides guidance for workforce planning activities, provides support to program managers, and collects program performance data. Components, such as U.S. Customs and Border Protection, the Transportation Security Administration, and the U.S. Coast Guard sponsor specific acquisition programs. The head of each component is responsible for oversight of major acquisition programs once the programs complete delivery of all planned capabilities to end users. Component Acquisition Executives within the components are responsible for overseeing the execution of their respective portfolios. Program management offices, also within the components, are responsible for planning and executing DHS’s individual programs. They are expected to do so within the cost, schedule, and performance parameters established in their APBs. If they cannot do so, programs are considered to be in breach and must take specific steps, as noted above. Figure 2 depicts the relationship between acquisition managers at the department, component, and program level. In 2016, we found that DHS had not effectively implemented or adhered to its review process for major acquisitions and recommended that DHS reinstate the Joint Requirements Council (JRC) to review and approve acquisition requirements and assess potential duplication of effort across the department. DHS established a JRC to develop and lead a component-driven joint requirements process for the department. In March 2016, DHS revised its policy instruction to reflect the addition of the JRC as an acquisition oversight body. Among other responsibilities, the JRC is to provide requirements-related advice and validate key acquisition documentation to prioritize requirements and inform DHS investment decisions among its components. The JRC chair is a member of the Acquisition Review Board and advises the board on capability gaps, needs, and requirements at key milestones in the acquisition life cycle. In March 2019, we reported that the JRC could better fulfill its mission by identifying overlapping or common requirements, and by making recommendations to senior leadership to inform budget decisions and help ensure that DHS uses its finite investment resources wisely. We will continue to monitor the JRC’s efforts through GAO’s high risk work. In May 2009, DHS established policies that describe processes for testing the capabilities delivered by the department’s major acquisition programs. The primary purpose of test and evaluation is to provide timely, accurate information to managers, decision makers, and other stakeholders to reduce programmatic, financial, schedule, and performance risks. We provide an overview of each of the 29 programs’ test activities in the individual program assessments presented in appendix I. DHS testing policy assigns specific responsibilities to particular individuals and entities throughout the department: Program managers have overall responsibility for planning and executing their programs’ testing strategies, including scheduling and funding test activities and delivering systems for testing. They are also responsible for controlling developmental testing, which is used to assist in the development and maturation of products, manufacturing, or support processes. Developmental testing includes engineering- type tests used to verify that design risks are minimized, substantiate achievement of contract technical performance, and certify readiness for operational testing. Operational test agents are responsible for planning, conducting, and reporting on operational test and evaluation to identify whether a system can meet its key performance parameters and provide an evaluation of the operational effectiveness, suitability, and cybersecurity of a system in a realistic environment. Operational effectiveness refers to the overall ability of a system to provide a desired capability when used by representative personnel. Operational suitability refers to the degree to which a system can be placed into field use and sustained satisfactorily. Operational cybersecurity refers to the degree to which a system is able to accomplish its mission in a cyber-contested environment. The operational test agents may be organic to the component, another government agency, or a contractor, but must be independent of the developer to present credible, objective, and unbiased conclusions. The Director, Office of Test and Evaluation is responsible for approving major acquisition programs’ operational test agent and test and evaluation master plans, among other things. A program’s test and evaluation master plan must describe the developmental and operational testing needed to determine technical performance and operational effectiveness, suitability, and cybersecurity. As appropriate, the Director is also responsible for observing operational tests, reviewing operational test agents’ reports, and assessing the reports. Prior to a program’s ADE 3, the Director provides the program’s acquisition decision authority a letter of assessment that includes an appraisal of the program’s operational test, a concurrence or non-concurrence with the operational test agent’s evaluation, and any further independent analysis. As an acquisition program proceeds through its life cycle, the testing emphasis moves gradually from developmental testing to operational testing. See figure 3. DHS has established a planning, programming, budgeting, and execution process to allocate resources to acquisition programs and other entities throughout the department. DHS uses this process to produce the department’s annual budget request and multi-year funding plans presented in the FYHSP report, a database that contains, among other things, 5-year funding plans for DHS’s major acquisition programs. According to DHS guidance, the 5-year plans should allow the department to achieve its goals more efficiently than an incremental approach based on 1-year plans. DHS guidance also states that the FYHSP articulates how the department will achieve its strategic goals within fiscal constraints. At the outset of the annual resource allocation process, the department’s Office of Strategy, Policy, and Plans and Office of the Chief Financial Officer provide planning and fiscal guidance, respectively, to the department’s components. In accordance with this guidance, the components should submit 5-year funding plans to the Chief Financial Officer. These plans are subsequently reviewed by DHS’s senior leaders, including the DHS Secretary and Deputy Secretary. DHS’s senior leaders are expected to modify the plans in accordance with their priorities and assessments, and they document their decisions in formal resource allocation decision memorandums. DHS submits the revised funding plans to the Office of Management and Budget, which uses them to inform the President’s annual budget request—a document sent to Congress requesting new budget authority for federal programs, among other things. In some cases, the funding appropriated to certain accounts in a given fiscal year remains available for obligation and can be carried over to subsequent fiscal years. Figure 4 depicts DHS’s annual resource allocation process. Federal law requires DHS to submit an annual FYHSP report to Congress at or about the same time as the President’s budget request. Two offices within DHS’s Office of the Chief Financial Officer support the annual resource allocation process: The Office of Program Analysis and Evaluation (PA&E) is responsible for establishing policies for the annual resource allocation process and overseeing the development of the FYHSP. In this role, PA&E develops the Chief Financial Officer’s planning and fiscal guidance, reviews the components’ 5-year funding plans, advises DHS’s senior leaders on resource allocation issues, maintains the FYHSP database, and submits the annual FYHSP report to Congress. The Cost Analysis Division is responsible for reviewing, analyzing, and evaluating acquisition programs’ LCCEs to ensure the cost of DHS programs are presented accurately and completely, in support of resource requests. This division also supports affordability assessments of the department’s budget, in coordination with PA&E, and develops independent cost analyses for major acquisition programs and independent cost estimates upon request by DHS’s Under Secretary for Management or Chief Financial Officer. Of the 27 programs we assessed with approved APBs, 25 are on track to meet their current schedule and cost goals as of August 2019. Of these 25 programs, 11 programs revised their schedule and cost goals in response to a prior breach of their APBs or to incorporate program changes. Of the 27 programs, two programs breached their schedule or cost goals between January 2018 and August 2019, and as of August 2019 had not yet re-baselined. This shows improvement from our prior review where seven programs were in breach. In addition, some programs, although currently on track to meet their goals, are nonetheless facing risks of breaching schedule or cost goals, or have plans to revise their baseline in the future. Further, as a result of the fiscal year 2019 partial government shutdown, five programs received approval for schedule adjustments, and other programs reported difficulty obligating funds before the end of the fiscal year. Finally, our analysis showed that seven programs are projected to experience an acquisition funding gap in fiscal year 2020, but, according to program officials, these gaps will be mitigated. We also reviewed two programs that were early in the acquisition process and planned to establish department-approved schedule and cost goals during our review. However, these programs were delayed in getting department approval for their initial APBs for various reasons; therefore, we excluded them from our assessment of whether programs were on track to meet schedule and cost goals. We plan to assess these programs in our future reviews; however, we provide more details on these two programs in the individual assessments in appendix I. Table 3 summarizes our findings regarding the status of major acquisition programs meeting their schedule and cost goals, and we present more detailed information after the table. We found that 25 of 27 programs we reviewed with department-approved APBs were on track to meet their current baseline schedule and cost goals as of August 2019. Of these, 11 programs met schedule and cost goals established prior to December 2017. Six of these programs are in the process of revising their baselines or plan to revise their baselines in the near future to account for program changes or to add capabilities. For example, the U.S. Coast Guard’s Fast Response Cutter and National Security Cutter programs plan to revise their baselines because they received additional funding to procure more cutters than reflected in their current baselines. Program officials said these programs are planning to update their APBs in fiscal year 2020 to reflect these changes. In addition, as shown in table 3, five of the 25 programs that met schedule and cost goals had only recently established initial APBs (between January 2018 and August 2019). Three of these five—Customs and Border Protection’s Biometric Entry-Exit program and Border Wall System Program, and the U.S. Coast Guard’s Polar Security Cutter—are new Level 1 major acquisition programs and as of August 2019 their combined life cycle costs were approximately $15 billion. In addition, DHS recently approved baselines for two Transportation Security Administration programs—Advanced Technology and Credential Authentication Technology. These programs were previously projects under the Passenger Screening Program, but according to Transportation Security Administration officials, transitioned into standalone programs to better align program office staffing to capabilities and focus on mitigating capability gaps, among other things. Eleven of the 25 programs that we found to be on track to meet current schedule and cost goals revised schedule and cost goals between January 2018 and August 2019. DHS leadership approved revised baselines for these programs for two primary reasons: to remove the program from breach status or to account for program changes, or both. Five of the 11 programs that revised their baselines did so in response to a breach of their cost or schedule goals and were subsequently removed from breach status. See table 4. DHS leadership approved revised baselines for these five programs following various actions by the program offices such as: Customs and Border Protection’s Automated Commercial Environment breached its cost and schedule goals in April 2017, which Customs and Border Protection officials attribute to an underestimation of the level of effort needed to complete development. The program revised its approach to developing remaining functionality by removing some capability from the program’s baseline and delaying development until funding is provided. As shown in table 4, the full operational capability date was delayed. The program’s total life-cycle cost increase is primarily attributed to a change in how threshold cost goals were calculated. Customs and Border Protection’s Medium Lift Helicopter re- baselined following a schedule breach of its ADE 3, among other things. As part of the re-baselining efforts, the program revised its cost goals to remove personnel costs and update the aircraft operational hours, among other things, which then resulted in a cost decrease of $515 million. Officials reported that the effect of the breach on the program’s schedule was minimal because the program was able to make adjustments to its testing schedule to assess multiple aircraft concurrently. DHS Management Directorate’s Homeland Advanced Recognition Technology re-baselined following multiple delays in awarding contracts and issues stemming from a subsequent bid protest. The re-baseline included a cost goal decrease resulting from an enhanced solution for biometric data storage. U.S. Coast Guard’s H-65 Conversion - Sustainment Program re- baselined to address delays which USCG officials primarily attributed to underestimating the technical effort necessary to meet requirements. As part of the re-baseline, the program also added a service life extension program to extend aircraft service life by replacing obsolete components. The program’s total life-cycle cost threshold decreased by approximately $200 million from its prior APB. Coast Guard officials attribute the decrease to the program’s ability to reduce labor costs, among other things, by synchronizing the service life extension program with other aircraft upgrades. U.S. Citizenship and Immigration Services’ Transformation program re-baselined in June 2018—lifting a strategic pause that limited new program development for 18 months. The program’s revised APB reflects a re-organization of the Transformation program as well as a new development strategy. The program breached its schedule in September 2016 when it failed to upgrade U.S. Citizenship and Immigration Services’ application processing information system to include applications for naturalization. In addition, between January 2018 and August 2019, DHS leadership approved revisions to six programs’ baselines that were not prompted by a breach. These programs either planned to revise their baselines to incorporate changes in technology, among other things, or to make changes to their scope. Customs and Border Protection’s Biometric Entry-Exit program revised its schedule goals in March 2019—after establishing an initial baseline in May 2018—to remove ADE 2C, the decision event when low-rate initial production is typically approved. Customs and Border Protection’s Border Wall System Program revised its baseline in August 2018 to replace sections of the border wall system in the San Diego sector. In addition, in May 2019 the program received approval for an additional baseline to extend the border wall system in the Rio Grande Valley sector. Customs and Border Protection’s Multi-role Enforcement Aircraft revised its baseline to increase the program’s quantity from 16 to 29 aircraft. The 16 aircraft from the prior APB provided maritime interdiction capabilities. The additional 13 aircraft are for air interdiction capabilities. Cybersecurity and Infrastructure Security Agency’s National Cybersecurity Protection System Program revised its baseline in January 2018 to inform ADEs for the program’s information sharing and intrusion-prevention capabilities and to account for schedule and cost changes after bid protests. However, the program updated its APB again in October 2018 to address an error found in the LCCE. Specifically, the LCCE that provided the basis for the program’s APB cost goals did not accurately account for the program’s sunk costs. In addition, the program added an additional 2 years of costs to its LCCE and revised its approach to estimating threshold costs. Once revised, the program’s total life-cycle cost threshold increased by more than $1.7 billion (41 percent) from the program’s January 2018 APB. The program’s full operational capability date was extended by two years to March 2021. Cybersecurity and Infrastructure Security Agency’s Next Generation Networks Priority Services revised its baseline in April 2018 to add capability to provide priority access for landline telephone calls to select government officials during emergencies. As a result, the program’s full operational capability date was extended by 3 years—to December 2025—and total acquisition costs increased by $68 million (10 percent). Transportation Security Administration’s Technology Infrastructure Modernization program revised its baseline in July 2019 to de-scope the program and narrow the definition of full operational capability. DHS leadership reported that by the time the program had delivered functions needed to meet the needs of end users, the Transportation Security Administration had updated and improved its legacy systems. As a result, costs decreased by $15 million (1 percent) and the program achieved full operational capability 3 years earlier than previously planned. Between January 2018 and August 2019, two programs breached their schedule or cost goals—down from seven programs in our previous assessment. As of August 2019, neither of these programs had revised their baselines. Customs and Border Protection’s Integrated Fixed Towers program declared a schedule breach of the program’s baseline in February 2019 as a result of delays in negotiations with the Tohono O’odham Nation—a sovereign Native American Nation—regarding access to tribal lands to construct towers and deploy systems. Customs and Border Protection subsequently reached an agreement with the Nation in March 2019. As of September 2019, the program was in the process of revising its APB to adjust deployments within the Nation’s land. Program officials anticipate the program’s full operational capability date will slip from September 2020 to March 2021 as a result of these actions. Transportation Security Administration’s Electronic Baggage Screening Program updated its LCCE in August 2019 which exceeds its baseline operations and maintenance (O&M) cost threshold. Transportation Security Administration officials attribute the program’s cost breach to an increase in maintenance costs related to sustaining screening technologies longer than initially planned. As of September 2019, the program’s revised APB, which TSA officials said will address the O&M cost increase, had not yet been approved. In addition, some of the programs on track as of August 2019 are facing risks that might lead to schedule slips or cost growth in the future. For example, U.S. Coast Guard’s Offshore Patrol Cutter may experience cost increases and schedule slips in the future. Specifically, the program’s shipbuilder reported damages from Hurricane Michael in October 2018 that have resulted in a long-term degradation of its ability to produce the Offshore Patrol Cutters at the previously estimated cost and schedule. As of August 2019, the Coast Guard was still assessing the shipbuilder’s report on the damage sustained and the potential effect on the Offshore Patrol Cutter program. U.S. Coast Guard’s Polar Security Cutter met established cost and schedule milestones between January 2018 and August 2019, but program officials stated that they anticipate a schedule slip because delivery of the lead ship in the awarded contract is two months after the program’s APB threshold date. We previously found that the program is at risk of experiencing future schedule delays and cost growth. The program’s schedule is driven by the need to address a potential gap in icebreaking capabilities once the Coast Guard’s only operational heavy polar icebreaker reaches the end of its service life as early as 2023. As a result, planned delivery dates are not informed by a realistic assessment of shipbuilding activities. We also found that the program is at risk of costing more than estimated because its LCCE—while adhering to most cost estimating best practices—is not fully reliable as it did not quantify the range of possible costs over the entire life of the program. Customs and Border Protection’s Biometric Entry-Exit program plans to re-baseline and achieve ADE 3—which will authorize full-rate production—in September 2019. However, program officials stated that not all testing will be completed to inform the ADE 3. As a result, DHS leadership will not have data related to the Biometric Entry-Exit system’s resiliency to cyberattacks before making this decision. We provide more information in the individual program assessments in appendix I, and we will continue to monitor these programs in future assessments. Due to a lapse in appropriations for fiscal year 2019, the federal government partially shut down from December 22, 2018, to January 25, 2019. Most Level 1 and Level 2 acquisition program staff were furloughed during the partial government shutdown, which affected the execution of these programs. As a result, in March 2019, DHS’s Under Secretary for Management, in coordination with PARM, authorized Component Acquisition Executives to request up to a 3-month extension for any program schedule milestone date, and inform PARM of any proposed changes in writing. PARM officials stated that they developed this process to mitigate program schedule risks since the government shutdown was beyond the control of program officials. Five programs requested and received approval from DHS leadership to extend schedule milestones by 3 months. Of these, three programs reported that the 3month extension will allow the programs to stay on track to meet their adjusted milestones—Federal Emergency Management Agency’s Logistics Supply Chain Management System, Customs and Border Protection’s Biometric Entry-Exit, and U.S. Coast Guard’s Medium Range Surveillance Aircraft programs. However, Coast Guard officials stated that the Offshore Patrol Cutter program requested approval to extend the program’s ADE 2C milestone to enable Coast Guard officials time to assess the shipbuilder’s report on damage caused by Hurricane Michael before determining the next steps for the program. The Cybersecurity and Infrastructure Security Agency’s Continuous Diagnostics and Mitigation program received approval to extend two schedule milestones—initial operational capability for two segments of the program—because the program experienced delays as a result of the partial government shutdown. In addition, DHS leadership previously directed the program to conduct an ADE 2B for a new segment by March 2019. The ADE 2B has been delayed 9 months to December 2019 to allow the program additional time to complete required acquisition documentation to inform the ADE. Programs also reported experiencing other effects of the partial government shutdown. Specifically, officials from several programs identified challenges in obligating funds by the end of the fiscal year due to the truncated timeframe. For example, Transportation Security Administration’s Electronic Baggage Screening Program officials reported that as a result of the partial government shutdown, contract awards had been delayed. These officials explained that contracting obligation activities from the component were compressed into the last two quarters of fiscal year 2019 and the program had to compete for contracting officer resources within the limited timeframe. Based on the information presented in the 2020-2024 FYHSP report to Congress, DHS’s acquisition portfolio is not affordable over the next 5 years, meaning that the anticipated funding will not be adequate to support the programs. But our analysis found the reported acquisition funding gaps may be overstated when additional information is taken into account. For example, the fiscal year 2020-2024 FYHSP report contained acquisition affordability tables for 21 of the 27 programs we assessed that have approved APBs. Of the 21 programs included in the FYHSP report, 11 were projected to have an acquisition affordability gap in fiscal year 2020. However, some of the cost information used to develop these projections was outdated since the FYHSP report—which was issued in August 2019—relied on cost estimates developed in April 2018. Therefore, we updated the analysis using the programs’ current LCCEs based on the approved scope of the program, as of August 2019 (as presented in the individual assessments in appendix I). In addition, we discussed funding gaps with program officials to determine additional funding sources, such as fees collected, funding from previous fiscal years that remained available for obligation—known as carryover funding, funds provided by components, or funding received above what was originally requested. Based on our analysis, we found that seven programs may have acquisition funding gaps in fiscal year 2020 rather than the 11 identified in the FYHSP report. However, the affordability gap for all seven programs we identified may be overstated because program officials reported that these programs either had carryover funding, received funding above what was requested, or anticipate receiving funding from the component to mitigate the affordability gap, as shown in table 5. Further, officials from several programs in our review told us that the programs were projected to experience a funding gap that could cause future program execution challenges, such as cost growth, or that programs were taking steps to mitigate funding gaps. For example, Customs and Border Protection’s Biometric Entry-Exit program—which is primarily fee-funded—conducted an affordability analysis that showed projected fees had declined. To mitigate risks of a potential affordability gap, program officials stated the number of officers to conduct enforcement activities at airport departure gates was reduced and the program is working with the component to identify other sources of funding. In addition, DHS Management Directorate’s Homeland Advanced Recognition Technology program reported that the program will use carryover funding to address the program’s affordability gap in fiscal year 2020. However, the program will also need to defer development of some additional capabilities to 2021 to remain affordable. In addition, officials from Customs and Border Protection’s Border Wall System Program stated the program is mitigating future acquisition funding gaps, in part by not developing its baseline until after funding amounts are determined. According to officials, this was necessary to mitigate program risks due to uncertainty in funding; however, through DHS’s resource allocation process, the program has requested $5 billion each year from fiscal year 2020 to fiscal year 2024. We elaborate on programs’ affordability over the next 5 years in the individual program assessments in appendix I. Traceability, which DHS policy and acquisition best practices call for, helps ensure that program goals are aligned with program execution plans, and that a program’s various stakeholders have an accurate and consistent understanding of those plans and goals. We found that the cost and performance goals in the acquisition programs’ approved APBs generally traced to the estimated costs identified in LCCEs and key performance parameters identified in operational requirements documents. That is, information in the APB matched the document required to be used as the basis for the baselines. In contrast, the schedule goals in the approved APBs generally did not trace to the Integrated Master Schedule (IMS), as required by the DHS acquisition management instruction and as a best practice identified in GAO’s Schedule Assessment Guide. Similarly, we found the required basis for the cost and performance goals is consistently identified in DHS acquisition management policy and guidance, whereas the basis for the schedule goals is not. We found that cost and performance goals in approved APBs generally traced to estimated costs in LCCEs and key performance parameters in operational requirements documents. However, schedule goals were generally not traceable to the IMSs, as required by DHS acquisition management instruction and as identified as a best practice in GAO’s Schedule Assessment Guide. Of the 27 programs we assessed with established baselines, 21 established or revised their APBs after DHS updated its acquisition management instruction in March 2016, which was the most current version of the guidance when we initiated our review. Table 6 shows the results of our analysis for the traceability of baselines to cost, schedule, and performance documents for those 21 programs. As shown in table 6, the APB goals traced to the key performance parameters in the operational requirements documents for all 21 programs that we reviewed. Generally, the APB goals traced to the costs in the LCCEs, though we found that three programs were not traceable. For example: The APB total life-cycle cost goals for Custom and Border Protection’s Tactical Communications Modernization program traced to the program’s LCCE, but the separate acquisitions and O&M costs were not traceable. The Transportation Security Administration’s Electronic Baggage Screening Program did not include sunk costs in the LCCE, and as a result the APB cost goals did not trace. In contrast, we could trace all schedule events and dates in the approved APBs to the programs’ IMS for only six of 21 programs. There was variation in how the programs’ APBs lacked traceability to the IMS. For example: The IMS for the Customs and Border Protection’s Border Wall System Program estimates the full operational capability dates to be between October 2021 and December 2021, whereas the approved APB includes an objective date of October 2022 and a threshold date of December 2022. The APB for the U.S. Citizenship and Immigration Services’ Transformation program does not identify a source for the schedule baseline. Program officials told us that they do not have an IMS and instead they use the schedule in the program’s release roadmap, a document that information technology programs use to communicate how they will iteratively deliver features. However, schedule events identified in the APB, such as full operational capability, were not identified in the release roadmap. Similarly, we found programs that developed an IMS but did not include all future APB milestones, such as Cybersecurity and Infrastructure Security Agency’s Continuous Diagnostics Mitigation and Transportation Security Administration’s Credential Authentication Technology. According to GAO’s Schedule Assessment Guide, schedules should be verified to ensure that they are vertically traceable—that is, verified to ensure the consistency of dates, status, and scope requirements between different levels of the schedule and management documents. Further, this guide states that a schedule baseline signifies a consensus of stakeholders on the required sequence of events, resources, and key dates. The IMS is more accurate when stakeholders agree on the underlying assumptions. These stakeholders would include, for example, program offices, end users, and component and DHS leadership. Further, DHS acquisition policy requires programs to obtain review and approval of LCCEs and operational requirements documents from various stakeholders within components and DHS headquarters. However, DHS acquisition policy states that approval of IMSs is based on DHS guidance and component policy and that program managers will provide the IMS to DHS in support of the acquisition review process. Officials from PARM and the Office of the Chief Financial Officer told us that the components vary in their capacity to develop schedules and assess schedule risks and there is a lack of expertise within the department to review program schedules. The lack of traceability between IMSs and schedule goals in the APB indicates that DHS does not have an appropriate oversight process in place to ensure APBs trace to schedule goals in the IMSs, in accordance with DHS policy and GAO’s best practices. Without this traceability, DHS cannot ensure that the understanding of program schedules among different stakeholders is consistent and accurate. As a result, DHS leadership may be approving program schedule goals that do not align with program execution plans. We found that LCCEs and operational requirements documents are consistently identified as the basis of cost and performance goals in DHS’s acquisition management policy and guidance. However, we also found that the documents do not consistently require that an IMS be used as the basis of schedule goals. Specifically, DHS’s acquisition management instruction and DHS’s Systems Engineering Life Cycle Guidebook—which outlines the technical framework for DHS’s acquisition management system—differ regarding the source of APB schedule milestone dates. Table 7 summarizes our findings on DHS’s acquisition policy and guidance related to developing APB cost, schedule, and performance goals. DHS’s acquisition management instruction states that the APB should trace to the IMS, which is consistent with GAO’s Schedule Assessment Guide. This instruction differs from the guidance in the Systems Engineering Life Cycle Guidebook, which in contrast, directs programs to use the APB as an input when developing the IMS. PARM officials said they were unaware of the inconsistency and confirmed that the IMS should provide the basis of APB schedule goals, as identified in DHS’s acquisition management instruction. PARM officials also acknowledged that the information related to schedule development should be consistent across all of DHS’s policies, instructions, and guidebooks. Conflicting agency-wide policy and guidance can lead to a lack of clarity and consistency on how programs develop their schedules. In addition, the lack of a well-developed schedule can contribute to poor acquisition outcomes, such as increased costs and delayed delivery of capabilities needed by end users. As previously noted, DHS’s 2019 update to its acquisition management directive and associated instruction addressed a GAO recommendation related to better defining requirements before establishing acquisition program baselines. PARM officials told us they plan to update the Systems Engineering Life Cycle Guidebook by the end of calendar year 2019 to account for the revisions in the acquisition management directive and associated instruction. At that time, they also plan to correct the inconsistency related to the documents used to develop APB schedule goals. Since we began reviewing DHS’s portfolio of major acquisitions in 2015, the department has strengthened implementation of its policies to improve acquisition oversight. These efforts have begun to yield better results as the performance of DHS’s major acquisition portfolio has improved compared to our last review. As DHS major acquisition policy has evolved over time, the department has put in place oversight and approval processes that help ensure cost and performance goals are clear, consistent, and trace to key acquisition documents serving as the basis for those goals. However, opportunities remain for DHS to provide better oversight of major acquisition programs’ schedule goals, as we found that these goals generally did not trace to the integrated master schedules per DHS policy. When schedule goals are not traceable, DHS decision makers cannot be sure that the schedule presented is consistent and accurate. Until DHS develops an oversight process to ensure schedules are developed and updated appropriately, the department cannot ensure that its most expensive acquisition programs are able to deliver capabilities needed by end users when promised. In addition, we found inconsistencies within DHS’s major acquisition policy and system engineering guidance in identifying the basis of schedule goals. Without consistent schedule development guidance, DHS has no way of knowing that programs establish schedules in a consistent manner and in accordance with GAO’s scheduling best practices. We are making the following two recommendations to DHS. The Secretary of Homeland Security should ensure that the Undersecretary for Management develops an oversight process to confirm that programs’ schedule goals are developed and updated in accordance with GAO’s Schedule Assessment Guide, to include ensuring traceability between APB schedule goals and IMSs. (Recommendation 1) The Secretary of Homeland Security should ensure that the Undersecretary for Management revises the schedule development guidance in the Systems Engineering Life Cycle Guidebook to state clearly that an IMS should be used as the basis for APB schedule goals. (Recommendation 2) We provided a draft of this report to DHS for review and comment. DHS’s comments are reproduced in appendix III. DHS also provided technical comments which we incorporated as appropriate. In its comments, DHS concurred with both of our recommendations and identified actions it planned to take to address them. We are sending copies of this report to the appropriate congressional committees and the Acting Secretary of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This appendix presents individual assessments for each of the 29 programs we reviewed. Each assessment presents information current as of August 2019. They include standard elements, such as an image, a program description, and summaries of the program’s progress in meeting cost and schedule goals, performance and testing activities, and program management-related issues, such as staffing. The information presented in these assessments was obtained from the Department of Homeland Security (DHS) documentation, answers to our questionnaire by DHS officials, interviews with program officials, and includes our analysis of program information. Each assessment also includes the following figures: Fiscal Years 2020–2024 Affordability. This figure compares the funding plan presented in the Future Years Homeland Security Program report to Congress for fiscal years 2020-2024 to the program’s current cost estimate. We use this funding plan because the data are approved by DHS and Office of Management and Budget, and was submitted to Congress to inform the fiscal year 2020 budget process. The data do not account for other potential funding sources, such as carryover funding. Acquisition Program Baseline (APB) Thresholds vs. Current Estimate. This figure compares the program’s cost thresholds from the initial APB approved after DHS’s acquisition management policy went into effect in November 2008 and the program’s current DHS-approved APB to the program’s expected costs as of August 2019. The source for the current estimate is the most recent cost data we obtained (i.e., a department-approved life-cycle cost estimate, updated life-cycle cost estimates submitted during the resource allocation process to inform the fiscal year 2020 budget request, or a fiscal year 2019 annual life-cycle cost estimate update). Schedule Changes. This figure consists of two timelines that identify key milestones for the program. The first timeline is based on the initial APB DHS leadership approved after the department’s current acquisition management policy went into effect. The second timeline identifies when the program expected to reach its major milestones as of August 2019 and includes milestones introduced after the program’s initial APB. Dates shown are based on the program’s APB threshold dates or updates provided by the program office. Test Status. This table identifies key recent and upcoming test events. It also includes DHS’s Director, Office of Test and Evaluation’s assessment of programs’ test results, if an assessment was conducted. Staffing Profile. This figure identifies the total number of staff a program needs (measured in full time equivalents) including how many are considered critical and how many staff the program actually has. Lastly, each program assessment summarizes comments provided by the program office and identifies whether the program provided technical comments. Page left blank intentionally. AUTOMATED COMMERCIAL ENVIRONMENT (ACE) CUSTOMS AND BORDER PROTECTION (CBP) The ACE program is developing software that will electronically collect and process information submitted by the international trade community. ACE is intended to provide private and public sector stakeholders access to information, enhance the government’s ability to determine whether cargo should be admitted into the United States, increase the efficiency of operations at U.S. ports by eliminating manual and duplicative trade processes, and enable faster decision making. Program completed operational testing in June 2018, but cybersecurity was not tested. Collections functionality will remain in the legacy system until additional funding is provided for development. GAO last reported on this program in May 2018 and March 2018 (GAO-18-339SP, GAO-18- 271). Following a cost and schedule breach in April 2017, CBP separated the ACE program’s Collections functionality—which collects and processes duties owed on imported goods—from its Core functionality to permit deployment of the other post-release capabilities, such as Liquidations and Reconciliation. CBP previously reported that officials were not versed in the complexities of collection in the legacy system and underestimated the level of effort required to integrate Collections capabilities into ACE. In August 2018, the program received Department of Homeland Security (DHS) approval to defer Collections functionality as an unfunded requirement. CBP officials said the Collections functionality will remain in the legacy system until funding for development is provided. ACE continued deployment of the Core functionality and updated acquisition documents including the program’s acquisition program baseline (APB) and life-cycle cost estimate (LCCE) to reflect the program changes. DHS leadership approved the program’s updated APB in November 2018—removing the program from breach status. The program achieved full operational capability (FOC) for Core functionality and received acquisition decision event (ADE) 3 approval in November 2018— approximately 2 years later than initially planned. Although the program removed costs associated with Collections functionality, the program’s total APB cost threshold increased by more than $500 million from its prior APB. This cost increase is primarily the result of a change in the way the program’s threshold costs were calculated. CBP officials estimated the total cost of decoupling Collections from ACE’s remaining functionality to be $30 million. In March 2019, the program received funding and approval for ADE 2B for the first of four planned releases of Collections functionality, but did not receive funding for the remaining releases. CBP officials applied for Technology Modernization Funds (TMF). However, in September 2019, CBP officials stated that a decision on TMF funding had not yet been made. CBP officials estimated that it would take 18 months to move Collections into ACE. In June 2019, the program updated its LCCE to inform the budget process—the LCCE includes some costs for Collections functionality, but the total cost is not yet known. Customs and Border Protection (CBP) AUTOMATED COMMERCIAL ENVIRONMENT (ACE) When DHS leadership re-baselined the ACE program in 2013, the program adopted an agile software development methodology to accelerate software creation and increase flexibility in the development process. The ACE program office oversees agile teams that conduct development and O&M activities. Staffing needs for ACE have decreased in the last year, which CBP officials attribute to the program completing most development efforts. These officials explained that staff from prior agile development teams were shifted to sustainment teams. In June 2019, CBP officials told GAO that, while ACE has some critical staffing gaps, these gaps have not affected program execution. CBP officials also stated that they plan to use existing contracts to address staffing needs for the Collections functionality, once funding for development is received. CBP officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. CUSTOMS AND BORDER PROTECTION (CBP) The BEE program is intended to verify the identities of travelers leaving the United States at air, land, and sea ports of entries using biometric data, such as facial recognition. The program has developed a capability to match photos of departing travelers to their passport photos or photos obtained upon a traveler’s arrival into the United States to identify foreign nationals that stay in the United States beyond their authorized periods of admission. CBP is currently focused on the air segment. Program deploying capabilities beyond approved quantity without approval from leadership. CBP pursuing public/private partnerships to reduce costs. GAO last reported on this program in May 2018 and February 2017 (GAO-18-339SP, GAO-17-170). In May 2018, the Department of Homeland Security (DHS) leadership approved BEE’s initial acquisition program baseline (APB) which established the cost, schedule, and performance parameters for the air segment. DHS leadership subsequently granted the BEE program acquisition decision event (ADE) 2A approval for this segment and directed the program to return for a combined ADE 2B/C. DHS leadership delayed the program’s ADE 2B decision—which will authorize the program to initiate development of the air segment—from October 2018 to December 2018 to allow for the completion of the test and evaluation master plan (TEMP). However, in October 2018, CBP officials told GAO that the facial matching service was ready to support nationwide deployment, and the program was on track to reach its initial operational capability (IOC) of supporting 30 international flights per day by December 2018. DHS leadership approved the program’s request to remove ADE 2C—which would authorize low-rate production—from its APB and granted the program ADE 2B in December 2018. In March 2019, DHS leadership approved the program’s updated APB, which reflected schedule changes related to the TEMP, schedule slips related to the fiscal year 2019 partial government shutdown, and removal of ADE 2C. The program’s APB costs goals remained the same. CBP officials said the program plans to re-baseline and achieve ADE 3—which will authorize full-rate production—in September 2019. However, in June 2019, CBP officials told GAO the program has continued to deploy capabilities to airports and airlines—beyond those needed to achieve IOC. The BEE program is primarily funded by fees. Congress provided that half the amount collected from fee increases for certain visa applications from fiscal years 2016 through 2025—up to $1 billion—would be available to DHS until expended for the development and implementation of the BEE system. In February 2018, Congress extended this period through fiscal year 2027. CBP officials said the current funding structure poses challenges because fees fluctuate based on immigration rates. The program conducted an affordability analysis in 2018 that showed projected fees had fallen from $115 million per year to $56 million per year. To address the funding gap, the program reduced the number of officers conducting enforcement activities at airport departure gates and is working with CBP to identify other sources of funding. . Customs and Border Protection (CBP) Since 1996, several federal statutes have required development of an entry and exit system for foreign nationals. DHS has been exploring biometric exit capabilities since 2009 and an Executive Order issued in March 2017 directed DHS to expedite the implementation of the BEE system. CBP is pursuing public/private partnerships in which airlines and airports invest in the equipment to collect biometric data to reduce program costs and improve the passenger boarding process. In September 2019, CBP officials told GAO they have received commitment letters from 28 airports and airlines since March 2018 and officials expect to operate within the airports with the highest volume of international flights by October 2021. CBP officials also told GAO that the program works independently with airlines and airports and does not seek any component or department approvals before proceeding to deploy technologies. These officials stated they proceed in this manner because program stakeholders have been highly engaged since the program’s ADE 1, internal testing results have been positive, and the congressional mandate necessitates expediency. CBP officials said the program’s current staffing level is manageable, but they will need more staff in the future to help manage planned partnerships with airlines and airports. CBP provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. CUSTOMS AND BORDER PROTECTION (CBP) The border wall system is intended to prevent the illegal entry of people, drugs, and other contraband by enhancing and adding to the 654 miles of existing barriers along the U.S. southern border. CBP plans to create a border enforcement zone between a primary barrier—such as a fence—and a secondary barrier. To establish the enforcement zone, the wall system may also include detection technology, surveillance cameras, lighting, and roads for maintenance and patrolling. Program establishes baselines as funding is received, but does not have a cost estimate to support funding plan. Current baselines do not account for all DHS and DOD border wall system construction efforts. GAO last reported on this program in July 2018 and May 2018 (GAO-18-614, GAO-18- 339SP). The Department of Homeland Security (DHS) plans to establish cost, schedule, and performance goals for each individual segment of the border wall system in separate acquisition program baselines (APB) as funding becomes available. The program’s current APBs were approved in May 2019 and account for segments funded in fiscal years 2018 and 2019, totaling nearly 123 miles of border wall system. DHS leadership approved a revised APB for the two segments funded in fiscal year 2018. This included cost and schedule goals for the replacement of an existing 14 miles of primary and secondary barriers in San Diego. It also refined the cost goals for an initial 60 mile segment in the Rio Grande Valley (RGV), because in the 2018 and 2019 Consolidated Appropriations Acts, Congress prohibited use of funds for construction in areas constituting about 4 miles. The program’s total cost for these efforts is nearly $2.2 billion. DHS leadership approved an initial APB for a second segment of nearly 53 miles in RGV in response to funding received in fiscal year 2019. The program’s total cost for this segment is approximately $2.6 billion. However, the design for this segment has not yet been approved, which could affect APB costs or schedule or both. In June 2019, to inform the budget process, the program developed a cost estimate that appears much greater than its APB goals because it reflects DHS’s funding request to Congress—not the current plans of the program. DHS officials reported that they did not have a cost estimate to support the requested amounts because the program develops acquisition documentation after funding becomes available. The current APBs do not account for related construction efforts that may limit oversight of the development of the entire border wall system. For example, in November 2018, CBP leadership was granted approval to oversee a segment replacing about 48 miles of primary pedestrian wall. Further, in February 2019, DHS requested that the Department of Defense (DOD) assist with the construction of infrastructure along the southern border. DOD agreed to provide support and is using $2.5 billion of DOD’s fiscal year 2019 funds to support these efforts. In September 2019 DOD officials identified an additional $3.6 billion, if needed. CBP officials told GAO that they provided a prioritized list of segments and construction standards to DOD, but said that they have limited insight into DOD’s planned efforts. 05/19 FY 2018 APB revised/ FY19 initial APB approved 03/23 FY 2018 segments full operational capability (FOC) Customs and Border Protection (CBP) The Border Wall System Program was initiated in response to an Executive Order issued in January 2017 stating that the executive branch is to secure the southern border through the immediate construction of a physical wall on the southern border of the United States. To expedite the acquisition planning process, CBP officials said they leveraged expertise from staff that worked on previous border fencing programs and were familiar with implementation challenges, such as land access. CBP intends to prioritize segments based on threat levels, land ownership, and geography, among other things. CBP plans to continue coordinating with the U.S. Army Corps of Engineers for engineering support and for awarding and overseeing the construction contracts. CBP officials stated that land access and acquisition issues are significant challenges and could affect the program’s ability to meet its schedule goals. CBP officials reported that the program has sufficient staff to manage the program’s work based on the funding received to date. The program’s unfilled staffing gaps are not yet funded positions. CBP officials stated that they will hire additional staff to fill the vacant positions once funding becomes available. CBP officials reviewed a draft of this assessment and provided no comments. CROSS BORDER TUNNEL THREAT (CBTT) CUSTOMS AND BORDER PROTECTION (CBP) The CBTT program is intended to help CBP identify, acquire, and implement operational services and technologies necessary to obtain subterranean domain awareness along the United States land border. These technologies will help CBP address existing gaps in the prediction, detection, confirmation, investigation, and remediation of cross border tunnels. CBP’s analysis of alternatives for detection capabilities identified a solution and CBP will conduct future analysis. Program performed two technology demonstrations, and CBP officials determined technologies were sufficient. GAO last reported on the program in August 2018 and May 2017 (GAO-18-550, GAO-17-474). In August 2015, the Department of Homeland Security’s (DHS) Under Secretary for Management (USM) granted the CBTT program acquisition decision event (ADE) 1 approval. The program initiated work on an analysis of alternatives (AoA) in March 2016, which considered technologies to detect four CBP classifications of illicit tunnels—rudimentary, sophisticated, mechanically bored, and interconnecting tunnels—but yielded no results. Program leadership and stakeholders subsequently determined that the AoA should be refocused to address tunnel detection threats in seven high-risk operational areas and broadened to incorporate newer tunnel detection technologies, among other things. In May 2018, the AoA was completed and, based on its results, CBP identified a preferred system—a variation of a legacy tunnel detection system used by the Department of Defense (DOD). In June 2018, DHS leadership directed the program to continue technology demonstrations of upgrades to the legacy tunnel detection system in order to mitigate technical and operational risks and refine program requirements, including identification of the areas where the capability will be deployed. At that time, DHS leadership directed the program to return to the acquisition review board for a combined ADE 2A and 2B to establish an initial acquisition program baseline (APB) for tunnel detection capability. CBP officials said the program now plans to pursue only ADE 2A when it returns to the acquisition review board, per DHS’s revised acquisition policy. As of September 2019, the program had not yet completed key acquisition documents that will support the program’s APB. CBP officials told GAO that the program experienced delays in updating the acquisition documents—including the operational requirements document—for the detection capability as a result of continued work with stakeholders. The program continues to work with stakeholders to refine end- user requirements, determine testing needs, and complete a technical assessment. CBP officials told GAO that the program plans to use an incremental acquisition approach to address the other capability gaps. They added that the incremental approach is necessary because the capability gaps the program intends to address are broader than one system can cover. Customs and Border Protection (CBP) CROSS BORDER TUNNEL THREAT (CBTT) In 2008, CBP began collaborating with the DHS Science and Technology Directorate, other federal partners, and private industry to develop and acquire tunnel detection technology. In September 2012, the DHS Inspector General found that CBP did not have the technological capability to detect illicit cross-border tunnels routinely and accurately. DHS leadership subsequently approved the CBTT Mission Needs Statement, which identified six capabilities—predict the location of illicit tunnels; detect the presence of suspected tunnels and tunneling activity and project the trajectory of a discovered tunnel; confirm a tunnel’s existence and map its location and measurements; investigate and exploit tunnels and tunnel activity; remediate discovered tunnels; and coordinate information sharing on tunnel threats. CBP officials stated that the CBTT Concept of Operations (CONOPS) was approved in June 2019. CBP officials also stated that the development of the CONOPS was informed by market research and AoA activities. CBP officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. INTEGRATED FIXED TOWERS (IFT) CUSTOMS AND BORDER PROTECTION (CBP) The IFT program helps the Border Patrol detect, track, identify, and classify illegal entries in remote areas. IFT consists of fixed surveillance tower systems equipped with ground surveillance radar, daylight and infrared cameras, and communications systems linking the towers to command and control centers. CBP plans to deliver or upgrade approximately 48 IFT systems across six areas of responsibility (AoR) in Arizona: Nogales, Douglas, Sonoita, Ajo, Tucson, and Casa Grande. System acceptance test completed in Sonoita AoR; all systems accepted by program. Border Patrol requested CBP add camera suites to address tower reductions in the Ajo and Casa Grande AoRs. GAO last reported on this program in May 2018 and November 2017 (GAO-18-339SP, GAO-18-119). The program declared a potential schedule breach in December 2017 because the program did not receive funding from the Department of Homeland Security (DHS) to address new IFT requirements, including camera upgrades and replacement of existing tower systems deployed in Tuscon and Ajo under a legacy program. In January 2018, CBP officials updated the program’s affordability analysis to reflect a reduction of IFT tower deployments—which mitigated the potential schedule breach. Specifically, a resolution passed within the Tohono O’odham Nation—a sovereign Native American Nation—that reduced the number of IFT tower systems CBP can deploy on the Nation’s land from 15 to 10. This reduction mitigated the funding shortfall that had put the program at risk of not achieving full operational capability (FOC) in September 2020. In February 2019, CBP declared a schedule breach of the program’s current acquisition program baseline (APB) as a result of delays in the negotiations with the Tohono O’odham Nation regarding access to tribal lands to construct towers and deploy IFT systems in the Ajo and Casa Grande AoRs. CBP subsequently reached an agreement with the Nation in March 2019. DHS leadership directed the program to revise its APB to reflect changes in tower deployments. CBP officials told GAO they submitted a revised APB to DHS leadership in June 2019, but as of September 2019 it had not yet been approved. CBP officials anticipate the program’s FOC date will slip to March 2021 as a result of these actions. In June 2019, the program updated its life-cycle cost estimate (LCCE) to inform the budget process. The updated LCCE includes estimated costs for camera upgrades and accounts for the reduction in IFT systems. CBP completed deployments in the Sonoita AoR in October 2017 and replaced legacy systems in the Tucson and Ajo AoRs in September 2018 and December 2018, respectively. In January 2015, Border Patrol requested the program prioritize replacing these legacy systems because the technology was obsolete and more expensive to maintain than IFT technology planned for deployment in other AoRs. 10/15 Initial operational capability (Nogales) Customs and Border Protection (CBP) INTEGRATED FIXED TOWERS (IFT) When CBP initiated the IFT program, it decided to procure a non-developmental system, and it required that prospective contractors demonstrate their systems prior to CBP awarding the contract. The program awarded the contract to EFW, Inc. in February 2014, but the award was protested. GAO sustained the protest and CBP reevaluated the offerors’ proposals before it decided to re-award the contract to EFW, Inc. As a result, EFW, Inc. could not initiate work at the deployment sites until fiscal year 2015. According to CBP officials, the number of IFT towers deployed to a single AoR is subject to change based on Border Patrol assessments. Border Patrol was briefed and approved the reduction of towers within tribal lands. To mitigate capability gaps resulting from the tower reduction, Border Patrol requested the program deploy two additional IFT camera suites in Ajo. DHS leadership directed CBP to develop a border technology plan that includes IFT capabilities. According to CBP officials, the plan calls for an additional 11 AoRs and 35 IFTs. Although the program has not yet received funding for expansion to the 11 AoRs, in September 2018, CBP officials stated they began updating acquisition documents. CBP officials also stated the program does not have a staffing gap, but will require additional staff if funding for the expansion to the 11 AoRs is received. CBP officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. MEDIUM LIFT HELICOPTER (UH-60) CUSTOMS AND BORDER PROTECTION (CBP) UH-60 is a medium-lift helicopter that CBP uses for law enforcement and border security operations, air and mobility support and transport, search and rescue, and other missions. CBP’s UH-60 fleet consists of 20 aircraft acquired from the U.S. Army in three different models. CBP previously acquired 4 modern UH-60M aircraft and converted 6 of its older 16 UH-60A aircraft into more capable UH-60L models. CBP is replacing the remaining 10 UH-60A with reconfigured Army HH-60L aircraft. Flight acceptance testing for the first reconfigured aircraft completed in February 2018. Program is assessing additional medium lift capability requirements. GAO last reported on this program in May 2018 (GAO-18-339SP). In July 2018, Department of Homeland Security (DHS) leadership granted the program acquisition decision event (ADE) 3 approval and approved the replacement of CBP’s remaining UH-60A aircraft for reconfigured Army HH-60L aircraft. CBP will begin replacing its UH-60A model aircraft on a one-to-one basis as the reconfigured Army HH-60Ls are delivered. DHS leadership previously approved the transfer of three reconfigured HH-60Ls. According to CBP officials, the ADE 3 approval to replace the remaining seven aircraft was based on the evaluation of an initial reconfigured prototype, which was delivered in 2018. CBP officials anticipate that the second and third reconfigured HH-60Ls will be delivered in fiscal year 2020. The program re-baselined as part of the ADE 3 approval process, removing it from breach status. The program previously experienced cost increases after accommodating a change in DHS’s appropriations structure and schedule slips because of a directive from DHS to develop a comprehensive border plan, which contributed to delays in getting approvals for some of the documents required for ADE 3. The program also anticipated delays in delivery for the second reconfigured HH-60L because of a redesign to be compliant with federal aviation regulations. DHS leadership and CBP officials determined that the effect of the schedule breach was minimal because the program was able to adjust its schedule so that the second and third reconfigured HH-60Ls can be accepted concurrently. The program still plans to achieve full operational capability (FOC) in September 2022 once all 10 of the reconfigured HH-60L aircraft are accepted and deployed. The program updated its life-cycle cost estimate (LCCE) to inform the program’s revised acquisition program baseline (APB). The program’s acquisition cost thresholds increased by nearly $100 million, and the operations and maintenance (O&M) cost thresholds decreased by approximately $15 million. These changes reflect updates to aircraft operational hours and the results of the Army’s annual obsolescence study, among other things. The updated LCCE also removes personnel costs included in the program’s initial APB, which CBP officials previously told GAO are funded through a separate, central funding account for all of CBP’s air and marine assets. Customs and Border Protection (CBP) MEDIUM LIFT HELICOPTER (UH-60) In July 2018, DHS leadership directed CBP to address requirements for additional medium-lift capability, including coordinating with Department of Defense and DHS stakeholders, such as the U.S. Coast Guard, that also maintain a fleet of H-60 aircraft. CBP officials stated a desire to replace its other medium lift helicopters as they are retired from the fleet with additional reconfigured HH-60L aircraft. This would not increase the overall number of medium lift helicopters, but would increase the number of UH-60 aircraft. If the number of UH-60 aircraft increases, the program will need to seek approval from DHS and extend its FOC date. In April 2019, CBP updated its interagency agreement with the Army to support completing the program’s currently approved quantity. According to CBP officials, this agreement could support acquiring additional reconfigured HH-60Ls if approved by DHS. CBP previously acquired UH-60 as a part of its Strategic Air and Marine Program (StAMP). In July 2016, DHS leadership designated UH-60 as a separate and distinct major acquisition program. In October 2018, CBP officials told GAO they continue to maintain a consolidated program office where the same staff from StAMP support all remaining acquisitions, including UH-60. CBP officials said they have refined the program’s staffing profile and taken steps to mitigate the gap. For example, in June 2019, CBP officials said they had hired four new employees and established a memorandum of agreement with CBP’s Office of Acquisition for matrixed support to assist with developing acquisition documents, as needed. CBP officials stated that as of August 2019, DHS’s Joint Requirements Council validated a requirement for 35 total Medium Lift Helicopters, and the program office is working on a strategy to achieve that inventory target. CBP officials also provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. MULTI-ROLE ENFORCEMENT AIRCRAFT (MEA) CUSTOMS AND BORDER PROTECTION (CBP) MEA are fixed-wing, multi-engine aircraft that can be configured to perform multiple missions including maritime, air, and land interdiction, as well as signals detection to support law enforcement. The maritime and air interdiction MEA are equipped with search radar and an electro-optical/infrared sensor to support maritime surveillance and airborne tracking missions. MEA will replace CBP’s fleet of aging C-12, PA-42, and BE-20 aircraft. Air interdiction configuration is operationally effective and suitable with limitations; cyber testing is not complete. Program developing requirements for next configuration; pursuing total of 38 MEA. GAO last reported on this program in May 2018 (GAO-18- 339SP). In February 2019, Department of Homeland Security (DHS) leadership approved a revised acquisition program baseline (APB), which increased the program’s quantity to 29 MEA: 16 previously approved maritime interdiction MEA and 13 additional air interdiction MEA. CBP officials told GAO they also requested approval to acquire all remaining air interdiction MEA. However, in April 2019, DHS leadership directed CBP to complete follow-on operational test and evaluation (OT&E) of the air interdiction configuration and undergo an acquisition decision event (ADE) 3 review before the program could receive full-rate production approval. DHS leadership previously approved CBP’s request to procure additional aircraft in the air interdiction configuration that exceeded the program’s initial baseline of 16 MEA. Specifically, DHS leadership approved procurement of MEA 17 in September 2017 after congressional conferees agreed to an additional aircraft beyond DHS’s budget request. In addition, DHS leadership approved MEA 18-20 in August 2018. CBP officials told GAO it was necessary to procure additional MEA to maintain the production schedule for already ordered aircraft. CBP officials accepted delivery of MEA 16 in February 2019—completing delivery of all maritime interdiction configured MEA. CBP officials said the program experienced a few months delay in delivery of MEA 13-16 because the contractor began laying off staff prior to the program receiving DHS leadership approval to acquire MEA 18-20. According to CBP officials, the program will need to receive ADE 3 approval to procure the remaining air interdiction MEA before the end of September 2019 to avoid future production issues. The program’s revised APB extends the program’s full operational capability (FOC) date by nearly 7 years, to account for the production and delivery of the air interdiction aircraft. The program updated its life-cycle cost estimate (LCCE) in September 2018 to inform its revised baseline. This estimate decreased by approximately $1.4 billion from the program’s previous LCCE due to a reduction in the number of total aircraft—from the program’s proposed end state of 38 MEA to the 29 included in its revised APB—and planned flight hours. Customs and Border Protection (CBP) MULTI-ROLE ENFORCEMENT AIRCRAFT (MEA) In April 2016, CBP identified capability needs in three additional mission areas and proposed increasing the program’s total to 38 MEA by adding 13 air (reflected in the February 2019 APB), six land interdiction MEA, and three signals detection MEA. The Joint Requirements Council endorsed CBP’s findings, but recommended CBP develop a number of requirements documents—including an operational requirements document (ORD)—to fully validate the findings. In June 2019, CBP officials said they had begun developing requirements for the land interdiction MEA—the next configuration the program plans to pursue. CBP previously acquired MEA as a part of its Strategic Air and Marine Program (StAMP). In July 2016, DHS leadership designated MEA as a separate and distinct major acquisition program. In October 2018, CBP officials told GAO they continue to maintain a consolidated program office where the same staff from StAMP support all remaining acquisitions, including MEA. CBP officials said they have refined the program’s staffing profile and taken steps to mitigate the gap. For example, in June 2019, CBP officials said they had hired four new employees and established a memorandum of agreement with CBP’s Office of Acquisition for matrixed support to assist with developing acquisition documents, as needed. CBP officials previously told GAO that the staffing gap contributed to delays in developing acquisition documentation for the air interdiction MEA. CBP officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. CUSTOMS AND BORDER PROTECTION (CBP) The NII Systems Program supports CBP’s interdiction of weapons of mass destruction, contraband such as narcotics, and illegal aliens being smuggled into the United States, while facilitating the flow of legitimate commerce. CBP officers use large- and small-scale NII systems at air, sea, and land ports of entry; border checkpoints; and international mail facilities to examine the contents of containers, railcars, vehicles, baggage, and mail. CBP is evaluating technologies to increase efficiencies and address capability gaps. Staffing challenges pose risk to current program execution and planning for follow-on to NII program. GAO last reported on this program in May 2018 (GAO-18-339SP). The NII Systems program is on track to meet its approved cost and schedule goals. The Consolidated Appropriations Act of 2019 included $570 million of acquisition funding for the NII program—$520 million above the President’s budget level. CBP officials told GAO they plan to use the additional acquisition funding primarily to increase scanning capability at land points of entry along the southwest border by recapitalizing some large-scale capabilities and deploying additional small-scale capabilities. The program updated its life-cycle cost estimate (LCCE) in June 2018. The program’s acquisition costs remain within its acquisition program baseline (APB) cost thresholds and continue to decrease. Compared to the prior year’s estimate, the program’s acquisition costs decreased by $81 million and operations and maintenance increased by $33 million. However, the LCCE update only estimated costs through fiscal year 2026—9 years short of the program’s final year. In June 2019, CBP officials told GAO that they were in the process of updating the program’s LCCE. These officials stated that they plan to extend the LCCE through the program’s final year and adjust program costs based on program changes made in response to the additional funding received. CBP plans to deploy full operational capability (FOC) quantities of 342 large- and 5,455 small-scale NII systems in fiscal year 2020—4 years earlier than the program’s current APB threshold date. In November 2018, Department of Homeland Security (DHS) leadership decided that once FOC quantities for large and small-scale systems are deployed, CBP will initiate a transfer of the NII program to the operational activity for sustainment efforts. In addition, once FOC quantities are deployed, DHS leadership determined that CBP may adjust large- and small-scale NII deployment quantities in excess of FOC with similarly capable systems to address changing capacity needs and emerging threats. CBP is assessing requirements to address capability gaps, such as increased throughput. In June 2019, CBP officials reported that some technologies being assessed can be procured through the current NII program because CBP considers them to be similarly capable systems. However, these officials also told GAO that CBP is developing acquisition documents to inform a follow-on NII program for other technologies. Customs and Border Protection (CBP) CBP is in the process of assessing requirements to inform the follow-on NII program. In March 2017, the Joint Requirements Council (JRC) validated a capability analysis report (CAR) that assessed capability gaps in NII operations to assist with identifying potential upgrades to existing systems and developing requirements for future systems. DHS leadership approved a new NII Mission Needs Statement (MNS) in August 2018, which updated the capability gaps identified in the CAR and described mission needs and capabilities to address the gaps. The JRC endorsed the MNS, but recommended that CBP address cybersecurity threats and vulnerabilities as requirements and solutions evolve, and also include the Transportation Security Administration—which leverages some of the same equipment to perform their mission—in defining requirements, among other things. CBP officials told GAO that they are developing acquisition documentation to inform acquisition decision event 1 for the follow-on NII program planned for September 2019, including a concept of operations and an initial cost estimate. CBP’s ability to successfully execute the existing NII Systems program and plan for future efforts may be at risk because of understaffing. As of September 2019, the program continued to face a staffing gap of approximately 21 percent. CBP officials said that they plan to mitigate the gap with government personnel from other offices within the component and with contractor support. CBP officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. REMOTE VIDEO SURVEILLANCE SYSTEM (RVSS) CUSTOMS AND BORDER PROTECTION (CBP) RVSS helps the Border Patrol detect, track, identify, and classify illegal entries across U.S. borders. RVSS consists of daylight and infrared video cameras mounted on towers and buildings with communications systems that link to command and control centers. From 1995 to 2005, CBP deployed approximately 310 RVSS towers along the U.S. northern and southern borders, and initiated efforts to upgrade legacy RVSS towers in Arizona in 2011. Diesel generators that power relocatable towers cause vibrations that could impact mission operations. Once funded, program plans to award a contract for additional deployments along the southwest border. GAO last reported on this program in May 2018 and November 2017 (GAO-18-339SP, GAO-18-119). In April 2016, Department of Homeland Security (DHS) leadership elevated RVSS from a level 3 program—which focused on upgrading legacy RVSS in Arizona—to a level 1 program after approving CBP’s plan to expand deployments to the Rio Grande Valley (RGV) sector and adding an additional 6 sectors along the southwest border—Laredo, Del Rio, Big Bend, El Paso, El Centro, and San Diego. DHS leadership approved the program to move forward with deployments at two Border Patrol stations within the RGV, which can be completed as options under the program’s existing contract, if exercised. However, DHS leadership also directed the program to re-baseline to account for its expanded scope and conduct an acquisition decision event (ADE) 2A to obtain approval for additional deployments. CBP officials previously told GAO the program anticipated conducting its ADE 2A and obtaining DHS leadership approval for an acquisition program baseline (APB) establishing cost, schedule and performance goals for the expanded program by December 2018. As of September 2019, the program had not yet received approval for key acquisition documents to conduct ADE 2A, including the APB, but CBP officials anticipate approval of these documents by March 2020. CBP officials primarily attribute these delays to a lack of funding for the additional deployments. CBP officials said the upcoming APB will include only deployments to Arizona and the RGV sector to align with funding received. Future deployments will require additional APB updates, which CBP officials said would be developed as funding becomes available. In June 2019, the program updated its life-cycle cost estimate (LCCE) to inform the budget process. The updated LCCE included the expansion to the 6 sectors along the southwest border, relocatable RVSS towers, and operations and maintenance costs for previously fielded systems. However, CBP officials told GAO the LCCE is in the process of another update, which will inform the upcoming APB and include the expansion across additional sectors across southwest border and upgrades to legacy RVSS towers. Customs and Border Protection (CBP) REMOTE VIDEO SURVEILLANCE SYSTEM (RVSS) In July 2013, CBP awarded a firm fixed-price contract for a commercially available, non-developmental system. This contract covered the program’s initial scope to deploy upgraded RVSS in Arizona and included options for some initial work within the RGV sector. According to CBP officials, the program will need to award a new contract to cover expansion to the remaining six sectors along the southwest border. CBP officials drafted the request for proposals for the new contract, but it cannot be released until funding is received. CBP officials said the program is experiencing challenges in the RGV sector related to land acquisition. The U.S. Army Corps of Engineers is leading efforts to acquire land for RVSS and other border security programs, including the Border Wall System Program (BWSP). CBP officials told GAO that the RVSS program is coordinating with BWSP on its planned deployments within the RGV sector. Program officials anticipate that some RVSS towers will be co-located within the border wall. In the interim, CBP officials said the program is using short-term agreements with landowners to place relocatable towers in areas where border wall construction is planned. These officials reported that the short-term agreements provide flexibility for the placement of towers and can be completed more quickly than permanent agreements. CBP officials stated that the program’s current staffing plan was based on receiving funding for the expansion to RGV. Program officials said they will address the staffing needs once additional funding is received, but current operations have not been affected by the staffing gap. CBP officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. CUSTOMS AND BORDER PROTECTION (CBP) The TACCOM program is intended to upgrade land mobile radio infrastructure and equipment to support approximately 95,000 users at CBP and other federal agencies. It is replacing obsolete radio systems with modern digital systems across various sectors located in 19 different service areas, linking these service areas to one another through a nationwide network, and building new communications sites to expand coverage in five of the 19 service areas. CBP officials reported that prior software issues have been addressed. Program continues to face staffing challenges due to competition from the private sector, among other things. GAO last reported on this program in May 2018 (GAO-18-339SP). In September 2018, the TACCOM program achieved full operational capability (FOC)—nine months later than initially planned. However, in July 2018, the program’s operational test authority (OTA) conducted a survey of end users and concluded that there were still large gaps in coverage the TACCOM capabilities were intended to address. CBP officials stated that limited funding has affected the program’s ability to address the remaining gaps in coverage. Department of Homeland Security (DHS) leadership previously approved a re-baseline of the TACCOM program in November 2017 after it experienced a schedule slip and cost growth. In July 2017, CBP officials notified DHS leadership that the program would not achieve FOC as planned due to issues related to federal information security requirements. In addition, the program experienced cost growth as a result of increased contractor labor costs and support for facilities and infrastructure. In November 2017, DHS’s Chief Financial Officer (CFO) approved the program’s revised life-cycle cost estimate (LCCE). At that time, DHS‘s CFO noted that the program’s estimate exceeded its available funding and requested that the program address the affordability gap before it was re-baselined. Nevertheless, DHS leadership approved the program’s revised acquisition program baseline (APB). CBP officials subsequently identified errors in the approved APB cost threshold tables and provided revised amounts, which are presented here. In September 2018, program officials told GAO that they completed an affordability analysis and submitted it to CBP and DHS leadership. CBP officials reported that the funding the program received in 2018 and carryover funds from prior years decreased the program’s affordability gap. However, CBP reported that in future years, funding gaps will require the program to reduce operations and maintenance requirements to match the appropriated funding and will continue to limit the program’s ability to address coverage gaps. Customs and Border Protection (CBP) CBP officials told GAO that in January 2018, the program moved from a mission support office to a joint program office under Border Patrol as part of CBP’s reorganization. The goal of this move was to make CBP land mobile radio capabilities seamless by combining the mission critical voice functions within Air and Marine Operations, the Border Patrol, and the Office of Field Operations—the TACCOM program’s primary customers—under one organizational leader: the Border Patrol Chief. In September 2018, CBP officials told GAO that the program reorganized staff within the program as it transitioned to an office under Border Patrol. CBP officials reported that hiring and retaining qualified land mobile radio engineers and information technology technical staff is a challenge because of lengthy hiring timeframes and competition with the private sector. CBP officials stated that the TACCOM upgrades improved interoperability, coverage, capacity, reliability and encryption to provide critical communications support to the agents and officers who secure the Nation’s borders. The program continues to provide LMR System Maintenance to include operation, sustainment and performance monitoring to ensure reliable and consistent border protection communications. CBP officials also provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. CUSTOMS AND BORDER PROTECTION (CBP) TECS (not an acronym) is a law-enforcement information system that has been in place since the 1980s and that helps CBP officials determine the admissibility of persons entering the United States at border crossings, ports of entry, and prescreening sites located abroad. CBP initiated efforts to modernize TECS to provide users with enhanced capabilities for accessing and managing data. Costs increased by $400 million in revised cost estimate due to extended sustainment timeframe. CBP working to address and prevent major system outages. GAO last reported on this program in May 2018 (GAO-18-339SP). Department of Homeland Security (DHS) leadership approved the fourth version of the program’s acquisition program baseline (APB) in July 2016. In this APB, CBP split full operational capability (FOC) into two separate operational capability milestones to better reflect the program’s activities at its primary and secondary data centers. CBP delivered operational capability at the primary data center and transitioned all remaining TECS users to the modernized system in December 2016. CBP delivered operational capability at the secondary data center in June 2017—as scheduled. This data center provides redundant TECS access to minimize downtime during system maintenance or unscheduled outages. However, not all test results were available in time for the program’s acquisition decision event (ADE) 3 decision. In August 2017, DHS leadership directed CBP to conduct follow-on operational test and evaluation (OT&E) activities to address known issues and conduct cybersecurity OT&E. The program completed follow-on OT&E in October 2018. DHS’s Director, Office of Test and Evaluation (DOT&E) completed an assessment of the test results in June 2019—which is intended to inform acquisition decisions. In June 2019, the program’s annual life-cycle cost estimate (LCCE) was updated in accordance with DHS’s guidance to include operations and maintenance (O&M) costs for 10 years past the program’s planned FOC date. The updated LCCE includes program costs through fiscal year 2028—7 years longer than the prior LCCE and the program’s current APB cost goals. However, the LCCE update does not include estimated costs for all program plans, such as migrating the data centers to a cloud infrastructure. CBP officials plan to incorporate these costs into future LCCE updates when requirements are better defined. The program’s O&M costs increased and exceeded the program’s APB O&M cost threshold by approximately $400 million. DHS officials stated that the additional O&M costs do not constitute a cost breach because the program is considered to be in O&M phase of the acquisition life cycle. Customs and Border Protection (CBP) Since the program has completed development, CBP is focused on ensuring that the modernized TECS system works as intended by addressing operational issues as they are identified. For example, in January 2017, TECS Modernization experienced a major outage that resulted in airport delays. CBP officials previously said that they continually monitor system health through a 24/7 operations center and have established a group dedicated to address system issues. In November 2017, DHS’s Office of Inspector General (OIG) found that CBP took sufficient steps to resolve the January 2017 outage, but underlying issues could result in future outages, including inadequate software capacity testing and deficient software maintenance. The OIG made five recommendations for CBP to implement improvements. CBP concurred with four of the recommendations but did not concur with a recommendation regarding CBP’s need to ensure staff make timely notifications of critical vulnerabilities to operating systems. CBP reported that the program’s notification activities were within DHS’s vulnerability management policy windows for testing and deploying software patches that were not deemed critical. Further, in September 2017, the DHS OIG found that nearly 100 outages, periods of latency, or instances of degraded service, were reported for TECS Modernization applications between June 2016 and March 2017, and recommended that CBP develop a plan to address factors that contributed to these challenges. CBP concurred with the recommendations. CBP officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. CONTINUOUS DIAGNOSTICS AND MITIGATION (CDM) CYBERSECURITY AND INFRASTRUCTURE SECURITY AGENCY (CISA) The CDM program aims to strengthen cybersecurity of the federal government’s networks by continually monitoring and reporting vulnerabilities at more than 65 civilian agencies. CDM provides four capabilities: Asset Management reports vulnerabilities in hardware and software; Identity and Access Management focuses on user access controls; Network Security Management will report on efforts to prevent attacks; and Data Protection Management will provide encryption to protect network data. Program revised its key performance parameters to better align with cybersecurity standards. The program began using a new contract vehicle and is hiring additional staff to support new capabilities. GAO last reported on this program in May 2018 (GAO-18-339SP). According to CISA officials, as a result of the 2019 partial government shutdown, the program experienced delays that impacted the program’s ability to achieve initial operational capability (IOC) for Identity and Access Management and Network Security Management capabilities as planned. In response, Department of Homeland Security (DHS) leadership approved a 3-month extension to both milestones. As a result, the IOC threshold date for Identity and Access Management capabilities was extended to and later achieved in June 2019. The IOC threshold date for Network Security Management was extended to December 2019. The program updated its life-cycle cost estimate (LCCE) in April 2019 to inform the budget process. This estimate exceeds the program’s current operations and maintenance (O&M) and total life-cycle cost thresholds by approximately $300 million and $100 million, respectively. The program’s cost increase is primarily attributed to evolving requirements described in the explanatory statements accompanying recent Appropriations Acts and the Office of Management and Budget (OMB). Specifically, CISA officials said the program received $110 million above the Presidential Budget Request and noted this was to accelerate procurement of CDM capabilities for additional agencies not in the original program scope and accelerate mobile and cloud computing visibility across the .gov domain, among other things. In addition, the program received funding in 2018 and 2019 after OMB directed that the CDM program cover certain costs of sustaining licenses for supported agencies, which CISA officials estimate will cost the program an additional $62 million. The program also estimates that O&M costs for these additional requirements will require a total of an additional $79 million in future years. In May 2019, CISA officials said the program is updating key acquisition documentation, such as its acquisition program baseline (APB) and LCCE, to inform acquisition decision event (ADE) 2B for Data Management Protection capabilities. They noted that the updated acquisition documents will account for the increased demand for CDM services. The program previously planned to achieve this ADE 2B by March 2019. However, due in part to the partial government shutdown, the program now plans to achieve the ADE 2B in 2020. Cybersecurity and Infrastructure Security Agency (CISA) CONTINUOUS DIAGNOSTICS AND MITIGATION (CDM) The CDM program updated its acquisition plan to reflect a change in strategy for procuring CDM tools and services. Previously, the program used blanket purchase agreements established by the General Services Administration (GSA) Federal Supply Schedule. CISA officials told GAO that in February 2018 the program began using an existing GSA government-wide acquisition contract and as of August 2019, the program has awarded 5 of 6 planned task orders to obtain CDM tools and services on behalf of participating agencies. According to CISA officials, the new acquisition strategy is intended to provide greater flexibility in contracting for current capabilities and to support future capabilities. Participating agencies will also be able to order additional CDM-approved products or services from GSA’s schedule for information technology equipment, software, and services. The program previously used the term “phases” and renamed the phases in the fall of 2018 to align with the associated capabilities it deploys. CISA officials explained that a phased deployment implied a serial implementation; however, CDM capabilities can be deployed in parallel. The program is not currently experiencing workforce challenges. The program received approval for 29 new positions to address staffing needs for the Network Security Management and Data Protection Management capabilities. Officials plan to fill those positions in fiscal years 2019 and 2020. CISA officials stated that in addition to efforts identified in this assessment, the program continues to manage its budget to ensure program costs match available funding and is leveraging the collective buying power of federal agencies and strategic sourcing to continue achieving government cost savings on CDM products. CISA officials also provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. NATIONAL CYBERSECURITY PROTECTION SYSTEM (NCPS) CYBERSECURITY AND INFRASTRUCTURE SECURITY AGENCY (CISA) NCPS is intended to defend the federal civilian government from cyber threats. NCPS develops and delivers capabilities through a series of “blocks.” Blocks 1.0, 2.0, and 2.1 are fully deployed and provide intrusion-detection and analytic capabilities across the government. The NCPS program is currently deploying EINSTEIN 3 Accelerated (E3A) to provide intrusion-prevention capabilities and plans to deliver block 2.2 to improve information sharing across agencies. Program capabilities determined to be operationally suitable, effective, and cyber resilient with limitations. Staffing challenges may impact program execution. GAO last reported on this program in May 2018 (GAO-18-339SP). In February 2018, the Department of Homeland Security’s (DHS) Under Secretary for Management (USM) granted NCPS acquisition decision event (ADE) 3 approval for E3A to transition to sustainment and ADE 2C approval for block 2.2 to deploy additional capabilities. DHS’s USM also directed NCPS to address several issues identified during test events that informed the ADEs, including the following: For EA—Conduct follow-on operational test and evaluation (OT&E) by March For block 2.2—Review the operational requirements document (ORD) and concept 2019 to assess cybersecurity, among other things. of operations (CONOPS) to ensure they accurately reflect the mission environment and processes, review current and planned capabilities to ensure they will adequately address the ORD and CONOPS, and conduct another operational assessment (OA) prior to initial OT&E. The program revised its acquisition program baseline (APB) in January 2018 in preparation for the ADEs. However, the program updated its APB again in October 2018 to address an error found in the life-cycle cost estimate (LCCE), to add an additional 2 years of program costs, and to revise the approach to estimating threshold costs. Specifically, the LCCE that provided the basis for the program’s APB cost goals did not accurately account for the program’s sunk costs. Once corrected, the program’s total life-cycle cost threshold was $5.9 billion—more than $1.7 billion more than in the program’s January 2018 APB. CISA officials reported that while correcting the sunk costs increased the APB cost goals, the change did not affect estimating future costs and, therefore, will not impact program affordability. In March 2019, to inform the budget process, the program updated its corrected LCCE—which is within its current APB cost goals. In the program’s January 2018 APB, the ADE 3 date for block 2.2 slipped by 2 years— from March 2019 to March 2021—compared to its prior APB. According to CISA officials, this milestone was revised due to bid-protest-related delays involving the award of the program’s development, operations, and maintenance contract. CISA officials said that due to several protests, the award was delayed until June 2018— nearly 3 years later than planned. Cybersecurity and Infrastructure Security Agency (CISA) NATIONAL CYBERSECURITY PROTECTION SYSTEM (NCPS) A intrusion-prevention capabilities have been primarily provided through sole source contracts with internet service providers and a contract to provide basic intrusion-prevention services. In December 2015, Congress required DHS to make available for use by federal civilian agencies, certain capabilities, such as those provided by NCPS’s EA at approximately 93 percent of federal civilian agencies and departments and, in October 2018, CISA officials reported that NCPS was up to 95 percent, with mainly small and micro organizations remaining. CISA officials said they are working with the various agencies to migrate agency email to a cloud environment, but each department and agency requires a unique solution and coordination can be a challenge. In April 2019, CISA officials reported that if the program’s staffing gap is not addressed, the program may experience a delay in meeting mission requirements. CISA officials told GAO that the federal hiring process and DHS’s lengthy suitability screening process have made recruitment efforts challenging because qualified candidates often find other employment while waiting for these processes to be completed. In addition, CISA officials anticipate workforce challenges if, in the future, they are not able to use compensation flexibility for cybersecurity specialists. CISA officials reviewed a draft of this assessment and provided no comments. NEXT GENERATION NETWORKS PRIORITY SERVICES (NGN-PS) CYBERSECURITY AND INFRASTRUCTURE SECURITY AGENCY (CISA) NGN-PS is intended to address an emerging capability gap in the government’s emergency telecommunications service, which prioritizes phone calls for select officials when networks are overwhelmed. CISA executes NGN-PS through commercial telecommunications service providers, which addresses the government’s requirements, as they modernize their own networks. Full operational capability for wireless capabilities delayed by 3 years to incorporate design changes in network. New program for acquisition of data and video capabilities to begin in fiscal year 2020. GAO last reported on this program in May 2018 (GAO-18-339SP). The NGN-PS program is developing and delivering prioritized voice capability in three increments: increment 1 maintains current priority service on long distance calls as commercial service providers update their networks; increment 2 delivers wireless capabilities; and increment 3 is intended to address landline capabilities. In October 2018, Department of Homeland Security (DHS) leadership granted the NGN-PS program acquisition decision event (ADE) 3 for increment 1. At that time, the program also declared full operational capability (FOC) for increment 1. Once operational, capabilities acquired by NGN-PS are transferred to CISA’s Priority Telecommunications Service program. In April 2018, DHS leadership approved a revised acquisition program baseline (APB) for NGN-PS and subsequently authorized the program to initiate development of increment 3. The previous APB included only costs and schedule milestones associated with increments 1 and 2. The revised APB modified the program’s cost and schedule goals to include goals for increment 3 and updates to cost goals previously established for increments 1 and 2. Specifically, the program’s total acquisition cost threshold increased by $68 million. This change reflects $144 million in additional costs to develop landline capabilities and a cost savings of approximately $100 million on previous increments, among other things. Program officials primarily attributed the cost savings on increment 1 to design changes implemented by a commercial service provider within its network. In addition, according to program officials, the increment 2 FOC goal was revised in the updated APB to allow additional time for a commercial service provider to incorporate design changes into its network. As a result, the FOC date for increment 2 slipped 3 years to December 2022. The program plans to achieve FOC for increment 3 in December 2025. The program updated its life-cycle cost estimate (LCCE) in February 2019. The updated LCCE includes operations and maintenance (O&M) costs, although the APB does not. Officials said this is not considered a breach because the O&M costs include staffing outside of O&M phase activities. Cybersecurity and Infrastructure Security Agency (CISA) NEXT GENERATION NETWORKS PRIORITY SERVICES (NGN-PS) In October 2018, DHS leadership approved the separation of the development of capabilities for data and video priority services into a new acquisition program. DHS leadership approved the decision because data and video capabilities are different than landline priority, and the addition of these capabilities would significantly extend the expected end date of the NGN-PS program. CISA officials anticipate establishing a preliminary baseline for the data and video capabilities in early fiscal year 2020. NGN-PS was established in response to an Executive Order requiring the federal government to have the ability to communicate at all times during all circumstances to address national security issues and manage emergencies. A Presidential Policy Directive issued in July 2016 superseded previous directives requiring continuous communication services for select government officials. According to CISA officials, the new directive validates requirements for the voice phase and was used to develop requirements for the data and video phase. In May 2019, the program reported four critical staffing vacancies, including two new positions. The program reported that it continues to have difficulty filling a systems engineer billet, which program officials attribute to the lengthy federal hiring process, DHS’s suitability screening process, and the fiscal year 2019 partial government shutdown. To mitigate the impact of the staffing gap on program execution, the program leverages contract support and staff from the Priority Telecommunications Service program. In addition to activities identified in this assessment, CISA officials stated that the program will continue planning for data and video priority in future budget years. CISA officials also said that service providers undergo annual network service verification testing and that the program is currently making progress in hiring for numerous positions. CISA officials also provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. HOMELAND ADVANCED RECOGNITION TECHNOLOGY (HART) HART will replace and modernize DHS’s legacy biometric identification system—known as IDENT—which shares information on foreign nationals with U.S. government and foreign partners to facilitate legitimate travel, trade, and immigration. The program plans to develop capabilities in four increments: increments 1 and 2 will replace and enhance IDENT functionality; increments 3 and 4 will provide additional biometric services, as well as a web portal and new tools for analysis and reporting. Program updated operational requirements document and revised its key performance parameters. Program is taking steps to address challenges as a result of a shortfall in staff with technical skillsets. GAO last reported on this program in May 2018 (GAO-18-339SP). In May 2019, DHS leadership approved a revised acquisition program baseline (APB) for the HART program, removing it from breach status, after the program experienced a schedule slip in June 2017. Specifically, the HART program declared a schedule breach when officials determined the program would not be able to meet its initial APB milestones. HART officials attributed the schedule slip to multiple delays in awarding the contract for increments 1 and 2 and a subsequent bid protest—which GAO denied. The program initiated work with the contractor in March 2018 and revised key acquisition documents, including its acquisition program baseline (APB) and life-cycle cost estimate (LCCE), to reflect program changes. For example, officials revised these documents to account for schedule delays and the contractor’s solution for enhanced biometric data storage. Specifically, the contractor plans to deliver services using a cloud-based solution rather than through DHS’s data centers. The HART performance work statement shows delivering services through the cloud provides greater flexibility to scale infrastructure supporting services at a lower cost. The program’s initial operational capability (IOC) date—when all customers will transition from using IDENT to HART—slipped 2 years to December 2020. This is a significant challenge because IDENT is at risk of failure and additional investments are necessary to keep the system operational. HART’s full operational capability (FOC) date—when the program plans to deploy enhancements of biometric services and new tools for analysis and reporting—slipped nearly 3 years to June 2024. HART’s total APB cost thresholds decreased by approximately $2 billion, which officials primarily attribute to the less expensive cloud-based solution and removal of IDENT upgrade costs, among other things. However, officials identified a risk that costs associated with the cloud-based solution could increase because technical requirements were not fully developed when the LCCE informing the revised APB was developed. As a result, HART is at risk for a future cost breach once these technical requirements are better defined. The affordability surplus from fiscal years 2020 through 2024 may be overstated because, according to officials, projected funding covers both IDENT and HART. HOMELAND ADVANCED RECOGNITION TECHNOLOGY (HART) In April 2019, following the passage of the Cybersecurity and Infrastructure Security Agency (CISA) Act of 2018, the transfer of CISA’s Office of Biometric Identity Management (OBIM)—which includes the HART program—to DHS’s Management Directorate was implemented. The transfer was informed by a working group including OBIM, DHS’s MGMT, and CISA subject matter experts. In June 2019, HART officials told GAO they are currently planning for increments 3 and 4, which will provide new and enhanced capabilities, analytics, and reporting, and additional biometric modalities and services, among other things. In June 2019, HART officials released a request for information for increments 3 and 4, which will inform the program’s acquisition plan and statement of work for a request for proposal. At the direction of DHS leadership, HART program officials coordinated with DHS’s Chief Technology Officer to assess the skills and functions of staff necessary to execute the program. In its August 2019 staffing plan, the program reported workforce risks, including a potential shortfall in staff with technical skillsets; however, officials stated that they are mitigating the shortfall, in part, by providing training activities for current staff. In June 2019, HART officials noted that the federal hiring process and DHS’s lengthy security clearance process have made recruitment efforts challenging. HART officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. LOGISTICS SUPPLY CHAIN MANAGEMENT SYSTEM (LSCMS) FEDERAL EMERGENCY MANAGEMENT AGENCY (FEMA) LSCMS is a computer-based tracking system that FEMA officials use to track shipments during disaster-response efforts. It is largely based on commercial-off- the-shelf software. FEMA initially deployed LSCMS in 2005, and initiated efforts to enhance the system in 2009. According to FEMA officials, LSCMS can identify when a shipment leaves a warehouse and the location of a shipment after it reaches a FEMA staging area near a disaster location. LSCMS found operationally effective and operationally suitable with limitations, but not cyber secure. Program transitioned to cloud data storage and plans to conduct annual cybersecurity testing. GAO last reported on this program in May 2018 (GAO-18- 339SP). In September 2019, Department of Homeland Security (DHS) leadership granted the program approval of acquisition decision event (ADE) 3 and acknowledged the program’s achievement of full operational capability (FOC). DHS leadership previously denied the program’s request for acquisition decision event ADE 3 and FOC approval until issues with the system’s backup server were resolved. Program officials reported that the program addressed these issues in August 2019. In November 2017, DHS leadership approved a revised acquisition program baseline (APB) after the LSCMS program experienced a schedule slip because of the 2017 hurricane season. FEMA officials said the need to deploy LSCMS personnel in support of response and recovery efforts during multiple hurricanes—Harvey, Irma, and Maria—jeopardized the program’s ability to complete all required activities as planned. Specifically, the program was unable to complete follow-on operational test and evaluation (OT&E) to achieve ADE 3 and FOC by its initially planned APB dates of September 2018 and December 2018, respectively. The program was able to retain most of its initial schedule by working with its operational test agent (OTA) to adjust the follow-on OT&E plan, which significantly reduced the scope of dedicated testing needed to complete follow-on OT&E. Specifically, the OTA collected operational data during the 2017 hurricane response efforts, which allowed them to assess approximately two-thirds of the performance measures required for follow-on OT&E. In December 2018, the program updated its life-cycle cost estimate (LCCE), which is within the program’s APB cost thresholds. The program’s operations and maintenance (O&M) costs decreased in part because the program plans to transition LSCMS data storage from a physical facility to a cloud environment. The updated LCCE also estimates costs for conducting technology refreshes annually instead of every 5 years, which FEMA officials said will make the program’s future funding needs more stable as the program moves into sustainment. Federal Emergency Management Agency (FEMA) LOGISTICS SUPPLY CHAIN MANAGEMENT SYSTEM (LSCMS) The LSCMS program previously experienced significant execution challenges because of prior poor governance. FEMA initially deployed the enhanced LSCMS in 2013 without DHS leadership approval, a DOT&E letter of assessment, or a DHS-approved APB documenting the program’s costs, schedule, and performance parameters, as required by DHS’s acquisition policy. DHS’s Office of Inspector General also found that neither DHS nor FEMA leadership ensured the program office identified all mission needs before selecting a solution. In response, DHS leadership paused all LSCMS development efforts in April 2014 until the program addressed these issues, among others. FEMA subsequently completed an analysis of alternatives and developed an APB based on this assessment. DHS leadership approved the program’s initial APB in December 2015 and authorized FEMA to resume all LSCMS development and acquisition efforts in March 2016. In July 2019, FEMA reported that the program had initiated the hiring process for its vacant positions. In July 2019, FEMA officials told GAO one of the positions had already been filled. According to FEMA officials, the program revised its methodology for completing its most recent staffing profile to reflect the current and future staffing needs of the program. FEMA officials said that the current staffing levels will not change significantly after the program achieves FOC, as there will be a continued need for regular updates to the system. FEMA officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. NATIONAL BIO AND AGRO-DEFENSE FACILITY (NBAF) SCIENCE AND TECHNOLOGY DIRECTORATE (S&T) The NBAF program is constructing a state-of-the-art laboratory in Manhattan, Kansas to replace the Plum Island Animal Disease Center. The facility will enable the Department of Homeland Security (DHS) and the Department of Agriculture (USDA) to conduct research, develop vaccines, and provide enhanced diagnostic capabilities to protect against foreign animal, emerging, and zoonotic diseases that threaten the nation’s food supply, agricultural economy, and public health. Program is on track to meet May 2021 initial operational capability date. DHS and USDA have developed a transition plan and are coordinating on commissioning efforts. GAO last reported on this program in May 2018 (GAO-18-339SP). The NBAF program was originally planned be a joint operation between DHS and USDA, with DHS taking the lead on construction and operation of the facility. However, the President’s budget request for fiscal year 2019 proposed transferring operational responsibility for NBAF, which includes operational planning and future facility operations, to USDA. In the Joint Explanatory Statement for the Consolidated Appropriations Act of 2018, congressional conferees specified that DHS would retain responsibility for completing construction of NBAF. As a result, DHS will continue to oversee and manage activities required to complete construction and achieve initial operational capability (IOC), which is facility commissioning. USDA will then be responsible for achieving full operational capability (FOC), including operational stand- up of the facility and all subsequent operations. The program’s acquisition program baseline (APB) has not yet been updated to reflect the change in responsibility for achieving FOC and to remove operational costs, which will now be budgeted for by USDA. NBAF officials said the transition introduces cost and schedule risks to the program because highly integrated activities—such as commissioning and operational stand-up—are now being managed by two different agencies, but DHS and USDA will continue to coordinate through the transition process. NBAF officials told GAO that construction activities thus far—such as pouring concrete for the main laboratory—have proceeded as anticipated and the program is on track to meet its APB cost and schedule goals through IOC, planned for May 2021. According to NBAF officials, the program has already received full acquisition funding for the facility construction efforts through federal appropriations and gift funds from the state of Kansas. The program previously planned to use operations and maintenance funding to support operational stand-up activities and awarded a contract for operational planning. However, beginning in fiscal year 2019, DHS will no longer request operations and maintenance funding for NBAF, as all such funding and activities will be the responsibility of USDA. Congressional conferees noted that $42 million in funding to USDA is to address operational stand-up activities and other initial costs to operate and maintain the facility. The Consolidated Appropriations Act of 2019 also authorized DHS to transfer personnel and up to $15 million in certain funds to USDA for contracts and associated support of the operations of NBAF. Science and Technology Directorate (S&T) NATIONAL BIO AND AGRO-DEFENSE FACILITY (NBAF) NBAF officials reported that they are coordinating with USDA officials, the commissioning agent, and federal regulators responsible for awarding the registrations needed for NBAF to conduct laboratory operations to determine how the final commissioning report will be structured to support FOC and federal certification to begin laboratory operations. In June 2019, DHS and USDA signed a memorandum of agreement that established plans to transfer NBAF operational responsibility from DHS to USDA. The memorandum establishes responsibilities related to costs and funding, requirements for establishing NBAF, and considerations for interagency coordination once NBAF is operational, among other things. For example, some USDA staff will participate in the NBAF commissioning process, but they will be integrated with DHS’s onsite construction oversight team to maintain the integrity of DHS’s existing oversight approach for the NBAF construction/ commissioning contract. The memorandum of agreement also states that DHS, in consultation with USDA, will plan for the appropriate timing and necessary mechanism to transfer identified DHS employees to USDA for NBAF activities. According to NBAF officials, DHS plans to transfer staff from both the Plum Island Animal Disease Center and the program’s on-site construction oversight team to USDA to preserve institutional knowledge. USDA was appropriated $3 million in the Consolidated Appropriations Act of 2018 to begin hiring NBAF operational staff and the memorandum of agreement notes that USDA will work with DHS to increase staffing in fiscal year 2019 as required by the construction commissioning schedule. In April 2019, the program’s staffing assessment was updated to reflect program needs from fiscal year 2019 through IOC. At that time, the NBAF officials reported that the program is fully staffed. NBAF officials reviewed a draft of this assessment and provided no comments. ADVANCED TECHNOLOGY (AT) TRANSPORTATION SECURITY ADMINISTRATION (TSA) The AT Program supports the checkpoint screening capability by providing capability to detect threats in the passenger’s carry-on baggage, including explosives, weapons, and other prohibited items. The AT-1 and AT-2 X-ray systems screen carry-on baggage providing threat detection capabilities for a wide range of threats. AT-2 Tier I and Tier II systems provide enhanced detection capabilities and improved image resolution. Computed technology (CT)—which offers enhanced three-dimensional imaging and detection capabilities over the currently deployed AT system—is also being procured through AT program. Both AT and CT units have experienced challenges achieving performance goals. Procurement and deployment of CT units will transfer to Checkpoint Property Screening System program. GAO last reported on AT as a part of the Passenger Screening Program in May 2018 (GAO-18- 339SP). In February 2018, Department of Homeland Security (DHS) leadership approved transitioning existing Passenger Screening Program (PSP) projects—including AT—into stand-alone programs to better align program office staffing to capabilities and focus on mitigating capability gaps, among other things. In fiscal year 2018, TSA determined that CT is the best technology available to address rapidly evolving threats in the transportation sector. As a result, TSA determined it would leverage the AT program to initiate the acquisition of CT systems. In December 2018, DHS leadership approved an acquisition program baseline (APB) for AT as a standalone program, which included cost and schedule goals for AT and CT that were presented separately. For AT, fiscal year 2018 and prior year costs were not included in the APB cost goals because those costs are considered sunk costs for PSP. AT does not have any acquisition costs because full operational capability for AT was achieved in 2016 under PSP. AT’s operations and maintenance (O&M) costs— which total $590 million—are related to maintaining AT-1 and AT-2 X-ray systems and incorporating upgrades to enhance detection capability and increase passenger volume through AT-2 Tier I and Tier II systems. When DHS leadership approved the APB, they also approved the acquisition decision event (ADE) 3—authorizing the procurement of CT units in fiscal year 2019 only. The APB includes acquisition costs for the fiscal year 2019 procurements but it does not identify any O&M costs for CT. In March 2019, DHS leadership acknowledged the AT program’s ADE 3 for AT-2 Tier II. The program previously achieved full operational capability (FOC) for AT-2, but ADE 3 was not achieved primarily because one the program’s key performance parameters (KPP) needed to be refined. The AT program’s surplus from fiscal years 2020-2024 may be overstated in DHS’s funding plan to Congress because costs associated with CT were not previously included in the AT cost estimate. However, the AT and CT costs in the affordability assessment are combined here. The purchase of CT units will become a separate acquisition for the fiscal year 2021 programming and budget cycle with an updated cost estimate. Transportation Security Administration (TSA) ADVANCED TECHNOLOGY (AT) TSA intends to transition the procurement and deployment of CT units, among other things, to the Checkpoint Property Screening System (CPSS), which, as of August 2019, had not yet been established. CPSS is a separate acquisition program that is intended to address capability gaps in passenger screening technologies. Through CPSS, TSA plans to eventually deploy CT to all checkpoints and replace AT X-ray technology. According to TSA officials, Automated Screening Lane (ASL) technologies have been managed by the AT program since March 2019. TSA is not incurring acquisition costs for ASLs, but the source of funding for O&M costs is unclear. DHS leadership directed TSA to begin tracking ASL maintenance and repairs to inform future budget requests, among other things. TSA officials stated that one of the program’s vacant positions has not yet been funded. To mitigate the staffing gap, TSA officials stated they are disbursing tasks among existing staff until the position is filled. TSA officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. CREDENTIAL AUTHENTICATION TECHNOLOGY (CAT) TRANSPORTATION SECURITY ADMINISTRATION (TSA) The CAT system is used to verify and validate passenger travel and identification documents prior to entering secure areas in airports. CAT reads data and security features embedded in identification documentation (ID), verifies security features are correct, and displays authentication results to the operator. The CAT system also verifies the passenger has the appropriate flight reservation to progress through security screening and enter the secure area, among other things. Program met its key performance parameters, but needs to address cyber resiliency and other issues. CAT system will require regular updates to address changes to state identification documentation. GAO last reported on CAT as part of the Passenger Screening Program in May 2018. (GAO-18-339SP). In February 2018, the Department of Homeland Security (DHS) approved transitioning existing Passenger Screening Program (PSP) projects, including CAT, into stand-alone programs to better align program office staffing to capabilities and focus on mitigating capability gaps, among other things. In December 2018, DHS leadership approved an acquisition program baseline (APB) for CAT as a stand-alone program. The APB reflected a revised testing and deployment strategy. Specifically, TSA no longer intends to pursue separate deployments of CAT for TSA Pre® and standard lanes. TSA concluded that the separate approach would extend the overall schedule to deploy CAT units to the field and was an inefficient use of resources. In February 2019, DHS leadership granted the program acquisition decision event (ADE) 3 for procurement and deployment of CAT units and acknowledged the program’s initial operational capability (IOC) based on the fielded units. TSA now plans to achieve full operational capability (FOC) in September 2022—more than 1 year earlier than previously planned for standard lanes, but 8 years later than initially planned under PSP. According to TSA officials, the program recently accelerated its deployment schedule to meet existing and emerging threats. The program developed an initial life-cycle cost estimate (LCCE) to inform the APB and ADE 3 and updated the estimate in June 2019 to inform the budget process. The program’s June 2019 LCCE reflects an O&M cost decrease of over $80 million, which TSA officials attribute to a reduction in enhancements needed to accelerate deployments. The program was not included in DHS’s funding plan to Congress for fiscal years 2020-2024 because the program is no longer expected to receive acquisition funding. TSA officials stated that they are working with TSA’s Chief Financial Officer and the CAT vendor to identify and mitigate any funding issues that may arise as the program moves into production. Transportation Security Administration (TSA) CREDENTIAL AUTHENTICATION TECHNOLOGY (CAT) TSA officials stated that CAT is expected to be TSA’s primary identification verification method by the end of fiscal year 2019. However, TSA officials said the CAT system will require regular updates to address changes to state IDs. In November 2018, TSA officials reported that states are in the process of adopting new requirements identified in the REAL ID Act of 2005. Among other things, the Act establishes minimum security standards for ID issuance and production, and prohibits federal agencies from accepting IDs from states not meeting these standards unless the Secretary of Homeland Security has granted the issuing state an extension of time to meet the requirements. TSA officials said that the current manual process of verifying a passenger’s ID against their boarding pass will be used if CAT units are unavailable and between system updates. In May 2019, the program reported two critical staffing vacancies. TSA officials reported that these positions have been filled. TSA officials reviewed a draft of this assessment and provided no comments. ELECTRONIC BAGGAGE SCREENING PROGRAM (EBSP) TRANSPORTATION SECURITY ADMINISTRATION (TSA) Established in response to the terrorist attacks of September 11, 2001, EBSP tests, procures, and deploys transportation security equipment, such as explosives trace detectors and explosives detection systems, across approximately 440 U.S. airports to ensure 100 percent of checked baggage is screened for explosives. EBSP is primarily focused on delivering new systems with enhanced screening capabilities and developing software upgrades for existing systems. Follow-on testing completed in January 2019; initial results show improvement in effectiveness. EBSP is pursuing a new procurement strategy for two types of detection systems. GAO last reported on this program in May 2018 (GAO-18-339SP). In August 2019, TSA declared a cost breach of EBSP’s current acquisition program baseline (APB) due to increased maintenance costs. The program previously revised its APB in May 2016 to account for budget reductions and to implement the program’s strategy to prioritize funding to extend the life of screening technologies, among other things. TSA has implemented these changes through ongoing maintenance and system upgrades, to include detection algorithm updates. DHS officials reported that this strategy has improved security effectiveness and operational efficiencies at a lower cost than replacing legacy systems with new systems. However, this approach increased the number of systems that are out-of-warranty and increased the maintenance needed to sustain these systems. This new strategy, coupled with increased maintenance activities, resulted in an operations and maintenance (O&M) cost increase exceeding the program’s APB O&M cost threshold. As of September 2019, the program’s revised APB, which TSA officials said will address the O&M cost increase, had not yet been approved. In January 2018, DHS leadership approved the program’s request to deploy an explosives detection system with an advanced threat detection algorithm. TSA officials reported that they achieved initial operational capability (IOC) of these systems in February 2018; this is the program’s final APB milestone. TSA leadership subsequently approved the program to deploy detection algorithm updates to fielded systems. Based on the program’s July 2019 life-cycle cost estimate (LCCE), the program is projected to face an acquisition funding gap of $29 million over the 5-year period. However, the program’s total projected funding gap, including O&M, is expected to be approximately $223 million. TSA officials told GAO that one of their primary challenges is funding, and that to mitigate anticipated funding gaps, the program may shift other projects from one fiscal year to another or cancel them altogether. Transportation Security Administration (TSA) ELECTRONIC BAGGAGE SCREENING PROGRAM (EBSP) As of July 2019, EBSP has 1,678 explosives detection systems and 2,477 explosives trace detectors deployed nationwide. In February 2018, DHS leadership approved the program’s updated acquisition plan, which reflects a new procurement strategy. Under the new procurement strategy, the program will transition from procuring systems with different sizes and speeds to two types: (1) inline systems that integrate with a baggage handling system and are linked through a network, and (2) stand-alone systems that may be integrated with a baggage handling system, but not linked to a network. In addition, TSA officials reported that the new strategy reflects updates to EBSP’s vendor qualification process, which is intended to improve collaboration with vendors so they can develop more technically mature systems. In March 2018, DHS leadership approved a pilot effort in which TSA’s Chief Acquisition Executive (CAE) provides oversight of changes to deployed systems, including algorithm updates. According to TSA officials, this process is intended to limit some steps in the formal oversight process so capabilities can be deployed more rapidly. DHS leadership plans to assess this pilot process to determine its effectiveness. In May 2019, the program reported that the five vacant positions impact the program’s performance and execution schedules at times. To mitigate the staffing gap, program officials said that current staff are temporarily assuming additional duties. TSA officials stated that issues identified in DOT&E assessments were corrected, and that follow-on test activities were conducted and resulted in favorable evaluations and capability deployment. TSA officials also provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. TECHNOLOGY INFRASTRUCTURE MODERNIZATION (TIM) TRANSPORTATION SECURITY ADMINISTRATION (TSA) The TIM program was initiated to address shortfalls in TSA’s threat assessment screening and vetting functions by providing a modern end-to-end credentialing system. The TIM system will manage credential applications and the review process for millions of transportation workers and travelers by supporting screening and vetting for Transportation Worker Identification Credential (TWIC) and TSA Pre®. Program achieved full operational capability for TWIC and TSA Pre® capabilities. Program met its four key performance parameters. GAO last reported on this program in May 2018 and October 2017 (GAO-18-339SP, GAO-18-46). In November 2018, Department of Homeland Security (DHS) leadership approved the TIM program’s request to descope and change its definition of full operational capability (FOC) to include only the TWIC and TSA Pre® capabilities. By the time TIM had fully delivered capabilities for TWIC and TSA Pre®, TSA had made ongoing updates and improvements to the remaining legacy vetting and credentialing systems to meet security and mission demands, which had also sufficiently met end user needs. According to TSA officials, any additional system development would produce redundant functionality. Going forward, the program plans to continue to modernize the legacy systems and to achieve additional efficiencies. The program updated its key acquisition documents, including its acquisition program baseline (APB) and life-cycle cost estimate (LCCE) to reflect the change in scope. In July 2019, DHS leadership approved program’s revised APB. DHS leadership granted the program acquisition decision event (ADE) 3 and acknowledged the program’s achievement of FOC—fulfilling TSA Pre® and TWIC mission needs for vetting and credentialing—in August 2019. DHS leadership previously approved a revised APB for the TIM program in September 2016. Prior to the approval of the program’s 2016 APB, DHS leadership paused new development for 22 months after the program breached its APB goals for various reasons including technical challenges. In July 2019, DHS headquarters conducted an independent cost assessment to inform ADE 3, which TSA adopted as the program’s LCCE. The revised LCCE reflected the program’s reduced scope. The program’s APB acquisition cost goal decreased by nearly $220 million from the program’s 2016 APB. The reduction in costs is primarily attributed to the reduction in the program’s scope. However, the program’s operations and maintenance APB cost goals increased by $205 million primarily due to maintenance of legacy systems to address user needs. Transportation Security Administration (TSA) TECHNOLOGY INFRASTRUCTURE MODERNIZATION (TIM) In October 2017, GAO found that TSA had not fully implemented several leading practices to ensure successful agile adoption. GAO also found that TSA and DHS needed to conduct more effective oversight of the TIM program to reduce the risk of repeating past mistakes. DHS concurred with all 14 GAO recommendations to improve program execution and oversight, and identified actions DHS and TSA can take to address them. As of September 2019, TSA addressed all but one recommendation— to ensure DHS leadership reached consensus on, documented, and implemented oversight and governance changes for agile program reviews. TSA reported a critical staffing gap of four FTEs in 2019, including a manager position to adapt initiatives to agile business and development processes. TSA officials stated that the staffing gap has had minimal impact on program execution. To mitigate the gap, the program is leveraging support from contractors and matrixed staff. TSA officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. FAST RESPONSE CUTTER (FRC) UNITED STATES COAST GUARD (USCG) The USCG uses the FRC to conduct search and rescue, migrant and drug interdiction, and other law enforcement missions. The FRC carries one cutter boat on board and is able to conduct operations in moderate sea conditions. The FRC replaces the USCG’s Island Class patrol boat and provides improved fuel capacity, surveillance, and communications interoperability with other Department of Homeland Security (DHS) and Department of Defense assets. FRC found operationally effective and suitable, and all key performance parameters validated. Defect in ship structure found, requiring changes in production and retrofits to cutters already delivered. GAO last reported on this program in May 2018 and March 2017 (GAO-18-339SP, GAO-17-218). The FRC program is on track to meet its current cost and schedule goals. USCG officials told GAO the program is revising its acquisition program baseline (APB) in 2019 to reflect an increase in FRCs. The USCG previously planned to acquire 58 FRCs and, as of August 2019, 35 had been delivered and another 21 were on contract. However, in fiscal years 2018 and 2019, congressional conferees supported funds for the acquisition of 4 additional FRCs to begin replacing 6 cutters currently operating in the Middle East. To account for the increase of up to 6 additional FRCs, USCG officials stated that they are revising the program’s acquisition documents and anticipate completing these updates by the end of calendar year 2019. To inform the budget process, the program updated its life-cycle cost estimate in June 2019 to reflect the additional 4 cutters that have been funded. The updated estimate remains within the program’s current APB cost thresholds. USCG officials stated that the contractor—Bollinger Shipyards LLC—is meeting the program’s current delivery schedule and the program is on track to achieve full operational capability (FOC) for the original 58 cutters by March 2027, as planned. However, the program’s FOC date will likely be extended to account for the delivery of the additional cutters in the revised APB. The program’s initial operational capability (IOC) date previously slipped due to a bid protest related to the program’s initial contract award—now known as the phase 1 contract—and the need for structural modifications. USCG officials attributed a subsequent 5-year slip in the program’s FOC date to a decrease in annual procurement quantities under the phase 1 contract. In May 2014, the USCG determined that it would procure only 32 of the 58 FRCs through this contract and initiated efforts to conduct full and open competition for the remaining 26 vessels— known as phase 2. In May 2016, the USCG awarded the phase 2 contract to Bollinger Shipyards LLC for the remaining 26 FRCs. Under the phase 2 contract, the USCG can procure 4 to 6 FRCs per option period. For fiscal year 2019, the USCG reported that it exercised an option for 6 FRCs. According to USCG officials, the phase 2 contract will need to be modified to increase the total quantity allowed under the current contract and account for the additional FRCs, but as of July 2019 the modifications had not been made. United States Coast Guard (USCG) FAST RESPONSE CUTTER (FRC) The USCG continues to work with Bollinger Shipyards LLC to address issues covered by the warranty and acceptance clauses for each ship. For example, in the fall of 2017, USCG officials reported identifying a latent defect that would affect the FRC’s ability to achieve its intended 25-year structural fatigue life. USCG officials said cracks were found in the interior steel structure of two FRCs, prompting a class-wide inspection. Upon further analysis, the USCG determined that the fatigue issues were due to faulty design assumptions and identified 12 areas of structural weakness that will require reinforcements to the ship’s interior steel structure. In response, USCG officials stated that the contractor developed corrective actions—ranging in complexity from adding bracket supports to removing and replacing large sections of steel—that have been approved by the USCG. USCG officials further stated that corrections are being incorporated during production, but FRCs that have already been delivered will need to be retrofitted during regular maintenance periods, scheduled through 2025. These officials added that these defects do not affect current operations. In addition, the contractor is undertaking retrofits for nine of the 10 engine issues covered by the warranty that are affecting the fleet—such as leaking exhaust pipes—and a prototype solution for the remaining issue is being assessed. As of June 2019, USCG officials reported the FRC’s warranty has resulted in $123 million in cost avoidance. In July 2019, USCG officials stated they had filled the one critical staffing gap and were in the process of hiring staff to address the remaining staffing gaps. USCG officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. H-65 CONVERSION/SUSTAINMENT PROGRAM (H-65) UNITED STATES COAST GUARD (USCG) The H-65 aircraft is a short-range helicopter that the USCG uses to fulfill its missions, including search and rescue, ports and waterways security, marine safety, and defense readiness. The H-65 acquisition program consists of eight discrete segments that incrementally modernize the H-65 aircraft fleet. The program is currently focused on the service life extension program (SLEP) and upgrades to the automatic flight control system (AFCS) and avionics. H-65 aircraft failed to meet two key performance parameters in testing; has not yet tested cyber resiliency. Program to synchronize upgrades into scheduled maintenance periods. GAO last reported on this program in May 2018 (GAO-18-339SP). In March 2018, Department of Homeland Security (DHS) leadership approved the program’s revised acquisition program baseline (APB), removing it from breach status, which USCG officials primarily attributed to underestimating the technical effort necessary to meet requirements. DHS leadership also granted the program approval for ADE 2C for low-rate initial production of the avionics and AFCS upgrades and ADE 2B for the addition of a SLEP. The SLEP is expected to extend the flight hour service life of each aircraft from 20,000 flight hours to 30,000 flight hours by replacing obsolete aircraft components. USCG officials stated the USCG plans to operate the H-65 aircraft until 2039 so that the USCG can prioritize funding for the Offshore Patrol Cutter. The USCG also plans to align its next helicopter acquisition effort with the Department of Defense’s future vertical lift acquisition plans. The program’s current APB reflects the restructured program schedule which synchronizes the SLEP with the avionics and AFCS upgrades. Specifically, the new program structure calls for completing the SLEP and upgrades to AFCS and avionics during the same scheduled maintenance period. This structure allows the USCG to leverage accessibility of components the program intends to replace as part of the SLEP while the aircraft is being assembled to accommodate the avionics and AFCS upgrades. As a result, USCG officials reported that the program will avoid some labor costs and will reduce the risk of damaging AFCS and avionics components which would need to be removed during the SLEP. In its current APB the program’s full operational capability (FOC) date was extended by nearly 2 years to September 2024, primarily to incorporate the SLEP. The program’s total life-cycle cost threshold decreased by approximately $200 million from its March 2014 APB, which USCG officials attributed to decreased labor costs, among other things. USCG officials told GAO they were in the process of updating the program’s key acquisition documents to inform the program’s ADE 3 decisions for full rate production of the avionics and AFCS upgrades and the SLEP. In July 2019, USCG officials said they do not plan to update the program’s APB for the upcoming ADEs because the program is on track and does not require changes to its cost, schedule, or performance goals. United States Coast Guard (USCG) H-65 CONVERSION/SUSTAINMENT PROGRAM (H-65) The USCG awarded contracts to Rockwell Collins—the original equipment manufacturer of the legacy AFCS and avionics—for continued development of the AFCS and avionics upgrades in July 2016 and March 2017, respectively. USCG officials said they expect delivery of the upgrades to the fleet in May 2020. USCG officials said there is risk involved with extending the aircrafts’ service life beyond 20,000 flight hours since it has never been done by other agencies that operate the H-65. USCG officials stated that the aircraft manufacturer, Airbus, assisted the USCG’s chief aeronautical engineer in identifying parts that need replacement. As part of the program’s revised acquisition strategy, the USCG plans to synchronize the SLEP with the avionics and AFCS upgrades and conduct this work during the programmed depot maintenance cycles in fiscal years 2020 through 2024. USCG officials reported that this strategy allows the program to leverage the engineering and program management contractors already in place and ensures SLEP component availability before production support from Airbus ends in 2018. In April 2019, the USCG reported the program had one critical staffing gap—a deputy program manager. USCG officials reported the program filled the position in August 2019. USCG officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. LONG RANGE SURVEILLANCE AIRCRAFT (HC-130H/J) UNITED STATES COAST GUARD (USCG) The USCG uses HC-130H and HC-130J aircraft to conduct search and rescue missions, transport cargo and personnel, support law enforcement, and execute other operations. Both aircraft are quad-engine propeller-driven platforms. The HC-130J is a modernized version of the HC-130H, which has advanced engines, propellers, and equipment that provide enhanced speed, altitude, range, and surveillance capabilities. Design of new mission system processor is complete; USCG officials reported all key performance parameters met. Transfer of surplus HC-130H aircraft to other agencies delayed. GAO last reported on this program in May 2018 (GAO-18-339SP). As of July 2019, the USCG has yet to complete a more than 4-year effort to revise the acquisition program baseline (APB)—to account for significant program changes. Specifically, the USCG decided to pursue an all HC-130J fleet and, in fiscal year 2014, Congress directed the transfer of 7 HC-130H aircraft to the U.S. Air Force. The USCG was in the process of upgrading these aircraft but canceled further HC-130H upgrades. In September 2017, Department of Homeland Security (DHS) leadership directed the USCG to submit the revised APB by January 2018. As of July 2019, USCG officials had revised key acquisition documents such as the program’s life-cycle cost estimate (LCCE) and operational requirements document (ORD)—which will inform the program’s revised APB—but USCG officials told GAO the APB is not expected to be approved until August 2019. USCG officials said the re-baseline has been delayed, in part, because Congress directed the USCG to conduct a multi-phased analysis of its mission needs. In November 2016, the USCG submitted the results of its analysis for fixed-wing aircraft, which confirmed the planned total quantity of 22 HC-130J aircraft and an annual flight-hour goal of 800 hours per aircraft. The results of the analysis are reflected in the program’s revised LCCE, which DHS approved in June 2019. However, the USGC plans to decommission the HC-130H fleet by the end of fiscal year 2022, which may result in a capability gap since the program’s revised LCCE indicates that the fleet will consist of only 14 HC-130J aircraft in fiscal year 2022. In addition, the program’s revised ORD includes a full operational capability (FOC) date—when all 22 aircraft are operational and assigned to USCG air stations—of September 2033. The revised FOC date is more than 6 years beyond the program’s current threshold date of March 2027. GAO previously reported that the program was at risk of not meeting its previously planned FOC date because the USCG had not requested adequate funding. The program’s revised LCCE acquisition costs decreased in part because costs associated with the initially planned HC-130H improvements were removed. However, the program’s operations and maintenance costs increased by over $800 million over the program’s previous estimate, which is primarily attributed to a 13-year increase in the life expectancy of the HC-130J aircraft. United States Coast Guard (USCG) LONG RANGE SURVEILLANCE AIRCRAFT (HC-130H/J) In December 2013, Congress directed the transfer of seven HC-130H aircraft to the U.S. Air Force for modifications—which consist of upgrades and installing a fire retardant delivery system—and subsequent transfer to the U.S. Forest Service. This direction factored into the USCG’s decision to pursue an all HC-130J fleet. However in August 2018, Congress directed that the U.S. Air Force transfer the modified aircraft to the state of California, Natural Resources Agency, for use by the Department of Forestry and Fire Protection. USCG officials reported seven aircraft will be transferred to the state of California, Natural Resources Agency, and the USCG does not plan to retain the surplus aircraft. As of July 2019, no HC-130H aircraft have been transferred. The USCG plans to procure a total of 22 HC-130Js. In July 2019, USCG officials reported 13 HC-130J aircraft had been delivered and USCG had awarded contracts for three more. At that time, the USCG also had 14 HC-130Hs in its inventory. The USCG planned to remove four of the HC-130Hs from service in 2019 as HC-130Js and C-27Js are delivered. USCG officials said the program is not experiencing any workforce issues as a result of its staffing gap. The program filled the one critical vacancy in August 2019 and is in the process of hiring staff to fill an additional vacancy. USCG officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. MEDIUM RANGE SURVEILLANCE AIRCRAFT (HC-144A/ C-27J) UNITED STATES COAST GUARD (USCG) The USCG uses HC-144A and C-27J aircraft to conduct all types of missions, including search and rescue and disaster response. All 32 aircraft—18 HC-144A aircraft and 14 C-27J aircraft—are twin-engine propeller driven platforms. The interior of both aircraft are able to be reconfigured to accommodate cargo, personnel, or medical transports. New mission system processor installed on five HC-144A aircraft. Program challenges related to purchasing spare parts and accessing technical data are improving. GAO last reported on this program in May 2018 (GAO-18- 339SP). In April 2019, Department of Homeland Security (DHS) leadership approved a change to the program’s current acquisition program baseline (APB) to adjust the program’s schedule milestones as a result of the fiscal year 2019 partial government shutdown. USCG officials told GAO that delays in funding limited contracted work for the program during the shutdown. USCG officials stated that the program could not recover from the lost time and, in response, DHS leadership authorized the program’s request for a 3-month extension on the program’s future APB milestones. The current APB was approved in August 2016 to reflect the restructuring of the HC-144A acquisition program. The USCG initially planned to procure a total of 36 HC-144A aircraft, but reduced that number to the 18 it had already procured after Congress directed the transfer of 14 C-27J aircraft from the U.S. Air Force to the USCG in fiscal year 2014. The program’s APB divides the program into two phases. Phase 1 includes acceptance of the 18 HC-144A aircraft and upgrades to the aircraft’s mission and flight management systems. Phase 2 includes acceptance of and modifications to the C-27J aircraft to meet the USCG’s mission needs. In July 2019, USCG officials said that the program had completed upgrades on five HC-144A aircraft and plans to complete upgrades on all HC-144As by September 2021. For phase 2, the USCG has accepted all 14 C-27Js from the U.S. Air Force and plans to complete the modification of these aircraft by June 2025 to achieve full operational capability (FOC). To inform the budget process, in June 2019 the program updated its life-cycle cost estimate (LCCE), which is within its current APB cost thresholds. The program’s total life-cycle cost decreased by approximately $115 million. USCG officials attribute the decrease to refinement of the cost estimate based on actual costs, changes to the schedule for the mission system upgrades, and a delay in operating missionized C-27Js—which reduces the total estimated aircraft flight hours—among other things. USCG officials said that they plan to delay operation of missionized C-27Js to ensure adequate logistics support is available for the aircraft. In addition, congressional conferees supported $18 million in fiscal year 2018 for the USCG to purchase a flight simulator for training purposes. According to USCG officials, prioritizing the procurement of the flight simulator in fiscal year 2018 addressed C-27J training needs and provided over $15 million in cost savings for the program. United States Coast Guard (USCG) MEDIUM RANGE SURVEILLANCE AIRCRAFT (HC-144A/C-27J) GAO previously found that the program faced challenges purchasing spare parts and accessing technical data for the C-27J, which was affecting the USCG’s ability to transition the aircraft into the fleet. USCG officials told GAO that these issues are improving. Specifically, they stated that program awarded two contracts for spare parts to third-party suppliers in early 2018 and purchased spare parts in bulk in 2017 to maintain the fleet. In July 2019, USCG officials said the program has been able to stock sites well enough to keep assets available for use, and will continue to work with the contractors to address the issue. USCG officials said that a contract was awarded to the original equipment manufacturer in April 2017 that allows the USCG appropriate rights to the technical data. Also, in August 2019, USCG officials told GAO they received all C-27J technical data in the Air Force’s possession, including operations and maintenance manuals, as part of the transfer of 14 C-27J aircraft from the Air Force to the Coast Guard. USCG officials told us that the program updated its acquisition plan in February 2018 to incorporate the procurement of a new full-motion flight simulator training device for the C-27J aircraft. The USCG received funding to purchase a flight simulator in fiscal year 2018 and plans to begin instructor training on the device in August 2019. In July 2019, USCG officials told GAO that the program’s staffing is not negatively impacting program execution. USCG officials explained that they have filled four of the program’s reported staffing vacancies and plan to fill the remaining position soon. USCG officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. NATIONAL SECURITY CUTTER (NSC) UNITED STATES COAST GUARD (USCG) The USCG uses the NSC to conduct search and rescue, migrant and drug interdiction, environmental protection, and other missions. The NSC replaces and provides improved capabilities over the USCG’s High Endurance Cutters. The NSC carries helicopters and cutter boats, provides an extended on-scene presence at forward deployed locations, and operates worldwide. Follow-on operational testing was completed in 2018, but unmanned aerial surveillance aircraft testing was delayed. The USCG continues to address issues identified with the NSC propulsion system. GAO last reported on this program in May 2018 and April 2017 (GAO-18-339SP, GAO-17- 218). In November 2017, Department of Homeland Security (DHS) leadership approved a revised acquisition program baseline (APB), which accounted for the addition of a ninth NSC to the program of record. The USCG originally planned to acquire eight NSCs; however, in fiscal year 2016 Congress appropriated funds specifically for the production of a ninth NSC. Congressional conferees subsequently included in fiscal year 2018 $540 million and $635 million to be immediately available and allotted to contract for production of a 10th NSC and purchase of long lead time materials and production of an 11th NSC, respectively. According to USCG officials, the USCG awarded a contract to produce the ninth NSC in December 2016 and awarded a production contract for the 10th and 11th NSCs in December 2018. As of August 2019, eight NSCs have been delivered and the remaining three NSCs are under contract for production. USCG officials reported that the program is currently on track to meet its current APB schedule and anticipate delivery of the ninth NSC in September 2020. However, the program’s full operational capability (FOC) date is expected to be extended until 2024 as a result of the anticipated delivery of the 11th NSC in January 2024. According to USCG officials, the program’s acquisition documentation, including the APB, is being revised to reflect the additional NSCs and these updates are expected to be complete by July 2020. To inform the budget process, the program updated its LCCE to include the 10th and 11th NSCs. As a result, the program’s life-cycle costs exceed the current APB thresholds. Despite this cost growth, the program’s total life-cycle cost is still less than the program’s initial estimate for eight ships. USCG officials attribute the overall decrease to more accurate estimates and reduced operations and maintenance (O&M) costs. The program’s current APB cost thresholds already reflect cost growth that occurred earlier in the program, when the program implemented several design changes to address equipment issues. As of September 2017, 12 equipment systems had design changes, which USCG estimated cost over $260 million. This work includes structural enhancements on the first two NSCs and the replacement of the gantry crane, which aids in the deployment of cutter boats. United States Coast Guard (USCG) NATIONAL SECURITY CUTTER (NSC) According to program officials, the USCG relies on the Navy to request funding for and provide certain systems on the NSC such as the Close In Weapon System, which includes a radar-guided gun used to protect against anti-ship cruise missiles. USCG officials reported that some of these Navy systems may not be available in time to support the production of the ninth, 10th and 11th NSCs, since these cutters were unplanned additions to the NSC program and the Navy had not included funding for some of these systems in its budget requests. According to program officials, they are working with the Navy to identify options to mitigate this issue. Officials stated that an option being considered is constructing the NSCs with space available for the Navy equipment to be installed after delivery. USCG officials said the program’s staffing vacancies had not negatively affected program execution and, as of September 2019, all three vacancies had been filled. The program’s staffing profile represents staffing requirements through NSC 11, and USCG officials reported that the program office would need to reassess future staffing requirements if the USCG acquires additional NSCs. USCG officials stated that with the exception of small unmanned aerial surveillance aircraft, follow-on OT&E testing is completed. Additional testing are planned in fiscal year 2020. A comprehensive update of the program’s LCCE is being drafted to reflect costs of the 10th and 11th NSC. The program will base the cost goals of the next revision to the APB on this update. The next revision of the APB will include a revised FOC date based on delivery of the 11th NSC in January 2024. USCG officials also provided technical comments on a draft assessment, which GAO incorporated as appropriate. OFFSHORE PATROL CUTTER (OPC) UNITED STATES COAST GUARD (USCG) The USCG plans to use the OPC to conduct patrols for homeland security, law enforcement, and search and rescue operations. The OPC is being designed for long-distance transit, extended on-scene presence, and operations with deployable aircraft and small boats. It is intended to replace the USCG’s aging Medium Endurance Cutters (MEC) and bridge the operational capabilities provided by the Fast Response Cutters and National Security Cutters. Shipyard sustained damage in Hurricane Michael, expected to result in program cost and schedule changes. USCG assessing the effects from hurricane and plans to identify a path forward in early fiscal year 2020. GAO last reported on this program in May and July 2018 (GAO-18-339SP, GAO-18-629T). In May 2018, the Department of Homeland Security (DHS) approved a revised life-cycle cost estimate (LCCE) for the OPC program, which officials said reflects a refinement of the OPC design and planned systems—including a weight increase of 27 percent—and the incorporation of actual contract data, among other things. The USCG is not reporting a cost increase because the amount of OPC acquisition costs that the program plans to fund, approximately $10.3 billion, remains within the program’s acquisition program baseline (APB) cost thresholds. However, the revised LCCE included a shift of some costs that were previously planned to be funded by the program to other sources, such as other parts of the USCG or the U.S. Navy. This government-furnished equipment, which is now estimated to cost nearly $2 billion, will largely be funded by the U.S. Navy, according to USCG officials. Overall, the total program acquisition costs increased by approximately $1.7 billion from the previous estimate. In October 2018, the shipbuilder, Eastern Shipbuilding Group, suffered damage as a result of Hurricane Michael. The shipbuilder reported to the USCG in May 2019 that it can no longer afford the estimated costs associated with the OPC contract without assistance from the government. In January 2019, the shipbuilder resumed construction of the lead ship, but the damages sustained have resulted in a long-term degradation of their ability to produce the OPCs at the previously estimated cost and schedule. The shipbuilder has projected hundreds of millions of dollars in increased contract costs—which it attributes to anticipated skilled labor shortages and a loss of production efficiencies—and a 9- to 12-month delivery delay for each of the first nine ships. Despite these anticipated cost increases and schedule delays, as of July 2019, USCG officials said they had not formally notified DHS leadership of a potential cost or schedule breach because they are continuing to assess how to move forward. DHS leadership granted the program a 3-month extension to achieve its acquisition decision event (ADE) 2C in December 2019 to mitigate impacts from the fiscal year 2019 partial government shutdown. USCG officials said they are preparing for the ADE 2C, but also are using the additional time to assess the shipbuilder’s report, analyze estimates, and determine a path forward by early fiscal year 2020. United States Coast Guard (USCG) OFFSHORE PATROL CUTTER (OPC) According to USCG program officials, they have established a team with representatives from DHS, USCG, and the U.S. Navy to assess the impact of Hurricane Michael and determine a way forward. As part of its assessment, these officials said they are evaluating a number of options, including modifications to the original contract. Regardless of the path forward, USCG officials stated the program will likely need congressional approval of the contracting strategy and financial resources necessary to execute the new plan. USCG officials stated that DHS leadership will review the program’s status and determine whether to authorize the construction of OPC 2 and the purchase of initial materials needed for OPC 3 at the program’s ADE 2C. USCG officials stated that they anticipate the exercise of a contract option for the construction of OPC 2 and the materials for OPC 3 will be delayed as the program and shipbuilder continue to assess the impact of the hurricane on OPC production. The OPC program is continuing to increase staffing as the program matures and production activities increase. In July 2019, USCG officials said the program has a staffing gap of five FTEs, none of which are critical. Officials said they were in the process of hiring staff to fill these positions. USCG officials provided technical comments on a draft of this assessment, which GAO incorporated as appropriate. POLAR SECURITY CUTTER (PSC) UNITED STATES COAST GUARD (USCG) The PSC program—formerly designated as the Heavy Polar Icebreaker—is intended to assist the USCG in maintaining access to Arctic and Antarctic polar regions. The USCG requires its icebreaking fleet to conduct multiple missions, including defense readiness; marine environmental protection; ports, waterway, and coastal security; and search and rescue. The USCG plans to acquire three PSCs to recapitalize its heavy polar icebreaker fleet, which currently consists of one operational ship. DHS identified three critical technologies in its June 2019 technology readiness assessment of the program. Program awarded a $750 million detail design and construction contract to VT Halter Marine in April 2019. GAO last reported on this program in May and September 2018 (GAO-18-339SP, GAO-18- 600). In January 2018, Department of Homeland Security (DHS) leadership approved the program’s initial acquisition program baseline (APB), establishing cost, schedule, and performance goals. The program achieved a combined acquisition decision event (ADE) 2A/2B in February 2018, which authorized the initiation of development efforts. However, in September 2018, GAO found that the program’s schedule and cost estimates are optimistic. Specifically, GAO found that the program’s planned delivery dates are not informed by a realistic assessment of shipbuilding activities. Instead, the schedule is driven by the potential gap in icebreaking capabilities once the USCG’s only operational heavy polar icebreaker reaches the end of its service life. As a result, the program is at risk of experiencing schedule delays. Similarly, GAO found that the program’s life-cycle cost estimate (LCCE) adheres to most cost estimating best practices but is not fully reliable. This was due, in part, to the cost estimate not quantifying the range of possible costs over the entire life of the program. As a result, the program is at risk of costing more than estimated. In April 2019, the program awarded a $746 million contract to VT Halter Marine for the detail design and construction of the lead PSC. According to USCG officials, the program is revising both the program schedule and cost estimate with information from the shipbuilder. For example, delivery of the lead ship in the awarded contract is anticipated in May 2024—2 months after the program’s APB threshold date. In addition, the program updated its LCCE in June 2019 to inform the budget process, but this estimate does not reflect cost changes as a result of the contract award. USCG officials acknowledged the schedule and cost risks identified by GAO and plan to address these risks as part of the acquisition documentation updates. From 2013 through 2019, the program received $1.035 billion in funding—$735 million in USCG appropriations and $300 million in Navy appropriations. USCG officials stated that the lead ship is fully funded but any funding gaps in the future may result in delays to delivery of the two follow-on ships. United States Coast Guard (USCG) POLAR SECURITY CUTTER (PSC) The USCG established an integrated program office and ship design team with the Navy and, in 2017, DHS, the USCG, and the Navy entered into several agreements that outline major roles and responsibilities, including the Navy’s role in contracting on behalf of the Coast Guard. The ship design team provided technical oversight for the development of the PSC’s concept designs, which the USCG used to inform the ship’s specifications and program’s life-cycle cost estimate. According to USCG officials, as of July 2019, the USCG and the Navy established a project residence office of three staff at the shipbuilder’s facility in Pascagoula, Mississippi to provide oversight of shipbuilding efforts. In April 2019, USCG reported that it is increasing the required staffing level for the program as it matures, with 5 FTEs added in fiscal year 2019. According to program officials, as of July 2019, three of these five vacancies—including the commanding officer and executive officer of the project resident office—have been filled. USCG officials said the remaining positions were being addressed by active duty USCG staff and through the civilian hiring process. In September 2018, GAO made six recommendations to DHS, the USCG, and the Navy to address risks GAO identified with the PSC program. As of August 2019, three of the six recommendations remain open. USCG officials stated that the PSC program awarded a contract for the detail design and construction of up to three cutters to VT Halter Marine in April 2019—ahead of schedule. USCG officials added that the program has either addressed or is in the process of addressing all of GAO’s recommendations contained in GAO-18-600, including an update to the schedule and cost estimate to reflect the award to VT Halter Marine. USCG officials also provided technical comments on a draft assessment, which GAO incorporated as appropriate. UNITED STATES CITIZENSHIP AND IMMIGRATION SERVICES (USCIS) The Transformation program was established in 2006 to transition USCIS from a fragmented, paper-based filing environment to a consolidated, paperless environment for electronically processing immigration and citizenship applications. The program is delivering system capability through releases that either deploy electronic, web-based application forms or improve system functionality. Program revised key performance parameters to reflect the program’s new baseline. Program reorganized to leverage USCIS expertise and focus on system functionality. GAO last reported on this program in May 2018 and July 2016 (GAO-18-339SP, GAO-16- 467). In June 2018, Department of Homeland Security (DHS) leadership approved Transformation’s revised acquisition program baseline (APB) and subsequently removed the program from breach status—lifting a strategic pause that had limited new program development for 18 months. The program experienced a schedule breach in September 2016 when it failed to upgrade to USCIS’s application processing information system to include applications for naturalization. The new baseline modified the program’s cost, schedule, and performance parameters and reflects changes to the way the program delivers capabilities and a new acquisition strategy. Specifically, the new APB revised the scope of the Transformation program to focus on improving functionality—such as application processing time. Under the prior strategy, the program was focused on adding new applications or forms—from four separate lines of business—to the upgraded processing system. The program plans to complete major development work in September 2019 and achieve full operational capability (FOC) in March 2020. Despite the 18-month pause in development, the program’s FOC dates slipped only 1 year from its previously revised APB. In August 2019, USCIS officials reported that the program is on track to meet its revised schedule goals. In its revised APB, the program’s acquisition cost threshold decreased from its previous APB by approximately $200 million primarily because the program shifted costs to operations and maintenance (O&M) to align with DHS’s new common appropriations structure. As a result of this shift in costs and because the new APB extended the program’s life cycle by 2 years, O&M costs increased by nearly $800 million from the program’s previous APB. In June 2019, the program updated its LCCE again to inform the budget process, which is within its APB cost thresholds. In September 2016, the Transformation program breached its schedule baseline when persistent system deficiencies forced the program to revert 84,000 monthly applications for naturalization forms from an upgraded application information system to a legacy platform. USCIS officials said the program had previously prioritized an ambitious release schedule over needed functionality. In response, USCIS dismantled the program office and repositioned Transformation under the USCIS Office of Information Technology so the program could leverage additional engineering expertise. According to officials, the program has also focused on activities like prototyping and beta testing forms, and is deploying updates as targeted changes to specific forms or functionality rather than major system upgrades. The program previously made significant changes after it experienced a 5-month delay in 2012. DHS attributed this delay to weak contractor performance and pursuing an unnecessarily complex system, among other things. To address these issues, the Office of Management and Budget, DHS, and USCIS determined the program should implement an agile software development methodology and increase competition for development work. These changes were reflected in the program’s April 2015 revised baseline. In July 2019, the program office reported that it is working to fill staffing vacancies, but the gap has not had a negative impact on program execution. In the meantime, the program is mitigating the gap with existing staff and contractors. However, officials noted that if positions remain unfilled, the program could experience schedule delays, among other things. USCIS officials reviewed a draft of this assessment and provided no comments. The objectives of this audit were designed to provide congressional committees insight into the Department of Homeland Security’s (DHS) major acquisition programs. We assessed the extent to which (1) DHS’s major acquisition programs are on track to meet their schedule and cost goals and (2) current program baselines trace to key acquisition documents. To address these questions, we selected 29 of DHS’s 80 major acquisition programs. We selected all 17 of DHS’s Level 1 acquisition programs—those with life-cycle cost estimates (LCCE) of $1 billion or more—that had at least one project, increment, or segment in the Obtain phase—the stage in the acquisition life cycle when programs develop, test, and evaluate systems—at the initiation of our audit. Additionally, we reviewed 12 other major acquisition programs—including 6 Level 1 programs that either had not yet entered or were beyond the Obtain phase, and 6 Level 2 programs that have LCCEs between $300 million and less than $1 billion—that we identified were at risk of not meeting their cost estimates, schedules, or capability requirements based on our past work and discussions with DHS officials. Specifically, we met with representatives from DHS’s Office of Program Accountability and Risk Management (PARM)—DHS’s main body for acquisition oversight— as a part of our scoping effort to determine which programs (if any) were facing difficulties in meeting their cost estimates, schedules, or capability requirements. The 29 selected programs were sponsored by eight different components, and they are identified in table 8, along with our rationale for selecting them. To determine the extent to which DHS’s major acquisition programs are on track to meet their schedule and cost goals, we collected key acquisition documentation for each of the 29 programs, such as all LCCEs and acquisition program baselines (APB) approved at the department level since DHS’s current acquisition management policy went into effect in November 2008. DHS policy establishes that all major acquisition programs should have a department-approved APB, which establishes a program’s critical cost, schedule, and performance parameters, before they initiate efforts to obtain new capabilities. Twenty- seven of the 29 programs had one or more department-approved LCCEs and APBs between November 2008 and August 31, 2019. We used these APBs to establish the initial and current cost and schedule goals for the programs. We then developed a data collection instrument to help validate the information from the APBs and collect similar information from programs without department-approved APBs. Specifically, for each program, we pre-populated data collection instruments to the extent possible with the schedule and cost information we had obtained from the APBs and our prior assessments (if applicable) to identify schedule and cost goal changes, if any, since (a) the program’s initial baseline was approved and (b) December 2017—the data cut-off date of our 2018 assessment. We shared our data collection instruments with officials from the program offices to confirm or correct our initial analysis and to collect additional information to enhance the timeliness and comprehensiveness of our data sets. We then met with program officials to identify causes and effects associated with any identified schedule and cost goal changes, including changes as a result of the fiscal year 2019 partial government shutdown. Subsequently, we drafted preliminary assessments for each of the 29 programs, shared them with program and component officials, and gave these officials an opportunity to submit comments to help us correct any inaccuracies, which we accounted for as appropriate (such as when new information was available). Additionally, in July 2018 and July 2019, we obtained copies of the detailed data on affordability that programs submitted to inform the fiscal year 2019 and 2020 resource allocation processes. We also obtained copies of any annual LCCE updates programs submitted in fiscal years 2018 and 2019. For each of the 27 programs with a department-approved APB, we compared (a) the most recent cost data we collected (i.e., a department-approved LCCE, the detailed LCCE information submitted during the resource allocation process, an annual LCCE update, or an update provided by the program office) to (b) DHS’s funding plan presented in the Future Years Homeland Security Program (FYHSP) report to Congress for fiscal years 2020-2024, which presents 5-year funding plans for DHS’s major acquisition programs, to assess the extent to which a program was projected to have an acquisition funding gap. Through this process, we determined that our data elements were sufficiently reliable for the purpose of this engagement. The FYHSP reports information by the department’s new common appropriation structure, which created standard appropriation fund types including (1) procurement, construction, and improvements and (2) operations and support. We refer to these types of funding as (1) acquisition and (2) operations and maintenance throughout this report. current version of the guidance when we initiated our review. We reviewed each program’s most recent APB to determine whether the APB referenced the documents that were used as the basis of its cost, schedule, and performance parameters. We asked program officials to provide the underlying documentation if the APB did not reference a document. We then compared the APB cost, schedule, and performance parameters to the information in the underlying documents. Specifically, we compared the approved LCCE to the APB objective and threshold cost values, the operational requirements document to the APB key performance parameters, and the integrated master schedule to the APB schedule goals. We determined that the cost and performance goals for a program were traceable if the information from the underlying documentation was the same as the cost and performance parameters in the APB. We determined that program schedule goals were traceable to the integrated master schedule, if all future baseline milestones identified in the APB were identified in the integrated master schedule. In addition, the milestone date from the integrated master schedule was within the range of the objective and threshold schedule goals identified in the APB. We did not include programs in our analysis with APBs approved before DHS updated its acquisition policy in March 2016 because they were developed under previous guidance when the requirements for developing APBs were different. We also did not include the APBs approved after DHS updated its acquisition policy in February 2019 because the update was not in place when we initiated this review. In addition, we interviewed officials from headquarters organizations, including PARM, to discuss how policies related to developing APBs are being implemented and clarify requirements for establishing APB parameters. We interviewed component and program officials to identify causes of inconsistencies between the approved APB and documents that provided the basis for approved cost, schedule, and performance parameters. We conducted this performance audit from April 2018 through December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact listed above, Rick Cederholm (Assistant Director), Alexis Olson (Analyst-in-Charge), Whitney Allen, Leigh Ann Haydon, Khaki LaRiviere, Sarah Martin, and Kelsey Wilson made key contributions to this report. Other contributors included Mathew Bader, Andrew Burton, Erin Butkowski, John Crawford, Aryn Ehlow, Lorraine Ettaro, Laurier R. Fish, Alexandra Gebhard, Elizabeth Hosler-Gregory, Stephanie Gustafson, Jason Lee, Claire Li, Ashley Rawson, Jillian Schofield, Roxanna Sun, Anne Louise Taylor, and Lindsay Taylor. Homeland Security Acquisitions: Opportunities Exist to Further Improve DHS’s Oversight of Test and Evaluation Activities. GAO-20-20. Washington, D.C.: October 24, 2019 High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas. GAO-19-157SP. Washington, D.C.: March 6, 2019. Coast Guard Acquisitions: Polar Icebreaker Program Needs to Address Risks before Committing Resources. GAO-18-600. Washington, D.C.: September 4, 2018. DHS Acquisitions: Additional Practices Could Help Components Better Develop Operational Requirements. GAO-18-550. Washington, D.C.: August 8, 2018. Southwest Border Security: CBP Is Evaluating Designs and Locations for Border Barriers but Is Proceeding Without Key Information. GAO-18-614. Washington, D.C.: July 30, 2018. Coast Guard Acquisitions: Actions Needed to Address Longstanding Portfolio Management Challenges. GAO-18-454. Washington, D.C.: July 24, 2018. Homeland Security Acquisitions: Leveraging Programs’ Results Could Further DHS’s Progress to Improve Portfolio Management. GAO-18-339SP. Washington, D.C.: May 17, 2018. DHS Program Costs: Reporting Program-Level Operations and Support Costs to Congress Would Improve Oversight. GAO-18-344. Washington, D.C.: April 25, 2018. Border Security: Additional Actions Could Strengthen DHS Efforts to Address Subterranean, Aerial, and Maritime Smuggling. GAO-17-474. Washington, D.C.: May 1, 2017. Homeland Security Acquisitions: Identifying All Non-Major Acquisitions Would Advance Ongoing Efforts to Improve Management, GAO-17-396. Washington, D.C.: April 13, 2017. Homeland Security Acquisitions: Earlier Requirements Definition and Clear Documentation of Key Decisions Could Facilitate Ongoing Progress. GAO-17-346SP. Washington, D.C.: April 6, 2017. Homeland Security Acquisitions: Joint Requirements Council’s Initial Approach Is Generally Sound and It Is Developing a Process to Inform Investment Priorities. GAO-17-171. Washington, D.C.: October 24, 2016. Homeland Security Acquisitions: DHS Has Strengthened Management, but Execution and Affordability Concerns Endure. GAO-16-338SP. Washington, D.C.: March 31, 2016. Homeland Security Acquisitions: Major Program Assessments Reveal Actions Needed to Improve Accountability. GAO-15-171SP. Washington, D.C.: April 22, 2015. Homeland Security Acquisitions: DHS Should Better Define Oversight Roles and Improve Program Reporting to Congress. GAO-15-292. Washington, D.C.: March 12, 2015. Homeland Security Acquisitions: DHS Could Better Manage Its Portfolio to Address Funding Gaps and Improve Communications with Congress. GAO-14-332. Washington, D.C.: April 17, 2014. Homeland Security: DHS Requires More Disciplined Investment Management to Help Meet Mission Needs. GAO-12-833. Washington, D.C.: September 18, 2012.", "summary": "Each year, the DHS invests billions of dollars in a diverse portfolio of major acquisition programs to help execute its many critical missions. DHS plans to spend more than $10 billion on these programs in fiscal year 2020 alone. DHS's acquisition activities are on GAO's High Risk List, in part, because of management and funding issues. The Explanatory Statement accompanying the DHS Appropriations Act, 2015 included a provision for GAO to review DHS's major acquisitions on an ongoing basis. This report, GAO's fifth review, assesses the extent to which: (1) DHS's major acquisition programs are on track to meet their schedule and cost goals, and (2) current program baselines trace to key acquisition documents. GAO assessed 27 acquisition programs, including DHS's largest programs that were in the process of obtaining new capabilities as of April 2018, and programs GAO or DHS identified as at risk of poor outcomes. GAO assessed cost and schedule progress against baselines; compared APB cost, schedule and performance parameters to underlying documents used in establishing baselines; and interviewed DHS officials. As of August 2019, 25 of the 27 Department of Homeland Security (DHS) programs GAO assessed that had approved schedule and cost goals were on track to meet current goals. The remaining two programs breached their schedule or cost goals. This represents an improvement since GAO's last review. However, GAO found that some of the programs that were on track as of August 2019 are at risk of not meeting cost or schedule goals or both in the future. For example, the U.S. Coast Guard's Offshore Patrol Cutter program faces potential cost increases and schedule slips in the future as a result of damages to the shipbuilder's facility from Hurricane Michael in October 2018. Traceability, which is called for in DHS policy and GAO scheduling best practices, helps ensure that program goals are aligned with program execution plans, and that a program's various stakeholders have an accurate and consistent understanding of those plans and goals. Of the 27 programs GAO assessed, 21 had established baselines after DHS updated its acquisition policy in March 2016 (the most current version of the policy at the beginning of this review). GAO found that the 21 programs' baseline cost and performance goals generally traced to source documents, such as life-cycle cost estimates and planned performance outcomes. However, schedule goals did not generally match up to the programs' integrated master schedules (IMS), as required by DHS acquisition management instruction and as a best practice identified in GAO's Schedule Assessment Guide (see figure). The lack of traceability between IMSs and schedule goals in the approved acquisition program baselines (APB) indicates that DHS does not have appropriate oversight processes in place to ensure that schedules are accurately reflected in program baselines, in accordance with DHS policy and GAO's best practices. Therefore, DHS cannot ensure that the understanding of program schedules among different stakeholders, including component and DHS leadership is consistent and accurate. As a result, DHS leadership may be approving program schedule goals that do not align with program execution plans. GAO is making two recommendations, including that DHS put in place an oversight process to ensure that programs' schedule goals are developed and updated according to GAO's scheduling best practices. DHS concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "For decades, fingerprint analysis was the most widely used biometric technology for positively identifying arrestees and linking them with any previous criminal record. However, beginning in 2010, the FBI began incrementally replacing the Integrated Automated Fingerprint Identification System (IAFIS) with Next Generation Identification (NGI). NGI was not only to include fingerprint data from IAFIS and biographic data, but also to provide new functionality and improve existing capabilities by incorporating advancements in biometrics, such as face recognition technology. As part of the fourth of six NGI increments, the FBI updated the Interstate Photo System (IPS) to provide a face recognition service that allows law enforcement agencies to search a database of criminal photos that accompanied fingerprint submissions using a photo of an unknown person—called a probe photo. The FBI began a pilot of NGI-IPS in December 2011, and NGI-IPS became fully operational in April 2015. NGI-IPS users include the FBI and selected state and local law enforcement agencies, which can submit search requests to help identify an unknown person using, for example, a photo from a surveillance camera. When a state or local agency submits such a photo, NGI-IPS uses an automated process to return a list of candidate photos from the database. The number of photos returned ranges from 2 to 50 possible candidate photos from the database, depending on the user’s specification. According to the FBI, in fiscal year 2018, NGI-IPS returned about 50,000 face recognition search results to law enforcement agency users, a decrease from about 90,000 search results in fiscal year 2017. Figure 1 describes the process for a search requested by state or local law enforcement. In addition to the NGI-IPS, the FBI has an internal unit called Facial Analysis, Comparison and Evaluation (FACE) Services that provides face recognition capabilities, among other things, to support active FBI investigations. FACE Services has access to NGI-IPS, and can also search or request to search databases owned by the departments of State and Defense and 21 states, which use their own face recognition systems. Figure 2 shows which states partnered with the FBI for FACE Services requests, as of May 2019, according to the FBI. Unlike NGI-IPS, which primarily contains photos obtained from criminal justice sources, these external systems primarily contain photos from state and federal government databases, such as driver’s license photos and visa applicant photos. According to the FBI, the total number of face photos available in all searchable repositories for FACE Services is over 641 million. Biometric images specialists for FACE Services manually review any photos received from their external partners before returning a photo as an investigative lead to the requesting FBI agents. No more than two photos are returned as a lead after the specialist for FACE Services completes the review. However, according to FACE Services officials we met with during our May 2016 review, if biometric images specialists determine that none of the databases returned a likely match, they do not return any photos to the agents. According to the FBI, from August 2011 (when searches began) through April 2019, FACE Services received 153,636 photos of unknown persons (often called probe photos) from FBI headquarters, field offices, and overseas offices, which resulted in 390,186 searches of various databases in attempt to find photo matches of known individuals in these databases. Federal agency collection and use of personal information, including face images, is governed primarily by two laws: the Privacy Act of 1974 and the privacy provisions of the E-Government Act of 2002. The Privacy Act places limitations on agencies’ collection, disclosure, and use of personal information maintained in systems of records. The Privacy Act requires that when agencies establish or make changes to a system of records, they must notify the public through a system of records notice (SORN) in the Federal Register. According to Office of Management and Budget (OMB) guidance, the purposes of the notice are to inform the public of the existence of systems of records; the kinds of information maintained; the kinds of individuals on whom information is maintained; the purposes for which they are used; and how individuals can exercise their rights under the Privacy Act. The E-Government Act of 2002 requires that agencies conduct Privacy Impact Assessments (PIAs) before developing or procuring information technology (or initiating a new collection of information) that collects, maintains, or disseminates personal information. The assessment helps agencies examine the risks and effects on individual privacy and evaluate protections and alternative processes for handling information to mitigate potential privacy risks. OMB guidance also requires agencies to perform and update PIAs as necessary where a system change creates new privacy risks, for example, when the adoption or alteration of business processes results in personal information in government databases being merged, centralized, matched with other databases or otherwise significantly manipulated. Within DOJ, preserving civil liberties and protecting privacy is a responsibility shared by departments and component agencies. As such, DOJ and the FBI have established oversight structures to help protect privacy and oversee compliance with statutory and policy requirements. For example, the FBI drafts privacy documentation for its face recognition capabilities, and DOJ offices review and approve key documents developed by the FBI—such as PIAs and SORNs. We reported in May 2016 that the FBI did not (1) update the NGI-IPS PIA in a timely manner when the system underwent significant changes, or (2) develop and publish a PIA for FACE Services before that unit began supporting FBI agents. However, DOJ and the FBI have since taken steps to review and publish PIAs more quickly. As discussed in our 2016 report, consistent with the E-Government Act and OMB guidance, DOJ developed guidance that requires initial PIAs to be completed at the beginning of development of information systems and any time there is a significant change to the information system in order to determine whether there are any resulting privacy issues. In accordance with this guidance, FBI published a PIA at the beginning of the development of NGI-IPS in 2008, as required. However, the FBI did not publish a new PIA or update the 2008 PIA before beginning to pilot NGI-IPS in December 2011 or as significant changes were made to the system through September 2015. During the pilot, the FBI used NGI- IPS to conduct over 20,000 searches to assist in investigations. Similarly, DOJ did not approve a PIA for FACE Services when it began supporting investigations in August 2011. As a new use of information technology involving the handling of personal information, it too required a PIA, according to the E-Government Act, as well as OMB and DOJ guidance. Figure 3 provides key dates in the implementation of these face recognition capabilities and the associated PIAs. DOJ approved the NGI-IPS PIA in September 2015 and the FACE Services PIA in May 2015—over 3 years after the NGI-IPS pilot began and FACE Services began supporting FBI agents with face recognition services. Among other factors, implementation of the NGI-IPS pilot constituted a significant change in the FBI’s use of the technology that, consistent with the E-Government Act and OMB guidance required DOJ/FBI to update the PIA. Similarly, DOJ/FBI acknowledged that FACE Services began supporting FBI investigations in 2011, which involved storing photos in a new work log and also performing automated searches instead of manual searches. As a new use of information technology involving the handling of personal information, it too required a PIA. While DOJ and the FBI updated the internal drafts of these PIAs, the public remained unaware of the department’s consideration for how the FBI uses personal information in the face recognition search process. Given the issues we identified, we recommended that DOJ assess the PIA development process to determine why PIAs were not published prior to using or updating face recognition capabilities. Although DOJ officials did not concur with this recommendation, they did agree that all DOJ processes may be reviewed for improvements and efficiencies. In November 2018, DOJ officials told us that they had reviewed the PIA development process and determined that one reason the FBI’s face recognition PIAs were not completed more quickly was because the FBI and DOJ engaged in an extensive PIA revision process. As a result, DOJ reported that it implemented a pilot in 2018 to expedite the PIA approval process, which included developing a PIA approval template, conducting DOJ’s review earlier in the process, and focusing the review solely on legal sufficiency instead of a more comprehensive review that included less significant editorial changes. According to DOJ, this new process has significantly reduced the time required between the completion of the PIA process by the FBI and the review by DOJ. Further, DOJ reported that it has applied the same process to other DOJ components since December 2018, and that the pilot is evolving into an operational process. We will continue to monitor DOJ’s implementation of its review process changes. We reported in May 2016 that DOJ did not publish a SORN, as required by the Privacy Act, that addresses the collection and maintenance of photos accessed and used through the FBI’s face recognition capabilities, in a timely manner. The DOJ published the SORN on May 5, 2016—after completion of our review—even though those capabilities were in place since 2011. According to OMB guidance then in effect, the SORN “must appear in the Federal Register before the agency begins to operate the system, e.g., collect and use the information.” However, from 2011 through May 2016, the agency collected and maintained personal information for these capabilities without the required explanation of what information it was collecting or how it was used. For example, at the time of our review, the existing version of the SORN that covered FBI’s face recognition capabilities was dated September 1999. According to DOJ officials, it did not address the collection and maintenance of photos accessed and used through NGI for the FBI’s face recognition capabilities but rather discussed fingerprint searches. Given that DOJ did not publish the SORN in a timely manner, we recommended DOJ develop a process to determine why a SORN was not published for the FBI’s face recognition capabilities prior to using NGI-IPS, and implement corrective actions to ensure SORNs are published before systems become operational. DOJ agreed, in part, with our recommendation and submitted the SORN for publication after we provided our draft report for comment. According to DOJ, it continues to review and update its pre-existing SORNs on an ongoing basis and is continually improving the scope and efficiency of its privacy processes. However, as of May 2019, DOJ had not taken actions to address our recommendation. Further, in April 2019, DOJ stated that with respect to transparency, a published PIA will provide much the same information that would be contained in a SORN and may provide it in a timelier manner. However, according to OMB guidance, the purpose of the SORN is to inform the public of the existence of systems of records; the kinds of information maintained; the kinds of individuals on whom information is maintained; the purposes for which they are used; and how individuals can exercise their rights under the Privacy Act. Further, PIAs and SORNs both contain information key to providing the public with information about the collection of their personal information, among other things. We continue to believe that by assessing the SORN development process and taking corrective actions to ensure timely development of future SORNs, DOJ would be better positioned to provide the public with a better understanding of how personal information is being used and protected by DOJ components. The Criminal Justice Information Services Division (CJIS), which operates FBI’s face recognition capabilities, has an audit program to evaluate compliance with restrictions on access to CJIS systems and information by its users, such as the use of fingerprint records. However, at the time of our May 2016 review, it had not completed audits of the use of NGI-IPS or FACE Services searches of external databases. We reported that state and local users had been accessing NGI-IPS since December 2011 and had generated IPS transaction records since then that would enable CJIS to assess user compliance. In addition, we found that the FACE Services Unit had used external databases that included primarily civil photos to support FBI investigations since August 2011, but the FBI had not audited its use of those databases. Standards for Internal Control in the Federal Government calls for federal agencies to design and implement control activities to enforce management’s directives and to monitor the effectiveness of those controls. In May 2016, we recommended that the FBI conduct audits to determine the extent to which users of NGI-IPS and biometric images specialists in FACE Services are conducting face image searches in accordance with CJIS policy requirements. DOJ partially concurred with our recommendation. Specifically, DOJ concurred with the portion of our recommendation related to the use of NGI-IPS. In March 2017, DOJ reported that the FBI began assessing NGI-IPS requirements in participating states in conjunction with its triennial National Identity Services audit, and by February 2018 had conducted eight NGI-IPS audits, which found no significant findings of noncompliance. In February 2018, DOJ provided us with copies of the final audit results for one state and its NGI-IPS audit reference guide. The FBI reported that it conducted an audit of FACE Services in September 2018. According to FBI documentation, the purpose of the audit was to determine the extent to which specialists in FACE Services conducted face image searches in accordance with FBI privacy laws and policies. The scope of the audit focused on determining adherence to policies which govern the appropriate use of NGI-IPS, including those for policy development as well as authorized requests and responses. The FBI reported that it finalized the audit report in April 2019, which concluded that the Face Services Unit is operating in accordance with privacy laws and policies. Further, the FBI stated in May 2019 that audits of FACE Services will continue on a triennial basis and that it conducts triennial audits of states that use NGI-IPS. As a result, DOJ has fully implemented our recommendation. In May 2016, we reported that prior to accepting and deploying NGI-IPS, the FBI conducted testing to evaluate how accurately face recognition searches returned matches to persons in the database. However, we found that the tests were limited because they did not include all possible candidate list sizes and did not specify how often incorrect matches were returned. According to the National Science and Technology Council and the National Institute of Standards and Technology at the time, the detection rate (how often the technology generates a match when the person is in the database) and the false positive rate (how often the technology incorrectly generates a match to a person in the database) are both necessary to assess the accuracy of a face recognition system. The FBI’s detection rate requirement for face recognition searches at the time stated that when the person exists in the database, NGI-IPS shall return a match of this person at least 85 percent of the time. However, we found that the FBI only tested this requirement with a candidate list of 50 potential matches. In these tests, 86 percent of the time, a match to a person in the database was correctly returned. The FBI had not assessed accuracy when users requested a list of 2 to 49 matches. According to FBI, a smaller list would likely lower the accuracy of the searches as the smaller list may not contain the likely match that would be present in the larger list. Further, FBI officials stated during our May 2016 review that they had not assessed how often NGI-IPS face recognition searches erroneously match a person to the database (the false positive rate). If false positives are returned at a higher than acceptable rate, law enforcement users may waste time and resources pursuing unnecessary investigative leads. In addition, we concluded that by conducting this assessment the FBI would help ensure that it is sufficiently protecting the privacy and civil liberties of U.S. citizens enrolled in the database. Therefore, we recommended that the FBI conduct tests of NGI-IPS to verify that the system is sufficiently accurate for all allowable candidate list sizes and ensure that both the detection rate and the false positive rate are identified for such tests. In comments on our draft report in 2016, and reiterated during recommendation follow-up in May 2019, DOJ did not concur with this recommendation. DOJ officials stated that the FBI has performed accuracy testing to validate that the system meets the requirements for the detection rate, which fully satisfies requirements for the investigative lead service provided by NGI-IPS. As of May 2019, DOJ has not taken action to address the recommendation. We continue to believe that the recommended action is needed. Such action would allow the FBI to have more reasonable assurance that NGI- IPS provides leads that help enhance, rather than hinder, criminal investigations and that helps protect the privacy of citizens. As noted above, a key focus of our recommendation is the need to ensure that NGI-IPS is sufficiently accurate for all allowable candidate list sizes. As we reported, although the FBI tested the detection rate for a candidate list of 50 photos, they did not do such tests when NGI-IPS users request smaller candidate lists—specifically between 2 and 50 photos. Further, according to the FBI Information Technology Life Cycle Management Directive, testing needs to confirm the system meets all user requirements. Because the accuracy of NGI-IPS’s face recognition searches when returning fewer than 50 photos in a candidate list is unknown, the FBI is limited in understanding whether the results are accurate enough to meet NGI-IPS users’ needs. In comments on our May 2016 report, DOJ officials also stated that searches of NGI-IPS produce a gallery of likely candidates to be used as investigative leads, not for positive identification. As a result, according to DOJ officials, NGI-IPS cannot produce false positives and there is no false positive rate for the system. We disagree with DOJ. According to the National Institute of Standards and Technology, the detection rate and the false positive rate are both necessary to assess the accuracy of a face recognition system. Generally, face recognition systems can be configured to allow for a greater or lesser number of matches. A greater number of matches would generally increase the detection rate, but would also increase the false positive rate. Similarly, a lesser number of matches would decrease the false positive rate, but would also decrease the detection rate. Reporting a detection rate of 86 percent without reporting the accompanying false positive rate presents an incomplete view of the system’s accuracy. We reported in May 2016 that FBI, DOJ, and OMB guidance all required annual reviews of operational information technology systems to assess their abilities to continue to meet cost and performance goals. For example, the FBI’s Information Technology Life Cycle Management Directive required an annual operational review to ensure that the fielded system is continuing to support its intended mission, among other things. In May 2016, we reported that the FBI had not assessed the accuracy of face recognition searches of NGI-IPS in its operational setting—the setting in which enrolled photos, rather than a test database of photos are used to conduct a search for investigative leads. According to FBI officials, at the time of our May 2016 review, the database of photos used in its tests was representative of the photos in NGI-IPS, and ongoing testing in a simulated environment was adequate. However, according to the National Institute of Standards and Technology, as the size of a photo database increases, the accuracy of face recognition searches performed on that database can decrease due to lookalike faces. At the time of our review, FBI’s test database contained 926,000 photos while NGI-IPS contained about 30 million photos. We concluded that by conducting an operational review of these systems, FBI officials would obtain information regarding what factors affect the accuracy of the face recognition searches, such as the quality of the photos in the database, and if NGI-IPS is meeting federal, state, and local law enforcement needs. As a result, we recommended the FBI conduct an operational review of NGI-IPS, at least annually, that includes an assessment of the accuracy of face recognition searches and take actions, as necessary, to improve the system. In May 2016, DOJ concurred with this recommendation and has taken steps to seek input from its users. For example, the FBI submitted a staff paper through the fall 2016 Advisory Policy Board Process to solicit feedback from its users. Specifically, officials said the paper requested feedback on whether the face recognition searches of the NGI-IPS are meeting their needs, and input regarding search accuracy. According to FBI officials, no users expressed concern with any aspect of the NGI-IPS meeting their needs, including accuracy. DOJ reported that it repeated this process in the fall of 2017. Although FBI’s action of providing working groups with a paper presenting our recommendation is a positive step, FBI’s actions do not fully meet the recommendation. FBI’s paper was presented as informational, and did not result in any formal responses from users. We disagree with FBI’s conclusion that receiving no responses on the informational paper fulfills the operational review recommendation, which includes determining that NGI-IPS is meeting user’s needs. In addition, in May 2019, the FBI stated that it will be working with the National Institute of Standards and Technology on annual operational testing and that such testing meets the intention of this recommendation. However, the proposed testing, while promising, will not occur in an operational environment. As such, we continue to believe the FBI should conduct an operational review of NGI- IPS at least annually, as we recommended. In May 2016 we reported that FBI officials had not assessed the accuracy of face recognition systems operated by external partners. Specifically, before agreeing to conduct searches on, or receive search results from, these systems, the FBI did not ensure the accuracy of these systems was sufficient for use by FACE Services. Standards for Internal Control in the Federal Government calls for agencies to design and implement components of operations to ensure they meet the agencies mission, goals, and objectives, which, in this case, is to identify missing persons, wanted persons, suspects, or criminals for active FBI investigations. As a result, we recommended the FBI take steps to determine whether each external face recognition system used by FACE Services is sufficiently accurate for the FBI’s use and whether results from those systems should be used to support FBI investigations. In comments on our draft report in 2016, and reiterated during subsequent recommendation follow-up, DOJ officials did not concur with this recommendation. DOJ officials stated that the FBI has no authority to set or enforce accuracy standards of face recognition technology operated by external agencies. In addition, DOJ officials stated that the FBI has implemented multiple layers of manual review that mitigate risks associated with the use of automated face recognition technology. Further, DOJ officials stated there is value in searching all available external databases, regardless of their level of accuracy. We acknowledge that the FBI cannot and should not set accuracy standards for the face recognition systems used by external partners. We also agree that the use of external face recognition systems by the FACE Services Unit could add value to FBI investigations. However, we disagree with DOJ and continue to believe that the FBI should assess the quality of the data it is using from state and federal partners. We also disagree with the DOJ assertion that manual review of automated search results is sufficient. Even with a manual review process, the FBI could miss investigative leads if a partner does not have a sufficiently accurate system. The FBI has entered into agreements with state and federal partners to conduct face recognition searches using hundreds of millions of photos. Without assessments of the results from its state and federal partners, the FBI is making decisions to enter into agreements based on assumptions that the search results may provide valuable investigative leads. For example, the FBI’s accuracy requirements for criminal investigative purposes may be different than a state’s accuracy requirements for preventing driver’s license fraud. By relying on its external partners’ face recognition systems, the FBI is using these systems as a component of its routine operations and is therefore responsible for ensuring the systems will help meet the FBI’s mission, goals and objectives. Until FBI officials can assure themselves that the data they receive from external partners are reasonably accurate and reliable, it is unclear whether such agreements are beneficial to the FBI, whether the investment of public resources is justified, and whether photos of innocent people are unnecessarily included as investigative leads. Chairman Cummings, Ranking Member Jordan, and Members of the Committee, this concludes my prepared statement. I would be happy to respond to any questions you may have at this time. For questions about this statement, please contact Gretta Goodwin at (202) 512-8777 or GoodwinG@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Dawn Locke (Assistant Director), Jason Jackson (Analyst-In- Charge), Jennifer Beddor, Ann Halbert-Brooks, Eric Hauswirth, Paul Hobart, Richard Hung, Susanna Kuebler, Kay Kuhlman, Tom Lombardi, and Dina Shorafa. Key contributors for the previous work that this testimony is based on are listed in the previously issued product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Technology advancements have increased the overall accuracy of automated face recognition over the past few decades. This technology has helped law enforcement agencies identify criminals in their investigations. However, there are questions about the accuracy of the technology and the protection of privacy and civil liberties when face recognition technologies are used to identify people for investigations. This statement describes the extent to which the FBI (1) ensures adherence to laws and policies related to privacy regarding its use of face recognition technology, and (2) ensures its face recognition capabilities are sufficiently accurate. This statement is based on GAO's May 2016 report regarding the FBI's use of face recognition technology (GAO-16-267) and includes agency updates to GAO's recommendations. To conduct its prior work, GAO reviewed federal privacy laws, and DOJ and FBI policies and operating manuals. GAO interviewed officials from the FBI and the departments of Defense and State, which coordinate with the FBI on face recognition. GAO also interviewed two state agencies that partner with the FBI to use multiple face recognition capabilities. For updates, GAO reviewed FBI data, as well as materials provided by DOJ and the FBI on the status of GAO's recommendations. In May 2016, GAO found that the the Department of Justice (DOJ) and the Federal Bureau of Investigation (FBI) could improve transparency and oversight to better safeguard privacy and had limited information on accuracy of its face recognition technology. GAO made six recommendations to address these issues. As of May 2019, DOJ and the FBI had taken some actions to address three recommendations—one of which the FBI has fully implemented—but has not taken any actions on the other three. Privacy . In its May 2016 report, GAO found that DOJ did not complete or publish key privacy documents for FBI's face recognition systems in a timely manner and made two recommendations to DOJ regarding its processes for developing these documents. These included privacy impact assessments (PIA), which analyze how personal information is collected, stored, shared, and managed in federal systems, and system of records notices, which inform the public about, among other things, the existence of the systems and the types of data collected. DOJ has taken actions to expedite the development process of the PIA. However, DOJ has yet to take action with respect to the development process for SORNs. GAO continues to believe both recommendations are valid and, if implemented, would help keep the public informed about how personal information is being collected, used and protected by DOJ components. GAO also recommended the FBI conduct audits to determine if users of FBI's face recognition systems are conducting face image searches in accordance with DOJ policy requirements, which the FBI has done. Accuracy . GAO also made three recommendations to help the FBI better ensure the accuracy of its face recognition capabilities. First, GAO found that the FBI conducted limited assessments of the accuracy of face recognition searches prior to accepting and deploying its face recognition system. The face recognition system automatically generates a list of photos containing the requested number of best matched photos. The FBI assessed accuracy when users requested a list of 50 possible matches, but did not test other list sizes. GAO recommended accuracy testing on different list sizes. Second, GAO found that FBI had not assessed the accuracy of face recognition systems operated by external partners, such as state or federal agencies, and recommended it take steps to determine whether external partner systems are sufficiently accurate for FBI's use. The FBI has not taken action to address these recommendations. GAO continues to believe that by verifying the accuracy of both systems—its system, and the systems of external partners—the FBI could help ensure that the systems provide leads that enhance criminal investigations. Third, GAO found that the FBI did not conduct an annual review to determine if the accuracy of face recognition searches was meeting user needs, and recommended it do so. In 2016 and 2017 the FBI submitted a paper to solicit feedback from system users. However, this did not result in formal responses from users and did not constitute a review of the system. GAO continues to believe that conducting such a review would help provide important information about potential factors affecting accuracy of the system. In its May 2016 report, GAO made three recommendations related to privacy, one of which has been implemented. GAO also made three recommendations related to accuracy that the FBI is still working to address.", "document_type": "gao"}
{"report": "The FHLBank System comprises 11 federally chartered banks. The FHLBanks represent 11 districts and are headquartered in Atlanta, Boston, Chicago, Cincinnati, Dallas, Des Moines, Indianapolis, New York City, Pittsburgh, San Francisco, and Topeka (see fig. 1). Each FHLBank is cooperatively owned by its members––such as commercial and community banks, thrifts, credit unions, and insurance companies. As of December 31, 2017, the number of member institutions in each district varied widely, as did the total amount of assets each FHLBank held (see table 1). Each FHLBank has a board of directors made up of member directors and independent directors. As shown in figure 2, the Federal Home Loan Bank Act (as amended by HERA) and its regulations set forth a number of requirements for FHLBank board directors. As of October 2018, each FHLBank board had 14–24 directors, for a total of 194 directors (see table 2). Of the 194, 108 were member directors and 86 were independent directors, including 24 public interest directors. Each board elects a chair and vice chair who serve 2-year terms. As of October 2018, of the 11 board chairs, six were member directors and five were independent directors, including two public interest directors (see table 3). Each FHLBank has a president who reports to the bank’s board of directors, but no representatives from bank management may serve on the boards. To implement requirements in HERA, in December 2010 FHFA issued the Minority and Women Inclusion rule to set forth minimum requirements for FHLBank diversity programs and reporting. Among other things, the 2010 rule required each bank to create its own Office of Minority and Women Inclusion (OMWI) or designate an office to perform duties related to the bank’s diversity efforts, and establish policies related to diversity and inclusion, including policies on nominating board directors. The 2010 rule also requires FHLBanks to submit an annual report to FHFA on their diversity efforts. FHFA also evaluates the quality of corporate governance by board directors as part of its on-site annual examinations and off-site monitoring of FHLBanks. For example, FHFA’s examination includes reviewing the bank boards’ responsibilities, board and committee meeting minutes, and the boards’ oversight of the banks’ operations and corporate culture. Our previous work on diversity includes reports on Federal Reserve Banks’ board diversity, FHLBank board governance, women on corporate boards, and diversity in the financial services sector. In 2011, we found limited diversity among the boards of the 12 Federal Reserve Banks. We recommended that the Board of Governors of the Federal Reserve System encourage all Reserve Banks to consider ways to help enhance the economic and demographic diversity of perspectives on boards, including by broadening potential candidate pools. The recommendation was implemented in December 2011. In a 2015 report on FHLBank board governance, we found that FHFA and FHLBanks had taken steps to increase board diversity, including creating regulations that encouraged the banks to consider diversity in board candidate selection and developing processes to identify and nominate independent directors. In a 2015 report on women on corporate boards, we found that while the share of women on boards of U.S. publicly traded companies had increased, reaching complete gender balance could take many years. We identified factors that might hinder women’s increased representation on boards, including boards not prioritizing recruiting diverse candidates and low turnover of board seats. In addition, in 2017 we reported that representation of women and minorities at the management level in the financial services sector showed marginal or no increase during 2007–2015. Since our 2015 report on FHLBank board governance, FHFA has taken additional actions to encourage diversity on FHLBank boards, including adding a requirement for the banks to report board demographics, clarifying expectations for board elections outreach, requesting the creation of a system-wide board diversity task force, and allowing some banks to add an independent director. FHFA has a limited role in overseeing FHLBanks’ board diversity, according to FHFA staff, because that is not part of the agency’s statutory responsibilities. While FHFA reviews the list of independent director nominees for FHLBank boards to ensure that the nominees meet all eligibility and qualification requirements, board directors are not FHLBank employees. Rather, they form the oversight body of each bank. In contrast, FHFA has a larger role in monitoring diversity efforts related to the workforce and suppliers of the banks. For example, the agency’s annual examination manual contains a section that covers such efforts. FHFA oversight of diversity efforts also includes reviewing the FHLBanks’ annual reports on diversity efforts, which the banks are required to submit under HERA. In adopting its Minority and Women Inclusion rule of 2010 to implement this requirement, FHFA stated that it would analyze and include information from the banks’ annual reports in the agency’s own annual report to Congress. The banks’ annual reports initially included data related to their workforce and supplier diversity efforts. In May 2015, FHFA amended the 2010 rule and added two reporting requirements for the annual reports: (1) data on gender and race/ethnicity of board directors (which the directors would voluntarily self-report), and (2) information on the banks’ outreach efforts (such as to promote diversity when nominating and soliciting director candidates). FHFA stated in its 2015 amendments that it intended to use the director data to analyze future trends in board diversity and the effectiveness of each bank’s policies and procedures to encourage board diversity. FHFA also clarified expectations on FHLBank diversity efforts in a 2016 amendment to its regulation related to bank board directors as well as in guidance and communications to FHLBanks. Clarifying scope of election outreach activities. According to FHFA staff, FHLBanks had inquired if the existing regulation would prohibit the banks from conducting outreach to or recruiting of diverse board candidates in the nomination or solicitation process. FHFA regulation restricts FHLBanks from advocating for a particular member director candidate or influencing the board election for member and independent directors. According to FHFA staff, to address these concerns, the agency amended the regulation in 2016 to clarify that the banks may conduct outreach to find diverse board director candidates. FHFA staff added that the regulation amendment also made clear that the banks may fulfill the regulatory requirement to encourage consideration of diversity in nominating or soliciting candidates for board director positions without violating restrictions on advocating for particular director candidates. Guidance. FHFA provided FHLBanks with guidance related to diversity, including board diversity. For example, the agency provided guidance on the roles and duties of the banks’ OMWI officers and the scope of diversity regulations. FHFA provided the banks a template to report newly required data on the gender and race/ethnicity of board directors. To help banks prepare their annual reports, in June 2018 FHFA also developed an annual report template that outlines and describes the contents of the required reporting elements. The template includes sections for individual FHLBanks to present data on board composition by diversity categories and to describe past and future outreach activities and strategies to promote board diversity and outcomes from the bank’s activities. Communications. FHFA has communicated guidance and discussed board diversity issues with FHLBank boards and with staff involved in the banks’ board diversity efforts. For example, FHFA staff gave presentations at meetings during which FHLBank board directors shared information on board diversity efforts. The staff noted FHFA’s OMWI director generally attends the semi-annual conferences of the banks’ OMWI officers, during which she discusses diversity issues such as the roles and responsibilities of these officers and the scope of the FHFA regulations. Furthermore, FHFA OMWI and other offices developed and implemented some strategies to help FHLBanks maintain or increase board diversity. In 2016, FHFA OMWI staff met with FHLBanks and requested that the banks create a Bank Presidents Conference Board Diversity Task Force to share practices to promote board diversity. The staff said that they act as facilitators and informal advisors and may provide technical assistance to the system-wide task force—for example, by developing a list of practices related to board diversity. Also, as encouraged by FHFA, starting in 2017, each bank has a representative (a board director or the bank president) on the task force. Also, based on FHFA’s 2016 annual FHLBank board analysis, the FHFA Director approved requests from three FHLBanks to add an independent director seat for their 2017 boards to help maintain or increase board diversity. FHFA extended the offer to the other banks (except Des Moines, as its board was undergoing restructuring after the merger with Seattle). FHFA staff said in preparation for their 2017 FHLBank board analysis, they informally monitored the gender and minority status of the additional independent director seats filled by the seven banks that accepted the offer. Six of the seats were filled by women (of whom two were minorities) and one seat was filled by a minority male, according to FHFA staff. FHFA staff also told us the FHFA Director has some discretion on the number of director seats based on an individual bank’s circumstances, including the request to maintain diversity. For example, in 2018, one FHLBank requested to retain its female board vice chair to help preserve diversity and institutional knowledge on its board. FHFA granted the bank’s request to keep the director for another year. FHFA staff told us that FHFA has considered issuing guidance in two areas, but that these areas do not represent immediate priorities for their diversity efforts. Specifically, FHFA OMWI staff stated that the office intended to develop an examination module on board diversity, but this is not the office’s high priority for 2019. As previously noted, FHFA’s current examination manual includes a section that covers FHLBanks’ workforce and supplier diversity efforts. But, the manual does not consider board diversity-related issues in as much detail as the supplier and workforce section. For example, it covers FHFA’s review of the quality of corporate governance by board directors and mentions the consideration of diversity for potential board director candidates. Also, the 2015 rule amendments noted that the agency intended to develop guidance to further elaborate on its expectations related to outreach activities and strategies for the banks’ board directors. FHFA staff told us that they would like to focus on ongoing diversity efforts and gather more information before starting new efforts. At the overall FHLBank board level, the share of female directors increased from 18 percent (34 directors) in 2015 to 23 percent (44 directors) in October 2018 (see fig. 3). This represented a continuation of an upward trend. For example, we previously reported a 16 percent share (31 female directors) in 2014. Each FHLBank had at least two female board directors in October 2018, but some boards had higher shares of female directors than others. As shown in figure 4, four banks—Chicago, Des Moines, Dallas, and Pittsburgh—had four or more female board directors (representing 22–38 percent of the boards). In comparison, seven banks had two or three female directors (representing 14–20 percent). Additionally, FHLBanks varied in how many female directors were added from 2015 to October 2018—one bank added two, six each added one, and four added none. For additional information on the number of board directors by bank and by gender from 2015 to October 2018, see appendix II. Women have some representation in board leadership positions. In October 2018, two FHLBanks—Des Moines and Pittsburgh—had female vice chairs of their respective board. Another bank (San Francisco) had a female vice chair of its board in 2016 and 2017. In 2015, we reported that one bank (Atlanta) had a female board chair. Additionally, each bank’s board has committees (such as the Audit Committee and the Risk Management Committee) with committee chairs and vice chairs. Ten of the 11 banks had board committees with at least one female chair or vice chair in October 2018. The share of women who chaired board committees was the same as the share of women on the overall FHLBanks boards in October 2018—23 percent. We compared female representation on FHLBank boards to that of other corporate boards and that of senior management in the financial services sector. Women constituted 23 percent of FHLBank boards in October 2018 and 22 percent of boards of the companies in the Standard and Poor’s 500 in 2017, as reported by Institutional Shareholder Services. Our analysis of the most recently available EEOC data found that the share of women in senior management positions in the financial services industry in 2016 was 29 percent. The share of women on FHLBank boards was 19 percent in the same year. Senior management in the financial services sector represents a pool of comparable candidates that could provide directors for FHLBank boards. The share of directors who self-identified as racial/ethnic minorities increased from 2015 to 2017, but the size of the increase is unclear due to the number of directors who did not report this information. Board directors voluntarily submit demographic information, including race/ethnicity. Some directors might have chosen not to self-identify their race/ethnicity. At the overall FHLBank board level, the share of directors who self- identified as racial/ethnic minorities increased from 2015 to 2017 (see fig. 5). Eleven percent (20 directors) of FHLBank board directors self- identified as racial/ethnic minorities in 2015 and 15 percent (30 directors) in 2017. Four percent (7 directors) did not self-identify in 2015 and 8 percent (15 directors) in 2017. The increase in the number of directors who identified as racial/ethnic minorities shows an upward trend from 10 percent (19 directors) in 2014, as we reported in 2015. The number of directors who self-identified as racial/ethnic minorities varied by bank. As shown in figure 6, all 11 FHLBanks had at least one minority director on the board in 2017, and six banks had three or more minority directors. Ten of the 11 banks each added one minority director during 2015–2017. For additional information on the number of board directors by bank and by race/ethnicity in 2015–2017, see appendix II. More specifically, as seen in table 4, in 2017, 9 percent (18 directors) identified as African-American, 4 percent (8 directors) identified as Hispanic, 2 percent (3 directors) identified as Asian, and 1 percent (1 director) identified as “other.” Racial/ethnic minorities have limited representation in board leadership positions. As of October 2018, one FHLBank had a vice chair of its board who identified as a minority. In 2017, another bank had one vice chair of its board who identified as a minority. We compared the FHLBank boards’ share of racial/ethnic minorities to those of corporate boards and senior management in the financial services sector. In 2017, 15 percent of the FHLBank board directors identified as racial/ethnic minorities, as previously noted. This compares to 14 percent on boards of directors of companies in the Standard and Poor’s 500 in 2017, according to Institutional Shareholder Services, and 12 percent in senior management of the financial services industry in 2016, based on our analysis of EEOC data. In 2016, the share of minority directors on FHLBank boards was 13 percent. Board demographic data collection processes vary by FHLBank, which may contribute to the differences in the number of directors who did not self-identify their gender, race/ethnicity, or both. FHFA has not reviewed the banks’ varying processes to determine whether some processes were more effective, such as whether the practices allowed banks to more effectively identify and follow up with directors who may have forgotten to respond. All directors at three banks self-reported their gender and race/ethnicity in 2015–2017, but some directors at the other eight banks did not self-identify this information. However, we could not determine whether those directors deliberately chose not to self-report this information or inadvertently did not respond to the data collection forms or questions. As allowed by FHFA regulation, FHLBanks varied in the data collection forms they used, questions they asked, and methods they used to distribute forms to board directors to obtain self-reported gender and race/ethnicity information. For example, the three banks with complete data from all directors each used different data collection forms. One bank collected gender and race/ethnicity as a voluntary section of its annual board director skills assessment, which was filled out by each director. Two banks distributed a separate data collection form at a board director meeting or through an online survey, which might have included a mechanism for tracking which directors had not responded to the survey. The other eight banks, which had incomplete demographic data, also used varying data collection processes. Of these, four banks distributed their data collection forms during a board meeting or through an e-mail, and the other four banks used online surveys. Of the 11 banks, six included an option on their forms to mark “opt not to self-identify,” while five included similar language as part of the form indicating that completing the form is voluntary. Although some banks had similar approaches to data collection, such as using an online survey, it is unclear whether certain approaches helped some banks to obtain more complete data despite directors’ right to opt out of self-reporting demographic information. FHFA has implemented some efforts on improving the quality of the data FHLBanks report to the agency, but FHFA staff told us that such efforts have not included a review of how the banks collect board director demographic data. For example, FHFA created templates to help banks report board data and board-related content, and its data reporting manual focused on reporting data related to the banks’ workforce, supplier base, and financial transactions. However, none of these documents discussed processes for collecting board director demographic data. According to FHLBank staff, FHFA’s instructions on board director data collection are limited to what is stated in the regulation. That is, banks should collect data on their board directors’ gender and race/ethnicity using EEOC categories, and such data should be voluntarily provided by the directors without personally identifiable information. FHFA’s 2015 regulation amendments require FHLBanks to compare the board demographic data with prior year’s data and provide a narrative of the analysis. FHFA also stated in the amendments that it intended to use the director data to establish a baseline to analyze future trends of board diversity. Additionally, federal internal control standards state that agency management should use quality information to achieve their objectives. Quality information would include complete and accurate information that management can use to make informed decisions in achieving key objectives. By obtaining a better understanding of the different processes FHLBanks use to collect board demographic data, FHFA and the banks could better determine which processes or practices could contribute to more complete data. For example, there may be practices that could help banks more effectively follow up with directors who might have missed the data collection forms or questions. More complete board demographic data could help FHFA and the banks more effectively analyze data trends over time and demonstrate the banks’ efforts to maintain or increase board diversity. FHLBanks report some challenges that may slow or limit their efforts to increase board diversity, which include low levels of diversity in the financial sector; member institutions not prioritizing diversity; balancing the need for diversity with retaining institutional knowledge; and competition for women and minority candidates. Despite these challenges, the banks have taken several steps to help increase board diversity. According to FHLBank representatives, including board directors, the FHLBank boards face challenges that may slow or limit their efforts to increase diversity, including the following: Low levels of diversity in the financial sector. Twelve representatives from nine FHLBanks told us that the pool of eligible women and minority board candidates is small in the banking and financial sector. For example, five representatives emphasized that the majority of member institutions have chief executive officers (CEO) who are white males. In particular, one director told us that out of the hundreds of member institutions affiliated with his FHLBank, he knew of only six female CEOs. Directors representing five banks also noted that the pool of eligible, diverse candidates in senior management positions in the financial services sector can be even smaller in certain geographic areas. As a result, it can be particularly challenging for some banks to fill member director seats because, by statute, candidates for a given FHLBank board must come from member institutions in the geographic area that the board seat represents. For example, one director said that the pool of such candidates is especially small in rural areas. In 2015, FHFA told us that the overall low levels of diversity in the financial services sector, including at FHLBank member institutions, increased the challenges for improving board diversity. However, representatives of corporate governance organizations with whom we spoke told us that the financial services sector does not face unique challenges. Representatives also said that qualified women and minority candidates are present in the marketplace. Our analysis of 2016 EEOC data found that the representation of women in senior management in the financial services sector was within 1 percentage point of the share of women in senior management in the private sector overall, and minority representation was within 4 percentage points. Member institutions may not always prioritize diversity in director elections. As previously discussed, member institutions nominate member director candidates and vote for the member director and independent director candidates. Ten representatives from eight FHLBanks stated that member institutions may prioritize other considerations over diversity when they nominate and vote on board candidates, such as name recognition or a preference for candidates who are CEOs. One director told us that the member banks may not be as interested in diversity as the FHLBanks. Another director emphasized that FHLBanks are trying to change attitudes and embed diversity in the member institutions’ operations. He characterized this process as a marathon, not a sprint. Board directors with whom we spoke also stressed that FHFA regulations do not allow the FHLBank boards to exert influence over how member institutions vote. Board directors can emphasize the importance of diversity to member institutions but cannot in their official capacity campaign for specific candidates. Balancing the need for diversity with retaining institutional knowledge. Directors from five banks told us that they aim to balance bringing in new women or minority directors with retaining the valuable institutional knowledge of incumbent directors. One director added that new board directors face a steep learning curve. Thus, the directors at some banks will recruit new directors only after allowing incumbent directors to reach their maximum number of terms (which could translate to several years). As we reported in 2015, FHFA staff acknowledged that low turnover, term lengths, and the need to balance diversity with required skills posed challenges to the FHLBank board diversity. In our 2016 report on women on corporate boards, relevant stakeholders acknowledged this as a challenge because directors with longer tenure possess knowledge about a company that newer directors cannot be expected to possess. Competition for women and minority candidates. Board directors from five FHLBanks told us that they face competition as they seek to recruit women and minority candidates. For example, a director from one bank told us that his board encouraged a potential female candidate to run for a director seat. However, the candidate felt she could not accept the opportunity because of her existing responsibilities on the boards of two publicly traded companies. While these challenges can apply to member and independent directors, representatives from all 11 FHLBanks emphasized that it can be particularly challenging to find and elect female or minority member directors. Our analysis of FHLBank board director data confirmed that across 11 FHLBank boards, female representation was lower among member directors (13 directors or 12 percent) than independent directors (31 directors or 36 percent) in October 2018. FHFA stated in this review and in 2015 that they are aware of the potential difficulty of identifying diverse candidates for member directors and that greater board diversity likely would be achieved with independent directors. Since 2015, FHLBanks have taken actions to help increase board diversity, including developing and implementing practices and strategies that target board diversity in general and member directors specifically. As previously discussed, at the request of FHFA, the banks established the Bank Presidents Conference Board Diversity Task Force. The purpose of the task force is to develop recommendations for advancing board diversity and to enhance collaboration and information sharing across FHLBank boards. Each bank is represented by a board director or the bank president. Representatives meet regularly to discuss challenges, recommend practices, and receive training. One task force representative told us that her participation on the taskforce has helped demonstrate to her board and bank that diversity matters. Others mentioned that the ability to share practices and learn from other banks was a great benefit. As part of its work, the task force developed a list of practices that FHLBanks have used or could use to improve board diversity (see text box). According to bank staff, the list was approved by the presidents of each bank and distributed to bank staff. The practices can be generally summarized into three categories—emphasizing the importance of diversity; assessing skills diversity; and seeking new ways to find candidates—which are generally similar to the commonly cited practices for improving board diversity we identified in 2015. Summary of Practices Developed by Bank Presidents Conference Board Diversity Task Force of the Federal Home Loan Banks Include references to diversity on the bank website, in appropriate publications, in presentations about the bank, and particularly in all election materials. Educate current board members on the business case for diversity. Educate member institutions on the business case for diversity through member meetings, newsletter articles, etc. to help develop a more diverse member base and help groom new leaders. Perform a skills assessment of current board skills and areas of expertise and determine skill sets and expertise needed. Review the term limits of current directors and determine the possible loss of continuity if multiple incumbent directors leave the board in a short period of time. Build a pool of diverse member and independent candidates. Conduct outreach to regional and national business organizations, such as trade associations, women and minority business groups, and professional organizations, to ask for referrals of possible candidates and form relationships prior to a board election. Seek an additional independent board seat from the Federal Housing Finance Agency. Example of Diversity Statement in an Election Announcement for a Federal Home Loan Bank The Federal Home Loan Bank of New York (FHLBNY) included the following statement in its 2017 director election announcement package: “The FHLBNY’s Board of Directors consists of a talented group of dedicated individuals that benefits from, among other things, demographic (including gender and racial) diversity, and we expect that this will continue in the future. As you consider potential nominations for Member Directorships and give thought to persons who might be interested in Independent Directorships, please keep diversity in mind. Your participation in this year’s Director Election process is greatly appreciated, and will help continue to keep the Board and the FHLBNY diverse and strong.” Emphasizing the importance of diversity. All 11 FHLBanks included statements in their 2017 election announcements that encouraged voting member institutions to consider diversity during the board election process. Six banks expressly addressed gender, racial, and ethnic diversity in their announcements. One female director with whom we spoke said that she was encouraged to run for a board seat after reading an election announcement in 2013 that specifically called for candidates with diverse backgrounds. All 11 FHLBanks also referenced their commitment to diversity on their websites, including posting diversity and inclusion policies, describing diversity missions, or including board statements on diversity. Directors we interviewed from all 11 FHLBanks told us that their bank conducted or planned to conduct diversity training for board directors. The training sessions covered topics such as the business case for diversity and unconscious bias. Additionally, board directors from two banks discussed efforts to encourage member institutions to increase diversity, such as holding a panel on the importance of diversity at the annual member conference. In 2015, we found that demonstrating a commitment to diversity in ways similar to these is a first step towards addressing diversity in an organization. Assessing skills diversity. Nine FHLBanks performed board skills assessments annually or biennially. These assessments asked directors to evaluate their knowledge of specific topic areas. FHFA regulation allows each bank to annually conduct a skills and experience assessment and, if applicable, inform members before elections of particular qualifications that could benefit the board. In 2015, we found that conducting a skills assessment was a commonly cited practice for boards seeking to increase representation of women and minorities. The other two FHLBanks conducted board self-assessments annually, focused on board effectiveness and organization, but did not evaluate the skills of their individual directors. All 11 FHLBanks also reported regularly reviewing the remaining terms of current directors to determine the possible loss of continuity. Seeking new ways to find candidates. Representatives from 10 FHLBanks noted that their banks maintain a pool of diverse director candidates for future open positions. FHLBanks described using various methods to build these pools. All 11 banks described outreach to trade organizations, industry groups, universities, and nonprofit organizations when looking to identify women and minority candidates. For example, FHLBank of Pittsburgh identified 15 organizations in its district that actively promote diversity and the inclusion of women and minorities in business to specifically target in 2017. Directors from seven banks also reported hiring a search firm or consultant to help them identify women and minority candidates. These activities are consistent with commonly cited practices described in our 2015 work that boards can use to reach out beyond the typical pool of applicants. As previously mentioned, seven FHLBanks requested or were offered an additional independent director seat by FHFA. According to FHFA staff, four of the seats were filled by white females, two were filled by minority females, and one was filled by a minority male. Example of a Diversity Practice Focused on Member Directors In 2017, the Federal Home Loan Bank of San Francisco developed a Member Director Diversity Outreach Plan. The plan included eight steps that provide timelines and specific assignments for directors and bank management. For example, steps include conducting early outreach to trade organizations where women and minority directors might participate, individual director outreach to potential candidates, and developing a list of prospective candidates in case of vacancy appointments. Following the implementation of this plan, member institutions elected one female director and one minority director to fill the vacant member director seats. Fill interim seats with women and minority candidates. FHLBanks can appoint women or minority candidates to fill interim member director seats. By regulation, when a director leaves the board in mid- term, the remaining board directors may elect a new director for the remaining portion of the term. For example, the FHLBank of Pittsburgh reported electing a minority director in 2017 to fill a vacant member director seat. One director told us that when a female or minority director is elected for an interim term, the election increases the likelihood of the director being elected by the member institutions for a following full term. Conduct mentoring and outreach. FHLBank board directors also can use their personal networks to conduct outreach and mentor potential candidates. Current directors can pledge to identify and encourage potential women and minority candidates to run for the board. For example, one director told us that his board emphasizes the need for directors to pay attention to potential women and minority candidates they meet. This director said he had personally contacted qualified potential candidates and asked them to run. Another director noted that women and minority directors are likely to know other qualified candidates with diverse backgrounds. These directors can identify and refer individuals in their networks. Another director emphasized the importance of member directors conducting outreach to member institutions. Member directors have the most interaction with the leadership of member institutions and can engage and educate them on the importance of nominating and electing diverse member directors. Look beyond CEOs. Additionally, FHLBanks can search for women and minority candidates by looking beyond member bank CEOs. By regulation, member directors can be any officer or director of a member institution, but there is a tendency to favor CEOs for board positions, according to board directors, representatives of corporate governance organizations, and academic researchers with whom we spoke. The likelihood of identifying a woman or minority candidate increases when member institutions look beyond CEOs to other officers, such as chief financial officers or board directors. For example, the FHLBank of Des Moines expanded its outreach to women and minority candidates to include board directors at member institutions. In 2017, a female director who is a board member of her member institution was elected. The Housing and Economic Recovery Act of 2008 emphasized the importance of diversity at the FHLBank System, and FHFA and FHLBanks have undertaken efforts to encourage diversity at the banks’ boards. In particular, FHFA plans to use data it collects on the gender and race/ethnicity of board directors as a baseline to analyze trends in board diversity. While FHFA regulation allows directors to choose not to report this information, the banks’ varying data collection processes did not always allow banks to accurately account for missing information (as in the case of directors forgetting to respond to the data questions or fill out forms). Reviewing the processes the banks use to collect the demographic data could help FHFA and the banks identify practices to produce data that would better allow FHFA to track trends in board diversity. FHFA could work with FHLBanks (potentially through the system-wide Board Diversity Task Force) to conduct such a review. The Director of FHFA’s Office of Minority and Women Inclusion, in consultation with FHLBanks, should conduct a review on each bank’s processes for collecting gender and race/ethnicity data from boards of directors and communicate effective practices to FHLBanks. (Recommendation 1) We provided a draft of this report to FHFA and each of the 11 FHLBanks for review and comment. In its comments, reproduced in appendix III, FHFA agreed with our recommendation. FHFA commented that it intends to engage with FHLBanks’ leadership in 2019 to discuss board data collection issue and address our recommendation. FHFA also stated that it plans to request that the Board Diversity Task Force explore the feasibility and practicability for FHLBanks to adopt processes that can lead to more complete data on board director demographics. In addition, four FHLBanks provided technical comments, which we incorporated as appropriate. The other seven FHLBanks did not have any comments. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Acting Director of FHFA, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or ortiza@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines the (1) extent to which the Federal Housing Finance Agency (FHFA) has taken steps to encourage board diversity at the Federal Home Loan Banks (FHLBank); (2) trends in diversity composition (gender, race, and ethnicity) for the boards of individual FHLBanks; and (3) challenges FHLBanks face and practices they use in recruiting and maintaining a diverse board. While diversity has many dimensions, this report focuses on gender, race, and ethnicity. To understand the steps FHFA has taken to encourage FHLBank board diversity, we reviewed relevant laws and regulations related to FHLBank boards, including FHFA regulations on director elections and diversity reporting requirements. For example, we reviewed the relevant sections in the Housing and Economic Recovery Act of 2008 pertaining to FHFA and the banks and FHFA’s 2010 Minority and Women Inclusion rule and its 2015 amendments. We also reviewed other FHFA and bank documentation related to board director elections and diversity considerations. For example, we reviewed FHFA’s annual board director analysis for 2016–2018 to identify actions the agency took to help maintain or increase the number of female or minority directors at the FHLBank boards. Additionally, we interviewed FHFA staff to understand the agency’s role in overseeing FHLBank board diversity and the agency’s efforts in helping the banks maintain or increase board diversity. To describe trends in FHLBank board diversity, we analyzed gender and race/ethnicity data self-reported by board directors in FHLBanks’ annual reports to FHFA as of the end of 2015, 2016, and 2017. The banks’ annual reports use the gender and race/ethnicity classifications from the Employer Information Report (EEO-1) of the Equal Employment Opportunity Commission (EEOC). The EEO-1 report race/ethnicity categories are Hispanic or Latino, White, Black or African-American, Native Hawaiian or Other Pacific Islander, Asian, Native American or Alaska Native, and Two or More Races. The Hispanic or Latino category in EEO-1 incorporates Hispanics or Latinos of all races. For our report, we used the following categories: Hispanic, White, African-American, Asian, and “Other.” We included only non-Hispanic members under White, African-American, Asian, and “Other.” We included Asian American, Native Hawaiian or Pacific Islander under the Asian category, and we included Native American or Alaskan Native, and Two or More Races under “Other.” To provide more recent data on gender composition, we also analyzed data on the gender of directors who were on boards as of October 17, 2018. Specifically, we compiled a list of board directors who started or continued their terms on the boards in 2018, based on board director information from the banks’ 2017 Form 10-K filings with the Securities and Exchange Commission (SEC). The filings include the names and brief biographies of board directors, which we used to derive the gender data for directors. For example, if directors were referred to as “Mr.” in the Form 10-Ks, we counted them as male. If they were referred to as “Ms.,” we counted them as female. We then confirmed with each FHLBank the compiled list of board directors, as of October 17, 2018. Because some directors did not self-identify their gender in 2015–2017 annual reports, we also used information in the banks’ 2014–2016 Form 10-Ks to derive data on the gender of the banks’ board directors. As a result, we were able to report the gender information for all FHLBank board of directors from 2015 through October 2018. We separately requested the names of the chairs and vice chairs for the committees of each bank’s board as of October 26, 2018. We then derived the gender of the chairs and vice chairs for these committees based on the information in the banks’ Form 10-Ks. To analyze data on board director race/ethnicity, we relied on FHLBanks’ 2015–2017 annual reports. However, we were not able to use banks’ Form 10-Ks to derive data on race/ethnicity for board directors who did not self-identify race/ethnicity in the annual reports because the 10-Ks do not include such information. We also requested and analyzed from each bank data on the gender and race/ethnicity of their board chair and vice chair as of October 17, 2018. We assessed the reliability of the data from the banks’ annual reports and Form 10-Ks through electronic testing, a review of documentation, and interviews with knowledgeable agency staff, and we determined these data to be sufficiently reliable for describing the overall trends and composition of gender and race/ethnicity at the FHLBank boards, except the data for directors who did not self- identify their race/ethnicity, as discussed in the report. We also compared the most recently available demographic information on FHLBank board directors with the demographic composition of senior management in the financial services industry and the overall private sector (excluding financial services), based on data from the 2016 EEO-1 report from EEOC. Senior management in the financial services industry represents a pool of comparable candidates that could provide directors for FHLBank boards. The EEO-1 report data are annually submitted to EEOC by most private-sector firms with 100 or more employees. The data include gender and race/ethnicity of the employees by job category. We included workforce from all sites of multi-establishment companies (companies with multiple locations). Consequently, the analysis included in this report may not match the analysis found on EEOC’s website, which excludes workforce from sites of multi-establishment companies with less than 50 employees. In our analysis of senior management-level diversity in the financial services sector, we included companies in the finance and insurance industry categorized under code 52 of the North American Industry Classification System. We assessed the reliability of the data from the EEO-1 report through electronic testing, a review of documentation, and interviews with knowledgeable agency staff. We determined these data to be sufficiently reliable for comparing the composition of gender and race/ethnicity in the financial services sector and the overall private sector with that of the FHLBank boards. Furthermore, to provide a general comparison of FHLBank board diversity composition with corporate boards of U.S. companies, we reviewed research that discussed data related to diversity at corporate boards of U.S. companies in recent years. In addition, from each FHLBank, we requested and reviewed the instrument they used to collect gender and race/ethnicity information from their board directors. We also obtained and reviewed information on the methods the banks used to distribute and collect the data collection instruments, and any instructions FHFA provided to the banks or that the banks provided to the board directors on collecting this information. We reviewed relevant information from the banks’ annual reports and relevant regulations on collecting and submitting board directors’ gender and race/ethnicity information. We also compared the banks’ data collection processes with relevant federal internal control standards. To determine the challenges the FHLBanks face and practices they use to recruit and maintain a diverse board, we interviewed staff at FHLBanks and FHFA to learn about the Bank Presidents Conference Board Diversity Task Force and the list of diversity practices compiled by the task force. We reviewed and analyzed the banks’ 2017 annual reports to learn about the most recent practices the banks implemented. We also reviewed the banks’ websites and bank documents, such as election materials and skills assessments for all 11 banks. In addition, we conducted semi- structured interviews with 10 board directors and one bank president, who act as representatives on the system-wide board diversity task force. We also conducted semi-structured interviews with a nongeneralizable sample of FHLBank board chairs from six banks (Atlanta, Boston, Des Moines, Pittsburgh, San Francisco, and Topeka). We selected these banks to achieve variation in board diversity composition (share of women and minority directors), asset size, and geographic locations. In these interviews, we asked directors and staff about the challenges their banks faced as they sought to increase or maintain diverse boards. We also asked about their participation on the task force, the task force diversity practices, and any other practices their banks had implemented related to board diversity efforts. To determine if the task force diversity practices generally followed commonly cited practices used to improve board diversity, we compared the task force practices against commonly cited practices we identified in previous work in 2015. To verify that the practices we identified in 2015 were still relevant and useful, we interviewed three academics and representatives of four organizations that advocate for board diversity, including gender and racial/ethnic diversity. We selected these external stakeholders based on their research and experience related to increasing board diversity and referrals from others knowledgeable in the field. In our interviews with external stakeholders, we also asked about the challenges that financial organizations or other publicly traded companies may face as they work to increase or maintain board diversity. We compared these answers to the challenges that FHLBank representatives described. We conducted this performance audit from July 2018 to February 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Anna Maria Ortiz, (202) 512-8678, ortiza@gao.gov. In additional to the individual named above, Kay Kuhlman (Assistant Director), Anna Chung (Analyst in Charge), Laurie Chin, Kaitlan Doying, Jill Lacey, Moon Parks, Barbara Roesmann, Jessica Sandler, and Jena Sinkfield made key contributions to this report.", "summary": "The FHLBank System consists of 11 regionally based banks cooperatively owned by member institutions. In 2018, each FHLBank had a board of 14–24 directors. Member directors are nominated from member institutions and independent directors from outside the system. Member institutions vote on all directors. At least two independent directors on a board must represent consumer or community interests. FHFA is the regulator of the FHLBanks. GAO was asked to review FHLBanks' implementation of board diversity and inclusion matters. This report examines (1) steps FHFA took to encourage board diversity at FHLBanks; (2) trends in gender, race, and ethnicity on FHLBank boards; and (3) challenges FHLBanks face and practices they use to recruit and maintain diverse boards. GAO analyzed FHLBank data on board demographics, reviewed policies and regulations, and reviewed previous GAO work on diversity at FHLBanks and the financial services industry. GAO interviewed FHFA and FHLBank staff and a nongeneralizable sample of FHLBank board directors and external stakeholders knowledgeable about board diversity. The Federal Housing Finance Agency (FHFA) has taken formal and informal steps to encourage board diversity at Federal Home Loan Banks (FHLBank) since 2015. For example, FHFA required FHLBanks to add board demographic data to their annual reports; clarified how banks can conduct outreach to diverse board candidates; and allowed some banks to add an independent director. Since 2015, the share of women and minority directors on the boards of FHLBanks increased (see figure). The number of women directors increased from 34 in 2015 to 44 in October 2018, and the number of minority directors increased from 20 in 2015 to 30 in 2017, based on most recently available data. Trends for minority directors were less clear, because the banks' varying data collections processes did not always allow them to determine the extent to which directors opted out or forgot to answer data collection forms. FHFA stated that it planned to use board data to establish a baseline to analyze diversity trends. A review of the banks' data collection processes would help identify whether practices exist that could help improve the completeness of the data. FHLBanks reported they continued to face some challenges to their efforts to promote board diversity, especially among member director seats. The challenges include (1) balancing the addition of new women or minority directors with retaining the institutional knowledge of existing directors; and (2) competing with other organizations for qualified female and minority board candidates. Despite reported challenges, FHLBanks have taken measures to promote board diversity, such as establishing a task force to promote board diversity through information sharing and training. Individually, the FHLBanks emphasized the importance of diversity in election materials, built pools of diverse candidates, and conducted outreach to industry and trade groups. They also took actions to increase diversity specifically among member directors, including filling interim board seats with women and minority candidates and encouraging directors to personally reach out to potential women and minority candidates. GAO recommends that FHFA, in consultation with FHLBanks, review data collection processes for board demographic information and communicate effective practices to banks. FHFA agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "The Tobacco Control Act specifies the total amount of tobacco user fees that FDA is authorized to assess and collect each fiscal year (beginning with fiscal year 2009) and stipulates those fees must be used for FDA’s tobacco regulation activities. FDA collected about $4.5 billion in tobacco user fees from fiscal year 2010 through fiscal year 2018, according to FDA budget documents, and has ongoing authority to assess and collect $712 million from tobacco manufacturers and importers annually starting in fiscal year 2019. See table 1 for the total user fees the Tobacco Control Act authorized FDA to collect, by fiscal year. All of FDA’s activities related to regulating tobacco products—including activities aimed at preventing youth use of tobacco products, educating the public about tobacco products and the risks associated with their use, and issuing regulations on the marketing and advertising of tobacco products—are funded through tobacco user fees, as required by the Tobacco Control Act. FDA’s Center for Tobacco Products (CTP), which was established by the act, is responsible for executing FDA’s tobacco regulation responsibilities. Within CTP, the two main offices involved in carrying out FDA’s tobacco user fee responsibilities are the Office of Management and the Office of Compliance and Enforcement. CTP’s Office of Management staff duties include—but are not limited to—calculating individual tobacco manufacturer’s and importer’s market share quarterly within each tobacco product class, as well as completing FDA’s year-end reconciliation process to ensure its market share calculations for each fiscal year are based on complete and accurate data. CTP’s Office of Compliance and Enforcement staff are involved in FDA’s efforts to implement and enforce the Tobacco Control Act by (1) informing tobacco manufacturers and importers that they must pay the required quarterly tobacco user fee, if they have not done so, by the due date, and (2) working to obtain voluntary compliance, or taking advisory or enforcement actions, when manufacturers or importers continue to fail to comply with the user fee requirements. FDA’s Office of Financial Management is responsible for calculating the quarterly assessments for each tobacco product class, and for activities related to the billing and collection of tobacco user fees. For example, FDA’s Office of Financial Management generates quarterly invoices for individual manufacturers and importers based on CTP’s market share calculations. Additionally, this office processes tobacco user fee payments received and works with CTP’s Office of Compliance and Enforcement to help collect user fee payments from tobacco manufacturers and importers that do not pay a quarterly assessment by the due date. The Tobacco Control Act establishes requirements regarding the calculation, billing, and collection of tobacco user fees. Calculation. For each fiscal year, total user fees are to be allocated in 1. Class allocation. The amount of total user fees for a fiscal year (e.g., $635 million for fiscal year 2017) is allocated among the different tobacco product classes subject to user fees; this allocation is based on each class’s share of the gross domestic volume of tobacco products introduced into the U.S. market. 2. Individual allocation. The amount of user fees allocated to each manufacturer or importer is proportional to its market share within a given class of tobacco products. For example, a manufacturer with 50 percent of the cigarette market would be required to pay 50 percent of user fees allocated for the cigarette product class. The act specifies that no manufacturer or importer of tobacco products shall be required to pay a user fee in excess of the percentage share of such manufacturer or importer. See figure 1 for a summary of the tobacco user fee allocation process under the Tobacco Control Act. Notifications to each manufacturer or importer of the amount of its quarterly tobacco user fee assessments are to be sent at least 30 days before the end of the quarter for which the assessment is made. Collection. Tobacco user fee payments are due the last day of each quarter. If a manufacturer or importer does not pay its user fee assessments by the last day of the relevant quarter, the act states that tobacco product shall be deemed adulterated. Since the enactment of the Tobacco Control Act, FDA has issued several final rules (regulations) regarding its process to calculate, bill, and collect tobacco user fees, including the following: In 2014, FDA issued a final rule requiring tobacco manufacturers and importers to submit to FDA the information needed to calculate individual tobacco user fees, starting with fiscal year 2015. This rule applied to the four classes of tobacco products FDA initially regulated: cigarettes, snuff, chewing tobacco, and roll-your own tobacco. Fiscal year 2015 was the first year for which FDA obtained the data directly from manufacturers and importers to calculate individual tobacco user fee assessments. In 2016, FDA issued a final rule extending FDA’s regulatory authority to all tobacco products, including pipe tobacco and cigars (but excluding accessories of newly deemed products). Using its deeming authority, FDA issued another final rule requiring that pipe tobacco and cigar manufacturers and importers submit to FDA the information required to calculate user fees for these tobacco product classes. FDA began collecting tobacco user fees from the pipe tobacco and cigar classes in fiscal year 2017. See appendix I for a timeline of events related to FDA tobacco product user fees. FDA’s process for calculating, billing, and collecting user fees involves five steps. First, FDA collects the data needed to calculate the quarterly user fee allocations for each tobacco product class and, within each class, for individual manufacturers and importers. For its quarterly class allocation calculations, FDA collects data on the total volume (units) of tobacco products introduced into the U.S. market for each tobacco product class from the Department of the Treasury’s Alcohol and Tobacco Tax and Trade Bureau (TTB)—these data are published on the TTB website. FDA also collects data from individual manufacturers and importers on the volume of and federal excise taxes paid for their tobacco products introduced into the U.S. market in each product class. Tobacco companies submit these data to FDA as part of required monthly report submissions. Second, FDA uses the TTB data it collected to calculate the quarterly class allocations. Third, FDA calculates the user fees owed by individual manufacturers or importers within a given product class, based on their market share in each tobacco product class and the quarterly class allocation it previously calculated. Fourth, FDA bills—that is, generates and mails user fee invoices to—tobacco product manufacturers and importers each quarter. Fifth, FDA collects user fee payments. User fees that are not received by FDA by the last day of the quarter are considered late, and are subject to financial charges beginning 30 days past the invoice due date and for each 30-day period that the assessment remains unpaid. Figure 2 shows the steps in FDA’s process to calculate, bill, and collect user fees. From fiscal year 2015—the first year that FDA obtained data directly from manufacturers and importers to calculate user fee assessments—through fiscal year 2017—the most recently available data at the time of our analysis—FDA assessed and collected about $1.8 billion in tobacco user fees. During this time, the vast majority of the total user fees assessed and collected each fiscal year were from manufacturers and importers of cigarettes. See figure 3 for user fees that FDA assessed, by product class, for fiscal years 2015 through 2017. FDA’s process is designed to ensure the quarterly user fees it calculates, bills, and collects each fiscal year are complete and accurate. This process is also designed to ensure user fee invoices are billed to tobacco manufacturers and importers in a timely manner and to help the agency ensure user fee payments are collected in a similar manner. Additionally, FDA has designed procedures to retroactively adjust its quarterly individual user fee calculations to include relevant excise tax data not reported to FDA at the time these calculations were completed. The agency’s year-end reconciliation process is designed to make these adjustments to ensure that the user fees assessed for a given fiscal year are complete and accurate. Calculation. FDA’s process related to its quarterly individual user fee calculations includes procedures to ensure its individual quarterly user fee assessments are complete and accurate. Tobacco manufacturers and importers provide monthly reports to FDA on the volume of and excise taxes paid on tobacco products introduced into the U.S. market, and those data are reviewed by CTP’s Office of Management for accuracy. If CTP identifies incomplete data or inaccurate reporting, it will contact the appropriate manufacturer or importer in an attempt to resolve discrepancies (e.g., differences between what the company reported to FDA and the supporting document it provided) prior to calculating individual market share for the quarterly billing cycle. However, according to agency officials, if the team is unable to resolve any discrepancies by the time it must submit market share percentages to FDA’s Office of Financial Management for the quarterly billing process, it uses the potentially incomplete or inaccurate data for its market share calculations. FDA officials stated that the agency may make adjustments to individual market shares and resulting user fees based on late or amended data it receives from manufacturers and importers after that data is received. While this is an option, FDA generally relies on its year-end reconciliation process to make all adjustments resulting from late or amended data received at one time, according to FDA officials. Billing: FDA’s process related to quarterly tobacco user fee billing includes procedures to ensure the invoices it creates for individual tobacco manufacturers and importers are complete and accurate—based on CTP’s market share percentages calculated using the monthly excise tax data submitted to FDA by manufacturers and importers—and mailed in a timely manner. For example, FDA’s billing procedures provide for quarterly user fee assessments to be calculated automatically in FDA’s Tobacco Billing Portal. Collection: FDA’s process related to the collection of quarterly tobacco user fees includes procedures to help it ensure quarterly tobacco user fee payments received are complete, accurate, and timely recorded. FDA has also designed mechanisms to identify and collect payment from tobacco manufacturers and importers who do not pay their invoices by the quarterly user fee due date (i.e., the last day of the applicable fiscal year quarter). For example, FDA has an internal system that is designed to generate alerts to warn staff of unpaid invoices that are approaching 30, 60, and 90 days past due so FDA can issue notification letters to inform the tobacco manufacturers and importers that their invoices are overdue and provide instructions for making a payment. (See appendix II for additional information on FDA’s process for the calculation, billing, and collection of tobacco user fees.) Outside of its tobacco user fee calculation, billing, and collection cycle, FDA’s procedures state that FDA will review TTB data to develop a list of current tobacco permit holders that may be subject to user fees. According to FDA officials, reviewing this list helps the agency ensure it has included all manufacturers and importers within relevant tobacco product classes in its individual quarterly user fee calculations. FDA procedures state that CTP’s Office of Management contacts the permit holders that have not reported monthly data to FDA, if identified, to inform them that (1) they are required to report monthly data to FDA for purposes of making user fee market share calculations, and (2) the permit holder may be required to pay quarterly tobacco user fees as a result of these data. User Fee Adjustments: FDA has also designed procedures to retroactively adjust its quarterly individual user fee calculations to include relevant excise tax data that were misreported or not reported to FDA at the time these calculations were completed. Individual quarterly user fee assessments are based on the market share of manufacturers and importers within each tobacco product class. As a result, FDA needs to recalculate all individual market share percentages within a given class of tobacco products if it receives new or amended data related to the excise taxes paid by manufacturers and importers in that class, to ensure compliance with the Tobacco Control Act. According to FDA’s procedures, FDA may recalculate its individual quarterly market share percentages to include changes identified by late or amended data submissions from individual tobacco manufacturers and importers, and FDA will recalculate market shares to include changes identified during its year-end reconciliation process. Late or amended data submissions. According to FDA’s procedures, FDA can receive data from tobacco manufacturers and importers that did not previously submit monthly data to FDA and were therefore excluded from FDA’s initial quarterly market share calculations. FDA can also receive late or amended data from tobacco manufacturers and importers that previously reported incomplete or inaccurate monthly data to FDA. According to FDA, in some instances, these late or amended data are data that FDA had requested during its monthly review process, but were received after FDA completed its quarterly market share calculations. According to FDA, companies may also voluntarily provide updated reports that the company itself determined were a correction to previously submitted data. Year-end reconciliation based on annual tax records from TTB and U.S. Customs and Border Protection (CBP). FDA’s procedures state that it will make annual adjustments to user fees for each fiscal year as part of its year-end reconciliation process. FDA’s procedures state that, by FDA request, TTB and CBP will provide an annual report listing the tobacco excise taxes paid by each manufacturer and importer subject to the tobacco user fee requirement. FDA officials stated that FDA submits an annual request to TTB and CBP for their records of the excise taxes paid by each tobacco permit holder in the six relevant tobacco product classes for the prior fiscal year. As of July 2019, FDA officials stated that because TTB and CBP have up to 3 years to update and finalize their data files, CTP plans to update its procedures to include two reconciliation processes for each fiscal year. According to FDA officials, the first reconciliation, the year-end reconciliation process, would begin immediately following the end of a fiscal year, and the second reconciliation would occur 3 years after that fiscal year ends. FDA’s Year-End Reconciliation Process Following the close of each fiscal year, the Food and Drug Administration’s (FDA) Center for Tobacco Products (CTP) initiates the year-end reconciliation process by requesting official records from the Alcohol and Tobacco Tax and Trade Bureau (TTB) and U.S. Customs and Border Protection (CBP). This process is designed to ensure that the tobacco user fees assessed that year are complete—that is, that all manufacturers and importers subject to user fees were assessed user fees—and accurate—that is, that the user fees assessed each quarter were based on accurate market share information. As designed, the year-end reconciliation includes steps for FDA to compare the information the agency used to calculate quarterly user fees with independent information obtained from TTB and CBP on the individual tobacco manufacturers and importers who paid tobacco excise taxes (to ensure FDA has a complete list of those who should pay user fees) and the amounts paid (to ensure FDA used the right amounts to calculate market share and user fees). According to FDA officials, the year-end reconciliation is designed to identify and make any needed corrections to its individual market share calculations based on findings of new, amended, or missing excise tax payments using the annual tax data provided by TTB and CBP (see sidebar). For example, FDA officials stated that FDA will use the data obtained from TTB and CBP to help identify tobacco manufacturers or importers that should have been assessed user fees but were not, due to the companies not reporting monthly data to FDA as required (non- reporters). This process also enables FDA to identify and address fraudulent reporting by tobacco manufacturers and importers who knowingly failed to submit or submitted false information in monthly forms sent to FDA. If FDA recalculates its quarterly market share percentages based on findings of new or amended excise tax data, FDA’s procedures specify that FDA will then make necessary adjustments to individual tobacco user fee assessments. FDA officials stated that FDA can apply necessary market share adjustments to individual user fee assessments in a subsequent quarterly invoicing cycle, or after the agency completes its year-end reconciliation process based on annual tax data from TTB and CBP. According to FDA officials, in order to limit the need to re-invoice companies multiple times outside of the regular billing cycle, FDA prefers to send the adjusted invoices out once the year-end reconciliation process is complete. However, the officials stated that they can make changes outside of the year-end reconciliation. For example, the officials reported making adjustments to individual user fee assessments for the cigar class once, for the first quarter of fiscal year 2017. In that instance, after receiving updated reports from two cigar companies that had initially reported incorrect excise tax data to FDA, FDA officials stated that the agency (1) recalculated the market share of cigar manufacturers and importers based on the amended data FDA received from both companies and (2) made necessary adjustments to the market share percentages and associated user fees for that class for that quarter. According to FDA, the agency has not completed its year-end reconciliation process since completing the reconciliation for fiscal year 2015—the first year that FDA obtained data directly from manufacturers and importers to calculate user fee assessments. FDA designed the year- end reconciliation process to ensure the agency’s individual user fee calculations are based on complete and accurate data and accurately reflect the market share of each tobacco manufacturer and importer. FDA procedures state that FDA will conduct an annual adjustment for each fiscal year using data received from TTB and CBP for individual manufacturers and importers. According to FDA officials, the agency has been unable to complete the reconciliation process for fiscal years 2016 through 2018 because it identified problems with the quality of data it had initially received from TTB and CBP for those years. FDA officials stated that the agency has worked with TTB and CBP, and officials believe they have determined the reasons for the data problems. Specifically, FDA officials said that changes in both the TTB and CBP internal data systems affected the data fields that FDA needs to complete the reconciliation process. As of July 2019, FDA officials had received revised excise tax and volume data for fiscal years 2016 and 2017 from CBP and TTB. They also received revised 2018 data from CBP and had requested, but not yet received, revised 2018 data from TTB. FDA officials said that once they have received the remaining 2018 data and determined that the data from both agencies are of sufficient quality, they will be able to perform the annual reconciliation process for those fiscal years. According to FDA officials, before they can be certain the data are of sufficient quality, the agency needs to modify its internal data system to accommodate a new CBP data format, and then run the data through the updated system. As of July 2019, FDA projects these modifications to its data system will be completed by the end of calendar year 2019. Once the modifications are finished, FDA projects it will complete the reconciliation process for fiscal year 2016 within 3 to 6 months, and then complete the reconciliation for fiscal years 2017 and 2018 in 3- to 6- month intervals consecutively after that. While FDA has identified the steps to perform the year-end reconciliation process for fiscal years 2016 through 2018, it could also face delays in the future, because it does not have reasonable assurance that it will receive quality data from TTB and CBP in a timely manner to complete the reconciliation process for future years. According to FDA officials, their efforts to obtain the data they need from TTB and CBP have focused on fiscal years 2016 through 2018, and they have not determined procedures or time frames for obtaining data from TTB and CBP for future years. However, according to FDA officials, the agency was considering possible actions for obtaining data in future years. One possible option the agency was exploring was the possibility of FDA gaining direct access to CBP’s and TTB’s data systems to obtain the data needed for the year- end reconciliation. According to officials, as of July 2019, CBP had offered this direct access to its data, and the officials expect to pursue this option with TTB officials for similar access. In addition, the agency reported efforts to schedule meetings with TTB and CBP to discuss establishing memorandums of understanding, or other written agreements, that would establish expectations—such as time frames and data format—with the agencies to obtain the quality data needed for the year-end reconciliation. As of September 2019, FDA reported it had scheduled a meeting with CBP officials and was working to schedule a meeting with TTB officials, but the agency had not yet determined procedures or time frames for obtaining the needed data from these agencies for future years. Federal internal control standards call for agencies to use quality information to achieve their objectives. As part of this standard, agencies obtain relevant data from reliable sources in a timely manner and process these data into quality information that supports their internal control system. Federal internal control standards also call for agencies to externally communicate the quality information necessary to achieve its objectives. As part of this standard, agencies communicate quality information externally through reporting lines so that external parties can help the entity achieve its objectives and address related risks. For example, information communicated includes significant matters relating to risks, changes, or issues that impact the agency’s internal controls. Consulting with TTB and CBP, determining procedures and time frames for FDA to receive the quality data it needs in future years, and documenting them in a written agreement would help to address this risk. Without completing the year-end reconciliation process in a timely manner, FDA cannot ensure that the data it uses to calculate individual user fees are complete and accurate. Until it works with TTB and CBP and resolves this issue, FDA is at increased risk that user fees may not be properly assessed on individual tobacco manufacturers and importers based on their market share of each tobacco product class. FDA collects user fees from tobacco manufacturers and importers for its tobacco regulation activities—including important activities such as educating the public about the risks associated with the use of tobacco products and preventing youth use of these products. The agency has designed a process with several steps for assessing these fees, including a year-end reconciliation, to ensure that the calculations are complete and accurate—that is, that all companies subject to user fees pay them, and that no companies are assessed fees in excess of their market share. However, for several years, FDA has faced serious delays obtaining the quality data it needs from TTB and CBP to complete the year-end reconciliations, according to FDA. Until FDA consults with these agencies to determine and document the procedures and time frames that will allow FDA to obtain the quality data it needs to complete this key step in a timely manner, the agency risks repeating these delays. Without performing its year-end reconciliation, FDA is at increased risk of allowing some companies—such as those who did not report information to FDA or who did not report accurate information—to not pay their required share of user fees, while other companies pay too much. The Commissioner of FDA should consult with TTB and CBP to determine and document—for example in Memorandums of Understanding or other written agreements—procedures and time frames for FDA to receive quality data from TTB and CBP that will allow FDA to complete its reconciliation process in a timely manner. (Recommendation 1) We provided a draft of this product to HHS for comment. In its comments, reproduced in appendix III, HHS generally agreed with our recommendation. The agency commented that it recognized GAO’s thorough review of FDA’s tobacco user fee program and stated that it is critically important for FDA to have a tobacco user fee collection program that is accurate, complete, and predictable. FDA also stated that it has prioritized making the necessary enhancements to its internal data system to accommodate the new format of TTB and CBP data files, and that these changes are on track to be completed by the end of 2019. HHS also provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Health and Human Services, FDA Commissioner, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or DeniganMacauleyM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This appendix provides additional information on the Food and Drug Administration’s (FDA) process, which is designed to ensure the quarterly user fees it calculates, bills, and collects each fiscal year are complete and accurate and that user fee invoices are billed to and collected from tobacco manufacturers and importers in a timely manner. Calculation. FDA’s process related to its quarterly user fee calculations include procedures to ensure the tobacco product class and individual manufacturer and importer allocations are accurate. According to FDA’s procedures, at the start of each fiscal year, FDA Office of Financial Management, Division of User Fees staff (1) calculate the percentage share for each tobacco product class and (2) enter these percentage shares into FDA’s User Fee System, which automatically calculates quarterly class allocations for each tobacco class. According to FDA officials, FDA’s Office of Financial Management, Division of User Fees staff, as well as Center for Tobacco Products’s (CTP) Office of Management, User Fee Management Team staff, review the class allocation calculations to verify the percentage shares were accurately calculated for each tobacco product class before entering the class percentages into FDA’s User Fee System. Prior to calculating individual market share percentages that are the basis for individual user fees, the User Fee Management Team within CTP’s Office of Management reviews the monthly data reported to FDA by tobacco manufacturers and importers for accuracy. According to FDA’s procedures, the CTP User Fee Management Team checks to ensure that the volume and excise tax data reported on each FDA form 3852 are accurate based on the accompanying supporting documents. According to FDA procedures, if the CTP User Fee Team identifies incomplete or inaccurate monthly reports, it contacts the appropriate tobacco manufacturers or importers to request the missing documentation or an amended FDA form 3852 and tries to resolve any inaccuracies prior to calculating individual market share for the quarterly billing cycle. Billing. According to FDA’s procedures, the CTP User Fee Management Team submits market share percentages to the FDA Office of Financial Management, Division of User Fees in the month prior to the date that invoices are to be issued. For example, for the first quarterly invoicing cycle (October through December), the CTP User Fee Management Team would submit market share percentages on November 15 and invoices would be mailed by the Division of User Fees by December 1. Using the market share data, the Division of User Fees calculates the quarterly user fee amount assessed to individual manufacturers and importers within each tobacco product class as part of its quarterly invoicing process. FDA officials stated that, prior to creating quarterly invoices, the Division of User Fees reviews CTP market share data to ensure it received all necessary data. Prior to mailing quarterly invoices to individual tobacco manufacturers and importers, FDA officials stated that division staff verifies that the invoices created are complete and accurate by comparing the invoice information to the CTP market share data. Collection. FDA’s Office of Financial Management, Division of User Fees utilizes different mechanisms to identify and notify tobacco manufactures and importers who do not pay their invoices by the quarterly user fee due date (i.e., the last day of the applicable fiscal year quarter). According to FDA’s procedures, the Division of User Fees uses a program within FDA’s User Fee System—referred to as the Dunning Tracker—to track relevant invoice data, including the date user fee payments are due and the amounts owed. The Dunning Tracker is designed to generate alerts to warn division staff of unpaid invoices that are approaching 30, 60, and 90 days past due so they can issue Dunning notification letters—which inform the tobacco manufacturers and importers that their invoices are overdue and provide instructions for making a payment. The Dunning notification letters also inform tobacco manufacturers and importers of the amount of additional charges assessed based on the number of days that the payment is late. According to FDA officials, division staff verify that a Dunning notification letter is issued for each tobacco manufacturer or importer with an outstanding invoice and that the appropriate charges have been assessed. According to FDA officials, the Division of User Fees also maintains an arrears list—a list of tobacco manufacturers and importers who have not paid their quarterly user fees on time. FDA’s procedures provide that the Division of User Fees will share the arrears list with the CTP Office of Compliance and Enforcement to assist that office’s efforts to obtain compliance with the user fee requirements. FDA officials stated that the office monitors the arrears list and takes enforcement action when appropriate. The officials said that the office will first issue information letters, separate from the Dunning notification letters, to each tobacco manufacturer and importer on the arrears list to try to obtain voluntary compliance on the user fee payments owed. FDA officials stated that if the office is unable to obtain compliance after it issues the information letter, it may take further action, such as notifying the delinquent company that all tobacco products manufactured and imported by it are adulterated. Agency officials told us that, in 2014, FDA notified three individual tobacco manufacturers that all the tobacco products they manufactured were adulterated due to these companies’ failure to pay their tobacco user fees. According to FDA’s procedures, the Division of User Fees refers delinquent debt to the Department of Health and Human Services (HHS) Program Support Center when outstanding invoices reach 90 days past due. The Program Support Center will pursue collection efforts per its standard procedures and issues two reports each month to the Division of User Fees to inform it of which debts have been collected and which are uncollectable. In addition to the contact named above, Kim Yamane (Assistant Director), Matthew Byer (Analyst in Charge), Sam Amrhein, Julie Flowers, Jackie Hamilton, Derry Henrick, Vikki Porter, and LaDonna Towler made key contributions to this report.", "summary": "Tobacco use causes more than 480,000 deaths each year, according to the Department of Health and Human Services (HHS). To protect the public, the Family Smoking Prevention and Tobacco Control Act granted FDA, an agency within HHS, authority to regulate tobacco products. To fund FDA's tobacco regulation activities—such as those aimed at preventing youth use of tobacco products—the act authorizes FDA to assess and collect a specified total amount of user fees from tobacco manufacturers and importers each fiscal year. The total amount of user fees are to be allocated based on the individual manufacturers' and importers' market share in six FDA-regulated tobacco product classes. GAO was asked to review FDA's tobacco user fees. This report examines FDA's process for the calculation, billing, and collection of these fees. GAO reviewed the relevant law and regulations, as well as FDA policies and procedures, and interviewed FDA officials. In fiscal year 2017, the latest data available at the time of our analysis, the Food and Drug Administration (FDA) assessed about $635 million in user fees to tobacco manufacturers and importers of six classes of FDA-regulated tobacco products—cigarettes, snuff, chewing tobacco, roll-your-own tobacco, pipe tobacco, and cigars. (See figure.) FDA has a process that is designed to ensure accurate calculation, billing, and collection of tobacco user fees. However, the agency has not completed a key step in this process—its year-end reconciliation—since doing so for fiscal year 2015. FDA procedures provide that the agency will conduct this year-end reconciliation annually after receiving necessary data from the Department of the Treasury's Alcohol and Tobacco Tax and Trade Bureau (TTB) and U.S. Customs and Border Protection (CBP). FDA relies on this year-end reconciliation to ensure that its user fee calculations are based on complete and accurate data—that is, that all manufacturers and importers subject to tobacco user fees were assessed fees correctly, based on accurate market share data. Incomplete or inaccurate data for one manufacturer or importer affects the market share—and the user fee amount—for all other manufacturers and importers in its product class. FDA has not completed this year-end reconciliation in recent years because of delays in obtaining the quality data it needs from TTB and CBP. While FDA has reported receiving most of the data for fiscal years 2016 through 2018 and has plans for completing the reconciliation for those years, the agency faces a risk of repeating delays in its reconciliation efforts in the future because it does not have reasonable assurance that it will receive quality data in a timely manner moving forward. Until FDA consults with TTB and CBP to determine and document the procedures and time frames that will allow FDA to obtain the quality data it needs to complete this key step in a timely manner, the agency risks repeating these delays. GAO is recommending that FDA consult with TTB and CBP to determine and document procedures for FDA to obtain quality data so the agency can complete its annual reconciliation process in a timely manner. HHS agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "Air ambulance providers use either helicopters or fixed-wing aircraft, as shown in Figure 1, depending on where and how far they are transporting patients. Helicopters are generally used for transports from the scene of the accident or injury to the hospital or for shorter-distance transports between hospitals. Helicopter bases may be at hospitals, airports, or other types of helipads, and a provider may need to fly from its base to the scene or a hospital to pick up the patient being transported. Air ambulance providers typically respond to calls for helicopter transports within a certain area around their bases in part to ensure appropriate response times. Fixed-wing aircraft are generally used for longer-distance transports between hospitals. Fixed-wing bases are at airports, and the patient is transported by ground ambulance to and from the airports. Air ambulance providers respond to emergencies without knowing patients’ health insurance coverage, such as whether the patient has private insurance, Medicare, Medicaid, or no insurance. According to our previous analysis of information from eight selected air ambulance providers, in 2016, Medicare patients received 35 percent of helicopter transports, privately-insured patients received 32 percent, Medicaid patients received 21 percent, uninsured patients received 9 percent, and patients with other types of coverage such as automobile and military- sponsored insurance received a small percentage. Relatively few patients receive air ambulance transports, but those patients who do generally have no control over the decision to be transported by air ambulance or the selection of the air ambulance provider, as shown in Figure 2. For privately-insured patients, this means they cannot necessarily choose to be transported by air ambulance providers in their insurers’ network and can potentially receive a balance bill from the providers for the difference between the price charged by the provider and the amount paid by the insurer. This amount is in addition to copayments, deductibles, or other types of cost-sharing that patients typically pay under their insurance. Air ambulance providers are prohibited from sending balance bills to Medicare and Medicaid patients, while uninsured patients might be held responsible by the air ambulance provider for the entire price charged. With many types of health care services, both health care providers and insurers have incentives to negotiate and enter into contracts that specify amounts that providers will accept as payment in full, thereby avoiding the potential for balance bills for those services. Insurers can offer—and health care providers may be willing to accept—payment rates that are much lower than the providers’ charged amounts because the providers may receive more patients as an in-network provider. Furthermore, when patients are choosing insurance plans, they may consider how many or which providers are in-network, particularly for providers such as hospitals or certain physicians. The emergency nature of most air ambulance transports, as well as their relative rarity and high prices charged, reduces the incentives of both air ambulance providers and insurers to enter into contracts with agreed- upon payment rates, which means air ambulance providers may be more often out-of-network when compared with other types of providers. Decisions by first responders and physicians on which air ambulance provider to call are typically not based on the patient’s insurance plan, meaning that being in-network may not increase air ambulance providers’ transport volume. As a result, according to stakeholders we spoke to, if insurers offer payment rates that are much lower than the air ambulance providers’ charged amounts, the air ambulance providers may be less willing than other health care providers to accept those payment rates. Furthermore, given the relative rarity of air ambulance transports, patients may not anticipate needing air ambulance transports and may not choose insurance plans based on which or how many air ambulance providers are in insurers’ networks. Approaches by states or the federal government to limit balance billing may target providers, insurers, or both. Examples of approaches described in research on balance billing include a cap on the amount that providers can charge or a requirement for insurers to pay the full amount charged by providers. However, according to the research, targeting just providers or insurers can result in undesired outcomes. Capping the amount providers can charge could result in insurers that underpay for services, which could lead some providers to reduce service or exit the market altogether. Conversely, requiring insurers to pay the full amount charged by providers could result in providers that overcharge for services, which could lead to higher premiums charged to patients. The authority of states to address issues related to air ambulance balance billing is affected by the following federal laws: Airline Deregulation Act of 1978 (ADA): A provision in this law preempts state-level economic regulation—i.e., regulating rates, routes, and services—of air carriers authorized by DOT to provide air transportation. In general, courts have held that air ambulances are considered to be air carriers under the ADA’s preemption provision, and courts, DOT, and state attorneys general have determined specific issues related to the air ambulance industry that can and cannot be regulated at the state level. McCarran-Ferguson Act of 1945: This act affirmed that states have the authority to regulate the business of insurance. For example, states may review insurers’ health insurance plans and premium rates. In instances of balance billing, states can determine whether the insurer paid a provider in accordance with its policy for paying for out-of-network services. Employee Retirement Income Security Act of 1974 (ERISA): ERISA provides a federal framework for regulating employer-based pension and welfare benefit plans, including health plans. Although states may regulate health insurers, ERISA preemption generally prevents states from directly regulating self-insured employer-based health plans. In 2017, as previously mentioned, we reported on the increase in prices charged by helicopter air ambulance providers and on the lack of data on the factors that may be affecting prices charged. We also found only limited information was available related to several key aspects of the industry, ranging from basic aspects—such as the composition of the industry by type of air ambulance provider, the prices charged by air ambulance providers, and the number of overall transports—to the more complex, such as the extent of contracting between air ambulance providers and insurers or the extent of balance billing to patients. Given DOT’s authority to oversee certain aspects of the industry, we made four recommendations to DOT in 2017 to increase transparency and obtain information to better inform their oversight of the air ambulance industry: (1) communicating a method to receive air ambulance complaints, including those regarding balance billing; (2) taking steps to make complaint information publicly available; (3) assessing available federal and industry data to determine what information could assist in the evaluation of future complaints; and (4) considering consumer disclosure requirements for air ambulance providers, such as established prices charged and the extent of contracting with insurers. DOT has taken steps to respond to the first two recommendations, including adding information to its website describing how air ambulance complaints can be registered and used by DOT. It has also listed the number of air ambulance complaints filed with DOT each month starting in January 2018—23 air ambulance complaints have been filed with DOT through November 2018. DOT has not yet acted on the remaining two recommendations. Air ambulance providers added helicopter bases from 2012 through 2017, according to our analysis of the ADAMS data. Specifically, there were 752 bases in the 2012 data and 868 bases in the 2017 data. When we compared the data for each year, there were 554 bases in both years of data (i.e., existing bases), 314 bases in the 2017 data only (i.e., new bases), and 198 bases in the 2012 data only (i.e., closed bases); the new and existing bases are shown in Figure 3. This addition in bases also increased the total area served by helicopter bases by 23 percent. Several air ambulance providers told us about their decisions to open new bases. For example, one air ambulance provider told us that one way it evaluates the need for a new base in an area is to ask hospitals in that area about the number of transports they typically require and the length of time it takes helicopters to arrive to pick up patients. Along with adding helicopter bases, air ambulance providers also added fixed-wing bases from 2012 through 2017, according to our analysis of the ADAMS data. Specifically, there were 146 bases in the 2012 data and 182 bases in the 2017 data. When we compared the data for each year, there were 114 bases in both years of data (i.e., existing bases), 68 bases in the 2017 data only (i.e., new bases), and 32 bases in the 2012 data only (i.e., closed bases); the new and existing bases are shown in Figure 4. Both the existing and new bases are more prevalent in the Western and Southern parts of the United States. Given that fixed-wing aircraft are used for longer-distance transports and that patients are brought to the base rather than picked up by fixed-wing aircraft, we did not measure the area or any changes in the area served by fixed-wing bases, which are usually airports. Based on our previous work, we further analyzed two trends related to where air ambulance providers have chosen to locate their new bases. New bases in rural areas: About 60 percent of the new helicopter bases and about half of the new fixed-wing bases in the ADAMS data were in rural areas. We previously reported that some helicopter air ambulance providers told us that the lower population density in rural areas leads to fewer transports per helicopter at rural bases. They also said that, despite the lower population density, rural areas may have greater need for air ambulance transports. This may be due to, for example, the closure of some rural hospitals and the establishment of regional medical facilities, such as cardiac and stroke centers that provide highly specialized care. New bases in areas with existing coverage: For just under half of the new helicopter bases in the ADAMS data, the area served overlapped with existing air ambulance coverage by more than 50 percent. On one hand, according to some stakeholders we spoke to, the new helicopters may help enhance available services by, for example, being able to respond to a call if the existing ambulance resources are in use or otherwise unavailable. On the other hand, as we have previously reported, some air ambulance providers told us that when helicopters are added to bases in areas with existing coverage, those helicopters are not serving additional demand. As a result, the same number of transports is spread out over more helicopters, reducing the average number of transports per helicopter. The FAA Reauthorization Act of 2018, which became law in October 2018, requires the FAA to assess the availability of information to the general public related to the location of heliports and helipads used by helicopters providing air ambulance services and to update current databases or, if appropriate, develop a new database containing such information. This could provide additional information about base locations going forward. In the FAIR Health data on air ambulance transports for privately-insured patients, about two-thirds of the approximately 13,100 and 20,700 transports with information on network status were out-of-network in 2012 and 2017, respectively. (See Table 1.) The proportions were similar for both helicopter and fixed-wing transports in each year. The proportion of out-of-network air ambulance transports in the FAIR Health data set is higher than what research shows for ground ambulance transports and other types of emergency services. For example, one study found that 51 percent of ground ambulance transports in 2014 were out-of-network, and the same study and another one found that 14 and 22 percent of emergency department visits in 2014 and 2015 involved out-of- network physicians, even at in-network hospitals. Air ambulance providers and insurers we spoke to confirmed that their proportion of out-of-network transports was high in 2017, but some also reported they have recently been entering into more network contracts. For example, one of the large independent air ambulance providers and a national insurer entered into a contract that covered patients in five states as of August 2018. These contracts could decrease the extent of out-of- network transports and balance billing in the future for these states. Increases in the prices charged for air ambulance transports may exacerbate the financial risks related to balance billing for those with private insurance. In 2017, the median price charged by air ambulance providers for a transport was approximately $36,400 for a helicopter transport and $40,600 for a fixed-wing transport, according to our analysis of FAIR Health data. The prices charged in 2017 were an increase of over 60 percent from 2012, when the median price charged was approximately $22,100 for a helicopter transport and $24,900 for a fixed- wing transport. There is limited information on what insurers pay for out- of-network services. While out-of-network transports may result in balance billing, the FAIR Health data we analyzed do not indicate the extent to which patients received balance bills and, if so, the size of the bills. In addition, as we previously reported, there is a lack of comprehensive national data about the extent and size of balance bills, and air ambulance providers are generally not required to report such data. However, some states have attempted to collect information from patients about balance billing for air ambulance services. Therefore, to provide insights into potential balance bill amounts, we reviewed data on consumer complaints that two of our selected states had received about specific incidents of balance billing for 2014 through 2018. Data for Maryland contained about two dozen complaints with information on the specific amount of balance bills, and those amounts ranged from $12,300 to $52,000. Data from North Dakota contained three dozen complaints with information on the specific amount of balance bills, and those amounts ranged from $600 to $66,600, though all but one amount was over $10,000. Given that providers may agree to reduce amounts that patients would otherwise owe or insurers may increase their payments to providers, along with limited national data, the extent to which patients actually pay the full amounts of balance bills received is also unclear. Generally, officials from air ambulance providers we spoke to said that they first encourage patients to appeal to their insurers for increased payment. If these appeals do not fully address the balance bill, the providers may offer various payment options. For example, officials from one air ambulance provider said that it offers a discount of up to 50 percent off the balance bill if the patient pays the remaining 50 percent immediately. Alternatively, the provider requests detailed financial information—such as income, obligations and debts, and medical bills—to determine whether to potentially offer other discounts or a payment plan. This process can take multiple months, and officials from another air ambulance provider said patients who do not respond to letters and calls may be more likely to be referred to a collections process. Air ambulance providers we spoke with said that they use discretion on how much assistance to offer, and not all patients receive discounts after providing all relevant documentation. Even with discounts, according to data from some air ambulance providers we spoke with, the amount patients pay can still be in the thousands of dollars. Four of our selected states attempted to limit balance billing through the regulation of insurers (Montana, New Mexico, North Dakota, and Texas). Additionally, four states have attempted to limit balance billing through education and public pressure on stakeholders (Florida, Maryland, New Mexico, and North Dakota). Four of the six states we selected—Montana, New Mexico, North Dakota, and Texas—have attempted to limit balance billing by air ambulance providers through the regulation of insurers, as shown in Table 2. Three states have faced challenges in federal district court related to whether their attempts to limit balance billing by air ambulance providers are preempted by the federal ADA. As of January 2019, the case in New Mexico was dismissed on procedural grounds, and the cases in North Dakota and Texas have been decided. The hold-harmless requirement and dispute resolution process established by Montana’s law is an example of how states are attempting to limit balance billing by regulating the business of insurance. Under the hold-harmless requirement, the financial risk for potential balance billing is transferred from patients to the insurer by limiting the patients’ out-of- pocket costs to their cost-sharing responsibilities. However, according to state officials, the dispute resolution process established by this law had not yet been used as of December 2018. The requirement and process apply to transports for patients covered by Montana-regulated insurance plans. It does not apply to transports for individuals in most self-insured plans subject to ERISA, nor does it apply to transports for individuals, such as tourists, covered by insurance plans regulated by other states. The stated purpose of the law establishing this process is to prevent state residents from incurring excessive out-of-pocket expenses in air ambulance situations in a manner that is not preempted by the ADA. Officials in Montana and North Dakota reported receiving fewer consumer complaints about balance billing after implementing their laws to limit balance billing. One reason for this decrease in consumer complaints, according to officials in Montana, was that uncertainty over the possible effects of the law has made most air ambulance providers more willing to enter into contract negotiations with insurers. The officials added that shortly after the law’s enactment, a large insurer and a large air ambulance provider entered into a network contract. Additionally, another air ambulance provider in Montana confirmed that although it had provided out-of-network transports, it had not sent balance bills to patients since the law took effect. Officials in both states could not comprehensively report the extent to which instances of balance billing may have decreased in their state. As required by FAA Reauthorization Act of 2018, the Secretary of Transportation has taken steps to form an advisory committee on air ambulance patient billing. DOT issued a solicitation in December 2018 for applications and nominations for membership on this advisory committee. The committee is to consist of representatives from state insurance regulators, health insurance providers, patient advocacy groups, consumer advocacy groups, and physicians specializing in emergency, trauma, cardiac, or stroke care, among others. The Act directs the advisory committee to issue a report within 180 days of its first meeting and to make recommendations that address the following, among other things: The disclosure of charges and fees for air ambulance services; Options and best practices for preventing balance billing—such as improving network and contract negotiation, dispute resolutions between health insurers and air medical service providers, and explanations of insurance coverage; Steps that states can take to protect consumers consistent with current legal authorities regarding consumer protection; and The recommendations from our 2017 report, including any additional data that DOT should collect from air ambulance providers and other sources to improve its understanding of the air ambulance market and oversight of the industry. Officials in three selected states—Florida, New Mexico, and North Dakota—have provided information to educate consumers and other stakeholders about balance billing for air ambulance transports. The Florida Office of the Insurance Consumer Advocate and the New Mexico Office of Superintendent of Insurance reviewed air ambulance transports in their states and issued public reports with recommendations to improve transparency and education, among other recommendations. Florida’s report, issued in June 2018, recommends that insurers and air ambulance providers improve transparency about the availability of in-network air ambulance providers in a given area and provide information about rate justifications and billing practices to help consumers anticipate potential out-of-network costs. New Mexico’s report, issued in January 2017, recommends educating emergency room physicians and other health care providers about the impact of air ambulance bills on consumers and on how to select in-network air ambulance providers. Additionally, since 2017, the North Dakota Insurance Department has produced a publicly available guide showing which air ambulance providers are in-network with the three insurers in the state. This guide is part of the state’s requirement that, for non-emergency transports, hospitals inform patients about the network status of air ambulance providers. Although the three large independent air ambulance providers we spoke with told us that non-emergency transports comprise only a small percentage of air ambulance transports, officials in North Dakota said some dispatchers and first responders reported using the guide to call in-network air ambulance providers when possible for emergency transports. Finally, one additional selected state—Maryland—has increased public awareness of air ambulance balance billing, which has generated public pressure on air ambulance providers and insurers to encourage the two sides to negotiate contracts. The Maryland Insurance Administration convened a public meeting in September 2015 with the goal of raising public awareness about air ambulance balance billing in the state. The meeting involved statements from patient, air ambulance, hospital, and insurer stakeholders. One of the large independent air ambulance providers said that public pressure following the meeting, as well as subsequent engagement from the state insurance commissioner, were factors in securing a contract with a large insurer in the state. We provided a draft of this report to the Department of Health and Human Services and DOT for review and comment. The Department of Health and Human Services told us they had no comments on the draft report, and DOT provided technical comments that we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Administrator of the Centers for Medicare & Medicaid Services, the Secretary of the Department of Transportation, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact James Cosgrove, Director, Health Care at (202) 512-7114 or cosgrovej@gao.gov or Heather Krause, Director, Physical Infrastructure at (202) 512-2834 or krauseh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the contacts named above, Lori Achman (Assistant Director), Heather MacLeod (Assistant Director), Corissa Kiyan-Fukumoto (Analyst-in-Charge), William Black, George Bogart, Stephen Brown, Krister Friday, Matthew Green, Barbara Hansen, Giselle Hicks, and Vikki Porter made key contributions to this report. Air Ambulance: Data Collection and Transparency Needed to Enhance DOT Oversight. GAO-17-637. Washington, D.C.: July 27, 2017. Air Ambulance: Effects of Industry Changes on Services Are Unclear. GAO-10-907. Washington, D.C.: Sept. 30, 2010.", "summary": "Air ambulances provide emergency services for critically ill patients. Relatively few patients receive such transports, but those who do typically have no control over the selection of the provider, which means privately-insured patients may be transported by out-of-network providers. The Joint Explanatory Statement accompanying the 2017 Consolidated Appropriations Act includes a provision for GAO to review air ambulance services. Among other objectives, this report describes (1) the extent of out-of-network transports and balance billing and (2) the approaches selected states have taken to limit potential balance billing. GAO analyzed a private health insurance data set for air ambulance transports with information on network status and prices charged in 2017 (the most recent data available). Although this was the most complete data identified, the data may not be representative of all private insurers. In addition, GAO interviewed officials in six states (Florida, Maryland, Montana, New Mexico, North Dakota, and Texas) selected in part for variation in approaches to limit balance billing and location. GAO also interviewed air ambulance providers, health insurers, and Centers for Medicare & Medicaid Services and Department of Transportation (DOT) officials. DOT provided technical comments on a draft of this report, which GAO incorporated as appropriate, and the Department of Health and Human Services had no comments. Privately-insured patients transported by air ambulance providers outside of their insurers' provider networks are at financial risk for balance bills—which, as the figure shows, are for the difference between prices charged by providers and payments by insurers. Any balance bills are in addition to copayments or other types of cost-sharing typically paid by patients under their insurance coverage. According to GAO's analysis of the most complete data identified for air ambulance transports of privately-insured patients, 69 percent of about 20,700 transports in the data set were out-of-network in 2017. This is higher than what research shows for ground ambulance transports (51 percent in 2014 according to one study) and other emergency services. Air ambulance providers that GAO spoke with reported entering into more network contracts recently, which could lower the extent of out-of-network transports in areas covered by the contracts. While out-of-network transports may result in balance billing, the data GAO analyzed do not indicate the extent to which patients received balance bills and, if so, the size of the bills. In addition, as GAO reported in 2017, there is a lack of national data on balance billing, but some states have attempted to collect information from patients. For example, GAO reviewed over 60 consumer complaints received by two of GAO's selected states—the only states able to provide information on the amount of individual balance bills—and all but one complaint was for a balance bill over $10,000. Patients may not end up paying the full amount if they reach agreements with air ambulance providers, insurers, or both. The amounts of potential balance bills are informed in part by the prices charged. GAO's analysis of the data set with transports for privately-insured patients found the median price charged by air ambulance providers was about $36,400 for a helicopter transport and $40,600 for a fixed-wing transport in 2017. The six states reviewed by GAO and others have attempted to limit balance billing. For example, the six states have taken actions to regulate insurers, generate public attention, or both. As required by recent federal law, the Secretary of Transportation has taken steps to form an advisory committee to, among other things, recommend options to prevent instances of balance billing.", "document_type": "gao"}
{"report": "Opioids, such as hydrocodone and oxycodone, can be prescribed to treat both acute and chronic pain. Opioids can pose serious risks when they are misused. These risks include addiction, overdose, and death. As a result, opioids are classified as controlled substances, which means that their use and disposal are subject to additional oversight by DEA. Some studies suggest that the majority of patients who received prescriptions for opioids often do not use a large portion of the drugs dispensed. A study that surveyed U.S. adults who had received opioids found that approximately 60 percent of patients who were no longer using the medication had unused opioids. Two studies reported that over one- half of patients did not use all of the opioids prescribed to them after surgery; these studies found that patients reported leaving 15 to 20 pills unused, representing 54 percent to 72 percent of the opioids they were prescribed. Another study on patient opioid use after a cesarean section and thoracic surgery found that most patients, 83 percent and 71 percent respectively, used less than half of the total opioids they were prescribed. There is no federal law or regulation imposing requirements for how patients are to dispose of unused opioids. However, DEA, FDA, and EPA all have authorities and initiatives related to patient disposal of opioids. DEA regulations specify three take-back options that patients can opt to use to dispose of their unused controlled substances: take-back events, permanent collection sites, and mail-back programs. DEA hosts semi- annual events called National Prescription Drug Take-Back Days, where temporary collection sites are set up in locations such as police stations. Advertisements encourage community participation in the events and educate the community on safe disposal of unused medications, including opioids. DEA also registers collectors and provides information to the public about the location of permanent collection sites for take-back, such as at local retail pharmacies or hospital pharmacies, and sets requirements for the provision of postage-paid envelopes that patients can use to mail unused drugs to a collector for destruction. DEA regulations establish a standard for the destruction of controlled substances that applies to DEA registrants, which can destroy opioids on patients’ behalf. DEA registrants include pharmaceutical companies that manufacture controlled substances, health care providers who prescribe them, and pharmacies that dispense them. The standard for destruction requires that controlled substances maintained or collected by DEA registrants be rendered non-retrievable. This means that the physical and chemical conditions of the controlled substance must be permanently altered, thereby rendering the controlled substance unavailable and unusable for all practical purposes. According to DEA, as of May 2019, the only method currently used to meet this standard is incineration, and DEA rulemaking states that DEA will not evaluate, review, or approve methods used to render a controlled substance non-retrievable. FDA has broad authority under the Federal Food, Drug, and Cosmetic Act to evaluate whether a drug is safe and effective and ensure the benefits of drugs outweigh the risks. FDA may require manufacturers to develop a risk evaluation and mitigation strategy (REMS) for drugs with serious safety risks, including the risk of abuse, to ensure that the benefits outweigh the risks. Under one REMS, for example, manufacturers of opioids intended for outpatient use must make training available to health care providers involved in the treatment and monitoring of patients who receive opioids. The training must contain certain elements, including how providers should counsel patients and caregivers about the safe use and disposal of these opioids, among other things. In October 2018, the SUPPORT Act authorized FDA to, at its discretion, require specific packaging or disposal systems as a part of certain drugs’ REMS. For drugs with a serious risk of overdose or abuse, FDA may require the drug to be made available for dispensing to certain patients with “safe disposal packaging” or a “safe disposal system” for purposes of rendering the drug non-retrievable in accordance with DEA regulations. Before imposing these requirements, FDA must consider the potential burden on patient access to the drug and the health care delivery system. As of May 2019, FDA had not imposed any REMS requirements using the new SUPPORT Act authority. Under the Resource Conservation and Recovery Act (RCRA), EPA has authority to regulate the generation, transportation, treatment, storage, and disposal of hazardous waste, including certain discarded opioids. However, hazardous waste pharmaceuticals generated by households are not regulated as hazardous waste even if the waste would otherwise be considered hazardous. Opioids and other household waste pharmaceuticals collected through a take-back option are also exempt from most hazardous waste regulations, provided certain conditions are met. Some states and localities have imposed additional requirements for pharmaceutical disposal, such as requirements for drug manufacturers to manage or fund the disposal of collected household pharmaceuticals. According to DEA, FDA, and EPA, patients should use take-back options to dispose of unused opioids, whenever feasible. Only if take-back options are not feasible, FDA recommends flushing opioids on FDA’s flush list down the toilet to remove them from the home as soon as possible. For opioids not on the flush list, the agencies recommend placing the drugs in the household trash mixed with an unpalatable substance. (See fig. 1). Officials from FDA said that the primary goal of these recommendations is to remove dangerous substances from the home as soon as possible to reduce accidental poisoning, which also may address issues related to intentional misuse. FDA officials explained that the agency has not measured the effects of its recommendations for disposing of opioids on opioid misuse, as of May 2019, because it is difficult to establish a causal link between the recommendations and any reductions in misuse. DEA, FDA, and EPA recommend using a take-back option as the preferred method for patients to dispose of unused prescription opioids. Under this method, patients can bring unused opioids to DEA’s semi- annual take-back events or to DEA-registered permanent collection sites, or use mail-back to deliver opioids to a DEA-registered collector for destruction. When patients use these take-back options, the drugs they dispose of are ultimately incinerated, which is the only method that DEA officials said is known to render the drugs non-retrievable, that is, permanently and irreversibly destroyed. Our analysis of DEA and U.S. Census Bureau data shows that as of April 2019, 71 percent of the country’s population lived less than 5 miles from a permanent collection site, and in 42 states, at least half of the population lived within 5 miles of a site. (See fig. 2). This number has increased since our April 2017 report, when we found that about half of the country’s population lived less than 5 miles away from a site. Our analysis also shows that 90 percent of the population lived within 15 miles of a site, though in rural areas only 57 percent lived within 15 miles. In addition, two studies found that patients were willing to bring unused opioids to a take-back location as long as it was located within 5 to 8 miles of their home address. If take-back options are not feasible, FDA recommends flushing the opioids on its flush list down the toilet, because a single dose can be fatal to a child or a pet. Flushing is a permanent way to remove opioids from the home. FDA confirmed that as of June 2019, 11 of 14 drugs on the flush list are opioids, which represents about three-quarters of the approved opioid active ingredients intended for outpatient use (see sidebar). Some portion of drugs that are flushed down the toilet ultimately enter surface and wastewater streams. However, a 2017 FDA study on the environmental impact of drugs listed on the flush list concluded that flushing these opioids has negligible effects on the environment and human health, particularly relative to the amount of opioids that are excreted after taking them as prescribed, because not all of the drug is metabolized. (See text box for a summary of the effects of disposal options on the environment.) Environmental Effects of Disposal Options The environmental impact of opioid disposal depends on the method used—take-back options, flushing, or trash. According to Environmental Protection Agency (EPA) and Drug Enforcement Agency (DEA) officials, disposal of drugs through take-back options results in disposal by permitted incineration, which fully destroys the active form of the drugs. EPA officials told us that flushing or placing opioids in the trash can introduce active opioids into wastewater streams, groundwater, and surface waters. Incineration of Drugs from Take-Back Options. Opioids disposed of using take-back options are destroyed by incineration, which, according to DEA officials, is the only method currently used to meet its non-retrievable standard for destruction. EPA officials told us that based on data from DEA, the amount of household pharmaceutical waste gathered and incinerated during DEA’s semi-annual take-back events is small compared to the total amount of waste one incinerator burns on an average day. EPA officials recommended take-back options as the preferred method of opioid disposal. Flushing. Opioids enter the water supply when excreted by patients who take opioids as prescribed and when patients intentionally flush unused opioids down the toilet. EPA officials told us that most wastewater treatment facilities are not designed to eliminate opioids from wastewater streams. Further, measureable concentrations of opioids have been reported in surface and ground water sources around the world. Trash. Disposal of unused opioids in the trash often introduces opioids into landfills. Studies in scientific literature show that pharmaceutical ingredients have been observed in the water that passes through landfills, called leachate. Similar to opioids that are flushed, opioids in landfill leachate can end up in wastewater streams and other water sources, according to EPA officials. If an opioid is not on the FDA flush list and a take-back option is not feasible, the agencies direct patients to take a series of steps to dispose of their opioids in household trash by: (1) mixing the drugs in an unpalatable substance such as dirt, cat litter, or used coffee grounds, (2) placing the mixture in a sealed container or plastic bag, and (3) throwing the container in the trash. An EPA official said that mixing the drugs with an unpalatable substance is meant to deter misusers from searching through the trash to retrieve the drugs. Disposal of opioids in the trash—either with an unpalatable substance or in-home disposal product—removes them from the home, but this option may not be permanent and the drugs still may be available for misuse. Drugs that are disposed in the trash ultimately are introduced to landfills, where they can escape landfill containment and enter wastewater streams or ground water sources. FDA’s website notes the availability of commercial products for disposing of unused opioids and other drugs in the home. FDA officials stated that, as of May 2019, the agency had not evaluated the effectiveness of these products or made any recommendations related to their use, but they are aware that patients may opt to use these products. These products, known as in-home disposal products, are proprietary substances that patients can mix with their unused drugs, including opioids, before disposing of them in the trash. In-home disposal product vendors told us they sell or donate their products to pharmacies, local law enforcement, and community groups, which then distribute them to patients. A representative from a group that distributes these products, the AmerisourceBergen Foundation, noted that in-home disposal products may be a convenient option for patients for whom take-back options are not feasible, and marketing materials from a product vendor instruct patients to use their product if a take-back option is not available. Vendors indicate that their products can prevent misuse of opioids by rendering drugs non-retrievable at home and by motivating patients to dispose of unused opioids. According to DEA officials, rendering opioids non-retrievable by using an in-home disposal product is challenging, because the drugs have a variety of chemical and physical properties and potencies. Furthermore, according to DEA officials, a lethal dose of fentanyl can be as low as 250 micrograms in adults—and lower in children—underscoring the importance of effective disposal. Some vendors have presented evaluations of their commercial products. A recent comprehensive review of eight in-home disposal products raised concerns about the credibility of vendors’ evaluations and concluded that additional independent laboratory analysis is needed to fully examine product performance and assess how well these products achieve stated goals. Our review of evaluations from three vendors found that the studies contained some inconsistencies and gaps in the evaluation methods used, raising questions about the studies’ conclusions that the products are effective for disposing of opioids. In some cases, studies included detailed, but inconsistent, methods. For example, in four studies about one product, the researchers concluded that the product deactivated most of an opioid dissolved in water. However, one of the earlier studies reported that whole pills did not dissolve in water, which could impact the results, but later studies did not include similar data. In other cases, companies’ evaluations were summaries of results that did not provide enough information to independently verify or assess whether the products deactivate opioids and prevent misuse. For example, one company’s research documents presented images of a mixture as evidence that the drugs had degraded, rather than results of a test measuring if drugs were still detectable. In addition, the studies included little information about the products’ effectiveness at treating mixtures of multiple drugs at the same time, a scenario that stakeholders have referred to as “real world” use testing. Disposal methods—when patients use them promptly—remove unused opioids from the home and therefore can be effective at reducing opioid misuse. FDA officials said that the federally recommended methods for disposing unused opioids are intended to remove these substances from the home as soon as possible, and stated that as long as individuals dispose of opioids promptly rather than storing them, then FDA has achieved its goal. However, the studies we reviewed suggest that most patients do not dispose of unused opioids using a federally recommended method. Specifically, three studies examined how patients disposed of unused opioids and found that between 12 percent and 41 percent of patients disposed of them using a federally recommended method. For example, one of the studies found that of 570 survey respondents who had unused opioids, 12 percent of respondents reported using a take-back option, 14 percent reported that they flushed them down the toilet, and 6 percent reported that they threw them in the trash after mixing with an unpalatable substance. Other studies we reviewed show that take-back options are often used to dispose of drugs other than opioids. Two studies found that less than 10 percent of the catalogued drugs brought to DEA take-back days were controlled substances, which included opioids, while another study weighed drugs brought to take-back events and permanent collection sites and reported less than 3 percent were controlled substances, including opioids. The same study found that annually, controlled substances disposed of at take-back events and permanent collection sites accounted for about 0.3 percent of those dispensed in the area, and concluded that take-back events may have a minimal impact on reducing the availability of unused opioids for misuse. Studies indicate that patients who receive an in-home disposal product may be more likely to dispose of unused opioids, but they may also be less likely to use federally recommended options like take-back or flushing. Two studies in our review found that patients who receive an in- home disposal product have reported that they are more likely to dispose of unused opioids than those who did not receive the product. Use of in- home disposal products—which may not be effective at permanently destroying drugs—may deter patients from using federally recommended options, like take-back, that have been proven effective. For example, one of these studies found that only one of the 70 patients who received an in-home disposal product used a take-back option for disposal, despite the study taking place in a state where we estimated that 77 percent of the population lived less than 5 miles from a permanent collection site. Studies indicate that patients are often unaware of federally recommended disposal options. Three of the 25 studies we reviewed suggest that many patients were not aware of federally recommended methods for disposing of opioids. For example, a study of cancer patients who received opioid prescriptions reported that more than three- quarters of these patients were unaware of proper opioid disposal methods. Another 2016 study of 1,032 patients found that nearly half of the respondents did not recall receiving information on proper disposal from pharmacists, medication packaging, or media outlets. Studies also indicate that patients choose not to dispose of unused opioids, and that they knowingly participate in the majority of opioid misuse. Five of the studies we reviewed found that between one-quarter and three-quarters of patients stored unused opioids for future use or had misplaced their unused opioids. For example, one of these studies found that 49 percent of survey respondents kept or planned to keep unused opioids for future use, and 14 percent were likely to let a family member use their opioid medications in the future. Federal data about the sources of misused opioids indicate that patients are complicit with most misuse. SAMHSA estimates that 5 percent of people nationwide who misused opioids in 2017 took these drugs from someone else without asking. In contrast, SAMHSA estimates that 85 percent of opioid misuse occurs with the patient’s knowledge or active participation, either through the patient misusing his or her own prescription by taking the drug for pain other than for which it was prescribed or by giving or selling the prescribed opioids to another person. (See fig. 3). To motivate patients to use federally recommended methods to dispose of unused opioids, FDA and some physician organizations have created educational materials on safe disposal methods. For example, FDA launched a public awareness campaign called “Remove the Risk” on April 25, 2019—complete with educational materials such as public service announcements, social media posts, fact sheets, and other web-based content. AMA representatives reported that the AMA has provided physicians with educational material on drug disposal and prescribing. Specifically, AMA representatives told us that the association has compiled a two-page document for physicians containing information about drug disposal, links to DEA information on nearby permanent collection sites and take-back events, and FDA guidance on safe disposal of medications. This document included recommendations for physicians to talk to patients about safe use of prescription opioids, remind patients to store their medications in a safe place out of reach from children, and have a conversation with patients about the most appropriate ways to dispose of expired, unwanted, or unused opioids. The AmerisourceBergen Foundation has also partnered with communities to promote safe opioid disposal by providing education about take-back options and commercial in-home disposal products to patients. A representative from the Foundation explained that its Safe Disposal Support Program provides non-profit organizations or municipalities with commercial in-home disposal products, which then can be distributed free of charge to other organizations, individuals, or households. It also recommends that patients use take-back options when available. The representative said that organizations are to demonstrate to patients how these products work either through a brief in-person demonstration at an event or through a video. According to the representative, these products and demonstrations help people reflect on what is in their home and needs to be disposed of, either using a product or a take-back option. Despite such efforts, little is known about the extent to which stakeholders’ efforts to educate the public are effective in increasing use of federally recommended disposal methods. FDA officials said that they are not aware of the extent to which providers are familiar with all disposal methods or the extent to which providers discuss the importance of proper disposal with patients. As part of FDA’s REMS requirements for outpatient opioids, manufacturers must make training available to health care providers involved in the treatment and monitoring of patients who receive opioids, which includes information about the need to communicate with patients about disposal of unused drugs. FDA officials said that opioid manufacturers must assess the effectiveness of their REMS, including an assessment of prescribers’, other health care providers including pharmacists’, and patients’ understanding of the key risk messages conveyed through the educational materials. FDA expects to receive the next REMS assessment with the results of these analyses in 2020. The AMA has not been able to measure the effects of its recommendations, but provided anecdotal feedback from its members that many physicians do not consistently speak to their patients about disposal. FDA officials and AMA representatives indicated that in addition to educating patients on opioid disposal methods, focusing efforts on reducing the amount of unused opioids would be an effective approach for reducing misuse and abuse. For example, FDA officials said that adding packaging configurations that contain smaller quantities of certain opioids could help prescribers to more carefully consider the amount of opioid pain medication they prescribe. This in turn may reduce the number of unused opioids available in the home that could be inappropriately accessed by family members or visitors, and could potentially reduce the risk for misuse and abuse. Representatives from the AMA explained that it and other organizations are working to provide opioid prescribing resources and guidance to help physicians effectively manage patients’ pain, which representatives said will reduce the number of unused opioids available for misuse. FDA officials and a researcher also noted that dispensing opioids in packaging that makes it easy to count the number of unused pills may help patients identify intentional misuse. The FDA and EPA provided technical comments on a draft of this report, which we incorporated as appropriate; the DEA did not have comments. We are sending copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, the Administrator of the DEA, the Administrator of the EPA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. James Cosgrove, (202) 512-7114 or cosgrovej@gao.gov. In addition to the contact named above, individuals making key contributions to this report include Leslie V. Gordon (Assistant Director), A. Elizabeth Dobrenz (Analyst-in-Charge), Sam Amrhein, Jieun Chang, Diana Chung, Kaitlin Farquharson, and Dennis Mayo. Also contributing were Giselle Hicks, Cynthia Khan, and Ethiene Salgado-Rodriguez.", "summary": "In 2017, an estimated 11.1 million Americans misused a prescription pain reliever, which included opioids. This misuse contributes to opioid abuse and death, which has quintupled from 1999 to 2017; about 17,000 people died from prescription opioid overdoses in 2017. Government agencies and stakeholders have attempted to address the potential for misuse and abuse by facilitating safe disposal of unused prescription opioids and other drugs. The SUPPORT for Patients and Communities Act enacted in 2018 included a provision for GAO to review patient disposal of unused opioids, among other things. This report examines (1) federally recommended and other available methods patients may use to dispose of unused prescription opioids, and (2) what is known about patients' use of these methods. To do this work, GAO examined peer-reviewed, academic literature on outcomes for prescription opioid disposal; reviewed federal agency documentation; interviewed federal agency officials, independent researchers, and stakeholder group representatives—such as those from the American Medical Association; and analyzed DEA data as of April 2019 on permanent drug collection sites. GAO also interviewed representatives of three companies that manufacture commercial in-home disposal products and reviewed publicly available documents about these products. The Food and Drug Administration (FDA), Drug Enforcement Administration (DEA), and Environmental Protection Agency (EPA) recommend that patients dispose of unused presciption opioids by bringing them to DEA-registered collection sites or a DEA take-back event, or using mail-back programs. As of April 2019, 70 percent of the U.S. population lived less than 5 miles from permanent collection sites, which are often located at pharmacies. If collection sites, take-back events, or mail-back programs are not feasible, FDA recommends quickly and permanently removing the most dangerous prescription opioids, such as hydrocodone and fentanyl, from the home by flushing them down the toilet. For all other prescription opioids, the agencies recommend disposal in the trash after mixing them with unpalatable substances, such as cat litter. Commercial products to facilitate in-home disposal also exist, and FDA is aware that patients may opt to use these products for disposal in the trash. Available studies suggest that many patients are unaware of federally recommended disposal methods or choose not to dispose of unused prescription opioids. For example, five studies found that between one-quarter and three-quarters of patients stored unused opioids for future use or had misplaced their unused opioids. Further, federal data indicate that 85 percent of intentional misuse occurs with the patient's knowledge—for example, when a patient sells or gives away unused prescription opioids. To educate and motivate patients to dispose of unused opioids, FDA launched a public awareness campaign called “Remove the Risk” in April 2019. Also, FDA and other stakeholders have created educational materials for patients and providers on safe opioid disposal.", "document_type": "gao"}
{"report": "First, DOD is facing substantial supply chain challenges that are hindering the readiness of the F-35 fleet. Specifically, spare parts shortages throughout the F-35 supply chain are contributing to F-35 aircraft being unable to perform as many missions or to fly as often as the warfighter requires. The F-35’s unique supply chain is central to DOD’s strategy to sustain the growing fleet. Rather than owning the spare parts for their aircraft, the Air Force, Navy, and Marine Corps, along with international partners and foreign military sales customers, share a common, global pool of parts. This construct for the F-35 supply chain was intended to ease the logistical burden and provide economies of scale for the military services and international partners; however, the global pool does not have enough spare parts. Specifically, from May through November 2018, F-35 aircraft across the fleet were unable to fly about 30 percent of the time due to parts shortages, as compared with a program target of 10 percent. Below is pictured an F-35B aircraft conducting training aboard a ship. Our work found that several factors contribute to these parts shortages, including F-35 parts that are breaking more often than expected, and DOD’s limited capability to repair parts when they break. Specifically, as of April 2019, the F-35 program was failing to meet four of its eight reliability and maintainability targets—which determine the likelihood that the aircraft will be in maintenance rather than available for operations— including metrics related to part removals and part failures. For instance, we reported at that time that the special coating on the F-35 canopy that enables the aircraft to maintain its stealth had failed more frequently than expected, and the manufacturer was unable to produce enough canopies to meet demands. These reliability challenges are exacerbated by DOD’s limited capability to repair broken parts at the military depots. The capabilities to repair parts are currently 8 years behind schedule. DOD originally planned to have repair capabilities at the depots ready by 2016, but as we reported in April 2019, the depots will not have the capability to repair all parts at expected demand rates until 2024. As a result, the average time taken to repair an F-35 part was more than 6 months, or about 188 days, for repairs completed between September and November 2018—more than twice as long as planned. At that time, there was a backlog of about 4,300 spare parts awaiting repair at depots or manufacturers. We have also reported on other challenges that DOD faces related to its supply chain, including challenges in supporting deployed F-35 aircraft around the world, in clarifying how scarce parts will be distributed, in establishing a plan for a global supply chain network, and in maintaining accountability for spare parts. Figure 2 depicts many of these and other challenges that DOD faces related to the F-35 supply chain. DOD has not fully implemented seven of our recommendations related to its supply chain challenges: Revise sustainment plans: In October 2017, we reported that DOD’s reactive approach to planning for and funding the capabilities needed to sustain the F-35 resulted in significant readiness challenges— including delays in the establishment of part repair capabilities at the depots—and placed DOD at risk of being unable to leverage the capabilities of the aircraft it had purchased. We recommended that DOD revise its sustainment plans to ensure that they include the key requirements and funding needed to fully implement the F-35 sustainment strategy. Conduct a comprehensive review of the F-35 supply chain: While DOD had ongoing efforts to increase the availability of spare parts, we found in April 2019 that DOD would likely continue to face challenges because the program was not planning for the quantity of parts necessary in its spare parts projections to meet warfighter requirements. Simply purchasing more F-35 parts may not be a viable solution for DOD, given the affordability concerns the program faces. These complex problems necessitate a comprehensive approach by DOD, or it is at risk that the F-35 will not be able to conduct the full range of intended missions. We recommended that DOD conduct a comprehensive review of the F-35 supply chain to determine what additional actions are needed to close the gap between warfighter requirements for aircraft performance and the capabilities that the F- 35 supply chain can deliver, in light of the U.S. services’ affordability constraints. Develop a process to modify the afloat and deployment spare parts packages: DOD purchases certain packages of F-35 parts years in advance to support aircraft on deployments, including on ships—called afloat and deployment spare parts packages. In April 2019, we reported that continued modifications to parts and aircraft can make such packages out-of-date by the time F-35 units deploy, and that the F-35 program did not have a process and funding in place to change out mismatched parts. This could put the military services at risk of not having the parts they need to support future deployments. We recommended that DOD develop a process to modify afloat and deployment spare parts packages, to include reviewing the parts within the packages to ensure that they match deploying aircraft and account for updated parts demand, and aligning any necessary funding needed for the parts updates. Mitigate risks related to operating and sustaining the F-35 in the Pacific: In March 2018, we issued a classified report on DOD’s initial transfer of F-35s to a Marine Corps base in Japan that, among other things, described the warfighting capabilities the F-35 brought to the Pacific and assessed operational challenges the Marine Corps faced. In April 2018, we publicly reported on the recommendations from this classified report, including our recommendation that the Marine Corps assess the risks associated with key supply chain-related challenges related to operating and sustaining the F-35 in the Pacific, and that it determine how to address those risks. Revise the business rules for prioritizing scarce F-35 parts: In April 2019, we reported that there was uncertainty about how the program will prioritize scarce F-35 parts among global participants. While the F-35 program had developed a set of business rules, those rules lacked clarity and detail. Absent comprehensive business rules, the F-35 program could face challenges in transparently allocating parts to support competing U.S. and international requirements. We recommended that DOD revise the business rules for the prioritization of scarce F-35 parts across all program participants so as to clearly define the roles and responsibilities of all stakeholders, the process for assigning force activity designations, and the way in which deviations from the business rules will be conducted. Complete a detailed plan for the establishment of the global network for moving F-35 parts: In April 2019, we reported that DOD’s networks to move F-35 parts around the world to the United States and international participants were immature. Because the F- 35 program did not fully recognize the complexity of establishing a global network for moving F-35 parts, this network is now several years behind schedule and there is risk that it will not be fully capable to support an expanding fleet. We recommended that DOD complete a detailed plan for the establishment of the global network for moving F-35 parts that outlines clear requirements and milestones to reach full operational capability, and that includes mechanisms to identify and mitigate risks to the F-35 global spares pool. Clearly establish how DOD will maintain accountability for F-35 parts: In April 2019, we reported that in its rush to field aircraft and its heavy reliance on the prime contractor, DOD had not consistently followed DOD guidance for property accountability. Simply put, DOD did not have records of all the F-35 spare parts it had purchased; where those parts were located; and how much the military services had paid for them. We recommended that DOD issue a policy consistent with DOD guidance that clearly establishes how DOD will maintain accountability for F-35 parts within the supply chain, and identify the steps needed to implement the policy retrospectively and prospectively. DOD concurred with these recommendations and has made some progress in addressing them, including issuing a revised life cycle sustainment plan in January 2019. In addition, DOD has taken actions to increase the availability of spare parts, such as efforts to improve the reliability of parts and incentivize manufacturers to repair parts. Second, DOD continues to face challenges with the F-35’s Autonomic Logistics Information System (ALIS). ALIS is a complex information technology system supporting operations, mission planning, supply-chain management, maintenance, and other processes. It is intended to provide the necessary logistics tools to F-35 users as they operate and sustain the aircraft. For supply chain management, for example, ALIS is supposed to automate a range of supply functions—including updating the status of parts, generating supply work orders, and communicating critical data about parts. However, we reported in April 2019 that these capabilities were immature, resulting in numerous challenges and the need for maintainers and supply personnel at military installations to perform time-consuming, manual workarounds in order to manage and track parts. We reported that one Air Force unit estimated that it spent the equivalent of more than 45,000 hours per year performing additional tasks and manual workarounds because ALIS was not functioning as needed. In our prior work we identified several challenges associated with ALIS, including the following examples (see table 1). We have made six recommendations since 2014 to help DOD address ALIS-related challenges. DOD generally concurred with these recommendations. It addressed two by developing a plan that prioritizes ALIS risks and creating a training plan for ALIS. However, DOD has not taken action on four of our recommendations. These are: Establish a performance-measurement process: In September 2014, we reported that ALIS had experienced recurring problems, including user issues and schedule delays, and was a risk that could adversely affect DOD’s sustainment strategy. But we found that DOD did not have a process to determine and address the most significant performance issues with ALIS based on user requirements, which could limit its ability to effectively and efficiently address performance issues and identify root causes of those issues. We recommended that DOD establish a performance-measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements. Incorporate cost-estimating best practices: In April 2016, we reported that DOD’s $16.7 billion life cycle cost estimate for ALIS was not fully credible since DOD had not performed key analyses as part of the cost-estimating process. We recommended that DOD conduct uncertainty and sensitivity analyses consistent with cost-estimating best practices. Ensure that future cost estimates use historical data: In April 2016, we also reported that DOD’s ALIS cost estimate was not fully accurate because DOD did not use historical cost data, including actual cost data from ALIS and data from other comparable programs. We recommended that DOD ensure that future estimates of ALIS costs use historical data as available and reflect significant program changes consistent with cost-estimating best practices. Test the operation of the F-35 when disconnected from ALIS: In March 2018, we issued a classified report on DOD’s initial transfer of F-35s to a Marine Corps base in Japan that, among other things, described the warfighting capabilities the F-35 brought to the Pacific and assessed any operational challenges the Marine Corps faced. In April 2018, we publicly reported on the recommendations from this classified report, including our recommendation that the F-35 program test operating the F-35 disconnected from ALIS for extended periods of time in a variety of scenarios, to assess the risks related to operating and sustaining the aircraft, and determine how to mitigate any identified risks. We are currently conducting a review of ALIS, assessing how DOD is managing current and future issues related to the system. We plan to complete this review in early 2020. Third, at the core, DOD’s current sustainment challenges have largely resulted from insufficient planning. We have found that DOD lacks information about the technical characteristics and costs of the F-35, which will impair its ability to plan for the long-term sustainment of the F- 35 fleet. The current F-35 sustainment strategy states that the primary contractor will provide logistics support for the aircraft. In October 2017, we reported that while DOD planned to enter into 5-year, fixed-price, performance-based contracts with the prime contractor in the next few years, DOD did not have full information on F-35 technical characteristics or costs to enable it to effectively negotiate those contracts. Specifically, certain technical aspects of the aircraft remained immature or uncertain, including reliability measures that are lagging behind operational requirements. As previously discussed, in April 2019 we reported that the F-35 program was still not on track to meet its targets for four out of eight reliability and maintainability metrics, and that the program had not taken adequate steps to ensure that those targets would be met. DOD officials told us that there would be inherent risk in signing a long-term, performance-based contract before reliability and maintainability data were more fully known, as those data would influence how much aircraft performance should cost. In addition, DOD did not have full visibility into the actual costs of some key sustainment requirements that are considered cost-drivers within the program, such as the actual costs of parts and repairs. Thus, DOD had relied on projected parts reliability and pricing to formulate cost estimates. Actual costs of sustainment requirements can change significantly from initial projections. For instance, we reported that, between the program’s 2014 and its 2015 estimates, the costs of initial spare parts over the life cycle increased by $447 million. The lack of cost information continues to be a challenge for DOD, as we reported in April 2019. DOD officials have stated that they need to know actual costs in order to improve both their confidence in the estimates and their understanding of how cost is related to performance. Below is pictured an F-35A aircraft being refueled. Further, DOD lacks the technical data from the prime contractor needed to fully understand the technical characteristics of the F-35 aircraft and enable potential competition of future sustainment contracts. Technical data include the blueprints, drawings, photographs, plans, instructions, and other documentation required to adequately produce, operate, and sustain weapon systems. Technical data are critical for weapon systems such as F-35 aircraft, as they provide DOD with the information necessary to support the fleet. In April 2019, we found that challenges related to readiness and costs were driving DOD to begin to develop an option for DOD-led supply chain management as a potential alternative to the performance-based contracts through which the prime contractor would provide logistics support. The DOD-led option would require the department to obtain significant amounts of technical data on F-35 parts from the manufacturers of those parts; however, at that time DOD was facing challenges in obtaining the needed data. DOD has not fully implemented 10 of our recommendations related to these issues: Develop a long-term Intellectual Property strategy: In September 2014, we reported that DOD had not identified all of the technical data it needs from the contractor, and at what cost, to enable competition of future sustainment contracts, which put the program at risk of not having the flexibility to make changes to its sustainment strategy. We recommended that DOD develop a long-term Intellectual Property strategy to include, but not be limited to, the identification of current levels of technical data rights ownership by the federal government and all critical technical data needs and their associated costs. Assess whether the program reliability and maintainability targets are still feasible: In April 2019, we reported that the F-35 program continued to fall short of meeting performance targets for half of its reliability and maintainability metrics. Program officials said that those targets need to be reevaluated to determine more realistic performance targets, but they had not taken action to do so. We recommended that DOD assess whether the program’s reliability and maintainability targets are still feasible, and revise accordingly. Identify specific and measurable reliability and maintainability objectives: In April 2019, we reported that the F-35 program’s plan for improving reliability and maintainability did not address the four under-performing metrics. Specifically, the guidance the program has used to implement this plan does not define specific, measurable objectives for what the desired goals for F-35 reliability and maintainability performance should be. As long as these metrics continue to fall short, the military services may have to settle for aircraft that are less reliable and more costly to maintain than originally planned. We recommended that DOD identify specific and measurable reliability and maintainability objectives in its guidance. Link reliability and maintainability improvement projects to the associated objectives: In April 2019, we reported that the F-35 program had not aligned its planned reliability and maintainability improvement projects with reliability and maintainability goals, which could put the program at risk of not meeting those goals. We recommended that DOD identify and document in guidance which reliability and maintainability improvement projects will achieve the identified objectives. Prioritize funding for reliability and maintainability improvement: In April 2019, we reported that the F-35 program office had estimated potential life-cycle cost savings of more than $9.2 billion from implementing the reliability and maintainability improvement projects in its plan, but had not prioritized or dedicated funding in its budget necessary to carry out the projects. As a result, projects had been prematurely suspended or delayed. We recommended that the F-35 program office prioritize funding for the reliability and maintainability improvement plan. Re-examine the metrics DOD will use to hold the contractor accountable: In October 2017, we reported that DOD might not be using the appropriate performance metrics under trial performance- based agreements to achieve desired outcomes or hold the contractor accountable for performance. We recommended that DOD re- examine the metrics that it will use to hold the contractor accountable under the fixed-price, performance-based contracts, to ensure that such metrics are objectively measurable, are fully reflective of processes over which the contractor has control, and drive desired behaviors by all stakeholders. Delay entering into multi-year, fixed-price, performance-based contracts: In October 2017, we reported that DOD was moving quickly toward negotiating longer-term performance-based contracts without a sufficient understanding of the actual costs and technical characteristics of the aircraft, which put DOD at risk of overpaying for sustainment support that is not sufficient to meet warfighter requirements. We recommended that, before DOD enters into multi- year, fixed-price, performance-based contracts, it ensure that it has sufficient knowledge of the actual costs of sustainment and technical characteristics of the aircraft at system maturity. Obtain comprehensive cost information for F-35 spare parts: In April 2019, we reported that DOD did not have comprehensive cost information for individual F-35 spare parts, and that it faced challenges in obtaining this information from the prime contractor. This lack of cost information impedes DOD’s ability to develop a complete understanding of the costs for the F-35 system and to effectively negotiate with the prime contractor for sustainment support. We recommended that DOD develop a methodical approach to consistently obtain comprehensive cost information from the prime contractor for F-35 spare parts within the supply chain. Formalize a methodology for recording military service funds spent on F-35 parts: In April 2019, we reported that the military services could not track the funds that they had spent for the purchase of F-35 spare parts to the actual parts on their financial statements, thereby hindering DOD’s financial improvement and audit readiness efforts. We recommended that DOD complete and formalize a methodology for the U.S. services to use in recording on their financial statements the funds spent on F-35 parts within the global spares pool. Clearly define the F-35 supply chain management strategy: In April 2019, we reported that DOD was caught between two distinct sustainment concepts—the program’s official contractor-provided logistics support construct and DOD’s effort to develop options for DOD-led supply chain management. Until DOD clearly defines its strategy for managing the F-35 supply chain in the future, the F-35 program will lack the certainty and unity of effort necessary to meaningfully improve supply chain performance and reduce costs. We recommended that DOD clearly define the strategy by which it will manage the F-35 supply chain in the future and update key strategy documents accordingly, to include any additional actions and investments necessary to support that strategy. DOD concurred with all of these recommendations. Seven of the preceding recommendations were made earlier this year, and we recognize that it will take time for DOD to implement them. However, DOD’s attention to each of these recommendations is important to improving its long-term sustainment planning. In summary, DOD’s costs to purchase the F-35 are expected to exceed $406 billion, and the department expects to spend more than $1 trillion to sustain its F-35 fleet. Thus, DOD must continue to grapple with affordability as it takes actions to increase the readiness of the F-35 fleet and improve its sustainment efforts to deliver an aircraft that the military services and partner nations can successfully operate and maintain over the long term within their budgetary realities. DOD’s continued attention to our recommendations will be important as it balances these goals. We will continue to monitor DOD’s efforts to implement our recommendations. Chairmen Garamendi and Norcross, Ranking Members Lamborn and Hartzler, and Members of the Subcommittees, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have questions about this testimony, please contact Diana Maurer, Director, Defense Capabilities and Management, at (202) 512-9627 or maurerd@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Alissa Czyz and Kasea Hamar (Assistant Directors); Jon Ludwigson, Vincent Buquicchio, Tracy Burney, Desiree Cunningham, Jeff Hubbard, Justin Jaynes, Amie Lesser, Sean Manzano, Jillena Roberts, Michael Silver, Maria Staunton, Tristan T. To, Cheryl Weissman, and Elisa Yoshiara. F-35 Joint Strike Fighter: Action Needed to Improve Reliability and Prepare for Modernization Efforts. GAO-19-341. Washington, D.C.: April 29, 2019. F-35 Aircraft Sustainment: DOD Needs to Address Substantial Supply Chain Challenges. GAO-19-321. Washington, D.C.: April 25, 2019. F-35 Joint Strike Fighter: Development Is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved. GAO-18-321. Washington, D.C.: June 5, 2018. Warfighter Support: DOD Needs to Share F-35 Operational Lessons Across the Military Services. GAO-18-464R. Washington, D.C.: April 25, 2018. Military Aircraft: F-35 Brings Increased Capabilities, but the Marine Corps Needs to Assess Challenges Associated with Operating in the Pacific. GAO-18-79C. Washington, D.C.: March 28, 2018. F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency. GAO-18-75. Washington, D.C.: October 26, 2017. F-35 Joint Strike Fighter: DOD’s Proposed Follow-on Modernization Acquisition Strategy Reflects an Incremental Approach Although Plans Are Not Yet Finalized. GAO-17-690R. Washington, D.C.: August 8, 2017. F-35 Joint Strike Fighter: DOD Needs to Complete Developmental Testing Before Making Significant New Investments. GAO-17-351. Washington, D.C.: April 24, 2017. F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities. GAO-16-390. Washington, D.C.: April 14, 2016. F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016. F-35 Joint Strike Fighter: Preliminary Observations on Program Progress. GAO-16-489T. Washington, D.C.: March 23, 2016. F-35 Joint Strike Fighter: Assessment Needed to Address Affordability Challenges. GAO-15-364. Washington, D.C.: April 14, 2015. F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.: September 23, 2014. F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26, 2014. F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014. F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013. F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but Affordability Challenges and Other Risks Remain. GAO-13-500T. Washington, D.C.: April 17, 2013. F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013. Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks. GAO-12-437. Washington, D.C.: June 14, 2012. Joint Strike Fighter: Restructuring Added Resources and Reduced Risk, but Concurrency Is Still a Major Concern. GAO-12-525T. Washington, D.C.: March 20, 2012. Joint Strike Fighter: Implications of Program Restructuring and Other Recent Developments on Key Aspects of DOD’s Prior Alternate Engine Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19, 2011. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011. Joint Strike Fighter: Restructuring Should Improve Outcomes, but Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.: March 15, 2011. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "DOD's F-35 Lightning II fighter aircraft provides key aviation capabilities to support the U.S. National Defense Strategy. The F-35 is also DOD's most costly weapon system, with U.S. sustainment costs estimated at more than $1 trillion over its life cycle. As of October 2019, there were more than 435 U.S. and international F-35 aircraft in operation, with more than 3,300 aircraft expected to be fielded throughout the life of the program. While there is little doubt that the F-35 brings unique capabilities to the U.S. military, DOD faces significant challenges in sustaining a growing fleet. This statement discusses F-35 sustainment challenges. It also summarizes GAO's open recommendations related to these challenges. This statement is based on previously published work since 2014 related to F-35 acquisition, sustainment, affordability, ALIS, operations, and the global supply chain. The Department of Defense (DOD) faces challenges in sustaining a growing F-35 fleet. This statement highlights three challenges DOD has encountered related to F-35 sustainment, based on prior GAO work (see figure). As a result of these challenges, F-35 performance has not met warfighter requirements. While DOD works to address these issues, it must also grapple with affordability. DOD has determined that it will need to significantly reduce F-35 sustainment costs—by 43 percent per aircraft, per year in the case of the Air Force—in order for the military services to operate the F-35 as planned. Continued attention to GAO's recommendations in these areas will be important as DOD takes actions to improve F-35 sustainment and aircraft performance for the warfighter. GAO has 21 recommendations related to the challenges described in this statement that DOD has not fully implemented. DOD generally concurred with all 21 recommendations. Continued attention to these recommendations is needed by DOD to successfully operate and sustain the F-35 fleet over the long term within budgetary realities.", "document_type": "gao"}
{"report": "IRS Form 990-series return or notice must be filed by most organizations exempt from income tax under Internal Revenue Code section 501(a), and certain political organizations and nonexempt charitable trusts. TE/GE uses Form 990 reporting for promoting compliance and enforcing federal tax law for tax-exempt organizations (see appendix II for a copy of the Form 990 and a list of its schedules). Form 990 asks for information about an organization such as: employees, governance, and compensation; revenue and expenses; assets and liabilities; employment tax compliance; and specific organizational issues, such as lobbying by charities and private foundations. TE/GE redesigned the Form 990 for the first time in nearly 30 years for tax year 2008, and has made subsequent changes to the form (see appendix III for a summary of the changes). For tax year 2017, which is the most recent year of completed filing data, organizations filed 319,183 Form 990s. Beyond the basic Form 990, other versions include: Form 990–EZ, Short Form Return of Organization Exempt from Income Tax. This form reduces the filing burden on small tax-exempt organizations. Organizations with less than $200,000 in gross receipts and less than $500,000 in total assets may use it. For tax year 2017, 232,764 Form 990-EZ’s were filed. Form 990–N, Electronic Notice (e-Postcard) for Tax-Exempt Organizations Not Required to File Forms 990 or 990–EZ. Most small organizations whose annual gross receipts are normally $50,000 or less may file Form 990-N. For tax year 2017, 652,280 Form 990-N’s were filed. Form 990–PF, Return of Private Foundation. In addition to private foundations, nonexempt charitable trusts treated as private foundations are required to file Form 990-PF. For tax year 2017, 113,658 Form 990-PF’s were filed. Certain larger organizations are required to electronically file their returns. The Taxpayer First Act of 2019 requires all organizations to electronically file Form 990’s for tax years beginning after July 1, 2021. TE/GE can assess financial penalties for failing to file a required Form 990. As an employer, or if an exempt organization generates unrelated business income, additional tax reporting requirements may apply, such as for employment tax or unrelated business income. A 2017 TE/GE reorganization created CP&C to provide a centralized approach to compliance planning, examination selection and assignment and planning and monitoring activities. CP&C has three groups as follows: 1. Issue Identification and Special Review identifies and develops issues for examinations or compliance activities and certain criteria for examination selection. 2. Classification and Case Assignment uses IRS staff known as “classifiers” to review returns for examination under different examination sources (see appendix IV). Classification is the process of determining whether a return should be selected for compliance activities, what issues should be the primary focus of the compliance activity, and the type of compliance activity that should be conducted. 3. Planning and Monitoring develops an annual work plan and monitors performance. The work plan details the number of examination starts, closures and other measures. It develops classification requests to ensure that enough returns are available to meet work plan goals. TE/GE’s Compliance Governance Board (Governance Board) oversees TE/GE’s compliance program, including CP&C operations such as approving priority issue areas—known as compliance strategies. The Governance Board also reviews program goals, considers metrics and reporting, and reviews the performance of compliance strategies. The Governance Board has five TE/GE executives plus counsel who are voting members as well as three non-voting members. The Exempt Organizations examinations group is responsible for compliance activities. Examinations have various outcomes for an organization. The most severe outcome is revocation of tax exempt status. Taxes—such as employment or excise—may be assessed as a result of an examination. In fiscal year 2019, approximately $131 million in taxes were assessed. TE/GE conducts compliance contacts—non-examination correspondence such as compliance checks and soft letters—that are used to handle some compliance issues. For example, compliance checks determine whether specific reporting or filing requirements have been met. A “soft letter” notifies an organization of changes in tax-exempt law or potential compliance issues. A response to these letters is not required. TE/GE also reviews tax-exempt hospitals for compliance with certain community benefit requirements. In fiscal year 2019, TE/GE closed 1,470 compliance checks, sent 3,955 soft letters, and closed 750 hospital reviews. Compliance checks and hospital reviews can result in an examination while responses to soft letters may result in a compliance check. TE/GE identifies exempt organization returns for examination from many sources and categorizes examinations into three groups, known as portfolios: (1) Data Driven Approaches, (2) Referrals and Other Casework, and (3) Compliance Strategies. All three rely on data, to some extent, to make decisions on selecting returns for examination. This portfolio uses analytical models and queries based on quantitative criteria to identify potential examinations. TE/GE has three separate models that review exempt organization data from Forms 990, 990-EZ, and 990-PF for compliance. The models “score” returns for examination based on potential noncompliance. The Form 990, 990-EZ and 990-PF models have 354 unique queries. For purposes of this report, a query reviews databases to identify responses on returns that may indicate noncompliance because they do not meet certain criteria or expected values, such as exceeding a dollar threshold. Exempt Organizations Examination staff developed many of the queries, based on information collected on the Form 990 after it was redesigned for tax year 2008, according to TE/GE officials. As queries were developed, staff tested and used them to identify certain potentially noncompliant populations and to identify returns that were flagged by multiple queries. Starting in fiscal year 2016, TE/GE began using queries in models. The models use a scoring system that applies weights, or points, to each query result to generate a score—which for the Form 990 model has ranged from zero to more than 50—for a return. The models also screen out returns that are approaching a statute of limitations date, if the organization is not active, or has a current or recent examination history. Since November 2017, staff have been able to submit potential compliance issues for consideration through an online submission portal for Governance Board approval. CP&C has the option of considering whether these ideas result in model changes, according to IRS officials. Twice a year, each model is run using the latest data, and generates a Model Score Sheet (MSS). The MSS is a ranked list of returns that score above a minimum threshold. A classifier uses the ranking to identify returns for potential examination. Although the models screen for examination status and statute of limitations, a TE/GE official said the classifier also checks whether the statute of limitations date is near and whether the organization recently had undergone an examination or compliance check, as well as whether the return was identified under another selection method. This official explained that a classifier checks the criteria because conditions may have changed since the model’s last run. The classifier selects returns to fulfill a stocking plan, which identifies the number and type of returns to be examined to meet work plan requirements. See figure 1. Aside from the three models, TE/GE also uses other methods and data to identify and develop compliance work. The Data Driven Approaches portfolio includes approaches that TE/GE developed in partnership with IRS’s RAAS division. The partnership began in 2016 and continues today, according to IRS officials. The portfolio also includes some of the queries that TE/GE ran prior to fiscal year 2016 for examination selection. Some of these examinations remained open as of fiscal year 2019. Although not all of the returns selected for examination in this portfolio rely on data for examination selection, we describe them all below. Referrals. Referrals are complaints about exempt organization noncompliance made by third parties, including the public and other IRS offices or divisions. Post Determination Compliance. Sampling and queries are used to identify organizations that file Form 1023-EZ. Claims. Claims are requests for tax refunds, adjustments of tax paid, or credits not previously reported or allowed. Form 990 Queries (pre-model): These queries were run prior to fiscal year 2016. Some of these examinations remained open as of fiscal year 2019. Training. TE/GE uses these examinations, selected based on various methods, to teach examiners. Other Projects. TE/GE initiated these examinations under older compliance projects, using a variety of selection methods. The Compliance Strategies portfolio consists of compliance issues that originated from a Compliance Issue Submission Portal for TE/GE staff. The strategies are approved by the Governance Board, which results in adding the compliance strategy to the work plan. In fiscal year 2019, TE/GE closed examinations under three compliance strategies, including private foundation loans, and for-profit entities that converted to 501(c)(3) organizations. Returns are selected using sampling or other uses of data. Table 1 shows examinations closed for the three portfolios. Once an examination is underway, an examiner may expand it to include an organization’s returns for other tax years or other types of returns such as employment tax returns. IRS refers to these additional examinations as “pick-ups,” each of which is counted as a separate examination. Examiners must obtain manager approval to expand an examination. Examiners are required to check that an organization filed all returns that are required. If the examiner finds that a return was not filed—such as an employment tax return—and is unable to secure the return, he or she may prepare a “dummy” return called a substitute for return (SFR). The organization’s activities, records, and documents may then be examined. In 2017, TE/GE hired a contractor to assess aspects of the exempt organization process for examination selection, with a focus on the Form 990 model. In January 2018, the contractor released a report on the development and operation of the models. The contractor released a second report in July 2018 on the Form 990 model performance. The contractor found the model was not always identifying the “next best case” as TE/GE intended because scores did not consistently predict certain measures of noncompliance. Across both reports, the contractor made 17 recommendations, which we discuss later in this report (see appendix V). As of March 2020, TE/GE implemented one recommendation on model update submissions and part of another on hiring assessments. In September 2019, TE/GE initiated another study with the same contractor—with a planned release of the report in September 2020—on developing alternatives to the Form 990 model. Since the Form 990 model was first run for fiscal year 2016, the percentage of examinations closed that were identified by using data, such as through models or queries, has increased each year, as shown in figure 2. Almost half of these examinations are from the models. This increased reliance on using data in selecting returns for examination offers potential efficiencies. For example, a potential efficiency from using data to find possible noncompliance could mean fewer steps for staff who classify returns. Ultimately, this could allow TE/GE to shift staff from classifying returns to doing compliance activities such as examinations to confirm any actual noncompliance. Another potential efficiency would be selecting more examinations that find changes to the return. To measure the outcomes of examinations, TE/GE computes a “change rate,” or the percentage of closed examinations with a change to the return. In general, a higher change rate indicates that more examinations found noncompliance. Examinations selected using data have a slightly better change rate than other selection sources (84 percent versus 82 percent) for closures in fiscal years 2016 through 2019. Similar to all examinations that used data, the change rate for examinations selected using data through the Form 990 model (87 percent) was higher than the change rate for other selection sources (82 percent) in fiscal years 2016 through 2019. However, we found evidence that the changes identified in examinations did not clearly result from using the Form 990 model’s scoring system. Specifically: The model has not improved change rates compared to pre-model Form 990 queries. A higher model score is not associated with a higher change rate. Most examination changes credited to the model come from pick-up returns and SFR’s that examiners identify rather than from primary returns identified by the model score. The scoring generated by the Form 990 model has not improved change rates compared with the Form 990 queries that TE/GE used prior to the model. The change rates for both the Form 990 model and the pre-model queries, for fiscal years 2016 through 2019, was 87 percent. Similarly, for the last 2 fiscal years, the change rate for all Form 990 models was roughly equivalent to the change rate for other selection sources of exempt organization examinations. As shown in table 2, the models had a slightly higher change rate in fiscal year 2018, and a slightly lower change rate in 2019, compared to the other sources. Form 990 model scores for returns do not consistently predict examination change rates based on our analysis of examination closures since the model’s first run in 2016 through fiscal year 2019; the scores better predicted the rate at which returns were selected for examination. See figure 3. The figure shows little relationship between model scores and change rates; change rates remained relatively flat as model scores increase. While change rates were slightly higher for the less than 1 percent of returns scoring 45 or above relative to lower-scoring returns, TE/GE only examined 65 returns during fiscal years 2016 through 2019 that scored this high. The overall correlation between model scores and change rates is -.02. A TE/GE official said that it is not difficult to find a small issue on a return, which allows for a change regardless of score. To attempt to measure the severity of an examination change, TE/GE developed a weighted disposal score (WDS). However, TE/GE does not have documented criteria or justifications for how the weights were developed. A TE/GE official acknowledged that TE/GE has not used WDS because of questions about how consistently the weights have been developed. If WDS was to be used as a measure, TE/GE would need to ensure the adequacy of the support for the related weights and scores. According to TE/GE’s fiscal year 2020 Program Letter, the model relies on quantitative criteria, “which allows TE/GE to allocate resources that focus on issues that have the greatest impact.” To the extent that a higher model score does not predict a higher change rate, the model is not selecting returns with the greatest impact. Further, taxes assessed per return also indicate that examinations are not having the greatest impact. For fiscal years 2016 through 2019, the examinations credited to the model averaged $2,460 in proposed tax assessments per return, compared with an average of $19,042 for the rest of the exempt organization examinations. TE/GE acknowledged that its scoring methods are limited because it does not utilize modern data practices. It contracted for a study, to be completed in September 2020, of alternative model architectures and scoring methods that incorporate best practices for using criteria and options for scoring returns. As shown in table 3, the Form 990 model scoring did not account for most closed examinations and examination changes credited to the model during fiscal years 2016 through 2019. Rather, examinations of “pick-up” returns and substitutes for returns (SFRs) accounted for most closed examinations and produced a higher change rate than examinations of primary returns scored by the model. Examiners find these other returns during examinations of returns identified by the model. The higher change rates for pick-up and SFR returns compared to the primary returns identified by the model support TE/GE’s policy to examine all pick-up returns and SFRs that meet examination criteria. However, this raises questions about how well the model identifies noncompliant returns. Given the lower change rate for the returns the model scored, the queries for noncompliance on the Form 990 may not be effective. While the model includes queries on noncompliance related to “pick-up” issues such as unfiled employment tax returns, the necessary data were not available to allow us to analyze how often these queries identified the primary return for potential noncompliance. As discussed later, an analysis of queries could provide insight into the validity of the model. Internal control should be an integral part of an agency’s operational processes and structure to help managers achieve their objectives on an ongoing basis. When evaluating implementation, management determines if the control exists and is operational. A deficiency in implementation exists when no such control is present or is not implemented correctly, preventing objectives from being met. Documentation is required to show the effective design, implementation, and operation of an internal control system. The level and nature of documentation can vary based on the size of the agency and the complexity of its processes. Management exercises judgment in determining the extent of documentation that is needed. TE/GE has not fully implemented or documented internal controls for analyzing data for examination selection, meaning it cannot be assured that its selection decisions will produce the desired outcomes. The internal controls range from two controls that TE/GE adequately documented and implemented to seven others where TE/GE did not. The seven include five controls presented as sequential steps in using data for making selection decisions as well as two controls addressing timely documentation of Internal Revenue Manual (IRM) sections and risk management. The first internal control TE/GE implemented involved assessing staff competence. To ensure competence in using data to make decisions, TE/GE officials contracted with data specialists for modeling expertise to incorporate statistical and machine learning into examination selection. Bringing in this modeling expertise was an important step because exempt organization examinations staff, rather than statisticians or data analysts, initially developed the examination selection models, according to TE/GE officials. TE/GE also provided documents on training and basic duties for staff when analyzing data. What are Internal Controls and Why Do They Matter? One way federal agencies can improve accountability in achieving their missions is to implement an effective internal control system. Effective internal control comprises the plans, methods, policies, and procedures used to fulfill objectives on an ongoing basis. It serves as the first line of defense in safeguarding assets and increases the likelihood that an agency will achieve its objectives while adapting to changing environments, demands, risks, and priorities. Effective internal control provides reasonable, not absolute, assurance that an organization will meet its objectives. The second internal control involved communicating inside and outside of TE/GE. Internally, TE/GE staff could provide feedback through an online compliance issue submission portal in fiscal year 2018. Submissions may become compliance strategies or model queries. As for external communication, TE/GE collaborated on data-related issues in an IRS- wide group and with statistical specialists in the RAAS division. For example, RAAS identified potential data sources for compliance issues and drew samples for certain compliance strategies to test rates of noncompliance. In addition, to show how it communicates essential information with staff and outside parties, TE/GE provided examples on disseminating guidance and examination accomplishments, including examination starts and closures. TE/GE did not fully implement and document internal control over the processes and data used to select returns for examination. These processes cover five key steps for using data to decide which returns to select for examination (see figure 4). Effective internal controls would enable TE/GE to show how feedback and lessons learned in Step 5 can help it better determine how to create and use quality information (Step 3) and what decisions to make (Step 4) when pursuing the established objectives (Step 1). However, TE/GE has not defined measurable objectives or undertaken regular evaluations to assess progress toward objectives. Although TE/GE was able to describe its approach for accessing relevant and reliable data, processing those data into quality information and using the data to make decisions, it was not able to fully document how its control processes worked, as discussed below. Since its 2017 reorganization, TE/GE has not established measurable objectives to select exempt organization returns for examination (see figure 5). Specifically, TE/GE has not produced formal objectives that are aligned with its mission and the IRS strategic plan, are expressed in quantitative terms, and are related to examination selection and program outcomes. TE/GE documents, including Program Letters and Business Performance Reviews, refer to outcomes that could constitute objectives—such as improving the models and advancing data analytics to drive decisions about identifying and addressing existing and emerging areas of noncompliance—but they do not identify them as such. TE/GE officials acknowledged the need to establish measurable objectives. They said their efforts are evolving and they need to improve analytical abilities to help assess the capacity for meeting objectives. For example, one official said they are working to establish objectives at the onset of a compliance strategy. Without measurable and defined objectives, TE/GE cannot effectively analyze how well it selects returns for examination and lacks a clear vision of what it is trying to achieve. A lack of measurable objectives also hinders implementing other internal controls, such as evaluating performance or assessing risk, as discussed later. The IRM has procedures for processing Form 990 data, which include controls over acceptance and transmission of the data (see figure 6). TE/GE provided data that showed error rates for electronically filed returns filed in 2019 were between 1 and 4 percent. However, taxpayer or transcription error rates for paper returns filed in 2019 were between 19 and 32 percent of filed returns, depending on the version of the Form 990. TE/GE was not able to show that it regularly reviews and remediates such errors to ensure the reliability of Forms 990 data. However, under the Taxpayer First Act of 2019, electronic filing of all Forms 990 will be required for tax years starting July 2, 2021. This change should remediate the known errors from paper-filed returns and increase data reliability. We found several issues with TE/GE’s processing of queries in the Form 990, 990-EZ and 990-PF models that affect the validity or reliability of the scores that the models generate to rank returns for examination selection (see figure 7). As a result, TE/GE cannot ensure that the model scores properly rank the returns for examination selection. Specifically, TE/GE does not consistently assign point values for the queries used to generate the model scores and inform selection decisions. We also found errors in TE/GE’s documentation of the queries, which lead to redundant queries, and inflated model scores. Finally, TE/GE has no control procedures to ensure consistent testing of proposed queries. Inconsistent Point Values for Queries Raise Concerns about Model Scores We estimate that for 24 percent of queries (83 queries) from the models, TE/GE staff did not assign point values for queries consistent with its definitions for the four categories (see table 4). Not implementing the defined point values puts the model scores at risk of inconsistent scoring and examination selection. We found three types of queries involved with the inconsistent assignment of point values. 1. Miscategorized queries were not assigned to the category that matches TE/GE’s definition. These occurred because TE/GE has not documented specific rules for query categorization. As a result, we found an estimated 7.4 percent of queries (26 queries) where TE/GE staff overrode the category definitions when assigning points without documenting the reasons. Absent the reasons, TE/GE cannot ensure consistent treatment of similar queries. In our sample, these override decisions included assigning: Three queries to the Speculative category, which is worth five points, when the definitions supported the Automatic category, which is worth 10 points. TE/GE officials said they did this to offset potentially confusing language in the return lines or instructions. One query to the Automatic category rather than the Speculative category supported by the definitions. TE/GE officials said they used the higher point value category to increase the chance of selection so that certain Form 990-PF attachments, which the queries do not cover, would be more likely to be considered for examination. 2. Queries could fit into more than one category based on TE/GE’s definitions. We estimate that 16 percent (55) of the queries could fit in more than one category. Of these, 18 in our sample could have been placed in the Missing Schedule/Form category. In addition, we found one query in our sample that TE/GE labeled as having a duplicate but one query was assigned to the Automatic category worth 10 points and the other was assigned to the Inconsistencies category worth one point. TE/GE officials acknowledged that some queries could fit in more than one category. When we asked why certain queries for missing schedules and forms were not categorized as such, these officials described a hierarchy of missing forms based on being subject to penalties and interest, such as employment tax returns, and their associated categories. They did not document or consistently implement this hierarchy as queries identifying the same missing form sometimes were in different categories. 3. Sliding scale queries whose point values differ from those stated in TE/GE’s model documentation. We found nine queries with sliding scale point values that involved Form 1099 information returns. The sliding scales reduce point values based on the severity of the compliance issue, such as reducing the query point values if the organization filed a low number of information returns. TE/GE did not provide documentation about the rationale and associated definitions for these queries. Without documentation on the different treatment of these queries, TE/GE is not transparent about the rationale for assigning points through a sliding scale to support its model scoring. TE/GE officials said they have not updated definitions and criteria for using the categories and sliding scales because of a decision to keep the model operating as is and to update documentation as time permits. After our preliminary analyses, TE/GE provided updated definitions for the four categories, and descriptions of the sliding scales that were used for queries. However, these definitions and descriptions do not include any decision rules or criteria that document how to apply them. Further, the sliding scale descriptions do not offer definitions for words like “low,” when referring to the volume of information returns filed. Definitions that are incomplete and not always followed when assigning point values raise concerns about consistency and transparency in scoring returns for examination selection. TE/GE’s assignments affect scores and whether a return is placed on the MSS for examination consideration. Inconsistent or invalid assignment of point values may distort the potential for examination. For example, of the nine miscategorized queries we analyzed in our sample, we determined that if their categorizations were corrected, hits on three of the queries would make a return eligible for the MSS and hits on two others may make a return eligible, depending on the other queries the return hit. Changes to two queries would have made returns no longer eligible for the MSS. Query Documentation Has Errors That Forestall Valid Analysis of Queries We estimate that about 27 percent (96 queries) of the queries in the models had errors in the documented descriptions. Query descriptions detail the logic and data used from specific forms and line numbers that the queries scan. The errors we found include: references to older versions of the forms as well as omissions of form lines used in the query; and query descriptions that did not match programming code. To address these differences, TE/GE proposed corrections to the query descriptions. A TE/GE official said re-visiting the query documentation is part of the contractor’s 2020 study and that TE/GE does not have a timeline for correcting the documentation. In addition to errors, the descriptions also use inconsistent language, which prevents easy identification of queries by issue. For example, to identify all queries related to excess benefit transactions, one must manually search different fields for terms such as “excess benefit,” “excessive benefit,” and “EBT” (excess benefit transaction). Furthermore, TE/GE’s database fields only capture one issue per query. Since many queries involve multiple issues, these fields cannot be used to fully inventory the queries. These errors and inconsistencies in the query descriptions occurred because TE/GE has no procedures for regular reviews of queries as forms or laws change. TE/GE Compliance Governance Board (Governance Board) members review query descriptions prior to implementation but do not review details of the queries in the context of the entire model. Further, TE/GE procedures only require review of programming code before queries are sent to the Governance Board. Review of the code once it is integrated into the model program is optional, according to TE/GE procedures. The errors and inconsistent descriptions prevent TE/GE from having a comprehensive and accurate inventory of queries within and across models. Without regular reviews, TE/GE cannot be assured that its programming code is correct and that any analyses of the performance of queries or the models as a whole are valid. When we asked about the lack of regular reviews of queries, TE/GE officials said they plan to implement reviews but did not provide us with a plan or timeframes for doing so. Another effect of not having a comprehensive and accurate inventory is that TE/GE cannot analyze query performance and identify queries that look for the same compliance issue to prevent redundancies and to ensure valid and consistent scoring. As a result, we found queries that address the same or similar issues with the same criteria, inflating scores for returns and making selection for examination more likely. Our analysis of the July 2019 Form 990 model run showed 90 pairs of queries, involving 78 unique queries that hit together at least 90 percent of the time. By having two queries that rely on the same criteria, returns accumulate extra points for the same behavior. For example, all 910 returns that hit an employment tax query also hit a query that shares some of the same criteria and thresholds. As a result, these returns accumulated 10 points rather than five points, making them eligible for the MSS. Aside from our sample, we found queries seeking certain organizations with political campaign activities and political expenditures that would total 15 points in the Form 990 model. Queries identifying these same activities and expenditures would total 30 points in the Form 990-EZ model. TEGE’s contractor recommended in 2018 that TE/GE eliminate “redundant” queries, which is similar to our finding. TE/GE officials said they do not believe the redundant queries are duplicates and they are awaiting the results of the contractor’s study in 2020 before making changes. Until TE/GE resolves the extent to which it has redundant queries, it cannot do a valid analysis of whether its queries identify the most noncompliant returns. TE/GE Lacks Procedures and Criteria for Testing Proposed Model Queries TE/GE has no procedures requiring the testing of proposed model queries. Even so, based on our sample of the new queries in the fiscal year 2018 Form 990 model, TE/GE would be able to provide evidence of tests for an estimated 94 percent of all new queries. However, TE/GE also does not have procedures for how to conduct testing or what data to use. The testing that has been performed consisted of running the query on certain tax years of returns to count the number of returns flagged, according to a TE/GE official. TE/GE does not run the queries on data from closed examinations to see whether the queries would identify known compliance issues that justify an examination. Interactions with existing queries are not tested. When considering new queries, Governance Board members see the number of returns flagged by each query during testing, but have no criteria to determine whether a query flags an appropriate number of returns. A TE/GE official said TE/GE does not believe it needs to document procedures for testing. In the absence of procedures and standards, TE/GE cannot ensure that testing of new queries is done consistently with appropriate data sources and research standards. By only testing the number of returns that a query flags, TE/GE cannot validate that proposed queries can effectively identify the noncompliance that would be worth examining. Using tested, validated and documented data is a critical step in ensuring that research is proper, reliable, and accomplished in accordance with expectations, according to the IRM. Without testing queries on reliable data, and making adjustments based on criteria, TE/GE risks implementing queries that do not produce reasonable numbers of hits that are worth pursing through examinations. For examination sources that used data other than the models, we found that TE/GE did not always document its processing of data into quality information. We identified common “start-to-finish” segments to this processing of data, including: submitting a proposal and supporting data to find noncompliance; reviewing the potential data sources and queries or thresholds to be used as examination selection criteria; and recommending the proposed effort for approval through the appropriate executives. On one hand, TE/GE provided documentation of the required approvals for these segments in processing data for five compliance strategies. These strategies included examining loans by private foundations and collecting information on organizations that exceed investment income limitations. On the other hand, TE/GE did not provide similar start-to-finish documentation on processing quality information from other examination sources that use data outside of the models; examples include research projects under the Data Driven Approaches portfolio and projects that use queries under the Referrals and Other Casework portfolio (see table 1). Over several discussions, TE/GE did not explain why it did not fully document such projects. By not fully documenting how it processes data into quality information, and by not linking such processes to measurable objectives, TE/GE cannot ensure that it is analyzing quality information in selecting examinations. For the compliance strategies, TE/GE showed evidence of using the quality information to decide which returns to select for examination, such as for Governance Board decisions. However, TE/GE did not provide documentation on how it made selection decisions using data for other projects that use queries (see figure 8). In addition, TE/GE did not use quality information to decide how frequently to run the model. TE/GE decided to run the Form 990 models twice per year without analyzing the effects. Moreover, we found that the time between runs is inconsistent. Since the Form 990 model’s first run, the time between runs ranged from 84 days to 251 days. Since returns are ranked on the MSS, eliminations result in the classification staff selecting lower scoring returns. The average score for examined returns was 27.1 for the list that was used for 84 days compared to 23.2 for the list used for 251 days. To the extent that TE/GE ensures that its model scores are as reliable and valid as possible, analyzing data could help TE/GE identify the frequency of model runs that maximizes the use of model scores to guide decisions on examination selection. For example, analyzing Form 990 filing patterns could help identify the optimal timing of model runs, allowing for adequate time remaining under the statute of limitations. TE/GE does not regularly evaluate its models and other selection processes that use data. In particular, model scores for all returns are not retained or are inconsistent from year to year which limits the ability to conduct evaluations. Furthermore, TE/GE does not evaluate reasons why some selected returns are not examined, which could help improve selection methods (figure 9). TE/GE Has No System for Regularly Evaluating Examination Selection Decisions TE/GE has not regularly evaluated its examination selection decisions that rely on data to improve its selection methods. While TE/GE commissioned the contractor evaluations of its Form 990 model, it has no documented process for continued evaluations of the model or any evaluations of other sources, such as research projects, that rely on data to select returns for examination. For its compliance strategies, not enough examinations have closed under the strategies to warrant evaluations yet, according to a TE/GE official. Data limitations have challenged evaluation efforts, according to a TE/GE official. To address this, TE/GE started capturing more detailed data on examination outcomes; however no evaluations of outcomes have resulted. The officials noted that they have been spending more time reporting and monitoring compared to analyzing and evaluating, which they said needs to occur more often. Without evaluation, TE/GE cannot ensure that its use of data to select returns is working as intended. In addition to not evaluating selection decisions and their outcomes, TE/GE has also not addressed the Form 990 model deficiencies the contractor previously identified. In its 2018 reports, the contractor made 17 recommendations (see appendix V for the status of each recommendation). A TE/GE official said it had not acted on many recommendations because all examination selection strategies are being evaluated with the transition to the Compliance Planning and Classification (CP&C) office. TE/GE initiated another study in 2019 with the same contractor to address its 2018 recommendations among other tasks. As of March 2020, TE/GE implemented one recommendation and part of another, deferred action on nine recommendations until after the contractor finishes the new study, deferred action on three due to other reasons, and did not clearly provide a status for two. In addition, TE/GE will likely not implement the other recommendation. According to contract documentation, the study will explore architectures and alternative designs to the model, propose up to three compliance actions other than examinations, and recommend measures to monitor the actions’ effectiveness. TE/GE expects a final report by September 2020. To the extent that TE/GE has not implemented the contractor’s recommendations from 2018, the related deficiencies identified in the Form 990 model will have persisted for more than 2 years by the time the contractor issues its 2020 report. Unless TE/GE documents its consideration and action of the recommendations, the value of the contractor’s work is diminished and possible improvements may be overlooked. TE/GE Has Not Retained Complete Data to Allow for Full Evaluation of Its Models Until recently, TE/GE did not retain model scores for each return and query performance data that would be useful for evaluation. The January 2018 contractor report recommended that TE/GE save model data. For its July 2018 report, the contractor had to recreate historical scoring data for its evaluation. TE/GE officials said they increased storage space and saved the fiscal year 2018 data. When we asked for these data, TE/GE officials said that each time they run a model, they overwrite the old data. The officials said they did not have space on their server to save all of the data. Instead, TE/GE had been saving the MSS’s for each run. However, the MSSs have only limited value for evaluating the model and queries. Specifically, the MSS for each model run contains score information for only about 20,000 returns (out of about 300,000 scored) that have a certain minimum score and hit queries in certain categories. Further, the MSS does not contain data on model queries that are flagged. In September 2019, TE/GE officials said the Research, Applied Analytics and Statistics division provided temporary server storage space to save model data through September 2020 while the contractor assesses TE/GE’s models. Starting with the July 2019 model run, TE/GE is saving score and query performance data for all filed returns. In January 2020, TE/GE officials told us they developed a way to save data on query hits for all returns run through the model. However, TE/GE has not provided documentation to show exactly what data will be saved over the long term for all filed returns run through the model. Without complete historical data on model scores and query hits, TE/GE cannot assess the full performance of its models. Such data would facilitate an analysis of the queries, and whether they identified returns with changes or related pick- up returns. Historical Data on Examination Outcomes Lack Consistency, Which Complicates Evaluation TE/GE does not analyze consistent multi-year data on examination outcomes, which would facilitate evaluation of its use of data in selecting returns for examination. TE/GE officials said they use historical data— such as change rates—to determine the success of an examination source. TE/GE provided historical data on examination starts, closures, and pick-up returns covering 2 years but did not provide data beyond that and change rates were not always included. Further, TE/GE has used different methods to organize and report examination outcomes over the years. These differences in reporting outcomes affect TE/GE’s data in the following ways: Starting with fiscal year 2018, data on exempt organizations examinations include federal, state and local employment tax examinations. Prior to 2018, TE/GE reported these employment tax examination data separately. After TE/GEs reorganization in 2017, it grouped examinations into portfolios and changed the portfolio definitions during 2018. As of March 2020, TE/GE has not produced a consistent method of summarizing of historical data. TE/GE officials acknowledged data limitations, and said they are working to implement recommendations from a 2019 study to improve capturing examination data. TE/GE officials said the staff member analyzing data has been doing so for many years, allowing them to reconcile the data. However, this poses a risk that other IRS or oversight entities cannot reconcile the data. According to internal control standards, agencies should establish effective methods for retaining organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel, as well as contingency plans to respond to sudden personnel changes. TE/GE’s inconsistent data limit its ability to conduct evaluations. These inconsistent data also prevent TE/GE from establishing baselines or targets for examination outcomes such as change rates to help measure the success of its selection methods. In fiscal year 2019, TE/GE did not examine about 20 percent of the exempt organization returns that had been selected for examination. Although this rate of non-examined returns has improved in recent years, TE/GE has not analyzed data to explore why the rate has improved and how to reduce it further. Our analysis showed that almost 30 percent of these returns were not examined because they were too close to the statute of limitations date. TE/GE officials did not have a reason why the returns were sent to the field for examination if the statute date was so close. TE/GE officials said they do not regularly analyze reasons for non-examined returns. They said they have analyzed only the number of non-examined returns by manager and area. In addition, TE/GE officials said they implemented new guidance in fiscal year 2019 for staff who make decisions to not examine returns, which is intended to improve the information they have on these decisions. As of fiscal year 2019, TE/GE began tracking certain non-examined returns by project code but has not committed to analyzing the data. Non-examined returns are not an efficient use of resources, as the time spent reviewing and rejecting these returns—even if minimal—reduces the time staff have for conducting examinations. Routinely analyzing reasons for non-examined returns, as well as related data, could help TE/GE identify actions to reduce the number of returns that are sent to the field but are then declined for examination by a manager or examiner. TE/GE did not annually update procedures on examination selection and databases in certain IRM sections since the May 2017 reorganization. The Internal Revenue Manual (IRM) states that procedures in IRM sections must be annually reviewed and updated as needed. TE/GE released updated IRM sections for two of the three groups in CP&C. It released a section on the Issue Identification and Research in September 2018, and one on the Classification and Case Assignment procedures in September 2019. However, these sections do not cover the steps the model classifier takes when reviewing returns from the MSS. As of December 2019, no IRM section has been released on the Planning and Monitoring group. As such, TE/GE staff does not always have official information on roles and responsibilities for new entities and processes created since May 2017. For certain updated or new IRM sections, TE/GE did not release interim guidance while those sections awaited approval. IRS requires issuance of interim guidance to address deviations from the IRM, even if temporary. Instead of developing interim guidance, TE/GE officials stated that, in the wake of the reorganization, they decided to use desk guides, such as for the IRM section on classification and case assignment processes. However, TE/GE did not update its desk guides on processes until more than 2 years after the reorganization. Furthermore, the desk guides do not cover the specific duties of the model classifier, or the steps for classification of returns identified for compliance strategies. IRM guidance states that management must develop and maintain documentation on data systems; collection and analysis; and responsibilities for data collection, input and analysis. Timely documentation of new procedures and responsibilities improves the accuracy and reliability of IRM content. According to the IRM, when the IRM and related guidance are not current, TE/GE increases the risk that staff follow incorrect procedures, use guidance that is not transparent to the public, administer tax laws inconsistently, and misinform taxpayers. Good federal government practice requires risk management, without which, TE/GE could undercut its use of data to enhance decisions on examination selection. Although the use of data in examination selection has the potential to improve efficiencies in classifying and examining returns to identify noncompliance, any new endeavor carries risks. TE/GE did not identify any TE/GE-specific risks that could undercut its success in using data to select exempt organization returns for examination. As of December 2019, the TE/GE risk register identified 12 risks, ranging from aging technology and infrastructure to employee engagement and morale. One risk— data access and analytics—involved using data in general decision making at the IRS level rather than TE/GE decisions about examination selection or its related models. TE/GE officials said they are analyzing and responding to this risk under the IRS- wide risk management process. TE/GE did not document why it did not identify any TE/GE-specific risks in using data for examination selection. Our report discusses a number of deficiencies that could be potential risks to TE/GE using data in selecting returns for examination. For example, TE/GE lacks program objectives that would be necessary to identify and assess risks. We also found weaknesses in how TE/GE processes and analyzes data to inform examination selection and how it evaluates selection decisions. Further, the IRM states that TE/GE’s Compliance Governance Board (Governance Board) should consider risks in its decisions and we saw that risks were considered in documents proposing examination selection criteria to the Governance Board. We did not find evidence that TE/GE’s risk management process recorded these risks for analysis and any response if needed. After we shared our concerns about the lack of identified risks, TE/GE officials noted that TE/GE participates in mitigation steps as identified by the IRS Risk Office. TE/GE officials also mentioned CP&C representation in an IRS pilot program designed to explore ways to better select employment tax cases. While such actions could be a component of a risk management strategy, it is incomplete and it is unclear how this initiative would help TE/GE identify, analyze, and mitigate risk. Not identifying and managing risks identified in this report leaves TE/GE open to errors and examination selection decisions that are potentially not transparent or not fair. As such, without objectives and a consistent and documented process for identifying and managing risks, TE/GE cannot effectively address risks that may hamper its efforts to use data to enhance its compliance work. Increasingly constrained resources underscore the importance of TE/GE’s efforts to efficiently identify and examine exempt organization returns that have the highest noncompliance potential. TE/GE has developed ways to use data to aid in examination selection. However, opportunities exist to strengthen internal controls to help ensure that data used are reliable, decision rules are clear and documented, and objectives are identified and being achieved. TE/GE should take several steps to improve the reliability and validity of the models. These steps include improving documentation of decision rules and criteria for scoring; regularly reviewing model documentation and programing; testing new queries and their interaction with existing queries; retaining model and query data; and periodically evaluating the performance of selection methods. In absence of regular evaluation of its examination selection decisions, TE/GE misses opportunities for improving its selection processes. Deficiencies that TE/GE’s contractor already identified provide an opening for improving its models. Without consistent historical data, TE/GE will be limited in assessing progress and making improvements. A review of the reasons why certain returns selected for examination are not examined is an example of an evaluation that could help inform process improvements. Ensuring that all procedures are current and accurate would reduce the potential for employees following incorrect procedures and administering tax laws inconsistently. TE/GE’s lack of identified risks from using data in examination selection precludes TE/GE from analyzing and responding to those risks. By taking actions to further strengthen these internal controls, TE/GE could enhance its efforts to identify and examine the most noncompliant exempt organizations and enhance IRS’s oversight of tax exempt organizations and help maintain the integrity of the charitable sector and the larger exempt community. We are making the following 13 recommendations to IRS: The Commissioner of Internal Revenue should document measurable objectives for using data in selecting exempt organization returns for examination. (Recommendation 1) The Commissioner of Internal Revenue should document and consistently use clear criteria and decision rules on assigning point values to queries, using categories and sliding scales. (Recommendation 2) The Commissioner of Internal Revenue should require a regular review of query descriptions and programming to ensure their accuracy and minimize queries that flag the same or similar compliance issue. (Recommendation 3) The Commissioner of Internal Revenue should develop procedures and criteria to test new queries prior to implementation in the models. (Recommendation 4) The Commissioner of Internal Revenue should more fully document how TE/GE processes data and uses data to make examination selection decisions for sources outside of the model such as research projects and other projects that use queries. (Recommendation 5) The Commissioner of Internal Revenue should conduct an analysis to identify the optimal interval between model runs. (Recommendation 6) The Commissioner of Internal Revenue should establish a process for regularly evaluating selection decisions and related outcomes for the models and other processes that use data to select returns for examinations. (Recommendation 7) The Commissioner of Internal Revenue should document consideration or action on recommendations from its 2018 and 2020 contractor assessments. (Recommendation 8) The Commissioner of Internal Revenue should document how score and query data for all returns in the models will continue to be saved over the long term. (Recommendation 9) The Commissioner of Internal Revenue should ensure that historical data on examination outcomes are consistently defined and used when doing analysis of examination outcomes. (Recommendation 10) The Commissioner of Internal Revenue should routinely analyze the reasons for not examining selected returns and identify any necessary actions to address the reasons. (Recommendation 11) The Commissioner of Internal Revenue should annually review and update procedures as needed in relevant IRM sections on examination selection and issue interim guidance until the affected IRM sections are updated. (Recommendation 12) The Commissioner of Internal Revenue should document why TE/GE has not identified any risks in its risk register for using data to select exempt organization returns for examination. If risks are subsequently identified, TE/GE should document how it plans to analyze and address them. (Recommendation 13) We provided a draft of this report to IRS for review and comment. IRS provided written comments, which are reproduced in appendix VI and summarized below. Of our 13 recommendations, IRS agreed with 12 and disagreed with one. IRS also provided technical comments, which we incorporated as appropriate. IRS disagreed with our recommendation on ensuring that historical data on examination outcomes are consistently defined (Recommendation 10), pointing out that its raw data are consistently defined in its information systems. Our concern, however, is with how the outcome data are reported and analyzed, which inhibits understanding of outcome trends over time. In response to IRS comments, we added language to the final recommendation to more clearly focus on the consistency of the outcome data used and analyzed over the years. In addition, although IRS agreed with our recommendation to more fully document how TE/GE processes and uses data to make examination selection decisions outside of the model (Recommendation 5), IRS said that it would provide documentation on a project (other than compliance strategies) that is approved by the Governance Board. While we look forward to such documentation, we are primarily interested in IRS documenting a system for how it processes and uses data to select returns for examinations for projects outside of the model, regardless of Governance Board approval. As discussed in the report, IRS has such a system for projects in its compliance strategies portfolio, which could provide a framework to follow. Similarly, IRS agreed to analyze return due dates of the filing populations commonly associated with the examinations (Recommendation 6). We will be interested to see how that analysis helps IRS to determine the optimal interval between model runs, which is the focus on our recommendation. We are sending copies to the appropriate congressional committees, the Secretary of the Treasury, the Commissioner of Internal Revenue, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9110 or mctiguej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. This report assesses (1) the use of data to select tax-exempt organization returns for examination; and (2) the process the Tax Exempt and Government Entities (TE/GE) division has established to select returns for examination. To assess the use of data to select tax-exempt organization returns for examination, we reviewed data from the Internal Revenue Service’s (IRS) Returns Inventory and Classification System (RICS) for fiscal years 2016 to 2019. Table 5 defines the variables and measures we analyzed. We analyzed aggregated data at the project code level, and we grouped project codes by examination source (for example, examinations from referrals occurred under several project codes). Based on our testing of the data and review of documentation and interviews, we determined that the data were reliable for purposes of assessing TE/GE’s selection processes. We analyzed outcomes from the Form 990, Return of Organization Exempt from Income Tax model. We used RICS data and Model Score Sheets (MSS), for examinations closed from October 1, 2015 through September 30, 2019. Each model run generates an MSS, which is a ranked list of Form 990s that hit certain types of queries and have a minimum score. We matched Form 990 scores from the MSS with selection information and examination outcomes in RICS for examinations closed under all project codes, though the data presented in objective one is specific to examinations started under the Form 990 project code. We used source codes—which indicate whether an examination was a pick-up, substitute for return or primary return—to analyze what types of examinations produced the highest change rates under the Form 990 model project code. To inform this work, we reviewed recent TE/GE contractor assessments of exempt organization examination selection and the Form 990 model. To assess the process that TE/GE has established to select returns for examination, we reviewed internal controls steps in Standards for Internal Control in the Federal Government (Green Book). Given TE/GE’s emphasis on using data in examination selection, we identified five internal control steps related to analyzing data to select returns for examination to address our objectives. We selected four other internal controls because they constitute practices common to all five steps in the selection process. These are presented in table 6. Define objectives in measurable terms so performance in achieving objectives can be assessed. (Green Book (GB) 6.04) Obtain relevant data from reliable internal and external sources in a timely manner based on identified information requirements. (GB 13.04) Process the obtained data into quality information that supports the internal control system (i.e., using data in decision making); use quality information to achieve the entity’s objectives; and document policies on the responsibilities for data collection, input, and analysis. (GB 13.05, 13.01, and 12.02) Use the quality information to make informed decisions in achieving key objectives. (GB 13.05) Evaluate performance (outcomes) for key objectives and take actions to remediate deficiencies. (GB 13.05, 16.03, and 17.06) Develops, maintains, and updates in a timely fashion documentation on the responsibilities for data collection, input and analysis for using data in decision making. (GB 12.02 and 12.05 and IRM) Defines risk tolerances in specific and measurable terms, considers internal and external factors to identify risks, analyzes risks to estimate significance, and designs specific actions for response. (GB 6.09, 7.04, 7.05, and 7.09) Ensures that personnel possess competence to meet responsibilities as well as understand the importance of effective data analysis in decision making. (GB 4.04) Communicates necessary information to enable personnel to perform key roles for analyzing data in decision making and with external parties. (GB 14.03 and 15.20) To identify criteria specific to IRS, we reviewed the Internal Revenue Manual (IRM), which provides standards and guidance similar to the criteria we identified. We shared the Green Book and IRM criteria with TE/GE, as well as our expectations of the documentation that would show adherence to these criteria. Our assessment focused on examination sources developed after the 2017 reorganization and sources that rely on data for selection (such as models and projects that use queries). Examination sources that did not rely on data, such as claims, were not assessed. We reviewed the referrals classification process to consider how data might be used to enhance it. We analyzed TE/GE documents such as Program Letters, Business Performance Reviews, desk guides, memorandums, work plans, performance data, contractor reports and training documents. In addition, we assessed documents—such as meeting minutes and research results—showing the development and approval of data queries and projects used in examination selection. We reviewed the MSSs for the Form 990 model, and procedures for the Form 990 model, the Form 990-EZ, Short Form Return of Organization Exempt from Income Tax, and Form 990-PF, Return of Private Foundation, models. We selected a generalizable stratified random sample of 114 of the 354 unique queries in the three models (see table 7). Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval (e.g., the margin of error is +/- 10 percentage points). This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. Our sample is designed to control the margin of error of attribute estimates within the overall scope query sample as well as the combined Form 990 query sample (a combination of strata 2 and 4 plus certainty selections). The sample was designed as follows. There is one certainty stratum for Form 990-PF queries where we selected a 100 percent sample (i.e. a census), and this stratum does not have a margin of error. We selected these queries with certainty because of the smaller population size in this stratum. For remaining strata, we selected the necessary sample size to achieve an overall 95 percent confidence interval for attribute (percentage) estimates with a margin of error of about +/-10 percentage points under proportionate allocation. In addition, the sample size was increased in strata 2 and 4 (combining Form 990 model queries) to achieve the necessary sample size for a 95 percent confidence interval with a margin of error of about +/-10 percentage points within this group. For the sampled queries, we compared their category and descriptions as provided in the model documentation, with TE/GE’s definitions of the categories to assess whether the query was categorized appropriately. We also compared the query descriptions with the forms to assess whether the referenced lines were relevant to the query. Additionally, we reviewed the model programming code to check for errors and consistency with the query descriptions. For query categorizations that did not match TE/GE’s definitions or queries that appeared to have errors in the descriptions or programming, we asked TE/GE to review and explain its decisions. To identify potentially redundant queries, we analyzed output from the July 2019 Form 990 model run, the only one available at the time of our analysis. Within our sample, we reviewed 36 of the 104 newly added queries in the fiscal year 2018 model. Specifically, we reviewed approval documentation and meeting minutes to test whether two levels of management and the Compliance Governance Board approved new queries, consistent with TE/GE procedures. We also reviewed evidence that TE/GE tested each query prior to its approval for inclusion in the models. We held two telephone focus groups with the nine classifiers who review exempt organizations referrals. We asked questions about the data and resources they use to classify referrals, how they convey their results, and how they are provided feedback. We interviewed officials from the Compliance Planning and Classification office and IRS’s Research, Applied Analytics and Statistics division who worked on several compliance research initiatives. We met regularly with TE/GE to share ongoing assessments. We conducted this performance audit from November 2018 to June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The figure below shows the text of the 2019 version of Form 990, Return of Organization Exempt from Income Tax. A list of schedules for the Form 990 is provided in table 8 following the form. The remaining pages of the Form 990 are available at IRS’s website, accessed March 23, 2020: https://www.irs.gov/pub/irs-pdf/f990.pdf. Most exempt organizations are required to file an annual form to report their activities, structure, revenue and expenses, and other items. The organization’s classification under the Internal Revenue Code and its gross receipts and total assets, determines which form must be filed. Most organizations file one of the following: Form 990, Return of Organization Exempt from Income Tax; Form 990-EZ, Short Form, Return of Organization Exempt from Form 990-PF, Return of Private Foundation or Section 4947(a)(1) Trust Treated as Private Foundation. The Internal Revenue Service (IRS) last redesigned the Form 990 series for tax year 2008. The redesign added 14 schedules to the existing two, and reflected changes in the tax-exempt sector and tax law. Some changes from the redesign were phased in and implemented for tax year 2008 and 2009 filings. We summarized changes as found in the “What’s New” section of the form instructions for each of the three Form 990 types and for each year. We grouped the changes into two categories as defined by: New or revised question(s): The addition of new lines, check boxes, narratives or schedules. This includes changes to accommodate new laws or reporting requirements, such as new reporting thresholds or standards. Instructions and format: New descriptions or details in the instructions, such as specifying examples or how to provide certain information to IRS. This also includes changes that affect order of lines or schedules, but not the content. For the Form 990 changes since the redesign, IRS made 56 changes to the form or its instructions for tax years 2009 through 2019 (see table 9 below). These changes include three to the 2018 form implementing new excise taxes on net investment income of certain colleges and universities and on certain tax-exempt organization executive compensation. Aside from new electronic filing requirements for tax years beginning July 2, 2019, the 2019 form did not have any changes. In addition to the 56 changes, IRS made 95 clarifications to existing lines or instructions, or revisions to definitions from tax years 2009 through 2018. These clarifications provide more specific definitions or other details. Further, several of the schedules had additions. For example, the Patient Protection and Affordable Care Act led to additional reporting on Schedule H, Hospitals, to fulfill requirements that hospitals report on each of their facilities and conduct a Community Health Needs Assessment every 3 years. Most of the Form 990-EZ’s 27 changes occurred in tax years 2009 through 2012, of which 12 were for 2011 and several of them focused on compensation reporting. IRS also made 27 clarifications for 2009-2013. Public Law 115-97 did not affect Form 990-EZ. There were no changes to the 2019 form. See table 10. For the Form 990-PF, IRS made the fewest changes compared to Forms 990 and 990-EZ, with only 11 changes and four clarifications for tax years 2009 through 2019. The Form 990-PF had three changes prompted in 2018 by Public Law 115-97. Electronic filing requirements apply to Form 990-PF for tax years starting July 2, 2019, but there were no other changes for the 2019 form. See table 11. Appendix IV describes the general examination selection process for exempt organization returns, and specific classification steps that apply to certain returns. The annual work plan is the foundation for identifying and assigning returns for examination. The Compliance Planning and Classification (CP&C) office follows various steps to identify returns to fulfill the work plan, which end in the assignment of returns for potential examinations to field work groups. The intended process is in figure 11 and discussed below. Annual work plan. CP&C’s Planning and Monitoring group develops the annual work plan. The work plan provides estimates of examination starts and closures. It also has estimates for the number of hours to be spent per return examination and the number of days to complete an examination. Planning and Monitoring develops estimates at the project code level, which corresponds to a specific examination source or project such as the Form 990 model. The Tax Exempt and Government Entities’ (TE/GE) Compliance Governance Board approves the work plan. TE/GE provides a summary of the work plan in its annual Program Letter. Stocking report. The Planning and Monitoring group uses the work plan to issue “stocking” reports to guide classifiers on types of returns to identify for potential examination. Planning and Monitoring considers available examiners, and progress in meeting work plan numbers. The report lists the number of returns needed by grade, project code, and classification source. Classification. Classifiers review stocking plans to identify returns for potential examination. Classifiers are to eliminate returns for consideration if the (1) return is approaching its statute of limitation date, (2) organization has been examined in the last 3 years, or (3) organization is under a compliance check. Establishing the return and initial case building. If classifiers identify examination potential, they establish returns in the Audit Information Management System and Reporting Compliance Case Management System (RCCMS). The returns are sent for initial case building—developing paperwork to initiate the examination— according to a TE/GE official. Virtual shelf. Established returns and the initial case material are sent to the virtual shelf, which is an electronic inventory of returns that may be assigned for examination. Certain referrals, claims, compliance strategies, and other returns are prioritized, according to a TE/GE official. Returns remain on the shelf until assigned for examination or otherwise closed due to statute of limitations, according to a TE/GE official. Examination assignment. Functional Assignment Coordinators pull returns from the virtual shelf to fulfill field group work requests. Returns on the virtual shelf that matched a work order undergo additional case building before delivery to field examination groups. Monitoring. Planning and Monitoring staff regularly review reports that compare work plan goals with current work, and run algorithms to forecast upcoming work. These reviews are intended to ensure that sufficient work is available for assignment, excess work is not created, and returns approaching statute of limitations are identified. The monitoring informs new stocking reports. Classification steps vary depending on how a return was identified for potential examination. For returns identified with queries or models, classifiers check a limited set of criteria once a return is identified. For returns identified through other sources, such as referrals, the classifier also reviews facts and circumstances about potential noncompliance in returns. We focus here on examination sources that rely on data—such as models or queries—and referrals. Referrals are complaints of exempt organization noncompliance made by third parties, including the public and other parts of the Internal Revenue Service. We describe referrals classification because it is one of the top sources of exempt organizations examinations. The models are run to identify returns with potential noncompliance and lists them on a Model Score Sheet (MSS). The MSS is a ranked list of returns by scores from the model. According to a TE/GE official, the classifier: works down the list, starting with the highest scores, to fill stocking checks whether the return was also identified for a compliance strategy; and. eliminates returns based on the statute of limitations and recent examination activity. For some projects in the Compliance Strategies portfolio, a query is run or returns are sampled to identify a population meeting indicators of potential noncompliance. Then, the classifier uses the stocking report to select returns with certain geographic or case grade criteria and eliminates returns based on statute of limitations, recent examination status, and resolving non-filing issues, according to a TE/GE official. TE/GE classifiers do a triage to review and eliminate referrals that are not relevant to tax administration or do not have substantiated information. The triage classifier sorts referrals and reviews the following: organization status (for example, already revoked or terminated); examination history of the organization; and evidence of substantial inurement or private benefit, non-exempt activities, or material employment tax or unrelated business income that would result in a significant tax assessment. Referrals that pass triage are either sent to classification or, if they deal with political issues, are sent to a committee of three TE/GE managers, who vote on a selection decision. For all referrals, the classifier researches the referral. Research sources include websites, external databases, and IRS taxpayer account databases. The classifier may look at the organization’s website, information about officers, or prior examination history. Referrals with examination potential are either assigned immediately or placed on the virtual shelf. Referrals that must be immediately assigned include those with strong indicators of fraud, illegal or illicit activities (including terrorism), or referrals from whistleblowers, or certain other IRS divisions. Other referrals are labeled as high, medium or lower priority, based on potential for revocation or significant tax assessments. The Tax Exempt and Government Entities division (TE/GE) hired a contractor to review the effectiveness of its Form 990 examination selection model. The contractor prepared two reports. The first, delivered in January 2018, makes recommendations on the model process, the computing environment, and performance measures. The second, delivered in July 2018, makes recommendations to more effectively and efficiently identify returns for examination, such as through the model. Within the two reports, the contractor made 17 recommendations. Table 12 lists the recommendations and the status of each. In September 2019, TE/GE initiated another study, anticipated to be completed in September 2020. The study focuses on developing alternatives to enhance the models. The study will explore architectures and alternative designs for the model and propose alternative compliance actions to examinations and recommend measures to monitor their effectiveness. In addition to the contact named above, Tom Short (Assistant Director), Lindsay Swenson (Analyst-in-Charge), Ann Czapiewski, George Guttman, Amalia Konstas, Krista Loose, Alan Rozzi, Cynthia Saunders, Andrew J. Stephens, and Sonya Vartivarian made key contributions to this report.", "summary": "Exempt organizations often provide charitable services, or in some instances, membership benefits in furtherance of an exempt purpose. They generally do not pay federal income tax. IRS examines exempt organization returns (Form 990 and others) to address noncompliance, which may promote confidence in the tax exempt sector. In 2016, IRS started using three analytical models using Form 990 data to identify potential noncompliance and select returns for examination. GAO was asked to review IRS's use of Form 990 data. This report assesses (1) IRS's use of data to select returns for examination and, (2) the process IRS has established for selecting returns. GAO analyzed (1) examination data from fiscal years 2016 through 2019 including results from the largest Form 990 model, and (2) model documentation for a generalizable sample. GAO interviewed IRS officials and assessed IRS policies and procedures using relevant standards for internal control. The Internal Revenue Service (IRS) used data to select almost 70 percent of its examinations of Form 990 returns in fiscal year 2019. Almost half of these examinations were selected using models that score returns for potential noncompliance (see figure). Of the returns examined that were selected using the model, 87 percent resulted in a change to the return, indicating that IRS identified noncompliance. GAO found that the model did not improve change rates compared to prior selection methods and a higher model score is not associated with a higher change rate. IRS has not fully implemented or documented internal controls in its established processes for analyzing data for examination selection. For example: IRS has not defined measurable objectives for using data to select returns for examination . Without measurable objectives, IRS cannot assess how well it is doing or fully implement other internal controls. IRS's models have deficiencies affecting the validity and reliability of return scoring and selection . IRS has incomplete definitions and procedures and did not always follow its definitions when assigning point values for identifying potential noncompliance for examination. As a result, return scoring by the models is not always consistent. IRS did not consistently document the processing and use of data in decision-making on examination selection . Without such documentation, IRS cannot support its use of data in examination selection in all cases. IRS does not regularly evaluate examination selection. Examination data were inconsistent across years and IRS only tracks one prior year of data. IRS also did not save data on all returns that the models scored. Without data and regular evaluations, IRS cannot assure that its models are selecting returns as intended and that deficiencies are identified and corrected. GAO makes 13 recommendations, including that IRS establish objectives, revise model documentation, fully document processing and using data in decisions, and regularly evaluate examination selection. IRS agreed with all recommendations except one related to evaluating examination selection methods using consistent historical data over time. GAO continues to believe that this recommendation is valid as discussed in the report.", "document_type": "gao"}
{"report": "American Samoa consists of five volcanic islands and two coral atolls in the South Pacific, about 2,600 miles southwest of Hawaii (see fig. 1). American Samoa has a combined land area of 76 square miles, slightly larger than Washington, D.C. Approximately 98 percent of the population of American Samoa lives on the main island of Tutuila, and most economic activity (including tuna canning) and government operations take place in and around the harbor of the capital city, Pago Pago, on Tutuila (see fig. 2). Most of Tutuila consists of rugged terrain with little level land. With a significant portion of its population and infrastructure located in low-lying coastal areas, American Samoa faces the risk of tsunamis and other coastal hazards. In September 2009, a tsunami following a magnitude 8.1 earthquake left 34 people dead in American Samoa, and caused severe damage to homes, businesses, and water and electrical infrastructure. In February 2018, Tropical Storm Gita struck the territory, causing damage with at least 50 percent of American Samoan residents facing some level of loss to property, according to American Samoa Department of Commerce estimates. The American Samoa government estimates that the disaster caused nearly $200 million in damages to public and private property. In response to both natural disasters, the federal government issued major disaster declarations and assisted with recovery efforts. The 2010 U.S. Census found American Samoa’s population to be 55,519, a decrease of 3 percent from its 2000 population. Individuals who are neither U.S. citizens nor U.S. nationals, most of them from the Independent State of Samoa, constituted approximately 35 percent of the territory’s population in that year. BEA most recently estimated American Samoa’s 2018 population to be approximately 58,000. The 2010 census also reported that American Samoa’s median household income remained well below, and its poverty rate well above, that of the United States. In 2009, American Samoa’s median household income was $23,892, 47 percent of the U.S. median household income, while its poverty rate was 57.8 percent, nearly four times the U.S. rate of 15.1 percent. U.S. interest in the Samoan islands began in 1872 with efforts by the U.S. Navy to establish a naval station in Pago Pago Harbor. A U.S.-British- German protectorate over all Samoan islands ended in 1899, when the islands that constitute American Samoa were placed under U.S. control. The U.S. Naval Station in the territory was established in 1900. From 1900 through 1904, the U.S. government negotiated control over American Samoa, and the U.S. Navy subsequently took responsibility for federal governance of the territory. In 1951, governance was transferred to the Secretary of the Interior. In 1960, American Samoa residents adopted their own constitution, but amendments to the constitution may be made only by an act of Congress. Persons born to non-U.S. citizen parents in American Samoa are U.S. nationals but may apply to become naturalized U.S. citizens. In addition, U.S. non-citizen nationals from American Samoa have the right to travel freely, live, and work throughout the United States. American Samoa exercises authority over its immigration system and customs through locally adopted laws. While American Samoans may serve in the U.S. military, they do not have voting representation for legislation passed before the full U.S. Congress, including legislation setting the minimum wage in American Samoa. The United States provides assistance to the American Samoa government, including funding the majority of its revenue. In fiscal year 2018, the American Samoa government’s financial audit reported that U.S. federal grants provided approximately $150 million of $246 million in total American Samoa government revenue. Ranked by approximate grant expenditures, the largest federal grantors were the Departments of Health and Human Services ($43 million), Agriculture ($33 million), Interior ($30 million), Education ($28 million), Transportation ($18 million), and Homeland Security ($5 million). The federal minimum wage was first enacted as part of the Fair Labor Standards Act of 1938 (FLSA). The FLSA specified that for industries engaged in commerce or in the production of goods for commerce, its policy was to correct and, as rapidly as practicable, to eliminate labor conditions detrimental to the maintenance of the minimum standard of living necessary for health, efficiency, and general well-being of workers without substantially curtailing employment or earning power. Since 1938, there have been nine amendments to the FLSA establishing new minimum wages and usually raising the rate through a series of steps over 2 to 4 years. The FLSA was amended in 1956 to provide for American Samoa minimum wages to be established through a special industry committee (SIC) process similar to that used in Puerto Rico and the U.S. Virgin Islands. Federal policy called for the minimum wage rates for industries in American Samoa to reach the federal level as rapidly as was economically feasible without substantially curtailing employment. The final SIC, which recommended minimum wages to be applied in 2005 and 2006, recommended minimum wages for 18 industry categories. These 18 industry categories remain in existence for present-day minimum wages in American Samoa. Since 2007, U.S. federal law has determined minimum wages in American Samoa. In 2007, Congress passed the Fair Minimum Wage Act of 2007, which eliminated the SICs and created a schedule of increases to American Samoa minimum wages that has since been revised and applied over a number of years. The Fair Minimum Wage Act of 2007 amended the FLSA, raising the federal minimum wage in a series of three steps from $5.15 to, effective July 2009, $7.25 per hour. The amended provision also eliminated the SICs in American Samoa and introduced a schedule for raising the minimum wages, by equal amounts, until all 18 minimum wage categories in American Samoa reached the federal level. According to the U.S. Department of Labor, when the law was enacted, nearly 80 percent of eligible American Samoa workers earned less than $7.25 per hour. The initial Fair Minimum Wage Act of 2007 schedule, which called for $0.50 annual increases, would have increased all American Samoa minimum wages to the current federal level by May 2016. After the initial (2007) schedule, each subsequent law revising the schedule of minimum wage increases for American Samoa extended the projected dates for American Samoa minimum wages to reach the federal level. Measures adopted in 2009 and 2010 retained the $0.50 increases but delayed their application, so that convergence between the American Samoa minimum wages and the federal level would have occurred in 2018 rather than 2016. Subsequent measures—applying increases every third year and reducing each increase from $0.50 to $0.40—delayed convergence of American Samoa minimum wages with the federal level by more substantial intervals. The current schedule establishes increases of $0.40 every 3 years for all 18 industry categories in American Samoa, with the most recent increase in September 2018 and the next increase scheduled for September 2021. If American Samoa minimum wages continue to increase by $0.40 every 3 years as scheduled, and if the current federal level does not increase, the highest minimum wage in American Samoa, for the stevedoring industry, will reach the federal level in 2027, while the lowest minimum wage, for the garment manufacturing industry, will reach the federal level in 2036. Minimum wages for the largest employer overall, government, and the largest private-sector employer, the fish canning and processing industry, will reach the federal level by 2036 and 2033, respectively. Table 1 shows past and projected minimum wages in American Samoa for these industries. (App. III shows the current federal minimum wage in American Samoa by industry.) Since 1957, American Samoa minimum wages have risen, first as recommended by SICs and then in accordance with schedules set by legislation. However, with the exception of 1986, when the highest American Samoa minimum wages—for fish canning and processing, petroleum marketing, and stevedoring—converged with the federal level of $3.35, American Samoa minimum wages have remained below the federal level (see fig. 3). American Samoa’s economy largely contracted during the past decade. Adjusted for inflation, gross domestic product declined by 18.2 percent from 2007 to 2017, though it increased by 2.2 percent in 2018. According to the American Samoa Department of Commerce, the 2018 uptick is likely to be temporary, partly reflecting reconstruction activity for Tropical Storm Gita. Changes in government spending and the tuna canning industry, including disaster-related federal funding and cannery closures, have impacted American Samoa’s economy. From 2007 to 2018, American Samoa employment varied by year without a clear trend, while workers’ inflation-adjusted earnings generally declined. American Samoa continues to depend on the territorial government and tuna canning industry as key sectors. The American Samoa government continues efforts to diversify the economy, and in recent years, these efforts have centered on the development of a new industry, telecommunications. Employment did not exhibit a clear trend, but varied from year to year from 2007 to 2018. Specifically, it ranged from about 16,000 to about 20,000 with a peak year in 2009. In 2018, employment was at the same level as it was in 2007, at about 17,000. Figure 5 shows the trend in employment in American Samoa over this period. In addition, we analyzed data from alternative sources, which also showed that employment lacked a clear trend from year to year. According to American Samoa Statistical Year Book data, employment ranged from about 14,000 to 19,000 from 2007 to 2017 with a peak in 2010. According to the U.S. Census Bureau’s County Business Pattern data, which mostly excludes certain groups such as the public sector, private sector employment ranged from about 7,000 to 10,000 from 2008 to 2017, with a peak in 2009. For more information, see appendix IV. Average earnings of employed workers contracted from 2007 to 2018 when adjusted for inflation. For the overall period from 2007 to 2018, average inflation-adjusted earnings fell by about 11 percent (from about $11,000 to about $10,000), reflecting an increase in average annual earnings of about 29 percent and an increase in prices of about 44 percent. For the most recent year available, 2017 to 2018, average inflation-adjusted earnings was almost unchanged—growing by about 1 percent. Figure 6 shows the trend in earnings in American Samoa from 2007 to 2018. For more information, see appendix IV. The territorial government and tuna canning industry are important sectors of American Samoa’s economy, contributing almost half of American Samoa’s employment and GDP. The American Samoa government and the tuna canning industry have historically employed the largest numbers of workers in American Samoa. In 2018, the government sector employed about 42 percent of the American Samoa’s workforce and the tuna cannery employed about 14 percent (see fig. 7). The territorial government continues to be the largest employer, while the tuna canning industry continues to be the largest private sector employer. The government and the tuna canning industry also remain large contributors to GDP in American Samoa. In 2017, government and manufacturing (primarily composed of tuna canning) contributed 42 percent of American Samoa’s total GDP (see fig. 7). The tuna canning industry plays a key role contributing to the territory’s trade, primarily through exports. According to U.S. Census Bureau data, processed tuna annually accounted for over 88 percent of exports from American Samoa to the United States from 1995 to 2018. According to American Samoa government officials, government and the tuna canning industry are the two main pillars of the economy and sustain other industries across the territory. The territory’s component units, including the Lyndon B. Johnson Tropical Medical Center, American Samoa Community College, American Samoa Power Authority, and American Samoa Telecommunications Authority, provide healthcare, higher education, utility, and telecommunications services, respectively. The tuna canning industry provides direct and indirect benefits to other industries. American Samoa Department of Commerce officials stated that the remaining cannery generates demand for support industries such as transportation and warehousing, retail and wholesale, and construction. American Samoa government officials also noted that the cannery’s large demand for shipping, transportation, and energy might reduce the cost of these services for the entire territory. In 2017, canned tuna constituted over 90 percent of American Samoa’s exports, and fish for processing constituted over 35 percent of American Samoa’s imports (see fig. 8). To reduce the territory’s dependence on its government and the tuna canning industry, the American Samoa government continues its efforts to diversify the economy. According to the American Samoa government, the territory’s dependence on the government and the tuna canning industry has exposed the economy to external risks, including changes in federal grant funding and global competition in the tuna canning industry. To reduce this dependence, the government has developed plans to diversify the economy. American Samoa’s economic development implementation plan for fiscal years 2014 to 2017 and economic development strategy for 2018 to 2022 outline economic development goals for sectors such as transportation and tourism, as well as action items to achieve these goals. The American Samoa government has identified ecotourism as an economic opportunity because the island’s mountains, tropical rainforests, coral reefs, and National Park may be attractive to tourists (see fig. 9). However, the American Samoa government has cited the federal restrictions on competition in passenger air carrier service to American Samoa as an impediment to developing the tourism sector. The United States restricts foreign airlines from carrying U.S. domestic passengers or cargo between U.S. locations, other than as part of a through trip involving a foreign location (cabotage), unless authorized by the U.S. Department of Transportation on the basis of specific criteria. According to the American Samoa government, as of August 2019, there are two passenger air flights per week between American Samoa and the United States (via Hawaii), with a third weekly flight added during peak travel seasons. American Samoa’s 2016 Workforce Innovation and Opportunity Act Unified Plan targets the development of five industries: fisheries and agriculture, telecommunications and information technology, manufacturing, visitors, and handicrafts. The plan notes that American Samoa is experiencing emigration of workers to the United States, countered in part by immigration of tuna cannery workers from neighboring islands to American Samoa. The plan cites low wages as a reason that high-skilled members of the labor force leave the territory. In recent years, the American Samoa government’s efforts to diversify the economy have centered on the development of the telecommunications industry. The government has made major investments in telecommunications infrastructure over the past 5 years. American Samoa Telecommunications Authority officials told us that they have managed the development of the territory’s telecommunications infrastructure projects. Completed in 2015, the Broadband Linking the American Samoa Territory (BLAST) project replaced the territory’s copper infrastructure with a fiber optic network capable of delivering high-speed data, voice, and cellular backhaul services. The U.S. Department of Agriculture’s Rural Utility Service funded the over $90 million project with an approximately $81 million grant and $10 million loan. According to American Samoa Telecommunications Authority officials, the Hawaiki cable project, completed and activated in 2018, added bandwidth to the BLAST network by connecting the territory via an underwater cable branch to the main Hawaiki cable trunk in Hawaii. The officials stated that the Hawaiki cable is a 15,000 kilometer, high- capacity underwater cable connecting Australia and New Zealand to the mainland United States, American Samoa, and Hawaii. The American Samoa government invested approximately $30 million to acquire its connection to the Hawaiki cable, using funding from American Samoa’s 2018 general revenue bond series. According to American Samoa Telecommunications Authority officials, other ongoing, multi-million dollar projects to enhance the territory’s telecommunications infrastructure include projects to upgrade BLAST bandwidth distribution and replace the territory’s 2G network with LTE technology. The American Samoa government believes that the newly activated Hawaiki cable and BLAST fiber optic network have raised the territory’s potential to develop new industries tied to telecommunications, including information communication technology and business process outsourcing. According to an American Samoa Department of Commerce survey of over 50 public and private stakeholders, 64 percent of respondents—the largest share—identified information communication technology as one of the most promising economic development opportunities for the territory. The next four most promising opportunities identified by approximate share of respondents (in parentheses) included “Attracting investors for capital investment projects (58 percent), “General Tourism” (47 percent), “Ecotourism” (47 percent), and “Federal Programs” (47 percent). American Samoa government officials acknowledge that despite progress made, American Samoa’s telecommunications industry is still at an early stage of development. The American Samoa government seeks to attract new telecommunications businesses, including a proposed call center, by identifying various competitive advantages for locating in American Samoa. American Samoa Department of Commerce officials stated that these advantages include an English-(American) speaking workforce with the lowest labor costs in the United States, and the territory’s qualification as an on-shoring location for call centers and other business process outsourcing operators. American Samoa Department of Commerce and American Samoa Telecommunications Authority officials stated that they are currently developing a territorial broadband strategy and proof-of- concept for a call center industry, expected to be released in mid-2020. Additionally, American Samoa Telecommunications Authority officials expect the Territorial Bank of American Samoa, opened in October 2016, to support the efforts to develop the telecommunications industry by encouraging investment in financial technology businesses. American Samoa Telecommunications Authority officials stated that the bank is partnering with the authority to develop internet banking services, which are expected to be offered in the next 2-3 years. American Samoa’s tuna canning industry faces multiple challenges, including increased competition and minimum wage increases, which led to cannery closures from 2007 to 2018. The companies that experienced the closures explained that minimum wage increases were a factor in the closures but not a main factor. With the closures, employment of cannery workers decreased, but inflation-adjusted earnings of cannery workers who maintained their jobs increased. StarKist Co. now operates the single remaining cannery in American Samoa, StarKist Samoa, but faces financial challenges. In addition to increased competition and labor market challenges, the industry faces other challenges, such as lower wages relative to those in American Samoa for cannery workers in other tuna-exporting countries. However, American Samoa offers the tuna canning industry advantages relative to the U.S. mainland and other countries, including lower wages compared to those in the U.S. mainland as well as duty-free access to the U.S. canned tuna market, according to StarKist Samoa officials. American Samoa’s tuna canning industry experienced cannery closures from 2007 to 2018 that adversely impacted the economy in that time period, as mentioned earlier. (For a timeline of selected events related to American Samoa’s tuna canning industry, see app. V.) StarKist Co., Chicken of the Sea, and Samoa Tuna Processors, which is owned by Tri Marine International (Tri Marine), have each operated or closed canneries in American Samoa over the years, as follows. StarKist Co. StarKist Co. (headquarters in Pittsburgh, Pennsylvania) has operated a cannery, StarKist Samoa, in American Samoa since 1963. StarKist Samoa is the one remaining cannery on the island, as mentioned earlier (see fig. 10). As of June 2018, StarKist Samoa employed 2,439 hourly wage workers. Chicken of the Sea. Chicken of the Sea (headquarters in El Segundo, California) operated a cannery in American Samoa, which it closed in September 2009. According to CRS, in the 1950s, the Department of the Interior contracted with Van Camp Seafood Company to move onto the island and develop a fish processing plant. Thai Union closed the Chicken of the Sea Samoa Packing cannery in American Samoa in September 2009. According to Chicken of the Sea officials, limited tuna supply was a key factor in the decision to close the cannery. The American Samoa minimum wage increases were a minor factor, but not as significant as other factors related to tuna supply, labor availability, logistics, and utility costs in contributing to the cannery’s closure. The company relocated its canning operations to the U.S. state of Georgia while outsourcing the more labor-intensive processes, including cleaning and cooking tuna loins (a low-tariff U.S. import), to countries with lower labor costs. By relocating to Georgia, Chicken of the Sea noted that it improved flexibility in sourcing and processing fish from multiple locations depending on where supply was readily available. Tri Marine International (Tri Marine). Tri Marine (headquarters in Bellevue, Washington) acquired the former Chicken of the Sea cannery in American Samoa in October 2010, undertook a multi- million dollar investment to renovate and expand it, and opened the new facility under the name Samoa Tuna Processors in January 2015. However, Tri Marine suspended its canning operations in American Samoa indefinitely in December 2016, primarily in response to highly competitive price setting across the global tuna canning industry, according to Tri Marine. Tri Marine explained that the American Samoa minimum wage increases were a minor factor—not as significant as rising price competition and high production costs, such as for utilities—in contributing to Samoa Tuna Processors’ closure. The company subsequently transferred its canned tuna sourcing operations from American Samoa to Thailand, Peru, and the Solomon Islands to take advantage of decreased production costs. According to a report by the Pacific Islands Forum Fisheries Agency, in 2018, StarKist Co. signed a 10-year lease agreement to use Tri Marine’s Samoa Tuna Processors facility for StarKist Samoa operations. According to a Tri Marine official, in 2019, the Bolton Group (Italy) completed its acquisition of Tri Marine. The acquisition did not include Samoa Tuna Processors, and the Tri Marine ownership change did not affect the 10-year lease agreement between StarKist Co. and Samoa Tuna Processors, according to the official. From 2007 to 2018, cannery employment in American Samoa fell from about 4,500 in 2007 to 2,469 in 2018, a decline of 45 percent. Most of the decline occurred in the period between 2007 and 2010, with the closure of the Chicken of the Sea cannery. Figure 11 shows the trend in cannery employment in American Samoa over this period. The inflation-adjusted earnings of cannery workers in American Samoa who have maintained their jobs during this period have increased by more than inflation. In American Samoa, the vast majority of cannery workers earn close to the minimum wage. Moreover, the hourly wage of minimum wage cannery workers has increased by more than inflation since 2007. Specifically, during this period, the minimum wage has risen by 70 percent (from $3.26 to $5.56, from the first half of 2007), while prices have increased by 44 percent. However, this analysis does not include those workers who have lost employment or have had hours cut. StarKist Co. faces continuing financial challenges because of legal issues, as follows. In 2019, StarKist Co. was sentenced to pay a criminal fine of $100 million, the statutory maximum, for its role in a conspiracy to fix prices for canned tuna sold in the United States. This fine amounts to almost three times StarKist Samoa’s cost of labor in 2018. According to StarKist Co.’s General Counsel, the company will potentially have to close the cannery in American Samoa and move operations to a foreign country to afford to pay the fine for price-fixing. For its role in price-fixing, StarKist Co. has faced—and may continue to face— lawsuits from wholesalers, food service companies and retailers, and customers. For example, in January 2019, StarKist Co. announced that its portion of a settlement with Walmart was $20.5 million, based on a combination of cash payment and certain commercial terms. In addition, in September 2017, StarKist Co. agreed to pay a $6.3 million penalty resulting from violations of federal environmental laws, according to the U.S. Department of Justice. The U.S. Department of Justice and the U.S. Environmental Protection Agency reached an agreement with StarKist Co. and StarKist Samoa, requiring a series of upgrades to reduce pollution, improve safety measures, and comply with important federal environmental laws at their tuna processing facility in American Samoa, the department reported. American Samoa’s tuna canning industry faces multiple challenges in addition to scheduled minimum wage increases. One challenge is rising competition in the global tuna canning industry, as the value of foreign processed tuna exports to the United States has increasingly exceeded the value of American Samoa processed tuna exports to the United States (see fig. 12). Specifically, tuna industry officials stated that firms in the U.S. canned tuna market are highly competitive in price setting as opposed to differentiating their product lines. A tuna canning industry official stated that price competition and the financial pressures of the recent anti-trust judgements have forced the U.S. canned tuna market into a cost-cutting environment. According to the same tuna canning industry official, firms must look to lower costs related to labor, energy usage, and shipping to remain competitive in the U.S. market. The official stated that firms implicated in the price-fixing scheme have agreed as part of a legal settlement resulting from a lawsuit to supply their product at lower prices. This puts more pressure on firms to implement cost-saving measures to maintain their U.S. market shares. For example, StarKist Samoa has implemented cost-saving measures to reduce labor and energy costs and has also raised prices and relocated business off the territory. American Samoa’s tuna canning industry also faces other challenges, as described below. Competitors’ canning production strategies. According to StarKist Co. officials, StarKist Co.’s main competitors implement a supply chain production process that spans more than one country. Conversely, StarKist Samoa’s full production process still occurs in American Samoa (see fig. 13). According to StarKist Samoa officials, the cost savings between a fully U.S.-based manufacturing process and an outsourced manufacturing process is substantial and places American Samoa at a distinct disadvantage. According to StarKist Samoa officials, StarKist Co.’s main competitors use a model that outsources the workforce-intensive process to extremely low-wage countries. They explained that StarKist Co.’s competitors clean, cook, and freeze the tuna before importing it—subject to an average tariff of $11 per metric ton—into the mainland United States, where it is then thawed and packaged. Furthermore, our analysis of the global tuna industry suggests that, under certain assumptions, this model can improve cost savings and competitiveness. See appendix VI for the results of our analysis of the global tuna industry and more details about the assumptions we used. Tuna canning industry officials also stated that a new production process combined with lower labor costs for packaging tuna in foreign countries decreases American Samoa’s competitiveness as a location of operation. Lower wages for cannery workers in other countries, relative to those in American Samoa. According to a tuna canning industry official, tuna canneries have moved operations from American Samoa to Thailand, Peru, and the Solomon Islands, in part because of the lower labor costs. According to an industry official, one prominent tuna exporting country offers wages as low as $10 dollars per day, whereas a full- time worker in 2020 at the cannery in American Samoa would earn over $44 per day. Upcoming minimum wage increases in American Samoa. Upcoming minimum wage increases in American Samoa will increase labor costs for the tuna canning industry. According to data provided by StarKist Samoa, most cannery workers in American Samoa would be impacted by a minimum wage increase. Specifically, over 90 percent of StarKist Samoa’s employment (roughly 2,200 workers) could be affected by the next minimum wage increase scheduled for September 30, 2021. At 2018 levels of employment, labor costs could increase by about $2 million at 2021 minimum wage levels. Decreased direct access to tuna supply. A number of factors have decreased direct access to tuna supply. The Pacific Remote Islands Marine National Monument regulations have had the biggest impact on tuna supply to the cannery, according to StarKist Co. officials. Also according to Starkist Co. officials, marine monuments in the region have closed fishing grounds to U.S. purse seine vessels that historically delivered tuna to local canneries in American Samoa, and the Rose Atoll Marine National Monument reduced fishing grounds in U.S. waters around American Samoa that were very important to the American Samoa longline fleet. In 2017, National Marine Fisheries Service removed a regulatory exemption that had allowed certain large U.S. longline vessels to fish in portions of the American Samoa Large Vessel Prohibited Area. Delivery volume from a Chinese tuna supplier that used to send fishing boats to supply canneries in American Samoa directly has decreased significantly as a result of China paying subsidies to Chinese fishing vessels in the Pacific, according to StarKist Co. officials. The subsidy draws potential tuna suppliers from the American Samoa market to the Chinese market, the officials stated. American Samoa offers the tuna canning industry certain competitive advantages relative to the U.S. mainland and other countries, as follows. Lower wages for cannery workers in American Samoa relative to those on the U.S. mainland. American Samoa offers lower labor costs relative to the U.S. mainland. For example, while the 2020 minimum wage for fish canning and processing in American Samoa is $5.56 per hour, Georgia’s wage for manufacturing is $15 per hour. Tariff-free access to the U.S. canned tuna market. According to StarKist Co. officials, U.S. trade policies provide tariff-free access to the U.S. market for processed tuna from American Samoa, while foreign suppliers generally are subject to tariffs for these goods. On average, foreign suppliers’ canned or pouched tuna is subject to an average tariff rate of 12 percent. However, U.S. trade agreements with certain countries may provide those countries tariff-free or reduced-tariff access to the United States. Tax credits provided by the federal and local government. The American Samoa tuna canning industry receives both federal and local tax benefits. U.S. tax policies have reduced federal taxes on income earned by qualifying U.S. corporations investing in American Samoa. Under the Internal Revenue Code, qualifying American Samoa tuna canneries have received an economic development credit for U.S. corporate income taxes. StarKist Samoa reported saving $5.9 million in 2016 through this tax credit. Canneries in American Samoa have also benefited from exemptions from local taxes. According to American Samoa government officials, the local tax exemption has allowed StarKist Samoa to reduce its corporate tax liability to the American Samoa government to 20–25 percent of the amount owed. According to American Samoa government officials, the total corporate and excise tax revenue loss to the American Samoa government is estimated to be $15–20 million annually. Federal procurement opportunities related to canned tuna. According to StarKist Samoa officials, operating in American Samoa offers access to certain U.S. government contracts that require U.S.-sourced and -processed fish, and allows them to comply with Buy American requirements. However, according to the officials, most school districts that enter into such contracts waive the Buy American requirements because StarKist Co. is the only tuna company that qualifies, and as a result, competitive bids reveal that the cost of domestic product is significantly higher than the cost of non-domestic product. StarKist Samoa reported that $15.8 million or 4 percent of its revenue in 2018 was from federal procurement that included contracts with the U.S. Department of Agriculture and the U.S. military. The American Samoa government and Chamber of Commerce both view the minimum wage increases as conflicting with sustainable economic development. Both expressed concerns about the reliance of American Samoa’s economy on the tuna canning industry and the potential negative impact of minimum wage increases on the remaining cannery in American Samoa. The American Samoa government stated that it supports a minimum wage that its economy can support. While the American Samoa government noted that it is committed to ensuring that the people of American Samoa can meet the basic cost of living, it stated that the impact of upcoming minimum wage increases on StarKist Co. would be extensive. The American Samoa government predicts that it would take years for the economy to recover if StarKist Co. should cease operations in American Samoa, and suggested that the burden of any economic impact would be on the federal government. The American Samoa government specified challenges that it believes StarKist Co. currently faces, including recent federal fines, decreasing supply of tuna, higher infrastructure costs in American Samoa compared to those of other countries, and increased regulation costs by the U.S. Coast Guard and U.S. Environment Protection Agency. In October 2019, the American Samoa Minimum Wage Task Force, commissioned by the Governor of American Samoa, provided us with its findings and recommendations. It reported that American Samoa's economy is unique and starkly different from the economies of all U.S. states and territories, and that, aside from the American Samoa government, the remaining and only pervasive economic driving force in the territory is StarKist Samoa. It also noted that its main objective is to express to the U.S. Congress the importance of involving the territory in the process of determining the applicable minimum wage for American Samoa. The task force identified various policy options and recommended that a combination of a moratorium on minimum wage increases and special industry classification or a special industry committee would increase and maximize the opportunity for local stakeholder participation. These have been long-standing positions of the American Samoa government. In response to a prior report, the American Samoa government requested we convey its position to the U.S. Congress to postpone the minimum wage increases. In response to another prior report, the American Samoa government recommended the pursuit of a U.S. Department of Labor-constituted committee similar to a special industry committee. While the American Samoa Chamber of Commerce noted that its employers support fair minimum wages for their workers, it stated that it supports any delay in minimum wage increases for the cannery until another economic option is feasible. The American Samoa Chamber of Commerce explained that while data show a shift in employment away from the cannery, StarKist Samoa still provides significant financial benefits to American Samoa in the form of decreasing fuel and shipping costs. The American Samoa Chamber of Commerce predicts that any increase in minimum wage could force the closure of StarKist Samoa and drive American Samoa into a recession. Public and private sector employers and workers we interviewed commented on the impact of minimum wage increases, including potential benefits and challenges. Potential positive impact on the livelihood of workers. Multiple employers and workers we met with stated that increasing the minimum wage would have a positive impact on the livelihood of workers. For example, a worker said that minimum wage increases have helped offset the increasing prices of imported products and excise tax products. Another worker stated that minimum wage increases help people to meet their community and church financial obligations. Some employers and workers noted that minimum wage increases improve customers’ ability to pay bills and their likelihood of using necessary services. Potential negative impact on the remaining cannery. Multiple workers and employers we met with generally stated that minimum wage increases could lead to a potential negative impact on StarKist Samoa. Multiple workers stated that such impact could result in a loss of jobs and increases in shipping costs, among other things. Some public employers were concerned that minimum wage increases could lead to the closure of the remaining cannery, and one of them stated that the potential closure was the main factor in the minimum wage increase discussion. One public worker stated that StarKist Samoa closing the remaining cannery is a major concern because the company is the main source of tax revenue. Another public worker added that having already seen a cannery close after minimum wage increases has raised concerns that it might happen again with StarKist Samoa. In addition, a private employer stated that after the Samoa Tuna Processors cannery closed in 2016, the employer’s retail sales decreased sharply and the economy now relies on the remaining cannery, StarKist Samoa. Increased recruitment and retention of workers. Some employers and workers we met with noted that a higher minimum wage could lead to increased recruitment and retention. For example, multiple employers noted the challenges of recruiting and retaining skilled workers given the low wages on the island, which often compel such workers to leave the island for better opportunities. One employer said that it could not recruit without minimum wage increases. Another employer stated that even low-paid workers often leave the island to obtain better pay in higher-paying countries. Some employers and workers noted that the lack of staff, especially nurses and teachers, has led to challenges, such as a negative impact on healthcare and education on the island. One of these employers stated that the minimum wage is too low and there is a lack of good teachers on the island. This employer was upset that the local school did not have a math teacher, noting that teachers leave or simply do not come to work because the pay is too low. One of these workers stated that nurses have moved off-island because their pay is too low and because overwork has contributed to potential health hazards. Keeping American Samoan youth on the island. Multiple employers and workers we met with were concerned that the current minimum wage was insufficient to keep younger American Samoans on the island, especially those who are college-educated. For example, an employer stated that there is a lack of young talent because there are no jobs on the island and pay is low. Another employer stated that some American Samoans earn degrees abroad and come back to American Samoa, but find that they cannot advance their careers on the island and leave again after 1-2 years. Some workers we met with spoke as parents about their children leaving the island, and became emotional upon sharing that they did not anticipate their children returning. Wage stagnation versus wage compression. While some workers we met with said they believed that a lack of an increase would lead to wage stagnation, other workers, as well as employers, we met with said they believed that an increase would lead to wage compression. Some workers noted not receiving pay increases despite working for an employer for many years and obtaining certifications. For example, a worker stated that if it were not for minimum wage increases, the worker would not receive any pay raises. However, another worker was upset that colleagues who had just started working were receiving more money than those who had been with the employer for many years. Funding for minimum wage increases. Multiple employers and workers said they were concerned about how future minimum wage increases could be funded. For example, a public employer noted that it did not oppose the minimum wage increases because the current minimum wage was barely enough to survive on, but was concerned about where the funding and revenue to sustain the increases would come from. Another employer stated that identifying additional funds for minimum wage increases is a major challenge. This employer noted that the company had not yet laid off workers to fund minimum wage increases, but might have to consider it. Public and private sector employers and workers we interviewed also commented on the following topics related to minimum wage increases: Proposed alternatives. Multiple employers and workers suggested alternative ways of increasing minimum wages in American Samoa. For example, an employer stated that minimum wages should be set based on the actual conditions of American Samoa rather than on what it believed to be an arbitrary federal schedule, and a worker stated that the U.S. Department of Labor reviewing the minimum wage and making changes is preferred to scheduled changes. Minimum wage levels. In addition, while cannery workers we met with generally agreed that the current minimum wage was sufficient, other workers, as well as some employers, stated that the current minimum wage and the scheduled minimum wage increases were insufficient. While cannery workers generally noted that they were happy to have previous minimum wage increases, they were fearful that future increases could lead to a loss of hours or complete job loss should the cannery close. However, other workers disagreed. For example, one worker explained that minimum wage increases did not keep pace with the cost of living. Another worker stated that 40 cents every 3 years is only about 13 cents per year, which the worker considered insufficient. Some employers and workers became emotional when speaking about their own financial situations or those of their workers, relatives, or fellow American Samoans. Varying degrees of impact on the livelihood of workers. While public employers generally stated that the impact of the minimum wage increases on their workers was minimal, private employers noted varying degrees of impact on their workers. Some public employers stated that the majority of American Samoa government workers are paid above the minimum wage, and noted that the minimal impact was a result of the local government raising the minimum wage to $5 ahead of the 2018 minimum wage increase to $5.21. Potential positive impact on the economy if the remaining cannery closes. Some private employers stated that there could be a positive impact on the economy if the remaining cannery closes. For example, a private employer stated that the American Samoa economy is so used to having the cannery as its pillar that it has not truly tried to diversify the economy. This employer added that American Samoa needed to continue shifting away from the cannery and toward the rest of the private sector. Another private employer stated that the American Samoa economy is better off without the remaining cannery and that an economist’s analysis of the cannery’s true impact on the rest of the economy is needed. Cost of living. Multiple workers, as well as some employers, were concerned that minimum wage increases could lead to increases in the cost of living, with some noting that the cost of living in American Samoa is already high because living on a remote island requires a high amount of imported goods. While a public employer stated that business owners pass the cost of minimum wage increases to customers, a private employer stated that business owners are unable to do so because of competition. Another employer noted concerns about inflation, stating that minimum wage increases might drive up prices and rent. Cultural considerations. Multiple employers and workers cited the importance of considering American Samoa’s unique culture when setting minimum wage increases. While some workers and an employer noted that the cost of living in American Samoa is unique in that communal land and living off the land through fishing and gardening could minimize housing and food costs, others noted that community and church financial obligations are significant costs. One employer stated that these costs could amount to up to a quarter of worker’s paycheck. A worker stated that nonresidents, like many of the cannery workers, pay much higher medical costs; and an employer stated that foreign workers pay much higher housing costs. Other economic challenges. Multiple public and private employers and workers cited an array of economic challenges other than minimum wage increases, including the high cost of living on the island and increased taxes and fees. For example, one employer stated that American Samoa government taxes and fee increases are more of a challenge than minimum wage increases. We provided a draft of this report to the U.S. Departments of Commerce, the Interior, and Labor, and the American Samoa government for comment. The Department of Commerce provided technical comments, which we incorporated as appropriate. The Department of Labor informed us that it had no comments. In its comments, reproduced in appendix VII, the Department of the Interior said it would be beneficial to the American Samoa government if we provided information on all potential alternatives to setting minimum wages, including the once-utilized special industry committees. Such a study was beyond the scope of this report, which focused on (1) economic trends including changes in employment and earnings since the minimum wage increases in American Samoa began in 2007, (2) the status of the tuna canning industry, and (3) stakeholder views on the minimum wage increases. In its comments, reproduced in appendix VIII, the American Samoa government noted that the draft report did not reference findings and recommendations of the American Samoa Minimum Wage Task Force, commissioned by the Governor of American Samoa. The task force recommended the establishment of a special industry committee and a moratorium on minimum wage increases to allow ample time for such a special industry committee to form. We have added information on these findings and recommendations. We are sending copies of this report to the appropriate congressional committees, the U.S. Secretaries of Commerce, the Interior, and Labor, and the Governor of American Samoa. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact David Gootnick at (202) 512-3149, or gootnickd@gao.gov; or Oliver Richard at (202) 512-8424, or richardo@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IX. This report updates our previous reports on the impact of minimum wage increases in American Samoa and examines (1) economic trends including changes in employment and earnings since the minimum wage increases in American Samoa began in 2007, (2) the status of the tuna canning industry, and (3) stakeholder views on the minimum wage increases. To examine economic trends including changes in employment and earnings, we analyzed gross domestic product data from the U.S. Bureau of Economic Analysis; tax and administrative data from the American Samoa government; and employment, earnings, and wage data gathered through an employer questionnaire that we submitted to American Samoa’s tuna canning industry. To examine the status of the tuna canning industry, we estimated changes in employment and earnings using the employer questionnaire, analyzed tuna trade data from the U.S. Census Bureau, and interviewed tuna cannery industry representatives and stakeholders. To examine stakeholder views on the minimum wage increases, we conducted interviews with officials from the American Samoa government and American Samoa Chamber of Commerce, and employers and workers from the public and private sectors. The federal sources generally used to generate data on employment and earnings in the United States, including the Current Population Survey and the Current Employment Statistics program, do not cover American Samoa. Therefore, we collected our own data on employment and earnings in American Samoa. Consistent with our prior reports, we reported on employment and earnings from 2007 to the most recent year available. Employment and earnings figures are based on our analysis of combined worker data from various sources. We used employer-level data that we obtained from the American Samoa Department of Commerce and Department of Treasury to measure the annual employment of the American Samoa government and its component units: (1) American Samoa Community College, (2) LBJ Tropical Medical Center Authority, (3) American Samoa Power Authority, and (4) American Samoa Telecommunications Authority. We used tuna canning industry employers’ responses to our employer questionnaire to estimate cannery employment and earnings. We used individual-level tax records that we received from the American Samoa Department of Treasury to measure annual employment and earnings in American Samoa’s private sector excluding the canneries. To adjust earnings for inflation, we relied on the Consumer Price Index (CPI) as provided by the American Samoa government. To estimate employment and earnings for non-cannery workers in the private sector, we relied on individual-level tax data that we obtained from the American Samoa Departments of Commerce and Treasury. We restricted the sample to tax records received for tax years 2005 through 2018. We excluded tax records that contained invalid values in the variables that uniquely identify employers and workers. We also excluded records that contained non-numeric values in Social Security withholdings. Together, these records accounted for less than 2 percent of all tax records in the sample between 2005 and 2018, and accounted for less than 1 percent of total gross wages during this period. In addition, we excluded a small number of tax records—26 out of over 130,000 total records during this period—that reported zero annual earnings under gross wages, Social Security, and Medicare wages. In addition, in less than 100 cases, we adjusted the reported gross wages of workers if the worker had reported Medicare or Social Security wages but had reported gross wages that were very extreme in value (for example, zero or over $300,000), under the assumption that these were data errors. We estimated annual employment by summing the number of workers reported by each employer, for employers for which there was at least one tax record reporting positive wages for a given year. Under this approach, it is important to note that if a worker had multiple employers, the worker was counted more than once. Because of data limitations, we did not include data for tax year 2015 in our analysis of employment and earnings in American Samoa. We excluded this year because, according to the American Samoa government, individual-level tax records for that year are incomplete. Consistent with this observation, we found that the data contained lower counts of employers and workers in the private sector excluding the canneries in tax year 2015 than in any other tax year between 2005 and 2018. With the exception of tax year 2015 data, we found the data on employment and earnings sufficiently reliable for the purposes of our reporting objectives. For more details on our methodology for estimating employment and earnings in comparison to our methodology used in previous reports, see appendix IV. To examine the status of the territory’s key private sector industry—tuna canning—we estimated changes in employment and earnings by submitting an employer questionnaire to American Samoa’s tuna canning industry. In accordance with other federal employment surveys and with our employer questionnaires for our 2010, 2011, 2014, and 2016 reports on the impact of minimum wage increases in American Samoa, our employer questionnaire requested employment and wage data for mid- June pay periods in 2016, 2017, and 2018 from American Samoa’s tuna canning industry—in this case, the territory’s one remaining cannery, StarKist Samoa. In our 2016 report, we asked for employment in the mid- January 2016 pay period. We used the 2016, 2017, and 2018 data to update and extend the time series of employment and earnings data received from our prior employer questionnaires provided to American Samoa’s tuna canning industry. We found the data collected through the employer questionnaire for prior reports and this report sufficiently reliable for the purposes of our reporting on changes in American Samoa employment and earnings from 2007 to 2018. Data based on employers’ questionnaire responses include the reported numbers of hourly workers as well as their annual earnings at the canneries as of June in the given year. The questionnaire asked separately for data regarding workers paid an hourly wage and workers paid an annual salary. For hourly wage workers, respondents were asked to provide the number of workers paid at each wage rate. For salaried workers, respondents were asked the number of full-time and part-time workers paid at each salary level. In compiling the questionnaire-based earnings data for a given year, we assumed that all hourly cannery workers earned the minimum wage for that year and worked all year. When the minimum wage changed midyear, we assumed that the original wage applied for the first half of the year and the revised wage for the second half of the year. To adjust earnings for inflation, we relied on the American Samoa CPI. Using employer questionnaire data, we determined the number of workers that would be affected by future minimum wage increases because their wages were at or below future scheduled minimum wage increases. We estimated the cost of future scheduled minimum wage increases by calculating the cost to the cannery of increasing each worker’s wages to scheduled levels. This estimate assumed that workers worked full-time and all year (i.e., 2,080 hours) and that the minimum- wage increase would not affect the wages of workers currently earning more than the minimum wage. In addition, we interviewed cannery representatives and industry experts to obtain their views on competitive challenges facing the industry, including changes in minimum wage rates, access to fishing grounds, and preferential trade status. To illustrate other potential tuna production scenarios, we developed a model where tuna production relocates from the current status quo in American Samoa to one of two alternative scenarios of loining or canning tuna, or both, in other locations. Changes in labor and tariff costs are compared to the status quo scenario in American Samoa. The model uses assumptions based on the tuna canning industry employment count from the employer questionnaire responses and information obtained during interviews with tuna cannery employers. (See app. VI for the results of our analysis and more details about the assumptions we used.) This model is an update of the model we used for our December 2016 report. To examine stakeholder views on the minimum wage increases, we conducted interviews with officials from the American Samoa government and American Samoa Chamber of Commerce, and employers and workers from the public and private sectors. During our fieldwork trip to American Samoa in October 2019, we conducted interviews with government officials, employers, other private sector representatives, and workers to obtain views and information on the minimum wage increases. In total, we conducted 15 interviews: five employer interviews (the American Samoa government and three of its component units, and StarKist Samoa), two employer group interviews (private employers that are American Samoa Chamber members and ethnic business employers), and eight worker group interviews. For the primary American Samoa government and StarKist Samoa, we conducted two worker group interviews for each. In the group interviews, we followed a standard protocol that asked for participants’ views on the impact of the minimum wage increases. We interviewed a nongeneralizable sample of employers and workers selected on the basis of key industry information from prior GAO reports and employment data from the American Samoa government. Specifically, we selected the following employers and their workers: (1) the American Samoa government, (2) StarKist Samoa, (3) American Samoa Medical Center, (4) American Samoa Community College, and (5) American Samoa Power Authority. To supplement these employers and workers, we requested that the American Samoa Chamber of Commerce identify additional employers and their workers on the basis of criteria related to the tuna canning, construction, and retail industries, among other things. The American Samoa Chamber of Commerce arranged a group of 15 employers and their workers belonging to its membership and related to the tuna canning, construction, and retail industries, as well as a group of eight employers related to the territory’s ethnic (including Filipino, Chinese, Korean) business community. Overall, the number of participants in each group interview ranged from four to 20, for a total of over 100 participants. The range in number of participants applies to all of the interviews, regardless of their composition. In addition, we reviewed data and interviewed officials from the U.S. Departments of the Interior, Commerce, and Labor. We also reviewed U.S. minimum wage laws and other relevant laws and regulations. We did not review the extent to which laws were properly enforced or implemented. The scope of our study also does not include workers in the underground economy, which would include employers that may not comply with laws, including tax, minimum wage, immigration, and other laws. We did not review compliance with laws as part of this study. We conducted this performance audit from June 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 2 summarizes key federal laws regarding minimum wages in American Samoa. Figure 14 shows a U.S. Department of Labor poster outlining federal minimum wage requirements for American Samoa employers subject to the Fair Labor Standards Act (FLSA). According to the department, all such employers are required to post this information in conspicuous places in every establishment where employees subject to the FLSA’s minimum wage provisions are employed to permit them to readily observe it. Table 3 compares American Samoa employment data from 2007 to 2018 obtained on this review, data reported in GAO-17-83, and data from the American Samoa Statistical Yearbook 2017, an annual report produced by the American Samoa Department of Commerce. Data obtained on this review were composed of American Samoa tax records, including individual-level data, and responses from a questionnaire submitted to StarKist. Data reported in GAO-17-83 relied on aggregate data and did not include individual-level tax records. The data in the tables derived from these different sources are broadly consistent, but there are differences in certain years. The largest gaps between the alternate sources are in 2009 and 2012. According to the American Samoa Department of Commerce, some temporary government workers are not reflected in the data reported in GAO-17-83. We also compared American Samoa private sector employment data that we obtained and analyzed to County Business Patterns private sector employment data, collected by the U.S. Census Bureau. Private sector employment data that we analyzed indicated 2,000 to 3,000 more workers employed than County Business Patterns private sector employment data, depending on the year. According to the U.S. Census Bureau, this may largely be because the County Business Patterns data capture employment during the week of March 12, while the tax data include employment throughout the year. In addition, given that the American Samoa manufacturing sector is largely composed of the tuna canning industry, we also compared cannery data that we obtained on this review to County Business Patterns manufacturing data, which County Business Patterns reports in selected years. As table 4 shows, cannery employment data are in a similar range as County Business Patterns manufacturing data. Table 5 compares American Samoa workers’ earnings data from 2007 to 2018 obtained on this review and data reported in GAO-17-83. As shown, the data are broadly consistent. We also compared American Samoa workers’ earnings data from 2007 to 2018 obtained on this review to County Business Patterns data. In general, average earnings estimates in the County Business Patterns data are somewhat higher than in the American Samoa tax data, as shown in table 6. However, both series show growth in earnings over the period of 2008 to 2017 of approximately 20 to 30 percent. As an additional test of the reliability of the individual-level tax data, we examined trends in the distribution of worker-level earnings. A prior GAO report found that the minimum wage increases narrowed the gap between lower- and higher-paid workers in American Samoa from 2007 to 2009. We first examined whether the tax data also show that the gap narrowed during this period and then examined trends through 2018. One limitation of this analysis was that it was restricted to the private sector excluding the canneries. Therefore, any patterns that we documented in this sector may not reflect changes to the American Samoa workforce as a whole. We began our analysis in 2006 to provide information before the Fair Minimum Wage Act of 2007. To measure a worker’s annual earnings, we summed all of the worker’s gross wages from his or her tax records in a given tax year. According to the tax data, in 2006, workers at the 50th percentile of annual earnings earned $6,031. This amount is only 8 percent higher than what full-time workers would have earned if they were continuously employed at the lowest minimum wage that was in effect in American Samoa in 2006 ($2.68 per hour). In comparison, in the same year workers at the 90th percentile earned $18,747, or 3.1 times higher than the 50th percentile. We found that earnings at the 50th percentile experienced a larger increase than earnings at the 90th percentile from 2006 through 2009, and this ratio fell to 2.7. Figure 15 depicts trends in the gap between lower- and higher-paid workers in American Samoa from 2006 through 2018, as measured using the tax data by the ratio between the 90th and 50th percentiles of earnings. Overall, from 2006 through 2018, this gap fell by 17 percent, from 3.1 to 2.6. The decline is attributable to a 48 percent increase in earnings for workers at the 50th percentile, compared to only a 23 percent increase for workers at the 90th percentile. The following events highlight changes in American Samoa’s minimum wages and the status of the tuna canning industry from 2007 to 2019: 2007. Fair Minimum Wage Act includes a provision to incrementally increase American Samoa minimum wages to the federal level ($7.25 per hour). Special industry committees that previously set minimum wages in American Samoa are disbanded. Minimum wages in American Samoa rise by $0.50 as federally mandated. Minimum wage for fish canning and processing workers becomes $3.76. May 2008. Minimum wages in American Samoa rise by $0.50 as federally mandated. Minimum wage for fish canning and processing workers becomes $4.26. May 2009. Minimum wages in American Samoa rise by $0.50 as federally mandated. Minimum wage for fish canning and processing workers becomes $4.76. September 2009. Chicken of the Sea closes its cannery in American Samoa. The company relocates its canning facilities to the U.S. state of Georgia while outsourcing labor-intensive processes to countries with lower labor costs. The Samoa earthquake and tsunami cause severe damage and leave 34 people dead in American Samoa. The federal government issues a disaster declaration and assists with tsunami recovery efforts. October 2010. Tri Marine International acquires former Chicken of the Sea facility in American Samoa, located adjacent to the StarKist Samoa cannery. American Samoa government grants Tri Marine International exemption from local taxes for 10 years. December 2012. American Samoa government grants StarKist Samoa exemption from local taxes for 10 years. January 2015. Tri Marine International opens $70 million Samoa Tuna Processors cannery after large capital investments in prior years to renovate and expand the former Chicken of the Sea cannery. September 2015. Minimum wages in American Samoa rise by $0.40 as federally mandated. Minimum wage for fish canning and processing workers becomes $5.16. December 2016. Tri Marine International indefinitely suspends operations at its Samoa Tuna Processors cannery in American Samoa. September 2017. StarKist Co. agrees to pay a $6.3 million penalty resulting from violations of federal environmental laws. October 2017. StarKist Samoa temporarily halts operations for 5 weeks because of fish supply setbacks and equipment upgrades. February 2018. According to American government estimates, Tropical Storm Gita causes nearly $200 million in damages to public and private property. The federal government issues a disaster declaration and assists with disaster recovery efforts. May 2018. According to a report by the Pacific Islands Forum Fisheries Agency, StarKist Co. signs 10-year lease agreement with Tri Marine International to sub-lease its Samoa Tuna Processors facility for use in StarKist Samoa operations. September 2018. Minimum wages in American Samoa rise by $0.40 as federally mandated. Minimum wage for fish canning and processing workers becomes $5.56. September 2019. StarKist Co. is sentenced to pay a criminal fine of $100 million for its role in price fixing. Although American Samoa’s tuna canning industry faces multiple challenges in addition to scheduled minimum wage increases, American Samoa offers the tuna canning industry certain competitive advantages relative to the U.S. mainland and other countries. To illustrate tuna canning costs for other business models, we compared the labor and tariff costs associated with three potential business models for the cannery operations currently used by firms in the global tuna industry. The following analysis provides cost estimates for the three possible models, assuming constant total production under each model. Our analysis considers only labor costs and tariffs to show the effect of variation across different models. Our analysis excludes other associated costs, including transportation and refrigeration, as well as costs associated with establishing multiple production locations. Therefore, we assume that shipping costs and other conversion costs, such as for electricity usage, are identical. We also assume that fixed costs for starting operations in a new location (i.e., search costs) are equal to zero. We assume the alternative country is Thailand, on the basis of prior related reports and interviews with relevant officials and stakeholders. All of the tariff and tax assumptions used in our analysis are based on input from tuna canning industry officials. Model A (maintaining all loining and canning in American Samoa): This is the current production process for the remaining cannery operating in American Samoa. Tuna processing currently performed in American Samoa remains entirely in American Samoa. The cannery located in American Samoa hires local and foreign workers to loin—clean, cook, and cut—and can the fish. With an estimated workforce of 2,000 employees in American Samoa, the associated labor cost was an estimated $23 million in 2019. The canned tuna from American Samoa is exported directly to the United States and, according to cannery officials who utilize this model, such canned tuna is eligible for tariff-free access to the U.S. market. The cannery, which is a qualified domestic corporation, according to cannery officials, receives an estimated $5 million as a federal tax credit. Model B (relocating loining to Thailand or another country with low labor costs and canning processed loins in the U.S. states): This is the current production process for a firm operating a cannery outside of American Samoa. The loining operation—the most labor-intensive part of the operation—would move to a country with low labor costs, such as Thailand, where the fish would be loined, sealed in pouches, and frozen. The loined, frozen fish would then be exported to the U.S. mainland, where it would be canned. With an estimated workforce of 1,700 employees in a country with low labor costs making $1.25 per hour, the associated labor cost would be $4.4 million; and with an estimated workforce of 300 employees in the U.S. mainland at $15 per hour, the associated labor cost would be $9.4 million. Therefore, the total associated labor cost in 2019 for this model would be $14 million. No workers would remain in American Samoa, and 300 workers would be employed on the U.S. mainland. The imported fish would carry an average tariff of $11 per metric ton. This model assumes that the firm operating outside of American Samoa would not qualify for the American Samoa economic development credit. Model C (relocating all loining and canning to Thailand or another country with low labor costs): This is an alternative production process for operating canneries outside of American Samoa. Tuna processing currently performed in American Samoa would relocate to a foreign country with low labor costs. All operations, including loining and canning the fish, would take place in this foreign country. With an estimated workforce of 2,000 employees in a country with low labor costs making $1.25 per hour, the associated labor cost in 2019 would be $5 million. No workers would remain in American Samoa and no workers would be employed in the U.S. mainland. The imported fish would carry an average tariff of 12 percent of export revenue. This model assumes that a firm operating outside of American Samoa would not qualify for the American Samoa economic development credit. Figure 16 shows that, considering labor and tariff costs along with tax credits, Model A has higher costs than Model B. Model B presents cost savings; however, importing processed loins to the United States would incur tariffs, and wages for canning in any of the 50 U.S. states would be higher than in competing tuna processing countries. Model C presents the highest combined labor and tariff costs and would result in an estimated 2,000 fewer jobs in American Samoa. Table 7 below shows how revenue and labor and trade costs are computed for each model. The following are GAO’s comments to the Department of Interior’s letter. With respect to paragraph 4 of the U.S. Department of the Interior’s letter above, the suggested further study was beyond the scope of this report, which focused on (1) economic trends including changes in employment and earnings since the minimum wage increases in American Samoa began in 2007, (2) the status of the tuna canning industry, and (3) stakeholder views on the minimum wage increases. In addition to the contacts named above, Emil Friberg (Assistant Director), Benjamin Bolitzer (Assistant Director), Justine Lazaro (Analyst in Charge), Samuel Huang, James Boohaker, Carl Nadler, Debbie Chung, Christopher Keblitis, Sara Daleski, Martin De Alteriis, and Alex Welsh made key contributions to this report. American Samoa: Alternatives for Raising Minimum Wages to Keep Pace with the Cost of Living and Reach the Federal Level. GAO-17-83. Washington, D.C.: December 2, 2016. American Samoa and the Commonwealth of the Northern Mariana Islands: Economic Indicators Since Minimum Wage Increases Began. GAO-14-381. Washington, D.C.: March 31, 2014. American Samoa and Commonwealth of the Northern Mariana Islands: Employment, Earnings, and Status of Key Industries Since Minimum Wage Increases Began. GAO-11-956T. Washington, D.C.: September 23, 2011. American Samoa and Commonwealth of the Northern Mariana Islands: Employment, Earnings, and Status of Key Industries Since Minimum Wage Increases Began. GAO-11-427. Washington, D.C.: June 23, 2011. American Samoa and Commonwealth of the Northern Mariana Islands: Wages, Employment, Employer Actions, Earnings, and Worker Views Since Minimum Wage Increases Began. GAO-10-333. Washington, D.C.: April 8, 2010.", "summary": "In 2007, Congress passed legislation that established a schedule of periodic increases that would have raised all minimum wages in American Samoa to the current federal level ($7.25 per hour) by 2016. However, subsequent legislation has postponed or reduced scheduled minimum wage increases. The most recent minimum wage increase in American Samoa occurred on September 30, 2018, but all minimum wages in American Samoa are not scheduled to converge with the current federal level until 2036. Pub. L. No. 111-5, enacted in February 2009, included a provision for GAO to report periodically on the economic impact of minimum wage increases in American Samoa. This report examines (1) economic trends including changes in employment and earnings since the minimum wage increases in American Samoa began in 2007, (2) the status of the tuna canning industry, and (3) stakeholder views on the minimum wage increases. GAO analyzed federal and American Samoa data for 2016 through 2018, and interviewed employers and workers in American Samoa selected on the basis of employment levels, among other criteria. Commenting on a draft of this report, the American Samoa government suggested creating a committee to set minimum wages in the territory and a moratorium on minimum wage increases until the committee is formed. The Department of the Interior suggested GAO conduct further study, including on the use of a committee to set minimum wages. The suggested further study was beyond the scope of this report. American Samoa's economy largely contracted during the past decade. Adjusted for inflation, gross domestic product declined by 18.2 percent from 2007 to 2017, and increased by 2.2 percent in 2018 (see fig.). While American Samoa employment varied by year from 2007 to 2018, workers' inflation-adjusted earnings generally declined. American Samoa's economy continues to depend on the territorial government and tuna canning industry as key sectors. Changes in government spending and the tuna canning industry, including cannery closures, have impacted American Samoa's economy. To reduce the territory's dependence on the government and the tuna canning industry, the American Samoa government continues its efforts to diversify the economy. American Samoa's tuna canning industry faces multiple challenges, including increased competition and minimum wage increases, which led to cannery closures from 2007 to 2018. The companies that experienced the closures explained that minimum wage increases were a factor in the closures, but not a main factor. With the closures, employment of cannery workers decreased but inflation-adjusted earnings of cannery workers who maintained their jobs increased. StarKist Co. now operates the single remaining cannery in American Samoa, StarKist Samoa, but faces financial challenges. In addition to increased competition and labor market challenges, the industry faces other challenges, such as lower wages relative to those in American Samoa for cannery workers in other countries. However, American Samoa offers the tuna canning industry advantages relative to the U.S. mainland and other countries, including lower wages compared to those in the U.S. mainland as well as duty-free access to the U.S. canned tuna market, according to StarKist Samoa officials. The American Samoa government and the American Samoa Chamber of Commerce (the Chamber) view the minimum wage increases as conflicting with sustainable economic development, but employers and workers GAO interviewed noted benefits and challenges presented by minimum wage increases. The government supports setting a minimum wage that the economy can support, while the Chamber supports delaying minimum wage increases for the cannery. Employers and workers GAO interviewed noted a potential positive impact on the livelihood of workers but a potential negative impact on the remaining cannery, among other things.", "document_type": "gao"}
{"report": "In 2017, three major hurricanes made landfall in the United States and historic wildfires struck California. According to FEMA, the 2017 hurricanes and wildfires collectively affected 47 million people—nearly 15 percent of the nation’s population. See figure 1 for a timeline of these major disasters. When disasters hit, state and local entities are typically responsible for disaster response efforts. The Stafford Act establishes a process by which the Governor of the affected state or the Chief Executive of an affected Indian tribal government may request a presidential major disaster declaration to obtain federal assistance. According to the DHS National Response Framework—a guide to how the federal government, states and localities, and other public and private sector institutions should respond to disasters and emergencies—the Secretary of Homeland Security is responsible for ensuring that federal preparedness actions are coordinated to prevent gaps in the federal government’s efforts to respond to all major disasters, among other emergencies. The framework also designates FEMA as the lead agency to coordinate the federal disaster response efforts across 30 federal agencies. The National Response Framework identifies 14 emergency support functions that serve as the federal government’s primary coordinating structure for building, sustaining, and delivering disaster response efforts across more than 30 federal agencies. Each function defines specific mission areas—such as communication, transportation, and energy—and designates a federal department or agency as the coordinating agency. For example, provision of assets and services related to public works and engineering, such as temporary roofing or power, are coordinated by USACE within DOD. See Appendix II for more information about emergency support function responsibilities across the federal government. FEMA’s Response Directorate coordinates disaster response efforts through mission assignments—work orders that it issues to other federal agencies to direct them to utilize their authorities and the resources granted to them under federal law in support of direct assistance to state, local, tribal, and territorial governments. Mission assignments are authorized by the Stafford Act, and agencies may fulfill these assignments through federal contracts. FEMA made 1,515 mission assignments for the 2017 hurricanes and California wildfires, and total obligations for these mission assignments were more than $7.8 billion as of January 2018, according to FEMA. See figure 2 for a depiction of the mission assignment process under a notional scenario of removing derelict marine vessels—boats and ships damaged during a hurricane and that are determined to be inoperable. The National Response Framework states that when an Emergency Support Function is activated in response to an incident, the primary agency for that emergency support function is responsible for executing contracts and procuring goods and services as needed, among other things. For example, DOD and USACE are the coordinators for Emergency Support Function 3—public works and engineering—and as part of this role, these agencies are responsible for emergency contracting support for lifesaving and life-sustaining services. As such, during the 2017 disasters, USACE obligated funds on contracts in support of its assigned mission of public works and engineering by restoring the electrical grid in Puerto Rico following Hurricane Maria and removing debris following the California wildfires. In its role as the lead coordinator of federal disaster response efforts across federal agencies, FEMA’s contracting workforce plays a key role in post-disaster contracts. FEMA’s contracting efforts are supported by its contracting workforce within FEMA’s Office of the Chief Procurement Officer (OCPO). In our prior work, we found that FEMA’s contracting workforce had grown significantly since Hurricane Katrina, but the agency struggled with attrition at times. While the majority of FEMA’s contracting workforce is located in headquarters, contracting officers are also located in each of FEMA’s 10 regional offices. See figure 3 for the location of FEMA’s 10 regional offices as well as the states and territories for which each one is responsible in terms of fulfilling National Response Framework duties. In addition, FEMA can deploy members of its Disaster Acquisition Response Team (DART), a group whose primary purpose is to support contract administration for disasters. There are two DART teams under FEMA’s Expeditionary branch, each comprised of contracting officers, contracting specialists, and quality assurance specialists. Figure 4 shows how FEMA’s contracting workforce is organized. In headquarters, FEMA’s contracting officers support a variety of functions, such as contracting for information technology needs, activities to prepare for and mitigate disasters, and disaster response. In the field, the disaster and field operations division manages contracting for disaster response efforts including: Logistics: delivering goods and services to support disaster survivors and communities, including life-sustaining commodities such as meals, blankets, and electricity generators, Response: coordinating capabilities needed immediately following a disaster, such as air and ground evacuation services and emergency sheltering, and Recovery: primarily supporting rebuilding efforts, including technical assistance programs. Regional contracting officers serve as the first response for contracting if a disaster occurs in their region. During a disaster, the regional offices can request additional contracting support from headquarters if needed. Contracting officers are typically located in each regional office’s mission support division, which provide essential administrative, financial, information technology, and acquisition support for the region. Each region is headed by a Regional Administrator who reports directly to the head of FEMA, the FEMA Administrator. In response to a 2009 DHS Inspector General Report, FEMA created a formal agreement to establish a new role for FEMA’s OCPO to oversee regional contracting staff. The Inspector General report found that regional contracting officers only reported to their respective supervisor in the region—who usually are not contracting officers—with no formal link to FEMA’s OCPO. The Inspector General recommended that only contracting officials should manage the technical performance of contracting officers. The report stated that having the contracting officer’s performance and career advancement controlled by someone who is not a contracting professional was an internal control risk and created a potential conflict-of-interest situation for the contracting officer. A subsequent 2011 agreement between the regions and headquarters states that a FEMA OCPO official will be the contracting officers’ performance reviewer and that the regional supervisors will continue to manage regional contracting officials’ day-to-day activities. As a result, regional contracting officers have a dual reporting chain to both FEMA OCPO in headquarters and to their supervisor within the region. In September 2015, we identified challenges with how the agreement was being implemented, particularly in that it heightened the potential for an environment of competing interests for the regional contracting officers. Specifically, we found that being physically located in a regional office where their regional supervisor is not a contracting professional gave contracting officers less standing to resist requests to perform duties outside of a contracting officer’s responsibilities or to resist pressure from program officials to make certain decisions. Further, we found that FEMA had not updated its 2011 agreement, even though the agreement states that FEMA OCPO and the regions will revisit it each year. We recommended that the FEMA Administrator direct FEMA OCPO and the regional administrators to revisit the 2011 agreement to, among other things, add details about the extent of operational control headquarters and regional supervisors should exercise to minimize potential competing interests experienced by regional contracting officers, and further detail headquarters and regional supervisors’ roles and responsibilities for managing regional contracting officers to improve coordination and communication. We also recommended, and FEMA agreed, that it establish a plan to review this agreement on an annual basis. As of January 2019, FEMA had not implemented these recommendations. After a major disaster is declared, FEMA establishes a joint field office, a temporary office through which it coordinates disaster response and recovery efforts with state and local governments and organizations. Once the need for disaster response and recovery ends and a joint field office is closed, the contracts supporting the disaster are returned to the cognizant regional contracting office. Congress enacted the Post-Katrina Emergency Management Reform Act of 2006 (PKEMRA) after shortcomings were identified in preparation for and response to Hurricane Katrina—one of the largest and most destructive natural disasters in U.S. history, which hit the Gulf Coast in 2005. PKEMRA included several provisions related to contracting, including: Contracting preference for local vendors. PKEMRA amended the Stafford Act to provide a contracting preference for local vendors. Specifically, for contracts or agreements with private entities, the provisions of the act state, in part: in general, for major disaster assistance activities, agencies shall provide a preference, to the extent feasible and practicable, to organizations, firms, and individuals residing or doing business primarily in the area affected by the major disaster or emergency; they may be set aside for local vendors, which means that only vendors residing or primarily doing business in the declared disaster area are allowed to compete for an award; those not awarded to local vendors shall be justified in writing in the contract file. After the enactment of PKEMRA, changes were made to the FAR to implement provisions regarding the award of set-aside contracts to local vendors. Figure 5 displays the steps a contracting officer must take to implement the preference for awarding post-disaster contracts to a local vendor based on related laws and regulation. Use of noncompetitive contracts using the urgency exception. Agencies are generally required to use full and open competition— achieved when all responsible sources are permitted to compete— when awarding contracts. The Competition in Contracting Act of 1984 recognizes that full and open competition is not feasible in all circumstances and authorizes contracting without full and open competition under certain conditions, such as in cases with an unusual and compelling urgency and the government would be seriously injured unless the agency is permitted to limit the number of sources from which it solicits offers (“urgency exception”). When DHS awards disaster contracts non-competitively based on the urgency exception, PKEMRA, as implemented in the Homeland Security Acquisition Regulation, restricts the period of performance to 150 days, unless the Head of Contracting Activity determines that exceptional circumstances apply. For other uses of the urgency exception, the FAR’s period of performance limit is generally no more than one year. Generally, exceptions to full and open competition must be supported by written justifications that contain sufficient facts and rationale to justify use of the specific exception. Depending on the proposed value of the contract, the justifications require review and approval at successively higher approval levels within the agency. Use of advance contracts. PKEMRA requires FEMA to establish advance contracts, which are typically needed to quickly provide life- sustaining goods and services, such as tarps and meals, in the immediate aftermath of disasters. While not required under PKEMRA, USACE also establishes advance contracts for supplies and services (e.g., generators for its temporary power mission) using its independent statutory authorities for emergency management, such as Section 5 of the Flood Control Act of 1941. In addition, DLA has an interagency agreement with FEMA to provide disaster commodities and services, including fuel. As such, DLA also has some advance contracts in place. In December 2018, we found that FEMA and USACE were the primary users of advance contracts. As of June 30, 2018, federal agencies obligated at least $5 billion through post-disaster contracts to support disaster response and recovery efforts after hurricanes Harvey, Irma, and Maria and the 2017 California wildfires. USACE and FEMA awarded over three quarters of the reported obligations on post-disaster contracts. However, data on post-disaster contracting are not comprehensive due to changes in the criteria for establishing and closing a NIA code and DHS’s inconsistent implementation of the criteria for closing codes. Specifically, we found DHS closed the codes for Hurricanes Harvey and Irma less than a year after the storms hit, compared to prior hurricanes when the NIA codes remained open for at least 5 years. As of June 30, 2018, federal agencies obligated at least $5 billion through post-disaster contracts in response to the three 2017 hurricanes and the California wildfires. Data on obligations for the California wildfires are limited to those contracts identified by two selected agencies in our review—FEMA and USACE—because no NIA code was established in FPDS-NG to track contracts specifically for the wildfire events at a government-wide level. The obligations on post-disaster contracts accounted for more than half of the $9.5 billion in contract obligations on contracts related to the three hurricanes and the 2017 California wildfires, with the remainder of the dollars obligated on advance contracts. See figure 6 for details on post-disaster and advance contract obligations by event. FEMA and USACE accounted for more than three quarters of the total obligations on post-disaster contracts for the three hurricanes. Because there was no NIA code for the 2017 California wildfires, we cannot identify government-wide obligations in FPDS-NG and, therefore, do not know which agencies had the highest contract obligations for the two wildfire events. Figure 7 provides details on known obligations on post-disaster contracts, by agency. About 63 percent of the obligations on post-disaster contracts, or $3.1 billion, was for services. See figure 8 for a breakdown of services and products by 2017 disaster. Five services across the 2017 disasters comprised nearly 80 percent of total obligations for services on post-disaster contracts. Contracts for repair and maintenance services comprised 38 percent of total obligations on post-disaster contracts for services, largely driven by the $1 billion obligated to support the power restoration effort in Puerto Rico following Hurricane Maria. Following Hurricanes Harvey and Irma, agencies primarily awarded post-disaster contracts for management support functions, such as call center services. See figure 9 for the top post- disaster contract services across the three hurricanes and the California wildfires. Of the $1.8 billion agencies obligated on goods through post-disaster contracts, 28 percent was on contracts for subsistence, such as food and water. Nearly 30 percent, or more than $530 million, of all obligations on post-disaster contracts for goods was on contracts for electric wire and power distribution equipment, almost all of which was for the power mission in Puerto Rico following Hurricane Maria. See Figure 10. Across all three hurricanes and the California wildfires, we found that the competition rate—the percentage of total obligations reported under competitive contracts—was about 75 percent for post-disaster contracts. This is an increase from the past since we previously found that the competition rate in the immediate aftermath of Hurricane Katrina was about 53 percent. Contracting for disaster relief and recovery efforts presents unique circumstances in which to solicit, award, and administer contracts. Under the FAR, agencies are generally required to use full and open competition when soliciting offers, with some exceptions. As discussed earlier, an agency may award a contract without full and open competition, for example when the need for goods and services is of such an unusual and compelling urgency that the federal government faces the risk of serious financial or other type of loss, unless the agency is permitted to limit the number of sources from which it solicits offers (“urgency exception”). When using the urgency exception, the FAR requires agencies to request offers from as many potential sources as practicable. Based on FPDS-NG data, we found that about 47 percent of obligations on post-disaster contracts were on contracts citing the urgency exception, with 63 percent of those obligations on contracts coded in FPDS-NG as using “limited competition.” Among our selected contracts, we also found that contracting officers implemented the urgency exception to seek offers from as many sources as possible in different ways. Of the 11 contracts in our sample that cited the urgency exception, five included abbreviated award time frames in the justification documentation. The full extent of disaster contracting—for both advance and post- disaster contracts—related to the 2017 disasters is unknown due to changes in the criteria for establishing and closing a NIA code in FPDS- NG and DHS’s inconsistent implementation of the updated criteria for closing codes. The NIA code data element in FPDS-NG was established following landfall of several major hurricanes in 2005 to enable consistent tracking of emergency or contingency-related contracting. Contracting officers select the applicable NIA code in FPDS-NG when entering related contract information into the system. Officials at GSA—the agency responsible for operating and maintaining FPDS-NG—stated there is little to no cost or administrative burden associated with establishing or maintaining a NIA code. Based on a memorandum of agreement (the agreement), GSA, DHS, and DOD are jointly responsible for determining when a NIA code should be established and closed. DHS delegated its role, on behalf of civilian agencies for disaster or emergency events, to its Office of the Chief Procurement Officer (DHS OCPO), and DOD, on behalf of military departments and defense agencies for contingency operations, delegated its role to the Defense Contract and Pricing office. The agreement outlines criteria DHS and DOD should consider in making determinations to establish and close a NIA code. We identified changes in the criteria for establishing and closing a NIA code between a June 2012 agreement and a June 2018 update that superseded and replaced it. According to DHS OCPO officials, the agencies updated the agreement to incorporate lessons learned (such as adding that events should have a procurement impact as criteria for establishing a NIA code), and because it had not been revisited in 6 years. See table 2 for criteria from the agreements, changes in 2018, and examples of potential implications of those changes that we identified related to emergency or disaster events. The June 8, 2012 agreement criteria applied to the establishment of NIA codes for the 2017 disasters, while the June 1, 2018 updates applied to determinations to close or extend the NIA codes after this date for the 2017 disasters. DHS OCPO requested that a NIA code be established for each of the 2017 major hurricanes (Harvey, Irma, and Maria). However, the codes for Harvey and Irma closed on June 30, 2018, less than a full year after the hurricanes hit. The code for Maria was scheduled to close on December 15, 2018, and in August 2018 we began raising questions about the planned or actual NIA code closures for the three 2017 hurricanes. Since December 2018, DHS OCPO provided two additional extensions for Maria, with the code now valid through June 15, 2019, about 21 months after that hurricane made landfall. In contrast, the NIA code for Hurricane Sandy, which made landfall in October 2012, remained open until December 2017, more than 5 years after the disaster. The NIA code for Hurricane Katrina, which made landfall in August 2005, remained open until August 2018, 13 years after the disaster. We observed that DHS OCPO requested NIA codes for Hurricanes Florence and Michael in 2018, although we did not review the data associated with those events. After we sent this report to the agencies for comment on February 15, 2019, the agencies allowed the codes for Florence and Michael to expire, on March 15, 2019 and April 12, 2019, respectively. DHS OCPO officials offered several different rationales to support their decision to close the NIA codes for the 2017 hurricanes and cited the changes to the criteria in the 2018 agreement for closing the codes. However, we found that these rationales were inconsistent with the criteria in the agreement, did not consider key user needs, and did not fully explain the decisions to close these codes. For example: DHS OCPO officials told us that NIA codes for disasters should be closed when agencies no longer use the special emergency procurement authority such that the procurement thresholds—such as the simplified acquisition and micro purchase thresholds—return to the general (non-emergency) procurement thresholds in the FAR. Further, when FEMA requested to keep the codes open, DHS OCPO questioned why agencies would need to continue tracking with a NIA code after the thresholds had returned to general procurement thresholds. DHS officials stated that the updated agreement put an emphasis on this criterion; however, our analysis indicated that was not consistent with 2018 agreement, which includes multiple criteria and is not limited to this factor. Further, the agreement does not provide additional emphasis on one criterion over others. DHS OCPO officials stated that the purpose of the NIA code is to track federal procurement related to response, not recovery efforts. However, both the 2012 and 2018 agreements specifically state that the NIA code is intended to track disaster response and recovery efforts. Further, according to the National Response Framework and National Disaster Recovery Framework, we found that there are no clear lines of distinction between the start and end date of these two efforts, and often these stages of the process overlap. Additionally, FEMA officials from the Recovery Support Function Leadership Group’s Program Management Office stated that they use the NIA code to track government-wide contracting related to recovery efforts. The Recovery Support Function Leadership Group, an interagency body chaired by FEMA, tasked the Program Management Office with providing accountability and transparency of projects and outcomes for the 2017 disasters, among other things. DHS OCPO officials pointed to the Digital Accountability and Transparency Act of 2014 as providing alternatives to FPDS-NG. The Digital Accountability and Transparency Act of 2014 required improvements in the quality of data on federal spending, including disaster spending, by making data more accessible and transparent, such as by improving the quality of data submitted by federal agencies to USASpending—an online tool that tracks federal grant, loan, contract, and other awards. However, we found that USASpending provided some information on contract obligations using disaster response and recovery funds but does not separate obligations by disaster event. Further, our prior work on the Digital Accountability and Transparency Act of 2014 has found limitations with the data agencies provide, notably the completeness and accuracy of data. Specifically, we found that agencies routinely provided award descriptions in an abbreviated way and lacked clarity needed to compare data across the federal government. Moreover, we found inconsistencies in agencies’ ability to track contract actions by disaster. While FEMA has the capacity to provide contract information by disaster through a centralized contract tracking tool, USACE officials stated that they use a decentralized tracking process where they reach out to the districts and centers to identify and track disaster contracts without a NIA code. Prior to the June 30, 2018 decision to close the NIA codes for Harvey and Irma, DHS OCPO officials told us they found that the number of actions FEMA was making for these events had decreased. Our analysis of the NIA codes showed that components across ten departments, including within DHS and DOD, were executing contracts related to Harvey and Irma in June 2018. When we requested supporting documentation and analysis, DHS OCPO officials provided some correspondence with FEMA but did not provide government-wide data analysis to identify what other agencies were awarding and executing contracts related to these events. DHS OCPO officials stated they also sought input from DOD through the Defense Pricing and Contracting Office on whether to keep the codes open. According to DHS officials, DOD deferred to DHS on the decision because DHS was responsible for establishing the codes. Further, DOD officials did not provide evidence that would allow us to determine whether they assessed which defense components were executing contracts related to these events or sought the input of the components that were doing so, such as USACE and the Navy. FPDS-NG—a public, government-wide database of federal procurements—offers a resource the federal government can use to create recurring and special reports for key users, such as the President, Congress, executive agencies, and the general public. The NIA code in FPDS-NG provides consistent tracking and government-wide visibility into contracting related to disaster events through a publicly available database. Without clear criteria for establishing and closing NIA codes that consider the needs of data providers and users, such as FEMA, and the high visibility of the event being tracked and a mechanism to ensure consistent implementation of these criteria, insight into disaster contracting may be limited. Additionally, federal internal control standards state that management should use quality information, communicate quality information internally, and communicate quality information externally to achieve objectives. Management should accomplish this by considering appropriate methods for communicating externally, such as to the President, Congress, and the general public. As noted above, the 2018 agreement no longer includes the 2012 criteria that a NIA code can be closed if the NIA no longer has high visibility and there is no other interest in the NIA code. In our discussions with officials, DHS OCPO could not provide a rationale for these changes and the rationale is also not included in the updated agreement. Prior to DHS OCPO’s decision to close the codes for Hurricanes Harvey and Irma, a senior FEMA procurement official requested that they remain open, in part because of the high visibility of these events. As such, this official stated that there will be continued interest in the 2017 hurricanes including inquiries from Congress, which will require agency officials to pull data for interested parties, as that data can no longer be tracked and identified through public databases, such as FPDS-NG and USASpending. DHS OCPO officials denied FEMA’s request, pointing to the criteria in the 2018 agreement, which does not include consideration of the visibility of the event or key user needs. As the federal agency responsible for coordinating disaster response and recovery, FEMA is well positioned to understand the level of national and political interest in tracking procurement information for a disaster or emergency event. Yet, it is unclear why neither the 2012 nor the updated 2018 agreements included a role for or consideration of key users, such as FEMA and Congress. Further, as noted above, FEMA program officials expressed concern over closing the Harvey and Irma codes because they had planned to use the codes to assess recovery efforts for the 2017 disasters. As we have previously reported, it can take years to fully account for federal contract obligations related to response and recovery after a hurricane. Once a NIA code is closed, there is no publicly available, government-wide system to track contract obligations for specific events. Moreover, DHS OCPO officials were unable to provide data analysis conducted using available data from prior events to determine historical patterns in federal contracting obligations for disasters prior to closing the codes for Hurricanes Harvey and Irma. Figure 11 illustrates the lack of insight we have into disaster contracting activities related to the 2017 hurricanes, in comparison to what we know about prior storms with high federal procurement obligations. Further, using the description field in FPDS-NG, we found that between July 1 and September 30, 2018, after the NIA codes were closed, agencies obligated at least $136 million on contracts for Hurricane Harvey and $123 million on contracts for Hurricane Irma. While this provides some important insights regarding the continued contracting activity related to these hurricanes, the description field in FPDS-NG cannot be relied on to provide a full picture. Some agencies may include event- specific information in the description field; however, we found that, for the 2017 hurricanes, about 65 percent of contract obligations linked to a NIA code did not include event-specific information in the description. Without reopening the NIA codes for Hurricanes Harvey and Irma, and, to the extent practicable, retroactively populating the NIA codes for contract actions supporting response and recovery for these hurricanes during the period they were closed, decision makers are missing important information to understand the procurement impact of these disasters. Retroactively entering NIA code information is not unprecedented. For example, based on our analysis, the NIA codes for the 2005 hurricanes were established in October 2005, and contracting officers retroactively entered data for contracts related to these events which occurred as early as August of that year to enable full insight into contracting for these disasters. Based on the contracts we reviewed and officials we spoke with responsible for the planning of these contracts, we found that agencies experienced challenges planning for post-disaster contracts, especially when it came to contracting with local vendors. Additionally, FEMA also experienced challenges with requirements development—in that program officials did not always provide well-defined or sufficiently specific requirements for post-disaster contracts. However, FEMA has taken steps to address its challenges with requirements development, but it is too soon to tell the extent to which these steps will address the challenges we identified. Steps to Implement Local Vendor Preference, as Outlined in the Post-Katrina Emergency Management Reform Act and the Federal Acquisition Regulation (FAR) Step 1: Identify the set-aside area in accordance with FAR § 26.202-1—Local Area Set-Aside and § 6.208—Set-asides for Local Firms During a Major Disaster or Emergency Step 2: Conduct market research to determine whether there are qualified vendors in the set-aside area. Step 3: Issue a solicitation that provides for local vendor preference to the extent feasible and practicable either through the use of a set-aside or an evaluation preference. Step 4: Review offers based on evaluation criteria in the solicitation. If using a local area set-aside, review information from potential vendors to determine if they reside or primarily do business in the set-aside area in accordance with FAR § 52.226-3—Disaster or Emergency Area Representation. Step 5: Award contract to qualified vendor. If the vendor selected is not local or no qualified vendors are in the set-aside area, justify the decision in writing. determine that a vendor resides or primarily does business in the local justify in writing awards that they made to vendors outside the set- aside area. For the contracts we reviewed, contracting officials at FEMA correctly identified the local area for six set-aside contracts across the three hurricanes, and USACE correctly identified the local area for two set- aside contracts in Puerto Rico. However, based on the interviews we conducted during our review, USACE contracting officials were not consistently aware of the specific regulation for doing so and did not correctly identify the local area for two other USACE contracts awarded in support of the California wildfires. When awarding a local area set-aside or using an evaluation preference for local vendors, FAR § 26.202-1 states that a major disaster area can span several counties in several contiguous states, but need not include all the counties in the disaster area, and cannot extend beyond the counties designated in a Presidential disaster declaration. Figure 12 provides an example of a disaster declaration that depicts which counties could be included in the set-aside area. For all six local area set-aside FEMA contracts—awarded in response to Hurricanes Harvey, Irma, and Maria—we reviewed, FEMA officials defined the local area in accordance with regulation. This was an improvement from what we previously found. Specifically, in 2015, we found that FEMA contracting officers were confused about the definition of the set-aside area and recommended that the FEMA Administrator provide new or updated guidance to ensure all contracting officers are aware of requirements concerning contracting with local vendors, among other things. DHS concurred, and FEMA updated its annual disaster contracting webinar training to reiterate the requirement and clarify how to determine the geographic area using the disaster declaration. For the two local area set-aside USACE contracts awarded, officials responsible for those contracts told us that when awarding these contracts, they were not aware of the regulatory requirements for defining the geographic area of the local area set-aside. However, as the presidential disaster declaration for Hurricane Maria included the entire island of Puerto Rico, the local set-aside area covered the entire island. As a result, officials met the set-aside area requirement in accordance with regulation, even though they noted that they were not familiar with the requirement at the time. Officials told us they became aware of the regulation after conducting research pursuant to a protest related to the use of local vendor preference. We also reviewed two other USACE contracts that were used to support the debris removal mission following the California wildfires. Contracting officials stated that they conducted market research on the availability of local contractors, and they ultimately did not find qualified local firms. However, based on a review of contract file documentation, we found that USACE officials did not identify the local area in accordance with regulation for these contracts. Instead they used congressional districts that overlapped with impacted areas to identify the local area. We found that the areas USACE identified included areas outside of the geographic area defined by the presidential disaster declaration for the California wildfires. Contracting officials responsible for these debris removal contracts stated they were not aware of a policy or regulation for how to identify the geographic area for a local area set-aside, but that their office had internally determined the use of congressional districts impacted by a disaster to be the preferred method. A senior USACE official told us that there is no agency supplemental guidance or related training regarding the use of local vendor preference for contracts supporting disaster recovery and response, only that they expect USACE contracting officials to comply with the FAR. Without additional guidance or related training, contracting officers may be unaware of how to define the geographic area for a local area-set aside in accordance with regulation and may miss opportunities to support improving the local economies of disaster impacted areas by giving preference in awarding contracts to local vendors to the extent feasible and practicable, per the Stafford Act. Despite contracting officers having a high degree of discretion to determine that an offeror qualifies as a “local firm,”—that is, a firm that resides or primarily does business in the designated set-aside area— contracting and legal officials at both FEMA and USACE told us they were unsure what or how much information is sufficient to determine that an offeror qualifies as a local firm under the FAR. After contracting officials have identified the geographic boundaries of the local “major disaster or emergency area” and included required clauses in the solicitation and issued it as a local area set-aside, offerors must represent in their offer that they reside or primarily do business in the set-aside area. Specifically, FAR § 52.226-3(c) outlines two criteria a contracting officer should use to determine whether an offeror is to be considered “local.” If an offeror does not meet these first two criteria, FAR § 52.226- 3(d) provides eight additional criteria contracting officers may consider to make this determination (see sidebar). under FAR § 52.226-3(c) An offeror is considered to reside or primarily do business in the set-aside area if, during the last 12 months, 1) the offeror had its main operating office in the area; and 2) that office generated at least half of the offeror’s gross revenues and employed at least half of the offeror’s permanent employees. If the offeror does not meet the criteria under FAR § 52.226-3(c) consider other factors listed in FAR § 52.226-3(d) including: 1) Physical location(s) of the offeror’s permanent office(s) and date any office in the set-aside area(s) was established; 2) Current state licenses; 3) Record of past work in the set-aside area(s); 4) Contractual history the offeror has had with subcontractors and/or suppliers in the set-aside area; 5) Percentage of the offeror’s gross revenues attributable to work performed in the set-aside area; 6) Number of permanent employees the offeror employs in the set-aside area; 7) Membership in local and state organizations in the set-aside area; and 8) Other evidence that establishes the offeror resides or primarily does business in the set-aside area. Of the eight local area set-aside contracts we reviewed, two were impacted by bid protests—which is when an offeror challenges an award or proposed award of a contract or a solicitation—related to the FAR criteria for determining that an offeror qualifies as a local firm. The following protests show examples of the criteria agencies reviewed to determine whether a firm resided or primarily did business in a set-aside area. FEMA contract for food: In a protest of the award of a contract for food on the basis that FEMA improperly determined the protester failed to meet the requirements in FAR§ 52.226-3(d), the protester stated it met the requirements of FAR § 52.226-3(d), because it had (1) done past work in the set-aside area; (2) maintained a warehouse in the set-aside area; (3) maintained a contractual history with subcontractors in the set-aside area; and (4) maintained a current state license and filed a franchise tax return. FEMA denied, the protest stating that the evidence the protester provided was not sufficient to qualify as “residing or primarily doing business” in the local area. USACE Blue Roof contract: To support the Blue Roof mission— which provides temporary blue plastic roofs for disaster-impacted residences to prevent further damage and allow homeowners to arrange for permanent repairs—following Hurricane Maria in Puerto Rico, contracting officials awarded two post-disaster contracts. In a protest of the awards filed with GAO, the protestor argued, among other things, that one of the awardees did not meet local firm criteria in FAR § 52.226-3(c). USACE had assessed information on the awardee, including its local business address in the System of Award Management and other documentation of prior work in Puerto Rico, prior to award and determined that the awardee met Stafford Act criteria for award to a local vendor. USACE officials told us that, after the protest was filed, they further assessed information on the awardee in question and determined that it was a subsidiary of a larger national company. According to USACE officials, in order to quickly continue work on the Blue Roof mission, which had increased in scale, USACE negotiated pricing with the protestor while the protest was ongoing and made a third award under the solicitation. The protestor withdrew the protest. Contracting and legal officials at FEMA and USACE described difficulty in determining whether a vendor resides or primarily does business in the local set-aside area and cited a lack of clarity and different interpretations of the FAR. Based on conversations with the agencies’ legal officials, we found that USACE and FEMA applied the eight criteria in FAR § 52.226- 3(d) differently. FEMA officials told us that in determining whether a firm is local, if the first two criteria are not met, they evaluate an offeror’s information related to the eight criteria in FAR §52.226-3(d) to see if the first two criteria can be met with this additional information. They added that they look to see if the firm’s main operating office is in the set-aside area and if that office generated at least half of the offeror’s gross revenues and employed at least half of its permanent employees, but stated that the eight criteria do not need to be met within the last 12 months. Alternatively, USACE officials told us that in determining if a firm is local, if the first two criteria are not met, they evaluate an offeror’s information against the eight criteria in FAR § 52.226-3(d) independent of the two criteria described under FAR § 52.226-3(c). Legal officials at both USACE and FEMA stated that the FAR criteria should be clarified. Further, agencies’ varying application of the criteria increases the risk that an offeror may be considered local by some agencies, but not others. FEMA legal officials told us that contracting officers have been instructed to ask offerors for information on a local firm status in post-disaster solicitations. USACE legal officials explained that it is not always clear what specific information or documents provide the necessary information to meet the criteria under FAR § 52.226-3. For example, it may not be clear what documentation adequately demonstrates the number of permanent employees the offeror employs in the set-aside area, or the percentage of the offeror’s gross revenue earned in the set-aside area. The Office of Federal Procurement Policy provides overall direction of government-wide procurement policies, regulations, procedures, and forms for executive agencies. However, Office of Federal Procurement Policy staff told us that they have not provided additional guidance or clarification related to this FAR clause. Federal internal control standards state that management should use quality information to achieve objectives. Management should accomplish this by identifying information requirements, collecting relevant data from reliable sources, and processing data into quality information to be communicated internally and externally. Without clarifying guidance, contracting and legal officials will likely continue to have varying interpretations on how to implement the FAR criteria for determining that an offeror qualifies as a local firm. When contracts for major disaster or emergency assistance activities are not awarded to local vendors, the Stafford Act, as implemented in the FAR, requires that the decision be justified in writing in the contract file. Contracting officers at three of the four agencies included in our review— FEMA, USACE, and the Coast Guard—did not consistently justify in writing the award of selected contracts to non-local vendors. Specifically, 12 of the 14 contracts in our review that were not awarded to local vendors did not contain the required written justifications in the files (see table 3). DLA included written justifications for the use of non-local vendors, as required. After the 2017 disasters, FEMA identified the absence of justifications for the use of non-local vendors as an area for improvement. According to FEMA officials, they subsequently released guidance and a pre-solicitation memorandum to assist contracting officers in identifying what documentation related to local vendor preference is required in a contract file. FEMA officials told us they expect these steps will improve compliance with the requirement to document the justification for using non-local vendors going forward. While the Coast Guard provided a memorandum ahead of the 2017 disaster response that addressed the use of local vendors, it did not reference the requirement under the Stafford Act, as implemented in the FAR, to justify in writing the use of non-local vendors. A senior USACE official told us the agency had not issued any guidance to address requirements for contracting with local vendors and was not aware of any guidance issued at the department level. USACE legal officials noted the lack of written justification may be due to abbreviated timeframes under which post-disaster contracts are awarded. However, we found that USACE contracts included consolidated justification documents outlining rationales for the use of limited competition or abbreviated solicitation timeframes, but they did not include justifications for the use of non-local vendors. Without additional guidance or tools, contracting officials may not be aware that they are required to include written justifications for the use of non-local vendors in contract files, and federal agencies are at risk of not complying with the Stafford Act requirement to do so. Contracting officers responsible for the FEMA contracts we selected and senior procurement officials stated that during disaster response they received post-disaster requirements packages that were lacking in technical specificity or were otherwise deficient, but FEMA has begun to address this challenge. Program officials communicate contract requirements to contracting officers through requirements documents that include, among other items, a statement of work describing goods or services to be provided by an offeror, market research, and an independent government cost estimate. Contracting officials explained that when they received deficient documents, they had to conduct additional work to refine the requirements before soliciting for the contract—such as spending time assisting program officials to develop the required documentation. This additional work may add time to already tight award time frames for post-disaster contracts. When compared to large dollar value acquisitions, post-disaster contracts are awarded on significantly abbreviated time frames. For example, among the 12 FEMA contracts we assessed, time frames between the submission of a resource request and award date ranged from 1-26 days. This is faster than suggested; FEMA’s Procurement Administrative Lead Time guidance suggests preparation time frames of 60-300 days for new procurements based on the nature and value of an action. We found instances where FEMA program offices provided inaccurate or untimely estimates of the quantities of goods or services needed for the contracts we reviewed, in some cases leading to additional time and efforts spent to meet the need. For example: After Hurricane Harvey, FEMA awarded contracts to supply a food bank. Officials told us the initial requirement from the food bank through the program office to the contracting officer was expressed in terms of “truck loads” but did not specify, for example, how large the truck should be, or how many pallets should be loaded per truck. FEMA ultimately awarded three contracts to meet the post-disaster need—the first contract had a period of performance of 4 days and, according to FEMA officials, was intended to meet initial needs for food while the program and contracting officials determined the full scope of the requirement. The second contract—a $37 million contract with a period of performance of 52 days—was intended to fulfill the remaining requirement. However, due to miscommunication of the requirement as documented in the contract files and according to a program official responsible for the contracts, FEMA needed to award a third contract for an additional 2.5 months and $23 million to meet the need. Due to the value of the contracts, FEMA deemed that the subsequent contract required a new solicitation and award, rather than a modification to the existing contracts, thereby increasing the time and effort required of procurement personnel to meet the post- disaster need for food. In response to Hurricane Maria, FEMA awarded four post-disaster contracts for self-help tarps—which are used to cover small areas of roof damage. Of these contracts, two were terminated for convenience, both of which were included in our sample. The terminations were due in part to a national supply shortage. FEMA officials told us that under one of the contracts included in our review, at the request of the Commonwealth of Puerto Rico through program officials, FEMA ordered 500,000 40-foot-by-40-foot tarps, which differ from the size of the tarps normally ordered and stocked by the agency. Due to the supply shortage, FEMA received none, but officials noted that the impact of not receiving the tarps was minimal because the agency had initially overestimated the total number of tarps needed. Since the 2017 disasters, FEMA has started to address the issues with requirements development. Specifically, in 2018, FEMA officials told us the agency used portfolio managers in the field to assist with developing requirements for disaster response. Previously, in 2017, portfolio managers told us they supported the National Response Coordination Center but did not deploy to the disasters. Organizationally housed within FEMA’s OCPO, portfolio managers we spoke with told us they provide general templates for and guidance on acquisition documents for program officials to use and are primarily responsible for supporting steady-state acquisitions included in FEMA’s Master Acquisition Planning Schedule. Additionally, portfolio managers told us they provide informal, optional, “brown bag” training sessions for program officials. FEMA OCPO officials told us that they receive more requests for portfolio manager assistance than they can support, as the portfolio management section only maintains up to six staff. FEMA OCPO officials noted, however, that the agency expected to award an acquisition support contract to expand portfolio management capabilities. While the use of portfolio managers is an important step, it is too soon to tell the extent to which the use of portfolio managers in the field will address FEMA’s challenges with requirements development for post-disaster contracts. The agencies we reviewed each have a process for identifying lessons learned following a disaster, and we found they used these processes for the 2017 disasters. While agencies have identified actions they plan to take in response to the lessons they found following the 2017 disasters, additional challenges remain. Specifically, the agencies in our review encountered interagency contracting coordination challenges during the mission assignment process. Further, FEMA identified disaster contracting workforce shortages. FEMA, USACE, Coast Guard, and DLA each have processes for identifying lessons learned within their agencies through after-action reports. These reports identify lessons learned and areas for improvement and may be completed following a training exercise or a real-world event. Through these processes, agencies identified lessons learned during the 2017 disasters. Table 4 lays out each agency’s practice or requirement for identifying lessons learned and key findings— those related to contracting and mission assignments during the 2017 disasters. FEMA has also taken steps to identify interagency lessons learned by leading the Emergency Support Function Leadership Group and developing a mechanism to regularly report to the Secretary of Homeland Security. This group consists of the national emergency support function coordinators from each of the functions (such as transportation and firefighting), along with FEMA headquarters and regional officials. This body of senior officials is tasked with coordinating responsibilities and resolving operational and preparedness issues relating to interagency response activities in support of the National Response Framework. According to its charter, the group is required to carry out post-incident and after-exercise critiques, and perform substantive reviews of after- action reports, with recommendations for federal interagency partners to address shortfalls. Following the 2017 disasters, in May 2018, the Emergency Support Function Leadership Group identified 19 corrective actions, including improvements to mission assignment submission documents. Federal internal control standards state that communicating internally is key to an entity achieving its objectives. Further, as part of this communication, management should receive quality information about the entity’s operational processes that flows up the reporting lines from personnel to help management achieve the entity’s objectives. FEMA officials stated that there are processes, such as data calls, in place to solicit input from agencies. However, we noted, and FEMA officials agreed, that there is no formal reporting mechanism to the leadership group, and that it is up to the representatives from these agencies to raise issues for the group’s consideration. However, this is not consistently happening within the Coast Guard because it does not have a formal reporting process for soliciting input from officials directly involved in responding to these disasters to share with the Emergency Support Function Leadership Group. Coast Guard officials stated that they actively collect input during and immediately after an event or incident response, and that Coast Guard responders are able to provide input and issues through their chain of command at any time, but there is no formal process for reporting to the interagency group. During the course of our review, USACE officials did not provide information that indicated they had a formal reporting process for soliciting input from officials directly involved in responding to these disasters to share with the Emergency Support Function Leadership Group. Some senior level USACE officials responsible for the agency’s public works and engineering mission stated that they were unsure of the process for raising concerns to the Emergency Support Function Leadership Group and that officials were sometimes hesitant to raise issues to the group. However, in response to our draft report, USACE stated it has a formal process called the USACE Remedial Action Program for soliciting input from officials directly involved in the agency’s response and recovery following a disaster. As discussed later, we will follow up with USACE as part of our recommendation follow-up process. While Emergency Support Function Leadership Group member agencies may raise issues to the group, additional opportunities exist within these agencies to enhance the lines of communication from responders to the senior officials that comprise this leadership group. For example, some of the interagency challenges we identified in our review were not identified by this group, such as challenges in managing state and local expectations of federal response, which is discussed in more detail below. Also, USACE officials told us that some of the interagency challenges they cited following the 2017 disasters related to the mission assignment process were still present during the response to Hurricane Florence, which struck the Carolina coast in 2018. Formal processes for Emergency Support Function agencies—such as the Coast Guard and USACE—to solicit and share input from officials directly involved in the response and recovery efforts would help ensure the Emergency Support Function Leadership Group does not miss additional opportunities to improve disaster response. As the federal disaster coordinator, FEMA obtains requirements from states and localities and tasks the appropriate federal agencies, based on their emergency support function, through the mission assignment process. The agency assigned to a specific mission is then responsible for fulfilling those requirements, and may use contracts to do so. For example, the Coast Guard fulfills its pollution mitigation mission by executing contracts, and utilizes its own workforce to execute its search and rescue mission. USACE officials we spoke with raised concerns about the mission assignment process for the debris removal and power restoration missions related to the 2017 disasters. Specifically, USACE officials noted concerns about coordination between state, local, and federal partners for the contracts we reviewed. USACE debris removal mission: In December 2018, we found that USACE and California state officials reported different expectations related to USACE’s debris removal contracts following the wildfires, such as what structures would be removed from private properties and what levels of soil contamination would be acceptable. USACE removed more than 2.2 million tons of debris from more than 4,500 properties following the northern California wildfires. Due to the size and scope of this mission, USACE used both its advance contracts and additional post-disaster contracts for debris removal. According to USACE officials, they relied on FEMA, the lead for coordinating federal disaster response, to manage communication with states and localities and to identify and manage expectations about the scope of work to be performed using their debris removal contracts. USACE officials cited challenges with communicating to state and local officials what the agency was permitted to do under its mission assignment. For example, USACE officials told us that local officials believed that USACE would replace soil removed as part of its debris removal efforts; however, this was not part of the mission assignment from FEMA. Further, officials added that different environmental standards created confusion regarding what types of soil should be removed. For example, Napa County officials said that USACE’s mission required them to ensure that no contaminated soil remained on the properties, without regard for the naturally occurring levels of arsenic and asbestos in Napa area soil. As a result, Napa County officials said that USACE removed more soil than was necessary. However, following discussions with Napa County officials, USACE obtained site-specific samples from some properties to understand pre-existing contamination levels prior to further debris removal. USACE power restoration mission: Hurricane Maria destroyed much of the electricity grid in Puerto Rico, leaving millions without power and resulting in the longest blackout in U.S. history. To restore power to its 3.3 million people, Puerto Rico requested federal assistance with its power grid. To coordinate this effort across all stakeholders, FEMA established a unified command structure—which included the federal agencies, the Puerto Rican government and its contractors, and utility companies providing mutual assistance. According to FEMA officials, this structure allowed stakeholders to target priority work, ensure crews could access the work areas, and identify the needed materials. USACE officials stated that they received direction from FEMA and had limited direct interaction with Puerto Rican officials. However, despite this structure, USACE officials noted that changing direction from FEMA contributed to inefficiencies in contract management. For example, the scope of power restoration work Puerto Rico was requesting changed several times—such as from transmission work to distribution. These changes necessitated adjustments in contractor workforce configurations and contributed to idle time and equipment, according to officials. FEMA’s mission assignment policy designates a Federal Disaster Recovery Coordinator as the person responsible for facilitating disaster recovery coordination and collaboration among federal, state, local, tribal, and territorial governments; the private sector; and voluntary, faith-based, and community organizations. However, neither FEMA’s mission assignment policy nor its guide—which provides guidance on how to open and close mission assignments—provide additional details on how that coordination is to take place. Further, FEMA’s Response Directorate—the office that oversees the mission assignment process— was unable to identify at what level this coordination should occur. USACE and Coast Guard officials also noted that the mission assignment process does not account for other contracting considerations, such as demobilization, which occurs when contractor personnel leave the work site and return to their headquarters. According to USACE and Coast Guard officials, demobilization is required to be completed by the end of the contract’s period of performance; therefore, contracting officers need to know when the mission will end so that they can build adequate time for demobilization into the contract. Coast Guard pollution mitigation mission: Under this mission, the Coast Guard is responsible for responding to threats to public health, welfare, or the environment caused by actual or potential oil and hazardous materials incidents. Coast Guard officials told us that mission timing and the length of requirements were not communicated by FEMA in a timely manner. They told us that they contacted FEMA multiple times to determine if its mission assignment would be continued, but they did not receive an answer until shortly before the end of a contract’s period of performance. As a result, officials told us they were unsure whether they would need to demobilize contractors before completing the work, which created uncertainty about the availability of subcontractors. A FEMA Response Directorate official stated that these issues are coordination and planning concerns that should be worked out in advance between FEMA and the mission assigned agency. Ultimately, FEMA extended the Coast Guard’s mission assignment for pollution mitigation following Hurricane Maria four times. Figure 13 depicts the number of times Coast Guard’s mission was extended by FEMA. USACE power restoration mission: USACE officials cited similar challenges during the power restoration mission in Puerto Rico following Hurricane Maria. For example, USACE officials stated they typically begin planning for demobilization as soon as a mission begins. However, in this instance, officials did not know the eventual end date in order to plan for demobilization activities. Officials added that demobilization may take about 30 days, but USACE cannot extend contracts or obligate funds without a FEMA mission assignment extension. For example, if the mission assignment is scheduled to end on June 30, contracting officials would need to direct the contractor to begin demobilization as early as May 31. Officials stated that a mission assignment extension or option period of 30 days beyond the anticipated mission end date would facilitate demobilization and reduce any undue burden or concern around demobilization efforts. FEMA’s mission assignment guide does not provide a process or mechanism to follow up on the status of a mission once it is assigned. A FEMA official stated that the Response Directorate is responsible for informing their leadership of expiring mission assignments and contacting the mission-assigned agency to make them aware of the impending expiration, but that there is no standard time frame for doing so. Further, the official stated that, in some cases, FEMA may be performing this work a few days before a mission is set to expire. However, officials at USACE and Coast Guard told us they are dependent upon FEMA to reissue, clarify, or extend mission assignments. Further, the FEMA official told us that contracting considerations—such as the time needed for a contractor to mobilize and demobilize—are not necessarily built into the period of performance of a mission assignment. FEMA identified issues related to the mission assignment process, both during the 2017 disasters and following Hurricane Sandy in 2012. For example, in its 2013 Hurricane Sandy After-Action Report, FEMA found that the mission assignment process was not optimally set up to quickly surge resources to the field in a large-scale incident. To address these challenges, FEMA convened an Executive Steering Committee to update the mission assignment process, among other actions, and subsequently updated its mission assignment policy in 2015. Following the 2017 disasters, the Emergency Support Function Leadership Group identified challenges related to the mission assignment process and made recommendations to: (1) ensure response officials are properly trained on their department or agency’s statutory authorities and FEMA’s mission assignment process, and (2) develop specific recommendations to the FEMA Response Directorate on ways to reform mission assignment submission documents. These recommendations have been assigned to working groups within the Emergency Support Function Leadership Group, which plans to track the status until they are implemented. While these actions may improve the mission assignment process, they do not specifically address the issues we identified related to coordination and contracting. While the emergency support functions lay out agencies’ general responsibilities, agencies are dependent upon FEMA’s mission assignment process to further define how to perform their roles. Federal internal control standards state that management should implement control activities through its policies. These control activities include periodically reviewing policies, procedures, and related control activities for continued relevance and effectiveness in achieving the entity’s objectives or addressing related risks. Further, these standards also state that communicating internally and externally are key to achieving an entity’s objectives. As part of its internal controls, entities should evaluate the methods to communicate quality information throughout and outside of the entity on a timely basis. While FEMA revised its mission assignment guide in 2017, it still does not require FEMA to lay out coordination responsibilities in detail when assigning a mission. Without a mission assignment policy and related guidance that better incorporates contracting considerations, such as demobilization, and requires FEMA to clearly define coordination responsibilities with federal, state, and local stakeholders during the mission assignment process, federal agencies may encounter challenges fulfilling their assigned missions and may not fulfill their disaster response and recovery missions efficiently. During the 2017 disasters, FEMA leveraged contracting staff from its regions, headquarters, and the DART teams—FEMA’s deployable contracting workforce. However, FEMA’s after-action report and officials we spoke with cited workforce shortages as a continuing challenge for disaster response and recovery. For example, officials we spoke with in several regional offices stated that there are only one to three contracting officers per region. Further, information provided by FEMA OCPO shows that eight of FEMA’s 10 regional offices have only one permanent full- time contracting official. Some of FEMA’s regional offices have additional contracting staff through FEMA’s Cadre of On-Call Response/Recovery Employees, but this varies from region to region. Regional offices are responsible for managing post-disaster contracts, even if regional procurement staff were not involved in the initial award of those contracts, according to FEMA officials. As noted in table 4 above, FEMA’s after-action report recommended increasing contract support capacities; however, it did not provide a specific plan to do so. According to FEMA officials, the agency’s workforce needs have not been assessed since a FEMA workforce analysis pilot conducted in 2014. We have identified several key principles that strategic workforce planning should address, including: determining the critical skills and competencies that will be needed to achieve current and future programmatic results, and developing strategies that are tailored to address gaps in the number, deployment, and alignment of human capital approaches for enabling and sustaining the contributions of all critical skills and competencies. Further, in our review of FEMA’s 2014 analysis, we found that FEMA evaluated contracting workforce needs, but did not specifically consider contracting workforce needs in the regional offices or address DART employees. The analysis was based on 5 years of workload data and conducted at the task or activity level, such as performing market research prior to making a contract award. However, the analysis did not prioritize skills or mission needs, nor did it identify critical competencies. In September 2018, FEMA procurement officials told us that, based on the 2014 analysis, they planned to hire 57 additional contracting staff. Officials noted that FEMA’s general operation funding does not support these additional hires, thus the agency plans to hire these staff as Stafford Act employees for 2-year appointments using disaster funding. While this is an important step, it is unclear when these staff will be hired or how they will be allocated across FEMA OCPO. For example, as of July 2018, FEMA OCPO had 72 vacant positions, including key leadership positions and contracting specialists. Without assessing its current contracting workforce needs—including staffing levels, mission needs, and skill gaps—and developing a plan to address these gaps that includes time frames, FEMA will not know whether it has the appropriate number of contracting officials with the key skills needed to meet its mission and is not likely to be well-positioned to respond to future disasters. Contracting during a disaster can pose a unique set of challenges as officials face a significant amount of pressure to provide life-sustaining goods and services to survivors as quickly as possible. Given the scale and consecutive nature of the 2017 disasters, disaster contracts— particularly post-disaster contracts—played a key role in the response and recovery efforts. In these situations, it is important that the federal government be accountable for the contracting decisions it makes and the money it obligates, support the local economy and survivors as effectively as possible, and implement lessons learned before the next disaster strikes. Regarding accountability for the contracting decisions it makes and dollars obligated following disasters, without the ability to track disaster contracts using a NIA code in FPDS-NG, agencies, Congress, and the public lack full insight into post-disaster contracts. Providing clear criteria for establishing and closing the NIA code that accounts for the needs of users and consistently implementing these criteria will help ensure insight into high-visibility disaster events. Further, the ability to identify and track contracting dollars for disasters through a publicly available database, such as FPDS-NG, can reduce the burden on agencies to provide these data for interested parties, including Congress and other users, and offer a resource for historical data across major disasters. To help meet the needs of the local economy as effectively as possible, using a contracting preference for vendors in a disaster-affected area is an important component to early recovery efforts. Without guidance or training to ensure contracting officers are aware of the regulatory definition of the local area, agencies may miss opportunities to provide financial support to local vendors. Additionally, without clarifying how contracting officers determine whether offerors reside or primarily do business in a disaster area for the purposes of a local area set-aside, contract officials will remain uncertain on how to implement related FAR criteria. Similarly, guidance and tools to help ensure contracting officials are aware of the requirement to provide preference to the extent feasible and practicable to local vendors, including the need for written documentation on the use of non-local vendors for post-disaster contracts, will help ensure agencies comply with the requirement to do so. Taken together, these actions could enhance compliance with the Stafford Act provisions related to the award of contracts to local businesses in the disaster area, which could help jump-start the local economy. With regards to implementing lessons learned before the next disaster strikes, large scale disasters, like those that occurred in 2017, require effective coordination across emergency support function agencies. Given the Emergency Support Function Leadership Group’s responsibility to identify gaps or seams in the federal government’s efforts to respond to disasters, it is essential that the group have accurate and up-to-date information. Formal processes for soliciting and sharing information to communicate lessons learned to this group would help enhance agencies’ abilities to identify and address weaknesses in disaster response. Further, incorporating contracting considerations, such as demobilization, into the mission assignment policy, could enhance federal agencies’ ability to fulfill their disaster response and recovery missions efficiently. Lastly, without an assessment of FEMA’s contracting workforce needs, FEMA is at risk of not having a sufficient contracting workforce during a disaster. We are making a total of 10 recommendations, including one to DHS, one to the Office of Federal Procurement Policy, two to FEMA, three to the Army, two to the Coast Guard, and one to GSA (in coordination with DOD and DHS). The Administrator of the General Services Administration, in coordination with the Secretaries of Defense and Homeland Security, should jointly revisit and assess the extent to which the criteria in the 2018 NIA code Memorandum of Agreement, including criteria for closing NIA codes, meet long-term visibility needs for high visibility events and account for the needs of users, such as FEMA, other agencies, and the Congress. At a minimum, the agreement should include criteria that take into account the roles of the federal agencies involved in response and recovery and provide a process that ensures consistent consideration and implementation of the criteria. (Recommendation 1) Until the NIA code Memorandum of Agreement between the General Services Administration and the Departments of Defense and Homeland Security is revised, the Secretary of Homeland Security should, in coordination with the Department of Defense and the General Services Administration, keep the existing NIA code for Hurricane Maria open, reopen the other NIA codes established for 2017 and 2018 hurricanes (Hurricanes Harvey, Irma, Florence, and Michael), and request that agencies retroactively enter NIA codes for contract actions for Hurricanes Harvey and Irma made after June 30, 2018, for Hurricane Florence made after March 15, 2019, and for Hurricane Michael made after April 12, 2019 into FPDS-NG to adequately capture contract obligations, to the extent practicable. (Recommendation 2) The Secretary of the Army should direct the Commanding General of the U.S. Army Corps of Engineers to provide guidance or related training to ensure contracting officers are aware of the regulatory definition of “local area”. (Recommendation 3) The Administrator of the Office of Federal Procurement Policy should provide additional clarification on how contracting officers should determine whether offerors reside or primarily do business in a disaster area for the purposes of a local area set-aside contract. (Recommendation 4) The Commandant of the Coast Guard should provide guidance and tools for contracting officials to use to ensure requirements concerning contracting with local vendors, including justification requirements for the use of non-local vendors, are consistently met. (Recommendation 5) The Secretary of the Army should direct the Commanding General of the U.S. Army Corps of Engineers to provide guidance and tools for contracting officials to use to ensure requirements concerning contracting with local vendors, including justification requirements for the use of non- local vendors, are consistently met. (Recommendation 6) The Secretary of the Army should direct the Commanding General of the U.S. Army Corps of Engineers to establish a formal process to solicit input from officials directly involved in the agency’s response and recovery following a disaster and to share that input with the Emergency Support Function Leadership Group. (Recommendation 7) The Commandant of the Coast Guard should establish a formal process to solicit input from officials directly involved in the agency’s response and recovery following a disaster and to share that input with the Emergency Support Function Leadership Group. (Recommendation 8) The FEMA Administrator should take the lead to work together with the Coast Guard and the U.S. Army Corps of Engineers to revise the mission assignment policy and related guidance to better incorporate consideration of contracting needs, such as demobilization, and to ensure clear communication of coordination responsibilities related to contracting. (Recommendation 9) The FEMA Administrator should assess its workforce needs—including staffing levels, mission needs, and skill gaps—for contracting staff, to include regional offices and DART; and develop a plan, including timelines, to address any gaps. (Recommendation 10) We provided a draft of this report to DOD, DHS, GSA, and OMB for review and comment. In written comments provided by DOD, DHS, and GSA (reproduced in appendixes III, IV, and V), as well as an email response from OMB, the agencies concurred with nine of the 10 recommendations. They generally provided steps they plan to take to address these recommendations. As discussed further below, USACE described actions it stated were sufficient to fully address the seventh recommendation, the steps described by FEMA would not fully meet the intent of the tenth recommendation, and DHS did not concur with our second recommendation. In response to the seventh recommendation as written in our draft report—to establish a formal process to solicit input from officials directly involved in the agency’s response and recovery following a disaster and to share that input with the Emergency Support Function Leadership Group—in its comments, USACE concurred and stated it has a formal process and it considered the recommendation completed. USACE noted that its Remedial Action Program solicits input from officials involved in response and recovery efforts and added that USACE shares findings from this program with the Emergency Support Function Leadership Group throughout the year and annually during the senior leaders seminar. During the course of our review, USACE did not provide information that indicated that they had such a formal process. As part of our recommendation follow-up process, we will request documentation regarding the process and how it solicits and shares information to the Emergency Support Function Leadership Group. In response to the tenth recommendation that FEMA assess its workforce needs—including staffing levels, mission needs, and skill gaps—for contracting staff, to include regional offices and DART; and develop a plan, including timelines, to address any gaps, FEMA stated that its Office of the Chief Component Procurement Officer assesses its workforce on an annual basis, with the last assessment conducted in January 2019. FEMA also noted that it entered into a contract for acquisition support services and plans to hire Cadre of On-Call Response and Recovery employees to provide dedicated support during disasters. Following FEMA’s response, we requested and received the FEMA Office of the Chief Component Procurement Officer’s 2019 workforce assessment. As with FEMA’s 2014 workforce analysis, the 2019 assessment calculated the number of employees needed based on the estimated time to complete a task. However, the assessment did not include an analysis of mission needs or skill gaps, and the assessment provided does not specify whether it includes the needs of regional offices and DART. FEMA estimates that it will implement this recommendation in September 2019, and we will continue to monitor FEMA’s planned efforts through our recommendation follow-up process. DHS did not concur with the draft report’s second recommendation regarding NIA codes. In its response, with regards to extending existing NIA codes and reinstating expired NIA codes, DHS stated that it is bound by the memorandum of agreement with GSA and DOD, unless or until all three signatory agencies agree to revise or suspend the agreement. We recognize that all three agencies are bound by the agreement, and also recommended in the first recommendation that GSA, DOD, and DHS jointly revisit the agreement. GSA concurred with this recommendation in its written comments reproduced in Appendix V. In an email sent from an official within DOD’s Defense Pricing and Contracting Office, DOD concurred. DHS did not respond to our first recommendation. As such, we have revised the second recommendation to state that DHS take action in coordination with DOD and GSA. We also note that the memorandum of agreement states that extending expiring or already expired NIA code end date is appropriate, in part, when two or more agencies do not have a reasonable alternative method of identifying and internally tracking those emergency acquisitions. We discuss in our report how once the NIA code is closed, there is no publicly available, government-wide system to track contract obligations for specific events. We also discuss how, using the description field (which does not provide a full picture) in FPDS-NG, agencies obligated more than $250 million on contracts for Hurricanes Harvey and Irma during the three months after the NIA codes for these two hurricanes were closed. Given this, we continue to believe DHS should consider reopening the codes for Hurricanes Harvey and Irma, in coordination with DOD and GSA. Moreover, in its response to the second recommendation DHS further stated that FEMA’s Office of the Chief Component Procurement Officer (who is not currently a party to the memorandum of agreement), believes the recommendation to extend the NIA codes for 2018 Hurricanes Michael and Florence goes beyond the scope of this audit. While the main focus of this report is the 2017 hurricanes and California wildfires, we discuss Hurricanes Florence and Michael in this draft with respect to the NIA codes, as the same issues and concerns we raised apply regardless of the year of the hurricane. However, after we sent the draft to the agencies for comment, the agencies let the codes for Hurricanes Florence and Michael expire on March 15, 2019 and April 12, 2019, respectively. We therefore revised the second recommendation to recommend that the codes for Hurricanes Florence and Michael should be reopened (rather than kept open). In its written comments, DHS also stated that neither DHS nor FEMA can unilaterally direct other agencies to retroactively enter FPDS-NG data for Hurricanes Harvey and Irma. We acknowledge this and have revised the recommendation to recommend that DHS request, rather than direct, other agencies to retroactively enter the information, to the extent practicable. As we state in the report, the NIA codes for the 2005 hurricanes were established in October 2005, and contracting officers retroactively entered data for contracts related to these events to enable full insight into contracting for these disasters. DHS further stated that retroactively entering data into FPDS-NG is not practical and places an unreasonable burden on contracting staff, and that the draft did not support the case that there were any benefits to be gained. We recognize that there is some burden associated with the recommendation, thus we recommended that DHS request agencies take action to the extent practicable. In terms of benefits, the report identifies benefits in terms of providing decision makers with important information to understand the procurement impact of such disasters. DOD and DHS also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the U.S. Army Corps of Engineers Director of Contracting, the Director of the Defense Logistics Agency, the Secretary of Homeland Security, the Administrator of the Federal Emergency Management Agency, the Federal Emergency Management Agency’s Chief Procurement Officer, the Commandant of the Coast Guard, the Administrator of the General Services Administration, the Director of the Office of Management and Budget, and the Administrator of the Office of Federal Procurement Policy. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. This report specifically addresses the use of post disaster contracts and: (1) assesses the extent to which federal agencies obligated funds on post-disaster contracts in response to the 2017 major disasters; (2) assesses the extent to which selected agencies experienced challenges in the planning process for selected post-disaster contracts; and (3) describes selected agencies’ lessons learned as a result of the 2017 major disasters and assesses the extent to which they have taken action to address them. To identify the extent to which federal agencies obligated funds on post- disaster contracts in response to the 2017 disasters, we reviewed Federal Procurement Data System-Next Generation (FPDS-NG) data through June 30, 2018, the most recent and complete data at the time of our review. We adjusted the obligation data to constant fiscal year 2018 dollars using the Fiscal Year Gross Domestic Product price index. We identified hurricane obligations using the national interest action (NIA) code, as well as the contract description. Data on obligations for the California wildfires is limited to those contracts, if any, identified by the agencies with the highest obligations on post- disaster contracts for the hurricanes—the Federal Emergency Management Agency (FEMA), U.S. Army Corps of Engineers (USACE), Defense Logistics Agency (DLA), and the U.S. Coast Guard (Coast Guard)—because no NIA code was established in FPDS-NG. Coast Guard officials stated that they did not execute any contracts in response to the 2017 California wildfires. DLA officials stated that they maintain contracts, which for the most part provide inventory replenishment for DLA and the U.S. Forest Service within the U.S. Department of Agriculture, but they were unable to provide data on contracts awarded or executed specifically for the two wildfire disasters in the scope of our review. Therefore, our analysis only captures obligations for FEMA and USACE reported contracts related to the 2017 California wildfires. To determine which obligations were made through the use of post- disaster contracts versus advance contracts, we reviewed documentation provided by FEMA and USACE identifying the advance contracts they have in place and that were used in support of the 2017 disasters. We analyzed the FPDS-NG data against these contracts to identify obligations on post-disaster contracts and compared these to obligations on advance contracts by disaster. We analyzed competition procedures used and the types of goods and services procured for post-disaster contracts. In addition to advance contracts for disaster response, agencies can leverage other existing contract vehicles. For example, to respond to its pollution mitigation functions under emergency support function 10, the Coast Guard awards task orders off of its portfolio of basic ordering agreements. For the purposes of this report, post-disaster contracts include all contract awards and orders that were not identified by FEMA or USACE as advance contracts. To assess the extent to which disaster contract obligations can be tracked through FPDS-NG using the NIA code, we identified prior hurricane events with the highest contract obligations from 2005 through September 2018. We analyzed the data to determine when the highest level of federal contract obligations occurs following a hurricane. We also assessed the process for establishing and closing a NIA code. Specifically, we reviewed the criteria in the 2012 and 2018 memorandums of agreement between DHS, DOD, and the General Services Administration, and interviewed officials involved in the process. We assessed the reliability of FPDS-NG data by reviewing existing information about the FPDS-NG system and the data it collects— specifically, the data dictionary and data validation rules—and performing electronic testing. We also compared FPDS-NG data to the contract files in our review. Specifically, to review our selected post-disaster contracts for data reliability, we compared items such as, the extent competed, the use of a local area set-aside, NIA code, and termination status, based on the contract information and the information in FPDS-NG. Based on the steps we took, we determined the FPDS-NG data were sufficiently reliable for the purposes of describing agencies’ post- disaster contract obligations. To assess the extent to which agencies experienced challenges in the planning of selected post-disaster contracts, we reviewed relevant laws and regulations, including the Post-Katrina Emergency Management Reform Act (PKEMRA), the Federal Acquisition Regulation (FAR), the Robert T. Stafford Disaster Relief and Emergency Assistance Act (Stafford Act), as well as agency policy and guidance. We identified a non-generalizable sample of 23 post-disaster contracts from the four agencies with the highest post-disaster obligations based on FPDS-NG data as of March 31, 2018—DHS’s FEMA, DOD’s USACE, DOD’s DLA, and DHS’s Coast Guard. We selected contracts across the four major 2017 disasters included in our scope (Hurricanes Harvey, Irma, and Maria, as well as the California wildfires) based on four selection criteria—(1) contracts using the urgency exception to full and open competition; (2) contracts using a local area set-aside; (3) contracts awarded to small businesses; and (4) contracts terminated for cause or convenience. Our goal in this selection was to ensure we selected a range of contracts within each of these four criteria so as to assess the extent to which these contracts implemented certain laws and regulations. Specifically, we selected contracts based on the use of urgency and local area set-asides in order to assess agencies’ implementation of relevant PKEMRA, Stafford Act and FAR criteria for post-disaster contracts. Because the obligations for local area set-aside contracts was low across all federal agencies, about 5 percent of total post-disaster obligations, we selected contracts that were awarded to small business vendors as a proxy to identify other awards to local vendors. Finally, we selected terminated contracts to assess additional challenges related to post- disaster contracts, such as the availability of contracted services and supplies and the requirement setting process. Based on these criteria, we selected 12 FEMA, 7 USACE, 2 DLA, and 2 Coast Guard contracts. Findings based on information collected from the 23 contracts cannot be generalized to all post-disaster contracts. Additional details on our selected contracts can be found in table 5. To assess how agencies used the urgency exception to full and open competition, we reviewed selected contracts for the inclusion of a justification and approval for other than full and open competition including sole source justifications and exclusion of sources justifications. To assess the extent to which agencies provided preference to local vendors for post-disaster contracts, we reviewed selected contract files for the use of a set-aside or an evaluation preference listed in the contract solicitation, and the inclusion of justifications for contracts not awarded to local vendors. Additionally, we reviewed applicable agency guidance and interviewed contracting and senior procurement officials across all four agencies regarding their use of local area set-asides, including the means by which they define the geographic set-aside area and determine that an offeror primarily resides or does business in the set-aside area. We also met with officials from the Office of Management and Budget’s Office of Federal Procurement Policy to discuss relevant FAR criteria. To assess how FEMA program offices develop and deliver requirements packages for use by contracting officers and the extent to which those packages are sufficiently specific to allow contracting officers to issue a contract solicitation, we interviewed contracting, program, and senior procurement officials responsible for the contracts in our selection sample. We discussed the specificity of initial versus final requirements, the nature of requirements changes, the process of requirements development, and training provided to program officials regarding the requirements development process. We also reviewed new post-disaster awards at FEMA to determine time frames between resource request to award on average for post-disaster contracts. We compared these findings to relevant agency guidance on acquisition planning. To describe lessons learned selected agencies identified related to the use of post-disaster contracts and assess the extent to which agencies have taken action to address them, we reviewed available completed after-action reports from the 2017 and prior disasters, including the Hurricane Sandy FEMA After-Action Report, the 2017 Hurricane Season FEMA After-Action Report, USACE’s Temporary Emergency Power Mission After Action Review for Hurricane Matthew, USACE’s Puerto Rico After Action Review, USACE’s Northern California Wildfires Debris Removal Mission After Action Review, the Coast Guard’s 2017 Hurricane Season Strategic Lessons Learned After Action Report, and the Defense Logistics Agency’s 2017 Hurricane After Action Meeting papers. We also reviewed findings from the Emergency Support Function Leadership Group related to interagency lessons learned. As part of our review, we identified requirements for agencies to document or practices agencies use to document lessons learned following a disaster, agency specific and interagency lessons learned specific to post-disaster contracts and mission assignments, and recommendations or actions planned by the agencies to address them. We reviewed federal internal control standards and the Emergency Support Function Leadership Group charter and the standard operating procedures for its Preparedness Evaluation/Corrective Action Working Group. To describe challenges related to coordination with state and local officials on the use of post-disaster contracts, we interviewed FEMA, USACE, DLA, and Coast Guard officials. To obtain perspectives and examples from state and local government officials involved in disaster response, we interviewed officials in California on the use of federal contracts. We also met with state and local officials in Texas, Florida, Puerto Rico, and the U.S. Virgin Islands to discuss the federal response to the 2017 hurricanes more broadly. The information gathered from these officials is not generalizable to all officials. To describe challenges related to the mission assignment process, we interviewed FEMA, USACE, and Coast Guard officials, including officials from FEMA’s Response Directorate and the contracting officials from USACE and the Coast Guard that awarded the contracts these agencies used to fulfill their missions. We also reviewed the mission assignment documents, where FEMA assigned USACE and Coast Guard missions and laid out their responsibilities. To assess workforce challenges, we reviewed DHS’s 2014 workforce assessment, which identified gaps in FEMA’s contracting workforce. We also obtained information from FEMA on its current contracting workforce in headquarters, regional offices, Disaster Assistance Response Team, and joint field offices. We also interviewed FEMA contracting officials to obtain their perspectives and experiences during the 2017 disaster season. We conducted this performance audit from March 2018 to April 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The National Response Framework identifies 14 emergency support functions (ESF) and designates a federal department or agency as the coordinating agency for each function. ESFs are the federal government’s primary coordinating structure for response, and under this structure, the Federal Emergency Management Agency (FEMA) acts as the federal coordinating agency. In addition to the contact named above, Janet McKelvey (Assistant Director), Katherine Trimble (Assistant Director), Caryn E. Kuebler (Analyst in Charge), Lindsay Taylor, and Sarah Tempel were principal contributors. In addition, the following people made contributions to this report: Emily Bond, Lorraine Ettaro, Suellen Foth, Julia Kennon, Carol Petersen, Sylvia Schatz, Alyssa Weir, and Robin Wilson.", "summary": "Federal contracts play a key role in timely response and recovery efforts following disasters. While federal agencies, such as FEMA and USACE, may have advance contracts in place for obtaining goods and services following disasters, agencies may also award post-disaster contracts. GAO was asked to review the federal government's response to three major hurricanes in 2017, as well as the 2017 California wildfires. This report addresses, among other objectives, the extent to which (1) federal agencies obligated funds on post-disaster contracts in response to the these events, and (2) selected agencies experienced challenges in the planning of selected contracts. GAO analyzed data from the Federal Procurement Data System-Next Generation; selected a non-generalizable sample of 23 post-disaster contracts based on factors such as if the contract was set aside for award to a local contractor; reviewed federal regulations and agency guidance; and interviewed agency officials. Following hurricanes Harvey, Irma, and Maria and the 2017 California wildfires, federal agencies obligated at least $5 billion in post-disaster contracts—which are awarded after disasters hit— to support disaster response and recovery efforts. The U.S. Army Corps of Engineers (USACE) and the Federal Emergency Management Agency (FEMA) comprised over three-quarters of reported post-disaster contract obligations as of June 30, 2018 (see figure). However, the full extent of post-disaster contracting related to the 2017 disasters is unknown due to the Department of Homeland Security's (DHS) inconsistent implementation of the criteria for closing a national interest action (NIA) code. This code allows agencies to track data on contract actions related to national emergencies, providing government-wide insight into response and recovery efforts. DHS closed the codes for Harvey and Irma on June 30, 2018, less than a year after those hurricanes hit. In contrast, the codes for prior hurricanes were open for at least five years, with Katrina remaining open for 13 years. Based on a review of 23 contract files from FEMA, USACE, the Defense Logistics Agency, and the Coast Guard, GAO identified challenges in the planning of selected contracts. For example, GAO found USACE officials were not consistently aware of the regulation that defines “local area.” GAO also found that contracting officers at FEMA, USACE, and the Coast Guard did not consistently write justifications for awards to non-local vendors outside the disaster area, as required. FEMA developed guidance to address this, but the Coast Guard and USACE have not issued guidance or tools to address this requirement. Without addressing planning challenges, agencies may miss opportunities to award contracts to local businesses in the disaster area to the extent feasible and practicable, which could help jump-start the local economy. GAO is making 10 recommendations, including that DHS reopen NIA codes for Hurricanes Harvey and Irma; USACE provide guidance on the local area definition; and the Coast Guard and USACE provide guidance to ensure contracting requirements for the use of non-local vendors are met. Agencies concurred with 9 recommendations. DHS did not agree that NIA codes should be reopened. GAO continues to believe DHS should do so, to the extent practicable, as discussed in the report.", "document_type": "gao"}
{"report": "The Ethics in Government Act of 1978 was enacted to preserve and promote the accountability and integrity of public officials, and the institutions of the federal government. The act requires political appointees and high-ranking government officials to complete a public financial disclosure report to help prevent and mitigate conflicts of interest for the purpose of increasing public confidence in the integrity of government. The act also established restrictions on postemployment activities of certain employees, and created OGE. The primary mission of the executive branch ethics program is to prevent conflicts of interest on the part of executive branch employees. The executive branch ethics program is a shared responsibility across government (see figure 1). OGE is the supervising ethics office for the executive branch and sets policy for the entire executive branch ethics program. Executive branch agency heads are responsible for leading their agency’s ethics program. Agency leaders are ultimately responsible for their organizations’ ethical culture. Their actions can demonstrate the level of commitment to ethics and set a powerful example for their employees. Designated Agency Ethics Officials (DAEO) and other agency ethics staff carry out ethics program responsibilities and coordinate with OGE. Inspectors General and the Department of Justice are authorized to investigate potential violations of criminal statutes pertaining to ethics. Executive branch employees are individually responsible for understanding and complying with the requirements of ethics laws and regulations, and are collectively responsible for making ethical conduct a standard of government service. Executive branch employees are ultimately responsible for understanding and abiding by the various ethics laws. Generally, executive branch employees are prohibited from working on government matters that will affect their personal financial interest or the financial interests of a spouse or minor child; general partner; any organization in which they serve as an officer, director, or trustee; and any person or organization with whom they are negotiating or have an arrangement for future employment. Executive branch employees are also subject to criminal statutes prohibiting bribery and illegal gratuities; civil statutes requiring public financial disclosure; and employee standards of conduct, such as acting at all times in the public’s interest, serving as good stewards of public resources, and refraining from misusing their office for private gain. Agency Offices of Inspectors General (OIG) have a responsibility to investigate potential ethics violations. Among our three case study agencies, since January 2017, the HHS and Interior OIG have investigated potential travel and ethics issues involving political appointees while the SBA OIG did not initiate any similar investigations. The HHS OIG investigated the former Secretary of HHS’s use of chartered and commercial aircraft and found that it did not always comply with applicable federal travel regulations and HHS policies and procedures. In response to its OIG’s findings, HHS implemented additional steps for political appointees’ travel approval. Since January 2017, the Interior OIG has initiated five investigations into potential ethics violations involving the former Secretary of the Interior. As of March 1, 2019, three investigations related to the former Secretary were completed. As a result of the first completed investigation, the Interior OIG found that “incomplete information” about the former Secretary’s travel and use of chartered flights during 2017 was provided to the DAEO for review. The other two completed investigations found no evidence that the former Secretary violated ethics laws. Two investigations remained open as of March 2019. Interior’s DAEO described multiple strategies that were implemented to address issues observed within the ethics program after he was hired in April 2018, such as establishing weekly meetings with the former Secretary of the Interior to discuss ethics matters. Executive Branch political appointees are subject to more ethics restrictions than other executive branch employees. Appointees make or advocate policy for a presidential administration or support those positions. Appointees generally serve at the pleasure of the appointing authority and do not have the civil service protections afforded to other federal employees. There are four major categories of political appointees: Presidential Appointees with Senate confirmation (PAS); presidential appointees; noncareer employees in the Senior Executive Service (SES); and Schedule C employees. The most recent Plum Book, which was published on December 1, 2016, identified about 4,000 political appointee positions from these four major categories across the entire executive branch as of June 30, 2016 (see figure 2). The Plum Book identifies presidentially appointed positions within the federal government using data from the Office of Personnel Management. It is published every 4 years just after the presidential election, alternately, by the Senate Committee on Homeland Security and Governmental Affairs and the House Committee on Oversight and Government Reform. In addition to the ethics laws for executive branch employees, several recent presidential administrations have issued an order requiring political appointees in executive branch agencies to sign an ethics pledge. Some of the restrictions in the ethics pledge relate to areas already covered under existing ethics provisions, such as restrictions on accepting gifts and postemployment restrictions. Political appointees may receive an ethics pledge waiver from the President or his designee of certain or all ethics restrictions and authorizations enabling them to participate in otherwise prohibited activities. Political appointees that sign the pledge are contractually bound to adhere to its restrictions. If violated, the restrictions in the pledge could only be enforced through civil actions. To foster transparency, federal law permits members of the public to access various government records. OGE provides online access to certified copies of public financial disclosure reports for PAS and certain other executive branch employees, as well as any applicable ethics agreements, certification of compliance for the ethics agreement, and certificates of divestiture for PAS. OGE also provides online access to copies of ethics pledge waivers for appointees at agencies. Members of the public can use this information to assist in holding government officials accountable for carrying out their duties free from conflicts of interest. OPM, PPO, and two nongovernmental organizations provide some data on political appointees serving in the executive branch, but the data have limitations that impede their usefulness. The Senate Homeland Security and Governmental Affairs Committee and the House Oversight and Government Reform Committee publish OPM data on political appointees after each presidential election in the Plum Book. Data include name, title, type of appointment, salary, and location of employment. The data reflect the positions and the individuals who are filling the positions at a single point in time, about 5 months prior to the report’s publication. While the data are comprehensive and publicly available, they are not timely. Because the Plum Book is a snapshot in time, it does not reflect changes that occur in between publications, such as changes to who is holding a certain position, the position title, and vacancies. OPM also maintains more timely data on federal personnel; however, these data are not comprehensive or publicly accessible for identifying individuals serving in political appointee positions. OPM maintains data in the Executive and Schedule C System and the Enterprise Human Resources Integration (EHRI) system—the latter serves as OPM’s primary repository for human capital data. We found both systems have limitations, several of which were also identified by OPM officials. The Executive and Schedule C System is not comprehensive. It includes data on Schedule C and noncareer SES political appointees, but generally does not include data on presidential appointees or PAS. Publicly available EHRI data do not identify political appointees, either at the individual or group level. In addition, the EHRI source data is not publicly available. Political appointees can be identified from a combination of multiple variables, but these combinations are not consistent within or across appointee types. OPM provided some data on political appointees serving in the executive branch as of June 2018 from the Executive and Schedule C System. We reviewed the data and found errors and omissions. For example, we found instances in which individuals appeared to be holding political appointee positions that they departed several months prior and individuals known to currently hold political appointee positions were not identified. We also found that the data are incomplete. For example, the data did not include information on political appointee positions within the EOP. The EOP provides data to OPM only every 4 years for inclusion in the Plum Book. In addition to OPM, the White House maintains timely data on political appointees that are likely more comprehensive than OPM’s data but are not publicly available. Historically, PPO maintained data on political appointees as part of its responsibilities to recruit, vet, and place political appointees in positions across the government. PPO data on political appointees have not been made publicly available by the Trump, Obama, or Bush administrations. According to former officials from the Bush and Obama administrations, PPO maintained and used data on political appointees to carry out its responsibilities. For example, during the Obama administration, PPO established a database to help with filling political appointee positions and managing the overall appointee process. The database included preliminary information on candidates, such as names, application status, and where the applicant was in the vetting process. After a position was filled, the database tracked information such as the name of the appointee, position, federal department or agency, and start and departure dates. The primary limitation of the data was that departure dates of political appointees were unreliable. The former Obama administration official attributed this limitation to the lack of a process for agencies to formally notify PPO when an appointee left a position. To address this gap, PPO met regularly with staff in federal agencies to review data for accuracy. There are requests by members of the public to obtain data on political appointees serving in the executive branch. For example, between January 2017 and November 2018, OPM received approximately 32 requests through the Freedom of Information Act (FOIA) for data on political appointments across federal agencies. According to OPM officials, requests for data on political appointees are common and tend to increase at the start of a new administration. Former PPO officials also stated that when they served at PPO they received requests for data on political appointees serving in the executive branch. In the absence of comprehensive and timely data on political appointees serving in the executive branch, two nongovernmental organizations—the Partnership for Public Service and ProPublica—stated that they collect and report some data themselves. The Partnership for Public Service primarily tracks and reports data on PAS appointments, which are compiled from publicly available sources such as Congress.gov and agency websites. According to the Partnership for Public Service, accurately tracking departure dates is the most significant limitation. Some PAS departures, such as cabinet level officials, are typically reported in the media; however, lower-level PAS departures may not be reported. ProPublica collects and reports data on all types of political appointees serving in the executive branch. To obtain and compile its data, ProPublica makes FOIA requests to OPM and departments and agencies across the executive branch for political appointee staffing lists. ProPublica also makes requests for other data, such as financial disclosure forms through an administrative process required by the Ethics in Government Act of 1978. ProPublica said it has had more than 166,000 unique visitors to its database since it launched in March 2018. According to officials at ProPublica, one limitation is that they rely on agency responses to FOIA requests and therefore the data may not be comprehensive or timely. The public has an interest in knowing who is serving in the government and making policy decisions. The Office of Management and Budget (OMB) stated that transparency promotes accountability by providing the public with information about what the government is doing. In a 2009 memorandum, OMB directed agencies to make information available online and to use modern technology to disseminate useful information, rather than waiting for specific requests under FOIA. Although some data are publicly available on political appointees and FOIA requests can be used to varying effect to obtain data on political appointees, neither option results in comprehensive, timely, and publicly available data. Until the names of political appointees and their position, position type, agency or department name, start and end dates are publicly available at least quarterly, it will be difficult for the public to access comprehensive and reliable information. Making such information available would promote transparency. The public, including independent researchers, the media, and nongovernmental organizations, can use these data to perform independent analyses to identify gaps and challenges for filling political appointee positions or to identify potential conflicts of interest. Such analyses would also facilitate congressional oversight of executive branch appointees by providing a comprehensive and timely source of information on political appointees. As of March 2019, no agency in the federal government was required to publicly report comprehensive and timely data on political appointees serving in the executive branch. As the leader of human resources and personnel policy, OPM is positioned to collect, maintain, and make political appointee data publicly available on a frequent and recurring basis. However, OPM is limited in its ability to provide comprehensive data, in part because it does not regularly receive data from each agency that has political appointees, such as the EOP, which has approximately 225 political appointee positions based on the 2016 Plum Book. PPO is positioned to make more comprehensive data on political appointees publicly available. However, PPO is reestablished with each new presidential administration, which could be a barrier to establishing a consistent process for maintaining and publishing data on a recurring basis. Ultimately, it is a policy decision as to which agency is best positioned to report comprehensive and timely data on political appointees. All three agencies we reviewed—HHS, Interior, and SBA—generally used appropriate internal controls to ensure they met basic ethics program requirements, such as financial disclosure, though two of the agencies— Interior and SBA—could do more to strengthen their ethics programs. SBA and Interior had not fully documented some of their procedures for ethics training and the ethics pledge, respectively. In implementing their ethics programs, each agency addressed human capital issues and workforce continuity challenges; however, we found that vacancies and staff turnover had negative effects on Interior’s ethics program. For the full results of our assessment of agencies’ internal controls, see appendix II. All three agencies we reviewed met the minimum statutory and regulatory requirement to have written procedures for financial disclosure. Federal law requires agencies to develop written procedures to collect, review, and evaluate financial disclosure reports (see sidebar). Each agency established financial disclosure processes in addition to what is required to reduce the risk of political appointees performing agency work while they may have conflicts of interest. For example, prior to an HHS political appointee’s first day, the HHS process requires the appointee’s financial disclosure report to be submitted and reviewed, and any potential conflicts be either resolved or identified, and an ethics agreement put in place with a timeline for conflict of interest resolution. This process aims to ensure that appointees are in compliance with ethics laws and regulations when they begin government service, rather than 30 days or more into their appointment. File a new entrant public financial disclosure report within 30 days of assuming a public filing position. If appointed to a position requiring Senate confirmation, file a nominee report within 5 days of transmittal of the President’s nomination to the Senate for confirmation. File a termination report within 30 days of leaving office. HHS and SBA have additional processes that include written procedures which reflect OGE’s guidance for reviewing reports, such as following up with appointees when a financial disclosure report appears incomplete. OGE officials told us that engaging with an appointee during the review process allows agencies to confirm that the appointee understands and completes each required item. These interactions are also an opportunity to provide ethics counseling and establish a relationship with appointees who may be new to government service. Interior instituted a process in June 2018 that requires ethics officials to interview new appointees, review their financial disclosure report, and complete a financial disclosure checklist prior to certification. In reviewing a nongeneralizable sample of political appointees at each of the three agencies, we found that nearly all political appointees filed financial disclosure reports on time, with four exceptions of non-PAS appointees from our Interior and SBA samples (see table 1). In one case, an Interior appointee who was required to file both a new entrant and termination report did not do so. According to Interior ethics officials, the office mistakenly determined that the appointee was excluded from public filing requirements. An individual who does not serve more than 60 days in a calendar year is not required to file a new entrant or a termination financial disclosure report; however, this political appointee served for 63 days. Three appointees—two from SBA and one from Interior—filed new entrant reports past their due dates. Late filing heightens the risk of appointees performing agency work while having conflicts of interest; however, none of the three appointees filed more than 30 days after the due date or the last day of an extension, and therefore were not subject to a late filing fee. For example, one Interior appointee received a 30-day extension to file a new entrant report, but filed it 4 days late. One SBA appointee received an extension exceeding the maximum time—90 days—that an agency may grant to any filer and consequently filed 2 days late. According to SBA ethics officials, the appointee was given a 92-day extension because the due date was miscalculated. A second SBA appointee filed a report 1 day past the due date. We did not find timeliness issues with any reports filed by appointees at HHS or filed by PAS appointees at Interior or SBA. Agency ethics officials generally reviewed appointees’ financial disclosure reports in a timely manner. However, agencies followed up with non-PAS political appointees’ to varying degrees when their financial disclosure reports were potentially missing information. For example, SBA followed up with an appointee to confirm that the appointee had not inadvertently omitted information, such as a retirement plan, from the financial disclosure report because the appointee reported having previous long- term employment. HHS asked for and received clarifying information from an appointee who reported compensation for legal work but did not report individual clients. However, Interior ethics officials told us they did not follow up with two appointees in our sample who reported having no previous outside employment. Interior officials acknowledged that the reports were neither reviewed nor certified properly. According to Interior’s new Designated Agency Ethics Official (DAEO), the June 2018 update to Interior’s review process was implemented in response to deficiencies within its financial disclosure program. HHS and Interior had written procedures for initial ethics training as required, but SBA did not until February 2019. Federal regulation requires agencies to establish written procedures for providing initial ethics training beginning in January 2017 (see sidebar). Carry out an ethics education program to teach employees how to identify government ethics issues and obtain assistance in complying with ethics laws and regulations. Establish written procedures, which the DAEO must review each year, for providing initial ethics training. HHS’s and Interior’s written procedures reflect the requirements of initial ethics training. For example, both agencies’ procedures describe time frames for providing initial ethics training to political appointees no later than 3 months after their appointment date, as well as the method for doing so. Prior to February 2019, SBA did not have adequate written procedures in place to address the requirement that became effective in January 2017. SBA’s written procedures now reflect the requirements of initial ethics training. Now that SBA officials have formally documented procedures, they can have reasonable assurance that the procedures are implemented as intended and that all required appointees are provided initial ethics training. Interior’s and HHS’s ethics programs track and maintain documentation of dates that political appointees received initial ethics training. During the time of our review, SBA did not adequately document political appointees’ training dates. For example, ethics officials at Interior manually record training dates in a spreadsheet shared between Interior’s ethics office, Office of Human Resources, and the White House Liaison. HHS requires appointees to confirm in writing that they completed initial ethics training. According to SBA ethics officials, the previous Alternate DAEO informally documented the dates that political appointees received training in her personal notes. Standards for internal control state that management should document significant events, and that documentation and records should be properly managed, maintained, and readily available for examination. Allowing one individual to control all key aspects of documenting an event puts the program at risk of errors. As of February 2019, SBA officials had developed a tracking sheet and a certificate for appointees to sign that indicates they completed initial ethics training. We plan to assess the implementation of the tracking sheet to confirm that SBA is using the tracker to hold appointees accountable by documenting their completion of initial ethics training requirements. By developing and implementing a mechanism, such as a tracking sheet, SBA can have reasonable assurance that political appointees meet the requirement to take initial ethics training. Our review of agency documentation, including SBA’s informal documentation, found that political appointees completed required initial ethics training on time. Also, all three agencies provided the required additional live ethics briefing for PAS appointees together with initial ethics training. In addition to required training, all three agencies provided examples of other ways they have reminded appointees about their personal ethical responsibilities. For example: In advance of the holiday season, Interior provided supplementary training to political appointees on restrictions on accepting gifts. SBA used its agency-wide newsletter during the March Madness college basketball tournament to remind employees that they are prohibited from gambling in the workplace. HHS updated its ethics website to highlight Hatch Act rules in preparation for upcoming elections. Political appointees we reviewed at each agency had signed the required ethics pledge prescribed in Executive Order 13770, “Ethics Commitments by Executive Branch Appointees.” However, nine Interior appointees’ and one HHS appointee’s pledges were not timely signed. For example, the former Secretary of the Interior signed the pledge 19 days after his appointment. According to an Interior ethics official, the political appointees were directed to sign the pledge at the start of their appointments, but did not do so. Interior’s new DAEO told us in October 2018 that Interior now requires all appointees to sign the pledge on their first day as a condition of continuing their employment; however, this procedure has not been formally documented. The non-PAS HHS appointee signed the pledge 9 days after his permanent appointment date. While the restrictions under the pledge are enforceable by civil action, there are no legal consequences, such as fines or penalties, for failing to timely sign the pledge. for all appointees, a 2-year ban on involvement in “particular matters” involving former employers and clients; for former lobbyists, a 2-year ban on involvement on particular matters on which he or she lobbied; and for appointees who leave government service, a 5-year ban on lobbying agencies in which they served. The President or his designee may grant a waiver of any of the restrictions contained in the executive order. As of March 2019, 32 executive branch appointees—not including White House appointees— received limited waivers of the pledge. Interior’s then acting solicitor and principal deputy solicitor signed a limited waiver of certain restrictions on lobbying activities for one appointee in our sample upon the appointee’s departure from the agency in July 2017. However, according to Interior ethics officials, the official from the Solicitor’s Office did not have authority to grant a waiver. Furthermore, Interior’s ethics office was not included in the decision to grant the waiver, although Interior ethics officials ultimately notified the appointee when they became aware that the waiver was legally invalid. According to the DAEO, Interior is updating and documenting its ethics program processes and procedures, including new processes to sign ethics pledges and grant waivers, but did not provide a time frame for completion. We discuss Interior’s efforts to document overall ethics program processes and procedures later in this report. We found that all of the agencies we reviewed are addressing human capital issues and workforce continuity challenges to varying extents to achieve the goals and objectives of the ethics program. Standards for internal control state that management can help ensure operational success by having the right personnel for the job on board and maintaining a continuity of needed skills and abilities. Standards for internal control also state that management has a responsibility to obtain the workforce necessary to achieve organizational goals. HHS and Interior reported challenges to recruiting and retaining ethics staff with the necessary knowledge, skills, and abilities. All of the reviewed agencies reported varying levels of effort to address vacancies, skills gaps, and succession planning. HHS reported vacancies in its ethics program as well as challenges in recruiting and hiring; however ethics program officials took actions to mitigate negative effects of the vacancies. As of October 1, 2018, HHS’s Ethics Division had six vacancies out of 32 full-time positions (a vacancy rate of approximately 19 percent), including the Alternate DAEO position. HHS officials told us that a senior attorney was assigned to assume the duties of the Alternate DAEO position for six months in 2018. HHS ethics officials told us that the 2017 government-wide hiring freeze and workforce reduction plan affected their efforts to fill vacancies. However, ethics officials also told us that, as of October 1, 2018, four people had tentatively accepted offers to fill vacancies. HHS ethics officials told us that applicants for ethics attorneys and specialist positions generally do not have a background in federal government ethics laws. As a result, Ethics Division officials said that it must invest time and resources to train new hires, who attend and review OGE trainings, participate in monthly interagency ethics meetings, and take HHS-specific ethics training. HHS ethics officials told us that new ethics program hires are assigned work from across the spectrum of ethics subject matter and trained one- on-one by senior staff. To address staffing shortages and prepare for potential attrition, the HHS ethics officials said they cross-train staff members and assign back-up team members to support HHS’s operating and staff divisions. In addition, to track potential staff attrition or retirement, the ethics officials told us that the Ethics Division uses OPM’s Federal Employee Viewpoint Survey data collected from HHS employees. However, the data only give the Ethics Division a general sense of the number of personnel that are planning to leave or retire. HHS Ethics Division officials said they use survey data because there is a general sensitivity related to asking about retirement and delays in planned retirements that could affect recruiting and hiring replacements. Interior’s ethics office also reported vacancies and challenges in recruiting and hiring that contributed to the issues in the ethics program. As of November 2018, the Interior ethics office reported that out of 14 full-time positions, four were vacant (a 29 percent vacancy rate). All vacancies were ethics attorney positions. Interior reported an ongoing transformation of the department’s ethics program and officials said that the vacancies resulted from prioritizing the staffing at individual bureaus— such as the National Park Service and Fish and Wildlife Service—instead of the department-level ethics office, which is responsible for overseeing the bureaus’ ethics programs and providing ethics services to employees at the Office of the Secretary, the Office of the Solicitor, and to all of Interior’s political appointees. Interior’s ethics officials said that the high vacancy rate in their ethics office affected its ability to properly collect and review financial disclosure forms—one of the main responsibilities of the federal ethics program. According to Interior’s new DAEO, the office received an influx of financial disclosure reports during the presidential transition, but was unprepared to handle them. Furthermore, during 2017 one official was responsible for reviewing and certifying more than 300 public financial disclosure forms. The official was unable to balance proper and timely review of forms with other responsibilities that also included reviewing and certifying more than 800 confidential disclosure forms. In the Interior Inspector General’s 2018 report on Interior’s Major Management Challenges, ethics staffing was identified as a limitation, as staffing shortages could lead to delays in reviewing appointees’ financial disclosure documentation. While the single Interior official was experienced in reviewing financial disclosure forms, Interior officials stated that there was not enough management support, training, or resources provided to properly review financial disclosure forms in 2017. According to the DAEO, a new supervisory ethics official for financial disclosure forms was hired in September 2018 as part of a proposed and ongoing organizational restructuring of Interior’s ethics office. In addition, Interior posted a job announcement for a second ethics attorney and now has two ethics specialists for financial disclosures. The DAEO stated that the ethics program also plans to increase the number of ethics officials that review and certify financial disclosures, and has established new program goals, such as improving ethics staff competencies for technical review of financial disclosure reports. Interior ethics officials also reported that the government-wide hiring freeze affected their ability to hire staff and address ethics program staff continuity. To build capacity within the ethics program and create a strong ethical culture at the agency and bureau levels, the Acting Deputy Secretary recommended in May 2017 that Interior develop a structure and staffing plan to have a full-time ethics official for every 500 employees by fiscal year 2020. On October 26, 2018, Interior officials stated that the ethics program was implementing the Acting Deputy Secretary’s staffing plan. However, OGE benchmarking guidance states that there is no “right” ratio for the number of ethics staff per employee, and that agencies should determine their ratio based on certain aspects of individual ethics programs, such as the scope of potential conflicts and the complexity of financial disclosure reports. Interior officials could not explain how the ratio was determined nor provide a strategy for achieving the goal or evaluating whether the ratio is meeting the needs of the department in the future. We have previously identified leading practices for human capital management; these practices include that agencies should determine the workforce skills and competencies needed to achieve current and future goals and objectives as well as identify and develop strategies to address gaps. In addition, agencies should continually assess and improve human capital planning and investment, and assess the impact on accomplishing the mission. Without having a better understanding of resource needs and documenting how to properly allocate and determine needed resources, Interior may not accurately estimate its needs and may not be best positioned to assess and strengthen its ethics workforce to achieve program goals and objectives. Moreover, staff turnover at the Interior ethics office also reduced institutional knowledge. For example, Interior’s ethics office could not produce the documentation of the policies and procedures that support its ethics program—an internal control requirement—such as documenting and providing written responses to ethics queries and the tools used to ensure short and long-term continuity of operations. However, the ethics office previously provided documented evidence of some of these policies and procedures in its response to OGE’s 2016 program review. Interior ethics officials stated that the OGE response was produced prior to the DAEO retiring and drafted by staff who no longer work at Interior. Standards for Internal Control also require agencies to document key processes and procedures to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel, as well as a means to communicate that knowledge as needed to external parties, such as external auditors. Both HHS and SBA provided documentation of ethics program policies and procedures while Interior did not provide documentation. Since there was no formal documentation of the ethics program’s policies and procedures, Interior ethics officials stated that the ethics office will document them as part of its organizational restructuring plans. As of March 2019, Interior officials had not provided this documentation. For example, the ethics program is to ensure that all ethics related advice, legal analyses, and conclusions are documented. However, without Interior completing the documentation of its policies and procedures and making them accessible to staff, institutional knowledge may be lost, and there is greater risk of not achieving the goals and objectives of the ethics program. SBA did not report challenges to recruiting or staff continuity in part because of the small size of the ethics program. SBA’s ethics program is administered by three full-time officials and during our review, the DAEO position was vacant for more than 3 months due to the retirement of the previous DAEO. However, the Alternate DAEO assumed the responsibility for managing the ethics program until a new DAEO was hired in August 2018. Ethics officials reported that the program could draw upon a pool of field attorneys previously designated to perform collateral ethics duties to temporarily address disruptions in staffing. To address continuity and succession, SBA ethics officials reported that a headquarters staff attorney was detailed to the ethics program to prepare for the possible retirement of its current Alternate DAEO. Strong ethics programs are critical to ensuring public trust in government and the integrity of actions taken on the public’s behalf. The executive branch ethics program is a shared responsibility across government. Political appointees, in particular agency heads, have a personal responsibility to exercise leadership in ethics. Some data are available on political appointees serving in the Executive Branch but the data have limitations that impede their usefulness. To facilitate independent review and analysis related to political appointees, members of the public need access to information on who is serving in political appointee positions. Otherwise, they are limited in their ability to discern whether appointees are performing their duties free of conflict. Information on the political appointees serving in the executive branch at any point in time would also facilitate congressional oversight. Both OPM and PPO are positioned to report these data, but there are some benefits and drawbacks of each agency’s current capacity that will need to be considered. Ultimately, it is a policy decision as to which agency is best positioned to report comprehensive and timely data on political appointees. Further, a robust internal control system is critical for agency ethics programs to achieve their mission of preventing conflicts of interest on the part of their employees. Without effective internal controls, agency ethics programs cannot reasonably assure that they are mitigating the risk—or the appearance of—public servants making biased decisions when carrying out the governmental responsibilities entrusted to them. During the course of our review SBA took steps to establish written procedures for initial ethics training, but still needs to complete the implementation of procedures to track and verify that all political appointees meet ethics training requirements. As Interior continues to reorganize its ethics program, improved strategic workforce planning can help to accurately assess its needs, maintain continuity, and achieve program goals and objectives. Finally, ensuring that Interior’s ethics processes and procedures are fully documented and easily accessible to staff can help mitigate the risk of reduced institutional knowledge, and can improve the ability to communicate with external parties. Congress should consider legislation requiring comprehensive and timely information on political appointees serving in the executive branch to be collected and made publicly accessible. (Matter for Consideration 1) We are making a total of three recommendations, including one to SBA, and two to Interior. The Administrator of the Small Business Administration should implement procedures to track and verify that required employees complete initial ethics training and that completion of this training is documented. (Recommendation 1) The Secretary of the Interior should direct the Departmental Ethics Office, in conjunction with the Chief Human Capital Officer, to develop, document, and implement a strategic workforce planning process that aligns with its ongoing departmental reorganization and that is tailored to the specific needs of the ethics program. As part of this process, Interior should monitor and assess the critical skills and competencies that its ethics program needs presently and is projected to need in the future. (Recommendation 2) The Secretary of the Interior should ensure that the department’s ethics program policies and procedures are documented and easily accessible to program staff. (Recommendation 3) We provided a draft of this report for comment to the Department of Justice (DOJ), the White House Counsel’s Office at the Executive Office of the President (EOP), the Department of Health and Human Services (HHS), the Department of the Interior (Interior), the Inspector General of the Department of the Interior (OIG), the Office of Government Ethics (OGE), the Office of Personnel Management (OPM), and the Small Business Administration (SBA). Interior, SBA, and OGE provided written comments, which are reproduced in appendixes IV, V, and VI respectively. Interior officials concurred with our recommendations and described steps they are taking to begin addressing them. In our draft report, we made two recommendations to SBA. Our first recommendation was that SBA establish written procedures for initial ethics training as required. SBA officials did not agree or disagree with this recommendation, but during their review of the draft report, they provided documentation to show that they had established written procedures in line with our draft recommendation. As such, we revised our final report to include the actions taken by SBA in February 2019 and to delete our recommendation to establish written procedures for initial ethics training. With regard to our second draft recommendation to SBA, which remains in our final report as our first recommendation, SBA again did not agree or disagree with the recommendation. SBA officials provided documentation to support that they have taken initial steps to address our recommendation to implement procedures to track and verify completion of initial ethics training by political appointees. We plan to assess the implementation of these new procedures to confirm that, in operation, these procedures meet the intent of our recommendation. In addition to the written comments we received, SBA, HHS, OGE, and OPM provided technical comments, which we incorporated as appropriate. DOJ and the Interior OIG had no comments on the draft report. EOP did not respond to our request for comments. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 1 day from the report date. At that time, we will send copies to the appropriate congressional committees, the Acting Attorney General of DOJ, the White House Counsel, the Secretary of HHS, the Acting Secretary of the Interior, the Acting Inspector General at the Interior, the Director of OGE, the Acting Director of OPM, the SBA Administrator, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or nguyentt@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. Our objectives were to evaluate the extent to which (1) existing data identify political appointees serving in the executive branch at any point in time, and (2) selected agencies use appropriate internal controls to reasonably ensure that their ethics programs are designed and implemented to meet statutory and regulatory requirements. To evaluate the extent to which data identifying political appointees serving in the executive branch at any point in time exist, we first synthesized requirements for reporting and developed criteria for comprehensive and timely reporting. We reviewed relevant laws and standards, and the United States Government Policy and Supporting Positions (Plum Book). We used the Office of Management and Budget’s Open Government Directive (M-10-06) memorandum to develop criteria for transparency and public availability. We interviewed officials from the Office of Personnel Management (OPM) to understand the extent to which data they collect on current political appointees are comprehensive, timely, and reportable. OPM provided data on the political appointees serving in the federal government between January 2017 and June 2018. We also requested and obtained information from OPM on the volume of Freedom of Information Act requests for data on political appointees to assess demand for this type of data. To further evaluate public demand for political appointee data, we interviewed two nongovernmental organizations that track political appointees in the executive branch, ProPublica, and the Partnership for Public Service. We gathered information on the public’s demand for information regarding political appointees, and the use and limitations of data. Both organizations provided statistics quantifying public demand, including number of unique visitors to their website and media impressions. Media impressions are any viewing of or interaction with a piece of content. We requested information or interviews with the Office of Presidential Personnel (PPO) and several White House Liaisons to understand how they track, maintain, and use data on political appointees serving in the executive branch. A senior leader at PPO and one White House Liaison acknowledged our request for an interview but deferred to the White House Counsel’s Office. As well, an ethics officer indicated they would be unable to facilitate the exchange of information with the White House Liaison Office in their agency. The White House Counsel’s Office did not acknowledge requests for information or interviews. We interviewed former senior PPO officials from the two previous administrations to understand how they tracked, maintained and used data on political appointees. To identify internal control processes and determine the extent to which selected agencies use appropriate controls to ensure their ethics programs are designed and implemented to meet statutory and regulatory requirements, we first identified four case study agencies. We selected a range of case study agencies based on the number and type of political appointees as well as the strength of their ethics programs, as determined by Office of Government Ethics (OGE) reviews. Using data from the 2016 Plum Book, we identified the total number of political appointee positions within each agency or department across the following four categories: presidential appointees with Senate confirmation (PAS), presidential appointees, noncareer members of the Senior Executive Service, and Schedule C appointees. We selected the Executive Office of the President (EOP) as a case study agency because EOP has the largest number of presidential appointees, and because OGE has not recently conducted a program review of EOP. According to OGE, ethics program reviews are a primary means of conducting systematic oversight of executive branch ethics programs. OGE completed a review of each agency between January 2014 and January 2018. Since the White House Counsel’s Office did not acknowledge receipt of our notification letter we could not review EOP’s practices. To allow for more comparability among case studies, we excluded agencies and departments that did not have at least one PAS, and one presidential appointee or noncareer member of the Senior Executive Service. From the remaining list of departments and agencies, we excluded those with nine or fewer total political appointee positions. We divided the remaining agencies into two groups: large agencies with more than 100 political appointees and small agencies with fewer than 100 political appointees. To ensure we observed a range of practices, we selected a large agency with no recommendations in its most recent OGE program review—the Department of Health and Human Services and an agency with multiple unaddressed recommendations from its most recent OGE program review—the Department of the Interior. To select our final case study, we used human resources data from OPM’s FedScope tool to determine the number of employees at each agency as of September 2017. We limited our selection to noncabinet agencies with between 2,000 and 10,000 employees. Out of the four remaining agencies, we randomly selected the Small Business Administration. To evaluate the extent to which the three reviewed agencies have and use appropriate internal controls to reasonably ensure that the objectives of their ethics programs are achieved, we reviewed selected principles from Standards for Internal Control in the Federal Government based on our review, analysis and professional judgment as to which were relevant to effectively execute an executive branch ethics program. Selected internal control principles included: 3.01: Management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives; 4.01: Management should demonstrate a commitment to recruit, develop, and retain competent individuals; 10.01: Management should design control activities to achieve objectives and respond to risks; and 14.01: Management should internally communicate the necessary quality information to achieve the entity’s objectives. Reviewed agencies confirmed that these internal control principles were relevant to effectively execute their ethics program. We provided each agency with an identical set of questions based on the selected internal control principles and components. We used agency responses to questions and supporting documentation to evaluate whether agencies’ policies and processes to oversee ethics compliance for political appointees were consistent with the internal control principles. We used a nongeneralizable random sampling method to select political appointees whose documentation we would review for compliance with certain ethics requirements. Agencies provided data detailing the political appointees within the agency at any point in time beginning January 20, 2017 and as of January 28, 2018. To assess the reliability of the data, we asked each agency’s officials about how the data were obtained, where the data came from, and what steps, if any, they each took to assure the accuracy and completeness of the data. Officials at each agency knowledgeable about their data provided responses. Based on those responses, we determined that the data were sufficiently reliable to indicate each agency’s political appointees, with start and end dates, for use in selecting a sample of appointees at each agency. Within each agency, we used random sampling to identify up to three PAS appointees and up to nine non-PAS appointees, including up to three appointees that separated from the agency during the time frame above. Each case study agency completed a data collection instrument that identified the applicable ethics requirements for each selected appointee. Each agency provided documentation to communicate how those requirements were met for each appointee. We reviewed the documentation to determine whether agency internal controls were sufficient to ensure that certain ethics program requirements were met. In addition, we conducted interviews with agency ethics officials, as needed, to discuss documentation provided. We also conducted several interviews with OGE officials to inform how we developed the data collection instrument and evaluate appointee compliance in alignment with OGE’s principles and practices. Our review of political appointees’ documentation was limited to testing the sufficiency of the agencies’ ethics program processes and procedures. We did not review financial disclosure forms with the intent of identifying conflicts of interest nor did we perform a conflict of interest analysis. Also, because we used a nongeneralizable sample of political appointees, results from the sample cannot be used to make inferences about all the agencies’ political appointees. We conducted this performance audit from October 2017 to February 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Use of Internal Controls in Reviewed Agencies’ Ethics Programs Has the agency established an organizational structure for its ethics program? Management should demonstrate a commitment to recruit, develop, and retain competent individuals. Are agency ethics program staff evaluated? Are agency ethics program staff’s expectations developed and documented? Does the agency commit resources to the ethics program? Does the agency recruit, develop, and train ethics program staff? Does the agency prepare alternate or contingency plans for ethics program staff attrition, succession, or other potential disruptions to staff levels? Management should design control activities to achieve objectives and respond to risks. Does the agency have goals and objectives for the ethics program? Are these goals and objectives documented? Does the agency have processes and procedures in place to support the goals and objectives of the ethics program? Does the agency have processes and procedures in place to ensure political appointees who are not Presidential Appointees with Senate Confirmation do not undertake an activity that represents an actual or apparent conflict of interest? Does the agency have processes and procedures in place to ensure that political appointees receive required training? Management should internally communicate the necessary quality information to achieve the entity’s objectives. Does the agency communicate ethics program related information to political appointees? Signed the Executive Order 13770, “Ethics Pledge” Presidential Appointee with Senate confirmation (PAS) nominee financial disclosure report filed no later than 5 days after nomination by the President PAS nominee signed an Ethics Agreement to address identified conflicts of interest Non-PAS new entrant financial disclosure report filed within 30 days of assuming the duties of the position, or within extension of time for filing Received live ethics briefing within 15 days of appointment (PAS only) Termination financial disclosure report filed within 30 days of leaving government (if appointee departed from the agency) Because we used a nongeneralizable sample of political appointees, results from the sample cannot be used to make inferences about all of the agencies’ political appointees. In addition to the above contact, Melissa Wolf and Carol Henn (Assistant Directors), Erinn L. Sauer (Analyst-in-Charge), Caitlin Cusati, Ann Czapiewski, Robert Gebhart, Travis Hill, James Lager, Brittaini Maul, Steven Putansu, Mary Raneses, Andrew J. Stephens, and Mackenzie D. Verniero made major contributions to this report.", "summary": "Federal agencies' ethics programs seek to prevent conflicts of interest and safeguard the integrity of governmental decision-making. GAO was asked to review compliance with ethics requirements for political appointees in the executive branch. This report examines the extent to which (1) existing data identify political appointees serving in the executive branch, and (2) selected agencies use internal controls to reasonably ensure that their ethics programs are designed and implemented to meet statutory and regulatory requirements. GAO reviewed available data on political appointees. GAO also reviewed three case study agencies selected to provide a range in agency size and number of political appointees. GAO reviewed ethics documentation for a nongeneralizable sample of political appointees at the three agencies at any point between January 2017 and 2018 and interviewed officials from the agencies and two non-governmental organizations. There is no single source of data on political appointees serving in the executive branch that is publicly available, comprehensive, and timely. Political appointees make or advocate policy for a presidential administration or support those positions. The Office of Personnel Management (OPM) and two nongovernmental organizations collect, and in some cases, report data on political appointees, but the data are incomplete. For example, the data did not include information on political appointee positions within the Executive Office of the President. The White House Office of Presidential Personnel (PPO) maintains data but does not make them publicly available. The public has an interest in knowing the political appointees serving and this information would facilitate congressional oversight and hold leaders accountable. As of March 2019, no agency in the federal government is required to publicly report comprehensive and timely data on political appointees serving in the executive branch. OPM is positioned to maintain and make political appointee data publicly available on a timely basis but is limited in its ability to provide comprehensive data. PPO has more comprehensive data but may not be positioned to publish data on a recurring basis. Ultimately, it is a policy decision as to which agency is best positioned to report comprehensive and timely data on political appointees. All three agencies GAO reviewed generally used appropriate internal controls to ensure they met basic ethics program requirements, though two of the agencies could take actions to strengthen their ethics programs. The Departments of Health and Human Services (HHS), and the Interior (Interior), and the Small Business Administration (SBA) all have procedures for administering their financial disclosure systems. HHS and Interior had procedures for providing initial ethics training as required beginning in January 2017. Prior to February 2019 SBA did not have written procedures for initial ethics training and did not adequately document political appointees' training dates. SBA's written procedures now reflect the requirements of initial ethics training and SBA developed a tracking sheet to indicate appointees completed training. GAO will assess the implementation of the tracking sheet to confirm the process is sufficient for documenting appointees' completion of initial ethics training. Interior's ethics program has human capital and workforce continuity challenges. Interior reported that four out of 14 full-time positions were vacant. Interior officials attributed the vacancies to a recent transformation of the ethics program and prioritizing the staffing at individual bureaus such as the National Park Service. However, vacancies affected the ethics program's ability to properly document policies and procedures as well as file and review financial disclosure forms. According to Interior officials, steps are being taken to address vacancies and document policies and procedures. However, GAO found that a more strategic and documented approach would enable Interior to better manage human capital, fill key positions, and maintain institutional knowledge. Congress should consider legislation requiring the publication of political appointees serving in the executive branch. GAO also recommends three actions: SBA should document that training was completed; Interior should conduct more strategic planning for its ethics workforce and document ethics program policies and procedures. SBA neither agreed nor disagreed with GAO's recommendation, but provided documentation that partially addresses the recommendation. Interior agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "FAA issues aircraft registrations according to eligibility requirements prescribed by federal statute in support of International Civil Aviation Organization requirements that every aircraft engaged in international air navigation must bear its appropriate nationality and registration marks. Specifically, the law requires that the aircraft may not be registered under the laws of a foreign country and must be owned by (1) a citizen of the United States, (2) a foreign citizen lawfully admitted for permanent residence in the United States, (3) a noncitizen corporation that is organized and doing business under the laws of the United States or a state if the aircraft is based and primarily used in the United States, or (4) the U.S. government, District of Columbia government, or the government of a U.S. state, territory, or possession. By law and FAA policy, FAA imposes safety obligations on all owners of registered aircraft. To meet these obligations, an owner must maintain current information about the identity and whereabouts of the operators of an aircraft and location and nature of the aircraft’s operation on an ongoing basis. In doing so, the owner is to retain the ability to provide the operator with safety-critical information in a timely manner, and to obtain information responsive to FAA inquiries, including investigations of alleged violations of FAA regulations. Such information supports FAA’s ability to carry out its oversight obligations under U.S. and international law. FAA’s aircraft registry is an owner registry; it is not intended to include aircraft operator information. Only an aircraft’s owner may apply for registration, and a registration is not valid if the interest of the applicant in the aircraft was created by a transaction that was not entered into in good faith, but rather was made to avoid registration requirements. In addition, anyone who knowingly and willfully submits documents to FAA with false, misleading, or fraudulent information could be subject to criminal penalties and revocation of the aircraft registration. To register an aircraft for a 3-year period, in addition to a $5 application fee, applicants must submit to FAA at least two primary documents: (1) a completed application form and (2) a bill of sale or other evidence of aircraft ownership. A sample aircraft registration submission for an individual owner is shown in figure 1 below. For additional information about required documentation based on registration type, see appendix III. According to FAA officials, in 2018 FAA received approximately 71,000 registration applications. FAA also issues dealer certificates, also known as dealer licenses, in support of aviation commerce. Individuals and legal entities who are U.S. citizens can apply for an aircraft dealer certificate. The dealer certificate is valid for 1 year at a cost of $10 for the initial certificate and $2 for additional certificates. The certificates allow manufacturers and dealers to demonstrate and merchandize aircraft for prospective buyers and to make flight tests without a standard aircraft registration certificate. A dealer may obtain one or more certificates and may use a certificate for any aircraft the dealer owns. Dealer certificates require the applicant to be a U.S. citizen, identify an established place of business in the United States, provide a mailing and physical address, and substantially engage in manufacturing or selling of aircraft. Among other things, a dealer certificate is generally valid when the dealer, his or her agent or employee, or prospective buyer within the United States operate the aircraft, and only for flights that are required for testing of the aircraft or necessary for, or incident to, the sale of the aircraft. In 2018, there were 9,864 dealer certificates in the aircraft registry, primarily issued to corporations, limited liability companies (LLC), or individuals. FAA’s aircraft registration application form identifies eight registration types, including individual, corporation, and government. In 2018, there were 294,221 aircraft registered with FAA across all registration types (see fig. 2). The various registration types are associated with different types of aircraft ownership structures. Individuals who are U.S. citizens or resident aliens can register aircraft in the United States as individual owners or as part of a legal entity, such as a corporation or LLC. Legal entities that meet certain requirements can also register aircraft in the United States. For most types of legal entities, the entity must qualify as a U.S. citizen. For example, a corporation may own and register an aircraft as a U.S. citizen if (1) it is organized under the laws of the United States or a state, District of Columbia, or a territory or possession of the United States; (2) the president and at least two-thirds of the board of directors and other managing officers are citizens of the United States; (3) it is under the actual control of citizens of the United States; and (4) at least 75 percent of the voting interest is owned or controlled by persons that are citizens of the United States. Depending on the type of legal entity, additional requirements may apply, and in some cases additional documentation must be provided to FAA. For some legal entities, the registered owners of aircraft may not be the beneficial owners—the persons who ultimately own and control an aircraft. See appendix III for further information about the types of registrations and an additional ownership structure, along with associated documentation requirements beyond the aircraft registration application form, bill of sale, and $5 registration fee. If necessary, a corporation may use a voting trust to establish the fourth element of citizenship noted above for the purposes of registering an aircraft. Generally, a voting trust legally transfers the voting control in the corporation from a foreign citizen to a U.S. citizen who holds those interests in trust; however, the exact requirements are governed by the law of the state in which the trust is created. FAA regulations have included requirements around the use of voting trusts since 1980. When promulgating the relevant regulations, FAA explained that use of a voting trust allows a domestic corporation to come within legal compliance by placing the “voting interest of the stock of the corporate applicant . . . in the hands of U.S. citizens as voting trustees that the trustees have a valid, independent, and bona fide control of the voting interest.” As a result, if a voting trust is used by the domestic corporation to meet the fourth element of citizenship, the corporation must submit to FAA a copy of the voting trust agreement, which identifies the voting interests and must be binding upon all parties to the transaction, as well as an affidavit from each voting trustee, which represents that the voting trustee is an independent actor. A sample aircraft registration submission for a corporation using a voting trust is shown in figure 3 below. Trusts are not a registration type on the FAA aircraft registration application form; however, trusts are a legal structure that may own property such as an aircraft and therefore may be used to register an aircraft. As of June 2019, according to FAA data, there were 11,364 trusts in the aircraft registry. Depending on whether the trustee is an individual or an entity as well as on the specific terms of the trust, the aircraft’s owner in the FAA registry may be listed as an individual or as a corporation (see fig. 4). A trust may own and register an aircraft if each of the trustees is a U.S. citizen or resident alien, and 75 percent of the control of the trust must be vested in U.S. citizens or resident aliens. Specifically, each trustee must affirm that trust beneficiaries who are not U.S. citizens or resident aliens do not have more than 25 percent of the aggregate power to influence or limit the exercise of the trustee’s authority. However, foreign citizens who are not resident aliens may have more than 25 percent of the beneficial interest in the trust. Trusts for which foreign citizens have a majority of the beneficial interest are generally referred to as “noncitizen trusts,” even though legal title in the aircraft remains owned by one or more U.S. citizen or resident alien trustees. In a 1979 rulemaking, FAA cited “increased activities of foreign investors in aircraft financing” as a reason for updating its regulations related to noncitizen trusts. In the ensuing decades, FAA experienced problems obtaining important operational and maintenance information concerning aircraft owned by noncitizen trusts from the owner trustees, prompting FAA in 2011 to begin a review of its policies and practices regarding the registration of such aircraft. After a series of public meetings and receipt of written public comments, FAA issued a notice of policy clarification for noncitizen trusts in 2013. Among other things, the policy clarification confirmed that the “FAA does not consider the status of the trustee as the owner of the aircraft under a trust agreement as having any differing effect on its responsibilities for regulatory compliance issues compared to other owners of a U.S.-registered aircraft,” and that “FAA is not aware of any basis for treating one type of owner—such as a trustee under a noncitizen trust—differently from any other owner of a civil aircraft on the U.S. registry when considering issues of regulatory compliance.” FAA collects, stores, and makes publicly available aircraft registration information. FAA collects basic aircraft registration data from the application form, which are available and searchable on FAA’s website or in imaged records in portable document format (PDF). FAA data available on its website include aircraft registration number (tail or N- number), serial number, aircraft make and model, owner name, owner’s address, and registration status. According to FAA officials, FAA stores scanned images in two key systems: (1) aircraft records, which includes documents such as registration application forms and bills of sale, and (2) ancillary files, which includes documents such as trust agreements. FAA officials told us that aircraft record files are accessible to the LEAU, FAA LEAP, and aviation safety inspectors who access aircraft records files via a web-based portal. Ancillary files must be accessed on-site at the FAA Aeronautical Center in Oklahoma City, Oklahoma. The LEAU has direct access to the ancillary files and provides aircraft record and ancillary file information to law-enforcement agencies, FAA LEAP, and aviation safety inspectors. Additionally, all records are accessible to the public in FAA’s public documents room located at the FAA Aeronautical Center or upon request. Figure 5 shows collection, storage, and availability of FAA’s aircraft registration documentation. Within FAA’s Aviation Safety office, the Flight Standards Service manages the Civil Aviation Registry and is the primary user of aircraft registry information. Registry staff process registrations for U.S. civil aircraft, issue aircraft registration numbers, and record conveyances affecting interest in aircraft. Internal FAA users of registration information include officials from ASH, LEAP, and SEIT, and aviation safety inspectors. FAA LEAP and SEIT coordinate closely with registry officials to request registration information in support of their missions on security and law-enforcement assistance. Apart from FAA, major users of aircraft registry information are organizations serving the aviation industry, international civil aviation agencies, federal safety officials, and law- enforcement agencies (see table 1). In 1964, FAA issued updated aircraft registration regulations and set the aircraft registration fee at $5. In 1988, Congress passed the Federal Aviation Administration Drug Enforcement Assistance Act of 1988 (FAA DEA Act), which declared that it is FAA policy to assist law-enforcement agencies in the enforcement of laws relating to the regulation of controlled substances and, among other things, required FAA to promulgate regulations that would require individuals to provide their driver’s license number and entities to provide a tax identification number in their registration application. In 1990, FAA issued a proposed rulemaking that, among other things, required a driver’s license number for an individual and a tax identification number for others. In 2005, FAA issued a notice of proposed rulemaking withdrawal, stating that it fulfilled the requirements of the FAA DEA Act, with certain exceptions, through changes to its system and procedures used by the FAA Civil Aviation Registry, such as by providing law-enforcement agencies access to the registry data. With regard to the requirement to provide a driver’s license number or tax identification number, FAA determined that the requirement would be detrimental to users of aircraft records and potentially to the aircraft owners, and cause an unnecessary burden on aircraft owners and government, and that this information was not necessary for law-enforcement agencies to carry out their responsibilities. In 2010, to improve the quality of registry data and to provide more accurate information to law-enforcement agencies and other users, FAA started requiring aircraft registration renewal. Such renewals must occur every 3 years. In 2018, the FAA Reauthorization Act of 2018 required FAA to modernize the Civil Aviation Registry’s information technology (IT) systems. The act also required FAA to initiate a rulemaking to extend the registration duration for noncommercial general aviation aircraft from 3 to 7 years. Beneficial ownership and legal information can assist law-enforcement and safety authorities by identifying those natural persons who may be responsible for the underlying activity of concern, or who may have relevant information to further an investigation. The Financial Action Task Force (FATF)—an international standards-setting body for combating money laundering, financing of terrorism, and other related threats to the integrity of the international financial system—has examined how legal and beneficial ownership information can assist law- enforcement and other competent authorities. FATF was established by the group of seven economic summit partners, known as the G7, of which the United States is a member, and the Treasury’s Office of Terrorist Financing and Financial Crimes leads the U.S. delegation to FATF. FATF developed a series of 40 recommendations, last updated in 2019, that are recognized as the international standard for combating of money laundering and the financing of terrorism and proliferation of weapons of mass destruction. Specifically, FATF Recommendations 24 and 25 call on member countries to ensure the availability of adequate, accurate, and timely information on the beneficial ownership of corporate vehicles that can be accessed by competent authorities in a timely fashion. To the extent that such information is made available, it may help financial institutions and other organizations to implement the due-diligence requirements on corporate vehicles including to identify the beneficial owner and to identify and manage financial crimes risks, including sanctions requirements. Internal controls help entities fulfill their mission and objectives while safeguarding assets and ensuring proper stewardship of public resources. According to federal internal control standards, managers are responsible for an effective internal control system, which increases the likelihood that an entity will achieve its objectives. Additionally, managers are responsible for proactively managing risks, including fraud risks and misconduct such as waste and abuse, to facilitate the entity’s mission and strategic goals by ensuring that taxpayer dollars and government services are being used for their intended purposes. The Fraud Reduction and Data Analytics Act of 2015, enacted in June 2016, required federal agencies to establish financial and administrative controls for managing fraud risks. These requirements are aligned with leading practices outlined in A Framework for Managing Fraud Risks in Federal Programs (Fraud Risk Framework). GAO’s Fraud Risk Framework outlines leading practices to prevent, detect, and respond to fraud risks. As depicted by the larger circle for prevention in the sidebar, preventive activities generally offer the most cost-efficient use of resources, since they enable managers to avoid costly and inefficient recovery activities following fraudulent transactions. Therefore, leading practices for strategically managing fraud risks emphasize risk-based preventive activities. FAA reviews registry applicant information for completeness and compliance with regulations—generally accepting self-certification of eligibility and aircraft ownership—but does not verify this information or collect key information on applicants and aircraft owners, according to our review of the registry process. This limits FAA’s ability to prevent fraud and abuse in aircraft registrations, which has enabled aircraft-related criminal, national security, or safety risks, according to our case-study review. Specifically, FAA’s review of aircraft registrations and dealer certifications primarily focuses on ensuring that applicants provide required documents and that forms are complete. Additionally, FAA requires limited personally identifiable information (PII), and it generally does not use that information to verify applicant information. The registry is further vulnerable to fraud and abuse when applicants register aircraft using opaque ownership structures that limit transparency into beneficial owners of aircraft. FAA’s approach has focused on obtaining and recording the required documents, and consequently, FAA has not identified fraud risks, their likelihood and impact, the suitability of controls, and other aspects of a fraud risk assessment that would support fraud prevention activities. As a result, FAA is limited in its ability to ensure registrant eligibility and prevent fraud and abuse and associated criminal, national security, and safety risks involving U.S.-registered aircraft. FAA generally accepts certification by applicants of their eligibility and aircraft ownership and performs limited review of applicant information to identify potential fraud or abuse. Specifically, FAA requires applicants to submit signed documents that attest to the requirements relevant to their registration type, including U.S. citizenship, resident alien status, or eligibility as a noncitizen corporation. Where owners are LLCs or trusts, applicants submit documentation that the entity is organized under U.S. or state laws. Additionally, applicants must submit evidence of aircraft ownership, such as a bill of sale, and attest to their ownership of the aircraft. According to FAA policy, by signing the application form, applicants certify to the truthfulness and accuracy of the information provided and that they understand that knowingly and willfully submitting documents to FAA with false, misleading, or fraudulent information could subject the person to criminal penalties and revocation of the aircraft registration. FAA collects applicants’ name and address, although according to officials, it accepts this information as factually valid and does not make an attempt to detect intentional fraud at the time of application. FAA does not require or collect other PII, such as the applicant’s date of birth or driver’s license information for individual applicants, or taxpayer identification numbers and state of incorporation for legal and corporate entities, for identity verification or record keeping. FAA collects some PII in the airmen registry, such as for pilot licensing, but it does not use this information for aircraft registration verification purposes. Use of PII is a key way federal programs verify the identity and eligibility of potential beneficiaries. FAA’s policy is to review documents for acceptability during the initial registration. This includes, for example, checking for internal discrepancies within the documents submitted, ensuring that documents are complete, and that the self-certification is signed. For previously registered aircraft, FAA also reviews prior bill of sale documents for inconsistencies in the chain of ownership. Where owners are corporations with complex ownership structures, such as LLCs that are owned by other LLCs, registry officials may request review by FAA’s legal counsel to confirm eligibility. FAA’s legal counsel may also review documentation provided by noncitizen corporations as well as trust agreements and related documents for registrations involving noncitizen trusts, statutory trusts, and corporations using voting trusts to meet U.S. citizenship requirements at the time of registration. In these cases, according to FAA officials, FAA legal counsel reviews documentation to ensure that the entity is organized under U.S. or state laws and may periodically perform spot checks by contacting a Secretary of State office to confirm the existence of an entity. However, where the owner is a U.S.-citizen corporation, FAA generally does not request or review articles or certificates of incorporation to ensure the entity is organized under U.S. or state laws. In addition, FAA does not require or review additional documentation for individual, partnership, and government registration types. For these applicants, FAA checks (1) all sections of the application form for completeness, (2) chain of ownership, and (3) that applicants self-certify their U.S. citizenship. Further, according to FAA officials, when FAA informs applicants of its unfavorable determination, such as when reviewing LLC documentation, for example, applicants are generally provided an opportunity to remedy deficiencies and resubmit their applications. According to FAA officials, FAA applies the same scrutiny to resubmissions as it does to initial applications. In addition, FAA does not review documents for eligibility when individuals certify that there have not been any changes since initial registration. As with aircraft registrations, FAA does not verify dealer identity, check for prior relevant violations, or enforce requirements associated with dealer certificates, such as verifying that dealers are substantially engaged in manufacturing or selling aircraft or only operating domestically, except when delivering an aircraft to a foreign purchaser. Furthermore, FAA regulations do not prescribe enforcement mechanisms to ensure continued dealer eligibility once approved or at the time of certificate renewal. Law-enforcement and FAA LEAP agents told us that dealer certificates is an area in need of greater oversight because dealer certificate applications have been falsified similar to other aircraft registrations, as discussed below. Additionally, FAA LEAP agents told us that they have identified instances of dealers acting as nominees on behalf of foreign entities, registering aircraft under their U.S. dealer certificate. The use of a nominee is an invalid means to register an aircraft, including for dealers. FAA LEAP agents noted that, in their experience, this practice may have enabled otherwise ineligible foreign entities to meet aircraft registration citizenship requirements. In our case studies and interviews with FAA, we identified examples of fraudulent registrations and potential abuse of the registry that occurred within the context of FAA’s current practice of limited verification and review of applicant information. In addition, our analysis of address data and investigation of selected addresses highlights the risks of abuse arising from FAA’s approach of not verifying address information. The examples below illustrate some of the risks associated with FAA not verifying: (1) applicant identity, (2) ownership, and (3) address information. Applicants falsified identities and registration self-certification. A 2017 case involving an aircraft registered through a falsified identity illustrates inherent risks of not verifying applicants’ information and identities, such as through PII or other checks. According to FAA documents, an applicant registered an aircraft as an LLC owner with supporting documents identifying two individual members. In registration documents, the applicant provided the name of a stolen identity for the first LLC member’s name and “John Doe” for the second. FAA accepted the registration information as factually valid and the aircraft remained legitimately registered for about 1 year. A DEA and FAA LEAP investigation of aircraft operating outside the United States eventually discovered the falsification. When FAA LEAP agents contacted the first named individual of the LLC, he affirmed that he was not a member of the LLC, never owned an aircraft, and never executed any documents to register an aircraft in his individual capacity or on behalf of a business entity. FAA LEAP determined that the stolen identity had been used to submit aircraft registration paperwork without the individual’s knowledge or consent. Accordingly, FAA revoked the aircraft registration, finding that the registration was invalid because the applicant’s interest in the aircraft was created by a transaction that was not entered into in good faith. This revocation was associated with a broader effort by DEA and FAA involving international operations of multiple U.S.-registered aircraft that resulted in aircraft and cocaine seizures, discussed later in this report. Aircraft broker fraudulently registered multiple aircraft for bank loan fraud scheme. A 2013 case involving an aircraft sales broker and dealer who was convicted of making a false statement to FAA in registering aircraft, among other convictions, illustrates risks associated with FAA’s reliance on self-certification and limited review of ownership information. In this case, the broker submitted fraudulent registration applications and bills of sale to FAA using forged signatures for over 20 aircraft as part of a multi-million-dollar bank fraud scheme. FAA accepted the broker’s self- certification as factually valid. The broker used the registration certificates that FAA had provided as an asset to support a loan application that resulted in a $3 million bank loan for his failing aircraft sales business. The bank uncovered the fraud over a year after the sales broker first submitted to the bank fraudulent aircraft registration documents to execute the bank loan. A subsequent investigation by the Federal Bureau of Investigation revealed the extent of the fraud, namely that the main thrust of the fraud scheme was to pledge 22 aircraft as collateral, which neither the broker nor his company owned, in order to obtain money from the bank. As a result of the fraud, some of the rightful owners of the aircraft experienced difficulty reinstating aircraft registrations in their names. For example, one owner told federal investigators that he could not fly his aircraft for 2 years because the registration of his aircraft was in the name of the fraudulent broker. This aircraft broker was also a licensed dealer, who held and renewed a dealer certificate during the time he was perpetrating his illicit scheme submitting fraudulent aircraft registrations to FAA. Noncompliant addresses. We also identified registrations with potentially noncompliant addresses and addresses that did not match USPS postal verification data in our analysis of FAA’s publicly available and ancillary registry data files. Our analysis illustrates noncompliance risks associated with FAA’s approach of not verifying physical address information as well as safety and security risks associated with FAA’s ability to readily identify or contact owners when issues arise. FAA regulations require that owners submit physical address information in their application forms. According to FAA policy, a physical address is needed so that the owner can be located, if necessary, for security or safety reasons. According to FAA officials, FAA will accept the use of a mail drop or a registered agent’s address as a mailing address, provided the physical address is included. However, our analysis of 2018 physical and mailing address data shows that over 2,000 (about 1 percent) of addresses list a mail drop location without a physical address, which does not comply with FAA’s requirement. We selected seven of these cases for further verification using online and subscription database research, including three for site inspection. In our review of seven selected cases based on categories of addresses and locality, we identified three cases in which a physical address was not provided by registrants. Through a site inspection for one of the selected cases, we were able to confirm a UPS Store location was provided as the mailing address, and no physical address was provided as required by FAA policy. (See sidebar.) For the remaining two cases, the registrants provided the addresses of the registered agents that likely facilitated the application on behalf of the registrants, but no physical addresses were provided. The address of one of these registered agents is the same address we identified in a case study discussed later in this report. In that case, FAA registry officials were not able to get in contact with the owner, who used a registered agent address, after the aircraft had crashed outside the United States. The aircraft was being operated by a foreign government, following its seizure on drug trafficking charges. FAA sent multiple letters to the owner to deregister the aircraft and also when the aircraft registration was expiring, but all were returned as refused by the registered agent. As discussed later, the use of a registered agent address may provide a layer of anonymity in aircraft ownership and pose challenges when FAA or law-enforcement agencies need to contact registered owners. The address of this mail drop location was used in one of the aircraft registration cases we selected for postal address verification, inconsistent with Federal Aviation Administration policy. Additionally, we selected five dealer addresses for further review. We found that in three cases physical addresses were provided on the certificate application forms as required. In two remaining cases, we cannot make any conclusions regarding the validity of the physical addresses provided because we could not confirm through our online and subscription databases whether the companies were or were not located at the physical addresses provided to the registry. In addition to fraud and abuse risks posed by limited verification and review of applicant information, the registry faces risks associated with nominee registrations. As noted above, use of a nominee is an invalid means to register an aircraft and involves a person or business acting on behalf of an ineligible owner, as shown in the following example. Fraudulently registered aircraft linked to notorious cartel. A 2016 case involving the use of a nominee to register an aircraft on behalf of an ineligible owner illustrates risks of registration fraud by individuals and entities misrepresenting their aircraft ownership. In this case, law- enforcement officials received information that an aircraft was in the process of being purchased by a foreign national. A U.S. corporation, acting on behalf of entities known to have ties to the Sinaloa Cartel, purchased the aircraft, filed registration documents for it, and represented itself as the aircraft owner. According to court documents, by registering as the aircraft owner, the nominee corporation concealed the otherwise ineligible non–U.S. citizen ownership of the aircraft by entities with Mexican drug cartel ties. FAA accepted the registration and registered the aircraft in 2014. A law-enforcement agency, which was aware of the scheme, seized the aircraft shortly after final payment was made on it. Law-enforcement investigation into this case also revealed that some of the same entities had previously been involved in similar schemes involving aircraft purchases and registration associated with drug trafficking. The aircraft was subsequently forfeited to the federal government because its registration was fraudulent and it was purchased with assets derived from wire fraud, money laundering, or other unlawful activities. As part of its IT modernization effort, FAA identified some risks to the aircraft registry, such as financial fraud and terrorist access. FAA officials have also pointed to various FAA LEAP and law-enforcement activities directed at managing these risks, as discussed later in this report. These are reactive measures, and the current process—which accepts applicant information at face value—is not designed to identify and prevent fraud and abuse. Preventive activities generally offer the most cost-efficient use of resources because they enable managers to avoid a costly and inefficient “pay-and-chase” approach. According to federal internal control standards, managers should identify, analyze, and respond to risks. Furthermore, GAO’s Fraud Risk Framework emphasizes risk-based preventive activities that are based on a comprehensive, documented risk assessment that identifies risks, assesses them, and develops a strategy to address analyzed risks, including periodic assessments to evaluate continuing effectiveness of the risk response. To identify risks, managers should consider the types of risks, including both inherent and residual risks. To assess risks, managers should estimate the significance of a risk by considering the magnitude of impact, likelihood of occurrence, nature, and tolerance of the risk. Managers should then design overall risk responses for the analyzed risks based on the significance of the risk and defined risk tolerance. According to FAA officials, FAA has not conducted such an assessment, which would better position it to design and implement risk-based preventive and other controls to manage these risks. As our case studies and illustrative examples demonstrate, this has enabled illicit actors to defraud and abuse the registry, with criminal and national security consequences. In addition, federal internal control standards call for agency management to design control activities to achieve objectives and respond to risks, including designing a variety of transaction controls, which may include verifications, reconciliations, and authorizations. As discussed in the Fraud Risk Framework, a leading practice to effectively prevent instances of potential fraud is for managers to take steps to verify reported information, particularly self-reported data and other key data necessary to determine eligibility. According to FAA officials, the law directs FAA to register an aircraft or issue a dealer certificate that meets eligibility requirements, but does not require FAA to verify the accuracy of the information included in the registration application. Yet without such a review to verify applicants’ information, FAA cannot be assured it is appropriately determining eligibility for the approximately 71,000 applications the registry processes annually. In turn, this limits FAA’s ability to prevent fraud and abuse of the registry from registrants engaged in illicit activities. Aircraft Registration and Dealer Fees Aircraft registration costs $5 and a dealer certificate costs $10 for initial application and $2 for additional certificates. While these fees are attractive to aircraft owners and dealers for economic reasons, we previously determined that the registration fee, in place since 1964, did not cover the cost of reviewing and processing an application. Considering only inflation adjustment, the $5 fee would be $41 in 2019 dollars, which may still be short of what the Federal Aviation Administration (FAA) would need to cover its expenses. FAA has been working to increase registration-related fees since 2013. According to FAA officials, FAA is evaluating regulatory strategy in light of registry information technology modernization and considering other regulatory priorities. According to FAA officials, although they have the authority to collect information for verification purposes, they do not have the tools and resources to do so. With respect to tools, as noted earlier, FAA is making plans to modernize registry operations by implementing streamlined and automated processes where registration information is submitted electronically. According to FAA officials, this is expected to improve online data availability and allow for cross-checking information with other data sources, such as other government databases. With respect to resources, FAA collects a fee that is intended to cover registration processing activities. However, the registration fee has remained the same—$5—since 1964, and for many years has not covered FAA costs associated with registration processing. In a 1993 report, we estimated that FAA had forgone about $6.5 million in fees since 1968 because the registration fee did not cover the cost of reviewing and processing an application. Since that time, U.S. taxpayers have subsidized the processing of aircraft registrations and dealer certificates, including legal analysis, and covering the costs of labor, technology, postage, and other direct and indirect expenses. GAO’s federal user fee guide states that fee collections should be sufficient to cover the intended portion of program costs over time, including factors such as inflation. (See sidebar.) Without a fee that keeps pace with inflation and covers the cost of collecting and verifying applicant information for these high-value assets, FAA passes these costs on to U.S. taxpayers and limits the resources available for applicant verification. Individuals or entities may use opaque ownership structures—a legitimate means to register aircraft—to disguise potential ineligibility or hide illicit activity, according to our illustrative case and intermediary research, and interviews with FAA and law-enforcement officials. Opaque ownership structures are legitimate business structures that are widely used by corporations and individuals to facilitate commerce as well as for asset and tax management. However, we identified cases where these structures were used to name legal entities or trusts as the owner of an aircraft to disguise potential ineligibility or provide layers of anonymity in support of illicit activity. The lack of transparency related to these registrations also creates challenges for safety and law-enforcement investigators seeking information about beneficial owners of aircraft to support timely investigations, according to these officials. On the basis of interviews with FAA LEAP, SEIT, and law-enforcement officials, we identified four types of ownership structures that can be used to register an aircraft so that the beneficial owner is not transparent. The four types can be used alone or in combination and include the use of (1) LLCs, (2) shell companies, (3) noncitizen trusts, and (4) U.S. citizen corporations using voting trusts. According to our analysis of the registry’s calendar year 2018 data, although not mutually exclusive, there were 54,549 aircraft registered to LLCs; approximately 2,300 aircraft registered to likely shell companies; 3,300 registered as noncitizen trusts, and 4,200 registered to U.S. citizen corporations using voting trusts. The four types of opaque ownership structures are often established by intermediaries— individuals and entities that facilitate aircraft registration for a fee, such as by establishing legal structures and submitting aircraft registration applications and renewals. (See sidebar.) The use of intermediaries adds a layer of opacity to aircraft registrations. Intermediaries may not know, and most are not required to know, beneficial owners of aircraft they help to register. However, intermediaries that are banks are required to establish due diligence procedures for accepting and monitoring their clients as part of banks’ anti-money-laundering requirements under the Bank Secrecy Act and its amendments. To obtain beneficial ownership information, banks must identify and verify the identity of any individual who owns 25 percent or more of a legal entity, and an individual who controls the legal entity. Other intermediaries are not required to establish due-diligence procedures for accepting and monitoring their clients. Another approach that adds opacity to aircraft registrations is when applicants use the address of a registered agent—a person or entity authorized to accept service of process or other important legal and tax documents on behalf of a business—as the applicant’s address. Although the use of opaque ownership structures, intermediaries, and registered agents can serve legitimate purposes, they can also be abused in the context of aircraft registration to disguise potential ineligibility or hide illicit activity, according to our analysis of registry data and research. (See app. IV for additional details on the use of opaque ownership structures for aircraft registration.) In our analysis of illustrative cases involving U.S.-registered aircraft and our intermediary research, we identified examples where opaqueness and complexities of aircraft registrations using the ownership structures hindered FAA’s ability to prevent abuse of the registry to facilitate other criminal activity. In these examples, intermediaries used mechanisms allowable under current registration requirements to register aircraft, sometimes using multiple ownership structures for the same registration. The first example, based on our review of FAA registration records, illustrates opaqueness of information contained in FAA registration records and includes the use of multiple intermediaries and jurisdictions for an aircraft associated with asset forfeiture. The second example illustrates the use of an intermediary in establishing opaque ownership structures for several aircraft involved in illicit activities, including actions subject to U.S. sanctions. Use of multiple intermediaries and jurisdictions to obscure ownership of aircraft. According to our review of registry documents for this case, an intermediary registered the aircraft in 2010 using a noncitizen trust, providing limited information about the corporate trustor, whose beneficial owner was a high-net-worth foreign national. To register the aircraft, the intermediary—a bank providing corporate owner trustee services for aircraft registrations—established the noncitizen trust. The trust agreement identified the trustor as a company established in the British Virgin Islands. The trustor’s address for correspondence was listed as a post office box in Switzerland, with an email address indicating another trust company. Signatures of two trustors, identified as directors of two other apparent intermediary companies, were illegible and omitted printed names of individuals (see fig. 6). In 2019, the foreign national consented to the forfeiture of this aircraft and other property to DOJ in exchange for the release of certain other frozen assets, with both parties agreeing that the agreement did not constitute a finding of guilt, fault, liability, or wrongdoing. Use of an intermediary to obscure ownership of multiple aircraft. Between 2011 and 2018, an intermediary set up various corporations to facilitate aircraft registrations. The intermediary was an attorney who established the corporations using a registered agent service and also established voting trusts for those corporations to meet U.S. citizenship requirements for the aircraft registrations. Acting as director of these corporations, which have indicators of being shell companies, he registered two aircraft in 2011 and 2013. In 2019, individuals associated with these companies were sanctioned by OFAC as part of a U.S. sanctions program. Specifically, the individuals were designated in connection with paying bribes and involvement in a corruption scheme designed to take advantage of Venezuela’s currency exchange practices. The intermediary facilitated an aircraft sale about a month prior to the OFAC sanction designation for one aircraft and resigned from his position as director of the other company upon the OFAC announcement. Another aircraft registered by a company with the assistance of this intermediary in 2012 was seized in 2016 and forfeited to the U.S. government as part of the black-market currency exchange scheme. The investigation revealed that the aircraft had been purchased by a U.S. corporation whose sole beneficial owner was a Venezuelan individual using proceeds from a scheme that involved black-market currency exchange. The U.S. government seized the aircraft, alleging it was purchased with assets traceable to money laundering or other illegal activities, and the aircraft was later forfeited. Through our research on intermediaries, we identified another aircraft in which this intermediary had been similarly involved. Registration documents for this aircraft indicate a pattern of activity associated with potential trade-based money laundering. We are making a referral to DHS HSI for further investigation to determine whether individuals associated with the aircraft may have engaged in unlawful activity. Opaque ownership structures pose challenges for law-enforcement investigations. According to the 2018 National Money Laundering Risk Assessment, federal law-enforcement agencies noted that misuse of legal entities posed a significant money laundering risk and that law- enforcement efforts to uncover beneficial owners of companies can be resource-intensive, especially when ownership trails lead outside the United States or involve numerous layers. Law-enforcement officials across multiple agencies and FAA ASH, LEAP, and SEIT officials noted that challenges identifying beneficial owners of aircraft can impede their investigations. According to FAA LEAP agents, it is an ongoing challenge for them to identify beneficial owners. For example, according to FAA LEAP agents, a secretary of a company frequently registers aircraft on the company’s behalf and it takes time to determine the identity of the company’s beneficial owner. Limited PII in the registry records further impedes law-enforcement efforts. FAA LEAP agents and law-enforcement officials from DHS HSI and DEA described challenges they experience in their investigative work because aircraft registration records do not contain relevant PII, as noted above. For example, according to LEAP agents, they experience daily challenges identifying individuals without PII, particularly those with common names, hyphenated names, and multiple last names. This can be particularly difficult when aircraft are registered through legal structures, and, as DHS HSI officials noted, penetrating through the layers of ownership can take time, slowing down investigations. Further, one DEA official stated that without PII, identifying beneficial owners of aircraft is a challenge in his investigations, and in two cases he was ultimately unable to identify beneficial owners of aircraft. In prior work, we reported on challenges that law-enforcement officials face in their investigations when information is not available, particularly company ownership information such as names of directors or officers. As discussed earlier, the FAA DEA Act required FAA to promulgate regulations—in consultation with other federal agencies, law-enforcement officials, and representatives of the general aviation industry—that would require individuals to provide driver’s license and taxpayer identification numbers, but did not require applicants to provide date of birth. FAA’s approach, however, did not require applicants to submit driver’s license and taxpayer identification numbers. In part to serve the aviation community, which relies on publicly available registration information for the purchase and sale of aircraft, in 2005 FAA determined that adding PII to the records would require restricting access to them and therefore it would be detrimental to users of aircraft records, burdensome on aircraft owners and the government, and not necessary for law enforcement. FAA’s IT Modernization The Federal Aviation Administration (FAA) is making plans to modernize its information technology (IT) infrastructure for the registry, including potentially revising relevant regulations. According to FAA, it plans, among other things, to (1) enhance service delivery through process improvement and automation for near real-time access to accurate information; (2) utilize technology to reduce or eliminate mail, fax, or paper-driven service requests, processing, and information delivery; and (3) utilize technology to mine data to support risk-based decision-making, including the use of business intelligence algorithms to eliminate fraud, inaccurate information, and inappropriate use. In a May 2019 report, the Department of Transportation (DOT) Office of Inspector General (OIG) assessed FAA’s efforts and plans and determined that the agency has not identified costs, schedule, or an acquisition strategy for IT modernization. DOT OIG recommended, among other things, that FAA develop a timeline for making key decisions to implement IT modernization. See Department of Transportation, Office of Inspector General, FAA Plans To Modernize Its Outdated Civil Aviation Registry Systems, but Key Decisions and Challenges Remain, AV2019052 (May 8, 2019) We recognize the concerns for federal agencies associated with collecting and storing PII as well as the potential burden for applicants to submit such information. However, according to FAA officials, the IT modernization for which FAA is currently in its planning stages is intended to provide FAA the technical capability to adjust the level of access to registry records for various users, restricting PII access for some while allowing broader access to authorized users such as law-enforcement agencies. (See sidebar.) Industry associations and corporate registry users we interviewed expressed concerns about client privacy; however they also indicated openness to future technology improvements of FAA systems. Additionally, as noted earlier, use of PII is a key way federal programs verify the identity and eligibility of potential beneficiaries. Including in the planning stages of IT modernization basic elements of PII such as name, date of birth, physical address, and a driver’s or pilot’s license could provide FAA with the initial capability to verify applicant information while it develops a risk-based approach informed by its risk assessment. According to federal internal control standards, managers should use quality information to achieve the entity’s objectives, including obtaining relevant data from reliable internal and external sources in a timely manner. By not collecting and recording PII at the time of application and renewal, FAA has limited assurance of registrants’ eligibility, and lacks information that could support its oversight and law-enforcement officials’ ability to identify relevant persons and entities as part of investigations involving registered aircraft. As with applicant PII, FAA does not require applicants to submit information on beneficial owners of aircraft—individuals and certain entities that own more than 25 percent of the aircraft. In addition to the federal internal control standards for managers to use quality information to achieve the entity’s objectives, U.S. implementation of international standards for combating money laundering and terrorism financing would need to ensure availability of adequate, accurate, and timely information on beneficial ownership of high-value assets. By not collecting and recording information on beneficial owners in an electronic format that facilitates data analytics, FAA has limited assurance of registrants’ eligibility, and lacks information that could support its oversight and law- enforcement officials’ ability to identify relevant persons and entities as part of investigations involving registered aircraft. FAA makes some use of registry information on a case-by-case basis to detect potential fraud and abuse. FAA LEAP agents, in addition to supporting law-enforcement officials by providing access to registry information and specialized guidance related to aviation issues, have conducted registry analyses to identify suspicious and potentially illicit actors. For example, in 2018, FAA LEAP agents and registry officials started a project to flag aircraft registrations for FAA LEAP monitoring when applications are filed by entities or individuals, such as multiple shell companies associated with a certain individual, suspected of abusing registry processes. Additionally, one FAA LEAP agent told us that he reviews aircraft registrations filed the previous day and checks them against other information sources to determine suspicious activity, sharing leads identified through this analysis with law-enforcement officials for further investigation. However, this case-by-case review is limited to the data and information FAA currently collects, and is further hindered by a data format that does not support data analytics for fraud and abuse detection. FAA collects some information that could support fraud and abuse detection and oversight. As described earlier, FAA collects information on aircraft owners from the registration application, such as name and address, and these data are searchable and electronically analyzable. In April 2018, FAA also began tracking aircraft registrations that use voting trusts to meet U.S. citizenship requirements and trusts with noncitizen trustors, which are opaque ownership structures discussed earlier. This included recording in ancillary files the names of individuals and entities with potentially significant responsibilities for aircraft ownership, such as trustors and voting trustees. Additionally, according to FAA and some industry officials, the 3-year registration renewal implemented in 2010 has helped improve the quality of registry data that FAA collects. According to FAA officials, in addition to updating owner address information, registration renewal improves data quality as it prompts (1) reports of unreported aircraft sales, (2) new registrations due to ownership changes, and (3) cancelations due to destruction, scrapping, and exports. However, the benefits of registration renewal for data-quality purposes could diminish when the renewal period for noncommercial general aviation aircraft changes from 3 to 7 years, in alignment with new requirements from the FAA Reauthorization Act of 2018. Nevertheless, most of the information that FAA collects in the ancillary files and elsewhere is not recorded in a format that facilitates data analytics, according to our review of FAA’s registry system. Specifically, data on individuals and legal entities with potentially significant responsibilities for aircraft ownership such as trustors, beneficiaries, stockholders, directors, and managers are stored as imaged PDF records that, due to information-system limitations, cannot facilitate data analytics. For example, information on LLC directors and managers as well as directors, managers, and stockholders of U.S. citizen corporations that use voting trusts is stored in imaged records. Our intermediary research identified an aircraft registered to a company whose sole stockholder was subject to U.S. sanctions; however, FAA currently stores data on foreign stockholders of U.S. citizen corporations that use voting trusts in PDF records, preventing it from being able to conduct data analysis to identify such individuals or entities for all registrations. Such data may be useful in identifying entities and individuals subject to U.S. sanctions, as discussed below. Additionally, the current system configuration limits FAA to viewing individual records within the ancillary files. This configuration prevents agency officials from tracking aircraft registration numbers—a common identifier—across records or linking them to the registration data portion of the registry. Further, FAA internally tracks noncitizen trusts and U.S. citizen corporations using voting trusts as one category within registry data, preventing analysis and monitoring of each group of registrations. Lastly, FAA stores records of declarations of international operations— requests that expedite registration processing for aircraft intending to travel outside the United States—as imaged PDF records, so information about the aircraft, owner’s name, departure and destination locations, date of intended travel, and name of the individual submitting the declaration are not in a format that facilitates data analytics. According to 2017–2018 analysis of information from declarations of international operations with checks against flight history data, FAA SEIT identified patterns of activity that could be used in support of safety and law- enforcement investigations, as discussed later in this report. Furthermore, due to manual data entry and lack of verification, the registry’s postal data may not support effective data analytics and oversight. FAA staff also have the option to override the formatting prompts produced by its address validation software. Our analysis of 2018 physical and mailing address data found that about 25,000 (9 percent) of all registrant addresses did not match a valid address in the USPS postal verification data, while just over 300 (about 3 percent) of all dealer addresses did not match. Of the seven aircraft registration cases we selected based on address category and locality, we found three registrant addresses that indicated a registry data-quality issue and one that did not. Specifically, our review of the application forms for two registrants showed that a physical address was provided by the registrants, but was not recorded in the physical address file. In another case, our review of five registration records for one company showed that FAA revoked registrations for the five aircraft in 1971, but did not deregister them until 2019, sending deregistration notification letters to the original address, which were returned as undeliverable. We did not find any noncompliance in the last case and, based on our review of aircraft registration documents, determined that a change of address form was provided to FAA following the most-recent renewal, but the new address had not yet been updated at the time we received the physical address data. As described earlier, FAA is taking steps to modernize its IT system for the registry because it is outdated. According to a recent DOT OIG report, the system had its last significant upgrade in 2008, is approaching the end of its service life, suffers intermittent outages, and uses an outdated programming language. According to FAA, the future system is expected to streamline and automate processes, allow for the submission of electronic forms, improve online data availability, and implement additional security controls, such as software that can cross-check aircraft registrations with other government databases. In December 2018 and June 2019, FAA issued requests for information to conduct a market survey and to develop a strategy based on feedback received, respectively. As of November 2019, FAA was making plans to issue a request for proposal, but did not identify specific time frames. Registry system modernization presents an opportunity to mitigate data format limitations as FAA designs new systems and controls. According to federal internal control standards, managers should use quality information to achieve the entity’s objectives. Managers can do that by designing processes and identifying information requirements needed to achieve objectives and address risks as well as by processing obtained data into quality information that supports the internal control system. This could include electronically analyzable information from declarations of international operations and information on owners and related individuals and entities with potential significant responsibilities for aircraft ownership such as beneficial owners, trustors, trustees, stockholders, directors, and managers. Without analyzable data on significant parties involved in aircraft registrations that can be linked through a common identifier, FAA is limited in its ability to exercise its domestic and international oversight functions and fully support safety and law-enforcement investigations. Use of data analytics to detect suspicious activity, anomalies, or patterns is one of the leading practices identified in GAO’s Fraud Risk Framework. However, registry officials primarily use collected data to send automated notifications, such as for aircraft renewals, and current use of data to support oversight is limited, in part hindered by data format limitations described earlier. In addition, registry officials do not analyze various external data sources against registry data to detect patterns of potential fraud or abuse. Risk indicators identified through such analyses may serve as points of inquiry for a broader fraud risk assessment, or for further examination of conduct that may pose criminal, national security, or safety risks. To demonstrate how FAA could identify registrations with indicators of potential fraud or abuse that may enable criminal activity, national security, and safety risks, we analyzed aircraft registry and related data. Specifically, we analyzed aircraft registry data from publicly available and ancillary files, as well as matched registry data against other datasets to identify (1) registrations using registered agent address, (2) registrations using opaque ownership structures, (3) aircraft registration addresses located in countries identified by the Department of State as associated with major illicit drug production and money laundering, (4) OFAC data on individuals and entities subject to U.S. sanctions, and (5) NTSB safety accident and incident reports. Based on this analysis, we found over 17,000 registrations out of approximately 300,000 registrations associated with one or more risk indicators for fraud or abuse. The majority of registrations (over 15,000 or about 90 percent) were associated with one risk indicator, about 2,000 registrations (10 percent) were associated with two risk indicators, and the remaining 140 (1 percent) were associated with three or more risk indicators. The results of our various analyses are described below. Use of registered agent address. As discussed earlier, registered agents are authorized to accept legal documents on behalf of a business. According to FAA officials, FAA will accept the use of a registered agent’s address as a mailing address, provided the owner’s physical address is also included. Our analysis of registry data identified cases where a registered agent’s address was recorded as the registrant’s physical address. The registry data do not specifically identify registered agents, but by analyzing address information for calendar year 2018, we identified at least 4,080 cases using registered agents’ addresses. For one of the registered agents we were able to confirm, we identified 965 associated registrations, including about 300 registrations associated with characteristics of a likely shell company or that were a noncitizen trust or a U.S. citizen corporation using a voting trust. Further, for this one registered agent, we identified about 280 unique business names, associated with about 760 registrations, which used this registered agent’s address on aircraft registration applications. Additionally, based on our analysis of postal address data provided by FAA as well as verification of selected cases, we identified and confirmed through site inspections two additional registered agents whose addresses were used in over 100 registrations and over 3,220 registrations, respectively. Use of registered agent addresses, when not accompanied by physical address information, particularly in combination with opaque ownership structures, provides a layer of anonymity to beneficial owners of aircraft and may mask ineligibility or illicit actors. Noncitizen trusts and U.S. citizen corporations using voting trusts. We reviewed internal FAA trust data from April 2018 through May 2019— the full range of data available at the time of our review—to identify the number of registrants that were noncitizen trusts or were U.S. citizen corporations using a voting trust. In total, we found about 6,800 such registrations contained in the registry data. Of these registrations, two were associated with individuals subject to U.S. sanctions, four were associated with an FAA revocation or suspension, and 16 appeared to be shell companies. FAA regulations allow for registrations using noncitizen trusts and U.S. citizen corporations using voting trusts as valid means of enabling registrants to meet FAA’s citizenship requirements. However, as discussed earlier and according to FAA and law-enforcement officials, registrations using noncitizen trusts and U.S. citizen corporations using voting trusts may also mask ineligibility or illicit actors. Consistent with their program-management responsibilities, if FAA registry officials detect aircraft owners, dealers, or intermediaries potentially abusing registration requirements or abusive use of noncitizen or voting trusts, they may send them warnings of denial of future services if observed abusive actions continue. For example, if registry officials suspect that an entity applying for registration is misrepresenting its citizenship, officials could request citizenship information as appropriate for the president, board of directors, and managing officers. If the inquiry results in a determination that the entity does not qualify as a citizen, FAA could deny the application or issue a letter of apparent ineffectiveness for an existing registration. However, according to FAA officials, they take mitigation actions on a case-by-case basis because they do not have a systematic way to analyze data and detect potential fraud and abuse. Department of State country lists associated with major illicit drug production and money laundering. We analyzed registry address data using lists of countries associated with major illicit drug production and money laundering published by the Department of State to identify aircraft registrations associated with such countries. We found 251 registrations with addresses located in countries on the Department of State’s list of money laundering jurisdictions that were registered as noncitizen trusts or corporations using voting trusts. Countries identified in the Department of State’s lists do not necessarily indicate that a registration is associated with criminal activity. However, the risk of abuse or illicit activity with these registrations may be increased when combined with the use of opaque ownership structures, another risk indicator that, according to FAA and law-enforcement officials, may mask ineligibility or illicit activity. U.S. sanctions. We analyzed and matched registry data to U.S. sanctions data that contain information on blocked assets and sanctioned entities and individuals. Through this data analysis as well as illustrative case and intermediary research, we identified six aircraft owned by entities subject to Venezuela-related U.S. sanctions from 2017 to February 2019. These six aircraft involved registrations established by intermediaries using noncitizen trusts or by U.S. citizen corporations using voting trusts, where aircraft were beneficially owned by noncitizen trustors or stockholders of companies using voting trusts to meet U.S. citizenship registration requirements. However, as discussed earlier, trust agreements that contain information on aircraft owners and related individuals and entities with potentially significant responsibilities for aircraft ownership are stored in PDF format that are not electronically analyzable, potentially inhibiting detection of sanctioned individuals or entities. Additionally, our analysis identified limitations in the sharing of sanctions information within FAA, specifically between the aircraft registry and dealer records. These limitations present the risk of registry abuse or illicit activity through sanctions violations while potentially impeding effective coordination between FAA and Treasury’s OFAC, which administers U.S. sanctions programs. On the basis of U.S. national security and foreign policy goals, OFAC can impose controls on transactions and block or freeze assets under U.S. jurisdiction, including aircraft. By blocking an asset such as an aircraft, its title remains with the targeted individual or entity; however, these individuals and entities cannot exercise the powers and privileges normally associated with ownership unless authorized by OFAC. Certain activities related to the use of the aircraft may violate the relevant sanctions program. Additionally, OFAC regulations generally prohibit persons and entities within the United States from engaging in transactions involving blocked property—including U.S-incorporated companies and aircraft—of sanctioned individuals and entities. OFAC-Sanctioned Aircraft One of the U.S.-registered aircraft about which Treasury’s Office of Foreign Assets Control (OFAC) notified the Federal Aviation Administration (FAA) was used as part of an illicit narcotics trafficking scheme. According to its 2017 announcement, OFAC designated a high-ranking Venezuela government official as a Specially Designated Narcotics Trafficker pursuant to the Foreign Narcotics Kingpin Designation Act (“Kingpin Act”) for playing a significant role in international narcotics trafficking. According to OFAC, the sanctioned official used a front man who laundered drug proceeds and purchased assets. In addition to a network of international companies, according to OFAC, the front man owned or controlled five U.S. companies, including a limited liability company (LLC) that registered an aircraft with FAA and used a voting trust to meet U.S. citizenship requirements. As part of its action, OFAC identified the U.S.-registered aircraft and the LLC as blocked property. FAA deregistered the aircraft in 2019 after registration renewal documentation submitted to FAA contained numerous errors. However, because the flags placed on sanctioned individuals’ and entities’ registration records do not extend to dealer records, FAA issued a dealer certificate to the blocked LLC after the OFAC designation and without coordination with OFAC, according to FAA records and officials. The blocked LLC held the dealer certificate for a year until the certificate expired. (See app. I.) FAA relies on OFAC to share information on sanctions and does not check whether applicants and aircraft are subject to U.S. sanctions or blocking at registration, at renewal, or on a periodic basis. Specifically, FAA does not proactively obtain and use OFAC data to detect (1) blocked aircraft, (2) entities or individuals subject to sanctions, or (3) those with potentially significant responsibilities for aircraft ownership, such as intermediaries registering on behalf of blocked aircraft or entities. Our analysis of the six cases revealed that OFAC officials initiated coordination with FAA, notifying FAA about four of the six cases. According to FAA officials, when FAA finds out about a blocking action from OFAC, it internally flags registry records and will withhold registration processing actions until further communication with OFAC. However, according to FAA officials, FAA does not have the authority to deny or revoke a registration solely because the registration is associated with an individual subject to OFAC sanctions. Accordingly, in those instances, FAA would register the aircraft or the aircraft’s registration would remain valid. In addition, although FAA flags sanctioned individuals’ and entities’ registry records, the flags do not extend to dealer certificate records. As a result, sanctioned individuals or entities flagged in aircraft registration records are not flagged by FAA for OFAC coordination before receiving a dealer certificate, which could allow operation of blocked aircraft under that certificate. One of the six cases we identified illustrates the criminal and national security risks involved with the use of U.S.-registered aircraft by OFAC-sanctioned individuals and entities, as well as risk-management challenges associated with dealer certificates. (See sidebar.) OFAC efforts to identify aircraft assets associated with sanctioned individuals and entities can encounter obstacles. According to OFAC officials, they search the publicly available FAA registry to identify aircraft for potential blocking. Where OFAC is aware that a sanctioned individual has control of a company, and the company had directly registered an aircraft, a search of the public database can provide relevant information about the aircraft. However, according to OFAC officials, identifying aircraft is more challenging when, for example, a voting trust or a shell company is the registered owner. As a result, OFAC does not have all the information from FAA it might need to support its investigations or enforcement when aircraft associated with sanctioned entities and individuals are not readily identifiable. FAA’s IT modernization provides an opportunity for FAA to link flagged records across aircraft registration and dealer systems and to proactively check OFAC sanctions data. OFAC provides information on individuals and entities subject to sanctions on its website that can be checked using online searches or by downloading data, but FAA officials said that checking sanctions designations would require resources and extend processing time for aircraft registrations. However, automated linkages across aircraft registration and dealer systems, and checks of OFAC information, could be achieved through FAA IT modernization, which aims to automate near-real time access to accurate information. An aspect of the modernization project could involve automatically cross-referencing sanctions data, which are dynamic and updated in real time in response to U.S. sanctions programs, with aircraft registration information on owners and related individuals and entities with potentially significant responsibilities for aircraft ownership, such as intermediaries. FAA noted that it does not have authority to deny or revoke a registration based solely on an OFAC sanctions designation. Nevertheless, records that are flagged across aircraft registration and dealer systems, as well as awareness of blocked aircraft, sanctioned owners, or intermediaries doing business with sanctioned entities, would help to ensure coordinated actions with OFAC. Such coordination would allow OFAC to seek a delay from FAA of the registration or dealer certification, to alert law- enforcement agencies to determine aircraft location, or to coordinate with its U.S. partner agencies on investigations as appropriate. By not linking flagged records across systems and not proactively checking OFAC sanctions data, FAA and OFAC may be unaware of, and therefore not well-positioned to manage, risks associated with registration of blocked aircraft, sanctioned entities, or intermediaries operating in violation of U.S. sanctions. In addition, FAA misses opportunities to address abuse of the registry for illicit purposes, as well as to provide information to OFAC in support of U.S. efforts to curb drug trafficking, corruption, and other illicit activity. Aircraft primarily operating outside the United States. According to our analysis of NTSB data, we identified 303 cases of U.S.-registered aircraft involved in accidents and incidents outside the United States from calendar years 2010 to 2018. According to FAA officials and our illustrative case research, U.S.-registered aircraft that are primarily based and operated outside the United States may be associated with risk of registration abuse. For example, FAA SEIT and LEAP officials told us that they were aware of numerous cases of aircraft operated primarily outside the United States that were registered to nominee buyers. In addition, they noted international operation of aircraft that were associated with illicit activity and registration violations such as bills of sale identifying foreign owners and cloned registrations. A 2010 case involving a U.S.-registered aircraft seized for alleged drug trafficking by the Panamanian government highlights registration violation risks related to aircraft primarily operating outside the United States. After Panama seized the aircraft, it was turned over to the country’s civil aviation authority (CAA), which registered the aircraft in Panama and painted a Panamanian registration number on it. According to FAA officials, the CAA did not seek to deregister the aircraft from the United States, and the new registration was likely invalid under international law. According to FAA officials, the Panamanian CAA operated the aircraft for about 1 year before it crashed. During that time, the aircraft remained registered to the original U.S. owner at a registered agent address. FAA sent multiple letters to the owner to deregister the aircraft and also when the aircraft registration was expiring, but all were returned as refused by the registered agent. Multiple Safety Violations Contributed to the Crash of an Aircraft Primarily Operating Outside the United States Our research identified a case where safety violations contributed to a fatal accident in the Caribbean involving a U.S.-registered aircraft in 2016. A Jamaican aviation training center was operating the aircraft since 2015 and at the time of the crash. The accident investigation by Jamaican authorities identified multiple safety deficiencies as the causes and contributing factors of the crash. This included falsified aircraft maintenance records, an engine replacement that did not conform to aircraft model and type, and the use of non-U.S.-certified maintenance programs. (See app. I.) Furthermore, aircraft that are based and primarily operated outside the United States may pose safety risks by not meeting FAA aircraft maintenance standards. Once registered with FAA, aircraft owners must continue to meet eligibility requirements and, along with operators, comply with certain maintenance responsibilities in order to operate, regardless of their location. According to FAA officials, U.S.-registered aircraft operating outside the United States may receive less scrutiny and inspections from other countries’ CAAs, and nefarious actors prefer a U.S. registration when aircraft are inspected abroad. Additionally, FAA SEIT and LEAP officials told us that they were aware of many U.S.- registered aircraft primarily operating in Latin American countries that may not be following required U.S. maintenance programs, thus posing aviation safety risks. One of our case studies highlights safety risks related to U.S.-registered aircraft that are primarily based and operated outside the United States. (See sidebar.) In another example involving 2011 and 2013 FAA examinations, an FAA maintenance inspector conducted inspections of U.S.-registered helicopters and airplanes located in Panama at the request of the Panama CAA and found multiple violations. According to FAA, the inspection of 16 aircraft initially found that, in addition to registration issues such as flying with a temporary registration, ten aircraft had maintenance issues, including maintenance performed by nonauthorized personnel. At least seven of the issues identified during this inspection resulted in FAA enforcement actions. According to this official, two of the aircraft had significant maintenance concerns and were not airworthy. On the basis of his experience inspecting aircraft domestically, safety violations among the aircraft inspected in Panama were more significant. In combination with other data sources and information, flight history data can provide indications of safety risks associated with aircraft based and primarily operated outside the United States. However, according to registry officials, they do not use these data to identify such risks. To examine specific registrations based on the entire risk-indicator data analysis, we also reviewed randomly selected aircraft registrations across each overall risk-indicator category. Our review of 20 selected registrations generally confirmed the risk-indicator characteristics we had identified for analysis. We did not identify further indicators of risk as part of this review except for the OFAC cases described earlier. Analysis of various data sources, alone or in combination, can help detect patterns of potential fraud or abuse. As demonstrated by our analysis, FAA data, such as postal addresses, information on dealers, noncitizen corporations, intermediaries, and entities with significant responsibilities for aircraft ownership, among others, along with various external databases could be used for such a purpose. FAA also has access to flight history data, currently used on an ad hoc basis, but which could also serve for (1) routine oversight functions such a verifying aircraft are based and primarily operating in the United States for certain registrant types or (2) to detect patterns of activity associated with declarations of international operations that could be used in support of safety and law- enforcement investigations. In addition, our analysis of registry data against external data sources, such as OFAC sanctions lists, illustrates the utility of such analyses for detecting registrant risks. FAA currently does not use internal or external information for such analysis or to assist in safety or law-enforcement oversight responsibilities across multiple aircraft, registrations, or dealer certificates. This is due, in part, to data limitations, but also because, according to registry officials, their role is primarily focused on recording of aircraft registration information. Aircraft registration data made available through IT modernization, as well as other currently available data, could support ongoing monitoring and risk- based oversight by FAA. Federal internal control standards call on managers to establish and operate activities to monitor the internal control system and evaluate results. By not analyzing available internal and external data, FAA is missing opportunities to identify registrant risks, conduct oversight, and safeguard the registry from potential fraud and abuse. Furthermore, while FAA registry officials may take risk-based mitigation actions, such as by sending warnings letters or denying services if abusive actions are detected, it generally does not take such action. According to FAA officials, the registry focuses on recording information, while it is currently the responsibility of other FAA organizations, such as ASH, LEAP, and SEIT, to detect fraud. However, federal internal control standards require managers to respond to risks by remediating internal control deficiencies on a timely basis. Without timely and measured risk-based mitigation actions, the aircraft registry continues to be vulnerable to fraud and abuse. In this context, as the key program office, aircraft registry is best positioned to manage fraud and abuse risks—by preventing, detecting, and responding to risks—in close coordination with stakeholder organizations such as ASH, LEAP, and SEIT. FAA and law-enforcement agencies have a variety of enforcement mechanisms to respond to instances of suspected fraud and abuse in aircraft registrations. For example, FAA can use administrative actions, such as aircraft registration suspensions and revocations, and law- enforcement agencies can use civil actions and criminal prosecutions to seize aircraft, among other enforcement actions. Law-enforcement agencies such as DEA, DHS HSI, and DOT OIG have authority to investigate criminal activity and take actions to seize aircraft when warranted. Recognizing the need for better dialogue and coordination, in August 2017 FAA LEAP agents launched the Aircraft Registry Task Force to discuss ideas and solutions for dealing with potentially fraudulent aircraft registrations and to improve FAA processes to assist the law-enforcement community. The first meeting, in August 2017, included participants from FAA—aircraft registry officials, legal counsel, ASH, LEAP, and SEIT—as well as other federal agencies, including DEA and DHS HSI. This meeting was the first time these various units came together to discuss aircraft registry vulnerabilities. FAA and law-enforcement officials presented cases associated with fraudulent aircraft registrations, highlighting safety implications. Participants also discussed issues related to deregistration, and aircraft seizures, among others. According to aircraft registry officials and FAA LEAP agents, the task force meeting discussions resulted in several changes, including revisions to the signature block in the aircraft application form, addition of a separate registration type for LLCs for tracking purposes, and sharing of declarations of international operations with FAA LEAP and SEIT. Specifically, regarding modifications to the signature block, in 2018 FAA added a statement requiring applicants to certify that information they provide is true and accurate while also identifying specific penalties for false information. The subsequent task force meeting, held in October 2018, included only FAA participants. Aircraft registry officials, legal counsel, ASH, LEAP, and SEIT, among others, discussed follow-up from the previous meeting and covered topics associated with ongoing concerns such as falsification of registration documents, incomplete applications, and proof of citizenship, among others. According to FAA officials, since the October 2018 meeting, the task force has not met. FAA and DEA have also established informal mechanisms to address registration violations and safety risks associated with aircraft based and operated outside the United States. For example, in 2016 and 2017, DEA and FAA LEAP and SEIT officials conducted a joint initiative at the request of the government of Guatemala to examine multiple U.S.- registered aircraft located in Guatemala. According to FAA, a total of 81 U.S.-registered aircraft were inspected through this effort as of April 2017. During the inspections, FAA identified more than 25 registration violations and numerous safety violations resulting in approximately 31 condition notices. Additionally, authorities seized eight aircraft with an approximate value of $2.5 million as well as over 400 kilograms of cocaine. According to FAA, registration violations identified during this effort included inconsistencies with trust agreements and associated documentation, violations involving U.S. corporations having individuals listed as president who do not meet U.S. citizenship requirements, and documentation allowing non-U.S. citizens to control U.S.-citizen entities that had registered aircraft. Since then, according to FAA officials, on the basis of the results of this initiative, DEA and FAA officials have conducted similar visits to other countries in Latin America and the Caribbean. The visits typically include training for local CAA officials on authorities to inspect U.S.-registered aircraft, ramp checks of U.S.- registered aircraft located in these countries, and maintenance inspections. FAA and DHS HSI also use informal collaboration mechanisms to support law-enforcement investigations. According to DHS HSI officials, they have a robust relationship with an FAA LEAP agent with whom they communicate on a daily basis. This agent has helped to investigate aircraft sale transactions and other cases and also provided leads to DHS HSI officials. Declarations of International Operations The Convention on International Civil Aviation requires registration certificates for international operations. The Federal Aviation Administration’s typical registration process takes 16–20 working days, during which applicants may fly domestically using a temporary registration. Registry officials have put in place declarations of international operations for applicants to notify the registry of the intent to operate internationally thereby expediting typical processing time to the same day or next day. FAA registry officials have been sharing expedited registration filings— declarations of international operations to expedite registration processing for aircraft intending to travel internationally—with FAA LEAP and SEIT officials for monitoring and analysis purposes. (See sidebar.) However, this informal collaboration does not extend to FAA sharing of declarations of international operations with DHS HSI or DEA. According to law- enforcement officials, declarations of international operations present challenges. Specifically, DEA officials noted that expedited registrations limit the amount of time law enforcement can effectively query appropriate sources of information to determine that payment for the aircraft is not derived from illicit proceeds. In addition, according to DEA officials, expedited registrations shorten the amount of time investigators have to determine whether the aircraft is being used to facilitate drug crimes and to identify beneficial owners of the aircraft, which, as discussed earlier in this report, can be a time-consuming process. The lack of notification about declarations of international operations further compounds these challenges. DHS HSI officials explained that they have experienced challenges not receiving information from expedited registrations, which could have allowed some illicit actors to expediently move or export aircraft out of the country, including as part of trade-based money laundering or trafficking schemes. According to these officials, aircraft can be purchased with illicit proceeds to launder money as well as used to smuggle illicit cargo such as persons, cash, cigarettes, and liquor. DHS HSI officials stated that, in one case, which resulted in aircraft seizure, the aircraft potentially could have been seized 2 years earlier if they had received declaration of international operations at the time of aircraft registration. Additionally, according to DHS HSI officials, information from declarations of international operations could help to generate leads, including information on planned travel to countries that are associated with illicit drug trafficking or money laundering. For example, they noted that in investigations of trade-based money laundering schemes, information from declarations of international operations can be used to check against shipping export declarations and trade data from other countries. Separately, in our analysis of aircraft registered to entities subject to U.S. sanctions described earlier, we found that five of the six aircraft registrations received expedited processing. Although not a precise indicator of actual travel, information from declarations of international operations could provide timely information about potential planned movement of aircraft in time-sensitive situations as well as bring awareness for longer-term investigative purposes. Expedited registrations provide more immediate opportunity to move aircraft out of the country and information on applicants’ intention to do so, which can inform monitoring and law-enforcement action. However, FAA does not provide declarations of international operations to DHS HSI or DEA. Without declarations of international operations, these law-enforcement entities may be missing opportunities to generate leads that would ultimately support FAA’s interests in addressing abuse of the registry for illicit purposes and support detection and response to potential trade-based money laundering and other cross-border schemes. Our prior work on interagency collaboration identified practices that can help enhance and sustain collaboration among federal agencies, including written agreements and use of liaison positions. Agencies that articulate their agreements in formal documents, such as memorandums of understanding, can strengthen their commitment to working collaboratively. Additionally, articulating a common outcome and roles and responsibilities in a written document can facilitate coordination. Similarly, the use of liaison positions, when an employee of one organization is assigned to work primarily or exclusively with another agency, can enhance coordination. For example, by providing direct access to agency information, liaison positions have helped to facilitate sharing of information and coordination of missions and activities. As relatively new and unofficial collaboration mechanisms, the Aircraft Registry Task Force and other efforts have not been fully utilized or leveraged some of the enhanced collaboration practices such as written agreements or liaison positions at law-enforcement agencies. While FAA LEAP agents coordinate with law-enforcement officials, these are not liaison positions as suggested by leading practices for collaboration, wherein an employee is assigned to or works primarily with another agency and has direct access to agency staff and information, and arrangements are formally outlined, such as in memorandums of understanding. Rather, FAA LEAP agents are assigned to FAA and do not have formal agreements for collaboration. The Aircraft Registry Task Force holds potential for FAA to work collaboratively internally and externally by formalizing various informal coordination efforts, such as international inspections by FAA and DEA and sharing of declarations of international operations with law-enforcement agencies, to bring together varied perspectives, functions, and skill sets necessary to mitigate aircraft registry vulnerabilities going forward. Leading practices in risk management also call for involvement of relevant stakeholders as part of risk-assessment and risk-mitigation activities. In the FAA context, the aircraft registry is best positioned to develop preventive measures and controls in coordination with FAA LEAP, SEIT, and law-enforcement stakeholders. FAA’s aircraft registry, the largest in the world, is preferred by aircraft owners for safety, economic, and financial reasons. Accordingly, the integrity of owner information for registry users is important to support these benefits. It is also important to ensure the registry is not exploited for fraudulent purposes or to support illicit activity involving U.S.- registered aircraft. FAA’s current process does not include strong controls to prevent ineligible registrants and potential fraud and abuse, instead allowing registrants to self-certify their information with limited independent review. A comprehensive registry risk assessment could help to manage risks of fraud and abuse, which enable criminal, national security, and other risks. Such a risk assessment, which considers inherent and residual risks as well as determination of likelihood, impact, and risk tolerance, would support the development of a risk-based strategy and approach to guide registry actions in preventing, detecting, and responding to fraud and abuse risks. To support its eligibility determinations, FAA currently obtains limited PII from individual registrants, aircraft dealers, or those entities (e.g., trustors) who might have a significant role in aircraft registrations. Additionally, the registry lacks information about beneficial owners of aircraft. Further, the registry generally accepts self-certification of eligibility and aircraft ownership and does not verify the information it receives. Such an approach may be appropriate for the majority of law-abiding registrants, but it leaves the registry vulnerable to exploitation by those who wish to circumvent eligibility requirements, disregard safety standards, or pursue criminal activities. Limited transparency into who beneficially owns aircraft has also precluded FAA from maximizing its collaboration with partners in the law-enforcement and safety communities to support detection and investigation of criminal, national security, and safety risks associated with registered aircraft. U.S. taxpayers have subsidized the costs of aircraft registration for several decades. Without a change to aircraft registration and dealer fees, the costs of FAA labor, technology, coordination, and risk-based oversight for these high-value assets would continue to be borne by the public and limit resources available for applicant verification. The absence of more and electronically analyzable information has substantially hindered FAA’s ability to use the registry as a tool to detect potential fraud and abuse and to oversee registered aircraft. As part of its ongoing IT modernization, FAA has an opportunity to collect such data and record them in a format that facilitates data analytics. These data could help FAA detect potential fraud and abuse and conduct preventive, risk-based monitoring and oversight of aircraft registrations as well as dealer certifications to ensure the integrity of the registry. They would also support a risk-based approach for verifying information provided by some registry applicants as well as for taking corrective actions. Additional information would position FAA to more broadly prevent, detect, and respond to risks associated with the aircraft registry and to facilitate data analytics by FAA and stakeholders for oversight, safety, and law- enforcement purposes. For example, FAA officials could analyze data patterns for potential fraud and abuse, as well as share data across dealer and aircraft records and to check OFAC sanctions data to ensure that they coordinate about owners with sanctions designations, as appropriate. Lastly, FAA lacks formal agreements with other federal entities to respond to risks. Specifically, FAA can provide additional support to law- enforcement and safety investigations by sharing quality information about individuals and entities with potentially significant responsibilities in aircraft registrations, as well as other registration information, such as declarations of international operations. FAA’s Aircraft Registry Task Force positions FAA to work collaboratively internally—among officials from the aircraft registry, legal counsel, ASH, LEAP, and SEIT—and with external law-enforcement to share information and to take advantage of collaborative mechanisms to formalize coordination. We are making the following 15 recommendations to FAA: The Administrator of FAA should conduct and document a risk assessment that considers inherent and residual fraud and abuse risks that may enable criminal, national security, or safety risks. (Recommendation 1) The Administrator of FAA should determine impact, likelihood, and risk tolerance as part of a risk assessment. (Recommendation 2) The Administrator of FAA should develop a strategy that outlines specific actions to address analyzed risks, including periodic assessments to evaluate continuing effectiveness of the risk response. (Recommendation 3) The Administrator of FAA should collect and record information on individual registrants, initially including name, address, date of birth, and driver’s license or pilot’s license, or both, with subsequent PII elements informed by the risk assessment, once completed. (Recommendation 4) The Administrator of FAA should collect and record information on legal entities not traded publicly—on each individual and entity that owns more than 25 percent of the aircraft; for individuals: name, date of birth, physical address, and driver’s license or pilot’s license, or both; and for entities: name, physical address, state of residence, and taxpayer identification number. (Recommendation 5) The Administrator of FAA should verify aircraft registration applicants’ and dealers’ eligibility and information. (Recommendation 6) The Administrator of FAA should increase aircraft registration and dealer fees to ensure the fees are sufficient to cover the costs of FAA efforts to collect and verify applicant information while keeping pace with inflation. (Recommendation 7) The Administrator of FAA should ensure, as part of aircraft registry IT modernization, that information currently collected in ancillary files or in PDF format on (1) owners and related individuals and entities with potentially significant responsibilities for aircraft ownership (e.g., beneficial owners, trustors, trustees, beneficiaries, stockholders, directors, and managers) and (2) declarations of international operations is recorded in an electronic format that facilitates data analytics by FAA and its stakeholders. (Recommendation 8) The Administrator of FAA should link information on owners and related individuals and entities with significant responsibilities for aircraft ownership through a common identifier. (Recommendation 9) The Administrator of FAA should, as part of IT modernization, develop an approach to check OFAC sanctions data on owners and related individuals and entities with potentially significant responsibilities for aircraft ownership for coordination with OFAC and to flag sanctioned individuals and entities across aircraft registration and dealer systems. (Recommendation 10) The Administrator of FAA should use data collected as part of IT modernization as well as current data sources to identify and analyze patterns of activity indicative of fraud or abuse, based on information from declarations of international operations, postal addresses, sanctions listings, and other sources, and information on dealers, noncitizen corporations, and individuals and entities with significant responsibilities for aircraft ownership. (Recommendation 11) The Administrator of FAA should develop and implement risk-based mitigation actions to address potential fraud and abuse identified through data analyses. (Recommendation 12) The Administrator of FAA should develop mechanisms, including regulations if necessary, for dealer suspension and revocation. (Recommendation 13) The Administrator of FAA, in coordination with relevant law-enforcement agencies, should enhance coordination within the Aircraft Registry Task Force through collaborative mechanisms such as written agreements and use of liaison positions. (Recommendation 14) The Administrator of FAA, in coordination with relevant law-enforcement agencies, should develop a mechanism to provide declarations of international operations for law-enforcement purposes. (Recommendation 15) We provided a draft of this product to DOT, DOJ, DHS, and Treasury for review and comment. DOT provided written comments, which are reproduced in appendix V. DOT concurred with our recommendations. Specifically, DOT stated that it supports other government agencies in addressing illegal activities and enforcing U.S. sanctions and agreed that enhancements to the accuracy of registry information would expedite enforcement actions and reduce the risk of ineligible aircraft registrations. FAA and DHS provided technical comments, which we incorporated as appropriate. DOJ and Treasury did not have any comments. We are sending copies of this report to the appropriate congressional committees, the Secretary of Transportation, the Attorney General, the Secretary of Homeland Security, the Secretary of the Treasury, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at 202-512-6722 or shear@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. We conducted illustrative case research related to U.S.-registered aircraft generally covering the 2010–2018 period, including over 1,200 publications and reports from cases investigated by law-enforcement agencies, news articles, and agency and safety investigation reports. We selected six case studies for in-depth review across three categories of risk enabled by aircraft registration fraud and abuse—criminal activity, national security, and safety (see app. II for additional details on the selection methodology). All selected cases are intended for the purpose of illustrating fraud and abuse vulnerabilities associated with the aircraft registration process. These cases may not represent all existing vulnerabilities and are not generalizable to the Federal Aviation Administration (FAA) registry population as a whole. From 2010 to 2011, an aircraft sales broker obtained multiple registration certificates from FAA for aircraft he did not rightfully own or possess. According to court records associated with this case, the broker submitted to FAA fraudulent registration applications and bills of sale with forged signatures for 22 aircraft as part of a multi-million-dollar bank fraud scheme. He used the registration documents that FAA provided as an asset to support a loan application that ultimately resulted in an approximately $3 million bank loan used to float his failing aircraft-sales business. The bank uncovered the fraud over a year after the sales broker first submitted the fraudulent aircraft registration documents to execute the loan. A subsequent investigation by the Federal Bureau of Investigation revealed the extent of the fraud, namely that the main thrust of the fraud scheme was to pledge as collateral 22 aircraft that neither the broker nor his company owned, in order to obtain money from the bank. Court records reveal that law-enforcement officials interviewed some of the rightful owners of the aircraft, who stated that the aircraft were always in their possession and they had never sold the aircraft to the fraudulent broker. These owners identified the signatures on the bills of sale used to register the aircraft as forged. In 2013, the broker pled guilty to bank fraud, making a false statement to a federally insured financial institution, and making a false statement to FAA in the registration of aircraft. As a result of the fraud, some of the rightful owners of the aircraft experienced difficulty in reinstating the aircraft registrations in their name. For example, one owner told federal investigators that he could not fly his aircraft for 2 years because the registration of his aircraft was in the name of the fraudulent broker. Another owner stated that he incurred thousands of dollars in legal fees to reinstate the registration of the aircraft in his name. Additionally, the court ordered the broker to pay approximately $2.4 million in restitution to the bank. In 2014, a U.S.-registered aircraft was seized by and subsequently forfeited to the U.S. government in 2016 because the aircraft had been fraudulently registered and it was purchased with assets derived from wire fraud, money laundering, or other unlawful activities, according to court records associated with this case. The registration was found to be fraudulent because at the time of registration, the applicant was not the true owner of the aircraft. Rather, the U.S. corporation that registered the aircraft acted as a nominee to purchase and register the aircraft on behalf of entities known to have ties to the Sinaloa Cartel, one of the world’s most notorious criminal enterprises. Law-enforcement officials were aware of the scheme and seized the aircraft shortly after final payment was made on it. Court records reveal that this corporation had been previously investigated for violations related to false and fictitious U.S. registration of aircraft on behalf of a criminal organization, and that the corporation’s owner was well known to members of law-enforcement agencies for his suspected role in multiple illegal activities. The aircraft was ultimately forfeited to the U.S. government because it had been purchased with proceeds traceable to illegal activities. In 2012, an intermediary established a U.S. corporation for a foreign national beneficial owner, and the company registered the aircraft. The foreign national was engaged in the black-market currency exchange, which is a common scheme used in trade-based money laundering. In this case, the foreign national conspired with another individual to fraudulently purchase millions of dollars in Venezuela at a rate preferred by the Venezuelan government that was reportedly established as a control to prevent capital flight from Venezuela. Court records show that the aircraft was purchased with illicit proceeds from this fraudulent scheme. In 2016, U.S. law enforcement seized the aircraft, and in 2018 it was forfeited to the U.S. government. In 2017, as the result of a multiyear investigation, the Department of the Treasury’s Office of Foreign Assets Control (OFAC) designated the Executive Vice President of Venezuela as a Specially Designated Narcotics Trafficker pursuant to the Foreign Narcotics Kingpin Designation Act for playing a significant role in international narcotics trafficking. According to the 2017 OFAC announcement on this case, this Venezuelan government official facilitated shipments of narcotics with the final destinations of Mexico and the United States, including control over airplanes and ports used in drug trafficking in Venezuela. According to OFAC, in previous government positions, this official oversaw and partially owned large narcotics shipments destined for the United States. Further, this official also used a front man who laundered drug proceeds and purchased assets. In addition to a network of international companies, according to OFAC, the front man owned or controlled five U.S. companies, including a limited liability company (LLC) that registered an aircraft with FAA using a voting trust to meet U.S. citizenship requirements. As part of its action, OFAC also designated the front man for providing material assistance, financial support, or goods or services in support of the international narcotics trafficking activities of, and acting for or on behalf of, the Venezuelan Executive Vice President. OFAC also identified as blocked property the U.S.-registered aircraft as well as the LLC used to register the aircraft. According to FAA officials, the agency does not have the legal authority to deny a registration solely because of a sanctions designation. OFAC notified FAA of the designation, and FAA flagged the aircraft in its system. FAA deregistered the aircraft in 2019 after registration renewal documentation submitted to FAA contained numerous errors. However, because the flags placed on sanctioned individuals’ and entities’ registration records do not extend to dealer records, FAA issued a dealer certificate to the blocked LLC after the OFAC designation and without coordination with OFAC, according to FAA records and officials. The blocked LLC held the dealer certificate for a year until the certificate expired. In 2011, an aircraft registered to a U.S. citizen with a registered agent address disappeared and was reported to have crashed off the coast of Panama with six fatalities. At the time of the crash, the government of Panama was operating the aircraft while it was still under the U.S. registration of the owner. According to FAA officials and documents we reviewed, the aircraft was in the possession of the Panamanian government because it had been seized by Panamanian authorities in 2010 on allegations that it had been used to traffic narcotics from Panama into Colombia. According to an FAA official knowledgeable about this case, as part of the seizure, a Panamanian court assigned the aircraft to the Panamanian civil aviation authority, which then registered the aircraft in Panama and painted a Panamanian registration number on it. However, the Panamanian civil aviation authority did not take the actions to first deregister the aircraft in the United States, so the new registration was likely invalid under international law. When told this by an FAA official, Panamanian authorities removed the Panamanian registration number from the plane and replaced it with the original N-number. FAA sent multiple letters to the owner to deregister the aircraft and also when the aircraft registration was expiring, but all were returned as refused by the registered agent. According to an FAA official we interviewed about this case, the Panamanian civil aviation authority operated the aircraft under U.S. registration for approximately 1 year until its crash. According to this official, at the time of the crash the aircraft was reportedly operated by the Panamanian civil aviation authority for the purposes of radar maintenance missions in that country. In 2016, an aircraft registered to a U.S.-based LLC crashed in the Caribbean, resulting in fatal injuries to all three people aboard. According to the accident report, the aircraft was operated by a foreign entity, an aviation training center located in Jamaica. The Jamaican civil aviation authority, the entity responsible for investigating the accident, found multiple safety deficiencies as the causes and contributing factors of the fatal crash. These deficiencies include the aircraft’s engine replacement not conforming to its design type; engine parts showing signs of wear ranging from worn to extremely worn conditions exhibiting heavy corrosion; and falsified maintenance records. FAA, by law, imposes safety obligations on all owners of aircraft. To meet these obligations, an owner must maintain current information about the identity and whereabouts of the actual operators of an aircraft and location and nature of the operation on an ongoing basis, thereby allowing that owner to provide the operator with safety-critical information in a timely manner, and to obtain information responsive to FAA inquiries, including investigations of alleged violations of FAA regulations. Such information is an essential element in FAA’s ability to carry out its oversight obligations under U.S. and international law. The safety deficiencies cited in the accident report indicate that, as the registered owner of the aircraft, the LLC may not have been fulfilling its safety obligations. Our objectives were to assess the Federal Aviation Administration’s (FAA) (1) actions to prevent fraud and abuse in aircraft registrations, (2) ability to detect potential fraud and abuse in aircraft registrations, and (3) actions and coordination with law-enforcement entities to respond to aircraft registry–related fraud and abuse risks. To address all objectives, we reviewed laws, regulations, and FAA policies pertaining to the aircraft registration eligibility requirements and processes. We also reviewed standard operating procedures, policy statements, and guidance for staff charged with processing aircraft registrations and addressing administrative compliance actions— including FAA Order 2150.3C issuing enforcement actions per its compliance and enforcement program, FAA Aircraft Examiner’s Guidelines outlining the steps for processing aircraft registrations, and published International Civil Aviation Organization civil aviation standards. We also reviewed prior GAO reports and Department of Transportation (DOT) Office of Inspector General (OIG) reports regarding the quality and utility of registry data, risks, and ongoing challenges associated with the registry’s information technology (IT) system. For all objectives, we interviewed FAA officials from: aircraft registry, legal counsel, FAA’s Security and Hazardous Materials Safety (ASH), FAA’s Law Enforcement Assistance Program (LEAP), and FAA’s Special Emphasis Investigation Team (SEIT). We also interviewed aviation safety, foreign policy, and law-enforcement officials to obtain broader perspectives, where applicable, on the registration process, challenges, and vulnerabilities, including officials from the National Transportation Safety Board (NTSB), the Department of the Treasury’s (Treasury) Office of Foreign Assets Control (OFAC) and Internal Revenue Service Criminal Investigations, the Department of Justice’s (DOJ) Drug Enforcement Administration (DEA), the Department of Homeland Security’s (DHS) Homeland Security Investigations (HSI), and DOT’s OIG. We interviewed aviation industry associations, selected based on a range of aviation interests, such as general aviation and equipment leasing. We also interviewed aircraft registry intermediaries—individuals and entities that facilitate aircraft registrations for others—such as trust companies, banks, and a registered agent, selected based on our analysis of aircraft registry data across types of intermediaries and number of registrations. We also reviewed relevant international standards on countering money laundering and issues related to transparency of corporate structures and beneficial ownership of assets. We performed a descriptive analysis of the registry data from calendar year 2010 through 2018. To do this, we first performed an in-depth review of the calendar year 2018 registry master data—which contains the most-current registration information for our review period—and selected key fields such as aircraft registration number and registrant name information for further analysis. For the remaining calendar years 2010 to 2017 annual files, we focused on identifying any substantive differences occurring between years for the selected key fields. We developed frequencies of the selected key fields to determine the number of registered aircraft, registration types and ownership structures (such as corporations, trusts, and dealers) used to register aircraft, and registration status across the 9-year period of our review. In September 2018 we conducted a site visit to the FAA Registry facility located at the Mike Monroney Aeronautical Center in Oklahoma City, Oklahoma. During the site visit, we interviewed officials from FAA’s major components responsible for processing aircraft registrations and addressing administrative compliance actions, including registry data analysts and managers for the aircraft and airmen systems, FAA ASH officials, and an Office of the Chief Counsel attorney. We also observed firsthand the registry’s process for receiving, sorting, scanning, and recording aircraft registration and renewal application packages. To determine potential fraud and abuse in aircraft registration and FAA actions to prevent them, we analyzed and synthesized a variety of information, including agency reports, registration, postal, and sanctions data, and news articles, among other sources. Our review of information generally spanned fiscal years 2010 through 2018. To identify illustrative cases of potential fraud and abuse, we conducted a literature review that included sources such as Lexis Nexis news articles, DOJ press releases, and investigative reports published by DOT OIG, FAA LEAP, Internal Revenue Service Criminal Investigations, and DHS HSI. We also searched the NTSB publicly available online database of aviation accidents and incidents for examples of safety-related cases. Our literature search yielded over 900 publications and over 300 aviation accident reports for further screening. We then applied two levels of criteria to filter the results for case narrative selections. For the first level, we identified 66 cases from fiscal years 2010 to 2018 involving U.S.- registered aircraft related to three categories of risk enabled by fraud and abuse—criminal activity, national security, and safety. Next, we performed a secondary level of review and selected 28 illustrative cases that included case details, such as entity names and aircraft registration numbers, to facilitate further research including legal review to ensure that selected case studies were adjudicated by a court of law, where applicable. Of those 28 cases, we selected six case studies for in-depth review. We also drew examples from our research of intermediaries of the registry, including selected banks, trust companies, and registered agents. For our in-depth research of these cases, we reviewed available information contained in the FAA Civil Aviation Registry, FAA Electronic Document Retrieval System, and ancillary files; aircraft flight plans; NTSB accident report information; state business registration data; court records; and GAO’s internal resources that included a mix of government and corporate databases. All selected cases are intended for the purpose of illustrating fraud and abuse vulnerabilities associated with the aircraft registration process and may not represent all existing vulnerabilities, nor are they generalizable to the FAA registry population as a whole. To further determine potential fraud and abuse in aircraft registrations, we analyzed FAA aircraft registry address data from calendar year 2018. Using registry address information, we performed a match to United States Postal Service (USPS) data to identify examples of potentially unverified and noncompliant addresses provided to the registry. To analyze postal address data, we used the address fields contained in the FAA registry master and dealer data to verify address information and identify examples of invalid addresses provided to the registry in calendar year 2018, which is the most-current registry data included in our review. Additionally, we obtained data from an internal registry physical address report that we then matched to the calendar year 2018 registry master data to replace mail drop boxes with physical address information, where available. We then performed a match of this updated address file to the USPS Address Matching System as of June 2019 to identify examples of potentially invalid addresses. Our match results revealed a number of commercial mail drop locations, including post office boxes, and addresses that did not match to the postal data. We selected seven aircraft registration addresses and five dealer addresses (total of 12 match results) using a randomized list filtered by locality. We then manually verified the match results for these selected cases using publicly available online geo-mapping tools such as Google Maps and company listings such as White Pages. On the basis of the results of those searches, we selected three aircraft registrations and three dealer certifications that highlight examples of potentially noncompliant addresses provided to the registry in violation of FAA regulations and policy. We conducted subscription database searches and reviewed FAA registration documents for these selected cases based on categories of addresses, such as mail drop boxes, and verified three addresses selected based on locality through site inspections by GAO investigators. Finally, we analyzed the costs associated with aircraft and dealer certificate registrations. To do this, we reviewed an FAA internal report that assessed the costs of FAA’s registration processing, and compared proposed fees to the current fee values for aircraft registrations and dealer certificates. We also reviewed GAO’s federal user fee guide provision that states that fee collections should be sufficient to cover the intended portion of program costs over time, including accounting for factors such as inflation. We reviewed a prior 1993 GAO report in which we determined that the registration fee, in place since 1964, did not cover the cost of reviewing and processing a registration application. Finally, we performed an inflation analysis of the 1964 fee level adjusted for inflation based on the Consumer Price Index. To assess FAA’s ability to detect potential fraud and abuse in aircraft registrations, we examined FAA aircraft registry data collection and storage as well as oversight actions based on registry information and data. We also conducted data mining and matching to identify registrations with indicators of potential fraud or abuse that may enable criminal activity, national security, and safety risks by analyzing FAA aircraft registry master data from calendar years 2010 through 2018, as well as other registry-based and external data sets. We selected five risk indicators, which were informed by interviews with FAA and law- enforcement officials and our background research, for analysis of registry-related data and for matching to a selection of external data sets. We analyzed FAA aircraft registry data to identify registrations with characteristics that matched one or more risk indicators, such as registrations using opaque ownership structures—corporation- and trust- based ownership that disguises the beneficial owner—and registration addresses in countries identified by the Department of State as associated with major illicit drug production and money laundering, among other factors. The risk indicators do not prove fraud or that any unlawful activity has occurred. Alone or together, the risk indicators may serve as points of inquiry for further examination of conduct that may run counter to the interests of the federal government by posing potential criminal, national security, or safety risks. On the basis of the results of our risk-indicator analysis using registry data, we selected a total of five items as potential risk indicators. We selected three risk indicators based on public and internal aircraft registry data. We compared the registry master data to the list of countries published in the latest Department of State narcotics control and financial crimes watch lists. Additionally, we reviewed nonpublic extracts of FAA registry voting trusts used by U.S. citizen corporations and noncitizen trusts from April 2018 through May 2019—the most complete data available at the time of our review—due to their opaque ownership structures and potential for abuse as registration vehicles. We also performed an analysis of types of intermediaries and selected a registered agent as a risk indicator based on confirmed misuse of its address as a means for corporate entities to register aircraft. To establish our population of corporate entities for outreach, we selected four corporate codes contained in the registry data. Next, we developed selection criteria that included geographic distribution (U.S.-based or foreign-based); registrant size based on thresholds that reflect the distribution of registered aircraft (small, medium, or large); and finally, registrant type (bank, trust company, or registered agent). Based on these criteria, we randomly selected two U.S.-based banks and four U.S.- based and foreign trust companies to interview. To identify registered agents, which are not specifically coded in the registry data, we summarized the registry address information and selected all entities with two or more aircraft registrations per address for further screening. We then randomly selected one established registered agent entity for outreach. We analyzed extracts from two external selected data sources for the risk indicator data matching—Treasury OFAC lists of sanctioned entities and individuals, and an NTSB accidents and incidents report—covering the period January 2010 through March 2019, where available. To do this, we used key fields to match the selected data sources to the FAA registry master and trust data, and selected additional risk indicators based on our analysis of the match file results. We matched aircraft registry data to the OFAC lists of sanctioned entities and individuals as of March 2019 to identify aircraft, individuals, and entities subject to U.S. sanctions. We combined five cases identified from our OFAC data match with one additional case identified through our illustrative case and intermediary research to report on our findings of U.S.-sanctioned individuals and aircraft. We included all NTSB-reported accidents and incidents of U.S.- registered aircraft taking place outside the United States as a safety risk indicator. Using the FAA registry aircraft registration number and registrant name fields as the primary match keys, we performed a final merge of all risk indicators identified through our multiple analysis steps described above. Our combined risk flag match returned over 17,000 records, which we used to develop totals for each risk indicator category that we identified. Next, we randomized the list generated from our combined match and applied criteria to filter cases for further review. These criteria included cases with multiple risk indicators, as well as prioritization of risk based on a combined evaluation across all risk indicator categories, among other filters. In total, we selected 20 cases for agency follow-up and in-depth file reviews based on a comprehensive assessment of risk flag categories described above. However, without reviewing a generalizable sample of cases across all categories, we were unable to determine the extent of risk such cases may represent as a proportion of total registrations. Therefore, we used the results of our file reviews for these 20 cases solely to illustrate examples of the risk indicators that we identified. We assessed the reliability of each data set described above for the purposes of generating high-level totals, as well as identifying and tracking potential risk-indicator cases across time. To do this, we performed electronic tests using reports from eight information systems to determine the completeness and accuracy of key fields contained in the data files. We also submitted to the overseeing offices for all eight information systems general data-quality questions regarding the purpose of the data, their structure, definitions and values for selected fields, automated and manual data-quality checks to ensure the accuracy of the data, and limitations. Overall, we found that the data were generally reliable for the purpose of performing a cross-comparison of current registrations associated with safety and compliance violations over the nine-year period of our review. To assess FAA’s actions and coordination with law-enforcement agencies to respond to registration-related risks, in addition to the interviews noted above, we reviewed FAA policies pertaining to the aircraft registration process and documents about FAA and law-enforcement efforts to address registry-related vulnerabilities. We reviewed FAA enforcement actions and government-wide data on aircraft seizures. To generate government-wide totals for aircraft seizures and forfeitures over time, we obtained data extracts from the DOJ Consolidated Asset Tracking System and DHS Customs and Border Protection Seized Assets and Case Tracking System from fiscal years 2010 through 2018. We limited our Consolidated Asset Tracking System data request to aircraft adjudicated as either seized and forfeited, or seized and substituted for cash forfeiture, while the report from the Seized Assets and Case Tracking System contains all seizures recorded by Customs and Border Protection during our review period. Therefore, the reports represent different populations, and we opted to report the totals for the two databases separately. Where feasible, we assessed the reliability of data in each system described above for the purposes of generating high-level totals. Our data-quality testing of selected data elements showed that the primary fields of interest were well-populated and sufficiently reliable for our purposes. We conducted this performance audit from November 2017 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We conducted our related investigative work in accordance with investigation standards prescribed by the Council of the Inspectors General on Integrity and Efficiency. In addition to an aircraft registration application form, evidence of ownership, and $5 registration fee, the Federal Aviation Administration (FAA) requires additional documentation based on the type of individual or entity that owns the aircraft, as discussed in table 2 below. Opaque ownership structures are legitimate business structures that are widely used by corporations and individuals to facilitate commerce as well as for asset and tax management. However, the lack of transparency related to aircraft registrations using opaque ownership structures also creates challenges for safety and law-enforcement investigators seeking information about beneficial owners to support timely investigations. The Financial Action Task Force (FATF) and other international organizations have determined that beneficial ownership information can be obscured through, among other things, the use of shell companies (which can be established with various forms of ownership structures) especially in cases where there is foreign ownership that is spread across jurisdictions; complex ownership and control structures involving many layers of shares registered in the name of other legal entities; formal nominee shareholders and directors where the identity of the beneficial owner is undisclosed; trusts and other legal arrangements that enable a separation of legal ownership and beneficial ownership of assets; and use of intermediaries in forming legal entities, including professional intermediaries. Shell companies, one of the opaque ownership structures, may be formed for legitimate purposes to obtain financing prior to starting operations. In the aircraft ownership context, shell companies may own aircraft by holding title for registration purposes. However, shell companies may also be used to conceal the beneficial owner’s identity for illicit purposes. For example, according to Federal Aviation Administration (FAA) officials, some aircraft registrations have “stacked” company ownership, where shell companies own each other. Such ownership arrangement can be used for illicit purposes to conceal the identity of foreign-based beneficial owners and create challenges for investigators, according to law- enforcement officials. Further, shell companies may use a registered agent’s mailing address on their aircraft application forms, further obscuring aircraft ownership information. Table 3 describes the four opaque ownership structures, their legitimate uses, and how they can be vulnerable to abuse, according to our illustrative case and intermediary research, and interviews with FAA and law-enforcement officials. In the example and figure below, we illustrate opaqueness and complexities of aircraft registrations using intermediaries and opaque ownership structures. It is based on an actual case from our review of aircraft registration documents and research from corporate filings and other databases. Apparent shell company and noncitizen trust used to register aircraft for unknown foreign beneficial owner. In this case, a foreign company obtained U.S. aircraft registration through an intermediary, using opaque ownership structures. This is allowable under current registration requirements and there is no identified wrongdoing in this case. The application, depicted in figure 13, shows the involvement of an intermediary, who used various legal entities and took a number of steps to facilitate aircraft registration for a beneficial owner who is unknown. The intermediary listed himself as the director of a corporation, N003 Inc., which was established using a company that provides company formation and registered agent services. Among other indicators, N003 Inc. appeared to be a shell company established shortly before the filing of the aircraft registration. The intermediary also used the mailing address of the registered agent as the owner’s address on the aircraft registration application. Further, the intermediary established a noncitizen trust for aircraft ownership. The trust agreement identified N003 Inc. as the owner trustee of the aircraft, and a foreign corporation, DEF Ltd., as the trustor. As such, the role of the intermediary, the use of apparent shell company and noncitizen trust ownership structures, and use of the registered agent’s mailing address worked to obscure the foreign beneficial owner of the aircraft while facilitating access to U.S. aircraft registration. Rebecca Shea, (202) 512-6722 or shear@gao.gov In addition to the contact named above, Tonita Gillich (Assistant Director), Irina Carnevale (Analyst-in-Charge), James Ashley, Priyanka Sethi Bansal, Gary Bianchi, Daniel Bibeault, Kimberley Bynum, Steven Campbell, Colin Fallon, Robert Graves, Ying Long, Olivia Lopez, Maria McMullen, James Murphy, George J. Ogilvie, Sean Peck, and April Van Cleef made key contributions to this report.", "summary": "The U.S. aircraft registry, managed by FAA, maintains information on approximately 300,000 civil aircraft. FAA issues aircraft registration to individuals and entities that meet eligibility requirements, such as U.S. citizenship or permanent legal residence. Registry fraud and abuse hinders the ability of law-enforcement and safety officials to use the registry to identify aircraft and their owners who might be involved in illicit or unsafe operations. GAO was asked to examine registry fraud and abuse. This report assesses FAA's actions to (1) prevent, (2) detect, and (3) respond to fraud and abuse risks in aircraft registrations. GAO reviewed relevant laws, regulations, and FAA policies; reviewed reports, DOJ press releases, and court cases that illustrated risks associated with the registry; analyzed aircraft registry data from fiscal year 2010 through 2018 to identify registrations with risk indicators; and interviewed FAA registry, legal, law-enforcement liaison, and safety officials, as well as officials from DOJ and DHS. To register civil aircraft, the Federal Aviation Administration (FAA) generally relies on self-certification of registrants' eligibility and does not verify key information. According to GAO's review of the registry process, there are risks associated with FAA not verifying applicant identity, ownership, and address information. The registry is further vulnerable to fraud and abuse when applicants register aircraft using opaque ownership structures that afford limited transparency into who is the actual beneficial owner (i.e., the person who ultimately owns and controls the aircraft). Such structures can be used to own aircraft associated with money laundering or other illegal activities (see example in figure). FAA has not conducted a risk assessment that would inform its eligibility review and collection of information to manage risks. Without a risk assessment, FAA is limited in its ability to prevent fraud and abuse in aircraft registrations, which enable aircraft-related criminal, national security, or safety risks. FAA makes some use of registry information to detect risks of fraud and abuse, but the format of the data limits its usefulness. Specifically, most data on individuals and entities with potentially significant responsibilities for aircraft ownership, such as trustors and beneficiaries, are stored in files that cannot be readily analyzed due to system limitations. As FAA modernizes its information-technology systems, it has an opportunity to develop data analytics capabilities to detect indicators of fraud and abuse in the registry. FAA takes administrative actions, such as registration revocations, to respond to registration violations and coordinates with law-enforcement agencies on investigations and enforcement actions such as aircraft seizures. Since 2017, FAA has coordinated with the Departments of Justice (DOJ) and Homeland Security (DHS) as part of an Aircraft Registry Task Force to address aircraft registry vulnerabilities. However, this coordination is informal, and other mechanisms for joint enforcement actions, sharing of information, and use of liaison positions are not in place, GAO is making 15 recommendations to FAA, including that it collect and verify key information on aircraft owners; undertake a risk assessment of the registry; leverage information-technology modernization efforts to develop data analytics approaches for detecting registry fraud and abuse; and formalize coordination mechanisms with law-enforcement agencies. FAA agreed with all recommendations.", "document_type": "gao"}
{"report": "The Dodd-Frank Act was enacted to promote the financial stability of the United States by improving accountability and transparency in the financial system and protecting consumers from abusive financial services practices, among other purposes. To help detect and prevent securities misconduct, section 961 of the Dodd-Frank Act promotes complete and consistent performance of SEC staff examinations, investigations and reviews, and appropriate supervision of these activities through internal supervisory controls. SEC has submitted eight annual reports to Congress under section 961, all of which stated that both its internal supervisory controls and its staff procedures were effective for the period under review. In addition, all such reports stated that no significant deficiencies in internal supervisory controls were identified. Section 961 does not define “internal supervisory control.” SEC has defined internal supervisory controls as the processes established by management to monitor that the procedures applicable to staff (that is, established day-to-day procedures to be followed by the employees within the applicable programs) are consistently being performed according to policy and procedures, and also remain reasonable, adequate, and current. SEC is the primary regulator of the U.S. securities markets and is responsible for protecting investors, maintaining fair, orderly, and efficient markets, and facilitating capital formation. To fulfill this mission, SEC requires public companies to disclose meaningful financial and other information to the public, examines firms it regulates, and investigates potential violations of the federal securities laws. SEC is organized into five divisions and 24 offices. SEC’s approximately 4,400 staff are located in Washington, D.C., and in 11 regional offices. As discussed previously, four divisions and offices are subject to section 961 of the Dodd-Frank Act (see table 1). SEC formalized its Section 961 Working Group in 2017. The primary purposes of the Working Group are to enhance the efficiency and effectiveness of SEC’s processes related to section 961 compliance and to enhance coordination and information sharing among the divisions and offices. The Working Group is a staff-level group comprising one or more representatives from each of the four divisions and offices subject to section 961 as well as the Office of the Chief Operating Officer. These staff are responsible for carrying out the Working Group’s responsibilities, which include establishing a common understanding and consistent approach to compliance; creating a means to share information and ideas to improve the efficiency and effectiveness of section 961 compliance activities; discussing best practices to streamline procedures and documentation of internal control testing and reporting; and developing and updating guidance related to implementing section 961. Standards for Internal Control in the Federal Government provides the overall framework for establishing and maintaining internal control in federal agencies. Agency management is responsible for adapting the framework for an agency. Furthermore, an agency may use the framework to organize its development and implementation of internal controls and implement its standards throughout the agency or at an office level. Five interrelated components and associated principles establish requirements for developing and maintaining an effective internal control system: Control environment: The control environment is the foundation for an internal control system. It provides discipline and structure, which affect the overall quality of internal control. It influences how objectives are defined and control activities are structured. The oversight body and management establish and maintain an environment throughout the entity that sets a positive attitude toward internal control. Risk assessment: Management assesses the risks facing the entity as it seeks to achieve its objectives. This assessment provides the basis for developing appropriate risk responses. Management assesses risks the entity faces from external and internal sources. Control activities: Control activities are the actions management establishes through policies and procedures to achieve objectives and respond to risks in the internal control system, which includes the entity’s information system. Information and communication: Management uses quality information to support the internal control system. Effective information and communication are vital for an entity to achieve its objectives. Entity management needs access to relevant and reliable communication related to internal and external events. Monitoring: Internal controls are dynamic and have to be adapted continually to risks and changes an entity faces. Monitoring the internal control system is essential in helping internal control remain aligned with changing objectives, environment, laws, resources, and risks. Internal control monitoring assesses the quality of performance over time and promptly resolves the findings of audits and other reviews. Corrective actions are a necessary complement to control activities to achieve objectives. To be effective, an agency’s internal control system must incorporate the five components of internal control in an integrated manner throughout its operations and on an ongoing basis. Once in place, internal control provides reasonable, not absolute, assurance of meeting agency objectives. When evaluating the design of internal control, management determines if controls individually and in combination are capable of achieving an objective and addressing related risks. To the extent a control does not fully achieve an objective or address related risks, it is deficient, and such deficiencies may be associated with a control’s design or operation. A deficiency in design exists when a control necessary to meet a control objective is missing, or an existing control is not properly designed so that even if the control operated as designed, the control objective would not be met. A deficiency in operation exists when a properly designed control does not operate as designed or the person performing the control does not possess the necessary authority or competence to perform the control effectively. In addition to the requirements under section 961 of the Dodd-Frank Act, SEC must establish and maintain effective internal control and financial management systems that meet the objectives of the Federal Managers’ Financial Integrity Act of 1982 (FMFIA). FMFIA requires agencies to annually assess and report on the internal controls that protect the integrity of their programs and whether financial management systems conform to related requirements. In addition, FMFIA requires agencies to provide an assurance statement regarding the effectiveness of the agency’s internal controls. SEC’s internal controls for financial management systems are not included in this report because they are reported in our annual financial audit of SEC. In addition, all of SEC’s internal controls—including those which constitute internal supervisory controls—are in scope for FMFIA. In response to section 961 of the Dodd-Frank Act, the Working Group put in place a framework that provides guidance for division and office staff responsible for assessing the effectiveness of internal supervisory controls (control framework). The control framework draws on external sources such as federal internal control standards as well as internal documents such as SEC’s Reference Guide for Compliance with Section 961 of the Dodd-Frank Act, the Risk Management and Internal Control Review Reference Guide from the Office of the Chief Operating Officer, and the charter for the Working Group. These internal documents include definitions, criteria, and other guidance and together compose SEC’s control framework. For example, the control framework includes time frames for when divisions and offices should assess their internal supervisory controls and report findings to Congress (see fig. 1). SEC’s control framework consists of three phases—risk assessment, internal supervisory control testing, and communication of results—during which division and office staff conduct activities to systematically assess and report on the effectiveness of their internal supervisory controls (see fig. 2 for examples). Changes to SEC’s control framework since our last review (which focused on fiscal years 2013–2015) include revisions to key guidance documentation and reclassification of some controls (as nonsupervisory controls). The Working Group revised elements of its control framework documentation since our last review. First, the Working Group streamlined the Reference Guide for Compliance with Section 961 by removing direct guidance—for example, steps staff should take to assess the design and operation of internal supervisory controls—and replaced it with references to the Risk Management and Internal Control Review Reference Guide. Second, the Working Group also updated other information such as the agency’s definition for internal supervisory control. Third, some divisions and offices changed which controls they considered to be internal supervisory controls subject to section 961 assessments. As stated previously, SEC defines internal supervisory controls as the processes established by management to monitor that procedures applicable to staff (the established day-to-day procedures to be followed by the employees within the applicable program) are consistently being performed according to policy and procedures, and also remain reasonable, adequate, and current. Division and Office officials elaborated further, stating they only consider controls that are supervisory in nature and directly related to the consistent and complete execution of examinations of registered entities, enforcement investigation, or reviews of corporate financial securities filings to be internal supervisory controls relevant to section 961. More specifically, OCIE reduced the number of controls it classified as internal supervisory controls from 40 in fiscal year 2015 to 14 in fiscal year 2018 by reclassifying some controls as nonsupervisory controls and by consolidating others (see table 2). For example, OCIE no longer classifies examination program strategy and selection controls (such as development and dissemination of examination program goals) as internal supervisory controls. Therefore, the controls are no longer assessed under section 961. OCIE officials explained that the strategy and selection of controls are performed by management and related to the selection of registrants for examinations, and not to staff conducting examinations consistently with professional competence and integrity. Similarly, the number of internal supervisory controls Corporation Finance maintained decreased from 25 in fiscal year 2015 to eight in fiscal year 2018. Corporation Finance officials told us that they determined that certain controls previously considered relevant to section 961 did not represent processes that fall within the core function of reviewing corporate financial securities filings and thus should not be considered internal supervisory controls under section 961. Enforcement maintained 25 internal supervisory controls from fiscal year 2015 to fiscal year 2018, while OCR had 13–14 internal supervisory controls during the same period. As of the end of fiscal year 2018, SEC’s control framework continued to reflect key components of internal control. We compared the framework against federal internal control standards. Specifically, we assessed whether the control framework reflected the five components of internal control—control environment, risk assessment, control activities, information and communication, and monitoring. We determined that SEC’s control framework included attributes covering each of the components. For example, the framework included oversight structures to monitor the design and operation of division and office internal supervisory controls, assigned responsibilities to division and office staff, incorporated steps for staff to follow to assess risks and test internal supervisory controls, and included mechanisms to correct deficiencies and report findings to internal and external stakeholders (such as Congress). See table 3 for additional examples that illustrate how the control framework reflected relevant standards. Divisions and offices have not developed written policies and procedures to ensure that they systematically assess the effectiveness of procedures applicable to staff who perform examinations of registered entities, enforcement investigations, and reviews of corporate financial securities filings. As mentioned previously, the report required under section 961 of the Dodd-Frank Act must include an assessment of the effectiveness of both internal supervisory controls and staff procedures. Division and office officials told us that they used findings and conclusions from their internal supervisory control assessments to support their conclusions that staff procedures were effective. As discussed earlier, SEC defines internal supervisory controls to include two types of processes used by managers: (1) those used to monitor whether staff follow existing procedures and (2) those used to monitor whether the procedures remain reasonable, adequate, and current. We found that SEC’s assessments of internal supervisory controls did not directly assess the effectiveness of staff procedures for three primary reasons. First, the controls included in SEC’s assessment generally consist of processes that monitor whether staff follow existing procedures, not processes that monitor whether the procedures remain reasonable, adequate, and current. Second, SEC’s assessments of internal supervisory control focus on evaluating the extent to which managers executed the controls for which they are responsible. Although the controls monitor whether staff follow underlying procedures, the control assessments do not directly address whether those underlying staff procedures are designed to effectively achieve their stated objectives (for example, identifying and mitigating securities misconduct by securities market participants). Lastly, documentation of division and office internal supervisory control assessments did not speak to how division and office staff reached conclusions that procedures applicable to staff were effective. In addition to findings from internal supervisory control assessments, SEC officials also told us about policies and procedures, compliance testing, and other activities that provide information regarding the effectiveness of staff procedures. Corporation Finance officials further elaborated by stating that there is no single or discrete assessment to test the effectiveness of staff procedures. Rather, the officials explained that the division relies on activities performed throughout the year that contribute to the evaluation of the effectiveness of staff procedures. Examples of activities all or some divisions and offices referenced included the following: Enforcement, Corporation Finance, OCIE, and OCR officials told us that senior management from each division or office monitor the effectiveness of their programs throughout the year to help assess the effectiveness of staff procedures. Examples of monitoring activities include discussions with staff and subject-matter experts who perform examinations of registered entities, enforcement investigations, and reviews of corporate financial securities filings. OCR and Corporation Finance provided examples of documentation for these activities. Enforcement, Corporation Finance, OCIE, and OCR provided documentation that showed they developed review teams, task forces, projects, or initiatives that review specific policies or risks, which can result in updates to procedures. Corporation Finance, OCIE, and OCR officials told us that they have implemented reviews and redesigns of their policies and procedures through periodic reviews of their respective program manuals. See table 4 below for additional examples of activities that divisions and offices referenced as assessments of the effectiveness of staff procedures. The activities mentioned above could provide valuable information for staff who perform examinations of registered entities, enforcement investigations, and reviews of corporate financial securities filings, but they do not represent systematic assessments for the purposes of section 961. In particular, these activities varied between divisions and offices, mostly were implemented on an irregular basis, and were not established through written policies or procedures. In addition, none of the divisions and offices provided documentation linking the results of these, or any other, activities to the conclusions in SEC’s annual reports to Congress under section 961, each of which have stated that SEC’s staff procedures were effective for the period under review. Furthermore, only Corporation Finance officials told us that they discuss the effectiveness of staff procedures with their Director when they present their annual internal supervisory control assessment findings. As stated previously, the control framework includes an oversight structure, timelines, evaluation criteria, and documentation requirements, and SEC considers its control assessments under the framework to represent assessments of the effectiveness of staff procedures. However, SEC has not developed detailed policies, procedures, or guidance for assessing the effectiveness of staff procedures for the purposes of section 961. For example, the activities that divisions and offices referenced as assessing the effectiveness of staff procedures were not established through written policies for section 961-reporting purposes. And, existing guidance documents such as the Reference Guide for Compliance with Section 961 do not include steps or documentation requirements for assessing staff procedures. Federal internal control standards state the importance for agency management to establish policies and procedures to achieve objectives. Because divisions and offices lack written policies and procedures for assessing the effectiveness of staff procedures, each uses informal methods and varied processes instead of a systematic approach that document how each division and office reached its conclusions (that staff procedures were effective) in SEC’s annual section 961 report to Congress. Establishing written policies and procedures for systematically assessing the effectiveness of staff procedures would provide SEC with greater assurance that the procedures were effective in the context of section 961 and would help divisions and offices meet objectives. To evaluate the extent to which SEC’s internal supervisory controls met federal internal control standards and SEC guidance, we evaluated a non-generalizable sample of internal supervisory controls. We assessed whether (1) controls were designed to address objectives and respond to risks and (2) control activities were implemented through policies. We discuss below our findings related to the 39 internal supervisory controls that SEC identified as related to section 961. See appendix II for an example of the template we used to evaluate the controls. All 39 internal supervisory controls that we evaluated incorporated design elements to achieve SEC’s control objectives and respond to risks that SEC identified. We assessed the overall design of selected internal supervisory controls against four design elements identified in federal internal control standards: Control activities should respond to identified objectives and risks, Appropriate types of control activities should be used, Control activities should be designed at the appropriate levels of the organization (Director, Assistant Director, Branch Chief, etc.), and Control activity duties should be segregated where practical. We found that, for the selected controls, each division and office designed control activities to respond to identified objectives and risks by identifying the risks addressed by each control and the control objective (how a control will address the associated risk) in their risk and control matrixes. In their risk and control matrixes, the divisions and offices also have established characteristics identified by relevant standards as important for designing appropriate controls, including the control frequency, control owner, and whether a control is automated or manual, preventive or detective, and key or secondary. To ensure that control activities are designed at the appropriate levels, each division and office identified control owners in their risk and control matrixes and in the control descriptions they identified the job title of staff responsible for executing the controls. Finally, the divisions and offices segregated control duties in cases in which the need for such segregation was apparent. For example, a second review by a higher-level official was included in some controls that required approval decisions. For the results of our control design assessments, see appendix III. Ten of the 39 controls we evaluated lacked key information needed to help ensure execution of the control activities (see table 5). Federal internal control standards state that documentation is required for the effective design, implementation, and operating effectiveness of an entity’s internal control system, including documentation of internal control responsibilities through policies. We assessed SEC’s documented control activities against three key attributes identified in federal internal control standards: Establishment of procedures to support control execution, Assignment of responsibility for control execution, and Establishment of time frames for control execution. Two or three of the selected controls from each division and office did not incorporate key execution attributes, as seen in table 5. For the results of our control design assessments, see appendix III. Descriptions for many control activities did not specify procedures to be performed or, in some cases include time frames, but all controls we assessed assigned responsibility for control execution (see table 6). More specifically, 10 of the 39 controls had no requirement to document execution of the control activities. For example, one Enforcement control and two Corporation Finance controls intended to monitor compliance with timeliness metrics did not include a requirement to document whether the control activities had been executed—that managers completed the review of the timeliness reports, noted if any cases were nearing the time frame threshold, or took appropriate actions in response. In addition, three of the 39 controls we reviewed did not include the control activity attribute of follow-up actions to be taken. For example, the Corporation Finance timeliness controls discussed above also did not establish follow-up actions for cases in which a team or individual neared the timeliness threshold. Follow-up actions could include emailing or calling relevant staff when a timeliness threshold was within a certain number of days of being breached. The divisions and offices did not establish operational procedures for how the control activities would be performed in three of the 39 controls we reviewed. For example, an OCIE control intended to track enrollment and completion of new examiner training lacked underlying procedures for identifying or tracking training progress of new employees. The divisions and offices did not establish time frames for executing control activities in three of the 39 controls we reviewed. For example, while the Corporation Finance timeliness controls discussed above identified the reports to be reviewed, one of the two controls did not specify when the reports should be reviewed. By not incorporating key control attributes into their control activities, SEC may not have reasonable assurance that internal supervisory controls are effectively implemented. Some of the controls with weaknesses in one or more of the control attributes lacked documentation of the controls’ execution, which hindered our ability to test whether the controls operated as intended, as discussed in the next section. For example, two of the timeliness controls for Corporation Finance, described above, did not include a documentation requirement, and no documentation of control execution was created. In lieu of reviewing documentation of control execution, for SEC’s assessment of the effectiveness of its internal supervisory controls, the divisions and offices asked supervisors twice a year (by email) whether they had executed this control weekly over the course of the year. Staff from some divisions and offices said the reason that control activity attributes were not included in some of the controls was because policies and procedures had been long established and orally communicated, but not written into the control activities. Based on Standards for Internal Control in the Federal Government, SEC developed a reference guide to provide guidance for identifying, documenting, and monitoring controls. The reference guide states that internal control activities should be written to describe the actual activities performed to meet the control objective, and at a minimum, identify control procedures and how they are to be executed, establish a documentation requirement for control execution, and assign responsibility and establish time frames for control execution. Following SEC guidance for developing control activities could help divisions and offices ensure evidence exists of control execution and better enable control monitoring by SEC, and oversight by external parties, such as GAO and the SEC Inspector General. In turn, better control monitoring would help ensure that SEC’s internal supervisory controls are effectively implemented and that procedures necessary to achieve organizational objectives are followed. Furthermore, enhancing control activity descriptions would provide SEC greater assurance that staff have the information necessary to effectively implement the controls. We selected 18 of 39 internal supervisory controls across the four divisions and offices to assess whether they operated as intended in fiscal year 2018. (See figure 3 for an overview of how we determined they operated as intended, partially operated as intended, or did not operate as intended.) As an example of how we conducted these assessments, we reviewed one OCIE control that called for manager approval at three points of an examination and additional assistant director approval to close the examination, as described in OCIE’s control documentation. To assess whether this control operated as intended, we selected and reviewed a random, generalizable sample of examinations in OCIE’s internal system to determine whether all of the control’s activities—in this case, management approvals—had been executed. We could not assess some of the controls we selected because SEC did not provide sufficient documentation to allow us to determine whether the control operated as intended. For example, two of four Corporation Finance controls did not include a documentation requirement for weekly monitoring of staff compliance with internal policy. As a result, documentation did not exist for us to assess whether supervisors executed these control activities throughout the year. For more information on how we determined whether controls were operating as intended, see appendix I. Of the 15 controls we could assess, 13 operated as intended and two partially operated as intended (see table 7). We could not assess three controls because sufficient documentation was not provided. More specifically, a control documentation requirement was not established for the three controls—as identified through our assessment of the control’s design, described earlier. We determined that two OCIE controls partially operated as intended. For example, while we found that 20 percent of sampled OCIE examinations were not approved within the designated deadline, all examinations were closed and included all required elements (see table 8). We were unable to assess three of 18 selected controls because the divisions and offices did not provide sufficient documentation on the execution of control activities (see table 9). We found these controls lacked a documentation requirement for control execution in their control activity descriptions and did not produce sufficient documentation, which prevented us from determining whether these controls operated as intended. For example, two of four Corporation Finance controls did not include a requirement to document execution of the control activity— weekly monitoring of staff compliance with internal policy. Because these controls did not produce documentation of weekly monitoring throughout the year as prescribed in the control activity frequency, we did not receive documentation to allow us to assess whether supervisors executed these control activities on a weekly basis or, in some cases, at all. Additionally, we could not assess one selected OCIE control involving tracking of new employee training. To help detect and prevent securities misconduct, section 961 of the Dodd-Frank Act requires SEC to assess the effectiveness of both its internal supervisory controls and the procedures applicable to staff who perform examinations of registered entities, enforcement investigations, and reviews of corporate financial securities filings. While SEC has established a framework for systematically assessing the effectiveness of its internal supervisory controls, it has not established a framework for systematically assessing the effectiveness of staff procedures or documenting how SEC reached related conclusions about the procedures in its annual reports to Congress under section 961. Creating written policies and procedures to systematically assess the effectiveness of staff procedures and documenting the results of such assessments would provide SEC with greater assurance that the staff procedures are effective, a key objective of section 961. Every control we reviewed incorporated design elements to achieve SEC’s control objectives and respond to risks that it identified. However, nine of the 39 controls did not incorporate one or more key attributes that would help ensure execution of the control, including documentation requirements, detailed procedures, identification of follow-up actions, assignment of responsibility for control execution, and time frames for control execution. Following SEC guidance for developing detailed control activities could help divisions and offices ensure evidence of control execution and better enable control monitoring by SEC and external parties, such as GAO and the SEC Inspector General. In turn, better control monitoring would help ensure that SEC’s internal supervisory controls are effective and that procedures necessary to achieve organizational objectives are followed. Furthermore, enhancing control activity descriptions would provide SEC greater assurance that staff have the information necessary to effectively implement the controls. We are making the following five recommendations to SEC. The SEC Chair should direct the Directors of the Division of Corporation Finance, Division of Enforcement, Office of Compliance Inspections and Examinations, and Office of Credit Ratings to develop written policies and processes to systematically assess the effectiveness of staff procedures (procedures applicable to staff who perform examinations of registered entities, enforcement investigations, and reviews of corporate financial securities filings). Examples of elements SEC could include in the policies and processes are the steps necessary to conduct such assessments, including time frames in which the assessments should be performed and reviewed; assignment of responsibilities related to the assessments; requirements for documenting assessments; and steps for staff to take to mitigate and report deficiencies identified as a result of the assessments. (Recommendation 1) The Director of the Division of Corporation Finance should ensure that all internal supervisory controls include documentation requirements, detailed procedures, identified follow-up actions, implementation time frames, and assignment of control execution responsibility, in accordance with SEC guidance and federal internal control standards for implementing control activities through documented policies. (Recommendation 2) The Director of the Division of Enforcement should ensure that all internal supervisory controls include documentation requirements, detailed procedures, identified follow-up actions, implementation time frames, and assignment of control execution responsibility, in accordance with SEC guidance and federal internal control standards for implementing control activities through documented policies. (Recommendation 3) The Director of the Office of Compliance Inspections and Examinations should ensure that all internal supervisory controls include documentation requirements, detailed procedures, identified follow-up actions, implementation time frames, and assignment of control execution responsibility, in accordance with SEC guidance and federal internal control standards for implementing control activities through documented policies. (Recommendation 4) The Director of the Office of Credit Ratings should ensure that all internal supervisory controls include documentation requirements, detailed procedures, identified follow-up actions, implementation time frames, and assignment of control execution responsibility, in accordance with SEC guidance and federal internal control standards for implementing control activities through documented policies. (Recommendation 5) We provided a draft of this report to SEC for review and comment. In written comments (reproduced in appendix VI), SEC agreed with our findings and concurred with our recommendations. In addition, SEC provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Chair of SEC, and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-8678 or clementsm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. This report focuses on activities that fall within the purview of the Division of Corporation Finance (Corporation Finance), Division of Enforcement (Enforcement), Office of Compliance Inspections and Examinations (OCIE), and Office of Credit Ratings (OCR) at the Securities and Exchange Commission (SEC)—to which we refer collectively as the divisions and offices. We examined (1) the extent to which SEC’s internal supervisory control framework during fiscal years 2016–2018 reflected federal internal control standards; (2) how SEC evaluated the effectiveness of staff procedures in fiscal year 2018; (3) the extent to which selected controls in fiscal year 2018 were designed consistent with relevant standards; and (4) the extent to which selected controls operated as intended in fiscal year 2018. For our first objective, we obtained and reviewed relevant documentation on SEC’s internal supervisory control framework for fiscal years 2016– 2018 and interviewed division and office staff responsible for developing and updating the framework. We then assessed this framework against Standards for Internal Control in the Federal Government and determined the extent to which the framework reflected these standards. Specifically, we assessed the framework against the five components of internal control—control environment, risk assessment, control activities, information and communication, and monitoring—and the 17 principles associated with these components. We compared information on changes SEC made to its internal supervisory control framework with information from our previous review and federal internal control standards to determine the extent to which the framework continued to reflect internal control standards. For our second objective, we reviewed policies, procedures, and guidance documents (for fiscal year 2018) relating to SEC assessments of the effectiveness of procedures applicable to staff who perform examinations of registered entities, enforcement investigations, and reviews of corporate financial securities filings. We also interviewed SEC staff to obtain an understanding of the steps and activities that divisions and offices take to assess the effectiveness of staff procedures. We intended to assess how SEC assessed staff procedures to determine the extent to which SEC’s assessments reflected federal internal control standards. However, as discussed in the report, we found SEC did not have a framework for assessing the effectiveness of staff procedures. We therefore examined policies, procedures, and guidance, but did not assess them against the components and principles associated with the federal standards for internal control. For our third objective, we used the policies, procedures, and control objectives to determine if the design of selected division and office internal supervisory controls in place during fiscal year 2018 was consistent with federal internal control standards and SEC guidance for designing internal controls. We developed an evaluation template and used it to assess selected controls from each division and office by having multiple analysts conduct independent reviews and then reached a final consensus by conducting a joint review with the same analysts. We used Standards for Internal Control in the Federal Government and The Committee of Sponsoring Organizations of the Treadway Commission’s Internal Control – Integrated Framework to develop our template. We also reviewed documents and interviewed staff to obtain a thorough understanding of the internal supervisory controls used to oversee the processes for conducting examinations of registered entities, enforcement investigations, and reviews of corporate financial securities filings. We selected for our review a non-generalizable sample of 53 controls in place during fiscal year 2018—13 controls in Corporation Finance, 15 controls in Enforcement, 11 in OCIE, and 14 in OCR. We grouped these controls into sets because some underlying staff processes had multiple associated controls. In cases in which we selected a control that was part of a set, we would review every control in the associated set. We selected controls and control sets that SEC designated as being associated with processes that have the highest risk or potential impact on achieving stated objectives until we reached our target of 10–15 controls per division or office. Some control sets also contained controls that were not related to section 961. Therefore, to fully assess complete control sets associated with underlying processes, our selection contained some controls that were not related to section 961. However, in this report we only discuss and include analysis for those controls that SEC identified as related to section 961, which comprises 39 controls—eight in Corporation Finance, 10 in Enforcement, eight in OCIE, and 13 in OCR. For our fourth objective, we developed an evaluation template for each control and conducted independent primary and secondary reviews to reach a final consensus on the operation of each control. The template was created using SEC’s control activities and related policy and procedural documents we received as part of our design assessment. We used the template to determine the extent to which the execution of controls met the design criteria. Depending on the extent to which they met criteria established from control design documents, the selected controls were grouped under one of the following categories: (1) operated as intended, (2) partially operated as intended, (3) did not operate as intended, and (4) could not be assessed because control documentation did not exist due to design weaknesses, was not received, or was not relevant. Because the nature of controls varied, we evaluated controls by applying the factors below in conjunction with professional judgment. We focused on whether deficiencies would affect the implementation and operation of controls. For controls that operated as intended, we determined that the divisions and offices provided documentation demonstrating that all control activities were executed for the instances of control implementation we reviewed. We considered controls to have partially operated as intended if the documentation provided supported that only some control activities were executed or if at least one control activity did not operate as intended, but the overall control was executed for most instances. We did not identify any controls that did not operate as intended. This determination would have applied to controls for which we received sufficient documentation to assess the control’s operation and for which the divisions and offices did not execute all control activities in most instances. For controls that we could not assess, we did not receive sufficient documentation that would enable us to make a determination of whether the control was executed or operated as intended. For these controls, we also used the results of our design assessments to determine whether the controls included a documentation requirement that would enable us to assess whether they operated as intended. We judgmentally selected a non-generalizable sample of 18 controls across all four divisions and offices from the population of 39 internal supervisory controls we reviewed in the third objective. We selected these controls based on factors such as whether they were classified as key to achieving objectives, high-risk, or having high potential impact on achieving stated objectives or likelihood of failure. We then created a generalizable, random sample of cases to review for eight controls, and we reviewed all instances for the remaining controls because they occurred annually or had few instances. In some cases, we conducted on-site testing in which we assessed samples of cases for controls by demonstrations of the divisions and offices’ internal systems. We conducted this performance audit from October 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix illustrates the template we used to assess the design of selected controls from each division and office that we reviewed at the Securities and Exchange Commission (SEC). For each control, we reviewed policies, procedures, and control objectives to determine if the design of the selected internal supervisory controls was consistent with federal internal control standards and SEC guidance for designing internal controls. To assess the extent to which design of the Securities and Exchange Commission’s (SEC) internal supervisory controls was consistent with federal internal control standards and SEC guidance for designing internal controls, we reviewed 39 internal supervisory controls across the four divisions and offices in place during fiscal year 2018. We used the policies, procedures, and control objectives to determine if the controls’ designs were consistent with the standards and guidance. This appendix illustrates the template we used to assess the operation of selected Securities and Exchange Commission internal supervisory controls. For each control, we compared control activity descriptions, including policy and procedure documents to determine whether selected controls operated as intended. As part of our review, we tested 18 internal supervisory controls across four divisions and offices at the Securities and Exchange Commission (SEC) to determine whether they operated as intended. Controls were assessed using SEC’s control activity descriptions, including related policy and procedure documents. For controls that operated as intended, SEC provided documentation demonstrating that all control activities were executed. We considered controls to have partially operated as intended if the documentation supported that only some control activities were executed or if at least one control activity did not operate as intended, but the overall control was executed. We did not identify any controls that did not operate as intended, but this would have applied to controls for which we received sufficient documentation and the divisions and offices did not execute all control activities. Controls that we could not assess lacked sufficient documentation that would have enabled us to determine whether they operated as intended. In addition to the contact named above, Kevin Averyt (Assistant Director), Christopher Ross (Analyst in Charge), Aaron A. Colsher, Justin Fisher, Efrain Magallan, Marc Molino, Kirsten Noethen, Barbara Roesmann, and Farrah Stone made key contributions to this report.", "summary": "Section 961 of the Dodd-Frank Wall Street Reform and Consumer Protection Act directs SEC to assess and report annually on internal supervisory controls and procedures applicable to staff performing examinations, investigations, and securities filing reviews. The act also contains a provision for GAO to report on SEC's internal supervisory control structure and staff procedures. GAO's last report was in 2016 ( GAO-17-16 ). This report examines SEC's internal supervisory control framework and assessment of staff procedures, the design of selected controls, and the operation of selected controls. GAO analyzed SEC's internal supervisory control framework and related policies and guidance and evaluated the design and execution of a non-generalizable sample of controls selected because they addressed high-risk processes. As of fiscal year 2018, the Securities and Exchange Commission's (SEC) internal supervisory control framework—which provides guidance for division and office staff responsible for assessing the effectiveness of internal supervisory controls —reflected federal internal control standards. GAO determined that SEC's framework included elements covering each of the five components of internal control—control environment, risk assessment, control activities, information and communication, and monitoring. However, SEC does not have written policies or guidance to ensure that relevant SEC divisions and offices systematically assess the effectiveness of procedures applicable to staff who perform examinations of registered entities, enforcement investigations, and reviews of corporate securities filings. Establishing such policies would provide SEC greater assurance that these procedures are effective at achieving their objectives. All the SEC controls GAO evaluated were designed consistent with standards, and a majority operated as intended. SEC guidance and federal internal control standards state that (1) controls should be designed to address objectives and respond to risks and (2) control activities should be implemented through policies, including documentation requirements, and include detail to enable management to monitor control execution. Control design. All 39 controls GAO evaluated included design elements to achieve SEC's control objectives and respond to risks it identified. However, 10 of these 39 controls did not include key attributes, such as requirements to document, and set time frames for, control execution (see fig.). Control operation. GAO could not assess the operation of three of 18 selected controls because documentation of control execution did not exist. Of the remaining controls, 12 operated as intended and three partially operated as intended. Examples of controls that operated as intended include SEC's approval of examinations and tracking of investigations. By more consistently following SEC guidance and federal internal control standards for developing control activities, including documentation requirements, relevant SEC divisions and offices would enhance their ability to monitor and ensure the effectiveness of their internal supervisory controls. Legend: Corporation Finance = Division of Corporation Finance; Enforcement = Division of Enforcement; OCIE = Office of Compliance Inspections and Examinations; and OCR = Office of Credit Ratings. Source: GAO analysis of Securities and Exchange Commission (SEC) documents. | GAO-20-115 GAO is making five recommendations to SEC related to developing policies to assess the effectiveness of staff procedures and ensuring that all relevant divisions and offices follow SEC guidance and federal internal control standards for implementing control activities through documented policies. SEC agreed with the recommendations.", "document_type": "gao"}
{"report": "Federal agencies and our nation’s critical infrastructures rely on information technology systems that are highly complex and dynamic, technologically diverse, and often geographically dispersed. This complexity increases the difficulty in identifying, managing, and protecting the numerous operating systems, applications, and devices comprising their systems and networks. Further, federal systems and networks are at an increased risk of attack. This is due to those systems often being interconnected with other internal and external systems and networks, including the internet. Cloud computing relies on internet-based interconnectivity and resources to provide computing services to customers, while intending to free customers from the burden and costs of maintaining the underlying infrastructure. As federal agencies increasingly use cloud computing to perform their missions, the implementation of effective information security controls becomes more important. The effective implementation of a standardized process for securing cloud environments could reduce risks to agency systems and information maintained on an agency’s behalf. The Federal Information Security Modernization Act of 2014 (FISMA) was enacted to provide a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support federal operations and assets. The act requires federal agencies to develop, document, and implement an information security program, and evaluate the program’s effectiveness. FISMA also requires OMB to develop and oversee the implementation of policies, principles, standards, and guidelines on information security in federal agencies, except with regard to national security systems. The law assigns OMB the responsibility of requiring agencies to identify and provide information security protections commensurate with assessments of risk to their information and information systems. In addition to implementing an agencywide security program, FISMA requires agencies to ensure the security of information and systems maintained by or on behalf of the agency. The law also applies to systems used or operated by a contractor or other organization on behalf of the agency, such as IT resources provided via cloud services. In December 2010, OMB issued a plan for improving IT management that included provisions for a decision framework to migrate IT services to cloud environments. Since then, OMB has developed cloud computing requirements, issued a number of cloud-related documents, and established FedRAMP. OMB cloud-related documents include: Federal Cloud Computing Strategy, which was intended to accelerate the government’s use of cloud computing by requiring agencies to evaluate safe, secure cloud computing options before making any new investments. Security Authorization of Information Systems in Cloud Computing Environments, which established FedRAMP in December 2011. 2019 Federal Cloud Computing Strategy, issued in June 2019, updates the 2011 Federal Cloud Computing Strategy and provides agencies with additional guidance on implementing cloud solutions and emphasizes cloud security as one of the three pillars of successful cloud adoption. In addition, the FedRAMP PMO established a framework for authorizing cloud services and guidance to help participants, including all agencies, implement it. According to the program management office, the framework is based on NIST guidance that agencies are supposed to follow. In addition to the framework, the program management office issued guidance on how agencies can leverage existing security authorization packages. Agencies can select different cloud services to support their missions. These services can range from a basic computing infrastructure on which agencies run their own software, to a full computing infrastructure that includes software applications. In defining cloud service models, NIST identifies three primary models, as follows: Infrastructure as a Service (IaaS). The cloud service provider delivers and manages the basic computing infrastructure of servers, software, storage, and network equipment. The agency provides the operating system, programming tools and services, and applications. Platform as a Service (PaaS). The cloud service provider delivers and manages the infrastructure, operating system, and programming tools and services, which the agency can use to create applications. Software as a Service (SaaS). The service provider delivers one or more applications and all the resources (operating system and programming tools) and underlying infrastructure, which the agency can use on demand. In addition, agencies can choose from a variety of arrangements for obtaining cloud services (called cloud deployment models), ranging from a private cloud for one organization to sharing a public cloud. NIST identified the following four cloud deployment models: Private cloud. The service is set up specifically for one organization, although there may be multiple customers within that organization and the cloud may exist on or off the customer’s premises. Community cloud. The service is set up for organizations with similar requirements. The cloud may be managed by the organizations or a third-party and may exist on or off the organization’s premises. Public cloud. The service is available to the general public and is owned and operated by the service provider. Hybrid cloud. The service is a composite of two or more of the three deployment models (private, community, or public) that are bound together by technology that enables data and application portability. These deployment models differ from each other in the number of consumers they serve, the nature of various consumers’ data that may be present in the cloud environment, and the amount of control consumers have over their data. A private cloud can allow for its consumers to have ultimate control in selecting who has access to that cloud environment. Community clouds and hybrid clouds allow for a mixed degree of consumers’ control and knowledge of other consumers. A public cloud allows access by all interested consumers, but, in doing so, should not allow one consumer who uses it to know or control data that belong to other consumers of that environment. Established by OMB and managed by GSA, the FedRAMP program is intended to provide a standardized approach to securing systems, assessing security controls, and continuously monitoring cloud services used by federal agencies. According to GSA, this approach is a “do once, use many times” framework that potentially lowers government costs, eliminates duplications, and ensures the consistent application of federal security requirements. The goals of FedRAMP are to: ensure that cloud-based services used by government agencies have adequate safeguards in place; eliminate the duplication of effort to assess security controls, and reduce risk management costs; and enable rapid and cost-effective procurement of information systems/service for federal agencies. The program’s key participants are the FedRAMP PMO, JAB, federal agencies, cloud service providers, and third-party assessor organizations. FedRAMP PMO. FedRAMP’s PMO is headed by GSA and serves as the facilitator of the program. The office’s responsibilities include managing the program’s day-to-day operations, creating guidance and templates for agencies and cloud service providers to use for developing, assessing, authorizing, and continuously monitoring cloud services per federal requirements (e.g., FISMA). JAB. The JAB is made up of chief information officers from the Department of Defense (DOD), DHS, and GSA. It is the primary governing and decision-making body of the program. The JAB is responsible for defining and establishing FedRAMP baseline security controls and accreditation criteria for third-party assessment organizations. The JAB is also responsible for issuing a provisional authorization to operate (P-ATO) for cloud services it determines will be leveraged across most of the federal government. Federal agencies. They are consumers and, in some cases, providers of cloud services. Agencies are responsible for ensuring that cloud services which process, transmit, or store government information, use FedRAMP’s baseline security controls before they issue subsequent authorizations for using those cloud services. Cloud service providers (CSP). These providers include commercial firms and some federal agencies that offer cloud services to agencies. Providers are required to meet the FedRAMP security requirements and implement the program’s baseline security controls. Providers work with an independent third-party assessment organization to conduct an initial system assessment, create security assessment documentation per the program’s requirements, and comply with federal requirements for incident reporting, among others. Third-party assessment organizations. These FedRAMP accredited assessors perform initial and periodic assessments of cloud providers’ controls to ensure they meet the program’s requirements. In addition, these assessors must be accredited through FedRAMP if they are assessing a cloud provider seeking a provisional authorization from the JAB. For details on the roles and responsibilities of other entities involved with the program, see table 6 in appendix II. In December 2015, the FedRAMP PMO developed a security assessment framework that is to be followed by the cloud service providers (providers) and agencies seeking to authorize cloud services through the program. In addition to outlining roles and responsibilities, the framework provides agencies and cloud service providers with guidance on elements key to issuing authorizations for using cloud services through the program. These elements are critical to developing the information system or cloud service authorization package. Authorization packages include, but are not limited to the following artifacts: a control implementation summary, the security plan, the security test plan and assessment report, and remedial actions plan. These artifacts are described in table 1. FedRAMP provides agencies with two options for authorizing cloud services. The first option, called a JAB authorization, involves the agency authorizing the cloud service based on a provisional authorization issued by the board. The second option, called an agency authorization, involves the agency issuing an authorization after either sponsoring a cloud service provider through FedRAMP, or by leveraging another agency’s FedRAMP authorization of that cloud service provider. Using either of these options, the agency is to review the authorization package for that cloud service prior to issuing its authorization. In reviewing the package, the agency is to consider the cloud service’s system impact level (low impact, moderate impact, or high impact), and deployment model, among other things, to help determine which authorization option is more appropriate. After an agency has reviewed the package and made a risk-based decision to authorize a cloud service for use, it is to formally document this decision in an authorization letter. The agency official authorizing the cloud service must provide a copy of the letter to the FedRAMP PMO. The PMO uses the information to verify agency use and keep other agencies informed of any changes to a provider’s authorization. As of July 2019, all 24 CFO Act agencies participated in FedRAMP. According to the program management office’s documentation, from June 2017 through July 2019, these agencies’ use of FedRAMP authorizations increased from 390 authorizations to 926 authorizations. Specifically, the number of JAB authorizations increased from 155 to 317—a 105 percent increase. Further, the total number of agency sponsored and –leveraged authorizations increased, from 235 to 609—a 159 percent increase. Figure 1 illustrates the increase in the number of FedRAMP authorizations for the 24 agencies from June 2017 through July 2019. Survey responses from 23 of 24 CFO Act agencies indicated that the highest number of cloud service authorizations through FedRAMP were for Software as a Service. Software as a Service accounted for 331 of the 590 reported authorizations or 56 percent. For the other two services, Infrastructure as a Service and Platform as a Service, agencies reported issuing 153 authorizations (26 percent) and 106 authorizations (18 percent), respectively. Figure 2, depicts the authorizations by agency and cloud service and shows that 18 of 23 agencies issued more authorizations for Software as a Service than Platform as a Service or Infrastructure as a Service. In addition, while agencies are consumers of cloud services, some agencies also serve as cloud service providers to other federal agencies. Four of 24 agencies reported that they served as cloud service providers to other federal agencies in FY 2017. All four agencies reported that their cloud services received authorizations that were approved through FedRAMP and used by other federal agencies. These four agencies reported a total of seven cloud services with an agency authorization and one cloud service with a provisional authorization from the JAB. OMB required all agencies to use FedRAMP for authorizing cloud services by June 2014, and by June 2017, all of the 24 CFO Act agencies were using the program. However, the agencies also used cloud services that were not authorized through the program. In responding to our survey, the majority of the agencies (15 of 24) reported that they used cloud services that were not authorized through FedRAMP. For instance, one agency reported that it used 90 cloud services that were not authorized through FedRAMP and the other 14 agencies reported using a total of 157 cloud services that were not authorized through FedRAMP. Seven agencies responded that they only use cloud services authorized through FedRAMP. Two agencies did not provide a response for this question. Agencies provided varying explanations for using cloud services that were not authorized through FedRAMP. For example, officials from two of the agencies stated that they were unable to identify providers authorized through the program that could meet their unique needs. An official from a third agency noted that the efforts to meet the program’s requirements were labor-intensive and that it was too expensive for the providers to become compliant with FedRAMP. In addition, that official stated that providers did not want to pursue FedRAMP compliance unless they had enough demand from federal customers. An official from a fourth agency stated that some of that agency’s cloud services were considered to be private and, thus, did not need to be authorized through the program. Nevertheless, according to that official, the agency performed its own authorization actions to ensure that FedRAMP requirements were met. In a similar example, an official at another agency noted that it took a significant amount of time for a provider to complete the FedRAMP process and that the agency had to issue its own authorization while the provider was going through the process. That authorization had not yet been approved through FedRAMP. The survey responses of cloud service providers were consistent with the agencies’ responses and indicated that multiple agencies were using cloud services that were not authorized or approved through FedRAMP. For example, 31 of 47 providers that responded to our survey reported that, during FY 2017, agencies had used their cloud services and those services were not authorized by FedRAMP. According to one cloud service provider, agencies were using 30 of its cloud services that were not authorized through FedRAMP. Another cloud service provider reported that agencies were using nine of its cloud services that were not authorized through the program. Officials from the FedRAMP program management office also provided several reasons why agencies did not use the program for all of their cloud services. For example, one PMO official indicated agencies had misperceptions of the program, its process, and resources required for a FedRAMP authorization. The official also specified that agencies did not use the program for all their cloud services because of internal resource constraints based on other competing agency priorities. Based on our work, another potential reason that agencies authorize cloud services outside of the FedRAMP program is that OMB has not adequately monitored compliance with this requirement. As mentioned earlier, OMB has issued a number of policies encouraging agencies to adopt cloud computing solutions and requiring agencies to use FedRAMP for authorizing cloud services. Nevertheless, OMB has not monitored agencies’ compliance or held agencies accountable for complying with the requirement to ensure that agencies are using the program to authorize their cloud services. According to an OMB technical specialist, the office collects and reviews data from the FedRAMP Marketplace to monitor agencies’ use of the program. However, the office does not collect data on the extent to which federal agencies are using cloud services authorized outside of the program or oversee agencies’ compliance with using FedRAMP. As a result, if OMB does not monitor or hold agencies accountable for using the FedRAMP program, OMB and federal agencies have reduced assurance that security controls required by the program are being consistently implemented. Additionally, OMB may lack information on agencies’ needs for cloud services. Although the four selected agencies included key documents supporting FedRAMP’s authorization process, they did not consistently include key information in those documents. Specifically, these four agencies did not consistently or fully address required information in system security plans, security assessment reports, and remedial action plans. In addition, the agencies did not always prepare their authorizations approving the use of cloud services. FedRAMP recommends that agencies use the FedRAMP Control Implementation Summary (CIS) when leveraging cloud services for their systems. In addition, FedRAMP specifies that agencies are to use NIST guidance when addressing their individual or shared control implementation responsibilities when leveraging cloud services. All 10 authorization packages we reviewed contained a summary, which identified agencies’ control implementation responsibilities as well as that of the cloud service providers. An objective of system security planning is to improve the protection of information system resources. A system security plan provides an overview of the security requirements for a system or cloud service and describes the controls that are in place or planned to meet those requirements. To identify controls that an agency will need to document on its security plan, the agency reviews the CIS which lists both the agency and CSP’s security control responsibilities. Further, NIST guidelines state that federal agencies’ system security plans should identify: an explicitly defined authorization boundary for the system, how the system operates in terms of mission and business processes, the security categorization of the system including supporting rationale, the operational environment of the system and connections to other information systems, the security controls in place or planned for meeting security requirements, including a rationale for supplementing controls, and a review and approval by the authorizing official or designated representative prior to plan implementation. As shown in table 2, the four selected agencies had documented security plans for 10 systems. However, the agencies had not consistently addressed the required information in their plans. As illustrated above, the security plans for the nine selected systems did not fully address all required information. For example, three plans partially identified the operational environment of the system, such as identifying external connections which could include the cloud service the agency system was leveraging. In addition, nine plans did not fully address the extent to which security controls were in place, including those listed as the agency’s responsibility. Further, agencies did not provide complete support that their authorizing officials had reviewed and approved the plans for five systems. Specifically, agencies provided signed letters indicating that the agencies initially approved the plans. However, agencies did not provide documentation to show that subsequent changes to the system security plan after the date of the signed letters were reviewed and approved by the authorizing official. Additionally, one agency had an expired letter. Until agencies fully address required information in their security plans, including the controls relied on by the cloud service provider, they have reduced assurance that security controls are in place and operating as intended. NIST specifies that organizations document the results of security assessments in a security assessment report. According to FedRAMP’s guidance, agencies are to use the Control Implementation Summary to identify controls that are their responsibility and assess agency-specific controls, inclusive of any agency controls that are shared with providers. The security assessment report is to summarize the control testing and describe whether the tested controls were effectively in place. As shown in table 3, agencies did not always summarize the testing of controls on security assessment reports. The four agencies prepared security assessment reports for each of the 10 selected systems. However, agencies summarized the results of control tests for only three of the 10 systems reviewed. USAID summarized the test results in the security assessment report for the agency system we reviewed, but the other three agencies did not consistently summarize their results. For example, HHS did not summarize test results for three controls for one system and six controls for another system. GSA did not summarize tests results for 17 controls for one of its systems. If security assessment reports do not fully summarize the test results, agencies may have limited assurance that the controls intended to protect agency data in the cloud environment are in place and operating effectively. A remedial action plan assists agencies in identifying, assessing, prioritizing, and monitoring progress in correcting security weaknesses that are found in information systems. NIST guidelines specify that organizations develop a remedial action plan, also referred to as a plan of action and milestones, to document the organization’s planned actions to correct weaknesses or deficiencies noted during the assessment of security controls of the information system. In addition, FedRAMP guidance stated that all agencies should follow FISMA which requires agencies to have a process for documenting remedial actions to address any deficiencies in the information security policies, procedures, and practices. OMB requires that remedial action plans include the following information: a description of the specific weakness; the name of the office or organization responsible for resolving the weakness; an estimate of the funding required to resolve the weakness, including the anticipated source of funding; an estimated completion date for resolving the weakness; key milestones with estimated completion dates; any changes to the key milestones and completion dates; the source of the identified weakness (e.g. security assessment, program review, inspector general audit, etc.); and the status of the corrective action (ongoing, completed, etc.). As shown in table 4, the four selected agencies documented remedial action plans for each of the selected systems, but did not consistently identify required information. As illustrated above, three plans partially identified the office responsible for addressing the weakness. Two plans did not include changes to information regarding key milestones and completion dates and two partially included the information. Further, two agencies partially identified the source of the weakness for three systems while a third agency did not identify any sources for the selected system. Until agencies include all required elements in their remedial action plans, they will be less likely to effectively assess, prioritize, and monitor efforts to resolve weaknesses in their systems. OMB defines an authorization to operate as an official management decision where a federal official or officials authorize the operation of information system(s) and accept the risk to agency operations and assets, individuals, and other organizations based on the implementation of security and privacy controls. OMB requires agencies to use FedRAMP processes when granting authorizations to operate for their use of cloud services. According to FedRAMP PMO guidance, authorizing officials should document the authorization of (1) the agency system supported by the cloud service and (2) the cloud service used by the agency. Additionally, the agency should provide a copy of its authorization letter for the cloud service (cloud service authorization letter) to the FedRAMP program management office so that the office can verify the agency’s use of the service and keep agencies informed of any changes to a provider’s authorization status. As shown in Table 5, agencies did not consistently prepare and provide the FedRAMP PMO with the cloud service authorization letter. GSA prepared both system and cloud service authorization letters for its two selected systems. However, the other three agencies did not consistently prepare the letters. Specifically, USAID did not consistently prepare letters authorizing the cloud service and the system supported by the cloud service. In addition, HHS and EPA did not consistently prepare letters authorizing their use of the cloud services. Further, EPA, HHS, and USAID did not consistently provide the FedRAMP PMO with authorization letters for cloud services. Although GSA and an HHS component, CDC, provided cloud service authorization letters to the FedRAMP PMO, only HHS included the requirement to provide the letter to the FedRAMP PMO in its guidance. Three of the four selected agencies did not include this requirement in their guidance. Not including this requirement in their security guidance could be a potential reason for agencies’ inconsistent implementation. If agencies do not provide copies of their cloud service authorization letters to the program management office, the office may not have accurate information on which agencies are using approved cloud services. Further, the lack of such information could result in the office being delayed in notifying agencies when a service provider’s authorization has been revoked or a provider has experienced a security incident. Agencies provided various reasons for not including required information in FedRAMP authorization documents. Such reasons included the agency was restricted from documenting proprietary information concerning the cloud service provider’s portion of the shared control in the security plan and the agency was tracking all remedial actions, but the agency did not include them in the plan it provided to us. By not including the required information, agencies have reduced assurance that controls over cloud services have been effectively implemented. FedRAMP participants identified a number of the program’s benefits, such as improved security of agencies’ data and increased efficiency for providers to obtain authorizations. Participants also cited a number of challenges, such as the agency resources needed for authorizing a cloud service or the resources needed by the provider to implement the program’s requirements. To address challenges, GSA has taken steps to improve the program, but its guidance on FedRAMP’s requirements and participant’s responsibilities was not always clear and the program’s process for monitoring the status of security controls over cloud services was limited. FedRAMP participants indicated that implementing certain elements of the program were challenging. Participants specifically identified the authorization process, remedial actions, and time and resources as key challenges. Authorization process and requirements. Complex authorization process. Surveyed participants—agencies and cloud service providers—responded that simplifying the agency authorization process would help them to better understand and manage their ongoing authorizations and continuous monitoring efforts. For example, 17 of 23 agencies, responding to this question, identified the agency authorization process as an area for improvement as did 30 of 47 surveyed cloud service providers. Survey respondents indicated that the agency authorization process should be streamlined to be less-restrictive and time-consuming. Agencies also reported that overcoming the complexity of the authorization process was one of their largest hurdles. According to the Director of FedRAMP, the FedRAMP PMO encourages agencies to streamline their agency authorization processes to be less- restrictive and time-consuming. Limitations with reviewing authorization packages. Agencies also identified reviewing authorization packages as a challenge. Agencies reported in the survey and during interviews that there were limitations in their ability to review cloud security packages prior to selecting a cloud service provider. Agencies that are currently using or want to evaluate specific FedRAMP authorized cloud services are able to access FedRAMP security packages directly through the FedRAMP Secure Repository, located on OMB MAX portal. However, agencies are given a 30-day period to access packages, which one agency official stated is too short of a time period for them to properly review documentation. Although access is limited to 30 days, agencies are able to renew the access by sending an email to the FedRAMP program management office. The Director of FedRAMP indicated that agencies can work directly with cloud service providers to obtain additional permissions to the package to save, print, email, post, publish, or reproduce. In addition, agencies expressed challenges with restrictions on downloading the packages, which limited their ability to automate their review of packages and subsequent monitoring of changes to the services security posture. Agencies also cited challenges with sharing review-related information due to the restrictive nature of cloud service nondisclosure agreements. The Director of FedRAMP mentioned that agencies can work directly with cloud service providers to obtain additional access permissions to their packages. Lack of uniform guidance for selecting cloud services. Federal agencies suggested that uniform guidance on authorization packages could assist FedRAMP customers in making better risk-based decisions in selecting cloud services. Agency officials we interviewed stated the quality and reviews of authorization packages approved through FedRAMP varied. Officials stated that inconsistencies in both FedRAMP agency and JAB provisional authorization packages have required some agencies to perform additional work. According to the officials, while the JAB process takes longer, the review appears to be more detailed than the agency process. Officials noted that improving guidance on reviewing authorization packages could help with the consistency and quality of the agency package reviews. The FedRAMP PMO has taken action and published guidance during our engagement to address more details of the authorization process. In addition, according to the Director of FedRAMP, the FedRAMP PMO launched a series of training events between February 2018 and June 2019 that provided detailed guidance into the package review process. Need for improved collaboration and coordination. Participants also identified opportunities for improving collaboration and coordination. Federal agencies suggested that improved collaboration among federal agencies in leveraging cloud services could provide transparency on the cloud service providers and the services other agencies are using. This could inform agencies on whether those services could be adopted to fit the need of their missions. Agencies also mentioned that FedRAMP PMO could improve its coordination across federal agencies and cloud service providers to provide consistent information and help facilitate opportunities to improve the program. For example, three participants suggested improving cross-agency collaborations for cloud authorizations. Additionally, one survey participant noted that improved collaboration within the cloud service provider community could provide a better understanding of the impacts and associated cost of potential changes to program’s policies or requirements before they are made. According to officials from the FedRAMP PMO, their standard practice is to solicit feedback from industry and agency stakeholders prior to release of significant guidance. They added that they plan to continue collaborating with agency and industry partners. Remedial action process. In responding to our survey, 9 of 23 agencies reported that the lack of clarity on actions taken to resolve weaknesses in systems supporting cloud services was a major or moderate challenge. Specifically, two agencies cited this area as a major challenge and seven as a moderate challenge. Two agencies suggested that the program management office could make improvements by providing better visibility and traceability of the remedial action process to inform agencies on the risks associated with a cloud service. Participants responded that the remedial action process could be improved by having structured procedures for aggregating system vulnerabilities and deficiencies. This would provide agencies with better information on weaknesses identified by cloud service providers or their third party assessors in order to better consider risks prior to the purchase or use of cloud services. Additionally, agencies cited the need for improvements to the consistency of remedial action plans. Specifically, agencies cited the need for a consistent format and content of remedial action plans among security packages. Further, one cloud service provider stated that outcome-based performance metrics were a better measure of monitoring the status and effectiveness of the ongoing authorization and assessment of cloud services, as opposed to only relying on remedial action plans. According to the Director for FedRAMP, the FedRAMP PMO developed additional remedial action guidance in February 2018 and a dedicated webpage specific to the remedial action process in January of 2018. Additionally, the Director noted that for all JAB provisional authorizations, the FedRAMP PMO and JAB analyzes raw data on vulnerability scans and provides a one-page summary report that is available to agencies within the OMB MAX portal. Commitment of time and resources to complete and maintain an agency authorization. The amount of time to complete an agency authorization to operate for a cloud service was cited as one of the most challenging aspects of FedRAMP. In responding to our survey, six agencies cited the commitment of time and resources for agency authorizations as a major challenge; five agencies identified it as a moderate challenge; and six as a minor challenge. One responding agency mentioned that the time and costs associated with completing and maintaining an ongoing agency authorization was burdensome to both the agency and cloud vendor. This burden was due to a lack of allocated agency resources to continue implementing the program’s requirements. In response to this challenge, the program management office has streamlined the authorization process for low-risk systems to allow for risk-based decisions that can reduce the time and resources required for an agency authorization. In addition, 36 of 47 cloud service providers responding to our survey indicated that the significant amount of resources required to implement the program’s requirements for an authorization was a major or moderate challenge. Additionally, JAB technical representatives identified many of the challenges and opportunities for improving the program that agencies and cloud service providers identified. In addition, the officials stated that the FedRAMP PMO is aware of these issues and has taken steps to address them. According to the JAB technical representatives, the FedRAMP PMO’s program intended improvements include, but are not limited to, updates to guidance and education resources, plans to automate the continuous monitoring process with vulnerability scanning tools, and reduced time and costs associated for completing the authorization process for both customer agencies and cloud service providers. According to the Director for FedRAMP, the FedRAMP PMO has continued to make enhancements based on industry and agency feedback. The official reported that numerous guidance documents, relating to continuous monitoring, the agency authorization process, and FedRAMP designations have been released during our engagement. The official also mentioned that the PMO actively seeks feedback from stakeholders and that additional opportunities for FedRAMP training was available. GSA has taken a number of steps to improve FedRAMP. Among other things, the office has provided updated instructions for completing authorization packages and established and updated its training portal to help agencies and cloud service providers better understand the steps required for obtaining an authorization. In addition, the office has taken steps to streamline the authorization process and provided additional guidance on continuous monitoring of security controls over cloud services. Nevertheless, FedRAMP’s requirements and guidance on implementing controls were not always clear and the program’s process for monitoring the status of security controls over cloud services was limited. Clarity in program requirements and responsibilities. Agencies reported challenges with understanding FedRAMP’s requirements and the process for granting an agency authorization. Specifically, agencies cited the need for clearer guidance on requirements and agency responsibilities for completing and maintaining an authorization. Eight agencies reported the clarity of FedRAMP requirements associated with the agency authorization process as a moderate challenge; whereas nine identified it as a minor challenge and no agencies reported it as a major challenge. Five agencies reported this was not a challenge. In addition, 20 of 24 surveyed agencies indicated that additional guidance describing roles and responsibilities would be very or moderately useful to their participation in FedRAMP. Further, 37 of 47 cloud service providers specified that additional guidance for describing the security roles and responsibilities between agencies and cloud service providers was needed. Both agencies and cloud service providers commented that existing guidance for using the program does not fully address control implementation roles and responsibilities and that a process should be established to address these issues. Officials from selected agencies also indicated that responsibilities were not always clearly detailed. Specifically, HHS, GSA, and USAID officials stated that guidance for using FedRAMP could be clearer on helping define roles and responsibilities between agencies and providers in implementing security controls for cloud services. The JAB technical representatives we interviewed acknowledged that while control implementation responsibilities between the agency and cloud service provider are defined in the Control Implementation Summary, in some cases, shared responsibilities are not clearly delineated. The JAB technical representatives stated that the unclear shared responsibilities could lead to inconsistent implementation of certain controls between the agency and its provider. According to the Director of FedRAMP, it is the cloud service providers’ responsibility to ensure the spreadsheet identifying control responsibilities are completed accurately and consistently. Our analysis of agency documentation of required information in authorization packages found that the cause of selected agencies’ gaps in required information for security plans, security assessment reports and remedial action plans were due in part, to unclear guidance for implementing their control responsibilities. If responsibilities are not clear, agencies may have reduced ability to ensure that controls over the cloud services they authorized are in place and effective. Limited capabilities for continuously monitoring security controls. FedRAMP’s continuous monitoring process does not allow for an automated review of control requirements by agencies with security management tools. According to NIST SP 800-137, security continuous monitoring is maintaining an ongoing awareness of information security, vulnerabilities, and threats to support organizational risk management decisions. In addition, NIST mentions that timely, relevant, and accurate information is vital, particularly when resources are limited and agencies must prioritize their efforts. According to the program’s officials, they will be working with NIST to incorporate automation into the authorization process. Based on our work and survey responses from agencies and cloud service providers, a number of weaknesses with the program’s continuous monitoring process existed. For example, copy-protected PDFs, Word documents, and Excel spreadsheets comprised the remedial action plans and other documents supporting continuous monitoring of FedRAMP cloud service provider controls. Because of the static nature of the documents, including restrictions on copying information concerning cloud service provider controls, the documents could not be readily integrated with agencies’ automated security management tools in providing ongoing awareness of control implementation. Further, agency staff would have to spend time manually accessing and reviewing the documents each time they needed to determine the status of a cloud service’s implementation of a particular control. Agency personnel would also have to confirm that the documents they reviewed were the most current version. According to the Director of FedRAMP, agencies may request unrestricted access to the security package directly from the provider. Agencies’ survey responses also indicated that: 1) remedial action plans, used in continuous monitoring, were not updated consistently, 2) the manual process did not allow for automated data feeds into their continuous monitoring tools, and 3) restrictions on copying documents reduced information sharing within the agency. Further, 21 of 23 agencies responded that FedRAMP’s continuous monitoring of cloud security controls was a needed area of improvement. Cloud service providers also reported difficulties (36 of 47) with implementing continuous monitoring which could highlight the need for further improvements. In response, the Director of FedRAMP indicated that as of October 30, 2018, the FedRAMP PMO consolidated all continuous monitoring guidance documents, templates, and blog posts to a single webpage for ease of access by program stakeholders. JAB technical representatives also acknowledged challenges with implementing continuous monitoring such as difficulties with using continuous monitoring reports to assess the security posture of a cloud service. According to JAB technical representatives, agencies are responsible for reviewing continuous monitoring reports from the cloud service providers, but not all agencies could effectively conduct continuous monitoring. For example, an agency’s continuous monitoring efforts could be affected from not receiving a timely notification that its cloud service provider has uploaded the required monthly continuous monitoring updates, including updates to remedial actions. According to the Director of FedRAMP, the OMB MAX portal provides the capability for agencies to receive automatic notifications when there is an update to the continuous monitoring. Agencies can enable updates by selecting the “Watch this Page” option in the menu bar. While the FedRAMP PMO recommends agencies to enable this feature, agencies were not aware of the feature. As a result, agencies may not be aware that such updates have taken place and tend to be reliant on a providers’ ability to ensure that effective security practices are in place. The JAB technical representatives commented that as cloud services evolve and mature, the continuous monitoring process needs to become more automated and user-friendly to provide real-time awareness of the security status of cloud services. Until the PMO allows for more options to automate continuous monitoring, agencies may have less assurance that they will receive timely information on the extent that controls are being effectively implemented for the cloud services they are using. In addition, as more federal agencies move toward DHS’s Continuous Diagnostics and Mitigation program, automation may become even more important. Although federal agencies increased their use of FedRAMP, they continued to authorize the use of cloud services that had not been approved through the program. While OMB requires agencies to use FedRAMP to authorize the use of cloud services, it did not monitor or ensure that agencies used the program to authorize cloud services. As a result, agencies have less assurance that security controls over cloud services have been consistently implemented. The selected agencies did not fully address key elements necessary for implementing the FedRAMP authorization process. Agencies did not consistently address required information for implementing controls, summarizing control tests, and tracking corrective actions. In addition, agencies also did not always provide the FedRAMP PMO with their cloud service authorization letters. By not fully addressing these elements, agencies have less assurance that they have effectively implemented security controls intended to protect their data in cloud environments and that those controls operating as intended. FedRAMP participants identified a number of benefits as well as challenges with the program. Among other benefits, several agencies indicated that FedRAMP improved of the security of their data. However, participants identified challenges with the program and areas where the program could be improved. GSA has taken a number of actions toward improving and furthering the program’s progress, nonetheless unclear guidance and limitations with FedRAMP’s continuous monitoring process could hamper the program’s effectiveness and result in agencies implementing the program unevenly. We are making a total of 25 recommendations—1 recommendation to OMB and 24 recommendations to the 4 selected agencies in our review, including additional recommendations to GSA as the FedRAMP program lead. The Director of OMB should establish a process for monitoring and holding agencies accountable for authorizing cloud services through FedRAMP. (Recommendation 1) The Administrator of GSA should direct the Director of FedRAMP to clarify guidance to agencies and cloud service providers on program requirements and responsibilities. (Recommendation 2) The Administrator of GSA should direct the Director of FedRAMP to improve the program’s continuous monitoring process by allowing more automated capabilities, including for agencies to review documentation. (Recommendation 3) The Administrator of GSA should update security plans for selected systems to include the description of security controls and reviews and approvals plan. (Recommendation 4) The Administrator of GSA should update the security assessment report for the selected system to identify the summarized results of control effectiveness tests. (Recommendation 5) The Administrator of GSA should update the list of corrective actions for selected systems to identify the responsible office and estimated funding required and anticipated source of funding. (Recommendation 6) The Administrator of GSA should develop guidance requiring that cloud service authorization letters be provided to the FedRAMP program management office. (Recommendation 7) The Secretary of HHS should direct the Director of CDC to update the security plan for the selected system to identify the authorization boundary, the system operational environment and connections, a description of security controls, and the individual reviewing and approving the plan and date of approval. (Recommendation 8) The Secretary of HHS should direct the Director of CDC to update the security assessment report for the selected system to identify the summarized results of control effectiveness tests. (Recommendation 9) The Secretary of HHS should direct the Director of CDC to update the list of corrective actions for the selected system to identify the specific weaknesses, funding source, changes to milestones and completion dates, identified source of weaknesses, and status of corrective actions. (Recommendation 10) The Secretary of HHS should direct the Administrator of CMS to update the system security plans for selected systems to identify a description of security controls. (Recommendation 11) The Secretary of HHS should direct the Administrator of CMS to update the security assessment report for selected system to identify the summarized results of control effectiveness tests. (Recommendation 12) The Secretary of HHS should direct the Administrator of CMS to update and document the CMS remedial action plan for the selected system to identify the anticipated source of funding. (Recommendation 13) The Secretary of HHS should direct the Administrator of CMS to prepare letters authorizing the use of cloud services for the selected systems and submit the letters to the FedRAMP program management office. (Recommendation 14) The Secretary of HHS should direct the Director of NIH to update security plans for selected systems to identify the authorization boundary, system operation in terms of mission and business processes, operational environment and connections, and a description of security controls. (Recommendation 15) The Secretary of HHS should direct the Director of NIH to update the security assessment report for selected systems to identify summarized results of control effectiveness tests. (Recommendation 16) The Secretary of HHS should direct the Director of NIH to update the NIH list of corrective actions for selected systems to identify estimated funding and anticipated source of funding, key milestones with completion dates, and changes to milestones and completion dates. (Recommendation 17) The Secretary of HHS should direct the Director of NIH to submit the division’s letters authorizing the use of cloud services for the selected systems to the FedRAMP program management office. (Recommendation 18) The Administrator of EPA should update security plan for the selected operational system to identify a description of security controls, and the individual reviewing and approving the plan and date of approval. (Recommendation 19) The Administrator of EPA should update the security assessment report for the selected operational system to identify the summarized results of control effectiveness tests. (Recommendation 20) The Administrator of EPA should update the list of corrective actions for the selected operational system to identify the specific weakness, estimated funding and anticipated source of funding, key remediation milestones with completion dates, changes to milestones and completion dates, and source of the weaknesses. (Recommendation 21) The Administrator of EPA should prepare the letter authorizing the use of cloud service for the selected operational system and submit the letter to the FedRAMP program management office. (Recommendation 22) The Administrator of EPA should develop guidance requiring that cloud service authorization letter be provided to the FedRAMP program management office. (Recommendation 23) The Administrator of USAID should update the list of corrective actions for the selected system to include the party responsible for addressing the weakness, and source of the weakness. (Recommendation 24) The Administrator of USAID should prepare the letter authorizing the use of cloud service for the selected system and submit the letter to the FedRAMP program management office. (Recommendation 25) We provided a draft of this report to OMB and the 24 CFO Act agencies for review and comment. In response, we received comments from OMB and the four agencies (GSA, HHS, EPA, and USAID) to which we made recommendations. Specifically, in comments provided via email on October 15, 2019, an OMB Associate General Counsel stated that OMB neither agreed nor disagreed with our draft recommendation that it establish a process for monitoring and enforcing agency compliance with its guidance on using FedRAMP. The official asserted that OMB does not have a mechanism for enforcing agencies’ compliance with its guidance on FedRAMP. However, we believe OMB can and should hold agencies accountable for complying with its policies. Policies without accountability mechanisms present the risk that the benefits expected from their implementation will likely not be realized. To ensure our position is clearly stated, we modified the recommendation to state that OMB should establish a process for monitoring and holding agencies accountable for authorizing cloud services through FedRAMP. In addition, the OMB Associate General Counsel stated that the report did not appropriately reflect FedRAMP’s progress. We disagree. Although identifying the program’s progress was not one of our objectives, we highlighted several areas throughout the report where progress was achieved such as the agencies’ increasing use of the program to authorize cloud services and the development of additional guidance and training opportunities for using the program. The OMB Associate General Counsel also commented on the duration of the audit. Additionally, OMB commented that our use of surveys on agencies and cloud service providers’ use of FedRAMP did not address whether the program was meeting its overall objectives, but presented more of a perception. As discussed in the scope and methodology for this review, and consistent with our objectives, the purpose of the surveys was to obtain program participants’ views on the benefits, challenges, and their use of the program. Additionally, our review, as designed, including our timelines, allowed us the opportunity to best assess the implementation of the program. OMB also provided technical comments, which we have incorporated into our report as appropriate. In its written comments, GSA concurred with each of our six recommendations. The agency stated that it is developing a plan to address the recommendations. GSA’s comments are reprinted in appendix IV. In written comments, HHS concurred with each of our 11 recommendations. One operating division, CDC, noted that our observations were narrowly focused on authorization artifacts and did not take their FISMA compliant authorization process into account. We disagree. Our reviews of their FedRAMP authorization processes included procedures for reviewing security practices that are required under FISMA. The department stated that it would work with its operating divisions to address our recommendations. HHS’s comments are reprinted in appendix V. The agency also provided technical comments, which we incorporated into the report as appropriate. EPA provided written comments, in which it disagreed with the findings for two recommendations, partially agreed with the findings for one recommendation and disagreed with two other recommendations. EPA disagreed with the finding supporting our recommendation to update the security plans for the two selected systems to identify specific required information The agency stated that one of the systems we selected for review was no longer in production and not used for EPA's operations. Nevertheless, the agency stated that its chief information security officer would coordinate with the agency’s information security officers to ensure that security plans for the systems used to support its operations include all required information. We acknowledged in the report that EPA discontinued the system after we completed our review of the system’s authorization package. However, our recommendation in the draft report did not clearly convey that it was intended only for the operational system. Thus, we revised the recommendation to specify the system in operation. EPA disagreed with the finding supporting our recommendation to update the security control assessment report for one of the selected systems to identify the summarized results of control effectiveness tests. The agency stated that it used a FedRAMP certified third-party assessor that provided full documentation of control test results. However, neither the security assessment report nor other documents that EPA provided to us summarized information on how the agency tested the effectiveness of its corrective actions to rectify a critical control that had previously failed. As a result, EPA had limited assurance that it had effectively implemented a control that was intended to protect agency data in the cloud environment. Accordingly, we believe that our recommendation is warranted. EPA partially agreed with the finding supporting our recommendation to update the list of corrective actions for the selected systems to identify specific required information. The agency stated that one of the systems we selected for review was no longer in production and not used for EPA's operations. In addition, the agency said that the Chief Information Security Officer would coordinate with agency information security officers to ensure that plans of corrective actions and milestones include all required information, as appropriate. We acknowledged in the report that EPA discontinued its use of the system after we completed our review of the system’s authorization package. However, our recommendation in the draft report did not clearly convey that it was intended only for the operational system. As a result, we revised the recommendation to specify the system in operation. EPA disagreed with our recommendation that the agency prepare letters authorizing the cloud services for the selected systems and submit the letters to the FedRAMP program management office. The agency stated that one of the systems we selected for review was no longer in production and not used for EPA's operations. We acknowledged in the report that EPA had discontinued the system after we completed our review of the system's authorization package. However, our recommendation in the draft report did not clearly convey that it was intended only for the operational system. We have revised the recommendation accordingly. EPA also stated that it prepares and sends authorization letters for cloud services to the FedRAMP PMO. However, at the time of our review, the FedRAMP PMO stated it had not received the cloud service authorization letter from EPA for the selected operational system. We believe that our revised recommendation for EPA to prepare and send the cloud service authorization to the FedRAMP PMO for the operational system is warranted. EPA disagreed with our recommendation that the agency develop guidance requiring cloud service authorization letters to be provided to the FedRAMP program management office. The agency stated that it had a standard operating procedure in which the EPA Chief Information Security Officer forwards the letters to the FedRAMP program management office. However, the agency did not provide us a copy of the standard operating procedure or otherwise demonstrate that it had such an operating procedure. Thus, we continue to believe that the recommendation is warranted. EPA’s comments are reprinted in appendix VI. The agency also provided technical comments, which we incorporated into the report, as appropriate. Further, in written comments, USAID concurred with two of our three recommendations, but did not concur with the third. Specifically, USAID concurred with the two recommendations for the agency to update the list of corrective actions for the selected system and prepare the letter authorizing the use of cloud services supporting the system and submit it to the FedRAMP program management office. However, USAID did not concur with our recommendation to update the system security plan for the selected system to identify the authorization boundary, system operational environment and connections, and a description of security controls. The agency provided additional information that it had documented the authorization boundary, system operational environment and connections, and security controls for the selected system. Upon our review of the information, we agreed that the agency had sufficiently documented these items. Accordingly, we revised our report to reflect the agency’s actions and withdrew the recommendation from the report. USAID’s comments are reprinted in appendix VII. In addition to the aforementioned responses, two agencies—the Department of Veterans Affairs and the Social Security Administration— provided written responses stating that they had no comments on the draft report. These agencies’ responses are reprinted in appendixes VIII and IX, respectively. Also, the Department of Justice provided technical comments, which we incorporated into the report as appropriate. Sixteen CFO agencies provided emails stating that they had no comments on the draft report. These agencies were the Departments of Agriculture, Commerce, Defense, Education, Energy, Homeland Security, Housing and Urban Development, the Interior, Labor, State, Transportation, and the Treasury; as well as the National Aeronautics and Space Administration, National Science Foundation, Nuclear Regulatory Commission, and Office of Personnel Management. We did not receive a response from one agency—the Small Business Administration. We are sending copies of this report to appropriate congressional committees, the Director of the Office of Management and Budget, the 24 CFO Act agencies; and other interested parties. This report will also be available at no charge on our website at http://www.gao.gov. If you or your staff have any questions on matters discussed in this report, please contact Gregory C. Wilshusen at (202) 512-6244 or WilshusenG@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix X. Our objectives were to determine the extent to which 1) federal agencies used FedRAMP to authorize the use of cloud services, 2) selected agencies addressed key elements of the program’s authorization process, and 3) program participants identified FedRAMP benefits and challenges. The scope of our review included the 24 agencies covered by the Chief Financial Officers Act. To address the three objectives, we developed one survey for the 24 agencies and another survey for 83 cloud service providers identified by the FedRAMP Program Management Office (PMO) as participating in the program. We administered these web-based surveys between April and November 2018. We sent two follow-up email messages to all nonrespondents and subsequently attempted to contact the remaining nonrespondents by telephone or email at least twice more. To inform our survey questions and options, we designed our questionnaire based on FedRAMP PMO documentation and interviews with the 24 agencies and cloud service providers. We pretested the surveys with three major federal agencies, three cloud service providers, and one internal GAO group. We requested that agency chief information officers and chief information security officers review and confirm the results of the survey. We received completed surveys from 24 of 24 agencies (a 100 percent response rate) for our agency survey and 47 of the 83 cloud survey providers identified (a 57 percent response rate) for our cloud service provider survey. Not all survey respondents provided answers to all survey questions. With any survey, error can be introduced with respect to measurement of concepts, representation of respondents, and other factors, and we took steps to minimize these errors. We conducted a nonresponse bias analysis to determine whether certain cloud service providers might have been more or less likely to respond to the survey than others. Specifically, we examined whether a cloud service provider’s service model (e.g., Software as a Service, Infrastructure as a Service, Platform as a Service), impact level (e.g., high, moderate, low), or deployment model (e.g., government, hybrid, private) was related to whether the CSP responded to the survey. We found that a higher share of cloud service providers that provide Software as a Service (SaaS) responded to the survey than those that provide Infrastructure as a Service (IaaS). In addition, we found that a higher share of cloud service providers that deployed in the government community cloud responded to the survey than those that deployed in the public cloud. These results suggest that cloud service providers that utilize certain service or deployment models were more likely to reply to the survey than others. As a result, the responses of the cloud service provider survey represent only those cloud service providers that participated in this survey, and are not generalizable to cloud service providers as a whole. Despite these limitations, the survey results provide insight into the experiences and views of cloud service providers that did respond. In addition to the surveys, to address our first objective, we examined 2017, 2018, and 2019 Joint Authorization Board (JAB) and agency authorization data from the 24 agencies to determine if there were an increase, decrease, or no change in the usage of the program. We also interviewed knowledgeable officials from the 24 agencies and FedRAMP PMO to obtain their views on the program. To address our second objective, we selected four agencies from the 24 agencies based on those with the highest and lowest amount of FedRAMP PMO reported FedRAMP authorizations as of June 15, 2017. We selected the four agencies by dividing them into three equal groups of eight agencies based on the highest to lowest number of FedRAMP PMO reported service authorizations. We selected at least one agency with the highest number of authorizations through FedRAMP in each group, unless we conducted prior FedRAMP work with the agency. Given that two agencies in the third group had the same number of services authorized, we selected both agencies as one had a higher number of reported provisional authorizations through the FedRAMP Joint Authorization Board process and the other had the higher number of reported authorizations through the FedRAMP agency process. To avoid a duplication of our efforts given limited resources, we excluded DOD because another GAO team was reviewing the department’s cloud- related efforts, which included leveraging FedRAMP authorizations. As a result, we selected the Department of Health and Human Services, General Services Administration, the Environmental Protection Agency, and the United States Agency for International Development for our review. Because HHS is a large federated agency, we selected three operating divisions for a more detailed review. The three operating divisions included the Centers for Disease Control and Prevention (CDC), Centers for Medicare and Medicaid (CMS), and National Institutes of Health (NIH). We selected these divisions based on their extensive usage of cloud service providers authorized through FedRAMP. To select the agency systems’ authorization packages for review, we first identified six cloud services based on FedRAMP PMO data that indicated as of June 15, 2017, the 24 agencies used these cloud services the most. We then requested the selected agencies to provide us with an inventory of systems that relied on the six cloud services in fiscal years 2017 and 2018. From these inventories, we selected 10 agency systems. However, due to sensitivity concerns, we are not disclosing the names of the systems in this report. The case studies we selected are not generalizable to the other agencies covered by the Chief Financial Officers Act. However, it may show the potential FedRAMP issues other agencies face. For each agency system, we reviewed security authorization documentation, including: cloud service provider documentation, such as the Control Implementation Summary on agency and cloud service provider responsibilities to determine the extent agencies documented selected core controls and consistently documented responsibilities in the system security plan; security plans to determine the extent to which plans documented and implemented selected identified core security controls, and met FedRAMP and National Institute of Standards and Technology (NIST) elements; security assessment reports to determine if the effectiveness of selected core controls had been assessed and operating as intended; the extent to which agencies documented remedial action plans for selected systems to determine if they met FedRAMP or Office of Management and Budget (OMB) elements; and authorization letters to determine the extent appropriate officials approved a cloud service and agency system for use. To select identified core controls as part of our authorization documentation review, we identified and selected 24 security controls from the 97 identified core controls. Then, to determine the agencies’ compliance with the FedRAMP authorization process to assure the protection of agency data, we compared the authorization documentation with the Federal Information Security Modernization Act of 2014, the Federal Risk and Authorization Management Program guidance, including the program’s Security Assessment Framework, OMB guidance, and NIST Special Publication 800-53 Revision 4. Each authorization package area was examined and reviewed by an analyst and each conclusion was corroborated by a second analyst. Where there was disagreement in the assessment, analysts discussed their analysis and reached a consensus. In addition, we interviewed security representatives and management officials from our selected agencies to determine the effectiveness of the FedRAMP authorization process in reviewing the controls necessary for securing agency data in the cloud, and potential rationale for deficiencies identified in authorization documentation. We also interviewed FedRAMP PMO and OMB staff on their efforts related to the FedRAMP authorization process. To address our second and third objectives, we also interviewed JAB technical representatives to obtain their views on the benefits and challenges of FedRAMP. Additionally, we obtained information about how the JAB technical representatives reviewed authorization packages. To determine the reliability of the data used to select agencies and of other data to address our three objectives, we assessed the following: FedRAMP program management office points of contact list provided for active cloud service providers and federal agency users of FedRAMP, FedRAMP program management office data on the 24 CFO Act agencies’ fiscal years 2017, 2018, and 2019 JAB and agency authorizations, FedRAMP program management office data on cloud service provider participation and agency usage of FedRAMP as of June 15, 2017, Agency inventory of systems relying on selected cloud services, Cloud service provider authorization documentation contained within Cloud service provider and agency reported third-party assessment organizations’ security assessment reports, and Agency plans of actions and milestones. To assess the reliability of the information received and reviewed on the FedRAMP marketplace, we collected and reviewed information on agencies’ quality control procedures and asked program officials relevant questions on the FedRAMP authorization log standard operating procedure. We reviewed GSA program officials’ responses to our data reliability questions such as: how the information was generated, how current the data provided was, how frequently it was updated, and how the data was accurately and consistently entered into the system used. The limitation FedRAMP officials noted was that the data generated was based on voluntarily provided authorization to operate letters submitted to the FedRAMP program management office by each of the CFO Act agencies. To ensure that the agency systems we reviewed relied on selected cloud service provider products, we had agencies confirm their use of the service supporting the agency’s system. We then compared the selected services with agencies’ annual FISMA reporting to OMB along with system security documentation (e.g. system security plans) to determine whether the cloud service services we selected were applicable to the selected agency system. A limitation with this method of selection is if an agency’s inventory is inaccurate, we would need to reselect a system. For this review, one agency’s inventory and system was incomplete resulting in removing that agency system from our selection. To confirm agencies’ virtual access to packages in OMB’s repository or a cloud service provider’s repository, we obtained screen captures of web portal contents from the FedRAMP PMO. We compared these screen captures with our own virtual access to the packages. We also obtained additional information from the FedRAMP PMO on how it ensures the accuracy and reliability of the cloud service provider package information. One limitation of this method is that cloud service providers could update documentation where access was outside of OMB MAX portal, and the PMO may not be immediately aware of package updates. To verify the accuracy and reliability of plans of actions and milestones provided by agencies, we compared the agency’s plans of actions and milestones with required OMB elements. We also requested that agencies describe how they generated the plans of action and milestones provided to us, identify the quality control procedures used, and any limitations to the data they provided. We evaluated the materiality of the information we obtained and compared it to our audit objectives. We assessed the reliability of the information by reviewing related documents and internal controls such as agency policies and procedures as well as examining packages stored in OMB’s MAX portal and cloud service provider repositories. We also interviewed knowledgeable agency officials. Through these methods, we concluded that the information was sufficiently reliable for the purposes of our reporting objectives. We conducted this performance audit from November 2016 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: FedRAMP Roles and Responsibilities Roles and responsibilities Issues policies which define the key requirements and capabilities of the FedRAMP program. Oversees and reports on agencies’ implementation of information security requirements, including implementation of FedRAMP. Develops processes for agencies and providers to request FedRAMP security authorization; Creates a framework for agencies to leverage security authorization packages; Establishes a centralized and secure repository for authorization packages that agencies can leverage to grant security authorizations; Coordinates with the National Institute of Standards and Technology ( NIST) and American Association for Laboratory Accreditation to implement a formal conformity assessment to accredit assessors; Develops templates for standard contract language and service level agreements , Memorandum of Understanding and/or Memorandum of Agreement; and Is led by GSA and serves as a liaison to ensure effective communication among all participants. Defines and updates the FedRAMP security authorization requirements; Approves accreditation criteria for third-party assessment organizations; Reviews security assessment packages of cloud service providers to grant provisional authorizations; Ensures provisional authorizations are reviewed and updated regularly; and Notifies agencies of changes to or removal of provisional authorizations. Advises FedRAMP on FISMA compliance guidance and assists in developing the standards for the accreditation of independent third-party assessment organizations (3PAO). Distributes FedRAMP information to federal CIOs and other representatives through cross- agency communications and events. Assists government-wide and agency-specific efforts to provide adequate, risk-based and cost- effective cyber security; Coordinates cyber security operations and incident response; Develops continuous monitoring standards for ongoing cyber security of federal Information systems; and Develops guidance on agency implementation of the Trusted Internet Connection program with cloud services. Gregory C. Wilshusen, (202) 512-6244 or wilshuseng@gao.gov. In addition to the individual named above, Sara Ann W. Moessbauer, (Director), Larry Crosland (Assistant Director), Rosanna Guerrero (Analyst-in-Charge), Sher’rie Bacon, Nabajyoti Barkakati , Christina Bixby, David Blanding, Chris Businsky, Fatima Jahan, David Plocher, Dana Pon, Carl Ramirez, Cynthia Saunders, and Priscilla Smith made significant contributions to this report.", "summary": "Federal agencies use internet-based (cloud) services to fulfill their missions. GSA manages FedRAMP, which provides a standardized approach to ensure that cloud services meet federal security requirements. OMB requires agencies to use FedRAMP to authorize the use of cloud services. GAO was asked to review FedRAMP. The objectives were to determine the extent to which 1) federal agencies used FedRAMP to authorize cloud services, 2) selected agencies addressed key elements of the program's authorization process, and 3) program participants identified FedRAMP benefits and challenges. GAO analyzed survey responses from 24 federal agencies and 47 cloud service providers. GAO also reviewed policies, plans, procedures, and authorization packages for cloud services at four selected federal agencies and interviewed officials from federal agencies, the FedRAMP program office, and OMB. The 24 federal agencies GAO surveyed reported using the Federal Risk and Authorization Management Program (FedRAMP) for authorizing cloud services. From June 2017 to July 2019, the number of authorizations granted through FedRAMP by the 24 agencies increased from 390 to 926, a 137 percent increase. However, 15 agencies reported that they did not always use the program for authorizing cloud services. For example, one agency reported that it used 90 cloud services that were not authorized through FedRAMP and the other 14 agencies reported using a total of 157 cloud services that were not authorized through the program. In addition, 31 of 47 cloud service providers reported that during fiscal year 2017, agencies used providers' cloud services that had not been authorized through FedRAMP. Although the Office of Management and Budget (OMB) required agencies to use the program, it did not effectively monitor agencies' compliance with this requirement. Consequently, OMB may have less assurance that cloud services used by agencies meet federal security requirements. Four selected agencies did not consistently address key elements of the FedRAMP authorization process (see table). Officials at the agencies attributed some of these shortcomings to a lack of clarity in the FedRAMP guidance. Program participants identified several benefits, but also noted challenges with implementing the FedRAMP. For example, almost half of the 24 agencies reported that the program had improved the security of their data. However, participants reported ongoing challenges with resources needed to comply with the program. GSA took steps to improve the program, but its FedRAMP guidance on requirements and responsibilities was not always clear and the program's process for monitoring the status of security controls over cloud services was limited. Until GSA addresses these challenges, agency implementation of the program's requirements will likely remain inconsistent. GAO is making one recommendation to OMB to enhance oversight, two to GSA to improve guidance and monitoring, and 22 to the selected agencies, including GSA. GSA and HHS agreed with the recommendations, USAID generally agreed, EPA generally disagreed, and OMB neither agreed nor disagreed. GAO revised four recommendations and withdrew one based on new information provided; it maintains that the remaining recommendations are warranted.", "document_type": "gao"}
{"report": "BLM leases federal lands to private entities for oil and gas development generally through auctions. In the auctions, if BLM receives any bids that are at or above the minimum acceptable bid amount of $2 an acre— called bonus bids—the lease is awarded to the highest bidder (leases obtained in this way are called competitive leases). Tracts of land that do not receive a bid at the auction are made available noncompetitively for a period of 2 years on a first-come, first-served basis (leases obtained in this way are called noncompetitive leases). The government collects revenues from oil and gas leases under terms and conditions that are specified in the lease, including rental fees and royalties. Annual rental fees are fixed fees paid by lessees until production begins on the leased land or, when no production occurs, until the end of the period specified in the lease. For federal oil and gas leases, generally the rental rate is $1.50 per acre for the first 5 years, and $2 per acre each year thereafter. Once production of the resource starts, the lessees pay the federal government royalties of at least 12.5 percent of the value of production. Oil and gas parcels are generally leased for a primary term of 10 years, but lease terms may be extended if, for example, oil or gas is produced in paying quantities. A productive lease remains in effect until the lease is no longer capable of producing in paying quantities. The fiscal system refers to the terms and conditions under which the federal government collects revenues from production on leases, including from payments specified in the lease (e.g., royalties and rental payments). We reported in December 2013 that, since 1990, all federal coal leasing has taken place through a lease-by-application process, where coal companies propose tracts of federal lands to be put up for lease by BLM. BLM is required to announce forthcoming lease sales, and the announcement notes where interested stakeholders can view lease sale details, including bidding instructions and the terms and conditions of the lease. BLM leases a tract to the highest qualified bidder, as long as its bonus bid meets or exceeds $100 per acre and BLM’s confidential estimate of fair market value. Annual rental fees are at least $3 an acre, and royalties are 8 percent of the sale price for coal produced from underground mines and at least 12.5 percent of the sale price for coal produced from surface mines. Tracts are leased for an initial 20-year period, as long as the lessee produces coal in commercial quantities within a 10-year period and meets the condition of continued operations. Bonds can help ensure lands affected by energy development are properly reclaimed, that is, according to BLM, restored to as close to their original natural states as possible. Bonds provide funds that can be used by the relevant regulatory authority to reclaim such lands if the operator or other liable party does not do so. For oil and gas developed on federal lands, BLM requires operators to provide a bond before certain drilling operations begin. Wells are considered orphaned and fall to BLM to reclaim if they are not reclaimed by their operators, there are no other responsible or liable parties to do so, and their bonds are too low to cover reclamation costs. For surface coal mining, the Surface Mining Control and Reclamation Act of 1977 (SMCRA) requires operators to submit a bond to either Interior’s Office of Surface Mining Reclamation and Enforcement (OSMRE) or an approved state regulatory authority before mining operations begin for development on federal or nonfederal lands. Among other bonding options, coal operators may choose to self-bond, whereby the operator promises to pay reclamation costs. Royalties that companies pay on the sale of oil and natural gas extracted from leased federal lands and waters constitute a significant source of revenue for the federal government. The Federal Oil and Gas Royalty Management Act of 1982 requires, among other things, that Interior establish a comprehensive inspection, collection, and fiscal and production accounting and auditing system for these revenues. In particular, the act requires Interior to establish such a system to provide the capability of accurately determining oil and gas royalties, among other moneys owed, and to collect and account for such amounts in a timely manner. To accomplish this, Interior tasks its Office of Natural Resources Revenue (ONRR) with collecting and verifying the accuracy of royalties paid by companies that produce oil and gas from over 26,000 federal leases. Each month, these oil and gas companies are to self-report data to ONRR on the amount of oil and gas they produced and sold, the value of this production, and the amount of royalties that they owe to the federal government. To ensure that the data provided to ONRR are accurate and all royalties are being paid, ONRR relies on its compliance program. Under this program, ONRR initiates compliance activities by selecting companies and properties for review to assess the accuracy of their royalty data and their compliance with all relevant laws and regulations. Under the Minerals Leasing Act of 1920, Interior is authorized to collect royalties on oil and gas produced on federal lands, and BLM is required to ensure that operators producing oil and gas take all reasonable precautions to prevent the waste of oil or gas developed on these lands. While most of the natural gas produced on leased federal lands and waters is sold and therefore royalties are paid on it, some is lost during production for various reasons, such as leaks or intentional releases for ongoing operational or safety procedures. Natural gas that is released for operational or safety procedures is released directly into the atmosphere (vented) or burned (flared). In addition to gas that is lost during production, some natural gas may be used to operate equipment on the lease (lease use). We use the term natural gas emissions to refer to vented, flared, and lease use gas collectively. Interior has generally exempted operators from paying royalties on reported natural gas emissions, and so such emissions represent a loss of royalty revenues for the federal government. Venting and flaring natural gas also has environmental implications as it adds greenhouse gases to the atmosphere—primarily methane and carbon dioxide. Natural gas consists primarily of methane, and methane (which is released through venting) is 34 times more potent by weight than carbon dioxide (which is released through flaring) in its ability to warm the atmosphere over a 100-year period, and 86 times more potent over a 20-year period, according to the Intergovernmental Panel on Climate Change. Key federal lease terms are the same as they were decades ago, and Interior has not adjusted lease terms for inflation or other factors, such as changes in market conditions, which may affect the government’s fair return. In addition, preliminary observations from our ongoing work indicate that federal oil and gas lease terms and practices differ from those of selected states, with selected state governments generally charging higher royalty rates on production on state lands than the federal government charges for production on federal lands. We have previously recommended that Interior should establish procedures for determining when to conduct periodic assessments of the oil and gas fiscal system, including how the federal government’s share of revenues compares with those of other resource owners. Interior has established procedures for determining when to conduct periodic assessments of the oil and gas fiscal system, and according to its policy, BLM plans to complete the next assessment in late 2019. Key federal lease terms are the same as statutory minimums established decades ago. For onshore oil and gas leases, the minimum royalty rate of 12.5 percent has been in place since 1920, and minimum bonus bids and rental rates are currently set at the statutory minimums established in 1987. For coal, the royalty rate for surface mining is set at the statutory minimum set in the Mineral Leasing Act. We previously found that royalty rates for oil and gas leases have not been adjusted to account for changes in market conditions, and our preliminary analysis for our ongoing work suggests that adjusting rental rates for inflation could generate increased federal revenues. We reported in December 2013 that Interior offers onshore leases with lease terms— terms lasting the life of the lease—that have not been adjusted in response to changing market conditions, potentially foregoing a considerable amount of revenue. Energy markets have also changed since federal oil and gas lease terms were established. For example, we reported in June 2017 that, according to the U.S. Energy Information Administration, almost all of the recent increase in overall oil and gas production had centered on oil and gas located in shale and other tight rock geologic formations, spurred by advances in production technologies such as horizontal drilling and hydraulic fracturing. In addition, we estimate that, based on preliminary observations, the rental rate would be $2.91 per acre if it were adjusted for inflation, which would have generated about $3.6 million for the first year for new leases issued in fiscal year 2018, or an additional $1.8 million. In June 2017, we reported that raising federal royalty rates for onshore oil, gas, and coal resources could decrease oil and gas production on federal lands by either a small amount or not at all but could increase overall federal revenue, according to studies we reviewed and stakeholders we interviewed. The two oil and gas studies we reviewed for that report modeled the effects of different policy scenarios on oil and gas production on federal lands and estimated that raising the federal royalty rate could increase net federal revenue from $5 million to $38 million per year. One of the studies stated that net federal revenue would increase under three scenarios that modeled raising the royalty rate from the current 12.5 percent to 16.67 percent, 18.75 percent, or 22.5 percent. The other study noted that the effect on federal revenue would initially be small but would increase over time. The two coal studies we reviewed for our June 2017 report analyzed the effects of different policy scenarios on coal production on federal lands, and both studies suggested that a higher royalty rate could lead to an increase in federal revenues. Specifically, one study suggested that raising the royalty rate to 17 percent or 29 percent might increase federal revenue by up to $365 million per year after 2025. The other study suggested that increasing the effective rate could bring in an additional $141 million per year in royalty revenue. However, we reported that the extent of these effects was uncertain and depended, according to stakeholders, on several other factors, such as market conditions and prices. Based on preliminary observations from our ongoing work, federal onshore lease terms and practices for oil and gas development differ from those of selected states (see table 1). For example, selected state governments tend to charge higher royalty rates for oil and gas development on state lands than the federal government charges for production on federal lands. For coal production, we reported in June 2017 that royalty rates charged by selected states were generally the same as federal rates. Royalty rates for the six states representing over 90 percent of total federal oil, gas, and coal production in fiscal year 2015 ranged from 8 to 12.5 percent for surface coal and from 8 to 10 percent for underground coal. Other factors influence the competitiveness of the development of oil and gas resources on federal land versus nonfederal land. We also reported in June 2017 that some stakeholders we spoke with stated that there was already a higher regulatory burden for oil and gas companies to develop resources on federal lands than on nonfederal lands. For coal, BLM officials stated that—assuming the royalty rate was the same—the main difference between federal and nonfederal coal was the additional regulatory burden of producing on federal lands. In our ongoing work examining the oil and gas lease permitting process, our preliminary interviews indicate that drilling permit fees are higher for federal lands than for the states we reviewed. However, operators we interviewed said that the filing fee was not an important or major factor in their decisions to apply for federal drilling permits. In addition to regulatory differences, in June 2017 we reported that a few stakeholders told us that competitiveness of federal lands for development depends on the location of the best resources—such as areas with low exploration and production costs. We also reported in June 2017 that most areas with major U.S. tight oil and shale gas plays—areas of known oil and gas sharing similar properties—and major U.S. coal basins do not overlap with federal lands. We have reported on steps Interior has taken to assess its oil and gas fiscal system—the terms and conditions under which the federal government collects revenues from production on leases—and have made recommendations intended to help ensure that the federal government receives a fair return on its oil and gas resources. For example, in September 2008, we found that Interior had not evaluated the federal oil and gas fiscal system for over 25 years and recommended that a periodic assessment was needed. In response to our September 2008 report, Interior contracted for a study that was completed in October 2011 and compared the federal oil and gas fiscal systems of selected federal oil and gas regions to that of other resource owners. However, in December 2013, we reported that Interior officials said that the study was not adequate to determine next steps for onshore lease terms. Interior has considered making changes to improve its management of federal oil and gas resources. For example, in April 2015, BLM sought comments on a number of potential reforms to the oil and gas leasing process, including changing royalty rates, but took no further action. In November 2016, BLM did issue the Methane and Waste Prevention Rule, which incorporated flexibility for the bureau to make changes to onshore royalty rates, as we recommended in December 2013. Officials told us in October 2018 that they were not aware of BLM issuing any recent competitive leases with a royalty rate higher than 12.5 percent. In addition, in March 2017, the Secretary of the Interior established the Royalty Policy Committee (committee), which was to be comprised of stakeholders representing federal agencies, states, Indian tribes, mining and energy, academia, and public interest groups. The purpose of the committee was to advise the Secretary on the fair market value of mineral resources developed on federal lands, among other issues. The committee met four times over the 2 years it was in effect and approved recommendations related to Interior’s oversight of its oil and gas programs. This included two recommendations to conduct studies that compare the U.S. oil and gas fiscal system to certain other countries’ fiscal systems. However, a U.S. District Court found that the establishment of the committee violated the law and prohibited Interior from relying on any of the committee’s recommendations. Interior has established procedures for assessing the oil and gas fiscal system. In December 2013, we found that Interior did not have documented procedures for determining when to conduct additional periodic assessments of the oil and gas fiscal system, and we recommended that Interior put such procedures in place. Further, we reported that documented procedures could help Interior ensure that its evaluations take relevant factors into consideration. These factors may change over time as the market for oil and gas, the technologies used to explore and produce oil and gas, or the broader economic climate changes. In August 2016, in response to our recommendation, Interior reported that it had developed documented procedures for conducting assessments of the oil and gas fiscal system, fully implementing our recommendation. To meet this recommendation, BLM established a fiscal assessment policy that describes actions it will take every 3 years and every 10 years. Based on this policy, the next assessment is expected to be completed in late 2019. According to the policy, every 3 years BLM plans to conduct a review of the oil and gas fiscal systems of the states with significant oil and gas leasing activity where there is also significant federal onshore leasing activity. The policy states that every 10 years— depending on available appropriations—Interior plans to co-sponsor with the Bureau of Ocean Energy Management an independent study of government take from lease and development of federal oil and gas resources. In February 2019, as part of our ongoing work examining oil and gas leases, BLM officials told us that the bureau had contracted for an external fiscal assessment in 2018 and that the report would be completed in mid-2019. According to Interior officials, the study is undergoing final review. We have reported that weaknesses with bonds for coal mining and for oil and gas development pose a financial risk to the federal government as laws, regulations, or agency practices have not been adjusted to reflect current economic circumstances. We have also reported that BLM has no mechanism to pay for reclaiming well sites that operators have not reclaimed. We reported in March 2018 that self-bonding for coal mining creates a financial risk for the federal government. If specific conditions are met, SMCRA allows states to let an operator guarantee the cost for reclaiming a mine on the basis of its own finances—a practice known as self- bonding—rather than by securing a bond through another company or providing collateral, such as cash, letters of credit, or real property. We reported that as of 2017, eight states held coal self-bonds worth over $1.1 billion. In the event a self-bonded operator becomes bankrupt and the regulatory authority is not able to collect sufficient funds to complete the reclamation plan, the burden could fall on taxpayers to fund reclamation. According to stakeholders we interviewed for our March 2018 report, self- bonding for coal mining presents a financial risk to the federal government for several reasons. It is difficult to (1) ascertain the financial health of an operator, in part, because greater financial expertise is often now needed to evaluate the complex financial structures of large coal companies as compared to when self-bonding regulations were first approved in 1983; (2) determine whether an operator qualifies for self- bonding; and (3) secure a replacement for existing self-bonds when an operator no longer qualifies. For example, some stakeholders we interviewed told us that the risk from self-bonding is greater now than when OSMRE first approved its self- bonding regulations in 1983; at that time, the office noted there were companies financially sound enough that the probability of bankruptcy was small. However, according to an August 2016 OSMRE policy advisory, three of the largest coal companies in the United States declared bankruptcy in 2015 and 2016, and these companies held approximately $2 billion in self-bonds at the time. Because SMCRA explicitly allows states to decide whether to accept self-bonds, eliminating the risk that self-bonds pose to the federal government and states would require SMCRA to be amended. In our March 2018 report, we recommended that Congress consider amending SMCRA to eliminate self-bonding. Interior did not provide written comments on the report. We reported in September 2019 that bonds held by BLM have not provided sufficient financial assurance to prevent orphaned oil and gas wells on federal lands. Specifically, we reported that BLM identified 89 new orphaned wells from July 2017 through April 2019, and 13 BLM field offices identified about $46 million in estimated potential reclamation costs associated with orphaned wells and inactive wells that officials deemed to be at risk of becoming orphaned in 2018. Although BLM does not estimate reclamation costs for all wells, it has estimated reclamation costs for thousands of wells whose operators have filed for bankruptcy. Based on our analysis of these estimates, we identified two cost scenarios: low-cost wells typically cost about $20,000 to reclaim, and high-cost wells typically cost about $145,000 to reclaim. In our September 2019 report, based on our cost scenarios described above, we found that most bonds (84 percent) that we were able to link to wells in BLM data are likely too low to fund reclamation costs for all the wells they cover. Bonds generally do not reflect reclamation costs because most bonds are set at regulatory minimum values, and these minimums have not been adjusted to account for inflation since they were first set in the 1950s and 1960s, as shown in figure 1. In addition, these minimums do not account for variables, such as the number of wells they cover, or other characteristics that affect reclamation costs, such as increasing well depth. In addition to the wells identified by BLM as orphaned over the last decade, in our September 2019 report we identified inactive wells at increased risk of becoming orphaned and found their bonds are often not sufficient to reclaim the wells. Our analysis of BLM bond value data as of May 2018 and ONRR production data as of June 2017 revealed that a significant number of inactive wells remain unplugged and could be at increased risk of becoming orphaned. Specifically, we identified 2,294 wells that may be at increased risk of becoming orphaned because they have not produced since June 2008 and have not been reclaimed. Since these at-risk wells are unlikely to produce again, an operator bankruptcy could lead to orphaned wells unless bonds are adequate to reclaim them. In our September 2019 report, we stated that if the number of at-risk wells is multiplied by our low-cost reclamation scenario of $20,000, it implies a cost of about $46 million to reclaim these wells. If the number of these wells is multiplied by our high-cost reclamation scenario of $145,000, it implies a cost of about $333 million. When we further analyzed the available bonds for these at-risk wells, we found that most of these wells (about 77 percent) had bonds that would be too low to fully reclaim the at-risk wells under our low-cost scenario. More than 97 percent of these at-risk wells have bonds that would not fully reclaim the wells under our high-cost scenario. Without taking steps to adjust bond levels to more closely reflect expected reclamation costs, BLM faces ongoing risks that not all wells will be completely and timely reclaimed, as required by law. We recommended in our September 2019 report that BLM take steps to adjust bond levels to more closely reflect expected reclamation costs. BLM concurred with our recommendation. However, while BLM stated it had updated its bond review policy, it is unclear whether the updated policy will improve BLM’s ability to secure bond increases. In addition to fulfilling its responsibility to prevent new orphaned wells, it falls to BLM to reclaim wells that are currently orphaned, and BLM has not always been able to do so quickly. For example, we reported in September 2019 that there were 51 wells that BLM identified as orphaned in 2009, and that they had not been reclaimed as of April 2019. As noted above, BLM faces significant estimated potential reclamation costs associated with orphaned wells and inactive wells. The Energy Policy Act of 2005 directs Interior to establish a program that, among other things, provides for the identification and recovery of reclamation costs from persons or other entities currently providing a bond or other financial assurance for an oil or gas well that is orphaned, abandoned, or idled. In our September 2019 report we described one way in which BLM may be able to accomplish this is through the imposition of user fees, such as at the time an operator submits an application for permit to drill or as an annual fee for inactive wells. Some states, such as Wyoming, have dedicated funds for reclaiming orphaned wells. According to one official we interviewed with the Wyoming Oil and Gas Conservation Commission, the Commission has reclaimed approximately 2,215 wells since 2014 under its Orphan Well Program, which is funded through a conservation tax assessed on the sale of oil and natural gas produced in the state. Developing a mechanism to obtain funds from operators for such costs could help ensure that BLM can reclaim wells completely and timely. In commenting on a draft of our September 2019 report, BLM stated that it does not have the authority to seek or collect fees from lease operators to reclaim orphaned wells. We continue to believe a mechanism for BLM to obtain funds from oil and gas operators to cover the costs of reclamation of orphaned wells could help ensure BLM can completely and timely reclaim these wells, some of which have been orphaned for at least 10 years. Accordingly, in our September 2019 report, we recommended that Congress consider giving BLM the authority to obtain funds from operators to reclaim orphaned wells and requiring BLM to implement a mechanism to obtain sufficient funds from operators for reclaiming orphaned wells. In May 2019, we found that ONRR had begun implementing several initiatives to help the agency operate more effectively, according to ONRR officials. For example, in March 2017, ONRR initiated Boldly Go, an effort to assess its organizational structure and identify and implement potential improvements. ONRR was also in the process of implementing a new electronic compliance case management and work paper tool referred to as the Operations and Management Tool. According to ONRR documents, this tool was to combine multiple systems into one and was intended to serve a variety of functions. ONRR documents stated that the tool is designed to be a single, standardized system that reduces manual data entry, creates a single system of record for ONRR case data, offers checks to eliminate data entry errors, and provides greater transparency for outside auditors. The agency also introduced a new auditor training curriculum in April 2018. In our May 2019 report, we also found that ONRR reported generally meeting its annual royalty compliance goals for fiscal years 2010 through 2017. However, we found that while ONRR’s fiscal year 2017 compliance goals could be useful for assessing certain aspects of ONRR’s performance, they may not have been effectively aligned with the agency’s statutory requirements or its mission to account for all royalty payments. For example, ONRR’s fiscal year 2017 compliance goals did not sufficiently address its mission to collect, account for, and verify revenues, in part, because its goals did not address accuracy, such as a coverage goal (e.g., identifying the number of companies or percentage of royalties subject to compliance activities over a set period). We stated that by establishing a coverage goal that aligns with the agency’s mission, ONRR could have additional assurance that its compliance program was assessing the extent to which oil and gas royalty payments were accurate. Overall, we made seven recommendations, including that ONRR establish an accuracy goal that addresses coverage that aligns with its mission. Interior concurred with our recommendations. We issued reports in October 2010 and July 2016 that included several recommendations regarding steps Interior should take to better account for and manage natural gas emissions associated with oil and gas development. In October 2010, we reported that data collected by Interior to track venting and flaring on federal leases likely underestimated venting and flaring because they do not account for all sources of lost gas. For onshore federal leases, operators reported to Interior that about 0.13 percent of produced gas was vented or flared. Estimates from the Environmental Protection Agency and the Western Regional Air Partnership showed volumes as high as 30 times higher. We reported that economically capturing onshore vented and flared natural gas with then-available control technologies could increase federal royalty payments by $23 million annually. We also found limitations in how Interior was overseeing venting and flaring on federal leases, and made five recommendations geared toward ensuring that Interior had a complete picture of venting and flaring and took steps to reduce this lost gas where economic to do so. Interior generally concurred with our recommendations. In July 2016, we found that limitations in Interior’s guidance for oil and gas operators regarding their reporting requirements could hinder the extent to which the agency can account for natural gas emissions on federal lands. Without such data, Interior could not ensure that operators were minimizing waste and that BLM was collecting all royalties that were owed to the federal government. We recommended, among other things, that BLM provide additional guidance for operators on how to estimate natural gas emissions from oil and gas produced on federal leases. BLM concurred with the recommendation. Interior has taken steps to implement our past recommendations regarding the control of natural gas. Accounting for natural gas is important for ensuring that the federal government receives all royalties it is due and because methane—which comprises approximately 80 percent of natural gas emissions—is a potent greenhouse gas that has the ability to warm the atmosphere. In addition, we reported in July 2016 that increased oil production in recent years has resulted in an increase in flared gas in certain regions where there is limited infrastructure to transport or process gas associated with oil production. In November 2016, Interior issued regulations intended to reduce wasteful emissions from onshore oil and gas production that were consistent with our recommendations. In June 2017, however, Interior postponed the compliance dates for relevant sections of the new regulations and then suspended certain requirements in December 2017. Interior subsequently issued revised regulations in September 2018 that are not consistent with the findings and recommendations in our prior work. In our prior work and preliminary observations in our ongoing work, we have found that some states have requirements that are more stringent than BLM’s regarding accounting for and managing natural gas emissions. For example, we reported in July 2016 that North Dakota targeted the amount of gas flared from two geologic formations in the state by imposing restrictions on the amount of gas operators may flare from existing and new sources. We also reported that North Dakota requires operators to include a gas capture plan when they apply to drill a new oil well. According to state officials we interviewed for our report, gas capture plans help facilitate discussions between oil producers and firms that process and transport gas and have improved the speed at which new wells are connected to gas gathering infrastructure. In the course of our ongoing work, we obtained documents indicating that per its regulations, North Dakota requires all gas produced and used on a lease for fuel purposes or that is flared must be measured or estimated and reported monthly, and that all vented gas be burned and the volume reported. In addition, based on preliminary observations in our ongoing work, Colorado and Texas both charge royalties on vented and flared gas volumes. In the course of our ongoing work, we obtained documents indicating that the Colorado Oil and Gas Conservation Commission, which regulates oil and gas activity in the state, addresses both venting and flaring as well as leaks. Colorado officials we interviewed with the State Land Board told us in September 2019 that, since 2018, the state charges royalties on all vented and flared gas volumes, with certain exceptions. These officials told us that prior to 2018, vented and flared gas could be exempt from royalties, but that it was uncommon. In addition, in Texas, a state official we interviewed told us that vented or flared volumes must be reported monthly and that charging royalties on these volumes increases revenues. Chairman Lowenthal, Ranking Member Gosar, and Members of the Subcommittee, this completes my prepared testimony. I would be pleased to respond to any questions you may have at this time. If you or your staff have any questions about this testimony, please contact Frank Rusco, Director, Natural Resources and Environment at (202) 512-3841 or RuscoF@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony. GAO staff who made key contributions to this testimony are Quindi Franco (Assistant Director), Marie Bancroft (Analyst-In- Charge), Antoinette Capaccio, John Delicath, Jonathan Dent, Elizabeth Erdmann, Glenn C. Fischer, Emily Gamelin, William Gerard, Cindy Gilbert, Holly Halifax, Richard P. Johnson, Christine Kehr, Michael Kendix, Greg Marchand, Jon Muchin, Marietta Mayfield Revesz, Dan Royer, and Kiki Theodoropoulos. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Interior oversees energy production on federal lands and waters and is responsible for ensuring taxpayers receive a fair return for access to federal energy resources. Oil, gas, and coal on federal lands provide an important source of energy for the United States; they create jobs; and they generate billions of dollars in revenues that are shared between federal, state, and tribal governments. However, when not managed properly, energy production on federal lands can create risks to public health and the environment, such as contaminated surface water. In February 2011, GAO designated Interior's management of federal oil and gas resources as a program at high risk for fraud, waste, abuse, and mismanagement or the need for transformation. This testimony discusses GAO's work related to ensuring a fair return on resources from federal lands. To do this work, GAO drew on reports issued from May 2007 through September 2019 and preliminary observations from ongoing work. GAO reviewed relevant federal and state laws, regulations, and policies; analyzed federal data; and interviewed federal, state, and industry officials, among others. GAO's prior and ongoing work found challenges related to ensuring a fair return for oil, gas, and coal developed on federal lands in areas, including the following: Oil, Gas, and Coal Lease Terms and Conditions. Key federal lease terms are the same as they were decades ago, and Interior has not adjusted them for inflation or other factors that may affect the federal government's fair return. In June 2017, GAO reported that raising federal royalty rates—a lease term that defines a percentage of the value of production paid to the government—for onshore oil, gas, and coal resources could decrease production on federal lands by a small amount or not at all but could increase overall federal revenue. Also, preliminary observations from GAO's ongoing work indicate that selected states charge royalty rates for oil and gas produced on state lands at a higher rate than the federal government charges for production on federal lands. Oil, Gas, and Coal Bonding. GAO found in September 2019 that oil and gas bonds do not provide sufficient financial assurance because, among other things, most individual, statewide, and nationwide lease bonds are set at regulatory minimum values that have not been adjusted for inflation since the 1950s and 1960s (see figure). Further, GAO reported in March 2018 that coal self-bonding (where an operator promises to pay reclamation costs without providing collateral) poses financial risks to the federal government. Bonds provide funds that can be used to reclaim lands—restore them as close to their original natural states as possible—if an operator or other liable party does not do so. Natural Gas Emissions. In October 2010, GAO reported that data collected by Interior likely underestimated venting and flaring because they did not account for all sources of lost gas. GAO reported that economically capturing vented and flared natural gas could increase federal royalty payments by $23 million annually and made recommendations to help Interior better account for and manage emissions. In November 2016, Interior issued regulations consistent with GAO's recommendations, but Interior has since issued revised regulations, which are inconsistent with GAO's recommendations. For the reports discussed in this testimony, GAO has made 20 recommendations and three matters for congressional consideration. Interior has taken steps to implement a number of these recommendations, but 10 recommendations and two matters for congressional consideration remain unimplemented, presenting opportunities to continue to improve management of energy resources on federal lands.", "document_type": "gao"}
{"report": "The U.S. older adult population is growing and is projected to steadily increase in the coming decades. By 2060, the U.S. Census Bureau projects that adults 65 or older will make up nearly one-quarter of the total U.S. population. In addition to the overall growth in this population, the number of adults 85 or older is expected to nearly triple, from 6.4 million in 2016 to 19 million in 2060 (see fig. 1). Several federal nutrition assistance programs serve older adults, which are overseen by HHS’s Administration for Community Living (ACL) and USDA’s Food and Nutrition Service (FNS). The characteristics of older adults served by these programs vary, as do the types of assistance provided, the numbers of participants, and the amounts of federal expenditures (see table 1). The nutrition assistance programs serving older adults are overseen by ACL and FNS’s national and regional offices and are generally administered by state and local entities. The ACL and FNS national offices allocate funding and develop program regulations and guidance, and their respective regional offices provide support, such as technical assistance and training, to state agencies. State agencies implement the programs directly or through local entities. In the four programs that provide meals and monthly food packages to participants, state agencies work with regional and local agencies, such as government entities or private nonprofit organizations, to provide nutrition assistance to participants (see fig. 2). Specifically, in FNS’s two programs, state agencies work directly with local providers, while in ACL’s two programs, states work with regional level area agencies on aging, which generally contract with local providers. Area agencies on aging are public or private nonprofit entities that are responsible for planning and delivering services to older adults within their geographic service area. The Dietary Guidelines for Americans and the Dietary Reference Intakes (DRIs) are the two federally supported scientific bodies of work that provide broad information and guidance on the nutritional needs of healthy populations to help individuals maintain health and prevent nutrition-related chronic diseases. The dietary guidelines are developed by HHS and USDA and summarized in a federal policy document that focuses on providing practical nutritional and dietary information and guidance for Americans ages 2 and older. Overall, the 2015-2020 Dietary Guidelines recommend the consumption of a variety of vegetables, fruits, grains (at least half of which are whole grains), and protein, as well as fat-free or low-fat dairy and oils—sources of essential fatty acids and vitamin E. They also recommend foods and beverages that limit saturated and trans fats, as well as added sugars and sodium. Developed by the National Academies of Sciences, Engineering, and Medicine, the DRIs are a set of values used to plan and assess diets and nutrient intakes in both the United States and Canada, and the DRIs also provide scientific support for the development of the dietary guidelines. Specifically, the DRIs provide nutrient intake recommendations at levels considered safe for consumption of a wide range of nutrients, including vitamins, such as vitamins A and C; minerals, such as sodium and iron; and macronutrients, such as fiber and fat. The majority of older adults in the U.S. have chronic conditions, and evidence shows that nutrition is associated with the development of such conditions. Older adults are the fastest growing segment of the population, and they also have the greatest prevalence of chronic conditions. For example, according to the most recent data available from the Centers for Disease Control and Prevention (CDC), 62 percent of older adults 65 and older had more than one chronic condition in 2016, such as diabetes or heart disease, compared to 18 percent of adults ages 18 to 64. Although the risk of developing chronic conditions increases with age, research has shown that poor nutrition is a contributor to negative health outcomes, including many chronic conditions. For example, research shows that over- and under-consumption of certain nutrients, in addition to physical inactivity, is associated with the development of chronic conditions, including certain cancers, obesity, heart disease, and diabetes. The CDC reported that, in 2016, nutrition- related chronic conditions, including heart disease and stroke, were among the leading causes of death for older adults 65 and older in the United States, with heart disease accounting for 25 percent of deaths among this population. At the same time, research shows that nutrients and diet can prevent, delay, or assist in managing many chronic conditions, and individuals with certain chronic conditions may have different nutritional needs compared to healthy individuals. For example, according to research reviewed during development of the dietary guidelines and DRIs: diets low in sodium that also replace some carbohydrates with protein or unsaturated fats lower blood pressure and cholesterol levels, both reducing the risk of developing heart disease and helping to manage it; consumption of certain types of dietary fats, such as omega-3 fatty acids found in fish and flaxseed, for example, may help prevent or manage heart disease; increased consumption of fiber reduces total blood cholesterol, and high cholesterol is both a chronic condition as well as an increased risk for developing other chronic conditions, such as heart disease and stroke; and decreased consumption of foods high in added sugars, saturated fats, and sodium helps reduce the risk of diabetes, stroke, or heart attack. Research has shown that certain age-related changes may impair older adults’ ability to meet their nutritional needs, potentially resulting in negative health outcomes. According to a study conducted by the Academy of Nutrition and Dietetics, physiological changes that occur with age, such as decreased metabolism and reductions in muscle mass and nutrient absorption, may make it difficult for older adults to meet their nutritional needs. Research reviewed to develop the dietary guidelines also indicates that older adults experience a decline in calorie or energy needs as they age, due in part to decreased physical activity. As a result of reduced energy needs, older adults exhibit less hunger and also experience changes in taste sensation and sense of smell, all of which may lead to decreased food consumption, according to the Academy of Nutrition and Dietetics study. Inadequate consumption of certain nutrients, such as potassium, may lead to increased risk of negative health outcomes, including the development of chronic conditions, as noted earlier. Age-related physical or mental impairments also may impact older adults’ ability to meet their nutritional needs, potentially resulting in negative health outcomes. The Older Americans Act defines disability to include a physical or mental impairment, or combination of the two, that results in substantial functional limitations to certain major life activities, including self-care and mobility, among other things. An HHS official we spoke with noted that some older adults’ inability to perform daily activities— which can include eating, walking, or leaving the home to obtain groceries or meals, because of a physical or mental impairment—can contribute to inadequate nutrition. According to the CDC, age-related declines in cognitive functioning, such as the ability to reason and remember, may affect some older adults’ ability to leave their homes and shop for food, hindering their ability to meet their nutritional needs. Further, HHS reported that older adults with age-related physical impairments, such as impaired mobility and vision, may have difficulty opening, reading, and using food packaging, limiting their ability to prepare food. According to an Academy of Nutrition and Dietetics study, older adults with a physical impairment, such as an inability to chew or swallow food, may have reduced ability to consume nutrients, which, as previously noted, may increase their risk of negative health outcomes. Older adults may also require the use of medication, which may impact their ability to absorb or consume nutrients and meet their nutritional needs. For example, according to the National Institute on Aging, common side effects of certain medications can include reduced appetite and dry mouth, which may make it difficult to chew and swallow. In addition, some medications require older adults to limit their consumption of certain foods, such as citrus fruit, as consumption of these foods may change the effectiveness of the medications or cause other negative health outcomes. However, such restrictions may impact older adults’ ability to obtain the nutrients commonly found in those foods. Further, some older adults experience food insecurity, and therefore have limited access to adequate food and nutrients, which research has shown may lead to negative health outcomes. According to research reviewed to develop the dietary guidelines, food insecurity is a leading nutrition- related public health issue that compromises nutrient intake, potentially resulting in an increased risk of developing a chronic condition, as well as difficulty managing chronic conditions. USDA reported that 8 percent of U.S. households with an older adult and 9 percent of U.S. households in which an older adult lived alone experienced food insecurity in 2017—the most recent year for which data are available. According to HHS, food insecure older adults are more likely to experience negative health outcomes than their food secure counterparts. For example, research has shown that older adults who are food insecure consume lower amounts of essential nutrients and are more likely to experience negative health outcomes, like diabetes or physical or mental impairments. The federal nutrition guidelines—the dietary guidelines and Dietary Reference Intakes (DRIs)—provide broad nutrition guidance for healthy populations. However, the guidelines do not address the nutritional needs of older adults, including the majority of older adults in the United States who have multiple chronic conditions. Specifically, the guidelines focus on the foods and nutrients healthy individuals need to maintain health and prevent nutrition-related chronic conditions, which limit their applicability to older adults who already have chronic conditions. According to the scientific report for the 2015-2020 Dietary Guidelines, the guidelines are expected to evolve to address public health concerns and the nutritional needs of specific populations. Further, a report from a DRI working group indicates that the growth of the older adult population and the prevalence of chronic conditions in this group highlight the importance of understanding how nutrition can help to address chronic conditions. Although DRI researchers recently took steps to examine research on the relationship between nutrition and chronic conditions, they noted in a March 2019 report that current research on this issue is somewhat limited. At the same time, the federal nutrition guidelines do not address the varying nutritional needs of older adults of different ages and instead focus on guidelines for broad age groups. Specifically, the dietary guidelines provide information by gender on the nutrient needs of all adults 51 or older, and the DRIs provide this information by gender for older adults 51 through 70 and 71 or older. However, research has shown that these broad age categories do not account for how needs change with age among older adults, particularly for those 71 or older. For instance, according to the Academy of Nutrition and Dietetics study, the nutrient needs of older adults can be wide-ranging given the various changes that may occur with aging, such as those associated with reduced energy needs. Further, according to a summary report on the DRIs, physiological functioning, such as nutrient absorption, varies greatly after age 70. HHS officials similarly noted that nutritional needs change with each stage in life, and the needs of older adults who are in their 60s and those who are in their 90s or older may be substantially different. Additionally, researchers note that information on the varying nutritional needs of the different age groups of older adults is limited. For instance, the advisory committee that developed the 2015-2020 Dietary Guidelines noted that more data are needed on older adults’ diets, particularly for those 71 or older, and the degree to which age-related changes affect older adults’ ability to establish and maintain proper nutrition. Similarly, researchers at the Jean Mayer USDA Human Nutrition Research Center on Aging—one of the largest research centers studying nutrition and aging in the United States—told us that research on different age groups has been hindered in part by limitations in national nutrition and health data on older adults, and adults 85 or older, in particular, despite the projected growth of this age group. HHS officials said they intend to include a focus on nutritional guidance for older adults in the 2025-2030 Dietary Guidelines update, but they have not yet documented their plans to do so. Broadly, HHS and USDA officials told us they intend to address the nutritional needs of individuals across the entire lifespan in future updates to the dietary guidelines. USDA is leading the 2020-2025 Dietary Guidelines update, which will include guidance for those individuals in the earliest stages of life. HHS officials said that when they lead the 2025-2030 Dietary Guidelines update, they intend to include a focus on nutritional guidance for older adults. However, HHS has not yet documented this intention, such as through a formal plan. As noted, older adults’ nutritional needs can vary with age and many face certain challenges that additional nutrition guidance could help address, such as the management of chronic conditions or age-related changes, yet guidance currently falls short in part because of limited research evaluating older adults’ nutritional needs. In its Strategic Plan for fiscal years 2018-2022, HHS notes that one of the department’s objectives is to prevent, treat, and control communicable diseases and chronic conditions. As previously noted, the dietary guidelines are also expected to evolve to address public health concerns and the nutritional needs of specific populations. A plan for incorporating a focus on older adults in a future dietary guidelines update, such as one that addresses their various needs based on available research on this population and identifies existing information gaps, could help ensure federal nutrition guidelines better address the nutritional needs of this population. The four federal nutrition assistance programs that we reviewed and that provide meals and food directly to older adults have federal nutrition requirements, while two other programs we reviewed that provide older adults with benefits to purchase food do not. Specifically, HHS’s congregate and home-delivered meal programs and USDA’s Child and Adult Care Food Program (CACFP) have nutrition requirements for older adults’ meals, and the Commodity Supplemental Food Program (CSFP) has nutrition requirements for the monthly food package provided to older adults. Two other federal programs—USDA’s Supplemental Nutrition Assistance Program (SNAP) and Senior Farmers’ Market Nutrition Program—provide older adults with benefits to purchase food, and neither program has specific nutritional requirements that must be met when purchasing food. The four programs with nutrition requirements used the federal nutrition guidelines—the Dietary Guidelines for Americans—as the basis for their nutrition requirements. These guidelines are also the basis for nutrition requirements in other federal nutrition assistance programs, such as those that serve children. As discussed earlier, the current guidelines provide broad guidance on nutrition for healthy populations and therefore serve a role in health promotion for all individuals. Several of the nutrition assistance programs that have nutrition requirements for meals or food served to older adults also require other services to help ensure older adults’ nutritional needs are met. These services include nutrition education, screenings and assessments, and the use of nutrition professionals. Three of the four selected nutrition assistance programs serving older adults that have nutrition requirements also require nutrition education to support efforts to meet older adults’ nutritional needs. These programs are HHS’s congregate and home-delivered meal programs and USDA’s CSFP, which provides monthly food packages. See figure 5 for examples of nutrition education materials from selected states. To help promote health and delay adverse health conditions among older adults, area agencies on aging, either directly or through their local providers, are required to provide nutrition education to congregate and home-delivered meal participants. According to HHS regional officials we spoke with, there are no requirements for the frequency or type of nutrition education that must be provided, though as officials in one region noted, programs are encouraged to provide education that is science- based. According to the nationwide evaluation of the congregate and home-delivered meal programs, almost half of state agencies surveyed in 2014 required area agencies on aging, either directly or through their local providers, to provide nutrition education at least quarterly, and about one-quarter of state agencies require it to be provided semi-annually or annually. Officials from two of the four state agencies told us local providers educate participants in a variety of ways, including by directly sharing nutrition-related information about specific menu items or meals offered to participants or by partnering with other entities, such as universities, to help educate older adults on nutritional well-being. State agencies overseeing CSFP food packages must also establish a nutrition education plan and ensure that local providers provide nutrition education to program participants. For example, providers must include information about the nutritional value and use of the foods provided in the food package and should account for specific ethnic and cultural characteristics of program participants. USDA regional officials and state agency officials overseeing CSFP in three of the four states told us that providers generally use USDA’s household foods fact sheets—which includes food product descriptions, general food storage information, recipes, and nutritional information—to provide nutrition education to CSFP participants. State officials in our selected states also noted other methods CSFP providers used to support nutrition education. For example, officials in one of the states told us one of their distribution sites provides nutrition education materials in 17 languages to accommodate the different cultural backgrounds of the population it serves. Officials in another state we visited told us some of their provider sites partner with universities, inviting staff from the university’s nutrition program to the provider site to share and discuss nutrition information with participants. Both of HHS’s congregate and home-delivered meal programs require states to ensure area agencies on aging or local providers conduct nutrition screenings and assessments of participants to help identify health risks. According to HHS data for fiscal year 2016, the most recent year for which data are available, just over one-fifth (347,002) of the 1.6 million congregate meal participants served and more than one-half (496,729) of the 868,382 home-delivered meal participants served were deemed at high nutrition risk. HHS officials stated that there is no federal policy or requirement on how assessments are conducted or their frequency, and states have the flexibility to determine their own process for assessing the nutritional needs of participants. However, HHS provides a tool that states may use for these assessments. See sidebar for the Federal Nutrition Screening tool used to determine a person’s nutrition risk. According to the nationwide evaluation of the congregate and home-delivered meal programs, over half of area agencies on aging and local providers of congregate and home-delivered meal programs had a formal process for assessing nutritional needs. Further, HHS regional officials we spoke with suggested that these assessments generally occur annually. Across the four selected states we visited, the majority of area agencies on aging conducted nutrition screenings and assessments, with the frequency varying from every 6 months to every few years. The Older Americans Act requires states to prioritize certain groups with high social and economic needs, such as those who are low-income, minorities, or isolated, and two area agencies on aging told us they use nutrition risk screenings and assessments to address malnutrition and identify those individuals who fall in these categories. HHS’s congregate and home-delivered meal programs require the use of nutrition professionals, such as registered dieticians, to help local providers meet the nutritional needs of older adults—primarily through menu reviews to verify that each menu is following federal nutrition requirements, according to HHS officials. According to the nationwide evaluation of the congregate and home-delivered meal programs, at least one-half of the state agencies, area agencies on aging, and local providers used the services of a nutrition professional to help meet the nutritional needs of older adults. In the four selected states, three state agencies had a nutrition professional on staff or contracted with a nutrition professional who worked with area agencies on aging to review menus, and in the other state, a nutrition professional was on staff or contracted for by area agencies on aging or local provider sites. In addition to menu reviews, nutrition professionals in the four selected states were also involved in activities such as training meal providers or providing nutrition education and counseling to participants. As part of HHS’s oversight of the congregate and home-delivered meal programs, regional officials meet with state staff and review state plans and other program information, but these efforts do not require states to provide documentation that meals served to participants comply with the programs’ nutrition requirements. State agencies are responsible for monitoring area agencies on aging’ implementation of these programs and ensuring that meals are consistent with the programs’ nutritional requirements. HHS regional offices, in turn, conduct oversight of the nutrition programs through its reviews of states. HHS’s guidance directs regional staff to collect information from states on the use of nutrition professionals in these programs. However, HHS’s guidance does not direct regional staff to systematically review or collect any other information from states, such as approved menus, to confirm that meals served to participants are consistent with the programs’ nutrition requirements. A recent national evaluation of meals provided through the congregate and home-delivered meal programs, however, indicates that state oversight of meals’ consistency with program nutrition requirements may have limitations. According to the 2017 evaluation, while program meals generally contributed positively to participants’ diets, the meals were higher in sodium and saturated fat than the recommended limits. For example, the diets of the majority of congregate and home-delivered meal participants included adequate amounts of a range of vitamins and minerals, with the exception of magnesium and calcium. However, a majority of participants had intakes of sodium and saturated fat from these meals that exceeded the dietary guidelines’ recommended limits. Specifically, 94 percent of congregate meal participants and 69 percent of home-delivered meal participants had sodium intakes from program meals that exceeded the dietary guidelines’ recommended limit. Likewise, 89 percent of congregate meal participants and 72 percent of home-delivered meal participants had saturated fat intakes from program meals that exceeded the recommended limit, despite the role state agencies play in monitoring programs to ensure meals meet federal nutrition requirements. According to the evaluation, overconsumption of sodium and saturated fat may pose a public health concern. Information obtained from the selected states we visited also suggests that state oversight of congregate and home-delivered meals’ consistency with program nutritional requirements may have limitations. Specifically, some selected states did not utilize a nutrition professional at the state level to help ensure meals served through the programs met federal nutrition requirements. For example, in one state, the state-level nutrition professional position was vacant and, officials from an area agency on aging we spoke with confirmed that state-level monitoring of menus for compliance with nutrition requirements had not occurred due to the vacancy. Area agency on aging officials added that the vacancy has also meant that state staff are not available to train or provide guidance to area agencies on the programs’ nutrition requirements. In the other state, officials from an area agency on aging told us the state agency has not focused on oversight of providers’ menus. HHS is responsible for overseeing its federal nutrition assistance programs to ensure compliance with the programs’ nutrition requirements. More complete information on state efforts to assess meal consistency with federal nutrition requirements could help HHS assure that meals served to program participants are meeting those requirements. In USDA’s CACFP, which provides meals to older adults at adult day care centers, USDA regional offices review states’ monitoring of local providers for consistency with federal meal pattern requirements. States are required to review each entity involved in the CACFP at least once every 3 years. During these reviews, state staff must assess provider compliance with federal requirements, which includes a review of a sample of the provider’s menus to ensure they comply with federal meal pattern requirements. Through federal management evaluations, USDA regional staff review states’ monitoring of the program, including their reviews of menus to ensure compliance with meal pattern requirements, and conduct onsite reviews at both the state agency and local provider level. Regional staff told us they review all states at least once every 3 years. However, USDA regional officials told us they lack information on how the program is working at adult day care centers, in part because its onsite reviews of adult day care providers are generally limited, unlike on the child care side of the program. According to USDA officials, the majority of state agencies oversee both child care and adult day care CACFP providers, and USDA’s criteria for selecting providers for onsite reviews focus on those providers receiving the highest reimbursement amounts. According to regional officials, because CACFP serves a significantly greater number of meals to children than to adults, providers receiving the highest reimbursement amounts are those serving meals in child care sites in the majority of states. Thus, federal onsite reviews of providers serving meals to older adults in adult day care centers generally have been limited. USDA’s regional officials told us that because they have not done onsite reviews at most adult day care centers recently, they lack information on how the program is working in those centers. USDA officials in four of the seven regional offices told us they receive few questions or requests for technical assistance from state agencies or providers operating the program in adult day care centers. However, our discussions with providers in the four selected states suggest that they face challenges operating the program in these centers and addressing the varying needs of participants they serve, such as those with physical and mental impairments, and may benefit from additional information or assistance. USDA is statutorily required to review state agency and provider compliance with regulations governing program administration and operation of certain nutrition assistance programs, including CACFP. Further, USDA guidance notes that its management evaluations are critical for monitoring state agency program compliance and improving program operations by providing a basis for assessing the administration of the CACFP and developing solutions to challenges in program operations. Without taking action to ensure on-site reviews of adult day care centers participating in CACFP are conducted more consistently, USDA may be missing an opportunity to identify and help address challenges adult day care centers face in operating the program, such as challenges meeting varied needs of participants. Such efforts could help them better assess the extent to which centers are meeting the nutritional needs of the older adults they serve and to better target technical assistance. For USDA’s CSFP, which provides monthly food packages to older adults, USDA regional office oversight includes reviews of state agencies’ monitoring of local providers and visits to local providers, covering all states at least once every 3-5 years. Regional staff indicated that they review monthly participation data, food inventory reports, and state plans as part of their oversight of the program. As part of their visits with local providers, regional officials told us they open and review food packages at local sites to ensure packages include the required food components and assess the types of nutrition education provided to participants, such as recipes or cooking classes. The growth in the older adult population has led to an increased demand for nutrition programs to serve them, and some providers told us they faced challenges meeting the nutritional needs of this population. From 2009 through 2018, the population of adults 60 or older grew by 31 percent. Federal funding for certain nutrition assistance programs serving older adults has not increased at the same rate as the population. Specifically, during that same time period, federal funding for HHS’s congregate and home delivered meal programs grew by 13 percent. HHS officials told us that with the increased demand for these programs and relatively flat federal funding, some providers have been unable to maintain the same level and quality of service that they have historically provided. According to state officials and providers in three of the four selected states we visited, the increased demand for older adult nutrition programs has resulted in waiting lists, in particular for the home-delivered meal program. For example, state officials in one selected state we visited told us they have large waiting lists in their state for the home-delivered meal programs due to a higher demand for services. They indicated that, in the absence of other changes, they will only be able to serve new people through attrition of current program participants. One provider in the same state said they have a waiting list of more than 12,000 older adults for their home-delivered meal program. Another provider told us they are currently serving about 10 percent of the older adult population in their area, although the need for these services is greater, and they have continually had a waiting list for their home-delivered meal program. Some providers have leveraged additional funding sources to decrease waiting lists and expand the reach of their congregate and home- delivered meal programs. Specifically, in two of the four states we visited, some providers said they have received additional funding to support nutrition and other services for older adults through a local property tax— called a millage tax. In one of these states, a local provider told us that the local millage tax provided $9.8 million for older adult services in 2018. Officials noted that these funds allowed providers to add new meal routes and decrease waiting lists for home-delivered meals, as well expand the capacity of senior centers to serve more older adults through nutrition and other programs. In three of the four selected states, some providers reported partnering with various entities, including grocery stores, local farmers, and others to obtain food at low or no cost or serve more older adults, which helped them to meet the increased demand for the congregate and home- delivered meal programs. For example, in one state, the area agency on aging that directly provides meals joined a larger consortium of organizations to purchase food at a lower cost from a food vendor. In another state we visited, a provider we spoke with reported that the majority of its food for older adults’ meals came from food donations provided by local grocery stores and food banks and through a program in which local farmers dedicate some of their produce for donation. This provider indicated that food donations saved them $140,000 in food costs in 2018 (see fig.6). Providers we spoke with in the four selected states reported challenges meeting older adults’ needs for certain meal accommodations, and both providers and state officials that administer the congregate and home- delivered meal programs as well as the CACFP meal program across the four states reported a need for additional information from the federal agencies overseeing these programs. As previously noted, the majority of older adults in the United States now have more than one chronic condition and older adults may have physical or mental impairments—all factors that may necessitate certain accommodations to ensure meals meet their nutritional needs. Although some providers we spoke with have taken steps to mitigate challenges meeting these needs, some reported that they continue to face challenges, such as the lack of skilled chefs and other resources, to make such accommodations. Providers of HHS’s congregate and home-delivered meal programs in three of the four states said they faced challenges making meal accommodations to meet the dietary needs of older adult participants with chronic health conditions. As previously noted, 62 percent of older adults 65 and older had more than one chronic health condition in 2016—the most recent year for which data are available. Eight of the 14 congregate and home-delivered meal providers across the selected states we visited said they do not tailor meals to meet participants’ special dietary needs— for example, due in part to limited resources and capacity. For example, four providers told us it is cost prohibitive to tailor meals. At one site we visited that does tailor meals, local officials told us that their vendor charges more for tailored meals because of the additional work involved to customize meals to meet the needs of participants with specific health conditions. Another provider said that some chefs lack the skills needed to prepare such meals. For example, the provider said that although some older adults need mechanically soft or pureed meals because of oral health issues, staff may lack the skills to produce those meals. Federal restrictions on reimbursing liquid meals may make providing such meals cost-prohibitive, according to officials in selected states. For example, state and local officials and a provider in two selected states said that program participants who are unable to chew, swallow, or digest solid foods due to various health conditions, may need such meals, yet these meals do not qualify for federal meal reimbursement. According to HHS officials, while a liquid meal does not qualify for meal replacement, states may use federal funds dedicated to providing nutrition education, counseling, and other aging services to purchase these meals. Some of these program providers in the selected states used additional funding sources to help them make meal accommodations for program participants with special dietary needs, and HHS also funds awards that can be used for this purpose. For example, an area agency in one selected state we visited received a grant from a local foundation to provide some of their home-delivered meal participants with special dietary meals, including for those with renal conditions and diabetes for up to 3 months. Similarly, another provider used a grant to provide liquid meals to home-delivered meal participants who needed them. Since 2017, HHS has also awarded grants to support innovative projects that enhance the quality, effectiveness, and outcomes of the congregate and home-delivered meal programs, and some of the projects have focused on providing meal accommodations for certain program participants. For example, a grantee in one state used these grant funds to develop and deliver modified meals appropriate for home-delivered meal participants with reduced dental function. Another state grantee created new medically-tailored meals for program participants transitioning from hospital to home. According to HHS officials, the department has seen positive preliminary results from the innovation grants, but does not currently have a centralized location that compiles information for congregate and home- delivered meals providers on promising approaches for making meal accommodations for participants with special dietary needs. HHS officials said they have shared some information on the projects through webinars and conferences and provided links to webinar materials on the National Resource Center on Nutrition and Aging website—funded by HHS. Further, HHS officials noted that they posted additional relevant materials, such as a toolkit focused on lowering sodium in meals, on the Center’s website. However, these materials are not compiled in one location on the Center’s website, which may hinder meal providers’ ability to locate all of the relevant information HHS has compiled. State officials and providers across the four selected states said that federal guidance on accommodating the special dietary needs of older adult program participants is limited and additional support would be helpful. HHS is responsible for collecting and disseminating information on older adults. Providing information on promising practices and available opportunities may help support providers’ efforts to accommodate the special dietary needs of some older adults participating in these programs. State and local entities administering USDA’s CACFP in adult day care centers in the four selected states reported that they face challenges providing meal accommodations to meet the nutritional needs of program participants. Officials in three selected states said they believe the federally-required meal patterns do not fully address older adults’ nutritional needs, including those with special dietary needs. For example, milk is a federally-required component of breakfasts and lunches served through the program, though officials from three selected states said that milk can be problematic for older adults because many are lactose-intolerant or do not like drinking milk. Further, officials in one state said that the meal pattern includes a significant amount of carbohydrates, which is inconsistent with the needs of older adults who have diabetes. Although CACFP requires adult day care centers to serve meals consistent with federal meal pattern requirements or a participant’s plan of care, which may include medically-prescribed meal accommodations, state officials reported some older adults face barriers to obtaining medical documentation of meal accommodation needs. Specifically, officials from two selected states said that some participants may not have access to medical providers, and officials from one of those states explained that a visit to a medical provider is sometimes cost- prohibitive for those with limited incomes. Officials in two of the four selected states said adult day care meal providers have used available federal options that allow older adults to tailor their own meals to meet their nutritional needs, though officials also noted that these options have limitations. For example: State officials in one selected state said they encourage adult day care centers to implement the federal “offer versus serve” option. This option allows adult participants, including older adults, to decline, for example, up to two of the five meal components required with a lunch—milk, fruits or vegetables, grains, and meat or meat alternate. According to USDA guidance, this option may reduce waste and give adults more choices. However, officials in this state noted that making choices is sometimes difficult and time-consuming for program participants with cognitive impairments, such as Alzheimer’s disease or dementia. State officials in another state said that the federal family-style meal service option, which allows older adults to serve themselves from communal platters of food with assistance from supervising adults, if needed, also provides older adults with the ability to tailor meals to meet their needs. However, state officials in this state noted this meal service approach also creates challenges with feeding certain older adults appropriately. For example, this approach makes it harder to meet the needs of those with particular dietary or functional requirements, such as those who have specific nutritional needs due to chronic conditions or those with swallowing or chewing issues. State officials and adult day care providers across all four selected states said that federal guidance for providing meals to older adults in adult day care centers is limited, and providers in two of the states said they lack information on ways to address some of the challenges associated with providing meals that meet the nutritional needs of older adults in these centers. For example, providers noted that information on promising practices for serving the differing needs of older adults in these centers, including those with special dietary needs and those with functional limitations, would assist their efforts to meet participants’ nutritional needs. State officials or providers in all four selected states said that FNS’s efforts to provide guidance and trainings are more focused on the child care component of the CACFP than the adult day care component. USDA officials confirmed their efforts to provide guidance to meal providers have been primarily focused on the child care side of the program in light of the larger number of participants served. Although USDA provides some guidance and information to address the adult component of the CACFP, some CACFP entities serving older adults may not be aware of these resources, and information on promising practices or other resources to help providers meet the varying needs of older adults is more limited. USDA officials said CACFP guidance and trainings address the implementation of adult meal pattern requirements and existing flexibilities with these requirements, such as allowable substitutions for milk. USDA also produced a handbook specifically for adult day care centers in 2014 to help assist providers in these centers. However, USDA officials said that awareness of existing guidance and trainings available may be lacking, in part, because turnover for CACFP providers is high and new providers may not be aware of existing resources. Some providers also said that more information on how to address the special dietary needs and functional limitations of some participants would be helpful, as USDA’s existing guidance and trainings focus on standard adult meal pattern requirements. For example, while the 2014 handbook includes information on meal patterns and different serving methods to provide meals, it does not include information specific to meeting the differing needs of older adults in these centers. In October 2019, USDA officials told us that they are in the process of updating this handbook to reflect new policies, guidance, and promising practices for addressing the needs of older adults. USDA officials also stated that they are in the process of reviewing a promising practice to address meal accommodations for older adults with varying needs. USDA is responsible for providing training and technical assistance to states in order to assist state agencies with program management and facilitate effective operation of the program. Without awareness of existing resources and additional guidance and information to help adult day care providers address the challenges they face meeting the nutritional needs of the older adults they serve, providers may continue to be limited in their ability to do so. USDA, state, and local officials administering the CSFP said that the federal requirements for foods provided in each monthly food package limit the extent to which providers can tailor or alter the foods provided to accommodate individual participants’ nutritional needs; though some approaches and recent changes help address this challenge. For example, two food package providers we spoke with said they use other methods of food delivery along with the food package such as a pantry or grocery store-style model, which allows participants to come to a site and choose from a variety of foods that meet the requirements (see fig. 7). USDA also recently issued updated federal requirements for the type and quantity of foods provided in the food package, which department officials said provide more variety to be more useful to older adults. As previously noted, some regional USDA officials told us that early feedback from states on the changes has been positive, though states have until November 2019 to implement the new requirements. For example, USDA officials in one regional office said states provided positive feedback on the introduction of new food items, such as lentils. Providers reported ongoing program administration challenges, such as staffing constraints, which to some extent challenge their efforts to meet the nutritional needs of older adults. For example, state and local officials and providers of the congregate and home-delivered meal programs across three of the four selected states said they face challenges finding and retaining a sufficient number of staff for program operations, which could include preparing and serving meals, and delivering meals. Four of the 14 providers of these programs reported that they struggle to offer competitive wages and benefits, which hinders their ability to hire and retain staff. To help overcome staffing constraints, some providers partnered with various entities. For example, in all four selected states, providers of the congregate and home-delivered meal programs established partnerships with entities such as colleges and local businesses to solicit volunteers to help with program operations. In one state, a provider partnered with a local college’s nursing program and students volunteered to assist with assessments for home-delivered meal participants. In another state, staff from a local police department volunteer and deliver meals to home- delivered meal participants in one area. One meal provider said that the efforts of volunteers, who donate their time and cover expenses for gas and vehicle insurance to help provide home-delivered meals to participants, are worth $100,000 in annual support to their program. This provider noted that they would be unable to operate the program without volunteers. See figure 8 for pictures of volunteers helping to prepare food in selected states. Providers of the CSFP food packages and congregate and home- delivered meal programs in three selected states we visited also reported challenges obtaining transportation to bring older adults to meal and food distribution sites and deliver meals and food packages to older adults, though some have found ways to mitigate these challenges. For example, providers in three selected states said a lack of transportation options prevents some older adults from visiting congregate meal sites as well as food package distribution sites, as public transportation is not always available and many older adult participants do not drive. According to local officials in one state, transportation is also a challenge for the home- delivered meal program, particularly in rural areas, because the distance between participants’ homes affects the cost of delivering meals. Similarly, officials at one local agency on aging said providers in its area would like to serve more people, but are unable to add additional routes because of transportation costs. To help mitigate transportation challenges and manage associated costs, some providers in the selected states have adjusted meal services and found alternative ways to transport clients to meal service sites. For example, to help control transportation costs, three providers in two selected states changed from delivering one hot meal daily to delivering multiple frozen meals once a week to home-delivered meal participants. In addition, one provider partnered with a local meal delivery service that used FedEx to deliver 10 home-delivered meals every 2 weeks to program participants. To help alleviate transportation challenges that older adults face getting to meal sites, three providers in two states partnered with private companies to provide participants with rides to and from meal sites for a minimal fee. Another provider used grant funds they received from their state to purchase vans they then used to provide older adults with transportation to and from the meal sites. Some providers also reported challenges accommodating the varied dietary preferences of different groups of older adults, as preferences sometimes vary by age and cultural or ethnic background, and being responsive to these preferences can increase the likelihood that meals will help older adults meet their nutritional needs. For example, HHS officials, as well as local providers in three out of the four selected states said the dietary preferences of adults in their 60s sometimes vary greatly from the preferences of adults in their 90s. Local officials in two states said that providers of congregate and home-delivered meal programs in their states noted that “older old” adults may prefer meals that include meat and potatoes, while “younger old” adults may prefer lighter meals, such as those consisting of soups and salads. In addition, providers in three selected states we visited told us they serve many older adults from diverse cultural or ethnic backgrounds, or with dietary preferences, such as a vegetarian diet, or who do not eat certain foods because of their religious beliefs. To meet the varied dietary preferences of the older adults they serve, and increase the likelihood that meals will help participants meet their nutritional needs, some providers reported taking various approaches. For example, one congregate meal site we visited offered a lunch entree choice of either meat and potatoes or a sandwich wrap with vegetables. Another congregate meal site offered a hot lunch, plus a soup and salad bar, in a restaurant-like setting. Providers also tried to incorporate certain foods on their menus that reflect the cultural or ethnic preferences of participants. For example, the adult day care provider and the congregate and home-delivered meal providers we visited in one selected state in the South all noted that their menus aim to include certain foods associated with their regional culture, such as red beans and rice. By 2060, older adults are expected to make up nearly one-quarter of the total U.S. population. HHS and USDA play important roles in promoting the health of this growing population both through administration and oversight of federal nutrition assistance programs that serve older adults and efforts to update federal nutrition guidelines, which serves as the basis for nutrition requirements in these programs. While federal nutrition guidelines provides broad guidance on nutrition for healthy populations, they do not address the varying nutritional needs of older adults, such as those who have common chronic conditions or face age-related changes. The 2025-2030 Dietary Guidelines update is expected to include a focus on nutritional guidance for older adults, but no formal plan to include this focus has been developed. A plan to incorporate the varied needs of older adults into the dietary guidelines could assist older adults with making their own dietary decisions and help providers of nutrition assistance programs better meet older adults’ nutritional needs. Further, HHS and USDA administration and oversight of the nutrition assistance programs is not fully addressing some of the challenges states and local providers indicated hinder their efforts to meet older adults’ nutritional needs. For example, providers we spoke with faced challenges meeting older adults’ needs for certain meal accommodations, and information from HHS and USDA regarding promising approaches to meeting those needs is limited or not sufficiently disseminated. Further, both HHS and USDA’s efforts to oversee older adult meal programs have limitations that affect information available at the federal level needed to ensure programs are meeting older adults’ nutritional needs. We are making the following five recommendations. The Administrator of ACL should work with other relevant HHS officials to document the department’s plan to focus on the specific nutritional needs of older adults in the 2025-2030 update of the Dietary Guidelines for Americans, which would include, in part, plans to identify existing information gaps on older adults’ specific nutritional needs. (Recommendation 1) The Administrator of ACL should direct regional offices to take steps to ensure states are monitoring providers to ensure meal consistency with federal nutrition requirements for meals served in the congregate and home-delivered meal programs. (Recommendation 2) The Administrator of FNS should take steps to improve its oversight of CACFP meals provided in adult day care centers. For example, FNS could amend its approach for determining federal onsite reviews of CACFP meal providers to more consistently include adult day care centers. (Recommendation 3) The Administrator of ACL should centralize information on promising approaches for making meal accommodations to meet the nutritional needs of older adult participants in the congregate and home-delivered meal programs, for example in one location on its National Resource Center on Nutrition and Aging website, to assist providers’ efforts. (Recommendation 4) The Administrator of FNS should take steps to better disseminate existing information that could help state and local entities involved in providing CACFP meals meet the varying nutritional needs of older adult participants, as well as continue to identify additional promising practices or other information on meal accommodations to share with CACFP entities. (Recommendation 5) We provided a draft of this report to HHS and USDA for review and comment. In its written comments, HHS agreed with our three recommendations to ACL (Recommendations 1, 2, and 4). In response to our first recommendation, HHS stated that ACL plans to work with the Office of Disease Prevention and Health Promotion and other relevant HHS officials and agencies to document HHS’s plans to emphasize the specific and varying nutritional needs of older adults in the 2025-2030 update. HHS also stated that ACL plans to acquire the services of a registered dietician with specialized expertise in older adults’ nutritional needs. In response to our second recommendation, HHS stated that ACL’s program and evaluation offices will collaborate on the development of plans to ensure state compliance with federal requirements. In response to our recommendation that ACL centralize information on promising practices, HHS stated that ACL will award a contract in fiscal year 2020 for a new National Resource Center on Nutrition and Aging to, among other things, centralize information on promising approaches so nutrition services providers can access it easily. HHS’s comments are reproduced in appendix II. In oral comments, USDA officials, including the Directors of the FNS Child Nutrition Program Monitoring and Operational Support Division and the Child Nutrition Program Nutrition Education, Training, and Technical Assistance Division generally agreed with our two recommendations to FNS (Recommendations 3 and 5). In response to our recommendation to improve CACFP oversight, FNS officials agreed with the intent of improving oversight of CACFP meals provided in adult care centers. These officials also noted that activities and changes in this area must be consistent with statutory and regulatory requirements, balanced with current priorities given the size of the program, and mindful of resources available to perform additional oversight. While we recognize that the CACFP serves fewer adults than children and that FNS oversight resources are limited, we believe that FNS is in a position to identify the best way to improve its oversight of CACFP meals provided in adult day care centers while taking into consideration the availability of its resources. In response to our recommendation to share additional information with state and local CACFP entities, FNS officials stated that there is existing guidance and information on the adult component of the CACFP, which it communicates through multiple channels. These officials said that some states and localities may be unaware of these resources, in part, because of high turnover among staff who administer these programs. FNS officials acknowledged that they could do more to increase awareness of existing resources, as well as continue to identify and share new practices to help entities providing CACFP meals in adult day care centers address challenges associated with providing meals that meet nutritional needs of older adults. USDA also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretaries of HHS and USDA and interested congressional committees. The report will also be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Our report examines (1) the relationship of older adults’ nutrition to health outcomes and the extent to which federal nutrition guidelines address older adults’ nutritional needs; (2) the extent to which federal nutrition assistance programs serving older adults have nutrition-related requirements and how these requirements are overseen; and (3) challenges program providers face in meeting the nutritional needs of older adults. In addition to the methods discussed below, to address all three research objectives we reviewed relevant federal laws, regulations and guidance. To provide context for all three research objectives, we examined federal projections of growth in the older adult population covering the time period of 2016 through 2060. We relied on the U.S. Census Bureau’s projections of the U.S. population by various demographic traits including age, sex, race, Hispanic origin, and nativity. We assessed the reliability of these data by reviewing technical documentation describing the methodology, assumptions, and inputs used to produce the 2017 National Population Projections, upon which the 2020-2060 estimates are based. We determined these data to be sufficiently reliable for the purposes of our report. To provide context on the federal nutrition assistance programs serving older adults, we examined federal data on expenditures and participation in these programs for the most recent fiscal year available. For the congregate and home-delivered meal programs, we relied on State Program Report data from fiscal year 2017, the most recent data available at the time of our review, from the U.S. Department of Health and Human Services’ (HHS) AGing Integrated Database. These data are submitted on an annual basis by states to HHS’s Administration for Community Living (ACL). For program expenditure and participation data for the Child and Adult Care Food Program, Commodity Supplemental Food Program, Senior Farmers’ Market Nutrition Program, and Supplemental Nutrition Assistance Program (SNAP), we relied on fiscal year 2018 data from the U.S. Department of Agriculture’s (USDA) National Data Bank and submitted through USDA’s Food and Nutrition Service (FNS) grantee reports. We also relied on fiscal year 2017 data from USDA’s Characteristics of SNAP Households report on the number of older adult participants in SNAP, the most recent year for which these data were available. To assess the reliability of these data, we interviewed FNS officials and reviewed relevant technical documentation. We determined that these data were sufficiently reliable for the purposes of our report. To address our first objective on what is known about the relationship between older adults’ nutrition and health outcomes, we conducted a literature search to identify relevant peer-reviewed studies on the relationship between nutritional needs and health outcomes of older adults covering the time period of 2013 through 2018. We searched research databases, such as ProQuest, Scopus, and Ebsco (AgeLine, EconLit, and CINAHL), using search terms such as nutrition and aging and dietary guidelines for seniors. We reviewed the results of the search to identify publications that (1) included a literature review and synthesis of studies on the connection between nutrition and health outcomes for older adults, including the factors that may affect older adults’ nutritional needs, such as age-related changes and (2) emphasized the general diet-health relationship among broad populations of older adults. Because these broader studies were most relevant to our objective, we excluded studies that (1) focused on the relationship between a specific food or nutrient and a single health outcome (e.g., salt and cardiovascular disease) or (2) studied a narrow group of older adults (e.g., residents of a single U.S. state or region). We conducted detailed reviews of these studies to assess the soundness of the reported methods and the credibility and reliability of the conclusions drawn by the authors, and deemed them to be sufficiently credible, reliable, and methodologically sound for the purposes of our report. To help inform all of our research objectives and gather information about nutrition assistance programs that provide meals and food packages to older adults at the local level, we conducted visits to 25 local meal and food distribution sites in four states: Arizona (5 sites), Louisiana (10 sites), Michigan (6 sites), and Vermont (4 sites) between December 2018 and March 2019. We interviewed officials from a variety of entities involved in administering these programs in each of the states, including 20 state and area agencies on aging and 20 local providers; observed meal services and food distribution; and held conversations with older adult program participants. We selected states and local sites within those states based on a high percentage of adults 60 or older, and to ensure variation across the sites in geographic location, urban and rural location, percentage of older adults in poverty, and program provider and site type. We visited a wide variety of site locations including, but not limited to, senior centers, community centers, adult day care centers, and senior housing. Because we relied on a nongeneralizable sample of sites and states, the views of the entities we interviewed do not represent the views of all providers of federal nutrition assistance programs providing meals and food packages to older adults or participants in those programs. Prior to each selected state visit, we gathered information from state and area agencies on aging responsible for administering these programs using semi-structured interview questions. We collected information on state and area agency on aging roles in administering nutrition assistance programs for older adults, federal nutrition requirements in these programs, oversight and monitoring of programs, partnerships to help meet the nutritional needs of older adults, outreach efforts, assistance from federal agencies, and challenges in administering the programs and meeting the nutritional needs of the older adult populations served. At each site, we gathered information from local providers and participants using semi-structured interview questions. We collected information on program provider operations; characteristics of the population served; efforts to meet the nutritional needs of the population served, other nutrition-related services; challenges with meeting the nutritional needs of the population and efforts to address them; outreach efforts; and assistance received from regional, state, and federal agencies. We also collected perspectives on food received and program impacts on health outcomes from those participating at sites. In addition, at each site we observed food and meal delivery and the approximate number of participants and staff operating the site. To inform all three research objectives, we interviewed officials from HHS’s Administration for Community Living and USDA’s Food and Nutrition Service in their national office and all of their regional offices. We also interviewed a broad range of national groups, including advocacy, research, and service provider organizations involved in nutrition assistance programs serving older adults. These included AARP, Feeding America, Food Research and Action Center, Jean Mayer USDA Human Nutrition Research Center on Aging, Mathematica Policy Research, Meals on Wheels America, National Academies, National Association of Area Agencies on Aging, National Association of Nutrition and Aging Services Programs, National Association of States United for Aging and Disabilities, National Commodity Supplemental Food Program Association, and National Council on Aging. To inform our first objective on the extent to which federal nutrition guidelines address older adults’ nutritional needs, we reviewed the federal guidance reports that detail the nutrition requirements for Americans, including those reports supporting the 2015-2020 Dietary Guidelines for Americans and the body of work on the Dietary Reference Intakes. To obtain information specific to our second objective on how nutrition assistance programs serving older adults are overseen, we reviewed relevant federal program documents on monitoring and oversight of these programs. In addition, we reviewed relevant studies conducted on behalf of HHS that evaluated the impact of its nutrition assistance programs on older adults’ nutrition. These studies evaluated program participants’ diet quality and nutrient intake, as well as program administration, among other things. We assessed the reliability of results in these evaluations by interviewing officials responsible for conducting these evaluations. In addition to the contact above, Rachel Frisk and Theresa Lo (Assistant Directors), Claudine Pauselli (Analyst-in-Charge), Jessica Ard, and Vernette G. Shaw made key contributions to this report. Also contributing to this report were Priyanka Sethi Bansal, Tim Bushfield, Daniel Concepcion, Kathleen van Gelder, Sarah Gilliland, Isabella Guyott, Serena Lo, Stacy Ouellette, Amber Sinclair, Joy Solmonson, Almeta Spencer, Curtia Taylor, Adam Wendel, and Sirin Yaemsiri.", "summary": "The U.S. population is aging and, by 2030, the U.S. Census Bureau projects that one in five Americans will be 65 or older. Recognizing that adequate nutrition is critical to health, physical ability, and quality of life, the federal government funds various programs to provide nutrition assistance to older adults through meals, food packages, or assistance to purchase food. This report examines (1) the relationship of older adults' nutrition to health outcomes and the extent to which federal nutrition guidelines address older adults' nutritional needs, (2) nutrition requirements in federal nutrition assistance programs serving older adults and how these requirements are overseen, and (3) challenges program providers face in meeting older adults' nutritional needs. GAO reviewed relevant federal laws, regulations, and guidance and conducted a comprehensive literature search; visited a nongeneralizable group of four states—Arizona, Louisiana, Michigan, and Vermont—and 25 meal and food distribution sites, selected for a high percentage of adults 60 or older, and variations in urban and rural locations, and poverty level; and interviewed officials from HHS, USDA, states, national organizations, and local providers. Research shows that nutrition can affect the health outcomes of older adults. Federal nutrition guidelines provide broad guidance for healthy populations, but do not focus on the varying nutritional needs of older adults. Department of Health and Human Services (HHS) data show that the majority of older adults have chronic conditions, such as diabetes or heart disease. Research shows that such individuals may have different nutritional needs. As older adults age, they may also face barriers, such as a reduced appetite, impairing their ability to meet their nutritional needs. HHS plans to focus on older adults in a future update to the guidelines, but has not documented a plan for doing so. Documenting such a plan could help ensure guidelines better address the needs of the population. Of the six federal nutrition assistance programs serving older adults, four have requirements for food that states and localities provide directly to participants, and federal agencies oversee states' monitoring of these requirements. In HHS's and U.S. Department of Agriculture's (USDA) meal programs, states must ensure meals meet requirements. Yet, HHS does not gather information from states, such as approved menus, to confirm this, and localities in two of the four selected states said state monitoring of menus was not occurring. Further, USDA regional officials told GAO they lack information on how meal programs operate at adult day care centers as they primarily focus on other sites for their on-site reviews. Additional monitoring could help HHS and USDA ensure meal programs meet nutritional requirements and help providers meet older adults' varying needs. In the states GAO selected, meal and food providers of the four nutrition programs with nutrition requirements reported various challenges, such as an increased demand for services. Providers in three of the four states reported having waiting lists for services. Providers of HHS and USDA meal programs in all four states also reported challenges tailoring meals to meet certain dietary needs, such as for diabetic or pureed meals. HHS and USDA have provided some information to help address these needs. However, providers and state officials across the four states reported that more information would be useful and could help them better address the varying nutritional needs of older adults. GAO is making five recommendations, including that HHS develop a plan to include nutrition guidelines for older adults in a future update, and that HHS and USDA improve oversight of meal programs and provide additional information to meal providers to help them meet older adults' nutritional needs. HHS and USDA generally concurred with our recommendations.", "document_type": "gao"}
{"report": "In 1968, Congress created NFIP, with the passage of the National Flood Insurance Act, to help reduce escalating costs of providing federal flood assistance to repair damaged homes and businesses. According to FEMA, NFIP was designed to address the policy objectives of identifying flood hazards, offering affordable insurance premiums to encourage program participation, and promoting community-based floodplain management. To meet these policy objectives, NFIP has four key elements: identifying and mapping flood hazards, floodplain management, flood insurance, and incentivizing flood-risk reduction through grants and premium discounts. NFIP enables property owners in participating communities to purchase flood insurance and, in exchange, the community agrees to adopt and enforce NFIP minimum floodplain management regulations and applicable building construction standards to help reduce future flood losses. A participating community’s floodplain management regulations must meet or exceed NFIP’s minimum regulatory requirements. Insurance offered through NFIP includes different coverage levels and premium rates, which are determined by factors that include property characteristics, location, and statutory provisions. NFIP coverage limits vary by program (Regular or Emergency) and building occupancy (for example, residential or nonresidential). In NFIP’s Regular Program, the maximum coverage limit for one-to-four family residential policies is $250,000 for buildings and $100,000 for contents. For nonresidential or multifamily policies, the maximum coverage limit is $500,000 per building and $500,000 for the building owner’s contents. Separate coverage is available for contents owned by tenants. NFIP also offers Increased Cost of Compliance coverage for most policies, which provides up to $30,000 to help cover the cost of mitigation measures following a flood loss when a property is declared to be substantially or repetitively damaged. Through NFIP, FEMA maps flood hazard zones on a Flood Insurance Rate Map, which participating NFIP communities must adopt. According to FEMA, floodplain management standards are designed to prevent new development from increasing the flood threat and to protect new and existing buildings from anticipated flooding. FEMA has a division responsible for flood mapping activities and policy and guidance, but stakeholders from various levels of government and the private sector participate in the mapping process, as appropriate. A community’s Flood Insurance Rate Map serves several purposes. They provide the basis for setting insurance premium rates and identifying properties whose owners are required to purchase flood insurance. Since the 1970s, homeowners with federally backed mortgages or mortgages held by federally regulated lenders on property in a special flood hazard area have been required to purchase flood insurance. Others may purchase flood insurance voluntarily if they live in a participating community. The maps also provide the basis for establishing minimum floodplain management standards that communities must adopt and enforce as part of their NFIP participation. As of May 2020, 22,487 communities across the United States and its territories voluntarily participated in NFIP by adopting and agreeing to enforce flood-related building codes and floodplain management regulations. FEMA supports a variety of community-level flood mitigation activities that are designed to reduce flood risk (and thus NFIP’s financial exposure). These activities, which are implemented at the state and local levels, include hazard mitigation planning; adoption and enforcement of floodplain management regulations and building codes; and use of hazard control structures such as levees, dams, and floodwalls or natural protective features such as wetlands and dunes. FEMA provides community-level mitigation funding through its HMA grant programs. In addition, FEMA’s Community Rating System is a voluntary incentive program that recognizes and encourages community floodplain management activities that exceed the minimum NFIP requirements. Flood insurance premium rates are discounted to reflect the reduced flood risk resulting from community actions that meet the three goals of reducing flood damage to insurable property, strengthening and supporting the insurance aspects of NFIP, and encouraging a comprehensive approach to floodplain management. At the individual property level, mitigation options include property acquisition—or “buyouts”—to either demolish a building for green space or relocate a building to a low flood risk area, elevation, or floodproofing. Acquisition and demolition (acquisition) is one of the primary methods by which states or localities use FEMA funding to mitigate flood risk. Through this process, a local or state government purchases land and structures that flooded or are at risk from future floods from willing sellers and demolishes the structures. The community restricts future development on the land, which is maintained as open space in perpetuity to restore and conserve the natural floodplain functions. According to FEMA officials, an advantage of property acquisition is that it offers a permanent solution to flood risks, whereas other mitigation methods make properties safer from floods but not immune. Property acquisition and demolition is a voluntary process, and property owners are paid fair market value for their land and structures. Acquisition is typically done on a community-wide scale, purchasing several or all properties in an at-risk neighborhood. Acquisition projects typically require building consensus from property owners and sustained communication and collaboration between residents and the government executing the project. Acquisition and relocation (relocation) refers to purchasing a structure and moving it to another location instead of demolishing it. Through this process, state or local governments use FEMA funding to help purchase land from willing sellers and assist the property owners with relocating the structure. The structure must be sound and feasible to move outside of flood-prone areas. Relocation is a voluntary process and property owners are paid fair market value for their land. Elevation involves raising a structure so that the lowest occupied floor is at or above the area’s base flood elevation. Structure elevation may be achieved through a variety of methods, including elevating on continuous foundation walls; elevating on open foundations, such as piles, piers, or columns; and elevating on fill. Structures proposed for elevation must be structurally sound and capable of being elevated safely. Further, elevation projects must be designed and adequately anchored to prevent flotation, collapse, and lateral movement of the structure from flooding, waves, and wind. Floodproofing falls into two categories: dry floodproofing and wet floodproofing. Dry floodproofing involves sealing a structure to prevent floodwater from entering. Examples of dry floodproofing measures include using waterproof coatings or coverings to make walls impermeable to water, installing waterproof shields, and installing devices that prevent sewer and drain backup. Dry floodproofing is appropriate only where floodwaters do not exceed three feet, the speed of flood waters is low, and the duration of flooding is relatively short because walls and floors may collapse from the pressure of higher water levels. Wet floodproofing involves changing a structure to allow floodwaters to enter and exit with minimal damage. Wet floodproofing is used in parts of a structure that are not used as living space, such as a crawlspace, basement, or garage. Examples of wet floodproofing measures include installing flood openings in the foundation and enclosure walls below the base flood elevation, using flood-resistant building materials and furnishings located below the base flood elevation, and either elevating or floodproofing all utility systems and associated equipment to protect them from damage. FEMA administers three HMA grant programs that can be used to fund flood mitigation projects: the Hazard Mitigation Grant Program (HMGP), Pre-Disaster Mitigation (PDM), and Flood Mitigation Assistance (FMA). Eligible HMA applicants include states, territories, and federally recognized tribal governments. Local communities cannot apply directly to FEMA for HMA funding but instead must collaborate as sub-applicants with their state, territory, or tribal government and then receive funding through that entity. Certain nonprofit organizations can act as sub- applicants but only under HMGP. Generally, individuals may not apply for HMA funding, but they may benefit from a community application. Applicants to all three programs must have FEMA-approved hazard mitigation plans. FEMA evaluates HMA applications based on technical feasibility and cost-effectiveness, among other factors. In fiscal year 2019, HMA awarded $859 million in funding. Eligible activities differ for the three programs but must be consistent with FEMA’s National Mitigation Framework. The Hazard Mitigation Grant Program helps communities implement hazard mitigation measures following a presidential major disaster declaration to improve community resilience to future disasters. HMGP provides funding to protect public or private property through various mitigation measures based on state or tribal priorities. Mitigation project examples include acquisition, relocation, retrofitting structures to minimize damages from various natural hazards, and elevating flood prone structures. HMGP recipients (states, territories, and federally recognized tribal governments) are primarily responsible for prioritizing, selecting, and administering state and local hazard mitigation projects. According to FEMA guidance, although individuals may not apply directly to the state for assistance, local governments engage interested property owners during the application process. A formula based on the size of the presidential disaster declaration determines the amount of money available to HMGP. Pre-Disaster Mitigation seeks to reduce overall risk to the population and structures from future natural hazard events, while also reducing reliance on federal funding in future disasters. PDM grants fund mitigation plans and eligible projects that reduce or eliminate long-term risk to people and property from natural disasters, such as property acquisition, property elevation, earthquake hardening, and construction of tornado and high-wind safe rooms. Generally, local governments (i.e., sub-applicants) submit mitigation planning and project applications to their state, territory, or federally recognized tribal government (i.e., applicants) for review and prioritization. The state, territory, or federally recognized tribal government then submits one PDM grant application to FEMA for consideration. Annual Congressional appropriations fund these grants, and FEMA awards them on a nationally competitive basis. In fiscal year 2019, Congress appropriated $250 million to PDM, which was the program’s final year of funding. In 2018, Congress passed the Disaster Recovery Reform Act, which included amendments to PDM, which FEMA calls the Building Resilient Infrastructure and Communities program. According to FEMA officials, this program is replacing PDM in fiscal year 2020 and will be funded through the Disaster Relief Fund as a 6 percent set-aside from the estimated total amount of grants for each major disaster declaration. FEMA has solicited public input on the program and said it expects to release a notice of funding opportunity in summer 2020. Flood Mitigation Assistance is designed to reduce or eliminate flood insurance claims by funding cost-effective flood mitigation projects that reduce or eliminate long-term risk of flood damage to structures insured under NFIP. Typical projects may include acquisition of RL properties, elevation of buildings, and neighborhood-scale flood defense investment. Generally, local communities will sponsor applications on behalf of homeowners and then submit the applications to their state. A state or federally recognized tribal government must submit the grant applications to FEMA. Annual Congressional appropriations fund FMA grants, and FEMA awards them on a nationally competitive basis. FMA appropriations have remained relatively stable at about $175 million for fiscal years 2016 through 2019. RL properties present a financial challenge for NFIP. FEMA has three definitions for such properties that vary slightly to meet the specific needs of different programs: NFIP Repetitive Loss refers to an NFIP-insured structure that has incurred flood-related damage on two occasions during a 10-year period, each resulting in at least a $1,000 claim payment. FEMA uses the NFIP RL definition for insurance purposes related to the Community Rating System, for local hazard mitigation plans, and for eligibility determinations for preferred risk policies and individual assistance. FMA Repetitive Loss refers to an NFIP-insured structure that (a) has incurred flood-related damage on two occasions in which the cost of repair, on average, equaled or exceeded 25 percent of the value of the structure at the time of each such flood event; and (b) at the time of the second incidence of flood-related damage, the flood insurance policy contained Increased Cost of Compliance coverage. FEMA uses this definition for FMA purposes, as these properties are eligible for the largest federal cost share for mitigation, up to 90 percent. This is also the same definition NFIP uses to approve an Increased Cost of Compliance payment. Severe Repetitive Loss refers to an NFIP-insured structure that has incurred flood-related damage for which (a) four or more separate claims have been paid that exceeded $5,000 each and cumulatively exceeded $20,000; or (b) at least two separate claim payments have been made under such coverage, with the cumulative amount of such claims exceeding the fair market value of the insured structure. FEMA has two severe RL definitions for mitigation and insurance, which are similar except that the insurance definition includes only residential structures, while the mitigation definition includes all structures. FEMA uses the severe RL definition for grant eligibility and cost share, the Community Rating System, and insurance rate setting. HMGP is the largest of FEMA’s three HMA programs and, unlike the others, it is based on the amount of disaster assistance a state or territory receives following a presidential disaster declaration (see table 1). PDM and FMA are smaller grant programs that receive annual appropriations and are not directly tied to an immediately preceding disaster. Because these programs do not require an immediate disaster declaration, FEMA considers them pre-disaster programs, as their intent is to mitigate potential damage before disasters occur. HMGP and PDM can be used for projects that mitigate the risk of many hazards, including flood, wind, fire, earthquake, and drought, but FMA can only be used to mitigate the risk of flood (see table 1). Furthermore, FMA funds can only be used to mitigate properties that are insured by NFIP, but HMGP and PDM funds can be used to mitigate properties without NFIP coverage. Properties mitigated in a special flood hazard area, where the structure remains on the parcel, must maintain a flood insurance policy after project completion. HMA grants fund a variety of methods to mitigate the flood risk of properties, including acquisition, elevation, relocation, and floodproofing. In most cases, HMA grants cover up to 75 percent of the project cost, and the grantee generally must contribute the remainder using nonfederal funds (although there are some exceptions, discussed below). However, PDM will cover up to 90 percent of project costs for communities that meet FEMA’s definition of small and impoverished. Moreover, FMA will cover up to 90 percent for projects that mitigate RL properties and up to 100 percent for severe RL properties. Funding levels for the three programs have varied over time because they have depended on disaster declarations and annual appropriations (see fig. 1). HMGP is the largest of the three programs—adjusted for inflation, annual HMGP grants have reached $2.9 billion, while PDM and FMA have never exceeded $300 million. According to FEMA officials, the estimated annual funding for the Building Resilient Infrastructure and Communities program, the successor to PDM, will average $300 million to $500 million, as it will be funded by a 6 percent set aside of annual estimated disaster grant expenditures. HMA funding also varies by state. Louisiana has obligated the most funding. After adjusting for inflation, it has obligated more than $3.1 billion from all three programs since HMGP was created in 1989, followed by California ($2.0 billion), Texas ($1.8 billion), New York ($1.6 billion), and Florida ($1.5 billion), while the bottom 18 states and territories each obligated less than $50 million (see fig. 2). Because HMGP is the largest program and is tied to presidential declarations, these totals reflect, in part, the extent to which states and territories have experienced natural disasters in this time period. Typically, recipients of federal mitigation grants must use nonfederal funds to meet cost share requirements because federal law prohibits the use of more than one source of federal disaster recovery funding for the same purpose. However, according to FEMA, some federal programs are exempt from these requirements due to authorizing statutes and therefore may be used in concert with HMA funds. Department of Housing and Urban Development’s Community Development Block Grant (CDBG) program. The Department of Housing and Urban Development awards CDBG funds to state and local governments to support a variety of community and economic development needs. According to FEMA’s HMA Cost Sharing Guide, HMA applicants may use several categories of CDBG funds as a source of project cost share, as long as the project meets Department of Housing and Urban Development rules. CDBG Disaster Recovery funds are the most frequently used form of HMGP cost share from a federal agency, according to FEMA. FEMA Increased Cost of Compliance coverage. NFIP offers Increased Cost of Compliance coverage, which provides up to $30,000 for policyholders to fund mitigation efforts on their property if they experience substantial damage or if their structure is an RL property. Between 1997 and 2014, the vast majority (99 percent) of Increased Cost of Compliance claims met the substantially damaged property definition, according to a 2017 report from the University of Pennsylvania. Unlike CDBG, which is awarded to states and local governments, Increased Cost of Compliance is awarded directly to individuals. According to FEMA, it is eligible as an HMA nonfederal cost share because it is considered a direct contract between the insurer and policyholder. FEMA allows recipients to assign their funds to the community as part of a collective mitigation project, and the community is then obligated to provide HMA funding to any property owner who contributed Increased Cost of Compliance dollars toward the nonfederal cost share. As of September 2019, FEMA had closed more than 38,000 Increased Cost of Compliance claims with dates of loss since 1997, totaling more than $877 million. Small Business Administration disaster loans. Small Business Administration disaster loans provide up to $200,000 for repairing or replacing a primary residence and $40,000 for repairing or replacing personal items that have been affected by a disaster. The interest rate cannot exceed 4 percent for applicants unable to access credit elsewhere, and cannot exceed 8 percent for all others. Secondary or vacation homes are not eligible, but qualified rental properties may be eligible under the Small Business Administration’s business disaster loan program, which offers loans of up to $2 million. According to FEMA guidance, these loans can serve as a source of cost share if HMA grants are disbursed early enough; however, the differing award timelines often make these funding sources incompatible. Further, disaster loans may not be eligible in conjunction with HMA funds due to duplication of benefits, but general-purpose Small Business Administration loans are not subject to this restriction, according to FEMA. In addition to FEMA’s three HMA programs, other federal, state, and local programs have helped acquire properties. Community Development Block Grants. In addition to its use as a cost- share complement to HMA grants, states and communities can use CDBG Disaster Recovery funding as a stand-alone source of property acquisition funds, according to the Department of Housing and Urban Development. Availability of CDBG Disaster Recovery funds is subject to supplemental appropriations following a presidential disaster declaration and must be used in response to that specific disaster. CDBG Disaster Recovery funds are disbursed to state and local governments and not to individuals directly. However, the governmental recipient can award CDBG Disaster Recovery funds to private citizens, nonprofits, economic development organizations, businesses, and other state agencies. The Bipartisan Budget Act of 2018 appropriated funding for CDBG, of which the Department of Housing and Urban Development allocated almost $6.9 billion for CDBG mitigation funds for the first time, as a result of the 2015 to 2017 disasters. Unlike CDBG Disaster Recovery funds, which the recipient must use in response to a specific disaster, recipients may use CDBG Mitigation funds to mitigate risks from future disasters. U.S. Army Corps of Engineers’ National Nonstructural Committee. The Army Corps of Engineers (Corps) conducts a range of mitigation measures through the National Nonstructural Committee, including acquisitions, elevations, relocations, and floodplain mapping. Nonstructural refers to measures that attempt to mitigate the consequences of floods, as opposed to structural measures intended to prevent floods from occurring. According to the Corps, except for limited research funding, it does not offer grants for flood risk management projects, and large projects generally require specific authorization from Congress. However, the Corps’ Continuing Authority Program allows it to execute smaller projects at its discretion. For example, for one of the programs, the federal government funds 65 percent of a project’s cost, and the project sponsor must provide all land, easement, rights-of-way, relocations, and disposal areas required for the project. The sponsor’s cost share includes credit for provision of the requirements above and pre-approved work-in-kind, but at least five percent must be provided in cash. Department of Agriculture’s Natural Resources Conservation Service Emergency Watershed Protection Program. The Federal Agriculture Improvement and Reform Act of 1996 enables the Emergency Watershed Protection Program to purchase floodplain easements on residential and agricultural land for flood mitigation purposes and to return the land to its natural state. For agricultural and residential land, this program pays up to the entire easement value and also funds property demolition or relocation, according to the Department of Agriculture. Land generally must have flooded in the past year or twice within the previous 10 years to be considered eligible. State and local acquisition programs. While state and local governments are active participants in federal acquisition projects, some have also developed their own acquisition programs. These programs vary on the extent to which they rely on federal funds, if at all. For example: The Harris County Flood Control District, a special purpose district, in Texas acquired about 3,100 properties between 1985 and 2017, according to a 2018 report from Rice University, using a combination of FEMA grants, Corps funds, and local dollars. Charlotte-Mecklenburg Storm Water Services, a joint city-county utility in North Carolina, has acquired more than 400 homes since 1999. Initially, it primarily used federal funds, but now it uses almost solely stormwater fees and other local revenue to fund acquisitions. The utility’s Quick Buys program allows it to acquire properties soon after a flood, before homeowners invest in repairs, whereas federal acquisitions often occur after property owners have begun rebuilding, according to FEMA officials. New Jersey, through its Blue Acres program, plans to acquire up to 1,300 properties damaged by Superstorm Sandy. The program has used state funds, including $36 million in bonds, as well as more than $300 million in federal funding received from multiple agencies. Since 1989, the primary means by which FEMA has mitigated flood risk at the property level has been by funding property acquisitions. Acquisitions accounted for about 75 percent of FEMA’s $5.4 billion in flood mitigation spending, adjusted for inflation, from 1989 to 2018 (see fig. 3). Most of the remaining spending was used to elevate properties, with smaller amounts used to floodproof and relocate properties. The average federal cost-per-property was $136,000 for acquisitions and $107,000 for elevations, according to 2008-2014 FEMA data. As seen in figure 4, FEMA-funded property acquisitions have fluctuated over time but have generally increased since FEMA’s HMA programs began. For example, from 1989 through 1992—the first four years of HMGP funding and prior to the creation of PDM and FMA—less than $8 million, adjusted for inflation, was obligated for property acquisitions each year, resulting in fewer than 200 acquisitions each year (see fig. 4). The highest acquisition funding generally was associated with years that had significant flood events, such as Superstorm Sandy (2012) and Hurricanes Harvey, Irma, and Maria (2017). From fiscal years 1989-2018, approximately $3.3 billion of property acquisition funding, adjusted for inflation, occurred through HMGP, resulting in the acquisition of 41,458 properties (see fig. 5). HMGP represented about 90 percent of all property acquisitions and 82 percent of all acquisition funding, with PDM and FMA representing the remainder. As a result, most FEMA-funded acquisitions occurred following flood events. Most of the funding, adjusted for inflation, for HMGP’s and PDM’s flood mitigation projects has been for property acquisition (83 percent and 89 percent of total funds, respectively), while most FMA funding has been for elevation (49 percent). Although FEMA mitigated more than 57,000 properties for flood risk from 1989 to 2018, including more than 46,000 through acquisition, the number of nonmitigated RL properties increased from 2009 to 2018. Figure 6 shows that this growth in the number of RL properties has outpaced efforts to mitigate their flood risk. From 2009 through 2018, FEMA’s inventory of new RL properties grew by 64,101. During this period, FEMA mitigated 4,436 RL properties through its three HMA programs, and an additional 15,047 were mitigated through other federal or state programs. As a result, the number of nonmitigated RL properties increased by 44,618—more than double the number of RL properties that were mitigated in that time period. States varied in the extent to which they mitigated high-risk properties, including RL properties, between 1989 and 2018. While FEMA does not require a property to be an RL property to receive flood mitigation funding, the number of properties mitigated by a state relative to its population of RL properties provides context to its flood mitigation progress. For example, some states with large numbers of RL properties, such as Texas, Louisiana, Florida, and New York, mitigated few properties relative to their numbers of RL properties (see table 2). Other states, such as Missouri and North Carolina, have far fewer RL properties but have mitigated more properties relative to their numbers of RL properties. States also varied in their methods for flood mitigation (see table 2). For example, while property acquisition accounted for 81 percent of mitigated properties nationwide, it represented closer to half of mitigated properties in Virginia, New Jersey, and Florida and only 19 percent in Louisiana. According to some FEMA and local officials, high property values in some regions can make acquisitions cost prohibitive and other mitigation methods such as elevation more attractive because they do not incur the cost of purchasing the land. Many other factors could affect mitigation, including homeowners’ preferences. Further, the voluntary nature of FEMA’s HMA programs may limit states’ ability to acquire properties with known flood risk. According to FEMA, acquisition permanently addresses flood risk because, unlike elevation or floodproofing, it moves individuals and structures away from flood risk rather than mitigating a structure in place. In a subsequent report, we plan to explore in more detail the factors, including homeowner demand for acquisition, that have affected the extent to which states have used acquisition to mitigate flood risk. NFIP represents a fiscal exposure to the federal government because its premium rates have not kept pace with the flood risk of the properties it insures. Addressing this imbalance would mean reducing the flood risk of the insured properties, increasing premium revenue, or some combination of both. Despite FEMA’s efforts to mitigate its insured properties’ flood risk, premium rates for many properties do not reflect the full estimated risk of loss. As we have reported previously, mitigation alone will not be sufficient to resolve NFIP’s financial challenges; structural reforms to the program’s premium rates will also be necessary. NFIP’s total annual flood claim payments have grown in recent years, potentially indicating an increase in flood risk. For example, the eight years of the highest annual NFIP claims have all occurred since 2004, with particularly catastrophic flood events accounting for much of these claims: In 2005, claims reached $17.8 billion ($23.3 billion, adjusted for inflation), largely due to Hurricanes Katrina, Rita, and Wilma. In 2012, claims reached $9.6 billion ($10.7 billion, adjusted for inflation), largely due to Superstorm Sandy. In 2017, claims reached $10.5 billion ($11.0 billion, adjusted for inflation), largely due to Hurricanes Harvey, Irma, and Maria. These severe weather events appear to be contributing to the long-term increases in claims paid by NFIP, as would be expected with infrequent but severe events. As seen in figure 7, the amount of claims paid per policy, adjusted for inflation, does not show a steady increase in claims but rather substantial spikes in certain years associated with catastrophic flooding events. RL properties have contributed heavily to NFIP’s claims and, as noted earlier, the number of RL properties continues to rise despite FEMA’s mitigation efforts. Of the $69.7 billion in claims NFIP paid out from 1978 to 2019, $22.2 billion was for flood damage sustained by RL properties (32 percent). The frequency and intensity of extreme weather events, such as floods, are expected to increase in coming years due to climate change, according to the U.S. Global Change Research Program and the National Academies of Sciences. Further, numerous studies have concluded that climate change poses risks to many environmental and economic systems and a significant financial risk to the federal government. For example, according to the November 2018 National Climate Assessment report, the continued increase in the frequency and extent of high-tide flooding due to sea level rise threatens America’s trillion-dollar coastal property market. According to the National Oceanic and Atmospheric Administration, minor flood events (sometimes referred to as nuisance flooding) also are projected to become more frequent and widespread due to climate change. While it is uncertain the exact extent to which flood risk has changed and will continue to change, NFIP’s fiscal exposure will persist as long as premium rates do not keep pace with flood risk. As we have been reporting since 1983, NFIP’s premium rates do not reflect the full risk of loss because of various legislative requirements and FEMA practices. To set premium rates, FEMA considers several factors, including location in flood zones, elevation of the property relative to the community’s base flood elevation, and characteristics of the property, such as building type, number of floors, presence of a basement, and year built relative to the year of the community’s original flood map. Most NFIP policies have premium rates that are deemed by FEMA to be full-risk rates, which FEMA defines as sufficient to pay anticipated losses and expenses. However, FEMA’s overall rate structure may not reflect the full long-term estimated risk of flooding, as discussed below. Subsidized rates. NFIP offers some policyholders subsidized rates—that is, rates that intentionally do not reflect the full risk of flooding. These premium rates are intended to encourage the widespread purchase of flood insurance by property owners and encourage floodplain management by communities. Subsidized rates generally are offered to properties in high-risk locations (special flood hazard areas) that were built before flood maps were created. FEMA staff said they have begun increasing rates for certain subsidized properties as prescribed under the Biggert-Waters Flood Insurance Reform Act of 2012 and the Homeowner Flood Insurance Affordability Act of 2014. In addition, the percentage of subsidized policies is decreasing. According to FEMA data, the percentage of NFIP policies receiving subsidized rates dropped from about 22 percent in July 2013 to about 17 percent in June 2019. In 2013, we recommended that FEMA obtain elevation information to determine full-risk rates for subsidized properties. As of January 2020, FEMA had not fully implemented this recommendation but was in the process of doing so. For example, FEMA had requested proposals from third-party vendors for obtaining the elevation information and was reviewing these proposals. This information remains necessary for FEMA to determine the adequacy of its premium rates and the costs of any subsidization. It will also allow Congress and the public to understand the amount of unfunded subsidization within the program and the federal fiscal exposure it creates. Grandfathered rates. FEMA allows some property owners whose properties are remapped into higher-risk flood zones to continue to pay the premium rate from the lower-risk zone. FEMA data show that about 9 percent of NFIP policies were receiving a grandfathered rate as of June 2019. In 2008, we recommended that FEMA collect data to analyze the effect of grandfathered policies on NFIP’s fiscal exposure. As of February 2020, FEMA officials said they had not fully implemented this recommendation but were in the process of doing so. The officials told us they had finished collecting data on grandfathered policies and that they planned to analyze it as they completed efforts to update their premium rate setting approach. Collection and analysis of data on grandfathered policies will help FEMA understand and communicate the extent to which these policies are contributing to NFIP’s fiscal exposure. Rates designated full-risk. As we reported in 2008 and 2016, it is unclear whether premiums FEMA considers to be full-risk actually reflect the full long-term estimated risk of loss. For example, NFIP full-risk premium rates do not fully reflect the risk of catastrophic losses or the expenses associated with managing them. Private insurers typically manage catastrophic risk using capital, reinsurance, and other instruments, such as catastrophe bonds, and include the associated expenses in premium rates. By contrast, FEMA has traditionally managed catastrophic risk by relying on its authority to borrow from Treasury. In January 2017, FEMA began purchasing reinsurance to transfer some of its flood risk exposure to the private reinsurance market. However, FEMA has not accounted for these expenses in setting its NFIP premium rates. Reinsurance could be beneficial because it would allow FEMA to recognize some of its flood risk and the associated costs up front through the premiums it must pay to the reinsurers rather than after the fact in borrowing from Treasury. However, because reinsurers must charge FEMA premiums to compensate for the risk they assume, reinsurance’s primary benefit would be to manage risk rather than to reduce NFIP’s expected long-term fiscal exposure. Congress has directed FEMA to provide discounted premium rates to promote affordability for policyholders but did not provide FEMA with dedicated funds to pay for these subsidies. As a result, premium revenue has been insufficient to pay claims in some years, requiring borrowing from Treasury to make up for the shortfall. While Congress passed reforms to NFIP in 1994 and 2004, neither set of actions sufficiently addressed program revenue. In 2005, Hurricanes Katrina, Rita, and Wilma hit the Gulf Coast and resulted in NFIP borrowing nearly $17 billion from Treasury to pay claims (see fig. 8). In July 2012, Congress passed the Biggert-Waters Flood Insurance Reform Act, which contained significant reforms to NFIP’s premium rates. But a few months later, Superstorm Sandy occurred, pushing NFIP’s debt to $24 billion. Following policyholders’ concerns about the rate increases authorized by the 2012 act, Congress slowed the pace of many of these rate increases in 2014 with the Homeowner Flood Insurance Affordability Act. In the fall of 2017, Hurricanes Harvey, Irma, and Maria occurred, prompting additional borrowing from Treasury and causing NFIP to reach its borrowing limit. In response, Congress canceled $16 billion of NFIP’s debt in October 2017, which allowed NFIP to pay claims from these storms. Since September 2017, NFIP has been operating under a series of short-term authorizations, the most recent of which expires in September 2020. As of March 2020, NFIP’s debt remained at $20.5 billion. To improve NFIP’s solvency and enhance the nation’s resilience to flood risk, we suggested in 2017 that Congress could make comprehensive reforms that include actions in six areas. We reported that it was unlikely that FEMA would be able to repay its debt and that addressing it would require Congress to either appropriate funds or eliminate the requirement that FEMA repay the accumulated debt. However, eliminating the debt without addressing the underlying cause of the debt—insufficient premium rates—would leave the federal taxpayer exposed to a program requiring repeated borrowing. To address NFIP’s fiscal exposure, there are two general approaches: decrease costs or increase revenue. Decreasing costs to the program in the form of claims involves mitigating insured properties’ flood risks. Mitigation can be very costly, but there will be some properties for which the cost to mitigate will be outweighed by the benefit of reduced flood risk and, ultimately, fiscal exposure. Mitigation may be a cost-effective option for those properties for which full-risk rates would be cost-prohibitive. Increasing revenue would require reforms to NFIP’s premium rates. FEMA has begun increasing rates on subsidized properties. But, as we suggested in 2017, Congress could remove existing legislative barriers to FEMA’s premium rate revisions. Members of Congress and others have raised concerns about such reforms because raising premium rates may make coverage unaffordable for some policyholders. To address these concerns, we suggested that all policies include full-risk premium rates, with targeted, means-based, appropriated subsidies for some policies. This would improve the program’s solvency while also addressing affordability concerns. Assigning full-risk premium rates to all policies would remove subsidies from those who do not need them, helping improve solvency. It would also more accurately signal the true flood risk to property owners and enhance resilience by incentivizing mitigation measures, such as acquisition. Means-based subsidies would ensure that property owners who needed help would get it, and an explicit appropriation for the subsidies would make their true cost transparent to taxpayers. We maintain that a comprehensive approach that includes mitigation and rate reform is needed to address NFIP’s fiscal exposure. Because several categories of NFIP premium rates do not reflect the full risk of flood loss, FEMA has had to borrow $36.5 billion from Treasury to pay claims from several catastrophic flood events since 2005. To address this, some have suggested additional funding to mitigate RL properties. While we acknowledge that mitigation is part of the solution, we maintain that a more comprehensive approach is necessary to address the program’s fiscal exposure. We have made two recommendations to FEMA that, if implemented, could help inform Congress’ efforts to reform NFIP. In 2008, we recommended that FEMA collect information on grandfathered properties and analyze their financial effect on NFIP, and in 2013, we recommended that FEMA obtain elevation information on subsidized properties. By implementing these recommendations, FEMA would better understand NFIP’s fiscal exposure and be able to communicate this information to Congress. Further, we suggested in 2017 that Congress take a comprehensive approach to reforming NFIP. One important first step would be to implement full-risk premium rates for all policies, with appropriated means-based subsidies for some policies. Full-risk premium rates would remove subsidies from those who do not need them, helping improve solvency, and also more accurately signal the true flood risk to property owners and incentivize efforts to mitigate flood risk. Further, means- based subsidies would ensure that property owners who need help will get it, and having Congress explicitly appropriate for the subsidies would make the true cost of the subsidy transparent to taxpayers. While this would be an important step to putting NFIP on a sustainable path, comprehensive reform of the program should also address the other issues we have identified, including mitigating the flood risk of insured properties. We provided a draft of this report to the Department of Homeland Security for its review and comment. The agency provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report addresses the Federal Emergency Management Agency’s (FEMA) National Flood Insurance Program (NFIP). Our objectives were to examine (1) funding programs available for property acquisitions, (2) FEMA’s flood mitigation efforts, and (3) factors contributing to NFIP’s fiscal exposure. To describe funding programs available for property acquisitions, we reviewed authorizing legislation, the Code of Federal Regulations, and FEMA guidance and manuals, including the Hazard Mitigation Assistance Guidance and Cost Share Guide, to identify program characteristics, eligibility requirements, and application guidelines. To identify funding for these programs, we analyzed FEMA’s project-level Hazard Mitigation Assistance (HMA) data from its Enterprise Applications Development Integration and Sustainment system, which FEMA uses to track mitigation projects funded through its HMA grant programs. To summarize Increased Cost of Compliance coverage, which NFIP policyholders can use to fund mitigation efforts, we analyzed FEMA’s NFIP claims database to identify the number and amount of such claims. We also interviewed the FEMA officials responsible for administering these grant programs. Further, we identified other federal agency programs that can fund property acquisitions or meet cost share requirements and reviewed their authorizing legislation and their relevant federal regulations. Finally, to identify examples of state and local programs that have been used to fund property acquisitions, we reviewed academic reports, including from the University of North Carolina and Rice University. To review FEMA’s flood mitigation efforts, we analyzed FEMA’s project- level HMA data from the “Mitigation Universe” of its Enterprise Applications Development Integration and Sustainment system. We analyzed several variables in this dataset, including number of properties, federal share obligated, mitigation type category, grant program area, grant program fiscal year, and state. For the analyses by mitigation type category, we excluded projects (79 percent of the total records) that did not include a flood mitigation activity (those with values of “Other” or “Pure Retrofit”). Of the remaining records, 98 percent were “Pure,” meaning all properties within each project were of a single mitigation method type (acquisition, elevation, floodproof, or relocation). The remaining 2 percent were “Mixed,” indicating a project contained at least one acquisition and at least one elevation but could also contain other mitigation methods. For analyses by grant program area, we treated projects funded through the Severe Repetitive Loss and Repetitive Flood Claims grant programs as being part of the Flood Mitigation Assistance program and projects funded through the Legislative Pre-Disaster Mitigation program as being part of the Pre- Disaster Mitigation program. For data on the number of flood mitigated properties, we used the final number of properties mitigated by a project. For data on funding, we used the federal share of the project’s obligated funding. To analyze mitigated and nonmitigated repetitive loss (RL) properties, we summarized FEMA’s RL property mitigation report, which tracked the cumulative number of RL properties by year from June 2009 through June 2018. To describe the number of RL properties by state, we analyzed FEMA’s list of RL properties as of August 31, 2019, which included every property that at any point FEMA had designated as an RL property under any of its three definitions. The list included properties that had since been mitigated, as well as those that are no longer insured by NFIP. To examine factors contributing to NFIP’s fiscal exposure, we analyzed FEMA’s claims dataset as of September 30, 2019. This dataset includes the more than 2 million claims paid to NFIP policyholders since the beginning of the program. We excluded records whose status was “open” or “closed without payment.” Further, we excluded records whose year of loss was before 1978 because FEMA officials told us that that was the first year they considered their claims data to be reliable and complete. To identify factors that contribute to NFIP’s fiscal exposure and illustrate how this fiscal exposure has materialized and changed over time, we reviewed several of our previous reports and the Department of the Treasury’s statements of public debt. Finally, to summarize how flood risk could change in the future, we reviewed our previous reports on climate change. In general, we adjusted for inflation any dollar figures that we compared or aggregated across multiple years and indicated this accordingly. To do this, we used the Bureau of Labor Statistics’ Consumer Price Index for All Urban Consumers. To assess the reliability of all of the datasets we analyzed for this report, we requested and reviewed preliminary versions of the data and accompanying data dictionaries. We used the data dictionary to identify potential variables for use in our analyses and output statistics on these variables (e.g., frequencies of values, number of blanks or zero values, minimum, maximum, and mean) to identify any potential reliability concerns such as outliers or missing values. We met with relevant FEMA officials to discuss each of the data sets to understand how FEMA collected, used, and maintained the data; the reliability and completeness of key variables; reasons for any potential discrepancies we identified; and whether our understanding of the data and approach to analyzing them were accurate and reasonable. After these meetings, we requested updated versions of the data and updated our analyses accordingly. We determined that all data elements we assessed were sufficiently appropriate and reliable for this report’s objectives. We conducted this performance audit from January 2019 to June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. January 1983: We recommended that FEMA improve its rate-setting process to ensure adequate income for NFIP and suggested that Congress either limit FEMA’s borrowing for extraordinary losses or establish an emergency fund for such losses, and pay for NFIP subsidies with appropriations. March 1994: We found that NFIP’s premium income was insufficient to meet expected future losses because of subsidized rates and suggested that Congress consider how any changes in premium rates would affect policyholder participation. September 1994: National Flood Insurance Reform Act. Developed a mitigation assistance program and expanded the mandatory purchase requirement. June 2004: Flood Insurance Reform Act. Authorized grant programs to mitigate properties that experienced repetitive flooding losses. August-October 2005: Hurricanes Katrina, Rita, Wilma. Caused $17.1 billion in NFIP claims. FEMA debt to Treasury increased to $16.9 billion in fiscal year 2006. March 2006: We added NFIP to our high-risk list. October 2008: We recommended that FEMA collect data to analyze the effect of grandfathered policies on NFIP’s fiscal exposure. November 2008: We identified three options for addressing the financial impact of subsidies: increasing mitigation efforts; eliminating or reducing subsidies; and targeting subsidies based on need. June 2011: We suggested that Congress allow NFIP to charge full- risk premium rates to all property owners and provide assistance to some categories of owners to pay those premiums. July 2012: Biggert-Waters Flood Insurance Reform Act. Required FEMA to increase rates for certain subsidized properties and grandfathered properties; create a NFIP reserve fund; and improve flood risk mapping. October 2012: Superstorm Sandy. Caused $8.8 billion in NFIP claims. FEMA debt to Treasury increased to $24 billion in fiscal year 2013. February 2013: We added limiting the federal government’s fiscal exposure by better managing climate change risks to our high-risk list. July 2013: We recommended that FEMA obtain elevation information to determine full-risk rates for subsidized policyholders. March 2014: Homeowner Flood Insurance Affordability Act. Reinstated certain rate subsidies removed by the Biggert-Waters Flood Insurance Reform Act of 2012; established a new subsidy for properties that are newly mapped into higher-risk zones; restored grandfathered rates; and created a premium surcharge that would be deposited into the NFIP reserve fund. October 2014: We recommended that FEMA amend NFIP minimum standards for floodplain management to encourage forward-looking construction and rebuilding efforts that reduce long-term risk and federal exposure to losses. July 2015: We recommended that the Mitigation Framework Leadership Group establish an investment strategy to identify, prioritize, and guide federal investments in disaster resilience and hazard mitigation-related activities. August-October 2016: Hurricane Matthew and Louisiana floods. Caused $3.1 billion in NFIP claims. FEMA debt to Treasury debt increased to $24.6 billion in early fiscal year 2017. April 2017: We suggested that Congress make comprehensive reforms to NFIP that include actions in six areas: (1) addressing the debt; (2) removing legislative barriers to full-risk premium rates; (3) addressing affordability; (4) increasing consumer participation; (5) removing barriers to private-sector involvement; and (6) protecting NFIP flood resilience efforts. August-September 2017: Hurricanes Harvey, Irma, and Maria. Caused $10 billion in NFIP claims. FEMA reached the limit of its Treasury borrowing authority of $30.4 billion. September 2017: NFIP’s last long-term authorization ended, resulting in a string of short-term reauthorizations. October 2017: Congress canceled $16 billion of NFIP’s debt to enable FEMA to continue paying flood claims. This reduced FEMA’s debt to Treasury to $20.5 billion. March 2020: FEMA’s debt to Treasury remained at $20.5 billion. September 2020: NFIP’s current short-term authorization ends. In addition to the contact named above, Patrick Ward (Assistant Director), Christopher Forys (Analyst in Charge), Emily Bond, Christina Cantor, William Chatlos, Eli Dile, Lijia Guo, Holly Halifax, Laura Ann Holland, Yann Panassie, Stephen Ruszczyk, Jessica Sandler, Joseph Silvestri, Jena Sinkfield, and Kelsey Wilson made key contributions to this report.", "summary": "NFIP has faced significant financial challenges over the years, highlighted by a rise in catastrophic flood events and its $20.5 billion debt to Treasury. Contributing to these challenges are repetitive loss properties—those that have flooded and received a claim payment multiple times. Acquiring and demolishing these properties is one alternative to paying for repeated claims, but questions exist about the cost, efficiency, and effectiveness of this approach. GAO was asked to review FEMA's property acquisition efforts as a means of addressing NFIP's financial challenges. This report examines (1) funding programs available for acquisitions, (2) FEMA's flood mitigation efforts, and (3) factors contributing to NFIP's fiscal exposure. To conduct this work, GAO reviewed FEMA guidance and other documentation; analyzed FEMA data sets related to NFIP policies and claims, repetitive loss properties, and mitigation projects; and interviewed FEMA officials. The Federal Emergency Management Agency (FEMA) administers three grant programs that can fund efforts to mitigate the flood risk of properties insured by the National Flood Insurance Program (NFIP). Together, these three programs funded $2.3 billion in mitigation projects from fiscal years 2014 through 2018. The largest program's funding is tied to federal recovery dollars following presidential disaster declarations, while the other two programs are funded each year through congressional appropriations. States and localities generally must contribute 25 percent of the cost of a mitigation project, but some other federal program funds can be used for that purpose. One example of such a project is property acquisition—purchasing a high-risk property from a willing property owner, demolishing the structure, and converting the property to green space. From 1989 to 2018, FEMA has helped states and localities mitigate more than 50,000 properties; however, the number of nonmitigated repetitive loss properties (generally meaning those that flooded at least twice in 10 years) has grown. Mitigation efforts varied by state. Property acquisition accounted for about 80 percent of mitigated properties nationwide, but, in some states, elevation (raising a structure) was more commonly used. In addition, some states (e.g., Missouri and North Carolina) mitigated a high number of properties relative to their numbers of repetitive loss properties, while others (Florida, New York, Louisiana, and Texas) mitigated a low number. While these efforts can reduce flood risk and claim payments, the federal government's fiscal exposure from NFIP remains high because premium rates do not fully reflect the flood risk of its insured properties. NFIP has experienced several catastrophic flood events in recent years, and the frequency and severity of floods is expected to increase. However, NFIP's premium rates have not provided sufficient revenue to pay claims. As a result, FEMA still owed Treasury $20.5 billion as of March 2020, despite Congress cancelling $16 billion of debt in 2017. As GAO has reported in the past (GAO-17-425), Congress will need to consider comprehensive reform, including mitigation and structural changes to premium rates, to ensure NFIP's solvency. GAO suggested in GAO-17-425 that Congress make comprehensive reforms to NFIP to improve the program's solvency. Given NFIP's continued debt growth, GAO maintains that comprehensive reform warrants consideration.", "document_type": "gao"}
{"report": "The NASA Authorization Act of 2010 directed NASA to develop SLS, to continue development of a crew vehicle, and to prepare infrastructure at Kennedy Space Center to enable processing and launch of the launch system. To fulfill this direction, NASA formally established the SLS launch vehicle program in 2011. Then, in 2012, NASA aligned the requirements for the Orion program with those of the newly created SLS and EGS programs. Figure 1 provides details about each SLS hardware element and its source as well as identifies the major portions of the Orion spacecraft. In order to facilitate Congressional oversight and track program progress, NASA establishes an agency baseline commitment—the cost and schedule baselines against which the program may be measured—for all projects that have a total life cycle cost of $250 million or more. NASA refers to these projects as major projects or programs. When the NASA Administrator determines that development cost growth within a major project or program is likely to exceed the development cost estimate by 15 percent or more, or a program milestone is likely to be delayed from the baseline’s date by 6 months or more, NASA replans the project and submits a report to this committee—the Committee on Science, Space, and Technology of the House of Representatives—and the Committee on Commerce, Science, and Transportation of the Senate. Should a major project or program exceed its development cost baseline by more than 30 percent, the program must be reauthorized by the Congress and rebaselined by NASA in order for the contractor to continue work beyond a specified time frame. NASA tied the SLS and EGS program cost and schedule baselines to the uncrewed first mission—known now as Artemis-1—originally planned for November 2018. The Orion program’s cost and schedule baselines are tied to a crewed second mission—known as Artemis-2—planned for April 2023. In April 2017, we found that given combined effects of ongoing technical challenges in conjunction with limited cost and schedule reserves, it was unlikely that these three programs would achieve the originally committed November 2018 launch readiness date. Cost reserves are for costs that are expected to be incurred—for instance, to address project risks—but are not yet allocated to a specific part of the project. Schedule reserves are extra time in project schedules that can be allocated to specific activities, elements, and major subsystems to mitigate delays or address unforeseen risks. We recommended that NASA confirm whether the November 2018 launch readiness date was achievable and, if warranted, propose a new, more realistic Artemis-1 date and report to Congress on the results of its schedule analysis. NASA agreed with both recommendations and stated that it was no longer in its best interest to pursue the November 2018 launch readiness date. Subsequently, NASA approved a new Artemis-1 schedule of December 2019, with 6 months of schedule reserve available to extend the date to June 2020, and revised the costs that it expects to incur (see table 1). In June 2019, we found that within 1 year of announcing a delay for the first human spaceflight mission, senior NASA officials acknowledged that the revised Artemis-1 launch date of December 2019 was unachievable and the June 2020 launch date (which takes into account schedule reserves) was unlikely. These officials estimated that there were 6 to 12 months of schedule risk associated with this later date, which means the first launch may occur as late as June 2021 if all risks are realized. As we found in June 2019, this would be a 31-month delay from the schedule originally established in the programs’ baselines. Officials attributed the additional schedule delay to continued production challenges with the SLS core stage and the Orion crew and service modules. NASA officials also stated that the 6 to 12 months of risk to the launch date accounts for the possibilities that SLS and Orion testing and final cross-program integration and testing at Kennedy Space Center may result in further delays. As we noted in our report, these 6 to 12 months of schedule risk do not include the effects, if any, of the federal government shutdown that occurred in December 2018 and January 2019. In commenting on our June 2019 report, NASA stated that its Lunar 2024 planning activities would include an Artemis-1 schedule assessment. However, in July 2019, NASA reassigned its senior leaders responsible for human spaceflight programs. The NASA Administrator stated in August 2019 that, as a result, the agency does not plan to finalize schedule plans for Artemis-1 until new leadership is in place at the agency. Additional details follow on the status of each program, including cost, schedule, and technical challenges. SLS. As we found in June 2019, ongoing development issues with the SLS core stage—which includes four main engines and the software necessary to command and control the vehicle—contributed to the SLS program not being able to meet the June 2020 launch date. Officials from the SLS program and Boeing, the contractor responsible for building the core stage, provided several reasons for the delays. These reasons include the underestimation of the complexity of manufacturing and assembling the core stage engine section—where the RS-25 engines are mated to the core stage—and those activities have taken far longer than expected. Since our June 2019 report, based on our review of the program’s most recent status reports, NASA has reported progress across many parts of the SLS program. For example, NASA has delivered the four RS-25 engines to Michoud Assembly Facility. NASA has also completed qualification testing of all components of the boosters and reports that there is schedule margin remaining for the booster deliverables. In addition, NASA reports that Boeing has made continued progress and expects that the core stage will be complete and ready for testing in December 2019. Completion of the core stage will represent a significant milestone for the program. In June 2019, we found that that SLS program has been underreporting its development cost growth since the December 2017 replan. This underreporting is because of a decision to shift some costs to future missions while not adjusting the baseline costs downward to reflect this shift. The SLS development cost baseline established in August 2014 for Artemis-1 includes cost estimates for the main vehicle elements—stages, liquid engines, boosters—and other areas. According to program officials, because of the December 2017 replan process, NASA decided that costs included as part of the SLS Artemis-1 baseline cost estimate would be more appropriately accounted for as costs for future flights. Thus, NASA decided not to include those costs, approximately $782 million, as part of the revised SLS Artemis-1 cost estimate. However, NASA did not lower the $7 billion SLS development cost baseline to account for this significant change in assumptions and shifting of costs to future flights. This decision presents challenges in accurately reporting SLS cost growth over time. NASA’s decision not to adjust the cost baseline downward to reflect the reduced mission scope obscures cost growth for Artemis-1. In June 2019, we found that NASA’s cost estimate as of fourth quarter fiscal year 2018 for the SLS program indicated development cost growth had increased by $1 billion, or 14.7 percent. However, our analysis showed that development cost growth actually increased by $1.8 billion or 29.0 percent, when the development baseline is lowered to account for the reduced mission scope. Essentially, NASA is holding the baseline costs steady, while reducing the scope of work included in current cost estimates (see figure 2). As NASA determines its new schedule for the first mission, it is likely this cost growth will increase as additional time in the schedule leads to additional costs. In our June 2019 report, we recommended that the SLS program calculate its development cost growth using a baseline that is appropriately adjusted for scope and costs NASA has determined are not associated with the first flight, and determine if the development cost growth has increased by 30 percent or more. NASA agreed with the recommendation and NASA officials stated that they plan to implement the recommendation when new leadership is in place for the human space exploration programs. Looking ahead, based on our review of the program’s most recent status reports, completing core stage manufacturing and integration and green run testing will be the critical path—the path of longest duration through the sequence of activities in the schedule—for the SLS program. During green run testing, NASA will fuel the completed core stage with liquid hydrogen and liquid oxygen and fire the integrated four main engines for about 500 seconds. The green run test carries risks because it is the first time that several things are being done beyond just this initial fueling. For example, it is also the first time NASA will fire the four main engines together, test the integrated engine and core stage auxiliary power units in flight-like conditions, and use the SLS software in an integrated flight vehicle. In addition, NASA will conduct the test on the Artemis-1 flight vehicle hardware, which means the program would have to repair any damage from the test before flight. Orion. While the Orion program’s schedule performance is measured only to the Artemis-2 mission, we found in June 2019 that the program was not on schedule to support the June 2020 launch date for the first mission. This was due to delays with the European Service Module and component issues for the avionics systems for the crew module, including issues discovered during testing. We found that these specific problems were resolved by the time of our report, but had already contributed to the inability of the program to meet the June 2020 launch date. Since we last reported, as of August 2019, the Orion program has completed significant events including completing the crew module and the service module prior to integration and conducting a test to demonstrate the ability to abort a mission should a life-threatening failure occur during launch. The program is tracking no earlier than October 2020 for an Artemis-1 launch date but that does not reflect the ongoing agency-wide schedule assessment noted above. In June 2019, we found that the Orion program has reported development cost growth but is not measuring that growth using a complete cost estimate. In summer 2018, the Orion program reported development cost growth of $379 million, or 5.6 percent above its $6.768 billion development cost estimate. Program officials explained that the major drivers of this cost growth were the slip of the Artemis-1 launch date, which reflected delays in the delivery of the service module; Orion contractor underperformance; and NASA-directed scope increase. However, during our review, Orion program officials originally stated that this cost estimate assumes an Artemis-2 launch date of September 2022, which is 7 months earlier than the program’s agency baseline commitment date of April 2023 that forms the basis for commitments between NASA, the Congress, and Office of Management and Budget. Subsequently, during the review, program officials told us that its cost projections fund one of those 7 months. In either case, NASA’s current cost estimate for the Orion program is not complete because it does not account for costs that NASA would incur through April 2023. As of September 2019, the program was targeting October 2022 for the Artemis-2 launch. In June 2019, we recommended that the Orion program update its cost estimate to reflect its committed Artemis-2 baseline date of April 2023. In its response, NASA partially agreed with our recommendation. NASA stated that providing the estimate to the forecasted launch date— September 2022—rather than to the committed baseline date of April 2023 is the most appropriate approach. However, by developing cost estimates only to the program’s goals and not relative to the established baseline, the Orion program is not providing NASA or the Congress the means of measuring progress relative to the baseline. We continue to believe that NASA should fully implement this recommendation. Looking ahead, based on our review of the program’s most recent status reports, there is an emerging issue that may delay schedule further for the first mission. Namely, there is the risk of damage to the Orion capsule during travel to and from integrated testing at Plum Brook Station in Ohio. The program office is studying whether it will be able to safely transport the integrated crew and service modules via the Super Guppy airplane as planned or if it will have to use an alternate airplane. We will continue to monitor this effort. Beyond Artemis-1, the Orion program must also complete development efforts for future missions. For example, the Artemis-2 crew module will need environmental control and life support systems, system updates from Artemis-1, and updated software to run these new elements. EGS. At the time of our June 2019 report, the EGS program was expecting to have facilities and software ready by the planned June 2020 launch date. We found that the program had overcome many challenging development hurdles that led to previous schedule delays. These hurdles included completing and moving the Mobile Launcher—a platform that carries the rocket to the launch pad and includes a number of connection lines that provide SLS and Orion with power, communications, coolant, fuel, and stabilization prior to launch—into the Vehicle Assembly Building for the multi-element verification and validation processes. Since our June 2019 report, the program is now targeting an Artemis-1 launch date of August 2020. According to NASA officials, the delay is primarily driven by challenges encountered installing ground support equipment on the Mobile Launcher and developing software, and does not reflect the ongoing agency-wide schedule assessment. The program has operated within the costs established for the June 2020 launch date, $3.2 billion, but officials stated that NASA is reevaluating the program’s development cost performance and will establish an updated baseline when new leadership is in place. Moving forward, based on our review of the program’s most recent status reports, the program has to complete the multi-element verification and validation process for the Mobile Launcher and Vehicle Assembly Building and complete its two software development efforts. Additionally, the EGS program is responsible for the final integration of the three programs. NASA officials stated that the 6 to 12 months of risk to the June 2020 launch date includes risk associated with EGS completing this integration that includes test and checkout procedures after SLS and Orion components arrive. Officials explained that the EGS risk is based on a schedule risk analysis that considered factors such as historical pre- launch integrated test and check out delays and the learning curve associated with a new vehicle. As previously stated, our prior work has shown that the integration and test phase often reveals unforeseen challenges leading to cost growth and schedule delays. NASA is currently embarking on an aggressive goal to return humans to the lunar surface in 2024. To achieve this goal, NASA not only needs SLS, Orion, and EGS to have completed their first two test missions, but is also developing several new systems. These new systems include a Lunar Gateway that will orbit the moon, landers that will transport astronauts from the Gateway to the lunar surface, and new space suits. Human spaceflight projects face inherent technical, design, and integration risks because they are complex, specialized, and are pushing the state of the art in space technology. Moreover, these programs can be very costly and span many years, which means they may also face changes in direction from Administrations and the Congress. Meeting the 2024 goal will also be challenging given the effort needed to better manage SLS, Orion, and EGS, coupled with the addition of the new programs, which are likely to compete for management attention and resources. Nevertheless, our past work has identified a range of actions that NASA can take to better position its human spaceflight programs for success. Today I would like to highlight three lessons from the SLS, Orion, and EGS programs that NASA can apply to improve the management of its human spaceflight programs. Enhance Contract Management and Oversight to Improve Program Outcomes. Over the past several years, we and the NASA Office of the Inspector General have identified shortcomings related to NASA’s management and oversight of its human spaceflight contracts. These shortcomings have left NASA ill-positioned to identify early warning signs of impending schedule delays and cost growth, reap the potential benefits of competition, and achieve desired results through contractor incentives. In July 2014, we found that NASA allowed high-value modifications to the SLS contracts to remain undefinitized for extended periods—in one instance a modification remained undefinitized for 30 months. Undefinitized contract actions such as these authorize contractors to begin work before reaching a final agreement with the government on terms and conditions. We have previously found that while undefinitized contract actions may be necessary under certain circumstances, they are considered risky in part because the government may incur unnecessary costs if requirements change before the contract action is definitized. Because lack of agreement on terms of the modification prolonged NASA’s timeframes for definitizing, the establishment of contractor cost and schedule baselines necessary to monitor performance was delayed. Specifically, we found in July 2014 that, in most cases, the SLS program did not receive complete earned value management data derived from approved baselines on these SLS contracts. Earned value, or the planned cost of completed work and work in progress, can provide accurate assessments of project progress, produce early warning signs of impending schedule delays and cost overruns, and provide unbiased estimates of anticipated costs at completion. In July 2014, we also found the SLS program could be in a favorable position to compete contracts for the exploration upper stage, the upper stage engine, and advanced boosters that it expected to use on future variants of the launch vehicle. At that time, except for the RS- 25 engines, NASA’s contracting approach for the SLS program did not commit the program beyond the hardware needed for the second mission, and we found that moving forward the agency would be in a position to take advantage of the evolving launch vehicle market. We found that an updated assessment of the launch vehicle market could better position NASA to sustain competition, control costs, and better inform the Congress about the long-term affordability of the program. We recommended that before finalizing acquisition plans for future capability variants, NASA should assess the full range of competition opportunities and provide to the Congress the agency’s assessment of the extent to which development and production of future elements of the SLS could be competitively procured. NASA agreed with the recommendation, which we have identified as among those that warrant priority attention. Since we made that recommendation, NASA has awarded a sole- source contract for the upper stage engine and agency officials told us in July 2018 that they planned to incorporate additional booster development under the existing contract. This further limits an opportunity for competition for the program. Our body of work on contracting has shown that competition in contracting is a key element for achieving the best return on investment for taxpayers. We have found that promoting competition increases the potential for acquiring quality goods and services at a lower price and that noncompetitive contracts carry the risk of overspending because, among other reasons, they have been negotiated without the benefit of competition to help establish pricing. In July 2016, we found that the lack of earned value management data for the SLS Boeing core stage contract persisted. Without this information, some 4.5 years after contract award, the program continued to be in a poor position to understand the extent to which technical challenges with the core stage were having schedule implications or the extent to which they may have required reaching into the program’s cost reserves. In October 2018, the NASA Office of Inspector General reported that NASA does not require Boeing to report detailed information on development costs for the two core stages and exploration upper stage, making it difficult for the agency to determine if the contractor is meeting cost and schedule commitments for each deliverable. The NASA Office of Inspector General found that given the cost-reporting structure, the agency is unable to determine the cost of a single core stage. Internally, Boeing tracks all individual costs but submits a combined statement of labor hours and material costs through the one contract line item for all its development activities. NASA approximates costs based on numerous monthly and quarterly reviews with the contractor to track the progress of each individual deliverable. The NASA Office of Inspector General made a number of recommendations aimed at improving reporting relative to the core stage contract. Among these was a specific recommendation to separate each deliverable into its own contract line item number for tracking performance, cost, and award fees. NASA concurred with this recommendation and is currently renegotiating the core stage contract with Boeing. In June 2019, we found that NASA’s approach to incentivizing Boeing for the SLS stages and Lockheed Martin for the Orion crew spacecraft have not always achieved overall desired program outcomes. NASA paid over $200 million in award fees from 2014-2018 related to contractor performance on the SLS stages and Orion spacecraft contracts, but the programs continue to fall behind schedule and incur cost overruns. For example, in its December 2018 award fee letter to Boeing in which the contractor earned over $17 million in award fees, NASA’s fee determination official noted that the significant schedule delays on this contract have caused NASA to restructure the flight manifest for SLS. For the Lockheed Martin Orion contract, the contractor earned over $29 million for the award fee period ending April 2017. NASA noted that Lockheed Martin was not able to maintain its schedule for the crew service module and that the contractor’s schedule performance had decreased significantly over the previous year. In June 2019, we reported that our past work shows that when incentive contracts are properly structured, the contractor has profit motive to keep costs low, deliver a product on time, and make decisions that help ensure the quality of the product. Our prior work also shows, however, that incentives are not always effective tools for achieving desired acquisition outcomes. We have found that, in some cases, there are significant disconnects between contractor performance for which the contractor was awarded the majority of award fees possible without achieving desired program results. Additionally, we have found that some agencies did not have methods, data, or performance measures to evaluate the effectiveness of award fees. As part of our June 2019 report, we recommended that NASA direct the SLS and Orion programs to reevaluate their strategies for incentivizing contractors and determine whether they could more effectively incentivize contractors to achieve the outcomes intended as part of ongoing and planned contract negotiations. NASA agreed with the intent of this recommendation and stated that the SLS and Orion program offices reevaluate their strategies for incentivizing contract performance as part of contracting activities including contract restructures, contract baseline adjustments, and new contract actions. We will continue to follow-up on the actions the agency is taking to address this recommendation after its ongoing contract negotiations are complete. Minimize Risky Programmatic Decisions to Better Position Programs for Successful Execution. Through our reviews of NASA’s human spaceflight programs, we have found that NASA leadership has approved programmatic decisions that compound technical challenges. These decisions include approving cost and schedule baselines that do not follow best practices, establishing insufficient cost and schedule reserves, and operating under aggressive schedules. As a result, these programs have been at risk of cost growth and schedule delays since NASA approved their baselines. In July 2015, we found that NASA generally followed best practices in preparing the SLS cost and schedule baseline estimates for the limited portion of the program life cycle covered through launch readiness for the first test flight of SLS. However, we could not deem the cost estimate fully reliable because it did not fully meet the credibility best practice. While an independent NASA office reviewed the cost estimate developed by the program and as a result the program made some adjustments, officials did not commission the development of a separate independent cost estimate to compare to the program cost estimate to identify areas of discrepancy or difference. In addition, the program did not cross-check its cost estimate using an alternative methodology. The purpose of developing a separate independent cost estimate and cross-checking the estimate is to test the program’s cost estimate for reasonableness and, ultimately, to validate the cost estimate. In July 2016, we found that the Orion program’s cost and schedule estimates were not reliable based on best practices for producing high-quality estimates. For example, the cost estimate lacked necessary support and the schedule estimate did not include the level of detail required for high-quality estimates. Therefore, we recommended that NASA perform an updated joint cost and schedule confidence level analysis including updating cost and schedule estimates in adherence with cost and schedule estimating best practices, which we have identified as among those recommendations that warrant priority. NASA officials have stated that they have no plans to implement our recommendation. In commenting on the July 2016 report, NASA stated that the agency reviewed, in detail, the Orion integrated cost/schedule and risk analysis methodology and determined the rigor to be a sufficient basis for the agency commitments. However, without sound cost and schedule estimates, decision makers do not have a clear understanding of the cost and schedule risk inherent in the program or important information needed to make programmatic decisions. We continue to believe that NASA should fully implement our recommendation. In our 2017 High-Risk Report, we highlighted concerns that all three programs—SLS, Orion, and EGS—were operating with limited cost reserves, limiting each program’s ability to address risks and unforeseen technical challenges. For example, we found in July 2016 that the Orion program was planning to maintain low levels of cost reserves until later in the program. The lack of cost reserves at that time had caused the program to defer work to address technical issues to stay within budget. Also in our 2017 High-Risk Report, we highlighted concerns regarding each program managing to an aggressive internal NASA launch readiness date. This approach creates an environment for programs to make decisions based on reduced knowledge to meet a date that is not realistic. For example, the EGS program had consolidated future schedule activities to prepare the Mobile Launcher—the vehicle used to bring SLS to the launch pad—to meet its internal goal. The program acknowledged that consolidating activities—which included conducting verification and validation concurrent with installation activities—increased risk because of uncertainties about how systems not yet installed may affect the systems already installed. Officials added, however, that this concurrency is necessary to meet the internal schedule. Subsequently, as discussed above, NASA delayed its committed launch readiness date. Improve Transparency into Costs for Long-term Plans. As we previously reported, a key best practice for development efforts is that requirements need to be matched to resources (for example, time, money, and people) at program start. In the past, we have found that NASA programs, including the Constellation Program, did not have sufficient funding to match demanding requirements. Funding gaps can cause programs to delay or delete important activities and thereby increase risks. In addition, since May 2014, we have found there has been a lack of transparency into the long-term costs of these human spaceflight programs. As discussed above, the EGS and SLS programs do not have a cost and schedule baseline that covers activities beyond the first planned flight. In addition, as previously noted, the Orion program does not have a baseline beyond the second planned flight. As a result, NASA is now committing to spend billions of taxpayer dollars for missions that do not have a cost and schedule baseline against which to assess progress. To that end, we have made recommendations in the past on the need for NASA to baseline these programs’ costs for capabilities beyond the first mission; however, a significant amount of time has passed without NASA taking steps to fully implement these recommendations. Specifically, among those recommendations that we have identified as warranting priority attention, in May 2014, we recommended that, to provide Congress with the necessary insight into program affordability, ensure its ability to effectively monitor total program costs and execution, and to facilitate investment decisions, NASA should: Establish a separate cost and schedule baseline for work required to support the SLS for the second mission and report this information to the Congress through NASA’s annual budget submission. If NASA decides to fly the SLS configuration used in the second mission beyond that mission, we recommended that it establish separate life cycle cost and schedule baseline estimates for those efforts, to include funding for operations and sustainment, and report this information annually to Congress via the agency’s budget submission. Establish separate cost and schedule baselines for each additional capability that encompass all life cycle costs, to include operations and sustainment. This is important because NASA intends to use the increased capabilities of the SLS, Orion, and EGS well into the future. As part of the latter recommendation, we stated that, when NASA could not fully specify costs due to lack of well-defined missions or flight manifests, the agency instead should forecast a cost estimate range— including life cycle costs—having minimum and maximum boundaries and report these baselines or ranges annually to Congress via the agency’s budget submission. In its comments on our 2014 report, NASA partially concurred with these two recommendations, noting that much of what it had already done or expected to do would address them. For example, the agency stated that establishing the three programs as separate efforts with individual cost and schedule commitments met the intent of our recommendation. NASA also stated that its plans to track and report development, operations, and sustainment costs in its budget to Congress as the capabilities evolved would also meet the intent of the recommendation. In our response, we stated that while NASA’s prior establishment of three separate programs lends some insight into expected costs and schedule at the broader program level, it does not meet the intent of the two recommendations because cost and schedule identified at that level is unlikely to provide the detail necessary to monitor the progress of each block against a baseline. Further, we stated that reporting the costs via the budget process alone will not provide information about potential costs over the long term because budget requests neither offer all the same information as life-cycle cost estimates nor serve the same purpose. Life-cycle cost estimates establish a full accounting of all program costs for planning, procurement, operations and maintenance, and disposal and provide a long-term means to measure progress over a program’s life span. We continue to believe that NASA should fully implement these recommendations. As NASA considers these lessons, it is important that the programs place a high priority on quality, for example, holding suppliers accountable to deliver high-quality parts for their products through such activities as regular supplier audits and performance evaluations of quality and delivery. As we found in June 2019, both the SLS and Orion programs have struggled at times with the quality of parts and components. For example, the Orion contractor has had a number of issues with subcontractor-supplied avionics system components failing during testing that have required time to address. NASA has highlighted concerns over the contractor’s ability to manage its subcontractors and the resulting significant cost, schedule, and technical risk impacts to the program. And the SLS program faced setbacks after its contractor did not verify the processes that its vendors were using to clean the fuel lines, resulting in delays to resolve residue and debris issues. Chairwoman Horn, Ranking Member Babin, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any question that you may have at this time. If you or your staff have any questions about this testimony, please contact Cristina T. Chaplain, Director, Contracting and National Security Acquisitions at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this statement include Molly Traci, Assistant Director; John Warren; Sylvia Schatz; Ryan Stott; and Chad Johnson. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "NASA is undertaking a trio of closely related programs to continue human space exploration beyond low-Earth orbit. These three programs include a launch vehicle, a crew capsule, and the associated ground systems at Kennedy Space Center. All three programs are working towards a launch readiness date of June 2020 for the first mission. NASA then plans for these systems to support future human space exploration goals, which include seeking to land two astronauts on the lunar surface. GAO has a body of work highlighting concerns over NASA's management and oversight of these programs. This statement discusses (1) the cost and schedule status of NASA's human spaceflight programs and (2) lessons that NASA can apply to improve its management of its human spaceflight programs. This statement is based on eight reports issued from 2014 to 2019 and selected updates as of September 2019. For the updates, GAO analyzed recent program status reports on program progress. The National Aeronautics and Space Administration's (NASA) three related human spaceflight programs are in the integration and test phase of development, a phase of the acquisition process that often reveals unforeseen challenges leading to cost growth and schedule delays. Since GAO last reported on the status of these programs in June 2019, each program has made progress. For example, the Orion program conducted a key test to demonstrate the ability to abort a mission should a life-threatening failure occur during launch. As GAO found in June 2019, however, the programs continue to face significant schedule delays. In November 2018, within one year of announcing an up to 19-month delay for the three programs—the Space Launch System (SLS) vehicle, the Orion crew spacecraft, and Exploration Ground Systems (EGS)—NASA senior leaders acknowledged the revised launch date of June 2020 is unlikely. In addition, any issues uncovered during integration and testing may push the date as late as June 2021. Moreover, GAO found that NASA's calculations of cost growth for the SLS program is understated by more than 750 million dollars. GAO's past work has identified a number of lessons that NASA can apply to improve its management of its human spaceflight programs. For example, NASA should enhance contract management and oversight to improve program outcomes. NASA's past approach in this area has left it ill-positioned to identify early warning signs of impending schedule delays and cost growth or reap the benefits of competition. In addition, NASA's approach to incentivizing contractors through contract award fees did not result in desired outcomes for the SLS and Orion programs. Further, NASA should minimize risky programmatic decisions to better position programs for successful execution. This includes providing sufficient cost and schedule reserves to, among other things, address unforseen risk. Finally, realistic cost estimates and assessments of technical risk are particularly important at the start of an acquisition program. But NASA has historically provided little insight into the future cost of these human spaceflight programs, limiting the information useful to decision makers. GAO has made 19 recommendations in these eight prior reports to strengthen NASA's acquisition management of SLS, Orion, and EGS. NASA generally agreed with GAO's recommendations, and has implemented seven recommendations. Further action is needed to fully implement the remaining recommendations.", "document_type": "gao"}
{"report": "According to USDA, beginning with NSLP’s authorization in 1946, the federal government has gradually built an array of nutrition assistance programs designed to help the most vulnerable populations meet their food needs. Currently, eight of USDA’s nutrition assistance programs are targeted to providing food to children, as noted in table 1. USDA oversees the child nutrition programs at the federal level, and state agencies and local organizations play key roles in program administration and implementation. The Improper Payments Information Act of 2002 (IPIA), as amended, requires agencies to estimate improper payments for programs and activities identified as being susceptible to significant improper payments, implement corrective actions, and report on their results for these programs, among other things. An improper payment is any payment that should not have been made or that was made in an incorrect amount (including overpayments and underpayments) under statutory, contractual, administrative, or other legally applicable requirements. It includes any payment to an ineligible recipient, any payment for an ineligible good or service, any duplicate payment, any payment for a good or service not received (except for such payments where authorized by law), and any payment that does not account for credit for applicable discounts. Reducing improper payments—such as payments to ineligible recipients or duplicate payments—is critical to safeguarding federal funds. The Office of Management and Budget (OMB) provides guidance to federal agencies on effectively measuring, reporting, and reducing their improper payment rates. USDA reports annual improper payment estimates for four child nutrition programs: the school meals programs—NSLP and SBP—as well as WIC, and CACFP. IPIA, as amended, requires agencies to review all programs and activities at least once every 3 years and identify those that may be susceptible to significant improper payments. Federal law also requires agencies’ Inspectors General to annually assess and report on whether agencies complied with six criteria listed in the Improper Payments Elimination and Recovery Act of 2010 (IPERA), as amended, related to improper payments. These criteria are (1) publish an agency financial statement in the form required by OMB guidance; (2) conduct program- specific improper payment risk assessments, if required; (3) publish improper payment estimates, if required; (4) publish corrective action plans for programs and activities deemed susceptible to significant improper payments; (5) publish and meet annual improper payment reduction targets; and (6) report an improper payment rate of less than 10 percent for each program and activity for which an improper payment estimate was published. Federal law requires agencies with 3 or more consecutive years of noncompliance findings by their Inspectors General to submit to Congress a reauthorization proposal or a proposal for statutory changes necessary to bring programs into compliance. FNS has taken various actions to improve the integrity of the child nutrition programs in response to findings from our prior work. Over the last 6 years, we issued five reports on the school meals programs, WIC, and SFSP, which included recommendations to FNS intended to improve the integrity of these programs. In response, FNS has addressed many of these recommendations, though additional actions are needed. In 2014, we issued two reports on school meals that found multiple opportunities for FNS to improve school meals program integrity and oversight, all of which FNS has since acted on. Specifically, in January 2014, we recommended that FNS take two different actions aimed at providing assistance to improve state oversight of local school food authority (SFA) administration of the programs; and in May 2014, we recommended that FNS take multiple actions to improve oversight and enhance verification processes that ensure only children who meet income requirements receive free and reduced price school meals. In January 2014, we reported that FNS had provided a significant amount of guidance and training to help states with oversight of local SFAs that directly provide meals to children in schools, but that certain aspects of the guidance may have hindered state oversight of program compliance. (See fig. 1 for entities involved in school meals oversight.) For example, we found evidence indicating that FNS’s guidance allowing states to focus their oversight on providing technical assistance to SFAs, rather than documenting instances of noncompliance and requiring corrective actions to address them, may have resulted in some SFAs that were not fully meeting requirements being certified as in compliance. According to Standards for Internal Control in the Federal Government, federal agencies should have policies and practices in place to provide reasonable assurance that programs are operating in compliance with applicable laws and regulations. Without documentation of noncompliance and requirements for corrective actions, SFAs may not have adequate information on the types of ongoing compliance issues and the need to take corrective actions. Further, FNS may lack information on areas that are problematic across SFAs, which could be the focus of future technical assistance efforts. In 2014, FNS substantially revised and updated the process through which states conduct program oversight—the administrative review—and in our January report, we also found that states reported a need for more information and training related to monitoring SFA financial management. Specifically, we reported that, previously, states had not been required to assess SFA financial management during monitoring reviews, but that states were now responsible for reviewing several aspects of SFA financial management, such as their nonprofit food service accounts and indirect costs. We surveyed all of the states, and over three-fourths reported the need for additional guidance or training from FNS on SFA financial management. We found that while FNS had provided some assistance to states on the new requirements related to SFA financial management, FNS officials had not collected information from all states on their needs in this area. Because state reviews are the key tool used to ensure the integrity of the school meals programs, if state reviewers are unable to effectively review SFA financial management, the federal government will lack assurance that SFAs are complying with federal requirements in this area. In our January 2014 report, we recommended that the Secretary of Agriculture direct the Administrator of FNS to (1) clarify to states the importance of documenting compliance issues found during administrative reviews and requiring corrective actions to address them, and (2) assess all states’ needs for information to improve their ability to oversee SFA financial management and provide assistance to meet identified needs. FNS officials generally agreed with our recommendations and have since addressed them. For example, FNS issued a memo on July 11, 2014, to all regional and state directors reiterating the importance of documenting review findings and any resulting technical assistance and corrective actions. Also in that month, FNS completed its initial efforts to systematically assess all states’ needs for information to improve their ability to oversee SFA financial management. Further, in 2015 and 2016, FNS discussed financial management issues with states during a national meeting and held three national training sessions and a webinar focused on reviewing SFA financial management. In our May 2014 report on school meals, we found that FNS had taken steps to help identify and prevent children ineligible for free or reduced price meals from receiving those benefits, but additional opportunities existed to enhance the application verification process and strengthen program integrity. For example, we reported that school districts are required to verify applications for free and reduced price meals if they are deemed to be questionable, known as for-cause verification. Some school districts were not conducting any for-cause verifications and FNS guidance did not provide indicators or describe scenarios that could assist school districts in identifying questionable applications. Further, FNS’s data on the outcomes of applications verified for cause were combined with data on the outcomes of applications verified for other reasons, limiting FNS’s ability to use these data to assess the effectiveness of for- cause verifications. Standards for Internal Control in the Federal Government direct agencies to design control activities to ensure management’s directives are carried out. Without FNS analysis of data on the outcomes of for-cause verifications, or provision of additional guidance on applications that may merit for-cause verification, some school districts may have continued to overlook these applications, potentially hindering program integrity. In our May 2014 report, we recommended that the Secretary of Agriculture take multiple actions to improve integrity of the school meals programs through additional verification of applications, including that USDA evaluate the data collected on for-cause verification outcomes, and, if appropriate, provide additional guidance for conducting for-cause verification that includes possible indicators of questionable or ineligible applications. FNS took actions in response to all of our recommendations. For example, FNS reported in March 2017 that it analyzed the data on verification outcomes and did not find that any benefit in integrity and oversight would be gained by requiring the reporting of for-cause verification outcomes separately. However, FNS also reported that it disseminated additional guidance in August 2014 for conducting for-cause verifications, which included criteria for identifying possible indicators of questionable or ineligible applications. In 2013 and 2014, we issued two reports on WIC that found multiple opportunities for FNS to improve program integrity and oversight, many of which FNS has since addressed. Specifically, in February 2013, we recommended that FNS review federal monitoring reports on state WIC program administration to assess program risks at a national level, and in December 2014, we recommended that FNS take multiple actions to improve federal WIC oversight and assist states’ efforts to prevent and address online sales of WIC formula. In our February 2013 report, we found that FNS regularly assisted and monitored states’ administration of WIC but needed to improve agency oversight of states’ policies and procedures for determining WIC applicants’ income eligibility for the program. We reported that while federal regulations define criteria that must be used to determine applicants’ income eligibility for WIC, state and local agencies are also given some discretion. We found that FNS generally had not focused its assistance to states on key income eligibility requirements for which states have discretion, such as determination of family size and the time period of income assessed, in the years preceding our report. However, through its monitoring reports, FNS had identified problems with, or concerns about, income eligibility determination policies or procedures in one-third of the states reviewed. Standards for Internal Control in the Federal Government indicate that management should identify, analyze, and respond to risks related to achieving defined objectives and note that risk identification methods may include consideration of deficiencies identified through audits and other assessments. At the time of our review, FNS officials said that they planned to begin regularly reviewing monitoring findings at the national level to identify areas of program risk and target assistance to states accordingly; however, officials did not indicate when those reviews would begin. Without conducting a complete review of its state monitoring findings, FNS lacked information it could potentially use to target additional assistance and clarification on income eligibility determination to states and help ensure overall program integrity. In our February 2013 report, we recommended that the Secretary of Agriculture direct FNS to develop a timeline for reviewing its federal monitoring reports on state WIC program administration to assess program risks at a national level and target assistance to states. FNS officials concurred with our recommendation, and FNS has since addressed it. Specifically, in that year, FNS staff developed a process to use an automated report to identify areas in need of correction or improvement that were found during its monitoring reviews of WIC conducted across the country. The report went into production on November 1, 2013, and FNS reported that staff would review the reports quarterly to assess the frequency of findings in each policy and program area and respond by providing policy clarification, training, or other corrective actions to states. A posting from late June 2014 included the container size in the title and stated: “I am looking to sell 5 [brand name] 12.5oz cans (NOT OPENED) because is super picky and does not want to drink it no matter what i do. will drink the kind for some reason. I told my WIC office to switch me to another brand but they say it might take 3 months. Im asking 35$ but best offer will do since the brand I buy is from so Im not looking to make a profit here if you consider each can is 16$ at the store. please text if interested!! A posting from early July 2014 included the brand, type, and container size in the title and stated: “I have 7 powder cans of they dnt expire for another year at least just got them from my wic n we ended up switching formulas so its $65.oo for pick up all 7 cans or $70 if i have to drive.” In December 2014, we reviewed the online sale of infant formula provided to WIC participants, a practice prohibited by WIC program rules, and concluded that FNS had provided limited assistance to states in preventing and addressing these sales. We found that FNS had not conducted any nationwide studies on the extent of online sales of WIC formula by program participants, though information gathered from state WIC officials and our own limited monitoring suggested that some WIC formula was offered for sale online. (See sidebar.) The use of the internet as a marketplace had substantially increased in the years preceding our report; therefore, actions needed to ensure WIC participants did not inappropriately use infant formula had changed as well. Yet, we found that FNS had not studied cost-effective techniques for monitoring potential online sales of WIC benefits. Standards for Internal Control in the Federal Government note that agencies should identify, analyze, and respond to significant changes that could impact the internal control system. However, FNS had not directed states to inform participants that selling WIC formula, including online, is against program rules, which could lead to participants making these sales and unknowingly using program resources inappropriately. Further, we noted that although states are responsible for controlling participant violations— including sales of WIC benefits—FNS is responsible for determining compliance with the WIC statute and regulations. However, we reported that FNS had not required states to describe procedures for controlling these violations in their WIC state plans, leaving the agency without assurance that efforts were taking place nationwide. Through interviews with state and local WIC agency officials from 12 states for our December 2014 report, we found that states varied in their approaches and the amount of resources devoted to monitoring attempted WIC formula sales, and some expressed concerns about the return on investment for these efforts. Because WIC participants purchase the same brands and types of infant formula from stores as non-WIC customers, monitoring attempted online sales of WIC formula can present a challenge. State officials we spoke with cited additional challenges to monitoring online sales, including the difficulty of identifying WIC participants in online posts that allow sellers to remain relatively anonymous, and as a result, some expressed concerns about the return on investment for these monitoring efforts. Standards for Internal Control in the Federal Government suggest that agencies consider both benefits and costs when designing and implementing internal controls. However, because FNS had not assessed the nationwide extent of online sales of WIC formula by program participants, nor determined cost-effective approaches for identifying and addressing these sales, FNS and the states were poorly positioned to strike the appropriate balance of costs and benefits when determining how to target their resources to ensure program integrity. In our December 2014 report, we recommended that the Secretary of Agriculture direct the Administrator of FNS to (1) instruct states to inform participants that they are not allowed to sell WIC food benefits, including online; (2) require states to inform FNS of their procedures for identifying attempted sales of WIC food benefits and analyze the information to ascertain the national extent of state efforts; and (3) collect information to help assess the national extent of attempted online sales of WIC formula and determine cost-effective techniques states can use to monitor online classified advertisements. FNS agreed with our recommendations and took several steps to address them, though the agency has yet to fully address the third. Specifically, FNS promulgated final regulations that were effective in May 2016 requiring state agencies to inform applicants and participants about the prohibition against the sale of WIC food benefits, including online. Further, in April 2015, FNS issued guidance directing states to articulate their policies and procedures for identifying and monitoring online sales of WIC benefits in their state plans; and in July 2018, an FNS contractor completed a study analyzing state efforts in this area. Also in that month, an FNS contractor completed a study intended to provide information to help FNS address our third recommendation that the agency assess the prevalence of online sales of WIC formula and identify cost-effective techniques states can use to monitor and prevent them. However, FNS indicated that it would not be releasing the study to states, in part because it included information that was investigative in nature. In April 2019, FNS officials indicated that they are currently developing guidance on best practices and cost-effective techniques identified in the report to disseminate to WIC state agencies later in 2019. Informing states of cost-effective techniques for monitoring and preventing online WIC formula sales would address our recommendation. In May 2018, we reviewed the SFSP, which generally provides food to children in low-income areas during periods when schools are closed for vacation, and assessed several aspects of the program, including participation. (See fig. 2 for an SFSP breakfast we observed during a site visit to one of three states we visited.) We found that nationwide, the total number of meals served to children in low-income areas through the SFSP increased from 113 to 149 million (about 32 percent) from fiscal year 2007 through 2016. FNS directs states to use the number of meals served, along with other data, to estimate the number of children participating in the SFSP. However, we found that participation estimates had been calculated inconsistently from state to state and year to year. Recognizing this issue, in 2017, FNS clarified its instructions for calculating participation estimates to help improve their consistency, noting that these estimates are critical for informing program implementation and strategic planning. However, we determined that the method FNS directed states to use would continue to provide unreliable estimates of participation, hindering the agency’s ability to use them for these purposes. Standards for Internal Control in the Federal Government state that agencies should maintain quality data and process it into quality information that is shared with stakeholders to help achieve agency goals. In our May report, we made four recommendations to FNS to improve the integrity of the SFSP, including that FNS take steps to improve its estimate of children’s participation in the SFSP by addressing, at a minimum, identified issues that continued to limit the reliability of the estimate. FNS officials generally agreed with our recommendations, and the agency has since provided information on actions it has planned, or begun to take, to address them. For example, in March 2019, FNS reported that it plans to complete an evaluation of how SFSP participation is calculated by summer 2020. We will continue to monitor FNS’s progress in addressing our SFSP recommendations. In fiscal year 2018, USDA reported improper payments for the child nutrition programs totaling an estimated $1.8 billion, or just over 1 percent of the $151 billion in improper payments federal agencies estimated government-wide in that year. GAO has reported improper payments as a material weakness in internal control in its reports on the U.S. government’s consolidated financial statements, noting that improper payments have consistently been a government-wide issue and reducing these payments is critical to safeguarding federal funds. Since fiscal year 2013, the school meals programs have consistently reported the highest improper payment rate estimates across the child nutrition programs. For example, in recent years, USDA reported annual improper payment rate estimates of about 15 percent and 24 percent for the NSLP and SBP, respectively, compared to about 5 percent and 1 percent for WIC and CACFP, respectively. The estimated total amount of improper payments in the school meals programs are also high, and these programs, along with WIC, are included on OMB’s list of programs with over $100 million in annual monetary losses. The USDA Office of Inspector General’s (OIG) most recent report on the department’s compliance with improper payment requirements, which assessed fiscal year 2017, found that the four child nutrition programs for which USDA estimates improper payments were noncompliant with improper payment requirements. The reasons for noncompliance varied, as the OIG noted that USDA has yet to develop a methodology to report a complete improper payment estimate for CACFP, and corrective actions taken in the other child nutrition programs have not yielded the desired reductions in estimated improper payments. According to our 2018 report, the four child nutrition programs contributed to the government-wide total of 58 programs in 14 federal agencies that agency inspectors general found were noncompliant with improper payment requirements in fiscal year 2017. Further, the four child nutrition programs had been reported as noncompliant for 7 years. We also noted that USDA was one of three federal agencies with programs reported as noncompliant for 3 or more consecutive years that had not notified Congress of their noncompliance, as required, despite prior recommendations that we, and the OIG, had made to USDA to do so. However, USDA submitted a letter to Congress in June 2018 that reported these programs’ noncompliance and described the agency’s planned actions to bring them into compliance. Over time, USDA has undertaken a variety of corrective actions aimed at reducing improper payments in the child nutrition programs, yet the estimated improper payment rates for these programs remained generally steady until fiscal year 2018. For that year, USDA changed what it considers to be an improper payment in the school meals programs, resulting in improper payment estimates that are substantially lower than, and not comparable to, those from prior years. According to USDA, FNS made this change after evaluating its definition of improper payments for the school meals programs and determining that the agency would no longer include a previously identified source of error in its estimates. According to FNS officials, FNS implemented this change after consultation with OMB, and FNS also briefed the USDA OIG on the change in advance of implementation. The USDA OIG has not yet released its report assessing USDA’s fiscal year 2018 compliance with improper payment requirements. To help ensure that annual estimates are produced for all child nutrition programs susceptible to significant improper payments, a 2018 USDA OIG report recommended that FNS complete an SFSP risk assessment for improper payments taking into account all of the risk factors identified by OMB as likely to contribute to improper payments. Although FNS’s 2017 SFSP risk assessment concluded that the program was at low risk for significant improper payments, the OIG found that FNS’s assessment was insufficient because it did not consider multiple risk factors regarding program vulnerabilities and improper payments that OMB requires be taken into account. The OIG reviewed SFSP’s payment structure, monitoring results, and investigations and media cases regarding fraud, and found that these suggest the program is vulnerable to significant improper payments. FNS concurred with the OIG’s recommendation. In April 2019, a senior FNS official indicated that the agency completed a risk assessment for SFSP in response to the OIG’s recommendation, determined that the program is at a high risk of improper payments, and is currently developing a methodology for measuring improper payments in the program. Chairman Roberts, Ranking Member Stabenow, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Kathryn A. Larin, Director, Education, Workforce, and Income Security Issues at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony include Rachel Frisk (Assistant Director) and Theresa Lo (Analyst in Charge). In addition, key support was provided by David Barish, Daniel Flavin, Alex Galuten, Sheila R. McCoy, Jean McSween, Almeta Spencer, and Matt Valenta. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In fiscal year 2018, the federal government provided about $30 billion for USDA's child nutrition programs, including the school meals programs, WIC, and SFSP, among others. In that year, the federal government spent almost $14 billion on the largest of these programs, the National School Lunch Program, which supported the provision of meals to about 30 million children. Federal, state, and local entities play important roles in administering the child nutrition programs and ensuring program integrity. For example, USDA annually estimates improper payments in these programs, which are an indicator of program integrity, and states monitor implementation of the programs by local organizations that directly provide food and services to participants. This testimony discusses (1) actions USDA has taken to address GAO's prior recommendations related to program integrity in the child nutrition programs and (2) improper payments in these programs. This testimony is based on prior GAO reports on child nutrition programs issued from 2013 through 2018, recent GAO and USDA reports on improper payments, and updates GAO obtained in March and April 2019 from USDA officials on actions related to GAO's prior recommendations and improper payments in child nutrition programs. The U.S. Department of Agriculture (USDA) has taken steps, or is planning steps, to improve the integrity of the child nutrition programs in response to recommendations from GAO's prior work. For example: School meals. In 2014, GAO identified several opportunities for USDA to improve school meals oversight and integrity. For example, through GAO's survey of states, over three-fourths reported a need for USDA guidance on monitoring the financial management of local entities that provide meals to children in schools—an area we reported states were newly required to review. GAO recommended that USDA assess states' needs for information in this area. USDA did this assessment and provided related guidance and training to states. Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). In 2013 and 2014, GAO identified several ways that USDA could improve program integrity and oversight in WIC, which provides food benefits to individuals who are low-income. For example, GAO found that USDA had not used its own monitoring findings on state policies for determining applicants' income eligibility to target assistance to states, and recommended that USDA do so. In response, USDA developed a process for reviewing and acting on its monitoring results. Summer Food Service Program (SFSP). In 2018, GAO identified several opportunities for USDA to improve program integrity in the SFSP, which provides food to children in low-income areas when schools are closed for vacation. For example, GAO found that USDA did not collect reliable data on children's participation in the program and that estimates were calculated inconsistently from state to state and from year to year. GAO recommended that USDA take steps to improve the reliability of these estimates and take additional actions to improve program integrity. USDA recently reported plans to address GAO's recommendations. USDA reported improper payments for four child nutrition programs totaling an estimated $1.8 billion in fiscal year 2018, or just over 1 percent of the $151 billion in improper payments that agencies estimated government-wide. GAO has reported that reducing improper payments—which generally include payments that should not have been made or were made in an incorrect amount—is critical to safeguarding federal funds. Since fiscal year 2013, the school meals programs have consistently reported the highest improper payment rates across the child nutrition programs. Over time, USDA has taken a variety of corrective actions aimed at reducing improper payments in child nutrition programs, yet estimated improper payment rates for these programs remained generally steady until fiscal year 2018. For that year, USDA changed what it considers to be an improper payment in the school meals programs, resulting in improper payment estimates that are substantially lower than those from prior years. The Office of Management and Budget (OMB) provides guidance to federal agencies on measuring and reporting improper payment rates, and USDA reported that it made this change after consultation with OMB. GAO made 14 recommendations to USDA in its prior reports on child nutrition. USDA generally concurred with the recommendations and has addressed nine, taken some steps to address one, and is planning to address the remaining four.", "document_type": "gao"}
{"report": "CBP’s Office of Field Operations (OFO) is responsible for inspecting pedestrians, passengers, and cargo at 110 land POEs, which have a combined total of 173 crossings (see figure 1). OFO has 20 field offices nationwide that oversee the operations of all POEs within their designated areas of responsibility. Travelers seeking entry to the United States through a land POE are required to present valid travel documents. In response to a recommendation from the 9/11 Commission and the Intelligence Reform and Terrorism Prevention Act of 2004, DHS and the Department of State implemented the Western Hemisphere Travel Initiative, which requires all travelers to present documents that denote identity and citizenship, such as a passport, when entering the United States. Foreign nationals may have particular travel document requirements, such as a visa or other entry permit, which vary based on such factors as nationality and the purpose of travel. See table 1 for examples of the types of acceptable documents for travelers coming into the United States through land POEs. There are also documentary requirements for commercial vehicles with cargo seeking entry into the United States. The Trade Act of 2002, as amended, establishes requirements for commercial vehicles with cargo to electronically submit information to CBP at least 1 hour in advance of arrival at a land POE. The information required includes data on the vehicle (e.g., Vehicle Identification Number or license plate number), the shipper, the carrier, scheduled date and time of arrival, and the description and weight of the cargo, among other things. Commercial vehicles with cargo valued less than $2,500 are considered “informal entries” that are exempt from the advance cargo information reporting requirements. CBP inspects travelers and cargo seeking to enter the country through land POEs. These inspections involve a targeting process in which CBP uses law enforcement databases to identify and target higher-risk passengers, pedestrians, commercial vehicles, and cargo before arrival at a land POE. Targeting. CBP uses law enforcement, intelligence, and other enforcement data to identify higher-risk individuals, vehicles, or cargo for additional scrutiny upon their arrival at a land POE. Most cargo-carrying commercial vehicles must submit an electronic manifest (e-manifest) with information on the shipment to CBP at least 1 hour in advance of arrival at a land POE. CBP personnel at the POEs are to use the e-manifest and CBP’s Automated Targeting System to identify high-risk inbound cargo. The Automated Targeting System is a decision support tool that compares traveler, cargo, and conveyance information against law enforcement, intelligence, and other enforcement data using risk-based targeting scenarios and assessments. It draws on many law enforcement, intelligence, and other enforcement databases, including the Terrorist Screening Database, the Department of Justice’s National Crime Information Center, the Social Security Administration Death Master File, and the National Insurance Crime Bureau’s private database of stolen vehicles. CBP policy requires that high-risk cargo be targeted for additional research and analysis and generally will also require the high- risk cargo to undergo a secondary examination once it arrives at the POE. In addition, CBP personnel at the POEs or field offices may review seizure and arrest reports, and other law enforcement information to identify individuals or vehicles that have associations with known criminals and place a “lookout” on them in TECS, CBP’s system for processing travelers. TECS will flag travelers with lookouts for additional inspection if they arrive at the land POE. CBP personnel at the POEs or field offices may also use this information to develop products on recent trends that can help inform inspections. Once passengers, pedestrians, and commercial vehicles arrive at a land POE, CBP has various processes for inspecting them, including preprimary, primary, and secondary inspections, as explained below (see figure 2). Preprimary. In the preprimary area, both commercial vehicles and passenger vehicles will generally pass through radiation portal monitors that are designed to detect radiation and help prevent the smuggling of nuclear material into the United States (see figure 3). In the passenger vehicle environment, the preprimary area also contains license plate readers and Radio Frequency Identification (RFID) readers to capture information on vehicles and RFID-enabled travel documents. Examples of RFID-enabled travel documents include passport cards and border crossing cards. When a vehicle enters the preprimary inspection lane, a sensor grid determines that a vehicle has entered the lane. The sensors deploy a flash strobe that illuminates the area and license plate reader cameras take a picture of the front and rear of the vehicle. The information associated with the license plate number is run against law enforcement databases to alert the officer during the primary inspection if there is a potential issue with the vehicle or its occupants. Similarly, as a vehicle approaches the primary inspection area, travelers are directed to hold up their RFID travel documents to be read by RFID readers. Some land POEs may also have RFID readers for pedestrians. See figure 4 for examples of a license plate reader and RFID reader. The preprimary area is also used to direct travelers to different lanes according to the type of travel documents they have. For example, CBP may use signs to designate specific lanes for travelers with RFID or other machine readable documents (“Ready lanes”) or for trusted travelers (see figure 5). Primary inspection. During the primary inspection, CBP officers inspect travelers, vehicles, and cargo to determine compliance with U.S. law and admissibility to the United States. A CBP officer is to examine travel documents to ensure their validity and visually match the traveler to the photo identification to confirm the traveler’s identity. All travelers’ names and license plates generally are to be screened against law enforcement databases. As previously discussed, this screening process may begin in the preprimary area when license plate and RFID readers collect data on vehicles and travelers with RFID travel documents. CBP officers may also manually enter data on travelers and vehicles during the primary inspection. A CBP officer is to interview travelers to obtain a declaration of citizenship, the purpose of travel, and items acquired outside the United States. For commercial vehicles, the CBP officer may also review the manifest and the results of targeting, if any. All CBP officers conducting primary inspections are to wear personal radiation detectors— small devices designed to be worn on a belt—to help detect radiation and help ensure the safety of officers and the traveling public. If the inspection cannot be completed at the primary inspection location, a more thorough inspection is required and the travelers, vehicles, or cargo are to be referred for secondary inspection. Travelers, vehicles, or cargo can be directed to secondary inspection for a wide range of issues, including when: radiation is detected (either on the traveler or from his or her vehicle), the traveler does not have required travel documents, the officer has questions about the validity of travel documents, the traveler’s information matches information that may be of concern from law enforcement or intelligence data, or the officer suspects that the traveler is carrying contraband. Foreign visitors to the United States (with the exception of Canadian citizens and Mexican citizens using border crossing cards) may also be referred to secondary inspection to complete processing of their admission records, referred to as Form I-94s. Additionally, CBP selects passenger vehicles at random to be sent to a secondary inspection for a Compliance Examination (COMPEX). COMPEX is a program designed to help measure the effectiveness of CBP’s inspections and is discussed in more detail later in this report. Secondary inspection. A secondary inspection may include a CBP officer conducting further questioning of travelers or additional examination of the traveler, vehicle, or cargo. CBP may use canines, non- intrusive inspection (NII) X-ray, Gamma-ray, or radiation detection equipment, or physically examine the traveler, vehicle, or cargo. CBP may also examine a traveler’s electronic devices, such as computers, tablets, and mobile phones. To examine cargo, CBP may require the contents to be offloaded. When foreign visitors are referred to a secondary inspection to process Form I-94 admission records, CBP officers are to conduct interviews and additional database screening, including biometric checks of fingerprints. CBP policy calls for documentation, immigration, and other admissibility issues to be resolved before a traveler or vehicle is permitted to enter the country. Below, figure 6 shows a canine examination and figure 7 shows an example of NII equipment and scans of vehicles with indicators of contraband smuggling. CBP also has additional processes to enhance preprimary, primary, or secondary inspections at land POEs, including: Canines. CBP has canines that can detect concealed humans, narcotics, currency, firearms, and agriculture products. Depending on availability, land POEs may deploy officers with canines to walk among the vehicles in preprimary waiting to reach an inspection booth. Canines may also be used in the pedestrian and commercial vehicle environments. As previously mentioned, canines are also used for some secondary searches. Anti-Terrorism Contraband Enforcement Teams. These teams conduct special operations that focus on anti-terrorism and the interdiction of narcotics, alien smugglers, and fraudulent documents, among other contraband. For example, at one POE we visited, members of the Anti-Terrorism Contraband Enforcement Team told us they often walk among the passenger vehicles in the preprimary area to look for indicators of illicit activity. Tactical Terrorism Response Teams. These teams provide immediate counterterrorism response capabilities at some land POEs. Members of Tactical Terrorism Response Teams receive counterterrorism training and are responsible for interviewing known and suspected terrorists at ports of entry to help determine admissibility and collect intelligence. Blitzes and other local practices. CBP officers at land POEs may perform “blitzes”, in which inspections are enhanced for a period of time. For example, CBP officials told us that blitzes may include looking in all vehicle trunks during the primary inspection or sending additional vehicles for NII (X-ray) exams during a certain period of time. Officers at the POEs we visited also discussed other local initiatives to enhance inspections. For example, one POE we visited used NII to screen all commercial vehicles. Another POE we visited partnered with the local authority that manages an international bridge to deploy license plate readers for commercial vehicles before the vehicles enter the bridge into the United States. The bridge authority uses the license plate reading to check if the commercial vehicle has submitted the required e-manifest to CBP; only those commercial vehicles that have submitted the required e-manifests are allowed to cross. Officials from CBP told us that, in the future, CBP and the bridge authority plan to deploy additional technology in the preprimary area on the non-U.S. side of the border, including facial recognition and NII. In addition, CBP has plans to make future improvements to inspection processes. For example, CBP is conducting tests to use facial recognition technology as part of inspections at land POEs. According to CBP, facial recognition technology may enhance its ability to detect imposters by matching facial images of those arriving with images on file. CBP began a facial recognition test in the passenger vehicle environment at the Anzalduas, Texas land POE in August 2018 and expects the test to run for up to 1 year. In September 2018, CBP initiated a project at the Port of San Luis, Arizona to demonstrate the feasibility of acquiring photos of all arriving pedestrians and comparing those photos to photos on file. Subsequently, in October 2018, CBP officials stated they extended this demonstration project to the Port of Nogales, Arizona. According to CBP, these pedestrian demonstration projects built upon an earlier pilot project at the Port of Otay Mesa, California, which ran from February through May 2016. Testing this technology is one of CBP’s key efforts in developing the capability to fulfill DHS’ statutory responsibility to collect biometric information from arriving and departing aliens. CBP has numerous directives, handbooks and other official instructions that specify policies and procedures for inspections at land POEs. However, many of these documents have not been reviewed and updated as required by OFO’s January 2016 OFO Policy Management Handbook. This guidance states that all of OFO’s policies must be reviewed and updated, as necessary at least once every 3 years to help ensure the timely provision of uniform and relevant policy. In some cases, the policy documents issued by OFO or its program offices have not been reviewed and updated for almost two decades. See table 2 below for a list of such policies we identified that have not been reviewed and updated to reflect changes in processes since their issuance consistent with OFO’s policy management requirements. As a result of policies not being reviewed and updated by OFO, these policies, as currently written, do not fully reflect changes in technology, operating conditions, or inspection processes. For example: The 2008 policy on processing travelers and vehicles at land POEs does not include information on the Consolidated Secondary Inspection System, the current system used to record secondary inspections. It also directs officers to follow guidance in the Inspector’s Field Manual, which has since been discontinued. The 1999 Compliance Measurement directive refers to procedures for a paper-based system, while the system is now electronic, according to officials. The 2004 Personal Search Handbook does not incorporate the 2015 National Standards on Transport, Escort, Detention, and Search policy that prohibited CBP officers from observing personal cavity searches conducted by medical personnel. The 1999 Narcotics Interdiction Handbook and the 2002 canine policies do not address fentanyl. Fentanyl is a synthetic opioid that requires special handling and has been a main contributor to the recent spike in overdose deaths in the United States, according to the Centers for Disease Control and Prevention. OFO’s Planning, Program, Analysis, and Evaluation (PPAE) Quality Assurance Enterprise Division (QAED) is responsible for monitoring that each program office review and update, as needed, the policies for its programs. QAED has an internal tracking system and sends out reminders to CBP program offices about policies that need to be reviewed, and updated, if necessary. QAED officials acknowledged that many policies need to be updated because some are almost 20 years old and many technological and other changes have occurred that may not be described in existing policies. CBP officials stated that they are in the process of updating some policies, including the 1999 Compliance Measurement directive, the 2002 Canine Enforcement Program Handbook, the 2004 Personal Search Handbook, and the 2008 Primary Processing of Travelers and Vehicles Seeking Entry to the United States at Land Ports of Entry directive. Officials attributed the lack of timely updating to several factors. OFO officials responsible for reviewing and updating policies said that the process can be time-consuming and difficult, as there may be many needed changes or may include conducting site visits to identify best practices and areas for improvement. In addition, QAED officials responsible for monitoring policy updates said QAED has 12 staff and is responsible for three OFO-wide mission areas in addition to policy management, as well as a number of other responsibilities within PPAE. Further, according to QAED officials, they do not have authority to require cognizant program offices to review and update their policies in line with the OFO Policy Management Handbook. QAED officials agreed that CBP and OFO could better ensure compliance with OFO’s policy updating requirements. OFO’s 2016 OFO Policy Management Handbook states that the timely provision of uniform and relevant policy facilitates informed decision- making at all levels of the organization and that an effective policy management program is critical to the success of any organization. By reviewing and updating as necessary all relevant policies related to land POE inspections consistent with OFO’s policy handbook, CBP could better ensure that officers have guidance needed to consistently and properly inspect vehicles and their passengers, pedestrians, and commercial vehicles. CBP uses various mechanisms at the port, field office, and national levels to monitor inspection activities at land POEs to help ensure that CBP officers are following policies and procedures. At the POE level, supervisors and port management monitor many of the inspection tasks in real-time by reviewing computer-based records and logs of inspections and observing inspections. CBP also provides tools to the ports to assist with supervisory monitoring efforts, such as Enforcement Link Mobile Operations Red Flag (ELMOrf)—a computer application that provides alerts to supervisors via mobile device when certain types of events occur during primary inspections that warrant supervisory oversight. Table 3 below provides key monitoring mechanisms CBP uses for its land POE inspections at the port level. At the field office level, field office staff may monitor land POE activities within their area of responsibility through periodic assessments of supervisor monitoring duties, such as inspection report reviews. In addition, all field offices have Integrity Officers tasked with identifying potential corruption and officer training issues at the ports. Table 4 below provides key monitoring mechanisms CBP uses for its land POE inspections at the field office level. CBP’s national level initiatives include its Self-Inspection Program (SIP) and the Operational Field Testing Division’s covert testing program. The Self-Inspection Program is an annual internal self-assessment of various CBP component offices and includes assessment of various inspection activities at POEs. Table 5 below provides key monitoring mechanisms CBP uses for its land POE inspections at the national level. CBP produces CBP-wide analyses of the SIP results it collects annually, but the analyses are not done in a manner—such as at the port level and over multiple years—that would allow CBP to identify potentially reoccurring deficiencies at individual POEs. The Management Inspections Division issues a report each year which provides comprehensive SIP results across CBP offices for that year and highlights compliance issues identified (referred to as the SIP Summary Analysis Report). Similarly, OFO issues an annual report which provides comprehensive results and highlights compliance issues identified across OFO’s programs for that year. See figure 8 for an overview of the SIP process. With regard to the 2018 SIP Summary Analysis Report, the Management Inspections Division reported that approximately 80 percent of all SIP worksheets, which document the results of the self-assessments, submitted across CBP in the 2018 cycle had no deficient conditions. The report also identified the six worksheets with the highest number of deficient conditions across OFO and the questions associated with the most corrective actions for those worksheets. For worksheets that the report did not highlight, additional summaries of the OFO data are provided, including the number of worksheets submitted and the number of worksheets reporting corrective actions. OFO’s SIP annual report also provides summaries of the SIP results, but with additional analysis specific to OFO. The 2018 OFO SIP annual report calculated an overall compliance rate of 92.4 percent across the 31,947 questions for worksheets completed by OFO that year. The report also provided summaries of data used to calculate compliance rates for each worksheet assigned to OFO and included trends in compliance rates for each over 3 years. Additionally, the report provided summaries of the data for each OFO field office that includes number of worksheets submitted, the number of deficient conditions in the given year, and the number of corrective actions for each POE under the field office. Beginning in 2017, the OFO report provided an analysis of any SIP worksheet question with a compliance rate below 90 percent in a given year and the actions planned or taken to increase future compliance. While these reports provide useful summary data of CBP’s monitoring of inspections activities and recommendations for increasing compliance for some programs and processes, our analysis of SIP results showed that opportunities exist for CBP to identify potential reoccurring deficiencies at individual land POEs over time. Specifically, our analysis of SIP results from 2013 through 2018 identified reoccurring instances of noncompliance at individual land POEs indicating the possibility that the corrective actions taken each year to address the deficiencies did not fully remediate them. We found that management at the land POEs with reoccurring instances of deficiencies took corrective actions each year to address the identified deficiencies, and in some instances, management proposed and implemented the same corrective action in multiple years to try to resolve the identified deficiency. While the Management Inspections Division and OFO reports provide some useful analysis to identify programs or specific activities across CBP to target for remediation each year, these reports have not positioned CBP to identify and more effectively address reoccurring deficiencies at individual POEs. Standards for Internal Control in the Federal Government provides that management should use quality information to achieve the entity’s objectives and management should process the obtained data into quality information that supports the internal control system. Furthermore, management should remediate identified internal control deficiencies on a timely basis and the audit resolution process is completed only after action has been taken that (1) corrects identified deficiencies, (2) produces improvements, or (3) demonstrates that the findings and recommendations do not warrant management action. Additionally, management, with oversight from the oversight body, is to monitor the status of remediation efforts so that they are completed on a timely basis. Management Inspections Division and OFO officials stated that their analyses are designed to identify systemic compliance issues across OFO. In addition, OFO officials stated that port management is responsible for addressing compliance issues of individual land POEs. However, without an analysis to identify reoccurring deficiencies at all individual land POEs, the Management Inspections Division and OFO are not well positioned to determine whether CBP may need to take additional or alternative actions to more effectively address the deficiencies at these ports. By enhancing analysis of the SIP data to include analysis at the port level over time, CBP could better identify potential reoccurring deficiencies with inspections at land POEs and could be better positioned to more fully remediate them and ensure compliance with inspection policies. CBP has produced comprehensive analyses of the results from some of its covert operational tests conducted at land POEs in fiscal years 2013, 2014 and 2018. These comprehensive assessments of aggregated covert test results provide analysis of trends, common vulnerabilities, and best practices used in inspections across land POEs; however, CBP has not developed comprehensive assessments for various other covert tests it conducted during this time frame. Of the 213 land POE tests conducted from fiscal years 2013 through 2018, 78 were included in comprehensive assessments. CBP’s Operational Field Testing Division (OFTD) is responsible for covertly assessing and evaluating the integrity of CBP’s personnel, technologies, and policies and procedures at land POEs. From fiscal years 2013 through 2018, OFTD conducted a variety of tests of inspections at land POEs including: fraudulent document and imposter tests, canine contraband detection tests, biological agent detection tests, NII equipment contraband detection tests, radiation detection capabilities tests, and assessments of Tactical Terrorism Response Teams. See figure 9 for an overview of the process for fraudulent document and imposter covert testing. For tests conducted from fiscal years 2013 to 2018, OFTD produced three comprehensive assessments related to tests it conducted at land POEs. One assessment compiled the results of 129 fraudulent document and imposter tests conducted at 10 land POEs and 14 airports in fiscal years 2012 and 2013. Another assessment covered 34 NII equipment tests conducted in fiscal years 2013 and 2014 at land POEs and seaports, of which nine of the tests were at land POEs. The third assessment, issued in 2018, covered 33 NII equipment tests conducted in fiscal year 2018 at six land POEs. While OFTD produced comprehensive assessments for these tests, OFTD did not comprehensively analyze the results of various other types of covert tests conducted from fiscal years 2013 through 2018. Such covert tests included 34 tests for canine detection of contraband, 11 for agricultural and biological agent detection, seven for radiation detection, and seven for Tactical Terrorism Response Team response. Additionally, OFTD conducted another 72 fraudulent document and imposter tests and six NII equipment tests over this time period that were not included in the comprehensive assessments described above. Overall, we found that 135 of 213 tests conducted from fiscal years 2013 through 2018 were not included in comprehensive assessments. For tests not included in comprehensive assessments, analysis of the test is limited to a test summary document that is produced following a test or group of tests conducted during a field visit to one location. The summaries identify officer actions during the test and record whether the test resulted in an interdiction of the test subject. Some of the summaries also include findings, identify leading practices, and provide recommendations to the POE where the test or tests were conducted to improve the inspections. While these summaries provide useful information, they encompass the results of tests at individual POEs and do not provide an evaluation of aggregated test results that could more broadly identify vulnerabilities, trends, and best practices across land POEs as provided in the comprehensive assessments. According to OFTD officials, they have drafted a policy and standard operating procedures that would address comprehensive analysis of covert testing results, but these have been in development for 3 years and have not been finalized. OFTD did not provide further details or documentation of the draft policy or procedures or a date for completion. Additionally, OFTD officials stated that in some cases they did not have a sufficient number of covert test results to conduct a comprehensive analysis. OFTD officials also stated that an additional comprehensive assessment of fraudulent document and imposter tests was not needed as OFTD completed this type of assessment in 2013 and no new findings were generated by subsequent tests. We recognize that the small number of certain tests limit OFTD’s ability to conduct comprehensive analyses. However, we found that from fiscal years 2013 through 2018 over half (135 of 213) of the tests conducted at land POEs were not included in a comprehensive assessment and a formalized policy could better position OFTD to be able to conduct these analyses moving forward. Further, our analysis of covert test interdiction rates suggests that additional periodic comprehensive analysis could help inform CBP management of vulnerabilities, systemic inspection deficiencies, leading practices observed, and ways to improve inspection processes. Moreover, the reasons for non-interdiction in the fraudulent document and imposter covert tests conducted since the last comprehensive assessment may be different due to changes in inspection technologies, training, personnel, or the threat environment. OFTD officials agreed and stated that another comprehensive assessment is being developed based on covert tests focused on facial recognition technologies. Standards for Internal Control in the Federal Government provides that management should implement control activities through policies, including documenting such policies. In addition, management should monitor the internal control system through ongoing monitoring and separate evaluations. These evaluations are to be used periodically and may provide feedback on the effectiveness of ongoing monitoring. Furthermore, management should evaluate and document issues identified through separate evaluations to identify internal control deficiencies and monitor changes in the internal control system. By implementing a policy for conducting periodic comprehensive analyses of its covert operational test results, CBP would be better positioned to understand the effectiveness of inspection policies, personnel, and technologies across land POEs over time. Furthermore, periodic analyses could help identify inspection vulnerabilities that may be occurring more broadly, trends in these vulnerabilities, and best practices in mitigating such vulnerabilities on a more consistent basis. CBP uses various sets of performance measures including organizational performance measures, internal performance measures, program and port-specific measures, and measures required by the National Defense Authorization Act for Fiscal Year 2017 (NDAA). CBP reports organizational measures externally to inform program management while internal measures track additional areas of performance to inform OFO management. In addition, some CBP programs and ports track measures specific to their performance at land POEs. DHS also reports measures that cover CBP’s efforts to detect illegal activity at land POEs as required by the NDAA. These performance measures generally reflect attributes of effective measures, however, CBP has not set an ambitious target for one measure—the land border interception rate. CBP tracks and externally reports the results of performance measures annually in its Organizational Performance Measures Overview. The Overview states that it serves as a tool for leadership to manage programs using performance information and includes performance measure descriptions, targets, results, and trends over time. CBP developed and reports on two measures that cover the detection of illegal activity among inbound passenger vehicle and cargo traffic at land POEs: (1) the estimated percentage of land border privately-owned vehicles with passengers who are compliant with laws, rules, and regulations; and (2) the percentage of inbound cargo identified as high-risk that is assessed or scanned prior to departure or at arrival at a U.S. air, land, and sea POE. CBP also tracks, but does not report, data on the percentage of high-risk inbound cargo assessed or scanned prior to departure or upon arrival at U.S. land POEs, which in fiscal year 2018 was 97.7 percent. See figures 10 and 11 for CBP’s reported results for these measures by fiscal year. CBP measures the percentage of privately-owned vehicles with passengers who are compliant with all federal, state, and local laws and regulations through its COMPEX program. COMPEX is a statistical survey in which vehicles cleared for entry into the United States by CBP are randomly selected for a comprehensive audit through a computer- generated random sample. CBP is to conduct an audit of the selected vehicles by doing a secondary inspection using a standardized system of checks to identify any violations that were missed during the routine inspection. Violations found in the COMPEX audits represent violations missed by CBP and are used by CBP to estimate the total number of violations missed by CBP operations. According to officials, CBP uses these data— along with data on violations CBP officers identify during the normal inspection process—to calculate the overall estimated percentage of land border privately-owned vehicles with passengers compliant with laws, rules, and regulations. As shown in Figure 10, CBP has set a target rate of 99.5 percent compliance. From fiscal years 2015 through 2018, CBP reported estimated rates of over 99 percent compliance. While CBP nearly met its target across all of these years, CBP plans to work with field office management and review COMPEX secondary inspection findings to identify noncompliance trends and identify the underlying reasons for noncompliance. In addition, CBP plans to develop materials to educate travelers on relevant laws and requirements. As previously discussed, in the cargo environment, CBP identifies potentially high-risk cargo through the Automated Targeting System. CBP then tracks the percentage of such cargo assessed or scanned prior to arrival or at a land POE. As shown in Figure 11, CBP has set a target rate of identifying 100 percent of potentially high-risk cargo. For fiscal years 2014 through 2017, CBP reported rates of 99 percent or higher, and in 2018, the rate was 97.9 percent. According to CBP, it did not meet its target rate of 100 percent in fiscal year 2018 because of challenges related to changes in high-risk status that occur en route, data entry errors, and logistical or scheduling errors. OFO plans to address these challenges by working with internal stakeholders to resolve status- tracking problems and information-processing errors and by working with shippers and carriers to rectify logistical and scheduling issues. In addition to its externally-reported organizational performance measures, OFO tracks two performance measures internally that relate to efforts to detect illegal activity among inbound traffic at land POEs: the percentage of individuals screened against law enforcement databases for entry into the United States and the land border interception rate for passengers in privately-owned vehicles with major violations. See figure 12 for CBP’s performance by fiscal year. CBP uses COMPEX data to estimate the land border interception rate for privately-owned vehicles containing passengers with major violations (interception rate). This represents the number of major violations in privately-owned vehicles at the border that CBP intercepts divided by the estimated total number of major violations. CBP tracked the percentage of individuals screened against law enforcement databases for entry into the United States across fiscal years 2013 through 2018, but plans to discontinue use of this measure beginning in fiscal year 2019 according to CBP officials. CBP officials stated that this measure was originally created to track progress toward electronic screening of travel documents as part of the Western Hemisphere Travel Initiative. This measure tracks the percentage of travelers screened against law enforcement databases using electronically readable documents. According to CBP officials, there have been a variety of technology infrastructure upgrades and changes to vehicle processing software at land POEs that have reduced the relevance of this measure for land POE operations and CBP plans to discontinue its use as a result. Some CBP programs that operate as part of the inspection process track performance data on the results of their program activities. For example, CBP tracks results from the Canine Program. Canine handlers are to enter performance data into the Canine Tracking System locally at land POEs. They track data on the numbers of days canine officers worked, searches conducted, and fines and arrests that result from canine searches. In addition, some land POEs track performance data on local efforts to detect illegal activity. For example, officials at one POE we visited track data on the numbers and types of seizures, arrests, and immigration enforcement actions that occur at the port. In 2018, DHS began reporting additional metrics to measure the effectiveness of border security at land POEs in response to the National Defense Authorization Act for Fiscal Year 2017 (NDAA). The NDAA requires DHS to produce an annual report for appropriate congressional committees, the Comptroller General, and certain other entities. This report is to include certain metrics to measure the effectiveness of border security between POEs, at POEs, in the maritime environment, and with respect to aviation assets and other air and marine operations in the land domain. DHS submitted the fiscal year 2017 Border Security Metrics Report in response to the NDAA requirement in May 2018. Nine of the metrics in DHS’s fiscal year 2017 report cover CBP’s efforts to detect illegal activity at land POEs, although many of these measures group land POE data with other types of ports. DHS reported data for 7 of these 9 metrics. In some instances, DHS reported that it did not have the specific data needed for a required metric and provided other available data instead. DHS reported data in response to the following required metrics related to land ports of entry in the fiscal year 2017 Border Security Metrics Report: total inadmissible travelers at ports of entry (DHS does not have a methodology to estimate total inadmissible travelers, and therefore presented data on known inadmissible travelers), refusal rate at ports of entry, illicit drugs seized at ports of entry, port of entry illicit drug seizure rate, major infractions at ports of entry (DHS does not have a methodology to estimate all major infractions, and therefore included data on known passenger infractions), cocaine seizures effectiveness rate at land ports of entry, and secondary examination rate. CBP did not leverage existing data from the COMPEX program to estimate all major infractions in the fiscal year 2017 Border Security Metrics Report, but began reporting these data in the fiscal year 2018 report. The NDAA requires DHS to report the number of infractions related to travelers and cargo committed by major violators who are interdicted by OFO at ports of entry and the estimated number of such infractions committed by major violators who are not so interdicted. In the fiscal year 2017 DHS Border Security Metrics Report, DHS reported the number of known major infractions at ports of entry. DHS also reported that they did not have a methodology to estimate the number of infractions among those who are not interdicted. However, CBP estimates the number of undetected major infractions through the COMPEX program. CBP officials stated there was likely a miscommunication within CBP that led to the DHS Office of Immigration Statistics—the DHS office that compiled the Border Security Metrics Report— not using COMPEX data to report the estimated number of major infractions in the 2017 Border Security Metrics Report. In addition, the DHS Office of Immigration Statistics was not aware that CBP’s COMPEX was applicable for purposes of reporting this metric. As a result of our review, DHS included an estimate of the number of major infractions not interdicted by CBP using data from the COMPEX program in the fiscal year 2018 Border Security Metrics Report. CBP organizational and internal performance measures for detecting illegal activity at land POEs generally reflect key attributes of effective performance measures that we previously identified. Based on our analysis of CBP’s organizational and internal performance measures, these measures generally reflect the key attributes listed in table 6. For example, CBP clearly defines its externally-reported organizational measures and presents baselines and trends in its Organizational Performance Measures Overview. In addition, CBP’s Organizational Performance Measures Overview provides linkage between its externally-reported organizational measures and DHS mission. CBP performance measures also have limited overlap with each other presenting new information beyond what other measures provide. Our analysis of CBP’s measures found that they focus on the commercial and passenger-owned vehicle environments and currently provide limited coverage of the pedestrian traveler environment. According to CBP officials, the agency is in the process of expanding the two COMPEX measures to include pedestrian travelers at land POEs, which would provide greater coverage of CBP’s core program activities for detecting illegal activity at land POEs. According to CBP officials, CBP began collecting COMPEX data for all pedestrian POEs in 2015. CBP officials stated they are in the process of reviewing the collected data and are working to refine the methodology and operational issues that may impact the reliability of the results. After CBP resolves these data issues, CBP will begin reporting the results of COMPEX audits in the pedestrian environment, according to CBP officials. Our analysis of CBP’s measures also found that CBP generally sets ambitious but realistic targets for its organizational and internal performance measures. However, CBP’s target for the land border interception rate is lower than the actual reported rate for fiscal years 2015 through 2018. We previously identified critical success factors for goal-setting and performance measurement efforts. Creating ambitious but realistic and measurable “stretch” goals based on current performance levels, among other things, supports the organization in achieving performance improvements. In addition, the Office of Management and Budget Circular A-11 states that agencies are expected to set ambitious goals to push them to achieve significant performance improvements beyond current levels. OFO officials stated they set the target for the land border interception rate following methodological changes OFO implemented in the COMPEX program in 2015. However since that time, OFO officials in the Strategic Transformation Office—the office that reviews and provides input into targets for CBP’s organizational performance measures—stated they have not reviewed this target because it is an internal measure and they do not review these as they would for the externally-reported organizational measures. Nevertheless, OFO officials stated they use this measure internally for performance management and to report results to OFO management. Because OFO sets a target for the interception rate and uses this measure internally, a more ambitious target for the measure would better encourage CBP to review its performance of inspection activities that impact the measure and challenge them to identify ways of improving performance. Inspecting travelers and cargo seeking entry to the United States through land POEs is critical to preventing terrorists and other inadmissible persons, as well as nuclear materials, narcotics, and other contraband, from entering the country. OFO has implemented processes and deployed technology to screen and examine travelers and cargo at POEs; however, by reviewing and updating its inspection policies in accordance with its own established time frames, CBP could better ensure that officers have guidance needed to consistently and properly inspect passengers, pedestrians, and commercial vehicles. Further, while CBP has taken steps to monitor compliance with inspection policies through the SIP and covert operational tests, it could more fully analyze the results. By identifying and addressing reoccurring SIP deficiencies at individual land POEs and implementing a policy to conduct periodic comprehensive analyses of covert test findings, CBP could be better positioned to enhance inspections and address vulnerabilities. Lastly, CBP has established various measures to assess the effectiveness of its inspections; however, establishing an ambitious and realistic target for its major violations interception rate could encourage additional improvements in performance. We are making the following four recommendations to CBP: The Commissioner of CBP should review and update policies related to land port of entry inspections in accordance with OFO guidance. (Recommendation 1) The Commissioner of CBP should analyze the results of the Self- Inspection Program over time and at a level necessary to identify and address potentially reoccurring inspection deficiencies at individual ports of entry. (Recommendation 2) The Commissioner of CBP should implement a policy to conduct periodic comprehensive analyses of covert test findings. (Recommendation 3) The Commissioner of CBP should develop a new target for the land border interception rate for passengers in privately-owned vehicles with major violations that sets an ambitious and realistic goal based on past performance. (Recommendation 4) We provided a draft of this report to DHS for its review and comment. DHS provided comments, which are reproduced in appendix I. In its comments, DHS concurred with the four recommendations. DHS also provided technical comments, which we incorporated as appropriate. With regard to the first recommendation that CBP update policies related to land POE inspections in accordance with OFO guidance, DHS stated that OFO has initiated a process to modernize handbooks, policy memoranda, and directives. With regard to the second recommendation that CBP analyze SIP results over time and at a level necessary to identify and address potentially reoccurring deficiencies at individual POEs, DHS stated that OFO plans to begin training on how to conduct this analysis so it may be conducted for 2021 SIP results. With regard to the third recommendation that CBP implement a policy to conduct periodic comprehensive analyses of covert test findings, DHS stated that CBP is in the process of writing a policy that will document procedures for comprehensive reporting, including periodic reviews of corrective actions taken to mitigate vulnerabilities. With regard to the fourth recommendation that CBP develop a new target for the land border interception rate, DHS stated that OFO will set a new target for fiscal year 2020 using data from the previous three fiscal years. If fully implemented, these actions will meet the intent of our recommendations. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Kirk Kiester (Assistant Director), Heather May (Analyst in Charge), Carl Barden, Michele Fejfar, Eric Hauswirth, Susan Hsu, Richard Hung, Jeff Love, Mara McMillen, Sasan J. “Jon” Najmi, and Jonathan Tumin made key contributions to this report.", "summary": "CBP, within the Department of Homeland Security (DHS), is the lead federal agency charged with a dual mission of facilitating the flow of legitimate travel and trade at the nation's borders while keeping terrorists and their weapons, criminals and their contraband, and inadmissible aliens out of the country. GAO was asked to review CBP's process for inspecting passenger vehicles, pedestrians, and commercial vehicles at land POEs to secure the border. This report examines to what extent CBP (1) has processes and policies for inspections, (2) monitors inspection activities, and (3) has measures to assess its efforts to detect illegal activity of passengers, pedestrians, and commercial vehicles at land POEs. To address these questions, GAO analyzed CBP documents and data related to inbound inspections; interviewed officials; and observed operations at a non-generalizable sample of seven land POEs, selected to reflect a range of traffic volumes and geographic locations, among other things. This is a public version of a sensitive report that GAO issued in June 2019. Information that DHS deemed sensitive has been omitted. U.S. Customs and Border Protection (CBP) has processes for inspecting passenger vehicles, pedestrians, and commercial vehicles at U.S. land ports of entry (POE). These processes include reviewing travel documents, screening against law enforcement databases, and using canines and X-ray equipment (see figure below). However, because CBP has not updated many of its policies—in a few cases for almost 20 years—they do not always reflect changes in technology or processes, such as those for conducting searches and handling fentanyl. By reviewing and updating policies, CBP could help ensure officers have guidance needed to consistently and properly perform inspections. CBP has various mechanisms at the port, field office, and national levels to monitor inspection activities at land POEs, but opportunities exist to enhance analysis of the results from its national level Self-Inspection Program (SIP) and covert operational testing. The SIP is an annual self-assessment that POEs are to conduct to determine compliance with CBP policies. CBP analyzes the results of the SIP annually to identify systemic compliance issues across CBP that year; however, it does not analyze noncompliance at individual POEs over time. By analyzing these data, CBP could better identify and address deficiencies at individual POEs. In addition, CBP has produced three comprehensive assessments, which analyzed aggregated results for certain types of covert tests, such as fraudulent document tests, conducted at land POEs in fiscal years 2013, 2014, and 2018. However, CBP has not done so for other types of tests, such as canine contraband detection tests, conducted from fiscal years 2013 through 2018. By implementing a policy for periodically conducting such analyses, CBP could identify vulnerabilities, trends, and best practices occurring more broadly. CBP uses various sets of measures to assess its efforts to detect illegal activity at land POEs. CBP performance measures generally reflect the key attributes of effective measures, but CBP does not set an ambitious and realistic target for one measure. CBP's target for the land border interception rate—the estimated percentage of major violations in privately-owned vehicles that CBP intercepts out of the projected total number of major violations—is lower than the actual reported rate for fiscal years 2015 through 2018. A more ambitious target for the interception rate would better encourage CBP to review past performance of inspection activities that impact the measure and challenge CBP to identify ways to improve performance . GAO recommends that CBP: (1) review and update policies related to land POE inspections in accordance with CBP guidance; (2) analyze the SIP results to identify and address reoccurring inspection deficiencies at individual POEs; (3) implement a policy to conduct periodic comprehensive analyses of covert test findings; and (4) develop a more ambitious target for the land border interception rate measure. DHS concurred.", "document_type": "gao"}
{"report": "The federal government plans to invest over $90 billion in IT in fiscal year 2019. Nevertheless, we have previously reported that investments in federal IT too often resulted in failed projects that incurred cost overruns and schedule slippages, while contributing little to the desired mission- related outcomes. For example: The tri-agency National Polar-orbiting Operational Environmental Satellite System was disbanded in February 2010 at the direction of the White House’s Office of Science and Technology Policy after the program spent 16 years and almost $5 billion. The Department of Homeland Security’s (DHS) Secure Border Initiative Network program was ended in January 2011, after the department obligated more than $1 billion for the program. The Department of Veterans Affairs’ Financial and Logistics Integrated Technology Enterprise program was intended to be delivered by 2014 at a total estimated cost of $609 million, but was terminated in October 2011. The Department of Defense’s Expeditionary Combat Support System was canceled in December 2012 after spending more than a billion dollars and failing to deploy within 5 years of initially obligating funds. The United States Coast Guard (Coast Guard) decided to terminate its Integrated Health Information System project in 2015. As reported by the agency in August 2017, the Coast Guard spent approximately $60 million over 7 years on this project, which resulted in no equipment or software that could be used for future efforts. Our past work has found that these and other failed IT projects often suffered from a lack of disciplined and effective management, such as project planning, requirements definition, and program oversight and governance. In many instances, agencies had not consistently applied best practices that are critical to successfully acquiring IT. Federal IT projects have also failed due to a lack of oversight and governance. Executive-level governance and oversight across the government has often been ineffective, specifically from CIOs. For example, we have reported that some CIOs’ roles were limited because they did not have the authority to review and approve the entire agency IT portfolio. In addition to failures when acquiring IT, our cybersecurity work at federal agencies continues to highlight information security deficiencies. The following examples describe the types of risks we have found at federal agencies. In September 2018, we reported that the Department of Education’s Office of Federal Student Aid exercised minimal oversight of lenders’ protection of student data and lacked assurance that appropriate risk- based safeguards were being effectively implemented, tested, and monitored. In August 2017, we issued a report stating that, since the 2015 data breaches, the Office of Personnel Management (OPM) had taken actions to prevent, mitigate, and respond to data breaches involving sensitive personal and background investigation information. However, we noted that the agency had not fully implemented recommendations made to OPM by DHS’s United States Computer Emergency Readiness Team to help the agency improve its overall security posture and improve its ability to protect its systems and information from security breaches. We reported in July 2017 that information security at the Internal Revenue Service had weaknesses that limited its effectiveness in protecting the confidentiality, integrity, and availability of financial and sensitive taxpayer data. An underlying reason for these weaknesses was that the Internal Revenue Service had not effectively implemented elements of its information security program. In May 2016, we found that the National Aeronautics and Space Administration, the Nuclear Regulatory Commission, OPM, and the Department of Veteran Affairs did not always control access to selected high-impact systems, patch known software vulnerabilities, or plan for contingencies. An underlying reason for these weaknesses was that the agencies had not fully implemented key elements of their information security programs. We reported in August 2016 that the information security of the Food and Drug Administration had significant weaknesses that jeopardized the confidentiality, integrity, and availability of its information systems and industry and public health data. Congress and the President have enacted various key pieces of reform legislation to address IT management issues. These include the federal IT acquisition reform legislation commonly referred to as the Federal Information Technology Acquisition Reform Act (FITARA). This legislation was intended to improve covered agencies’ acquisitions of IT and enable Congress to monitor agencies’ progress and hold them accountable for reducing duplication and achieving cost savings. The law includes specific requirements related to seven areas: Agency CIO authority enhancements. CIOs at covered agencies have the authority to, among other things, (1) approve the IT budget requests of their respective agencies and (2) review and approve IT contracts. Federal data center consolidation initiative (FDCCI). Agencies covered by FITARA are required, among other things, to provide a strategy for consolidating and optimizing their data centers and issue quarterly updates on the progress made. Enhanced transparency and improved risk management. The Office of Management and Budget (OMB) and covered agencies are to make detailed information on federal IT investments publicly available, and agency CIOs are to categorize their investments by level of risk. Portfolio review. Covered agencies are to annually review IT investment portfolios in order to, among other things, increase efficiency and effectiveness and identify potential waste and duplication. Expansion of training and use of IT acquisition cadres. Covered agencies are to update their acquisition human capital plans to support timely and effective IT acquisitions. In doing so, the law calls for agencies to consider, among other things, establishing IT acquisition cadres (i.e., multi-functional groups of professionals to acquire and manage complex programs), or developing agreements with other agencies that have such cadres. Government-wide software purchasing program. The General Services Administration is to develop a strategic sourcing initiative to enhance government-wide acquisition and management of software. In doing so, the law requires that, to the maximum extent practicable, the General Services Administration should allow for the purchase of a software license agreement that is available for use by all executive branch agencies as a single user. Maximizing the benefit of the Federal Strategic Sourcing Initiative. Federal agencies are required to compare their purchases of services and supplies to what is offered under the Federal Strategic Sourcing Initiative. In June 2015, OMB released guidance describing how agencies are to implement FITARA. This guidance was intended to, among other things: assist agencies in aligning their IT resources with statutory requirements; establish government-wide IT management controls to meet the law’s requirements, while providing agencies with flexibility to adapt to unique agency processes and requirements; strengthen the relationship between agency CIOs and bureau CIOs; and strengthen CIO accountability for IT costs, schedules, performance, and security. The guidance identifies a number of actions that agencies are to take to establish a basic set of roles and responsibilities (referred to as the common baseline) for CIOs and other senior agency officials and, thus, to implement the authorities described in the law. For example, agencies are to conduct a self-assessment and submit a plan describing the changes they intend to make to ensure that common baseline responsibilities are implemented. In addition, in August 2016, OMB released guidance intended, among other things, to define a framework for achieving the data center consolidation and optimization requirements of FITARA. The guidance directed agencies to develop a data center consolidation and optimization strategic plan that defined the agency’s data center strategy for fiscal years 2016, 2017, and 2018. This strategy was to include, among other things, a statement from the agency CIO indicating whether the agency had complied with all data center reporting requirements in FITARA. Further, the guidance states that OMB is to maintain a public dashboard to display consolidation-related costs savings and optimization performance information for the agencies. Congress has recognized the importance of agencies’ continued implementation of FITARA provisions, and has taken legislative action to extend selected provisions beyond their original dates of expiration. Specifically, Congress and the President enacted laws to: remove the expiration dates for the enhanced transparency and improved risk management provisions, which were set to expire in 2019; remove the expiration date for portfolio review, which was set to expire in 2019; and extend the expiration date for FDCCI from 2018 to 2020. In addition, Congress and the President enacted a law to authorize the availability of funding mechanisms to help further agencies’ efforts to modernize IT. The law, known as the Modernizing Government Technology (MGT) Act, authorizes agencies to establish working capital funds for use in transitioning away from legacy IT systems, as well as for addressing evolving threats to information security. The law also creates the Technology Modernization Fund within the Department of the Treasury, from which agencies can “borrow” money to retire and replace legacy systems, as well as to acquire or develop systems. Further, in February 2018, OMB issued guidance for agencies on implementing the MGT Act. The guidance was intended to provide agencies additional information regarding the Technology Modernization Fund, as well as the administration and funding of the related IT working capital funds. Specifically, the guidance encouraged agencies to begin submitting initial project proposals for modernization on February 27, 2018. In addition, in accordance with the MGT Act, the guidance provided details regarding a Technology Modernization Board, which is to consist of (1) the Federal CIO; (2) a senior IT official from the General Services Administration; (3) a member of DHS’s National Protection and Program Directorate; and (4) four federal employees with technical expertise in IT development, financial management, cybersecurity and privacy, and acquisition that were appointed by the Director of OMB. Congress and the President enacted the Federal Information Security Modernization Act of 2014 (FISMA) to improve federal cybersecurity and clarify government-wide responsibilities. The act addresses the increasing sophistication of cybersecurity attacks, promotes the use of automated security tools with the ability to continuously monitor and diagnose the security posture of federal agencies, and provides for improved oversight of federal agencies’ information security programs. To this end, the act clarifies and assigns specific responsibilities to entities such as OMB, DHS, and the federal agencies. Table 1 describes a selection of the OMB, DHS, and agency responsibilities. Beyond the implementation of FITARA, FISMA, and related actions, the administration has also initiated other efforts intended to improve federal IT and the nation’s cybersecurity. Specifically, in March 2017, the administration established the Office of American Innovation, which has a mission to, among other things, make recommendations to the President on policies and plans aimed at improving federal government operations and services. In doing so, the office is to consult with both OMB and the Office of Science and Technology Policy on policies and plans intended to improve government operations and services, improve the quality of life for Americans, and spur job creation. In May 2017, the Administration also established the American Technology Council, which has a goal of helping to transform and modernize federal agency IT and how the federal government uses and delivers digital services. The President is the chairman of this council, and the Federal CIO and the United States Digital Service Administrator are among the members. In addition, in May 2017, the President signed Executive Order 13800, Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure. This executive order outlined actions to enhance cybersecurity across federal agencies and critical infrastructure to improve the nation’s cyber posture and capabilities against cybersecurity threats. Among other things, the order tasked the Director of the American Technology Council to coordinate a report to the President from the Secretary of DHS, the Director of OMB, and the Administrator of the General Services Administration, in consultation with the Secretary of Commerce, regarding the modernization of federal IT. In response, the Report to the President on Federal IT Modernization was issued in December 2017 and outlined the current and envisioned state of federal IT. The report focused on modernization efforts to improve the security posture of federal IT. Further, it recognized that agencies have attempted to modernize systems but have been stymied by a variety of factors, including resource prioritization, ability to procure services quickly, and technical issues. The report provided multiple recommendations intended to address these issues through the modernization and consolidation of networks and the use of shared services to enable future network architectures. Further, in March 2018, the Administration issued the President’s Management Agenda, which laid out a long-term vision for modernizing the federal government. The agenda identified three related drivers of transformation—IT modernization; data, accountability, and transparency; and the workforce of the future—that are intended to push change across the federal government. The Administration also established 14 related Cross-Agency Priority goals, many of which have elements that involve IT. In particular, the Cross-Agency Priority goal on IT modernization stated that modern IT must function as the backbone of how government serves the public in the digital age. This goal established three priorities that are to guide the Administration’s efforts to modernize federal IT: (1) enhancing mission effectiveness by improving the quality and efficiency of critical services, including the increased utilization of cloud-based solutions; (2) reducing cybersecurity risks to the federal mission by leveraging current commercial capabilities and implementing cutting edge cybersecurity capabilities; and (3) building a modern IT workforce by recruiting, reskilling, and retaining professionals able to help drive modernization with up-to-date technology. On May 15, 2018, the President signed Executive Order 13833: Enhancing the Effectiveness of Agency Chief Information Officers. Among other things, this executive order was intended to better position agencies to modernize their IT systems, execute IT programs more efficiently, and reduce cybersecurity risks. The order pertains to 22 of the 24 Chief Financial Officers (CFO) Act agencies; the Department of Defense and the Nuclear Regulatory Commission are exempt. For the covered agencies, the executive order strengthened the role of agency CIOs by, among other things, requiring them to report directly to their agency head; serve as their agency head’s primary IT strategic advisor; and have a significant role in all management, governance, and oversight processes related to IT. In addition, one of the cybersecurity requirements directed agencies to ensure that the CIO works closely with an integrated team of senior executives, including those with expertise in IT, security, and privacy, to implement appropriate risk management measures. In the March 2019 update to our high-risk series, we reported that agencies still needed to complete significant work related to the management of IT acquisitions and operations. As government-wide spending on IT increases every year, the need for appropriate stewardship of that investment increases as well. However, we stated that OMB and federal agencies have not made significant progress since 2017 in taking the steps needed to improve how these financial resources are budgeted and realized. To address this issue, we highlighted the need for OMB and federal agencies to further implement the requirements of federal IT acquisition reforms, including the enhancement of CIO authority. Our update to the IT acquisitions and operations high-risk area also stressed that OMB and agencies needed to continue to implement our prior recommendations in order to improve their ability to effectively and efficiently invest in IT. Specifically, since fiscal year 2010, we have made 1,278 recommendations to address shortcomings in IT acquisitions and operations. As stated in our 2019 high-risk update, OMB and agencies should demonstrate government-wide progress by, among other things, implementing at least 80 percent of our recommendations related to managing IT acquisitions and operations. As of June 2019, OMB and agencies had fully implemented 768 (or 60 percent) of their 1,277 recommendations. Figure 1 summarizes the progress that OMB and agencies have made in addressing our recommendations compared to the 80 percent target. Overall, federal agencies would be better positioned to realize billions in cost savings and additional management improvements if they address these recommendations, including those aimed at implementing CIO responsibilities, reviewing IT acquisitions; improving data center consolidation; and managing software licenses. In all, various laws, such as FITARA and related guidance, assign 35 IT management responsibilities to CIOs in six key areas. These areas are: leadership and accountability, budgeting, information security, investment management, workforce, and strategic planning. In August 2018, we reported that none of the 24 agencies we reviewed had policies that fully addressed the role of their CIO, as called for by federal laws and guidance. In this regard, a majority of the agencies had fully or substantially addressed the role of their CIOs for the area of leadership and accountability. In addition, a majority of the agencies had substantially or partially addressed the role of their CIOs for two areas: information security and IT budgeting. However, most agencies had partially or minimally addressed the role of their CIOs for two areas: investment management and strategic planning. Further, the majority of the agencies minimally addressed or did not address the role of their CIOs for the remaining area: IT workforce. Figure 2 depicts the extent to which the 24 agencies addressed the role of their CIOs for the six areas. Notwithstanding the shortfalls in agencies’ policies addressing the roles of their CIOs, most agency officials stated that their CIOs are implementing the responsibilities even if the agencies do not have policies requiring implementation. Nevertheless, in their responses to our survey, the CIOs of the 24 selected agencies acknowledged that they were not always very effective in implementing the six IT management areas. Specifically, at least ten of the CIOs indicated that they were less than very effective for each of the six areas of responsibility. We believe that until agencies fully address the role of CIOs in their policies, agencies will be limited in addressing longstanding IT management challenges. Figure 3 depicts the extent to which the CIOs reported their effectiveness in implementing the six areas of responsibility. Beyond the actions of the agencies, however, shortcomings in agencies’ policies were also partially attributable to two weaknesses in OMB’s guidance. First, the guidance did not comprehensively address all CIO responsibilities, such as those related to assessing the extent to which personnel meet IT management knowledge and skill requirements and ensuring that personnel are held accountable for complying with the information security program. Correspondingly, the majority of the agencies’ policies did not fully address nearly all of the responsibilities that were not included in OMB’s guidance. Second, OMB’s guidance did not ensure that CIOs had a significant role in (1) IT planning, programming, and budgeting decisions; and (2) execution decisions and the management, governance, and oversight processes related to IT, as required by federal law and guidance. In the absence of comprehensive guidance, CIOs would not be positioned to effectively acquire, maintain, and secure their IT systems. In response to the survey conducted for our August 2018 report, the 24 agency CIOs also identified a number of factors that enabled and challenged their ability to effectively manage IT. Specifically, most agency CIOs cited five factors as being enablers to effectively carry out their responsibilities: (1) NIST guidance, (2) the CIO’s position within the agency hierarchy, (3) OMB guidance, (4) coordination with the Chief Acquisition Officer (CAO), and (5) legal authority. Further, three factors were cited by CIOs as major factors that have challenged their ability to effectively carry out responsibilities: (1) processes for hiring, recruiting, and retaining IT personnel; (2) financial resources; and (3) the availability of personnel/staff resources. As shown in figure 4, the five enabling factors were identified by at least half of the 24 CIOs and the three factors cited as major challenges were identified by at least half of the CIOs. Although OMB issued guidance aimed at addressing the three factors identified by a majority of the CIOs as major challenges, the guidance did not fully do so. Further, regarding the financial resources challenge, OMB recently required agencies to provide data on CIO authority over IT spending; however, its guidance did not provide a complete definition of that authority. In the absence of such guidance, agencies created varying definitions of CIO authority. Until OMB updates its guidance to include a complete definition of the authority that CIOs are to have over IT spending, it will be difficult for OMB to identify any deficiencies in this area and to help agencies make any needed improvements. In order to address challenges in implementing CIO responsibilities, we made three recommendations to OMB and one recommendation to each of the selected 24 federal agencies for each of the six IT management areas. Most agencies agreed with or had no comments on the recommendations. However, as of June 2019, none of the 27 recommendations had been implemented. We will continue to monitor the implementation of these recommendations. FITARA includes a provision to enhance covered agency CIOs’ authority through, among other things, requiring agency heads to ensure that CIOs review and approve IT contracts. OMB’s FITARA implementation guidance expanded upon this aspect of the legislation in a number of ways. Specifically, according to the guidance: CIOs may review and approve IT acquisition strategies and plans, rather than individual IT contracts; CIOs can designate other agency officials to act as their representatives, but the CIOs must retain accountability; CAOs are responsible for ensuring that all IT contract actions are consistent with CIO-approved acquisition strategies and plans; and CAOs are to indicate to the CIOs when planned acquisition strategies and acquisition plans include IT. In January 2018, we reported that most of the CIOs at 22 selected agencies were not adequately involved in reviewing billions of dollars of IT acquisitions. For instance, most of the 22 agencies did not identify all of their IT contracts. In this regard, the agencies identified 78,249 IT- related contracts, to which they obligated $14.7 billion in fiscal year 2016. However, we identified 31,493 additional IT contracts with combined obligations totaling $4.5 billion, raising the total amount obligated to IT contracts by these agencies in fiscal year 2016 to at least $19.2 billion. Figure 5 reflects the obligations that the 22 selected agencies reported to us relative to the obligations we identified. The percentage of additional IT contract obligations we identified varied among the selected agencies. For example, the Department of State did not identify 1 percent of its IT contract obligations. Conversely, eight agencies did not identify over 40 percent of their IT contract obligations. Many of the selected agencies that did not identify these IT contract obligations also did not follow OMB guidance. Specifically, 14 of the 22 agencies did not involve the acquisition office in their process to identify IT acquisitions for CIO review, as required by OMB. In addition, seven agencies did not establish guidance to aid officials in recognizing IT. We concluded that, until these agencies involve the acquisitions office in their IT acquisition identification processes and establish supporting guidance, they cannot ensure that they will identify all such acquisitions. Without proper identification of IT acquisitions, these agencies and their CIOs cannot effectively provide oversight of these acquisitions. In addition to not identifying all IT contracts, 14 of the 22 selected agencies did not fully satisfy OMB’s requirement that the CIO review and approve IT acquisition plans or strategies. Further, only 11 of 96 randomly selected IT contracts at 10 of the 22 agencies were CIO-reviewed and approved as required by OMB’s guidance. The 85 contracts that were not reviewed had a total possible value of approximately $23.8 billion. Until agencies ensure that CIOs are able to review and approve all IT acquisitions, CIOs will continue to have limited visibility and input into their agencies’ planned IT expenditures and will not be able to effectively use the increased authority that FITARA’s contract approval provision is intended to provide. Further, agencies will likely miss an opportunity to strengthen their CIOs’ authority and the oversight of acquisitions. As a result, agencies may award IT contracts that are duplicative, wasteful, or poorly conceived. As a result of these findings, we made 39 recommendations in our January 2018 report. Among these, we recommended that agencies ensure that their acquisition offices are involved in identifying IT acquisitions and issuing related guidance and that IT acquisitions are reviewed in accordance with OMB guidance. OMB and the majority of the agencies generally agreed with or did not comment on the recommendations. As of June 2019, 23 of the 39 of the recommendations had not been implemented. Data center consolidation efforts are key to implementing FITARA. Specifically, OMB established the FDCCI in February 2010 to improve the efficiency, performance, and environmental footprint of federal data center activities. The enactment of FITARA in 2014 codified and expanded the initiative. In addition, in August 2016, OMB issued a memorandum which established the Data Center Optimization Initiative (DCOI) and included guidance on how to implement the data center consolidation and optimization provisions of FITARA. Among other things, the guidance required agencies to consolidate inefficient infrastructure, optimize existing facilities, improve their security posture, and achieve cost savings. According to the 24 agencies covered by the initiative, data center consolidation and optimization efforts had resulted in approximately $4.7 billion in cost savings through August 2018. Even so, additional work remains to fully carry out the initiative. Specifically, in a series of reports that we issued from July 2011 through April 2019, we noted that, while data center consolidation could potentially save the federal government billions of dollars, weaknesses existed in several areas, including agencies’ data center consolidation plans, data center optimization, and OMB’s tracking and reporting on related cost savings. In April 2019, we reported that agencies continued to report mixed progress toward achieving OMB’s goals for closing data centers and realizing the associated savings by September 2018. Specifically, as of August 2018, over half of the agencies reported that they had met, or planned to meet, all of their OMB-assigned closure goals for tiered data centers by the deadline. Six agencies reported that they did not plan to meet their goals for tiered data centers. In addition, as of August 2018, 11 agencies reported that they had already met the goal for closing 60 percent of their non-tiered centers, three agencies reported that they planned to meet the goal by the end of fiscal year 2018, and nine agencies reported that they did not plan to meet the goal by the end of fiscal year 2018. In all, the 24 agencies reported a total of 6,250 data center closures as of August 2018, which represented about half of the total reported number of federal data centers. In addition, the agencies reported 1,009 planned closures by the end of fiscal year 2018, with an additional 191 closures planned through fiscal year 2023, for a total of 1,200 further closures. Further, in August 2018, 22 agencies reported that they had achieved $1.94 billion in cost savings for fiscal years 2016 through 2018, while two agencies reported that they had not achieved any savings. In addition to that amount, 21 agencies identified an additional $0.42 billion in planned savings through fiscal year 2018—for a total of $2.36 billion in planned cost savings from fiscal years 2016 through 2018. Nevertheless, this total is about $0.37 billion less than OMB’s goal of $2.7 billion for overall DCOI savings. From July 2011 through April 2019, we made a total of 196 recommendations to OMB and 24 agencies to improve the execution and oversight of the initiative. Most agencies and OMB agreed with our recommendations or had no comments. As of June 2019, 79 of these 196 recommendations had not been implemented. In our 2015 high-risk report’s discussion of IT acquisitions and operations, we identified the management of software licenses as a focus area, in part because of the potential for cost savings. Federal agencies engage in thousands of software licensing agreements annually. The objective of software license management is to manage, control, and protect an organization’s software assets. Effective management of these licenses can help avoid purchasing too many licenses, which can result in unused software, as well as too few licenses, which can result in noncompliance with license terms and cause the imposition of additional fees. As part of its PortfolioStat initiative, OMB has developed a policy that addresses software licenses. This policy requires agencies to conduct an annual, agency-wide IT portfolio review to, among other things, reduce commodity IT spending. Such areas of spending could include software licenses. In May 2014, we reported on federal agencies’ management of software licenses and determined that better management was needed to achieve significant savings government-wide. Of the 24 selected agencies we reviewed, only two had comprehensive policies that included the establishment of clear roles and central oversight authority for managing enterprise software license agreements, among other things. Of the remaining 22 agencies, 18 had policies that were not comprehensive, and four had not developed any policies. Further, we found that only two of the 24 selected agencies had established comprehensive software license inventories, a leading practice that would help them to adequately manage their software licenses. The inadequate implementation of this and other leading practices in software license management was partially due to weaknesses in agencies’ policies. As a result, we concluded that agencies’ oversight of software license spending was limited or lacking, thus, potentially leading to missed savings. However, the potential savings could be significant considering that, in fiscal year 2012, one major federal agency reported saving approximately $181 million by consolidating its enterprise license agreements, even when its oversight process was ad hoc. Accordingly, we recommended that OMB issue a directive to help guide agencies in managing software licenses. We also made 135 recommendations to the 24 agencies to improve their policies and practices for managing licenses. Among other things, we recommended that the agencies (1) regularly track and maintain a comprehensive inventory of software licenses and (2) analyze the inventory to identify opportunities to reduce costs and better inform investment decision making. Most agencies generally agreed with the recommendations or had no comments. As of June 2019, 27 of the 135 recommendations had not been implemented. Table 2 reflects the extent to which the 24 agencies implemented the recommendations in these two areas. We have consistently identified shortcomings in the federal government’s approach to cybersecurity. In particular, in a September 2018 report, we identified four major cybersecurity challenges: (1) establishing a comprehensive cybersecurity strategy and performing effective oversight, (2) securing federal systems and information, (3) protecting cyber critical infrastructure, and (4) protecting privacy and sensitive data. To address these challenges, we identified 10 critical actions that the federal government and other entities need to take. For example, in order to address the challenge of securing federal systems and information, we identified three actions that the agencies should take: (1) improve implementation of government-wide cybersecurity initiatives, (2) address weaknesses in federal information security programs, and (3) enhance the federal response to cyber incidents. Figure 6 depicts the 10 critical actions to address the four major cybersecurity challenges. As we have previously noted, in order to strengthen the federal government’s cybersecurity posture, agencies should fully implement the information security programs required by FISMA. In this regard, FISMA provides a framework for ensuring the effectiveness of information security controls for federal information resources. The law requires each agency to develop, document, and implement an agency-wide information security program. Such a program should include risk assessments; the development and implementation of policies and procedures to cost- effectively reduce risks; plans for providing adequate information security for networks, facilities, and systems; security awareness and specialized training; the testing and evaluation of the effectiveness of controls; the planning, implementation, evaluation, and documentation of remedial actions to address information security deficiencies; procedures for detecting, reporting, and responding to security incidents; and plans and procedures to ensure continuity of operations. Since fiscal year 2010, we have made 3,058 recommendations to agencies aimed at addressing the four cybersecurity challenges. These recommendations have identified actions for agencies to take to strengthen technical security controls over their computer networks and systems. They also have included recommendations for agencies to fully implement aspects of their information security programs, as mandated by FISMA. Nevertheless, many agencies continue to be challenged in safeguarding their information systems and information, in part, because many of these recommendations have not been implemented. Of the 3,058 recommendations made since 2010, 2,384 (or 78 percent) had been implemented as of June 2019, leaving 674 recommendations (or 22 percent) unimplemented. In order to determine the effectiveness of the agencies’ information security programs and practices, FISMA requires federal agencies’ inspectors general to conduct annual independent evaluations. The agencies are to report the results of these evaluations to OMB, and OMB is to summarize the results in annual reports to Congress. In these evaluations, the inspectors general are to frame the scope of their analyses, identify key findings, and detail recommendations to address the findings. The evaluations also are to capture maturity model ratings for their respective agencies. Toward this end, in fiscal year 2017, the inspector general community, in partnership with OMB and DHS, finalized a 3-year effort to create a maturity model for FISMA metrics. The maturity model aligns with the five function areas in the NIST Framework for Improving Critical Infrastructure Cybersecurity (Cybersecurity Framework): identify, protect, detect, respond, and recover. This alignment is intended to help promote consistent and comparable metrics and criteria and provide agencies with a meaningful independent assessment of their information security programs. The maturity model is designed to summarize the status of agencies’ information security programs on a five-level capability maturity scale. The five maturity levels are defined as follows: Level 1 (Ad hoc): Policies, procedures, and strategy are not formalized; activities are performed in an ad-hoc, reactive manner. Level 2 (Defined): Policies, procedures, and strategy are formalized and documented but not consistently implemented. Level 3 (Consistently Implemented): Policies, procedures, and strategy are consistently implemented, but quantitative and qualitative effectiveness measures are lacking. Level 4 (Managed and Measurable): Quantitative and qualitative measures on the effectiveness of policies, procedures, and strategy are collected across the organizations and used to assess them and make necessary changes. Level 5 (Optimized): Policies, procedures, and strategy are fully institutionalized, repeatable, self-generating, consistently implemented and regularly updated based on a changing threat and technology landscape and business/mission needs. According to this maturity model, Level 4 (managed and measurable) represents an effective level of security. Therefore, if an inspector general rates an agency’s information security program at Level 4 or Level 5, then that agency is considered to have an effective information security program. For fiscal year 2017, the inspectors general for six of the 23 civilian CFO Act agencies reported that their agencies had an effective agency-wide information security program. Specifically, for the five function areas in the NIST Cybersecurity Framework, most inspectors general reported that their agencies were at Level 3 (consistently implemented) for the identify, protect, and recover functions, and at Level 2 (defined) for the detect and respond functions. Table 3 shows the individual maturity ratings for each covered agency. In its efforts toward strengthening the federal government’s cybersecurity, OMB also requires agencies to submit related cybersecurity metrics as part of its Cross-Agency Priority goals. In particular, OMB developed the IT modernization goal so that federal agencies will be able to build and maintain more modern, secure, and resilient IT. A key part of this goal is to reduce cybersecurity risks to the federal mission through three strategies: manage asset security, protect networks and data, and limit personnel access. The key targets supporting each of these strategies correspond to areas within the FISMA metrics. Table 4 outlines the strategies, their associated targets, and the 23 civilian CFO Act agencies’ progress in meeting those targets, as of June 2018. In conclusion, by addressing the high-risk areas on improving the management of IT acquisitions and operations and ensuring the cybersecurity of the nation, the government has the opportunity to both save billions of dollars and advance the efficiency and effectiveness of government services. Most agencies have taken steps to execute key IT management and cybersecurity initiatives, including implementing CIO responsibilities, requiring CIO reviews of IT acquisitions, realizing data center consolidation cost savings, managing software assets, and complying with FISMA requirements. The agencies have also continued to address the recommendations that we have made over the past several years. Nevertheless, further efforts by OMB and federal agencies to implement our previous recommendations would better position them to improve the management and security of federal IT. To help ensure that these efforts succeed, we will continue to monitor agencies’ efforts toward implementing the recommendations. Chairman Connolly, Ranking Member Meadows, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Carol C. Harris, Director of Information Technology Acquisition Management Issues, at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Kevin Walsh (Assistant Director), Meredith Raymond (Analyst-in-Charge), Chris Businsky, and Rebecca Eyler. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The federal government plans to spend over $90 billion in fiscal year 2019 on IT. Even so, IT investments have too often failed or contributed little to mission-related outcomes. Further, increasingly sophisticated threats and frequent cyber incidents underscore the need for effective information security. To focus attention on these concerns, GAO's high-risk list includes both the management of IT acquisitions and operations and cybersecurity. This statement summarizes federal agencies' progress in improving the management and ensuring the security of federal IT. It is primarily based on GAO's reports issued between July 2011 and April 2019 on (1) CIO responsibilities, (2) CIO IT acquisition review requirements, (3) data center consolidation efforts, (4) the management of software licenses, and (5) cybersecurity. The Office of Management and Budget (OMB) and federal agencies have taken steps to improve the management of information technology (IT) acquisitions and operations and ensure federal cybersecurity through a series of initiatives. As of June 2019, federal agencies had fully implemented 60 percent of the 1,277 IT management-related recommendations that GAO has made to them since fiscal year 2010. Likewise, agencies had implemented 78 percent of the 3,058 security-related recommendations that GAO has made since 2010. Even with this progress, significant actions remain to be completed. Chief Information Officer (CIO) responsibilities . Laws such as the Federal Information Technology Acquisition Reform Act (FITARA) and related guidance assigned 35 key IT management responsibilities to CIOs to help address longstanding challenges. In August 2018, GAO reported that none of the 24 selected agencies had established policies that fully addressed the role of their CIO, as called for by laws and guidance. GAO recommended that OMB and each of the 24 agencies take actions to improve the effectiveness of CIOs' implementation of their responsibilities. As of June 2019, none of the 27 recommendations had been implemented. CIO IT acquisition review . According to FITARA, covered agencies' CIOs are required to review and approve IT contracts. Nevertheless, in January 2018, GAO reported that most of the CIOs at 22 covered agencies were not adequately involved in reviewing billions of dollars of IT acquisitions. Consequently, GAO made 39 recommendations to improve CIO oversight for these acquisitions. As of June 2019, 23 of the recommendations had not been implemented. Consolidating data centers . OMB launched an initiative in 2010 to reduce data centers. According to 24 agencies, data center consolidation and optimization efforts had resulted in approximately $4.7 billion in cost savings through August 2018. Even so, additional work remains. GAO has made 196 recommendations to OMB and agencies to improve the reporting of related cost savings and to achieve optimization targets. As of June 2019, 79 of the recommendations had not been implemented. Managing software licenses . Effective management of software licenses can help avoid purchasing too many licenses that result in unused software. In May 2014, GAO reported that better management of licenses was needed to achieve savings, and made 136 recommendations to improve such management. As of June 2019, 27 of the recommendations had not been implemented. Ensuring the nation's cybersecurity . While the government has acted to protect federal information systems, GAO has consistently identified shortcomings in the federal government's approach to cybersecurity. The 3,058 recommendations that GAO made to agencies since 2010 have been aimed at addressing cybersecurity challenges. These recommendations have identified actions for agencies to take to fully implement aspects of their information security programs and strengthen technical security controls over their computer networks and systems. As of June 2019, 674 of the recommendations had not been implemented. Since fiscal year 2010, GAO has made about 1,300 recommendations to OMB and agencies to address shortcomings in IT acquisitions and operations, as well as approximately 3,000 recommendations to agencies to improve the security of federal systems. These recommendations addressed, among other things, implementation of CIO responsibilities, oversight of the data center consolidation initiative, management of software license efforts, and the efficacy of security programs and technical controls. Implementation of these recommendations is essential to strengthening federal agencies' acquisitions, operations, and cybersecurity efforts.", "document_type": "gao"}
{"report": "We have previously reported that multiple federal programs provide or support early learning and child care, but the CCAMPIS program is the only one designed specifically to support the participation of low-income parents in postsecondary education by funding child care services. Education awards CCAMPIS competitive grants for up to 4 years to colleges to either support existing campus-based child care programs or establish new programs. Grant funds are primarily intended to help students who receive or are eligible to receive federal Pell Grants, but grantees may also serve low-income graduate students or low-income foreign students. Education reported that CCAMPIS grantees received about $15 million in fiscal year 2017 and about $33 million in fiscal year 2018. HHS administers other key federally funded programs that subsidize child care that may assist college students: the Child Care and Development Fund (CCDF), Temporary Assistance for Needy Families (TANF), and Head Start. CCDF is the primary source of federal funding dedicated to helping low-income families pay for child care. Parents must generally be working or attending a job training or education program to receive CCDF child care subsidies. States have flexibility to establish program eligibility criteria and other priorities within the program’s broad federal requirements. According to the HHS fiscal year 2020 budget justification, the CCDF program provides about $8.2 billion in federal funds per year for child care. In fiscal year 2017, the latest year for which preliminary data were available, CCDF provided child care assistance to about 1.3 million children each month. TANF is a federal block grant to states that supports cash assistance and a variety of other benefits and services to low-income families with children. States may use their TANF funds to directly fund child care, both for families receiving TANF cash assistance and for other low-income families in the state. In 2017, 9 percent of federal TANF funds used—or $1.5 billion—was spent directly for child care, while states spent $2.3 billion in maintenance of effort funds directly on child care, according to the HHS fiscal year 2020 budget justification. In addition, states transferred $1.3 billion in federal TANF funds to CCDF in fiscal year 2017. Head Start grants are awarded directly to public and private nonprofit and for-profit preschool and child care providers. The purpose of the Head Start program is to promote the school readiness of low-income children through the provision of educational, health, and other services. Most Head Start participants are 3- and 4-year-old children, but through the Early Head Start program, many infants and toddlers also receive early education and child care services. In fiscal year 2017, Head Start provided about $9.6 billion in grants and other services, and the program served over 1 million children. Under Title IV of the Higher Education Act of 1965, as amended, the federal government offers students financial assistance to help pay for their education. To be eligible for most federal student aid, a student must demonstrate financial need. Students are eligible for federal need based aid if the cost of attending a school is more than a family’s expected financial contribution. A family’s expected contribution is an approximation of the financial resources a family has available to help pay for a student’s postsecondary education expenses. The cost of attendance is calculated by each school using elements set forth in federal law. In addition to expenses such as tuition, fees, and room and board, the cost of attendance may include a dependent care allowance for students who incur such costs—including for child care— while in school. Being eligible for a dependent care allowance increases the student’s total cost of attendance, which could make the student eligible for additional financial assistance. Federal student aid is awarded primarily through grants and loans. Grants: Federal Pell Grants are the primary federal grant aid available to low-income undergraduate students with financial need. The maximum allowable Pell Grant was $6,095 for the 2018-2019 school year. A student’s expected family contribution is a key determinant of Pell Grant eligibility. Federal Direct Loans: Education provides loans to undergraduate and graduate students both with and without financial need. The maximum amount an undergraduate student may borrow in federal student loans is based on the student’s year in school and dependency status (see table 1). Students are classified as either financially dependent on their parents or financially independent. Students with dependent children are categorized as independent students for the purpose of calculating federal student aid. In addition, the total amount of grants and scholarships plus the total amount of federal student loans a student receives cannot exceed the total cost of attendance at his or her school. As a result, some students may be eligible for a lower federal loan amount than the maximum allowable amount, after grant and scholarship aid are factored in. For example, if an independent, first-year student’s total cost of attendance is $20,000, and the student receives $12,000 in grant and scholarship aid, the student can take out no more than $8,000 in federal student loans, which is less than the first-year limit of $9,500. Under federal law, schools participating in federal student aid programs are required to disclose certain consumer information, including information about college costs and the availability of federal student aid. We have previously reported that schools are increasingly using their websites to share consumer information, according to Education officials. Schools must also post a tool on their websites to help students estimate their cost of attendance based on their individual circumstances. Student parents comprised about 20 percent of undergraduate students, and many had characteristics that Education has reported can affect their likelihood of staying enrolled in school and completing a degree, such as being a single parent and working full time. In 2015-2016, an estimated 22 percent of undergraduate students (4.3 million of 19.5 million) were parents, according to our analysis of Education’s nationally representative NPSAS data. This percentage has remained close to one-quarter since 2003-2004, peaking at nearly 26 percent in 2011-2012. In addition, about 55 percent (2.4 million) were single parents and 44 percent (1.9 million) were working full-time while enrolled (see fig. 1). About 23 percent (nearly 1 million) were single parents working full-time while enrolled. In addition, undergraduate student parents in 2015-2016 were older than other students and mostly female, and a higher percentage were African- American compared to students without children. The average age of undergraduate student parents was 33, compared to 24 for all other undergraduates. A relatively small proportion of undergraduate student parents—15 percent—was age 23 or younger. Most student parents were female (71 percent). An estimated 23 percent of undergraduate student parents were African-American, compared to 13 percent of all other undergraduates (see app. II for additional information on student parent characteristics). Education data indicate that a lower percentage of undergraduate student parents earned a degree compared to students without children. According to our analysis of the 2009 BPS data—a 6-year follow-up survey of the cohort of first-time students in the 2003-2004 school year— an estimated 52 percent of undergraduate student parents left school without a degree within 6 years, compared to 32 percent of students without children (see fig. 2). Compared to students without children, a higher percentage of undergraduate student parents were enrolled in private for-profit schools, programs of two years or less, and online programs, according to NPSAS data for 2015-2016. An estimated 25 percent of undergraduate student parents were enrolled in programs taught entirely online, compared to 7 percent of all other undergraduates (see fig. 3). Undergraduate student parents had fewer financial resources available to fund their education than students without children, according to NPSAS data for 2015-2016. An estimated 67 percent of undergraduate student parents in 2015-2016 had an expected family contribution of zero, compared to 31 percent of students without children. Student parents also had an average expected family contribution of $9,180, compared to $17,506 for students without children. Accordingly, about half of student parents received a federal Pell Grant, compared to 35 percent of all other undergraduates. In addition, a higher percentage of student parents rely on federal student loans compared with other students. Approximately 62 percent of undergraduate student parents used federal student loans for their education, compared to 50 percent of students without children. About half of student parents had childcare expenses, in addition to their education and other living expenses. An estimated 45 percent reported paying for child care in 2015-2016, paying an average of about $490 per month (see fig. 4). An estimated 56 percent of student parents had a child age 5 or younger. However, about 60 percent of undergraduate student parents were enrolled in schools that did not offer on-campus childcare for students. CCAMPIS grantees reported that about 3,320 student parents received subsidized child care services for at least one academic term during the 2016-2017 school year, the most recent year for which performance data were available. The 85 schools that submitted CCAMPIS program data for this time period were about evenly split between 2-year (42) and 4- year (43) schools. The average amount awarded to each school for the year was approximately $182,000. Grantees reported that there were more children of CCAMPIS-eligible parents on waiting lists to receive child care services (over 4,200 children) than the number of children served by the 85 schools (about 4,000). Many of the children on waiting lists were infants and toddlers (65 percent). Most CCAMPIS participants in 2016-2017 were female and low-income undergraduate students, according to data reported by grantees. Further, most participants were undergraduates who either received or were eligible to receive federal Pell Grants (85 percent). About 10 percent were low-income graduate students. Almost 80 percent of CCAMPIS participants were female. A majority of female CCAMPIS participants attended 2-year schools (53 percent). In contrast, most male participants were enrolled in 4-year schools (70 percent). Grantee reported data also indicate that about half of CCAMPIS participants were single parents, although most male students served by the grant were married (78 percent). Just under half of CCAMPIS participants were white, 25 percent Hispanic or Latino, and 15 percent were Black or African-American. Grantees reported using CCAMPIS funds to subsidize a variety of child care services, either provided on-campus or in the community. Almost all grantee schools (84) reported using CCAMPIS funds to subsidize full-time child care, while 72 funded part-time child care (see fig. 5). Fewer schools funded before- or after-care or care during the evening (18 schools) or weekends (5 schools). Many grantees also reported funding parenting classes (e.g., workshops on time management and family nutrition) and meetings (e.g., student parent advisory board meetings). Grantees funded other activities with their CCAMPIS grants, such as student advising, free finals week child care, and child health screenings, according to grantee data. While some schools paid for the entire cost of child care for CCAMPIS participants, most provided partial subsidies using a sliding fee scale. Among the students that grantees reported as receiving a CCAMPIS- funded child care subsidy, over 75 percent had some out-of-pocket child care expenses (2,091 of 2,754). The median amount students paid out- of-pocket each month was about $160, after receiving about $385 per month in grant-funded subsidies. CCAMPIS grants can help schools address the demand for child care that their on-campus child care centers have not been able to accommodate. For example, prior to receiving the CCAMPIS grant, the on-campus child care center at one 2-year school on the West Coast served children age 2.5 to 5 years, according to school officials. With CCAMPIS grant funding, officials said they were able to expand on-campus child care for school- age children (ages 5-13). They said the grant also allows the school to offer drop-in child care when local elementary schools are closed. In another case, to help meet demand for child care among student parents that its on-campus child care center could not accommodate, an official from a 4-year school in the Rocky Mountain region said the school has established relationships with approximately 20 community-based child care centers and used CCAMPIS funds to help students pay for child care provided by these off-campus centers. These CCAMPIS grantees told us they also used grant funds to offer students supportive services in addition to subsidized child care. For example, the 2-year school on the West Coast runs a family resource center that provides free baby clothes, diapers, wipes, college textbooks, and school supplies for students and their children. The 4-year school has used CCAMPIS funds to pay for a graduate student to provide home visits for student parents who have concerns about their children's development or behavior. These schools also relied on funding from other sources to support student parents. For example, officials from the 2-year school we spoke with said the school uses local funds to host weekly faculty-led playgroups and state funding to increase student parent access to food pantries and housing assistance and to host evening parenting workshops led by a marriage and family counselor. In its budget justification to Congress, Education reports on the progress that CCAMPIS grantees make toward meeting the program’s performance goals; however, flaws in its calculations prevented Education from reporting reliable results. Education reports information on three performance measures for CCAMPIS participants: their persistence in school, the federal cost for each persistent student, and their graduation rate. The persistence rate for students participating in the CCAMPIS program is the percentage of program participants who receive child care services that remain in postsecondary education at the end of the academic year, according to Education’s published definition. To calculate this measure Education’s explanation states that it includes any student that has remained enrolled in school at the end of the school year, transferred from a 2-year to a 4-year school during the school year, or graduated during the school year. However, Education’s calculations did not produce results that align with this definition of persistence; specifically, the agency’s calculations did not identify students who remained enrolled until the end of the school year. Education counted a student as persisting if the grantee reported the student as enrolled and participating in the CCAMPIS program in either the fall or the winter terms and did not consider whether students were also enrolled in another term during the year. As a result, a student who was enrolled and participating in CCAMPIS during the fall term and withdrew from school during the spring term was counted as having persisted in school. Further, while Education’s calculation included students who graduated at some point during the school year, it did not include students who transferred from a 2-year to a 4 year school. Using Education’s definition, we recalculated the percentage of CCAMPIS participants who persisted until the end of the 2016-2017 school year. Specifically, we limited our analysis to students who grantees reported as having participated in CCAMPIS during either the fall or winter term and persisted to the spring term. While Education reported a persistence rate of about 74 percent in its fiscal year 2020 budget justification to Congress, our recalculation indicated that the persistence rate was an estimated 82 percent. The flaws in Education’s persistence rate calculation meant the agency also reported unreliable results for the federal cost per CCAMPIS participant who persisted in school. Given our recalculation of the persistence rate for students enrolled in both 2-year and 4-year schools, we calculated that the cost per CCAMPIS participating student who persisted during the 2016-2017 school year was about $7,550. Education reported this cost as $5,625 in its fiscal year 2020 budget justification to Congress. Education defines its graduation rate measure as the percentage of CCAMPIS program participants enrolled in 2-year schools who graduate from postsecondary education within 3 years of enrollment. According to Education’s published definition of this measure, it is intended to be consistent with Education’s standard graduation rate reported by all 2- year schools that receive federal student aid funds. Education does not calculate or report the graduation rate for CCAMPIS participants enrolled in 4-year schools. However, Education’s calculations did not produce results that aligned with its published graduation rate definition. To correctly calculate the graduation rate, based on its definition, Education would need to track the enrollment of a cohort of CCAMPIS participating students in 2-year schools who started school in the same year. This would allow Education to follow these students over 3 years to identify how many of them graduated during this time period. Instead, Education included in its calculation students that participated in CCAMPIS at any point during a 3- year period regardless of when they first enrolled in school. Education does not currently collect data from CCAMPIS grantees that indicate when students first enrolled in school, which it would need to accurately calculate the percentage of CCAMPIS program participants enrolled in 2- year schools who graduate within 3 years of enrollment. Education officials said that they were concerned that collecting such student enrollment information could be overly burdensome for grantees. Education officials acknowledged that they had not accurately defined this performance measure in the fiscal year 2020 budget justification to Congress. Specifically, Education officials said that although the published definition of the CCAMPIS graduation rate states it is consistent with the agency’s standard graduation rate measure, program officials actually calculate something different. Officials said that because they do not collect data on when students first enroll in school, they calculated the percent of CCAMPIS participants who graduated within 3 years of receiving CCAMPIS subsidies instead. While this alternative could be used as a CCAMPIS outcome measure, Education’s calculations did not align with this definition because they did not organize students into cohorts based on when they first started receiving CCAMPIS subsidies. Education has the data to do this, but would need to revise its calculations. Without either collecting the student enrollment data needed to calculate a standard 3-year graduation rate or accurately defining and calculating a different metric, Education is unable to report reliable college completion results for CCAMPIS participants. Having accurate performance measures is critical to assessing the effectiveness of the CCAMPIS program. Federal standards for internal controls state that management should ensure that measurements achieve the appropriate level of precision and accuracy for their reporting purposes. These federal standards also state that when communicating with external parties, managers should share quality information to help the entity achieve its objectives. However, Education has not calculated a persistence rate or graduation rate that accurately reflects the CCAMPIS program’s performance measures, as the agency has publicly defined them. As a result, the agency is unable to give a reliable accounting of CCAMPIS performance in its budget justification to Congress. Reporting unreliable performance information about the CCAMPIS program affects Education’s ability to manage the program and Congress’ ability to make informed funding and program decisions. College students may benefit from other key federal programs that fund child care services for low-income families—CCDF, TANF, and Head Start—but little is known about the extent to which they benefit. Child Care and Development Fund: HHS does not track how many families use CCDF child care subsidies specifically to pursue postsecondary education, as this is an optional program activity, according to HHS officials. HHS tracks and reports on child care subsidy use for training and education as a broader category. For fiscal year 2016, in about 6 percent of families receiving child care subsidies a parent was enrolled in training or education, and in an additional 7 percent of families a parent was enrolled in training or education while also employed, according to state reported data. These data also show that states differed in the extent to which parents pursuing training or education received such subsidies. For example, three states provided CCDF subsidies during an average month in 2016 to only a small number of families where a parent was not employed while pursuing education or training (less than one-half of one percent). In contrast, two states provided CCDF subsidies to about 20 percent of families where a parent was pursuing education or training while not employed. Some states have established policies that restrict postsecondary students’ access to CCDF funds, according to our analysis of an HHS report containing information on key state CCDF policies as of 2017. Specifically, four states have policies that limit students who are pursuing postsecondary education from receiving child care subsidies. Nine additional states do not allow access to child care subsidies for full-time students, unless they also meet work requirements. For example, Arizona, Kentucky, Pennsylvania, and Washington require full-time students to work 20 hours each week in addition to attending school. States have implemented other policies that affect CCDF subsidy access for postsecondary students. Program length: Eight states limit the length of time students may receive child care subsidies for enrollment in a postsecondary program. For example, Alabama, Kansas, New Hampshire, and Wisconsin limit postsecondary programs to 24 months. Program type: Ten states place restrictions on the type or nature of the postsecondary program students may pursue. For example, states may limit approved programs to vocational programs. Almost all states exclude graduate level programs. Academic Achievement: Four states have policies related to the minimum grade point average students must maintain to receive child care subsidies. For example, Illinois requires that students that do not work 20 hours per week maintain a 2.5 average. In 2016, HHS issued an informational memorandum with examples of policies and practices that could help states support parents who need child care assistance to participate in education programs. Such strategies included limiting the number of hours students were required to work and ensuring student parents are aware of child care services. See the text box for an example of how one school reported it is using CCDF funds to assist student parents. Example of using the Child Care and Development Fund (CCDF) to subsidize child care for student parents New York state uses CCDF funds to offer child care subsidies to students enrolled in its State University of New York (SUNY) and City University of New York (CUNY) schools. These colleges partner with nonprofit child care providers and receive CCDF funds to provide child care subsidies to income-eligible students. Schools receive additional state funds to help pay for child care operating costs, such as staff salaries, supplies, and meals for children. A school official from one of the state’s community colleges told GAO that CCDF subsidizes care during time students are in class. School officials submit students’ class schedules to the state when applying for benefits on students’ behalf. Students pay out of pocket for any time they elect to enroll their children in care that is in addition to scheduled class time. However, eligible families may not receive a CCDF child care subsidy, as states often do not have sufficient funds to serve all eligible families. HHS officials said that states must prioritize three types of eligible families: families with very low incomes, families with children with special needs, and families who are experiencing homelessness. Temporary Assistance for Needy Families: Student parents may also be eligible to receive child care subsidies from their state’s TANF program, but it is unclear how many students benefit from these subsidies. HHS officials said that although they track the amount of TANF funds states use to help families pay for child care, HHS does not collect information that would allow it to identify how many families are using child care to pursue postsecondary education. According to NPSAS data, an estimated 4 percent of undergraduate student parents reported that a member of their household received TANF assistance during either 2013 or 2014. For an example of how one school reported it is using TANF funds to assist student parents, see the text box. Example of using Temporary Assistance for Needy Families (TANF) to subsidize child care for student parents A TANF-funded program in Arkansas called the Career Pathways Initiative assists student parents with child care costs. This program also offers financial assistance for school-related expenses and a number of other supportive services. To access child care assistance from the Career Pathways Initiative, student parents must have an income at or below 250 percent of the poverty level or receive another state service, such as Medicaid. According to an official from one community college in the state, in order to receive a child care subsidy at that school, students must also work at least one hour per week. Research on the Career Pathways Initiative found that, of the nearly 30,000 low-income participants enrolled in Arkansas community colleges between 2006 and 2013, more than 52 percent graduated with a degree or certificate. This is more than double the 24 percent completion rate of all Arkansas community college students who did not participate in the program. Metis Associates and the Arkansas Research Center, “College Counts Evidence of Impact: A Research Analysis of the Arkansas Career Pathways Initiative.” January 2018. Head Start: Student parents may also enroll their children in Head Start programs, and some colleges have received Head Start grants or partnered with local Head Start programs to connect eligible student parents with services. HHS officials said they do not, however, collect information from Head Start grantees to identify how many grantees partner with colleges or how many Head Start grantees themselves are colleges. They also said they cannot quantify the number of student parents with children enrolled in Head Start programs because that information is not collected by the Office of Head Start, as this is not a primary purpose of the program. At many Head Start programs— particularly those located in early learning or child care centers—services are only available on a part day or part week basis, which may not align with a student’s school or work schedules. See the text box for an example of how one school reported it is supporting its student parents with Head Start funds. Example of using Head Start to subsidize child care for student parents A community college district in the Northwest that comprises two campuses, has received a Head Start grant for approximately the past 25 years. According to school officials, in 2018, the district managed 9 Head Start and Early Head Start centers located across the county, including centers on each of the district’s community college campuses. According to school officials, the district used Head Start funding to offer family well-being services for student parents, including helping families find housing, providing referrals for mental health counseling, and providing bus passes. In addition, the program connected families with medical and dental services. For the 2017-2018 school year, the district reported that over 88 percent of children enrolled in its Head Start centers were up-to-date on dental and medical screenings. In certain circumstances, a student parent may be eligible to receive additional federal student aid to help pay for child care. Students with dependent children in paid child care are allowed to request a dependent care allowance as part of their financial aid calculation, but whether it provides them with additional financial aid depends largely on other school costs. For example, at higher-cost schools, these students may already be eligible for the maximum amount of federal student loans before adding this allowance. In these situations, requesting a dependent care allowance would not increase the amount of federal student loans available to the student because they have already reached the maximum. At lower-cost schools, such as community colleges, costs may be low enough to allow student parents to access additional federal student loans by adding a dependent care allowance. According to our analysis of 2016 NPSAS data, an estimated 2.6 million student parents nationwide were eligible for a lower federal loan amount than the maximum allowable loan amount, so that adding a dependent care allowance might make them eligible for a higher federal loan amount. Figure 6 illustrates how adding a dependent care allowance can affect a student’s federal student loan amount at a school with a relatively low cost of attendance. In this example, adding a $3,000 dependent care allowance to a student’s cost of attendance increases the amount of federal student loans the student can borrow without exceeding the maximum amount available ($9,500 for a first-year, independent undergraduate student). At a higher-cost school, however, a student may already be eligible for the maximum possible loan amount, so adding a dependent care allowance would not affect how much the student could take out in federal student loans. Officials from seven of the 13 schools we interviewed said that adding a dependent care allowance would more likely increase the amount of federal student loans a student can borrow, rather than increase a student’s access to grant or scholarship aid. However, school officials we interviewed who recently added dependent care allowances to students’ financial aid calculations said that students with a dependent care allowance may, in some cases, receive additional grants from the state or school. Officials at most of the 13 schools we contacted said they receive relatively few requests for a dependent care allowance, generally ranging from zero to 47 in 1 year. Officials at the eight schools that had included this allowance in recent years reported different ways of determining the amount of the allowance. At two of these schools, officials said they allot a fixed amount for the dependent care allowance. Officials at the other six schools said allowance amounts are flexible and based on students’ documented child care expenses, and can vary depending on the number of children in child care. Fixed. One school in the Northwest surveys local child care providers annually to determine the community standard rate each year and bases its dependent care allowance amount on the average market value in the area, according to a school official. This rate is two tiered. The first tier is for children ages 0-5 and is $552 per month and the second is for children ages 6-12, with a monthly allowance of $276. At a school in the Midwest, an official said that the school provides a fixed allowance amount to all students who indicate they have a dependent child on the Free Application for Federal Student Aid (FAFSA). The allowance amount is based on the student’s enrollment status (e.g., $900 per school year for a student enrolled full-time). The school included a dependent care allowance for 30 percent of students who received financial aid in 2017-2018, according to a school official. Flexible. At one school in the South, students can request a dependent care allowance based on their actual child care expenses, according to a school official. Financial aid officials at the school use their judgment to determine if the request is reasonable for the community and may request documentation for requests exceeding $2,500 per semester. For example, students with more than one child may spend more than $2,500 per semester on child care. An official at another school in the West said it does not set limits for the allowance, but financial aid counselors use their judgment to counsel students if the requested amount looks too high for the student’s circumstances. The average allowance amount at this school is between $600 and $1,000 per month. Not all students may want to increase their student loans to finance their child care costs while in school, but access to additional federal student loans could be a useful option for those students who need it. We previously reported that officials at a national association of community colleges said that low-income students often use federal loans to help them pay for basic living expenses. These loans can be a valuable resource for some students who need additional funds to support themselves while in college, but some school officials cautioned that loans may not be the best choice for all students, and may worsen the financial position of already vulnerable students. However, two recent studies of 2-year students examined how federal financial aid improved students’ college outcomes. One study found that federal financial aid helped reduce the drop-out risk for some students, while another study found that students who received federal student loans had completed more college credits and earned higher grades than those who did not. About two-thirds of the college websites we reviewed (40 of 62) did not include information on their websites about the option to include a dependent care allowance in financial aid calculations. While schools are required to post certain college cost information on their websites and inform students about the availability of financial aid, they are not required to inform students about the dependent care allowance. At 29 of these 40 schools, the average net price for a low-income student is low enough that some students may qualify for additional loan amounts with the addition of a dependent care allowance. We reviewed the websites of schools that were CCAMPIS grant recipients. As CCAMPIS grant recipients, these schools serve students with a demonstrated need for child care services, and have shown an interest in helping students with their child care needs. Given that most of these schools do not provide information online about the option to include a dependent care allowance, other schools without the same focus on student parents may be even less likely to make information about this option available. If schools are not consistently informing students about the option to access additional federal student aid, student parents who could benefit may not be aware the option exists, and therefore not apply for additional aid that could help them pay for child care. Among the 22 schools that did provide information about the dependent care allowance on their websites, we found that the details they provided varied. They ranged from a general statement on the existence of the allowance to explicit instructions on how to request it, and, in some cases, the specific dependent care allowance amounts the school would provide. Three of the 22 schools that discussed the dependent care allowance on their websites did not post any instructions on how to add the allowance to financial aid calculations, and the instructions posted on the other 19 school websites varied. Such instructions included directing students to contact the financial aid office and submitting a financial aid award appeal. Among the 13 schools at which we conducted interviews, six schools included information about the dependent care allowance on their websites and seven did not. Officials at two schools that publicized the allowance on their websites said that the schools also took other steps to inform current students about the dependent care allowance. For example, one school official said the school references the allowance in emails to students about the on-campus child care center. Officials at the other schools—including the seven schools that did not include information on their websites—said that they did not use any other method to inform current students about the dependent care allowance. Further, none of the 13 schools made information publicly available to prospective students using anything other than the schools’ website, according to school officials. Additionally, although not generalizable, there was a relationship between those schools that used their websites to inform student parents about the option to include a dependent care allowance and whether they had provided this allowance in recent years. All six schools that provided dependent care allowance information online also reported including this allowance in the financial aid calculations of some students in recent years. Of the seven schools that did not include dependent care information online, just two of them reported that they had provided any dependent care allowances in recent years. Education uses its Federal Student Aid (FSA) Handbook—a comprehensive annual guide to regulatory and administrative requirements for federal student aid programs—to instruct school financial aid officials on how to incorporate the dependent care allowance in a student’s financial aid calculations. However, the handbook does not encourage schools to make information readily available to students via school websites about the option to increase federal student aid to help pay for child care or what steps they need to take to request it. Posting this information on school websites would make it more easily accessible to students, including prospective students who may not have access to publications located on campus. Education has used its handbook to encourage schools to adopt other suggested practices, such as informing students about how to save money on textbooks by either renting them or purchasing used copies. Moreover, Education officials said that they could include language in the handbook encouraging schools, as a best practice, to include information about the dependent care allowance on school websites along with other college cost information. Federal standards for internal control state that management should consider the availability of information and the extent to which information is readily available to the intended audience. Because the dependent care allowance can affect how much financial aid a student can access, making this information accessible on school websites would help ensure enrolled and prospective students are aware of all of their financial aid options. Student parents face many obstacles to completing college, including paying for child care, and are less likely to complete school than students without children. The CCAMPIS program offers financial assistance that can provide key support to help student parents complete college. However, because Education is not accurately calculating its CCAMPIS performance measures, the agency is not reporting reliable information on program outcomes. As a result, it is difficult for Education and Congress to evaluate the effectiveness of the program and make informed funding and program decisions. Federal student aid can be an important resource available to help student parents—who have fewer financial resources than other students—pay for child care while enrolled in school, but only if students are aware of the option to increase aid to help cover child care costs. Without information made widely available on school websites, student parents who could benefit may not know they can obtain additional aid. Moreover, the challenges this population faces in completing college make it especially important that they know about the types of assistance available to them. This information is particularly important for prospective students as they consider costs among different schools. Encouragement from Education for schools to provide information about the dependent care allowance on their websites could offer student parents more complete information about the financial aid resources available to them and how to request additional aid that could ultimately help them remain in school and graduate. We are making the following three recommendations to Education. The Assistant Secretary for Postsecondary Education should correctly calculate its CCAMPIS program persistence rate and cost per persisting student measures. (Recommendation 1) The Assistant Secretary for Postsecondary Education should either collect the CCAMPIS participant enrollment data needed to calculate a standard 3-year graduation rate or accurately define and calculate a different college completion measure. (Recommendation 2) The Chief Operating Officer of Federal Student Aid should encourage schools—through appropriate means, such as the FSA Handbook—to inform students via school websites about the availability of the dependent care allowance and how to request the allowance. (Recommendation 3) We provided a draft of this report to HHS and Education for review and comment. HHS provided technical comments, which we incorporated as appropriate. Education provided written comments, which are reproduced in appendix III. In its comments, Education stated that the report inaccurately characterizes the CCAMPIS performance data as unreliable and disagreed with the recommendation to correctly calculate its CCAMPIS persistence rate and cost per persistent student measures (recommendation 1). Regarding this first recommendation, Education acknowledged one error in its persistence rate calculation that affected the accuracy of both the persistence rate and cost per persistent student measures that it reported in its fiscal year 2020 budget justification to Congress. The agency noted that it plans to correct this error in its fiscal year 2021 budget justification. In addition, Education stated that it would explore a different model for calculating the persistence rate. While we support Education’s plans to correct the error it acknowledged and explore another model for calculating the persistence rate, Education’s persistence rate calculation has additional errors that the agency needs to correct to accurately calculate the CCAMPIS program’s persistence rate. For example, as we stated in the report, Education’s calculations did not include students who transferred, which the agency has reported should be included in its persistence rate measure. Moreover, we identified other technical errors in the numerator and denominator of Education’s formulas. For example, when calculating the persistence rate for CCAMPIS participants, Education counted students who declined to participate in the CCAMPIS program. We continue to believe that it is important for Education to report reliable program information to oversee and monitor the program and to provide accurate information to Congress. To do this, Education needs to take additional action to address all of the errors in its persistence calculations. Education disagreed with the recommendation to collect the CCAMPIS participant enrollment data needed to calculate a standard 3-year graduation rate or accurately define and calculate a different college completion measure (recommendation 2). Education stated that it could address our concerns with a modification to the description of the measure published in the agency’s budget justification. Specifically, Education said it plans to clarify that, for graduation rate data published for fiscal year 2020 and prior years, the term “within 3 years of enrollment” means within 3 years of enrolling in the CCAMPIS program. However, as we stated in the report, Education’s calculations do not align with this measure either. As for future years, the agency stated that it will explore transitioning to a new model of tracking CCAMPIS students over time, which, as described, would be consistent with Education’s standard graduation rate. However, Education noted that it must carefully balance the need to collect more informative and reliable data from grantees with the need to avoid adding unnecessary reporting burdens. We recognize that collecting the enrollment data needed to calculate the standard graduation rate could place a burden on grantee schools. Our recommendation therefore gives Education the option to define a different college completion measure and calculate it correctly. We continue to believe that Education needs to take steps to either collect the necessary enrollment data to calculate a standard 3-year graduation rate or correctly calculate a modified college completion measure. Education disagreed with the recommendation to encourage schools— through appropriate means, such as the FSA Handbook—to inform students via school websites about the availability of the dependent care allowance and how to request the allowance (recommendation 3). Education stated that it believes it would be inappropriate to indiscriminately encourage all schools to encourage student parents to borrow additional loans without considering a student’s individual financial circumstances. We did not suggest that schools should encourage all student parents to borrow additional loans to pay for child care. Instead, we recommended that Education encourage schools to make students aware of this potential option—which federal law makes available to students—to allow them to make informed financial decisions based on their personal circumstances. We made this recommendation because we found that schools were not consistently sharing information with students about the dependent care allowance or how to request one. We further recognized in the report that not all students may want to increase their student loans to finance their child care costs while in school; however, access to additional federal student loans could be a useful option for those students who may need it, so we believe students should be aware of this potential option. Education also stated that it would be inappropriate for the agency to require schools to take actions that could erode their student loan repayment and default rates. We did not recommend that Education require schools to take any action; rather, we recommended that Education encourage schools to inform students about a potentially available federal resource. In addition, Education did not provide any evidence that being aware of or using the dependent care allowance would negatively affect student loan repayment or default rates. Further, access to additional financial resources can help some students succeed in school if it allows them to work less and study more. For example, as cited in the report, recent research suggests that additional federal financial aid, including student loans, can lead to improved academic outcomes for some students. Education also expressed concerns about students borrowing more and noted there are numerous federal, state, local, and private options that offer low-income students affordable or no-cost child care. Education noted that the federal Child Care and Development Fund (CCDF) provides significant resources for student parents. However, as we noted in the report, some states either fund very few families pursuing education or training or have implemented policies that restrict access to CCDF subsidies for college students. Education also noted that many colleges, as well as countless faith-based organizations offer affordable or no-cost child care to low-income students. However, we found that nearly half of student parents reported paying for child care, with costs averaging about $490 per month. Moreover, even colleges that received a CCAMPIS grant had significant waiting lists for assistance and reported more children on waiting lists for CCAMPIS assistance than children receiving subsidized care from the CCAMPIS program. Finally, Education noted that the Federal Student Aid Handbook already contains information about the dependent care allowance and its inclusion in students’ financial aid calculations. While the handbook does include information to help school financial aid administrators implement a dependent care allowance appropriately, it is not a resource directed at student parents. For this reason, we recommended that Education encourage schools to take steps to inform students about the dependent care allowance and how to request one. We continue to believe that it is important for Education to encourage schools to inform student parents about the availability of the dependent care allowance and how to request it. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services, the Secretary of Education, appropriate congressional committees, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0534 or emreyarrasm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines 1) What is known about the characteristics and degree completion of undergraduate students with dependent children? 2) What is known about the Child Care Access Means Parents in School grant program and how reliable is Education’s reported outcome information? 3) What is known about student parent access to other key federal programs that help low-income families pay for child care? 4) To what extent do selected schools that serve student parents publicize information about the option to increase federal student aid to help pay for child care? This appendix provides details of the data sources used to answer these questions, the analysis we conducted, and any limitations to our analysis. Student Parent Characteristics and Degree Completion To examine the characteristics and degree completion of undergraduate student parents, we analyzed data from the Department of Education’s (Education) National Postsecondary Student Aid Study (NPSAS) from the 2015-2016 school year, the most recent year available. NPSAS data contain nationally representative, detailed demographic and financial aid data for college students enrolled in postsecondary education programs. These data come from institutional records, government databases, and interviews with students. We also analyzed Beginning Postsecondary Students Longitudinal Study (BPS) data from 2008-2009. BPS tracks students over a 6-year period and collects both survey and transcript data. The most recently completed BPS cohort first enrolled in postsecondary education in the 2003-2004 school year. We assessed the reliability of the NPSAS and BPS data by reviewing existing information about the data and the system that produced them. We also interviewed agency officials knowledgeable about the data. We determined these data to be reliable for our purposes. Because the NPSAS and BPS data are based on probability samples, estimates are calculated using the appropriate sample weights provided, which reflect the sample design. Unless otherwise noted, all percentage estimates from the NPSAS data analysis have 95 percent confidence intervals within plus or minus 3.8 percentage points of the percent estimate, and all number estimates from the NPSAS data analysis have 95 percent confidence intervals within plus or minus 9 percent of the estimate. Similarly, all percentage estimates from the BPS data analysis have 95 percent confidence intervals within plus or minus 3.7 percentage points of the percent estimate. We compared 95 percent confidence intervals for both NPSAS and BPS data to identify statistically significant differences between specific estimates and the comparison groups. The information collected from the interview portions of the NPSAS and BPS studies, such as the variables measuring whether students have children in paid child care and a student’s monthly child care costs, is self-reported and is not based entirely on federal determinations or cross- verified with outside sources. Students’ monthly child care costs may be prone to more error than simpler yes/no questions. Child Care Access Means Parents in School Grant Program To determine what is known about the Child Care Access Means Parents in School (CCAMPIS) grant program, we reviewed relevant program information and federal laws and regulations, and interviewed Education officials knowledgeable of the program. To provide illustrative examples of how selected colleges and universities use CCAMPIS grant funding to help students pay for child care, we interviewed officials from two schools. We selected these two schools based on expert and agency recommendations and research. We also considered level of degree program (2-year and 4-year) and geographic diversity. We also conducted descriptive analysis of the performance data that CCAMPIS grantees reported to Education for the 2016-2017 school year, the most recently available performance data at the time of our review. Education provided us with performance information from the 85 colleges and universities that received their first year of grant funding in fiscal years 2013 and 2014. At the time of our review, Education had not yet collected performance data for the 2017-2018 school year, which would be the first project year for the 62 schools that were awarded CCAMPIS grants in fiscal year 2017. Education collects annual performance information from CCAMPIS grantees using annual performance reports. Grantees report both summary information for all participating students as well as detailed information—listed separately—for each participating student. Education officials said that they do not use the summarized participant data for performance calculations because of inconsistencies they identified in grantees’ reported data. Instead, Education uses the detailed information grantees provide for each student. This student-level data includes student demographic information, the number of children served, CCAMPIS child care subsidies received, and child care fees paid. Grantees also report each student’s CCAMPIS participation and academic enrollment during four academic terms (fall, winter, spring, and summer). To assess the reliability of the CCAMPIS performance data, we reviewed related program documentation, interviewed knowledgeable agency officials, and conducted electronic data testing for missing data, outliers, and logical errors. When we reviewed the student-level data, we identified instances of incomplete and inconsistent data that affected which students could be identified as participating in the CCAMPIS program. To address these concerns, we excluded from our analysis students that grantees reported as having 1) declined to participate in CCAMPIS for each of the four academic terms, 2) no enrollment information for any of the four academic terms, and 3) an enrollment code not included in Education’s report instructions. We discussed our methodology for identifying program participants with Education officials, who agreed with our approach. We also omitted from our analysis any student for whom grantees reported duplicate information. After these corrections, we determined that CCAMPIS student-level performance data were sufficiently reliable for the purpose of describing participant characteristics. We determined that selected summary variables reported elsewhere in grantees’ performance reports were similarly reliable for the purpose of describing child care services funded and number of children on waiting lists. We also examined Education’s calculations underlying the CCAMPIS program’s performance measures that the agency reported in its fiscal year 2020 budget justification to Congress and assessed them against federal internal control standards related to data quality. Because of the flaws we identified in Education’s calculations, we developed our own calculations of Education’s performance measures using the 2016-2017 CCAMPIS program performance data. Student Parents Access to Other Key Federal Child Care Programs To examine student parents’ access to other key federal programs that assist low-income families with child care costs, we focused on the Child Care and Development Fund (CCDF), Temporary Assistance for Needy Families (TANF), and Head Start and reviewed relevant federal laws and regulations, agency guidance, and program documents. To describe the extent to which states have established CCDF program policies that limit postsecondary students’ access to child care subsidies, we summarized information published in the CCDF Policies Database Book of Tables: Key Cross-State Variations in CCDF Policies as of October 1, 2017. To provide illustrative examples of how selected schools use these programs to help college students pay for child care, we interviewed school officials from three colleges and universities that also received CCAMPIS grants. We selected schools based on expert and agency recommendations and research. We also considered level of degree program (2-year and 4- year) and geographic diversity. To assess the extent to which selected schools are publicizing information about the option to increase federal student aid to help pay for child care, we reviewed the websites of the 62 schools that first received a CCAMPIS grant in fiscal year 2017. These were the most recently awarded CCAMPIS grants at the time of our review. In order to review comparable information across all schools, we developed a standardized data collection instrument that we used to examine the availability of information on the option to include a dependent care allowance. We developed the instrument after reviewing the websites of 22 schools and interviewing officials from four schools to learn more about their practices for informing students about the dependent care allowance. We selected these four schools because they did not include information about the dependent care allowance on their websites, students attending these schools borrowed federal student loans, and at least one-third of enrolled students were age 25 or older. We conducted our review from October through December 2018. One analyst recorded information in the data collection instrument and another analyst checked and verified it. We collected complete information for all 62 schools and analyzed the information across schools. We did not, as a part of our review of school websites, assess the schools for compliance with any laws or regulations. Instead, this review was intended to understand what information is made available to students on school websites. To better understand these 62 schools and their practices, we examined additional federal data and interviewed financial aid officials from 13 of the 62 schools to obtain additional information about school practices for incorporating the dependent care allowance into students’ financial aid calculations. The results from our website reviews and school interviews are not generalizable. We selected these schools to achieve a mix of schools that did and did not publicize the availability of the dependent care allowance on their websites, as well as degree levels (2-year and 4- year), and geographic diversity. We also considered the cost of attendance for the average low-income student, after grant or scholarship aid. We also analyzed the characteristics of all 62 schools using 2016-2017 data, the latest available, from Education’s Integrated Postsecondary Education Data System (IPEDS), and examined the characteristics in the context of our website analysis. We also interviewed federal officials from Education about the information the agency provides to schools about the dependent care allowance. We assessed the reliability of the IPEDS data by reviewing existing information about the data and the system that produced them, and determined they were reliable for our purposes. We assessed Education’s practices against federal internal control standards for communicating with external parties. We conducted this performance audit from April 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the individual named above, Michelle St. Pierre (Assistant Director), Karissa Robie (Analyst-in-Charge), Jennifer Cook, and Marissa Jones Friedman made key contributions to this report. Also contributing to this report were, Deborah Bland, Kevin Daly, Nisha Hazra, Gina Hoover, Michael Kniss, Sheila R. McCoy, Jean McSween, Brittni Milam, John Mingus, Jessica Orr, Joshua Paul, Benjamin Sinoff, and Adam Wendel.", "summary": "Student parents face many challenges, including paying for child care, that can make it difficult for them to complete a degree. The federal government supports student parents through Education's CCAMPIS program, which provides colleges funding for child care services, and federal student aid, which can also help students pay for child care. GAO was asked to provide information on student parents and the federal programs that support these students. This report examines, among other objectives, what is known about the characteristics and degree completion of undergraduate students with children; what is known about the CCAMPIS program and how reliable Education's reported outcomes are; and to what extent selected schools publicize the option to increase federal student aid to help pay for child care. GAO analyzed 2009 and 2016 federal student data (the most recent available) and CCAMPIS program performance data, reviewed how the 62 schools that were awarded CCAMPIS grants in 2017 publicized the student aid option to help pay for child care, and reviewed relevant federal laws and regulations and agency documents. GAO interviewed officials from Education and selected schools. More than one in five undergraduate students were raising children, and about half of student parents left school without a degree, according to Department of Education (Education) data. In 2015-2016, an estimated 22 percent of undergraduates (4.3 million of 19.5 million) were parents. An estimated 55 percent of student parents were single parents, 44 percent were working full-time while enrolled, and 64 percent attended school part-time. Undergraduate student parents had fewer financial resources to fund their education than students without children. Nearly half of student parents reported paying for child care, with monthly costs averaging about $490. A higher percentage of student parents left school without a degree (52 percent) compared to students without children (32 percent) as of 2009 (the most recent data available). Education's Child Care Access Means Parents in School (CCAMPIS) program helped about 3,300 students pay child care costs for about 4,000 children in 2016-2017. Another 4,200 children were on waiting lists to receive assistance. Most CCAMPIS participants paid some child care fees after receiving subsidies—the median payment each month was about $160. Education measures participants' persistence in school and graduation rate to assess the performance of the CCAMPIS program. However, flaws in its calculations of these two measures prevented Education from reporting reliable results, making it difficult for Education and Congress to evaluate the program's effectiveness. Some student parents could be eligible to increase their federal student loans to help pay for child care by asking their schools to include an allowance for dependent care expenses in their financial aid calculations. However, schools do not always publicize this allowance to current and prospective students. GAO reviewed the websites—where schools post other college cost information—of schools serving student parents and found that about two-thirds of these websites did not mention the allowance. Schools are not required—and Education does not encourage them—to inform student parents about the allowance. As a result, eligible student parents may be unaware of this option to request additional financial support to help them complete their degree. GAO is making three recommendations to Education to correct its CCAMPIS persistence and graduation rate calculations and to encourage schools to inform students about the option to increase federal student aid to help pay for child care. Education disagreed with GAO's recommendations, but described plans to improve its performance calculations. GAO continues to believe additional actions are warranted. (617) 788-0534 or emreyarrasm@gao.gov .", "document_type": "gao"}
{"report": "Disaster response can involve many federal, state, territorial, tribal, private sector, and voluntary organizations. The National Response Framework describes how the federal government, states and localities, and other public and private sector institutions should respond to disasters and emergencies. For example, state, local, tribal, and territorial governments are to play the lead roles in disaster response and recovery. Local emergency agencies—police, firefighters, and medical teams—are to be the first responders. In serving individuals who have disabilities and others who have access or functional needs, disaster responders at all levels are responsible for ensuring compliance with any applicable requirements for equal opportunity and non-discrimination. Federal agencies become involved in responding to a disaster when effective response and recovery are beyond the capabilities of the state and local governments. The Robert T. Stafford Disaster Relief and Emergency Assistance Act (Stafford Act) authorizes federal funding and support to assist states and localities in responding to a disaster. This federal support is available under the Stafford Act when the President declares a major disaster or emergency in response to a request by the governor or by the chief executive of a tribal government. Under the National Response Framework, DHS is the federal agency with primary responsibility for coordinating disaster response, and within DHS, FEMA has lead responsibility. In addition to DHS, at least 29 other federal agencies carry out disaster assistance programs and activities. The National Response Framework identifies 15 emergency support functions (ESFs)—such as communication, transportation, and energy— and designates a federal department or agency as the coordinating agency for each function. Under the National Response Framework, FEMA is designated as the coordinating agency for ESF-6, which includes mass care, emergency assistance, temporary housing, and human services. The National Response Framework also designates primary and support agencies for each ESF. Both FEMA and the Red Cross are the primary agencies for ESF-6. As co-primary agencies, FEMA and the Red Cross are responsible for working closely to coordinate mass care and related services across sectors, including identifying resource needs, organizations with mass care capacity to address those needs, and establishing strategies to address resource gaps (see fig. 1). According to ESF-6, Red Cross also provides technical assistance to FEMA and serves as its principal mass care subject matter expert. The Red Cross works with FEMA to provide such assistance to state and local partners, according to FEMA. In addition, the Red Cross and FEMA facilitate the mobilization of resources and coordination within the whole community for the provision of mass care services. The Red Cross role in ESF-6 has shifted over time. At the time of Hurricane Katrina, Red Cross was a primary agency, but in the 2008 update to ESF-6 it became a support agency. However, in a 2013 update, Red Cross was shifted back to the primary agency role and given new responsibilities such as working with FEMA to identify available mass care capacity, anticipate mass care requirements, and establish strategies to address gaps in coordination. These responsibilities, among others, remain in effect under the current ESF structure. FEMA and Red Cross coordinate mass care with the support of other federal agencies such as USDA, the Department of Health and Human Services, and the Department of Defense (DOD), as well as voluntary organizations and partners at the state and local levels. There are also over a dozen federal agencies named as having supporting roles in ESF- 6 (see app. I for a list of ESF-6 support agencies). For example, DOD and its Army Corps of Engineers provides construction and engineering support for temporary housing and sheltering, including inspecting shelter facilities to ensure accessibility and suitability. In addition, ESF-6 names over 50 members of the National Voluntary Organizations Active in Disaster (NVOAD) that provide a wide range of services in support of mass care and other ESF-6 activities, including the Salvation Army, Southern Baptist Convention Disaster Relief, and Feeding America. State and local governments are vital to mass care provision and assessing their own communities’ response capabilities. According to ESF-6, local government agencies coordinate with voluntary organizations and the private sector to coordinate activities that meet immediate needs of disaster survivors. When those needs exceed local resources, the state may provide additional support. When these resources are insufficient, federal assistance may be requested through the FEMA regional office. We found in 2018 that FEMA faced a number of challenges that slowed and complicated its response efforts to the 2017 hurricanes, especially Hurricane Maria in Puerto Rico. The sequential and overlapping timing of the three hurricanes strained staffing resources and created logistical challenges in deploying additional assistance (see fig. 2). In particular, FEMA had already deployed staff and resources to support the response efforts for Hurricane Harvey in Texas when the other major hurricanes made landfall shortly thereafter. Moreover, FEMA’s response efforts in Puerto Rico and the U.S. Virgin Islands were complicated by a number of factors, including their distance from the continental United States and limited local preparedness for a major hurricane. We have previously reported that there is increasing reliance on the federal government for disaster assistance as the number of natural disasters increases and that costs will likely continue to rise as the climate changes. FEMA identified key findings related to mass care in its After-Action Report for the 2017 hurricanes, noting differences in shelter populations across the states, as well as the duration of shelter stays (see fig. 3). FEMA also reported facing challenges transitioning survivors out of group shelters in a timely fashion. In order to qualify for federal emergency preparedness funding, states and eligible urban areas (grantees) are required to regularly submit information to FEMA on their ability to respond to a disaster. Specifically, grantees first identify their own capability targets—such as for sheltering disaster victims—through the Threat and Hazard Identification and Risk Assessment, and then assess their progress toward these targets annually in the Stakeholder Preparedness Review (capabilities assessments). In fiscal year 2018, FEMA awarded $402 million to states and territories through the State Homeland Security Program, and $580 million to urban areas through the Urban Area Security Initiative, both of which require grantee capability assessments. FEMA provides guidance and technical assistance to state and local partners in their self-assessment efforts. According to officials, FEMA does not conduct its own evaluations of state, local, and voluntary organizations’ capabilities. FEMA and Red Cross established joint operation centers where they co- located with key partners such as the Salvation Army and NVOAD for each of the 2017 hurricanes, which facilitated coordination of shelter, feeding, and supply distribution. In addition to co-locating at FEMA’s National Response Coordination Center in Washington, D.C., FEMA, the Red Cross, and key mass care partners also co-located in state and local emergency operations centers (see fig. 4). Our prior work has found co-location of staff enhances interagency collaboration. Co-location contributed to relationship-building that facilitated communication and coordination of mass care services, according to FEMA, Red Cross, and emergency management officials in all four states we visited. See figure 5 for examples of how various agencies and sectors prepared food and supplies for mass care operations. Co-location meant workers could communicate face-to-face, as key partners needed to collaborate and communicate resource requests to FEMA and other agencies. In the U.S. Virgin Islands, DOD provided airplanes that enabled workers to fly between the islands to attend face- to-face meetings, according to FEMA regional officials. According to officials in two states we visited, this type of face-to-face communication facilitated building relationships. Moreover, officials in one state told us that co-location enabled them to communicate survivor needs directly to FEMA, which could then provide assistance. This was especially critical when power and cell phone service were out, particularly in Puerto Rico and the U.S. Virgin Islands, which experienced prolonged power outages and disabled electronic communications. Officials from federal agencies and the Red Cross described some additional benefits of co-location: USDA Food and Nutrition Service (FNS) officials said co-located ESF- 11 (Agriculture and Natural Resources) staff in the National Response Coordination Center provided food inventories to staff at the ESF-6 desk. Red Cross officials said they were able to quickly obtain supply trucks after Hurricane Harvey in Texas because the Red Cross had representatives at FEMA’s National Response Coordination Center. As we previously reported, DOD provided high-water vehicles, amphibious vehicles, and boats to transport supplies for the Red Cross and support FEMA logistics efforts. Officials in one state noted that in-person communication was especially useful for coordinating mass care when FEMA’s on-line system for submitting resource requests could not be used (see text box). Web Emergency Operations Center Resource requests can be communicated through the Federal Emergency Management Agency’s (FEMA) Web Emergency Operations Center (WebEOC), an electronic system that processes and tracks resource requests from state or local governments. WebEOC supports emergency management processes and functions by providing a real-time operating picture for FEMA headquarters, regions, and federal, state, local, and tribal strategic partners. In 2015, the Department of Homeland Security’s Office of Inspector General (OIG) found that WebEOC was not sufficiently integrated with key agency systems and could cause delays in providing disaster assistance. In 2017, WebEOC was used in two of our four selected states, and a predecessor system to WebEOC was used by one of FEMA’s regional offices, according to officials in these areas. WebEOC was useful in tracking resource requests, but in-person communication was more helpful for coordinating mass care, according to FEMA regional officials. In cases where staff could not access WebEOC, requests to FEMA were presented on paper, according to state officials. In 2018, FEMA reported that it had provided every state with FEMA WebEOC accounts so state users could submit resource requests directly to FEMA. WebEOC also allows FEMA to share aggregated data, such as shelter counts and feeding information, according to FEMA officials. Federal officials and partners in Texas, Florida, Puerto Rico, and the U.S. Virgin Islands described many challenges they encountered in coordinating mass care. While the concurrence and intensity of the 2017 hurricanes presented many unforeseen challenges, several state and local governments and voluntary organizations told us about issues related to mass care coordination and planning. As a result, some supply distribution, sheltering, and feeding needs went unmet. Miscommunication: Miscommunication among disaster workers affected supply distribution. For example, FNS officials reported challenges with delivering baby formula for about 28,000 infants in Puerto Rico through FEMA. One shipment of baby formula was lost and discovered frozen and unusable in Puerto Rico because FEMA officials were not aware that the products had been delivered, according to FNS’ 2018 After-Action Report. The report also stated that some perishable infant formula and food remained at a port in Florida several weeks after delivery. FEMA officials told us they shipped nearly 400 containers of infant formula and food in the first 3 months after Hurricane Maria, but that competition for port clearances made it challenging to coordinate, prioritize, and track supplies. As a result, some who needed these supplies may not have received them. FEMA officials also noted that they believe survivor needs were met by a combination of disaster relief supplies and the restoration of capacity at grocery stores. According to FNS’2018 After- Action Report, their officials met with FEMA and completed training on FEMA’s logistics system in 2018 to be able to better track future shipments of these products. Insufficient shelter staff: In Texas and Florida, emergency managers we spoke with described having unprecedented numbers of residents needing shelters but not enough staff initially to operate them. To address this gap, they said they relied on members of the state National Guard or local government and community organizations to staff shelters, but in some instances, shelters continued to have insufficient numbers of workers. To improve shelter staffing for future disasters, emergency managers in Florida told us they are working on training additional county employees to serve as shelter staff. Serving individuals who have disabilities: Public shelters faced challenges in some cases serving individuals who have disabilities, as we previously reported. For example, we reported in 2019 that some individuals who have disabilities faced challenges accessing services from local shelters, including restrooms. In another example, the lack of a quiet space in public shelters for individuals with autism negatively impacted their mental health, according to officials from an advocacy group. Extensive damage to hurricane shelters: In Texas, Puerto Rico, and the U.S. Virgin Islands, Hurricanes Harvey, Maria, and Irma damaged many buildings planned for use as hurricane shelters, according to emergency management and local government officials in these areas. As a result, some remaining shelters were at maximum capacity. In some cases, survivors and staff had to relocate to alternate sites during the hurricanes. For example, an arena in Humacao, Puerto Rico, and a Department of Human Services building in the U.S. Virgin Islands served as shelters when intended shelter buildings were destroyed by Hurricanes Maria and Irma, respectively (see fig. 6). Damaged roads and communications infrastructure: Damaged and flooded roads and the affected terrain in all four states contributed to challenges in distributing supplies, especially in Puerto Rico and the U.S. Virgin Islands. In Puerto Rico, FEMA received complaints from municipalities that food was not reaching neighborhoods in need. Impassable roads and no ability to communicate challenged FEMA’s plans, which had designated certain partners to distribute meals. Several weeks after Hurricane Maria hit, FEMA redesigned its distribution strategy, which included identifying the most vulnerable municipalities and having liaisons from the Puerto Rico Emergency Management Agency and the municipalities help coordinate the distribution. This enabled food to reach neighborhoods in need. Insufficient supplies: According to Puerto Rico Department of Education officials, FEMA was initially reluctant to provide water to schools serving as shelters because the schools were supposed to have their own water supply from the Puerto Rico Department of Education’s warehouses. However, Puerto Rico Department of Education officials said they only had enough water for shelter residents for 30 days. The agency requested help to meet additional needs, but FEMA did not have enough water or food boxes to help supplement the schools’ supply. There were several thousand people sheltered in the schools, but according to these officials, the Puerto Rico Department of Education was responsible for providing food and water to survivors whether or not they were shelter residents. Once the Puerto Rico Department of Education officials met with FEMA and demonstrated their need for water, they were able to secure supplies from FEMA. Early relocation of survivors to hotels: In Texas, the early relocation of survivors from shelters to eligible hotels under FEMA’s Transitional Sheltering Assistance program challenged mass feeding operations, according to two Texas emergency management officials and representatives of two voluntary organizations. As a result, some survivors did not receive food assistance, as described below. FEMA’s Transitional Sheltering Assistance program and the Department of Agriculture’s Disaster Supplemental Nutrition Assistance Program (D- SNAP), while not considered to be a central part of mass care under the National Response Framework, provide assistance to survivors after disasters and provide services that may intersect with mass care activities. According to officials in Texas and Florida, some aspects of how these programs were implemented contributed to unmet needs. Transitional Sheltering Assistance program: After the initial response effort ends and mass shelters close, FEMA’s Transitional Sheltering Assistance program is intended to provide short-term sheltering assistance to survivors who are still unable to return home. States request FEMA approval for Transitional Sheltering Assistance when they determine there is a need for short-term assistance. According to officials in Texas, the Transitional Sheltering Assistance program was activated earlier than they expected before mass shelters closed, resulting in survivors leaving early to stay in program-eligible hotels. According to these officials, the early activation resulted in the inability to track where survivors were located and where survivors needed assistance. According to an official at a voluntary organization, survivors in program-eligible hotels were going without food and some were eating coffee grounds in their hotel rooms because they had no food and no money to purchase food. Officials from state agencies and voluntary organizations that could provide assistance told us they could not get information from the hotels about how many survivors were guests at specific hotels, due to the hotels’ reluctance to provide guests’ information. When voluntary organizations tried to set up feeding operations at hotels, some hotels did not want the organizations to set up feeding operations on hotel premises, according to organizational representatives. One state official also said some hotels did not allow food distribution because of concerns about food sitting in rooms or the hotels’ preference that their guests use their restaurant facilities. D-SNAP: D-SNAP provides temporary food assistance for households affected by a natural disaster. D-SNAP usually begins after grocery stores have re-opened and families are able to purchase and prepare food at home. USDA’s FNS offers guidance to states that choose to operate a D-SNAP program on where and how to operate D-SNAP registration sites, including guidance on serving individuals who have disabilities and the elderly. For example, FNS guidance states that D- SNAP registration sites should offer extra cooling measures in a special waiting area for individuals who have disabilities and the elderly, and move these individuals to the front of regular registration lines. FNS’ After-Action Report identified, and state and county officials in Texas and Florida said they observed, D-SNAP registration sites that did not appropriately serve elderly individuals or those who have disabilities, such that some elderly survivors fainted while waiting in the heat. In one state we visited, officials from a local voluntary organization said the state government did not work with community-based groups to identify local D-SNAP registration sites. As a result, D-SNAP registration sites did not align with where survivors needed assistance, and according to these officials approximately 50,000 applicants came to one site and were turned away after waiting for hours in the heat. To help address these challenges, some elderly individuals and individuals with disabilities in Florida were allowed to register for D-SNAP over the phone in December 2017 and in May 2018, according to a state official. While the National Response Framework indicates that many agencies participating in disaster response formalize their responsibilities in written agreements, we found that key mass care partners did not have such agreements or that they did not clearly outline responsibilities at the time of the 2017 hurricanes. Although Red Cross has written agreements with some state and local partners, counties we visited in Florida, Texas, and the U.S. Virgin Islands—states where Red Cross shelters disaster victims—did not have written agreements that clearly specified what mass care services would be provided by the Red Cross. In Florida, several counties we visited did not have formal agreements with the Red Cross during the 2017 hurricane season. In lieu of a formal agreement, one of the counties had an email from the Red Cross stating that the Red Cross could support one of 15 shelters, according to officials and documents we reviewed. In some cases, even when written agreements were established, there were still unclear roles and expectations. For example, another Florida county did have an agreement in place, but county officials said they found out after the 2017 hurricane season started and shortly before Hurricane Irma that the Red Cross could support only eight shelters—a substantial decrease from previous years. Further, when counties did have written agreements with the Red Cross, the agreements did not always clearly define responsibilities. The agreements also did not specify how and at what point sheltering and feeding needs and capabilities should be communicated by the Red Cross to counties, which exacerbated challenges in providing these services after the hurricanes. After the 2017 hurricane season, officials in three states we visited said they have been working toward clarifying responsibilities in written agreements. Red Cross officials also said they have been developing letters of intent with local government partners since 2017, which describe what services can be provided by the Red Cross in these localities. However, our review of some of these new finalized agreements found they lack consistency and detail in what each of the parties can deliver regarding sheltering, feeding, and supply distribution. For example, Red Cross’ agreement with one Florida county specifies it can operate two shelters for about 1,000 residents, while its agreement with another county states it will “support shelters as resources allow.” Red Cross officials said written agreements may be difficult to change as needs and capabilities change over the course of the response to a disaster. Outside of written agreements, Red Cross officials said they collaborate with government agencies in other ways, such as participating in mass care exercises to create a shared understanding of mass care roles and work on jointly-developed response plans. Red Cross officials also told us that they need to be clearer with local jurisdictions about what they can and cannot provide, and that they need to reach mutual understanding with local governments about shared planning assumptions, such as the peak shelter population and what the Red Cross could provide within specified timeframes. According to Red Cross officials, neither they nor local governments established clear expectations in the past. In August 2017, the Red Cross launched a nationwide readiness initiative focusing on mass care planning discussions with local governments. This initiative also includes clarifying planning assumptions with local governments on a recurring basis. FEMA provides some guidance to states and localities about how to effectively coordinate with mass care partners, as well as a training course that encourages establishing written agreements. FEMA’s training materials for the mass care planning and operations course describe the differences in types of agreements that states and localities might establish with mass care partners, and specifically suggest defining the roles and responsibilities of each party. In addition, FEMA has helped developed tools for stakeholders to use when specifically coordinating mass care operations, such as the Multi-Agency Feeding Support Plan Template. This tool guides states, voluntary organizations, and other partners to clearly establish roles and responsibilities related to specific aspects of feeding, including the delivery of supplies and networking with other organizations to identify unmet needs. FEMA officials noted that all of their mass care templates encourage this type of planning for roles and responsibilities. However, FEMA guidance and training materials do not suggest detailing the specific responsibilities of each entity for mass care services in the written agreements. For example, the guidance does not explicitly prompt states and localities to use their written agreements to specifically establish how much shelter and feeding assistance an agency, government, or organization can provide. Our prior work has found that clarifying responsibilities through written agreements is critical to effective interagency collaboration. When an agency, government, or organization does not specifically indicate how much shelter and feeding assistance it can provide in a disaster, its partners may have unfounded expectations. For example, in Texas, officials in one city said when one large mass shelter first opened, there were only a small number of Red Cross volunteers, which was insufficient to operate and manage a shelter with tens of thousands of survivors; this was short of city officials’ understanding that Red Cross would fully staff the location from the beginning. Without further guidance from FEMA on how to establish effective written agreements, unmet expectations between state and local partners and voluntary organizations may persist and place disaster survivors at risk. Our prior work has also found that federal agencies engaged in collaborative efforts need to create the means to evaluate their activities in order to identify areas for improvement. In addition, federal internal control standards state that management should establish an organizational structure, assign responsibility, and delegate authority to key roles in order to achieve objectives. Moreover, the organizational structure should be evaluated periodically in order to meet the objectives and adapt to new situations. FEMA is responsible for coordinating and supporting the federal response to major disasters and relies significantly on the Red Cross as its co-primary agency under ESF-6. While FEMA and the Red Cross conduct after-action reviews following certain major disasters, including for the 2017 hurricane season, these reviews are focused on response and recovery efforts and do not include a broader review of roles and responsibilities of the co-primary agencies. Based on its findings on the 2017 hurricane season, FEMA called for some revisions to the National Response Framework and ESF annexes related to coordination across sectors. Accordingly, FEMA is currently revising the framework, which is considered a living document to be regularly reviewed to reflect experience gained from its use. However, FEMA has not proposed revisions to ESF-6 as part of its current review of the National Response Framework and ESF annexes. Specifically, FEMA has not reviewed whether the current structure of ESF-6 leadership roles and responsibilities is best suited for coordinating mass care, or whether there are responsibilities that should be shifted. ESF-6 is unique among ESFs in that it has a voluntary organization serving as a co-primary agency. Further, the Red Cross’ role under ESF-6 has changed multiple times since Hurricane Katrina. According to FEMA officials, FEMA is not required to review ESF-6 leadership roles and responsibilities, and instead focuses on the overall improvement of mass care delivery, including mass care activities and services. However, FEMA’s ESF Leadership Group noted that it was not always clear which agency that is part of an ESF is best suited to carry out a task. Evaluating collaborative efforts can help key decision makers within the agencies obtain feedback for improving both policy and operational effectiveness. Moreover, the National Response Framework is considered a living document, and DHS plans regular reviews to evaluate consistency with existing and new policies, evolving conditions, and the experience gained from its use. As we have previously reported, in disasters in which the federal government is involved, the extent and effectiveness of the Red Cross’s activities could have a direct impact on the nature and scope of the federal government’s activities. Given the challenges experienced with mass care during the response to the 2017 hurricanes, FEMA is missing an opportunity to identify areas for improvement and strengthen interagency coordination by not reviewing ESF-6 leadership roles and responsibilities. Many FEMA, Red Cross, local government officials, and representatives from local voluntary organizations we interviewed emphasized the importance of pre-existing relationships among established partners in coordinating mass care during the 2017 hurricanes. Relationships between these established mass care partners were often formed during non-disaster periods through regular conference calls and mass care training exercises. For example, officials in all four state emergency management departments we visited described positive relationships developed with FEMA staff through regular joint training exercises. FEMA’s Voluntary Agency Liaisons (VALs) help facilitate relationships between FEMA and established mass care partners. For example, VALs serve as contacts for non-governmental organizations active in disasters on a routine basis and during disaster response. In one FEMA regional office, officials said VALs serve as mass care specialists and regularly participate in calls with mass care partners. While such pre-existing relationships among established mass care partners facilitated mass care coordination, officials from voluntary organizations that did not have pre-existing relationships—unaffiliated organizations—reported challenges connecting with established mass care organizations, such as FEMA and the Red Cross, to share knowledge that could have informed response efforts. During the 2017 hurricane response, officials from unaffiliated organizations such as local advocacy groups and faith-based organizations told us they experienced challenges sharing critical information regarding needs, resources, and capabilities with established mass care organizations. These coordination challenges affected their ability to provide mass care services to certain populations. For example: A group of community organizations in Florida representing low- income and migrant populations had information on the location of people needing assistance, but reported difficulties in locating FEMA and Red Cross officials with whom to share that information. Representatives of a community group that assists victims of domestic violence in the U.S. Virgin Islands said there was no centralized way to share critical information and no plan for how to best address the issues facing these survivors. For example, they said the Red Cross had mapped damaged areas but was not sharing that information with community groups that could have provided assistance. This group said these maps could have been used to help locate people who were at particular risk. Red Cross officials stated that they experienced challenges in sharing damage assessment information in the U.S. Virgin Islands due to technology issues, which prevented them from being able to share these data securely with other organizations. Representatives from several faith-based organizations in multiple states told us they had food, water, and supplies, as well as local knowledge of need. Two of these representatives said FEMA and the Red Cross did not share information with them as to where they had already distributed supplies. This information was important so as to not duplicate efforts and to ensure those who still needed supplies were not overlooked, according to these representatives. Some migrant populations in all four areas we visited were hesitant to seek or receive assistance from federal, state, and local government agencies due to their undocumented immigration status, according to emergency management officials and community group representatives. Officials from multiple local voluntary organizations said they knew where migrant populations were located and what types of assistance they needed; they were trusted by these populations, but had difficulty finding FEMA or Red Cross representatives for sharing this information. Established mass care partners, including FEMA and the Red Cross, may not share information with unaffiliated organizations due to concerns about privacy, according to officials. Local governments also may not receive such information, because FEMA shares it with the states and the states are responsible for determining when to share it with local governments, according to FEMA officials. Local governments and unaffiliated organizations told us, however, that they do not need personally identifiable information, and that aggregated information about overall resource needs in certain locations would be sufficient for their purposes. For example, county officials in two states told us it was difficult to get FEMA data that would have helped them target areas for assistance, including those that other agencies might not have been able to reach. Similarly, the leader of a group that coordinates local voluntary organizations said they only needed aggregate-level data to identify needs in different counties. In addition, the Red Cross told us that mass care partners could access certain information from their RC View portal, which provides situational awareness information that supports resource requests and needs assessments. However, the Red Cross did not share such information with all its partners during the 2017 hurricanes because the technology was not yet ready. As of May 2019, Red Cross officials told us they are working on providing access to their RC View portal for several key partners, and that they intend to expand access to RC View to additional organizations in the future. ESF-6 states that Red Cross, in conjunction with FEMA, will facilitate the mobilization of private sector partners for the provision of mass care services. FEMA’s most recent strategic plan emphasizes the importance of a whole community approach to disaster response because individuals and local communities are the true first responders in a disaster. FEMA guidance states that the integration of non-traditional responders (which may include unaffiliated organizations) providing mass care services may be necessary during severe disasters. Federal internal control standards also emphasize the importance of communicating externally to key stakeholders. By not engaging in information sharing with unaffiliated organizations, FEMA and the Red Cross may miss opportunities to more accurately and efficiently coordinate mass care. As a result, those in need may not receive critical assistance in a timely way. Red Cross’ Training for Staff Deployed to Disaster Areas Red Cross provides training for its staff and volunteers deployed to disaster areas. This training includes information on the area of deployment, the nature of the disaster, and any cultural sensitivities they need to be aware of, according to Red Cross officials. However, unfamiliarity with local traditions and norms challenged Red Cross personnel when they arrived at disaster sites in 2017, and some local governments and community groups said this affected mass care coordination. Red Cross officials said they initially did not have enough Spanish speakers in Puerto Rico during the response to Hurricane Maria, for example. To address this need, they used Spanish-speaking workers from the International Red Cross community in Mexico and South America to assist with mass care coordination, according to Red Cross officials. As a result of challenges encountered during the 2017 hurricane season, Red Cross officials said that they have made changes to their approach intended to increase their engagement with the Latino community. This effort includes having materials translated into Spanish. To counter concerns among some disaster survivors about providing immigration status information, Red Cross officials said they have taken steps to clarify that the Red Cross does not collect this information. Information on the mass care capabilities of state and local jurisdictions that FEMA collected in 2016 and 2017 was not specific enough to aid the agency in its response to the 2017 hurricanes, according to FEMA’s After- Action Report and agency officials. The reporting process at the time of the 2017 hurricanes did not require grantees to report specific estimates of their current capabilities for providing mass care, which resulted in an incomplete picture of capabilities. With regard to mass care capabilities, FEMA did not ask grantees to report the number of people they could shelter, or how long they could maintain sheltering operations. For example, one state affected by the 2017 hurricanes identified gaps in the state’s capability to provide cots, blankets, laundry facilities, kitchens, and shelter facilities, but did not quantify the shortfall in its assessment submitted in December 2016. In addition, it was optional for grantees to describe deficiencies in their mass care capabilities at the time of the 2017 hurricanes, according to FEMA officials. One grantee affected by the 2017 hurricanes had indicated in its assessment from December 2016 that there were gaps in several mass care capabilities, such as shelter equipment and training for family reunification. However, this grantee chose not to include an additional description of what those gaps were. As a result of these limitations, FEMA and its grantees did not have specific information on state, territorial, and urban mass care capabilities or gaps at the time of the 2017 hurricanes. Officials from several states told us they were not aware of capabilities assessments being used during the response to the 2017 hurricanes, but some said this information could have been useful. For example, an official in one state said the information could be used for resource targeting. In submissions from the year following the 2017 hurricanes, 35 state and territorial grantees did not provide gap descriptions for mass care, which were optional at the time. According to FEMA, the agency recognized the limitations of the capabilities assessment data it had been collecting and began revising its methodology prior to 2017. FEMA’s After-Action Report for the 2017 hurricanes stated that one reason the agency began revising its capabilities assessment methodology was to provide more actionable information to use during response. Revisions were implemented for the 2018 reporting period that could result in FEMA collecting more specific and descriptive data on mass care capabilities, such as the number of people for whom the grantee can provide shelter, food, water, and relocation assistance as part of mass care (see table 1). FEMA’s 2018 guidance encouraged grantees to use a standardized format developed by FEMA, which allows grantees to insert community- specific numbers into a template when they report capability targets and estimates. The new standardized format also generates a quantitative statement of a grantee’s capability gaps (see table 2). Other new changes in FEMA’s revised approach will also allow the agency to collect more specific information on mass care capabilities. For example, starting in 2018, grantees were required to: Report the extent to which capabilities have been lost, built, or sustained over the previous year. Describe intended approaches for addressing capability gaps and sustaining capabilities built, including investments in resources. Describe the extent to which funding sources contributed to building or sustaining capabilities and improving disaster outcomes. Rate their level of confidence (1-5 scale) in the accuracy of their capability assessment for each target. These data elements have the potential to inform both disaster planning and response operations. FEMA revised its methodology for collecting capabilities assessment data in 2018, but it does not collect key information that could better inform its mass care planning. FEMA does not specifically require grantees to solicit the input of key partners in assessing mass care capabilities, according to officials, even though mass care generally depends on the work of such organizations. For example, the Salvation Army and the Southern Baptist Convention Disaster Relief often play key roles in mass care feeding, and the Red Cross manages sheltering in many locations, but they are not always included in mass care capabilities assessments submitted by grantees. FEMA officials told us that the new methodology should naturally foster engagement between grantees and their stakeholders, which should provide a better understanding of local capabilities for sheltering and feeding. According to these officials, under the new framework, FEMA requires grantees to report the number and type of government agencies and nongovernment organizations that participated in estimating capabilities (see fig. 7). However, by not requiring that grantees solicit input from organizations that provide mass care, or that grantees name specific organizations in their submissions, FEMA may rely on capabilities assessments developed without consultation with voluntary organizations providing key mass care services. We found that two of the six grantees included in our review did not report participating with the Red Cross, faith-based organizations, or other VOAD groups, in their 2018 assessments. An official from one of these jurisdictions confirmed that they had never reached out to voluntary organizations to take part in the assessment process, due to staff turnover and lack of time, despite relying on these organizations for providing mass care. An official from another jurisdiction said it is detrimental not to have voluntary partners’ input when preparing capabilities assessments because these partners are critical to providing mass care and play vital roles in disaster response. According to FEMA’s guidance, all organizations—not just government agencies—should be involved in preparedness efforts, and grantees should involve stakeholders throughout the process. FEMA’s guidance encourages a whole-community approach in which grantees include community stakeholders and subject-matter experts in estimating capabilities. Further, federal internal control standards emphasize the importance of designing systems for obtaining information that help an agency achieve its objectives. Without including key mass care providers when estimating capabilities and naming them in their capabilities assessments, grantees and FEMA may not collect reliable mass care capability estimates, or know who to contact in response to a disaster. States and localities may not be able to efficiently allocate their own resources to areas of unmet need and may be more reliant on outside resources during disaster response, which could have implications for the allocation of federal resources. FEMA reviews grantees’ capabilities assessments using standard checklists, but does not have a systematic process for providing feedback to grantees on their submissions in order to improve the usefulness of the information in them. FEMA officials use the checklists to assess the completeness and reasonableness of the submissions. Specifically, FEMA regional officials use the checklists to look for outliers, inconsistencies, invalid information, and inputs that to do not align with FEMA guidance or information that does not pass a “common sense” check. For example, one 2018 checklist we reviewed included comments from FEMA that the grantee’s capabilities assessment was only partially “complete and reasonable” because it showed no gaps for most capabilities, which might suggest that the targets it set are too low. FEMA officials told us that if the checklist identifies shortcomings in a grantee’s assessment, the regional office will send the assessment back to the grantee and communicate what needs to be changed. However, regional offices vary in their approaches to following up with grantees to obtain more information when potential issues are identified, and FEMA has not provided them with written guidance to standardize this feedback process. FEMA officials from two regional offices told us that the headquarters and regional preparedness divisions discussed follow-up protocols by phone, but they did not provide documentation that identified conditions or considerations for when to follow up with grantees or provide feedback. As a result, grantees may not receive consistent feedback from FEMA on their assessment of mass care capabilities and the information provided may remain incomplete. Rather than systematically providing feedback on the content of capabilities assessments, FEMA officials told us that they focus on identifying areas in which they can provide support to grantees. Their view is that communities know more about their own capabilities than the federal government does, and that it would not be appropriate to suggest major changes to the submitted assessments. Officials from one FEMA region said they view these submissions as self-assessments that are used for maintaining relationships with states and to help states better understand their capabilities and gaps. Officials also said that the FEMA regional office or the national preparedness office, or both, examine grantees’ disaster scenarios described in the assessments, the grantees’ self-assessed scores, and areas of grantee strengths and weaknesses to determine how FEMA can better support them. FEMA officials said they also phone grantees after each submission cycle to discuss challenges, including how to improve FEMA’s technical assistance and support, and how to make the process more useful for grantees. State officials we spoke to said that especially since the 2017 hurricanes, they have received more upfront guidance from FEMA than previously. Generally, FEMA’s support to grantees includes published guidance, annotated examples, technical assistance webinars, and a help desk for phone and email assistance. In 2018, FEMA also began piloting readiness visits where FEMA regional officials met with state and local grantees to discuss capability gaps identified in their assessments, according to officials. However, officials from three of the six grantees included in our review said that they did not receive key feedback from FEMA about their mass care capabilities assessments that would have been useful. An official in one state said it did not receive helpful feedback from FEMA prior to the 2017 hurricane season and, in particular, the official would have liked FEMA to confirm whether the state had completed its assessment correctly and completely, or if other information was needed. Officials from another state said that they did not receive any substantive feedback on their 2017 assessment. Officials from one urban area grantee said they did not receive technical feedback on areas of least readiness, and noted it would be helpful if FEMA could provide insight on the information provided in cases where the grantee had assigned a low confidence level in its capability assessment. Officials from four of the six grantees we spoke with said they would like additional clarity about the process from FEMA. For example, one state official said that understanding how FEMA uses capabilities information would have helped the grantee know how to improve its responses; get other agencies to participate more in the process; and solicit better, more tailored information from partners. This official noted that FEMA addressed this issue in 2019 by sharing more information about how it uses capabilities information. An official from another state said the state preparedness office would like input about how to obtain information from other agencies and how to assess capabilities at the local level. FEMA has an opportunity to use its review of capability assessments to improve its ability to assist with future disasters. After reviewing the 2018 submissions that used the new methodology, FEMA officials told us they are planning to develop criteria for evaluating future submissions and establish a regular process for providing feedback. By not systematically following up with grantees thus far, FEMA limits the extent to which it can build and supplement the emergency preparedness capabilities of these grantees. According to FEMA, it routinely analyzes capabilities assessment information for this purpose. FEMA has a strategic goal that involves supporting emergency managers in building the capacity to self- evaluate, monitoring the completion of improvement actions, and sharing insights. Providing feedback to grantees, including on the effective use of capability assessments as well as potential pitfalls, may help grantees develop their capability assessments and inform plans for how FEMA and the grantee will respond to disasters. Without clear protocols for providing feedback, grantees and FEMA may not possess complete, accurate, and reliable information on communities’ mass care capabilities, which will limit the effectiveness of the capability assessment process in contributing to the goal of national preparedness. The 2017 hurricane season presented unprecedented challenges for mass care service providers, and for survivors in Florida, Puerto Rico, Texas, and the U.S. Virgin Islands. While many partners coordinated extensively on the mass care response to 2017 hurricanes, unmet needs in sheltering, feeding, and supply distribution should spur FEMA and the Red Cross to consider the sufficiency of current agreements, especially with state and local governments. In particular, the 2017 hurricanes highlighted the importance of state and local governments understanding the services that mass care providers can deliver, particularly when disasters are severe or overlapping. Without FEMA providing more targeted guidance to help states and localities develop specific written agreements with voluntary organizations providing mass care services, expectations for what these organizations can provide may be unclear, putting disaster victims at risk. Moreover, without proactively considering the roles and responsibilities that the federal disaster framework establishes for agencies and organizations coordinating mass care, DHS lacks assurance that responsibilities are assigned to the entities best suited to carry them out. In addition, mass care coordination efforts during the 2017 hurricane season illustrated the importance of appropriately sharing information about capabilities and resources as part of advance preparation. During a disaster, local community groups are often the most informed about where needs exist, but also may not be connected with established mass care partners. Further leveraging community groups could prove vital for meeting mass care needs in a large-scale disaster, especially for the most vulnerable populations. FEMA does not explicitly require grantees to involve key mass care providers in their capabilities assessments. This may make it difficult for grantees to be well informed as to what they are actually capable of delivering locally. Further, FEMA has not documented a consistent, systematic approach to following up with partner governments on their reporting of mass care capabilities, while some grantees have said that additional feedback would be useful for preparedness and response efforts. As a result, some grantees may be ill-prepared to meet the mass care needs of the public during future disasters. We are making a total of six recommendations, including the following recommendation to the Secretary of Homeland Security: To strengthen the mass care response to future disasters, the Secretary of Homeland Security should direct FEMA to periodically review the current structure of ESF-6 leadership roles and responsibilities for coordinating mass care. (Recommendation 1) In addition, we are making the following four recommendations to the FEMA Administrator: To better clarify what mass care services voluntary organizations can provide, especially for severe or overlapping hurricanes, FEMA should strengthen its guidance to state and local governments to emphasize the importance of clearly defining roles and responsibilities related to mass care when state and local governments develop written agreements with partner organizations. This could include creating a guidance document or memo that calls attention to the issue and brings together existing resources, such as the Multi-Agency Feeding Plan Template and training materials, in a comprehensive and accessible manner. (Recommendation 2) To ensure assistance reaches all survivors, FEMA should develop mechanisms for the agency and its partners to leverage local community groups, such as conducting regular outreach to communicate and share aggregate information with these groups. (Recommendation 3) To ensure more accurate mass care capability assessments, FEMA should require grantees to solicit capabilities information from key mass care service-delivery providers in making capability estimates and identify these providers in their submissions. (Recommendation 4) To build the emergency preparedness capabilities of grantees, FEMA should develop systematic, documented protocols to determine the conditions under which it will follow up and provide feedback to grantees about mass care capability assessments. (Recommendation 5) We are also making the following recommendation to the American Red Cross: To ensure assistance reaches all survivors, Red Cross should develop mechanisms for it and its partners to leverage local community groups, such as conducting regular outreach to communicate and regularly share aggregate information with these groups. (Recommendation 6) We provided a draft of this report to DHS and the American Red Cross (Red Cross) for review and comment. DHS and American Red Cross provided written comments, which are reproduced in appendices II and III, and described below. In addition to its formal letter, DHS provided technical comments, which we incorporated as appropriate. We also provided relevant excerpts of the draft report to third parties, such as state and local government agencies and voluntary organizations we interviewed. These third parties provided technical comments, which we incorporated as appropriate. In its formal letter, DHS concurred with four of our recommendations and did not concur with one recommendation. Specifically, DHS and FEMA did not concur with our recommendation that FEMA should require grantees to include key mass care service-delivery providers in making capability estimates and identify these providers in their submissions. The letter noted the importance of involving stakeholders and subject matter experts at multiple levels of government and across sectors in order to develop complete and accurate assessments. However, DHS and FEMA said that requiring communities to include the key mass care providers in capabilities assessments is not the most effective approach for achieving this outcome. Because grantees cannot control which partners participate, DHS and FEMA said implementing this recommendation would increase the burden on grantees and could put certain communities at a disadvantage. In addition, DHS and FEMA said that because capabilities assessments are not limited to mass care, such a requirement may have unintended consequences for other partners. Instead, the letter stated that FEMA plans to continue working with the mass care community to identify the best solution, including encouraging collaboration at all levels of government. We modified our recommendation to address their concern. Specifically, we clarified that FEMA should require grantees to solicit information from key mass care partners and to identify these partners in their submission. This change acknowledges that grantees cannot compel partners to participate, but they can, at a minimum, invite such partners to participate in the process. We continue to believe that grantees should be required to make an effort to include mass care providers in developing their mass care capability assessments, as this is vital for developing high quality assessments. FEMA has emphasized the importance of having an active relationship and ongoing communication with key partners before disasters strike. In its Strategic Plan, FEMA states that pre-disaster coordination and communication among partners is critical to improve response and recovery outcomes. Thus, we do not believe it would be an undue burden to reach out to such partners as part of the capability assessment process. With regard to the remaining recommendations, DHS and FEMA described steps they have taken or plan to take to address the issues raised. While DHS concurred with recommendation 1 to direct FEMA to periodically review the ESF-6 leadership roles and responsibilities, the department considers this issue to be resolved because FEMA routinely conducts after-action reports and recently established a working group focused on performance metrics and corrective actions. We agree that these actions are important parts of effectively overseeing and evaluating ESF activities and results. While these efforts may address the responsibilities of ESF agencies, they may overlook the overall leadership roles of ESF agencies. In order to fully implement the recommendation, DHS and FEMA would also need to establish a process for reviewing the structure of ESF leadership roles on a regular basis. In concurring with recommendation 3, DHS and FEMA detailed several approaches they use to connect with local resources, including collaborating with VOAD groups at national, state, and local levels, and indicated that they consider this recommendation already implemented. Given the information gathered from several unaffiliated organizations in areas affected by the 2017 disasters, it is clear there is more work to be done in terms of sharing critical information about mass care needs and resources. Therefore, we continue to encourage FEMA to develop additional mechanisms to enhance outreach to organizations that may not be aware of existing approaches such as collaboration with the VOAD groups. Red Cross agreed with our recommendation to leverage local community groups through outreach and information-sharing. Red Cross noted several ongoing activities to engage such community groups and said the organization intends to continue expanding outreach, data-sharing, and engagement initiatives. We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, American Red Cross, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512- 7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Department of Homeland Security - Federal Emergency Management Department of Homeland Security - Federal Emergency Management Support Agencies with Roles Directly Related to Mass Care (Feeding, Sheltering, Supply Distribution, and Family Reunification): Corporation for National and Community Service Department of Defense/ U.S. Army Corps of Engineers Department of Health and Human Services Department of Homeland Security Department of Veterans Affairs National Center for Missing & Exploited Children National Voluntary Organizations Active in Disaster (National VOAD) In addition to the contact named above, Scott Spicer (Assistant Director), Amy Moran Lowe (Analyst-in-Charge), Grace Cho, and Michael Walton made key contributions to this report. Also contributing to this report were Joel Aldape, Aditi Archer, Susan Aschoff, James E. Bennett, Deirdre Gleeson Brown, Alicia Cackley, Sarah Cornetto, Elizabeth Curda, Chris Currie, Kelly DeMots, Erin Guinn-Villareal, Camille Henley, Denton Herring, Sara Schibanoff Kelly, James Lawson, Matthew T. Lowney, Sheila R. McCoy, Jean McSween, Amanda R. Parker, Sara Pelton, Brenda Rabinowitz, Michelle Sager, Brian Schwartz, Almeta Spencer, Manuel Valverde, Jr., and Su Jin Yon.", "summary": "Three catastrophic hurricanes affected more than 28 million people living in Texas, Florida, Puerto Rico, and the U.S. Virgin Islands in 2017. Hurricanes Harvey, Irma, and Maria—which all made landfall within four weeks—caused a combined $265 billion in damage, and led to unprecedented demands for food and shelter, according to FEMA. FEMA and the Red Cross are the primary agencies responsible for coordinating mass care under the federal disaster response framework. GAO was asked to review their efforts. This report examines (1) FEMA's and the Red Cross' coordination of mass care in response to the 2017 hurricanes, and (2) FEMA's support and use of assessments of mass care capabilities for the 2017 hurricanes. GAO reviewed relevant federal laws, federal frameworks, and written agreements between federal, state, or local governments and various voluntary organizations providing mass care services. GAO also interviewed state, territorial, local, and voluntary organization officials in Florida, Puerto Rico, Texas, and the U.S. Virgin Islands; as well as officials from Red Cross, FEMA, other relevant federal agencies, and voluntary organizations. Following the three major U.S. hurricanes in 2017, disaster relief efforts of the Federal Emergency Management Agency (FEMA) and the American Red Cross (Red Cross) benefitted from locating key partners in the same place. In-person coordination was critical to maintaining communication in Puerto Rico and the U.S. Virgin Islands given the prolonged power outages and damage to public structures (see photo). However, some needs related to mass care—such as shelter, food, and supply distribution—were unmet. For example, local officials in Texas said flooded roads prevented trucks from delivering supplies. Providers encountered challenges in part because state and local agreements with voluntary organizations did not always clearly detail what mass care services could be provided. Additionally, FEMA guidance and training materials do not explicitly encourage states and localities to include in their written agreements the specific assistance each agency or organization can provide. This limits the benefits of mass care coordination and may put disaster victims at risk. State, territorial, and local grantees of federal disaster preparedness grants are required to regularly submit information on their capabilities to FEMA, and FEMA has provided related guidance and technical assistance. However, the information some grantees provided to FEMA was not specific enough to aid its response in 2017. Moreover, FEMA does not require grantees to specify the organizations providing mass care services in their capabilities assessments. Also, FEMA does not have systematic protocols for providing feedback to grantees to improve their assessments. These limitations hinder FEMA's efforts to strengthen emergency preparedness. GAO is making six recommendations, including that FEMA emphasize the importance of defining roles and responsibilities in its guidance to states and localities, require them to solicit information from key mass care providers in assessing capabilities, and develop protocols for providing feedback to grantees on capability assessments. FEMA agreed with all but one of GAO's recommendations; GAO maintains its recommendations are valid.", "document_type": "gao"}
{"report": "This section provides an overview of (1) the impact of nuclear or radiological events, (2) U.S. efforts to combat nuclear or radiological smuggling, (3) STC program goals and phases, (4) how the STC program operates, and (5) STC program activities. We previously reported that a terrorist’s use of either an improvised nuclear device or a radiological dispersal device could have devastating consequences, including not only loss of life but also enormous psychological and economic impacts. An improvised nuclear device is a crude nuclear bomb made with highly enriched uranium or plutonium. A radiological dispersal device —frequently referred to as a dirty bomb— would disperse radioactive materials into the environment through a conventional explosive or through other means. Depending on the type of radiological dispersal device, the area contaminated could be as small as part of a building or a city block or as large as several square miles. If either type of device were used in a populated area, hundreds of individuals might be killed or injured from the explosion or face the risk of later developing health effects because of exposure to radiation and radioactive contamination. U.S. efforts to counter nuclear or radiological threats are considered a top national priority. Federal agencies that have a role in combating nuclear or radiological smuggling are responsible for implementing their own programs under the GNDA. The GNDA comprises programs run by U.S. agencies, including DHS, the FBI, and NNSA, as well as partnerships with local, state, tribal, and territorial governments; the private sector; and international partners. These programs are designed to encounter, detect, characterize, and report on nuclear or radiological materials that are “out of regulatory control”, such as those materials that have been smuggled or stolen. Under DHS’s reorganization, there is no longer a specific directorate in charge of GNDA responsibilities, according to CWMD officials. However, CWMD officials said that GNDA responsibilities, such as identifying gaps in current nuclear detection capabilities, will be distributed throughout CWMD components. CWMD initiated the STC program with three primary goals: (1) enhance regional capabilities to detect and interdict unregulated nuclear and other radiological materials, (2) guide the coordination of STC cities in their roles defined by the GNDA, and (3) encourage participants to sustain their nuclear or radiological detection programs over time. According to the Program Management Plan, for each city, the STC program consists of three phases that provide for the development, integration, and sustainment of nuclear or radiological detection capability by cities to support state, local, and tribal operations. Phase 1: Development of initial operating capability. CWMD provides a mechanism for cities to develop initial operating capability to detect and report the presence of nuclear or radiological materials that are out of regulatory control. During phase 1, efforts focus on satisfying the immediate needs of state and local agencies in developing detection and reporting capabilities. This phase of the implementation is expected to take 3 years. Phase 2: Integration. CWMD provides additional resources to cities to allow them to develop enhanced detection, analysis, communication, and coordination functionality. These resources build on the integration of state and local capabilities with U.S. government activities and the GNDA that existed prior to cities’ participation in the STC program or were established during phase 1. This phase is expected to take about 2 years. Phase 3: Sustainment. CWMD provides indirect support to cities to sustain their capabilities. CWMD maintains a relationship with local program operators through assistance with alarm response and subject matter expertise. For example, it provides advice to cities on training, practice exercises, and questions as they arise. As of March 2019, Chicago and Houston are in phase 1 of the program, the National Capital Region is in phase 2, and New York—New Jersey and Los Angeles—Long Beach are in phase 3. The STC program operates as a cooperative agreement between CWMD and eligible cities. Accordingly, a substantial amount of interaction is expected between CWMD and program participants. A full cooperative agreement package for the STC program includes a notice of funding opportunity, notice of financial assistance award (assistance award), and general guidance documents for the program. It also includes requirements for cities to develop performance metrics for achieving key program tasks, such as purchasing equipment and conducting training, and to submit quarterly financial and performance reports. CWMD seeks applications for the program through a notice of funding opportunity, which lays out eligibility criteria and other requirements. According to CWMD officials, after New York—New Jersey was accepted into the STC program, CWMD opened up eligibility for the program to cities in DHS’s Urban Area Security Initiative (UASI) identified as having the highest risk for a terrorist attack. In the application process, one local government entity applies as the principal partner for the city (e.g., the New York Police Department is the principal partner for New York— New Jersey). Once CWMD accepts a city into the program, the city receives an assistance award, which details the approved budget for the year and may include an approved purchase plan. DHS prefers that a lead agency within the city distributes funds or any equipment purchased with program funds to the other state and local partners, such as police departments of neighboring jurisdictions, fire departments, or public health officials, among others. According to CWMD officials, every year cities in the program must apply for the next increment of funding from the program; if a city’s application is approved, it receives an amendment to its assistance award. There is a 5-year period of performance— corresponding to phases 1 and 2—under which the cities are eligible to receive and obligate funding. CWMD officials told us that they can grant an extension to cities to obligate the funds if they have not been able to do so within the original 5-year period. In phase 3 of the program, CWMD may provide technical assistance or subject matter expertise to cities but no further funding. Cities in the STC program may spend their funds on nuclear and radiological detection equipment, training, and administrative program costs, among other things. Several types of detection equipment may be approved for purchase. Personal radiation detectors (PRD) are wearable radiation detectors, approximately the size of a cell phone. When exposed to elevated radiation levels, the devices alarm with flashing lights, tones, vibrations, or combinations of these. Most PRDs numerically display the detected radiation intensity (on a scale of 0 to 9) and thus can be used to alert the officer of a nearby radiation source. However, they typically are not as sensitive as more advanced detectors and cannot identify the type of radioactive source. Radiation detection backpacks are used for primary screening and for conducting wide area searches, according to CWMD officials. These officials said the size of the detector contained within the backpack allows the operator greater detection sensitivity as compared to a PRD. CWMD officials also said these devices are especially useful for screening a large venue for radiological materials prior to occupancy by the public. Radiation isotope identification devices are radiation detectors that can analyze the energy spectrum of radiation, which enables them to identify the specific radioactive material emitting the radiation. Such devices are used to determine if detected radiation is coming from a potential threat or from naturally occurring radioactive material, such as granite. Mobile detection systems contain larger detectors. Typically, mobile detection systems interface with a laptop computer to display alarms and analysis, and are capable of both detection and identification. This type of system may be mounted on vehicle platforms, such as cars, trucks, vans, boats, or helicopters. Figure 2 shows examples of such equipment. Such equipment and associated training are the basis for the capability provided through the STC program. Officials we interviewed in one STC city told us that in order to operate the equipment, law enforcement, fire, health, and other state and local personnel must take training on the process for screening and for resolving alarms related to suspected nuclear or radiological material. As shown in figure 3, primary screening is the first step of the process: if an officer is able to determine the source of the alarm and deems it a nonthreat, then the case is resolved. According to CWMD officials, PRDs often detect nuclear or radiological materials that do not actually pose threats, such as radiation from medical treatments and from naturally occurring substances such as granite. An officer who is not able to determine the source of the alarm should initiate a secondary screening process; according to CWMD officials, secondary screening varies by locality. Officers with advanced training conduct secondary screening by using equipment such as radiological isotope identification devices to identify the type of source material detected. If, after secondary screening, officers still suspect a threat, they can contact technical “reachback,” which is a system that puts officers on the ground in communication with off-site specialists and resources. This technical reachback can provide greater expertise, including the ability to analyze the energy spectrum detected during screening and improve identification of the source and nature of the potential threat. CWMD officials said that the technical reachback may occur at the state and local or national level. State and local technical reachback procedures may vary, but national level technical reachback is standardized with 24-hour call centers run by the Department of Energy or U.S. Customs and Border Protection. According to CWMD officials, at any point in the screening process, if a secondary screening device is utilized, it is standard protocol for the officer to alert the FBI of the incident. If a threat is suspected, the FBI can deploy a team that is trained to respond to such a threat. DHS’s CWMD does not collect information to fully track cities’ use of STC funds for approved purposes and to assess the cities’ performance in the program. Specifically, CWMD tracks cities’ spending using program funds and some performance data through quarterly reports that it collects from cities, but does not collect other key data to track itemized expenditures and to assess how effectively cities achieved key performance metrics and program milestones or how they performed in exercises or drills that simulate a nuclear or radiological threat. CWMD tracks cities’ spending using program funds through quarterly financial reports it collects from cities, according to CWMD officials, but does not collect other key data to ensure that funds are spent for approved purposes and not spent on unrelated program activities. Specifically, CWMD provides each city eligible for additional funding an assistance award every year that includes an approved budget for spending categories such as program staff and equipment, but CWMD officials told us that CWMD does not track itemized expenditures to ensure that program funds were spent according to this budget. According to CWMD’s program agreements with cities, cities must have written approval from DHS in advance of spending obligated program funds for all equipment purchases in the amount of $5,000 or more per unit cost. However, CWMD officials told us that because of time and resource constraints, they do not collect data that cities maintain in their internal systems on the expenditures they actually made with program funds, even though CWMD’s program agreements with cities typically specify that CWMD or DHS’s Grants and Financial Assistance Division (GFAD) may access these data at any time. Furthermore, although GFAD officials told us that CWMD, in conjunction with the Grants Officer at GFAD, has the authority to conduct programmatic and financial audits and site visits to cities, these audits are infrequent and limited in their ability to ensure that cities’ expenditures were in accordance with CWMD’s approved purchase plans, which take into account program goals and objectives. According to these officials, in the program’s history, GFAD has conducted a total of two desk audits in two STC cities—New York—New Jersey and Los Angeles—Long Beach. GFAD initiated these two audits in 2015 and, according to GFAD officials, examined a small random sample of purchases. GFAD officials said they do not currently plan to conduct any additional audits in STC cities because of resource constraints. The extent of CWMD’s tracking of cities use of STC program funds is not consistent with federal internal control standards, which state that program management should design control activities to achieve objectives, such as comparing actual performance to planned or expected results and analyzing significant differences. However, according to CWMD officials, CWMD does not compare information on expenditures to cities’ approved purchase plans. As a result, DHS does not know the dollar amounts cities actually spent on program purchases. By regularly collecting detailed information from cities on expenditures made using program funds and comparing that information to approved purchase plans, CWMD would have greater assurance that cities spent funds as approved and that the expenditures are in keeping with program goals and objectives. Because CWMD does not regularly collect or maintain data on how cities spent program funds, we requested that it ask cities for these data and provide them for our review. Table 1 summarizes STC program funds obligated to and spent by each city and shows that New York—New Jersey spent about three-quarters of all STC funds—about $110 million of the $145 million cities spent as of June 30, 2018. As discussed above, New York—New Jersey was the pilot city for the program and was not subject to the $30 million limit on program funding. In addition to program funds, CWMD provided cities with nonmonetary assistance in the form of training, among other things. These data also show that cities spent most STC funds on equipment purchases. Specifically, about two-thirds of STC funds spent were for equipment to detect nuclear or radiological threats—about $95 million of the $145 million spent. Among the four cities that have purchased equipment, the largest equipment purchase category was PRDs, at over $40 million. Cities also reported purchasing equipment such as backpacks that contain radiation detectors; radiation isotope identification devices, which identify the type of radiation that is emitted from a source; and mobile systems that detect radiation from a vehicle on the ground or in the air. In addition, cities spent STC funds on training, staff, and contracts for training and other services, according to the data. Collectively, cities spent about 6 percent of program funds on training, 3 percent on staff, and 14 percent on contracts for training and other services. (See table 2.) CWMD tracks some performance data in quarterly reports it collects from cities, but it does not collect data to ensure that key performance metrics and program milestones identified in the Program Management Plan are achieved. For example, the quarterly reports CWMD collects from cities show the quantities of equipment, by type, that cities purchased with STC funds over the course of the program (see table 3), but these reports do not show whether the quantities of equipment met cities’ targets for equipment purchases. In addition, these reports do not show how much cities spent to purchase equipment for the program. CWMD’s notices of funding opportunity require cities to identify and submit key performance metrics for measuring progress against their objectives and a schedule of program milestones as part of their application to the STC program. According to the CWMD officials, each STC city submitted a Gantt chart—which plots planned activities over time—as part of its initial application. However, over the course of the program, CWMD found this tool had limited value and later gave each city the latitude to manage its program timeline as it deemed appropriate. In addition to the Gantt charts, CWMD officials said they provided cities with templates to develop checklists to document their progress against their objectives and compare their progress to planned actions. However, CWMD officials told us that they view this checklist as a guide to help cities plan rather than a firm program requirement, and cities have not submitted these checklists. Until CWMD requires cities to submit checklists or equivalent information on their progress in the STC program, it will not have complete information on how cities are performing compared to the key performance metrics and program milestones they identified for themselves. CWMD does not consistently collect information on how cities performed during STC program-funded exercises and drills that test cities’ ability to detect a simulated nuclear or radiological threat. CWMD’s notices of funding opportunity entered into after 2007 generally state under program performance reporting requirements that cities must submit operational reports, such as exercise after-action summaries. CWMD officials told us that they have provided STC cities with a template for preparing after- action reports—which assess a city’s performance during an exercise and include improvement plans following exercises that the program funded. These reports and plans could provide greater insight than quarterly performance reports on the effectiveness of cities’ capabilities. Nonetheless, available performance data show that CWMD did not enforce this requirement and that cities have submitted very few after- action reports. In their quarterly performance reports, the four cities other than New York—New Jersey reported completing 231 drills and exercises but only five after-action reports and one improvement plan. Officials from New York—New Jersey, whose performance reporting requirements differ from those of other cities according to CWMD officials, said that they complete over 100 drills and exercises per year but do not complete after-action reports because of the amount of paperwork that would be required. CWMD officials said that they did not enforce the requirement to submit after-action reports and improvement plans because they felt they could not force cities to report this information. Officials also told us that even though cities are aware of requirements in CWMD’s notices of funding opportunity to provide these reports and plans, cities may be reluctant to complete them because they could highlight weaknesses in their capabilities. We have previously found that a leading practice to promote successful data-driven performance reviews includes participants engaging in rigorous and sustained follow-up on issues identified during reviews. Until CWMD more fully assesses cities’ performance by consistently enforcing reporting requirements on how cities performed during exercises, it cannot assess the extent to which cities could effectively detect or deter a nuclear or radiological threat. DHS’s CWMD does not have assurance that cities can sustain threat detection and deterrence capabilities gained through the STC program, and cities anticipate funding challenges once STC program funding ends. Specifically, CWMD has not enforced sustainment planning requirements and has taken limited action to help cities sustain their capabilities, even though encouraging sustainment is one of its primary program goals. Cities anticipate funding challenges that will adversely affect their ability to sustain capabilities after the program. CWMD identified a key goal related to sustainment of cities’ nuclear or radiological detection program overtime in its Project Management Plan and requires cities to plan for sustainment. However, CWMD has not enforced sustainment planning requirements and has taken limited action to help cities sustain capabilities. CWMD’s program agreements generally require cities to submit plans describing how they will sustain capabilities gained through the program. For example, some of CWMD’s program agreements state that these sustainment plans must (1) explain how the city will support and sustain STC capabilities after completing the program, (2) describe potential sources of future financial support, and (3) commit to obtaining future financial assistance beyond CWMD support. However, CWMD accepted sustainment plans from four cities that did not identify how they will sustain capabilities once program funding ended. Each of the cities’ plans clearly state that they will have difficulty sustaining the program without additional federal funds. (See fig.4.) We also found that three of the four sustainment plans submitted to CWMD provide little detail about the specific equipment or training cities expect they will need after program funding ends. CWMD, however, did not take steps to address these concerns because CWMD officials said that they viewed finding alternative sources of funding to sustain capabilities as the cities’ responsibility. CWMD officials told us that they provide some ongoing technical assistance to cities in the sustainment phase of the program, but this assistance does not include additional funding. Thus far, New York—New Jersey is the only city of the two cities in the sustainment phase that has received technical assistance. Furthermore, CWMD did not consistently take steps to ensure that cities planned for sustainment when making purchasing decisions. As previously noted, program agreements generally require sustainment plans. Under CWMD’s Project Management Plan, CWMD expects cities to submit those sustainment plans to CWMD within 24 months of their initial award date. However, New York—New Jersey and Los Angeles— Long Beach did not submit their sustainment plans until many years after they began to receive STC funding. New York—New Jersey, for example, did not submit a draft sustainment plan until 2015, nearly 8 years after the city initially received funding because CWMD did not include a sustainment plan requirement for the city until its award for fiscal year 2011 and allowed 36 months to complete a sustainment plan. Similarly, Los Angeles—Long Beach did not submit a draft sustainment plan until 2017—5 years after the city initially received funding. In its program agreement with Los Angeles—Long Beach, CWMD required that a sustainment plan be submitted within 18 months of the award date, but CWMD did not enforce this requirement and accepted a sustainment plan from Los Angeles—Long Beach that was significantly delayed. It is unclear whether New York—New Jersey and Los Angeles—Long Beach ever finalized their draft sustainment plans. CWMD identified sustainment as a program goal but has not enforced its own requirements related to this goal or taken steps to analyze the risks sustainment challenges pose to its program’s success. Federal internal control standards state that program management should identify, analyze, and respond to risks related to achieving the defined objectives. Unless CWMD analyzes risks related to sustainment, works with cities to address these risks, and enforces sustainment planning requirements for cities that join the program in the future, program participants could see their radiological detection programs and related capabilities deteriorate over time. Officials from all five cities raised concerns to us about their ability to maintain capabilities over time without a dedicated source of funding once STC program funding ends. For example, New York—New Jersey officials told us that they informed CWMD they would not be able to maintain capabilities past 2021 without additional funds. Houston conducted an analysis of the funds needed to sustain the program and estimated that it would generally need over $1 million per year, primarily to replace equipment. City officials also said that they are already experiencing challenges that will have implications for funding and sustainment of the program. For example, Chicago officials said they are facing challenges regarding funding for training. These officials said CWMD told them that the company that conducted training in the other STC cities—at no cost to those cities—will no longer be the designated training entity. But a new training company has not been put in place. CWMD has not communicated a new plan for training Chicago’s officers on equipment that has already been purchased, and Chicago officials told us that they do not have additional funds to purchase training. Chicago officials said that if they do not receive future years of funding to conduct training on the already-purchased equipment, their planned capabilities could go to waste. According to several city officials, cities cannot rely on other DHS grant programs or federal grant programs or local sources of funding to sustain the STC program. Specifically, the officials said that cities’ ability to obtain funds from DHS’s UASI for sustainment may be limited, in part because of ineligibility by some partner agencies within an STC city. For example, law enforcement agencies in Santa Ana, California, received support from the STC program as part the Los Angeles—Long Beach city region, but they would not be eligible for UASI funds because Santa Ana is not in the Los Angeles—Long Beach UASI region. Moreover, UASI funds may not be sufficient to meet demand from cities. Houston city officials said that in fiscal year 2017, the city had requested $40 million in UASI funds from the UASI Committee, which distributes UASI funds in each city. But the committee had only $23 million to disperse to Houston. According to CWMD officials, other DHS grant programs within the Federal Emergency Management Agency—such as the Homeland Security Grant Program— may not provide a guaranteed source of consistent funding. Further, CWMD, NNSA, FBI, and city officials that we interviewed said they were not aware of any other federal grant program that cities could utilize to sustain nuclear or radiological detection capabilities. At a local level, several city officials said that there are competing funding priorities, such as preventing school shootings and addressing the opioid crisis, that require more money and attention because they affect the local community more directly every day. DHS has not (1) fully developed potential changes or documented a plan for making changes to the STC program; (2) identified the basis for such changes; and (3) clearly communicated with the cities, raising concerns about how the changes will impact them. CWMD officials told us that the agency is considering several potential changes to the STC program that would broaden its geographic reach and scope, but it has not fully developed or documented these changes and does not have a strategy or plan for implementing them. According to these officials, CWMD has not made any final decisions about potential changes and therefore has not developed any formal strategic documents. Based on our interviews with CWMD and city officials and some limited information in DHS’s fiscal year 2019 budget justification, we found that CWMD is considering making the following changes to the STC program: New program goals. CWMD officials told us that the STC program’s new goals would be to (1) enhance regional capabilities to detect, analyze, report, and interdict nuclear and other radioactive threats; (2) provide defense in large geographic regions; and (3) maximize deployment of detection equipment to nonfederal agencies to support federal nuclear detection priorities. The first program goal is one of the original program goals. However, CWMD officials said that under this proposal, CWMD would no longer include encouraging cities to sustain capabilities over time as a program goal because CWMD has discussed centralizing acquisition of detection equipment. Expansion of the program’s geographic coverage. Although legacy cities would still receive support under the new version of the STC program, CWMD officials said that the new program would provide national coverage and would include detection and deterrence activities in regions well outside of cities that UASI identified as having the highest level of threat and risk for a terrorist attack. Prior to proposing this change, CWMD had included in DHS’s fiscal year 2018 budget justification its intent to select a sixth and seventh city to participate in the program by the end of fiscal year 2018, which CWMD officials told us did not occur. In DHS’s fiscal year 2019 budget justification, CWMD stated its intent to support the development of nuclear or radiological detection capability for broader regions. Centralized acquisition of detection equipment. Instead of providing funding to STC cities to purchase detection equipment directly, CWMD officials told us that they would plan to centralize the acquisition process and purchase equipment on behalf of cities and regions. CWMD officials told us that they expect most of this equipment to be PRDs. A greater role for other agencies. CWMD officials said that although the STC program would remain a CWMD-only program, CWMD expects to work closely with the FBI, NNSA, and other DHS components, such as the U.S. Coast Guard and U.S. Customs and Border Protection, to detect and deter nuclear or radiological threats. Currently, according to CWMD officials, CWMD is working with the FBI and NNSA on a Domestic Detection Concept of Operations to coordinate their capabilities and functions. In addition, CWMD officials said that they plan to align the STC program with the existing FBI stabilization program, which responds to nuclear or radiological threats that have been detected. According to CWMD officials, CWMD would rely on FBI-led stabilization teams for guidance on selecting and distributing detection equipment for the STC program. Each stabilization team would have a partner STC program office to test, calibrate, and distribute detection equipment and to train operators, and the STC program would provide funding to cities to maintain these offices. Inclusion of chemical and biological weapon detection and deterrence within the program’s scope. The Countering Weapons of Mass Destruction Act of 2018 includes chemical and biological weapon detection and deterrence under the scope of CWMD but limits the STC program to detecting and deterring nuclear or radiological threats. CWMD officials told us that they had planned to add chemical and biological detection and deterrence efforts to the STC program, but such a change would now require a statutory change. The changes that CWMD is considering making to the STC program would be significant in scope. However, CWMD officials confirmed that CWMD has not documented these potential changes for key stakeholders, such as cities or partner agencies or provided strategic documents to describe how it plans to implement any changes. FBI officials we interviewed said that although the FBI supports greater coordination between CWMD and FBI-led stabilization teams, these programs will remain distinct and independent, with separate and dedicated lines of funding and personnel. These officials also said that CWMD and the FBI will not share equipment or technicians. According to NNSA officials, there is no new role defined for NNSA in the STC program, although NNSA leadership has asked its Radiological Assistance Program to contribute to the STC program where possible. NNSA officials also said that NNSA and CWMD will continue to coordinate on how information flows at a federal level if a nuclear or radiological threat has been detected. CWMD officials told us that they first introduced potential program changes to five STC cities at a meeting in February 2018 and met with leadership from these cities in August 2018 to discuss these changes further. In November 2018, we contacted officials from the STC cities to determine whether they understood how the STC program would continue. Officials from the STC cities made statements that indicated confusion and uncertainty about the future of the program. For example: Officials from one city told us they believed that changes to the STC program would apply only to new cities joining the program, even though CWMD officials told us that the changes would affect all cities going forward. Officials in another city told us that they left the August meeting with the impression that the changes presented were only preliminary proposals up for discussion and that the program could evolve in any number of directions. However, documents CWMD provided to us during interviews show CWMD’s intention to make several of the specific changes described above, even though the agency’s proposals for the STC program have not yet been finalized. Officials in most cities told us they believed that CWMD may provide them separate funding under the new program for sustaining capabilities developed to date, but CWMD officials told us that no final decisions had been made regarding future support for legacy cities. Most city officials we interviewed said that the August meeting provided a high-level overview of potential changes and little detail on how such changes would be implemented or affect city operations. Our past work has discussed the importance of strategic planning. We have reported that, among other things, strategic plans should clearly define objectives to be accomplished and identify the roles and responsibilities for meeting each objective. By developing a written strategic plan (or implementation plan) for any potential changes to the STC program, CWMD would provide clarity on what specific changes are planned and how CWMD plans to implement them. For example, given the uncertainty around the future direction of the program, a written strategy would help shed light on the exact role that CWMD envisions for partner federal agencies and how it plans to utilize these partnerships to acquire and distribute equipment. In October 2018, we briefed staff on the Senate Committee on Homeland Security and Governmental Affairs and House Committee on Homeland Security on our ongoing work, including our preliminary findings on the benefits of (1) developing an implementation plan for potential changes to the STC program and (2) assessing the effect of changes on the program. The recent Countering Weapons of Mass Destruction Act of 2018, signed into law on December 21, 2018, requires that CWMD develop an implementation plan that among other things, identifies the goals of the program and provides a strategy for achieving those goals. The act requires CWMD to submit this implementation plan to Congress by December 21, 2019. In addition, the law requires a subsequent report assessing effectiveness and proposing changes for the program, which could provide clarity on how proposed changes would align with STC program strategy and how CWMD plans to implement them. CWMD is also required to consult with and provide information to appropriate congressional committees before making any changes to the STC program, including an assessment of the effect of the changes on the capabilities of the STC program. CWMD has not identified a clear basis for making program changes, and the extent to which these changes can be attributed to new priorities under DHS’s reorganization is unclear. CWMD officials told us that they have not conducted any studies or analyses that would justify making changes to the program. In DHS’s fiscal year 2019 budget justification, CWMD discussed the importance of using the STC program to build capabilities far outside the immediate target areas, (i.e., cities) and the need to detect threats along the air, land, or sea pathways into and within the country that terrorists could potentially use to reach their targets. However, according to CWMD officials, CWMD has not identified a change in the nature or level of nuclear or radiological threats to explain its intent to move from its original city-focused model for the STC program to a more national approach. In addition, as stated above, CWMD does not collect information to fully assess the performance of cities currently in the program and therefore does not have a performance-based rationale for changing its program goals. CWMD officials said that the uncertainty surrounding making changes reflect a program under transition within an agency under transition—that is, the reorganization from DNDO to CWMD. The Countering Weapons of Mass Destruction Act of 2018 requires that before making changes to the STC program, the Assistant Secretary of CWMD brief appropriate congressional committees about the justification for proposed changes. This briefing is to include, among other things, an assessment of the effect of changes, taking into consideration previous resource allocations and stakeholder input. This new requirement would provide DHS an opportunity to identify the basis for potential changes. Assessing such changes could provide more reasonable assurance that they would strengthen the program and not result in unintended consequences, such as reducing capabilities in current cities. CWMD has not clearly communicated with the cities currently in the STC program about the status of potential program changes, raising concerns among these cities about how the changes will impact them. Although CWMD officials told us that the STC program would still support cities currently in the program, CWMD has not communicated to cities the levels of funding or other resources they can expect to receive going forward under the new version of the program. Notably, CWMD has not explained how expanding the program’s geographical coverage would affect cities currently in the program, including any effect on the availability of resources for these cities. City officials told us that they had several concerns, including the following, about CWMD’s potential changes for the STC program: Ability to choose equipment that meets a city’s needs. Some city officials we interviewed expressed concerns that the potential changes could detract from their ability to decide which types of equipment and support would best meet their needs. For example, officials in one city expressed concern that their planned calibration laboratory, which is used to maintain equipment, could become obsolete if CWMD chose to distribute PRDs that differ from the type the city currently uses. Furthermore, some city officials questioned whether CWMD and local FBI-led stabilization teams could adequately assess the specific equipment needs of state and local partner agencies within current STC cities. FBI officials told us that they do not assess the equipment needs of state and local partner agencies, but instead share information with those partners should they wish to acquire similar resources in order to maintain state, local, and federal capabilities. Scope of the program. Several city officials said concerns arose when CWMD requested that STC cities test toxic compound meters in 2018, raising questions about the scope of the program. These devices are designed to detect the presence of certain chemical weapons, but the STC program does not include detecting or deterring chemical weapons. Therefore, several officials felt that the request to test the devices was outside the scope of their mission. CWMD officials said that although the meters were not connected with the STC program, it made sense to reach out to the STC cities as CWMD already had a relationship with the cities and they were deemed appropriate locations. Role of the FBI. Some city officials told us that they had heard from CWMD that the FBI could play an expanded role in secondary screening in the future, which they felt could be problematic because of the FBI’s limited staff presence in field locations. FBI officials we interviewed said that they did not plan to conduct additional secondary screening in the future; instead they plan to formalize the secondary screening process that is already in place in STC cities. According to FBI officials, the bureau would always respond to situations requiring a threat assessment. Effect on future funding, including for sustainment activities. CWMD recently informed National Capitol Region officials that they would not receive an expected fifth year of funding because of planned program changes. City officials said that this change came as a surprise to them and now they will only be able to buy approximately 90 percent of the equipment they had originally planned to purchase. In addition, these officials said that they planned to use much of the fifth year funding for sustainment activities, such as training classes, and that this loss would adversely affect their current sustainment plans. CWMD officials said that under the new program, CWMD will take responsibility for sustaining the nuclear or radiological detection equipment distributed to cities, but, as described above, these officials said that no final decisions have been made regarding future support for legacy cities. Several city officials said that CWMD had not adequately responded to their concerns and that there has been less communication from CWMD about the STC program since 2017 as a result of the DHS reorganization. Further, several city officials said that they expected CWMD to set up quarterly meetings with STC city leadership following the August meeting, but they had not received any notifications about additional meetings. CWMD officials told us that they intend to have more frequent meetings with STC city leadership in the future but were unable to schedule a meeting during the first quarter of fiscal year 2019. Federal internal control standards state that management should externally communicate the necessary quality information to achieve the entity’s objectives. If CWMD does not clearly communicate to the cities how the existing program will operate until a new program is developed and implemented, these cities could face difficulties planning for the future and achieving the program’s detection and deterrence objectives. DHS’s STC program has taken steps to address a top-priority threat to national security by providing high-risk cities with resources to develop nuclear or radiological detection capabilities. However, in implementing the program, CWMD does not collect key data to track itemized expenditures and to assess how effectively cities achieved key performance metrics and program milestones or how well they performed in exercises or drills that simulate a nuclear or radiological threat. By regularly collecting detailed information from cities on expenditures made using program funds and comparing that information to approved purchase plans, CWMD would have greater assurance that cities spent funds as approved, and consistent with program goals, and that the expenditures are in keeping with program objectives. In addition, until CWMD requires cities to submit checklists or equivalent information on their progress in the STC program, it will not have complete information on how cities are performing compared to the key performance metrics and program milestones they identified for themselves. Further, until CWMD more fully assesses cities’ performance by consistently enforcing requirements, as applicable, that cities report on how they performed during exercises, it cannot assess the extent to which cities could effectively detect or deter a nuclear or radiological threat. CWMD identified sustainment as a program goal but has not enforced its own requirements related to this goal or taken steps to analyze the risks sustainment challenges pose to its program’s success. Unless CWMD analyzes these risks, works with cities to address them, and enforces sustainment planning requirements for future cities, program participants could see their radiological detection capabilities deteriorate over time. CWMD officials told us that the agency is considering several potential changes to the STC program that would broaden its geographic reach and scope, but it has not fully developed or documented these changes and does not have a strategy or plan for implementing them. The Countering Weapons of Mass Destruction Act of 2018 requires that the Secretary of Homeland Security develop a strategy and implementation plan for the STC program and a subsequent report assessing effectiveness and proposing changes for the program, which could provide clarity on how proposed changes would align with STC program strategy and how CWMD plans to implement them. CWMD also has not provided a clear basis for proposed program changes. The act further requires that, before making changes, the Assistant Secretary of CWMD brief appropriate congressional committees about the justification for proposed changes, which should include an assessment of the effect of changes. This new requirement could help ensure that changes will strengthen the program and not result in unintended consequences, such as reducing capabilities in current cities. In the meantime, CWMD has not clearly communicated how its proposed changes will impact cities currently in the STC program, raising concerns among these cities about how the changes will impact them. If CWMD does not clearly communicate to the cities how the existing program will operate until a new program is developed and implemented, these cities could face difficulties planning for the future and achieving the program’s detection and deterrence objectives. We are making the following four recommendations to CWMD: The Assistant Secretary of CWMD should ensure that the office regularly collects detailed information from cities on expenditures made using program funds and compares that information to approved purchase plans to ensure that these funds were spent as approved, consistent with program goals, and that the expenditures are in keeping with the objectives of the program. (Recommendation 1) The Assistant Secretary of CWMD should more fully assess cities’ performance by collecting information from cities on achieving key performance metrics and program milestones and enforcing reporting requirements on performance during exercises. (Recommendation 2) The Assistant Secretary of CWMD should analyze risks related to sustaining detection capabilities, work with cities to address these risks, and enforce sustainment planning requirements for future cities. (Recommendation 3) The Assistant Secretary of CWMD should clearly communicate to cities how the existing program will operate until a new program is developed and implemented. (Recommendation 4) We provided a draft of this product to DHS, the FBI, and NNSA for review and comment. In its comments, reproduced in appendix I, DHS concurred with our recommendations in the draft report. DHS identified actions it would take to address these recommendations, including revising quarterly reporting requirements to include detailed information on expended funds, performance metrics, program milestones, and exercise activities. In addition, DHS said it would engage with cities to procure and distribute equipment and to refurbish or replace it when appropriate, and would conduct on-site senior-level meetings with all current STC cities to continue discussions about new procedures, partnerships, and sustainment of capability. We believe these actions, if implemented as described, would address the intent of our recommendations. DHS also provided technical comments, which we incorporated as appropriate. The FBI and NNSA told us that they had no comments on the draft report. We are sending copies of this report to the appropriate congressional committees, the Secretary of Homeland Security, the Secretary of Energy, the Assistant Attorney General for Administration of the Department of Justice, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Ned H. Woodward (Assistant Director), Keya Cain (Analyst in Charge), and Alexandra Jeszeck made key contributions to this report. Chris P. Currie, Pamela Davidson, R. Scott Fletcher, Juan Garay, Tom James, Benjamin Licht, Greg Marchand, Cynthia Norris, and Kiki Theodoropoulos also contributed to this report.", "summary": "Countering the threat that a terrorist could smuggle nuclear or radiological materials into the United States is a top national security priority. In fiscal year 2007, DHS initiated the STC program to reduce the risk of the deployment of a nuclear or radiological weapon by establishing capability in state and local agencies to detect and deter such threats. Since the program began, five participating cities have spent almost $145 million in program funds. GAO was asked to review the STC program. This report examines (1) the extent to which DHS tracks cities' use of program funds and assesses their performance; (2) what assurance DHS has that cities can sustain capabilities gained through the STC program and the challenges, if any, that cities face in sustaining such capabilities; and (3) potential changes to the STC program and how DHS plans to implement them, the basis for these changes, and the extent to which DHS has communicated with cities about the impact of making changes. GAO reviewed DHS documents, conducted site visits to all cities in the program, and interviewed DHS and city officials. The Department of Homeland Security (DHS) does not collect information to fully track cities' use of Securing the Cities (STC) program funds for approved purposes and to assess their performance in the program. To reduce the risk of successful deployment of nuclear or radiological weapons in U.S. cities, the program establishes local threat detection and deterrence capabilities. DHS tracks cities' spending of program funds and some performance data through cities' quarterly reports but does not collect other data on itemized expenditures and to assess how effectively cities achieved performance metrics and program milestones or how they performed in drills that simulate a threat. For example, DHS does not compare information on expenditures to the purchase plans it approved for cities. As a result, DHS does not know the dollar amounts cities actually spent on program purchases. Expenditure data GAO requested show that cities spent most funds on detection equipment—that is, $94.5 million of the $144.8 million cities spent through June 30, 2018. By regularly collecting expenditure information from cities and comparing it to approved purchase plans, DHS could better ensure these funds were spent consistent with program goals. DHS does not have assurance that cities can sustain threat detection and deterrence capabilities gained through the STC program. DHS has not enforced planning requirements for sustaining those capabilities and has taken limited action to help cities do so, although encouraging sustainment is one of its primary program goals. Officials from the five cities in the program told GAO that they anticipate funding challenges that will adversely impact their ability to sustain capabilities over time. For example, several city officials said they cannot rely on other DHS or federal grant programs or local sources of funding once STC funding ends. Unless DHS analyzes risks related to sustainment, works with cities to address these risks, and enforces sustainment-planning requirements for cities in the program in the future, program participants could see their radiological detection programs and related capabilities deteriorate. DHS has not (1) fully developed potential changes or documented a plan for making changes to the STC program; (2) identified the basis for such changes; and (3) consistently communicated with cities, raising concerns about how the changes will impact them. DHS officials told GAO that the agency is considering several potential changes to the STC program that would broaden its geographic reach and scope and centralize acquisition of detection equipment, among other things, but it has not fully developed or documented these changes and does not have a strategy or plan for implementing them. A law enacted in December 2018 requires DHS to develop an implementation plan for the STC program. The law's requirements would provide DHS an opportunity to identify the basis for potential changes, and assessing such changes would provide more reasonable assurance that they would strengthen the program. Further, most city officials GAO interviewed said that in an August 2018 meeting, DHS provided a high-level overview of potential changes and little detail on how such changes would be implemented or affect city operations. If DHS does not clearly communicate to cities how the program will operate under potential changes, these cities could face difficulties planning for the future and achieving the program's detection and deterrence objectives. GAO is making four recommendations including that DHS regularly collect detailed information from cities on program expenditures; analyze risks related to sustainment, work with cities to address these risks, and enforce sustainment-planning requirements for cities in the program; and clearly communicate to cities how the existing program will operate until a new program is in effect. DHS concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "TSA is responsible for securing the nation’s civil aviation system, which includes domestic and foreign air carrier operations to, from, within, or overflying the United States, as well as the foreign point-to-point operations of domestic air carriers. Air carriers are responsible for implementing TSA security requirements predominantly through TSA- approved security programs. These requirements for air carriers include, among other things, measures related to the screening of passengers, baggage, and cargo; training of employees in security and screening procedures; testing employee proficiency in screening; and access to aircraft. In addition, TSA may impose additional requirements in the form of security directives or emergency amendments when more immediate action on behalf of air carriers is necessary. Whereas security programs include standing regulatory requirements, directives are not intended to be permanent in nature and are expected to eventually be canceled, for example, should the threat or vulnerability cease to exist. If TSA determines that safety and the public interest require the incorporation of measures from directives into security programs, TSA will amend the programs after providing affected air carriers with notice and an opportunity for comment. TSA may impose directives based on the following: Threat information. Directives may focus on addressing specific threats. For example, in June 2017, TSA announced new security requirements in a directive on international aviation security that included, among other requirements, heightened screening of personal electronic devices larger than a cellphone for air carriers operating last point of departure flights to the United States. The directive was based on intelligence that terrorists were attempting to smuggle explosive devices in various consumer items (e.g., laptops). Events. Terrorist attacks, both successful and foiled, can also lead to the issuance of directives. For example, in response to a terrorist plot in July 2017, TSA issued security directives and emergency amendments in September 2017 requiring air carriers transporting cargo to the United States from last point of departure airports in Turkey to submit advance cargo data to DHS. Results of foreign airport assessments and air carrier inspections. TSA may issue directives requiring air carriers to implement security measures to account for vulnerabilities at foreign airports identified during TSA assessments (e.g., inadequate perimeter fencing). Through its foreign airport assessment program, TSA determines whether foreign airports that provide service to the United States maintain and carry out effective security measures. TSA does not have authority to impose or otherwise enforce security requirements at foreign airports and, therefore, often seeks to address security vulnerabilities it identifies by working with domestic and foreign air carriers to implement security measures to mitigate any identified vulnerabilities, as appropriate, while also working with the foreign governments to address the vulnerabilities. Measures required by directives to mitigate vulnerabilities identified during foreign airport assessments include screening passengers at the boarding gate and posting guards around parked aircraft. Air carriers must implement the security measures set forth in applicable directives in addition to other requirements imposed and enforced by TSA to remain compliant with TSA security requirements. However, TSA may approve the use of an alternative measure used in place of an existing measure required by a directive if TSA determines the alternative measure will achieve the required level of security. For example, an air carrier may request to use a different screening technology than specified in a directive, which TSA could approve if it determines the security outcome is commensurate, according to TSA officials. To ensure that air carriers meet applicable security requirements, including those imposed through directives, TSA conducts inspections of domestic and foreign air carriers. As of March 2019, there were 46 TSA directives related to air carrier operations at last point of departure airports in effect. These directives most often applied to passenger operations in specific foreign locations (see fig. 1). The characteristics of the 46 directives vary in a number of ways. For example: Of these directives, 25 were for foreign air carriers and 21 were for domestic air carriers. More than half of the current directives were issued prior to 2014, and most have a stated duration of 2 years or less. According to TSA officials and corroborated by our analysis, threat-driven directives, just over 60 percent of all directives, are generally in effect for about a year. Our analysis also shows that all directives with 3-year durations pertain to cargo-related threats, which TSA officials said are unlikely to change in the near term. However, foreign airport vulnerability- driven directives may have time horizons of about 2 years because, according to TSA officials, it could take foreign governments or airport authorities longer than 1 year to take corrective actions to address the deficiencies. About 30 percent of directives apply to air carrier operations worldwide and 70 percent apply to air carrier operations at airports in certain countries. Specifically, there are 33 directives that apply to specific countries in Asia, Africa, the Caribbean, Central America, or the Middle East. The security policies the directives address also vary and include passenger screening (23 directives), cargo (23), checked baggage (12), and aircraft security (12), among others. Although TSA generally issues directives with expiration dates, it may decide to renew the directive based on the threat or vulnerability. TSA has renewed or updated the 46 directives related to air carrier operations at last point of departure airports an average of five times through its review process. TSA has developed a process for reviewing directives that requires intra- agency coordination across TSA offices, and we found that the agency generally implemented this process in the 43 reviews it conducted from January 2017 to March 2019. However, TSA has not defined when or how it is to coordinate with air carriers and other industry stakeholders in reviewing directives. In addition, when TSA officials have coordinated with domestic and foreign air carriers, they have not documented the input air carriers provided. Further, TSA has not defined the process for cancelling or incorporating directives into air carrier security programs and certain directives are longstanding. TSA issued a management directive in 2012 and associated standard operating procedures in 2016 to guide the development and review of directives, among other policies. The management directive provides high-level TSA policy for the development, external coordination, and issuance of, among other things, directives. Further, the management directive describes the roles and responsibilities individual TSA offices have when developing directives, which are shown in table 1. The standard operating procedures describe the process that TSA is to apply to ensure that subject-matter experts coordinate to identify the problem and formulate solutions while obtaining appropriate stakeholder input from air carriers and their associations. TSA is to develop and review directives in accordance with steps identified in the TSA management directive and associated standard operating procedures, which include creating a team, developing a problem statement and options, drafting the policy document, and obtaining interoffice and management approval. Figure 2 shows how TSA is to apply this process to the development and review of directives. The directive development process can take weeks if, for example, the directive is merely expanding the applicable locations from an existing directive, or several months, as was the case of the broad-scoped worldwide directive regarding personal electronic devices and other international aviation security measures. Based on our review of TSA documents and meetings with TSA officials, TSA has generally adhered to its internal process to update or cancel directives in the 43 reviews conducted from 2017 to March 2019. Key steps of this process include the following: Initiate review process and create team. TSA initiates the directive review process because of (1) new intelligence, (2) feedback received from air carriers, (3) new information received from foreign airport assessments or air carrier inspections, or about 90 days before a directive is to expire, according to TSA officials. After initiation, TSA’s standard operating procedures state that all TSA offices that have equity in the security policy subject matter are to be invited to participate in the directive review team. TSA may also include other DHS components or government agencies in the team. According to our review of TSA documentation, in all 43 reviews TSA created an interoffice team that included Policy, Plans, and Engagement; Global Operations; and Chief Counsel. Our analysis also shows that at least 28 reviews included TSA Intelligence and Analysis. Further, certain teams reviewing vulnerability- driven directives included TSA field staff, such as TSA international industry representatives, TSA representatives, and regional operations center managers who have responsibility for the overall planning and conduct of assessments and air carrier inspections at foreign airports. In addition, according to TSA officials and corroborated by TSA documentation, they coordinated as needed with other federal partners— including DHS, the State Department, where TSA has a liaison embedded, and the National Security Council. Develop problem statement and options. To understand the environment and the nature of the threat, the team is to request a threat summary from TSA Intelligence and Analysis and, based on the intelligence summary, prepare a problem statement outlining the threat and vulnerability. The team is also to develop a proposed solution to the problem statement, and the team may decide to propose to either update or cancel the directive through an action memo written for TSA leadership. TSA officials stated that criteria for updating and canceling directives include whether the threat or vulnerability remains, intelligence, feedback from air carriers, and the results of air carrier inspections and airport assessments. Updates can result in a renewal of the policy with no significant changes or a revision to the security measures. All reviews developed a problem statement and documented proposed solutions in action memos that also included draft updates to the directives, as applicable. Further, Intelligence and Analysis officials stated that they provided the team with updated threat information and recommendations on whether the directive required a change or could be canceled. Obtain final approval and disseminate directive. If the team does not decide to cancel a directive, the completed drafts are to be routed to TSA offices for review and then to the administrator or assistant administrator for final approval. After final approval, TSA is to post worldwide directives to DHS’s Homeland Security Information Network. However, if the directive is country or region-specific TSA officials stated that they post an announcement on the network that the affected air carriers should contact their TSA international industry representatives for more information. According to our file review, TSA documented interoffice approval to the updates or cancellations for at least 41 of the 43 reviews. Further, the teams obtained administrator or assistant administrator approval in all 43 reviews. TSA headquarters officials and international industry representatives as well as air carrier representatives confirmed that directives are posted to the Homeland Security Information Network. TSA’s Standard Operating Procedures for Security Policy Development, Coordination, and Issuance requires TSA officials to obtain input from key stakeholders and representatives of affected regulated parties (e.g., air carriers), as appropriate, as shown in figure 2. However, the standard operating procedures do not explain what “as appropriate” means. Figure 3 shows a TSA international industry representative briefing foreign air carrier representatives on the 2017 international aviation security emergency amendment. TSA is also to incorporate key stakeholder input into the final draft as appropriate. TSA officials stated that they generally obtain mostly informal feedback from domestic air carriers and their associations during quarterly meetings with industry or through air carriers’ regular coordination with TSA international industry representatives. However, TSA officials stated that the extent to which they include air carriers and aviation associations in the review process varies. For example, TSA officials may share drafts of the directives with the air carriers for feedback or decide to only discuss the content of the directive at a high level, depending on the threat or vulnerability, air carriers involved, whether the changes needed are time-sensitive, and countries involved. While TSA’s standard operating procedures state that TSA is to coordinate with air carriers and other industry stakeholders, the feasibility of doing so when issuing or updating directives (particularly when the time frame is short and security measures must reach the industry rapidly due to a specific threat or recent event) is limited, according to TSA officials. These officials noted that engagement is more likely to take place when a directive is up for renewal or is being updated. Representatives from domestic air carriers confirmed that TSA has coordinated with them but also told us that the coordination has been inconsistent. Officials from four of the five domestic air carriers (three passenger and one all-cargo air carrier) and two associations representing domestic air carriers we met with told us that coordination with TSA on directives has improved since 2017. The air carrier representatives also stated that coordination with their TSA international industry representatives on directives was helpful. For example, all three domestic passenger air carriers we met with stated that TSA international industry representatives coordinated closely with them during the multiple revisions of the 2017 directive pertaining to international aviation security and that TSA made changes based on the feedback or approved alternative security measures they requested. However, representatives from both passenger and all-cargo domestic air carriers and an association that represents them identified ways that TSA coordination has been inconsistent when reviewing directives. For example, representatives from one of these air carriers stated that TSA sometimes coordinates with them when revising directives but generally seeks feedback from the same one or two air carriers that fly globally or operate out of the most last point of departure airports and does not always coordinate with air carriers that do not have a large global operation. In addition, a representative from another air carrier told us that TSA only coordinated with them after they insisted on being included in the process to revise a security directive; TSA did not proactively seek their input. Similarly, representatives from an association told us that TSA did not coordinate with them on the 2018 revision of a security directive issued to increase security requirements applied to cargo shipments originating in, transiting through, or transferring from Egypt until the association first reached out and that the process was not fully transparent. Although TSA verbally shared anticipated changes, representatives from the association were not clear what the new language would say or what it meant. While TSA sometimes includes domestic air carriers in the directive review process, foreign air carriers are generally not included, according to their representatives. Representatives from four of the five foreign air carriers we met with told us that they have a productive relationship with their TSA international industry representative and that TSA has made changes to emergency amendments based on alternative security measures they have requested. However, representatives from all five foreign air carriers noted that TSA generally does not solicit their input when reviewing emergency amendments. Representatives from the association that represents foreign air carriers told us that TSA’s coordination is sporadic; sometimes TSA would coordinate with industry when revising directives, and other times TSA would not—even though such coordination was necessary, in their view. For example, the representatives from this association stated that TSA has not consistently provided them with draft directives to review prior to issuance. These officials also stated that TSA coordination usually comes after they request being included in the process. All three international industry representative groups responsible for coordinating with foreign air carriers confirmed that TSA generally does not include their air carriers or the association that represents them when revising emergency amendments. Instead of coordinating with TSA, foreign air carriers may provide their input to domestic code-share partners, according to one TSA international industry group and representatives from a domestic air carrier. Representatives from both domestic and foreign air carriers and their associations identified negative effects of inconsistent coordination with TSA during the directive review process and stated that improved coordination would lead to more efficient and effective security measures. For example, according to representatives from six air carriers and two associations we met with, TSA did not include them at all or early enough in the directive review process. These carriers and associations identified a number of issues with the revised directives because of this lack of coordination, such as directives that were vague, less effective, or difficult for carriers to implement. For example, representatives from an association and one air carrier noted that cargo directives are not always effective because they do not fully account for how cargo moves around the world (e.g., shippers may transport cargo by truck from one country to another before loading it onto a U.S.-bound aircraft to avoid security measures specific to certain foreign airports). Representatives from two air carriers provided an example of vague requirements in directives related to aircraft cabin search procedures that has led to TSA international industry representatives and inspectors offering different interpretations of the same requirement. As a result, representatives of the air carriers said they do not know how to implement, and have at times been found in violation of, the requirement. In addition, according to representatives from one foreign air carrier, had TSA included them and other foreign air carriers early on in the review process, the changes to the 2017 emergency amendment pertaining to international aviation security measures would have been more efficient and effective. For example, within 3 months of issuance, TSA revised the directive twice to, among other things, change screening requirements for personal electronic devices (e.g., allowing for alternative screening methods). According to representatives from this air carrier, TSA could have reduced or eliminated the need for such revisions had TSA officials better coordinated with air carriers. Moreover, representatives from one association stated that when TSA does not involve them or the air carriers in the directive review process, TSA is missing an opportunity to implement the most effective security measures and may actually inadvertently create security vulnerabilities. TSA’s 2018 Administrator’s Intent states that TSA is to coordinate with external customers early and often for diverse perspectives and to develop trusted relationships to grow opportunities for mission success. Moreover, the Administrator’s Intent has a goal to effectively secure and safeguard the transportation system through contributions from a diverse and interconnected community of stakeholders, which includes actively seeking stakeholder input. The goal further states that coordinating with industry and other partners will enable timely and well-informed decisions and increase security effectiveness. In addition, TSA’s Standard Operating Procedures for Security Policy Development, Coordination, and Issuance requires TSA officials to obtain input from key stakeholders and representatives of affected regulated parties (e.g., air carriers), when developing the problem statement, developing options, and drafting the directive (as appropriate), as discussed above. TSA is also to incorporate key stakeholder input into the final draft as appropriate. TSA officials identified several reasons why coordination with air carriers and their association may be inconsistent. For example, TSA does not have guidelines that are specific as to how it is to coordinate with air carriers and their associations, and coordination can be difficult to define. In addition, the level of coordination with industry stakeholders is to some extent driven by the discretion of TSA administrators and assistant administrators. As the personnel in these positions change, so too does the level of expected coordination with industry. According to TSA officials, they cannot write specific requirements for each of the over 200 air carriers with U.S.-bound operations and necessarily must choose which air carriers to seek input from. In addition, TSA officials noted that they coordinate with one or two domestic air carriers that chair the security committee within the association that represents both passenger and all-cargo air carriers. Further, TSA officials may decide not to share much information at all with air carriers owned and operated by certain foreign governments because of potential security concerns. Although TSA’s Standard Operating Procedures for Security Policy Development, Coordination, and Issuance require TSA officials to obtain input from air carriers and key stakeholders, the current procedures do not provide clear guidance on the circumstances under which coordination should occur. Better defining (e.g., develop guiding principles) how to coordinate with air carriers and other stakeholders during the review of directives and implementing such guidance would help TSA ensure that it more consistently coordinates with air carriers over time, addresses air carriers concerns, and issues directives that enable air carriers to effectively secure their operations against the identified threats or vulnerabilities. When TSA officials have coordinated with domestic and foreign air carriers, they have not documented the input air carriers provided. Based on our review of the 43 directive reviews TSA conducted from 2017 to March 2019, TSA officials did not document the input they have received from air carriers. TSA did provide us with emails and appointments with associations and air carriers to obtain their input during revisions to the 2017 directives pertaining to international aviation security, but this documentation did not capture a summary of the discussions or stakeholder concerns. TSA’s Standard Operating Procedures for Security Policy Development, Coordination, and Issuance requires that stakeholder and regulated party input be documented and include the entity consulted, date, location, and a brief summary of the discussion and specific stakeholder input, to include any concerns. In addition, Standards for Internal Control in the Federal Government states that effective documentation assists in management’s design of internal control by establishing and communicating who, what, when, where, and why of internal control execution to personnel. Documentation also provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel, as well as a means to communicate that knowledge as needed to external parties, such as external auditors. According to TSA headquarters officials, TSA does not document its coordination with air carriers and their associations because the feedback that it solicits and receives from air carriers and associations is mostly informal. TSA officials stated that for the 2017 directives pertaining to international aviation security, for example, they had to adjudicate many requests through dialogue with air carriers and their associations but the discussions were not documented, as it would have been too burdensome. However, TSA officials stated that most directives do not have the broad scope or apply to as many air carriers as the 2017 directive pertaining to international aviation security. Documenting the input provided by air carriers during the directive review process, even if the input is deemed informal, would better ensure that TSA provides insight on shared air carrier views or concerns, and retains knowledge about who, what, when, where, and why coordination occurred. In addition, TSA would be able to reference documented information for decision-making purposes, which could help ensure that TSA is consistently coordinating with air carriers during the review of directives and addressing their concerns. In general, directives are not meant to be permanent, and TSA has canceled some of them in recent years. Specifically, of the total of 78 directives related to air carrier operations at last point of departure airports in effect at some point from fiscal year 2012 to March 2019, 46 remain current while 32 were canceled for a variety of reasons (see fig. 4). One reason TSA might cancel a directive is if the agency incorporates the directive’s security measures into air carrier security programs. When this occurs, TSA initiates the directive review process and the directive will be canceled simultaneously with the security program change taking effect, according to TSA officials. TSA officials stated that they follow a similar process when they cancel a directive and include that directive’s security measures in a new directive. As a result, there is no lapse in security measure requirements. Although TSA has canceled some directives, others are longstanding. According to TSA officials, they have incorporated threat-based directives into air carrier security programs but not foreign airport vulnerability- based directives because the latter are site-specific and would not apply to all air carriers. However, as shown in figure 5, more than half (25 of 46) of directives related to last point of departure airports have been in effect for more than 5 years, and about one quarter (12) were threat- based. According to TSA officials, the threat pertaining to these directives still exists. Further, certain security measures predate the issuance of the directives that remain in effect. As shown in figure 4, the security measures within one-third (12) of the canceled directives were incorporated into new directives. According to TSA officials, there are security measures in certain directives that predate the creation of TSA in 2001. Representatives of the air carriers and associations we met with identified directives that have, in their view, persisted for too long, which can create redundant and confusing security requirements. Specifically, half of the air carrier representatives we met with told us that some directive requirements conflict with requirements in the air carriers’ security programs, are redundant, or could be incorporated into the security programs. According to representatives from one air carrier, without an exit strategy or plan to help TSA determine when it can cancel directives, the directives may be in effect beyond their useful time frame and are in some instances outdated or redundant. For example, representatives from this air carrier stated that directives require air carriers to identify baggage in a manner to thwart an attack in which passengers check their baggage with explosives in it but do not board the plane. However, given advancements in screening technology, such security measures are no longer required, according to these representatives. In addition, according to representatives from another air carrier, there are often conflicts between the directives and the security programs, which may cause confusion and sometimes misinterpretation of security requirements. Further, representatives from a third air carrier and one association also told us that there is value in incorporating directives into air carrier security programs because it removes the uncertainty involved, and air carriers can better plan for security requirements. TSA headquarters and field officials told us that there are directives that can be incorporated into air carrier security programs. For example, TSA headquarters officials stated that they have identified several such directives, including a 2012 emergency amendment and a 2017 security directive and emergency amendment related to passenger international aviation security; a 2014 security directive regarding the handling of items containing liquids, aerosols, and gels (e.g., personal hygiene products) brought into the aircraft cabin by passengers; and security directives and emergency amendments pertaining to cargo from certain Middle Eastern and African countries. Further, three groups of TSA international industry representatives told us that TSA should incorporate certain directives into security programs. Further, they stated that certain directives overlap, have outdated requirements, or contradict each other. For example, they highlighted overlap between requirements found in the 2012 emergency amendment and 2017 emergency amendments related to passenger international aviation security, as well as the air carriers’ security programs. Both emergency amendments have security requirements pertaining to passenger screening, aircraft security, and catering. According to one group of international industry representatives, there is confusion among themselves and air carriers over which emergency amendment supersedes the other. Although TSA officials have identified directives that they may be able to cancel by incorporating them into security programs, TSA does not have a defined process for doing so. TSA’s standard operating procedures provide step-by-step guidance for issuing new or revised security requirements through the directive review process, but it does not provide similar guidance for incorporating directives into security programs. Specifically, TSA officials have not resolved how they will accomplish key steps in incorporating certain long-standing directives into the security programs. For example, TSA officials stated that they are considering incorporating a 2011 security directive and emergency amendment pertaining to security measures for cargo from Yemen. However, TSA officials are unclear how they might request comments from air carriers because not all air carriers transport cargo from that country. Further, TSA officials stated that they have not determined whether or how they might incorporate vulnerability-driven directives into security programs. In addition, according to TSA officials, TSA’s reorganizations, personnel changes, and limited staff availability have delayed efforts to incorporate longstanding directives into security programs. TSA officials stated they have been attempting to incorporate the 2012 international aviation security emergency amendment into the security programs for foreign air carriers for the past 10 years. Specifically, in 2012 TSA consolidated over 20 worldwide threat-based emergency amendments issued from 2001 to 2012 into one emergency amendment covering a number of different types of security measures with the plan to next incorporate it into the security program, according to TSA officials. However, since that time, TSA has renewed the emergency amendment 13 times, each time with a new expiration date. TSA officials stated that it is easier to renew directives to ensure that the security measures remain in place than to incorporate them into security programs. Despite these challenges, TSA officials stated that they are mapping out how to incorporate certain directives into air carrier security programs. Further, they may be able to develop the changes to the programs and draft action memos for the TSA Administrator to approve by the end of 2019, according to these officials. As of July 2019, TSA officials had identified the directives it first planned to migrate into security programs and begun the process. However, these officials had not yet finalized plans for doing so. TSA Management Directive 2100.5 provides high-level TSA policy for the development, external coordination, and issuance of security programs and directives. It states that during the creation of all directives (i.e., security directives and emergency amendments), a sunset date will be assigned. This date is to serve as the date where a decision will be made by the agency to either cancel the directive or convert it into a security program change. Factors for this decision will include a comprehensive intelligence review, assessment of risk-based relevance, and operator performance and compliance. According to the management directive, this lifecycle analysis will ensure that directives are not permanent in nature and that the security program change process is routinely used as the vehicle for long-term regulatory requirements. However, the management directive does not preclude continuation of a directive, and TSA may decide to renew the directive, as appropriate. Further, according to the standard operating procedures associated with this management directive, the goal of the policy development process is to enhance TSA’s ability to make sound and timely policy decisions. In addition, Standards of Internal Control in the Federal Government states that management should define objectives clearly to enable the identification of risks and define risk tolerances. This involves clearly defining what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement. By defining the process for cancelling or incorporating directives into security programs, including expected time frames, and taking actions to implement this process, as applicable, TSA could better ensure that it clarifies and streamlines the security requirements for air carriers that operate at last point of departure airports in a timely manner and in a way that uses limited resources efficiently. Further, taking these steps would help ensure that requirements in directives that should become permanent are incorporated into security programs. Given that terrorist groups continue to target international aviation, it is paramount that TSA effectively update and issue security directives and emergency amendments in response to threats. For the approximately 300 airports in foreign countries offering last point of departure flights to the United States, TSA may issue directives when immediate action on behalf of air carriers is necessary and has developed a review process for these directives, but it has not defined the circumstances under which TSA is to coordinate with air carriers and other industry stakeholders throughout the process. Better defining (e.g., develop guiding principles) how TSA is to coordinate with air carriers and implementing such guidance would help TSA ensure that it more consistently coordinates with air carriers over time, air carriers concerns are addressed, and it issues directives that enable air carriers to effectively secure their operations against the identified threats or vulnerabilities. In addition, documenting the input provided by air carriers during the directive review process would help TSA better ensure that it captures stakeholder views or concerns and retains knowledge about who, what, when, where, and why coordination occurred. TSA would also be able to reference documented information for decision-making purposes, which could help ensure that TSA is consistently coordinating with air carriers during the review of directives and addressing their concerns. Further, TSA has not always canceled longstanding directives or incorporated them into air carrier security programs. However, according to TSA Management Directive 2100.5, directives are not meant to be permanent. Recognizing that threat-driven exigent circumstances may preclude consultation, better defining the process for cancelling or incorporating directives into security programs, including expected time frames, and taking actions to implement this process, as applicable, could better ensure that TSA clarifies and streamlines the security requirements for air carriers that operate at last point of departure airports in a timely manner and in a way that uses limited resources efficiently. We are making the following three recommendations to TSA: The Administrator of TSA should ensure that the Assistant Administrator for Policy, Plans, and Engagement and the Assistant Administrator for Global Operations better define (e.g., develop guiding principles) how TSA is to coordinate with air carriers and other stakeholders during the review of security directives and emergency amendments, and implement such guidance (Recommendation 1). The Administrator of TSA should ensure input provided by air carriers and other stakeholders is documented during the security directive and emergency amendment review process (Recommendation 2). The Administrator of TSA should ensure that the Assistant Administrator for Policy, Plans, and Engagement defines a process for cancelling or incorporating security directives and emergency amendments into security programs, including time frames, and take action to implement this process, as applicable (Recommendation 3). We provided a draft of our report to DHS for review and comment. In written comments, which are included in appendix I and discussed below, DHS concurred with our three recommendations and described actions taken to address them. DHS also provided technical comments, which we have incorporated into the report, as appropriate. With respect to our first recommendation that TSA better define how to coordinate with air carriers and other stakeholders during the review of security directives and emergency amendments, and implement such guidance, DHS stated that TSA is developing a process for more formal and consistent coordination with air carrier and industry association stakeholders. With regard to our second recommendation that TSA document the input provided by air carriers and other stakeholders during the security directive and emergency amendment review process, DHS stated that TSA will require international industry representatives and other TSA officials to keep records of all communications related to review and feedback on directives. TSA officials plan to incorporate substantive feedback into action memos associated with the review of directives. With respect to our third recommendation that TSA define a process for cancelling or incorporating security directives and emergency amendments into security programs, DHS stated that TSA will establish milestones at which TSA will conduct a formal review to determine if long- standing directives should be consolidated into a security program or otherwise cancelled. We are sending this report to the appropriate congressional committees and to the acting Secretary of Homeland Security. In addition, this report is available at no charge on the GAO website at http://gao.gov. If you or your staff members have any questions about this report, please contact William Russell at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. William Russell (202) 512-8777 or russellw@gao.gov. In addition to the contact above, Kevin Heinz (Assistant Director), Paul Hobart (Analyst-in-Charge), Charles Bausell, Michele Fejfar, Sally Gilley, Eric Hauswirth, Tom Lombardi, and Adam Vogt made key contributions.", "summary": "Approximately 300 airports in foreign countries offer last point of departure flights to the United States. When threat information or vulnerabilities at foreign airports indicate an immediate need for air carriers to implement additional security measures, TSA may issue new or revise existing security directives (for domestic air carriers) and emergency amendments (for foreign air carriers). The TSA Modernization Act includes a provision for GAO to examine TSA's review process for directives that apply at last point of departure airports. This report (1) identifies key characteristics of the TSA directives and (2) assesses TSA's process to review directives. GAO reviewed TSA policies and procedures, analyzed TSA program information, and interviewed TSA officials and representatives from a nongeneralizable sample of 10 air carriers, selected to represent carriers with high numbers of U.S.-bound flights, and three industry associations. As of March 2019, there were 46 Transportation Security Administration (TSA) security directives and emergency amendments (i.e., directives) in effect related to air carrier operations at foreign airports. Twenty-eight directives addressed threats (e.g., explosives in laptops) and 18 pertained to vulnerabilities identified at foreign airports (e.g., inadequate perimeter fencing). TSA reviews directives, but its process does not fully define how to coordinate with industry representatives and TSA has not incorporated the security measures of many longstanding directives into air carrier security programs in accordance with TSA policy. Representatives from four domestic air carriers stated that coordination with TSA on directives has improved. However, representatives from six air carriers and two associations indicated that TSA has issued revised directives that are vague or difficult to implement—which, for example, contributed to TSA officials offering different interpretations of aircraft cabin search requirements—because TSA did not sufficiently include them in the review process. Better defining how TSA coordinates with air carriers and other stakeholders would help ensure that TSA issues directives that enable air carriers to effectively secure their operations against the identified threats or vulnerabilities. In addition, when TSA officials have coordinated with air carriers, they have not documented the input provided. Documenting the input could help ensure that TSA is consistently addressing air carrier concerns and retaining knowledge about who, what, when, where, and why coordination occurred. Further, TSA policy states that directives are not intended to be permanent and are expected to eventually be canceled or incorporated into security programs. GAO analysis found that TSA issued more than one half (25) of the directives prior to 2014, meaning they have been in effect for more than 5 years. Several have been in effect for more than 10 years (see figure). As of July 2019, TSA officials had begun the process to migrate directives into security programs as deemed appropriate, but had not yet finalized their plans for doing so. Defining the process for incorporating directives into security programs, including expected timeframes, and taking actions to implement this process, as applicable, could better ensure that TSA clarifies and streamlines security requirements in a timely manner. GAO recommends that TSA (1) better define how to coordinate with air carriers when reviewing directives, (2) document air carrier input, and (3) define a process, including time frames, for cancelling or incorporating security measures from directives into security programs. DHS concurred with all three recommendations.", "document_type": "gao"}
{"report": "Available federal law enforcement data show a range of threats and assaults against the four federal land management agencies’ employees in fiscal years 2013 through 2017. The severity of these incidents ranged from threats conveyed over the telephone to attempted murder and included an incident in which an employee was stabbed outside a federal building. The number of incidents of threats and assaults varied by agency. For example, for fiscal years 2013 through 2017 BLM data included 88 incidents of threats and assaults against BLM FWS data included 66 incidents of threats and assaults against FWS Forest Service data included 177 incidents of threats and assaults against Forest Service employees; and Park Service data included 29 incidents of threats and assaults against Park Service employees. Further, FBI data for fiscal years 2013 through 2017 show that the FBI initiated under 100 domestic terrorism investigations into potential threats to federal land management agencies. Our analysis of the FBI data showed that the majority of the domestic terrorism investigations involved BLM. Additionally, the majority involved individuals motivated by anti- government ideologies. For example, the FBI investigated one case in which a BLM law enforcement officer received more than 500 harassing phone calls and several death threats after a subject posted personal information about the officer on the social media platform Twitter. However, the number of actual threats and assaults against federal land management employees is unclear and may be higher than what is represented in available data, because not all incidents of threats and assaults against land management agency employees are captured in the agencies’ databases. There are several reasons why this may be the case. Specifically, some incidents of threats and assaults are investigated by local or state law enforcement and may be recorded in their data systems rather than in the land management agencies’ systems. Additionally, officials from two agencies we interviewed said that when a single incident involved multiple offenses, the less serious offenses are unlikely to be recorded in the data system and, therefore, the entirety of what occurred may not be captured. Further, land management agency employees do not always report all incidents of threats. For example, some field unit employees said that in certain circumstances, they consider receiving threats as a normal part of their job. Some officials also described being threatened while off duty, such as being harassed in local stores or being monitored at their home, and they said that in some cases they did not report the incident because it was a common occurrence. However, even in more high-profile incidents, agency officials told us that employees may not always report threats to agency law enforcement. For example, agency officials we interviewed cited specific incidents around the time of the 2016 armed occupation of FWS’s Malheur National Wildlife Refuge that they did not necessarily report to their agency’s law enforcement. These incidents included individuals holding anti-government beliefs who followed a teenage girl wearing a BLM shirt around the local grocery store and threatened to burn her house down, and agency employees who had shots fired over their heads while working in the field. According to officials at two agencies, many employees were traumatized by the Malheur occupation and some did not return to work, including some who transferred to other agency field units. Federal land management agencies use various approaches to protect their employees from threats and assaults, including deploying agency law enforcement officers to protect employees and resources and building relationships with external law enforcement entities and the public. Specifically, when necessary, agencies deploy additional law enforcement officers to assist their local officers. For example, during the armed occupation of the Malheur National Wildlife Refuge, FWS officials reported deploying FWS law enforcement officers from around the country to field units in western states to provide additional security for FWS employees. Agency officials we interviewed also told us that they build relationships with local, state, and other federal agency law enforcement entities to help protect employees and resources in the field and to assist with coordinating law enforcement responses. Such relationships are important because not all field units have a law enforcement officer, and those that do often rely on local law enforcement for assistance in responding to incidents of threats or assaults against agency employees. For example, officials we interviewed at a field unit in Nevada stated that during a high-profile court case involving the agency, the Las Vegas Metropolitan Police Department kept a patrol car outside the field unit for several days to help ensure field unit employees’ safety. Finally, officials at several field units we visited stated that their law enforcement officers are focused on educating, rather than policing, visitors. Agency officials we interviewed cited several factors that can affect their ability to protect employees. Specifically, agency officials noted that employees are required to interact with the public as part of their official duties and may wear uniforms, which makes them easily recognizable and can put them at risk of being threatened or assaulted. (See figure 1.) Additionally, agency officials stated that it can be difficult to protect employees because, as part of their field work, employees may be dispersed across hundreds of miles of federal lands and may be located hours or days away from the nearest agency law enforcement officer. For example, as of fiscal year 2018, BLM had 194 field law enforcement officers to cover the 245 million acres of land managed by BLM. Further, the number of agency field law enforcement officers at all four land management agencies declined from fiscal year 2013 through fiscal year 2018. For example, BLM experienced a decrease of 9 percent, while the Forest Service experienced a decrease of 22 percent, the largest decrease among the four agencies. Finally, agency officials we interviewed said that the risk to employee safety posed by individuals holding anti-government sentiments can be unpredictable and that incidents of threats and assaults against employees by such individuals are generally sporadic. The four federal land management agencies have completed some but not all of the facility security assessments on their occupied federal facilities as required by the ISC Standard. Agency officials cited various reasons for not doing so, including lack of resources, training, and expertise. Not complying with the ISC Standard’s requirement to complete facility security assessments on all occupied facilities could leave federal agencies exposed to risks in protecting their employees and facilities. While FWS has a plan to complete its assessments, BLM, the Forest Service, and the Park Service do not. Specifically: FWS. FWS has conducted five facility security assessments on its approximately 465 occupied facilities. According to FWS headquarters officials, FWS employees have limited physical security expertise to conduct facility security assessments; therefore, the agency has developed a plan to meet the ISC Standard’s requirement using contractors. BLM. BLM has conducted 21 facility security assessments on its approximately 280 occupied facilities, but officials do not know when they will complete the remaining assessments and do not have a plan to do so. Forest Service. The Forest Service has conducted at least 135 facility security assessments on its approximately 1,135 occupied facilities, but officials do not know when they will complete the remaining assessments and do not have a plan for doing so. Park Service. The Park Service has conducted at least 148 facility security assessments on its approximately 1,505 occupied facilities, but officials do not know when they will complete the remaining assessments and do not have a plan to do so. The ISC Standard requires that agencies conduct assessments using a methodology that meets, among other things, two key requirements: (1) consider all of the undesirable events (e.g., arson and vandalism) identified in the ISC Standard as possible risks to facilities, and (2) assess the threat, vulnerability, and consequence for each of these events. The Forest Service’s methodology meets these two requirements and utilizes an ISC-compliant facility security assessment methodology developed by the U.S. Department of Agriculture. The Park Service’s methodology partially meets the requirements because it does not include a step to assess the consequences of specific undesirable events, as required by the ISC Standard. BLM and FWS have not yet established methodologies for conducting facility security assessments, although officials we interviewed from each agency stated that they intend to develop an ISC- compliant methodology. Specifically, BLM officials told us that they plan to hire a security manager who will develop an assessment methodology but did not know when the manager would be hired. FWS officials we interviewed provided a high-level description of what they expected to be included in their new methodology. However, FWS’s description did not indicate that the agency would evaluate the consequences of specific undesirable events, as required by the ISC Standard. Without developing a plan for conducting all of the remaining facility security assessments and using a methodology that complies with ISC requirements, agencies may not identify the risks their facilities face or identify the countermeasures they could implement to mitigate those risks. Based on these findings, we made a total of six recommendations to the four land management agencies, including that BLM, the Forest Service, and the Park Service each develop a plan to conduct all required facility security assessments agency-wide; The Park Service update its facility security assessment methodology to address the consequences of specific undesirable events in order to comply with requirements in the ISC Standard; and BLM and the Forest Service each develop facility security assessment methodologies that comply with requirements in the ISC Standard. The four land management agencies generally concurred with our recommendations and provided examples of actions they plan to take to address our recommendations, including revising policies and developing new tools, training, and data system modules. Chairwoman Haaland, Republican Leader Young, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Anne-Marie Fennell at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff members who made key contributions to this testimony are Casey L. Brown (Assistant Director), Tanya Doriss (Analyst in Charge), Charles W. Bausell, Charles A. Culverwell, John W. Delicath, Emily E. Eischen, Cindy K. Gilbert, Richard P. Johnson, Vanessa E. Obetz, Dan C. Royer, and Breanna M. Trexler. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "A 2014 government report predicted that the rate of violent domestic extremist incidents would increase. In recent years, some high-profile incidents have occurred on federal lands, such as the armed occupation of a FWS wildlife refuge in 2016. Federal land management agencies manage nearly 700 million acres of federal lands and have law enforcement divisions that protect their employees and secure their facilities. This testimony summarizes GAO's September 2019 report on how land management agencies protect their employees and secure their facilities (GAO-19-643). In that report, GAO examined, among other things, for the four federal land management agencies, (1) what is known about the number of threats and assaults against their employees and (2) the extent to which agencies met federal facility security assessment requirements. For the report, GAO analyzed available government data on threats and assaults; examined agencies' policies, procedures, and documentation on facility security assessments; compared the agencies' methodologies against ISC requirements; and interviewed land management agency, ISC, and FBI officials. Data from the four federal land management agencies—the Forest Service within the U.S. Department of Agriculture and the Bureau of Land Management (BLM), Fish and Wildlife (FWS), and National Park Service (Park Service) within the Department of the Interior—showed a range of threats and assaults against agency employees in fiscal years 2013 through 2017. For example, incidents ranged from telephone threats to attempted murder against federal land management employees. However, the number of actual threats and assaults is unclear and may be higher than what is captured in available data for various reasons. For example, employees may not always report threats because they consider them a part of the job. Federal Bureau of Investigation (FBI) data for fiscal years 2013 through 2017 also showed that the FBI initiated under 100 domestic terrorism investigations into potential threats against federal land management agencies. The majority of these FBI investigations involved BLM, and the majority involved individuals motivated by anti-government ideologies. The four federal land management agencies have not completed all of the facility security assessments on their occupied federal facilities as required by the Interagency Security Committee (ISC). Officials at the four agencies said that either they do not have the resources, expertise, or training to conduct assessments agency-wide. FWS has a plan to complete its assessments, but BLM, the Forest Service, and the Park Service do not. Such a plan could help these agencies address the factors that have affected their ability to complete assessments. The ISC also requires that agencies conduct assessments using a methodology that meets, among other things, two key requirements: (1) consider all of the undesirable events (e.g., arson and vandalism) identified as possible risks to facilities, and (2) assess the threat, vulnerability, and consequence for each of these events. The Forest Service's methodology meets these two requirements and the Park Service's methodology partially meets the requirements, but BLM and FWS have not yet established methodologies for conducting facility security assessments. Without developing a plan for conducting all of the remaining facility security assessments and using a methodology that complies with ISC requirements, agencies may not identify the risks their facilities face or identify the countermeasures—such as security cameras or security gates—they could implement to mitigate those risks. In its September 2019 report, GAO made six recommendations: that BLM, the Forest Service, and the Park Service develop a plan for completing facility security assessments; and that BLM, FWS, and the Park Service ensure their facility security assessment methodologies comply with ISC requirements. The agencies generally concurred with the recommendations.", "document_type": "gao"}
{"report": "Transnational criminal organizations and terrorist organizations use a variety of money laundering schemes to disguise the origin and destination of their illicit proceeds and integrate their assets in legitimate financial entities. According to the U.S. government’s 2018 National Strategy for Combating Terrorist and Other Illicit Financing, the criminal activities in the United States that generate the largest share of illicit proceeds for laundering are fraud, drug trafficking, human smuggling, human trafficking, organized crime, and government corruption. FATF has identified three primary methods of money laundering: the laundering of money through the financial system, the physical movement of money (such as through cash couriers), and TBML. FATF has defined TBML as “the process of disguising the proceeds of crime and moving value through the use of trade transactions in an attempt to legitimize their illicit origins.” The volume of international trade is significant and has grown over time. According to the World Trade Organization, in 2018, there was $19.67 trillion in international merchandise trade and $5.63 trillion in international services trade. Although international trade offers many economic opportunities for the United States and other countries around the world, the number and complexity of international trade transactions present a number of risks and vulnerabilities that make them susceptible to abuse by criminal and terrorist organizations. For example, the large volume of international trade complicates detection of individual illicit transactions. In the United States alone, on a typical day in fiscal year 2019, almost 79,000 containers and $7.3 billion worth of goods entered the country through ports of entry, according to U.S. Customs and Border Protection (CBP). Similarly, different studies have noted that the increasingly complex nature of international trade—with the movement of goods and services around the world and the use of various financing and payment structures—makes detecting suspicious transactions difficult. TBML schemes can involve misrepresenting the price, quantity, or type of goods or services in trade transactions, but other types of TBML schemes, such as the Black Market Peso Exchange, do not need to rely on this type of misrepresentation. In misrepresentation schemes, the parties involved in the trade transaction may under or over invoice goods or services; issue multiple invoices for the same goods or services; provide more or less goods or services than the declared amount, including in some cases providing no goods or services; or falsely describe the types of goods or services provided. Through these types of misrepresentation, value can be transferred from one party to another and the illicit origins of criminal proceeds obscured. In a hypothetical TBML scheme involving the misrepresentation of the price of goods, a criminal organization in Country A needs to launder the proceeds from its criminal activity and move these proceeds to Country B. To accomplish this, the criminal organization will use the illicit proceeds to purchase 100,000 cell phones worth $100 each. The criminal organizations will then make arrangements to export the 100,000 cell phones to a co-conspirator in Country B. However, the criminal organization in Country A, will fraudulently invoice the cell phones at $10 each rather than $100 each. Thus, the co-conspirator in Country B pays a total of $1 million for the cell phones, rather than their true value of $10 million. The co-conspirator then sells the cell phones at their true market value of $10 million in Country B resulting in the criminal organization having successfully transferred $9 million in value from Country A to Country B through TBML. Figure 1 illustrates how such a price misrepresentation scheme works. Similarly, the criminal organization can transfer value through misrepresentation of the quantity or type of goods being exported. For example, the criminal organization can invoice its co-conspirator for 50,000 cell phones, but actually ship 100,000 phones, or it can claim that it is shipping different, lower value items such as USB flash drives. Under a hypothetical Black Market Peso Exchange scheme, a criminal organization operating in Country A, which uses dollars, will take the dollar proceeds of its criminal activities to a currency broker’s representative that has access to currency reserves from Country B (pesos). At the same time, in Country B, an import company will contact the currency broker seeking dollars to pay for goods that it wishes to import from Country A. The currency broker uses the dollars provided by the criminal organization to pay exporters in Country A on behalf of the importer in Country B. The importer receives and sells the goods in Country B and pays the currency broker in pesos. The currency broker then pays the criminal organization in Country B in pesos, completing the transfer of its proceeds. Thus, the criminal organization has successfully shifted the value of its proceeds from Country A to Country B without having to physically move money, or transfer funds through the banking system, from Country A to Country B. Figure 2 shows such a Black Market Peso Exchange scheme involving the United States and Colombia. TBML differs from other crimes, such as trade or customs fraud, that may occur in connection with trade and the movement of goods, according to Treasury officials. Organizations and individuals involved in TBML exploit vulnerabilities in international trade to move value across international borders in an attempt to disguise the origin, nature, or source of illicit proceeds, which may derive from a variety of predicate crimes. According to Treasury officials, while offenses like smuggling and fraud may resemble TBML, they differ in purpose. For example, smugglers attempt to evade detection or the payment of custom fees, duties or taxes while moving legitimate, illicit, or restricted goods across borders. Similarly, in frauds involving the (purported) purchase or sale of goods, one of the parties to the transaction seeks to deceive another one for financial gain. In TBML, the scheme may be accomplished using fraudulent documents, such as false invoices, but this is not a necessary part of the scheme, nor does it alone represent TBML. In TBML schemes that involve misrepresenting the price, quantity, or type of goods, both the buyer and seller normally understand that the goods shipped or funds paid may differ from what is stated in the supporting documents. Within the United States, a number of laws and regulations are used to combat TBML. The Bank Secrecy Act, which was passed in 1970, and implementing anti-money laundering (AML) regulations provide the legal and regulatory framework for preventing, detecting, and deterring money laundering in the United States. The Bank Secrecy Act regulations generally require banks and other financial institutions, such as money service businesses, securities broker-dealers, and certain types of insurance companies, among others, to, for example, collect and retain various records of customer transactions, verify customers’ identities at the time of account opening, maintain AML programs, and report suspicious transactions or cash transactions over a certain amount. In addition, the Trade Facilitation and Trade Enforcement Act of 2015, signed into law in 2016, addressed trade facilitation and trade enforcement issues such as import safety, the protection of intellectual property, and the prevention of the evasion of duties, among other things. Further, individuals can be prosecuted under U.S. law, such as section 1956 of title 18 of the United States Code, for money laundering, including TBML schemes. For example, under section 1956, defendants can be prosecuted for money laundering activities, including those involving falsely classifying goods or entering goods by means of false statements. Within the U.S. government, a number of agencies play a role in working with international partners to combat money laundering more broadly, as well as TBML specifically. These include DHS, DOJ, State, and Treasury and their component agencies and offices. DHS: Within DHS, ICE’s Homeland Security Investigations (HSI) investigates financial crimes and money laundering cases, including those involving TBML. HSI has established a TTU that seeks to identify global TBML trends, provide investigation support to HSI and other law enforcement efforts, and conduct ongoing analysis of trade data provided through partnerships with TTUs that it has helped establish in other countries. CBP is responsible for enforcing U.S. trade laws, facilitating compliant trade, collecting revenue, and protecting the U.S. economy and consumers from harmful imports and unfair trade practices. As part of its mission, CBP conducts targeting of high-risk shipments that may involve trade violations, including violations linked to TBML schemes. DOJ: The Drug Enforcement Administration (DEA) and the Federal Bureau of Investigation both conduct investigations of criminal organizations that may use TBML to launder their illicit proceeds. In addition, the DOJ Criminal Division’s Money Laundering and Asset Recovery Section and U.S. Attorney’s Offices throughout the country prosecute cases involving money laundering crimes, including TBML schemes. State: State’s Bureau of International Narcotics and Law Enforcement Affairs (INL) leads State’s AML technical assistance efforts with international partners. In this role, INL works in global and regional forums to promote the implementation of international AML standards. INL also funds AML assistance programs in countries around the world. Finally, INL publishes the annual International Narcotics Control Strategy Report, which includes an analysis of countries identified as “major money laundering countries.” In addition to INL, State’s Bureau of Economic and Business Affairs and Bureau of Counterterrorism also play a role in State’s AML and countering the financing of terrorism (CFT) efforts. Treasury: Treasury’s Financial Crimes Enforcement Network (FinCEN) collects, analyzes, and disseminates the financial intelligence information it collects pursuant to the Bank Secrecy Act to support efforts to combat financial crime, including money laundering. FinCEN is responsible for administering the Bank Secrecy Act and coordinating with federal and state regulatory agencies on AML/CFT efforts. Additionally, FinCEN serves as the Financial Intelligence Unit (FIU) of the United States, which entails gathering and analyzing Suspicious Activity Reports (SAR) and other financial information relevant to money laundering, terrorist financing, and other financial crimes, as well as disseminating the results of this analysis to law enforcement and other competent authorities. A number of other Treasury agencies and offices also play a role in efforts to combat money laundering, including TBML. For example, Treasury’s Office of Technical Assistance (OTA) provides assistance to partner countries to help strengthen their efforts to combat economic crimes. Treasury’s Office of Terrorist Financing and Financial Crimes is the policy coordination office for illicit finance and develops and implements U.S. government strategies to combat all forms of illicit finance domestically and internationally. Internal Revenue Service Criminal Investigation investigates tax crimes and other financial crimes, including those associated with TBML schemes. It has lead authority for investigating criminal violations of the Bank Secrecy Act. Internationally, the U.S. government participates in a number of bodies that address issues related to TBML, including the Egmont Group, FATF, UNODC, and the WCO. The Egmont Group: The Egmont Group, formed in 1995, is composed of FIUs from 164 jurisdictions. The organization seeks to foster information exchange among its members to support efforts to combat money laundering and terrorist financing. In addition, the Egmont Group provides training and technical assistance to its member FIUs. FinCEN represents the United States at the Egmont Group. The Egmont Group’s Secretariat is located in Canada. FATF: FATF is an intergovernmental body, formed in 1989, that sets internationally recognized standards for developing AML/CFT regimes and assesses the ability of member jurisdictions to meet these standards. In addition, FATF works to identify specific money laundering methods and promotes international cooperation in disrupting and dismantling those money laundering schemes. FATF’s membership includes 37 jurisdictions and two regional organizations—the European Commission and the Gulf Cooperation Council. Treasury’s Office of Terrorist Financing and Financial Crimes heads the United States delegation to FATF. The FATF Secretariat is located in Paris, France. UNODC: UNODC is an agency within the United Nations, formed in 1997, that works to combat illicit drugs and other international crime in more than 150 countries throughout the world. As part of its mandate, UNODC carries out the Global Program against Money Laundering, Proceeds of Crime and the Financing of Terrorism. Through this program, UNODC seeks to strengthen the ability of United Nations member states to implement measures against money laundering and the financing of terrorism and to assist them in detecting, seizing, and confiscating illicit proceeds. State is the lead agency representing the United States at UNODC. UNODC is headquartered in Vienna, Austria and has field offices in 20 countries, as well as liaison offices in New York and Brussels, Belgium. WCO: The WCO, established in 1952, is an intergovernmental body whose mission is to enhance the effectiveness and efficiency of customs administrations around the world and to help them in their dual role of facilitating international trade while also promoting security. WCO’s membership includes customs agencies from 183 countries. CBP is the lead agency representing the United States at WCO. The WCO’s Secretariat is located in Brussels, Belgium. Different types of criminal and terrorist organizations use TBML to disguise the origins of their illicit proceeds and fund their operations. In some cases, these organizations may manage the TBML schemes directly, and in other cases, they may enlist the services of professional money launderers. Drug trafficking organizations. Drug trafficking organizations throughout Latin America, including in Colombia and Mexico, have used TBML schemes for decades to launder the proceeds from illegal drug sales. These organizations make billions of dollars from the sale of illegal drugs in the United States and elsewhere. Although much of these revenues remain with the ultimate sellers of the illegal drugs in the United States, significant amounts of illicit proceeds are sent back to drug trafficking organizations in supplier countries, including through TBML schemes. For example, in a 2017 reporting cable on Colombia’s cocaine economy, State noted that U.S. law enforcement agencies and independent economists have estimated that somewhere between $5 billion to $10 billion in cocaine proceeds are laundered back to Colombia each year, frequently using TBML schemes. U.S. government reporting, including Treasury’s 2020 National Strategy for Combating Terrorist and Other Illicit Financing and DEA’s 2019 National Drug Threat Assessment, and various U.S. officials noted that a key trend related to TBML that has occurred in recent years is the increasing involvement of Chinese criminal organizations in TBML globally, including in the United States. Chinese money laundering networks are working increasingly with Mexican drug cartels to assist the cartels in laundering drug proceeds. In addition, U.S. government reporting, including the 2018 National Money Laundering Risk Assessment, and U.S. officials noted Chinese criminal gangs are using TBML schemes to repatriate proceeds from the sale of synthetic opioids in the United States and around the globe. Other criminal organizations. In addition to drug trafficking, criminal organizations have used TBML schemes to launder proceeds from a range of other crimes, including illegal mining, human trafficking, and the sale of counterfeit goods. For example, criminal organizations in Colombia have used TBML to disguise the origins of illegally mined gold, in exchange for funds, according to U.S. Embassy Bogotá and Colombian government officials we interviewed. Corrupt government officials. In certain countries, senior government officials and government entities have used TBML schemes to disguise profits derived from corrupt practices, according to U.S. government reporting. For example, FinCEN has reported that senior government officials in Venezuela have used TBML as part of schemes to steal money from the Venezuelan government’s food distribution program. Terrorist organizations. Terrorist organizations, including Hezbollah and the Revolutionary Armed Forces of Colombia (known by its Spanish acronym FARC), have also used TBML schemes to launder funds. For example, a number of U.S. officials and knowledgeable sources have noted that Hezbollah operates a number of TBML schemes in the Tri-Border Area in South America, where Argentina, Brazil, and Paraguay meet, which help to fund the terrorist organization’s activities around the world. Criminal and terrorist organizations use a range of TBML schemes with varying levels of complexity. In many instances, these organizations combine TBML techniques with other forms of money laundering, such as bulk cash smuggling and the laundering of funds through the banking system. The U.S. government, foreign governments, and international bodies have identified a number of different examples of the types of TBML schemes that occur. For example: In one case described in Treasury’s 2018 National Money Laundering Risk Assessment and ICE press releases, HSI led an investigation, known as Operation Fashion Police, which targeted businesses in the Los Angeles Fashion District that were suspected of being involved in Black Market Peso Exchange schemes to launder the proceeds of illegal drug sales on the behalf of international drug cartels. As a result of the investigation, two owners of a textile company pled guilty to using the business to receive bulk cash that they knew or believed to be the proceeds of narcotics trafficking and part of a Black Market Peso Exchange scheme. The two individuals received approximately $370,000 in cash delivered on four separate occasions as payment for goods shipped to Mexico, Guatemala, and other countries in Latin America. Operation Fashion Police, along with several related investigations, also resulted in the seizure of tens of millions of dollars in bulk cash stashed at warehouses in the Los Angeles area. In one case identified by Treasury, DOJ indicted seven co- conspirators for participating in an international TBML scheme. The individuals are alleged to have used family-owned import-export businesses in Long Island and Miami and to launder millions of dollars in illegal drug proceeds. As part of the scheme, the defendants are alleged to have taken in bulk cash deliveries from drug dealers in the United States and disguised the transfer of money to South America and elsewhere through the actual and purported purchase and export of mobile phones. In another case, according to U.S. government information provided to FATF, Colombian drug cartel representatives in the United States deposited proceeds from illegal drug sales into the U.S. financial system. The cartel then used these funds to buy gold from Colombia, which it imported into the United States. The cartel representatives in the United States then melted down the gold and recast and enameled the gold to disguise it as low value items such as nuts and bolts. The cartel then exported the disguised gold back to Colombia where it was melted down once again and the process was repeated. Through this scheme, the cartel was able to use the same gold to justify multiple payments to its representatives in Colombia, thus transferring proceeds from its U.S. operations. In Australia, according to U.S. Embassy Canberra officials, Chinese criminal organizations give Australian dollars from drug sales to individual Chinese nationals, known as Daigou shoppers, who pose as retail shoppers and use the funds to purchase various items in Australia on behalf of buyers in China who want to purchase higher quality foreign goods. The Daigou shoppers then ship the items to the buyer or deliver them by hand. The buyers in China then pay the Chinese criminal organizations, in Chinese yuan, for the items. Through this TBML scheme, the criminal organizations are able to move their proceeds to China without going through the financial system. Finally, in Benin, Lebanese financial institutions linked to Hezbollah were involved in schemes that used TBML to launder funds and move criminal proceeds through West Africa and back to Lebanon, according to State reporting in its 2015 International Narcotics Control Strategy Report. The criminals using these schemes wired funds from Lebanon to the United States to buy used cars, which were then shipped to Benin and sold throughout West Africa. The criminals then combined the profits from the sale of these cars with the proceeds from drug sales in Europe and subsequently sent the funds back to Lebanon via bulk cash smuggling and deposited the funds into the Lebanese financial system. According to information from different U.S. agencies, international bodies, and partner countries, criminal and terrorist organizations use a wide variety of goods in TBML schemes, but HSI analysis has found the most common items are precious metals, automobiles, clothes and textiles, and electronics (see fig. 3). As of 2018, HSI reported that approximately 70 percent of its TBML-related casework involved these four types of goods. However, criminal and terrorist organizations use any number of different goods in TBML scje,es. For example, U.K. government officials told us about a scheme involving the misrepresentation of dental equipment as books in a series of exports from the United States to the United Kingdom. In addition to international trade in goods, available evidence indicates that TBML schemes, at times, involve international trade in services. According to HSI, under some TBML schemes, shell companies are created that issue invoices for consulting or other professional services which are used to justify the international movement of funds as payment for the invoiced services. U.S. agencies and other sources have noted the potential for TBML schemes involving services such as consulting, accounting, and web design, among others. Various U.S. agencies, international bodies, and knowledgeable sources have identified a number of “red flags” that may indicate TBML schemes. For example, table 1 includes a list of nine red flag indicators that HSI has identified related to TBML schemes. U.S. agencies have identified a number of countries around the world as being at risk for money laundering more generally and TBML specifically. For example, State’s annual International Narcotics Control Strategy Report (INCSR) identifies “major money laundering countries,” as required by the Foreign Assistance Act. Over the last 5 years, the INCSR has identified, on average, almost 80 countries as being major money laundering countries. In addition, State has identified countries that face TBML-specific risks in the country reports included within the INCSR each year. For example, in our review of the 2019 INCSR, we found that State had cited TBML risks in 26 countries or territories in a number of different regions of the world. Previously, HSI conducted an analysis of TBML- related SARs filed by financial institutions with FinCEN in fiscal year 2012. Of the 474 TBML-related SARs that financial institutions filed during this period, HSI found that 93 different countries or territories were referenced with the five most frequently mentioned being Nigeria, Hong Kong, Mexico, Venezuela, and Panama. More recently, in 2019, HSI identified Mexico, China, Colombia, the United Arab Emirates, Ecuador, Peru, Venezuela, and the United Kingdom as its key countries of TBML concern. In addition to identifying different countries that are vulnerable to money laundering, the U.S. government and FATF, among others, have identified free trade zones as particular areas of risk for TBML. In a 2010 report on money laundering vulnerabilities in free trade zones, FATF identified approximately 3,000 free trade zones located in 135 countries and noted they had systemic weaknesses making them susceptible to money laundering and terrorist financing. These weaknesses included less stringent AML/CFT reporting requirements, relaxed oversight by responsible government authorities, and weak procedures for inspecting goods, among other things. Similarly, the 2019 INCSR notes that the 114 free trade zones in Colombia are vulnerable to TBML due to inadequate regulation, supervision, and transparency. Available evidence from the U.S. government, international bodies, and knowledgeable sources suggests that the amount of TBML occurring globally is substantial and has increased in recent years. State has reported that the amount of money laundered through TBML schemes may potentially be up to hundreds of billions of dollars globally, every year. Some U.S. officials and knowledgeable sources believe that, based upon available evidence, TBML is likely one of the largest forms of money laundering. In addition, as countries have strengthened their controls to combat other forms of money laundering, various U.S. government reports and officials, as well as knowledgeable sources have stated that there are indications that criminal organizations and terrorist organizations have increased their use of TBML to launder their funds. For example, FinCEN has reported that since the Mexican government increased restrictions on U.S. dollar cash deposits at Mexican financial institutions in 2010, Mexican drug cartels appear to have increasingly turned to TBML as an alternative means of repatriating profits from U.S. drug sales. Similarly, in Australia, as controls on large cash deposits at ATMs have increased since 2017, criminals have increased their use of TBML to hide their profits, according U.S. officials at Embassy Canberra. In addition, the 2020 National Strategy for Combating Terrorist and Other Illicit Financing notes that there has been a steady decrease in seizures related to bulk cash smuggling from 2012 through 2018 and states that this decrease could indicate that criminal organizations are increasingly turning to other means to move illicit money, including TBML. Although various observers believe the magnitude of TBML is large, specific estimates of the amount of TBML occurring around the world are unavailable. A number of academic studies have sought to quantify various aspects of illicit financial flows and money laundering. Although the results of such studies can shed light on the potential volume of TBML, none of those we identified in our literature review sought to develop estimates of TBML specifically. In addition, the studies we reviewed all had certain methodological limitations. We found, based upon our review of relevant literature, that academic studies seeking to quantity potential illicit financial flows do not provide the exact extent of TBML. These studies capture activities that are generally broader than TBML, such as tax avoidance, trade price manipulation, or trade misinvoicing, which demonstrates the difficulty in estimating the exact magnitude of TBML activity. For example, one academic researcher analyzed U.S. Census Bureau trade data over time to estimate money moved in and out of the United States through trade price manipulation, which involves prices showing up outside of an expected range. The stated objectives of trade price manipulation in this study include not only TBML, but also income tax avoidance or evasion, among other things. Therefore, measurement of trade price manipulation is generally broader than that of TBML. For 2018 alone, this researcher estimated that trade price manipulation accounted for approximately $278 billion moved out of and $435 billion moved into the United States. Global Financial Integrity, a nonprofit organization dedicated to studying the cross-border flow of illegal money, has analyzed International Monetary Fund and United Nations data to develop an estimate of potential trade misinvoicing between developing and advanced economies. In a 2019 report, it calculated the illicit financial flows to and from 148 developing countries from 2006 to 2015. For 2015, it estimated that potential trade misinvoicing to and from these 148 developing countries were between $0.9 trillion and $1.7 trillion. Global Financial Integrity defines trade misinvoicing as a method for moving money illicitly across borders that involves the deliberate falsification of the value, volume, or type of commodity in an international commercial transaction of goods or services by at least one party to the transaction. Therefore, measurement of trade misinvoicing is generally broader than that of TBML. Appendix II provides additional details on our literature review and efforts to quantify illicit financial flows, including TBML. Certain international bodies, such as UNODC, and other organizations have produced estimates on the amount of criminal proceeds and the volume of money laundering more broadly. For example, in 2011, UNODC conducted a meta-analysis of the results of various studies and estimated that in 2009 the amount of funds available for laundering, including TBML, was likely around 2.7 percent of global gross domestic product, or $1.6 trillion. However, the report’s authors noted that the studies reviewed in the meta-analysis contained a range of methodological issues and information gaps. FinCEN data on SARs related to TBML can also provide an indication of the potential volume of TBML activity that financial institutions have detected. In 2010, FinCEN issued an advisory on TBML that found that financial institutions had filed over 17,000 SARs related to potential TBML between January 2004 and May 2009, involving over $276 billion worth of transactions. In addition, we analyzed FinCEN data from more recent years, using a different methodology, and found financial institutions had filed 7,044 SARs related to TBML from 2014 to 2018, including 1,673 in 2018. FinCEN officials noted that the number of TBML-related SARs is a small portion of the total of 9.6 million SARs it received over this period. However, FinCEN officials also acknowledged that financial institutions may not have enough information on many trade transactions to determine whether there is suspicious activity and whether that suspicious activity is potentially related to TBML schemes. In addition, FinCEN officials noted that suspicious activity related to TBML schemes could be reported under different categories. Officials and reporting from relevant international bodies and selected partner countries, and knowledgeable sources have recommended that governments consider a number of different practices to strengthen their efforts to detect and combat TBML. After reviewing and analyzing these sources, we identified and grouped these recommended practices into the following five categories: (1) partnerships between governments and the private sector, (2) training in detecting and combatting TBML, (3) sharing information through interagency collaboration, (4) international cooperation through information and knowledge sharing, and (5) further research on challenges, such as potential impediments to combatting TBML. In addition, we identified examples of steps the United States and other countries have taken in line with these practices. Officials and knowledgeable sources also noted some potential difficulties to implementing some of the recommended practices that have been identified. Reporting from relevant international bodies and certain partner countries, and knowledgeable sources have proposed that governments develop partnerships with the private sector to combine and collectively analyze information needed to identify potential TBML schemes and trends. Through these partnerships, representatives from the private and public sector could meet on a regular basis to share information on suspicious activity that may warrant further investigation. For example, FATF’s guidance paper Best Practices on Trade Based Money Laundering stated that governments should consider conducting periodic joint meetings with the private sector to discuss emerging TBML trends. Governments can also provide feedback to private sector entities on what information is helpful as they conduct investigative work. FATF standards on information sharing state that anti-money laundering authorities should provide feedback to financial institutions to assist them with complying with AML requirements in the countries in which they are operating. For example: U.S. example: In 2017, FinCEN publicly launched the “FinCEN Exchange” to enhance information sharing between FinCEN, law enforcement agencies, and financial institutions. FinCEN invites financial institutions to voluntarily participate. As of December 2018, FinCEN had convened more than a dozen briefings with law enforcement agencies across the country, involving more than 40 financial institutions. According to FinCEN officials, through the FinCEN Exchange, the U.S. government and the private sector are able to exchange information on priority illicit finance threats, including TBML. For example, according to Treasury officials, FinCEN convened a FinCEN Exchange focused on TBML in San Antonio, Texas in April 2018. According to Treasury’s 2018 National Strategy for Combating Terrorist and Other Illicit Financing, the information provided by financial institutions through the FinCEN Exchange briefings has assisted FinCEN in targeting TBML networks. Other country example: In 2015, the United Kingdom established the Joint Money Laundering Intelligence Task Force as a collaborative mechanism between the U.K. government and the private sector to share and collectively analyze information on money laundering and economic crime threats. The task force brings together a range of private and public sector organizations, including law enforcement agencies and financial institutions. According to U.K. officials, TBML is one of the four priority areas of the task force. The task force has established six expert working groups led by representatives of the financial sector, including a TBML expert working group. Among other things, the TBML expert working group offers experts witness statements on TBML to support criminal prosecutions. In addition to sharing information with and providing feedback to financial institutions, several knowledgeable sources and reports from international bodies stated that these partnerships should also include a broad range of private sector entities involved in international trade. Several knowledgeable sources have highlighted the need for other private sector entities involved in international trade, such as shipping companies, freight forwarders, and customs brokers, to play a role in working with governments to identify TBML activities. One knowledgeable source noted that broader partnerships are important because banks and other financial institutions have a limited ability to detect indicators of potential TBML in a majority of trade transactions. For example, according to the Wolfsberg Group, 80 percent of international trade is conducted through open-account trade. With open-account trade, the transaction is not financed by a bank. Banks are generally not involved beyond processing the buyer’s payment to the seller and do not typically receive supporting documentation related to the transaction. Thus, financial institutions have limited visibility over open-account transactions and thus limited ability to identify suspicious activity. Several knowledgeable sources and reports from certain partner countries also acknowledged that challenges exist to creating partnerships with the private sector. They emphasized that for these partnerships to be successful, governments should ensure all participants trust that any information they share will be handled appropriately. For example, one knowledgeable source noted that countries could develop standards for information sharing between banks, while providing assurances about data security, privacy, and confidential commercial information. In addition, several knowledgeable sources and reports from partner countries stated that countries should address challenges related to privacy laws that prohibit banks from sharing client information or barriers restricting government agencies from sharing intelligence information with private sector partners. Relevant international bodies, including FATF, and knowledgeable sources stated that given the complexity of and difficulty in detecting TBML, governments could consider providing additional training to relevant government officials on techniques to detect and counter the threat. Governments would provide the training to government agencies, such as customs and tax collection agencies, tailored to meet the specific requirements and needs of different government authorities. Several knowledgeable sources and reports from international bodies noted that governments should also conduct events and other outreach activities to educate private sector entities. Some stated that such events and outreach activities could help increase the capacity of personnel at banks and other financial institutions to identify the characteristics, emerging trends, and new methods of TBML. According to FATF’s guidance paper on TBML, governments could organize conferences on the topic, or develop materials to help inform staff of various private sector organizations who monitor suspicious financial activity and potential TBML risks. For example: U.S. example: In 2018, FinCEN organized a conference on TBML for several U.S. agencies involved in combatting TBML, including HSI, CBP, and Internal Revenue Service Criminal Investigation, in addition to government officials from partner countries and non-government participants. The conference provided presentations on a range of issues related to TBML, such as the vulnerabilities in the gold industry that make it susceptible to TBML and the evolution of the Black Market Peso Exchange. In 2019, FinCEN organized an additional conference focused on TBML and bulk cash smuggling. Other country example: The Mexican government is working with State/INL to develop anti-money laundering experts and to build an AML task force. INL also created a training program to certify compliance officers, state auditors, prosecutors, analysts, and regulators in Mexico City on TBML. Several U.S. embassy officials noted that some partner countries needed to account for additional factors when creating TBML-specific training. They stated that before receiving TBML training, some partner countries needed to build more basic foundational skills. For example, U.S. embassy officials in Colombia stated that their priority is to provide Colombian prosecutors with more basic training on prosecutorial skills, such as presenting oral arguments, before offering advanced training, such as how to build a TBML case. Several knowledgeable sources, partner country officials, and international body reports we reviewed recommended that governments share information and data through domestic interagency collaboration to combat TBML. According to United Kingdom officials and an international body report, sharing trade data and relevant financial information, such as SARs, through an interagency approach is critical because TBML and its predicate crimes often cut across multiple agencies and their authorities and responsibilities. Agencies also bring different skill sets to investigations, such as expertise on customs enforcement, financial crimes, and trade data analysis. To foster interagency collaboration, several knowledgeable sources stated that governments could consider creating multi-agency task forces or mechanisms to address the challenges posed by TBML. For example: U.S. example: The El Dorado Task Force is an interagency investigative body that consists of 55 law enforcement agencies in New York and New Jersey, including federal agents, state and local police investigators, intelligence analysts, and federal prosecutors. The task force contains 12 groups, including one focused specifically on TBML. Officials from the El Dorado Task Force stated that as an interagency task force, it is able to utilize the respective expertise of various agencies and analyze multiple sources of information, such as international trade and Bank Secrecy Act data, in its investigative work. Other country example: The United Kingdom created the National Economic Crime Centre, which involves officials from multiple agencies, including law enforcement and regulatory bodies. The National Economic Crime Centre’s mission is to strengthen and prioritize the U.K. government’s coordination efforts by combining operational capabilities, data, and intelligence to target economic crime. To target specific crimes, the National Economic Crime Centre has created working groups, including a TBML one, to further cooperation and build expertise. Several U.S. embassy officials and host country officials stated that some countries may be hesitant to share information with all of the agencies involved in combatting TBML. These officials noted that issues such as corruption and lack of trust between agencies might limit the willingness and ability of countries to share information. For example, several Colombian government officials stated that corruption in their government limits the number of counterparts from other agencies that they can trust to collaborate with on combatting TBML. Several officials from certain partner countries, knowledgeable sources, and reports we read stated that trade partners could share trade data and relevant financial information with each other through bilateral or multilateral partnerships. Officials and international body reports also emphasized how important it is for countries to see both sides of trade transactions in order to detect anomalies that might reveal TBML activities. FATF reports noted governments could work together to create a secure system or mechanism that countries could use to exchange trade data and financial information. According to the Asia/Pacific Group on Money Laundering’s APG Typology Report on Trade Based Money Laundering, governments could coordinate international capacity building efforts with partner country counterparts, such as sharing strategies on combatting TBML and emerging trends related to TBML. For example: U.S. example: As part of its TTU program, HSI has established a formalized bilateral mechanism with a number of partner countries, particularly in the Western Hemisphere, to exchange and conduct ongoing analysis of trade data to facilitate the detection of suspicious TBML-related activities. By sharing these data, HSI and each of its partner TTUs are able to see import and export data for goods moving between the United States and the partner country. Other country example: The Paraguayan government has taken initial steps to coordinate with several countries in the region to try to increase the sharing of trade information, including Chile, Uruguay, and Argentina. According to a U.S. embassy official in Paraguay, the Paraguayan government also participates in a regional security mechanism with Brazil, Argentina, and the United States to address broader regional security threats, including money laundering activities. Figure 4 shows photos from Ciudad del Este, Paraguay, on Paraguay’s border with Brazil and Argentina, a region that has been identified by U.S. and Paraguayan officials as a key hub of TBML activity. U.S. officials and knowledgeable sources, however, noted several challenges to international cooperation related to technology and data uniformity. For example, officials from HSI stated that while international cooperation is critical to combat TBML, changes in government administration and technological limitations affect the continuity and the commitment to information sharing with foreign partners. In addition, U.S. officials and reports we reviewed stated that countries could consider enhancing and creating more uniformity in their data collection efforts so that they could use the data more effectively to combat TBML. For example, U.S. embassy officials and knowledgeable sources stated that countries need a common formatting or trade transactions identifier to allow countries to match import and export data more easily. HSI and partner country officials noted that, without a common identifier, they have faced difficulties connecting the import and export sides of trade transactions as they have sought to analyze trade data to identify potential cases of TBML. In addition, while some U.S. officials and knowledgeable sources see arrangements for sharing trade data between multiple countries as a possible means of improving detection of TBML-related activities, U.S. officials said that a lack of trust among countries complicates such efforts. U.S. officials and officials from countries we visited noted that countries might be reluctant to share their trade data more widely through multilateral mechanisms due to perceived risks the sharing of such important information might have on their commercial competitiveness. These officials noted the difficulty in creating a multilateral TTU because of these limitations. Multiple knowledgeable sources, as well as reports from international bodies, stated that governments could conduct further research on challenges that reduce their ability to combat TBML effectively, including potential impediments. According to the Asia/Pacific Group on Money Laundering’s report on TBML, developing a comprehensive strategy would help governments to address key challenges to combat TBML while also facilitating legitimate trade. In addition, one partner country report highlighted the need for an ongoing assessment of TBML to address challenges as the threat continues to evolve. For example: U.S. example: In 2015 and 2018, Treasury produced the National Money Laundering Risk Assessment, identifying the money laundering threats and risks, including TBML, which confront the United States. The assessments also identify the challenges U.S. agencies face in combating money laundering. For example, the 2018 assessment found that merchants sometimes knowingly accept illicit payments in exchange for trade goods without reporting the transactions and individuals can abuse their professional position at financial institutions by ignoring suspicious transactions. Other country example: In 2017, the Government of Singapore worked with private sector entities to identify and assess key issues that Singapore faced related to money laundering. As a result of that study, in 2018, the government produced the Best Practices for Countering Trade-Based Money Laundering report. The study found that, for example, banks should periodically conduct a risk assessment on risk factors related to TBML and test TBML red flags for effectiveness. Several knowledgeable sources stated that international bodies could examine any challenges and provide additional guidance to member countries on combatting TBML. According to Treasury officials, FATF is currently examining operational challenges related to TBML to provide additional guidance to member countries on combatting it. These officials indicated that this new study should provide an updated definition of TBML to better distinguish money laundering activity from other criminal activity. Additionally, an official from Treasury’s Office of Terrorism and Financial Intelligence said the best practices in FATF’s 2008 report were still relevant and that FATF has produced other reports since then related to TBML, such as its 2010 report on money laundering vulnerabilities in free trade zones. In the report, FATF noted a number of challenges related to combating TBML in these zones. For example, it reported that relaxed oversight and lack of data collection in free trade zones make them vulnerable to these schemes. Knowledgeable sources and reports from international bodies and a partner country also recommended further research about other impediments that challenge the ability of governments to combat TBML. For example, reports from international bodies and a partner country highlighted the ease with which shell companies can be established in many jurisdictions and the lack of transparency regarding the beneficial owners of such shell companies. According to FATF and various U.S. officials, criminal organizations can use shell companies to funnel illicit money through accounts that obscure the source of the funds. FATF recommends in its international standards that countries take measures to ensure relevant authorities have timely access to information on the ownership and control of legal persons. DHS, DOJ, State, and Treasury provide a variety of support to partner countries to assist in combating TBML, including establishing information- sharing methods, funding training and technical assistance, and providing ongoing law enforcement cooperation. The U.S. government’s primary partnership effort focused specifically on combating TBML is HSI’s TTU program. Under the program, HSI has set up TTUs in 17 partner countries. HSI established the first TTU with Colombia in 2005 and the most recent one with New Zealand in 2019. HSI’s goal with the TTU program is to exchange trade data with its partner TTUs to allow agencies in each country to work together to better identify anomalies in trade data that may indicate TBML. For example, through the analysis of shared trade data, HSI and a partner TTU may be able to determine if there is a discrepancy between the reported value of goods when they leave the United States and the reported value of the goods when they arrive in the partner country (and vice versa). There are four key steps that HSI and a partner country undertake in establishing a TTU, according to HSI officials: As a precondition for setting up a TTU, a country must have a Customs Mutual Assistance Agreement or similar information sharing agreement in place with the United States. HSI then negotiates a memorandum of understanding (MOU) with the relevant counterpart agency setting out the details of the partnership. Once the partner country signs the MOU, HSI provides the partner TTU access to its specialized system for analyzing trade data—the Data Analysis and Research for Trade Transparency System (DARTTS). HSI also provides training to the partner TTU on the system’s use. Table 2 shows the partner countries participating in the TTU program and how often HSI and each country share data. In addition to the TTU program, U.S. agencies have established other methods for sharing information with partners overseas that support efforts to combat money laundering, including TBML. For example, U.S. officials at Embassy Canberra reported that HSI had set up a pilot program in which the U.S. government shares its Reports of International Transportation of Currency or Monetary Instruments with the Australian Border Force. By comparing the U.S. information with what the Australian Border Force collects, the Australian Border Force has been able to identify and apprehend a number of bulk cash smugglers, according to Embassy Canberra officials. U.S. agencies have also worked to organize a number of ongoing or ad hoc forums for sharing information related to transnational crime, including money laundering and other economic crime. For example, DOJ’s Office of Overseas Prosecutorial Development, Assistance and Training has organized, with State support, two sessions of the Transnational Criminal Organizations Working Group, which brings together officials from the United States, Colombia, and Mexico to participate in specialized training and to develop joint strategies and best practices for combating transnational criminal organizations that threaten the three countries. According to an Office of Overseas Prosecutorial Development, Assistance and Training official at Embassy Bogotá, combating money laundering, including TBML, was a focus of the group’s most recent session in June 2019. State and Treasury’s OTA have funded a range of foreign assistance programs in partner countries that provide training and technical assistance related to combating money laundering and economic crimes. State allocated approximately $90 million in fiscal years 2014 through 2018 to programs to counter financial crimes and money laundering throughout the world. According to State, this funding supported a range of programs, including programs to assist countries in drafting legislation and regulations; training bank regulators and examiners, financial investigators, prosecutors, and judges; and strengthening the ability of FIUs in partner countries to receive, analyze, and disseminate suspicious activity reports, among other things. Although State has not funded any programming that focused exclusively on TBML during this period, it reported that it allocated approximately $5 million in fiscal years 2014 through 2018 for programs that included a substantial amount of information on the investigation, enforcement, or prosecution of TBML. For example, according to State, it has funded a series of projects to reform Peru’s criminal justice system that, among other things, helped strengthen the country’s ability to fight TBML. More recently, in fiscal year 2019, State noted that it has allocated approximately $5 million to the WCO for a project focused specifically on TBML. According to State, through this program, WCO will build the capacity of customs agencies to detect and deter smuggling and misreporting used to facilitate TBML. Treasury’s OTA allocated approximately $20 million in fiscal years 2014 through 2018 for projects to counter economic crimes throughout the world. Through these projects, OTA funds advisors—either a resident advisor who remains in the host country for several years, or a group of intermittent advisor who travel to the host country for short-term assignments. According to Treasury, these projects support the implementation of AML/CFT legal and regulatory regimes, as well as host government institutions, that are able to combat economic crimes. Although OTA has not funded any projects focused specifically on TBML, it stated that OTA advisors routinely discuss with their country partners the different methods that criminals use to launder money, including TBML. According to OTA, its assistance has addressed TBML to varying degrees in a number of projects. For example, OTA helped Peru’s tax and customs authorities to develop training for the Peruvian National Police Money Laundering Unit on how to best use customs databases to identify potential leads in TBML cases. Law enforcement agencies, including DEA, HSI, the Federal Bureau of Investigation, and Internal Revenue Service Criminal Investigation, have also posted personnel overseas that collaborate with law enforcement officials from the host country to work on cases related to TBML. For example, according to HSI data, the agency has opened TBML investigations supported by its personnel at embassies in a number of countries, including Colombia, Mexico, the United Kingdom, the Netherlands, the Dominican Republic, Singapore, and Spain. U.S. law enforcement personnel have also set up U.S.-supported vetted units in partner countries. For example, DEA has established Sensitive Investigative Units in a number of countries, such as Colombia and Paraguay. DEA partners with these units to investigate and disrupt various aspects of drug trafficking organizations’ operations, including money laundering activities. Over time, HSI’s work with partner TTUs has helped in the successful disruption of certain TBML schemes. For example, HSI reported that the Panamanian TTU provided analysis to support an investigation that successfully disrupted an illicit tobacco smuggling ring involving several Panamanian companies. The investigation led to four arrests and the seizure of over $10 million in cigarettes. In another case, HSI reported that HSI and the Peruvian TTU worked together to support an investigation that disrupted a TBML scheme involving the import of illegally mined gold into the United States from Peru. While HSI and other U.S. government officials have stated the TTUs in some countries have played an important role in certain investigations, the TTU program has faced challenges that limited its results in disrupting TBML schemes, including: Insufficient resources or support for the partner TTUs. In recent years, the U.S. government has not provided any funding directly to partner TTUs to support their activities, according to HSI officials. These officials noted that while HSI does not obligate funds to directly support partner TTUs, the agency will fund the travel expenses for its personnel to travel to a foreign country to provide training to a partner TTU. Previously, State had provided a limited amount of funding to certain partner TTUs, including for training and the purchase of computer software, according to State officials. However, State officials reported that State has not provided any funding for partner TTUs since fiscal year 2013, because insufficient evidence of the program’s effectiveness and various programming obstacles have led the department to prioritize funding for other anti-money laundering and crime prevention programs over the TTU program. For example, State officials noted that limited support from some U.S. embassies and a lack of HSI staff posted at them negatively affected the TTU program at times. However, State officials noted that they are generally supportive of the TTU concept and would consider providing further funding for the program, if HSI can demonstrate program results. HSI officials noted that they have not sought State funding for the TTU program in recent years, but would be interested in discussing State’s expectations regarding program results and pursuing State funding going forward. U.S. and partner country officials also noted that host governments have not always dedicated the necessary personnel and information technology resources to ensure the effective operations of the TTUs. For example, HSI officials stated that a lack of funding for partner TTUs has contributed to technology gaps between U.S. and partner country systems. Slow expansion of program and limited geographic range. Although HSI has established the goal of expanding the TTU program, the expansion has slowed over the last few years and it operates mainly in Latin America, despite the range of countries around the world that face risks related to TBML. HSI officials stated they have had discussions with several additional countries about establishing TTUs, but have not yet been able to finalize agreements with a number of these countries, resulting in only two new TTUs being set up over the last 3 years. Delays in launching partner TTUs and lapses in their operation. The TTU program has experienced delays in launching TTUs after HSI and the partner governments have signed the MOUs. For example, HSI officials at Embassy Canberra noted that HSI signed the MOU with Australia to establish its TTU in 2012, but it did not become fully operational until 2017. According to HSI officials, this delay was due to significant coordination challenges within the Australian government. Several TTUs have also experienced lapses in their operations. For example, the TTU in Argentina launched in 2006, but the two countries halted information sharing between 2011 and 2015. According to HSI officials, this halt in information sharing was because of U.S. concerns with corruption in the Argentinian government at that time. Differences in objectives between HSI and partner TTUs. HSI officials noted that one limitation in the TTU program is that partner TTUs frequently focus on revenue collection issues and place less priority on disrupting TBML schemes than HSI does. For example, partner TTUs may seek to identify instances of customs fraud, which can reduce duties collected by customs agencies on imported goods, but they may not pursue the investigation further to disrupt the criminal organizations involved in the scheme. Limited authorities and lack of interagency coordination in TTU partner countries. Partner TTUs generally operate within their countries’ custom agencies, which frequently do not have their own law enforcement authorities, according to HSI and other U.S. officials. As a result, they must coordinate with law enforcement partners within their countries to be effective. However, HSI officials noted that such coordination does not always take place. For example, HSI officials in Mexico stated that the Mexico TTU has had limited effectiveness because of a lack of sufficient cooperation between Mexican customs and law enforcement officials. Similarly, in Brazil, HSI officials noted information sharing with that country’s TTU has been delayed because the TTU lacks ready access to trade data and must purchase it from a different Brazilian government agency. Data sharing and connectivity. HSI and partner government officials have also noted issues about uploading partner trade data into DARTTS and ensuring these data are in a compatible format. For example, an HSI official in the United Kingdom described a delay of several months in uploading data from the United Kingdom into DARTTS because of data formatting issues. In addition, U.S. officials at Embassy Canberra noted that the Australian TTU has frequently experienced connectivity problems with DARTTS that have challenged the TTU’s ability to upload its data to the system. In addition, HSI and partner TTU officials noted that there are certain limitations in DARTTS, including difficulties in working with cross-border data, that reduce its effectiveness as a tool for HSI and partner TTUs to use in identifying potential cases of TBML. DHS noted that details on these limitations are sensitive and we did not include the specifics in this report. Although the TTU program has faced a number of challenges, HSI has not taken key management steps that could help guide its efforts, including developing a strategy and a performance monitoring framework. Because the TTU program involves partnerships between HSI and foreign governments, HSI has varying levels of ability to address these challenges through independent action. However, by developing a strategy and a performance monitoring framework, HSI could assess how best to plan for and address these challenges in order to maximize the program’s effectiveness. HSI officials stated that they have not produced any sort of planning or strategy documents specifically for the TTU program. HSI has produced a strategic plan for fiscal years 2016 through 2020 that references the TTU program. For example, the strategy notes that HSI plans to, “continue to provide operational, analytical, technical, and targeting support on trade- based money laundering and illicit funding investigations being conducted by HSI field offices and partner TTUs.” However, the strategy includes only limited references to the TTU program’s operations. According to HSI officials, for the TTU program specifically, they only conduct informal, periodic planning, such as identifying countries that they would like to prioritize for inclusion in the TTU program. DHS Directive 101-01 establishes requirements for planning, budgeting, programming, and executing for the department and its component agencies. Among other things, the directive requires agency heads, including the Director of ICE, to establish planning processes and methods to oversee program management and risk management activities for the programs and operations under their purview. HSI officials noted that in addition to the HSI strategic plan, they have used some documents, such as FATF’s 2008 report on best practices for combating TBML, to guide the TTU program, but have not prioritized the development of a strategy for the TTU program because of resource constraints. Without such a strategy, however, HSI lacks an important tool to guide its operations, including how best to work with its partner TTUs to identify potential cases of TBML, prioritize potential cases for further investigation, and successfully conduct these investigations. In addition, without a strategy, HSI cannot effectively plan how to grow the TTU program, where appropriate, and establish TTUs in additional priority countries. Although developing a strategy would require an investment of resources, a strategy would help ensure HSI is utilizing its limited resources effectively to achieve the TTU program’s goals over the long term. According to HSI officials, the HSI TTU tracks some information on the results of domestic investigations, including the number of TTU-related cases initiated and arrests made, but it does not have a performance monitoring framework, with specified metrics, that allows it to track the results of its work with partner TTUs. HSI officials also stated they have not conducted any evaluations of the factors that increase or decrease the TTUs’ effectiveness. As part of its requirement on planning, programming, budgeting, and execution, DHS Directive 101-01 states that, among other things, the objective of the execution phase is to account for cost and performance to determine if value has been delivered to stakeholders. The directive also notes that annual analysis and reporting of financial expenditures and performance measure results are key deliverables during the execution phase. HSI officials acknowledged that a performance monitoring framework would be beneficial, but they have prioritized other operational issues because of limited resources. In addition, they noted designing a performance monitoring framework that would allow HSI to measure and evaluate the results achieved through its work with partner TTUs would be challenging because, among other things, enforcement efforts of partner TTUs are not within their control and they do not have access to all partner country information. According to HSI officials, they instead rely on measures such as the number of trade records uploaded into DARTTS and the number of foreign users of DARTTS, among other things. However, without a performance monitoring framework for the TTU program, HSI lacks important information on what successes the program has achieved and how to replicate them with other partner TTUs. In addition, HSI lacks key information on areas where the program is not achieving its intended results and what adjustments to make in response. As with the development of a strategy, working to establish a performance monitoring framework would entail an investment of resources, but once completed it could help HSI in assessing how to maximize the impact of its resource investments in the TTU program. In addition, the performance monitoring framework could help demonstrate results to other stakeholders, such as State, that may wish to consider providing support to the TTUs in partner countries. The U.S. government has worked with FATF, the Egmont Group, UNODC, and the WCO to combat TBML. Among other things, the U.S. government has worked with these international bodies to develop anti- money laundering standards, share information regarding TBML methods and specific cases, and provide training and technical assistance to strengthen the ability of countries to combat TBML. As a member of FATF, the U.S. government has supported the organization’s efforts to develop internationally recognized standards for combating money laundering, terrorist financing, and the financing of the proliferation of weapons of mass destruction. FATF’s standards, updated in 2019, include 40 recommendations. According to FATF, it designed these recommendations to set out the critical measures that countries should establish to: identify the risks, and develop policies and domestic coordination; pursue money laundering, terrorist financing, and the financing of apply preventive measures for the financial sector and other establish powers and responsibilities for the competent authorities (such as investigative, law enforcement and supervisory authorities) and other institutional measures; enhance the transparency and availability of beneficial ownership information of legal persons and arrangements; and facilitate international cooperation. To date, FATF’s standards do not include any specific reference to TBML. However, Treasury officials from the U.S. government’s delegation to FATF stated that the standards are designed to provide a robust framework to help competent authorities prevent, detect, and mitigate against the misuse of global trade and combat all forms of money laundering, including TBML. For example, the officials noted that FATF’s third recommendation identifies the need for countries to criminalize money laundering, which would include TBML activity. The U.S. government also works with FATF to conduct mutual evaluations of member countries. FATF designed these evaluations, which are periodic peer reviews for each country, to provide a detailed assessment of a country’s technical compliance with the FATF standards and the effectiveness of its AML/CFT systems. These evaluations may at times highlight issues related to TBML in countries. For example, FATF’s 2014 mutual evaluation of Spain found a significant number of cases involving TBML, particularly those associated with value added tax or other tax fraud schemes. The U.S. government has also supported FATF’s development of several reports on TBML, including a 2006 report on types of TBML schemes and a 2008 report on best practices for detecting TBML. More recently, FATF published various other reports addressing issues relevant to combating TBML, including the 2010 Money Laundering Vulnerabilities of Free Trade Zones, the 2015 Money Laundering/Terrorist Financing Risks and Vulnerabilities Associated with Gold, and the 2018 Professional Money Laundering. These reports provide a range of guidance to countries on how to detect and combat TBML. FinCEN has worked with its fellow FIUs in the Egmont Group to exchange tactical, operational, and strategic information to assist in efforts to combat money laundering, including TBML. As part of its work with Egmont Group partners, FinCEN shares information on particular cases in response to requests from fellow FIUs, proactively shares relevant information with other FIUs, and requests information from FIUs. According to FinCEN officials, Egmont Group membership is critical to information sharing in support of FinCEN analysis and U.S. law enforcement cases because it provides assurances that members have the appropriate policies and procedures in place to respond to and protect sensitive information. FinCEN and its FIU counterparts follow the Egmont Group’s Principles for Information Exchange Between Financial Intelligence Units, in addition to the law of each jurisdiction, to foster cooperation while sharing information securely. Generally, Egmont Group members use a dedicated computer system that the organization has developed, the Egmont Secure Web, to share information securely. FinCEN officials stated that they respond to about 1,000 information requests a year from other Egmont Group members. For example, at the request of a foreign FIU, FinCEN conducted research on an import/export company suspected of involvement in TBML, summarizing relevant SARs and identifying other relevant information on the subjects. FinCEN’s assessment determined the potential use of a TBML scheme and use of shell companies to obfuscate the flow of funds. FinCEN has also supported the Egmont Group’s efforts to provide training to member FIUs on issues related to money laundering and terrorism financing. For example, FinCEN has helped develop and deliver Egmont Group-sponsored training to FIU analysts on how to understand complex financial data. However, Treasury officials stated that the Egmont Group has not provided any TBML-specific training. Although the Egmont Group has not sponsored TBML-specific training for FIUs, FinCEN officials noted that FinCEN has hosted officials from several partner FIUs at the TBML conferences it held in 2018 and 2019 and has provided its own TBML- related training to partner FIUs. For example, in October 2019, FinCEN provided TBML-related training to Mexico’s FIU. Finally, FinCEN has supported the Egmont Group’s development of relevant guidance documents. For example, the Egmont Group developed, in partnership with FATF, a 2013 report called Money Laundering and Terrorist Financing through Trade in Diamonds. According to the report, the two bodies decided to undertake the research because they had (1) never conducted in-depth research on the diamond trade and associated money laundering and terrorist financing risks and (2) a number of participants in the bodies had noted indications that the diamond trade was being exploited for money laundering and terrorist financing purposes. More recently, in July 2018, the Egmont Group produced an additional report with FATF, Concealment of Beneficial Ownership, which also discussed certain TBML schemes. The U.S. government also partners with UNODC in its work to combat illicit drugs and international crime, including TBML. Among other things, State has provided funding to UNODC’s Global Program against Money Laundering, Proceeds of Crime and the Financing of Terrorism. Through the program, UNODC has provided training and technical assistance to a range of member states throughout the world. For example, as part of the program, UNODC places AML experts in countries for up to a year to serve as mentors. These mentors provide a range of support, such as helping countries establish functioning FIUs. UNODC also conducts shorter-term workshops and training sessions, such as mock trial training for law enforcement officers, prosecutors, and judges to enhance their ability to investigate and prosecute money laundering cases. In addition, according to UNODC, under the program, it has developed model legislation that United Nations members can use in setting AML/CFT legal regimes in their countries that are consistent with FATF standards. The U.S. government has also supported certain UNODC programs that have specifically addressed issues related to TBML. According to a UNODC official in Colombia, UNODC has worked with State INL and HSI to provide training for governments in the region to increase expertise on TBML. The official said that UNODC is prioritizing TBML-specific trainings, particularly to build TBML knowledge amongst new prosecutors. In addition, UNODC headquarters officials noted that State INL has supported the development of a program on TBML that UNODC is planning in the Caribbean. The U.S. government also works with the WCO to develop and strengthen the role of customs administrations in tackling TBML. Among other things, CBP has supported WCO’s efforts to develop enforcement tools, guidance and best practices, and training for member countries. For example, CBP has supported the WCO’s development of its Cargo Targeting System. The system, which is available to all WCO members, is designed to assist customs agencies in conducting automated risk assessments of import, export, and transshipment cargo in order to identify high risk shipments that warrant further investigation. With WCO support, several customs agencies also developed the “Compendium of Customs Operational Practices for Enforcement and Seizures,” a tool that provides practical examples for improving enforcement and seizure practices. With CBP support, WCO has produced a number of guidance and best practices documents that can support efforts to combat TBML. For example, in a 2018 report, the WCO described a number of best practices that customs administrations could consider for combating illicit financial flows via trade misinvoicing. In addition, in 2019, the WCO and the Egmont Group developed a Customs-FIU Cooperation Handbook that provides their members guidance and best practices for enhancing global collaboration efforts between customs agencies and FIUs. Finally, the WCO has provided training for its member countries to deter illicit activities and combat TBML. For example, through the WCO, HSI special agents with AML and TBML expertise have conducted workshops to assist WCO member countries in their operational efforts. The WCO also organized a joint workshop with the Organization for Economic Co- operation and Development in 2019 that was designed to raise awareness among customs agencies, FIUs, and law enforcement agencies about TBML related to gems and precious metals. In 2019, the WCO also agreed to launch a two-year counter-TBML effort entitled “Project TENTACLE,” according to CBP officials. The project will include the delivery of TBML workshops to WCO members through 2021, as well as five operational customs activities that follow each workshop. This project will focus on the Asia/Pacific, Africa, and South America regions. State INL has provided funding for Project TENTACLE, in coordination with experts from ICE and CBP. WCO officials noted the lack of training that many customs administrations have on TBML, and the need for regularized training on the subject. TBML poses significant national security risks to the United States. Criminal and terrorist organizations use TBML schemes to disguise the origins of billions of dollars in funds generated by their illicit activities. Given the national security threat that TBML poses, it is crucial that the U.S. government develop an effective response to combat it. Because TBML is international in nature and frequently involves complex, difficult to detect schemes that cut across international borders, it is important that the U.S. government respond through domestic efforts and collaborate with partner countries and international bodies to address the problem. As the U.S. government’s primary partnership program focused on combating TBML, the TTU program plays a key role in these efforts to collaborate with other countries. Although the TTU program has achieved some successes, it has also faced a number of challenges. However, HSI has not taken key management steps to address those challenges and to strengthen the TTU program. HSI, for example, has not established a strategy for the TTU program. Because HSI does not have such a strategy, it lacks an important guide for its efforts to maximize the effectiveness of its existing TTU partnerships and to prioritize efforts to expand the program to other countries. HSI also does not have a performance monitoring framework that tracks the results of its work with partner TTUs. Without such a framework, HSI does not have a means of systematically tracking progress toward program goals and identifying areas that need adjustments to improve program results. We are making two recommendations to DHS: The Secretary of Homeland Security should direct the Director of ICE to develop a strategy for the TTU program to ensure that ICE has a plan to guide its efforts to effectively partner with existing TTUs, and to expand the program, where appropriate, into additional countries. (Recommendation 1) The Secretary of Homeland Security should direct the Director of ICE to develop a performance monitoring framework for the TTU program that would enable the agency to systematically track program results and how effectively it is achieving the program’s goals. (Recommendation 2) We provided a draft of the report to DHS, DOJ, State, and Treasury. DHS, State, and Treasury provided technical comments, which we incorporated as appropriate. DOJ noted that it had no comments on the draft. DHS also provided written comments, which are reproduced in appendix III. In its comments, DHS stated it concurred with our recommendation that the Secretary of Homeland Security direct the Director of ICE develop a strategy for the TTU program, but did not concur with our recommendation to develop a performance monitoring framework for the program. In its response to our recommendation regarding a strategy for the TTU program, DHS noted that HSI has a strategic plan for fiscal years 2016 through 2020 that addresses the TTU program. However, it stated that the TTU program would develop, as a complement to the HSI strategic plan, a document that outlines emerging threats and challenges, as well as existing metrics that are used to track program results for the TTU. In noting it did not concur with our recommendation to develop a performance monitoring framework for the TTU, DHS stated the TTU program already collects a number of statistics each fiscal year related to its program results and can use these statistics to demonstrate program results. DHS also stated that while the TTU program’s primary mission is to establish partnerships and provide foreign law enforcement with information tools to facilitate the exchange of data between TTUs, HSI has limited ability to track the activities of partner TTUs and cannot dictate the enforcement actions partner countries take. In our report, we acknowledge that the HSI TTU tracks some information on the results of domestic investigations, as well as other information, such as the number of records in DARTTS. We also acknowledge that because the TTU program involves partnerships between HSI and foreign governments, HSI does not have the ability to independently control all aspects of the program’s performance. However, we believe that further action by HSI to establish a performance monitoring framework is warranted for the following reasons. First, although HSI has noted examples of statistics it can use to measure the performance of the TTU program, it does not have a formally documented framework or process for measuring its performance or reporting performance results. Second, while the TTU program has identified a few indicators it uses in assessing performance, it has not established any indicators with goals for which to measure its results against, making it challenging to assess whether HSI is making progress to achieve the program’s goals. Third, even though HSI has some measures, such as the number of TTU-related cases it has initiated or arrests made, HSI officials acknowledged that the agency does not track information on what role the TTU actually played in these cases. As a result, HSI cannot establish the extent to which the TTU, rather than a different HSI office, has contributed to any of the measures. Fourth, although we recognize that HSI does not have the ability to dictate what actions partner TTUs will take and may not have access to all relevant partner country information, HSI does have opportunities to take further action to monitor the outputs of its work with partner TTUs. For example, HSI could work with partner TTUs to collect information more systematically on successful cases that they have initiated. HSI could also collect information on factors that reduced the ability of partner TTUs to successfully pursue cases. Other U.S. agencies have conducted performance monitoring and evaluations on programs that rely on partnership and collaboration with foreign governments. We continue to believe in the need for a rigorous performance monitoring framework for the TTU program, a key U.S. government effort in combatting TBML. We note that HSI could potentially integrate a performance monitoring framework into the strategy it plans to develop in response to our first recommendation. For example, DHS stated in its comments that HSI plans to document the metrics it will use to measure the TTU program’s results in that strategy. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Acting Secretary of Homeland Security, the Secretary of State, the Secretary of the Treasury, and the Attorney General. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-8612 or GianopoulosK@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made contributions to this report are listed in appendix IV. This report examines (1) what the available evidence indicates about the types and extent of international trade-based money laundering (TBML) activities, (2) the practices international bodies, selected countries, and knowledgeable sources have recommended for detecting and combating TBML, and (3) the extent to which U.S. Immigration and Customs Enforcement has effectively implemented the TTU program and the steps the U.S. government has taken to collaborate with international partners to combat TBML. To address all three objectives, we analyzed relevant data and documentation from the Departments of Homeland Security (DHS), Justice (DOJ), State (State), and the Treasury (Treasury). For example, we reviewed U.S. government documents that discuss risks associated with TBML, including Treasury’s 2015 and 2018 National Money Laundering Risk Assessment and 2015 and 2018 National Terrorist Financing Risk Assessment and the Drug Enforcement Administration’s annual National Drug Threat Assessment. In addition, we reviewed U.S. government strategy documents that provide information on the extent and types of TBML, including Treasury’s 2018 National Strategy for Combating Terrorist and Other Illicit Financing and State’s annual International Narcotics Control Strategy Report (Volume II). We also analyzed other U.S. government reporting on TBML, including TBML- related advisories from Treasury’s Financial Crimes Enforcement Network (FinCEN), selected cables from U.S. embassies describing TBML issues in their host country, and summary analyses from Immigration and Customs Enforcement Homeland Security Investigation’s (HSI) Trade Transparency Unit (TTU). Finally, we conducted interviews with officials from DHS, DOJ, State, and Treasury in Washington, D.C. We also selected a nongeneralizable sample of six countries to study in greater depth. We conducted fieldwork in three of these countries: Colombia, Paraguay, and the United Kingdom. During our fieldwork in each country, we interviewed U.S. embassy officials from DHS, DOJ, State, and Treasury. In each country, we also interviewed host country officials, including TTU, law enforcement, financial intelligence unit, and financial regulatory agency officials. In addition, in Paraguay, we traveled to Ciudad del Este to observe commercial activity and border operations on Paraguay’s border with Brazil and Argentina. For the other three countries we selected—Australia, Mexico, and Singapore—we conducted work remotely. We interviewed, via telephone, U.S. embassy officials in Australia and Mexico, and obtained written responses from U.S. officials at Embassy Singapore. To select these six countries, we considered several criteria, including (1) the type and extent of TBML risk, (2) the types and level of U.S. collaboration with the country, (3) the presence of U.S. agencies that work on TBML in the country, (4) the extent to which the country had implemented recommended practices to identify and combat TBML (with a goal of covering a range of levels of adoption), and (5) the country’s location (with a goal of covering a range of geographic regions). The team also considered additional factors based on recommendations from knowledgeable sources, such as selecting countries with differing levels of capacity to respond to the TBML threat. To determine what available evidence indicates about the types and extent of international TBML, we analyzed documentation from relevant international bodies including the Egmont Group of Financial Intelligence Units (the Egmont Group) the Financial Action Task Force (FATF), the United Nations Office on Drugs and Crime (UNODC) and the World Customs Organization (WCO). For example, we reviewed these reports: FATF’s 2006 Trade Based Money Laundering and 2008 Best Practices Paper on Trade Based Money Laundering; the Egmont Group’s and FATF’s 2013 Money Laundering and Terrorist Financing through Trade in Diamonds; UNODC’s 2011 Estimating Illicit Financial Flows Resulting from Drug Trafficking and Other Transnational Organized Crimes; and WCO’s 2018 Illicit Financial Flows via Trade Mis-invoicing. To gather further information regarding the types and extent of international TBML activities, we conducted 15 interviews, covering a nongeneralizable sample of individuals knowledgeable about TBML and efforts to combat it, including academic researchers, think tank officials, private sector representatives from trade organizations and individual companies, and former U.S. government officials. Throughout this report, we refer to these individuals as “knowledgeable sources.” In selecting these knowledgeable sources, we conducted initial research to identify individuals or organizations that had conducted research related to TBML and prioritized those whose work was frequently cited by other sources. We also requested recommendations from U.S. agencies and the knowledgeable sources we spoke with regarding other individuals or organizations we should meet with during our work. In selecting these knowledgeable sources, we sought to choose people with different types of experiences studying and working on issues related to TBML to get a range of perspectives. We also conducted a literature search for studies from peer-reviewed journals, conference papers, dissertations, government reports, industry articles, and think tank publications that sought to quantify the amount of TBML activities. We also asked for recommendations on relevant publications as part of our initial meetings with U.S. agencies and knowledgeable sources. We examined summary level information about each piece of literature, and then from this review, identified articles that were germane to our report. A GAO economist then evaluated the methods used in the research and a GAO methodologist performed a secondary review and confirmed the summarized research findings. We reviewed 10 studies published between January 2009 and July 2019 that were relevant to our research objective on what the available evidence indicates about the extent of international TBML activities. We also reviewed one additional article published in 1999, which was frequently cited in other articles as a pioneer of measuring money laundering and included it in our review. To identify the practices international bodies, selected countries, and knowledgeable sources have recommended for detecting and combating TBML, we conducted a literature review to find relevant studies and other reports prepared by international bodies, industry groups, think tanks, academics, and foreign governments. We then analyzed these studies and reports to identify recommendations they made regarding practices for detecting and combating TBML. To gather further information regarding recommended practices for detecting and combating TBML and potential challenges in implementing such practices, we interviewed U.S. representatives of FATF and the Egmont Group, conducted interviews with UNODC officials, and obtained written responses to a set of questions from the WCO. We also spoke with U.S. embassy officials in five of the countries we selected for our nongeneralizable sample and obtained written responses from U.S. embassy officials in the sixth country. In addition, we spoke with host country officials in three of those countries. Finally, we spoke with selected knowledgeable sources. Through our work, we identified a range of recommended practices related to detecting and combating TBML. We grouped these recommended practices into five categories. We also identified examples of the steps that the U.S. government and other countries have taken to implement practices in each of these five categories. To examine the extent to which U.S. Immigration and Customs Enforcement has effectively implemented the TTU program, we collected information on HSI’s TTU program, including data on HSI’s TTU partner countries, the details on the TTU program’s operations, and documentation on the data system HSI developed to support the TTU program—the Data Analysis and Research for Trade Transparency System (DARTTS). We also evaluated HSI’s management of the TTU program by comparing the steps it had taken to establish a strategy and performance monitoring framework to requirements that DHS has established related to planning, programming, budgeting, and execution. To identify the steps HSI had taken, we interviewed HSI officials and reviewed relevant documentation on the TTU program. To examine the steps U.S. agencies have taken to collaborate with international partners to combat TBML, we also obtained and analyzed foreign assistance data, for fiscal years 2014 through 2018, from State on financial crimes and money laundering assistance programs it funded and from Treasury’s Office of Technical Assistance (OTA) on economic crimes assistance programs it funded. To assess the reliability of these data, we reviewed available documentation and interviewed knowledgeable U.S. officials. We determined that the State and Treasury OTA assistance data were sufficiently reliable for our purposes to present summary information on funding for assistance programs. We also reviewed other relevant U.S. government documentation describing training, technical assistance, or other support that U.S. agencies provided to partner countries to assist them in combating TBML or money laundering more broadly. For example, we reviewed selected performance reports for State anti-money laundering programs and selected end-of-project reports for Treasury OTA economic crimes programs. To gather information on the U.S. government’s collaboration with international bodies, we reviewed documentation from the Egmont Group, FATF, UNODC, and WCO describing the key activities of the bodies. Finally, as part of our work for this objective, to learn more about U.S. agencies’ work with partner countries and international bodies to combat TBML, we also interviewed U.S. officials in Washington, D.C. and interviewed U.S. embassy and host government officials in partner countries. We conducted this performance audit from January 2019 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. To determine the extent of trade-based money laundering (TBML) activities, we conducted a literature search for studies that sought to quantify potential illicit financial flows, including TBML. We considered existing studies from peer-reviewed journals, conference papers, dissertations, government reports, industry articles, and think-tank publications identified through searches the GAO librarian conducted of various databases, such as EconLit, Social SciSearch, and Scopus. We also asked for recommendations on relevant publications as part of our initial meetings with U.S. agencies and knowledgeable sources. After conducting the searches and relying on recommendations, we started the review with 82 studies. To assess the methodological quality of the studies, we relied on generally accepted social science standards. We examined summary level information about each piece of literature, and then from this review, identified 14 articles that sought to quantify potential illicit financial flows, including TBML. A GAO economist evaluated the methods used in the research, eliminated some research if the methods were not appropriate or not rigorous, and then summarized the research findings. In addition, a GAO methodologist performed a secondary review and confirmed our reported analysis of the finding. We further eliminated four studies and eventually identified 10 studies published between 2009 and 2019 that were relevant to our research objective on what the available evidence indicates about the extent of international TBML activities. We also identified one additional article published in 1999, which other articles frequently cited as a pioneer method of measuring money laundering, and included it in our review. See table 3 below for the list of studies included in our analysis. We found that estimating the extent of money laundering is a challenging task given that criminals seek to hide their illegal activities. Still, economic and statistical models have been developed that attempt to quantify the extent of such activities using various published datasets. However, none of the studies we identified in our literature review sought to develop estimates of TBML specifically and all the studies we reviewed capture activities that are generally broader than TBML to include tax avoidance, trade price manipulation, or trade misinvoicing, which demonstrates the difficulty in estimating the magnitude of TBML activity. In addition, according to the literature we reviewed, the studies we identified all had certain methodological limitations. We found that studies seeking to quantify potential money laundering activities, including TBML, have typically relied on one of four methods: (1) Walker gravity model, (2) unit price analysis, (3) trade mirror analysis, or (4) a theoretical model. One of the first researchers that attempted to measure money laundering is John Walker. In a paper published in 1999, he used what became known as the Walker gravity model to estimate the amount of money laundering globally. The gravity model states that the amount of trade from place A to place B depends on the size of the population in A, the “attractiveness” of B to people based in A, and the distance between the two places. The Walker model based the “attractiveness” of a place on four assumptions: (1) foreign countries with a tolerant attitude towards money laundering will attract a greater proportion of the funds than more vigilant countries; (2) high levels of corruption or conflict will deter money launderers, because of the risks of losing their funds; (3) countries with high levels of gross national product per capita will be preferred by money launderers, since it would be easier to “hide” their transaction; and (4) other things being equal, geographic distance, and linguistic or cultural differences, work as deterrents to money launderers. According to the literature we reviewed, the Walker gravity model has several limitations. First, because the flows of money laundering are unobservable, it is not possible to assess the quality of the formula. Second, although some factors in the attractiveness indicators are plausible, they are still arbitrary. Third, the researcher acknowledged that these figures represent only an interim set of results to show the types of output that would be derived from a fully developed model. These estimates are not his best and final estimates of money laundering around the world. Because of these limitations and considering the estimates are based on data that date to 1995, we did not present the estimates in the report. However, considering the importance of the Walker gravity model in the literature on measuring money laundering, we discussed this model in the report to provide context on methods used to quantify potential money laundering activities. A researcher used the unit price analysis to analyze U.S. trade data to quantify the magnitude of suspicious trade transactions. The database contains information at the transaction level that is reported to the U.S. Census Bureau from Shipper’s Export Declarations and U.S. Customs Service Entry Summary forms. The model follows the International Revenue Service’s definition of suspicious prices, which, according to the researcher, is defined as prices that are outside of the upper- or lower- quartile price range for each commodity in each country. He then aggregated the total dollar amount to come up with an estimate of the amount of suspicious trade. The researcher found that in 2018, total money moved out of United States through under-valued exports and over-valued imports was approximately $278 billion. Total money moved into the United States through over-valued exports and under-valued imports was approximately $435 billion. According to the literature we reviewed and information we received from the Census Bureau, we found that the unit price analysis approach has several limitations. First, the Census Bureau edits raw trade data received from Customs and Border Protection by automatically correcting unit prices that fall outside of its price parameters, which it establishes using industry analysis, input from public and private entities, and trend data. Of the total amount of export and import records in a specific month, roughly 18 percent to 22 percent contain some type of editing, according to the Census Bureau. The edited data with some extreme unit prices (those that fall outside of price parameters set by the Census Bureau) already “corrected” creates issues for the unit price analysis, which relies on identification of extreme unit prices. Second, the use of lower- or upper-quartile as price filters is somewhat arbitrary. For example, another study noted a fundamental weakness is that unit price analysis depends on the existence of a benchmark against which “abnormality” can be assessed. A lower benchmark would, in most product categories, produce more prices flagged as suspicious. Moreover, estimates from the unit price analysis also include other types of illicit activities in addition to TBML, such as income tax avoidance or evasion, among others. Therefore, this measurement of suspicious trade is generally broader than that of TBML. In addition, because of their focus on identifying suspicious prices, these estimates exclude other types of TBML that may not utilize over- or under-invoicing techniques, such as the Black Market Peso Exchange. The third approach, adopted by Global Financial Integrity and several other scholars, uses trade mirror analysis to estimate the amount of trade misinvoicing. This approach compares what country A reports as an export to country B and what B reports as an import from A (or vice versa). The calculation assumes the price and volume declared to both countries authorities would match after accounting for insurance and freight costs, and that any further difference between the trades reported by the countries indicates trade misinvoicing. In its latest report, Global Financial Integrity measured trade misinvoicing using two datasets. First, Global Financial Integrity relied on the International Monetary Fund’s (IMF) Direction of Trade Statistics and selected bilateral trade reports for 148 developing countries trading with 36 advanced economies from 2006 to 2015. Global Financial Integrity calculated potential trade misinvoicing as the import and export gaps, netted of the insurance and freight costs differentials. Second, Global Financial Integrity used United Nations Comtrade data to calculate trade gaps, where Comtrade gaps are calculated for each of the Harmonized System six-digit commodity classes available. Global Financial Integrity found that over the 10-year period of this study, potential trade misinvoicing amounted to between 19 and 24 percent of developing country trade on average. For 2015, it estimated that potential trade misinvoicing to and from these 148 developing countries were between $0.9 trillion and $1.7 trillion. According to the literature we reviewed, the Trade Mirror Analysis approach also has several limitations. First, alternative, legitimate reasons for import and export gaps may exist. For example, a researcher noted that “price volatility, transit and merchant trade, and the use of bonded warehouses can result in large trade data discrepancies arising from legitimate trade.” Another researcher also noted that major differences in customs import valuation methodologies and customs administration fees could contribute to trade data discrepancies. Moreover, accurate records may not always exist, especially in developing economies. Second, according to one researcher, the IMF and the United Nations, whose data these studies draw on, warn that the statistics cannot be reliably used in this way. The IMF says, “we caution against attempting to measure by using discrepancies in macroeconomic datasets…. fficial estimates of trade misinvoicing cannot be derived by transforming trade data from the IMF Trade Statistics and/or United Nations Comtrade, either by individual country or in aggregate.” Moreover, Global Financial Integrity defines trade misinvoicing as the fraudulent manipulation of the price, quantity, or quality of a good or service to shift money across international borders. Therefore, this measurement of trade misinvoicing is generally broader than that of TBML. However, certain types of TBML schemes are likely not included in the estimate of trade misinvoicing. For example, Black Market Peso Exchange schemes are likely not included because they do not require falsification of the price, quantity, or quality of a good or service. Another study sought to account for various factors that may lead to simple import-export discrepancies. The analysis focuses on under- reporting of Italian exports and over-reporting of Italian imports. The authors used a linear mixed model, where the dependent variable is the discrepancy in mirror statistics. The authors adopted a “residual approach,” in which the model controls for the main legal determinants of mirror statistics gaps, and the estimate residuals are proxy measures of the illegal component of such discrepancies. Using this approach, the authors were able to calculate irregular trade flows at country-sector level and rank countries and sectors by their risk levels. This approach uses economic theory to determine how much launderers would launder if they acted in an economic rationally manner. One study developed a theoretical model for estimating money laundering in the United States and the 15 countries that were in the European Union at the time. According to a researcher, the model assumes that “agents have the option to work partly in the legal economy and partly in the illegal economy. They face transaction costs in the legal sector and costs of being detected in the illegal sector. Two types of firms produce with two different technologies a legal good and an illegal good. The government sets fines, can influence the probability of detection, and can influence the liquidity of the economy. There is a liquidity constraint. If households want more liquid funds, they must engage in the illegal sector. The ‘optimal’ money laundered depends on the labor services allocated to the legal and illegal sector and on the prices and on the quantities of both goods.” The model uses parameters for the U.S. economy and for the European Union macro area and creates simulations to generate equilibrium allocations for money laundering. According to one study, this model has the advantage of having a solid micro-foundation, which helps to identify rational laundering behavior. However, the model is highly theoretical and has various unrealistic assumptions. For example, according to the model, without liquidity constraint in the economy, there would be no money laundering. Moreover, one of the parameters used in the model—the probability of being detected—is calibrated using data for the Italian economy from 1998 through 2000. Given the limitations discussed above and because the data date to 1998, we did not present the estimates in the report. However, considering that the theoretical model is one of the methods frequently discussed in the literature on measuring money laundering, we discussed this model in the report to provide context on methods used to quantify potential money laundering activities. In addition to the contact named above, Juan Gobel (Assistant Director), Ming Chen (Assistant Director), Ryan Vaughan (Analyst-in-Charge), Joyce Y. Kang, Pamela Davidson, Leia Dickerson, Neil Doherty, Toni Gillich, Jeff Harner, Georgette Hagans, Grace Lui, Dan Luo, and Aldo Salerno made key contributions to this report.", "summary": "TBML involves the exploitation of international trade transactions to transfer value and obscure the origins of illicit funds. Various observers have noted that although TBML is a common form of international money laundering, it is one of the most difficult to detect due to the complexities of trade transactions and the sheer volume of international trade, among other things. This report examines (1) what the available evidence indicates about the types and extent of international TBML activities, (2) the practices international bodies, selected countries, and knowledgeable sources have recommended for detecting and combating TBML, and (3) the extent to which ICE has effectively implemented the TTU program and steps the U.S. government has taken to collaborate with international partners to combat TBML. GAO analyzed U.S. agency and international body data and documentation, conducted a literature review, and interviewed U.S. officials and selected knowledgeable sources. Different types of criminal and terrorist organizations use trade-based money laundering (TBML) to disguise the origins of their illicit proceeds and fund their operations. TBML schemes can rely on misrepresenting the price, quantity, or type of goods in trade transactions, but other methods are also used. For example, some drug trafficking organizations from Latin America have used a type of TBML scheme known as the Black Market Peso Exchange (BMPE) to launder funds. BMPE schemes involve merchants who—wittingly or not—accept payment in illicitly derived funds, often from third parties to a trade transaction, for exports of goods. In carrying out TBML schemes, criminal and terrorist organizations use various goods, including precious metals and automobiles (see fig.). U.S. officials and other sources have identified a number of countries as being at particular risk for TBML schemes. Available evidence indicates that the amount of TBML occurring globally is likely substantial. However, specific estimates of the amount of TBML occurring around the world are not available. Officials and reporting from relevant international bodies and selected partner countries, and knowledgeable sources recommended various practices for countries to consider to combat TBML, which GAO grouped into five categories: (1) partnerships between governments and the private sector, (2) training, (3) sharing information through domestic interagency collaboration, (4) international cooperation, and (5) further research on challenges to combating TBML. The U.S. government's key international effort to counter TBML is the Trade Transparency Unit (TTU) program under the Department of Homeland Security's (DHS) Immigration and Customs Enforcement (ICE). ICE set up TTUs in 17 partner countries with the goal of exchanging and analyzing trade data to identify potential cases of TBML. While TTUs have played a role in some TBML investigations, the TTU program has experienced various challenges, including lapses in information sharing between ICE and the partner TTUs, differing priorities between ICE and partner TTUs in pursuing TBML investigations, and limitations in the data system that ICE and the TTUs use. However, ICE has not developed a strategy to increase the effectiveness of the TTU program or a performance monitoring framework to assess the results of its work with partner TTUs. As a result, ICE does not have a clear guide on how best to operate the TTU program and cannot make management decisions based on program results. In addition to the TTU program, the U.S. government collaborates with partner countries and international bodies through a range of other activities, such as developing international anti-money laundering standards, providing training and technical assistance, establishing information-sharing methods, and providing ongoing law enforcement cooperation. GAO recommends that DHS develop (1) a strategy to maximize TTU program effectiveness and (2) a performance monitoring framework for the TTU program. DHS concurred with the first, but did not concur with the second recommendation, citing data it already collects and challenges it faces. GAO continues to believe the recommendation is valid, as discussed in the report.", "document_type": "gao"}
{"report": "The FirstNet-AT&T network contract and its associated task orders define the requirements AT&T must meet. The contract currently involves five task orders, four of which relate directly to the network’s deployment. Task orders 1 and 2 (actions complete). Required AT&T to develop and deliver individual network deployment plans for each of the 56 states, territories, and the District of Columbia (hereafter, states). The governor of each state had the opportunity to review the plan and opt in to allow FirstNet and AT&T to build the network in their state. All governors opted in by the applicable deadline. The result of this process was a state deployment plan that included state-specific commitments made by AT&T. Task order 3 (actions ongoing). Requires AT&T to deploy, operate, and maintain the network’s “core” and all of its functions, and provide for the development of device and application ecosystems for the network. A network core consists of national and regional data centers and other elements that store, process, and secure network users’ traffic (activity), and interface with federal, state, and local networks. AT&T deployed the core in March 2018. The network uses the spectrum reserved for public-safety use (“Band 14”), as well as the spectrum that AT&T’s existing, commercial network operates on. When Band 14 spectrum capacity is not being used by public-safety users, AT&T can use the excess capacity for its non-public-safety, commercial-network users. As such, among the functions that task order 3 provides for are capabilities that allow prioritizing a public- safety user’s network access and traffic over other users and, when necessary, preempting other users altogether. These functions are commonly referred to as “priority and preemption.” Task order 4 (actions ongoing). Requires AT&T to deploy the network’s Band 14 coverage in the states, including building the “radio-access network” in each state that connects to the network’s core and backhaul (which carries network users’ traffic) and fulfilling the state-specific commitments. Radio-access networks consist of cell towers, sites, and other elements that connect network-users’ devices to the network core. This task order also requires AT&T to provide 72 “deployable” cellular assets—meaning, transportable equipment (typically in a vehicle) that can provide additional network coverage when needed—dedicated solely for FirstNet network users. The task order also provides for access to at least 300 additional deployables in AT&T’s fleet. The contract and task orders 3 and 4 outline a phased approach for deploying the network’s capabilities and coverage (in both non-rural and rural areas), with five “initial” operating-capability phases that build to a “final” operating capability expected in 2023, as well as ongoing performance, maintenance, and continuous improvement through 2042. As described further below, each phase provides for increased capabilities and coverage—and some outline goals for network user adoption—and AT&T must meet certain required milestones in each phase to receive payment for that phase from FirstNet. Figure 1 depicts the phased timeline for task orders 3 and 4. The Band 14 spectrum on which AT&T is building the network is a key component that differentiates it from other commercial networks, as the network’s full capabilities and functionality are only available via Band 14. For example, certain high-power user equipment can transmit at stronger signals; this signal increase can only be done using the Band 14 spectrum. However, at its expected final operating capability, the network using Band 14 spectrum will not cover the entire country. Public-safety network users will also have access to the non-Band 14 LTE spectrum that AT&T uses for its existing, commercial network (with priority and preemption), though this spectrum does not have all the full capabilities of Band 14, as in the high-power user equipment example above. According to AT&T, when including this non-Band 14 spectrum, the network will cover 76.2 percent of the U.S. geographically and around 99 percent of the population. Network users are to also have access, by request, to deployable assets that can provide temporary coverage when needed, such as in remote and wilderness areas that will not have permanent coverage. AT&T has met, and exceeded, the first required nationwide network- coverage milestone. According to FirstNet documentation, AT&T is required to meet certain coverage milestones in both non-rural and rural areas and by the end of March 2019, AT&T had met the requirement to provide at least 20 percent of the total expected Band 14 coverage in both non-rural and rural areas. The Band 14 coverage milestones that AT&T is contractually required to meet to receive payment increase each year through March 2023, when AT&T is to have completed 100 percent of the total expected Band 14 coverage. For example, by March 2021, the coverage milestones are 80 percent of the total expected Band 14 coverage in both non-rural and rural areas and by March 2022, 95 percent. Per the terms of the contract, prior to meeting the first milestone, AT&T provided initial coverage via its existing, commercial wireless network and made 72 deployables (such as mobile cell sites on trucks) available for network users. AT&T fulfilled the deployables requirement through a combination of deployables built specifically for network users and others allocated from AT&T’s existing fleet of deployables used for disaster relief. Specifically, to complete the first coverage milestone, AT&T delivered Band 14 coverage in about 63 percent of the total square miles required by 2023 in non-rural areas, and about 21 percent of the total square miles required by 2023 in rural areas, according to FirstNet documentation. For meeting this milestone, FirstNet paid AT&T approximately $1.2 of the $6.5 billion. Since completing this milestone, AT&T has continued to expand coverage and, according to FirstNet officials, is also on track to meet the next coverage milestone (due March 2020) early, although FirstNet was in the process of completing final verification and validation activities as of September 2019. AT&T constructed or “delivered” (i.e., these sites are all on-air) thousands of Band 14 cell sites to produce the level of coverage needed to meet the March 2019 milestone. Since then, according to FirstNet documentation, AT&T has continued adding Band 14 sites, delivering—as of July 2019— more than one-third of the total Band 14 cell sites planned for the entire network. AT&T may deliver these cell sites through a combination of constructing new sites, retrofitting existing AT&T sites, or acquiring or contracting with local providers, such as rural telecommunications carriers. Although FirstNet tracks the status of planned cell sites (such as which sites are undergoing environmental policy review or are currently operational, or on-air), cell sites are not an explicit part of the contractual coverage milestones required for AT&T to receive payment. That is, AT&T’s payment is not contingent upon getting a certain number, type, or location of cell sites on-air, but rather the amount of coverage (in square miles) provided on a nationwide level by these sites. While AT&T met the first coverage milestone and has delivered more than a third of the planned cell sites nationwide, AT&T also has state- specific commitments. These commitments or targets, like the delivery of sites, are not explicit contractual payment milestones. AT&T and the states negotiated the commitments during the state opt-in process, and AT&T delineated them in the state plans. For example, among states in our review, AT&T made commitments regarding the number of Band 14 cell sites, including new cell sites, and future coordination with state, local, or tribal authorities to discuss governance or priority coverage areas, among other things. According to our analysis of FirstNet documentation, progress toward meeting state-specific coverage commitments has varied. For example, among our case-study states as of July 2019, AT&T’s progress meeting the total coverage commitment in non-rural areas ranged from approximately 20 percent complete in one state to nearly 100 percent in others. In comparison, AT&T’s coverage progress in rural areas ranged from about 14 percent complete in one state to about 91 percent in another. Likewise, AT&T’s progress meeting state-specific commitments for delivery of Band 14 cell sites has varied across states. For example, in our case-study states, AT&T delivered between 9 and 71 percent of the total committed Band 14 cell sites as of July 2019. According to FirstNet documentation and officials, variances in state progress are allowable, as the contractual payment requirements focus on outcomes related to nationwide milestones. FirstNet documentation specifies that if the nationwide payment milestone was met, regardless of the amount of coverage that was deployed in a specific state, FirstNet deemed AT&T to have fulfilled that phase for all states. Moreover, FirstNet officials explained that multiple factors can contribute to delays or variance in progress across states, including natural and man-made disasters, subcontractor issues that AT&T must work through with local partners, and technical challenges common to cellular networks, such as degraded performance due to mixing of radio-frequency signals. Furthermore, FirstNet officials explained that AT&T has the first 5 years of the contract to meet all commitments made to the states. AT&T is on track to meet the first adoption milestone, which is to have a certain number of devices connected or subscribed onto the network (“device connections”) by the end of March 2020. FirstNet uses device connections as a proxy for adoption and has set or “forecasted” monthly targets that build up to the nationwide connections expected by March 2020. Our analysis of FirstNet documentation indicates that AT&T is making progress in meeting the monthly nationwide targets leading up to March 2020. Specifically, we found that AT&T was at approximately 165 percent of the July 2019 target. See figure 2 for a comparison of actual nationwide device connections versus the forecasted targets by month through July 2019. Furthermore, while AT&T must meet the nationwide device-connection milestone to receive payment for the phase ending March 2020, the targets are to be prorated depending on the month that AT&T meets the corresponding nationwide coverage requirement. Thus, if AT&T meets this requirement early (i.e., before March 2020), then the required adoption milestone is to be reduced accordingly. For example, if AT&T completes the coverage milestone in September 2019, then it would be required to meet a corresponding adoption target for that timeframe. While AT&T is on track to meet the nationwide, forecasted device- connection targets that serve as the payment milestone, our analysis found that there is variation in who is adopting the network. The targets are broken out by device connections associated with “primary” versus “extended-primary” users in different states. FirstNet defines primary users as those in the law-enforcement, fire, and emergency medical- services disciplines, whereas extended primary encompasses a myriad of other types of public-safety entities. For example, according to our analysis of FirstNet documentation, there are extended-primary users from transit agencies; public-utility and tow-truck companies; school districts; a state child-protective-services agency; airports; and television- media news outlets. Nationwide, with regard to primary users, AT&T was at 196 percent of the July 2019 target. For extended-primary users, AT&T was at approximately 106 percent of the nationwide target. These device connections are also distributed amongst the different types of public- safety entities. For example, for primary device connections, AT&T was at more than twice the forecasted nationwide target for law enforcement, as of July 2019. Our analysis also shows that there is wide variance in where adoption is occurring. Specifically, we found that AT&T is exceeding the device connection targets forecasted in certain states but lagging in others. Among our case-study states as of July 2019, for example, device connections for primary users in one state were more than 5 times the target, whereas in another state, AT&T had met only 33 percent of the target by July 2019. Adoption by extended-primary users among our case-study states also varied, with one state at 3 times the target compared to only 7 percent of the target met in another. Many types of devices are connected to the network and users’ experiences with network performance can vary based on the specific device they use. According to FirstNet documentation as of April 2019, 93 device types, 47 of which are Band-14 capable, were vetted and published on the list of devices certified for use on the network maintained by Commerce’s National Institute of Standards and Technology. Our analysis found that a variety of devices and device models are being used on the network, including smartphones, mobile hotspots, trunk modems, laptops, and tablets. As of July 2019, the most prevalent type of device was smartphones. FirstNet has acknowledged that user experiences on the network may vary depending on the type and model of device. Some public-safety officials we interviewed described inferior experiences on certain types or models of devices. In at least one case, AT&T worked with the public-safety entity to address identified device performance issues. Aside from device connections, FirstNet also tracks and has reported— via press releases, board presentations, and its most recent annual report to Congress—on the number of public-safety entities that have started using the network. For example, in April 2019, FirstNet reported to Congress that more than 7,000 public-safety agencies were using the network. This number represents agencies with at least one device connection, which may indicate piloting of the network. For example, one agency we interviewed had only about 2 dozen of its approximately 1,300 total devices on the network. Similarly, officials from multiple other public- safety agencies explained they were in the piloting phase (i.e., testing a small number or types of devices to gauge network performance) and that they were using or would continue to use another carrier for broadband services to ensure effective redundancy and emergency planning. According to FirstNet officials, AT&T provides the count of public-safety agencies at periodic program-review meetings and documents it in a required contract deliverable. We analyzed this deliverable and were able to approximate FirstNet’s reported numbers. FirstNet employs a variety of mechanisms to manage and oversee AT&T’s deployment of the network and monitor contract performance. We found that many of FirstNet’s approaches to managing and overseeing AT&T’s network deployment and contract performance generally align with the key contract-oversight practices identified in federal acquisition regulations and other government, academic, and industry guidance on contract oversight that we reviewed, as shown in table 1. We analyzed the key performance indicators and other documentation related to all 46 quality assurance elements that FirstNet monitors as of April 2019 and found that AT&T’s performance was rated as “excellent” in over half of these elements but “unsatisfactory” in almost a quarter. Regarding the number of unsatisfactory ratings, FirstNet officials stated that these ratings did not raise concerns given where AT&T was in the deployment lifecycle at the time of our review. That is, the rating may measure performance on an item that was not yet contractually due. For example, AT&T cannot achieve an excellent rating for certain elements that relate to coverage deployment until it is closer to the network’s final operating capability, expected in March 2023. Relatedly, according to FirstNet documentation as of April 2019, FirstNet had issued only one corrective action report since awarding the contract. According to FirstNet officials at the time of our review, although FirstNet has rejected or requested corrections to some items submitted by AT&T, no other concerns have risen to this level because they have been successful in resolving issues at lower levels first. FirstNet’s oversight activities leading up to the March 2019 coverage milestone were the first wherein it had to validate AT&T’s delivery of Band 14 coverage. FirstNet’s methodology for doing so included verifying AT&T’s prediction of the signal strength at which the necessary throughput—or, capacity, the amount of data transported successfully in a given time period—would be achieved, and reviewing AT&T’s lab and field tests. FirstNet then engaged in a process to verify the validity of AT&T’s coverage-prediction maps to ensure they were an acceptable representation of coverage in the field. Finally, FirstNet confirmed that the on-air coverage as compared to the expected total coverage at the network’s final operating capability met the contractual requirement. FirstNet’s methodology did not include conducting its own coverage tests in the field. According to FirstNet officials, FirstNet does not perform independent verification of network coverage in the field because FirstNet officials believe the contract provides an appropriate level of detail within the contractual deliverables and supporting information that is used to validate and verify the coverage milestones. While many of FirstNet’s contract-oversight mechanisms generally align with key practices, we found that some have weaknesses that limit their effectiveness. Specifically, FirstNet lacks: (1) a reliable master schedule to review, (2) communication with relevant stakeholders regarding contract oversight, and (3) meaningful information on end-users’ satisfaction to gauge performance quality. Key practices for contract oversight call for tracking the contractor’s performance and progress toward the expected schedule. Furthermore, GAO’s Schedule Assessment Guide identifies 10 best practices associated with effective scheduling, and they are grouped into 4 characteristics of a reliable schedule—comprehensive, well-constructed, credible, and controlled. The contract cites this guide when detailing the schedule’s requirements. As described above, AT&T must provide a current master schedule to FirstNet monthly. However, we found that FirstNet’s use of the schedule AT&T provides is limited because, based on our assessment, it only partially or minimally meets the characteristics of a reliable schedule, as shown in table 2 and described further below. Comprehensive. We found that the schedule did not reflect all of the work to be performed, precluding a comprehensive view of the entire program. For example, although a master schedule should be a comprehensive plan of all government, contractor, and subcontractor work that must be performed to complete the project, the schedule did not capture all government (e.g., FirstNet) activities or cover the entire contract period. Our schedule guide notes that management should be aware of how long government activities take because they often have a clear effect on schedules. An integrated master schedule should reflect all efforts necessary to successfully complete the program. Failing to include all work for all deliverables, regardless of whether they are the government’s responsibility or the contractor’s, can hamper program members’ understanding of the complete plan. Further, our analysis showed that there was a 1:1 detail-to-milestone ratio, meaning there was 1 detail activity for every milestone in the schedule, which is a low level of planning detail. Activities contained in the schedule did not always have manageable or reasonable durations; for example, over 50 percent of remaining activities had durations greater than 2 standard working months, with 25 percent of those having durations greater than 1 year. Our schedule guide notes that, for a schedule to provide a more accurate view of progress, longer activities should be broken down into smaller efforts where possible. While some of these activities had long durations because FirstNet expects AT&T to plan them in the future, some were not designated as such and had no other noted justification. Moreover, the schedule did not show any resources (i.e., labor, materials, travel, facilities, equipment, etc.). Our schedule guide also notes that resources must be considered in the creation of a schedule because their availability directly affects an activity’s duration, and a schedule without resources implies their unlimited supply and availability. Well-constructed. We found that the schedule had a high number of date constraints and an unreasonable amount of total float (or slack). For example, 60 percent of remaining activities and milestones in the schedule had “start-no-earlier-than” constraints. These date constraints confine the schedule by preventing tasks from starting earlier even if predecessor activities are completed ahead of schedule, which prevent the constrained activities from taking advantage of possible savings being introduced by predecessor activities. Our schedule guide recommends minimizing and justifying (in documentation) date constraints because they override the schedule’s logic and restrict how planned dates respond to accomplished effort. Schedules with constrained dates can portray an artificial view of the program and begin to look more like calendars than schedules. Moreover, over 50 percent of remaining activities had total float greater than 2 standard working months, with the average being over 200 days. In other words, activities in the schedule can slip an average of 200 working days before delaying the project’s finish date. Our schedule guide notes that without accurate values of total float, the schedule cannot be used to identify activities that could be permitted to slip and thus release and reallocate resources to activities that require more resources to be completed on time. Finally, while we found that the schedule had continuous critical paths, there was not enough detail activities to track the work necessary to achieve project milestones. Credible. We found that there was no risk analysis performed for the schedule. Our schedule guide notes that data about program risks should be incorporated into a statistical simulation to predict the level of confidence in meeting a program’s completion date; to determine the contingency, or reserve of time, needed for a level of confidence; and to identify high-priority risks. Additionally, our schedule guide notes that a schedule should be (1) “horizontally traceable,” meaning that it should link products and outcomes associated with other sequenced activities; such links are commonly referred to as “hand- offs” and serve to depict the relationships between different program elements and verify that activities are arranged in the right order, and (2) “vertically traceable,” meaning data are consistent between different levels of the schedule. Our analysis found that the schedule responded when significant delays were introduced into the planned activities; that is, when we tested the robustness of the schedule by extending activities’ durations, forecasted dates recalculated appropriately. However, as described above, we found that the schedule did not capture all activities or provide sufficient detail, meaning it cannot be fully traceable horizontally. We also found that, in general, the schedule provided good vertical traceability—that is, dates were traceable between status reports and the schedule. However, when we compared other reported information to the schedule, there were instances where this traceability was not the case. For example, one monthly report stated that baseline information was included for all tasks and milestones of a particular task order, but we found that the schedule did not in fact include this information. Vertical traceability provides assurance that the representation of the schedule to different audiences is consistent and accurate. Controlled. We found that the schedule was updated regularly using actual progress and logic by trained AT&T personnel, with supporting documentation and review procedures. We also found that not all activities in the schedule had baseline dates. According to FirstNet officials, portions of the schedule are baselined on a rolling basis once the next requirements traceability matrixes are created. However, some activities with no baseline dates had already begun or been completed. Further, FirstNet officials stated that no “basis document” exists for the baselined schedule. Our schedule guide notes that a corresponding basis document is important because it explains the overall approach to the program, defines custom fields in the schedule file, details assumptions used in developing the schedule, and justifies constraints, lags, long activity durations, and any other unique features of the schedule. Furthermore, while AT&T was submitting schedule variance information, it covered only tasks that had been baselined, when the majority of activities in the schedule were missing baseline dates. Without formally established baseline-schedule start and finish dates to measure performance against, FirstNet is limited in how it can use the schedule to identify or mitigate the effect of unfavorable performance. Overall, FirstNet officials said they are not concerned about the gaps in the AT&T master schedule for a variety of reasons. Namely, officials stated that FirstNet entered into a contract with AT&T that lays out specific milestones that AT&T must meet or it does not receive payment. Accordingly, they said that the summary level of detail is sufficient for FirstNet’s purposes, as AT&T’s program management office determines what activities are appropriate to track to meet those milestones and AT&T maintains its own, more detailed schedule. They further added that given the firm-fixed price nature of the contract, it is not practical or helpful for FirstNet to collect information on the resources for AT&T’s deliverables; if it takes AT&T 50 or 50,000 individuals to complete the requirement that decision is for AT&T to determine. As such, although the contract cites GAO’s schedule guide when detailing the schedule’s requirements, FirstNet excluded requirements related to resources. Similarly, FirstNet excluded requirements related to schedule risk analysis primarily, according to FirstNet officials, because risks to the established schedule milestones were largely considered when evaluating AT&T’s proposal prior to contract award. Finally, FirstNet officials highlighted that the schedule is not the only measure for progress and reporting, noting that it employs many other mechanisms to monitor and oversee AT&T’s progress and performance, and discusses the schedule during program management review and other meetings with AT&T. However, the contract itself states that FirstNet is responsible for ensuring the overall success of the network and that, to do so, its responsibilities after contract award include overseeing the program schedule. Regarding resources in particular, the contract also states that these responsibilities include managing schedule resources. Thus, while it may not be necessary for FirstNet to collect information from AT&T on every resource detail, as FirstNet has stated, it is nevertheless important for FirstNet to gain an understanding of the overall resources needed to complete the work. This understanding could include, for example, evidence that sufficient resources were assigned to activities in the more detailed schedule that AT&T maintains. Our schedule guide notes that resources must be considered in the creation of a schedule—and it is important that FirstNet have sufficient insight into those resources—because their availability directly affects an activity’s duration. Regarding schedule risk analyses, consideration of risks to the milestones prior to contract award may not serve as a substitute for a risk analysis of the current schedule, which would include detail on activities and risks that could not have been known or fully understood prior to the award. Finally, while FirstNet utilizes a variety of other mechanisms to oversee AT&T’s performance, having a more detailed master schedule from AT&T would strengthen FirstNet’s use of the schedule as a management and oversight tool. For example, such a schedule could improve FirstNet’s insight into the activities driving AT&T’s deployment of the network and completion of requirements, how each activity relates to others, and any potential risks. It could also provide FirstNet with additional information that could help it and AT&T manage tradeoffs and make decisions to maximize the program’s success across the entire country. Key practices for contract oversight call for communicating appropriate information to relevant stakeholders and reporting on monitoring results. Additionally, the 2012 Act requires FirstNet to consult—via a designated single point of contact (SPOC) in each state—with regional, state, local, and tribal jurisdictions regarding a host of activities, such as: ongoing compliance review and monitoring of the management and operation of the network; practices, procedures, and standards for the management and operation of the network; terms of service for use of the network; radio-access network build out, placement of cell towers, and coverage areas; and assignment of priority and selection of entities seeking use of the network. Furthermore, the contract requires AT&T to report, by state, on the state- specific commitments made as a result of the state opt-in process. Portions of this report are to be shareable with states, and it is to detail the deadline by which the commitments will be fulfilled, the status of fulfilling them, and include evidence of the state’s satisfaction with progress. Beginning April 2018, AT&T is required to deliver this report semi-annually. Although two such state-specific commitment reports were due as of July 2019, only one has been completed by AT&T and accepted by FirstNet. Additionally, according to FirstNet officials as of October 2019, the report was not shared with the states. Numerous state, local, and tribal stakeholders we interviewed described having had very little contact with FirstNet or being generally dissatisfied with the level or quality of information they had received from FirstNet and AT&T. These officials said that FirstNet had communicated little to no information on AT&T’s progress deploying the network in their area, or if and how FirstNet was monitoring performance. For example, many officials said that they had limited interaction with FirstNet beyond public relations emails or events promoting the network, or noted that their interactions lacked substantive information and details that would be of more value. The SPOCs were particularly dissatisfied with the lack of transparency surrounding the contractual requirements or FirstNet’s oversight of progress to date. Many of these state officials noted that the level of communication and information shared by FirstNet post contract award stood in stark contrast to the level of engagement prior to the state’s opt-in decision. Numerous state, local, and tribal stakeholders we interviewed said that additional information on AT&T’s deployment and FirstNet’s oversight would be helpful or that greater transparency was needed. Officials wanted additional information on, among other things: contract requirements, milestones, and progress; technical details on the network including operational status and location of cell sites; subscribers within the official’s agency or agencies across the state that had adopted the network; and FirstNet’s oversight activities and results, including assurance from FirstNet that network coverage and performance had been verified. Even public-safety officials who were pleased with their experiences on the network to date or their relationship with FirstNet representatives reported that having more information was important. In the absence of this type of information, many public-safety entities we contacted expressed concern that they did not know whether FirstNet was holding AT&T accountable. For example, several officials indicated they did not know whether FirstNet or AT&T was “running the show.” State, local, and tribal stakeholders we interviewed gave a variety of reasons for wanting greater transparency on contractual requirements and oversight. Numerous public-safety officials said that they needed to know this information for tactical response and planning, or state and local contracting purposes. For example, some local public-safety officials described wanting to have basic information on the contract coverage phases in their states so that they could confidently plan out equipment lifecycles. Additionally, many SPOCs said that there was a duty for FirstNet as the contracting agency to oversee that state-specific commitments were met. Many SPOCs also stated that their attempts to obtain more information from FirstNet or AT&T per the agreed-upon commitments had been delayed. At times, when they reached out to FirstNet, they were directed back to AT&T, or vice versa. Numerous stakeholders agreed that given the nature of the network as a public resource—involving public investment and funds, with the expressed purpose of serving public safety—they expected greater transparency from both FirstNet and AT&T. FirstNet officials provided several reasons for not communicating the additional information cited by the stakeholders we spoke to and for not reporting on monitoring results. In particular, FirstNet officials told us there is no contractual requirement to communicate or share information collected, including any performance information or monitoring results, with any stakeholders or network users. However, its Public Safety Advocacy team serves as the primary interface to the public-safety community and conducts considerable outreach to stakeholders, as described above. Regarding the SPOCs, the officials further said that they believe the 2012 Act’s consultation requirement applied only to the initial planning stages (namely, the development of the request for proposal prior to contract award). As such, they do not believe they are legally obligated to continue to communicate specifically as identified in the 2012 Act. Additionally, FirstNet has stated that much of the information AT&T provides is proprietary and, therefore, cannot be disclosed to stakeholders. Finally, regarding the state-commitments report, FirstNet officials have said that FirstNet shares subsets of this information with states that request it during consultative interactions with FirstNet and in coordination with AT&T, but does not routinely share the full report to protect confidential commercial or trade-secret information. While the 2012 Act does require consultation to occur “in developing requests for proposals,” it also states “and otherwise carrying out its responsibilities,” suggesting a broader application than just the initial planning stages, which is FirstNet’s interpretation. Moreover, while there are valid concerns about disclosing proprietary information and statutory prohibitions on doing so, there are opportunities for FirstNet to communicate additional information in ways it deems appropriate. For example, communicating how it oversees AT&T, the mechanisms it employs, and the performance areas it monitors could be done in a manner that does not disclose proprietary AT&T information, as these are government activities. Additionally, a state official and some local government officials we spoke to said that certain AT&T commercial information (e.g., the location of cell towers) could already be publicly available through local permitting offices. Further, federal internal-control standards note that management may select appropriate methods for external reporting, meaning management can consider what methods are appropriate for different audiences when communicating and reporting information. Finally, the contract states that except as specifically indicated or with explicit written permission from FirstNet, AT&T’s deliverables documentation shall not contain proprietary information or have any restriction on reproduction and/or distribution, suggesting that upon awarding the contract, FirstNet recognized the value of limiting these instances. Industry guidance on project management that we reviewed—and which is cited in the contract—notes that analyses of high-profile project failures highlight the importance of stakeholder engagement. It also notes that communicating with stakeholders in an appropriate way can mean the difference between a project’s success and failure. Stakeholders’ lack of information on the program and FirstNet’s oversight of AT&T can make it difficult for stakeholders to assess what benefits have, or have not, been realized, which may affect their enthusiasm and continued support of the program. This scarcity of information has also left them speculating about other matters such as what, if any, oversight FirstNet conducts of AT&T. By not communicating additional information and reporting on monitoring results, FirstNet could be unknowingly reinforcing nascent skepticism of the program overall and of itself as the entity charged with holding AT&T accountable. Key practices for contract oversight call for obtaining information on end- users’ satisfaction that can be used as a metric to gauge performance quality. For example, industry guidance on program management emphasizes that end-users’ satisfaction is a powerful metric that should be obtained to gauge program quality, noting that the benefits, product, or service delivered is best evaluated by those who receive it. While FirstNet collects some information—via its QASP monitoring, as described above—that could relate to end-users’ satisfaction, these metrics provide limited insight into users’ experiences. For example, although AT&T surveys some customers to ask them whether they would recommend FirstNet services to a colleague to satisfy a QASP requirement, a user could recommend the service not because they are satisfied but because they have limited alternatives. Indeed, while many state and local public-safety officials we spoke to were pleased with their experience migrating to or piloting the network, numerous officials told us about experiences that fell short of their expectations for a public-safety broadband network backed by the government. Numerous officials told us that they had concerns about misleading or disorganized sales tactics from AT&T representatives. For example, while some officials said that their AT&T representative had been candid in explaining the limited available coverage in their area, many officials told us about instances when AT&T representatives had shown them maps depicting more coverage than actually existed or that were insufficiently granular for their mission work. Similarly, while many officials recounted positive experiences with network coverage or performance or AT&T representatives, many also described instances when equipment failed to work or perform as expected during piloting phases or exercises. In some instances, these officials stated that FirstNet or AT&T representatives explained, after the fact, that differences in user experience were to be expected depending on the device model or subscriber identity module (SIM) card being employed. Specifically, FirstNet or AT&T officials explained that the optimal performance could only be achieved when Band 14 devices connected to a Band 14 cell site. According to FirstNet officials, the best experience will be when subscribers use a Band 14-capable FirstNet-ready device with a FirstNet SIM card while in a Band 14 coverage area. The officials said any other combination could result in slightly degraded performance or features being unavailable. This is notable given that Band 14 coverage is still limited and generally state and local public-safety officials do not have insight as to where these sites were located or when, if ever, coverage will be expanding, as previously discussed. As stated above, at its final operating capability, the network utilizing Band 14 spectrum will not cover the entire country. Many officials also expressed concerns about the network’s quality of service, priority, and preemption capabilities over the long run or during a catastrophic event. They speculated about the type or expanding number of subscribers allowed on the network or whether at some point in the future, the network would become saturated because non-public safety organizations or individuals (either extended-primary users or non-verified public-safety subscribers) were being granted priority and preemption capabilities. Exacerbating these concerns, many officials noted that they did not have insight into who had subscribed even within their own agency or state, or lacked confidence in how FirstNet or AT&T verifies individuals’ public-safety status, based on anecdotal experiences. Further, some officials also raised concerns about their inability to test the network during congested periods or simulate catastrophic power failures and lack of insight into if or how AT&T had hardened the network. Many officials discussed or shared after-action reports or their testing results with us, and several communicated that they had shared or would be willing to share such information with FirstNet as well to support validation of the network’s actual performance. According to FirstNet officials, the key performance indicators identified via the QASP are the performance quality measures, not end-users’ satisfaction. They also stated that “disincentive” payments embedded in the contract serve as an incentive for AT&T to ensure end-users’ satisfaction. Specifically, if AT&T does not meet user adoption (i.e., device connection) goals specified in the contract, it has to make payments to FirstNet on a timetable identified in the contract. Additionally, according to FirstNet officials, they informally hear information on end- users’ satisfaction and the network’s performance through many of the engagements its Public Safety Advocacy team conducts, which they can informally share with AT&T. However, disincentive payments (and the user-adoption goals tied to them) may be a limited reflection of end-users’ satisfaction for various reasons. For example, users may continue to subscribe to the service not because they are satisfied with it but because agency procurement lifecycles and budgets prevent them from changing providers, or because they find it difficult to break a sales contract, have already sunk costs into the transition, or lack alternatives in the market. Additionally, if AT&T perceives that the value derived from its commercial customers’ use of the excess Band 14 spectrum capacity is greater than the disincentive payment it must make to FirstNet, it may view making the payment as an acceptable tradeoff. Alternatively, aggressively pursuing sales contracts with potential public-safety users to avoid the payments may not be welcomed by the public-safety community, which could result in negatively, not positively, affecting end-users’ satisfaction, as some public-safety network users we spoke to said it had. Finally, while the informal collection and sharing of information on satisfaction can be valuable, it does not serve as a formal performance-quality measure, which could provide FirstNet with additional recourse should issues arise. End-user adoption is both a goal of the program and how AT&T plans to fund the $40 billion of investment in the network. Adoption may be driven by satisfaction in addition to need. Ultimately, end-users’ dissatisfaction could affect the success of the program. Thus, FirstNet’s lack of formal insight into end-users’ satisfaction hampers its ability to take actions that could increase the program’s chance of succeeding. By not obtaining and using this information to inform its oversight or related activities, FirstNet could be missing an opportunity to increase assurance of the program’s long-term success. The FirstNet public-safety broadband network has the potential to save lives every day. Since beginning their 25-year partnership, AT&T has made progress deploying the network and meeting contractual milestones and goals, and FirstNet has employed a variety of mechanisms—many of which align with key practices—to oversee AT&T’s performance. However, the success of the network depends not only on AT&T’s contract execution and FirstNet’s oversight but also on the confidence of the end users, the nation’s first responders. As FirstNet enters the next phases of its partnership with AT&T, it could reduce the risks to the network’s long-term success by strengthening its schedule oversight; increasing transparency, communication, and reporting of additional information to states and other public-safety stakeholders; and obtaining and using meaningful information on the satisfaction of the first responders for whom the network is intended. We are making the following four recommendations to FirstNet: FirstNet’s Chief Executive Officer should take steps to ensure that the integrated master schedule for the program is developed and maintained in accordance with the best practices provided in GAO’s Schedule Assessment Guide. (Recommendation 1) FirstNet’s Chief Executive Officer should identify additional information about the program, including FirstNet’s oversight and monitoring activities, that can be shared with public-safety stakeholders and periodically communicate and report this information to them. (Recommendation 2) FirstNet’s Chief Executive Officer should share relevant portions of the accepted state-specific commitment reports with the states, as specified in the contract. (Recommendation 3) FirstNet’s Chief Executive Officer should, in consultation with public- safety stakeholders and its contractor, as appropriate, identify and obtain periodic information or meaningful indicators on end-users’ satisfaction that would serve as a metric to gauge performance quality, including the effect of the FirstNet network and products on public-safety operations. (Recommendation 4) We provided a draft of the sensitive report to FirstNet for review and comment. FirstNet’s comments on the sensitive report are reprinted in appendix II. In these comments, FirstNet stated that it agreed with all of our recommendations; will take appropriate additional steps to apply lessons learned and address our concerns; and will continue to find ways to improve transparency with and feedback from its stakeholders, in addition to refining the integrated master schedule. Separately, FirstNet also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Chief Executive Officer of FirstNet, the Secretary of Commerce, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines the extent to which (1) AT&T is meeting the established milestones for deploying the nationwide public-safety broadband network, including coverage and adoption goals, via its contract with the First Responder Network Authority (FirstNet), and (2) FirstNet is overseeing AT&T’s deployment of the network in accordance with key practices. To assess progress toward the coverage and adoption milestones, we reviewed the FirstNet-AT&T network contract, corresponding task orders, and relevant documentation contained in FirstNet’s contract files, including information or “deliverables” submitted by AT&T that had been reviewed by FirstNet for contract compliance. We also reviewed additional FirstNet documentation, such as board-meeting materials, annual reports to Congress, press releases, fact sheets, and official blog postings. We reviewed the Middle Class Tax Relief and Job Creation Act of 2012 (the 2012 Act), which created FirstNet as an independent authority charged with establishing a nationwide public-safety broadband network that would, among other things, be deployed in phases that included substantial coverage milestones in rural areas. Within the contract, we identified the various coverage and adoption milestones and focused our analysis primarily on task order 3, phase 3 (which spanned March 31, 2018, to March 30, 2019) and task order 4, phase 2 (which spanned October 1, 2018, to March 30, 2019) milestones. We focused on these task orders because they are most relevant to the network’s coverage deployment and adoption, and on these phases because they were the phases under way at the time we began our review. We did not review activities or progress as described in AT&T deliverables dated beyond September 2019 given the timing of our review. We also did not make any conclusions about progress toward the final phases of these task orders. However, we did assess the master schedule to determine its reliability and validity for planning and tracking progress toward the final phases as described further below and in our report. The contractual deliverables that we reviewed in some cases included detailed data broken out by state and public-safety discipline. In particular, we analyzed data that indicated progress toward nationwide and state Band 14 network coverage (in square miles); cell site delivery; monthly adoption targets (i.e., device connections) by discipline; and types of devices connected. When analyzing these data, in all cases, we used the most currently available data at the time of our request for the information, and we report data as of September 2019. Although all data were the most currently available as of September 2019, because the deliverables have varying cycles for when AT&T is contractually required to report the information, we specify throughout the report the “as of” period these data represent. We assessed the reliability of these data by asking FirstNet officials questions about how they review the deliverables and about data sources, quality, and timeliness, as well as by electronically testing the dataset for missing or invalid entries. We removed a small number of missing or invalid entries from our analysis of device types and models and count of public-safety agencies. We did not assess AT&T’s underlying systems or databases, nor did we interview AT&T officials about their protocols for producing this data. We found these data reliable for the purpose of describing FirstNet’s current and projected progress toward coverage and adoption milestones for the related task orders and phases. To further assess deployment progress, we conducted case studies of seven states to illustrate and obtain greater context on variations in state- level coverage and adoption. We selected our case-study sample to include states that had very high-density counties; relatively large numbers of low-population density counties; high poverty rates (due to budgetary challenges public-safety entities may face); varying levels of progress in cell site delivery as of January 2019 (the most currently available data at the time of our selection); and geographic diversity and tribal lands. In total, the selected states represent almost a third of the contract dollars allocated for network coverage deployment. Our case- study analyses included reviewing and comparing the deployment plans and commitment letters for these seven states (detailing the agreed-upon, state-specific commitments AT&T made to these states) against the deliverables describing the progress AT&T made on some of these commitments, as of July 2019. It also included interviewing state, local, and tribal officials and first responders from these states, as described further below. The case studies and stakeholders’ views illustrate experiences with FirstNet’s deployment of the network across a wide cross section of geographies and network users to date but are not generalizable to those of all FirstNet stakeholders or the network as a whole. We also interviewed FirstNet officials to obtain their perspectives on AT&T’s progress and factors that may explain the variance across states. To examine FirstNet’s oversight efforts, we reviewed the FirstNet-AT&T network contract and documentation contained in FirstNet’s contract files, as well as additional FirstNet documentation. In addition to the material described above, this documentation included, for example, the Quality Assurance Surveillance Plan, requirements traceability matrixes, verification reports, memos, Contract Administration Plan, FirstNet Acquisition Manual, guidance documents on contract management and procedures, and FirstNet officials’ written responses to questions we posed. For the same reasons described above, we focused primarily on material related to task order 3, phase 3 and task order 4, phase 2. We interviewed FirstNet officials to obtain greater context on FirstNet’s oversight mechanisms and their use, and to observe FirstNet’s verification activities and the platform it uses to manage its contract files. Further, we reviewed key acquisition and contract-oversight practices established in the Federal Acquisition Regulation and the Commerce Acquisition Regulation, as well as the Commerce Acquisition Manual and other academic and industry guidance. We also reviewed the 2012 Act and federal standards for internal control. We selected those practices that were most appropriate given FirstNet’s contract approach (i.e., Indefinite-Delivery/Indefinite-Quantity, Firm-Fixed-Price contract vehicle) and the stage of the acquisition process FirstNet was in during the course of our review. We assessed FirstNet’s oversight efforts against these practices. We also compared the network’s integrated master schedule, which AT&T provides to FirstNet, to scheduling best practices in GAO’s schedule guide. Collectively, these best practices are organized into four characteristics of a reliable schedule. A schedule is considered reliable if each of the four characteristics is substantially or fully met; if any of the characteristics are not met, or minimally or partially met, the schedule cannot be considered reliable. We reviewed the schedule as of its status date January 31, 2019, which represented the latest status update to the schedule at the time we began our schedule analysis. In reviewing the schedule, we also reviewed the schedule dictionary, work breakdown structure, and program management review or monthly progress reports dated October 2018 to January 2019, among other documents. We provided our criteria and draft schedule analyses to FirstNet for review. To inform both of our objectives, we conducted about 40 interviews with state, local, and tribal officials and first responders. These interviews represented almost 30 different states’ single point of contact (SPOC) to FirstNet or their designees, and over 30 different state, local, or tribal public-safety entities. The public-safety entities we interviewed included police and fire departments, sheriffs’ offices, emergency medical-services providers, and emergency-management agencies, among others. We interviewed the SPOC from each of our case-study states and received information from other SPOCs (or a designee) via a multi-state focus-group discussion and written responses to the semi-structured discussion questions and prompts we posed. A GAO moderator led the discussion to establish ground rules and keep participants focused on the specified issues within the discussion time frame. We selected state, local, and tribal public-safety entities within our case-study states to interview. To select the state and local public-safety entities to interview, we reviewed the AT&T subscription management report provided to FirstNet as of February 2019 (the most current available at the time of our selection) and asked the SPOCs for recommendations within their state. Generally, we selected among the largest subscribers (meaning, the most number of devices on the network) in each of the primary public-safety disciplines (law enforcement, fire, emergency-medical services) in each state, and selected others to ensure representation among urban, suburban, and rural areas. To select the tribal entities to interview we asked the National Tribal Emergency Management Council for a recommendation in each state. Not all public-safety entities accepted our interview requests. Among our case-study states, we conducted a site visit in one state region. We selected this region for our visit because of the concentration of subscribers within reasonable geographic proximity to each other. For additional context, during this visit we also met with the FirstNet Public Safety Advisors that serve the state and attended a FirstNet presentation and town hall meeting hosted by the local chapter of the Association of Public Safety Communications Officials. Because stakeholders varied in their expertise with various topics, not every stakeholder provided an opinion on every topic. Throughout this report we refer to “some” stakeholders if officials from 3–5 entities, “several” if 6–9, “many” if 10–19, and “numerous” if 20 or more expressed the view. Finally, for additional perspective we also interviewed the National Public Safety Telecommunications Council because of its role as a federation of organizations whose mission is to improve public-safety communications and interoperability. As noted above, stakeholders’ views are not generalizable to those of all FirstNet stakeholders. In addition to the contact named above, Sally Moino (Assistant Director); Nalylee Padilla (Analyst in Charge); David Aja; Melissa Bodeau; Andrew Burton; Mark Goldstein; Yvette Gutierrez; David Hooper; Jason Lee; Andrew Stavisky; Hai Tran; William Woods; and Friendly Vang-Johnson made key contributions to this report.", "summary": "Public-safety officials such as police officers and firefighters rely on communications systems to do their jobs. The Department of Commerce's FirstNet must establish a nationwide public-safety broadband network for use by these officials. In March 2017, FirstNet awarded a 25-year, multibillion-dollar contract to AT&T to deploy, operate, and maintain the network. AT&T must meet milestones specified in the contract, such as for providing network coverage and for the network's adoption. FirstNet's oversight of AT&T's progress and performance is critical given the contract's scope and duration. GAO was asked to review FirstNet's progress and oversight. GAO examined the extent to which (1) AT&T is meeting milestones for the network's coverage and adoption and (2) FirstNet is overseeing AT&T in accordance with key practices. GAO analyzed FirstNet and AT&T documentation; assessed FirstNet's oversight efforts against key contract-oversight practices identified in federal regulations and other government, academic, and industry guidance; and assessed the program's master schedule against GAO best practices. GAO interviewed FirstNet officials, and selected state, local, and tribal officials and first responders representing a variety of viewpoints. Although not generalizable, they provided useful perspectives. AT&T is meeting—or on track to meet—all nationwide, contractual network coverage and usage (adoption) milestones for the First Responder Network Authority (FirstNet) public-safety broadband network. AT&T has met the first nationwide coverage milestone (20 percent of the final expected coverage by March 2019), but coverage varies across states. Similarly, AT&T is on track to meet the first nationwide adoption milestone (which is to have a certain number of devices connected to the network by March 2020). AT&T has exceeded adoption targets in most states but lags in others. According to FirstNet officials, variances by state are allowable, as the key milestones are nationwide. FirstNet uses various mechanisms to oversee AT&T; many of which align with key contract-oversight practices. For example, FirstNet uses a quality assurance surveillance plan to evaluate AT&T's performance. However, GAO found that FirstNet lacked (1) a reliable master schedule to review, (2) communication with relevant stakeholders regarding contract oversight, and (3) meaningful information on end-users' satisfaction to gauge performance quality. Schedule. AT&T is required to provide a current master schedule to FirstNet monthly, but the schedule only partially or minimally meets the characteristics of a reliable schedule per GAO best practices. For example, the schedule only partially captures all activities or the duration or sequence of activities. Key practices call for tracking a contractor's progress toward the expected schedule. Having a more detailed schedule to review could improve FirstNet's insight into AT&T's deployment and strengthen FirstNet's use of the schedule as a management tool. Stakeholder communication. Numerous public-safety officials GAO interviewed were dissatisfied with the level or quality of information received from FirstNet, noting that FirstNet had communicated little to no information on AT&T's progress or FirstNet's oversight. FirstNet officials said there is no contractual requirement to share such information, but key practices call for communicating appropriate information to relevant stakeholders and reporting on monitoring results. The lack of information has left stakeholders speculating about what, if any, oversight FirstNet conducts; sharing more information about the oversight FirstNet conducts could improve public-safety sentiment for and support of the program. End-users' satisfaction. FirstNet collects some information that could relate to end-users' satisfaction, but this information provides limited insight into users' experiences. For example, AT&T surveys some users to ask whether they would recommend FirstNet services, but a user might do so due to limited alternatives, not satisfaction. Although end-users' satisfaction is not a performance quality measure in the contract, key practices call for using end-user satisfaction information as a metric to gauge performance quality. By not using this information to inform FirstNet's oversight or related activities, FirstNet could be missing an opportunity to increase assurance of the program's long-term success. This is a public version of a sensitive report that GAO issued in December 2019. Information that FirstNet deemed proprietary has been omitted. GAO is making four recommendations, including that FirstNet ensure the schedule aligns with GAO best practices, share additional oversight and other information with appropriate stakeholders, and utilize end-user satisfaction information to gauge performance. FirstNet agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "The Rogue River-Siskiyou National Forest, located mainly in southwestern Oregon and extending into northern California, encompasses nearly 1.8 million acres. The west side of the forest lies within the Klamath-Siskiyou ecoregion, which is known for its ecological diversity, with 28 coniferous tree species and numerous rare and endemic plants. The forest also contains diverse topography, with steep terrain and rugged geological features across several mountain ranges, including the Klamath Mountains, Siskiyou Mountains, Cascade Range, and Coast Range. Access to the forest is limited, due to many roadless areas and over 340,000 acres of wilderness, including the 180,000-acre Kalmiopsis Wilderness, where the Chetco Bar Fire began. Cities and communities in Oregon near the fire include Brookings and Gold Beach—along the coast of the Pacific Ocean—as well as Agness, Cave Junction, and Selma in Curry and Josephine counties. Figure 1 shows the final perimeter of the fire in southwest Oregon. The part of southwestern Oregon where the Rogue River-Siskiyou National Forest is located is a fire-adapted ecosystem, meaning that most native species and plant communities have evolved with fire, and many are adapted to or dependent on periodic wildfires. The historic fire interval in the area where the Chetco Bar Fire occurred varied, as did the historic severity of fires, according to a Forest Service ecologist. The forest experienced a number of fires over the 30 years before the Chetco Bar Fire occurred. In 1987, the Silver Fire burned nearly 100,000 acres. Fifteen years later, in 2002, the Biscuit Fire burned nearly 500,000 acres, including areas previously burned by the Silver Fire. The Chetco Bar Fire started in the areas burned by both the Silver and Biscuit Fires. In 2018, the year after the Chetco Bar Fire, the forest experienced another large fire, the Klondike Fire, which burned about 175,000 acres, abutting the burn scar of the Chetco Bar Fire in some places. The occurrence of large fires in the western United States has been increasing, while, at the same time, fire seasons have been increasing in length, according to recent assessments. Some of these assessments have found that these increases are due in part to climate change, which has contributed to increasing temperatures and droughts in the West, as well as a later onset of fire-season-ending rains. We have previously found that the cost of disasters, including wildfires, is projected to increase as extreme weather events such as droughts become more frequent and intense due to climate change. Moreover, land use practices have increased the risk that severe and intense wildfires will affect people and communities. As we have previously described, land use practices over the past century have reduced forest and rangeland ecosystems’ resilience to fire. In particular, fire suppression—with 95 percent or more of fires suppressed for nearly a century—and timber harvesting and reforestation have contributed to abnormally dense accumulations of vegetation, and these accumulations can fuel uncharacteristically large or severe fires. In some parts of southwestern Oregon, significant vegetation has built up, according to Forest Service and other documents. As a result, southwestern Oregon, as well as other parts of the country, is under high to very high risk from fire, according to a risk assessment and Forest Service presentation. At the same time, development in and around wildland areas continues to increase, placing more people, businesses, and infrastructure at risk of being affected by fires. Because a single firefighting entity may not be able to handle all wildfires in its jurisdiction, agencies in the United States use an interagency incident management system that depends on the close cooperation and coordination of federal, state, tribal, and local fire protection agencies. The Forest Service is the predominant federal firefighting agency in terms of funding. Other federal firefighting agencies include the Bureau of Indian Affairs, BLM, Fish and Wildlife Service, and National Park Service. Federal and nonfederal firefighting entities generally share their firefighting personnel, equipment, and supplies and work together to fight fires, regardless of who has jurisdiction over the burning lands. Agreements between cooperating entities govern these firefighting efforts and contain general provisions for sharing firefighting assets and costs. On a large wildfire, firefighting efforts generally fall into two phases—initial attack and extended attack. The initial attack phase consists of the efforts to control a fire during the first “operational period” after the fire is reported, generally 24 hours. While the majority of fires on Forest Service land are controlled and suppressed during initial attack, some fires require further firefighting efforts. Such additional efforts are referred to as extended attack. The Forest Service and its interagency cooperators use an incident management system designed to provide appropriate leadership of firefighting efforts. There are five types of incidents, ranging in complexity from type 5 (least complex) to type 1 (most complex). The fire’s complexity determines the type of incident commander and management team assigned. For example, for a type-5 incident, the incident commander may be a local employee qualified to direct initial attack efforts on a small fire with two to six local firefighters. In contrast, for a type-1 incident, the incident commander is one member of a highly qualified incident management team, often with more than 500 firefighters and other personnel. There are sixteen interagency type-1 incident management teams that operate nationwide and are typically deployed to fires for 14-day assignments. In addition, the Forest Service has four type-1 incident management teams under its National Incident Management Organization (NIMO). The Forest Service calls these “short” teams; each team has seven full-time members, but they can add additional members as needed. NIMO teams generally handle complex fires, including long-duration fires, so as not to tie up critical firefighting personnel over a long time. A single incident management team, under the direction of the agency administrator (the line officer, such as the forest supervisor or district ranger, responsible for management of the incident), is typically in charge of a fire, but the incident management system may be expanded into a unified command structure when multiple jurisdictions are involved. This structure brings together incident commanders from the relevant jurisdictions to facilitate a coordinated and integrated interagency response. In such cases, members of the unified command work together to develop a common set of incident objectives and strategies, maximize the use of firefighting assets, and enhance the individual jurisdictions’ efficiency. Once assigned to a fire, an incident management team works with local line officers and fire management staff to determine the strategy and tactics to use in managing the fire. The strategy is the overall plan designed to control the fire; for example, to protect structures and contain the fire within a certain geographic area. Tactics are actions taken to accomplish the objectives set out in the strategy. For example, the fire may be attacked directly, with firefighters working at the fire’s edge to extinguish it. If direct attack is not possible, practical, or safe—because the fire is burning too intensely or on very steep slopes, for example— firefighters may choose to attack it indirectly. In such cases, firefighters typically select an area away from the fire and construct a “fireline,” where vegetation is cleared in an effort to stop the fire’s spread at that point or slow it sufficiently to allow firefighters to attack directly. Firefighters often incorporate geographic features such as roads, rocky areas, ridgelines, and rivers into firelines to increase their effectiveness. In some cases firefighters conduct burnout operations, in which they intentionally set fire to fuels between a fireline and the main fire perimeter to slow or contain a rapidly spreading fire by depriving it of fuel. In carrying out strategies and tactics, firefighters use a variety of firefighting assets, both on the ground and in the air. Ground-based assets include firefighting crews, wildland fire engines, and machinery such as bulldozers, which firefighters use to help construct firelines. When providing personnel to fight fires, the Forest Service and other federal agencies generally rely on a “militia” strategy whereby personnel within each agency are trained to serve in firefighting roles when needed, in addition to performing their day-to-day work responsibilities. Air-based assets include helicopters and fixed-wing air tankers. Helicopters generally drop water directly on a fire, whereas air tankers generally drop fire retardant ahead of the fire, often near a fireline that has been constructed, to slow a fire’s spread. Air tankers range in size from small single-engine air tankers, which are maneuverable but carry only small amounts of retardant, to large aircraft such as converted DC-10s or Boeing 747s—referred to as “very large air tankers”—which can carry substantial amounts of retardant but whose use can be limited in mountainous terrain because of their size. The level of risk that decision makers and firefighters are willing to accept in any given situation depends on the experience and training of those involved. Overall, agency firefighting doctrine emphasizes safety above all other concerns; Forest Service policy, for example, states, “In conducting wildland fire suppression, responsible officials shall give first priority to the safety of firefighters, other personnel, and the public.” Firefighters and other personnel who respond to wildland fire incidents are required to complete training to help them identify risks as well as develop appropriate strategies and tactics to respond to different situations. The Chetco Bar Fire grew slowly in the summer of 2017 before undergoing a period of rapid growth driven by strong, hot winds. In response, the Forest Service and other agencies undertook various firefighting strategies and tactics over different phases of the fire, described below. Figure 2 provides a timeline of the fire’s key events. In the initial phase (July 12-13, 2017), the Chetco Bar fire was relatively small and inaccessible. When the fire was first detected on July 12, it was estimated to be between one quarter and one half acre in size, burning in remote, steep terrain in the Kalmiopsis Wilderness in the Rogue River- Siskiyou National Forest. The fire’s initial location was several miles from the closest road access point. No properties or other “values at risk” (such as structures, other property, and natural and cultural resources that could be damaged by a wildfire) were in the immediate vicinity of the fire, according to Forest Service documents and officials. The Forest Service was notified of the Chetco Bar Fire at 2:43 p.m. on July 12 and, at 4:14 p.m., four Forest Service firefighters rappelled from a helicopter to assess the fire. The rappellers landed on a ridge above the fire to create a helispot (a temporary helicopter landing area) so that additional firefighters and equipment could more easily be brought to the fire. The rappellers requested and received permission from the district ranger for chainsaw use in the Kalmiopsis Wilderness to prepare the helispot, and they worked on cutting trees and clearing brush until late that evening, according to Forest Service documents and national forest officials. The rappellers estimated that the helispot was 60 percent cleared by the end of the first day, according to national forest officials. While the rappellers were working, the Forest Service helicopter returned to its base near Grants Pass, Oregon, to attach a bucket to drop water onto the fire. In the meantime, two helicopters from the Oregon Department of Forestry headed to the fire. The three helicopters dropped about 17,000 gallons of water the first day, according to Forest Service documents. Forest Service officials said these water drops were intended to slow the spread of the fire while the rappellers worked to clear the helispot. Anticipating that the helispot would be completed shortly, the Forest Service ordered two 20-person crews to assist in firefighting efforts the next day. As the rappellers set up camp for the night, incident command radioed them to say that the fire appeared to be holding at about three quarters of an acre. The next morning, July 13, the Forest Service brought in four additional rappellers to continue working on the helispot throughout the morning and into the afternoon (see fig. 3). One of the rappellers walked the perimeter of the fire and determined that the fire had grown to about 10 acres overnight. While the rappellers were working, two helicopters dropped about 18,000 gallons of water that day and a single engine air tanker dropped 1,200 gallons, according to a Forest Service document. The crew bosses for the two crews that had been ordered the previous day flew over the fire early afternoon of July 13, according to Forest Service documents. They estimated the fire had grown to about 15 acres and observed a number of spot fires (smaller fires separate from the main fire) caused by burning material rolling down the hill. They expressed safety concerns about bringing crews into that area and also determined the helispot needed more work before a helicopter could land safely. Since the crews would need to be shuttled in by helicopter, the crew bosses decided not to bring in the requested crews, according to officials. Later that day, the incident commander requested a helicopter to remove the eight rappellers from the fire because of safety concerns and a low probability of success at containing the fire, according to the incident commander and Forest Service documents. The rappellers said that it was taking much longer to complete the helispot than initially anticipated and they did not have a good safety zone or escape route. They also noted that there was unburned vegetation on the slope between the fire and the helispot they were constructing—a dangerous situation if the fire started to spread quickly. The rappellers were removed by 5:00 p.m., at which time the helicopters also stopped dropping water. Figure 4 shows the ignition point of the Chetco Bar Fire and the fire’s growth as of July 13, 2017. In the second phase of the fire, Rogue River-Siskiyou National Forest officials assigned a type-3 incident management team to manage the response to the Chetco Bar Fire, following the unsuccessful initial attack. Forest Service documents indicated that fire behavior was moderate over the next several weeks, averaging around 150 acres of growth per day. The Chetco Bar Fire was a relatively low-priority fire during this phase, since it was far from values at risk and it remained within the Kalmiopsis Wilderness, while other fires in the region were threatening communities and resources, according to Forest Service documents and incident management team officials. Because firefighters had been unable to suppress the fire during initial attack, national forest officials said they anticipated, based on knowledge of previous fires in the area, that the Chetco Bar Fire would become a long-term incident. The type-3 incident management team completed a long-term assessment and began working to contain the fire using long- term, indirect strategies. Under the type-3 team, crews scouted potential locations to fight the fire and started building firelines some distance away, approximately 6 miles from the fire and outside of the wilderness boundary, according to a Forest Service document and an incident management team official. Several additional fire crews were assigned to work on the fire during this time, with staffing fluctuating between approximately 40 and 140 people per day. As the type-3 team’s 2-week rotation was ending, national forest officials decided to bring in a NIMO team to assume command of the fire. Officials said they brought in a NIMO team because it consisted of type-1-qualified staff who could be staffed on the fire for longer than 2 weeks, and the team could expand or contract as needed. The NIMO team took command of the fire on July 29, with the fire estimated at 2,181 acres in size, and started updating the type-3 team’s long-term assessment and developing a long-term implementation plan. The plan identified 13 trigger points, referred to as “management action points,” to help guide decision-making on protecting high values at risk if certain conditions were met. For example, the plan laid out actions to prevent the fire from crossing the Chetco River—the first trigger point identified—and actions to be taken if the fire crossed the river. The NIMO team continued the type-3 team’s efforts to construct a series of firelines away from the main fire and, according to a team summary document, completed all of the firelines by August 17. Forest Service officials told us that for these firelines to be effective, firefighters would have needed to burn the vegetation between the lines and the fire itself (known as a burnout). National forest and NIMO team officials said that the teams had not yet taken this step because they considered it an unnecessary risk as long as the fire remained north of the Chetco River. These officials said that burnout operations pose risks if the fire set by firefighters burns in a different direction than intended, and such operations can unnecessarily burn a larger area of the forest if the fire does not reach the burnout. Therefore, one national forest official said firefighters will prepare firelines but not conduct burnout operations until the incident management team determines they are needed—particularly since safety risks can be associated with conducting burnout operations. Figure 5 shows the Chetco Bar Fire’s growth from July 14 through August 16, 2017. As the fire burned into August, hotter and drier weather created conditions for more active fire behavior in the third phase of the fire. Chetco Effect winds developed in mid-August 2017, causing the Chetco Bar Fire to rapidly expand and intensify (see sidebar). The Forest Service was aware of the potential for such winds, as fire behavior modeling and the July 2017 long-term assessment showed the potential for these winds to increase fire behavior dramatically by mid-August. The winds, combined with dry fuels and heavy vegetation, created conditions that led to extreme fire behavior. Chetco Effect Winds Chetco Effect winds, also known as Brookings Effect winds, are warm, dry, and strong winds flowing down the Chetco River Basin toward Brookings, Oregon (see figure below). Such winds are more broadly referred to as Foehn or downslope winds, other examples being the Santa Ana winds in southern California and the Diablo winds in northern California. Chetco Effect winds can happen any time and generally occur two to four times a year, according to the National Weather Service. The Chetco Effect winds first occurred the evening of August 15 and morning of August 16, but the fire remained north of the Chetco River. When the winds returned the evening of August 16 and morning of August 17, the fire crossed the river and began expanding rapidly, in part because heavy vegetation on the south side of the river fueled the fire under the winds. Many officials and stakeholders said nothing could be done to moderate the fire’s behavior when the Chetco Effect winds were in effect. The fire increased in size from 8,500 acres on August 17 to 91,551 acres on August 21 (see fig. 6). As a result, the Chetco Bar Fire became a much higher priority fire, according to Forest Service documents. The NIMO team ordered additional crews on August 17, in anticipation of conducting burnout operations along 10 miles of fireline in an attempt to slow the fire, according to Forest Service documents. However, the Chetco Effect winds caused the fire to move rapidly toward and past the fireline before the Forest Service could conduct the planned burnouts. Even though the fireline was completed prior to being overrun by the fire, national forest officials told us that the weather conditions were not favorable for burnout operations, as the winds would have blown the burnout fires back toward private timberlands and populated areas. The winds also caused embers to fly far ahead of the fire during this time, creating spot fires 1 to 2 miles or more ahead of the main flame front. On August 18, the Chetco Bar Fire began spreading from national forest onto private timberlands and unincorporated areas containing homes. As the fire began to threaten homes and other structures, the NIMO team directed firefighters to take appropriate action to try to protect those structures, if fire behavior allowed. For example, between August 18 and 21, Forest Service documents indicated that firefighters cleared brush around several structures and homes in a small community known as Wilderness Retreat and along two Forest Service roads. On August 19, the fire burned rapidly toward Wilderness Retreat and firefighters conducted an emergency burnout, which successfully protected the community, according to a NIMO team document and national forest officials. Around this time in another area, the Chetco Bar Fire burned six primary residences and more than 20 other structures, according to state and Forest Service documents. On August 20, the fire traveled 6 miles toward Brookings in a single day, and threatened more than 3,000 homes during this phase. As the Chetco Bar Fire burned toward Brookings, the NIMO team notified the Curry County Sheriff that residents would need to be evacuated. However, the rapid spread of the fire provided limited time to notify residents and conduct evacuations, according to a NIMO team document and national forest officials. The Curry County Sheriff’s Office issued the first evacuation notices on August 18, and additional evacuation notices were issued between August 19 and 21. As the fire expanded, the NIMO team ordered additional firefighting assets, increasing the ground assets assigned from 65 firefighters and 1 fire engine on August 17, 2017, to 788 firefighters and 90 fire engines by August 21. However, some assets ordered were not available because they were assigned to other fires in the region. In addition to ground assets, additional aircraft were ordered and assigned to assist the firefighting effort—such as two large and one very large air tankers, which dropped retardant on the fire on August 17 and August 18. The incident management team had requested two additional air tankers, but the requests were cancelled since aircraft were unavailable, according to a Forest Service document. Some ordered drops from air tankers also were cancelled because of poor visibility from smoke. Six helicopters were ordered during this phase, four of which were assigned to the fire, but the helicopters also were unable to fly due to smoke, according to flight communication logs and an incident management senior official. With the Chetco Bar Fire’s rapid growth, national forest officials decided to order a type-1 incident management team on August 21. Since mobilizing the team would take time, a type-2 team already in the vicinity was brought in to assist the NIMO team on August 19. The type-1 team arrived on August 23 and assumed command on August 26, according to a team document. In the fourth phase, the Chetco Bar Fire continued to burn actively through the end of August and into September 2017, but the rate of its spread generally slowed. However, high temperatures and low humidity contributed to the fire growing from 97,758 acres on August 22 to 191,067 acres on September 22 (see fig. 7). Evacuations continued in the early part of this phase, threatening more than 8,500 homes during parts of September, but evacuation orders began to be lifted as the risk to homes declined. During this phase, the Forest Service ordered more firefighting assets, resulting in over 1,700 firefighters in total assigned to the fire. Between September 6 and 19, the fire began expanding to the east and the fire was divided into an east and west zone, with separate incident management teams assigned to each zone. Firefighters constructed firelines to the south and west of the fire. Forest Service documents indicated the agency put in 128 miles of fireline cut by bulldozers and 52 miles of hand cut fireline, and used 141 miles of existing roads and 25 miles of natural features as firelines. Air tankers and helicopters continued supporting firefighters, dropping over 950,000 gallons of water, 55,000 gallons of retardant, and 10,000 gallons of gel during this phase, according to Forest Service documents. However, smoke from the fire hampered air operations, with one type-1 team reporting it was unable to conduct air operations for about half of the days it was in command (August 26 through September 9). Firefighters gained substantial control of the fire during this phase, going from 0 percent containment on August 22 to 97 percent containment by September 22. In mid- to late-September, the weather started to change, with cooler days and more moisture, which helped to moderate the fire’s behavior. By September 23, the area had received several inches of rain, which nearly contained the fire, according to an incident management team document. Firefighting assets were released as the fire was contained. The Chetco Bar Fire was declared fully contained on November 2—nearly 4 months after it was detected. The fire burned a total of approximately 191,197 acres, according to the Forest Service’s Burned Area Emergency Response (BAER) report (see fig. 8). Forest Service officials and stakeholders we interviewed raised a number of concerns about the Forest Service’s response to the Chetco Bar Fire. Many of these concerns related directly to the Forest Service’s response to the fire; some related to broader agency programs that may have had an effect on fire behavior. We grouped these concerns into five categories: (1) aggressiveness of firefighting response, (2) availability of firefighting assets, (3) communication with cooperators, (4) communication with the public, and (5) timber harvest and other fuel reduction activities. The Forest Service has taken steps that may help address some of the concerns, such as those related to communication. Agency officials and stakeholders expressed differing views about some of the concerns and whether changes were necessary. Some national forest officials and many stakeholders we interviewed said that the Forest Service was not aggressive enough in fighting the Chetco Bar Fire before the Chetco Effect winds arrived in mid-August. Several of these stakeholders said if the Forest Service had used more aggressive firefighting strategies and tactics, the agency could have prevented the fire from getting as large as it did and threatening homes. Some of these officials and stakeholders raised concerns about whether incident management teams and line officers appropriately balanced the risks of different firefighting decisions during the fire. Some said the strategies and tactics taken early on may have put hundreds of firefighters and the public at risk later in the fire. National forest and incident management team officials said that in attempting to suppress the Chetco Bar Fire, they adopted firefighting strategies and tactics that considered firefighter safety, the values at risk, and the probability of success. National forest officials said that when deciding how to respond to the fire, they prioritized firefighter safety and also considered the likelihood that a particular response would be successful, in accordance with 2017 Forest Service guidance. As previously discussed, in the early stages of the Chetco Bar Fire, firefighters expressed concerns about their safety and the likelihood of success of certain tactics. In addition, national forest officials noted that after the rappellers asked to be pulled out of the fire and other firefighters expressed safety concerns, line officers were hesitant to send in additional firefighters. Other officials and stakeholders said the area where the Chetco Bar Fire started is very dangerous, with some noting that it is one of the most dangerous areas in the region and possibly the country to fight fire. Specific concerns about the aggressiveness of the Forest Service’s response included the following: Number of firefighters. Some officials and several stakeholders raised concerns about the Forest Service not sending in more firefighters at the beginning of the Chetco Bar Fire to try to contain it before it threatened homes. In response, national forest officials said that the four rappellers that were sent on the first day were part of an 18-person crew stationed near Grants Pass, Oregon. They were the only crew members available to respond on July 12, as the remaining crew members had just returned from another fire assignment, and firefighters are generally required to take 2 days off after completing a standard 14-day fire assignment. As previously noted, safety concerns also factored into decisions to remove the rappellers and not add crews on the second day of the fire. Absence of smokejumpers. Some stakeholders raised concerns that the Forest Service did not send smokejumpers into the Chetco Bar Fire in its early stages, saying that smokejumpers may have been more effective at suppressing the fire when it was small. In response, national forest officials said that the rappellers who were sent to the fire were located much closer to the ignition point than the closest smokejumpers and were able to respond more quickly. These officials also said that rappellers can be more effective in rough terrain with heavy timber, since they do not need an open space to land with parachutes and can be dropped closer to the fire. Use of helicopters. Several stakeholders raised concerns about the Forest Service stopping the use of helicopters to drop water on the fire after the rappellers were removed. According to interagency guidance and Forest Service officials, water drops are not as effective at containing a fire without crews on the ground (to build firelines, for example), and they did not want to expose helicopter crews to unnecessary risk for actions that were unlikely to be effective. In addition, officials said that the water drops were causing burning logs and other debris to roll down the hill and create spot fires. Interagency guidance discusses the importance of coordinating air and ground firefighting tactics, noting that the effectiveness of aircraft is dependent on the deployment of ground assets. Use of indirect strategies. Several stakeholders raised concerns about incident management teams not engaging the fire more directly in the first several weeks rather than constructing fireline miles away. Some of these stakeholders described this indirect approach as a “watch and wait” or “let it burn” approach. In response, officials said that they looked for locations and opportunities to fight the fire directly, but the fire’s remote location and rugged terrain made this difficult. One official estimated it would have taken firefighters 2 days to hike to the fire because of the distance and trail conditions. Number of burnout operations. Several officials raised concerns about the Forest Service not conducting burnout operations before the Chetco Effect winds arrived in mid-August. However, as previously noted, officials stated that there are risks in conducting such operations. Limited use of chainsaws. Some national forest officials raised concerns about limited use of chainsaws in the Kalmiopsis Wilderness, saying this prevented them from making quicker progress in constructing fireline. For example, two national forest fire management officials said that in trying to clear a wilderness trail to use as a fireline, the crew used handsaws rather than chainsaws after the initial attack, which made the task more difficult and time consuming. Limited action to protect homes. Several stakeholders raised concerns about incident management teams not doing more to protect homes, stating that firefighters and equipment in the vicinity of homes that later burned were not used to help protect those homes. In response, national forest and headquarters officials said that although the agency tries to prevent fires from reaching homes, protecting homes and other private structures is the responsibility of state and local entities. Moreover, headquarters officials noted that Forest Service firefighters are not trained or equipped to defend structures. Forest Service officials said that since the Chetco Bar Fire, the agency has expanded tools that may help address some of these concerns for future fires. They noted that some of these tools were not widely available at the time of the Chetco Bar Fire but are becoming more common. In particular, the Forest Service has an evolving risk management assistance program aimed at improving decision-making on fires by developing a strategic evaluation process. This program includes risk- management assistance teams that can be deployed to fires to assist with key decisions and exercises to help incident management teams and line officers analyze different firefighting options, according to program documents. For example, the Forest Service developed a tradeoff analysis tool through which decision makers assess different firefighting options and rate them according to how well they address firefighter safety, public safety, and values at risk. During the 2018 Klondike Fire, national forest officials said they brought in a risk-management team to facilitate analysis of firefighting options and included cooperators in the discussions. Officials said these discussions helped everyone understand the risks and tradeoffs of various firefighting options, adding transparency to the process. Several officials and stakeholders raised concerns about the number of firefighting assets assigned to the Chetco Bar Fire. According to Forest Service documents and officials, firefighting assets were stretched thin fighting other fires in the region, and there were a number of times throughout the Chetco Bar Fire when assets, such as management teams, crews, and helicopters, were requested but were unavailable (see table 1). For example, an incident management team that was heading to the Chetco Bar Fire was diverted to the Eagle Creek Fire, which was threatening homes and other structures near Portland, Oregon. Further, some officials said limited availability of certain firefighting assets with specific capabilities, such as infrared drones that can “see” through smoke or cloud cover, hindered their ability to fight the fire when visibility was limited. Some officials also emphasized the importance of having more long-term fire analysts assigned to national forests and incident management teams to help develop and interpret fire behavior models and long-term assessments that, in turn, could help protect people and values at risk. However, other officials said that having additional assets likely would not have made a significant difference in the response to the Chetco Bar Fire because of the difficult terrain where the fire started and because of the Chetco Effect winds. Beyond their specific concerns with the Chetco Bar Fire, some stakeholders also observed the Forest Service would likely benefit from having additional firefighting assets in the future, as the frequency and intensity of fires are likely to increase. Forest Service officials acknowledged that there were not enough firefighting assets in 2017, given the number of large fires that year. As a result, they said they had to make difficult decisions regarding prioritizing assets, with fires threatening life and property receiving higher priority. Forest Service officials said that the agency is working to increase the number of some types of firefighting assets. For example, headquarters officials said that the agency was in the process of developing a drone program. In addition, officials said that the agency is working on increasing the availability of some assets, such as air tankers and helicopters, through the use of different contracting authorities. Several officials and stakeholders raised concerns about communication among the various cooperators before and during the Chetco Bar Fire. In particular, some said that differences in firefighting approaches—due in part to cooperators’ differing missions, responsibilities, and priorities—had not been fully clarified in advance, leading some cooperators to express frustration with the Forest Service’s response to the fire. For example, according to some officials and stakeholders, the Oregon Department of Forestry and Coos Forest Protective Association generally place more emphasis on protecting timberlands than the Forest Service, and this sometimes leads to differences in the agencies’ preferred approaches to responding to fires. For example, when determining where to construct a fireline, Forest Service officials may identify a location aimed to keep a fire from reaching homes, whereas cooperators from the Oregon Department of Forestry or Coos Forest Protective Association may prefer a location that also protects timberlands. In addition, some stakeholders said that the frequent rotation of incident management teams—generally about once every 2 weeks—made it difficult for local cooperators to coordinate with those teams. One official noted that rotation of teams can make it difficult to build trust and maintain good communication with cooperators and the public. However, Forest Service headquarters officials said that the agency has studied the structure and use of incident management teams in the past, and the agency has not identified a better approach. Several officials and some stakeholders noted lessons learned from the Chetco Bar Fire. For example, they cited the need to do more pre-season fire planning, such as meeting with cooperators before the fire season begins to discuss coordination among agencies and planning how they might respond to fires in certain situations. Some also noted the need to improve communication and transparency with cooperators during fires, such as through the use of risk-management assistance teams previously discussed. Officials and stakeholders said that communication among cooperators in the region has improved since the Chetco Bar Fire, helping to develop a shared understanding of the potential firefighting response in different locations and under different conditions. Many officials and several stakeholders said the Forest Service did not provide sufficient or timely information to the public about the danger from the Chetco Bar Fire and what the agency was doing to fight it. In particular, several officials raised concerns about the Forest Service waiting to hold its first public meeting until over a month after the fire was detected. Several officials and some stakeholders said that in the absence of sufficient information, misinformation and rumors—such as incorrect information on evacuations in certain areas—spread, leading to frustration, anger, and fear on the part of the public. Officials and stakeholders said another lesson learned was the importance of communicating accurate and timely information through various means, including public meetings and social media. Officials and stakeholders told us that the Rogue River-Siskiyou National Forest is taking steps to help ensure that it communicates more effectively during fires. For example, national forest officials said that since the Chetco Bar Fire, they have increased their level of communication with local communities. Officials also said they are now more proactive in monitoring social media and ensuring they post correct information on fires, among other things. As a result, officials and stakeholders said that public perception of the 2018 Klondike Fire was much more positive than of the Chetco Bar Fire, even though both fires burned more than 175,000 acres. fueled the Chetco Bar Fire and made firefighting efforts more dangerous by leaving snags (standing dead trees) that could injure or kill firefighters. Following wildfires, the Forest Service may consider whether to leave burned trees and allow the burned area to recover naturally or to harvest some of those trees—called salvage harvesting—with the intention of generating funds to help pay for the recovery of natural resources or infrastructure, such as trails or roads, among other purposes. Considerable scientific uncertainty exists about whether and how quickly harvested areas recover compared with unharvested areas. Disagreement also exists about the extent salvage harvesting generates funding, considering the cost of planning, preparing, and administering sales of salvaged trees. Following the Chetco Bar Fire, the Forest Service determined that 13,626 acres of the burned area were potentially available for salvage harvesting. These areas had 50 to 100 percent tree mortality and were in areas of the Rogue River-Siskiyou National Forest where timber harvesting aligned with existing management objectives, according to an official. The Forest Service narrowed the area that it proposed putting up for salvage harvesting to 4,090 acres, removing areas that lacked economically viable timber, were inaccessible to logging equipment, were in roadless areas, or had sensitive wildlife habitat, among other factors. The total number of acres the Forest Service offered for salvage harvesting was 2,194 acres across 13 sales, according to an official. Of the 13 salvage sales offered, eight were sold, totaling 1,957 acres, and five were not sold. Of these five offers, three did not receive bids, and two were dropped by the Forest Service due to market changes or other considerations. In contrast, several Forest Service officials and some stakeholders said that higher levels of timber harvest and fuel reduction would not have made a large difference in the Chetco Bar Fire because of the fire’s intensity and rate of spread under the Chetco Effect winds. Several said that if there had been more timber harvest, the forest might have been replanted in ways that could have made the fire worse. Specifically, when replanting is done following timber harvest, trees may be planted more densely and uniformly than would occur if vegetation were allowed to grow back naturally, according to a Forest Service ecologist and some stakeholders. In addition, slash (debris from logging operations) is sometimes left on the ground after timber harvest, which can fuel future fires. As a result, areas where timber has been harvested may burn more severely during future fires, according to some officials and stakeholders. Rogue River-Siskiyou National Forest officials said the forest has been carrying out many fuel reduction activities and has exceeded its fuel reduction target every year from fiscal year 2014 through fiscal year 2019 (see appendix I for a map of past timber harvests and other fuel reduction activities). As part of its fuel reduction efforts, the forest is creating some larger breaks in vegetation by connecting areas where fuel reduction activities have taken place, according to officials. Further, national forest officials are maintaining some firelines that were built during previous fires, including the Chetco Bar Fire, to aid in their response to future fires. Agency officials said these efforts are part of a broader effort to move towards spatial fire planning, where areas at risk and effective places to contain wildfires are identified before fires start. Forest Service officials and stakeholders we interviewed and reports and other documents we reviewed identified a variety of effects the Chetco Bar Fire had on local communities and resources. We grouped these effects into four categories: (1) homes and infrastructure, (2) public health, (3) local businesses and workers, and (4) natural and cultural resources. Most of the identified effects were negative, although some positive short- and long-term effects were identified. For example, the Chetco Bar Fire damaged habitat for many wildlife species, but some species that prefer burned landscapes likely benefitted from the fire, according to officials. The Chetco Bar Fire destroyed six homes and damaged one home, according to Forest Service and state documents. The fire also threatened over 8,500 homes, causing more than 5,000 residents to be evacuated over the course of the fire, according to Forest Service documents. In addition, Forest Service and state documents stated that the fire destroyed more than 20 other structures and damaged at least eight more, such as garages and other outbuildings. After a severe wildfire, soil erosion can increase and cause adverse effects. As fires burn, they destroy plant material, such as roots and leaves, that help prevent erosion during severe rainstorms. Plant roots help stabilize the soil, and leaves slow runoff by allowing water to seep into the soil. In some severe fires, burning vegetation creates a gas that penetrates the soil. As the soil cools, this gas condenses and forms a waxy coating that causes the soil to repel water. Rainwater and melted snow can then flow across these surfaces and cause erosion. Erosion can reduce water quality and damage roads. In addition, because burned soil does not absorb as much water as unburned soil, seeds have a harder time germinating, and surviving plants find it more difficult to obtain moisture. the 63 miles of trails within the fire perimeter. Further, a campground within the national forest was partially damaged and closed to the public while being repaired. Erosion following the Chetco Bar Fire also washed approximately 40,000 cubic yards of sediment into the Port of Brookings Harbor. A port official said that dredging the harbor is estimated to cost $4 million. The official noted that the commission governing the port was pursuing grants, such as disaster grants from the Federal Emergency Management Agency, to help with dredging costs but was unsure whether total costs could be covered. Local officials said that post-fire erosion could also negatively affect drinking water infrastructure, since the Chetco Bar Fire burned about 80 percent of Brookings’ watershed. Brookings received a grant to evaluate the fire’s effect on the city’s water system, according to a local official. The city hired a consultant, who reported in June 2018 that the quality of the water was generally excellent and that no significant water quality effects from the fire had been observed. People with existing lung disease may not be able to breathe as deeply or vigorously as they normally would during exposure to high levels of particulate matter. Healthy people may also experience these effects. susceptibility to respiratory infections and aggravate existing respiratory diseases, such as asthma and chronic bronchitis. smoke (see sidebar). Most healthy individuals recover quickly from smoke exposure and will not experience long-term health effects, according to an Environmental Protection Agency document; however, the smoke exposure effects are more sudden and serious for sensitive groups, including children, older adults, and people with existing heart or lung disease. Local health officials and a national forest official also raised concerns about the potential long-term effects of exposure to wildfire smoke, but little data exist on such effects. The Forest Service reported that four towns in the vicinity of the Chetco Bar Fire experienced, on average, about 9 days of unhealthy or worse air quality, although the severity and duration of wildfire effects on air quality varied by town (see fig. 10). Of these towns, Brookings had the most days—three—measured as “hazardous,” the worst category. The four towns also experienced about 5 days, on average, that were measured as being unhealthy for sensitive groups. Many residents also experienced mental and emotional effects from the Chetco Bar Fire, according to local health officials and some stakeholders. A local health official said that some residents experienced post-traumatic stress disorder after the fire, with some residents becoming hypervigilant of smoke and sirens. Some stakeholders noted that the 2018 Klondike Fire, which burned nearby, led to additional mental and emotional stresses for those affected by the Chetco Bar Fire. The Chetco Bar Fire’s effects on local businesses and workers included damage to the tourism and logging industries. Local businesses lost revenue in the short term because of decreased summer tourism during the Chetco Bar Fire, according to some documents and many stakeholders. According to estimates from the Oregon Tourism Commission, businesses—including tourism-dependent ones such as hotels and restaurants—lost over $1 million in both Curry and Jackson counties, and businesses in Josephine County lost over $160,000 during the 2017 fire season. For example, the Oregon Shakespeare Festival canceled nine outdoor performances because of wildfire smoke, resulting in losses estimated at about $600,000, according to a company document. In addition, one vineyard in Cave Junction lost an estimated $10,000 to $20,000 in revenue because of reduced tasting room sales and vacation rentals, according to an Oregon vineyard association spokesperson. The decrease in tourism also had short-term negative effects on workers in the tourism industry. According to a report, workers in Curry County lost income, in part due to employee furloughs, because of wildfires in 2017. Another document cited that Josephine County lost an estimated 100 jobs in 2017 because of the Chetco Bar Fire. Following the fire, the governor of Oregon created the Chetco Bar Fire Recovery Council to help the region recover from the fire. The council assessed economic damage, identified recovery needs, and identified potential state funding for those needs. For example, in November 2017, the council identified a potential need for state economic development funds to assist local businesses. However, the council reported in March 2018 that three businesses affected by the fire had received federal loans from the U.S. Small Business Administration and that there was no longer a clear need for state economic development funds. In addition, some stakeholders we interviewed and documents we reviewed raised concerns that if summer wildfire smoke became common in southern Oregon, it could have a long-term negative effect on tourism. However, a 2019 report found that wildfire smoke had a minimal effect on people’s willingness to consider traveling to southern Oregon in the future. One local business has set up air quality monitors at a tourist attraction to inform tourists of the current air quality. The Chetco Bar Fire burned 14,130 acres of nonfederal timberlands, according to the Forest Service’s BAER report. One privately owned lumber company was particularly hard hit, with the fire burning about 10,000 acres of its timberlands, according to company representatives. This loss was about 10 percent of the company’s timberlands and represented about 5 years of its average harvest. Following the fire, the company salvage-harvested approximately 6,000 acres of the burned timber, which company representatives said provided some short-term economic benefits for the company and, according to one stakeholder, also temporarily increased employment for loggers and truck drivers in the area. However, the long-term effects of the fire on the company are unknown. One representative said, depending on future market conditions, the loss of timber from the Chetco Bar Fire could lead the company to lay off employees or could jeopardize its future. The severity of the Chetco Bar Fire varied across the forest, which led to varied effects on soil and vegetation. As shown in figure 11, within the perimeter of the Chetco Bar Fire, burn severity ranged as follows: unburned or very low (19 percent, or 36,027 acres); low (40 percent, or 76,613 acres); moderate (34 percent, or 64,545 acres); and high (7 percent, or 14,012 acres). The severity with which soil burns during a fire affects both the potential for erosion following the fire and the severity of damage to vegetation. Areas of the Chetco Bar Fire that burned at moderate and high severity had increased potential for erosion, according to the BAER report. As previously discussed, post-fire erosion damaged roads and other infrastructure. Further, the BAER report noted that severely burned areas may have lower soil productivity and vegetation growth. However, most of the native vegetation in the area is adapted to fire and is likely to recover over time, according to the BAER report. Moreover, a Forest Service ecologist said the Chetco Bar Fire helped create a more diverse forest structure (characterized as a mosaic of different species and age classes) that benefits many plant and animal species (see fig. 12). For example, nine sensitive plant species found in the area burned by the Chetco Bar Fire thrive in early post-fire ecosystems, according to a Forest Service document. Further, officials said rapid regrowth of vegetation, such as a moss that thrives after fires, helped reduce erosion and limit potential future damage to roads and trails. Forest Service officials and documents noted that they did not expect widespread, long-term negative effects on vegetation from the Chetco Bar Fire, but they identified two negative effects: Invasive plants. More than a thousand individual invasive plants (such as noxious weeds) were introduced to an approximately 13,000- acre area of the national forest during the Chetco Bar Fire, mainly via firefighters’ boots and equipment. Invasive plants can, in some cases, displace native plants, compromise the quality and quantity of habitat for wildlife and fish, and increase wildfire risk. A national forest official said that it is labor intensive and costly to eradicate invasive plants because they have to be pulled out by hand. The official said the agency does not have the resources to remove all of the invasive plants brought in during the fire and is prioritizing removal of those that are the fastest growing, most disruptive, and affect the most highly valued resources. In addition, the National Forest Foundation administered a $7,000 grant to remove invasive plants on 10 of the affected acres in June and July 2019. Redwood stands. The Rogue River-Siskiyou National Forest contains the northernmost naturally occurring coast redwood tree stands, and the Chetco Bar Fire burned about 12 percent of the total area of redwood stands within the forest, or about 60 acres, according to a Forest Service ecologist. However, most of the area burned at low severity, though parts burned at moderate or high severity. The ecologist said redwoods are adapted to survive fire, noting that larger trees will usually resprout from dormant buds under the bark along the entire length of the trunk (see fig. 13). Smaller trees and larger trees burned at high severity can be killed at the top but are often able to resprout. In the short-term, the Chetco Bar Fire killed or damaged habitat for many wildlife species, although the exact effect of the fire on wildlife is unknown, according to a Forest Service official. Most wildlife species are expected to recover, but the effects on some threatened and sensitive species could be longer lasting, according to Forest Service documents and officials. For example, half of the 13 known northern spotted owls—a species that is federally listed as threatened under the Endangered Species Act—living within the perimeter of the fire were estimated to have died from the fire, according to a Forest Service biologist. In addition, this biologist said the fire’s effect on the population of a seabird called the marbled murrelet, as well as on two mammals—Pacific marten and fisher—is unknown, although it negatively affected their habitats. National forest officials said the Chetco Bar Fire also likely benefitted some wildlife species because the mosaic landscape resulting from the fire is preferred by some wildlife, including deer, elk, migratory birds, butterflies, and woodpeckers. For example, black-backed woodpeckers thrive in partly burned areas because they eat wood-boring beetles that feed on recently burned trees. Erosion resulting from the Chetco Bar Fire likely had short-term negative effects on fish populations, including the threatened coho salmon, according to the BAER report. Sediment in the water makes it harder for fish to breathe and can smother their eggs. In addition, over time, increased sediment in streams and rivers can disrupt salmon migration because salmon use their sense of smell to navigate to their native stream to spawn, and sediment can mask that smell. Some stakeholders said they were concerned that the loss of shade from trees might lead to warmer river water, thereby harming salmon. However, a Forest Service biologist said that vegetation near the river has regrown since the fire and there is no indication that the temperature of the river water has increased. The fire may provide some long-term benefits for salmon and other fish species. Specifically, erosion following the fire is likely to increase the supply of downed trees and coarse gravel in streams and rivers, which provide places for fish to lay their eggs and hide, according to a study and a Forest Service biologist. Some cultural resources—including archaeological sites, historic structures, and areas significant to contemporary Native American tribes—were negatively affected by the Chetco Bar Fire. The Forest Service reported that 130 known and recorded Native American archaeological sites were located within the perimeter of the Chetco Bar Fire, 49 of which the agency characterized as isolated sites containing one to three stone artifacts. The effect of the Chetco Bar Fire on known and recorded sites—and on any cultural sites not previously identified—is not fully known. Following the fire, as part of its BAER report, the Forest Service assessed some of these sites, including a prehistoric Native American village site and an area culturally important to Native American tribes. This report noted a number of cultural artifacts, such as arrowheads and tools, that were discolored by the fire or were displaced or moved during or after the fire by, for example, soil disruption caused by trees falling or roots burning and collapsing. The report also stated additional damage could occur in the future; for example, increased erosion could further damage some cultural sites, and vegetation loss could make artifacts more visible, increasing the potential for looting and vandalism. To help mitigate some of the effects, the Forest Service planted some of the burned area with native grass seed to reestablish ground cover and reduce erosion. In addition to the fire damaging cultural resources, a Forest Service archaeologist said fire suppression activities caused some damage. For example, Native American arrowheads and tools were unearthed when a bulldozer constructed a fireline. The archeologist said that they took precautions to minimize suppression impacts on cultural resources, for instance by avoiding using heavy equipment in areas where cultural resources were known to be located. We provided a draft of this report to the Departments of Agriculture and the Interior for review and comment. In an email dated April 17, 2020, the Forest Service, responding on behalf of the Department of Agriculture, said it generally agreed with the draft report. The Forest Service also provided a technical comment, which we incorporated. The Department of the Interior told us it had no comments on the report. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Agriculture, the Secretary of the Interior, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. Figure 14 shows the timber harvests and other fuel reduction activities— such as thinning vegetation or conducting prescribed burns—done in the area of the Chetco Bar Fire from 2008 through 2017. Anne-Marie Fennell, (202) 512-3841 or fennella@gao.gov In addition to the individual named above, Jonathan Dent (Assistant Director), Lesley Rinner (Analyst-in-Charge), Elizabeth Jimenez, and Jesse Lamarre-Vincent made key contributions to this report. Philip Farah, Ellen Fried, Richard P. Johnson, John Mingus, Edward J. Rice, Sara Sullivan, and Elizabeth Wood made additional contributions.", "summary": "A wildfire known as the Chetco Bar Fire began in the summer of 2017 in southwest Oregon and burned more than 190,000 acres over nearly 4 months. Since the fire began in a national forest, the Department of Agriculture's Forest Service played a key role in managing the firefighting response. Because the fire also threatened other lands, state and private firefighting entities were also involved. GAO was asked to review the Forest Service's response to and the effects of the Chetco Bar Fire. This report describes (1) key events of the Chetco Bar Fire and the Forest Service's firefighting response, (2) key concerns raised by Forest Service officials and stakeholders about the Forest Service's response, and (3) effects of the fire on local communities and resources. GAO reviewed federal documents related to key events and the response, such as incident action plans and daily status summaries; analyzed reports on effects of the fire; and visited burned areas. GAO also interviewed Forest Service, state, and local officials involved in the response, as well as other stakeholders—such as representatives of nongovernmental organizations and community members—to discuss key concerns and effects of the fire. To identify the stakeholders, GAO reviewed documents and interviewed Forest Service officials and stakeholders, who suggested others to interview. The Chetco Bar Fire was first reported in July 2017, burning in the Rogue River-Siskiyou National Forest in Oregon. Because of the remote, steep terrain, initial Forest Service attempts to fight the fire at close range were unsuccessful. The fire grew slowly over the next month. Firefighters, directed by the Forest Service, responded in various ways, such as by constructing “firelines”—clearing vegetation—in an effort to stop the fire's spread. In mid-August, strong, hot winds caused the fire to expand rapidly, from 8,500 acres to more than 90,000 acres over several days, threatening thousands of homes. Firefighters continued constructing firelines and dropped water and retardant on the fire to try to contain it. In September, the weather changed and cooler days and rain moderated the fire. Firefighers fully contained the fire in November (see figure). Forest Service officials and stakeholders raised a number of key concerns about the Forest Service's response to the Chetco Bar Fire. For example, some said that if the Forest Service's response had been more aggressive, it might have kept the fire from growing and threatening homes. Forest Service officials said that in making firefighting decisions, they prioritized firefighter safety and considered the likelihood that a particular response would be successful. The agency has taken steps to improve decision-making for future wildfires, such as developing a tradeoff analysis tool to help decision makers assess firefighting options. Forest Service officials, stakeholders, and documents identifed various effects of the fire. Some of these sources cited negative effects including destruction of six homes, damage to roads and trails, and damage to habitat for the northern spotted owl. However, the fire likely improved habitat for some species, such as woodpeckers that eat beetles that feed on burned trees, according to officials.", "document_type": "gao"}
{"report": "Uranium enrichment is the process of increasing the concentration of the uranium-235 isotope relative to uranium-238 in a quantity of uranium. Natural uranium consists of approximately 0.7 percent of the fissile uranium-235 isotope, while uranium used in commercial nuclear power reactors generally consists of 3 to 5 percent uranium-235 and uranium for nuclear weapons requires a higher concentration of uranium-235. In addition, as a nuclear reactor operates, some of the uranium in the reactor fuel is converted to plutonium, which can also be used as a weapons material when it is separated from other elements of the irradiated, or spent, fuel through a process known as reprocessing. Plutonium and enriched uranium are “special nuclear material” under the Atomic Energy Act. The processes for obtaining such material— enrichment and reprocessing—are called sensitive nuclear technologies. Under the Treaty on the Nonproliferation of Nuclear Weapons, which came into force in 1970, non-nuclear weapon state parties to the treaty may not acquire nuclear weapons and must conclude a Comprehensive Safeguards Agreement (CSA) with the International Atomic Energy Agency (IAEA). IAEA is an independent international organization affiliated with the United Nations that has the dual mission of promoting the peaceful uses of nuclear energy and verifying, through a set of technical measures called safeguards, that nuclear technologies and materials are not diverted from peaceful uses to military purposes. Most countries have also brought into force an Additional Protocol to their CSAs, which provides IAEA with a broader range of information on the country’s nuclear and nuclear-related activities than under a CSA alone and gives the agency’s inspectors access to an expanded range of locations. For example, the Additional Protocol requires states to declare the location and status, among other things, of uranium mines and uranium and thorium mills. Under a CSA alone, material in mining or ore processing activities (e.g., uranium at mines and mills) is not subject to the agency’s safeguards as it is not yet suitable for enrichment. The United States promotes universal adoption of the Additional Protocol as a policy, but it is not a requirement for the conclusion of a nuclear cooperation agreement with the United States. Figure 1 shows the safeguards arrangements of the partners with which the United States has nuclear cooperation agreements. Section 123 of the AEA establishes a framework for civilian nuclear cooperation agreements, which are a prerequisite for the export of certain nuclear material and equipment, including major components of nuclear reactors. The United States has 23 such agreements with other nations and entities. Section 123 generally requires that nuclear cooperation agreements include nine nonproliferation conditions, such as a guarantee from the cooperating party that transfers will not be used for any military purpose. The President may exempt an agreement from any of these requirements, provided that the president determines that the inclusion of any such requirement would be seriously prejudicial to United States nonproliferation objectives or otherwise jeopardize the common defense and security. See Table 1 for a list of the nine requirements. Section 123 of the AEA also requires that State supply the President with an unclassified Nuclear Proliferation Assessment Statement (NPAS) for each proposed agreement, accompanied by a classified annex prepared in consultation with the Director of National Intelligence. The NPAS describes how the agreement meets AEA nonproliferation requirements and usually includes an overview of the other party’s nuclear energy program and related infrastructure, nonproliferation policies, and relations with countries of proliferation concern. Section 123 also lays out requirements for informing congressional committees and obtaining congressional review. It requires that the President submit any proposed agreement along with the NPAS to the House Committee on Foreign Affairs and the Senate Committee on Foreign Relations for consultation for a period of at least 30 days of continuous session. The proposed agreement, with the NPAS, must subsequently be submitted to Congress as a whole (and referred to the abovementioned committees) for a period of 60 days of continuous session, during which the committees consider it and submit recommendations to the House and Senate, respectively, as to whether to approve the agreement. As a general matter, the agreement may then be brought into effect unless a joint resolution of disapproval is enacted before the end of this period. Section 123 also requires the President to keep the abovementioned committees “fully and currently informed” of any initiative or negotiations relating to a new or amended agreement for peaceful nuclear cooperation. Figure 2 depicts the stages and time frames for negotiation and conclusion of nuclear cooperation agreements. Another section of the AEA, Section 57(b), governs the direct or indirect engagement or participation in the development or production of special nuclear material outside the United States. Under this provision, DOE regulates exports of commercial nuclear technology and assistance. DOE has promulgated these regulations at 10 C.F.R. Part 810; authorizations under these regulations are accordingly referred to as “Part 810 authorizations.” Activities authorized under section 57(b) may not require a nuclear cooperation agreement. The Secretary of Energy signed seven “Part 810” authorizations for the export of nuclear technology to Saudi Arabia between December 2017 and February 2019. For more information about Part 810, see table 2. In negotiating nuclear cooperation agreements, the United States has sometimes pursued nonproliferation measures beyond the nine conditions specified by the AEA. For example, the agreement that the United States concluded with the United Arab Emirates (UAE) in 2009 includes a provision in which UAE agreed to forswear enrichment and reprocessing capabilities. This broad restriction on any enrichment and reprocessing, which became known as the “gold standard,” goes beyond the enrichment and reprocessing restriction required by Section 123 of the AEA, because it applies to all nuclear material rather than just U.S.- obligated material. U.S.-obligated material includes material transferred by the United States or material used in, or produced through, the use of material or facilities transferred by the United States. Following the conclusion of the UAE agreement, the NSC deliberated requiring the so- called “gold standard” for all nuclear cooperation agreements as a policy, but ultimately adopted a policy of pursuing it on a case-by-case basis. The nuclear cooperation agreement that the United States concluded with Taiwan in 2014 included a similar provision. By contrast, the agreement concluded with Vietnam the same year includes a political commitment, rather than a legal one, not to acquire enrichment and reprocessing capabilities. In addition to the roles of State, DOE, and the NSC discussed previously, additional U.S. agencies such as Commerce, DOD, and NRC are involved in matters related to international nuclear cooperation and the negotiation and conclusion of a nuclear cooperation agreement. Table 2 describes agency roles related to nuclear cooperation. Stakeholders we interviewed identified various potential nonproliferation benefits and concerns related to negotiating a nuclear cooperation agreement with Saudi Arabia. Specifically, stakeholders identified the following benefits: A nuclear cooperation agreement would limit production of weapons-usable material. Several stakeholders told us that a nuclear cooperation agreement with Saudi Arabia would give the United States the opportunity to directly restrict Saudi Arabia’s proliferation potential. For example, a U.S.-Saudi nuclear cooperation agreement would include a term required by the AEA that would limit Saudi Arabia’s production of weapons-usable material by prohibiting Saudi Arabia from separating plutonium accumulated in any reactor supplied under the agreement without U.S. consent. According to some stakeholders, other potential supplier countries likely would not impose such restrictions as conditions of supplying Saudi Arabia with nuclear materials or equipment. Cooperation would help the United States retain influence. Several stakeholders noted that nuclear cooperation with Saudi Arabia could help revitalize the United States as a global nuclear supplier, which would help the United States retain its current influence over global nonproliferation norms and rules. For example, as a global nuclear supplier, the United States would have greater influence in international nuclear forums such as the Nuclear Suppliers Group, which establishes nonproliferation guidelines. According to one stakeholder, the United States’ political leverage to promote strong global nonproliferation norms depends upon the United States’ retaining a leadership role in nuclear energy. Another stakeholder said that nuclear cooperation agreements provide the United States with influence over countries’ proliferation decisions. For instance, this stakeholder said that nuclear cooperation agreements include legal conditions that reinforce the legal obligations of the Treaty on the Nonproliferation of Nuclear Weapons and create an additional disincentive to violate those conditions or withdraw from the treaty. Stakeholders we interviewed also identified several proliferation concerns that U.S.-Saudi nuclear cooperation may not mitigate, and could potentially aggravate. According to these stakeholders, concerns include the following: Concerns about stated Saudi nuclear weapon ambitions and commitment to obligations. Some stakeholders expressed concern over Saudi officials’ stated interest in acquiring nuclear weapons. As previously noted, senior Saudi officials have said publicly that there could be conditions under which the country would seek to acquire nuclear weapons or develop a nuclear weapons program. For example, Saudi Crown Prince Mohammed bin Salman said publicly in 2018 that if Iran develops or obtains a nuclear weapon, Saudi Arabia would also work to do so. In 2009 and 2012, respectively, King Abdullah and Prince Turki al- Faisal were reported to have made similar statements. Some stakeholders said that the intent behind such statements was to send a message about Saudi Arabia’s posture toward Iran, but some other stakeholders said that lower-lever Saudi officials have also indicated that the country is open to pursuing nuclear weapons. Several stakeholders said that such statements should be taken seriously as indicators of Saudi nuclear weapons ambitions. One stakeholder said that such statements raise concerns as to Saudi Arabia’s commitment to its obligations under the Treaty on the Nonproliferation of Nuclear Weapons. This stakeholder also said that Saudi Arabia has demonstrated willingness to disregard the terms of transfers of U.S. conventional arms to the country, calling into question whether the country could be trusted to abide by the terms of the nuclear cooperation agreement. Concerns about the extent to which a nuclear cooperation agreement would mitigate the risks of a Saudi weapons program. Several stakeholders questioned whether the terms of an agreement would meaningfully restrict proliferation behavior. For example, notwithstanding the provision of Section 123 of the AEA that prohibits a partner country from using U.S.-obligated material or equipment for weapons purposes, some stakeholders said that another risk of nuclear cooperation is that it would provide Saudi Arabia with the infrastructure and knowledge to produce nuclear material for a future weapons program. In addition, some stakeholders said that there were questions as to whether the United States could enforce the terms of an agreement if it was breached—for example, whether in practice the United States would be able to retrieve U.S.-obligated nuclear material from another country. One stakeholder also noted that the terms of a nuclear cooperation agreement would only be relevant in mitigating proliferation risks if Saudi Arabia contracted with a U.S. company to build the reactors. If Saudi Arabia purchases reactors from other suppliers, its nuclear program will not be bound by the section 123-mandated restrictions of a nuclear cooperation agreement with the United States, since those restrictions only apply to U.S.-obligated material. Concerns about the thoroughness of a U.S. assessment of Saudi proliferation risks. Some stakeholders raised concerns about whether the NPAS process would adequately assess Saudi proliferation risks. We have previously identified weaknesses in the NPAS process related to interagency consultation and a robust, transparent review process. As described above, an NPAS for a U.S.-Saudi nuclear cooperation agreement would be expected to include an overview of Saudi Arabia’s nuclear energy program and related infrastructure, nonproliferation policies, and relations with countries of proliferation concern. An NPAS would also include an analysis of the adequacy of safeguards and other control mechanisms to ensure that assistance provided under the U.S.- Saudi agreement is not used to further any nuclear weapons effort. Some stakeholders said that it would be important for the NPAS for Saudi Arabia to address the questions regarding the country’s stated intentions to develop a nuclear weapons program. One stakeholder questioned whether an NPAS would provide a sufficient assessment of Saudi nuclear proliferation behavior or potential because the statutory requirement for intelligence community input into the NPAS is narrowly worded. Specifically, the addendum that the intelligence community is to provide to each NPAS is required to contain a comprehensive analysis of the country’s export control system with respect to nuclear-related matters, including interactions with other countries of proliferation concern and the actual or suspected nuclear, dual-use, or missile-related transfers to such countries, but the requirement does not call for the intelligence community to assess the country’s intent to develop nuclear weapons. State officials declined to tell us whether they had begun drafting an NPAS in anticipation of an agreement with Saudi Arabia. However, State officials noted that their engagement with the intelligence community in the development of an NPAS goes beyond the requirements of that statute, but they also said that the legal requirement was limited. Concerns about regional proliferation risks and undermining of global nonproliferation norms. Several stakeholders expressed concerns about the regional and international nonproliferation implications of a U.S.-Saudi nuclear cooperation agreement. For example, several stakeholders said that an agreement without restrictions on enrichment and reprocessing could lead to the renegotiation of the agreement with the UAE. The agreement with the UAE, which includes a commitment to forswear enrichment and reprocessing, also contains a provision that would allow the UAE to request renegotiation of its agreement if another country in the region concludes a less restrictive agreement with the United States. Several stakeholders also raised the concern that a nuclear cooperation agreement without additional nonproliferation conditions would undermine U.S. and global nonproliferation norms by sending the message that such norms were negotiable. For example, in addition to the Additional Protocol being a mechanism to prevent diversion of nuclear material, many stakeholders said that insisting on the Additional Protocol was critical and emphasized the importance of the Additional Protocol as a global nonproliferation norm. Several stakeholders also questioned the premise that supplying Saudi Arabia’s nuclear program would allow the United States to retain influence over international nonproliferation norms. One stakeholder said that the United States has not been a significant nuclear exporter for decades and has nonetheless retained its influence. The United States and Saudi Arabia have not made significant progress toward a nuclear cooperation agreement because of persistent differences between the parties over nonproliferation conditions, including U.S. insistence that Saudi Arabia conclude an Additional Protocol with IAEA and that Saudi Arabia agree to restrictions on enrichment and reprocessing, based on our analysis of available information. The United States and Saudi Arabia first held formal nuclear cooperation negotiations in 2012, during which the United States provided a draft agreement text to Saudi officials that included the nine nonproliferation conditions required under Section 123 of the AEA, according to NNSA officials. In that round of negotiations, Saudi officials accepted “the vast majority” of the conditions in the draft text, according to NNSA officials; these officials estimated that approximately three pages of the text remained to be negotiated. NNSA officials told us that the areas of disagreement include provisions required by the AEA. In the next formal negotiations in 2018, there was no progress in resolving the remaining issues, and no changes to the text of the agreement were made at the time, according to agency officials. The areas of disagreement that were not resolved in 2012—including those regarding provisions required by the AEA—remained unresolved as of January 2020, according to agency officials. These areas of disagreement include: Additional Protocol. The United States has urged Saudi Arabia to conclude an Additional Protocol with IAEA, according to a September 2019 letter from the Secretary of Energy to the Saudi Minister of Energy, Industry, and Mineral Resources and based on public statements by the Secretary of Energy and another government official. Several former agency officials and other stakeholders said that Saudi Arabia has expressed an unwillingness to conclude an Additional Protocol with IAEA. Restriction on enrichment and reprocessing. According to public statements by agency officials, the United States supports a permanent restriction on enrichment and reprocessing. According to the Secretary’s September 2019 letter and to former officials we interviewed, however, the United States may be willing to accept a temporary restriction on enrichment and reprocessing in its negotiations with Saudi Arabia. According to these former officials, such a temporary restriction would allow the United States and other countries more time to work with Saudi Arabia to reach agreement on mutually acceptable terms. However, one stakeholder said that this option would not be attractive to Saudi Arabia and would not be useful to the United States as a nonproliferation measure because an existing nuclear cooperation agreement and any nuclear infrastructure that it would have enabled would reduce U.S. leverage to influence Saudi enrichment and reprocessing decisions in the future. Despite the lingering disagreement on certain provisions between both countries, NNSA officials told us in November 2019 they believed the negotiations had made progress since 2012 because the continued interactions with Saudi officials over this time were useful in advancing Saudi understanding of the United States’ position on the nonproliferation conditions of a potential agreement. We are unable to characterize Saudi views on the status of the negotiations or on other aspects of our review, because State did not respond to our repeated requests for assistance in facilitating travel to Saudi Arabia and interviews with relevant Saudi officials. We also did not receive a response to our written request to the Saudi ambassador to the United States for an opportunity to interview relevant Saudi officials about the negotiations. Agency management of U.S.-Saudi nuclear cooperation negotiations remains unclear with regard to agency roles and informing Congress. We were unable to confirm U.S. agency roles at a range of U.S.-Saudi interactions where nuclear cooperation was or may have been discussed. We were also unable to determine whether the agencies kept the relevant congressional committees fully and currently informed of the negotiations. The roles various U.S. agencies have played in U.S.-Saudi nuclear negotiations remain unclear because DOE and State did not provide us with information to clarify or corroborate such roles. According to a State official and DOE officials, State would have “by definition” led any negotiations and without State present, any interactions between U.S. and Saudi officials on nuclear cooperation did not constitute negotiations. The AEA stipulates that State conduct any nuclear cooperation negotiations but does not define “negotiations.” According to one stakeholder, during an NSC meeting in late 2017, during which nuclear cooperation with Saudi Arabia was discussed, the NSC made a decision to reinforce established agency roles, including specifying that State would lead any negotiations. We were unable to confirm whether NSC made such a decision because NSC did not respond to our requests for interviews or documentation. However, through our interviews with State, DOE, and NRC officials, we determined that representatives of each agency participated in the 2012 and March 2018 formal nuclear cooperation negotiations with Saudi Arabia. State and DOE officials did not provide information that we requested about interactions between the United States and Saudi Arabia, such as the dates and agency participants. However, despite the limited cooperation from State and DOE, we were able to identify through our analysis of documentation and interviews with other stakeholders, a range of interactions between the United States and Saudi Arabia where nuclear cooperation was or may have been discussed. The interactions we were able to identify during which potential nuclear cooperation was discussed are as follows: five bilateral meetings, including a September 2018 meeting in Washington, D.C., a December 2018 meeting in Saudi Arabia, and an August 2019 meeting in Washington, D.C.; a Civil Nuclear Energy Roundtable in Saudi Arabia in December 2017, a commercial nuclear mission to Saudi Arabia in April 2018, in partnership with DOE; and the letter from the Secretary of Energy to his Saudi counterpart in September 2019 conveying U.S. positions on nonproliferation conditions for U.S.-Saudi nuclear cooperation. We also identified five interactions where the U.S. Secretary of Energy and Saudi officials may have discussed nuclear cooperation, including a phone call in November 2017 and meetings on the sidelines of four events: the IAEA General Conference in Austria in September 2017, the Bilateral Energy Dialogue in Saudi Arabia in December 2017, the World Economic Forum in Switzerland in January 2018, and the Future Investment Initiative in Saudi Arabia in October 2019. Figure 3 illustrates U.S.-Saudi negotiations and other interactions, and appendix II includes a detailed list of the interactions we were able to identify. Because State and DOE did not cooperate with our information requests, we cannot confirm that the interactions we identified constitute all of the interactions between the United States and Saudi Arabia on potential nuclear cooperation since 2012. Furthermore, we were unable to determine whether the agencies followed the established roles in the other interactions with Saudi Arabia where nuclear cooperation was or may have been discussed because NSC, State, and DOE did not respond to our requests for information to clarify these matters. Specifically, with the exception of the April 2018 commercial nuclear mission to Saudi Arabia, we were unable to determine whether State or other agency officials authorized, were present for, or were aware of a number of DOE–led interactions with Saudi Arabia described above. In addition, State and DOE officials declined to confirm whether State authorized the September 2019 letter from the Secretary of Energy to his Saudi counterpart regarding U.S. positions on the nonproliferation conditions for nuclear cooperation. It is unclear whether the agencies kept the relevant committees fully and currently informed of U.S.-Saudi negotiations. State officials stated that they consistently provide information to Congress, but the limited information they provided to us does not support this position. As previously stated, section 123 of the AEA requires that the President keep certain congressional committees “fully and currently informed of any initiative or negotiations relating to a new or amended agreement for peaceful nuclear cooperation.” State officials told us during our May 2019 interview that they consistently provided information to Congress on the nuclear cooperation negotiations and other interactions with Saudi Arabia. However, neither State nor DOE provided documentation within the time frame of our review to support these statements. DOE did not respond to our request for information on any dates or related details of any congressional briefings related to U.S.- Saudi nuclear cooperation negotiations. State did not respond to our initial request in May 2019 for information on dates and related details of any congressional briefings it held on U.S.-Saudi nuclear cooperation negotiations. However, in January 2020, after reviewing a preliminary draft of this report, State officials provided a list of congressional briefings on U.S. nuclear cooperation initiatives since 2013. We reviewed this list and identified two briefings specifically focused on nuclear cooperation negotiations with Saudi Arabia: one held in January 2018 for House Committee on Foreign Affairs staff and another held in May 2019 for House Committee on Oversight and Reform staff. State officials also noted that U.S.-Saudi nuclear cooperation may have been discussed in other State briefings that focused on nuclear cooperation in general or with other countries, such as briefings to the House Committee on Foreign Affairs and Senate Committee on Foreign Relations in July 2019 and November 2019. State officials declined to discuss the details of any congressional briefings with us, including the participating agencies, substantive issues, and other details. Consequently, we could not establish the extent and substance of information the agencies provided to Congress on U.S.-Saudi nuclear cooperation negotiations. After State did not provide us with the information we requested, we reached out to a number of current and former staff of the House Committee on Foreign Affairs and Senate Committee on Foreign Relations, representing both parties. Through our interviews with eight of these staff, we were able to identify one congressional briefing by the agencies in December 2017 on the status of U.S.-Saudi nuclear cooperation negotiations. However, based on our interviews with congressional staff, we were unable to identify the dates of any other briefings by the agencies on the U.S.-Saudi nuclear cooperation negotiations. Notably, based on our review of the documentation and interviews with congressional staff, it does not appear that the agencies provided a briefing to the House Committee on Foreign Affairs or Senate Committee on Foreign Relations until more than a year after the last formal U.S.-Saudi nuclear cooperation negotiations in March 2018. Current and former congressional staff we interviewed also described their frustration in trying to obtain information, beyond briefings, from the agencies on the status of the negotiations. Several current and former congressional committee staff we interviewed told us that they learned of developments in the U.S.-Saudi negotiations through the press or from representatives of the nuclear industry, rather than directly from the agencies, despite having asked the executive branch to keep them informed of any developments. For example, one former staff member of a relevant committee told us that they learned of the March 2018 formal negotiations just days before the meeting through a press article. Another former congressional committee staff member said that since late 2017, the agencies have only provided information to Congress about the negotiations in response to forceful measures, such as holds on nominations or legislation. According to many of the current and former congressional staff we interviewed, this stands in contrast to past practice in which agencies regularly briefed the committees on nuclear cooperation negotiations without coercion, and sometimes even initiated the meetings. State and DOE provided Congress with contradictory justifications for not providing such information to Congress, according to our review of documents and interviews with congressional staff. For example, one congressional committee staff member told us that agency officials said they were not obligated to keep the committee currently and fully informed of negotiations because the United States was not in negotiations with Saudi Arabia. On another occasion, when pressed by members of Congress in congressional hearings, an agency official said he could not discuss nuclear cooperation negotiations with Saudi Arabia because negotiations were ongoing. Specifically, in September 2019, the Assistant Secretary of State for International Security and Nonproliferation stated in a hearing that he could not get into details of nuclear cooperation negotiations with Saudi Arabia because the negotiations were ongoing. These contradictory justifications may have led to inconsistency in the agencies providing information to Congress on nuclear cooperation negotiations. By committing to regularly scheduled, substantive briefings to Congress on nuclear cooperation initiatives and negotiations, State and DOE could enhance transparency and build confidence with Congress on nuclear cooperation, preemptively address congressional concerns about cooperation with certain countries, and support congressional oversight on nonproliferation matters. Former congressional staff, including those involved in drafting Section 123(e) in 2008—the “fully and currently informed” provision—said the intent of the provision was to promote transparency on the status of any nuclear cooperation negotiations to the congressional committees of jurisdiction to lay the groundwork for congressional consideration of any agreement. However, some former congressional staff said that the provision allows for broad interpretation and that it may be up to Congress to more clearly define the “fully and currently informed” requirement. By specifying, through an amendment to the AEA, its expectations for timeliness and information provided by the agencies on nuclear cooperation negotiations and initiatives, Congress could have better assurance that it will get the information it needs for its oversight of nuclear nonproliferation matters. State officials told us that they consistently provided information to Congress on the nuclear cooperation negotiations and other interactions with Saudi Arabia. They later provided a list of congressional briefings on U.S. nuclear cooperation initiatives since 2013 but did not specify what was discussed. Based on this limited information, it is unclear whether the briefings by State kept Congress fully and currently informed of developments in the negotiations with Saudi Arabia, and congressional staff provided us with examples of having to find information on the negotiations from other sources, such as press articles. NNSA is a separately organized agency within the Department of Energy, with responsibility for its nuclear weapons and nonproliferation programs, among other things. transparency and build confidence with Congress on nuclear cooperation, preemptively address concerns about cooperation with certain countries, and support congressional oversight on nuclear nonproliferation matters. Former congressional staff involved in drafting the “fully and currently informed” provision said that its intent was to promote transparency and lay the groundwork for congressional consideration of any agreement. However, some said that this provision allows for broad interpretation of the “fully and currently informed” requirement. By specifying, through an amendment to the AEA, its expectations for timeliness and information provided by the agencies regarding nuclear cooperation negotiations and initiatives, Congress could have better assurance that it will get the information it needs for its oversight of nuclear nonproliferation matters. Congress should consider amending the Atomic Energy Act to require regularly scheduled briefings, for instance, on a quarterly basis, and specify expectations for the content of such briefings, such as potential difficulties in negotiating nonproliferation conditions with partner countries. The Secretary of State, in coordination with the Secretary of Energy, should commit to regularly scheduled, substantive briefings for the House Committee on Foreign Affairs and the Senate Committee on Foreign Relations on all initiatives and negotiations related to nuclear cooperation in order to enhance transparency and establish greater confidence with Congress on nuclear cooperation matters. (Recommendation 1) We provided a draft of this report to the Secretaries of State, Energy, Defense, and Commerce, and to the Chairman of the NRC for review and comment. In its written comments, reproduced in appendix III, State neither agreed nor disagreed with our findings, and concurred with our recommendation. State also noted in its response that it is already implementing the recommendation; specifically, that it conducted briefings on nuclear cooperation in 2018 and 2019 to Congress. However, as we noted in our report, because State officials declined to discuss the details of these briefings, we could not establish the extent and substance of information the agencies provided to Congress on U.S.-Saudi nuclear cooperation negotiations. Furthermore, as we reported, staff of the relevant congressional committees we interviewed were able to identify only one briefing on U.S.-Saudi nuclear negotiations and several staff expressed frustration in trying to get information about the negotiations, including learning of developments through the press. NRC also provided written comments, which are reproduced in appendix IV; NRC neither agreed nor disagreed with our recommendation. DOE provided technical comments, which we incorporated as appropriate. DOD and Commerce did not have any comments. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of State, the Secretary of Energy, the Secretary of Defense, the Secretary of Commerce, the Chairman of the Nuclear Regulatory Commission, and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. NNSA is a separately organized agency within the Department of Energy, with responsibility for its nuclear weapons and nonproliferation programs, among other things. Saudi nuclear cooperation negotiations and any areas of disagreement, we also reviewed official documentation such as agency correspondence to Saudi officials and transcripts of congressional hearings. In addition, we submitted to the Saudi Ambassador to the United States a written request for an opportunity to interview relevant Saudi officials about the negotiations, but did not receive a response. To examine U.S. agency management of the negotiations, including how the agencies have informed Congress about the negotiations, we reviewed official documentation such as agency correspondence to Saudi officials, certain export authorization application packages, dates of congressional briefings on nuclear cooperation, and agency documentation related to U.S. government advocacy for U.S. businesses related to nuclear cooperation with Saudi Arabia. We also requested a list of dates and participants of U.S.-Saudi interactions pertaining to nuclear cooperation, as well as materials used for briefings, if any, by the agencies to Congress. The agencies provided us with limited information in response to some categories we requested and did not provide information in other categories. Specifically, beginning in May 2019, we requested from the Departments of State and Energy and the National Security Council (NSC) basic factual information on license applications for the transfer of nuclear technology to Saudi Arabia; the dates of any discussions or negotiations between U.S. and Saudi officials; the U.S. and Saudi agencies, offices, and representatives present at such meetings; and the types of records produced from such meetings. DOE provided us with information on the license applications, and State and DOE provided us with limited information on their general processes relating to the negotiation of agreements. State officials also provided a list of congressional briefings on U.S. nuclear cooperation initiatives since 2013 in January 2020, after reviewing a preliminary draft of this report, but declined to discuss the details of the briefings with us, including the participating agencies, substantive issues, and other details that would have allowed us to establish the extent of information provided to Congress on U.S.-Saudi nuclear cooperation negotiations. Furthermore, neither agency nor NSC provided substantive information in any of the other categories we requested; in order to complete this review within a time frame responsive to the needs of our congressional requesters, we adjusted our audit objectives to focus on examining the status of the negotiations and management of the negotiations process. Because State, NSC, and DOE did not provide information to fully address these adjusted objectives, we obtained documentation and information from other agency officials and over 30 other stakeholders, including, as previously noted, former senior U.S. government officials, current and former congressional staff, and nuclear industry representatives and knowledgeable nongovernmental experts who have followed the negotiations. We conducted our work from April 2019 through April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Since the United States and Saudi Arabia signed a memorandum of understanding on nuclear energy cooperation in 2008, there have been a variety of interactions between the United Sates and Saudi Arabia regarding potential nuclear cooperation between both countries, as well as other developments related to such cooperation. The Atomic Energy Act (AEA) does not define “negotiations.” In this report, we use “formal nuclear cooperation negotiations” and “formal negotiations” to signify sessions where parties aim to agree on specific terms and conditions in the text of an agreement. We use the term “interactions” for all U.S.-Saudi encounters on potential nuclear cooperation other than the two formal negotiations explicitly identified by agency officials. Table 3 provides information on dates we identified of formal U.S.-Saudi negotiations; other U.S.-Saudi interactions; National Security Council meetings to discuss policy and related matters on U.S.-Saudi negotiations; agency briefings to Congress on the negotiations; and other related developments, including developments in Saudi Arabia related to its planned nuclear power program. See table 3 for more information. In addition to the individual named above, other key contributors to this report were William Hoehn, Assistant Director; Alisa Beyninson, Analyst in Charge; Antoinette Capaccio; Tara Congdon; Camille Pease; Steven Putansu; Dan Royer; Sara Sullivan; and Madeline Welter.", "summary": "U.S. policy has long sought to balance U.S. civilian nuclear exports with the nation's obligation to ensure that they are not used to proliferate nuclear weapons. The Atomic Energy Act (AEA) provides a framework for certain civilian nuclear exports and outlines the requirements for nuclear cooperation agreements, including that certain nonproliferation conditions be met; that State conduct negotiations with the technical assistance and concurrence of DOE; and that the President keep certain congressional committees fully and currently informed of negotiations or initiatives. This report describes, among other things, (1) the status of U.S.-Saudi negotiations and any areas of disagreement and (2) what is known about U.S. agency management of the negotiations. GAO reviewed the AEA and documentation of interactions between U.S. and Saudi officials regarding nuclear cooperation. GAO received limited information from State and DOE officials during the review but interviewed over 30 other stakeholders, including former senior executive branch officials, former congressional staff, and others with knowledge of and insights into nuclear cooperation issues and the negotiations. Since 2008, when the United States and Saudi Arabia signed a memorandum of understanding on nuclear energy cooperation, the current and prior U.S. administrations have engaged in discussions and negotiations about nuclear cooperation with the Saudi government. However, these negotiations are stalled; the two countries have not been able to resolve disagreements on several nonproliferation conditions, including Saudi Arabia agreeing to enrichment and reprocessing restrictions and signing an Additional Protocol with the International Atomic Energy Agency (IAEA), which would allow IAEA to obtain additional information about and access to Saudi nuclear activities. U.S. agency management of the negotiations with Saudi Arabia remains unclear in two areas regarding AEA requirements—(1) that the Department of State (State) conduct negotiations, with the technical assistance and concurrence of the Department of Energy (DOE), and (2) that certain congressional committees be informed. First, it is unclear which U.S. agencies were present at or aware of various interactions where nuclear cooperation was or may have been discussed, except for the formal negotiations in 2012 and 2018 and a commercial mission coordinated with State. GAO was able to identify eight interactions where nuclear cooperation was discussed and five more interactions where nuclear cooperation may have been discussed (see figure). Note: Interactions depicted in this figure include meetings, phone calls, and a letter, among other things. Second, GAO was unable to determine whether the agencies kept the committees fully and currently informed. GAO identified two briefings on the negotiations—in December 2017 and January 2018—to the relevant committees, but it does not appear that these committees were briefed until more than a year after the March 2018 formal negotiations. According to congressional staff, Congress on occasion learned of developments through non-agency sources and had to apply forceful measures, including holds on nominations, to get information from the executive branch. By committing to regular briefings to Congress on nuclear cooperation negotiations and initiatives, State could better support congressional oversight on nuclear nonproliferation matters. In addition, congressional staff have said the AEA allows for broad interpretation of the “fully and currently informed” requirement. By specifying, through an amendment to the AEA, its expectations for timeliness and information provided by the agencies on nuclear cooperation negotiations and initiatives, Congress could have better assurance that it receives the information it needs for oversight of nuclear nonproliferation matters. GAO believes that Congress should consider amending the Atomic Energy Act to require regularly scheduled briefings. GAO is also making a recommendation that the Secretary of State commit to regularly scheduled, substantive briefings to the relevant congressional committees. State concurred with our recommendation.", "document_type": "gao"}
{"report": "In 2014, Congress issued two separate mandates to DOT to develop strategies related to challenges facing the U.S.-flag fleet, specifically: The Secretary of Transportation was directed to develop a national maritime strategy with recommendations to, among other things, help U.S.-flag vessels remain competitive. The Secretary of Transportation and the Maritime Administration (MARAD) within DOT were directed to develop, in collaboration with DOD, a national sealift strategy to ensure the long-term viability of U.S.-flag vessels and U.S.-citizen mariners. As we reported in August 2018, according to MARAD and DOD officials, MARAD has been working on a single draft maritime strategy to meet both mandates because the broader national maritime strategy would need to encompass the national sealift strategy, as well. While there is no statutory deadline for the completion of the national sealift strategy, in the John S. McCain National Defense Authorization Act for Fiscal Year 2019, the statutory deadline for the national maritime strategy was extended from February 2015 to February 2020. In our August 2018 report, we noted that MARAD officials had completed a draft strategy in 2016, but they told us that the strategy was subject to the new administration’s review. At that time, MARAD and DOT officials told us that they viewed the existing draft strategy as pre-decisional and could provide no timeline for when they planned to move the strategy forward. In our report, we concluded that the delay in submitting the strategy to Congress had resulted in decision-makers not having the information they needed and recommendations from the agency to inform policy-making in this area. We recommended that DOT complete the national maritime strategy and establish time frames for its issuance. DOT concurred with our recommendation. In our recent discussions with DOT officials after passage of the John S. McCain National Defense Authorization Act for Fiscal Year 2019, they told us that DOT now plans to meet the new statutory deadline and issue the strategy by February 2020. Stakeholders we spoke with for our August 2018 report identified two primary challenges to ensuring that the U.S.-flag fleet would continue to meet DOD’s national defense needs. First, they described maintaining the financial viability of U.S.-flag vessels participating in MSP as a challenge. Second, stakeholders identified a potential shortage of U.S. citizen mariners available to crew the government-owned reserve fleet during a military activation as a challenge, in part due to the declining numbers of U.S.-flag vessels that employ these mariners. In our August report, we noted that MARAD had identified some options to address the competitiveness of U.S.-flag vessels and the long-term viability of the U.S.-citizen mariners—issues that are very similar to the key challenges identified by stakeholders. However, DOT and MARAD officials had stated that they were not yet ready to address the feasibility of these options, or formally propose them. According to MARAD officials, the relative cost of operating a U.S.-flag vessel compared to a foreign-flag vessel has increased in recent years, making it more challenging for vessel operators to remain economically viable under the U.S. flag. In our August report we found that financial support to U.S.-flag vessels through both the MSP stipend and the government cargo preference requirements has helped ensure a sufficient number of internationally trading U.S.-flag vessels are available to meet DOD’s most recently stated cargo capacity needs from such vessels. On the other hand, according to MARAD officials, the additional cost of operating a U.S. flag vessel compared to a foreign-flag vessel has increased—from about $4.9 million annually in 2009 and 2010 to about $6.2 to $6.5 million currently—making it harder for such vessels to remain financially viable. This cost differential results primarily from the rising relative costs of employing U.S. versus foreign mariners as crew. Compounding the increasing costs of operating U.S. flag vessels, the volume of government cargo—a key source of revenue for many U.S.- flagged vessels—has fallen in recent years as the international military presence of the United States and funding for food aid overseas have both declined. In response to these challenges, Congress increased the MSP stipend from $3.5 million to $4.99 million per vessel from fiscal year 2016 to 2017. MARAD officials said this increase has temporarily stabilized the financial situation of MSP vessel operators. However, they added that trends in operating costs and government cargo suggest that the ability to retain an adequate number of financially-viable U.S.-flagged vessels will remain an ongoing challenge. MARAD officials identified the following options as having potential to reduce the costs of operating a U.S.-flag vessel—which would in turn make U.S.-flag vessels more competitive in the international cargo market: MARAD is part of a U.S. Registry Working Group looking at a range of actions to decrease the time and cost of bringing vessels under the U.S. flag, including the cost of meeting Coast Guard requirements. For example, the group is looking at a recommendation for the broader application of internationally recognized vessel standards to U.S.-flag vessels to meet Coast Guard requirements. In the current strategic plan for 2017 through 2021, MARAD identified two areas of reform—mariner income-tax relief and liability insurance reform—that could reduce the crew costs of operating under a U.S. flag. According to MARAD officials, some stakeholders have recommended that MARAD consider requesting the elimination of a tax on U.S.-flag vessels receiving maintenance overseas to reduce maintenance costs for U.S.-flag vessels. In general, maintenance and repairs on U.S.-flag vessels not conducted at U.S. shipyards are subject to a statutory 50 percent ad valorem tax on the cost of maintenance performed in a foreign country. According to 12 of the 14 MSP vessel operators we spoke with for our August report, U.S. shipyards are typically more expensive than foreign shipyards or may not be close to the vessel’s location or route, so they typically choose to pay the tax and have the maintenance performed overseas. Four MSP vessel operators we spoke to stated that they send U.S.-flag vessels to U.S. shipyards for maintenance when it makes sense from a logistical and financial perspective. MARAD officials we spoke to said they are considering the effect of eliminating the tax, a step that would reduce costs for vessel operators but would potentially negatively affect the financial viability of U.S. shipyards, which the law was designed to assist. However, MARAD officials stated that they have not yet evaluated these trade-offs. MARAD and DOD’s Transportation Command (Transportation Command) officials have also identified—but not officially proposed—several options to address the decline in government cargo carried on U.S.-flag vessels, which would also make U.S.-flag vessels more competitive by providing more revenues. In our August 2018 report, Transportation Command officials and ship operators to whom we spoke told us that they consider access to U.S. government cargo to be a critical means of sustaining U.S.-flag vessels. Transportation Command and MARAD officials stated that one way to increase the amount of commercial cargo on U.S.-flag vessels would be to require that certain energy export commodities, such as oil or liquefied natural gas, be carried on U.S.-flag vessels. While this option has been considered in the past, it would require new legislation and would potentially have a negative impact on the export market for liquefied natural gas. In 2015, we analyzed the potential effects of a requirement that U.S. liquefied natural gas exports be carried on U.S.- built and -flagged vessels. We found that such a requirement could potentially increase the number of U.S.-flag vessels by 100 over the course of many years; however, due to their higher operating costs, this would increase the cost of transporting liquefied natural gas from the United States, decrease the competitiveness of U.S. liquefied natural gas in the world market, and in turn, reduce demand for U.S. liquefied natural gas. MARAD officials stated that another option would be increasing the percentage of other cargo, such as food aid, that civilian agencies are required to transport on U.S.-flag vessels. This would also require an amendment to existing legislation and would also have trade-offs, since cargo requirements such as these can result in higher shipping costs that can negatively affect the missions of civilian agencies, in particular food aid agencies. Another option identified by MARAD officials to address declining government cargo volumes would be to increase the MSP stipend to replace some of the government support previously provided through cargo preference requirements, as was done for fiscal year 2017. The second challenge identified by stakeholders related to maintaining adequate sealift for defense needs is the potential shortage of U.S.- citizen mariners available to crew the government-owned reserve fleet during a crisis. The government’s reserve fleet vessels are held in reduced operating status with minimal crew in peacetime. When put into full operating status—such as for a surge related to a wartime effort— these vessels need additional crew, and DOD counts on mariners working on oceangoing U.S.-flag vessels to meet this need. MARAD and DOD have raised concerns about the sufficiency of U.S.-citizen mariners to meet this need. For example, in January 2018, in a statutorily mandated report, MARAD’s Maritime Workforce Working Group estimated a shortage of over 1,800 mariners in the case of a drawn-out military effort, although it also recommended data improvements to increase the accuracy of the count of available mariners. Specifically, in this report, the working group estimated approximately 11,768 qualified and available U.S.-citizen mariners as of June 2017— 1,839 less than the 13,607 mariners the working group estimates would be needed for sustained operation of the reserve and commercial fleet. The working group based its identification of 11,768 existing qualified U.S.-citizen mariners on the number of U.S.-citizen mariners actively sailing on U.S.-flag commercial and government-owned oceangoing vessels. For the vessels in full operating status, the working group accounted for 2 mariners employed for each crew position. The double crew, which according to MARAD officials is typical for a commercial U.S.-flag vessel operating in international trade, allows each mariner, over the course of a year, to work for 6 months on the vessel and take 6 months of earned leave. The working group assumed that during a military activation, commercial operations would continue at the same level as during peacetime—but that some U.S-citizen mariners currently working on commercial vessels would be willing to reduce the amount of earned leave they took in order to work on government-owned reserve vessels. The working group analyzed this scenario by changing the ratio of crew positions to crew from 2 to 1.75. As illustrated in figure 1, under this scenario, with an average of 26 crew positions per vessel, between 6 and 7 mariners per existing commercial oceangoing U.S.-flag vessel are made available to crew the reserve fleet. According to the working group’s methodology, given the size of the current U.S.-flag oceangoing fleet and the number of currently employed mariners on this fleet, there are enough U.S.-citizen mariners to crew the reserve fleet during an initial surge, but not for a sustained activation, during which the working group estimated that the reserve vessels themselves would need a double crew to allow for crew rotations. This need for crew rotations on the reserve vessels led the working group to estimate a shortage of 1,839 U.S.-citizen mariners. Moreover, the working group’s report found that the shortage of mariners may be understated if some of the estimated available mariners are unable or unwilling to continue sailing during times of national emergency, as available mariners are not required to crew the reserve fleet. Although the working group concluded that there is a shortage of mariners for sustained operations, its report also details data limitations that cause some uncertainty regarding the actual number of existing qualified mariners and, thus, the extent of this shortage. The working group’s approach—driven, in part, by limitations of the U.S. Coast Guard’s database that tracks mariner credentials—did not count any qualified mariners who are no longer employed on U.S.-flag oceangoing vessels or who are employed on other types of vessels but may have the required credentials. In fact, according to the working group’s analysis, over 15,000 mariners listed in the U.S. Coast Guard’s database have unlimited credentials but are unaccounted for, as they are neither currently employed on large, oceangoing vessels nor serving as civil- service mariners committed to government-owned vessels. The working group stated that the availability and continuing proficiency of these mariners remains unknown. MARAD officials emphasized to us, however, that mariners who have not worked on the right types of vessels for more than 18 months are likely to need additional training before they would be qualified to crew the reserve fleet during a military activation. The working group’s report contains several recommendations related to improving information on the number of available and willing mariners. These recommendations include replacing the Coast Guard database with one that would enable a more accurate account of available mariners, and establishing a periodic survey of the U.S.-citizen mariner pool to allow MARAD to determine, with reasonable certainty, how many qualified mariners would be available and willing to sail on U.S.- government reserve vessels if called upon to do so. The report concluded that until these agencies improve the tracking of licensed mariners who may be available to crew the government-owned reserve vessels when activated into full operating status, the extent to which there is a shortage of mariners for defense needs will remain unclear. The lack of information on the extent to which there is a shortage of mariners limits the U.S. government’s ability to effectively plan for such needs. In January 2018, MARAD’s administrator testified that MARAD is working with the Coast Guard and the maritime industry to better track licensed mariners who may no longer be sailing but could serve in a time of crisis, and in March 2018, MARAD officials told us they are taking steps to initiate a new survey of mariners, as recommended in the working group’s report. In its report, the working group also identified options to address the challenge of ensuring a sufficient number of U.S.-citizen mariners for defense needs. It identified two actions that could help increase the number of U.S.-citizen mariners—one focused specifically on mariners and the other focused more broadly on the merchant marine, which encompasses U.S.-flag vessels and U.S.-citizen mariners. However, the working group’s report did not discuss specific costs or trade-offs related to either action or elaborate any further on them. The report identified the following actions: MARAD should develop a broad-based reserve program that would identify and support qualified mariners willing to sail in commercial and government-owned vessels during an emergency. MARAD would provide limited financial assistance in training mariners and maintaining credentials, in turn for which mariners who participate would be obligated to sail in the event of a defense need. MARAD and other U.S. government agencies should support a healthy merchant marine (which encompasses U.S.-flag vessels and U.S.-citizen mariners). The government should fully support programs including MSP, requiring the government to ship certain cargo on U.S flag vessels, the Jones Act, and government chartering of privately owned vessels. If DOD determines that national needs require more mariners and vessels than can be provided through current programs, those programs should be expanded to meet such needs. In conclusion, the U.S.-flag fleet is increasingly facing challenges that threaten its ability to meet future defense needs. In response to congressional mandates, MARAD has been working on a national maritime strategy and plans to issue one by February 2020. However, until such a strategy is in place, decision-makers will have limited information to make important policy choices that consider all the relevant tradeoffs associated with this complex issue. Chairman Mast, Ranking Member Garamendi, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions. If you or your staff have any questions about this testimony, please contact Andrew Von Ah, Director, Physical Infrastructure, at (202) 512- 2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Alwynne Wilbur (Assistant Director), Stephanie Purcell, (Analyst in Charge), Bonnie Ho, Christopher Jones, and Amy Rosewarne. Other staff who made key contributions to the report cited in the testimony are identified in the source product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The U.S. government relies on U.S.-flag vessels to transport cargo and provide a pool of U.S.-citizen mariners who could be called upon to support defense needs in times of war or crisis. Through financial support and by requiring government agencies to ship certain cargo on U.S. flag vessels, the United States has supported the viability of the U.S.-flag fleet. However, concern has grown about the fleet's future sustainability. In 2014, Congress mandated that DOT develop national strategies to address this issue. This statement summarizes GAO's August 2018 report on challenges in sustaining the U.S. flag fleet for defense purposes and DOT's efforts to draft a national maritime strategy that addresses these challenges. Specifically, it discusses: (1) the status of the mandated national strategies and (2) challenges that stakeholders identified related to sustaining the U.S.-flag fleet and options DOT has considered for addressing them. For the August 2018 report, GAO reviewed relevant laws, regulations, reports, and studies. GAO also analyzed data on international government cargo and interviewed officials from DOT and DOD, vessel operators, and other stakeholders. For this statement, GAO spoke to DOT officials for an update on the status of the strategy. The Department of Transportation (DOT) is still finalizing the national maritime strategies that were called for in two separate mandates by Congress in 2014. According to DOT officials, DOT has been working on a single draft maritime strategy to meet both mandates. This strategy is intended to address how to make vessels registered to the United States (U.S.-flag vessels) more competitive in the international cargo market. It is also intended to address how to ensure the long-term viability of U.S.-flag vessels and U.S.-citizen mariners. The Department of Defense (DOD) counts on U.S.-citizen mariners that work on U.S.-flag vessels to crew the government-owned reserve fleet during a crisis. In an August 2018 report, GAO concluded that by not completing the strategy or establishing a timeline for completing it, DOT had delayed providing decision-makers the information they needed to address challenges facing the U.S. flag fleet. Subsequently, with the passage of the John S. McCain National Defense Authorization Act for Fiscal Year 2019, Congress extended the deadline for the strategy to February 2020. According to DOT officials, DOT will issue the strategy by the new deadline. Stakeholders GAO spoke with for its August 2018 report identified two primary challenges to ensuring that the U.S.-flag fleet would continue to meet DOD's national defense needs: (1) maintaining the financial viability of the U.S.-flag fleet, which is threatened by the increasingly higher costs of operating U.S. vessels compared to foreign flag vessels and a decrease in government cargo being shipped internationally; and (2) a potential shortage of U.S. citizen mariners available to support defense needs, in part due to the declining numbers of U.S.-flag vessels that employ these mariners. For example, the number of U.S. flag vessels involved in international trade declined from 199 vessels at the end of 1990 to just 82 vessels by the end of 2017. DOT officials have identified some options to make U.S.-flag vessels more competitive, increase the amount of commercial cargo on U.S. flag vessels, and address a potential shortage of U.S.-citizen mariners, although they are not ready to assess their feasibility or formally propose these options. To address the challenge of maintaining the financial viability of U.S.-flag vessels, DOT has identified options such as changing regulations to decrease the costs of bringing a ship under the U.S. flag and requiring that certain energy export commodities, such as oil or liquefied natural gas, be carried on U.S.-flag vessels. To address the potential shortage of U.S.-citizen mariners, DOT convened a working group to determine how many mariners would be needed to meet defense needs. The working group estimated a shortage of over 1,800 U.S.-citizen mariners in the event of a sustained military activation, although it also recommended data improvements to increase the accuracy of the count of available mariners. In addition, the working group identified two actions that could help increase the number of U.S.-citizen mariners: (1) developing a reserve program to identify and support qualified mariners willing to sail to support defense needs during an emergency and (2) expanding programs and requirements that support U.S.-citizen mariners, such as requirements that government agencies must ship certain cargo on U.S. flag vessels. In the August 2018 report, GAO recommended that DOT complete the national maritime strategy and establish time frames for its issuance. DOT concurred with the recommendation.", "document_type": "gao"}
{"report": "SBA has not fully addressed deficiencies we have previously identified for the WOSB program, and these deficiencies are affected by SBA’s ongoing implementation of changes to the program authorized by the National Defense Authorization Act of 2015 (2015 NDAA). As of early June 2019, SBA had implemented one of the three changes to the program authorized in the 2015 NDAA. Specifically, in September 2015 SBA published a final rule to implement sole-source authority (to award contracts without competition), effective October 2015. The two other changes—authorizing SBA to implement its own certification process for WOSBs and requiring SBA to eliminate the option for firms to self-certify that they are eligible for the WOSB program—had not been implemented. On May 14, 2019, SBA published in the Federal Register a proposed rule that eliminates the self-certification option and describes a potential certification process to be administered by SBA. SBA officials have stated that the agency will not eliminate self-certification until the new certification process for the WOSB program is in place, which they expect to implement by June 2021. In addition, SBA has not fully addressed WOSB program oversight deficiencies described in our March 2019 report and first identified in our 2014 report. We reported that SBA did not have formal policies for reviewing the performance of its four approved third-party certifiers (private entities approved by SBA to certify the eligibility of WOSB firms), including their compliance with their agreements with SBA. Further, we found that SBA had not developed formal policies and procedures for, among other things, reviewing the monthly reports that certifiers submit to SBA. We recommended that the Administrator of SBA establish comprehensive procedures to monitor and assess the performance of the third-party certifiers in accordance with their agreements with SBA and program regulations. While SBA has taken some steps to address our recommendation, including conducting a compliance review of the certifiers in 2016, SBA officials said in June 2018 that SBA had no plans to conduct further compliance reviews until the final rule implementing the new certification process was completed. By waiting to improve its oversight of the WOSB program, SBA cannot provide reasonable assurance that certifiers are complying with program requirements and cannot improve its efforts to identify ineligible firms or potential fraud. In addition, the implementation of sole-source authority in light of these continued oversight deficiencies can increase program risk. Consequently, we maintain that our recommendation should be addressed. SBA also has not fully addressed deficiencies related to eligibility examinations that we described in our March 2019 report and first identified in our October 2014 report. We found that SBA lacked formalized guidance for its eligibility examination processes and that the examinations identified high rates of potentially ineligible businesses. As a result, we recommended that SBA enhance its examination of businesses that register for the WOSB program to ensure that only eligible businesses obtain WOSB set-asides. Specifically, we suggested that SBA should take actions such as (1) completing the development of procedures to conduct annual eligibility examinations and implementing such procedures; (2) analyzing examination results and individual businesses found to be ineligible to better understand the cause of the high rate of ineligibility in annual reviews and determine what actions are needed to address the causes, and (3) implementing ongoing reviews of a sample of all businesses that have represented their eligibility to participate in the program. SBA has taken some steps to implement our recommendation, such as including written policies and procedures for WOSB program eligibility examinations in a standard operating procedure and a Desk Guide. However, SBA does not collect reliable information on the results of its annual eligibility examinations. In addition, SBA continues to have no mechanism to look across examinations for common eligibility issues to inform the WOSB program. As we noted in 2014, by not analyzing examination results broadly, the agency is missing opportunities to obtain meaningful insights into the program, such as the reasons many businesses are deemed ineligible. Further, SBA still conducts eligibility examinations only of firms that have already received a WOSB award. Restricting the samples in this way limits SBA’s ability to better understand the eligibility of businesses before they apply for and are awarded contracts, as well as its ability to detect and prevent potential fraud. We recognize that SBA has made some effort to address our recommendation by documenting procedures for conducting annual eligibility examinations of WOSB firms. However, without maintaining reliable information on the results of eligibility examinations, developing procedures for analyzing results, and expanding the sample of businesses to be examined to include those that did not receive contracts, SBA limits the value of its eligibility examinations and its ability to reduce ineligibility among businesses registered to participate in the WOSB program. Leading fraud risk management practices state that federal program managers should design control activities that focus on fraud prevention over detection and response, to the extent possible. The deficiencies in SBA’s oversight of the WOSB program limit SBA’s ability to identify potential fraud risks and develop any additional control activities needed to address these risks. As a result, the program may continue to be exposed to the risk of ineligible businesses receiving set- aside contracts. In addition, in light of these continued deficiencies, the implementation of sole-source authority without addressing the other changes made by the 2015 NDAA could increase program risk. For these reasons, we maintain that our previous recommendation that SBA enhance its WOSB eligibility examination procedures should be addressed. In addition, similar to previous findings from SBA’s Office of Inspector General, our March 2019 report found that about 3.5 percent of contracts using a WOSB set-aside were awarded for ineligible goods or services from April 2011 through June 2018. At that time, SBA was not reviewing contracting data that could identify this problem and inform SBA which agencies making awards may need targeted outreach or training. As a result, we found that SBA could not provide reasonable assurance that WOSB program requirements were being met and that the program was meeting its goals. We recommended that SBA develop a process for periodically reviewing the extent to which WOSB program set-asides are awarded for ineligible goods or services and use the results to address identified issues, such as through targeted outreach or training to agencies making awards under the ineligible codes. In early May 2019, SBA said that it had initiated such efforts. In September 2018, we found that although SBA had adopted criteria and guidance for a risk-based approach to certifying and recertifying firms for the HUBZone program in March 2017, the extent to which it conducted a risk assessment to inform its approach was unclear. In 2015, we found that SBA lacked key controls for its recertification process and recommended that SBA assess the process. In 2009, SBA increased documentation requirements for certification but not recertification (which determines continued program eligibility every 3 years). In March 2017, SBA changed its recertification criteria and guidance to require firms with $1 million or more in HUBZone contract awards to provide documentation to support continuing eligibility. During our work for the September 2018 report, SBA officials stated they had completed a risk assessment of the HUBZone recertification process, but did not provide us with documentation on when they performed the risk assessment, which risks were identified and considered, or what analysis established the $1 million threshold. As of May 2019, SBA had not provided documentation showing that it had performed the risk assessment, but we maintain that an assessment of the recertification process would help inform a risk- based approach to reviewing and verifying information from firms that appear to pose the most risk to the program. In addition, SBA had not provided documentation showing that a technology-based solution designed to address some of the ongoing challenges with the recertification process had been implemented. SBA officials had previously estimated this solution would be available first in spring 2017 and then by the end of calendar year 2017. We also found in our September 2018 report that, based on our review of case files for a nongeneralizable sample of 12 firms in Puerto Rico that received HUBZone certification between March 2017 and March 2018, SBA did not consistently document or follow its policies and procedures for certification reviews: SBA did not have complete documentation in nine of 12 cases. SBA officials described alternative procedures they used to determine firms’ eligibility, but SBA had not updated its internal policy manuals to reflect these procedures, and analysts did not document use of such procedures in the files we reviewed. As a result, SBA did not have reasonable assurance that firms met HUBZone criteria. In four of 12 cases, SBA did not follow its policy to conduct three levels of review (by an analyst, a senior analyst, and the program director or deputy) when determining whether to approve or deny a firm. It was unclear to what extent SBA reviewed staff compliance with certification and recertification review procedures. SBA provided an assurance letter stating that it evaluated the Office of HUBZone’s internal controls and concluded the controls were effective, but the letter did not specify what steps SBA took for the evaluation. We recommended that SBA (1) update its internal policy manuals for certification and recertification reviews to reflect existing policies and procedures not currently in written guidance and (2) conduct and document reviews of staff compliance with procedures associated with HUBZone certification and recertification. In response to our report, SBA said that it planned to update its internal policies on certification and recertification by issuing a procedural notice and to begin reviewing and documenting staff compliance with the updated procedures outlined in the notice. However, as of May 2019, SBA had not provided documentation showing that it had completed these planned actions. In September 2018, we found that for fiscal year 2017, SBA revised the methodology for its Small Business Procurement Scorecard, which is used to assess federal agencies’ progress toward small business procurement goals. SBA made revisions to address requirements specified in the National Defense Authorization Act for Fiscal Year 2016. SBA (1) reduced the share of the total scorecard grade devoted to prime contracting achievement, which is the dollar amount of contracts awarded directly to small businesses, and (2) added an element calculating changes in the number of small businesses receiving prime contracts. SBA made two additional revisions—with input from other agencies’ representatives—to increase the share of subcontracting achievement results and the share of the peer review of required activities designed to facilitate small business procurement. In July 2018, officials said they had begun developing a plan to evaluate the effects of the revised scorecard methodology but did not provide a draft plan. Conducting a well-designed and comprehensive evaluation could aid SBA in determining whether the scorecard is an effective tool for helping to achieve the agency’s strategic goals. In our September 2018 report, we also found that the published fiscal year 2017 scorecards originally contained errors, including an incorrect grade and numeric score for one agency, and SBA does not have a process to ensure that scorecard results are published accurately. Although SBA later corrected the errors, it did not initially document that scorecards had been changed, which is inconsistent with SBA’s policy on information quality. SBA officials said that errors occurred in the process of formatting scorecards for publication. Errors in the published scorecards—and the initial lack of disclosure about corrections—weaken data reliability and may undermine confidence in scorecard data. We recommended that SBA (1) design and implement a comprehensive evaluation to assess scorecard revisions and (2) institute a process for reviewing scorecards for accuracy prior to publication and a mechanism for disclosing corrected information. Since our report, SBA has proposed a two-phase program evaluation of the scorecard. SBA officials said that they plan for phase one to include a report to Congress on the impact of the small business procurement goal program for Chief Financial Officers Act agencies and to provide a recommendation on continuing, modifying, expanding, or terminating the scorecard program. SBA plans to provide the phase one report in September 2019. In phase two, SBA plans to conduct a program evaluation that investigates the effectiveness of the small business contracting scorecard on federal agency small business contracting goal achievement. SBA has not provided a time frame for phase two. With respect to the second recommendation, SBA officials said that SBA has developed a procedure that includes a prepublication review process for procurement scorecards. The officials said the procedure identifies responsibilities, provides for an independent peer review, and includes supervisory review. Officials said the procedure also includes measures for post-publication review and corrections. We will review supporting documentation for this new procedure to assess whether this recommendation can be closed as implemented. Chairman Rubio, Ranking Member Cardin, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact William Shear, Director, Financial Markets and Community Investment at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Andrew Pauline (Assistant Director), Paige Smith (Assistant Director), Winnie Tsen (Assistant Director), and Jennifer Schwartz. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Federal agencies conduct a variety of procurements that are reserved for small business participation through small business set-asides. These set-asides can be for small businesses in general, or they can be specific to small businesses that meet additional eligibility requirements in programs such as those for WOSB or HUBZone. SBA administers both the WOSB and HUBZone programs. SBA also produces an annual Small Business Procurement Scorecard to measure how much contracted spending federal agencies allocate to small businesses and whether the federal government is meeting its goals for awarding contracts to small businesses. GAO issued three reports between September 2018 and March 2019 on SBA contracting programs (see GAO-18-666 , GAO-18-672 , and GAO-19-168). This testimony is primarily based on these three reports and discusses prior GAO findings and SBA's progress on implementing GAO's recommendations on (1) the WOSB program, (2) the HUBZone program, and (3) SBA's procurement scorecard. To update the status of prior recommendations, GAO reviewed updates from SBA and interviewed officials. The Small Business Administration (SBA) has not fully implemented GAO's prior recommendations to address oversight deficiencies in the Women-Owned Small Business (WOSB) and Historically Underutilized Business Zone (HUBZone) programs and to improve evaluation of its procurement scorecard. GAO maintains that its recommendations should be addressed. Women-Owned Small Business Program. In its March 2019 report, GAO found that SBA had not addressed WOSB program oversight deficiencies identified in GAO's 2014 report (GAO-15-54). For example, GAO had found that SBA did not have procedures related to reviewing the performance of the four third-party certifers—private entities approved by SBA to certify the eligibility of WOSB firms—as well as information the certifiiers submitted to SBA. GAO recommended that SBA establish procedures to assess the performance of the certifiers and the information they submitted. While SBA conducted a compliance review of the certifiers in 2016, SBA said in June 2018 that it had no plans to conduct further compliance reviews until a final rule implementing a new certification process was completed. SBA officials said that they expected the rule to be implemented by June 2021. By waiting to improve its oversight of the WOSB program, SBA cannot provide reasonable assurance that certifiers are complying with program requirements and cannot improve its efforts to identify ineligible firms or potential fraud. HUBZone Program. In September 2018, GAO reported that it had reviewed case files for a nongeneralizable sample of 12 firms in Puerto Rico that received HUBZone certification between March 2017 and March 2018 and found that SBA did not consistently document or follow its policies and procedures for certification reviews. For example, SBA did not have complete documentation in nine of 12 cases and did not follow its policy to conduct three levels of review when determining whether to approve or deny a firm in four of 12 cases. As a result, SBA did not have reasonable assurance that firms meet HUBZone criteria. SBA said that it planned to implement GAO's recommendations that SBA (1) update internal policy manuals for certification and recertification and (2) conduct and document reviews of staff compliance with relevant procedures. However, as of May 2019, SBA had not provided documentation showing that it had completed these planned actions. Small Business Procurement Scorecard. For fiscal year 2017, SBA revised the methodology for its Small Business Procurement Scorecard, which assesses the efforts of federal agencies to support contracting with small businesses. For example, one revision reduced the share of the total scorecard grade devoted to prime contracting achievement (the dollar amount of contracts awarded directly to small businesses). GAO recommended in September 2018 that SBA design and implement a comprehensive evaluation to assess the scorecard revisions. Since that report was issued, SBA has proposed but not yet implemented a two-phase evaluation of the scorecard to include an evaluation of the scorecard's effect on federal agencies achieving small business contracting goals. SBA said that it expects to complete phase one by September 2019 and has not provided a time frame for phase two.", "document_type": "gao"}
{"report": "In June 2018, DOD’s Chief Management Officer established the Community Services Reform Task Force to perform a business case analysis to determine whether consolidating the defense resale organizations would result in efficiencies. The task force conducted its work from July 2018 through November 2018, during which it collected financial and other data from the four resale organizations and conducted workshops with subject matter experts from the resale organizations. In November 2018, the military departments were given an opportunity to review the business case analysis, provide comments, and indicate whether they concurred with the analysis. In its business case analysis, the task force recommended consolidating the four defense resale organizations into a single organization. The task force stated that consolidation would eliminate duplication that currently exists across the resale organizations and increase the competitiveness, or financial viability, of defense resale, which has seen sales declines in recent years. Specifically, the task force recommended that a single chief executive officer or director be responsible for leading the organization and report to a single board of directors. The task force also recommended that separate leadership positions for commissary operations and exchange operations be established, and that a chief administrative officer manage the business functions that are common to the current resale organizations, such as information technology (IT), human resources, marketing, and finances. Figure 1 shows the task force’s recommended organizational chart for the consolidated resale organizations. The task force estimated that the time frame for consolidating the four defense resale organizations would be 5 years for implementation, and that consolidation would result in “net savings” (i.e., estimated savings minus estimated costs) ranging from about $690 million to $1.3 billion during the first 5 years, followed by annual net savings of approximately $390 million to $670 million every year thereafter. Specifically: Estimated savings: The task force estimated that consolidating the four defense resale organizations would result in savings in three areas: (1) reduction of the cost of goods sold in the commissaries and the exchanges; (2) reduction of the cost of goods and services that are not sold but are necessary for operating stores (e.g., plastic shopping bags and custodial services); and (3) reduction of payroll costs by eliminating redundant personnel. According to the task force, consolidation would result in estimated savings of $1.4 billion to $2.1 billion over the first 5 years, followed by annual savings of $470 million to $750 million. Estimated costs: The task force estimated that consolidating the defense resale organizations would result in costs from four areas: (1) development of new, common IT systems; (2) severance pay for separating employees and retention bonuses to incentivize employees to remain; (3) operation of a transformation management office, supported by private contractors, to implement the consolidation; and (4) costs to convert DeCA to a non-appropriated fund organization. According to the task force, consolidation would result in estimated costs of $700 million to $810 million over the first 5 years, followed by annual costs of $80 million. DOD’s Chief Management Officer in March 2019 and the Deputy Secretary of Defense in August 2019 both approved the results of the business case analysis and directed that plans be made for consolidation, pending congressional action to remove the statutory prohibition on consolidating the commissary and exchange systems. The task force may have overestimated the expected savings from reducing the “cost of goods sold” (i.e., the cost of purchasing products that are resold in commissaries or exchanges) and underestimated the expected costs from IT consolidation and headquarters relocation. The task force estimated that most of the savings (i.e., about 70 percent annually) to be achieved from consolidating the four defense resale organizations would result from reducing the cost of goods sold. According to the task force’s business case analysis, retailers often pay different costs for identical products, and mergers are an opportunity for retailers to compare costs across a larger combined organization and make decisions that maximize savings. In the case of a consolidated defense resale organization, the task force stated in its business case analysis that savings could be achieved by implementing what the task force called category management reforms and by obtaining the lowest cost for identical products sold by both commissaries and exchanges. Task force officials added that one board of directors and one chief executive officer overseeing the consolidated resale organization would be more likely to achieve savings than the current, individual boards of directors and chief executive officers of the resale organizations. GAO-18-151SP. DeCA and one or more exchange organization, or Two or more of the exchange organizations (and not DeCA). In addition to overlap in identical products sold, task force officials told us that savings from category management reforms are dependent, in part, on the amount of overlap in vendors that sell products to the resale organizations. Specifically, task force officials stated that there are opportunities to reduce cost of goods sold through negotiations with vendors that sell items to both DeCA and the exchange organizations. For example, task force officials stated that the consolidated organization could negotiate better prices with a vendor that sells family-size items to DeCA and single-size items to the exchange organizations, even though those items are not identical. Industry Benchmarks The industry benchmarks used by the task force are based on proprietary data gathered and owned by Boston Consulting Group based on its experience working with mergers and category management reforms in the private sector retail industry. These benchmarks were presented as a percentage of cost of goods sold; specifically, the task force estimated that savings from obtaining the lowest cost for identical items were from 1 to 1.5 percent of the cost of goods sold, and savings from category management efforts were from 2.5 to 4 percent of the cost of goods sold. We did not review and evaluate the underlying data that were used to develop the proprietary benchmarks. Based on this information, the task force calculated the estimated savings that would result from reducing the cost of goods sold by multiplying the total cost of goods sold for all four resale organizations in fiscal year 2017 ($9.5 billion) by industry benchmarks developed by Boston Consulting Group (see sidebar for more information on these benchmarks). This calculation showed an estimated annual savings of $329 million to $517 million from reducing the cost of goods sold. However, additional information from the task force suggests this savings estimate may be overstated because there is limited overlap in the products DeCA sells (i.e., groceries and household goods) and the products the exchange organizations sell (i.e., goods and services similar to retail stores). According to the task force, about $2.2 billion of DeCA’s cost of goods sold in fiscal year 2017 were for products also sold by at least one of the exchange organizations, which is equivalent to about 23 percent of the total cost of goods sold for the four resale organizations. This differs from the data provided in the business case analysis, which stated that 62 percent of the total cost of goods sold was for identical products sold by two or more resale organizations; however, that figure also includes products sold by two or more exchange organizations and not DeCA (we further discuss product overlap among the exchange organizations below). Given the more limited product overlap between DeCA and the exchange organizations, it is unclear whether using the total cost of goods sold for all four resale organizations as the basis for estimating savings was appropriate. Additionally, the business case analysis did not fully identify the amount of vendor overlap that exists between DeCA and the three exchange organizations, but the data that were provided in the business case analysis suggest that limited vendor overlap exists. Specifically, the business case analysis provided data for 10 vendors that sell to the defense resale organizations, but those vendors represent less than 20 percent of the cost of goods sold to DeCA and the exchange organizations in fiscal year 2017. Further, only 5 of the 10 vendors identified in the business case analysis sold goods to both DeCA and the exchange organizations, and their cost of goods sold accounted for about 10 percent ($972 million) of the total cost of goods sold for the four resale organizations ($9.5 billion). Based on these data, the extent of vendor overlap between DeCA and the exchange organizations—and, as a result, how much can be saved through category management reforms by consolidating DeCA and the exchange organizations—is unclear. Although the task force stressed the importance of a conservative estimate in both its business case analysis and in meetings with us, our assessment of the assumptions and methodology for estimating savings from the cost of goods sold found that a more conservative approach could have been used to better ensure estimated savings were not overstated. For example, one method could have been to multiply the benchmarks by the cost of goods sold for just the three exchange organizations (about $5.5 billion in fiscal year 2017, per the task force), as data provided by the task force indicate that about 67 percent of the cost of goods sold for the exchange organizations in fiscal year 2017 were for identical products sold by at least two exchange organizations. Another method, which task force officials suggested after we shared our concerns about their methodology, could have been to multiply the benchmarks by the cost of goods sold for the exchange organizations, plus the portion of DeCA’s cost of goods sold that overlaps with at least one exchange organization (about $2.2 billion in fiscal year 2017, per the task force). Either method would be more conservative than the one adopted in the business case analysis and would yield a savings estimate that is about 20 to 40 percent lower, but would be more consistent with the task force’s assertion that consolidation savings are dependent on the amount of overlap among the merging organizations. DOD policy states that an economic analysis should base its analysis of benefits on facts and data whenever possible. Additionally, our Assessment Methodology for Economic Analysis states that an economic analysis should examine the effects of an action by considering relevant alternatives and justifying what the world would be like under each alternative; describe and justify the analytical choices, assumptions, and data used; and assess how plausible adjustments to each important analytical choice and assumption affect the estimates of savings. Ensuring that the estimates for cost of goods savings are accurate is particularly important, as they account for approximately 70 percent of the task force’s overall savings estimate from consolidation. However, the task force did not fully identify and analyze in its business case analysis how many identical products are sold by both DeCA and the exchange organizations or how many vendors sell products to both DeCA and the exchange organizations. According to task force officials, they did not provide data on product overlap between DeCA and the exchange organizations because it would not change their savings methodology or estimate, and they did not provide more information on vendor overlap because of the proprietary nature of that data. However, the amount of product and vendor overlap that exists across the four resale organizations will have a direct effect on the amount of savings to be achieved from consolidation, as acknowledged by the task force. Without the task force reassessing the approach it used to estimate savings from the cost of goods sold and, if necessary, making adjustments to those estimates, decision makers in DOD and Congress may lack confidence in the reliability of the task force’s savings estimates in the business case analysis and will not have complete information as they consider defense resale consolidation. Based on our analysis of the business case analysis, we found that DOD’s task force may have underestimated the expected costs of consolidating the four defense resale organizations in two areas: (1) the development of new, common IT systems and (2) the location of a new headquarters for the consolidated organization. The task force estimated in its business case analysis that most of the costs (i.e., about 50 percent annually) of consolidating the four defense resale organizations will result from developing new, common IT systems to support the consolidated organization. In the business case analysis, the task force stated that it worked with the four resale organizations to calculate a cost estimate of $292 million to $352 million for developing five types of IT systems that are needed for the consolidated organization: merchandising, store inventory management, financial management and general ledger, transportation and logistics, and ecommerce. According to the business case analysis, the task force’s cost estimates for developing new, common IT systems for the consolidated organization were to be based on data provided by the resale organizations on recent or projected costs for replacing similar systems or performing major upgrades to existing systems, when available. For example, the task force’s estimate for the merchandising system was approximately $115 million, which the business case analysis stated is based on a $35 million estimate provided by AAFES, a $23.5 million estimate provided by NEXCOM, a $15 million estimate provided by MCCS, and a $41 million estimate provided by DeCA. However, the task force’s cost estimate for IT consolidation may be understated because it is based, in part, on less expensive minor IT system upgrades and partial replacements, according to the resale organizations. Based on our analysis of information provided to us by the resale organizations, about $140 million (about 40 percent) of the overall IT cost estimate was based on what the resale organizations described as minor upgrades or partial replacements. Specifically, while MCCS confirmed the cost estimates attributed to them in the business case analysis were for total IT system replacement costs, the other three resale organizations—AAFES, NEXCOM, and DeCA— disagreed with the task force’s characterization that all the data used to calculate IT system estimates represented costs for replacements or major upgrades. AAFES told us that the cost estimates cited in the business case analysis for its merchandising, financial management and general ledger, and transportation and logistics systems reflected minor upgrades of specific modules within the overall systems, and the cost to replace or upgrade the entire system would be significantly higher. NEXCOM stated that the cost estimates for upgrading its merchandising, store inventory management, and financial management and general ledger systems were for minor upgrades, not replacements or major upgrades, as stated by the task force. DeCA told us that the estimate for replacing its store inventory management system only represented 1 year of costs, even though DeCA plans to incur replacement costs through at least 2022. According to the task force, the task force and the resale organizations agreed on the methodology for estimating IT costs, and the subject matter experts from the resale organizations provided the cost data used in the business case analysis. However, based on information provided by the resale organizations, it appears that the task force may not have always based its cost estimate on replacement or major upgrade costs— consistent with the key assumption that new IT systems would be developed for the consolidated resale organization—but, rather, used minor upgrade or partial replacement costs in some cases. Specifically, task force officials told us they believed the estimates provided by the resale organizations were too high to be minor upgrades or partial replacements, based on their understanding of IT requirements for resale operations. Further, task force officials stated that their overall IT cost estimate was likely overstated, not understated. For example, they stated that their estimate is higher than what is typically spent for a private sector consolidation of similar size. However, the task force stated that it did not use private sector IT cost estimates in its business case analysis because it determined that public sector IT costs would likely be higher than private sector IT costs. Additionally, task force officials told us that some planned spending on existing IT systems by the four resale organizations would not be necessary as a result of consolidation. However, the business case analysis does not quantify how much future spending could be reduced or factor those reductions into the IT cost estimate. According to the GAO Cost Estimating and Assessment Guide, cost estimates are developed based on assumptions that are defined to establish the baseline conditions the estimate will be built from. Additionally, our Assessment Methodology for Economic Analysis states that an economic analysis should define an appropriate baseline that represents the best assessment of what the world would be like under that alternative. Thus, estimating costs that reflect the baseline conditions is a key step in developing a sound cost estimate. Additionally, we have previously reported that federal IT investments frequently fail or incur cost overruns and schedule slippages. As such, high-quality data are imperative for ensuring proper management and oversight of IT investments. The task force’s IT cost estimate is particularly important, as it represents about 50 percent of the total estimated costs for defense resale consolidation. Until the task force consults with the resale organizations to reassess the methodology for estimating IT costs, decision makers in DOD and Congress may not have a reliable and complete understanding of the estimated costs for the implementation of new, common IT systems, which is information DOD and Congress need as they consider defense resale consolidation. According to the task force, there would be costs associated with relocating AAFES, NEXCOM, MCCS, and DeCA to a new headquarters location, to include relocating existing personnel, hiring new personnel, and obtaining real estate. Although no relocation options were presented in the business case analysis, task force officials told us there are multiple options for where to locate the headquarters of a consolidated resale organization. One option cited by the task force would be to create a new headquarters in the Washington, DC, area, which would be the most expensive option, as it would likely involve acquiring new real estate and hiring personnel in a high-cost region. Another option cited by the task force would be to locate all exchange operations and staff at the existing AAFES headquarters in Dallas, TX, and maintain commissary operations and staff at the existing DeCA headquarters at Fort Lee, VA. This option would likely be less expensive, as personnel and available real estate are already present at both locations. In January 2020, task force officials also told us that an even less expensive option they might consider is maintaining commissary and exchange headquarters staff at their current locations, but having personnel work for the consolidated organization, rather than for DeCA or the exchange organizations. Despite the potential for relocation costs, the task force did not include a range of cost estimates for different relocation options in its business case analysis. According to task force officials, relocation cost estimates were not included because the headquarters location has not been chosen, and costs will vary widely depending on the chosen location. While actual relocation costs will depend on the chosen headquarters location, this fact does not prevent the task force from presenting a range of cost estimates in advance of that decision being made. Task force officials also said that including relocation cost estimates would not have changed the conclusion of the business case analysis. However, without a range of relocation cost estimates, we were unable to assess the effect of relocation costs on the conclusion of the business case analysis. DOD policy states that an economic analysis should quantify the costs associated with each alternative under consideration whenever possible so that they may be included in the economic analysis calculation. Additionally, our Assessment Methodology for Economic Analysis states that an economic analysis should quantify the important costs, where feasible, to inform decision makers about the economic effects of a proposed action. Without developing and providing a range of relocation cost estimates from the least expensive option to the most expensive, decision makers in DOD and Congress will not be fully informed about the costs of consolidation, which is necessary information for deciding whether to consolidate the four defense resale organizations. The military departments officially concurred with the task force’s business case analysis for consolidating the four defense resale organizations. However, the military departments also provided written comments that detailed concerns with fundamental aspects of the business case analysis, to include: the use of proprietary industry benchmarks; estimated savings, costs, and timeline of the consolidation; and the proposed governance structure for the new resale organization. In an April 2019 report to Congress that summarized the business case analysis, DOD stated that the military departments agreed with the consolidation. However, the report did not disclose the military departments’ comments and concerns on the business case analysis, which are relevant as Congress considers defense resale consolidation. In their written comments, the military departments either stated concerns about the consolidation or included critical comments from the exchange organizations—all of which opposed the consolidation. Specifically: The Army concurred with the business case analysis but noted that funding for morale, welfare, and recreation programs must be preserved or increased as a result of the consolidation. In addition, the Army’s comment letter included as an attachment written comments from AAFES, which expressed opposition to the consolidation and detailed concerns with the business case analysis. For example, AAFES stated that the business case analysis relied on unverifiable, proprietary industry benchmarks that overstated the benefits of consolidation, underestimated the costs and time to consolidate, and did not account for recent efforts by the resale organizations to reduce costs by collaborating on a purchasing alliance. The Air Force also concurred with the business case analysis, but noted in its comments that mergers and acquisitions have historically cost more, taken longer, and saved less than originally expected. As a result, the Air Force recommended that a phased implementation plan be followed to guard against financial risk. The Air Force also stated that morale, welfare, and recreation funding currently provided by the exchanges should be maintained while opportunities are examined to reduce the need for appropriated funding. The Navy initially non-concurred with the business case analysis in December 2018. In its comment letter, the Navy stated that the task force’s analysis was flawed beyond repair and included comments from NEXCOM and MCCS that also opposed consolidation. For example, the exchanges’ comments stated that the expected cost savings were overstated, that potential inefficiencies from consolidation were not discussed, and that the resale organizations could achieve cost savings through greater collaboration without the need for consolidation. NEXCOM and MCCS also stated concern that the task force’s savings projection relied heavily on unverifiable industry benchmarks. In addition, MCCS expressed concern that consolidation could result in unexpected costs from separating exchange operations from the rest of MCCS operations, which also include the Marine Corps’ morale, welfare, and recreation and family programs. In January 2019, the Navy changed its position to concur subject to several significant comments and clarifications, and attached a letter detailing comments and concerns similar to those it submitted with its original non-concurrence in December 2018. Officials from the resale organizations further articulated their concerns about the business case analysis when they met with us. For example, resale officials told us they are concerned that savings are overstated, that costs are understated, and that the proprietary benchmarks used by the task force are unverifiable or may not be applicable to the public sector. Exchange officials also stated that they are worried about the effect of consolidation on morale, welfare, and recreation funding generated by the exchanges. Specifically, exchange officials are concerned that exchange revenue currently used for morale, welfare, and recreation programs could be used to pay for consolidation expenses or to reduce the amount of appropriated funds allocated to the commissaries. Despite the concerns detailed in the comments from the military departments and resale organizations, DOD did not include them in its April 2019 report to Congress summarizing the results of the business case analysis. The National Defense Authorization Act for Fiscal Year 2019 required DOD to include in its report the recommendations of the Secretaries of the military departments regarding the plan to consolidate the defense resale organizations. When we asked the task force why DOD did not provide Congress with the comments and concerns cited by the military departments and the resale organizations, officials stated that they were advised by DOD’s Office of General Counsel not to include the comments because they contained information that may have disclosed DOD’s deliberative process. Task force officials also stated that the savings, cost, and timeline estimates in the business case analysis were conservative, and that the proprietary industry benchmarks are based on years of experience by Boston Consulting Group and similar to those cited by prior studies. Regarding the purchasing alliance formed by the resale organizations to reduce their cost of goods sold, task force officials stated they do not believe such efforts to reduce costs will be sustained without a single chief executive officer and board of directors to ensure those efforts continued, as recommended in the business case analysis. Finally, the task force stated in the business case analysis that any savings achieved from consolidation could be used to increase morale, welfare, and recreation funding or reduce appropriations used to fund DeCA, and that decisions on how to allocate savings will be made by the proposed board of directors. According to task force officials, some of the concerns articulated by the military departments and the exchanges could be motivated by a general opposition to consolidation. However, without a more complete reporting of the military departments’ perspectives on consolidation and the task force’s response to those comments, Congress may be unaware of the views various organizations within DOD have regarding the business case analysis, which is relevant information as Congress considers defense resale consolidation. Moreover, fully reporting the comments and concerns could strengthen trust and collaboration among the task force, military departments, and resale organizations on any future resale reforms. Four defense resale organizations currently operate about 240 commissaries (operated by DeCA) and 2,500 exchange facilities (operated by AAFES, NEXCOM, and MCCS) worldwide to provide reduced-priced groceries and retail goods and services to DOD servicemembers, their families, and retirees. DeCA operations are funded in part by appropriations, which have totaled approximately $1.3 billion in recent years. By law the commissary and exchange organizations must be operated separately. In November 2018, a DOD task force completed a business case analysis and concluded that consolidating the four defense resale organizations into a single organization would result in several hundred million dollars in annual cost savings. However, we found that the task force’s projected savings from reducing the cost of goods sold may be overestimated, and that projected costs for IT development and headquarters relocation may be underestimated. Further, while the military departments concurred with the task force’s recommendation to consolidate, DOD did not fully share their comments and concerns about the business case analysis with Congress. DOD’s proposed consolidation will cost several hundred million dollars, take years to implement, and involve multiple DOD organizations. Given the cost and complexity of the proposed defense resale consolidation, DOD can ensure that Congress has the reliable information it needs to consider consolidation by reviewing and updating savings and cost estimates and sharing comments and concerns from the military departments. We are making the following four recommendations to DOD. The Secretary of Defense should ensure that the DOD Chief Management Officer direct the task force to reassess its approach to estimating savings from cost of goods sold—to include reassessing its use of the cost of goods sold for all four defense resale organizations rather than, for example, just for the three exchange organizations—and make any necessary adjustments to its savings estimates for consolidation and provide that updated information to Congress. (Recommendation 1) The Secretary of Defense should ensure that the DOD Chief Management Officer direct the task force, in consultation with the resale organizations, to reassess its methodology for estimating IT costs of consolidation, and make any necessary adjustments to its range of IT cost estimates and provide that updated information to Congress. (Recommendation 2) The Secretary of Defense should ensure that the DOD Chief Management Officer direct the task force to develop a range of cost estimates for relocating the defense resale organizations, and adjust its range of cost estimates for consolidation and provide that updated information to Congress. (Recommendation 3) The Secretary of Defense should ensure that the DOD Chief Management Officer provide additional written information to Congress on the comments and concerns from the military departments and resale organizations on the task force’s November 2018 business case analysis, as well as the task force’s response to those comments and concerns. (Recommendation 4) We provided a draft of this product to DOD for review and comment. In its comments, reproduced in appendix III and summarized below, DOD concurred with the first three recommendations and did not concur with the fourth recommendation. DOD also provided technical comments, which we incorporated as appropriate. DOD stated in its letter that it continues to firmly believe that consolidation of above-store operations of DeCA and the military exchanges is the right path forward and that it intends to move forward with this effort. DOD also requested that we consider the first three recommendations as implemented, based on information provided in the letter and as detailed below. Regarding the first recommendation for the task force to reassess its approach to estimating savings from cost of goods sold—to include reassessing its use of cost of goods sold by all four defense resale organizations rather than, for example, just the three exchange organizations—and make any necessary adjustments to its savings estimates for consolidation and provide that updated information to Congress, DOD stated that it had reassessed its approach and found that there is significant overlap and, therefore, savings opportunity in products sold by DeCA and the exchanges. DOD also provided revised savings estimates that exclude DeCA’s cost of goods sold from its methodology that show net savings ranging from $309 million to $739 million in the first 5 years of consolidation, followed by $255 million to $457 million per year thereafter. These figures are about 44 percent to 55 percent lower than the business case analysis’s estimate for the first 5 years and about 32 to 35 percent lower per year thereafter. By providing these revised savings estimates, we believe that DOD has addressed the intent of the recommendation. With regard to the second recommendation for the task force to reassess its methodology for estimating IT costs of consolidation, and make any necessary adjustments to its range of IT cost estimates and provide that updated information to Congress, DOD stated that the task force followed up with AAFES, NEXCOM, and DeCA to get an update on the cost estimates these entities expressed concern about to us. However, according to DOD, those resale organizations were unable to provide alternate data to use in place of the numbers in the business case analysis. DOD further stated in its letter that because no alternative data were provided, the department will continue to use the estimate in the business case analysis and will reengage with the resale organizations to develop more detailed IT design plans and make any necessary updates to the IT cost estimates as integration planning moves forward. As DOD develops its more detailed IT design plans and associated cost estimates, we will follow up with the department, including the resale organizations, to determine whether this recommendation has been addressed. In commenting on the third recommendation for the task force to develop a range of cost estimates for relocating the defense resale organizations, and adjust its range of cost estimates for consolidation and provide that updated information to Congress, DOD provided three possible courses of action, along with corresponding cost estimates. These possible courses of action, from least expensive to most expensive, are: (1) maintain operations at all four existing locations (no cost); (2) maintain commissary operations at DeCA headquarters, perform all exchange functions at AAFES headquarters, and close the NEXCOM and MCCS headquarters (one-time costs of $5.5 million and recurring annual costs of $1.3 million); and (3) create a new headquarters to perform all commissary and exchange operations near Washington, D.C. (one-time costs of $19.6 million and recurring annual costs of $19.7 million). DOD stated that consolidation would still result in financial benefits, even if the department chooses the most costly of these courses of action. By providing these cost estimates, we believe that DOD has addressed the intent of the recommendation. While we have determined that DOD has met the intent of the first and third recommendations, we also note that, in its comment letter, the department questioned some aspects of our analysis and conclusions regarding the first three recommendations. We stand by our analysis and conclusions and offer the following response: DOD stated in its comment letter that modifying the business case analysis’s approach to cost of goods savings would result in an incorrect use of benchmarks and go against industry best practice. For example, DOD stated that estimating savings by using the cost of goods sold for just the three exchange organizations, or for the three exchange organizations plus a portion of DeCA, would be flawed. However, as noted in our report, the latter method was recommended to us by task force officials when we raised concerns about the accuracy of the task force’s savings estimates in the business case analysis. In addition, multiplying the benchmarks by the cost of goods sold for the exchange organizations, as opposed to for all four resale organizations, would be more consistent with the assertion in the business case analysis that consolidation savings are dependent on the overlap among the merging organizations. DOD questioned the accuracy of some of our figures in the report by providing different data on product and vendor overlap between DeCA and the exchange organizations. However, this information was not included in the task force’s business case analysis or offered to us during the course of our audit. In addition, when we asked for supporting documentation that would allow us to validate the new figures, DOD did not provide any. DOD stated in its comments that excluding all or including only a portion of DeCA’s cost of goods sold implies that there is no opportunity to achieve savings between DeCA and the exchanges. Our report does not make this assertion, but rather offers a methodology that would result in a more conservative savings estimate, consistent with the data presented in the business case analysis and provided by task force officials, to better ensure that estimated savings were not overstated. As noted above, DOD did not concur with the fourth recommendation for the department’s Chief Management Officer to provide additional written information to Congress on the comments and concerns from the military departments and resale organizations on the task force’s November 2018 business case analysis, as well as on the task force’s response to those comments and concerns. DOD stated in its written response to our report that the department considered all the comments submitted in its decision-making process and that all of the military department secretariats agreed with above-store consolidation, despite their comments on the business case analysis. DOD further stated that the military department comments regarding the business case analysis were shared with congressional committee professional staff, and DOD suggested in its letter that this recommendation be closed. However, DOD’s written response did not provide information on which comments were shared, whether those comments were communicated in writing or orally, or which committee or committees received information on the comments. In their written comments on the business case analysis, the military departments detailed concerns with fundamental aspects of the analysis, to include: the use of proprietary industry benchmarks; estimated savings, costs, and timeline of consolidation; and the proposed governance structure for the new resale organization. We continue to believe that implementing this recommendation would help ensure that Congress has the full information it needs as it considers defense resale consolidation and would also help strengthen trust and collaboration among the various DOD stakeholders involved in defense resale, particularly given their role in any consolidation, should one occur. We will follow up with DOD as part of our regular recommendation follow- up process. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; and the Secretaries of the Army, Air Force, and Navy. In addition, the report is available at no charge on our website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2775 or FieldE1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. The National Defense Authorization Act for Fiscal Year 2016 mandated that the Department of Defense (DOD) report on DOD’s plan to achieve budget neutrality for commissaries and exchanges—which DOD interpreted as ending the use of appropriated funding—and included a provision for us to assess DOD’s report. In November 2016, we found that DOD’s May 2016 report did not provide a plan for achieving budget neutrality. DOD reported that it would not be able to eliminate fully the use of appropriated funds for defense resale, but the department did not provide detailed information supporting that conclusion. Instead, the report stated that DOD expected to achieve $2 billion in cost savings over a 5-year period from fiscal year 2017 through fiscal year 2021. However, we found that the report did not include any assumptions, methodology, or specific time frames related to initiatives that would lead to these savings. We recommended that DOD provide information to Congress to support its conclusion about budget neutrality and develop a plan for achieving reductions to defense resale appropriations. DOD concurred with our recommendations, but as of February 2020 had not addressed them. In March 2017, we reported on DOD’s commissary operations, including the extent to which the Defense Commissary Agency (DeCA) had assurance that it was maintaining the desired savings rate for its customers. DeCA’s desired savings rate—which at the time of our March 2017 report was 30 percent and is now 23.7 percent—shows how much a customer can expect to save on grocery purchases at a commissary in comparison to purchases at other local grocery stores. We found that DeCA lacked reasonable assurance that it was maintaining its desired savings rate for commissary customers because of weaknesses in its methodology for calculating the savings rate. For example, the methodology did not use a random sample of overseas commissaries or account for seasonal and geographic variations in item prices. We also found that DeCA’s business model departed from practices generally employed by commercial grocery stores. For example, DeCA did not assess the contribution of the sale of each product to a given store’s total sales in determining which products to sell, and it had not conducted cost- benefit analyses for its use of stocking and custodial service contracts or product distribution options across all commissaries. We recommended that DOD (1) address limitations identified in its savings rate methodology; (2) develop a plan with objectives, goals, and time frames to improve efficiency in product management; and (3) conduct comprehensive cost-benefit analyses for service contracts and distribution options. As of February 2020, DOD had addressed the first two recommendations but had not addressed the third recommendation. We assessed DOD’s business case analysis on consolidating the four defense resale organizations against the five key elements of an economic analysis, as described in our Assessment Methodology for Economic Analysis (see table 1). Elizabeth A. Field, (202) 512-2775 or fielde1@gao.gov In addition to the contact named above, Suzanne Perkins (Assistant Director), Geoffrey Peck (Analyst-in-Charge), Pedro Almoguera, Noah Gerber, Mae Jones, Matthew Kienzle, Amie Lesser, Felicia Lopez, and Jeanne Sung made key contributions to this report.", "summary": "DOD operates about 240 commissaries and 2,500 exchanges that sell groceries and retail goods and services to servicemembers, their families, and retirees. Commissaries and exchanges are operated by four resale organizations, and in November 2018 a DOD task force completed a business case analysis on consolidating those organizations. The National Defense Authorization Act for Fiscal Year 2020 included a provision for GAO to review DOD's business case analysis. This report evaluates the extent to which (1) DOD's business case analysis for consolidating the four resale organizations provided reliable savings and cost estimates and (2) the military departments concurred with the business case analysis and DOD shared their accompanying comments with Congress. GAO evaluated the business case analysis against DOD- and GAO-identified key elements of economic analyses; reviewed comments on the business case analysis; and interviewed DOD officials. A Department of Defense (DOD) task force's business case analysis for consolidating the defense resale organizations—the Defense Commissary Agency (DeCA), the Army and Air Force Exchange Service, the Navy Exchange Service Command, and Marine Corps Community Services—may not provide reliable savings and cost estimates. These organizations sell groceries and retail goods to servicemembers, their families, and retirees. The task force recommended consolidating the four resale organizations into a single organization, estimating “net savings” (i.e., savings minus costs) of about $690 million to $1.3 billion during the first 5 years. However, the task force may have overestimated savings and underestimated costs. Savings from reducing the cost of goods sold. The task force estimated that DOD would save several hundred million dollars annually by reducing the cost of purchasing goods that are resold in stores. Specifically, the task force multiplied the fiscal year 2017 total cost of goods sold for all four resale organizations by industry benchmarks, reasoning that mergers lead to more savings when merging organizations sell a high amount of identical products. However, task force data show that DeCA and the exchange organizations have limited identical products; the overlap between DeCA products and those of at least one exchange organization amounts to less than one-third of the total cost of goods sold. Thus, multiplying the benchmarks by the total cost of goods sold for all four organizations may not have been appropriate. Information technology (IT) costs. The task force estimated the costs of developing new, common IT systems to operate a consolidated resale organization to be between $326 million and $401 million, about 50 percent of estimated consolidation costs. The task force stated that it based IT cost estimates on data resale organizations provided for major upgrades or system replacements. But GAO found that about 40 percent of the IT cost estimate was based on minor upgrades or partial replacements, not major upgrades or system replacements. Thus, the estimate may be understated. Headquarters relocation costs. According to the task force, there will be costs if DOD decides to relocate the four defense resale organizations to a new headquarters location. However, the task force did not include cost estimates for relocation in its business case analysis. According to federal law, the operation of the commissary and exchange systems may not be consolidated unless authorized by Congress. Until the task force reassesses and updates, as necessary, its savings and costs estimates, DOD and Congress will not have reliable information to consider resale consolidation. The military departments officially concurred with the business case analysis, but provided written comments detailing fundamental concerns with the analysis, such as the use of proprietary industry benchmarks and the estimated savings and costs. In April 2019, DOD reported to Congress that the military departments agreed with consolidation, but did not disclose the accompanying comments. Without more complete reporting of those comments, Congress has limited visibility of the views of the organizations involved in a potential consolidation. GAO is making four recommendations, including that DOD reassess and update as necessary its estimates for consolidation savings and costs, and provide additional information to Congress on the military departments' comments on the November 2018 business case analysis. DOD concurred with three recommendations and provided updated estimates. DOD did not concur with the last recommendation. GAO continues to believe providing such information is beneficial, as discussed in the report.", "document_type": "gao"}
{"report": "DOT is made up of nine modal administrations and the Office of the Secretary of Transportation (OST), each of which has its own mission, primarily focused on enhancing mobility and safety. Among other activities, modal administrations oversee financing and grant funding programs specific to their modes (e.g., roads, transit, rail). OST oversees the formulation of national transportation policy and promotes intermodal transportation. In this latter role, OST administers programs that provide grants to projects that can represent multiple transportation modes: roads, bridges, transit, rail, or ports. In fall 2015, DOT created the Build America Transportation Investment Center within OST—a predecessor to the Bureau. This new center was created to be a single point of contact and coordination for project sponsors seeking to apply for finance programs and explore innovative financing, in recognition of the fact that sponsors can face difficulties navigating multiple modal administrations to apply for funding or financing for a single project. In 2015, DOT was required by law to establish a finance bureau to align, coordinate, and consolidate certain surface transportation funding and financing programs. The Bureau—located within OST—is led by an Executive Director, who is responsible for managing and overseeing the daily activities, decisions, operations, and personnel of the Bureau. The Executive Director is appointed by the Secretary and then approved by the President. Three financing programs and one grant funding program were moved into the Bureau. Collectively, these programs provide billions of dollars of support to transportation projects across the country, as described below: TIFIA. TIFIA provides direct loans, loan guarantees, and standby lines of credit to surface transportation projects of national or regional significance. Eligible projects include a variety of projects such as highways, intermodal stations, and passenger rail. The fundamental goal of TIFIA is to leverage federal funds by attracting substantial private and other non-federal co-investment, and the legislation creating TIFIA stated that the program can do so by complementing existing resources to fill market gaps. For fiscal year 2018, the FAST Act authorized $285 million in funding to cover the federal government’s cost of providing financing and administering the program. According to DOT, $1 of TIFIA’s budget authority generally allows DOT to provide more than $10 in credit assistance, so $285 million in funding authority could support approximately $2.9 billion in assistance. TIFIA has provided over $31 billion in financing to 79 projects since its creation in 1998. The Federal Highway Administration (FHWA) administered TIFIA before it was moved to the Bureau. RRIF. RRIF provides direct loans and loan guarantees to finance the development of railroad infrastructure, such as rehabilitating passenger equipment and acquiring or rehabilitating track and bridges. Created in 1998, the RRIF program is authorized to provide up to $35 billion in credit assistance, and RRIF dedicates part of this funding to providing vital access to financing for smaller, short-line and regional railroads, which have historically lacked the access to private financing. The RRIF statute permits appropriations of budget authority to be used for the cost of providing financing, but appropriations acts have typically prohibited the use of appropriations for such purposes. This prohibition, however, was not included in the fiscal year 2018 Consolidated Appropriations Act and appropriations were, for the first time, made available to pay the cost of providing financing. RRIF loans totaling over $5 billion have supported 39 projects as of February 2019. The Federal Railroad Administration (FRA) administered RRIF before it was moved to the Bureau. PAB for Highway and Surface Freight Transfer Facilities. PAB provides private-sector developers of certain types of surface transportation projects with access to tax-exempt financing. In contrast to TIFIA and RRIF, where the federal government directly provides loans and other forms of credit assistance, PAB does not directly provide financing but enables a state or city to borrow on behalf of private companies and nonprofits. PAB does, however, impose costs on the federal government through forgone tax revenues. The total amount of PAB for surface transportation is limited by statute to $15 billion, and the Secretary of Transportation allocates this available capacity among qualified projects. As of February 2019, DOT had allocated about $10.3 billion in PAB to 27 projects. A different office within OST previously administered PAB. Infrastructure for Rebuilding America (INFRA). The FAST Act authorized DOT to award $4.5 billion in discretionary grants for nationally significant freight and highway projects for fiscal years 2016 through 2020. In response, DOT developed the INFRA grant funding program. States and local governments are among the eligible entities that may apply for INFRA grants. DOT may fund freight or highway projects that meet statutory requirements, such as reserving at least 25 percent of available funds for rural areas. In June 2018, DOT announced its most recently proposed INFRA grants totaling nearly $1.5 billion for 26 projects. The FAST Act also created the Council on Credit and Finance (Council) to review and make recommendations to the Secretary on applications for DOT’s financing programs, regularly review projects that have received financing, and conduct other duties the Secretary establishes. The Council is mostly comprised of DOT political appointees, including the Deputy Secretary of Transportation, Under Secretary of Transportation for Policy, and Administrators of FRA, FHWA, and the Federal Transit Administration (FTA). The FAST Act outlined specific responsibilities for the Bureau, some of which relate to administering the above programs. The responsibilities include the following, grouped into five broad categories: Administering the application evaluation process for certain Establishing procedures for analyzing and evaluating applications for these programs, as well as for documenting major decisions in the application evaluation process through a decision memorandum or similar mechanism that provides a clear rationale for such decisions Streamlining the approval processes for the above programs Providing assistance to project sponsors seeking funding or financing Making credit assistance programs more accessible Providing technical assistance, upon request, for proposed public- private partnerships and environmental reviews and permitting, among other areas Promoting innovative-financing best practices: Developing and monitoring best practices for state authorities and practices, standard contracts, and analytical tools Improving environmental reviews and permitting: Serving as DOT’s liaison on the Council on Environmental Quality Coordinating efforts to improve the efficiency and effectiveness of the environmental review and permitting process Identifying, developing, and tracking metrics for permit reviews and decisions Sharing information on procurement costs and risks: Developing procurement benchmarks for projects receiving assistance under the above programs, and collecting and publishing information on procurement benchmarks; to the extent possible, the benchmarks should establish maximum thresholds for cost increases and schedule delays, establish uniform ways to measure these changes, and be tailored to different types of project procurement Developing guidance to require and publish value for money and after-action reports findings for public-private partnerships seeking assistance from the Bureau programs The conference report accompanying the FAST Act noted that the Bureau would serve as a one-stop shop for states and local governments, and to serve in this capacity, the report highlights the Bureau’s role to work with individual project sponsors as the Bureau administers financing programs, as well as its broader role to help reduce costs and uncertainty with environmental reviews and permitting and procurement. The FAST Act also gave the Secretary of Transportation authority to consolidate or eliminate different offices within DOT as it creates the Bureau. DOT established an organizational structure for the Bureau and created a consolidated process for it to use when working with sponsors to evaluate applications for financing programs and provide assistance. Creating this process helped the Bureau make progress on two of its FAST Act responsibilities, and overall, DOT’s initial steps were important actions that allowed it to open and operate the Bureau. Since the Bureau was established in 2016, it has made more limited progress on its other responsibilities, including promoting innovative-financing best practices for certain types of projects. Although we recognize that it is a relatively new office that in many ways remains a work in progress, the Bureau lacks a plan to guide its ongoing and future efforts and has not established performance indicators to measure its outcomes and assess progress. DOT designed and established the Bureau in the year after the FAST Act’s enactment. DOT established internal committees and hired a consultant to produce an initial implementation plan to establish the Bureau. To create this plan, the consultant analyzed existing staffing and processes, interviewed internal and external stakeholders, and examined organizational structures at public and private sector entities, among other things. DOT prioritized several areas for this initial work, including consolidating existing processes for evaluating applications for finance programs and providing assistance, that were important to opening and operating the Bureau as well as assuming control of the financing programs. As part of its work to develop a structure for the Bureau, DOT’s initial implementation plan set out guiding principles for what the Bureau aims to achieve: mobilizing available financial resources for high-impact transportation projects in the United States; identifying and encouraging innovative best practices in project planning, financing, delivery, and monitoring; clearing roadblocks to provide financing and grants more quickly and transparently, with a streamlined user interface and less uncertainty, complexity, and cost for project sponsors; and ensuring the protection of public resources through efficient leveraging of taxpayer money and the development of a creditworthy portfolio of projects. DOT also created an organizational structure for the Bureau and laid out the Bureau’s relationships to other offices in DOT. When the Bureau opened in July 2016, DOT appointed an Acting Executive Director, filled 29 positions with staff from other DOT offices, and created two offices within the Bureau, all of which generally aligned with the initial implementation plan. The Outreach and Project Development Office largely aligns with the Bureau responsibility to provide assistance to sponsors, which includes providing technical assistance on public-private partnerships and federal requirements to specific project sponsors as they prepare to apply for funding and financing. The Bureau’s Credit Programs Office largely aligns with the Bureau’s responsibility to administer the application evaluation process for certain programs through its work on underwriting, risk management, and portfolio management. DOT decided to also leverage other DOT offices within OST and modal administrations to carry out some of the Bureau’s work. Bureau officials told us this was a more efficient approach because it used the expertise and support of existing DOT offices rather than duplicating this expertise and support. For example, DOT used staff in OST that administer another competitive grant funding program to administer the INFRA grant program. See appendix II for more detail on the Bureau’s organizational structure and staffing. In its initial work, DOT also created a consolidated process for the Bureau to use when working with project sponsors pursuing TIFIA and RRIF financing, including standardized steps for evaluating applications for financing programs as well as providing assistance. Applications for the PAB program, which was also moved into the Bureau, go through many of the same application evaluation steps as TIFIA and RRIF, especially in the latter phases. DOT’s work to create this process aligned with two responsibilities given to the Bureau in the FAST Act: Administering the application evaluation process for certain programs Providing assistance to project sponsors seeking funding or financing DOT, in creating this process, set out steps that the Bureau would follow when working with sponsors. In the first two phases—initial engagement and project development—the Bureau provides assistance to project sponsors as they consider and navigate the financing programs. In those phases, a single point of contact works with sponsors to share information on the Bureau and provide assistance as sponsors develop materials to apply for financing programs. In the remaining phases of the process, Bureau staff and other DOT officials evaluate financing applications. During the creditworthiness review for a TIFIA or RRIF loan, for example, Bureau staff and independent advisors conduct an in-depth review of the project, including the sufficiency of a proposed repayment stream or collateral pledged. Throughout the process, the Credit Review Team—a decision-making body composed of Bureau and other DOT staff—votes at three points whether to advance a project seeking a TIFIA or RRIF loan and votes once for PAB allocations. During a later phase in the process, the Council then votes whether to recommend that an application advance to the Secretary for approval. The phases and steps in the Bureau’s process are summarized in figure 1 below. In creating this consolidated process, DOT also sought to improve and streamline the process, as called for in the FAST Act. Overall, DOT officials and documentation stated that these improvements, described below, should allow the Bureau to gather more information and better assist sponsors in the early phases of the process as well as identify and address potential issues earlier in the process. Single point of contact in the initial engagement and project development phases. The Bureau provides a single point of contact to assist sponsors during the early phases of the process. With a single point of contact, the Bureau aims to provide a streamlined interface with DOT for a sponsor. Furthermore, Bureau documents show that the single point of contact works with the sponsor to identify specific technical assistance needs—such as help completing environmental review requirements—and then develops a roadmap for providing this assistance as the sponsor develops its draft application. The point of contact can also help to resolve any conflicting requirements; for example, Bureau officials said the point of contact can facilitate discussions with a project sponsor and modal administrations on which Buy America requirements apply for a multi-modal project, as the requirements may differ across modes. Bureau officials said the Bureau’s work in these phases builds off the functions of the Bureau’s predecessor, the Build America Transportation Investment Center, which the initial implementation plan shows reached out to some sponsors interested in federal financing and connected them to the TIFIA, RRIF, and PAB programs, as well as the work of the former TIFIA Joint Program Office and RRIF Office. In contrast, the Bureau now more formally connects early assistance to later phases where the Bureau evaluates financing applications, all within the same office. Combined process for the creditworthiness review, application review, and Council review phases. The Bureau’s process combined the various review processes previously used by the three separate offices—in FHWA, FRA, and OST—to administer the three financing programs—TIFIA, RRIF, and PAB, respectively. For example, before this new process was implemented, a sponsor seeking a TIFIA loan and a RRIF loan would have to submit two applications to two offices and then work through two different processes; now a sponsor can submit one application to the Bureau and work through a single process for both loans. Our analysis of DOT and Bureau documents found that the reviews conducted in these phases are largely built off and resemble previously used processes. For example, the initial implementation report shows that previously the offices administering TIFIA, RRIF, and PAB were each required to brief the Council’s predecessor at different steps for each program, while the new process requires briefings to the Council at the same step for each program. Formalized decision-making body that monitors and advances projects through phases. The Credit Review Team—the new, primary decision-making body within the Bureau—plays a key role in deciding when projects can advance from one phase to another. For example, the team reviews a project’s initial materials for a TIFIA or RRIF loan and then votes on whether the project is ready to advance to the creditworthiness review phase. According to Bureau documents, the team’s predecessor, a less formal working group, did not review projects until after the creditworthiness review began. Bureau documents show that the Credit Review Team is meant to meet weekly, in contrast to its predecessor organization, which met monthly. According to Bureau officials, this more frequent meeting schedule allows the Bureau to expedite its decision-making. Since DOT designed and established the Bureau, the Bureau has made more limited progress in its first 2 years on addressing additional responsibilities assigned to it by the FAST Act, as listed and described below. Bureau officials spoke generally about plans to continue making progress on these responsibilities in the future, and pointed out that the Bureau is still a relatively new office that remains a work in progress. However, Bureau officials were unable to provide written plans or timelines for these additional efforts. Promoting innovative-financing best practices. The Bureau has started to address this responsibility by employing the expertise of modal administration staff. The Bureau signed an agreement with FHWA in October 2016 to leverage the expertise of FHWA’s long- standing Office of Innovative Program Delivery rather than duplicate these efforts in the Bureau. Since signing the agreement, the Bureau and FHWA have jointly developed or updated a number of resources for public-private partnerships, building on FHWA’s existing work. This includes conducting on-site trainings for state entities and updating two model contract guides. Progress with other modal administrations has been more limited. For example, Bureau staff told us they have worked with FTA to start to identify gaps and jointly produce materials, such as an upcoming public-private partnership procurement guide. Though the Bureau does not have a signed agreement with FTA, Bureau officials said they want to sign one. Bureau officials said that they have started speaking with officials at other modal administrations to identify opportunities but that it will take time to identify gaps and develop tools in innovative financing for rail, maritime, and aviation. Improving environmental reviews and permitting. Bureau officials said they have relied on the expertise of DOT’s Infrastructure Permitting Improvement Center to carry out responsibilities to improve environmental reviews and permitting, rather than duplicate this expertise in the Bureau. The Center’s stated mission is to improve the performance of federal environmental review and permitting of infrastructure projects. As a result, Bureau officials said the Center carries out several specific responsibilities directed to the Bureau in the FAST Act, including serving as DOT’s liaison to the Council on Environmental Quality and tracking metrics for permit reviews and decisions in a public dashboard. According to Bureau officials, the Infrastructure Permitting Improvement Center and the Bureau also jointly hired an environmental expert. This environmental expert’s duties include supporting broad efforts to improve the efficiency and effectiveness of these processes in the Center and providing technical assistance to ensure that environmental reviews on specific projects move forward in the Bureau. Bureau officials told us that the Bureau does not have a written plan to document its efforts to fulfill the Bureau’s FAST Act environmental review and permitting responsibilities, beyond the position description for the environmental expert, because both offices are under the direction of the Under Secretary. However, the position description does not mention the Bureau or provide a sequence or timeline to fulfill these responsibilities that could help ensure continued progress. Sharing information on procurement costs and risks. The Bureau has not taken steps to collect or share information on procurement costs and risks, though documents show it has coordinated with FHWA to take some preliminary planning steps. For its FAST Act responsibility to develop, collect, and publish procurement benchmarks, the Bureau and FHWA published a preliminary paper in June 2017 that identified the types of procurement information to collect and publish, identified existing information sources for highway projects, and outlined possible next steps. However, Bureau officials told us that much work remains to identify specific cost and schedule information to collect from project sponsors and ultimately publish procurement benchmarks for projects across modes. The FAST Act also directs the Bureau to require sponsors procuring a project as a public-private partnership to conduct and publish value for money assessments and after-action reports, but the Bureau has not taken steps to do so. Bureau officials stated that additional efforts to address these responsibilities will require additional work and resources. Bureau officials could not provide a written plan or schedule for these future efforts. Several factors, including some outside the Bureau’s control, have affected the Bureau’s ability to more fully carry out its responsibilities in its 2 years of operation. First, there have been changes in leadership. After the presidential transition in early 2017, many DOT leadership positions, including many members of the Council, were vacant until new political appointees were put in place. Bureau documents show that the Council did not meet for 2 months, and Bureau officials told us that career staff sat on the Council to enable it to meet and resume voting on applications until appointees were confirmed. In addition, the Bureau’s Executive Director stepped down in November 2017. The Bureau is currently trying to fill that position through a second job announcement. With the Executive Director position vacant, Bureau officials told us that the Deputy Assistant Secretary and 3 senior officials from the Bureau and OST have fulfilled the day-to-day activities of that leadership role in the interim. Bureau officials told us that the lack of an Executive Director has had an effect on setting long-term plans for the Bureau; such planning is part of the duties of that position. Some stakeholders we spoke to stressed the importance of having an Executive Director in place so Bureau staff can quickly elevate issues or make decisions that currently need to be made by higher-level officials. Second, the Bureau has had a number of vacant positions since it was opened. Based on Bureau documents and discussions with Bureau officials, we determined that between 8 and 11 positions in its current organizational chart were vacant during 2018. During the government- wide hiring freeze in early 2017, the Bureau could not fill any vacancies, but several positions remained vacant before and after the hiring freeze, and two former Bureau officials said that the Bureau remained understaffed into mid-2017. The positions vacant during 2018 changed over time due to attrition, but two positions that remained vacant throughout this period are the transit-oriented development and project finance specialists. When asked about the vacancies in early 2018, Bureau officials said that they had originally wanted to fill the Executive Director position before filling other vacancies but later decided to start filling some critical vacancies. In July 2018, Bureau officials discussed their strategy for filling some vacant positions in response to immediate needs and in October 2018 said they intended to fill all vacant positions. Throughout this period, Bureau officials verbally shared these staffing priorities with us but did not provide a written plan or strategy for prioritizing the Bureau’s vacancies. Bureau officials said they do not have a timeline to fill remaining vacant positions in part due to limited human capital resources to draft position descriptions and conduct other parts of the hiring processes. DOT’s efforts to establish the Bureau and its processes were guided by an initial implementation plan. However, subsequent work by the Bureau to address its responsibilities and continue its implementation efforts is ongoing without the benefit of a plan and associated timelines. Key practices for organizational transformations state that an agency must set implementation goals and a timeline and ensure that top leadership drives the transformation, as such a transformation could take years to complete. Bureau officials have developed general priorities and approaches that they said have been communicated to staff through regular meetings and use specific performance plans to guide work in certain areas. However, without detailed written plans with implementation goals and timelines, the Bureau risks not being able to sustain the progress it has made in the last 2 years and ensure that it implements all of its statutory responsibilities in a timely manner. Finally, though the consultant’s report recommended that the Bureau develop indicators to track its performance, the Bureau has not established any indicators or measures to track progress in accomplishing its guiding principles or mission to be a “one-stop shop.” Federal standards for internal control and key practices for organizational transformations stress the importance of setting measurable objectives and developing performance measures to assess progress. The consultant’s initial implementation plan identified a number of potential performance indicators for the Bureau, including customer satisfaction. Bureau officials said they currently track data on projects through early assistance and application evaluation. However, Bureau officials said they do not want to use certain indicators, such as those that measure how long different parts of the process take, as they could create incentives to move projects ahead before they are ready. However, our prior work shows that to counter such incentives as well as to help an agency avoid drawing the wrong conclusions about its effectiveness, an agency could use multiple indicators rather than any one indicator to assess progress. Concerns about one indicator might be countered by information from other indicators. For example, to help offset incentives to move projects ahead before they are ready, an indicator for how long different parts of the process take could be considered along with an indicator that also measures the ratio of projects that were and were not returned to staff to gather additional information. Without establishing or beginning to use performance indicators that measure the Bureau’s performance rather than the progress of individual projects as it currently does, the Bureau will not know if it is achieving its guiding principles or meeting the mission set out in the conference report that it serve as a “one-stop shop” that advances projects. Sponsors we interviewed had mixed views on the Bureau’s process for evaluating applications and providing technical assistance, including views on whether the process was quick or streamlined. Selected sponsors had a generally positive experience with the PAB application evaluation process. However, for TIFIA and RRIF, selected sponsors had more mixed experiences and identified challenges with the application evaluation process, including the length and uncertainty of the process, changes to requirements or terms, and unclear goals and risk appetite— that is, how much risk an agency is willing to accept to achieve its goals— for the programs. Bureau officials identified limitations to providing more certainty to sponsors for each of these challenges and noted that the Bureau cannot control all the factors, such as a sponsor’s responsiveness or changes to a project’s proposed financing, surrounding the application evaluation process. However, the Bureau has also not determined how it will improve or streamline its process by, for example, consistently soliciting feedback from sponsors, nor has it outlined the goals and appetite for risk for TIFIA and RRIF. As discussed earlier, DOT created a consolidated process for evaluating applications for its financing programs. Selected sponsors we interviewed that applied for a PAB allocation since the Bureau was created had a generally positive experience with the PAB application evaluation process. In particular, sponsors of the four PAB projects we selected said the process was quick and streamlined. For example, each sponsor said the process met or exceeded its schedule expectations for receiving a PAB allocation. In addition, these sponsors said the process was simple to follow and that the simplicity was an important strength. One sponsor found its point of contact’s efforts to clearly explain information requirements early in the process as useful to understand the Bureau’s expectations. DOT officials also said that PAB applications can move relatively quickly as they, in contrast to TIFIA and RRIF, do not create a direct financial risk for DOT or the federal government since DOT’s role is limited to approving the use of tax-exempt bond authority. Selected sponsors that applied for TIFIA and RRIF financing had mixed views on their overall experiences with the Bureau’s application evaluation process. Some sponsors had positive experiences to share. Among sponsors of six projects we selected, two sponsors said they believed the application evaluation process was streamlined, and five sponsors said it was somewhat streamlined. Some sponsors based their responses on comparing the Bureau’s process to the processes previously used to administer TIFIA and RRIF, while other sponsors focused on whether the process was efficient. For example, one sponsor that was new to TIFIA and that believed the process was streamlined said the Bureau was thorough but did not ask repetitive questions and that the process was not overly onerous. In terms of speed, two sponsors said the process was quick, two said the process was somewhat quick, and three said the process was not quick. Among sponsors of the six projects we selected and ten additional sponsors and stakeholders that had experience with some part of the Bureau’s application evaluation process, five sponsors and one stakeholder found the responsiveness of the Bureau’s staff to questions or issues as most useful, and several sponsors also praised individual staff or cited the professionalism and commitment of Bureau staff. Despite these positive comments, sponsors and stakeholders we interviewed also identified challenges with the application evaluation process for TIFIA and RRIF and offered some suggestions to improve the process, including how to further streamline the process. Based on our interviews, the most common challenges involved uncertainty related to the overall length of the application process, changes to the Bureau’s requirements or terms for loans, and the goals and risk appetite for the financing programs. We and others have previously reported on some of these challenges for TIFIA or RRIF. Length and uncertainty of process. Four sponsors and one stakeholder said the overall length of the application evaluation process creates a challenge when seeking and planning for credit assistance. This challenge predated the Bureau as we similarly reported in 2012 and 2016, before the Bureau was created, that project sponsors cited the length of the application evaluation process for the TIFIA and RRIF programs respectively as challenges. Furthermore, seven sponsors and three stakeholders we spoke with also said the Bureau should refine or further streamline the application evaluation process. For example, one sponsor said it faced an uncertain timeline when its project awaited Credit Review Team approval and that it was not informed by the Bureau when the meeting would be held. The Bureau instituted regular Credit Review Team and Council meetings to give sponsors a greater sense of certainty and transparency on when DOT would be voting to advance a project. Another sponsor said it took the Bureau over 3 months to procure independent advisors to help with the Bureau’s creditworthiness review, though Bureau officials said it takes about 6 weeks to procure these advisors. In our analysis of six selected TIFIA and RRIF projects, we found that five projects signed their credit agreements between 3 and 6 months later than was anticipated when the project was in creditworthiness review, according to Bureau documents for each project. Our analysis also found that the processing time for steps in the process varied, including steps that may be more within the Bureau’s control. For example, the number of days between a project’s receiving approval by the Council and the Secretary ranged from same-day approval to 43 days. Though some slowdowns can result from factors that are out of the Bureau’s control, sponsors we interviewed discussed the overall effect of slowdowns to projects. For example, sponsors of two projects said application slowdowns led to cost increases and a schedule delay for one project. To improve the application evaluation process, three sponsors and one stakeholder said the Bureau could provide tailored schedules for a project for each phase of the process. One stakeholder also said the Bureau could add certainty and transparency by providing information on how long different phases generally take, information that this stakeholder said it had not received when working with the Bureau, though this is a customary practice when seeking financing in the private sector. Bureau officials pointed out limitations to providing or predicting formal schedules and timelines for the process for specific projects. Bureau officials said many factors influence how quickly a project can advance through the application evaluation process for TIFIA and RRIF, primarily the quality of the project’s credit and overall complexity. In addition to these primary factors, Bureau officials said an application’s processing time can be affected by a sponsor’s responsiveness to requests or whether the sponsor is concurrently negotiating other agreements. Bureau officials said they do not tell a sponsor the specific date of the Credit Review Team or Council meeting on which its project will be reviewed, but instead tell a sponsor what information is needed and by when to reach the next meeting. The Bureau takes this approach because a sponsor may, for example, provide incomplete information, meaning the project would have to wait to be discussed at a meeting that is later than expected. Furthermore, the dates of Council meetings often change due to the members’ schedules, and the Bureau does not want to cause a sponsor undue alarm if the date changes. Bureau officials said they provide a general schedule to a sponsor once a project enters creditworthiness review and use this schedule as a starting point to build a tailored schedule for a project. We found that this general schedule uses historical data to show how long steps in the process could take, but this schedule uses steps and decisions for the process used for TIFIA that pre-dated the Bureau. Bureau officials also said they may informally identify ways to expedite the process where appropriate for a specific project, but that these enhancements affect primarily lower-risk projects. Changing requirements or terms for loans. Six sponsors said changing requirements or terms during the application evaluation process created a challenge of having to navigate new expectations during the process. For example, two sponsors said they had to make changes to terms and conditions for loans late in the process. Specifically, one of the sponsors said it would have preferred to learn about the Bureau’s policy related to certain terms earlier in the process rather than have to accept an unexpected change late in the process, after it has committed time and resources to the process. One sponsor said certain terms developed by the Bureau’s underwriting team, which conducts the creditworthiness review, had to be restructured following review by the Credit Review Team. Another sponsor said the Bureau changed or introduced new requirements after it began the application evaluation process, including what was required at particular steps, but did not provide reasoning for its changes. To address such challenges, four sponsors and two stakeholders said the Bureau could better accommodate projects with different revenue streams by, for example, creating different standard terms and contract templates. Bureau officials described factors that can result in changes to the tentative terms and conditions during the application evaluation process for a project. For instance, if a project’s scope or construction cost estimates change significantly in ways that affect the financial assumptions for a project, the Bureau must reevaluate the project and make changes to the terms and conditions accordingly. Bureau officials said they try to balance providing certainty and flexibility but lean toward providing flexibility; for instance, the Bureau will try to accommodate a sponsor that changes the proposed financing for a project, which then may result in changes to terms as the Bureau reevaluates the project’s risk. In addition, the terms and conditions discussed for a project are tentative until they are approved by the Credit Review Team, Council, and Secretary. According to Bureau officials, sponsors can advance through the application process more quickly and with greater certainty by agreeing to use the Bureau’s standard credit terms—that is, agreeing to the terms and conditions in a template provided by the Bureau as opposed to choosing to negotiate with the Bureau with those terms and conditions as a starting point. Finally, Bureau officials said they were developing two additional standard loan templates to post on the Bureau’s website with the two existing loan templates for projects with different financing structures and revenue streams. Unclear program goals and risk appetite. Many sponsors we interviewed said the Bureau did not clearly convey the program goals or appetite for risk for its TIFIA and RRIF programs. Eight sponsors and one stakeholder cited the Bureau’s approach toward risk as creating a challenge for sponsors to determine if their projects fit the Bureau’s programs. Four sponsors said the Bureau required strict terms and conditions in its credit agreements that seemed excessive, and one sponsor said such strict terms can impose additional costs on a sponsor without materially improving credit quality since a project must have an investment-grade credit rating. One sponsor stated that the lack of clarity on goals and appetite for risk for its project, coupled with other challenges, led the sponsor to withdraw from seeking financing. According to the sponsor, while the programs were created to fill market gaps, it is not clear whether the Bureau’s financing programs currently seek to provide financing to lower risk projects that have a high-quality credit rating or to higher risk projects that are unable to secure financing in the private markets. Similarly, a May 2017 Congressional Research Service report noted that a significant portion of RRIF financing has gone to passenger rail projects since 2008, though the program was primarily created to support freight rail projects, and that the size of loans and some of the risks for passenger rail assistance differ from the assistance historically provided for freight rail. One sponsor we spoke with said it would be helpful if the Bureau and the Council shared information with sponsors regarding DOT’s appetite for risk when evaluating projects, similar to how commercial banks can share a risk profile framework. Bureau officials said DOT’s financing programs and their treatment of risk have evolved over the past decade based on changes to private markets and lessons learned by DOT in working on projects that faced bankruptcy. According to Bureau officials, the Bureau has also changed its standard terms and conditions, as any lender would do, over time. However, Bureau officials said the Bureau lacks an external statement that communicates its goals and appetite for risk for its financing programs. Bureau officials told us they have developed a draft risk appetite statement for internal use. Officials said this risk appetite statement is imbedded in draft credit-risk guidelines the Bureau is developing to use to enable more consistent review of individual projects applying for financing. The officials noted that this draft statement is short and general by design because TIFIA and RRIF can finance a wide range of projects. Furthermore, Bureau officials said it would be difficult to create a public risk appetite statement, as suggested by the consultant, that did not constrain their flexibility to finance a range of projects, particularly as the Bureau seeks to further diversify its portfolio and assist a variety of projects. In lieu of a public risk appetite statement, the Bureau encourages sponsors to meet with its staff early to assess whether a project would be a good fit for its financing programs. However, Bureau officials agreed that it could be beneficial for the Bureau to issue a public statement that conveys how it intends to balance its financing portfolio and support varying types of risks and projects that seek assistance. Given the challenges identified by sponsors, we found that the Bureau has not developed an approach to assess how effectively its application evaluation process works for TIFIA and RRIF, including what in the process is challenging and what works well. In particular, Bureau officials said they have not formally analyzed the amount of time it takes for projects to proceed through the process due to concerns that assessing speed and efficiency may not be appropriate to track for all projects. For example, a sponsor may not need financing immediately and thus choose to proceed at a slower pace. Also, while Bureau officials said it would be beneficial to formally solicit and analyze the satisfaction of sponsors that have closed loans, the Bureau has not implemented a mechanism to systematically solicit feedback on sponsors’ experiences, including any challenges. Federal standards for internal control state that management should design control activities to achieve its objectives. Control activities include reviews of an agency’s programs or activities to compare actual results to objectives and expected results, for example by evaluating the amount of time projects take in each step of the process. Federal standards for internal control also state that an agency should externally communicate information to achieve its objectives; this communication includes receiving information through reporting lines from external parties to help ensure effective operations. In addition, Office of Management and Budget guidance to agencies that manage financing programs also states that effective oversight relies on robust data collection and reporting systems that include, for instance, metrics from collected feedback on customer service or overall applicant satisfaction. As noted above, the Bureau cannot control all the factors and circumstances surrounding the application evaluation process. However, officials have stated that the Bureau seeks to expand and diversify the types of projects that access the TIFIA and RRIF programs, and one of the Bureau’s own guiding principles is to clear roadblocks to provide financing more quickly and transparently and to have a consistent application process. Without a mechanism to formally examine how to improve and further streamline the process, the Bureau may be missing an opportunity to address any recurring challenges with the process or with how the Bureau communicates with sponsors, a situation that could discourage sponsors from the seeking financial assistance from these programs. Moreover, the Office of Management and Budget has directed agencies that manage financing programs to establish acceptable risk thresholds to balance policy goals with risks and costs to the taxpayer, and to monitor the program’s progress toward achieving policy goals within those acceptable risk thresholds. Federal standards for internal control also call for management to define objectives or goals clearly to enable the identification of risks and define risk tolerances. These standards also call for management to externally communicate the necessary information to achieve its goals. In the initial implementation plan, the Bureau’s consultant recommended that the Bureau publicly issue a risk appetite statement that specified acceptable types of risks and projects DOT would support. We have previously reported that setting an organizational risk appetite is an example of a good practice agencies can take to align risk management processes to goals and objectives. We also reported that by not clearly defining and communicating its appetite for risk, an agency could be taking risks well beyond management’s comfort level or be passing up opportunities by assuming its leaders were risk averse. In addition, a former DOT official we interviewed said DOT and the Bureau should have an in-depth conversation about the risk in its portfolio of projects to help decide what risks are tolerable and, thus, help the Bureau better decide the risks it can accept for individual projects. Without clearly defining and communicating to the public the goals and appetite for risk for TIFIA and RRIF programs, the Bureau may be missing an opportunity to make its application process more transparent. Moreover, by issuing a public statement that clearly communicates the types of risks DOT is willing to accept, sponsors would be in a better position to determine if the TIFIA and RRIF programs would be a feasible option for their projects before committing resources to applying. Since it opened in July 2016, the Bureau has provided technical assistance to sponsors for 119 distinct projects, based on our analysis of Bureau data. As of August 2018, about half of projects were in the early phases of working with the Bureau. In total, 56 projects were in initial engagement or project development, the phases during which the Bureau provides technical assistance to sponsors (see table 1). By mode, rail and highway projects comprised about half of all projects. The amount of technical assistance and level of interaction between the Bureau and project sponsor in the initial engagement and project development phases varied, based on the sponsor’s experience using DOT’s financing programs and the project’s complexity. For example, one sponsor we interviewed met with the Bureau to discuss the expected timing to apply for and receive a TIFIA loan; this sponsor did not seek additional technical assistance in project development as it had previously received a TIFIA loan and had completed work to comply with federal requirements for the project, including the environmental review and permitting work. Another sponsor we interviewed was new to the Bureau’s financing programs and met with the Bureau to learn more generally about the requirements for the different programs and the application process. Half of the sponsors we interviewed were satisfied with the Bureau’s technical assistance, but some sponsors expressed concerns including the following: Ability and willingness to move projects forward. In our interviews with 16 sponsors that received technical assistance from the Bureau, 8 said they were satisfied with the technical assistance provided by the Bureau, and 9 said that the Bureau functioned as a one-stop shop to access financing and funding programs and technical assistance. However, six sponsors said the Bureau’s technical assistance was slightly helpful or not helpful in clearing roadblocks to provide credit and grants more quickly and transparently. For example, one sponsor said its project experienced delays over a period of several months as it made multiple attempts to obtain specific, actionable feedback from the Bureau on its materials to better understand what was needed to advance in the Bureau’s process. Lack of clarity on RRIF program eligibility. In our interviews with sponsors, a recurring concern included a lack of clarity from the Bureau on eligibility requirements for the RRIF program, in particular for sponsors seeking financing for transit-oriented development projects. For example, from information gathered from sponsors of 10 inactive projects, we found that four were transit-oriented development projects that became inactive because the Bureau determined them to be ineligible. Sponsors of two of these projects said they were initially told their projects would be eligible, but after continuing to work with the Bureau for 5 to 6 months, the sponsors said their transit-oriented development projects were determined to be ineligible for the RRIF program. In addition, sponsors of these two projects said they faced difficulty reconciling differences found in the Bureau’s transit-oriented development eligibility guidance for the RRIF program and transit-oriented development guidance issued by modal administrations for other programs. For example, one sponsor said it felt that the Bureau’s guidance did not clearly outline the eligibility requirements for transit-oriented development for the RRIF program and that it would help if the Bureau provided greater clarity about what kinds of development around rail stations would be eligible. In response to these concerns, the Bureau has begun taking steps that could help address them. For example, the Bureau is working to develop an expedited application process—RRIF Express—for RRIF projects that meet certain criteria. As we and the DOT Office of Inspector General have previously reported, sponsors have identified challenges with RRIF that, in some cases, have deterred them from applying to the program, so steps taken by the Bureau to expand use of the program are of particular interest to many sponsors of potential rail projects. Despite these efforts, as stated earlier, the Bureau does not have a written plan to guide its continued implementation efforts, and it does not have a formal mechanism to examine how it could improve its process for working with sponsors. Such a plan and mechanism could help the Bureau better understand and appropriately address sponsors’ concerns with the Bureau’s provision of technical assistance. As discussed earlier, the FAST Act required the Bureau to document major decisions in the application evaluation process and provide a clear rationale for its decisions. Federal standards for internal control also call for management to internally communicate the necessary quality information to achieve its objectives; this communication includes providing management quality information that is necessary for effective oversight. We reviewed documents for six TIFIA and RRIF projects and found the Bureau documented each decision to approve these projects and provided a clear rationale for those decisions. To document decisions about whether to advance and approve these projects, the Bureau used formal meeting agendas and notes from the Credit Review Team and Council meetings and internal memorandums. For example, the Bureau used internal memorandums to record the Secretary’s signature of approval to extend credit to a project. To document the rationale in support of these decisions, the Bureau used internal reports and memorandums. For example, to support its decisions to invite or not invite a project sponsor to submit a formal application, the Credit Review Team provided a description of how the project satisfied program requirements like having a preliminary rating opinion letter and how the project satisfied program creditworthiness standards including the sufficiency of the repayment source or collateral. However, in our review of four projects that received PAB allocations, we found that while the Bureau documented its decision about whether to advance and approve each application, it did not document a clear rationale to support that decision. Specifically, the Bureau recorded decisions in Credit Review Team and Council meeting materials and the approval letter sent to the sponsor. To evaluate a PAB application, the Bureau reviews the application against statutory eligibility requirements and the availability of PAB allocation capacity. We found that the Bureau’s documents in the PAB evaluation process lacked a clear rationale in support of decisions. Specifically, the documents summarized information from the application but did not articulate whether or how the Bureau determined that this summarized information from the application satisfied PAB eligibility and availability requirements. We found that this occurred because the Bureau lacks a policy to document the rationale for how a project meets statutory and DOT requirements in order to advance a PAB application. DOT officials said determining whether a project meets requirements to receive a PAB allocation can be self-evident, and therefore, the application itself can be sufficient documentation. However, absent a documented rationale to support its decisions, it is not immediately clear what information the Bureau cited or used to make decisions about applications through the process. As a result, DOT, the Bureau, and the PAB program could be exposed to risks. For example, we previously reported that programs that do not have defined application review procedures may not review applications consistently and thereby leave the program vulnerable to questions about the integrity of the process. Moreover, as the PAB program nears the $15 billion allocation limit, recording the rationale—including the effect of a proposed allocation—would help ensure DOT’s decision makers receive up-to-date information needed to make informed decisions and manage the program. With the creation of the Bureau, transportation projects seeking financing from DOT have a new, central point of contact for assistance. A concerted initial-planning effort enabled the Bureau to open and start working with project sponsors in just over 6 months after federal law called for its creation. The Bureau has made varied progress on its statutory responsibilities since it was created over 2 years ago. This situation underscores the need to sustain momentum beyond an initial implementation effort, in order to give ongoing planning and attention to additional priorities and tasks and to identify possible improvements based on early experiences. The Bureau was given a challenging task— to serve as a one-stop shop that provides a number of different services and diverse technical resources. However, without an implementation plan and performance indicators, it may not be able to sustain its progress and prioritize its efforts. In response to congressional direction for the Bureau to make changes to streamline the application evaluation process for DOT’s financing programs, the Bureau created a new, consolidated process to accept and evaluate applications. However, the Bureau has not developed an approach to examine whether opportunities for further streamlining and improvement exist. Furthermore, absent clarity about the Bureau’s appetite for risks for its financing programs, sponsors lack information to know if they should invest time and resources applying for TIFIA or RRIF for their projects. Without examining the Bureau’s process and communicating its appetite for risk, the Bureau may be missing an opportunity to address any recurring challenges that may undermine the purpose and availability of its programs. Finally, for the PAB program, the Bureau does not have a policy to document its rationale justifying decisions and that lack of a rationale may leave the Bureau open to challenges regarding its decisions. By providing the rationale for its decisions, the Bureau could engender more trust in these decisions and increase the program’s transparency. We are making the following five recommendations to DOT: The Under Secretary of Transportation for Policy should ensure that the Build America Bureau develop a detailed implementation plan that sets goals and a timeline for the Bureau’s continued efforts, fills vacancies in the Bureau, and prioritizes and sequences work to carry out the multiple responsibilities given to the Bureau in the FAST Act. (Recommendation 1) The Under Secretary of Transportation for Policy should ensure that the Build America Bureau develop performance indicators to assess the Bureau’s progress toward meeting its guiding principles or mission as a “one-stop shop.” (Recommendation 2) The Under Secretary of Transportation for Policy should ensure that the Build America Bureau develop a mechanism to assess the Bureau’s application evaluation process for TIFIA and RRIF and identify and address opportunities to improve and further streamline the process. This evaluation should include mechanisms to solicit feedback from project sponsors that sought financing. (Recommendation 3) The Under Secretary of Transportation for Policy should ensure that the Build America Bureau develop and adopt a public statement that outlines DOT’s and the Bureau’s policy goals and appetite for risk for the TIFIA and RRIF financing programs. (Recommendation 4) The Under Secretary of Transportation for Policy should ensure that the Build America Bureau establish a policy to document a clear rationale to support decisions made in the PAB application evaluation process to explain why an allocation should or should not be approved. (Recommendation 5) We provided a draft of this report to the Department of Transportation for review and comment. In its comments, reproduced in appendix III, DOT concurred with our recommendation to develop performance measures (Recommendation 2) and to assess its application review process (Recommendation 3). DOT did not fully concur with our recommendations to develop a detailed implementation plan (Recommendation 1), adopt a public statement of its policy goals and risk appetite for its financing programs (Recommendation 4), and establish a policy to document the rationale for decisions in the PAB process (Recommendation 5). In its comments, DOT did not provide reasons for disagreeing with these three recommendations. We continue to believe that it is important for DOT to implement these recommendations to help the Bureau prioritize and complete its continued implementation efforts and to help improve the transparency of the Bureau’s processes and decisions for evaluating applications. DOT also provided one technical comment, which we incorporated. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. The Fixing America’s Surface Transportation Act (FAST Act) required that the Department of Transportation (DOT) establish a finance bureau to coordinate and consolidate certain surface transportation funding and finance programs. The FAST Act also included a provision for GAO to review the Bureau’s actions to establish procedures for evaluating applications for programs it administers and provide a clear rationale for major decisions in the application evaluation process. We assessed (1) the progress DOT made to establish the Bureau and carry out its responsibilities; (2) the Bureau’s process for evaluating applications and providing technical assistance, including obtaining the views of sponsors and stakeholders; and (3) whether the Bureau, when evaluating applications, has provided a clear rationale for its decisions. In the second objective, we focused on the Bureau’s work evaluating applications and providing technical assistance because these two responsibilities aligned with the mandate for GAO and were responsibilities the Bureau has made the most progress on. To examine DOT’s progress establishing the Bureau, we reviewed DOT and Bureau documents—90-day and yearly implementation progress reports to Congress, operating procedures, job descriptions and position postings for vacant positions, and budget requests—to determine DOT’s plans and progress organizing and staffing the Bureau. We also analyzed reports, including an initial implementation plan, created by a consultant DOT hired in 2016 to help it create and organize the Bureau, and we reviewed the FAST Act and appropriations acts to identify DOT authorities to eliminate and consolidate offices and transfer funds and staff in order to establish the Bureau. We interviewed former and current DOT and Bureau officials to understand DOT’s goals and priorities, coordination with modal administrations, challenges or successes, and key next steps for the Bureau. We selected former DOT and Bureau officials who played key roles to establish or work in the Bureau or that were recommended in our interviews. We also interviewed select associations and advisors about their interactions with the Bureau to date, including observations on its creation, organization, and staffing. We selected associations representing project sponsors that have sought or could seek assistance from the Bureau, that vary in mode and sponsor type, and that vary in terms of experience working with the Bureau since July 2016. We selected advisors that have experience working with multiple project sponsors and that worked with the most sponsors of recently closed TIFIA and RRIF loans. At the end of this appendix, these selected organizations are included in table 2, which lists the individuals and organizations interviewed for this report. In addition, to determine DOT’s and the Bureau’s progress in carrying out responsibilities set out for the Bureau in the FAST Act, we examined DOT and Bureau documents, such as the Credit Programs Guide and Build America Bureau Processes and Governance Manual, and procedures, documents, and other information publicly available on the Bureau’s website. We supplemented this information with interviews with DOT and Bureau officials to understand the progress the Bureau made for each responsibility and how the Bureau prioritized its approach to fulfilling these responsibilities overall. We also used these interviews to understand the Bureau’s timeline or strategy for fulfilling each responsibility in the future or the cause of no action to date for responsibilities on which the Bureau has taken limited or no action, as well as to understand what metrics or performance measures DOT established to track its progress or outcomes for these responsibilities. We also asked stakeholders we interviewed—including select former DOT officials, associations, and advisors, selected as described above— about their observations on the Bureau’s progress in carrying out these responsibilities. We compared DOT’s and the Bureau’s efforts to federal standards for internal control and key practices for organizational transformations. To assess the Bureau’s process for evaluating applications and providing technical assistance, we reviewed the Credit Programs Guide and other Bureau documents and interviewed Bureau officials to determine the phases and steps in the process. We also reviewed these documents and interviewed Bureau officials to understand the changes DOT made to combine and consolidate existing processes. Our review of the process of evaluating applications included semi-structured interviews with selected project sponsors and stakeholders to understand their experiences using the application evaluation process, experiences working with the Bureau, and comparisons of the application process before and after the Bureau was created, if applicable. First, we selected sponsors for the 10 projects for which we reviewed application documents, as described below, to determine whether the Bureau provided a clear rationale for its decisions. Second, we selected other stakeholders—including advisors and associations (as described above) and projects sponsors with experience applying for DOT financing both before and after the Bureau was created. Among these project sponsors, we selected three projects that had multiple loans; used special authorities or agreements (i.e., master credit agreement); or employed public-private partnerships to deliver projects. Five additional project sponsors, selected as part of other samples described in this appendix, had experience with some part of the TIFIA or RRIF application evaluation process under the Bureau, so we asked these sponsors questions on this part of the process. We analyzed the interview responses by categorizing them based on the extent to which respondents said the process was quick, streamlined, and transparent; what in the process was most useful and most challenging; suggestions for improving the process; and overall satisfaction or dissatisfaction with the process. Furthermore, our review of the Bureau’s process for providing technical assistance included analyzing the Bureau’s data to describe the projects that have sought assistance from the Bureau since it opened by mode, location, type of financing pursued, and step reached in the application process. For technical assistance, we focused on project-specific assistance provided by the Outreach and Project Development Office before a project enters the creditworthiness review phase—referred to as initial engagement and project development. We reviewed the Bureau’s data on projects from April 2018 as well as updated data from August 2018. To assess the reliability of these data, we reviewed relevant documents and interviewed Bureau officials responsible for overseeing the data to learn how information was entered, maintained, and reviewed. We also reviewed relevant data elements for missing data, outliers, and obvious errors. Based on these steps we determined that the data were sufficiently reliable for the purpose of describing the number and type of projects that worked with the Bureau and selecting project sponsors to interview. We also conducted semi-structured interviews with project sponsors to understand their experiences working with the Bureau during the initial engagement and project development phases. In these interviews, we asked sponsors whether the Bureau serves as a single DOT point of contact and provides access to its finance programs with greater speed and transparency; for projects no longer seeking assistance from the Bureau, we asked about the reasons for doing so. Among project sponsors actively working with the Bureau, we identified 32 projects that began working with the Bureau after it was created in July 2016, that had met with or been in contact with the Bureau in the 6 months prior to April 2018, and that the Bureau ranked as 2 or higher on its readiness scale. Of these projects, we selected 13 sponsors to ensure variety in project status (i.e., initial engagement, project development, creditworthiness), mode, total project cost, prior experience with DOT’s financing programs, and location. Among project sponsors no longer actively working with the Bureau, we identified 10 projects that began working with the Bureau after it was created in July 2016 and had at least two interactions with the Bureau, based on available data. We selected 5 of these sponsors to interview to ensure variation in mode and location. For the Bureau’s provision of technical assistance, we categorized the responses to questions in terms of which interactions with the Bureau were most useful and most challenging, suggestions for improving the process, and overall satisfaction or dissatisfaction. For inactive project sponsors, we categorized responses according to reasons the project became inactive or withdrew from working with the Bureau, and what other financing, if any, was used for the project. Table 2 below lists project sponsors and other organizations we interviewed. Overall, we assessed the Bureau’s process for evaluating applications and providing technical assistance and the collected evidence against federal standards for internal control and Office of Management and Budget’s guidance for agencies that manage financing programs. To assess whether the Bureau provided a clear rationale for its decisions when evaluating applications, we reviewed the Credit Programs Guide and other Bureau documents to identify steps and major decision points and accompanying documents in the application evaluation process. We identified 5 major decision points for TIFIA and RRIF and 3 major decision points for PAB. We also used these documents to identify evaluation criteria for each major decision point (i.e., the information or requirements that the Bureau says must be considered at each decision point) to use to assess whether the Bureau provided a clear rationale for each decision point. We confirmed our list of steps and major decision points, as well as accompanying documents, with Bureau staff responsible for the financing programs. We did not examine whether the Bureau documented decisions for the grant funding program it administers, and we have previously evaluated this program and also have work in progress to evaluate it. To assess whether the Bureau followed these procedures and documented major decisions and rationale, we selected projects that went through most of the application process after the Bureau updated its process in September 2016. For TIFIA and RRIF, these are projects that completed the first or second decision point—being invited to enter creditworthiness or being invited to submit a formal application—and had signed credit agreements as of March 31, 2018. We selected all three projects that completed the first decision point and had signed credit agreements. We selected 3 of the 5 projects that completed the second decision point and had signed credit agreements to ensure variation in type of sponsor (e.g., state or local government, private entity), mode, and size of loan. For PAB, we selected all four projects that submitted an application after September 2016 and received an allocation as of March 2018. For each selected project, we reviewed Bureau documents, including meeting agendas and summaries, memos, summaries of financial analyses, and letters to sponsors. Two GAO staff independently reviewed these documents to determine if the Bureau documented and provided a clear rationale for each major decision point, comparing the documents against practices in the Bureau’s application evaluation process and federal standards for internal control. Using Bureau documents, we also calculated how much time it took for each project to move between each step and decision point and determined whether each project met its anticipated financial close date. We did not compare the amount of time it took for these projects to complete the application process to projects that received financing before DOT created the Bureau because the steps and decision points for the application process changed. However, we interviewed Bureau officials to understand the application evaluation process and the 10 projects we selected. We also drew on past GAO work and that of others to understand past findings and challenges for the financing programs before the Bureau was created. The consultant’s initial implementation plan for the Build America Bureau (Bureau)—created by the consultant while working with the Department of Transportation’s (DOT) internal committees—outlined an organizational structure with responsibilities and roles for its positions. Most positions resided in three offices that administer specific programs or provide technical assistance to sponsors. The Outreach and Project Development Office works to educate project sponsors about how they can best combine DOT’s financing and funding programs as well as innovative project delivery approaches. The implementation plan envisioned a director to manage the office, general project development lead positions to conduct outreach and provide assistance to sponsors on specific projects, and specialized project development lead positions with expertise in a particular area, such as rail or maritime, to help sponsors with more complex projects and to provide technical assistance to other sponsors and staff in the Bureau. The plan also envisioned best practices positions with expertise in public-private partnerships, transit-oriented development, or federal permitting. The Credit Programs Office administers the application processes for the Transportation Infrastructure Finance and Innovation Act (TIFIA) and Railroad Rehabilitation and Improvement Financing (RRIF) programs. The implementation plan envisioned a director to manage the office with the remaining positions split among three areas: underwriting positions to review and evaluate project applications, portfolio management positions to manage existing credit agreements, and risk management positions to evaluate project- specific risks, conduct audit activities, and carry out other risk and budget activities. Underwriting staff, for example, conduct an in-depth review of a project application that includes evaluating the plan of finance and feasibility of the revenue stream pledged to repay credit assistance or sufficiency of other pledged collateral. For the Infrastructure for Rebuilding America (INFRA) Grants Office, the structure envisioned a director and additional positions to administer the competitive grant program. Beyond these offices, the initial implementation plan proposed an Executive Director, as required by statute, to lead the Bureau’s work and positions to support the entire Bureau. The organizational structure also included additional positions to provide full-time legal support to the Bureau, which are housed in DOT’s Office of General Counsel. Our analysis—based on Bureau documents and discussions with Bureau officials—shows that when the Bureau opened in July 2016, 7 months after the Fixing America’s Surface Transportation Act (FAST Act) was enacted, it largely followed the envisioned structure. When the Bureau opened in July 2016, DOT detailed or transferred 29 staff to run the Bureau. Twenty-five of these staff filled positions in the Bureau’s three offices, and the four remaining staff filled positions in the Office of General Counsel that provided dedicated legal services to the Bureau. These staff came from other parts of DOT as follows: Federal Highway Administration (FHWA). DOT detailed 16 staff from FHWA’s TIFIA Joint Program Office to the Bureau, primarily to work in the Bureau’s Credit Programs Office. DOT also detailed three attorneys from FHWA’s Office of the Chief Counsel to the Office of General Counsel. Federal Railroad Administration (FRA). DOT transferred five staff from FRA to the Bureau’s Credit Programs Office. Federal Transit Administration (FTA). DOT transferred one attorney from this modal administration to the Office of General Counsel. Maritime Administration. DOT transferred one staff member from this modal administration to the Bureau to work in the Outreach and Project Development Office. Office of the Secretary of Transportation (OST). DOT transferred the remaining three staff from the Build America Transportation Investment Center to work in the Outreach and Project Development Office and in Bureau leadership and support roles. DOT, in opening the Bureau, did not fill any of the positions in the INFRA Grants Office. According to current and former DOT officials, DOT used staff in OST that administer another competitive grant funding program to administer the first round of INFRA grants, as noted above. This decision also allowed DOT to move quickly to make grants for the first round of funding. At the same time, DOT officials told us that no funding was provided specifically to administer the INFRA program, so hiring staff to fill those envisioned positions would have diverted resources from other Bureau priorities. In addition, one OST staff person who both worked on the INFRA program and managed the Private Activity Bonds (PAB) program continued to manage PAB after the Bureau took over administration of that program while staying in OST. DOT also decided to leverage other DOT offices and modal administrations to carry out some of the Bureau’s work. Bureau officials stated that this model allows the Bureau to realize efficiencies by using the expertise and support of existing DOT offices rather than duplicating this expertise and support. Figure 2 summarizes the DOT offices that the Bureau interacts with, based on our analysis of Bureau and DOT documents and interviews with Bureau officials. Support provided by other offices within OST: As noted above, the Office of Infrastructure Finance and Innovation administers the INFRA program, leveraging the experience and knowledge of staff in that office that administer another competitive grant program. The Bureau also coordinates with the Infrastructure Permitting Improvement Center on its FAST Act responsibilities related to environmental reviews and permitting. Expertise from DOT’s modal administrations: Designated liaisons in FRA, FTA, FHWA, and the Maritime Administration coordinate with the Bureau to help assess project readiness or identify issues on projects applying for financing, such as ongoing litigation or work remaining on environmental reviews. Liaisons are funded by their modal administration and told us that they spend anywhere from 10 to 75 percent of their time serving as a liaison to the Bureau. The FAST Act gave DOT authority to consolidate or eliminate offices and positions when creating the Bureau. When the Bureau opened in July 2016, DOT eliminated the FRA office that administered RRIF and the Build America Transportation Investment Center as staff and functions transferred to the Bureau. DOT also plans to eliminate the TIFIA Joint Program Office—the office that FHWA staff detailed to the Bureau formerly worked in. According to DOT officials, the FHWA staff from that office are fully integrated and working in the Bureau; however, these staff will remain FHWA employees until DOT completes actions to transfer funds and staff to the Bureau and formally eliminate that office. See below for more detail on the transfer of funds and staff. DOT officials said it was easier to eliminate FRA’s RRIF office than the TIFIA Joint Program Office because the RRIF office did not have dedicated administrative funding like the TIFIA office did and FRA employees worked on RRIF as one of several duties. After opening and operating the Bureau, DOT made minor changes to the initial organizational structure. According to DOT officials, the Bureau has evolved and changed since it began operations—as would occur for any new office—and its current structure differs in various ways from its initial structure. Based upon the Bureau’s early experience, it eliminated 7 proposed positions: 1 position providing legal support, 3 positions for outreach to sponsors, 2 for addressing risk management, and 1 for managing the Bureau’s portfolio. The Bureau decided to eliminate the outreach positions because despite earlier findings that DOT’s TIFIA and RRIF programs were underutilized, officials discovered that more sponsors than expected were interested in those financing programs. The Bureau also added 5 positions that had not been initially proposed: 2 underwriter positions and 3 positions that work across individual Bureau offices. These cross-Bureau positions handle several duties, including budget, human resources, and procurement issues for the Bureau, working closely with the Office of the Under Secretary for Policy. Funding for the Bureau currently comes from three sources, though DOT officials said they want to consolidate all funding for the Bureau in OST. First, 12 positions are funded through appropriations from general revenues to OST specifically for the Bureau. The President’s budget request has requested funding to support these 12 positions since fiscal year 2017. Second, 23 positions for the TIFIA program are funded through appropriations from the Highway Trust Fund. This funding cannot be used for positions that do not work on matters involving the TIFIA program, unless it is formally transferred to the Bureau, according to DOT. Third, the remaining 8 positions identified in the Bureau organizational chart are not carried out by Bureau employees. Instead, they are carried out by contractors and employees supported by other units of DOT, an approach that Bureau officials said is consistent with the missions of those other units and the Bureau. For instance, FHWA funds two positions in the Outreach and Project Development Office, outside of funding for TIFIA. DOT’s initial ability to transfer funds under the FAST Act to support the Bureau ended in December 2017; according to Bureau officials, this impaired the Bureau’s ability to finish steps to formally consolidate staff who are paid from the Highway Trust Fund. Due to how funds for TIFIA are authorized to FHWA in the FAST Act, DOT needed to receive transfer authority beyond December 2017 so that it could maintain its ability to pay Highway Trust-funded employees in future years after they are formally transferred to OST and paid from OST’s budget. In early 2018, DOT’s ability to transfer funds was extended in the fiscal year 2018 Consolidated Appropriations Act. DOT provided information to the appropriations committees on transferring funds and consolidating offices, as required in statute, and is awaiting a response from these committees. See figure 3 below for position titles, locations in the organization, and funding sources as of October 2018. The Bureau has had many vacant positions since it opened in July 2016, based on our interviews with current and former DOT officials and our review of Bureau documents. In the 6 months after the Bureau opened, DOT filled some positions, including competitively selecting an Executive Director. Then, in early 2017, DOT and other executive branch agencies were subject to a hiring freeze for about 3 months. However, in the time since the end of the hiring freeze, we found that the Bureau has continued to have many vacant positions (see fig. 4). The Executive Director position has been vacant since the person previously in that role stepped down in November 2017. DOT posted an unsuccessful announcement for this position in November 2017, followed by a second announcement in April 2018 that largely matched the earlier announcement. Beyond the Executive Director, the Bureau has had between 8 and 11 vacant positions in its organizational structure throughout 2018. Some positions, such as the Deputy Executive Director position, have never been filled. Other positions were filled but became vacant as staff left the Bureau for other opportunities. According to our analysis of Bureau documents, 16 of the 29 staff who were detailed or transferred to work in or for the Bureau when it was created in July 2016 remained in the Bureau as of August 2018. DOT and Bureau officials said that DOT did not want to fill vacant positions in the Bureau before filling the Executive Director position, as hiring is one of that position’s duties. Therefore, between fall 2017 and spring 2018, while the Executive Director position was vacant, DOT did not actively fill other vacancies, instead taking a “wait and see” approach, according to DOT and Bureau officials. However, in spring 2018, DOT and Bureau officials said they identified 5 critical vacancies to fill but were not able to provide a written document that laid out a hiring plan or sequence for filling the remaining positions. As of October 2018, Bureau officials said they had filled 5 positions and are in various stages of filling all the remaining vacant positions, either planning to write position descriptions, working with human resources to post jobs, or are in the hiring process. Finally, according to DOT and Bureau officials, DOT continues to use other OST staff to administer INFRA because of uncertainties related to the Bureau’s funding sources. However, DOT and Bureau officials said that many members of the team that oversees the INFRA evaluation process are also members of the Council on Credit and Finance, so the Bureau has an indirect role in the program. The Bureau has used detailees and contractors to fill vacant positions in the Outreach and Project Development Office. This office, unlike the Credit Programs Office, did not have an existing program or a large existing office to fill its positions from. Since July 2016, four detailees from other parts of DOT have filled positions in the Outreach and Project Development Office—the project development or specialized project development lead positions—on short, 4 to 6 month terms. Two of these detailees were reassigned permanently to these positions in the Bureau in summer 2018, and the other two detailees returned to their prior roles. Recently, the Bureau filled one additional such positon with a 2-year detailee from the Federal Aviation Administration. Finally, the Bureau filled two other positions with staff provided through an interagency agreement with the John A. Volpe National Transportation Systems Center effective through fiscal year 2020. In addition to the contact named above, Steve Cohen (Assistant Director); Joanie Lofgren (Analyst in Charge); Lauren Friedman; David Hooper; Lauren Lynch; Ned Malone; Malika Rice; Amy Rosewarne; and Michael Sweet made key contributions to this report.", "summary": "Constructing surface transportation projects can be long endeavors and involve multiple DOT offices. The 2015 Fixing America's Surface Transportation Act (FAST Act) required DOT to establish a finance bureau to consolidate certain funding and financing programs. The FAST Act further required that DOT improve procedures for evaluating applications for these programs—including providing a clear rationale for decisions and streamlining the process. The FAST Act also gave this finance bureau other responsibilities such as promoting best practices for innovative financing. In response, DOT opened the Build America Bureau in July 2016. The FAST Act included a provision for GAO to review the Bureau. This report assesses, among other things, (1) progress DOT made to establish the Bureau and carry out its responsibilities, (2) the Bureau's process for evaluating applications, and (3) whether the Bureau provided a clear rationale for decisions in that process. GAO reviewed federal laws and Bureau documents and interviewed DOT officials and selected stakeholders, including 28 project sponsors selected so projects varied by mode, cost, and outcome. The Department of Transportation (DOT) has taken initial steps to establish the Build America Bureau's (Bureau) organizational structure and to create a process to help the Bureau carry out some of its responsibilities since it was created in 2016. However, the Bureau lacks a plan to guide its ongoing and future efforts. Initial steps included creating a consolidated process to evaluate applications for three financing programs: Transportation Infrastructure Finance and Innovation Act (TIFIA), Railroad Rehabilitation and Improvement Financing (RRIF), and Private Activity Bonds (PAB). DOT largely based this consolidated process on prior practices used for individual programs but also sought to improve and streamline the process. For example, DOT formed a decision-making body that meets more frequently than a predecessor group to quickly address issues and to decide when to advance projects through the process. However, progress has been more limited in implementing other responsibilities, such as promoting best practices for innovative financing. While some of the lack of progress can be attributed to factors such as changes in leadership and staff, the Bureau lacks a plan with implementation goals and a timeline to guide its ongoing and future efforts and also lacks performance indicators to assess its progress. Without these tools, the Bureau may face difficulties prioritizing work to carry out other responsibilities and maintaining momentum throughout continued implementation efforts and any future changes in leadership and staff. While the Bureau has taken steps to improve and streamline the application evaluation process, it does not have a mechanism to assess how well the process works—including what is challenging and what works well. Project sponsors GAO interviewed had mixed views on the Bureau's application evaluation process and whether it was streamlined. Selected sponsors that applied for TIFIA and RRIF financing identified challenges with the process, including the length of the process and changes to requirements or terms for a loan. For example, sponsors said the Bureau took longer than it had estimated to procure external advisors to help conduct its evaluation of applications. According to the sponsors, such delays and uncertainty led to cost increases for two projects and construction delays for one project. Bureau officials noted that many factors outside the Bureau's control influence the length of the application evaluation process, such as changes to a project's scope and construction cost estimates. However, the Bureau has not taken steps, such as consistently soliciting feedback from sponsors, to assess how to further improve and streamline its process. Without taking such steps, the Bureau is missing an opportunity to further streamline the process and to ensure that any challenges do not discourage sponsors from seeking the Bureau's financing programs. GAO found that the Bureau provided a clear rationale for decisions to advance or approve projects in the TIFIA and RRIF programs but did not do so for the PAB program. While DOT did document the decisions made in each step of the application evaluation process for the PAB program, the lack of a documented rationale to support these decisions leaves that program open to questions about the integrity of its process, as it is not immediately clear how the Bureau determined that an application satisfied requirements and what information was used to support decisions that advanced projects. GAO is making five recommendations, including that the Bureau develop a plan to guide its efforts and assess ways to further improve the application evaluation process. DOT concurred with two but did not fully concur with three of the recommendations and provided no rationale. GAO continues to believe the recommendations are valid as discussed in the report.", "document_type": "gao"}
{"report": "Consistent with the discretion afforded by the APA, Regulations.gov and agency-specific comment websites use required and optional fields on comment forms to collect some identity information from commenters. In addition to the text of the comment, agencies may choose to collect identity information by requiring commenters to fill in other fields, such as name, address, and email address before they are able to submit a comment. Regardless of the fields required by the comment form, the selected agencies all accept anonymous comments in practice. Further, because the APA does not require agencies to authenticate submitted identity information, neither Regulations.gov nor the agency-specific comment websites contain mechanisms to check the validity of identity information that commenters submit through comment forms. Regulations.gov and agency-specific comment websites also collect some information about public users’ interaction with their websites through application event logs and proxy server logs, though the APA does not require agencies to collect or verify it as part of the rulemaking process. This information, which can include a public user’s Internet Protocol (IP) address, browser type and operating system, and the time and date of webpage visits, is collected separately from the comment submission process as part of routine information technology management of system security and performance, and cannot be reliably connected to specific comments. Seven of 10 selected agencies have documented some internal guidance associated with the identity of commenters during the three phases of the public comment process: intake, analysis, and response to comments. However, the focus and substance of this guidance varies by agency and phase of the comment process. As shown in table 1, for selected agencies that have guidance associated with the identity of commenters, the guidance most frequently relates to the comment intake or response to comment phases of the public comment process. The guidance for these phases addresses activities such as managing duplicate comments (those with identical or near-identical comment text but varied identity information) or referring to commenters in a final rule. Agencies are not required by the APA to develop internal guidance associated with the public comment process generally, or identity information specifically. Within the discretion afforded by the APA, the 10 selected agencies’ treatment of identity information during the comment intake, comment analysis, and response to comments phases of the public comment process varies. Selected agencies differ in how they treat identity information during the comment intake phase, particularly in terms of how they post duplicate comments, which can lead to identity information being inconsistently presented to public users of comment systems. With regard to the comment intake phase in particular, the variation in how agencies identify duplicate comments and post comments results in identity information being inconsistently presented on Regulations.gov or the agency-specific websites. Generally, officials told us that their agencies either (1) maintain all comments within the comment system or (2) maintain some duplicate comment records outside of the comment system, for instance, in email file archives. For example, according to officials of one participating agency—the Wage and Hour Division (WHD)—all duplicate comments are stored in Regulations.gov. Our analysis of WHD comments did not suggest that any comments were missing from Regulations.gov. However, in one example, almost 18,000 duplicate comments were included in attachments under one individual’s name in the comment title. While all of the comments are included within 10 separate attachments, none of the identity information included with these comments can be easily found without manually opening and searching all 10 attachments, most of which contain approximately 2,000 individual comments. Selected agencies’ treatment of identity information during the comment analysis phase also varies. Specifically, program offices with the responsibility for analyzing comments place varied importance on identity information during the analysis phase. Finally, all agencies draft a response to comments with their final rule, but the extent to which the agencies identify commenters or commenter types in their response also varies across the selected agencies. Our analysis of Regulations.gov and agency-specific comment websites shows that the varied comment posting practices of the 10 selected agencies are not always documented or clearly communicated to public users of the websites. In part to facilitate effective public participation in the rulemaking process, the E-Government Act of 2002 requires that all public comments and other materials associated with a given rulemaking should be made “publicly available online to the extent practicable.” Additionally, key practices for transparently reporting open government data state that federal government websites—like those used to facilitate the public comment process—should fully describe the data that are made available to the public, including by disclosing data sources and limitations. We found that the selected agencies we reviewed do not effectively communicate the limitations and inconsistencies in how they post identity information associated with public comments. As a result, public users of the comment websites lack information related to data availability and limitations that could affect their ability to use and make informed decisions about the comment data and effectively participate in the rulemaking process themselves. Public users of Regulations.gov seeking to submit a comment are provided with a blanket disclosure statement related to how their identity information may be disclosed, and are generally directed to individual agency websites for additional detail about submitting comments. While additional information is provided in the Privacy Notice, User Notice, and Privacy Impact Assessment for Regulations.gov, public users are not provided any further detail on Regulations.gov regarding what information, including identity information, they should expect to find in the comment data. Additionally, there is not enough information to help public users determine whether all of the individual comments and associated identity information are posted. Available resources on Regulations.gov direct public users to participating agencies’ websites for additional information about agency-specific review and posting policies. Seven of the eight participating agencies’ websites direct public users back to Regulations.gov and the Federal Register, either on webpages that are about the public comment process in general, or on pages containing information about specific NPRMs. Three of these participating agencies—the Environmental Protection Agency (EPA), Fish and Wildlife Service (FWS), and Food and Drug Administration (FDA)—do provide public users with information beyond directing them back to Regulations.gov or the Federal Register, but only FDA provides users with details about posting practices that are not also made available on Regulations.gov. The eighth participating agency—the Employee Benefits Security Administration (EBSA)—does not direct public users back to Regulations.gov, and instead re-creates all rulemaking materials for each NPRM on its own website, including individual links to each submitted comment. However, these links go directly to comment files, and do not link to Regulations.gov. While EBSA follows departmental guidance associated with posting duplicate comments, which allows some discretion in posting practices, the agency does not have a policy for how comments are posted to Regulations.gov or its own website. Further, in the examples we reviewed, the content of the NPRM-specific pages on EBSA’s website does not always match what is posted to Regulations.gov. Because participating agencies are not required to adhere to standardized posting practices, Regulations.gov directs public users to participating agency websites for additional information about posting practices and potential data limitations. However, these websites do not describe the limitations associated with the identity information contained in publicly posted comments. As allowed for under the APA, all of the participating agencies in our review vary in the way in which they post identity information associated with comments—particularly duplicate comments. However, the lack of accompanying disclosures may potentially lead users to assume, for example, that only one entity has weighed in on an issue when, actually, that comment represents 500 comments. Without better information about the posting process, the inconsistency in the way in which duplicate comments are presented to public users of Regulations.gov limits public users’ ability to explore and use the data and could lead users to draw inaccurate conclusions about the public comments that were submitted and how agencies considered them during the rulemaking process. Both nonparticipating agencies use comment systems other than Regulations.gov and follow standardized posting processes associated with public comments submitted to their respective comment systems, but the Securities and Exchange Commission (SEC) has not clearly communicated these practices to the public. Although it appears to users of the SEC website that the agency follows a consistent process for posting duplicate comments, at the time of our June 2019 report, this practice had not been documented or communicated to public users of its website. In contrast, FCC identifies its policies for posting comments and their associated identity information in a number of places on the FCC.gov website, and on its Electronic Comment Filing System (ECFS) web page within the general website. Regarding comments submitted to rulemaking proceedings through ECFS, public users are informed that all information submitted with comments, including identity information, will be made public. Our review of ECFS comment data did not identify discrepancies with this practice. Although the public comment process allows interested parties to state their views about prospective rules, the lack of communication with the public about the way in which agencies treat identity information during the posting process, particularly for duplicate comments, may inhibit users’ meaningful participation in the rulemaking process. While the APA does not include requirements for commenters to provide identity information, or for agency officials to include commenters’ identity as part of their consideration of comments, key practices for transparently reporting open government data state that federal government websites— like those used to facilitate the public comment process—should fully describe the data that are made available to the public, including by disclosing data sources and limitations. In our June 2019 report, we made eight recommendations. Specifically, we recommended that five of the selected agencies establish a policy for posting comments, and that those five agencies plus three others take action to more clearly communicate their policies for posting comments, particularly with regard to identity information and duplicate comments. The eight agencies generally agreed with our recommendations and identified actions they planned to take in response, such as developing policies for posting duplicate comments and communicating those in various ways to public users. Since issuing our June 2019 report, SEC has taken action that is responsive to the recommendation we made to it. Specifically, in September 2019, SEC issued a memorandum that reflects SEC’s internal policies for posting duplicate comments and associated identity information. In addition, SEC has communicated these policies to public users on the SEC.gov website by adding a disclaimer on the main comment posting page that describes how the agency posts comments. Chairmen Portman and Lankford, Ranking Members Carper and Sinema, and Members of the Subcommittees, this concludes my prepared remarks. I would be happy to answer any questions you may have at this time. For further information regarding this testimony, please contact Seto J. Bagdoyan, (202) 512-6722 or bagdoyans@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony are David Bruno (Assistant Director), Elizabeth Kowalewski (Analyst in Charge), and Dahlia Darwiche. Other individuals who also contributed to the report on which this testimony is based include Enyinnaya David Aja, Gretel Clarke, Lauren Kirkpatrick, James Murphy, Alexandria Palmer, Carl Ramirez, Shana Wallace, and April Yeaney. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Federal agencies publish on average 3,700 proposed rules yearly and are generally required to provide interested persons (commenters) an opportunity to comment on these rules. In recent years, some high-profile rulemakings have received extremely large numbers of comments, raising questions about how agencies manage the identity information associated with comments. While the APA does not require the disclosure of identifying information from a commenter, agencies may choose to collect this information. This testimony summarizes GAO's June 2019 report on public comment posting practices (GAO-19-483). In that report, GAO examined (1) the identity information collected by comment websites; (2) the guidance agencies have related to the identity of commenters; (3) how selected agencies treat identity information; and (4) the extent to which selected agencies clearly communicate their practices associated with identity information. The agencies were selected on the basis of the volume of public comments they received on rulemakings. For this testimony, GAO obtained updates on the status of recommendations made to the selected agencies. The Administrative Procedure Act (APA) governs the process by which many federal agencies develop and issue regulations, which includes the public comment process (see figure). In June 2019, GAO found that Regulations.gov and agency-specific comment websites collect some identity information—such as name, email, or address—from commenters who choose to provide it during the public comment process. The APA does not require commenters to disclose identity information when submitting comments. In addition, agencies have no obligation under the APA to verify the identity of such parties during the rulemaking process. GAO found in the June 2019 report that seven of 10 selected agencies have some internal guidance associated with the identity of commenters, but the substance varies. This reflects the differences in the way that the selected agencies handle commenter identity information internally. GAO also found that the selected agencies' practices for posting public comments to comment websites vary considerably, particularly for duplicate comments (identical or near-identical comment text but varied identity information). For example, one agency posts a single example of duplicate comments and indicates the total number of comments received, but only the example is available to public users of Regulations.gov. In contrast, other agencies post all comments individually. As a result, identity information submitted with comments is inconsistently presented on public websites. The APA allows agencies discretion in how they post comments, but GAO found that selected agencies do not clearly communicate their practices for how comments and identity information are posted. GAO's key practices for transparently reporting government data state that federal government websites should disclose data sources and limitations to help public users make informed decisions about how to use the data. If not, public users of the comment websites could reach inaccurate conclusions about who submitted a particular comment, or how many individuals commented on an issue. In June 2019, GAO made recommendations to eight of the selected agencies regarding implementing and communicating public comment posting policies. The agencies generally agreed with the recommendations and identified action they planned to take in response. Since the June 2019 report, one agency has implemented GAO's recommendation.", "document_type": "gao"}
{"report": "VA has purchased health care services from community providers since as early as 1945. In general, veterans may be eligible for community care when they are faced with long wait times or travel long distances for appointments at VA medical facilities, or when a VA medical facility is unable to provide certain specialty care services, such as cardiology or orthopedics. In general, community care services must be authorized in advance of when veterans access the care. Currently, there are several community care programs through which VA purchases hospital care and medical services for veterans, including the Choice Program. In implementing the VA MISSION Act, VA plans to consolidate four of its community care programs for veterans under the Veterans Community Care Program, which is expected to go into effect by June 2019. (See table 1.) VA also provides health care services to veterans and other eligible beneficiaries through community providers under additional benefit programs. These benefit programs include the Civilian Health and Medical Program of the Department of Veterans Affairs (CHAMPVA) and the Camp Lejeune Family Member Program, among others. After implementing the VA MISSION Act, VA will continue to operate the community care programs for other eligible beneficiaries, such as CHAMPVA and others, as it has historically done. Appendix I contains more information about VA’s community care programs. The amount of funding VA receives to provide its health care services is determined during the annual appropriations process. In preparation for the process, VA develops an estimate of the resources needed to provide its health care services—known as its health care budget estimate—for two fiscal years. This budget estimate is one step in a complex, multistep budget formulation process, which culminates in an appropriation request for VA health care that updates the earlier, advance appropriation request for the upcoming fiscal year and an advance appropriation request for the next fiscal year in the President’s annual budget request to Congress. VA’s health care budget estimate includes the total cost of providing health care services, including direct patient costs, as well as costs associated with management, administration, and maintenance of facilities. VA uses its Enrollee Health Care Projection Model (EHCPM) to estimate the majority of resources needed to meet the expected demand for health care services, and uses other methods for the remaining services. VA uses the EHCPM to make projections 3 and 4 years into the future for budget purposes based on data from the most recent fiscal year. For example, in 2017, VA used data from fiscal year 2016 to develop its health care budget estimate for the fiscal year 2019 request and advance appropriation request for fiscal year 2020. The EHCPM’s estimates are based on three basic components: (1) the projected number of veterans who will be enrolled in VA health care, (2) the projected quantity of health care services enrollees are expected to use, and (3) the projected unit cost of providing these services. Each component is subject to a number of complex adjustments to account for the characteristics of VA health care and the veterans who access VA’s health care services. (See fig. 1.) VA uses other methods to estimate resources needed for the remaining portion of its budget estimate. This portion of the budget includes the state home per diem program, CHAMPVA, and other health care programs for veterans and other eligible beneficiaries, as well as health- care-related initiatives proposed by the Secretary of Veterans Affairs or the President. (See app. II for more information about the other methods VA uses in developing its health care budget estimate.) VHA generally starts to develop a health care budget estimate approximately 10 months before the President submits the budget to Congress, which should occur no later than the first Monday in February. The budget estimate changes during the 10-month budget formulation process, in part, due to successively higher levels of review in VA and OMB before the President’s budget request is submitted to Congress. (See table 2.) The Secretary of Veterans Affairs considers the health care budget estimate developed by VHA when assessing resource requirements among competing interests within VA, and OMB considers overall resource needs and competing priorities of other agencies when deciding the level of funding requested for VA’s health care services. OMB passes back decisions, known as a “passback,” to VA and other agencies on their budget estimate, along with funding and policy proposals to be included in the President’s budget request. VA has an opportunity to appeal the passback decisions before OMB finalizes the President’s budget request. Concurrently, VA prepares a congressional budget justification that provides details supporting the policy and funding decisions in the President’s budget request. As of fiscal year 2017, VA primarily receives funding for all health care it provides or purchases through the following appropriation accounts: Medical Services: health care services provided to eligible veterans and other beneficiaries in VA facilities and non-VA facilities, among other things. Medical Community Care: health care services that VA authorizes for veterans and other beneficiaries to receive from community providers. Medical Support and Compliance: the administration of the medical, hospital, nursing home, domiciliary, supply, and research activities authorized under VA’s health care system, among other things. Medical Facilities: the operation and maintenance of VHA’s capital infrastructure, such as the costs associated with nonrecurring maintenance, leases, utilities, facility repair, laundry services, and groundskeeping, among other things. Separate from VA’s health care appropriation accounts, the Veterans Access, Choice, and Accountability Act of 2014 provided $10 billion in funding for the Choice Program, which was implemented in early fiscal year 2015 and authorized until funds were exhausted or through August 7, 2017, whichever occurred first. However, VA received additional authority and funding to maintain the Choice Program through June 6, 2019, when the new Veterans Community Care Program is expected to go into effect. VA expects that the new Veterans Community Care Program will be primarily funded through the Medical Community Care appropriation account. Our analysis of VA budget justification data shows that from fiscal year 2014 through fiscal year 2018, the total amount VA actually obligated for community care increased 82 percent, from $8.2 billion to $14.9 billion. Since VA implemented the Choice Program in fiscal year 2015, the share of VA’s obligations for community care relative to VA’s total obligations for health care services increased through fiscal year 2018, from about 14 to 19 percent of VA’s total obligations for health care services. By fiscal year 2021, VA estimates that the total amount obligated for community care will increase to $17.8 billion, an increase of about 20 percent from the $14.9 billion in actual obligations for fiscal year 2018. (See fig. 2.) As figure 2 shows, the largest increase in actual obligations for community care occurred from fiscal years 2015 through 2016, when they increased by $3.4 billion, from $8.9 billion to $12.3 billion. According to VA officials, this increase in obligations during this period reflected veterans’ expanded use of community care through the Choice Program, as more providers participated in the provider networks established by third-party administrators or entered into provider agreements with VA facilities. (Fig. 3 provides information on VA’s obligations for community care by the Choice Program and by other community care programs.) The increase in actual obligations for community care from fiscal year 2016 through fiscal year 2017 was also largely due to expanded use of community care through the Choice Program. VA officials attributed this increase to efforts to obligate as much of the available Choice Program funding as possible before the anticipated end of the Choice Program in August of 2017. From fiscal years 2017 through 2018, obligations for community care continued to increase, but the increase was partially due to greater use of other community care programs, according to VA officials. From fiscal years 2014 through 2018, the increases in total actual obligations for VA community care were driven largely by increases in obligations for outpatient and inpatient services. Over this time period, VA’s actual obligations for outpatient services increased by $2 billion, from $2.3 billion to $4.3 billion, and actual obligations for inpatient services increased by $818 million, from $1.8 billion to $2.7 billion. Figure 4 illustrates how outpatient and inpatient services accounted for most of VA’s total community care obligations for fiscal year 2018. VA estimated that from fiscal years 2019 through 2021, obligations for community care will increase to $17.8 billion, which VA officials said are attributable to the new eligibility criteria under the VA MISSION Act. The authority for the Choice Program ends June 6, 2019, after which the new Veterans Community Care Program—which consolidates VA’s community care programs under the VA MISSION Act—will be expected to begin. For comparison purposes, the largest increase in obligations for services provided at VA medical facilities is estimated to occur between fiscal years 2020 and 2021. VA officials said this increase is attributable, in part, to efforts related to hiring and telehealth in response to the eligibility criteria under the VA MISSION Act. Our analysis of VA data on authorizations for community care shows that the number of veterans authorized to use community care increased 41 percent from fiscal years 2014 through 2018. (See fig. 5.) The approximately 1.8 million veterans authorized to use community care in 2018 represented about 30 percent of all veterans accessing VA health care services that year (approximately 6.2 million veterans). By fiscal year 2021, VA officials told us that they estimate that at least 1.8 million veterans will still use community care. Our analysis of VA data also shows that after being authorized for care, veterans’ utilization of certain community care services increased from fiscal years 2014 through 2018. Over this time period, a number of outpatient services experienced increases of more than 200 percent in utilization, especially chiropractic visits (418 percent, from 143,000 to 743,000 visits), physical therapy visits (252 percent, from 857,000 to 3 million visits), and non-mental health related office visits (243 percent, from 651,000 to 2.2 million visits). In comparison, our analysis found relatively smaller increases in veteran utilization for certain inpatient services. For example, the utilization for surgical inpatient stays increased about 39 percent—from 253,000 to 352,000 bed days. VA first developed a separate budget estimate for community care to inform the President’s fiscal year 2017 budget request. Beginning with the President’s fiscal year 2018 budget request, VA updated its EHCPM to develop over 75 percent of its community care budget estimate and used other methods to develop the remainder. Subsequent changes were made to the community care budget estimates developed by the EHCPM for fiscal years 2018 and 2019 through successively higher levels of review in VA and OMB. VA first developed a separate budget estimate of the resources it would need for community care—as distinct from the care provided in VA medical facilities—in order to inform the President’s fiscal year 2017 budget request for VA. Prior to this fiscal year 2017 budget request, VA developed a single budget estimate of the resources needed to provide all VA health care services, regardless of whether these services were purchased from community providers or delivered in VA medical facilities, because all these services were to be funded through the same appropriation account. According to VA officials, at the time a separate community care appropriation account and budget estimate were unnecessary, because community care accounted for a relatively small portion of VA’s overall health care budget. However, once the medical community care appropriation account was established in fiscal year 2017, VA began developing a separate budget estimate for community care, as required by law. To develop its first estimate of the resources needed for community care for fiscal year 2017, VA made adjustments to existing estimates for total demand for care—both in VA medical facilities and community care combined—developed by the EHCPM. At the time, VA used the EHCPM to estimate the resources needed to provide VA health care services to veterans, including inpatient, outpatient, and long-term care. However, the EHCPM did not make separate estimates for community care and care provided at VA facilities; according to VA officials, VA adjusted the EHCPM estimates by assuming that for each service, the share of total utilization and costs devoted to community care would be the same as they had been in the most recently completed fiscal year. In addition, after this adjustment, VA made additional changes to the community care budget estimate, which resulted in a net increase of $2.5 billion. Nearly all of this increase reflected an anticipated impact of the expanded access under the Choice Program, according to VA officials. Overall, this approach accounted for about 75 percent of the $12.3 billion community care budget estimate that informed the President’s budget request for fiscal year 2017. To develop the remaining portion of its community care budget estimate, VA used methods other than the EHCPM that, according to VA officials, were used historically to develop estimates of the resources needed for the state home per diem program and benefit programs. For example, VA develops budget estimates for certain services under the state home per diem program by creating projections of the amount of care to be provided using information about the size and demographic characteristics of the enrolled veteran population and projections of the unit cost of providing one day of care using recent cost experience. According to VA officials, VA was able to continue using these other methods, because the services under these programs have been provided through community providers and not VA medical facilities. While methods for each program vary, in general, these methods are based on each program’s historical utilization and costs. (See app. II for additional information on the methods VA uses to develop the budget estimates for each of these community care programs.) Beginning with the President’s fiscal year 2018 budget request, VA updated its EHCPM directly to estimate most of the resources needed to purchase community care for veterans. Specifically, VA updated the EHCPM to estimate the amount of resources needed to purchase a set of more than 40 community care services that have accounted for over 75 percent of VA’s total community care budget estimates of $12.6 billion for fiscal year 2018 and $12.4 billion for fiscal year 2019. These health care services were grouped into seven service types and include outpatient care, inpatient care, and long-term care. (See app. III for a list of the health care services). Of these services, outpatient services typically accounted for the largest share of VA’s community care budget estimate. For the remainder of community care services—including services provided under the state home per diem program and benefit programs— VA did not use the EHCPM and instead continued to use the other methods it has historically used to develop budget estimates for these services. (See fig. 6.) VA made several changes to the EHCPM to develop most of its community care budget estimate. Historically, the EHCPM estimated resources needed to meet the total expected demand for VA health care—a combination of care provided in VA medical facilities and through community care programs. VA updated the EHCPM to determine the proportion of demand met by community care by projecting enrolled veterans’ expected utilization of community care and the expected costs of purchasing these services. In what follows, we describe five major changes made to the EHCPM allowing VA to estimate the budgetary resources needed for community care. 1. Reliance on community care services. The EHCPM has historically accounted for the extent to which enrolled veterans would be projected to obtain health care services through the VA as opposed to other health care programs or insurers—referred to as reliance on VA health care. VA updated the EHCPM so that it can further account for the extent to which enrolled veterans would be expected to use VA’s community care programs as opposed to using care in VA’s medical facilities. Each year, the EHCPM determines reliance on VA community care based on a combination of historical experience—or the extent to which community care was used in prior fiscal years— and on the projected impact of new VA policies and operational guidance. For example, for the fiscal year 2019 budget estimates, the EHCPM projected reliance on VA care to be about 38 percent, of which 14 percent would be met through community care. Thus, the EHCPM projected reliance on VA’s community care programs to be about 5.3 percent for all care enrolled veterans are projected to use in fiscal year 2019. 2. Accounting for difference in community providers’ efficiency delivering inpatient services. VA also updated the EHCPM so that community care utilization projections account for the fact that veterans receiving inpatient care through community providers generally have relatively shorter lengths of inpatient stays compared with veterans receiving care at VA medical facilities. According to officials from VA and its actuarial consultant, community providers on average have historically performed better than VA providers on national benchmarks measuring how well providers manage the length of inpatient stays, while not affecting quality of care. To account for this difference, VA uses an adjustment factor when projecting utilization of inpatient services based on potentially avoidable days of care for community providers. 3. Comparing projected utilization with actual utilization for community care services. VA developed an adjustment factor for the EHCPM’s utilization estimates to account specifically for the differences between projected utilization and actual utilization of community care for the most recently completed fiscal year of data. According to VA officials, the difference typically reflects utilization behavior among providers or patients that are difficult to estimate based solely on historical data—such as changes in local practice patterns (e.g., providers choosing to use magnetic resonance imaging versus x-rays). To account for this behavior, VA compares projected and actual utilization and creates an “actual-to-expected” adjustment factor for each health care service to account for the difference. 4. Projecting unit costs for community care services. VA updated the EHCPM so that it could estimate what are known as the unit costs of purchasing community care services for veterans. In general, the unit cost of a community care service comprises the payment made to the provider (known as direct patient costs), as well as the indirect costs associated with administration and overhead. Indirect costs include (1) the fees paid to the contractors for administrative responsibilities for the Choice Program, (2) VA billing and processing costs and care coordination costs associated community care programs, and (3) certain costs associated with the VA Central Office that support community care (e.g., the salaries for officials from the Office of Community Care and other VA Central Office officials). 5. Accounting for community care service complexity and inflation. VA made other changes to the EHCPM’s unit cost projections for community care. For example, VA updated the EHCPM so that it accounts for costs associated with changes in the complexity—that is, the level of resources required to deliver—of health care services VA purchases from community providers. Officials from VA and its actuarial consultant noted that more complex services require relatively more resources to deliver, such as more expensive equipment (e.g., magnetic resonance imaging); more provider time; or higher-cost providers, such as surgeons. Officials anticipate that most services that VA purchases in the community will increase in complexity, leading to higher projected unit-costs for community care. VA also updated the EHCPM so that its unit cost estimates for community care account for inflation in the cost of labor and equipment. VA’s community care budget estimates are reviewed at successively higher levels at VA and OMB to inform the President’s budget request for VA. VA identified several changes made during the review process to its estimates projected by the EHCPM for fiscal years 2018 and 2019; these changes were due to the availability of more current information related to utilization and costs, among other factors. For fiscal year 2018, changes resulted in a budget request for VA community care in the President’s budget request that was approximately $1 billion lower than VA’s original EHCPM budget estimate of $10.7 billion. These changes included the following: A $996 million decrease reflecting the availability of more current information showing that an anticipated increase in utilization due to the Choice Program was too high. A $600 million decrease reflecting the availability of more current information showing that overhead costs initially allocated to community care in the data used in the EHCPM were too high. A $180 million decrease accounting for VA’s implementation of a new law that reduces VHA’s use of community care for examinations determining veterans’ disability ratings. A $500 million increase accounting for a court ruling that affected veteran eligibility for reimbursement of emergency community care, which was expected to increase utilization. A $250 million increase reflecting the availability of more current information that indicated administrative costs for the Choice Program in the data used in the EHCPM were too low. For fiscal year 2019, changes resulted in a budget request for VA community care in the President’s budget request that was nearly $1 billion higher than VA’s original EHCPM budget estimate of $8.6 billion. These changes included the following: A $1.7 billion increase reflecting more current information indicating that community care administrative costs and the utilization levels in the data used in the EHCPM were too low. A $1 billion increase accounting for a delay in the timing of the implementation of community care network contracts. According to VA officials, this resulted in the continued use of reimbursement rates in community care that were higher than Medicare reimbursement rates. A $1.8 billion decrease that reflected VA’s implementation of a new policy that changed the timing of community care obligations from when a veteran is authorized to use community care to the when a claim for actual services is paid. Our analysis of data included in VA’s budget justifications shows that in fiscal years 2017 and 2018, VA obligated $1.2 billion and $2.2 billion more for community care than originally estimated at the time of the President’s budget requests for those years. In both years, VA’s actual obligations for both the Choice Program and other community care programs were higher than estimated. (See table 3.) According to VA officials, the higher-than-estimated obligations for the Choice Program for fiscal year 2017 were driven, in part, due to changes in Choice Program policies and a large increase in the cost per authorization for care. In the case of other community care programs, VA officials told us that the higher-than-estimated obligations for both fiscal years 2017 and 2018 were driven, in part, by local practice patterns (e.g., providers choosing to use magnetic resonance imaging versus x-rays) and the capacity of VA medical facilities to provide services. As discussed later in this report, VA also received and reallocated additional funding to purchase community care in fiscal years 2017 and 2018, which contributed to actual obligations being higher-than-estimated obligations. Our analysis of VA’s obligations by service type shows that in fiscal year 2017, VA’s higher-than-estimated obligations for community care were primarily for outpatient and inpatient services, as shown in table 4. In fiscal year 2018, the higher-than-estimated obligations for community care were primarily for outpatient services, while there was an overall decrease in obligations for inpatient services. (See table 5.) Additionally, for some service types, VA’s actual obligations were lower than estimated in fiscal years 2017 and 2018. To obligate $13.6 billion for community care in fiscal year 2017 and $14.9 billion in fiscal year 2018—amounts that were $1.2 billion and $2.2 billion higher, respectively, than what VA originally estimated for its budget request, and what VA received in its annual appropriation—VA requested and received additional Choice Program funding outside of the annual appropriations process. VA also reallocated funding from other sources, including unobligated funding from a prior fiscal year and collections, to pay for the other community care programs. Specifically, the $13.6 billion and $14.9 billion VA obligated for community care in fiscal years 2017 and 2018, respectively, came from the following sources: Choice Program. For both fiscal years, VA obligated from its remaining funding and prior-year recoveries from the previous fiscal years, and requested and received additional funding three times outside of the annual appropriations process. (Table 6 below summarizes the time frames during which VA requested and received additional appropriations for the Choice Program outside of the annual appropriations process for fiscal years 2017 and 2018.) Other community care programs. For both fiscal years, VA obligated from its annual appropriation and transferred a portion of its overall collections from its Medical Care Collections Fund to the medical community care account. In addition, for fiscal year 2018, VA used unobligated funding and prior-year recoveries from fiscal year 2017. We provided a draft of this product to VA and OMB for comment. VA provided technical comments, which we incorporated as appropriate. OMB had no comments. We are sending copies of this report to the Secretary of Veterans Affairs, the Director of the Office of Management and Budget, appropriate congressional committees, and other interested parties. This report is also available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or silass@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. While the majority of veterans utilizing Department of Veterans Affairs’ (VA) health care services receive care in VA-operated medical facilities, veterans may also obtain services from non-VA providers in the community—known as community care—through one of several community care programs aimed at helping to ensure that veterans receive timely and accessible care. In implementing the VA MISSION Act, VA plans to consolidate four of its community care programs for veterans—dialysis contracts, individually authorized care, the Patient- Centered Community Care Program, and the Veterans Choice Program— under the Veterans Community Care Program, which is expected to go into effect by June 2019. In addition, VA has several other community care programs that serve veterans and programs that provide health care services to other eligible beneficiaries, including a veteran’s spouse or dependent child. Dialysis contracts. When dialysis services—a life-saving medical procedure for patients with permanent kidney failure—are not feasibly available at VA medical facilities, veterans may be referred to one of VA’s contracted dialysis providers, and veterans may receive dialysis at local clinics on an outpatient basis, or at home (if the contractors offer home- based dialysis services). Individually authorized care. When a veteran cannot access a particular specialty care service from a VA medical facility—either because the service is not offered, the veteran would have to wait too long for an appointment, or the veteran would have to travel a long distance to a VA medical facility—VA medical facility staff may request an individual authorization for the veteran to obtain the service from a community provider who is willing to accept VA payment. Patient-Centered Community Care. VA contracted with two third-party administrators to develop regional networks of community providers of specialty care, mental health care, limited emergency care, and maternity and limited newborn care when such care is not feasibly available from a VA medical facility. To be eligible to obtain care from Patient-Centered Community Care providers, veterans must meet the same criteria that are required for individually authorized care. Veterans Choice Program. VA modified its Patient-Centered Community Care contracts with the two third-party administrators to implement the Veterans Choice Program. This program allows eligible veterans to obtain health care services from community providers if the veteran meets certain criteria, including when a veteran cannot receive care within 30 days from the veteran’s or physician’s preferred date, or face an unusual or excessive burden in traveling to a VA medical center. Agreements with federal partners and academic affiliates. When services are not available at VA medical facilities, VA may obtain specialty, inpatient, and outpatient health care services for veterans through different types of sharing agreements—those with other federal facilities (such as those operated by the Department of Defense and the Indian Health Service), those with Tribal Health Programs, and those with university-affiliated hospitals, medical schools, and practice groups (known as academic affiliates). Emergency care. When emergency community care is not preauthorized, VA may reimburse community providers for emergency care for eligible veterans for a condition related to a service-connected disability, and for eligible veterans for a condition not related to a service- connected disability. Foreign Medical Program. The Foreign Medical Program is VA’s health care benefits program for eligible veterans who are residing or traveling abroad and have a service-connected disability. State Home Per Diem Program. Under the State Home Per Diem Program, states provide care for eligible veterans in three different types of programs: nursing home, domiciliary, and adult day health care. Camp Lejeune Family Member Program. The Camp Lejeune Family Member Program is for family members of veterans that lived or served at U.S. Marine Corps Base Camp Lejeune, North Carolina, for no fewer than 30 days between January 1, 1957, and December 31, 1987, and were potentially exposed to drinking water contaminated with industrial solvents, benzene, and other chemicals. The program provides health care to veterans who served on active duty at Camp Lejeune and to reimburse eligible Camp Lejeune family members for health care costs related to one or more of 15 specified illnesses or medical conditions specified in law. Children of Women Vietnam Veterans Health Care Benefits Program. This program provides health care benefits to female Vietnam veterans’ birth children who the Veterans Benefits Administration has determined to have a covered birth defect. This program is not a comprehensive health care plan and only covers those services necessary for the treatment of a covered birth defect and associated medical conditions. Civilian Health and Medical Program of the Department of Veterans Affairs (CHAMPVA). CHAMPVA is a comprehensive health care program that provides health care coverage for spouses, children and primary caregivers of veterans who are permanently and totally disabled from a service-connected disability. CHAMPVA functions similarly to traditional health insurance, with most care in the program delivered using non-VA community providers. Spina Bifida Health Care Benefits Program. This program provides health care benefits to certain Korea and Vietnam veterans’ birth children who have been diagnosed with spina bifida. The Department of Veterans Affairs (VA) and its actuarial consultant use the Enrollee Health Care Projection Model to develop most of the department’s estimate of the resources needed to meet the expected demand for VA’s health care services. VA uses other methods to estimate the remaining resources needed. This remaining portion includes community care programs for veterans and other eligible beneficiaries, including the State Home Per Diem Program and the Civilian Health and Medical Program of the Department of Veterans Affairs (CHAMPVA). State Home Per Diem Program. This program pays per diem for state- provided care for eligible veterans in three different types of programs: domiciliary, nursing home, and adult day health care. For state home domiciliary and nursing care, categorized as institutional care, VA creates budget projections based on historical funding data. For state home adult day health care, categorized as non-institutional care, VA’s budget estimates are based on projections of the amount of care provided— which is known as workload—and the unit cost of providing a day of this care. VA projects the demand for non-institutional care services using information about the size and demographic characteristics of the enrolled veteran population. VA projects unit cost for non-institutional care services by calculating unit-cost increases observed from recent experience and then using this information to project future unit costs. VA multiplies the workload estimates, unit-cost estimates, and the number of days in the fiscal year to develop an estimate of the amount of resources needed for non-institutional care. CHAMPVA. CHAMPVA provides health care coverage for spouses and children of veterans who are permanently and totally disabled from a service-connected disability. CHAMPVA functions similarly to traditional health insurance—most care within CHAMPVA is delivered using non-VA community providers. Therefore, developing estimates of the resources needed for CHAMPVA requires factoring in utilization patterns and cost inflation that are generally outside of VA’s control. Budget estimates for CHAMPVA are developed using a formula that computes the predicted number of users and costs per-member per-year. VA works with its actuarial consultant to generate projections of CHAMPVA users that incorporate changes related to the population of disabled veterans and projections of expected increases and decreases in the CHAMPVA- eligible population. In addition, the actuarial consultant projects the costs per-member per-year, which is calculated by dividing the most current fiscal year data on total CHAMPVA expenditures by the number of actual users. Trends are then incorporated to predict the future costs per- member per-year, which is multiplied by projections of the number of CHAMPVA users to develop CHAMPVA budget estimates. Using its Enrollee Health Care Projection Model (EHCPM), the Department of Veterans Affairs (VA) developed estimates for 79 health care services—available in VA medical facilities or through community care—for the fiscal year 2019 President’s budget request. As shown in table 7, VA developed separate estimates for the 43 services that were available through community care. Some of these 43 services were only available through community care. These services were primarily long- term care, including nursing home care provided at community nursing homes, home hospice care, home respite care, homemaker or home health aid programs, and purchased skilled nursing care. The Department of Veterans Affairs (VA) and its actuarial consultant use the Enrollee Health Care Projection Model (EHCPM) to develop most of the department’s budget estimate to meet the expected demand for VA’s health care services. This estimate includes the services that VA purchases from non-VA community providers through its various community care programs, including the Veterans Choice Program (Choice Program). Based on our interviews with various VA officials, VA’s Office of Enrollment and Forecasting provided utilization and cost data from fiscal year 2016 community care claims from four different sources for use in the 2017 EHCPM, which was used to project the fiscal year 2019 budget estimate. (See fig. 7.) Specifically, the Office of Enrollment and Forecasting—which is responsible for compiling the claims data used in the EHCPM—obtained community care claims data, including Choice Program claims, from VA’s Fee Basis Claims System. In addition, the Office of Enrollment and Forecasting worked with VA’s Allocation Resource Center to gather additional utilization and cost data from Choice Program claims processed outside the Fee Basis Claims System, and other data needed for the 2017 EHCPM. Specifically, the Allocation Resource Center compiled claims data for those Choice Program claims paid through expedited payments. The Allocation Resource Center also pulled data on dual eligible veterans, from the Department of Defense’s Medical Data Repository, and indirect costs associated community care claims (for example, costs associated with care coordination or claims processing) from VA’s Managerial Cost Accounting system. In addition to the contact named above, Rashmi Agarwal (Assistant Director), Aaron Holling (Analyst-in-Charge), Chad Clady, and Kate Tussey made key contributions to this report. Also contributing were Krister Friday, Jacquelyn Hamilton, and Muriel Brown. Veterans Choice Program: Further Improvements Needed to Help Ensure Timely Payments to Community Providers. GAO-18-671. Washington, D.C.: September 28, 2018. Veterans Choice Program: Improvements Needed to Address Access- Related Challenges as VA Plans Consolidation of its Community Care Programs. GAO-18-281. Washington, D.C.: June 4, 2018. VA’s Health Care Budget: In Response to a Projected Funding Gap in Fiscal Year 2015, VA Has Made Efforts to Better Manage Future Budgets. GAO-16-584. Washington, D.C.: June 3, 2016. Veterans’ Health Care: Proper Plan Needed to Modernize System for Paying Community Providers. GAO-16-353. Washington, D.C.: May 11, 2016. Veterans’ Health Care Budget: Improvements Made, but Additional Actions Needed to Address Problems Related to Estimates Supporting President’s Request. GAO-13-715. Washington, D.C.: August 8, 2013. Veterans’ Health Care: Improvements Needed to Ensure That Budget Estimates Are Reliable and That Spending for Facility Maintenance Is Consistent with Priorities. GAO-13-220. Washington, D.C.: February 22, 2013. Veterans’ Health Care Budget: Better Labeling of Services and More Detailed Information Could Improve the Congressional Budget Justification. GAO-12-908. Washington, D.C.: September 18, 2012. Veterans’ Health Care Budget: Transparency and Reliability of Some Estimates Supporting President’s Request Could Be Improved. GAO-12-689. Washington, D.C.: June 11, 2012. VA Health Care: Estimates of Available Budget Resources Compared with Actual Amounts. GAO-12-383R. Washington, D.C.: March 30, 2012. VA Health Care: Methodology for Estimating and Process for Tracking Savings Need Improvement. GAO-12-305. Washington, D.C.: February 27, 2012. Veterans’ Health Care Budget Estimate: Changes Were Made in Developing the President’s Budget Request for Fiscal Years 2012 and 2013. GAO-11-622. Washington, D.C.: June 14, 2011. Veterans’ Health Care: VA Uses a Projection Model to Develop Most of Its Health Care Budget Estimate to Inform the President’s Budget Request. GAO-11-205. Washington, D.C.: January 31, 2011. VA Health Care: Challenges in Budget Formulation and Issues Surrounding the Proposal for Advance Appropriations. GAO-09-664T. Washington, D.C.: April 29, 2009. VA Health Care: Challenges in Budget Formulation and Execution. GAO-09-459T. Washington, D.C.: March 12, 2009. VA Health Care: Long-Term Care Strategic Planning and Budgeting Need Improvement. GAO-09-145. Washington, D.C.: January 23, 2009. VA Health Care: Budget Formulation and Reporting on Budget Execution Need Improvement. GAO-06-958. Washington, D.C.: September 20, 2006.", "summary": "VA continues to focus on the use of community care to address challenges with veterans' access to health care services at VA medical facilities. In fiscal year 2019, VA plans to consolidate the Veterans Choice Program and several other community care programs under a single new Veterans Community Care Program. GAO and others have previously reported on past challenges VA has faced regarding the reliability, transparency, and consistency of its budget estimates for health care. GAO was asked to review VA's use of community care and efforts to develop budget estimates for this care. This report describes (1) trends in obligations for and utilization of VA's community care programs since fiscal year 2014, (2) how VA develops its community care budget estimate and any subsequent changes made to this estimate, and (3) how VA's actual obligations for community care compared with estimated obligations for fiscal years 2017 and 2018. GAO reviewed actual obligation and utilization data for fiscal years 2014 through 2018, as well as estimated obligations for fiscal years 2019 through 2021. GAO also reviewed available VA documentation on the methods and data used to develop VA's community care budget estimate that informed the President's budget request for fiscal years 2017 through 2019. GAO also interviewed VA officials and contractors responsible for developing these estimates, and OMB staff responsible for the federal budget. VA and OMB reviewed a draft of this report. VA's technical comments were incorporated as appropriate. To help ensure that veterans are provided timely and accessible health care services, the Department of Veterans Affairs (VA) may purchase care from non-VA providers, known as community care. VA obligated $14.9 billion for community care in fiscal year 2018, an increase of $6.7 billion (about 82 percent) since fiscal year 2014. The number of veterans authorized to use community care increased from 1.3 million to 1.8 million during this period. By fiscal year 2021, VA estimated obligations to increase to $17.8 billion, and officials estimate at least 1.8 million veterans will continue to use this care. Note: VA estimated obligations for fiscal year 2019 to reflect $1.8 billion in anticipated savings as a result of a VA policy change regarding the timing of certain community care obligations. VA uses a projection model to estimate the majority of resources needed to provide health care services. Beginning with the President's fiscal year 2018 budget request, VA updated its model to estimate the resources needed to purchase over 40 community care services accounting for over 75 percent of VA's community care budget estimate. These services include outpatient and inpatient care, among others. For the remainder of its community care budget estimate, which includes nursing care in state-operated homes, VA uses other methods based on historical utilization. VA's budget estimate is successively reviewed at VA and the Office of Management and Budget (OMB) to inform the President's budget request. VA identified several changes made during the review process to its budget estimate for fiscal years 2018 and 2019 to reflect more current information related to utilization and costs, among other factors. VA's actual obligations for community care for fiscal years 2017 and 2018 were $1.2 billion and $2.2 billion higher, respectively, than originally estimated. According to VA officials, this occurred for several reasons, including policy changes and increased costs for the Veterans Choice Program. To support higher obligations, VA requested and received additional funding for the Veterans Choice Program outside the annual appropriations process and used other funding sources, such as unobligated amounts from prior fiscal years.", "document_type": "gao"}
{"report": "Promoting respect for human rights is a goal of U.S. foreign policy. The United States considers the advancement of human rights when providing security assistance to foreign countries. Providing training on human rights issues and international humanitarian law to foreign security forces can further U.S. credibility and interests. For example, such training could help maintain local populations’ cooperation with U.S. security efforts by curbing potential abuses by partner country forces. Human rights abuses by U.S.–backed forces can damage the local population’s support for the United States’ strategic aims, according to guidance from the U.S. Army. The United States provides military equipment and training, including human rights training, to partner countries through a variety of security cooperation and assistance programs authorized by statutes, some of which are codified within Title 10 and Title 22 of the U.S. Code. Human rights training is incorporated into broader security cooperation and assistance efforts. DOD and State share responsibility for developing policy for, managing, and implementing human rights training. Title 10 programs are generally overseen by DOD. Title 22 programs primarily fall under State. According to DOD and State officials, most Title 22 human rights training is implemented by DOD. DOD integrates human rights concepts into various types of training and assistance, including “train and equip” programs and defense institution building. Train and equip programs provide training, equipment, and small–scale military construction activities intended to build the capacity of partner nations’ military forces. Defense institution building activities are security assistance programs intended to empower partner nation defense institutions to establish or re-orient their policies and structures to make their defense sector more accountable, effective, and responsive to civilian control, among other things. Some of the authorities under which DOD and State provide human rights training to partner countries require such training when security assistance is provided. For example, one of the more recent and significant changes to security assistance legislation was the 2017 NDAA, which enacted a new chapter in Title 10 of the U.S. Code containing authorities related to security cooperation. Among other things, the 2017 NDAA replaced multiple capacity building authorities with a new statute codified at 10 U.S.C. § 333 (Section 333). All Section 333 programs are required to include elements that promote observance of and respect for human rights and fundamental freedoms, rule of law, and the law of armed conflict, as well as respect for civilian control of the military. Prior to the 2017 NDAA, a similar requirement was mandated for security assistance delivered under the Global Train and Equip program (then codified at 10 U.S.C. § 2282), which required that U.S. assistance pursuant to this authority include “elements to promote observance of and respect for human rights and fundamental freedoms and respect for legitimate civilian authority.” Section 333 covers a greater range of security assistance programs—for example, counternarcotics assistance—than did Section 2282. Other authorities include human rights considerations in their authorizing language. For example, in 1976, Congress established the International Military Education and Training (IMET) program codified within Title 22. The program provides education and training to foreign military personnel with the objectives of professionalizing military forces and increasing respect for democratic values and human rights. In 1990, Congress expanded the objectives of the IMET program to include fostering greater understanding of and respect for civilian control of the military, contributing to responsible defense resource management, and improving military justice systems and procedures in accordance with internationally recognized human rights. State and DOD refer to the expanded IMET objectives as Expanded IMET (E-IMET). Table 1 lists key authorities through which DOD and State provide human rights training to foreign security forces. In addition to human rights training, U.S. agencies consider human rights records when providing certain assistance. The Foreign Assistance Act of 1961, as amended, prohibits assistance to a unit of a foreign government’s security forces if the Secretary of State has credible information that such unit has committed a gross violation of human rights. DOD–funded training programs are covered by a similar provision. These requirements are commonly referred to as Leahy laws. As we have previously reported, these laws and the corresponding policies developed to enforce and supplement these laws are intended to leverage U.S. assistance to encourage foreign governments to prevent their security forces from committing human rights violations and to hold their forces accountable when violations occur. To address requirements under both the State and DOD Leahy laws, State has established a process for vetting potential recipients of U.S. security assistance training. State’s Bureau of Democracy, Human Rights, and Labor (DRL) is responsible for overseeing this vetting process and for developing human rights vetting policies, among other duties. DOD incorporates human rights training as part of a wide range of assistance programs that involve a number of DOD entities in different capacities. (See table 2). State incorporates rule of law assistance and human rights training as part of a wide range of assistance programs that involve a number of State entities in different capacities. (See table 3). DOD operates a number of education facilities that provide training to foreign security forces and many include human rights–related material in their curriculum. However, there are a few training providers that deliver the majority of human rights training through courses explicitly focused on such topics as well as in courses and residential programs that include related material. In addition, State provides some human rights training through the International Law Enforcement Academies (ILEA). Defense Institute of International Legal Studies (DIILS): DIILS is housed under DSCA and is DOD’s lead resource for providing legal education and rule of law engagement training to foreign military personnel and civilian defense officials. DIILS delivers its training primarily through either in-residence courses—for which members of foreign security forces attend trainings at the DIILS campus—or through mobile education training that is delivered to foreign military forces overseas. DIILS provides three types of training: (1) core rule of law training in the United States and abroad, (2) defense institution building, and (3) mandated human rights training delivered under Section 333. DIILS is the only institute to provide the mandated human rights training delivered under Section 333. DOD officials said there are no plans for other facilities to be certified to meet these training requirements. Mandated Human Rights Training Provided by DIILS: In response to the increased demand for mandated human rights training, DIILS created a three–tiered training model to deliver mandated human rights training, according to DIILS officials, who also noted that DIILS is in the early stages of applying the model. The three–tiered training model categorizes mandated human rights training according to basic, intermediate, and advanced trainings. Basic training includes a 2-hour block of scripted coursework which is dedicated to general topics covering human rights and is appropriate when providing training to military units who are not dealing with a combat environment, for example. Military officials without legal training or nonattorney civilian personnel—including contractors— may conduct this training. Intermediate and advanced training is typically 8 or 16 hours of training, respectively, and instruction is provided by DIILS staff and other military attorneys. According to DIILS officials, each intermediate or advanced training is intended to be tailored for the recipient military unit based on an assessment of its duties and the lethality of any equipment provided through the security assistance. Western Hemisphere Institute for Security Cooperation (WHINSEC): WHINSEC, also operated by DOD, provides professional education and training, including human rights training, for military and law enforcement personnel from countries in the Western Hemisphere. The Institute’s Center for Human Rights and Democracy promotes human rights education and training through international programs and partnerships. Curriculum developed by the Center includes topics such as the lawful use of lethal force, due process under international human rights law, and violence against women and vulnerable groups. Examples of WHINSEC’s Human Rights Training: To meet its statutory requirement to provide human rights training, WHINSEC provides a mandatory, 10-hour training on human rights for every student. This training covers five objectives: (1) human rights, (2) the rule of law, (3) due process (4) civilian control of the military, and (5) the role of the military in a democratic society. Additionally, WHINSEC students are required to take an ethics course that builds on the material covered in the human rights and democracy classes. WHINSEC also includes human rights–related material in a number of other courses. For example, the Counter Transnational Threats course focuses on threat interdiction activities using simulated exercises and scenarios. WHINSEC officials explained that one such scenario involves students conducting a simulated raid of a drug lab. (See fig. 1). During the exercise, students encounter armed and unarmed criminals, along with civilians. The simulation is intended to create real–world human rights scenarios for students to assess and apply lessons learned from classroom–based human rights training. Center for Civil–Military Relations (CCMR): CCMR is a DOD organization within the Naval Postgraduate School. CCMR was designed to support the goals of E-IMET and strengthen civil–military relationships through a variety of education and training programs. Additionally, CCMR focuses on defense institution building activities provided under DOD’s Title 10 authority. Like DIILS, CCMR delivers in-residence programs and mobile education training. Examples of CCMR’s Human Rights Training: CCMR officials said that human rights–related material is included in many CCMR programs, although it is not always an explicit focus. For example, although the Maritime Security Program does not explicitly focus on human rights, CCMR staff said that human rights–related topics are integrated into various aspects of the program. One of the program’s modules focuses on how to apply the appropriate use of force when enforcing international and maritime law. CCMR staff said they use practical scenarios to prompt discussion among classroom participants on techniques to avoid use of lethal force. Participants might discuss how to respond if a potential suicide vessel is approaching a ship, including the use of barriers or other deterrents to prevent potential terrorist activity without use of lethal force. Additional DOD Training Providers: A number of other DOD facilities provide training to eligible foreign security forces that includes human rights–related material. These facilities include: Regional Centers: DOD operates five regional centers of strategic studies, whose main purpose is to engage senior leaders in partner countries. A common topic taught at Regional Centers includes civil– military relations, which generally contains information related to human rights. Judge Advocate General (JAG) schools: JAG schools train students on the rules of armed conflict and international humanitarian law; international students may attend these schools, according to DOD officials. Service War Colleges: The service war colleges educate representatives of foreign security forces at a general level about U.S. laws and policies. Human rights–related material may be included, although DOD officials acknowledged such material is peripheral to the main mission. Defense Institute of Security Cooperation Studies (DISCS): International partners who are interested in Foreign Military Sales management participate in human rights training at DISCS. According to DOD officials, DISCS trains hundreds of foreign partners each year on military sales. State’s Bureau of International Narcotics and Law Enforcement Affairs (INL): State INL funds human rights–related training that is delivered by ILEAs. The ILEAs are a global network of training centers with a mission to support emerging democracies; help promote U.S. interests through international cooperation; and promote social, political, and economic stability by combating crime. According to State, this mission is met through strengthening the rule of law and stressing respect for human dignity in law enforcement. ILEAs represent a major component of training provided to foreign law enforcement entities, but do not represent all human rights–related law enforcement training supported by State. DOD was unable to provide aggregate data on the extent of human rights training for foreign security forces. According to agency officials, DOD does not systematically track all human rights training in DOD systems. As a result, DOD officials noted they were unaware of the full scope of the agency’s human rights training. DOD officials said it is challenging to track human rights training because many courses and training activities might include human rights content. DOD training activities are tracked in the Security Cooperation Training Management System (SC-TMS). However, the tracking is focused on the training overall rather than on any one component of the training conducted, such as human rights. For example, a course at a Regional Center might include human rights– related topics in a civil–military relations class but DOD is not able to identify such a course in SC-TMS or elsewhere as one that could be considered human rights training. DOD officials noted that while DOD is not required to track all human rights training, DSCA and DIILS have systems in place intended to track the provision of human rights training mandated by Section 333, as described below. DSCA uses a case management system to track the mandated human rights training that DIILS provides under Section 333. However, limitations in the implementation of this system have led to questions about the completeness of the data. The case management system is used across DOD to track and manage a range of security assistance programs, in addition to DIILS training. The system is designed so that the implementing entity enters information into the case management system about the training or other security assistance programming provided. However, DOD has not designated DIILS as an implementing agency with authority to enter or edit data in the case management system. As a result, for many years DIILS has relied on a different entity to enter human rights training data into the system. DIILS officials said the U.S. Navy’s agent for international education and training acted as the implementing agency and entered data in the system for DIILS. Due to DIILS’ inability to enter data or make changes in the case management system, DIILS officials told us they have been unable to ensure that data on DIILS training are properly entered. In addition, although DSCA is the DOD entity with oversight responsibilities for ensuring that Section 333 human rights training is provided as appropriate, DSCA officials acknowledged that they did not consistently take steps to monitor the accuracy and completeness of data on the DIILS–provided Section 333 human rights training. DSCA officials said that most of the DIILS trainings likely were entered into DOD’s data system because policy and procedures for capturing training records require it, such as the requirements spelled out in DOD’s Security Assistance Management Manual. However, DSCA officials said they do not have assurance that all trainings were entered as a matter of practice because they lack a process to regularly review whether the training data were captured as required. DOD officials said as of fiscal year 2019, DSCA and DIILS are taking steps to enable DIILS to enter human rights training data directly into the case management system as an implementing agency, but this is still an ongoing effort and not yet operational. In addition, as part of broader changes implemented in 2019 related to how DIILS is funded, the Navy agent is no longer entering information into the case management system about training DIILS provides under Section 333. In the meantime, DIILS continues to track the provision of training using an internal spreadsheet, according to officials, and plans to enter training data into the case management system when they get access as an implementing agency. Federal standards for internal control state that management should use quality information and design appropriate types of control activities in the entity’s information systems to achieve objectives and ensure quality external reporting. In the case of human rights training, DOD officials acknowledged that they do not have a process to ensure that information on mandated human rights training is systematically and accurately entered into its tracking systems. Without such a process, DOD is limited in its ability to monitor compliance with the statutory requirement that Section 333 assistance include a human rights training component. DOD tracks and reports funding for mandated human rights training at a global level, but not by country and program, although DOD is taking steps to do so. DSCA has published periodic reports that include global funding information for Section 333 activities, including the mandated human rights training. In 2016, Congress required the Director of DSCA to publish quarterly monitoring reports on the status of funding allocated for Section 333 activities. DSCA published three quarterly monitoring reports in fiscal year 2018, which identified the amount of unobligated funds, disbursements, and unliquidated obligations for Section 333 activities. According to the monitoring report from the third quarter of fiscal year 2018, year-to-date unobligated funds for human rights training totaled over $2 million dollars, disbursements totaled about $17,000, and unliquidated obligations totaled about $200,000. The funding data for human rights training is generally reported globally in these reports, not by a specific program or country. DOD could not provide the information we requested on funds obligated and disbursed for mandated human rights training, by program and country, for fiscal years 2015 through 2018. DSCA officials said they could not provide these data because it was challenging to pull this type of information from their systems in a usable way. Further, DOD officials noted that their previous accounting system made it challenging to obtain funding data easily. DSCA and DIILS transitioned to a new accounting system in 2017 which, according to DSCA officials, was expected to provide more detailed information on the status of funding for human rights training. However, DOD officials said that the transition to the new accounting system introduced errors in the data and DIILS staff are still working through a learning curve in adopting the new system. Under the new accounting system, DIILS is to enter information using a unique program and task- naming convention. DSCA officials said the new accounting system, when fully implemented, is expected to allow both DSCA and DIILS to track funds according to the specific recipient country and Section 333 security assistance program, which would better enable DOD and others to effectively monitor the status of funds dedicated to these efforts. State officials said they rely on DOD to track funding and information on the Title 22 authorities that DOD implements, including IMET, which State officials said is its most substantial source of human rights–related training for foreign military forces. DOD provided information on the funding for certified E-IMET courses in recent years. However, according to DOD officials, not all E-IMET courses are related to human rights. State INL maintains data on human rights–related training delivered by ILEAs, which is a major component of training provided to foreign law enforcement entities. In September 2018, we reported that while INL collects data for certain types of police training, such as training provided through the ILEA program, they do not have reliable information readily available on police trained through INL–funded projects. We recommended that State develop and implement a process to collect more reliable data on the number of police trained in El Salvador, Guatemala, and Honduras, the geographic focus of that review. State concurred with our recommendation and stated that it is in the process of developing specific indicators related to police training. According to our review of State data on human rights–related training delivered by ILEAs, State supported human rights training for over 5,400 law enforcement personnel from over 100 countries at ILEAs from fiscal years 2015 through 2017. (See fig. 2.) State identified 31 trainings provided by ILEAs that included human rights topics. (See table 4). According to State, the course that received the most funding—Law Enforcement and Leadership Development—is not expressly focused on human rights but is a 6-week long course that includes human rights concepts in different modules. State provided approximately $34.4 million for such training to foreign law enforcement entities at ILEAS from fiscal years 2015 through 2017. (See fig. 3.) Although officials at both agencies identified examples of past monitoring and evaluation (M&E)–related efforts for security assistance programs, DOD and State officials acknowledged that they have not assessed the effectiveness of human rights training for foreign security forces provided as part of such programs. DOD. DOD officials cited student surveys and after-action reports—which are summaries of the training events, training outcomes, challenges encountered, and further actions to be taken that are prepared by course facilitators—as examples of M&E–related efforts: At DIILS, course facilitators use surveys to solicit student feedback on courses and on the relevance of the course materials. They also use after-action reports, which, according to officials, provide continuity and capture lessons learned from human rights training in partner countries for DIILS facilitators who will be traveling to those countries in the future. At CCMR, according to CCMR officials, training facilitators prepare after-action reports for each course that involves human rights content. They also solicit input from the security cooperation officers in the country where the training took place. At the U.S. Africa Command, officials also said that they prepare after-action reports on DIILS–provided mandated human rights training, which they share with DIILS. Officials said these reports often discuss improvements needed with regard to logistics planning for human right training that DOD provides in African countries. State. Examples of related M&E efforts that State has conducted include a multi-year survey of IMET and evaluations of some security assistance programs. For example, State and DOD funded a survey of IMET graduates which DOD entities conducted and covered the period from 2007 through 2014. The multi-year survey measured, among other things, if graduates reported an improved understanding of internationally recognized human rights. According to DOD officials, DOD is beginning to develop a new M&E approach for DOD’s security assistance programs. However, DOD has not established a timeline for evaluating the effectiveness of human rights training for foreign security forces that is often included as part of such assistance. The 2017 NDAA, enacted in December 2016, requires DOD to conduct assessment, monitoring, and evaluation of its security assistance programs and activities. The steps DOD is taking to implement the 2017 NDAA M&E requirements include: Policy guidance: DOD issued Instruction 5132.14: Assessment, Monitoring, and Evaluation Policy for the Security Cooperation Enterprise in January 2017. The instruction states that M&E will foster accurate and transparent reporting to key stakeholders on the outcomes and sustainability of security cooperation and improve returns on DOD security cooperation investments. The new M&E requirements are intended to include centralized, independent, and rigorous evaluations of significant security cooperation initiatives to examine their relevance, effectiveness, and sustainability, among other things. DOD officials said that they planned to develop additional guidance to meet the mandated M&E requirements for security assistance, which includes human rights training. Security assistance guidelines: Based on new security assistance guidelines, DOD developed templates for documents that combatant commands are required to complete when planning security assistance activities. These templates for initial assessment and initiative design documents (including for rule of law and human rights training) incorporate M&E into design and planning of security assistance programs and activities. Geographic combatant commands are required to submit these documents to DSCA for projects that are developed in fiscal year 2019 and will be implemented beginning in fiscal year 2020. Draft evaluation agenda: In 2018, DOD prepared a draft evaluation agenda which outlines notional timeframes for evaluations. However, DOD officials could not specify when they plan to finalize the agenda, and as of April 2019 could not tell us when DOD planned to begin monitoring and evaluating human rights training for foreign security forces because they have not developed a timeline for doing so. According to DOD officials, DOD is in the initial phase of developing its overall approach to monitoring and evaluating security assistance, of which human rights training is a small part. The 2019 NDAA, enacted in 2018, requires, as a condition for expending 50 percent of DOD operations and maintenance funds made available for Section 333 assistance, that DOD establish a written plan describing, among other things, evaluation activities planned for security assistance programs for fiscal year 2019. In addition, according to the Office of Management and Budget’s monitoring and evaluation guidelines for the federal government entities providing foreign assistance, agencies should establish annual monitoring and evaluation objectives and timetables to plan and manage the process of monitoring, evaluating, analyzing progress, and applying learning toward achieving results. Developing a timeline for implementing its activities to monitor and evaluate the effectiveness of human rights training, which could be done as part of DOD’s monitoring and evaluation of its broader security assistance efforts, would provide greater assurance that DOD will complete M&E requirements. According to State officials, they have not established a plan, with a clear timeline, for evaluating the effectiveness of human rights training provided as part of IMET. Officials from State’s Bureau of Political–Military Affairs (PM) acknowledged that State’s responsibilities for IMET include M&E of the program. According to these officials, PM is in the initial phase of developing M&E of its security assistance programs, including IMET. They stated that for this reason PM is not currently planning to evaluate human rights training provided under IMET. Although DOD implements IMET, PM has overall responsibility for the program. According to State’s January 2018 Guidance for the Design, Monitoring and Evaluation Policy at the Department of State, it is essential that bureaus and independent offices have comprehensive plans for monitoring and evaluating all their programs and projects, and the plans should include, among other things, an implementation schedule. An M&E plan with a clear timeline for human rights training provided under IMET will better position State and DOD to determine the effectiveness of a significant component of U.S. human rights training for foreign militaries and identify areas for improvement. Additionally, an evaluation of the effectiveness of the human rights training would provide other important stakeholders, including Congress, with evidence to better inform decisions about U.S.–funded human rights training provided under IMET. Such an evaluation could be done as part of State’s broader effort to evaluate IMET. According to DOD and State officials and outside experts we interviewed, there are several challenges to achieving human rights objectives—such as a decrease in human rights violations or promoting greater respect for human rights—through training alone. Such challenges include tailoring training to the partner nation, integrating it into operational training, and a lack of capabilities and accountability systems on the part of partner nations. Agency officials and outside experts we spoke with stated that it can be challenging to tailor human rights training to the partner nation, the unit receiving assistance, and, when appropriate, the type of equipment being provided. DIILS has developed a three–tiered training model to meet the requirements of Section 333, as discussed above, and DIILS officials stated that they work to tailor trainings to the extent possible, including by selecting trainers with experience relevant to the equipment that the U.S. government provides and adding additional training when needed. However, agency officials and experts stated that DIILS, as a small entity, has limited capacity to tailor human rights trainings for specific situations, especially since DIILS must cover certain material to meet the Section 333 requirements. In addition, DIILS’ ability to tailor training is limited because, according to agency officials, mandated human rights training—typically a classroom course—is generally added to a security assistance package for a partner nation once the planning process has been completed. Since the human rights training is not integrated when the security assistance is planned, it is not generally feasible to adjust the training after the fact to address a specific situation in a given partner country, according to DOD officials. DSCA officials acknowledged that most human rights training is not sufficiently tailored to the needs of the recipient countries and that they have not yet fully incorporated human rights training considerations into security assistance planning. These officials said more work remains to be done to ensure that assistance under the Section 333 authority include comprehensive human rights training designed to meet specific partner nation needs. Agency officials and outside experts we interviewed stated that it can be challenging to achieve human rights objectives through human rights training as currently delivered because mandated human rights training is typically delivered as a stand–alone course in a classroom setting, rather than integrated into operational training. Agency officials stated that integrated training can be more effective because it would expose participants to practical skills that could help them comply with human rights concepts and avoid human rights violations during military or law enforcement operations. For example, State officials said that operational training on how to run a checkpoint while respecting human rights principles is likely to be more effective than training slides that outline international treaties on human rights. Agency officials and outside experts also stated that partner nations may lack capabilities and accountability systems. A military justice system might not hold responsible soldiers who commit human rights violations. A partner nation may lack equipment, experienced personnel, and planning for precision targeting to avoid civilian casualties. Further, partner nations may lack the political will to focus on human rights, and poorly–resourced security forces might see human rights as a low priority. Agency officials and outside experts said that without defense institution building that would address some of these broader systemic issues, human rights training may be less likely to have an effect in some countries. Finally, agency officials noted that in some instances, competing priorities necessitate prioritizing U.S. national security interests when providing security assistance, with human rights receiving less emphasis. Instilling respect for human rights in our foreign partners is important to achieving U.S. foreign policy goals. Human rights training that DOD and State provide is one means to do so, but DOD and State are unable to provide a comprehensive accounting of the full array of human rights training they support. With the demand for human rights training increasing as a result of Section 333, a process to ensure training information is systematically tracked would provide DOD greater assurance that it is complying with the statutory requirement to provide human rights training as a component of Section 333 assistance. Furthermore, DOD and State are not able to provide stakeholders, including Congress, with an evaluation of the effectiveness of human rights training the agencies support. Without monitoring and evaluation, decision–makers may be unable to identify whether human rights training provided through Section 333, IMET, and other authorities is achieving objectives and whether it could be adjusted for greater effectiveness. We are making a total of three recommendations, including two to DOD and one to State. Specifically: The Secretary of Defense should direct the Director of the Defense Security Cooperation Agency to establish processes to ensure that information on the provision of Section 333 mandated human rights training is systematically and accurately entered into its tracking systems. (Recommendation 1) The Secretary of Defense should direct the Under Secretary of Defense for Policy to develop a timeline for implementing its activities to monitor and evaluate the effectiveness of human rights training for foreign security forces. (Recommendation 2) The Secretary of State, in consultation with the Secretary of Defense, should develop a plan with a clear timeline for monitoring and evaluating the effectiveness of human rights training for foreign security forces provided under IMET. (Recommendation 3) We provided a draft of this report for review and comment to DOD and State. DOD concurred with the two recommendations directed to the Secretary of Defense and identified actions it plans to take to address them. Regarding the recommendation to monitor and evaluate human rights training, DOD stated that it would do so as part of monitoring and evaluating its broader security assistance efforts. DOD’s written comments are reproduced in appendix II. State disagreed with the recommendation directed to the Secretary of State. State’s written comments are reproduced in appendix III. In its comments, State acknowledged that human rights training is a vital element of IMET programs and agreed with the need to monitor and evaluate the effectiveness of training—including human rights training— delivered through IMET. However, the department stated that it did not agree to separately conduct monitoring and evaluation of human rights training for IMET participants. Our recommendation for State to develop a plan with a timeline to evaluate the effectiveness of human rights training provided under IMET does not call for a separate evaluation. State could meet the intent of our recommendation through evaluating the effectiveness of human rights training as part of its broader efforts to monitor and evaluate IMET. We added a statement to the report to that effect. We are sending copies of this report to the appropriate congressional committees, the Secretaries of Defense and State, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have questions about this report, please contact Jennifer Grover at 202-512-7141 or groverj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. This report (1) describes the entities through which the Department of Defense (DOD) and the Department of State (State) provide training for foreign security forces on human rights and international humanitarian law; (2) assesses the extent to which DOD and State track the provision of and funding for the training; and (3) examines the extent to which DOD and State have evaluated the effectiveness of the training; and (4) provides DOD, State, and outside expert views on human rights training. To address these objectives, we reviewed laws, guidance, budget documents, course catalogs, and agency data. We also interviewed agency officials in Washington, D.C., and at DOD geographic combatant commands. In addition, we conducted site visits at three facilities that provide human rights training: the Center for Civil–Military Relations (CCMR) in Monterey, California; the Defense Institute of International Legal Studies (DIILS) in Newport, Rhode Island; and the Western Hemisphere Institute for Security Cooperation (WHINSEC) in Fort Benning, Georgia. To address the structures through which DOD and State provide training for foreign security forces on human rights and international humanitarian law, we also reviewed course catalogs and interviewed DOD officials from several DOD entities, including the Defense Security Cooperation Agency; the Office of the Undersecretary of Defense for Policy; U.S. Africa Command; U.S. Indo-Pacific Command; and CCMR, DIILS, and WHINSEC. At State, we interviewed officials from the Bureaus of Political–Military Affairs; Democracy, Human Rights, and Labor; and International Narcotics and Law Enforcement Affairs; and the Office of Foreign Assistance Resources. To address what is known about tracking and funding for the training, including whether and how DOD comprehensively tracks human rights training, we reviewed DOD guidance and interviewed DOD officials and training providers. With the 2017 National Defense Authorization Act (NDAA) consolidating authorities—codified at 10 U.S.C. § 333—and the resulting increase in demand for the human rights training DIILS provides under that authority, we then focused on the ways in which that training and its funding is tracked in DOD systems. We reviewed agency documents, including congressional notifications and quarterly monitoring reports, to review how the training data are reported. We also reviewed federal internal control standards to determine what responsibilities agencies have related to information collection and communication. To assess the extent to which DOD and State have evaluated the effectiveness of the training, we reviewed monitoring and evaluation (M&E) policy and guidance documents and other relevant documents. We interviewed DOD and State officials about their current and planned actions to monitor and evaluate human rights training as well as examples of M&E-related efforts for security assistance programs that include human rights training. We also reviewed legislation, including the 2017 and 2019 NDAAs, which outline M&E requirements for DOD’s security assistance. In addition, we reviewed State’s January 2018 Guidance for the Design, Monitoring and Evaluation Policy at the Department of State to determine M&E requirements for State. To collect information on DOD, State, and outside expert perspectives of human rights training provided to foreign security forces, we conducted individual semistructured interviews with selected stakeholders, including agency officials and outside experts, who consisted of former government officials and representatives of nongovernmental organizations and think tanks. To identify outside experts, we asked stakeholders, including current government officials, to recommend other stakeholders we should speak with (i.e., snowball sampling). In our interviews, we collected information on perspectives of factors that could potentially enhance the effectiveness of human rights training and challenges to achieving human rights objectives through such training. The information we obtained from these stakeholders cannot be generalized across all stakeholders. We conducted this performance audit from February 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Jennifer A. Grover, 202-512-7141 or groverj@gao.gov. In addition to the contacts named above, Juan Gobel, Christina Werth, Emily Desai, Sada Aksartova, James McCully, David Payne, Neil Doherty, John Hussey, Mark Dowling, and Rachel Stoiko contributed to this report.", "summary": "The U.S. government seeks to advance human rights when it provides security assistance to foreign countries. Such assistance includes DOD– and State–supported human rights and international humanitarian law training for foreign security forces. The NDAA for Fiscal Year 2017 consolidated multiple capacity building authorities, now codified at 10 U.S.C. § 333. DOD implements most U.S. human rights training for foreign security forces. Congress included a provision in the NDAA for Fiscal Year 2018 for GAO to review human rights training for foreign security forces. This report, among other objectives, (1) describes the entities through which DOD and State provide such training, (2) assesses the extent to which DOD and State track the provision of and funding for such training, and (3) examines the extent to which DOD and State have evaluated the effectiveness of the training. GAO reviewed laws, regulations, guidance, agency training and funding data, and course catalogs, and interviewed agency officials. Several entities within the Departments of Defense (DOD) and State (State) are involved in human rights training. DOD's Defense Security Cooperation Agency (DSCA) conducts program management for DOD's efforts to build the capacity of foreign security forces. The human rights training required by 10 U.S.C § 333 is provided exclusively by the Defense Institute of International Legal Studies (DIILS), a DOD entity. DOD operates a number of other educational entities that provide training to foreign security forces, and many include human rights–related material in their curriculum or through operational exercises. (See figure.) DOD does not systematically track human rights training and, as a result, only limited information is available on the provision of and funding for these activities. Without a process to ensure systematic and accurate tracking of human rights training data, DSCA is limited in its ability to monitor its compliance with the training–related provision of the National Defense Authorization Act (NDAA) for Fiscal Year 2017. State relies on DOD to track human rights training for military forces and tracks some training and funding data for police. DOD and State have not assessed the effectiveness of human rights training for foreign security forces, according to agency officials. The NDAA for Fiscal Year 2017 required DOD to conduct monitoring and evaluation of its security assistance programs. DOD has taken initial steps to develop monitoring and evaluation policies but officials stated that they have not yet determined when DOD will evaluate human rights training. State officials said they do not know when the agency will begin monitoring and evaluating human rights training provided under the International Military Education and Training program, a large source of funding for such training. Monitoring and evaluation would enable DOD and State to determine the effectiveness of U.S.–provided human rights training for foreign security forces. GAO is making three recommendations, including that the Secretary of Defense establish a process to systematically track mandated human rights training and develop a timeline for implementing monitoring and evaluation. DOD agreed. GAO also recommends that the Secretary of State develop a plan with a timeline to monitor and evaluate such training. State disagreed. GAO continues to believe the recommendation is valid as discussed in the report.", "document_type": "gao"}
{"report": "FEMA is the federal agency primarily responsible for assisting state and local governments, private entities, and individuals to prepare for, mitigate, respond to, and recover from natural disasters, including floods. Floods are the most frequent natural disasters in the United States, causing billions of dollars of damage annually. In 1968, Congress passed the National Flood Insurance Act, which created NFIP, to address the increasing amount of flood damage, the lack of readily available insurance for property owners, and the cost to the taxpayer for flood-related disaster relief. Since its inception, NFIP has served as a key component of FEMA’s efforts to minimize or mitigate the damage and financial impact of floods on the public, as well as to limit the need for federal assistance after floods occur. A primary goal of NFIP is to minimize flood-related property losses by making flood insurance available on reasonable terms and encouraging its purchase by commercial and residential property owners who need flood insurance protection. The program focuses on areas in communities that are at the highest risk of flooding, known as special flood hazard areas. As of November 2019, 22,436 communities across the United States and its territories voluntarily participated in NFIP by adopting and agreeing to enforce flood-related building codes and floodplain management requirements. FEMA uses community assistance visits and community assistance contacts to oversee community enforcement of NFIP requirements. Community assistance visits are on-site assessments of a community’s floodplain management program and its knowledge and understanding of NFIP’s floodplain management requirements. During the visit, FEMA also helps the community remedy any program deficiencies or violations. Some visits are conducted by FEMA regional office staff and others by state floodplain management personnel, through funding from FEMA’s Community Assistance Program (State Support Services Element). Community assistance contacts are usually done by telephone, and their purpose is to establish or re-establish contact with an NFIP community regarding any existing problems or issues and to offer assistance if necessary. These contacts generally include a broad discussion of the community’s floodplain management activities, as well as any outstanding deficiencies and violations and community actions taken to resolve them. NFIP regulations allow FEMA to place a community on probation or to suspend the community from the program if it does not meet or enforce NFIP requirements. After a flood, local officials in communities that participate in NFIP must determine whether the proposed repairs to a damaged building are above or below FEMA’s threshold for substantial improvement or repair of substantial damage. Substantial improvement refers to any reconstruction, rehabilitation, addition, or other improvement of a structure that equals or exceeds 50 percent of the market value of the structure before the start of the construction. Repair of substantial damage means that the cost of restoring the structure to its pre-damage condition equals or exceeds 50 percent of the market value of the structure before the damage occurred. Substantially improved and substantially damaged buildings must be brought into compliance with NFIP requirements for new construction, including the requirement that lowest floors be elevated above the level indicated by the current NFIP flood map. These requirements help reduce future flood risk by elevating or otherwise mitigating properties at risk of flooding. FEMA officials generally do not conduct substantial damage assessments themselves but offer communities tools they can use to collect information and perform damage assessments. When a building insured under NFIP suffers a flood loss and is declared substantially damaged, the owner of the building can apply to receive up to $30,000, on top of any claim payment, to help rebuild according to current NFIP requirements, under a program called Increased Cost of Compliance. In 1990, FEMA implemented a voluntary rating system to recognize and encourage community floodplain management activities that exceed the minimum NFIP requirements. Communities may apply to join CRS if they are in full compliance with the minimum NFIP floodplain management requirements. As of June 2017, about 5 percent of NFIP communities participated in CRS, and more than 69 percent of all flood insurance policies were written in CRS communities. Communities are grouped into classes based on their ratings and can move up in ratings by earning CRS credits for activities such as increasing public information about flood risks, preserving open space, taking steps to reduce flood damage, and preparing residents for floods. The three goals of the CRS program are to reduce flood damage to insurable property by reducing existing buildings’ risk of flood damage and by protecting new buildings from current and future flood hazards; strengthen and support the insurance aspects of NFIP, in particular by encouraging communities to implement NFIP flood maps and increasing residents’ awareness of flood risk so they purchase and maintain flood insurance policies; and foster a comprehensive approach to floodplain management, such as by ensuring that new development does not cause adverse impacts elsewhere in the watershed or on other properties. As the community earns credits for additional flood-mitigation activities, residents and property owners in special flood hazard areas become eligible for increased NFIP policy premium discounts. Each CRS class improvement produces a 5 percent greater discount on flood insurance premiums for properties in the special flood hazard area, up to a maximum of 45 percent. FEMA contracts with a private company to administer many aspects of the CRS program. This contractor verifies the activities of communities on a 5-year cycle, though some communities may be visited on a 3-year cycle as their CRS class and discount improve. Communities can lose discounts if they do not sustain their activities. Communities in Texas and Florida made up 2 percent and 6 percent, respectively, of all NFIP communities nationwide, and their residents purchased almost half of all NFIP policies in force in 2019 (see fig. 1). After Hurricanes Harvey and Irma, property owners in Texas, Florida, and other states made about 98,000 flood insurance claims to NFIP and received a total of almost $10 billion. According to FEMA, Hurricane Harvey required a disaster response that was the largest in Texas state history. Nearly 80,000 homes had at least 18 inches of floodwater, and 23,000 of those had more than 5 feet. Older homes that were not built to minimum NFIP standards sustained the greatest damage. In Florida, Hurricane Irma caused widespread damage to residential and commercial buildings and infrastructure, and flood damage occurred particularly in low-lying areas. Community participation in NFIP is voluntary, but communities must join NFIP for their residents to purchase flood insurance through the program. To join NFIP, communities must adopt and enforce FEMA-approved building standards, floodplain management strategies, and floodplain management regulations to reduce future flood damage. FEMA relies on the communities to notify it of changing flood hazards and help update flood hazards on NFIP flood maps. (See figure 2 for an example of how development can increase flood risk.) Communities designate a floodplain administrator, who may be a local member of the community, such as a building inspector, community zoning official, engineer, or planner, or an entity contracted by the community, such as a county, regional planning agency, another jurisdiction or authority, or a private firm. 44 C.F.R. § 60.2(h). base flood elevations, or the elevation to which FEMA anticipates floodwater will rise during a flood (see fig. 3). Communities must require permits for all development in special flood hazard areas. The permit requirement includes both the construction of buildings or other structures and other land operations, such as mining, paving, excavation, or drilling, which can increase the risk of flooding by obstructing floodwater flows. Development must not increase the flood hazard on other properties. NFIP requires communities to regulate development to ensure that new development does not increase the risk of flooding for surrounding properties. 44 C.F.R. § 60.3. elevated to or above the base flood elevation indicated on the NFIP flood map. FEMA allows elevation on fill; elevation on posts, piers, or columns; or elevation on walls or a crawlspace (see fig. 4). Some communities set standards higher than what is required by NFIP. For example, Harris County, Texas, and Key West, Florida, require new or substantially improved construction to be elevated 2 feet and 1 foot, respectively, above NFIP’s base flood elevation level. In addition, several communities in Florida have cumulative substantial improvement rules. The rules require property owners who make substantial improvements over a period of time to a home built before the community implemented NFIP flood maps to elevate or bring the home into NFIP compliance. Several FEMA studies show that homes that are rebuilt above the base flood elevation suffer less damage in subsequent floods. Challenges expressed by some community officials whom we interviewed included difficulty enforcing NFIP requirements after a storm, retaining experienced floodplain management staff, and implementing updated NFIP flood maps. Difficulty inspecting buildings after a flood. Officials in several communities discussed the challenges related to inspecting buildings for substantial damage after a flood. In one community, inspectors had difficulty assessing flood damage because officials allowed construction to begin immediately and without a building permit. Floodplain officials in two communities said insurance adjustors may pay claims before inspectors have assessed damage, hindering inspectors’ ability to determine if repairs will exceed 50 percent of the home’s value if the homeowner begins to repair damage before the inspection. Challenges retaining floodplain management staff. In eight of the 19 communities we visited, officials cited difficulties obtaining or retaining sufficient staff to perform work such as conducting substantial damage assessments or fulfilling CRS paperwork requirements. For example, one floodplain official told us that after a major storm, the small floodplain management office was overwhelmed with trying to inspect damaged buildings to determine which would require rebuilding to current NFIP standards. Another community we visited did not have a full-time floodplain manager and relied on its building department, which is responsible for issuing building permits, to implement NFIP requirements. Officials said that retaining floodplain management staff is challenging due to factors such as the overwhelming amount of work that had to be performed after a hurricane and low prioritization of floodplain management in noncoastal communities. Two officials said that floodplain management is a difficult job, which can lead to high turnover of staff. Difficulty adopting new NFIP flood maps. Officials in three communities said the introduction of a new flood map can create difficulties. For example, an official said a new flood map can increase the size of the special flood hazard area and require more property owners to buy flood insurance. Another official said that new maps also can raise the base flood elevation, which can raise the cost of insurance premiums. A community official said that his community has been working with FEMA to revise a map for a few years and noted that some property owners in the community planned to challenge the new maps, further delaying adoption. FEMA’s primary method of verifying community compliance with NFIP requirements is through community assistance visits. These visits, along with community assistance contacts—which are in-depth discussions that can be conducted by telephone—are intended to help FEMA prevent, identify, and mitigate deficiencies in a community’s floodplain management. According to FEMA’s guidance, FEMA or state specialists who conduct these visits are to take the following steps (see fig. 5): Prepare for the visit. Specialists prepare for the visit by learning about the characteristics of the community and its prior history with NFIP in order to identify potential issues. Conduct the visit. Specialists tour the community, meet with local officials, and inspect files, among other activities. During the tour, specialists make observations, such as noting for later file inspection whether new structures or structures undergoing major repair meet permit documentation and base flood elevation requirements, and whether major new developments will divert flood water from special flood hazard areas. The specialists meet with local officials to assess the community’s floodplain management program and to provide technical assistance. Specialists also inspect the community’s files to assess the documentation and activities of its floodplain management program. Document findings. Within 30 days of the visit, the specialists are to enter information obtained from the visit, including specific information on deficiencies and violations, into FEMA’s Community Information System. Follow up with the community. After completing the visit, the specialists who conducted the visit are to ensure that the community resolves deficiencies and violations found during the visit in a timely manner. Specialists are to consider additional action, including enforcement actions, if deficiencies remain. In our visits to NFIP communities, officials told us that community assistance visits generally were consistent with the process we found documented in FEMA’s guidance. For example, community officials said specialists toured the floodplains to observe structures (such as new construction, renovations, and waterfront developments) and inspected community files, including permits and elevation certificates. The community officials said specialists generally spent from 1 to 7 days on site performing their reviews. Until recently, FEMA’s guidance documents stated that its goal was to visit all communities it considered to be high-risk every 5 years. FEMA designated some communities as high-risk based on factors including the community’s size, number of flood insurance policies, and number of previously damaged structures. Lower-risk communities were designated to receive a community assistance contact, training, or other contact without regard to time frame. FEMA officials with whom we spoke noted that the risk factors used to designate communities had not been updated since 2010. As a result, according to FEMA officials, in 2019 FEMA began developing a new selection tool that includes updated criteria and focuses on the risk of flooding in a community, the opportunity for a community to improve resilience, and the level of interest a community has in improving its floodplain management. An early version of the tool was released for testing in 2019. FEMA officials said that they and the states started using the new tool to select communities for the annual community visit cycle that began in July 2019. FEMA officials said that while they no longer have a goal of visiting high-risk communities once every 5 years, they do not anticipate conducting fewer visits than before. FEMA officials also noted that communities requesting to participate in CRS will be prioritized for a community assistance visit. From January 2008 through July 2019, FEMA met the 5-year goal for 13 percent of high-risk communities in Florida and 5 percent of such communities in Texas (see fig. 6). FEMA records also indicated that approximately 13 percent of high-risk communities in Florida and 31 percent in Texas did not receive a community assistance visit in that period. However, most high-risk communities in the two states were visited at some point during the overall time period. About 87 percent of high-risk communities in Florida and about 69 percent in Texas received at least one visit during that period. FEMA officials said that one reason for the limited number of visits to some high-risk communities is that FEMA resources, including state specialists, can be diverted to assist with disaster recovery efforts. FEMA officials also said that it is a challenge to visit all high-risk communities in states with a large number of NFIP communities, such as Texas and Florida, but they generally do not have the same challenge in states with fewer communities. FEMA officials said that in 2019 they employed about 120 specialists nationally, and that state grants allowed for another 130 state specialists to be divided among all states. Based on our analysis of FEMA’s data for Florida and Texas, FEMA regional staff completed about 20 percent of the visits and state specialists and others completed the remaining 80 percent. A FEMA official told us that the agency has been considering using methods other than community visits (such as checking in with communities 12–18 months after a flood) to verify compliance with NFIP requirements. However, as community assistance visits currently remain FEMA’s primary tool for ensuring compliance, the limited number of visits it has conducted in high-risk communities hinders its ability to provide such oversight. For example, it hinders FEMA’s ability to prevent, identify, and mitigate deficiencies in communities’ implementation of NFIP requirements, which, in turn, can limit their ability to prevent or limit future flood losses. According to FEMA guidance, specialists should document their community assistance visits, including information on any deficiencies and violations found during the visit, in FEMA’s Community Information System within 30 days of the visit. If a deficiency or violation is found, the specialists are to close out the record of the community visit after any deficiencies and violations have been addressed. The guidance further states that during the course of the visit, specialists should collect documentation that thoroughly supports their findings. Such documentation helps monitor a community’s progress toward resolving its floodplain management issues and, if needed, support any enforcement actions. Our review of FEMA records of community assistance visits in Florida and Texas from 2008 through 2019 showed that about one-third of all records remained open for a year or longer, and in some cases records stayed open for 5 years or more (see fig. 7). For example, around 29 and 23 percent of community assistance visits conducted in Florida and Texas, respectively, remained open for 3 or more years. In Florida, 4 percent remained open for 8 years or more. FEMA headquarters officials told us that they were unsure whether individual records remained open due to unresolved deficiencies and violations or because the specialist who conducted the visit failed to close the record. The officials also noted that specialists who enter information into the Community Information System about deficiencies and violations may not understand the importance of noting specific details and, as a result, may exclude details in many cases. As such, the level of detail can vary from one visit record to another depending on the individual entering the data. FEMA officials told us that turnover of state floodplain specialists and community floodplain managers could be a reason that many records remained open for an extended period. For example, they said turnover among state specialists could result in visit records remaining open because the staff responsible for closing a visit record no longer worked for the state. They also said that turnover among community floodplain managers could result in deficiencies remaining open for extended periods because there was no one in the community to address them. Furthermore, they said that because of the high turnover of community floodplain managers, they want to find other ways of monitoring community compliance with NFIP requirements. FEMA officials told us that another reason visit records can remain open for longer periods of time is FEMA’s approach to community oversight. The officials said that they would rather work with a community to resolve any deficiencies and consider steps such as suspension and probation to be a last resort. As a result, FEMA guidance does not include a maximum number of days a deficiency can remain open before beginning enforcement action, such as probation or suspension. Standards for internal control in the federal government state that management should use quality information to achieve the entity’s objectives. Without appropriate steps to ensure that it has reliable and timely information on community assistance visits, FEMA cannot readily determine if open records indicate a recordkeeping problem, a community deficiency that needs to be addressed, or something else. As a result, FEMA’s ability to determine if communities have been following NFIP requirements is hindered and the agency may miss opportunities to prevent future flood losses. Immediately after a flood, local floodplain management officials may assess the extent of damage to individual properties and determine whether damage is substantial enough that certain structures must be rebuilt to current NFIP requirements. As stated earlier, a substantially damaged property is one requiring repair work that costs 50 percent or more of a structure’s pre-flood market value. Local officials usually assess substantial damage to a property in three stages. Initial assessment. Local officials conduct initial assessments of flood-damaged properties—typically by driving through affected areas—to gauge the number of buildings affected and extent of damage. Preliminary damage assessments. These assessments are performed by FEMA or state officials, along with community officials. They are intended to broadly characterize the extent of damage. Local officials charged with performing building inspections and making substantial damage determinations may find the results of these assessments useful for identifying areas where significant damage has occurred and to coordinate their substantial damage inspections. Substantial damage assessment. Local officials conduct substantial damage assessments on the most severely damaged structures. These assessments are more in depth than the initial review and generally involve identifying damage to a property, estimating the cost to fix that damage, and determining whether the damaged structure can be classified as substantially damaged. State and FEMA representatives can assist local officials in performing these assessments, as they did after Hurricanes Harvey and Irma. FEMA also recently began offering communities an updated version of its substantial damage estimator tool, a software template designed to help officials assess damage more quickly and consistently. Figure 8 illustrates the process for declaring properties to be substantially damaged after a flood. While FEMA may provide assistance in conducting damage assessments, NFIP guidance documents state that community floodplain management officials are responsible for estimating the cost to repair and the market value of the structure, determining which properties are substantially damaged, and notifying property owners of their determination. As noted earlier, NFIP requires property owners to bring any substantially damaged buildings located in a special flood hazard area into compliance with minimum NFIP requirements, if they choose to rebuild. This could mean elevating their structure to reduce the risk of future flood damage or losses. For example, several officials from NFIP communities we visited commented that properties raised to or built at higher elevations following floods prior to 2017 received less flood damage during the events of 2017. Commercial and residential property owners with NFIP flood insurance who wish to rebuild a property that has been declared substantially damaged must work with the insurance company through which they purchased their NFIP policy to process their NFIP claim, and then must obtain permits from their community for repair work. As noted previously, these policy holders may be eligible to receive additional funding through NFIP’s Increased Cost of Compliance program—currently up to $30,000 beyond the claim payment—to help with the cost of bringing their home into compliance with current NFIP standards. FEMA does not have ready access to data on substantial damage assessments outside of community assistance visits, which we noted above are FEMA’s primary mechanism for NFIP community oversight. For example, we requested data from FEMA on the number of substantial damage assessments performed after Hurricane Harvey in Texas and Hurricane Irma in Florida in 2017. FEMA headquarters officials said that the data were not readily available and they would have to reach out to the regional offices to provide the figure, which took several months. In addition, FEMA regional officials said in August 2019 that they were still assessing the total number of properties that were substantially damaged in Texas in 2017 and that it would take approximately 12 to 24 months to collect these data. They estimated that local NFIP officials and state contractors in Texas performed 27,000 substantial damage assessments with FEMA assistance after Hurricane Harvey. FEMA regional officials also said that following Hurricane Irma, FEMA floodplain management specialists helped train local officials for, or assisted local communities in conducting, 20,206 substantial damage assessments in Florida. According to FEMA, as of December 2019, approximately 2,232 properties had been declared substantially damaged as a result of Hurricane Irma, 86 percent of which had been brought into compliance with NFIP regulation. FEMA officials could not tell us how many substantial damage assessments were conducted in Texas after Hurricane Harvey almost 2 years after the hurricane in part because FEMA does not have ready access to community data on substantial damage assessments. To access data on substantial damage assessments, FEMA headquarters officials first need to ask FEMA regional officials to request data from NFIP communities, and then wait for the communities to compile and send the data to the regional offices. FEMA officials also can review data on individual substantial damage assessments during community assistance visits. FEMA officials said they have not centralized or automated their collection of information on substantial damage assessments for several reasons. FEMA officials said that, in their view, the community is responsible for gathering and maintaining this information as a condition of its NFIP participation, and they consider the communities to be owners of those data. Furthermore, they said that centralized collection of substantial damage data would involve data privacy issues and be a drain on limited resources for disaster relief. However, FEMA officials expressed concern that some communities might not be consistently maintaining documentation of the substantial damage assessments. FEMA officials told us that they have two initiatives underway to help NFIP communities and FEMA staff collect data on substantial damage assessments: Substantial damage estimator tool. Updates to the substantial damage estimator tool, discussed earlier, should help communities collect data more consistently and better document assessments, according to FEMA officials. Community officials can use the tool to evaluate flood damage to residential and nonresidential structures and enter information such as structure type and address. The tool also includes a square-footage calculator and now allows photographs or other files to be attached to the completed assessment. Staff guidance. New staff guidance, which officials said will be implemented sometime in 2020, explicitly outlines for NFIP floodplain managers and FEMA staff the information NFIP communities should collect and maintain when performing substantial damage assessments. The guidance was created to address what FEMA officials believed were shortcomings in existing guidance to communities, which may have made some NFIP communities reluctant to conduct substantial damage assessments and enforce the requirements for those deemed substantially damaged. The new guidance also establishes time frames for data collection at the NFIP community level. While these steps may improve the quality of FEMA’s data on substantial damage assessments, federal internal control standards state that management should obtain relevant data from reliable sources in a timely manner based on the identified information requirements and obtain data on a timely basis so that they can be used for effective monitoring. If FEMA headquarters and regional offices do not have ready access to such data beyond the data collected during community assistance visits, they will be hindered in their ability to evaluate community compliance with NFIP requirements. FEMA also may be hindered in its ability to measure the effectiveness of substantial damage assessments, such as the extent to which substantially damaged homes are rebuilt according to NFIP requirements. It is especially important for FEMA to monitor community compliance with the process for assessing substantially damaged properties because this is the system FEMA uses to mitigate flooded properties and reduce the risk of future losses. If FEMA does not know how effectively this process operates, it could miss opportunities to use the process to reduce the financial exposure of NFIP. NFIP communities that we visited reported varying levels of access to NFIP claims data and information. According to FEMA guidance, the agency should provide local officials with information on their community that includes the number of flood insurance policies in force, dollar amount of coverage, and the number of claims. NFIP communities also can access information on publicly available data on claims payouts. Officials in some communities we visited were able to access claims data while officials in other communities were not. Some officials with whom we spoke were unsure whether access was permissible. For example, an NFIP community official in Texas said that FEMA told his office they could not provide him with the data when he asked for it. Another Texas official said that typically the communities do not have access to data on flood losses and claims paid. In Florida, an official said that she was able to access some data on NFIP claims in her area as long as the community did not use the data to make substantial damage determinations. Community officials told us that it would be helpful for them to access NFIP claims data after a flood. For example, a number of community floodplain managers told us that having NFIP claims information from FEMA would benefit their flood recovery efforts because it would allow them to better target their substantial damage assessments and make that process more efficient. Officials from other NFIP communities that we visited stated that claims data could help them identify property owners who were likely to start to rebuild and ensure they obtained permits, which can be difficult to determine otherwise. Another group of community officials said that claims data for their community indicated NFIP paid out more than the community’s own estimated value of the insured homes in their community, indicating there may have been more substantially damaged homes than they identified. FEMA officials acknowledged confusion among communities concerning their access to NFIP claims data and said they have been working to address it, noting that they must ensure compliance with the Privacy Act of 1974, under which the agency can share certain data only with organizations that have a programmatic need for the information. Officials also said they have been working to streamline the process through which NFIP communities can request claims data. For example, they said they have been considering the most efficient methods for sharing data with local communities that require post-disaster flood information while protecting the privacy of the data. In addition, FEMA officials said they have been drafting guidance—which they expect to be issued in 2020—and a new form for community data requests. They said their intent is that once communities are approved for access to a certain type of data, they would not have to reapply for subsequent requests. FEMA officials said the guidance will provide communities with access to NFIP claims data on a property-by-property basis. Federal standards for internal control state that management should externally communicate necessary quality information to achieve the agency’s objectives and address related risks. While FEMA has taken positive steps toward reducing confusion surrounding communities’ access to claims data, at the time of our review FEMA had not yet finalized new guidance. As a result, we were unable to evaluate the potential of these tools to effectively resolve communities’ confusion over appropriate access to claims information. Until FEMA clarifies the process for communities to access claims data on properties within their community, FEMA may be missing an opportunity to provide communities with data that they would find helpful in the substantial damage assessment process. FEMA relies on communities participating in NFIP to follow its floodplain management requirements, which are designed to reduce the risk of flood damage and the resulting cost to taxpayers. Community assistance visits are the agency’s primary tool for ensuring that communities implement these requirements. However, in Texas and Florida FEMA often has not conducted such visits to high-risk communities and lacks complete data on the results. As a result, FEMA’s ability to ensure that the communities follow NFIP requirements is limited. In addition, FEMA does not have ready access to data on substantially damaged properties and the related documentation, which hinders its ability to determine if an NFIP community has followed NFIP substantial damage assessment procedures and correctly identified all substantially damaged homes. This, in turn, limits FEMA’s ability to evaluate NFIP’s effectiveness. Finally, confusion exists among some NFIP communities regarding their access to NFIP claims data, potentially limiting the benefit such data could provide to those communities in identifying substantially damaged properties and ensuring all repairs of flood damage are done to NFIP community standards. We are making a total of four recommendations to FEMA: The Administrator of FEMA should assess different approaches, in addition to community assistance visits, for using existing resources to ensure communities’ compliance with NFIP requirements. This should include analyzing alternatives to community assistance visits. (Recommendation 1) The Administrator of FEMA should identify appropriate steps to ensure it has complete, up-to-date, and reliable records of community assistance visits, including information on why some visit records remain open for a significant period of time. (Recommendation 2) The Administrator of FEMA should ensure that communities are consistently collecting data on their substantial damage assessments and that FEMA has a way to readily access those data to evaluate community compliance with NFIP requirements for rebuilding substantially damaged properties. (Recommendation 3) The Administrator of FEMA should clarify with NFIP communities its policies on sharing data on NFIP claims and provide such information to those communities as needed. (Recommendation 4) We provided a draft of this report to the Department of Homeland Security for review and comment. In its comments, the Department of Homeland Security concurred with our recommendations. FEMA also provided technical comments, which we incorporated, as appropriate. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report (1) describes the requirements that communities participating in the National Flood Insurance Program (NFIP) must meet and the challenges they face in doing so, (2) examines the extent to which the Federal Emergency Management Agency (FEMA) uses community visits to ensure communities follow requirements, and (3) examines how FEMA oversees community implementation of NFIP requirements for conducting substantial damage assessments. This report focuses on NFIP communities in Florida and Texas that were affected by Hurricanes Irma and Harvey in 2017. For all three objectives, we reviewed FEMA guidance and regulations for communities participating in NFIP and in FEMA’s Community Rating System. We interviewed officials from FEMA’s Federal Insurance and Mitigation Administration, as well as officials in two FEMA regional offices in Georgia and Texas. We also visited 18 communities in Texas and Florida, and an additional community in Louisiana, that were affected by flooding in the 2017 hurricanes. We conducted structured interviews with officials in these communities. We selected these communities to represent a mix of large and small communities and because they participate in FEMA’s Community Rating System. The officials we interviewed included floodplain managers, emergency management coordinators, watershed managers, and representatives of homebuilder associations. We also interviewed representatives of four national and state floodplain associations, and three additional experts—two academic experts and a city official—with significant knowledge of NFIP and flooding issues. For our first objective, we analyzed the responses of these officials to identify the most commonly cited challenges. For our second objective, we analyzed data on community assistance visits in Florida and Texas from FEMA’s Community Information System from January 1, 2008, through July 30, 2019, and spoke with FEMA and community officials. To determine whether FEMA carries out the community assistance visits in accordance with its own guidance, we reviewed FEMA’s guidance for specialists to prepare for, conduct, and follow up on the visits. We also spoke with FEMA and other officials about their experience with the visits to determine whether FEMA and state specialists generally followed FEMA’s guidance. To determine the extent to which FEMA met its goal of visiting high-risk communities once every 5 years, we compared the data in the Community Information System on community visits against the lists of Tier 1 (high-risk) and Tier 2 (lower- risk) communities provided by FEMA. We also analyzed the data to determine the length of time that records from the community visits were left open, and whether the records were complete. While we noted that the data in the Community Information System were at times incomplete, we found the data reliable enough to identify the frequency of community assistance visits and issues with data entry. For our third objective, to examine how FEMA oversees community implementation of NFIP requirements for conducting substantial damage assessments, we reviewed FEMA policies and guidance, including NFIP Floodplain Management Requirements outlined in 44 C.F.R. Parts 59 and 60. We also reviewed FEMA’s Substantial Improvements Substantial Damage Desk Reference (FEMA 758-P) and FEMA flood-mitigation requirements. We examined FEMA’s NFIP post-flood processes and procedures related to substantial damage assessments. We reviewed FEMA data on the number of substantial damage assessments performed in Florida and Texas after Hurricanes Irma and Harvey as well as the number of damaged properties that received increased cost of compliance funding. We discussed with community officials their experiences conducting substantial damage assessments and the challenges they faced in doing so. We also reviewed literature to identify actions taken by NFIP communities after a flood, and we reviewed FEMA documentation to determine the actions taken by FEMA and NFIP communities after a flood. We also compared FEMA’s practices for collecting data for effective monitoring and communication against federal standards for internal controls. We conducted this performance audit from October 2018 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Alicia Puente Cackley, (202) 512-8678 or cackleya@gao.gov. In addition to the contact named above, Patrick Ward (Assistant Director), Leah DeWolf (Analyst in Charge), Audrey Blumenfeld, Tarik Carter, Anar Jessani, Angela Pun, Jessica Sandler, Jennifer Schwartz, and Jena Sinkfield made key contributions to this report. William Chatlos and Yann Panassie provided technical assistance.", "summary": "NFIP's effectiveness depends in part on communities implementing FEMA requirements on floodplain management and post-disaster rebuilding efforts. GAO was asked to undertake a comprehensive evaluation of federal disaster preparedness, response, and recovery efforts. This report examines (1) requirements NFIP communities must meet and challenges they face, (2) FEMA's use of community visits to ensure compliance, and (3) how FEMA oversees community implementation of NFIP requirements for conducting substantial damage assessments. GAO analyzed FEMA data on oversight visits and substantial damage assessments from January 2008 through July 2019. GAO also interviewed floodplain managers in 19 communities in Texas, Florida, and Louisiana, and officials from FEMA and floodplain management organizations. The Federal Emergency Management Agency (FEMA) requires communities participating in the National Flood Insurance Program (NFIP) to adopt FEMA floodplain maps; limit flooding caused by new development; and require that substantially damaged structures meet elevation requirements (see figure). Community floodplain officials cited challenges, including difficulty inspecting buildings after a flood, staff turnover, and adopting new NFIP flood maps. FEMA primarily uses community assistance visits to monitor compliance with NFIP requirements. The visits include evaluations of recent construction. Until 2019, FEMA's goal was to visit all communities considered to be high-risk every 5 years. However, FEMA did not meet this goal in Texas or Florida in 2008–2019 because of a lack of resources. Many high-risk communities received only one visit in this period, and some were not visited at all. Without regular monitoring, FEMA's ability to ensure communities comply with requirements is limited. FEMA and state specialists also are to close out records of these visits in FEMA's tracking system if they find no deficiencies or violations, or when the community has resolved any issues. However, in Florida and Texas GAO found that records for many visits remained open for several years, and FEMA staff were unsure whether this indicated unresolved deficiencies or incomplete recordkeeping. Unreliable recordkeeping hinders FEMA's ability to assess community compliance with NFIP requirements. After a flood, one key community responsibility is to assess whether flood damage on a property was substantial (50 percent or more of the property's value). In such cases, the community must ensure the properties are rebuilt to current NFIP standards. However, FEMA generally does not collect or analyze the results of these assessments, limiting its ability to ensure the process operates as intended. Furthermore, FEMA has not clarified how communities can access NFIP claims data. Such data would help communities target substantial damage assessments after a flood. GAO is making four recommendations to FEMA: The agency should (1) assess different approaches for ensuring compliance with NFIP requirements, (2) ensure data on community visits are up-to-date and complete, (3) ensure communities collect data on substantial damage assessments, and (4) clarify policies on data sharing between FEMA and NFIP communities. FEMA concurred with the recommendations.", "document_type": "gao"}
{"report": "The use of ESG factors has emerged as a way for investors to capture information on potential risks and opportunities that otherwise may not be taken into account in financial analysis. ESG factors like climate change impacts and workplace safety may affect a company’s expected financial performance and thereby its value to shareholders. See table 1 for examples of ESG factors. ESG standard-setting organizations were created to improve transparency and consistency in companies’ disclosure of ESG information. Several independent and nonprofit organizations have created voluntary frameworks companies may use to disclose on ESG issues, as shown in table 2. Frameworks are generally comprised of single-issue categories that contain several specific disclosure topics related to that category. SEC rules and regulations generally require public companies to disclose, among other things, known trends, events, and uncertainties that are reasonably likely to have a material effect on the company’s financial condition or operating performance, as well as potential risks to investing in the company. SEC considers information to be material if there is a substantial likelihood that a reasonable investor would consider it important in making an investment decision in the context of the total mix of available information. Public companies disclose information on an ongoing basis through annual 10-K filings, quarterly 10-Q filings, and definitive proxy statements, among other disclosure requirements. Regulation S-K contains SEC integrated disclosure requirements for 10-K filings and other periodic reports filed with SEC. Staff in Corporation Finance are to selectively review 10-K filings for compliance with requirements outlined in Regulation S-K and other applicable accounting standards and form requirements. While federal securities laws generally do not specifically address the disclosure of ESG information, Regulation S-K’s disclosure requirements for nonfinancial information apply to material ESG topics. Regulation S-K also includes prescriptive requirements for disclosure of certain topics considered to be ESG topics, such as board composition, executive compensation, and audit committee structure. Corporation Finance’s legal and accounting staff review filings through seven offices organized by industry, and office managers assign different levels of reviews to 10-K filings, such as full reviews (which include financial and legal reviews) and financial-only reviews. The Sarbanes- Oxley Act of 2002 requires SEC to review the financial statements of each reporting company at least once every 3 years, which informs, among other factors, how Corporation Finance selects and determines the extent to which 10-K filings are reviewed. In conducting these reviews, Corporation Finance staff may provide comments to a company to obtain additional information, clarification on the company’s disclosure, or to significantly enhance its compliance with applicable reporting requirements. Comments depend on the issues that arise in a particular filing, and staff may request that a company provide additional information to help them better evaluate disclosures. SEC occasionally issues interpretive releases on topics of general interest to the business and investment communities, which reflect the Commission’s views and interpret federal securities laws and SEC regulations. For example, in 2010, SEC issued the Commission Guidance Regarding Disclosure Related to Climate Change, which described how existing disclosure requirements could apply to climate change-related information and how companies may consider climate disclosures in required filings. In 2018, SEC also issued the Commission Statement and Guidance on Public Company Cybersecurity Disclosures, outlining how existing reporting requirements could apply to cybersecurity-related risks and incidents. These interpretive releases do not establish new reporting requirements. Instead, they identify items in existing laws and regulations that may be most likely to require disclosure on these topics, such as description of the company’s business and potential risk factors that may affect the company. Institutional investors with whom we spoke generally agreed that ESG issues can have a substantial effect on a company’s long-term financial performance. All seven private asset managers and representatives at five of seven public pension funds said they seek ESG information to enhance their understanding of risks that could affect companies’ value over time. Representatives at the other two pension funds said that they generally do not consider ESG information relevant to assessing companies’ financial performance. While investors with whom we spoke primarily used ESG information to assess companies’ long-term value, other investors also use ESG information to promote social goals. A 2018 US SIF survey found that private asset managers and other investors, representing over $3.1 trillion (of the $46.6 trillion in total U.S. assets under professional management), said they consider ESG issues as part of their mission or in order to produce benefits for society. Institutional investors we interviewed identified various ways they use ESG disclosures to inform their investment decisions and manage risks related to their investments. Protecting long-term investments by monitoring companies’ management of ESG risks. Some investors with whom we spoke noted that they primarily make long-term investments in passively managed funds, which may prevent them from making investment decisions based on ESG information. However, 10 of 14 investors said that their focus on long-term factors that drive value leads them to monitor or influence companies’ management of ESG issues to protect their investments. Investors generally said they use ESG disclosures to determine which ESG issues companies monitor and to assess how companies manage these risks. Nearly all investors said ESG issues can be important to a company’s operations and performance over time. For example, seven of 14 investors said they used ESG disclosures to identify companies that were less transparent than their peers or appeared to be outliers in their industries, such as having less board diversity than their peers. Investors then engaged with these companies to discuss their risk- management strategies, encourage disclosure on ESG issues, or provide information about what kind of disclosure they would find useful. Informing shareholder votes. Most investors with whom we spoke said they use ESG information to inform their votes as shareholders at annual shareholder meetings, either through a proxy advisory firm or independently. Specifically, nine of 14 investors said that ESG information informs how they vote on directors’ nominations to the board and other proposals at public companies’ annual meetings. For example, representatives from two large public pension funds said they withhold votes for directors if they determine that a company’s board had not effectively disclosed issues, such as climate risk or executive performance metrics. Creating ESG funds or portfolios. Five of 14 investors we interviewed said they created ESG-focused investment funds or portfolios with goals such as promoting social responsibility and environmental sustainability. In creating these funds and portfolios, investors generally review companies’ ESG disclosures to determine which companies to include or exclude from these funds or portfolios. For example, two private asset managers said they created ESG funds or portfolios to attract investors focused on social goals, such as faith-based investors, while representatives from one pension fund said they had worked with an asset manager to create a low- emissions index intended to support the Paris Agreement’s goals. Divesting. Some investors we interviewed said they typically would not divest based on a company’s ESG disclosures, and three said that ESG information could lead them to divest. A mid-size asset manager noted that the firm works with companies to improve their disclosures rather than divest. Conversely, representatives from one mid-size pension fund said they found that buying or selling shares is a more efficient method for changing corporate behavior than the lengthier strategy of engaging companies in dialogue. Additionally, a large asset manager said that its portfolio managers sell shares if a company’s ESG performance or response to engagement is poor. Although some studies report that the quantity and quality of ESG disclosures generally improved in the last few years, 11 of 14 investors with whom we spoke said they seek additional ESG disclosures from companies to address gaps and inconsistencies, among other issues. Investors described challenges with understanding and interpreting both quantitative and narrative disclosures. Quantitative disclosures. Investors cited examples of inconsistencies in companies’ quantitative disclosures that limit comparability, including comparability among companies that disclose on the same ESG topics. Specifically, investors described challenges such as the variety of different metrics that companies used to report on the same topics, unclear calculations, or changing methods for calculating a metric. For example, five of 14 investors said that companies’ disclosures on environmental or social issues use a variety of metrics to describe the same topic. A few studies have reported that the lack of consistent and comparable metric standards have hindered companies’ ability to effectively report on ESG topics, because they are unsure what information investors want. In addition, some investors said that companies may change which metrics they use to disclose on an ESG topic from one year to the next, making disclosures hard to compare within the same company over time. Narrative disclosures. Most investors noted gaps in narrative disclosures that limited their ability to understand companies’ strategies for considering ESG risks and opportunities. For example, some investors noted that some narrative disclosures contained generic language, were not specific to how the company addressed ESG issues, or were not focused on material information. For example, two private asset managers said that companies may provide boilerplate narratives or insufficient context for their quantitative disclosures, and representatives from one pension fund said that the fund would like additional disclosures on cybersecurity but has found that most disclosures on this topic are generic and not very helpful. Additionally, most institutional investors said that there is fragmentation in the format or location of companies’ ESG disclosures, which can make this information hard to compile and review. However, these investors generally said that it is more important for companies to focus on providing disclosures than on how or where the disclosures are presented. These investors said that they are able to purchase access to compiled data from third-party data providers to use in their analysis of companies’ ESG disclosures. Regarding how investors seek ESG disclosures, nearly all institutional investors with whom we spoke said they engage with companies to request additional ESG disclosures through meetings, telephone calls, or letters. Some investors said that companies’ responsiveness, which can include producing ESG presentations for investors and discussing ESG information on earnings calls, varied by size because larger companies have more resources to respond to investor engagement. Engagement also can be complicated by conflicting investor demands, as well as the proliferation of standards and surveys. According to representatives from an industry group that we interviewed, the large number of demands for specific ESG information from investors and third parties can pose a challenge to companies as they prioritize how to respond. For example, one company said it receives diverse requests for information that indicate that those investors do not agree on what issues are most important. Some investors seek additional ESG disclosures by submitting shareholder proposals, which are requests from shareholders that the company take action on a specific issue or issues. These proposals are generally presented for a shareholder vote at public companies’ annual meetings. However, shareholder proposals can be withdrawn before coming to a vote when the company reaches an agreement with the shareholder who submitted the proposal prior to the annual meeting. Our analysis of a generalizable sample of companies listed on the S&P 1500 found that in 2019, an estimated 10 percent of companies received one or more shareholder proposals and an estimated 5 percent of companies received one or more shareholder proposals related to increasing ESG disclosures. For the ESG-related proposals in our sample, on average about 28 percent of shareholders voted in favor of these proposals and no proposals received more than 50 percent of the vote. As shown in table 3, the companies in our sample received a total of six proposals requesting additional ESG disclosures on a variety of social and governance topics. Most of these proposals were submitted to large companies. Investors that submitted proposals included one public pension fund, one labor organization, three socially focused asset managers, and one higher education endowment. All of the private asset management firms and representatives from three of seven pension funds we interviewed said they do not use shareholder proposals as a means to influence companies’ ESG disclosures. One of these pension funds said they have found filing shareholder proposals unnecessary after engaging in dialogue with companies. However, representatives from four of seven pension funds said they have filed shareholder proposals to seek additional ESG disclosures. Two large pension funds said they have found filing shareholder proposals an important engagement method for getting companies’ attention on ESG issues, while the other two funds noted that it was rare for them to file a proposal. Similarly, studies and reports we reviewed indicated that shareholder proposals are concentrated among a relatively small number of shareholders and that the number of proposals has been declining in the last 5 years. For example, a law firm’s analysis of shareholder proposals filed with companies listed on the S&P 1500 in 2019 reported that 10 investors submitted over half of all proposals. This report also found that faith-based investors and socially focused asset managers, who seek to advance social causes in their investments, submitted the majority of environmental and social proposals in both 2018 and 2019. In addition, this analysis showed that the total number of shareholder proposals, including withdrawn proposals, submitted annually declined each year from 2015 to 2019. As the total number of proposals has declined, shareholder proposals related to environmental and social issues constituted over 45 percent of proposals each year from 2015 to 2019. While studies found that during this same time period shareholder support increased for these environmental and social proposals that went to a vote, shareholder support for most of them remained below 30 percent. Representatives from public companies with whom we spoke said they use several methods and consider multiple factors when deciding which ESG topics to report. Most companies (10 of 18) noted that legal and regulatory requirements were their primary consideration when determining which ESG factors to disclose. In addition, nearly all companies (15 of 18) told us they conduct some form of stakeholder engagement when determining what ESG information beyond regulatory requirements to report. As part of the engagement process, companies generally said they reach out to investors, representatives of communities they operate in, and other interested stakeholders to solicit their opinions about which ESG factors are important to them. Some companies described their ESG stakeholder engagement process as part of their broader company-wide outreach efforts, while others told us they hired outside firms to conduct this engagement on their behalf. In addition to stakeholder outreach, most companies (11 of 18) told us they perform assessments to determine which ESG topics to include in their regulatory filings or other reports. As part of these assessments, companies review a wide array of potential risks and identify the ones that would have the most impact on their business. In addition to requirements, outreach and assessments, most companies (nine of 18) told us they review ESG disclosure frameworks, such as GRI and SASB, to inform their consideration of which ESG factors to disclose. Similar to deciding which ESG topics to disclose, most companies (10 of 18) told us they also rely on legal and regulatory requirements when determining where to disclose ESG information. Specifically, companies said they identify those ESG factors that should be included in the 10-K or proxy statement according to SEC requirements, and publish information on these factors in their regulatory filings. In addition, some companies (six of 18) told us that they view their voluntary sustainability report as complementary to their regulatory filings. Specifically, four companies said they view their sustainability reports as a place to publish relevant ESG information that may not necessarily be material under the SEC definition and is therefore not included in regulatory filings. Lastly, some companies also told us that their voluntary sustainability reports provide an opportunity to disclose information that is of interest to ESG-focused investors or non-investor stakeholders. For example, some companies (five of 18) told us they use these reports to reach a broader stakeholder audience beyond investors, including employees and customers, when writing their sustainability reports. In addition to the regulatory and voluntary reporting that we reviewed, representatives from all 18 companies said they communicate ESG information in other ways. For example, most companies (13 of 18) said they also publish issue-specific ESG reports, most commonly on climate change. Most companies (12 of 18) also said they include ESG information on their company websites, because information could be updated more frequently and include more dynamic content, such as videos. Finally, most companies (11 of 18) told us they have developed ESG-focused presentations for investors, and some companies (four of 18) said they have begun including ESG information in their traditional investor communications, such as quarterly earnings calls and stockholder bulletins. To assess the amount and characteristics of the ESG information companies report, we reviewed regulatory filings and voluntary reports issued by 32 large and mid-size public companies in eight industries. For each company, we reviewed two types of regulatory filings (10-K and the definitive proxy statement), annual reports (when distinct from the 10- K), and voluntary sustainability reports (where available). Of our selected companies, 25 published voluntary sustainability reports and 21 published annual reports separate from their 10-Ks. Using keyword search terms, we searched these documents to identify disclosures related to eight broad ESG factors and 33 more-specific disclosure topics under these factors (see fig. 1). We selected ESG factors from among those that a range of market observers frequently cited as important to investors or potentially material and selected ESG topics by reviewing ESG disclosure frameworks. For more information about this methodology, see appendix I. As shown in figure 2, we identified disclosures on six or more of the eight ESG factors for 30 of the 32 companies in our sample and identified 19 companies that disclosed information on all eight factors. All selected companies disclosed at least some information on factors related to board accountability and resource management. In contrast, we identified the fewest companies disclosing on human rights and occupational health and safety factors. With regard to the 33 more-specific ESG topic disclosures we examined, 23 of 32 companies disclosed on more than half of them. The topics companies disclosed most frequently were related to governance of the board of directors and addressing data security risks. Conversely, based on disclosures we identified, we found that companies less frequently reported information on topics related to the number of self-identified human rights violations and the number of data security incidents. In addition, we found that companies most frequently disclosed information on narrative topics and less frequently disclosed information on quantitative topics. There are several reasons why a company may not have disclosed information on a specific ESG topic, including that the topic is not relevant to its business operations or material. Figure 3 compares the amount of disclosure on the 33 ESG topics within and across the selected industries. We identified the most disclosure on the group of topics related to board accountability, climate change, and workforce diversity and the least amount on topics related to human rights. SEC requires companies to report certain governance information in their proxy statements in advance of shareholder meetings where shareholders elect members of the company’s board of directors, which may help explain why board accountability topics are the most reported across industries in our sample. Additionally, differences in disclosure can result, in part, from the relevance of an ESG topic to a particular industry. For example, more companies in the airline and oil and gas industries disclosed information on climate change, while more companies in the internet media and banking industries disclosed information on data security. We identified disclosures on fewer topics by companies in the internet media industry than the other industries we assessed. None of the four internet media companies in our sample issued a stand-alone sustainability report. As discussed below, most companies tended to include more extensive ESG disclosures in their sustainability reports than in their regulatory filings. Figure 4 illustrates how the amount of disclosures on the 33 ESG topics compared across the four types of documents we reviewed. We found that companies generally reported information on a wider variety of ESG topics in their voluntary sustainability reports. Specifically, with the exception of a few topics, when companies disclosed information on an ESG topic, they most frequently did so in their sustainability reports. Certain ESG topics were reported more frequently in regulatory filings. For example, nearly all selected companies reported ESG information related to their board of directors in their proxy statements. Additionally, we found that companies disclosed on risks related to climate change, data security, hiring employees, and resource management in their 10- Ks, which includes a risk factors section where companies are required to discuss the most significant factors that make investment in the company speculative or risky. As discussed earlier, some investors with whom we spoke said they seek additional narrative disclosures from companies whose disclosures contained generic language or did not provide specific details about how the company manages ESG-related risks or opportunities. Among the 33 ESG topics we reviewed, 16 were topics for which companies reported a narrative rather than quantitative disclosure. We categorized these narrative disclosures as either generic or company-specific (see fig. 5 for examples). We defined company-specific disclosures as those that discussed specific ways that ESG-related risks and opportunities could affect the company’s operations or specific steps the company takes to manage or respond to the ESG-related risks or opportunities. We defined disclosures that did not include such specific details as generic disclosures. As a result, such generic disclosures can be considered applicable to the reporting company as well as to many of its peers. According to two reports, companies may choose not to disclose more detailed information for a particular ESG topic for several reasons, including concerns that such disclosures would put the company at a competitive disadvantage or expose it to legal liability. For 11 of the 16 narrative topics, among companies for which we identified disclosures on these topics, at least 75 percent disclosed company-specific information (see fig. 6). For certain topics, such as those related to companies’ actions to add new directors to the board and promote diversity and inclusion, most companies disclosed information and nearly all of those companies reported company-specific information. In contrast, for other narrative topics, such as addressing data security risks and describing climate-related risks and opportunities, we identified company-specific information for less than two-thirds of disclosing companies. In addition, for one narrative topic, describing obstacles that might limit the company’s ability to hire the talent it needs, less than one- third of disclosing companies reported company-specific information. We also found that disclosures we identified in companies’ 10-K filings were less likely to be company-specific than those in the other three types of documents we reviewed. Though most of the narrative ESG disclosures we reviewed contained company-specific details, these disclosures varied in the amount of detail they provided about how a company manages ESG-related risks and opportunities (see fig. 7). In particular, some companies’ disclosures included details about specific steps the company was taking to manage an ESG-related risk or opportunity and details about the results of such efforts, while others did not. To the extent that some companies provided more detailed disclosures, those companies’ disclosures could be of greater usefulness to investors trying to understand the ESG risks facing a company or the steps the company was taking to manage ESG risks. We identified inconsistencies in how companies disclosed on some of our selected quantitative ESG topics, which may limit investors’ ability to compare these disclosures across companies. Specifically, we found instances where companies defined terms differently or calculated similar information in different ways. We most frequently identified these inconsistencies in quantitative topics associated with climate change, personnel management, resource management, and workforce diversity. For quantitative topics related to data security, human rights, and occupational health and safety, five or fewer of the 32 companies in our sample disclosed information on these topics, limiting comparisons across companies. As previously discussed, some investors told us that one of the reasons they seek additional ESG disclosures is because it is difficult to compare disclosures across companies. SEC also noted in a 2016 concept release that sought comment on modernizing certain disclosure requirements in Regulation S-K that consistent disclosure standards can increase the efficiency with which investors process the information. Additionally, three of the most commonly used ESG disclosure frameworks—GRI, SASB, and TCFD—have a stated goal to help companies disclose information in a way that allows investors to compare information among companies. Despite this focus on comparable reporting from investors, regulators, and standard-setters, we identified instances where companies reported certain quantitative metrics differently from one another for some ESG topics. For example, in workforce diversity disclosures, some companies reported their employee demographics using broad groupings, such as “minority” or “ethnically diverse,” while others reported by specific racial or ethnic groups. Similarly, some companies defined greenhouse gas emissions differently. Most companies combined carbon dioxide and other greenhouse gases when reporting emission data, but a few reported carbon dioxide emissions alone. We also identified instances of companies using different calculation methods or units of measure when reporting information related to climate change and resource management. For example, companies used different base years when calculating their reduction in greenhouse gas emissions, limiting their comparability. Some companies reported reductions year-over-year, while many reported reductions over multiple years with no consistency within or across industries. For example, airline companies we reviewed reported emission reductions with base years ranging from 1990 to 2017. Similarly, when disclosing total water withdrawal, eight companies used metric units of measure while two companies used imperial units of measure. Companies that used the same ESG framework did not always disclose on ESG topics in a consistent manner. Specifically, we identified the types of inconsistencies discussed above in quantitative disclosures among those companies using the GRI framework. For example, we identified four different methods for reporting workforce diversity among companies that reported using the GRI framework to develop their disclosures. The GRI framework does not specify the method for reporting diversity information, as it does for certain other topics. SEC staff generally use a principles-based approach to overseeing public companies’ disclosures of nonfinancial information, including information on ESG topics. Under this approach, SEC staff rely primarily on companies to determine what information is material and requires disclosure in their SEC filings, such as the 10-K filing. SEC officials noted that companies are ultimately responsible for the disclosures they provide to investors, and they have liability for their disclosures under federal and state securities laws. While federal securities laws generally do not specifically address the disclosure of ESG information, Regulation S-K’s disclosure requirements for nonfinancial information apply to material ESG topics. Corporation Finance officials noted that their reviews of public companies’ 10-K filings are not a checklist review for compliance with securities regulations. Instead, these reviews are meant to identify and address potentially significant disclosure issues, such as nondisclosure of information that the Corporation Finance review team believes is material and therefore may influence an investor’s investment decision. Some Corporation Finance review staff told us that in their reviews of public companies’ 10-K filings they generally defer to companies’ determinations about which ESG information is relevant to their business and should be disclosed. Review staff also generally said they perform company- and industry-specific research as part of their review, including company websites, web searches for news articles, and earnings calls that may identify material ESG information. In a January 2020 statement that addressed climate change and environmental disclosures, the SEC Chairman reiterated his view that SEC’s approach to disclosure on these topics should continue to be rooted in materiality, including providing investors with insight regarding the company’s assessments and plans for addressing material risks to its business operations. The Chairman’s statement also noted that this approach is consistent with the Commission’s ongoing commitment to ensure that current disclosures on these issues provide investors with a mix of information that facilitates well-informed capital-allocation decisions. Corporation Finance has provided its review staff with internal review guidance that highlights relevant issues to consider, while emphasizing the use of professional judgment when reviewing companies’ 10-K and other filings. Staff use internal procedural guidance that provides steps for conducting and documenting reviews of filings. While this guidance does not include specific instructions for reviewing ESG disclosures, staff are instructed to conduct background research on companies and industries to determine if there is material information, such as potential risks, that may be relevant to a company’s filing. As noted above, according to review staff, this company-specific research could include ESG information. In addition, Corporation Finance has distributed internal review guidance on a few ESG-related topics. This guidance illustrates how existing disclosure requirements may apply to a given topic and offers information for staff to consider when conducting background research and performing filing reviews. In cases where the SEC review team identifies a potential disclosure deficiency related to an ESG or other topic, they may issue a comment letter to the company to request additional information or additional disclosures when necessary. Most review staff with whom we spoke said ESG-related information generally does not rise to the level of comment unless they identify material information during background research that may be relevant to the company’s operations. In April 2019, Corporation Finance reallocated responsibilities for reviewing nonfinancial information in 10-K filings, which also can include ESG information, from attorneys to accountants. Corporation Finance officials cited resource constraints, which reduced the number of attorneys within the Division, as a factor in this decision. While review teams vary by industry group and company, attorneys previously held primary responsibility for reviewing nonfinancial disclosures, whereas accountants primarily reviewed financial statements and related disclosures in 10-K filings. SEC staff provided training to accountants on how to conduct these reviews, which outlined Regulation S-K reporting requirements for nonfinancial disclosures and highlighted areas for staff to consider in various sections of the 10-K. Two of six accounting review staff with whom we spoke noted that this training was thorough and said they refer to training materials when conducting 10-K filing reviews. Additionally, most accounting review staff told us they can consult legal staff within their industry offices during reviews as necessary. According to Corporation Finance officials, attorneys may still participate in reviews of 10-K filings. Accounting staff also noted that they previously reviewed nonfinancial information within the context of financial disclosures as part of their financial reviews of 10-K filings. Corporation Finance has conducted assessments of samples of public companies’ 10-K filings to examine the amount and type of disclosure on selected ESG topics. Overall, Corporation Finance staff found that most sampled companies included disclosure of selected ESG topics within 10- K filings and told us they did not issue additional guidance or interpretive releases on these topics following these assessments. Climate change disclosures: In 2012 and 2014, SEC staff issued mandated reports to the Senate Committee on Appropriations that assessed the compliance of climate change disclosures included in a sample of 60 companies’ 10-K filings in selected industries. The Committee had required these reviews following SEC’s issuance of its interpretive release on climate change disclosures in 2010. SEC staff found that most sampled companies included climate-related information within their 10-K filings with varying levels of detail. Since 2014, Corporation Finance has conducted additional internal assessments on these topics that have resulted in findings consistent with previous reviews. Additional ESG-related disclosures: In recent years, Corporation Finance staff conducted additional assessments of disclosures related to some ESG topics. These assessments involved staff reviewing the disclosures of a sample of companies’ filings and evaluating compliance with disclosure requirements. Corporation Finance found that while the level of detail among disclosures varied, nearly all companies included the relevant ESG topic within their filings. Additionally, Corporation Finance staff outlined action items for the Division, such as providing comments to companies as appropriate and monitoring press reports for information that may be material for companies to disclose. In addition to internal assessments, SEC has taken steps to identify significant emerging disclosure issues through the creation of the Office of Risk and Strategy within Corporation Finance. According to Corporation Finance officials, this office was created in February 2018 and was allocated additional resources in October 2019 to support its risk surveillance function, in which it identifies emerging issues that may be material for public companies by reviewing press articles, speeches, and information from other sources such as industry experts. According to Corporation Finance officials, once the office identifies an issue that may present material disclosure risks, it may perform research and analysis that can determine whether further internal or external guidance may be necessary. Corporation Finance officials also noted these efforts may result in additional guidance to review staff based on topics identified. Investors and market observers have proposed a range of policy options to improve the quality and usefulness of ESG disclosures. These options include legislative or regulatory action to require or encourage certain ESG disclosure practices, as well as private-sector approaches, such as industry-developed frameworks and stock-exchange listing requirements. These policy options can pose important trade-offs in relation to the extent to which they impose specific new disclosure requirements or encourage companies to voluntarily adopt certain ESG disclosure practices. For example, while new ESG-related requirements may help achieve greater comparability in ESG disclosures across companies and reduce investor demands on public companies, voluntary approaches may provide more flexibility to companies while limiting potential costs associated with disclosing ESG information that may not be relevant for their business. Some institutional investors and market observers have proposed new legislative or regulatory requirements to enhance public companies’ ESG disclosures. These actions could take the form of new requirements for specific ESG disclosures, a new SEC regulation that endorses the use of an ESG disclosure framework, or new SEC interpretive releases on ESG disclosure topics. Some market observers have recommended that SEC issue new rules requiring issue-specific ESG disclosures, such as disclosures related to climate change. For example, one investor association said that it has supported various petitions and requests for rulemaking at SEC on environmental and human capital issues. SEC has taken steps to consider these types of issue-specific ESG disclosures. For example, in August 2019, SEC proposed including disclosure topics related to human capital resources and management in the description of business section of Regulation S-K. The rule has not been finalized, but in comment letters to SEC on the proposed rule, some organizations requested more line-item disclosures and metrics on this topic. Gender Pay Gap Disclosure Requirements in the United Kingdom (UK) In 2017, the UK required issue-specific disclosure rules for large companies to report the difference in average pay for male and female employees, according to a report by the UK House of Commons’ Business, Energy, and Industrial Strategy Committee. An intended benefit of gender pay gap disclosure is achieving greater equity in pay by gender and improved economic performance among UK companies, according to this committee report. However, the committee found in its 2018 review of this reporting that some companies were unsure how to account for alternative compensation, such as child care vouchers and bonuses, and that additional guidance was necessary to help companies standardize their disclosures. The committee’s report also recommended that the government mandate narrative disclosures where companies explain their action plan for closing any gender pay gap they may have. As previously mentioned, most investors told us they seek comparable information across companies, which line-item disclosure requirements may facilitate. Increasing comparability across companies also may reduce investor demands on companies, which have been increasing the last 5 years, according to most companies with whom we spoke. Additionally, requiring ESG disclosures in companies’ regulatory filings— rather than across multiple locations—could reduce information disparities between large and small investors, because the information would be located in a single place that was readily available to everyone. For example, some third-party data providers, which compile ESG information from various sources, may be prohibitively expensive to individual investors and small advisors, according to a study commissioned by the Department of Labor. One impediment to improved ESG disclosures that some institutional investors, companies, and market observers with whom we spoke cited was the lack of consensus around what information companies should be disclosing. Focusing on issue-specific ESG disclosure rules could allow SEC to enhance disclosures on the most pressing issues that may have more consensus, according to two academics we interviewed. As previously discussed, our review found that several ESG factors were commonly disclosed by companies across industries, including board accountability, climate change, and workforce diversity. On the other hand, regulatory requirements that necessitate new or additional disclosures may increase compliance costs for companies. None of the 18 companies with whom we spoke had quantified the costs associated with their ESG reporting. However, companies generally said that collecting and reporting ESG information required input from employees across the company. Three companies said ESG reporting represented an increasing opportunity cost as employees spent more time on reporting and away from business activities. Data not used in regular business operations or data that required outside assurance were the most costly disclosures, according to some companies. In addition, some market observers have noted that issue-specific rules can become outdated as issues evolve and that these types of disclosures would reduce flexibility for companies. Line-item or issue- specific disclosures also may not be relevant for all companies, possibly resulting in large volumes of immaterial information. According to one academic, compelling companies to disclose on issues that may not be relevant to them could distract companies from using resources on the relevant disclosures. Other market observers recommended that SEC issue a new rule endorsing one or more comprehensive ESG reporting frameworks, such as SASB or GRI, for companies’ reporting of material ESG issues. SEC has required the use of frameworks in other rulemakings, such as rules related to companies’ evaluation and disclosure of their internal controls. For that rule, SEC endorsed the Committee of Sponsoring Organizations of the Treadway Commission (COSO) Framework as satisfying regulatory requirements. In its evaluation of several countries’ reporting policies, the United Nations Environment Programme recommended regulators use existing international standards and guidelines when developing sustainability reporting policies. Regulations that endorse one or more frameworks could maintain flexibility for companies, because companies could choose which parts of the framework are relevant to their businesses. In addition, frameworks can be updated over time without necessitating new rulemaking in contrast to issue-specific requirements that could become outdated. Some institutional investors and companies with whom we spoke noted the importance of flexibility if there were to be any new regulation for ESG disclosures. Additionally, frameworks could encourage companies to disclose on a wide range of ESG issues. Most investors told us they focused on a broad array of ESG issues in their analyses. European Union Directive Endorsement of ESG Frameworks A 2014 European Union directive that endorsed companies’ use of existing frameworks to report how they manage social and environmental challenges has needed several updates to improve comparability across companies, according to a report by the European Securities and Markets Authority (ESMA). In 2017 and 2019, the European Commission issued voluntary guidelines for the directive that encouraged companies to use an established disclosure framework to make nonfinancial information easier to report and compare, according to ESMA. However, respondents to a 2019 survey by ESMA said that among other obstacles, the lack of specificity in the directive’s requirements and the use of various frameworks contributed to a lack of comparability among companies’ environmental, social, and governance (ESG) disclosures. As a result, ESMA recommended the European Commission amend the directive to include both general principles for reporting ESG information as well as a set of specific, universal disclosures. However, companies reporting based on different frameworks may limit comparability across companies, and there was not consensus on which framework companies should use. While some institutional investors told us they supported SASB’s framework, investors also mentioned other frameworks such as GRI, TCFD, and CDP. In a 2019 survey of 46 global institutional investors, a consulting firm found that agreeing on ESG standards that are relevant to companies’ performance was a challenge. Additionally, the Chamber of Commerce noted that companies said in roundtable discussions that the lack of universally accepted ESG reporting standards was a major challenge to effective ESG reporting. There have been initiatives recently to standardize ESG frameworks. However, a project to improve comparability across frameworks found that there were already high levels of agreement between climate change disclosures standards and that standard-setting organizations needed to more clearly communicate how their standards were interconnected. Additionally, companies reporting under a framework may choose not to disclose certain ESG information, which could result in less comparability. As previously discussed, among the company disclosures we reviewed, we identified instances of calculation inconsistency among quantitative disclosures for companies that reported information according to GRI— the most prevalent reporting framework in our sample—because GRI does not always include prescriptive disclosure recommendations and sometimes allows for different calculation methods. Some institutional investors and companies with which we spoke indicated that additional SEC interpretative releases addressing how ESG topics fit within existing disclosure requirements could be helpful. These releases can highlight the importance of ESG disclosures without requiring a rule change, because they clarify without changing the existing disclosure requirements. Some investors and SEC review staff said that interpretive releases serve as a good reminder for companies to consider ESG issues in their disclosures. Interpretive releases also maintain flexibility for companies to disclose the information that is material for each company. However, two market observers noted that because these releases do not create new disclosure requirements, they may not have much impact on ESG disclosures on their own. About half of the companies told us previous SEC releases had been helpful, but most investors said disclosures on these issues remain inconsistent. Eight of 18 companies said SEC’s previous releases on climate change and cybersecurity had helped create an even playing field for companies or underscored the need for more transparency on these issues, among other things. However, two investors and one international organization noted that the release on climate change did not appear to expand disclosure of climate change risk among U.S. companies. As previously discussed, SEC staff reviewed samples of company’s disclosures on climate change and found that most sampled companies included climate-related information within their 10-K filings with varying levels of detail. As a result, SEC staff decided against recommending that the Commission issue additional releases. Some institutional investors, companies, and market observers have cautioned against legislative and regulatory intervention in ESG disclosures and have recommended private-sector approaches to improve companies’ ESG disclosures. One advantage of private-sector approaches is that because they are voluntary, they provide companies with flexibility. Some investors and companies said flexibility was important in ESG reporting because the relevance of ESG issues can vary by company and change over time. Conversely, because ESG disclosures remain voluntary under these approaches, companies may choose not to use them in their reporting. Private-sector approaches could include industry-developed frameworks and stock exchange listing requirements. Some market observers with whom we spoke recommended that industries develop their own industry-specific ESG framework. For example, Edison Electric Institute and the American Gas Association partnered to develop standards to guide electric and natural gas companies’ ESG reporting. According to the American Gas Association, the framework was created to provide the financial sector with more uniform and consistent ESG data and information. SASB’s framework also provides industry-specific standards, covering 77 different industries. Industry-specific standards focus on ESG issues that industry representatives believe are relevant to that industry. Some investors, companies, and market observers said that ESG issues vary by industry and therefore industry-specific standards are preferred. As previously discussed, we identified some differences in the amount of disclosures on specific ESG topics between industries. Agreed-upon industry-specific standards provide consensus across various stakeholders and provide comparability of ESG disclosures across companies, according to some market observers, which also may reduce investor demands on companies. One disadvantage of relying on industries to create standards is that some industries may be diverse and unable to find consensus on standards. For example, two companies told us that their unique business model does not fit into one industry group. Company and trade association interests also may conflict with those of investors and other stakeholders. According to two academics with whom we spoke, individual companies do not have an incentive to work towards standardized ESG reporting standards and will not do so on their own. In some countries, stock exchanges have used ESG disclosure listing requirements to try to improve companies’ disclosures. The United States has several stock exchanges that list publicly traded companies, and none have extensive ESG disclosure listing requirements. NASDAQ produces a voluntary ESG reporting guide for companies and the New York Stock Exchange, as a subsidiary of the Intercontinental Exchange, has declared its support for ESG disclosures of its listed companies, but neither requires such ESG reporting to be listed on its exchange. Johannesburg and Tokyo Stock Exchange Listing Requirements Stock exchanges in Japan and South Africa are examples where listing requirements have been implemented to improve public companies’ environmental, social, and governance (ESG) reporting in those countries. According to officials from Japan’s Financial Services Agency, listing requirements on the Tokyo Stock Exchange have helped change how Japanese companies disclose ESG-related information and engage in proactive risk management. Similarly, officials from the Johannesburg Stock Exchange said that its listing requirements have had a positive impact on companies’ integrated reporting, which includes ESG information. However, these officials stated that other factors also have contributed to the increase in integrated reporting in South Africa. These include an understanding by local companies of how ESG factors affect their day-to-day operations and increased investor interest in ESG disclosures. According to research comparing integrated reporting in 10 countries, a number of factors contributed to South African companies high-quality integrated reports, including a framework for integrated reporting developed by a local nonprofit organization to assist companies in meeting the listing requirements. according to two industry studies. One third-party data provider noted that listing requirements provide an incentive—listing on the exchange— for companies to report on ESG issues. However, competition between U.S. stock exchanges could give companies alternative listing opportunities if one stock exchange enacted ESG disclosure listing requirements. According to officials from the Johannesburg Stock Exchange, as commercial entities, stock exchanges may choose to avoid imposing mandatory listing requirements on companies because they would risk losing listings that generate revenue to other exchanges or discouraging companies from listing publicly. Finally, some institutional investors, companies, and market observers noted that it was too early to prescribe standards for ESG disclosures, because there is not consensus among companies, investors, and market observers on which ESG issues should be disclosed. The marketplace should be given time to resolve these issues, according to these market participants and observers. Government officials in the United Kingdom and Japan and industry association representatives from South Africa noted that increased investor interest prompted more meaningful ESG disclosures from companies in their countries. However, they said that nonfinancial reporting requirements can be a catalyst for changing attitudes towards ESG disclosures. We provided a draft of this report to SEC for review and comment. SEC provided written comments that are reprinted in appendix II. SEC also provided technical comments, which we incorporated as appropriate. In its written comments, SEC generally concurred with our findings and stated that our report will contribute to the ongoing discussion around ESG disclosures among public companies, investors, and policy makers. SEC also highlighted some of its related activities, such as issuing interpretive releases on climate change and cybersecurity and soliciting public comments on disclosure requirements. In addition, SEC reiterated its commitment to materiality as the foundational principle for public company disclosure requirements. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 4 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Chairman of the Securities and Exchange Commission, and other interested parties. In addition, the report will be available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or clementsm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines (1) why and how investors have sought additional environmental, social, and governance (ESG) disclosures; (2) how public companies’ disclosures of selected ESG factors have compared within and across selected industries; (3) steps the Securities and Exchange Commission (SEC) staff have taken to assess the effectiveness of the agency’s efforts to review the disclosure of material ESG factors; and (4) the advantages and disadvantages of policy options that investors and market observers have proposed to improve ESG disclosures. To obtain information about why and how investors have sought additional ESG disclosures, we reviewed relevant reports and studies by academics, investment firms, and others published in the last 5 years. We identified these reports and studies through interviewing investors and market observers, reviewing sources cited in documents we obtained, and conducting internet searches. These reports and studies provided investor perspectives on issues related to ESG disclosures, including how investors use ESG disclosures, the types of ESG disclosures investors seek from companies, and investors’ use of shareholder proposals to request ESG information. In addition, we selected a nongeneralizable sample of 14 institutional investors and conducted semi-structured interviews with them to obtain information and perspectives on how and to what extent they incorporate ESG information into their investment decisions, why they do or do not incorporate ESG information, and why and how they engage with companies around these disclosures. Institutional investors include public and private entities that pool funds on behalf of others and invest the funds in securities and other investment assets. For our sample, we selected private-sector asset management firms and public pension funds of varying size: four large private asset management firms (each with more than $1 trillion in worldwide assets under management as of December 31, 2018); three mid-sized private asset management firms (each with from $500 billion to $1 trillion in worldwide assets under management as of December 31, 2018); three large public pension funds (each with more than $100 billion in total assets as of September 30, 2018); and four mid-sized public pension funds (each with from $40 billion to $100 billion in total assets as of September 30, 2018). To get a mix of regional perspectives, we incorporated geographic location into our selection when possible. For example, we selected at least one of the seven public pension funds from each of four U.S. census regions (Northeast, South, Midwest, and West). The information collected from this sample of institutional investors cannot be generalized to the larger population of all institutional investors. To obtain information about the extent to which investors have used shareholder proposals to promote improved ESG disclosures, we analyzed proposals submitted to a stratified random sample of 100 companies listed as of October 4, 2019, on the S&P Composite 1500, which combines three indices—the S&P 500, the S&P MidCap 400, and the S&P SmallCap 600 (see table 4). For our sample, we refer to companies appearing in the S&P 500 as large, companies in the S&P MidCap 400 as mid-sized, and companies in the S&P SmallCap 600 as small. With this probability sample, each company on the S&P Composite 1500 had a nonzero probability of being included, and that probability could be computed for any company. We stratified the population into three groups on the basis of company size, and each sample element was subsequently weighted in the analysis to account statistically for all the members of the population, including those that were not selected. All sample estimates in this report are presented along with their 95 percent confidence intervals. For each company in our sample, we obtained and reviewed its definitive proxy statement for the annual meeting that took place in calendar year 2019 to identify shareholder proposals. Using a data collection instrument, we analyzed each shareholder proposal submitted to a company in our sample to determine if it was related to ESG disclosures, what type of ESG disclosure it was requesting (environmental, social, or governance), and what type of investor (such as individual, labor union, or pension fund) requested the proposal. For any company in our sample that disclosed one or more shareholder proposals in its definitive proxy statement, we obtained and reviewed the company’s 8-K that included the number of votes each proposal received at the company’s annual meeting. We then calculated the percentage of votes in favor of the proposal, using the number of votes shareholders cast in favor of the proposal divided by the sum of votes cast in favor, against, and to abstain. We downloaded these SEC filings from its online Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system. To compare public companies’ ESG disclosures within and across industries, we identified and analyzed disclosures related to eight ESG factors by 32 large and mid-sized public companies across eight industries. First, we judgmentally selected eight ESG factors by reviewing ESG factors frequently cited by a range of market observers (such as ESG standard-setting organizations, academics, nonprofits, and international organizations) as being important to investors or possibly material for companies in several industries and through discussions with market observers, including two ESG standard-setting organizations and one investor association. We selected eight factors that were among the most frequently cited, including at least two from each of the three categories of ESG (environmental, social, and governance). The eight ESG factors we selected were (1) climate change, (2) resource management (water and energy), (3) human rights, (4) occupational health and safety, (5) personnel management, (6) workforce diversity, (7) board accountability, and (8) data security. We then judgmentally selected 33 specific topics to represent company disclosures on the eight ESG factors. Among these 33 specific topics, we selected 16 narrative disclosure topics that companies can address by providing a narrative discussion of ESG-related risks and opportunities and their management of them and 17 quantitative disclosure topics that companies can address by providing numbers and percentages. We selected these topics by reviewing four ESG disclosure frameworks and identifying commonly occurring disclosure topics associated with the selected ESG factors. For a list of the ESG factors and topics we selected, see figure 1 in the body of the report. We then selected a nongeneralizable sample of 32 large and mid-sized public companies to review their disclosures on the eight ESG factors and 33 ESG topics. First, we judgmentally selected eight industries from which to select public companies. We identified industries that were likely to disclose information on the selected ESG factors; had multiple companies included in the S&P 500; and, when taken together, represented a diverse range of industry sectors. The eight industries we selected were (1) airlines, (2) beverages, (3) biotechnology and pharmaceuticals, (4) commercial banks, (5) consumer retail, (6) electric utilities, (7) internet media and services, and (8) oil and gas production. We used industry classifications from the Standard Industrial Classification system, which SEC’s Division of Corporation Finance uses as a basis for assigning review responsibilities for industry groups. We then selected four public companies within each of these eight industries for a total of 32 companies. We selected four companies per industry that were among the eight largest in terms of market capitalization and that, when considered collectively within industries, provided representation across different U.S. regions. We limited our selection to U.S. public companies that were traded on either of the two largest American stock exchanges. The information collected from this sample of public companies cannot be generalized to the larger population of all public companies. We reviewed recent regulatory filings for these companies and voluntary reports, such as corporate social responsibility reports, to identify relevant disclosures on the selected ESG topics. We reviewed companies’ 2018 10-Ks, 2019 definitive proxy statements (which typically covered the same reporting period as the 2018 10-K), and 2018 annual reports (when different from the 10-K). We also reviewed companies’ most recent sustainability reports available on their websites, accessed from July through December 2019. We defined a sustainability report as a voluntary, stand-alone document that provided information on sustainability and other issues related to environmental, social, and governance factors. Companies can use other means to report ESG information, such as their websites or issue-specific company reports. We did not include single-issue documents or information included on websites that was not also part of the sustainability report. There are several reasons why a company may not disclose information on a specific ESG topic; for example, the topic may not be relevant to its business operations or the company may not consider it to have a significant enough impact on its financial performance to warrant disclosure. To identify relevant disclosures, we searched each document for a list of keywords related to each of the eight ESG factors to help identify passages likely to contain ESG disclosures on the 33 specific ESG topics. We selected these keywords by reviewing the 33 topics we selected and identifying unique terms associated with them. We categorized each narrative disclosure as being generic or company-specific. We categorized a narrative disclosure as company-specific if it included details about how ESG-related risks and opportunities affect the company’s specific operations or how the company manages these risks or opportunities. Otherwise, we characterized the narrative disclosure as generic. Generic narrative disclosures are disclosures that could apply to the reporting company as well as to many of its peers. We considered each disclosure as a whole and, if it provided some company-specific information, we categorized the disclosure as company-specific. In addition, we conducted semi-structured interviews with representatives of 18 of the 32 selected companies to obtain their perspectives on how they determine what ESG information to disclose, where to disclose it, and the benefits and challenges of ESG reporting. We requested interviews with all 32 of the selected companies, but eight companies declined and six companies did not respond to our request. For those that did not respond, we made at least three requests by email. We interviewed at least one company from each of the selected industries. Furthermore, through the semi-structured interviews with investors described above, we obtained investors’ perspectives on characteristics of ESG disclosures that may limit their usefulness to investors. To understand SEC’s current regulatory framework for overseeing public companies’ disclosures, we reviewed relevant laws and regulations, such as Regulation S-K and the Sarbanes-Oxley Act of 2002. To review SEC’s efforts related to ESG disclosures, we reviewed relevant SEC policies and procedures, such as internal guidance and SEC’s interpretive releases to public companies on climate change and cybersecurity disclosures. We also reviewed SEC’s 2012 and 2014 reports on climate change disclosures to the U.S. Senate Committee on Appropriations. We reviewed additional internal SEC assessments on selected ESG- related topics to obtain information on steps taken by SEC to review ESG disclosures. To obtain information on how staff conduct reviews of annual 10-K filings and ESG information, we interviewed SEC officials from the Division of Corporation Finance and a nongeneralizable sample of 15 review staff from the same division (six attorneys, six accountants, and three office chiefs). For our sample, we judgmentally selected staff in industry groups in accordance with those selected for our sample of public companies and with varying levels of tenure at SEC. The information collected from this sample of SEC review staff cannot be generalized to the larger population of all SEC review staff. To identify relevant policy proposals to improve ESG disclosures, we reviewed reports and public statements from investors, ESG standard- setting organizations, and other groups that provided their perspectives on the current state of ESG disclosures and potential policy proposals, including advantages and disadvantages of these proposals. For example, we reviewed letters submitted by various groups to SEC in response to its 2016 request for public comment on possible changes to regulation S-K, as well as press releases by large asset management firms. We conducted searches of government and academic literature for research on ESG disclosures from the previous 5 years. We searched the internet and various databases, such as ProQuest Newsstand Professional and Scopus. Using broad search terms, we identified articles related to our research objectives that provided useful context and discussion topics for interviews with market observers, investors, and companies. We also identified relevant reports and studies through investor and market observer interviews, by reviewing sources cited in documents we obtained, and through internet searches. In addition, we reviewed reports and studies on international ESG disclosure requirements to identify and obtain information about relevant policy approaches implemented in other countries. We interviewed government officials in the United Kingdom and Japan and stock exchange and industry association representatives from South Africa to obtain their perspectives on the quality of ESG disclosures in their countries and the advantages and disadvantages of their current ESG disclosure laws and policies. We selected these countries for interviews because each had implemented one or more of the ESG policies that had been discussed as potential policy proposals by investors and market observers in the United States. Finally, we interviewed a nongeneralizable sample of 13 market observers selected to represent a range of stakeholders, including ESG standard-setting organizations, academics, and representatives of industry and investor groups, to obtain their perspectives on issues and policy options related to ESG disclosures. We selected these market observers through studies and reports of companies ESG disclosures that identified leading observers with subject matter expertise and through referrals obtained during interviews for this study. We also used information obtained from our interviews with investors and companies to inform our analysis for this objective. We conducted this performance audit from January 2019 to July 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Michael Clements at (202) 512-8678 or clementsm@gao.gov. In addition to the contact named above, John Fisher (Assistant Director), Katherine Carter (Analyst in Charge), Emily Bond, Rachel DeMarcus, David Dornisch, Justin Fisher, Christopher Lee, Elizabeth Leibinger, Efrain Magallan, Adam Martyn, Patricia Powell, Jena Sinkfield, Tyler Spunaugle, Winnie Tsen, and Jack Wang made key contributions to this report.", "summary": "Investors are increasingly asking public companies to disclose information on ESG factors to help them understand risks to the company's financial performance or other issues, such as the impact of the company's business on communities. The Securities and Exchange Commission requires public companies to disclose material information—which can include material ESG information—in their annual 10-K filings and other periodic filings. GAO was asked to review issues related to public companies' disclosures of ESG information. This report examines, among other things, (1) why investors seek ESG disclosures, (2) public companies' disclosures of ESG factors, and (3) the advantages and disadvantages of ESG disclosure policy options. GAO analyzed 32 large and mid-sized public companies' disclosures on 33 selected ESG topics. Among other criteria, GAO selected companies within eight industries that represented a range of sectors in the U.S. economy and selected ESG factors that were frequently cited as important to investors by market observers. GAO also reviewed reports and studies on ESG policy proposals and interviewed 14 large and mid-sized institutional investors (seven private-sector asset management firms and seven public pension funds), 18 public companies, 13 market observers (such as ESG standard-setting organizations, academics, and other groups), and international government, stock exchange, and industry association representatives. Most institutional investors GAO interviewed (12 of 14) said they seek information on environmental, social, and governance (ESG) issues to better understand risks that could affect company financial performance over time. These investors added that they use ESG disclosures to monitor companies' management of ESG risks, inform their vote at shareholder meetings, or make stock purchasing decisions. Most of these institutional investors noted that they seek additional ESG disclosures to address gaps and inconsistencies in companies' disclosures that limit their usefulness. GAO's review of annual reports, 10-K filings, proxy statements, and voluntary sustainability reports for 32 companies identified disclosures across many ESG topics but also found examples of limitations noted by investors. Twenty-three of 32 companies disclosed on more than half of the 33 topics GAO reviewed, with board accountability and workforce diversity among the most reported topics and human rights the least. Disclosure on an ESG topic may depend on its relevance to a company's business. As shown in the figure, most companies provided information related to ESG risks or opportunities that was specific to the company, though some did not include this type of company-specific information. Additionally, differences in methods and measures companies used to disclose quantitative information may make it difficult to compare across companies. For example, companies differed in their reporting of carbon dioxide emissions. Policy options to improve the quality and usefulness of ESG disclosures range from legislative or regulatory action requiring or encouraging disclosures, to private-sector approaches, such as using industry-developed frameworks. These options pose important trade-offs. For example, while new regulatory requirements could improve comparability across companies, voluntary approaches can provide flexibility to companies and limit potential costs.", "document_type": "gao"}
{"report": "The organization that is now FAMS was created in 1961 to counter hijackers. The Aviation and Transportation Security Act, enacted in November 2001, established TSA as the agency responsible for civil aviation security and transferred FAMS along with other aviation security- related responsibilities from the Federal Aviation Administration to TSA. Among other things, the Act expanded FAMS’s mission and workforce in response to the September 11, 2001, terrorist attacks. Specifically, the Act authorizes TSA to deploy air marshals on every passenger flight of a U.S. air carrier and requires TSA to deploy air marshals on every such flight determined by the TSA Administrator to present high security risks—with nonstop, long-distance flights, such as those targeted on September 11, 2001, considered a priority. As of August 2019, FAMS had thousands of employees and 20 field offices across the United States. FAMS’s Field Operations Division consists, in part, of these field offices, which are divided into regions overseen by Regional Directors. A Supervisory Air Marshal in Charge (SAC) manages each field office, assisted by a Deputy Supervisory Air Marshal in Charge or Assistant Supervisory Air Marshals in Charge, depending on the size of the field office. SFAMs typically oversee squads of air marshals in the field offices. FAMS’s Flight Operation Division consists of the Systems Operation Control Section, among other groups. The Systems Operation Control Section is responsible for planning and preparing air marshals’ schedules, which are based on 28-day cycles known as roster periods. It is also responsible for monitoring all FAMS missions. For example, its Mission Operations Center is responsible for providing real-time support to air marshals performing missions by resolving mission-related issues, including last-minute scheduling changes. The senior leader of FAMS is the Executive Assistant Administrator / Director of FAMS. Given that there are many more U.S. air carrier flights each day than can be covered by air marshals, FAMS uses a concept of operations to set forth its methodology for deploying air marshals. FAMS’s concept of operations prioritizes flights that it considers higher risk, such as those for which a known or suspected terrorist is ticketed. FAMS refers to these flights as Special Mission Coverage (SMC) and, according to FAMS Flight Operation Division officials, FAMS typically learns of them no more than 72 hours in advance of flight departure and sometimes less than an hour before departure time. According to Flight Operations Division officials, in March 2018 FAMS adopted a new concept of operations that expanded the number of SMCs. To cover SMCs, FAMS uses air marshals scheduled to standby status, who report to their home airport and fly upon notification. If no air marshals in standby status are available, FAMS may reassign air marshals from regularly scheduled missions or air marshals who were not scheduled to fly at that time. FAMS has established scheduling guidelines intended to balance mission needs with air marshals’ quality of life. Specifically, Systems Operation Control Section officials maintain guidelines detailing parameters for shift length and rest periods when scheduling air marshals to fly missions. Exceptions to these guidelines are permitted to meet mission needs and the Mission Operations Center is not restricted by the guidelines when addressing mission scheduling issues, such as flight delays. For an overview of FAMS’s scheduling guidelines for shift length and rest, see figure 1. Air marshals are expected to be available to work as needed, 24 hours a day. To compensate air marshals for the demands of their position, air marshals receive law enforcement availability pay, which provides eligible TSA law enforcement officers, including air marshals, a 25 percent increase in their base pay for working or being available to work an annual average of 2 hours or more of unscheduled overtime per regular workday. In addition to law enforcement availability pay, certain air marshals are eligible to receive overtime pay after working more than 85.5 hours in a single 14-day pay period. Based on FAMS 2019 human capital data, approximately 85 percent of FAMS employees are law enforcement officers (e.g., air marshals). FAMS’s law enforcement workforce is largely White, male, and 40 years of age or older. As of August 2019, 68 percent of FAMS law enforcement employees identified as White, followed by 14 percent Hispanic or Latino, 12 percent Black or African American, 3 percent Asian, 1 percent American Indian or Alaskan Native, and 1 percent identified as Other or more than one race. Also as of August 2019, approximately 94 percent of FAMS law enforcement employees were male, approximately 76 percent were aged 40 or older, and approximately 51 percent have been with the agency since 2002. See figure 2. Air marshals report being concerned about their health. Air marshals in all six offices we visited stated that health issues are a key quality of life concern. The most common health issues air marshals raised in discussion sessions with us were extreme fatigue, mental health issues, difficulty maintaining a healthy diet, and increased frequency of illness. In addition, OPM’s FEVS survey asked FAMS employees whether they “believe they are protected from health and safety hazards.” DHS estimates that in fiscal year 2018—the most recent year for which complete FEVS results are available—less than half (44 percent) of FAMS employees believed they were protected from health and safety hazards. Moreover, during the 6-year period from fiscal year 2013 through 2018—a period during which the number of FAMS employees decreased by 17 percent—the number of workers’ compensation claims filed by FAMS employees nearly quadrupled, from 71 claims to 269 claims. From fiscal year 2013 through 2019, thirteen air marshals died while employed with FAMS, one of whom died while on duty covering a flight. According to FAMS officials, five of the thirteen deaths were caused by suicide; and FAMS officials did not know the cause of death for the other eight. Concerns about air marshals’ health are long-standing. For example: In 2008, a FAMS Medical Issues Working Group reported that air marshals had experienced various types of health issues—poor physical fitness as well as musculoskeletal injuries and upper respiratory infections. The Working Group also noted that air marshals’ disrupted sleep patterns often resulted in fatigue and long hours and made it difficult for air marshals to work out and maintain healthy eating habits. In 2012, the FAMS-commissioned Harvard sleep and fatigue study— which included a literature review, an analysis of air marshals’ work schedules, and a survey of air marshals—reported that shift work schedules, like air marshals’ flight schedules, can cause significant acute and chronic sleep deprivation which in turn can adversely affect their personal health, such as increasing the risk of heart disease. The study also reported that sleep deprivation degrades air marshals’ ability to think quickly, make good decisions, and to recognize when fatigue impairs performance and safety. In 2013, a FAMS review of air marshals’ fitness noted that air marshals were experiencing high injury rates when taking their physical fitness assessments and declining overall health and wellness. FAMS officials attributed air marshals’ declining overall health and wellness in part to the increasing age of air marshals. FAMS has had initiatives in place to assess air marshals’ health. For example, since 2004 FAMS has required that individual air marshals obtain a medical examination at least every 2 years. In addition, FAMS has operated a Health, Fitness, and Wellness Program since 2015 and a Hearing Conservation Program since 2017. However, FAMS maintains limited health information in a data system. Since 2004, FAMS has gathered information on individual air marshals’ health to help ensure employees meet its medical standards. Specifically, FAMS has required that air marshals obtain a medical examination from private, FAMS-approved clinics at least every 2 years. According to FAMS policy, these exams are to assess air marshals’ cognitive, physical, psychomotor, and psychological abilities and include certain cardiac, pulmonary, audiometric, and visual tests. FAMS’s Medical Programs Section—an office staffed with one part-time physician, five nurses, and three administrative staff—is responsible for helping ensure that air marshals obtain their required medical examinations. The office also follows up if an exam indicates an air marshal may have a health issue that may affect their ability to perform their duties, such as a sleep disorder or high blood pressure. Clinicians who conduct the periodic medical examinations provide the Medical Programs Section a medical report, which they use to determine if an air marshal is medically qualified to perform the essential functions of the position in a safe and efficient manner. Air marshals deemed unqualified to perform one or more essential functions of the position, with or without reasonable accommodation, are subject to administrative actions, such as being placed on light or limited duty status and possibly non-disciplinary removal based on medical inability to perform the essential function of the position. FAMS officials report, however, that they have not entered air marshals’ medical information, including their medical qualification status, into a data system because medical information is protected by law and their existing data system—the Federal Air Marshal Information System (FAMIS) is not suitable to maintain medical information. Instead, the Medical Programs Section maintains the results of air marshals’ medical exams—including their qualification status—in paper files. Medical Programs Section officials explained that because medical information about air marshals are not in a data system, reviewing and compiling information to obtain a comprehensive assessment—such as the number of air marshals who are medically qualified—would be resource-intensive. Medical Programs Section officials noted that it would be helpful to be able to analyze air marshals’ health records to identify any trends across the workforce. FAMS officials report that by the end of September 2020 the Medical Programs Section plans to review and evaluate software platforms that would be suitable for medical data. However, these same officials reported that, as of September 2019, the work on this initiative had been verbal and informal so they were not able to provide documentation of this effort. OPM’s 2018 report on human capital management highlights the importance of using data to conduct workforce analyses to help identify and properly address human capital challenges. Without information about the number and proportion of the FAMS workforce who are medically qualified, FAMS management has a limited understanding of its workforce’s ability to fly missions and fulfill their duties. Further, FAMS management cannot readily identify trends among its workforce and therefore is also limited in their ability to identify any problems and make better-informed workforce planning decisions. In May 2015, FAMS initiated a Health, Fitness, and Wellness Program intended to address concerns with air marshals’ fitness and injury rates and improve air marshals’ overall health and wellness. According to FAMS policy, the program is intended to provide opportunity, resources, and education necessary to enhance mission readiness and promote workplace wellness. For example, FAMS requires air marshals to participate in a health and fitness assessment twice a year to measure their fitness including cardio-respiratory endurance, muscular strength, muscular endurance, and flexibility. FAMS physical fitness instructors administer the assessment and record the results in FAMIS, such as the number of pushups an air marshal can complete in one minute. Since February 2016, FAMS has used these data to track air marshals’ mandatory participation in the assessments and to identify individual air marshals who do not maintain their fitness levels or show improvement. However, it has not used these data to analyze trends in the fitness of the workforce as a whole. FAMS officials noted that analyzing these data could provide some indication of the state of the workforce, but they have not done so because these data provide a limited snapshot and other information would need to be considered to provide a full understanding of the workforce’s well-being. Two other aspects of the program are the establishment of Health Fitness and Wellness Coordinators and an optional Health Risk Assessment. FAMS Health, Fitness, and Wellness Program coordinators are responsible for engaging with air marshals to promote a culture of wellness, build an inclusive fitness community at each location, and provide health, fitness, and wellness recommendations. The national coordinator of the Health, Fitness, and Wellness Program is also responsible for providing oversight of the program, ensuring program effectiveness, and providing FAMS leadership with program reports and assessments when requested. According to FAMS documents, the optional Health Risk Assessment is intended to help air marshals identify modifiable health risk factors. The assessments are completed by air marshals and reviewed by a certified occupational health nurse. Air marshals then meet with FAMS Medical Programs Section staff to discuss their health and recommendations to promote health and wellness, and prevent disease. FAMS officials report that in 2015, they completed eight Health Risk Assessments; however, since then no additional air marshals have requested this assessment. Medical Programs Section officials stated that few air marshals took advantage of this option because air marshals prefer to obtain health services outside of the agency (i.e. with private providers) to maintain their privacy. In August 2017, FAMS established a Hearing Conservation Program to provide a coordinated approach to prevent hearing loss due to noise exposure in the work environment and to be compliant with federal regulations. According to FAMS documentation, air marshals are regularly or intermittently exposed to gunshot noise such as during training activities. Through this program FAMS has provided training about the adverse effects of noise and administered baseline audiograms and annual testing of air marshals. FAMS physicians are to evaluate data from the hearing screenings and conduct follow-up with individual air marshals when there is a change in the test results. FAMS officials report that they maintain these test records in the Medical Programs Section’s paper files for individual air marshals. As of July 2019, FAMS estimated that about two-thirds of air marshals had obtained baseline audiograms. FAMS officials report that they do not have plans to analyze air marshals’ audiogram results in the aggregate. Instead, FAMS officials plan to review the program at least annually to identify any enhancements that could improve program efficiency and effectiveness. FAMS began more closely monitoring certain workforce-wide data in response to management concerns that arose in 2016 about the rising costs associated with workers’ compensation claims. In 2016, it began to more closely monitor the number and costs of workers’ compensation claims. In February 2019, FAMS hired a safety specialist to begin analyzing available information on air marshals’ on-the-job injuries in an effort to identify ways to prevent them from occurring, according to FAMS officials. Although FAMS monitors certain information on workers’ compensation claims and has plans to further monitor workplace injuries, it has not used or planned to use other information it collects to assess the health of its workforce in a comprehensive manner that would enable it to look for broader health trends and risks. As previously discussed, FAMS collects and reviews in-depth health information on each air marshal at least every 2 years. However, it has not analyzed this information to distill trends across the workforce because, according to FAMS officials, it would be difficult given that FAMS maintains individual air marshals’ medical information in paper files. Similarly, FAMS routinely collects data from air marshals’ health and fitness assessments but has not used these data to identify any workforce-wide trends because, as discussed above, FAMS officials state that these data would provide a limited snapshot of air marshals’ fitness. Further, although FAMS began collecting data from hearing screenings in 2018, officials indicated that they do not have any plans to analyze these data for the workforce as a whole. Furthermore, since 2015, the National Coordinator for the Health, Fitness, and Wellness Program is responsible for providing program assessments when requested but, as of July 2019, FAMS leadership has not requested any such reports. There is evidence of interest within FAMS in information about the overall health of the workforce. In 2017, the FAMS Advisory Council asked the Medical Programs Section to report on the health and wellness of the workforce. According to documents we reviewed, in March 2017, Medical Programs Section officials reported to the advisory council that air marshals’ most common medical restrictions were due to mental health and cardiac conditions and the most common work-related medical issues were orthopedic issues resulting from training-related injuries. However, Medical Programs Section officials told us their assessment was not derived from an analysis of air marshals’ medical data but rather relied on anecdotal information gathered from on-call nurses fielding calls from sick air marshals and providing routine occupational health case management. OPM’s 2018 report on human capital management highlights the importance of using data to conduct workforce analyses to help identify and properly address human capital challenges. The FAMS Medical Programs Section and other offices regularly collect information about individual air marshals’ illnesses and injuries as well as health and fitness information but FAMS management is not analyzing it to inform decisions and address any potential health risks. If FAMS management analyzed this information in a manner consistent with relevant policies and requirements, they would be better positioned to identify medical, health, and fitness issues among the entire workforce, make informed workforce planning decisions, and take steps they deemed warranted, such as providing targeted education or revising its policies. Further, in February 2018, OPM identified “enhancing productivity through a focus on employee health” as a key priority within human capital management for the federal workforce. Four months later, in June 2018, TSA identified “care for our people” as a leadership principle and directed leaders to prioritize employee welfare. In November 2019, FAMS management officials provided us with a statement that said, in part, that “understanding the overall health and wellness of our air marshals is paramount.” They further stated that they now plan to create a working group to identify options to monitor the health of the workforce as a whole. They did not provide any timeframes or documentation of this effort. However, if implemented, this could be a good first step toward assessment of the overall health of the FAMS workforce. Without information on the overall health and fitness of the FAMS workforce, FAMS management is not well positioned to prioritize employee health and welfare or ensure that it deploys a workforce capable of fulfilling its national security mission. Air marshals in each of the six field offices we visited stated that schedule unpredictability—short-notice changes to their start times, missions, and at-home days—was a key quality of life issue. Air marshals explained that they have experienced changes to their scheduled mission days and non-mission days—such as in-office training and scheduled days off—so they could cover mission needs that came up on short notice. In addition, air marshals in four of the six field offices we visited explained that they have been taken off of their scheduled missions on short notice so they could cover higher-risk missions. Air marshals in all six field offices stated that schedule unpredictability has made it difficult to manage their personal commitments. For example, air marshals described some challenges planning and attending family events, maintaining personal relationships, obtaining childcare, and scheduling doctor’s visits for themselves and their children. Air marshals in one office also described anxiety about the possibility of missing a phone call asking them to report for a mission and about their ability to arrive to work on time when given short notice. Air marshals, supervisors, and FAMS management we met with explained that changes to FAMS’s deployment strategy in March 2018 that increased the number of SMCs have increased schedule unpredictability. According to Flight Operations Division officials, FAMS typically does not learn of these missions more than 72 hours in advance. Our analysis of FAMS data shows that the average number of SMCs per roster period more than tripled after FAMS implemented its new concept of operations in March 2018, and air marshals’ SMC-related schedule changes more than doubled during the same period. FAMS has taken some steps to mitigate the impacts of SMCs on air marshals’ schedules as follows: Implemented a standby shift and increased the number of air marshals on standby. FAMS Flight Operations Division officials report that they implemented a standby shift to staff SMCs in June 2018. According to Flight Operations Division officials, FAMS typically staffed SMCs using air marshals scheduled to domestic and international missions, recovery shifts, or ground-based duties prior to the implementation of the standby shift. Flight Operations Division officials also report that they increased the number of scheduled standby shifts in an effort to curtail schedule unpredictability. Based on our review of FAMS data, the number of scheduled standby shifts more than tripled from June 2018 to December 2018. According to these officials, scheduling air marshals on standby shifts is intended to improve schedule predictability by reducing the frequency that air marshals have their planned work schedules adjusted so they can cover SMCs. Expanded to multiple standby shifts with staggered start times and modified standby shift start times. According to Flight Operations Division officials, field office SACs reported that FAMS frequently adjusted air marshals’ scheduled start times for the single standby shift in response to SMC requests. To reduce this schedule unpredictability, Flight Operations Division officials reported that in November 2018, they began scheduling air marshals to multiple standby shifts per day with staggered start times, rather than just one shift per day. These officials stated that they received positive feedback regarding this change during management’s subsequent field office visits. We asked air marshals in four of the six field offices we visited for their perspectives on the effectiveness of this change during discussion sessions and received mixed feedback. Air marshals in two field offices stated that they thought this change had improved SMC scheduling by reducing the number of changes to standby shift start times. However, air marshals in each of these four field offices stated that Mission Operations Center personnel do not always observe air marshals’ scheduled standby shift hours. Systems Operation Control Section officials noted that the magnitude of adjustments to air marshals’ standby shift start times is not always significant. To further reduce schedule unpredictability, FAMS also began modifying standby shift start times for some of its field offices in December 2018. Flight Operations Division officials stated that they modify standby shift start times for individual field offices based on specific SMC timing trends in field offices. According to Flight Operations officials, they analyzed air marshals’ scheduled standby shift start times and actual start times both before and after these changes and concluded that they were reducing start time variance. For example, they found that between October 28, 2018, and November 24, 2018—a period during which they report using one standby shift—approximately 46 percent of actual standby shift start times deviated from scheduled start time by 4 or more hours. Between June 9, 2019, and July 6, 2019, after FAMS Flight Operation Division officials reported having expanded to multiple standby shifts and adjusted start times for individual offices, FAMS officials found that approximately 33 percent of actual standby shift start times deviated from scheduled start times by 4 or more hours. Flight Operations Division officials stated that these changes have reduced the frequency of SMCs covered by air marshals not in standby status. Our analysis of FAMS data on SMC-related schedule changes shows that FAMS reduced the need to make changes to the schedules of air marshals that were not on recovery or standby shifts in order to staff SMCs. Additionally, Flight Operations Division officials stated that they continue to monitor data on SMC start times to identify the optimal standby shift start times to reduce scheduling unpredictability. Improved coordination with field offices. In April 2019, FAMS management issued guidance aimed at improving coordination between the Mission Operations Center and field offices to reduce schedule unpredictability. First, the guidance requires that the Mission Operations Center obtain field office approval prior to adjusting an air marshal’s standby shift start time by more than 2 hours in order to staff an SMC. Second, in situations where FAMS receives a SMC request with more than 24 hours’ notice and there are no available air marshals scheduled to standby, Mission Operations Center and field office personnel are to use air marshals scheduled to recovery shifts (if they are available and at the field office’s discretion) before pulling air marshals from non-SMC missions to cover the request. According to FAMS management, this latter change is intended to reduce the number of non-SMC missions dropped to cover SMCs. FAMS management and Flight Operations Division personnel monitor some information about air marshals’ planned and actual schedules. According to Flight Operations Division officials, they routinely monitor average scheduled shift length, average actual shift length, and average scheduled rest for domestic and international missions through monthly field office-specific reports. These officials stated that field office SACs and other FAMS management officials use the reports to understand characteristics like the mission tempo in each field office. Our analysis of air marshals’ work hours as recorded on their time sheets demonstrated that air marshals’ shift lengths were generally consistent with scheduling guidelines for selected roster periods, but in each period a few shifts were not. Additionally, our analysis of air marshals’ regular days off showed that air marshals generally received 8 days off per roster period—consistent with FAMS scheduling guidelines—for the periods we analyzed. The details of that analysis are presented in appendix II. Domestic missions. Generally, FAMS schedules air marshals to shifts that range between 6.5 and 10 hours on days that they fly domestic missions, but the Mission Operations Center has the authority to extend shift lengths to 12 hours. During the four roster periods we reviewed, air marshals’ domestic mission shifts were generally shorter than 10 hours. Specifically, during the 28-day roster periods we examined in fiscal year 2019, we estimate that air marshals exclusively worked shifts lasting 10 hours or less approximately 87 percent of the time. Air marshals worked one or more shifts that extended beyond the scheduling guideline of 10 hours about 13 percent of the time. For example, during the 28-day roster periods we examined in fiscal year 2019, we estimate that air marshals worked at least one shift between 10 hours and 12 hours about 10 percent of the time and worked at least one shift that was greater than 12 hours approximately 3 percent of the time. See figure 3 for the results of our analysis of domestic mission shifts. International missions. Scheduling guidelines for international missions vary based on factors like mission destination, and some missions are not subject to a maximum duration. Given the guideline variation for international missions, we examined actual international missions against the highest international mission shift length specified by the guidelines— 18 hours—as well as guidance that requires the Mission Operations Center to consider scheduling alternatives when a delay causes an international mission shift to last beyond 20 hours. Air marshals generally worked in accordance with guidelines for international missions. Specifically, we found that air marshals generally worked shifts that lasted fewer than 18 hours during the four roster periods we analyzed. During the 28-day roster periods we examined in fiscal year 2019, we estimate that air marshals exclusively worked shifts lasting 18 hours or less approximately 71 percent of the time. Air marshals worked one or more shifts lasting more than 18 hours about 29 percent of the time. For example, during the 28-day roster periods we examined in fiscal year 2019, we estimate that air marshals worked at least one shift between 18 and 20 hours approximately 24 percent of the time and worked at least one shift greater than 20 hours about 11 percent of the time. See figure 4 for the results of our analysis of international mission shifts. FAMS management’s monthly reports on average shift lengths do not provide insight into the extent air marshals are working hours consistent with scheduling guidelines. For example, FAMS management reports for the roster periods we analyzed for fiscal years 2018 and 2019 showed that the average domestic mission shift lasted between about 6.5 and 7.5 hours. While these average times are below the 10-hour guideline for domestic mission shifts, these data are not granular enough to determine whether any air marshals worked shifts that exceeded scheduling guidelines. With regard to international missions, because FAMS’s guidelines vary more widely depending on the specifics of the mission, a single average of all international mission durations is even less useful in determining the extent to which air marshals’ work hours were consistent with applicable guidelines. For example, one FAMS management report stated that the average international mission shift length between October 29, 2017, and November 25, 2017—the first period we examined in fiscal year 2018— was 12 hours and 55 minutes. Although this average exceeds the scheduling guideline of 12 hours for international mission shifts to North and Central American destinations that do not include an overnight layover, this average is less than the guideline of 15 hours for international mission shifts to North and Central American destinations that include an overnight layover. As a result, the average shift length would not have made clear how often guidelines were being observed. FAMS’s scheduling guidelines allow for exceptions to accommodate operational needs, but more information on actual work hours could improve FAMS management’s insight into how air marshals’ quality of life is being balanced against mission needs. For example, FAMS management’s reports could include other statistics that would provide more insight into air marshals’ domestic mission shifts, such as minimum or maximum actual shift lengths or the extent of variation across actual shift lengths. Flight Operations Division officials explained that they do not monitor other statistics that could provide more insight into actual work hours because they had not identified a need to do so but stated that they could and added that more information could be helpful. Standards for Internal Control in the Federal Government requires that management use quality information to achieve the entity’s objectives by, for example, processing its data into quality information that management uses to make informed decisions. Without monitoring the extent to which air marshals’ shifts and rest periods are consistent with scheduling guidelines, FAMS management is not well positioned to determine if scheduling guidelines are serving their purpose to balance air marshals’ quality of life with FAMS’s operational needs to execute its mission, nor can it determine the extent to which air marshals are working beyond the guidelines. As a result, the agency may not be able to successfully manage risks of potentially decreased alertness and focus when air marshals perform their duties. FAMS has not made its scheduling guidelines available to all air marshals. During our visits to a non-generalizable sample of field offices, many FAMS personnel—including field office management, SFAMs, and air marshals—stated that they did not have access to scheduling guidelines. Rather, several air marshals stated that they learned of the scheduling guidelines through discussions with immediate supervisors and interactions with the Mission Operations Center. Air marshals in two field offices we visited stated that they had asked for a copy of the guidelines but were never provided one. Air marshals told us it would be helpful to have access to the guidelines so that they can understand how FAMS schedules its shifts. When we asked why the guidelines were not available to employees, Systems Operation Control Section officials reported that they were previously unaware that the field office SACs did not have access to the guidelines. In response, in June 2019, they provided Field Operations Division leadership with a document outlining the guidelines for distribution to field office SACs. However, according to Systems Operation Control Section officials, they did not explicitly direct the field office SACs to further disseminate the guidelines to air marshals in their respective field offices. As of July 2019, Systems Operation Control Section officials were not aware to what extent the document was disseminated beyond the field office SACs, if at all. FAMS scheduling guidelines are intended to balance mission needs with air marshals’ quality of life. As discussed above, these guidelines include specific parameters for shift length and rest periods when air marshals fly missions. Further, exceptions to these guidelines are permitted to meet operational needs. Standards for Internal Control in the Federal Government provides that management should implement control activities, such as FAMS scheduling guidelines, and that it is helpful for management to communicate them to personnel so they can implement them for their assigned responsibilities. Furthermore, the FAMS- commissioned Harvard sleep and fatigue study states that policies concerning work hours and scheduling need to be well communicated. Without access to the scheduling guidelines, air marshals and their supervisors may not be aware of management’s intended balance between mission needs and air marshals’ quality of life. Further, they may not feel empowered to request schedule changes that may be needed to ensure air marshals are sufficiently rested to carry out their mission. Some field office SFAMs we spoke to in our discussion sessions were not clear about protocols that require Mission Operations Center personnel to obtain their approval before making certain adjustments to air marshals’ schedules. FAMS protocols state that the Mission Operations Center can extend an air marshal’s domestic mission shift to 12 hours or reduce rest following a domestic shift to 10 hours. However, the Mission Operations Center must first obtain the approval of a field office SFAM before extending an air marshal’s domestic mission shift beyond 12 hours or reducing rest below 10 hours. SFAMs we discussed this issue with during our six site visits had varying levels of knowledge about their authority or involvement in approving such changes. For example, individual SFAMs in two field offices we visited told us they were aware of the requirements but in two other field offices, SFAMs stated that they did not have any say in adjustments to air marshals’ schedules, regardless of the circumstances. SFAMs were also unaware of field offices’ authority to remove air marshals from missions on short notice. FAMS protocols authorize, and Systems Operation Control Section officials confirmed, that field office SFAMs can remove air marshals from a mission the day of or day before the mission. However, there were SFAMs that were unaware of this in each of the four field offices where we discussed the topic. Some SFAMs had the understanding that management officials—either field office SACs or other management officials outside of field offices—or Mission Operations Center personnel must make these decisions. Systems Operation Control Section officials explained that field office SFAMs do not have access to the Standard Operating Procedure that sets forth these protocols, nor have they provided written guidance on the protocols. Systems Operation Control Section officials stated that they have not given supervisors access to these protocols or written guidance on them because they chose to communicate protocols through verbal briefings. Systems Operation Control Section officials explained that they follow the protocols and had not previously seen a need to share them more widely, but acknowledged that doing so would increase transparency. It is important that SFAMs have access to protocols outlining their role and authority so that they can carry out their job. Standards for Internal Control in the Federal Government provides that management should implement control activities through policies by, for example, communicating to personnel the policies and procedures so that the personnel can implement the control activities for their assigned responsibilities. Furthermore, the FAMS-commissioned Harvard sleep and fatigue study states that policies concerning work hours and scheduling need to be well communicated. Providing SFAMs with written information on these protocols that detail their involvement and authorities in making decisions that affect air marshals’ quality of life would provide clarity for SFAMS, who we found to be uncertain about their authorities in this regard. From fiscal years 2016 through 2018, FAMS employees filed 230 EEO complaints with TSA’s Civil Rights Division (CRD), though employees may have reported additional discrimination complaints through other means. CRD is responsible for receiving and handling FAMS employees’ EEO complaints. During this 3-year period, the number of EEO complaints CRD handled regarding FAMS employees was proportional to the number of complaints handled for employees across all of TSA, relative to the size of each workforce. Specifically, in 2018 the ratio of total complaints to total number of employees was 2.8 percent for FAMS and 2.1 percent for TSA. Although reporting to CRD is the only means for FAMS employees to file an EEO complaint, they may choose to report discrimination to their manager or to other entities including the DHS OIG or TSA’s Anti- Harassment Program, which is overseen by the National Resolution Center. The Anti-Harassment Program can take immediate action intended to stop the discriminatory behavior by, for example, separating the employees involved in the complaint. FAMS employees may also choose to report to CRD as well as one or more of the other available means. Once an employee files a complaint with any of these entities, agency officials are to follow processes to investigate the allegation to determine if the complaint is substantiated or not substantiated. See appendix III for a description of the four venues through which FAMS employees can raise discrimination complaints, including what is known about the number and nature of complaints received through each venue in fiscal years 2016 through 2018. We found that some FAMS employees may choose not to report an allegation of discrimination to any of these venues. For example, air marshals in five of the six field offices we visited indicated that they may not file a discrimination complaint because they were concerned about retaliation. Additionally, air marshals in three discussion sessions indicated that some FAMS employees may prefer to handle an allegation of discrimination themselves by speaking directly with the person involved. Further, representatives of a FAMS employee group and the professional association representing federal law enforcement officers we met with stated some FAMS employees may choose not to report an allegation of discrimination to any of these venues. As such, the 230 EEO complaints may underestimate the total number of incidents of alleged discrimination within FAMS. FAMS’s 2012 action plan identified a number of existing TSA and FAMS efforts already in place at that time—such as providing certain training— and stated FAMS’s commitment to continuing and improving these existing efforts with a goal to enhance organizational and cultural initiatives regarding diversity and equal employment opportunities. Consistent with FAMS’s 2012 plan, DHS, TSA, and FAMS have provided EEO and diversity training to FAMS employees and offered several forums for air marshals to raise concerns about discrimination. Training. Since 2003, DHS and TSA have required all employees— including air marshals—to complete training intended to, among other things, prevent discrimination. These include mandatory annual DHS training, TSA new-hire training, and some optional TSA training. For example since 2003, TSA has required new employees to complete a course called Introduction to Civil Rights which provides an overview of civil rights, EEO laws, and TSA’s related complaint process. In addition, as of December 2006, DHS has required all employees to complete annual No FEAR Act training to inform employees of their rights and responsibilities with regard to discrimination in the workplace. FAMS management officials told us that educating the workforce about discrimination is important because education promotes and opens communication avenues within FAMS that were previously underutilized. TSA has also provided training beyond these required courses. For example, CRD officials told us that at the start of each fiscal year they work with FAMS management to identify FAMS field offices where concerns about discriminatory behavior have been raised. CRD officials stated that they have then provided in-person tailored trainings based on the field offices’ needs. Additionally, in August 2019, TSA’s Anti- Harassment Program provided FAMS leadership with an overview of the program—including defining harassment and manager and employee responsibilities. According to CRD and FAMS officials, they are in the process of developing additional courses that could be helpful to preventing discrimination, including civility courses, coaching through conflict, and crucial conversations training. Venues. FAMS has venues for air marshals to raise issues, such as concerns about discrimination. Specifically, in 2002 FAMS created “Field Office Focus Groups;” in 2006 FAMS established an Ombudsman position; and in 2011 FAMS created EEO points of contact in FAMS field offices. FAMS Field Office Focus Groups. During the early ramp-up of FAMS after September 11, 2001, FAMS established an internal initiative called “Field Office Focus Groups” to provide a venue for employees to raise issues, such as concerns about discrimination, to field office management through group discussions. We reviewed Field Office Focus Group meeting minutes from all 20 field offices from October 2016 through December 2018. During these meetings, discrimination-related issues were discussed in two field offices. For example, in one focus group air marshals inquired about their recourse when they believe management has retaliated against them. FAMS Ombudsman. FAMS established a FAMS-specific Ombudsman position in 2006. The FAMS Ombudsman is responsible for answering inquiries about agency policies and helping employees identify options to resolve workplace concerns, such as concerns about discrimination. The FAMS Ombudsman we met with told us they have fielded inquiries about discrimination but they do not keep records on the number of inquiries. The Ombudsman estimated that between May 2018, when assuming the Ombudsman position, and July 2019 the office received, on average, eight calls per month from air marshals on various topics, some of which involved inquiries about discrimination. In these cases the Ombudsman explained that they had informed individuals of the resources available to them as well as the 45-day time frame to file an EEO complaint with CRD if they chose to do so. Air marshals in five of the six field offices we visited reported being aware of the Ombudsman position. EEO Points of Contact in all FAMS field offices. According to FAMS officials, in 2011, FAMS began to establish EEO points of contact in FAMS’s 20 field offices. FAMS officials report that these points of contact are intended to provide ready, onsite referrals to CRD staff and facilitate access to information about EEO and diversity training opportunities. As of August 2019, FAMS officials told us that all FAMS field offices have at least one EEO point of contact and several field offices have more than one. The FAMS 2012 action plan highlighted additional efforts to prevent discrimination but FAMS has not fully implemented or maintained these efforts. According to FAMS leadership, they have not fully implemented or continued the efforts they set forth in the 2012 action plan because the changeover in FAMS leadership since 2012 resulted in a loss of focus on implementing the plan. For example, the plan called for each FAMS field office to develop an EEO/diversity action plan to strengthen the current workplace environment. Each plan was to emphasize four principles: leadership commitment, recruitment and resourcing, career development and enhancement, and employee engagement/workplace culture. As of July 2019, none of the field offices had a diversity action plan in place. In addition, the 2012 action plan called for FAMS to continue to convene diversity focus groups. In 2010 and 2011, FAMS conducted 10 diversity focus groups to solicit input from the workforce related to recruitment, retention, discrimination, harassment, and retaliation, according to FAMS officials. However, FAMS has not held these diversity focus groups since 2011. Further, in 2007, TSA established what is now the Diversity and Inclusion Change Agents Council, which serves as a venue where TSA employees, including air marshals, can promote diversity. In the 2012 action plan, FAMS planned to have all levels of FAMS employees, including senior leadership, such as SACs and Assistant Supervisory Air Marshals in Charge, represented on the council. However as of 2019, two air marshals are the FAMS representatives on this council. Concerns with discrimination persist among air marshals. For example, FAMS employees’ fiscal year 2018 FEVS survey responses related to issues of discrimination were consistently less positive than those of DHS and TSA employees overall, although the proportion of EEO complaints among FAMS’s workforce is similar to TSA’s as a whole. Specifically, DHS estimates that less than half (44 percent) of FAMS employees feel they can disclose a suspected violation without fear of reprisal. Further, FAMS employees’ positive responses were lower than TSA and DHS employees’. Similarly, a smaller estimated percent of FAMS employees believe that prohibited personnel practices are not tolerated (FAMS 54 percent, TSA 60 percent, and DHS 62 percent). Further, as described earlier, air marshals in five of the six field offices we visited raised concerns about potential retaliation for reporting discrimination. For example, one air marshal expressed concern that they might be given undesirable travel schedules as retaliation if they filed a complaint. Finally, according to employee exit surveys conducted by TSA in fiscal years 2012 through 2018, of the 342 FAMS respondents who completed a survey, 26 (about 8 percent) cited that a reason for leaving was diversity or inclusion barriers in the workplace. Given these indications of concerns about discrimination in the FAMS work environment, it is important that FAMS management reaffirm and strengthen its efforts to prevent discrimination. The Equal Employment Opportunity Commission’s Management Directive 715 requires agencies to take appropriate steps to establish a model EEO program and identifies six essential elements for a model EEO program, including demonstrated commitment from agency leadership and proactive prevention of unlawful discrimination. Further, it is DHS’s stated objective to develop and maintain a high performing workforce in part by promoting a culture of transparency, fairness, and equal employment opportunity throughout the DHS workforce. By taking steps to renew its commitment to the goals and initiatives in its 2012 action plan, such as updating and following through on its 2012 action plan, FAMS management can demonstrate leadership commitment to the prevention of discrimination. Doing so could better ensure it proactively addresses and reduces concerns of discrimination among its workforce. Federal air marshals are deployed worldwide to protect civil aviation against the risk of terrorist violence. Although FAMS has taken some steps to address air marshals’ quality of life issues, FAMS management does not have information about the number and proportion of the workforce who are medically qualified, which limits their understanding of the workforce’s ability to fulfill its duties. Further, FAMS has not assessed the overall health of its workforce by analyzing available data, which would allow it to identify any health and fitness trends or risks among its workforce, take steps to mitigate these risks, make informed workforce planning decisions, and prioritize employee welfare to ensure that it deploys a workforce capable of fulfilling its national security mission. FAMS does not monitor the extent to which air marshals’ actual work hours are consistent with scheduling guidelines, limiting its ability to determine if air marshals’ quality of life is being balanced with the agency’s operational needs. FAMS also has not shared these scheduling guidelines with air marshals or provided guidance outlining authorities and procedures for changing air marshals’ schedules with field offices. Sharing these guidelines would improve the ability of air marshals and their supervisors to address quality of life issues related to long shifts and inadequate rest. Finally, although FAMS has taken steps to prevent discrimination, FAMS employees have continued to file discrimination complaints indicating that at least the perception of discrimination persists. By taking steps to reaffirm and strengthen its efforts to prevent discrimination, such as updating and following through on its 2012 action plan, FAMS management could better ensure it proactively addresses and reduces concerns of discrimination consistent with DHS’s objective of developing and maintaining a high performing workforce through fairness and equal employment opportunity. We are making the following six recommendations to FAMS: The Executive Assistant Administrator / Director of FAMS should identify and utilize a suitable system that provides information about air marshals’ medical qualification status. (Recommendation 1) The Executive Assistant Administrator / Director of FAMS should develop and implement a plan to assess the health and fitness of the FAMS workforce as a whole, including trends over time. (Recommendation 2) The Executive Assistant Administrator / Director of FAMS should identify and implement a means to monitor the extent to which air marshals’ actual shifts and rest hours are consistent with scheduling guidelines. (Recommendation 3) The Executive Assistant Administrator / Director of FAMS should provide all air marshals access to scheduling guidelines, including workday length and rest periods. (Recommendation 4) The Executive Assistant Administrator / Director of FAMS should disseminate or otherwise provide supervisory air marshals access to guidance that outlines authorities and procedures for changing an air marshal’s work schedule. (Recommendation 5) The Executive Assistant Administrator / Director of FAMS should take steps to reaffirm and strengthen efforts to prevent discrimination by, for example, updating and following through on its 2012 action plan and renewing leadership commitment to the plan’s goals. (Recommendation 6) We provided a draft of our report to DHS for comment. In written comments, which are included in appendix IV, DHS concurred with our six recommendations and described steps they plan to take to address them, including estimated timeframes for completion. With regard to our first recommendation that FAMS identify and utilize a suitable system that provides information about air marshals’ medical qualification status, DHS officials stated that FAMS is evaluating case management software to track this information and plans to pursue funding for this effort in fiscal year 2021. This action, if fully implemented, should address the intent of this recommendation. With regard to our second recommendation that FAMS develop and implement a plan to assess the health and fitness of the FAMS workforce as a whole, DHS officials stated that FAMS recently established a team to develop a plan for assessing workforce health and wellness issues. Adopting and implementing a plan that assesses the health and fitness of the FAMS workforce as a whole should address the intent of this recommendation. With regard to our third recommendation that FAMS identify and implement a means to monitor the extent to which air marshals’ actual shifts and rest hours are consistent with scheduling guidelines, DHS officials stated that FAMS will begin tracking air marshals’ actual hours and examine the extent to which air marshals’ actual and scheduled hours vary. This information could be helpful, for example, in assessing air marshals’ schedule predictability. However, to address the intent of this recommendation, FAMS would need to monitor the extent that air marshals’ actual work and rest hours are consistent with FAMS’s scheduling guidelines. With regard to our fourth recommendation to provide all air marshals access to scheduling guidelines, according to DHS officials, FAMS will provide air marshals ongoing access to the guidelines. Similarly, with regard to our fifth recommendation to provide supervisory air marshals access to guidance that outlines authorities and procedures for changing an air marshal’s work schedule, according to DHS officials, FAMS will provide supervisors ongoing access to scheduling authorities and procedures. These actions, if fully implemented, should address the intent of these recommendations. With regard to our sixth recommendation that FAMS reaffirm and strengthen efforts to prevent discrimination, DHS officials stated that FAMS plans to review the goals of its 2012 action plan and develop steps to strengthen efforts to prevent discrimination. If fully implemented, these actions should address the intent of this recommendation. We are sending copies of this report to the appropriate congressional committees and to the Acting Secretary of Homeland Security, Administrator of TSA, Executive Assistant Administrator / Director of FAMS, and other interested parties. In addition, this report is available at no charge on the GAO website at http://gao.gov. If you or your staff have any questions concerning this report, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. The objectives of this report are to (1) assess the extent to which the Federal Air Marshal Service (FAMS) has taken steps to address air marshals’ health concerns; (2) assess the extent to which FAMS has taken steps to address air marshals’ concerns about their work schedules; and (3) describe what is known about the number of discrimination complaints FAMS employees have reported to the Transportation Security Administration (TSA) and FAMS and assess the extent to which TSA and FAMS have taken steps to prevent discrimination in the workplace. To address all three objectives, we visited a non-generalizable sample of six FAMS field offices in: Atlanta, Georgia; Dallas, Texas; Los Angeles, California; Newark, New Jersey; New York, New York; and Seattle, Washington. We chose these field offices to capture variation in the following factors: the number of special mission coverage trips (SMCs) in fiscal year 2018; the rate of schedule changes by field office in fiscal year 2018; the rate of equal employment opportunity complaints by field office for fiscal years 2015 through 2018; the number of employees in each field office as of September 2018; field office location; and results from the Office of Personnel Management’s (OPM) 2018 Federal Employee Viewpoint Survey (FEVS). To obtain a range of perspectives on quality of life issues, work schedules, and discrimination within FAMS, we conducted discussion sessions with air marshals as well as separate discussion sessions with supervisory federal air marshals (SFAMs) in each field office. We conducted a total of ten discussion sessions with air marshals. We initially conducted one discussion session with air marshals in the Seattle field office—where we spoke with approximately 15 air marshals—and one discussion session with air marshals in the Dallas field office—where we spoke with approximately 30 air marshals. Following these discussion sessions, we developed a standardized list of questions used to facilitate two discussion sessions with approximately 10 air marshals each, in each of the remaining four field offices (Atlanta, Los Angeles, Newark, and New York). We also conducted a total of six discussion sessions exclusively with SFAMs—one session in each field office that we visited. The discussion session in the Seattle field office consisted of two SFAMs, while all others consisted of approximately 10 SFAMs. Following discussion sessions with SFAMs in the Seattle and Dallas field offices, the team developed a standardized list of questions that was used by a moderator in meetings with SFAMS in the remaining four field offices. For discussion sessions with air marshals and SFAMs, we requested that each field office make available a diverse group of participants, to include women and minorities. These were semi-structured discussions, led by a moderator who followed a standardized list of questions and allowed for unstructured follow-up questions. The results from these group discussions are not generalizable to air marshals or SFAMs who did not participate in them, but they provided a range of perspectives from about 125 air marshals and about 50 SFAMs spanning the six FAMS field offices we visited. In each field office we visited we also interviewed field office management officials about these same topics. Finally, we interviewed field office operations staff in four of the six field offices about their role in scheduling air marshals. To obtain additional perspectives on these topics, we interviewed a TSA employee group (Women Executives at FAMS); a professional association representing federal law enforcement officers, including air marshals (the Federal Law Enforcement Officers Association); and the FAMS Ombudsman. To address the first objective about air marshals’ health concerns, we reviewed prior research on FAMS workforce issues including our past reports on challenges associated with FAMS’s workforce; a 2012 FAMS- commissioned Harvard Medical School study on air marshal sleep and fatigue; and reports from FAMS working groups that examined medical issues and physical fitness. To identify air marshals’ current concerns about health issues, we asked air marshals about any quality of life issues they face during discussion sessions. We then performed a content analysis of the results and identified key issues relating to health that were raised during the discussion sessions. One of our analysts conducted this analysis, tallying the number of discussion sessions in which certain health issues were discussed by air marshals. A different analyst then checked the information for accuracy, and any initial disagreements were discussed and reconciled by the analysts. We also analyzed results of OPM’s FEVS for FAMS, TSA, and DHS employees in 2018—the most recent data available at the time of our review. We analyzed FEVS question number 35, which asks survey participants if “Employees are protected from health and safety hazards on the job.” We assessed the reliability of the FEVS data by reviewing OPM’s 2018 FEVS Technical Report and reviewing confidence intervals for the data points we included in this report. We determined that the data we used were sufficiently reliable for use in the analysis presented in this report. We also analyzed FAMS’s workers’ compensation claim data for FAMS employees for fiscal years 2013 (when FAMS reviewed air marshals’ physical fitness) through 2018 (the most recent full fiscal year of data available). We assessed the reliability of the claim data by interviewing cognizant FAMS officials, obtaining information about the data systems that maintain these data, and conducting checks for missing and out of range values. We determined that the data we used were sufficiently reliable for use in the analysis presented in this report. To identify steps FAMS has taken to address air marshals’ health concerns, we asked FAMS management, SFAMs, and air marshals we met with in headquarters and field offices to identify efforts to assess and promote air marshals’ health—such as programs, policies, and practices. We reviewed documentation related to these efforts including FAMS’s policies outlining medical standards for air marshals and its Health, Fitness, and Wellness program, as well as FAMS analyses of health issues among air marshals, workers’ compensation claims, and on-the- job injuries. For example, we examined (a) minutes from two FAMS meetings when FAMS Medical Programs Section officials reported on medical and health issues among air marshals; (b) summary information from TSA’s Occupational Safety, Health, and Environment Division describing air marshals’ worker compensation claims from fiscal years 2015 through 2018; (c) an analysis of injuries and illnesses reported by air marshals from calendar years 2016 through 2018. We also reviewed information about FAMS practices for maintaining medical and health information about air marshals. We compared FAMS’s efforts to address air marshals’ health concerns to OPM strategies for human capital management and a TSA strategic planning document from June 2018. To address the second objective to examine the extent to which FAMS has taken steps to address air marshals’ concerns about their work schedules, we reviewed FAMS documents outlining scheduling guidelines for shift length and rest periods, protocols for adjusting air marshals’ schedules, and FAMS management reports with statistics on air marshals’ planned and actual schedules. We analyzed data from FAMS’s Aircrews data system on the number of SMC missions and the number of changes made to air marshals’ schedules in order to cover SMCs between November 2016 and June 2019. We also analyzed data from FAMS’s Aircrews data system on the number of scheduled standby shifts between June 2018—when FAMS began scheduling air marshals to standby shifts to staff SMCs—and August 2019. We assessed the reliability of these data by reviewing documentation regarding the source of this data and by obtaining information from knowledgeable agency officials about its accuracy and completeness. We found these data to be sufficiently reliable for use in our analysis. To identify the lengths of air marshals’ shifts when they flew missions, we analyzed 808 air marshal time sheets. We first selected four separate 28-day periods, known as roster periods, during which air marshals flew missions. Our analysis included air marshals scheduled to fly or on recovery shifts on 11 or more days during the selected roster periods. This resulted in a total of 7,981 roster periods worked by air marshals as our population of interest. To help ensure the sample included air marshals from field offices that had high rates of SMCs for each roster period, we stratified our population into eight mutually exclusive strata based on the roster period and the percentage of each field office’s missions that were SMCs in each roster period. We then randomly selected a stratified sample of 101 air marshals from each roster period proportionally allocated across the SMC percentage strata within each roster period. Using these data for these air marshals, we analyzed the length of air marshals’ shifts when they flew domestic and international missions to identify shifts that were (1) consistent with or (2) exceeded scheduling guidelines. For example, we analyzed time sheets to estimate the percentage of roster periods worked by air marshals that included one or more shifts longer than 10 hours. We also analyzed time sheets to estimate the percentage of roster periods worked by air marshals that included one or more shifts between 10 and 12 hours and to estimate the percentage of roster periods worked by air marshals that included one or more shifts longer than 12 hours. We also examined the number of air marshals’ regular days off. Specifically, we analyzed air marshals’ time sheets to estimate the percentage of roster periods worked by air marshals that included less than 8 regular days off. In performing this analysis, we did not count days as regular days off when air marshals reported receiving a regular day off but also reported time worked for the same day, unless the time worked was carryover from a prior workday. In conducting these time sheet analyses, we took steps to minimize issues that might affect data reliability. Specifically, we identified time and attendance sheets that included errors that would impact our analysis— such as those with missing values—and either excluded them or obtained corrected information from FAMS. We excluded a total of 44 of the 404 roster periods initially selected in our sample. We also performed an analysis to ensure that by excluding these timesheets we did not introduce bias into our sample. We found no evidence of bias and concluded the sample data was sufficiently reliable for the purposes of producing population estimates. The results of our analysis are generalizable to the roster periods analyzed. To identify steps FAMS has taken to address air marshals’ concerns about their schedules, we interviewed management officials from FAMS’s Flight Operations Division about their efforts to (1) monitor air marshals’ shifts and rest against scheduling guidelines and (2) make scheduling protocols available to staff. We compared FAMS’s actions to address air marshals’ scheduling concerns to two principles in Standards for Internal Control in the Federal Government related to the need to implement control activities and use quality information to achieve an entity’s objectives. To address the third objective about discrimination, we reviewed FAMS, TSA, and DHS policies related to discrimination and interviewed FAMS, TSA, and DHS officials to understand how FAMS employees report discrimination complaints. Specifically, we met with officials in TSA’s Civil Rights Division (CRD), TSA’s Anti-Harassment Program, FAMS Incident Activity Coordination and Trends Unit, and DHS OIG. We also examined the number and characteristics of discrimination complaints reported by FAMS employees from fiscal year 2016 through fiscal year 2018—the most recent 3 full years of data available at the time of our review. Specifically, we analyzed record-level data on discrimination complaints filed or reported by FAMS employees to TSA’s CRD, TSA’s Anti- Harassment Program, and FAMS’s Incident Activity Coordination and Trends Unit. We also obtained information from the DHS OIG on individual complaints they received that involved FAMS employees and included complaints of discrimination. Generally, we analyzed the date of the complaint, type of allegation, basis of the discrimination, and outcomes. We assessed the reliability of the data from TSA’s CRD, TSA’s Anti-Harassment Program, and FAMS’s Incident Activity Coordination and Trends Unit by interviewing cognizant TSA and FAMS officials, obtaining information about the data systems that maintain these data, and conducting checks for missing and out of range values. We determined that the data we used was sufficiently reliable for use in the analysis presented in this report. To examine the proportion of the FAMS and TSA workforces who alleged discrimination relative to the size of these workforces, we compared the number of complaints handled by TSA’s CRD for fiscal years 2016, 2017 and 2018 to the total number of employees during the same fiscal years. We assessed the reliability of the TSA’s CRD data by interviewing cognizant TSA officials and obtaining information about the data system that maintains these data. We determined that the data we used was sufficiently reliable for use in the analysis presented in this report. To identify steps TSA and FAMS have taken to prevent discrimination in the workplace, we interviewed TSA and FAMS management, SFAMs, and air marshals we met with during our site visits. We then analyzed documentation related to the identified efforts such as minutes from all 20 FAMS Field Office Focus Group meetings between October 2016 and December 2018 as well as DHS and TSA training materials related to preventing discrimination. To identify air marshals’ current perspectives about discrimination, we asked air marshals in our discussion sessions about the processes for reporting discriminatory behavior as well as their perspectives on discriminatory behavior within FAMS. We then performed a content analysis of the results and identified key issues that were raised during the discussion sessions, including air marshals’ comments regarding their experiences related to retaliation for reporting discrimination. One of our analysts conducted this analysis, tallying the number of discussion sessions in which certain issues were discussed by air marshals. A different analyst then checked the information for accuracy. We then determined the extent to which certain key issues were raised among the sessions. In addition, we analyzed results of OPM’s FEVS for FAMS, TSA, and DHS employees in 2018. Specifically, we analyzed FEVS question number 17, which asks survey participants if employees “Can disclose suspected violation without fear of reprisal.” We also analyzed FEVS question number 38, which asks survey participants if “Prohibited personnel practices are not tolerated.” As noted above, we assessed the reliability of the FEVS data and determined that the data we used was sufficiently reliable for use in the analysis presented in this report. We also analyzed data from TSA’s employee exit survey results for FAMS employees from fiscal years 2012 through 2018—the period for which full year data were available since the DHS OIG review. Specifically, we examined the extent to which employees’ reasons for leaving included diversity or inclusion barriers in the workplace. We assessed the reliability of the exit survey data by obtaining information about how the data are collected from TSA officials. We determined that the data we used were sufficiently reliable for use in the analysis presented in this report. We compared TSA’s and FAMS’s efforts to prevent discrimination in the workplace to the Equal Employment Opportunity Commission’s Management Directive 715. This policy requires agencies to take appropriate steps to establish a model equal employment opportunity (EEO) program and identifies six essential elements for a model EEO program. In addition, we compared TSA’s and FAMS’s efforts to DHS’s and TSA’s strategic planning documents which both include an objective to develop and maintain a high-performing workforce. We conducted this performance audit from July 2018 to January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Federal Air Marshal Service’s (FAMS) scheduling guidelines state that each air marshal is scheduled to receive a minimum of 60 hours of rest around 2 consecutive regular days off each week, or a total of 8 regular days off each 28-day roster period. FAMS Flight Operations officials stated that there are exceptions that may prevent an air marshal from being scheduled to receive 2 regular days off each week, such as international deployments that last 6 or more days and travel to and from training programs that last 6 or more days. Additionally, FAMS management officials and air marshals that we interviewed stated that air marshals may be asked to cover flights for which a potentially high-risk passenger has been ticketed—known as Special Mission Coverage deployments—on their scheduled regular days off if no other air marshals are available. Furthermore, FAMS Flight Operations officials stated that FAMS may ask air marshals to receive non-consecutive regular days off due to operational needs. We analyzed air marshals’ regular days off as recorded on their timesheets to determine the extent that they were consistent with these scheduling guidelines. Specifically, we analyzed a generalizable sample of air marshals’ timesheets for two roster periods in fiscal year 2018 and two roster periods in fiscal year 2019. We found that air marshals generally received 8 regular days off in the roster periods we analyzed. Specifically, during the 28-day roster periods we examined in fiscal year 2019, we estimate that air marshals received 8 regular days off approximately 98 percent of the time. However, some air marshals did not receive all 8 regular days off. Specifically, during the 28-day roster periods we analyzed in fiscal year 2019, we estimate that air marshals received 7 regular days off approximately 2 percent of the time. See figure 5 for results of our analysis. There are four venues through which Federal Air Marshal Service (FAMS) employees can raise discrimination complaints. One of these venues is the Transportation Security Administration’s (TSA) Civil Rights Division (CRD) which is responsible for receiving and handling FAMS employees’ equal employment opportunity (EEO) complaints. Although reporting to CRD is the only means for FAMS employees to file an EEO complaint, they may choose to report discrimination in other venues. Specifically, they may report discrimination to their manager, TSA’s Anti-Harassment Program—which is overseen by TSA’s National Resolution Center, or the Department of Homeland Security’s (DHS) Office of Inspector General (OIG). FAMS employees may also choose to report to CRD as well as to one or more of the other available entities. Table 1 describes what is known about the number and nature of complaints received through each venue in fiscal years 2016 through 2018. In addition to the contact named above, Claudia Becker (Assistant Director), Anne Akin (Analyst-in-Charge), Enyinnaya Aja, James Ashley, Carl Barden, Taiyshawna Battle, Edda Emmanuelli-Perez, Eric Hauswirth, Yvonne Jones, Jesse Jordan, Ellie Klein, Thomas Lombardi, Diona Martyn, Sam Portnow, Minette Richardson, Forrest Rule, Raymond Sendejas, Michael Silver, and Adam Vogt also made key contributions to this report.", "summary": "In the wake of 9/11, terrorists continue to target aircraft and airports, underscoring the ongoing threat to civil aviation and the need for effective security measures. FAMS deploys air marshals on selected flights to address such threats and is a key component of TSA's approach to aviation security. However, longstanding challenges faced by FAMS's workforce could impact its ability to carry out its mission. GAO was asked to review FAMS workforce issues. This report addresses (1) the extent to which FAMS has taken steps to address air marshals' health concerns, (2) the extent to which FAMS has taken steps to address air marshals' concerns about their work schedules, and (3) the number of discrimination complaints FAMS employees have reported and the extent to which FAMS has taken steps to prevent discrimination. GAO analyzed TSA and FAMS policies; documentation of efforts to address air marshals' quality of life issues; and FAMS data on missions, schedules, and discrimination complaints. GAO also interviewed TSA and FAMS officials, including FAMS management and air marshals in a non-generalizable sample of six FAMS field offices selected to capture a breadth of perspectives. Air marshals continue to express concerns about their health, but the Federal Air Marshal Service (FAMS) has not comprehensively assessed the health of its workforce. Air marshals in all six field offices we visited noted health issues, such as sleep deprivation, as a key quality of life concern. FAMS has taken steps to assess air marshals' individual health, such as requiring medical exams, but has not comprehensively assessed the overall health of its workforce and has not developed a plan to do so. FAMS officials stated that it would be difficult to analyze air marshals' medical records because they are not stored electronically, though they are researching options to do so. FAMS could develop and implement a plan to analyze the employee health data it already collects to identify workforce trends, and use this information to better promote employee welfare consistent with Transportation Security Administration (TSA) leadership principles. FAMS has taken some steps to address air marshals' concerns about their work schedules. In March 2018, FAMS revised its deployment strategy to expand coverage of certain high risk missions that it typically learns of 72 hours in advance. Following this, changes to air marshals' schedules to accommodate these missions more than doubled. In response, FAMS altered how it staffs these missions and reports that these modifications have reduced schedule changes. FAMS also maintains shift length and rest period guidelines intended to balance mission needs with air marshals' quality of life. However, FAMS does not monitor the extent to which air marshals' actual work hours are consistent with guidelines because it has not identified a need to do so. As a result, it cannot determine how frequently air marshals work beyond guidelines and is not well-positioned to manage risks associated with long work hours. From fiscal years 2016 through 2018, FAMS employees filed 230 discrimination complaints with TSA's Civil Rights Division, though employees may have reported additional discrimination complaints through other means. In 2012, FAMS adopted an action plan to address discrimination and has taken some steps called for in the plan, such as sustaining a FAMS Ombudsman position. However, due to a loss of management focus on the plan, FAMS has not fully implemented other planned efforts, such as holding diversity focus groups. Taking steps to reaffirm its efforts to prevent discrimination would demonstrate leadership commitment to reducing concerns of discrimination within FAMS. GAO is making six recommendations to FAMS, including that it implement a plan to assess the health of the FAMS workforce, monitor the extent that air marshals' shifts are consistent with guidelines, and strengthen efforts to prevent discrimination. DHS concurred with all six recommendations.", "document_type": "gao"}
{"report": "VA is responsible for providing benefits to veterans, including health care, disability compensation, and various types of financial assistance. In fiscal year 2019, VA received a total budget of $201.1 billion, and the largest discretionary budget in its history—$86.6 billion, about $20 billion higher than in 2015. The department operates one of the largest health care delivery systems in the nation through its Veterans Health Administration (VHA), with 172 medical centers and more than 1,000 outpatient facilities organized into regional networks. VA has faced growing demand by veterans for its health care services, with the total number of veterans enrolled in VA’s health care system rising from 7.9 million to more than 9 million from fiscal year 2006 through fiscal year 2017. In fiscal year 2019, VHA received $73.1 billion of VA’s $86.6 billion discretionary budget. In addition to providing health care services, VA provides cash benefits to veterans for disabling conditions incurred in or aggravated by military service. To carry out its mission, VA spends tens of billions of dollars to procure a wide range of goods and services, including medical supplies; to construct hospitals, clinics, and other facilities; and to provide the information technology (IT) to support its operations. We have made hundreds of recommendations to improve VA’s management and oversight of the services it provides to veterans. Specifically, since 2000, we have made 1,225 recommendations to VA. While VA has implemented most of the recommendations, a number remain open, as of April 2019. Specifically, more than 125 recommendations related to VA health care remain open, including 17 recommendations that have remained open for 3 years or more; 15 recommendations related to improving VA acquisition management remain open, including 1 recommendation that has remained open for 3 years or more; and 12 recommendations related to management of disability claims workloads. In 2017, we began sending letters to VA and appropriate congressional committees identifying priority recommendations for VA to implement in order to significantly improve its operations. We categorized these recommendations into nine areas: (1) veterans’ access to timely health care; (2) veterans’ community care program; (3) human capital management; (4) information technology; (5) appeals reform for disability benefits; (6) quality of care and patient safety; (7) national policy documents; (8) contracting policies and practices; and (9) veterans’ access to burial options. Since we designated VA health care as a high-risk area in 2015, VA has begun to address each of the identified five areas of concern related to managing risks and improving VA health care: (1) ambiguous policies and inconsistent processes; (2) inadequate oversight and accountability; (3) IT challenges; (4) inadequate training for VA staff; and (5) unclear resource needs and allocation priorities. Since our 2017 High-Risk Report, ratings for all five criteria remain unchanged as of March 2019. Specifically, the leadership commitment and action plan criteria remain partially met. Although VA has experienced leadership instability over the past 2 years in several senior positions, a new Secretary was confirmed in July 2018. Secretary Wilkie has demonstrated his commitment to addressing the department’s high- risk designation by, among other things, creating an office to direct an integrated, focused high-risk approach and communicating to VA leaders the importance of addressing our recommendations and working with GAO. The Secretary’s actions, to date, have allowed the department to maintain its leadership commitment rating as of March 2019. The action plan criterion also remains partially met as of March 2019. In March 2018, VA submitted an action plan to address the underlying causes of its high-risk designation, but the plan did not clearly link actions to stated outcomes and goals or establish a framework to assess VA’s progress. VA officials told us that instead of revising the March 2018 action plan, it will incorporate its plans to address the high-risk designation into the department’s current initiatives. Specifically, VA is currently implementing the VHA Plan for Modernization, through which the department intends to modernize VA’s structure, culture, governance, and systems through organizational improvements. VA officials have indicated that the VHA Plan for Modernization is intended, among other things, to address the high-risk areas for VA health care. VA officials also told us they are currently developing operational plans for the VHA Plan for Modernization, and these plans will include goals, time frames, and metrics, among other things. VA estimates that the operational plans will be complete by September 2019. The monitoring, demonstrated progress, and capacity criteria remain unmet since our 2017 High-Risk Report. In order to address the monitoring and demonstrated progress criteria, VA’s ongoing revisions to its action plan need to include the addition of certain essential components, including metrics, milestones, and mechanisms for monitoring and demonstrating progress in addressing the high-risk areas of concern. VA’s capacity rating also remains not met. Though the department took steps to establish offices, workgroups, and initiatives to address its high-risk designation, many of these efforts are either in the initial stages of development or resources have not been allocated. For each of the five identified areas of concern related to managing risks and improving VA health care, ratings reflect the level of progress VA has made to address them. Ambiguous policies and inconsistent processes. Since our 2017 High-Risk Report, ratings for all five criteria remain unchanged for this area of concern as of March 2019. Leadership commitment: partially met. In September 2017, we reported that VHA had approximately 800 national policies, the majority of which were outdated. VHA reported reducing the number of national policies by 26 percent, and work continues in this area. In addition, VHA established an inventory of approximately 55,000 local policies as of October 2017. In October 2018, VHA noted its plans to determine who is responsible for monitoring implementation of national and local policy, as well as the alignment between these levels of policy. At that time, VHA also discussed its future plans to monitor the implementation and alignment of national and local policy and update its national policy directive by the end of June 2019. Additionally, VA has implemented a structure for leadership input into the policy process, such as at the VHA Chief of Staff level. However, senior leadership has lacked the stability needed to ensure issued policy meets agency goals. Capacity: not met. Since 2017, VA has issued an updated directive on policy management, and put in place procedures to train staff and obtain input from all levels on policy development. However, VA continues to face challenges in this area because it is reliant on contracts and information technology resources, which if delayed, can impede progress toward meeting goals. Action plan: partially met. Since 2017, VA has further refined its root cause analysis for this area of concern. In June 2017, VA also identified the following as enterprise-wide root causes of its high-risk designation: disjointed strategic planning; poorly defined roles, responsibilities, and decision authorities; poor horizontal and vertical integration; lack of reliable data and analysis; ineffective human capital management; and inadequate change management. VA relied on these root cause analyses as the foundational drivers for the VHA Plan for Modernization. However, VA has not used these analyses to develop and prioritize appropriate milestones and metrics in the action plan. Monitoring: not met. Since the March 2018 action plan lacked specific metrics and mechanisms for assessing and reporting progress, it is not clear how VA is monitoring its progress. Demonstrated progress: not met. Our work continues to indicate VA is not yet able to show progress in this area. Since its 2015 high-risk designation, we have made 50 new recommendations in this area of concern, 32 of which were made since our 2017 report was issued. For example, In November 2017, we reported that, due in part to misinterpretation or lack of awareness of VHA policy, VA medical center officials did not always conduct or document timely required reviews of providers when allegations were made against them. We also found that VHA was unable to reasonably ensure appropriate reporting of providers to oversight entities such as state licensing authorities. As a result, VHA’s ability to provide safe, high quality care to veterans is hindered because other VA medical centers, as well as non-VA health care entities, may be unaware of serious concerns raised about a provider’s care. We recommended that VHA direct medical centers to document and oversee reviews of providers’ clinical care after concerns are raised, among other recommendations. All of our recommendations remain open. As of January 2019, VA estimated completing the recommended revisions to its policy and audit processes in August 2019 and August 2020, respectively. In July 2018, we reported that VA collected data related to employee misconduct and disciplinary actions, but data fragmentation, reliability issues, and inadequate guidance impeded department-wide analysis of those data. Thus, VA management is hindered in making knowledgeable decisions regarding the extent of misconduct and how it was addressed. We recommended that VA develop and implement guidance to collect complete and reliable misconduct and associated disciplinary-action data department-wide, whether through a single information system, or multiple interoperable systems. VA concurred with this priority recommendation, which remains open. VA reported that it expects to implement one or more information systems that will collect misconduct and associated disciplinary action data in January 2020. Inadequate oversight and accountability. Since our 2017 High-Risk Report, ratings for one criterion improved and four remain unchanged for this area of concern as of March 2019. Leadership commitment: partially met. VA has made organizational changes, including establishing the Office of Integrity, to standardize and streamline the agency’s oversight of its programs and personnel. However, since 2017, the lack of stability in the Under Secretary for Health position has hindered its ability to demonstrate sustained commitment to improving this area of concern. Capacity: not met. VA has begun to implement capacity-building initiatives directed at improving oversight and accountability. For example, VHA’s Office of Internal Audit and Risk Assessment, a key component of the department’s oversight and accountability model, began conducting audits in 2018. However, according to VA’s action plan, the department has yet to allocate resources for this office, such as sufficient staff to carry out its activities. Action plan: partially met. In March 2019, the rating for this criterion improved to partially met. In 2018, VA conducted an analysis of the root causes contributing to findings of inadequate oversight and accountability, an important step in identifying the underlying factors contributing to this area of concern. addresses gaps in physician staffing, including those for mental health providers, which may affect veterans’ access to care, among other issues. We recommended that VHA should develop and implement a process to accurately count all physicians providing care at each medical center, including physicians who are not employed by VHA. VHA did not concur with this recommendation, which we reiterated in our priority recommendation letter. In a series of reports from 2012 through 2018, GAO found VA’s wait time data unreliable for primary and specialty care, as well as for care in the community. GAO also found that VA did not measure the full wait times that veterans experience in obtaining care across these settings. Specifically, in December 2012, we made two recommendations to VA to improve the reliability and oversight of wait time measures, both of which are designated as priority, and remain open. Similarly, in June 2018, we reported that VHA could not systematically monitor the timeliness of veterans’ access to Veterans Choice Program care because it lacked complete, reliable data to do so. Specifically, we found (1) a lack of data on the timeliness of accepting referrals and opting veterans in to the program, (2) inaccuracy of clinically indicated dates, which are used to measure the timeliness of care, and (3) unreliable data on the timeliness of urgent care. We recommended that VA take steps to improve the timeliness and accuracy of data on veterans’ wait times for care and its oversight of the future community care program that will consolidate other community care programs with the Veterans Choice Program, whose authority sunsets on June 6, 2019. VA concurred with eight of the 10 recommendations related to these findings, all of which remain open. VA reported that, in order to improve wait times data accuracy under the Veterans Community Care Program, it intends to implement several initiatives through September 2019. In September 2018, we reported on the timeliness of third-party administrators’ payments to community providers under VA’s largest community care program, the Veterans Choice Program. Although VA has taken steps to improve the timeliness of claim payments to these providers, VA is not collecting data or monitoring compliance with third-party administrators’ customer service requirements for provider calls. This could adversely affect the timeliness with which community providers are paid, possibly making them less willing to participate and affecting veterans’ access to care. We recommended that VA collect data on and monitor compliance with its requirements pertaining to customer service for community providers. VA agreed with the recommendations, but has not yet implemented them. In November 2018, we reported that VHA’s suicide prevention media outreach activities declined in recent years due to leadership turnover and reorganization. Additionally, we found that VHA did not assign key leadership responsibilities or establish clear lines of reporting for its suicide prevention media outreach campaign, which hindered its ability to oversee the campaign. In April 2019, VA implemented one of the recommendations by providing a new oversight plan for its suicide prevention media outreach campaign. It plans to implement the remaining recommendation by working with communications experts to develop metrics, targets, and an evaluation strategy to improve its outreach efforts. In April 2019, we reported that VHA’s appraisal process for assessing medical center director performance relies heavily on medical center performance information. VHA designed the Strategic Analytics for Improvement and Learning (SAIL) system to provide internal benchmarking of medical center performance and to promote high quality health care delivery across its system of regional networks and medical centers. SAIL was evaluated in 2014 and 2015 by VHA and an external contractor, respectively, but VHA has not assessed the recommendations from those evaluations, or taken action on them. The evaluations, which found issues related to the validity and reliability of SAIL and its ratings for measuring performance and fostering accountability, together included more than 40 recommendations for improvement. Without ensuring that the recommendations resulting from these previous evaluations are assessed and implemented as appropriate, the identified deficiencies may not be adequately resolved, and VHA’s ability to hold officials accountable for taking the necessary actions may be diminished. VA concurred with the two recommendations we made to address these findings, both of which remain open. Information technology challenges. Since our 2017 High-Risk Report, ratings for one criterion regressed, one improved, and three remain unchanged this area of concern as of March 2019. Leadership commitment: not met. In March 2019, the rating for this criterion declined to not met. In January 2019, the Senate confirmed a new VA Chief Information Officer (CIO). This is the fourth official to lead VA’s IT organization since our 2017 High-Risk Report, and the frequent turnover in this position raises concerns about VA’s ability to address the department’s IT challenges. Capacity: not met. In May 2018, VA awarded a contract to acquire the same commercial electronic health record system as the Department of Defense (DOD). However, VA is early in the transition and its actions are ongoing. Additionally, VA has developed a strategy for decommissioning its legacy IT systems, which are tying up funds that could be reallocated for new technology to enable improved veteran care, but has made limited progress in implementing this effort. Action plan: partially met. In March 2019, the rating for this criterion improved to partially met. In 2018, VA conducted an analysis to identify the root causes of IT challenges, which informed the goals in its action plan. However, VA’s action plan contained significant information gaps, including missing interim milestone dates. These information gaps raise questions about VA’s commitment to addressing IT-related root causes and need to be addressed before we can consider this criterion met. Monitoring: not met. The March 2018 action plan lacked specific metrics and mechanisms for assessing and reporting progress. Demonstrating progress: not met. Our work continues to indicate VA is not yet able to show progress in this area. Since its 2015 high-risk designation, we have made 14 new recommendations in this area, 12 of which were made since our 2017 report was issued. For example: In June 2017, to address deficiencies we found related to VA’s pharmacy system, we recommended that VA take six actions to provide clinicians and pharmacists with improved tools to support pharmacy services to veterans and reduce risks to patient safety. This included assessing the extent to which the interoperability of VA and DOD’s pharmacy systems impacts transitioning service members. VA generally concurred with these recommendations, all of which remain open. In April 2019, we testified that from 2001 through 2018, VA pursued three efforts to modernize its health information system— the Veterans Health Information Systems and Technology Architecture (VistA). (See Fig. 2.) However, these efforts resulted in high costs, created challenges ensuring the interoperability of health data, and ultimately did not result in a modernized VistA. Specifically, in December 2017, we reported that VA obligated over $1.1 billion for contracts with 138 contractors during fiscal years 2011 through 2016 for two modernization initiatives, an Integrated Electronic Health Record program with the DOD and VistA Evolution. We have ongoing work that examines the cost to VA of VistA and the department’s actions to transition from VistA to a new electronic health record system. Regarding the department’s most recent effort, the Electronic Health Record Modernization, we testified in April 2019 that the governance plan for this program was not fully defined, which could jeopardize its fourth attempt to modernize its electronic health record system. VA plans to implement the same electronic health record system the DOD is currently deploying. The new system is intended to be the authoritative source of clinical data to support improved health, patient safety, and quality of care provided by VA. VA has not fully implemented our priority recommendation calling for the department to define the role of the Interagency Program Office in the governance plans for acquisition of the department’s new electronic health record system. VA concurred with this recommendation and reported that the Joint Executive Committee, a joint governance body, approved a role for the Interagency Program Office, but as of April 2019 VA has yet to provide us with documentation of this development. We also testified in April 2019 that VA has not yet fully addressed the recommendation we made in September 2014 to expedite the process for identifying and implementing an IT system for the Family Caregiver Program. We reported in September 2014 that the Family Caregiver Program, which was established to support family caregivers of seriously injured post-9/11 veterans, has not been supported by an effective IT system. Specifically, we reported that, due to limitations with the system, the program office did not have ready access to the types of workload data that would allow it to routinely monitor workload problems created by the program. Without such information, the program’s workload issues could persist and impact the quality and scope of caregiver services, and ultimately the services that veterans receive. VA concurred with our recommendation and subsequently began taking steps to implement a replacement system. However, the department has encountered delays and reported recently initiating an effort to implement a new IT system to support the program based on existing commercially available software. We have ongoing work to evaluate VA’s effort to acquire a new IT system to support the Family Caregiver Program. Inadequate training for VA staff. Since our 2017 High-Risk Report, ratings for one criterion improved and four remain unchanged for this area of concern as of March 2019. Leadership commitment: not met. VA officials have reported progress in establishing a process to develop an enterprise-wide annual training plan to better ensure that VA staff are adequately trained to provide high-quality care to veterans. However, the actions necessary to complete and implement this training plan are not reflected in VA’s March 2018 action plan for the training area of concern, raising questions about the process through which it will be developed. The lack of progress in setting clear goals for improving training demonstrates that VA lacks leadership commitment to address our concerns in this area. Capacity: not met. VA has created working groups and task forces— such as the Learning Organization Transformation Subcommittee in the National Leadership Council—with specific responsibilities. However, VA’s ability to demonstrate capacity is limited because, according to VA’s March 2018 action plan, the department relies on external contractor support services to meet training goals. Action plan: partially met. In March 2019, the rating for this criterion improved to partially met. VA completed a root cause analysis for training deficiencies, which informed the goals underlying its action plan. However, the action plan continues to have deficiencies identified in 2017. For example, not all goal descriptions correspond to planned actions and the action plan lacks detail about how and which data will be collected to assess progress. Monitoring: not met. The March 2018 action plan lacked specific metrics and mechanisms for assessing and reporting progress. Demonstrated progress: not met. Our work continues to indicate that VA is not yet able to show progress in this area. Since its 2015 designation, we have made 11 new recommendations in this area of concern, 3 of which were made since our 2017 report was issued. For example, in April 2018 we reported that, while the department has recommended training for patient advocates—staff members who receive and document feedback from veterans or their representatives—it has not developed an approach to routinely assess their training needs or monitored training completion. The failure to conduct these activities increases VA’s risk that staff may not be adequately trained to advocate on behalf of veterans. As a result, we recommended VHA develop an approach to routinely assess training needs and monitor training completion. VA concurred with our recommendations, which remain open. Unclear resource needs and allocation priorities. Since our 2017 High-Risk Report, ratings for one criterion improved and four remain unchanged for this area of concern as of March 2019. Leadership commitment: partially met. In December 2017, a VA Chief Financial Officer (CFO) was confirmed after the department spent over 2.5 years under an interim CFO. In addition, VA is in the process of establishing a new office to estimate workforce resource requirements. Capacity: not met. VA has established functions intended to inform cost analyses of major VA initiatives, including a new financial management process to replace its outdated financial systems. However, it is unclear in its action plan the extent to which VA has identified the resources needed to establish and maintain these functions. Action plan: partially met. In March 2019, the rating for this criterion improved to partially met. Since our 2017 High-Risk Report, VA conducted a root cause analysis of this area of concern. However, VA’s action plan lacks metrics for monitoring progress and does not include all of VA’s ongoing actions, such as efforts to assess current and future regional demand for veterans’ health care services. Monitoring: not met. Since VA’s action plan lacks specific metrics and mechanisms for assessing and reporting progress, it is not clear how VA is monitoring its progress. Demonstrating progress: not met. Our work continues to indicate VA is not yet able to show progress in this area. Since its 2015 designation, we have made 16 new recommendations in this area of concern, 10 of which were made since our 2017 report. For example: In May 2017, we reported identifying several limitations with VA’s clinical productivity metrics and statistical models for tracking clinical efficiency; this limits VA’s ability to assess whether resources are being used effectively to serve veterans. Specifically, we found that productivity metrics may not account for all providers or clinical services, reflect the intensity of clinical workload, and reflect providers’ clinical staffing levels. Additionally, we found that efficiency models may also be adversely affected by inaccurate workload and staffing data. As a result, VA cannot systematically identify best practices to address low productivity and inefficiency as well as determine the factors VA medical centers commonly identify as contributing to low productivity and inefficiency. We made four recommendations to address these findings; three of which VA implemented in the spring of 2018 by improving productivity metrics and staffing and workload data. To implement the remaining recommendation, VA should establish a process to oversee medical centers’ plans for addressing low clinical productivity and inefficiency. In August 2018 we reported that VA medical centers face challenges operating their Sterile Processing Services programs— notably, addressing workforce needs, such as lengthy hiring time frames and limited pay and professional growth potential. VHA’s Sterile Processing Services workforce challenges pose a potential risk to VA medical centers’ ability to ensure access to sterilized medical equipment. Until VHA examines these workforce needs, VHA won’t know whether or to what extent the reported challenges adversely affect VA medical centers’ ability to effectively operate their Sterile Processing Services programs and ensure access to safe care for veterans. We recommended that VA examine workforce needs and take action based on this assessment, as appropriate. VA concurred with this recommendation, which remains open. In light of numerous contracting challenges that we have identified, and given the significant investment in resources to fulfill its critical mission of serving veterans, we added VA acquisition management as a new high- risk area in 2019. VA has one of the most significant acquisition functions in the federal government, both in dollar amount of obligations and number of contract actions. Specifically, about a third of VA’s discretionary budget in fiscal year 2018, or about $27 billion, has been used to contract for goods and services. We have identified challenges in the following areas of concern related to VA’s acquisition management: (1) outdated acquisition regulations and policies; (2) lack of an effective medical supplies procurement strategy; (3) inadequate acquisition training; (4) contracting officer workload challenges; (5) lack of reliable data systems; (6) limited contract oversight and incomplete contract file documentation; and (7) leadership instability. Outdated acquisition regulations and policies. VA’s procurement policies have historically been outdated, disjointed, and difficult for contracting officers to use. In September 2016, we reported that (1) the acquisition regulations contracting officers currently follow have not been fully updated since 2008 and (2) VA had been working on completing a comprehensive revision of its acquisition regulations since 2011. VA’s delay in updating this fundamental source of policy has impeded the ability of contracting officers to effectively carry out their duties. We recommended in September 2016 that VA identify measures to expedite the revision of its acquisition regulations and clarify what policies are currently in effect. VA concurred with this priority recommendation and, as of January 2019, had rescinded or re-issued updated policy memoranda for all information letters, which VA previously used to provide guidance that was temporary in nature. VA has also made some progress in updating its acquisition regulations, but more work remains to be done over the next several years. As of April 2019, VA reports that 15 of the 41 parts in its acquisition regulations update were published as final rules, 10 were issued as proposed rules for public comment, and the remainder are at an earlier stage of the rulemaking process. All parts are scheduled to be out for public comment by March 2020, but the final rules are not expected to be published until April 2021. Lack of an effective medical supplies procurement strategy. VA’s program for purchasing medical supplies has not been effectively executed, nor is it in line with practices at leading hospitals. To support more efficient purchasing of medical supplies for its 172 medical centers that serve the needs of about 9 million veterans, VA launched the Medical Surgical Prime Vendor-Next Generation (MSPV-NG) program in December 2016. MSPV-NG was part of VA’s overall effort to transform its supply chain and achieve $150 million in cost avoidance. In November 2017, we reported that VA’s approach to developing its catalog of supplies was rushed and lacked key stakeholder involvement and buy-in. It also relied on establishing non-competitive blanket purchase agreements for the overwhelming majority of products, resulting in low utilization by medical centers. VA had set a target that medical centers would order 40 percent of their supplies from the MSPV-NG catalog, but utilization rates were below this target with a nationwide average utilization rate across medical centers of about 24 percent as of May 2017. This low utilization adversely affected VA’s ability to achieve its cost avoidance goal. We recommended in November 2017 that VA develop, document, and communicate to stakeholders an overarching strategy for the program. VA concurred with this priority recommendation and is developing strategies to address it. First, in February 2019, VA developed and documented a new, overarching acquisition strategy for its Medical Surgical Prime Vendor (MSPV) program, and has begun the process of communicating it to key stakeholders, including clinical and logistics staff. Further, VA is developing a separate strategy to involve clinicians in developing requirements with plans to complete a pre-pilot of this strategy by September 2019. In response to a congressional request to assess these and other program changes, we recently began a review of VA’s MSPV program. Inadequate acquisition training. VA acquisition training, at times, has not been comprehensive nor provided to staff that could benefit from it. A 2006 statute required, and a 2016 Supreme Court decision (Kingdomware Technologies, Inc. v. United States) reaffirmed, that VA is to give preference to veteran-owned small businesses when competitively awarding contracts—a program known as Veterans First. In September 2018, we reported that training on VA’s Veterans First policy did not address some of its more challenging aspects. For example, many of the contracting officers we interviewed were uncertain about how to balance the preference for veteran-owned small businesses with fair and reasonable price determinations when lower prices might be found on the open market. In addition, VA provided several installments of online training sessions on the Veterans First policy to contracting officers but did not make them mandatory. As a result, only 52 percent of VA contacting officers completed the follow-up training by the spring of 2018. We recommended in September 2018 that VA provide more targeted training to contracting officers on how to implement the Veterans First policy, particularly in the area of making fair and reasonable price determinations, and assess whether this training should be designated as mandatory. VA concurred, and in April 2019, VA’s Chief Acquisition Officer (CAO) stated that VA is taking steps to make this training mandatory. VA also reported that its Acquisition Academy will provide Veterans First training to all contracting staff on May 30, 2019. Contracting officer workload challenges. The majority of our reviews since 2015 have highlighted workload as a contributing factor to the challenges that contracting officers face. Most recently, in September 2018, we reported that about 54 percent of surveyed VA contracting officers said their workload was not reasonable and found that workload stresses have exacerbated the struggles that they face implementing the department’s Veterans First policy. In addition, in September 2016, we reported that VHA contracting officers processed a large number of small dollar-value actions to support medical center operations, many of which involve emergency procurements of routine items to support immediate patient care. Contracting officers and the department’s Acting CAO told us that these frequent and urgent small-dollar transactions reduce contracting officers’ efficiency and ability to take a strategic view of VHA’s overarching procurement needs. We reported in November 2017 that emergency procurements accounted for approximately 20 percent—$1.9 billion—of VHA’s overall contract actions in fiscal year 2016. Figure 3 shows the percent of VHA contract actions designated as emergencies in fiscal year 2016 by each network contracting office. We recommended in November 2017 that VHA network contracting offices work with medical centers to identify opportunities to more strategically purchase goods and services frequently purchased on an emergency basis. VA concurred with this recommendation and recently offered to provide us with a demonstration of the supply chain dashboard that VA uses to track items purchased on an emergency basis, which we plan to attend by the end of May 2019. VA also agreed to conduct an analysis of its purchase card spending to identify items that should be purchased through its MSPV program. VA expects to complete this analysis by July 2019. If implemented, this would allow for both greater contracting officer efficiency and cost savings. For example, based on a similar recommendation we made in 2012, VA began more systematically employing strategic sourcing in FY 2013, and in subsequent fiscal years reported about $10 billion in savings over a 5-year period. Lack of reliable data systems. The lack of accurate data has been a long-standing problem at VA. In September 2016, we reported that VA had not integrated its contract management and accounting systems, resulting in duplicative efforts on the part of contracting officers and increased risk of errors. We and VA’s Inspector General each recommended that VA perform data checks between the two systems. VA concurred with this recommendation and some VA contracting organizations have made efforts to address this risk. Further, VA reported in March 2019, that it plans to adopt a new integrated financial and contract management system, which it plans to install VA-wide over a 9- year period, with the final site receiving the system in 2027. Limited contract oversight and incomplete contract file documentation. VA has had difficulty ensuring that its contracts are properly monitored and documented. In September 2018, we reported that, although VA obligated $3.9 billion to veteran-owned small businesses in fiscal year 2017, its contracting officers were not effectively monitoring compliance with key aspects of the department’s Veterans First policy, such as limits on subcontracting (which ensure that the goal of the program—to promote opportunities for veteran-owned businesses—is not undermined). In many cases, we found that clauses requiring compliance were not included in the VA’s contracts and orders with veteran businesses because the contracting officers either forgot to include them or were unaware of the requirement. The contracting officers we spoke with also said that they do not have sufficient time or knowledge to conduct oversight. Through limited reviews, VA has identified a number of violations that would warrant a broader assessment of the fraud risks to the program. We recommended in September 2018 that VA establish a mechanism to ensure that mandatory subcontracting-related clauses be consistently incorporated into set-aside contracts with veteran-owned businesses and that VA conduct a fraud risk assessment for the Veterans First program. VA concurred with these recommendations and is taking steps to implement them. For example, VA reported in April 2019 that it had made modifications to its electronic contract management system to ensure the clauses would be included in set-aside contracts and anticipated completing testing of the modifications in May 2019. We also reported in September 2016 that a number of VA contract files we reviewed were missing key documents, increasing the risk that key processes and regulations were not followed. We recommended that VA focus its internal compliance reviews to ensure that required contract documents are properly prepared and documented. VA concurred with this recommendation. Since then, VA has made policy changes that revised its processes for compliance reviews of contract documentation. We are currently following up with VA to obtain the results of its compliance reviews to determine if VA has fully implemented this recommendation. Leadership instability. We have previously reported, most recently in September 2018, that procurement leadership instability has made it difficult for the VA to execute and monitor the implementation of key acquisition programs and policies. For example, changes in senior procurement leadership, including the CAO and VHA’s Chief Procurement and Logistics Officer, occurred during the implementation of MSPV-NG and similar instability in leadership affected the MSPV-NG program office itself. Overall, the MSPV-NG program office has had four directors, two of whom served in an acting capacity, since its inception in 2014. To address this instability, we recommended in November 2017 that VA appoint a non-career employee as the CAO and prioritize the hiring of the MSPV-NG program office’s director position on a permanent basis. VA concurred with these recommendations and implemented them in 2018. Stable leadership should help bring consistent and much needed direction to the MSPV-NG program, but we recently identified other areas within the VA where sustained leadership is also needed. For instance, in September 2018, we reported there have been six Acting Directors within the past 2 and a half years within an oversight office that helps assess whether VA is in compliance with aspects of its Veterans First policy. We designated improving and modernizing federal disability programs as high risk in 2003. An estimated one in six working-age Americans reported a disability in 2010. Many of these Americans need help finding or retaining employment, or rely on cash benefits if they cannot work. Three of the largest federal disability programs—one run by VA— disbursed about $270 billion in cash benefits to 21 million people with disabilities in fiscal year 2017. However, federal disability programs, including VA’s, struggle to meet their needs. In particular, VA struggles to manage its disability claims workloads, and, when determining whether individuals qualify for disability benefits, VA relies on outdated eligibility criteria. Managing disability claims workloads. Since our 2017 High-Risk Report, our assessment of ratings for all five criteria remains unchanged for this area of concern for VA as of March 2019. Leadership commitment: met. VA has maintained leadership focus on managing initial disability claims and appeals workloads through various initiatives to improve benefits processing and reduce backlogs. Enhancing and modernizing VA’s disability claims and appeals processes are goals in its 2018–2024 strategic plan. Capacity: partially met. VA has continued building the capacity to process initial disability claims, such as using an electronic system to distribute claims ready for decisions to available staff. On appeals, VA is reforming its process, onboarding hundreds of new staff, and implementing new technology. However, as we reported in March 2018, VA’s appeals plan does not provide reasonable assurance that it will have the capacity to implement the new process and manage risks. VA agreed with our recommendation to better assess risks associated with appeals reform and took some steps to address risks, such as limited testing of the new process. However, as of April 2019 VA has not fully addressed this recommendation. For example, VA has not developed plans to fully address risks, such as veterans choosing more resource-intensive options at higher rates than expected. Action plan: partially met. VA continues to implement plans to reduce the initial disability claims backlog. For appeals reform, VA submitted its appeals plan in November 2017 and provided several progress reports throughout 2018. In March 2018, we reported that VA’s plan for implementing a new disability appeals process did not explain how VA would assess the new process compared to the legacy process, and did not fully address risks associated with implementing a new process. We made two recommendations to improve VA’s disability benefit appeals process, including that VA (1) clearly articulate in its appeals plan how it will monitor and assess the new appeals process compared to the legacy process, and (2) ensure that its appeals plan more fully addresses related risks, given the uncertainties associated with implementing a new process. As of April 2019, VA has taken actions to address our recommendations, although key steps remain. For example, VA has not fully articulated detailed steps and time frames for assessing the relative performance of the new and legacy appeals processes. Without this assessment, VA cannot determine the extent to which the new process will achieve final resolution of veterans’ appeals sooner than the legacy process. Monitoring: partially met. VA monitors the timeliness of initial disability claims and legacy appeals, and has set timeliness goals for some, but not all, of the appeal options under the new process. VA’s plans also signal how it intends to monitor the allocation of staff for concurrent workloads in its legacy and new appeals processes. However, as of April 2019, VA has yet to specify a complete set of balanced goals for monitoring the new and legacy appeals processes (including timely and accurate processing of appeals while ensuring veteran satisfaction). Demonstrated progress: partially met. VA reported it reduced the backlog of initial disability claims from 611,000 in March 2013 to about 81,000 at the end of fiscal year 2018. However, VA’s Inspector General reported in September 2018 that VA overstated its performance by only reporting about 79 percent of the backlog. For appeals, VA addressed some gaps in its plan for implementing appeals reform, in accordance with our 2017 and 2018 recommendations, and has prioritized processing of legacy appeals. However, as of September 2018, VA still had a backlog of about 396,000 legacy appeals. Updating disability benefit eligibility criteria. Since our 2017 High-Risk Report, VA’s ratings for the action plan and monitoring criteria regressed while the other three remain unchanged as of March 2019. Leadership commitment: met. VA has sustained leadership focus on updating its Veterans Affairs Schedule for Rating Disabilities (VASRD)—used to assign degree of disability and compensation levels for veterans with military service-connected injuries or conditions—to reflect advances in medicine and labor market changes. Capacity: partially met. In August 2017, VA officials told us that it had taken actions to hire more staff for the regulations updates and leverage outside researchers to evaluate veterans’ loss of earnings in the current economy. However, as of September 2018, the agency was still working to hire these staff. Moreover, VA’s current earnings loss study covers only 8 of over 900 diagnostic codes and 2 of 15 body systems. VA needs to continue its current hiring and earnings loss planning efforts to ensure it has the capacity to comprehensively update the VASRD. Action plan: partially met. In March 2019, the rating for this criterion declined to partially met. As of April 2019, VA’s efforts to update the VASRD included new plans to conduct earnings loss studies. Veterans Benefits Administration officials stated they completed a study for eight diagnostic codes under two body systems, and the agency is determining whether its current approach for evaluating earnings loss is applicable to updating other diagnostic codes. However, we lowered VA’s prior rating of met to partially met because its latest August 2018 updated plan, issued since our 2017 High-Risk Report, provided limited detail on key planned activities, potentially jeopardizing its third attempt at modernization over the past decade. For example, VA’s plans do not indicate how and when VA will assess the applicability of its current approach, and does not include plans for updating earnings loss information for the remaining diagnostic codes and body systems. Monitoring: partially met. In March 2019, the rating for this criterion declined to partially met. According to VA officials, VA continues to track its progress toward finishing the medical updates by fiscal year 2020 and has updated its project plan to reflect delayed time frames. However, we lowered VA’s prior rating for this criterion from met to partially met because VA’s plans have changed since our last update, and although it is conducting a study to update earnings loss information for some diagnostic codes and body systems, its plan does not include timetables for monitoring these or future updates to earnings loss information. Demonstrated progress: partially met. VA reported that as of December 2018, it promulgated final regulations for 6 of 15 body systems, proposed regulations for 2, and is reviewing draft regulations for the remaining 7. However, VA has fallen about 4 years behind in its efforts to fully update the VASRD and has not completed earnings loss updates. Several other government-wide high-risk areas include VA and its operations. These areas include (1) improving the management of IT acquisitions and operations, (2) strategic human capital management, (3) managing federal real property, and (3) ensuring the cybersecurity of the nation. Improving the management of IT acquisitions and operations. The executive branch has undertaken numerous initiatives to better manage the more than $90 billion that is annually invested in IT across the government. However, our work shows that federal IT investments, including those made by VA, too frequently fail or incur cost overruns and schedule slippages while contributing little to mission-related outcomes. Thus, in 2015, we added improving the management of IT acquisitions and operations to the High-Risk List. To address the portion of the high-risk area for which it is responsible, VA should, among other things, implement our past recommendations on improving IT workforce planning practices and establishing action plans to modernize or replace obsolete IT investments. In August 2018, for example, we found that VA’s policies did not fully address the role of its CIO consistent with federal laws and guidance in the areas of IT workforce, IT strategic plan, IT budgeting, and IT investment management. Until VA fully addresses the role of the CIO in all of its policies, it will be limited in addressing longstanding IT management challenges. We recommended that VA’s IT management policies address the role of the CIO for key responsibilities in the four areas we identified. VA concurred with this recommendation, which remains open. Strategic human capital management. This area was added to our High-Risk List in 2001 and continues to be at risk today because mission-critical skills gaps both within federal agencies and across the federal workforce are impeding the government from cost-effectively serving the public and achieving results. As of December 2018, VA reported an overall vacancy rate of 11 percent at VHA medical facilities, including vacancies of over 24,000 medical and dental positions and around 900 human resource positions. Also, with 32 percent of the VA workforce eligible to retire in the next 5 fiscal years, VA must address these mission-critical skill gaps and vacancies that we continue to identify in our work. In December 2016, for example, we found that VHA’s limited human resources capacity combined with weak internal control practices has undermined VHA’s human resources operations and its ability to improve delivery of health care services to veterans. Further, VHA is challenged by inefficiencies in its performance management processes, including the lack of a performance appraisal IT system, which prevents it from identifying trends and opportunities for improvement. VHA can better support medical centers by establishing clear lines of accountability for engagement efforts, collecting and leveraging leading practices, and addressing barriers to improving engagement. We made three recommendations to VA to improve its performance management system. VA partially concurred with these recommendations, which remain open. Managing federal real property. Since federal real property management was placed on the High-Risk List in 2003, the federal government has given high-level attention to this issue. However, federal agencies, including VA, continue to face long-standing challenges, including (1) effectively disposing of excess and underutilized property, (2) relying too heavily on leasing, (3) collecting reliable real property data for decision making, and (4) protecting federal facilities. In January 2019, for example, we reported that VA has enhanced its data collection on vacant properties, but the agency does not collect information needed to track and monitor disposal projects at the headquarters level. Without information on the status of disposal projects, VA cannot readily track and monitor its progress and identify areas where facilities’ managers may need additional assistance. As a result, we recommended that VA improve its procedures related to disposal of excess and underutilized property to help local facility managers plan, implement, and execute projects to dispose of those properties. In addition, VA should collect key information on the status of these disposal projects to help manage the process and identify areas where management attention is needed. VA concurred with the three recommendations we made related to these findings, all of which remain open. Ensuring the cybersecurity of the nation. We have designated information security as a government-wide high-risk area since 1997. We expanded this high-risk area in 2003 to include protection of critical cyber infrastructure and, in 2015, to include protecting the privacy of personally identifiable information. Federal agencies and our nation’s critical infrastructures are dependent on IT systems and electronic data to carry out operations and to process, maintain, and report essential information. The security of these systems and data is vital to public confidence and national security, prosperity, and well- being. Because many of these systems contain vast amounts of personally identifiable information, agencies must protect the confidentiality, integrity, and availability of this information. In addition, they must effectively respond to data breaches and security incidents when they occur. In May 2016, for example, we found that VA had developed a risk assessment for their selected high-risk systems, but had not always effectively implemented access controls. These control weaknesses included those protecting system boundaries, identifying and authenticating users, authorizing access needed to perform job duties, and auditing and monitoring system activities. Weaknesses also existed in patching known software vulnerabilities and planning for contingencies. An underlying reason for these weaknesses is that the key elements of information security programs had not been fully implemented. VA concurred with all of our five recommendations related to improving its cybersecurity controls. However, two recommendations—which specifically call for the department to conduct security control assessments and develop a continuous monitoring strategy—remain open. In November 2018, the department’s inspector general reported that VA had made progress in developing, documenting, and distributing policies and procedures to support its security program, but identified IT security as a major management challenge due to the persistence of deficiencies. For example, the inspector general identified significant deficiencies related to access, configuration management, change management, and service continuity. In addition, VA’s financial statement auditor reported deficiencies in the department’s IT security controls as a material weakness for financial reporting purposes. The auditor has reported IT security controls as a material weakness for more than 10 years. Since his confirmation in July 2018, Secretary Wilkie has demonstrated his commitment to addressing the department’s high-risk designations by, among other things, creating an office to direct an integrated approach for high-risk concerns and communicating to VA leaders the importance of addressing our recommendations. Additionally, VA leadership has also encouraged senior leaders to meet with GAO subject matter experts from acquisition, performance, human capital, and financial management, among other areas, to discuss leading practices and VA’s modernization efforts. In addition, senior leaders from GAO and VA meet regularly to identify and address the root causes of high-risk issues, and discuss the status of our recommendations and VA’s efforts to address them. Fully addressing these issues will require sustained leadership attention on these issues as well as leadership stability—something that VA has not had in recent years. In particular, in the 2 years prior to Secretary Wilkie’s confirmation, VA experienced leadership instability with senior- level vacancies in key positions, including the Under Secretary for Health, CIO, and Deputy Under Secretary for Health for Community Care. In addition to sustained leadership, VA must develop action plans for addressing the high-risk issues. As noted earlier, VA officials have stated that they are currently working to address our high-risk concerns through the implementation of the VHA Plan for Modernization. The plan, which identifies high-level implementation targets through 2020, provides a framework to address the Secretary’s four priorities: (1) improving training and customer service; (2) implementing the VA MISSION Act and improving veterans’ access to care; (3) connecting the VA’s electronic health records system to the DOD’s to ensure a continuum of care for transitioning service members; and (4) transforming VA’s business systems. As part of this effort, VA is focused on “10 lanes of effort,” including transitioning to the same electronic health record system the DOD is currently deploying, and transforming its business systems— including its human resource management, finance and acquisition management, and supply chain functions—to improve the quality and availability of services at VA medical centers. In closing, VA has launched several significant efforts to address many of the underlying management challenges it faces, including transforming its electronic health record and financial management systems, updating its medical surgical prime vendor program, and implementing the VA MISSION Act. Any one of these efforts would be a significant undertaking for an agency given their scope, time frames, and costs, and VA is attempting to concurrently implement them. If successful, these efforts could be transformative for VA. Sustained congressional oversight of VA’s efforts will also be needed. We stand ready to support this oversight through continued monitoring of VA’s efforts as it ensures that the modernization efforts integrate and address many of the concerns that led to the designation of various VA areas as high risk. Chairman Pappas, Ranking Member Bergman, and Members of the Subcommittee, this concludes my statement. I would be pleased to respond to any questions you may have. For further information about this statement, please contact Debra A. Draper at (202) 512-7114 or draperd@gao.gov or Sharon M. M. Silas at (202) 512-7114 or silass@gao.gov for VHA health care issues; Shelby S. Oakley at (202) 512-4841 or oakleys@gao.gov for VA acquisition management issues; or Elizabeth H. Curda at (202) 512-7215 or curdae@gao.gov for VA disability claims issues. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement were Ann Tynan, Mark Bird, David Bruno, Keith Cunningham, Cathleen Hamann, Lisa Gardner, Steven Lozano, William Reinsberg, Maria Storts, Jamie Whitcomb, Amanda Cherrin (Analyst-in-Charge), Kate Tussey, Jeff Hartnett, and Teague Lyons. Vikki Porter and Jacquelyn Hamilton also contributed to this statement. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "VA is responsible for providing benefits and services to veterans, including health care, disability compensation, and various types of financial assistance. In fiscal year 2019, VA received a total budget of $201.1 billion and a discretionary budget of $86.6 billion—the largest in VA's history—to carry out its mission. GAO, along with the VA Inspector General and other entities, continues to identify significant deficiencies in VA's governance structures and operations—all of which can affect the care provided to our nation's veterans. This testimony focuses on the status of VA's efforts to address GAO's high-risk designations and open GAO recommendations in the following areas: VA health care, acquisition management, and disability claims workloads and benefit eligibility criteria, among other areas. It is primarily based on GAO's March 2019 high-risk update and a body of work that spans more than a decade. The Department of Veterans Affairs (VA) has longstanding management challenges. As a result, GAO added several VA programs to its High-Risk List. This list focuses attention on government operations that are most vulnerable to fraud, waste, abuse, or mismanagement, or in need of transformation. These include managing risks and improving VA health care, VA acquisition management, and improving and modernizing VA disability programs, including managing claims and updating eligibility criteria. VA health care was designated high risk in 2015 due to concerns about VA's ability to ensure the cost-effective and efficient use of resources to improve the timeliness, quality, and safety of health care for veterans. GAO identified five areas of concern: (1) ambiguous policies and inconsistent processes; (2) inadequate oversight and accountability; (3) information technology challenges; (4) inadequate training for VA staff; and (5) unclear resource needs and allocation priorities. VA's efforts to address each of these areas have been impeded by leadership instability. However, since his July 2018 confirmation, Secretary Wilkie has demonstrated his commitment to address the department's high-risk designations. His actions to date have allowed the department to maintain its leadership commitment rating of partially met in GAO's 2019 High-Risk update. VA also partially met the action plan criteria. As of March 2019, it did not meet the other three criteria for removal from the High-Risk List (agency capacity, monitoring, and demonstrated progress). This is, in part, because GAO continues to have audit findings that illustrate that the five areas of concern have not been fully addressed. For example: In a series of reports from 2012 through 2018, GAO found VA's wait time data unreliable for primary and specialty care as well as for care in the community. GAO also found that VA did not measure the full wait times that veterans experience in obtaining care across these settings. In November 2017, GAO reported that VA medical center officials did not always conduct or document timely required reviews of providers when allegations of wrongdoing were made against them. In April 2019, GAO found that VA's governance plan for modernizing its electronic health record system was not fully defined, potentially jeopardizing its fourth attempt at modernization. In April 2019, GAO reported that VA's appraisal process for assessing medical center director performance relies heavily on a system with long-identified deficiencies that remain unaddressed, thus diminishing VA's ability to hold officials accountable. In its 2019 High-Risk Report, GAO added VA acquisition management as a high-risk area in light of the department's numerous contracting challenges and the significant federal investment in serving veterans. To date, GAO has identified challenges in the following areas: (1) outdated acquisition regulations and policies; (2) lack of an effective medical supplies procurement strategy; (3) inadequate acquisition training; (4) contracting officer workload challenges; (5) lack of reliable data systems; (6) limited contract oversight and incomplete contract documentation; and (7) leadership instability. For example, as of May 2019, VA does not have updated acquisition regulations and officials expect to have a full update by 2021; a process which has been in place since 2011. GAO designated improving and modernizing federal disability programs, including VA's program, as high risk in 2003. GAO identified two areas of concern related to VA: (1) managing disability claims workload and (2) updating disability benefit eligibility criteria. As a result of these concerns, veterans may not have their disability claims and appeals processed in a timely manner. GAO reported in March 2018 that VA is making a major effort to reform its appeals process by onboarding new staff and implementing new technology. However, its appeals planning process does not provide reasonable assurance that it will have the capacity to successfully implement the new process and manage risks. VA agreed with GAO's recommendation to better assess risks associated with appeals reform. VA leadership has committed to addressing GAO's high-risk concerns and has launched several transformational efforts. For example, VA is currently implementing the Veterans Health Administration Plan for Modernization, a framework that aims to modernize the department, as well as the VA MISSION Act of 2018. This Act requires VA to consolidate programs that allow veterans to receive care outside VA. If successful, these efforts could be transformative for VA. However, such success will only be achieved through sustained leadership attention and detailed action plans that include metrics and milestones to monitor and demonstrate VA's progress. Sustained congressional oversight will also be essential. Since 2000, GAO has made more than 1,200 recommendations to reduce VA's high-risk challenges, and VA has implemented approximately 70 percent. GAO will continue to monitor VA's progress in implementing the remaining open recommendations.", "document_type": "gao"}
{"report": "Mobile devices, such as smartphones and tablets, use wireless networks to enable voice and data communications. These wireless networks comprise several components. For example, cell sites—a base station equipped with an antenna—receive and transmit radio signals to mobile devices. In addition to traditional “macro cells,” 5G networks also use smaller wireless infrastructure, known as “small cells,” which can be installed on existing structures, including macro towers, buildings, utility poles, or streetlights. Base station controllers manage communications between the cell site and the mobile switching center, or the “core network.” The core network then directs the communication to landline phones, other cell phones, or the internet. Finally, backhaul facilities, such as fiber optic cables or microwaves, transport the communications (See fig. 1.). Cell sites use radio frequency spectrum to receive and transmit radio signals to and from mobile devices. Spectrum is a finite natural resource used to provide a variety of communication services to businesses and consumers, as well as federal, state, and local governments. Businesses and consumers use spectrum for a variety of wireless services including mobile voice and data, Wi-Fi- and Bluetooth-enabled devices, broadcast television, radio, and satellite services. Federal, state, and local governments’ uses of spectrum include national defense, law enforcement communication, air traffic control, weather services, military radar, and first responder communications. The frequency bands have different characteristics that make them more or less suitable for specific purposes. For example, different bands have different limits to the amount of information that they can carry, known as “data capacity,” and different levels of ability to effectively penetrate or bend around physical obstacles and cover distances, known as “propagation.” Regarding wireless communication: Low-band spectrum (generally defined as under 1 gigahertz (GHz)) typically has relatively low data capacity but has propagation characteristics that enable transmission over longer distances and penetration of buildings and other physical barriers better than higher bands. Mid-band spectrum (generally defined as between 1 GHz and 6 GHz) tends to provide greater data capacity than low bands and has better propagation qualities than higher bands. High-band spectrum (generally defined as those above 24 GHz) allows for high data capacity but has relatively limited propagation, to the point that bands at higher frequencies (according to FCC, those above 95 GHz) are most prone to obstruction by natural or manmade objects, such as trees or glass. In the United States, two federal agencies are primarily responsible for managing spectrum. FCC is the federal agency responsible for allocating spectrum for consumer and commercial purposes (as well as state and local government uses), assigning spectrum licenses to those entities, and making spectrum available for use by unlicensed devices. Licensing assigns frequencies of spectrum, in a specific area, and—generally speaking, according to FCC officials—to a specific entity, such as a telecommunications company. NTIA is responsible for establishing policy on regulating federal spectrum use. NTIA assigns frequencies to government agencies, maintains federal spectrum use databases for those assignments, and oversees, in cooperation with other relevant federal agencies, which spectrum bands reserved for federal government use might be made available for commercial use. Approximately every 10 years since around the early 1980s, wireless carriers have deployed a new generation of technology, and each development has changed how people and businesses use mobile communication. These technologies can bring greater speed and capabilities to mobile networks and can provide new revenue streams for carriers and economic gains for national economies. For example, a trade organization representing carriers reported that U.S. leadership in developing and deploying 4G brought in significant economic benefits, adding billions of dollars to the U.S. economy. The carriers we spoke with are currently developing and deploying 5G networks, which will allow for enhanced mobile broadband, offering greater speed and higher data capacity than previous generations of mobile wireless networks. Carriers in the United States are currently deploying 5G as “hybrid” 5G, which uses 5G technologies in combination with existing 4G networks to improve the networks’ speed by enhancing the technology that connects a user device to a core network. In the future, carriers that want to deploy “standalone” 5G will have to replace their existing 4G network infrastructure with new 5G equipment to enhance the core network. The new, standalone 5G networks will allow for additional enhanced capabilities, such as lower “latency,” and will be better able to support other advanced use cases (see fig. 2). Carriers have thus far deployed limited 5G services as hybrid 5G networks in the United States, and are taking different approaches with regard to spectrum use for 5G. For example, some carriers told us that they are relying on low-band spectrum for their 5G network. Other carriers are using high-band spectrum in limited locations. In both cases, customers need to purchase new 5G-capable smartphones to use these hybrid 5G networks. In general, telecommunications networks, including mobile networks, provide important economic and educational opportunities to communities, but different socioeconomic groups and groups in different geographic areas have historically received different levels of access to telecommunications services, leading to a disparity called the “digital divide.” A number of factors explains the digital divide. For example, as we have reported in the past, rural areas tend to have conditions—such as low population density or difficult terrain—that can increase the costs for carriers to deploy and maintain networks. Furthermore, lower-income households may have access to the necessary infrastructure for service but may not be able to afford the service. Experts we convened told us that the lack of sufficient access to mid- band spectrum is a key challenge to deploying 5G, noting that mid-band spectrum is particularly important for 5G deployment because of its network characteristics and potential to be interoperable with other 5G networks worldwide. Experts stated that the availability of mid-band spectrum to carriers in the United States is not yet sufficient to meet carriers’ needs for 5G network deployment because of existing congestion within the band. Experts stated that carriers will need a mix of low-, mid-, and high-band spectrum when deploying 5G networks because of the network characteristics unique to each spectrum band. For example, one expert noted that mid-band spectrum provides 5G network characteristics that cannot be achieved using solely low- or high-band spectrum. Signals using mid-band spectrum have better propagation (i.e., ability to effectively penetrate or bend around physical obstacles and cover distances) than signals using high-band spectrum (see fig. 3) and carry more data than low-band spectrum. Global harmonization of spectrum, or the use of the same spectrum bands among countries around the world, helps ensure that 5G devices will work across countries. Countries that harmonize spectrum for 5G may benefit by making international travel and communication more convenient. For example, consumers from countries that deploy 5G using the same spectrum bands will have the benefit of roaming across networks. Spectrum harmonization also creates economies of scale that can reduce the costs of manufacturing wireless devices and deploying network equipment. Countries, including the United States, have identified specific frequencies in mid-band spectrum that may be used for 5G. However, experts told us that, as currently allocated, mid-band spectrum is highly congested, leading to an insufficient amount available for carriers to deploy 5G networks in the United States. According to NTIA officials, current mid-band spectrum users—known as “incumbents”—include federal government users that have primary access rights to the spectrum and face challenges in readily transitioning to new or less favorable spectrum bands. For example, agencies’ existing technologies may be designed specifically for their existing spectrum bands. Additionally, according to FCC officials, it is becoming increasingly challenging to relocate federal users out of a spectrum band entirely and into a new band due to a variety of factors, including concerns about potential interference as well as greater spectrum use in recent years. According to experts, large consecutive portions of spectrum will be necessary for commercial users deploying 5G networks. Using smaller or non- consecutive portions of spectrum may limit the capability of the network. According to FCC officials, FCC has taken several actions to make additional spectrum available for carriers planning to deploy 5G networks. Some examples of FCC’s actions to make low-, mid-, and high-band spectrum available for 5G deployment include: Low-Band: FCC concluded an auction in 2017 for low-band 600 megahertz (MHz) spectrum licenses, assigning 70 MHz for licensed wireless operations. Such spectrum auctions allow FCC to use competitive bidding to choose from among two or more applications for a spectrum license. Mid-Band: FCC issued a Report and Order in July 2019 that made spectrum licenses within the 2.5 GHz band accessible to nonfederal users. High-Band: FCC held its first auctions for high-band 5G spectrum in the 24 GHz and 28 GHz bands in 2018 and 2019. FCC also began an auction of 3,400 MHz of spectrum in the upper 37, 39 and 47 GHz bands in December 2019; bidding in this auction concluded on March 5, 2020. FCC officials told us that they are aware that mid-band spectrum will be particularly important for 5G deployment despite congestion amongst federal users in this spectrum range. FCC is taking steps to make some additional mid-band spectrum available. For example, in February 2020, FCC announced that it had adopted new rules to auction 280 MHz of mid- band spectrum, which can be used for 5G purposes. This spectrum, (3.7 to 3.98 GHz) is currently being used primarily by satellite operators. In March 2020, FCC released a public notice seeking comment on procedures to be used for the auction of this spectrum, which is currently scheduled to begin on December 8, 2020. As another example, according to FCC, in April 2020 FCC provided for expanded Wi-Fi use in 1,200 MHz of spectrum in the 6 GHz band. Such advanced Wi-Fi networks, FCC told us, will be capable of working hand-in-hand with commercial networks to enable robust 5G device broadband connectivity and may be able to help alleviate commercial wireless network congestion. Other activities, according to FCC, include: (1) opening a proceeding in the 3.1 to 3.55 GHz band to consider potential shared use between federal operations and flexible use commercial services; and (2) authorizing a private entity to deploy a low-power terrestrial nationwide network in certain frequencies that will make available additional spectrum for advanced wireless services, including 5G. FCC and NTIA also developed a spectrum-sharing framework in the 3.5 GHz band that will increase the availability of this mid-band spectrum targeted globally for 5G. This framework separates users into three hierarchical “tiers,” giving differing priority access to the spectrum (3.55 GHz to 3.7 GHz, also known as the “Citizens Broadband Radio Service”) depending on the user’s tier. The first tier includes incumbents, such as federal users (e.g., U.S. Navy radar systems) and a number of commercial users. These users receive first priority and protection from all other users. Tier two users—referred to as “Priority Access Licensees”—are, according to FCC, wireless users that obtain licenses at auction or, following the auction, via secondary markets. These users, which can include wireless carriers, have access to the same mid-band spectrum when a tier one user is not using the spectrum, but FCC officials said these users will need to move to another frequency when a nearby tier one user accesses the same frequency. Third tier users access the band as available. FCC officials stated that this spectrum-sharing framework will allow for increased spectral utilization of mid-band spectrum in a band like 3.5 GHz and that individual users (e.g., the public using mobile devices) will not notice any difference in their network connection. According to FCC officials, the technology supporting this spectrum-sharing framework is now authorized for full commercial deployment. They also said that FCC certified administrators in January 2020 to coordinate this framework, which will allow for full commercial operation, and that FCC has scheduled to begin auctioning licenses for tier two users on July 23, 2020. Other federal agencies are also involved with managing spectrum in the United States. For example, NTIA, which manages federal spectrum use, is working with FCC on the technical design and implementation of the spectrum-sharing framework discussed above. NTIA is also seeking to identify additional spectrum for 5G, in conjunction with FCC. According to FCC, some of the most useful portions of mid-band spectrum are already occupied by a federal incumbent and FCC is limited in its ability to make this spectrum available for commercial use. According to NTIA officials, the agency is focused on meeting the spectrum requirements set forth in the Making Opportunities for Broadband Investment and Limiting Excessive and Needless Obstacles to Wireless Act (MOBILE NOW Act) of 2018, which requires NTIA and FCC to prepare a report by 2022 identifying potential spectrum for future use. For example, NTIA is currently studying the feasibility of spectrum sharing in the 3.45 to 3.55 GHz band. Overall, FCC’s efforts, in conjunction with NTIA, to date have primarily made more high-band spectrum available for 5G purposes. According to the Department of Commerce’s 2019 Annual Report on the Status of Spectrum Repurposing, 84 percent (4,950 MHz out of 5,863 MHz) of the spectrum made available by FCC and NTIA has been within high-band. According to the report, 12 percent (709 MHz of 5,863 MHz) of the spectrum FCC and NTIA have made available has been within mid-band. NTIA officials said there has been more of a focus on repurposing high- band spectrum because there is a far greater amount of this spectrum available for use and fewer incumbent users within this spectrum. Further, NTIA officials stated that these amounts are a snapshot as of the time the 2019 report was issued and the ratios will change, as additional spectrum is made available. Other recent FCC actions, including those described above, may make more mid-band spectrum available in the future. For example, FCC told us that it has a number of active proceedings that could make additional mid-band spectrum available to commercial users. To guide its 5G-related efforts, including spectrum management, FCC has developed its Facilitate America’s Superiority in 5G Technology Plan (5G FAST Plan). This Plan includes three “key components: (1) pushing more spectrum into the marketplace; (2) updating infrastructure policy; and (3) modernizing outdated regulations.” According to FCC officials, the 5G FAST Plan represents FCC’s strategy for supporting 5G, and these key components are FCC’s broad strategic goals for 5G. However, FCC has not laid out in the 5G FAST Plan how it will implement and assess progress toward the three key components. Our past work on strategic planning has identified related leading practices. These include identifying: (1) specific and measurable performance goals to show progress toward broad strategic goals; (2) the activities (also known as strategies) the agency will take to make progress toward its goals; and (3) related performance measures to assess actual progress made toward the performance goals. Although FCC’s 5G FAST Plan notes actions or strategies FCC has taken regarding managing spectrum for 5G, it does not clearly identify specific and measurable performance goals and related measures for spectrum management related to 5G deployment. For example, the plan notes that FCC’s actions on the 2.5 GHz, 3.5 GHz, and 3.7 - 4.2 GHz bands could make up to 844 MHz available for 5G, but these strategies are not related to any identified performance goals or measures. Without such strategic planning efforts, it is unclear if these actions will be sufficient to address the challenges experts raised about the lack of mid-band spectrum for 5G. Additionally, establishing performance goals and measures would allow FCC to assess its spectrum management strategies and track the progress it is making toward its goals. Further, according to FCC officials, the priorities noted in the 5G FAST Plan were not developed with outside entities, such as NTIA or other relevant stakeholders, including carriers. Leading practices, as identified in our previous work, show that successful organizations base their strategic planning, to a large extent, on the interests and expectations of their stakeholders, which could include other federal agencies, Congress, and others. Thus, involving stakeholders in the strategic planning process helps ensure that the agency’s efforts are targeted at the highest priorities. According to FCC officials, it can be difficult to set goals for specific amounts of spectrum to be made available because of, in part, the fast- changing nature of the telecommunications industry. While we recognize that setting such goals, measures, or strategies may be difficult, our past work on strategic planning has found that there is no more important element in results-oriented management than an agency’s strategic planning effort. This effort is the starting point and foundation for defining what the agency seeks to accomplish, identifying the strategies it will use to achieve desired results and then determining how well it succeeds in reaching results-oriented goals and achieving objectives. Proactively developing performance goals, strategies and measures— with the involvement of relevant stakeholders—to manage spectrum demands associated with 5G deployment would help ensure that sufficient amounts of spectrum in consecutive portions are made available to avoid delaying the deployment and limiting the capabilities of 5G networks. Additionally, by incorporating these key elements into its strategic planning for 5G, FCC would be able to assess its progress in managing spectrum, particularly the congested mid-band spectrum that is important to 5G deployment. Experts we convened told us that 5G deployment, especially high-band 5G networks, will likely widen the existing digital divide, particularly between urban and rural areas, as well as within urban areas. Experts and stakeholders told us that 5G using high-band spectrum is likely to be first deployed in areas already equipped with much of the necessary infrastructure (i.e., fiber and power). Experts said these areas are generally more urban, densely populated, high-income areas as opposed to rural or low-income areas. Stakeholders told us that rural areas will see 5G deployed mostly on lower frequencies, on which signals can propagate further but which cannot carry the same bandwidth (i.e., data throughput) as higher frequencies can. Within urban settings, experts said that high-band 5G networks are more likely to be deployed in commercially viable areas, including those parts of a city that already are equipped with fiber and power and, presumably, already benefit from the most advanced mobile broadband services available. For example, an expert representing the wireless industry stated that only about 10 percent of the District of Columbia would receive 5G services using high- band spectrum, as it would be cost-prohibitive for the carriers to install 5G using this spectrum beyond that 10 percent of the city. Experts told us that individuals without access to 5G networks will not be able to take advantage of the use cases that 5G promises, including the high-speed connections offered by enhanced mobile broadband. Experts stated that this situation will greatly affect, among other things, the economic and educational opportunities that 5G promises to make possible. We have previously reported on the digital divide, or the varying levels of access to technologies such as internet and wireless services among different socioeconomic groups, as well as groups in different geographic areas. For example, as we have reported in the past, rural areas tend to have conditions—such as low population density or difficult terrain—that can increase the costs for carriers to deploy and maintain networks in those areas. Furthermore, lower-income households may be located in areas with access to the necessary infrastructure for certain services but may not be able to afford them. The challenge for some households to afford the most advanced mobile communications services would become worse if carriers charge more for 5G services. An Expert’s Perspective on the Digital Divide “Employers who want to ensure that their workforce has access to 5G, or their factory floor has access to 5G, won't locate in communities that don't have those services, thereby exacerbating the existing digital divide.” FCC has taken steps to address digital divide issues, with some of these efforts potentially affecting the digital divide as it relates to 5G deployment. For example, according to FCC officials, FCC issued a recent order approving a merger between two mobile carriers that included certain service requirements to increase 5G access nationwide. FCC told us that the merged company will face significant financial penalties if it fails to meet these requirements. Additionally, FCC has established financial support that may be used for 5G-related efforts. The Universal Service Fund provides financial support to carriers through different programs, each targeting a particular group of telecommunications carriers or consumers. For example, one of these programs, the High Cost Program, provides support in rural or remote areas where the customer base is relatively small and the cost of installing infrastructure is high. According to FCC officials, the support provided by the Universal Service Fund can evolve over time to address emerging technologies, including 5G. For example, the officials stated that in response to recent hurricanes in the Caribbean, the Universal Service Fund is currently being used to support deployment of fiber and power in parts of Puerto Rico and the U.S. Virgin Islands, which will help support future deployment of 5G in those areas. Further, in December 2019, the FCC Chairman announced his intention to establish the “5G Fund,” which would make up to $9 billion in Universal Service Fund support available to carriers to deploy 5G services in rural areas of the United States. In April 2020, FCC issued a Notice of Proposed Rulemaking and Order seeking comments on the framework for the 5G Fund and on approaches to identifying eligible areas for support. Although FCC’s actions could help address the digital divide, its existing planning documents for 5G do not include key elements that would allow FCC to understand the effects of these efforts as they relate to 5G deployment. Neither FCC’s strategic plan for 2018 through 2022 nor its 5G FAST Plan include specific performance goals—or related strategies and measures—that would allow FCC to assess the effectiveness of its efforts to close the digital divide associated with 5G deployment. For example, FCC’s strategic plan for 2018 through 2022 includes a strategic goal and performance goals to close the digital divide, but the performance goals are not specific or measurable. Further, neither the strategic plan nor the 5G FAST Plan include specific performance measures regarding the effects of 5G on the digital divide. Moreover, while FCC’s strategic plan states that a strategy to help close the digital divide is that it will set rules to encourage and facilitate the development of 5G networks, the strategy is not associated with specific performance goals or measures regarding the effects of 5G on the digital divide. Additionally, the 5G FAST Plan identifies a number of current and future strategies for FCC but does not include specific performance goals or measures that would allow it to understand what those strategies are intended to achieve and the effects those strategies are having on the digital divide as 5G networks are deployed. These omissions are contrary to leading practices of results-oriented organizations identified in previous GAO work. These leading practices call for performance goals and related strategies and measures, as we previously described. Such leading practices, as previously noted, include identifying: (1) specific and measurable performance goals to show progress toward broad strategic goals; (2) the activities (also known as strategies) the agency will take to make progress toward its goals; and (3) related performance measures to assess the results of the strategies and actual progress made toward the performance goals. FCC officials said that they are focusing on reducing the digital divide and have set high-level goals, but have not established goals specific to 5G. However, by establishing specific and measurable performance goals for 5G with related strategies and measures, FCC will have greater assurance that it has properly planned actions to effectively address the likely adverse effects on the digital divide as 5G networks are deployed. For example, specific goals for 5G will help FCC assess the effectiveness of its recent decision to make $9 billion in Universal Service Fund support available to carriers to deploy 5G services. Experts told us that deploying 5G infrastructure will be very costly for carriers. According to international standards established for 5G networks, to support all the new capabilities of 5G, carriers will need to replace their 4G core networks with new 5G equipment. These standalone 5G networks will provide new capabilities such as ultra- reliable low latency communications that could enable the development of new, more advanced use cases. In the meantime, carriers are currently deploying hybrid 5G, which uses existing 4G network infrastructure, but they still must make some upgrades to their 4G equipment and in some instances, deploy additional cell sites, such as small cells, to provide hybrid 5G service. These small cells can be installed on existing structures, such as buildings or streetlights. See figure 4 for examples of small cells. Some carriers are deploying 5G using low-band spectrum, which is much less costly to deploy because carriers can use their existing 4G cell towers. However, low-band spectrum does not enable the same data speeds as other types of 5G. As discussed previously, the United States has made a large amount of high-band spectrum available for 5G. The use of such spectrum increases the cost of 5G infrastructure deployment because it requires more small cell installations. Experts suggested that, because of the increased costs, carriers may limit deployment of high- band 5G network equipment to high-density areas such as sections of cities or stadiums. Moreover, a recent Defense Innovation Board report referenced a preliminary study that indicated that carriers would have to install approximately 13 million base stations, at a cost of approximately $400 billion, to deliver 5G service using this high-band spectrum to 72 percent of the population. In addition to installing the actual cell site equipment, each small cell site has costs associated with it: Fiber: Experts told us that fiber deployment is critical to the success of 5G. They noted that getting enough fiber in place to support the large increase in small cells will require a massive infrastructure deployment. For example, experts stated that currently there is not enough fiber in the ground in most places to support 5G. The fiber network must also have the capacity to handle the increased traffic from 5G. Experts told us that the new fiber needed for 5G will be costly to install, both in urban and rural areas. For example, installing fiber in urban areas can be costly due to local rules and difficulty accessing the right-of-way. In rural areas, fiber deployment costs are high because carriers must install fiber over longer distances to reach customers. Power: In addition to fiber, new cell sites also require a power source. While some small cells are being installed on light poles that have an existing power source, an expert noted that sometimes these are only powered on at night. Another expert noted that carriers may need to install back-up power sources, in case of a power outage. Permitting: When installing a new cell site, carriers generally must seek approval from the federal, state, or local government that controls the right-of-way or property where the cell site is to be located. This may require carriers to pay permitting fees or meet certain aesthetic requirements. Experts told us that the fact that different cities have different permitting regimes drives up the cost to build infrastructure. For example, experts told us that making sure small cells meet different localities’ requirements for design, dimension, and other aesthetic requirements is difficult for carriers and could slow deployment. However, other experts noted that these local permitting processes enable local governments to ensure 5G is being deployed in such a way that would benefit their citizens. For example, according to a report by the National League of Cities, San Jose, California created a tiered pricing structure to encourage carriers to cover more of the city. The city plans to use some of the revenues from this permitting process to help close the digital divide in the city, for example by allowing people to check out devices at libraries. Experts’ Perspectives on Permitting Costs “There are several thousand municipalities in the United States, each of which has different paperwork, processes, and payment mechanisms for siting small cells, which increases 5G deployment cost and time and inhibits the rollout of the networks.” “5G deployment requires many more small cells than 4G required macro cells, thereby increasing the reliance on public right-of-ways. As a result, we have seen cities restricted from charging market rates for permitting. However, cities play a really crucial role in ensuring that the deployment of these technologies and the use of public right-of-ways is coupled with public interest obligations.” Carriers’ current financial condition will also affect how they deploy 5G, including where and what type of networks they deploy. Experts noted that carriers in the United States may not currently have the capital required to fund large-scale deployments of 5G due, in part, to the costs of recent business decisions. Additionally, an expert mentioned that carriers would likely only be able to afford to deploy high-band spectrum in small sections of cities. Another expert predicted that carriers may not replace existing 4G equipment until it becomes less reliable over time, leading to a more comprehensive roll-out of 5G in 6 to 8 years. A representative from a carrier said that while it did not anticipate any additional revenue from 5G deployment, carriers still must deploy 5G because to not do so would place their companies at extreme risk of losing large numbers of customers, potentially eroding their revenue base. Experts’ Perspectives on Economic Challenges to 5G Deployment “Most of the major carriers in the United States aren't in great financial shape. They've leveraged up and have made acquisitions into new spaces. …The challenge for these carriers is do they have the billions of dollars required for another large-scale capital program, or do they just sort of put it out there because it’s good PR.” “Right now, from an economic standpoint, there isn't a strong case for making this enormous investment in 5G and it feels like it's more of a defensive play by carriers.” Experts also noted that consumers are not always willing to pay more for 5G service, which reduces carriers’ ability to recoup their deployment investments. For example, one expert questioned whether consumers in the United States were willing to pay anything more for 5G, and another noted that carriers are currently charging the same price for 5G and 4G service. Additionally, experts told us that there is no clear use case currently developed for 5G in the United States, besides enhanced mobile broadband. The 5G use cases often cited, such as remote surgery or autonomous vehicles, are unlikely to be developed in the near future. Without such use cases, they said, carriers lack a strong business case for deploying 5G. Other experts noted, however, that 4G use cases— such as social media or ride sharing apps—did not exist when carriers started to deploy 4G, but were developed after 4G was in place. Similarly, experts predicted that 5G use cases would be developed after 5G networks were available in the United States. To help reduce the cost of deploying 5G infrastructure, FCC has taken steps to expedite the permitting and review of small cells. For example, FCC issued a Programmatic Agreement for the Collocation of Wireless Antennas, which reduces the regulatory approval process for collocating small cells that are on existing infrastructure, such as utility poles. In addition, FCC adopted an order and declaratory ruling regarding state and local government reviews of small cell applications, which set parameters for fees and time frames for these reviews. This Order went into effect in January of 2019; however, it is currently being challenged in federal court. FCC also adopted an order that, among other things, exempted the construction of small cells from compliance with federal historic preservation and environmental review that were applied to large macro towers. A recent federal court decision, overturned the exemption and the FCC repealed the section of the order. In addition to the steps FCC has taken to limit regulatory and permitting costs, experts suggested that carriers could consider sharing their network infrastructure to reduce their capital expenditure for deploying 5G. Through infrastructure sharing agreements, two or more carriers share infrastructure such as radio antennas or fiber to deliver service to users. This sharing reduces deployment costs for carriers and allows them to deploy in areas where the costs would normally be prohibitive, such as rural areas. Such sharing agreements can increase choices for consumers, as more carriers can afford to operate in areas they would not normally be able to. FCC officials said that industry is already moving toward greater shared infrastructure and FCC’s efforts are designed to promote it. However, such sharing agreements may have the potential to decrease competition, if not well monitored. In addition, a carrier may not be willing to share infrastructure with other carriers for fear of losing its competitive advantage. For example, according to a report by the Body of European Regulators, a carrier that is the only one offering service in a certain area could lose competitive advantage and not be rewarded for its investments in the area under a sharing agreement. Infrastructure sharing is common in other countries; the same report by the Body of European Regulators found that carriers in 14 European countries had active sharing agreements with joint deployment in place. For example, in Spain, some carriers share their mobile networks in areas with fewer than 175,000 inhabitants. According to the report, 5G will further incentivize network sharing, as carriers need to deploy more small cells and fiber. Experts told us, however, that such sharing agreements were uncommon in the United States. Instead, carriers typically install their own network infrastructure, leading to overlapping networks and higher overall deployment costs. FCC officials said they recognize the benefits of infrastructure sharing, especially for 5G, but said that the decision about whether to share infrastructure is ultimately up to each carrier. Officials noted that carriers will use their own economic and engineering analysis in determining how to deploy 5G. 5G networks could create significant economic benefits for the United States, as companies develop products and technologies to access 5G’s new capabilities. Carriers currently face challenges to deploying 5G, however, which could delay or even limit the United States’ opportunity to realize those benefits. FCC has taken a number of actions regarding 5G deployment, but it has not clearly developed specific and measurable performance goals and related measures–with the involvement of relevant stakeholders, including NTIA–to manage the spectrum demands associated with 5G deployment. This makes FCC unable to demonstrate whether the progress being made in freeing up spectrum is achieving any specific goals, particularly as it relates to congested mid-band spectrum. Additionally, without having established specific and measurable performance goals with related strategies and measures for mitigating 5G’s potential effects on the digital divide, FCC will not be able to assess the extent to which its actions are addressing the digital divide or what actions would best help all Americans obtain access to wireless networks. We are making the following two recommendations to FCC: The Chairman of FCC should develop, in coordination with NTIA and other relevant stakeholders, specific and measurable performance goals—with related strategies and measures—to manage spectrum demands associated with 5G deployment. (Recommendation 1) The Chairman of FCC should develop specific and measurable performance goals—with related strategies and measures—to determine the effects 5G deployment and any mitigating actions may have on the digital divide. (Recommendation 2) We provided a draft of this report to FCC and NTIA for review and comment. FCC provided written comments, which we have reprinted in appendix II. FCC and NTIA also provided technical comments, which we incorporated as appropriate throughout our report. In its written comments, FCC neither agreed nor disagreed with our recommendations. FCC described the challenges associated with developing performance goals for managing the spectrum demands associated with 5G deployment. Specifically, FCC stated that such goals could limit the options available to manage spectrum demands. Instead, FCC stated that it adopts specific and measurable performance goals— with related strategies and measures—during ongoing rulemakings, which allow FCC to establish engineering, economic, or other technical outcomes. We acknowledge in our report that setting specific and measurable performance goals, strategies, and measures can be challenging, but continue to believe such strategic planning would benefit FCC’s spectrum management efforts. We did not identify what specific and measurable performance goals, strategies, and measures FCC should develop because FCC is in the best position to make such determinations. However, as we describe in our report, FCC still has not engaged in this strategic planning effort. Our past work has found that there is no more important element in results-oriented management than an agency’s strategic planning effort. That effort should be the starting point and foundation for FCC to define what it seeks to accomplish, identify the strategies it will use to achieve desired results, and then determine how well it succeeds in reaching results-oriented goals and achieving objectives. Related to our recommendation for FCC to develop specific and measurable performance goals to determine the effects 5G deployment and any mitigating actions may have on the digital divide, FCC noted that it is taking regulatory actions and providing funds designed to reduce the digital divide. FCC further said that it remains committed to promoting robust 5G deployment nationwide and, consistent with our recommendation, will continue to explore new ways to evaluate how it may impact efforts to close the digital divide. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 15 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Chairman of the FCC, the Secretary of the Department of Commerce, and other interested parties. In addition, the report will be available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or Vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines the challenges, and the federal government’s efforts, related to 5G deployment with regard to: (1) managing spectrum; (2) closing the digital divide; and (3) addressing economic issues. These were the top three challenges to 5G deployment identified by the 17 experts we convened. To identify challenges to 5G deployment, we issued a brief questionnaire to 146 GAO-identified stakeholders with knowledge of 5G networks. These stakeholders included officials from the federal government, as well as representatives from academia, industry, and consumer groups. Stakeholders were identified by reviewing previous GAO reports and through background research and were selected to provide a range of perspectives on 5G deployment. We asked these stakeholders to identify challenges to deploying 5G networks in the U.S. and received 23 responses. We conducted a content analysis to categorize the responses into a final set of 13 challenges. See table 1 for a list of the challenges. Because content analysis relies on the judgment of coders to determine whether qualitative data reflects particular categories, we took several steps to ensure that this judgment remained objective, accurate, and consistent. These steps included using independent coders from two different mission teams within GAO to ensure consistent judgment of categories. The independent coders were in general agreement on the challenges categories. On the basis of this high level of agreement between coders, as well as a review by a third independent analyst, we are confident that our content analysis represents an objective, accurate, and consistent assignment of these coding categories. We then convened a meeting of 17 experts to discuss the above challenges to 5G deployment. Our meeting of experts was held at the National Academies of Sciences, Engineering, and Medicine (NASEM) in October 2019 over one-and-a-half days. Staff from NASEM assisted us in identifying experts for the meeting. To identify the experts appropriate for this meeting, NASEM relied on staff experience and professional judgment drawn from its Computer Science and Telecommunications Board. We selected the final panel of experts in consultation with NASEM staff with the goal of ensuring that a broad range of views was represented from multiple 5G-related areas, such as those of wireless carriers, academia, and consumer and industry groups. See table 2 for a list of the experts that participated in the meeting. The meeting was moderated by GAO staff who guided the experts through questions about each challenge to 5G deployment. The experts also discussed potential actions the federal government could take to address those challenges and were asked to identify the most significant challenges to 5G deployment. This meeting of experts was planned and convened with the assistance of NASEM to better ensure that a breadth of expertise was brought to bear in its preparation; however, all final decisions regarding meeting substance and expert participation are the responsibility of GAO. Any conclusions and recommendations in GAO reports are solely those of the GAO. The meeting was recorded and transcribed to ensure that we accurately captured the experts’ statements, and we reviewed and analyzed the transcripts as a source of evidence. We edited experts’ quotations from the transcripts for clarity and conciseness to include in this report. In addition to the experts we spoke to on the panel, we also interviewed 16 stakeholders—as well as officials from the Federal Communications Commission (FCC), the National Telecommunications and Information Administration (NTIA), and the National Institute of Standards and Technology—to further understand the challenges to 5G deployment. We selected these stakeholders based on our prior telecommunications work, other 5G literature, and recommendations from stakeholders we interviewed to provide a range of perspectives on 5G deployment. Stakeholders were from two universities, four industry associations, five wireless carriers, as well as five local governments and organizations. To identify these local governments, we selected a group of cities to include those where wireless carriers have announced they will launch 5G services, and selected a mix of cities where there was local opposition to 5G, as well as cities with state or local laws regarding small cell permitting. We then selected lower-population density cities in the same states as those cities, using U.S. Census data. To select these cities, we identified the county with the median population density of the state, and then selected the city which holds the county seat. We attempted to contact all the selected cities and were able to schedule and hold interviews with representatives from Los Angeles, California; Jacksonville, Florida; Greenville, Illinois; and Naples, Florida. The information we obtained from these interviews is not meant to be generalizable to other cities’ experiences, but is meant to provide illustrative examples of actual 5G deployment. See table 3 for a complete list of stakeholders we interviewed. Finally, to assess the federal government’s actions to address challenges to 5G deployment, we reviewed relevant statutes and literature, along with reports and documents from FCC and the Department of Commerce. For example, we reviewed FCC reports and orders related to 5G networks, the Department of Commerce’s Spectrum Repurposing Report, along with planning reports such as FCC’s Facilitate America’s Superiority in 5G Technology Plan (5G FAST Plan) and FCC’s and the Department of Commerce’s strategic plans. In addition, we interviewed FCC and NTIA officials about their efforts to address 5G deployment challenges. We compared FCC efforts to address 5G deployment challenges to its own strategic goals and relevant leading practices for performance management identified in our prior body of work. We conducted this performance audit from February 2019 to June 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Andrew Von Ah, (202) 512-2834 or vonaha@gao.gov. In addition to the contact named above, Keith Cunningham (Assistant Director); Daniel Paepke (Analyst in Charge); Oluwaseun Ajayi; Carol Bray; Vijay D’Souza; Wayne Emilien; Jonathan Felbinger; Richard Hung; Catrin Jones; Michael Krafve; Kaelin Kuhn; Hannah Laufe; Dan Luo; Neelaxi Lakhmani; Brian Mazanec; Jamilah Moon; Cheryl Peterson; Erika Prochaska; Malika Rice; Oliver Richard; Pamela Snedden; Andrew Stavisky; Hai Tran; Christopher Turner; Tatyana Walker; and Michelle Weathers made key contributions to this report.", "summary": "As the latest generation of mobile communications, 5G networks are expected to provide faster connections to support consumer, industry, and public sector services. While private sector carriers deploy 5G networks, FCC has a role in managing deployment challenges, such as how to allocate low-, mid-, and high-band spectrum for 5G use. GAO was asked to review 5G deployment challenges. This report examines challenges and the federal government's efforts related to 5G deployment with regard to managing spectrum for 5G and closing the digital divide, among other things. GAO, with assistance from the National Academies of Sciences, Engineering, and Medicine, convened a meeting of 17 experts from academia, industry, and consumer groups; reviewed relevant statutes, literature, and FCC documentation; and interviewed FCC and other relevant federal officials, along with stakeholders that include various localities, wireless carriers, and industry associations. Approximately every 10 years since the early 1980s, wireless carriers have deployed a new generation of wireless communication technology. This decade is no different, as carriers are now developing and deploying 5G networks, which offer greater speed and higher data capacity than previous generations of mobile wireless networks. Carriers in the United States are currently deploying “hybrid” 5G, which uses 5G technologies in combination with existing 4G networks to improve the networks' speed. In the future, carriers may deploy “standalone” 5G, which relies exclusively on 5G equipment to allow for additional enhanced capabilities (see fig. 1). Radio frequency spectrum is a finite natural resource used to provide a variety of communication services to businesses and consumers, as well as to federal, state, and local governments. The frequency bands—often referred to as low-band, mid-band, and high-band spectrum—have different characteristics that make them more or less suitable for specific purposes. Experts GAO convened said that mid-band spectrum is highly congested, leading to an insufficient amount available for carriers to deploy their 5G networks in the United States. The experts stated that to avoid delays in 5G deployment, the commercial sector needs access to more mid-band spectrum. These experts highlighted the need for mid-band spectrum for 5G due to mid-band's use internationally and because of its properties. Mid-band spectrum allows for higher data capacity than lower bands and can penetrate physical obstacles over long distances—a property known as “propagation”— better than higher bands (see fig. 2). The Federal Communications Commission (FCC) has some efforts under way to make additional mid-band spectrum available but so far has primarily made high-band spectrum available for 5G because it is more readily available. Making more mid-band spectrum available to the commercial sector will be challenging, as current mid-band spectrum users include federal government users that may not be able to readily transition to new or less favorable spectrum bands. FCC's planning document for 5G includes a section on making additional spectrum available but does not clearly identify specific and measurable performance goals or measures to manage the spectrum demands for 5G. Without such strategic planning efforts, FCC will be unable to determine the effectiveness of its spectrum management efforts, particularly related to the congested mid-band spectrum that is critical to 5G deployment. The experts GAO convened also stated that 5G deployment would likely exacerbate disparities in access to telecommunications services, known as the “digital divide.” Specifically, experts as well as stakeholders GAO interviewed said that 5G using high-band spectrum—which allows for high data capacity—is likely to be first deployed in areas already equipped with much of the necessary infrastructure. Experts said the areas with existing infrastructure are generally urban, densely populated, high-income areas as opposed to rural or low-income areas. Further, within urban settings, experts said that high-band 5G networks are more likely to be deployed in commercially viable areas, including those parts of a city that already are equipped with fiber and power and, presumably, already benefit from the most advanced mobile broadband services available. FCC has taken steps to address the digital divide, including a recent announcement to make up to $9 billion in funding available to carriers to deploy 5G in rural areas of the United States. However, FCC has not developed specific and measurable performance goals with related strategies and measures to assess how well its actions are mitigating the added effects 5G deployment will have on the digital divide. FCC should develop specific and measurable performance goals with related strategies and measures to: (1) manage spectrum demands for 5G and (2) determine the effects 5G deployment and any mitigating actions may have on the digital divide. FCC indicated that setting spectrum goals could unnecessarily limit its options but did not agree or disagree with GAO's recommendations. GAO continues to believe that well-considered strategic planning would benefit FCC's efforts.", "document_type": "gao"}
{"report": "From 1944 through 1988, the production of plutonium at Hanford generated about 525 million gallons of radioactive and hazardous waste. Some of the waste was dumped directly into the soil, some was encased in drums or other containers and buried, and about 54 million gallons were stored on-site in 177 underground tanks. Some of the waste stored in the underground tanks is “high-level waste” (HLW) mixed with hazardous chemicals that is to be vitrified—a process in which the waste is immobilized in glass—prior to disposal. “Low-activity waste” (LAW) is EM’s term for the portion of the tank waste with low levels of radioactivity. EM estimates that LAW comprises more than 90 percent of the volume in the tanks but contains less than 10 percent of the radioactivity. EM currently plans to treat much of Hanford’s tank waste in the WTP. The WTP is the most technically complex and largest construction project within EM. As figure 2 shows, the WTP consists of facilities that are designed to separate waste into low-activity and high-level waste streams. Once completed, the WTP is to treat the HLW and a portion of the LAW in separate facilities using vitrification. The WTP consists of the following facilities: Pretreatment Facility. This facility is to receive the waste from the tanks and separate it into HLW and LAW. Under the original WTP design, all waste must first pass through this facility before it can be treated. Tank waste to be sent to the pretreatment facility for processing must meet specific physical and chemical characteristics, known as waste acceptance criteria, and the waste must be certified as having met these criteria before transfer from the tanks to the pretreatment facility. For example, WTP waste acceptance criteria may stipulate that waste meet certain requirements for chemical composition, particle size, and density in order to be handled by the pretreatment facility. Construction of this facility as originally designed is about 40 percent complete. LAW Vitrification Facility. This facility is designed to receive the LAW and immobilize it by vitrification. The canisters of vitrified waste will be permanently disposed of at another facility on the Hanford Site. Construction of this facility is nearing completion, and EM plans to complete commissioning of the facility no later than December 31, 2023. As currently designed, this facility would only have capacity to treat a third to half of the LAW currently in the waste tanks. EM is analyzing alternatives for treating the remaining LAW, known as supplemental LAW. HLW Vitrification Facility. This facility is designed to receive the HLW and immobilize it by vitrification. The canisters of vitrified waste will be stored on-site until a final repository is established. Construction of this facility is about 40 percent complete. Effluent Management Facility. The Effluent Management Facility is being built to evaporate much of the liquid waste produced during LAW processing and vitrification at the LAW Facility. Design work on this facility is nearly complete and construction is under way. EM plans to complete construction of this facility in December 2021. Analytical Laboratory. This facility will be used to analyze the waste at various stages of treatment, such as testing samples of the vitrified waste to ensure that it meets certain criteria and regulatory requirements for disposal. Construction of this facility is complete and EM has begun startup and commissioning activities. Balance of Facilities. These facilities consist of the 22 support facilities that make up the plant infrastructure, such as cooling water systems and silos that hold glass-forming materials. Construction of these facilities is nearing completion, and EM has begun startup and commissioning activities. The WTP has faced hundreds of technical challenges since the early years of the project. These challenges ranged from effectively mixing the waste prior to treatment to addressing potential erosion in the facility piping. We have reported on these challenges in the past and have made numerous related recommendations to EM. For example, in 2003 we found that BNI and outside experts had concerns about the technology for separating the waste—including problems associated with mixing the waste during separations and evaporating water from the waste—and they proposed more testing to resolve those challenges. We recommended that EM consider further testing to resolve those challenges before moving forward with construction of the pretreatment facility. In early 2007, EM decided to build a pilot-scale facility for the WTP to fully test pretreatment technologies before completing the full- scale design of the facility. Similarly, in 2006 we found that the WTP continued to face numerous technical challenges and that many of the technical challenges still had not been addressed even though EM was moving forward with construction on the pretreatment facility. We recommended that EM resolve the technical challenges before moving forward. EM agreed and took steps to ensure that the design of each WTP component was at least 90 percent complete before construction or installation. In December 2012, we found that the WTP continued to face significant technical challenges, even though construction was 55 percent complete, and we recommended that EM not resume construction of the pretreatment facility until the issues had been fully resolved. Because of these ongoing challenges, in December 2012, EM’s WTP Engineering Division issued a memorandum that recommended that all activities affecting design, construction, and installation of structures, systems, and components be stopped. According to the memorandum, stopping work would help ORP avoid future nuclear safety and quality compromises and substantial rework. Instead of stopping all work at the WTP, ORP management stopped work only on those facilities that faced the most significant technical challenges, namely, the pretreatment and HLW facilities. As we discuss in this report, EM has not yet resumed construction on the pretreatment and HLW facilities. In 2015, we reported that because of ongoing problems hampering the progress of the pretreatment facility at Hanford, EM was pursuing other pretreatment alternatives (such as feeding the waste from the tanks directly to the vitrification facilities) but had not properly defined the mission need for the analysis or developed a reliable life-cycle cost estimate for the alternatives being analyzed. We recommended that EM revise its analysis to consider a variety of alternatives without limiting potential solutions and that EM further limit construction activities on the pretreatment facility until aggressive risk mitigation strategies are developed and employed to address the technical challenges. EM opted to change the alternative pretreatment approach it had been pursuing and in 2018 began design work on a different alternative pretreatment approach. In April 2018, we reported that seven of nine ORP quality assurance experts expected rework would be needed for existing facilities, including the pretreatment facility. In that report we noted that according to three experts with knowledge about maintenance programs, BNI had not established a fully effective WTP quality assurance program, particularly for the pretreatment facility and HLW facility, and as a result, structures, systems, and components at these facilities have deteriorated and been damaged. We recommended that EM (1) determine the full extent to which problems exist in all WTP structures, systems, and components, (2) stop work in areas where quality assurance problems are recurring until ORP’s Quality Assurance Division can verify that the problems are corrected and will not recur, and (3) revise ORP’s organizational structure so that the quality assurance function is independent of ORP upper management. As of March 2020, EM had implemented one of our three recommendations (revising ORP’s organizational structure), but had not yet fully implemented the other recommendations. Cleanup of the Hanford Site is governed by two main documents. The 1989 Hanford Federal Facility Agreement and Consent Order—or Tri- Party Agreement (TPA)—is an agreement among DOE, Ecology, and the Environmental Protection Agency. The TPA lays out a series of legally enforceable milestones for completing major activities in Hanford’s waste treatment and cleanup process. The 2010 Consent Decree, as amended, resolves certain disputes between Ecology and DOE and addresses a subset of cleanup activities, including completing the construction and achieving initial operations of the WTP and retrieving waste from specified single-shell tanks. Among other things, the consent decree requires DOE to do the following: Begin treating LAW by 2023; Substantially complete the construction of the pretreatment facility by Start WTP operations by 2036. The TPA requires DOE to complete the treatment and vitrification of all HLW and LAW in the Hanford tanks by 2047. In addition to oversight by Ecology and the Environmental Protection Agency, DNFSB is responsible for, among other things, reviewing the design of new defense nuclear facilities at DOE’s sites, including the WTP. DNFSB, established in 1988, provides independent analysis, advice, and recommendations to the Secretary of Energy—in the Secretary’s role as operator and regulator of DOE’s defense nuclear facilities—to ensure adequate protection of public health and safety at these facilities. DNFSB is not authorized to issue regulations binding on DOE apart from establishing reporting requirements. Instead, DNFSB uses both informal interactions and formal communications with DOE to help ensure that its concerns are addressed. DOE Order 413.3B establishes program and project management requirements for the acquisition of capital assets with the purpose of delivering projects within budget, on time, and capable of meeting mission performance. EM is required to manage its cleanup projects in accordance with this order. In particular, Order 413.3B requires EM to conduct an AOA that is consistent with the 22 AOA best practices we identified. DOE also has an AOA guide, which describes suggested approaches for DOE and its contractors to be consistent with the 22 best practices for an AOA process. The 22 best practices compile common AOA policies and guidance used by different government and private- sector entities and incorporate experts’ comments. These best practices include the following: Define mission need, Develop AOA time frame, Establish AOA team, Define selection criteria, Weight selection criteria, Include baseline (or status quo) alternative, and Develop a life-cycle cost estimate for each viable alternative. From early fiscal year 2013 until the end of fiscal year 2018, EM spent about $752 million to maintain the pretreatment facility and resolve technical challenges. Over half of the $752 million went toward overhead, oversight, and other costs to maintain the partially constructed facility. The remaining costs went toward resolving technical challenges. Design and construction of the pretreatment facility is on hold, and DOE’s budget request for fiscal year 2020 states that EM plans to continue “limited activities” on the pretreatment facility to keep the facility in a preservation and maintenance mode. However, officials told us that EM does not have a cost estimate for completing the pretreatment facility, and EM has no plans to develop such an estimate in the near future. From early fiscal year 2013—when work involving design and construction of the pretreatment facility was suspended—until the end of fiscal year 2018, EM spent about $752 million on the pretreatment facility. Among other things, EM used this funding for resolution of the technical challenges that led to the suspension of the facility’s construction, overhead and project management, equipment purchase and management, facility maintenance, BNI award and contract modification fees, and EM oversight. (See fig. 3.) Less than half of the $752 million spent on the pretreatment facility in fiscal years 2013 through 2018 went toward resolving technical challenges associated with the facility. According to EVM system reports, EM spent approximately $323 million—or 43 percent of the $752 million in total costs—on costs incurred by BNI to resolve the technical challenges. This includes activities such as identifying research tasks needed to resolve the technical challenges and performing testing, as well as the cost of subcontracts to assist BNI in resolving the technical challenges. Pretreatment Facility Lifetime Overhead Costs In fiscal year 2019, Bechtel National, Inc. (BNI), the prime contractor for Hanford’s Waste Treatment and Immobilization Plant (WTP)— including the pretreatment facility—allocated $1.5 billion in overhead costs to the pretreatment facility in its Earned Value Management system (for fiscal years 2001 through 2014) that had previously been recorded in non-facility specific accounts. What we refer to as overhead BNI refers to as project services allocation or shared services, which according to officials at the Office of River Protection includes both traditional overhead costs (such as light and power), as well as the cost of common activities for multiple facilities and the management system used for those facilities. Prior to fiscal year 2015, overhead costs for the entire WTP were recorded in non-facility specific accounts. In fiscal year 2015, BNI changed the way that it accounts for these costs by allocating overhead costs to each individual facility; however, at the time, this change was only made for future overhead costs for the entire WTP. In June 2019, BNI also applied this change to pre-2015 costs, which brings BNI’s total pretreatment facility costs, from the beginning of the contract in December 2000 through July 2019, to $3.4 billion—$1.5 billion of which are overhead costs. However, this allocation of cost to each facility from the project level shared services accounts did not change the overall cost of the WTP project. because of design changes. According to ORP officials, EM did not pay a termination fee for procurements that were terminated because of the vendor going out of business; however, for other terminated procurements, EM might have to pay additional costs if the vendor submits a claim for compensation to BNI, for which BNI in turn seeks reimbursement from EM. In either case, there may be additional costs related, for example, to picking up and transporting items. Facility Maintenance. About $18.8 million—or 2 percent of the total $752 million—went towards the costs of general facility maintenance. According to ORP officials, facility maintenance includes activities such as maintaining building access controls, maintaining installed components, cleaning up waste from birds, removing snow and trash, and conducting periodic walks of the facility to determine the condition of materials in the building, among other things. Award fees are an amount of money added to a contract, which a contractor may earn in whole or in part by meeting or exceeding subjective criteria stated in an award fee plan typically related to areas within quality, technical ingenuity, cost-effective management, program management, and other unquantifiable areas. Award fees in the context of this report refer to money earned by BNI based on its performance in carrying out work on the pretreatment facility. In this report, contract modification fees refer to money negotiated between BNI and EM based on a change in the contract agreed to by both parties. $153 million—the facility’s highest costs from fiscal year 2013 through fiscal year 2018. (See fig. 4.) Contributing to the fiscal year 2017 costs was a one-time $60 million contract modification fee for both the pretreatment facility and the high-level waste facility that was negotiated between BNI and EM. According to EM officials, EM and BNI negotiated this fee for work completed by BNI in previous years for which it had not been paid a fee. This work included developing facility designs, resolving technical issues, and conducting reviews and research studies. Design and construction of the pretreatment facility have been on hold since 2012. At the time construction was halted, BNI estimated that construction of the facility was about 40 percent complete. In July 2018, the U.S. Army Corps of Engineers reported that construction of the facility was still about 40 percent complete. In a tour of the facility in May 2019, we observed that construction remains on hold and that EM is instead using the space inside the partially constructed building to conduct worker training exercises. Additionally, DOE’s budget request for fiscal year 2020 states that EM plans to continue “limited activities”—such as maintaining the existing facility, storing uninstalled equipment, and maintaining records for quality assurance—on the pretreatment facility to keep the facility in a preservation and maintenance mode. ORP officials told us in September 2019 that EM does not plan to restart design and construction activities on the pretreatment facility until alternatives for pretreating HLW have been analyzed. According to EM officials, EM does not have an updated cost estimate for completing the pretreatment facility, as required under DOE Order 413.3B. This order requires EM to develop, maintain, and document cost estimates in a manner consistent with methods and best practices identified in GAO’s Cost Estimating and Assessment Guide, as well as other documents, including the Office of Management and Budget’s Circular A-11, prior to DOE approving a performance baseline change. EM’s last independently verified approved cost estimate for completing the entire WTP was completed in 2006. At that time, EM estimated that completing the pretreatment facility would cost approximately $2.5 billion. However, the pretreatment facility has surpassed that amount. Specifically, through fiscal year 2018, EM spent about $3.8 billion on the facility, including approximately $3 billion spent prior to halting construction in 2012 and $752 million spent in fiscal years 2013 through 2018. EM was in the process of updating the cost estimate in 2012 when construction of the pretreatment facility was suspended, and therefore EM’s update to the cost estimate was suspended as well. ORP officials told us that they do not have plans to complete a cost estimate for the pretreatment facility. According to these officials, they cannot complete a cost estimate for the pretreatment facility until EM has made a decision about the future of the facility and, if necessary, BNI develops design changes to address technical challenges. The officials explained that the development of design changes depends on the prioritization of funding. They also explained that ORP’s highest funding priority is to begin vitrifying some LAW as soon as possible by bypassing the pretreatment facility using alternative technologies and sending the separated LAW directly to the WTP’s LAW vitrification facility—an approach known as Direct-Feed Low-Activity Waste (DFLAW). Officials told us that ORP’s second highest funding priority is the completion of the HLW facility and that the pretreatment facility will not be a priority until EM has made a decision on which pretreatment methods to use going forward and updated the design changes for the facility as needed. After EM halted construction on the pretreatment facility in 2012, EM began working with BNI to address the longstanding technical challenges associated with the design and construction of the pretreatment facility. According to July 2019 correspondence between EM and BNI, both parties consider these technical challenges to be resolved, and according to ORP officials, pretreatment facility engineering and design followed by its construction may now continue. However, based on our interviews with EM and BNI officials, EM has not yet designed or engineered the solutions. In addition, according to DNFSB officials, the DNFSB does not consider the technical challenges to be resolved yet, though it continues to review EM’s efforts. In late 2012, EM halted construction of the pretreatment facility, and EM and BNI began work to resolve technical issues. In November 2012, EM formed a design completion team responsible for resolving the technical challenges. In May 2014, EM asked BNI to submit a plan for resolving the challenges and resuming construction of the pretreatment facility. EM ultimately identified eight key categories of technical challenges to be resolved before resuming construction of the pretreatment facility (see table 1 for a list of the eight categories, and see app. III for a more detailed description of each category of technical challenges). The majority of these categories involved portions of the pretreatment facility intended to manage the HLW. For example, one category EM identified involves preventing hydrogen from building up in the facilities’ piping and vessels, which could cause an explosion. Another category involves preventing corrosive waste from eroding treatment equipment, which could cause a leak of radioactive materials. In June 2014, BNI formed eight teams to address each category of technical challenges. For example, to address the technical challenges associated with mixing the waste in the pretreatment facility using a technology known as pulse-jet mixing, the design completion team developed a plan to standardize and test a new design to address pulse- jet mixing challenges. Similarly, to address concerns about the potential weaknesses in equipment and piping located in rooms inaccessible to humans once operations begin (known as black cells), EM formed a black cell analysis team. BNI submitted interim updates to EM on the proposed resolution of specific challenges as BNI addressed them. For example, in December 2017, BNI informed EM of its resolution of the challenges related to facility ventilation. Similarly, in September 2018, BNI informed EM of its resolution of the challenges related to the black cells. BNI sent similar correspondence on the other six categories of technical challenges to EM throughout 2019. According to EM officials, EM and its contractors provided DNFSB documentation and briefings on the resolution of the technical challenges. In June 2019, BNI informed EM that it considered all eight categories of technical challenges to be resolved. In July 2019, EM subsequently informed BNI that it agreed with BNI’s conclusions that the technical challenges were resolved. According to ORP officials, “resolved” means that all the required studies, calculations, and testing have been completed and demonstrated to independent experts and EM that (1) the issue is fully understood so that no further research is needed and (2) a solution is ready for detailed design. Although EM and BNI consider the technical challenges associated with the pretreatment facility to be resolved, EM and BNI have not yet designed or engineered the solutions. BNI acknowledged early in the process that resolution of the technical challenges would involve not only a conceptual solution, but also subsequent design, engineering, and, in some cases, testing of the solutions before construction could resume on the pretreatment facility. For example, in its June 2014 plan for addressing the challenges, BNI noted that prior to making a decision to proceed with construction of the facility, it would need to conduct a number of additional steps, including updating the designs of the pretreatment facility and assessing the nuclear safety basis and the contract implications for the updated designs. In addition, ORP officials told us that proposed revisions to the pretreatment facility would require negotiation with Ecology. As of February 2020, EM and BNI had not yet begun developing these required designs and engineering changes and have no plans to do so until a decision is made on the future of the facility. According to EM officials, ORP’s current priorities are to begin DFLAW operations and to conduct an analysis of alternatives related to the treatment of the HLW. These next steps could involve significant work and potential rework to the facility. According to EM officials, resolving the technical challenges likely will require BNI to change its designs for the pretreatment facility and conduct significant rework in portions of the facility that have been completed. ORP officials said that they expect this design work to be significant and do not expect it to be complete enough to proceed with the construction of the facility until at least 2022, depending on the availability of funding to support the design work. BNI’s plan going forward includes a number of steps related to updating the pretreatment facility designs. As a result of this significant engineering work still ahead, as we reported in May 2015, EM likely will have to conduct rework of the existing facility (which is 40 percent built), leading to further cost increases and schedule delays. For example, BNI will need to redesign any existing components and systems that have become obsolete since EM halted construction or that need to be reworked to accommodate the technical solutions. In addition, DNFSB officials have begun reviewing EM’s proposed solutions, but they said that they do not consider the technical challenges to be resolved. Although EM does not require DNFSB approval to restart construction of the pretreatment facility, ORP officials said that they consider the next step in the process to be DNFSB review of their solutions. DNFSB officials, on the other hand, said that the process used to review issues is as follows: (1) DNFSB raises a concern; (2) EM comes up with a conceptual solution, presents it to DNFSB, and receives feedback; and (3) EM then comes up with a design solution, presents it to the DNFSB, and receives feedback. According to DNFSB officials, because they have not been able to review the updated engineering and design plans, they are not in a position to approve the proposed solution. Since 2012, DNFSB has been reviewing EM’s proposed technical solutions as part of its role to provide independent advice and recommendations to DOE regarding the protection of public health and safety at DOE facilities. As of December 2019, DNFSB had officially commented on one of EM’s proposed solutions—related to technical challenges surrounding the pulse-jet mixers—and noted simply that EM’s and BNI’s work “strengthens the technical foundation” for using the mixers and that DNFSB would “continue to follow the design process.” With regard to the remaining challenges, DNFSB officials said that for some, additional deficiencies needed to be addressed. For others, DNFSB officials said they were reviewing the details of EM’s proposed solution or needed additional information from EM. For two of the categories of technical challenges, DNFSB officials said they considered them to be operational rather than safety issues and therefore DNFSB would not review EM’s proposed solutions. (See table 2.) To begin treating LAW by 2023 as required, EM began pursuing pretreatment alternatives in 2013 and has spent about $428 million on developing these alternatives for LAW pretreatment capabilities that were originally planned for the pretreatment facility. We reported in May 2015 that in analyzing alternative LAW pretreatment approaches, EM did not meet two key steps outlined in best practices and DOE internal guidance—define mission need and develop a life-cycle cost estimate for its alternatives. We recommended that EM revise its mission need and its cost estimates for the alternatives being reviewed. In April 2019, EM began analyzing alternatives for treating HLW, and EM officials stated that this analysis of HLW treatment alternatives would follow best practices. However, as of February 2020, EM did not yet have a well- defined mission need statement for its HLW treatment AOA, nor did it have life-cycle cost estimates related to the pretreatment facility, as called for by best practices. In addition, Ecology, a key regulatory stakeholder for the Hanford cleanup, has raised concerns about the AOA as well as EM’s engagement with regulators during this process. In 2013, to meet its deadline to begin treating LAW by 2023, EM began work on a strategy to bypass the pretreatment facility and instead separate out some of the LAW to remove most of the radioactivity from the tank waste. This approach, called DFLAW, has involved several different activities since 2013 such as constructing separate facilities and infrastructure to accomplish this work, as well as modifying existing facilities: Direct-Feed Low-Activity Waste Modifications and Effluent Management Facility. EM has spent $272 million on modifications to the WTP to support the DFLAW approach, including designing and constructing the Effluent Management Facility. The Effluent Management Facility is intended to manage the high volume of contaminated liquid generated through the processing of LAW. This capability was originally designed to be located in the pretreatment facility. Low-Activity Waste Pretreatment System. In fiscal years 2014 through 2018, EM spent approximately $146 million on the Low- Activity Waste Pretreatment System. The Low-Activity Waste Pretreatment System included designing a permanent facility to receive and treat liquid waste, separating out the less radioactive portion from the underground tanks in preparation for direct feed to the WTP’s LAW facility. This function was originally intended to be accomplished by the pretreatment facility. In November 2017, ORP ordered work on this permanent facility to be suspended because, according to EM officials, the cost estimates for completing it had become too high and the urgency of meeting the pending treatment deadline too great. Tank Side Cesium Removal System (TSCR). EM spent about $6 million for work on a demonstration of the TSCR technology in fiscal year 2018 after suspending the Low-Activity Waste Pretreatment System. TSCR will be built next to an underground double-shelled waste tank and will filter waste directly from the tank to remove solids and cesium. The resulting waste will be pumped to a different underground tank for storage until it can be sent to the LAW facility for vitrification. This would enable the rest of the waste to be fed directly to the WTP’s LAW facility. ORP plans this demonstration project to be complete as early as 2021 and then, depending on the results, ORP could decide to build additional TSCR units near other tank farms to treat more of the tank waste. In addition to DFLAW, EM briefly pursued a smaller-scale pretreatment approach—known as the Test Bed Initiative—in which low-level waste was drawn directly out of the underground tanks (using existing processes and commercial facilities), grouted on site, and shipped to a disposal facility in Texas. EM spent about $4.8 million in fiscal years 2016 through 2018 to design the technology and treat 3 gallons of waste from the underground tanks. EM suspended the Test Bed Initiative in June 2019. In total, EM has spent about $428 million developing these alternative pretreatment approaches for LAW, in addition to the $752 million spent on the pretreatment facility since 2012. (See fig. 5.) EM began exploring alternative LAW pretreatment approaches as early as 2006 in connection with its analysis of options for treating the supplemental LAW at Hanford. In September 2013, in an effort to make progress while working to resolve technical challenges on the pretreatment facility, EM announced plans to pursue these alternative LAW pretreatment approaches and received funding to do so. However, as we reported in May 2015, EM did not properly define the mission need for the analysis or develop a reliable life-cycle cost estimate for the alternatives it analyzed prior to selecting its preferred alternative: First, in May 2015, we found that EM had developed a narrow statement of mission need that effectively excluded other potential alternatives from being considered. This, we noted, was contrary to DOE requirements in DOE Order 413.3B and our best practices for an AOA process, which specify that statements of mission need should not identify a particular solution such as equipment, facility, or technology, to allow the analysis the flexibility to explore a variety of alternatives without limiting potential solutions. We noted that by narrowly defining the mission need in this way, EM effectively narrowed the range of acceptable options and excluded from consideration other alternatives to expediting waste treatment and addressing the potential danger posed by the leakage of waste from the tanks. Second, we noted in May 2015 that in choosing its current approach to treating LAW, EM did not develop a life-cycle cost estimate for its Low-Activity Waste Pretreatment System approach and did not develop life-cycle cost estimates for all of the alternatives before choosing its course of action. Our AOA best practices and DOE’s AOA Guide call for developing a life-cycle cost estimate for each alternative, including all costs from inception of the project through design, development, deployment, operation, maintenance, and disposal. We recommended in our May 2015 report that EM revise its mission need statement and life-cycle cost estimate for the Low-Activity Waste Pretreatment System. EM opted to change this alternative pretreatment approach and in 2018 began designing and building TSCR, as noted above. EM did not undertake an AOA process before making that decision; instead, EM chose to pursue TSCR, a technology similar to one being used at the Savannah River Site in South Carolina. EM officials said the decision to move forward on these LAW pretreatment alternatives without an AOA process was based on the urgency of the upcoming requirement to begin treating LAW by 2023. We continue to believe that as EM pursues additional treatment alternatives, EM should properly define the mission need for the analysis and develop a reliable life-cycle cost estimate for the alternatives it is analyzing. In April 2019, EM initiated an AOA for treating the HLW in the tanks at Hanford and plans to conclude the review and report its findings in September 2020. According to the review team’s September 2019 study plan, the review is to analyze 15 alternatives, including completing the pretreatment facility as planned, repurposing the pretreatment facility, and changing the current approach to pursue other pretreatment options. Some of the other options the review team plans to explore include sending HLW directly from the underground tanks to the HLW facility for treatment, building alternate HLW pretreatment facilities, and shipping the HLW to the Savannah River Site in South Carolina for treatment. (See appendix IV for a list of the alternatives being analyzed.) EM officials said that in undertaking this AOA for HLW treatment alternatives, they plan to meet best practices for an AOA process and those in DOE’s AOA Guide. They noted that, consistent with these AOA best practices, EM has developed a time frame to complete the review, established a review team, and defined and weighed selection criteria against which to compare the alternatives. However, based on our review, as of February 2020, EM had not yet met two key steps—defining mission need and developing a life-cycle cost estimate for the baseline alternative—that are among the best practices we identified for an AOA process. First, EM has not yet defined the mission need, which is the first element in a successful AOA and is called for in DOE’s guidance for conducting an AOA. One ORP official said that a succinct definition of the mission need for the AOA does not exist but is or can be deduced from the documents provided to the contractor conducting the analysis. An official from DOE’s Office of Project Management confirmed that there is no mission need statement and noted that because the WTP began prior to the DOE Order requiring a mission need statement, there is no such statement for the WTP or for the current AOA. We have previously noted that defining the mission need is the first step in the AOA process in order to ensure that the AOA process does not favor one solution over another. We have also previously noted that when a concise set of objectives is established, it can ensure that the decision-making process stays open to a range of potential options. Second, as we noted earlier in this report, EM does not have an updated cost estimate for the baseline (or status quo) alternative of completing the pretreatment facility. As such, it is uncertain if or how EM will have a life-cycle cost estimate to compare the baseline alternative to the other alternatives it is analyzing. One of the best practices for an AOA process calls for the inclusion of the cost to pursue the baseline alternative (in this case, the cost of completing the existing pretreatment facility), to provide a basis of comparison among alternatives. However, EM officials told us that they do not intend to update EM’s cost estimate for completing the existing pretreatment facility because it is not a priority for ORP; instead, ORP’s priority is beginning DFLAW operations. Without a life-cycle cost estimate for EM’s baseline alternative, decision makers will not have a complete picture of the costs and will have difficulty comparing the alternatives because comparisons may not be based on accurate information. Without a defined mission need and a complete cost estimate for the baseline alternative, EM’s AOA for HLW treatment alternatives will be missing key elements that are necessary to provide decision makers with the information needed to make the best decision going forward. EM’s analysis and the subsequent decisions that are made based on that analysis could be undermined as a result. Officials from Ecology have raised concerns about EM’s lack of progress on finishing the original pretreatment facility and EM’s shifting focus on the pretreatment mission. In a letter to EM in May 2019, Ecology’s director outlined a series of concerns related to the pretreatment mission and stated that Ecology is not “conceding to, accepting, or acquiescing in any alternative path forward that is different than what has been agreed to in the TPA and Amended Consent Decree between our two agencies.” In September 2019, ORP informed Ecology that a serious risk had arisen that DOE might be unable to meet certain Amended Consent Decree milestones related to, among other things, the construction of the pretreatment facility. In the same month, ORP agreed to participate in “holistic negotiations” to identify a new path forward for treating and disposing of Hanford’s tank waste. As part of this agreement, the parties involved—EM, Ecology, and the Environmental Protection Agency—could use the services of a mediator to assist with negotiations, which may be completed by July 31, 2020. Ecology officials also said that EM has not adequately consulted with them while making important decisions about the pretreatment mission and facility. In particular, in January 2020, Ecology officials told us that they had not been engaged early, often, or appropriately by EM regarding EM’s changing plans to pretreat the tank waste and that they were concerned about the possible negative impacts of EM diverting its resources away from completing the pretreatment facility. According to Ecology officials, they have been invited to key EM meetings but have not been properly engaged in the decision-making process. In an October 2019 presentation to panelists from the National Academies of Sciences, Engineering, and Medicine, Ecology officials noted their frustration with “too many ideas that did not work out, resulting in long delays.” In December 2019, because of concerns that EM was not providing access to all of the information needed to make timely regulatory decisions, Ecology issued a determination requiring EM to provide information as required by the TPA within 30 days. In January 2020, after EM failed to provide the information, Ecology fined EM $1 million and reiterated that without access to this crucial data, it was nearly impossible for Ecology to independently verify compliance with cleanup regulations. According to officials at EM headquarters, engagement with Ecology is a priority, and ORP officials said that since 2018, their engagement with Ecology has improved. In particular, ORP officials noted that Ecology has had representatives on a joint team tasked with exploring the options to be examined under the HLW AOA and has a representative on the AOA review team to observe the deliberations. In September 2019, we outlined a risk-informed framework for making cleanup decisions and recommended that EM incorporate this framework into its cleanup policy across the entire DOE complex. DOE agreed with this recommendation but has yet to respond with a plan to implement it. In that report, we state that the risk-informed decision-making framework can be applied to a range of cleanup decisions, from selecting a cleanup approach at a single site to prioritizing cleanup activities across sites. The risk-informed decision-making framework consists of several steps, including engaging with stakeholders such as Ecology throughout the decision-making process. In that report, we noted that the goal of engaging stakeholder groups in a risk-informed cleanup decision should be to incorporate their viewpoints and seek their acceptance of the decision-making process as transparent and legitimate, rather than to obtain their concurrence with the final decision. We also found that this can best be accomplished when EM seeks stakeholders’ input and buy-in to the process by providing meaningful opportunities for engagement early in the process, communicating throughout the process, and providing transparent, understandable information about the science and rationale behind the final decision. Doing so can help improve the likelihood that stakeholders will view the decision-making process as fair and legitimate. By following the steps outlined in our risk-informed decision-making framework as it makes decisions about the future of the Hanford pretreatment facility, EM and stakeholders would have greater assurance that EM’s decision-making process is transparent, participatory, and credible. After nearly 20 years and with over $11 billion spent since EM awarded the contract to design and build the WTP, the WTP is not complete and has faced numerous technical challenges, cost overruns, and schedule delays. According to a recent study by the U.S. Army Corps of Engineers and EM’s Hanford Lifecycle Report, the largest and most complex portion of the WTP—the pretreatment facility—is unlikely to be completed as designed and scheduled. Since the early years of the project, we have recommended that EM stop moving ahead on the pretreatment facility until it resolves the numerous technical challenges or conducts a reliable analysis of alternatives and determines a risk-informed, cost-effective path forward. However, EM has yet to fully implement these recommendations. EM officials reported that the technical challenges that have plagued the project for years have been solved, but EM has not developed the design and engineering changes needed to implement the solutions. Instead, EM is focusing on analyzing alternatives to accomplish the mission of the pretreatment facility and officials have stated that this analysis will follow best practices we have identified and DOE guidance. EM’s current AOA of HLW treatment alternatives is still under way, and officials told us that they intend to follow best practices for developing an AOA. However, as of February 2020, the AOA still lacks at least two key elements of the best practices. First, without a clear statement of mission need, it is unclear on what basis decision makers will consider and assess the alternatives being considered. Second, without an updated life-cycle cost estimate to complete the pretreatment facility, it is unclear whether the HLW pretreatment alternatives being analyzed represent a better path forward than completing the partially constructed pretreatment facility as originally planned. Without these key elements of an AOA, EM’s ultimate decision may not be the best option or be credible with stakeholders. Throughout this decision-making process, EM’s engagement with Ecology has not met the expectations of the regulator, resulting in fines and further delays as all parties participate in an ongoing, mediated negotiation on a path forward. By following the steps outlined in our risk-informed decision-making framework as it makes decisions about the future of the pretreatment facility, EM can ensure that its regulators have greater assurance that EM’s decision-making process is transparent, participatory, and credible. We are making the following two recommendations to DOE: The Secretary of Energy should direct the Assistant Secretary of Environmental Management to ensure that EM’s final AOA for HLW pretreatment at the Hanford Site includes a definition of mission need and life-cycle cost estimates for the baseline or status quo alternative, as called for in the best practices for an AOA process we have identified and DOE guidance. (Recommendation 1) The Secretary of Energy should direct the Assistant Secretary of Environmental Management to follow the steps outlined in GAO’s risk- informed decision-making framework as EM makes decisions about the future of the pretreatment mission; in particular, engaging the Washington State Department of Ecology in the AOA process, communicating with them throughout the process, and providing them with transparent information about the rationale behind the final decision. (Recommendation 2) We provided a draft of this report to the Secretary of the Department of Energy. In its written comments, reproduced in appendix VI, DOE concurred in principle with our recommendations and outlined a plan to address the recommendations by December 31, 2020. DOE also provided additional technical comments, which we have incorporated into the report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to this report are listed in appendix VII. Our report examines (1) the cost of pretreatment efforts from fiscal year 2013 through fiscal year 2018 and the status of the pretreatment facility, (2) the status of the technical challenges facing the pretreatment facility, and (3) the steps the U.S. Department of Energy’s (DOE) Office of Environmental Management (EM) is taking to begin treating waste by 2023 as required and the extent to which EM has engaged with regulators. To determine the cost of the pretreatment facility, we reviewed Earned Value Management (EVM) status reports from Bechtel National, Inc. (BNI) and fiscal year totals for EM’s oversight costs and BNI’s award and contract modification fees for the pretreatment facility for fiscal years 2013 through 2018 provided by officials in EM’s Office of River Protection (ORP), which oversees the construction of the Waste Treatment and Immobilization Plant (WTP) at Hanford. BNI’s EVM status reports give actual costs for the work performed categorized by a number of different activities, such as engineering to design the pretreatment facility and the acquisition of plant equipment items to be installed in the pretreatment facility. For reporting purposes, we combined BNI accounts with similar activity descriptions and renamed them. To determine the activities included in the accounts, we reviewed both the Work Authorization Document, which describes activities covered by each account used in BNI’s EVM status reports, as well as descriptions of major accomplishments achieved each fiscal year included in the summary status report. To determine the cost of alternative pretreatment efforts, we reviewed EVM status reports for the Direct-Feed Low-Activity Waste project, the Low-Activity Waste Pretreatment System, and the Tank Side Cesium Removal project for fiscal years 2014 through 2018. Because the Test Bed Initiative project did not use an EVM system until fiscal year 2018, we reviewed invoiced costs data for that project for fiscal years 2016 through 2018. To gain context on the planned capabilities of these projects, we reviewed project presentations for pretreatment alternatives and interviewed ORP and BNI officials to learn more about the progress made in developing each project. To assess the reliability of all cost data for both the pretreatment facility and alternative pretreatment efforts, we reviewed documentation and officials’ responses related to data-gathering processes, data storage systems, and data limitations for each of the relevant sources to ORP. Based on this, we found all of the data sources to be sufficiently reliable for our reporting objectives. Finally, to determine the extent to which EM has established a cost estimate to complete the pretreatment facility that is consistent with DOE policy set out in DOE Order 413.3B, we interviewed officials about EM’s cost estimate to complete the facility. To examine the status of technical challenges facing the pretreatment facility and to gather information pertaining to obstacles and risks of project completion, we reviewed the following documents: ORP’s 2018 briefing to the Washington State Department of Ecology (Ecology) regarding the status of challenges, BNI’s 2018 briefing about the status of the pretreatment facility, The U.S. Army Corps of Engineers’ 2018 report on the status of the The Defense Nuclear Facilities Safety Board’s (DNFSB) 2017 technical report on WTP hazards. We also interviewed officials from EM, regulators at Ecology, officials from the U.S. Environmental Protection Agency, and contractor officials who are working to resolve these challenges to better understand the status of the technical challenges, as well as any concerns they might have. In addition, we interviewed officials from DNFSB—an independent agency that provides analysis, advice, and recommendations to the Secretary of Energy regarding the adequate protection of public health and safety at DOE’s defense nuclear facilities—regarding DNFSB’s assessment of the technical challenges and what additional steps, if any, DOE needs to take to resolve the challenges. To examine the steps EM is taking to begin treating waste by 2023 as required, we visited the WTP construction site at Hanford in May 2019 to observe the status of the construction of the pretreatment facility and pretreatment alternatives. We reviewed historical documentation, such as technical reports summarizing testing, and studies conducted by EM and its contractors. These reports included Washington River Protection Solutions’ 2014 low-activity waste (LAW) alternatives analyses summary and its 2011 conceptual design report, and CH2M HILL Hanford Group’s 2006 LAW First Study. We interviewed DOE officials from headquarters to discuss the status of and future plans for the WTP and DOE officials from ORP at Hanford to gather information about the project. We also interviewed ORP contractors regarding their ongoing and planned efforts related to pretreatment of the tank waste and regulator officials from Ecology to better understand their concerns and priorities. To analyze the extent to which EM is following guidance and best practices as it conducts its analysis of alternatives (AOA) of high-level waste (HLW) treatment alternatives, we first interviewed DOE officials and reviewed available documentation associated with DOE’s ongoing AOA to determine the status of the draft AOA. We then reviewed the steps EM is taking and compared them against DOE’s project management requirements (DOE Order 413.3B) and guidance (DOE Analysis of Alternatives Guide) and the best practices for an AOA process that we identified in our prior work. Because EM was conducting its own AOA concurrent with our review, we selected two key best practices in an AOA process—define mission need and develop a life-cycle cost estimate for the baseline (or status quo) alternative—because these two steps are requisite for completing the remaining steps of an AOA. These steps are also essential to ensuring that the other 20 best practices and the results of the AOA are credible and based on accurate information. We also noted best practices that EM officials noted EM has met thus far. In addition, we compared EM’s decision-making process, in particular its stakeholder engagement, against a framework for risk-informed decision- making we developed in our prior work. We developed this framework in 2019 to assist agencies in identifying and implementing the essential elements of risk-informed decision-making. To create the framework, we synthesized key concepts from relevant literature and input from experts who participated in a meeting convened by the National Academies of Sciences, Engineering, and Medicine. We conducted this performance audit from February 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Office of Environmental Management’s contractor for its pretreatment facility, BNI, tracks its costs in its Earned Value Management system. Costs are tracked through the use of different accounting codes that represent the costs of different types of activities. For reporting purposes, we combined BNI accounts with descriptions of similar activities and renamed them. The table below lists the labels we used, the BNI account codes included in each label, and selected examples of activities described for each accounting code. Appendix III: Technical Challenges Facing the Department of Energy’s Waste Treatment and Immobilization Plant’s (WTP) Pretreatment Facility Description Inadequate pulse jet mixing can lead to the accumulation of solids in process vessels, resulting in generation and accumulation of hydrogen and potentially leading to explosions. Settled sludge layers will rise in temperature, increasing the hydrogen generation rate. Up to 16 of the 177 underground tanks at Hanford contain large-size plutonium particles that could settle onto internal surfaces of the pulse-jet mixer vessels, which use compressed air to mix the waste. If the pulse-jet mixers could not then resuspend settled particles, an uncontrolled nuclear chain reaction known as a criticality accident could occur. In the Pretreatment facility and High-Level Waste (HLW) facility, the accumulation of hydrogen gas in piping and small vessels can occur after the loss of off-site power or after an interruption of a transfer of waste due to operator error and during normal operation in isolated pipe sections, potentially causing an explosion. Accumulating solids in pulse-jet mixing vessels could cause excessive air to be discharged in the vessels. This discharge could cause premature erosion of vessel surface bottoms, all of which are located in nonmaintanable areas called black cells. In addition, pulse-jet mixing vessels may need structural modifications to account for abnormal environmental conditions, such as seismic events. Because of uncertainties in waste feed characteristics, the vessel and piping design in the Pretreatment facility and HLW facility may require revisions to account for the amount of wear the equipment will need to withstand. Excessive wear could damage plant equipment and result in interruption of operations or leakage of material from vessels and piping. The potential incorporation of a Standards High Solids Vessel into Pretreatment requires a detailed study to determine the feasibility and optimization of this design change. An additional opportunity is created to revisit the capability to perform In service inspections in order to underpin resolution of erosion/corrosion questions. The Project has not established an in-service inspection program. Once WTP operations begin, equipment in black cells within the Pretreatment facility and HLW facility must last for the WTP’s 40-year expected design life without maintenance because significant failures of components installed in the black cells could impact the throughput and mission duration of the WTP. Potential weaknesses in equipment and piping located within black cells must be identified before WTP operations begin to ensure that timely repairs can be conducted, should failure of these components occur. Ventilation systems in the Pretreatment facility, HLW facility, and Low-Activity Waste facility must be able to contain radioactive material that could be released from primary confinement. The structural integrity of some internal vessel components in these facilities could be compromised if seismic or other events beyond the design basis occur. The ventilation system must survive a release of radioactive material without shutdown, plugging, or blowing out filters to continue to provide confinement. Appendix IV: High-Level Waste (HLW) Alternatives Being Analyzed by the Office of River Protection Description HLW is received, characterized, and pretreated in HLW Feed Preparation Facility; contaminated liquids produced in the process are concentrated in a new HLW Effluent Management. HLW is sampled, characterized, and staged in the tanks. HLW is then pretreated in HLW Feed Preparation Facility; contaminated liquids produced in the process are concentrated in HLW Effluent Management Facility. Same as previous alternative with some processes performed at a higher temperature. HLW is transferred to the pretreatment facility for preparation and staging; then leached, washed, and concentrated in the HLW Feed Preparation Facility. Contaminated liquids produced in the process are concentrated in the pretreatment facility. HLW treated using alternative technologies such as grouting or steam reforming. Would require technology development, research and development, lab testing and technology readiness assessment. HLW is immobilized within existing tanks using alternative technologies. Would require technology development, research and development, lab testing and technology readiness assessment. HLW in the tanks located furthest away from the Waste Treatment and Immobilization Plant (in the western portion of the site) is pretreated and treated in new west area HLW Feed Preparation Facility, HLW Effluent Management Facility, and HLW vitrification facilities. HLW is received, characterized, and pretreated in in HLW Feed Preparation Facility; contaminated liquids produced in the process are concentrated in new facilities. Pretreatment facility repurposed to treat low-activity waste. HLW is transferred to compliant mediums for transfer to Savannah River Site for treatment and vitrification. Fuels Material Examination Facility would be retrofitted to provide pretreatment capabilities. HLW vitrification facility is abandoned; pretreatment facility is repurposed to pretreat and vitrify HLW. HLW is pretreated and vitrified at a near-tank mobile facility or in a centrally located facility using bulk vitrification technology. Would require technology development, research and development, lab testing and technology readiness assessment. Same as the second alternative above with added filtering capability. HLW is sampled, characterized, and staged in the tanks. Contaminated liquids produced in the process are concentrated in HLW Effluent Management Facility. Same as previous alternative with added step of concentrating the HLW in the HLW Effluent Management Facility before sending it to be vitrified. The guidance below is meant as an overview of the key principles that lead to a successful AOA process and not as a “how to” guide with detailed instructions for each best practice identified. Conforming to the 22 best practices helps ensure that the preferred alternative selected is the one that best meets the agency’s mission needs. Not conforming to the best practices may lead to an unreliable AOA, and the customer will not have assurance that the preferred alternative best meets the mission needs. Table 6 shows the 22 best practices. David C. Trimble, (202) 512-3841 or trimbled@gao.gov In addition to the contact named above, Amanda K. Kolling (Assistant Director), Jeffrey T. Larson (Analyst in Charge), Mark Braza, Kelly Friedman, Richard P. Johnson, Gwen Kirby, and Alan K. Smith made key contributions to this report.", "summary": "The Hanford Site in Washington State contains large quantities of nuclear waste. EM has been building the Waste Treatment and Immobilization Plant—which consists of multiple facilities, including a key pretreatment facility—to treat a large portion of the nuclear waste at Hanford. Under way since 2000 and costing over $11 billion to date—$3.8 billion of that spent on the pretreatment facility—the plant has faced technical challenges, cost overruns, and schedule delays. In late 2012, work on the pretreatment facility stopped until technical challenges could be resolved. In 2018, the U.S. Army Corps of Engineers reported that at current annual funding levels, completing the pretreatment facility on time would not be possible. Senate Report 116-48 accompanying the National Defense Authorization Act for fiscal year 2020 included a provision for GAO to review this project. This report examines (1) the cost of pretreatment efforts from fiscal year 2013 through fiscal year 2018, (2) the status of the technical challenges facing the pretreatment facility, and (3) the steps EM is taking to start treating waste by 2023 as required, among other things. GAO toured the facility, analyzed EM documents and expenditure data, and interviewed EM officials. The Department of Energy's (DOE) Office of Environmental Management (EM) spent $752 million in fiscal years 2013 through 2018 on the pretreatment facility at the Hanford Site in Washington State. This facility was to separate nuclear waste into two streams for treatment in other site facilities. However, EM stopped design and construction of the facility in 2012 due to technical challenges. According to expenditure data, over half of the $752 million EM spent was for overhead, oversight, procurements, and facility maintenance. The rest was spent resolving the technical challenges. DOE's fiscal year 2020 budget request states that EM plans to continue “limited activities”—such as maintaining the existing facility and storing uninstalled equipment—while construction remains on hold. After working to address pretreatment facility technical challenges since 2012, EM and its contractor consider these challenges—ranging from facility ventilation concerns to preventing explosions during waste treatment—to be conceptually resolved. However, EM has not yet designed, engineered, or tested solutions to the challenges. In addition, the Defense Nuclear Facilities Safety Board—an independent agency that provides analysis, advice, and recommendations regarding safety at DOE's defense nuclear facilities—does not consider the challenges resolved pending additional information and, in some cases, additional design and engineering work by EM. To begin treating waste by 2023 as required, EM has been pursuing alternatives to the pretreatment facility. Since 2013, EM has spent over $400 million pursuing alternatives for low-activity waste pretreatment capabilities originally planned for the pretreatment facility. However, as GAO reported in May 2015, EM did not properly define a mission need statement or a life-cycle cost estimate prior to selecting its preferred alternative for treating low-activity waste, consistent with analysis of alternatives best practices and DOE policy, and GAO recommended EM revise its analysis. In April 2019, EM began an analysis of alternatives for treating high-level waste, which EM expects to be completed in September 2020. However, as of February 2020, EM had not yet defined a mission need for this new analysis of alternatives and did not have a life-cycle cost estimate for its baseline alternative. Without these, decision makers will not have the information they need to make the best decisions for pretreating high-level waste, and EM cannot assure decision makers that alternative approaches meet mission needs. GAO is making two recommendations, including that DOE ensure that its analysis of alternatives for pretreatment of high-level waste include a mission need statement and a life-cycle cost estimate for the baseline alternative. DOE concurred in principle with both recommendations.", "document_type": "gao"}
{"report": "EEOC provides leadership and guidance to federal agencies on all aspects of the federal government’s EEO program. EEOC assures federal agency and department compliance with EEOC regulations, provides technical assistance to federal agencies concerning EEO complaint adjudication, monitors and evaluates federal agencies’ affirmative employment programs, develops and distributes federal sector educational materials, conducts training for stakeholders, provides guidance and assistance to Administrative Judges who conduct hearings on EEO complaints, and adjudicates appeals from administrative decisions made by federal agencies on EEO complaints. EEOC’s MD-715 requires agencies to take appropriate steps to establish a model EEO program and to ensure that all employment decisions are free from discrimination. It also sets forth the standards by which EEOC will review the sufficiency of agency Title VII and Rehabilitation Act programs, which include periodic agency self-assessments and the removal of barriers to free and open workplace competition. Under MD-715, federal agencies, and any subordinate component that enjoys autonomy from its parent agency, are required to submit annual MD-715 EEO program status reports to EEOC. Completed MD-715 reports include: Part F: Requires designated agency officials to certify that the agency has completed an annual self-assessment (Part G) and established plans to correct any program deficiencies (Part H), as well as conducted comprehensive barrier analyses and established plans to eliminate identified barriers (Part I). Part G: Contains a self-assessment checklist for an agency to assess its compliance with essential EEO program elements to operate a model EEO program and identify any deficiencies. Part H: Describes the agency’s plans to address identified deficiencies. Part I: Shows identified EEO triggers and barriers for race, gender, and national origin; how the agency plans to address them; and who the responsible officials are. Part J: Contains the agency’s affirmative action plan for individuals with disabilities and individuals with targeted disabilities. Additionally, federal agencies are required to identify and eliminate barriers that impede free and open competition in their respective workplaces. EEOC reporting requirements state that a barrier is an agency policy, procedure, practice, or condition that limits or tends to limit employment opportunities for members of a particular gender, race, or ethnic background, or for individuals based on disability status. According to EEOC’s instructions, many employment barriers are built into the organizational and operational structures of an agency, and are embedded in the day-to-day procedures and practices of the agency. Agencies are also required to identify EEO program deficiencies and develop plans to address them. According to EEOC’s instructions, deficiencies are weaknesses in an agency’s EEO program where agency officials need to provide more attention. For example, a deficiency might be that the EEO director is not under the direct supervision of the agency head, or that an EEO Director or Officer lacks a regular, effective means of informing the agency head and other top management of the effectiveness, efficiency, and legal compliance of the agency’s EEO program. EEOC’s Office of Federal Operations instructs agencies on how to complete their MD-715 reports, provides training and technical assistance, and offers additional informal assistance, such as sharing best workplace practices. Based on agency MD-715 reports, EEOC includes assessments of agency progress in its Annual Report on the Federal Workforce, and in notice and feedback letters addressed to individual agencies. In addition, according to EEOC officials, EEOC meets with each agency every 3 years to review the status of its compliance with federal EEO laws, regulations, and management directives. If EEOC determines that areas of noncompliance exist in an agency’s program, it may take compliance actions. Compliance actions include requiring the agency to provide an update on the status of its plans to correct deficiencies in its MD-715 submission, or to submit a Compliance Report to EEOC explaining the agency’s progress in correcting deficiencies within 6 months of the date of a feedback letter. If agencies do not comply, EEOC may choose to initiate its noncompliance process, which could include conducting a program evaluation, issuing a notice to the Secretary of Homeland Security, or publicly identifying DHS as a noncompliant agency. As part of its noncompliance process, EEOC has conducted program evaluations of DHS components. In 2013, EEOC initiated a program evaluation at TSA to determine the adequacy and appropriateness of the EEO complaint framework in all offices and directorates within TSA. EEOC reported in 2014 that TSA’s EEO complaint process was adequate and complied with its regulations. However, EEOC made eight recommendations to TSA, including one that called for TSA to modify the EEO information in its training materials and presentations. In response to EEOC’s report, TSA submitted a letter to EEOC stating that it planned to address all of the EEOC recommendations and had already taken steps to implement several of them. In addition, in 2018, EEOC conducted a multiagency program evaluation that included U.S. Customs and Border Protection (CBP). It found that CBP had no women serving in positions that involve intercepting prohibited commodities or persons, and that women comprised only 5 percent of its border patrol agents. EEOC stated that the report’s recommendations may help CBP with its hiring efforts. CRCL, through the Deputy Officer for EEO and Diversity, is responsible for processing complaints of discrimination; establishing and maintaining EEO programs; fulfilling reporting requirements as required by law, regulation, or executive order; evaluating the effectiveness of EEO programs; leading the department’s diversity management program; and preparing and submitting DHS’s annual MD-715 report to EEOC. According to EEOC policy, a second-level reporting component is one that enjoys autonomy from its parent agency, and has 1,000 or more employees. EEOC instructions require second-level reporting components submit MD-715 reports to their agency headquarters for inclusion in the agency-wide report in addition to submitting them directly to the EEOC. DHS’s Headquarters EEO Office, a part of CRCL, implements the EEO program for all headquarters employees and applicants. DHS has nine second-level reporting components, including DHS Headquarters, that are required to submit individual MD-715 reports to EEOC. Each component has an office headed by a director charged with implementing its EEO program. Figure 1 shows the officials who are primarily responsible for EEO at DHS. CRCL develops DHS’s annual MD-715 EEO program status report and submits it to EEOC. CRCL works with components to gather and analyze necessary data and information, and to perform the required MD-715 exercises that are ultimately used to complete the overall DHS MD-715 report. CRCL includes components’ identified deficiencies in the DHS- wide MD-715 report. The Secretary of Homeland Security (or its designee) and CRCL’s Deputy Officer for EEO and Diversity are to certify DHS’s MD-715 report before CRCL sends the report to EEOC. Figure 2 illustrates DHS’s MD-715 report development process. This process includes conducting a self-assessment checklist of DHS’s and its components’ efforts to achieve a model EEO program and barrier analysis to eliminate identified EEO barriers. DHS has taken steps to follow EEOC’s guidance by using and analyzing various information sources, investigating possible causes of potential barriers or triggers, and planning activities to address and eliminate barriers. EEOC MD-715 guidance calls for federal agencies to continually work towards preventing all forms of discrimination and eliminating barriers that may impede free and open competition in the workplace. Figure 3 shows the barrier identification and elimination steps under MD- 715. DHS generally uses the information sources that EEOC guidance recommends in addition to workforce data to help identify potential barriers. As directed by EEOC guidance, DHS analyzes its workforce data to help identify triggers or indicators of potential EEO barriers by comparing the racial, national origin, gender, and disability profiles of its total workforce, and for various occupational categories to relevant civilian labor workforce data. In fiscal year 2017, DHS analyzed all available workforce data including: Total Workforce – Distribution by Race/Ethnicity, Gender, and Participation Rates by Major Occupations – Distribution by Race/Ethnicity, Gender, and Disability, and Applicants and Hires by Major Occupations – Distribution by Race/Ethnicity, Gender, and Disability. In addition to analyzing workforce data, in each of the fiscal years 2014 through 2017, DHS utilized the U.S. Office of Personnel Management’s Federal Employee Viewpoint Survey (FEVS) and DHS’s employee exit survey results to help identify and address barriers. For example, CRCL, in DHS’s fiscal year 2017 MD-715 report, used FEVS and exit survey results to help investigate the possible causes of higher-than- expected nonretirement separations for white females and several other ethnic and racial groups. According to the report, possible causes included the lack of advancement opportunities, insufficient work/life programs, and the lack of alternate work schedules. During our small group discussions, DHS employee groups told us that through the MD-715 report development process, they helped identify and address triggers and barriers. For example, Special Emphasis Program Managers we spoke with told us that DHS components conduct climate surveys to obtain input from employees on workforce practices every 1 or 2 years. Further, several DHS components’ MD-715 reports referenced soliciting employee input, such as obtaining Disability Employment Program Managers’ input via quarterly disability employment advisory council meetings where they share best practices and discuss issues and topics including barriers. Our review of DHS’s MD-715 reports showed that DHS identified three department-wide triggers in fiscal years 2014 through 2017. The three triggers were (1) high rate of nonretirement separations for certain groups, particularly white women; (2) low participation rates of women and various ethnic and racial groups in the permanent workforce; and (3) low participation rates of individuals with disabilities and targeted disabilities. Subsequent to its trigger identification and department-wide barrier analysis, from fiscal years 2014 through 2017, DHS identified three barriers: (1) problems with supervision/management, lack of advancement opportunities, lack of alternate work schedules, insufficient work/life programs, and personal/family related reasons causing higher- than-expected nonretirement separations for white females and several ethnic and racial groups; (2) the geographic location of jobs which has contributed to the low hiring rates of racial groups in certain major occupations; and (3) medical and physical requirements of law enforcement positons, such as the ability to engage in moderate to arduous physical exertion, which limit the eligibility of some applicants with targeted disabilities. DHS identified these barriers by analyzing component and DHS level workforce data and reviewing DHS FEVS and exit survey results. DHS identified barriers in its MD-715 reports for fiscal years 2014 through 2017. However, EEOC noted that for fiscal years 2015 to 2017, DHS had not identified any policies, procedures, practices, or conditions causing (1) low hiring rates for women in certain major occupations, and (2) the high separation rate of employees with disabilities. As stated in EEOC’s guidance, barrier elimination is a vital step to addressing identified barriers and working towards the goal of making the federal government a model employer. To address and eliminate identified barriers, EEOC’s instructions direct agencies to include in their MD-715 reports measurable objectives, an action plan that includes planned activities and completion dates, as well as officials responsible for overseeing the plan, and a summary of accomplishments. Since our 2009 recommendations, DHS has included interim milestones in its MD-715 reports. Our 2009 report showed that DHS had modified nearly all of its target completion dates. We recommended that DHS identify essential activities and establish interim milestones necessary for the completion of all planned activities to address identified barriers to EEO. In its fiscal year 2011 MD-715 report to EEOC, DHS identified essential activities and established interim milestones. Based on its MD- 715 reporting for fiscal years 2014 through 2017, DHS has continued to identify planned activities and establish interim milestones. Our review of DHS’s MD-715 reports from fiscal years 2014 through 2017 also shows that DHS has planned activities and targeted completion dates to address each identified barrier, and each trigger for a potential barrier. For example, to address the low participation rates of women and several ethnic and racial groups in DHS’s overall workforce, DHS’s planned activities included researching where to conduct outreach for the identified groups, and producing a plan to integrate data from the multiple applicant data-tracking systems used across DHS. DHS’s outreach activities included identifying colleges and universities with large populations of underrepresented groups, identifying relevant job fairs in selected service areas, and conducting focus group meetings with employees from underrepresented groups to determine how to improve recruitment and retention, among other events. These events were initiated in 2011, but are to be reviewed and updated annually. For example, DHS reported that it develops a “Top 25” list of annual outreach and recruitment activities that include law enforcement focused events. DHS also reported developing a framework in 2016 for applicant flow data analysis—important for identifying and addressing potential recruitment and outreach barriers. In 2017, activities included conducting more robust department-wide analysis of applicant data. Many of the activities were initiated in prior years and target dates for completion were met. To address the high nonretirement separation rate of certain groups, notably white women, DHS’s planned activities included updating and augmenting previously instituted exit survey methods, and identifying and implementing retention interventions. Further, in its fiscal year 2014-2017 reports, DHS has identified essential activities, established interim milestones, and met recurring interim milestones for its planned activities. For example, DHS reported that it planned to research where to conduct outreach for groups in occupations with underrepresentation. DHS components completed this outreach activity in 2012, and components and facilities are to annually identify (1) colleges with substantial populations of underrepresented groups, (2) relevant job fairs in the service area, and (3) relevant local affinity groups and community groups, among other outreach activities. Additionally, DHS’s Office of the Chief Human Capital Officer (OCHCO) has lead responsibility for implementing a multiyear plan for targeted recruitment of applicants from identified underrepresented groups. OCHCO completed its initial multiyear plan in 2012 and is to annually update its established goals for intern programs, job fairs, and local advertising. All four selected DHS components have taken steps to follow EEOC guidance to conduct barrier analyses. Of the components, Federal Law Enforcement Training Centers (FLETC), the U.S. Secret Service (Secret Service), the Transportation Security Administration (TSA), and U.S. Citizenship and Immigration Services (USCIS), only one, TSA, identified any EEO barriers. However, each of the components identified triggers and analyzed potential barriers by reviewing workforce data (i.e., data on total workforce, new hires, and mission critical occupations) and comparing the data to relevant benchmarks, reviewing various information sources to help identify possible barriers that may be resulting in the current condition highlighted by the analysis of workforce data, and reporting action plans and time frames for addressing potential or actual barriers. The Secret Service’s fiscal year 2017 MD-715 report showed that after analyzing demographic data to identify triggers, the Secret Service used FEVS data to identify potential barriers to the employment of individuals with disabilities in occupations where the triggers were identified. In addition, USCIS stated in its fiscal year 2017 MD-715 report that its review of exit survey data provided reasons that men, Hispanics, and whites left the agency, but data were inconclusive regarding the continuing underrepresentation of those groups. USCIS also reported that it would continue analyzing exit data in fiscal year 2018. In fiscal year 2017, TSA identified two barriers in its MD-715 report—(1) medical and physical restrictions limit opportunities for individuals with disabilities and individuals with targeted disabilities in Transportation Security Officer and Federal Air Marshal occupations, and (2) women are not applying to Transportation Security Officer or Federal Air Marshal positions at the same rate as men. TSA reported that it analyzed workforce data and policies, procedures, and practices related to recruiting, hiring, and promotions to try to determine what may be contributing to low participation rates for women and individuals with disabilities. TSA also interviewed employees involved in those processes, and conducted focus groups with supervisors and leadership at airports and field offices. TSA’s plans to address barriers include developing a communication plan to promote TSA programs that support persons with disabilities and with targeted disabilities; making sure training modules are accessible; conducting training to increase awareness of unconscious bias towards working with individuals with disabilities; and working with its human capital office and others to assist with recruiting and hiring to more effectively target women. Although FLETC, Secret Service, and USCIS did not identify EEO barriers in fiscal year 2017, they each developed action plans that identified activities designed to help address and correct undesired conditions, identified responsible officials, and set time frames for addressing the conditions. Examples of selected components’ plans and activities include: FLETC. To address low participation rates of persons with targeted disabilities in the permanent workforce, FLETC-planned activities include working with human resource specialists to identify data and timelines needed to create reports in its applicant data flow system that would help identify any barriers in the selection process, and working to resolve issues concerning applicant flow data in the applicant pool. Secret Service. To address low participation rates of certain groups in the general workforce and new hires, planned activities include quarterly tracking and reporting ethnicity, race, and gender data net changes, hires, resignations, and retirements. Other activities would involve working closely with the Office of Human Resources Talent and Employee Acquisition Management Division in recruitment activities. USCIS. To address the lower-than-expected participation rate of certain groups in the permanent workforce—for example, white males and females and Hispanic males—planned activities include conducting comprehensive applicant flow data analysis of the top five major occupational categorizes, and administering and analyzing a bi- annual EEO and Diversity Climate Survey. DHS has provided training for its components on how to conduct EEO barrier analysis. In 2016 and 2018, DHS trained DHS component EEO officials on methods for identifying the root of specific triggers in the workplace, as well as steps for eliminating identified barriers. According to DHS’s analysis of participant training evaluations, the majority of participants believed they would be able to apply what they learned from the training. In 2017, DHS provided a 2-day barrier analysis training to agency and component affirmative employment practitioners that introduced various barrier analysis methods. It included an exercise involving a hypothetical federal agency. Based on our review of participant evaluations, participants were satisfied with the training. DHS reported improvements in EEO indicators in its MD-715 reports from fiscal years 2014 through 2017. DHS cited its higher FEVS scores under employee engagement. For example, although DHS’s employee engagement remained 7 percent below the government-wide average, it increased from 54 percent in 2014 to 60 percent in 2017. According to DHS, this score was largely driven by TSA and U.S. Customs and Border Protection employees, who accounted for 46.8 percent of DHS’s completed surveys. Our review of DHS’s workforce data from fiscal years 2014 through 2017 showed that every minority group as well as individuals with disabilities and individuals with targeted disabilities had been trending in a positive direction since fiscal year 2014. Further, DHS officials told us that minority representation was up 3 percent and female representation was up 2 percent since 2015. In addition, DHS has produced barrier analysis reports that address underrepresentation of women and various ethnic and racial groups. In 2018, DHS completed a barrier analysis report on Hispanic employment in General Schedule pay scale grades 12 and higher, as required by EEOC and the U.S. Office of Personnel Management. The report identified several potential triggers, such as Hispanic women separating from DHS, and related barriers, such as possible harassment of Hispanic employees and women, and glass walls. DHS also developed action plans focused on enhancing elder and family care programs, offering training on preventing harassment in the workforce, increasing recruitment into job series with substantial promotion opportunities, and ensuring interview panels were diverse and interviewers properly trained. Although DHS has reported positive trending in various underrepresented groups, DHS officials said they were unable to fully identify the barriers contributing to the underrepresentation of women in its workforce despite conducting the required barrier analysis. In 2014, DHS conducted a barrier analysis of women in law enforcement to help identify any barriers. While specific barriers were not identified, DHS’s report, Women in Law Enforcement Study, provided insight into why DHS employed lower rates of female law enforcement officers than federal government-wide. For example, study participants shared anecdotal instances of where they or their colleagues did not pursue promotional opportunities because they perceived their work environment made them choose between the job and family. The study also highlighted steps DHS could take to help address its underrepresentation of women, such as being more creative in its approach to attracting qualified women through use of social media, and by creating more family-friendly environments. According to EEOC, one important tool in examining the fairness and inclusiveness of the federal government’s recruitment efforts is applicant flow data. By reviewing the yield of an agency’s recruitment effort, the organization can reassess and improve its effort to reach all segments of the population. EEOC guidance states that having department-wide applicant flow data could aid in analyzing differences in selection rates among different groups for a particular job. In July 2017, EEOC informed DHS that the agency’s applicant flow data were incomplete. DHS has reported challenges in collecting department-wide data that could help identify potential barriers. EEOC found that DHS’s workforce data tables do not always contain all of the agency’s applicant flow data. According to EEOC, without such data, it becomes much more difficult to pinpoint the specific policies, procedures, or practices in which barriers might be embedded. DHS does not have a consolidated applicant flow data system. According to DHS, four of its components use one system (USA Staffing), while five other components use a different system (Monster Government Solutions). Office for Civil Rights and Civil Liberties (CRCL) officials told us DHS is developing a new system to integrate applicant flow data department-wide. However, the officials could not give us a time frame for when the system is expected to be completed. As a work-around, DHS explained that it obtains these data directly from each component that uses Monster Government Solutions. CRCL officials said they will report complete applicant flow data in fiscal year 2019. In addition to creating a model EEO environment, progress in eliminating EEO barriers can help DHS avoid costs related to workplace disputes. According to EEOC guidance, the elimination of barriers may help an agency avoid expensive costs, such as back pay awards, compensatory damages, and attorney’s fees, from findings of discrimination. EEOC found 81 instances of discrimination from fiscal years 2014 through 2017 resulting in DHS paying nearly $30 million to cover judgments, awards, and settlements for these EEO cases, or an average of $7.4 million per year. These expenses were nearly equal to the average annual cost of DHS’s EEO program, which DHS estimated at about $7.63 million in fiscal year 2019. DHS does not have complete performance metrics or mechanisms for tracking progress towards eliminating its identified EEO barriers. For example, CRCL does not maintain numerical objectives or goals for eliminating barriers involving certain EEO groups, such as workplace satisfaction of white females or the retention rate of women in law enforcement positions. According to CRCL officials, they are not required to establish performance metrics or mechanisms for tracking progress towards eliminating barriers beyond what is included in the department- wide MD-715 report. DHS reported one performance measure for its EEO program—the percent of timely merit Final Agency Decisions (FADs). Standards for Internal Control in the Federal Government states that management should establish specific and measureable objectives, and ways to assess progress including performance metrics and milestones. It also states that management should design control activities to achieve objectives and respond to risks. Such control activities may include the establishment and review of performance metrics. Further, EEOC guidance states that agencies are not prevented from establishing additional practices that exceed its requirements. DHS officials acknowledged that their EEO program performance measurement does not reflect all the work that they do. According to CRCL officials, CRCL has proposed additional performance measures for its MD-715 activities, but they were rejected by DHS’s Office for Policy because they were not directly related to national security or public safety. DHS’s Office for Policy is responsible for approving new performance measures. CRCL officials told us that adopting hiring goals for individuals with disabilities and individuals with targeted disabilities—which had previously been identified as potential barriers—has been beneficial in garnering support and commitment towards meeting them. They said that DHS incorporated these goals into its efforts and initiatives to increase the recruitment, hiring, advancement, and retention of individuals with disabilities. Implementing performance metrics could help DHS better assess its progress in eliminating EEO program barriers. As shown in table 1, our analysis of DHS’s MD-715 reports found that DHS did not meet about a quarter of the compliance measures for a model EEO program for each fiscal year from 2014 through 2017. Specifically, over this 4-year period, DHS did not meet 26 percent of its compliance measures (128 out of 487). The largest percentage of unmet measures occurred under the model EEO essential element D—which focuses on proactive steps taken by an agency to prevent unlawful discrimination—where about 53 percent or 21 of 40 measures were unmet. According to DHS officials, in Part G of its MD-715 report, DHS includes deficiencies identified and reported at the component level as well as deficiencies directly attributable to the department. For example, in each of the fiscal years 2015 through 2017, DHS reported that it did not meet a compliance measure under element D that senior managers successfully implement EEO action plans and incorporate EEO action plan objectives into agency strategic plans. Specifically, in fiscal years 2015 through 2017, DHS noted that USCIS had not met this measure, and in fiscal year 2017, the Federal Emergency Management Agency (FEMA) and DHS Headquarters did not meet this measure. Our analysis of components’ MD-715 reports showed that components did not meet 9 percent of the compliance measures for a model EEO program from fiscal years 2014 through 2017. Specifically, over this 4- year time frame, components had a combined total of 369 program deficiencies out of a total of 4,229 compliance measures. DHS Headquarters, one of the nine second-level reporting components, accounted for 36 percent of deficient measures (134 of 369), while the other eight components accounted for 64 percent (235 of 369) of deficient measures. Examples of DHS’s deficient measures included EEO directors not under the direct supervision of the agency head, and the lack of established timetables or schedules for the agency to review its employee development and training programs for systemic barriers that may be impeding full participation in training opportunities by all groups. DHS and its components did not have action plans to address some of their self-identified deficiencies from fiscal years 2014 through 2017. Specifically, DHS did not have action plans to address 56 percent, or 72 of the 128 reported deficiencies, and components did not have action plans to address nearly half, or about 179 of the 369 deficiencies reported by all of the components during the four year period. For example, in fiscal year 2017, four out of nine DHS components—U.S. Customs and Border Protection (CBP), DHS Headquarters, FEMA, and Federal Law Enforcement Training Centers (FLETC)—did not have action plans to ensure that their EEO directors report directly to their agency heads. EEOC guidance requires agencies to demonstrate meaningful progress toward the removal of deficiencies, and to develop action plans for how agencies will attain the essential elements of a model EEO program. Specifically, for each deficient measure, agencies are to develop an action plan for correcting the deficiency. The plan should identify and briefly describe the deficiency; provide a measurable objective, including the reason for the deficiency, and target date for completion; identify officials responsible for overseeing implementation of planned activities to accomplish the objective; and provide for a yearly update on status of activities until objective is completed (i.e., the deficiency is removed). In addition, Standards for Internal Control in the Federal Government states that management should design control activities to achieve objectives. Control activities, such as policies or procedures to enforce directives, can help identify if a required activity is not being achieved (e.g., action plan completion) or implemented. The four selected DHS components told us that they do not have standard operating procedures for completing their MD-715 reports, including review and assessment of deficiencies and action plans, but have various processes in place to review their reports for accuracy and completeness. For example, according to USCIS, its process for the preparation and review of its MD-715 report includes providing a self- assessment checklist to each of its subcomponents; a review of their responses for accuracy; and follow-up with subcomponents to address any questions. In addition, USCIS stated that its MD-715 report undergoes multiple levels of reviews by subject matter experts and managers that include collaboration with human capital and chief counsel, and obtaining review and approval from the Director and other agency officials. The other three components—FLETC, TSA, and Secret Service—also reported having MD-715 review processes in place, including report review and approval by senior management; however, none specifically cited a review and approval of action plans to address reported deficiencies. CRCL officials told us that DHS and its components’ MD-715 reports met EEOC requirements for action plans for fiscal years 2014 through 2017 by providing explanations for, or briefly stating plans to address, the majority of their deficiencies rather than developing action plans identifying how each deficiency would be addressed. During our review of the MD-715, we noted that the Part G self-assessment checklist form gave respondents the option of providing a brief explanation in a comment box on the form or completing an action plan for each deficiency in Part H of the MD-715 report. For fiscal year 2018, EEOC revised its MD-715 report form and instructions to clarify that a plan is required for each identified program deficiency. For example, one component responded to a measure that asks whether an agency implemented an adequate data collection and analysis system that permits tracking of the information required by MD-715 and these instructions by stating “selected offices were currently working on the initiative.” The same component responded to a measure that asks whether an agency tracked recruitment efforts and analyzed efforts to identify potential barriers in accordance with MD-715 standards by stating, “The inconsistencies with the reporting of applicant flow data will need to be addressed to help identify potential barriers.” Another component responded to this measure by stating that, while its participation in various events are tracked, a clear, concise, and efficient system to track and analyze recruitment efforts according to MD-715 standards is currently not in place. Neither component provided a plan for how these deficient measures would be addressed. EEOC continues to identify areas of noncompliance in DHS component EEO programs. For example, in fiscal year 2017, EEOC noted that three of four selected components had areas of noncompliance. The areas of noncompliance included (1) failure to timely issue FADs, and (2) not establishing timetables or schedules to review its merit program policies and procedures, employee recognition awards programs, and employee developmental training programs for potential barriers. Developing policies or procedures, in consultation with the Deputy Officer for EEO and Diversity, to help ensure component EEO programs have action plans with measurable objectives for addressing deficiencies could help DHS components better comply with EEOC requirements. DHS continues to report insufficient staffing to support its EEO program. In 2009, we reported that, according to CRCL, DHS modified the target dates for planned activities to address identified barriers primarily because of staffing shortages in both CRCL and the Office of the Chief Human Capital Officer. We also reported that DHS had not conducted barrier analyses of policies, procedures, and practices that were established or used after fiscal year 2004 because of resource limitations, such as staffing and limited funding to contract for this activity. According to CRCL’s MD-715 and Notification and Federal Employee Antidiscrimination and Retaliation (No FEAR) Act reports from fiscal years 2014 through 2017, certain aspects of DHS’s EEO program did not have sufficient staffing. In addition, in fiscal years 2014 through 2017, DHS reported that staffing shortages contributed to it not meeting its target for the percent of timely decisions on discrimination complaints. In fiscal year 2017, DHS reported deficiencies for five out of seven staffing measures in its MD-715 report. In February 2019, CRCL officials told us they lacked staffing to issue timely decisions on discrimination complaints, to increase the number of mediators in the alternative dispute resolution program, and to provide them with training. From fiscal years 2014 through 2017, DHS and its components reported funding and staffing challenges in components’ EEO programs. As shown in table 2, DHS and its components reported that certain aspects of components’ EEO programs do not have sufficient funding or staffing from fiscal years 2014 through 2017. EEOC guidance states that an agency must provide its EEO program with sufficient budget and staffing to be able to successfully implement various activities, including (1) conducting a self-assessment of the agency for possible program deficiencies; (2) conducting a thorough barrier analysis of its workforce; and (3) ensuring timely, thorough, and fair processing of EEO complaints. CRCL and component EEO officials told us that they do not have formal staffing models to assess appropriate staffing of their EEO program sections. CRCL officials explained that each component EEO program section is unique with its own assessments and measures by the leaders in charge of their funding and staffing resources. Using these informal processes to identify staffing needs, CRCL and component EEO officials told us that they have requested additional staffing and funding to address some of their EEO program deficiencies from their top leadership. However, they said that additional staffing has not been granted. Our analysis of DHS’s congressional budget justifications show that DHS’s EEO program funding requests have decreased each year from nearly $8 million in fiscal year 2016 to nearly $7 million in fiscal year 2019. CRCL officials told us that DHS’s overall resources for the EEO program have not significantly increased. A staffing model could be a computer-based formula that estimates the number of staff needed to conduct varying numbers of EEO activities, such as processing a certain number of complaints or providing a certain number of training courses on an annual or ad hoc basis. As we have reported, a staffing model is a helpful tool that could better justify requests for resources to top leadership. Staffing models can identify resources required to enable program delivery to a sufficient degree and in a timely manner, or to adapt to changes in program delivery. According to DHS, the department has contracted support to help components develop models for Mission Support areas as part of a larger effort to ensure that all positions are eventually covered by a staffing model. Developing and utilizing a formal staffing model for its EEO program could help CRCL better identify, request, and obtain the staff it needs. Further, developing staffing models, in collaboration with the Deputy Officer for EEO and Diversity, would help components to better assess the staff they need. DHS did not respond timely to EEOC’s findings of noncompliance and EEOC did not follow up with DHS concerning the untimely response. In July 2017, in its most recent review of DHS compliance with EEOC requirements, EEOC reported nine areas of noncompliance in DHS’s EEO program. For example, EEOC found that DHS lacked resources to process EEO complaints and to conduct trend analyses of workforce data. EEOC stated in its feedback letter to DHS that it would initiate its noncompliance process if DHS did not submit a report explaining the agency’s progress in correcting its EEO program deficiencies by January 2018. However, according to DHS officials, due to an administrative oversight, DHS was unaware of EEOC’s July 2017 feedback letter until October 2018, when we asked about it. In February 2019, DHS submitted a report to EEOC that responded to each area of noncompliance. EEOC officials told us that it had not initiated its noncompliance process against DHS, but that it had placed DHS in its queue for agencies to be held in noncompliance. As discussed earlier, DHS Headquarters, a second-level reporting component, is required by EEOC to submit a separate MD-715 report to EEOC. However, DHS’s Headquarters EEO Office did not submit a separate MD-715 report to EEOC during fiscal years 2014 through 2017. DHS Headquarters EEO Office staff told us that the office had not submitted the required reports due to staff vacancies, including its EEO director position. They explained that the component’s EEO data and information were subsumed in DHS’s department-level MD-715 submission. In October 2018, CRCL filled its Headquarters EEO director position, which had been vacant for 8 months. DHS officials told us they plan to submit Headquarters’ fiscal year 2018 MD-715 report to EEOC by the due date. In February 2019, EEOC officials told us that DHS could be subject to EEOC’s noncompliance process if the report is not received. As shown in figure 4, CRCL and the Strategic, Recruitment, Diversity, and Inclusion (SRDI) Office support and oversee components in their efforts to identify and address EEO barriers. For example, CRCL convenes an EEO council consisting of EEO directors from each component that meets monthly and, among other things, shares best practices for identifying and addressing barriers. In addition, CRCL hosts EEO and Diversity Training Conferences for EEO staff that includes barrier analysis training. Further, CRCL provides midyear feedback to component EEO officials on components’ planned action items and plans for inclusion in their respective MD-715 reports. For example, based on our review of the CRCL statistician’s notes, during feedback meetings with components in 2017, he suggested that components consider opportunities for improving their draft MD-715 reports. The notes show that at least two out of nine components—CBP and DHS Headquarters—were given feedback to conduct more robust barrier analyses. SRDI supports component EEO program efforts to address EEO barriers related to recruitment, hiring, veterans, and individuals with disabilities. For example, to increase the participation of women in law enforcement across the department, SRDI held a joint hiring event in Dallas based on its analysis that a large number of female veterans live in Texas. According to SRDI officials, SRDI also assists DHS components with their evaluations of their human capital policies, procedures, or practices that may represent EEO barriers, such as awards, promotions, and career development. For example, in fiscal year 2016, SRDI analyzed the representation of the DHS Senior Executive Service Candidate Development Program applicant pool by various ethnic and racial groups, and by actual selectee participation. When it found that the representation rate of women decreased from 32.5 percent in the application stage to 23.4 percent in the selection stage, SRDI stated that the results, among other things, triggered the need for further analysis. Two cohorts later in 2018, the representation rate of women increased from 23.3 percent to 41.4 percent in the selection/participant stage. Further, CRCL and SRDI officials said they collaborate on a number of EEO activities to identify and address EEO barriers. For example, SRDI works together with CRCL to provide input for completing MD-715 report sections that address human capital-related EEO barriers. In addition, SRDI and CRCL worked together to conduct a barrier analysis of Hispanic employee representation. DHS components told us that its collaboration practices are generally working well and provided examples. In our interviews of nine DHS components, they told us they are generally satisfied that DHS has: clearly defined its short- and long-term outcomes, bridged the organizational cultures of participating agencies, clearly defined roles and responsibilities for participating agencies, included all relevant DHS participants when identifying and addressing EEO barriers, funded and staffed its collaborative mechanisms, such as monthly EEO council meetings, and documented its agreements on how participating agencies will be collaborating in identifying and addressing barriers. All nine components told us that CRCL regularly meets with them and provides guidance on identifying and addressing barriers. Four components specifically stated that CRCL provided assistance for reviewing and processing EEO data. For example, USCIS officials said that CRCL’s statistician provided direction on analyzing workforce data when conducting barrier analysis. Components also said that they find the training and technical assistance provided by CRCL helpful, and specifically commented that DHS’s EEO and Diversity Training Conferences have helped improve their barrier analyses. While DHS components are generally satisfied with DHS’s collaboration practices, some components provided examples of collaboration practices that could be improved. Three components—CBP, the U.S. Secret Service (Secret Service), and USCIS—told us that collaboration on funding or staffing efforts could be improved. For example, USCIS officials said CRCL lacks sufficient staffing to provide needed training, tools, and assistance to components to meet new MD-715 reporting requirements. Three components—CBP, the Transportation Security Administration, and USCIS—cited the lack of written guidance and agreements regarding collaboration between CRCL and components as areas that could be improved. For example, USCIS officials said that its collaborative efforts with CRCL were guided by informal best practices, feedback, and guidance, but having formal written guidance and agreements could clarify roles and responsibilities for identifying and addressing component EEO triggers and barriers. CRCL officials and component EEO officials stated that component EEO directors report directly to their respective component heads and not to CRCL. While CRCL requires components to meet to discuss midyear updates on their EEO efforts, CRCL officials explained that DHS components are responsible for developing, certifying, and submitting their own MD-715 reports to EEOC. They also said that if EEOC finds areas of noncompliance in DHS components’ EEO programs, EEOC requires DHS components to submit their compliance reports directly to EEOC. In fiscal year 2017, EEOC provided notice to six out of eight DHS components for having areas of noncompliance in their EEO programs. For five out of six DHS components, EEOC required components to establish plans to correct deficiencies, submit compliance reports explaining the agency’s progress in correcting these deficiencies, and showing meaningful progress in implementing its plans within 6 months. We found that three out of five DHS components—CBP, FEMA, and USCIS—did not submit timely compliance reports in response to EEOC’s feedback letters. Due to an administrative oversight, CBP officials explained that the component did not submit a compliance report that was due in February 2018 until we asked about it during our review. Although CBP submitted the report in March 2019, the report did not include plans to correct three out of seven areas of noncompliance. As of July 2019, CBP has taken steps to address the areas of noncompliance but has not yet responded to EEOC. As a result, CBP remains at risk of EEOC initiating the noncompliance process against it. FEMA also did not submit a compliance report that was due in February 2018 until we asked about it during our review. In June 2019, FEMA responded to EEOC’s feedback letter and included plans to correct three areas of noncompliance. FEMA’s response stated that the component would provide another update on its plans to correct these areas to EEOC in October 2019. FEMA’s response also stated that it would update its actions on 12 other areas of noncompliance in its fiscal year 2018 MD- 715 report. EEOC guidance states that an agency’s EEO Director ultimately is responsible for ensuring equal opportunity throughout the entire agency. In addition, Standards for Internal Controls in the Federal Government states that management should implement control activities through policies. According to CRCL officials, CRCL does not have policies and procedures to ensure that components have addressed EEOC’s feedback letters completely and timely. CRCL officials said CRCL does not have the authority to ensure components’ responses completely and timely address EEOC’s feedback letters. They explained that components interact directly with EEOC and are not required to discuss EEOC’s feedback with CRCL. CRCL officials further said that components may address EEOC’s feedback in their MD-715 reports instead of sending compliance reports to EEOC. For example, in response to the EEOC’s 2017 feedback letter, in its MD-715 report for fiscal year 2017, the U.S. Coast Guard discussed ways to assist DHS with improving its issuance of Final Agency Decisions. CRCL reported in its MD-715 reports from fiscal years 2015 through 2017 that it had authority for components’ EEO programs. A DHS delegation of authority order states that CRCL can recommend EEO program improvements to the component head before he or she responds to EEOC’s feedback letters. In addition, CRCL could use its existing practices to discuss EEOC’s feedback letter with components, such as midyear update meetings and monthly council meetings. However, CRCL officials stated they did not meet to discuss EEOC’s feedback letters with components in 2018. EEOC officials told us they send component feedback letters to both the component and CRCL, and invite CRCL officials to participate in component site visits. They also explained that DHS could be found noncompliant if a component’s EEO program does not comply with EEOC guidance. Developing policies and procedures for responding completely and timely to EEOC’s feedback letters may help the department comply with EEOC guidance. While DHS officials told us that ensuring DHS components’ compliance with MD-715 guidance is EEOC’s responsibility, EEOC officials explained that DHS’s responsibility equaled the responsibility that EEOC has to ensure DHS components’ compliance with MD-715 guidance. In addition, MD-715 guidance states that federal agencies, such as DHS, have the primary responsibility to ensure nondiscrimination in employment. Our prior work has found that an agency can benefit from periodically evaluating its organizational structure. Additionally, Standards for Internal Control in the Federal Government states that agency management should establish an organizational structure to achieve the agency’s objectives. According to these standards, an effective management practice for attaining this outcome includes periodically evaluating the organizational structure to ensure that it meets its objectives. As we previously discussed, EEOC found areas of noncompliance in the EEO programs of six out of eight DHS components, and two of the six components did not have plans to correct all of the areas of noncompliance until we asked about them during our review. While CRCL officials told us that they lack authority to certify that components’ MD-715 reports comply with MD-715 guidance, EEOC guidance states that an agency’s EEO Director ultimately is responsible for ensuring equal opportunity throughout the entire agency. EEOC guidance allows DHS components to report to either the Deputy Officer for EEO and Diversity or the Secretary of Homeland Security. However, DHS has not taken steps—in consultation with EEOC and other agencies as relevant—to analyze options to address EEO program management weaknesses, such as analyzing alternatives for granting additional authorities to the Deputy Officer for EEO and Diversity to ensure DHS components comply with MD-715 guidance, and assessing benefits and trade-offs of each alternative. In the absence of these steps, DHS may not be positioned to effectively manage its EEO program. As the third largest U.S. government department, the challenges DHS has faced to fully implement effective EEO programs may result in widespread negative consequences, including monetary expenses borne by the agency in connection with workplace disputes and decreased morale and productivity resulting from the ineffective and inefficient use of human capital resources. MD-715 requires DHS and its components to report annually on the status of their EEO activities and include plans that set forth steps they will take to correct deficiencies or improve EEO efforts. From fiscal years 2014 through 2017, DHS and its components have reported deficiencies in their EEO programs and identified EEO barriers in their workforces. We found areas for improvement in DHS and its components’ EEO programs that could help ensure success and compliance with MD-715. Specifically, DHS does not have complete performance metrics for the department’s EEO program, including a mechanism for tracking progress towards eliminating barriers. Developing performance metrics for the department’s EEO program could help improve progress in eliminating identified EEO barriers. In addition, DHS and its components reported that they lack action plans for addressing deficiencies in their MD-715 reports. Developing policies and procedures could help DHS component EEO Directors correct deficiencies in their EEO programs. DHS and its components also reported that areas of their EEO programs do not have sufficient staffing to successfully implement EEO activities. Developing formal staffing models could help DHS and its components better assess their resource needs to correct their deficiencies and eliminate their barriers. Further, from fiscal years 2014 through 2017, EEOC found areas of noncompliance in DHS and its component EEO programs. Without developing policies and procedures for responding completely and timely to EEOC’s feedback letters, DHS components may not correct areas of noncompliance and remain at risk of financial penalties and lost employee potential. Finally, DHS has not taken steps to address the key EEO program management weaknesses. Analyzing options for granting additional authorities to the Deputy Officer for EEO and Diversity can help position DHS to ensure its components are complying with MD-715 guidance. The commitment of DHS’s leadership is essential to successfully addressing these issues. By focusing leadership attention on developing performance metrics, policies and procedures, and staffing models, DHS and its components can help improve their EEO programs by making progress towards eliminating barriers, obtaining sufficient staffing, and addressing areas of noncompliance. We are making the following six recommendations to DHS: 1. The Secretary of Homeland Security should develop performance metrics for the department’s EEO program including a mechanism for tracking progress towards eliminating barriers. (Recommendation 1) 2. DHS component EEO Directors, in consultation with the Deputy Officer for EEO and Diversity, should develop policies and procedures to help ensure that their component EEO programs have action plans for addressing deficiencies in their MD-715 reports. (Recommendation 2) 3. The Deputy Officer for EEO and Diversity should develop a formal staffing model for its EEO program. (Recommendation 3) 4. DHS component EEO Directors, in collaboration with the Deputy Officer for EEO and Diversity, should develop component formal staffing models. (Recommendation 4) 5. The Deputy Officer for EEO and Diversity should develop policies and procedures for responding in a complete and timely manner to EEOC’s feedback letters. (Recommendation 5) 6. The Secretary of Homeland Security—in consultation with CRCL and EEOC, and other agencies and components, as relevant—should analyze options for granting additional authorities to the Deputy Officer for EEO and Diversity to ensure DHS components comply with MD-715 guidance, including the authority of the Deputy Officer for EEO and Diversity to certify components’ MD-715 reports. (Recommendation 6) We provided a draft of this report to DHS and to EEOC for review and comment. In its official comments, reproduced in appendix I, DHS agreed with all six of our recommendations, and DHS and EEOC provided separate technical comments to the draft of our report, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or jonesy@gao.gov, or Christopher P. Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Yvonne D. Jones, (202) 512-6806 or jonesy@gao.gov. Christopher P. Currie, (404) 679-1875 or curriec@gao.gov. In addition to the contacts named above, Clifton G. Douglas, Jr. (Assistant Director), Luis E. Rodriguez (Analyst-in-Charge), Adam J. Brooks, Carla D. Brown, Andrew Howard, Haley Klosky, and Steven Putansu made key contributions to this report.", "summary": "EEOC's Management Directive 715 requires that, to attract and retain top talent, federal agencies are to identify EEO barriers in their workforces and deficiencies in their EEO programs, execute plans to address them, and report annually to EEOC. In 2009, GAO reported that DHS had opportunities to better identify and address barriers to EEO in its workforce, and made recommendations which DHS has taken action to address. GAO was asked to provide an update on DHS's efforts to identify and address barriers to EEO in its workforce. This report examines the steps DHS has taken to (1) identify and address barriers to EEO in its workforce, (2) identify and address EEO program deficiencies, (3) address areas of noncompliance in its EEO program identified by EEOC, and (4) oversee and support component EEO programs. GAO reviewed DHS's and its components' policies, procedures, practices, and reports for their EEO programs for fiscal years 2014 through 2018, interviewed DHS and its component EEO officials, and assessed DHS employee survey results. GAO also reviewed EEOC's feedback on DHS's and its components' EEO programs, and interviewed EEOC officials. The Department of Homeland Security (DHS) has identified barriers to equal employment opportunity (EEO) and has plans to address them, but lacks performance metrics for tracking its progress towards eliminating these barriers. DHS identified three barriers from fiscal years 2014 through 2017: (1) problems with supervision/management, lack of advancement opportunities, and lack of alternate work schedules, among other things, causing higher-than-expected nonretirement separations for white females and several ethnic and racial groups; (2) the geographic location of jobs, which has contributed to low hiring rates of racial groups in certain major occupations; and (3) the medical and physical requirements of various law enforcement positions, such as the ability to engage in moderate to arduous physical exertion, which limit the eligibility of some applicants with targeted disabilities. While DHS reports some improvements in employee engagement and representation of minorities and women, it does not have complete performance metrics, such as the retention rate of women in law enforcement positions. Implementing performance metrics could help DHS better assess its progress in eliminating barriers. DHS and its components have identified various deficiencies in their EEO programs, but lack policies and procedures for developing action plans and formal staffing models to address some deficiencies. DHS components did not have action plans to address nearly half (179 out of 369) of the deficiencies self-reported by all components from fiscal years 2014 through 2017. For example, in fiscal year 2017, four DHS components did not have action plans to ensure that their EEO directors report directly to their agency heads. Developing policies and procedures to help ensure components' EEO programs have action plans for addressing deficiencies could help DHS components better comply with Equal Employment Opportunity Commission (EEOC) requirements. Developing and utilizing formal staffing models for their EEO programs could help DHS and its components to better identify, request, and obtain the staff they need. For example, DHS and its components reported that staffing challenges contributed to some of their EEO program deficiencies, and acknowledged they lack formal models to use their existing staffing to address the deficiencies. DHS has plans to address the nine areas of noncompliance in its EEO program identified by EEOC. For example, in its most recent review of DHS compliance with EEOC requirements, EEOC identified that DHS did not provide complete demographic data on new hires and promotions in its report to EEOC in fiscal year 2016. DHS officials told us that the department plans to report the data by collecting complete data from DHS components in fiscal year 2019. DHS's EEO and human capital offices assist and support DHS components in identifying and addressing EEO barriers. However, the EEO office lacks policies and procedures to ensure components respond timely and completely to areas of noncompliance identified in EEOC feedback letters. Additionally, DHS EEO officials said they lack authority to ensure components' compliance with EEOC requirements. Without addressing these issues, DHS may not be effectively positioned to manage its EEO program. GAO is making six recommendations, including: develop performance metrics for the department's EEO program; develop DHS and component formal staffing models; and analyze options for granting additional authorities to the most senior official for EEO and Diversity. DHS concurred with our six recommendations and described actions the department plans to take to address them.", "document_type": "gao"}
{"report": "This section provides an overview of the Hanford Site, including DOE’s progress cleaning up the site, and DOE’s requirements and organizational structure for managing and overseeing cleanup and S&M activities at the site. Located in southeastern Washington State, the Hanford Site is one of the most contaminated nuclear waste sites in North America. The site covers 586 square miles upriver from the cities of Richland, Pasco, and Kennewick. The Columbia River flows through about 50 miles of the site. The River Corridor and Central Plateau represent the two main geographic areas for cleanup work. See Figure 1 for a map of the Hanford Site. DOE’s primary goal for cleaning up Hanford is to protect the Columbia River from contamination now and in the future and to restore groundwater. Since cleanup began in 1989, DOE has made progress towards these goals, including remediating 1,342 of 2,032 waste sites, demolishing 889 of 1,715 excess facilities, removing 18.5 million tons of contaminated soil and debris from areas along the Columbia River, and treating 20 billion gallons of contaminated groundwater. DOE’s most recent schedule estimate for completing cleanup of the Hanford Site is 2078, although final decisions for many cleanup actions have not yet been made. RL’s current overarching set of near-term cleanup goals and priorities—outlined in its 2020 Vision—include initiating the transfer of radioactive sludge from the K basin, cleaning up highly contaminated soils underneath the 324 building, and completing demolition of the Plutonium Finishing Plant, which is among Hanford’s most contaminated nuclear facilities. Table 1 includes a list and summary descriptions for each of the contaminated excess facilities we selected for our review. A more detailed discussion of the scope for our review is presented in appendix I. The objectives for conducting S&M of contaminated excess facilities are to ensure adequate containment of any contaminants left in place; provide physical safety and access controls; and maintain the facility in a manner that will minimize risk to human health and the environment. S&M requirements are derived primarily from nuclear facility safety regulations and DOE orders concerning occupational safety, environmental protection, security, and emergency response planning. DOE orders also require that nuclear facility maintenance plans address aging degradation and obsolescence and that surveillance inspections be conducted to detect malfunction and deterioration and determine whether the structural integrity of contaminated excess facilities is threatened. Under the TPA Action Plan, DOE has established an S&M plan for each of the key excess facilities. The S&M plan identifies the facility and associated structures covered by the plan and the specific inspection activities and frequencies to be conducted. For the other excess facilities, S&M requirements are established through provisions of the cleanup contract which require that the contractor perform the S&M activities necessary to maintain them in a safe and compliant condition. Due to the wide variation in types of contaminated excess facilities and associated hazards and risks, RL uses a graded approach that allows for differences from facility to facility regarding the frequency and extent of inspections and associated structural integrity engineering evaluations. Structural integrity engineering evaluations are conducted to determine the adequacy, structural integrity, and soundness of structures and their components. Inspections are conducted using a procedural checklist comprising a list of functional areas from the facility’s inspection plans or procedures, which personnel performing inspections are to evaluate. Inspection checklists can include, among other things, structural integrity (an integral part of excess facility inspections), animal and water intrusion, electrical hazards, and ground subsidence. The S&M plans for contaminated excess facilities require interior walk-through inspections generally on an annual basis but that can vary depending on the facility. Typically, these inspections follow a designated path intended to represent conditions that might be present in areas of the facility that are not visually inspected. In addition, a qualified structural engineer conducts an inspection of the roof integrity—recognized as the most likely risk of failure for the contaminated excess facilities—and other associated structures at those facilities. The frequency, extent of future inspections, and recommendations resulting from these periodic inspections are to be documented by the structural engineer. The program offices at DOE headquarters, RL, and Hanford contractors have overlapping roles and responsibilities for managing and overseeing the cleanup and S&M of Hanford excess facilities. These include: Office of Environmental Management: DOE established EM in 1989 to address the environmental legacy of 50 years of nuclear weapons production and government-sponsored nuclear energy research across the country. EM is responsible for the cleanup of large amounts of radioactive wastes, spent nuclear fuel and nuclear material, contaminated soil and groundwater, and the decommissioning and demolition of contaminated excess facilities at various sites. EM offices involved with oversight of contaminated excess facilities cleanup and S&M activities include: Field Operations Oversight/Chief of Nuclear Safety Office. This office has responsibility for strengthening federal oversight of EM’s cleanup mission, including maintaining operational awareness of field office sites’ operations oversight and implementation of nuclear safety requirements, including requirements for S&M. The Standards and Quality Assurance Office. This office assists with headquarters review of deactivation and decommissioning project planning documents, configuration management and controls, and S&M programs. Office of Enterprise Assessments. This independent office, which reports directly to the Office of the Secretary, is responsible for implementing DOE’s Independent Oversight Program for safety and security in accordance with various DOE policies and orders. Through this program, the office conducts appraisals of the adequacy of DOE policy and requirements and the effectiveness of DOE and contractor line management performance in safety and security. The Office of Environment, Safety, and Health Assessments. This office is responsible for conducting assessments to provide information on programs and performance in protecting DOE workers, the public, and environment from hazards present at DOE sites and operations. It also conducts special reviews and studies of safety and emergency management topics and activities where warranted based on circumstances or performance or as directed by DOE management. Hanford Site. RL is responsible for managing and overseeing non- tank waste cleanup activities at Hanford—including S&M of excess facilities—in the Central Plateau area and for completion of some remaining cleanup work in the River Corridor. RL management and oversight includes verification that work is performed in a safe, secure, and quality manner that protects the public, the worker, and the environment and complies with contractual requirements. Project and Facilities Division. This division is responsible for managing and overseeing the cleanup and S&M of Hanford’s excess facilities. Operations Oversight Division. This division has primary responsibility for day-to-day oversight to ensure cleanup work is performed in compliance with requirements for safety, quality assurance, and quality control. This includes ensuring that S&M activities follow approved plans and procedures and that the contractor corrects any deficiencies identified during facility inspections. Site Stewardship Division. This division manages the Long Term Stewardship Program that includes overseeing S&M of the six cocooned reactors. Cleanup Contractor. Private firms under contract to DOE perform the cleanup and S&M work at Hanford. Central Plateau Cleanup. Since 2008, cleanup and S&M of most of the contaminated excess facilities discussed in this report have been performed under the Plateau Remediation Contract by C2HM HILL Plateau Remediation Company. The S&M activities for excess facilities, including how often and what parts of the facility are inspected, are determined by the contractor as necessary to meet contract requirements. Mission Support. Mission Support Alliance is the contractor for the Long Term Stewardship Program and is responsible for ongoing S&M activities for the six cocooned reactors; these activities are expected to last for at least 75 years. DOE has taken some actions to evaluate the causes of the PUREX tunnel 1 collapse, but has not determined the programmatic causes that contributed to the tunnel collapse, such as by completing an accident investigation or a root cause analysis. In addition, DOE headquarters’ recommendations to improve S&M of contaminated excess facilities and the availability of information on the condition of at-risk areas within these facilities have not been fully implemented. RL has taken some actions to evaluate the physical causes that contributed to the PUREX Tunnel 1 collapse, but has not determined the programmatic causes that led to the collapse, such as by completing an accident investigation or a root cause analysis, among other things. Specifically, after the collapse, RL took several actions to comply with a 2017 Washington State Department of Ecology Administrative Order. In this order, the Washington State Department of Ecology determined that RL and the Hanford cleanup contractor were not operating and maintaining the PUREX Tunnel 1 to achieve compliance with the site’s hazardous waste permit and failed, among other things, to keep the operation of the tunnel undisturbed until closure of the facility. The Administrative Order required RL to take several corrective actions to address violations outlined in the Administrative Order, including determining the cause of the PUREX Tunnel 1 collapse. To fulfill the 2017 Administrative Order corrective action, the cleanup contractor performed an engineering evaluation to determine the structural conditions that led to the collapse of PUREX Tunnel 1. However, the contractor noted in the evaluation that due to the risks of exposure to high radiation levels and urgency to seal the collapsed area, there was insufficient information available to determine the causes of the collapse. Instead, the evaluation identified three potential causes of the collapse, with the most likely cause being deterioration and decay of the tunnel’s timber structure. The state accepted these findings from the engineering evaluation as satisfying the requirements in the Administrative Order corrective action that RL identify the causes of the collapse. Notably, the 2017 structural engineering evaluation of Tunnel 1 conducted after the tunnel collapse did not include a root cause analysis to determine the underlying programmatic causes that contributed to DOE not performing previously recommended structural assessments or detecting through regular S&M activity the imminent collapse of PUREX Tunnel 1 collapse. DOE had been aware of concerns with the structural integrity of Tunnel 1 since the 1970s. These concerns lead to the completion of structural assessments in the late 1970s, early 1980’s, and in 1991, when it was recommended that the tunnel be reassessed again in 10 years. Due to elevated risk of contamination and radiation exposure to inspectors, subsequent structural integrity assessments were completed using existing information from prior evaluations, including testing of tunnel structural material, instead of collecting updated information through physical inspections to determine if the PUREX tunnels were structurally sound for continued use, according to RL officials. Figure 2 illustrates the timeline of events related to the tunnels, showing that while the structural integrity of Tunnel 1 was raised several times over the last 40 years and it was recommended in 1991 to assess the tunnel again by 2001, an assessment did not occur until after the May 2017 PUREX Tunnel 1 collapse as part of the corrective actions required by the state. DOE’s order on accident investigations contains requirements to initiate an investigation into both the individual and organizational (programmatic) root and contributing causes of events resulting in, but not limited to, a fatality of an employee or member of the public or serious injury requiring hospitalization; loss of control of radioactive material or environmental release of hazardous material; or at least $2.5 million in damage to property or in costs for cleaning, decontaminating, renovating, replacing or rehabilitating. According to RL officials, RL did not initiate such an investigation into programmatic causes because management concluded that the PUREX Tunnel 1 collapse did not reach these threshold requirements. However, according to RL officials’ written responses to our questions about incident, the costs of responding to the PUREX Tunnel 1 collapse and stabilizing the tunnel exceeded $10 million. DOE Order 232.2A, Occurrence Reporting and Processing of Operations Information, also requires the investigation, categorization, and analysis of reportable occurrences by facility representatives and contractors using a graded approach in accordance with locally approved procedures for implementing the requirements of this order. For an occurrence such as the May 2017 PUREX tunnel collapse, which constituted noncompliance with regulatory requirements that created the potential for actual harm, DOE’s order and related guidance indicates that a causal analysis should have been performed to identify the root causes, including the programmatic causal factor or factors that, if corrected, would prevent similar future occurrences. According to the cleanup contractor’s condition report on the PUREX tunnel collapse, the contractor initially classified the incident as a significant event because it was categorized as an operational emergency and significant by default. According to this report, under the contractor’s reporting procedures, such a classification requires the performance of a root cause analysis to determine the causes and corrective actions with the intent of preventing recurrence. The contractor’s condition report related to the incident notes that RL waived the performance of a root cause analysis in favor of a less rigorous apparent cause analysis to determine the structural factors that led to the collapse of PUREX Tunnel 1. According to a written explanation provided to us by RL management, while the tunnel collapse was due to structural degradation, RL’s first priority was stabilizing the tunnel to mitigate the potential for further collapse, and a programmatic root cause analysis to determine the cause was not warranted. In this written response, RL did not provide any explanation for why a programmatic root cause analysis was not warranted. In an email, RL’s Operations and Oversight Division facility representative granted the cleanup contractor’s request for a waiver from conducting a root cause analysis and concurred with their assertion that an apparent cause analysis was more appropriate. Based on this direction, a root cause analysis was not performed. A root cause analysis, performed by either DOE headquarters or RL in accordance with the requirements of DOE’s orders on accident investigations and occurrence reporting, would have included an assessment of the underlying programmatic factors that contributed to the collapse of PUREX Tunnel 1. For example, a root cause analysis would determine why PUREX facility inspections that only include visual observations of the surface areas around the tunnels were insufficient in identifying the likelihood of the imminent collapse of PUREX Tunnel 1; why a recommendation made in 1991 for an engineering evaluation to be completed by 2001 to determine if the tunnel was still structurally sound for continued use was not completed; or why RL did not make stabilization or cleanup of the tunnel a higher priority. By conducting a root cause analysis to determine any programmatic weaknesses that contributed to the collapse of PUREX Tunnel 1, and taking action to address any identified weaknesses, DOE would have greater assurance that another, similar event will not take place at Hanford. In June 2017, shortly after the PUREX Tunnel 1 collapse, EM initiated an Extent of Condition Review to investigate program weaknesses and risks in regard to contaminated excess facilities at three DOE sites, including Hanford. Although EM’s 2017 Extent of Condition Review concluded that, overall, the S&M processes for excess facilities were adequate in mitigating risks, EM’s review identified some weaknesses and made four recommendations to improve the S&M of contaminated excess facilities and availability of information on these facilities’ condition. Specifically, two of these four recommendations addressed weaknesses in inspections of facilities and improving information about the condition of excess facilities: A comprehensive review should be conducted to identify high-risk areas within excess facilities where inspections have not been conducted for over 5 years. The results of the review should be used to inform the risk management process used to prioritize actions and projects. For excess facilities for which limited areas may be used for ongoing operations or storage of nuclear materials, the S&M of the unused areas should be reviewed to assure long-term integrity and stability that is comparable to facilities that are excess. RL has not fully implemented these two recommendations. RL has taken some actions, including commissioning an engineering team to evaluate the structural integrity of some facilities similar to the PUREX tunnels that may pose a future threat of collapse. However, this evaluation of the structural integrity of Hanford’s contaminated excess facilities was not comprehensive and did not include an evaluation of the structural integrity of all excess facilities of concern that may be at risk of structural failure. For instance, the scope of the evaluation was focused on 27 underground waste storage structures in the Central Plateau, such as cribs, tanks and trenches, which were constructed prior to PUREX Tunnel 1. In addition, this evaluation was largely based on old data and did not include any physical or non-physical inspection and testing to verify if a facility or part of a facility needed to be stabilized or prioritized for cleanup, according to RL officials. In addition, although recommended in EM’s 2017 Extent of Condition Review, to date, RL has not taken action to direct the cleanup contractor to carry out comprehensive inspections at all contaminated excess facilities, and there are areas of some facilities that still have not been entered, either physically or by remote means, to conduct internal inspections. RL officials told us that they generally agree that inspections of aging facilities should include evaluations of their structural integrity. According to these officials, there have been ongoing discussions about such inspections, including how often and in what areas to conduct them. Officials said these decisions would need to be determined on a case-by- case basis depending on the safety consequences of potential incidents. They also stated that RL has prioritized removing hazards to reduce potential threats to human health and the environment to reduce future surveillance and maintenance costs and preparing the canyon areas and other facilities for final cleanup. According to EM headquarters officials, the 2017 Extent of Condition Review recommendations were intended to be considered as opportunities for improvement which site management could incorporate as deemed appropriate. EM officials explained that there is no requirement for sites to take action to implement the review recommendations or track their progress. However, by not taking actions to implement the Extent of Condition Review recommendations, RL will continue to lack information about the condition of high-risk areas within contaminated excess facilities where inspections have not been conducted for several years and will miss opportunities to identify and address any deteriorating conditions that could lead to the collapse of another contaminated excess facility. The Hanford contractor is generally conducting surveillance inspections of most contaminated excess facilities as required. However, EM’s 2017 Extent of Condition Review and our review found that the cleanup contractor did not conduct comprehensive inspections at all contaminated excess facilities and that there are areas of some facilities that personnel infrequently or never enter, physically or by remote means, to conduct interior inspections. In addition, although EM’s 2017 Extent of Condition Review team noted that they observed examples where appropriate S&M activities were taking place at contaminated excess facilities, the team also acknowledged that such activities do not assure the EM sites’ S&M programs are adequate to prevent mishaps, as evidenced by the collapse of PUREX tunnel. Further, DOE headquarters offices responsible for the evaluation of DOE site activities have not conducted any specific assessments or audits focusing on management and oversight of Hanford S&M activities since 2013. According to EM’s 2017 Extent of Condition Review and our review of inspection reports at selected facilities, routine surveillance inspections of Hanford’s contaminated excess facilities are being conducted and the EM review concluded that Hanford’s surveillance inspections were generally adequate. However, this same EM review, as well as our review, identified weaknesses in Hanford’s inspection program. DOE orders require sites to clearly address aging degradation and obsolescence and to conduct surveillance inspections at contaminated excess facilities to detect malfunction and deterioration and determine whether the structural integrity of contaminated excess facilities is threatened. Once DOE determines that a facility is excess to mission needs, the disposition phase of a contaminated excess facility’s life cycle usually includes deactivation, decommissioning, and S&M activities, followed by decontamination and demolition. According to RL officials, a graded approach—taking into account the risks posed at each contaminated excess facility— can be used to tailor S&M activities, including the frequency of facility inspections. In addition, S&M plans and procedures are prepared by DOE and implemented by the contractor, who determines the frequencies and areas of contaminated excess facilities included in surveillance inspections. EM’s 2017 Extent of Condition Review found that at three EM sites, including Hanford, contaminated excess facilities surveillance inspections were adequate and overall ensured that the S&M programs were mitigating risks. Additionally, the review found that the sites were giving appropriate attention to roof integrity through the S&M process. Roof structural integrity is a key concern at contaminated excess facilities, as the roof serves as protection against spread of contamination and represents the most likely failure risk and safety risk for workers. Further, in our review of selected contaminated excess facilities, we found that the Hanford cleanup contractor has conducted annual surveillance inspections of most of these facilities and has taken action to ensure the structural integrity of some contaminated excess facilities. For example, RL’s responses to our questionnaire indicated that for 16 of the 18 contaminated excess facilities we selected for our review, the contractor conducts interior inspections of structural integrity on a periodic basis. In addition, we found that between 2008 and 2018, the contractor annually inspected three of the four contaminated excess facilities we selected for our in-depth reviews. However, RL responses to our questionnaire revealed concerns with completeness of structural integrity evaluations and the structural integrity of some facilities. For five of 18 facilities, RL officials identified structural integrity or degradation which could lead to the potential release of hazardous or nuclear materials, such as the May 2017 partial collapse of PUREX Tunnel 1, as a concern. RL responses also indicated that engineering analyses to evaluate structural integrity had been conducted for 13 of the 18 facilities; however, at 10 of these facilities some areas were not included in the evaluation due to concerns about worker safety from radiological or other hazards. Further, EM’s 2017 Extent of Condition Review, other recent DOE reports, and our review of inspection reports for selected contaminated excess facilities found several instances in which the cleanup contractor did not conduct comprehensive surveillance inspections at all excess contaminated facilities, including infrequently or never entering portions of some facilities, either physically or by remote means, to conduct interior structural integrity evaluations. REDOX. According to the 2015 Canyon Risk Mitigation Plan, the REDOX canyon is not accessed during routine S&M activities. This report also notes that the canyon deck area is expected to be highly contaminated, is not inspected, has not been entered in more than 50 years, and structural conditions are unknown. The canyon deck is located in the central portion of the canyon building and is isolated from other areas of the facility by thick reinforced concrete walls and floors. It is located above the facility process cells that were used to extract plutonium. According to RL officials, these process cells and other parts of the main canyon building are not accessed during routine walkthrough inspections due to high levels of radioactive contamination. Furthermore, in the contractor’s 2016 annual inspection of the REDOX facility complex, the contractor did not evaluate three annexes of the canyon facility for structural integrity, according to RL’s response to our questionnaire. According to RL officials, the contractor did not carry out these evaluations of the annexes because RL plans to complete their final cleanup in the near term. However, according to a 2016 DOE planning document, the schedule for conducting the cleanup of the annexes is unknown, and RL officials told us it may be several more years before cleanup begins. Because these annexes are not inspected for structural integrity, RL and the cleanup contractor may not have sufficient information regarding their condition for planning purposes, such as assessing if immediate maintenance is required to stabilize a structure or prioritizing an annex for immediate cleanup. In addition, according to a 2012 DOE report, because the canyon was not deactivated after shutdown in the 1960s, information is very limited and there is a significant level of uncertainty about the conditions inside the building. According to the EM’s 2017 Extent of Condition Review, despite ongoing S&M activities, if facility deterioration continues and is left unaddressed, the condition of the facility could present a threat to human health and the environment, as well as increase the costs of S&M in the near term. PUREX. According to EM’s 2017 Extent of Condition Review, parts of the main PUREX facility are not physically inspected, including the canyon deck. The canyon deck is in the central portion of the main canyon building and is isolated from the surrounding areas of the facility by thick, reinforced concrete walls and floors and has not been entered in more than 10 years, according to the Hanford cleanup contractor’s 2015 Canyon Risk Mitigation Plan report. According to this report, conditions within this space are unknown, and high contamination levels are expected. Due to lack of information and concerns about this area, the 2015 Canyon Risk Mitigation Plan recommended—for data-gathering and planning purposes— inspecting this area either physically or remotely, if physical entry is not possible due to high levels of radiation. This report also stated that future cleanup work could not be initiated in this area without sufficient information related to the condition of the canyon deck. In addition, a 2019 engineering evaluation of the facility determined that degradation may not be fully addressed by S&M activities and the risk of release of hazardous substances will increase as degradation continues or goes undetected. Figure 3 shows the main PUREX plant and auxiliary facilities. 216-Z-9 Crib. According to the EM’s 2017 Extent of Condition Review, due to the highly contaminated nature of 216-Z-9 Crib, inspections of this facility are limited to external surveillance of the roof and looking down the facility stairwell to the trench area of the crib. However, a 2006 inspection of the interior of the crib utilized a remote controlled device to inspect and determine that the structural integrity of the facility’s roof was suspect. This inspection recommended that the roof be inspected for structural integrity every 5 years; however RL did not direct the contractor to inspect the facility until 2016. Furthermore, according to RL officials, when the facility was inspected in 2016 and then again in 2018, the inspections did not include an engineering evaluation or use of non-physical engineering or robotic tools to inspect the structural integrity of the roof, as was done in 2006, to determine if the facility was safe for continued use. Despite the lack of an engineering evaluation or interior inspection of the roof, the 2016 and 2018 inspection reports gave the facility a passing grade for structural integrity—raising questions about both the basis and reliability of this assessment. RL officials told us they did not instruct the contractor to conduct such an evaluation because recent visual surveillance inspections of the outside of the crib roof did not indicate that structural failure was imminent. However, in its January 2019 structural integrity assessment of contaminated excess facilities at risk of collapse, the contractor reported that this facility was among 11 facilities needing further evaluation. Plutonium Finishing Plant 241-Z-361 Settling Tank. According to RL’s response to our January 2019 questionnaire, the interior of the Plutonium Finishing Plant 241-Z-361 Settling Tank is not inspected. RL’s response noted that although there are concerns regarding the structural integrity of the facility, the facility is safe for continued use. However, RL’s response is not consistent with prior studies on the condition of the tank. To support the questionnaire response, RL referred to the 2018 Documented Safety Analysis and a 1997 Structural Integrity Assessment for Plutonium Finishing Plant 241-Z- 361 Settling Tank. The 2018 Documented Safety Analysis concludes that the tank is in a structurally degraded condition but is not considered at risk of imminent failure. However, the 1997 Structural Integrity Assessment that DOE used to support the conclusion in its Documented Safety Analysis determined it was not possible to accurately assess the condition of concrete in the facility and there were uncertainties associated with the strength of its structural steel. The 1997 report also concluded that deteriorating conditions of the facility could lead to the leakage of radioactive waste material, further accelerating the degradation through corrosion and conditions that could result in the collapse of the tank. Notably, a subsequent 1999 video inspection revealed cracking in the interior roof, dissolving of the interior steel liner, and deterioration of the concrete sidewall of the tank. Despite these documented concerns about the structural integrity of the facility, RL officials that we spoke with could not provide a specific reason for why the interior of this facility has not been inspected. Most recently, a structural integrity initial assessment performed for the contractor in January 2019 identified the Plutonium Finishing Plant 241-Z-361 Settling Tank as the top priority among 11 contaminated excess facilities needing further evaluation to determine if the facility is structurally sound for continued use. This report stated that the facility is currently in a structurally degraded condition, with severe deterioration of the construction materials supporting the structure. 224B Concentration Facility. This facility is contaminated from past operations and parts of the facility are not physically inspected, according to the 2015 Canyon Risk Mitigation report. In addition, according to a 2015 RL briefing report, the facility’s roof is aging and will likely require replacement within 5 years. According to RL officials, the roof of this facility has not been replaced, and according to RL’s response to our questionnaire, no significant maintenance or structural work has been conducted since 2008 and none is needed or planned based upon the current condition of the facility. However, RL’s response to our questionnaire indicates that RL has not conducted a structural integrity engineering evaluation of the facility to support this conclusion. According to RL officials, they are currently in the process of developing a plan to complete decommissioning and decontamination of the facility. Under the TPA, the plan is to be submitted by the end of September 2020. However, RL officials told us that even with regulatory approval of the plan, DOE likely will use additional funding to pursue other near-term cleanup priorities rather than clean up the 224B Concentration Facility. According to EM’s 2017 Extent of Condition Review, other recent DOE reports, and our review of inspection reports for selected contaminated excess facilities, gaps in S&M activities are, in some cases, due to access challenges at the facilities. According to the EM 2017 Extent of Condition Review, not all facility areas are inspected regularly due to difficulty of access or elevated risk of contamination or exposure, or because those areas are in such a degraded condition they are not safe to enter. However, the contractor has demonstrated the capability to use engineering or robotic evaluations to inspect or determine the structural integrity of the facility, or parts of the facility, and verify whether it needs to be stabilized or prioritized for cleanup. For example, such analyses were done at the PUREX tunnels and at the 216-Z-9 Crib, as noted above. Despite this capability, RL management has not directed the cleanup contractor to perform such inspections for some of Hanford’s contaminated excess facilities or parts of facilities. According to RL officials, decisions on regularity and types of inspections and structural evaluations will depend on the known risks associated with the facility. Without directing the contractor to routinely conduct comprehensive inspections to gather crucial information on the condition of contaminated excess facilities, RL cannot ensure that it is meeting all of DOE’s S&M requirements—such as addressing aging degradation and obsolescence of facilities—and preventing other potential events similar to the PUREX tunnel collapse. DOE headquarters offices have conducted some assessments of RL cleanup work but have not conducted any assessments or audits focused on RL’s oversight of the cleanup contractor’s S&M activities since 2013. EM’s Field Operations Oversight/Chief of Nuclear Safety Office and DOE’s Office of Enterprise Assessments are required to conduct independent oversight to the extent necessary to evaluate the effectiveness of DOE field office oversight of contractor activities, including activities needed to maintain contaminated excess facilities in a safe and compliant condition pending their final cleanup. We reviewed 21 DOE HQ oversight reports on RL activity from the past 5 years and determined that none of these assessments or audits focused on RL’s management and oversight of the contractor’s S&M activities for contaminated excess facilities. We spoke with DOE officials from two headquarters offices responsible for independent oversight of DOE field offices—the EM Field Operations Oversight/Chief of Nuclear Safety Office and the Office of Enterprise Assessments. Officials with both headquarters offices confirmed that neither office has conducted a specific assessment or audit focusing on RL’s management and oversight of S&M activities for contaminated excess facilities in the last 5 years. Officials with the Office of Enterprise Assessments told us that, given the limited resources available to conduct oversight, they have to prioritize and be selective about the reviews they plan to conduct in a given year, and conducting an in-depth assessment of RL’s oversight of Hanford S&M activity has not been a priority with that office. In December 2018, the office considered whether to conduct a formal assessment of RL oversight of Hanford S&M activity, but decided that such an assessment was not needed. However, the projected overall time in S&M mode underscores the importance that S&M be adequate to maintain facility safety during the final stages of cleanup operations through a seamless transition to the final disposition of the facility to protect human health and the environment. We found that S&M requirements for selected contaminated excess facilities will continue for decades. Specifically, our review of 18 contaminated facilities at Hanford found that many of these facilities were determined to be excess between the 1960s and the late 1980s and transitioned into S&M status at that time. Notably, our review of these facilities shows that several of them do not have planned cleanup completion dates and for those with cleanup completion dates, cleanup is scheduled to be completed between 1 and 6 decades in the future. Table 2 shows the dates for when the 18 contaminated excess facilities transitioned into S&M mode and how long RL will need to continue S&M activities until cleanup is completed. As S&M of Hanford’s contaminated excess facilities is expected to continue for many decades, conducting an effective S&M program is essential to minimize the risks of potential releases of contamination that could harm the environment or human health before cleanup is completed. Notably, RL has not established final cleanup dates for several of the 18 contaminated excess facilities included in our review. DOE, however, has not conducted independent reviews of S&M oversight activity necessary to determine whether weaknesses exist in RL’s management and oversight of the Hanford Site contractor’s S&M activities for these facilities. Without prioritizing and conducting periodic assessments or audits focused on RL’s management and oversight of the Hanford Site contractor’s S&M activities for contaminated excess facilities, DOE does not have assurance that RL is overseeing S&M activity in a way that ensures contaminated excess facilities are being inspected and maintained in a safe and compliant condition pending final cleanup. RL seeks to balance risks with other factors, such as legally enforceable milestones, available budget, and stakeholder interests, to prioritize cleanup activities that support achieving its overarching Hanford Site cleanup goals, according to RL officials and planning documents. While EM has overall responsibility for managing DOE’s cleanup program, including deactivation and demolition of excess facilities, it has delegated prioritization of cleanup activities to the sites through the annual budget process. As part of the process, EM requests sites develop and submit a site-specific Integrated Priority List to EM management. The Integrated Priority List is based on a number of site-specific factors, including regulatory commitments, agreements with EPA and states, and risks to worker safety and the environment. According to RL officials, EM does not provide specific written guidance for the sites to follow in developing their priority lists, other than a list of seven general factors. RL officials told us that more specific guidance is not necessary because site management needs the flexibility in setting and adjusting cleanup priorities to reflect changes in site conditions and other evolving circumstances as they arise. Since 2017, RL and the Hanford cleanup contractor have been using a new site-wide risk-informed tool, known as the Project Evaluation Matrix, to help inform decisions on which cleanup priorities to include in the Integrated Priority List. The matrix is used to produce a prioritized listing of the stabilization, waste removal, and other activities that need to be completed as part of the deactivation and decommissioning of the contaminated excess facilities and their associated buildings, structures, and waste sites. RL and cleanup contractor officials described the matrix as a broad, overarching tool to aid in establishing a qualitative basis by which they can determine and agree on cleanup priorities that are planned to be executed within the next 1 to 5 years. Neither the Washington State Department of Ecology nor the Environmental Protection Agency is directly involved in the development of the rankings in the matrix. The contractor’s guidance document explains that the risk evaluation process used to develop the matrix rankings involves a number of steps. It starts with the data collection phase, during which RL and the contractor collect information on site conditions from a variety of sources, such as historical records, safety assessments, subject matter experts, and S&M activities. This information is then used to develop relative ranking scores for the various cleanup and S&M activities using weighted scores for three criteria: (1) risk reduction; (2) mortgage reduction/cost avoidance; and (3) TPA milestones/regulatory drivers. The initial scores also take into consideration other factors such as potential consequences of failure and overall project lifecycle costs. After developing an initial risk ranking of cleanup projects and activities, the contractor works with RL management to evaluate the initial results and make adjustments as necessary to reflect comments, changes in conditions, or new work scope. The risk rankings are then updated and used by RL to inform decisions on which projects to prioritize in its Integrated Priority List budget submission to EM. As funding decisions are made and cleanup work proceeds, risks are reassessed and the process starts again. RL officials explained that planned cleanup priorities established in the Integrated Priority List can be adjusted as necessary to reflect information learned through S&M activities and changes in site conditions. For example, routine annual S&M inspections at one facility identified concerns with the integrity of the roof. Based on these concerns, a structural analysis was performed by the cleanup contractor, and RL adjusted its priorities for fiscal year 2016 to include replacing the facility’s roof. Similarly, RL may also modify its planned cleanup priorities to reflect changes in site conditions, such as completing cleanup of a facility or taking actions to stabilize a facility pending its final disposition. For example, based on structural evaluations completed after the partial collapse of Tunnel 1 in May 2017, RL elevated interim stabilization of both PUREX tunnels as among its top priorities in fiscal years 2018-2019. The ability of RL management to establish and adjust cleanup priorities depends on the availability of quality information on site conditions that is reliable, complete, and current. One source of information for this process is annual and routine S&M activities for Hanford’s contaminated excess facilities. These activities, such as facility inspections, structural integrity evaluations, and radiological monitoring, help provide management with updated information on potential changes in site conditions that may lead to an adjustment in previously planned priorities. As discussed above, however, both EM’s 2017 Extent of Condition Review and our review found that parts of certain contaminated excess facilities that may be at risk for structural deterioration—such as the REDOX annexes—are not included in the routine surveillance inspections and have not been inspected within the past 5 years, or longer. We also identified instances where structural integrity evaluations for some facilities, such as for the 216-Z-9 crib and the Plutonium Finishing Plant 241-Z-361 Settling Tank, appear to have relied on outdated information and reached determinations seemingly inconsistent with the contractor’s more recent analyses and conclusions. By conducting comprehensive surveillance inspections of Hanford’s contaminated excess facilities, DOE would have greater assurance that RL and the contractor’s process for identifying cleanup priorities reflects the current status of the potential human health and environmental risks present at such facilities. At Hanford, RL has made progress in cleaning up approximately 800 excess facilities, and six major plutonium production reactors are now cocooned and waiting final dispositioning. Despite efforts to mitigate risks and cleanup excess facilities, significant vulnerabilities remain at Hanford due to, among other things, the degrading state of hundreds of contaminated excess facilities still requiring cleanup. Given the pivotal role of the S&M program in ensuring that aging and degrading contaminated excess facilities do not collapse or fail to contain radioactive or hazardous material, it is important that this program is functioning effectively and that any weaknesses are addressed in a timely manner. The partial collapse of PUREX Tunnel 1 was a clear signal that there are flaws in the S&M program at Hanford. By conducting a root cause analysis to determine any programmatic weaknesses that contributed to the causes of the PUREX Tunnel 1 collapse, and taking action to address any identified weaknesses, DOE will have greater assurance that another, similar event will not occur at Hanford. Additionally, the PUREX Tunnel 1 event demonstrates that RL and the cleanup contractor need complete and updated information regarding the condition of aging contaminated excess facilities to determine if facilities should be stabilized to prevent structural failure or prioritized for cleanup. This information can only be acquired by routinely completing comprehensive surveillance inspections, to include, if necessary, engineering evaluations including the use of remote controlled probes. Without directing the contractor to conduct routine and comprehensive inspections to gather crucial information on the condition of contaminated excess facilities, RL cannot ensure that it is meeting all of DOE’s S&M requirements—such as addressing aging degradation and obsolescence of facilities—and preventing other potential events similar to the PUREX tunnel collapse. Furthermore, because DOE headquarters offices have not prioritized and conducted any assessments or audits focused on RL’s oversight of the cleanup contractor’s S&M activities within the past 5 years or since the PUREX Tunnel 1 collapse, they are missing an opportunity to identify and address any Hanford S&M program weaknesses that may have led to the collapse. We are making the following three recommendations to DOE: The Assistant Secretary of DOE’s Office of Environmental Management should direct RL to conduct a root cause analysis to identify any programmatic causes that may have led to the collapse of PUREX Tunnel 1. (Recommendation 1) The Assistant Secretary of DOE’s Office of Environmental Management, while ensuring the protection of DOE workers, the public, and the environment, should ensure that RL directs the Hanford Site cleanup contractor to explore using robotic or other means to routinely complete comprehensive surveillance inspections of contaminated excess facilities to identify aging degradation and obsolescence of facilities and take timely action as warranted. (Recommendation 2) The Secretary of Energy should ensure DOE headquarters offices responsible for the oversight of EM sites’ field offices conduct an assessment of RL’s management and oversight of the Hanford Site contractor’s surveillance and maintenance activity for contaminated excess facilities. Based on the results of this assessment, DOE headquarters offices should consider whether such assessments should be conducted on a periodic basis. (Recommendation 3) We provided a draft of this report to the Secretary of the Department of Energy. In its written comments, reproduced in appendix III, DOE agreed with the report’s findings and concurred with our recommendations. In addition, DOE described ongoing and planned actions to address our recommendations by December 31, 2020. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees; the Secretary of Energy; the Director, Office of Management and Budget; and other interested parties. In addition, the report will be available at no charge on the GAO website at www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. This report reviews issues related to the cleanup, inspection and maintenance of Hanford’s contaminated excess facilities, such as the Plutonium Uranium Extraction Plant (PUREX), and how the Department of Energy (DOE) and the Richland Operations Office (RL) prioritizes and schedules cleanup and ensures that the Hanford Site contractor inspects and maintains these facilities. The objectives of our review were to (1) examine actions DOE has taken to evaluate the causes of the PUREX Tunnel Collapse, 2) examine the extent to which DOE ensures that the contractor’s surveillance and maintenance of Hanford’s contaminated excess facilities meet DOE requirements, and (3) describe how DOE determines the priority ranking and schedule for cleanup of Hanford’s excess facilities. To examine actions DOE has taken to address the PUREX Tunnel Collapse and the extent to which DOE ensures that the contractor’s surveillance and maintenance (S&M) of Hanford’s contaminated excess facilities meets DOE requirements, we reviewed DOE orders, policies, RL procedures, and documents that describe DOE’s S&M requirements. We also obtained and reviewed DOE evaluation reports and assessments of S&M activities and operations at Hanford facilities; these include the Office of Environmental Management’s (EM) 2017 Extent of Condition Review for Excess Facilities and historic S&M assessment reports on PUREX tunnel structural stability. To describe how DOE determines the priority ranking and schedule for Hanford cleanup work of Hanford’s contaminated excess facilities, we reviewed federal environmental regulations, legal agreements, planning documents from DOE and the Hanford cleanup contractor, DOE directives and guidance, and reports by the Consortium for Risk Evaluation with Stakeholder Participation and others on ways to consider risk in making cleanup decisions. These include, but are not limited to, the Tri-Party Agreement (TPA) and associated Action Plan; EM’s Fiscal Year 2020 budget request; RL’s 2015 Vision and 2020Vision, which include high-level cleanup priorities and goals; the Hanford cleanup contractor’s Project Evaluation Matrix and its associated guideline; and RL’s Integrated Priority List. For all objectives, we also interviewed DOE officials with RL, the DOE Office of Inspector General at Hanford, and DOE headquarters offices, including the Office of Enterprise Assessments and EM’s Office of Safety, Security, and Quality Assurance. In addition, we interviewed Hanford cleanup contractors, officials from the Washington State Department of Ecology, and officials from the Defense Nuclear Facilities Safety Board. Due to the large number of Hanford contaminated excess facilities requiring cleanup (approximately 800), we focused our review on 18 contaminated excess facilities. These contaminated excess facilities represent the majority of the excess facilities cleanup effort and include some of the most challenging of the non-tank waste cleanup efforts remaining at Hanford, according to DOE officials. We chose key excess contaminated facilities as identified in the TPA because, among other things, DOE and its regulators identify these facilities in Section 8 of the agreement as presenting sufficient potential environmental concern that coordination of the decommissioning process with cleanup activities under the agreement was deemed necessary. We also selected the five other contaminated excess facilities because DOE identified them as having 1) high risks to the environment, workers, and public safety, 2) high annual S&M costs, and 3) high disposition costs. See Table 1 in the report for summary descriptions of each facility we selected. To gather information about RL’s planning on S&M activities at Hanford and estimated costs for fiscal year 2019, we administered a questionnaire to RL facility representatives responsible for overseeing the cleanup contractor’s implementation of S&M for contaminated excess facilities. For each facility, the representatives were asked whether there was an S&M plan for the facility, when it was developed, and when it was most recently updated. We also asked about the type and frequencies of facility inspections, whether the facility included areas where structural integrity was a concern, if any structural integrity evaluations had been conducted, and whether any significance corrective or preventative maintenance had been performed. We also asked them to explain the facility representative’s role in overseeing that the contractor was conducting S&M activities in accordance with the applicable plan and DOE requirements. A copy of the complete questionnaire is included in appendix II. We conducted two pretests of the questionnaire with RL officials in November and December 2018, and we revised it in response to their comments. During this process, we sought to ensure that (1) the questionnaire questions were clear and unambiguous, (2) terminology was used correctly, (3) the questionnaire did not place an undue burden on respondents, and (4) respondents had sufficient information to answer the questions. For the questionnaire we identified an initial set of 21 contaminated excess facilities based on the following criteria: (1) whether they were a key facility identified by DOE and its regulators in Section 8 of the Tri- Party Agreement Action Plan and (2) whether we considered them to be a contaminated excess facility that poses high risks to the environment, workers, and public safety; (3) whether it has potentially high annual surveillance and maintenance costs; and (4) whether it has high final disposition costs based on information we gathered from DOE. After further correspondence with RL officials, we agreed that three of the contaminated excess facilities on our list could be deleted because they were not in S&M mode, as cleanup was completed for one facility, one was undergoing active cleanup, and the other was in operational status. We sent the questionnaire by email in a password-protected Word document to which respondents could return electronically after marking checkboxes or entering responses into open-answer boxes. We sent the questionnaire with a cover letter to DOE officials on January 10, 2019, with a request to complete and return it by January 31, 2019. By February 25, 2019, we received completed questionnaires for each of the 18 selected contaminated excess facilities. In addition, to provide further context for all objectives, we conducted in- depth reviews regarding S&M of selected Hanford facilities. For these reviews, we selected four high-risk facilities: PUREX, REDOX, the 224B Concentration Facility, and the 216–Z-9 Crib. We used a judgmental (non-probability) sample to select four contaminated excess facilities for in-depth review. These facilities have been identified by DOE, the DOE Office of Inspector General, or the Consortium for Risk Evaluation with Stakeholder Participation as contaminated excess facilities with concerns regarding high risks to the environment, workers, and public safety and risk of potential release of radioactive material and other hazardous materials due to aging degradation and weakening structural integrity. In addition, these contaminated excess facilities are moderate- to high-risk priority facilities for cleanup, according to the contractor’s June 2018 Project Evaluation Matrix, but not scheduled to start cleanup for at least 5 years. For these reviews, we examined DOE documents, including inspection records dating back to the start of fiscal year 2008 through the end of fiscal year 2018 to determine if inspections were occurring, and interviewed RL officials. We conducted this performance audit from March 2018 to January 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the individual named above, Ned Woodward, Assistant Director; Tara Congdon; Justin Fisher; Richard Johnson; Michael Meleady; Peter Ruedel; Sara Sullivan; and Roxanne Sun made key contributions to this report.", "summary": "DOE's Hanford site in Washington State contains thousands of contaminated excess facilities and waste sites that remain to be cleaned up. In May 2017, a partial roof collapse at a waste storage tunnel facility for one of the former plutonium nuclear processing plants raised questions about the S&M of Hanford's excess facilities and how RL prioritizes cleanup of these facilities. GAO was asked to review DOE's cleanup of Hanford's contaminated excess facilities, including how DOE ensures that the Hanford Site contractor inspects and maintains facilities. This report examines, among other things, (1) DOE's actions to evaluate the causes of the PUREX tunnel collapse, and (2) the extent to which DOE ensures that S&M of Hanford's contaminate excess facilities meet DOE requirements. GAO reviewed DOE documents, administered a questionnaire to collect S&M information about 18 selected facilities representing the majority of the Hanford facilities cleanup effort, conducted in-depth reviews of selected Hanford facilities, and interviewed DOE and Hanford cleanup contractor officials. The Department of Energy (DOE) has taken some actions to evaluate the physical causes that contributed to the May 2017 partial collapse of the Plutonium Uranium Extraction (PUREX) Tunnel 1, but has not determined the programmatic causes that led to the collapse, such as by completing an accident investigation or a root cause analysis, among other things. For example, although an engineering evaluation of the tunnels was completed at the request of the State of Washington, Richland Operations Office (RL) officials told GAO an accident investigation was not initiated because the event did not meet threshold requirements in a DOE order that includes, among other things, damages or costs exceeding $2.5 million. However, GAO's analysis shows that the costs of responding to the event and stabilizing the tunnel were about $10 million. At the contractor's request, RL also waived performance of a root cause analysis, which DOE guidance states is typically required for such a significant event, and agreed to a less rigorous analysis of the potential physical causes of the event. By conducting a root cause analysis to determine any programmatic weaknesses that contributed to the collapse of PUREX Tunnel 1, and taking action to address any identified weaknesses, DOE will have greater assurance that another, similar event will not take place. According to a DOE report and GAO's review, although the Hanford contractor is generally conducting routine surveillance inspections of contaminated excess facilities, these inspections have weaknesses and GAO found that DOE has not ensured requirements are fully met. Specifically, DOE orders require that processes be in place to ensure that inspections are conducted to detect deterioration and determine whether the structural integrity of facilities is threatened. A December 2017 DOE report and GAO's review found that the surveillance and maintenance (S&M) inspections at several facilities were not comprehensive and that there are areas of some facilities that personnel infrequently or never enter—physically or by remote means—to conduct inspections. For example, parts of the Reduction-Oxidation Facility have not been entered in more than 50 years and structural conditions are unknown. Without conducting comprehensive inspections, RL cannot ensure that it is meeting all of DOE's S&M requirements, such as addressing aging degradation and obsolescence of some facilities, and preventing other potential events similar to the PUREX tunnel collapse. In addition, GAO's review of oversight reports since 2013 by DOE headquarters offices responsible for evaluating field office operations found that none of these assessments focused on RL's management and oversight of the contractor's S&M activities. DOE's Oversight Policy requires DOE to conduct independent oversight to the extent necessary to evaluate the effectiveness of DOE field office oversight of contractor activities. Without conducting periodic assessments or audits focused on RL's management and oversight of the contractor's S&M activities for contaminated excess facilities, DOE does not have assurance that RL is overseeing S&M activity in a way that ensures these facilities are inspected and maintained in a safe and compliant condition pending final cleanup. GAO recommends that DOE (1) analyze the programmatic root causes of the tunnel collapse, (2) routinely conduct comprehensive inspections of contaminated excess facilities and take timely action as warranted, and (3) assess RL oversight of S&M of Hanford excess facilities. DOE agreed with GAO's recommendations and stated that it is taking steps to implement all of them by December 2020.", "document_type": "gao"}
{"report": "The Marine Corps’ Marine Helicopter Squadron One (HMX-1) currently uses a fleet of 23 helicopters to support the President in the national capital region, the continental United States and overseas. In April 2002, the Navy began developing a replacement helicopter later identified as the VH-71. Schedule delays, performance issues, and a doubling of estimated acquisition costs from $6.5 billion to $13 billion prompted the Navy to terminate the VH-71 program in 2009. Our prior work found that the VH-71 program’s failure to follow acquisition best practices was a critical factor in the program’s poor performance that led to its ultimate termination. In the case of the VH-71, the Navy had a faulty business case, did not perform appropriate systems engineering analysis to gain knowledge at the right times, and failed to make necessary trade-offs between resources and requirements even after years of development. Because of these failures, the program was unable to achieve a stable design and experienced significant cost growth and schedule delays. Although the prior replacement program was terminated, the need for a replacement helicopter remained. As a result, the Navy initiated a follow- on replacement program in 2010. In April 2012, the Secretary of Defense approved the Navy’s plan based on the modification of an in-production helicopter to meet Navy requirements. The VH-92A is expected to provide improved performance, survivability, and communications capabilities, while offering increased passenger capacity when compared to legacy helicopters. In May 2014, the Navy competitively awarded a contract to Sikorsky to develop the VH-92A, which included options for production. The $2.7 billion contract includes a fixed-price incentive (firm target) Engineering and Manufacturing Development (EMD) phase and a firm-fixed price production phase with options for three lots for 17 helicopters, spares and support equipment. Under the EMD phase, Sikorsky has delivered two development test helicopters which were used in an operational assessment that was completed in April 2019. Additionally, Sikorsky has delivered three of four System Demonstration Test Article (SDTA) production representative helicopters that are being used in developmental testing and that will also be used to evaluate the VH-92A’s operational effectiveness and suitability during the program’s Initial Operational Test and Evaluation (IOT&E). The fourth SDTA helicopter is to be delivered in May 2020 and will also be used to conduct IOT&E. In June 2019, the Assistant Secretary of the Navy, Research, Development and Acquisition (RD&A) approved the program to begin low-rate initial production of the helicopters and authorized the program to exercise the contract options for the first two low-rate production lots. Shortly thereafter, the Navy exercised the Lot l option for 6 helicopters, initial spares, and support equipment for $542 million. Those helicopters, initial spares and support equipment are all to be delivered in calendar year 2021. In February 2020, the Navy exercised the Lot ll option for $471 million for 6 additional helicopters and associated support equipment. All of these helicopters and support equipment will be delivered in calendar year 2022. The Navy had planned for two years of low-rate initial production of 6 helicopters each year followed by one year of full-rate production for the remaining 5 helicopters. The Navy’s acquisition strategy in support of the production decision included a change in that plan with the re-designation of full-rate production as a third lot of low-rate production. A key reason for the change is that the planned full-rate production run of 5 helicopters was too small to achieve the potential cost benefits of full-rate production, which typically involves purchasing a sufficient number of helicopters to decrease unit cost. This revised strategy would also enable the Navy to award the third production lot seven months earlier than the originally planned May 2021. Before obligating the funding available for the second lot, the program office had to brief the Assistant Secretary of the Navy (RD&A) on various elements of the VH-92A’s performance. The program office is required to obtain approval from the Assistant Secretary of the Navy (RD&A) for the procurement of the last lot (Lot lll) with a decision brief that includes, among other things, the status of IOT&E. Building a VH-92A helicopter involves work at three facilities. To begin the production process, Sikorsky takes an S-92A helicopter from its commercial production line in Coatesville, Pennsylvania and flies it to a dedicated VH-92A modification facility in Stratford, Connecticut. Once there, Sikorsky removes some components, such as circuit breaker panels, engines, and main and tail rotor blades and replaces them with VH-92A components. Additionally, Sikorsky modifies the helicopter to accommodate VH-92A specific subsystems, including racks and wiring for a Navy-developed mission communications system (MCS). Sikorsky then flies the helicopter to a dedicated facility in Owego, New York where it integrates the MCS, installs the executive cabin interior, paints the helicopter, and conducts final testing before delivering the helicopter to the government. See figure 1 for a depiction of modifications of the commercial S-92A helicopter to the VH-92A presidential helicopter. We have reported annually on the Navy’s effort to replace the current fleet of presidential helicopters since 2011. Our reports highlighted, in part, the extent to which the Navy used the lessons learned from the failed VH-71 program—the need to balance requirements, costs, and schedule and the importance of establishing a knowledge-based program that is aligned with acquisition best practices—in its new effort. For example, our 2011 report found that while the replacement program was early in its development cycle, the Navy’s initial efforts appeared to reflect the intent to pursue a best practices aligned knowledge-based acquisition. Following the program’s entry into the EMD phase of acquisition in April 2014, we found that the Navy’s reliance on mature technologies, selection of an existing helicopter for use in the program, and award of a fixed-price incentive type contract reduced risk. As to be expected with a major system development effort, however, we found the program still faced a number of technical challenges. In four reports issued from 2016 to 2019, we found that the Navy continued making progress in developing the VH- 92A helicopter while managing design, integration and technical challenges. Some key technical risk and challenges that we previously identified are summarized in table 1. We discuss the current status of the Navy’s efforts to address these challenges later in the report. In April 2019, the Navy estimated that the VH-92A would cost about $4.9 billion to develop and produce and about $15.6 billion to operate and support the helicopters through fiscal year 2062. Overall, the Navy’s $20.5 billion estimate reflects a 10-percent reduction from the program’s 2014 baseline estimate (see table 2). The Navy and contractor officials worked together to remain within the program’s April 2014 cost baseline, in part, by keeping requirements stable, limiting the design changes, and taking advantage of cost saving initiatives. For example, the Navy has not added any key performance requirements to the fixed-price incentive contract since it was awarded in 2014. The Navy has, however, implemented a small number of design changes to add an additional cockpit display and increase the height of the upper portion of the forward aircraft door. Previously, we found that cost saving initiatives included leveraging the Federal Aviation Administration’s airworthiness certification process, optimizing work processes, and reducing the movement of helicopters between contractor sites. In addition, the Navy attributes the reduction in cost to support the VH- 92A fleet to using a planned maintenance interval concept as the basis for its April 2019 cost estimate. Program officials explained that the April 2014 baseline estimate was based on the approach used to maintain the current fleet of VH-3D and VH-60N presidential helicopters. For these helicopters, the contractor carries out depot-level maintenance by disassembling, inspecting, and reassembling them at its maintenance depot. However, for the VH-92A, the Navy intends to perform depot-level maintenance itself through scheduled inspections at its own presidential helicopter support facility, which was designed to support this approach. As a result, the Navy expects to be able to support the VH-92A fleet in a more cost-effective manner while ensuring the availability of the helicopter to perform its mission. The program has made progress addressing technical risks and performance challenges we discussed in prior reports and deficiencies confirmed during the April 2019 operational assessment. According to program officials, solutions for these performance shortfalls, except for the landing zone suitability issue, have been developed and successfully tested during integrated testing and will be evaluated during the 3-month IOT&E test scheduled to begin in June 2020. The program is pursuing options to achieve landing zone suitability that include possible changes in operational procedures, helicopter design, and lawn surface treatments. If design modifications are required, they will not be implemented until after IOT&E. As a result, the Navy may not be able to fully demonstrate that the VH-92A helicopter meets all its key requirements until after the test program is complete. Further, IOT&E results may also identify additional issues that may require additional design or software changes. Depending on the severity of the issues, the Navy may need additional time to test and incorporate changes into the helicopter, including those helicopters currently in production. The program office has mitigated or reduced risk on some technical issues we discussed in prior reports. For example, according to program documents, the program has mitigated the risk in the following areas: helicopter start procedures, electromagnetic environment effects/ electromagnetic pulse and cybersecurity. The Navy assessed these capabilities during earlier developmental test and during the operational assessment, which concluded in April 2019; subsequently, the Navy approved the program to enter into production. However, the operational assessment confirmed other known performance shortfalls—specifically those associated with the MCS—that, if not corrected, could prevent the program from meeting certain operational requirements. The MCS replaces the communications suite currently used by the in- service fleet and provides VH-92A passengers, pilots, and crew with simultaneous short- and long-range secure and non-secure voice and data communications capabilities. As such, its performance is critical for the VH-92A to meet its mission. To conduct its operational assessment, the Navy used two development test helicopters and a developmental version of MCS software with known performance and capability limitations. The operational assessment confirmed these MCS-related performance limitations, including dropped communication connections. Navy officials noted that these and other MCS-related performance shortfalls could, if not addressed, reduce the helicopter’s availability to perform its transport mission and lower overall reliability, among other operational requirements. Overall, the operational assessment confirmed 24 MCS-related performance limitations. According to program officials, they have incorporated or identified fixes to 22 of the 24 issues, which they are now testing on SDTA helicopters. In turn, these fixes are expected to be incorporated into MCS software that will be tested during IOT&E. According to program officials, the remaining two MCS issues are related to bandwidth and an unreliable off-aircraft network configuration affecting on-aircraft system performance. According to those officials, the VH-92A is already equipped with a wide-band line-of-sight system that provides high bandwidth, though with coverage limitations. The program is conducting market research on how to provide the helicopter with increased bandwidth with increased coverage. The remaining two issues were assessed earlier as having a serious (but not critical) impact to mission accomplishment. In addition to the MCS deficiencies, the helicopter experienced problems with other components during the April 2019 operational assessment. For example, the mission and maintenance data computer repeatedly sent out false warning alarms/notifications, which affected the reliability and required the aircrew to spend extra time troubleshooting or switch to a backup helicopter. A software update to help address this issue is planned for the computer prior to IOT&E. The program is also still working to demonstrate the ability of the helicopter to meet a key system capability requirement to land the helicopter without damaging landing zones (including the White House South Lawn). For landing zone suitability, the program’s objective has been to assess the downwash and exhaust effects on the landing zone. In a September 2018 training event, the Navy found that VH-92A’s exhaust damaged a landing zone. Program officials stated that the training event did not represent a typical operational scenario since the lawn was exposed to the helicopter’s exhaust for a longer period than it would be under normal operating conditions. The program is studying solutions to minimize risk of landing zone damage including possible changes in operational procedures, helicopter design, and lawn surface treatments. For example, the contractor developed a prototype design change to the helicopter’s auxiliary power unit to deflect exhaust. Flight testing of the prototype design change was conducted in March 2020 with analysis of the results expected in April 2020. Navy officials stated the contractor is also conducting testing to determine if changes in helicopter and/or engine operating procedures can mitigate the risk of landing zone damage. According to both program officials and contractor representatives, a decision on potential solutions will be made prior to IOT&E. If design modifications are required, they will not be implemented until after IOT&E. Initial operational testing of the VH-92A, which will be used to evaluate operational effectiveness and suitability of the helicopter, training system, support equipment, upgraded MCS software and other changes implemented to address previously identified issues, is now scheduled to be conducted between June and September 2020. As such, IOT&E will be conducted about 3 months later than we reported in 2019, but is expected to be completed by the threshold (latest acceptable) date in the Navy’s April 2014 baseline. Program officials attributed the 3-month delay to the need to develop MCS hardware and software changes that are currently being tested. Should IOT&E demonstrate that efforts to address the MCS performance issues or other previously identified issues are insufficient—or if the testing identifies new issues that result in the program being unable to meet its operational requirements—then the program may need to identify, test and incorporate changes into the VH- 92A’s design and into the helicopters already in production, further delaying the program and increasing associated costs. As previously noted, the first delivery of the helicopters ordered under the first production option is scheduled to begin in April 2021. As a result of the revised IOT&E test schedule, the program office has also delayed the initial operational capability (IOC) milestone, which clears the helicopter to enter service, by 3 months to January 2021. This new date represents a total delay of 6 months from the original date but still remains within the IOC threshold date established in April 2014. Figure 2 compares the program’s 2019 schedule with the 2014 baseline schedule and the 2018 schedule we reported on last year. Program officials acknowledged that if there is a delay in the program that results in the program breaching a schedule threshold established in its acquisition baseline, they would need to submit a program deviation report to the Assistant Secretary of the Navy (RD&A). In turn, the program may need to keep certain staff in place longer than originally planned, potentially increasing program costs. However, program officials told us that the program can cover any additional costs with existing funding. Further, Navy officials stated that should IOC be delayed, the Navy will continue to use its existing fleet of presidential helicopters as the VH-92A transitions into the HMX-1 fleet. Navy officials indicated that the transition process will be gradual, and that the existing fleet is sufficiently funded until HMX-1 completes the transition. We are not making any recommendations in this report. We provided DOD with a draft of this report. DOD provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense and the Secretary of the Navy. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or DiNapoliT@gao.gov. Contacts points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix l. In addition to the contact above, Bruce H. Thomas, Assistant Director; Marvin E. Bonner; Bonita J.P. Oden; Alexander Webb; Peter Anderson; Robin Wilson; and Marie Ahearn made key contributions to this report. Presidential Helicopter: Program Continues to Make Development Progress While Addressing Challenges. GAO-19-329. Washington, D.C.: April 11, 2019.* Presidential Helicopter: VH-92A Program Is Stable and Making Progress While Facing Challenges. GAO-18-359. Washington, D.C.: April 30, 2018.* Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-17-333SP. Washington, D.C.: March 30, 2017.* Presidential Helicopter: Program Progressing Largely as Planned. GAO-16-395. Washington, D.C.: April 14, 2016.* Presidential Helicopter Acquisition: Program Established Knowledge- Based Business Case and Entered System Development with Plans for Managing Challenges. GAO-15-392R. Washington, D.C.: April 14, 2015.* Presidential Helicopter Acquisition: Update on Program’s Progress toward Development Start. GAO-14-358R. Washington, D.C.: April 10, 2014. Department of Defense’s Waiver of Competitive Prototyping Requirement for the VXX Presidential Helicopter Replacement Program. GAO-13-826R. Washington, D.C.: September 6, 2013. Presidential Helicopter Acquisition: Program Makes Progress in Balancing Requirements, Costs, and Schedule. GAO-13-257. Washington, D.C.: April 9, 2013. Presidential Helicopter Acquisition: Effort Delayed as DOD Adopts New Approach to Balance Requirements, Costs, and Schedule. GAO-12-381R. Washington, D.C.: February 27, 2012. Defense Acquisitions: Application of Lessons Learned and Best Practices in the Presidential Helicopter Program. GAO-11-380R. Washington, D.C.: March 25, 2011. *GAO issued these reports on the VH-92A program in response to a provision in National Authorization Defense Act of 2014.", "summary": "The mission of the presidential helicopter fleet is to provide safe, reliable, and timely transportation in support of the President. The Navy plans to acquire a fleet of 23 VH-92A helicopters to replace the current Marine Corps fleet which has been in use for more than 40 years. Delivery of production VH-92A helicopters is scheduled to begin in April 2021 and be completed in January 2023. The National Defense Authorization Act of 2014 included a provision for GAO to report annually on the acquisition of the VH-92A helicopter. This report, GAO's sixth related to the provision, examines (1) the extent to which the program is meeting cost goals and (2) performance and schedule challenges that the program has experienced. To conduct this work, GAO compared the Navy's April 2019 cost estimates for acquiring and maintaining the new helicopters and October 2019 program schedule information to its April 2014 acquisition baseline. GAO reviewed development test results and status reports from the program. GAO also interviewed officials from the program office, Navy test organizations, and the contractor. GAO is not making any recommendations in this report. The Navy estimates the cost to develop, procure, and maintain the VH-92A ® over its 40-year operational life to be just over $20.5 billion, or about 10 percent less than the Navy's 2014 baseline estimate (see table). Navy and contractor officials worked to remain within the program's April 2014 cost baseline estimate, in part, by keeping program requirements stable, limiting design changes, and taking advantage of cost saving initiatives. The Navy also plans to use Navy personnel and facilities to perform depot-level maintenance for the VH-92A fleet, rather than sending the helicopters back to the contractor as is currently done. The program has made progress addressing technical risks and performance challenges GAO discussed in prior reports; however, an April 2019 operational assessment confirmed several other risks that could affect the helicopter's ability to meet its reliability and availability requirements. For example, Navy officials stated that the assessment confirmed known limitations with the mission communications system. Upgraded software intended to address those limitations is to be evaluated during the initial operational test and evaluation scheduled to be conducted between June and September 2020. The results of that testing could impact the Navy's planned January 2021 decision to begin using the helicopters as part of the presidential helicopter fleet.", "document_type": "gao"}
{"report": "This section provides information on the electricity provider, impact of 2017 hurricanes, and status of electricity restoration in Puerto Rico and the U.S. Virgin Islands. Also, it describes FEMA’s Public Assistance Program. Electricity provider. PREPA is a public power utility owned by the Commonwealth of Puerto Rico and a monopoly supplier of electricity in the commonwealth. It is also one of the nation’s largest public power utilities, serving approximately 1.5 million customers. PREPA was approximately $9 billion in debt prior to Hurricanes Irma and Maria, and its electric power infrastructure was known to be in poor condition, largely due to underinvestment and poor maintenance practices. In May 2018, we found that inadequate management of PREPA’s financial condition contributed to Puerto Rico’s persistent deficits. Specifically, PREPA did not update or improve its electric generation and transmission systems, which hampered their performance and led to increased costs of producing electricity that it did not fully pass on to consumers. In addition, Puerto Rico’s economy is in a prolonged period of economic contraction, and according to U.S. Census Bureau estimates, its population declined from a high of approximately 3.8 million people in 2004 to 3.3 million people in 2017, a decline of 12.8 percent. Along with the declining population, demand for electricity declined 18 percent from 2007 to 2017, according to PREPA. Impact of 2017 hurricanes. Hurricanes Irma and Maria in September 2017 left Puerto Rico’s entire electricity grid inoperable, according to the economic and disaster recovery plan for Puerto Rico. According to a report by a working group that included utilities and national laboratories, among others, because of the extended and unprecedented damage, a significant portion of the generation, transmission, and distribution system must be rebuilt, including high-voltage transmission lines that often survive lower category hurricanes. While Puerto Rico’s population had already been declining, the migration of people from Puerto Rico accelerated following the hurricanes, according to PREPA. Status of electricity restoration. According to PREPA, it took roughly 11 months for power to be restored to all of the customers able to receive power safely in Puerto Rico following the hurricanes. A PREPA official told us that PREPA’s estimates of customers with power restored are based on the number of meters that it knows are served by a given power line and on the number of meters it can read currently. Power has been restored to 100 percent of customers that are able to receive power safely, but this does not mean that all pre-storm customers have power restored, as some structures may not have been deemed safe for power restoration, according to PREPA officials. Figure 1 shows the percentage of customers with electricity restored in Puerto Rico beginning in January 2018 when PREPA was able to start estimating this information. Although PREPA estimates that electricity had been restored to all customers by August 2018, in some instances electricity service has been supported by temporary generators, and outages have continued. For example, as of December 11, 2018, USACE was supporting seven generators that were supporting micro grids for the island municipalities of Vieques and Culebra. These islands had previously been served by an undersea transmission line connecting the islands to PREPA’s main grid on Puerto Rico. According to the U.S. Energy Information Administration, total electricity sales in Puerto Rico returned to pre–Hurricane Maria levels as of April and May 2018, although residential electricity sales appear to continue to lag historical levels, reflecting some continued outages. Electricity provider. VIWAPA, a public utility, is a monopoly provider of electric power services in the U.S. Virgin Islands and serves approximately 55,000 customers throughout the territory. Like PREPA, VIWAPA faced financial challenges before the hurricanes. The USVI Hurricane Recovery and Resilience Task Force Report noted that VIWAPA has a 17 percent non-payment rate across its customer base, a significant unfunded pension liability, and long-term debt commitments of $265 million. In addition, the report states that the U.S. Virgin Island’s energy system faces many challenges that have led to higher rates and a historically unreliable grid. These include an aging, inefficient, and oversized infrastructure and heavy reliance on imported fossil fuels. The report also says that peak demand declined 18 percent from 2011 through 2017, driven by a variety of factors, including population decline. In addition, the report says that VIWAPA’s high energy rates and reliability issues have led some customers— particularly larger commercial and industrial ones—to leave the grid. Impact of 2017 hurricanes. Hurricanes Irma and Maria damaged more than 90 percent of VIWAPA’s aboveground power lines and over 20 percent of VIWAPA’s generation capacity, according to the USVI Hurricane Recovery and Resilience Task Force Report. Specifically, the hurricanes damaged more than 20,000 poles and 1,100 miles of transmission and distribution lines, according to the report. Although 90 percent of VIWAPA’s above ground power lines were damaged, this was far fewer than the miles of transmission and distribution lines damaged in Puerto Rico. Electricity status. According to VIWAPA, following the hurricanes, it took roughly 5 months for power to be restored to all of the eligible customers in the U.S. Virgin Islands. Eligible customers were those whose homes were safely able to receive power. Some homes had suffered substantial damage to their electrical infrastructure from the hurricanes and were not able to receive power safely until their electrical equipment was repaired. VIWAPA’s estimates of customers with power restored are based on the number of meters that VIWAPA knows are served by a given power line, as VIWAPA’s automated system for determining the percentage of customers without power was destroyed and is still being restored, according to a FEMA official. Although electricity service has been restored, electricity demand has not recovered to prestorm levels. According to the USVI Hurricane Recovery and Resilience Task Force Report, VIWAPA’s peak demand—the maximum energy load consumed by customers at any point in a year—was approximately 107 megawatts before the storms, but as of May 2018 it was 66 megawatts. The report says that demand will likely rebound to some degree as the territory rebuilds and recovers; however, it is unclear how quickly or by how much. FEMA, in leading the coordination of federal disaster response efforts, provides assistance through its Public Assistance Program to state, territorial, local, and tribal governments and certain types of private nonprofit organizations to assist them in responding to and recovering from major disasters or emergencies. FEMA Public Assistance Program funds can be provided for emergency work, such as for emergency protective measures that must be done immediately to protect public health and safety; permanent work, which includes the restoration of disaster-damaged management costs, which include indirect costs, administrative expenses, or other expenses that are not directly chargeable to a specific project and that a recipient or subrecipient incurs in administering and managing Public Assistance awards. Generally, emergency work takes place for about 6 months following a disaster, while permanent work can take place over a decade, according to FEMA officials. FEMA can provide grants for both emergency and permanent work, and it can also provide direct federal assistance for emergency work. Under direct federal assistance, federal agencies directly perform or contract for the emergency work. FEMA’s Public Assistance Program allows for the federal government to provide direct assistance at the request of the state, territorial, and local governments when the impact of an incident is so severe that the state, territorial, and local governments lack the capability to perform or contract eligible emergency work. Under the Public Assistance Program and the Stafford Act, FEMA may mission assign—issue a work order that directs another federal agency, such as DOE or USACE to utilize its authorities and the resources granted to it under federal law—in support of this direct assistance to state, local, and territorial governments. The Community Disaster Loan program provides loans to local governments that have suffered substantial loss of tax and other revenue in areas included in a major disaster declaration. The loan funding may be used for existing essential municipal functions and expanded functions required to meet disaster-related needs, but not for capital improvements or repair or restoration of damaged public facilities. Federal agencies provided traditional support to restore electricity in response to Hurricanes Irma and Maria in both Puerto Rico and the U.S. Virgin Islands—such as providing temporary power for critical facilities. They also provided unprecedented support in Puerto Rico by helping to coordinate efforts to repair Puerto Rico’s electricity grid rather than primarily supporting the local utility’s efforts. Factors that affected the electricity grid restoration efforts in Puerto Rico and the U.S. Virgin Islands included logistical constraints, availability of materials, the financial condition of local utilities, and the unprecedented and extensive role of federal agencies. Appendix I provides timelines of federal and other efforts to support electricity grid restoration in Puerto Rico and the U.S. Virgin Islands after the 2017 hurricane season. Federal agencies assisted in the restoration of electricity after Hurricanes Irma and Maria in a variety of ways. FEMA provided billions in grants and direct federal assistance for electricity restoration. DOE provided subject matter expertise and coordination assistance. USACE provided temporary emergency power in Puerto Rico and the U.S. Virgin Islands. In addition, FEMA and USACE undertook unprecedented roles to help coordinate and directly assist with grid restoration in Puerto Rico. Grants, direct federal assistance, and loans from FEMA. FEMA provided billions in grants and direct federal assistance to support electricity restoration in Puerto Rico and the U.S. Virgin Island through its Public Assistance Program. As public utilities, both PREPA and VIWAPA are eligible applicants for federal assistance through FEMA’s Public Assistance Program for the repair, restoration, and replacement of public facilities damaged or destroyed by a major disaster. As of July 20, 2018, FEMA had obligated approximately $3.2 billion for direct federal assistance through mission assignments and Public Assistance grant funds for electricity restoration in Puerto Rico and approximately $795 million for the U.S. Virgin Islands. This includes $2 billion that FEMA obligated for direct federal assistance through mission assignments to USACE for temporary emergency power and grid restoration efforts in Puerto Rico. In the U.S. Virgin Islands, FEMA obligated $63 million for direct federal assistance related to electricity restoration, most of which was obligated to USACE and DOE. Table 1 shows FEMA funding obligations for electricity restoration efforts in Puerto Rico and the U.S. Virgin Islands. In addition, FEMA provided $75 million to VIWAPA through the Community Disaster Loan program as of July 20, 2018, according to FEMA officials. FEMA officials said that the most common use for Community Disaster Loan funds is payroll, and other examples of eligible uses include employee benefits, facilities maintenance costs, and normal operating materials. Coordination and technical assistance from DOE. DOE received mission assignments from FEMA and deployed staff from its headquarters, site offices, and power marketing administrations to provide subject matter expertise and technical assistance in support of electricity grid damage assessments and power restoration efforts in both Puerto Rico and the U.S. Virgin Islands. According to DOE officials, DOE’s primary role in the response efforts on Puerto Rico and the U.S. Virgin Islands was coordination and provision of subject matter experts, as is typical for DOE’s role as the lead agency for the energy sector emergency support function. In Puerto Rico, however, DOE provided more advisors for a longer period of time than would be typical because of the extent of the damage to the electricity grid in Puerto Rico and PREPA’s limited capacity to respond, according to DOE officials. Specifically, DOE staffed up to 12 project support advisors to Puerto Rico from October 18, 2017, to August 8, 2018, and one supply chain support advisor from December 18, 2017, to March 16, 2018. These advisors provided subject matter expertise to USACE by reviewing construction plans and providing recommendations for prioritization, and scheduling and assisting in inventory management for incoming electrical grid equipment, among other things, according to DOE. In addition, in the U.S. Virgin Islands DOE deployed a team of 36 people from the Western Area Power Administration along with trucks and materials to help rebuild the electricity grid through a FEMA mission assignment. DOE officials told us that the department is also providing ongoing support on how to improve grid resilience as part of grid restoration and recovery efforts in both Puerto Rico and the U.S. Virgin Islands. Temporary power from USACE. USACE provided temporary emergency power for critical facilities in Puerto Rico and the U.S. Virgin Islands. These temporary emergency power missions provided and maintained generators to deliver electricity to critical public facilities, such as hospitals and relief centers. After receiving a FEMA mission assignment to provide temporary emergency power in Puerto Rico, USACE deployed its Emergency Power Planning and Response Team, USACE government employees, soldiers from the 249th Engineer Battalion, and contractors. USACE installed a record number of emergency electric generators in Puerto Rico—over 2,300—through the end of May 2018. The previous record was 310 emergency generators installed in response to Hurricane Katrina. On May 17, 2018, FEMA approved the extension of the USACE mission assignment for emergency power to November 30, 2018. This extension permitted USACE to continue its support for the more than 700 generators still in use throughout Puerto Rico at that time. FEMA later extended the mission assignment until April 8, 2019. As of December 11, 2018, USACE was supporting 24 generators in Puerto Rico, seven of which were supporting micro grids for the island municipalities of Vieques and Culebra. In the U.S. Virgin Islands, USACE installed 180 generators as a part of its temporary emergency power mission. USACE’s temporary emergency power mission for the U.S. Virgin Islands was completed in February 2018, and USACE is no longer supporting generators there. Unprecedented Roles by FEMA and USACE in Puerto Rico. In addition to the typical roles federal agencies undertake in restoration activities, FEMA and USACE undertook unprecedented roles in Puerto Rico because of the severe and widespread impacts of Hurricane Maria and PREPA’s limited capacity. For the first time in its history, FEMA undertook the role of helping to coordinate major electricity grid restoration because PREPA did not have the necessary capability, capacity, or structure to respond, according to FEMA officials. FEMA officials also noted that PREPA’s workers were not only engaged in restoration work but were also victims dealing with the same post- hurricane effects as the rest of the population. As part of its response, FEMA mission assigned USACE to lead federal efforts to repair Puerto Rico’s electricity grid—a role USACE had not played in the past during a domestic disaster response. Specifically, on September 30, 2017, the FEMA Administrator tasked USACE with leading the planning, coordination, and integration of the grid restoration. FEMA assigned USACE to lead federal efforts and provide direct support for grid restoration because PREPA was overwhelmed and had liquidity issues and USACE had the structures in place to award contracts with and bring in grid restoration crews, according to FEMA officials. In order to carry out its mission assignment, USACE issued contracts to bring lineworkers and materials to Puerto Rico to support the reinstallation and repair of transmission and distribution lines, among other power restoration activities. As of June 30, 2018, USACE had obligated approximately $1.5 billion on these contracts. Figure 2 shows USACE and its contractors working to restore electricity in Puerto Rico. USACE’s grid restoration mission assignment from FEMA ended on May 18, 2018, because, according to FEMA officials, power had been restored to approximately 98 percent of customers and PREPA, with its remaining contractors, had adequate capability to do the remaining work. In addition to the federal response, PREPA issued its own contracts to bring in additional lineworkers, received assistance from the New York State Utility Contingent, and requested and received mutual assistance from other utilities. PREPA did not initially reach out for mutual assistance. About 6 weeks following Hurricane Maria, on October 31, 2017, PREPA formally requested aid from other utilities on the mainland through the American Public Power Association and the Edison Electric Institute. The electric power industry sent two individuals to Puerto Rico on November 3, 2017 and they began assessing storm damage and working with PREPA, FEMA, USACE, and DOE officials to develop a restoration plan. On November 22, 2017, the Governor of Puerto Rico appointed one of these individuals as Power Restoration Coordinator to oversee the multipronged restoration effort. According to the Power Restoration Coordinator, as a first step he worked to create an incident command structure, and incident management teams began arriving in December. Once the incident command structure was in place, the industry deployed additional crews, equipment and materials in January to accelerate the ongoing restoration efforts across the island. As discussed previously, local utilities are typically responsible for restoring service, with federal agencies providing financial and other support. In contrast, approximately half of the lineworkers working to restore the electricity grid in Puerto Rico were USACE or USACE contractors at the peak of restoration efforts in February 2018, as shown in figure 3. FEMA established a unified command structure to coordinate efforts of federal agencies, PREPA, PREPA’s contractors, the New York State Utility Contingent, and utilities providing mutual assistance to PREPA, to help target priority work, ensure that crews could get to the work, and identify needed materials. Figure 4 shows the unified command structure. According to documents we reviewed and our interviews with officials and representatives, the most commonly cited factors that affected federal electricity grid restoration efforts in Puerto Rico and the U.S. Virgin Islands included (1) logistical challenges, (2) availability of materials, (3) financial condition of local utilities and poor condition of existing infrastructure, and (4) the extensive and unprecedented role of federal agencies. Logistical challenges. Responding to disasters on islands presents a number of logistical challenges. Specifically, according to federal officials, getting the crews, equipment, and materials needed to support restoration efforts to an island was more difficult and time- consuming than doing so on the mainland. This includes prepositioning assets, such as generators, and delivering equipment and materials in advance of a storm. The difficulties were greater in the days following the hurricanes since neither the ports nor the airports in Puerto Rico and the U.S. Virgin Islands had power, which prevented the delivery of materials to the islands. In Puerto Rico, the Port of San Juan reopened for daylight operations 3 days after Hurricane Maria made landfall; every airport and seaport had limited capacity after reopening for approximately 7 days post-landfall, according to FEMA’s 2017 Hurricane Season After-Action Report. Federal officials in the U.S. Virgin Islands told us that they faced further delays locating key supplies because of inadequate labelling of shipping containers at the port. For example, some containers were marked only as disaster supply equipment, which did not sufficiently identify the contents within them. According to USACE’s 2018 Remedial Action Program Senior Leader Briefing, USACE lacked the expertise and capabilities to manage the large operational logistics requirements to support the Puerto Rico and U.S. Virgin Islands response. Availability of materials. The sequence of three hurricanes making landfall in the United States in 2017 and the need to restore electricity service in Texas, Florida, and elsewhere, in addition to Puerto Rico and the U.S. Virgin Islands, complicated the restoration effort in the two territories. Since utilities in all affected areas were acquiring materials to restore electricity service, demand for these materials increased and available supplies were generally low; in some instances materials were only available as they were manufactured. Few, if any, materials were stockpiled locally on Puerto Rico. In addition, some of the equipment used in Puerto Rico was not standard in the continental United States and required ordering of specialized materials, resulting in delays in the restoration process. The U.S. Virgin Islands also faced supply issues, which became worse once grid recovery work in Puerto Rico began. Financial condition of local utilities and poor condition of existing infrastructure. Electric utilities in both Puerto Rico and the U.S. Virgin Islands were insolvent, which led to a lack of maintenance and presented its own challenges for restoring the grids after the storms. Specifically, PREPA was approximately $9 billion in debt before the 2017 hurricane season, with annual costs that exceeded its revenues. Puerto Rico’s electric power infrastructure was in poor condition before the 2017 hurricane season largely because of PREPA’s underinvestment and poor maintenance practices. For example, PREPA canceled its vegetation management program because of its financial situation; this contributed to the destruction of transmission and distribution lines when the hurricane arrived, according to FEMA officials. Similarly, in the U.S. Virgin Islands, financial challenges contributed to the extent of the damage to grid infrastructure. Specifically, VIWAPA officials told us that VIWAPA’s financial challenges prevented certain infrastructure improvements and a large proportion of the electricity poles were at or above their weight capacity, increasing the likelihood and extent of wind damage during the hurricanes. According to VIWAPA officials, VIWAPA was aware that there were too many lines and heavy transformers on old poles, but was not in a position to address this concern prior to the hurricanes. Extensive and unprecedented role of federal agencies. FEMA did not anticipate or plan for the extensive role that it and USACE played in grid restoration in Puerto Rico. According to FEMA’s after action report for the 2017 hurricane season, FEMA’s planning assumptions for a hurricane, earthquake, or tsunami striking Puerto Rico and the U.S. Virgin Islands underestimated the actual requirements. As discussed above, prior to Hurricane Maria in Puerto Rico, USACE had never worked on a large-scale power restoration as part of a domestic disaster response and did not have expertise in this area, according to USACE officials. This affected grid restoration efforts. For example, USACE did not have a grid restoration contract in place to immediately initiate grid repair efforts, according to USACE officials. Rather, USACE issued an order off of a pre-existing contract that it had under its public works and engineering support function to bring electric utility lineworkers to Puerto Rico. According to USACE officials, a bid protest against one of USACE’s contracts delayed its ability to increase the contract to bring more lineworkers to Puerto Rico. In addition, the contract review and approval process USACE used to obtain supplies took longer than it would typically take utilities to get supplies, according to FEMA officials we interviewed. According to USACE officials, USACE followed federal acquisition regulations, which is a slow process compared to private party purchases. USACE officials said that USACE is considering looking at what would be needed to create an advance grid restoration contract. FEMA, USACE, and DOE identified potential actions to address these challenges. According to its after action report, FEMA plans to establish a standing interagency Power Task Force to coordinate with DOE, USACE, and state and local governments and provide crisis planning for the energy sector emergency support function to support the restoration of power during future national response efforts. USACE’s 2018 Remedial Action Program Senior Leader Briefing made recommendations to improve contingency contracting and operational logistics, among other things. Specifically, recommendations included that USACE review existing and planned advance contracts and make adjustments as necessary to increase capacity and improve capabilities, and that USACE work with FEMA to convene an interagency logistics planning team and identify logistics contracting gaps and propose government and private sector solutions. DOE’s after action report for the 2017 hurricane season says that the lessons learned from the response to Hurricane Maria may prompt some programmatic improvements to the energy sector emergency support function roles and responsibilities related to island response, among other potential improvements. In addition, the report states that because of the extensive damage to grid infrastructure and the length of the restoration and recovery, there is an increasing need to incorporate resilience and hardening into restoration, recovery, and mitigation planning and execution. We provided a draft of this report to the Department of Defense (DOD), the Department of Homeland Security (DHS), DOE, and the governments of Puerto Rico and the U.S. Virgin Islands for review and comment. In its comments, reproduced in appendix II, DHS indicated that a top priority of DHS, FEMA and the entire federal government has been to provide life safety and life-sustaining resources to Puerto Rico and the U.S. Virgin Islands, including efforts to restore power and stabilize critical infrastructure. DHS, DOD, and DOE also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of Energy, the Secretary of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. See figures 5 and 6 for a timeline of federal and other efforts to support electricity grid restoration in Puerto Rico and the U.S. Virgin Islands after the 2017 hurricane season. In addition to the contact named above, Quindi Franco (Assistant Director), Marya Link (Analyst in Charge), Janice Ceperich, William Gerard, Cindy Gilbert, Joseph Maher, David Marroni, Bolko Skorupski, Sheryl Stein, and Jarrod West made key contributions to this report.", "summary": "In 2017, Hurricanes Irma and Maria damaged much of the electricity grids' transmission and distribution systems in USVI and Puerto Rico. The hurricanes left most of USVI's 106,405 people and all of Puerto Rico's 3.3 million without power and resulted in the longest blackout in U.S. history. Under the National Response Framework, electric utilities are responsible for repairing infrastructure and restoring service. They often use mutual assistance—voluntary partnerships with other electric utilities—to bring in additional resources to help restore electricity. Federal agencies provide financial assistance; help coordinate the federal response; and in severe emergencies, provide logistical support, such as assisting in damage assessments and location and transportation of repair crews and equipment. GAO was asked to review the federal response to the 2017 hurricanes. This report provides information on federal support for restoring the electricity grids in Puerto Rico and USVI and factors affecting this support. GAO has ongoing work examining federal support to improve grid resilience in Puerto Rico. GAO reviewed agency documents and funding data through July 20, 2018, the most recent data available; interviewed officials from FEMA, DOE, and USACE; and conducted site visits to Puerto Rico and USVI. Federal agencies supported efforts to restore electricity in the U.S. Virgin Islands (USVI) and Puerto Rico through the types of support they traditionally provide following disasters and, in Puerto Rico, in some unprecedented ways. USVI. Federal agencies provided traditional federal support to the electric utility's restoration efforts. For example, the Federal Emergency Management Agency (FEMA) provided financial assistance through its Public Assistance Program, and the Department of Energy (DOE) provided subject matter expertise to assist the local utility. In addition, the U.S. Army Corps of Engineers (USACE) provided generators for hospitals and other critical facilities. FEMA obligated about $795 million for these efforts as of July 20, 2018. According to the local utility, it took about 5 months for power to be restored to all customers with structures deemed safe for power restoration. Puerto Rico. In addition to the traditional types of support, FEMA and USACE undertook unprecedented roles of helping to coordinate and directly assist with grid restoration in Puerto Rico. FEMA requested that USACE lead federal grid repair efforts because of the scale of the damage and because the Puerto Rico Electric Power Authority (PREPA) did not have the capacity to respond, according to FEMA officials. FEMA obligated about $3.2 billion for electricity restoration efforts as of July 20, 2018, and PREPA estimated that it took roughly 11 months for power to be restored to all customers with structures deemed safe for power restoration. Various factors affected federal support for electricity grid restoration, according to officials GAO interviewed and documents reviewed. For example, getting the crews and materials needed to islands was more difficult and time-consuming than on the mainland. In Puerto Rico, PREPA was insolvent, which presented challenges for restoring the grid. For example, PREPA canceled its vegetation management program; this contributed to the destruction of the grid when the hurricane arrived, according to FEMA officials. In addition, FEMA did not anticipate or plan for the extensive federal role in grid restoration in Puerto Rico, and USACE did not have a contract in place to immediately initiate grid repair efforts, according to USACE officials. FEMA and USACE identified potential actions to address these challenges, such as reviewing advance contracts.", "document_type": "gao"}
{"report": "We found that the National Biodefense Strategy and associated plans bring together all the key elements of federal biodefense capabilities, which presents an opportunity to identify gaps and consider enterprise- wide risk and resources for investment trade-off decisions. However, challenges with planning to manage change; limited guidance and methods for analyzing capabilities; and lack of clarity about decision- making processes, roles, and responsibilities while adapting to a new enterprise-wide approach could limit the success of the Strategy’s implementation. The National Biodefense Strategy and its associated plans bring together the efforts of federal agencies with significant biodefense roles, responsibilities, and resources to address naturally-occurring, accidental, and intentional threats. The Strategy and plans also provide processes for collecting and analyzing comprehensive information across the enterprise, an important step toward the kind of enterprise-wide strategic decision-making we have called for. The Strategy defines the term “biothreat” broadly to include all sources of major catastrophic risk, including naturally-occurring biological threats, the accidental release of pathogens, and the deliberate use of biological weapons. Officials we interviewed noted that this is the first time that the federal government has identified activities across the whole biodefense enterprise and assessed resources and gaps to address multiple sources of threat regardless of source. The Strategy also outlines high-level goals and objectives to help define priorities. NSPM-14, which was issued to support the strategy, established a structure and process by which federal agencies can assess enterprise-wide biodefense capabilities and needs, and subsequently develop guidance to help inform agency budget submissions. NSPM-14 lays out, in broad strokes, a process to identify biodefense efforts and assess how current resources support the Strategy, how existing programs and resources could better align with the Strategy, and how additional resources, if available, could be applied to support the goals of the Strategy. As shown in figure 1, this process begins through a data call with participating agencies documenting all biodefense programs, projects, and activities within their purview in a biodefense memorandum. In interviews, officials from participating agencies stated that the NSPM- 14 processes constitute a new approach to identifying gaps and setting budget priorities for biodefense, and that they viewed the approach as generally well designed. Additionally, agency officials said that the assessment and joint policy guidance development process outlined in NSPM-14 offered some promise for helping agencies identify the resources necessary to achieve the Strategy’s goals. Nevertheless, officials from all of the agencies we interviewed, even those with the most optimistic views on the leadership and governance structure design, tempered their responses with the caveat that implementation is in such early stages that it remains to be seen how effective these structures will actually be once tested. In our February 2020 report, we also identified challenges that if not addressed could hinder enterprise-wide biodefense efforts. Specifically, although the Strategy and associated plans establish the foundation for enterprise risk management, we and biodefense agency officials identified multiple challenges that could affect the Strategy’s implementation. These include challenges individual agencies faced during the initial data collection process as well as a lack of planning and guidance to support an enterprise-wide approach. In our analyses and interviews, we found that parts of the process in the first year were underdeveloped, raising questions about (1) the plans to support change management practices and ensure that early-implementation limitations do not become institutionalized in future years’ efforts; (2) guidance and methods for meaningfully analyzing the data; and (3) the clarity of decision-making processes, roles, and responsibilities. Challenges adapting to new procedures. During our interviews, agency officials reported challenges they faced in the first-year’s data collection effort. These challenges may have led to incomplete data collection, but are not wholly unexpected given they occurred in the context of the individual agencies and officials adapting to new procedures and a broader cultural shift from how they have approached their biodefense missions in the past. Officials told us that because of the learning involved the first time through the process, agencies may not have submitted complete or detailed information about their biodefense programs. Some officials we interviewed voiced concern that this first-year effort could set a poor precedent for these activities in future years if the challenges are not acknowledged and addressed. For example, an official noted that committing to the first-year’s results as the “baseline” for future years of the Strategy’s implementation could compound or institutionalize the issues encountered in the first year. Officials from HHS and Office of Management and Budget staff stressed that this process will be iterative, with the first year being primarily about outlining the existing biodefense landscape. Our prior work on organizational transformations states that incorporating change management practices improves the likelihood of successful reforms and notes that it is important to recognize agency cultural factors that can either help or inhibit reform efforts. However, the agencies involved in implementing the Strategy do not have a plan that includes change management practices that can help prevent these challenges from being carried forward into future efforts, and help reinforce enterprise-wide approaches, among other things. To address this issue, we recommended the Secretary of HHS direct the Biodefense Coordination Team to establish a plan that includes change management practices—such as strategies for feedback, communication, and education—to reinforce collaborative behaviors and enterprise-wide approaches and to help prevent early implementation challenges from becoming institutionalized. HHS concurred with this recommendation. Guidance and methods for analyzing data. We found a lack of clear procedures and planning to help ensure that the Biodefense Coordination Team is prepared to analyze the data, once it has been collected, in a way that leads to recognition of meaningful opportunities to leverage resources in efforts to maintain and advance national biodefence capabilities. In particular, HHS (1) has not documented guidance and methods for analyzing the data, including but not limited to methods and guidance for how to account for the contribution of nonfederal capabilities; and (2) does not have a resource plan for staffing and sustaining ongoing efforts. Specifically, we found that the processes for the Biodefense Coordination Team to analyze the results of all the individual agency data submissions and identify priorities to guide resource allocation were not agreed upon or documented prior to the agency efforts and continue to lack specificity and transparency. In our interviews, officials from four agencies said they were uncertain about fundamental elements of the implementation process, including how information gathered will be used to identify gaps and set priorities. Additionally, the initial effort to collect information on all programs, projects, and activities focused on existing federal activities and did not include a complete assessment of biodefense capabilities at the nonfederal level ̶ capabilities needed to achieve the goals and objectives outlined in the Strategy. Officials we interviewed also expressed concern about the resources that the Biodefense Coordination Team had available to it, both in the first year and on an ongoing basis. The officials told us that not all agencies were able to provide a full-time detailee to help support the team. We have previously reported that agencies need to identify how interagency groups will be funded and staffed. However, officials from multiple agencies told us that the initial planning for the staffing and responsibilities for the Biodefense Coordination Team had not been finalized. Without a plan to help ensure sufficient resources and mitigate resource challenges for ongoing efforts, the Biodefense Coordination Team risks not having the capacity it needs to conduct meaningful analysis, which would undermine the vision created by the Strategy and NSPM-14. To address these issues, we recommended the Secretary of HHS direct the Biodefense Coordination Team to (1) clearly document guidance and methods for analyzing the data collected from the agencies, including ensuring that nonfederal resources and capabilities are accounted for in the analysis, and (2) establish a resource plan to staff, support, and sustain its ongoing efforts. HHS concurred with both recommendations. Roles and responsibilities for joint decision-making. The governing bodies overseeing the National Biodefense Strategy’s implementation— the Biodefense Steering Committee and Biodefense Coordination Team—did not clearly document key components of the assessment process and roles and responsibilities for joint decision-making in the first year of NSPM-14 implementation. This raises questions about how these bodies will move from an effort to catalog all existing activities to decision- making that accounts for enterprise-wide needs and opportunities. For example, officials from multiple agencies were not certain how the governing bodies would make joint decisions regarding priority-setting and the allocation of resources, how they would assign new biodefense responsibilities if gaps were identified, and to what extent the Biodefense Steering Committee could enforce budgetary priorities, if at all. We also found a lack of shared understanding and agreement about how the interagency process would work to align resources toward any identified gaps and reconfigure resources for any identified redundancies or inefficiencies. Additionally, we found that Presidential memorandums guiding the process did not detail specific decision-making principles or steps for reaching consensus or even for raising decision points about how to best leverage or direct resources across the enterprise in response to any gaps or inefficiencies. Similarly, agency officials we interviewed were not clear how this process would work, how decisions would be made, or how agencies would agree to take on new responsibilities to bridge gaps to achieve the Strategy’s goals. Further, the governing bodies have not fully defined the roles and responsibilities for making enterprise-wide decisions that affect individual agency budgets and for enforcing enterprise-wide budget priorities. As with other parts of the NSPM-14 implementation process, the details regarding specific roles and responsibilities for directing and enforcing budget decisions lack detail and specificity. Additionally, officials from four agencies stated that the charter for the Biodefense Coordination Team has not been finalized, further delaying the articulation of roles and responsibilities and the ability to establish a shared agenda and common operating picture. As a result, some officials remain skeptical of the effectiveness of any decisions made. We previously reported that effective national strategies should help clarify implementing organizations’ relationships in terms of leading, supporting, and partnering. In the context of the Strategy, that includes how enterprise-wide decisions about leveraging or directing resources to fill gaps and reduce inefficiency will be made and by whom. Similarly, our previous work has found that articulating and agreeing to a process for making and enforcing decisions and clarifying roles and responsibilities can improve the clarity surrounding a shared outcome, and that articulating these agreements in formal documents can strengthen agency commitment to working collaboratively and provide the overall framework for accountability and oversight. Uncertainty around the mechanisms to identify enterprise-wide priorities along with the lack of clearly documented and agreed upon processes, roles, and responsibilities for joint decision-making jeopardize the Strategy’s ability to enhance efficiency and effectiveness of the nation’s biodefense capabilities. To address this issue, we recommended that the Secretary of HHS direct the Biodefense Coordination Team to clearly document agreed upon processes, roles, and responsibilities for making and enforcing enterprise-wide decisions. HHS concurred. In conclusion, the current COVID-19 outbreak demonstrates that responding to the ever-changing nature and broad array of biological threats is challenging. The National Biodefense Strategy calls for the need to improve state, local, tribal, territorial, private sector, federal, regional, and international surveillance systems and networks to contain, control and respond to biological incidents. As the current coronavirus outbreak continues to cross regional and international borders, the federal government must take necessary steps to protect the American public. At the same time, we must not lose sight of the next threat. The National Biodefense Strategy and NSPM-14 put in place a framework to be able to assess threats and make difficult decisions about how to apply limited resources to achieve the best benefit. However, the Strategy is only as good as its implementation. Taking the necessary steps to address the recommendations we have made regarding managing this cultural change, analyzing data, ensuring sufficient resources to maintain implementation efforts, and clearly articulating roles and responsibilities for joint decision-making will better position our nation for the threats we face today and in the future. Chairwoman Maloney, Ranking Member Jordan, and Members of the Committee, this concludes our prepared statement. We would be happy to respond to any questions you may have at this time. If you or your staff has any questions concerning this testimony, please contact Christopher P. Currie at (404) 679-1875, CurrieC@gao.gov or Mary Denigan-Macauley at (202) 512-7114, DeniganMacauleyM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Kathryn Godfrey (Assistant Director), Susanna Kuebler (Analyst- In-Charge), Michele Fejfar, Eric Hauswirth, Tracey King, and Jan Montgomery. Key contributors for the previous work that this testimony is based on are listed in each product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "GAO has reported on the inherent fragmented nature of the federal and nonfederal resources needed to protect the nation from potentially catastrophic biological threats. GAO called for a strategic approach to help the federal government better leverage resources and manage risk The White House issued the National Biodefense Strategy and the Presidential Memorandum on the Support for National Biodefense to promote a more efficient and coordinated biodefense enterprise. The National Defense Authorization Act for Fiscal Year 2017 included a provision that GAO review the strategy. This testimony highlights key findings from our February 2020 report, which analyzed the extent to which the Strategy and related implementation efforts are designed to allow an enterprise-wide approach. Issued in September 2018, the National Biodefense Strategy (Strategy) and implementation plan, along with National Security Presidential Memorandum-14 (NSPM-14), are designed to enhance national biodefense capabilities. NSPM-14 established a governance structure composed of relevant federal agencies and chaired by the Secretary of Health and Human Services (HHS) to guide implementation. It also required federal agencies with biodefense responsibilities to collect and assess data on their biodefense activities to, among other things, identify gaps. There are a number of challenges, however, that could limit long-term implementation success. Among other things, there was no documented methodology or guidance for how data are to be analyzed to help the enterprise identify gaps and opportunities to leverage resources, including no guidance on how nonfederal capabilities are to be accounted for in the analysis. Agency officials were also unsure how decisions would be made, especially if addressing gaps or opportunities to leverage resources involved redirecting resources across agency boundaries. Although HHS officials pointed to existing processes and directives for interagency decision making, GAO found there are no clear, detailed processes, roles, and responsibilities for joint decision-making, including how agencies will identify opportunities to leverage resources or who will make and enforce those decisions. As a result, questions remain about how this first-year effort to catalogue all existing activities will result in a decision-making approach that involves jointly defining and managing risk at the enterprise level. Without clearly documented methods, guidance, processes, and roles and responsibilities for enterprise-wide decision-making, the effort runs the risk of failing to move away from traditional mission stovepipes toward a strategic enterprise-wide approach that meaningfully enhances national capabilities. In the February 2020 report, GAO made four recommendations to the Secretary of HHS, including working with other agencies to document methods for analysis and the processes, roles, and responsibilities for enterprise-wide decision making. HHS concurred with all the recommendations and described steps to implement them.", "document_type": "gao"}
{"report": "As shown in table 1 the cost of counting the nation’s population has been escalating with each decade. The 2010 Census was the most expensive in U.S. history at about $12.3 billion, and was about 31 percent more costly than the $9.4 billion 2000 Census (in 2020 dollars). According to the Bureau, the total cost of the 2020 Census in October 2015 was estimated at $12.3 billion and in October 2017 that cost estimate grew to approximately $15.6 billion, approximately a $3 billion increase. Additionally, Bureau officials told us that while the estimated cost of the census had increased to $15.6 billion, it was nevertheless managing the 2020 Census to a lower cost of $14.1 billion. Bureau officials explained that the $14.1 billion includes all program costs and contingency funds to cover risks and general estimating uncertainty. The remaining $1.5 billion estimated cost is additional contingency for “unknown unknowns”—that is, low probability events that could cause massive disruptions—and several what-if scenarios such as an increase in the wage rate or additional supervisors needed to manage field operations. Moreover, as shown in figure 1, the average cost for counting a housing unit increased from about $16 in 1970 to around $92 in 2010 (in 2020 constant dollars). At the same time, the return of census questionnaires by mail (the primary mode of data collection) declined over this period from 78 percent in 1970 to 63 percent in 2010. Declining mail response rates have led to higher costs because the Bureau sends temporary workers to each non-responding household to obtain census data. Achieving a complete and accurate census has become an increasingly daunting task, in part, because the population is growing larger, more diverse, and more reluctant to participate in the enumeration. In many ways, the Bureau has had to invest substantially more resources each decade to conduct the enumeration. In addition to these external societal challenges that make achieving a complete count a daunting task, the Bureau also faces a number of internal management challenges that affect its capacity and readiness to conduct a cost-effective enumeration. Some of these issues—such as acquiring and developing IT systems and preparing reliable cost estimates—are long-standing in nature. At the same time, as the Bureau looks toward 2020, it has faced emerging and evolving uncertainties. For example, on March 26, 2018, the Secretary of Commerce announced his decision to add a question to the decennial census on citizenship status which resulted in various legislative actions and legal challenges. Ultimately, the case was heard by the U.S. Supreme Court, which, in a June 26, 2019, ruling, prevented the addition of the question because the Court found that the evidence Commerce provided in the case did not match the Secretary’s explanation. In addition, the Fourth Circuit Court of Appeals remanded other legal challenges to the district court on June 24, 2019, for further legal action, which is yet to be resolved. According to Bureau officials, on June 28, 2019, Commerce asked the Bureau to put its scheduled July 1 start date for printing questionnaires on hold while it considered legal implications of the Supreme Court ruling. On July 2, 2019, Commerce told the Bureau to proceed with printing questionnaires and other materials without the citizenship question on them. As of July 5, 2019, the Department of Justice (DOJ) indicated that, although printing was continuing without the citizenship question, DOJ was evaluating legal options to include the question. On July 11, 2019, the President announced that instead of collecting this information from the census questionnaire, he ordered all federal agencies to provide data on citizenship status to Commerce using legally available federal records. We have not analyzed this decision or its implications, if any, for how the Bureau will tabulate its official counts. We will continue to monitor developments for Congress. The Bureau also faced budgetary uncertainties that, according to the Bureau, led to the curtailment of testing in 2017 and 2018. However, the Consolidated Appropriations Act, 2018 appropriated for the Periodic Censuses and Programs account $2.544 billion, which more than doubled the Bureau’s request in the President’s Fiscal Year 2018 Budget of $1.251 billion. According to the explanatory statement accompanying the act, the appropriation, which is available through fiscal year 2020, was provided to ensure the Bureau has the necessary resources to immediately address any issues discovered during operational testing, and to provide a smoother transition between fiscal year 2018 and fiscal year 2019. The availability of those resources enabled the Bureau to continue preparations for the 2020 Census during the 35 days in December 2018 to January 2019 when appropriations lapsed for the Bureau and a number of other federal agencies. Moreover, the Consolidated Appropriations Act, 2019 appropriated for the Periodic Censuses and Programs account $3.551 billion. According to Bureau officials, this level of funding for fiscal year 2019 is sufficient to carry out 2020 Census activities as planned. Importantly, the census is conducted against a backdrop of immutable deadlines. In order to meet the statutory deadline for completing the enumeration, census activities need to take place at specific times and in the proper sequence. Thus, it is absolutely critical for the Bureau to stay on schedule. Figure 2 shows some dates for selected decennial events. The Bureau has begun to open its area census offices (ACO) for the 2020 Census. It has signed leases for all 248 ACOs, of which 39 of the offices will be open for the address canvassing operation set to begin in August 2019 where staff verifies the location of selected housing units. The remaining 209 offices will begin opening this fall. In 2010 the Bureau opened 494 census offices. The Bureau has been able to reduce its infrastructure because it is relying on automation to assign work and to record payroll. Therefore there is less paper—field assignments, maps, and daily payroll forms—to manually process. For the 2020 Census, the Bureau is refining its recruiting and hiring goals, but tentatively plans to recruit approximately 2.24 million applicants and to hire over 400,000 temporary field staff from that applicant pool for two key operations: address canvassing, and nonresponse follow-up, where they visit households that do not return census forms to collect data in person. In 2010 the Bureau recruited 3.8 million applicants and hired 628,000 temporary workers to conduct the address canvassing and nonresponse follow-up field operations. According to Bureau officials, it has reduced the number of temporary staff it needs to hire because automation has made field operations more efficient and there is less paper. As of June 2019, the Bureau reported that for all 2020 Census operations it had processed about 430,000 applicants. In addition, the Bureau was seeking to hire approximately 1,500 partnership specialists by the end of June 2019 to help increase census awareness and participation in minority communities and hard-to-reach populations. As of July 9, 2019, the Bureau’s latest biweekly reporting indicated that it had hired 813 partnership specialists as of June 22, 2019. Moreover, as of July 10, 2019, Bureau officials told us that another 830 applicants were waiting to have their background checks completed. According to Bureau officials, hiring data are based on payroll dates generated biweekly, while background check data are tracked internally. Therefore, according to Bureau officials, more current hiring data were not available as of July 10, 2019 to indicate whether the Bureau had met its June 30 hiring goal. Among other things, partnership specialists are expected to either provide or identify partners to help provide supplemental language support to respondents locally in over 100 different languages. We will continue to monitor the Bureau’s progress in meeting its partnership specialist staffing goals and addressing any turnover that takes place. Hiring partnership specialists in a timely manner and maintaining adequate partnership specialist staffing levels are key to the Bureau’s ability to carry out its planned outreach efforts, especially to hard-to-count communities. Moreover, Bureau officials also stated that the current economic environment (i.e., the low unemployment rate compared to the economic environment of the 2010 Census) has not yet impacted their ability to recruit staff. The Bureau will continue to monitor the impact of low unemployment on its ability to recruit and hire at the local and regional levels. For the 2020 Census, the Bureau is substantially changing how it intends to conduct the census, in part by re-engineering key census-taking methods and infrastructure, and making use of new IT applications and systems. For example, the Bureau plans to offer an option for households to respond to the survey via the internet and enable field-based enumerators to use applications on mobile devices to collect survey data from households. To do this, the Bureau plans to utilize 52 new and legacy IT systems, and the infrastructure supporting them, to conduct the 2020 Census. A majority of these 52 systems have been tested during operational tests in 2017 and 2018. For example, the Bureau conducted its 2018 End-to- End test, which included 44 of the 52 systems and was intended to test all key systems and operations in a census-like environment to ensure readiness for the 2020 Census. Nevertheless, additional IT development and testing work needs to take place before the 2020 Census. Specifically, officials from the Bureau’s Decennial Directorate said they expect that the systems will need to undergo further development and testing due to, among other things, the need to add functionality that was not part of the End-to-End test, scale system performance to support the number of respondents expected during the 2020 Census, and address system defects identified during the 2018 End-to-End test. To prepare the systems and technology for the 2020 Census, the Bureau is also relying on substantial contractor support. For example, it is relying on contractors to develop a number of systems and components of the IT infrastructure, including the IT platform that is intended to be used to collect data from households responding via the internet and telephone, and for non-response follow-up activities. Contractors are also deploying the IT and telecommunications hardware in the field offices and providing device-as-a-service capabilities by procuring the mobile devices and cellular service to be used for non-response follow-up. In addition to the development of technology, the Bureau is relying on a technical integration contractor to integrate all of the key systems and infrastructure. The contractor’s work is expected to include, among other things, evaluating the systems and infrastructure and acquiring the infrastructure (e.g., cloud or data center) to meet the Bureau’s scalability and performance needs; integrating all of the systems; and assisting with technical, performance and scalability, and operational testing activities. In February 2017, we added the 2020 Decennial Census as a high-risk area needing attention from Congress and the executive branch. This was due to significant risks related to, among other things, innovations never before used in prior enumerations, the acquisition and development of IT systems, and expected escalating costs. Among other things, we reported that the commitment of top leadership was needed to ensure the Bureau’s management, culture, and business practices align with a cost-effective enumeration. We also stressed that the Bureau needed to rigorously test census-taking activities; ensure that scheduling adheres to best practices; improve its ability to manage, develop, and secure its IT systems; and have better oversight and control over its cost estimation process. Our experience has shown that agencies are most successful at removal from our High-Risk List when leaders give top level attention to the five criteria for removal and Congress takes any needed action. The five criteria for removal that we identified in November 2000 are as follows: Leadership Commitment. The agency has demonstrated strong commitment and top leadership support. Capacity. The agency has the capacity (i.e., people and resources) to resolve the risk(s). Action Plan. A corrective action plan exists that defines the root causes and solutions, and that provides for substantially completing corrective measures, including steps necessary to implement solutions we recommended. Monitoring. A program has been instituted to monitor and independently validate the effectiveness and sustainability of corrective measures. Demonstrated Progress. The agency has demonstrated progress in implementing corrective measures and in resolving the high-risk area. These five criteria form a road map for efforts to improve, and ultimately address, high-risk issues. Addressing some of the criteria leads to progress, while satisfying all of the criteria is central to removal from the list. As we reported in the March 2019 high-risk report, the Bureau’s efforts to address the risks and challenges for the 2020 Census had fully met one of the five criteria for removal from the High-Risk List—leadership commitment—and partially met the other four, as shown in figure 3. Additional details about the status of the Bureau’s efforts to address this high-risk area are discussed later in this statement. The 2020 Census is on our list of high-risk programs because, among other things, (1) innovations never before used in prior enumerations are not expected to be fully tested, (2) the Bureau continues to face challenges in implementing IT systems, (3) the Bureau faces significant cybersecurity risks to its systems and data, and (4) the Bureau’s cost estimate for the 2020 Census was unreliable. If not sufficiently addressed, these risks could adversely impact the cost and quality of the enumeration. Moreover, the risks are compounded by other factors that contribute to the challenge of conducting a successful census, such as the nation’s increasingly diverse population and concerns over personal privacy. The basic design of the enumeration—mail out and mail back of the census questionnaire with in-person follow-up for non-respondents—has been in use since 1970. However, a lesson learned from the 2010 Census and earlier enumerations is that this traditional design is no longer capable of cost-effectively counting the population. In response to its own assessments, our recommendations, and studies by other organizations, the Bureau has fundamentally re-examined its approach for conducting the 2020 Census. Specifically, its plan for 2020 includes four broad innovation areas: re-engineering field operations, using administrative records, verifying addresses in-office, and developing an internet self-response option (see table 2). If they function as planned, the Bureau initially estimated that these innovations could result in savings of over $5 billion (in 2020 constant dollars) when compared to its estimates of the cost for conducting the census with traditional methods. However, in June 2016, we reported that the Bureau’s initial life-cycle cost estimate developed in October 2015 was not reliable and did not adequately account for risk. As discussed earlier in this statement, the Bureau has updated its estimate from $12.3 billion and now estimates a life-cycle cost of $15.6 billion, which would result in a smaller potential savings from the innovative design than the Bureau originally estimated. According to the Bureau, the goal of the cost estimate increase was to ensure quality was fully addressed. While the planned innovations could help control costs, they also introduce new risks, in part, because they include new procedures and technology that have not been used extensively in earlier decennials, if at all. Our prior work has shown the importance of the Bureau conducting a robust testing program, including the 2018 End-to-End test. Rigorous testing is a critical risk mitigation strategy because it provides information on the feasibility and performance of individual census-taking activities, their potential for achieving desired results, and the extent to which they are able to function together under full operational conditions. To address some of these challenges we have made numerous recommendations aimed at improving reengineered field operations, using administrative records, verifying the accuracy of the address list, and securing census responses via the internet. The Bureau has held a series of operational tests since 2012, but according to the Bureau, it scaled back its most recent field tests because of funding uncertainties. For example, the Bureau canceled the field components of the 2017 Census Test including non-response follow-up, a key census operation. In November 2016, we reported that the cancelation of the 2017 Census Test was a lost opportunity to test, refine, and integrate operations and systems, and that it put more pressure on the 2018 End-to-End test to demonstrate that enumeration activities will function under census-like conditions as needed for 2020. However, in May 2017, the Bureau scaled back the operational scope of the 2018 End-to-End test and, of the three planned test sites, only the Rhode Island site would fully implement the 2018 End-to-End test. The Washington and West Virginia sites would test just one field operation. In addition, due to budgetary concerns, the Bureau delayed ramp up and preparations for its coverage measurement operation (and the technology that supports it) from the scope of the test. However, removal of the coverage measurement operation did not affect testing of the delivery of apportionment or redistricting data. Without sufficient testing, operational problems can go undiscovered and the opportunity to improve operations will be lost, in part because the 2018 End-to-End test was the last opportunity to demonstrate census technology and procedures across a range of geographic locations, housing types, and demographic groups under decennial-like conditions prior to the 2020 Census. We reported on the 2018 End-to-End test in December 2018 and noted that the Bureau had made progress addressing prior test implementation issues but still faced challenges. As the Bureau studies the results of its testing to inform the 2020 Census, it will be important that it addresses key program management issues that arose during implementation of the test. Namely, by not aligning the skills, responsibilities, and information flows for the first-line supervisors during field data collection, the Bureau limited its role in support of enumerators within the re-engineered field operation. The Bureau also lacked mid-operation training or guidance, which, if implemented in a targeted, localized manner, could have further helped enumerators navigate procedural modifications and any commonly encountered problems when enumerating. It will be important for the Bureau to prioritize its mitigation strategies for these implementation issues so that it can maximize readiness for the 2020 Census. To manage risk to the 2020 Census the Bureau has developed hundreds of risk mitigation and contingency plans. Mitigation plans detail how an agency will reduce the likelihood of a risk event and its impacts, if it occurs. Contingency plans identify how an agency will reduce or recover from the impact of a risk after it has been realized. In May 2019, we reported that the Bureau had identified 360 active risks to the 2020 census as of December 2018—meaning the risk event could still occur and adversely impact the census. Of these, 242 met the Bureau’s criteria for requiring a mitigation plan and, according to the Bureau’s risk registers, 232 had one (see table 3). In addition, 146 risks met the Bureau’s criteria for requiring a contingency plan and, according to the Bureau’s risk registers, 102 had one. Bureau guidance states that these plans should be developed as soon as possible after a risk is added to the risk register, but it does not establish a clear time frame for doing so. Consequently, some risks may go without required plans for extended periods. We found that, as of December 2018, some of the risks without required plans had been added to the Bureau’s risk registers in recent months, but others had been added more than 3 years earlier. We reviewed the mitigation and contingency plans in detail for six risks which the Bureau identified as among the major concerns that could affect the 2020 Census. These included cybersecurity incidents, late operational design changes, and integration of the 52 systems and 35 operations supporting the 2020 Census. We found that the plans did not consistently include key information needed to manage the risk. For example, the Bureau’s contingency plan for late operational design changes did not include activities specific to the three most likely late operational design changes—including removal of the citizenship question as a result of litigation or congressional action—that the Bureau could carry out to lessen their adverse impact on the enumeration, should they occur. We found that gaps stemmed from either requirements missing from the Bureau’s decennial risk management plan, or that risk owners—the individuals assigned to manage each risk—were not fulfilling all of their risk management responsibilities. Bureau officials said that risk owners are aware of these responsibilities but do not always fulfill them given competing demands. Bureau officials also said that they are managing risks to the census, even if not always reflected in their mitigation and contingency plans. However, if such actions are reflected in disparate documents or are not documented at all, then decision makers are left without an integrated and comprehensive picture of how the Bureau is managing risks to the census. We made seven recommendations to improve the Bureau’s management of risks to the 2020 Census, including that the Bureau develop mitigation and contingency plans for all risks that require them, establish a clear time frame for plan development, and ensure that the plans have the information needed to manage the risk. Commerce agreed with our recommendations and said it would develop an action plan to address them. We have previously reported that the Bureau faces challenges in managing and overseeing IT programs, systems, and contractors supporting the 2020 Census. Specifically, we have noted challenges in the Bureau’s efforts to manage, among other things, the schedules and contracts for its systems. As a result of these challenges, the Bureau is at risk of being unable to fully implement the systems necessary to support the 2020 Census and conduct a cost-effective enumeration. To help improve its implementation of IT for the 2020 Census, the Bureau revised its systems development and testing schedule. Specifically, in October 2018, the Bureau organized the development and testing schedule for its 52 systems into 16 operational deliveries. Each of the 16 operational deliveries has milestone dates for, among other things, development, performance and scalability testing, and system deployment. According to Bureau officials in the Decennial Directorate, the schedule was revised, in part, due to schedule management challenges experienced, and lessons learned, while completing development and testing during the 2018 End-to-End test. The Bureau has made initial progress in executing work against its revised schedule. For example, the Bureau completed development of the systems in the first operational delivery—for 2020 Census early operations preparations—in July 2018, and deployed these systems into production in October 2018. However, our current work has determined that the Bureau is at risk of not meeting several near-term systems testing milestones. As of June 2019, 11 systems that are expected to be used in a total of five operational deliveries were at risk of not meeting key milestones for completing system development, performance and scalability testing, and/or integration testing. These 11 systems are needed for, among other things, data collection for operations, business and support automation, and customer support during self-response. Figure 4 presents an overview of the status for all 16 operational deliveries, as of June 2019. The at-risk systems previously discussed add uncertainty to a highly compressed time frame over the next 6 months. Importantly, between July and December 2019, the Bureau is expected to be in the process of integration testing the systems in 12 operational deliveries. Officials from the Bureau’s integration contractor noted concern that the current schedule leaves little room for any delays in completing the remaining development and testing activities. In addition to managing the compressed testing time frames, the Bureau also has to quickly finalize plans related to its IT infrastructure. For example, as of June 2019, the Bureau stated that it was still awaiting final approval for its Trusted Internet Connection. Given that these plans may impact systems being tested this summer or deployed into production for the address canvassing operation in August 2019, it is important that the Bureau quickly addresses this matter. Our past reporting noted that the Bureau faced significant challenges in managing its schedule for system development and testing that occurred in 2017 and 2018. We reported that, while the Bureau had continued to make progress in developing and testing IT systems for the 2020 Census, it had experienced delays in developing systems to support the 2018 End-to-End test. These delays compressed the time available for system and integration testing and for security assessments. In addition, several systems experienced problems during the test. We noted then, and reaffirm now, that continued schedule management challenges may compress the time available for the remaining system and integration testing and increase the risk that systems may not function or be as secure as intended. The Bureau has acknowledged that it faces risks to the implementation of its systems and technology. As of May 2019, the Bureau had identified 17 high risks related to IT implementation that may have substantial technical and schedule impacts if realized. Taken together, these risks represent a cross-section of issues, such as schedule delays for a fraud-detection system, the effects of late changes to technical requirements, the need to ensure adequate time for system development and performance and scalability testing, contracting issues, privacy risks, and skilled staffing shortages. Going forward, it will be important that the Bureau effectively manages these risks to better ensure that it meets near-term milestones for system development and testing, and is ready for the major operations of the 2020 Census. The risks to IT systems supporting the federal government and its functions, including conducting the 2020 Census, are increasing as security threats continue to evolve and become more sophisticated. These risks include insider threats from witting or unwitting employees, escalating and emerging threats from around the globe, and the emergence of new and more destructive attacks. Underscoring the importance of this issue, we have designated information security as a government-wide high-risk area since 1997 and, in our most recent biennial report to Congress, ensuring the cybersecurity of the nation was one of nine high-risk areas that we reported needing especially focused executive and congressional attention. Our prior and ongoing work has identified significant challenges that the Bureau faces in securing systems and data for the 2020 Census. Specifically, the Bureau has faced challenges related to completing security assessments, addressing security weaknesses, resolving cybersecurity recommendations from DHS, and addressing numerous other cybersecurity concerns (such as phishing). Federal law specifies requirements for protecting federal information and information systems, such as those systems to be used in the 2020 Census. Specifically, the Federal Information Security Management Act of 2002 and the Federal Information Security Modernization Act of 2014 (FISMA) require executive branch agencies to develop, document, and implement an agency-wide program to provide security for the information and information systems that support operations and assets of the agency. In accordance with FISMA, National Institute of Standards and Technology (NIST) guidance, and Office of Management and Budget (OMB) guidance, the Bureau’s Office of the Chief Information Officer (CIO) established a risk management framework. This framework requires system developers to ensure that each of the Bureau’s systems undergoes a full security assessment, and that system developers remediate critical deficiencies. According to the Bureau’s risk management framework, the systems expected to be used to conduct the 2020 Census will need to have complete security documentation (such as system security plans) and an approved authorization to operate prior to their use. As of June 2019, according to the Bureau’s Office of the CIO: Thirty-seven of the 52 systems have authorization to operate, and will not need to be reauthorized before they are used in the 2020 Census Nine of the 52 systems have authorization to operate, and will need to be reauthorized before they are used in the 2020 Census Five of the 52 systems do not have authorization to operate, and will need to be authorized before they are used in the 2020 Census One of the 52 systems does not need an authorization to operate. Figure 5 summarizes the authorization to operate status for the systems being used in the 2020 Census, as reported by the Bureau in June 2019. As we have previously reported, while large-scale technological changes (such as internet self-response) increase the likelihood of efficiency and effectiveness gains, they also introduce many cybersecurity challenges. The 2020 Census also involves collecting personally identifiable information (PII) on over a hundred million households across the country, which further increases the need to properly secure these systems. Thus, it will be important that the Bureau provides adequate time to perform these security assessments, completes them in a timely manner, and ensures that risks are at an acceptable level before the systems are deployed. We have ongoing work examining how the Bureau plans to address both internal and external cyber threats, including its efforts to complete system security assessments and resolve identified weaknesses. FISMA requires that agency-wide information security programs include a process for planning, implementing, evaluating, and documenting remedial actions (i.e., corrective actions) to address any deficiencies in the information security policies, procedures, and practices of the agency. Additionally, the Bureau’s framework requires it to track security assessment findings that need to be remediated as a plan of action and milestones (POA&M). These POA&Ms are expected to provide a description of the vulnerabilities identified during the security assessment that resulted from a control weakness. As of the end of May 2019, the Bureau had over 330 open POA&Ms to remediate for issues identified during security assessment activities, including ongoing continuous monitoring. Of these open POA&Ms, 217 (or about 65 percent) were considered “high-risk” or “very high-risk.” While the Bureau established POA&Ms for addressing these identified security control weaknesses, it did not always complete remedial actions in accordance with its established deadlines. For example, of the 217 open “high-risk” or “very high-risk” POA&Ms we reviewed, the Bureau identified 104 as being delayed. Further, 74 of the 104 had missed their scheduled completion dates by 60 or more days. According to the Bureau’s Office of Information Security, these POA&Ms were identified as delayed due to technical challenges or resource constraints to remediate and close them. We previously recommended that the Bureau take steps to ensure that identified corrective actions for cybersecurity weaknesses are implemented within prescribed time frames. As of late May 2019, the Bureau was working to address our recommendation. Until the Bureau resolves identified vulnerabilities in a timely manner, it faces an increased risk, as continuing opportunities exist for unauthorized individuals to exploit these weaknesses and gain access to sensitive information and systems. The Bureau is working with federal and industry partners, including DHS, to support the 2020 Census cybersecurity efforts. Specifically, the Bureau is working with DHS to ensure a scalable and secure network connection for the 2020 Census respondents (e.g., virtual Trusted Internet Connection with the cloud), improve its cybersecurity posture (e.g., risk management processes and procedures), and strengthen its response to potential cyber threats (e.g., federal cyber incident coordination). Federal law describes practices for strengthening cybersecurity by documenting or tracking corrective actions. As previously mentioned, FISMA requires executive branch agencies to establish a process for planning, implementing, evaluating, and documenting remedial actions to address any deficiencies in their information security policies, procedures, and practices. Standards for Internal Control in the Federal Government calls for agencies to establish effective internal control monitoring that includes a process to promptly resolve the findings of audits and other reviews. Specifically, agencies should document and complete corrective actions to remediate identified deficiencies on a timely basis. This would include correcting identified deficiencies or demonstrating that the findings and recommendations do not warrant agency action. Since January 2017, DHS has been providing cybersecurity assistance (including issuing recommendations) to the Bureau in preparation for the 2020 Census. Specifically, DHS has been providing cybersecurity assistance to the Bureau in five areas: management coordination and executive support, including a cybersecurity threat intelligence and information sharing enhancement through, among other things, a DHS cyber threat briefing to the Bureau’s leadership; network and infrastructure security and resilience, including National Cybersecurity Protection System (also called EINSTEIN) support; incident response and management readiness through a Federal Incident Response Evaluation assessment; and risk management and vulnerability assessments for specific high value assets provided by the Bureau. In the last 2 years, DHS has provided 42 recommendations to assist the Bureau in strengthening its cybersecurity efforts. Among other things, the recommendations pertained to strengthening cyber incident management capabilities, penetration testing and web application assessments of select systems, and phishing assessments to gain access to sensitive PII. Of the 42 recommendations, 10 recommendations resulted from DHS’s mandatory services for the Bureau (e.g., risk management and vulnerability assessments for specific high value assets). The remaining 32 recommendations resulted from DHS’s voluntary services for the Bureau (e.g., Federal Incident Response Evaluation assessment). Due to the sensitive nature of the recommendations, we are not identifying the specific recommendations or specific findings associated with them in this statement. In April 2019, we reported that the Bureau had not established a formal process for documenting, tracking, and completing corrective actions for all of the recommendations provided by DHS. Accordingly, we recommended that the Bureau implement a formal process for tracking and executing appropriate corrective actions to remediate cybersecurity findings identified by DHS. As of late May 2019, the Bureau was working to address our recommendation. Until the Bureau implements our recommendation, it faces an increased likelihood that findings identified by DHS will go uncorrected and may be exploited to cause harm to agency’s 2020 Census IT systems and gain access to sensitive respondent data. Implementing a formal process would also help to ensure that DHS’s efforts result in improvements to the Bureau’s cybersecurity posture. The Bureau faces other substantial cybersecurity challenges in addition to those previously discussed. More specifically, we previously reported that the extensive use of IT systems to support the 2020 Census redesign may help increase efficiency, but that this redesign introduces critical cybersecurity challenges. These challenges include those related to the following: Phishing. We have previously reported that advanced persistent threats may be targeted against social media web sites used by the federal government. In addition, attackers may use social media to collect information and launch attacks against federal information systems through social engineering, such as phishing. Phishing attacks could target respondents, as well as Bureau employees and contractors. The 2020 Census will be the first one in which respondents will be heavily encouraged to respond via the internet. This will likely increase the risk that cyber criminals will use phishing in an attempt to steal personal information. According to the Bureau, it plans to inform the public of the risks associated with phishing through its education and communication campaigns. Disinformation from social media. We previously reported that one of the Bureau’s key innovations for the 2020 Census is the large-scale implementation of an internet self-response option. The Bureau is encouraging the public to use the internet self-response option through expanded use of social media. However, the public perception of the Bureau’s ability to adequately safeguard the privacy and confidentiality of the 2020 Census internet self-responses could be influenced by disinformation spread through social media. According to the Bureau, if a substantial segment of the public is not convinced that the Bureau can safeguard public response data against data breaches and unauthorized use, then response rates may be lower than projected, leading to an increase in cases for follow-up and subsequent cost increases. To help address this challenge, the Bureau stated that it plans to inform the public of the risks associated with disinformation from social media through its education and communication campaigns. Ensuring that individuals gain only limited and appropriate access to 2020 Census data. The Bureau plans to enable a public- facing website and Bureau-issued mobile devices to collect PII (e.g., name, address, and date of birth) from the nation’s entire population— estimated to be over 300 million. In addition, the Bureau is planning to obtain and store administrative records containing PII from other government agencies to help augment information that enumerators did not collect. The number of reported security incidents involving PII at federal agencies has increased dramatically in recent years. Because of these challenges, we have recommended, among other things, that federal agencies improve their response to information security incidents and data breaches involving PII, and consistently develop and implement privacy policies and procedures. Accordingly, it will be important for the Bureau to ensure that only respondents and Bureau officials are able to gain access to this information, and enumerators and other employees only have access to the information needed to perform their jobs. Ensuring adequate control in a cloud environment. The Bureau has decided to use cloud solutions as a key component of the 2020 Census IT infrastructure. We have previously reported that cloud computing has both positive and negative information security implications and, thus, federal agencies should develop service-level agreements with cloud providers. These agreements should specify, among other things, the security performance requirements—including data reliability, preservation, privacy, and access rights—that the service provider is to meet. Without these safeguards, computer systems and networks, as well as the critical operations and key infrastructures they support, may be lost; information—including sensitive personal information—may be compromised; and the agency’s operations could be disrupted. Commerce’s Office of the Inspector General recently identified several challenges the Bureau may face using cloud-based systems to support the 2020 Census. Specifically, in June 2019, the Office of the Inspector General identified, among other things, unimplemented security system features that left critical 2020 Census systems vulnerable during the 2018 End-to-End Test and a lack of fully implemented security practices to protect certain data hosted in the 2020 Census cloud environment. Officials from the Bureau agreed with all eight of the Office of Inspector General’s recommendations regarding 2020 Census cloud-based systems and identified actions taken to address them. Ensuring contingency and incident response plans are in place to encompass all of the IT systems to be used to support the 2020 Census. Because of the brief time frame for collecting data during the 2020 Census, it is especially important that systems are available for respondents to ensure a high response rate. Contingency planning and incident response help ensure that, if normal operations are interrupted, network managers will be able to detect, mitigate, and recover from a service disruption while preserving access to vital information. Implementing important security controls, including policies, procedures, and techniques for contingency planning and incident response, helps to ensure the confidentiality, integrity, and availability of information and systems, even during disruptions of service. Without contingency and incident response plans, system availability might be impacted and result in a lower response rate. The Bureau’s CIO has acknowledged these cybersecurity challenges and is working to address them, according to Bureau documentation. In addition, we have ongoing work looking at many of these challenges, including the Bureau’s plans to protect PII, use a cloud-based infrastructure, and recover from security incidents and other disasters. Since 2015, the Bureau has made progress in improving its ability to develop a reliable cost estimate. We have reported on the reliability of the $12.3 billion life-cycle cost estimate released in October 2015 and the $15.6 billion revised cost estimate released in October 2017. In 2016 we reported that the October 2015 version of the Bureau’s life-cycle cost estimate for the 2020 Census was not reliable. Specifically, we found that the 2020 Census life-cycle cost estimate partially met two of the characteristics of a reliable cost estimate (comprehensive and accurate) and minimally met the other two (well-documented and credible). We recommended that the Bureau take specific steps to ensure its cost estimate meets the characteristics of a high-quality estimate. The Bureau agreed and has taken action to improve the reliability of the cost estimate. In August 2018 we reported that while improvements had been made, the Bureau’s October 2017 cost estimate for the 2020 Census did not fully reflect all the characteristics of a reliable estimate. (See figure 6.) In order for a cost estimate to be deemed reliable as described in GAO’s Cost Estimating and Assessment Guide and thus, to effectively inform 2020 Census annual budgetary figures, the cost estimate must meet or substantially meet the following four characteristics: Well-Documented. Cost estimates are considered valid if they are well-documented to the point they can be easily repeated or updated and can be traced to original sources through auditing, according to best practices. Accurate. Accurate estimates are unbiased and contain few mathematical mistakes. Credible. Credible cost estimates must clearly identify limitations due to uncertainty or bias surrounding the data or assumptions, according to best practices. Comprehensive. To be comprehensive an estimate should have enough detail to ensure that cost elements are neither omitted nor double-counted, and all cost-influencing assumptions are detailed in the estimate’s documentation, among other things, according to best practices. The 2017 cost estimate only partially met the characteristic of being well- documented. In general, some documentation was missing, inconsistent, or difficult to understand. Specifically, we found that source data did not always support the information described in the basis of estimate document or could not be found in the files provided for two of the Bureau’s largest field operations: Address Canvassing and Non- Response Follow-Up. We also found that some of the cost elements did not trace clearly to supporting spreadsheets and assumption documents. Failure to document an estimate in enough detail makes it more difficult to replicate calculations, or to detect possible errors in the estimate; reduces transparency of the estimation process; and can undermine the ability to use the information to improve future cost estimates or even to reconcile the estimate with another independent cost estimate. The Bureau told us it would continue to make improvements to ensure the estimate is well- documented. The 2017 life-cycle cost estimate includes much higher costs than those included in the 2015 estimate. The largest increases occurred in the Response, Managerial Contingency, and Census/Survey Engineering categories. For example, increased costs of $1.3 billion in the response category (costs related to collecting, maintaining, and processing survey response data) were in part due to reduced assumptions for self- response rates, leading to increases in the amount of data collected in the field, which is more costly to the Bureau. Contingency allocations increased overall from $1.35 billion in 2015 to $2.6 billion in 2017, as the Bureau gained a greater understanding of risks facing the 2020 Census. Increases of $838 million in the Census/Survey Engineering category were due mainly to the cost of an IT contract for integrating decennial survey systems that was not included in the 2015 cost estimate. Bureau officials attribute a decrease of $551 million in estimated costs for Program Management to changes in the categorization of costs associated with risks. Specifically, in the 2017 version of the estimate, estimated costs related to program risks were allocated to their corresponding work breakdown structure (WBS) element. Figure 7 shows the change in cost by WBS category for 2015 and 2017. More generally, factors that contributed to cost fluctuations between the 2015 and 2017 cost estimates include: Changes in assumptions. Among other changes, a decrease in the assumed rate for self-response from 63.5 percent in 2015 to 60.5 percent in 2017 increased the cost of collecting responses from nonresponding housing units. Improved ability to anticipate and quantify risk. In general, contingency allocations designed to address the effects of potential risks increased overall from $1.3 billion in 2015 to $2.6 billion in 2017. An overall increase in IT costs. IT cost increases, totaling $1.59 billion, represented almost 50 percent of the total cost increase from 2015 to 2017. More defined contract requirements. Bureau documents described an overall improvement in the Bureau’s ability to define and specify contract requirements. This resulted in updated estimates for several contracts, including for the Census Questionnaire Assistance contract. However, while the Bureau has been able to better quantify risk; in August 2018 we also reported that the Secretary of Commerce included a contingency amount of about $1.2 billion in the 2017 cost estimate to account for what the Bureau refers to as “unknown unknowns.” According to Bureau documentation these include such risks as natural disasters or cyber attacks. The Bureau provides a description of how the risk contingency for “unknown unknowns” is calculated; however, this description does not clearly link calculated amounts to the risks themselves. Thus, only $14.4 billion of the Bureau’s $15.6 billion cost estimate has justification. According to Bureau officials, the cost estimate remains at $15.6 billion; however, they stated that they are managing the 2020 Census at a lower level of funding—$14.1 billion. In addition, they said that, at this time, they do not plan to request funding for the $1.2 billion contingency fund for unknown unknowns or $369 million in funding for selected discrete program risks for what-if scenarios, such as an increase in the wage rate or additional supervisors needed to manage field operations. Instead of requesting funding for these contingencies upfront the Bureau plans to work with OMB and Commerce to request additional funds, if the need arises. According to Bureau officials they anticipate that the remaining $1.1 billion in contingency funding included in the $14.1 billion will be sufficient to carry out the 2020 Census. In June 2016 we recommended the Bureau improve control over how risk and uncertainty are accounted for. This prior recommendation remains valid given the life-cycle cost estimate still includes the $1.2 billion unjustified contingency fund for “unknown unknowns”. Moreover, given the cost growth between 2015 and 2017 it will be important for the Bureau to monitor cost in real-time, as well as, document, explain and review variances between planned and actual cost. In August 2018 we reported that the Bureau had not been tracking variances between estimated life-cycle costs and actual expenses. Tools to track variance enable management to measure progress against planned outcomes and will help inform the 2030 Census cost estimate. Bureau officials stated that they already have systems in place that can be adapted for tracking estimated and actual costs. We will continue to monitor the status of the tracking system. According to Bureau officials, the Bureau planned to release an updated version of the 2020 Census life-cycle estimate in the spring of 2019; however, they had not done so as of June 28, 2019. To ensure that future updates to the life-cycle cost estimate reflect best practices, it will be important for the Bureau to implement our recommendation related to the cost estimate. The difficulties facing the Bureau’s preparation for the decennial census in such areas as planning and testing; managing and overseeing IT programs, systems, and contractors supporting the enumeration; developing reliable cost estimates; prioritizing decisions; managing schedules; and other challenges, are symptomatic of deeper organizational issues. Following the 2010 Census, a key lesson learned for 2020 that we identified was ensuring that the Bureau’s organizational culture and structure, as well as its approach to strategic planning, human capital management, internal collaboration, knowledge sharing, capital decision- making, risk and change management, and other internal functions are aligned toward delivering more cost-effective outcomes. The Bureau has made improvements over the last decade, and continued progress will depend in part on sustaining efforts to strengthen risk management activities, enhancing systems testing, bringing in experienced personnel to key positions, implementing our recommendations, and meeting regularly with officials from its parent agency, Commerce. Going forward, we have reported that the key elements needed to make progress in high-risk areas are top-level attention by the administration and agency officials to (1) leadership commitment, (2) ensuring capacity, (3) developing a corrective action plan, (4) regular monitoring, and (5) demonstrated progress. Although important steps have been taken in at least some of these areas, overall, far more work is needed. We discuss three of five areas below. The Secretary of Commerce has successfully demonstrated leadership commitment. For example, the Bureau and Commerce have strengthened this area with executive-level oversight of the 2020 Census by holding regular meetings on the status of IT systems and other risk areas. In addition, in 2017 Commerce designated a team to assist senior Bureau management with cost estimation challenges. Moreover, on January 2, 2019, a new Director of the Census Bureau took office, a position that had been vacant since June 2017. With regard to capacity, the Bureau has improved the cost estimation process of the decennial when it established guidance including: roles and responsibilities for oversight and approval of cost estimation processes, procedures requiring a detailed description of the steps taken to produce a high-quality cost estimate, and a process for updating the cost estimate and associated documents over the life of a project. However, the Bureau continues to experience skills gaps in the government program management office overseeing the $886 million contract for integrating the IT systems needed to conduct the 2020 Census. Specifically, as of June 2019, 14 of 44 positions in this office were vacant. For the monitoring element, we found to track performance of decennial census operations, the Bureau relied on reports to track progress against pre-set goals for a test conducted in 2018. According to the Bureau, these same reports will be used in 2020 to track progress. However, the Bureau’s schedule for developing IT systems during the 2018 End-to-End test experienced delays that compressed the time available for system testing, integration testing, and security assessments. These schedule delays contributed to systems experiencing problems after deployment, as well as cybersecurity challenges. In the months ahead, we will continue to monitor the Bureau’s progress in addressing each of the five elements essential for reducing the risk to a cost-effective enumeration. Over the past several years we have issued numerous reports that underscored the fact that, if the Bureau was to successfully meet its cost savings goal for the 2020 Census, the agency needed to take significant actions to improve its research, testing, planning, scheduling, cost estimation, system development, and IT security practices. As of June 2019, we have made 106 recommendations related to the 2020 Census. The Bureau has implemented 74 of these recommendations, 31 remain open, and one recommendation was closed as not implemented. Of the 31 open recommendations, 9 were directed at improving the implementation of the innovations for the 2020 Census. Commerce generally agreed with our recommendations and is taking steps to implement them. Moreover, in April 2019 we wrote to the Secretary of Commerce, providing a list of the 12 open 2020-Census-related recommendations that we designated as “priority.” Priority recommendations are those recommendations that we believe warrant priority attention from heads of key departments and agencies. We believe that attention to these recommendations is essential for a cost-effective enumeration. The recommendations included implementing reliable cost estimation and scheduling practices in order to establish better control over program costs, as well as taking steps to better position the Bureau to develop an internet response option for the 2020 Census. In addition to our recommendations, to better position the Bureau for a more cost-effective enumeration, on March 18, 2019, we met with OMB, Commerce, and Bureau officials to discuss the Bureau’s progress in reducing the risks facing the census. We also meet regularly with Bureau officials and managers to discuss the progress and status of open recommendations related to the 2020 Census, which has resulted in Bureau actions in recent months leading to closure of some recommendations. We are encouraged by this commitment by Commerce and the Bureau in addressing our recommendations. Implementing our recommendations in a complete and timely manner is important because it could improve the management of the 2020 Census and help to mitigate continued risks. In conclusion, while the Bureau has made progress in revamping its approach to the census, it faces considerable challenges and uncertainties in implementing key cost-saving innovations and ensuring they function under operational conditions; managing the development and testing of its IT systems; ensuring the cybersecurity of its systems and data; and developing a quality cost estimate for the 2020 Census and preventing further cost increases. For these reasons, the 2020 Census is a GAO high-risk area. Going forward, continued management attention and oversight will be vital for ensuring that risks are managed, preparations stay on track, and the Bureau is held accountable for implementing the enumeration, as planned. Without timely and appropriate actions, the challenges previously discussed could adversely affect the cost, accuracy, schedule, and security of the enumeration. We will continue to assess the Bureau’s efforts and look forward to keeping Congress informed of the Bureau’s progress. Chairman Johnson, Ranking Member Peters, and Members of the Committee, this completes our prepared statement. We would be pleased to respond to any questions that you may have. If you have any questions about this statement, please contact Robert Goldenkoff at (202) 512-2757 or by email at goldenkoffr@gao.gov or Nick Marinos at (202) 512-9342 or by email at marinosn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other key contributors to this testimony include Ty Mitchell (Assistant Director); Lisa Pearson (Assistant Director); Jon Ticehurst (Assistant Director); Emmy Rhine Paule (Analyst in Charge); Christopher Businsky; Jackie Chapin; Jeff DeMarco; Rebecca Eyler; Adella Francis; Scott Pettis; Lindsey Pilver; Kayla Robinson; Robert Robinson; Cindy Saunders; Sejal Sheth; Kevin R. Smith; Andrea Starosciak; and Umesh Thakkar. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Bureau is responsible for conducting a complete and accurate decennial census of the U.S. population. The decennial census is mandated by the Constitution and provides vital data for the nation. A complete count of the nation's population is an enormous undertaking as the Bureau seeks to control the cost of the census, implement operational innovations, and use new and modified IT systems. In recent years, GAO has identified challenges that raise serious concerns about the Bureau's ability to conduct a cost-effective count. For these reasons, GAO added the 2020 Census to its High-Risk list in February 2017. GAO was asked to testify about the reasons the 2020 Census remains on the High-Risk List and the steps the Bureau needs to take to mitigate risks to a successful census. To do so, GAO summarized its prior work regarding the Bureau's planning efforts for the 2020 Census. GAO also included preliminary observations from its ongoing work examining the IT systems readiness and cybersecurity for the 2020 Census. This information is related to, among other things, the Bureau's progress in developing and testing key systems and the status of cybersecurity risks. The 2020 Decennial Census is on GAO's list of high-risk programs primarily because the Department of Commerce's Census Bureau (Bureau) (1) is using innovations that are not expected to be fully tested, (2) continues to face challenges in implementing information technology (IT) systems, and (3) faces significant cybersecurity risks to its systems and data. Although the Bureau has taken initial steps to address risk, additional actions are needed as these risks could adversely impact the cost, quality, schedule, and security of the enumeration. Innovations. The Bureau is planning several innovations for the 2020 Census, including allowing the public to respond using the internet. These innovations show promise for controlling costs, but they also introduce new risks, in part, because they have not been used extensively, if at all, in earlier enumerations. As a result, testing is essential to ensure that key IT systems and operations will function as planned. However, citing budgetary uncertainties, the Bureau scaled back operational tests in 2017 and 2018, missing an opportunity to fully demonstrate that the innovations and IT systems will function as intended during the 2020 Census. To manage risk to the census, the Bureau has developed hundreds of mitigation and contingency plans. To maximize readiness for the 2020 Census, it will also be important for the Bureau to prioritize among its mitigation and contingency strategies those that will deliver the most cost-effective outcomes for the census. Implementing IT systems. The Bureau plans to rely heavily on IT for the 2020 Census, including a total of 52 new and legacy IT systems and the infrastructure supporting them. To help improve its implementation of IT, in October 2018, the Bureau revised its systems development and testing schedule to reflect, among other things, lessons learned during its 2018 operational test. However, GAO's ongoing work has determined that the Bureau is at risk of not meeting near-term IT system development and testing schedule milestones for five upcoming 2020 Census operational deliveries, including self-response (e.g., the ability to respond to the 2020 Census through the internet). These schedule management challenges may compress the time available for the remaining system development and testing, and increase the risk that systems will not function as intended. It will be important that the Bureau effectively manages IT implementation risk to ensure that it meets near-term milestones for system development and testing, and that it is ready for the major operations of the 2020 Census. To its credit, the Bureau is also working with the Department of Homeland Security (DHS) to support its 2020 Census cybersecurity efforts. For example, DHS is helping the Bureau ensure a scalable and secure network connection for the 2020 Census respondents and to strengthen its response to potential cyber threats. During the last 2 years, as a result of these activities, the Bureau has received 42 recommendations from DHS to improve its cybersecurity posture. GAO recently recommended that the Bureau implement a formal process for tracking and executing appropriate corrective actions to remediate cybersecurity findings identified by DHS. Implementing the recommendation would help better ensure that DHS's efforts result in improvements to the Bureau's cybersecurity posture. In addition to addressing risks which could affect innovations and the security of the enumeration, the Bureau has the opportunity to improve its cost estimating process for the 2020 Census, and ultimately the reliability of the estimate itself, by reflecting best practices. In October 2017, the 2020 Census life-cycle cost estimate was updated and is now projected to be $15.6 billion, a more than $3 billion (27 percent) increase over its earlier estimate. GAO reported in August 2018 that although the Bureau had taken steps to improve its cost estimation process for 2020, it needed to implement a system to track and report variances between actual and estimated cost elements. According to Bureau officials, they planned to release an updated version of the 2020 Census life-cycle estimate in the spring of 2019; however, they had not done so as of June 28, 2019. To ensure that future updates to the life-cycle cost estimate reflect best practices, it will be important for the Bureau to implement GAO's recommendation related to the cost estimate. Over the past decade, GAO has made 106 recommendations specific to the 2020 Census to help address these risks and other concerns. The Department of Commerce has generally agreed with these recommendations and has taken action to address many of them. However, as of June 2019, 31 of the recommendations had not been fully implemented. While all 31 open recommendations are important for a high-quality and cost-effective enumeration, 9 are directed at managing the risks introduced by the Bureau's planned innovations for the 2020 Census. To ensure a high-quality and cost-effective enumeration, it will be important for the Bureau to address these recommendations. Over the past decade, GAO has made 106 recommendations specific to the 2020 Census to help address issues raised in this and other products. The Department of Commerce has generally agreed with the recommendations. As of June 2019, 31 of the recommendations had not been fully implemented.", "document_type": "gao"}
{"report": "For over 20 years, Congress has enacted various laws, and federal agencies have issued guidance, that call for agencies to perform workforce planning activities to help ensure the timely and effective acquisition of IT. These laws and guidance focus on the importance of (1) setting the strategic direction for workforce planning, (2) analyzing the workforce to identify skill gaps, (3) developing strategies to address skill gaps, and (4) monitoring and reporting on progress in addressing skill gaps. For example: The Clinger-Cohen Act of 1996 requires agency chief information officers (CIO) to annually (1) assess the requirements established for agency personnel regarding knowledge and skills in information resource management and the adequacy of such requirements for facilitating the achievement of performance goals; (2) assess the extent to which the positions and personnel at executive and management levels meet those requirements; (3) develop strategies and specific plans for hiring, training, and professional development to address any deficiencies; and (4) report to the head of the agency on the progress made in improving information resources management capability. The E-Government Act of 2002 requires the Director of OPM, in consultation with the Director of OMB, the CIO Council, and the Administrator of General Services to (1) analyze, on an ongoing basis, the personnel needs of the federal government related to IT and information resource management; and (2) identify where current IT and information resource management training do not satisfy personnel needs. In addition, the law requires the Director of OMB to ensure that agency heads collect and maintain standardized information on their IT and information resources management workforce. In 2010, OMB issued its 25-point plan for IT reform and outlined several action plans to build workforce capabilities, including capabilities for acquisition and program management. For example, the plan directed OPM to create a specialized career path for IT program managers. In addition, OMB stated that it would work with OPM to provide agencies with direct hiring authority for program managers. OMB also tasked agencies with identifying program management competency gaps and reporting to OMB on those gaps. Subsequent to the 25-point plan, in July 2011, OMB released guidance for agencies to develop specialized IT acquisition cadres. Among other things, this guidance required agencies to analyze current acquisition staffing challenges; determine if developing or expanding the use of cadres would improve program results; and outline a plan to pilot or expand cadres for an especially high-risk area, if the agency determined that such an effort would improve performance. Further, in November 2011, OPM issued guidance for developing career paths for IT program managers. OPM’s career path guide was to build upon its IT Program Management Competency Model released in July 2011 by serving as a roadmap for individuals interested in pursuing a career in this area. In addition, the roadmap was to provide employees and their supervisors with a single-source reference to determine appropriate training opportunities for career advancement. In December 2014, Congress enacted legislation commonly referred to as FITARA. Among other things, the law aims to ensure timely progress by federal agencies toward developing, strengthening, and deploying IT acquisition cadres consisting of personnel with highly specialized skills in IT acquisition, including program and project managers. Almost all of the 24 CFO Act agencies (other than the Department of Defense (Defense)) are required to update their annual acquisition human capital plans to address how they are meeting their human capital requirements to support timely and effective acquisitions. To assist agencies in implementing the provisions of FITARA and to build upon agency responsibilities under the Clinger-Cohen Act of 1996, OMB issued guidance to agencies in June 2015. In doing so, OMB directed agencies (other than Defense) to, among other things, (1) develop a set of competency requirements for staff, including leadership positions; and (2) develop and maintain a current workforce planning processes to ensure that agencies can anticipate and respond to changing mission requirements, maintain workforce skills in a rapidly developing environment, and recruit and retain the talent needed to accomplish their missions. Each agency is to conduct an annual self-assessment of its conformity with these requirements and develop an implementation plan describing the changes it will make. The Federal Cybersecurity Workforce Assessment Act of 2015 required OPM, with support from the National Institute of Standards and Technology, to establish a coding structure to be used in identifying all federal civilian and noncivilian positions that require the performance of IT, cybersecurity, or other cyber-related functions. The act also required agencies, in consultation with OPM, the National Institute of Standards and Technology, and the Department of Homeland Security (DHS), to then utilize this coding structure to annually assess, among other things, the IT, cybersecurity, and other cyber-related work roles of critical need in their workforce. In April 2016, OPM issued an update to agency chief human capital officers stating that it had recently revalidated the need to continue working to close skill gaps in certain government-wide high-risk mission critical occupations, including those in the cybersecurity and the science, technology, engineering and mathematics functional area. OMB released its Federal Cybersecurity Workforce Strategy in July 2016. Among other things, the strategy cited the need for agencies to examine specific IT, cybersecurity, and cyber-related work roles, and identify personnel skills gaps, rather than merely examining the number of vacancies by job series. The strategy identified several actions that agencies could take to identify workforce needs, expand the cybersecurity workforce through education and training, recruit and hire highly skilled talent, and retain and develop highly skilled talent. In July 2016, OMB issued updated policy for the planning, budgeting, governance, acquisition, and management of federal information, personnel, equipment, funds, IT resources, and supporting infrastructure and services. Among other things, OMB’s updated policy requires an agency’s chief human capital officer, CIO, chief acquisition officer, and senior agency official for privacy to develop a set of competency requirements for staff and develop and maintain a current workforce planning process. Further, in September 2016, OPM updated its guidance regarding the annual submission of agencies’ mission critical occupation resource charts. These charts are to identify current staffing levels, staffing targets, projected attrition, actual attrition, and retirement eligibility in government-wide and selected agency-specific mission critical occupations. While these laws and guidance focus on IT workforce planning, other broader initiatives have also been undertaken to improve federal human capital management. For example, we and OPM have developed human capital management models that call for implementing workforce planning practices that can facilitate the analysis of gaps between current skills and future needs. In addition, the models call for the development of strategies for filling the gaps, as well as planning for succession. Further, our Standards for Internal Control in the Federal Government stress that management should consider how best to retain valuable employees, plan for their eventual succession, and ensure continuity of needed skills and abilities. Based on the aforementioned laws, guidance, and initiatives, in November 2016, GAO issued an evaluation framework to support the assessment of whether selected federal agencies are adequately assessing and addressing gaps in IT knowledge and skills. The framework identifies four workforce planning steps and supporting activities that address (1) setting the strategic direction for IT workforce planning, (2) analyzing the IT workforce to identify competency and staffing gaps, (3) developing and implementing strategies to address the gaps, and (4) monitoring and reporting progress in addressing the gaps. We have previously reported that effectively addressing mission critical skill gaps in IT requires a multifaceted response from OPM and agencies. Specifically, our high-risk update in February 2013 noted that OPM and agencies would need to use a strategic approach that (1) involves top management, employees, and other stakeholders; (2) identifies the critical skills and competencies that will be needed to achieve current and future programmatic results; (3) develops strategies that are tailored to address skill gaps; (4) builds the internal capability needed to address administrative, training, and other requirements important to support workforce planning strategies; and (5) includes plans to monitor and evaluate progress toward closing skill gaps and meeting other human capital goals using a variety of appropriate metrics. In January 2015, we reported that the Chief Human Capital Officers Council had identified skill gaps in six government-wide occupations including IT/cybersecurity and contract specialist/acquisition. We noted, however, that the effort had shortcomings, and that it would be important for the council to use lessons learned from these initial efforts to inform subsequent ones to identify skill gaps. We also reported that key features of OPM’s efforts to predict emerging skill gaps beyond those already identified were in the early planning stages, and OPM and selected agencies could improve the manner in which they address skill gaps by strengthening their use of quarterly data-driven reviews. Further, we reported that individual agencies across the federal government have not always effectively planned for IT workforce challenges. For example, In May 2014, we concluded that the Social Security Administration’s (SSA) IT human capital program had identified skills and competencies to support certain workforce needs, but lacked adequate planning for the future. The agency had developed IT human capital planning documents, such as an Information Resources Management plan, and skills inventory gap reports that identified near-term needs, such as skill sets for the following 2 years. Nevertheless, SSA had not adequately planned for longer-term needs because its human capital planning and analysis were not aligned with long-term goals and objectives and the agency did not have a current succession plan for its IT efforts. Accordingly, we recommended that SSA identify long-term IT needs in its updated human capital operating plan. The agency agreed with, and subsequently implemented the recommendation. In August 2016, we determined that the Department of Veterans Affairs (VA) had performed key steps, such as documenting an IT human capital strategic plan and regularly analyzing workforce data. However, the agency had not tracked and reviewed historical and projected leadership retirements and had not identified gaps in future skill areas. We recommended that the agency track and review historical workforce data and projections related to leadership retirements, and identify IT skills needed beyond the current fiscal year, to assist in identifying future skills gaps. The agency concurred with our recommendations and has partially implemented them by identifying the IT skills it needed beyond the current fiscal year. In November 2016, as a part of the review in which we developed the IT workforce planning framework discussed previously, we assessed five agencies—the Departments of Commerce (Commerce), Defense, Transportation (Transportation), the Treasury (Treasury), and Health and Human Services (HHS)—against the eight key workforce planning activities. While all five agencies had demonstrated important progress in either partially or fully implementing key workforce planning activities, each had shortfalls. For example, only one agency (Defense) had implemented a workforce planning process, none had identified IT competency gaps for their entire workforce, and three (Defense, Transportation, and Treasury) were performing some level of monitoring toward the closure of identified skill gaps. We reported that, until the agencies fully implemented key workforce planning activities, they would have a limited ability to assess and address gaps in knowledge and skills that are critical to the success of major IT acquisitions. As a result, we recommended that the agencies implement the eight IT workforce planning activities to facilitate the analysis of gaps between current skills and future needs, the development of strategies for filling the gaps, and succession planning. Defense partially agreed with our recommendations and the other four agencies agreed with our recommendations. An updated assessment of actions to implement our recommendations is described in our evaluation of agencies’ implementation of key IT workforce planning activities in appendix II. In May 2018, as part of a review of the National Aeronautics and Space Administration’s (NASA) approach to overseeing and managing IT, we found that the agency had partially implemented five of the eight key IT workforce planning activities and had not implemented three. For example, NASA had not assessed competency and staffing needs regularly or reported progress to agency leadership. We reported that, until the agency implemented the key IT workforce planning activities, it would have difficulty anticipating and responding to changing staffing needs. As a result, we recommended that NASA fully implement the eight key IT workforce planning activities. The agency disagreed with our recommendation stating that its workforce improvement activities were already underway. Nevertheless, implementing the workforce planning activities discussed in this report could enhance and complement the agency’s ongoing and future efforts. In a June 2018 report on the progress of agencies’ efforts to implement the requirements of the Federal Cybersecurity Workforce Assessment Act of 2015, we noted that most CFO Act agencies had developed baseline assessments to identify cybersecurity personnel within their agencies that held certifications. However, because agencies had not consistently defined the workforce and the National Initiative for Cybersecurity Education had not developed a list of appropriate certifications, efforts such as conducting the baseline assessment to determine the percentage of cybersecurity personnel that hold appropriate certifications had yielded inconsistent and potentially unreliable results. Further, we reported that, while most CFO Act agencies had developed procedures for assigning cybersecurity codes to positions, several agencies had not addressed activities required by OPM to implement the requirements of the Federal Cybersecurity Workforce Assessment Act. As a result, we made 30 recommendations to 13 agencies to develop and submit their baseline assessments and to fully address the required activities in OPM’s guidance in their procedures for assigning employment codes to cybersecurity positions. Of the 13 agencies, seven agreed with the recommendations made to them, four did not state whether they agreed or disagreed, one agency agreed with one of the two recommendations made to it, and one did not provide comments on the report. As of July 2019, the agencies had implemented 20 of the recommendations. In August 2018, as part of a government-wide review of CIO responsibilities, we reported that CIOs are responsible for assessing agency IT workforce needs and developing strategies and plans for meeting those needs. However, we noted that the majority of the agencies minimally addressed or did not address the role of their CIOs in the area of IT workforce and reported major challenges related to their IT workforce. Specifically, 19 agencies’ policies had not addressed their CIOs’ role in conducting annual assessments of IT management and skill requirements and the remaining five agencies had only partially addressed this responsibility. We noted that the shortcomings in agencies’ policies were attributable, at least in part, to incomplete guidance from OMB. Consequently, we recommended that OMB issue guidance that addresses the IT workforce responsibilities of CIOs that were not included in existing guidance. OMB partially agreed with the recommendation and has not yet implemented it. We also recommended that 24 agencies ensure that their IT management policies address the role of their CIOs in the IT workforce management area. Of the 24 agencies, 14 agreed with the recommendations, five had no comments, five partially agreed, and one disagreed. We are monitoring the status of the agencies’ actions to implement our recommendations. In March 2019, as part of an update on the status of agencies’ progress in implementing the requirements of the Federal Cybersecurity Workforce Assessment Act, we reported, among other things, that most of the 24 CFO Act agencies had not completely or accurately categorized work roles for IT positions within the 2210 IT management occupational series (IT management). The agencies reported that this was, in part, because they may have assigned the associated codes in error or had not completed validating the accuracy of the assigned codes. We noted that, by assigning work roles that are inconsistent with the IT, cybersecurity, and cyber-related positions, the agencies were diminishing the reliability of the information they needed to improve workforce planning. We made recommendations to 22 agencies to take steps to address the inaccuracies. Of these agencies, 20 agreed with the recommendations, one partially agreed, and one did not agree with one of the two recommendations. As of August 2019, three of the agencies have implemented their recommendation, and two of the agencies have implemented one of their two recommendations. We continue to believe that all of the recommendations are warranted. As previously noted, GAO issued an IT workforce planning framework that includes eight key activities, based on federal laws, guidance, and best practices. Implementing these activities is critical to adequately assessing and addressing gaps in IT knowledge, skills, and abilities that are needed to execute a range of management functions that support agencies’ missions and goals. The eight key workforce planning activities are identified in table 1. None of the 24 agencies that we reviewed had fully implemented all eight IT workforce planning activities. In this regard, nearly all of the agencies had partially implemented, substantially implemented, or fully implemented three of the workforce planning activities (develop competency and staffing requirements, assess competency and staffing needs regularly, and assess gaps in competencies and staffing). However, most agencies had minimally implemented or did not implement the five other workforce planning activities (including efforts to establish a workforce planning process and address staffing gaps). Figure 1 shows the agencies’ overall implementation of each of the eight key IT workforce planning activities, as of May 2019. Further, some agencies had made more progress than others. Specifically, while five agencies (Defense, Department of State (State), VA, Small Business Administration (SBA), and SSA) fully implemented or substantially implemented three or more activities, 11 agencies did not fully implement any of the activities, and 15 agencies did not implement three or more activities. Figure 2 identifies the extent to which each of the 24 agencies had implemented the eight workforce planning activities. In addition, appendix II provides our assessment of each agency’s implementation of the activities. To fully implement the establish and maintain an IT workforce planning process activity, an agency should have a documented IT workforce planning process that describes how the agency will implement key IT workforce planning activities, including those identified in the IT workforce planning framework. The process should also define the CIO’s and others’ roles and responsibilities for implementing the activities; align with mission goals and objectives; and address both the agency-level and component-level workforce, including how the agency is to maintain visibility and oversight into component-level workforce planning efforts (as applicable). In addition, the agency should periodically update the process. Only one of the 24 CFO Act agencies had fully implemented this activity. Specifically, one agency had fully implemented the activity (Nuclear Regulatory Commission (NRC)); one agency had substantially implemented the activity (Defense); two agencies had partially implemented the activity (Department of Housing and Urban Development (HUD), and SBA); 12 agencies had minimally implemented the activity (U.S. Department of Agriculture (Agriculture), Commerce, Department of Energy (Energy), HHS, DHS, Department of the Interior (Interior), Department of Labor (Labor), State, Transportation, Treasury, VA, and SSA); and eight agencies did not implement the activity (Department of Education (Education), Department of Justice (Justice), Environmental Protection Agency (EPA), General Services Administration (GSA), NASA, National Science Foundation (NSF), OPM, and U.S. Agency for International Development (USAID)). NRC fully implemented the activity. In February 2016, NRC developed a strategic workforce plan that addressed all key IT workforce planning activities in our framework. In addition, the process was aligned with the agency’s goals and objectives. Further, the process included general roles and responsibilities, including for the Office of the Chief Human Capital Officer, Senior Management, and its component offices. Moreover, the agency’s Management Directive 9.22 further defined the Chief Information Officer’s roles and responsibilities with regards to IT workforce planning. In addition, NRC has periodically updated the process. For example, the agency updated the process in July 2017 to better integrate its workload projection, skills identification, human capital management, individual development, and workforce management activities. Defense substantially implemented the activity. The agency’s June 2018 Human Capital Operating Plan addressed how Defense plans to implement the workforce planning activities for its functional communities, including the IT functional community. In addition, the plan defined the CIO’s roles and responsibilities and was aligned with the agency’s goals and objectives. Further, the plan documented how the agency will maintain oversight of and visibility into functional community planning efforts. However, it called for the functional communities to develop strategic workforce plans to further define their workforce planning process and the IT functional community has not yet completed its plan or provided a time frame for completion. With respect to maintaining the process, Defense periodically updated its IT workforce process—the June 2018 plan replaced the process identified in the agency’s previous workforce plans. SBA partially implemented the activity. In April 2018, SBA released its IT Workforce Plan for fiscal years 2018 through 2020 that addressed how the agency intends to implement all of its IT workforce planning activities, and was aligned with the agency’s mission goals and objectives. In addition, in April 2018, the agency released its IT Change Management and Communication Plan that defined the CIO’s IT workforce planning roles and responsibilities and was aligned with the agency’s mission goals and objectives. However, as it is a new process, SBA had not updated it as of May 2019. Interior minimally implemented the activity. Interior issued a policy in 2016 that directed its bureaus to develop IT workforce plans, which the agency stated that it intends to use to develop an agency-wide IT workforce plan. The policy identified efforts that should be addressed in the plans, including most of the IT workforce planning activities. However, as of May 2019, the bureaus’ plans and the agency-wide plan had not been completed. Officials in the Office of the CIO stated that they expect to finalize all of the plans by the end of fiscal year 2019. GSA did not implement the activity. Officials in the Human Capital Strategic Planning Division stated that GSA followed the process described in OPM’s IT workforce planning guidance; however, the agency did not document this in policy and had not developed any other documentation to guide its implementation of workforce planning activities. To fully implement the develop competency and staffing requirements activity, an agency should develop a set of competency requirements for all or most of its IT workforce, including leadership positions. In addition, the agency should develop staffing requirements, which include projections over several years. Most of the agencies had fully or substantially developed competency and staffing requirements. Specifically, 12 agencies had fully implemented the activity (Defense, Education, HUD, State, Transportation, Treasury, VA, GSA, NASA, SBA, SSA, and USAID), four agencies had substantially implemented the activity (Agriculture, Commerce, HHS, and DHS), and eight agencies had partially implemented the activity (Energy, Interior, Justice, Labor, EPA, NSF, NRC, and OPM). State fully implemented the activity. State developed competency requirements for its IT workforce, including for both its foreign and civil services. In addition, State developed staffing requirements for its IT staff, including projections over several years. Specifically, it developed staffing requirements for its mission critical occupations, which include IT management, in response to OPM’s requirement to submit this information annually. DHS substantially implemented the activity. DHS developed competency requirements for two of the agency’s four IT functional groups. According to officials in the Office of the CIO, the agency expects to finalize competency requirements for the remaining two groups by the end of fiscal year 2019. In addition, DHS developed staffing requirements for its IT staff, including projections over several years. Specifically, it developed staffing requirements for its mission critical occupations, which include IT management, in response to OPM’s requirement to submit this information annually. OPM partially implemented the activity. OPM did not develop competency requirements. However, the agency developed staffing requirements for its IT staff, including projections over several years. Specifically, it developed staffing requirements for its mission critical occupations, which include IT management. To fully implement the assess competency and staffing needs regularly activity, an agency should periodically assess competency needs for all or most of its IT workforce. In addition, the agency should periodically assess staffing needs for all or most of its IT workforce. Most of the agencies periodically assessed staffing needs, but did not assess competency needs. Specifically, three agencies had fully implemented the activity (Defense, VA, and SSA); 20 agencies had partially implemented the activity by periodically assessing IT staffing needs; however, these agencies did not periodically assess competency needs (Agriculture, Commerce, Education, Energy, HHS, DHS, HUD, Interior, Justice, Labor, State, Transportation, Treasury, GSA, NASA, NSF, NRC, OPM, SBA, and USAID); and one agency did not implement the activity (EPA). VA fully implemented the activity. VA assessed competency needs annually as a part of its professional development planning process. For example, the agency performed an assessment in fiscal year 2017, which led it to add project management as a competency for all IT staff. In addition, in fiscal year 2018, VA’s assessment resulted in adding two new competencies—data analytics and risk management. Further, VA annually assessed staffing needs for its IT staff in response to the annual OPM reporting requirement to do so. Commerce partially implemented the activity. The agency initially developed its competency requirements in January 2016, but had not since updated its needs. On the other hand, Commerce annually assessed staffing needs for its IT staff in response to the OPM reporting requirement to do so. EPA did not implement the activity. EPA did not develop competency needs for its IT workforce. In addition, the agency could not provide documentation showing that it had regularly assessed staffing needs for its IT staff. To fully implement the assess gaps in competencies and staffing activity, an agency should periodically assess gaps in competencies for all or most of its IT workforce. Further, the assessment should be performed based on the agency’s current competency needs. In addition, the agency should periodically assess gaps in staffing for all or most of its IT workforce. Most agencies took steps to assess competency and staffing gaps. Specifically, two agencies had fully implemented the activity (VA and SSA); nine agencies had substantially implemented the activity (Agriculture, Defense, DHS, HUD, State, Transportation, GSA, NASA, and SBA); 12 agencies had partially implemented the activity by periodically assessing IT staffing gaps, but not periodically assessing competency gaps (Commerce, Education, Energy, HHS, Interior, Justice, Labor, Treasury, NSF, NRC, OPM, and USAID); and one agency had minimally implemented the activity (EPA). SSA fully implemented the activity. SSA assessed gaps in its competencies for its IT management staff biennially starting in fiscal year 2014. In addition, SSA annually assessed staffing gaps for its IT staff in response to the OPM reporting requirement. HUD substantially implemented the activity. HUD assessed competency gaps for its IT management staff biennially; it began doing so in fiscal year 2014. However, HUD did not assess competency needs regularly; thus, it could not ensure that the gap assessments reflect the agency’s current competency needs. HUD annually assessed staffing gaps for its IT staff in response to the OPM reporting requirement. Education partially implemented the activity. Education did not assess gaps in competencies for its IT staff. However, the agency annually assessed staffing gaps for its IT staff in response to the OPM reporting requirement. EPA minimally implemented the activity. EPA did not assess competency gaps because, as previously stated, the agency did not develop competency requirements. In addition, while EPA assessed staffing gaps in 2018, it did not provide documentation showing that it had assessed staffing gaps prior to or since then. To fully implement the develop strategies and plans to address gaps in competencies and staffing activity, an agency should develop strategies and plans, including specific actions and milestones, to address identified competency gaps. In addition, the agency should develop strategies and plans, including specific actions and milestones, to address identified staffing gaps. Most agencies did not develop strategies and plans to address competency and staffing gaps. Specifically, four agencies had substantially implemented the activity (Defense, State, VA, and SBA), one agency had partially implemented the activity (Agriculture), six agencies had minimally implemented the activity (HUD, Transportation, EPA, GSA, SSA, and USAID), and 13 agencies did not implement the activity (Commerce, Education, Energy, HHS, DHS, Interior, Justice, Labor, Treasury, NASA, NSF, NRC, and OPM). State substantially implemented the activity. State identified strategies to address high-priority IT competency gaps, including developing additional training, conducting quarterly reviews of IT workforce issues, and improving hiring processes; however, it had not developed plans, including actions and milestones, for how it would carry out the strategies. With respect to staffing, State identified strategies and plans to address them in its Five-Year Workforce and Leadership Success Plan for Fiscal Years 2016 through 2020. For example, State identified using special hiring initiatives, such as its Pathways Programs, to address staffing gaps. In addition, State developed the Foreign Affairs IT Fellowship Program, which is intended to recruit students by offering internships. Agriculture partially implemented the activity. In 2019, Agriculture developed strategies, which included providing training and developing career paths, to address competency gaps identified for two of 13 IT functional roles; however, the agency did not develop associated plans, including actions and milestones. Further, Agriculture did not develop strategies to address gaps for the other 11 IT functional roles because the agency had not assessed gaps for those roles. With respect to staffing, in 2019, Agriculture identified strategies to address staffing gaps identified for two of its IT functional roles, including collaborating with universities. However, it did not develop plans to carry out the strategies. In addition, Agriculture did not develop strategies and plans to address gaps in staffing for its other 11 IT functional roles. HUD minimally implemented the activity. HUD’s Office of the CIO developed a training plan for fiscal years 2017 through 2018, which identified training courses to address specific technical competency gaps. However, HUD has not updated its competency needs regularly to ensure that the plan and underlying gap assessment reflect the agency’s current competency needs. With respect to staffing, HUD did not develop strategies and plans to address gaps. DHS did not implement the activity. DHS did not develop strategies and plans to address either competency or staffing gaps. To fully implement the implement activities that address gaps activity, an agency should execute its strategies and plans to address identified gaps in competencies and staffing. In addition, the agency should implement other efforts to assist with addressing competency and staffing needs, including the following efforts identified in FITARA: IT acquisition cadres, cross-functional training of acquisition and program personnel, career paths for program managers, plans to strengthen program management, and the use of special hiring authorities. Most of the agencies minimally implemented strategies and plans to address competency and staffing gaps. Specifically, two agencies had substantially implemented this activity (Defense and VA), seven agencies had partially implemented the activity (HHS, DHS, State, Treasury, SBA, SSA, and USAID), and 15 agencies had minimally implemented the activity by implementing workforce efforts identified in FITARA, but not implementing strategies and plans to address its identified competency and staffing gaps primarily because they had not developed strategies and plans to address identified gaps (Agriculture, Commerce, Education, Energy, HUD, Interior, Justice, Labor, Transportation, EPA, GSA, NASA, NSF, NRC, and OPM). VA substantially implemented the activity. VA implemented strategies and plans to address gaps in competencies. For example, in its Office of Information and Technology Training Gap Analysis report, VA identified actions taken to address the prior year’s competency gaps. These actions included developing additional training courses, as well as providing on-the-job training activities. However, VA did not provide documentation showing that it had implemented strategies and plans to address identified staffing gaps. With respect to the efforts identified in FITARA that can assist with addressing competency and staffing needs, VA implemented an IT acquisition cadre, developed plans to strengthen program management, developed a career path for program managers, and used special hiring authorities to hire IT staff. SSA partially implemented the activity. SSA implemented strategies to address gaps in competencies. For example, according to its gap closure report, the agency closed competency gaps by providing training to existing staff, hiring new staff, and hiring contractors with needed skills. However, SSA did not implement strategies and plans to address staffing gaps because it had not yet developed them. With respect to the efforts identified in FITARA that can assist with addressing competency and staffing needs, SSA used special hiring authorities to hire eight IT specialists in fiscal year 2018. However, SSA did not implement others, including IT acquisition cadres, cross- functional training of acquisition and program personnel, career paths for program managers, and plans to strengthen program management. GSA minimally implemented the activity. GSA did not develop strategies and plans to address identified gaps in competencies or staffing. With respect to the efforts identified in FITARA that can assist with addressing competency and staffing needs, GSA implemented efforts to provide cross-functional training for acquisition and program personnel and used special hiring authorities to hire IT staff. However, the agency did not implement others, including plans to strengthen program management or career paths for program managers. To fully implement the monitor the agency’s progress in addressing competency and staffing gaps activity, an agency should track progress in implementing strategies and plans to address competency gaps. In addition, the agency should track progress in implementing strategies and plans to address staffing gaps. Most agencies did not establish processes for monitoring progress in addressing competency and staffing gaps. Specifically, three agencies had partially implemented the activity (Defense, VA, and SBA), five agencies had minimally implemented the activity (HUD, State, Transportation, SSA, and USAID), and 16 agencies did not implement the activity (Agriculture, Commerce, Education, Energy, HHS, DHS, Interior, Justice, Labor, Treasury, EPA, GSA, NASA, NSF, NRC, and OPM). SBA partially implemented the activity. SBA established an IT Workforce Steering Committee which monitored progress made in implementing, among other things, strategies and plans to address competency and staffing gaps. However, the agency did not monitor whether the strategies and plans led to a closure in gaps. State minimally implemented the activity. While State monitored its progress in implementing recommended actions to address competency gaps, the agency did not monitor whether the actions led to closing gaps. With respect to staffing, State did not monitor progress in addressing gaps because it did not develop strategies and plans to close staffing gaps. GSA did not implement the activity. GSA did not track progress in addressing competency gaps because the agency did not assess competencies to identify such gaps. Further, GSA did not monitor its progress in addressing staffing gaps because it did not develop strategies and plans to close the gaps. To fully implement the report to agency leadership on progress activity, an agency should periodically report to agency leadership on progress in implementing strategies and plans to address gaps in competencies. In addition, the agency should periodically report to leadership on progress in implementing strategies and plans to address gaps in staffing. However, most of the agencies did not establish processes for reporting their progress in addressing competency and staffing gaps. Specifically, three agencies had partially implemented the activity (Defense, VA, and SBA), three had minimally implemented the activity (HUD, SSA, and USAID), and 18 did not implement the activity (Agriculture, Commerce, Education, Energy, HHS, DHS, Interior, Justice, Labor, State, Transportation, Treasury, EPA, GSA, NASA, NSF, NRC, and OPM). VA partially implemented the activity. VA reported to agency leadership on its progress in addressing competency gaps, including the closure of gaps, and the actions planned and taken to address the gaps. However, VA did not report on progress in addressing staffing gaps because it did not implement strategies and plans to address such gaps. HUD minimally implemented the activity. HUD reported to agency leadership on the closure of competency gaps from fiscal year 2014 through fiscal year 2016. However, the agency did not monitor or report on its progress in implementing strategies and plans to address gaps in competencies. With respect to staffing, HUD did not report on its progress in addressing gaps because it did not implement strategies and plans to close staffing gaps. DHS did not implement the activity. DHS did not periodically report to agency leadership on its progress in addressing competency or staffing gaps. The agency did not do so because it did not develop strategies and plans to address competency and staffing gaps. Agency officials cited various factors that limited their progress in implementing the key IT workforce planning activities. For example, six agencies, including DHS and NRC, reported that they had not completed key activities because they were reliant on finishing other prerequisite activities. For example, officials in DHS’s Office of the CIO stated that they had not updated their IT competency needs because they had not yet finished identifying competency requirements for all of the agency’s role-based groups; four agencies, including HHS and NASA, reported that they had other workforce related priorities, including those related to the Cybersecurity Workforce Assessment Act; three agencies, including GSA and USAID, reported that they lacked resources to perform the activities; and two agencies (OPM and Interior) reported that leadership turnover affected their implementation of workforce planning activities. Until agencies make it a priority to implement all of the key IT workforce planning activities, they will likely have a limited ability to assess and address gaps in the knowledge and skills that are critical to the success of major acquisitions. As a result, it will be difficult for agencies to anticipate and respond to changing staffing needs and control human capital risks when developing, implementing, and operating critical IT systems. The majority of the agencies made significant progress implementing three activities—develop competency and staffing requirements, assess competency and staffing needs regularly, and assess gaps in competencies and staffing—and in doing so took important steps towards identifying the workforce they need to help them achieve their mission, and the gaps that need to be addressed. In contrast, most agencies only minimally implemented or did not implement the remaining five activities, increasing the risk that they will not address the gaps. Agencies’ limited implementation of the IT workforce planning activities has been due, in part, to not making IT workforce planning a priority, despite the laws and guidance which have called for them to do so for over 20 years. Until this occurs, agencies will likely not have the staff with the necessary knowledge, skills, and abilities to support the agency’s mission and goals. We are making a total of 18 recommendations to federal agencies—one recommendation to 18 agencies. The Secretary of Agriculture should ensure that the agency fully implements each of the eight key IT workforce planning activities it did not fully implement. (Recommendation 1) The Secretary of Education should ensure that the agency fully implements each of the seven key IT workforce planning activities it did not fully implement. (Recommendation 2) The Secretary of Energy should ensure that the agency fully implements each of the eight key IT workforce planning activities it did not fully implement. (Recommendation 3) The Secretary of Homeland Security should ensure that the agency fully implements each of the eight key IT workforce planning activities it did not fully implement. (Recommendation 4) The Secretary of Housing and Urban Development should ensure that the agency fully implements each of the seven key IT workforce planning activities it did not fully implement. (Recommendation 5) The Secretary of the Interior should ensure that the agency fully implements each of the eight key IT workforce planning activities it did not fully implement. (Recommendation 6) The Attorney General should ensure that the agency fully implements each of the eight key IT workforce planning activities it did not fully implement. (Recommendation 7) The Secretary of Labor should ensure that the agency fully implements each of the eight key IT workforce planning activities it did not fully implement. (Recommendation 8) The Secretary of State should ensure that the agency fully implements each of the seven key IT workforce planning activities it did not fully implement. (Recommendation 9) The Secretary of Veterans Affairs should ensure that the agency fully implements each of the five key IT workforce planning activities it did not fully implement. (Recommendation 10) The Administrator of the Environmental Protection Agency should ensure that the agency fully implements each of the eight key IT workforce planning activities it did not fully implement. (Recommendation 11) The Administrator of the General Services Administration should ensure that the agency fully implements each of the seven key IT workforce planning activities it did not fully implement. (Recommendation 12) The Director of the National Science Foundation should ensure that the agency fully implements each of the eight key IT workforce planning activities it did not fully implement. (Recommendation 13) The Chairman of the Nuclear Regulatory Commission should ensure that the agency fully implements each of the seven key IT workforce planning activities it did not fully implement. (Recommendation 14) The Director of the Office of Personnel Management should ensure that the agency fully implements each of the eight key IT workforce planning activities it did not fully implement. (Recommendation 15) The Administrator of the Small Business Administration should ensure that the agency fully implements each of the seven key IT workforce planning activities it did not fully implement. (Recommendation 16) The Commissioner of the Social Security Administration should ensure that the agency fully implements each of the five key IT workforce planning activities it did not fully implement. (Recommendation 17) The Administrator of the U.S. Agency for International Development should ensure that the agency fully implements each of the seven key IT workforce planning activities it did not fully implement. (Recommendation 18) We are not making new recommendations to six agencies—Commerce, Defense, HHS, Transportation, Treasury, and NASA—because we previously made recommendations to these agencies to address the key IT workforce planning activities. We provided a draft of the report to the 24 CFO Act agencies for their review and comment. Of the 18 agencies to which we made a recommendation in this report, 13 agencies (Energy, DHS, HUD, Interior, Labor, State, VA, GSA, NSF, OPM, SBA, SSA, and USAID) agreed with the recommendation; one agency (Education) partially agreed with the recommendation; three agencies (Agriculture, Justice, EPA) neither agreed nor disagreed with the recommendation; and one agency (NRC) did not agree with our findings. We also received technical comments from a number of the agencies, which we have incorporated into the report, as appropriate. In addition, of the six agencies to which we did not make recommendations in this report, two (Defense and the Treasury) provided comments on the report and the remaining four (Commerce, HHS, Transportation, and NASA) responded that they did not have any comments on the report. The following 13 agencies agreed with our recommendations: In written comments (reprinted in appendix III), Energy concurred with our recommendation. The agency stated that it plans to fully implement all of the IT workforce planning activities, and described recently completed and intended efforts to do so. For example, the agency stated that it completed the development of competency and staffing requirements in May 2019. In addition, the agency said it expects to finish developing an IT workforce planning process in December 2020. While the efforts described represent positive steps toward fully implementing the IT workforce planning activities, Energy did not provide supporting documentation for the activities it said were completed. As a result, we did not change our ratings for these activities. In its written comments (reprinted in appendix IV), DHS concurred with our recommendation and stated that it remains committed to fully implementing all of the IT workforce planning activities. Further, the agency stated that it had completed developing competency requirements and assessing gaps for its two remaining IT role based groups. However, the agency did not provide documentation to support its completion of these activities. As a result, we did not change our ratings for the activities. DHS also stated that the Office of the Chief of Staff Workforce Engagement Division, within the Office of the CIO, plans to work with the agency’s Chief Information Officer Council and the Office of Chief Human Capital Officer to form an integrated project team by January 30, 2020. According to DHS, this project team will be charged with discussing the agency’s IT workforce planning strategy and outlining an action plan to ensure the strategy addresses all of the key IT workforce planning activities. DHS also provided technical comments which we incorporated, as appropriate. In written comments (reprinted in appendix V), HUD concurred with our recommendation and stated that it plans to fully implement the remaining workforce planning activities. In its written comments (reprinted in appendix VI), Interior agreed with our recommendation. The agency stated that it has begun taking steps to implement the IT workforce planning activities and plans to fully implement the remaining activities. In its written comments (reprinted in appendix VII), Labor concurred with our recommendation. The agency stated that it had made significant progress since the completion of our review and had fully implemented seven of the eight IT workforce planning activities. For example, the agency described efforts to review position descriptions, including identifying key IT competency areas. In addition, the agency stated that it assessed competency and skills needs, and critical IT skill gaps, as part of an IT workforce supply analysis. Further, Labor stated that, in June 2019, it developed hiring approval and prioritization templates, which require a current workforce and competency assessment, and identified IT competencies with each hiring request. The agency added that hiring managers perform a job analysis prior to posting open positions, and that this includes identifying key IT competencies for each position. Moreover, Labor stated that, in June 2019, the Secretary approved the use of direct hire authority for IT Specialists. In addition, the agency said that the Office of the CIO and the Chief Human Capital Officer finalized an action plan in March 2019 that identified strategies to address IT workforce gaps. Further, it stated that progress had been monitored in weekly discussions with and oversight from the Chief Information Officer and Chief Human Capital Officer. However, while the actions described indicate progress toward fully implementing the workforce planning activities, the agency did not provide evidence to support the actions it said it had taken. As a result, we did not change our ratings for the activities. In written comments (reprinted in appendix VIII), State agreed with our recommendation and described steps it said the agency is taking to implement the IT workforce planning activities. These steps included developing an IT strategic workforce plan that it expected to finalize by the end of fiscal year 2019. Further, the agency stated that it had substantially implemented the report to agency leadership on progress in addressing the competency and staffing gaps activity, which we assessed as not implemented. As evidence, the agency stated that departmental leadership is briefed regularly on efforts made to address IT competency gaps. However, State did not provide supporting documentation for these activities. As a result, we did not change our rating for the activities. In written comments (reprinted in appendix IX), VA concurred with our recommendation. However, the agency said it believed that it had fully implemented each of the five IT workforce planning activities we rated as less than fully implemented. Specifically, With regard to establishing and maintaining an IT workforce planning process, VA stated that its Office of Information and Technology had fully implemented a workforce planning process, including developing and implementing strategies to address gaps in competencies and staffing. The agency submitted two documents as supporting evidence: the Office of Information and Technology’s Human Capital Management Recruitment Strategy, which we reviewed during our engagement and determined did not sufficiently address the criteria; and the Office of Information and Technology’s Human Capital Strategic Plan for fiscal years 2014 through 2020, a document that it had not previously provided to us. We reviewed this document but have questions we need to follow up on with VA to determine whether the agency has fully implemented the activity. As a result, we did not change our rating for this activity. With regard to developing strategies and plans to address gaps in competencies and staffing, VA stated that, for projected staffing gaps, it has developed initial plans for deploying internal employee growth mechanisms. In addition, the agency stated that, because it anticipates no authorized staffing growth for fiscal years 2020 and 2021, the primary focus of its workforce strategies will be on delivering IT services in a growing environment while experiencing no authorized staff growth. Further, the agency stated that, due to its low vacancy rate, its emphasis will change from filling gaps to sustaining services while controlling workforce attrition. While the actions described may be sufficient to fully implement the activity, VA did not provide documented plans to address projected staffing gaps; as a result, we did not change our rating for this activity. With regard to implementing activities that address gaps, the agency stated that its Office of Information and Technology Human Capital Management Recruitment Strategy outlines talent acquisition approaches leveraged within the office to address staffing gaps. We analyzed this document during our review and, as noted in our report, found that it identified actions taken to address the prior year’s gaps, but it did not provide documentation showing that VA had implemented strategies and plans to address projected staffing gaps. As a result, we did not change our rating for this activity. With regard to monitoring the agency’s progress in addressing competency and staffing gaps, the agency stated that it has fully implemented the activity because it believes it has fully implemented the aforementioned dependent activities. However, as previously stated, we did not change our ratings for the other activities based on information that VA provided. Accordingly, we did not change our rating for this activity. With regard to reporting to agency leadership on progress in addressing competency and staffing gaps, VA stated that, in June 2019, its Office of Information and Technology briefed the agency’s Chief Information Officer and senior leadership on the preliminary results of data collection that is expected to ultimately result in a staffing model which accurately depicts the current array of the office’s workforce, requirements to perform the mission, functions, task assigned, and the associated staffing gap. However, the agency did not provide documentation supporting this activity. As a result, we did not change our partially implemented rating designation for the activity. In written comments (reprinted in appendix X), GSA agreed with our recommendation and stated that it has established a project team to implement the remaining workforce planning activities. In comments provided via email on September 12, 2019, the Liaison to GAO in NSF’s Office of the Director, Office of Integrative Activities, stated that the agency agreed with our recommendation. The liaison added that NSF had recently completed an iteration of an IT workforce plan that is to inform its processes going forward, and address many of the IT workforce planning activities. The liaison also stated that NSF recognizes the importance of IT workforce planning and will continue to implement improvements to its processes in this area. OPM provided written comments (reprinted in appendix XI) in which the agency stated that it concurred with the recommendation. In addition, the agency stated that, to address its shortcomings, it has partnered with GSA’s IT Modernization Center of Excellence to assess the current state of its IT workforce planning activities. The agency stated that this effort is intended to assist with identifying and addressing gaps. In its written comments (reprinted in appendix XII), SBA agreed with the recommendation. The agency stated that its Office of Human Resource Solutions and the Office of the CIO will continue unified efforts to fully implement the remaining seven key IT workforce planning activities noted in our report. SBA added that it expects to complete the efforts by the end of fiscal year 2021. SBA also provided technical comments which we incorporated, as appropriate. SSA provided written comments (reprinted in appendix XIII) in which it agreed with the recommendation. The agency stated that it planned to finish developing an IT Workforce Strategy by the end of fiscal year 2019, which is to provide a framework to address its future IT workforce needs. In addition, the agency stated that, in fiscal year 2020, it expects to begin implementation of activities to address our findings. SSA also provided technical comments which we incorporated, as appropriate. In written comments (reprinted in appendix XIV), USAID stated that it concurred with the recommendation. The agency said that it was taking actions to fully implement each of the seven IT workforce planning activities that we identified as not fully implemented. USAID added that it expects to complete these actions by the end of the first quarter of fiscal year 2021. One agency—Education—partially agreed with the recommendation. Specifically, in written comments (reprinted in appendix XV), Education stated that it has taken actions to address the workforce planning activities. For example, with regard to the assess competency and staffing needs regularly activity, the agency stated that, in fiscal years 2018 and 2019, it conducted assessments of competency and staffing needs for employees coded as cybersecurity employees. However, the agency did not provide supporting documentation, including documentation showing that it had assessed or updated competency needs since they were originally developed. As a result, we did not change our rating for the activity. For the assess gaps in competencies and staffing activity, Education stated that it conducted a two-part competency assessment of all employees with cybersecurity responsibilities in March 2019. However, the agency did not provide documentation of the assessment. As a result, we did not change our rating for the activity. With regard to developing strategies and plans to address gaps in competencies and staffing, Education stated that, in April 2019, it submitted to OPM its action plan to address competency and staffing gaps identified in its Cybersecurity Work Roles of Critical Need report. However, the agency did not provide documentation of the plan. As a result, we did not change our rating for the activity. In addition, the agency described its planned efforts to fully implement the remaining IT workforce planning activities, including developing an IT workforce planning process and monitoring and reporting on progress in addressing competency and staffing gaps. Three agencies commented on our findings but did not state whether they agreed or disagreed with our recommendations: In comments provided via email on September 6, 2019, the Director of Strategic Planning, Policy, E-government and Audits in Agriculture’s Office of the CIO stated that the agency concurred with our findings. In addition, the agency provided technical comments, which we have incorporated in the report as appropriate. In comments provided via email on August 26, 2019, an official from Justice’s Office of the CIO stated that the agency concurred with our findings. In comments provided via email on September 5, 2019, the GAO liaison coordinator for EPA’s Office of Mission Support provided comments on the findings. The agency stated that, in April 2019, it submitted two action plans to address Cybersecurity Work Roles of Critical Need to OPM which it believes address the eight IT workforce planning activities. For example, with regard to the establish and maintain a workforce planning process activity, the agency stated that the workforce action plans present a model on how the agency plans to fill critical needs related to IT and application project management, and information systems security. While the action plans describe efforts to be performed to address gaps for specific work roles of critical need, they do not describe an overall IT workforce planning process for the agency, to include how the agency will continue to develop its competency and staffing requirements, assess for gaps, and develop strategies and plans to address the gaps. As a result, we did not change our rating for the activity. Further, with regard to the remaining workforce planning activities, the agency stated that the action plans, which it had not previously provided during the course of our review, include actions and milestones focusing on evaluating skill gaps and assessing current training and development opportunities. However, the agency did not provide documentation of the underlying IT competency requirements or competency gap assessments used to identify the gaps. As noted in our report, if an agency has not developed competency requirements, it is not able to implement the subsequent activities relating to competencies. On the other hand, the agency has developed staffing requirements, and as a result we have updated our rating for the staffing evaluation criteria within the develop strategies and plans to address gaps in competencies and staffing activity. However, EPA did not provide documentation showing that it had implemented the strategies and plans to address staffing gaps, or monitored and reported on progress in addressing staff gaps. As a result, we did not change our ratings for these activities. One agency did not agree with our findings. Specifically, in its written comments (reprinted in appendix XVI), NRC stated that it did not agree with the findings that it had not developed an IT workforce planning process or IT competency requirements. With regard to the IT workforce planning process, we noted in our report that NRC had developed a workforce planning process that addressed all the key IT workforce planning activities; however, we stated that the process did not define the Chief Information Officer’s roles and responsibilities for implementing the activities or how the plan aligns with mission goals and objectives. In its response, the agency stated that its Management Directive 9.22, which was not provided to us during our review, defines the Chief Information Officer’s roles and responsibilities for implementing activities, including workforce planning by developing and maintaining the agency’s IT/Information Management Strategic Plan and enterprise IT/Information Management roadmap in alignment with the NRC Strategic Plan, and reviewing all positions with IT responsibilities requested in the budget request to ensure the positions meet the ongoing requirements of the agency. We reviewed the directive and determined that it addresses the Chief Information Officer’s roles and responsibilities. In addition, NRC identified parts of its workforce planning process, that it believes addresses alignment with mission and goals. We reviewed these parts, and agree with NRC that the plan addresses alignment with mission and goals. We have incorporated the change into this report, including changing the rating from partially implemented to fully implemented for this activity. As a result, we modified the recommendation from fully implementing eight activities NRC did not implement to fully implementing seven activities it did not fully implement. With regard to developing competency requirements, the agency stated that it specifies competencies for all IT positions in its position descriptions. However, NRC did not provide documentation of the position descriptions or the related competencies. As a result, we are not changing our not implemented rating for this activity. NRC also noted that it has joined other federal agencies to develop career paths and competency models for 64 IT security roles across the federal government, and that this effort is scheduled to be completed in October, at which time the agency will decide which of the models to adopt. In addition, the following two agencies to which we made recommendations in prior reports provided comments. In its written comments (reprinted in appendix XVII), Defense stated that it concurred with the overall contents of the report. In comments provided via email on September 5, 2019, an official from Treasury’s Office of the CIO stated that the agency agreed with all but two of our findings in this report, associated with three of the activities. First, the agency disagreed with our finding that it minimally implemented the establish and maintain a workforce planning process activity, stating that it has a department-wide workforce planning process that includes the IT workforce. However, while the agency issued a policy in 2013, which we reviewed during our engagement, that directs bureaus to annually conduct workforce planning, it did not define a process for doing so. In addition as we further note, in 2018, the agency issued guidance addressing workforce planning issues for bureaus to consider in developing their own processes. However, this does not constitute an IT workforce planning process. Since Treasury did not provide any additional evidence of an IT workforce process, we are not changing our rating for this activity. Second, Treasury disagreed with our finding that it did not implement the activities associated with monitoring and reporting on its progress in addressing competency and staffing gaps. Specifically the agency stated that it has designed and begun implementing a new governance structure for workforce management that reinforces the monitoring and reporting of workforce related issues to agency leadership during quarterly performance reviews. However, as we note in our report, the monitoring and reporting activities are dependent on the developing strategies and plans to address competency and staffing gaps activity which Treasury has yet to implement. Until Treasury develops such strategies and plans, it cannot monitor and report on their progress. We are sending copies of this report to interested congressional committees, the Director of the Office of Management and Budget, the secretaries and agency heads of the departments and agencies addressed in this report, and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. Should you or your staffs have any questions on information discussed in this report, please contact me at (202) 512-4456 or HarrisCC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XVIII. Our objective was to examine the extent to which federal agencies are effectively implementing information technology (IT) workforce planning activities. To address this objective, we relied on practices from GAO’s IT workforce planning framework as criteria. The framework identifies eight key IT workforce planning activities that, when effectively implemented, can facilitate the success of major acquisitions. These activities are listed in table 2. To ensure consistent understanding and application of the activities in our evaluations, we reviewed the supporting laws, policy, and guidance for each activity and identified specific evaluation criteria. The criteria are listed in table 3. We reviewed IT workforce planning policies and other workforce planning documentation for each of the 24 Chief Financial Officers Act of 1990 agencies, including workforce planning processes; competency requirements; annual mission critical occupation resource charts required by the Office of Personnel Management (OPM) which document staffing requirements and gap assessments; strategies and plans to address gaps; and reports on progress in addressing gaps. For the six agencies for which we previously performed IT workforce planning assessments, we reviewed the previously reported information and obtained and analyzed updates, as appropriate. We compared the information obtained to our evaluation criteria and identified gaps and their causes. We also interviewed cognizant officials from each of the 24 agencies, to discuss their implementation of the IT workforce planning activities and causes for any gaps. Our review focused on the agency’s IT workforce planning efforts at the agency level, including the extent to which the agency maintained visibility and oversight into component-level IT workforce planning. Based on our assessment of the documentation and discussions with agency officials, we assessed each agency’s implementation of our evaluation criteria as: fully implemented—the agency provided evidence which showed that it fully or largely addressed the elements of the criteria. partially implemented—the agency provided evidence that showed it had addressed at least part of the criteria. not implemented—the agency did not provide evidence that it had addressed any part of the criteria. To determine an overall rating for each of the eight key workforce planning activities, we summarized the results of our assessments of the evaluation criteria. Specifically, we assessed each activity as: fully implemented—the agency fully implemented both of an activity’s evaluation criteria. substantially implemented—the agency fully implemented one of an activity’s evaluation criteria and partially implemented the other evaluation criteria. partially implemented—the agency fully implemented one of an activity’s evaluation criteria and did not implement the other criteria, or partially implemented both of an activity’s evaluation criteria. minimally implemented—the agency partially implemented one of an activity’s evaluation criteria and did not implement the other evaluation criteria. not implemented—the agency did not implement either of an activity’s evaluation criteria. We assessed the staffing evaluation criteria for the develop competency and staffing requirements, assess competency and staffing needs regularly, and assess gaps in competencies and staffing activities as fully implemented if agencies provided evidence of a complete mission critical occupation resource chart to meet OPM reporting requirements and were able to demonstrate that the mission critical staff represented most or all of their IT workforce. In addition, we assessed the competency evaluation criteria for these activities as fully implemented if agencies provided evidence that they performed them for most or all of their IT workforce. For the implement activities that address gaps activity, we assessed agencies as having fully implemented the evaluation criteria on other efforts if they provided evidence as having implemented at least four of the efforts identified in the Federal Information Technology Acquisition Reform Act (FITARA). We rated this evaluation criteria as partially implemented if agencies provided evidence of having implemented fewer than four of the efforts. Finally, in making our assessments, we also considered the extent to which an agency had implemented prerequisite activities. For example, to implement the competency evaluation criteria for the develop strategies and plans to address gaps activity, the agency needed to have also implemented the competency evaluation criteria for the assess gaps in competencies and staffing activity. We did not assess any activity higher than the prerequisite activity. We also determined if there was a common factor which led to the rating for a particular activity. For example, we noted whether most agencies partially implemented an activity because they had fully implemented one of the evaluation criteria, but had not implemented the other criteria. To determine the reliability of staffing data in the mission critical occupation resource charts, we reviewed the charts for obvious errors and for completeness and obtained clarification from agencies on identified errors. We determined that the data were sufficiently reliable for the purpose of this report, which was to determine the extent to which agencies had implemented the key activities. We conducted this performance audit from January 2018 to October 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix contains assessments of the extent to which the 24 Chief Financial Officers Act of 1990 agencies implemented each of the eight key IT workforce planning activities identified in GAO’s information technology (IT) workforce planning framework. In addition to the individual named above, the following staff made key contributions to this report: Sabine Paul (Assistant Director), Scott Borre (Analyst in Charge), Rebecca Eyler, Cassaundra Pham, Thomas B. Rackliff, and Marshall Williams, Jr.", "summary": "The federal government annually spends over $90 billion on IT. Despite this large investment, projects too frequently fail or incur cost overruns and schedule slippages while contributing little to mission-related outcomes. Effectively implementing workforce planning activities can facilitate the success of major acquisitions. GAO was asked to conduct a government-wide review of IT workforce planning. The objective was to determine the extent to which federal agencies effectively implemented IT workforce planning practices. To do so, GAO compared IT workforce policies and related documentation from each of the 24 Chief Financial Officers Act of 1990 agencies to activities from an IT workforce planning framework GAO issued. GAO rated each agency as having fully, substantially, partially, minimally, or not implemented for each activity. GAO supplemented its reviews of agency documentation by interviewing agency officials. Federal agencies varied widely in their efforts to implement key information technology (IT) workforce planning activities that are critical to ensuring that agencies have the staff they need to support their missions. Specifically, at least 23 of the 24 agencies GAO reviewed partially implemented, substantially implemented, or fully implemented three activities, including assessing gaps in competencies and staffing. However, most agencies minimally implemented or did not implement five other workforce planning activities (see figure). Agencies provided various reasons for their limited progress in implementing workforce planning activities, including competing priorities (six agencies), and limited resources (three agencies). Until agencies make it a priority to fully implement all key IT workforce planning activities, they will likely have difficulty anticipating and responding to changing staffing needs and controlling human capital risks when developing, implementing, and operating critical IT systems. GAO is making recommendations to 18 of the 24 federal agencies to fully implement the eight key IT workforce planning activities. Of the 18 agencies, 13 agreed with the recommendations, one partially agreed, three neither agreed nor disagreed, and one disagreed with the findings and provided evidence which led to a modification to its recommendation, as discussed in this report. For all of the remaining recommendations, GAO continues to believe that they are all warranted.", "document_type": "gao"}
{"report": "Space systems generally involve one of four types of interrelated segments that are needed to make a space capability fully functional. As illustrated in figure 1, they include (1) space components—namely the satellites, (2) ground components, including satellite control systems and data processing subsystems and facilities, (3) user equipment, such as radios/terminals, needed by the warfighter to use the capability, and (4) launch vehicles and facilities. DOD space systems are acquired under the same acquisition policies as other weapons systems. However, as we found in July 2016, space systems are different from other acquisitions in some ways. For example, space has more programs of joint interest than other areas, and includes varied stakeholders, such as civil agencies and multiple services. According to officials, in developing space systems once a satellite is launched, if there are problems it is essentially impossible to change the hardware, and software changes may not be an option. In addition, space programs typically use cutting-edge technologies that have to withstand the harsh space environment. Such technologies are rarely available as off-the-shelf systems from the commercial market and must be developed following a specific process overseen by specially- trained DOD acquisition personnel. Data from the Under Secretary of Defense for Acquisition and Sustainment’s Office of Human Capital Initiatives show that, as of June 2018, DOD employed about 170,000 military and civilian personnel who are designated as acquisition personnel and are responsible for acquiring weapon systems, such as aircraft, ships, tanks, and space systems. DOD tracks the characteristics, education, training, and experience of these acquisition personnel in DOD’s acquisition workforce data system—Data Mart—where they are tracked as belonging to 1 of 15 acquisition career fields. Table 1 shows a list of these acquisition career fields. Contractor and FFRDC personnel often support DOD acquisition efforts. For the purpose of this report, “contractor” refers to support service contractors who provide technical and administrative support rather than prime contractors who develop and produce weapon systems or products. FFRDCs are not-for-profit entities sponsored and funded primarily by DOD to fulfill research and development, engineering, and analytic needs that cannot be met as effectively by existing government or contractor personnel. Nonprofit, university-affiliated, or private industry organizations operate the FFRDCs through contracts or other agreements with federal agencies. DOD procures FFRDC services by staff years of technical effort. The total amount of FFRDC services time that DOD is permitted to obtain is set annually by Congress. For fiscal year 2018, DOD was authorized to use available funds for FFRDCs for not more than 6,030 staff years of technical effort. Authorized staff years of technical effort are allocated among the military services’ organizations that act as the primary sponsors for each FFRDC, which then prioritize what work the FFRDC will perform according to the allocation level received. In general, managers in the contractor and FFRDC organizations direct the daily activities of their respective personnel, while DOD military and civilian personnel oversee their work. Over the years, GAO has highlighted the importance of workforce management. Since 2001, GAO has included strategic human capital management as a government-wide high-risk area. More recently, we found that having the right workforce mix with the right skill sets is critical to achieving DOD’s mission, and that it is important for DOD, as part of its strategic workforce planning, to conduct gap analyses of its critical skills and competencies. Strategic workforce planning—an integral part of human capital management—is an iterative, systematic process that helps organizations determine if they have staff with the necessary skills and competencies to accomplish their strategic goals. As shown in table 2, many DOD offices play key roles in strategic workforce planning activities. DOD does not have comprehensive information about its space acquisition workforce—including the size, mix, and location of this workforce. DOD does not have this information because, among other things, DOD has not clearly identified its space programs, and its workforce data systems are not configured to identify space acquisition personnel. In the absence of comprehensive DOD data, we sought to obtain an understanding of the extent of this workforce. We aggregated data from individual DOD organizations and estimate that at least 8,000 military, civilian, contractor, and FFRDC personnel were working on space acquisitions in multiple locations across the United States at the end of 2017. While this information represents only a snapshot in time, it provides insight into the extent of the space acquisition workforce. Given DOD’s recent decision to stand up a United States Space Command and to establish a consolidated Space Development Agency in 2019, along with the President’s directive for DOD to submit a legislative proposal for a United States Space Force, having knowledge about which personnel are involved with military space acquisitions and where these personnel are located will be important to DOD’s planning efforts. DOD collects data on its acquisition workforce, but does not collect and maintain comprehensive and complete data on the size, mix, and location of the military, civilian, contractor, and FFRDC personnel working on space acquisitions. According to the military services’ Directors of Acquisition Career Management, DOD manages its acquisition workforce by career field, such as program management and engineering, and not by the type of product being acquired, such as space systems. They told us that, in their view, the acquisition skills needed for an acquisition program—such as those for program management, engineering, and contracting—are largely the same regardless of the product type. However, officials acknowledged that it takes some time for personnel to learn the nuances of acquiring a specific type of product. We identified three factors that hinder DOD’s ability to collect comprehensive data on its space acquisition workforce. Together, they impede DOD from maintaining a complete and accurate understanding of the size, mix, and location of its space acquisition workforce. DOD does not maintain a complete list of its space acquisition programs. Officials from the office of the Assistant Secretary of the Air Force for Acquisition and the service-level acquisition career managers told us that DOD does not maintain a list of the universe of space acquisition programs. In addition, the budget document that DOD submits to Congress specific to space programs, which could possibly serve as an alternative source of such information, identifies programs that have large amounts of funding by name, but aggregates information for smaller programs without identifying them individually. While DOD does not maintain a complete list of space acquisition programs, during the course of our review we found that the military services were generally able to identify space acquisition programs. DOD does have a definition of space systems. Specifically, according to a DOD Directive, space systems include all systems related to making a space capability operational—that is programs acquiring satellites, satellite ground systems (including satellite control and data processing), receivers/user segments (including terminals and radios), and launch systems—but specifies that terminals that are embedded as part of a platform (i.e. aircraft, ship, or tank) are excluded. However, DOD officials had difficulty identifying some programs, particularly those in the user segment. For example, the Air Force’s Space Fence program, which is developing ground radar as a part of the space surveillance network that detects and tracks space objects, is included as a space program in DOD’s budget documents. Officials from the Program Executive Office that staffs personnel to the program initially told us they did not consider it a space program since it is a series of ground-based radars. They subsequently determined that it is a space program since the radar will track space objects and provide data for space situational awareness. DOD data systems are not currently configured to identify space acquisition personnel. We examined three data sources that have information on the different personnel groups in the acquisition workforce, and found that none of them can identify space acquisition personnel. The Office of Human Capital Initiatives within the Office of the Under Secretary of Defense for Acquisition and Sustainment uses the Data Mart system to track the education, experience, and training of military and civilian acquisition-coded personnel working in the 15 acquisition functional career fields shown in table 1. DOD has taken periodic steps to enhance the data captured in this system. For example, in 2009 DOD began tracking whether acquisition personnel in the business career field were working on financial management or cost estimating. In 2014, DOD started to track personnel with expertise in contracting with small businesses, and expanded its efforts to track personnel with expertise in international acquisitions. However, this system does not currently identify personnel staffed to or supporting space acquisitions or any other type of product acquisition. The Office of the Under Secretary of Defense for Personnel and Readiness tracks contractor data using the Enterprise-wide Contractor Manpower Reporting Application system to provide DOD management information on contracted services obtained by each military service and defense agency. The system includes data on the number of hours of service each contractor provides to the government, which could be used to approximate the number of contractor personnel used to perform the work. However, the system does not track the type of acquisition programs being supported, such as space acquisition programs. In addition, the data are self-reported by service contractors and concerns exist regarding potential underreporting. For example, we reported in March 2018 that the military services estimated that the Enterprise-wide Contractor Manpower Reporting Application included fiscal year 2016 contractor data for 80 percent of Army contracts and 75 percent of Navy contracts; the percentage of Air Force contracts was unknown. The Director of Laboratories and Personnel within the Office of the Under Secretary of Defense for Research and Engineering tracks information on FFRDCs, such as the staff years of technical effort provided each year, to ensure that DOD stays within its congressionally mandated limit. Each FFRDC sponsor organization provides an annual report of their staff years of technical effort and funding to DOD. However, DOD officials told us that sponsoring organizations do not identify what type of acquisition program their respective FFRDC personnel support, such as space acquisition programs. Personnel supporting space acquisitions are dispersed across a variety of organizations and may also support non-space programs. Each of the military services we reviewed has program executive offices, research labs, or other organizations that support both space and non- space acquisitions. DOD officials told us that functional career field leaders in each of the organizations, such as the engineering or the contracting directorates, assign personnel to space or non-space programs on an as-needed basis, which could make it difficult for DOD to determine which and how many personnel should be included in the space acquisition workforce. Five of the 10 space acquisition programs we reviewed—1 Air Force, 1 Navy, and 3 Army—were managed by organizations that were primarily responsible for developing and acquiring non-space programs. Air Force—The Space Fence program is staffed by the Air Force Life Cycle Management Center’s Program Executive Office for Battle Management. The Center primarily supports non-space programs, such as fighters, bombers, tankers, and presidential aircraft. Navy—The Mobile User Objective System is managed by the Space and Naval Warfare Systems Command, which primarily manages non-space programs that provide enterprise information system and command, control, communications, computers, and intelligence capabilities. Army—The Joint Tactical Ground Station program is managed by the Army’s Program Executive Office for Missiles and Space. The office primarily manages a variety of missile programs—such as close combat, cruise, and integrated air and missile defense programs—that are non-space programs. Similarly, the Secure, Mobile, Anti-Jam, Reliable, Tactical-Terminal and the Transportable Tactical Command Communications programs are managed by the Army’s Program Executive Office for Command, Control, Communications-Tactical. This office primarily manages a variety of information systems to provide tactical communication for the service, which may or may not be space programs. Officials told us that the three Army programs we reviewed were also supported by other, separate Army organizations, such as the Army Contracting Command for contracting support; the Army’s Aviation and Missile Research, Development, and Engineering Center for engineering support; and the Army Materiel Command for logistics support. These organizations provide support to space and non-space programs on an as-needed basis. The Administration, Congress, and DOD are discussing a variety of approaches for strengthening the government’s space operations, including the establishment of one or more new organizations. In June 2018 the President directed DOD to begin the process of establishing a new military branch focused on space that is separate from and equal to the other military departments, Army, Navy, and Air Force. In an August 2018 report to the Congress on the organizational and management structure needed for the national space components, DOD described the immediate steps that it plans to take to implement the President’s direction while waiting for Congressional authorization to create the new military branch. These steps include establishing a new United States Space Command to further its space warfighting capabilities, as well as developing plans to establish a consolidated Space Development Agency to rapidly develop and field next generation space capabilities. DOD has described the general areas of focus planned for these new organizations; however, many specifics are still to be determined. DOD has announced that a committee of senior DOD leaders is expected to identify which of the current space activities will be consolidated into these new space organizations. In addition, the President’s February 2019 Space Policy Directive now requires DOD to submit a legislative proposal to establish a United States Space Force as a new armed service within the Air Force. DOD announced it delivered a legislative proposal to Congress on March 1, 2019. The lack of comprehensive information about DOD’s space programs and the acquisition personnel supporting those programs affects DOD’s ability to assess gaps in the overall capabilities of its space acquisition workforce and determine whether it has sufficient internal capability and critical knowledge or skills for its space acquisitions. Moreover, it hampers DOD’s ability to make decisions related to establishing the United States Space Command, a new Space Development Agency, or potentially the United States Space Force. This includes determining the appropriate number and mix of acquisition personnel that are needed for the new organizations, as well as which military and civilian personnel should be assigned to them. According to federal internal control standards, an agency, such as DOD, should have relevant, reliable, and timely information in order to run and control operations, including managing the workforce. Air Force Director of Acquisition Career Management officials stated that having a process for identifying space acquisitions personnel would be beneficial. As we reported in July 2003, the success of merging personnel during organizational transformations is more likely when the best individuals are selected to meet the skills and competencies needed for the new organization’s goals. In the absence of readily available comprehensive data from DOD, we collected and aggregated data from multiple DOD space organizations and found that at least 8,000 personnel were in the space acquisition workforce at the end of 2017. However, our data set is not complete. For example, the National Reconnaissance Office, which DOD officials told us has a significant number of personnel working on space acquisitions, is not included in our analysis. In addition, our count only includes personnel that spent 50 percent or more of their time working on space acquisitions; therefore any personnel who spent less than 50 percent of their time on space acquisitions was not included. Furthermore, it is important to note that our data provide a snapshot of the workforce as of December 31, 2017. According to DOD officials, the size and mix of the workforce can change based on the number of programs and where programs are in the acquisition process. The military and civilian personnel data we collected are expressed as number of people. The contractor and FFRDC personnel data are expressed as full-time equivalents and staff-years of technical effort equivalents, respectively. Size of Workforce: Based on data we collected from multiple DOD space acquisition organizations, at least 8,000 military, civilian, contractor, and FFRDC personnel supported DOD’s space acquisitions as of December 31, 2017 (see figure 2). Military and civilian personnel comprised about 64 percent of the total space acquisition workforce, the vast majority of which support Air Force acquisitions. The remaining 36 percent of the workforce is contractor and FFRDC personnel that support DOD’s space acquisition activities. The Air Force has the largest number of military and civilian personnel because the Air Force has primarily been responsible for DOD’s space acquisitions and develops programs for all four segments of space capability, including launch services for the most critical national security space satellites. The Navy is responsible for systems that provide satellite communications across DOD as well as its user segments, while the Army and other DOD components primarily focus their efforts on developing their user segment systems or other space-related projects. Workforce Mix: Based on data we collected from multiple DOD space acquisition organizations, the mix of military, civilian, contractor, and FFRDC personnel that each military service and agency had supporting their respective space acquisition programs varied considerably (see figure 3). Military and civilian personnel comprised between 54 and 63 percent of the Air Force’s, Army’s, and Navy’s space acquisition workforce and 94 percent of the other DOD components’ workforces. Contractors and FFRDC personnel made up the remainder of the workforce. The Air Force relies more heavily on FFRDC personnel as a percentage of its workforce than the Army, Navy, and other DOD components. According to Air Force officials, the Space and Missile Systems Center—the Air Force’s major space acquisition organization—has relied heavily on FFRDC support for space engineering and technical expertise since its founding in the 1950s. The Army and Navy primarily rely on contractors for their remaining support. These contractors mainly provide technical expertise, such as engineering services, to support military and civilian personnel. Some contractors also support program management and business and administration activities, such as cost estimating. Figure 4 provides detailed examples of how personnel support two space acquisition programs included in our review. Locations of Workforce: Based on data we collected from multiple DOD space acquisition organizations, space acquisition personnel work at over 20 organizations located across the United States. Figure 5 shows the primary locations of DOD’s space acquisition organizations. About 45 percent of the overall space acquisition workforce is located at the Air Force Space and Missile Systems Center in Los Angeles, California. The Army space acquisition workforce is located primarily at Redstone Arsenal in Huntsville, Alabama, and Aberdeen Proving Ground, Maryland. The Navy space acquisition workforce is located at the Space and Naval Warfare Systems Command in San Diego, California, and a few other locations. DOD faces several challenges related to hiring, assigning, and retaining qualified personnel to work on space acquisition programs, similar to the challenges it faces more generally with the acquisition workforce. However, some of the challenges are magnified because almost half of the military and civilian space acquisition workforce is concentrated in Los Angeles, California, which has a higher cost of living than many other areas in the United States, and where competition with private industry for personnel with space acquisition experience is high. DOD is taking steps to address these challenges where possible. DOD officials told us that one of the primary workforce challenges DOD faces is its ability to hire qualified people to work on space acquisitions. They said that DOD is competing with private industry and other federal agencies for top talent in several acquisition career fields. Attracting Candidates with Technical Expertise. DOD officials stated that it is particularly difficult to attract people with certain technical expertise, such as cybersecurity and systems engineering, because they are in high demand in both government and private industry. Air Force officials said the government cannot match the salaries offered by industry. For example, the Launch and Test Range System program office told us that a shortage of trained and qualified cybersecurity personnel exists both within the government and industry. Our prior work has described how maintaining cybersecurity personnel is a challenge government-wide, and that, according to DOD officials, even when DOD cybersecurity positions are filled, it may not necessarily be with the right expertise since it is a specialized area. Hiring in Areas with Higher Costs of Living. Air Force officials at the Space and Missile Systems Center said that hiring challenges are further exacerbated for space acquisition organizations that are located in areas with higher costs of living. They said, for example, that prospective employees often visit the center in Los Angeles, California, and, after assessing the local cost of living, decide not to accept a job offer. DOD is taking steps to address its hiring challenges. To address difficulties in obtaining personnel with sufficient technical experience, some officials told us that they typically hire the best candidate available—who may lack some of the desired technical skills— and provide them with on-the-job and formal training to increase their technical knowledge and skills. To better compete with higher salaries offered by other potential employers, several officials told us they offer tuition reimbursement as a recruiting incentive. Air Force officials told us that in areas with higher costs of living they focus their recruiting efforts on the local area because local candidates already understand the higher costs of living challenges for the area and are more likely to have support systems in place to manage such costs. Beyond the concerns expressed about hiring personnel, Air Force Space and Missile Systems Center officials expressed concerns that some functional areas within the space acquisition workforce face challenges assigning experienced personnel—personnel with the appropriate knowledge and skill set to perform the work—that are already hired to space acquisition programs. For example, contracting career field officials at the center noted that the space acquisition workforce does not have enough mid-level personnel who understand the detailed steps and documentation required in the acquisition process. In particular, the Air Force Space and Missile Systems Center reported that at the end of January 2018, the number of mid-level civilian and military personnel working in the contracting functional career field was 50 less than the number authorized. According to contracting career field officials at the center, a large number of mid-level procurement contracting officer positions were vacant, and senior procurement managers were picking up the corresponding workloads rather than performing their staff development and strategic planning tasks. Furthermore, officials from the Air Force’s Space and Missile Systems Center program management functional office also expressed concern that the bulk of the military personnel assigned to the program management positions were more junior in rank than the Center was authorized by the Air Force to obtain. Figure 6 shows the level of the Air Force Space and Missile Systems Center personnel that filled its program management positions as of January 2018. Junior officers typically have less experience managing acquisition programs than more senior officers. The military services are taking steps to manage the effects of military and civilian personnel skills and experience gaps, to some degree, by having contractor personnel perform the work. For example, the Air Force Space and Missile Systems Center’s contracting functional office used four contractor personnel to support its pricing work. DOD has also experienced challenges with retaining some space acquisition personnel, especially those within their first few years of joining federal government service that had obtained certain acquisition- related experience or authorities. For example, contracting career field officials at the Air Force Space and Missile Systems Center said that they have difficulty retaining contracting officers once they receive their contract warrant authority because they can obtain a higher compensation package from private industry companies. Receiving contract warrant authority is considered an indication that the individual gained sufficient skills and experience to be able to perform the work involved in writing, awarding, and managing contracts. Officials also stated that some personnel leave after obtaining security clearances required to perform their work because private companies working on government contracts pay more to qualified individuals with clearances. Officials from the Air Force Space and Missile Systems Center and Army Space and Missile Defense Command also told us that they have difficulty retaining engineers. They said some engineers have left because they were not satisfied with being used as generalists to oversee the work of FFRDC or contractor personnel, rather than being used to perform hands-on engineering work. Officials also stated that this situation is not unique to space acquisitions—government engineers seldom get to design, develop, or build systems as the hands-on engineering work is primarily performed by prime contractors. Air Force Space and Missile Systems Center officials said they are trying to help the government engineers understand how to influence decisions and be more effective in working as part of the space engineering acquisition team, which would include military, civilian, contractor, and FFRDC personnel. Officials from various functional career fields at these Air Force and Army locations noted that limited promotion opportunities for civilian personnel in space acquisitions also cause retention challenges. For example, the Air Force Space and Missile System Center has 53 management (General Schedule 15) positions; however, Center officials told us that the turnover rate for these higher-level positions is low. Officials reported that some mid-level program management personnel seek and accept promotions at other non-space acquisition offices or in other geographical locations that have more promotion opportunities. Some Air Force Space and Missile Systems Center and Army officials noted that retention incentives are used to help retain staff. This includes student loan repayments, and recognition incentives, such as monetary or time-off awards tied to performance. Air Force Space and Missile Systems Center officials also said that they are working to realign current civilian acquisition personnel at the center under the Civilian Acquisition Workforce Demonstration project, which they believe will help attract, retain, and motivate high-quality civilian personnel for the acquisition workforce. DOD space systems and the personnel who work to acquire them remain critical components of national security and key resources. As DOD takes steps toward establishing the United States Space Command, its Space Development Agency, and potentially the United States Space Force, it will be essential to understand the size, mix, and location of the space acquisition workforce. However, DOD does not collect and maintain this type of comprehensive data on its space acquisition workforce. Although we were able to pull together information on the space acquisition workforce, the data represent a snapshot of the workforce at one point in time, and are not complete since acquisition personnel working on National Reconnaissance Office space programs and those who spent less than 50 percent of their time working on space acquisitions were not included. Taking steps to identify and routinely track accurate information on space acquisition programs and the organizations and personnel that support those programs would provide several benefits to DOD. In particular, it would better position DOD to assess whether it has the appropriate number and mix of military, civilian, contractor, and FFRDC personnel working on space acquisitions and to make adjustments if necessary. Further, it would better position DOD to make decisions on which acquisition personnel will support or transition into the United States Space Command or the new Space Development Agency, since DOD has not clearly defined what acquisition functions may or may not be handled by these new organizations. Finally, comprehensive data on the space acquisition workforce would also be beneficial to support DOD’s development of its legislative proposal regarding the establishment of the United States Space Force. We are making the following two recommendations to DOD: The Secretary of Defense should direct the military services and other DOD components to identify the universe of space acquisition programs, as well as the various organizations that support these programs, and report this information to Congress. In doing so, DOD should implement procedures to maintain and periodically update the list. (Recommendation 1) The Under Secretary of Defense for Acquisition and Sustainment, in conjunction with the Under Secretaries of Defense for Research and Development and for Personnel and Readiness, should collect and maintain data on acquisition-coded military and civilian personnel that support space acquisition programs and related activities—including those that may do so less than full time—as well as track the contractor and FFRDC workforce general levels of effort supporting space acquisition programs and related activities and the total resources annually committed to perform that work. (Recommendation 2) We provided a draft of this report to DOD for review and comment. DOD provided written comments (reproduced in appendix II) on our draft report. In those comments, DOD concurred with our first recommendation to identify the universe of space acquisition programs, as well as the various organizations that support these programs, and report this information to Congress. DOD did not concur with our draft second recommendation to collect and maintain data on the space acquisition workforce. DOD stated that the manner in which personnel data are captured in its human resource and development systems makes it difficult to identify, collect, and maintain data on the military and civilian personnel working on space acquisition programs. Further, DOD raised concerns over contractual limitations on collecting and maintaining data on contractor and FFRDC personnel supporting space acquisitions. In light of these concerns, we made changes to the draft recommendation. We believe the language of our final recommendation will better facilitate implementation by DOD. With regard to our second recommendation, we continue to believe that taking steps to identify military and civilian personnel supporting space acquisition programs would support DOD’s strategic workforce planning, particularly considering DOD’s recent legislative proposal for establishing the United States Space Force. For example, we acknowledge that the current personnel data system used to track military and civilian acquisition personnel has limitations, but we believe taking steps to make minor modifications to the system to facilitate identifying and routinely tracking accurate information on these two segments of the space acquisition workforce would provide several benefits to DOD. Most importantly, it would help DOD make decisions on how many and which military and civilian acquisition personnel should be assigned to the new space organizations—namely the Space Development Agency, the United States Space Command, and the United States Space Force. With regard to DOD’s comment that our recommendations do not recognize that DOD personnel have been shifted into and out of space acquisition programs, we recognize that acquisition personnel have been moved across programs and support space and non-space acquisitions. However, we continue to believe that DOD should have better information on military and civilian acquisition personnel. In particular, knowing which personnel have space acquisition backgrounds could enhance the productivity and effectiveness of DOD’s space acquisition efforts. As a result, we did not make a change to our second recommendation as it relates to military and civilian space acquisition personnel. However, in consideration of the concerns raised by DOD about tracking data on contractor and FFRDC personnel who are supporting space acquisition activities, we modified our second recommendation. It was not our intention to have DOD undertake significant modifications to the relevant contracts to obtain data on these segments of the space acquisition workforce. However, understanding the extent to which space acquisition programs rely on contractor and FFRDC personnel for support could be useful in helping DOD determine the right number and mix of military and civilian personnel needed in the new space organizations. As a result, we modified the language of our second recommendation to focus on tracking the contractor and FFRDC workforce general levels of effort supporting space acquisition activities and the resources spent to obtain this assistance, rather than—as we stated in our draft recommendation—tracking the individuals who perform such work. However, we continue to believe that collecting and maintaining more robust data on that workforce will support DOD’s planning efforts and better inform Congress. DOD also expressed concern that our report may be equating statements of officials at the staff- and operational-level to military service- and DOD- level officials. We reviewed statements attributed to DOD officials throughout our report. Where necessary, we clarified attributions to better reflect the appropriate level of the officials with whom we discussed the corresponding information during our review. DOD also provided technical comments on our draft report, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees; the Acting Secretary of Defense; and the Secretaries of the Air Force, Army, and Navy. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or ludwigsonj@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. A House Report related to the National Defense Authorization Act of Fiscal Year 2017 contained a provision for GAO to review the current state of the Department of Defense’s (DOD) space systems acquisition workforce. This report examines (1) what DOD knows about the size, mix, and location of its space acquisition workforce, and (2) the challenges, if any, DOD faces in hiring, staffing, and retaining space acquisition workforce personnel. For the purpose of this report, we defined the space acquisition workforce broadly to include military, civilian, contractor, and Federally Funded Research and Development Center (FFRDC) personnel working on space acquisition programs and related efforts. To determine what DOD knows about the size, mix, and location of the space acquisition workforce, we met with officials from DOD’s Office of Human Capital Initiatives, the Air Force, the Army, the Navy, and 4th Estate’s Director of Acquisition Career Management to obtain information that is collected on the space acquisition workforce. We were told by each of these officials that DOD does not have a group of personnel officially designated as the space acquisition workforce. They stated that DOD has separate mechanisms for collecting military, civilian, contractor, and FFRDC workforce data and that none of these systems contained the level of granularity we would need to identify all personnel working on space acquisitions. Specifically, the sources we discussed were DOD’s Data Mart system, a central repository for military and civilian acquisition workforce data, as well as workforce data systems maintained by DOD components that feed into the Data Mart system; the Enterprise-wide Contractor Manpower Reporting Application system for contractor services data; and FFRDC data maintained by military components. We collected data on the size, mix, and location of the space acquisition workforce from the space organizations performing space acquisition activities. The Directors of Acquisition Career Management for the military services and the 4th Estate defense agencies provided a list of organizations that could be working on space acquisitions based on DOD’s 2017 space system definition, which states that a space system includes all areas related to making a space capability operational—that is programs acquiring satellites, satellite ground systems (including satellite control and data processing), receivers/user segments (including terminals and radios), and launch systems. It also specifies that terminals are included unless they are embedded as part of a platform (i.e., aircraft, ship, or tank). We contacted each of the identified space organizations to verify that they had personnel working on space acquisitions based on this definition. Three of the organizations we originally contacted stated their organizations did not work on any space acquisition programs based on the definition. We did not include these organizations in our data gathering efforts. We also identified other organizations that worked on space acquisitions through discussions with acquisition management officials from the Army and included these organizations in our data gathering efforts. We asked each space organization to identify the number of military and civilian personnel working on space acquisition activities for 50 percent or more of their work time as of December 31, 2017. We used the threshold of 50 percent or more of the time to be consistent with the DOD definition of the acquisition workforce, which requires personnel to work 50 percent or more of their work time on acquisition activities to be counted as part of that workforce. DOD officials could not identify the number of contractor and FFRDC personnel working on space acquisitions. Therefore, for contractor and FFRDC personnel, we asked for the number of full-time equivalencies and staff-years of technical effort equivalencies, respectively, provided as support to space acquisitions. We requested that the personnel data be categorized by acquisition career field. We collected data from each DOD component as follows: The Air Force Director of Acquisition Career Management provided military and civilian workforce data from the Air Force’s Acquisition Career Management System that feeds into Data Mart for all Air Force organizations where the entire organization works on space acquisitions. These organizations were the Air Force Space Command and the Networks Family of Advanced Beyond Line of Sight Terminals Division within the Air Force Life Cycle Management Center’s Program Executive Office for Command, Control, Communications, Intelligence and Networks. The Deputy Director identified other space programs that are managed by the Air Force Life Cycle Management Center, but could not identify which military and civilian personnel were supporting those programs because the workforce data system is not configured to identify personnel by product types. In addition, the Deputy Director could not provide data on the number of contractor or FFRDC personnel working on any space acquisition program. We contacted these organizations directly to collect additional military, civilian, contractor and FFRDC workforce data: Air Force Space Command; Air Force Space and Missile Systems Center; Program Executive Office Command, Control, Communications, Program Executive Office Battle Management; and Air Force Research Laboratory. These organizations provided personnel data from their respective manpower sources, such as personnel data systems or manning documents. To assess the reliability of the data, we discussed the data and sources used to compile the data with Air Force officials; reviewed the data for logical inconsistencies; compared the data received from the Air Force workforce data system to data from Air Force Space and Missile Systems Center briefing documents; and compared relevant data received from individual space organizations with data from the Air Force Research Laboratory Space Vehicle Directorate. We collected military, civilian, contractor and FFRDC workforce data directly from the following Army organizations performing space acquisition activities: Army Space and Missile Defense Command; Program Executive Office Missiles and Space; Program Executive Office Command, Control and Program Executive Office Intelligence, Electronic Warfare and Sensors; Communications-Electronics Research, Development and U.S. Army Aviation and Missile Research Development and Army Contracting Command. These organizations provided personnel data from their respective manpower sources, such as personnel data systems or manning documents. To assess data reliability, we discussed the data and sources used to compile the data with Army officials, and reviewed the data for logical inconsistencies. We collected military, civilian, contractor and FFRDC workforce data directly from the following Navy organizations: Space and Naval Warfare Systems Command; Program Executive Office Space Systems; Space and Naval Warfare Systems Center Pacific; and Space and Naval Warfare Systems Center Atlantic. These organizations provided personnel data from their respective manpower sources, such as personnel data systems or manning documents. The Naval Research Laboratory and the Navy’s Program Executive Office for Command, Control, Communications, Computers and Intelligence were originally identified as performing space acquisition activities; however, officials stated they did not have any personnel working on space acquisition activities for at least 50 percent of their time. To assess data reliability, we discussed the data and sources used to compile the data with Navy officials, and reviewed the data for logical inconsistencies. We collected military, civilian, contractor, and FFRDC workforce data directly from: Defense Contract Management Agency; and Missile Defense Agency. To assess data reliability, we obtained information on the data and sources used to compile the data with the agencies’ officials and reviewed the data for logical inconsistencies. The Defense Advanced Research Projects Agency was originally identified as performing space acquisition activities; however, officials stated they did not have any personnel working on space acquisition activities for at least 50 percent of their time. We determined the workforce data were sufficiently reliable to provide estimates of the general size and mix of the space acquisition workforce. To assess any challenges DOD faces in hiring, staffing, and retaining its space acquisition workforce, we interviewed officials from multiple levels within DOD and the Air Force, Army and Navy. In addition to discussing the challenges with the majority of the military service space organizations listed above, we also met with the following DOD organizations: Office of Cost Assessment and Program Evaluation; and Defense Acquisition University. To gather additional insight into the challenges faced at the program office level, we also interviewed officials from a non-generalizable sample of 10 space acquisition programs from the Air Force, Army, and Navy. The selected programs included different types of space acquisitions— such as satellites and launch systems—with a range of dollar values and phases of acquisition. During our review, the Air Force and Army had other space acquisition programs in addition to the ones we selected, whereas the Navy had one space acquisition program according to service officials. The selected programs from each military service included: Advanced Extremely High Frequency (space segment) Evolved Expendable Launch Vehicle (launch segment) Launch and Test Range System (launch segment) Protected Tactical Enterprise Service (ground segment) Space Fence (ground segment) United States Nuclear Detonation Detection System (ground segment) Joint Tactical Ground Station (ground system) Secure, Mobile, Anti-Jam, Reliable, Tactical–Terminal (user segment) Transportable Tactical Command Communications (user segment) Mobile User Objective System (space segment) We also reviewed prior DOD and other space acquisition studies, including reports from the Defense Science Board, Institute for Defense Analyses, Office of Management and Budget, and the RAND Corporation. We conducted this performance audit from November 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Jon Ludwigson (202) 512-4841 or ludwigsonj@gao.gov. In addition to the contact named above, Cheryl K. Andrew (Assistant Director), Peter W. Anderson, R. Eli DeVan, Lorraine R. Ettaro, Lisa L. Fisher, Miranda Riemer, Anne Louise Taylor, and Lauren M. Wright made key contributions to this report.", "summary": "DOD plans to spend about $65 billion from fiscal year 2019 to 2023 on space acquisition programs—including satellites, launch vehicles, ground components, and user equipment. DOD's space acquisition personnel perform a variety of activities, such as preparing and reviewing acquisition documents, to manage or oversee programs that develop or procure space capabilities. DOD recently announced it plans to establish a new Space Development Agency and a United States Space Command. A House Report accompanying a bill for the 2017 National Defense Authorization Act contained a provision for GAO to review DOD's space acquisition workforce. This report examines, among other things, what is known about the size, mix, and location of that workforce. GAO collected data from DOD's acquisition workforce data systems and multiple space acquisition organizations. GAO interviewed officials from these organizations and from a non-generalizable sample of 10 space acquisition programs, representing a range of dollar values and stages in the acquisition process. The Department of Defense (DOD) does not routinely monitor the size, mix, and location of its space acquisition workforce. However, data GAO collected and aggregated from multiple DOD space acquisition organizations show that at least 8,000 personnel in multiple locations nationwide were working on space acquisition activities at the end of 2017 (see figure). Also as shown, military and civilian personnel comprise the majority of the overall workforce, while contractor and Federally Funded Research and Development Center personnel also provide support. Several factors hinder DOD's ability to collect data needed for a comprehensive view of its space acquisition workforce: DOD does not maintain a complete list of its space acquisition programs; DOD's workforce data systems are not configured to identify personnel working on space acquisition activities; and DOD space acquisition personnel are dispersed across organizations and some personnel support both space and non-space programs. Without complete and accurate data, DOD cannot assess gaps in the overall capabilities of the space acquisition workforce. Identifying space programs and collecting such data would also better position DOD to ensure that the appropriate space acquisition personnel are assigned to the new Space Development Agency and the United States Space Command. Finally, comprehensive data on the space acquisition workforce would also be beneficial to support DOD's efforts related to its recent legislative proposal regarding the establishment of the United States Space Force. GAO recommends that DOD (1) identifies the universe of its space acquisition programs and the organizations that support them and (2) collects and maintains data on the workforce that supports these programs. DOD agreed with the first recommendation, but not the second. GAO revised the second recommendation to address DOD's concerns.", "document_type": "gao"}
{"report": "In 1992, the Prescription Drug User Fee Act (PDUFA) was enacted, in part, to provide additional funds for FDA to support the process of reviewing NDAs. PDUFA authorized FDA to collect user fees from drug sponsors to supplement its annual appropriation for salaries and expenses. PDUFA has been reauthorized every 5 years since 1992; most recently PDUFA VI reauthorized the prescription drug user fee program from fiscal year 2018 through fiscal year 2022. As part of each reauthorization process, FDA identifies goals in a commitment letter to Congress. In general, these goals identify a percentage of certain types of applications that FDA is expected to review within specified time frames, including goals for the time the agency takes to complete reviews of different types of NDAs upon initial submission and resubmission. For example, in its commitment letters for PDUFA V and VI, FDA committed to completing its initial review of 90 percent of priority NDAs that involve previously marketed or approved active ingredients within 6 months of receipt. As previously noted, four key features of NDAs are linked to drug development and review processes. For initial NDA reviews, the time frames for FDA’s review that would meet its PDUFA V and VI commitments—its PDUFA goals—vary and are linked to three key features of the NDA. (See table 1.) The target time frame for the initial review of any specific NDA under these user fee commitments reflects the goals associated with all three of the key features. The fourth key feature of NDAs is whether they qualify for one of FDA’s expedited programs. Whether designated as priority or standard, FDA may determine that NDAs for drugs intended to treat serious or life- threatening conditions qualify for development and review under one or more expedited programs. These programs confer specific benefits with the potential to help reduce the development or review time needed to bring a drug to market. For example, some expedited programs provide for more intensive drug development guidance from FDA officials or allow the applicant to submit completed sections of the NDA for review before submitting the entire application. FDA’s expedited programs include accelerated approval, breakthrough therapy designation, and fast track designation. (See table 2.) NDAs must include substantial evidence of a drug’s effectiveness, which is typically drawn from clinical trials. In traditional clinical trials, patients receiving a new drug are often compared with patients receiving a placebo or a different drug. To maximize data quality, these clinical trials are usually randomized (patients are randomly assigned to either the group receiving the new drug or a comparison group) and double-blinded (neither the patients nor the investigators know who is receiving a particular treatment). According to FDA, although this type of study design is often the most powerful tool for evaluating the safety and effectiveness of new drugs, many traditional clinical trials are becoming more costly and complex to administer. Additionally, according to FDA, many new drugs are not easily evaluated using traditional approaches. For example, drugs intended for patients with rare diseases are difficult to evaluate due to the limited number of patients affected by the disease and available for study. The Cures Act was enacted on December 13, 2016, to accelerate the discovery, development and delivery of new treatments—including drugs—for patients. Among other things, the Cures Act includes provisions for FDA to evaluate and facilitate the use of evidence from sources other than traditional clinical trials to support safety and effectiveness determinations for new drugs. For example, FDA was directed to evaluate the potential use of evidence based on data that is routinely collected outside of traditional clinical trials from sources such as electronic health records, medical claims data, and disease registries; evidence from such data sources is referred to as real-world evidence. In the commitment letter associated with PDUFA VI, which was enacted on August 18, 2017, the agency agreed to certain goals relating to the use of real-world evidence in regulatory decision-making and also agreed to certain activities intended to facilitate the development and application of an additional source of evidence known as model-informed drug development. Although these nontraditional sources of evidence were included in NDAs prior to the enactment of the Cures Act and PDUFA VI, at the time this legislation was enacted, most of them were not widely used. For example, according to FDA officials, the NDAs that included real-world evidence were generally for drugs to treat oncology diseases or rare diseases. Our analysis of the 637 original NDAs submitted from fiscal years 2014 through 2018 indicates that divisions differed in the proportions of NDAs they reviewed that had any one of three key features that are linked to time frames for initial review under FDA’s PDUFA goals. As examples: 6 percent of the NDAs reviewed by the dermatology and dental division had a priority review designation, while 56 percent of the NDAs reviewed by the anti-infective division had a priority review designation; 4 percent of the NDAs reviewed by the anesthesia, analgesia, and addiction division involved a new molecular entity, while 52 percent of the NDAs reviewed by the neurology division involved one; and None of the NDAs reviewed by the transplant and ophthalmology division involved a major amendment, while 36 percent of the applications reviewed by the gastroenterology and inborn errors division involved one. (See fig. 1. App. IV provides more detailed information about differences between divisions in the number and proportion of NDAs with these key features.) We also found differences between divisions in the proportion of NDAs that they reviewed under an expedited program—the fourth key feature of NDAs. For example, none of the NDAs reviewed by the metabolism and endocrinology division qualified for one or more expedited programs, while 52 percent of the NDAs reviewed by the antiviral division qualified for one or more expedited programs. (See fig. 2. App. V provides more detailed information about differences between divisions in the number and proportion of NDAs that qualified for one or more expedited programs.) It is not unexpected that divisions differ in the proportion of their applications with key features linked to FDA’s time frames for review or qualification for expedited programs because the divisions are responsible for different products. For example, some divisions, such as the oncology divisions, regulate products for conditions that are more likely to be serious or life-threatening, and therefore the NDAs reviewed by these divisions are more likely to qualify for priority review designation and expedited programs, compared with other divisions, such as the dermatology and dental division. Our analysis of review times for the 637 original NDAs submitted from fiscal years 2014 through 2018 shows that FDA divisions differed in the number of days they took to complete their initial reviews. For example, the median time taken to complete an initial review of an NDA by the anti- infective division was about 2 months faster than the median time taken by the gastroenterology and inborn errors division. (For more information about initial review times, see app. VI.) We found, however, that these differences in initial review times largely reflected key features of the NDAs reviewed by the divisions, particularly those features linked to FDA’s time frames for review under its PDUFA goals. We analyzed initial review times using a statistical regression with two variables reflecting key features of the NDAs—target time frame for review of the application under FDA’s PDUFA goals (in days, from FDA’s receipt of the NDA to FDA’s targeted date for completion of the initial review) and number of expedited programs (0, 1, or 2 or more)—along with division as independent variables. We found that each of these variables was a significant determinant of initial review times. Specifically, our regression analysis shows that on average The shorter the target time frame for initial review of the NDA under FDA’s PDUFA goals, the shorter the initial review, and this target time frame was responsible for the majority of variation in initial review times. The greater the number of expedited programs for which the NDA qualified, the shorter the time FDA took to complete the initial review. Controlling for the effects of these key NDA features, however, we found that most of the divisions’ average review times were similar to (within 2 weeks of) each other. In contrast, the hematology and oncology divisions reviewed applications a bit more rapidly—about 2 or 3 weeks faster—than other divisions. Figure 3 illustrates the results of our analyses. The panel on the left shows the variation in the divisions’ actual average review times. The panel on the right shows the estimated average review times, after accounting for key application features, that is, what the review times would have been if each division had reviewed equal numbers of applications with these key features. We asked FDA officials what might contribute to somewhat faster review times by the hematology and oncology divisions, and FDA officials told us that a number of variables could have contributed to these differences. For example, the officials told us that applicants differ in their level of experience, which can affect the quality of the NDA or the speed of response to FDA’s requests for information; applications differ in complexity; and the oncology and hematology divisions could differ from others in their risk/benefit considerations. As previously noted, some divisions, such as the oncology divisions, regulate products for conditions that are more likely to be serious or life-threatening compared with other divisions, such as the dermatology and dental division, and risk/benefit considerations can differ across conditions that vary in how serious or life- threatening they are. For example, the potential benefits of drugs that carry substantial risks for dangerous side effects would likely be weighed differently if the drug is intended to address a life-threatening illness for which there is no other treatment than if the drug is intended to address an illness that is not life-threatening or for which there is an alternative treatment. FDA has several initiatives underway to evaluate and facilitate FDA review divisions’ and drug sponsors’ use of evidence derived from sources other than traditional clinical trials to support NDAs. (See table 3 for a description of these different evidence sources and each initiative.) According to FDA officials, implementing these initiatives can help ensure that when drug sponsors utilize these sources of evidence in NDAs, the evidence is of sufficient quality to be used in regulatory decision-making and that there is consistency across FDA review divisions in their evaluation of the evidence. FDA officials also said that although complex innovative trial designs might replace traditional clinical trials as evidence in NDAs, real-world evidence is more likely to be used to supplement clinical trial data. Although the initiatives are not restricted to any particular type of disease or patient population, according to FDA officials, some initiatives may be more relevant for certain types of diseases or patient populations than others. For example, according to FDA officials: real-world evidence may be most relevant for diseases that have outcomes that are consistently collected in the health care system. clinical outcome assessments (one aspect of patient-focused drug development) may be most relevant for diseases that are chronic, symptomatic, or affect functioning and activities of daily living. complex innovative trial designs may be most relevant for situations in which the population size is small or limited, such as pediatric populations, or where there is an unmet medical need, such as rare diseases. Our review of FDA documentation and interviews with FDA officials show that FDA has taken steps to implement each of these five initiatives. These steps include conducting public workshops with key stakeholders, issuing guidance for industry and FDA staff, initiating pilot programs, and developing FDA staff capacity, including by providing training and other educational resources. (See table 4 for examples of key activities by initiative.) These and future planned activities—including issuing additional guidance and revising relevant FDA policies and procedures— are intended to address deliverables for FDA to accomplish through 2021 that are outlined in the Cures Act and the PDUFA VI commitment letter. According to FDA officials, the agency intends to meet these deliverables, though, according to these officials, some of the activities implemented under the initiatives, such as certain pilot programs, will likely extend beyond 2021. Although implementation is still in progress for all of the initiatives, FDA officials reported some outcomes. For example, since the launch of the model-informed drug development pilot program, the agency has received two NDA supplements that incorporated model-informed drug development concepts discussed during pilot program meetings. Additionally, officials told us there has been a recent increase in investigational new drug submissions utilizing complex innovative trial designs. FDA officials also reported an increase in biomarker submissions under the drug development tool qualification program, and continued growth of the clinical outcome assessment qualification program. FDA expects that fully implementing the initiatives will lead to further increases in the use of evidence from sources other than traditional clinical trials. We provided a draft of this report to the Department of Health and Human Services for review and comment. The department provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of the Department of Health and Human Services, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. To determine (1) how Food and Drug Administration (FDA) divisions differ in the proportion of new drug applications (NDA) they review with key features linked to review time goals and expedited programs and (2) how FDA review divisions differ in the time taken to complete initial reviews and the extent to which key features of NDAs contribute to those differences, we analyzed data from FDA. We also interviewed FDA officials about the data and their review processes. We obtained data regarding all NDAs submitted to FDA’s Center for Drug Evaluation and Research (CDER) from fiscal years 2014 through 2018. These data included information about features that distinguish NDAs from one another, including which division was responsible for the review. The data also included information through March 31, 2019, about the dates when FDA received and completed a review of each NDA, along with the target dates for completion of review under FDA’s goals in commitment letters associated with the Prescription Drug User Fee Act (PDUFA) reauthorizations for fiscal years 2013 through 2017 (PDUFA V) and fiscal years 2018 through 2022 (PDUFA VI). To ensure meaningful analysis of review times, we excluded NDAs for which FDA had not completed an initial cycle of review. Of 686 NDAs submitted in fiscal years 2014 through 2018, the applicant withdrew 10 NDAs prior to completion of FDA’s initial review and 39 NDAs were still under FDA review as of March 31, 2019, leaving 637 NDAs for which FDA had completed an initial review. To assess the reliability of these data, we conducted a series of electronic and logic tests to identify missing data or other anomalies. These analyses were informed by our review of relevant documentation and interviews with knowledgeable FDA officials. As part of our assessment of reliability, we worked with FDA to identify and correct information about certain NDAs in a small number of instances in which we identified discrepancies. Using these methods, we determined that the remaining data were sufficiently reliable for our purposes. Unless otherwise specified, the results we present are statistically significant at the 0.05 level. To determine how FDA divisions differ in the proportion of NDAs they review with key features linked to FDA’s time frames for initial reviews and expedited programs, we conducted a series of chi-square tests comparing the distributions of the 637 NDAs with and without specific features across divisions. These key features included: whether the NDA had a priority review designation (a designation applied by FDA if the product would provide a significant therapeutic improvement in the safety and effectiveness of the prevention, diagnosis, or treatment of a serious condition when compared to available drugs) or instead had a standard designation; whether the NDA did or did not involve a new molecular entity—an active ingredient that had not previously been marketed or approved for use as a drug in the United States, whether the NDA did or did not involve a major amendment (a submission, while a pending NDA is under FDA review, of additional information that may include a major new clinical safety or efficacy study report or major new analyses of studies, among other things); and whether the NDA did or did not qualify for an expedited program (accelerated approval, breakthrough therapy designation, or fast track designation), programs intended to help reduce the time involved in developing or reviewing certain drugs that have the potential to treat serious or life-threatening conditions. (See table 5 for relevant statistics from these chi-square tests.) To determine how FDA review divisions differ in the time taken to complete initial reviews, we conducted a preliminary regression analysis of 637 NDAs with the number of days an FDA division took to complete its initial review as the dependent variable and division as a single independent variable. We defined the time to complete a review as the number of days from FDA’s receipt of the NDA to the agency’s completion of the initial review by taking regulatory action. To determine the extent to which key NDA features contributed to differences between divisions in the time taken to complete initial reviews, we conducted a multiple regression analysis of the number of days FDA took to complete its initial review with division as an independent variable, along with two other independent variables to control for the key NDA features: Target time frame for initial review of the NDA under FDA’s PDUFA goals. Three key NDA features are linked to time frames for FDA’s initial review under its PDUFA goals—whether the NDA was priority or standard, did or did not involve a new molecular entity, and did or did not involve a major amendment. To control for these three features simultaneously, we counted the number of days from FDA’s receipt of the NDA until FDA’s target date for completion of the initial review under FDA’s PDUFA goals, and used that variable—the target time frame for review under FDA’s PDUFA goals—as an independent variable. We identified five NDAs for which FDA’s review time was exceptionally long in comparison to the target time frame for review under its PDUFA goals, and we asked FDA officials about them. FDA officials stated that these reviews were substantially delayed because of complicated manufacturing site issues, complicated legal and regulatory issues, or emerging public health issues requiring last minute advisory committee meetings—conditions that we deemed sufficiently unusual to exclude these five NDAs from further statistical analyses of review times. Number of expedited programs for which the NDA qualified. Another key NDA feature is whether it qualified for one or more expedited programs, programs with the potential to help reduce the development or review time needed to bring a drug to market. We controlled for this feature by including number of expedited programs (0, 1, or 2 or more) as an independent variable in our multiple regression analysis. Thus, we tested the effect of division on initial review times for 632 NDAs while controlling for the target time frame for review under FDA’s PDUFA goals and qualification for expedited programs. (See tables 6 and 7 for relevant statistics from this multiple regression analysis.) Our multiple regression analysis allowed us to test a specific hypothesis about the effect of division on review times, namely, whether divisions differed in their review times after controlling for the key features of NDAs. This regression analysis did not test a model of review times—that is, we did not attempt to identify all variables that affect review times, nor did we seek to identify the specific set or combination of variables within our data that had maximum explanatory power. Our analyses indicated that variation remained in initial review times, even after we controlled for these variables. It is important to note that an array of factors might be expected to influence review times, including not just those factors that were captured in our analysis, but also factors such as state of the science and quality of the application. With data from 632 NDAs distributed unevenly across 15 divisions, meaningful tests of additional variables or their interactions were not possible. Nonetheless, we conducted exploratory analyses that included other potentially relevant variables in addition to the target time frame for review under FDA’s PDUFA goals, number of expedited programs, and division. In separate regression analyses, we examined (a) the fiscal year in which FDA received the NDA and (b) whether the application was a BLA, an NDA based on information from studies conducted by the applicant, or an NDA based on at least some information from studies not conducted by or for the applicant. We did not find evidence of a consistent effect of either of these additional factors on review times, but in light of the number of NDAs, we cannot exclude the possibility that one or more of these factors affects review times. In a third exploratory analysis, we examined the outcome of the initial review—(a) approval; (b) tentative approval, which FDA grants if the NDA meets requirements for approval, but cannot be approved due to a patent or exclusivity period for a listed drug; or (c) issuance of a letter to the applicant called a complete response letter, in which FDA describes the specific deficiencies the agency identified and recommends ways to make the application viable for approval. This analysis suggested that NDAs that were approved for marketing at the end of the initial cycle of review were reviewed slightly faster on average than other NDAs, but this result should be viewed with caution because a small number of NDAs with certain initial review outcomes were distributed unequally. For example, very few of the NDAs (11) reviewed through one or more expedited programs resulted in tentative approval. The Food and Drug Administration’s (FDA) Center for Drug Evaluation and Research (CDER) divisions differed in the total number of days they took to complete reviews of 637 new drug applications (NDA) submitted from fiscal years 2014 through 2018 and completed by March 31, 2019. (See fig. 4.) Importantly, these times reflect differences associated with the number of completed review cycles, FDA’s target time frames for review under its goals in commitment letters associated with the Prescription Drug User Fee Act (PDUFA) reauthorizations for fiscal years 2013 through 2017 (PDUFA V) and fiscal years 2018 through 2022 (PDUFA VI), and number of expedited programs. Number of review cycles. The number of cycles of review to which the NDAs we examined were subject was largely dependent on factors that were not under FDA’s control, namely, the applicant’s actions and timing. When a cycle of review ends with an FDA action, that action can be (a) approval, which allows the applicant to market the drug, (b) tentative approval, which FDA grants if the NDA meets requirements for approval, but cannot be approved due to a patent or exclusivity period for a listed drug, or (c) issuance of a letter to the applicant called a complete response letter, in which FDA describes the specific deficiencies the agency identified and recommends ways to make the application viable for approval. The applicant may respond to either tentative approval or a complete response letter by resubmitting a revised application, triggering a new cycle of review; it is up to the applicant to decide whether to resubmit the application. In addition, NDAs that were submitted earlier in time would have a greater chance of being resubmitted and reviewed by March 31, 2019, than applications submitted later in time. The number of completed review cycles ranged from one to four cycles: 637 NDAs went through a completed first (initial) cycle review; 99 of those 637 NDAs went through a completed second cycle review; 20 of those 99 NDAs went through a completed third cycle review; 3 of those 20 NDAs went through a completed fourth cycle review. Target time frames for review. Review times reflect differences in time frames for review under FDA’s PDUFA goals. The target time frames for review ranged from less than 6 months to 15 months for the first cycle and from less than 2 months to 9 months for later cycles of review. Number of expedited programs. These review times also reflect differences associated with the number of FDA’s expedited programs for which NDAs qualified. In general, these expedited programs are designed to help reduce the development or review time needed for drugs intended to treat serious or life-threatening conditions. Two of the Food and Drug Administration’s (FDA) expedited programs for new drugs intended to treat serious or life-threatening conditions— breakthrough therapy designation and fast track designation—must be requested by the drug sponsor. These programs are intended to help reduce the development or review time needed to bring a drug to market by offering benefits such as more intensive drug development guidance from FDA officials or by allowing the applicant to submit completed sections of the NDA for review before submitting the entire application. The request is normally made while the drug sponsor is conducting clinical trials or when seeking FDA’s permission to collect clinical trial data, although the request may also be made when submitting a new drug application (NDA) or while the NDA is under review. FDA’s Center for Drug Evaluation and Research (CDER) divisions are responsible for determining whether requests qualify for these expedited programs based on evidence the drug sponsors provide in support of the requests. To qualify for breakthrough therapy designation, the drug sponsor must present preliminary clinical evidence involving one or more clinically significant endpoints that indicate that the drug may demonstrate substantial improvement over available therapies. To qualify for fast track designation, the drug sponsor must either provide evidence demonstrating the drug’s potential to address unmet need or document that the drug is designated as a qualified infectious disease product. FDA may grant or deny the request, or the drug sponsor may withdraw the request before FDA renders a decision. If FDA grants the designation, the drug sponsor may subsequently withdraw from the designation, or FDA may rescind either designation if the drug no longer meets the qualifying criteria. We obtained data regarding all requests for breakthrough therapy and fast track designations submitted to CDER from fiscal years 2013 through 2018. These data included information about which division was responsible for the review and the outcome of the request—whether it was granted or denied or whether the drug sponsor withdrew the request before FDA reached a decision. To assess the reliability of these data, we conducted a series of electronic and logic tests to identify missing data or other anomalies. These analyses were informed by our review of relevant documentation and interviews with knowledgeable FDA officials. Using these methods, we determined that the data were sufficiently reliable for our purposes. We examined these data to determine whether there were any material differences between divisions in the frequency of possible outcomes. Our analyses focused on the outcomes and did not allow us to determine whether divisions differed in their application of the stated criteria. Breakthrough therapy designation. We found few differences across divisions in the frequency of the possible outcomes of requests for breakthrough therapy designation: Of 634 requests for breakthrough therapy designation (including nine requests submitted with or after the NDA submission), 39 percent were granted, 48 percent were denied, and 13 percent were withdrawn by the drug sponsor before FDA reached a decision. Divisions differed widely in the number of requests for breakthrough therapy designation they received, from 0 for the nonprescription drug division to 102 for one of FDA’s two oncology divisions. With two exceptions, the numbers of these requests that were granted, denied, or withdrawn for each division were similar to what would be expected based on the overall frequency of the possible outcomes. Requests to the hematology division were withdrawn more frequently than requests to other divisions (32 percent) and that division denied requests less frequently (17 percent) than other divisions. The neurology division denied more (81 percent), and granted fewer (13 percent), requests for breakthrough therapy designation than other divisions. Within the time period we studied, the drug sponsor withdrew from breakthrough therapy designation after it was granted in six cases and FDA rescinded the designation in 14 cases. Fast track designation. Similarly, we found few differences across divisions in the frequency of the possible outcomes of requests for fast track designation: Of 965 requests for fast track designation (including 35 requests submitted with or after the NDA submission), 71 percent were granted, 24 percent were denied, and 5 percent were withdrawn by the drug sponsor before FDA reached a decision. Again, divisions differed widely in the number of requests for fast track designation they received, from 2 for the nonprescription drug division to 133 for the neurology division. The numbers of these requests that were granted, denied, or withdrawn for each division were generally similar to what would be expected based on the overall frequency of the possible outcomes, although the anti-infective division granted more (91 percent), and denied fewer (6 percent), requests for fast track designation than other divisions. Within the time period we studied, no drug sponsor withdrew from fast track designation after it was granted, nor did FDA rescind any such designation. Pursuant to the Prescription Drug User Fee Act (PDUFA) and its subsequent reauthorizations, the Food and Drug Administration (FDA) collects user fees from drug sponsors to supplement its annual appropriation for salaries and expenses. As part of each reauthorization process, FDA identifies goals in a commitment letter to Congress, including goals for the time the agency takes to complete reviews of different types of drug applications upon initial submission and resubmission. In general, these goals identify a percentage of certain types applications that FDA is expected to review within specified target time frames. For initial NDA reviews—reviews of the NDA as originally submitted—FDA’s target time frames for review that would meet its PDUFA goals vary and are linked to three key NDA features that reflect the drug or the applicant’s action: (1) whether or not the application receives priority review designation, which indicates that the drug could provide significant therapeutic improvements in the safety and effectiveness of the prevention, diagnosis, or treatment of a serious condition when compared to available drugs; (2) whether or not the application involves a new molecular entity—an active ingredient that has not been previously marketed or approved for use in the United States; and (3) whether or not the applicant submitted a major amendment while the NDA was pending, that is, while under FDA’s review. The target time frame for review for any specific NDA reflects all three of these features. Reviews are conducted by one of the agency’s Center for Drug Evaluation and Research (CDER) divisions, each of which specialize in a specific group of drug products, such as hematology or neurology. As shown in table 8, divisions differed in the numbers and proportions of NDAs they reviewed that had the features linked to time frames for review under FDA’s PDUFA goals. The Food and Drug Administration (FDA) may determine that NDAs for drugs intended to treat serious or life-threatening conditions qualify for one or more expedited programs. These programs confer specific benefits with the potential to help reduce the development or review time needed to bring a drug to market, for example, some expedited programs provide for more intensive drug development guidance from FDA officials or allow the applicant to submit completed sections of the NDA for review before submitting the entire application. FDA’s expedited programs include accelerated approval, breakthrough therapy designation, and fast track designation. Reviews are conducted by one of the agency’s Center for Drug Evaluation and Research (CDER) divisions, each of which specialize in a specific group of drug products, such as hematology or neurology. As shown in table 9, divisions differed in the proportions of NDAs they reviewed that qualified for expedited programs. The Food and Drug Administration’s (FDA) Center for Drug Evaluation and Research (CDER) divisions differed in the total number of days they took to complete initial reviews of new drug applications (NDA) received from fiscal years 2014 through 2018 and completed by March 31, 2019. (See fig. 5.) These review times reflect differences associated with FDA’s target time frames for initial review under its goals in commitment letters associated with the Prescription Drug User Fee Act (PDUFA) reauthorizations for fiscal years 2013 through 2017 (PDUFA V) and fiscal years 2018 through 2022 (PDUFA VI). These target time frames for review are linked to specific features of the NDA and ranged from less than 6 months to 15 months for the initial review. These review times also reflect differences associated with the number of expedited programs for which NDAs qualified. John E. Dicken, (202) 512-7114 or dickenj@gao.gov. In addition to the contact named above, William Hadley (Assistant Director), Geri Redican-Bigott (Assistant Director), Aubrey Naffis (Analyst- in-Charge), and Kristen Joan Anderson made key contributions to this report. Also contributing were Sam Amrhein, Todd D. Anderson, Leia Dickerson, Kaitlin Farquharson, Rich Lipinski, and Ethiene Salgado- Rodriguez.", "summary": "Before a drug can be marketed in the United States, FDA must determine that the drug is safe and effective for its intended use through a review of evidence that a drug sponsor—the entity seeking to market the drug—submits in an NDA. The review is conducted by one of FDA's divisions (17, at the time of GAO's review) that each specialize in a specific group of drug products, such as hematology products. NDA reviews are complex, and may involve not only an initial review, but also reviews of resubmissions if the initial review does not result in approval. Under FDA's PDUFA commitments, FDA's goal is to complete reviews of 90 percent of NDAs within specific time frames linked to key features of the NDAs. GAO was asked to examine NDA review times across FDA's divisions. In this report, GAO examines (among other things) differences between FDA divisions in the key features of the NDAs they review and initial review times, as well as the extent to which key NDA features contribute to these differences. GAO analyzed data from FDA's Center for Drug Evaluation and Research regarding 637 NDAs submitted from fiscal years 2014 through 2018. These data also included biologic license applications submitted to the center. GAO excluded NDAs that were withdrawn by the applicant before FDA completed a review, as well as NDAs for which FDA had not completed a review by March 31, 2019. GAO also interviewed FDA officials about the agency's review process and these review times. Four key features of new drug applications (NDA) are linked to the time the Food and Drug Administration (FDA) takes to complete initial reviews of NDAs. Three key NDA features determine the time frames for initial review that would meet FDA's goals under the Prescription Drug User Fee Act (PDUFA) and its reauthorizations, which authorize FDA to collect user fees from drug sponsors: Whether or not the NDA qualifies for the priority review program, which is generally an expedited program for drugs that provide significant therapeutic improvements in the prevention, diagnosis, or treatment of a serious condition when compared to available drugs. The PDUFA goal for review of a priority NDA is 4 months less than for an otherwise similar standard NDA, for which the goal is to complete the review in 10 months. Whether or not the NDA involves a new molecular entity (an active ingredient that has not been previously marketed or approved in the United States). The PDUFA goal for review of an NDA with a new molecular entity is 2 months longer than for an NDA without one. Whether or not the applicant submits a major amendment (additional or new information, such as a major new clinical study) while the NDA is under review. The PDUFA goal for a review of an NDA may be extended by 3 months if the applicant submits a major amendment. The fourth key NDA feature is whether or not it qualified for one or more of three other expedited programs for drugs intended to treat serious or life-threatening conditions. GAO's analysis of 637 NDAs submitted from fiscal years 2014 through 2018 indicated that the proportion of NDAs with these key features differed among FDA review divisions. For example, 6 percent of the NDAs reviewed by the dermatology and dental division had a priority designation, compared to 56 percent for the anti-infective division. FDA has reported that some divisions, such as the oncology divisions, generally regulate products for conditions that are more likely to be serious or life-threatening, and, therefore, those products may be more likely to qualify for priority designation and other expedited programs. GAO found that FDA's divisions differed in the average number of days they took to complete an initial review of NDAs, and these differences largely reflected the key features of the NDAs they reviewed. GAO's analysis shows that the time FDA took to complete an initial review of NDAs was affected by (1) the target time frame for completion of the review under the agency's PDUFA goals, (2) the number of expedited programs for which the NDA qualified, and (3) the division performing the review. GAO also found that the target time frame for review was largely responsible for differences in initial review times. Specifically, NDAs with key features that resulted in shorter target time frames for review under FDA's PDUFA goals had shorter initial review times. Controlling for the effects of these target time frames and the number of expedited programs for which the NDA qualified, GAO found that most of the divisions' average review times were similar to (within 2 weeks of) each other.", "document_type": "gao"}
{"report": "In the United States, HHS is the lead federal agency responsible for public health. Its responsibilities include preparing for, mitigating, responding to, and recovering from public health emergencies. Within HHS, ASPR and CDC prepare for and respond to infectious disease outbreaks. ASPR leads and coordinates national preparedness and response to outbreaks in the United States. It also coordinates and supports advanced research and development, manufacturing, and procurement and deployment of medical countermeasures, such as vaccines, drugs, therapies, and diagnostic tools that can be used in the event of a potential public health emergency to protect the public from harm. CDC monitors and responds to outbreaks by, among other things, studying the link between infection and health; monitoring and reporting cases of infection; and providing guidance to the public, travelers, and health care providers. During public health emergencies, CDC may operate an Emergency Operations Center (EOC) for monitoring and coordinating its response to emergencies—including infectious disease outbreaks of Ebola, Zika, and pandemic influenza—in the United States and abroad. The EOC staff helps with directing specific incident operations; acquiring, coordinating, and delivering resources to incident sites; and sharing incident information with the public. Other agencies perform additional work related to infectious diseases. For example, FDA monitors and protects the blood supply, and NIH makes grant awards that support research related to diseases and modeling. ASPR, CDC, and FDA have different approaches to modeling. In the cases of Zika, Ebola, and pandemic influenza, CDC and ASPR are two key agencies that conduct federal infectious disease modeling efforts. As of February 2020, ASPR had a centralized modeling unit staffed by about nine people, who are a mix of federal and contract employees, according to ASPR officials. At CDC, however, modeling is decentralized and integrated into the individual centers that make up the agency. Some staff work full time on modeling, while others spend part of their time on other tasks. In addition, some of CDC’s modeling efforts are conducted externally. According to CDC, approximately 70 staff members participated in modeling studies, as of October 2018. Of those staff, CDC’s Health Economics and Modeling Unit employed about 10 modelers who have worked on Ebola and other diseases. For Zika, CDC officials responding to Zika said most modeling work was done by one modeler in CDC’s Division of Vector-Borne Diseases, a part of the National Center for Emerging and Zoonotic Infectious Diseases. CDC influenza officials said influenza modeling is conducted by six or seven members of CDC’s Influenza Division. Agency infectious disease modeling activities are not limited to Ebola, Zika, or pandemic influenza. Agency efforts to protect the nation from disasters and emergencies can be organized into two elements: preparedness and response. Infectious disease modeling is one tool used to inform a wide range of decisions related to outbreak preparedness and in response to an outbreak. In the context of infectious disease outbreaks, ASPR and CDC perform work on preparedness and response. For example, ASPR leads the Public Health Emergency Medical Countermeasures Enterprise (PHEMCE), an interagency group that helps develop medical countermeasures—FDA- regulated products including drugs, or devices that may be used in the event of a potential public health emergency to protect the public from harm. CDC may activate its EOC to assist with the response during an outbreak. For example, during the 2014-2016 West Africa Ebola outbreak, CDC activated its EOC in July 2014 to help coordinate activities. CDC personnel were deployed to West Africa to assist with response efforts, including surveillance, data management, and laboratory testing. Since the 1980’s, emerging infectious diseases have resulted in more recurrent disease outbreaks, causing an increasing number of human infections. Emerging infectious diseases have at least one of the following characteristics: they are newly recognized, have emerged in new areas, are newly affecting many more individuals, or have developed new attributes. Some of these diseases—including Ebola and Zika—are zoonotic pathogens, meaning they spread from animals to humans. Zoonotic pathogens can be carried from an animal to a human by another animal, such as a mosquito, chicken, or bat, which is known as a vector. Such pathogens sicken approximately 1 billion people annually. According to the World Health Organization, Ebola causes an acute, serious illness, which is often fatal if untreated. Ebola is introduced into human populations through close contact with the blood and other bodily fluids of infected animals. Humans spread Ebola through direct contact with the bodily fluids of infected individuals or objects contaminated with these fluids. Ebola symptoms include fever, muscle pain, vomiting, diarrhea, impaired kidney and liver functioning, and, in some cases, internal and external bleeding. There have been five Ebola outbreaks since 2014, including the 2014-2016 West Africa outbreak which caused more than 28,600 cases and 11,325 deaths. Since 2018, there has been an ongoing outbreak in the Democratic Republic of the Congo. Figure 1 provides a timeline of Ebola outbreaks since 2014. Zika is a virus that is primarily transmitted through mosquito bites. It can cause symptoms such as fever, rash, conjunctivitis (red eyes), and joint and muscle pain. It can also be transmitted from mother to child during pregnancy, or around the time of birth, or from person to person through sexual contact or blood transfusion. Many infected people do not have symptoms or will only experience mild symptoms. The Zika outbreak that began in 2015 affected individuals infected with the virus in ways that had not been seen with previous outbreaks of the disease. Specifically, during the 2015-2016 outbreak, Zika infection in pregnant women was linked to microcephaly and other severe brain defects, according to CDC. CDC officials said this was the first time in more than 50 years that an infectious pathogen has been identified as the cause of birth defects. Zika was also linked to other problems, such as miscarriage, stillbirth, and Guillain-Barré syndrome, an uncommon disorder affecting the nervous system. In the Western Hemisphere, the first cases of locally- transmitted Zika were confirmed in Brazil in May 2015. In December 2015, locally-transmitted Zika was reported in Puerto Rico. On January 22, 2016, CDC activated its Emergency Operations Center to respond to outbreaks of Zika occurring in the Americas and to increased reports of birth defects and Guillain-Barré syndrome in areas affected by Zika. Within the continental United States, the first locally-transmitted cases were confirmed in Florida in June 2016. The World Health Organization declared Zika a Public Health Emergency of International Concern from February to November 2016. In the spring of 2009, a novel influenza virus emerged, known as influenza A (H1N1)pdm09. According to CDC, it was detected first in the United States and quickly spread across the world, causing a pandemic or global outbreak of a new influenza A virus. This new virus contained a combination of influenza genes not previously identified in animals or people. The virus was very different from other H1N1 viruses circulating at the time, so seasonal influenza vaccines offered little cross-protection against infection with the new H1N1 virus, according to CDC. A vaccine against the new virus was produced, but it was not available in large quantities until late November—after the peak of illnesses during the second wave in the United States. CDC activated its EOC on April 22, 2009, to manage the H1N1 response. From April 12, 2009, to April 10, 2010, CDC estimated there were about 60.8 million cases, 274,304 hospitalizations, and 12,469 deaths in the United States due to the new H1N1 virus. According to CDC, few young people had any existing immunity—as detected by antibody response—to the virus, but nearly one-third of people over 60 years old had antibodies against it, likely from exposure to an older H1N1 virus. Multiple strains of influenza can infect humans, including strains that originate in animals. According to CDC, human infections with an Asian lineage avian influenza A (H7N9) virus were first reported in China in March 2013. During an epidemic that lasted from October 1, 2016, through September 30, 2017, the World Health Organization reported 766 human infections with H7N9 virus, making it the largest H7N9 epidemic. From 2013 to December 7, 2017, there were 1,565 humans infected with Asian lineage H7N9 reported by the World Health Organization. According to CDC, while the risk posed by H7N9 virus to the public’s health was low, the agency was concerned about its pandemic potential. Agencies use infectious disease models to answer a variety of public health questions, including those related to outbreak preparedness and response. A model is a physical, mathematical, or logical representation of a system, phenomenon, or process that allows a researcher to investigate that system, phenomenon, or process in a controlled way. For example, the classic Susceptible-Infected-Recovered or “SIR” model divides a population into three categories: 1) susceptible to the disease, S; 2) infected and infectious, I; and 3) recovered or removed from the infected or susceptible population, R. This model uses equations to determine how many people move between these three categories. The equations contain parameters—numerical descriptors of the disease based, for example, on experiment, expert opinion, or statistics of an ongoing or past outbreak. The equations allow the researcher to estimate how many people are or could be affected by the disease. For example, for past Ebola outbreaks, models estimated that after 40 days, about 44 percent of the population in close contact with infected individuals was susceptible to infection, 31 percent was infected, and 22 percent was recovered. Based on these parameters, equations for transfer between categories, and underlying demographics of the community, an epidemiologist could use the model to estimate how many people within a given town could be susceptible, infected, or removed from the categories of susceptible or infected (due to death or recovery and immunity). Based on model estimates and if a vaccine was available, CDC officials said the decision maker could plan for a specific number of vaccine kits and additional medical staff and supplies to treat infected patients. Models can also help agency officials anticipate future outbreaks, forecast the spread or severity of a disease, and predict the effects and costs of different intervention options. After an outbreak, models can help sort out what happened, what drove the outbreak, and how it compared to past outbreaks. Other tools are available to accomplish some of these tasks, but models are particularly useful when existing data are not sufficient to answer a given question, or when agencies need to integrate data from disparate sources. Infectious disease models can be put into two broad categories: Statistical models. This type of model identifies relationships or patterns that can be used to describe what is occurring or predicts what may occur in the future based on what has occurred in the past. Statistical models tend to use a large amount of data, such as past observed events, to forecast future events, such as disease occurrence, but do not require a fundamental understanding of biological processes or human behavior. They can predict outcomes when causes are not known or understood and when scientific understanding of a disease is limited. They tend to use large amounts of data on past events to forecast future events. Statistical models do not provide full explanations about an infectious disease but may be used when epidemiologists have all or most of the data needed to test a hypothesis. Several benefits can be derived from statistical modeling, including the ability to control for multiple factors that might impact the outcome reviewed, and the ability to isolate the potential effect of infectious disease factors on a particular outcome. Mechanistic models. Mechanistic models rely heavily on scientific evidence and theory related to infectious diseases, and the understanding of disease dynamics or human behavior from prior knowledge—such as biological processes or interactions between people—to represent known processes. They use basic infectious disease science to inform public health guidance and provide insights into outbreak emergence, spread, and control. For example, population-based models can simulate the course of an epidemic by dividing the population into different categories, such as susceptible, infected, and recovered. Mechanistic models can project the likely course of disease transmission, calculate and predict the effect of proposed interventions, and take into account variable conditions, such as human behavior. Both statistical and mechanistic models can range from simpler to more complex. A simpler model may, for example, have fewer parameters (inputs) or equations than a more complex model. According to CDC modelers and an expert, a simpler model may be run with a variety of software, ranging from spreadsheet software to more sophisticated software, whereas more complex models are usually run using sophisticated statistical or mathematical programming languages. As a model becomes more complex, it can become harder to describe, recreate, and understand its internal functioning. Modeling is identified as a beneficial tool in various national plans for disease response and biodefense. These plans do not define the extent to which modeling should occur or how models should be developed for policy, resource allocation, or planning purposes. See table 1 for examples of relevant national plans. CDC and ASPR use models primarily to answer questions from decision makers. CDC and ASPR officials told us, and documents show, that modeling is one source of information that may inform such decisions, along with sources such as expert opinion, surveillance, other prior work on the disease, and an official’s own knowledge. CDC modelers and officials said there is no “rule” as to when to use models, and in some situations, it may not be considered useful. For example, CDC did not use modeling when issuing a travel notice for an Ebola outbreak in specific provinces in the Democratic Republic of the Congo, officials said. Instead, CDC based the travel order on an analysis that considered disease incidence and prevalence, public health infrastructure, and the availability of therapeutics, among other things. Similarly, CDC officials responding to Ebola said modeling may be undesirable when it would take too long to engage the necessary external subject matter experts or when modeling would detract from responding to a disease. CDC and ASPR modelers use models for a variety of purposes. CDC officials said modeling is done differently for each disease, and the amount and type of modeling varies across CDC centers, in part because some centers have less capacity to conduct modeling than others. According to a CDC internal report, the most frequent uses of infectious disease modeling at CDC are: guiding preparedness and response efforts; conducting economic analyses to evaluate the benefits of public health actions, thereby reducing illness and deaths from infectious diseases; understanding pathogen biology, disease transmission, and estimating disease burden; and assessing the effect of interventions and prevention strategies. ASPR modelers and officials said models have provided information about topics such as: resources, including protective equipment, needed to help respond to an Ebola outbreak; the number of therapeutics and vaccine doses needed to respond to Ebola, both in Africa and domestically; expected U.S. demand for Zika diagnostics; and the number of vaccine doses needed to mitigate the spread of pandemic influenza. ASPR modelers and officials said modelers tend to serve in a broad role that can include modeling, data analysis, or other tasks. For example, officials said a modeler could provide a team with day-to-day analytic support and not necessarily spend time developing models or use them. Additionally, ASPR maintains a Visualization Hub that can be used for outbreak planning and response, including outbreaks of pandemic influenza and other emerging infectious diseases (see fig. 2). CDC and ASPR modelers and officials said they generally initiate modeling in response to questions from decision makers. The modelers then work closely with epidemiologists and other subject matter experts to answer the questions. Modeling, according to CDC officials, may be used by individuals or groups within centers, such as division directors, branches, or teams to influence decisions. Who answers a particular question depends, according to ASPR modelers and officials, on the decision maker. Sometimes questions asked will not be within their mission—modelers may suggest such questions be sent to a more relevant agency or part of HHS. CDC and ASPR have modeled to answer a variety of public health questions relevant to Ebola, Zika, and pandemic influenza, and, at times, the results helped inform policy and planning decisions. Modelers and officials provided the following examples: Planning: ASPR modelers and officials said the bulk of the agency’s modeling is related to the planning, development, and deployment of medical countermeasures. For example, these modelers and officials said many clinical trials for vaccines and therapeutics were planned during the 2014-2016 Ebola outbreak response. As a part of these planning activities, ASPR modelers said modelers developed forecasts of future trajectories of disease incidence under a variety of conditions. These forecasts indicated a significant likelihood the disease incidence in Sierra Leone could decrease to a level that would significantly reduce the success of the trials, according to modelers. Additionally, at the beginning of the 2014-2016 Ebola outbreak response, CDC modelers received modeling questions related to the resources needed to effectively limit the spread of the disease, according to CDC documentation. CDC used models to predict the number of Ebola cases that could be expected over time with and without disease interventions such as Ebola treatment units, community care centers, and safe burials. On the basis of this information and other factors, including a United Nations document on Ebola needs, CDC leadership and other U.S. government officials recommended a rapid increase in Ebola response aid, according to CDC documentation. According to CDC documentation, later analyses demonstrated that this increase helped to greatly reduce the actual number of cases, compared to the likely number if prompt action had not been taken. Additionally, in response to the H7N9 influenza outbreak in 2017, ASPR modeled to determine when doses of influenza vaccine should be delivered and how many doses should be administered in order to mitigate a domestic outbreak. This model found that having a vaccine stockpile could be helpful in preventing disease and that a slow effort to administer an H7N9 vaccine could reduce the vaccine’s usefulness. Policy: During the Zika outbreak, CDC modelers and officials said they modeled to determine the potential effectiveness of using pesticides to remove insects from aircraft, trains, or ships. According to modelers and agency officials, the issue arose as concern about Zika virus grew, including from other countries and U.S. agencies, like the Department of Transportation and Department of Defense. The model indicated that humans are more likely than insects to transport Zika on airplanes, and officials therefore concluded that the use of pesticides on airplanes would not be an effective intervention. According to CDC modelers and officials, this modeling resulted in an additional sentence being added to World Health Organization policy, which stated that pesticide use was not expected to be effective. The extent of modeling conducted for Ebola, Zika, and pandemic influenza varied according to the question being asked, along with other factors as follows: Type of question: CDC and ASPR have used models to answer such questions as who should be prioritized for vaccination or treatment, how transmissible a disease is, and how effective certain interventions are likely to be, according to modelers and agency officials. For example, ASPR modelers and officials said they modeled to help estimate the resources needed to respond to an Ebola outbreak; the number of therapeutics and vaccine doses needed to respond to Ebola, both in Africa and the U.S; and the expected U.S. demand for Zika diagnostics. One ASPR official said that, during the 2009 pandemic influenza outbreak, modeling questions were used to provide decision makers with information on what might happen in a given situation. For example, models were used to provide information related to decisions on early vaccine distribution and how this intervention could affect the potential mortality rate. Time to model: How soon decision makers needed information also influenced the extent to which CDC and ASPR modeled. For example, if decision makers needed an answer in a week, modelers would inform the decision makers about how much of the answer they could provide within that time frame, ASPR modelers said. Similarly, CDC modelers and officials said that, in one instance, modelers had only 12 hours to provide decision makers with information. Even estimating the time needed to develop and conduct modeling could represent an additional challenge, according to CDC modelers responding to Zika. According to a CDC article on modeling to inform responses to novel influenza viruses, the amount of time required to develop and execute a model can vary from less than a week to more than a month. Agency officials concurred with these time frames. Personnel and data availability: The availability of qualified personnel was also a factor that affected how much modeling agencies conducted for the selected diseases. For example, CDC modelers and officials said the agency’s Division of Vector-Borne Diseases has focused its resources in other areas, such as building the capacity of states to address vector-borne diseases, and therefore had not invested in individuals with the right skill sets to conduct modeling for the Zika outbreak response. As a result, the division had to call on the three or four CDC modelers from outside of the division who were available to assist with the Zika outbreak response, which limited the amount of modeling that could be performed. Data challenges can also limit the types of modeling conducted. For example, when modeling for Zika, ASPR modelers said they used available information, but data quality and availability limited their ability to model. More data typically become available as an outbreak progresses, but models may be most helpful at the beginning of an outbreak when critical decisions need to be made (see fig. 3). CDC and ASPR do not keep a list of all modeling conducted, and we therefore cannot quantify the extent of their efforts in terms of a number of models. ASPR modelers and officials said modeling is typically one small aspect of the way the agency carries out its mission. One ASPR official said models are never the sole source of information for decision-making. According to NIH officials, NIH does not conduct or fund internal modeling for decision-making purposes. NIH’s Fogarty International Center has conducted self-initiated, internal modeling to answer questions generated from research, and from ideas from Center-held workshops. Two NIH institutes—the National Institute of General Medical Sciences and the National Institute of Allergy and Infectious Diseases—along with NIH’s Fogarty International Center have awarded grants for external modeling research for our selected diseases. However, NIH officials said these efforts were intended to advance science, not for policy or outbreak response. CDC and ASPR modelers and officials said they considered modeling results to a limited extent when making decisions about resource allocation. While modeling can help determine the amount of particular resources needed during an infectious disease outbreak, CDC modelers and officials said it is not central to their resource allocation planning. For example, CDC modelers and officials noted that while a model could inform a decision maker about how many diagnostic testing supplies would be needed based on the range of predicted cases, this would be one input among many into the decision. Decision makers would also consider whether there are other diagnostic test supplies for similar diseases that could be used, the extent of laboratory testing capacity, or the longevity of those supplies. Models can be used to help plan for the cost of interventions by determining the numbers or types of interventions that can be used during a response to an infectious disease outbreak, according to CDC modelers and officials. It can also help decision makers recognize gaps in their ability to implement resource allocation decisions, according to CDC officials. For example, CDC leadership described how modeling input requirements spurred analysis of the factors limiting hospitals’ use of ventilators during a pandemic influenza outbreak. This work, according to CDC officials, helped determine the number of ventilators that should be included in the national stockpile. While modeling results are important to consider during a public health event, ASPR officials and modelers said it is also important to consider concrete financial estimates based on prior experience and whether recommended medical interventions or countermeasures are available or effective. For example, ASPR modelers and officials have occasionally been asked to analyze costs for medical countermeasures, but modelers and officials said that few medical countermeasures typically meet the requirements of decision makers, and existing medical countermeasures are typically unavailable for use in a response. ASPR modelers and officials noted that the usefulness of modeling to the decision maker in these instances is limited. In the event that they were asked to model for such questions, ASPR modelers and officials said time would also be a limiting factor in their analysis. CDC has also developed models to inform decision-making at the state level, specifically to assist state and local public health agencies in developing outbreak response plans. A professional organization of epidemiologists we contacted expressed some concerns with limitations of CDC models, specifically noting that state and local officials viewed CDC models as lacking the level of refinement needed for their state- and local-level planning needs. To follow up, we interviewed officials from a non-generalizable selection of five states based on their reported use of CDC models, the level of selected disease activity in the state, and geographic variation. Two of the five state health departments we contacted reported using one of CDC’s models for Ebola, Zika, or pandemic influenza. These two states confirmed that the usefulness of the CDC FluSurge pandemic influenza model was limited by unrealistic assumptions or a lack of predictive capability, but added that the models were useful to them when considering how to allocate resources or otherwise prepare for a severe pandemic. Officials from one state health department told us they had similar concerns with the CDC Ebola model regarding an unrealistic overestimate of the potential cases, but added that it was useful for informing staff allocation planning as part of their overall response. Officials from another state health department told us they used CDC’s Zika modeling results that indicated how many emergency room visits they could expect and what symptoms it would take to confirm a Zika infection. At the time, state officials said, commercial testing for Zika was not available, so this modeling was very helpful to health officials looking to recommend who hospitals should test based on the presence of Zika symptoms. State health department officials added that many other factors are considered when deciding on resource allocation, such as local leadership and willingness to embrace the public health response. The four HHS agencies that work on infectious disease modeling reported using multiple mechanisms to coordinate their efforts. However, they do not routinely monitor these efforts, evaluate their effectiveness, or report on them to identify areas for improvement. The four HHS agencies that work on infectious disease modeling—ASPR, CDC, FDA, and NIH—reported using multiple mechanisms to varying extents to coordinate such efforts. For example: Emergency Operations Center (EOC). During the response to an outbreak, CDC activates its EOC—a temporary, formal organizational structure for coordinating expertise within CDC and among agencies. The four HHS agencies—ASPR, CDC, FDA, and NIH—used EOCs to coordinate modeling efforts during responses to Ebola, Zika, and pandemic influenza outbreaks. For example, during the 2015-2016 Zika outbreak, CDC’s EOC served as the command center for monitoring and coordinating the response by bringing together CDC scientists with expertise in areas such as arboviruses (the category that includes Zika), reproductive health, birth defects, and developmental disabilities. CDC modelers and officials told us that they had weekly strategy meetings and briefings with response leadership within the EOC where they discussed which modeling questions to prioritize. In general, CDC modelers in the EOC were expected to coordinate with modelers from other agencies within and outside of HHS—such as ASPR, FDA, NIH, and the Department of Homeland Security—to produce timely estimates of cases, hospitalizations, and deaths. These estimates can inform response leadership and enable them to assess the speed and impact of the geographic spread of the pandemic. Modelers in the EOC also provide support to decision makers as they examine the potential effects of various response options. These options include when and how to deploy Strategic National Stockpile assets, such as influenza antiviral drugs and mechanical ventilators. We found the use of EOCs to be consistent with leading collaboration practices we have previously identified, such as defining and articulating a common outcome. Public Health Emergency Medical Countermeasures Enterprise (PHEMCE). The four HHS agencies also participated in PHEMCE, a federal interagency body formed by HHS in 2006 that coordinates the development, acquisition, stockpiling, and recommendations for use of medical products that are needed to effectively respond to a variety of high-consequence public health emergencies. PHEMCE is led by ASPR and also includes partners at the Departments of Defense, Veterans Affairs, Homeland Security, and Agriculture. PHEMCE’s 2017-2018 strategy and implementation plan, its most recent, identified Ebola, pandemic influenza, and emerging infectious diseases more broadly as high-priority threats. PHEMCE leadership could ask modelers to address questions related to these infectious diseases, according to ASPR modelers and officials. According to ASPR officials, such questions tend to support larger response- related efforts, and modeling results are often incorporated into final reports and products. According to ASPR officials, as of February 2020, the PHEMCE structure has been updated and it is unclear how modeling fits into the new structure. We found that coordination through PHEMCE is consistent with leading collaboration practices such as establishing mutually-reinforcing or joint strategies. Working groups. Modelers with the four HHS agencies have participated in working groups related to infectious disease modeling (see table 2). The use of working groups and similar bodies is consistent with leading collaboration practices that we have previously reported as useful for enhancing and sustaining interagency collaboration, such as identifying and addressing needs by leveraging resources. For example, CDC and ASPR modelers participated in the National Science and Technology Council’s Pandemic Prediction Forecasting Science and Technology Working Group, which facilitates coordination among numerous federal agencies. In 2016, this group produced a report that identified challenges in outbreak prediction and modeling for federal agencies and offered recommendations for federal actions to advance the development and effective application of outbreak prediction capabilities. Description This interagency working group, directed by the National Science and Technology Council, is responsible for analyzing the state of infectious disease modeling and prediction, and facilitating coordination among numerous federal agencies. According to CDC modelers and officials, as of October 2018, the charter for this group is no longer active, and it meets on a voluntary, ad hoc basis. According to CDC officials, this group connects modelers by holding seminars, managing an email list, and arranging for members to peer review one another’s models. This group had over 160 participants from various centers across CDC, as of June 2019. During the 2014-2016 Ebola and 2015-2016 Zika outbreaks, the Department of Health and Human Services’ (HHS) Office of the Assistant Secretary for Preparedness and Response (ASPR) established temporary modeling coordination groups that brought together government agencies and academics to share early modeling results and discuss pressing questions that could be answered through modeling, according to ASPR modelers and officials. A wide range of entities participated in these groups, including the four HHS agencies, other federal agencies such as the Departments of Defense and Homeland Security, universities, and foreign entities, such as the World Health Organization and the United Kingdom. According to ASPR modelers and officials, there are no plans to convene modeling coordination groups unless there is an ongoing infectious disease outbreak. Joint model development. ASPR and CDC modelers jointly developed some modeling products during outbreak responses. For example, during the 2014-2016 Ebola response, ASPR and CDC developed a model to estimate future numbers of Ebola patients needing treatment at any one time in the United States. According to a publication describing the model, policymakers have used it to evaluate responses to the risk for arrival of Ebola-infected travelers, and it can be used in future infectious disease outbreaks of international origin to plan for persons requiring treatment within the United States. Building these positive working relationships can help bridge organizational cultures by building trust and fostering communication, which facilitates collaboration and is vital in responding to emergencies. For example, in our 2011 report, we found that, through interagency planning efforts, federal officials built relationships that helped facilitate the federal response to the H1N1 influenza pandemic. Similarly, HHS officials said that federal coordination during the H1N1 pandemic was much easier because of these formal networks and informal relationships built during pandemic planning activities and exercises. Memoranda of understanding. The four HHS agencies have entered into various agreements through memoranda of understanding in order to define their relationships for coordinating infectious disease modeling (see table 3). Generally these memoranda were between individual agencies rather than department-wide. We found that the use of memoranda of understanding was consistent with leading collaboration practices, such as agreeing on roles and responsibilities. Our prior work found that agencies that articulate their agreements in formal documents can strengthen their commitment to working collaboratively. Similarly, CDC modelers and officials said that written agreements can reduce the possibility of misunderstandings or disagreements and help ensure that participants have a mutual understanding of collaboration goals. For example, in the absence of such written agreements, the potential for duplication is increased because agencies could be working on similar types of models without one another’s knowledge. Table 3. Selected Examples of Memoranda of Understanding for Coordinating on Infectious Disease Modeling Collaborating agencies The Office of the Assistant Secretary for Preparedness and Response (ASPR) and Centers for Disease Control and Prevention (CDC) ASPR and the Food and Drug Administration (FDA) Description From 2013 to 2018, CDC and ASPR had a memorandum of understanding to promote collaboration, provide expertise, and facilitate data and information exchange related to infectious disease modeling. This agreement expired in 2018. ASPR modelers and officials told us that, as of August 2019, it had not been updated, and there were no plans to do so. Despite this, according to CDC modelers and officials, the substance of the agreement is still being followed. CDC modelers and officials told us they continue to collaborate with ASPR modelers on the development of models that address questions of mutual interest. For example, for the ongoing Ebola response, CDC modelers and officials said they have kept ASPR informed on modeling efforts, and ASPR shares data on vaccine production that is included in one of the models. ASPR and FDA have a memorandum of understanding to promote collaboration and enhance knowledge and efficiency by providing for the sharing of information and expertise. This memorandum was in place from 2012 to 2017, and was then renewed in 2019. It remains valid unless modified by consent of both parties or terminated by either party immediately upon written notice in the event that a federal statute is enacted or a regulation is issued by a federal partner that materially affects the memorandum. According to FDA modelers and officials, the agreement facilitates collaboration related to FDA’s Medical Countermeasure Initiative and FDA’s role in supporting the HHS-led Public Health Emergency Medical Countermeasures Enterprise (PHEMCE). FDA modelers and officials told us that the agreement supports the frequent, ongoing collaborations between FDA and ASPR, including collaboration related to preparedness for emerging infectious diseases. However, FDA modelers and officials said, while no specific steps have been taken with regards to collaborating on infectious disease modeling under the agreement, modeling assistance could be provided in the future, if needed. Description From 2013-2018, ASPR had a memorandum of understanding with NIH’s Models of Infectious Disease Agent Study program to (1) enable Models of Infectious Disease Agent Study program researchers to work with ASPR as part of public health preparedness and response activities, (2) share data and information, and (3) support model development and use in the HHS modeling hub. This agreement has expired. ASPR modelers and officials told us that, as of August 2019, it has not been updated, and there were no plans to do so. Since 2015, CDC has had a memorandum of understanding with NIH’s Models of Infectious Disease Agent Study program, to promote collaboration and facilitate the exchange of data, tools (models), methods, and information. It was set to expire in February 2020. From 2013 to 2018, ASPR had separate memoranda of understanding with the Departments of Defense and Homeland Security to promote collaboration, provide expertise, and facilitate data and information exchange. The goals of the collaboration in both agreements were to explore ways to, among other things: share analytical approaches and efforts, such as modeling and simulation tools, in support of public health preparedness and response activities; provide personnel as needed to facilitate analytical efforts; and share data and information. These goals were similar to those laid out in the agreement between CDC and ASPR. These agreements expired in 2018. ASPR modelers and officials told us that, as of October 2019, they have not been updated, and there were no plans to do so. Forecasting competitions. CDC and NIH have sponsored formal forecasting competitions to improve modeling for Ebola, Zika, and seasonal influenza. According to a report from the National Science and Technology Council, controlled, multi‐center modeling contests and projects generate valuable insights. For example, they often show that simpler models perform as well as more complex models and that ensemble models, which combine the results of multiple models to predict an outcome, perform better than an individual model. Such competitions are consistent with a leading collaboration practice we previously reported: identifying and addressing needs by leveraging resources. In this case, such leveraging allowed CDC and NIH to obtain additional benefits and insights on models that may not otherwise be available. These modeling competitions can therefore help the HHS agencies better prepare for future outbreaks through coordination with participants. The following are examples of forecasting competitions sponsored by CDC or NIH: Ebola competition. NIH’s Fogarty International Center held an Ebola forecasting competition from August to December 2015, related to the 2014-2016 West African Ebola outbreak, to compare the accuracy of predictions from different Ebola models, among other things. According to NIH modelers and officials, lessons learned from the challenge were that (1) with regard to short-term incidence predictions, ensemble estimates were more consistently accurate than predictions by any individual participating model; (2) as expected, more accurate and granular epidemiological data improved forecasting accuracy; (3) the availability of contextual information, including patient-level data and situational reports, is important for accurate predictions; (4) the accuracy of forecasting was not positively associated with more complex models; and (5) coordination of modeling teams and comparison of different models is important to ensure robustness of predictions. According to NIH officials, based on these lessons and in response to the most recent Ebola outbreak, NIH has established a coordination group to share information about modeling and data sharing for this particular outbreak and a formal model comparison is underway under World Health Organization leadership. Aedes (Zika) competition. In 2019, CDC hosted a forecasting competition related to using models to predict the presence of Aedes mosquitoes, which is a vector for the Zika virus. Evaluating these models can, according to CDC, help clarify model accuracy and utility, the seasonal and geographical dynamics of these mosquitoes, and key directions for future research. According to CDC documentation, these advances can contribute to improved preparedness for arboviral invasion in the United States and in other regions where Aedes suitability may be limited and changing. CDC plans to evaluate forecasts for this competition in early 2020, as soon as final surveillance data for 2019 are available. FluSight (seasonal influenza) competition. CDC holds an annual seasonal influenza forecasting competition—known as FluSight—to facilitate efforts to engage external researchers to improve the science and usability of seasonal influenza forecasts. The results of the competition are evaluated by the CDC Influenza Division, which works with state and local partners to determine whether the results are useful to them and if there are other metrics, milestones, or targets that would be more helpful in making public health decisions. According to CDC officials in February 2020, the results from the FluSight competition are not directly incorporated into pandemic influenza forecasting because the most accurate seasonal influenza forecasts would not necessarily be the most accurate pandemic influenza forecasts. According to these officials, the overall lessons learned from the FluSight competition relate to how to quantify, visualize, and communicate model results and model accuracy, as well as the value of forecast ensembles to summarize multiple models. CDC officials said these lessons are incorporated into pandemic influenza forecasting plans. Coordination with academic and other modelers. CDC coordinated infectious disease modeling efforts with academic and other modelers through various means, including the following: Intergovernmental Personnel Act agreements. CDC has used agreements under the Intergovernmental Personnel Act of 1970 to collaborate with external experts on modeling efforts. For example CDC’s Division of Vector-Borne Diseases had an agreement from 2014 to 2017 to assign a CDC official to the Harvard T.H. Chan School of Public Health. The agreement was to help CDC integrate with a larger modeling community and provide the Harvard School of Public Health with expertise in arboviral diseases and applied public health. Vector-Borne Disease Centers of Excellence. CDC has funded the Vector-Borne Disease Centers of Excellence, which are engaged in modeling-specific projects. In 2017, CDC established five universities as regional centers of excellence to help prevent and rapidly respond to emerging vector-borne diseases across the United States. According to CDC, the goals of the centers are to build effective collaboration between academic communities and public health organizations at federal, state, and local levels for surveillance, prevention, and response, among other things. Support for other governmental entities. CDC has coordinated with other entities—such as state and local officials—to provide modeling tools, estimates of case counts, or effects of interventions during the Ebola, Zika, and pandemic influenza outbreaks. For example, CDC developed pandemic influenza models for state and local health departments to use in influenza pandemic planning activities. The tools are available on the CDC pandemic influenza website and from ASPR’s emergency preparedness information portal. As previously discussed, officials from two of the states we spoke with said they generally were unaware of the availability of the models. According to CDC modelers and officials, these models were developed in the mid- 2000s for pandemic influenza planning and remain useful but had not been a priority to update because they have not received a request to do so. Informal collaboration. CDC has engaged in a range of informal collaborations related to infectious disease modeling. According to CDC modelers and officials, modelers often develop relationships through conferences or other contacts. For example, CDC modelers and officials said they informally collaborated on Ebola modeling needs with academic institutions, as well as modelers and analysts in the World Health Organization and other U.S. government agencies, such as the Federal Emergency Management Agency. For example, CDC modelers and officials told us that model estimates produced under collaboration with academics helped inform decisions about how many beds to be ordered and delivered on the ground in West Africa during the 2014-2016 Ebola Outbreak. Similar to the forecasting competitions described above, such informal coordination mechanisms are consistent with the best practice of identifying and addressing needs by leveraging resources, thus obtaining additional benefits that may not be available if they were working separately. For example, we have previously reported that informal collaboration mechanisms—such as building relationships between key personnel and soliciting input for research projects—can provide the opportunity to leverage expertise. CDC and ASPR modelers and officials did not routinely monitor, evaluate, and report on coordination efforts for infectious disease modeling. While CDC did conduct after-action reviews for Ebola and Zika, which included a review of modeling efforts, such reviews are not routine outside of a response and do not examine modeling coordination between agencies. ASPR modelers and officials told us they saw no reason to monitor coordination efforts under the memorandum of understanding with CDC because such memoranda outline expectations rather than requirements. However, we have found that agencies that create a means to monitor, evaluate, and report the results of collaborative efforts can better identify areas for improvement. We have previously reported that progress reviews or after action reviews can be useful mechanisms for monitoring, evaluating, and reporting on collaborative efforts. For example, we previously reported that, to monitor, evaluate, and report on the status of achieving the Healthy People 2010 objectives, HHS held progress reviews in which the federal agencies with lead responsibilities for a focus area reported on the progress towards achieving the objectives. During these reviews, the participating agencies discussed the data trends, barriers to achieving the objectives, strategies undertaken to overcome barriers, and alternative approaches to attain further progress. By holding similar progress reviews in which CDC and ASPR evaluate and report on coordination efforts for infectious disease modeling, these agencies could be better positioned to identify and address challenges prior to infectious disease outbreaks occurring, which could lead to improved responses. Further, there is the potential for overlap and duplication of modeling efforts across agencies, which may not be identified if coordination efforts are not effectively being monitored, and which could lead to inefficiencies. The memorandum of understanding between CDC and ASPR had expired in 2018. Agency officials told us they had no plans to review or update the agreement. According to ASPR modelers and officials, the agreement has not been updated because it was not a priority and the substance of the expired agreement is being followed. However, without an active agreement in place that clearly defines the goals of the collaborative effort and the roles and responsibilities of participants, a lack of understanding and agreement becomes more likely, particularly as agencies’ priorities evolve over time. Our prior work on leading collaboration practices found that agencies that articulate their agreements in formal documents can strengthen their commitments to working collaboratively, and that such agreements are most effective when they are regularly reviewed and updated. Further, we found that the memorandum of understanding between ASPR and CDC was not fully implemented when it was active. For example, according to this agreement, CDC was to appoint a designee to participate in a steering committee related to modeling within HHS. However, ASPR modelers and officials told us that this steering committee was never formed because of changing leadership and priorities. They told us that HHS does not have any intention to form such a steering committee in the future. However, our past work shows creating a steering committee or other similar coordination mechanism could help facilitate monitoring of coordination efforts. We similarly found that other memoranda of understanding related to infectious disease modeling were not fully implemented. For example, although ASPR had a 2013-2018 memorandum of understanding with NIH’s Models of Infectious Disease Agency Study program, ASPR modelers and officials said they rarely use models funded by NIH, including those funded through the program. In particular, ASPR modelers and officials recalled only using one such model in recent years. That model, known as “FluTE,” is an influenza model that was used as part of a larger study on vaccine availability. However, ASPR modelers faced challenges in using this model. Specifically, these ASPR modelers and officials said the FluTE model initially was not compatible with ASPR’s computer system, so software engineers had to modify the source code to resolve the compatibility issue. The model did not have documentation describing its parameters, according to ASPR modelers and officials, so they had to read through the model’s source code to understand them. Similarly, regarding a separate agreement between ASPR and FDA, FDA modelers and officials said that, while there is ongoing information sharing, no specific steps have been taken with regard to collaborating on infectious disease modeling under the agreement. However, these modelers and agency officials said that modeling assistance could be provided in the future, if needed. We identified four elements of practices for developing and assessing models: (1) communication between decision maker and modeler, (2) description of the model, (3) verification, and (4) validation. We determined that CDC and ASPR generally followed these GAO-identified practices for 10 models we reviewed. However, for four of the 10 models, CDC modelers did not provide all of the details needed in the verification steps to reproduce their model results, which is inconsistent with HHS guidelines on transparency and reproducibility. According to our interviews with agency modelers and experts, along with our review of selected literature, there are no documented standards that prescribe the steps agencies must or should follow when developing and assessing models. However, based on our interviews and review, we identified four broad elements of the modeling process that modelers generally consider. They are: 1 communication between modelers and officials to refine questions to be addressed by the model, such as geographic spread of the disease and total cases of the disease; 2 description of the model, including detailed descriptions of assumptions and data sources used; 4 validation. Figure 4 outlines the model development and assessment process. Based on our assessment of 10 selected models, we found that CDC and ASPR generally took steps that corresponded to our four elements, and agency modelers generally agreed with our assessment of each model. See table 4 for more information on the elements. See appendix III for a list of models we reviewed and a complete list of the steps we identified that make up each element. Communication between modeler and decision maker. In all 10 agency models we reviewed, we found that agencies took all the steps we identified for communication between decision maker and modeler. In some cases, these steps were formalized, while in others they were informal. For example, CDC modelers responding to Ebola ensured communication with decision makers by following a memo template they developed, which has a section requiring modelers to communicate key aspects of their model. These modelers noted, however, that they would not follow all the steps in their memo template for models developed during an outbreak because of time constraints. CDC modelers responding to pandemic influenza noted they do not have formal best practices for communication about key model aspects to decision makers, and a CDC modeler responding to Zika highlighted the role of CDC’s Emergency Operations Center (EOC) in communication between decision makers and modelers, which is activated only during a response. ASPR modelers noted that—as a best practice—they hold a discussion for all new models, in which decision makers describe what they are looking for and modelers describe what they can provide. Description of the model. In nine of the 10 models we reviewed, modelers took all steps we identified for describing their model type, inputs, outputs, assumptions, and limitations. In one case, ASPR’s “flumodels” package, the agency did not carry out the step of describing the model’s limitations. ASPR modelers told us they did not do so because they expected the model’s intended users—primarily federal public health modeling experts—would understand the limitations of their model, an assumption we find reasonable. Verification. In six of 10 models reviewed, we found agency modelers followed most of the steps we identified for model verification. However, in four of the seven CDC models reviewed, CDC did not publish the model’s code, a part of model reproducibility and a model verification step. We examine CDC’s policy and efforts on reproducibility in more detail below. Validation. For four of the 10 models we reviewed, agencies performed few validation steps. In all three CDC pandemic influenza models we reviewed, and the ASPR Zika model, sensitivity analysis was the only validation step performed. CDC influenza modelers said they did not perform other validation steps because of a lack of comparable external models or applicable data which could be used for other types of model validation. For example, they said they could not validate their models using real-world data because they made projections for scenarios that did not come to pass (e.g., an unmitigated pandemic influenza outbreak). They said they have continued to look for comparable models that could be used to cross-validate their model estimates. ASPR modelers responding to the Zika outbreak also did not have access to comparable external models or applicable data to confirm their model projections, but have since attempted to validate their model. For the other six models we reviewed, agencies carried out most but not all validation steps. For example, CDC modelers responding to Zika also said they did not perform cross-validation (comparison of different model results to each other) for their Zika model because of a lack of comparable models. However, these ASPR and CDC Zika modelers said they have attempted to validate their model since its publication as new data emerges, and we found this occurred. Assessing Model Validity Assessing model validity means determining whether a model is sufficiently accurate for its purpose. Several methods are available, including the following: Modelers can compare the results of the model against real-world data the model was designed to predict. If there are no such data, another method is to determine how much the model projections change in response to changes in input data. This is known as model sensitivity analysis. Modelers can also withhold a part of the available data in building the model and then confirm the model can reproduce the withheld data. real-world data is to run the model along with a separate, independent model using the same input data, and comparing the outputs. CDC modelers and ASPR modelers responding to Zika followed identified practices and validated their model projections for the Zika outbreak, although their efforts yielded mixed results for model performance. CDC modelers responding to Zika attempted to estimate whether there was an enhanced risk of microcephaly in infants born to expectant mothers infected with Zika. Using data available during the initial stage of the outbreak, they calculated the enhanced risk to be between 0.88 and 13.2 percent if the mother was infected in the first trimester. In two subsequent studies using later data on the actual incidence of microcephaly as a result of the outbreak, other researchers found the enhanced risk was within the bounds of CDC modelers’ earlier projections: a 10 percent enhanced risk in one study and an 8.3 percent enhanced risk in the other. In the second case, ASPR modelers attempted to estimate potential new cases of Guillain-Barré syndrome, a rare disorder in which the body’s immune system attacks part of its own nervous system, in places burdened by Zika infection. Their initial projections were that there would be between 191 and 305 new cases in Puerto Rico, a three- to five-fold increase above the number normally expected. ASPR modelers attempted to verify these results themselves and found that the incidence did increase, but only two-fold, to 123 new cases. through independent performance evaluations. For example, agencies sometimes host modeling competitions, in which independent modelers compare the predictive performance of multiple models under controlled conditions using standardized data. The National Institutes of Health hosted an Ebola forecasting competition in 2015, and the Centers for Disease Control and Prevention (CDC) launched its FluSight competition in 2013. The Challenge of Modeling During an Outbreak. Early in the 2014-2016 Ebola outbreak, Centers for Disease Control and Prevention (CDC) officials faced the challenge of answering questions with limited data and time. In order to estimate the potential number of future cases and to aid in planning for additional disease-control efforts, CDC developed EbolaResponse, an Excel spreadsheet-based model that could forecast how interventions would impact the outbreak. Using EbolaResponse, CDC predicted in early September 2014 that 1.4 million cases of Ebola could occur in Liberia and Sierra Leone by January 2015, if the world health community did not increase interventions. These estimates included a correction factor intended to account for the underreporting of cases and that, according to officials, was to represent model uncertainty. Partly because of these estimates of rapidly increasing cases, CDC and others increased intervention by sending more treatment units, personnel, and medical supplies in late 2014. EbolaResponse was created to model the effects of intervention, and it later turned out to be unreliable for the 4-month forecast that CDC used to support its request for increased intervention. Independent analysis found that the model could forecast cases up to a month ahead well but could not provide any measure of uncertainty. Furthermore, the model was unable to make accurate forecasts much beyond 3 months, a limitation that was common among the models used during the outbreak. CDC later reported that roughly 8,500 cases, or 34 percent of the corrected EbolaResponse prediction of 25,000 cases, occurred in Liberia by the end of January 2015. We also found that CDC and ASPR modeling approaches varied somewhat, while generally remaining within the bounds of our identified practices. For example, all the agency modeling groups reviewed their model assumptions, but they also varied in whether this review was formal or informal and internal or external. CDC modelers responding to Ebola use a formal internal peer review process during non-outbreak periods, as well as a detailed checklist to ensure communication with decision makers, full consideration of model inputs and outputs, quantification of model uncertainty, and validation of the model. By contrast, CDC modelers responding to Zika told us they do not have a formal system for evaluating their models, and instead rely on their own review of model assumptions. ASPR and CDC pandemic influenza modelers told us their modeling approach also relied on peer review, but the review was done by external experts; informally for ASPR and formally for CDC pandemic influenza modelers. There are several reasons agency modeling approaches can vary. According to agency modelers, agency modeling practices can be influenced by the availability of time, data, and comparable models. For example, CDC pandemic influenza modelers and officials said they follow a shortened process when facing time constraints by documenting model development in a journal publication after the model has already been put to use. Similarly, CDC modelers responding to Ebola noted that, during a response, a lack of time may mean models are not reviewed through CDC’s formal clearance process; instead, a more informal review of model results may occur. CDC and ASPR modelers also described variation in the complexity of the models they use. They said they sometimes use both simple and complex models for the same disease and during the same outbreak. CDC modelers and officials responding to Ebola said that they preferred models run in spreadsheet programs for their transparency and communicability, whereas CDC influenza modelers mostly use dedicated statistical software programs to run models and spreadsheets for communicating with state and local health departments. ASPR modelers develop more complex prediction models so that they can be reused to answer more than one question, as opposed to models run in spreadsheet programs that are designed to answer one question. Experts and agency modelers generally agreed that infectious disease models should not be more complex than is necessary to answer the questions they were developed to address. A simpler model may be run on a variety of software programs, ranging from spreadsheet programs to specialized programming languages that can do statistical analysis. One downside of models run in spreadsheet programs, according to CDC influenza modelers, is that it is harder to conduct quality control measures. Two experts we spoke to, along with CDC Zika modelers, also expressed concerns with reliability and reproducibility of models run in spreadsheet programs. Since 2002, HHS agencies responsible for disseminating influential scientific, financial, or statistical information have been required to ensure methods used to develop this information are “reproducible.” A 2019 report from the National Academies of Sciences, Engineering, and Medicine noted that the scientific enterprise depends on the ability of the scientific community to scrutinize scientific claims and to gain confidence over time in results and inferences that have stood up to repeated testing. As part of this process of scrutiny, a study’s data and code should be made available so that the study is reproducible by others. The National Academies report defines reproducibility as obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis. Reproducibility is specifically addressed earlier in this section in our discussion of model verification, a step that requires making code available for independent review. HHS requires its component agencies to either follow HHS department guidelines on reproducibility or to ensure their own guidelines include a high degree of transparency about the data and methods used to generate scientific information. HHS guidelines require that, in a scientific context, agencies identify the supporting data and models for their published scientific information and provide sufficient transparency about data and methods that an independent reanalysis could be undertaken by a qualified member of the public. When asked whether CDC has specific policies related to reproducibility that would have applied to provision of model code in their published scientific research, CDC referred to its guidelines developed in response to the 2002 HHS Guidelines. However, CDC guidelines do not contain any reference to reproducibility, models, or provision of model code. CDC guidelines for review of scientific information provided to the public focus on completeness, accuracy and timeliness, data management and analysis, clarity and accuracy of presentation, and validity of interpretation of findings. CDC’s policy on public health research and non-research data management and access does not make any reference to reproducibility or model code. This lack of reference to reproducibility in CDC’s guidelines and policies is not in accordance with HHS guidelines. Our review found four instances in which CDC modelers did not provide model code when they published their models. CDC modelers said in some instances, issues with publication formats made the code difficult to share, they did not have time to produce a user-friendly version of the code, or they would share the code upon request. By contrast, ASPR modelers provided code for every model within our review when they published their models. While neither agency cited a specific HHS policy that required them to share model code, ASPR modelers noted that their internal peer review process typically includes sharing model source code with other modelers within PHEMCE. In our review of HHS guidelines and agency-specific guidance for these HHS guidelines, we found that, of three published agency guidance, two require reproducibility, or transparency for the methods used in the reports they issue to the public. Of these agencies, CDC was the only one that did not explicitly require transparency or reproducibility. The National Academies report noted that researchers have to be able to understand others’ research in order to build on it. This report also notes that the ability of qualified third parties to reproduce a model using published code is important because it can reveal mistakes in model code, which can lead to serious errors in interpretation and reported results. If researchers do not share an important aspect of their study, such as their model code, it is difficult to confirm the results of their research and ultimately produce new knowledge. One agency official acknowledged the importance of releasing model code, noting that HHS could benefit by ensuring policies across the agency are consistent regarding reproducibility and transparency in modeling. By not specifically addressing reproducibility in their policy on dissemination of scientific information, CDC risks undermining the reliability of the scientific information they disseminate to the public. Based on our review of documents and reports from agencies, as well as expert and agency interviews, we identified three categories of challenges that CDC modelers and officials and ASPR modelers faced when modeling for Ebola, Zika, and pandemic influenza, along with steps they took to address the challenges. The categories are data, resources, and communicating results. According to a 2016 report from the National Science and Technology Council (NSTC), obtaining timely and accurate data and information has long been a major challenge to an effective response during an infectious disease outbreak. One expert described reliable data as a modeler’s most limited resource. Until data of sufficient quality and quantity are available and usable, the predictive value of models will be limited. Agency modelers and officials provided examples of data-related challenges, which we categorize as follows: Data Access. Public health data, according to one expert, often has access restrictions. For example, ASPR modelers said their ability to access data during the 2014-2016 Ebola outbreak was reduced by a need to enter into agreements with data-owning countries in order to obtain patient data. Modelers said there were agreements between CDC and data owners, but further agreements would have been required for ASPR to obtain data because the agreements did not authorize CDC to share data with its partners. In addition to the example above, the lack of data sharing agreements during the 2014- 2016 Ebola outbreak response led to modeling projects being delayed, according to a CDC publication. ASPR modelers said their inability to obtain data without a data-sharing agreement made it challenging for them to developing a current, reliable estimate of Ebola incidence before modelers could start creating future estimates of disease incidence. They said that, as a result, they instead developed a statistical model, which provided less reliable estimates of future numbers of disease cases than they would have preferred. Modelers said they worked to address this challenge by obtaining data and indirect information through personal relationships with other modelers. In addition to the example provided above, CDC modelers and officials responding to Ebola described experiencing data access challenges. Data availability. Without sufficient data, models may be unable to identify an epidemic’s key drivers, which could result in misdirected intervention efforts. For example, ASPR modelers noted that during the 2015-2016 Zika outbreak response, there were substantial limits on available data, and data that were available could be unreliable and delayed. They said it was very difficult, and in many cases effectively impossible, to determine the accuracy of forecasting models for the evolving Zika outbreak. In addition, CDC officials and modelers responding to Ebola, Zika, and influenza described encountering limits on available data as an ongoing challenge. Steps that modelers said they have taken to address data availability challenges include designing models to use a minimum amount of data, building trust and communication with stakeholders who might be able to provide additional data, and updating data systems to provide all available information. According to CDC modelers, data availability will likely continue to pose a challenge to public health responses. Data collection. There is limited manpower during an infectious disease outbreak response, which can limit the health care system’s ability to collect data, according to CDC modelers and officials responding to Ebola and ASPR modelers. ASPR modelers said if a provider has to fill out a time-consuming form, then they will be delayed in treating the next patient. In order to address this challenge, CDC modelers and officials and ASPR modelers said data requesters should ask for the minimum amount of data needed. For example, CDC modelers and officials said they focus on understanding what data are essential, how they are collected, and the policy implications of reporting those data. A 2016 NSTC report recommended the federal government address this challenge by identifying questions likely to arise during an outbreak response, in order to help define and prioritize data collection and modeling goals. Data quality. Experts said creating models with low-quality data can result in inaccurate models that may not provide clear answers to decision maker questions. For example, CDC modelers and officials responding to the 2015-2016 Zika outbreak said the data quality varied, based on many factors such as surveillance systems that were doing different things and defining reporting Zika cases differently, and the availability of diagnostic testing. Because of data quality concerns, there were questions about whether modeling could be conducted, but through discussions modelers and agency officials said they were able to address challenges. To address such challenges, CDC modelers and officials responding to Zika said they worked to improve public data sharing, sent an official to the Pan- American Health Organization to help interpret data and understand the outbreak from an international perspective, and used modeling methods appropriate for data with high levels of uncertainty. In addition to the example provided above, CDC modelers and officials responding to Ebola, ASPR modelers, and experts described experiencing data quality challenges. Data integration. CDC modelers and officials responding to Ebola and Zika also faced the challenge of integrating multiple data sets, which may not be standardized or in a readily usable form. For example, CDC modelers and officials responding to Zika found it challenging to integrate data as the definition of the disease was refined over time. As the definition got more specific and monitoring systems became available, it was hard to establish data trends, these officials said. Further, there were variations in who would be tested, with all people who exhibited symptoms being tested in some areas, and only pregnant women in others, and also when data would be placed into a combined form and reported to state, national, or international officials, according to these officials. This integration issue may have complicated efforts to conduct modeling such as determining the risk of microcephaly in infants over time. In order to address this challenge, Zika modelers said they set up an online data repository to, among other things, standardize shared data. CDC modelers and officials responding to Ebola and Zika, along with experts, said finding staff with sufficient training to support modeling during an infectious disease outbreak represented an ongoing challenge. For example, CDC modelers responding to Zika said it can be difficult to find modelers with both an epidemiological background and skills in coding and mathematics. Modelers and agency officials said those who had the correct skills were in high demand, and it was difficult to fully engage them in the Zika outbreak response. They said they could have conducted more modeling or completed modeling efforts more rapidly if they had had access to more modelers with the right skills. To address this challenge, modelers participate in trainings on how to communicate what models can and cannot do, participate in working groups that support modeling efforts, employ the Intergovernmental Personnel Mobility Act Program, maintain collaborations with external partners, and host students and researchers. ASPR modelers said they faced personnel challenges in their modeling efforts but that they were wide-ranging and not specific to Ebola, Zika, or pandemic influenza. According to a 2016 NSTC report, time constraints make it challenging for researchers to keep up with scientific literature during an outbreak. CDC influenza modelers said they faced this challenge and that they conduct weekly searches for new influenza publications, which normally identify about 150 publications each week. To address this challenge, modelers said they conduct literature searches, share the responsibility of reviewing publications and informing others of their content, talk to experts, and attend conferences. Modelers said this challenge was more easily addressed than others. Communicating model results can be difficult and, as modelers and agency officials pointed out, decision makers will not give credence to results from a model they do not understand. Model results, according to CDC influenza modelers, are often nuanced and complicated, and officials have to think about what pieces of information are the most important to convey to a decision maker, the public, or health officials. Furthermore, as one expert noted, the complexities of modeling can get lost in translation, especially with the media, which may focus on only a worst-case scenario. When modeling for infectious diseases, appropriately communicating complex information has been described as a constant challenge, and CDC influenza modelers described it as their biggest challenge. CDC influenza modelers particularly noted the challenge of communicating uncertainty. CDC influenza and ASPR modelers said if decision makers did not understand the models, they could misunderstand the results, which, according to ASPR modelers, could lead to errors in decision making. CDC modelers and officials responding to Ebola and Zika, CDC influenza modelers, ASPR modelers, and experts described experiencing challenges communicating model results to decision makers. Clear communication may help prevent misunderstandings. For example, one review article said officials may not understand what models can and cannot do before an epidemic, and modelers may not be fully aware of a decision maker’s needs. An expert said there is a need to constrain the use of models intended to inform decisions so that the model does not over- or under-influence a decision maker. And, according to ASPR modelers, decision makers sometimes want a model to make a decision for them, although models can only inform the decision making process. They said this is less of a problem during an outbreak response, when decision makers know they have to act based on incomplete information. Some steps officials described taking to address communication challenges were similar across CDC and ASPR officials. For example, CDC modelers and officials and ASPR modelers said they took steps to improve communication, such as working to develop relationships outside of an outbreak and to improve how data are visualized. For example, ASPR modelers and officials said they provided decision makers with a website that displays an interactive influenza model known as ShinyFlu. The website lets users adjust a model to see how its results could change based on its inputs used. However, modelers said this only works if the decision maker is willing to engage with data. Other steps to address communication challenges were not discussed by all modelers we spoke to. For example, ASPR modelers said that, when they use models with high uncertainty, they do additional research to assess and communicate how a model could be misrepresenting a real- world problem. Additionally, CDC modelers responding to Zika and CDC influenza modelers said they sometimes use the language of weather forecasting—which provides information on the risk of an event occurring over a specified period of time—to help communicate model outcomes. For all 10 of the models we reviewed, modelers communicated all the information they had agreed to provide to decision makers, including information about model uncertainty. Agency modelers and officials said they provided this information through discussions with decision makers and by showing decision makers the results of multiple modeling situations to convey uncertainty. Infectious disease modeling is one tool that can provide decision makers with valuable information to support outbreak preparedness and response. In particular, modeling can help answer questions that are difficult to address in other ways because of practical, ethical, or financial reasons. Federal agencies have recognized the importance of modeling. CDC and ASPR reported using it to inform policy and planning questions and, to a more limited extent, to inform planning and the use of resources. HHS agencies that work on infectious disease modeling—ASPR, CDC, FDA, and NIH—reported using multiple mechanisms to coordinate their modeling efforts, including working groups, memoranda of understanding, and coordination with academic and other external modelers. The use of these mechanisms was consistent with many leading collaboration practices, such as defining and articulating a common outcome and addressing needs by leveraging resources. However, HHS does not routinely monitor and evaluate its coordination efforts, as called for by another leading collaboration practice, which limits the department’s ability to identify areas for improvement. Further, there is the potential for overlap and duplication of modeling efforts across agencies, which may not be identified if coordination efforts are not effectively being monitored, and could lead to inefficiencies. By holding progress reviews in which CDC and ASPR evaluate and report on coordination efforts for infectious disease modeling, these agencies could be better positioned to identify and address challenges prior to infectious disease outbreaks, which could lead to improved response efforts. CDC and ASPR modelers generally followed GAO-identified modeling practices, with the notable exception of model verification. Specifically, CDC did not make model code available to others for four of the seven CDC models we reviewed. HHS does not have a policy that requires its agencies to share model code, but it does require its component agencies to either follow its guidelines or ensure that their own guidelines include a high degree of transparency to facilitate reproducibility by qualified third parties. Without sharing code and other important information, CDC cannot ensure that its models are reproducible, a key characteristic of reliable, high-quality scientific research. In order to facilitate HHS infectious disease modeling efforts, we are making two recommendations. The Secretary of Health and Human Services should develop a mechanism to routinely monitor, evaluate, and report on coordination efforts for infectious disease modeling across multiple agencies. (Recommendation 1) The Secretary of Health and Human Services should direct CDC to establish guidelines that ensure full reproducibility of CDC’s research by sharing with the public all permissible and appropriate information needed to reproduce research results, including, but not limited to, model code. (Recommendation 2) We provided a draft of this report to the Department of Health and Human Services (HHS) for review and comment. In its comments, reproduced in appendix IV, HHS agreed with our recommendations and noted that it was developing a process to coordinate its infectious disease modeling efforts across its components. With regard to our second recommendation—that HHS should direct CDC to establish guidelines that ensure the full reproducibility of CDC’s research by sharing all permissible and appropriate information needed to reproduce research results, including, but not limited to, model code— HHS’s comments indicated that CDC believes it has already completed actions to implement this recommendation. For example, the HHS comments state that CDC has established policies such as “Public Access to CDC Funded Publications” and “Policy on Public Health Research and Nonresearch Data Management and Access” that ensure that results are made available to the public, as appropriate. However, as we state in our report, these policies do not contain any reference to reproducibility, models, or provision of model code and therefore do not fully address our recommendation. CDC also said in the HHS comments that its methods—including its practice of providing a copy of model code upon request—are in line with standard practice in the scientific community and peer- reviewed journals. However, in the four instances we identified where CDC modelers did not share code, code being available upon request was only one of the reasons cited. Further, this practice is inconsistent with those of the other HHS agencies we reviewed, and may limit the ability of external researchers to confirm the results of CDC’s research and ultimately produce new knowledge. As noted in our report, by not specifically addressing reproducibility in its policies on access to data and publications, CDC risks undermining the reliability of scientific information disseminated to the public. Therefore, we did not change our recommendation in response to HHS’s comments. We did, however, revise our report to include information on other HHS agency policies related to reproducibility. HHS also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, and to other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you are your staff have questions about this report, please contact Timothy M. Persons, Chief Scientist, at (202) 512-6888 or personst@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V. In conducting our review of infectious disease modeling by the Department of Health and Human Services (HHS) agencies, our objectives were to (1) examine the extent to which HHS has used various types of models to inform policy, planning, and resource allocation for public health decisions for selected infectious diseases, (2) examine the extent to which HHS coordinated their modeling efforts for selected infectious diseases, (3) examine the steps HHS generally took to develop and assess the performance of its models for the selected diseases and steps it applied to a selection of infectious disease models, and (4) describe the extent to which HHS has addressed challenges related to modeling for selected infectious diseases. For purposes of this review, we focused on HHS because of its focus on scientific and technical issues related to disease modeling, role in infectious disease outbreak preparedness and response activities, and use of modeling for policy and regulatory issues related to disease. Within HHS, we identified four agencies—HHS’s Office of the Assistant Secretary for Preparedness and Response (ASPR), the Centers for Disease Control and Prevention (CDC), National Institutes of Health (NIH), and Food and Drug Administration (FDA)—which may develop or use infectious disease models. To inform all four objectives, we selected three naturally-occurring infectious diseases that have pandemic or epidemic potential—Ebola virus disease (Ebola), Zika virus disease (Zika), and pandemic influenza—to use as examples of broader infectious disease modeling efforts. We selected these diseases based on document review, their inclusion on NIH’s pathogen priority list, modeling being conducted by HHS agencies, and interviews with experts that we selected based on their experience with infectious disease. Based on these steps, the team selected diseases that fit into one of the three categories on NIH’s pathogen priority list: the disease (1) can be transmitted easily from person to person, resulted in a high mortality rate and had the potential for major public health impact, might cause social disruption, and may require special action for public health preparedness (Ebola), (2) was moderately easy to disseminate, and required specific enhancements for diagnostic capacity and enhanced disease surveillance (Zika), or (3) was an emerging pathogen that could be engineered for mass dissemination in the future because of availability, ease of production and dissemination, and have the potential for high morbidity and mortality rates and major health impacts (pandemic influenza). To examine the types of models developed by HHS agencies to inform policy, planning, and resource allocation decisions, we reviewed documents from 2009—the year of the last pandemic influenza outbreak in the United States—to April 2019 to identify examples of models developed by the agencies for the three selected diseases. For context on and examples of the types of modeling that CDC and ASPR have conducted, we reviewed published articles that CDC and ASPR officials and experts provided to us or cited during the course of our review, such as articles identified during interviews which we later obtained. We also obtained selected internal memoranda, when available, that described models used in the Ebola virus outbreak. We did not include FDA and NIH in this review because FDA has a limited role in modeling, and NIH generally funds, rather than conducts, modeling. This review yielded articles and memoranda describing about 60 CDC and ASPR models. See appendix II for a bibliography of model publications reviewed. We then categorized the models using categories derived from a federal working group report to characterize the types of modeling conducted and the purpose of the modeling, when that purpose was identified. To analyze each study, one analyst initially coded each study, and each classification was then independently reviewed to verify that it had been correctly classified and to resolve any categorization discrepancies. We used these categories to describe types of modeling efforts undertaken by HHS agencies. Because we focused on studies published between 2009 and 2019, our findings are not generalizable to models that were developed outside of that time period. Additionally, because we relied on agency officials or reviews of relevant agency documents and publications to identify studies, we may not have captured all studies relevant to our scope. Further, because CDC and ASPR modelers and officials said that they do not publish every model they conduct, our review was not intended to develop an inventory of the modeling conducted during the time period. Therefore, we were unable to determine the extent to which the models we identified represented agency modeling efforts as a whole. To describe the extent of model use for public health decision making, we interviewed officials from HHS agencies identified as decision makers for conducting the response to these selected diseases—CDC, ASPR, and FDA—and officials who conducted the modeling. We also interviewed two NIH institutes and one center about funding for research related to modeling for the selected diseases. Additionally, we conducted semi- structured interviews of officials from five states concerning their use of models prepared by HHS agencies for decision making, among other topics. We selected these states based on a review of a CDC draft report on states’ use of CDC models, on the level of influenza activity experienced by states, and consideration of geographic variation by U.S. region. During our review, we sought to identify the common types of decisions that could be informed by models, as well as the considerations that could impact the extent to which a decision maker requests and uses models for specific types of decisions. Based on interviews with agency officials and our review of HHS models we identified examples of models that were used to make specific decisions during response and non- response times. Because we relied on officials to describe the extent to which models inform decision making, we may not have captured all relevant instances when models for the selected infectious diseases informed decision makers. To examine coordination and collaboration across HHS agencies, we reviewed documents describing HHS agencies’ collaboration and coordination mechanisms such as Memoranda of Understanding, descriptions of Emergency Operations Center procedures, and after- action reports following infectious disease outbreaks. We also conducted interviews with and requested information from HHS officials, asking them to provide information on their efforts to coordinate their infectious disease modeling activities. In this report, and in our past work, we define coordination broadly as any joint activity that is intended to produce more public value than could be produced when organizations act alone. We compared these actions to relevant selected collaboration leading practices: define and articulate a common outcome; establish mutually reinforcing or joint strategies; identify and address needs by leveraging resources; agree on roles and responsibilities; establish compatible policies, procedures, and other means to operate across agency boundaries; and develop mechanisms to monitor, evaluate, and report on results. Because we judgmentally selected a group of experts and diseases, the results of our review cannot be generalized to HHS coordination efforts for other infectious diseases. However, our assessment of collaboration and coordination activities did cover modeling efforts for the three selected diseases. To identify steps that are generally considered when modelers develop infectious disease models and assess their performance, we conducted semi-structured interviews with relevant experts from academia and other organizations and CDC and ASPR officials, and reviewed literature identified by experts. We used a snowball sampling approach to identify relevant experts and groups. We initially identified five infectious disease modeling experts through informal conversation with individuals working in the field, infectious disease modeling experts known through GAO work, as well as a review of websites, publications, and grants funded by NIH. Using a snowball sampling approach, we reviewed key literature related to the steps generally taken to develop models and assess their performance, consulted with infectious disease modeling experts, and interviewed agency officials to identify relevant groups, as well as individual experts, who could convey to us the steps generally taken during infectious disease modeling. Through literature searches, the team identified literature from public health journals or other major sources. The team applied personal background and knowledge in public health, infectious disease modeling, and statistics to help identify key sources. For the selected literature, we reviewed references and used a snowball approach to identify further relevant studies. Finally, we reviewed CDC guidance on decision making for data access and long-term preservation as it related to documentation standards. Based on our review of identified literature, we developed a data collection instrument to assess the extent to which CDC and ASPR used the steps for infectious disease model development identified by experts and in the literature. Through this data collection instrument, we gathered information about the elements of developing and assessing model performance and the steps that could be taken within each element. In order to develop the data collection instrument, based on our review of literature, we mapped out steps to develop and assess model performance, and developed broad categories of assessment elements. Within each assessment element, we included steps modelers could take as a part of each assessment element. For example, the data collection instrument included items that recorded model verification steps that might have been taken by modeler(s) within the broader model verification element. The instrument was reviewed by internal stakeholders, who provided feedback on its content. Prior to sending the data collection instrument to the agency, we filled in information on verification steps taken for each of the 10 selected models, based on provided model documentation to reflect steps we determined modelers took as a part of the model development and assessment process. In order to provide officials with this information, two analysts reviewed each model’s documentation, with one analyst providing an initial coding of the model and the other reviewing and verifying the first analyst’s findings. This method was first tested on one of the 10 selected models by two analysts independently coding information from the model’s documentation into the data collection instrument and then reviewing coding choices to reconcile any differences found. We then sent the instruments with filled-in information to CDC and ASPR modelers to receive their feedback concerning the steps taken to develop models and assess their performance, provide any missing information, and resolve any ambiguities. See Appendix III for a list of the 10 selected models reviewed and steps to develop and assess model performance included in the data collection instrument. The data collection instrument was intended to record whether a specific step had been taken, but did not assess the quality of the modeling steps. In order to determine steps CDC and ASPR took to develop and assess its models, we selected a non-generalizable sample of 10 models for review in our data collection instrument that demonstrated steps that HHS agencies took to develop models and assess their performance. The model selection process described above informed our selection of infectious disease models. To be selected for inclusion in our non- generalizable sample, the model had to be (1) developed by CDC, or ASPR officials or contractors; (2) developed to answer a question about Ebola, Zika, or pandemic influenza; and (3) used to inform public health decision makers during an outbreak or for preparedness activities. We selected 10 models that differed in form and answered different types of questions, which included studies prepared during both outbreak preparedness and response times, and covered topics such as the impact of vaccination programs on deaths and hospitalization. For Ebola and Zika, we focused on review of selected papers or memos produced since 2014 in order to capture the time period following the 2014-2016 Ebola and 2015-2016 Zika outbreaks. For pandemic influenza, we focused on papers and memos produced since 2009, when the H1N1 pandemic occurred in the United States. Because we selected from a group of models identified by HHS modelers and officials for Ebola, Zika, and pandemic influenza, the results of our review cannot be generalized to other diseases outside of the scope of this report. Furthermore, we requested models that informed public health decision making, and did not consider models that were not used for this purpose. Because we reviewed a non-generalizable sample of 10 models, the results of our review cannot be generalized to a larger population of models prepared by HHS agencies. To identify challenges associated with modeling for the selected infectious diseases, we reviewed documents and reports to identify modeling challenges and steps to address those challenges, and interviewed agency officials and modelers, and experts identified through the previously-described snowball sampling methodology. We used semi- structured interview protocols that included open-ended questions about challenges associated with infectious disease modeling and limitations associated with model development. Not all officials and experts we interviewed provided comments on every challenge or limitation. In addition, because we judgmentally selected a group of experts and diseases, the results of our review cannot be generalized to all infectious disease modeling efforts. We conducted this performance audit from May 2018 to May 2020, in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Carias, Cristina, et al. “Preventive Malaria Treatment for Contacts of Patients with Ebola Virus Disease in the Context of the West Africa 2014- 15 Ebola Virus Disease Response: An Economic Analysis.” The Lancet Infectious Diseases, vol. 16, no. 4 (April 2016): pp. 449-458. Christie, Athalia, et al. “Possible Sexual Transmission of Ebola Virus— Liberia, 2015.” Morbidity and Mortality Weekly Report, vol. 64, no. 17 (May 8, 2015): pp. 479-481. Martin I. Meltzer, et al. “Estimating the Future Number of Cases in the Ebola Epidemic - Liberia and Sierra Leone, 2014-2015.” Morbidity and Mortality Weekly Report, vol. 63, no. 3 suppl. (September 26, 2014): pp. 1-14. Meltzer, Martin I., et al. “Modeling in Real Time during the Ebola Response.” Morbidity and Mortality Weekly Report, vol. 65, no. 3 suppl. (July 8, 2016): pp. 85-89. Rainisch, Gabriel, et al. “Estimating Ebola Treatment Needs, United States.” Emerging Infectious Diseases, vol. 21, no. 7 (July 2015): pp. 1273-1275. Rainisch, Gabriel, et al. “Regional Spread of Ebola Virus, West Africa, 2014.” Emerging Infectious Diseases, vol. 21, no. 3 (March 2015): pp. 444-447. Undurraga, Eduardo A., Cristina Carias, Martin I. Meltzer, Emily B. Kahn. “Potential for Broad-Scale Transmission of Ebola Virus Disease during the West Africa Crisis: Lessons for the Global Health Security Agenda.” Infectious Diseases of Poverty, vol. 6, no. 159 (2017). Washington, Michael L., Martin I. Meltzer. “Effectiveness of Ebola Treatment Units and Community Care Centers Liberia, September 23- October 31, 2014.” Morbidity and Mortality Weekly Report, vol. 64, no. 3 (January 30, 2015): pp. 67-69. Adamski, Alys, et al. “Estimating the Numbers of Pregnant Women Infected with Zika Virus and Infants with Congenital Microcephaly in Colombia, 2015–2017.” Journal of Infection, vol. 76 (2018): pp. 529-535. Dirlikov, Emilio, et al. “Guillain-Barré Syndrome and Healthcare Needs during Zika Virus Transmission, Puerto Rico, 2016.” Emerging Infectious Diseases, vol. 23, no. 1 (January 2017): pp.134-136. Ellington, Sascha R., et al. “Estimating the Number of Pregnant Women Infected With Zika Virus and Expected Infants With Microcephaly Following the Zika Virus Outbreak in Puerto Rico, 2016.” JAMA Pediatrics, vol. 170, no. 10 (2016): pp. 940-945. Grills, Ardath, et al. “Projected Zika Virus Importation and Subsequent Ongoing Transmission after Travel to the 2016 Olympic and Paralympic Games—Country-Specific Assessment, July 2016.” Morbidity and Mortality Weekly Report, vol. 65, no. 28 (July 22, 2016): pp.711-715. Johansson, Michael A., et al. “Zika and the Risk of Microcephaly.” The New England Journal of Medicine, vol. 375 (July 7, 2016): pp.1-4. Johnson, Tammi L., et al. “Modeling the Environmental Suitability for Aedes (Stegomyia) aegypti and Aedes (Stegomyia) albopictus (Diptera: Culicidae) in the Contiguous United States.” Journal of Medical Entomology, vol. 54, no. 6 (November 7, 2017): pp. 1605-1614. Mitchell, Patrick K. et al., “Reassessing Serosurvey-Based Estimates of the Symptomatic Proportion of Zika Virus Infections.” American Journal of Epidemiology, vol. 188, no. 1 (January 2019): pp. 206-213. Mier-y-Teran-Romero, Luis, Mark J. Delorey, James J. Sejvar, Michael A. Johansson. “Guillain-Barré Syndrome Risk Among Individuals Infected with Zika Virus: a Multi-Country Assessment.” BMC Medicine, vol. 16, no. 67 (2018). Mier-y-Teran-Romero, Luis, Andrew J. Tatem, Michael A. Johansson. “Mosquitoes on a Plane: Disinsection Will Not Stop the Spread of Vector- Borne Pathogens, a Simulation Study.” PLoS Neglected Tropical Diseases, vol. 11, no. 7 (July 3, 2017). Reefhuis, Jennita, et al. “Projecting Month of Birth for At-Risk Infants after Zika Virus Disease Outbreaks.” Emerging Infectious Diseases, vol. 22, no. 5 (May 2016): pp. 828-832. Russell, Steven, et al. “Detecting Local Zika Virus Transmission in the Continental United States: A Comparison of Surveillance Strategies.” PLoS Currents Outbreaks (November 22, 2017). Watts. Alexander G., et al. “Elevation as a Proxy for Mosquito-Borne Zika Virus Transmission in the Americas.” PLoS ONE, vol. 12, no. 5 (May 24, 2017). Atkins, Charisma Y., et al. “Estimating Effect of Antiviral Drug Use during Pandemic (H1N1) 2009 Outbreak, United States.” Emerging Infectious Diseases, vol. 17. no. 9 (September 2011): pp. 1591-1598. Biggerstaff, Matthew, et al. “Estimates of the Number of Human Infections With Influenza A(H3N2) Variant Virus, United States, August 2011–April 2012.” Clinical Infectious Diseases, vol. 57, suppl. 1 (2013): pp. S12-S15. Biggerstaff, Matthew, et al. “Estimating the Potential Effects of a Vaccine Program Against an Emerging Influenza Pandemic—United States.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S20-S29. Carias, Cristina, et al. “Potential Demand for Respirators and Surgical Masks during a Hypothetical Influenza Pandemic in the United States.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S42-S51. Cauchemez, Simon, et al. “Role of Social Networks in Shaping Disease Transmission during a Community Outbreak of 2009 H1N1 Pandemic Influenza.” Proceedings of the National Academy of Sciences of the United States, vol. 108, no. 7 (February 15, 2011): pp. 2825-2830. Dawood, Fatimah S., et al. “Estimated Global Mortality Associated with the First 12 Months of 2009 Pandemic Influenza A H1N1 Virus Circulation: a Modelling Study.” The Lancet Infectious Diseases, vol. 12 (September 2012): pp. 687-695. Fung, Isaac Chun-Hai, et al. “Modeling the Effect of School Closures in a Pandemic Scenario: Exploring Two Different Contact Matrices.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S58-S63. Iuliano, A. Danielle, et al. “Estimates of Global Seasonal Influenza- Associated Respiratory Mortality: a Modelling Study.” The Lancet, vol. 391, no. 10127 (March 31, 2018): pp. 1285-1300. Jain, Seema, et al. “Hospitalized Patients with 2009 H1N1 Influenza in the United States, April–June 2009.” The New England Journal of Medicine, vol. 361, no. 20 (November 12, 2009): pp. 1935-1944. Kostova, Deliana, et al. “Influenza Illness and Hospitalizations Averted by Influenza Vaccination in the United States, 2005–2011.” PLoS ONE, vol. 8, no. 6 (June 19, 2013). Lafond, Kathryn E., et al. “Global Role and Burden of Influenza in Pediatric Respiratory Hospitalizations, 1982–2012: A Systematic Analysis.” PLoS Medicine, vol. 13, no. 3 (March 24, 2016). Meltzer, Martin I., Nancy J. Cox, Keiji Fukuda. “The Economic Impact of Pandemic Influenza in the United States: Priorities for Intervention.” Emerging Infectious Diseases, vol. 5, no. 5 (September-October 1999): pp. 659-671. Meltzer, Martin I., et al. “Estimates of the Demand for Mechanical Ventilation in the United States during an Influenza Pandemic.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S52-S57. O’Hagan, Justin J., et al. “Estimating the United States Demand for Influenza Antivirals and the Effect on Severe Influenza Disease during a Potential Pandemic.” Clinical Infectious Diseases, vol. 60, suppl. 1 (2015): pp. S30-S41. Presanis, Anne M., et al. “The Severity of Pandemic H1N1 Influenza in the United States, from April to July 2009: A Bayesian Analysis.” PLoS Medicine, vol. 6, no. 12 (December 8, 2009). Reed, Carrie, et al. “Estimates of the Prevalence of Pandemic (H1N1) 2009, United States, April-July 2009.” Emerging Infectious Diseases, vol. 15, no. 12 (December 2009): pp. 2004-2007. Reed, Carrie, Martin I. Meltzer, Lyn Finelli, Anthony Fiore. “Public Health Impact of Including Two Lineages of Influenza B in a Quadrivalent Seasonal Influenza Vaccine.” Vaccine, vol. 30 (2012): pp. 1993-1998. Reed, Carrie, et al. “Estimating Influenza Disease Burden from Population-Based Surveillance Data in the United States.” PLoS ONE, vol. 10, no. 3 (March 4, 2015). Rolfes, Melissa A., et al. “Annual Estimates of the Burden of Seasonal Influenza in the United States: A Tool for Strengthening Influenza Surveillance and Preparedness.” Influenza and Other Respiratory Viruses, vol. 12 (2018): pp. 132-137. Russell, K., et al. “Utility of State-Level Influenza Disease Burden and Severity Estimates to Investigate an Apparent Increase in Reported Severe Cases of Influenza A(H1N1) pdm09 – Arizona, 2015–2016.” Epidemiology and Infection, vol. 146 (June 14, 2018): pp. 1359-1365. Shrestha, Sundar S., et al. “Estimating the Burden of 2009 Pandemic Influenza A (H1N1) in the United States (April 2009–April 2010).” Clinical Infectious Diseases, vol. 52, suppl. 1 (2011): pp. S75-S82. Tokars, Jerome I., Melissa A. Rolfes, Ivo M. Foppa, Carrie Reed. “An Evaluation and Update of Methods for Estimating the Number of Influenza Cases Averted by Vaccination in the United States.” Vaccine, vol. 36 (2018): pp. 7331-7337. Appendix III: Ten Selected Infectious Disease Models and Questions from Data Collection Instrument Document describing model Meltzer, Martin I., Charisma Y. Atkins, Scott Santibanez, Barbara Knust, Brett W. Petersen, Elizabeth D. Ervin, Stuart T. Nichol, Inger K. Damon, Michael L. Washington. Estimating the Future Number of Cases in the Ebola Epidemic–Liberia and Sierra Leone, 2014-2015, MMWR. Volume 63, Number 3, September 26, 2014. Rainisch, Gabriel, Manjunath Shankar, Michael Wellman, Toby Merlin, and Martin I. Meltzer. Regional Spread of Ebola Virus, West Africa, 2014. Emerging Infectious Diseases. Volume 21, Number 3, March 2015. Asher, Jason. Forecasting Ebola with a Regression Transmission Model. Epidemics. Volume 22, 2018. Ellington, Sascha R., Owen Devine, Jeanne Bertolli, Alma Martinez Quiñones, Carrie K. Shapiro-Mendoza, Janice Perez-Padilla, Brenda Rivera-Garcia, Regina M. Simeone, Denise J. Jamieson, Miguel Valencia-Prado, Suzanne M. Gilboa, Margaret A. Honein, Michael A. Johansson. Estimating the Number of Pregnant Women Infected With Zika Virus and Expected Infants With Microcephaly Following the Zika Virus Outbreak in Puerto Rico, 2016. JAMA Pediatrics. Volume 170, Number 10, October 2016. Johansson, Michael A., Luis Mier-y‐Teran-Romero, Jennita Reefhuis, Suzanne M. Gilboa, and Susan L. Hills. Zika and the Risk of Microcephaly. New England Journal of Medicine. Volume 375, Number 1, July 7, 2016. Dirlikov, Emilio, Krista Kniss, Chelsea Major, Dana Thomas, Cesar A. Virgen, Marrielle Mayshack, Jason Asher, Luis Mier-y-Teran-Romero, Jorge L. Salinas, Daniel M. Pastula, Tyler M. Sharp, James Sejvar, Michael A. Johansson, Brenda Rivera-Garcia. Guillain-Barré Syndrome and Healthcare Needs during Zika Virus Transmission, Puerto Rico, 2016. Emerging Infectious Diseases. Volume 23, Number 1, January 2017. Biggerstaff, Matthew, Carrie Reed, David L. Swerdlow, Manoj Gambhir, Samuel Graitcer, Lyn Finelli, Rebekah H. Borse, Sonja A. Rasmussen, Martin I. Meltzer, Carolyn B. Bridges. Estimating the Potential Effects of a Vaccine Program against an Emerging Influenza Pandemic—United States, Clinical Infectious Diseases. Volume 60, Issue Supplement 1, 2015. Carias, Cristina, Gabriel Rainisch, Manjunath Shankar, Bishwa B. Adhikari, David L. Swerdlow, William A. Bower, Satish K. Pillai, Martin I. Meltzer, Lisa M. Koonin. Potential Demand for Respirators and Surgical Masks during a Hypothetical Influenza Pandemic in the United States. Clinical Infectious Disease. Volume 60, Issue Supplement 1, 2015. Reed, Carrie, Frederick J. Angulo, David L. Swerdlow, Marc Lipsitch, Martin I. Meltzer, Daniel Jernigan, and Lyn Finelli. Estimates of the Prevalence of Pandemic (H1N1) 2009, United States, April–July 2009, Emerging Infectious Diseases. Volume 15, Number 12, December 2009. Asher, Jason, Matthew Clay. Deterministic compartmental models for influenza with mitigations. R: “flumodels” package. Version: 1.0.7, April 24, 2017. Purpose: The Government Accountability Office has been asked by the Congress to review the Department of Health and Human Services’ agency efforts to model infectious disease. As part of our methodology, we selected and reviewed published papers and internal memoranda from the sources provided to us. We reviewed these sources to describe the steps taken to describe, verify, validate, and communicate results of these modeling efforts. The purpose of this inquiry is to provide the authors of the selected papers the opportunity to confirm, clarify, or provide additional information in the table below. Instructions: In the table below, we have two sets of columns: one set indicating GAO’s assessment of whether the document contained information about a step being taken. The second set of columns is for the authors of the selected paper to fill out. If you agree with information in the GAO columns, please indicate your concurrence in the Reviewer Comments column. Otherwise, please provide information accordingly. If a step is marked “Step taken” please review the entries we have made in the GAO Reviewer Comments column for accuracy and completeness and indicate your concurrence in the Reviewer Comments column. Please also provide additional supporting documentation if available. For any steps that were taken, but where we indicated either “not taken” or “not enough information to determine” in our review, please provide a description of the actual steps and any documentation you may have. If a step was not taken, please provide an indication as to why that step was not taken and, if possible, please provide supporting documentation. For example, if limited data availability impacted the ability to conduct a model validation step(s), then please include this information in the appropriate table cells. 10 Independent expert (internal or external) review of key programming 11 Debugging tests and checks for coding accuracy 12 Model’s code or Excel spreadsheet is available 13 Test model assumptions (i.e. confirming model assumptions are reasonable and appropriate for question), for example: Distributional assumptions about model residuals Form of the model 14 Model handling of input data/parameters is verified as correct (i.e. as intended by developers) 16 Sensitivity analysis (assessing impact of assumption/parameter uncertainty on output or model form) 17 Cross validation or between model comparisons: Compare results to other models that address the same problem 18 External validation: Compare model results to actual event data 19 Predictive validation: Compare model predictions for future events to actual outcomes. 21 Modelers supply customer with agreed upon information, which may vary depending on the model 22 Modeler provides customer with clear information on uncertainty in model results, such as inclusion of standard errors or confidence intervals, or qualitative explanations of uncertainty in the model results Assessment Steps Question: Do you think that the assessment elements identified in the table above sufficiently reflect the steps that should generally be taken to develop and assess the performance of models? Would you remove any steps, add any steps, or make any other adjustments to these steps in order to consider them best practices in assessing performance of models, generally? Please explain. In addition to the contact named above, the following individuals made contributions to this report: Sushil Sharma (Assistant Director), Charlotte E. Hinkle (Analyst-in-Charge), Sam Amrhein, Breanne Cave, Jehan Chase, Carol A. Gotway Crawford, Justin Cubilo, Karen Doran, Nancy Fasciano, Douglas G. Hunker, Dennis Mayo, Anika McMillon, Sarah Resavy, Edward Rice, Ben Shouse, Amber Sinclair, Walter Vance, Sarah Veale, and Richard Zarrella.", "summary": "Outbreaks of infectious diseases—such as Ebola, Zika, and pandemic influenza—have raised concerns from Congress about how federal agencies use modeling to, among other things, predict disease distribution and potential impacts. In general, a model is a representation of reality expressed through mathematical or logical relationships. Models of infectious diseases can help decision makers set policies for disease control and may help to allocate resources. GAO was asked to review federal modeling for selected infectious diseases. This report examines (1) the extent to which HHS used models to inform policy, planning, and resource allocation for public health decisions; (2) the extent to which HHS coordinated modeling efforts; (3) steps HHS generally takes to assess model development and performance; and (4) the extent to which HHS has addressed challenges related to modeling. GAO reviewed documents and interviewed HHS officials, state officials, and subject matter experts. GAO identified practices commonly used to assess infectious disease model performance and reviewed 10 selected modeling efforts to see if they followed these practices. Within the Department of Health and Human Services (HHS), the Centers for Disease Control and Prevention (CDC) and the Office of the Assistant Secretary for Preparedness and Response (ASPR) used models to inform decision-making during and after outbreaks of Ebola, Zika, and pandemic influenza. These agencies' modeling efforts informed public health planning, outbreak response, and, to a limited extent, resource allocation. Four CDC centers perform modeling. HHS agencies reported using multiple mechanisms to coordinate modeling efforts across agencies, but they do not routinely monitor, evaluate, or report on the extent and success of coordination. Consequently, they risk missing opportunities to identify and address modeling challenges—such as communicating clearly, and obtaining adequate data and resources—before and during an outbreak. As a result, agencies may be limiting their ability to identify improvements in those and other areas. Further, there is potential for overlap and duplication of cross-agency modeling efforts, which could lead to inefficiencies. CDC and ASPR generally developed and assessed their models in accordance with four steps GAO identified as commonly-recognized modeling practices: (1) communication between modeler and decision maker, (2) model description, (3) verification, and (4) validation. However, for four of the 10 models reviewed, CDC did not provide all details needed to reproduce model results, a key step that lets other scientists confirm those results. GAO found that CDC's guidelines and policy do not address reproducibility of models or their code. This is inconsistent with HHS guidelines and may jeopardize the reliability of CDC's research. This report also identifies several modeling-related challenges, along with steps agencies have taken to address them. GAO recommends that HHS (1) develop a way to routinely monitor, evaluate, and report on modeling coordination efforts across multiple agencies and (2) direct CDC to establish guidelines to ensure full reproducibility of its models. HHS agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "U.S. international trade agreements that cover USG procurement include the GPA and bilateral and regional FTAs. The revised GPA has 20 parties (including the EU) covering 48 WTO member countries (including the 28 EU member countries). Another 33 WTO members are observers; of these, 10 are in the process of acceding to the agreement. In addition to the GPA, the United States has 14 FTAs with 20 countries, four of which (Canada, Israel, Singapore, and South Korea) are also parties to the GPA. Almost all of the FTAs to which the United States is a party include provisions covering government procurement. The GPA aims to mutually open government procurement markets for goods, services, and construction services among its parties, according to the WTO. Under the GPA, foreign suppliers are able to compete alongside with U.S. suppliers for USG contracts covered by the agreement, and U.S. suppliers are able to compete for covered foreign government contracts in accordance with the framework established by the GPA. According to the office of the United States Trade Representative (USTR), to implement U.S. obligations under the international agreements that cover government procurement, the United States—generally (and not always) — waives preferential purchasing requirements for goods and suppliers from other countries that are parties to the agreements in covered procurements over a certain threshold. For example, USTR has waived the Buy American Act and other preferential provisions for eligible products in acquisitions covered by various trade agreements. However, Commerce officials noted that small business set-aside requirements are not waived nor are the provisions of the Berry Amendment. As part of our body of work on international government procurement, we have previously reported the following: The U.S. and EU government procurement markets are comparable in size, and each is larger than those of all other GPA and U.S. FTA partner countries combined. Some other parties to the agreements also have large government procurement markets, including Japan, South Korea, Canada, Mexico, and Norway. The government procurement chapters of the GPA and selected U.S. FTAs that we reviewed generally have similarities in text and commitments, possibly because key parties negotiated multiple agreements concurrently. However, the revised GPA generally provides more comprehensive market access than the selected FTAs we reviewed. The United States reported opening more procurement opportunities covered by the GPA to foreign firms than had other parties to the agreement. Data for 2010 showed that the United States reported $837 billion in GPA covered procurement. This amount is about twice as large as the approximately $381 billion reported by the next five largest GPA parties combined—the EU, Japan, South Korea, Norway, and Canada—even though total U.S. procurement is less than that of the other five parties combined. Previously, we reported on the opportunities available to U.S. and foreign firms seeking to compete for covered government procurement contracts in the countries that are parties to the agreements. In the current report, we analyze the value and number of actual contract awards, reported in procurement databases, including contracts covered under the GPA and NAFTA and those not covered. Covered contracts can be awarded to domestic firms, to firms from countries that are parties to the GPA and U.S. FTAs, or to other non-U.S. firms. Additionally, the Buy American Act does not apply to products that are purchased for use outside the United States, nor to the acquisition of services. Therefore, such contracts can be awarded without the application of Buy American Act domestic preference conditions to bids from any firm, including firms from non-GPA and non-FTA countries. To estimate foreign source procurement, we looked for information about where the goods and services that governments purchase are produced and the characteristics of the firms supplying those goods and services. We identified two types of primary data sources that could be analyzed to estimate foreign sourcing in government procurement: (1) government procurement databases to estimate direct cross-border central government procurement and (2) input-output tables merged with international trade data to estimate total procurement by all levels of government and the portion comprising imported goods and services. Government procurement databases collect information on contracts awarded by government entities to firms supplying goods and services. Except for Japan, all the countries in our analysis maintain online government procurement databases that can serve as a primary data source to generate statistics on their foreign source central government procurement. The USG and the other six main parties to the GPA and NAFTA use these databases to report to the WTO their required procurement statistics under the GPA. While Japan does not have a government procurement database, Japan’s central government collects procurement data from various ministry sources and reports the aggregated data to the WTO. As table 1 shows, the U.S. Federal Procurement Data System-Next Generation (FPDS-NG) provides more data fields that can be used as proxies for measuring foreign source procurement than the non-U.S. databases provide. FPDS-NG contains data on four potential proxy measures of foreign sourcing—firm location, firm ownership, product and service origin, and place of performance. The database for the EU and Norway and the databases for Canada and Mexico all contain contract award data related to firm location. South Korea’s database and Japan’s WTO submission on its 2013 procurement contain data on source country of goods and services. Therefore, two data fields, one reflecting firm location and the other reflecting country of product and service origin, appear to provide reasonable proxy measures of foreign source procurement, although neither is available across all data sources. (For more information on the characteristics of each government procurement database, see app. II.) Information about how much goods and services a country imports provides the basis for another approach to estimating what portion of all government procurement in that country is imported. The WIOD provides such information, giving us a second type of data and an alternative analytical approach for estimating foreign source government procurement. The WIOD links data on an economy’s supply chain interdependencies to data on its import and export flows, thus providing a proxy estimate of the share of imports in procurement by all levels of government. We based our method for analyzing linked input-output tables on an approach used by the European Commission which examines import penetration of government procurement within Europe. Unlike the contract data we analyzed from government procurement databases, the WIOD data capture procurement by all levels of government. However, the input-output tables are organized by industry, which requires a decision as to which industries make up the government sector in any given country’s economy. Some industries, like “public administration”, can safely be assumed to be part of the governmental sector in every country. Other industries, like education or health care, vary across countries in the degree to which they are part of the government sector, if at all. While this analytical method based on input-output tables can provide broad estimates of how much governments are purchasing imported goods and services, it relies on some important assumptions that may affect the reliability of the results. For example, it assumes that the goods and services purchased by all levels of government are imported to the same extent as they are when purchased by other industries in the same country. This assumption, known as the “proportionality assumption”, recognizes that results from this method may overestimate the share of imports in government procurement to the extent that the analysis does not capture attempts by the government sector to limit foreign sourcing in its procurement. On the other hand, other aspects of this method may underestimate the share of imports in government procurement. For example, the input-output data include intermediate inputs but do not include purchases for investment, such as some government assets because, according to the authors of the European Commission study, input-output tables do not have the data to distinguish between investments by the private and public sectors. Thus, the input-output data could exclude investment made through construction services like those purchased to build highways, schools, or other assets that have long-term use, services that are included in covered procurement under both the GPA and NAFTA. The value of U.S. government (USG) contracts awarded to firms located in the other six main parties to the GPA and NAFTA likely exceeds twice the estimated value of contracts from those parties to U.S. firms, but exact comparisons are not possible. The USG awarded contracts valued at about $12 billion to foreign-located firms in fiscal year 2015, of which less than half went to firms located in the other six main parties. Conversely, the government procurement data we analyzed indicated the central governments of these parties awarded almost $7 billion to foreign sources, of which less than a third were awarded to firms located in the United States or for goods or services from the United States. Over three- quarters of these U.S.-sourced contracts were awarded by Canada and Mexico. Only the USG’s procurement database contains data on firm ownership. Analyzing these data, we found that the USG awarded more, by reported contract value, to foreign-owned firms located abroad, than it awarded to U.S.-based subsidiaries of foreign-owned firms. This was mostly U.S. Department of Defense (DOD) contracts in support of the U.S. military presence in those countries. Overall, while available contract data enable broad cross-country comparisons, these data allow only limited insight into the effects on the U.S. economy of foreign sourcing of USG procurement. This is principally because the contract data do not capture the economic roles of firms awarded contracts and thus do not allow for a definitive assessment of the economic implications of foreign sourcing, as we discuss later in this report. In 2015 the USG awarded about 511,000 contracts valued at about $290.9 billion. Out of this total, about 47,000 contracts valued at about $12.1 billion were awarded to firms located outside the United States (as shown in the data by firm location). Similarly, the USG awarded about 50,000 contracts valued at about $16.5 billion for foreign goods and services (as shown by country of product and service origin). See table 2. Of the USG foreign source procurement awarded to firms in the other six main parties to the GPA and NAFTA, firms located in the EU received more than half in terms of contract value and slightly less than half by number. In 2015 the USG awarded about 10,000 contracts valued at about $5.3 billion to firms located in the other six main parties to the GPA and NAFTA (see table 2 above). This $5.3 billion is about 40 percent of the total value of USG contracts awarded to foreign-located firms. Firms located in the EU received almost 5,000 USG contracts valued at $2.8 billion. Firms located in Japan, South Korea, and Canada were awarded most of the remaining aggregate USG contract value ($1.1, $0.8, and $0.6 billion, respectively) and number of contracts (about 1,500, 600, and 2,900, respectively) awarded to firms in the other six main parties to the GPA and NAFTA. Firms located in Mexico and Norway received less than 1 percent of the aggregate USG contract value and number of contracts awarded to firms in the other six main parties. However, as table 2 also shows, the majority of foreign-sourced USG procurement, in terms of both value and number of contracts, went to firms located in countries that are not among the other six main parties to the GPA and NAFTA. Germany, Japan, and South Korea are among the top five countries whose firms received the most USG contract value in fiscal year 2015. However, countries in the Middle East, including Afghanistan, United Arab Emirates, and Saudi Arabia, were also among the countries whose firms were main recipients of USG procurement in terms of aggregate contract value (see app. III for additional information on USG foreign source procurement by country). Finally, table 2 shows that FPDS-NG data are similar when we use, instead of firm location, the alternative measure of foreign sourcing based on country of product and service origin. For example, the aggregate value of contracts awarded by the USG for goods and services originating in countries of the other six main parties was about 43 percent of the overall value of USG foreign source procurement—the same proportion we found when using firm location as proxy measure of foreign sourcing. In addition, as with the results based on firm location, most of the USG’s foreign source procurement as measured by country of product and service origin went to countries outside the other six main parties to international procurement agreements. Foreign-located firms can be either foreign-owned or U.S.-owned, just as U.S.-located firms can be either foreign-owned or U.S.-owned. Among the government procurement databases we used, only the FPDS-NG includes data on firm ownership. Some research on foreign sourcing in government procurement differentiates between direct and indirect cross- border procurement based on knowledge about both the location and ownership of the successful bidder: In direct cross-border procurement, the successful bidder is both foreign-owned and foreign-located. In indirect cross-border procurement, the successful bidder is a U.S.- based domestic subsidiary of a foreign-owned firm. According to a recent EU Commission study, between 2009 and 2015, the EU’s indirect cross-border government procurement was more than 5 times greater in terms of both value and number of contract awards than its direct cross-border government procurement. The study notes that indirect cross-border procurement is often high when direct cross-border procurement is low and suggests that may reflect actual or perceived barriers to cross-border bidding, which lead firms to rely on their locally based subsidiaries for cross-border sales. The study reported that indirect cross-border government procurement (foreign-owned, domestically located vendor) accounted for 21.9 percent of the number and 20.4 percent of the value of certain contract awards in the EU’s 28 countries, while direct cross-border government procurement (foreign-owned, foreign-located vendor) accounted for 1.7 percent of the number of contracts and 3 percent of contract value. In contrast to the findings of that EU Commission study, our analysis of FPDS-NG data shows that indirect cross-border procurement by the USG was smaller in terms of total award value and number of contracts than direct cross-border procurement. This indicates that foreign firms selling to the USG generally do not establish a local presence in the United States. Specifically, foreign-owned firms located in the United States (indirect cross-border procurement) received contracts valued at about $3.6 billion, or less than 1 percent of the value of all USG contracts. By contrast, firms that were both foreign-owned and foreign-located (direct cross-border procurement) received contracts valued at about $11.8 billion, or about 4 percent of the value of all USG contracts ($290.9 billion). Therefore, USG direct cross-border procurement was about three times greater than indirect cross border procurement for contracts awarded in fiscal year 2015. A possible explanation for this finding could be that foreign-owned and foreign-located firms are awarded more USG contracts in terms of value and number than U.S. subsidiaries of foreign-owned firms because those contracts are covered by international procurement agreements. Foreign- owned and foreign-located firms are awarded more USG contracts because they may bid for large-value GPA covered USG contracts at a higher rate than their U.S.-located counterparts, or they may generally be more competitive for such contracts. However, for contracts not covered under the GPA and NAFTA, the relative difference between the two groups of foreign-owned firms becomes smaller in terms of aggregate contract value. Therefore, the difference between direct and indirect cross-border procurement is likely not due to agreement coverage as one might expect. To better understand why the USG’s direct cross-border procurement was larger than its indirect cross-border procurement, we further analyzed the FPDS-NG data on firm location, firm ownership, and place of performance—where the services were performed or where the goods were produced. Based on firm location, as stated earlier, foreign-located firms were awarded about $12.1 billion in USG contracts. Measured by aggregate contract value, almost all of the USG contracts awarded to those firms were performed abroad (i.e., outside the United States)—$11.9 out of $12.1 billion or 98 percent. USG contracts performed abroad are commonly awarded to U.S.-located as well as to foreign-located firms. In 2015, the USG awarded contracts performed abroad valued at about $23.3 billion, of which about half was awarded to U.S.-located firms. In particular, as figure 1 suggests, while U.S.-located firms received contracts performed abroad valued at $11.4 billion, foreign-located firms were awarded USG contracts valued at $11.9 billion. Almost all of those USG contracts—$11.7 out of $11.9 billion or 98 percent—were awarded to firms that were foreign-owned as well as foreign-located (i.e., direct cross-border government procurement). The vast majority of the value of these USG contracts to foreign-owned, foreign-located firms was for DOD contracts performed abroad. In particular, DOD awarded about 84 percent of the value of USG contracts—$9.8 billion out of $11.7 billion— that were performed abroad and awarded to foreign-owned, foreign-located firms. The vast majority of those contracts ($7.5 billion or 77 percent) were covered under the GPA and NAFTA. (See app. III for a breakdown by agency of all USG contracts performed abroad and awarded in fiscal year 2015 to foreign-owned, foreign-located firms.) Foreign-owned firms located in six countries received the majority (57 percent) of DOD’s $9.8 billion in aggregate award value of contracts performed abroad. Specifically, firms located in three countries in the Middle East—Afghanistan, Saudi Arabia, and United Arab Emirates— together received 28 percent of that award value; firms in Japan and South Korea together received 18 percent; and firms in Germany received 11 percent. About a quarter of DOD’s $9.8 billion in aggregate award value were for purchases of fuel, oil, lubricant, and wax. About 9 percent were for education and training services, and about 7 to 8 percent each were for construction of buildings and housekeeping services. For example, fuel was the main product procured by DOD in United Arab Emirates, while in Saudi Arabia most DOD procurement was for education and training services. (See app. III for a breakdown of DOD contracts performed abroad and awarded to foreign-owned and foreign- located firms, by country.) Our analysis of available procurement contract data from 2015 shows that the central governments of the other six main parties to the GPA and NAFTA, apart from the USG, awarded contracts valued at about $170.5 billion. About 4,000 out of a total of 245,000 of these contracts with an estimated total value of about $6.5 billion were awarded to foreign sources, that is, to foreign-located firms or for imported products and services. Some of these contracts awarded by the other six main parties were covered by the GPA and NAFTA, while others were not. Furthermore, the central governments of the other six main parties awarded about 2,000 U.S.-sourced contracts worth about $1.8 billion (see fig. 2). U.S.-sourced contracts are contracts awarded to U.S.-located firms or for products made in the United States. Canada and Mexico awarded most of the U.S.-sourced contracts. Specifically, central government contracts awarded to U.S.-located firms by Canada and Mexico accounted for almost 80 percent of the value and number of all U.S.-sourced contracts. Over 60 percent of the value and number of U.S.-sourced contracts awarded by the central governments of the other six main parties were for the procurement of goods. In particular, Canada awarded more than 20 times more in contract value to purchase goods than it did to purchase services from U.S.-located firms. However, for contracts covered under trade agreements, the other six main parties collectively awarded more U.S.-sourced contracts for services than for goods; these contracts were awarded primarily by the EU and Mexico. U.S.-located firms were awarded virtually no construction services contracts. This result is consistent with our findings for procurement flows among all countries among the other six main parties to GPA and NAFTA and may be explained by the proxy measure used—firm location, which accounts only for direct cross-border procurement. For example, the EU commission paper cited previously finds that for construction works the share of direct cross-border procurement in the total value of awards was 1.7 percent compared with 12.3 percent for indirect cross-border procurement. The data available from the government procurement databases we analyzed provide relevant and useful information for assessing foreign sourcing in government procurement, but these data do not allow for precise cross-country comparisons based on the GPA provisions on rules of origin. Data and reporting on country of origin for goods and services is limited for a number of reasons. Most of the databases we analyzed contain fields on contract award value and type of contract, as well as fields on firm location or country of product or service origin—proxy measures of foreign sourcing that, as we have found, allow for broad cross-country comparisons. However, precise estimates from the available data are not possible because no single internationally accepted definition exists to distinguish procured goods and services that are “foreign” from those that are “domestic” and the information in government procurement databases is not uniform. There is no agreed- upon definition of the country of origin for goods and services for statistical reporting purposes in the GPA even though a similar term— country of production—is used in the 1994 GPA’s general principles on nondiscrimination. Instead, the GPA generally expresses that a party shall apply the rules of origin that it applies in the normal course of trade when determining the country of origin for goods and services in covered procurement. Another factor that limits cross-country comparisons of country of origin data by parties to the GPA is the recent revision to the GPA itself, which no longer requires the parties to provide country of origin statistics, as we previously reported. According to the 1994 GPA, parties were to provide statistics on the country of origin for products and services purchased by its entities, to the extent that such information is available. However, the revised GPA, which went into effect in 2014, does not require parties to report available information on the country of origin of purchased products or services. While all the GPA members included in our scope reported the amount of covered procurement to the WTO, only Japan (until 2013) reported statistics on the “nationality of the winning tenderer”. The WTO Committee on Government Procurement’s Work Programme on the Collection and Reporting of Statistical Data is currently examining the issues surrounding how countries define country of origin for the procurement of goods and services. Finally, while the United States collects a variety of relevant data on foreign sourcing, those data have certain limitations for cross-country comparisons since the data are collected for different purposes. While U.S. agencies collect country data on successful bidders and the country of origin of goods and services in response to the Buy American Act and report these in FPDS-NG, the agencies do not collect data on country of origin determinations in response to relevant provisions of the GPA or NAFTA. For example, the U.S. Federal Acquisitions Regulation (FAR), in implementing statutes including the Buy American Act, applies different tests to determine the country of origin of an end product and defines end products to include “domestic”, “foreign”, or “U.S.-made”. The test to determine country of origin for an end product under the Buy American Act is different from the test to determine country of origin in the procurement of an end product under trade agreements. According to the FAR, for manufactured products, the Buy American Act uses a two-part test to define a domestic end product: (1) the article must be manufactured in the United States, and (2) the cost of domestic components must exceed 50 percent of the cost of all the components. According to the FAR, for procurement under trade agreements, the test to determine “country of origin” is “substantial transformation”, (i.e., transforming an article into a new and different article of commerce, with a name, character, or use distinct from the original article). The substantial transformation test can also be used to determine whether a product is a U.S.-made end product. The FAR also defines a foreign end product as an end product other than a domestic end product. Therefore, under the FAR, contracting officers use different tests and different descriptors to designate country of origin. Since corresponding data fields for these descriptors are not available in FPDS-NG, the data do not allow for exact cross-country comparisons of foreign sourcing under the GPA and NAFTA. In all countries included in this report, available contract data do not allow for a definitive assessment of the economic implications of foreign sourcing in government procurement, such as impacts on wages and profits. As figure 3 shows, using the United States for illustrative purposes, foreign versus domestic sourcing in government procurement could be viewed in four different ways —firm location, firm ownership, product and service origin, and place of contract performance. For example, FPDS-NG data shows that a task order under a DOD contract for facilities support performed in Iraq reports the United Arab Emirates as the country of product and service origin for safety and rescue equipment, while also reporting the firm location and ownership as the United States. FPDS-NG data showed that another task order under the same contract, for housekeeping services, reports the place of performance as Kuwait but reports the United States as the country of product and service origin, the firm location, and the country of firm ownership. As another example from FPDS-NG data, a contract awarded by the U.S. Agency for International Development for internet services performed in Malawi and awarded to a foreign-owned business reports the United States as the country of service origin but the United Kingdom as the firm location. Each of the various different ways relevant to the sourcing of USG contracts can be viewed on a continuum based on the extent of foreign involvement associated with the production and service delivery processes. Country of firm location. As found in the procurement databases, suppliers can be located, for example, domestically in the United States or abroad. However, the economic effects related to the country of firm location depend on what is produced in the country relative to what is produced elsewhere. For example, the supplier may be an end product manufacturer doing less skill-intensive assembly and packaging, a high technology and skill-intensive manufacturing firm that substantially transforms a product that is subsequently used as an input in the production process, or a broker providing unskilled labor for product distribution. In each of these examples, the country of firm location could experience different economic effects from the awarded contract. Country of firm ownership. Suppliers could be domestically or foreign owned, and who owns the firm determines who accrues the firm’s profits. However, determining ownership is challenging because a supplier awarded a contract may have various ownership structures. For example, the supplier may be a sole proprietor or a corporation with shareholdings, subsidiaries, ultimate owners, or may be a participant in a corporate group. The supplier may have established a presence in the United States through a foreign-owned subsidiary or may participate in a partnership such as a joint venture with a U.S. firm. Country of product or service origin. Goods and services purchased under government procurement contracts may be domestically produced or imported. In this case, the effects can be analyzed in the same way as trade flows in general. However, the country of product or service origin is more challenging to determine for government procurement contracts compared with general trade in goods and services, since government contracts typically cover more than one good or service. Therefore, the country of origin for certain goods included in a contract may be different from the country of origin for other goods under the same contract. Country of contract performance. USG contracts can be executed within the United States or outside the United States. For example, the country of contract performance may determine where the service is delivered as opposed to the location or ownership of the firm that delivers the service. The place of performance may lead to benefits and costs accruing to the location where the contract is performed. For example, if a service is delivered or the products are produced outside the United States, the contract likely employs local labor and therefore benefits the local labor market. Because available data in government procurement databases do not specify the supplier firm’s economic role, the economic effects of the awarded contract remain uncertain. The potential effects of the awarded contract on other firms, workers, the government, or consumers in the domestic and foreign economies may vary depending on the supplier firm’s economic role. We estimate that foreign sourcing is generally a small share of government procurement for the United States and the other six parties to the GPA and NAFTA. Foreign sourcing by the USG and the other parties’ central governments, estimated by government procurement databases, varied in value from about 2 to 19 percent of overall central government procurement. Foreign sourcing by all levels of government, estimated by data on trade and public sector purchases by the United States and the other six main parties, shows that government imports ranged from about 7 to 18 percent of the goods and services purchased by these countries’ governments. In addition, our analysis of central government contract data found that foreign sourcing is sometimes but not always greater, in terms of value and number of contracts, for contracts covered by procurement agreements than for contracts not covered by those agreements. Our analysis of available data on firm location from government procurement databases shows that foreign sourcing in 2015 ranged in value from 2 to 19 percent of overall central government procurement (see fig. 4). The central governments of the EU, Mexico, and the United States awarded less than 5 percent of the aggregate value of their procurement contracts to foreign-located firms. The proportions for Canada and Norway were about 11 and 19 percent, respectively. Both Canada and Norway can be characterized as small, open economies bordering much larger, open trading partners, which may contribute to their relatively larger shares of foreign sourcing in central government procurement. Canada’s central government awarded about 10 percent of the value of all its contracts to firms located in the United States. Similarly, Norway’s central government awarded about 19 percent of the value of all its contracts to firms located in the EU. Our analysis of available data on country of product and service origin shows that Japan procured less from foreign sources (2 percent) than both the United States (6 percent) and South Korea (3 percent). See figure 5. We obtained similar results in terms of number of foreign-sourced contracts. Less than 5 percent of the number of central government contracts was sourced from abroad in the EU, Japan, South Korea, and Mexico. For the United States, Norway, and Canada, the numbers of foreign-sourced contracts based on firm location comprise higher percentages (9, 8, and 13 percent, respectively). Canada’s central government awarded about 9 percent of the total number of contracts it awarded to firms located in the United States. Similarly, Norway’s central government awarded about 7 percent of the total number of contracts it awarded to firms located in the EU. Except for the United States, most of the central governments of the other six main parties to the GPA and NAFTA awarded few construction services contracts to foreign-located firms. One possible explanation is that, given the higher dollar value threshold of contracts in this sector, foreign-owned firms may have a greater incentive to establish a local presence through subsidiaries in the host countries. The data in the non- U.S. databases do not provide enough information to explore that hypothesis. However, FPDS-NG data show that construction services contracts are the main contract type awarded to foreign-located firms by the USG, which awarded about 3,090 construction services contracts worth $1.8 billion (or about 20 percent and 8 percent of all construction services contracts, respectively) to foreign-located firms. Less than 1 percent of these contracts’ award value was for contracts performed in the United States and over 70 percent of these contracts’ award value was for contracts covered by the GPA and NAFTA. In addition, the USG awarded a roughly equal share (about 4 percent of all contracts in terms of value) of goods and services contracts to the other six parties to the GPA and NAFTA. Canada, on the other hand, awarded a relatively large percentage of the value of all goods contracts (30 percent) to firms located abroad. We also assessed the degree of foreign sourcing in terms of government import percentages to identify patterns in government procurement that may differ from those based on the location of the supplier and origin of goods and services. Using linked input-output tables and an alternative analytical approach, we were able to broadly estimate the domestic and foreign sources of inputs to the government sector for the United States and the six main parties to the GPA and NAFTA. This alternative approach to estimating foreign source government procurement is based on macroeconomic data on trade flows of goods and services between countries and the types of goods and services purchased by the public sector. Unlike the approach above based on government procurement contract data, this approach allows us to calculate broad estimates of domestic and foreign sourcing in procurement by all levels of government—central, state, and local. Table 3 shows our broad estimates based on a narrow definition of the government sector, which includes only “public administration”. In the table, the columns are the purchasing countries or the EU. The rows indicate where the goods or services are being purchased from. As the table shows, for all the countries and the EU, foreign sourcing generally accounts for a small portion of all governmental purchases. For example: Out of the estimated $1.2 trillion that the central, state, and local governments in the United States purchased, $100 billion was imported from outside the United States—a total foreign source percentage of about 9 percent, including $26 billion (2 percent) from the EU. Out of the $460 billion that the EU governments at every level purchased, $36 billion was imported from outside the EU—a total foreign source percentage of about 8 percent, including $10 billion (2 percent) from the United States. Out of the $178 billion that governments in Japan purchased, $12 billion was imported from outside Japan—a total foreign source percentage of about 7 percent. In general, the smaller economies in terms of government purchases— Canada, South Korea, Mexico, and Norway—imported a relatively larger percentage of such purchases than the United States, EU, and Japan. Specifically, Canada, South Korea, and Norway imported about 9 to 13 percent of their governments’ purchases. Mexico imported a notably large share of about 18 percent. Of the estimated $24 billion in purchases by Mexico’s government sector, about 6 percent was from the United States and about 3 percent from the EU. This inverse relationship between the size of an economy and the relative percentage import share of government purchases has been noted by others that have used the input-output methodology. Basing estimates of foreign source government procurement on the narrow definition of the government sector may not be as appropriate in countries where the government plays a large role in various additional sectors. Figure 6 shows the size of the government sector under the narrow definition as well as two broader definitions which add additional industries. The “typical definition” as defined in the EU study also includes the education and health care sectors. The “broad definition” also includes a portion of the energy and the telecommunications sectors. The relative sizes of the parties change under the different definitions, as shown in the figure. For example, while the EU government sector is less than half the size of the U.S. government sector under the narrow definition ($460 billion for the EU compared with $1,159 billion for the United States), under the broad definition they are comparable in size ($2.4 trillion for the EU and $2.6 trillion for the United States). Figure 7 shows the estimated percentages of each country’s and the EU’s government sector purchases that are imported under the narrow, typical, and broad definitions as described above. Under all three definitions, the United States and EU have some of the smallest percentages of imported government purchases, between 8 and 10 percent. Mexico has one of the largest percentages, between 17 and 22 percent. Canada and Norway are in the middle, from about 12 to 16 percent. For South Korea and Japan, the estimated percentages of government sector purchases that are imported increased under the broad definition—from 7 percent to 17 percent for Japan, and from 9 percent to 22 percent for South Korea. Our analysis of 2015 data from central government procurement databases finds evidence that foreign sourcing was sometimes, but not always, greater for contracts covered by the GPA and NAFTA than for contracts not covered by those agreements. Given the goals promoted by the GPA and NAFTA, one might expect that procurement covered by such agreements would likely result in a higher number or larger aggregate value of contracts awarded to foreign-located firms or for the purchase of foreign goods and services. For the United States and two of the other six main parties to the GPA and NAFTA—Mexico and South Korea—the results bore out that expectation: for all three, more central government foreign sourcing in terms of contract value occurred when procurement was covered by the agreements. However, our analysis also shows that for two other parties, Canada and Norway, the opposite was true; for the remaining two parties, the EU and Japan, we found little difference or could not calculate an estimate. Our previous work showed that only about a third of the estimated average annual government procurement at all levels of government from 2008 through 2012 was covered by the GPA and NAFTA ($1.5 out of $4.4 trillion). The available data from the government procurement databases that we analyzed show that the USG and the central governments of Mexico and South Korea awarded at least twice as much to foreign sources for contracts covered by international agreements—ranging from 2 to 6 percent of the value of covered contracts compared with less than 1 to 2 percent for non-covered contracts (see table 4). In particular, for contracts awarded by the USG, foreign-located firms received more than twice the value of covered compared with non-covered contracts—about $8.8 billion compared to $3.4 billion, respectively. Results for the USG are similar when looking at the amount of foreign source procurement based on product and service origin. Conversely, U.S.-located firms were awarded a higher aggregate value of non-covered contracts from the USG, compared with covered contracts. (See table 4.) For Canada and Norway, more central government foreign sourcing in terms of contract value occurred when procurement was not covered by trade agreements than when it was. For covered contracts, Canada’s central government awarded 1 percent of the value of all contracts to foreign-located firms compared with 10 percent of the value for non- covered contracts. Similarly, Norway awarded foreign-located firms more than 5 times more in non-covered than covered contracts as measured by aggregate contract value. For the EU and Japan, data on the value of foreign sourced contracts and their agreement coverage are either not available or incomplete. The available EU data have a significant number of foreign unclassified contracts and do not include contracts below the GPA threshold values, which limits the reliability of any comparison for covered versus non- covered contracts. In addition, Japan’s 2015 GPA submission of 2013 procurement data did not report on the amount of foreign source procurement broken out by covered and non-covered contracts, because, according to Japanese officials, this is not a GPA statistical reporting requirement. Therefore, we could not calculate a similar comparison of the value of covered versus non-covered procurement for Japan. Finally, with regard to the number of contracts awarded, our analysis of available data from country databases does not show a consistent relationship with international procurement agreement covered awards to foreign-located firms or for foreign-sourced goods or services. In South Korea and the United States, the number of contracts not covered by trade agreements and awarded for foreign sourced products was greater compared with covered contracts. Conversely, in Canada, EU, Mexico, and Norway, the number and share of contracts covered by trade agreements and awarded to foreign-located firms was greater compared with non-covered contracts. In percentage terms, foreign-located firms received the same share (9 percent) of covered and non-covered contracts awarded by the USG. We provided a draft of this product to USTR, Commerce, OMB, and GSA for comment. Commerce provided technical comments on this report, which we incorporated, as appropriate. USTR, OMB, and GSA did not comment on our draft report. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the U.S. Trade Representative, the Secretary of Commerce, the Director of the Office of Management and Budget, the Administrator of the General Services Administration, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8612 or gianopoulosk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report examines the extent of foreign sourcing in government procurement by the United States and the other six main parties to selected international procurement agreements. Under the World Trade Organization (WTO) Agreement on Government Procurement (GPA), the other main parties, besides the United States, are the European Union (EU), Japan, Canada, South Korea, and Norway. Under the North American Free Trade Agreement (NAFTA), the other main parties are Mexico and Canada. The report (1) provides alternative broad estimates of foreign sourcing by the U.S. government (USG) and the central governments of the other six main parties to the GPA and NAFTA, and (2) assesses foreign sourcing as a share of estimated central government procurement and of estimated procurement by all levels of government, and the extent to which central government contracts that are covered under the GPA and NAFTA are foreign-sourced. We analyzed data from two types of sources: (1) government procurement databases in Canada, the EU, South Korea, Mexico, Norway, and the United States, for 2015, and (2) 2014 trade data merged with data on the types of goods and services purchased by the public sector. Since Japan does not have a national procurement database, data for Japan were based on its WTO GPA submission for 2013, which is the last submission that contains information on its foreign sourcing in government procurement. We also interviewed cognizant government officials in Washington, D.C.; Ottawa, Canada; Mexico City, Mexico; Seoul, South Korea; and Tokyo, Japan, and reviewed available research literature to identify potential methods, sources, and data limitations. We also interviewed government officials at the EU mission in Washington, D.C. and exchanged information with officials knowledgeable of the EU government procurement database. We collected and analyzed data from the following five databases: for the United States, the Federal Procurement Data System-Next Generation (FPDS-NG); for the EU and Norway, Tenders Electronic Daily (TED); for Canada, Contract History; for Mexico, the Government of Mexico e-Procurement System CompraNet; and for South Korea, the South Korea ON-line E-Procurement System (KONEPS). Several of these government procurement databases included data on procurement at all levels of government—national, state, and local— while others did not. Therefore, we limited our analysis to data on central government procurement. For a detailed discussion of the characteristics of each database, see appendix II. To identify data fields that could be reasonably compared across databases, we followed a number of methodological steps: First, we looked for fields that capture the total award value of the contract at the time of award (2015); the type of contract in terms of goods, services, and construction services; the contract award date; the contract duration; and the type of tendering procedure. We took into account the following considerations: Units of analysis. We established appropriate units of analysis across databases. Several databases contained a number of fields that were potentially relevant to our work. Specifically, in FPDS-NG the unit of analysis is the contract award. The database contains data at the contract action level (contracts, task orders, and their modifications). We used contract awards for the number of reported contracts, but for certain data on indefinite delivery vehicles (IDVs) such as government-wide acquisition contracts, indefinite delivery contracts, and blanket purchase agreements, we relied on data for task orders awarded in fiscal years 2015 through July 2018 (see discussion of contract valuation and multiple-year, multiple-award contracts below) because they contained information on place of performance, country of product and service origin, and place of manufacture, which were the relevant fields for foreign sourcing. The TED database contains information on contract notices, contract award notices, and contract awards above certain thresholds set by relevant EU legislation. While the EU and Norway use contract award notices to estimate the value of covered procurement in their GPA statistical notifications, we used contract awards because they allowed us to estimate actual foreign sourcing. The databases for Canada, Mexico, and South Korea contain a contract identifier, which is the sole and unique unit of analysis that is available. Contract valuation. We established comparable fields across databases that represented the estimated maximum total value of a procurement awarded in 2015 over its entire duration. For FPDS-NG, we developed a methodology that is consistent with the methodology laid out in the revised GPA and avoids the inconsistencies of the revised U.S. methodology, which we previously reported. In particular, in October 2015, the Office of the U.S. Trade Representative (USTR) notified the WTO that the United States had revised its methodology for preparing GPA statistical reports on U.S. federal procurement. To more precisely reflect the value of the federal procurement market at the time of each report, the revised methodology presented the total amounts obligated under GPA covered contracts over a 6-year period—that is, the year the contract was awarded plus 5 years after the award. As we previously reported, the revised methodology has both advantages and disadvantages. It improves the accuracy of reporting but introduces a 6-year delay, whereas the revised GPA requires reporting within 2 years of the end of the reporting period. In addition, the revised valuation methodology is not consistent with the one used by other countries and creates an internal inconsistency: In measuring actual obligations for procurement contracts rather than the value at the time of award, the revised U.S. methodology is inconsistent with the methodology used by other large GPA members, such as the EU, Norway, Canada, and Mexico, which report contract values at the time of award rather than actual obligations or expenditures. The United States continues to report the number of covered contracts to the WTO based on their award value, which leads to an inconsistency between the reported numbers and values of reported U.S. government procurement contracts. The contracts comprising the reported value of covered procurement are determined at a later time under the revised methodology and can result in a different set of contracts being used to determine the reported value. Our current methodology uses base and all options value for all contracts awarded in fiscal year 2015 unless the contract was an IDV. For IDVs we used the base and all options value of task orders awarded in fiscal years 2015 through July 2018 under those IDVs to avoid overestimating the total value. We used the aggregate base and all options value for task orders under those contracts because the alternative—using the base and all options value on the base IDVs—is inflated due to problematic data entries for multiple awards. As a result, our methodology produces an estimate that is consistent with methods used by other parties, internally consistent, and in accordance with the methodology for valuation in the revised GPA. As we noted earlier, the result is close to the obligations value currently reported in the Trade Agreements Report used by USTR to report to the WTO. In TED, we used the contact award value field, because it captures the appropriate measure and according to EU documentation was corrected for errors in the data. For the EU and Norway, we found that for above-threshold procurement approximately 15 percent and 12 percent of the contract award values were missing, respectively. We took additional steps to address these missing values to generate estimates of the total contract award values. Specifically, we implemented a Predictive Mean Matching (PMM) multiple imputation methodology for the EU and used post-stratification estimation techniques for Norway. (See app. IV for more details on both methods.) However, we excluded the value of below-threshold procurement for the EU and Norway because it is reported on voluntary basis and suffers from missing and implausible values. In particular, for the EU, about 42 percent of the contract award values below threshold are missing and another 10 percent are below €1,000. For Norway, 80 percent of the contract award values below threshold are missing. Nevertheless, as a robustness check of the results from our analysis, we applied our imputation methodology discussed in appendix IV to the entire TED dataset and found that once those values are estimated, the amount of procurement awarded by the EU to U.S.- located firms increases by less than 10 percent. However, we do not consider the estimate sufficiently reliable to be included in our aggregate analysis. In Contract History, we used the contract value field because, according to Canadian officials, it includes the original total value of the contract at the time of the award. In addition, those officials noted that this field was used by Canada in its reporting of covered procurement for its WTO statistical notifications. In CompraNet, we used the contract amount field since, according to Mexican officials, this field reflects the total value of the contract award. In KONEPS, we used the total awarded value field, since it was the only field available for our analysis and contained the value awarded for a given year (see adjustments we made for multiple-year contracts below). Currency denomination. We converted contract values reported in different currencies in the databases into dollars using the period average exchange rate for 2015 as provided by the International Monetary Fund’s International Financial Statistics. Contract modifications or amendments. Since we defined the value of the award at the time of award, we selected contracts awarded in 2015 and excluded any subsequent modifications or amendments in all the databases. Contract types. We used the product and service classifications that each database used to group contracts by type. Different databases used different classification schemes, and we did not independently reclassify any contracts to a uniform classification system, since such a system does not exist and a concordance among all schemes is not possible. In FPDS-NG, we used the U.S. product and service codes to classify federal government contracts in product groups and categorized reported procurement as either goods, services, or construction services. In TED, we used the type of contract field, which categorized reported procurement as supplies, services, and works based on the EU common procurement vocabulary in TED. In Contract History, we used the grouping of goods, services, and construction, which Canadian officials provided to us based on the global shipment identification number codes and description in the database. In CompraNet, we used the type of contract field, which indicates if the contract is for goods, services, or public works. In KONEPS, the data on foreign procurement included goods only, and no classification scheme was available for foreign procurement contracts. Multiple-year, multiple-award contracts. Some countries’ procurement practices include contracts awarded for multiple years, and we accounted for the valuation of those contracts by estimating their total cumulative value over multiple years at the time of award in 2015. In FPDS-NG, we accounted for the value of multiple-award contracts by using the base and all options value of task orders awarded in fiscal years 2015 through 2018 for IDVs initially awarded in fiscal year 2015. In TED, available documents noted that member states can use alternative multiple-year tools such as framework agreements and dynamic purchasing systems for a certain time period or for repeat purchases, respectively. While the indicator field for these data in TED was not sufficiently populated for further analysis of those types of contracts, the contract valuation field we used had already accounted for the total value of the contract, and thus no further adjustment was warranted. Officials in Canada provided data on multiple-year contracts, including call-ups and standing offers. However, since the contract value field we used accounted for the total value of the contract, no further adjustment was needed. For Mexico, CompraNet contains information on framework agreements and multiple-year contracts, but since the contract value field indicated the total value of the contract award, no adjustment was needed. South Korea also uses multiple-year contracts, and we made several adjustments to estimate South Korea’s total value of 2015 awards. We identified multiple-year contracts in KONEPS in 2015; based on solicitation numbers, we then removed the value of contracts originally awarded in prior years, while adding the value of multiple year contracts with solicitations in 2015 and awards in 2016 and 2017. Type of tendering procedure. In all databases we included in our analysis contracts under open and limited tendering procedures. Second, we identified data fields among the five databases that could potentially be used as proxy measures of foreign sourcing in government procurement: contractor data related to firm location contractor data related to firm ownership data on country of product and service origin However, we did not identify a data field common to all five databases that could be used as a proxy measure of foreign sourcing. FPDS-NG contained data on all four measures listed above. TED, CompraNet, and Contract History contained contractor data related to firm location. KONEPS and Japan’s WTO submission on its 2013 procurement contained data on country of product and service origin. Therefore, two data fields—firm location and country of product and service origin—were available in two groups of countries as reasonable proxy measures of foreign source procurement. Finally, we analyzed the contract data from the government procurement databases by GPA coverage. Some databases contain a field for GPA coverage, the data for which we deemed to be reliable for our purposes; for the databases that did not, we developed a proxy measure for GPA coverage. FPDS-NG contains a field on trade agreement coverage, but we found it to be unreliable as reported in previous work; therefore, we constructed a method to identify GPA covered procurement using an approach that USTR confirmed is consistent with the steps applied by the USG in developing its GPA statistical notifications. TED contains an identifier for GPA covered procurement, and we used this field to estimate GPA covered procurement for Norway and the EU after taking steps to address missing values for this identifier using other information in the dataset. Contract History contains a field that lists all internal and international agreements applicable to a contract in Canada. Therefore, covered procurement includes all contracts covered under the GPA, NAFTA, and other Canadian international procurement agreements. For Mexico, CompraNet contains a data field on type of procedure, which indicates the eligible firms that can bid on a contract. The data in this field indicate that the contract is (1) open to national firms only; or (2) international procurement under trade agreements, that is, open to both national (Mexican) firms and foreign firms from FTA partner countries; or (3) international procurement open to national firms, foreign firms from FTA partners, and all other foreign bidders. We treated international procurement in CompraNet as a proxy for GPA covered procurement. We grouped all contracts awarded in 2015 into two categories: non-covered procurement, which includes contracts open to national firms only, and covered procurement, which includes contracts open to foreign bidders (i.e. all contracts in categories 2 and 3 described above). KONEPS does not have a data field that specifically identifies covered procurement. Therefore, we defined a proxy for covered procurement as procurement above the revised GPA thresholds by covered entities. However, we were unable to make an adjustment for goods and services excluded from the agreement, since KONEPS does not classify foreign procurement by product service codes. To analyze the extent to which central government contracts that are covered under the GPA and NAFTA are foreign-sourced, we compared the proportion of foreign-sourced award values for contracts covered under the GPA and NAFTA to the same proportion of foreign-sourced contracts, which are not covered by those agreements. Our analysis describing the relationships between trade agreement coverage and procurement award values did not account for additional factors and was limited due to the data available. As we previously reported, the countries within our scope represent over 90 percent of the GPA countries’ total government procurement. Moreover, we previously performed consistency checks across time periods for these countries and determined that covered procurement out of total central government procurement appeared relatively stable over time. However, a more robust test of the relationship between foreign sourcing and selected international agreement coverage would use a larger cross-section of data over time and control for factors such as types of goods and services procured, size of the economy, type of tendering procedure, and other specific details of each agreement, among others. To determine whether the procurement contract data from the five databases were reliable for our purposes, we identified in relevant countries the appropriate data sources used to prepare the countries’ and the EU’s submissions of statistical notifications to WTO and other government procurement reports. To ensure consistency between our methods for estimating foreign sourcing with the methods used by the countries and the EU in their estimates of covered procurement for their GPA statistical notifications, we discussed with government officials in Canada, Japan, Mexico, and South Korea their process and data used to create their statistical notifications and other WTO reports, and we took steps to replicate existing report totals of EU covered procurement. We performed a sensitivity check for the U.S. data in FPDS-NG, where more than one relevant data field was available, to determine whether the definitional differences in the data fields were likely to materially affect our results about foreign sourcing. The results were similar across all six fields that could be used as alternative proxy measures of foreign source procurement in FPDS-NG data (see app. III, tables 11 and 12). In addition, we conducted electronic tests of all five procurement databases to identify whether the data were complete and internally consistent. We determined that the country procurement databases were sufficiently complete and internally consistent after taking the additional steps for the EU and Norway as described earlier, related to missing contract award values (see app. IV). We also shared our analyses of the data with cognizant officials from the corresponding countries who were willing to verify our methodology and replicate our analysis. Procurement and trade officials and researchers in Canada, Mexico, South Korea and Japan answered our questions relevant to data quality including data collection, cross checks of data entries, access controls, internal reviews, primary users, completeness and updates to the data, missing values, reporting mistakes, electronic safeguards and procedures for follow-up if errors are found. In Canada and Mexico officials replicated and confirmed our methodology and results. Results for South Korea and Japan were consistent with alternative available official publications. The various limitations in the procurement contract data that we identified and addressed, to the extent possible, affected our ability to obtain precise estimates of foreign sourcing in government procurement, but they were not an impediment to using the data for broad comparisons of orders of magnitude. Such comparisons include the amount of foreign sourcing, measured using firm location and country of product and service origin, by the USG and central governments of the other six main parties to the GPA and NAFTA. The data also allowed broad comparisons of bilateral procurement flows among the parties, as well as comparisons by type of contract and agreement coverage, as available, for the seven parties to the GPA and NAFTA within our scope. To obtain information on the aggregate levels and percentages of procurement by all levels of government that are imported, we relied on input-output tables from the World Input Output Database (WIOD) for 2014. The input-output tables have an industry by industry format, with each country’s industries listed separately. The data in each table are derived from publically available data from both national statistics agencies and international organizations such as the United Nations and the Organisation for Economic Co-operation and Development. We relied on the WIOD to ensure that the combined data from different countries was collected to be consistent. These data do not allow for distinctions between different levels of government. To assess the reliability of estimates based on the WIOD data, we first reviewed available documentation for the database. In cases where we had questions, we received written responses from WIOD officials. In addition, we compared estimates based on the WIOD to estimates based on other databases and found similar results. In general, we found that the data were sufficiently reliable for our purposes. To estimate the level and percent of procurement from the database, we took the following steps. First, we identified the industries associated with the governmental sector. Then, for that industry (or combination of industries), we obtained both the total level of purchases (or inputs), and the inputs that came from within that country, or other countries of interest. To obtain an estimate for the EU, we combined the purchases over the 28 member countries then in the EU. In general, we followed a procedure outlined in a 2017 paper produced by the European Commission. In this paper, the authors describe how input-output tables can be used to measure cross-border penetration in public sector procurement. An essential step in our method is defining which industries make up the government sector. Moreover, because the composition of the government sector and the patterns of government purchases vary by country, different measures of the government sector are more appropriate for different countries—since what goods and services the government provides or performs affects what it procures from the private sector. For example, for the EU, the government funds the majority of services in the area of public administration, defense, social security, education, and health care. In contrast, the USG funds a smaller share of health care services. We followed the model laid out in the European Commission paper and defined the government sector in three ways: 1. Narrowly Defined – (O84) Public administration and defense; compulsory social security 2. Typically Defined – (O84) Public administration and defense; compulsory social security (Q) Human health and social work activities 3. Broadly Defined – (O84) Public administration and defense; compulsory social security (Q) Human health and social work activities (D35) Electricity, gas, steam and air conditioning supply (E36) Water collection, treatment and supply (E37-E39) Sewerage; waste collection, treatment and disposal activities; materials recovery; remediation activities and other waste management services (1/3) * (H49) Land transport and transport via pipelines (1/2) * (H53) Postal and courier activities (1/2) * (J61) Telecommunications However, our procedure deviated from the European Commission report with regard to an additional category of expenditure in the report, final consumption by government. As in our prior reports, we did not include this category. This category includes both spending on social benefits, health care, and education as well as spending on collective items such as defense. We did not include this category in prior reports partly due to data reliability concerns about consistency in measurement of spending on social benefits across countries. However, if we had included it, that would have caused our estimates of import penetration to be smaller, because the WIOD tables do not include any cross-border expenditures for this category. For example, the percentage for the United States would have changed from about 8 percent to about 4 percent. To construct consistent data from different countries over time, certain assumptions were made by the WIOD. An assumption that has important implications for our analysis is known as a “proportionality assumption,” which is typical in the construction of input-output tables. This assumption requires that the percentage of a product that is imported is constant across all industries. In the example provided by the WIOD: “If 20 percent of Czech absorption of electronics is sourced from Germany, then 20 percent of any Czech final or intermediate use of electronics is assumed to originate from Germany.” The WIOD has attempted to improve on the proportionality assumption by making it at a more disaggregated level, but according to the WIOD, the proportionality assumption remains a limitation of the data set and consequently of our analysis. Importantly for our analysis, the proportionality assumption implies that the results we obtained from this method may not capture attempts by the government sector to award a larger share of its procurement to domestic firms relative to other industries. Another important limitation for our analysis is the scope of the industry data reported by the WIOD. Specifically, the input-output data include intermediate inputs but exclude purchases by government for investment. Such purchases could include some government assets that would be considered procurement covered by the GPA and NAFTA. For example, the input-output data could exclude construction services like those government purchases to build highways or schools that have long-term use, which are procurements potentially covered by the GPA and NAFTA. Finally, while we followed a method described above that has been used to study procurement, there are alternative methods that could have also been used based on input-output data. For example, according to industry officials at the U.S. Bureau of Economic Analysis, the “Trade in Value Added” methodology is such a method, and such data are maintained by the Organisation for Economic Co-operation and Development. We conducted this performance audit from March 2017 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The following appendix contains descriptive comparative information about the five databases included in our review: for the United States, the Federal Procurement Data System-Next Generation (FPDS-NG); for the EU and Norway, Tenders Electronic Daily (TED); for South Korea, the South Korea ON-line E-Procurement System (KONEPS); for Mexico, the Government of Mexico e-Procurement System CompraNet; and for Canada, Contract History. For each database, we provide its formal name and function, contract and/or agency coverage, and data field(s) related to firm location, firm ownership, source country of goods or services, location of contract execution, contract valuation, trade agreement coverage, and type of contract in terms of goods, services or construction services. The following appendix provides supplemental information from our analysis of foreign sourcing by the United States in fiscal year 2015 based on data from the Federal Procurement Data System-Next Generation (FPDS-NG). FPDS-NG contains data on four potential proxy measures of foreign sourcing—firm location, firm ownership, product and service origin, and place of performance. The database contains six fields that correspond to these four proxy measures. See tables 11 and 12. For cross-country comparisons, we use two of the six measures—vendor country code (13QQ) and country of product and service origin (9E). We disaggregate the data by country and list the top 20 countries, which are recipients of USG contracts based on firm location. See tables 13 and 14. Since about 10 percent of USG contracts are performed outside the United States, we also provide a breakdown of those contracts that are awarded to foreign-owned and –located firms by agency. See table 15. Finally, since most of these contracts by contract value are awarded by the Department of Defense (DOD), we also provide a country breakdown of DOD contracts performed outside the United States and awarded to foreign-owned and –located firms. See table 16. To report on European Union (EU) procurement data in the Tenders Electronic Daily (TED) database for 2015, we took steps to address missing contract award values, which amounted to approximately 15.2 percent of the 38,233 in-scope contract award values. To address these missing contract award values, we implemented a multiple imputation methodology that imputes a range of values for each missing contract award value and allows for estimation of additional uncertainty induced by the imputation methodology. After determining that the data were likely to be conditionally missing at random, we used predictive mean matching (PMM) to address missing values as described below. We determined that using PMM was appropriate because it can provide more robust results when the relevant variable is not normally distributed; PMM, as a form of multiple imputation, allows us to assess the variability introduced through the process of addressing missing data; and PMM, when properly specified, does not distort averages or variance in the underlying data. As we discuss below, the method for addressing missing values used by the EU has none of these features. In PMM, a regression model is first fit to complete cases in the dataset to predict values for the variable of interest for the entire dataset (i.e., including complete and incomplete cases). These predicted values are used to identify complete observations (“donors”) that are close (a “match”) to a given observation that is missing a value for the variable of interest. The PMM model draws matches using the posterior predicted distribution of the regression model. When PMM is used in conjunction with multiple imputations, this process is repeated multiple (m) times for each missing value. As a result, each of m imputations may match to a different donor. The donor’s observed value for this variable is donated to fill the blank data cell—not the predicted value used to match to this donor. The strength of the predictive model used to identify these matches will affect variation in the set of m imputed values because a better predictive model will identify donors that have observed values more consistently close to their predicted values. In order to specify our PMM model, we first explicitly tested the Ordinary Least Squares regression model used to match donors as part of the process discussed above. We performed standard regression diagnostics, including an examination of the included variables and residuals to avoid overfitting. We found that our model was able to explain 86 percent of variation in contract award values and appeared to have well-behaved (homoscedastic) residuals. We drew m=30 imputed values for each missing observation using the PMM process described above, which allowed us to generate estimates of the total contract value amounts and measure the uncertainty induced in those estimates by the imputation methodology. We used these measures of uncertainty to construct 95 percent confidence intervals and express these values as a percentage relative to the estimate itself. To assess the quality and reliability of the multiple imputations that followed from this predictive model, we performed four main sensitivity checks, which are included in tables 17 and 18. 1. We examined the proportion imputed for each subset of the data that we planned to report. The column headed “Percent imputed contract awards” shows the proportion of the count of contracts in a given data subset that were imputed using the methodology described above. We looked to avoid any individual subset being substantially greater than the overall average of 15 percent imputed. In practice, we individually checked any subset exceeding 30 percent imputed. 2. We evaluated the level of uncertainty induced by the imputation methodology across important subgroups of the data. The column headed “95 percent confidence interval +/-” indicates the percent of the “Contract award value estimate” that, when added and subtracted to this estimate, forms the 95 percent confidence interval. The level of uncertainty expressed in the relative confidence interval results from between-imputation variance, which could indicate extreme or inconsistent matches. We looked for confidence intervals that were, in our judgment, narrow as a proportion of point estimates. In practice, nearly all of the subgroups we are choosing to report have confidence intervals smaller than plus or minus 3.5 percent of point estimates. 3. We evaluated the percent of imputed values across important subgroups of the data. “Percent of imputed value duplicates” is a diagnostic column to test for sparseness of imputation matches among the 30 imputed values for each imputed contract award. We determined the number of duplicate imputation draws among the 30 imputed values for each observation, which could indicate sparseness in the number of suitable matches or overfitting of the model. We intended to inspect any finding with more than about 5 out of 30 (17 percent) duplicated imputation draws; in practice, however, this threshold was not reached for any subsets of the data that we have chosen to report. 4. We compared estimates resulting from our imputation methodology to published EU reports across important subgroups of the data. “Alternative estimate (EU’s missing value methodology)” shows the results of replicating a methodology for correcting missing data described in EU documents and used for some EU reports. The EU methodology is based primarily on the average value of contracts that are present in the dataset. This provides a general point of comparison, allowing us to determine which subsets of the data are likely to be responsible for estimation differences with prior EU publications. This comparison methodology therefore provides a benchmark but not a diagnostic for the imputation models. There are several important differences between our imputation methodology and the EU’s methodology. a. Calculation of confidence intervals: The EU’s methodology results in the same value substituted for every contract award of a given type (construction goods, and services). As such, it is not possible to estimate confidence intervals for a given observation or group of observations using this methodology. In contrast, the multiple imputation models include estimates of uncertainty. b. Distortion of subgroup averages: The EU’s methodology is not sensitive to differences in group averages apart from contract type. As a result, it may distort subgroup averages. For example, if hypothetical Country A has services contracts that average $100 but the overall average for services contracts is $1,000, substituting the overall average into missing values for Country A as the EU methodology would have the effect of significantly distorting Country A’s characteristics. In contrast, the imputation models we used are designed to be sensitive to all significant reported differences in contract awards because we included all reported variables in our imputation models. To report on Norway procurement data in the TED database for 2015, we needed to take steps to address missing contract award values (153 of 1,319 missing, or about 11.5 percent). The scale of the missing values is thus smaller than for the EU data, while the dataset as a whole is too small, in our judgment, to support correction through an imputation model. Our statistical tests found no evidence that contract award values were conditionally missing at random. Thus, we assume that the data are missing completely at random and corrected the missing data using post- stratification estimation techniques. To do so, we treated the database of contract awards as the full population of such contract awards, which provides the full joint distribution of contract attributes. We treated the complete observations (88.5 percent of the total) as our sample of this population. Post-stratification adjusts the sampling weights for this sample so that the joint distribution of post-stratifying variables, which we selected based on our reporting needs, matches the known population joint distribution. Based on the resulting confidence intervals, we determined that the post- stratification sampling results in data are sufficiently reliable for subsets defined by foreign status and contract type (see table 20) or by foreign status and GPA coverage (see table 21). Kimberly Gianopoulos, (202) 512-8612 or gianopoulosk@gao.gov. In addition to the contact named above, Adam R. Cowles (Assistant Director), Marisela Perez (Analyst-in-Charge), Gergana T. Danailova- Trainor, Ben Bolitzer, Andrew Kurtzman, and Julia Kennon made major contributions to this report. James Ashley, Peter Choi, David Dayton, Christopher Keblitis, Grace P. Lui, John Yee, and Timothy Young provided technical assistance.", "summary": "Globally, government procurement constitutes about a $4 trillion market for international trade. However, little is known about foreign sourcing in government procurement—how much governments procure from foreign-located suppliers or how much they acquire in foreign-made goods. GAO was asked to review the extent of foreign sourcing in government procurement across countries. GAO focused on the United States and the other six main parties to the GPA and NAFTA, selected international agreements that open procurement markets on a reciprocal basis. This report, the fourth of a related series, (1) provides broad estimates of foreign sourcing by the U.S. government and central governments of the other six main parties, and (2) assesses foreign sourcing as a share of estimated central government procurement and of estimated procurement by all levels of government, and the extent to which central government contracts that are covered under selected international procurement agreements are foreign-sourced. GAO analyzed the most recent comparable data available from two sources: (1) government procurement databases used in Canada, the European Union, South Korea, Mexico, Norway, and the United States, for 2015, and (2) 2014 trade data merged with data on the types of goods and services purchased by the public sector. Since Japan does not have a government procurement database, data for Japan were based on its 2015 GPA submission of 2013 data. GAO also interviewed cognizant government officials in Washington, D.C.; Ottawa, Canada; Mexico City, Mexico; Seoul, South Korea; and Tokyo, Japan. The U.S. government awarded contracts valued at about $12 billion to foreign-located firms, of which about $5 billion went to firms with reported locations in the other six main parties to the World Trade Organization Agreement on Government Procurement (GPA) and the North American Free Trade Agreement (NAFTA) (see figure). Conversely, government procurement databases indicated the central governments of these parties awarded an estimated $7 billion to foreign sources, out of which about $2 billion was U.S.-sourced. Canada and Mexico awarded most of the U.S.-sourced contracts. GAO was able to determine that the U.S. government awarded more, by contract value, to foreign-owned firms located abroad than to foreign-owned, U.S.-located firms. Moreover, more than 80 percent of U.S. government contracts awarded to foreign-owned firms located abroad were Department of Defense contracts performed abroad. Overall, while available contract data enable broad cross-country comparisons, they do not necessarily show where the goods are produced, where the services are delivered, or where the profits go, among other economic effects. Foreign sourcing by the seven GPA and NAFTA parties within the scope of the study, using two alternative methods, is less than 20 percent of overall central government procurement. Foreign sourcing by central governments, estimated from government procurement databases of the United States and the other six main parties, varied in value by party from about 2 to 19 percent of overall central government procurement. Foreign sourcing by all levels of government, estimated from data on trade and public sector purchases, showed that the governments' imports likely ranged from about 7 to 18 percent of the goods and services the governments purchased. In addition, contract data show that U.S., South Korean, and Mexican central government foreign sourcing was greater in value under contracts covered by GPA and NAFTA than under noncovered contracts, but the opposite was true for Canada and Norway. For the European Union and Japan, GAO found little difference or could not calculate an estimate.", "document_type": "gao"}
{"report": "The Air Force operates several types of RPAs: the MQ-9 Reaper; RQ-4 Global Hawk; and RQ-170 Sentinel. The MQ-9 Reaper RPA community has about four times the number of pilots and eight times the number of sensor operators assigned as compared to the next largest RPA community (the RQ-4 Global Hawk). Additionally, the MQ-9 Reaper RPA provides persistent intelligence, surveillance, and reconnaissance and strike capabilities against high-value, fleeting, and time-sensitive targets. It is operated by an aircrew that includes an officer pilot and enlisted sensor operator. See figure 1. The Air Force RPAs operate remote split operations, which divides the control of the RPA among geographically separated units. Remote split operations employ a launch and recovery ground control station unit aircrew who controls the RPA’s take-off and landing at an overseas operating location while a crew based in the continental United States (i.e., the Mission Control Element unit) flies the RPA the remainder of the mission via electronic links. Remote split operations result in fewer personnel deployed overseas, consolidates flying multiple aircraft from one location, and as such, simplifies command and control functions as well as the logistical supply challenges for the weapon system. RPA operations include Active Duty and Air National Guard personnel and locations. Figure 2 shows the location of bases involved in RPA training and MQ-9 Reaper RPA operational locations with the active-duty sites bolded. Over nearly two decades, the number of combat lines and flying hours for RPAs has grown substantially. Specifically, in 2008, the Air Force flew 33 RPA combat lines but in 2015, the number had increased to 60 RPA combat lines. A combat line is the measure of the capability to provide near-continuous 24-hour flight presence of an RPA over a specific region on Earth, to include time flying to and from a specific target area. In doing so, the RPA can provide air action against hostile targets that are in close proximity to friendly forces, gather intelligence, or, if necessary, employ its weapons to strike identified targets. Additionally, the number of combat flying hours has also increased from calendar year 2000, as shown in figure 3 below, and reached 4 million cumulative combat hours in March 2019. In March 2016, General Herbert J. Carlisle, then-commander of Air Combat Command, testified to the Senate Armed Services Committee’s Subcommittee on Airland that the RPA enterprise has been “a victim of its own success” with “an insatiable demand for RPA forces” that was taxing the capability of the community. To meet the demand for RPA pilots, the Air Force has pursued efforts to increase the number of RPA pilots. For example, the Air Force trained traditional manned-aircraft pilots to fly RPAs and placed graduates of manned-aircraft pilot training into RPA training rather than in advanced manned-aircraft training. In 2010, the Air Force created a dedicated RPA pilot career field (i.e., 18X specialty code) and developed a training program for pilots who specialize in flying RPAs. In December 2013, there were 1,366 Air Force RPA pilots, of which 249 were dedicated RPA pilots (18 percent). Six years later, in December 2019, the number of total Air Force RPA pilots had grown to 1,768, with 1,127 of those being dedicated RPA pilots (64 percent). MQ-9 Reaper RPA pilots and sensor operators complete multiple phases of training designed to generate combat mission capable aircrews within approximately a year of starting training. First, the pilots initially attend RPA Flight Training in Pueblo, Colorado, and then Undergraduate RPA Training at Randolph Air Force Base, Texas, which includes instrument qualification in simulators and an RPA fundamentals course. Second, they complete MQ-9 Initial Qualification Training at the formal training unit at either Holloman Air Force Base in New Mexico, March Air Reserve Base in California, or Hancock Field Air National Guard Base near Syracuse, New York. Finally, they are assigned to an operational squadron, where they complete unit-specific Mission Qualification Training that can vary in length. According to officials at two RPA bases, their respective Mission Qualification Training was taking between six to 10 weeks or as much as 17 weeks to complete. MQ-9 Reaper RPA sensor operators go through a similar pipeline. They complete courses on aircrew fundamentals and the basics of being a sensor operator at Lackland Air Force Base, Texas, and Randolph Air Force Base, Texas, respectively. Then, they complete training at the MQ- 9 Reaper RPA formal training unit at Holloman Air Force Base, New Mexico; March Air Reserve Base, California; or Hancock Field, Syracuse, New York. Finally, they complete unit-specific Mission Qualification Training in the operational unit at which they are assigned after graduation. Figure 4 shows the MQ-9 Reaper RPA aircrew training pipeline. The Air Force does not have enough RPA pilots and sensor operators to meet its staffing targets, and it does not track its overall progress to access and retain sufficient quantities of RPA personnel that is needed to implement its combat-to-dwell policy as planned. More specifically, the Air Force has not consistently met its accession targets for RPA pilots and sensor operators and has had fewer RPA pilots and sensor operators than it has needed for most years between fiscal years 2016 through 2019. The Air Force has offered financial retention incentives to RPA pilots and sensor operators; however, it does not directly measure RPA pilot and sensor operator retention rates and retention concerns exist. Moreover, the Air Force does not track the overall progress being made from its accession and retention efforts to maintain a sufficient quantity of RPA pilots and sensor operators needed to implement as planned its combat-to-dwell policy—a policy intended to better balance RPA units’ time in combat operations with time spent away from those operations to accomplish other activities such as training. The Air Force met its accession targets for its RPA pilots in only one year during fiscal years 2015 through 2019 and it did not meet any of its sensor operator accession targets during those years. However, this is not a new trend. In 2014, we reported that the Air Force did not achieve its accession targets for RPA pilots in fiscal years 2012 and 2013 and recommended that the Air Force develop a tailored accession strategy for RPA pilots to help ensure that it can meet and maintain required staffing levels to meet its mission. The Air Force concurred with the recommendation and took steps to address accession issues for RPA pilots, such as having officers with RPA pilot experience serve at the U.S. Air Force Academy as instructors and as ROTC detachment commanders and instructors at several large, nationally recognized universities, thus giving attention to the career field among future Airmen. Because of these actions to address RPA accessions, the Air Force met the intent of our recommendation. Since then, however, the Air Force has not consistently met its annual accession targets from fiscal years 2015 through 2019, as shown in figure 5. As shown in figure 5, for the 5-year period between fiscal years 2015 and 2019, the average accession target fill rates for pilots and sensor operators were 95 and 88 percent, respectively. Air Force officials told us that they do not believe the RPA pilot career field is facing an accessions problem and thus there is no need to offer an accession bonus because the overall population of RPA pilots has been steadily growing year after year. These officials attribute the trend to the appealing RPA mission. Participants in 12 of 14 focus groups we conducted agreed that the ability to affect front line combat operations and missions every day was a positive aspect of the job. For sensor operators, Air Force officials told us that the number entering active-duty service reflects the number who had finished Basic Military Training and their first RPA-specific training course. These numbers would have been higher but Air Force officials stated they have determined that about 11 percent are disqualified during Basic Military Training sensitive skills screening. This screening involves identifying individuals upon entry into the service with behavioral or mental health issues and is used for, among other things, determining a trainee’s job classification and qualification for sensitive occupations. According to Headquarters Air Force officials, the 711th Human Performance Wing at Wright Patterson Air Force Base, Ohio, has ongoing research to help better identify the right types of airmen for RPA positions beyond the vocational aptitude battery test given to determine how qualified an enlistee is for certain occupations. They said that they expect the results of that research to be disseminated in early fiscal year 2021. According to Air Force data, the service has had fewer RPA pilots and sensor operators as compared to both their respective requirements and authorizations for almost the entire time between fiscal years 2016 through 2019. More specifically, the number of RPA pilot and sensor operator requirements has increased every year in support of the Air Force’s plan to create a new wing by 2024 that is needed to implement the combat-to-dwell policy. These Air Force requirements represent minimum essential resources needed to accomplish approved missions and functions that are valid, unconstrained, and realistic. After establishing the number of required positions, the Air Force fills these required positions to the extent possible based first on the number of those positions funded by Congress (i.e., authorizations) and then the number of trained and qualified personnel available to assign to those positions. Since fiscal year 2016, the overall number of authorized and assigned Air Force RPA pilots and sensor operators has increased. However, for a majority of the time in fiscal years 2016 through 2019, the Air Force’s number of assigned RPA pilots and sensor operators were less than both of their respective authorizations and requirements, as shown in figures 6 and 7. The overall number of assigned RPA pilots has increased; however, this trend has not been enough to meet the increased number of authorized positions in this RPA career field. For example, for RPA pilots, there was a 22-percent gap between authorizations (1,168) and assigned (908) in August 2015 which was similar to the 20-percent gap between authorizations (1,652) and assigned (1,320) in September 2019. The Air Force’s Rated Officer Retention Analysis report for fiscal year 2019 states that each of the four rated groups (pilots, combat system officers, air battle managers and RPA pilots) ended fiscal year 2019 in a deficit. Current projections indicate that the pilot deficit will continue into the near future. The report went on to say that while the number of assigned RPA pilots actually grew in fiscal year 2019, increases in the requirements for this career field reduced or negated the effect of the increase. Additionally, there was less than a 10 percent gap between the number authorized and assigned sensor operators during fiscal year 2016. However, by September 2019, a gap of 28 percent had developed (1,277 authorizations versus 919 assigned). To encourage the retention of RPA pilots and sensor operators, the Air Force has provided financial incentives for many years. For example, the National Defense Authorization Act for Fiscal Year 2017 authorized RPA pilots to receive aviation incentive pay up to $1,000 a month and an aviation retention bonus up to $35,000 to those who are willing to extend their service. In addition, the Air Force has offered a number of financial incentives to RPA sensor operators. At various times in January 2010 through November 2019, RPA sensor operators were eligible for monthly aviation incentive pay, critical skills incentive pay, or special duty assignment pay to address retention issues and have occasionally been eligible for Selective Retention Bonuses. In November 2019, the Air Force offered a Selective Retention Bonus to RPA sensor operators who were eligible to reenlist and had between 17 months to 6 years of military service. To measure long-term retention trends among pilots other than RPA pilots, the Air Force calculates two retention metrics—the Cumulative Continuation Rate and the Total Active Rated Service rate. However, the number of RPA pilots (i.e., Air Force Specialty Code 18X pilots) is still too few to have enough data to calculate reliably these standard retention metrics since the career field was not established until 2010. Officials at Headquarters Air Force and Air Combat Command told us that to calculate the Total Active Rated Service metric, the Air Force would need about 20 years of data; however, the RPA pilot career field is too new to have that amount of data. These RPA pilots have a 6 year Active Duty Service Commitment, which begins at the end of their undergraduate RPA training at Randolph Air Force Base. According to Air Force officials, the first group of 18X pilots’ service commitments ended in fiscal year 2019. Senior leaders at an RPA base we visited said that due to the newness of the RPA pilot 18X career field, the Air Force does not currently have enough historical data to help predict retention trends going forward. They also noted that until the combat-to-dwell policy is implemented, it is unknown what effect it will have on RPA personnel retention. According to Air Force officials, the Air Force tries to retain about 60 to 65 percent of those who have completed their initial service commitment and are eligible to be retained. However, this target is based on the average aviation retention bonus acceptance rates (i.e., the percentage of pilots accepting the retention bonuses) for healthy and established career fields where the number of required positions are not substantially increasing and which are able to meet between 95 to 100 percent of their staffing requirements. However, as previously discussed, RPA pilot requirements have increased about 74 percent in the 5 years from fiscal years 2015 through 2019. Therefore, these Headquarters Air Force officials stated that use of the 60 to 65 percent target may not be an appropriate target for RPA pilot retention. In the case of RPA pilots, if the Air Force met that target, Air Force officials said the service would still be understaffed due to the growing requirements, so the retention target would need to be higher. Further, they stated that while aviation retention bonus acceptance rates are leading indicators of retention, they are not measures of actual retention rates and there are limitations to using this approach. For example, one limitation is that pilots may choose to stay in the Air Force but not take the aviation retention bonus to exercise more control and flexibility over their career. In these cases, actual retention would be higher than the aviation retention bonus acceptance rate suggests. According to the Air Force’s annual Rated Officer Retention Analysis reports we reviewed, the combined aviation retention bonus acceptance rates for RPA pilots both with and without previous manned aircraft experience completing their initial service commitment were approximately 55 percent in fiscal year 2016, 64 percent in fiscal year 2017, and 60 percent in fiscal years 2018 and 2019. Our comparison of the aviation retention bonus acceptance rates for RPA pilots with previous manned aircraft experience to those without that experience suggests that the pilots without that experience have consistently had lower bonus acceptance rates, as shown in table 1. As far back as April 2014, we reported that there were indications the Air Force could be facing challenges retaining RPA pilots in the future. Despite the existence of incentive payments, pilots in seven of the 10 focus groups we conducted at that time indicated that retention of RPA pilots was or would be a challenge. We recommended that the Air Force develop a retention strategy that was tailored to the needs and challenges of the RPA pilots to help ensure the Air Force could meet and retain required staffing levels to meet its mission. The Air Force took some steps to address RPA pilot retention, such as expanding RPA operations to an additional base to increase assignment choices and decreasing the number of combat lines that RPA aircrews were flying to reduce their workload. Further, in July 2018, officials said that the Air Force established a new division at Headquarters to serve as a focal point for overseeing RPA personnel matters for the service. Because of these actions to address RPA retention, the Air Force met the intent of our recommendation. However, in our current review, we found indicators of concern regarding RPA pilot retention. For example, officials in varying leadership positions in the Air Force raised concerns about RPA pilot retention. Air Combat Command officials stated that they assume that about 30 percent of RPA pilots each year will have to be replaced due to attrition. Senior leaders at one RPA base that we visited told us that not having dwell time as a break from constant combat operations negatively impacts RPA personnel resiliency and retention. They said that to get a break from combat operations, RPA personnel turn to the Air National Guard or separate. They noted that people join the Air Force to see and do things, not to be exposed to constant combat operations in less than appealing locations. Further, according to RPA officials, personnel stated in exit interviews that they wanted more temporary duty opportunities, deployments, exercises, and other opportunities for better career development. Similarly, senior leaders at another location we visited said that the lack of training and leadership opportunities affects retention. They noted that there are hundreds of pilots at Creech Air Force Base, but only one wing commander, and this has a chilling effect given the limited leadership opportunities available. With regard to RPA sensor operators, Headquarters Air Force officials stated that the Air Force does not have an RPA-specific sensor operator retention goal, but rather it generally aims to retain about the same amount as other career enlisted aviator career fields have historically retained, which is about 70 percent. However, according to a February 2017 memorandum, the RPA sensor operators experienced a steady decline in retention since 2012. This memorandum requested Special Duty Assignment Pay for RPA sensor operators stating that airmen in this career field were placed under enormous personal and professional demands. It also stated that in a 2-year sample, 2014-2016, the Air Force Personnel Center reported a 31 percent reenlistment decrease for first term RPA sensor operators, a 7 percent decrease for second term RPA sensor operators, and a 16 percent decrease for career RPA sensor operators. Specifically, the memorandum said that in 2016 the reenlistment rates for RPA sensor operators were 44 percent, 54 percent, and 74 percent for first-term, second-term, and career RPA sensor operators, respectively. In comparison, these rates were 19 percent, 22 percent, and 16 percent lower than the average rate across all Air Force Career Enlisted Aviators. The Air Force approved this Special Duty Assignment Pay for RPA sensor operators effective in November 2017. Additionally, effective October 2018 and again in July 2019 and November 2019, RPA sensor operators were eligible to receive Selective Retention Bonuses. Coinciding with the start of these financial incentives in fiscal year 2018, Air Force data showed increases in RPA sensor operator reenlistment rates as compared to fiscal year 2017 reenlistment rates (see table 2). While Air Force data show improvements in RPA sensor operator reenlistment rates, officials we spoke with shared concerns about retention-related issues specifically regarding sensor operators. For example, a senior leader at one RPA base we visited said that there is an acknowledged retention problem within the sensor operator community citing one of the factors being the perception among sensor operators that private contractors pay more than the Air Force. An Air Force document justifying the Selective Retention Bonus states that contractors are targeting experienced RPA sensor operators for six-figure salaries of greater than $100,000 per year. Similarly, a senior leader at one RPA base we visited stated that contractors are paying sensor operators 2 to 4 times as much as the Air Force does, essentially making the Air Force a pipeline for RPA personnel to become government contractors. Moreover, participants in each of the senior RPA sensor operators (i.e., E5-E9) focus groups that we conducted told us that they thought the retention bonuses and financial incentives were too small to matter in their retention decision- making. In a questionnaire we administered to the 105 participants across the 14 focus groups, nearly half (19 of 41) of the sensor operators responded they were “somewhat dissatisfied” or “very dissatisfied” with their total compensation versus 20 percent (13 of 64) of pilots who responded they were “somewhat dissatisfied” or “very dissatisfied.” The Air Force does not track its overall progress of accessing and retaining sufficient quantities of RPA pilots and sensor operators needed to achieve its goal of implementing the combat-to-dwell policy in fiscal year 2024. Specifically, in a February 2018 briefing to Congress, the Air Force stated it planned to fully implement the combat-to-dwell policy in fiscal year 2024. Headquarters Air Force officials stated that in order to meet this 2024 goal, the Air Force is working to increase the number of trained RPA pilots and sensor operators through its accession, training, and retention efforts because they said it cannot implement the combat- to-dwell policy if it lacks sufficient quantities of available personnel. Several senior leaders at each of the locations we visited discussed the importance of achieving and sustaining a sufficient level of staffing that is needed to implement the dwell policy. One senior leader emphasized that the Air Force made “getting to dwell” its cornerstone promise. Officials stated that pilots and sensor operators are currently only able to accomplish training that can be done while completing combat missions because the RPA personnel are currently flying 24/7 combat missions. The January 2017 combat-to-dwell policy emphasized the need for the implementation of dwell time within the RPA community to allow these units to focus on either combat operations or training, but not both at the same time. This policy states that it is essential for preventing future risk to the mission and preserving the combat capability of the RPA force. Headquarters Air Force officials stated that they were hopeful that implementing the combat-to-dwell policy would improve quality of life and reduce burnout among RPA personnel by allowing them to take a break from combat operations to give them time to rest and train. Officials acknowledged that poor quality of life conditions for RPA personnel negatively affects retention. According to an Air Force instruction related to the RPA community, it is important to build a sustainable and healthy force and retention affects virtually all aspects of the Air Force’s effort to meet its goal of attaining the proper number of aircrew personnel. Further, it states that understanding the connection between the accession of new recruits, the training and production requirements of new aircrew members, and the ability of units to absorb newly trained aircrews into the structure and operations of the forces is critical to maintaining a healthy aircrew force and to achieve Air Force goals. However, the Air Force does not know its overall progress toward achieving its goal of having sufficient quantities of RPA pilots and sensor operators to implement the combat-to-dwell policy in fiscal year 2024 as planned. Thus far, Headquarters Air Force officials said that the Air Force has been focused on retaining as many RPA pilots and sensor operators as possible in an effort to meet the increasing staffing authorizations. The Standards for Internal Control in the Federal Government states that management should track achievements and actual performance, compare to plans, goals and objectives and analyze significant differences. Specifically, officials explained that it does not have a comprehensive metric (or set of metrics) which allows them to track changes in the number of its RPA pilots and sensor operators from its combined accession and retention efforts over a projected timeline. This prevents the Air Force from being able to compare its progress against its goal of having sufficient numbers of RPA pilots and sensor operators to fully implement the policy as planned by fiscal year 2024. The Air Force RPA officials stated that the Air Force does not have a metric (or set of metrics) that measures a “glide path to health and stability of the RPA workforce” by balancing both accessions and retention of RPA personnel in order to know when changes might be needed over time to achieve the goal of implementing the combat-to-dwell policy. Without such a metric (or a set of metrics), it is unclear whether the Air Force is on track to have enough RPA pilots and sensor operators to achieve implementation of its combat-to-dwell policy or to know if adjustments are needed to its accession and retention efforts or to the policy’s implementation timeframe. Taking such action is critical for the Air Force to be able to position itself to address long-standing RPA pilot and sensor operator shortages and documented challenges in the management of these communities through its combat-to-dwell policy. Absent such action, a key component of the Air Force’s workforce will not be well-positioned to meet its mission for the nation. The number of active-duty RPA pilot and sensor operator instructor positions required at the Holloman formal training unit are understated and do not reflect the current training instructor needs. More specifically, the number of instructor positions needed were developed using a 2009 program of instruction with a length of 49 training days and were never updated to reflect changes to the syllabus length, which as of July 2019, was 83 training days. Air Force documentation showed that if 100 percent of the formal training unit’s currently identified active-duty instructor positions were filled, they could provide only 47 percent of the total course instruction currently identified. To provide the rest of the course instruction, the formal training unit relies heavily on contractors. Air Force information shows that, as of July 2019, contractors provided 53 percent of instruction, active-duty personnel provided 27 percent, and 20 percent remained unaccomplished (i.e., not provided). The Standards for Internal Control in the Federal Government states that management should use quality information to make informed decisions to achieve its objectives. Quality information is, among other things, current, complete, and accurate. Further, a 2017 report to Congress on the implementation progress of the Air Force’s actions to ensure a sustainable RPA operational force stated having maximum instructor staffing was critical to generating new RPA pilots. However, the Air Force continues to use the out-of-date, inaccurate, and incomplete number of active-duty RPA pilot and sensor operator instructor position requirements that were originally developed based on the 2009 program of instruction. Without using quality information, the Air Force does not fully know the number of active-duty RPA pilot and sensor operator instructor positions necessary for sufficiently training RPA aircrews. As such, it may not be fully addressing the challenges affecting the training unit’s staffing and ability to produce the needed number of aircrews to support the continued demand for RPAs and the implementation of its combat-to-dwell policy as planned. Since fiscal year 2016, the Holloman formal training unit has been unable to meet the authorized instructor position staffing levels even though the numbers of those positions are based on an out-of-date number of training days from the 2009 program of instruction that underestimates actual instructor requirements. In 2015, top senior Air Force leaders developed the Get Well Plan, and the Secretary of the Air Force and other top senior leadership helped develop the plan’s two goals to staff 100 percent of the positions for (1) instructors at the RPA pilot school and (2) combat RPA pilots. In the March 2017 report to Congress, the Air Force again emphasized that maximum instructor staffing was critical to generating new RPA pilots and that it had achieved this goal as planned and it would stabilize and sustain the Get Well Plan’s goals into the future. We found that both the number of RPA pilot and sensor operator instructors assigned peaked at the end of 2016 and early 2017 in accordance with this Air Force goal. However, the assigned numbers of both RPA pilot and sensor operator instructors have not stabilized or been sustained and have fallen since that time as shown in figures 8 and 9. Specifically, authorized RPA pilot instructor positions within the three RPA training squadrons at Holloman Air Force Base (i.e., the 6th, 9th, and the 29th squadrons) were filled at 75 percent (110 of 147) as of September 2019. That fill rate is almost 20 percent less than the highest fill rate for these positions in March 2017 (137 of 147, or 93 percent). Similarly, authorized RPA sensor operator instructor positions within these same training squadrons as of September 2019 were filled at 58 percent (82 of 141), down from the highest fill rate of 91 percent (128 of 141) in November 2016. A training official explained that the inability to maintain the level of staffing, even when considering it was an underestimation of the true requirement, is an example of the issues experienced in the RPA community. He stated that when RPA pilots and sensor operators at squadrons leave the Air Force that means there are fewer of them overall available to conduct the missions and to be sent to the formal training unit to serve as instructors. Fewer instructors at the training unit means a greater workload on the instructors already there, which affects the morale of the instructors and may result in those individuals leaving the Air Force. It also limits the ability of the formal training unit to meet the expectations of producing newly trained aircrews that are supposed to fill the staffing need at the squadrons. Overall, this cycle contributes to the challenge the Air Force faces in being able to retain and produce RPA pilots and sensor operators. Moreover, the gap in instructor staffing is compounded by a majority of instructors arriving at the Holloman formal training unit not having prior operational squadron-level instructor experience, according to training officials. According to an Air Force instruction regarding RPA training, any aircrew member designated for instructor duties at a formal training unit should already be an instructor in the applicable aircraft. However, for example, at Holloman’s formal training unit, officials told us that for the training session from August 2019 to May 2020, 17 of 25 of the new incoming instructors did not have previous squadron-level instructor or evaluator experience. In these instances, they said the new instructors would need additional training to qualify them fully to teach certain classes. According to training officials, being an instructor at a formal training unit is not the same as being an instructor at an operational squadron. For example, in an operational squadron, an instructor is expected to take an individual that is fully qualified in the aircraft and get them up to speed on the squadron’s specific mission and to assist in increasing the squadron’s overall level of efficiency through continued supervised training. At the formal training unit, however, instructors are laying the foundation for new aircrew students that are not familiar with the aircraft, its operation, or its various mission sets. Officials stated that because the formal training unit is receiving inexperienced instructors rather than fully qualified ones, the training unit must provide more upgrade training to these student instructors to qualify them to teach any classes. While the instructors are going through the upgrade and any other training needed to become fully qualified, they are filling an instructor staff position but not fully contributing to the development of new RPA pilots or sensor operators. Air Force training officials acknowledge that staffing at its Holloman formal training unit is a concern and that they need more instructors. They said that shortening the length of training was one approach to addressing the instructor gap and, in June 2019, the commander of the 19th Air Force (Air Education and Training Command) directed syllabus modifications. According to training officials, the modifications suspended about 15 percent of the training and thereby, shortened the length of the course. These modifications are scheduled until the end of October 2020 unless deemed necessary to extend them into fiscal year 2021. In 2015, the Air Force developed over 140 initiatives to address quality of life challenges facing its RPA units but has not fully implemented them. While the Air Force has been aware that the RPA community faces such issues as work-related physical and mental ailments, lack of base services, and other challenges to its quality of life, long-standing concerns we have identified previously, as well as others, remain. The Air Force’s Air Combat Command established the Culture and Process Improvement Program (CPIP) in 2015 to identify and address stress and quality of life issues within the Air Force’s MQ-1 Predator and MQ-9 Reaper RPA communities. This effort collected nearly 2,500 inputs from the RPA community through surveys and in-person engagement. Following this input, the Air Force developed over 140 initiatives to address concerns in eight different areas, such as missions, quality of life, locations and basing options, and training. These initiatives varied widely in scope and specificity and they addressed the RPA enterprise, such as pilots, sensor operators, intelligence personnel, and maintainers across active-duty personnel and the Reserve component. In February 2018, the Air Force briefed Congress, reporting that 57 percent of CPIP initiatives were complete and 43 percent were ongoing. According to Air Force officials, examples of initiatives completed include: expanding RPA combat operations to Shaw Air Force Base, South Carolina, to provide additional assignment options; establishing an advanced weapons instructor course specifically for redesignating MQ-9 Reaper RPA squadrons from “Reconnaissance” to “Attack;” establishing a medal to specifically recognize the contributions of personnel that operate and support the RPA enterprise; and, authorizing RPA aircrews to log combat time when flying an aircraft within designated hostile airspace, regardless of the aircrew’s physical locations. The CPIP report finalized just over a year later in June 2019 states that the Air Force had achieved “an almost 90 percent solution” and the most significant of the initiatives had been accomplished. It went on to say that there were 17 initiatives remaining open at that time and that the Air Force would no longer track those initiatives because they had reached the point of diminishing returns. Additionally, the office established to track the CPIP initiatives was closed because Air Combat Command officials told us that the office is no longer needed and all remaining initiatives have been staffed to other offices of primary responsibility. However, in our review, we found examples of quality of life initiatives labeled complete where the objective had not yet been fully achieved. Examples we found include: an initiative to create a new MQ-9 RPA wing to be led by an RPA pilot was labeled with a status of “complete” even though Headquarters Air Force officials confirmed that no new MQ-9 Reaper RPA wing has yet been created; an initiative to have aircrews’ shiftwork schedules rotate every 4 to 6 months; however, each of the squadrons at the RPA operational bases we visited had a shift work schedule that rotated for 5 to 8 weeks; an initiative to grant appropriate clearances to allow medical and chaplain personnel into all RPA operational areas; however, at one location we visited, medical officials and a chaplain we spoke with said that they do not have the required clearance levels to meet with RPA personnel within their secured facilities; two initiatives to improve spousal opportunities, although one vaguely stated that the “Air Force should think big and think flexible as it needs to consider society’s shift to the two-income family” and the other called for providing better family services and support. However, we found that while these services may exist at RPA bases, they are not always accessible to RPA personnel or their families for a variety of reasons, as we discuss below; an initiative to provide childcare support for workers performing 24/7 operations, although we found childcare was not available at certain facilities we visited; and, an initiative to make Creech Air Force Base its own installation, add a Missions Support Group, and improve base infrastructure and services. Creech did receive its own command authority and is no longer an auxiliary facility under Nellis Air Force Base and a Mission Support Group was established in July 2019. However, its plans to create officer and non-commissioned officer housing and an additionally medical facility are not expected to be completed until between fall 2021 and fall 2022, according to a Creech official. According to Air Force officials, an initiative marked as “complete” means that the Air Combat Command CPIP office had completed its portion of the initiative and another Air Force entity had taken it over for further action as necessary and may still be in process. Therefore, the 57 percent of initiatives that the Air Force reported to Congress in February 2018 as completed and the “almost 90 percent solution” discussed in the June 2019 CPIP final report may not present a transparent account of what has been completed and what remains to be accomplished. Reporting planned tasks as “complete” as the Air Force did could create perception gaps regarding the effects of CPIP. Interviews we had with senior leaders at multiple bases yielded concerns that CPIP is effectively over without accomplishing key objectives and that CPIP is going to be perceived as a failed promise by the Air Force. Along with the CPIP initiatives developed in 2015 as discussed above, academic studies published since 2010 and our previous 2014 report on RPA job dissatisfaction identified challenges facing the RPA community. For example, in April 2011, a study by researchers at the U.S. Air Force School of Aerospace Medicine found that there are several important operational stressors to consider when assessing the health and well- being of RPA operators. More specifically, the researchers noted, for many operators that participated in the study, the most commonly cited stressors associated with occupational stress included, but not limited to, the following: (1) long hours and low manning; (2) frequently changing shift work and shift changes; (3) geographically undesirable locations; (4) limited base resources and rural settings; and (5) human-machine interface difficulties such as poor ergonomics and temperature control of work stations. The study concluded that it stood to reason such stressors could lead to both physical and psychological distress when faced on an unending basis. Three years after the issuance of that study, in April 2014, we reported that RPA pilots faced multiple, challenging working conditions, including work shifts that frequently rotate, long hours, and increased workloads. More specifically, we reported in 2014 that In seven of the 10 focus groups conducted at that time, RPA pilots said continuously rotating to new shifts disrupted their ability to spend time with their family and friends and caused sleep problems. They said that these changes to their sleep schedules resulted in significant fatigue both at home and when they returned to work. In seven of the 10 focus groups conducted at that time, RPA pilots described working long hours because, for example, they had to perform administrative duties and attend briefings in addition to flying their combat shifts. High work demands on RPA pilots limit the time they have available for training and development and negatively affects their work-life balance. During the course of our current review, we heard various positive comments about how RPA pilots or sensor operators like the RPA mission and being able to contribute on a daily basis to combat operations. However, as discussed below, we also found examples of how long-standing challenges that others and we reported about years ago regarding the physical and mental health of RPA personnel and the availability of base support services continue to exist. Shift Work and Sleep Issues In 12 of the 14 focus groups we conducted, participants stated that the frequent rotations are a key challenge of shift work and that their schedules rotated approximately every 5 to 8 weeks. However, members of the Human Performance Team at Creech Air Force Base stated studies have shown that it is better for individuals to stay on shifts for longer periods of time, such as 3 to 4 months, to allow their circadian rhythms to adjust. Additionally, focus group participants told us that rotating shift work is difficult for RPA personnel’s relationships. Participants in 13 of the 14 focus groups indicated that shift work has negatively affected their family or social life. Additionally, rotating shifts and the limited time with family creates a dilemma on weekends for personnel, especially for those on the midnight shift that covers roughly midnight to 8 a.m. These individuals must decide whether to maintain their work sleep schedule which limits time with family, or instead to align with their family’s sleep schedule which limits their ability to adapt to the work schedule. Some comments from participants include “I destroy my circadian rhythm to spend time with my kids” and “Shift work is disruptive to lives. It is hard to be tied into the community. Shift work can be really isolating.” Crew rest is compulsory for aircrew members prior to performing any aircraft operations. Aircrew members are individually responsible to ensure they obtain sufficient rest during a crew rest period. If crew rest is interrupted, individuals should immediately inform appropriate leadership and will either begin a new crew rest period or not perform flight duties. According to health officials at one of the bases, though, it is well known that RPA aircrew members often do not accurately report how much rest they get. Participants in one focus group agreed with this statement and said that they do not want to be restricted from flying and affect the mission and cause the work to fall on other squadron members. Participants in 12 of our 14 focus groups that we conducted stated that it is difficult to get adequate sleep. Sample participant comments include: “I can’t sleep anymore. Before the military, I could get 10 hours of sleep. Now it’s like 2-4. You’re physically and mentally exhausted.” “I feel perpetually tired. I haven’t felt healthy in years.” “We did an internal survey of how much sleep people on nights for months at a time were getting, and it was like 3-4 hours. And they are flying combat for 8-12 hours at a time.” Back, Eyes, and Other Physical Issues In 12 of 14 focus groups, participants said the working environment is harmful to health in areas such as the neck, back, eye, and hearing. Participant comments included: “I’ve been losing hearing …over the last 6 years from computer fans, air conditioning units, the use of multiple communication devices, etc.” “Just sitting in the seat for 8, 10, or 12 hours affects our posture. It is bad on our backs. I didn’t have lower back problems, and I work out a lot, but I started having lower back problems.” “My eyesight has been getting worse.” See figure 10 for an example of a pilot flying a simulated mission in an RPA cockpit. During our site visits for this review, participants in 14 focus groups that we conducted said that maintaining fitness was difficult. They said they are not motivated to work out as they are frequently exhausted after flying long shifts and then completing other extra duties as well. Further, participants in 11 of 14 focus groups told us that nutrition is difficult for RPA crews. For example, participants said that they consume energy drinks, soda, and sugary foods to stay awake during the midnight shift. Studies have shown negative psychological effects on RPA aircrews. An Air Force study from 2010 of the psychological attributes critical to the performance of RPA sensor operators noted it is important that RPA sensor operators be aware prior to training that they would be targeting and destroying enemy combatants. It stated that it was likely that some candidates might choose not to become sensor operators once they fully understand their role in precision-strike operations. These motivational attributes were not deemed critical to performance, but were deemed critical to retention and job satisfaction. Participants in 10 of our 14 focus groups we conducted said that some crew members—either themselves or others—did not initially understand what the job entails, such as killing. One focus group participant noted “the first time you know what you’re getting into emotionally is the first day of training at Holloman, which is too late because you already have wings.” Participants in 13 of 14 focus groups we conducted stated that witnessing or causing violence has a negative psychological impact but two-thirds of our survey respondents (66 of 105) said that the Air Force has not assessed their level of stress and fatigue related to their role as an RPA pilot or sensor operator. A study published in 2018 described how RPA aircrew members are affected by their own actions in combat as well as by connections with either people who they target or support on the ground regardless of the physical distance separating them. One focus group participant commented “F-16s drop and then go. For RPA aircrews, we get in and we are there for 20 hours. We watch who we employ weapons on, then get the battle damage assessment, including seeing body parts…on the ground.” RPA personnel stated that their base’s services are not consistently available to RPA aircrews rotating shifts to conduct missions 24 hours every day or to their families as they live in remote locations. Collectively, participants in all 14 focus groups we conducted expressed concerns about the availability of services such as medical services, childcare, spouse and family support services, and base locations and housing. Some level of health care is provided at each RPA base we visited, but the extent to which these services are available varies. For example, the Cannon Air Force Base mission briefing we received in June 2019 noted some “sustainability challenges” such as the base’s inadequate availability of specialty medical care. The briefing noted that the base had made over 2,000 referrals related to 10 areas of specialty medical care. Additionally, because these referrals were to facilities outside the local area, the base had incurred about $500,000 in travel reimbursements for this medical care—the highest of all Air Force locations—and about $21 million in TRICARE expenses per year, according to officials. Further, we found examples during our site visits of health services without adequate staffing. For example, during our visit to Shaw Air Force Base in May 2019, a medical technician stated that Shaw had two medical technicians for the RPA community though staffing documents state they are supposed to have six medical technicians and two doctors. At Creech Air Force Base, we visited the medical and dental facility and learned that a psychologist position had been unfilled for 9 months as of our visit in August 2019. We also found that the hours of available medical services are limited and not convenient for shift workers such as RPA aircrews. For example, officials at Creech stated occupational therapy is offered only once a month, optometry twice a month, and nutrition on an as-needed basis. In addition, Creech has two family health personnel, a behavioral health officer who is available every Wednesday and Friday, and one flight surgeon who comes over from Nellis Air Force Base is available twice a week. A 2018 internal assessment done for Creech leadership estimated that 20,714 man-hours are wasted each year due to personnel needing to obtain medical services, the equivalent of losing 11.5 people in a given year. To address health issues, Creech Air Force Base has a Human Performance Team that includes chaplains, religious affairs airmen, a psychologist, a mental health tech, and a physiologist. While team members are physically located at Creech, they told us that they are also responsible for RPA units at all the bases under the same wing, including Creech, Ellsworth Air Force Base, South Dakota; Whiteman Air Force Base, Missouri; and Shaw Air Force Base, South Carolina. Further, at Shaw Air Force Base, a religious affairs airman made similar comments about serving a large variety of military personnel, not just the RPA community and a chaplain at Cannon Air Force Base said that he can be assigned responsibility for up to as many as 2,000 to 3,000 people at a time. Childcare is not limited for 24/7 shift workers at certain facilities although a CPIP initiative called for childcare support for workers performing 24/7 operations, citing the Missile Care childcare program offered at Minot Air Force Base. To this end, the Air Force established two programs, RPA Care and RPA 2 Care. The RPA Care program provides additional care outside the normal work hours at no additional cost to members who are already purchasing full-time care from the Child Development Center. However, in 12 of 14 focus groups we conducted, participants said that they found childcare services were of low quality or limited for 24/7 shift workers. For example, Cannon Air Force Base has two Child Development Centers, but they operate Monday through Friday from 6 a.m.to 6 p.m., and focus group participants noted a long waiting list for admission. At Creech Air Force Base, there is no childcare on base and at Shaw Air Force Base, participants said it was difficult finding available childcare to aid RPA personnel working shiftwork. For example, one RPA aircrew member was permanently assigned to the day shift because of childcare issues. Spouse and Family Support Issues RPA personnel have complained about the issues associated with working at remote location, such as the Creech Air Force Base, Nevada, and Cannon Air Force Base, New Mexico, locations. In 9 of 14 focus groups, participants made various comments regarding the limited spousal opportunities and family support issues such as the following: “I got orders to Cannon…. The problem is I’ll be bringing my wife there who has no job opportunities. There will be a lot of military spouses competing for jobs. I’ve already decided I’ll leave at the end of my contract and then will go to the Guard. I’ve told my wife I’ll get out because I don’t want to hurt her quality of life.” “I loved the mission at Cannon, but the facilities and area and schools are absolutely terrible.” “I’m fed up with Cannon and this area in general.” RPA bases vary in housing available for personnel with Cannon and Creech Air Force Bases reporting inadequate housing situations. At Cannon, officials stated that lack of dormitory space was forcing first-term Airmen off base. During our visit in June 2019, Cannon housing officials provided a report that stated that the shortfall in dormitory space continues to put Airmen and the Air Force Special Operations Command mission at risk. The report said that the locations off base where first-term Airmen can afford to live are usually in the worst crime-ridden parts where there is a far greater propensity for trouble. This can create morale issues and a distraction from the mission, according to the report. Additionally, Creech Air Force Base does not have any permanent on- base housing. At Creech, unaccompanied first-term Airmen must live in the dormitories on Nellis Air Force Base, which is approximately 50 miles away. The remoteness of Creech Air Force Base and the lack of basic services offered only at Nellis Air Force Base creates an unusual level of stress brought on by the added time, effort, and expense Creech Airmen experience that those at almost every other continental United States installation do not. In fact, a 2018 internal assessment for Creech leadership calculated that a junior airman who must live at Nellis Air Force Base would have a one-way commuting time of 63 minutes if they drive a personal vehicle or 105 minutes if they take the shuttle. To help address the housing and access to medical facilities, Creech Air Force Base senior officials said that a plan to create officer and non- commissioned officer housing and a medical facility on the northwest side of Las Vegas has been approved, but it is not expected to be completed until between fall 2021 and fall 2022. Many of the RPA workforce issues we identified at the time of our 2014 review continue to exist today. These workforce issues include the challenges to the RPA workforce’s quality of life due to stressful working conditions, including work shifts that frequently rotate, long hours, and increased workloads. In 2017, we recommended that the Air Force should monitor the extent to which its RPA human capital efforts are achieving the Air Force’s overall programmatic goals. The Air Force had not implemented this recommendation as of February 2020. Because long- standing RPA quality of life and workforce management issues affecting RPA personnel continue to exist, we believe that this recommendation is still valid and would aid the Air Force in its efforts to address many of the challenges facing this career field. Therefore, we are not making any additional quality of life related recommendations. A healthy RPA workforce is one that balances supply with demand and addresses quality of life conditions to motivate and sustain performance and retention. Successful efforts to assess, train and retain RPA pilots and sensor operators would allow the Air Force to grow sufficient quantities of its RPA workforce to meet its goal of implementing its combat-to-dwell policy. While the total number of Air Force RPA pilots and sensor operators has increased between 2015 and 2019, the number of positions required to meet the constant demand is increasing at a faster pace. Additionally, the Air Force has not achieved its accession targets for pilots and sensor operators for most of those years. Moreover, the inability to use standard retention metrics due to the newness of the RPA pilot career field is hindering the Air Force’s ability to determine accurately if sufficient quantities of RPA personnel are remaining in the service to grow its RPA workforce. Further, the Air Force currently does not have a comprehensive metric (or set of metrics) to track the overall progress toward having sufficient numbers of RPA personnel through its accessions and retention of RPA personnel to meet its prescribed timeline for implementing its combat-to-dwell policy. This policy is intended to balance the time RPA units spend in combat with non-combat activities, to provide relief from those combat operations that it has conducted constantly for many years, to improve the quality of life of these RPA aircrew members. Without a metric, it is unclear whether the Air Force is on course to achieve implementation of its combat-to-dwell policy. As such, the Air Force cannot know if adjustments are needed specifically to that policy and its implementation timeline or to its overall personnel management efforts to access, train and retain sufficient numbers of RPA personnel. Further, the Air Force previously prioritized having maximum instructor staffing at the training units to help increase the production of new RPA aircrews. However, the number of instructor positions required at the RPA formal training unit at Holloman Air Force Base is out-of-date and does not reflect what is needed to teach the current training curriculum. Additionally, this formal training unit has consistently experienced staffing shortages since fiscal year 2016. As such, without updated information, the Air Force does not know the number of instructor positions necessary for sufficiently training RPA aircrews and it may not fully address the challenges affecting the training unit’s staffing and ability to produce the needed number of aircrews to support the continued demand for RPAs and the implementation of the combat-to-dwell policy as planned. The Air Force developed initiatives with its 2015 Culture and Process Improvement Program to address quality of life issues and other challenges affecting the RPA community, but has not fully implemented them. We also identified workforce management challenges in our previous work. We believe that our prior recommendation that the Air Force monitor its human capital efforts would help address these challenges. We believe the Air Force should implement our prior recommendation to aid the Air Force in its attempts to improve the quality of life issues that still exist within the RPA community. We are making the following two recommendations to the Secretary of the Air Force. The Secretary of the Air Force should ensure that a comprehensive metric (or set of metrics) is established to track the progress of its combined accession and retention efforts to obtain sufficient quantities of RPA pilots and sensor operators needed to achieve its objective of implementing the combat-to-dwell policy as planned. (Recommendation 1) The Secretary of the Air Force should ensure that the number of instructor positions needed at the RPA training unit at Holloman Air Force Base is updated by applying more complete, accurate and timely information to better reflect the training curriculum and instructor needs. (Recommendation 2) We provided a draft of this report to DOD for review and comment. In written comments reproduced in appendix III, the Department of the Air Force partially concurred with our first recommendation and concurred with our second recommendation. In concurring with our second recommendation to ensure the number of instructor positions needed at the RPA training unit at Holloman Air Force Base is updated, the Air Force noted that it has requested an updated study to determine the appropriate number of instructor positions. With regard to our first recommendation—to establish a comprehensive metric (or set of metrics) to track the progress of its combined accession and retention efforts—the Air Force noted that it already has efforts to monitor accession, production, and retention for RPA pilots and sensor operators. Additionally, it expects that standard retention metrics used in other rated career fields will provide increased utility as the RPA career field matures. The Air Force acknowledges in its comments, however, that these efforts could be better integrated to allow for greater analysis, to include tracking progress in meeting the combat-to-dwell policy by 2024. We continue to believe that in developing a specific metric (or set of metrics) the Air Force would be in a better position to evaluate the status of its combined accession and retention efforts to obtain the proper number of RPA personnel to achieve its combat-to-dwell implementation goal. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Secretary of the Air Force. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov . If you or your staff have any questions regarding this report, please contact me at (202) 512-3604 or farrellb@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV. To obtain the perspectives of Air Force remotely piloted aircraft (RPA) pilots and sensor operators regarding training, availability of services and support to RPA personnel and their families; quality of life issues; retention issues; and other challenges facing the RPA career field, we analyzed participants’ comments from 14 focus groups at three different RPA operational locations. These locations were: Shaw Air Force Base, South Carolina; Cannon Air Force Base, New Mexico; and Creech Air Force Base, Nevada. We selected Cannon and Creech Air Force Bases because they have the largest population of RPA operators in Air Force Special Operations Command and Air Combat Command, respectively. In addition, we selected Shaw Air Force Base to obtain the perspectives of RPA pilots and sensor operators working at a base with newly established RPA operations since 2018. To obtain a balance of perspectives from RPA pilots and sensor operators with varying levels of experience and responsibilities, we conducted focus group sessions with active-duty MQ-9 Reaper RPA pilots and sensor operators who were divided by their occupation, Air Force Specialty Code, and rank at the selected locations. Specifically, we used the following categories as shown in table 3 for the formation of the focus groups. The 14 focus groups we held ranged in size from five to 11 participants across the three sites with 105 total participants. We conducted five focus groups at Shaw Air Force Base; four focus groups at Cannon Air Force Base; and five focus groups at Creech Air Force Base. Of the 14 focus groups, eight focus groups were with RPA pilots and six focus groups were with RPA sensor operators. These sessions involved structured small-group discussions designed to gather in-depth information that is not easily obtained from other methods. We requested that our point of contact at each location gather approximately 8 to 12 participants to attend the five pre-defined focus groups. We conducted focus groups with RPA pilots and sensor operators separately because they have different roles and responsibilities and to encourage active participation and minimize the risk of participants being the same group as immediate supervisors. We segmented our groups by this characteristic in order to compare and contrast their perspectives on training, retention, and quality of life issues and to identify meaningful similarities and differences. Participants in the focus groups were not randomly selected by using a probability sampling method, but recruited by unit leadership based on shift availability and correspondence with the characteristics we requested. Because scheduling availability was the primary factor affecting participation, coupled with the fact that questions for focus group sessions were not shared in advance, we considered the risk of leadership selectively picking participants to be minimal. Methodologically, focus groups are not designed to (1) demonstrate the extent of a problem or to generalize results to a larger population, (2) develop a consensus to arrive at an agreed-upon plan or make decisions about what actions to take, or (3) provide statistically representative samples or reliable quantitative estimates. Instead, they are intended to generate in-depth information about the reasons for the focus group participants’ attitudes on specific topics and to offer insights into their concerns about and support for an issue. A facilitator who used a standard script and list of questions to guide the discussion and encourage participants guided the focus group participants to share their thoughts and experiences. We confirmed at the start of each session that participants met the inclusion criteria for the respective group. Due to the low numbers of 18X pilot participants at the O3-O5 rank and 11U/12U pilot participants at Cannon Air Force Base, we conducted a focus group of the available participants together instead of separately. Additionally, at Creech Air Force Base, we encountered three situations where participants were currently full-time Reserve pilots, but because all had former active-duty experience and dismissing them would result in too few participants in the group, we allowed them to stay in the focus groups in order to have a sufficient number of participants. This situation occurred in the O1-O2 18X pilot focus group, the O3-O5 18X pilot focus group, and the E5-E9 1U0XX sensor operator focus group. The core questions that the GAO facilitator asked during each of the focus groups are listed in table 4. During the focus group meetings, three GAO members independently took separate sets of detailed notes to document the participants’ comments. Afterward, each member’s notes were compiled into one final official record documenting the comments made in each of the focus groups we conducted. Then, these records were consolidated into one database to be used for coding each comment and to facilitate the team’s content analysis of all the comments. To identify common categories and themes from the participants’ comments across all focus groups, the team met, reviewed and discussed the official record of each of the 14 focus groups. From that meeting, the team identified 43 categories across seven areas of inquiry; see table 5 for a list of the categories and themes. Using the categories and themes identified, the team conducted a pre- test by having two groups of two coders independently code an identical subset of the comments to determine their levels of coding consistency and accuracy before attempting to code all 1,848 individual recorded comments. After the pretest, the two groups split the list of comments in half and each coder independently coded the comments contained in their list into the categories and themes under which the coder believed the comment fell. Once completed, the coders within each group met to discuss any discrepancies in each of their coding and to make any necessary adjustments in the coding. Where discrepancies could not be resolved between coders, an independent third team member determined which code would be used. Once the coding of all 1,848 comments was finalized, the team’s methodologist prepared a report that presented all comments that fell within each of the categories and themes. The team used this information as the basis for frequency tabulation and qualitative analysis of focus group comments. In addition to discussing the RPA pilots’ and sensor operators’ perspectives in a focus group setting, we administered a questionnaire to each participant at the end of each session before the participants were dismissed. All participants completed the questionnaire. A GAO methodologist with a social science background and knowledge of small group methods and survey administrations reviewed the focus group script and the questionnaire. In addition, we pre-tested both the focus group protocol and the questionnaire on our first site visit to Shaw Air Force Base and both were used again at the remaining RPA locations, Cannon and Creech Air Force Bases, without any changes. The Department of Defense (DOD), the military services, and organizations outside DOD have produced reports and studies that addressed issues associated with Air Force remotely piloted aircraft (RPA) personnel, including the following: Armour, Cherie, and Jana Ross. “The Health and Well-Being of Military Drone Operators and Intelligence Analysts: A Systematic Review.” Military Psychology, 2017. Bryan, Craig J., Tanya Goodman, Wayne Chappelle, Lillian Prince, and William Thompson. “Subtypes of severe psychological distress among US Air Force remote warriors: A latent class analysis.” Military Psychology, 2018. Campo, Joseph L. “Distance in War: The Experience of MQ-1 and MQ-9 Aircrew.” Air and Space Power Journal, 2015. Chappelle, Wayne L., Kent McDonald, Lillian Prince, Tanya Goodman, Bobbie N. Ray-Sannerud, and William Thompson. “Symptoms of Psychological Distress and Post-Traumatic Stress Disorder in United States Air Force “Drone” Operators.” Military Medicine, 2014. Chappelle, Wayne, Emily Skinner, Tanya Goodman, Julie Swearingen, and Lillian Prince. “Emotional reactions to killing in remotely piloted aircraft crewmembers during and following weapon strikes.” Military Behavioral Health, 2018. Chappelle, Wayne, Julie Swearingen, Tanya Goodman, Sara Cowper, Lillian Prince, and William Thompson. Occupational Health Screenings of U.S. Air Force Remotely Piloted Aircraft (Drone) Operators. Report, Wright-Patterson Air Force Base, OH: Air Force Research Laboratory, 2014. Chappelle, Wayne, Kent McDonald, and Raymond King. Psychological Attributes Critical to the Performance of MQ-1 Predator and MQ-9 Reaper U.S. Air Force Sensor Operators. Report, Brooks City-Base, TX: Air Force Research Laboratory, 2010. Chappelle, Wayne, Kent McDonald, Billy Thompson, and Julie Swearangen. Prevalence of High Emotional Distress and Symptoms of Post-Traumatic Stress Disorder in U.S. Air Force Active Duty Remotely Piloted Aircraft Operators (2010 USAFSAM Survey Results). Report, Wright-Patterson Air Force Base, OH: Air Force Research Laboratory, 2012. Chappelle, Wayne, Kent McDonald, Lillian Prince, Tanya Goodman, Bobbie N. Ray-Sannerud, and William Thompson. “Assessment of Occupational Burnout in United States Air Force Predator/Reaper “Drone” Operators.” Military Psychology, 2014. Chappelle, Wayne, Tanya Goodman, Laura Reardon, and Lillian Prince. “Combat and operational risk factors for post-traumatic stress disorder symptom criteria among United States Air Force remotely piloted aircraft “Drone” warfighters.” Journal of Anxiety Disorders, 2019. Chappelle, Wayne, Tanya Goodman, Laura Reardon, and William Thompson. “An analysis of post-traumatic stress symptoms in United States Air Force drone operators.” Journal of Anxiety Disorders, 2014. Cooke, Nancy J., Kristen Barrera, Howard Weiss, and Claude Ezzell. “Psychosocial Effects of Remote Operations.” In Remotely Piloted Aircraft Systems: A Human Systems Integration Perspective, by Nancy J. Cooke, Leah J. Rowe, Winston Bennett, Jr. and DeForest Q. Joralmon. West Sussex: John Wiley & Sons, 2017. Goodman, Tanya, Lillian Prince, Wayne Chappelle, and Craig Bryan. A Reassessment of Risk Factors and Frequency of Suicide Ideation Among U.S. Air Force Remote Warriors. Report, Wright-Patterson AFB, OH: Air Force Research Laboratory, 2018. Hardison, Chaitra M., Eyal Aharoni, Christopher Larson, Steven Trochlil, and Alexander C. Hou. Stress and Dissatisfaction in the Air Force’s Remotely Piloted Aircraft Community. Santa Monica, CA: RAND Corporation, 2017. Hijazi, Alaa, Christopher J. Ferguson, Harold Hall, Mark Hovee, F. Richard Ferraro, and Sherrie Wilcox. “Psychological Dimensions of Drone Warfare.” Current Psychology, 2017. Martin, Kiel M., Daniel J. Richmond, and John G. Swisher. “Sustaining the Drone Enterprise: How Manpower Analysis Engendered Policy Reform in the United States Air Force.” INFORMS Journal on Applied Analytics, 2017. Martin, Matt. “Remote-Split Operations and Virtual Presence: Why the Air Force Uses Officer Pilots to Fly RPAs.” 18th International Symposium on Aviation Psychology. Dayton, 2015. Ouma, Joseph A., Wayne L. Chappelle, and Amber Salinas. Facets of Occupational Burnout Among U.S. Air Force Active Duty and National Guard/Reserve MQ-1 Predator and MQ-9 Reaper Operators. Report, Wright-Patterson Air Force Base, OH: Air Force Research Laboratory, 2011. Terry, Tara L., Chaitra M. Hardison, David Schulker, Alexander C. Hou, and Leslie Adrienne Payne. Building a Healthy MQ-1/9 RPA Pilot Community: Designing a Career Field Planning Tool. Santa Monica, CA: RAND Corporation, 2018. Wood, III, Joe, et al. Prevalence of Posttraumatic Stress Disorder in Remotely Piloted Aircraft Operators in the United States Air Force. Report, Wright-Patterson Air Force Base, OH: Air Force Research Laboratory, 2016. Wood, III, Joe D, et al. “Relationship Between Spiritual Well-being and Post-traumatic Stress Disorder Symptoms in United States Air Force Remotely Piloted Aircraft and Intelligence Personnel.” Military Medicine, 2018. Brenda S. Farrell, (202) 512-3604 or farrellb@gao.gov In addition to the contact named above, key contributors to this report were Lori Atkinson, Assistant Director; Rebecca Beale, Brad Crofford, Caitlin Cusati, Felicia Lopez, Terry Richardson, Ophelia Robinson, Pamela Snedden, and John Van Schaik. Unmanned Aerial Systems: Air Force Pilot Promotion Rates Have Increased but Oversight Process of Some Positions Could Be Enhanced. GAO-19-155. Washington D.C.: February 7, 2019. Unmanned Aerial Systems: Air Force and Army Should Improve Strategic Human Capital Planning for Pilot Workforces. GAO-17-53. Washington D.C.: January 31, 2017. Unmanned Aerial Systems: Actions Needed to Improve DOD Pilot Training. GAO-15-461. Washington, D.C.: May 14, 2015. Air Force: Actions Needed to Strengthen Management of Unmanned Aerial System Pilots. GAO-14-316. Washington, D.C.: April 10, 2014.", "summary": "High demand and constant combat operations have created challenges for Air Force RPA pilots and sensor operators who conduct missions across the world. In January 2017, the Air Force approved a combat-to-dwell policy to better balance RPA units' time in combat with non-combat activities. It plans to fully implement the policy in 2024. Senate Report 115-262 included a provision that GAO review ongoing challenges in the Air Force RPA community. This report assesses, among other things, the extent to which the Air Force (1) met overall RPA pilot and sensor operator staffing targets and tracked its progress in implementing its combat-to-dwell policy and (2) identified and met instructor staffing levels at its RPA formal training unit. GAO analyzed selected Air Force accession, retention, and instructor staffing data; held non-generalizable focus groups at three RPA military bases; and interviewed officials at various levels of the RPA enterprise. The Air Force does not have enough pilots and sensor operators to meet its staffing targets for its unmanned aircraft—also called remotely piloted aircraft (RPA). It also does not track its overall progress in accessing and retaining enough RPA personnel needed to implement its combat-to-dwell policy, which is intended to balance RPA units' time spent in combat with non-combat activities. Officials stated that to fully implement combat-to-dwell the Air Force needs to access and retain more RPA personnel because since fiscal year 2016 it has had fewer RPA personnel than authorized (see figure for RPA sensor operator example). The Air Force has provided financial incentives to address retention of RPA personnel, but it does not yet have enough historical data to help predict RPA pilot retention trends going forward given the newness of the career field. Officials additionally expressed specific concerns about sensor operator retention particularly due to the possibility of lucrative private-sector jobs. Further, the Air Force does not have a comprehensive metric (or set of metrics) to know whether its accession and retention efforts are on track to generate the additional RPA personnel needed to implement its combat-to-dwell policy by 2024. Without a metric (or set of metrics), it is unclear whether any adjustments are needed to meet its implementation timeframes. The Air Force has not fully identified the number of RPA pilot and sensor operator instructor positions needed at its formal training unit and since 2016 has experienced instructor staffing shortages. Specifically, the number of instructor positions required is understated because they are based on a 2009 program of instruction with 49 training days while the current program of instruction is 83 training days. Moreover, since fiscal year 2016, the formal training unit has had fewer assigned instructors than authorized positions even though those numbers of instructor positions are underestimates of actual needs. To help address the effect of the instructor gap, officials temporarily reduced the length of training. Without updated information to inform the number of required instructors, the Air Force does not know the correct number of instructor positions necessary to train RPA aircrews to be ready to complete their mission. GAO recommends that the Air Force establish a comprehensive metric (or set of metrics) to track the progress of its efforts to access and retain enough RPA personnel needed to implement its combat-to-dwell policy, and update the number of required RPA instructor positions. The Air Force partially concurred with the first recommendation and concurred with the second one. GAO continues to believe the first recommendation is valid, as discussed in the report.", "document_type": "gao"}
{"report": "Effective communication is vital for first responders’ ability to respond to emergencies, ensure the safety of both their personnel and the public, and protect public and private property. For example, first responders use public safety communications systems to gather information, coordinate a response, and, if needed, request resources and assistance from neighboring jurisdictions and the federal government. First responders use several types of communications systems, such as LMR systems, commercial wireless services, and the FirstNet network. LMR systems. These systems are the primary means for first responders to use voice communications to gather and share information while conducting their daily operations and coordinating their emergency response efforts. LMR systems are intended to provide secure, reliable voice communications in a variety of environments, scenarios, and emergencies. Commercial wireless services. Public safety entities often pay for commercial wireless services to send data transmissions such as location information, images, and video. FirstNet network. FirstNet is working to establish a nationwide, dedicated broadband network for public safety use that is intended to foster greater interoperability among first responders, support important voice and data transmissions, and meet public safety officials’ reliability needs on a priority basis, including call “preemption.” FirstNet’s network is intended to complement LMR systems with broadband capabilities and does not serve as a substitute for mission-critical voice needs. Communications systems must work together, or be interoperable, to ensure effective communication. Emergency communications interoperability refers to the ability of first responders and public safety officials to use their radios and other equipment to communicate with each other across agencies and jurisdictions when needed and as authorized. First responders’ LMR systems operate by transmitting voice communications through radio waves at specific frequencies and channels within the electromagnetic spectrum. FCC is responsible for allocating spectrum for various purposes and assigning spectrum licenses in a specific area and to a specific entity such as a police department or a telecommunications company. As previously noted, an auction is one mechanism that FCC may use to assign spectrum licenses. According to FCC officials, due to certain restrictions in the Communications Act, FCC has used administrative procedures, not auctions, to assign licenses for public safety and non-commercial educational broadcast stations. Over the years, spectrum for public safety has expanded to new frequency bands, as previously available frequencies became congested and public safety needs for spectrum increased. As we have previously reported, congestion results from growth in the overall number of users and demand for spectrum dependent technologies and services. Because of the increased demand for spectrum, in 1971 FCC authorized public safety and business-industrial users to share a portion of the T- Band spectrum (470 to 512 megahertz) with television broadcast stations in 11 metropolitan areas. The 11 metropolitan areas, which are identified in figure 1, include almost all the most populous metropolitan areas in the United States. The entire T-Band is not available for public safety and business users in these 11 metropolitan areas to build and operate LMR systems, and the amount of spectrum varies in each area. FCC rules allow “base station transmitters”—the equipment that emits radio signals to communicate with mobile units—to be located within 50 miles from the geographic center of each metropolitan area, as shown in figure 1. In 2012, as part of the Middle Class Tax Relief and Job Creation Act of 2012 (the Act), FCC was required by statute to reallocate the T-Band spectrum currently used by public safety and commence the process for an auction by February 22, 2021. As part of the reallocation of the T- Band for the 11 metropolitan areas listed above, the proceeds from the required auction shall be available to NTIA to make grants to cover relocation costs for the relocation of public safety entities. The grants are to be funded by the auction proceeds for the purpose of helping cover these users’ relocation costs. According to FCC officials, the Act does not address the hundreds of business-industrial users also using the T- Band and does not set aside or identify replacement spectrum for public safety users. DHS officials told us that the Act does not provide a formal role for DHS in the T-Band spectrum auction or relocation of public safety users. While one purpose of spectrum auctions is to recover the public portion of the value of spectrum, FCC officials told us that the Act and its legislative history do not explain the purpose of the T-Band auction and relocation, and we confirmed the absence of legislative history for the auction mandate. According to FCC officials, there are approximately 925 public safety entities with licenses in the T-Band. Each of these entities holds at least one license, but in some cases may hold many licenses. For example, the State of Texas holds one public safety license in the T-Band in the Houston metropolitan area, while the New York City Police Department has 180 licenses in the New York City metropolitan area. The number of licenses held by each entity depends on the demand for the spectrum for LMR systems and the availability of spectrum in other bands allocated for public safety use. FCC estimates that public safety entities have approximately 3,000 stations within the T-Band. Additionally, FCC said that the T-Band also contains approximately 700 business-industrial users that occupy about 1,700 stations. Public safety officials in three of our four selected metropolitan areas— Boston, Los Angeles, and New York City—told us that they have not been able to identify alternative spectrum to relocate from the T-Band, a situation that raises questions about the feasibility of the auction and relocation. For example, all of the officials we interviewed from New York City police, fire, and emergency management departments said there is no spectrum available for them to relocate to. The officials noted that the New York City Police Department is the largest municipal police department in the country and that it relies on the T-Band to dispatch police for 911 calls. Additionally an official from Pasadena in the Los Angeles metropolitan area said that the spectrum allocated for public safety in the region is already crowded and that officials are unsure of where to relocate their emergency communication operations. Public safety officials from Boston, Los Angeles, and New York City metropolitan areas also said that FCC has not provided a plan or identified alternative spectrum for relocation. In 2013, in anticipation of the mandatory T-Band auction, FCC published a notice and solicited public comment to gather information on when, how, and under what circumstances to relocate public safety and business-industrial users of the T-Band. At that time, FCC asked commenters what alternative spectrum bands were potentially available for relocation of T-Band’s public safety users, and whether these users could relocate to other public safety bands including the 700 and 800 MHz bands. In response to FCC’s request for comment, NPSTC conducted an analysis and reported in 2013 that the 11 different metropolitan areas would face different likelihoods of relocating to alternative spectrum. NPSTC analyzed FCC data on T-Band licenses to determine the number of public safety licenses that would need to be relocated, and then compared the need for licenses to the available licenses in other spectrum bands that FCC has allocated for public safety use. Based on that analysis NPSTC concluded the following. In five of the 11 metropolitan areas, relocating public safety users from the T-Band would not be possible. Specifically, in addition to identifying the three metropolitan areas we discuss above (Boston, Los Angeles and New York City), NPSTC concluded that at least two other metropolitan areas (Chicago and Philadelphia) lacked sufficient spectrum in any band to relocate public safety’s existing T-Band operations. For the other six metropolitan areas (Pittsburgh, San Francisco, Washington, D.C., Dallas-Fort Worth, Houston, and Miami) NPSTC’s analysis found that these areas might have sufficient spectrum to relocate T-Band users, with the 700 MHz narrowband offering the greatest potential. These metropolitan areas have fewer public safety T-Band licensees needing to relocate. Representatives from a trade organization that represents business-industrial users of the T-Band told us that in five of these six metropolitan areas, business-industrial users hold more than half of T-Band licenses. Specifically, the representatives noted that approximately 95 percent of T-Band users in the Houston metropolitan area are business-industrial users and that in Pittsburgh, Washington, D.C., Dallas-Fort Worth, and Miami metropolitan areas more than 50 percent of the T-Band users are business-industrial users. Our interviews with selected local officials confirmed that public safety users in Dallas-Fort Worth (our fourth selected metropolitan area) have had success transitioning off the T-Band. Two of the three public safety licensees we talked with told us they had already transitioned off the T- Band and noted that it was unrelated to the required T-Band auction. For example, an official from the City of Dallas, which holds one public safety license in the T-Band, told us that in 2012 the city began replacing existing radios with new radios that did not operate on the T-Band. The official said the city stopped operating on the T-Band in 2013 and relocated operations onto another spectrum band where most of the city’s public safety communications operated. Another T-Band public-safety licensee from the Dallas-Fort Worth metropolitan area told us that although it has active licenses they were unaware of the required auction or need to relocate from the T-Band. FCC and DHS officials told us the analysis conducted by NPSTC was a good source of information about the potential negative effects of the T- Band auction on public safety users, including numbers related to licensing and potential cost. DHS officials told us that NPSTC has broad expertise in emergency communications, noting that it is a member of two federally supported organizations that promote the interoperability of emergency communications—the Public Safety Advisory Committee and SAFECOM. Additionally, SAFECOM worked with another federally supported emergency communications advisory group—the National Council of Statewide Interoperability Coordinators—to create a publicly available document on the T-Band auction and the potential effects on public safety and cited the NPSTC’s report in the assessment. The document, notes that insufficient spectrum alternatives leave few options for identifying replacement spectrum in several major metropolitan areas. Selected representatives from industry groups whose members are business-industrial T-Band users in the 11 T-Band metropolitan areas, such as the American Petroleum Institute and the Utilities Technology Council, also said they anticipate that there would not be alternative spectrum available if required to relocate. For example, representatives with the American Petroleum Institute said that there are staff at major refineries that use the T-Band on a daily basis for all plant operations including emergency response (firefighters and hazardous materials), control room, engineering, and maintenance, and that relocating to new spectrum would be challenging given the lack of available spectrum. These representatives noted that most of the refineries that use the T- Band are located in Houston, but there are also some facilities in the San Francisco, Los Angeles, and Philadelphia metropolitan areas. Public safety officials in Boston, Los Angeles, and New York City agreed that relocating LMR operations from one spectrum band to another can be costly, complicated, and time intensive given infrastructure and equipment needs. These officials told us that transitioning from the T- Band requires identifying and acquiring new sites to build towers, purchasing new radios, testing new systems, building other infrastructure, and training personnel on the new systems. NPSTC calculated in its 2013 report that the cost to relocate public safety operations in the 11 metropolitan areas would be approximately $5.9 billion. Their calculation includes the costs for the total estimated number of new towers, cables, antennas, and mobile, portable, and vehicular radios. In 2016, after updating its analysis, NPSTC’s second report confirmed that the conclusions from the 2013 report remain valid. According to FCC officials, in early 2019 they analyzed the costs for relocating public safety users from the T-Band and estimated the total cost would be between $5 and $6 billion. Officials from nearly all of the public safety entities we interviewed in the Boston and New York City metropolitan areas cited the NPSTC reports as the best source of publicly available cost calculations for relocating public safety users from the T-Band. Officials from nearly all of the public safety entities we interviewed in Boston, Los Angeles, and New York City told us that estimating relocation costs is and will remain difficult until alternative spectrum is identified. However a few selected public safety users provided us with high-level cost estimates for replacing LMR system components. For example, an official in Pasadena said a conservative estimate for those components would be $13 to $14 million; while public safety officials in New York City estimated component costs would be at least $1.8 billion. According to public safety officials in Morris County, New Jersey, and Yonkers, New York, the financial burden may be greater for less populated areas, despite the higher anticipated actual cost for more populated areas. For instance, public safety officials in Morris County, New Jersey, told us they estimated $30 million in relocation costs, which exceeds the county’s total annual capital project budgets (approximately $20 to 25 million). According to public safety users in the Boston, Los Angeles, and New York City metropolitan areas, costs for relocating LMR systems from the T-Band depend on a variety of factors including (1) equipment, (2) infrastructure, and (3) real estate. 1. Equipment. Transitioning to another spectrum band could require public safety users to purchase new equipment such as radios. Some radios can only operate on one spectrum band, so moving to a new band requires purchasing new radios that can operate on that band. Alternatively, users could purchase multi- band radios, which can operate on more than one radio frequency band. According to public safety officials we spoke with, multi-band radios might be the best option since it is not clear which frequencies they will ultimately be relocated to. However, they also noted that multi-band radios are substantially more expensive than single band radios. For example, officials with the Boston Fire Department told us a regular radio costs approximately $5,000 each while multi-band radios cost up to $8,000. These officials told us that relocating from the T-Band would mean replacing approximately 1,800 radios with multi-band units, meaning that just replacing the Boston Fire Department’s handheld and portable radios could cost more than $14 million. Additionally, public safety officials in Boston and New York City added that local building codes in those areas require buildings of a certain size to install equipment that amplifies wireless signals throughout a building and improves coverage. These systems help first responders, such as police and firefighters, communicate with each other in large buildings. 2. Infrastructure. Infrastructure costs could include new radio towers and antennas and fiber-optic cable systems. Because different radio frequencies have different characteristics and can cover different distances, depending on to which spectrum band public safety users are relocated, circumstances may require more radio towers and antennas. For example, officials with the Boston Fire Department told us that if space were available and they were to relocate from the T- Band to the 800 MHz public safety band, they would need additional radio towers. Specifically, these officials said their current system consists of 42 receivers and five transmitting sites and estimated that a system in the 800 MHz band would likely require up to 60 receivers and five-to-nine transmit sites. FCC officials told us that based on the characteristics of other spectrum bands allocated to public safety, users may need to build between two and three times as much infrastructure to provide the same coverage. The officials noted this would substantially increase relocation costs. Additionally, public safety officials in Boston and New York City told us they are able to use the T-Band to communicate in the tunnels beneath each city because of infrastructure investments like the T-Band specific radiating cables, which allow first responder’s radios to work underground. Officials from New York City police, fire, emergency- management department and the mayor’s office said that relocating to a new spectrum would require installing a new radiating cable system in hundreds of miles of subway, train, and vehicle tunnels. These officials estimated that replacing the radiating cable infrastructure alone would take at least a decade and cost over $1 billion. Officials added that replacing the infrastructure would involve closing subway lines for extended periods of time as the new cables are installed. 3. Real estate. Costs associated with buying or leasing new real-estate sites for towers and other radio equipment will also affect the cost estimate for public safety users. Officials from Boston, Los Angeles, and New York City told us that because of the characteristics of different spectrum bands, building a replacement system might require additional sites. Additionally, officials with New York City told us that identifying locations and negotiating leases for radio towers and spaces for other equipment including radio cabinets would likely be difficult due to the scarcity of and high costs of appropriate sites in New York City. Public safety officials in Boston, Los Angeles, and New York City added that relocating from the T-Band would require building and operating parallel systems to avoid disrupting emergency communications. This project would require some duplication of investments—for example, radio towers, radio cabinets, and antennas, among other equipment and infrastructure—during the transition. For example, officials in New York City police, fire, and emergency-management departments told us they would need to build a dual system that could require at least twice as much space for equipment. They also noted that the current sites are rent free because of existing arrangements, but they believe that it is unlikely that landlords will provide additional space rent free. These officials told us that even if FCC identified available spectrum for them to relocate to, they would be unable to build and test the systems in the 2-year time frame required by statute. For example, New York City officials estimated buildout and testing could take over a decade, which they indicated would also substantially increase the city’s cost. Public safety stakeholders in the Boston, Los Angeles, and New York City metropolitan areas told us that it is difficult to estimate the time needed to build new LMR systems, but estimates ranged from 2 to more than 10 years from the time that alternative spectrum was identified. They noted that these time frames would also depend on the availability of funding and on the complexity of the new systems to be designed, built, and tested. FCC officials also told us that the time and expense of relocating hundreds of licensees at thousands of sites is difficult to predict due to many local factors. For instance, FCC officials cited their ongoing experience relocating public safety licensees within the 800 MHz band which was originally estimated to take 3 years. However, based on certain factors such as the geographic location and interdependencies of communications systems, this relocation effort remains incomplete after 14 years. Public safety stakeholders we talked to told us that the T-Band is important for the interoperability of public safety equipment and said that maintaining interoperability on alternative spectrum would be a challenge. Boston officials told us interoperability is vital for public safety and the T- Band is the key for their interoperability capabilities. For example, these officials said the LMR systems that allow almost 170 local, county, state, and federal law enforcement agencies to communicate with each other use the T-Band. The officials said this network of LMR systems is the only way for all these entities to communicate on a daily basis and is also used for command and control for crisis response at major events such as the Boston Marathon. These officials credited this system on the T-Band for the successful response to the 2013 Boston Marathon bombing. Officials said the LMR system allowed first responders in neighboring jurisdictions to provide additional communication equipment and personnel during the ensuing manhunt. Similarly, officials from New York City told us the T- Band now provides the foundation for all first responder communications in the area. Officials said the September 11, 2001, terrorist attacks demonstrated the loss of life that can occur when first responders are unable to communicate with each other because there was no system in place to allow police, fire, and emergency medical services to easily communicate. As a result, officials said New York City has spent countless hours and millions of dollars to improve interoperability, and that the interoperable system currently in place is based on the T-Band. In December 2018, we reported that it is vital for first responders—such as police officers and firefighters—to have (1) timely communications; (2) sufficient capacity to handle the communications; and (3) interoperable communications systems that enable first responders to connect with their counterparts in other agencies and jurisdictions, even if their counterparts’ systems or equipment vendors differ. As noted previously, public safety users rely on LMR systems as their primary means to gather and share information. For public safety users that rely on the T-Band for interoperable communications and that lack alternative spectrum to build new interoperable systems, losing access to the T-Band would mean public safety officials in multiple large metropolitan areas would be unable to communicate with first responders within their community, neighboring jurisdictions, and the federal government. Public safety officials in Boston, Los Angeles, and New York City told us that the characteristics of the T-Band spectrum are ideal for reliable emergency communications and that moving to another spectrum band may present a challenge to reliability. Since different frequencies of radio waves have different characteristics, jurisdictions typically use the spectrum that is best suited for their particular location. The officials told us that the T-Band’s characteristics allow radio signals to penetrate buildings and across varied terrain and require less infrastructure investments, such as radio towers, than other frequency bands assigned for public safety use. Los Angeles County officials cited the characteristics of the T-Band as the primary advantage the current radio system has over other systems operating on other spectrum bands. They explained that the characteristics make it more suitable for challenging terrain on the forested, mountainous, and coastal areas of the county, than similarly equipped radio systems operating in other frequency bands. FCC has taken some preliminary steps to help facilitate the mandated relocation of public safety users from the T-band, such as imposing a T- Band license freeze, requesting public comments, and creating a fact sheet to notify stakeholders of the spectrum auction and prepare for the auction. In April 2012, FCC froze the processing of applications for new or expanded T-Band radio operations in an effort to avoid adding to the cost and complexity of the mandated public safety relocation. Affected applications included those seeking: (1) new T-Band licenses; (2) modifications to existing licenses by adding or changing frequencies or locations within the T-Band; (3) modifications to existing licenses by changing technical parameters—such as increases in bandwidth, power level, antenna height, or area of operation—in a manner that expands the station’s spectral or geographic footprint; and (4) any other modification that could increase the degree to which the 470–512 MHz band currently is licensed. Both public safety and business-industrial users we interviewed expressed concerns about the license freeze and said it has caused some uncertainty and in limited cases has affected their ability to maintain existing systems. For example, public safety officials from one department we interviewed in the Boston metropolitan area said the freeze has affected users’ ability to replace aging equipment, which has led to poor communications in the area. Additionally, representatives from one business-industrial user told us that Hurricane Harvey destroyed one of its LMR sites and that the entity was having trouble rebuilding a site elsewhere since FCC considers this action a major change and thus affected by the license freeze. FCC staff told us that the public notice announcing the license freeze specifically advised affected parties that they could request a waiver in unusual circumstances where the public interest so warrants, and that that no such request appears to have been filed in this instance. In addition, as discussed earlier, FCC sought public comment in February 2013 to gather information and specific proposals for reallocating and auctioning the T-Band. FCC officials said they continue to evaluate auction proposals from these comments. In October 2014, FCC released a report and order making 24 channels in the 700 MHz narrowband, previously held in reserve, available for public safety users. FCC concluded that given the significant increase in demand for 700 MHz narrowband spectrum, particularly in urban areas, these channels should be made available for use. Public safety users of the T-Band were given priority to these new channels if they committed to return an equal amount of T-Band channels and obtained the concurrence of the relevant regional-planning committees. According to NPSTC’s 2016 report, these 24 additional channels are beneficial but insufficient to relocate all current users of the T-Band. The report notes that channel insufficiency is particularly challenging in the five metropolitan areas where T-Band usage is the highest—Boston, Chicago, Los Angeles, New York City, and Philadelphia. Furthermore, one public safety official in the Los Angeles metropolitan area raised concerns about potential radio interference if relocated to another frequency. The official said that currently, because the T-Band is not used by neighboring jurisdictions, the city does not currently have to worry about frequency interference. By contrast, the 700 and 800 MHz band is currently occupied by public safety in neighboring Riverside and San Diego Counties. This means, according to the official, that building a new system operating in the 700-800 MHz band could potentially introduce interference issues. FCC also created a fact sheet in July 2016 with basic information on the statutory relocation requirement. The T-Band fact sheet states that the relocation shall be completed within 2 years of the auction’s completion date: the exact timing of the relocation deadline will depend on when the auction concludes. FCC officials told us the T-Band fact sheet is the only formal T-band auction guidance that they have provided. However, officials said that they have also met with several licensees to discuss T- Band issues. For example, according to officials, FCC has met with public safety entities from areas such as Los Angeles, Chicago, Boston, and New York City. DHS officials told us that while they have no formal role in the T-Band auction and relocation of public safety users, they provide this fact sheet when they are asked for details about the T-Band auction as a way to help raise awareness about the auction and relocation requirements. Although FCC has made efforts to provide guidance and information to T-band users regarding the mandated auction, as we discuss earlier in the report, we found that not all T-Band users we interviewed are aware of the upcoming auction or the need to relocate from the T-Band. FCC has not set a timeline for initiating the auction but has stated that it is committed under any scenario to ensure the continuity of T-Band licensee’s public safety mission-critical communications. According to FCC officials, as of March 2019, almost all T-Band licensees continue to operate on the T-Band spectrum, and FCC officials cited multiple factors for the limited progress in preparing for the T-Band auction: FCC has not determined how to address challenges stakeholders identified in response to FCC’s 2013 request for public comment, including the lack of available spectrum to relocate and the cost. For example, officials told us that they are taking a wait-and-see approach to see how many T-Band licensees relocate prior to the auction. However, as noted previously, FCC officials told us their analysis of other spectrum bands shows insufficient spectrum for relocating public safety entities from the T-Band. The officials told us that public safety operates on the T-Band in large metropolitan areas where other public safety spectrum is heavily used and that this reason is why the T-Band was allocated for LMR in these areas in the first place. The T-Band auction has raised complicated relocation questions. For example, select industry groups we spoke to whose members are business-industrial T-Band users expressed concern about the uncertainty of the spectrum auction requirements, since the Act was silent on business-industrial users, but they are constrained by the license freeze from replacing aging equipment. FCC previously told us that it had not determined whether business-industrial users would be required to relocate. However, in April 2019, FCC officials told us that it intends to implement the auction following the statute’s language. FCC officials stated that the Act does not expressly require it to auction spectrum licensed to business-industrial users, but officials also stated that FCC may decide that it has the authority to auction that spectrum under a different statutory provision. Before conducting the auction, FCC must issue a notice, which includes a public comment period, to determine the auction procedures and requirements. FCC officials told us they have not progressed beyond the preliminary conceptual stages and do not have a precise timeline for the pre-auction process or auction. The officials explained that if business-industrial users relocate, they would face similar relocation challenges to that of public safety users and the Act does not mention them as eligible for relocation grants. According to FCC officials, licenses for business-industrial users outnumber those of public safety users on the T-Band in some areas. According to FCC officials and a FirstNet official, public safety users on the T-Band may subscribe to services on FirstNet’s nationwide public safety broadband network, which offers some voice functionality. However, officials said the network currently does not accommodate the need of public safety users for mission-critical voice functionality. For example, FCC officials told us that FirstNet’s network is not a substitute for mission critical voice systems operated by public safety licensees in the T-Band because the network does not support such capabilities and because there is no plan or schedule in place for the network to begin offering such services. According to an official at FirstNet, this network is intended to complement LMR systems with broadband capabilities, not replace LMR systems in the near future. In the interim, public safety users electing to use FirstNet’s broadband network will need to continue to use LMR networks for their mission critical voice needs while evaluating whether their future voice needs require continued maintenance of their LMR networks or whether FirstNet broadband services could fulfill their wireless communications requirements. The amount of proceeds that may be generated from the T-Band auction—which are, according to FCC, expected to be the sole source of federal funding to help cover the relocation costs incurred by public safety entities—is likely to be less than the total relocation costs. FCC officials told us the T-Band has potentially low value because of limited demand by potential bidders in the auction. For example, FCC officials estimated that revenue for the entire T-Band would not exceed $2 billion. To reach this amount would require public safety and business-industrial users to relocate from the T-Band, which according to FCC estimates could cost between $9 and $10 billion. As discussed previously, representatives from a trade organization told us that in five of 11 metropolitan areas where public safety uses the T-Band, business-industrial users hold more than half of T-Band licenses. Because of the high numbers of business- industrial users in the T-Band, there may be less spectrum to auction than perhaps initially contemplated when the Act was passed, which would ultimately affect auction proceeds. If FCC were to decide that it has the authority to auction spectrum utilized by business industrial users under a different statutory provision, as explained above, proceeds would be higher. As discussed above, NTIA is to make grants to cover relocation costs for the relocation of public safety entities in accordance with the Middle Class Tax Relief Act. However, NTIA officials told us that the agency has no dedicated funding to administer such a program and must wait for auction proceeds to stand one up. The officials also said that only when the auction concludes will NTIA know the total amount available and how best to disburse those funds for relocating agencies. Thus, designing a grant program, notifying eligible parties of available grants, evaluating applications, and issuing awards must all take place during the statutory 2-year relocation period. If agencies require the funds before they can move to other frequencies, it is unlikely that this migration can meet the two-year deadline. NTIA officials also stated that until they design the grant program, they do not have any relevant information to provide public safety stakeholders. NTIA officials said they would provide information on the grant program and begin making grants as soon as possible given the statutory requirement for public safety users to relocate within 2 years of the auction’s conclusion. According to NTIA officials, because the requirements for NTIA’s grant program for public safety relocation costs have not yet been specified, it is unclear what expenses will be covered. As previously discussed, FCC and NPSTC each calculated the cost for relocating public safety users in the 11 metropolitan areas and each arrived at an estimate between $5 and $6 billion. FCC officials said because of the high relocation costs and likely low value of the T-Band’s being auctioned, there is a strong likelihood auction proceeds would not cover public safety relocation costs. Although the Act stipulates that auction proceeds shall be made available through grants in such sums necessary to cover costs for the relocation of public safety entities from the T-Band spectrum, FCC officials said the Act did not address what would happen if the auction generated insufficient funds to cover relocation costs. Consequently, public safety stakeholders from Boston, Los Angeles, and New York City expressed concern about moving forward with relocating. These stakeholders identified the uncertainty of what spectrum would ultimately be auctioned as one of the main reasons they were concerned they would be unable to fully cover their relocation costs. FCC officials stated that they recognize that the T-Band auction and relocation requirement present challenges for FCC and public safety entities—and potentially business-industrial users—particularly since spectrum for relocating all public safety users is limited to non-existent. However, these officials said they will design and conduct the spectrum auction, as required, unless the law is changed. In this case, FCC officials told us they provided Congress with information on the challenges associated with the auction. While FCC provided information to Congress, it did not suggest changes to law in this instance. As such, officials told us in March 2019 they were in the process of briefing key congressional committees on the challenges associated with the T-Band auction based on FCC analysis. According to this analysis, all T-Band auction scenarios would fail. FCC ran auction scenarios that looked at different options for relocating users and auctioning the T-Band used by public safety. These scenarios included relocating only public safety users, relocating public safety and business-industrial users, relocating public safety users, and reorganizing business-industrial users within the T-Band. In 2018, bills were introduced in both the House of Representatives and the Senate to repeal the requirement for FCC to reallocate and auction the T-Band. These bills were not enacted and expired at the end of the 115th Congress. However, in January 2019, a bill was introduced—and subsequently referred to a House subcommittee—to repeal the T-Band relocation and auction requirements. As of June 2019, no further action has taken place on the legislation. According to FCC’s strategic plan, one of FCC’s priorities is to protect public safety, and in particular, take steps to assist and safeguard the communications of our nation’s law enforcement officers and first responders. However, auctioning the T-Band spectrum, as FCC has been mandated to do, could hamper its ability to safeguard these communications. As mentioned above, the Act and its legislative history do not discuss the purpose of the T-Band auction. Public safety stakeholders in Boston, Los Angeles, and New York City told us they believe that there may have been an assumption the FirstNet network could absorb public safety users, but at this time the network does not support mission-critical voice capabilities first responders need. According to stakeholders in the Boston and New York City metropolitan areas, if the provision requiring the auction of public safety users’ T-band spectrum remains in effect and if the auction takes place, they could experience substantial harmful effects on their ability to maintain continuous and effective communications during an emergency. Officials representing seven public safety entities told us they favored Congress’ repealing the required T- Band auction for this very reason. For example, public safety officials in New York City said they believe the T-Band auction would severely negatively affect their ability to respond to emergencies and could lead to the loss of lives. In addition, officials with the Boston police department told us the T-Band is the lifeblood of police communications and the only way for almost 170 law enforcement departments in the Boston metropolitan area to communicate with one another on a daily basis and during major events. These officials said that auctioning the T-Band and forcing them to relocate and build a new system over several years would disrupt critical public safety communications and be disastrous. Since the passage of legislation requiring the relocation of public safety users from, and auction of, the T-band radio spectrum, the potential consequences of these actions have become far more apparent. If FCC conducts such an auction, it is unclear that all public safety users in the affected areas will be able to relocate. If alternative spectrum is not available, public safety would be jeopardized in some of the nation’s largest metropolitan areas. Even if alternate available spectrum can be found, public safety users are likely to bear significant costs associated with relocating and reestablishing interoperability. These costs could go well beyond the revenue produced by such an auction. Congress should consider legislation allowing public safety users continued use of the T-Band radio spectrum. (Matter for Consideration 1) We provided a draft of this report to the Department of Commerce, DHS, and FCC for review and comment. DHS and FCC provided technical comments, which we incorporated as appropriate. The Department of Commerce indicated that it did not have comments. We are sending copies of this report to the appropriate congressional committees, the Secretaries of Commerce and Homeland Security, and the Chairman of FCC. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or members of your staff have any questions about this report, please contact me at (202) 512-2834 or goldsteinm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix II. In addition to the individual named above, David Sausville (Assistant Director); Aaron Kaminsky (Analyst in Charge); Camilo Flores; Ray Griffith; Delwen Jones; Josh Ormond; Kelly Rubin; and Jessica Walker made key contributions to this report.", "summary": "First responders and others in 11 large metropolitan areas use radio systems operating in the T-Band since spectrum is limited in other bands. In 2012, FCC was required by statute to begin an auction of this T-Band public safety spectrum by February 2021 and to make the proceeds available to the National Telecommunications and Information Administration (NTIA) to develop and administer a grant program to help cover costs associated with relocating public safety users' radio systems. GAO was asked to review issues related to the required T-Band auction. This report examines, among other things: (1) the challenges selected first responders and local governments anticipate facing in relocating public safety communications from the T-Band and (2) the actions FCC has taken both to help facilitate the required T-Band relocation and to address identified challenges. GAO reviewed FCC's March 2019 congressional briefing and analysis on T-Band spectrum and conducted case studies in four cities selected based on the number of public safety licenses in each area, among other things. GAO reviewed relevant statutes and regulations, FCC documents, and T-Band studies conducted by a public safety organization. GAO interviewed FCC officials and other stakeholders, including first responders in case study cities. Public safety officials, such as police and fire fighters, in 11 metropolitan areas rely on radio systems that use the portion of spectrum known as the T-Band for mission critical voice communications. Selected stakeholders GAO interviewed, including first responders and officials in three of four areas selected as case studies, anticipate significant challenges in relocating public safety communications from the T-Band. For example, stakeholders in Boston, Los Angeles, and New York said the Federal Communications Commission (FCC) has not identified sufficient alternative spectrum. Additionally, two studies conducted by a public safety organization concluded these three areas and others may also have insufficient alternative spectrum (see figure below). Moreover, a recent FCC analysis showed that relocation options for public safety users are limited or nonexistent. Further, costs for relocating public safety users from the T-Band were calculated by FCC to be $5-to-$6 billion. Selected stakeholders said relocating their communication systems would require such things as new towers and radios as well as other infrastructure. FCC has taken limited actions to address challenges and assist public safety users of the T-Band with the mandatory relocation. For example, FCC has taken steps to notify stakeholders, but officials told GAO they have not begun planning the auction. FCC officials acknowledged challenges the auction and relocation requirements present. FCC officials explained that public safety entities were licensed to operate on the T-Band in large metropolitan areas because other public safety spectrum was already heavily used. In March 2019, FCC briefed Congress on the auction's challenges and concluded that all T-Band auction scenarios would fail. Nonetheless, FCC officials said the agency will conduct the auction unless the law is amended. While FCC provided information to Congress, it did not suggest changes to law in this instance. Stakeholders in two metropolitan areas said the auction could result in substantial harmful effects on their ability to maintain continuous and effective communications during an emergency. Congress should consider legislation allowing public safety users continued use of the T-Band spectrum.", "document_type": "gao"}
{"report": "The FAR, among other things, sets forth requirements that must be met before agencies can award contracts to prospective contractors. Beginning February 26, 2016, contracting officers are required to include a provision in all contract solicitations that require contractors to report information about unpaid federal taxes regardless of the contract value. Specifically, FAR § 52.209-11 incorporates the language from the fiscal years 2015 and 2016 appropriations acts that prohibits the government from entering into contracts with corporations with unpaid federal taxes that have been assessed, for which all judicial and administrative remedies have been exhausted or have lapsed, and that are not being paid in a timely manner pursuant to an agreement with the authority responsible for collecting the tax liability, where the awarding agency is aware of the unpaid tax liability, unless an agency has considered suspension or debarment of the corporation and made a determination such action is not necessary to protect the interests of the government. If the prospective contractor reports having unpaid federal taxes under this provision, the contracting officer must request additional information from the prospective contractor; in accordance with agency procedures, notify the officials responsible for debarment and suspension actions, commonly referred to as the suspension and debarment officials (SDO); and not award to the corporation unless an agency SDO has considered suspension or debarment of the corporation and has made a determination that suspension or debarment is not necessary to protect the interests of the government. Additionally, the FAR requires that contracting officers include in certain contract solicitations another provision for prospective contractors to report delinquent taxes. Specifically, contracting officers are also required to include FAR § 52.209-5 in contract solicitations in which the contract value is expected to exceed the simplified acquisition threshold, which was generally $150,000 at the time of our review, under which prospective contractors report delinquent federal taxes owed. This requirement has been in place since 2008. Under this provision, the prospective contractor must report whether it or any of its principals have, within the preceding 3-year period, been notified of “delinquent federal taxes” in an amount that exceeds $3,500. For purposes of this provision, “delinquent federal taxes” are those for which the tax liability is finally determined and assessed, with no pending administrative or judicial challenge, and all judicial appeal rights are exhausted; and the taxpayer is delinquent in making payment, unless enforced collection action is precluded (the taxpayer is not delinquent if the taxpayer has entered into an installment agreement and is making timely payments in compliance with the agreement terms). If the prospective contractor reports having federal tax debt under this provision, the contracting officer must (1) request additional information from the prospective contractor and (2) in accordance with agency procedures notify the officials responsible for debarment and suspension actions. Further, the contracting officer is not required to receive a suspension and debarment determination before contract award for tax debt reported under this certification. In addition, the FAR generally requires prospective contractors to register in SAM before a contract can be awarded. As part of registering in SAM, prospective contractors must make up to 54 representations and certifications, which must be updated as necessary but at least annually. Included among these is the federal tax debt FAR § 52.209-11 representation and § 52.209-5 certification. The representations and certifications in SAM must be kept current, accurate, and complete. Unpaid federal tax debts reported under FAR § 52.209-11 and delinquent federal taxes reported under § 52.209-5 do not automatically disqualify the prospective contractor from receiving a contract, but rather are used as part of the contracting officer’s responsibility determination of the prospective contractor. Contracting officers rely on the contractors’ representations and certifications in SAM to identify qualifying federal tax debts. Federal tax law generally prohibits the IRS from disclosing taxpayer data to other federal agencies for the purpose of determining whether potential contractors owe qualifying federal tax debt. As a result, contracting officers cannot verify a contractor’s tax-debt status by obtaining taxpayer information directly from the IRS without the contractor’s prior consent. In general, the federal pre–contract award process consists of the agency identifying its needs for goods and services, creating an acquisition plan, posting a solicitation that allows interested contractors to submit bids or proposals, and assessing and selecting a prospective contractor to meet its needs. Agency contracting personnel have a variety of pre–contract award responsibilities. As one of these responsibilities, the contracting officer is to identify the FAR provisions and clauses required to be included in contract solicitations based on various criteria, such as the contract type and contract value. For example, contracts expected to be above the simplified acquisition threshold are required to include § 52.209-5 in the solicitation. After the solicitation is issued and prospective contractors’ offers are obtained, the contracting officer, among other tasks, generally must verify that the prospective contractor is registered in SAM, and that the contractor is not suspended or excluded from doing business with the federal government prior to contract award. The contracting officer must also determine whether the prospective contractor is “responsible.” FAR § 9.104-1 requires that to be determined responsible, prospective contractors must have adequate financial resources to perform the contract, or the ability to obtain them; have a satisfactory record of integrity and business ethics; and be otherwise qualified and eligible to receive an award under applicable laws and regulations, among other things. As part of the responsibility determination, the contracting officer must also access, review, and document the prospective contractor’s applicable representations and certifications, including qualifying federal tax debt reported under § 52.209-11 and § 52.209-5. See figure 1 for an overview of the pre– contract award requirements related to tax debt. The IRS, which is located in the Department of the Treasury (Treasury) and led by a commissioner, may collect assets or payments, including federal contract payments to collect unpaid taxes, and these collections are referred to as a “levy.” The IRS will usually levy only after notifying the taxpayer in writing of the amount of the unpaid tax and the right of the taxpayer to request a hearing within a 30-day period before the levy occurs. However, if the taxpayer is a federal contractor, the taxpayer is given the opportunity for the hearing within a reasonable period after the levy. One way the IRS levies federal contractor payments is through the FPLP, which is an automated program that can collect overdue taxes through a continuous levy on certain federal payments processed by Treasury’s Bureau of the Fiscal Service (Fiscal Service). In addition to the FPLP, the IRS can also levy federal contractors manually. Specifically, the IRS may levy federal contractor payments directly from federal agencies to collect unpaid taxes. The five selected agencies we examined have established control activities to varying degrees to help contracting officers comply with federal laws and regulations related to identifying prospective contractors’ reported qualifying federal tax debt. These control activities include the following: Class Deviations: The five agencies issued class deviations from the FAR to implement the tax debt–related appropriations restriction prior to February 26, 2016. These class deviations generally required contracting officers to include an alternative provision in solicitations and, if a contractor reported having qualifying tax debt, to not award the contract without a written suspension and debarment determination from an agency SDO. For example, the Department of Defense, DOE, HHS, and VA issued class deviations as early as 2012 that required contracting officers to take two actions: (1) insert an alternate provision when issuing solicitations using appropriated funds and (2) obtain an SDO determination that suspension or debarment is not necessary to protect the interests of the government before awarding a contract to a contractor who reported qualifying tax debts. Policies and Procedures: VA, DOE, and HHS issued policies and procedures to varying degrees that generally direct contracting officers to the relevant sections of the FAR when assessing contractor responsibility. For example, both VA and DOE issued policies or guidance on determining contractor responsibility and including § 52.209-5 in solicitations where the value was expected to exceed the simplified acquisition threshold. In addition, agency officials who supervise contracting officers told us that contracting officers use contractors’ representations and certifications in SAM to identify qualifying federal tax debts and document their review of the information when determining contractor responsibility before contract award. For example, one of the Navy’s responsibility-determination templates requires contracting officers to notate that they verified, in SAM, that the prospective contractor did not report qualifying federal tax debts under FAR § 52.209-5. Further, the five agencies have also issued procedures outlining the SDO suspension and debarment referral and review process, as required by federal regulations. For example, HHS issued guidance on suspension and debarment that includes (1) relevant contact information, (2) required or optional documentation to include, and (3) potential causes for suspension or debarment, such as the contractor reported qualifying federal tax debt. Both the Army and Navy issued policy alerts informing contracting officers of the February 26, 2016, effective date of FAR § 52.209-11 and the requirement that an SDO determine that suspension or debarment is not necessary to protect the interests of the government before awarding a contract to a contractor who reported having tax debts under this provision. Contract-File Compliance Tools: The five agencies told us that contracting officers have tools available that help ensure required information, including information related to federal tax debt, is reviewed and documented in contract files. For example, contracting officer supervisors and policy officials at these agencies told us that contracting officers use agency contract-writing systems to assist with identifying and inserting required FAR provisions and clauses in the contract solicitation. HHS and VA contracting officer supervisors also told us contracting officers use contract-file checklists to ensure required FAR provisions and clauses are included in the contract solicitation. In addition, some of the five selected agencies’ contract- file checklists or memorandums we reviewed generally document that the contracting officer verified the prospective contractor’s SAM registration, and suspension and debarment status, and retrieved the relevant SAM representations and certifications before contract award. Further, some VA and DOE contract checklists we reviewed also document that the contracting officer considered tax debts reported under § 52.209-5 or federal tax debt in general (see fig. 2). Periodic Compliance Reviews of Samples of Contracts: The five agencies’ policy officials and contracting officer supervisors we interviewed told us they generally conduct compliance reviews on a sample of contract files before and after contract award to ensure that the required FAR provisions and clauses are inserted in contract solicitations, including peer-to-peer, management, and legal compliance reviews. Agency officials also told us this includes verifying that the contracting officer considered and documented the prospective contractors’ SAM representations and certifications before contract award. For example, the Army’s procurement management review program is designed to ensure regulatory and policy compliance, among other things, via oversight by a multilevel program that reviews each contracting activity every 3 years. Training: DOE and VA provide training that generally discusses contractor responsibility determinations and references the requirement that contracting officers inform the SDO when prospective contractors report that they have qualifying federal tax debt before contract award. The Department of Defense provides training on the causes for suspension, and the Navy SDO also provides training discussing the requirement to notify the SDO when prospective contractors report qualifying federal tax debt. HHS suspension and debarment staff we interviewed told us that they provide general suspension and debarment training that includes causes for suspension and debarment referrals, such as tax debt. Further, one Navy contracting office also provides training on inserting the tax-debt provision in all contract solicitations. We identified 1,849 contracts awarded by the five selected agencies in 2015 and 2016 to contractors that reported qualifying federal tax debt that potentially should have resulted in these agencies taking required follow- up actions before contract award, such as notifying the agency SDO of these tax debts. Specifically, according to our analysis of FPDS-NG and SAM data for this period, the five selected agencies potentially should have notified an SDO prior to awarding 1,849 contracts to contractors that reported having qualifying federal tax debt under their § 52.209-11 representation or § 52.209-5 certification, which we discuss further below. However, none of the five selected agencies’ SDOs we interviewed were notified of any instances in which a contracting officer identified a prospective contractor with these reported qualifying federal tax debts, and they did not receive any tax debt-related referrals within this period. Agency officials we interviewed were unable to explain why the SDOs were not notified without reviewing each of the 1,849 contract files. Because referrals were not made to an SDO before awarding the contract, agencies’ control activities do not appear to have operated effectively to identify contractors’ reported tax debt and to consider suspension and debarment when required. As a result, these contracts may have been awarded without required actions being taken—a potential violation of federal regulations and, in some cases, the Antideficiency Act. In addition, we reviewed a nongeneralizable sample of seven contracts where prospective contractors reported qualifying tax debts before receiving contract awards and identified two illustrative examples where agency control activities did not ensure regulatory compliance. The tax debts for these contractors were collectively more than $250,000, and historical IRS tax records include instances where the IRS had assessed a Trust Fund Recovery Penalty (TFRP), indicating willful failure to collect, account for, or pay taxes owed. Nonetheless, the contracting officers awarded these two contracts without taking required follow-up actions for these awards. These contractors were awarded more than $510,000 in contract obligations in total, in 2015 and 2016. In our analysis of the five selected agencies, we identified 143 contracts at four of the agencies that were awarded to contractors who reported qualifying federal tax debt in SAM under § 52.209-11 from February 26, 2016, through December 31, 2016. Table 1 shows the number of contract awards to contractors who reported qualifying federal tax debt under § 52.209-11 from February 26, 2016, through December 31, 2016, by selected agency. We did not identify contracts awarded by DOE during this period to similar contractors, and thus did not assess the operational effectiveness of the agency’s controls activities for compliance with its relevant class deviation. However, none of the four agencies that awarded these 143 contracts took required follow-up actions that potentially should have resulted from the contractor’s reporting qualifying tax debt before contract award. As mentioned earlier, when prospective contractors report having qualifying federal tax debt under § 52.209-11, the FAR requires that contracting officers (1) request that the contractor provide such additional information as the contractor deems necessary in order to demonstrate responsibility; (2) notify, in accordance with agency procedures, the SDO of the contractor’s reported qualifying federal tax debt, before award, for suspension and debarment review; and (3) not award the contract unless an SDO determines that further action is not required to protect the interest of the government. The FAR also requires that contracting officers possess or obtain information sufficient to determine whether the prospective contractor is responsible. As mentioned above, qualifying federal tax debts reported under this representation do not automatically disqualify the prospective contractor from receiving a contract, but rather are used as part of the contracting officer’s responsibility determination of the prospective contractor. In our review of contract-file documentation for seven contract awards to contractors that reported they had qualifying tax debt under either provision, we could determine for one case under this representation that the contracting officer did not take required follow-up actions to ensure compliance with federal regulations. We highlight this example in the sidebar to the left. Agency contracting officer supervisors we interviewed from the four selected agencies that awarded the 143 contracts discussed earlier told us that they were not aware of any instances in which a contracting officer identified a prospective contractor’s reported qualifying federal tax debt under § 52.209-11 and notified the SDO during this period. As mentioned, the SDOs we interviewed at these four agencies told us that they did not receive, nor were they aware of, any notifications to review prospective contractors that reported having qualifying federal tax debt during this period. All four of these SDOs told us that they track notifications to the SDO manually or via a case-management tracking system. Further, none of the agency officials we interviewed at the selected agencies were able to identify specific reasons a contracting officer would not notify an SDO of reported qualifying federal tax debt as required. Our analysis of the five selected agencies also identified 1,706 contracts awarded in 2015 and 2016 to contractors that reported having qualifying federal tax debt in SAM under § 52.209-5. Table 2 shows the number of contract awards to contractors that reported having qualifying tax debt under § 52.209-5 in 2015 and 2016, by selected agency. However, none of the five agencies that awarded these 1,706 contracts took required follow-up actions that potentially should have resulted from the contractor’s reporting qualifying tax debt before contract award. As mentioned above, as early as 2008, contractors were required to certify whether they had qualifying federal tax debt if, within the preceding 3-year period, they or any of their principals had been notified of “delinquent federal taxes” in an amount that exceeds $3,500 for which the liability remained unsatisfied. Also as previously mentioned, tax debts must only be reported under this provision if the tax liability is finally determined with no pending administrative or judicial challenge, all judicial appeal rights have been exhausted, enforcement action is not precluded, and the taxpayer is not in compliance with an installment repayment agreement. Qualifying federal tax debts reported under this certification do not automatically disqualify the prospective contractor from receiving a contract, but rather are used as part of the contracting officer’s responsibility determination. Further, contracting officers are to insert this FAR provision in solicitations where the value of the contract is expected to be greater than the simplified acquisition threshold. If a prospective contractor reports qualifying tax debt, contracting officers must request such additional information as the contractor deems necessary in order to demonstrate responsibility, and, prior to proceeding with the award, notify the agency’s SDO in accordance with agency procedures. While we cannot readily determine whether all 1,706 contract awards were out of compliance with federal regulations due to limitations in the data, as discussed earlier, our review of seven contract awards with reported qualifying tax debt under either provision identified an instance under this certification where we confirmed that the solicitation was above the simplified acquisition threshold and the contracting officer did not take follow-up actions to ensure compliance with federal regulations (see sidebar to the left). As mentioned, agency contracting officer supervisors we interviewed from the five agencies told us that they were not aware of any instances in which a contracting officer identified a prospective contractor’s reporting qualifying federal tax debt under § 52.209-5 and notified the SDO during this period. Further, SDOs we interviewed at these five agencies told us that they did not receive, nor were they aware of, any notifications identifying prospective contractors that reported qualifying federal tax debt under this FAR provision during this period. As mentioned earlier, four out of the five SDOs told us that they track SDO notifications, and none of the agency officials we interviewed identified specific reasons a contracting officer would not notify an SDO as required. When discussing these 1,849 contracts with agency officials, they were unable to explain whether or why their control activities did not operate effectively to ensure compliance with applicable federal laws and regulations. To do so, some of these officials told us that they would need to review the contract files for each of the 1,849 instances of potential noncompliance we identified. Specifically, the agency must confirm that (1) a solicitation was issued, and (2) the estimated value of the contract award was above the simplified acquisition threshold, when applicable, to determine whether the regulatory requirements applied. If the regulatory requirement applied to the contract award, the agency must then determine why their control activities did not operate effectively to ensure compliance. We plan to refer these contract awards to the appropriate agency’s Inspector General for review, and share them with the agencies at that time as well. Understanding why existing control activities potentially did not operate effectively will help these agencies ensure they are taking necessary steps to protect the interests of the government and avoid the misuse of appropriated funds in the future. The five selected agencies told us, in response to our review, they plan to take actions to improve control activities to identify contractors’ federal tax debts reported under § 52.209-11 and § 52.209-5. These planned actions include issuing new guidance, providing additional training, verifying that contracting officers considered reported tax debts in postaward compliance reviews, and updating preaward contract-file checklists to ensure compliance with federal laws and regulations. Some of the selected agencies also noted that the FAR requirements apply to all executive agencies and that a broader solution to accessing, identifying, and reviewing qualifying federal tax debt reported in SAM representations and certifications could be useful. Agency officials explained that contracting officers have to individually identify and review each relevant representation and certification—up to 54 representations and certifications—to become aware of the prospective contractor’s response before contract award. Further, agency officials told us that contractors’ responses are not easily identifiable in SAM and contracting officers can miss the contractor’s reported qualifying federal tax debt under § 52.209-11 and § 52.209-5. As mentioned earlier, accessing, reviewing and documenting the SAM representations and certifications is one part of the preaward contracting process and is one of the actions contracting officers are required to take as part of the contract award process. The SAM tax-related representations and certifications that must be reviewed before contract award are determined by various factors, including contract award value. See figure 3 for an overview of the general process to access, review, and identify prospective contractors’ qualifying tax debts reported in SAM. As mentioned earlier, GSA manages SAM, and while the GSA official we interviewed acknowledged the challenges raised by the selected agencies, this official noted that SAM representation and certification data are accessible to contracting officers for the purpose of reviewing qualifying federal tax debt reported by prospective contractors and taking any required follow-up actions. Nevertheless, this official noted that GSA is in the process of upgrading SAM, which may include changes to the representations and certifications. Standards for Internal Control in the Federal Government state that management should use high-quality information to achieve its objectives and that management should consider the accessibility of information and make revisions when necessary so that the necessary information is accessible. As GSA makes planned upgrades to SAM, it is in a position to consider improvements to SAM users’ experience with representations and certifications that may help executive-branch agency contracting officers more easily identify contractors’ reported qualifying federal tax debt under § 52.209-11 and § 52.209-5. Of the 120,000 federal contractors that were awarded contracts in 2015 and 2016, our analysis found that over 4,600 of them had unpaid taxes at the time they received the award. These contractors collectively owed $1.8 billion in unpaid taxes as of December 15, 2016, and received contract award obligations totaling $17 billion. We could not confirm, however, whether at the time of the contract awards these contractors’ unpaid taxes met the relevant legal definitions of qualifying federal tax debt under § 52.209-11 and § 52.209-5 due to limitations in the data. However, we were able to determine which debts likely met the definition of qualifying tax debt, and to determine those that did not meet the definition, as of December 15, 2016—a date after the contract award. Specifically, over 2,700 of these contractors had unpaid taxes that were all likely qualifying federal tax debt as of December 15, 2016. In addition, about 1,900 had unpaid taxes that were not qualifying federal tax debt. As previously noted, agencies are required by the FAR to consider contractors’ reported qualifying federal tax debt before awarding contracts. Generally, as mentioned earlier, agencies are not restricted from awarding contracts to contractors that report having qualifying federal tax debt if an agency SDO determines suspension and debarment of the contractor is not necessary to protect the interests of the government. We describe characteristics of the unpaid taxes and contract awards for these 4,600 contractors with unpaid taxes below. (See fig. 4.) We identified over 4,600 federal contractors that had unpaid taxes at the time they received a contract award in 2015 and 2016. However, we could not confirm whether these contractors’ unpaid taxes met the relevant legal definitions under § 52.209-11 and § 52.209-5 at the time of the contract award due to limitations in the data we obtained, as previously described. These 4,600 contractors received about $17 billion in contract awards and owed $1.8 billion in unpaid taxes as of December 15, 2016. The characteristics of these 4,600 federal contractors with unpaid taxes in December 15, 2016, are discussed below: Average and Total Debt Associated with Contractors with Unpaid Taxes: About 1,000 contractors had unpaid taxes of at least $51,000 each. These contractors collectively owed about 98 percent of the $1.8 billion in unpaid taxes we identified. About 1,900 contractors each had unpaid taxes between $3,500 and $51,000. They collectively owed about $30 million in taxes. About 1,700 contractors each had unpaid taxes over $100 but less than $3,500. They collectively owed about $2 million in taxes. Chief Financial Officers (CFO) Act Agencies Associated with Contractors with Unpaid Taxes: The 4,600 contractors with unpaid taxes as of December 15, 2016, received contract awards in our 2- year period from one or more of all 24 CFO Act agencies. Almost 1,500 contractors received contract awards from more than one agency. These contractors owed almost $600 million in unpaid taxes as of December 15, 2016 (see sidebar to the left). Although, as discussed above, we reviewed the control activities of five agencies, all executive-branch agencies are required by the FAR to consider the qualifying federal tax debt of prospective contractors before making an award. If a contractor is receiving awards from multiple federal agencies, the suspension and debarment determination of any agency SDO is relevant to other agencies considering the same contractor for an award. For example, as discussed earlier, we identified 1,849 contract awards by five selected agencies to contractors that reported qualifying tax debt before contract award, and none of these agency SDOs were notified. There were some instances where more than one agency made a contract award to the same contractor that reported having qualifying tax debts. These obligations might not have been made by multiple agencies if one of these agencies’ SDOs had been notified of the reported tax debt as required. Contractors with Unpaid Taxes and Associated with TFRP: We also identified about 600 contractors whose tax records indicate the IRS assessed a TFRP to the owner or officers associated with the contractor, as shown in the sidebar to the left. As mentioned previously, a TFRP indicates willful failure to collect, account for, or pay certain taxes owed. These 600 contractors had $200 million in unpaid taxes in December 2016. Having a TFRP does not disqualify a contractor from obtaining a contract, but it can be considered when the agency determines a prospective contractor’s responsibility under the FAR, according to agency contracting and suspension and debarment officials (SDO) we spoke with. We found that over 2,700 contractors owed about $350 million in unpaid taxes that likely met the relevant legal criteria for qualifying federal tax debt on December 15, 2016. However, few of those contractors reported having qualifying tax debts in SAM. Because the contracts were awarded before December 2016, we cannot determine whether these unpaid taxes met the relevant legal criteria under § 52.209-11 and § 52.209-5 for qualifying federal tax debt at the time of the contract award. However, because these tax debts were unpaid as of December 15, 2016, we determined they were likely qualifying tax debts because they were not being timely paid consistent with a collection agreement and appeared to be finally determined. These tax debts amounted to about 20 percent of the $1.8 billion in unpaid taxes we identified. The 2,700 contractors received almost $5 billion of the $17 billion in federal contract obligations for awards made to contractors with unpaid taxes. We examined the SAM § 52.209-11 representations and § 52.209-5 certifications for these over 2,700 contractors to determine whether they reported this debt as qualifying federal tax debt. We identified about 2,000 contractors that had completed a representation or certification, and, when applicable, met the tax-debt threshold for § 52.209-5. Of those 2,000, 93 percent (1,848) did not report their debt as qualifying federal tax debt, compared to fewer than 150 who did report qualifying federal tax debt under one or both tax-debt provisions (see sidebar to the left). Specifically: Over 1,300 contractors completed the § 52.209-11 representation in SAM (which took effect on Feb. 26, 2016), and less than two dozen of these contractors reported having qualifying federal tax debt under § 52.209-11 before receiving contract awards. Nearly 1,400 contractors completed the § 52.209-5 certification in SAM and as of December 15, 2016, had unpaid taxes over the certification threshold. Fewer than 140 of these contractors reported under § 52.209-5 that they had been notified of qualifying federal tax debt above $3,500 before receiving a contract award. The accuracy of contractors’ reported tax-debt status in SAM is critical to federal agencies’ ability to identify reported qualifying federal tax debt owed by prospective contractors. As described earlier, contracting officers generally rely on the contractors’ representations and certifications in SAM to identify qualifying federal tax debts. Contracting officers generally cannot verify a contractor’s tax-debt status by obtaining taxpayer information directly from the IRS without the contractor’s prior consent, because federal tax law generally prohibits the IRS from disclosing taxpayer data for this purpose. While contracting officers cannot independently verify whether federal contractors accurately report qualifying federal tax debt, any qualifying federal tax debt may be available for levy by the IRS, as discussed further below. We found that about 1,900 contractors had about $1.4 billion in unpaid taxes that did not meet the relevant criteria for qualifying federal tax debt on December 15, 2016, a date after which their contracts were awarded. Specifically, these unpaid taxes were not finally determined or were being paid in a timely manner consistent with a collection agreement as of December 15, 2016. If the status of these debts was the same at the time of contract award, then the contractors did not need to report them during the contracting process and agencies were not required to consider the debts before awarding the contract. Although we were able to determine that these unpaid taxes did not meet the legal definitions of qualifying federal tax debt as of December 15, 2016, we could not determine whether this was also the case at the time of the contract award. Federal agencies obligated $12 billion to these 1,900 contractors between 2015 and 2016, for awards made while the contractors owed taxes. Of these 1,900 contractors, about 1,400 owed $1.3 billion in unpaid taxes that were not finally determined on December 15, 2016. About 700 contractors owed $90 million in unpaid taxes that were being timely paid consistent with a collection agreement in December 15, 2016, due to installment agreements or offers-in-compromise accepted by the IRS. Through its FPLP, the IRS identified for levy most contractors we found to have likely qualifying federal tax debt, according to our analysis of IRS data. Specifically, of the over 2,700 executive-branch agency contractors with likely qualifying federal tax debt as of December 15, 2016, discussed above, the IRS identified over 2,000 for levy through the FPLP, a program administered by Treasury’s Fiscal Service. These 2,000 contractors collectively owed about $300 million of the roughly $350 million in likely qualifying federal tax debt. According to IRS data, the FPLP did not identify almost 700 of the 2,700 contractors we found to have likely qualifying federal tax debt as of December 15, 2016. These 700 contractors owed about $50 million in likely qualifying federal taxes. IRS officials responsible for the FPLP told us that they would need to review these instances to determine whether the contractors were eligible for levy as of December 15, 2016, and if so why they were not identified by the FPLP. We plan to share these cases with the IRS to determine whether the contractors were eligible for levy at that time and take any appropriate enforcement action. It is possible that the IRS did not identify these 700 contractors for levy through the FPLP because the IRS did not have access to their payments. The FPLP was developed as an automatic and efficient means for the IRS to collect delinquent taxes as payments were processed through the Fiscal Service. Accordingly, the FPLP can only levy federal agency payments processed by the Fiscal Service, but not all federal agencies process their payments through the Fiscal Service. As a result, payments disbursed by other means—such as payments that agencies make directly to contractors—are not included in the FPLP, although they can be levied by the IRS through other manual methods (see fig. 5). The IRS cannot readily identify which payments are made outside of the Fiscal Service, and such payments cannot be levied through the FPLP. While the IRS receives some information about contractor payments from agencies, it does not receive information that would allow it to comprehensively determine which payments are processed by the Fiscal Service and can be levied through the FPLP and which payments are not and must be levied manually. Specifically, executive-branch agencies, including those that do and do not process payments through the Fiscal Service, are required to report information to the IRS about some federal contracts through the IRS Form 8596 information return. Reporting agencies identify themselves on the Form 8596, and the IRS uses data from this form to identify federal contractors for potential levy. However, the Form 8596 information return lacks information on whether payments to federal contractors are processed by the Fiscal Service or through some other means. Without visibility into the payments made outside the Fiscal Service, the IRS is limited in its ability to identify nonparticipating agencies for outreach about the efficiencies of leveraging the FPLP to collect contractors’ unpaid taxes, as opposed to manual levies. Further, without information on agencies’ payment methods, the IRS cannot quickly identify payments that must be levied through manual methods. Expanding Form 8596 to include payment-method information could help the IRS identify which agencies to target for outreach and avoid delays in identifying contractor payments requiring manual levy. IRS officials told us the IRS has the legal authority to expand Form 8596 reporting requirements and would have to determine whether a change to add information on Fiscal Service processing of agency payments was warranted. In addition, we found the IRS is missing an opportunity to further enhance the FPLP levy process for certain contractor payments. Within the FPLP, the IRS has an expedited process to levy federal contractors and, as noted above, the IRS uses data from Form 8596 to identify federal contractors for potential levy. However, Form 8596 reporting requirements do not apply to federal contracts for which the amount obligated is $25,000 or less. When Form 8596 reporting requirements were initially established, this threshold was consistent with Federal Procurement Data System (FPDS) contract reporting requirements for agencies at the time. However, subsequent FAR amendments revised the reporting threshold from contracts over $25,000 to contracts over the micropurchase amount, which is currently set at $10,000. Because the Form 8596 reporting threshold is higher than FPDS reporting requirements, information about contracts in the $10,000 to $25,000 range is available in FPDS, but is not required to be shared with the IRS. Such information could help the IRS identify and use expedited levy procedures on federal contractors with contract obligations in the $10,000–$25,000 range. According to the IRS, an amendment to its regulations would be needed to align the Form 8596 reporting threshold with FPDS reporting requirements. Standards for Internal Control in the Federal Government state that management should use high-quality information to achieve the entity’s objectives. To do this, management obtains relevant data from reliable internal and external sources, processes the obtained data into high- quality information, and uses high-quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives. Without additional information about and from the agencies making these payments, the IRS may be missing opportunities to identify federal contractors for levy to enhance tax collections. Considering prospective contractors’ reported qualifying federal tax debt—in accordance with federal regulations—helps ensure federal agencies comply with federal appropriations law, supports the integrity of the contracting process, and protects the interests of the government. The five federal agencies we reviewed had control activities, such as policies, procedures, and training, to help ensure contracting officers consider prospective contractors’ reported qualifying federal tax debt before making an award. However, these controls were not always effective in ensuring that potentially required actions were taken. Determining the reasons the contracts we identified were awarded without appropriate consideration of contractors’ reported qualifying federal tax debt and taking additional steps to ensure tax debts are appropriately considered in future contract award decisions is necessary to ensure contracting opportunities are appropriately awarded. Improving accessibility of SAM representation and certification data to allow contracting officers to more easily identify and consider reported qualifying federal tax debt before contract award can help contracting officers meet required steps, such as referring them to the SDO. Federal tax law generally prohibits the IRS from disclosing taxpayer data to other federal agencies for the purpose of determining whether potential contractors owe qualifying federal tax debt. Consequently, federal agencies generally rely on contractors’ reported qualifying federal tax debt to detect any tax debt owed by their potential contractors. However, agencies cannot independently verify the accuracy of contractors’ reported qualifying federal tax debts when awarding contracts. This limitation heightens the importance of the IRS’s levy process for recouping revenue from businesses that have failed to pay their taxes in a timely way but are receiving federal contract dollars, and the recoupment of revenue can help reduce the tax gap. Accordingly, the IRS has opportunities to use available data to improve its detection and collection of qualifying federal tax debts owed by federal contractors, which can help enhance revenue collection and compliance. We are making 12 recommendations—two each to the Army, HHS, the Navy, and VA; one each to DOE and GSA; and two to the IRS. The Senior Procurement Executive for the Department of the Army should review the contracts we identified as being awarded to contractors that reported qualifying federal tax debt under FAR § 52.209-11 and (1) determine whether the contracting officer was required to consider the contractor’s reported tax debt; if so, (2) determine the reasons controls to identify and refer these contractors to the SDO before contract award did not operate effectively; and (3) design or modify controls to help ensure compliance with applicable regulations. (Recommendation 1) The Senior Procurement Executive for HHS should review the contracts we identified as being awarded to contractors that reported qualifying federal tax debt under FAR § 52.209-11 and (1) determine whether the contracting officer was required to consider the contractor’s reported tax debt; if so, (2) determine the reasons controls to identify and refer these contractors to the SDO before contract award did not operate effectively; and (3) design or modify controls to help ensure compliance with applicable regulations. (Recommendation 2) The Senior Procurement Executive for the Department of the Navy should review the contracts we identified as being awarded to contractors that reported qualifying federal tax debt under FAR § 52.209-11 and (1) determine whether the contracting officer was required to consider the contractor’s reported tax debt; if so, (2) determine the reasons controls to identify and refer these contractors to the SDO before contract award did not operate effectively; and (3) design or modify controls to help ensure compliance with applicable regulations. (Recommendation 3) The Senior Procurement Executive for VA should review the contracts we identified as being awarded to contractors that reported qualifying federal tax debt under FAR § 52.209-11 and (1) determine whether the contracting officer was required to consider the contractor’s reported tax debt; if so, (2) determine the reasons controls to identify and refer these contractors to the SDO before contract award did not operate effectively; and (3) design or modify controls to help ensure compliance with applicable regulations. (Recommendation 4) The Senior Procurement Executive for the Department of the Army should review the contracts we identified as being awarded to contractors that reported qualifying federal tax debt under FAR § 52.209-5. Specifically, the Senior Procurement Executive should determine whether each contract value was expected to exceed the simplified acquisition threshold when the solicitation was issued and, if so, (1) determine the reasons controls to identify and notify the SDO of these contractors before contract award did not operate effectively and (2) design or modify controls to help ensure compliance with applicable regulations. (Recommendation 5) The Senior Procurement Executive for DOE should review the contracts we identified as being awarded to contractors that reported qualifying federal tax debt under FAR § 52.209-5. Specifically, the Senior Procurement Executive should determine whether each contract value was expected to exceed the simplified acquisition threshold when the solicitation was issued and, if so, (1) determine the reasons controls to identify and notify the SDO of these contractors before contract award did not operate effectively and (2) design or modify controls to help ensure compliance with applicable regulations. (Recommendation 6) The Senior Procurement Executive for HHS should review the contracts we identified as being awarded to contractors that reported qualifying federal tax debt under FAR § 52.209-5. Specifically, the Senior Procurement Executive should determine whether each contract value was expected to exceed the simplified acquisition threshold when the solicitation was issued and, if so, (1) determine the reasons controls to identify and notify the SDO of these contractors before contract award did not operate effectively and (2) design or modify controls to help ensure compliance with applicable regulations. (Recommendation 7) The Senior Procurement Executive for the Department of the Navy should review the contracts we identified as being awarded to contractors that reported qualifying federal tax debt under FAR § 52.209-5. Specifically, the Senior Procurement Executive should determine whether each contract value was expected to exceed the simplified acquisition threshold when the solicitation was issued and, if so, (1) determine the reasons controls to identify and notify the SDO of these contractors before contract award did not operate effectively and (2) design or modify controls to help ensure compliance with applicable regulations. (Recommendation 8) The Senior Procurement Executive for VA should review the contracts we identified as being awarded to contractors that reported qualifying federal tax debt under FAR § 52.209-5. Specifically, the Senior Procurement Executive should determine whether each contract value was expected to exceed the simplified acquisition threshold when the solicitation was issued and, if so, (1) determine the reasons controls to identify and notify the SDO of these contractors before contract award did not operate effectively and (2) design or modify controls to help ensure compliance with applicable regulations. (Recommendation 9) The Administrator of GSA should coordinate with the appropriate SAM users, such as agency procurement officials, to identify potential updates to facilitate contracting officers’ identification of contractors that report qualifying federal tax debt under the § 52.209-11 representation and § 52.209-5 certification. (Recommendation 10) The Commissioner of the IRS should evaluate options to identify which contract payments federal agencies expect to be processed by the Fiscal Service, including amending the reporting requirements for Form 8596 to require federal agencies to include information about whether contractor payments are expected to be processed by the Fiscal Service. If the IRS amends Form 8596 reporting requirements, the IRS should (1) systematically note this information on taxpayer accounts to help the IRS identify which payments may be available for levy through the FPLP and which payments may be available for other (i.e., manual) levies and (2) analyze these data to help identify agencies that do not participate in the FPLP and inform its efforts to expand the number of agencies participating in the FPLP. (Recommendation 11) The Commissioner of the IRS should evaluate options to obtain comprehensive contract payment data above the existing FPDS-NG reporting threshold of $10,000, including assessing the costs and benefits of changing the current threshold for contracts that agencies are required to report to the IRS through Form 8596 information returns to be consistent with the existing reporting threshold for FPDS-NG, determine whether regulatory revisions are necessary, and change the reporting threshold, if appropriate. (Recommendation 12) We provided a draft of this report to the Department of Defense (for the Army and Navy), HHS, VA, DOE, GSA, the IRS and the Office of Management and Budget for review and comment. In written comments (reproduced in appendixes II–VI), the Department of Defense, HHS, VA, DOE, and GSA agreed with our recommendations. The IRS generally agreed with our recommendations (see appendix VII). The Office of Management and Budget had no comments. HHS and the Navy provided technical comments, which we incorporated as appropriate. The Department of Defense, HHS, VA, and DOE noted that they plan to review the contract awards identified in our review. In addition, several agencies described steps they will be taking to address our recommendations. For example, the Department of Defense noted that it plans to take corrective actions or add controls as necessary. HHS noted that it will assess internal controls and take appropriate action. VA noted that it will provide an action plan. DOE noted that it will design or modify controls for regulatory compliance, if necessary. GSA noted that it will work with the procurement community through established governance channels to identify potential approaches for drawing contracting officers’ attention to qualifying federal tax-debt information reported in SAM. The IRS noted its commitment to obtaining accurate information on potential levy sources and, accordingly, indicated it will review the benefits of expanding the information included on its Form 8596, along with other alternatives, to determine their feasibility, effectiveness, and relative burden. The IRS further noted that it will review the potential benefits and costs that would result from changing the current reporting threshold for contract payments, and submit its findings to the Office of IRS Chief Counsel to consider this addition to the IRS Priority Guidance Plan. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Health and Human Services, the Secretary of Veterans Affairs, the Secretary of Defense, the Secretary of the Navy, the Secretary of the Army, the Secretary of Energy, the Administrator of GSA, the Commissioner of Internal Revenue, the Director of the Office of Management and Budget, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6722 or shear@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. This report first examines the extent to which, in calendar years 2015 and 2016, (1) selected federal agencies had control activities that ensured contractors’ reported federal tax debts were considered before contract award. The remainder of the report assesses the same period; however, it focuses on all executive-branch agencies and examines the extent to which (2) federal contracts were awarded to contractors with federal tax debt, including the characteristics of those contracts and contractors, and (3) the Internal Revenue Service (IRS) identified selected federal contractors’ payments for levy. To identify the extent to which selected federal agencies had control activities that ensured contractors’ reported federal tax debts were considered before contract award (including task orders), we analyzed contract obligation information from the Federal Procurement Data System–Next Generation (FPDS-NG) and selected for our review the five agencies with the highest contract obligations associated with contract awards for 2015 and 2016, which is the period when contract award data were available at the time of our review. In addition, the revised FAR tax-debt provision went into effect during this period. Specifically, we selected the three civilian agencies with the highest obligations—the Departments of Energy (DOE), Health and Human Services (HHS), and Veterans Affairs (VA)—and, within the Department of Defense, the two agencies with the highest obligations—the Departments of the Army and Navy. The results of our review of these five selected agencies are not generalizable to all federal agencies. However, these five selected agencies awarded about 51 percent of contract obligations associated with contract awards for 2015 and 2016, which were the most-recent contract award data available at the time of our review, and during this period the newest Federal Acquisition Regulation (FAR) tax-debt provision was implemented. We reviewed selected agencies’ policies and procedures related to awarding contracts to prospective contractors that report they owe certain tax debts and met with agency officials to discuss how their agencies consider contractors’ reported federal tax debt before awarding a federal contract. Specifically, we met with agency officials who supervise contracting officers, such as the Head of Contracting Activity, Director of Contracts, or other contracting managers, policy and procurement officials, and suspension and debarment officials from the selected agencies. Additionally, we reviewed and analyzed applicable laws and regulations, as well as applicable policies and procedures from DOE, HHS, VA, the Navy, and the Army for considering contractors’ reported federal tax debt when awarding federal contracts. In addition, we interviewed staff from the Office of Management and Budget’s Office of Federal Procurement Policy and officials from the Interagency Suspension and Debarment Committee, and the Civilian Agency Acquisition Council to obtain an understanding of how the law is implemented through the FAR. We also met with the General Services Administration (GSA) to obtain an understanding of the System for Award Management (SAM), including the registration of prospective contractors and their reporting of certain federal tax debt to the representation requirement of FAR § 52.209-11 and the certification of § 52.209-5. As part of this work, we analyzed FPDS-NG contract award and SAM contractor registration data to identify instances where contractors reported having certain qualifying federal tax debt and received a contract award (including task orders). Specifically, we electronically matched FPDS-NG contract award data from 2015 and 2016 to the relevant contractors’ SAM registration. We then analyzed the relevant contractors’ representations and certifications most recently updated in SAM before the relevant contract award to identify all instances where contractors reported that they had a federal tax debt as defined in FAR § 52.209-11 or § 52.209-5 within our time frame. From the resulting list, we identified the contracts that selected agencies awarded to contractors that reported these qualifying federal tax debts. In addition, we reviewed a nongeneralizable sample of 15 contract awards selected from the five selected agencies to provide illustrative examples of the extent to which these agencies’ control activities ensured required actions were taken before contract award. These 15 contract awards were selected based on numerous criteria, including the prospective contractors’ (1) responses under FAR § 52.209-11 or § 52.209-5 in SAM before the new contract award, and (2) having tax debts as of December 15, 2016, that were not in a repayment agreement with the IRS. Further, when selecting contract awards that had a § 52.209-5 certification, we considered only contractors having at least $3,500 in tax debts as of December 15, 2016. We identified the relevant contractor population and then considered the following factors simultaneously to select the 15 case examples: unique contractor Taxpayer Identification Number across selected agency contracting office locations, the amount of tax debt owed by the prospective contractor, the amount of award obligations, and IRS assessment of a Trust Fund Recovery Penalty (TFRP). We selected case examples that represent a variety of these factors. We reviewed seven contract awards made to contractors that reported that they had certain tax debts and eight contract awards made to contractors that reported that they did not have certain tax debts as part of their § 52.209-11 representations and § 52.209-5 certifications in SAM. For these 15 contract awards, we reviewed pre–contract award documentation, which included tax debt–related representations and certifications retrieved by the selected agencies from SAM, and copies of historical tax transcripts and other records, such as revenue officers’ notes obtained from the IRS. For the case examples presented in this report, we rounded tax debt and contract obligation amounts, did not identify the awarding agency, and did not meet with awarding agency officials to discuss each contract award to protect sensitive taxpayer information. To determine the extent to which executive-branch agency contracts were awarded in 2015 and 2016 to federal contractors with federal tax debt, and characteristics of those contract awards and contractors, we electronically matched data from FPDS-NG on contract awards (including task orders) for all executive agencies with (1) data from SAM on contractors’ representations and certifications of their tax debt, and (2) data from the IRS on tax debts owed by these contractors. Specifically, we used the Data Universal Numbering System number to match data from FPDS-NG with contractor registration data from SAM to obtain additional information on these contractors, such as the contractors’ Taxpayer Identification Numbers and their representations and certifications of tax debt. Using the contractor Taxpayer Identification Number from SAM, we then matched our list of contractors with IRS data to identify our population of contractors that received a contract award and had unpaid federal tax debts. Our analysis included all of the executive-branch agencies. Further, our analysis describes some of characteristics of these debts, including the total amount of unpaid taxes, whether the contractors had a TFRP, and whether or not contractors had unpaid taxes that were timely paid or appeared to be finally determined, as of December 15, 2016, which was the time of our data extract. We also analyzed whether contractors that were assessed unpaid taxes in the IRS data reported having certain tax debts as part of their § 52.209- 11 representations and § 52.209-5 certifications in SAM. We reviewed the most-recent § 52.209-11 representation and § 52.209-5 certification prior to the relevant contract award. Our analysis may understate the population of contractors with tax debt to the extent that contractors repaid their tax debts before the timing of our data extract. Specifically, our analysis does not include any contractors that may have owed federal taxes at the time of a new contract award during this period, but that paid or otherwise resolved their tax debts before December 15, 2016. Additionally, our analysis focuses on contract awards made in 2015 and 2016, and not contract modifications made during this period. In 2015 and 2016, federal agencies obligated $400 billion in modifications to contracts made in 2014 or earlier, almost half of all federal contract obligations in this period. We identify contractors who potentially may have had federal tax debt meeting the definitions of tax debt under FAR § 52.209-11 and § 52.209-5 before the contract award, but cannot verify whether that was the case. To determine the extent to which the IRS identified selected federal contractors’ payments made for levy in 2015 and 2016, we identified the population of contractors that owed taxes at the same time they received a contract award during our period by matching FPDS-NG, SAM, and IRS Unpaid Assessment data, as described above. We then determined whether the tax debt had ever been levied or blocked by the Federal Payment Levy Program (FPLP) as of December 15, 2016, according to IRS data. We also interviewed IRS officials about levying federal contractor payments and reviewed Internal Revenue Manual sections and other relevant documents from the IRS. We assessed the reliability of FPDS-NG, SAM, and IRS Unpaid Assessment data by reviewing relevant documentation, interviewing knowledgeable agency officials, and performing electronic testing to determine the validity of specific data elements in the databases. We determined that these databases were sufficiently reliable for the purposes of our reporting objectives. In additional to the individual named above, Jonathon Oldmixon (Assistant Director), Gloria Proa (Analyst-in-Charge), Jennifer Felder, and Albert Sim made significant contributions to this report. Also contributing to this report were Scott Hiromoto, Barbara Lewis, Heather Miller, James Murphy, and Elizabeth Wood.", "summary": "The federal government obligated approximately $507 billion on contracts in fiscal year 2017. Businesses, including federal contractors, pay billions of dollars in taxes each year. Some businesses, however, do not pay owed taxes, contributing to what is known as the tax gap. Federal contractors owe some of the taxes that contribute to the tax gap, and, since 2015, federal law prohibits agencies, under certain circumstances, from using appropriated funds to contract with those who have qualifying tax debt. The IRS also has authority to levy certain payments of contractors with qualifying federal tax debt. GAO was asked to review issues related to federal contractors and tax debt. Among other things, GAO examined whether, in calendar years 2015 and 2016, (1) selected federal agencies had control activities that ensured contractors' reported federal tax debts were considered before contract award and (2) the IRS levied selected federal contractors' payments. GAO analyzed contract and IRS data from 2015 and 2016 (the most-recent data available), reviewed five agencies that represent 51 percent of contract obligations, and reviewed seven awards to contractors reporting tax debt. The five selected agencies GAO reviewed have control activities—such as policies and procedures—to help ensure they consider qualifying federal tax debts as defined by Federal Acquisition Regulation (FAR) § 52.209-11 and § 52.209-5 before awarding contracts. However, these controls were potentially ineffective in ensuring compliance with relevant laws and regulations. According to GAO's analysis, in 2015 and 2016 the Departments of Energy, Health and Human Services, and Veterans Affairs, and the Army and Navy, awarded 1,849 contracts to contractors that reported qualifying federal tax debts, such as delinquent debts over $3,500 (see table). When a contractor reports qualifying tax debts under these regulations, the contracting officer must take several actions, including notifying the agency suspension and debarment official (SDO). However, SDOs at all five agencies told GAO they did not receive any notifications of contractors reporting tax debt in this period. As a result, these contracts may have been awarded without potential required actions, indicating potential violations of federal regulations and, in some cases, appropriations law. GAO's nongeneralizable review of seven contracts illustrate two cases where contractors were collectively awarded more than $510,000 in contract obligations while having more than $250,000 in tax debt, including tax penalties for willful noncompliance with tax laws. Officials from the selected agencies were unable to explain why their control activities were potentially ineffective without reviewing each contract to determine whether FAR requirements were applicable and whether control activities were applied. Understanding why existing control activities did not operate effectively will help these agencies enhance controls to avoid future misuses of appropriated funds. GAO plans to provide information on the instances of potential noncompliance GAO identified to the selected agencies. Of the over 2,700 executive-branch contractors GAO found to have likely qualifying federal tax debt as of December 2016, the Internal Revenue Service (IRS) had identified over 2,000 for levy through its automated Federal Payment Levy Program (FPLP). However, the FPLP cannot levy all contractors because not all payments are processed by the system the FPLP uses. The data the IRS receives from agencies does not allow it to readily identify payments made using other systems—information the IRS needs for agency outreach about inclusion in the FPLP and to more quickly initiate a manual levy. With this information, the IRS may be able to improve its levy capacity and enhance tax collections. GAO is making 12 recommendations, including that selected agencies enhance controls for considering contactors' qualifying federal tax debt before awarding contracts and that the IRS evaluate options to obtain comprehensive contract-payment information. All the agencies generally agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "The federal government faces long-standing challenges in strategically managing its workforce. We first added federal strategic human capital management to our list of high-risk government programs and operations in 2001. Because skills gaps within individual federal agencies—as well as across the federal workforce—can lead to costly, less-efficient government, the issue has been identified as the focus of the Strategic Human Capital Management GAO high-risk area since February 2011. Our experience has shown that the key elements needed to make progress in high-risk areas are top-level attention by the administration and agency leaders grounded in the five criteria for removal from the High-Risk List, as well as any needed congressional action. The five criteria for removal are: (1) leadership commitment, (2) agency capacity, (3) existence of a corrective action plan, (4) program monitoring, and (5) demonstrated progress. Although Congress, OPM, and individual agencies have made improvements since 2001, federal human capital management remains a high-risk area because mission-critical skills gaps within the federal workforce pose a high risk to the nation. GAO, along with OPM and individual agencies, has identified mission critical skills gaps in numerous government-wide occupations. These skills gaps both within federal agencies and across the federal workforce impede the government from cost-effectively serving the public and achieving results. For example, the difficulties in recruiting and retaining skilled health care providers and human resource staff at Veterans Health Administration’s (VHA) medical centers make it difficult to meet the health care needs of more than 9 million veterans. As a result, VHA’s 168 medical centers have large staffing shortages, including physicians, registered nurses, physician assistants, psychologists, physical therapists, as well as human resource specialists and assistants. In October 2017, we reported that the VHA, within the Department of Veterans Affairs (VA), has opportunities to improve staffing, recruitment, and retention strategies for physicians that it identified as a priority for staffing, or mission-critical. For 2016, the top five physician mission- critical occupations were primary care, mental health, gastroenterology, orthopedic surgery, and emergency medicine. However, VHA was unable to accurately count the total number of physicians who provide care in its VA medical centers (VAMC). Additionally, VHA lacked data on the number of contract physicians and physician trainees. Five of the six VAMCs in our review used contract physicians or physician trainees to meet their staffing needs, but VHA had no information on the extent to which VAMCs nationwide use these arrangements. We also reported that VHA had not evaluated the effectiveness of its physician recruitment and retention strategies. One such strategy—hiring physician trainees—was weakened by ineffectual hiring practices, such as delaying employment offers until graduation. In February 2018, we reported that the Department of Homeland Security (DHS) had taken actions to identify, categorize, and assign employment codes to its cybersecurity positions, as required by the Homeland Security Cybersecurity Workforce Assessment Act of 2014; however, its actions were not timely and complete. While DHS has implemented four of our six recommendations from this report, two recommendations remain open. For example, DHS has not yet completed its efforts to identify all of the department’s cybersecurity positions and accurately assign codes to all filled and vacant cybersecurity positions. Further, it has not yet fully developed guidance to assist DHS components in identifying their cybersecurity work categories and specialty areas of critical need that align to the National Initiative for Cybersecurity Education framework. Without ensuring that its progress in identifying and assigning codes to its positions is accurately reported and it has guidance to fully assist components, DHS will not be positioned to effectively examine its cybersecurity workforce, identify its critical skill gaps, or improve its workforce planning. In March 2019, we reported that 24 federal agencies generally assigned work roles to filled and vacant positions that performed information technology, cybersecurity, or cyber-related functions as required by the Federal Cybersecurity Workforce Assessment Act of 2015. However, most agencies had likely miscategorized the work roles of many IT positions. Until agencies accurately categorize their positions, the agencies may not have reliable information to form a basis for effectively examining their cybersecurity workforce, improving workforce planning, and identifying their workforce roles of critical need. Skills gaps caused by insufficient number of staff, inadequate workforce planning, and a lack of training in critical skills are contributing to our designating strategic human capital management and other areas as high risk. (See table 1.) Skills gaps affect individual agencies but also cut across the entire federal workforce in areas such as cybersecurity and acquisition management. As our 2019 analysis of federal high-risk areas shows, in addition to Strategic Human Capital Management, skills gaps played a role in 16 of the other 34 high-risk areas we have identified. Insufficient numbers of staff with critical skills can be related to staff retirements as well as to inadequate recruitment and hiring. Moreover, if not carefully managed, anticipated retirements could widen skills gaps or open new ones, adversely affecting agencies’ capabilities. As shown in figure 1, more than 31 percent of federal employees on board by the end of fiscal year 2017 will be eligible to retire in the next 5 years. In March 2019, we reported on key talent management strategies that can help agencies better manage the current and future workforce. Below we focus on nine selected practices we identified related to recruiting, incentivizing, and engaging the federal workforce: Cultivate a diverse talent pipeline. In our prior work, we have noted the importance of active campus recruiting that goes beyond infrequent outreach to college campuses. Active campus recruiting includes developing long-term institutional relationships with faculty, administrators, and students. In addition, OPM guidance emphasizes that agencies should develop an inclusive approach to their talent acquisition strategies. This includes developing strategic partnerships with a diverse range of colleges and universities, trade schools, apprentice programs, and affinity organizations from across the country. Recruit continuously and start the hiring process early in the school year. The ability to hire students is critical to ensuring that agencies have a range of experience levels for succession planning and a talent pipeline to meet mission requirements. One of the key challenges agencies face in recruiting students is managing the timing of recruitment. The federal fiscal year begins on October 1—about when private sector firms we interviewed start recruiting on campus. Frequently, however, federal agencies have been unable to hire at this time of year because of the limitations of budget uncertainty. Yet if agencies wait to start the recruiting and hiring process until they receive funding, many graduates will have taken other job opportunities. Agencies can overcome these timing challenges by recruiting continuously and starting the hiring process early in the school year. To recruit continuously, Chief Human Capital Officers (CHCOs) from the U.S. Departments of Agriculture and Homeland Security said that they advertise funding-conditional positions throughout the year. Write user-friendly vacancy announcements. We previously reported that some federal job announcements were unclear. This can confuse applicants and delay hiring. In July 2018, OPM officials stated that agencies can develop more effective vacancy announcements when hiring managers partner with human resource (HR) staff. According to OPM, hiring managers can work with HR staff to identify the critical competencies needed in the job, develop a recruiting strategy, and ensure the job announcement accurately and clearly describes the required competencies and experience. To promote collaboration between hiring managers and HR staff, OPM is training agencies on the role of hiring managers in writing vacancy announcements, according to OPM officials. Strategically leverage available hiring and pay flexibilities. To help ensure agencies have the talent they need to meet their missions, we have found that federal agencies should have a hiring process that is simultaneously applicant friendly, sufficiently flexible to enable agencies to meet their needs, and consistent with statutory requirements, such as hiring on the basis of merit. Key to achieving this is the hiring authority used to bring applicants onboard. In March 2019, we reported that CHCOs cited the complex competitive examining process as a cause of the lengthy hiring time. This has been a long-standing concern. In our 2002 report on human capital flexibilities, we noted that for many years prior, federal managers had complained that competitive examining procedures were rigid and complex. Agencies can use a number of additional hiring authorities beyond competitive examining. These authorities can add flexibility to the process and CHCOs have expressed a desire for more. However, we previously found that agencies relied on only a small number of available authorities. In fiscal year 2014, of the 105 hiring authority codes used in total, agencies relied on 20 hiring authority codes to make around 90 percent of the new appointments. We recommended in 2016 that OPM use information from its reviews of agencies’ use of certain hiring authorities to determine whether opportunities exist to refine, consolidate, or expand agency-specific authorities, and implement changes where OPM is authorized, including seeking presidential authorization or developing legislative proposals if necessary. OPM agreed with our recommendation and has made progress in these areas, although more work is needed to follow through on planned actions to streamline authorities. For example, in December 2018, OPM said that it continues to research and examine streamlining opportunities, such as those identified in its July 2018 study on excepted service hiring authorities. However, OPM did not provide a time frame for implementation. In addition, in its March 2019 Congressional Justification for the Fiscal Year 2020 Budget Request, OPM included legislative proposals for new hiring authorities, such as authority for short-term appointments to allow agencies to appoint and compensate highly qualified experts to help agencies meet critical needs as well as a change to the criteria for granting direct hire authority. A variety of special pay authorities can help agencies compete in the labor market for top talent, but agencies only use them for a small number of employees. In fiscal year 2016, these incentives were used for less than 6 percent of employees. In December 2017, we reported that agencies can tap an array of special payments when they need to recruit or retain experts in engineering, cybersecurity, or other in-demand fields. These payments include, for example, incentives for recruitment or retention, or higher rates of pay for critical positions. We found that agencies reported that these payments were helpful, but few documented their effects, and OPM had not assessed their effectiveness. Further, in our March 2019 report, we found that less than 5 percent of employees received payments for recruitment or retention annually in the past 10 years. In December 2017, we made three recommendations to OPM, including for it to track the effectiveness of special payment authorities. OPM partially concurred with this recommendation, saying that agencies are in the best position to take this action. In December 2018, OPM stated that it established a baseline to measure changes in the use of special payment authorities over time, and that it is focused on government-wide, mission- critical occupations to help identify trends where there may be recruitment and retention difficulties. However, documents OPM provided gave no information on actions taken on this recommendation. We will continue to monitor OPM’s actions to implement this recommendation. This is one of 18 priority recommendations in GAO’s Priority Recommendations letter to OPM. Use relevant assessment methods and share hiring lists. In March 2019, we reported that CHCOs and OPM officials we interviewed stated that roadblocks to hiring the right skills include issues with assessment methods. Specifically, agencies may use methods that are less relevant for assessing the desired skills or agencies may experience issues incorporating multiple assessments in the hiring process. For example, one CHCO we interviewed said that her agency uses multiple-choice questions to assess candidates, but essay questions more effectively assess the skills she seeks. OPM issued guidance to agencies on how to use additional assessment methods, including how to rank applicants. Additionally, federal employee and management group representatives we spoke with said agencies could reduce the time of the assessment process by sharing hiring lists. The Competitive Service Act of 2015 allows agencies to share hiring lists, but agencies have only started to pilot the practice within departments, according to OPM officials we spoke with for our March 2019 report. OPM and agencies discussed sharing hiring certificates with the CHCO Council, and OPM is planning virtual training sessions on this topic. However, one federal employee group representative noted that to be consistent with merit principles, agencies may need to refresh the list every 2 to 3 months to give new candidates the opportunity to enter the application pool. Highlight agency mission and link to employees’ work. Agencies can help counter negative perceptions of federal work by promoting their missions and innovative work, according to experts and CHCOs we interviewed for our March 2019 report. For example, DHS’s CHCO told us that DHS provides “Day in the Life” information on its work to promote public awareness of how its everyday tasks tie in with its mission of protecting the United States. In addition, we have previously reported that high-performing organizations create a “line of sight” between individual performance and organizational results by aligning employees’ daily activities with broader results. Agencies can motivate and retain employees by connecting them to their agency’s mission, according to human capital experts and federal employee and management group representatives we interviewed. Employee responses to Federal Employee Viewpoint Survey (FEVS) indicate the federal government appears to be performing well in this area. In 2017, 84 percent of employees knew how their work related to the agency goals and priorities. Increase awareness of benefits and incentives, such as work-life programs. As shown in figure 2, the majority of federal employees were satisfied with compensation, and employees who participated in work-life programs were satisfied with those incentives. However, OPM’s 2018 Federal Work-Life Survey Governmentwide Report found that one of the most commonly reported reasons employees do not participate in work- life programs is lack of program awareness among employees and supervisors. Increase support for an inclusive work environment. An increasingly diverse workforce can help provide agencies with the requisite talent and multidisciplinary knowledge to accomplish their missions. In January 2005, we reported fostering a diverse and inclusive workplace could help organizations reduce costs by reducing turnover, increasing employee retention across demographic groups, and improving morale. We also reported that top management commitment is a fundamental element in the implementation of diversity management initiatives. Encourage details, rotations, and other mobility opportunities. In March 2019, we stated that CHCOs, human capital experts, and federal management groups said upward and lateral mobility opportunities are important for retaining employees. CHCOs also said that in some cases, lateral mobility opportunities such as rotations, details, and opportunities to gain experience in other sectors can help employees gain new skills more cost-effectively than training, particularly for rapidly changing skill sets such as those related to the sciences. Further, we previously reported that effective interagency rotational assignments can develop participants’ collaboration skills and build interagency networks. However, according to OPM data, few employees in 2017 moved horizontally because, according to federal manager group representatives and our previous work, managers are sometimes reluctant to lose employees. (See table 2.) We have previously made recommendations that could help address these challenges. For example in 2015, we recommended that OPM determine if promising practices, such as providing detail opportunities or rotational assignments to managerial candidates prior to promotion, should be more widely used across government. OPM partially concurred with this recommendation and agreed to work with the CHCO Council to explore more government-wide use of rotational assignments. However, OPM noted that agencies already have authority to take these actions. In June 2019, OPM officials told us they had discussed the scalability of promising practices for supervisors—specifically, details and rotational assignments and a dual career ladder—with members of the CHCO Council. OPM found these practices were being used at some agencies, but has not determined if these practices may be beneficial to other agencies. In conclusion, OPM has instituted numerous efforts to assist agencies’ in addressing mission-critical skills gaps within their workforces. This includes providing guidance, training and on-going support for agencies on the use of comprehensive data analytic methods for identifying skills gaps and the development of strategies to address these gaps. However, as of December 2018, OPM had not fully implemented 29 of our recommendations made since 2012 relating to this high-risk area. We will continue to monitor OPM’s efforts to implement our recommendations. Further, we have reported on numerous talent management strategies that can help agencies better manage the current and future workforce. Without these measures, the federal government’s ability to address the complex social, economic, and security challenges facing the country may be compromised. Chairman Lankford, Ranking Member Sinema, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff have any questions about this testimony, please contact Yvonne D. Jones, Director, Strategic Issues, at (202) 512-6806 or jonesy@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Clifton Douglas, Jr., Assistant Director; Christopher Falcone; Karin Fangman; Cindy Saunders, Alan Rozzi and Katherine Wulff. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Strategic human capital management plays a critical role in maximizing the government's performance and assuring its accountability to Congress and to the nation as a whole. GAO designated strategic human capital management as a government-wide, high-risk area in 2001. Since then, important progress has been made. However, retirements and the potential loss of leadership and institutional knowledge, coupled with fiscal pressures, underscore the importance of a strategic and efficient approach to acquiring and retaining individuals with critical skills. As a result, strategic human capital management remains on GAO's High-Risk List. This testimony is based on a large body of GAO work issued from May 2008 through May 2019. This testimony, among other things, focuses on key human capital areas where some actions have been taken but attention is still needed by OPM and federal agencies on issues including: (1) addressing critical skills gaps and (2) recruiting and hiring talented employees. GAO, along with the Office of Personnel Management (OPM) and individual agencies, has identified skills gaps in numerous government-wide occupations. According to GAO's 2019 analysis of federal high-risk areas, skills gaps played a role in 17 of the 35 high-risk areas. Causes vary but these skills gaps often occur due to shortfalls in one or more talent management activities such as robust workforce planning. Staffing shortages and the lack of skills among current staff not only affect individual agencies but also cut across the entire federal workforce in areas such as cybersecurity and acquisition management. Additionally, the changing nature of federal work and the high percentage of employees eligible for retirement could produce gaps in leadership and institutional knowledge, and threatens to aggravate the problems created from existing skills gaps. For example, 31.6 percent of permanent federal employees who were on board as of September 30, 2017, will be eligible to retire in the next 5 years with some agencies having particularly high levels of employees eligible to retire. GAO's work has identified a range of problems and challenges with federal recruitment and hiring efforts. Some of these problems and challenges include unclear job announcements and a lengthy hiring process. Further, the federal workforce has changed since the government's system of current employment policies and practices were designed. Strategies that can help agencies better manage the current and future workforces include: Manage the timing of recruitment . To address issues of funding uncertainty at the beginning of the fiscal year, agencies should recruit continuously, starting the hiring process early in the school year. Write user-friendly vacancy announcements . GAO has reported that some federal job announcements were unclear. This can confuse applicants and delay hiring. OPM stated that when hiring managers partner with human resources staff, agencies can develop more effective vacancy announcements. Leverage available hiring and pay flexibilities . To help ensure agencies have the talent they need, they should explore and use all existing hiring authorities. A variety of special pay authorities can help agencies compete in the labor market for top talent, but GAO has found that agencies only use them for a small number of employees. Increase support for an inclusive work environment . An increasingly diverse workforce can help provide agencies with the requisite talent and multidisciplinary knowledge to accomplish their missions. Encourage rotations and other mobility opportunities. Upward and lateral mobility opportunities are important for retaining employees, but few employees move horizontally because managers are sometimes reluctant to lose employees. Without these measures, the federal government's ability to address the complex social, economic, and security challenges facing the country may be compromised. Over the years, GAO has made numerous recommendations to agencies and OPM to improve their strategic human capital management efforts. Agencies have taken actions to implement some of these recommendations, but many remain open. GAO encourages OPM and the agencies to fully implement the recommendations.", "document_type": "gao"}
{"report": "VA has taken several steps in recent years to reduce health care disparities for minority veterans and advance health equity, but lacks mechanisms to ensure accountability for advancing health equity using these or other actions. In particular, VA established a program office and a dedicated steering committee to draft VA’s first action plan designed to achieve health equity. However, this initial action plan lacked performance measures and clear lines of accountability; as such, the extent to which it has been implemented and the progress made in achieving its goals is unknown. In 2012, VA established OHE to lead the department’s efforts to advance health equity and reduce health disparities throughout VA’s health care system. At that time, the Principal Deputy Under Secretary for Health— who reports to the Under Secretary for Health—identified health equity as a cross-cutting issue of the highest priority, and as such, he required the Director of OHE to report directly to him. OHE is responsible for several efforts, including providing education, training, research, communications and information; promoting common awareness about health care disparities and working to improve health care outcomes; and representing VA and serving as a liaison to other governmental and non- governmental organizations working to achieve health equity. OHE was also tasked with developing and maintaining a comprehensive action plan to achieve health equity in VA medical centers and improve VA’s overall quality of care. Also in 2012, VA’s Principal Deputy Under Secretary for Health created the Health Equity Coalition (HEC). Chaired by the Director of OHE, the HEC is a VA-wide steering committee that is comprised of officials from several VA program offices dedicated to areas such as patient care services, communications, research and development, and minority veteran issues. The HEC advises and assists OHE in developing and implementing plans, sets milestones to review progress to ensure timely completion of initiatives, and ensures that program offices commit appropriate organizational resources needed to meet these goals. Since its establishment, OHE has experienced changes in budget and staff levels from year to year. OHE’s core budget, which VA officials told us is spent on salaries, travel, and supplies, fluctuated between 2013 and 2019; staffing levels began decreasing in 2014 and subsequently increased in 2019. (See fig. 1.) VA officials told us that staff reductions were due to transfers and reassignments of OHE staff to other offices and positions, which coincided with a shifted focus from equity issues to other issues such as veteran wait times and modernization efforts. In addition, OHE was also repositioned to a lower level within the organization from VHA’s Office of the Principal Deputy Under Secretary for Health to the Office of the Deputy Under Secretary for Health for Organizational Excellence. In February 2014, VA released its first Health Equity Action Plan (action plan), drafted by OHE and the HEC to document VA’s approach for eliminating health disparities and achieving health equity. The action plan included five focus areas, or goals, in which VA intended to direct its efforts to improve the overall quality of care for all veterans, including minority veterans: Awareness: increase awareness of the significance of health disparities, their impact on the nation, and the actions necessary to improve health care outcomes for racial, ethnic, and underserved populations. Leadership: strengthen and broaden leadership for addressing health disparities at all levels. Health system and life experience: improve health and health care outcomes for racial, ethnic, and underserved populations. Cultural and linguistic competency: improve cultural and linguistic competency, and the diversity of the health-related workforce. Data, research, and evaluation: improve data availability; and coordination, utilization, and diffusion of research and evaluation outcomes. OHE and the HEC included between two to 14 “implementation activities” in each of the five focus areas to describe specific plans and tasks VA could undertake to advance health equity. For example, under the “data, research, and evaluation” focus area, examples of implementation activities included, “identify limitations of existing data, barriers to access to data, and data collection methodologies that affect VA’s ability to describe disparities,” and “develop a strategy for prioritizing identified disparities.” In addition, all five focus areas also included “success criteria” to measure success, a list of resources needed in order implement the activities, and identified stakeholders. However, despite documenting elements needed to make improvements in the five focus areas, the extent of VA’s progress in implementing the action plan and advancing health equity is unknown because the action plan lacks performance measures and clear lines of accountability. Performance measures and clear lines of accountability are among the criteria identified in GAO’s body of work on effectively managing performance under GPRA. Without such mechanisms, VA cannot be assured that the plan has been implemented or will ultimately be effective in addressing health equity. In particular, we found that VA’s plan did not include the following mechanisms: Performance measures. Our past work on effectively managing performance shows that performance measures should demonstrate how well the organization is meeting its goals and should be linked directly to offices that have responsibility for the program or activity. As previously noted, although VA’s action plan included a list of “success criteria”, such criteria were not measurable, and were not directly linked to the specific implementation activities or to the responsible lead office for any of the five focus areas. For example, under the “leadership” focus area, the action plan identified “development of process tools for monitoring in FY 2014” as one of the success criteria, but it was not clearly linked to one of this focus area’s specific implementation activities and did not identify who among the list of lead offices and stakeholders was responsible for it. under the “data, research, and evaluation” focus area, the action plan identified “developed standards for collecting data used to understand disparities” and “improved on-going data sharing between programs” as two of the success criteria, but they were not linked to one of this focus area’s implementation activities, nor were they specifically assigned to one or more of the lead offices and stakeholders listed as responsible for achieving them. Clear lines of accountability: Our past work on effectively managing performance also shows that designating a lead official or office to be responsible for coordinating efforts to achieve results for each goal or action creates clear lines of accountability. This is critical to implementing change to achieve goals and marshaling resources needed to improve management. In contrast, VA’s action plan listed for each of the five focus areas: a broad group of lead offices and stakeholders responsible for the entire focus area, in general, (for example, HEC members and their respective offices, VISN officials, and VA medical center directors) but did not designate specific offices or officials responsible for coordinating efforts to complete specific implementation activities. vaguely described resources—such as leadership support, time, money, and travel—needed to execute all of the implementation activities under each of the five focus areas, but did not designate specific lead offices or stakeholders responsible for committing specific resources needed to implement each activity. Without such performance measures or lines of accountability, VA lacked the means to measure specific progress in implementing and achieving the action plan’s goals. Moreover, according to VA officials, following the release of the action plan and the reduction in number of OHE staff, the frequency of HEC meetings decreased and the last regular meeting before it reconvened in January 2019 occurred in early 2015. As such, VA officials told us that there was no formal involvement or oversight following the release of the action plan to ensure that coalition members were meeting their responsibilities, including committing the organizational resources needed to ensure implementation. In recent years, there have been several recommendations from stakeholders, urging VA to provide OHE with the resources needed to fully implement its action plan. Specifically, the 2016 Commission on Care report recommended that VA commit additional resources to address the causes of the problem and ensure the action plan is fully implemented. The Secretary of VA at the time concurred with the Commission’s recommendation and said that VA would identify health equity leaders and clinical champions in each VISN and VA medical center who could catalyze and monitor actions to implement the action plan and further advance the elimination of health disparities. More recently, VA has signaled renewed interest in supporting the advancement of health equity by increasing OHE’s budget and staffing levels in fiscal year 2019, and reconvening the HEC in January 2019. According to OHE officials, the reconvened HEC has held regular meetings and approved an updated action plan in September 2019. In October 2019, OHE officials told us that the action plan had been sent to VHA leadership for review, which they anticipated would be completed within the first fiscal quarter of 2020. Both OHE and other VA programs fund research conducted by VA clinicians and staff to identify disparities in health care outcomes. However, VA officials and researchers have noted problems with the completeness and accuracy of the data on veterans’ race and ethnicity. These weaknesses, in turn, limit VA’s ability to assess and address health care disparities at the VA medical center level. VA funds research aimed at identifying health care outcome disparities involving minority veterans. According to OHE officials, annually OHE receives a research budget separate from its core budget, and can apply monies from this separate budget to any appropriate research activities it wishes to support. As an example, in fiscal year 2019, OHE officials told us it provided funds to the Quality Enhancement Research Initiative, VA’s Center for Health Equity, Research, and Promotion, and two VISNs. In addition to OHE-funded research, VA’s Health Services Research & Development (HSR&D) has spent about $12 million to fund research studies related to identifying and reducing disparities in health care outcomes between minority and other veterans since 2014. This research has identified disparities in health care outcomes for minority veterans. Research funded by HSR&D includes the following studies: A 2017 report focused on whether changes in the way VA delivered primary care were effective in addressing racial and ethnic disparities in health care outcomes. Using VA data from 2009 and 2014, the study found lower rates of control of hypertension and diabetes among veterans who were African American, Hispanic, American Indian/Alaska Native, and Native Hawaiian and other Pacific Islanders compared with White veterans. A 2016 report examined why minority veterans with mental health and substance abuse disorders are less likely to use mental health and substance abuse services, and to complete mental health and substance abuse treatment. The study, which used 2013 data, found health disparities between White veterans and Black, Hispanic, and American Indian or Alaskan Native veterans with mental health and substance abuse disorders on several quality measures, including access to care. The study also found disparities by race and ethnicity in patients’ experiences communicating with providers and office staff. A 2017 systematic review of 351 studies published between 2006 and February 2016 examined the prevalence of disparities in health care outcomes experienced by veterans, including health disparities based on race and ethnicity. This systematic review concluded that a large proportion of the research conducted has focused on differences between Black or African American and White veterans and suggested that future targeted research is needed to capture the unique characteristics of American Indian or Alaska Natives and Native Hawaiian or other Pacific Islanders. Despite VA’s funding of numerous studies to identify health disparities and to explore interventions to potentially reduce or eliminate them, health disparities continue to persist among VA’s patient population. HSR&D officials told us that VA has faced difficulties translating research into practice in clinical settings, including their research findings about disparities in health care outcomes. HSR&D officials told us that they have recently undertaken new efforts aimed at implementing research findings, including those focused on disparities in health care outcomes. Among these efforts is the development of a new program to provide additional funding (for up to two years) for research projects that are completed or close to completion so that researchers can: 1) develop tool kits that others can adopt, and 2) implement research in additional VA medical centers in order to facilitate the sharing of information about successes and failures to make the impact of research more effective. Generally, VA collects and records race and ethnicity information in veterans’ EHRs when they enroll in VA health care online, by mail, fax, or telephone applications, or through self-service touch-screen kiosks at VA medical centers. Intake clerks may also collect and record race and ethnicity information when assisting veterans with enrollment, as well as when checking a veteran in at a clinic for an outpatient appointment, or registering a veteran for an inpatient hospital admission. However, VA researchers and officials have identified weaknesses in the completeness and accuracy of VA’s patient data on race and ethnicity, which has raised data reliability concerns. (See fig. 2) VA cannot ensure that race and ethnicity information labeled in the EHR as self-reported is accurate. VA follows standards outlined by the Office of Management Budget, which state that self-reported information is the preferred method for obtaining an individual’s race and ethnicity, because it is more accurate than data collected by observation of a third party. However, a VA data expert with HSR&D’s Center for Health Equity Research and Promotion, and officials at one of the VISNs in our review told us that they are aware that intake clerks sometimes enter information based on observation, which may be inaccurate, because they feel uncomfortable asking veterans for their race and ethnicity information in case the veterans find it offensive. Adding further to potential inaccuracies, because VA’s EHR default setting automatically records all race and ethnicity information as self-reported, observational data are only accurately labeled as such if a clerk manually changes the default setting to ‘observational’. The VA data expert from HSR&D’s Center for Health Equity Research and Promotion told us that, based on her research, almost all of the information collected electronically in the EHR is automatically assigned as self-reported, the default setting, even when it is collected by observation of VA staff. This expert also told us that research efforts at VA medical centers have indicated that the default setting is rarely changed and that some clerks had never changed the setting because they do not know how. VA research has indicated that observational data is more accurate for Blacks or African Americans and Whites than other racial groups, and that studies focused on other racial groups may be especially vulnerable to misclassification bias. As such, VA lacks reasonable assurance that the identification of race and ethnicity as “self-reported” is accurate. Data on veterans’ race are often incomplete. Race and ethnicity information is collected as two separate categories in the EHR, and as previously stated, is generally obtained when a patient enrolls in VA health care, or seeks care at a VA medical center or clinic. Two VA researchers told us that ethnicity data—based on veterans’ designation of whether they are Hispanic or non-Hispanic—are often more complete than race data. They said that one reason for this is that the order in which the questions are asked may be problematic; specifically, the ethnicity question is asked first, followed by a second question to request a race designation. Veterans may self-report that they are “Hispanic” upon enrollment or check-in for an inpatient admission or outpatient medical visit, and then leave the race field empty because they believe that they have already provided this information. Missing data on race impedes VA’s ability to identify potential disparities in health care outcomes. Conflicting race and ethnicity information in a veteran’s medical records makes it difficult to determine which information is accurate. According to VA researchers we spoke with, because a patient’s race and ethnicity information is uploaded from his or her EHR after each inpatient admission and outpatient appointment, there can be multiple records for each patient’s race and ethnicity data in VA databases. These patient records often conflict with one another, and may result from the use of both observational and self-reported data. As such, officials stated that it can be difficult to determine which of the multiple race or ethnicity records are accurate. To account for the issues with completeness and accuracy, VA researchers have used various approaches. VA researchers we spoke with told us that while they use data entered into VA’s EHR, which are then uploaded into various databases, they also must use a variety of methods, often time-intensive, to enable the use of race and ethnicity data due to concerns about its completeness and accuracy. These methods include using veterans’ patient records that may be several years old and from multiple VA health care settings, and looking at patient race and ethnicity information captured across multiple years and VA facilities. In addition, researchers also described using multiple non-VA data sources to supplement VA’s race and ethnicity information, such as Medicare data, and data from the Department of Defense’s roster of veterans from recent military operations. VA officials and other stakeholders representing veterans’ interests have recognized the weaknesses in VA’s race and ethnicity data and the importance of improving those data in order to address disparities and improve health equity. For example, VA’s first action plan included a goal to improve data availability, supported by implementation activities to “identify limitations of existing data, barriers to access to data, and data collection methodologies that affect VA’s ability to describe disparities” and “identify strategies for capturing data on race, ethnicity, language…needed to stratify the results for all quality measures and to address disparities.” in 2016, the Commission on Care recommended that VA increase the availability, quality, and use of race, ethnicity, and language data to improve the health of minority veterans, as well as utilize systems that monitor trends in health status, patient satisfaction, and quality measures. in its 2015 annual report, the Advisory Committee on Minority Veterans recommended that VA enhance its existing data collection processes to include the reporting of race and ethnicity data for all benefits and utilization programs to ensure the identification of delivery gaps and potential disparate levels of service. Furthermore, in its 2017 annual report, the Committee again highlighted ongoing concerns with VA’s inconsistency in collecting race and ethnicity data and stated that it impedes VA’s ability to adequately identify health disparities and to ensure minority veterans are receiving quality care and services throughout VA’s system. VA is currently collaborating with the Department of Defense to implement a new EHR system. As yet, they have not yet addressed how the EHR will store race and ethnicity information. The new EHR system is to provide both departments with a common EHR platform that is intended to support the provision of seamless care and create a single health record for service members and veterans. VA officials from the Office of EHR Modernization told us that this collaboration is still in the very early stages and that while race and ethnicity information will be included in the new EHR system, the new EHR will take 10 years to fully implement. Data weaknesses, including incomplete and inaccurate data have limited VA’s ability to advance health equity and patient care in its medical centers, according to VA officials. Unlike VA researchers, who report being able to account for missing and inaccurate race and ethnicity data, most VA medical centers do not have the research staff and data specialists needed for these efforts, according to a VA researcher from HSR&D’s Center for Health Equity Research and Promotion and officials from a VISN included in our review. As a result, the inaccurate and missing data have limited the ability of VISN and VA medical center staff to identify and address disparities in health care outcomes by race and ethnicity at the medical center level. VISN officials we spoke to discussed the challenges they encountered when trying to obtain complete and accurate data on health care outcomes by race and ethnicity to identify disparities involving their minority veteran populations. For example, one VISN official told us that she began an effort to analyze disparities in health care outcomes by race and ethnicity in fiscal year 2018, but encountered challenges in obtaining complete and accurate data for minority veterans. She said she contacted OHE officials for assistance, who provided data for diabetes and hypertension by race and ethnicity, but these data were 2 years old and available only at the national level. According to the VISN official, complete and accurate health care outcomes data by race and ethnicity were not available for minority veterans that received care in her region. The official told us that she subsequently contacted both VISN-level and national data specialists, but was still unable to obtain the data to assess health care outcomes by veterans’ race and ethnicity at the regional or local level. other VISN officials we spoke to explained that they had a similar experience when they explored using race and ethnicity data to design a dashboard for a VISN-funded project to track efforts to address disparities in ambulatory care readmissions involving minority veterans. They also told us they too contacted regional and national data specialists, but were told that the readmissions data were missing and inaccurate by race and ethnicity and therefore not useable for their efforts. VA officials told us they are taking steps to provide VA medical centers with data on health care outcomes at their facilities. These officials told us that they are currently developing two health equity dashboards that will use VA’s data on race and ethnicity to provide information on health care outcomes, which would allow VA staff to identify any disparities in these outcomes at the VISN and VA medical center levels. The two health equity dashboards are in different stages of development and, as of September 2019, VA did not have a timeline for completion and implementation across VA medical centers for either dashboard. According to VA officials, the development of these dashboards will not address the accuracy and completeness of the race and ethnicity data in the VA’s EHR. In order to maximize the effectiveness of these dashboards, VA needs to ensure that underlying data weaknesses are addressed by ensuring that race and ethnicity data in the EHR are complete and accurate. VA collects patient experience feedback from veterans, including minority veterans, through the following surveys: The Survey of Healthcare Experiences of Patients (SHEP) is VA’s national, standardized, and publically reported patient experience survey that comprises up to 83 questions that are used to collect information about patients’ experiences in various inpatient care settings. The SHEP covers topics to assess patients’ perceptions of their experience using the Consumer Assessment of Health Providers and Systems Survey, which is the health care industry standard. According to VA officials, the response rate for the SHEP is just under 40 percent, and on average, 95 percent of respondents complete survey questions about their race and ethnicity. SHEP survey results are reported publically at the national, VISN, and VA medical center level. VA does not report survey data for specific racial and ethnic groups because, according to VA officials, the number of minority veterans responding to the SHEP is too small to report. In 2012, a memo establishing the OHE recommended that VA send the SHEP survey to a higher number of veterans from racial and ethnic minority groups so there are enough responses to report survey results by those minority groups. However, VA officials told us that they were not aware of this recommendation and had not addressed it. Currently, VA officials told us that VA staff can access SHEP data by race and ethnicity in four broad categories: Hispanic, White, African American or Black, and other, which includes American Indian or Alaskan Native, Asian, and Native Hawaiian or other Pacific Islander racial groups. VA officials told us these data can be accessed on VA’s intranet and are updated on a monthly basis. The Survey of Veteran Enrollees’ Health and Use of Health Care comprises questions about a range of issues, such as enrollee’s health status, insurance, VA and community health care use, and attitudes and perceptions of VA services. The survey is generally conducted on an annual basis, and achieved a 32 percent response rate in 2018. VA publically reports these survey results by race and ethnicity at the national level, and also provides survey results by race and ethnicity for each VISN. For example, one indicator in the 2018 survey results showed that Native Hawaiians were far less satisfied with their ability to get referrals compared to other minority groups. According to VA survey documentation, VA requires a minimum number of survey responses to draw conclusions across the VA enrollee population; the number of survey responses must be aggregated at the VISN level to meet this minimum number. Veterans Signals is a VA survey intended to collect immediate targeted feedback on veterans’ experiences with outpatient services on an ongoing basis. VA officials told us that about one to two million survey invitations are sent out via email each week to veterans who recently received outpatient services, and have provided their email addresses to VA. These short surveys include eight to nine questions and focus on a particular area related to veterans’ recent experiences with VA health care services, such as scheduling appointments, pharmacy wait times, and proficiency of provider communication about veteran concerns during appointments. VA officials told us that the surveys have a response rate of about 20 percent, and of the responses received, 44 percent of respondents provide their race and ethnicity information. VA officials told us that VA medical center staff have access to survey results in real-time and can review results by race and ethnicity for their individual medical centers. In addition to these surveys, VA collects patient experience feedback from veterans, including minority veterans, through its patient advocates located at its medical centers. Each of VA’s 172 VA medical centers is responsible for making at least one patient advocate available to respond to veterans’ feedback and for ensuring feedback is recorded in its Patient Advocate Tracking System (PATS)—an electronic system used to describe and track the resolution of veterans’ feedback across VA medical centers. Patient advocates enter veterans’ feedback in PATS and assign one or more issue codes that generally describe the nature of the feedback. Of the 21 patient advocates we interviewed across 12 VA medical centers, most said they generally do not include race and ethnicity information in PATS when filing a veteran’s complaint, but a few patient advocates said they will include such information if it pertains directly to the complaint. For example, a patient advocate told us she may include race and ethnicity information in PATS in the case of a concern that discrimination occurred. We found that some veteran complaints may not be consistently coded and reported under the correct PATS issue codes in a manner similar to inconsistencies we have identified in prior work. According to VA officials from the Office of Patient Advocacy, two PATS codes were created in 2017 that, in particular, may specifically apply to issues affecting minority veterans: (1) discrimination concerns, and (2) diversity concerns. Of the 21 patient advocates we interviewed about these two specific issue codes, nine were not familiar with or had never used them. VA officials told us that they expected to see patient advocates use these codes more often in 2019 as a result of updates to their patient advocate training curriculum, required for newly hired patient advocates and available to all others. Patient advocates that we interviewed often told us that they review PATS data to report systemic issues to their VA medical center leadership, and a few said they report on a weekly or monthly basis. Additionally, VISN patient advocate coordinators use the PATS data to determine whether there are any trends in PATS data across the medical centers in their networks. As one of the nation’s largest health care systems, VA has a unique opportunity to gain a better understanding of the reasons that disparities in health care outcomes occur. VA signaled its commitment to reducing disparities for racial and ethnic minorities and achieving health equity by establishing a responsible program office, creating an action plan, and funding research toward this goal. As the number of minority veterans receiving VA health care services continues to increase, it is important that VA enhances and strengthens its efforts to identify and address disparities in health care outcomes to ensure that all veterans receive equitable care. Despite these efforts, however, without including performance measures or lines of accountability, VA lacks the means to ensure any action plan will be fully implemented to achieve its goals. Further, weaknesses in race and ethnicity data due to problems with the completeness and accuracy continue to limit VA’s ability to identify and address disparities in health care outcomes at the VA medical center level. Although VA is developing equity dashboards to provide health care outcomes data by race and ethnicity at the VA medical center level, these efforts will not improve the completeness and accuracy of the race and ethnicity data in VA’s EHR. Until VA resolves known weaknesses with the completeness and accuracy of its race and ethnicity data, it will be limited in its ability to assess health equity for veterans receiving care at its facilities. We make the following two recommendations to VA: The Under Secretary for Health should ensure that any action plan for achieving health equity includes key elements for successful implementation by consistently applying criteria identified in GAO’s past work on effectively managing performance, including developing performance measures to assess progress and creating clear lines of accountability by designating specific offices or officials with responsibility for coordinating efforts to implement actions and committing resources necessary for achieving its goals and objectives. (Recommendation 1) To ensure the availability of information on health care outcomes by race and ethnicity throughout the VA health care system, the Secretary of Veterans Affairs should conduct an assessment to determine the completeness and accuracy of race and ethnicity data captured in VA’s electronic health record, and implement corrective actions as necessary to resolve any identified deficiencies. (Recommendation 2) We provided a draft of this report to the Department of Veterans Affairs and the Department of Defense for review and comment. The Department of Defense did not have comments. VA provided written comments, which are reprinted in appendix I. VA concurred with both of our recommendations—that any health equity action plan should include performance measures to assess progress and that VA should resolve weaknesses identified with the completeness and accuracy of race and ethnicity data. VA further provided information on how the agency intends to address our recommendations, with targeted completion dates of December 2020 and June 2021, respectively. We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs and the Under Secretary for Health, and the Secretary of the Department of Defense. In addition, the report is also available at no charge on GAO’s website at http://www.gao.gov. If you or your staff has any questions regarding this report, please contact me at (202) 512-7114 or draperd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Ann Tynan, Assistant Director; Michelle Paluga, Analyst-in-Charge; Jennie Apter, Romonda McKinney Bumpus, and Phil Steinberg made key contributions to this report. Also contributing were Kaitlin Farquharson and Ethiene Salgado-Rodriguez.", "summary": "According to VA, in 2016, racial and ethnic minority veterans represented about 22 percent of the total veteran population of 18.6 million. VA projects racial and ethnic minority veterans will make up 36 percent of its total veteran population by 2040. VA has identified racial and ethnic disparities in its health care outcomes, mirroring trends seen across the United States. House Report 115-188 included a provision for GAO to review whether VHA provides quality, equitable care for minority veterans. GAO's report examines, among other issues, (1) the extent to which VA has taken steps to advance health equity for racial and ethnic minority veterans, and (2) VA's efforts to use race and ethnicity data to identify and address disparities in health care outcomes involving minority veterans. GAO reviewed relevant documents, such as strategic and operational plans and peer-reviewed research studies; assessed VA's health equity action plan against criteria identified in GAO's body of work on effectively managing performance; and interviewed VA officials familiar with VA's health equity efforts, as well as race and ethnicity data. The Department of Veterans Affairs (VA) has taken steps to reduce disparities in health care outcomes linked to race and ethnicity, but lacks mechanisms to measure progress and ensure accountability for results. In 2012, VA established the Office of Health Equity to identify and address health care outcome disparities and to develop an action plan to achieve health equity. This office issued an action plan in 2014 that identified activities to make improvements in five focus areas, such as increasing awareness of the significance of disparities and strengthening leadership for addressing them. However, GAO found that the extent of VA's progress in implementing the action plan and advancing health equity is unknown because the action plan lacked performance measures and clear lines of accountability for specific offices. For example, although VA's action plan included a list of “success criteria” for each of the five focus areas, these criteria were not measurable, and were not linked to specific activities or to offices responsible for implementation. VA funds research efforts that have identified disparities in health care outcomes involving minority veterans, but rely on data that VA officials and researchers noted have weaknesses in completeness and accuracy. One concern is that race and ethnicity information can be labeled incorrectly in VA patients' electronic health records as ”self-reported”, a highly reliable method of collection, when data were actually collected based on the less reliable method of VA staff observation. Other reported concerns include missing values on patients' race and conflicting race and ethnicity information. VA researchers told GAO they account for some of these concerns by using data from other sources, such as Medicare, but such work-arounds are time intensive. Further, VA officials reported that data weaknesses limit their ability to identify and address disparities in health care outcomes in their medical centers. Despite recognizing weaknesses related to the quality of race and ethnicity data, VA has not implemented corrective actions to address them. Without doing so, VA medical center officials cannot readily identify and address disparities in health care outcomes by race and ethnicity. Note: Concerns about the completeness and accuracy of race and ethnicity information were raised by officials from VA's Office of Health Equity, Veterans Experience Office, and Health Services Research & Development. GAO is making two recommendations to VA to (1) ensure that any health equity action plan includes performance measures to assess progress, and clear lines of accountability designating responsibility to specific offices, and (2) conduct an assessment to determine how to address weaknesses identified with the completeness and accuracy of race and ethnicity data in the electronic health record, and implement corrective actions as necessary. VA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "AOC is responsible for the maintenance, operation, and preservation of the buildings that comprise the U.S. Capitol complex, as shown in figure 1. AOC is organized into the following 10 jurisdictions, each of which is funded by a separate appropriation: (1) Capitol Building, (2) Capitol Grounds and Arboretum (hereafter the Capitol Grounds), (3) Capitol Police Buildings, Grounds, and Security (hereafter the Capitol Police), (4) Capitol Power Plant, (5) House Office Buildings (hereafter the House), (6) Library Buildings and Grounds (hereafter the Library), (7) Planning and Project Management (PPM), (8) Senate Office Buildings (hereafter the Senate), (9) Supreme Court Building and Grounds, and (10) U.S. Botanic Garden (hereafter the Botanic Garden). PPM provides consolidated services to all of AOC’s jurisdictions, such as long-range facility planning, historic preservation, and architectural and engineering design services. In addition, PPM manages systems that span jurisdictions including electrical distribution and emergency generators. PPM is also the parent organization of the Division, which provides construction and facility management support to all of AOC’s jurisdictions. Established in the 1970s, the Division’s mission is to “support AOC jurisdictions serving their Congressional and Supreme Court clients by providing high quality construction and craftsmanship with seamless flexibility, best value, and extraordinary customer service, while protecting our national treasures.” The Division’s operations are funded through a mix of appropriations and project funding from the jurisdictions. Specifically, according to AOC officials, the agency’s appropriation for Capital Construction and Operations provides the salaries and expenses of up to 13 permanent staff. The salaries and expenses of the remaining staff, as well as other costs (such as materials) are covered by the project funding the Division receives from the jurisdictions. According to AOC officials, essentially, the jurisdictions hire the Division to execute work on their behalf, and the Division charges the jurisdictions for its expenses. As a result, the number and type of temporary employees the Division employs at any given time is directly related to the projects it is performing for the jurisdictions. As of October 2018, of the Division’s 162 employees, 12 were permanent employees responsible for executive management and administrative functions. The remaining 150 were temporary employees—124 trade workers and 26 construction support employees—that it hired under temporary (e.g., 13- or 24-month) appointments. The trade workers include electricians, plumbers, masons, woodcrafters and carpenters, cement finishers, sheet metal mechanics, painters and plasterers, hazardous material abaters and insulators, laborers, and warehouse and material handlers. The construction support employees include personnel who perform activities such as construction management, purchasing, and timekeeping. The Division’s temporary employees are eligible for benefits. By law, AOC is generally required to provide all temporary employees with “the same eligibility for life insurance, health insurance and other benefits” to temporary employees who are hired for periods exceeding one year. The benefits AOC’s temporary employees receive may differ from what other federal temporary employees in the executive branch receive since these benefits vary depending on the type of temporary appointment and the employing agency, among other things. For example, employees serving under an appointment limited to 1 year or less are generally not eligible for the Federal Employees’ Group Life Insurance program. As previously stated, the Division pays for the salaries and expenses of its temporary employees with project funding from the jurisdictions. That project funding covers both the Division’s direct and indirect costs. Direct costs are those directly attributed to and expended on a project, such as labor (i.e., trade workers) and materials. In contrast, indirect costs are necessary costs that are not directly attributable to a specific project, such as employee leave and training, as well as salaries for construction support employees, such as supervisors and purchasing agents. To pay for its indirect costs, the Division charges the jurisdictions what it calls an “indirect rate.” As of October 2018, the Division’s indirect rate was 0.85. The Division applies this rate to every direct labor-hour associated with a project it executes for the jurisdictions. For example, for a trade worker with a hypothetical hourly cost of $45, the Division charges the jurisdictions about $83, as shown in figure 2. For more information on the Division’s direct and indirect costs, see appendix II. Based on our analysis of the Division’s data for projects completed during fiscal years 2014 through 2018, the jurisdictions used the Division to varying degrees for projects that ranged widely in terms of cost, complexity, and duration. Cost: There was a wide range in the nominal cost of individual projects the Division completed during fiscal years 2014 through 2018. The smallest individual project cost about $1,100 in 2017 dollars to perform hazardous materials testing in the Ford Office Building for the House jurisdiction in fiscal year 2016. Larger projects may be done in phases and when combined can cost millions of dollars. For example, in 2015 the Division completed a lighting project at the James Madison Building for the Library jurisdiction in two phases at a total cost of about $9.8 million in 2017 dollars. Complexity: During this period, the Division’s projects ranged from work involving one type of activity or trade to work involving several phases or many trades. For example, the Capitol Power Plant jurisdiction used the Division for paint projects and a door replacement. Other projects included the construction of a lactation suite at the Ford House Office Building. For this 4-month project, the Division performed carpentry, electrical work, hazardous materials abatement, and other tasks in order to demolish an existing women’s restroom and build a lactation suite with an adjacent, smaller women’s restroom (see fig. 3). Duration: During this period, the jurisdictions used the Division for projects that varied from quick turnaround projects that took a few days to complete to longer, multi-year projects. Most (about 88 percent) of the projects were completed within one year, while about 4 percent were completed between 1 and 2 years, and about 8 percent took 2 or more years to complete. For example, the Senate jurisdiction used the Division for an elevator repair project in 2016 at the Russell Office Building that took one day to complete while smoke detector upgrades in the James Madison Building for the Library jurisdiction took over 5 years and were completed in 2014. We also identified several examples of projects that the Division did for jurisdictions in phases. Sometimes the duration of the phases were less than one year but when combined the work spanned multiple years. For example, the Division built additional office spaces for staff displaced by the House jurisdiction’s renovation of the Cannon Office Building. Each phase of the work was completed within one year, but the work spanned almost 2 years from November 2014 to August 2016. The extent to which each of the jurisdictions used the Division also varied. Based on our analysis of the Division’s data and discussions with the jurisdictions, the Library, House, and Senate jurisdictions were the primary users of the Division during fiscal years 2014 through 2018, comprising more than 90 percent of the total work by cost for completed projects, as shown in figure 4. The Division completed projects exceeding $1 million for each of these jurisdictions. While jurisdictions have the option to use their own staff or a contractor for projects, jurisdiction officials said they consider a range of factors when determining whether to use the Division. They most frequently cited the Division’s flexibility in responding to scheduling and scoping changes and the jurisdictions’ own internal capacity to execute a project. They less frequently cited other factors, such as the availability of appropriations. Schedule: Jurisdiction officials said the Division provided scheduling flexibility at no extra charge compared to using outside contractors. According to jurisdiction officials, when projects require a great deal of flexibility, the jurisdictions may be more likely to choose the Division over a contractor because the Division can start and stop work as needed and can work nights or on weekends if necessary to keep a project on schedule without charging extra fees. For example, work on projects may need to be stopped or delayed for a variety of reasons, such as for security purposes if there is a protest near the worksite, or during a presidential visit. Jurisdiction officials also noted that the Division can typically mobilize faster than a contractor, a consideration that can be an important factor in determining whether to use the Division. For example, jurisdiction officials noted that the Division’s employees can begin work faster than an outside contractor because they have employee identification badges that authorize access to most buildings across the Capitol complex without an escort. 'Contractors must obtain a badge prior to accessing a work site and require escorts in instances when they do not have an AOC or site- specific badge, and the process of obtaining a badge adds time to when a contractor can begin work. As another example, jurisdiction officials also told us that using a contractor requires that AOC develop full design specifications for a project, a process that takes time and resources. In contrast, the Division can execute work without full design specifications. For example, Capitol Power Plant officials told us they used the Division for renovations to their Administration Building because, according to the officials, the Division started the work sooner, without design specifications and thereby completed the project faster than a contractor likely could have. Capitol Power Plant officials explained that the work— which included new carpet and painting—was agreed upon with the Division without spending time developing detailed design specifications that would have been required to obtain a contractor for the work. Scope Changes: According to jurisdiction officials, the Division is typically more flexible than a contractor when dealing with issues that arise from unforeseen site conditions or changes to a project’s scope. For example, during the construction of the lactation suite discussed above, the Division uncovered lead paint in the walls, requiring the work to stop until the lead paint was removed. According to officials, contractors typically charge for making changes to a project’s scope, such as removing hazardous materials uncovered during construction or associated delays. The Division does not charge for making changes or associated delays. This flexibility is because the Division charges based on direct labor hours spent on a project, meaning its expenses are charged as they are incurred. Accordingly, while a project’s costs may increase if more labor is charged to a project, the Division also has the option of having its employees work on other projects if work on a particular project has to stop. Jurisdiction officials told us that the Division also works with the jurisdictions to save money on projects. According to officials, such savings were the case during a 2-year project that the Division completed at the Library jurisdiction’s Jefferson Building in 2018 with a cost of $3.5 million in 2017 dollars. The project involved reversing the direction of doors in high-occupancy areas to allow for more orderly evacuation of occupants in the event of an emergency, as shown in figure 5. It also involved replacing some of the building’s historic doors and associated hardware with replicas that meet modern safety standards. Officials told us the Division helped the jurisdiction save about $1.2 million (in current dollars) during the course of the project by identifying less expensive materials for the project than originally planned for. Internal staff: Jurisdiction officials also told us that they use the Division for projects when they lack the internal capacity to do so. Most of the jurisdictions have some trade workers, such as electricians and plumbers, on staff to handle their daily operations and maintenance needs. Jurisdictions may execute smaller projects with their own employees but may use the Division for projects beyond routine maintenance work that their own employees cannot fit into their schedules. For example, officials with the Senate jurisdiction told us that they have staff capable of performing cabinetry work but have used the Division in the past for cabinetry work so that their staff could focus on more routine maintenance work. Senate jurisdiction officials also told us that they primarily use their own staff for construction work, but will use the Division as an option to supplement their staff when the volume of the Senate jurisdiction’s own workload is higher than what can be handled internally. Skill and equipment: Jurisdictions may use the Division if they lack the skills or equipment to execute a particular project. Officials from five of the jurisdictions told us that they have staff within their jurisdiction who can execute small projects involving hazardous materials, such as lead paint abatement under 2 square feet in size. Larger projects have additional abatement requirements, and the jurisdictions have used the Division for these projects. As another example, the Capitol Grounds jurisdiction used the Division in 2016 to install the annual Christmas tree on the Capitol lawn because the jurisdiction lacked the necessary equipment to do so. The Botanic Garden jurisdiction, which does not employ any masons, used the Division for a project at its Conservatory in 2016 because of the Division’s masonry expertise. Officials with the Senate jurisdiction also cited the Division’s masonry expertise among other factors, such as the Division’s familiarity with the jurisdiction’s buildings, in selecting the Division to repair the steps at the Russell Senate Office Building in 2017, as shown in figure 6. Availability of appropriations: Jurisdiction officials told us that they might not use the Division if the work cannot be completed by the time the jurisdiction’s appropriations expire. Specifically, because the jurisdictions pay for the Division’s services as work is executed rather than upfront when the work is initiated, the jurisdictions must ensure that work by the Division can be completed before their appropriations expire. Jurisdiction officials told us that as a result, the Division may not be a realistic option when using 1-year appropriations near the end of the fiscal year. In contrast, when using a contractor, jurisdictions may obligate fixed period appropriations prior to the end of the fiscal year for work that will continue into the following fiscal year. Cost: Most jurisdiction officials said that a project’s cost was not a key factor they considered when determining whether to use the Division for a project. When the jurisdictions are considering using a contractor they are not required to obtain cost estimates from the Division first and generally do not do so. As a result, comprehensive information on the relative costs of using the Division compared to a contractor was not available. However, in cases where the jurisdictions told us they did obtain estimates from both the Division and a contractor, they said the cost to use the Division was sometimes more expensive than a contractor and sometimes less expensive, as illustrated in the following examples. Officials with the Supreme Court Building and Grounds jurisdiction told us they used the Division to install a new heating, ventilation, and air-conditioning system in one of its buildings after it obtained an estimate from a contractor. According to officials, the project required specialized skills that the Division’s trade workers did not have. However, once they received the contractor’s estimate, the jurisdiction officials determined it was cheaper to pay for the Division’s employees to get trained to do the project than using a contractor. Officials with the Senate jurisdiction told us they obtained cost estimates for lead abatement work from both the Division and a contractor several years ago. According to officials, the contractor’s estimate was less than that of the Division because the contractor proposed using different equipment for the project than the Division, and the jurisdiction used the contractor for the abatement. The jurisdiction officials we interviewed said they were generally satisfied with the Division’s services, including the quality of its work, and were particularly satisfied with the flexibility the Division offers. Officials from seven of the nine jurisdictions we interviewed also told us they would not suggest making changes to how the Division currently operates. Officials from two of the jurisdictions suggested the organizational and cost-allocation changes discussed below. According to Division officials, implementing those suggestions would have implications for its operations and structure, and would require additional research and evaluation to determine if they are feasible. Transfer positions from the Division to its parent organization, PPM: Officials from one jurisdiction suggested that the Division could lower its indirect rate by transferring payroll responsibility for some supervisory positions, such as its construction or safety managers, from the Division to PPM. As discussed above, because the Division does not receive an appropriation for the salaries and expenses of its temporary construction support employees, it pays for those costs by charging the jurisdictions for direct labor hours and also an “indirect rate.” Division officials told us that payroll responsibility for some construction support positions could be transferred to PPM and that this transfer would reduce the Division’s indirect rate because that rate increases by about 1.1 percent for each employee captured in the rate. Because PPM is the parent organization of the Division, this step would not reduce the total costs of projects to AOC as an organization; rather, it would transfer the responsibility for paying certain costs from the jurisdictions to PPM. According to AOC officials, this could have several effects. First, PPM would need to find a way to fund those positions, which would likely require an increase in its appropriations to cover additional positions. Second, transferring supervisory positions to PPM could mean those personnel could be tasked to support other AOC-wide efforts, rather than supervising and managing the day-to-day execution of the Division’s projects. Similarly, Division officials told us that transferring supervisory positions or support personnel such as purchasing agents to PPM could reduce the Division’s flexibility, such as its ability to hire additional supervisors or support personnel if its workload increases in the future. Make the Division’s indirect rate variable: Officials with that same jurisdiction suggested that the Division consider making its indirect rate (which as of October 2018 was a fixed rate of 0.85) a variable rate. Under a variable rate approach, projects would have different rates depending on their needs. For example, a project requiring only labor would be charged one rate, but a project requiring labor and additional services, such as the purchasing of materials, would be charged a higher rate. According to Division officials, charging the jurisdictions varying rates depending on the extent to which a project utilizes the Division’s resources could reduce the cost for some jurisdictions but increase it for others since the Division must charge enough to recover all of its costs. Division officials told us AOC evaluated this option in 2017 but decided against it. AOC determined that making the Division’s indirect rate variable would result in increased administrative burden because the Division would have to determine which projects and workhours would be variable and which would not. It would then need to track and assess them differently for each project. Provide additional on-site supervisors for complex multi-trade projects: For most projects, the Division provides supervisors who manage the day-to-day execution of multiple projects. However, jurisdictions have the option to pay, as a direct cost, for dedicated, on- site supervisors to oversee and manage their projects exclusively. Officials with one jurisdiction suggested that the Division make it standard practice for complex, multi-trade projects to have a dedicated, on-site supervisor. Division officials told us that having a dedicated, on-site supervisor works best for complex, multi-trade projects such as the East Phase of the House jurisdiction’s 13-month, $15 million child care center project that the Division completed in December 2018 (see fig. 7). According to Division officials, having dedicated, on-site supervisors day and night during construction enabled the project to remain on schedule and below budget because the supervisors were responsible for overseeing all construction activities and could immediately address questions or concerns that arose, thereby resulting in increased efficiency and cost savings. Division officials told us that while the project’s scope increased during execution, the Division was able to work additional nights and weekends to meet the project’s deadline. Even with additional scope, Division officials estimated that they have saved the House jurisdiction about $500,000 (in current dollars) on the project through increased oversight and by identifying areas of cost savings, such as purchasing less expensive lighting fixtures than called for in the design. The variability of the Division’s workload makes anticipating the necessary size (number of employees) and composition (mix of trades and number of employees within each trade) of its workforce challenging. AOC has reported to Congress that the primary drivers behind the size and composition of the Division’s workforce have been project demand and the availability of funding. As previously discussed, the Division’s workload is driven by projects the jurisdictions hire it to perform. Without projects to execute for the jurisdictions, the Division does not have funding to pay the salaries and expenses of most of its employees. Accordingly, the size of the Division’s workforce expands and contracts in response to the jurisdictions’ demand for work. For example, over the last 5 fiscal years, the size of the Division’s trade workforce has fluctuated between a high of 191 in fiscal year 2016 and a low of 121 in fiscal year 2018. During that period, the number of employees the Division employed within each trade also fluctuated. Several factors contribute to the variability of the Division’s workload and make determining its future workforce needs challenging. First, officials told us that the Division has no control over whether the jurisdictions use the Division for their projects. Second, even if a jurisdiction decides to use the Division, Division officials told us that projects are notional or uncertain until that jurisdiction signs a project agreement, among other things. Third, even with a signed agreement, jurisdictions can reduce a project’s scope or cancel it all together, a situation that can leave the Division searching to find work for the trade workers it planned to use for the project. Finally, differing project priorities also come into play, as both Division officials and representatives from three of the jurisdictions acknowledged that some projects and work for certain jurisdictions are a higher priority than others. According to officials, when priority or emergency projects arise, the timing and work for ongoing projects can be affected as trade workers are shifted to the priority or emergency. In some cases, the on-going project may continue at a slower pace with fewer workers and in other cases all work might be stopped for a period of time. Over the last several years the Division has made efforts to strategically manage its workforce to help ensure that it has the right number and composition of employees to meet the jurisdictions’ needs. Our prior work has identified certain practices that, when implemented, can help federal agencies strategically manage their human capital. These practices include: (1) involving managers and stakeholders in decision-making, (2) basing workforce decisions on current needs and future projections, (3) having strategies to address workforce gaps, and (4) monitoring progress. As discussed below, we found that the Division has taken steps that generally align with those practices. However, it does not have a formalized process for collecting information that it uses to project future workforce needs, and we note that several of the steps it has taken date to the time of the March 2017 layoffs or more recently. Involve managers and stakeholders in decision-making: The Division has taken steps to involve AOC’s management, including the superintendents of the jurisdictions, in managing its workforce given the variability of its workload. According to Division officials, its staff are in frequent contact with the jurisdictions and meet periodically with the jurisdictions to discuss the status of ongoing and future projects. The officials said that Division staff meet bi-weekly with the larger jurisdictions—such as the Senate, House, and Library—and monthly or as needed with others as well as with PPM on a weekly basis to discuss the status of projects and workforce needs. According to Division officials, this regular communication with the jurisdictions is their primary and most important method of identifying and addressing workload issues or concerns. Jurisdiction officials echoed the Division’s comments, noting that they are in frequent contact with staff from the Division or as needed. Base workforce decisions on current needs and future projections: Over the last several years, the Division has taken steps to improve how it collects and tracks information from the jurisdictions upon which to base its future workforce projections. Prior to 2015, the Division used a paper- based process to collect information on the jurisdictions’ work requests and tracked information on a spreadsheet. In 2015, the Division implemented a software tool called the Construction Division Management System (CDMS) to streamline that process, making it easier for the jurisdictions to submit requests for work. For example, using CDMS, the Division can now electronically collect information for ongoing projects from the jurisdictions, such as change orders and schedule updates, and the jurisdictions can electronically submit requests for cost or schedule information for future projects. According to Division officials, Construction Managers, who are familiar with the resource needs of individual projects, are responsible for updating and validating the information in CDMS—typically bi-weekly—and the information in CDMS is available to the jurisdictions to review and verify. More recently, in July 2017, the Division hired a scheduler to develop resource-loaded schedules for ongoing projects. This involves assigning labor, materials, equipment, and other resources to a project’s schedule. According to Division officials, currently, the Division develops resource- loaded schedules for about 70 percent of its workload as the projects that comprise its remaining workload are too small or short-term for such schedules. In addition, in October 2017, the Division began collecting additional information on the jurisdictions’ construction priorities through a monthly data call. As part of this data call, which the Division performs via email, the Division requests updated information from the jurisdictions on their current projects, such as the expected start date or whether minor tasks remain, and the status of potential future projects. Using the information the Division collects from the jurisdictions, officials told us it then forecasts its workload and workforce needs out over the succeeding 12 months. According to officials, those projections are an “art, not a science,” because of the uncertainties surrounding the Division’s workload. However, the Division has not formalized the process it uses to collect information about the jurisdictions’ construction priorities. Specifically, we found that the Division lacks a written set of procedures for the monthly data call discussed above to help ensure that staff understand who is responsible for collecting information, what information should be collected, and when that information should be collected. This lack of procedures led to a situation in July 2018 where, according to officials, the Division did not conduct that data call but has since set calendar reminders for key staff in an effort to help ensure they do not miss it again. While setting such reminders may have some benefit now, it does not ensure that others within AOC will execute that data call in the future. Formalized processes, such as written procedures, can help ensure that steps an agency is taking can be implemented in a predictable, repeatable, and accountable way. Such procedures are also a key component of internal control designed to provide reasonable assurance that an organization’s operations are effective and efficient. AOC officials agreed that a more formalized process for collecting information about the jurisdictions’ construction priorities could help ensure the data is collected consistently. It would also better position AOC management to ensure that the Division’s process will be implemented consistently and that the jurisdictions understand what information is expected of them. It could also provide reasonable assurance to AOC management and Congress that the Division is taking the steps necessary to manage its workload and basing its workforce projections on the most current information available. Have strategies to address workforce gaps: The Division has a number of strategies it can employ if the size and composition of its workforce are not aligned with its workload requirements. For example, officials told us the Division can utilize direct-hire authority to quickly fill positions if there is a shortage of employees with specific skillsets to meet the jurisdictions’ needs. Officials told us employees may also work overtime to meet the jurisdictions’ needs if the Division’s workload projections do not show a need to hire additional employees. In instances where there is a lack of work, officials told us the Division has the options of not renewing the appointments of its temporary employees; helping affected employees in finding positions in jurisdictions to the extent practicable; or, if necessary, lay off affected employees, as it did in March 2017. Division officials told us they are also exploring additional strategies to help address potential instances where the size and composition of its workforce are not aligned with its workload requirements moving forward. One potential strategy involves using the Division to help address AOC’s backlog of deferred maintenance and capital renewal, which AOC estimated in 2017 was about $1.4 billion. Another potential strategy involves working with the jurisdictions to establish more large projects that provide a stable amount of work over a period of time. An example of a recent such project is the East Phase of the O’Neill Child Care Center project. According to Division officials, around 25–30 trade workers worked at the site at any given time, providing stability and work for multiple trades. When work on other projects was delayed or did not materialize, the Division was able to move the trade workers to the child care project. Monitor progress: Over the last several years the Division has taken steps to monitor the accuracy of its workload and workforce projections by discussing its projections with AOC management, including the Architect of the Capitol and the superintendents of the jurisdictions, each month. According to officials, the Division began these monthly briefings for AOC’s management in December 2016, when its workload decreased due to the completion of work related to the renovation of the Cannon Office Building. During these briefings, Division staff provide the Architect of the Capitol and the superintendents with information on the Division’s active, committed, and potential projects over the next several months. According to Division officials, these briefings provide an opportunity to discuss with AOC’s management any issues or concerns they have with the Division’s workload. The Division employed the practices described above in the months leading up to the March 2017 layoff of the 30 temporary employees. Division officials told us that 5 to 6 months prior to March 2017, they anticipated a potential decline in the Division’s workload and worked with the jurisdictions to identify potential projects that the Division could execute, but sufficient additional projects did not materialize. During this process, the Division involved PPM, the jurisdictions, and AOC’s management, among others. The efforts to minimize the number of employees affected by any layoffs included identifying job openings within the jurisdictions that employees could apply for. According to officials, one employee was hired by the Senate jurisdiction, another by the Capitol Grounds jurisdiction, and a third by the Office of the Chief Administrative Officer in the House, prior to the layoff. During the course of our review, we observed that the Division employed these strategies. Specifically, Division officials told us that they anticipated there might be a potential decline in the Division’s workload in early 2019. The Division raised this potential with AOC’s management during the summer of 2018, and officials told us the issue was resolved once the House and Library jurisdictions identified several projects that the Division could execute beginning in 2019. AOC’s authority to appoint and remove its employees is governed by title 2 of the U.S. Code and AOC has established various practices and policy related to their terms of employment. We found that AOC generally followed its practices when it appointed 30 temporary employees and adhered to its policy when it subsequently laid them off in March 2017. Our review of the appointment letters for 27 of the 30 temporary employees laid off in March 2017 found that the letters specified that the position was temporary and was for a term not-to-exceed 13-months. We also found that 10 of the 27 appointment letters included language stating that the position was dependent on the availability of work or funding. As part of our review, we met with five of the nine employees that AOC rehired following the March 2017 layoffs, all five employees told us that they were aware of the temporary nature of their positions and of the fact that they could be laid off at any time due to lack of work. Human capital officials told us that in April 2017, they developed a standard appointment letter to communicate the terms of employment for temporary employees more consistently. This letter includes language explaining that temporary appointments may be terminated at any time due to a lack of work, lack of funds, or failure to meet management’s expectations. For a copy of AOC’s standard appointment letter for temporary employees, see appendix III. AOC may renew the employment of temporary employees at the end of their 13-month appointment based on project needs and the availability of funding, according to human capital officials. We found that the 13 month appointments for 26 of the 30 temporary employees were routinely renewed prior to their March 2017 layoff. Of the 26 temporary employees, 12 had been employed from 13 months to 5 years, 9 had been employed from 6 to 10 years, and 5 had been employed for more than 10 years. The remaining four had been employed for less than 13 months. Human capital officials told us that there is no limitation on the number of times an employee’s appointment may be renewed. To ensure that employees serving under temporary appointments understand the terms of their employment, human capital officials told us that since March 2014 employees who have had their appointments renewed sign a standard Extension of Temporary Appointment form. This form states the position is temporary, may be shorter or longer than 13 months, and may end at any time. For a copy of this extension form, see appendix IV. AOC’s layoff policy allows the Director of PPM, as delegated by the Architect of the Capitol, to lay off the Division’s temporary employees for lack of work, lack of funds, or failure to meet management’s expectations. The policy does not specify which factors AOC should consider in selecting employees to be laid off, thereby allowing the agency discretion in this area. Our review of the layoff letters for the 30 temporary employees laid off in March 2017 confirmed that AOC communicated to the employees that the layoff was due to a lack of work. In this particular situation, the Division officials said they determined the number of temporary employees needed to carry out its projected workload and considered two factors equally: (1) the employees’ performance and skillset and (2) the employees’ ability to work independently and as part of a team. Human capital officials told us that AOC’s offices of Employee and Labor Relations and its General Counsel reviewed the Division’s request, and found no human-capital or legal concerns. The human capital officials drafted letters notifying the 30 employees of their layoff, effective upon receipt. Division supervisors provided the letters to employees at the start of their shifts on March 21, 2017. Figure 8 provides summary information by trade on the 30 temporary employees that AOC laid off in March 2017. At the time of the March 2017 layoff, AOC did not have a policy that required the Division to notify the Division’s temporary employees of an impending layoff. Human capital officials told us that they did not provide the Division’s temporary employees with advance notice of their layoff because of concerns that such advance notice could result in an unproductive and disruptive work environment. In terms of notifying relevant employee unions, human capital officials said they provided 12- hour advance notification of the layoff to one employee union, in accordance with that union’s collective-bargaining agreement. The five rehired employees we interviewed told us they were caught off guard by the March 2017 layoffs. None of the 30 temporary employees filed grievances related to the layoff, according to human capital officials. Since the layoff, human capital officials told us they recognized that AOC did not have a consistent policy for providing advance notice of layoffs to temporary employees across AOC’s 10 jurisdictions. According to AOC’s Chief Human Capital Officer, some jurisdictions provided advance notice of layoffs to temporary employees while others did not. To provide consistency with such notification and in response to our inquiries, in October 2018 AOC issued guidance standardizing the notification period for temporary employees laid off due to lack of work or lack of funds across all jurisdictions. This guidance directs jurisdictions to provide all temporary employees with a notification period of 2-weeks prior to the effective date of being laid off for these reasons. It also provides jurisdictions the option to request administrative leave so that the temporary employee may stop work immediately and be paid during the two week notification period. The Division was created to serve as a flexible option that the jurisdictions can use to meet the facility needs of their congressional and Supreme Court clients. By design, the Division can hire employees if there is demand for its services and lay off employees, as it did in March 2017, if there is insufficient demand or project funding to pay them. In recent years, the Division has taken steps to more strategically manage its workforce and minimize disruptions to that workforce in part by increasing its communication with the jurisdictions. However, formalizing the process the Division uses to collect information on the jurisdictions’ construction priorities, such as by providing staff with a written set of procedures, which specifies what is required of staff and when, could help ensure that those staff consistently collect and use the best information to make decisions about the appropriate number of employees and the mix of trades. Formalizing that process in this manner could also help the Division provide reasonable assurance to AOC management and Congress that it is taking the steps necessary to manage its workload and basing its workforce projections on the most current information available. The Architect of the Capitol should formalize the process the Construction Division uses to collect information on the jurisdictions’ construction priorities each month, such as through developing written procedures. (Recommendation 1) We provided AOC with a draft of this report for review and comment. AOC responded with a letter in which it concurred with our recommendation and said it intended to address our recommendation later this year. AOC’s letter is reprinted in appendix V. AOC also provided technical comments, which we incorporated in the report as appropriate. We are sending copies of this report to the appropriate congressional committees and the Architect of the Capitol. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact us at (202) 512-2834 or rectanusl@gao.gov or (202) 512-7215 or gurkinc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Our objectives were to: (1) describe how the Architect of the Capitol’s (AOC) jurisdictions use the Construction Division (hereafter the Division) and the factors they reported considering when deciding whether to use the Division, (2) assess how the Division manages its workforce given the variability of its workload, and (3) assess whether AOC’s appointment and subsequent March 2017 layoff of temporary employees from the Division complied with applicable policy. To describe how the jurisdictions use the Division and the factors affecting this use, we obtained and analyzed data on projects the Division completed for the jurisdictions during fiscal years 2014 through 2018. We focused our discussion of these data to the cost, scope, and duration of projects and do not present information on the number of completed projects because of differences in how the jurisdictions identify projects. To assess the reliability of the Division’s data, we reviewed available documentation and interviewed agency officials. We determined that the Division’s project data were sufficiently reliable for the purposes of this report, which includes describing the type and cost of projects the Division completed for the jurisdictions over the last 5 fiscal years and identifying illustrative examples of those projects. For appropriate comparison, the costs of completed projects we present in our report have been adjusted for inflation and converted to 2017 dollars using the fiscal-year gross domestic product index, which is compiled by the U.S. Department of Commerce, Bureau of Economic Analysis. We attempted to obtain comparable data for projects where the jurisdictions used their own employees or a contractor, but these data were not readily available. With respect to the jurisdictions’ use of their own employees, the Capitol Building jurisdiction attempted to obtain this data for us, but the data that were available did not include the cost of all labor spent on projects. Further, according to AOC, the jurisdictions do not capture data on employees’ time spent on construction work so this data also included projects that were considered routine maintenance. With respect to the jurisdictions’ use of contractors, the data that were available also included purchase card transactions, among other unrelated costs. According to AOC, identifying just the contract costs of the jurisdictions’ construction projects would require that AOC conduct significant research and review every transaction associated with its contracts. To provide illustrative examples, we visited the sites of six projects that the Division was executing at the time of our review. To select these projects, we asked the agency to provide us with projects that would enable us to understand the nature of the work the Division performs for the jurisdictions. In addition to the 4 projects the agency provided, we selected 2 additional sites based on projects that were discussed during our interviews. During our visits, we met with Division officials and representatives from the jurisdictions to discuss the projects in detail. We visited the following projects: an abatement and insulation project at the Russell Senate Office repairs to the drainage system at the Russell Senate Office Building, the replacement of doors at the Library of Congress, demolition and construction activities associated with the construction of a new lactation suite at the Ford House Office Building, demolition and construction activities associated with the construction of a new child care center at the O’Neill House Office Building, and the replacement of light poles across the U.S. Capitol complex. We also interviewed officials from the Division and AOC’s 10 jurisdictions, including their respective superintendents. Except Planning and Project Management, we asked the jurisdictions if they had any suggestions for changing the Division’s operations. We did not ask Planning and Project Management because the Division is a component of that jurisdiction. We then discussed with Division officials the potential implications of making those changes. We did not independently evaluate the implications of implementing the superintendents’ suggestions as part of this review. To assess how the Division manages its workforce, we reviewed pertinent documents, such as AOC’s August 2017 report to Congress on the Division, the Division’s Organization and Operating Plan, user guides for the Construction Division Management System, and prior GAO reports. We also obtained and analyzed payroll data for the Division for fiscal years 2014 to 2018 and interviewed Division officials. To assess the reliability of the Division’s data, we interviewed agency officials. We determined that the Division’s payroll data were sufficiently reliable for the purposes of this report, which includes describing the size and composition of the Division’s workforce over the last 5 fiscal years. We compared the Division’s efforts to manage its workforce to strategic human capital-management activities or practices identified in our prior work and standards for internal control in the federal government. To assess whether AOC’s layoff of temporary employees from the Division in March 2017 complied with applicable policy, we reviewed relevant federal laws and agency policy, such as the Separation of Non- Permanent Employees Policy Memorandum (AOC Order 316-1). We also reviewed pertinent personnel documents, such as appointment letters, layoff letters, and Standard Form 50 personnel documentation. We compared AOC’s policy with AOC’s implementation during the March 2017 layoff of 30 temporary employees. We did not independently verify AOC’s application of the criteria used to determine which employees to lay off in March 2017. In addition, we interviewed officials from both AOC’s Human Capital Management Division and the Division. As part of our work, we requested interviews with the nine temporary employees that AOC subsequently rehired and interviewed the five who responded in order to obtain their perspective on AOC’s processes for laying off temporary employees. This information is not generalizable to all rehired temporary employees. We conducted this performance audit from March 2018 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Construction Division’s (hereafter the Division) costs include both direct and indirect costs. Direct costs are costs directly attributed to and expended on a project, such as labor (i.e., trade workers) and materials. Indirect costs are costs that cannot be directly attributed to a single project, such as costs associated with employee leave and training. Table 1 shows the components of the Division’s direct and indirect costs. To pay for its indirect costs, the Division charges the jurisdictions what it calls an “indirect rate” as part of the work it performs. As of October 2018, the Division’s indirect rate was 0.85. The Division applies this rate to every direct labor-hour associated with a project it performs for the jurisdictions. For example, a trade worker that the Division employs who has a hypothetical hourly cost of $45 also has an indirect cost of about $38. Accordingly, that trade worker’s total hourly cost, which the Division charges the jurisdictions, is about $83. The Division developed the methodology for its indirect rate in 2012, in consultation with the Architect of the Capitol’s (AOC) Chief Financial Officer and the jurisdictions, after it determined that its funding model at that time did not adequately recover costs that were not directly attributable to projects. According to the Division, the primary driver for developing this indirect rate was employee leave. Specifically, the Division’s employees earn about 11 hours of leave per pay period, and funds to cover that leave need to be recovered because they cannot be obligated and charged to a project at the time the leave is earned but prior to its being taken by the employee. The Division allocates its indirect costs among the jurisdictions, using statutory authorities available to the Architect of the Capitol. According to AOC officials, historically, the Division’s indirect rate was determined by staff within the Division. The rate was determined by looking at historical cost and project data over the two prior fiscal years. As of fiscal year 2019, AOC established a steering committee to determine the Division’s indirect rate. This committee is comprised of five members: AOC’s Chief Financial Officer, the Director of the Division, the superintendent of the House Office Buildings jurisdiction, and a superintendent from another large jurisdiction and a small jurisdiction. According to AOC officials, the Division’s indirect rate is now based on projected costs and projects for the current fiscal year, and this rate will be monitored and may be adjusted throughout the year to address potential gaps or overages in funding for the Division’s annual indirect costs. In addition to the contact above, key contributors to this report included Mary Crenshaw (Assistant Director); Maria Edelstein (Assistant Director); Melissa Bodeau; Sarah Cornetto; Patrick Dibattista; Camille M. Henley; Wesley A. Johnson; Efrain Magallan; Josh Ormond; Cheryl Peterson; Kelly Rubin; and Laurel Voloder.", "summary": "AOC is organized into 10 jurisdictions that operate and maintain the buildings and grounds of the U.S. Capitol complex. For projects such as renovations and repairs, the jurisdictions can use their own employees, a contractor, or AOC's Construction Division, which is staffed with trade workers such as electricians and plumbers. Most of the Division's staff are employed on a temporary basis and paid with funds the Division receives from the jurisdictions for projects it executes on their behalf. In March 2017, AOC laid off 30 of the Division's approximately 190 temporary employees, citing a lack of work from the jurisdictions. GAO was asked to review the Division's operations. This report examines the jurisdictions' use of the Division and the Division's management of its workforce, among other issues. GAO analyzed information on projects the Division completed during fiscal years 2014 through 2018, reviewed AOC policies, visited the sites of six projects that are illustrative of the work the Division performs for the jurisdictions, and interviewed AOC staff, including officials from AOC's 10 jurisdictions and five of the employees AOC laid off in 2017. The Architect of the Capitol's (AOC) Construction Division (hereafter the Division) is designed to serve as a flexible option that the 10 operational jurisdictions that comprise AOC can use to meet their facility needs. In their efforts to manage the buildings and grounds of the U.S. Capitol complex, AOC's jurisdictions have used the Division for projects that vary widely in cost, complexity, and duration (see figure). For example, over the last 5 fiscal years, the jurisdictions have used the Division for projects ranging in cost from about $1,000 to about $10 million and in scope from hazardous material testing to multiyear lighting-system upgrades. Jurisdiction officials cited the Division's flexibility in adjusting to scope and other changes to keep a project on schedule as one of the reasons they may decide to use the Division instead of an outside contractor. While jurisdiction officials said they were generally satisfied with the Division's services, officials from two jurisdictions suggested that the Division consider changing how it operates—for example, by transferring some positions to its parent organization in an effort to lower what it charges the jurisdictions. According to AOC officials, making changes such as this one to the Division's operations could have varying effects, such as increasing how much funding AOC would require from other sources beyond the jurisdictions. The Division has taken steps to strategically manage its workforce to help ensure that it has the right number and composition of staff to meet the jurisdictions' needs but has not formalized the process it uses for collecting information on the jurisdictions' construction priorities each month. Because the Division's workload is driven by projects the jurisdictions hire it to perform, such things as changes in projects' priorities and work to be performed make determining future workforce needs challenging. The Division's approach to managing its workforce generally aligns with practices that GAO has previously identified that help agencies strategically manage their human capital. This approach includes having strategies to address gaps if the size and composition of an agency's workforce are not aligned with its workload requirements. However, because the Division has not formalized the process it uses to collect information each month on the jurisdictions' construction priorities it may miss opportunities to obtain information that is critical to making informed decisions. The Division also cannot provide reasonable assurance to AOC management and Congress that it is taking the steps necessary to manage its workload and that it is basing its workforce projections on the most current information available. GAO recommends that AOC formalize the process the Division uses for collecting information on the jurisdictions' construction priorities each month, such as through developing written procedures. AOC concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "Private companies are generally prohibited by federal law from discriminating in employment on the basis of race, color, religion, sex, national origin, age, and disability status. Additionally, federal contractors and subcontractors are generally required to take affirmative action to ensure that all applicants and employees are treated without regard to race, sex, color, religion, national origin, sexual orientation, and gender identity, and to employ or advance in employment qualified individuals with disabilities and qualified covered veterans. EEOC enforces federal antidiscrimination laws, and OFCCP enforces affirmative action and nondiscrimination requirements for federal contractors. EEOC and OFCCP share some enforcement activities and have established a memorandum of understanding (MOU) to minimize any duplication of effort. The EEOC enforces Title VII of the Civil Rights Act of 1964, as amended, which prohibits employment discrimination on the basis of race, color, religion, sex, or national origin. EEOC also is responsible for enforcing other federal laws that prohibit discrimination in employment based on age and disability, among other characteristics. EEOC investigates charges of employment discrimination from the public, litigates major cases, and conducts outreach to prevent discrimination by educating employers and workers. EEOC also pursues a limited number of cases each year designed to combat systemic discrimination, defined by the agency as patterns or practices where the alleged discrimination presented by a complainant has a broad impact on an industry, profession, company, or geographic location. EEOC can also initiate a systemic investigation under Title VII with the approval of an EEOC commissioner provided the commissioner finds there is a reasonable basis for the investigation. In fiscal year 2018, EEOC resolved about 90,558 charges of discrimination, secured more than $505 million for victims of discrimination, and filed 199 lawsuits. The OFCCP within DOL is responsible for ensuring that about 200,000 federal contractor establishments comply with federal nondiscrimination and affirmative action requirements. Under Executive Order 11246 and other federal laws and regulations, covered federal contractors and subcontractors are prohibited from discriminating in employment on the basis of race, color, religion, sex, sexual orientation, gender identity, or national origin and are required to take affirmative action to help ensure that all applicants and employees are treated without regard to these factors. OFCCP also enforces Section 503, and the affirmative action provisions of VEVRAA, which require covered contractors to take affirmative action to employ and advance in employment qualified individuals with disabilities and covered veterans, respectively. OFCCP uses two approaches to ensure compliance with federal equal employment and affirmative action requirements—enforcement and compliance assistance. OFCCP’s enforcement program primarily involves conducting evaluations of contractors’ compliance with federal requirements and these evaluations represent the preponderance of agency activity. In 2015, OFCCP compliance officers conducted 2,345 compliance evaluations, which represented about 2 percent of federal contractor establishments in its jurisdiction. OFCCP has since significantly decreased the number of compliance evaluations it conducts. In fiscal year 2018, OFCCP completed 812 compliance evaluations, which is 65 percent fewer than in fiscal year 2015. Since fiscal year 2016, OFCCP has adopted a strategy of conducting fewer compliance evaluations and prioritizing larger systemic cases. Since OFCCP can only evaluate a small fraction of federal contractors each year, the agency also carries out compliance assistance efforts, including issuing guidance, conducting outreach concerning nondiscrimination requirements, and providing compliance assistance to contractors. OFCCP’s regulations generally require that covered contractors prepare and maintain an affirmative action program (AAP). Contractors must also comply with certain recordkeeping requirements; for example, under Executive Order 11246, covered contractors are required to maintain records pertaining to hiring, promotion, layoff or termination, rates of pay, and applications, among other records. Under OFCCP’s Executive Order 11246 regulations, an AAP is a management tool that is designed to ensure equal employment opportunity, with an underlying premise that the gender, racial, and ethnic makeup of a contractor’s workforce should be representative of the labor pools from which the contractor recruits and selects. An AAP must also include practical steps to address underrepresentation of women and minorities, such as goals for expanding employment opportunities to these groups in instances in which they are underrepresented. Companies must create an AAP for each business establishment—generally, a physical facility or unit that produces goods or services, such as a factory, office, or store for the federal contractor. Each year the federal government provides billions of dollars to organizations that provide social services to needy families and individuals. Some of these funds are provided through competitive grants to faith-based organizations (FBO), which may include religious groups, like churches, mosques, synagogues, and temples, or charitable organizations affiliated with religious groups. In some instances, FBOs believe it is necessary to hire only individuals who share their religious beliefs in order to carry out their mission. Title VII of the Civil Rights Act of 1964 generally prohibits employment discrimination based on religion. However, section 702(a) of the Act exempts FBOs with respect to basing employment decisions on religion, thereby permitting FBOs to intentionally, and exclusively, hire individuals who share their religious beliefs. In light of this exemption, FBOs that receive federal grant funding or that contract with the federal government have also generally been permitted to make employment decisions based on religion. OFCCP is responsible for ensuring that federal contractors comply with federal nondiscrimination requirements and provides compliance assistance to the entities it oversees, including guidance related to this exemption. There are, however, certain federal grant programs that are subject to statutory restrictions that prohibit recipients from using grant funding, in whole or in part, to discriminate or deny employment on the basis of religion, among other factors. In June 2007, the Department of Justice’s Office of Legal Counsel issued an opinion in a particular case stating that the Religious Freedom Restoration Act of 1993 (RFRA) could be reasonably construed to require an agency to exempt FBOs from statutory requirements that restrict federal grantees from hiring on the basis of religion. Pursuant to that opinion, and the RFRA, certain federal agencies have permitted FBOs that receive funding under a program that is subject to a statutory restriction on religious-based hiring to certify that they are exempt from such restrictions, allowing these FBOs to engage in religious-based hiring, provided that they do not discriminate on other bases. OFCCP and EEOC face challenges in conducting oversight efforts to ensure that employers meet applicable federal equal employment opportunity requirements. For example, in our September 2016 report, we found several shortcomings that limited OFCCP’s oversight efforts, including weaknesses in OFCCP’s compliance evaluation selection process, its reliance on voluntary compliance, and the lack of staff training. Also, in our November 2017 report, we found that OFCCP’s planned methodology for identifying equal employment disparities by industry, such as the technology sector, might not accurately identify industries at greatest risk of potential noncompliance with affirmative action and nondiscrimination requirements. Additionally, we reported that while EEOC had identified barriers to recruitment and hiring in the technology sector as a strategic priority, it had not consistently captured information identifying specific industries when conducting investigations. EEOC’s inability to capture this information using standard industry codes impeded its ability to conduct related analysis that could be used to more effectively focus its limited enforcement resources and outreach activities. In September 2016, we reported that about 22 percent of OFCCP’s compliance evaluations of supply and service contractors found violations of some type and about 2 percent had discrimination findings, since 2010 (see figure 1). When OFCCP found violations during compliance evaluations, it often resolved those violations with conciliation agreements that outlined remedial action that contractors agreed to take. As a result of our work, we made six recommendations (see table 1). The agency has taken action to fully implement three of our recommendations: (1) to address the risk geographic imbalances in compliance evaluation assignments; (2) to review outreach and compliance assistance efforts and identify options for improving information provided to federal contractors; and (3) assess existing contractor guidance for clarity. However, the agency has not taken action to fully implement our other three recommendations that focus on improving enforcement and compliance. With regard to the recommendations that have not been fully implemented, OFCCP has taken action to date as described below. Focus compliance evaluations on greatest violation risk. We found the process OFCCP used to select contractors for compliance evaluations could not ensure that contractors with the highest risk of noncompliance were being selected. OFCCP’s selection process was nonrandom and did not produce a generalizable sample of contractors for evaluation. As a result, OFCCP was unable to draw conclusions about noncompliance risk in the overall federal contractor population. While the selection process included consideration of a number of neutrally applied factors, such as alphabetical order, employee count at the establishment, contract value, or contract expiration date, OFCCP was not able to identify which of these factors, or any factors, are associated with risk of noncompliance. Thus, OFCCP was unable to quantify the extent to which federal contractors in its jurisdiction are noncompliant, and did not have reasonable assurance that it was focusing its efforts on those contractors at greatest risk of not following equal employment opportunity or affirmative action requirements. Because OFCCP only conducts evaluations about 2 percent of federal contractor establishments in its jurisdiction, without an effective risk-based contractor selection process, OFCCP may be missing opportunities to evaluate whether there is a significant segment of contractors who may be more likely to violate nondiscrimination and affirmative action requirements, leaving workers potentially vulnerable. OFCCP has taken steps to improve its contractor selection process, but has not fully implemented either this 2016 recommendation or a related recommendation we made in 2017 that it assess the quality of its proposed methods to incorporate consideration of disparities by industry before selecting contractors for compliance evaluation. Beginning in fiscal year 2020, contractors will be able to apply to the Voluntary Enterprise- wide Review Program (VERP), which aims to remove top-performing contractor participants from the pool of contractors scheduled for compliance evaluations. OFCCP also recently implemented a new scheduling list (the list of contractor establishments selected for evaluation) methodology based on research on closed cases from the previous five years (2014-2018). Thirty-three percent of the new scheduling list was comprised entirely of contractor establishments from the three industries with the highest rates of violation based on this sample of closed cases. However, the scheduling lists of the previous 5 years included nonrandom selections of contractor establishments that included a number of neutrally applied factors. If OFCCP’s goal is to prioritize contractors at highest risk of noncompliance, this new scheduling methodology may not achieve this, because contractors selected will be weighted towards prior neutrally applied selection factors, such as employee count, in addition to violation risk. Further, while VERP may remove some compliant contractors from the scheduling list pool, without overwhelming volunteer participation, it will do little to help identify those most likely to violate. Consequently, it remains unclear whether contractors with the highest risk of not following equal employment opportunity and affirmative action requirements will be selected for compliance reviews. Monitor affirmative action programs. In 2016 OFCCP relied significantly on voluntary compliance by federal contractors, and this approach could not ensure that contractors were complying with basic requirements like developing and maintaining an AAP. By signing a qualifying federal contract, covered contractors are required to develop an AAP within 120 days of contract commencement and update it annually. However, OFCCP had no process for ensuring that the tens of thousands of establishments that had signed a qualifying federal contract do so. OFCCP has taken steps towards implementing a mechanism to monitor AAPs but has not fully implemented this recommendation. In 2018 OFCCP contracted with an information technology vendor to develop a web-based portal to allow contractors to upload their AAPs electronically for convenience, increased compliance, and for OFCCP review and resource prioritization. Officials anticipate delivery of the portal by the close of fiscal year 2019. Simultaneously, according to officials, OFCCP has developed the necessary information collection request to obtain approval from OMB to collect all contractors’ AAPs annually. The agency anticipates that OMB approval will be timely to align with completion of the AAP portal. Facilitate timely compliance officer training. In 2016, we found that OFCCP may not be providing timely training for new compliance officers. According to OFCCP officials, budget constraints had made it difficult to hold timely centralized training for new compliance officers. In half of the regions we visited, compliance officers or management officials we spoke with noted that this training was not provided in a timely manner after new officers are hired. For example, one compliance officer told us they worked for 8 months before receiving formal training. In one district office, compliance officers we spoke with explained that the lack of uniform, timely training made compliance officers feel unprepared when they began their job. Further, without providing timely training to new compliance officers, OFCCP cannot ensure consistency in its enforcement efforts across its offices. OFCCP has taken steps to improve its training program, but has not fully implemented this recommendation. In 2018, OFCCP retained an expert consultant to assess its national training program and standardize its training development and evaluation process. The assessment was completed in 2019 and a plan of action was created to address any program gaps, according to agency officials. Officials reported that the plan of action was fully implemented in fiscal year 2019 and OFCCP obtained a 5 year International Association for Continuing Education and Training (IACET) accreditation for its program. OFCCP officials told us they are developing a learning management system that will allow new compliance officers easy access to training soon after the hiring. OFCCP plans for the system to include the development of course requirements by level of competence— basic, intermediate, and advanced. OFCCP officials told us they plan to roll out the new system in January 2020. In November 2017, we reported that the estimated percentage of minority technology workers had increased from 2005 to 2015, however, while we found statistically significant increases in the numbers of Asian and Hispanic workers, no growth had occurred for either female or Black workers (see figure 2). Further, female, Black, and Hispanic workers remain a smaller proportion of the technology workforce—mathematics, computing, and engineering occupations—compared to their representation in the general workforce. These groups have also been less represented among technology workers inside the technology sector—those companies that have the highest concentration of technology workers in such industries as computer systems design and software publishing—than outside the technology sector such as retail or finance companies. In contrast, Asian workers were more represented in these occupations than in the general workforce. As a result of our work, we made one recommendation to EEOC and five recommendations to OFCCP (see table 2). EEOC has taken action, but not fully implemented our recommendation on identifying missing standard industry classification data from its handling of charges. By providing guidance to contractors regarding the option to include more specific goals in their AAPs, OFCCP has taken actions to implement one of our six recommendations—to take steps toward requiring contractors to disaggregate demographic data for the purpose of setting placement goals in the AAP. The agency has not taken action to fully implement our other four recommendations that focus on improving oversight, as shown in table 2 and discussed below. With regard to the recommendations that have not been fully implemented, EEOC and OFCCP have taken action to date as described below. Capture standard industry classifications on charges. In our November 2017 report, we found that EEOC could not analyze charge data by industry to help identify investigation and outreach priorities. This was inconsistent with EEOC strategic planning documents and EEOC Inspector General reports which, had emphasized the importance of analyzing charge data by industry. EEOC’s inability to analyze charge data by industry limits EEOC’s ability to identify trends by industry sector and conduct sector-related analyses that could be used to more effectively to focus its limited enforcement resources and outreach activities. EEOC has taken some action towards addressing missing industry code data, but has not taken actions sufficient to fully implement this recommendation. As part of an effort to overhaul its data system, EEOC has begun developing an Employer Master List that will provide a source of employer information, including industry codes, but EEOC told us that it has not yet completed this effort. It anticipates this system will be more fully developed by spring 2020. Use data on closed evaluations to address delays. In our November 2017 report, we found that OFCCP did not analyze data on closed evaluations to understand the root causes of delays in its compliance review process that may be straining its resources and inhibiting OFCCP’s efforts to identify potential discrimination. This evaluation could help OFCCP determine whether changes are needed in its own internal policies and processes, as well as guide OFCCP’s selection of improved methods for obtaining complete, accurate, and timely documentation from federal contractors. OFCCP has taken actions but it does not fully address this recommendation. In June 2019, OFCCP officials reported that OFCCP’s procedures outlined in the Active Case Enforcement Directive (DIR 2011-01) caused delays in case closures, but it does not indicate that this conclusion resulted from the recommended analysis of internal process data from closed evaluations. OFCCP officials reported that the agency’s aged case rate—defined as a case which is open for more than 730 days and has not been referred for further enforcement— has dropped from 27.7 percent in fiscal year 2017 to 20.9 percent in fiscal year 2019, though they did not report any corresponding change in case outcomes. In September 2019, OFCCP officials told us they continue to study causes and how to address delays with effective policies that make the agency more efficient. Assess the methods used to consider industry disparities in compliance. In our November 2017 report, we found that OFCCP’s current methodology for identifying disparities by industry—using data from the American Community Survey—may not have accurately identified industries at greatest risk of potential noncompliance with nondiscrimination and affirmative action requirements. In its agency response to our November 2017 report, OFCCP officials reported that the agency was exploring the use of U.S. Census Bureau and administrative data to refine its selection process to focus on industries with a greater likelihood of noncompliance. OFCCP has taken some action, but has not fully implemented this recommendation. In January 2019, DOL officials reported that DOL had revised its scheduling methodology to include industries with the highest rates of violations. OFCCP published the scheduling list in March 2019 and its field offices started scheduling cases in May 2019. OFCCP stated it will continue to monitor results from this revised scheduling methodology to determine its effectiveness. It will be important for OFCCP to refine these methods based on its experiences with them. This new process is a step toward focusing efforts on industries at greater risk of potential noncompliance with nondiscrimination or affirmative action requirements. Evaluate establishment-based approach to compliance evaluations. In our November 2017 report, we found that OFCCP had made no changes to its establishment-based approach since OFCCP was founded in 1965. However, OFCCP officials acknowledged the changing nature of a company’s work can involve multiple locations and corresponding changes in the scope of hiring and recruitment. OFCCP has taken some action, but has not fully addressed this recommendation. In fiscal year 2019, OFCCP evaluated its current approach for identifying subcontractors for review. OFCCP stated that the current approach does not reliably include subcontractors in the pool from which contractors are scheduled because there is no government or public database that captures the complete universe of subcontractors and other important data. In June 2019, OFCCP submitted revisions to its process to the Office of Management and Budget (OMB) for approval. Evaluate the Functional Affirmative Action Program. In November 2017, we found that OFCCP had not evaluated its Functional Affirmative Action Program (FAAP)—an alternative affirmative action program for a business function or unit that may exist at multiple establishments or multi-establishment contractors. OFCCP offered the FAAP so that companies could move away from establishment-based reviews, which may be more appropriate for some multi-establishment contractors. However, few contractors participate in this program and the agency has not conducted an evaluation of it. OFCCP has taken some action, but has not fully implemented this recommendation. OFCCP has taken steps to encourage contractors to use the FAAP program without fully evaluating it as an alternative to the establishment-based program. Evaluating the FAAP could help OFCCP improve its ability to achieve its objectives and may provide broader insight for OFCCP’s overall enforcement approach. In our October 2017 report, we found that from fiscal years 2007 through 2015, 9 of the 117 potential FBOs we identified across HHS, DOJ, and DOL, certified that they were exempt based on RFRA from nondiscrimination laws related to religious-based hiring (see fig. 3). As a result, the nine FBOs were allowed to consider a prospective employees’ religious faith when making employment decisions. All nine of the FBOs were awarded funding by DOJ primarily through the agency’s Justice Programs, and collectively received approximately $3.2 million, which is less than 1 percent of the $804 million in grants that DOJ awarded that were subject to statutory restrictions from fiscal years 2007 to 2015. HHS, DOJ, and DOL awarded funding to at least 2,586 grantees through 53 grant programs that were subject to statutory restrictions on religious- based hiring. The number of relevant grant programs could be higher because GAO could not identify all such programs due to data limitations. We interviewed six of the nine faith-based grantees that certified that they were exempt from religious-based hiring restrictions. Each of the six grantees emphasized the importance of hiring someone of the same religious faith to assist with grant activities. For example, the grantees said that hiring someone with the same religious faith was critical to their mission and organizational success, and if the RFRA exemption were not available, they may not have sought the grant. We also interviewed grantees from five of 35 potential FBOs that did not certify that they were exempt from statutory restrictions based on religious-based hiring to see if they were aware of the potential for an exemption. The five grantees said that they did not recall seeing information about the exemption option in the grant application or grant award documentation. They said that they also may not have been looking for the information because they were not considering religion in their hiring decisions. HHS, DOJ, and DOL used various methods for informing grant applicants and recipients of the statutory restrictions on religious-based hiring and their processes for obtaining an exemption from such restrictions. Specifically: DOJ had made this information available on agency web pages as well as in the documentation that is provided to grant recipients. DOL had a web page dedicated specifically to explaining statutory restrictions on religious-based hiring to faith-based grant applicants and recipients, which also covers the process for seeking exemptions from the restrictions. In addition to providing information in grant announcements, HHS provided all Substance Abuse and Mental Health Services grant applicants seeking funds for substance abuse prevention and treatment services with a form that cites laws and regulations governing religious organizations that receive grant funding, including the regulation that outlines the exemption process. As we reported in 2016, DOJ, DOL, and HHS all required grantees that seek to make employment decisions based on religion to self-certify that they met requirements to be eligible for an exemption from statutory restrictions on religious-based hiring, but varied in how they reviewed and approved requests for approval. All three agencies required that faith- based grantees complete a form or some written request to demonstrate their eligibility for the exemption, but DOL is the only agency that reviewed and approved the requests. For example, DOL required that faith-based grantees submit their requests for the exemption for review and approval by the Assistant Secretary responsible for issuing or administering the grant. Conversely, while DOJ and HHS required that faith-based grantees submit a form or written request, respectively, neither reviewed nor approved the requests. On August 15, 2019, OFCCP proposed regulations intended to clarify the scope and application of the religious exemption to help religious employers with federal contracts and subcontracts and federally assisted construction contracts and subcontracts better understand their obligations. OFCCP proposes to add definitions of the following terms: exercise of religion; particular religion; religion; religious corporation, association, educational institution, or society; and sincere. In addition, the proposed rule states that the religious exemption should be construed to provide the broadest protection of religious exercise permitted under the Constitution and related laws, consistent with the administration policy to protect religious freedom. The stated intent of the proposed rule is to make clear that religious employers who contract with the federal government can condition employment on acceptance of or adherence to religious tenets, provided that they do not discriminate on other bases. Chairwoman Bonamici, Senior Republican Comer, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff have any questions about this testimony, please contact Cindy Brown Barnes, Director, Education, Workforce and Income Security Team at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Blake Ainsworth, Amber Yancey- Carroll, Melinda Bowman, Sheranda Campbell, Sarah Cornetto, Mary Crenshaw, Helen Desaulniers, Holly Dye, Michael Erb, Monika Gomez, LaToya King, Joel Marus, Diana Maurer, Heidi Neilson, James Rebbe, Katrina Taylor, Rosemary Torres Lerma, Kathleen van Gelder, and Betty Ward Zukerman. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Several federal laws, executive orders, and regulations seek to promote equal employment opportunity by prohibiting employers from discriminating in employment on the basis of race and gender, among other things, and generally require companies contracting with the federal government to comply with affirmative action and other equal employment opportunity provisions. The EEOC and OFCCP are the primary federal agencies that enforce these requirements. Although federal law also generally prohibits employment discrimination based on religion, faith-based organizations may hire based on religion. Some federal grant programs contain statutory restrictions prohibiting this practice; however, since a 2007 DOJ legal opinion, federal agencies have allowed faith-based grantees to use RFRA as a basis for seeking an exemption to allow religious-based hiring. GAO has issued three reports since September 2016 that address equal employment opportunity ( GAO-16-750 , GAO-18-69 , and GAO-18-164 ). This testimony is based on these three reports and discusses 1) OFCCP and EEOC's progress in addressing prior GAO recommendations and 2) equal employment opportunity exemptions for faith-based organizations. To update the status of prior recommendations, GAO reviewed agency guidance and documentation and interviewed agency officials. The Department of Labor's Office of Federal Contract Compliance Programs (OFCCP) and the Equal Employment Opportunity Commission (EEOC) face challenges in overseeing compliance by employers and federal contractors with applicable federal equal employment opportunity requirements. In its 2016 report, GAO made six recommendations to OFCCP and in its 2017 report made five additional recommendations to OFCCP and one to EEOC to strengthen program oversight. OFCCP has implemented four recommendations, but seven require additional agency action to be fully implemented, as does the one to EEOC. For example: In 2016, GAO found that OFCCP's oversight was limited by reliance on contractors' voluntary compliance with affirmative action plan requirements. OFCCP has taken steps to develop a new web portal for collecting those plans annually, but has not yet obtained Office of Management and Budget approval for the collection or launched the portal. GAO also found OFCCP's oversight was limited by a lack of timely staff training. OFCCP has taken steps to implement a new training curriculum, but has not yet implemented its new learning management system that will help ensure timely and regular training. In 2017, GAO found that EEOC had not consistently captured information on industry codes, which limits EEOC's ability to identify trends by industry sector and conduct sector-related analyses. EEOC has not yet completed development of its Employer Master List that will include industry codes. GAO also found that OFCCP's methodology for identifying equal employment disparities by industry might not accurately identify industries at greatest risk of noncompliance with affirmative action and nondiscrimination requirements. OFCCP has taken steps to develop a new methodology, but needs to further refine it to ensure that it will identify industries at greatest risk. From fiscal years 2007 through 2015, few faith-based grantees sought an exemption from nondiscrimination laws related to religious-based hiring under the Religious Freedom Restoration Act of 1993. In October 2017, GAO found that the Departments of Justice (DOJ), Health and Human Services (HHS), and Labor (DOL) had awarded funding to at least 2,586 grantees through at least 53 grant programs that restricted grantees from making employment decisions based on religion. The number of relevant grant programs could be higher because GAO could not identify all such programs due to data limitations. Across the three agencies, GAO identified 117 grantees that were potentially faith-based organizations (FBO). Of the 117 potential FBOs, nine DOJ grantees were FBOs certified as being exempt from statutory restrictions on religious-based hiring. All three agencies required grantees seeking an exemption to self-certify that they were eligible for the exemption, but the agencies' processes for reviewing and approving exemption requests varied. In August 2019, OFCCP issued a proposed rule to clarify the scope and application of the religious exemption to help organizations with federal contracts and subcontracts and federally assisted construction contracts and subcontracts better understand their obligations.", "document_type": "gao"}
{"report": "The size and complexity of states’ Medicaid programs have implications for program administration and oversight, including provider screening and enrollment. States have flexibility, within broad federal guidelines, in how they design, administer, and oversee their Medicaid programs. For example, states have the option to pay for care through fee-for-service (FFS) payments to participating providers, contract with managed care organizations (MCO) to deliver services based on a fixed amount per beneficiary, or a combination of both. In fiscal year 2018, total Medicaid spending was $629 billion, about half of which was estimated to be spent for services delivered under managed care. CMS and states each have a role to play in protecting the integrity of the Medicaid program and preventing fraud, waste, and abuse. States administer their Medicaid programs, including implementing federal requirements for screening and enrolling Medicaid providers. CMS has a role overseeing states’ compliance with federal requirements. CMS’s oversight activities include measuring improper payments in the Medicaid program, and conducting focused program integrity and desk reviews. Other federal and state entities also have a role in oversight of the Medicaid program. For example, state auditors—state agencies that typically conduct the annual single state audit of federal programs—may also conduct program integrity reviews and identify Medicaid improper payments. We have previously testified that state auditors are uniquely qualified to partner with CMS in its oversight of Medicaid. In our testimony, we noted that CMS could help improve program integrity by providing state auditors with a substantive and ongoing role in auditing state Medicaid programs. To limit payments to ineligible providers—such as those convicted of program-related fraud and abuse, or with a suspended or revoked medical license for reasons of bearing on professional competence or performance—federal regulations require states to screen and enroll all providers, whether the provider furnished, ordered, or referred services to an eligible beneficiary or whether the service was paid for under FFS or Medicaid managed care contracts. Providers subject to these requirements include individual practitioners—such as physicians, nurse practitioners, and physical therapists—as well as any physicians and other professionals who may only order or refer beneficiaries to services, but do not render services; for example, providers who only prescribe medications or order imaging services, such as an x-ray. Providers also include provider organizations—such as hospitals, group practices, and skilled nursing facilities—and providers and suppliers of medical equipment or goods. All providers must be screened when they (1) initially apply for and submit an application, and (2) upon reenrollment in a state’s Medicaid program. Further, states must screen all providers at least once every 5 years to revalidate their enrollment. States may rely on the results of providers’ screenings performed by the Medicare program or another state’s Medicaid program. States may also choose to delegate screening activities to vendors that screen providers on the states’ behalf or MCOs. If a state chooses to delegate screening activities, it must ensure that the screenings are conducted in accordance with the Medicaid program requirements. States must also collect certain information from providers to enroll them into their Medicaid programs, such as their Social Security numbers, dates of birth, and National Provider Identifiers, if applicable. States must also collect disclosure information for owners, managing employees, and others with controlling interests in provider organizations meeting certain criteria. For example, states must collect disclosure information for those with direct or indirect ownership totaling 5 percent or more, or who are agents or managing employees of a provider organization. These owners and others with controlling interests who are subject to disclosure requirements must undergo certain required screening activities, such as federal database checks, and states must perform these screening activities to enroll the provider organization. Federal regulations require states to perform several screening activities prior to enrolling providers. The provider’s categorical risk level for fraud, waste, and abuse determine the required screening activities. The screening activities may include conducting checks in federal databases; verifying licensure; and performing site visits and fingerprint-based background checks. In addition to required activities, states may also choose to conduct other screening activities in order to identify providers ineligible for participating in Medicaid. See figure 1 for an overview of Medicaid provider screening activities and appendix II for a full list of provider screening requirements. Risk-based screening. States must screen providers according to the provider’s categorical risk level for fraud, waste, and abuse. The regulations establish screening requirements for three risk levels—limited, moderate, and high risk—and each risk level includes a range of provider types. (See table 1.) In addition, providers’ risk levels can change. For example, limited- or moderate-risk providers may be categorized as high risk if the state Medicaid agency imposes a payment suspension based on a credible allegation of fraud, waste, or abuse. Federal database checks. States must confirm the identity of prospective providers, providers seeking revalidation, and individuals subject to disclosure requirements to determine if they have been excluded from participating in Medicaid by checking four federal databases: 1. the Social Security Administrations’ Death Master File (DMF); 2. the National Plan and Provider Enumeration System (NPPES); 3. the List of Excluded Individuals/Entities (LEIE); and 4. the General Services Administration’s System for Award Management (SAM). In addition, states must conduct at least monthly checks in the LEIE and SAM. States may also check other federal and state databases. For example, states may check CMS’s database containing Medicare provider enrollment data—the Provider Enrollment, Chain and Ownership System (PECOS)—prior to conducting their own database checks to determine if a provider is enrolled in Medicare and was previously screened. For providers enrolled in Medicare, states may choose to rely on the results of the screening conducted for the Medicare program and enroll the provider without conducting any further screening activities. For providers not enrolled in Medicare, states must screen the provider prior to enrolling them. (See table 2.) Licensure verification. States must verify that providers have a current, valid medical license in the states in which they are licensed. Further, states must confirm that the providers’ license does not have any limitations, such as a suspension or probation. Site visits and fingerprint-based criminal background checks. States must conduct on-site visits for moderate- and high-risk providers to verify that the information submitted is accurate and to determine providers’ compliance with federal and state enrollment requirements. Further, states must collect fingerprints from high-risk providers, and these providers must consent to a criminal background check. CMS developed the PERM to estimate the national Medicaid improper payment rate, including improper payments due to states’ non- compliance with provider screening and enrollment requirements. CMS computes the national improper payment rate as the weighted average of states’ improper payment rate estimates from the PERM using three key components of the Medicaid program: FFS, managed care, and beneficiary eligibility determinations. Each component of the PERM is estimated differently, and only the FFS component is used to oversee states’ compliance with provider screening and enrollment requirements. When calculating the FFS component, CMS measures improper payments in a sample of FFS claims, which record services provided. Specifically, CMS reviews the sample of FFS claims and examines related state documents to identify any errors resulting from a failure to meet federal and state policies, including provider screening and enrollment requirements. For example, CMS verifies the provider was eligible to render and bill for the services by reviewing provider information, including the provider’s name and license, and whether the provider was screened in accordance with risk-based screening requirements. Any FFS claims paid for services furnished, ordered, referred, or prescribed by a provider who was not screened in compliance with requirements or not enrolled with the state is considered an improper payment. The managed care component of the PERM measures any improper payments in the capitated payments that state Medicaid agencies make to MCOs on behalf of enrollees. It does not examine whether providers in managed care were appropriately screened and enrolled within a state. The eligibility component focuses solely on measuring improper payments related to state determinations of whether Medicaid enrollees meet categorical and financial criteria for Medicaid benefits. CMS conducts the PERM across all states on a 17-state, 3-year rotation cycle and computes an annual rolling average of improper payment rates from the 3 years of data. At the conclusion of each PERM cycle, CMS develops reports for each state, which include any findings related to provider screening and enrollment. Following each PERM cycle, states must prepare a corrective action plan to address errors found. CMS also conducts other oversight activities to protect the integrity of the Medicaid program and assess states’ compliance with Medicaid provider screening and enrollment requirements. These activities include the following: Focused program integrity reviews. CMS conducts these reviews to examine specific areas of Medicaid, including provider screening and enrollment and managed care. These reviews may include a full or partial review of states’ compliance with provider screening and enrollment requirements. Desk reviews. CMS conducts these off-site reviews on specific aspects of states’ program integrity activities, such as a state’s progress toward implementing corrective action plans in response to PERM findings and payments made to providers terminated from Medicaid. Officials from all seven selected states told us they faced challenges building the capacity and establishing the administrative processes needed to implement the new and expanded provider screening and enrollment requirements under PPACA and the 21st Century Cures Act. These challenges included establishing procedures for risk-based screenings, using federal databases and collecting information from providers, and screening and enrolling an increased volume of providers. Due, in part, to these challenges, officials from five selected states told us they have not yet implemented some of the requirements. Officials from all seven selected states described challenges building their capacity to conduct risk-based provider screenings prior to enrollment into their Medicaid programs. To incorporate database checks, site visits, fingerprint-based background checks, and other risk-based screening activities into state screening procedures; state Medicaid officials said they needed financial resources, leadership support, and time. Specifically, officials from the selected states told us they used one of the following three approaches to build capacity to implement the screening and enrollment requirements. 1. Developing new information technology systems. Officials from two of the seven selected states told us they developed new state information technology systems that automated screening and enrollment activities. For example, officials from one state told us they spent $5.9 million from 2015 through 2018 to develop a new provider screening and enrollment system that included an online provider application portal and automated screening activities, such as conducting database checks and flagging high-risk providers for site visits and fingerprint-based background checks. According to state officials, this new system helped the state implement provider screening and enrollment requirements, yielding efficiencies by allowing staff to focus on analyzing provider screening results rather than clarifying data entry errors and manually checking each database. (See fig. 2.) 2. Contracting with vendors. Officials from four other selected states told us they initiated or modified existing contracts with vendors to screen new provider applications and conduct revalidations on their behalf. For example, officials from one state told us their contract with a vendor resulted in screening and enrolling about 10,000 providers in 2018, more than five times the number the state had processed in the previous year. Another state used a vendor to revalidate more than 9,000 providers in 2016; about 12 percent of the state’s enrolled provider population. 3. Modifying existing procedures. Officials from our seventh selected state told us that they modified their existing state information technology system and procedures to manually screen and enroll providers. However, according to state officials, this approach has put pressure on their resources. Officials said that they were working to automate some database checks as much as possible without requiring services from a contractor. Officials from six of the seven selected states told us they experienced challenges using federal databases, and all seven of the states described challenges collecting required information for screening and enrollment. To mitigate challenges using federal databases, the state Medicaid agencies took actions including accessing data from alternate sources, manually verifying information, and collecting information from providers. (See table 3.) Recent CMS actions could also improve states’ ability to search databases. In April 2019, CMS officials told us they have partnered with the Treasury Department, which is conducting a pilot that will offer states access to its Do Not Pay Business Center services. Do Not Pay is a resource developed by the Treasury Department to detect and prevent improper payments. This resource allows federal agencies to automate screenings by searching for excluded parties using common identification numbers, such as Social Security numbers. Do Not Pay also allows users to search DMF, LEIE, and SAM from a single portal. CMS referred seven states, including two of our selected states, to take part in Treasury’s pilot. Officials from all seven selected states told us that they faced challenges collecting required information from providers for screening and enrollment, such as Social Security numbers or fingerprints. These states took steps—such as educating providers and developing or updating forms, procedures, and statutory provisions—to address some of the challenges associated with collecting the information necessary to screen and enroll providers. For example, one state told us that some providers have been hesitant to disclose Social Security numbers and their date of birth on applications, as well as other information that states are required to collect for enrollment. In response, the state has offered provider education on the requirements to facilitate collecting this information. State Medicaid officials also noted that their agencies worked with CMS, state legislatures, and state law enforcement agencies to implement fingerprint-based background check requirements to, for example, collect fingerprints and check them against Federal Bureau of Investigation records. Officials from two selected states told us their agencies did not have the authority under state law to collect fingerprints from providers or submit them to the Federal Bureau of Investigation prior to PPACA, and officials from one of these states told us changes to state statute were needed before they could implement this requirement. Officials from five of the seven selected states described challenges having sufficient capacity to screen an increased volume of providers and enroll certain provider types. Officials from one state told us that the new requirement to screen managed care providers more than doubled the number of providers the state needed to screen and enroll. Further, officials from three states told us about challenges obtaining information needed to conduct screenings from prescribers and other professionals who only order and refer services. Previously, such providers were not required to enroll in Medicaid and some were not responsive to the state Medicaid agency’s requests for information. Officials from the five selected states that faced these challenges told us they had taken steps to address these challenges. Yet, four of the five selected states that faced these challenges continued to make payments to these types of providers even though they were not enrolled in their Medicaid programs, because they wanted to maintain beneficiary access to the services. Managed care providers. Officials from three of the selected states told us they faced challenges enrolling managed care providers. For example, officials from one state told us that they could not process the large number of applications they needed to screen before enrolling these providers, and attempted to delegate some required database checks for managed care providers to its MCOs. However, officials told us these MCOs do not have state-level access to all required databases; therefore, the managed care providers have not been screened as required and are not all enrolled with the state. The officials told us the state has about 80,000 managed care providers to screen and enroll as part of implementing the 21st Century Cures Act requirements. However, officials said they have chosen to wait until the state launches a new information technology system that automates screenings before screening and enrolling these providers. Prescribers and other professionals who may only order and refer services to beneficiaries. Officials from three selected states told us they had not enrolled all prescribers and other professionals who may only order and refer services, but do not render them. These states have taken steps to address this challenge. Officials from one state told us they took steps to screen and enroll medical residents— hospital providers who are not providing services to Medicaid beneficiaries, but may prescribe medication during a beneficiary’s hospital stay. These officials told us they did not screen and enroll medical residents prior to PPACA, because of differences in licensure. Officials from all three states said they continue to pay for prescriptions written by these providers who are not enrolled in their Medicaid programs. CMS offers optional consultations that are tailored to support states’ implementation of the Medicaid provider screening and enrollment requirements. However, these consultations are optional, regardless of whether states have implemented the federal requirements. CMS also conducts several oversight activities—the PERM, focused program integrity reviews, and other activities—to oversee states’ compliance with provider screening and enrollment requirements. Collectively, these activities do not ensure CMS has comprehensive and timely information on the extent of states’ compliance with the requirements. In 2016, CMS began offering optional consultations tailored to support states’ implementation of Medicaid screening and enrollment requirements. Optional consultations include CMS contractor site visits to states that examine the extent to which states have implemented the requirements, and the data compare service to assist states with screening providers. While most states (38) have used one or more of these consultations, 13 states have not used any. Because some states do not avail themselves of the optional consultations, these consultations do not provide CMS with information on all states’ progress in implementing the requirements. Officials from some of the seven selected states reported limitations affecting their use of the consultations. CMS contractor site visits. One-third of states (17), including three of the seven selected states, participated in at least one multi-day CMS contractor site visit, as of June 2019. Officials from CMS and its contractor told us that during the site visit, the state completes a self- assessment, followed by the contractor’s assessment on the implementation status of all provider screening and enrollment requirements to identify requirements that have not been implemented and opportunities for improving the states’ screening and enrollment procedures. (See fig. 3 for a map of states that participated in the CMS contractor site visit.) After the visit, the contractor provided a report that summarizes the state’s status toward implementing each requirement, such as full, partial, nearly complete, and not started. (See fig. 4.) CMS officials consider any requirements that are not fully implemented as “opportunities for improvement.” CMS contractor site visits are not required and are not considered audits; the agency does not track states’ progress on implementing requirements and opportunities for improvement unless the state engages CMS in follow-up. At the time of their contractor site visit, 16 of the 17 states that opted for this service had not fully implemented all provider screening and enrollment requirements. Officials from the three selected states that received a CMS contractor site visit told us that the visit helped (1) accurately identify requirements that their state had not fully implemented, or (2) establish priorities for making changes. Two of these states found it helpful to learn from the contractor about other states’ best practices and requested a return visit. Officials from another state told us they were able to make positive changes to their screening and enrollment procedures immediately after the visit, such as improving documentation of database checks through the use of screenshots to record search results. Data compare service. About half of all states (25), including four of the seven selected states, used the data compare service as of June 2019— a process by which states submit a list of all their active providers, and CMS provides a full report of the state’s providers who were previously screened by Medicare, as well as identify providers the state may need to take action on, because, among other reasons, they were terminated from Medicaid. For the states that have used the data compare service, CMS officials reported being able to screen between 40 to 80 percent of their providers. Officials from the four selected states said it was also useful for testing their provider screening and enrollment procedures to see whether the service would identify any providers they should have excluded in their screening, and three of these states said it was useful for streamlining their provider revalidations. Additionally, officials from one state that had not yet used the service told us they would consider using it in the future for both of these purposes. For example, officials from one state reported that CMS’s data compare service screened half of the approximately 80,000 providers they needed to revalidate. (See fig. 5 for a map of states that opted for the data compare service.) However, officials from all seven selected states identified limitations of the data compare service that led some states to use the service less frequently and three states to not use the service at all. CMS officials acknowledged the three limitations reported by state officials: 1. Time for receiving results. The results from the data compare service were not timely enough to help states with screening newly enrolling providers. Officials from one state explained that some provider information may become outdated by the time the results are received 6 to 8 weeks later, which makes the service less useful than it could be. 2. Different Medicare and Medicaid address entries. The data compare service’s addresses reflected Medicare practice or billing locations that may be different from providers’ Medicaid addresses. Because these addresses do not match, they could not be relied upon for updating the state’s provider records or to help states conduct site visits required for screening and enrolling moderate- and high-risk providers. 3. Additional burden for manual enrollment systems. Officials from two selected states told us that manually extracting provider data from their system—including names, addresses, Social Security numbers, and National Provider Identifiers—and manually re-entering the results from CMS for each provider into their system was burdensome and resource-intensive, leading one of these states to stop using the service. CMS offers guidance and other supports to states on a regular and periodic basis, including monthly calls with states, and assigning states to a CMS contact (see sidebar). These services also assist states with implementing the Medicaid provider screening and enrollment requirements. Officials from all of our selected states told us the guidance and other supports were helpful. According to CMS officials, the extent to which states participate in these other supports varies, because the level of participation is optional. CMS officials also told us that they use these other supports, including monthly calls and ad hoc emails, to discuss progress and keep a record of information provided; however, the agency does not revisit or require corrective actions unless the state initiates it. The PERM and other methods CMS uses to oversee states’ efforts to screen and enroll Medicaid providers do not provide CMS with comprehensive and timely information on states’ compliance with the requirements. Some methods do not fully track whether states have enrolled all types of providers and are in compliance with all the requirements; other program integrity oversight methods have not been conducted on all states. Further, these methods do not ensure timely follow-up to address identified concerns. The PERM’s components—FFS, managed care, and beneficiary eligibility determinations—measure improper payments across all states; as previously noted, the FFS component is the only component CMS uses to assess states’ compliance with provider screening and enrollment requirements. However, using the PERM to oversee states’ compliance with the requirements has limitations, including the following: The PERM does not examine whether providers under contract with MCOs are appropriately screened and enrolled. The PERM assesses states’ compliance with the provider screening and enrollment requirements by reviewing provider information for claims paid under FFS; it does not review such information for services financed under managed care. Currently, the PERM does not examine ownership disclosure and certain other provider screening and enrollment requirements. CMS officials told us the agency plans to assess the feasibility of including ownership disclosure requirements in the PERM over the next 3 years. The PERM does not ensure that CMS identifies areas of non- compliance in a timely manner. CMS conducts the PERM in each state every 3 years, and states develop corrective action plans in response to findings from the PERM; thus, it may be years before CMS identifies—and states resolve—areas of non-compliance with the provider screening and enrollment requirements. (See fig. 6.) Although CMS follows up annually with states regarding their corrective action plans, it does not fully assess states’ progress toward implementing their plans until the next PERM cycle, which is 3 years later. Further, while four of our selected states had implemented all their corrective action plans regarding provider screening and enrollment requirements within 1 year of PERM findings, the other three states had not fully implemented their plans about 2 years after PERM findings. CMS officials emphasized that developing and tracking corrective action plans was a collaborative process and that states may change corrective action plans in response to competing priorities. CMS uses other methods to oversee states’ compliance with the provider screening and enrollment requirements—focused program integrity reviews and desk reviews—that are not optional and have resulted in findings. However, these methods do not provide the agency with comprehensive and timely information on states’ compliance with the requirements. Specifically, these methods have not been conducted in all states, performed in a timely manner, or included a systematic review of states’ compliance with all the provider screening and enrollment requirements. For example: Focused program integrity reviews. CMS has not conducted focused program integrity reviews examining specific areas in Medicaid for all states. Most of the reviews performed did not include a comprehensive or timely examination of states’ compliance with provider screening and enrollment requirements. Overall, CMS has conducted 42 focused program integrity reviews in fiscal years 2014 through 2018 in 39 states. Among these reviews, nine of the 42 focused program integrity reviews examined states’ compliance with provider screening and enrollment requirements, the last of which was completed in fiscal year 2015. CMS also conducted focused program integrity reviews on managed care for 34 of the 41 states with managed care expenditures in fiscal year 2017. However, nearly all of these reviews (33) were conducted prior to January 2018 when states were required to screen and enroll all managed care providers, as required by the 21st Century Cures Act. CMS also conducted seven focused reviews examining personal care services in seven states— which include examining screening and enrollment requirements for providers of these services. Desk reviews. Off-site desk reviews that examine specific aspects of states’ program integrity activities do not include a comprehensive or timely examination of states’ compliance with the provider screening and enrollment requirements. CMS has conducted desk reviews examining activities related to provider screening and enrollment, such as corrective actions states have taken in response to PERM findings. However, desk reviews on corrective action plans are limited to examining findings on provider screening and enrollment identified during the PERM and are not conducted until 3 years after the PERM has occurred. For example, CMS told us that in 2018 the agency conducted desk reviews on the 17 states that underwent the PERM in fiscal year 2015. CMS also conducted 35 desk reviews on potential payments to terminated providers since fiscal year 2014. The PERM and CMS’s other oversight methods do not provide CMS with sufficient or timely information about states’ screening and enrollment procedures for all Medicaid provider types, and most states with managed care expenditures have not undergone a managed care-focused program integrity review since the 21st Century Cures Act screening and enrollment provisions went into effect. The lack of complete information on whether states are screening and enrolling all providers according to requirements is inconsistent with federal internal controls on assessing risk, which note that management should consider the potential for fraud when identifying, analyzing, and responding to risks. Without complete information, CMS cannot ensure that only eligible providers are participating in the Medicaid program, leaving the program vulnerable to improper payments. Further, CMS does not obtain timely information on all states’ actions to address areas of non-compliance or track progress toward addressing these areas. The length of the PERM cycle—3 years—and time for performing corrective actions to address PERM findings, limits CMS’s awareness of states’ progress, or lack thereof, toward implementing requirements. As a result, CMS lacks assurance that states are addressing areas of non-compliance or if such actions are being taken in a timely manner. This is inconsistent with federal internal controls on monitoring, which note that management should remediate deficiencies in the internal control system on a timely basis. CMS has a range of activities that provide the agency with some knowledge of states’ implementation of required provider screening and enrollment under PPACA and the 21st Century Cures Act; however, the agency’s oversight activities are not designed to systematically examine compliance with all the requirements for all providers in a timely manner. Notably, the PERM does not examine managed care providers, and CMS’s assessment of compliance and monitoring of corrective actions are not timely, because they are based on the 3-year PERM cycle. Also, focused program integrity reviews—which may examine states’ oversight of MCOs and their compliance with provider screening and enrollment requirements for providers participating in managed care—have not been conducted on all states. The one activity that can provide CMS and states with a complete and timely assessment of states’ implementation with provider screening and enrollment requirements is optional. While CMS does some tracking of state-reported information on the status of states’ implementation of the requirements, this oversight does not include states that have not availed themselves of the support CMS provides. Since states are not required to participate in these optional consultations, the states that may face the greatest challenges with implementing the provider screening and enrollment requirements might not volunteer to participate in the consultations. Without a thorough review of states’ implementation of the provider screening and enrollment requirements, as well as processes to monitor states to ensure timely remediation of deficiencies, the agency lacks assurance that only eligible providers are participating in the Medicaid program, leaving the program at risk for improper payments. We are making the following two recommendations to CMS: The Administrator of CMS should expand its review of states’ implementation of the provider screening and enrollment requirements to include states that have not made use of CMS’s optional consultations. Similar to CMS’s contractor site visits, such reviews should include any necessary steps to address areas of noncompliance for all types of enrolled providers, including those under contract with MCOs. (Recommendation 1) The Administrator of CMS should annually monitor progress toward addressing any areas of noncompliance related to the provider screening and enrollment requirements for any state with one or more corrective action plans. (Recommendation 2) We provided a draft of this report to HHS for review and comment. In its written comments, HHS concurred with our recommendations; the full text of which are reproduced in appendix I. Regarding our first recommendation, HHS stated that it will reach out to states that have not yet participated in optional consultations to discuss their progress and outline steps that the states should take to come into full compliance with the provider screening and enrollment requirements. Regarding our second recommendation, HHS stated that it is in the process of instituting more frequent reviews of corrective action plans resulting from one of CMS’s oversight activities—the PERM—stating that such reviews will now be performed quarterly. However, HHS’s comments did not discuss monitoring areas of noncompliance that are identified through other oversight activities, such as focused program integrity reviews, which include reviews of states’ screening and enrollment of providers who are under contract with MCOs. We recommend that CMS annually monitor progress toward addressing any areas of noncompliance related to the provider screening and enrollment requirements, which would include areas of noncompliance identified through the PERM, optional consultations, and other oversight activities. HHS also provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services, the Administrator of CMS, appropriate congressional committees, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or at yocomc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to this report are listed in appendix III. Appendix II: Summary of Medicaid Provider Screening and Enrollment Requirements Description The state Medicaid agency must require all enrolled providers to be screened in accordance with applicable requirements. The state Medicaid agency must have a method for verifying that a provider is licensed without restrictions in accordance with the laws of that state. The state Medicaid agency must complete revalidation of enrollment for all providers, regardless of provider type, at least every 5 years. The state Medicaid agency must deny enrollment to any provider and disclosing entity that does not successfully pass or comply with the screening process, and the agency must terminate providers who no longer meet the requirements for enrollment. The state Medicaid agency must rescreen a provider who has been deactivated for any reason prior to the provider’s reactivation. The state Medicaid agency must share with providers who are terminated or denied enrollment the process for appealing the decision. The state Medicaid agency must conduct site visits for providers who are designated as moderate or high-risk levels. Fingerprint criminal background checks The state Medicaid agency must complete fingerprint-based criminal background checks for providers and disclosing entities in the high-risk category. The state Medicaid agency must confirm the identity and determine the exclusion status of providers, any person with an ownership or control interest, and any agent or managing employee of the provider. The state Medicaid agency must require all claims for payment for items and services that were ordered or referred to contain the National Provider Identifier of the physician or other professional who ordered or referred such items or services. Screening levels for Medicaid providers The state Medicaid agency must screen all initial applications based on a categorical risk level of “limited,” “moderate,” or “high.” The state Medicaid agency must collect an application fee from institutional providers during a new enrollment or revalidation, unless Medicare or another Medicaid agency has already collected an application fee. Allows CMS and states to implement temporary moratoria pausing the enrollment of new provider types in a given location. In addition to the contact named above, Leslie V. Gordon (Assistant Director), Kristin Ekelund (Analyst-in-Charge), Manuel Buentello, Drew Long, Giao N. Nguyen, and Chris Zakroff made key contributions to this report. Also contributing were Marissa Coloske, Vikki Porter, and Jennifer Whitworth.", "summary": "A crucial component of protecting the integrity of the Medicaid program is ensuring that only eligible providers participate in Medicaid. States' non-compliance with provider screening and enrollment requirements contributed to over a third of the $36.3 billion estimated improper payments in Medicaid in 2018. To improve the integrity of the Medicaid program, PPACA and the 21st Century Cures Act established new requirements for screening and enrolling providers and expanded enrollment to include additional provider types. In this report, GAO (1) describes challenges states faced implementing provider screening and enrollment requirements; and (2) examines CMS support for and oversight of states' implementation of these requirements. GAO reviewed federal laws and CMS guidance. GAO also reviewed CMS documents, including reports resulting from CMS oversight activities published from 2014 through 2018 for seven states. These states were selected based on their use of CMS's contractor site visits, among other things. GAO also interviewed officials from CMS and the seven selected states. Officials from seven selected states that GAO interviewed described challenges they faced implementing new Medicaid provider screening and enrollment requirements, established by the Patient Protection and Affordable Care Act (PPACA) in 2010 and the 21st Century Cures Act in 2016. These challenges included establishing procedures for risk-based screenings, using federal databases and collecting required information, and screening an increased volume of providers. Due in part to these challenges, officials from five of the seven selected states told GAO they had not implemented certain requirements. For example, one state plans to launch its new information technology system, which automates screenings, before it will enroll providers under contract with managed care organizations, as required under these laws. The Centers for Medicare & Medicaid Services (CMS)—the federal agency that oversees Medicaid—supports states' implementation of new requirements with tailored optional consultations, such as CMS contractor site visits that examine the extent of states' implementation. Yet, because these are optional, states that need support might not participate, and CMS would not have information on those states. CMS uses other methods to oversee states' compliance, such as, the Payment Error Rate Measurement (PERM) process for estimating improper payments, and focused program integrity reviews. PERM. This process assesses states' compliance with provider screening and enrollment requirements, but does not assess compliance for all providers and all requirements, and occurs once every 3 years. Focused program integrity reviews. These reviews examine specific areas in Medicaid, like state compliance with provider screening and enrollment requirements, but have not been done in all states. CMS conducted reviews in 39 states in fiscal years 2014 through 2018. Collectively, CMS's oversight methods do not provide it with comprehensive and timely reviews of states' implementation of the provider screening and enrollment requirements or the remediation of deficiences. As a result, CMS lacks assurance that only eligible providers are participating in the Medicaid program. GAO recommends that CMS (1) expand its review of states' implementation of provider screening and enrollment requirements to include states that have not participated in optional consultations; and (2) for states not fully compliant with the requirements, annually monitor the progress of those states' implementation. The Department of Health and Human Services, the department that houses CMS, concurred with both recommendations.", "document_type": "gao"}
{"report": "While there is no statutory definition for virtual currency, IRS guidance has described virtual currency as a digital representation of value that functions as a medium of exchange, a unit of account, or a store of value. Some virtual currencies can be used to buy real goods and services and can be exchanged for U.S. dollars or other currencies. A cryptocurrency is a type of virtual currency that employs encryption technology and operates on distributed ledger technology, such as blockchain. Distributed ledger technology allows for users across a computer network to verify the validity of transactions potentially without a central authority. For example, a blockchain is made up of digital information (blocks) recorded in a public or private database in the format of a distributed ledger (chain). The ledger permanently records, in a chain of cryptographically secured blocks, the history of transactions that take place among the participants in the network. For the purposes of this report, we use the term virtual currency as a broad term that includes both cryptocurrencies, which use distributed ledger technology, and digital units of exchange that do not use that technology but still meet IRS’s definition of a convertible virtual currency, as defined in Notice 2014-21. Figure 1 shows a simplified representation of how distributed ledger technology is used to circulate virtual currencies. Bitcoin, which emerged in 2009, is the first and most widely circulated blockchain-based cryptocurrency. Bitcoins are created through a process called mining. Bitcoin miners download software to solve complex equations to verify the validity of transactions taking place on the network, and the first miner to solve a problem is awarded coins in return. Once a problem is solved, the transactions are added as a new block to the distributed ledger. Users transact in virtual currencies electronically through a network, and may use virtual wallets to manage their virtual currency. Some virtual currencies can be used as investments and to purchase goods and services in the real economy. For example, some retailers accept virtual currency as a form of payment. Virtual currency exchanges provide a platform where users can transact in different types of virtual currencies or exchange them for government-issued currencies or other virtual currencies. The fair market value of some virtual currencies has changed dramatically over time. For example, according to one index, the average value of one bitcoin was just under $20,000 in mid-December 2017. By early February 2018, one bitcoin was valued at about $7,000, before falling below $4,000 in December 2018, and again rising to over $9,000 in November 2019. The size of the virtual currency market is unknown due to limitations in available data. For example, one recent analysis concluded that a widely cited source for data about bitcoin trading included exaggerated data that gave an inflated impression of the size of the actual market. Nonetheless, there are data that may provide some context for the size of this market: As of April 2019, 10 major virtual currency exchanges collectively handled an average daily trading volume in bitcoin of more than $500 million, according to Bitwise. For comparison, the Federal Reserve Banks’ Automated Clearing House (a traditional payment processor) processed $103 billion in payment transactions on average per day in 2018. According to one index, the total market capitalization of bitcoin, the most widely circulated virtual currency, is estimated to have ranged between $60 billion and $225 billion between December 2018 and October 2019. As of November 2019, Coinbase, a large U.S.-based cryptocurrency exchange, reports a user base of more than 30 million. According to economists at the Federal Reserve Bank of New York, a 2018 survey they conducted found that 85 percent of respondents had heard of cryptocurrencies, 5 percent currently or previously owned cryptocurrency, and 15 percent reported that they were considering buying cryptocurrency. Federal agencies, including CFTC, FinCEN, and SEC, have jurisdiction over various aspects of virtual currency markets and market participants. In May 2014, we reported on the federal financial regulatory and law enforcement agency responsibilities related to the use of virtual currencies and their associated challenges. These challenges include money laundering, transfers of funds across borders, and consumer and investor protection issues. We also reported on the regulatory complexity for virtual currencies and the approaches that federal and state regulators have taken to their regulation and oversight. For example, CFTC has taken the position that bitcoin and ether, another virtual currency, meet the definition of a commodity provided in the Commodity Exchange Act. SEC has determined that some virtual currencies may be designated as securities, based on the characteristics of how they are offered and sold. FinCEN determined that certain virtual currency businesses would be money transmitters under the Bank Secrecy Act, subject to regulation as money services businesses. According to IRS guidance, convertible virtual currencies—which have an equivalent value in real currency or act as a substitute for real currency— are to be treated as property for tax purposes. Among other things, this classification means that income, including gains, from virtual currency transactions is reportable on taxpayers’ income tax returns. Therefore, a payment for goods or services made using virtual currency may be subject to tax to the same extent as any other payment made in property. Figure 2 illustrates examples of how virtual currency transactions can affect taxes. Taxpayers using virtual currency must keep track of transaction-level information, such as the fair market value of the virtual currency at the time it was obtained, to determine tax basis and calculate gains or losses. The gain or loss from the sale or exchange of virtual currency is characterized as either a capital gain or loss or an ordinary gain or loss, depending on whether the virtual currency is held as a capital asset. Taxpayers are required to report their gains or losses from virtual currency on their tax returns, including Form 1040, U.S. Individual Income Tax Return, and Form 8949, Sales and Other Dispositions of Capital Assets, for capital gains or losses. Figure 3 shows one example of how using virtual currencies could result in a capital gain or loss. IRS has limited data on tax compliance for virtual currency use, partly because the forms taxpayers use to report their taxable income do not require them to identify whether the source of their income is from virtual currency use. Likewise, information returns that third parties, such as employers, financial institutions, or other entities, file to report taxpayer income or transactions do not include space for, or direction to, indicate if the income or transactions reported involved a virtual currency. In 2016, the Treasury Inspector General for Tax Administration (TIGTA) found that IRS had not developed a methodology for gathering data on virtual currency use in taxable transactions that would help to analyze the risk of noncompliance and to estimate its significance. TIGTA recommended that IRS revise third-party information returns to identify the amounts of virtual currency used in taxable transactions. IRS agreed with the recommendation, but stated that it faced other higher priority funding needs, and did not consider modifying information reporting forms to be a priority at the time. As of February 5, 2020, IRS has not implemented any changes to these information returns to include information about virtual currency use. However, IRS added a question about virtual currency to Schedule 1, Additional Income and Adjustments to Income, of Form 1040 for tax year 2019. Individual taxpayers use Schedule 1 to report additional income, such as capital gains, unemployment compensation, prize or award money, and gambling winnings. IRS added a question asking if taxpayers received, sold, sent, exchanged, or otherwise acquired any financial interest in any virtual currency during the tax year. Only taxpayers who are otherwise required to file Schedule 1 or who would answer “yes” to the question need to file this schedule. According to IRS officials responsible for examining tax returns, IRS’s focus is on ensuring taxpayers are reporting all of their taxable income and it is not necessary to distinguish between virtual currency transactions and other property transactions being reported. Because IRS forms have not required taxpayers to explicitly identify income from virtual currency, IRS uses data from other sources to inform compliance decisions and research. These sources include: Searches of tax return databases. For tax years 2013 to 2015, IRS searched electronically filed Forms 8949 to identify how often taxpayers included language in the property description to indicate the transaction likely involved bitcoin, the most widely traded virtual currency at the time. For the 3 years, IRS identified fewer than 900 taxpayers who reported virtual currency activity each year. IRS officials said that due to the time and resources required to generate these data, IRS did not generate these filing statistics for tax years 2016 or later. By comparing these data to the size of the bitcoin market, IRS concluded that many taxpayers were likely not reporting income from virtual currency use. Third party information reports. To address tax noncompliance risks for virtual currencies, in December 2016, IRS served a John Doe summons to Coinbase, a U.S.-based cryptocurrency exchange. After IRS later narrowed the scope of the summons, it requested identifying and transactional data for all Coinbase users with a U.S. address, U.S. telephone number, U.S. email domain, or U.S. bank account that transacted with Coinbase between January 1, 2013, and December 31, 2015 that had the equivalent of $20,000 in any one transaction type (a buy, sell, send, or receive) in any year during that period. According to an announcement posted on Coinbase’s website, on February 23, 2018, Coinbase notified approximately 13,000 customers that it expected to deliver information about their accounts to IRS within 21 days. In addition, IRS officials stated that IRS had received information returns from a small number of virtual currency exchanges for tax year 2017. Third-party reports of potential fraud. IRS also has access to information on potential fraud reported to IRS and FinCEN by third parties. Financial institutions and money services businesses, which could include virtual currency exchanges, are to file a Suspicious Activity Report (SAR) if they observe or identify suspicious financial activity. SAR reporting can help IRS in identifying potential income underreporting, money laundering, and other potential tax-related violations and crimes. IRS may also receive information about tax noncompliance involving virtual currencies from whistleblowers and other referral programs. Voluntary disclosures by taxpayers. In March 2019, IRS updated Form 14457, Voluntary Disclosure Practice Preclearance Request and Application, to include a space specifically for taxpayers to disclose that they have unreported virtual currency income. IRS’s Criminal Investigation division (CI) reviews the forms IRS receives to ensure they meet criteria of eligibility and timeliness, and that the disclosure does not apply to illegal sources of income. CI sends forms that meet the criteria to two of IRS’s civil operating divisions—Large Business & International (LB&I) and Small Business/Self-Employed (SB/SE)—for review. According to IRS officials, the addition of virtual currency to the form was made to assist IRS employees in routing the forms to the correct subject matter experts in the civil operating divisions. According to officials with IRS’s Research, Applied Analytics, and Statistics (RAAS) division, RAAS had begun some virtual currency research projects to better understand virtual currency tax compliance. One project, which RAAS completed, was to develop compliance profiles for taxpayers that LB&I had identified through its compliance efforts as having virtual currency activity. RAAS officials also said that they are enhancing their use of a range of third-party information reporting, including reporting of virtual currency activity, to improve IRS’s ability to assess compliance risks. These efforts focus on use of data from multiple sources to better understand evolving risks and improve estimates of compliance risk. These projects support LB&I, SB/SE, CI, and IRS’s broader research, analysis, and statistical reporting needs. Virtual currency has not been included in past National Research Programs (NRP)—IRS’s detailed study of voluntary tax compliance used as the basis for tax gap estimates. The most recent NRP study of individual tax returns was tax years 2011-2013, before virtual currencies became more widely used. RAAS officials said the time frame for the next NRP study of individual tax returns has not yet been determined, but virtual currency may be included in future NRP projects. In December 2013, IRS established the Virtual Currency Issue Team (VCIT) to study virtual currencies and related compliance issues. According to IRS officials, the VCIT aimed to learn about virtual currencies, educate examiners about them, and develop examination techniques to identify and address virtual currency tax compliance risks. In 2015, the VCIT provided two training lessons for examiners on the terminology, technology, and audit issues related to virtual currencies. The VCIT is made up of about 30 individuals and continues to meet periodically to discuss virtual currency issues. In July 2018, IRS announced the launch of a virtual currency compliance campaign within LB&I to address noncompliance related to individual taxpayers’ use of virtual currency through multiple education and enforcement actions, including outreach and examinations. The goals of the compliance campaign include identifying causes of noncompliance using feedback from examination results, using information to identify additional enforcement approaches to increase compliance and decrease taxpayer burden, and improving examiner knowledge and skills as related to virtual currency transactions. According to IRS officials, the compliance campaign was initiated, in part, to analyze large amounts of data received from third-party sources. As part of the campaign, IRS developed and delivered several online and in-person training classes on blockchain technology and virtual currencies to its examiners and other staff. The trainings included details on how to identify and understand blockchain transactions and provide examiners with information on how to seek additional information from taxpayers about possible virtual currency use. According to LB&I officials, as examiners provide feedback on what new issues they are seeing in cases involving virtual currency, they will schedule follow-up training sessions to address these new issues. LB&I has also reached out to a number of external stakeholder groups to gather information and better understand the tax concerns within the virtual currency community. For example, LB&I and the IRS Office of Chief Counsel have spoken to tax practitioner groups, state tax authorities, IRS Nationwide Tax Forum participants, and tax preparation software companies. According to IRS officials, the discussions they had with tax preparation software companies led to some adding questions to their programs asking taxpayers to enter virtual currency income when preparing their tax returns. The compliance campaign also aims to assist in developing a comprehensive IRS virtual currency strategy. In addition to leading the compliance campaign, LB&I is also leading a working group focused on cryptocurrency that includes members from across IRS, including LB&I, SB/SE, CI, and the Office of Chief Counsel. This working group reports to the IRS Enforcement Committee, which includes the Deputy Commissioner for Services and Enforcement and the commissioners for each of the operating divisions and CI. CI has been assisting in analyzing data received from third-party sources to look for potential investigative leads. According to CI officials, CI first reviews the data to identify any taxpayers who are already targets of CI investigations so that LB&I does not use the information in its civil enforcement efforts. The officials also said that they were reviewing information from large virtual currency users to identify any ties to criminal activity. However, according to IRS officials, since some of the data IRS has received predate a major uptick in virtual currency activity in 2017, the data that predate these developments are less valuable than more recent data would be, other than to understand the history of an individual’s virtual currency usage. IRS has also begun civil enforcement activities to address virtual currency noncompliance as part of the compliance campaign. In April 2019, LB&I was forwarding cases identified as likely involving virtual currency for examination classification, the process IRS uses to determine which returns to select to examine. Due to the time needed to complete examinations and to allow taxpayers time to exercise their rights, IRS officials said they do not have outcome data from these efforts yet. In July 2019, IRS began sending out more than 10,000 letters to taxpayers with virtual currency transactions. These letters stated that IRS is aware that the taxpayer may have a virtual currency account. They instructed the taxpayer to ensure that virtual currency income, gains, and losses have been reported appropriately and to file or amend returns as necessary. The letters also provide taxpayers with information on where they can find resources to help them understand their reporting obligations. According to IRS officials, CI works with a number of federal partners, including FinCEN and the Federal Bureau of Investigation (FBI), among others, in the routine course of its work, which may involve virtual currency issues. According to CI officials, virtual currency does not constitute a new program area that would require a new specific set of policies and procedures. Instead, traditional crimes that CI might investigate may be intertwined with virtual currency use. CI participates in virtual currency issue information sharing efforts through a number of groups. For example, CI is a monthly participant in the FBI’s National Cyber Investigative Joint Task Force, which brings agencies together to share intelligence and work large-scale cases jointly. CI also has agents on site at the National Cyber-Forensics and Training Alliance, a public-private partnership, and at the European Union Agency for Law Enforcement Cooperation. Both entities work on a variety of issues, including virtual currency issues. CI also participates in some multinational information sharing groups to address virtual currency issues as part of its broader criminal enforcement goals. For example, CI participates in the Joint Chiefs of Global Tax Enforcement (J5), a group of criminal intelligence and tax officials from Australia, Canada, the Netherlands, the United Kingdom, and the United States that launched in mid-2018 to focus on shared cross-national tax risks, including cybercrimes and virtual currency. Among the goals of the J5 are to lead the international community in developing a strategic understanding of offshore tax crimes and cybercrimes, and raise international awareness that the J5 are working together to address international and transnational tax crimes. Within the Department of the Treasury (Treasury), IRS works with Treasury’s Office of Tax Policy when developing any guidance or regulation, including for virtual currency. IRS also works with FinCEN with regard to IRS’s delegated authority to administer parts of the Bank Secrecy Act, including Report of Foreign Bank and Financial Accounts (FBAR) filings. For example, FinCEN provides training materials to SB/SE examination staff who may come across virtual currency issues in the performance of a Bank Secrecy Act examination. IRS and FinCEN officials also periodically discuss how to apply the Bank Secrecy Act and its implementing regulations to virtual currency transactions. Given IRS’s unique role in administering the federal tax system, it generally does not need to coordinate with other agencies outside of Treasury in developing or issuing virtual currency guidance or taking civil enforcement actions. According to IRS officials, the work of the virtual currency compliance campaign does not involve any other federal agencies. IRS first issued virtual currency guidance in 2014, in response to our recommendation. In 2013, we found that IRS had not issued guidance specific to virtual currencies and that taxpayers may be unaware that income from transactions using virtual currencies could be taxable. We recommended that IRS provide taxpayers with information on the basic tax reporting requirements for transactions using virtual currencies. In response to this recommendation, IRS issued Notice 2014-21 in March 2014 and published it in the Internal Revenue Bulletin (IRB) in the form of answers to frequently asked questions (FAQs). IRS solicited public input on Notice 2014-21 through several means. Within the notice, IRS requested comments from the public regarding other aspects of virtual currency transactions that should be addressed in future guidance by providing a physical and email address to which comments could be submitted. IRS reviewed more than 200 public comments it received to identify topics that were in need of further guidance. Our analysis of the public comments found that the most common topics concerned tax forms and reporting (64 comments), realization of income (45 comments), cost basis (33 comments), and general tax liability (29 comments). Other topics included the tax implications of hard forks and airdrops, mining, and foreign reporting. Virtual currency stakeholders we spoke with, such as tax practitioners, executives at virtual currency exchanges, advocacy groups, and industry representatives also identified these topics as in need of further guidance. Additionally, LB&I officials said they held several sessions to gather information from external stakeholders, such as tax practitioner groups and state tax authorities, to develop a better understanding of what was happening in taxpayer communities. In October 2019, IRS issued two forms of additional virtual currency guidance, which answered some questions previously raised by the public comments and virtual currency stakeholders. According to IRS, these guidance documents were intended to supplement and expand upon Notice 2014-21. Revenue Ruling 2019-24 addresses the tax treatment of hard forks and airdrops following hard forks. Specifically, the guidance discusses whether taxpayers have gross income as a result of (1) a hard fork, if they do not receive units of a new virtual currency; or (2) an airdrop of a new virtual currency following a hard fork if they receive units of new virtual currency. Additional FAQs provide further examples of how tax principles apply to virtual currency held as a capital asset. Topics addressed include what tax forms to use when reporting ordinary income and capital gains or losses from virtual currency; how to determine fair market value of virtual currencies; when virtual currency use results in taxable income; how to determine cost basis in several scenarios; and when a taxpayer may use the First-In-First-Out accounting method, known as FIFO, to calculate their gains. However, some virtual currency and tax stakeholders with whom we spoke expressed concern that the 2019 revenue ruling and FAQs leave many questions unanswered and provide confusing responses to others. Their concerns include the following: Clarity: According to some stakeholders, Revenue Ruling 2019-24 is unclear, mostly due to confusion surrounding IRS’s usage of technical virtual currency terminology and the situations meant to illustrate IRS’s application of the law to hard forks and airdrops. Several tax and virtual currency stakeholders we spoke with said these examples do not accurately explain how virtual currency technology works and therefore may not be helpful to taxpayers looking for guidance on the tax implications of income received as a result of hard forks or airdrops. In public remarks on the new guidance in October 2019, IRS’s Chief Counsel stated that terms are not used in a uniform way in the virtual currency industry, but IRS is interested in receiving comments on how virtual currency technology should be described. Additional topics in need of guidance: The revenue ruling and additional FAQs do not address several topics raised in the public comments and by stakeholders. For example, the guidance does not clarify foreign asset reporting requirements for virtual currency. The statutory provisions commonly known as the Foreign Account Tax Compliance Act (FATCA) require taxpayers and foreign financial institutions to report on certain financial assets held outside the United States. Regulations implementing the Bank Secrecy Act separately require taxpayers to report certain foreign financial accounts to FinCEN on the FBAR form. Some practitioners told us that it is unclear whether these requirements apply to virtual currency wallets and exchanges, as we discuss later in this report. Other topics not addressed in the 2019 guidance include mining, like-kind exchanges, and retirement accounts. According to an official from the IRS Office of Chief Counsel, IRS’s focus when developing the 2019 guidance was to assist individual taxpayers. Therefore, the topics addressed by the revenue ruling and FAQs were limited to the most common issues that would be applicable to most individual taxpayers. The official told us that if IRS were to develop additional virtual currency guidance in the future, it may focus on a different audience, such as taxpayers involved in virtual currency businesses or exchanges that could be subject to third-party information reporting. Another official stated that issuing guidance on certain topics, including like-kind exchanges, would have taken additional time, and these topics were therefore left unaddressed. IRS issues thousands of publications in a variety of different forms to help taxpayers and their advisors understand the law; however, IRS has stated that only guidance published in the IRB contains IRS’s authoritative interpretation of the law. Unlike with the virtual currency FAQs IRS issued in 2014 in the form of a notice, the 2019 FAQs were not published in the IRB. Therefore, the 2019 FAQs are not binding on IRS, are subject to change, and cannot be relied upon by taxpayers as authoritative or as precedent for their individual facts and circumstances. For FAQs not published in the IRB, tax practitioners have noted that sometimes IRS has included a disclaimer noting that the FAQs do not constitute legal authority and may not be relied upon. The new virtual currency FAQs do not include such a disclaimer. According to IRS officials, they did not include a disclaimer along with the new FAQs because the FAQs do not contain any substantial new interpretation of the law. IRS officials did not feel that a disclaimer about the limitations of the FAQs was necessary or that it would be helpful to taxpayers. However, the FAQs provide new information, such as a definition of the term “cryptocurrency” and an explanation of how taxpayers can track cost basis for virtual currency. As we have previously reported, clarity about the authoritativeness of certain IRS publications could be improved by noting any limitations, especially when FAQs provide information to help taxpayers comply with tax law. Additional explanatory language would help taxpayers understand what type of IRS information is considered authoritative and reliable as precedent for a taxpayer’s individual facts and circumstances. The first article in IRS’s Taxpayer Bill of Rights—“The Right to Be Informed”—states that taxpayers have the right to know what they need to do to comply with tax laws. The article further states that taxpayers are entitled to clear explanations of the laws and IRS procedures in all forms, instructions, publications, notices, and correspondence. As we have previously reported, just as taxpayers have the right to clear explanations in IRS instructions and publications, taxpayers should be alerted to any limitations that could make some IRS information less authoritative than others. Failing to note any limitations associated with particular guidance could lead to misinterpretation of nonauthoritative information from IRS. If taxpayers make decisions based on guidance that is nonauthoritative, including FAQs, those taxpayers’ confidence in IRS and the tax system could be undermined if the content is later updated and IRS challenges taxpayers’ positions. As we have noted in prior reports, taxpayers’ perception that IRS is fairly and uniformly administering the tax system helps further overall voluntary compliance and lowers IRS’s administrative costs. IRS does not receive information returns on some potentially taxable transactions involving virtual currency, which limits its ability to detect noncompliance. Some virtual currency exchanges send information returns to IRS and to customers that provide information about customers’ trading activity, but others do not. Financial institutions and other third parties are to report interest payments, property sales, and other transactions to both taxpayers and IRS using forms known as information returns. Form 1099-K, Payment Card and Third Party Network Transactions. Third parties that contract with a substantial number of unrelated merchants to settle payments between the merchants and their customers are required to issue a Form 1099-K for each merchant that meets the threshold of having more than 200 transactions totaling more than $20,000 in a year. Form 1099-B, Proceeds from Broker and Barter Exchange Transactions. Brokers use Form 1099-B to report transactions such as sales or redemptions of securities, regulated futures contracts, and commodities. For certain types of property, brokers must also report cost basis information on Form 1099-B if the information is required. Form 1099-MISC, Miscellaneous Income. Certain payments made in the course of a trade or business—including rents, prizes, and various other types of income—must be reported by the payer on Form 1099-MISC. For most types of income subject to reporting on Form 1099-MISC, payers must file the form only if they made payments totaling at least $600. According to our review of websites for nine major U.S.-based virtual currency exchanges, as of November 2019, two exchanges have policies posted online stating that they report information for some of their customers’ virtual currency transactions to IRS on Form 1099-K. One exchange states that it reports customers’ transactions on Form 1099-B, a more detailed information return that provides a breakdown of individual virtual currency transactions. Another exchange’s website states that it provides Forms 1099, but does not identify the form more specifically. Three exchanges’ websites have policies stating that the exchanges do not report customers’ transactions on tax forms. The remaining two exchanges do not state on their websites whether or not they file information returns or provide customers with tax forms. When transactions handled by third parties, such as virtual currency exchanges, go unreported on information returns, it is difficult for IRS to identify and address compliance risks. According to IRS officials and tax practitioners we interviewed, it is difficult for IRS to find out when taxable transactions involving virtual currency are occurring. As discussed earlier in this report, IRS’s virtual currency compliance campaign has identified more than 10,000 taxpayers who may not have properly reported virtual currency transactions on tax returns. However, the campaign likely has not identified all taxpayers with underreported virtual currency income. In addition, according to IRS officials, examining tax returns is more resource intensive than the automated processes IRS uses to match tax returns against information returns. For taxpayers, limited information reporting by third parties can make it difficult to complete tax returns. Tax practitioners told us that recordkeeping is a challenge for taxpayers who buy and sell virtual currencies. To report virtual currency income accurately under IRS guidance, taxpayers need to report information about each transaction, including cost basis and fair market value at the time virtual currency is disposed of, such as by selling it for cash or another virtual currency on an exchange. Some taxpayers may not keep their own records of virtual currency transactions, and as a result may lack easy access to the information that would be provided in third-party information returns. When taxpayers do keep these records, they may not know how to report virtual currency transactions on tax forms. As discussed earlier in this report, 64 of the public comments IRS received on Notice 2014-21 were about forms and reporting. For example, some of these 64 comments expressed uncertainty about how to calculate the fair market value of virtual currency at the time of sale; others requested assistance in determining which tax forms to use to report income from virtual currency transactions. Some virtual currency transactions are not subject to third-party reporting requirements. For example, unless owned by a U.S. payor (including a controlled foreign corporation), virtual currency exchanges operating outside the United States are not required to file information returns such as Forms 1099-K or 1099-B unless the customer or transaction has certain connections to the United States. Some transactions, such as transferring virtual currency directly to a merchant in exchange for goods, generally create no obligation to file any information returns. Other virtual currency transactions, such as sales of virtual currency for cash through virtual currency exchanges, may be subject to third-party reporting requirements. However, those requirements are not entirely clear, and people have interpreted them differently. Tax practitioners we spoke with generally stated that it is not clear whether current regulations require virtual currency exchanges to report customers’ trading activity on Forms 1099-K or 1099-B. According to IRS officials, virtual currency exchanges may be subject to the 1099-K reporting requirement if they fall into the legal category of “third party settlement organizations.” Exchanges are subject to the 1099-B requirement only if they are brokers or barter exchanges. IRS does not have an official position on whether virtual currency exchanges are required to report customers’ trading activity on Form 1099-B. There may also be ambiguity regarding when, if at all, reporting on virtual currency sales is required on Form 1099-MISC. Furthermore, even if exchanges are subject to the 1099-K, 1099-B, or 1099-MISC reporting requirements, these requirements do not cover all taxable transactions. Third-party settlement organizations are required to file Form 1099-K only for customers who make more than 200 transactions in a year that total more than $20,000. Taxable transactions below that threshold may not be reported. Separately, some transactions carried out by brokers do not need to be reported on Form 1099-B unless they involve cash. For example, taxpayers must report trades between different virtual currencies on tax returns, but brokers may not be required to report such trades on Form 1099-B. According to IRS, a virtual currency exchange would be required to file Form 1099-MISC if it has sufficient information, such as the recipient’s basis in the virtual currency, to determine whether a payment made to a recipient in exchange for virtual currency gives rise to income for that recipient. In addition, Forms 1099-K, 1099-B, and 1099-MISC do not always contain all the information that taxpayers need to file accurate tax returns or that IRS needs to monitor compliance. Form 1099-K provides information on the number and gross amount of payments made to the recipient, but does not provide information about individual transactions. Some tax practitioners we interviewed stated that taxpayers who receive Form 1099-K for virtual currency transactions may find the form unhelpful or confusing. Because the form does not identify specific transactions, it may be difficult to match the aggregate amounts reported on the form with taxpayers’ own records of virtual currency transactions. Form 1099-B does provide information about individual transactions, but does not always include or require cost basis information. According to IRS, a Form 1099-MISC that reports a payee’s gain does not provide information about that payee’s gross proceeds and basis. Some stakeholders we interviewed mentioned challenges that could make it difficult to implement information reporting at the individual transaction level. For example, it could be difficult to distinguish between taxable dispositions of virtual currency—such as the sale of virtual currency for U.S. dollars—and nontaxable events such as the transfer of virtual currency from a taxpayer’s account on an exchange to a personal wallet controlled directly by the same taxpayer. These stakeholders also told us that if exchanges were required to report cost basis information, additional challenges could include tracking the cost basis of virtual currency transferred between exchanges. However, as we have previously reported, cost basis reporting can be particularly valuable for tax compliance. IRS officials told us that they are studying the issue of third-party information reporting, and it is included in IRS’s priority guidance plan as of October 2019. We have reported that, in general, the extent to which taxpayers accurately report their income is closely aligned with the amount of income that third parties report to them and to IRS. For example, according to IRS data for tax years 2011-2013, taxpayers misreported more than half of their income for types of income subject to little or no third-party information reporting (see figure 4). Taxpayers misreported a much lower percentage of their income for types of income subject to at least some information reporting. Information returns that include details about individual transactions can assist taxpayers by providing information about how to report virtual currency income correctly. For example, in addition to providing transaction details, Form 1099-B instructs recipients where to report transactions on Form 8949 or Schedule D, which are forms used to report capital gains. By contrast, Form 1099-K does not include similar instructions. One of IRS’s strategic goals is to protect the integrity of the tax system by encouraging compliance through administering and enforcing the tax code. This goal includes identifying and planning for compliance risks proactively, including risks associated with the increasing complexity of the tax base. Further, internal control standards state that management should use quality information to achieve the entity’s objectives. Using quality information requires identifying information requirements and obtaining relevant data from reliable sources. As discussed above, IRS does not have quality information on many potentially taxable transactions involving virtual currency, in part because information reporting requirements for virtual currency exchanges are unclear, and in part because some information reporting does not include detailed information about specific transactions. As a result, some taxpayers may not be reporting virtual currency transactions properly on their tax returns or paying the full amount of tax owed on those transactions, contributing to the tax gap. As previously discussed, two overlapping reporting requirements apply to taxpayers who have foreign financial assets. These two requirements are the Report of Foreign Bank and Financial Accounts (FBAR) filings required under the Bank Secrecy Act and the separate reports required by the statutory provisions commonly known as the Foreign Account Tax Compliance Act (FATCA). The federal agencies that administer these requirements have not clarified how taxpayers who hold virtual currency should interpret them. Under FATCA, taxpayers have an obligation to report certain foreign financial accounts and other assets on IRS Form 8938, Statement of Specified Foreign Financial Assets, if the value of those assets exceeds a certain amount. FATCA was enacted in 2010 to reduce offshore tax evasion, and it also requires foreign financial institutions to report detailed information to IRS about their U.S. customers. Tax practitioners we interviewed told us that there is no generally accepted view about whether FATCA filing requirements apply to virtual currency holdings, and IRS has not publicly stated a position on how, if at all, FATCA requirements apply to virtual currency holdings for either taxpayers or institutions. Some practitioners stated that in the absence of guidance or information from IRS specifically addressing virtual currency and FATCA, some of their clients report foreign virtual currency accounts because the potential penalties for failing to report, if deemed to be required, are high. Additionally, several public comments on IRS Notice 2014-21 requested clarification from IRS about whether virtual currency holdings must be reported under FATCA. The FATCA filing requirements can be difficult for individual taxpayers to interpret, in part because FATCA was enacted before the use of virtual currency became more widespread, and it was not designed to cover nontraditional assets such as virtual currencies. For example, under FATCA, taxpayers must report accounts at foreign financial institutions. A taxpayer who holds virtual currency with an exchange based outside the United States may not know whether the exchange counts as a foreign financial institution under FATCA because this determination involves applying legal criteria to specific facts about how the exchange operates. Taxpayers must also report foreign nonaccount assets held for investment (as opposed to held for use in a trade or business), such as foreign stock and securities, foreign financial instruments, contracts with non-U.S. persons, and interests in foreign entities. IRS officials told us that in some situations, virtual currencies could be foreign nonaccount assets, depending on specific facts about how an individual taxpayer holds the virtual currency. However, a taxpayer holding virtual currency may not know whether the virtual currency is considered a specified foreign financial asset because this determination involves applying legal criteria to specific facts such as whether the virtual currency has a foreign issuer, which the taxpayer may not have sufficient information to determine. According to IRS officials, they have not issued guidance about virtual currency and FATCA because the instructions for Form 8938 clearly explain how taxpayers are to interpret FATCA requirements. However, those instructions do not mention virtual currency and do not provide information needed to determine whether virtual currency holdings must be reported. For example, the instructions state that a financial account is any depository or custodial account maintained by a foreign financial institution, but do not explain under what circumstances, if any, an account that holds virtual currency could be considered a depository or custodial account. IRS’s Taxpayer Bill of Rights states that taxpayers are entitled to clear explanations of the laws and IRS procedures in all tax forms, instructions, publications, notices, and correspondence. Furthermore, one of IRS’s strategic goals is to empower taxpayers by making it easier for them to understand and meet their filing, reporting, and payment obligations. Without information about how to interpret and apply FATCA requirements to situations involving virtual currency, taxpayers will not know whether they are required to report virtual currency held outside the United States. As a result, they may be underreporting, depriving IRS of data needed to address offshore tax evasion, or overreporting by filing forms that are not required. As we have previously reported, such overreporting creates unnecessary burdens, including financial costs, for taxpayers. Separate from the requirement to file Form 8938 under FATCA, regulations implementing the Bank Secrecy Act require reporting of financial accounts maintained with financial institutions located outside the United States on the FBAR form. FinCEN’s FBAR regulations predate the widespread use of virtual currency and do not specifically mention virtual currency. Consequently, tax practitioners have raised questions about whether taxpayers are required to include virtual currency holdings in FBAR filings. In correspondence and interviews, FinCEN officials have stated that, based on their understanding of the regulations, virtual currency does not need to be reported on the FBAR. For example, FinCEN officials told us that FinCEN provides a standard response when members of the public ask FinCEN’s Resource Center about reporting virtual currency on the FBAR. The response states, in part, “as of right now, reporting [virtual currency exchange accounts] on the FBAR is not required.” Likewise, in March 2019, FinCEN responded in writing to a question from the American Institute of Certified Public Accountants by stating that the FBAR regulations do not define virtual currency held in an offshore account as a type of reportable account. While FinCEN has provided responses to direct questions, it has not made information about whether foreign virtual currency accounts are subject to the FBAR requirement readily available, such as by posting this information on its website. FinCEN officials stated that FinCEN and IRS had issued a statement on IRS’s website in 2014 informing the public that virtual currencies did not need to be reported on the FBAR. However, the officials noted that the statement was no longer available on the website, but they did not say when it may have been removed or why. Neither IRS’s FBAR Reference Guide nor FinCEN’s instructions for filing the FBAR mention virtual currencies. Internal control standards state that management should externally communicate the necessary quality information to achieve the entity’s objectives. As part of this standard, management should communicate information that allows external parties, including the general public, to assist the entity in achieving its objectives. In the absence of a readily available official statement from FinCEN that virtual currencies are not reportable on the FBAR, users of virtual currency may be filing reports that are not legally required. According to some tax practitioners we interviewed, some individuals may report foreign virtual currency accounts on the FBAR even if they believe it is unlikely that they are required to report, because of the high penalties for failing to file required FBARs. Such filings can create financial costs and unnecessary recordkeeping and other burdens for these individuals. Virtual currencies can present challenges for enforcement of tax laws, both because they can be circulated without a central authority and because complying with current tax requirements can be confusing and burdensome. IRS has taken important steps to address these challenges, including issuing multiple sets of guidance to clarify how virtual currencies would be treated for tax purposes and carrying out a range of enforcement activities to address noncompliance. Although IRS’s 2019 virtual currency guidance addressed some issues left unresolved by its 2014 guidance, it did not address others, and it has also prompted new concerns among virtual currency stakeholders. Additionally, including information that the 2019 FAQs are not legally binding would enhance taxpayer understanding and could ultimately help enhance taxpayers’ confidence in IRS and the tax system. Currently, much trading activity in virtual currency goes unreported on information returns. In part, this lack of reporting may be because third parties are unclear about whether they are required to report. Limitations in what information returns report related to virtual currencies also constrain the utility of reported information. In general, information reporting is associated with high levels of compliance. Additionally, the rules for foreign asset reporting—specifically, the FBARs required by the Bank Secrecy Act and the separate reports required by FATCA—do not clearly address virtual currency, and tax professionals have raised questions about the applicability of these requirements to virtual currency. Clarifying the FATCA requirements and making a statement about the FBAR requirements readily available to the public would help reduce uncertainty about these rules and may result in reduced burden for some taxpayers who may be filing reports that are not required. We are making a total of four recommendations, including three to IRS and one to FinCEN. Specifically, The Commissioner of Internal Revenue should update the FAQs issued in 2019 to include a statement that the FAQs may serve as a source of general information but cannot be relied upon by taxpayers as authoritative since they are not binding on IRS. (Recommendation 1) The Commissioner of Internal Revenue should take steps to increase third-party reporting on taxable transactions involving virtual currency, which could include clarifying IRS’s interpretation of existing third-party reporting requirements under the Internal Revenue Code and Treasury Regulations, or pursuing statutory or regulatory changes. (Recommendation 2) The Commissioner of Internal Revenue should clarify the application of reporting requirements under FATCA to virtual currency. (Recommendation 3) The Director of FinCEN, in coordination with IRS as appropriate, should make a statement about the application of foreign account reporting requirements under the Bank Secrecy Act to virtual currency readily available to the public. (Recommendation 4) We provided a draft of this report to IRS, FinCEN, Treasury, SEC, and CFTC for review and comment. In its written comments, which are summarized below and reproduced in appendix II, IRS agreed with one and disagreed with two of the recommendations directed to it. In its written comments, which are summarized below and reproduced in appendix III, FinCEN agreed with the recommendation directed to it. IRS, Treasury, SEC, and CFTC provided technical comments, which we incorporated as appropriate. IRS agreed with the recommendation to take steps to increase third-party reporting on taxable transactions involving virtual currency (recommendation 2). IRS stated that it is working with Treasury to develop guidance on third-party reporting under section 6045 of the Internal Revenue Code for certain taxable transactions involving virtual currency. Such guidance, if it aims to increase third-party reporting, would address the intent of the recommendation. IRS disagreed with the recommendation to add a statement to the 2019 FAQs on virtual currency informing taxpayers that the FAQs provide general information but are not binding on IRS (recommendation 1). IRS stated that the FAQs are illustrative of how longstanding tax principles apply to property transactions. IRS also stated that IRS does not take positions contrary to public FAQs. We continue to believe that including such a statement would provide more transparency and help taxpayers understand the nature of the information provided in the FAQs. As we state earlier in this report, IRS has included disclaimer statements in other informal FAQs posted on its website. IRS could include a similar statement in the virtual currency FAQs at minimal cost. Alternatively, if IRS intends to be bound by the positions it takes in the current version of the virtual currency FAQs, as the response to this recommendation suggests, it could publish the FAQs in the Internal Revenue Bulletin. Doing so would render a disclaimer statement unnecessary and would satisfy the intent of the recommendation. IRS disagreed with the recommendation to clarify the application of FATCA reporting requirements to virtual currency (recommendation 3). IRS stated that U.S. exchanges and other U.S. businesses play a significant role in virtual currency transactions carried out by U.S. taxpayers, and therefore it is appropriate for IRS to focus on developing guidance for third-party reporting under section 6045, as discussed above. IRS also stated that guidance on FATCA may be appropriate in the future when the workings of foreign virtual currency exchanges become more transparent. We believe that, given the widespread uncertainty about the FATCA requirements among virtual currency stakeholders, it would benefit taxpayers for IRS to clarify these requirements to the extent possible with the information currently available. It may be appropriate to wait for future developments in the foreign virtual currency exchange industry before issuing detailed, thorough guidance on this issue. However, IRS could address the uncertainty about the FATCA requirements by clarifying in general terms how it believes they should be interpreted in situations involving virtual currency. In its comments, FinCEN agreed with the recommendation to make a public statement about whether virtual currency must be reported on the FBAR (recommendation 4). FinCEN confirmed in its letter that as of January 2020, its regulations do not require virtual currency held in an offshore account to be reported on the FBAR. Additionally, FinCEN stated that it will coordinate with IRS to determine the best approach to provide clarity to the public regarding the FBAR requirement. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of the Treasury, the Commissioner of Internal Revenue, the Director of the Financial Crimes Enforcement Network, the Chairman of the Securities and Exchange Commission, the Chairman of the Commodity Futures Trading Commission, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9110 or mctiguej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our objectives were to (1) describe what is known about virtual currency tax compliance; (2) describe the steps the Internal Revenue Service (IRS) has taken to address virtual currency tax compliance risks; (3) evaluate the extent to which IRS’s virtual currency guidance meets taxpayer needs; and (4) evaluate whether additional information reporting could assist IRS in ensuring compliance. To describe what is known about virtual currency tax compliance and the steps IRS has taken to address virtual currency tax compliance risks, we reviewed IRS documentation on the agency’s virtual currency tax enforcement efforts, including information about the legal summons IRS issued to Coinbase and the Large Business and International (LB&I) division’s virtual currency compliance campaign. We interviewed IRS officials in the Small Business/Self Employed (SB/SE) and LB&I operating divisions, as well as the Research, Applied Analytics, and Statistics division about any data the agency had on virtual currency tax compliance, challenges in collecting such data, and plans for data analyses. We also reviewed IRS forms that taxpayers may use to report virtual currency use. We interviewed officials from the Financial Crimes Enforcement Network, Commodity Futures Trading Commission, and Securities and Exchange Commission about coordination efforts that have been made across agencies regulating virtual currencies. We also interviewed tax practitioners, tax attorneys, virtual currency industry advocates, and virtual currency exchange executives about virtual currency tax compliance issues. We took a snowball sampling approach to identify the outside stakeholders we interviewed, which involved asking stakeholders we interviewed for recommendations of others we should contact to gain additional insight into virtual currency tax compliance, and we assessed their qualifications and independence. In total, we interviewed five individual stakeholders in addition to representatives of 10 entities with expertise in tax issues related to virtual currency. Although results from these interviews are not generalizable, they provide insight into what is known about tax compliance and the steps IRS has taken to address virtual currency tax compliance risks. To evaluate the extent to which IRS’s virtual currency guidance meets taxpayer needs, we identified and analyzed all of the guidance and statements IRS has published about tax compliance for virtual currencies. To identify these documents, we searched IRS’s website and interviewed IRS officials. According to IRS officials, Notice 2014-21, issued in March 2014, and Revenue Ruling 2019-24 and Frequently Asked Questions (FAQs), issued in October 2019, are the only IRS guidance specific to virtual currencies. We also reviewed and analyzed all of the public comments IRS had received on Notice 2014-21 as of August 19, 2019, to determine the concerns raised about virtual currency tax compliance. IRS sent us 229 public comments. We identified 25 of the comments as not applicable because they were not related to Notice 2014-21, were duplicate comments, or were otherwise not relevant. Two reviewers coded the content of the 204 applicable public comments and grouped them into 13 different thematic categories. We developed these categories based on the topics or issues that commenters identified. We assigned each separate issue raised by a comment to an existing category unless it did not relate to any of the existing categories, in which case we created a new category. We also recorded the date the comment was submitted and the occupation of the commenter, if specified in the comment. To assess the reliability of these data, we reviewed relevant documentation and consulted knowledgeable IRS officials. Specifically, we requested information from IRS’s Office of Chief Counsel to identify the quality controls in place to help ensure all comments are processed. We determined that the data were sufficiently reliable for our purposes. The information we obtained from these comments may not be representative of the viewpoints of the entire U.S. public. In addition, we interviewed the stakeholders mentioned above before IRS released new guidance in October 2019 to identify any taxpayer concerns, any compliance challenges with virtual currency tax obligations, and the extent to which the guidance provided in IRS’s Notice 2014-21 was meeting taxpayer needs. We reached out to these same stakeholders in October 2019, after IRS issued a new set of FAQs and Revenue Ruling 2019-24, to determine how these new guidance documents addressed taxpayers’ concerns. Of the five individuals and 10 groups we initially interviewed, we received responses regarding the new IRS guidance from four individuals and six groups. The information we obtained from these practitioners and exchanges is not generalizable to all practitioners and exchanges because we took a snowball sampling approach, but the information provides insight into the extent to which IRS’s virtual currency guidance is meeting the needs of taxpayers. To evaluate whether additional information reporting could assist IRS in ensuring compliance, we reviewed IRS’s requirements for information reporting for virtual currency transactions, including the laws and regulations for foreign asset reporting. We interviewed IRS officials in the SB/SE and LB&I operating divisions about how IRS’s third-party and taxpayer information reporting processes and current forms assist in IRS’s work to detect noncompliance for virtual currencies. We reviewed the websites of a judgmental selection of nine virtual currency exchanges for policies or statements about tax reporting, including whether the exchanges file Forms 1099-B or 1099-K. For the website review, we selected virtual currency exchanges that were based in the United States and that were likely, because of their size or public profile within the virtual currency industry, to have established policies regarding information reporting. For each exchange, we identified and categorized any statements on the exchange’s website regarding tax or information reporting, such as a statement that the exchange does not provide any tax forms to customers or a statement that the exchange provides information on a specific form to customers and IRS. We also interviewed the stakeholders mentioned above to determine what information is being reported to IRS and whether additional information reporting would help IRS and taxpayers with ensuring tax compliance. We interviewed executives from two exchanges to determine what burden, if any, information reporting does or could impose on exchanges and virtual currency users. We attempted to contact four additional exchanges but did not receive a response. Because we used a snowball sampling approach, the information we obtained from these virtual currency industry participants is not generalizable to all virtual currency industry participants. We conducted this performance audit from October 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Jeff Arkin (Assistant Director), Danielle Novak (Analyst-in-Charge), Theodore Alexander, Michael Bechetti, David Blanding, Jacqueline Chapin, Ed Nannenhorn, Bruna Oliveira, Kayla Robinson, and Andrew J. Stephens made key contributions to this report.", "summary": "Virtual currencies, such as bitcoin, have grown in popularity in recent years. Individuals and businesses use virtual currencies as investments and to pay for goods and services. GAO was asked to review IRS's efforts to ensure compliance with tax obligations for virtual currencies. This report examines (1) what is known about virtual currency tax compliance; (2) what IRS has done to address virtual currency tax compliance risks; (3) the extent to which IRS's virtual currency guidance meets taxpayer needs; and (4) whether additional information reporting on virtual currency income could assist IRS in ensuring compliance. GAO reviewed IRS forms and guidance and interviewed officials at IRS, FinCEN, and other federal agencies, as well as tax and virtual currency stakeholders. Taxpayers are required to report and pay taxes on income from virtual currency use, but the Internal Revenue Service (IRS) has limited data on tax compliance for virtual currencies. Tax forms, including the information returns filed by third parties such as financial institutions, generally do not require filers to indicate whether the income or transactions they report involved virtual currency. IRS also has taken some steps to address virtual currency compliance risks, including launching a virtual currency compliance campaign in 2018 and working with other agencies on criminal investigations. In July 2019, IRS began sending out more than 10,000 letters to taxpayers with virtual currency activity informing them about their potential tax obligations. IRS's virtual currency guidance, issued in 2014 and 2019, addresses some questions taxpayers and practitioners have raised. For example, it states that virtual currency is treated as property for tax purposes and that using virtual currency can produce taxable capital gains. However, part of the 2019 guidance is not authoritative because it was not published in the Internal Revenue Bulletin (IRB). IRS has stated that only guidance published in the IRB is IRS's authoritative interpretation of the law. IRS did not make clear to taxpayers that this part of the guidance is not authoritative and is subject to change. Information reporting by third parties, such as financial institutions, on virtual currency is limited, making it difficult for taxpayers to comply and for IRS to address tax compliance risks. Many virtual currency transactions likely go unreported to IRS on information returns, due in part to unclear requirements and reporting thresholds that limit the number of virtual currency users subject to third-party reporting. Taking steps to increase reporting could help IRS provide taxpayers useful information for completing tax returns and give IRS an additional tool to address noncompliance. Further, IRS and the Financial Crimes Enforcement Network (FinCEN) have not clearly and publicly explained when, if at all, requirements for reporting financial assets held in foreign countries apply to virtual currencies. Clarifying and providing publicly available information about those requirements could improve the data available for tax enforcement and make it less likely that taxpayers will file reports that are not legally required. GAO is recommending that IRS clarify that part of the 2019 guidance is not authoritative and take steps to increase information reporting, and that FinCEN and IRS address how foreign asset reporting laws apply to virtual currency. IRS agreed with the recommendation on information reporting and disagreed with the other two, stating that a disclaimer statement is unnecessary and that it is premature to address virtual currency foreign reporting. GAO believes a disclaimer would increase transparency and that IRS can clarify foreign reporting without waiting for future developments in the industry. FinCEN agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "Under T-2017, DHA reduced the number of TRICARE regions by merging the North and South regions to form the East region, which has approximately 6 million beneficiaries, while the West region remained the same with approximately 3.4 million beneficiaries (see figure 1). In July 2016, DHA awarded the East region contract to Humana Government Business, the incumbent South region contractor, and the West region contract to Health Net Federal Services, the incumbent North region contractor. The T-2017 contracts include five 1-year performance periods and are scheduled to expire on December 31, 2022. As a result of the changes in regional structure, the T-2017 contract transition included a transition from Health Net Federal Services (North region) and Humana Government Business (South region) to Humana Government Business in the East region as well as a transition from UnitedHealth Military & Veterans to Health Net Federal Services in the West Region. The start of the T-2017 transition was initially planned for August 2016, with a health care delivery start date of August 2017. However, due to bid protests filed against each contract, the transition start date was pushed out to January 1, 2017 with a health care delivery start date of October 1, 2017. To manage the T-2017 transition, DHA assigned individuals to lead the transition in each region, who were responsible for coordinating all major transition activities. The transition leads were supported by other staff, including contracting officers, contracting officer representatives, and subject matter experts. In addition, DHA established an organizational structure comprised of several groups to oversee the T-2017 transition from day-to-day oversight to leadership updates. The TRICARE Operations Manual, which is part of the managed care support contract, establishes transition guidance that includes requirements for both the incoming and outgoing contractors. The T- 2017 transition guidance focused on the incoming contractors’ readiness to perform in seven critical areas: (1) provider network, (2) referral management, (3) enrollment, (4) medical management, (5) claims processing, (6) customer service, and (7) management. For the T-2017 transition, DHA introduced two new oversight methods to ensure contractors’ readiness in the seven critical areas prior to the start of health care delivery. These methods and other guidance are outlined in the TRICARE Operations Manual and T-2017 contracts. The performance readiness validation (PRV) and performance readiness assessment and verification (PRAV)—referred to as PRV/PRAV—tested contractors’ functionality in the seven critical areas outlined in the TRICARE Operations Manual. For the PRV, contractors validated their own readiness for specific requirements within each area. For example, the contractor had to validate that it had a complete provider directory online and operational 60 days prior to the start of health care delivery at a 95 percent accuracy rate. The number of requirements varied by critical area. For the PRAV, DHA subsequently assessed and verified contractors’ validation prior to the start of health care delivery. DHA also established financial penalties—referred to as transition performance guarantees—for five of the seven critical areas. The T- 2017 contracts specify that if a contractor does not meet a transition- in requirement in any one of these five areas, DHA will assess a financial penalty (see table 1). In December 2016—prior to the start of the transition—DHA held transition specification meetings with the incoming and outgoing contractors to begin planning critical T-2017 transition activities. The incoming contractors were also required to provide DHA with an integrated master plan and an integrated master schedule outlining processes and specific steps for the transition as well as a risk management plan that identified risks to the successful execution of the contractor’s schedule. Contractors were required to provide weekly updates to DHA on the status of their transition schedule progress. In April 2018—several months after the transition had ended—DHA produced an “after action” report to identify best practices, lessons learned, and recommendations to improve future TRICARE contract transitions. DHA is currently in the process of developing its fifth generation of contracts, referred to as the T-5 contracts. As required by the NDAA 2017, DHA established a new preferred provider benefit option called TRICARE Select and terminated the TRICARE Standard and Extra benefit options by January 1, 2018. Prior to 2018, beneficiaries primarily had a choice between three basic options— TRICARE Prime (a managed care option), TRICARE Standard (a fee-for- service option), or TRICARE Extra (a preferred provider organization option). The TRICARE Standard and Extra options did not require beneficiaries to enroll. However, beneficiaries who choose the TRICARE Select option must enroll during an annual open enrollment period or within 90 days of experiencing a qualifying life event. Beneficiary cost sharing responsibilities were also modified for the new benefit option. The implementation of TRICARE Select delayed timeframes for the T- 2017 transition and was the primary challenge of the T-2017 transition, according to DHA and contractor officials. Because the T-2017 contracts were awarded prior to the enactment of the NDAA 2017, DHA had to incorporate TRICARE Select requirements into the ongoing T-2017 transition process, including developing updated guidance for contractors. As a result of the time needed to plan for and implement a new benefit, DHA delayed timeframes for the following key transition activities. DHA postponed the start of health care delivery by 3 months. DHA moved the start of health care delivery from October 1, 2017 to January 1, 2018 (see fig. 2). According to DHA officials, DHA made this change to align the start of health care delivery with the implementation of TRICARE Select to minimize the impact that two, successive changes could have had on the continuity of care for beneficiaries. On March 30, 2017—three months into the transition— DHA sent a letter to the contractors informing them of this decision. DHA also directed its incoming contractors to submit modified transition schedules and risk management plans. DHA had to delay the start of a planned enrollment freeze and lengthen its duration. According to DHA officials, in a typical transition, DMDC requires 3 to 4 days to make adjustments to beneficiaries’ records in the Defense Enrollment Eligibility Reporting System, including assigning beneficiaries to incoming contractors and regions for the T-2017 contracts. During this time, which is referred to as an enrollment freeze, contractors cannot access this system to process any enrollments. For the T-2017 transition, DHA and DMDC officials stated that, given the termination of two benefit options and the new enrollment requirements for TRICARE Select, DMDC needed additional time to adjust every beneficiary enrollment record (over 9 million). Therefore, DHA delayed the start of the T-2017 enrollment freeze from August to December 2017 and increased its duration from 3 to 4 days to 19 days—December 1-19, 2017 (see fig. 2). Contractors had less time to process enrollments and make other system changes. Once an enrollment freeze has ended, incoming and outgoing contractors have a designated period of time, referred to as a dual operations period, to process beneficiaries’ enrollments and make other systems changes, such as assigning Prime beneficiaries to a primary care manager (PCM). Due to the extended enrollment freeze, contractors had a shorter dual operations period—less than 2 weeks in December 2017 rather than 6 to 8 weeks beginning in August 2017 (see fig. 2). According to contractors, the shorter dual operations period for T-2017 transition contributed to a backlog of enrollment requests and PCM assignments that they were unable to process prior to the start of health care delivery. To mitigate the financial effect on beneficiaries, DHA issued point of service waivers and waived referral requirements for TRICARE Prime enrollees for both regions and provided an enrollment grace period for beneficiaries so they did not have to pay higher copayments for receiving care from non-network providers or care that was not referred by a PCM. DHA’s communications to TRICARE beneficiaries were delayed. TRICARE Select complicated and delayed DHA’s communications to beneficiaries about TRICARE program changes, which led to customer service problems after the start of health care delivery. DHA engaged in various efforts to inform beneficiaries of the new changes, such as through website updates, blog posts, and direct mailings. However, DHA’s “after action” report acknowledged that on multiple occasions its communication division posted incorrect information on its website because of changing policy language. In addition, DHA planned to send a direct mailing to beneficiaries to inform them of TRICARE program changes in October 2017. However, DHA and DMDC officials told us that this date was delayed due to the additional time needed to prepare for TRICARE Select. As a result, DHA mailed information to beneficiaries starting in December 2017, and some beneficiaries did not receive this mailing until after the start of health care delivery, according to DHA. An organization representing TRICARE beneficiaries told us that some beneficiaries were unaware of the various benefit changes that went into effect on January 1, 2018 because of inadequate communication from DHA. Contractors also told us that the delayed communication to beneficiaries contributed to the high volume of customer service calls they received after the start of health care delivery. DHA officials told us that they took several steps to minimize the risks these delays and the implementation of TRICARE Select created, including the use of various transition oversight meetings to discuss and track related challenges. For example, the regional transition management staff participated in a monthly Risk Review Board meeting to discuss concerns related to the schedule of transition activities, such as the impact of TRICARE Select on the time needed for performance testing in critical areas. DHA also discussed transition risks related to TRICARE Select during weekly meetings with contractors throughout the transition. Furthermore, in August 2017, DHA hosted an Enrollment Summit for all stakeholders involved with the transition and implementation of TRICARE Select, where they discussed the schedule of transition steps and the coordination needed to implement the interrelated T-2017 and NDAA 2017 requirements. In addition, DHA kept contractors informed about TRICARE Select as they developed the related policies. Beginning in June 2017, DHA provided contractors with draft guidance on the new benefit to keep them informed of potential changes and obtain their feedback. According to DHA, this also allowed contractors to plan for and begin implementing the program changes they would be required to make once the policies were finalized. DHA issued the final TRICARE Select policies to its contractors in late October 2017, which left contractors with less than 3 months to implement the finalized changes prior to the start of health care delivery on January 1, 2018. According to DHA officials and contractors, contractors ideally would have had the final TRICARE policies at the start of the 9-to-12 month transition period. During the T-2017 transition, outgoing and incoming contractors had disagreements over data transfers. According to DHA officials and contractors, DHA’s transition guidance to contractors was not always specific or accurate regarding the amount and type of data to be shared, as well as how these data should be transferred. Furthermore, according to contractors, DHA did not always resolve contractors’ guidance-related disagreements in a timely manner. Contractors said this contributed to delays in implementing some transition steps and problems after the start of health care delivery. DHA faced challenges related to the following data transfer issues: Referral and authorization data. The contractors in the West region disagreed on how many years of historical referral and authorization data the outgoing contractor would provide the incoming contractor because this was not specified in the guidance, according to the contractors and DHA’s contracting officers. While the contractors in the East region mutually agreed on the years of data to transfer, the West region contractors did not. As a result, the incoming West region contractor reached out to DHA for resolution on August 2, 2017 by letter, and continued to discuss it with DHA officials during weekly meetings, as documented in meeting minutes we reviewed. However, DHA did not address the issue until December 12, 2017, at which point DHA rejected the incoming contractor’s request for additional historical data because the outgoing contractor would not have enough time to provide it by the start of health care delivery on January 1, 2018. The incoming contractor reported that not receiving the anticipated historical referral information contributed to several problems related to referrals after the start of health care delivery. First, it contributed to delays in processing referrals within timeliness standards. Second, the lack of data made it difficult for contractors to help MTFs address customer referral inquiries, which negatively affected the contractor’s relationship with MTFs. Finally, the contractor had limited ability to resolve beneficiaries’ customer service questions related to referrals and had to reissue authorizations for some referrals. Claims data. The incoming and outgoing West region contractors also disagreed on which elements of claims data needed to be transferred. For example, the incoming contractor requested information from the claims notes section, which the outgoing contractor stated contained some proprietary information. According to the incoming contractor, this section typically contains information important for claims processing, such as medical necessity reviews— medical record reviews to determine that health care services are appropriate for payment. When the outgoing contractor refused to provide the claims notes, the incoming contractor raised the issue several times to DHA during weekly meetings and through letters, as documented in meeting minutes and correspondence we reviewed. However, DHA determined that the outgoing contractor did not need to provide the information requested, as the non-proprietary information was available in other claims data sections. According to the incoming contractor, without access to more detailed historical information from the claims notes, there were instances in which they were unable to adjust payment determinations for certain claims paid prior to transition, which resulted in provider and beneficiary dissatisfaction. Beneficiary payment information. The incoming contractors faced challenges obtaining payment information for TRICARE beneficiaries who paid their health insurance premiums using credit cards or electronic funds transfers. According to a contracting officer, DHA initially directed the outgoing contractor to transfer beneficiary payment data to the incoming contractor. However, the outgoing contractors told us that they were unable to transfer this data due to banking laws and proprietary information security standards. DHA agreed that the outgoing contractors could not legally transfer this information and resolved the problem by requiring incoming contractors to reach out directly to beneficiaries to obtain the payment information. According to incoming contractor officials, this created additional, unanticipated effort, since they had to contact beneficiaries for this information directly, which diverted transition resources, such as enrollment staff, away from ongoing transition activities. In addition, contractors reported that this put certain TRICARE plan beneficiaries at risk since those who did not resubmit their payment information risked disenrollment and gaps in health care coverage. The contractors and DHA made attempts to notify affected beneficiaries that they needed to contact the contractor to reestablish their automated premium payments. However, approximately 224,000 beneficiaries’ credit card or electronic funds transfer enrollments for premium payments did not continue after January 1, 2018. To give beneficiaries more time to provide this information, DHA provided a 150-day grace period for premium payments. Still, certain beneficiaries were disenrolled from TRICARE plans for failure to establish a recurring form of payment. For example, more than 15,000 beneficiaries were disenrolled in the East region. In its “after action” report, DHA acknowledged that it did not always provide specific and accurate requirements for data transfers in its transition guidance and that this should be addressed for the next transition. However, the report did not address the difficulties related to resolving contractors’ questions and disagreements on these issues. For example, DHA officials told us that they followed an informal process for tracking and handling issues raised by contractors during the transition, which was explained in the initial transition specifications meeting in December 2016. However, the outgoing and incoming contractors in the West region expressed concerns about this process, explaining that it was difficult to resolve issues, particularly with the amount of time it took for DHA to provide a response, such as with the referral and authorization disagreement. Federal standards for internal control note that an agency should implement control activities through policies, such as by providing guidance with greater specificity for data transfers. These standards also indicate that agencies should remediate deficiencies in a timely manner, such as the prompt resolution of contractors’ guidance-related disputes so as to not disrupt the transition schedule. Without more specific guidance and a process that ensures timely dispute resolution, DHA risks disagreements and delays for future contract transitions, which could hinder health care delivery. DHA experienced challenges executing its new T-2017 transition oversight methods—PRV/PRAV and performance guarantees—as planned because of fundamental problems with how some requirements were written and the implementation of TRICARE Select. As a result, some of the requirements were not feasible or effective in assessing contractors’ readiness for health care delivery. 1. Certain PRV/PRAV requirements were not feasible as originally written or were not aligned with the corresponding performance guarantee, according to DHA officials. For example, one of the PRV requirements in the critical area of medical management focused on testing the contractors’ web-based systems for exchanging information electronically with the government and providers, but this was not always possible as some information continues to be transferred in hard copy, such as by fax. In addition, the performance guarantee related to provider network development did not align with the corresponding PRV/PRAV requirements. A DHA official told us that aligning the performance guarantee and PRV/PRAV requirements would have resulted in a higher financial penalty for one of the contractors. 2. Contractors noted that some PRV/PRAV requirements were not complete or effective measures of readiness. For example, contractors told us that requirements for claims and referrals did not effectively test the actual volume of administrative tasks that they would have to process after the start of health care delivery. According to the West region contractor, one of the referral PRAV tests required contractors to demonstrate that they could process 300 referrals during DHA’s onsite review, whereas they typically need to process 9,000 referrals a day after the start of health care delivery. 3. The original PRV/PRAV requirements did not account for TRICARE Select, since the contracts were awarded prior to the enactment of the NDAA 2017. Furthermore, due to the delayed and extended enrollment freeze that ended on December 19, 2017, DHA determined that contractors could not demonstrate a fully operational enrollment system sixty days prior to the start of health care delivery as originally required. Additionally, the contractors had limited access to DHA’s information technology systems for testing scenarios that included TRICARE Select. As a result, contractors had to test the majority of the critical areas (claims, enrollment, customer service, and referral management) with information technology systems that did not include TRICARE Select, which was not a true test of their readiness. To address issues with feasibility and TRICARE Select, DHA modified the PRV requirements for four of the seven critical areas during transition. Specifically, DHA modified all of the PRV requirements for enrollment, referral management, and claims processing as well as one PRV requirement for medical management. DHA also waived the corresponding performance guarantees for the three of these critical areas that had such guarantees (enrollment, referral management, and claims processing). As a result, the contractors were not subjected to financial penalties for not meeting the requirements for these critical areas. According to DHA officials, the problems with the PRV/PRAV requirements experienced during the T-2017 transition occurred in part because DHA subject matter experts did not review the requirements prior to the release of the final request for proposal. As a result, officials said that it was not until after the contracts were awarded that subject matter experts determined that some of the requirements could not be performed as written. Nonetheless, DHA officials and contractors agreed that the PRV/PRAV processes are good conceptual measures, and should continue to be used for the next transition with improvements to their feasibility and effectiveness. Having subject matter experts review contractors’ readiness requirements for feasibility and contract alignment could help ensure that these requirements are appropriate measures of contractor readiness. In addition, DHA’s “after action” report included feedback and lessons learned from officials and contractors on the PRV/PRAV requirements, which DHA could incorporate for future transitions. Federal standards for internal control state that an agency should internally communicate quality information to enable personnel to perform key roles in achieving objectives. By considering lessons learned from this transition and having subject matter experts review the requirements, DHA would be able to better ensure their metrics are appropriate to prepare contractors for health care delivery. DHA reported that the T-2017 contractors had overall better performance meeting contract requirements after the start of health care delivery than the two previous generations of TRICARE contracts. Nonetheless, DHA has acknowledged that both T-2017 contractors did experience some problems meeting certain contract requirements. DHA addressed most of these problems through the issuance of corrective action requests, which require the contractors to submit and implement a corrective action plan (see table 2). One exception where DHA did not issue formal corrective action requests was for problems both contractors experienced with processing enrollment backlogs after the start of health care delivery due to the extended TRICARE Select enrollment freeze during transition. Although most of the problems have been resolved, some problems have persisted into the second year of health care delivery, which DHA and contractors reported they are continuing to address. Provider directory accuracy. Both contractors have continued to fall short of the requirement for 95 percent accuracy of their online provider directories—problems they also experienced during the transition. As of June 2019, the West region contractor’s directory was 76 percent accurate and the East region’s was 64 percent accurate, according to DHA officials. Both contractors expressed concern about the methodology used to assess their performance against this requirement and stated that the 95 percent standard is too high. DHA officials acknowledged that the 95 percent standard is high and that the provider directory corrective action requests may remain open indefinitely because of the high standard, though they continue to monitor the corrective action requests. Claims processing timeliness and accuracy. The East region contractor has struggled to meet timeliness and accuracy standards for processing claims. The contract requires contractors to process 98 percent of claims within 30 calendar days of receipt and 100 percent of claims within 90 days with a 98 percent accuracy rate. As of June 2019, the contractor was meeting the 30 day timeliness requirement and was close to meeting the 90 day timeliness requirement (99.99 percent within 90 days). However, the contractor continued to miss the performance standard for claims processing accuracy, according to DHA officials. DHA officials told us that the department had completed multiple on-site reviews and continues to monitor this issue to ensure the contractor improves its ability to meet claims processing standards. Contractor officials acknowledged that they needed to improve their oversight of claims functions and improve training and job aids with their claims processing subcontractor, which was a new partner for their T-2017 contract. A smooth transition of health care delivery between outgoing and incoming managed care support contractors helps ensure continuity of care for TRICARE beneficiaries. In the most recent transition, the need to concurrently implement a new benefit option—TRICARE Select— presented some unique challenges that delayed the transition timeline and limited DHA’s ability to ensure contractors’ readiness in certain areas. While the implementation of a new benefit option during the T-2017 contract transition was a one-time occurrence, our review highlighted weaknesses in DHA’s transition guidance and oversight that could pose challenges to future contract transitions. By improving the specificity of its transition guidance, revising its process for resolving contractors’ issues, and ensuring review of PRV/PRAV requirements for feasibility and effectiveness, DOD could mitigate these challenges and thus improve future transitions. We are making the following three recommendations to DHA: The Director of DHA should define data sharing requirements with more specificity in its transition guidance for outgoing and incoming contractors, including the time period covered and the types of data that must be shared. (Recommendation 1) The Director of DHA should revise the process the agency has in place for resolving issues raised between contractors during transition to ensure such issues are resolved within time frames that will not adversely affect the transition schedule. (Recommendation 2) The Director of DHA should incorporate lessons learned from this transition and ensure that subject matter experts review PRV/PRAV requirements and performance guarantees prior to the issuance of the request for proposal for the next transition. These requirements should be reviewed to ensure their feasibility and effectiveness for assessing contractor readiness. (Recommendation 3) We provided a draft of this report to DOD for comment. In its written comments, reproduced in appendix I, DOD generally agreed with our findings and concurred with our recommendations. DOD outlined steps the department will take to improve the next TRICARE contract transition, including revising the TRICARE Operations Manual to better define data sharing requirements, developing a process to ensure that all contractor questions are answered appropriately and in a timely manner, and ensuring SMEs are involved in writing the PRV/PRAV requirements. DOD also provided technical comments, which we incorporated as appropriate We are sending copies of this report to the Secretary of Defense and appropriate congressional committees. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or cosgrovej@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix II. In addition to the contact named above, Bonnie Anderson, Assistant Director; Rebecca Abela, Analyst-in-Charge; Cathleen Hamann; Jacquelyn Hamilton; Rianna Jansen; and Vikki Porter made contributions to this report.", "summary": "DOD contracts with private sector companies—referred to as managed care support contractors—to deliver health care services to its TRICARE program beneficiaries through networks of civilian providers. In July 2016, DOD awarded its fourth generation of TRICARE contracts, referred to as T-2017, for management of civilian providers in its two regions (East and West). For new TRICARE contracts, DOD provides a transition period—usually 9 to 12 months—for the incoming and outgoing contractors. During this time, the incoming contractors must take specific steps to prepare for health care delivery. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to review the T-2017 transition. This report examines (1) how the requirement to implement TRICARE Select affected the transition, (2) challenges DOD experienced executing the T-2017 transition process, and (3) how DOD addressed problems after the start of health care delivery. GAO reviewed and analyzed DOD guidance, contract requirements, and other relevant documentation, and interviewed DOD officials, TRICARE contractors, and other stakeholders. The implementation of a required new health care benefit option delayed aspects of the transition to the Department of Defense's (DOD) fourth generation of TRICARE managed care support contracts (T-2017). The National Defense Authorization Act for Fiscal Year 2017 required DOD to implement TRICARE Select, a new preferred provider benefit option. As a result, DOD delayed the start of health care delivery—the date the incoming T-2017 contractors would assume responsibility for managing health care—from October 1, 2017, to January 1, 2018, to align with the mandated implementation date for TRICARE Select. DOD also delayed and lengthened a planned period for the department to make changes to beneficiary information in TRICARE's eligibility system. According to DOD and its contractors, this delay contributed to problems with enrollment processing backlogs that were not addressed until several months after health care delivery began. DOD experienced challenges during the T-2017 transition that resulted from weaknesses with its transition guidance and oversight. Specifically, DOD's guidance does not always specify the amount and types of data outgoing contractors have to share with incoming contractors. This led to contractor disagreements over data transfers, which DOD did not always resolve in a timely manner. Contractors reported that these issues contributed to problems after health care delivery began for the T-2017 contracts, such as with processing referrals. DHA also determined that some of DHA's oversight requirements, such as for specialty care referrals, were not feasible or effective, which limited some testing of contractors' readiness for health care delivery. This occurred in part because DOD's relevant subject matter experts did not review the requirements. DOD addressed most of the problems that occurred after health care delivery began by requiring the contractors to develop and implement corrective action plans. DOD and contractors are addressing some problems that have persisted, including problems with the contractors' provider directory accuracy in both regions and claims processing in one region. DOD has an opportunity to avoid similar problems in the future by improving the specificity of its transition guidance and effectiveness of its oversight requirements. GAO is making three recommendations to improve future contract transitions, including that DOD improve the specificity of its transition guidance and have subject matter experts review oversight requirements. DOD concurred with GAO's recommendations and identified steps the department is taking to address them.", "document_type": "gao"}
{"report": "The Aviation and Transportation Security Act designated TSA as the primary federal agency responsible for security in all modes of transportation, which includes physical security and cybersecurity. Passenger rail operators, however, have the day to day responsibility for carrying out safety and security measures for their systems. Unlike the aviation environment, where TSA has operational responsibility for screening passengers and baggage for prohibited items prior to boarding a commercial aircraft, the agency has a limited operational role for securing mass transit (including rail). To secure passenger rail, TSA primarily partners with public and private transportation operators to address their security needs by conducting vulnerability assessments and sharing intelligence information and key practices, among other measures. The agency also engages with the passenger rail industry through associations, such as APTA and Association of American Railroads. Additionally, TSA is responsible for assessing the risk from terrorism and cyber threats to passenger rail, as well as other transportation modes. In addition to engaging with domestic passenger rail stakeholders, TSA’s Office of Policy, Plans, and Engagement is responsible for coordinating domestic and international multimodal transportation security polices, programs, directives, strategies, and initiatives, including conducting outreach to foreign stakeholders. TSA also engages with foreign stakeholders through TSARs. TSARs are primarily located in posts overseas and communicate with foreign government officials to address transportation security matters involving all modes of transportation, including aviation, rail, mass transit, highways, and pipelines. The TSAR role was originally created in response to the 1988 bombing of Pan Am Flight 103 over Lockerbie, Scotland, when the Aviation Security Improvement Act of 1990 was enacted, which provided that foreign security liaison officers were to serve as liaisons to foreign governments in carrying out U.S. government security requirements at specific airports. TSARs are responsible for ensuring the implementation of TSA’s requirements primarily as they relate to passenger and cargo air transportation departing the specific country en route to the United States. The primary focus of the role remains on aviation; however it has evolved over time to include maritime and land transportation. According to TSA, recent attacks overseas and online terrorist messaging point to public transportation systems, which include passenger rail systems, as continued high-value targets for terrorists. In general, passenger rail systems are open and designed to expedite the free flowing movement of large numbers of passengers through multiple stations. As such, these systems are inherently vulnerable to physical attacks (such as improvised explosive devices, active shooters, and chemical or biological attacks) due in part to factors such as high ridership, open access points, limited exit lanes, and fixed, publically available schedules. In addition, TSA has reported that risks increase in urban areas where multiple transportation systems and high volumes of travelers merge at intermodal stations. Transportation systems, including passenger rail systems, rely on technology and internet-connected devices to manage and secure certain business/enterprise functions, such as the operation’s website, communications, and reservations and ticketing mechanisms. They also increasingly rely on computer-networked systems for tracking, signals, and operational controls of transportation equipment and services. As dependence on these systems increases, so does risk to the system. Cyberattacks have the potential to significantly affect both business/enterprise systems and operational control systems. Business/Enterprise systems. Cybersecurity threats include ransomware, malware, phishing, and website attacks that may compromise sensitive information and affect an operator’s ability to communicate with passengers or engage in day-to-day business functions. Operational control systems. Cybersecurity threats, which may include malware or physical manipulation of a system, such as jamming signals or damaging equipment, include threats to the systems that control signaling and train speed. For example, attackers could attempt to access positive train control systems, a computer-based system designed to automatically slow or stop a train that is not being operated safely, to disrupt services. Unintentional cybersecurity threat sources may include failures in equipment or software due to aging or user errors, such as unintentionally inserting a flash drive infected with malware or clicking on a phishing email. Intentional cybersecurity threats may include corrupt employees, criminal groups, terrorists, and nations and may be used, for example, to achieve monetary gain, or for political or military purposes. Figure 1 shows examples of the types of physical and cyber threats passenger rail systems face. The NIPP outlines a risk management framework for critical infrastructure protection. In accordance with the Homeland Security Act of 2002, as amended, DHS created the NIPP in 2006 to guide the national effort to manage security risk to the nation’s critical infrastructure, including through coordination of agencies and 16 various critical infrastructure sectors, including transportation systems. Most recently updated in 2013, the NIPP uses a risk management framework as a planning methodology intended to inform how decision makers take actions to manage risk. The framework calls for public and private partners to conduct risk assessments. The NIPP defines risk as a function of three elements: threat, vulnerability, and consequence, as shown in Figure 2. Threat is an indication of the likelihood that a specific type of attack will be initiated against a specific target or class of targets. Vulnerability is the probability that a particular attempted attack will succeed against a particular target or class of targets. Consequence is the effect of a successful attack. In 2010, DHS, through TSA and the U.S. Coast Guard, developed the Transportation Systems Sector-Specific Plan to conform to requirements in the NIPP. Most recently updated in 2015, this plan describes shared goals, priorities, and activities to mitigate critical infrastructure risks, and acknowledges the increasing dependence of transportation companies on cyber systems for business, security, and operational functions. Regarding cybersecurity risks, DHS produced the Cybersecurity Strategy in 2018 to help execute its cybersecurity responsibilities during the next 5 years. In order to meet one of its objectives, DHS is to encourage the adoption of applicable cybersecurity best practices, including the NIST Framework for Improving Critical Infrastructure Cybersecurity (referred to as the NIST Cybersecurity Framework). The framework is a set of voluntary industry standards and best practices to help organizations manage security risks specific to cybersecurity. The framework consists of five functions: Identify, Protect, Detect, Respond, and Recover. When considered together, these functions provide a high-level view of an organization’s management of cybersecurity risk. NIST issued the framework in 2014 and updated it in April 2018. CISA, formerly DHS’s National Protection and Programs Directorate, manages the national effort to secure and protect against critical infrastructure risks, including cybersecurity risk, for all 16 critical infrastructure sectors, including transportation. CISA’s responsibilities include coordinating with sector-specific agencies to carry out its cybersecurity and critical infrastructure activities. According to TSA officials, TSA uses the TSSRA, the BASE, and threat assessments to assess risk elements for physical and cyber security in passenger rail. Such assessments may address different elements of risk—threat, vulnerability, or consequence—or the total risk for specific assets, such as airport perimeters and pipeline critical facilities. Table 1 below shows the type of risk element each assessment addresses, and whether the assessment addresses risks to intermodal stations or cybersecurity risk. TSSRA. TSA uses the TSSRA, a periodic risk assessment, to assess threat, vulnerability, and consequence for various attack scenarios across the five transportation modes for which TSA is responsible. The scenarios define a type of threat actor—including homegrown violent extremists and transnational extremists, such as Al Qaeda and its affiliates—a target, and an attack mode. For example, a scenario might assess the risk of attacks using varying types of weapons on passenger rail system assets. As part of the assessment process, TSA engages with subject matter experts from TSA and industry stakeholder representatives to compile vulnerabilities for each mode, and TSA analyzes both direct and indirect consequences of the various attack scenarios. According to TSA, the agency uses the TSSRA to provide strategic insights to inform the administration’s risk mitigation strategies, policy considerations, security countermeasures and programs, and resource allocation decisions. Our analysis of the TSSRAs issued during calendar years 2015 through 2017 indicates that TSA included intermodal station attack scenarios, but did not include cybersecurity scenarios. Specifically, the assessments featured various scenarios that targeted intermodal stations, which could include rail systems. For example, a scenario might describe attacks using various numbers of improvised explosive devices on an intermodal station. TSA did not include cybersecurity attack scenarios in the calendar year 2015, 2016, or 2017 assessments. According to the 2016 assessment and TSA officials we interviewed, threat experts have indicated that cyber threats, due to their unique nature and other factors, do not lend themselves to traditional TSSRA attack scenarios. However, as discussed below, the agency does conduct cyber threat assessments. Further, TSA’s Cybersecurity Roadmap 2018, states that, as one objective, the agency will include cybersecurity in its risk assessments for all modes. According to TSA officials, the implementation plan for the Roadmap, which was approved in September 2019, provides guidance and direction for meeting this objective. TSA officials confirmed that they plan to include basic cybersecurity scenarios for all modes in the 2020 TSSRA, and that they plan to engage with TSA mass transit experts and consult with industry experts as needed to inform future cyberattack scenarios. BASE. The BASE is a voluntary security assessment of national mass transit, passenger rail, and highway systems conducted by TSA surface transportation inspectors that addresses potential vulnerabilities, among other things. It consists of an assessment template with 17 security action items developed by TSA and the Federal Transit Administration that address, among other best practices, security training programs, risk information sharing, and cybersecurity. TSA developed this assessment in 2006 to increase domain awareness, enhance prevention and protection capabilities, and further response preparedness of passenger transit systems nationwide. The agency uses the BASE assessments to track progress in implementing specific security measures over time, offer technical assistance and share best practices to help improve the overall security posture of agencies, and inform transportation security grant funding by, among other things, identifying actions agencies have taken to reduce vulnerability. TSA officials stated that the most recent formal update to the assessment template began in 2014 and was fully implemented in 2015. The update included, among other changes, revised guidance for TSA surface inspectors and the addition of questions concerning active shooter events. In fiscal year 2016, the agency also developed a more targeted BASE assessment that focuses the assessment on an entity’s areas of concern as identified by surface inspectors in a previous BASE review of that operator. As of 2017, TSA had completed initial and follow-up assessments for the top 100 mass transit agencies in the country, which comprise approximately 80 percent of the ridership in the United States. TSA officials told us that their goal is to conduct follow-up assessments every one to three years. As previously shown in table 1, our analysis of the BASE template for mass transit and passenger rail indicates that it includes questions that address selected rail agency concerns about intermodal station security, and questions related to cybersecurity issues. Specifically, we found that while the template does not contain security action items or questions that directly refer to intermodal stations, questions in the template do correspond to topics that domestic rail agencies we interviewed identified as significant to intermodal station security, such as coordination among security forces, visible security measures, and establishing roles and responsibilities. For example, one BASE question asks if the agency’s system security plan has procedures or protocols for responding to security events with external agencies such as law enforcement or fire departments. This question corresponds to the challenge of coordination among security forces in intermodal stations identified by six of the seven agencies we interviewed. Cybersecurity is the focus of one of the security action items, which includes a series of general questions related to whether the transit agency has developed a comprehensive cybersecurity strategy. According to TSA officials, the agency added cybersecurity questions to the BASE in 2013 and the questions are intended to be a high level review. For example, the BASE addresses whether the transit agency has conducted a cybersecurity risk assessment, ensured employee training covers cybersecurity roles and threats, and established a protocol for reporting cyber incidents. It also provides a list of available cybersecurity resources for agencies to consult. Threat Assessments. TSA’s Intelligence and Analysis Office identifies security threats to mass transit and passenger rail systems through various threat assessments, including annual and semiannual Mass Transit and Passenger Rail Terrorism Threat Assessments and annual Cyber Modal Threat Assessments. TSA’s Mass Transit and Passenger Rail Terrorism Threat Assessment is produced annually and establishes the current mass transit passenger rail threat level and reviews terrorist threats against mass transit passenger rail for the past year. Threat information includes terrorist attacks on passenger rail trains, train tracks, buses, bus stops, and various stations. Additionally, the threat assessment analyzes intelligence gaps for the mass transit mode. TSA supplements the annual assessment with a semiannual threat assessment that reviews terrorist threats against mass transit and passenger rail for a 6-month period. Our analysis of threat assessments TSA issued for calendar years 2015 through 2019 indicates that they addressed stations, in general, and intermodal stations specifically, when they are the subject of an attack. For example, an attack on Manchester, England’s Victoria station, an intermodal station, was included in the 2018 Mass Transit and Passenger Rail Terrorism Threat Assessment. TSA’s Cyber Modal Threat Assessment reviews cyber threats to transportation over the course of the previous year, establishes cyber threat levels for the transportation modes for which TSA is responsible, and evaluates the threat through the next year or two. This annual assessment examines cyber threats to business and industrial control systems from state and non-state actors, including terrorist groups, pro-terrorist hacker groups, and hacktivists. Moreover, it analyzes incidents of cyberattacks and cyber espionage against U.S. and foreign transportation. Both assessments analyze threat actors and their capabilities, intent, and activities—including attacks occurring internationally—as well as tactics, techniques, and procedures that could be employed in future attacks. TSA calculates threat levels for transportation and cyber modes based on assessments of threat actor intent and capability. It may also issue additional situation-based products on emerging threats. TSA routinely shares these threat assessments with rail agencies and other stakeholders, such as industry security professionals. In addition to TSA’s risk assessment efforts, the agency coordinates with CISA, which conducts voluntary cybersecurity assessments as needed and requested by TSA and industry stakeholders. Specifically, CISA offers eight different voluntary cyber assessment options for public and private sector stakeholders, including mass transit and passenger rail agencies. Because CISA provides services to all 16 critical infrastructure sectors, including the transportation systems sector, officials noted that it must balance the resources it devotes to each sector. For example, CISA officials stated that they have conducted six Validated Architecture Design Review assessments on rail agencies since 2015, and currently have four pending requests from transportation agencies. The Validated Architecture Design Review evaluates systems, networks, and security services to determine if they are designed, built, and operated in a reliable and resilient manner. CISA officials also stated that they have conducted weekly vulnerability scans for one rail agency since 2015. While CISA coordinates with federal and private sector stakeholders to identify and address significant risks to critical infrastructure through its assessments, agency officials stated that they defer to TSA (as the co- sector specific agency for transportation) to take the lead in broader cyber initiatives and outreach to the transportation sector. For example, TSA officials stated that the agency included CISA in planning its cybersecurity workshops, a series of half-day workshops for surface transportation agencies to learn about cybersecurity resources from DHS and discuss nontechnical cybersecurity actions to improve their cybersecurity posture. According to TSA’s Cybersecurity Roadmap 2018, the agency plans to assess the resilience of the transportation modes to malicious cyber activity in conjunction with CISA, among other things. According to officials, TSA and CISA are collaborating or planning to collaborate on several cybersecurity assessments for passenger rail systems, including a cyber risk assessment for passenger rail cars and a cyber assessment of the mass transit and passenger rail mode. CISA officials told us that TSA, DHS’s Science and Technology Directorate, and CISA’s National Risk Management Center are in early phases of developing a cyber risk assessment for select passenger rail cars that they plan to produce in fiscal year 2020. CISA officials stated that they intend to address cyber vulnerability in the rail car assessments and plan to reach out to operators to discuss results. TSA officials told us that TSA and CISA also are considering a mass transit and passenger rail cyber assessment similar to one being developed for the pipeline mode. CISA officials stated that the planned pipeline assessment effort will include a total of 10 Validated Architecture Design Review assessments, in which TSA will help make arrangements with industry and will observe the process. TSA officials explained that expanding this effort to include passenger rail would depend on CISA’s availability to conduct assessments and balance demands in other sectors. CISA officials noted that they currently do not have the resources to support a similar plan for rail. TSA participates in APTA working groups that review and develop standards and recommended practices for passenger rail security, including those that apply to intermodal station security and cybersecurity. Specifically, from 2009 through 2019, APTA produced 45 documents related to security and emergency management standards and recommended practices, among other things. TSA is listed as a participant in 37 of the 45 documents. TSA officials noted that APTA working groups regularly review documents and issues related to security topics, including through monthly phone calls in some cases, and update them as needed. According to APTA’s Manual for the Standards Development Program, standards address safety-critical subjects and establish requirements that must be met by industry; recommended practices describe an established or generally recommended approach that does not rise to the level of a standard; and white papers are intended to provide information about complex issues that present the industry’s prevailing philosophy on the subject matter. For example: APTA offers a series of general standards, recommended practices, and white papers targeted at physical infrastructure protection at passenger facilities. These documents are not specifically directed at intermodal stations, but, according to our analysis and APTA officials, apply to such facilities as well as others. The documents address factors such as exterior door and window security, as well as securing mailrooms and utility openings, among other issues. Another APTA standard addresses security and emergency management considerations during planned special events, such as identifying transit hubs that are likely to be inundated with passengers going to and from the event. APTA offers cybersecurity recommended practices that are targeted at transit agencies in the early stages of starting a cybersecurity program, including how to obtain executive-level awareness and support and how to develop a cybersecurity awareness and training program. APTA also offers recommended practices for securing control and communications systems in transit environments, such as train control systems and fare collection systems. Table 2 provides additional examples of industry standards and key practice documents, as they relate to threats identified by domestic passenger rail stakeholders we interviewed. In addition to working with industry through APTA to develop standards and practices, TSA officials stated that the Surface Transportation Security Advisory Committee, which was established in 2019 to provide advice and recommendations to the TSA Administrator on transportation security matters, may serve as a mechanism for discussing or recommending key practices as the Committee develops. Officials noted that the Committee, which includes industry and community groups, could serve as a source for identifying forward looking best practices for rail security. The Committee held initial meetings in July 2019, October 2019, and January 2020, and proposed establishing subcommittees on topics such as cybersecurity and insider threats. None of the seven domestic rail agencies we contacted identified any security areas in which they felt recommended practices were missing. Officials from five agencies specifically commented on the usefulness of APTA publications. Officials from three agencies however, noted that many transit and rail agencies are still in the early stages of starting a cybersecurity program and that cybersecurity recommended practices are generally targeted at those agencies, as compared to agencies that already have a more sophisticated approach to cybersecurity. Officials from one agency further noted that publications related to the more technical aspects of cybersecurity (such as industrial control systems) can become outdated quickly as industry outpaces the development of security standards. TSA, CISA, and passenger rail agency officials we interviewed identified the NIST Cybersecurity Framework as the primary key practices document they reference for cybersecurity. Domestic and foreign rail agency, and industry association officials, as well as academic experts we interviewed noted that the possibility or likelihood of a cyberattack causing physical damage or harm to rail passengers or infrastructure is unlikely and largely hypothetical at this time. Academic experts we interviewed pointed to an incident in Poland in 2008 as one of the few, if only, known incidents in which a cyber-related attack on rail resulted in physical harm. In this incident, according to news reports, a Polish teenager modified a television remote control so that it could be used to control signals and switch points in a local tram system. Four vehicles derailed and 12 people were injured in the incident. Several rail agency officials and stakeholders we spoke with noted that successfully hacking into train control systems would require a highly sophisticated knowledge of the system. Officials further noted that train systems are designed to fail to safe mode and stop a train in the event of an abnormal signal, and that train operators have the ability to take over controls and manually stop trains if necessary. Officials from three rail agencies, however, stated that as agencies continue to adopt new technologies and systems become more interconnected, the potential for a cyberattack increases. Additionally, CISA officials and officials from one rail agency stated that, despite the lack of many incidents to date, protecting control systems is critical given the potential catastrophic impact of a successful attack. According to TSA officials, TSA identifies foreign passenger rail security standards and key practices through engagement in multilateral groups and by leveraging bilateral relationships. Examples of multilateral groups include the International Working Group on Land Transport Security and the European Association of Railway Police Forces (RAILPOL). The working group, established in 2006, consists of 19 member states, including the United States. It is intended as a framework for members to openly share best practices, exchange information, and contribute to the development of surface transportation security initiatives. For example, TSA and members of the working group developed a searchable database of international surface transportation security measures (known as the SMARTbox) as a resource for surface transportation professionals to gain insights into security practices used by their peers. RAILPOL, founded in 2004, is an international association of government railway police organizations. It has 22 members, including TSA and the Amtrak Police Department. Information about intermodal stations and cybersecurity can be identified and exchanged through both of these mechanisms. For example, representatives from the United Kingdom delivered a presentation on securing intermodal stations at a 2016 working group meeting, and both working group and RAILPOL meetings have included cybersecurity discussions. Figure 3 provides an image of St. Pancras International Station in London, an intermodal station where international, local, and long distance trains converge with the London Underground. Regarding bilateral engagement, TSA identifies foreign rail security standards and practices through one-on-one relationships with other countries. TSA officials noted that their level of engagement with other countries can depend on a variety of factors, including how much the countries have in common regarding transportation systems and threats, and whether or not there are formal agreements in place that allow for regular, detailed information sharing. While some relationships are ongoing, officials stated that TSA interactions with other countries are often situational or transactional—countries may reach out either directly to TSA or through the TSAR for information about a specific issue, such as perimeter protection for surface transportation. For example: TSA holds biannual meetings with Transport Canada, the Canadian government department responsible for transportation policies and programs. Discussion topics from the meetings in 2017 and 2018 included Canadian efforts to develop passenger rail regulations, results from TSA derailment device testing, and opportunities for collaboration. According to TSA officials, TSARs in several countries have facilitated engagement with foreign surface transportation officials, including passenger rail officials. For example, officials stated that one TSAR facilitated the use of TSA’s Exercise Information System for an exercise on the metro system in a foreign city, as well as joint rail security training at TSA facilities in the United States. Officials further noted that another TSAR has taken initiative to facilitate quarterly meetings between foreign government and TSA surface transportation officials, including research and development and passenger rail officials. In addition to quarterly meetings facilitated by the TSAR, TSA officials stated that they are in regular contact with research and development officials in one country to share testing information, such as the results of derailment device testing and explosives testing on railcars, and to discuss security issues related to unmanned aircraft systems. TSA officials also reported that representatives attended an APTA- sponsored study trip to Brussels and London after the 2016 and 2017 rail attacks in those cities, in part, to observe lessons learned from the attacks. Foreign governments and international rail associations also produce a variety of passenger rail security standards and key practice documents. Table 3 below provides examples of these documents and the types of threats they address. TSA officials noted that while multilateral forums provide valuable opportunities to communicate with other countries about evolving threats, emerging security technologies, and potential key practices, interest in forums such as the International Working Group on Land Transport Security has been in decline. For example, while the working group charter calls for annual meetings and quarterly conference calls, the full group has not met since 2016. TSA and foreign government officials we spoke with stated that interest in the working group may be in decline due to factors such as retirements of key officials and lack of engagement from certain countries. These officials also noted that, as leaders in rail security, they typically provide more information about key practices to other countries in large forums than they receive. Additionally, TSA officials noted that other countries frequently used the working group- developed SMARTbox initially, but that use declined in recent years in part due to its location on the Homeland Security Information Network because users may find it difficult to navigate. Further, eight of the 10 domestic and foreign rail agencies we interviewed said they were either unfamiliar with the application or did not use it. For example, officials from one domestic agency said that there was little incentive to contribute and that they found informal networks to be more useful for sharing information. In contrast, TSA and other officials we spoke to stated that bilateral relationships with trusted partners with similar sophisticated rail operations may allow for more detailed exchanges of current and emerging key security practices. TSA has provided limited guidance to TSARs on engagement with foreign passenger rail stakeholders through the TSAR Toolkit (or handbook), which states that TSARs should engage with officials involved in multiple modes of transit, including rail; however, the primary focus of the document is engagement with aviation stakeholders. TSA further provides comprehensive and specific guidance for TSAR aviation engagement as part of its foreign airport assessments and air carrier inspections, but does not do so for surface transportation. As discussed above, according to TSA officials, some TSARs have taken the initiative to facilitate meetings and share testing and training information related to surface transportation, including passenger rail. Passenger rail officials we talked to in one country stated that these TSAR-led initiatives served as a valuable source of information and communication with TSA. In addition, one TSA official cited the value of discussing preliminary testing findings, as well as new guidelines on topics such as security in station designs, which address concerns about security in public spaces. These efforts, however, are dependent on the individual initiative of each TSAR and are not universal. For example, one TSAR we interviewed stated that TSA’s expectations and priorities for surface transportation engagement were unclear and, as a result, he focused almost exclusively on aviation. TSA officials stated that they have focused TSAR guidance on aviation engagement because of the agency’s regulatory role in this area, which, as discussed above, includes foreign airport assessments and air carrier inspections. In lieu of detailed guidance on surface transportation, officials noted they defer to the individual TSARs on how or whether to engage foreign surface transportation stakeholders. Officials emphasized this individual approach and stated that in some countries, TSAR engagement on passenger rail security issues may be limited by legal or cultural barriers. Because rail (unlike aviation) does not directly connect to the United States in most cases, officials noted that there may be less incentive for some host countries to engage. Further, some countries may not have a rail system, or may not be as advanced in rail security policies and procedures, and therefore may be less able to offer key practices. In November 2019, TSA officials noted that they were considering adding guidance for engaging with surface transportation officials and addressing intermodal concerns to TSAR Regional Operational Implementation Plans. According to officials, these plans provide targeted guidance to TSARs for engagement within their specific regions. As of February 2020, officials stated that draft plans for two regions (Western Hemisphere and Africa/Middle East) were under review at TSA. Officials further stated that these drafts, and drafts for the remaining regions currently in development, would include surface transportation-related guidance. TSA officials stated that they hoped to complete all regions’ plans by the end of calendar year 2020, but they did not provide documentation for us to verify that the final plans would contain surface transportation guidance for TSARs. The 2018 TSA Administrator’s Intent document includes a goal to promote security partnerships across surface transportation systems by, in part, identifying and communicating best practices and lessons learned to stakeholders and international partners. In addition, the NIPP states that officials should share actionable and relevant information across the critical infrastructure community to build awareness and enable risk informed decision making. The TSAR Toolkit further states that, even in locations without modal connections to the United States, there is still great value in establishing key points of contact who can share best practices or facilitate the exchange of information in the event of an emergency in modes of transit outside of aviation. As the primary overseas point of contact for security matters involving all modes of transportation, TSARs are responsible for developing bilateral relationships and facilitating information sharing with foreign stakeholders, among other things. Further leveraging formal or informal bilateral relationships could allow TSA to obtain additional passenger rail security information. While several TSARs have individually taken initiative with regard to rail, without additional guidance from TSA, there is no assurance that they will engage in these exchanges with modes outside of aviation. As a result, TSA is less likely to be fully aware of key passenger rail security practices in other countries, such as those listed in table 3 above, among others. Moreover, specific guidance will also provide TSARs with clear expectations for engaging with stakeholders, and provide TSA with greater assurance that they are engaging in a consistent manner. TSA’s new Regional Operational Implementation Plans provide an opportunity for TSA to more clearly incorporate targeted guidance to encourage TSAR outreach and information sharing in specific areas. Recent efforts by TSARs in several countries demonstrate practices, such as opening lines of regular communication on surface transportation, including passenger rail, which could be replicated in other countries. According to TSA officials and domestic rail stakeholders we interviewed, TSA uses various mechanisms such as the Transit Policing and Security Peer Advisory Group, monthly conference calls with rail stakeholders, and the annual APTA roundtable meeting to share and discuss a range of security information with stakeholders, including information about standards and key practices. These mechanisms provide opportunities to discuss issues related to intermodal stations and cybersecurity key practices. TSA also shares information about key practices with domestic stakeholders through voluntary TSA programs such as BASE, the Intermodal Security Training and Exercise Program, and the Visible Intermodal Prevention and Response program. TSA officials provided information about how they incorporate information from foreign threats and attacks into these programs. Specifically: TSA officials noted that TSA initially developed the BASE program around standards that were produced by APTA and other industry partners following the 2004 terrorist attacks on commuter trains in Madrid and the 2005 terrorist attacks on the London subway system. According to TSA officials, the APTA standards and recommended practices, which evolve based on threats and lessons learned, form the basis for the BASE assessment template. One way in which TSA helps communicates these standards and practices to agencies is through the questions in the template. Officials noted that lessons learned from foreign rail security incidents have been used to further support certain security concepts in the BASE, such as assessment questions related to whether agencies engage in public outreach for security awareness (e.g. “If You See Something, Say Something”) and report suspicious activity. TSA officials reported that they consider overseas and domestic attack methods and tactics when planning Intermodal Security Training and Exercise Program exercises to raise awareness of emerging tactics and threats. These exercises are intended to share best practices and lessons learned, among other things. Officials noted that they recently incorporated cyber, chemical, and vehicle- ramming attacks into the program’s objectives based on recent domestic and overseas incidents, and that they shared resources, information, and best practices for security solutions. For example, officials reported conducting two regional exercises that focused on chemical threat elements as the result of a 2017 plot in Australia. TSA further reported hosting a series of vehicle ramming program workshops in the wake of attacks in New York City and Europe. According to TSA officials, TSA has not made any recent changes to the Visible Intermodal Prevention and Response program directly as a result of lessons learned or key practices resulting from a foreign rail security incident; however, officials said they regularly integrate information about foreign incidents and threats when planning program deployments. Officials also noted that the majority of current deployments are for surface transportation, which includes rail. Regarding cybersecurity, TSA has shared information about cybersecurity key practices, including the NIST Cybersecurity Framework, through a series of regional cybersecurity Intermodal Security Training and Exercise Program workshops since 2017. These “5N5” workshops listed five nontechnical cybersecurity actions an agency could take in 5 days, including: (1) develop familiarity with the NIST Cybersecurity Framework; (2) implement a unique password change policy; (3) understand the latest phishing and spam trends and how to message awareness; (4) differentiate access control among staff; and (5) report cybersecurity incidents. Six of the seven domestic rail agencies we spoke with were generally satisfied with TSA’s efforts to share security and key practice information; however officials from two of these six agencies also expressed concerns about timeliness and quality of cybersecurity information provided by TSA. For example, officials from one agency stated that they received limited cybersecurity information from TSA and that the information they did receive was of limited use because it was targeted at agencies without a sophisticated cybersecurity program. An official from another agency noted that while there were opportunities to discuss cybersecurity, the information provided was often general in nature and there was limited time for discussion in certain mechanisms because of the large number of people involved. This official also noted that while the information TSA provides is valuable and there are mechanisms available to share information about a range topics, discussions are typically related to security incidents and threats, as opposed to key practices. TSA officials acknowledged that the agency’s cybersecurity efforts were still in the early stages. They further noted that the implementation plan for the 2018 Cybersecurity Roadmap, which, among things, calls for improving information sharing and partnering with stakeholders to promote the adoption of best practices and industry and/or international standards, was only recently signed in September 2019. In addition to TSA’s information sharing mechanisms, domestic rail agency officials we spoke to reported learning about foreign key practices through personal experience and direct engagement with foreign rail counterparts. For example, officials from two agencies we spoke to hosted visits from foreign rail officials to study security measures, among other things. Officials from one agency noted they provided information to Hong Kong through APTA on key practices for managing large protest crowds in an urban transit environment. Officials from another agency noted that they participate in international information sharing surveys and research to learn about cybersecurity practices by foreign rail operators, and sent representatives to an international mass transit training forum on the development of threat, vulnerability, and risk assessments. Domestic rail agencies also identified several changes they have made to their physical security systems as a result of key practices or lessons learned from foreign rail incidents. For example: increasing random patrols and high visibility deployments of security officers, changing security camera placement to better capture station exits, and increasing security awareness messaging to employees and passengers. Additionally, officials from one agency noted that they revised subway evacuation plans to direct people towards areas less vulnerable to an attack after reviewing lessons learned from recent vehicle-based attacks in Europe. With regard to cybersecurity, one domestic agency we spoke to noted that recent wide-spread global cyberattacks reinforced challenges they have securing legacy Information Technology systems against threats such as ransomware threats. As a result, the agency is focused on identifying expiring technologies and replacing those that can no longer be patched or updated. Officials from another agency noted that they have increased the number of firewalls they use to further segment and protect systems. Table 4 below provides information on mechanisms that can be used to identify and share rail security key practice information, as identified by TSA and domestic stakeholders. While TSA has taken initial steps to share cybersecurity key practices and other information with passenger rail stakeholders, the BASE assessment, does not fully reflect the updated cyber key practices presented in the NIST Cybersecurity Framework, nor does it include the framework in a list of available cyber resources. As discussed above, TSA uses the BASE assessment to share security best practices with transit agencies, among other things. Our review of the BASE cybersecurity questions in the template found that they cover selected activities associated with three of the five functions outlined in the framework– Identify, Protect, and Respond. For example, the BASE asks agencies if they ensure training reinforces cybersecurity roles and responsibilities, which corresponds to the awareness and training category of the NIST Protect function. However, the remaining two functions—Detect and Recover—are not represented in the BASE. According to the framework, when considered together, these functions provide a high-level, strategic view of the life cycle of an organization’s management of cybersecurity risk. TSA officials stated that they regularly review the BASE and noted that the questions are intended to reflect both industry key practices and agency policy; however, they also stated that the agency has not updated the BASE cybersecurity questions since NIST released its Cybersecurity Framework in 2014. In January 2020, officials responsible for the BASE acknowledged that the cybersecurity questions should be updated to reflect the framework. TSA officials also noted that they would want to align changes to the BASE cybersecurity questions with any new guidance or direction provided by the newly established Surface Transportation Security Advisory Committee. As of January 2020, the Committee is in its initial start-up phase, and has not yet provided any reports or recommendations or published a timeline or project plan. Further, because the framework functions organize basic cybersecurity activities at their highest level, incorporating elements of all five functions into the BASE template should not require additional guidance from the Committee. The 2015 TSA Transportation Systems Sector-Specific Plan states that encouraging the adoption of the NIST Cybersecurity Framework across all transportation modes supports the plan’s goal to manage the security risks to the physical, human, and cyber elements of critical transportation infrastructure. The plan also states that encouraging the adoption of the framework contributes to several of the NIPP’s calls to action related to sharing actionable and relevant information. TSA considers the framework a best practice document. By updating the BASE cybersecurity questions to align more closely with the core functions in the NIST Cybersecurity Framework, TSA could better assist passenger rail and other operators in identifying current key practices and improving their cybersecurity posture. As a result, transit operators would be more aware of cybersecurity vulnerabilities and better prepared to reduce the impact from a cybersecurity incident. In addition, this would create a more consistent cybersecurity approach from TSA, since the agency promotes the framework through other mechanisms, such as the series of cybersecurity workshops, as noted above. Recent physical and cyberattacks in U.S. cities and Europe demonstrate the evolving nature of the threats to passenger rail and highlight the importance of working with both domestic stakeholders and foreign rail security partners. As such, TSA actively engages with domestic passenger rail stakeholders, but could do more to engage with foreign stakeholders. TSARs stationed abroad are well positioned to further leverage bilateral relationships with foreign passenger rail stakeholders, and several TSARs have taken initiative to do so. However, TSA provides only limited guidance to TSARs on surface transportation engagement. Without specific guidance, there is no assurance that TSARs will engage in these exchanges with modes outside of aviation. TSA’s new Regional Operational Implementation Plans provide an opportunity to more clearly incorporate targeted guidance to encourage TSAR outreach and information sharing in specific areas. Additionally, such guidance will provide TSA with greater assurance that TSARs are engaging with foreign stakeholders in a consistent manner. TSA uses various mechanisms to share security standards and key practice information with rail stakeholders, including through BASE assessments. The cybersecurity questions in the BASE template, however, do not fully reflect two of the five core areas identified in the NIST Cybersecurity Framework. By updating the BASE cybersecurity questions to align more closely with current key practices such as the framework, TSA could better assist passenger rail and other operators in improving their cybersecurity posture. As a result, transit operators would be more aware of cybersecurity vulnerabilities and better prepared to reduce the impact from a cybersecurity incident. We are making two recommendations to TSA. The TSA Administrator should ensure that the TSAR Regional Operational Implementation Plans include guidance on how TSARs are to engage with foreign surface transportation stakeholders, including passenger rail stakeholders. (Recommendation 1) The TSA Administrator should update the BASE cybersecurity template to ensure it reflects cybersecurity key practices, including the Detect and Recover functions outlined in the NIST Cybersecurity Framework. (Recommendation 2) We provided a draft of this report to DHS for review and comment. DHS provided written comments, which are reprinted in appendix II, and also provided technical comments, which we incorporated as appropriate. DHS concurred with both recommendations and described actions TSA plans to take to address them. Specifically, to address recommendation 1, TSA plans to draft an Operational Implementation Plan, which will provide guidance to TSARs for engaging with foreign surface transportation stakeholders, including in passenger rail security. According to TSA, this plan will also serve as the outline for the development of Regional Operational Implementation Plans, which will help align resources worldwide. To address recommendation 2, TSA plans to update the BASE Cybersecurity Security Action Item section to ensure it reflects the NIST Cybersecurity Framework Detect and Recover functions. These actions, if fully implemented by TSA, should address the intent of both recommendations. We are sending copies of this report to the appropriate congressional committees and the acting Secretary of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact Triana McNeil at (202) 512-8777 or McNeilT@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributors to this report are listed in appendix III. We asked domestic and foreign passenger rail agencies and foreign passenger rail stakeholders we interviewed to identify some security related key practices or lessons learned that they employ, including, but not limited to, intermodal stations and cybersecurity. Table 5 below provides examples of common security practices both domestic and foreign officials identified; table 6 shows several additional key practices foreign rail stakeholders cited. These tables are not intended to be a comprehensive list, but provide examples of key security practices utilized by selected domestic and foreign rail agencies. Figure 4 below shows an example of a Project Servator poster displayed during an exercise at St. Pancras International Station in London. As noted in table 6 above, foreign passenger rail stakeholders cited Project Servator as a key rail security practice. In additional to the contact named above, Christopher Ferencik (Assistant Director), Sarah Turpin (Analyst in Charge), Chuck Bausell, Benjamin Crossley, Suzanne Kaasa, Tracey King, Ronald La Due Lake, William Reed, and Adam Vogt made key contributions to this report.", "summary": "Recent physical and cyberattacks on rail systems in U.S. and foreign cities highlight the importance of strengthening and securing passenger rail systems around the world. TSA is the primary federal agency responsible for securing transportation in the United States. GAO was asked to review TSA's efforts to assess passenger rail risk, as well as its role in identifying and sharing security standards and key practices. This report addresses (1) TSA's efforts to assess risk; (2) the extent to which TSA works with U.S. and foreign passenger rail stakeholders to identify security standards and key practices; and (3) the extent to which TSA shares passenger rail security standards and key practices with stakeholders. GAO analyzed TSA risk assessments from fiscal years 2015 through 2019 and reviewed TSA program documents and guidance. GAO interviewed officials from TSA, and from seven domestic rail agencies, three foreign rail agencies, and two foreign government agencies. The results from these interviews are not generalizable but provide perspectives on topics in this review. The Transportation Security Administration (TSA) assesses passenger rail risks through the Transportation Sector Security Risk Assessment, the Baseline Assessment for Security Enhancement (BASE), and threat assessments. TSA uses the risk assessment to evaluate threat, vulnerability, and consequence for attack scenarios across various transportation modes. TSA surface inspectors use the baseline assessment, a voluntary security review for mass transit, passenger rail, and highway systems, to address potential vulnerabilities and share best practices, among other things. TSA works with U.S. stakeholders to identify security standards and key practices and identifies foreign standards and practices through multilateral and bilateral exchanges. However, TSA Representatives (TSARs), the primary overseas point of contact for transportation security matters, lack specific guidance on foreign rail stakeholder engagement. As a result, TSA is less likely to be fully aware of key practices in other countries, such as station security guidance. Specific guidance would provide TSARs with clear expectations and encourage more consistent engagement with foreign rail stakeholders. Public Awareness Campaign Canine Units Emphasize security awareness Detection of vapor from explosives TSA shares standards and key practices with stakeholders, including those related to cybersecurity, through various mechanisms including BASE reviews; however, this assessment does not fully reflect current industry cybersecurity standards and key practices. For example, it does not include any questions related to two of the five functions outlined in the National Institute of Standards and Technology's Cybersecurity Framework—specifically the Detect and Recover functions. Updating the BASE questions to align more closely with this framework would better assist passenger rail operators in identifying current key practices for detecting intrusion and recovering from incidents. GAO is making two recommendations: (1) that TSA update TSAR guidance to include engaging with foreign passenger rail stakeholders; and (2) that TSA update the BASE cybersecurity questions to ensure they reflect key practices. DHS concurred with both recommendations.", "document_type": "gao"}
{"report": "Historically, the federal government has had difficulties acquiring, developing, and managing IT investments. Further, federal agencies have struggled with appropriately planning and budgeting for modernizing legacy systems; upgrading underlying infrastructure; and investing in high quality, lower cost service delivery technology. The consequences of not updating legacy systems has contributed to, among other things, security risks, unmet mission needs, staffing issues, and increased costs. Security risks. Legacy systems may operate with known security vulnerabilities that are either technically difficult or prohibitively expensive to address. In some cases, vendors no longer provide support for hardware or software, creating security vulnerabilities and additional costs. For example, in November 2017, the Department of Education’s (Education) Inspector General identified security weaknesses that included the department’s use of unsupported operating systems, databases, and applications. By using unsupported software, the department put its sensitive information at risk, including the personal records and financial information of millions of federal student aid applicants. Unmet mission needs. Legacy systems may not be able to reliably meet mission needs because they are outdated or obsolete. For instance, in 2016, the Department of State’s (State) Inspector General reported on the unreliability of the Bureau of Consular Affairs’ legacy systems. Specifically, during the summers of 2014 and 2015, outages in the legacy systems slowed and, at times, stopped the processing of routine consular services such as visa processing. For example, in June 2015, system outages caused by a hardware failure halted visa processing for 13 days, creating a backlog of 650,000 visas. Staffing issues. In order to operate and maintain legacy systems, staff may need experience with older technology and programming languages, such as the Common Business Oriented Language (COBOL). Agencies have had difficulty finding employees with such knowledge and may have to pay a premium to hire specialized staff or contractors. For example, we reported in May 2016 that the Social Security Administration (SSA) had to rehire retired employees to maintain its COBOL systems. Further, having a shortage of expert personnel available to maintain a critical system creates significant risk to an agency’s mission. For instance, we reported in June 2018 that the Internal Revenue Service (IRS) was experiencing shortages of staff with the skills to support key tax processing systems that used legacy programming languages. These staff shortages not only posed risks to the operation of the key tax processing systems, but they also hindered the agency’s efforts to modernize its core tax processing system. Increased costs. The cost of operating and maintaining legacy systems increases over time. The issue of cost is linked to the three previously described consequences—either because the other issues directly raise costs or, as in the case of not meeting mission needs, the agency is not receiving a favorable return on investment. Further, in an era of constrained budgets, the high costs of maintaining legacy systems could limit agencies’ ability to modernize and develop new or replacement systems. During the course of our review, agencies reported that they consider several factors prior to deciding whether to modernize a legacy system. In particular, agencies evaluate factors, such as the inherent risks, the criticality of the system, the associated costs, and the system’s operational performance. Risks. Agencies consider the risks associated with maintaining the legacy system as well as modernizing the legacy system. For instance, agencies may prioritize the modernization of legacy systems that have security vulnerabilities or software that is unsupported by the vendor. However, limited system accessibility may also reduce the need to modernize a legacy system. For example, air-gapped systems, which are systems that are isolated from the internet, may mitigate a legacy system’s cybersecurity risk by preventing remote hackers from having system access. Conversely, we have also reported that air-gapped systems are not necessarily secure: they could potentially be accessed by other means than the internet, such as through Universal Serial Bus devices. Even so, removing the threat of remote access is a mitigation technique used by agencies such as the Nuclear Regulatory Commission (NRC). According to NRC, the agency reduced the riskiness of using computers with unsupported operating systems by putting these computers on isolated networks or by disconnecting them from networks entirely. Criticality. Agencies consider how critical the system is to the agency’s mission. Several agencies stated that they would consider how essential a legacy system is to their agencies’ missions before deciding to modernize it. For example, the Department of Health and Human Services (HHS) stated that, when deciding to modernize a legacy system, it considers the degree to which core mission functions of the agency or other agencies are dependent on the system. Similarly, Department of Energy (Energy) officials noted that the department is required to maintain several legacy systems associated with the storage of its nuclear waste. Costs. Agencies consider the costs of maintaining a legacy system and modernizing the system. For example, according to the Department of Veterans Affairs (VA), there are systems for which a life-cycle cost analysis of the legacy system may show that the cost to modernize exceeds the projected costs to maintain the system. Similarly, the Department of Defense (DOD) noted that, before deciding on a modernization solution, it is important to assess the costs of the transition to a new or replacement solution. An agency also may decide to modernize a system when there is potential for cost savings to be realized with a modernization effort. For example, HHS stated that it may pursue the modernization of a legacy system if the department anticipates reductions in operations and maintenance costs due to efficiencies gained through the modernization. Performance. Before making the decision to modernize, agencies consider the legacy system’s operational performance. Specifically, if the legacy system is performing poorly, the agency may decide to modernize it. For example, the Department of Transportation (Transportation) stated that, if a legacy system is no longer functioning properly, it should be modernized. In addition, HHS noted that the ability to improve the functionality of the legacy system could be a reason to modernize it. As previously mentioned, in May 2016, we reported that federal legacy IT investments were becoming increasingly obsolete. In this regard, agencies had reported operating systems that used outdated languages and old parts, which were difficult to replace. Further, we noted that each of the 12 selected agencies had reported using unsupported operating systems and components, which could create security vulnerabilities and additional costs. At the time, five of the selected agencies reported using 1980s and 1990s Microsoft operating systems that stopped being supported by the vendor more than a decade ago. We concluded that agencies were, in part, maintaining obsolete investments because they were not required to identify, evaluate, and prioritize investments to determine whether the investments should be kept as-is, modernized, replaced, or retired. We pointed out that the Office of Management and Budget (OMB) had created draft guidance that would require agencies to do so, but OMB had not committed to a firm time frame for when the guidance would be issued. As such, we made 16 recommendations to OMB and the selected federal agencies to better manage legacy systems and investments. Most agencies agreed with the recommendations or had no comment. However, as of May 2019, 13 recommendations had not been implemented. In particular, OMB has not finalized and issued its draft guidance on legacy systems. Until this guidance is finalized and issued, the federal government will continue to run the risk of maintaining investments that have outlived their effectiveness and are increasingly difficult to protect from cybersecurity vulnerabilities. Congress and the executive branch have initiated several efforts to modernize federal IT, including: Identification of High Value Assets. In a December 2016 memorandum, OMB observed that continued increases in computing power combined with declining computing and storage costs and increased network connectivity had expanded the government’s capacity to store and process data. However, OMB noted that this rise in technology and interconnectivity also meant that the federal government’s critical networks, systems, and data were more exposed to cyber risks. As a result, OMB issued guidance to assist federal agencies covered by the Chief Financial Officers Act in managing the risks to these assets, which it designated as High Value Assets. Subsequently, in December 2018, OMB issued a memorandum that provided further guidance regarding the establishment and enhancement of the High Value Asset program. It stated that the program is to be operated by DHS in coordination with OMB. Further, the new guidance expanded the program to apply to all agencies (i.e., agencies covered by the Chief Financial Officers Act, as well as those not covered by the act) and expanded the definition of High Value Assets. The guidance required agencies to identify and report these assets (which may include legacy systems), assess them for security risks, and remediate any weaknesses identified, including those associated with obsolete or unsupported technology. Assessment of federal IT modernization. On May 11, 2017, the President signed Executive Order 13800, Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure. This executive order outlined actions to enhance cybersecurity across federal agencies and critical infrastructure to improve the nation’s cyber posture and capabilities against cybersecurity threats. Among other things, the order tasked the Director of the American Technology Council to coordinate a report to the President from the Secretary of DHS, the Director of OMB, and the Administrator of the General Services Administration (GSA), in consultation with the Secretary of Commerce, regarding modernizing federal IT. As a result, the Report to the President on Federal IT Modernization was issued on December 13, 2017, and outlined the current and envisioned state of federal IT. The report focused on modernization efforts to improve the security posture of federal IT and recognized that agencies have attempted to modernize systems but have been stymied by a variety of factors, including resource prioritization, ability to procure services quickly, and technical issues. The report provided multiple recommendations intended to address these issues through the modernization and consolidation of networks and the use of shared services. In particular, the report recommended that the federal government prioritize the modernization of legacy IT by focusing on enhancing security and privacy controls for those assets that are essential for agencies to serve the American people and whose security posture is most vulnerable (i.e., High Value Assets). Enactment of the Modernizing Government Technology (MGT) Act. To help further agencies’ efforts to modernize IT, in December 2017, Congress and the President enacted a law to authorize the availability of funding mechanisms to improve, retire, or replace existing IT systems to enhance cybersecurity and to improve efficiency and effectiveness. The law, known as the MGT Act, authorizes agencies to establish working capital funds for use in transitioning from legacy systems, as well as for addressing evolving threats to information security. The law also created the Technology Modernization Fund, within the Department of the Treasury (Treasury), from which agencies can “borrow” money to retire and replace legacy systems, as well as acquire or develop systems. Subsequently, in February 2018, OMB issued guidance for agencies to implement the MGT Act. The guidance was intended to provide agencies additional information regarding the Technology Modernization Fund, and the administration and funding of the related IT working capital funds. Specifically, the guidance allowed agencies to begin submitting initial project proposals for modernization on February 27, 2018. In addition, in accordance with the MGT Act, the guidance provides details regarding a Technology Modernization Board, which is to consist of (1) the Federal CIO; (2) a senior official with IT technical expertise from GSA; (3) a member of DHS’s National Protection and Program Directorate; and (4) four federal employees with technical expertise in IT development, financial management, cybersecurity and privacy, and acquisition, appointed by the Director of OMB. As of February 2019, the Technology Management Fund Board had approved funds for seven IT modernization projects across five agencies: the Department of Agriculture, Energy, the Department of Housing and Urban Development (HUD), the Department of Labor, and GSA. For example, the board approved $20 million for HUD to modernize a mainframe and five COBOL-based applications that are expensive to maintain. According to the board’s website, without these funds, HUD would not have been able to pursue this project for several years. Issuance of the President’s Management Agenda. In March 2018, the Administration issued the President’s Management Agenda, which lays out a long-term vision for modernizing the federal government. The agenda identifies three related drivers of transformation—IT modernization; data, accountability, and transparency; and the workforce of the future—that are intended to push change across the federal government. The President’s Management Agenda identifies 14 related Cross- Agency Priority goals, many of which have elements that involve IT. In particular, the Cross-Agency Priority goal on IT modernization states that modern technology must function as the backbone of how government serves the public in the digital age. Further, the goal on IT modernization provides three priorities that are to guide the Administration’s efforts to modernize federal IT: (1) enhancing mission effectiveness by improving the quality and efficiency of critical services, including the increased utilization of cloud-based solutions; (2) reducing cybersecurity risks to the federal mission by leveraging current commercial capabilities and implementing cutting edge cybersecurity capabilities; and (3) building a modern IT workforce by recruiting, reskilling, and retaining professionals able to help drive modernization with up-to-date technology. As determined by our review of 65 critical federal legacy systems (see appendix II), the 10 most critical legacy systems in need of modernization are maintained by 10 different federal agencies whose missions are essential to government operations, such as emergency management, health care, and wartime readiness. These legacy systems provide vital support to the agencies’ missions. According to the agencies, these legacy systems range from about 8 to 51 years old and, collectively, cost approximately $337 million annually to operate and maintain. Several of the systems use older languages, such as COBOL and assembly language code. However, as we reported in June 2018, reliance on assembly language code and COBOL has risks, such as a rise in procurement and operating costs, and a decrease in the availability of individuals with the proper skill sets. Further, several of these legacy systems are also operating with known security vulnerabilities and unsupported hardware and software. For example, DHS’s Federal Emergency Management Agency performed a security assessment on its selected legacy system in September 2018. This review found 249 reported vulnerabilities, of which 168 were considered high or critical risk to the network. With regard to unsupported hardware and software, Interior’s system contains obsolete hardware that is not supported by the manufacturers. Moreover, the system’s original hardware and software installation did not include any long-term vendor support. Thus, any original components that remain operational may have had long-term exposure to security and performance weaknesses. Table 1 provides a generalized list of each of the 10 most critical legacy systems that we identified, as well as agency-reported system attributes, including the system’s age, hardware’s age, system criticality, and security risk. (Due to sensitivity concerns, we substituted a numeric identifier for the system names and are not providing detailed descriptions). Appendix III provides additional generalized agency- reported details on each of these 10 legacy systems. Given the age of the hardware and software in legacy systems, the systems’ criticality to agency missions, and the security risks posed by operating aging systems, it is imperative that agencies carefully plan for their successful modernization. Documenting modernization plans in sufficient detail increases the likelihood that modernization initiatives will succeed. According to our review of government and industry best practices for the modernization of federal IT, agencies should have documented modernization plans for legacy systems that, at a minimum, include three key elements: (1) milestones to complete the modernization, (2) a description of the work necessary to modernize the legacy system, and (3) details regarding the disposition of the legacy system. Of the 10 identified agencies with critical systems most in need of modernization, seven (DOD, DHS, Interior, Treasury, the Office of Personnel Management (OPM), the Small Business Administration (SBA), and SSA) had documented modernization plans for their respective critical legacy systems and three did not have documented plans. The three agencies that did not have documented modernization plans for their critical legacy systems were: (1) Education, (2) HHS, and (3) Transportation. Of the seven agencies with documented plans, DOD and Interior had modernization plans that addressed each of the three key elements. For example, Interior submitted documentation of both completed and forthcoming milestones leading to the deployment of the modernized system. The department also provided a list of the mandatory requirements for the updated system, as well as the work that needed to be performed at each stage of the project, including the disposition of the legacy system. Likewise, DOD provided documentation of the milestones and the work needed to complete the modernization of its legacy system. In addition, the documentation discussed the department’s plans for the disposition of the legacy system. While the other five agencies—Treasury, DHS, OPM, SBA, and SSA— had developed modernization plans for their respective legacy systems, their plans did not fully address one or more of the three key elements. For instance, DHS’s Federal Emergency Management Agency’s modernization plan for its selected legacy system described the work that the department needed to accomplish, but did not include the associated milestones or the disposition of the legacy system. Similarly, SBA included milestones and a plan for the disposition of the legacy system, but did not include a description of the work necessary to accomplish the modernization. Treasury, OPM, and SSA partially included one or more of the key elements in their modernization plans. For instance, OPM’s and SSA’s plans included upcoming milestones for one part of the initiative, but not the entire effort. Similarly, OPM’s modernization plans only described a portion of the work necessary to complete each modernization initiative. Further, none of these four agencies’ modernization plans included considerations for the disposition of legacy system components following the completion of the modernization initiatives. While agencies may be using development practices that minimize initial planning, such as agile, agencies should have high-level information on cost, scope, and timing. Table 2 identifies the seven agencies with documented modernization plans for their critical systems, as well as the extent to which the plans were sufficiently detailed to include the three key elements. (Due to sensitivity concerns, we substituted a numeric identifier for the system names.) The agencies provided a variety of explanations for the missing modernization plans. For example, according to the three agencies without documented modernization plans: Education’s modernization plans were pending the results of a comprehensive IT visualization and engineering project that would determine which IT systems and services could be feasibly modernized, consolidated, or eliminated; HHS had entered into a contract to begin a modernization initiative but had not yet completed its plans; and Transportation had solicited information from industry to determine whether the agency’s ideas for modernization were feasible. Of the five agencies which had plans that lacked key elements, officials within SSA’s office of the CIO stated that the agency has yet to complete its modernization planning, even though modernization efforts are currently underway. The officials said that they will update the planning documentation and make further decisions as the modernization effort progresses. Officials within DHS’s Federal Emergency Management Agency’s Office of the CIO stated that its plans for modernizing the system we reviewed (System 4) are contingent on receiving funding and being able to allocate staffing resources to planning activities. According to the officials, the agency is also integrating its plans for modernizing System 4 with the management of the rest of the agency’s systems. Similarly, Treasury officials stated that IRS’s efforts to complete planning for the remaining modernization activities have been delayed due to budget constraints. In addition, officials within OPM’s Office of the CIO stated that its modernization plan did not extend to fiscal year 2019 because there were changes in leadership during the creation of the plan, and because of uncertainty in funding amounts. While we recognize that system modernizations are dependent on funding, it is important for agencies to prioritize funding for the modernization of these critical legacy systems. In addition, Congress provided increased authority for agencies to fund such modernization efforts through the MGT Act’s Technology Modernization Fund and the related IT working capital funds. Until the agencies establish complete legacy system modernization plans that include milestones, describe the work necessary to modernize the system, and detail the disposition of the legacy system, the agencies’ modernization initiatives will have an increased likelihood of cost overruns, schedule delays, and overall project failure. Project failure would be particularly detrimental in these 10 cases, not only because of wasted resources, but also because it would prolong the lifespan of increasingly vulnerable and obsolete systems, exposing the agency and system clients to security threats and potentially significant performance issues. Further, agencies may not be effectively planning for the modernization of legacy systems, in part, because they are not required to. As we reported in May 2016, agencies are not required to identify, evaluate, and prioritize existing IT investments to determine whether they should be kept as-is, modernized, replaced, or retired. We recommended that OMB direct agencies to identify legacy systems needing to be replaced or modernized. As of April 2019, OMB had not implemented this recommendation. OMB staff stated that agencies were directed to manage the risk to High Value Assets associated with legacy systems in OMB’s December 2018 guidance. While OMB’s guidance does direct agencies to identify, report, assess, and remediate issues associated with High Value Assets, it does not require agencies to do so for all legacy systems. Until OMB requires agencies to do so, the federal government will continue to run the risk of continuing to maintain investments that have outlived their effectiveness. The 24 Chief Financial Officers Act agencies in our review identified a total of 94 examples of successful modernizations of legacy systems undertaken in the last 5 years. The initiatives were of several types, including those aimed at transforming legacy code into a more modern programming language, migrating legacy services (e.g., email) to the cloud, and re-designing a legacy mainframe to a cloud-based application. Among these examples, the five that we selected reflect a mix of different agencies, types of system modernization initiatives, and types of benefits realized from the initiatives. Table 3 provides details on the five examples of successful IT modernization initiatives, as reported by their respective agencies, as well as the reported benefits related to those initiatives. The five agencies attributed the success of their modernization initiatives to various factors, including: using automated technologies to examine programming code and perform testing (DOD and Treasury); testing the system thoroughly (SSA and Treasury); actively engaging the end users and stakeholders throughout the modernization process (SSA and Treasury); cultivating a partnership between industry and government (DOD); following management practices on change and life cycle management (Education); developing and implementing an enterprise-wide cost collection and data analysis process for commodity IT to track and measure progress against consolidation, optimization, and savings targets (DHS); creating an interface that was consistent across systems (SSA); having strong executive leadership and support (Treasury); and using agile principles to facilitate the team’s ownership of the project (Treasury). These factors are largely consistent with government and industry best practices. For example, we reported in 2011 on critical success factors associated with major acquisitions, including engaging stakeholders and having the support of senior executives. Similarly, OMB’s guidance on High Value Assets calls for agencies’ plans to address change management and life cycle management. Likewise, the Software Engineering Institute’s Capability Maturity Model® Integration for Development recommends that organizations engage stakeholders, practice effective change and life cycle management, and thoroughly test systems, among other practices. Further, our Information Technology Investment Management framework recommends involving end users, implementing change and life cycle management processes, and obtaining the support of executive leadership. Agencies that follow such practices are better positioned to modernize their legacy systems. Doing so will also allow the agencies to leverage IT to successfully address their missions. The 10 most critical federal legacy systems in need of modernization are becoming increasingly obsolete. Several agencies are using outdated computer languages, which can be difficult to maintain and increase costs. Further, several of these legacy systems are also operating with unsupported hardware and software and known security vulnerabilities. Most agencies did not have complete plans to modernize these legacy systems. Due to the criticality and possible cybersecurity risks posed by operating aging systems, having a plan that includes how and when the agency plans to modernize is vital. In the absence of such plans, the agencies increase the likelihood of cost overruns, schedule delays, and overall project failure. Such outcomes would be particularly detrimental because of the importance of these systems to agency missions. Successfully modernizing legacy systems is possible, as demonstrated by the five highlighted examples. Agencies attributed the success of their modernization initiatives to a variety of management and technical factors that were consistent with best practices. In the LOUO report that we are issuing concurrently with this report, we are making a total of eight recommendations to eight federal agencies to identify and document modernization plans for their respective legacy systems, including milestones, a description of the work necessary, and details on the disposition of the legacy system. We requested comments on a draft of this report from OMB and the 24 agencies included in our review. The eight agencies to which we made recommendations in the LOUO report agreed with our findings and recommendations. In addition, OMB and the 16 agencies to which we did not make recommendations either agreed with our findings, did not agree or disagree with the findings, or stated that they had no comments. Further, multiple agencies provided technical comments, which we have incorporated, as appropriate. The following eight agencies agreed with our recommendations: In written comments from Education, the agency stated that it concurred with the recommendation and indicated its intent to address it. Education’s comments are reprinted in appendix IV. In written comments from HHS on the LOUO version of this report, the agency stated that it concurred with the recommendation and intends to evaluate ways to provide its modernization plan, including milestones and a description of the work necessary to modernize the system. HHS also provided technical comments that we incorporated, as appropriate. HHS deemed some of the information in its original agency comment letter pertaining to particular legacy systems to be sensitive, which must be protected from public disclosure. Therefore, we have omitted the sensitive information from the version of the agency comment letter that is reprinted in appendix V of this report. In written comments, DHS stated that it concurred with our recommendation. DHS’s comments are reprinted in appendix VI. In comments received via email from Transportation’s Director of Audit Relations and Program Improvement on May 9, 2019, the agency stated that it agreed with our recommendation. In comments from Treasury’s Supervisory IT Specialist/Performance and Governance Analyst, received via email on May 17, 2019, the department stated that it agreed with our recommendation. In addition, Treasury’s component agency, IRS, provided written comments which stated that it agreed with the recommendation. The agency said it intends to develop a multiyear retirement strategy for its system to address the recommendation. In its written comments, IRS also stated that our draft report did not accurately convey that the legacy system replacement project is intended to only replace core components of its selected legacy system. The agency said that, even when the entire replacement project is completed, it will only address a portion of the work required to retire the legacy system. In response, we modified our discussion of this project in the report. IRS’s comments are reprinted in appendix VII. In written comments from OPM on the LOUO version of this report, the agency stated that it concurred with the recommendation and indicated its plans to address the recommendation. OPM also provided technical comments that we incorporated, as appropriate. OPM deemed some of the information in its original agency comment letter pertaining to particular legacy systems to be sensitive, which must be protected from public disclosure. Therefore, we have omitted the sensitive information in the version of the agency comment letter that is reprinted in appendix VIII. In written comments, SBA concurred with our recommendation and stated that it intends to include a description of the work necessary to modernize the legacy system in the initiative’s project plan. The agency estimated that it will address the recommendation by July 31, 2019. SBA deemed some of the information in its original agency comment letter pertaining to particular legacy systems to be sensitive, which must be protected from public disclosure. Therefore, we have omitted the sensitive information from the version of the agency comment letter that is reprinted in appendix IX. In written comments from SSA, the agency stated that it agreed with our recommendation. The agency added that it is modernizing its legacy system using agile software methods and a multiyear roadmap of development activities. The agency further stated that, as it completes its modernization work, it expects to retire most of the legacy software associated with System 10. SSA also provided technical comments that we incorporated, as appropriate. SSA’s comments are reprinted in appendix X. In addition, we received responses via email from 14 agencies to which we did not make recommendations. Of these agencies, three agreed with our findings and 11 stated that they did not have comments on the report. Two other agencies—HUD and the U.S. Agency for International Development—provided written comments in which they expressed appreciation for the opportunity to review the report, but did not state whether they agreed or disagreed with our findings. These agencies’ comments are reprinted in appendixes XI and XII, respectively. Further, in an email from OMB staff on May 22, 2019, the agency did not state whether it agreed or disagreed with our findings, but provided technical comments that we incorporated, as appropriate. We are sending copies of this report to the appropriate congressional committees; the Secretaries of the Departments of Agriculture, Commerce, Defense, Education, Energy, Health and Human Services, Homeland Security, Housing and Urban Development, Labor, State, the Interior, the Treasury, Transportation, and Veterans Affairs; the U.S. Attorney General (Department of Justice); the Administrators of the Environmental Protection Agency, General Services Administration, National Aeronautics and Space Administration, Small Business Administration, and the U.S. Agency for International Development; the Commissioner of the Social Security Administration; the Directors of the National Science Foundation and the Office of Personnel Management; and the Chairman of the Nuclear Regulatory Commission; and other interested parties. This report is also available at no charge on the GAO website at http://www.gao.gov. Should you or your staffs have any questions on information discussed in this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XIII. Our objectives were to (1) identify the most critical federal legacy systems in need of modernization and evaluate plans for modernizing them, and (2) identify examples of information technology (IT) legacy system modernization initiatives in the last 5 years that agencies considered successful. The scope of our review included the 24 agencies covered by the Chief Financial Officers Act of 1990. This report presents a public version of a “limited official use only” (LOUO) report that we are also issuing today. The Department of Homeland Security and the Department of the Interior determined that certain information in our original report should be protected from public disclosure. Therefore, we will not release the LOUO report to the general public because of the sensitive information it contains. The LOUO report includes eight recommendations that we made to eight agencies to document modernization plans for particular legacy systems, including milestones, a description of the work necessary, and details on the disposition of the legacy system. In this public version of the report, we have omitted sensitive information regarding particular legacy systems. Specifically, we have deleted systems’ names and other information that would identify the particular system, such as specific descriptions of the systems’ purposes and vulnerabilities. Although the information provided in this report is more limited, the report addresses the same objectives as the LOUO report and is based on the same audit methodology. We provided a draft of this report to agency officials to obtain their review and comments on the sensitivity of the information contained herein. We confirmed with the agency officials that this report can be made available to the public without jeopardizing the security of federal agencies’ legacy systems. To identify the most critical legacy systems in need of modernization, we first reviewed the agencies’ 2017 responses to congressional committees’ requests for information that identified the agencies’ top three legacy systems in need of modernization. We then asked the agencies to either confirm that those systems were still considered their top systems in need of modernization or update their lists to include the three systems most in need of modernization. All 24 agencies either confirmed or updated their lists of legacy systems most in need of modernization. This resulted in a collective list of 65 systems. However, due to sensitivity concerns, we are not disclosing the names of the systems in this report. Appendix II provides a generalized list of the systems. To develop a set of attributes for determining systems’ obsolescence and their need for modernization, we reviewed available technical literature, such as: General Services Administration’s Unified Shared Services Management’s Modernization and Migration Management (M3) Playbook and M3 Playbook Guidance, American Technology Council’s Report to the President on Federal IT Modernization, Office of Management and Budget’s Management of Federal High IBM Center for The Business of Government’s A Roadmap for IT Modernization in Government, and American Council for Technology-Industry Advisory Council’s Legacy System Modernization: Addressing Challenges on the Path to Success. We also consulted with system development experts within GAO and reviewed our prior report on federal legacy systems. Using these sources, we developed a set of 14 total attributes for determining systems’ obsolescence and their need for modernization. We then asked the agencies in our review to provide the associated details for the selected systems. We considered these details to rank the systems against the attributes that we compiled. We assigned point values to each system based on the systems’ agency-reported attributes. Table 4 details the nine attributes and associated point values and ranges we used to initially rank the legacy systems. We then totaled the assigned points for each legacy system and ranked the results from highest to lowest number of assigned points. While we had planned to select the top 20 systems with the most points for more detailed analysis, three systems were ranked in nineteenth place. As a result, we selected 21 systems for our review. We collected additional information on the 21 selected systems and performed a second round of analysis, scoring, and ranking. Based on the second set of scores, we identified the 10 systems with the highest scores as being the most critical legacy systems in need of modernization. We also supplemented our review with interviews of officials in the agencies’ offices of the Chief Information Officer and program offices for the selected legacy systems. Table 5 details the five attributes and associated point values and ranges we used to rank the legacy systems in the subsequent round of analysis. Table 6 lists these 10 selected systems according to their designated identifiers. However, due to sensitivity concerns, we substituted a numeric identifier for the name of each system. To evaluate agencies’ plans for modernizing the 10 federal legacy systems most in need of modernization, we requested that agencies provide us with the relevant plans. These modernization plans could have been contained within several types of documentation, since a system modernization could be a new system development, a system acquisition, or a renovation of the legacy system. For example, if an agency was acquiring a new system from a vendor, the plans for modernization could have been contained within an acquisition plan or a statement of work in a contract. Likewise, if an agency was developing a new system on its own, the modernization plans could have been within a project plan or design document. We reviewed government and industry best practice documentation on the identification and modernization of legacy systems, including: General Services Administration’s Unified Shared Services Management’s Modernization and Migration Management (M3) Playbook and M3 Playbook Guidance, American Technology Council’s Report to the President on Federal Office of Management and Budget’s Management of Federal High IBM Center for The Business of Government’s A Roadmap for IT Modernization in Government, and American Council for Technology-Industry Advisory Council’s Legacy System Modernization: Addressing Challenges on the Path to Success. Based on our reviews of these sources, we determined that agencies’ documented plans for system modernization should include, at a minimum, (1) milestones to complete the modernization, (2) a description of the work necessary to modernize the system, and (3) details regarding the disposition of the legacy system. We then analyzed agencies’ documented modernization plans for the selected systems to determine whether the plans included these elements. If an agency’s plans included milestones for only a portion of the initiative or only described a portion of the work necessary to complete the modernization, we assigned the agency a partial rating. Appendix III provides details on each of the selected systems and the agencies’ plans for modernizing them. To identify examples of successful IT legacy system modernization initiatives, we first asked each of the 24 agencies to provide us with examples of their successful modernization initiatives completed between 2014 and 2018. The agencies reported 94 examples of successful modernization initiatives. We also reviewed the agencies’ responses to congressional committees’ requests for information to determine other possible successful modernization initiatives at these agencies. Using the examples discovered in this process and the agency-provided examples, we then collected and reviewed documentation describing the modernization initiatives, such as case studies and the agencies’ written responses to our questions about the initiatives. We used our professional judgment to select examples that reflected a mix of different agencies, types of system modernization initiatives, and types of benefits realized from the initiatives. We ultimately included in our review those modernization initiatives that two or more members of our audit team selected as examples that reflected a mix of different agencies, types of system modernization initiatives, and types of benefits realized from the initiatives. We also coordinated with the selected agencies’ Offices of Inspector General to determine whether those offices had any past or current audit work that would contradict the agencies’ determination that the selected initiatives were successful. We conducted this performance audit from January 2018 to June 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Each of the 24 Chief Financial Officers Act agencies identified their agency’s most critical legacy systems in need of modernization. The agencies identified a total of 65 such systems. The agencies also identified various attributes of the legacy systems, including the systems’ age, hardware age, system criticality, and security risk. Table 7 provides a generalized list of the most critical legacy systems in need of modernization, as identified by the agencies, as well as selected factors related to each system’s age and criticality. (Due to sensitivity concerns, we substituted alphanumeric identifiers for the names of the agencies’ systems. Specifically, we assigned a number to identify each of the 10 most critical legacy systems in need of modernization that we discuss in this report and we assigned a letter or letters to identify the remaining 55 systems.) This appendix describes the 10 most critical legacy systems in need of modernization, as identified during our review. The profiles of each system describe (1) the system’s purpose, (2) the reason that the system needs to be modernized, (3) the agency’s plans for modernization, and (4) possible benefits to be realized once the system is modernized. The Department of Defense (DOD)—U.S. Air Force’s System 1 provides configuration control and management to support wartime readiness and operational support of aircraft, among other things. See figure 1 for a photograph of airmen maintaining an aircraft. The Department of Education’s (Education) System 2 processes and stores student information and supports the processing of federal student aid applications. Education first implemented System 2 in 1973. Agency officials stated that the system runs approximately 1 million lines of Common Business Oriented Language (COBOL) on an IBM mainframe. COBOL is a legacy language that can be costly to maintain. The department noted that 18 contractors are employed to maintain the COBOL programming language for this and another system. Education officials stated that the agency would like to modernize System 2 to eliminate reliance on COBOL, simplify user interactions, improve integration with other applications, respond to changing business requirements more quickly, and decrease development and operational costs. Education officials stated that the agency intends to modernize System 2 as part of its Next Generation Financial Services Environment initiative. This initiative is to modernize Federal Student Aid’s technical and operational architecture and improve the customer experience. The agency expects to consolidate all customer-facing websites and implement a new loan servicing platform to benefit federal student loans. Education has not developed a plan for the modernization of System 2. According to agency officials, these plans are pending the results of a comprehensive information technology (IT) visualization and engineering project that will determine which IT systems and services could be feasibly modernized, consolidated, or eliminated. While Education has not calculated the specific cost savings associated with modernizing System 2, the department anticipates potential cost savings, including decreased hardware and software licensing costs and decreased costs associated with changes to business rules. According to the agency, other potential benefits of modernizing this system include integration across the enterprise, improved cybersecurity and data protection, reduced system complexity, and improved system efficiency. The Department of Health and Human Services’ (HHS) System 3 is a clinical and patient administrative information system. HHS’s component, Indian Health Service’s (IHS) uses the system to gather, store, and display clinical, administrative, and financial information on patients seen in a clinic, hospital, or remotely through the use of telehealth and home visit practices. HHS officials stated that the modernization of System 3 is imperative. Specifically, the agency noted that the system’s technical architecture and infrastructure were outdated. This has resulted in challenges in developing new capabilities in response to business and regulatory requirements. Further, System 3 is coded in C++ and MUMPS. MUMPS is a programming language that HHS considers to be a legacy language. The agency noted that it has become increasingly difficult to find programmers proficient in writing code for MUMPS. Lastly, the system’s more than 50 modules were added over time to address new business requirements. The software is installed on hundreds of separate computers, which has led to variations in the configurations at each site. According to IHS, this type of add-on development becomes detrimental over time and eventually requires a complete redesign to improve database design efficiency, process efficiency, workflow integration, and graphical user interfaces. While the agency does not yet have modernization plans, in September 2018, HHS awarded a contract to conduct research for modernizing IHS’s health information technology (IT) infrastructure, applications, and capabilities. According to the department, the research will be conducted in several stages over the next year, and a substantial part of the research will be an evaluation of the current state of health IT across IHS’s health facilities. Once the research is conducted, in consultation with IHS and its stakeholders, the contractor will use the findings and recommendations to propose a prioritized roadmap for modernization. According to HHS, the agency will be completing the modernization initiative over the next 5 years, but anticipated that it may be able to begin to execute an implementation plan as early as 2020. With regards to potential cost savings, HHS noted that the modernization will take significant capital investment to complete and it is unknown whether the modernization will lead to cost savings. HHS officials stated that this modernization could improve interoperability with its health care partners, the Department of Veterans Affairs and the Department of Defense, and significantly enhance direct patient care. The Department of Homeland Security—Federal Emergency Management Agency’s (FEMA) System 4 consists of routers, switches, firewalls, and other network appliances (all referred to as devices) to support the connectivity of FEMA sites. According to the agency, System 4 needs to be modernized because there are significant cyber and network vulnerability risks associated with its end of life (i.e., no longer supported or manufactured by the vendor) devices. In particular, the system’s devices typically require replacement every 3 to 5 years from the date of purchase. Despite this, the majority of the hardware was purchased between 8 and 11 years ago. As of December 2018, about 545 of these devices were at the end of life. In a security assessment report performed in September 2018, System 4 received 249 security findings, of which 168 were high or critical risk to the system. Further compounding this issue, the agency is not certain exactly how many devices make up the system. In particular, FEMA officials stated that the vendor completed an inventory of devices in May 2018, but that inventory did not align with other inventory counts. As a result, the agency plans to develop an inventory reconciliation strategy and process to address this issue. FEMA intends to replace System 4’s devices in two phases. The first phase will target the agency’s smaller facilities, while the second phase is to address the larger facilities, which may require more complex installations. FEMA’s Office of the Chief Information Officer is conducting site surveys to better define requirements and cost estimates. While the agency has yet to develop finalized modernization plans for this initiative with milestones, DHS officials and contract information technology staff developed a list of future recommended activities that would help modernize the system as part of their November 2018 quarterly business review. Despite the lack of finalized plans, FEMA intends to replace 240 of the 545 devices that are at the end of support, if funds are available. The agency also intends to upgrade the remaining 305 devices in the future, if funds are available. The agency has not calculated the exact amount of cost savings. Once the system is completely updated and a lifecycle replacement operations and maintenance support plan is in place and funded, FEMA and DHS expect to realize cost savings based on new technology and increased throughput. Further, the agency stated that with new equipment, it would be able to meet mission requirements and take advantage of new technologies. In addition, replacing these unsupported devices would significantly reduce downtime and increase network availability. The Department of the Interior’s (Interior) System 5 is an Industrial Control System (ICS) Supervisory Control and Data Acquisition (SCADA) System that supports the general operation of dams and power plants on a particular river and its tributaries. The system serves its customers by, among other things, starting and stopping the generators, adjusting the output of electricity to assure electric grid stability, and monitoring the operating conditions of dam and power plant equipment. Figure 2 shows an example of an Interior dam. The system is approximately 18 years old and contains obsolete hardware that is not supported by the manufacturers. Further, according to a program official, the system’s original hardware and software installation did not include any long-term vendor support. Thus, any original components that remain operational may have had long-term exposure to security and performance weaknesses. In January 2014, the Director of National Intelligence testified that ICS and SCADA systems used in electrical power distribution provided an enticing target to malicious actors and that, although newer architectures provide flexibility, functionality, and resilience, large segments of the systems remain vulnerable to attack, potentially causing significant economic or human impact. Further, according to Interior’s system modernization plans, the agency needs to modernize the system in order to increase data collection capabilities and security. Specifically, the system is expected to interface with more plant equipment and collect and report on more data than it has in the past. According to Interior’s plans, the modernized system is expected to accommodate future growth requirements. The plans also support the complete replacement of the system’s obsolete hardware and software. The modernization plans also outline goals, milestones, and the work to be accomplished. The agency plans to complete the modernization by January 2020. By replacing the legacy system, Interior plans to realize a number of potential benefits, including annual cost savings of $152,000. In addition, the system will no longer run on obsolete, unsupported hardware. Furthermore, newer software and hardware are expected to allow for the automation of compliance tasks, increase system security, and expand system availability. According to the system’s fiscal year 2017 operational analysis, these benefits should create a more reliable system for both the agency and the customers of the networked hydroelectric dams. The Department of the Treasury’s Internal Revenue Service’s (IRS) System 6 contains taxpayer data. Many IRS processes depend on output, directly or indirectly, from this data source. System 6 was written in a now outdated assembly language code and Common Business Oriented Language (COBOL). The department and we have raised a number of concerns related to this system’s reliance on assembly language code and COBOL, the maintainability of the system, and staff attrition. For example, in May 2016, we reported that legacy systems using outdated languages may become increasingly more expensive and agencies may pay a premium to hire staff or contractors with the knowledge to maintain these systems. IRS plans to address these concerns by modernizing core components of System 6. The new system is intended to provide improved functionality. However, IRS is having trouble fully staffing the modernization effort, resulting in significant delays. While the agency has developed modernization plans, they are incomplete. For example, the plans’ milestones do not go past the current project and their descriptions of the work necessary to complete the project are at a higher level when outlining the goals of future stages. In May 2019, the agency stated that even when the current modernization effort is fully implemented, only a portion of the work required to retire the legacy system will have been completed. The agency has not provided a target date for decommissioning the legacy system. While IRS does not anticipate cost savings associated with the modernization of this system, it anticipates many internal and external benefits for both the taxpayer and the agency. In particular, according to the IRS’s Fiscal Year 2019 Capital Investment Plan, the benefits of modernizing this system include: (1) increased agility of agency response to changing taxpayer priorities and legislation; (2) reduced IT costs and complexity; (3) enhanced analytics and reporting to greatly improve compliance and issue resolution; and (4) reduced burden of manually intensive processes on IRS employees, by enabling automated calculations that currently are not possible. The Department of Transportation’s (Transportation) Federal Aviation Administration’s (FAA) System 7 contains information on aircraft and pilots. The system also provides information to other government agencies, including those responsible for homeland security and investigations of aviation accidents. According to Transportation, the system is DOS-based and needs to be updated to continue to efficiently meet its mission. Specifically, some of the core system components are mainframe applications that have been in operation since 1984. In addition, the system is running unsupported software, including one operating system that was last supported by the vendor in 2010. FAA is planning to implement a new system to streamline processes, allow for the submission of electronic applications and forms, automate registration processes, improve data availability, and implement additional security controls. However, the agency does not currently have a documented modernization plan. Officials stated that the agency is seeking alternatives to modernize the system and meet legislative requirements. FAA has asked interested vendors to respond to a request for information. According to the agency, the responses to this request are intended to inform strategic decisions about the modernization, and are planned to ultimately lead to proposed solutions from industry. While FAA has not calculated the specific cost savings associated with modernizing the system, the agency stated that it anticipates potential cost savings. Agency officials stated that they plan to have information on the anticipated cost savings in November 2019. The agency also expects that the modernized system will provide enhanced security. The Office of Personnel Management’s (OPM) System 8 consists of the hardware, software, and service components that support OPM’s information technology (IT) applications and services. This system supports the agency’s business functions and supports the agency in providing investigative products and services for more than 100 federal agencies. Modernizing this system is especially important due to past security incidents and persistent security concerns. Specifically, according to OPM, segments of the agency’s infrastructure were allowed to age beyond end of life and now pose a significant risk in performance and security to IT operations. Further, in October 2017, OPM’s Office of the Inspector General (OIG) reported that the agency’s IT environment contained many instances of unsupported software and hardware, where the vendor no longer provided patches, security fixes, or updates for the software. As a result, the OIG noted that there was increased risk that OPM’s IT environment contained known vulnerabilities that would never be patched, and could have been exploited to allow unauthorized access to data. In June 2015, OPM reported that an intrusion into its systems had affected the personnel records of about 4.2 million current and former federal employees. Then, in July 2015, the agency reported that a separate but related incident had compromised its systems and the files related to background investigations for 21.5 million individuals. At a June 2015 Congressional hearing, OPM’s Director stated that the modernization of the IT infrastructure was critical to protecting the agency’s data from adversaries. The Director also stated that it was not feasible to implement encryption on networks that were too old, but noted that OPM was taking other steps to secure the networks. OPM plans to modernize System 8 by upgrading hardware at the end of life, migrating off of legacy operating systems and support software, and augmenting the agency’s established policies and procedures. In fiscal year 2018, OPM completed software and hardware upgrades, including replacement of core switches, network end points, and laptops. In fiscal year 2019, the agency plans to continue its focus on refreshing aged IT infrastructure, so that its hardware components will have the proper vendor support. OPM developed multiple documents related to the planning of this modernization effort, including a modernization schedule, and its fiscal year 2019 budget justification. However, the modernization plans contained in these documents did not include details for the entire modernization effort. The milestones in these documents, for instance, were either no longer current or only contained milestones regarding one part of the project. While the budget justification did outline what it planned to accomplish in fiscal years 2018 and 2019, it did not mention the rest of the work needed to complete the infrastructure modernization. Similarly, the OIG has reported concerns regarding the agency’s plans to modernize its infrastructure. Most recently, in June 2018, the OIG reported that OPM was generally continuing in the right direction toward modernizing its IT environment, but the OIG had concerns with the agency’s plan for modernization and its overall approach to IT modernization. For example, the OIG was concerned that OPM’s planning documents did not identify the full scope of the modernization effort or contain cost estimates for the individual initiatives or the effort as a whole. The OIG planned to monitor and continue to report on the agency’s progress in modernizing its infrastructure. OPM anticipates realizing both financial and nonfinancial benefits with the modernization of its infrastructure. For example, as a part of its overall infrastructure modernization, the agency avoided approximately $16 million in costs as part of its data center consolidation efforts for fiscal year 2018. The agency also expects that cybersecurity and operational risks associated with end of life hardware will be reduced. To that end, the agency stated that remediating end of life hardware also should allow OPM the ability to address identified security vulnerabilities and avoid operational downtime, as support is more readily available. The Small Business Administration’s (SBA) System 9 is a system that, according to the agency, provides identification, authentication, and authorization services for several of the agency’s applications. According to the agency, the system was developed by SBA and originally implemented in 2002. Agency officials stated that System 9’s hardware and software are no longer supported by the associated vendors. Consequently, according to the agency, it is paying for extended support contracts that have increased operating costs for the system. Further, agency officials stated that the system resides on a platform that is scheduled to be decommissioned within the next year. In addition, the system is coded using a programing language that the agency considers to be a legacy programming language (among others). The agency’s documented modernization plan includes milestones to complete the modernization and plans for the disposition of the legacy system following system modernization; however, the plan does not include a description of the work necessary to complete the modernization. However, agency officials stated that it intends to replace the system’s functionality with login.gov. Login.gov was developed and is maintained by the General Services Administration as a single sign-on trusted identity platform. Login.gov provides identification and authentication for applications and is intended to offer the public secure and private online access to participating government programs. However, according to the agency, since login.gov does not provide authorization controls, SBA intends to develop additional software to provide authorization controls beginning in March 2019. According to the agency, it does not anticipate any cost benefits from modernizing System 9. However, the agency expects that the security and stability of the system will increase. The Social Security Administration’s (SSA) System 10 supports the provision of particular Social Security benefits to eligible people. Currently, SSA collects detailed information from the recipients in person, by telephone, and via the internet on multiple platforms (e.g., desktops and hand-held devices), and from internal and external interface methods. System 10 is comprised of many applications that collect information, make payments, and communicate with SSA’s clients. According to SSA’s October 2017 information technology modernization plan, the agency needed to modernize its core systems, including System 10, because of complications related to their age and original system design. SSA’s modernization plan indicates that, since implementation, these systems had been subjected to constant modifications to incorporate changes in legislation, regulations, and policy. Through the years, new technologies and capabilities had been integrated into the core systems and delivering new capabilities was becoming exorbitantly expensive. Further, most of the agency’s systems, including System 10, are generally unconnected to each other, creating functional silos servicing independent lines of business. According to the agency, navigating these systems is challenging, and copying beneficiary data from system to system can result in data becoming out of sync. According to the agency’s modernization plan, SSA intends to replace its core systems, including System 10, with new components and platforms, engineered for usability, interoperability, and future adaptability. Work accomplished over several years of incremental modernization has already resulted in moving a substantial portion of System 10 away from old technologies. For instance, according to SSA officials in the Office of the Deputy Commissioner, Systems, SSA moved System 10 to a modern, relational database platform and modernized aspects of the user interface. According to an SSA 5-year modernization roadmap, the agency is currently working to modernize and create web services as a part of the effort to consolidate SSA’s initial claims processes; however, the roadmap does not offer specific information about these efforts. As for its modernization planning efforts, SSA’s plans include overall modernization goals, a high-level overview of the planned system architecture, milestones for fiscal year 2018, and a description of the work that it had planned to accomplish in fiscal year 2018. However, the plans do not include either System 10-specific milestones or a description of the work necessary to modernize the legacy system beyond fiscal year 2018. Further, the document does not include plans for the disposition of the legacy system after modernization. According to officials in the Office of the Deputy Commissioner, Systems, the agency will update the planning documentation and make further decisions as the modernization effort progresses. SSA expects that modernizing System 10 will result in cost savings in addition to many other benefits. For instance, the agency expects that it will be able to save approximately $38 million from modernizing System 10 and other systems running in the agency’s mainframe environment. In addition, increased staff access to benefit recipients’ data will enable staff to review medical evidence faster and process claims more accurately, among other things. According to the agency’s modernization plan, the improvements to the system should improve productivity and service to the public, as well as reduce the number of improper payments due to technician error. Error! No text of specified style in document. Appendix VIII: Comments from the Office of Personnel Management Error! No text of specified style in document. Appendix IX: Comments from the Small Business Administration Error! No text of specified style in document. Error! No text of specified style in document. Error! No text of specified style in document. Error! No text of specified style in document. Appendix XIII: GAO Contact and Staff Acknowledgments Error! No text of specified style in document. In addition to the contact name above, the following staff made key contributions to this report: Dave Powner (Director), Kevin Walsh (Assistant Director), Jessica Waselkow (Assistant Director), Chris Businsky, Rebecca Eyler, Angel Ip, and Meredith Raymond.", "summary": "The federal government plans to spend over $90 billion in fiscal year 2019 on IT. About 80 percent of this amount is used to operate and maintain existing IT investments, including aging (also called legacy) systems. As they age, legacy systems can be more costly to maintain, more exposed to cybersecurity risks, and less effective in meeting their intended purpose. GAO was asked to review federal agencies' legacy systems. This report (1) identifies the most critical federal legacy systems in need of modernization and evaluates agency plans for modernizing them, and (2) identifies examples of legacy system modernization initiatives that agencies considered successful. To do so, GAO analyzed a total of 65 legacy systems in need of modernization that 24 agencies had identified. Of these 65, GAO identified the 10 most in need of modernization based on attributes such as age, criticality, and risk. GAO then analyzed agencies' modernization plans for the 10 selected legacy systems against key IT modernization best practices. The 24 agencies also provided 94 examples of successful IT modernizations from the last 5 years. In addition, GAO identified other examples of modernization successes at these agencies. GAO then selected a total of five examples to highlight a mix of system modernization types and a range of benefits realized. This is a public version of a sensitive report that is being issued concurrently. Information that agencies deemed sensitive has been omitted. Among the 10 most critical legacy systems that GAO identified as in need of modernization (see table 1), several use outdated languages, have unsupported hardware and software, and are operating with known security vulnerabilities. For example, the selected legacy system at the Department of Education runs on Common Business Oriented Language (COBOL)—a programming language that has a dwindling number of people available with the skills needed to support it. In addition, the Department of the Interior's system contains obsolete hardware that is not supported by the manufacturers. Regarding cybersecurity, the Department of Homeland Security's system had a large number of reported vulnerabilities, of which 168 were considered high or critical risk to the network as of September 2018. Of the 10 agencies responsible for these legacy systems, seven agencies (the Departments of Defense, Homeland Security, the Interior, the Treasury; as well as the Office of Personnel Management; Small Business Administration; and Social Security Administration) had documented plans for modernizing the systems (see table 2). The Departments of Education, Health and Human Services, and Transportation did not have documented modernization plans. Of the seven agencies with plans, only the Departments of the Interior and Defense's modernization plans included the key elements identified in best practices (milestones, a description of the work necessary to complete the modernization, and a plan for the disposition of the legacy system). Until the other eight agencies establish complete modernization plans, they will have an increased risk of cost overruns, schedule delays, and project failure. In the sensitive report, GAO is making a total of eight recommendations—one to each of eight agencies—to ensure that they document modernization plans for the selected legacy systems. The eight agencies agreed with GAO's findings and recommendations, and seven of the agencies described plans to address the recommendations.", "document_type": "gao"}
{"report": "As we reported in October 2019, the federal government has invested in projects that may enhance climate resilience but does not have a strategic approach for investing in high-priority climate resilience projects. Some federal agencies have made individual efforts to manage climate change risk within existing programs and operations, and these efforts may convey climate resilience benefits. For example, the U.S. Army Corps of Engineers’ civil works program constructs flood control projects, such as sea walls, that could convey climate resilience benefits by protecting communities from storms that may be exacerbated by climate change. However, even with individual agency efforts, federal investment in projects specifically designed to enhance climate resilience to date has been limited. As we stated in our Disaster Resilience Framework, most of the federal government’s efforts to reduce disaster risk are reactive, and many revolve around disaster recovery. As a result, we reported in October 2019 that additional strategic federal investments may be needed to manage some of the nation’s most significant climate risks because climate change cuts across agency missions and presents fiscal exposures larger than any one agency can manage. Our analysis shows the federal government does not strategically identify and prioritize projects to ensure they address the nation’s most significant climate risks. In addition, our October 2019 report discusses our past work that shows an absence of government-wide strategic planning for climate change. For example, in our March 2019 update to our high-risk list, we reported that one area of government-wide action needed to reduce federal fiscal exposure is in the federal government’s role as the leader of a strategic plan that coordinates federal efforts and informs state, local, and private- sector action. For this 2019 update, we assessed the federal government’s progress since 2017 related to climate change strategic planning against five criteria and found that the federal government had not met any of the criteria for removal from the high-risk list. Specifically, since our 2017 high-risk update, four ratings regressed to “not met” and one remained unchanged as “not met.” Also, although we have made 17 recommendations that address improving federal climate change strategic planning, as of August 2019, no action had been taken toward implementing 14 of those recommendations—including one dating from 2003. Our enterprise risk management framework calls for reviewing risks and selecting the most appropriate strategy to manage them. However, no federal agency, interagency collaborative effort, or other organizational arrangement has been established to implement a strategic approach to climate resilience investment that includes periodically identifying and prioritizing projects. Such an approach could supplement individual agency climate resilience efforts and help target federal resources toward high-priority projects. Six key steps provide an opportunity for the federal government to strategically identify and prioritize climate resilience projects for investment, based on our review of reports (including a National Academies report and the U.S. Global Change Research Program’s Fourth National Climate Assessment) that discuss adaptation as a risk management process, as well as on international standards, our past work (including our enterprise risk management criteria), and interviews with stakeholders. The six key steps are (1) defining the strategic goals of the climate resilience investment effort and how the effort will be carried out, (2) identifying and assessing high-risk areas for targeted resilience investment, (3) identifying potential project ideas, (4) prioritizing projects, (5) implementing high-priority projects, and (6) monitoring projects and climate risks. (See fig. 1.) In our October 2019 report, we used one domestic and one international example to illustrate these key steps: Louisiana’s Coastal Protection and Restoration Authority (CPRA) coastal master planning effort and Canada’s Disaster Mitigation and Adaptation Fund (DMAF). In the domestic example, to address the lack of strategic coordination, in 2005 the state of Louisiana consolidated coastal planning efforts previously carried out by multiple state entities into a single effort, led by CPRA. CPRA periodically identifies high-priority coastal resilience projects designed to address two primary risks: flooding and coastal land loss. To identify potential projects, CPRA sought project proposals from citizens, nongovernmental organizations, and others. To prioritize projects, CPRA used quantitative modeling to estimate project outcomes under multiple future scenarios of varied climate and other conditions and coordinated with stakeholders to understand potential project impacts. CPRA has published three coastal master plans in which it identified and evaluated potential projects. For example, in its 2017 Comprehensive Master Plan for a Sustainable Coast, CPRA identified $50 billion in high- priority projects to be implemented as funding becomes available. In the international example, in 2018 the Canadian government launched the DMAF, a financial assistance program, to provide $1.5 billion (in U.S. dollars) over 10 years for large-scale, nationally significant projects to manage natural hazard risks, including those triggered by climate change. Infrastructure Canada, the entity responsible for administering the DMAF, seeks project ideas from provinces and territories, municipal and regional governments, indigenous groups, and others. These entities apply directly to Infrastructure Canada for funding. According to Canadian officials, two committees of experts—one composed of experts from other federal departments and the other composed of nonfederal experts (e.g., urban planners and individuals with regional expertise)—provide feedback on potential projects. These projects are prioritized based on multiple criteria such as the extent to which they reduce the impacts of natural disasters. As we reported in October 2019, on the basis of our review of relevant reports and our past work, interviews with stakeholders, and illustrative examples, we identified two options—each with strengths and limitations—for focusing federal funding on high-priority climate resilience projects. The options are (1) coordinating funding provided through multiple existing programs with varied purposes and (2) creating a new federal funding source specifically for investment in climate resilience. In addition, our analysis of these sources identified opportunities to increase the climate resilience impact of these two funding options. A strength of coordinating funding from existing sources is access to multiple funding sources for a project. For example, one stakeholder we interviewed whose community used federal funding to implement large- scale resilience projects said that having multiple programs is advantageous because when funding from one program is not available— such as when the project does not match that program’s purpose or when there are insufficient funds—funds could be sought from another program. The state of Louisiana’s coastal master planning effort also uses multi-program coordination to fund projects. Specifically, funding for high-priority resilience projects identified in the master plan is provided via several federal and nonfederal programs designed for wetlands restoration, hurricane risk reduction, oil spill recovery, and community development, among other purposes. A limitation of that option, according to CPRA officials, is that coordinating funding from multiple sources could be administratively challenging and could require dedicated staff to identify programs, assess whether projects meet program funding criteria, apply for funds, and ensure program requirements are met. Alternatively, one strength of creating a new federal funding source, such as a federal financial assistance program that could provide loans or grants or a climate infrastructure bank, is that it could encourage cross- sector projects designed to achieve benefits in multiple sectors. For example, according to one stakeholder, such a funding source could allow experts from multiple sectors—such as infrastructure, housing, transportation, and health—to collaborate on projects, leading to more creative, comprehensive approaches to enhance community resilience. However, such a new funding source would have to be created, which would require congressional authorization. In addition, we identified opportunities to increase the climate resilience impact of federal funding options based on our review of our past work, related reports, an international standard, and the Louisiana and Canadian examples, as well as interviews with stakeholders: Using both existing and new funding options. Several stakeholders told us that using both funding options—multiple, existing federal programs with varied purposes and a new funding source for high-priority climate resilience projects—in a strategic, coordinated way could help increase the impact of federal investment. Two stakeholders told us that in practice, multiple, existing federal funding sources that are not specific to climate resilience could be coordinated to fund projects when their purposes and rules align and adequate funding is available. A funding source specifically for climate resilience could be used to fund proposed projects when no related program exists or when existing programs do not have sufficient funding available, according to these and other stakeholders. Helping ensure adequate and consistent funding. Several stakeholders we interviewed identified the need for adequate and consistent funding to implement high-priority climate resilience projects. For example, according to one stakeholder we interviewed, inconsistent, inadequate funding makes it difficult to complete large- scale projects and can lead to additional costs if significant delays occur during which existing work deteriorates. In addition to adequate and consistent funding, funding options should be designed to accommodate long-term projects since high-priority climate resilience projects can take multiple years to design and implement, according to two stakeholders we interviewed. Encouraging nonfederal investment. Several stakeholders we interviewed told us that the federal government could use a federal climate resilience investment effort to encourage nonfederal investment in high-priority climate resilience projects, thereby increasing the impact of federal investment. For example, several stakeholders identified the importance of a cost-share component so that funding recipients are invested in a project’s success. Canada’s DMAF encourages nonfederal investment by partially funding projects of national significance and requiring different levels of cost-share from funding recipients, ranging from 25 percent for indigenous recipients to 75 percent for private-sector and other for-profit recipients. Several stakeholders also identified potential funding mechanisms—for example, public-private partnerships and loan guarantees—that could leverage federal dollars to encourage additional investment in climate resilience projects by nonfederal entities, including the private sector. Encouraging complementary resilience activities. To increase the impact of federal investment in climate resilience, a federal investment effort presents an opportunity to encourage complementary resilience activities by nonfederal actors such as states, localities, and private- sector partners, based on interviews with several stakeholders, the Canadian example, and reports we reviewed. For example, this could include establishing conditions that funding recipients must meet in exchange for receiving federal funding. Alternatively, the federal government could use incentives (e.g., providing greater federal cost- share or giving additional preference in the project prioritization process) to encourage complementary resilience activities by nonfederal actors. Our Disaster Resilience Framework states that incentives can make long-term, forward-looking risk reduction investments more viable and attractive among competing priorities. The federal government could use these conditions and incentives to encourage several types of complementary resilience activities by nonfederal actors. For example, the federal government could encourage the use and enforcement of building codes that require stronger risk-reduction measures. In addition, a federal investment effort could provide an opportunity to encourage communities to limit or prohibit development in high-risk areas to minimize risks to people and assets exposed to future climate hazards. One example of this would be through zoning regulations. Another stakeholder suggested that communities receiving federal funding for resilience projects should be adequately insured against future climate risks so they have a potential source of funding for rebuilding in the event of a disaster. Allowing funds to be used at various stages of project development. Several stakeholders suggested that federal funds be used for multiple stages of project development—such as project design, implementation, or monitoring—to increase the impact of federal funds. For example, two stakeholders we interviewed told us that resilience projects can require significant amounts of design work to develop an implementable and effective project concept and that making funds available for project design could improve the quality of project proposals, thereby maximizing the impact of federal funds. In addition to providing federal funds for project design, one stakeholder suggested making federal funding available to measure project outcomes (e.g., how effectively projects increased resilience) to improve future decisions by both the federal government and others making resilience investments. Based on the findings of our October 2019 report, we recommended that Congress consider establishing a federal organizational arrangement to periodically identify and prioritize climate resilience projects for federal investment. Such an arrangement could be designed using the six key steps for prioritizing climate resilience investments and the opportunities to increase the climate resilience impact of federal funding options that we identified in our report. Chairwoman Castor, Ranking Member Graves, and Members of the Select Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Mark Gaffigan at (202) 512-3841 or gaffiganm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff members who made key contributions to this testimony and the underlying report are Joseph “Joe” Thompson (Assistant Director), Celia R. Mendive (Analyst in Charge),Taiyshawna Battle and Paige Gilbreath. Also contributing to this report were Alicia Puente Cackley, Colleen M. Candrl, Kendall Childers, Steven Cohen, Christopher Curry, Cindy Gilbert, Kathryn Godfrey, Holly Halifax, Carol Henn, Susan Irving, Richard Johnson, Gwendolyn Kirby, Joe Maher, Gregory Marchand, Diana Maurer, Kirk Menard, Tim Persons, Caroline N. Prado, William Reinsberg, Oliver Richard, Danny Royer, Jeanette Soares, Kiki Theodoropoulos, Sarah Veale, Patrick Ward, Jarrod West, Kristy Williams, Eugene Wisnoski, and Melissa Wolf. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Since 2005, federal funding for disaster assistance has totaled at least $450 billion, including a 2019 supplemental appropriation of $19.1 billion for recent disasters. In 2018 alone, 14 separate billion-dollar weather and climate disaster events occurred across the United States, with total costs of at least $91 billion, including the loss of public and private property, according to the National Oceanic and Atmospheric Administration. Disaster costs likely will increase as certain extreme weather events become more frequent and intense due to climate change, according to the U.S. Global Change Research Program, a global change research coordinating body that spans 13 federal agencies. In 2013, GAO included “Limiting the Federal Government’s Fiscal Exposure by Better Managing Climate Change Risks” on its high-risk list. The cost of recent weather disasters has illustrated the need to plan for climate change risks and invest in climate resilience, which can reduce the need for far more costly steps in the decades to come. This statement summarizes GAO’s findings from its October 2019 report on climate resilience and federal investment (GAO-20-127). Specifically, it focuses on (1) the extent to which the federal government has a strategic approach for investing in climate resilience projects; (2) key steps that provide an opportunity to strategically prioritize projects for investment; and (3) the strengths and limitations of options for focusing federal funding on these projects. To perform this work, GAO reviewed about 50 relevant reports and interviewed 35 stakeholders with expertise in climate resilience and related fields, including federal officials, researchers, and consultants. GAO also identified domestic and international examples of governments that invest in climate resilience and related projects. The federal government has invested in individual projects that may enhance climate resilience, but it does not have a strategic approach to guide its investments in high-priority climate resilience projects. In GAO’s March 2019 update to its list of federal programs at high risk for fraud, waste, abuse, and mismanagement, or most in need of transformation, GAO reported that one area of government-wide action needed to reduce federal fiscal exposure is in the federal government’s role as the leader of a strategic plan that coordinates federal efforts and informs state, local, and private-sector action. For this 2019 update, GAO assessed the federal government’s progress since 2017 related to climate change strategic planning against five criteria and found that the federal government had not met any of the criteria for removal from the high-risk list. Further, as of August 2019, no action had been taken to implement 14 of GAO’s 17 recommendations to improve federal climate change strategic planning. Additionally, no federal agency, interagency collaborative effort, or other organizational arrangement has been established to implement a strategic approach to climate resilience investment that includes periodically identifying and prioritizing projects. Such an approach could supplement individual agency climate resilience efforts and help target federal resources toward high-priority projects. Based on its review of prior GAO work, relevant reports, and stakeholder interviews, GAO found six key steps that provide an opportunity for the federal government to strategically identify and prioritize climate resilience projects for investment. These are (1) defining the strategic goals of the climate resilience investment effort and how the effort will be carried out, (2) identifying and assessing high-risk areas for targeted resilience investment, (3) identifying potential project ideas, (4) prioritizing projects, (5) implementing high-priority projects, and (6) monitoring projects and climate risks. GAO also identified two options—each with strengths and limitations—for focusing federal funding on high-priority climate resilience projects. The options are (1) coordinating funding provided through multiple existing programs with varied purposes and (2) creating a new federal funding source specifically for investment in climate resilience. In addition, GAO identified opportunities to increase the impact of federal funding options on climate resilience, including ensuring adequate and consistent funding and encouraging nonfederal investment in climate resilience. Congress should consider establishing a federal organizational arrangement to periodically identify and prioritize climate resilience projects for federal investment. Such an arrangement could be designed using the six key steps for prioritizing climate resilience investments and the opportunities to increase the climate resilience impact of federal funding options that GAO identified in its October 2019 report.", "document_type": "gao"}
{"report": "The Secretary of Defense may enter into ACSAs with authorized countries and international organizations for the reciprocal provision of logistic support, supplies, and services with the military forces of that country or international organization. DOD describes ACSAs as bilateral agreements that allow exchanges of logistic support, supplies, and services between the United States and partners in return for reimbursement in the form of cash or the reciprocal provision of support. As of February 2020, DOD had signed 125 ACSAs, including five that had expired, which span DOD’s six geographic areas of responsibility identified in table 1. For a full list of past and present ACSA partners, see appendix II. According to DOD, it uses ACSAs primarily during wartime, combined exercises, training, deployments, contingency operations, humanitarian or foreign disaster relief operations, certain peace operations under the United Nations Charter, or for unforeseen or exigent circumstances. For example, ACSAs can give a commander increased flexibility to address logistical shortfalls in a contingency environment. DOD officials noted that the agreements provide DOD with flexibility, enhanced readiness at minimal cost, and increased military effectiveness by allowing partners and allies to access U.S. logistics capabilities and practice mutual support procedures, which is particularly valuable in planning international exercises and coalition operations. For example, DOD established ACSAs with 70 new partners during Operations Enduring Freedom and Iraqi Freedom, which together covered the 14 years from 2001 through 2014. DOD signed an additional 15 ACSAs from 2015 through February 2020. Figure 1 shows the cumulative growth in the number of ACSAs over time. Under 10 U.S.C. §2342, DOD is authorized to enter into ACSAs with governments of NATO countries, subsidiary bodies of NATO, and international organizations. DOD can also enter into ACSAs with governments of non-NATO countries, but must first designate the country eligible for an ACSA by following a process that includes consulting with State, determining that the designation is in the interests of national security, and notifying Congress of its intent to make the designation. Within DOD, the OUSD (A&S) is the focal point for establishing ACSAs, as of December 2019, and officials from that office request State’s authority to negotiate an ACSA and coordinate designees with DOD— typically Combatant Command staff—to negotiate and sign ACSAs. DOD officials told us that the amount of time it takes to negotiate and sign an ACSA varies because of a number of factors. For example, a lack of urgency or the complicated legal context of a potential partner can extend negotiations. As a result, the amount of time it takes to negotiate and sign an ACSA has varied greatly, from less than 1 year to more than 25 years. After an agreement is signed, State is required to notify Congress about international agreements that enter into force, including ACSAs. Although, according to agency documentation, most ACSAs enter into force at the time they are signed, an ACSA may enter into force on a later date, depending on the conditions outlined in each agreement. According to State officials, ACSAs, like some other international agreements, may be applied provisionally (the agreement has been signed and transactions may be executed) prior to entering into force. Figure 2 illustrates the process by which DOD and State generally establish new ACSAs. The Secretary of Defense generally delegates the responsibilities of managing ACSA implementation to various components including the OUSD (A&S), Chairman of the Joint Chiefs of Staff (CJCS), defense agencies, military departments and service components, Combatant Commands, and subordinate unified commands. Responsibilities and procedures for implementing ACSA transactions are set forth in DOD guidance and regulations including CJCS Instruction (CJCSI) 2120.01D, DOD Directive 2010.9, and DOD’s Financial Management Regulation. For example, CJCSI 2120.01D calls for military departments and defense agencies to appoint primary ACSA program managers charged with maintaining financial and program records of all ACSA transactions. In addition to the primary guidance documents, DOD policy and legislation have modified ACSA implementation over time. For example, DOD issued memorandums in 2017, 2018, and 2019 to update or clarify requirements for managing ACSAs, and in October 2018, officials noted that DOD had begun a process to update each of the three primary guidance documents listed above. In addition, the NDAA for Fiscal Year 2020 was enacted on December 20, 2019, and Section 1203 modified the authorities related to ACSAs. The law includes a number of new requirements, including a requirement for the Secretary of Defense to designate an official who will have primary responsibility for overseeing and monitoring the implementation of ACSAs in coordination with the Under Secretary of Defense for Policy. Further, the law requires that, among other things, the Secretary of Defense shall prescribe regulations to ensure that adequate processes and controls are in place to provide for the accurate accounting of logistic support, supplies, and services received or provided under ACSAs. The legislation also instituted a new congressional notification requirement that DOD may not enter into an ACSA without notifying the appropriate congressional committees of its intent to do so at least 30 days in advance. DOD uses AGATRS as its system of record to create, track, and manage transactions executed under ACSAs. CJCSI 2120.01D requires the use of AGATRS to fully document all ACSA transfers of logistic support, supplies, and services. DLA has managed AGATRS since 2013, when, according to DLA officials, an updated version of the system was launched and historical data archived. As of November 2019, AGATRS included records of more than 31,000 ACSA sales and acquisitions orders authorized from fiscal years 2014 through 2019. According to DOD officials, AGATRS is the best source of automated information on ACSA transactions. According to DOD, it authorized more than 22,000 ACSA sale orders from October 2013 through September 2019 that provided approximately $5 billion of logistic support, supplies, and services for items ranging from water and fuel to bullets and munitions. Figure 3 shows examples of the types of support provided through ACSAs. According to AGATRS, more than 70 different DOD components executed ACSA order sales or acquisitions from October 2013 through September 2019. However, the seven components shown in table 2 accounted for about 92 percent of the reported total value and about 79 percent of the reported order volume. In addition to direct transactions, the retransfer of support may also occur under ACSAs. CJCSI 2120.01D describes these retransfers as transfers from the original recipient to another foreign government or international organization, or to any entity other than the officers, employees, or agents of the foreign country or international organization whose military originally received the logistic support, supplies, or services. DOD Directive 2010.9 prohibits the retransfer of ACSA support without the prior written consent of the U.S. government. DOD records indicate that it approved 11 ACSA retransfers with six different partners from 2003 through 2019. These approvals, listed in appendix III, involved at least 15 final foreign recipients. Eight of these recipients did not have an ACSA at the time of DOD’s authorization for a retransfer. For example, before DOD signed an ACSA with Saudi Arabia in 2016, DOD authorized a retransfer of general purpose bombs from the United Arab Emirates to Saudi Arabia to support its activities in Yemen. In August 2018, Congress amended 10 U.S.C. §2342 to prohibit DOD from using an ACSA to facilitate the transfer of logistic support, supplies, and services to a final recipient that has not signed an ACSA with DOD. DOD is responsible for providing information to Congress regarding its intent to designate non-NATO countries for an ACSA. Specifically, under 10 U.S.C. §2342, DOD must notify Congress of its intent to designate the government of a non-NATO country for an ACSA at least 30 days before making the designation. Of the 125 ACSAs DOD had signed as of February 2020, 21 were agreements with NATO countries and international organizations, which do not require congressional notification. For the remaining 104 agreements signed with the governments of non-NATO countries, DOD should have notified Congress at least 30 days before designating each country eligible for an ACSA. DOD records indicate that DOD transmitted notifications of its intent to designate at least 78 of the 104 countries as eligible for ACSAs. For these 78 ACSAs, we confirmed that notifications to Congress were dated on time, that is, at least 30 days before DOD signed the relevant agreements. However, as shown in figure 4, DOD did not have records of 26 of the 104 agreements for which DOD should have notified Congress, so we could not confirm whether the notifications had occurred. DOD estimates that these 26 notifications would have occurred between 1993 and 2009, with 20 being before or during 1996. According to DOD officials, DOD’s ACSA recordkeeping procedures are not documented and have changed over time, which contributes to gaps in DOD notification records. DOD officials told us that while they had endeavored to save notifications and signed agreements, they had not systematically tracked notifications for each partner. Neither DOD Directive 2010.9 nor CJCSI 2120.01D specifically call for DOD to track ACSA signature or congressional notification transmittal dates, but DOD officials noted that recordkeeping procedures such as scanning and maintaining documents should be part of commonly understood proper administration practices. In addition, several different DOD offices have been responsible for various aspects of ACSA management over the years. Each office, according to DOD officials, may have had different recordkeeping practices, including some that predated electronic records. Further, DOD officials had difficulties finding paperwork from offices not currently involved with ACSAs and those that no longer exist. Poor recordkeeping has affected DOD’s ability to provide Congress with full and accurate information about ACSAs. For example, DOD’s January 2019 report to Congress on ACSA activities included inaccurate and incomplete information on notification and signature dates, including some for which DOD did not have documentation. DOD included estimated Congressional notification transmittal dates for the agreements for which it could not locate supporting documentation. Moreover, DOD included incorrect ACSA signature dates in the report for 16 other agreements. DOD officials responsible for compiling the report told us that they made some of these errors because they used the inaccurate data available at the time. In November 2019, DOD officials told us that they intended to create a consolidated list of ACSA partners including the date of eligibility designations and agreement signatures for each partner to be kept updated through a joint effort by OUSD (A&S) and the Joint Staff. As of January 2020, DOD had not formalized these intentions in written guidance. Documenting and implementing recordkeeping procedures would help ensure that DOD can report accurate and complete information to Congress. While DOD is required to notify Congress about non-NATO partner eligibility for ACSAs, under 1 U.S.C. §112b (commonly referred to as “the Case-Zablocki Act”), State is required to notify Congress when any international agreement to which the United States is a party, other than a treaty, enters into force. Under the Case-Zablocki Act, State is required to provide this notification as soon as practicable after the agreement has entered into force, but in no event later than 60 days thereafter. In addition, the law requires any department or agency of the U.S. government that enters into any international agreement on behalf of the United States to transmit the text of such an agreement to State no later than 20 days after such agreement has been signed. Of the 125 signed ACSAs, State and DOD officials confirmed that, as of February 2020, 118 had entered into force and, as such, required State notification to Congress. State’s Office of the Assistant Legal Advisor’s Office of Treaty Affairs is responsible for receiving texts of signed international agreements from the agencies that signed them, for recordkeeping associated with such agreements, and for transmitting the texts of such agreements to Congress in accordance with the Case-Zablocki Act. As of February 2020, records for the 118 ACSAs that had entered into force indicate that State’s notifications to Congress for 68 (or 58 percent) were dated within 60 days, as required. However, 48 (or 41 percent) of the 118 notifications were late, that is, dated more than 60 days after entry into force, as shown in figure 5. According to agency records, these 48 agreements entered into force between 1995 and 2019. For two agreements that entered into force in 1983 and 2002, State records are insufficient to determine whether or not State notified Congress. For most of the 48 State notifications dated after the 60-day deadline, State attributed the delays to untimely DOD delivery of required information to State. Specifically, 32 (or 74 percent) of the 43 late notifications that included a reason for delayed transmittal attributed the cause to DOD elements having provided late or incomplete agreement information to State’s Treaty Office. As described above, because DOD enters into ACSAs on behalf of the United States, it must provide State the text of the agreements no later than 20 days after signing or otherwise concluding such an agreement, to facilitate State’s required notifications to Congress. However, DOD officials confirmed that they provided information on some ACSAs to State more than 20 days after signature. DOD officials and our analysis identified multiple causes that contributed to DOD’s providing information on newly signed ACSAs to State after the 20-day deadline: Procedural complications. Procedural complications can affect DOD’s ability to provide information to State within 20 days. For example, DOD officials noted that the standard DOD process to send a memo to State sometimes takes more than 20 days to complete. Further, for some agreements, DOD provided some information to State within 20 days, but did not include one or more necessary elements—such as a language certification if the agreement was signed in a language other than English—to determine whether such an agreement had been concluded. DOD officials told us that a significant amount of time can pass before they compile all the information State needs from DOD, resulting in State’s inability to send notifications within 60 days of entry into force, as required. Lack of experience. DOD officials told us that the relevant DOD officials had overlooked the responsibility to send information to State about newly signed ACSAs, at times because of a lack of experience. For example, they explained that DOD missed the 20-day deadline to send information to State about the 2017 ACSA signing with Mexico because it had been 10 years since officials from DOD’s Northern Command had negotiated an ACSA, and the officials had overlooked the requirement. Regarding two ACSAs about which State had not notified Congress as of September 2019, State officials told us they did not know those agreements had entered into force until we asked about their status. Subsequently, State notified Congress about one of these agreements in October 2019. For the second, as of February 2020, DOD had begun providing related information to State, and State was continuing to review related documentation to confirm that the agreement had entered into force. Inconsistent guidance. Our review of DOD’s guidance found inconsistent language describing when DOD should provide information to State about new ACSAs that could affect DOD’s transfer of such information. Specifically, the CJCSIs on international agreements and ACSAs note that DOD should provide State with information on new ACSAs no later than 20 days after an agreement is signed. However, DOD Directives on international agreements and ACSAs indicate that the relevant deadline is no later than 20 days after an agreement enters into force, which can be days or years after an ACSA is signed. DOD officials noted that the officials who drafted the guidance may not have understood the difference between the signing and entry into force of international agreements. Limitations in training. As of December 2019, DOD’s standard online training on ACSAs did not address responsibilities to share information about newly signed agreements with State. Specifically, while DOD’s two required training courses on ACSAs include some aspects of negotiation and signing new agreements, neither mentions DOD’s responsibility to report signed ACSAs to State. According to DOD officials, the requirement may be included during in-person training conducted by personnel from DOD’s Office of General Counsel for DOD’s combatant command officials. Congress depends on State and DOD for information to oversee the use of ACSAs, which DOD officials have cited as important tools for furthering national security interests, particularly involving activities with broad coalitions. Without timely notification of entry into force, Congress will not have full information about countries and international organizations to and from which DOD can and may already be using ACSAs to transfer logistic support, supplies, and services. CJCS Instruction 2120.01D contains policy and procedural guidance concerning the use of ACSA authorities, and addresses, among other things, maintenance of ACSA transaction orders. Specifically, the instruction establishes AGATRS as DOD’s system of record for the Joint Staff, Combatant Commands, and the Military Services to manage ACSA orders; describes processes to execute an ACSA order; and notes that AGATRS will be used to fully record all transfers of ACSA support, including documentation such as invoices. Additionally, federal standards for internal control state that management should use quality information to make informed decisions and achieve agency objectives. Quality information is defined as information that is accurate, complete, and provided on a timely basis, among other attributes, and should include relevant data obtained from reliable sources. However, based on our analysis of a generalizable sample of orders, we found that DOD’s ACSA system of record lacked quality data to track the status of ACSA order reimbursement. First, we found that DOD incorrectly recorded the reimbursement status in AGATRS of an estimated 7.3 percent of ACSA orders authorized from October 2013 through March 2018. For example, DOD recorded three of the 227 orders in our sample as completed, even though it had not received full reimbursement for them—including at least one order that it had ceased processing. DOD records included five orders recorded as incomplete despite having received full reimbursement. We also identified six orders that DOD either improperly categorized as ACSA transactions or orders that DOD should have cancelled because the related transaction never took place or was a duplicate. Second, DOD could not determine the reimbursement status of an estimated 12.2 percent of ACSA orders authorized from October 2013 through March 2018 in AGATRS. Based on our generalizable sample, DOD would not be able to locate records to verify the status of reimbursement for an estimated 1,100 ACSA orders with authorization dates ranging from this time period. With regard to the reimbursement status of these orders, a DOD official noted that DOD could not determine the status based on available information. As a result, DOD does not know if the orders have been reimbursed, were processed for reimbursement, or even took place. According to DOD officials, data quality lapses occur because DOD does not have a process in place to reconcile reimbursement information with data recorded in AGATRS. Although AGATRS is DOD’s system of record for ACSA transactions, DOD officials told us that the database does not have financial processing capabilities and is not integrated with DOD’s financial processing systems. As a result, ACSA personnel must manually update information in AGATRS as orders are processed in other financial systems, but do not always do so, according to DOD officials. A DOD official told us that the military services vary greatly in the extent to which they regularly populate AGATRS, and even within a service, some personnel are better than others at including complete information. DOD officials explained that personnel may delay or fail to update information in AGATRS for multiple reasons. First, personnel may be on temporary duty in an operational environment where they do not have a secure internet connection and thus cannot upload information into AGATRS. Second, short-term rotations of personnel in the field can result in delays as new personnel learn how to use AGATRS and process transactions. Third, after negotiating the transfer of support, drafting the order, and receiving a unique order number assignment in AGATRS, ACSA orders change frequently. These changes can include price adjustments that result in DOD or the partner deciding not to move forward with the transaction, or significantly revising it. In such situations, DOD officials told us that DOD should cancel orders in AGATRS, but does not always do so. Further, DOD does not have quality data to track the extent to which DOD processes ACSA transactions in accordance with statutory requirements. Under 10 U.S.C. §2345, payment-in-kind or exchange entitlements through ACSA transactions shall be satisfied within 12 months of the date of the delivery of logistic support, supplies, or services. However, DOD officials told us that they did not have the information necessary to track such compliance because AGATRS lacks a mechanism to track these data. DOD officials explained that AGATRS has a field to record the “delivery time” for an order, but that field does not require users to enter data in a standard format. Our review of AGATRS data found instances in which users left the field blank, entered date ranges as opposed to a single date, or entered text information about the delivery, such as how quickly it should occur. DOD officials noted that they could not use the information in this field to determine the extent to which orders were reimbursed within 12 months of delivery, as outlined in the statute. Instead of using the date of delivery, DOD officials stated, and our analysis confirmed, that DOD used an order’s date of authorization as an alternate metric to indicate whether an order was reimbursed within 12 months. However, DOD has transactions in which it delivers the support weeks or months after the order is authorized, according to DOD officials. When asked about such transactions, DOD officials acknowledged that the authorization date was not an appropriate alternate date to use to determine if ACSA orders were completed within 12 months of delivery. DOD has taken some steps—including several since we began our review—to improve tracking of ACSA orders in AGATRS, such as issuing memos reiterating requirements for personnel to use AGATRS, improving the system’s functionality, and updating relevant training. For example, in October 2018, DOD introduced additional categories of order status in AGATRS to track an order’s progress through the transaction process and in June 2019, DOD updated the AGATRS training course to reflect this and other updates to the system. Additionally, in October 2019, DOD updated AGATRS to help ensure that orders are assigned to appropriate DOD organizations and personnel in the system. According to DOD officials, as of October 2019, three military services were discussing processes that could improve record keeping and tracking for ACSA orders. For instance, U.S. Army officials told us that the Army had begun reconciling data from the service’s financial accounting system with information recorded in AGATRS to address data quality issues. However, DOD has not finalized or fully implemented most of these steps, which, even if implemented, would not address historical inaccuracies in DOD’s recorded data, according to DOD officials. According to DOD, from fiscal years 2014 through 2019, DOD used ACSAs to provide support valued at about $5 billion to foreign partners. Without a process to ensure that ACSA order data are accurate and without data to track the timeliness of transactions, DOD does not have sufficient information to oversee ACSA reimbursement. Section 2344(a) of Title 10 of the United States Code provides that the United States can use ACSAs to transfer logistic support, supplies, and services to partners in return for cash reimbursement or by replacement- in-kind or exchange of supplies or services of an equal value. DOD guidance and financial management regulations outline procedures for DOD to carry out these transactions and seek timely reimbursement. Additionally, federal standards for internal control state that management should perform ongoing monitoring as part of the normal course of operations to obtain reasonable assurance about the effectiveness of its internal controls. On the basis of a generalizable sample of ACSA orders recorded in AGATRS, we estimate that DOD received reimbursement for approximately 64 percent of ACSA orders recorded in AGATRS that it authorized from October 2013 through March 2018 (about 6,000), but did not receive full reimbursement for approximately 24 percent (about 2,300), as shown in figure 6. Some orders for which DOD did not receive full reimbursement included basic life support such as food, water, housing, and fuel, authorized in 2017. Further, DOD could not verify the accuracy of the reimbursement status for an estimated 12.2 percent of orders (about 1,100) recorded in AGATRS during this time period—meaning that for these orders, DOD could not verify whether it had requested or received reimbursement, or whether the transaction had occurred. The orders in this category included, for example, helicopter transportation authorized for a partner in 2015 and valued by DOD at almost $150,000. DOD officials identified several factors that contributed to unreimbursed ACSA orders, including: Lack of invoicing. DOD officials said that DOD had not received reimbursement for 39 of the 221 ACSA orders in our sample, valued by DOD at more than $700,000, because it had not sent invoices to request reimbursement from partners. According to the officials, DOD had not processed these orders for invoicing in part because it had not assigned the orders to the appropriate officials who manage financial processing. Officials from two military services told us that while they aim to have strong communication between the personnel who manage logistics and finance processes for ACSA orders, factors such as staff rotations, contingency environments, and delayed training may affect the efficiency of order processing. DOD officials also noted that missing or incorrect order information, such as an incorrect billing address for a partner nation, may delay invoicing. Delays from partner countries. For some unreimbursed orders in our sample, DOD officials explained that DOD had sent invoices to partner countries but, as of August 2019, had not received reimbursement. The average time from the date of invoice to the date of reimbursement was 208 days for reimbursed cash transactions in our sample of 221 orders authorized from October 2013 through March 2018, and the longest amount of time was 751 days. Lack of a monitoring process. According to DOD officials, DOD did not appropriately monitor the reimbursement status of some orders in our sample and does not have a process to monitor delinquent debt. For example, DOD officials explained that they could not verify reimbursement for some orders recorded as overdue in our sample because personnel had not closely monitored the status of these orders. Additionally, in response to our inquiries, DOD acknowledged that it would need to reassign certain overdue orders to appropriate officials for processing. Although AGATRS produces reports that identify overdue orders, DOD does not have an agency-wide process to monitor and take action on unreimbursed orders that become delinquent. DOD officials told us that the Defense Finance and Accounting Service (DFAS), responsible for some ACSA billing, sends letters to partners for delinquent ACSA bills 30, 60, and 90 days after the end of the billing period outlined under the terms of the ACSA. However, after 90 days, DOD does not have a standardized approach to continue seeking delinquent ACSA debt according to DOD officials. In 2018, DOD updated the section of its Financial Management Regulation that addresses the collection of debt owed by foreign entities, but according to DOD officials, DOD had not implemented the updated policy as of October 2019. Officials from DFAS explained that the policy had not been implemented because they were working with officials from the military services to evaluate possible debt collection procedures. Unless it takes steps to ensure that it processes and invoices ACSA orders as required, and seeks unpaid debt, DOD may not receive reimbursement for thousands of orders for which it has provided support. As of November 2019, DOD indicated that the department had authorized more than $1 billion in ACSA sale orders for which reimbursement is now overdue. Seeking reimbursement for these ACSA orders and implementing oversight processes will help ensure that the United States receives reimbursement for current and future orders under the terms of these agreements. In the past 5 years, DOD has exchanged billions of dollars in reimbursable ACSA support with military forces from more than 100 partner nations and international organizations through ACSA transactions. DOD uses ACSAs to exchange logistic support, supplies, and services with partners in a variety of circumstances, including international coalition efforts, such as those combating terrorist groups in Iraq and Syria. However, weaknesses in recordkeeping and management processes limit the extent to which agencies can (1) provide Congress with information requested for oversight and (2) monitor and secure reimbursement. First, DOD could not locate records related to required congressional notifications about designating 26 countries for an ACSA. Further, State transmitted almost half of its congressional notifications on ACSA entry into force after required deadlines, largely because DOD did not provide State with information about new agreements. Without full and timely information about new partners that DOD intends to designate for an ACSA or agreements that have entered into force, Congress will not be sufficiently informed to effectively oversee DOD’s use of ACSAs as an element of security cooperation. Second, DOD lacks quality data necessary for tracking ACSA orders and has not received reimbursement for thousands of orders. Our review of 227 transactions confirmed at least $26 million of unreimbursed overdue transactions, but, as of November 2019, DOD records include additional overdue ACSA transactions for support provided to partners dating back to 2011, which DOD values at more than $1 billion. By establishing procedures to improve ACSA recordkeeping and processes to seek reimbursement, DOD can help ensure that reliable information is available for reporting and oversight of activities to secure reimbursement of hundreds of millions of dollars of support provided to our partners. We are making a total of seven recommendations to DOD: The Secretary of Defense should ensure that written ACSA guidance includes recordkeeping procedures related to ACSA congressional notifications and signature dates to help enable the provision of complete information for Congress. (Recommendation 1) The Secretary of Defense should take steps, such as updating guidance, to help ensure the implementation of requirements related to providing information to State about newly signed ACSAs. (Recommendation 2) The Secretary of Defense should take steps to verify the accuracy of ACSA order statuses recorded in DOD’s system of record, and make corrections as appropriate. (Recommendation 3) The Secretary of Defense should implement a process to reconcile data in financial systems with the data and associated documents collected and stored in DOD’s ACSA system of record on a periodic basis. (Recommendation 4) The Secretary of Defense should develop and implement a mechanism to record and track the extent to which it is meeting required time frames to receive reimbursement for ACSA orders. (Recommendation 5). The Secretary of Defense should take steps to improve invoicing of ACSA orders, such as identifying ACSA orders recorded in DOD’s system of record that have not been invoiced and sending invoices to partner countries. (Recommendation 6) The Secretary of Defense should implement a process to monitor ACSA orders recorded as overdue in DOD’s system of record, and take steps to resolve outstanding reimbursements, as appropriate. (Recommendation 7) We provided a draft of this report to DOD and State for comment. In its comments, reproduced in appendix V, DOD concurred with the seven recommendations directed to it. DOD also provided information about actions it has taken to address recommendations 1 and 2. With respect to recommendation 1, DOD provided a copy of a February 2020 memorandum that outlines procedures to capture and preserve information about ACSA establishment, including the dates of DOD’s congressional notifications of intent to designate countries for ACSAs and agreement signature dates. With respect to recommendation 2, DOD provided a copy of a February 2020 memorandum that issued guidance related to DOD’s provision of ACSA information to State for State’s congressional notifications under the Case-Zablocki Act. We plan to follow up with DOD to learn about the distribution of these memoranda. State provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretaries of Defense and State, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6881 or bairj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Senate Report 115-262, accompanying a bill for the National Defense Authorization Act (NDAA) for Fiscal Year 2019, includes a provision for us to review several aspects of Acquisition and Cross-Servicing Agreement (ACSA) management, including information provided to Congress and Department of Defense (DOD) tracking of support and receipt of reimbursement. In this report, we examine the extent to which (1) agencies have provided information to Congress about ACSAs, and (2) DOD has tracked and received reimbursement for ACSA orders. To address these objectives, we reviewed legal authorities related to ACSAs in sections 2341-2350 of Title 10 of the United States Code, DOD policy and guidance on ACSA management and implementation, and DOD Inspector General (IG) reporting on DOD’s management of ACSAs. We analyzed DOD and Department of State (State) documentation related to congressional notifications and the establishment of ACSAs, DOD ACSA transaction data, and DOD’s Report to Congress Concerning Acquisition and Cross-servicing Activities for Fiscal Year 2018. We also discussed ACSA management, order tracking, and transactions, including for the Saudi-led Coalition, with DOD officials from the Air Force Central Command (AFCENT); Defense Finance and Accounting Services; Defense Logistics Agency (DLA), including DLA Energy; Office of the Chairman of the Joint Chiefs of Staff (OCJCS); Office of the Undersecretary of Defense for Acquisition and Sustainment (OUSD (A&S)); U.S. Air Force; U.S. Marine Corps; U.S. Army; and U.S. Central Command. With State officials from the Bureau of Political-Military Affairs and the Office of the Legal Adviser’s Office of Treaty Affairs, we discussed the process to establish international agreements, State’s notifications to Congress on ACSA entry into force, and recordkeeping for those notifications. We conducted fieldwork at AFCENT Headquarters at Shaw Air Force Base in Sumter, South Carolina, to discuss ACSA transactions related to support provided to the Saudi-led Coalition. To determine the extent to which agencies have provided information to Congress about ACSAs, we analyzed agency activities related to (1) DOD’s requirement to notify Congress of its intent to designate a country eligible for an ACSA and (2) State’s requirement to notify Congress no later than 60 days after the entry into force of international agreements, which includes ACSAs. First we reviewed DOD’s congressional notification requirements under 10 U.S.C. §2342. The law authorizes the Secretary of Defense to sign ACSAs with the governments of North Atlantic Treaty Organization (NATO) countries, subsidiary bodies of NATO, and the United Nations Organization or any regional international organizations without an official designation of eligibility. However, for countries that are not members of NATO, DOD must notify Congress of its intent to designate the government of a country eligible for an ACSA at least 30 days before making the designation. Agency records indicate that DOD had signed 125 ACSAs as of February 2020. We included these 125 agreements in our analysis because, according to DOD, each agreement is considered to be an ACSA although some are named as other types of mutual logistics support agreements. To determine the extent to which DOD addressed requirements for notifying Congress of its intent to designate a non-NATO country for the purposes of entering into an ACSA, we conducted a content review of ACSA documents to identify signature and notification dates for each relevant ACSA, calculated the number of days between them, and compared our results to DOD’s requirement to notify Congress of its intent to make a designation not less than 30 days before a country is designated. Although DOD is required to notify Congress at least 30 days before designating non-NATO countries for the purposes of entering into an ACSA, DOD officials told us that ACSA records do not include a precise designation date for each country. Therefore, we used ACSA signature dates as a proxy for designation dates. In addition, because some ACSAs are revised and re-signed over time, we planned to compare the date on which DOD transmitted notifications to Congress with the signature date of the first ACSA signed with each partner. However, DOD officials explained that they could not readily provide the signature dates of the first ACSA signed with each partner because they purposefully expunged electronic records related to expired or replaced agreements—which would have noted signature dates—to help ensure that officials planning ACSA transactions referenced the current version of the agreement. Although DOD did not systematically track the signature dates for agreements that had been revised and re-signed, we reviewed documents related to each ACSA partner, historical treaty records, and other agency documents and found the signature date for the first agreement DOD signed with each ACSA partner. We compared NATO accession dates with these first ACSA signature dates and determined that 19 ACSA partners were members or elements of NATO at the time the relevant ACSA was signed. An additional two ACSA partners were elements of other international organizations. Therefore, we determined that DOD had signed 21 of its 125 ACSAs with governments of NATO countries, subsidiary bodies of NATO, and other international organizations, which do not require an official designation of eligibility. Under the law, DOD was required to notify Congress at least 30 days prior to designating the remaining 104 countries for an ACSA. The Secretary of Defense typically submits these notifications to the Senate Committees on Armed Services and Foreign Relations and the House Committees on Armed Services and Foreign Affairs. We included in our analysis the 78 of these 104 countries for which DOD records included a copy of a dated notification letter addressed to at least one of these four committees. For these 78 countries, we compared DOD notification dates with the signature date of the initial agreement with each partner. DOD could not provide documentation of congressional notifications for the remaining 26 partners, which we excluded from our analysis. We also interviewed DOD officials from the OCJCS and the OUSD (A&S) to discuss DOD’s congressional notification process. Second, we analyzed State’s requirement under 1 U.S.C. §112b to notify Congress no later than 60 days after the entry into force of international agreements, which includes ACSAs. Under the law, often referred to as “the Case-Zablocki Act,” State is required to notify Congress of any international agreement to which the United States is a party, other than a treaty, as soon as practicable after the agreement has entered into force, but in no event later than 60 days thereafter. To determine the extent to which State had transmitted notifications about ACSA entry into force on or before the statutory 60-day deadline, we conducted a content analysis of DOD ACSA documents and State notification records to identify relevant entry into force and State notification dates. We then calculated the number of days between them and compared our results to State’s reporting requirement under 1 U.S.C. §112b. Of the 125 ACSAs that DOD had signed, State officials confirmed that, as of February 2020, 118 had entered into force and, as such, required notification to Congress of entry into force under the Case-Zablocki Act. We excluded the remaining seven signed ACSAs from our analysis as follows. First, we excluded three agreements DOD signed with Benin, Iraq, and Uruguay that, according to State and DOD officials, had not entered into force as of February 2020, and therefore did not yet require notification under the Case-Zablocki Act. Second, we excluded two ACSAs signed with Canada and the United Kingdom, for which State officials explained that the legal arrangements governing acquisition and cross-servicing transactions are contained in government-to government “chapeau agreements” regarding defense cooperation rather than in agency-level ACSA agreements more commonly used with other partners. According to officials, these chapeau agreements are supplemented by nonbinding, agency-level implementing procedures that are not separately subject to Case-Zablocki Act reporting to Congress. Third, we excluded two agreement for which, as of February 2020, State officials were reviewing agreement documentation to confirm potential entry into force prior to notifying Congress. For one of these two agreements, if State determines the agreement to be entered into force, the date of entry into force will be retroactively dated to the date of signature, per the terms of the agreement. The retroactive entry into force date for the agreement is more than 60 days before February 2020, so if the entry into force date is confirmed, the related notification to Congress under the Case-Zablocki Act would be late as compared to the 60-day deadline. The second of these two agreements was signed on January 31, 2020. For the 118 ACSAs that had entered into force and thus required State’s notification to Congress, we compared entry into force dates with notification dates to determine the extent to which State had provided notifications on or before its 60-day deadline. State provided documentation on entry into force notifications for all but two of the 118 relevant ACSAs. For these two agreements, signed in 1983 and 2002, State had no record of related notifications, so we were unable to conclude whether or not they had occurred. For the remaining 116 agreements, State provided copies of dated congressional notifications for 113 and notification dates from its Treaty Information Management System for three notifications for which copies of the letters were unavailable. We included in our analysis notifications that were transmitted to either the President of the Senate, the Speaker of the House, or both. We compared the date of these notifications with ACSA entry into force dates we verified using ACSA agreement documentation and State notification documents. We also analyzed information in State’s notification documents to determine the causes for late transmittals. We interviewed DOD officials from the OCJCS and the OUSD (A&S), and State officials from the Bureau of Political-Military Affairs and the Office of the Legal Adviser’s Office of Treaty Affairs to discuss State’s congressional notification process. To determine the extent to which DOD has tracked and received reimbursement for support provided through ACSA orders, we analyzed a generalizable sample of ACSA orders that DOD had authorized from October 2013 through March 2018 in the ACSA Global Automated Tracking and Reporting System (AGATRS). AGATRS is DOD’s system of record for management of ACSA transactions and designates orders as overdue if reimbursement is not completed within 12 months of the order authorization date. We selected a stratified random sample of 227 orders, which were sampled from a population of 9,761 orders within the population groups in table 3. Strata in table 3 are based on a combination of four features: order total (dollar amount); order status (completed versus incomplete); document upload requirement (required versus not required); and military service. With this probability sample, each order of the study population had a nonzero probability of being included, and that probability could be computed for any order. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95-percent confidence interval (e.g., the margin of error is plus or minus 7 percentage points). This interval would contain the actual population value for 95 percent of the samples we could have drawn. We calculated our sample analysis with survey software that accounts for the sample design (stratification and weights) and appropriate subpopulation reporting group statements. We designed stratification and sample sizes based on order status and document upload requirements to ensure that the 95-percent confidence intervals of attribute estimates (e.g., percentage of orders that have proper support) had margins of error within around +/- 10 percentage points for each of the following four reporting groups, which collapse over the following strata: complete orders, incomplete orders, document upload required, and document upload not required. We also designed stratification based on military service to ensure proportionate representation of each military service in our sample within each combination of order status and document upload requirement. All of the orders in our population had been authorized for 14 months, and thus should have been repaid according to DOD’s 12-month system requirement, at the time we conducted our review of the sample from May 2019 through June 2019. For this sample, we analyzed order information and coordinated with DOD to validate the reimbursement status recorded in AGATRS. On the basis of (1) reporting from the DOD Inspector General, (2) interviews with DOD officials, (3) DOD’s use of manual entry to populate the system, and (4) our review of DOD’s use of ACSA orders to process reimbursement for unpaid transactions with members of the Saudi-led Coalition, we determined that DOD’s data in AGATRS may not be fully reliable. DOD officials explained that although AGATRS was the single repository for global ACSA transaction data, the system was not integrated with any other DOD systems and thus relied on manual entry from personnel to populate ACSA order information. As a result, we took additional steps to determine the reliability of information reported in the system. Specifically, we requested a data report from DOD of all ACSA transactions recorded in AGATRS as of May 8, 2019. We reviewed supporting documentation and information recorded in AGATRS for each ACSA order in our sample to determine whether the data in the “order status” field were accurate. For the order status “completed,” which indicates that the ACSA order has been fully reimbursed, we reviewed available information to determine whether financial collection documentation had been recorded and compared the information in these documents to the information in AGATRS. We then took steps to verify with DOD the status of orders that (1) were recorded as “completed,” but for which we had not identified any financial documentation or the documentation did not contain sufficient information to verify reimbursement, and (2) were not recorded as “completed” as of the time of our review. Of the 227 orders in our sample, 138 fit into one of these two categories. For orders that were recorded as completed but did not have sufficient supporting documentation, we requested that DOD provide additional support. For orders that were recorded as incomplete, we requested that DOD verify whether the orders had been reimbursed, given that they had been in the system longer than 12 months and were categorized as overdue in the data report provided by DOD. DOD provided feedback on and validated the reimbursement status for 101 of the 138 orders sent for follow-up. DOD did not provide a response for the remaining 37 orders. DOD identified whether orders recorded as overdue in AGATRS had been partially reimbursed, which we incorporated into our calculation of unreimbursed dollar amounts for the orders in our sample. On the basis of this validation process, we report on whether ACSA orders authorized from October 2013 through March 2018 in AGATRS had been reimbursed or not fully reimbursed as of July 10, 2019, or whether DOD did not know the reimbursement status as of October 2019. We found that approximately 7 percent of the order status information recorded in AGATRS was inaccurate. For example, three of the 227 orders in our sample that DOD had recorded as “completed” were not fully reimbursed. Five of the 227 orders in our sample that DOD had recorded as incomplete were actually reimbursed; DOD uploaded supporting documents and closed these orders in AGATRS in response to our inquiry. Additionally, as described above, six of the 227 orders should not have been included in our scope but were misclassified in DOD’s system. We also found orders under the purview of DLA Energy that were partially or fully settled (i.e., reimbursed or reconciled by netting sales and purchases between the United States and the partner nation), but whose status had not been updated in AGATRS. DLA Energy officials told us that AGATRS does not have sufficient functions to capture DLA Energy’s fuel reconciliation process, in which sales and purchases with partners may be offset through specific implementing arrangements with the partners. In some cases DLA Energy provided us with the actual amounts, including unpaid amounts, but we were unable to verify this information further. In response to our verification questions, DOD took steps to correct some of the AGATRS data inaccuracies we identified. For instance, DOD reopened (i.e., redesignated as incomplete) some orders it had recorded as completed in AGATRS but for which it had not received full reimbursement. Similarly, DOD uploaded reimbursement information for orders that it had recorded in the system as incomplete, but for which it had received reimbursement. DOD also uploaded reimbursement information in AGATRS for ACSA orders from our sample that it had recorded as completed, but for which it lacked documentation to support that it had received reimbursement. Finally, DOD settled or requested and received reimbursement for five of the ACSA orders in our sample. We found that DOD data on ACSA transactions contained weaknesses that we describe in this report. Because of these weaknesses, we only used data from our sample in developing estimates on data quality and reimbursement. We checked all of the orders in our sample, and either verified or corrected them as needed, and report any data that could not be verified. Since our probability sample with verified and corrected information is generalizable to all in-scope orders, we were able to estimate population values based on the corrected sample information. We conducted this performance audit from September 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. As of February 2020, DOD had 120 signed ACSAs that span DOD’s geographic areas of responsibility. (See table 4.) The Department of Defense (DOD) describes the retransfer of logistic support, supplies, and services provided under Acquisition and Cross- Servicing Agreements (ACSA) as a transfer from the original recipient to another foreign government or international organization, or to any entity other than the officers, employees, or agents of the foreign country or international organization whose military originally received the support. DOD Directive 2010.9 prohibits the retransfer of ACSA support without the prior written consent of the U.S. government, obtained through applicable DOD channels. As of November 2019, DOD had information related to 11 ACSA transactions made with six different ACSA partners between 2003 and 2019 for which the United States approved retransfer of ACSA support, as detailed in table 5. In 2019, we presented preliminary observations to Congress about the extent to which the Department of Defense (DOD) had provided support to and requested reimbursement from the Saudi-led Coalition (SLC), and DOD’s use of Acquisition and Cross-Servicing Agreements (ACSA) to do so. This appendix describes those observations and provides updates as appropriate with information obtained during the course of our review. In March 2018, DOD received a congressional inquiry regarding DOD’s use of ACSAs to provide support to the SLC activities in Yemen. In November 2018, DOD informed Congress about (1) the legal justification for the provision of aerial refueling assistance to the SLC, since March 2015, and (2) the status of reimbursement. DOD reported that it had failed to process and seek reimbursement for some fuel and all aerial refueling support provided to members of the SLC from March 2015 through November 2018, and that it would use the ACSA authority to request retroactive reimbursement. Additionally, as of August 2019, DOD had not received full reimbursement for general purpose bombs provided through ACSA in April 2015. According to DOD officials, a Joint Staff Execute Order signed on March 27, 2015, directed DOD to provide aerial refueling support to the SLC, if requested, and stated that the support would be provided on a reimbursable basis either through foreign military sales (FMS) or an ACSA. The order also stated that, as of March 2015, Saudi Arabia had not signed an ACSA. Further, according to DOD officials, there was no FMS case through which DOD might have provided aerial refueling to Saudi Arabia in March 2015. Aerial refueling support includes flying hours to conduct refueling and the fuel exchanged. According to DOD officials, air crews recorded aerial refueling flight hours for members of the SLC at the time they occurred, but did not record them as related to SLC activities in Yemen or process them as reimbursable FMS or ACSA transactions. For fuel provided to SLC members during aerial refueling flights at this time, DOD documented and processed some, but not all, as ACSA transactions. DOD officials identified multiple factors, including inadequate planning and insufficient understanding of guidance in the Joint Staff order, that led to a process breakdown in which DOD did not invoice and request reimbursement. Following the congressional inquiry, DOD began a review of air tanker flight hours, Air Force fuel purchases, and data from Saudi Arabia to determine aerial refueling reimbursement charges for flying hours and fuel. Based on this review, DOD identified reimbursable amounts of more than $261 million for flying hours and $37 million for fuel provided to coalition members. Using this information, DOD requested retroactive reimbursement through the ACSA authority from the United Arab Emirates (UAE) and Saudi Arabia for the flight hours and fuel not previously reimbursed. According to DOD officials, DOD is treating these transactions as third-party transfers. According to DOD documents and officials, because Saudi Arabia did not have a signed ACSA prior to June 2016, UAE agreed to reimburse the United States for transactions supporting the SLC before this date. Saudi Arabia agreed to reimburse the United States for transactions after this date. As of February 28, 2019, UAE had submitted $103.7 million in retroactive reimbursement for air tanker flight hours and $15 million for fuel. In May 2019, DOD signed an agreement with Saudi Arabia for repayment of $151 million for aerial refueling support provided from June 2016 through September 2018. DOD and Saudi Arabia agreed that Saudi Arabia would submit payments in increments over the course of 12 months, after receiving approval from the crown prince, Mohammad bin Salman, and additional leadership in Saudi Arabia. As of February 2020, Saudi Arabia had submitted payment of approximately $114 million, according to DOD documents. A balance of about $37 million for flight hours remains unreimbursed as well as $22 million for fuel. In addition to aerial refueling support, in 2015 DOD provided about $2 million of general purpose bombs to UAE for which UAE had received U.S. approval for an ACSA retransfer to Saudi Arabia for operations in Yemen. However, DOD did not record this order in the ACSA system of record as required until August 2019 and, as of September 2019, had received reimbursement in the form of reciprocal support for only two- thirds of the value of the bombs initially provided. DOD officials told us that UAE planned to provide the remaining in-kind reimbursement in September 2020. In addition to the contact named above, Biza Repko (Assistant Director), Kathryn Bolduc and Jasmine Senior (Analysts-in-Charge), Joe Carney, Debbie Chung, Martin de Alteriis, Neil Doherty, Adrian Good, Sally Newman, Cary Russell, Sonya Vartivarian, and Nicole Willems made key contributions to this report.", "summary": "According to DOD, from fiscal years 2014 through 2019, it used ACSAs to provide billions of dollars of logistic support, supplies, and services to more than 100 partner countries. For example, this support included fuel and ammunition to assist international exercises and coalition operations, among other efforts. Senate Report 115-262 included a provision for GAO to review ACSA management. This report examines the extent to which (1) agencies have provided information to Congress about ACSAs, and (2) DOD has tracked and received reimbursement for ACSA orders. GAO conducted content analysis of DOD and State ACSA documents, and analyzed a generalizable sample of ACSA orders authorized from October 2013 through March 2018 and recorded in DOD's system of record for ACSA orders. An ACSA order, also referred to as a transaction, documents an exchange of support between the United States and a foreign partner. In addition, GAO interviewed agency officials and conducted fieldwork at Shaw Air Force Base in Sumter, South Carolina. While generally providing required information to Congress, poor recordkeeping by the Department of Defense (DOD) and late notifications by the Department of State (State) have limited the accuracy and timeliness of information provided to Congress on acquisition and cross-servicing agreements (ACSA). DOD and State have Congressional notification requirements pertaining to ACSAs—agreements through which DOD exchanges logistic support, supplies, and services with foreign partners in return for cash or in-kind reimbursement. Documents indicate that DOD provided notice to Congress before designating 78 of 104 countries eligible for an ACSA. However, DOD did not have records for the remaining 26, in part because it lacks documented recordkeeping procedures. While State generally notified Congress about ACSAs' entry into force, it transmitted 41 percent of them after the statutory deadline, largely because DOD did not provide required information to State. These gaps and issues have reduced the accuracy and timeliness of information provided to Congress about ACSAs. DOD has not maintained quality data to track ACSA orders and has not received reimbursement for thousands of orders. First, DOD does not have complete and accurate ACSA data. For example, for an estimated 12 percent of ACSA orders authorized from October 2013 through March 2018 in DOD's system of record, DOD could not determine whether it had received reimbursement for support provided to partners. According to DOD officials, such inaccuracies occur in part because DOD does not have a process to validate data in its system. Second, GAO estimates that DOD received full reimbursement for 64 percent of ACSA orders authorized from October 2013 through March 2018 (about 6,000 orders), but did not receive full reimbursement for 24 percent. Orders remain unpaid in part because DOD has not requested timely repayment or monitored reimbursement. These management weaknesses limit DOD's ability to obtain reimbursement for overdue ACSA orders, which, according to DOD, were valued at more than $1 billion as of November 2019. Note: These estimates are based on a generalizable sample of orders in which the United States provided support to foreign partners; have a margin of error of up to plus or minus 5.1 percentage points at the 95-percent confidence level; and represent the percentage of the number of orders, not the dollar value of orders. GAO is making seven recommendations to DOD to improve ACSA recordkeeping and reimbursement, through steps such as better monitoring, periodic data reconciliation, and timely invoicing. DOD agreed with all seven recommendations.", "document_type": "gao"}
{"report": "DHS, DOJ, and the federal judiciary have different roles in the federal criminal process for immigration-related prosecutions, as shown in table 1. For this report, we define immigration-related offenses as the offenses listed in the Attorney General’s April 2017 memorandum (see table 2). Criminal prosecution process. DHS and DOJ officials told us that DHS’s practices for referring cases for prosecution, and DOJ’s practices for prioritizing immigration-related prosecutions, vary by location along the southwest border. In general, individuals are prosecuted in the judicial district that corresponds with the location of their alleged criminal offenses. Each USAO prosecutes cases in one or more courts. USAOs coordinate with DHS and DOJ components, as well as the federal courts, to determine the types and number of cases, including immigration- related cases, each office will prosecute, according to DOJ officials. In particular, according to DOJ’s Principles of Federal Prosecution, the prosecutor has wide latitude in determining when, whom, how, and whether to prosecute for apparent violations of federal criminal law, and this broad discretion has been recognized on numerous occasions by the courts. Border Patrol officials on the southwest border told us that they receive training from the USAOs about the criminal prosecution process and that they use the prosecution priorities established by the USAO to determine whether to refer a case for criminal prosecution. In general, immigration-related cases referred to the USAO by Border Patrol follow the process described in figure 2. One-day prosecutions. In three federal judicial districts on the southwest border—Arizona, Texas Southern, and Texas Western—DOJ prosecutes defendants for improper entry in criminal proceedings that generally last one day or less, or one-day prosecutions (see figure 2). The volume of defendants prosecuted for improper entry per day in these districts varies depending on the volume of Border Patrol apprehensions and capacity limitations, among other things. Since 2017, there have been several federal directives related to DOJ’s prioritization of immigration-related prosecutions. They are summarized in table 3. Prosecutors in all five southwest border USAOs told us that, in response to the Attorney General’s 2017 directive and in coordination with DHS and other stakeholders, they took steps to prioritize immigration-related prosecutions in their respective jurisdictions. According to officials from the Office of the Attorney General, each USAO exercised its discretion in implementing the priorities identified in the memorandum. For example, some USAOs changed the threshold at which they would accept a prosecution referral for alien smuggling or illegal reentry. Officials in one USAO told us that, before the April 2017 memorandum, their office generally declined to prosecute alien smuggling cases involving fewer than six smuggled aliens. However, in response to the April 2017 memorandum, the office lowered its threshold to two smuggled aliens. Officials in another USAO said that in light of the April 2017 memorandum, they began accepting all referred illegal reentry cases that met the evidentiary standard. Previously, this office did not accept more than 150 illegal reentry defendants without a prior felony conviction per month. In particular, in response to the memoranda, all five USAOs along the southwest border prioritized improper entry referrals for prosecution. Some districts that were already prosecuting some improper entry cases, such as Arizona, were able to quickly increase such prosecutions by scaling their existing systems, according to USAO officials. Specifically, USAO officials in Arizona stated that their office began accepting referrals for first time improper entrants without aggravating circumstances in May 2017, in response to the April 2017 memorandum. In comparison, other USAOs created new processes to prosecute more improper entry cases because they were not previously accepting a significant number of such referrals. For example, prior to the Attorney General’s April 2017 memorandum, the USAO in the California Southern district did not prioritize the prosecution of improper entry cases. USAO officials in San Diego stated that in the spring of 2017, the USAO formed an immigration enforcement working group comprised of certain federal law enforcement entities in San Diego, including USMS, Border Patrol, and CBP OFO, to discuss potential actions the district might take to prioritize immigration enforcement. In May 2018, the Chief Judge in California Southern convened a criminal case management committee comprised of district and magistrate judges, court officials, USAO officials, USMS officials, and federal defenders, among others, to “identify and resolve problems” related to the increased prosecution of improper entry cases. These working groups collaborated to make decisions on issues such as the volume of improper entry cases the court could hear each day, how defendants in improper entry cases would meet with their attorneys, and how many defendants a public defender would represent in court each day. In July 2018, the San Diego court initiated a daily docket for misdemeanor improper entry cases. Regarding DOJ’s coordination with DHS, in four of the five southwest border districts, USAO officials told us that they informed local DHS partners, including local Border Patrol and OFO leadership, that their prosecution guidelines had changed in light of the 2017 memorandum and that they would accept more immigration-related cases for prosecution. As a result, Border Patrol and OFO referred more immigration-related cases to DOJ. Further, in response to the April 2018 zero-tolerance memorandum, Border Patrol issued guidance to all southwest border sectors instructing each sector to develop phased plans to refer all amenable apprehended adults to the USAO for improper entry prosecution, based on capacities of the USAO and the federal courts, and the sectors developed and implemented these plans. In general, these plans prioritized referrals of those individuals with a criminal history first, followed by those with no criminal history. For example, the plan for the Rio Grande Valley Border Patrol Sector stated that, incrementally, the Sector would increase prosecution referrals until attaining 100 percent prosecution on a timeline consistent with DOJ partners’ capacity. Regarding DOJ’s coordination with other stakeholders in the federal criminal process, including the federal courts, USMS, and public defenders, USAO officials told us that they coordinated at the local level to be able to increase immigration-related prosecutions, to the extent practicable. In particular, the federal judiciary held a border court conference in June 2018 and established a task force—including judges, public defenders, and DHS and DOJ representatives—to discuss issues related to changing prosecution priorities in southwest border districts. The task force met three times between July 2018 and April 2019. In addition, stakeholders told us they took other steps to accommodate the USAOs’ prioritization of such prosecutions. For example, Some courts added additional daily dockets or court sessions, or adjusted their use of facilities to accommodate the higher volume of cases being prosecuted. Court officials and magistrate judges we spoke with in all five southwest border districts told us that magistrate judges spent more time presiding over improper entry cases as the number of those cases increased. In McAllen, Texas, for example, court and USAO officials told us that the court added a second daily docket for misdemeanor improper entry cases in May 2018, and doubled the court’s capacity to hear such cases. In Las Cruces, New Mexico, court officials told us that there is one magistrate judge on duty each day for the docket that includes improper entry cases. Federal defenders in Las Cruces told us that stakeholders in Las Cruces, including the court, federal defenders, and USMS, met in spring 2018 and decided to use a second courtroom for magistrate judge duty—including improper entry cases—each day. One courtroom is used for an active proceeding while the other is used to meet and counsel defendants prior to their active court proceeding. In some locations, FDO told us that they developed new practices to provide representation to each defendant appearing in court. For example, the Federal Defender office in McAllen developed an “all hands on deck” process in May 2018, in which all available defenders meet individually with defendants in the courtroom before their initial appearance in court each day. In October 2018, we observed 14 Assistant Federal Public Defenders in McAllen meet with about 72 defendants during the hour before court; federal defenders we spoke with in McAllen said that the process we observed is their daily routine. In San Diego, federal defenders told us that in July 2018, they assigned a team to work full-time on improper entry cases. The team included six trial attorneys, two appellate attorneys, two legal assistants, two investigators, and one interpreter. The courts also increased their use of private defense attorneys appointed under the Criminal Justice Act and interpretation services due to the increased number of immigration-related cases. Several USAO districts were able to quickly increase the number of improper entry prosecutions in response to the Attorney General’s 2017 and 2018 memoranda, to the extent practicable, because such misdemeanor cases are less resource-intensive and less complicated to prosecute than felonies such as illegal reentry or alien smuggling, according to USAO officials in all five southwest border districts. Specifically, many improper entry cases were completed in one-day court proceedings in fiscal year 2018, and in some locations, the cases of 75 or more improper entry defendants were completed each day during a single court proceeding. In three of the five USAO districts—Arizona, Texas Southern, and Texas Western—improper entry prosecutions in fiscal year 2018 generally took place in one-day court proceedings. Based on our analysis of DOJ data, about 84 percent of the 62,000 improper entry cases filed in fiscal year 2018, or about 52,000 improper entry cases, took place in these three districts. We observed proceedings in Arizona and Texas Southern in July and October of 2018, respectively. These proceedings lasted approximately two hours, during which time 50 to 75 improper entry prosecutions were completed. In these proceedings, the initial hearing, presentation of evidence, plea, and sentencing took place during a single day—or a single morning or afternoon—in court. On the basis of our observations in Arizona and Texas Southern, as well as interviews with agency officials in Arizona, Texas Southern, and Texas Western between July 2018 and November 2018, first-time offenders without a prior criminal history typically pled guilty to the improper entry offense and were sentenced to time served. Those defendants remained in the custody of the arresting agency for the duration of the criminal court proceeding, according to Border Patrol and USMS officials at headquarters and agency officials in these three districts. At the time of our visits to the Arizona and Texas Southern districts, we observed judges sentence some defendants with a prior improper entry conviction to terms of imprisonment ranging from 10 to 180 days. The judge remanded these defendants to USMS custody to serve their sentence. In the other two USAO districts—California Southern and New Mexico— most improper entry prosecutions took place over the course of approximately one week, based on our observations of such prosecutions in California Southern and interviews with agency officials in California Southern and New Mexico in October and November 2018. Based on our analysis of DOJ data, about 16 percent of improper entry cases filed in fiscal year 2018, or about 10,000 cases, took place in these districts. After an initial appearance in court, the judge remanded the defendant to USMS custody and set a subsequent hearing for three to four days later. At the second hearing, the defendant typically pled guilty to the improper entry offense and the judge sentenced them. First-time offenders typically pled guilty to the improper entry offense and were sentenced to time served. The USAOs’ ability to increase improper entry prosecutions was also affected by different practices in the federal criminal process for improper entry cases in each of the five southwest border districts, as shown in table 4. In some locations, these practices affected the extent to which prosecutors could accept all improper entry cases referred for prosecution. According to officials from the Offices of the Attorney General and the Deputy Attorney General, DOJ contemplated such variation in its directives to federal prosecutors. Further, according to agency officials, practices for improper entry cases may change over time, depending on the priorities of various stakeholders in the federal criminal process, physical space limitations, or availability of resources such as interpreters, among other reasons. As of November 2018, Border Patrol referred nearly all single adults who could be charged with improper entry to the USAOs for prosecution in some districts, according to Border Patrol officials and Border Patrol’s operational guidance in those districts. In these locations, officials from Border Patrol, USAO, and the federal judiciary told us that they had sufficient capacity to process all such cases. In other districts, Border Patrol referred a lower percentage of single adults for prosecution for improper entry based on the ability of the USAO to accept such referrals or other factors, consistent with DHS’s May 2018 memorandum. For example, in Tucson, Arizona, the court generally allowed 75 improper entry cases per day at the time of our July 2018 visit. However, in McAllen, Texas, court officials told us that the court would hear as many improper entry cases as the USAO accepted for prosecution, which was as many as 200 cases per day, as of our October 2018 visit. At the time of our visits in July and October 2018, other considerations affecting the number of improper entry prosecutions included Border Patrol’s capacity to process case referrals (Texas Southern), restrictions on the number of daily defendants that the court could accommodate (Arizona, California Southern), and physical constraints, such as the number of seats for defendants in the courtroom (Texas Southern). In addition, public defense practices for misdemeanor improper entry cases varied across districts and, in some locations, affected the number of improper entry cases that the USAO could file each day. In California Southern and Arizona, each public defender represented a maximum of 4 or 6 defendants in court each day, respectively, in October 2018 and July 2018. In Texas Southern, one public defender may represent up to 100 defendants in court at a time, as of October 2018, according to defender office staff. Furthermore, local court rules or practices in some locations affected the number of improper entry cases that Border Patrol could refer or the USAO could file each day. For example, in California Southern, as of October 2018, the court required defendants to appear in court the next court day after their arrest. In addition, all defendants were required to undergo a medical screening for tuberculosis before their initial appearance in court. Border Patrol data indicate that the number of single adults referred to USAOs for prosecution more than doubled from fiscal year 2017 (about 49,700) to fiscal year 2018 (about 101,000), and was higher in fiscal year 2018 than in each of the four prior fiscal years. The total number of single adults Border Patrol apprehended varied from year to year over this time and Border Patrol data indicate that fewer single adults were apprehended in both fiscal years 2017 and 2018 than in each of the three prior fiscal years. However, the proportion of apprehended single adults that Border Patrol referred for prosecution was higher in fiscal year 2018 (38 percent) than in each of the four prior fiscal years (ranging from 20 to 24 percent) (see fig. 3). On the basis of our analysis of Border Patrol data, USAOs declined approximately 8 percent of Border Patrol’s criminal prosecution referrals in fiscal year 2018. In the four prior fiscal years, USAOs declined between 2 and 4 percent of such Border Patrol referrals. However, in fiscal year 2018, the number of cases Border Patrol referred for prosecution—and the number of cases that were accepted and prosecuted by USAOs—was also substantially higher compared to prior years, which was consistent with DHS and Border Patrol guidance to increase prosecution referrals to the extent practicable and consistent with DOJ partners’ and federal court capacity. The reasons for declinations varied and included timing and capacity-related reasons, according to Border Patrol’s data and officials. For example, defendants must generally appear before a judge within 48 hours of their Border Patrol apprehension and, according to Border Patrol officials, the remote locations of some apprehensions can make it difficult for Border Patrol to process, transport, and present defendants in court within the required timeframe. Border Patrol data indicate that apprehensions of single adults in fiscal year 2018 varied by U.S. Attorney district and, in general, Border Patrol referred a greater proportion of those apprehended for prosecution in districts with a relatively low number of apprehensions. Specifically, in the two districts with the fewest apprehensions (New Mexico and Texas Western, with about 10,000 and about 26,000 apprehensions, respectively), Border Patrol referred 80 and 75 percent of those apprehended for prosecution in fiscal year 2018. In the remaining three districts (Arizona, California Southern, and Texas Southern), each of which had more than 53,000 single adult apprehensions in fiscal year 2018, Border Patrol referred between 14 and 45 percent of those apprehended for prosecution in fiscal year 2018. According to Border Patrol officials in these three districts, various factors influenced the number of referrals to USAOs, including court capacity, availability of Border Patrol agents to prepare cases for referral, and USAO capacity to accept and prosecute cases, consistent with the Attorney General’s guidance to prioritize such prosecutions to the extent practicable. DOJ prosecuted more immigration-related cases—including improper entry, illegal reentry, and alien smuggling cases—in fiscal year 2018 than in each of the prior four fiscal years. Specifically, southwest border USAOs filed about 91,000 improper entry, illegal reentry, and alien smuggling cases in fiscal year 2018, compared to a prior four-year high of about 78,000 immigration-related cases filed in 2014. On the basis of our analysis of DOJ data, cases with a lead charge of improper entry comprised more than half of DOJ’s immigration-related cases filed each year from fiscal years 2014 through 2018. Further, the total number of cases filed with a lead charge of improper entry, illegal reentry, or alien smuggling increased between fiscal year 2017 and fiscal year 2018 in the five southwest border districts, consistent with the priorities in the April 2017 and April 2018 memoranda, although the magnitude of the increases varied. Figure 4 illustrates the number of cases filed by USAOs with a lead charge of improper entry, illegal reentry, or alien smuggling along the southwest border, as well as trends in such cases from fiscal years 2014 through 2018. From fiscal year 2014 through 2018, more than 95 percent of improper entry, more than 90 percent of illegal reentry, and more than 80 percent of alien smuggling cases ended in convictions. The majority of defendants for improper entry and illegal reentry cases from fiscal years 2014 through 2018 were Mexican nationals, although the proportion of defendants with nationalities other than Mexican increased in fiscal year 2018 relative to the prior four fiscal years. The majority of defendants for alien smuggling cases from fiscal years 2014 through 2018 were U.S. nationals. See appendices II and III for more detailed information on case dispositions and nationalities of defendants. Improper entry. DOJ data indicate that the total number of cases filed with a lead charge of improper entry in southwest border districts more than doubled between fiscal year 2017 and 2018, as illustrated in table 5. Figure 5 illustrates the number of improper entry cases filed by southwest border USAOs each month in fiscal years 2017 and 2018. In New Mexico and Arizona, the number of improper entry cases filed increased notably in June 2017. These districts generally did not prosecute first-time entrants for these misdemeanor offenses from 2014 until 2017, and changed their prosecution practices in response to the Attorney General’s April 2017 memorandum, according to USAO officials we spoke with in those districts. In Texas Southern, the number of improper entry cases filed increased notably in April 2018. Prior to April 2018, the USAO in McAllen allowed 40 to 50 improper entry prosecutions per day, according to USAO officials. The USAO removed this limitation in response to the Attorney General’s April 2018 memorandum. As of October 2018, this USAO accepts all prosecution referrals with sufficient evidence (on average, 100 to 200 improper entry prosecutions per day), according to officials. From fiscal year 2017 through 2018, improper entry cases filed in Texas Southern nearly tripled, from about 10,800 to about 30,100 cases. In Texas Western, the number of improper entry cases filed began to increase in March 2018, but to a lesser extent than other districts, and then decreased from July through September 2018. USAO officials attributed the increase to increased Border Patrol apprehensions and said that they accept all Border Patrol prosecution referrals, but the number of cases that the USAO receives depends on fluctuating Border Patrol apprehension numbers. In California Southern, the number of improper entry cases filed began to increase in May 2018. Prior to July 2018, California Southern did not have a court docket dedicated to prosecuting improper entry misdemeanor offenses. According to officials, following the Attorney General’s April 2018 memorandum, the San Diego district court, in coordination with the USAO, agreed to establish a daily improper entry docket with the capacity to hear initial appearances for 40 to 52 improper entry cases each day. Illegal reentry. DOJ data indicate that the number of cases USAOs filed with a lead charge of felony illegal reentry along the southwest border declined from fiscal years 2015 through 2017 before increasing by 2,669 cases from fiscal year 2017 through 2018. However, the number of illegal reentry cases filed in fiscal year 2018 (25,112) was lower than in fiscal year 2014 (31,670) or 2015 (28,480), and the magnitude of the increase in illegal reentry cases filed from fiscal year 2017 through 2018 (12 percent) was smaller than the increase in improper entry cases during the same period (130 percent). The number of cases filed with a lead charge of illegal reentry declined in Arizona each year between fiscal years 2015 and 2018, but increased or varied in other districts. Between fiscal year 2017 and 2018, illegal reentry cases filed increased most notably in Texas Western, where there were 69 percent more illegal reentry cases filed in fiscal year 2018 than in fiscal year 2017. Federal court and USAO officials in Texas Western attributed this increase in illegal reentry prosecutions to increased Border Patrol apprehensions and referrals for prosecution in fiscal year 2018. Table 6 illustrates illegal reentry cases filed, by fiscal year, from fiscal years 2014 through 2018. USAO officials attributed the changes in illegal reentry cases filed from fiscal year 2014 through 2018 to changes in prosecution practices as well as changes in the number of apprehensions. For instance, the New Mexico USAO removed a monthly limitation originally enacted in fiscal year 2016 on the number of illegal reentry cases filed they would accept following the April 2017 memorandum, according to Border Patrol and USAO officials. Other locations have varying thresholds and practices regarding accepting, charging, and prosecuting illegal reentry cases. USAO officials in New Mexico and Texas Western told us that they charge defendants with illegal reentry if the defendant has one prior deportation or one prior conviction for improper entry. Officials in three other districts told us that they generally require a more extensive criminal history—for instance, they might require multiple prior improper entry convictions—to charge illegal reentry. USAO officials in Arizona and California said that they file cases with a lead charge of illegal reentry that might ultimately end with improper entry convictions. For example, our analysis of EOUSA data indicates that of almost 12,000 illegal reentry cases filed in Arizona in fiscal year 2017, approximately 77 percent ended with an improper entry conviction and approximately 18 percent ended with an illegal reentry conviction. Figure 6 illustrates the number of cases filed with a lead charge of illegal reentry filed each month in fiscal years 2017 and 2018. Alien smuggling. DOJ data indicate that the number of cases filed with a lead charge of alien smuggling increased in four of the five southwest border districts from fiscal year 2017 through 2018. Officials from two USAO locations along the southwest border told us that they changed their thresholds for how many material witnesses (individuals being smuggled) must be present to accept an alien smuggling referral in response to the Attorney General’s April 2017 memorandum. For instance, the USAO in San Diego lowered the threshold for accepting alien smuggling referrals and, following the April 2017 memorandum, places equal priority on all alien smuggling referrals. Prior to the April 2017 memorandum, the USAO would have considered several factors when deciding whether to accept the referral, such as if there was a risk of harm to the material witnesses or whether the conviction could result in a significant term of imprisonment for the smuggler. Figure 7 illustrates the number of cases filed with a lead charge of alien smuggling each month over fiscal years 2017 and 2018. DOJ, DHS, and the federal judiciary realigned resources to support the prosecution priorities outlined in the April 2017 and April 2018 memoranda. Officials from USAOs, USMS, Border Patrol, federal courts, and federal defenders along the southwest border told us that they are using more personnel, physical space, or both to support increased immigration-related prosecutions than they were prior to DOJ’s prioritization of immigration enforcement in April 2017. When USAOs along the southwest border changed their prosecutorial priorities and realigned resources in response to the April 2017 and April 2018 memoranda, other agencies, such as USMS and the federal judiciary, also realigned resources to respond to and support increased immigration-related prosecutions. In some cases, these realignments affected their ability to conduct other activities. Officials from USMS and the courts told us that, as stakeholders in the federal criminal process, they are accustomed to reacting to changing conditions that may affect their operations. For example, these officials’ operations could be affected by changes in the number of Border Patrol apprehensions, changes in Border Patrol’s prosecution referral priorities, changes in the location of drug or human smuggling activity, and changing USAO prosecutorial priorities, among other things. USAOs. USAO officials in three locations stated that the more time prosecutors spend on reactive work—such as misdemeanor or felony immigration-related cases—the less time Assistant U.S. Attorneys (AUSA) have to work on other issue areas, including proactive cases that may take months or years of work to build, or civil cases. For instance, USAO officials from Texas Southern said that the high immigration caseload in McAllen affects AUSAs’ ability to prosecute other types of cases, such as Organized Crime Drug Enforcement Task Force cases, which tend to be long-term cases. According to USAO officials in San Diego, when prosecutors began accepting improper entry referrals in July 2018, there was a short-term decline in the number of prosecutions that were initiated for other cases. This decline mainly affected drug and alien smuggling cases, some of which were referred to state or local prosecutors, according to USAO officials in San Diego. As of October 2018, USAO officials in San Diego said that improper entry prosecutions were not affecting their ability to accept referrals for new felony prosecutions. USMS. According to USMS officials, each additional court docket, courtroom in use, or immigration-related defendant who appears in court requires judicial security support. USMS officials in all five southwest border locations told us that they took actions to meet the judicial security mission need, but that the increased prosecutions have strained their staff. USMS officials we spoke with in all five southwest border districts said that they reassigned deputies in fiscal year 2018 from proactive task forces (such as task forces dedicated to arresting individuals with active federal warrants) to judicial security court duty and detention security to support increased immigration-related prosecutions. In particular, USMS officials said that they assigned more deputies to judicial security court duty because of the increase in improper entry prosecutions. USMS officials we spoke with in several locations on the southwest border said that the increased judicial security duty has made it difficult for their deputies to meet their training requirements. They are concerned that the high demand for judicial security in southwest border districts may affect their ability to retain deputies. Officials from USMS in multiple locations along the southwest border told us that the increase in immigration-related prosecutions strained their existing detention space. For instance, California Southern required additional detention space for defendants in improper entry cases, and could not locate additional detention space nearby or within the judicial district. As a result, USMS officials told us in October 2018 that deputies may drive defendants to neighboring judicial districts, including California Central, Nevada, and Arizona, to detain them before and between court appearances. According to USMS officials, providing transportation for such defendants can comprise deputies’ entire shifts. Additionally, officials in the Texas Western district told us in November 2018 that due to the increase in immigration-related prosecutions in fiscal year 2018, detention facilities in Del Rio reached capacity. USMS transports prisoners up to seven hours one way to other detention facilities. Further, USMS received permission to triple-bunk prisoners (using three stacked beds, rather than two stacked beds) in Del Rio and El Paso, and to use additional temporary beds, such as cots, to house additional prisoners close to courthouses. As we have previously reported, the average daily population of USMS prisoners is directly influenced by, among other things, the activities and decisions of federal law enforcement, USAOs, and the federal judiciary. According to USMS data, the average daily population of immigration- related prisoners on the southwest border increased from 7,796 in May 2017 (a five-year low) to 11,668 in September 2018 (a five-year high). According to USMS officials with whom we spoke and documents we reviewed, in 2018, USMS sought additional detention space. In May 2018, USMS issued a public request for information to determine the availability of contractor owned and operated secure detention facilities on the southwest border. In October 2018, USMS signed intragovernmental agreements with two local detention facilities in Texas, adding approximately 655 available beds to its inventory. Additionally, USMS officials in Las Cruces told us that they had more detention space than they required for prisoners in New Mexico and that, following the April 2018 memorandum, USMS began to accept prisoners from other districts. Prior to the April 2018 memorandum, USMS in New Mexico had approximately 1,300-1,400 of their own prisoners in custody. As of November 2018, USMS in Las Cruces had approximately 1,800 prisoners in custody from New Mexico and approximately 500 prisoners in custody from neighboring districts. Border Patrol. Border Patrol agents support, and in some cases supplement, DOJ components in both prosecution and judicial security work. As of March 2019, in nine of nine southwest border sectors, Border Patrol reported that it had detailed agents to USAOs to assist with tasks like data entry and preparing court documents for immigration-related prosecutions. In addition, in seven of nine sectors, Border Patrol detailed agents to USMS locations to assist with judicial and detention security. The number of agents from Border Patrol that are detailed to assist DOJ components with immigration-related prosecutions generally varies based on the volume of prosecutions that the USAO receives and accepts and, in some sectors, based on available Border Patrol agent resources, according to Border Patrol officials. Following the Attorney General’s memoranda, Border Patrol increased the number of agents that it detailed to certain USAOs and USMS locations along the southwest border, both temporarily and on an ongoing basis, because of the increased volume of immigration-related prosecutions, according to Border Patrol officials. As of March 2019, Border Patrol sectors across the southwest border detailed from zero to four agents to perform USMS functions, and zero to five agents to perform USAO functions. The length of detail and duties assigned to Border Patrol agents detailed to USAOs and USMS vary by location, according to officials. Generally, when an assignment ends, Border Patrol agents return to their regular Border Patrol duties. Federal courts. Federal court officials we spoke with in five locations stated that they faced challenges resulting from the increased immigration caseload. For instance, court officials in Las Cruces said that, as of November 2018, staff in the clerk’s office often work on weekends to keep up with court scheduling and paperwork resulting from increased improper entry prosecutions. The Las Cruces court also implemented telework options for clerk staff to give them the option of working additional hours from home. Additionally, officials we spoke with from several courts reported that they had existing needs for judgeships, and the increasing immigration caseload placed additional strain on district and magistrate judges. For instance, the district court in Del Rio has one district judge and the number of illegal reentry prosecutions in fiscal year 2018 increased by almost 70 percent compared to fiscal year 2017. Court officials we spoke with in two locations told us that sentencing dates have been pushed out because of the increase in district judges’ caseloads. According to federal court officials in Del Rio, the district judge’s calendar is so full that, in some cases, a defendant’s sentencing might be pushed back far enough that the defendant has already served more jail time than the federal sentencing guidelines recommend by the time the defendant is sentenced. In addition, multiple court officials in multiple locations across the southwest border told us that increased immigration-related prosecutions, and particularly improper entry cases, increases strain on courtroom facilities and equipment and, in some instances, courts have to replace equipment and furniture more often. For example, in Tucson, Arizona, the improper entry courtroom can hold up to 75 improper entry defendants in restraints, such as handcuffs and/or leg restraints, at a time during morning improper entry proceedings, and court officials told us that the restraints worn by defendants cause damage to the chairs and benches in the courtroom. Defender services. FDO staff we spoke with in several southwest border districts told us that they dedicated more staff or staff time towards defendants in immigration-related cases and accommodated increased prosecutions within existing resources as of December 2018. For instance, defenders in Las Cruces stated that the court added a new docket for improper entry cases, and defense attorneys are at times scheduled to be in two courtrooms at once, and must cover for each other. Defenders in Las Cruces also told us that they have run out of physical office space for their staff. Federal defenders in McAllen said that the amount of time defense attorneys spend on improper entry interviews affects the time they can spend on felony cases. In addition, these defenders described the process of preparing 100 or more defendants for criminal proceedings each day as draining. Defenders in McAllen noted that they filed more continuances in fiscal year 2018 than in prior years as a result of the increased workload caused by the expanded improper entry docket. EOUSA, USMS, Border Patrol, and the federal judiciary temporarily surged personnel from locations across the United States to the southwest border to support increased immigration-related prosecutions. These agencies tracked some costs associated with those temporarily detailed personnel, among other costs associated with increased immigration-related prosecutions. Additionally, EOUSA announced plans to hire new attorneys to prosecute immigration-related offenses in May 2018, both on the southwest border and in the interior of the United States. EOUSA. In May and June 2018, DOJ announced plans to permanently hire 70 new AUSAs to prosecute immigration-related offenses both at the southwest border and in the interior of the U.S. Additionally, EOUSA officials told us that they subsequently received DOJ approval to hire 13 more AUSAs to work on immigration and border security issues on the southwest border. In fiscal year 2018, EOUSA expended about $9.8 million on personnel costs associated with these prosecutors—including 35 immigration crimes prosecutors in the interior of the United States, 42 immigration crimes prosecutors in the five southwest border districts, and 6 civil condemnation AUSAs working on the southwest border. EOUSA estimated that the fiscal year 2019 continuing personnel costs associated with these prosecutors would be about $17 million. In its fiscal year 2020 Congressional Budget Justification, EOUSA requested a $23.3 million increase in funding from Congress to sustain hiring and program operations that were initially funded in fiscal year 2018, including the immigration prosecutors. EOUSA also intends to allocate a portion of these 2020 funds to USAOs around the country with demonstrable workload challenges. EOUSA and USAO officials said that these permanent AUSA positions would support immigration prosecutions on the southwest border in the long-term. USAO officials we spoke with in all five southwest border districts between July and November 2018 said that they were in the process of hiring these immigration AUSAs. While EOUSA was in the process of permanently hiring new AUSAs, EOUSA temporarily surged Special Assistant U.S. Attorneys (SAUSAs) to southwest border districts that needed more prosecutors to handle the increased immigration caseload. Some of these SAUSAs prosecuted improper entry offenses specifically and others prosecuted any immigration-related case. Specifically, EOUSA solicited attorneys from other DOJ components, the Department of Defense (DOD), and CBP to serve as SAUSAs for immigration-related offenses along the southwest border. Beginning in June 2017, DOJ detailed 12 attorneys from non- southwest border USAOs and other DOJ components to prosecute immigration-related cases in all five districts on the southwest border. In fiscal years 2017 and 2018, EOUSA expended approximately $440,000 on travel and lodging for these 12 SAUSAs. In June 2018, DOD agreed to provide military attorneys to act as SAUSAs and support immigration-related prosecutions on the southwest border. DOD detailed a total of 21 military attorneys to the southwest border for approximately six months each between June 2018 and January 2019, according to EOUSA. According to USAO officials in New Mexico, which received five military SAUSAs, and California Southern, which received five, these SAUSAs provided key support that allowed these districts to increase improper entry and illegal reentry prosecutions beginning in June 2018. In fiscal year 2018, EOUSA estimated that it expended approximately $1,186,000 on salaries, travel, and lodging for these 21 SAUSAs. In some southwest border locations, CBP regularly provides SAUSAs to add prosecutor capacity to USAOs. For example, in four locations, CBP SAUSAs are the federal prosecutors for misdemeanor improper entry cases and appear daily in court to prosecute these cases. In San Diego, CBP SAUSAs began supporting the misdemeanor improper entry docket in July 2018, when the docket began. In New Mexico, 10 part-time CBP SAUSAs supported the improper entry docket temporarily between January and July 2018, which allowed New Mexico to begin prosecuting improper entry cases with no effect to the workload of its full-time AUSAs. CBP officials also said that CBP has provided full-time SAUSAs for a six or 12-month term to some USAOs on an ongoing basis, depending on USAO request and CBP workload. USAO officials have asked CBP for additional SAUSAs in San Diego and Yuma; as of April 2019, CBP officials said that due to CBP’s workload in these locations, they have not agreed to additional SAUSAs in these locations. USMS. From June through November 2018, USMS detailed deputies from non-southwest border locations to southwest border courts to support judicial security operations. Approximately 96 deputies participated in these temporary detail rotations, which lasted two to three weeks each, over the six month period. USMS established a budget code to track additional expenditures that USMS headquarters incurred related to implementing the April 2018 memorandum. These additional expenditures included travel and lodging costs for the detailed USMS deputies and transportation costs, among others. USMS reported approximately $1,149,000 in expenditures from May through December 2018 under this budget code. In its 2020 Congressional Budget Justification, USMS requested nearly $8 million from Congress for 35 positions to address departmental priorities and initiatives, including immigration enforcement. USMS officials said that their workload, including immigration prosecutions in fiscal year 2018 surpassed previous peak levels. For instance, USMS reported more immigration-related “prisoners received” in 2018 than in each of the prior five fiscal years. Border Patrol. Border Patrol established a budget code in April 2018 to track additional expenditures directly associated with implementing the April 2018 memorandum. In particular, according to Border Patrol budget officials and documentation, Border Patrol officials were to use the budget code to track expenditures related to detainee food, supplies, and transportation. In addition, the code was to be used for Border Patrol agent overtime expenditures and any travel expenditures that could be attributed to the April 2018 memorandum. From April 2018 through December 2018, Border Patrol reported approximately $2,316,000 in expenditures under this budget code. Federal judiciary. The federal judiciary sends visiting judges from other parts of the United States to southwest border districts to assist with judge caseloads, including immigration cases. For instance, the federal judiciary approved 67 visiting judge assignments from other parts of the U.S. to New Mexico and Texas Western in fiscal years 2017 and 2018; AOUSC reported expending approximately $114,000 on travel costs for these visiting judges. Federal courts along the southwest border also expended more funds on contracted interpreter services in fiscal year 2018 than in any of the prior four fiscal years. When a defendant does not speak English, courts may have interpreters on staff and courts may use contracted interpreter services. Court officials from multiple locations along the southwest border told us that contracted interpreter services became increasingly difficult to obtain following the increase in immigration-related prosecutions. According to federal judiciary documentation, there were 100,000 more court events, or defendant appearances before a judge, in southwest border courts requiring court interpreter services in fiscal year 2018 than there were in fiscal year 2017. Expenditures for contracted court interpreters increased by over $450,000 from fiscal year 2017 to fiscal year 2018 for southwest border courts. We provided a draft of the sensitive report to DOJ, DHS, and AOUSC for their review and comment. DOJ, DHS, and AOUSC provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Attorney General of the United States, the Acting Secretary of the Department of Homeland Security, the Administrative Office of the U.S. Courts, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or goodwing@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V. This appendix provides additional details on our objectives, scope, and methodology. Specifically, our objectives were to provide information on the following: 1. how the Department of Justice (DOJ) prioritized criminal prosecutions of immigration-related offenses in response to the Attorney General’s 2017 and 2018 memoranda; 2. what Department of Homeland Security (DHS) and DOJ data from fiscal years 2014 through 2018 indicate about criminal prosecutions of immigration-related offenses; and 3. resources that DOJ, DHS, and the federal judiciary used to implement increased immigration-related prosecutions. This report is a public version of the prior sensitive report that we provided to you in August 2019. DHS, DOJ, and the Administrative Office of U.S. Courts (AOUSC) deemed some of the information in the prior report as Law Enforcement Sensitive or For Official Use Only, which must be protected from public disclosure. Therefore, this report omits sensitive information about specific law enforcement, prosecutorial, and judicial practices along the southwest border, including certain courtroom security and agency staffing information. Although the information provided in this report is more limited, the report addresses the same objectives as the sensitive report and uses the same methodology. For all three objectives, we generally focused our review on the five U.S. Attorney Office (USAO) districts along the southwest border—Arizona, California Southern, New Mexico, Texas Southern, and Texas Western— because the Attorney General’s 2017 and 2018 memoranda specifically directed officials in these districts to prioritize improper entry prosecutions. Further, approximately 93 percent of all immigration-related prosecutions from fiscal years 2014 through 2018 took place in these districts. USAO districts and federal judicial districts have the same boundaries. U.S. Border Patrol (Border Patrol) sectors along the border are generally not contiguous with USAO districts. We visited three of the five districts and interviewed officials by telephone from the other two southwest border districts. Specifically, we conducted in-person site visits to Arizona in July 2018 and to California Southern and Texas Southern in October 2018. We selected these locations on the basis of several factors, including Border Patrol apprehension characteristics and DOJ prosecution practices. Specifically, to select the locations for our site visits, we considered DOJ’s history of prosecuting improper entry offenses in different locations, including whether districts implemented changes to their practices for prosecuting improper entry offenses in response to the Attorney General’s memoranda. For instance, we considered districts’ practices for prosecuting improper entry offenses and whether those practices changed in response to the April 2017 or April 2018 memoranda. In addition, we considered the number of Border Patrol apprehensions in each USAO district and changes in the number of apprehensions from fiscal years 2014 through 2018. We also considered factors such as whether DOJ, DHS, and federal court facilities are in close proximity, among other things. In the three districts we visited, we met with DOJ and federal court officials, including magistrate and district judges, to understand and observe their roles in the criminal prosecution process. We met with USAO, U.S. Marshals Service (USMS), Federal Defender Organizations (FDO), and federal court officials and observed federal criminal court proceedings in Tucson, Arizona; San Diego, California; McAllen, Texas; and Brownsville, Texas. We observed the criminal prosecution process from arrest to conviction and sentencing, including observations of district and magistrate court proceedings and USMS intake and holding facilities in federal courthouses. In addition, we observed U.S. Customs and Border Protection’s Border Patrol agents and Office of Field Operations (OFO) officers processing apprehended individuals and referring them for prosecution. We met with Border Patrol officials in Tucson, Arizona; McAllen, Texas; and San Diego, California. We met with OFO officials at ports of entry in Nogales, Arizona; San Ysidro, California; Hidalgo, Texas; and Brownsville, Texas. We also interviewed USAO, USMS, federal court, Border Patrol, and OFO officials who are involved in immigration prosecutions in Las Cruces, New Mexico in November 2018 (New Mexico district) and Del Rio, Texas in November 2018 (Texas Western district). Although the information we obtained from these site visits and interviews cannot be generalized to all locations along the southwest border, these interviews provided important insights and perspectives about immigration-related prosecutions and any process, volume, or resource changes in immigration-related prosecutions following the April 2017 and 2018 memoranda. To determine how DOJ prioritized immigration-related prosecutions, we obtained and reviewed operational guidance, policies, and memoranda describing how DOJ, DHS, and the federal judiciary implement such prosecutions along the southwest border. We also reviewed documentation to identify any changes to such practices associated with implementing the Attorney General’s April 2017 and the April 2018 memoranda. We reviewed training materials from the Executive Office of U.S. Attorneys (EOUSA) provided to some federal prosecutors regarding prosecuting immigration-related cases at a 2018 Border Security Coordinator conference and relevant U.S. Attorneys’ Bulletins from DOJ’s Journal of Federal Law and Practice, such as the July 2017 bulletin, Prosecuting Criminal Immigration Offenses, and the Justice Manual, which contains publicly available DOJ policies and procedures, including criminal prosecution procedures. In addition, we interviewed headquarters and district officials from DOJ, DHS, and the federal courts to obtain their perspectives on the Attorney General’s prioritization of immigration-related prosecutions and any changes in practices as a result of the two memoranda. Specifically, from DOJ, we interviewed officials from the Offices of the Attorney General and the Deputy Attorney General about the development and implementation of the April 2017 and April 2018 memoranda. We also interviewed officials from EOUSA about headquarters-level support to USAOs. We interviewed headquarters officials from USMS about how the Attorney General’s prioritization of immigration offenses affected USMS operations and about available data measuring such effects. from DHS, we interviewed Border Patrol and OFO headquarters officials about actions CBP components took in response to the Attorney General’s prioritization of immigration prosecutions and reviewed DHS, CBP, and Border Patrol memoranda and Border Patrol operational guidance related to the prioritization of immigration prosecutions. We also interviewed officials from U.S. Immigration and Customs Enforcement (ICE) about the effect of the Attorney General’s prioritization on ICE’s operations. from the Administrative Office of the U.S. Courts (AOUSC)—the federal judiciary agency that provides legislative, administrative, management, and program support to federal courts, among other functions—we interviewed officials in Washington, D.C. about the federal judiciary’s roles and responsibilities related to criminal immigration-related cases, including the roles of magistrate and district judges and public defenders. To determine what DHS and DOJ data indicate about prosecutions of immigration-related offenses, we analyzed record-level apprehension and prosecution referral data from Border Patrol’s Enforcement Integrated Database/e3 (e3), as well as record-level prosecution data from EOUSA’s CaseView from fiscal years 2014 through fiscal year 2018, the most recent data available at the time of our analysis. Border Patrol data. In reviewing the Border Patrol data, we determined that the majority of Border Patrol apprehensions (about 97 percent) from fiscal years 2014 through fiscal year 2018 took place along the southwest border. We excluded the small percentage of apprehensions nationwide that did not take place along the southwest border from our primary analysis, meaning that we excluded apprehensions in all districts but Arizona, California Southern, New Mexico, Texas Southern, or Texas Western from our primary analysis. We assigned each Border Patrol sector apprehension to its corresponding judicial district to maintain the judicial district as our unit of analysis for the apprehension and prosecution referrals data. For instance, if the El Paso Border Patrol sector referred a prosecution to the USAO in Las Cruces, New Mexico, we report that referral as occurring in the district of New Mexico. We matched data from e3’s apprehensions module with data from e3’s prosecutions module using an identifier that Border Patrol officials told us was unique to each apprehended individual to analyze those individuals that were and were not referred for criminal prosecution. Border Patrol’s apprehensions and prosecution referrals include individuals who are deportable and non-deportable, as determined by Border Patrol. According to Border Patrol officials, non-deportable individuals may be U.S. citizens, foreign nationals who have a valid visa, or individuals who otherwise may not be amenable to removal from the United States. We have included non-deportable individuals in our analysis because they may be referred for prosecution for immigration-related crimes, including alien smuggling. Appendix II includes information on Border Patrol apprehensions and prosecution referrals in each judicial district from fiscal years 2014 through 2018 and information on apprehensions and prosecution referrals by nationality, including U.S. citizens. We restricted our Border Patrol data analysis to apprehensions of non-juveniles who Border Patrol did not process as members of family units. In other words, we analyzed apprehensions and prosecution referrals of single adults. According to Border Patrol guidance and agency officials, e3 has system checks in place that do not allow members of family units to be referred for criminal prosecution. Prior to April 2018, Border Patrol officials said that individuals who were to be referred for prosecution were generally processed by Border Patrol as single adults whether or not they were apprehended with their minor children. In April 2018, an update to e3 allowed Border Patrol agents to separate one or more members of a family unit from that family unit and refer those individuals for prosecution. As stated previously, we included individuals that Border Patrol processed as single adults in our analysis of Border Patrol apprehensions. EOUSA data. In reviewing EOUSA record-level prosecution data from fiscal years 2014 through 2018, we determined that the majority of cases filed with an immigration-related lead charge (over 90 percent of cases with an immigration-related lead charge) took place along the southwest border. We excluded prosecutions that did not take place along the southwest border from our primary analysis; we report on them in an appendix. Additionally, we determined that improper entry, illegal reentry, and alien smuggling charges comprised approximately 99 percent of immigration-related cases filed on the southwest border from fiscal years 2014 through 2018. We excluded the other charges that the Attorney General listed in the April 2017 memorandum from our primary analysis. We analyzed EOUSA data based on the lead charge of the prosecution record. The lead charge is typically the most serious provable offense for which a defendant can be prosecuted, as determined by the USAO. We analyzed EOUSA data by fiscal year from fiscal years 2014 through 2018 to determine overall trends in immigration-related prosecutions over time. We also analyzed data by month in fiscal year 2017 and fiscal year 2018 to identify any changes in immigration-related prosecutions following the April 2017 and April 2018 memoranda. We interviewed knowledgeable USAO officials in southwest border districts level to understand how practices that they changed in response to the April 2017 and April 2018 memoranda were reflected in the data. We also analyzed the nationality of defendants based on lead charge for fiscal years 2014 through 2018, and for fiscal year 2018, to determine any changes in nationality of those prosecuted in the most recent fiscal year compared to prior fiscal years. We identified a population of defendants whose nationalities were listed as ‘unknown,’ in the EOUSA data. When USAOs are unable to determine the nationality of a defendant, officials entering the case data will list that nationality as ‘unknown.’ In appendix III, we report on the proportion of defendants with a nationality that is ‘unknown’ for alien smuggling cases because ‘unknown’ nationalities were relatively common for alien smuggling cases. We grouped the lead charges into offense categories based on the statute of the offense. We analyzed EOUSA data at the statutory level rather than by the individual charged offenses because EOUSA officials told us that USAOs may have differing data entry practices related to the level of specificity at which they enter lead charge data into CaseView. Additionally, EOUSA directed USAOs to ensure that improper entry, illegal reentry, and alien smuggling cases are entered into EOUSA’s data system on a monthly basis at the statute level in August 2017. Table 8 lists the specific offenses that we combined under their shared statute for our analysis. We assessed the reliability of Border Patrol and EOUSA data by testing for missing data and obvious errors, reviewing related documentation such as data dictionaries and guidance for entering data, and interviewing knowledgeable agency officials both at the headquarters level and in the three districts that we visited. We determined that the Border Patrol and the EOUSA data are sufficiently reliable for reporting on immigration- related prosecutions and individuals that Border Patrol apprehended and referred for criminal prosecution. To determine the resources used to implement increased immigration- related prosecutions, we obtained and reviewed DOJ, DHS, and federal judiciary documentation focused on any existing resources that agencies realigned to implement or support increased immigration prosecutions, as well as expenditures or additional personnel used to support the implementation of increased immigration-related prosecutions. For example, we reviewed memoranda of understanding between EOUSA and the Department of Defense (DOD) regarding DOD detailing attorneys to EOUSA to prosecute immigration-related offenses, as well as USMS intergovernmental agreements used to expand detention space. We also reviewed data from USMS on the unique prisoners received, average daily prisoner population, and total prisoner appearances in court to determine any changes in the volume of USMS prisoners from fiscal year 2014 through fiscal year 2018. We reviewed documentation from southwest Border Patrol sectors specifying the number of Border Patrol agents that those sectors detailed to USMS and USAO locations, as well as any changes in the number and duration of agents detailed to those locations following the April 2017 and April 2018 memoranda. To specifically identify expenditures or personnel for implementing increased immigration-related prosecutions, we reviewed agency documentation, such as documentation from expenditure tracking systems from USMS and Border Patrol. We interviewed agency budget and program officials from USMS’ Offices of Budget Formulation, Forecasting and Analysis, and General Counsel; EOUSA’s Office of Resource Management and Planning; AOUSC’s Office of the Financial Liaison and Analysis Staff; Border Patrol’s Office of Budget Execution; OFO’s Office of Budget Formulation; CBP’s Budget Office; and CBP’s Office of Chief Counsel. In instances where there was no explicit distinction between expenditures or personnel for specifically supporting immigration-related prosecutions and expenditures or personnel used to support other prosecutions, we identified the general account within which immigration-related prosecution costs would be included, and noted that those expenditures include costs for other prosecutions as well. In addition, where agencies identified that they used personnel resources to implement immigration- related prosecutions, we collected related documentation, such as expenditures for temporary details from other parts of the United States to the southwest border, as available, and spoke with district officials by telephone and during our site visits to better understand the use of these personnel resources. We also reviewed Congressional Budget Justifications for fiscal year 2020 to identify expenditures that agencies requested from Congress to support increased immigration-related prosecutions. We conducted this performance audit from May 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix provides additional detail on and characteristics of immigration-related prosecutions in the five U.S. southwest border districts: Arizona, California Southern, New Mexico, Texas Southern, and Texas Western. Each enclosure contains the following information: Description of the district. In this section, we provide a narrative description of the district, including prosecution practices in the district for improper entry, illegal reentry, and alien smuggling cases. We also provide information on the location of the federal courts in the district, U.S. Border Patrol (Border Patrol) sectors in the district, and the federal court circuit in which the district falls. Descriptions of the district reflect practices that were in place as of the date we observed prosecution practices or interviewed knowledgeable officials in the district, which generally ranged from July through November 2018. Cases filed. In this table, we show cases filed by the U.S. Attorney’s Office in the district with a lead charge of alien smuggling, improper entry, or illegal reentry from fiscal years 2014 through 2018. The lead charge is typically the most serious of the charged offenses at the time the U.S. Attorney’s Office files the case, according to Executive Office for U.S. Attorneys (EOUSA) officials. Table 9 describes these offenses. Dispositions. In this table, we show the dispositions of those cases with a lead charge of alien smuggling, improper entry, or illegal reentry from fiscal years 2014 through 2018, as of September 30, 2018, based on the year the case was filed by the U.S. Attorney’s Office. We have included a “pending” column for those cases that did not have a disposition, as of September 30, 2018. Apprehensions and prosecution referrals. In this table, we show prosecution referrals and declinations for those single adults that Border Patrol apprehended from fiscal years 2014 through 2018. In particular, this table includes single adults that Border Patrol apprehended and processed as an individual apprehension, not as a member of a family unit. The U.S. Attorney’s Office decides whether to accept or decline each case that Border Patrol refers for prosecution. We show those cases (each apprehended individual is one case) that Border Patrol referred to the U.S. Attorney’s Office for prosecution and the number of such cases that the U.S. Attorney’s Office declined to prosecute. Individuals whose immigration-related criminal cases are declined by a U.S. Attorney’s Office may be processed in administrative removal proceedings. Nationality and prosecution referrals. In this table, we show the number of prosecution referrals from Border Patrol to U.S. Attorneys’ Offices, by country of nationality. These include both referrals that U.S. Attorneys’ Offices accepted and those that U.S. Attorneys’ Offices declined. We also show, by country of nationality, the percent of individuals who were apprehended and referred for prosecution compared to all those apprehended. For example, if 100 Mexican nationals were apprehended and 50 were referred for prosecution, 50 percent of Mexican nationals apprehended were referred for prosecution. Cases filed by month. In this figure, we show the cases filed with lead charges of alien smuggling, improper entry, or illegal reentry each month from October 2016 through September 2018. We also show the timing of the Attorney General’s April 2017 memorandum, which prioritized immigration enforcement, and the Attorney General’s April 2018 memorandum, which instructed prosecutors on the southwest border to accept all improper entry referrals, to the extent practicable. Volume constraints: Yes; generally 75 improper entry cases per day in Tucson and 30 in Yuma. Percentages may not add to 100 due to rounding. Percentages may not add to 100 due to rounding. Percentages may not add to 100 due to rounding. This appendix provides additional detail on the nationality of defendants for improper entry, illegal reentry, and alien smuggling cases filed in U.S. southwest border federal judicial districts from fiscal years 2014 through 2018. We analyzed the nationality of defendants in cases filed with a lead charge of 8 U.S.C. § 1325 (improper entry), 8 U.S.C. § 1326 (illegal reentry after removal, or illegal reentry), and 8 U.S.C. § 1324 (alien smuggling) from fiscal year 2014 through fiscal year 2018 and for fiscal year 2018. Our analysis of Executive Office for U.S. Attorneys (EOUSA) data indicates that the majority of defendants for cases filed with a lead charge of improper entry and illegal reentry from fiscal year 2014 through 2018 were Mexican nationals. The majority of defendants in cases filed with a lead charge of alien smuggling over this time period were U.S. nationals. Improper entry (8 U.S.C. § 1325): From fiscal years 2014 through 2018, the majority of defendants in cases filed with a lead charge of improper entry were Mexican nationals. Our analysis of EOUSA data indicates that, in fiscal year 2018, the proportion of improper entry defendants who were Mexican nationals was lower than the fiscal year 2014 through 2018 time period, and the proportion of improper entry defendants who were Honduran or Guatemalan nationals was higher than the fiscal year 2014 through 2018 time period. The number of improper entry defendants who were Nicaraguan nationals increased substantially from fiscal year 2017 to 2018—from fewer than 70 defendants in fiscal year 2017 to more than 900 in fiscal year 2018. Figure 13 illustrates the nationalities of defendants with cases filed with a lead charge of improper entry, both from fiscal years 2014 through fiscal year 2018, and in fiscal year 2018. Illegal reentry (8 U.S.C. § 1326): The majority of defendants with cases filed with a lead charge of illegal reentry after removal from fiscal years 2014 through 2018 were Mexican nationals. Our analysis of EOUSA data indicates that, in fiscal year 2018, the proportion of illegal reentry defendants who were Mexican nationals was lower than in the fiscal year 2014 through 2018 time period, and the proportion of illegal reentry defendants who were Honduran or Guatemalan nationals was higher than in the fiscal year 2014 through 2018 time period. Figure 14 illustrates the nationalities of defendants with cases filed with a lead charge of illegal reentry, both from fiscal years 2014 through fiscal year 2018, and in fiscal year 2018. Alien smuggling (8 U.S.C. § 1324): The majority of defendants in cases filed with a lead charge of alien smuggling from fiscal year 2014 through fiscal year 2018 were U.S. nationals. Our analysis of EOUSA data indicates that, in fiscal year 2018, the proportion of defendants for alien smuggling who were U.S. nationals was lower than in the fiscal year 2014 through 2018 time period. Figure 15 illustrates the nationalities of defendants with cases filed with a lead charge of alien smuggling, both from fiscal years 2014 through fiscal year 2018, and in fiscal year 2018. This appendix provides additional detail on cases filed in the 89 non- southwest border judicial districts with a lead charge of 8 U.S.C. § 1325 (improper entry), 8 U.S.C. § 1326 (illegal reentry after removal), or 8 U.S.C. § 1324 (alien smuggling) from fiscal year 2014 through fiscal year 2018. Specifically, this appendix analyzes the number of cases filed with one of these lead charges in every district but Arizona, California Southern, New Mexico, Texas Southern, and Texas Western. Our analysis of Executive Office for U.S. Attorneys (EOUSA) data indicates that illegal reentry cases comprised the majority of immigration-related offenses in non-southwest border districts. From fiscal year 2014 through fiscal year 2018, about 14 percent of cases filed with a lead charge of illegal reentry were filed in non-southwest border districts. Figure 16 illustrates the number and trends in cases filed with a lead charge of improper entry, alien smuggling, or illegal reentry in non-southwest border districts from fiscal years 2014 through 2018. Our analysis of EOUSA data indicates that cases filed with a lead charge of illegal reentry in non-southwest border districts increased by approximately 26 percent between fiscal year 2017 and fiscal year 2018. Illegal reentry cases comprised approximately 91 percent of immigration- related cases filed in non-southwest border districts from fiscal years 2014 through 2018. Table 30 illustrates the number of illegal reentry cases filed by non-southwest border district and fiscal year. In addition to the contact named above, Kathryn Bernet (Assistant Director), Mary Pitts (Analyst-in-Charge), Isabel Band, Dominick Dale, Jan Montgomery, Heidi Nielson, Hiwotte Amare, Michele Fejfar, and Eric Hauswirth made key contributions to this work.", "summary": "In 2017 and 2018, the Attorney General directed federal prosecutors to prioritize prosecutions of immigration-related offenses, including improper entry into the United States, illegal reentry after a prior removal from the country, and alien smuggling, among other offenses. Most individuals prosecuted for such offenses are arrested by DHS's U.S. Border Patrol and referred to DOJ's USAOs for prosecution in federal court. GAO was asked to review the actions DOJ, DHS, and the federal judiciary took in response to the 2017 and 2018 memoranda. GAO reviewed (1) how DOJ prioritized prosecutions of immigration-related offenses in response to the Attorney General's memoranda, (2) what DHS and DOJ data from fiscal years 2014 through 2018 indicate about such prosecutions, and (3) resources that DOJ, DHS, and the federal judiciary used to support increased immigration-related prosecutions. GAO visited three of the five southwest border USAO districts and interviewed DOJ, DHS, and federal judiciary officials by phone from the other two districts. GAO also analyzed U.S. Border Patrol data on its arrests and prosecution referrals from fiscal years 2014 through 2018; analyzed Executive Office for U.S. Attorneys data on its prosecutions from fiscal years 2014 through 2018; and reviewed relevant laws and DOJ, DHS, and federal judiciary policies, operational guidance, and budget data. This is a public version of a sensitive report that GAO issued in August 2019. Information that DOJ, DHS, or the federal judiciary deemed sensitive has been removed. Department of Justice (DOJ) U.S. Attorney's Offices (USAO) in all five districts along the southwest border—Arizona, California Southern, New Mexico, Texas Southern, and Texas Western—have adopted prosecution priorities aligned with the Attorney General's prioritization of criminal immigration enforcement. In particular, all five USAOs prioritized misdemeanor improper entry cases in response to the Attorney General's 2017 and 2018 memoranda. Some USAOs, such as Arizona, were able to quickly increase such prosecutions using existing practices. In other districts, such as California Southern, USAOs had to establish new practices in coordination with other stakeholders in the federal criminal prosecution process—including the Department of Homeland Security (DHS), other DOJ components such as the U.S. Marshals Service (USMS), and the federal judiciary—before they could begin accepting a significant number of improper entry cases. Note: The lead charge is typically the most serious charged offense at the time the case is filed. The number of improper entry cases more than doubled from fiscal year 2017 (about 27,000) to fiscal year 2018 (about 62,000). In fiscal year 2018, about 84 percent of all improper entry cases filed were completed in districts with one-day improper entry court proceedings. In these proceedings, the initial hearing, presentation of evidence, plea, and sentencing took place in one day or less. DOJ, DHS, and the federal judiciary realigned resources to support the prosecution priorities outlined in the 2017 and 2018 memoranda, including personnel and physical space. In addition, agencies temporarily surged personnel to the southwest border. For example, USMS reassigned personnel from other enforcement areas to judicial security duties to support increased immigration-related prosecutions.", "document_type": "gao"}
{"report": "Table 1 provides an overview of tax-time financial products based on information gathered during our review. The tax-time financial products industry consists of four main groups of participants: banks, paid providers of tax preparation services, settlement service providers, and software developers. Providers of tax preparation services include paid tax return preparers or electronic return originators (ERO). Not all tax preparers are EROs, but because IRS generally requires returns to be filed electronically for tax preparers filing more than 10 returns, tax preparers generally work with or for an ERO that also may be a tax preparer. Paid preparers and EROs offer their services in-person, on the Internet, or through software sold to taxpayers. They generally offer different refund disbursement options to taxpayers and may partner with banks to offer tax-time financial products. Software developers provide software needed to file tax returns electronically and offer tax-time financial products through their software to taxpayers. The largest tax preparation companies have their own software that allows them to prepare returns as well as offer tax-time financial products. Applications for the products generally can be completed through the same software used to file the return. Banks provide tax-time financial products. They also may approve and process product applications and perform settlement services (discussed below). Settlement service providers serve as intermediaries in transactions to deliver tax-time products. They work with banks to accept and process applications for tax products; allocate payments due to paid preparers, other providers, banks, and taxpayers; and provide distribution instructions to banks. Some banks have affiliates that perform settlement services, and some banks perform these functions themselves. Figure 1 illustrates the roles of these groups, using the example of a refund transfer transaction. The purpose of federal banking supervision is to help ensure that banks throughout the financial system operate in a safe and sound manner and comply with banking laws and regulations in the provision of financial services. At the federal level, banks are supervised by one of the following three prudential regulators and CFPB: The Federal Reserve supervises state-chartered banks that opt to be members of the Federal Reserve System, bank holding companies and savings and loan holding companies (and the nondepository institution subsidiaries of those organizations), and nonbank financial companies designated for Federal Reserve supervision by the Financial Stability Oversight Council. FDIC supervises all FDIC-insured state-chartered banks that are not members of the Federal Reserve System as well as state savings associations and insures the deposits of all banks and thrifts approved for federal deposit insurance. OCC supervises federally chartered national banks, federal savings associations (federal thrifts), and federally chartered branches and agencies of foreign banks. CFPB has rulemaking authority to implement provisions of federal consumer financial law and enforces various federal laws and regulations governing consumer financial protection. CFPB also examines banks with more than $10 billion in assets and their affiliates and certain nonbanks for compliance with federal consumer financial laws, accepts consumer complaints on topics such as debt collection and other consumer financial products or services, and educates consumers about their rights under federal consumer financial laws. FDIC, the Federal Reserve, and OCC are required to conduct a full- scope, on-site risk-management examination of each of their supervised banks at least once during each 12-month period. The regulators may extend the examination interval to 18 months, generally for banks and thrifts that have less than $3 billion in total assets and that meet certain conditions (for example, if they have satisfactory ratings, are well capitalized, and are not subject to a formal enforcement action). The prudential regulators generally conduct consumer compliance examinations every 12–36 months and Community Reinvestment Act examinations every 12–72 months. The specific timing depends on a bank’s size and its previous consumer compliance and Community Reinvestment Act rating. But the Dodd-Frank Wall Street Reform and Consumer Protection Act transferred consumer protection oversight and other authorities over certain consumer financial protection laws from multiple federal regulators to CFPB. Additionally, for the transferred laws such as Truth in Lending Act (TILA) and Equal Credit Opportunity Act, CFPB has examination and primary enforcement authority for banks with assets of more than $10 billion and any affiliates of such institutions. The three prudential regulators also are responsible for supervising for compliance with federal consumer financial laws for insured depository institutions with total assets of $10 billion or less. For example, they examine depository institutions for compliance with consumer financial laws including the Fair Housing Act, the Servicemembers Civil Relief Act, and Section 5 of the Federal Trade Commission Act. FTC can enforce Section 5 of the Federal Trade Commission Act, which prohibits unfair or deceptive acts or practices affecting commerce, and TILA, which seeks to promote the informed use of consumer credit. TILA requires disclosures about the terms and cost of credit and standardizes the manner in which costs associated with borrowing are calculated and disclosed. FTC can enforce a number of additional statutes against certain entities; they include portions of the Gramm-Leach-Bliley Act, which requires financial institutions, including those providing tax-time financial products, to protect consumer data; the Telemarketing and Consumer Fraud and Abuse Prevention Act, which prohibits telemarketers from making misrepresentations in the sale of goods or services, which could include tax-time financial products; and the Military Lending Act, which provides important protections for servicemembers and their dependents seeking and obtaining certain types of consumer credit, including refund anticipation loans. The Office of Professional Responsibility within IRS is responsible for ensuring all tax practitioners (defined as certified public accountants, attorneys, enrolled agents, enrolled actuaries, appraisers, and enrolled retirement plan agents) and other individuals authorized to practice before IRS adhere to regulations relating to Circular 230, which governs practice before IRS. According to IRS, IRS is neither involved in offering, nor responsible for, tax-time financial products. Nonetheless, IRS stated that it addresses these types of products on its website because it is important for taxpayers to understand the terms of the loan products, which constitute an agreement between them and the third-party lender. Although IRS is not statutorily required to collect data on tax-time products, according to IRS officials, the agency retains information on use of the products. Specifically, IRS compiles information from tax returns that indicates whether the taxpayer also applied for a financial product. IRS also issues guidance to EROs on reporting these data through its Handbook for Authorized IRS e-File Providers of Individual Income Tax Returns (Pub. 1345). IRS makes the usage data publicly available on its website, and provides it on a biweekly basis to industry participants that are members of an IRS working group on security issues. In addition to researchers and consumer advocacy groups, federal entities also use these data, including the National Taxpayer Advocate, who leads IRS’s Taxpayer Advocate Service—an independent office in IRS whose objectives include mitigating systemic problems that affect large groups of taxpayers. As industry data on product use are generally limited, agencies and researchers rely on IRS for this information. Refundable tax credits include the Earned Income Tax Credit (EITC) and the Additional Child Tax Credit (ACTC). The credits are termed refundable because, in addition to offsetting tax liability, any excess credit over the tax liability is refunded to the taxpayer. EITC provides tax benefits to eligible workers earning relatively low wages. For tax year 2018, the maximum EITC amount available was $6,431 for taxpayers filing jointly with three or more qualifying children, and $519 for individuals without children. In 2017, EITC provided more than $65 billion to about 27 million taxpayers. ACTC is the refundable portion of the Child Tax Credit and provides tax relief to low-income families with children. The Protecting Americans from Tax Hikes Act of 2015 (PATH Act) made several changes to the tax law. One of its provisions stipulates that funds owed taxpayers claiming EITC or ACTC refunds for a tax year cannot be released before February 15 to allow IRS time to review these returns for potential fraudulent activity. This change became effective on January 1, 2017. For the 2018 tax filing season (January through April 2018), refunds for taxpayers who claimed these tax credits were not available in bank accounts or prepaid cards until February 27, 2018. IRS data on tax-time financial products for 2016–2018 do not accurately reflect product use and IRS has not updated reporting guidance to tax preparers. IRS data for 2008–2016 and information from industry participants and a consumer advocacy group’s reports suggest that trends in the market for tax-time financial products include the decline of refund anticipation loans and that refund transfers became the most used product. Industry data also indicate that product fees for refund transfers increased in 2018; multiple other fees can be associated with tax-time products. New tax-time products and product features continue to be introduced. Data collected by IRS are the primary source of information on the use of tax-time financial products and are used by federal entities, policymakers, regulators, researchers, and consumer groups. However, we identified some limitations in the IRS data related to use of refund anticipation loans, refund advances, and refund transfers. Despite limitations with IRS data on product use by tax year, our analysis of multiyear trends from these data, supplemented with data collected by the National Consumer Law Center and from Securities and Exchange Commission filings, suggests that use of refund anticipation loans declined, the refund advance was introduced while refund transfers have become the most used tax-time product. Applications for refund anticipation loans declined sharply from 2010 to 2012, according to IRS data and consumer groups reports. According to a 2010 study, the volume of refund anticipation loans peaked in 2002 with 12.7 million taxpayers. Volume began to decline at a faster rate between 2010 and 2011. According to a report by the National Consumer Law Center and the Consumer Federation of America, banks stopped offering the products in 2012 after the loans came under the scrutiny of federal banking regulators. IRS data continued to show use of refund anticipation loans after 2012 but with banks out of the market for refund anticipation loans, it is unclear what types of financial institutions were offering the loans. Consumer advocates with whom we spoke agree that nonbank lenders such as payday lenders likely offered the loans; however, we were not able to identify any. The consumer advocates, researchers, and industry participants with whom we spoke also were not able to provide us with any current information about these lenders. The IRS Taxpayer Advocate Office, the Financial Crimes Enforcement Network, and consumer advocates have long raised concerns about refund anticipation loans. For example, in 2007 the National Taxpayer Advocate expressed concerns about how the loans were offered to consumers and whether consumers adequately understood the product. Consumer advocates questioned the high interest rates the loans could carry, how loan fees reduced EITC benefits taxpayers received, and the ramifications of borrower default. In a 2008 advance notice of proposed rulemaking, IRS and the Department of the Treasury also shared concerns that refund anticipation loans offered tax preparers an incentive to fraudulently inflate refund claims and to market the loans to taxpayers who might not understand the full cost of the product. Banking regulators raised concerns as well. OCC and FDIC noted consumer protection and safety and soundness risks to banks that offered refund anticipation loans. FDIC encouraged consumers to have tax refunds directly deposited into their own bank accounts and raised concerns about other options that claimed to speed up a refund for a sizable cost, according to FDIC officials. The Office of Thrift Supervision, which had supervisory authority over federal thrifts at the time, ordered a medium-sized thrift to cease making refund anticipation loans in 2010. In part due to concerns expressed by OCC, national banks stopped offering the loans by 2010 and FDIC-supervised banks stopped offering them by 2012. An IRS decision also contributed to FDIC enforcement actions on refund anticipation loans. Before 2011, IRS used a tool called the debt indicator that acknowledged whether any of a taxpayer’s refund could be used to pay certain outstanding debts. IRS provided the debt indicator to tax preparers at the time the taxpayer’s return was filed electronically. Banks used the debt indicator in their underwriting tools to help determine a borrower’s likelihood of loan repayment. FDIC determined that without the debt indicator, a bank would have to develop and adopt a more robust underwriting process to make these loans in a safe and sound manner. According to FDIC, IRS’s elimination of the debt indicator created a safety and soundness concern because it removed a key data element used for determining a borrower’s ability to repay. Losing this information increased the risk of loss for lenders and at that time helped inform FDIC’s consent orders with two banks under its supervision to stop offering refund anticipation loans. In 2011 (the first tax season without the debt indicator), the number of returns with a refund anticipation loan indicator reported by IRS decreased to 1.17 million from 6.9 million in the prior year. IRS data continue to show use of refund anticipation loans after 2012, albeit at a much lower volume. For example, in 2016, IRS data show about 468,500 returns with a refund anticipation loan indicator and in 2017 the number appeared to spike to about 1.7 million. However, as discussed earlier, the data for these two years may be misleading because they likely conflate refund anticipation loans with refund advances. In 2018, IRS created a separate reporting category for refund advances and the 2018 data show about 356,000 returns with a refund anticipation loan indicator as of October 2018. Use of refund transfers—which allow for direct deposit of refund checks through temporary accounts that banks open for taxpayers—far exceeded use of refund anticipation loans and refund advances since 2008, according to IRS data. The number of taxpayers who used a refund transfer more than doubled from 2008 through October 2018 to exceed 21 million. As banks stopped offering refund anticipation loans in 2012, refund transfers (also known as refund anticipation checks) began to increase. Unlike other tax-time financial products generally only available early in the tax season (which generally runs through mid-April), refund transfers are usually available after April. However, IRS data on refund transfers since 2016 have limitations. Although a refund transfer is not required to get a refund advance, a number of industry experts told us that almost all taxpayers who apply for a refund advance also apply for a refund transfer. But because tax preparers could select only one product indicator when reporting use of tax-time financial products, they could report a refund advance or a refund transfer, but not both. As discussed previously, IRS made changes in 2018 to allow preparers to add information about other product use but has not issued explanatory material about the changes. In 2016, a few banks began offering refund advances to taxpayers. Refund advances are no-fee, nonrecourse loans. It is difficult to determine usage trends for this product, although available data indicate an increase in use from 2016 to 2017. First, accurate IRS data on refund advances are not available for 2016 and 2017 because IRS did not provide an option for tax preparers to report refund advance products. As previously discussed, IRS added a separate reporting category for refund advances in 2018. As of October 17, 2018, IRS data show about 1.65 million returns with a refund advance indicator. Second, publicly available data from industry and other sources (consumer advocacy and research organizations) are limited. According to data reported by the National Consumer Law Center, major tax preparation companies facilitated the sale of about 365,000 refund advances in 2016. According to industry sources, use increased to about 1.63 million in 2017, when one of the largest tax preparation companies began offering refund advances. Industry data for 2018 were not yet publicly available at the time of this report. Third, taxpayers often obtain refund advances and refund transfers in tandem. But as discussed previously, IRS reporting indicators did not include an option for reporting use of multiple products until 2018. Use of refund advances also may have increased in 2017 because tax preparers increased the size of the advances. One lender that offers refund advances to tax preparers told us that the driving factor in demand for refund advances was the available loan amount. The maximum advance amount that tax preparers offered taxpayers in 2016 was $750. In 2017, the maximum increased to $1,300. Most industry participants and consumer groups told us that they believe that provisions of the PATH Act requiring IRS to delay issuance of EITC or ACTC returns and associated refunds until after February 15 led to an increase in demand for refund advances. They said that the delay puts pressure on taxpayers eligible for EITC or ACTC who depend on getting their refund early in the tax season (a refund advance can help mitigate the impact of this delay). Others stated that an increase in demand due to the PATH Act is possible, but the correlation between the two cannot be determined. One industry provider suggested that increased demand for refund advances also could be the result of marketing by tax preparation companies. Our analysis of publicly available data about product fees for refund transfers showed that fees increased in 2018. In particular, our analysis of fee data collected by the National Consumer Law Center shows that in 2014–2017 refund transfer fees charged by paid tax preparers remained generally unchanged at between $32.95 and $34.95. According to fee information we were given during our undercover visits, paid tax preparers generally charged their customers $39.95 or $49.95 during the 2018 tax filing season for a refund transfer that sometimes included both federal and state tax refunds. In one case the fee was $65, which included a paper check disbursement. Also in 2018, we found that online providers of tax filing services and software charged online filers who prepared their own returns between $12 and $39.99 for a refund transfer. According to our analysis, factors that can affect the fee a taxpayer pays for a refund transfer include the following: Filing method. Our review of providers’ websites shows that taxpayers who filed their own returns online using preparer software paid an average fee of $31.13 in 2018, which was lower than the $39.95 or $49.95 that paid preparers charged their customers. Disbursement method. The manner in which the taxpayer chooses to receive a tax refund may affect the fee. For example, our review of industry literature indicates that one bank set the fee at $29.95 if the refund was disbursed to a prepaid card offered by an affiliate vendor or at $39.95 if the refund was directly deposited or disbursed as a check. Another bank gave tax preparers the option to offer a free refund transfer for disbursement onto a prepaid card, $15 for a direct deposit, or $20 for a paper check. Incentives offered to tax preparers by banks. Incentives from banks for tax preparers can increase fees for taxpayers. Our review of banks’ promotional materials for tax preparers also indicates that some bank providers offer tax preparers different fee structures for a product—that is, the preparers can charge a higher fee to earn a rebate. For example, one bank offered a tax preparer the option to provide a refund transfer to clients for $39 (which includes an $8 incentive paid to the tax preparer) or for $29 (no incentive payment). On their websites, two banks marketed the no-incentive option to tax preparers as a way to be competitive (by offering low-cost options to their customers). Using a refund advance. According to a report by the National Consumer Law Center, one bank set a higher fee for a refund transfer if taxpayers also applied for a refund advance. When taxpayers used only a refund transfer, the fee was $29.95 for the federal refund and an additional $9.95 for the state refund, for a total of $39.90. If the taxpayer also applied for a refund advance (a no-fee product), the refund transfer fee was $44.95. Thus, taxpayers paid $5.05 more for a refund transfer if they also received a refund advance. Our analysis found that, in addition to the product fee, taxpayers may be charged other fees when they use a refund transfer. State refund transfer. In some cases, the refund transfer fee covered the deposit of a federal and a state refund. In other cases, the fee only covered the federal refund. In these cases, if the taxpayer received a state refund, the tax preparer charged an additional fee of $10 or $12. Disbursement services. According to documentation we reviewed, a tax preparer may charge an additional fee of $25 if taxpayers choose to get their refund as a paper check or $7 for a cash transfer to a third party. Prepaid card use. The long-term use of prepaid cards used to disburse a refund may add to the overall cost of getting a tax product. We reviewed cardholder agreements and fee schedules for several prepaid cards commonly used to disburse funds from a tax refund and found they generally carry monthly fees of about $5. The issuer of the prepaid cards also may charge consumers a fee every time they access cash at automated teller machines, deposit more money onto the card, or do not use the card for a certain period of time. Software fees. Companies that design tax preparation software may charge a fee or fees associated with the tax product. Taxpayers may pay one or more of these fees when they use a refund transfer to receive their tax refund. The bank deducts these fees from the taxpayer’s refund after receiving funds from IRS or the state taxing authority. The fee categories are technology fee (up to $18 in our review), a transmission fee that may be a fixed amount (such as $2) or a variable amount, and a processing fee of $6. To determine how the fees associated with a refund transfer can affect the total tax preparation fees a provider may charge a taxpayer, we reviewed fee data we collected. We then identified the types and totals of fees generally associated with tax products and created four possible scenarios based on this analysis (see fig. 2). We designed two scenarios with online self-filers (taxpayer uses a refund transfer and taxpayer does not use a refund transfer) and two scenarios with paid preparers performing the filing (taxpayer uses a refund transfer and taxpayer does not use a refund transfer). Recent and emerging developments in the market for tax-time financial products include higher loan amounts and new products, according to our analysis of selected tax preparers’ websites and marketing materials, and information we were given during our undercover visits. For example, in 2018 refund advances became available to online filers. They previously were offered only to taxpayers who obtained paid tax preparation services in person (at a “storefront”). The maximum amount for a refund advance has continued to increase. In 2016, the maximum loan amount available to a taxpayer was $750. In 2018, the maximum loan amount available was $3,250 and for 2019, one preparer has offered an advance of up to $3,500. One industry participant told us that the industry in general is in a race to increase borrowing limits to remain competitive and attract more customers. In 2018, banks offered a new product that combines the features of a refund anticipation loan and a refund advance. The product allows the taxpayer to apply for a refund advance (up to a fixed amount) with no fee or finance charges, the option to apply for an additional loan with a fee (similar to a refund anticipation loan), or a combination of the two products known as a hybrid. For 2018, two banks offered this additional loan (not to exceed $1,000) at an annual percentage rate of 29.9 percent. For 2019, one bank offered taxpayers the option of a no-fee advance of up to $1,000, or an interest-bearing loan of $2,000, $3,000, or $5,000 based on the expected refund. The interest-bearing loans would carry an annual percentage rate of 26.07 percent in addition to a fee of $30–$75, depending on the loan amount. Also for 2019, one national tax preparation company has offered the option of a no-fee advance of up to $3,500 or a fee-based advance of up to $7,000, which would carry an annual percentage rate of 35.9 percent. In addition, demand for refund transfers has increased among online self- filers. As more people file their own tax returns by using web-based software, the number of refund transfers used by self-filers may continue to increase. Because few tax preparers offer refund advances to online self-filers, taxpayers are still more likely to get a refund advance from a paid tax preparer. Finally, issues relating to the applicability of TILA disclosure requirements to refund transfers could affect the market for tax-time products. According to representatives of two consumer advocacy organizations, deferment of tax preparation fees until the refund is received constitutes an extension of credit; therefore, refund transfers should be treated as loan products. Tax preparers and a policy research and education organization with whom we met do not believe that refund transfer fees meet the definition of a loan. Should regulators decide that a refund transfer constitutes an extension of credit, and would therefore be a credit transaction with a finance charge, refund transfers would become subject to provisions of TILA. These changes could affect taxpayers’ access to this product as well as product pricing. According to Securities and Exchange Commission filings of some tax preparers, if refund transfers were successfully characterized as such, the additional requirements and costs could limit their ability to offer these products to clients. Refund advances were promoted by providers as a fee-free, interest-free credit product, and thus TILA disclosure requirements are generally not considered applicable for them. However, new interest-bearing credit products announced for 2019 may be subject to consumer protection regulations. Using FDIC data, we conducted a multivariate regression analysis to examine the relationship between economic and demographic variables and tax-time financial product use. This approach allowed us to test the significance of the relationships between each variable and the likelihood of using tax-time financial products, while controlling for other factors. Lower-income households were more likely to use tax-time financial products than higher-income households, particularly when they used paid tax preparers to file their taxes, according to our analysis of 2017 FDIC data. More specifically, we estimated that households with incomes between $20,000 and $39,999 were more likely to use tax-time financial products to receive their tax refunds more quickly through paid tax preparers than households with incomes of $60,000 or more. For example, we estimated that households with incomes between $20,000 and $29,999 were 34 percent more likely to use tax-time financial products than households with incomes of $60,000 or more; and households with incomes between $30,000 and $39,999 were 61 percent more likely to use the products than households with income of $60,000 or more. Moreover, our analysis of FDIC data suggests that households that received EITC were more likely to use tax-time financial products, compared to households that did not receive EITC. Our results also suggest that wealth, as measured by homeownership, was associated with the household decision whether to use tax-time financial products. Homeowners were 34 percent less likely to use tax- time financial products than non-homeowners, controlling for other factors. Households of some minority groups were more likely to use tax-time financial products when filing tax returns than white households. For example, using FDIC data, we estimated that African-American households were 36 percent more likely to use tax-time financial products than white households after controlling for other factors. Other research (a 2013 study) found that African Americans were more likely to use refund anticipation loans than white individuals. According to our analysis of 2016 IRS data, which included information about tax-time financial product use and locality, use of tax-time financial products was more concentrated in some areas of the South and the West (see fig. 3). Our analysis of FDIC data further suggests that other characteristics associated with use of tax-time financial products include age and household type. For example, households headed by younger persons (15–39 years old) were more than twice as likely to use the products as households headed by older persons (60 or older), controlling for other factors. Households headed by single adults with families were more likely to use tax-time financial products than households headed by married couples. For example, according to our analysis of FDIC data, we estimated that households headed by unmarried females with families were 76 percent more likely to use tax-time financial products than households headed by married couples, controlling for other factors. Using IRS data from 2016, we found that a higher proportion of product users filed as unmarried heads of household, compared to the general tax filing population. Among those who used tax-time financial products, about 39 percent filed as single, 22 percent filed as married, and 37 percent as unmarried heads of household. Reasons to use tax-time financial products include more quickly obtaining cash from the expected tax refund, not having to pay tax preparation fees out of pocket, and obtaining cash more cheaply than with alternative short-term funding options, according to our review of federal and industry reports. Taxpayers generally might have to wait weeks for refunds from IRS: Taxpayers who file paper returns can expect to receive their refund about 6–8 weeks after the date on which IRS receives their return, according to IRS guidance. Taxpayers who file electronically generally can expect to receive their refunds within 21 days, or faster if they opt to have refunds deposited directly into their bank accounts. As previously discussed, IRS must delay payments of refunds on which EITC, ACTC, or both are claimed until at least February 15 of each year. Effectively, the refunds might not be disbursed to bank accounts (or prepaid cards) of tax filers until the end of the month. In contrast, users of tax-time products can obtain cash very quickly. For example, refund advance recipients generally receive loan funds within 24 hours of applying, and in some instances within the same hour they apply, according to selected tax preparer documents and websites that we reviewed. Refund transfer products also allow those who do not have the option of directly depositing refunds into a temporary account instead of waiting longer to receive a paper check. According to our analysis of IRS data from 2016, tax-time financial product users were more likely than other taxpayers to receive their tax refunds by direct deposit. Taxpayers may use tax-time financial products because they need cash quickly. Studies we reviewed found that product recipients tend to have pressing financial obligations. One study’s review of available literature from 2010 found that product recipients tend to live paycheck-to- paycheck or lack sufficient savings to cover prior, current, or future spending. Another study published in 2010 found that recipients use the products to pay for pressing financial obligations, both expected and unexpected, and for their tax preparation. According to the study, many users of tax-time products become delinquent on rent, utilities, and other expenses during the winter with the expectation that they will be able to pay obligations after receiving tax refunds. As one study found, the annual tax refund represents the largest single cash infusion received all year by about 40 percent of checking account holders. Lower-income taxpayers also use tax-time financial products to defer payment of fees related to tax return preparation, according to federal government and industry reports that we reviewed. Tax preparation fees vary greatly based on the tax forms used, including the EITC worksheet. One of the largest national tax preparation chains reported that its average tax preparation fee was between $205 and $240 in 2017. Free Filing Services The Internal Revenue Service (IRS) offers the following free filing services: Fillable forms. IRS offers forms that can be completed online and electronically submitted to IRS. The forms are available without age, income, or residency restrictions. Free file software. IRS, in partnership with the Free File Alliance (members of the tax software industry), provides free online filing options to eligible taxpayers. Twelve leading tax software providers make a version of their products available exclusively at IRS.gov for taxpayers with an adjusted gross income up to $66,000 (in 2018). Volunteer Income Tax Assistance. The program provides free basic income tax preparation with electronic filing by IRS- certified volunteers to qualified individuals, including to persons who earn $55,000 or less, have disabilities, or have limited proficiency in English. Tax Counseling for the Elderly. The program provides free tax preparation by IRS- certified volunteers to all taxpayers, particularly those 60 or older. Program volunteers specialize in pension and retirement-related issues unique to seniors. Consumers may perceive any costs associated with tax-time financial products and tax return preparation as lower than they actually may be because the costs are not paid out of pocket. Fees for the products and tax return preparation are deducted from the refund before it reaches the consumer. In general, studies have found that the transparency of a payment method affected the payer’s willingness to spend. One consumer advocacy organization representative posited that paying for tax-time financial products and tax preparation from a refund makes consumers less sensitive to the real cost of tax-time products and preparation services. Instead of using tax-time financial products to defer payment of tax preparation fees, lower-income taxpayers can access free filing services through several IRS programs (see sidebar). However, these options do not allow taxpayers to use tax-time financial products to access refunds faster. IRS estimates that about 70 percent of taxpayers are eligible to access its free filing software, and we estimated about 3 percent of taxpayers use this service. According to IRS officials, while IRS does not have a marketing budget to promote the free file programs, the predominant reason so few taxpayers use them is because there are many free tax preparation options on the market, such as tax preparation software. Taxpayers also may use paid tax preparers because they do not think they can fill out tax returns on their own, believe that preparers will help them receive higher refunds, or both, according to federal government and industry reports we reviewed. For taxpayers who did not use tax-time financial products, we did not find a clear association between paid tax preparation and higher average refunds. On the other hand, for taxpayers who used tax-time financial products, we found that average tax refunds were higher for taxpayers who filed through paid tax preparers than for taxpayers who self-filed online (see table 2). According to IRS data, nearly all taxpayers who used refund loan products filed their taxes through paid tax preparers, as refund advances were not available online until the 2018 tax filing season. There may be various reasons for the association between higher refunds, paid tax preparation, and product use. Those who use tax-time financial products tend to be eligible for tax credits such as EITC, which can increase the size of tax refunds. Fifty- four percent of EITC claimants used a paid preparer. However, a 2017 study found that the combination of paid tax preparation and tax-time financial product use was associated with relatively high incorrect tax payments (specifically, overpayments of EITC compared to online self- filing and product use or no product use). Furthermore, our analysis of IRS data found that taxpayers who used tax- time financial products received higher refunds on average than those who did not use tax-time financial products, regardless of tax filing method—although other factors might explain this association. For example, taxpayers who have high refunds have a greater incentive to use the products than taxpayers who have relatively small refunds or owe taxes. For lower-income taxpayers, tax-time products generally provide more cash at a lower cost than other small-dollar loan alternatives such as payday loans, auto title loans, and pawnshop loans, according to our review of federal government and industry reports. The amounts of alternative loan products are based on the value of the collateral the consumer provides. Average loan amounts are $150 for pawnshops, about $500 for payday loans, and under $1,000 for automobile title loans, according to industry statistics and CFPB and other studies. In contrast, refund advances were offered for up to $3,250 for the 2018 tax filing season. Furthermore, the alternative products generally include fees, unlike refund advances. For example, fees for payday loans generally range from $10 to $30 per $100 borrowed. Automobile title lenders generally charge a fixed price per $100 borrowed, with a common fee limit of 25 percent of the loan per month. In contrast, refund advances are offered at no cost to the consumer. Tax-time financial products also may be easier to access because, unlike alternative loans, they generally can be obtained without regard to credit history. However, tax-time financial products generally are only available during tax season. Loans provided by nonfinancial companies (often called fintech firms) are another source of short-term financing. However, fintech firms generally provide much larger loan amounts than tax-time financial products, and include fees, unlike refund advances. The federal banking regulators oversee banks that offer tax-time financial products and IRS sets standards of practice for certain service providers (including some tax preparers). While our nongeneralizeable review found that selected banks and tax preparers generally followed existing OCC and IRS disclosure requirements, some tax preparers’ disclosure practices may present challenges for consumers trying to compare product options. FDIC, the Federal Reserve, or OCC are responsible for the safety and soundness supervision of banks within their authority (which offer tax-time financial products) and may have supervisory authority over third-party service providers (which provide settlement services). We identified five banks that partnered with several national tax preparation chains in recent years to offer tax-time financial products (refund transfers and refund advances). Of the five banks, FDIC supervised one medium-sized and one small bank, OCC supervised two medium-sized banks, and Federal Reserve supervised one medium-sized bank. As previously discussed, FDIC, the Federal Reserve, and OCC are to conduct full-scope, on-site risk-management examinations of each of their supervised banks at least once in each 12–18 month period. FDIC officials told us that its regular safety and soundness examinations may include an examination of the bank’s tax-time financial product offerings. OCC officials told us that they examine tax-time financial products in every annual examination of the banks they supervise that offer these products. Because the five banks each has total assets of less than $10 billion, the three regulators also are responsible for enforcing compliance with federal consumer financial laws (such as TILA and the Electronic Fund Transfer Act) that govern disclosure requirements for certain tax-time financial products. Officials from the regulators told us that they received few complaints about tax-time financial products offered by their supervised banks. We discuss the disclosure requirements and compliance with the requirements in more detail later in this section. The regulators’ consumer compliance examiners also may review a bank’s tax-time financial products—if, for example, a bank offers a new product or there are a number of consumer complaints about a current product. Examiners employ a risk-focused approach with a focus on consumer harm in selecting products to evaluate for compliance with applicable consumer laws and regulations. Furthermore, compliance examiners may decide, based on the potential for consumer harm and a bank’s compliance management system, that there is enough residual risk to scope the product into the examination. FDIC officials said that a bank with a lot of activity in the market for tax-time financial products would have to assure examiners that it had performed appropriate due diligence. Regulators also can take other oversight actions, ranging from enforcement to raising awareness among consumers. In 2015, CFPB took an enforcement action, along with the Navajo Nation, to ban an owner of four tax preparation franchises from the market and levy civil penalties for understating refund anticipation loan rates and deceiving customers about the status of their tax refunds. Our search of CFPB’s complaint database did not identify any consumer complaints on tax-time financial products. CFPB published a blog post in February 2018 that describes the different tax-time financial product options and the process for obtaining them, and cautions consumers to consider all fees, charges, and timing associated with the products. FTC staff we interviewed told us that supervision authority over many financial services providers has been given to CFPB, but that FTC still has the authority to enforce many financial statutes and rules, including rules administered by CFPB. FTC brought an enforcement action in 2017 against an online tax preparation provider alleging that it failed to secure consumer accounts. FTC officials also told us that, while they received numerous complaints on tax-related issues, FTC’s complaint database does not separately classify complaints based exclusively on tax-time financial products. FTC also has issued guidance to educate consumers regarding tax- related scams and other consumer protection issues that arise during tax time, and to businesses, including tax professionals, to help them detect cyber threats. FTC also co-sponsors a series of educational events for consumers and businesses surrounding tax identity theft awareness week. Software companies we interviewed stated that they are subject to IRS regulations relating to electronic filing of tax returns. Software developers provide tax software to tax preparers so that they may file tax returns electronically and assist taxpayers in obtaining tax-time financial products. One software company told us that this involves working with IRS to ensure that returns can be electronically submitted, IRS can receive data, and the software is in compliance with IRS’s required data schemas. IRS officials said that IRS does not monitor or have direct oversight authority over tax-time financial products, but requires some paid tax preparers to meet standards of practice or other requirements. The extent to which IRS has oversight over paid preparers depends partly on whether the preparer is a tax practitioner or unenrolled preparer. Tax practitioners are subject to regulations (Circular 230) that establish standards of practice. For example, practitioners must return tax records to clients, exercise due diligence in preparing tax returns, and submit records and requested information to IRS in a timely manner. IRS officials told us that they monitor the suitability of these practitioners and their adherence to the rules. Additionally, certain tax practitioners known as enrolled agents generally are required to pass a three-part examination and complete annual continuing education, while attorneys and certified public accountants are licensed by states but are still subject to Circular 230 standards of practice if they represent taxpayers before IRS. Alternatively, unenrolled preparers—the remainder of the paid preparer population and the majority of paid preparers—generally are not subject to these requirements. In 2011, IRS issued final regulations to establish a new class of registered tax return preparers to support tax professionals, increase confidence in the tax system, and increase taxpayer compliance. However, the U.S. District Court for the District of Columbia ruled in 2013 and the U.S. Court of Appeals for the District of Columbia Circuit affirmed in 2014 that IRS lacked sufficient authority to regulate all tax preparers. IRS officials also told us that all authorized IRS e-file providers have to follow certain requirements to be able to file tax returns electronically. We found selected authorized IRS e-file providers generally followed the requirements established by IRS on the disclosure of product fees, and banks generally followed the disclosure guidance relating to tax-time financial products issued by OCC. (We conducted nongeneralizeable reviews of website content, industry documents, and disclosures made during our undercover visits.) Two of the five banks we reviewed are regulated by OCC. One of the two FDIC-supervised bank and the Federal Reserve-supervised bank told us that they voluntarily follow OCC guidance. More specifically, IRS established the following disclosure requirements for authorized IRS e-file providers, generally known as EROs, that relate to tax-time financial products: EROs must obtain taxpayers’ written consent before disclosing any tax return information to other parties in relation to an application for a tax product. EROs must ensure taxpayers understand that if they use a tax product, the refund will be sent to the bank and not to them. If taxpayers choose to use a fee-based loan, EROs must advise that the product is an interest-bearing loan and not an expedited refund. EROs must advise taxpayers that the bank may charge them interest, fees, or both, in the case of any shortages on the refund. EROs also must disclose all deductions to be made from the expected refund and the net amount of the refund. In 2015, OCC issued risk-management guidance for national banks that offer tax refund-related products. This guidance advises that banks should specify to customers, as applicable, the total cost of the tax product, separately from the tax preparation cost; that total costs will be deducted from and reduce the refund amount; that tax refunds can be sent directly to the taxpayer without the additional costs of a tax product; that customers with deposit accounts can receive their refund without incurring fees through direct deposit in about the same time as it would take to receive a tax refund-related product; and the ongoing periodic maintenance and transaction fees related to any product intended for long-term use. In addition, OCC’s guidance establishes that banks should clearly disclose all material aspects of the product in writing before the consumer applies or pays any fees for a tax-time financial product. Also, representatives of the American Coalition for Taxpayer Rights, a group representing the leading tax preparation, tax software, and bank providers, told us that its members signed a joint statement with attorneys general from six states on disclosure practices for refund transfers. The member providers agreed to explain to taxpayers the different options for filing and receiving a tax refund, including no-cost options, and the associated costs and features of each option. The providers also agreed to disclose the optional nature of the products, the timing of the refund, and to present the disclosures in a clear and conspicuous manner understandable by a reasonable consumer. Our nongeneralizeable review of documents received from selected banks and tax preparers found disclosures generally followed OCC guidance or IRS requirements, respectively. However, our review of these documents and selected tax preparer websites also found—and our undercover visits of selected tax preparers suggested—that the level of transparency on product fees varied and product fees and information were not always clearly disclosed. Bank documents were more likely than information provided by paid preparers (in person or online) to include more disclosures about the fees and terms of tax-time financial products. For example, of the 12 bank documents we reviewed, all disclosed that funds would be sent to the bank if the taxpayer used a tax product. Almost all the bank documents disclosed the fees associated with the product and all disclosed that the fees would be deducted from the refund. In contrast, while written disclosure is not required, less than one third of ERO documents disclosed that the taxpayer using a tax-time financial product would receive funds from the bank instead of IRS. However, almost all the documents are presented to taxpayers after returns have been prepared and preparers have determined that taxpayers qualified for a product. The timing of when a tax preparer makes these disclosures would pose a challenge for taxpayers looking to compare prices for different providers. That is, they would not learn of the total fees—partly because the paid preparer could not determine the amount of some tax preparation fees until well into the preparation of the tax return. A taxpayer trying to determine the cost of using a tax refund to pay for online tax preparation services only would be able to compare the prices of two of the eight online providers we reviewed. The remaining six did not disclose this fee in a prominent way—with some disclosures made in small print or requiring navigation through several pages after the product page—or at all. A taxpayer choosing to file taxes using the services of a paid tax preparer in a brick-and mortar-location, and opting to use the refund to pay for tax preparation fees, would be unlikely to be able to compare prices among different providers. For example, during six of our undercover visits, our investigators explicitly requested literature on product fees. However, the preparers stated that they did not have the literature available or only provided us with business cards and other promotional material. Our analysis shows that providers do not consistently explain products or disclose fees to taxpayers. For example, providers told us, and industry documents show, that a refund transfer is not required to get a refund advance. However, during our site visits, tax preparers tied the use of a refund transfer to a refund advance four out of five times. In two of these cases, the tax preparer included the fee for a refund transfer as part of processing an advance product, while in another two cases the tax preparer said that a refund transfer was required with the advance. Also, during our site visits, three of the nine tax preparers did not disclose the cost of a refund transfer. Appendix III provides more information on our analysis of bank and tax preparer disclosure practices. According to industry participants, only taxpayers expecting a refund can qualify for a tax product; consequently, the tax preparer generally cannot determine whether the taxpayer qualifies until after the tax return is completed. Once this is determined, the tax preparer must request the taxpayer’s consent to offer a tax product. EROs with whom we met told us they may disclose fee information at various points throughout the process of tax preparation, and do so verbally or through their in-store computer interface. Bank disclosures are provided to the taxpayer before the product application has been submitted. Some researchers and representatives from consumer advocacy organizations with whom we met were concerned about the timing of disclosures of tax-time financial product fees. Consumer advocates said disclosures given to taxpayers were inadequate, unhelpful, or timed in such a way as to prevent meaningful comparison shopping. Specifically, one consumer advocacy organization said that taxpayers they serve do not understand the fees associated with filing through preparers. Representatives from another consumer advocacy organization said that taxpayers do not know the total cost for tax-related financial products and services until they already have taken steps to file their returns. In its 2017 Report to Congress, the National Taxpayer Advocate recommended that IRS require all e-file participants offering tax-refund financial products to provide a standard “truth-in-lending” statement to help taxpayers better understand the terms of the refund anticipation loan product. IRS did not adopt the National Taxpayer Advocate’s recommendation but agreed that e-file providers should be transparent about the costs associated with the loan products offered to taxpayers as part of the return preparation process. As previously discussed, courts have determined that IRS does not have sufficient authority to regulate individuals who are solely tax preparers and not licensed by IRS—in effect, the majority of the paid preparer population. Previously, we asked Congress to consider legislation granting IRS the authority to regulate paid tax preparers, if it agreed that significant paid preparer errors existed. As of March 2019, this Congressional action we have recommended remains open. The lack of consistency about the timing of fee disclosures for tax-time financial products may add to the rationale for Congress to consider regulating preparers. Such statutory authority could allow IRS to require that tax preparers make tax-time financial product disclosures or ensure meaningful transparency in the sale of the products. For lower-income taxpayers with pressing financial obligations, tax-time financial products can offer an alternative to higher-cost short-term products such as payday loans. Taxpayers can purchase tax-time financial products from many tax preparers; however, according to our review of selected tax preparers and banks, the price and associated fees of these products can vary. And disclosure practices by some paid tax preparers may pose challenges for consumers looking to compare prices for different providers. IRS is an essential source for data on tax-time financial products, but to date IRS has offered limited options to tax preparers for accurately reporting usage of all available tax-time products. Furthermore, IRS has not informed tax preparers about changes made in reporting options and has not informed users of IRS’s product data about known issues with the data. Consequently, data on product usage are not reliable. Improving the quality of data collected on these products would help ensure that federal agencies, policymakers, regulators, consumer advocacy groups, and researchers have quality information to report on tax policy and consumer protection issues and inform their decision-making. We are making a total of two recommendations to IRS. The Commissioner of Internal Revenue Service should communicate data issues regarding the refund anticipation loan indicators for tax years 2016 and 2017 and the refund transfer indicators since tax year 2016—for example, by attaching explanatory material to the dataset. (Recommendation 1) The Commissioner of Internal Revenue Service should improve the quality of tax-time financial product data collected; for example, by allowing authorized e-file providers to indicate more than one type of tax- time financial product for each return or by informing tax preparers of the addition of new product definitions and instructions on how to accurately code the products. (Recommendation 2) We provided a draft of this report to IRS, FDIC, Federal Reserve, OCC, CFPB, and FTC for review and comment. IRS provided written comments, which are reproduced in appendix IV and discussed below. FDIC, Federal Reserve, OCC, CFPB, and FTC provided technical comments, which we incorporated as appropriate. In its comments, IRS concurred with both recommendations, and described how it planned to address them. In response to our first recommendation, IRS stated that it plans to provide the appropriate notations with the datasets. In response to our second recommendation, IRS stated that it plans to pursue programming changes and clarify instructions for tax return preparers to promote accurate coding of refund- related products. We believe that these actions, if implemented, would address our recommendations and improve the quality of data IRS reports on these products. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees and IRS, FDIC, Federal Reserve, OCC, and FTC. This report will also be available at no charge on our website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or clementsm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report (1) describes trends in the market for tax-time financial products and product fees and examines the reliability of IRS data on these trends, (2) describes characteristics of those who use tax-time financial products and factors that influence the decision to obtain the products, and (3) describes regulatory oversight of industry participants and the disclosure of information on product fees and terms. To examine trends in the use of tax-time financial products, we used 2008–2018 Internal Revenue Service (IRS) data compiled from tax filings to determine the types and use of these products. We assessed the reliability of these data by interviewing IRS officials about the controls and quality assurance practices they used to compile these data. We determined the data alone did not provide a reliable count of refund transfers, refund anticipation loans, or refund advances in 2016, 2017, and 2018, but were adequate to suggest general trends when supplemented with other information. To supplement the IRS data, we collected information from reports issued by the National Consumer Law Center, reviewed Securities and Exchange Commission filings for two selected tax preparers, and interviewed representatives from National Consumer Law Center and both tax preparers on the offerings of tax-time financial products. We selected these preparers because they are major providers of tax preparation services and tax products. To identify and review trends in product offerings, we reviewed the websites, promotional materials, and other industry literature including Securities and Exchange Commission filings of a nongeneralizeable selection of four providers of online tax preparation services, three tax preparers with physical locations that also offer services online, and four banks. We also discussed changes in the market and product offerings with nine of the industry providers with whom we met. We accessed provider websites before and during the 2018 tax season. The tax preparation firms were selected because they are national tax preparation chains, and the five banks were selected because they partnered with the national tax preparation chains and major developers of tax preparation software. In addition, we reviewed studies related to these products published by GAO, federal agencies, four consumer advocacy and research groups, and two academic researchers. We used these studies primarily to corroborate findings from our data analysis. We focused on studies from 2010 and later; however, we also reviewed an older report to gain a greater understanding of how the market for tax-time financial products evolved. We identified these studies through expert recommendations and citations in studies. To examine trends in fees for tax-time financial products, we collected fee-related information from several different sources (because of limited publicly available industry data). All of the information cannot be used to generalize our findings to the retail tax preparation industry. Product fees. For 2018, we collected information on product fees from six paid tax preparers and four banks. For tax years 2014 to 2017, we used product fee information as reported by the National Consumer Law Center. For 2018, we also reviewed fee data from six providers of online tax preparation software, two that provide services in person and online, and four that only provide services online. We selected these providers after conducting internet searches and reviewing reports by consumer advocates and federal agencies. Data elements included fees for refund transfers and refund advances. For 2018, data elements also included the dollar amount for the incentives banks offered tax preparers for each refund transfer sold. Ancillary product fees. We collected information on ancillary product fees from four tax preparers, four banks, and three software developers for tax years 2017 and 2018. Data elements included fees for disbursement methods such as prepaid cards and paper checks and other charges related to the use of a tax-time financial product such as technology and transmission fees. Tax preparation fees. We collected information on tax preparation fees from eight tax preparers with physical locations and eight online providers of tax preparation services for 2018. Data elements included fees for federal and state filing. Aggregate fees. We collected aggregate tax-time financial product, ancillary product, and tax preparation fee information from studies issued by consumer protection advocates. We collected the above information from websites, advertising materials, and public filings with the Securities and Exchange Commission of tax preparers, banks, and software developers. To identify some of the demographic and economic characteristics of product users, we used data from the Bureau of the Census and the Federal Deposit Insurance Corporation (FDIC) from 2011, 2013, 2015, and 2017 to conduct a multivariate regression analysis to determine the influence of individual characteristics on the decision to obtain a product. We statistically controlled for various income, education, and demographic factors. While the FDIC data contain a rich set of demographic and economic variables, they include limited data on characteristics specifically related to tax filing. To identify specific tax-filing characteristics associated with product use, we also used a probability sample of data from IRS from the 2014, 2015, and 2016 tax years to calculate the percentages of taxpayers who used tax-time financial products according to various tax-filing characteristics, including tax filing status and tax filing method. We also used the sample data to calculate the percentage of taxpayers who used free filing services, including free file software, programs, and fillable forms. We reviewed documentation on and conducted testing of the data we used and determined they were sufficiently reliable for reporting economic, demographic, and tax-filing characteristics associated with product use. For more detailed information on our analysis of characteristics associated with tax-time financial product use, see appendix II. To better understand user characteristics associated with the decision to obtain a tax-time financial product identified by our analysis, we reviewed relevant federal and industry reports on the financial needs of individuals with characteristics similar to taxpayers who obtained these products. We focused on reports from 2010 and later. We also reviewed our prior studies and studies from the Consumer Financial Protection Bureau (CFPB) on alternative credit products and compared their features and fees to those of tax-time financial products. In addition, we interviewed representatives from consumer groups, four Low-Income Taxpayer Clinics, and IRS’s Taxpayer Advocate Service to obtain their perspectives on characteristics associated with tax-time financial product users. To describe the regulatory oversight of industry participants associated with tax-time financial products, we reviewed relevant federal laws and regulations, and reports and guidance documents from IRS and federal regulators, including the CFPB, FDIC, the Board of Governors of the Federal Reserve System, Office of the Comptroller of the Currency (OCC), and Federal Trade Commission. We inquired about consumer complaint data related to tax-time financial products at the federal regulators and interviewed officials from the federal agencies and representatives from five tax preparation providers, five banks and bank affiliates such as settlement service providers, four consumer advocacy organizations, three software developers, two researchers, one provider of alternative financial services, and one industry group to gain their perspectives on the benefits and risks of the tax-time financial products and how any related concerns were being addressed. The tax preparation firms were selected because they are national tax preparation chains, and the five banks and three software developers were selected because they partnered with the national tax preparation chains. The four consumer advocacy organizations, two researchers, alternative financial service provider, and industry group were selected for their experience and to provide a range of perspectives. To review how product terms and fees are disclosed by tax preparers, in February 2018 GAO investigators acting in an undercover capacity visited a nongeneralizeable sample of nine randomly selected tax preparers in Washington, D.C., Maryland, and Virginia to inquire about tax-time financial products. We selected the two states and Washington, D.C. to ensure a mixture of state and local laws governing the products and providers. From the two states and Washington, D.C., we selected one metropolitan statistical area based on the concentration of product users and the proximity to lower-income households. We randomly selected three individual tax preparers in each of the three metropolitan statistical areas to visit, based on proximity to taxpayers in lower-income households and to ensure a mixture of urban and rural communities and company sizes. We visited offices of large tax preparation chains and single-office tax preparation businesses. Results cannot be used to generalize our findings to the retail tax preparation industry. Our investigators posed as taxpayers seeking tax preparation services who wanted to pay for the tax preparation fees with the expected refund or obtain an advance based on their anticipated tax refund. They requested available documents associated with tax preparation, refund advance and refund transfer products, and different disbursement options and fees. Because GAO investigators did not experience the tax preparation or the product application process, we were not able to assess the timing of any disclosures typically made after the tax return preparation process would begin. In addition, we received some consumer-facing disclosures and product agreements that were typically provided during the product application process from two tax preparers and two banks. We also conducted a content analysis of websites of eight selected tax preparers that offer tax-time financial products. The tax preparers were selected as national providers of tax preparation services with an online presence, and the results are not generalizeable to the retail tax preparation industry. Three of the providers offer tax preparation services online and through physical retail locations and five of the providers offer their services online only. We reviewed these websites to understand the extent to which they disclose fees to the taxpayer for tax preparation services, tax-time financial products, disbursement, and additional products or services, and to review the ease with which these disclosures are accessible. In addition to consumer-facing disclosures we received from providers with whom we met, we searched online for additional disclosures provided by the tax preparers and banks in our review and reviewed seven disclosures from two national tax preparation chains and 12 disclosures from five banks offering tax-time financial products. We then compared the disclosures against IRS and OCC requirements for disclosure for product terms and conditions. IRS established certain disclosure requirements for authorized IRS e-file providers. OCC instructs banks it supervises to make certain disclosures to product consumers. More specifically, we analyzed tax products and fee disclosures obtained from our undercover visits of selected tax preparers, online reviews, and directly from tax preparers and banks to determine the type and timing of disclosures made in these instances and whether they were consistent with IRS disclosure requirements and followed OCC guidance. We conducted this performance audit from July 2017 to April 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We conducted our related investigative work in accordance with standards prescribed by the Council of the Inspectors General on Integrity and Efficiency. This technical appendix outlines the development, estimation, results, and limitations of the econometric model and other data analysis we described in the report. We undertook this analysis to better understand the characteristics associated with the decision to obtain a tax-time financial product. Federal Deposit Insurance Corporation. To assess the characteristics associated with tax-time financial product use, we used data from the Federal Deposit Insurance Corporation’s (FDIC) National Survey of Unbanked and Underbanked Households for 2011, 2013, 2015, and 2017, which is a supplement of the Current Population Survey. We used the following variables on households and heads of households to examine how various demographic and economic characteristics are related to the use of tax-time financial products: Household income. Household type. Homeownership status. Race and ethnicity of the head of household. Educational attainment of the head of household. Age of the head of household. Head of household has children. Household used refund anticipation loan or a tax preparation service to receive a tax refund faster than the Internal Revenue Service (IRS) would provide it in the past 12 months. This is a dummy variable, which equals 1 if the household used products and 0 otherwise. A refund anticipation loan is a tax-time financial product. Based on our interviews and other research reports, refund anticipation loans and other tax-time financial products (including refund anticipation checks) may be used by consumers to get their tax return faster than IRS could provide it. We refer to this variable as “used tax-time financial product” for simplicity in the report, and we explain the relevant caveats and limitations below. This variable is the basis for the sample used for this analysis. See table 3 for the estimated distributions of these variables for all households, as well as households that used tax-time financial products in 2017. We also examined the relationship between the use of tax-time financial products and being unbanked, as well as the association between using tax-time financial products and alternative financial services (those offered outside the banking system). We used additional data from FDIC’s National Survey of Unbanked and Underbanked Households on the following variables: Household used other alternative financial services in the past 12 months, including nonbank check cashing, nonbank money orders, payday loans, and pawn shops. Household used prepaid card(s) in the past 12 months. Household was unbanked in the past 12 months. See table 4 for estimated distributions of household responses to questions related to unbanked status and usage of other alternative financial services for all households, as well as households that used tax- time financial products in 2017. IRS. To further identify tax-filing characteristics associated with tax-time financial product use and trends, we also used data from a probability sample of 2 percent of all electronically filed tax returns from IRS for tax years 2014, 2015, and 2016. In 2016, the sample size was 2,952,418, representing a population of 147,625,598 tax returns. According to IRS, the sample is representative of all electronically filed tax returns for the relevant tax years. In this sample, IRS provided data on the following variables: Tax filing method, including online (self-filed using tax software) or through a paid practitioner (including tax preparers with physical storefronts). Taxpayer used free filing services from IRS, including the Free File program and free fillable forms. Tax filing status, including single, married, and head of household. Disbursement options for tax refunds (direct deposit or paper check) or tax balance due. Tax refund amount. Tax year. Tax-time financial product use, including refund anticipation loans, refund anticipation checks, or no tax-time financial products. In tax year 2016, we estimated that about 18 percent of taxpayers used a tax-time financial product, plus or minus less than 1 percentage point. We also used IRS data from the Statistics of Income division for tax year 2016 to assess the geographical concentration of product use at the zip- code level. Zip code data from the IRS Statistics of Income division are based on population data that was filed and process by IRS in tax year 2016. Due to some data suppression from IRS for privacy purposes, zip codes with less than 100 tax returns are excluded from the data. As a result, in 2016 the total returns represented in the IRS zip code data are 145,302,140 and the number of tax returns with a tax-time financial product was 21,654,760, meaning about 15 percent of tax filing units in these data used a tax-time financial product. Regression analysis using FDIC data. Using FDIC data, we conducted a multivariate regression analysis to examine the relationship between each explanatory variable and tax-time financial product use. Specifically, we estimated multivariate logistic regression models. Regression models allow us to test significant relationships between economic and demographic variables and the likelihood of using tax-time financial products, while controlling for other factors. We used logistic regression models because our dependent variable is binary. The dependent variable represents whether a household used tax-time financial products. We collapsed “no” and “did not know/refused” into a single category for our regression analysis, so that the dependent variable is equal to 1 if the household used tax-time financial products and 0 otherwise. Logistic regressions allow the relationships between various characteristics and tax-time financial product usage to be described as odds ratios. Odds ratios that are statistically significant and greater than 1.00 indicate that households or heads of households with those characteristics are more likely to use tax-time financial products. Odds ratios that are less than 1.00 indicate that households or heads of households with those characteristics are less likely to use tax-time financial products. For categorical variables, this increase or decrease in the likelihood of product use is in comparison to an omitted category, or reference group. For example, the odds ratio for households headed by African Americans is statistically significant and 1.36. This implies that the odds of tax-time financial product use for households headed by African Americans are 1.36 times the odds of use for households headed by whites, holding other factors constant. Put another way, households headed by African Americans are about 36 percent more likely to use tax- time financial products than households headed by white individuals, if other conditions remain constant. This result and others are discussed further in the results section below. We also present 95 percent confidence intervals, which helps clarify the statistical significance of the odds ratios. Our baseline estimates were derived from logistic regressions that accounted for the survey features of the FDIC data. Our main regression results used data from the 2017 survey year. We also estimated logistic regressions using data from the 2015, 2013, and 2011 survey years, using the same variables when possible. Our baseline specification includes explanatory variables for race and ethnicity, education, age, household type, income, and homeownership. We used groups of indicator variables or categorical variables to control for all characteristics. In other specifications, we included controls for children, unbanked status, use of alternative financial services other than tax-time financial products, state indicators, and region indicators to check the robustness of our results. We also assessed the sensitivity of our analyses by restricting the analysis to households that only answered “yes” or “no” to tax-time financial product use. We excluded answers of “did not know/refused,” so that the dependent variable is equal to 1 if the household used tax-time financial products and 0 if the household did not use tax-time financial products. In a more limited analysis, we merged data from the 2017 FDIC data, which is the June 2017 supplement of the Current Population Survey, with the 2017 Annual Social and Economic Supplement, which is the March 2017 supplement of the Current Population Survey. We performed the additional analysis because the March 2017 supplement has data on tax-filing characteristics, including tax credits used by households. Given the structure of the Current Population Survey, some households were surveyed in both the March and June 2017 supplements, and those households comprise the sample used in this part of the analysis. We identified those represented in both supplements using household and person identifiers, as well as data on sex, race and ethnicity, and age. Using this merged sample, we estimated logistic regressions that both did and did not account for the survey features of the data. We included the same explanatory variables as our baseline estimates, along with indicators for use of the Earned Income Tax Credit, Additional Child Tax Credit, and Child Tax Credit. Analysis of IRS data. Using the 2 percent sample of IRS data, we estimated the percentages of tax filers with varying tax-filing characteristics by year and average refund amounts by year. All estimates are weighted at the tax filing unit level. Using the IRS’s zip code data from the Statistics of Income division for 2016, we calculated the number of total tax filing units and tax filing units who used tax-time financial product at the zip code level. Regression analysis using FDIC data. Our results have limitations and should be interpreted with caution. For example, our analysis identifies correlations between characteristics and tax-time financial product use and not causal relationships. Moreover, there may be variables that are correlated with tax-time financial product use that are not included in our models. For example, we are not able to account for community characteristics that may influence the decision to use the products due to data limitations. We used statistical tests for multicollinearity (high intercorrelations among two or more independent variables) and goodness of fit to check the validity of the model to the extent possible, given the use of complex survey data. Our analysis of the characteristics associated with the use of tax-time financial products uses a relatively small number of observations. For example, we observe 798 households that used these products in the 2017 survey year, representing about 2.4 percent of households (plus or minus 0.2 percentage points), and that is the benchmark utilization rate against which the results should be interpreted. Moreover, IRS data indicate that more than 20 million tax filers used tax-time financial products in 2016, representing about 20 percent of tax filers who filed their taxes electronically. These data sets use different units of analysis, and there can be multiple tax filers in one household, especially for those who use Earned Income Tax Credit. However, comparing the two suggests that the survey data may not include all users of tax-time financial products. Given the question used to measure the dependent variable, our analysis focuses on those who use tax-time financial products to get their tax refund more quickly. While a key reason people use tax-time financial products is to meet cash needs, there may be other reasons people use the products, including covering the cost of tax preparation. Our results may not generalize to other time periods. There have been a number of changes in the market for tax-time financial products in recent years. Our results may not generalize to all products currently available in the market. However, our results from 2017 are generally similar with the 2015, 2013, and 2011 survey years, despite a number of changes to the tax-time financial product market during these years. Our findings suggest that similar types of households have utilized tax-time financial products regardless of industry and market changes, particularly if households used paid preparers and tax-time financial products to expedite their tax refunds. Our analysis focuses on households that used tax-time financial products and accessed them through paid preparers. However, taxpayers also may have accessed specific types of tax-time financial products when they used online software to file their own taxes. For example, individuals who file their own taxes online may use the products to cover the cost of the software that helps them prepare their taxes. The characteristics of people who use products for these reasons may be different than what we found in our analysis. Analysis of IRS data. The IRS data are representative of tax returns filed electronically and not of tax returns filed by other means, including by paper. The results may not generalize to years for which we do not have data. The indicators in the data for specific types of tax-time financial products, including the indicators for refund anticipation loans and refund anticipation checks have some significant limitations. In tax years 2014– 2016, IRS only allowed tax-time financial products to be coded as refund anticipation loans or refund anticipation checks (that is, there was no code to indicate that two or more products were used together). However, there were some major changes in the industry during this period, particularly with regards to refund anticipation loans, that suggest that these indicators do not measure the same types of products over time. Given the limitations of the definitions of specific tax-time financial products, most of our analysis focuses on the universe of tax-time financial products in the IRS data and not on differences by specific types of products. Regression analysis using FDIC data. Our analysis suggests a number of economic and demographic characteristics are associated with tax- time financial product use, particularly when purchased through a tax preparer to expedite the tax refund, after controlling for other factors. In 2017, relatively lower-income households were more likely to use the products than higher-income households. Households headed by single women with families were more likely to use tax-time financial products than households headed by married couples. Furthermore, householders who owned their homes were less likely to use tax-time financial products. African American households were more likely to use the products compared to white households. Finally, relatively younger households were more likely to use the products than older ones. The results of the main specification of our logistic regression are presented in table 5. Our results for other specifications using 2017 data were generally similar. For example, adding an additional control for unbanked status did not substantively change the results. In alternative specifications that included an indicator for use of other alternative financial services, we found a significant and positive correlation between using tax-time financial products and other alternative financial services, including nonbank check cashing, nonbank money orders, payday loans, and pawn shops. Moreover, including state and region indicators did not substantively affect the results. Using the sample restricted to just “yes” and “no” responses also did not substantively change the results. Our results for other years were generally similar, with some exceptions. For example, in other survey years prior to 2017, we found that in addition to African American households, Native American households also were more likely to use tax-time financial products than white households. Moreover, education and children were significant correlates in prior survey years. Analysis of IRS data. We found that nearly 1 in 5 taxpayers who filed their taxes electronically used tax-time financial products each year from 2014 to 2016, while less than 3 percent of filers used free filing services available through IRS during the same period. We also found that in 2016, tax-time financial product use was associated with receiving tax refunds through direct deposit, which is a faster way to receive a tax refund than paper check. Users of tax-time financial products also were more likely to file as heads of household (tax filing status) than taxpayers who did not use tax-time financial products. Moreover, taxpayers who used the products received higher tax refunds on average than taxpayers who did not use the products, especially when they used paid tax preparers to file their taxes. Finally, analyzing the zip code of the filers, we found that use of tax-time financial product was concentrated in some areas of the South and the West. Our limited nongeneralizeable review of documents received from selected banks and tax preparers found disclosures generally followed Office of the Comptroller of the Currency (OCC) guidance or Internal Revenue Service (IRS) requirements for fees disclosure, respectively. However, we noted from our undercover visits of selected tax preparers that the extent and clarity of the disclosures offered to customers varied. Furthermore, in our review of selected tax preparers’ websites, we found that fees and information about products were not always clearly disclosed. All nine tax preparers we visited offered the option to pay for the tax preparation fees with the tax refund by using a refund transfer, but they did not always clearly communicate how these options work. For example, three preparers did not disclose the refund transfer fee, and in a few instances, the refund transfer was provided alongside a refund advance and we were not given the option to pay for the tax preparation fees out of pocket. In other cases, the refund transfer fee was disclosed, but the product was not always identified as optional (that is, not required for tax preparation). During six of our undercover visits, our investigators explicitly requested literature on product fees. However, the preparers either stated they did not have the literature available or only provided us with business cards and promotional material. The other three times we did not ask for, and were not offered literature on product fees, features, or terms. In two of our visits, the tax preparers offered our investigators a refund advance after we expressed an interest in getting the refund quickly. In another two visits, we were offered unsolicited refund advances. When offering the product, these four tax preparers bundled the refund advance with a refund transfer (an optional product). By adding a refund transfer, the tax preparer effectively added a fee-based product to the refund advance, a product that otherwise is free to the taxpayer. During one of the visits, we were offered a refund advance only after we specifically asked for it. We reviewed the websites of eight selected providers of tax preparation services. We found that while these providers generally disclosed product fees, these disclosures were not made in a consistent manner. For example, all eight of the websites we reviewed offered taxpayers the option to use the expected refund to pay for tax preparation fees. Most of the time, the fee associated with this option was not clearly disclosed on the website. Only two of the eight providers clearly disclosed this fee on the products page; the other six did not disclose the fee in a prominent way or at all. In addition, all five providers that offered refund advances fully disclosed fee information for this product. Three of the eight online tax preparation service providers had physical locations in addition to their online presence. Of these three, only one disclosed on its website the refund transfer fee for taxpayers who filed a return in-person at one of their offices. For the second preparer with a physical presence, the refund transfer fee quoted for the online service was significantly lower than the fee we were quoted for in-person services at an office. The third preparer with a physical and online presence did not disclose the refund transfer fee for either the in-person service or online filing. We received and reviewed seven disclosure documents originated by two national tax preparation companies both of which are electronic return originators (ERO) and 12 bank documents from five banks in the industry. We compared the disclosure documents against IRS requirements for disclosure of fees for tax products and we compared the bank documents to OCC guidance related to disclosure of product, disbursement, and additional fees. Both sets of documents in our nongeneralizeable review generally disclosed the product fees in accordance with IRS requirements or OCC guidance as appropriate. Bank forms, including disclosures, are presented to taxpayers once they have decided to apply for a tax product. This practice is consistent with OCC’s guidance, which states that the details of a product should be provided to consumers before they apply for it. However, our analysis found that almost all of these documents are presented to taxpayers after returns have been prepared and tax preparers have determined the taxpayers were qualified for a tax-time financial product. The timing of when a tax preparer make these disclosures would make it challenging for a taxpayer to compare product prices from different providers or make more informed purchasing decisions. Moreover, all the ERO documents we reviewed with information on refund advances disclosed that the taxpayer would be receiving a loan and not a refund. However, of the six ERO disclosure documents that disclosed fees, four disclosed additional fees that might be associated with tax refund products, such as disbursement fees. Of the 12 bank documents we reviewed, all disclosed that funds would be sent to the bank if taxpayers used a tax product. Almost all the documents disclosed the fees associated with the tax product and that the fees would be deducted from the refund. And four of five documents related to a loan product disclosed that the taxpayer would be receiving a loan and not a tax refund. The majority of the documents also disclosed that the taxpayer may receive the refund directly from the taxing authority without incurring additional costs and within the same time frame without using a tax product. All the tax preparer documents and the banks’ disclosure documents were brief and written in plain language. However, almost all the bank application documents were longer than four pages and included technical and industry language. Based on our document reviews of selected tax preparers and banks and as suggested by our undercover visits of nine selected tax preparers, the disclosure of fees for disbursing funds was inconsistent, particularly around prepaid cards. Prepaid cards are often used to disburse funds from a tax-time product. Based on our analysis of providers’ promotional content, in some cases a tax preparer will offer prepaid cards as the only disbursement option. The cards generally carry additional fees for long- term use (such as monthly, withdrawal, reload, and inactivity fees). Prepaid cards usually are reloadable and can be used to pay bills and make retail purchases. IRS does not have guidelines for disclosing fees for the long-term use of prepaid card. However, OCC requires that banks disclose if a tax product may be used on a long-term basis and disclose fees associated with extended use of the product. During our visits, seven of the nine tax preparers provided the option to have the tax refund deposited on a prepaid card. However, only two of the seven preparers noted any potential fee information associated with the short or long-term use of prepaid cards. These two preparers said that there was no additional charge to have the taxpayer’s refund deposited on a prepaid card, and the other five did not explain whether any fees would be charged for this transaction. Five of the seven preparers that offered a prepaid card explained that the card could be used for transactions other than receiving the tax refund. However, only two of the five disclosed any fee information associated with long-term use of the card. Another two of the five preparers referred our undercover agents to the issuer of the card for additional information. The remaining preparer did not disclose that additional fees would apply to long-term use of the card. Four of the eight tax preparation websites we reviewed disclosed partial information about fees related to the disbursement of funds to the taxpayer. Three of the eight websites only disclosed disbursement fee information related to use of prepaid cards. We found fee information in one of the eight websites only after doing a word search. Fees associated with the long-term use of prepaid card fees were not disclosed by three of the six preparers that offered this disbursement option. Two websites disclosed partial fee information and only one disclosed all the fees and terms associated with the long-term use of a prepaid card. Six of these websites advised the taxpayer to see the terms and conditions of the card, four included a link to the terms and conditions of the card, and two did not include a link. Bank documents generally disclosed the fees associated with different disbursement methods such as paper checks and prepaid cards; however, fees related to the long-term use of prepaid cards were not always disclosed. Almost half of the documents we reviewed that include the use of a prepaid card did not acknowledge that fees were associated with the long-term use of prepaid cards, while others included only partial information or a general statement that “fees may apply.” In addition to the contact named above, Karen Tremba (Assistant Director), Nathan Gottfried (Analyst in Charge), Jessica Artis, Maurice Belding, Evelyn Calderón, Farrah Stone, Kathleen McQueeney, Marc Molino, Neil Pinney, Barbara Roesmann, Jessica Sandler, Erinn Sauer, Erin Saunders-Rath, Michael Walton, and Helina Wong made significant contributions to this report.", "summary": "American taxpayers spent at least half a billion dollars in 2017 on financial products—issued by banks, through paid tax return preparers—to help them file taxes and get advances or loans against tax refunds. GAO was asked to review tax-time financial products. Among other things, GAO (1) described market trends and examined IRS data, (2) described characteristics of product users and factors that influence product use, and (3) described product disclosure practices. GAO reviewed fee and product usage data; conducted a multivariate regression analysis to determine user characteristics; and analyzed disclosures of selected providers that are national chains and those of their bank partners. GAO conducted nongeneralizeable undercover visits of nine randomly selected tax preparers in the Washington, D.C. area to understand how they communicate fees and terms to taxpayers. Preparers were selected to ensure a mixture of regulatory jurisdictions, among other factors. GAO reviewed laws, regulations, and guidance on the products, and interviewed IRS and other government officials and a nongeneralizeable selection of product and service providers, tax preparation companies, consumer groups, and academics. Trends in the market for tax-time financial products since 2012 include the decline of refund anticipation loans (short-term loans subject to finance charges and fees), the rise in use of refund transfers (temporary bank accounts in which to receive funds), and the introduction of refund advances (loans with no fees or finance charges). More recent product developments include increased online access to products for self-filers, higher refund advance amounts, the introduction of new products, and for tax year 2019, the reintroduction of fee-based loans. However, GAO identified some limitations in Internal Revenue Service (IRS) data on product use, including over- or under-counting of certain types of products. IRS has not communicated these data issues to users and has not updated guidance to tax preparers on how to report new product use. As a result, data users (including federal agencies and policymakers) have inaccurate information to inform their findings and decision-making. Lower-income and some minority taxpayers were more likely to use tax-time financial products, according to GAO analysis of 2017 data from IRS, the Bureau of the Census, and the Federal Deposit Insurance Corporation. Specifically, taxpayers who made less than $40,000 were significantly more likely to use the products than those who made more. African-American households were 36 percent more likely to use the products than white households. Product users tend to have immediate cash needs, according to studies GAO reviewed. For these users, tax-time financial products generally provide easier access to cash and more cash at a lower cost than alternatives such as payday, pawnshop, or car title loans. GAO's undercover visits with nine tax preparers, its review of selected provider websites, and review of documents obtained from selected banks and tax preparers found disclosures generally followed requirements for disclosing fees. However, disclosure practices by some paid tax preparers may pose challenges for consumers. For example: Preparers in GAO's review generally indicated that they present taxpayers with almost all of the documents with fee information after their tax returns have been prepared and the preparers determined the taxpayers qualified for a tax-time financial product. The timing of these disclosures would pose a challenge for taxpayers looking to compare prices for different providers. During six of nine undercover visits, GAO investigators explicitly requested literature on product fees but were not provided such information. Refund transfer fee information on websites GAO reviewed sometimes was presented only after the tax preparation process started, was in small print, or could be found only after navigating several pages. As a result, taxpayers may face challenges comparing prices. GAO is making two recommendations to IRS to make the collection of product use data more accurate and make data limitations known to users of the data. IRS concurred with both recommendations.", "document_type": "gao"}
{"report": "According to FCC, caller ID services became commonplace due to technology developed in the 1980s, and caller ID information transmitted with the call could generally be trusted by the call recipient. However, FCC found as voice service providers migrated to Internet Protocol (IP) networks, these technologies lessened the overall accuracy and reliability of the information presented to the call recipient. Caller ID allows the recipient of an incoming call to determine the telephone number of the caller and, in some cases, the name. This information helps the recipient make informed decisions about which calls to accept or ignore. While the number and name displayed on the caller ID may be associated with the caller, a caller can also deliberately falsify or “spoof” the information transmitted to the caller ID display to disguise the source of the call. Under the current telephone system, this information, true or false, is conveyed to the call recipient unless the caller requests that such information not be conveyed. Caller ID spoofing is widespread. Many instances of spoofing are legal. For example, spoofing is legally used by professionals such as doctors who want to use their cell phones to return calls to patients but choose to transmit their office number instead. Spoofing also often accompanies robocalls—an automated telephone call which delivers a recorded message. Certain types of robocalls are illegal, such as robocalls for sales pitches unless companies have consumers’ express written permission to call. In addition, telemarketers may not call home or mobile numbers that consumers have registered in the National Do Not Call Registry, which was established through legislation and is maintained by FTC—and they must transmit their telephone number and, if possible, their name, to the call recipient’s caller ID. According to FCC, advancements in technology have made it inexpensive and easy to make robocalls. As telecommunications systems have transitioned from traditional wireline services, to IP networks, the cost of making phone calls has dramatically decreased. IP-based voice services use existing internet connections to send phone calls, which may be cheaper than long distance phone charges associated with traditional phone service. Autodialers can be programmed to dial a long list of phone numbers in order to deliver millions of calls in a short period of time. These dialing systems, coupled with IP-based voice services, such as Voice over Internet Protocol (VoIP), enable telemarketers and scammers to make high volumes of calls from anywhere in the world. IP-based voice services have also made it inexpensive and easy to spoof caller IDs. According to an industry stakeholder, historically, the router systems used to spoof calls were physical devices located on site, which could be prohibitively expensive. However, software that is available for free can now be downloaded to enable a computer to function as a router. According to stakeholders, telemarketers and scammers can, with minimal cost, configure a router to display either a single spoofed number or a constantly changing set of numbers, making it appear as though calls originated in the United States even if they did not. (See fig. 1.) FCC, FTC, and DOJ each enforce different rules or laws related to caller ID spoofing. FCC enforces rules prohibiting anyone from causing the transmission of misleading or inaccurate caller ID information with the intent to defraud, cause harm, or wrongfully obtain anything of value. FCC also enforces rules requiring telemarketers to transmit caller ID information. FTC protects consumers against unfair or deceptive business acts or practices. FTC, similar to FCC, enforces rules requiring telemarketers to transmit their telephone number, and when available, the name of the telemarketer to a consumer’s caller ID service. DOJ enforces federal fraud statutes under which fines or imprisonment can be imposed against anyone who uses interstate telecommunications as part of a fraud scheme. DOJ can also take civil enforcement actions on FTC’s behalf. FCC and FTC each manage consumer complaint databases where consumers can file complaints about unwanted calls, robocalls, and violations of the Do Not Call Registry. In addition to government efforts, the telecommunications industry, including voice service providers and third party companies, have taken steps to counteract illegal spoofing. For example, some of these companies have developed or deployed applications (i.e., software programs, often referred to as apps) to defend against robocalls and other unwanted calls. This includes call blocking devices for landline telephones and various mobile applications that can label and block robocalls and other unwanted calls based on call patterns, consumer complaints or other means. While some carriers provide these services free, others may charge a fee. In addition, some carriers also work with analytics providers to analyze traffic on their networks. Beginning in 2017, FCC authorized voice service providers to block certain categories of unwanted calls before they reach consumers’ phones. Recently, FCC clarified that service providers can also, as a default, block calls identified as likely unwanted based on the provider’s reasonable analysis of call data unless consumers opt out of this service. Scammers use caller ID spoofing to facilitate a variety of financial fraud and other schemes, often in combination with robocalling. Based on our analysis of FCC, FTC, and DOJ enforcement cases and alerts from federal and state government agencies, as well as interviews with stakeholders, we identified three types of caller ID spoofing schemes. To Obtain Money or Information: Scammers have used caller ID spoofing to trick consumers into providing their financial or personal information or sending money such as via a debit or gift card. These scammers may spoof a name and phone number that looks familiar and trustworthy, such as that of a government agency, a company you do business with, or local number. Scams include telling call recipients they may be arrested or they owe money. For example, spoofed robocalls have been used as part of a wide-reaching scam in which callers spoofed IRS phone numbers and impersonated IRS staff to trick people into sending the scammers money for supposed unpaid taxes. IRS reported that from October 2013 through March 2019, the agency was contacted more than 2.4 million times by taxpayers who reported such calls, and more than 15,453 taxpayers reported losing about $75.1 million. (See fig. 2.) To Generate Telemarketing Leads: Unscrupulous telemarketers have used spoofing as part of an attempt to sell goods or services. In this scheme, consumers may receive a pre-recorded robocall with a sales pitch and be instructed to “press 1” to indicate interest, at which point the call recipient is transferred to a live operator. In one such scheme, more than 96-million spoofed robocalls were made over a 3- month period. These calls included pre-recorded messages falsely claiming to be from Hilton and other well-known travel companies; once consumers were transferred, live operators attempted to sell vacations not affiliated with the brands presented during the prerecorded message. To Harass: People have used spoofing to harass others. In some of these cases, people have used spoofing to cause another person’s caller ID to display a familiar or trusted phone number. In one case, an individual apparently placed 31 spoofed calls as part of a personal campaign to harass and stalk another person. These spoofed numbers appeared to be from the victim’s child’s school, among others. Spoofing is also one of several techniques used to place false calls to emergency response centers to elicit a police response to an address where no emergency exists. Callers have used spoofing to make it appear as if their call originated at or near the reported address. This practice, known as swatting, has resulted in death. For example, in one swatting case, a man was shot and killed by police who believed he was holding others at the address hostage. FCC and FTC consumer complaint data both show dramatic increases in recent years in the number of unwanted call complaints that specifically mention the term spoofing. According to our analysis of FCC and FTC complaint data, from 2015 through 2018, complaints to FCC that specifically referred to spoofing more than doubled and those received by FTC increased by more than four times. (See fig. 3). Several industry stakeholders we spoke with noted a growing trend in one particular type of spoofing, neighbor spoofing. Neighbor spoofing occurs when the caller ID is manipulated to display a phone number matching the area code and prefix (the first six digits) of the consumer’s phone number. Consumers may be more inclined to answer these calls because they appear to be local—perhaps from someone they know. Among FCC’s complaints that included both the caller’s and the call recipient’s phone numbers, the percentage that were indicative of 6-digit neighbor spoofing increased from 10 percent in 2015 to 15 percent in 2018; for similar FTC complaints, the percentage increased from 2 percent in 2015 to 16 percent in 2018; and a call blocking provider told us that its percentage of neighbor-spoofed robocalls increased from 2 percent in January 2016 to 23 percent in December 2018. One analytics provider told us there has been a shift recently from spoofing the first six digits to spoofing the first four and five, which the provider believed to be a reflection of scammers adjusting their methods as more people become aware of the original six-digit form of neighbor spoofing. From 2015 to 2018, FCC and FTC data show substantial increases in complaints indicative of four and five digit neighbor spoofing, with FCC complaints nearly doubling and FTC complaints increasing more than 10 times during this time period. FCC and prior GAO work have described several limitations with using complaint data as a means of measuring the extent of unwanted calls. For example, complaints might increase following consumer outreach regarding how to file a complaint or after news media coverage of a particular scam. In addition, not all consumers who experience problems file complaints, and not all complaints are necessarily legitimate or categorized appropriately. Further, a consumer could submit a complaint more than once, or to more than one agency, potentially resulting in duplicate submissions. Finally, while some consumers may use the term “spoof” when describing the complaint, others may not, either because they do not know they have been spoofed or are not familiar with the term. According to our analysis of FCC data, in 2018, 66 percent of all complaints that were indicative of neighbor spoofing did not include the term “spoof” in the complaint description. Nonetheless, FCC, FTC, and DOJ officials told us they use this complaint data to identify specific trends in types of scams that may help the agencies’ enforcement and public education efforts, which we discuss later in this report. Although we could not find industry data that estimated the total number of spoofed calls, available industry data suggest that the volume of unwanted calls and robocalls (of which illegally spoofed calls are a subset) has increased over the past several years. Using call patterns on their own networks or other means, voice service providers, call blocking applications and analytics providers track data on unwanted calls and robocalls. According to one company, these companies may have limited ability to detect or isolate spoofed calls, in part, because scammers may frequently change the numbers they use. In addition, stakeholders told us, because each of these companies analyzes their specific user base and may use different methods to identify and label robocalls and other unwanted calls, the number of unwanted calls each company estimated may be substantially different. For example, while one analytics company estimated 26.3 billion robocalls nationwide in 2018, another company estimated the number at nearly 48 billion. Similarly, one company estimated a 46 percent increase in robocalls from 2017 to 2018, while another estimated a 57 percent increase for the same time period. Despite these differences, all analytics and call blocking companies we interviewed reported that their estimates of the number of unwanted calls and robocalls have increased in recent years. Because there is no comprehensive data source on unwanted calls, robocalls, or spoofed calls, it is not possible to reliably estimate national trends. FCC has taken steps to seek input from industry and other stakeholders on how to better measure the extent of the unwanted call and spoofing problem. In a November 2017 Further Notice of Proposed Rulemaking, FCC sought comment on, among other things, what information should be collected to evaluate the effectiveness of efforts to combat these calls and whether FCC should adopt a reporting obligation for providers. FCC received numerous comments from voice service providers, their associations, and other stakeholders in response to this notice. One commenter expressed concern that a reporting obligation would be burdensome to providers or of little benefit to FCC, and other commenters stated the agency should instead continue to monitor trends in consumer complaints. More recently, in a June 2019 Declaratory Ruling, FCC adopted a recommendation from 2017 to prepare two reports—one in 2020 and a second in 2021—to measure the effectiveness of efforts to combat illegal robocalls. The ruling explicitly delegates authority to FCC staff to collect any and all relevant information and data from voice service providers necessary to complete these reports and states that the report should include authoritative data about the number of illegal robocalls. FCC, FTC, and DOJ officials all said that their agencies must prioritize which illegal spoofing activity to investigate and take enforcement action against because they do not have sufficient resources to pursue all such activities. FCC and FTC officials stated that while they review complaint data and other information, it would not be practical to open investigations related to every complaint. According to officials at all three agencies, given their limited resources, the agencies prioritize investigations based on the level of harm being perpetrated and the likelihood of being able to effectively bring an enforcement case. Such prioritization is consistent with standards for internal control in the federal government. Those standards call for agencies to estimate the significance of risks to achieving agency objectives—in this case objectives related to protecting the public from harm—and to use those estimates as a basis for responding to the risks. More specifically: In a 2015 letter to several members of Congress, the Chairman of the FCC stated that the agency is more likely to pursue enforcement action when a problem appears to be pervasive, represents a trend, involves an agency priority, affects many consumers, reflects particularly egregious abuse, or presents a security or safety concern. Focusing specifically on investigations and enforcement action related to caller ID spoofing, FCC officials told us that the agency’s three highest priorities are events that (1) threaten public safety; (2) involve very large numbers of spoofed calls; or (3) involve malicious scams or threats. FTC’s strategic plan for fiscal years 2018 to 2022 calls for the agency to target its enforcement efforts on those areas that cause the greatest amount of consumer harm. In line with this objective, FTC officials told us that the agency decides which consumer complaints to investigate based on the level of harm being perpetrated, as well as the likelihood of being able to effectively bring an enforcement case. DOJ’s Justice Manual states that serious violations of federal law must be prosecuted. DOJ officials told us that for fraud schemes that employ caller ID spoofing, the agency is more likely to charge a violation of one of the fraud statutes, such as mail fraud, wire fraud, computer fraud, or conspiracy, as well as the money laundering and identity theft statutes. Specifically with regard to wire and mail fraud cases, the Justice Manual states that serious consideration should be given to the prosecution of any scheme which in its nature is directed to defrauding a class of persons or the general public with a substantial pattern of conduct. FCC and FTC officials stated that there are significant challenges related to investigating spoofing cases that can affect which investigations they choose to pursue and limit the number of enforcement cases they are able to bring. For example, FTC officials stated that the use of VoIP technology enables fraudsters to easily change both their physical locations and the numbers they spoof, making it harder for FTC and other law enforcement agencies to track them down. An industry stakeholder said that the use of VoIP technology makes it difficult to determine even whether the call originated domestically or from overseas. Moreover, FCC officials stated that when spoofed calls originate wholly from a foreign jurisdiction, a lack of foreign cooperation can make it exceptionally difficult to follow a trail back to either the service provider that originated the call or the person or company making the calls. The officials explained that foreign cooperation may be lacking when the calls come from countries with which the United States does not have strong diplomatic relationships. The officials stated that because of this challenge, they are less likely to bring an enforcement case when calls originate wholly from a foreign jurisdiction, due to the low likelihood of successfully resolving such cases and the heightened use of limited staff resources required by such cases. Regardless of these challenges, FCC and FTC officials stated that their agencies have taken steps to improve their ability to investigate cases based overseas. For example, both agencies cited their outreach to the Indian government and the U.S.-India Business Council as well as their participation in the Unsolicited Communications Enforcement Network, a global network of law enforcement authorities and regulatory agencies that works to combat unsolicited communications. FCC, FTC, and DOJ officials identified 62 enforcement cases that they said involved spoofing or blocking of caller ID information, though DOJ officials stated that their list of enforcement cases was not comprehensive because DOJ’s enforcement database does not include an indicator for whether spoofing was employed as part of a fraud scheme. (For a description of these 62 cases, see app. II.) As noted below, these 62 cases are not representative of all of the cases the agencies have brought related to illegal robocalling. FCC officials provided us information on six cases—each of which the officials said involved spoofing or a caller’s blocking of their caller ID information—that the agency brought from April 2011 to September 2018. For example, one case involved a company that used spoofed robocalls to target elderly and low-income individuals to generate sales of health insurance coverage. The company’s high numbers of robocalls also disrupted an emergency medical paging service. FCC issued fines in five of these cases, and one pending case includes a proposed fine. FCC officials told us that since January 2004, the agency has initiated approximately 20 additional enforcement cases and has issued approximately 1,000 warnings, all for robocalling or Do-Not-Call violations under the Telephone Consumer Protection Act of 1991. FTC officials provided us information on 31 cases—each of which the officials said involved spoofing—that FTC brought—or that DOJ brought on FTC’s behalf—from April 2006 to June 2019. Examples of cases include several involving numerous calls to numbers on the National Do Not Call Registry and an incident in which a company impersonated government officials and help centers to make a sales pitch with false and misleading claims about an English-language learning course to Spanish-speaking U.S. consumers. Monetary judgments were issued in all but one of these cases. FTC officials told us that as of November 2019 the agency had brought 147 enforcement cases against Do Not Call and robocall violators. FTC officials also stated that FTC obtains injunctive relief in their Do Not Call, robocall, and spoofing cases, including court orders prohibiting the defendants from engaging in similar conduct, and in some cases, banning defendants from any telemarketing activity. Further, they stated the injunctive relief also includes reporting and compliance requirements to help FTC monitor defendants. FTC officials told us that the agency has obtained injunctive relief in all of its completed spoofing cases and that these injunctions provide strong deterrence and help stop illegal spoofing. DOJ officials provided us information on 25 cases—each of which the officials said involved spoofing—that the agency brought from May 2010 to August 2018. Several of these cases involved companies or individuals that used spoofing as part of a scheme to swindle money from people. For example, in one case, defendants used spoofing as part of a scheme to defraud and extort money from victims who were falsely told they had failed to accept and pay for products they had never ordered. Twenty cases had judgments that included prison time; 18 cases had monetary judgements. FCC and FTC have collected far less than has been assessed in fines or monetary judgements, but officials at both agencies stated that the amounts they have collected still serve both punitive and deterrent purposes. Specifically, FCC officials stated that thus far, FCC has collected $25,970 of the approximately $205 million in fines it assessed. This mostly represents full payment of a $25,000 fine FCC issued in January 2017, but FCC has yet to collect any portion of the more recent fines it has issued: a fine of $120 million it issued in May 2018 and a fine of approximately $82 million it issued in September 2018. FCC has referred both of these cases to DOJ for collection action. FCC officials noted that these large fines may not represent the amount that the defendants are able to pay, and that even payment of a fairly small fraction of a large fine could be enough to put a scammer out of business and serve as a substantial deterrent. FTC officials said that FTC has obtained a total of about $363 million in monetary judgments in its 31 spoofing cases. The officials said that many of these judgements were partially suspended based on the defendants’ ability to pay determined by a defendant’s net worth and assets. Further, the officials said if the defendant misrepresents his or her financial position, the entire judgment can become due under a clause that is part of the judgement. The officials said that as of August 14, 2019, FTC had collected about $31 million in its spoofing cases, and that this amount represents all or substantially all of the unsuspended judgments in those cases. Officials with DOJ’s Consumer Protection Branch said that the branch views monetary judgments as one piece of the deterrence equation for caller-ID-spoofing offenses. The officials stated that the low amounts collected suggest that other preventative measures, such as injunctive relief and imprisonment, must be employed to deter continued unlawful activity. FCC and FTC both favor some changes to law to enhance the effectiveness of their enforcement efforts. Specifically: In May 2019, FTC officials testified that the agency’s enforcement efforts are hindered by a statutory provision that prohibits the agency from taking action against telecommunications carriers, to the extent they are engaged in common carriage activities. FTC further testified that it would like this provision removed so that the agency could take enforcement action against carriers engaged in illegal telemarketing activities. In 2018, an FCC official publicly stated that a longer statute of limitations for enforcement of the Telephone Consumer Protection Act of 1991 would improve the agency’s enforcement efforts against knowing and willful violators of the act. Currently, that act has a 1-year statute of limitations, while the Truth in Caller ID Act of 2009 has a 2-year statute of limitations. FCC officials told us that harmonizing the two acts’ statutes of limitations to 2 years would help FCC’s enforcement efforts since spoofing often occurs with robocalling and the agency often uses the two statutes in tandem. A February 2019 FCC staff report on robocalls notes that FCC’s enforcement efforts can be hindered by the requirement that in many instances FCC must warn a party of apparent robocalling violations and can only proceed with a monetary penalty if the party subsequently commits the same type of violation, a requirement in the Communications Act that applies to the Telephone Consumer Protection Act of 1991. According to the report, this requirement enables a warned offender to incorporate under a new name to evade further detection and begin illegal activity anew. In contrast, the report notes, the Truth in Caller ID Act of 2009 allows FCC to directly issue a proposed monetary penalty without first issuing a warning. Similar to the statutes of limitations just discussed, FCC officials told us that since spoofing often occurs with robocalling and the agency often uses the two statutes in tandem, their enforcement efforts would benefit from the elimination of this statutory requirement. In 2019, bills were introduced in Congress that, if passed, would implement the changes in law that FCC and FTC have recommended and could potentially help address other challenges faced by FCC and FTC. For example, in July 2019, a bill was introduced in the Senate that would remove the provision prohibiting FTC from taking action against common carriers. Also in 2019, two different bills were introduced, one in the House and one in the Senate, that would, among other things, address issues with harmonization of the FCC statute of limitations and eliminate the FCC pre-penalty warning requirement with respect to illegal robocalling. In addition, one of these bills, the Telephone Robocall Abuse Criminal Enforcement and Deterrence Act (TRACED Act), would require DOJ, in consultation with FCC, to assemble an interagency working group to study and report to Congress on how to enhance enforcement against robocalls by examining issues like the types of laws, policies, or constraints that could be inhibiting enforcement of the Truth in Caller ID Act of 2009. The interagency working group would also be tasked with identifying existing and potential international policies and programs that could encourage and improve coordination between countries. We have reported in past work that collaborative mechanisms such as interagency working groups can help the federal government achieve many of the meaningful results it seeks to achieve, and that such mechanisms all benefit from certain key features, which raise issues to consider when implementing these mechanisms. As of November 2019, no federal legislation had been enacted on these issues. We found that FCC’s and FTC’s efforts to collaborate on spoofing investigations and enforcement actions align with seven key practices we have previously identified to enhance and sustain interagency collaboration. FCC and FTC officials explained that their close collaboration helps ensure that they share relevant information and avoid duplicating efforts. In addition, we found that DOJ’s collaboration with FCC and FTC aligns with five of the seven key practices. Although we did not find evidence that DOJ had taken steps in line with the other two key practices, officials at all three agencies stated that DOJ’s collaborative efforts were appropriate given its broader jurisdiction and wider focus. More specifically, we found that all three agencies have incorporated five key practices. Our prior work has found that one way agencies can incorporate three of these practices — (1) defining and articulating a common outcome, (2) establishing mutually reinforcing or joint strategies, and (3) agreeing on roles and responsibilities—is through a memorandum of understanding. In 2003, FCC and FTC agreed to a memorandum of understanding that calls for the agencies to cooperate and coordinate to implement consistent, comprehensive, efficient, and non-redundant enforcement of federal telemarketing statutes and rules. The memorandum also calls for the agencies to meet quarterly to discuss matters of mutual interest, share consumer complaints, and engage in joint enforcement actions when necessary. Consistent with the memorandum, FTC officials told us that FTC and FCC hold quarterly meetings to discuss how they are targeting robocalls and spoofing investigations and enforcement cases to avoid duplication. FTC and FCC officials stated that in addition, their collaboration with DOJ is enhanced through the participation of all three agencies in a monthly conference call hosted by the National Association of Attorneys General to coordinate efforts to combat illegal robocalls across the government. Although DOJ officials told us that DOJ does not have a memorandum of understanding with FCC or FTC regarding spoofing or robocall-related enforcement, officials we interviewed at all three agencies identified collaborative efforts that DOJ engages in that are consistent with the three key practices cited above. FCC and DOJ officials stated they are developing procedures to share information on a particular enforcement case, and that these procedures could be used on other cases as needed in the future. In addition, officials from all three agencies stated that DOJ’s participation in the monthly conference calls and additional informal outreach as needed was sufficient to ensure effective collaboration. With regard to the fourth and fifth key practices (4) identifying and addressing needs by leveraging resources, and (5) establishing compatible policies, procedures, and other means to operate across agency boundaries, FCC and FTC officials described regularly sharing information from their complaint databases, which is in line with these practices. FTC officials stated they regularly review FCC’s complaint information to help their enforcement efforts. Moreover, FTC has established policies and procedures whereby DOJ and FCC and other law enforcement entities have access to FTC’s complaint database, and FCC and DOJ officials stated that they frequently analyze FTC’s complaint database to inform their investigative decisions. Furthermore, DOJ officials stated that DOJ recently contributed funds to FTC to enhance capabilities to analyze the database. FCC and FTC have also leveraged resources by co-hosting a public event in 2018 on reducing robocalls and spoofing that included discussions of recent policy changes and enforcement actions to stop illegal robocalls. We found that FCC and FTC follow two additional key practices for collaborating on spoofing-related investigations and enforcement actions that DOJ does not: (1) developing mechanisms to monitor, evaluate, and report the results of collaborative efforts, and (2) reinforcing agency accountability for collaborative efforts through agency plans and reports. For example, FCC and FTC collaborated on a robocall report published by FCC in 2019 that discussed both agencies’ enforcement actions related to robocalls and spoofing, and each discussed their collaborative efforts related to robocalls in key agency documents related to accountability and performance. DOJ officials stated that they would be unlikely to have such materials specifically related to spoofing given the agency’s focus on fraud itself rather than spoofing or robocalling, which it views as a means to fraud. DOJ officials stated that DOJ’s general commitment to interagency collaboration is emphasized in its fiscal year 2020 budget submission to Congress and many press releases related to its enforcement cases. We reviewed DOJ’s budget submission and several DOJ press releases and found that they mention collaboration between DOJ and other agencies. FCC and FTC use a number of methods to educate consumers on ways to protect themselves against spoofed and other unwanted calls. According to FCC documentation, the agency has made combatting illegal robocalls and caller ID spoofing its top consumer protection priority and uses consumer education as a means to address this priority. Similarly, according to FTC’s chairman, consumer education is a critical element of FTC’s efforts to fulfill its consumer protection mission. The methods that FCC and FTC use—both independently and collaboratively—to educate consumers on ways to combat caller ID spoofing and unwanted calls include the following. Posting online consumer alerts, videos, blog posts, and other informative materials: Both FTC and FCC post information and warnings about caller ID spoofing scams on their websites. FTC, for example, developed Pass It On, a print- and web-based campaign to educate seniors about various types of scams that target seniors, including spoofing. FCC launched an animated video initiative on how to avoid spoofing scams and also posted a consumer alert about neighbor spoofing scams. The alert explains that scammers use such spoofing to increase the likelihood that consumers pick up the phone and provides tips such as to not answer calls from unknown numbers and to not provide any personal information to such callers. Additionally, FCC and FTC post other information, including tip cards and graphics such as those illustrated in figure 4. Visiting vulnerable communities: FCC has conducted speaking tours, such as tours through rural Appalachia and the Pacific Northwest in 2018 to educate communities about spoofing, and to build partnerships to help improve the effectiveness of future outreach efforts. Similarly, FTC has hosted briefings in underserved communities with law enforcement, consumers, and community advocates to place more attention on consumer protection issues such as spoofing and other types of fraud. We found that FCC’s and FTC’s consumer education efforts related to spoofing and other unwanted calls aligned with nine key practices for consumer education that we identified in our prior work (see table 1). For example, FCC and FTC have developed consistent and clear consumer education messages related to spoofing and unwanted calls: specifically, consumers: should not answer unknown calls; should not push any numbers if directed to do so; and should hang up immediately once it is clear that the caller is unknown. In addition, FCC and FTC officials have worked with credible messengers to help disseminate consumer education messages, including to potentially vulnerable populations. For example, since 2017, FCC has worked with the National Asian American Coalition to train grassroots volunteers to engage local community members and distribute educational tip cards printed by FCC in languages such as Mandarin Chinese, Korean, Vietnamese, and Tagalog. In addition, FTC has collaborated with AARP to develop three videos for Asian American and Pacific Islander communities on robocall, IRS, and Medicare scams. In addition, we found that, similar to their enforcement efforts, FTC and FCC’s efforts to collaborate on public education in this area are consistent with the seven key collaboration practices we discussed earlier in this report. For example, FCC and FTC agreed to a second memorandum of understanding in 2015 that states that the agencies will collaborate with each other on consumer and industry outreach and education efforts, as appropriate. FCC and FTC also collaborate with other entities, including federal, local, and private entities, to educate consumers on ways to combat spoofing. For example, FCC officials told us that beginning in October 2018, they collaborated with Department of Veterans Affairs officials to send out three joint emails (from November 2018 through March 2019) to veterans and veterans’ organizations on ways to protect themselves against illegal robocalls, including spoofed calls. These officials also noted that each email reached approximately 5.5 million targeted recipients. According to officials with industry groups, voice service providers, and FCC, the voice service provider industry has taken key steps towards successfully putting in place a caller ID verification system throughout much of the IP-based U.S. telephone network by the end of 2019. As discussed previously, the system is commonly referred to as STIR/SHAKEN or SHAKEN/STIR. According to the Alliance for Telecommunications Industry Solutions (ATIS), which spearheaded this industry-led effort along with the Session Initiation Protocol (SIP) Forum, the system is intended to enable voice service providers to verify that a caller has a right to use the caller ID transmitted with the call. Under the system, the voice service provider that first initiates the call onto the network (originating service provider) generates a digital signature that attaches to the phone call indicating that the caller has this right. This occurs only when the originating provider knows this information and is considered the highest level of verification, referred to by the industry as “attestation.” The signature is transmitted along with the call as it is routed from one service provider to another. The terminating service provider, which passes the call onto the call recipient, can verify that the signature was not tampered with before sending the call to the call recipient (see fig. 5). According to an FCC Notice of Proposed Rulemaking, as of June 2019, several major providers had deployed or were in the process of deploying the system on their own networks, and a few had started exchanging signed calls with a second provider. In addition, ATIS has announced a number of key steps taken to fully implement the system’s framework. For example, in September 2018, ATIS launched the system’s governance authority, whose board consists of representatives from a variety of U.S. voice service providers and relevant industry associations, and which, according to ATIS, is overseeing the system to ensure that it is effective and secure. In August 2019, ATIS issued a press release stating that the governance authority board had determined the requirements service providers must meet in order to get certificates to digitally sign calls and had contracted a private firm tasked with ensuring that only authorized service providers get these certificates. According to an industry official who worked on this effort, once most U.S. carriers deploy the system and are sharing information across their networks, the technical experts who developed the standards will be able to see how it works and improve and enhance the system through additional technical developments. Because it is not always possible for the originating service provider to determine whether the caller has a right to use the phone number that will be displayed, in addition to the top level of verification, the system was designed with a middle level and a lowest level of verification. The originating service provider digitally signs the call with the middle level of attestation when the provider has an established relationship with the caller but does not know whether the caller has the right to use the phone number it will display. According to ATIS officials, the originating service provider may use this level of attestation, for example, when a call comes from a corporate call center, which displays all outbound calls as originating from a central number or set of numbers. The originating service provider signs the call with the lowest level of attestation when it is responsible for originating the call onto its network but it does not have a relationship with the caller (such as when the call comes in from another country). When using either the middle or lowest level of attestation, the provider cannot determine if the call is spoofed. However, according to ATIS officials, the information that provided the basis for the attestation level is still likely to be helpful. For example, this information may better position the terminating service provider or call blocking and analytics apps to determine, in combination with other data the terminating service provider or such apps may have analyzed, whether to block or warn the consumer about the call. According to officials from several carrier associations or voice service providers, the new system should substantially improve the industry’s ability to combat spoofing and block unwanted calls by providing carriers with immediate verification information. These stakeholders, as well as FCC officials, also stated that enabling voice service providers to instantly identify the provider that initiated the call onto the network—through the digital signature attached to the call— could help facilitate federal investigations by accomplishing in an instant what can now take significant time and effort as the call must be traced back from provider to provider. One stakeholder who played a key role in the development of the system stated that as some U.S. service providers deploy this system and more calls are able to be verified, it is likely to incentivize other U.S. providers to deploy verification systems so that their calls will not stand out as unverified. This stakeholder said that the hope is that other countries, including those with many legitimate call centers that send calls to the United States, such as India, will also implement verification systems that eventually can be integrated with the U.S. system. And as more calls are able to be verified, the stakeholder explained that the system will become more valuable and useful. An ATIS representative and other stakeholders identified other examples of ongoing technical challenges and open issues: Information provided to consumers: The industry has not reached agreement about what, if any, information should be presented to call recipients to inform them that the call has or has not been verified. Stakeholders we spoke with noted that it is important to educate consumers on the limitations of any such information. For example, although a call may be verified, the provider cannot guarantee that the caller is not trying to defraud the call recipient—just that the caller is not using a spoofed phone number to do so. Further, if a provider is unable to verify the caller ID information, it does not necessarily mean the call is fraudulent or the caller has malicious intent. For these reasons, several industry stakeholders we spoke with emphasized that the information provided by this system can be most useful when combined with other methods service providers use to analyze call traffic to identify unwanted or illegal calls. IP-only system: Several stakeholders also emphasized that the system only works for calls carried entirely over IP networks, not those using traditional wireline networks. One industry group representing smaller providers that may use traditional wireline networks expressed concerns that its members may need more time to deploy the caller ID verification system because of the resources needed to transition to an IP network. This issue was discussed by industry stakeholders at FCC’s July 2019 summit on the caller ID verification system. One industry stakeholder stated that when calls that begin on a traditional wireline network are uploaded to an IP network, the originating service provider on that IP network will sign the call with the lowest level of verification, and that that information, in combination with analytics, will help providers to know whether these calls can be trusted. Verification of certain calls: As of June 2019, ATIS and industry stakeholders were also working to determine how to ensure that calls from 911 operators or video relay service calls for deaf and hard of hearing users are not blocked if providers are unable to verify the caller is authorized to use the phone number. Since 2013, FCC has taken several steps to encourage the industry’s caller ID verification initiative. In doing so, FCC’s efforts have aligned with federal guidance for agency participation in private-sector standards activities to help address national priorities. That guidance states that federal engagement in standards activities should aim to produce timely, effective standards that address legitimate regulatory, procurement, and policy objectives. The guidance also states that the federal government should assume an active role where necessary to ensure a rapid, coherent response to national challenges. Key steps FCC took to initiate and accelerate industry efforts—in line with the OMB guidance to produce timely and effective standards— are summarized below. In March 2013, FCC’s Chief Technology Officer presented a vision of developing a caller ID verification system to combat spoofing at an Internet Engineering Task Force meeting, later referred to as a “call to action” by a technology stakeholder who played a key role in developing this system. In July 2016, FCC’s Chairman issued a call to action for providers to accelerate their efforts to develop this system. FCC also called for responses detailing provider efforts. In December 2017, FCC directed one of its advisory bodies to, among other things, define criteria for selecting the system’s governance authority and recommend milestones for system deployment. Consistent with the guidance that federal engagement should aim to produce timely, effective standards, FCC’s Chairman urged service providers and standards groups to accelerate the development and deployment of these technical standards. In November 2018, the FCC Chairman sent letters to 14 U.S. providers and publicly demanded that that they adopt the caller ID verification system by the end of 2019. While the demand did not legally require providers to deploy the system, the Chairman stated that if industry’s progress lagged in 2019, FCC would take action to ensure widespread deployment. This demand and warning represent preliminary steps consistent with the guidance’s call for the federal government to assume an active role where necessary to ensure a rapid, coherent response to national challenges. In June 2019, FCC issued a notice of proposed rulemaking that would require all providers to implement the technical system if major providers fail to do so by the end of 2019. The notice also requested comments on how FCC should determine whether it is necessary to mandate implementation of the technical system and how to evaluate whether major voice service providers have met the FCC’s end of 2019 deadline for implementation. According to FCC officials and consistent with the federal guidance, FCC has engaged with ATIS, providers, and relevant technical stakeholders throughout their caller ID verification efforts. For example, FCC officials attended key meetings, and an FCC official submitted technical suggestions on standards development related to the caller ID verification system. ATIS representatives told us that FCC’s engagement in these technical efforts was helpful, as FCC was able to ask questions and prompt those working on the standards to consider some of the broader issues that various stakeholders would be concerned about and needed to be addressed. Furthermore, FCC is considering how, if at all, its role should evolve in the future. Notably, FCC’s June 2019 notice also requested comments on what role FCC should have in the governance of the caller ID verification system, how to encourage carriers that maintain some portion of their network on legacy technology to implement elements of the system, and how FCC and industry can best leverage this system to combat illegal calls originating outside of the United States. FCC also directed staff to develop two reports over the next 2 years that, among other things, provide information on the state of deployment of this caller ID verification system. FCC officials stated that their efforts related to these issues encompass more than what is in the proposed regulations, as FCC will continue to monitor the work of the governance authority, the progress of service providers’ implementation of the system, and industry’s efforts to improve the effectiveness of the system and address remaining technical issues. Moreover, at FCC’s July 2019 summit on the caller ID verification system, FCC’s Chairman stated that FCC is prepared to issue rules in 2020 mandating that major providers implement the caller ID verification system if these major providers do not meet the 2019 deadline. We provided a draft of this report to FCC, FTC, and DOJ for review and comment. Each agency provided technical and editorial comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Chairman of the FCC, the Chairman of the FTC, the Attorney General, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff any have questions about this report, please contact me at (202) 512-2834 or VonahA@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Appendix II: Summary of Federal Agencies’ Enforcement Actions Involving Telephone Calls that Allegedly Used Spoofed Caller ID alleged conduct. We selectively mention spoofing in this column to provide additional context for some of these cases. Andrew Von Ah, (202) 512-2834 or VonahA@gao.gov. In addition to the individual named above, other key contributors to this report were Alwynne Wilbur, Assistant Director; David Goldstein, Analyst- in-Charge; Mark Canter; Joshua Cicala; Jennifer Clayborne; Kristen Farole; Jeffery Haywood; Gina Hoover; Delwen Jones; Jenna Lada; Hannah Laufe; Harold Podell; Cheryl Peterson; and Malika Rice.", "summary": "Unwanted phone calls, which may also involve spoofing, consistently rank among the top consumer complaints to FCC and FTC. In recent years, consumers have lost millions of dollars—and been deceived into providing financial or other sensitive information or purchasing falsely advertised products—due to schemes using these calls. FCC, FTC, and DOJ have efforts aimed at combatting the fraudulent use of caller ID spoofing. Recently enacted federal legislation included a statutory provision for GAO to review federal efforts to combat the fraudulent use of caller ID spoofing. This report examines (1) what is known about caller ID spoofing schemes, including any recent trends; (2) federal agency enforcement and consumer education efforts; and (3) the status of industry efforts to develop technologies to combat spoofing, and FCC's role in these efforts. To address these objectives, GAO reviewed consumer complaint data from FCC and FTC from 2015 through 2018; reviewed investigation and enforcement information from FCC, FTC, and DOJ; and interviewed agency officials and representatives from 23 nonfederal stakeholders, including industry associations, voice service providers, call blocking and analytics services, mobile phone manufacturers, consumer groups, and a standards body. GAO also reviewed relevant agency documentation and assessed agency efforts against key practices for consumer education and interagency collaboration identified in GAO reports. Transmitting fake caller ID information with a phone call, also referred to as “spoofing,” is in many cases illegal—and is used in schemes to obtain money and personal information or generate telemarketing leads. Complaints submitted to the Federal Communications Commission (FCC) and the Federal Trade Commission (FTC), both of which work to protect consumers from spoofing, suggest that spoofing is a growing issue. FCC, FTC, and the Department of Justice (DOJ) identified 62 enforcement cases they have brought since 2006 involving spoofing. Enforcement can be challenging, as it can be difficult to identify the source of spoofed calls, and scammers may be based overseas. Nevertheless, GAO found that the agencies prioritize their spoofing-related enforcement actions based in part on the level of harm perpetrated against the public and generally follow key practices identified by GAO for effective collaboration. Additionally, FCC and FTC have proposed changes to law to enhance the effectiveness of their enforcement efforts, such as a change that would allow FCC more time to bring certain enforcement actions. Furthermore, FCC's and FTC's consumer education efforts related to spoofing align with key practices for collaboration and consumer education. For example, FCC and FTC have developed consistent and clear messages related to spoofing. Several major telecommunications carriers are taking key steps to put an industry-developed technical system in place designed to reduce spoofing by December 2019, which FCC has encouraged in line with federal guidance. This system is intended to enable carriers to verify whether a caller has a right to use the caller ID being transmitted with the call. Carriers can use this information to better determine whether to block or warn consumers about the incoming call. Stakeholders cautioned that the system cannot determine whether a caller has fraudulent intentions but only whether the caller is using a spoofed number. FCC has followed relevant federal guidance in participating in the development of this system by, for example, encouraging industry to accelerate deployment of the system, monitoring industry's progress, and providing input into the process.", "document_type": "gao"}
{"report": "Medicare beneficiaries with behavioral health conditions have a diverse range of conditions, of different severity, requiring different types of care. Beneficiaries with mild behavioral health conditions—such as mild depression—may require less complex care than beneficiaries with serious behavioral health conditions—such as schizophrenia—or with multiple interacting behavioral or physical health conditions. Subpopulations of Medicare beneficiaries also may face different behavioral health challenges. For example, dual-eligible beneficiaries— individuals eligible for both Medicare and Medicaid—are three times more likely to have been diagnosed with a major psychiatric disorder than non- dual beneficiaries. Medicare covers services for the diagnosis and treatment of behavioral health conditions, which includes the inpatient care covered by Part A and the physician services and outpatient care covered by Part B. Key behavioral health services in Medicare Part B include visits with a physician or other covered provider, partial hospitalization program services, annual depression screening, alcohol misuse screening and counseling, psychotherapy, screening, brief intervention, and referral to treatment services, and behavioral health integration services. Dual-eligible beneficiaries may be able to access additional behavioral health services through Medicaid that are not available through Medicare. Medicare covers behavioral health services delivered by a range of providers, including psychiatrists and physicians, clinical psychologists, licensed clinical social workers (LCSW), nurse practitioners, physician assistants, and clinical nurse specialists. In order to bill for services provided to Medicare beneficiaries, providers must enroll with CMS. Providers who do not want to enroll in the Medicare program may “opt out” of Medicare. Behavioral health providers have among the highest opt-out rates, with over 7,000 psychiatrists, psychologists, and LCSWs opting out of Medicare, representing nearly one-third of all providers who opted out of Medicare in 2017. Beneficiaries may still see these providers but must enter into a private contract with them. Medicare will not pay for any services furnished by providers who have opted out, so in these cases, beneficiaries must pay the provider’s entire charge out of pocket. According to researchers, psychiatrists have low participation rates across all forms of insurance, including Medicare, which may be explained, in part, by the reimbursement rates for time intensive treatments, low supply and high demand for psychiatry services, and high administrative burdens for solo practitioners to participate in insurance programs. CMS is required by law to provide information annually to Medicare beneficiaries about their coverage, including benefits and limitations on payment. Various factors affect how beneficiaries receive and process information about behavioral health conditions and their coverage options for behavioral health services. According to HHS, low health literacy is a key barrier that impacts individuals’ ability to comprehend health-related information. Moreover, researchers have found that low health literacy is associated with poor physical and mental health. More specific challenges facing individuals with behavioral health conditions include the stigma surrounding behavioral health conditions that may discourage individuals from seeking help or treatment. According to advocates for Medicare beneficiaries and individuals with behavioral health conditions, some individuals may have caregivers or other support for finding information and engaging in decision-making about their behavioral health care. According to CMS, one-third (36 percent) of Medicare beneficiaries in 2019 were enrolled in MA plans, which CMS pays on a monthly capitated basis to deliver all covered services needed by an enrollee. MA plans contract with provider networks to deliver care to Medicare beneficiaries and must meet CMS’s network adequacy standards. MA plans may employ care management and utilization management strategies. Care management may include case managers or care coordinators who work with enrollees and providers to manage the care of complex or high-risk enrollees, including those with behavioral health conditions. According to the MA plan officials we interviewed, prior authorization—a utilization management strategy—may be employed for high-cost treatments. Officials from all five MA plans told us that they may have difficultly recruiting behavioral health providers to participate in their network. One study found access to psychiatrists to be more limited than any other physician specialty in MA plan networks, with 23 percent of psychiatrists in a county included in network on average, compared to 46 percent of physicians in a county across all physician specialties in 2015. Our analysis of Medicare claims data shows that in 2018 approximately 5 million beneficiaries used behavioral health services through Medicare Part B. This represented about 14 percent of the more than 36 million fee- for-service (traditional) Medicare beneficiaries, and CMS paid providers about $3.3 billion for approximately 39.3 million behavioral health services in 2018. (See fig. 1.) Our analysis of claims data also shows that among Medicare beneficiaries who used behavioral health services in 2018, utilization of the services varied significantly. (See fig. 2.) The average number of services used by Medicare beneficiaries who used behavioral health services in 2018 was eight, while the median was three. Nearly half of all such beneficiaries used between two and seven behavioral health services during the year. Nearly one-third (30 percent) of beneficiaries using behavioral health services used one behavioral health service during the year. The 11 percent of beneficiaries who were the highest behavioral health service users used 19 or more behavioral health services (the 90th percentile) during 2018, and accounted for about half of all Medicare expenditures on behavioral health services. Our analysis also found that the services beneficiaries received largely fell into two broad categories in 2018: general patient consultations (53 percent of services) and psychiatry services, including psychotherapy (43 percent of services). Other services, such as central nervous system assessments and drugs administered by providers, accounted for about 5 percent of services. Beneficiaries receiving behavioral health care were largely diagnosed with a condition in at least one of five diagnostic behavioral health conditions categories, each of which contain multiple specific diagnoses. In 2018, 96 percent of all behavioral health services were for a primary diagnosis within one of these five categories. For example, the mood disorder category, which includes diagnoses such as depression and bipolar disorder, accounted for 42 percent of services provided. (See fig. 3.) Medicare claims data for 2018 show that some Medicare beneficiaries used behavioral health services to obtain treatment for SUDs. Seven percent of the behavioral health services in 2018 were for SUDs. Moreover, Medicare beneficiaries with SUDs represented 11 percent of beneficiaries using behavioral health services. On average, Medicare beneficiaries with SUDs used five behavioral health services in 2018, which is less than the number of behavioral health services used on average by all beneficiaries with a behavioral health diagnosis. Overall, beneficiaries under age 65 and dual-eligible beneficiaries were disproportionately represented among users of behavioral health services compared to the Medicare population. (See fig. 4.) In 2018, while beneficiaries under age 65 constituted 16 percent of all Medicare beneficiaries, they represented 34 percent of the Medicare beneficiaries who used behavioral health services and accounted for 42 percent of all behavioral health services paid for by Medicare that year. Similarly, while dual-eligible beneficiaries, many of whom are under age 65, constituted 20 percent of all Medicare beneficiaries, they represented 39 percent of the Medicare beneficiaries who used behavioral health services in 2018 and accounted for 45 percent of all behavioral health services paid for by Medicare. Finally, women constituted about 55 percent of all Medicare beneficiaries in 2018 and represented 62 percent of the beneficiaries who used behavioral health services that year. Our analysis of Medicare Part B claims shows that in 2018 two-thirds of behavioral health services (67 percent) were delivered to Medicare beneficiaries by behavioral health specialists: psychiatrists, psychologists, and licensed clinical social workers (LCSW). (See fig. 5.) Psychiatrists provided the most behavioral health services (31 percent), followed by LCSWs (19 percent), and psychologists (17 percent). A range of other providers delivered the remaining one-third of behavioral health services, including advanced practice providers (16 percent), primary care physicians (11 percent), other physicians (5 percent), and other providers (1 percent). As figure 5 shows, beneficiaries who were relatively high users of behavioral health services received a greater share of services from behavioral health specialists compared to all Medicare beneficiaries who used behavioral health services. Approximately three-quarters of services (78 percent) provided to the highest service users (those in the 90th percentile with 19 or more services per year) were delivered by behavioral health specialists: psychiatrists (31 percent), LCSWs (25 percent), and psychologists (22 percent). However, this pattern did not hold for Medicare beneficiaries with SUDs. Our analysis showed that beneficiaries with SUDs received 20 percent of their behavioral health services from a behavioral health specialist, and the other 80 percent of services were delivered by providers who did not specialize in behavioral health. See appendix I for additional information on Medicare behavioral health utilization. According to CMS officials, the agency’s overall strategy for providing information to beneficiaries about coverage of behavioral health services involves a variety of communication and outreach approaches. For example, CMS disseminates information to beneficiaries through written and online publications, Medicare.gov, scripted answers to questions through 1-800-MEDICARE, and social media. CMS is required by law to provide information to beneficiaries about coverage under Medicare. CMS annually mails out the Medicare & You handbook to all beneficiaries, and according to CMS officials, it mailed the handbook to 42.6 million households in 2018. The information provided in the publication includes descriptions of benefits and services, a summary of cost sharing, and the types of providers Medicare covers. According to CMS officials, Medicare.gov also includes information about covered benefits and a provider directory, although some may not be accepting new Medicare patients. According to CMS officials, the most comprehensive source of information on coverage for behavioral health services is contained in the publication Medicare & Your Mental Health Benefits, which is also available at Medicare.gov. We obtained statistics from CMS officials on the frequency with which Medicare beneficiaries requested copies of Medicare & You or the agency’s other publications or accessed the agency’s web-based tools to obtain information on Medicare coverage, including coverage for behavioral health services. The most frequently accessed in 2018 were Medicare & You, Medicare.gov, and the “What’s Covered?” smartphone application. (See table 1.) These sources of information cover Medicare more broadly and provide information about Medicare benefits in general, rather than those dealing specifically with behavioral health. Like CMS, MA plans use different approaches to provide information to the beneficiaries enrolled in MA plans, including publications, phone calls, and websites. According to officials from the five MA plans in our review, MA plans use multiple modes of communication to meet the preferences of their enrolled populations. MA plans are required to provide information to each enrollee at the time of enrollment and annually thereafter; for example, MA plans must share information about providers reasonably available to enrollees. MA plans are also required to provide marketing materials to CMS for review to ensure the adequacy and accuracy of the information in the materials. Two of the MA plans in our review offer digital health tools to their enrollees. One plan offers a tool that allows enrollees to communicate with case managers, and another plan provides enrollees access to test results, the ability to refill prescriptions and schedule appointments, as well as resources for patient education. According to CMS officials, the agency also uses other strategies for providing information to beneficiaries about coverage of behavioral health benefits. CMS officials stated that it partners with stakeholders to assist beneficiaries and caregivers seeking help with behavioral health conditions. For example, CMS officials described webinars and workshops it conducts to educate partners and stakeholders who educate and counsel Medicare beneficiaries. According to agency officials, the webinars cover a range of topics related to Medicare benefits and coverage, including behavioral health. The officials also told us that CMS partners with state health insurance programs to provide information about Medicare, including information to help Medicare beneficiaries understand their coverage. CMS officials also stated that the agency conducts public awareness and outreach campaigns to provide information to beneficiaries. Medicare & You—the most widely disseminated source of information on Medicare benefits and coverage—does not provide explicit information about coverage of services for beneficiaries with SUDs, although HHS and CMS have identified addressing SUDs as a top priority. We reviewed the fall 2019 edition of the Medicare & You publication and found that, while it does provide information on Medicare coverage for behavioral health services, it does not contain an explicit description of the services that may be covered for treatment of SUDs. CMS officials noted that printing the almost 43 million hard copies of the fall 2019 edition of Medicare & You started in July 2019, several months before the rule implementing expanded OUD coverage under Medicare was finalized. In December 2019, CMS officials updated the 2020 edition of Medicare & You to include information on the expanded OUD treatment authorized by the SUPPORT Act, which were finalized in November 2019, and became effective in January 2020. According to CMS officials, as of December 2019, this updated version was available on Medicare.gov, and CMS officials told us it will be sent to all individuals who become eligible for Medicare throughout calendar year 2020. We reviewed the updated 2020 web version of Medicare & You and found that a reference to opioid treatment was included; however, explicit information about Medicare’s coverage for other SUDs was not added. Although information on Medicare’s coverage for treating OUD is important, OUD represents only a subset of the SUDs for which Medicare beneficiaries may need treatment. Further, several of the advocates we interviewed noted that Medicare beneficiaries would benefit from clearer and more specific information about SUD coverage. According to data from SAMSHA, about one in 10 SUD cases is related to an OUD, while the rest are related to non-opioid substances. We asked CMS officials why the additions to Medicare & You relate only to OUDs, and they explained that it is the only distinct Medicare benefit category for substance abuse treatment. Officials also stated that while there is no category in Medicare & You for other SUDs specifically, the publication does note some related benefits, such as for counseling and services for behavioral issues, alcohol misuse screening, and behavioral health integrative services. However, the alcohol misuse screening benefit is specifically for beneficiaries who do not meet the criteria for alcohol dependency and covers brief counseling in a primary care setting. The description of behavioral health does not specify that SUDs are a behavioral health condition. The absence of information on Medicare’s coverage for SUDs in Medicare & You is inconsistent with HHS and CMS strategic priorities related to treatment for SUDs. The Department of Health and Human Services’ Fiscal Year 2018-2022 Strategic Plan includes among its strategic objectives reducing the impact of SUDs through treatment. Additionally, CMS has made addressing SUDs a top priority and has a stated commitment to treat SUDs, including OUDs. Beneficiaries lacking information on coverage of SUDs may be less likely to seek treatment. HHS and CMS have made addressing SUDs a priority. However, in its most widely disseminated publication on Medicare coverage and benefits, CMS does not provide explicit information about the program’s coverage for SUD treatment services. As a result, beneficiaries with SUDs may not be aware of this coverage and may not seek needed treatment. We are making the following recommendation to CMS: The Administrator of CMS should ensure that the Medicare & You publication includes explicit information on the services covered by the Medicare program for beneficiaries with a SUD. (Recommendation 1) We provided a draft of this report to HHS for review. HHS concurred with our recommendation, and provided written comments that are reproduced in app. II, and technical comments, which we have incorporated as appropriate. In its written comments, HHS stated it would explore opportunities to modify the Medicare & You handbook to ensure beneficiaries with substance use disorders are aware of the services covered by Medicare. HHS also reiterated some of the situations under which substance use disorder may be covered under Medicare, as well as its communication strategies and tools to ensure that beneficiaries and providers are aware of all of the services available under Medicare. We are sending copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, the Administrator of CMS, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at CosgroveJ@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix III. To produce the tables below describing the utilization of behavioral health services by Medicare beneficiaries and the providers furnishing these services, we analyzed the 2018 Medicare Part B claims file, the most recent data available at the time of analysis. Our analysis only includes Medicare beneficiaries in fee-for-service Medicare because similar reliable information was not available for beneficiaries enrolled in Medicare Advantage. In addition to the contact named above, Lori Achman (Assistant Director), N. Rotimi Adebonojo (Analyst in Charge), Todd Anderson, Sauravi Chakrabarty, Kelly Krinn, Rich Lipinski, Drew Long, Diona Martyn, Vikki Porter, and Caitlin Scoville made key contributions to this report.", "summary": "Behavioral health disorders often go untreated, potentially leading to negative health consequences. Behavioral health disorders include substance use or mental health disorders. Medicare provides coverage for behavioral health services. The Substance Use-Disorder Prevention that Promotes Opioid Recovery and Treatment for Patients and Communities Act enacted in 2018 included a provision for GAO to examine Medicare behavioral health services and how beneficiaries are informed of coverage and treatment options. This report (1) describes the utilization of behavioral health services by Medicare beneficiaries and the types of providers furnishing these services, and (2) examines how CMS provides information to beneficiaries about their coverage for behavioral health services. To describe service utilization and provider types, GAO analyzed 2018 Medicare claims data, the most recent data available. To examine how CMS shares information with beneficiaries, GAO reviewed CMS requirements for providing coverage information to beneficiaries, reviewed CMS publications, and interviewed CMS officials. GAO's analysis of Medicare claims data shows that in 2018 almost 5 million beneficiaries used behavioral health services—services for mental and substance use disorders. This represented about 14 percent of the more than 36 million fee-for-service (traditional) Medicare beneficiaries and reflects about $3.3 billion in spending. Additionally, about 96 percent of all behavioral health services accessed by Medicare beneficiaries in 2018, the latest data available, were for a primary diagnosis in one of five behavioral health disorder categories. (See figure.) Mood disorders, such as depression and bipolar disorders, accounted for 42 percent of services. SUD services accounted for about 7 percent of all services accessed by beneficiaries. Further, two-thirds of behavioral health services were provided by psychiatrists, licensed clinical social workers, and psychologists in 2018. The Centers for Medicare & Medicaid Services (CMS), the Department of Health and Human Services' (HHS) agency that administers Medicare, uses various approaches to disseminate information to Medicare beneficiaries about coverage for behavioral health services. As part of these efforts, CMS mails out Medicare & You —the most widely disseminated source of information on Medicare benefits—to all Medicare beneficiaries every year. GAO reviewed the fall 2019 and January 2020 editions of Medicare & You. While the January 2020 edition describes a new coverage benefit for beneficiaries with opioid use disorders, neither edition includes an explicit and broader description of the covered services available for substance use disorders. Both HHS and CMS have stated that addressing substance use disorders is a top priority. Given that coverage for substance use disorders is not explicitly outlined in Medicare's most widely disseminated communication, Medicare beneficiaries may be unaware of this coverage and may not seek needed treatment as a result.", "document_type": "gao"}
{"report": "The DOD nuclear enterprise includes strategic and nonstrategic nuclear forces and the supporting infrastructure and personnel to build, maintain, and control these assets. The strategic nuclear forces include a triad of Air Force ICBMs; Air Force nuclear-capable bomber aircraft; and Navy submarine-launched ballistic missiles carried by SSBNs; as well as associated nuclear munitions; air refueling; and NC3 capabilities. NC3 capabilities are a key part of the defense nuclear enterprise, used to support planning, situation monitoring, and communication of force direction between the President and nuclear forces. Consistent with the New Strategic Arms Reduction Treaty (New START), the United States has limited the number of deployed delivery systems for each of the three legs of the strategic nuclear triad (see fig. 1). The 2018 Nuclear Posture Review states that the triad’s synergy and overlapping attributes help ensure the enduring survivability of deterrence capabilities against attack and the capacity to hold at risk a range of adversary targets throughout a crisis or conflict. In addition to the strategic nuclear triad, the defense nuclear enterprise includes nonstrategic nuclear forces: forward-deployed fighters—referred to as dual-capable fighter aircraft―that are able to deliver conventional or nuclear munitions; their associated nuclear weapons; and the supporting infrastructure and personnel to build, maintain, and control nuclear assets. NC3 capabilities are fielded through a large and complex system comprising numerous land-, air-, and space-based components used to ensure connectivity between the President and nuclear forces. Responsibilities for managing NC3 are distributed among many DOD components including military departments, combatant commands, defense agencies, the Joint Staff, and the Office of the Secretary of Defense. NC3 capabilities provide the President with the means to authorize the use of nuclear weapons in a crisis. NC3 capabilities support five important functions: Force management: assignment, training, deployment, maintenance, and logistics support of nuclear forces and weapons before, during, and after any crisis. Planning: development and modification of plans for the employment of nuclear weapons and other operations in support of nuclear employment. Situation monitoring: collection, maintenance, assessment, and dissemination of information on friendly forces, adversary forces and possible targets, emerging nuclear powers, and worldwide events of interest. Decision making: assessment, review, and consultation that occur when the employment or movement of nuclear weapons is considered. Force direction: implementation of decisions regarding the execution, termination, destruction, and disablement of nuclear weapons. The NDERG is the principle integrated civilian–military governance body for the DOD nuclear enterprise. It was established in 2014 by the Secretary of Defense to ensure the long-term health of the nuclear enterprise by addressing resourcing, personnel, organizational, and enterprise policy issues identified in the 2014 nuclear enterprise reviews. The NDERG also maintains senior-leader awareness of ongoing issues of importance in the nuclear enterprise, ensures effective sustainment of these critical nuclear capabilities, and provides a forum for strategic-level coordination and integration of issues arising from other oversight committees and councils related to the nuclear enterprise. The NDERG consists of a group of senior officials chaired by the Deputy Secretary of Defense with the Vice Chairman of the Joint Chiefs of Staff as vice chair. The NDERG is supported by a Nuclear Deterrent Working Group, which meets biweekly and reviews the status of the implementation of the recommendations of the nuclear enterprise reviews, and a Nuclear Deterrent Senior Oversight Group, which meets quarterly and reviews any recommendations that the Working Group believes are ready for the NDERG to close. The Nuclear Deterrent Senior Oversight Group―co- chaired by the Deputy Assistant Secretary of Defense for Nuclear Matters, the Deputy Assistant Secretary of Defense for Nuclear and Missile Defense Policy, the Joint Staff Deputy Director for Strategic Stability, and a senior-level representative of the Director of CAPE―also receives annual briefings on DOD components’ assessments of their progress, reviews organizational changes, and discusses other issues related to the management, operations, and health of the nuclear enterprise—including human resources and culture, operational availability, sustainment, and modernization and recapitalization issues not directly addressed in other forums. The Deputy Secretary of Defense updates the Secretary of Defense on the NDERG’s progress as requested. In November 2014, the Secretary of Defense directed DOD to address the recommendations from the 2014 nuclear enterprise reviews and directed CAPE to track and assess implementation efforts. The Joint Staff, the Navy, the Air Force, offices within the Office of the Secretary of Defense, and U.S. Strategic Command support CAPE’s efforts. CAPE compiled the recommendations from the 2014 nuclear enterprise reviews. In total, CAPE identified 175 distinct recommendations from the three documents associated with the reviews. CAPE then identified 247 sub- recommendations within those recommendations, which were directed to multiple military services or other DOD components. For example, if a recommendation was directed to both the Air Force and the Navy, then one sub-recommendation was made to the Air Force and one to the Navy. CAPE then worked with the military services to identify offices of primary responsibility for implementing actions to address the recommendations, any offices with coordinating responsibility, and any resources necessary to implement each recommendation. CAPE has developed a centralized tracking tool to collect information on progress in meeting milestones and metrics. As shown in figure 2, the tracking tool includes fields for the underlying problem statement, or root cause, and for the recommendation and time frames with milestones for implementing the recommendation. The tracking tool also includes performance measures (referred to as metrics in the tracking tool) to assess both the progress (through “process metrics”) and the effectiveness of the implementation actions (through “outcome metrics”). The outcome metrics aid DOD in determining whether implemented recommendations have addressed the underlying problem that was the impetus for the original recommendation. The tracking tool contains hundreds of unique milestones and metrics, and additional milestones and metrics may be added as they are identified. The Air Force and the Navy also have developed their own methods of tracking their service-specific recommendations. In December 2016, the Deputy Secretary of Defense issued a memorandum that directed the transition of the tracking and analysis responsibilities related to implementing the recommendations of the 2014 nuclear enterprise reviews from CAPE to the military departments and other DOD components. However, CAPE remains responsible for providing guidance to inform the analyses conducted by the military departments and other DOD components, overseeing these analyses, and assessing recommendations for closure. The aim of these changes was to enhance ownership and embed the principles of robust analysis, continuous monitoring, and responsibility throughout the department. In January 2018, in response to a GAO recommendation, CAPE issued additional guidance to aid the military departments and other DOD components in identifying, assessing, and documenting risks associated with the 2014 recommendations. The guidance instructs components to document key risks, defined by CAPE as a risk that requires mitigation by the leadership of the DOD components or a risk that cannot be mitigated within a component’s existing authorities and resources—for example, one that cannot be mitigated within the Air Force or Navy and must be raised to a higher authority. As we reported in November 2018, in response to the January 2018 guidance for tracking risks, the Air Force and the Navy included in the centralized tracking tool information on key risks for the recommendations they were responsible for or an indication of the absence of any key risk. The Council on Oversight of the National Leadership Command, Control, and Communications System (NLC3S Council) was established by statute and is responsible for oversight of the command, control, and communications system for the national leadership of the United States. Additionally, as recommended in the 2015 NC3 report, the NLC3S Council reviews the recommendations from the report and assesses them for closure. The NLC3S Council is supported by the National Leadership Command Capabilities Executive Management Board, which comprises a Senior Steering Group and four working groups—Stakeholders, Resources, Assessments, and Nuclear Command and Control Issues. The Executive Management Board ensures that the council is informed of and presents issues that require senior leadership–level decisions. In 2018, the Secretary of Defense approved the designation of the Commander of U.S. Strategic Command as the NC3 enterprise lead with increased responsibilities for operations, requirements, and systems engineering and integration. At that time, the Secretary of Defense also approved the designation of the Under Secretary of Defense for Acquisition and Sustainment as the NC3 enterprise capability portfolio manager with increased responsibilities for resources and acquisition. In November 2018, we recommended that DOD update applicable guidance (such as the NLC3S Council’s and Executive Management Board’s charters) and identify whether there is a need to request changes to statutory or presidential guidance in order to clarify changes to roles and responsibilities for oversight of NC3. According to DOD officials, DOD is in the process of implementing these recommendations, with the intent of having the Commander of U.S. Strategic Command and the Under Secretary of Defense for Acquisition and Sustainment provide leadership with respect to NC3 capabilities, while the Executive Management Board maintains its role for those systems that primarily relate to non-NC3 systems, with all three entities reporting on their respective issues to the NLC3S Council. The NLC3S Council is co-chaired by the Under Secretary of Defense for Acquisition and Sustainment and the Vice Chairman of the Joint Chiefs of Staff. Members of the council include the Under Secretary of Defense for Policy; the Under Secretary of Defense for Research and Engineering; the Under Secretary of Defense for Intelligence; the Commander, U.S. Strategic Command; the Commander, North American Aerospace Defense Command/U.S. Northern Command; the Director, National Security Agency; and the DOD CIO. The DOD CIO also serves as the Secretariat for the NLC3S Council and tracks the implementation of recommendations from the 2015 NC3 report, among other activities. Additional organizations may participate in the NLC3S Council’s meetings to provide subject-matter expertise. Regular participants in the NLC3S Council include the Office of the Under Secretary of Defense (Comptroller); senior leaders from the Army, the Navy, and the Air Force; the Director, Defense Information Systems Agency; the Director, White House Military Office; and Director, CAPE. The 2014 nuclear enterprise reviews included Operations and Maintenance as 1 of 11 categories. Recommendations within this category are primarily related to the sustainment and maintenance of nuclear weapon systems. The reviews identified several Operations and Maintenance core issues related to, among other things, maintenance infrastructure, lack of leadership visibility into sustainment issues, fragmented logistics support, and aging systems and support equipment leading to parts obsolescence issues. Of the 175 recommendations included in the 2014 nuclear enterprise reviews, 30 were categorized as Operations and Maintenance. Other categories in the 2014 reviews, such as Investment and Personnel, also include some recommendations that are related to sustainment and maintenance. DOD conducts sustainment and maintenance on nuclear enterprise weapon systems to ensure that these systems are available to support current military operations and maintain the capability to meet future requirements. Sustainment of weapon systems comprises logistics and personnel services required to maintain and prolong operations of the weapon system. DOD conducts maintenance at two levels: field level and depot level. Field-level maintenance is performed at the unit level on the unit’s own equipment, requires a relatively fewer number of skill sets, and occurs more frequently. Depot-level maintenance includes the overhaul, upgrade, or rebuilding of equipment, occurs less frequently, and requires a greater number of skill sets. Depot maintenance includes inspection, repair, overhaul, or the modification or rebuild of end items, assemblies, subassemblies, and parts that, among other things, require extensive industrial facilities, specialized tools and equipment, or uniquely experienced and trained personnel that are not available in other maintenance activities. A number of DOD organizations are involved in the sustainment and maintenance of nuclear weapon systems. Some key organizations include the following: Defense Logistics Agency. The Defense Logistics Agency manages approximately one-fifth of the value of DOD’s overall inventory and provides billions of dollars in consumable items on an annual basis for depot maintenance conducted at defense industrial sites—Army and Marine Corps depots, Navy Fleet Readiness Centers and Navy shipyards, and Air Force Air Logistics Complexes—where combat vehicles, planes, helicopters, and ships are repaired and overhauled. Air Force Materiel Command. Air Force Materiel Command conducts research, development, test, and evaluation, and provides acquisition management services and logistics support necessary to keep Air Force weapon systems ready for war. One of six centers within Air Force Materiel Command, the Air Force Nuclear Weapons Center is the nuclear-focused center synchronizing all aspects of nuclear materiel management on behalf of the Air Force Materiel Command commander. Naval Sea Systems Command and Naval Air Systems Command. Naval Sea Systems Command’s affiliated Program Executive Offices—including the Program Executive Office for submarines and the Program Executive Office for the Ohio-class SSBN and its replacement, the Columbia-class SSBN—are responsible for life-cycle management of their assigned programs. Similarly, Naval Air Systems Command provides full life-cycle support of naval aviation aircraft, weapons, and systems. DOD continues to make progress in implementing the recommendations from the 2014 nuclear enterprise reviews and the 2015 NC3 report, but the key tracking tools used to provide visibility on the status of the recommendations from these reviews do not provide current and complete information. For example, expected completion dates for key metrics and milestones—key methods of evaluating the department’s progress—are not up to date. Additionally, the NDERG is working to develop an additional approach for tracking long-term risks and opportunities to monitor the health of the defense nuclear enterprise. Current and complete information regarding the status and metrics for enduring recommendations from the 2014 and 2015 studies would help inform this effort. DOD continues to make progress in implementing the recommendations of the 2014 nuclear enterprise reviews. As of our last report, in November 2018, DOD had closed 151 sub-recommendations. Based on our review of CAPE’s centralized tracking tool, the NDERG has closed five additional sub-recommendations since then. As a result, as of August 2019, the NDERG has closed 156 of the 247 sub-recommendations (see fig. 3). DOD continues to make progress in implementing the recommendations of the 2015 NC3 report. Since we last reported, in November 2018, DOD has closed two additional recommendations. As of August 2019, the NLC3S Council has closed seven of the 13 recommendations from the NC3 report (see fig. 4). The DOD CIO has provided guidance to improve the tracking and evaluation of DOD’s progress in implementing the recommendations of the 2015 NC3 report, in response to our second October 2017 recommendation. The military services and other DOD components have not kept information on the implementation status of the 2014 nuclear enterprise reviews’ recommendations and 2015 NC3 report’s recommendations current and complete. As we have previously reported, CAPE developed a centralized tracking tool to aid in evaluating the actions that have been taken to implement the recommendations from the 2014 nuclear enterprise reviews and inform senior leaders across the defense nuclear enterprise. DOD CIO collects information on the status of the 2015 NC3 report’s recommendations in a layout similar to that used for the 2014 recommendations. The military departments and other DOD components are responsible for tracking and evaluating the implementation status of the 2014 nuclear enterprise reviews’ recommendations; CAPE provides guidance to aid in these efforts. CAPE’s 2016 guidance indicates that the military departments and DOD components should, as appropriate, use metrics and milestones to analyze progress. The guidance also states that existing data should be used, where possible, to minimize the workload of this effort. The centralized tracking tool developed by CAPE is the primary means by which progress is tracked. For each of the hundreds of metrics and milestones identified, the tracking tool includes expected completion dates and indicates which have been met and which are behind schedule. The tool identifies both process metrics, to aid in assessing the progress of implementation efforts, and outcome metrics, to aid in determining whether implemented recommendations have addressed the underlying problem that was the impetus for the original recommendation. However our review has found, for those metrics and milestones that are behind schedule, many of the completion dates have not been updated to reflect when they are expected to be completed, even if years have passed since the original completion date lapsed. According to officials from CAPE, the original dates were left in the tracking tool to maintain visibility on how far past their initial expected completion dates these metrics and milestones had gone without being resolved. We previously found that the Air Force and Navy used their own tracking tools in addition to DOD’s centralized tracking tool. According to Air Force officials, they still are using their internal tracking tool to help them note progress within the Air Force before providing inputs to DOD’s centralized tracking tool. However, according to Navy officials, they are no longer maintaining their internal tracking tool, because they determined that those efforts were unnecessary and redundant with providing inputs to the centralized tracking tool for the relatively few recommendations that the Navy still has open. CAPE’s 2016 guidance indicates that the goals of monitoring the implementation of the 2014 nuclear enterprise reviews’ recommendations are to track progress toward addressing systemic issues and to assess changes in the overall health of the enterprise. This information provides stakeholders within the defense nuclear enterprise with key means of monitoring progress and evaluating the outcomes of these efforts. DOD’s approach has been to measure the effectiveness of actions taken by gathering supporting data and measuring the effectiveness of each recommendation separately. However, DOD officials have noted that some enduring recommendations—including recommendations associated with changing a service’s culture or morale—will take time to evaluate. In some cases, data related to outcome metrics may not be available to evaluate the effectiveness of actions taken until years after a service has taken key actions to address the recommendation. According to DOD officials, this framework was established to avoid prematurely assuming that actions taken have successfully addressed underlying problems. The need for the military departments and other DOD components to keep information current, particularly estimated dates for the completion of activities, has been emphasized at meetings of the Nuclear Deterrent Working Group. Further, a July 2018 memorandum from the Deputy Secretary of Defense reiterated that the components of the nuclear enterprise, which includes the Air Force and the Navy, will continue to track progress in implementing the recommendations from the 2014 nuclear enterprise reviews through 2020. According to officials from the Office of the Deputy Assistant Secretary of Defense for Nuclear Matters and CAPE, the use of the centralized tracking tool is likely to extend beyond 2020, and the Nuclear Deterrent Working Group—which supports the NDERG and its Nuclear Deterrent Senior Oversight Group—is using information from the centralized tracking tool to support additional work. In the context of transitioning from the current centralized tracking tool— which tracks the recommendations of the 2014 nuclear enterprise reviews—to enduring metrics used to characterize the health of the nuclear enterprise, as discussed later in this report, the Deputy Assistant Secretary of Defense for Nuclear Matters stated that it was not a good use of limited personnel resources to request that all metrics and milestones be updated. This is because many of the 2014 recommendations were minor and quickly closed. He noted that improved information about critical recommendations transitioning to enduring recommendations would be of use. The approach that the DOD CIO has established to track the recommendations from the 2015 NC3 report largely mirrors the approach developed for tracking the 2014 nuclear enterprise reviews’ recommendations. However, DOD CIO officials have noted that the 2015 NC3 report recommendations are more narrowly scoped than some of the recommendations from the 2014 reviews and therefore their tracking is less extensive. DOD CIO has issued guidance that requests that DOD components provide quarterly updates on the progress of implementing the recommendations. It specifies that the components should provide current metrics used to track progress, as well as key milestones, at a minimum by quarter, for the next year. The guidance further states that, as appropriate, both process metrics—to measure whether actions taken address a recommendation—and outcome metrics—to measure end results of interest—should be used. However, metric and milestone information for many of the recommendations in the tracking tool is out of date or incomplete. In particular, many of the recommendations do not have outcome metrics identified. DOD CIO’s guidance does request quarterly updates from the components and provides some information on content for those updates, but it does not specify that the information should be kept current and complete in the tracking tool. Therefore, information like process and outcome metrics may not be complete and kept current beyond the next year. Standards for Internal Control in the Federal Government states that an organization’s management should use high-quality information, which is defined as information from relevant and reliable data that is appropriate, current, complete, accurate, accessible, and provided on a timely basis. CAPE’s guidance provides a framework for information that DOD components should consider as they evaluate and track progress made for the 2014 recommendations. The guidance notes that, although the intent of the recommendations is enduring and the systemic issues identified by the 2014 nuclear enterprise reviews should be addressed, the specific approaches to the recommendations can be revised to address the recommendations more effectively. Similarly, the DOD CIO’s guidance provides a framework for information that DOD components should consider as they evaluate and track progress made for the 2015 NC3 report recommendations. For tracking both the 2014 nuclear enterprise reviews’ and 2015 NC3 report’s recommendations, DOD’s approaches are limited by the quality and completeness of the data collected and tracked in the centralized tracking tools. Specifically, CAPE’s general guidance for tracking the 2014 nuclear enterprise reviews’ recommendations does not include a specific requirement to periodically update the information to keep it current. DOD CIO’s guidance for tracking the 2015 NC3 report recommendations does request quarterly updates but does not specifically require information included in the tracking tool be complete. Without current and complete information—including revised dates for when metrics and milestones will be complete—the tracking tools used to track the 2014 and 2015 recommendations do not provide a complete and accurate picture of when tasks are expected to be completed, whether progress is still being made to address the many issues the department has identified, whether any efforts have stalled, or any additional challenges. Additionally, without an accurate picture of the department’s progress in addressing these recommendations, the Nuclear Deterrent Working Group has less information to leverage to support additional work to track enduring issues on behalf of the NDERG. In addition to tracking the 2014 recommendations, the July 2018 memorandum from the Deputy Secretary of Defense stated that stakeholders will develop metrics to capture long-term risks and identify opportunities for regular reporting to the NDERG. The NDERG Charter, issued in early June 2019, provides further direction to the Nuclear Deterrent Senior Oversight Group and its Nuclear Deterrent Working Group, including that members should develop metrics, data, tools, and briefing materials to support the NDERG efforts to identify, track, and address issues, risks, and opportunities across the nuclear enterprise. The charter further directs the Nuclear Deterrent Senior Oversight Group and Nuclear Deterrent Working Group members to recommend disposition of the long-term recommendations from the 2014 nuclear enterprise reviews and of the long-term efforts to achieve management, operations, and health outcomes directed by the 2018 Nuclear Posture Review. In order to address the direction from the July 2018 Deputy Secretary of Defense memorandum and the June 2019 NDERG Charter, DOD officials stated that the co-chairs of the Nuclear Deterrent Senior Oversight Group have been working with defense nuclear enterprise stakeholders to identify long-term issues that should be tracked to monitor the health of the enterprise. According to agency officials, they would like to adjust how long-term issues that relate to the enduring recommendations from the 2014 nuclear enterprise reviews are monitored. Examples include the need to sustain the current weapon systems until they are replaced, providing adequate funding for the acquisition of new systems, and improving the morale of nuclear forces. Since these recommendations are not expected to be closed as completed within the next few years, the Nuclear Deterrent Senior Oversight Group wants to find ways to improve how the recommendations can be tracked to monitor the health of the enterprise. According to DOD officials, they are currently working to identify relevant metrics from the existing tracking tool as well as existing data sources that might be leveraged to support the long-term monitoring of the health of the enterprise. This may be particularly helpful if the use of the existing tool is discontinued at some point after the 2020 time frame. The efforts of the military services and other DOD components to maintain current and complete information using the existing tracking tools for the 2014 and 2015 recommendations has the potential to aid the department. In particular, existing tools can be helpful for tracking and assessing both enduring recommendations from those reviews as well as additional efforts by the NDERG to assess and monitor the health of the nuclear enterprise. For example, existing outcome metrics can aid in the assessment of whether completed actions have addressed underlying issues that affect the health of the enterprise, identified risks can aid the department in addressing issues as they arise, and the use of the tools themselves can help maintain visibility across the DOD nuclear enterprise, including aiding the communication of timely information to senior leaders. DOD and the military services are experiencing challenges related to sustainment and maintenance of nuclear weapon systems—including challenges identified in recommendations from the 2014 nuclear enterprise reviews—and have ongoing and planned initiatives intended to mitigate these challenges. The military services face challenges related to operating weapon systems beyond their initial design life, parts availability and parts obsolescence, small fleet size, and the maintenance workforce. DOD and the services are mitigating sustainment and maintenance challenges through initiatives to increase parts availability and to improve depot-level maintenance, and through increased tracking of sustainment and maintenance problems. We reviewed sustainment and maintenance for the following nuclear weapon systems: Minuteman III. The Minuteman III ICBM is a strategic weapon system using a ballistic missile of intercontinental range. Missiles are dispersed in hardened silos to protect against attack and connected to an underground launch control center through a system of hardened cables. B-2 Spirit. The B-2 Spirit is a multirole bomber capable of delivering both conventional and nuclear munitions. B-52 Stratofortress. The B-52 Stratofortress is a long-range, heavy bomber that can perform a variety of missions. AGM-86B ALCM. The AGM-86B ALCM is a long-range, self-guided missile with a nuclear warhead that is carried by the B-52H bomber. E-4B NAOC. The E-4B NAOC is the primary survivable element of the National Military Command System through which the President, as Commander in Chief, and Secretary of Defense exercise national and nuclear command and control of military forces in day-to-day and crisis operations. In case of national emergency or destruction of ground command and control centers, the aircraft provides a highly survivable NC3 center to direct U.S. forces, execute emergency war orders, and coordinate actions by civil authorities. E-6B Mercury. The E-6B Mercury is a communications relay and, when manned, a strategic airborne command post aircraft. It provides survivable, reliable, and endurable airborne NC3 capabilities needed to direct, command, and control U.S. strategic nuclear forces. Ohio-class SSBN. The Ohio-class SSBNs are the most survivable leg of the strategic triad, serving as launch platforms for submarine- launched ballistic missiles. They are designed specifically for stealth and the precise delivery of nuclear warheads. Table 1 shows examples of sustainment challenges affecting these systems. According to DOD and service officials, while there are acquisition programs under way to replace most of these systems, the current nuclear enterprise systems remain necessary for years to come. The 2014 nuclear enterprise reviews included recommendations to sustain and maintain these systems until they are replaced, such as a recommendation to “fully fund increasing maintenance needs as the triad ages.” See appendixes II–VI for additional information and specific sustainment and maintenance challenges and initiatives for select systems. Almost all of the nuclear weapon systems we reviewed are experiencing challenges related to aging. Specifically, these weapon systems are being deployed beyond their originally intended service lives, which adds to the challenges of sustaining these systems. DOD, along with the Department of Energy, has undertaken an extensive, multifaceted effort to sustain and modernize U.S. nuclear weapons capabilities, including the nuclear weapons stockpile; the research and production infrastructure; and the NC3 system. Some of these sustainment efforts are directly linked to recommendations from the nuclear enterprise reviews of 2014 and the 2015 NC3 report. For example, the 2014 nuclear enterprise reviews recommended that the Air Force establish bomber and ICBM sustainment plans for aging platforms. The 2014 nuclear enterprise reviews also resulted in a recommendation to fully fund increasing maintenance needs as the nuclear triad ages. Table 2 provides additional examples of related recommendations from the 2014 reviews. According to DOD officials, as these nuclear weapon systems have aged they have required more maintenance in order to sustain them through their extended service lives, and they will continue to do so until they are replaced by new systems. For example, Air Force officials cited aircraft age as the major factor leading to corrosion and other airframe issues that the B-52 is experiencing. The first B-52 model was initially deployed in 1952, and the B-52H—the model currently in use—became operational in 1962. The Air Force now plans to sustain the B-52 until at least 2050, which will require increased maintenance and a series of modernization programs in the 2020s. The E-4B, first deployed in 1980, is also experiencing significant corrosion in the galley area, necessitating a fleet- wide galley replacement program. Neither the B-52 nor the E-4B have replacement programs identified. According to Air Force officials, aging components have also led to structural problems with the Minuteman III ICBM. The Minuteman III was deployed in 1970 with an original planned service life of 10 years. The Minuteman III is now expected to last until the 2030s, when it will be replaced by the Ground Based Strategic Deterrent system. In addition to the weapon systems, support components and support infrastructure are also experiencing age-related challenges. For example, according to Air Force officials, the support infrastructure for the Minuteman III in use today, known as the real property installed equipment, is the original infrastructure that was fielded with the Minuteman I weapon system in 1960, which reached operational capability in 1962. These officials stated that challenges at these facilities include corrosion, water intrusion, collapsed conduits, misaligned doors, and bulging walls. The need to sustain nuclear support equipment is reflected in a nuclear enterprise review recommendation to prioritize nuclear support and test equipment. Parts availability issues and parts obsolescence also affect maintenance on existing weapon systems across the nuclear enterprise. In many cases, the industrial base that produced specific parts for a weapon system is no longer active or is no longer producing the part, so when parts break there are no replacements available. For example, Air Force officials working to maintain the B-52 fleet told us that they have trouble finding suppliers who will produce the necessary parts for such an old airframe. Similarly, the Ohio-class SSBN program is experiencing challenges in sustaining submarines through their planned 42-year service life. The Ohio-class was initially intended to be operational for 30 years. Since it will be in service longer than expected, the Navy is finding that parts not originally intended to be replaced now need replacement. Navy officials stated that obsolescence has a greater impact for these parts that were never expected to fail and therefore do not have an industrial base to support replacements than for parts that the Navy has always planned to replace at some point during the Ohio-class service life. In certain scenarios, maintainers across several weapon systems have had to reengineer parts, because the original blueprints do not exist. Maintainers we spoke to reported long lead times to have parts fabricated and delivered, which extends the time that a system is offline for maintenance. The 2014 nuclear enterprise reviews included multiple recommendations to address parts obsolescence and availability problems in both the Air Force and the Navy, including the examples shown in table 3. Additionally, maintainers may cannibalize parts, a process by which parts are taken from one asset for use in another. This process is conducted during maintenance for both Air Force and Navy nuclear weapon systems. For example, according to Air Force officials, parts are routinely cannibalized from B-2 aircraft that are undergoing modifications so that they can be used on the operational B-2 aircraft. Similarly, Navy officials stated that parts are cannibalized from other classes of submarines to sustain Ohio-class SSBNs when replacement parts are not available elsewhere. Parts cannibalization has also occurred during engineered refueling overhauls. According to Navy officials, in the past, SSBNs completing refueling overhauls have cannibalized parts from SSBNs that are beginning to be overhauled. The final Ohio-class SSBN to undergo an overhaul, the USS Louisiana, will not have that option, because there will be no other SSBNs from which to cannibalize parts; all SSBNs except the USS Louisiana and USS Wyoming have already completed their overhauls. According to officials from the Office of the Chief of Naval Operations, they are not concerned about not being able to cannibalize parts for the remaining overhauls. Several legacy nuclear systems have a limited number of assets, which can create challenges for meeting operational requirements while at the same time conducting maintenance. In particular, the size of a fleet can create challenges when it becomes difficult or impossible to meet operational requirements. According to Air Force and Navy officials, maintenance challenges stemming from small fleet sizes particularly affect the B-2, E-4B, and E-6B weapon systems. Scheduling maintenance is one such challenge, because taking one aircraft down for maintenance will have a proportionally greater effect on the number of aircraft available for operations than it would for a larger fleet. For example, according to Air Force officials, the B-2 is experiencing challenges related to maintaining aircraft availability during the extensive modernizations that are being conducted, including integration of a new weapon and upgrades to its radar system. Scheduling this modernization process, part of the effort to sustain the B-2, is challenging given that there are only 20 aircraft in the fleet. Taking an aircraft down for maintenance limits the number of aircraft available for operational use by U.S. Strategic Command. Similarly, Air Force officials told us that the time needed for maintenance and modernization efforts on the E-4B was a primary factor leading to decreased aircraft availability of the E-4B, because of the small number of aircraft in the fleet—four in total. Having only four aircraft means that delays currently experienced during depot maintenance and installation of modifications have larger effects on the overall availability of the fleet. One aircraft unavailable as it undergoes these actions results in one quarter of the fleet being unavailable for operations. Additionally, unscheduled maintenance could exacerbate the issue of scheduling challenges and conflict with operational requirements. Having a small fleet with some systems in maintenance could also impede the force’s ability to surge if needed. The B-52 fleet has experienced a unique challenge, because it has recently been used extensively in conventional operations. According to Air Force officials, it takes time to change a B-52 configuration from conventional to nuclear to ready the aircraft for a nuclear mission, which may affect aircraft availability. According to officials from the Office of the Deputy Assistant Secretary of Defense for Nuclear Matters, reduced availability also negatively affects readiness through the reduction of training opportunities. Security-clearance backlogs for the maintenance workforce are a challenge with respect to certain nuclear weapon systems. Without at least a secret security clearance, maintainers may be limited in the activities they can perform on a nuclear system. For example, an Air Force official explained that without a clearance maintainers are not only limited in the activities they can perform on the B-2, but they cannot complete some of the training they need. To mitigate this challenge, the Air Force sometimes chooses to issue interim clearances. But in so doing unit commanders must accept additional risk. Specifically, since background investigations may not be complete at the time these interim clearances are issued, it is possible that someone who has been issued an interim clearance will ultimately be found ineligible for that security clearance due to information discovered during their background investigation. Similarly, there is a backlog of top secret clearances for missile-wing personnel working with the Minuteman III, including maintainers. Again, the services sometimes choose to issue interim clearances, but leadership must accept that risk, and interim clearances may have limitations. For example, according to officials from one of the missile wings we spoke with, a missileer in that wing with an interim top secret clearance can complete training for the Minuteman III but cannot be certified to be on a two-person alert team. The 2014 nuclear enterprise reviews included several recommendations to improve various issues related to workforce, including the examples shown in table 4. We have previously found that problems related to security-clearance backlogs and the resulting delays in determining clearance eligibility and issuing initial clearances can result in millions of dollars of additional costs to the federal government. We have also found that the backlogs can result in longer periods needed to complete national security–related contracts and lost opportunity costs if prospective employees decide to work elsewhere rather than wait to get a clearance. Further, we have found that the backlogs can result in diminishing quality of the work because industrial contractors may be performing government contracts with personnel who have the necessary security clearances but are not the most experienced and best-qualified personnel for the positions involved. Additionally, we identified the personnel security-clearance process as a high-risk area in March 2019 and we continue to monitor progress addressing the weaknesses in this area. The services have taken steps to ease the effect of parts availability issues and obsolescence. For example, partly in response to nuclear enterprise review recommendations, the Air Force has broadened the definition of the Minuteman III weapon system—a process the Air Force refers to as demarcation—and instituted programmed depot maintenance for the weapon system. The Air Force’s demarcation effort centralized parts funding and inventory management for all of the essential components of the Minuteman III and integrated the entire weapon system into a standard Air Force supply process. According to Air Force officials, the Air Force is also working with the Defense Logistics Agency to identify and catalog parts that previously had no identification numbers associated with them. Officials said that programmed depot maintenance is expected to result in a steady, predictive demand level for parts, which will help the Air Force ensure that parts are available and incentivize vendors to manufacture parts, including previously obsolete parts for which there was no steady source of supply. Additionally, both of these efforts are expected to reduce the likelihood that parts will be unavailable when needed. Navy officials explained that for the Ohio-class SSBN, when an industrial base supplier is not able to meet the need for certain obsolete parts, the Navy purchases enough parts to “stock the shelf” by including in one contract enough quantities of the part to last for the life of the SSBNs. Additionally, the Navy has developed programs such as the Trident Planned Equipment Replacement Program, which has identified over 300 critical parts and has them manufactured and ready to be used for replacement when SSBNs are undergoing planned maintenance. The Defense Logistics Agency has increased its support to the nuclear enterprise to help ensure that parts are available when they are needed. In 2015, the Defense Logistics Agency established a Nuclear Support Office from its headquarters staff to synchronize resources to ensure responsive support to the DOD nuclear enterprise. According to Defense Logistics Agency officials, the office has 13 people, three of whom are embedded at U.S. Strategic Command, Air Force Space Command, and Air Force Global Strike Command. In the Defense Logistics Agency’s 2018–2026 strategic plan, supporting the nuclear enterprise is the top objective. According to Defense Logistics Agency officials, they also have a series of new initiatives to increase materiel availability and accomplish activities such as paying for the cost of reverse engineering to fill in voids that exist in technical data for nuclear enterprise systems; working in additive manufacturing to set the standard for 3D printing and polymers across DOD and subsequently printing parts on demand; and identifying weaknesses in the industrial base and focusing investments in those areas. The focus of the material availability effort is presently to find out how to help the services when they cannot find a part and to address it in one of the initiatives. The Air Force and Navy have taken steps to improve depot-level maintenance across the nuclear enterprise. For example, the Air Force introduced programmed depot maintenance for the Minuteman III weapon system in 2014 and transformed ICBM weapon system sustainment processes into a standardized, integrated planning and support model. For the E-4B, according to E-4B program officials, the Air Force has initiated incentivized programmed depot maintenance gates that provide contractors additional financial incentive to complete increments of depot maintenance, as well as the entire depot maintenance process, on time or early. The E-4B program office is implementing this incentive structure in an effort to decrease the E-4B’s time spent in depot maintenance. Additionally, the Air Force has several initiatives under way to mitigate B- 2 sustainment and maintenance challenges, including increasing the intervals between depot-level maintenance and merging modernization and depot maintenance efforts so that the aircraft is down less and available more. In addition, there are multiple ongoing initiatives to improve the B-2’s supply chain, including using predictive analysis and forecasting tools to help determine how many spare parts to keep in stock. To sustain the Ohio-class SSBN fleet, the Navy has conducted engineered refueling overhauls on all SSBNs except for the USS Wyoming and USS Louisiana, the last two SSBNs to enter service. This major maintenance is intended to help sustain the Ohio-class SSBN fleet until its service life reaches 42 years and it is replaced by the Columbia- class SSBN. These engineered refueling overhauls have taken longer than originally anticipated. Navy officials attribute these delays to the submarines requiring more maintenance work than expected as well as some delays in acquiring parts. Over the past several years, DOD and the services have increased their attention to and tracking of nuclear weapon systems maintenance and sustainment issues. As we have previously found, DOD and the military services have taken steps to improve oversight of the nuclear enterprise in response to the 2014 reviews. For example, DOD has established or participated in a number of oversight organizations that aid in the management of the defense nuclear enterprise, including the NDERG, which was established in 2014 by the Secretary of Defense to ensure the long-term health of the nuclear enterprise by addressing issues identified in the 2014 nuclear enterprise reviews, including sustainment and maintenance-related issues. The Air Force and Navy have also taken actions to improve oversight of sustainment and maintenance. For example, the Air Force, through its Nuclear Mission Assessment effort, uses independent analyses of various data sources to recognize challenges within the Air Force nuclear enterprise, including sustainment and maintenance problems. Additionally, the Air Force implemented the Nuclear Weapon System Enterprise Review, which was developed in 2016 by the Air Force Nuclear Weapons Center with support from Air Force Materiel Command. According to Air Force documentation, the review provided timely insight into the comprehensive health of individual nuclear weapon systems and provided an assessment of how well the enterprise is performing. Nuclear weapon systems that were specifically reported on in the Nuclear Weapon System Enterprise Review included ALCM, Minuteman III, and NC3 systems. The Air Force modeled its Nuclear Weapon System Enterprise Review in part on assessment and reporting already completed for all aircraft, including the B-2 and B-52 bombers, through its Weapon System Enterprise Review briefings. Weapon System Enterprise Review metrics are tailored to each weapon system and have details on data such as cost, schedule, performance, and funding. These data are compiled into a quarterly briefing report for Air Force major commands and Air Force headquarters. According to Air Force officials, information included in the Nuclear Weapon System Enterprise Review was related to 10 recommendations from the 2014 nuclear enterprise reviews and the 2015 NC3 report. Tracking this information helped the Air Force to close out the recommendations assigned to Air Force Materiel Command. According to Air Force officials, as of July 2019 the Air Force had discontinued the use of the Nuclear Weapon System Enterprise Review. The officials said that Air Force Nuclear Weapons Center and Air Force Global Strike Command are currently collaborating on a replacement presentation focused on weapon system availability; however, this effort is not finalized. The officials further stated that the Air Force has transitioned to an Aircraft Availability Improvement Program construct with an aircraft readiness focus and is working to establish an equivalent for the nonflying weapon systems (i.e., Minuteman III and NC3). The Navy oversees its leg of the nuclear triad using the Navy Nuclear Deterrent Mission Oversight Council. The council is a senior Department of the Navy forum that is responsible for coordinating the Navy’s nuclear weapon activities (safety, security, reliability, and nuclear weapons incident response), operations, personnel, policy, material support, and oversight functions. According to Navy officials, the Navy Nuclear Deterrent Mission Oversight Council addresses long-term issues affecting the Navy’s nuclear enterprise and identifies and monitors risks associated with those issues, including the actions taken in response to sustainment and maintenance-related recommendations from the 2014 nuclear enterprise reviews. According to Navy officials, the Navy has also established an SSBN Sustainment Working Group and a Trident Planned Equipment Replacement Program Working Group to address Ohio-class sustainment and maintenance-related issues. DOD and the military services have made progress in addressing the recommendations from the 2014 nuclear enterprise reviews and the 2015 NC3 report. They have done so partially by establishing and improving a number of processes to aid in the sustainment of defense nuclear enterprise systems. The department is modifying the NDERG’s focus from monitoring the status of the 2014 recommendations to monitoring the long-term health of the enterprise. This shift in focus should position the NDERG to better perform its oversight functions as the principal integrated civilian–military governance body for the defense nuclear enterprise. This is important because many of the recommendations that remain open are focused on long-term sustainment of the enterprise or are designed to be closed only after progress in addressing the issues can be meaningfully evaluated. It is important that the department and the military services continue to use the successful tools they have created to monitor these efforts and leverage these tools (and the premises behind them) as they create new mechanisms to maintain senior-leader visibility of the defense nuclear enterprise. Providing current, complete, and relevant information on the status of service and DOD component actions to address recommendations and an understanding of metrics, milestones, and risks will allow senior leadership to maintain oversight of the department’s progress. In particular, such visibility will help senior leaders maintain awareness of the progress of efforts to address past failings, determine whether efforts are having the intended effects and achieving the desired outcomes of addressing root problems, and achieve the desired end states of a healthy defense nuclear enterprise. These existing processes can help inform additional processes the department develops to monitor the health of the nuclear enterprise. The collection and assessment of information to maintain the currency and completeness of information in existing tools may also allow the department to identify potential emerging issues that may negatively affect the vital programs, infrastructure, and personnel essential to the maintenance of an effective nuclear deterrent. We are making the following two recommendations to DOD: The Secretary of Defense should ensure that the Director of CAPE, in coordination with the Deputy Assistant Secretary of Defense for Nuclear Matters, the Deputy Assistant Secretary of Defense for Nuclear and Missile Defense Policy, and the Joint Staff Deputy Director for Strategic Stability, as co-chairs of the Nuclear Deterrent Senior Oversight Group, update the applicable guidance for methods of tracking and evaluating progress on implementation of the recommendations from the 2014 nuclear enterprise reviews, requiring DOD components to keep information—including any revised time frames—current. (Recommendation 1) The Secretary of Defense should ensure that the Under Secretary of Defense for Acquisition and Sustainment updates the applicable guidance for methods of tracking and evaluating progress on implementation of the recommendations of the 2015 NC3 report, requiring DOD components to keep information—including metrics for measuring progress and outcomes as well as any revised time frames that may extend out more than 1 year—complete and current. (Recommendation 2) We provided a draft of the classified report to DOD for review and comment. The department’s comments on the classified report are reprinted in appendix VII. In its comments, DOD concurred with both of our recommendations. DOD also provided technical comments on the classified report, which we incorporated as appropriate. In concurring with our first recommendation, DOD stated that the Nuclear Deterrent Senior Oversight Group co-chairs or, as necessary, the Deputy Secretary of Defense as the chair of the NDERG, will update the applicable guidance to ensure that time frames and other information associated with planned actions are kept up to date. In concurring with our second recommendation, DOD stated that the DOD CIO and, as appropriate, the Under Secretary of Defense for Acquisition and Sustainment as the NC3 capability portfolio manager, will update the applicable guidance to ensure that metrics, time frames, and other information associated with planned actions are kept up to date and complete. We are encouraged that DOD is planning to take these actions to address our two recommendations. We believe that providing current, complete, and relevant information on the status of service and other DOD component actions to address recommendations and an understanding of metrics, milestones, and risks will allow senior leadership to maintain oversight of the department’s progress. This may also allow DOD to identify potential emerging issues that may negatively affect the vital programs, infrastructure, and personnel essential to the maintenance of an effective nuclear deterrent. We are sending copies of this report to the appropriate congressional committees, and to the Secretary of Defense; the Under Secretary of Defense for Acquisition and Sustainment; the Chairman of the Joint Chiefs of Staff; the Secretaries of the Army, of the Navy, and of the Air Force; the Commander, U.S. Strategic Command; the Department of Defense Chief Information Officer; and the Director of the Office of Cost Assessment and Program Evaluation. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9971 or kirschbaumj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. Minuteman III is a strategic intercontinental ballistic missile (ICBM) weapon system that represents one leg of the nation’s nuclear triad. First deployed in 1970 with a planned service life of 10 years, the Minuteman III weapon system consists of missiles as well as 450 launch facilities and 45 launch control centers. The Minuteman III service life was extended since its deployment by various service-life extension programs. Launch facilities are connected to underground launch control centers through a system of hardened cables. A launch facility is an unmanned site that houses the missile and all equipment required to maintain the missile in a launch-ready configuration. These underground facilities have been considered part of the Minuteman III weapon system since 2014. Missile alert facilities are manned compounds that encompass the launch control center, a launch control support building, and a launch control equipment building. Missile alert facilities are crewed by security personnel, a cook, a facilities manager, and a launch crew. Launch crews, consisting of two officers, perform around-the-clock alert in the underground launch control center. See figure 5 for components of the Minuteman III weapon system. Nuclear command, control, and communications (NC3) systems and related procedures ensure launch crews in the launch control centers can receive and authenticate the President’s authorization for the use of nuclear weapons. In the event that connectivity is lost between a launch control center and an associated launch facility, other NC3 capabilities are available to carry out the direction of the President. For example, launch control centers aside from the one that lost connectivity can communicate with that launch facility as well as numerous other launch facilities. Further, an E-6B aircraft configured as an Airborne Command Post can transmit a launch command to the ICBM force through the Airborne Launch Control System capability. Minuteman III has undergone many life extension sustainment efforts to maintain its warfighting capabilities. The Air Force plans to sustain Minuteman III through 2030—50 years past its initial planned service life—and gradually draw down the weapon system before it is finally retired in 2036, as it is replaced by the Ground Based Strategic Deterrent ICBM weapon system. The Ground Based Strategic Deterrent has a planned initial operating capability date of 2029 and is to be fully deployed by 2036. Figure 6 provides a timeline of the expected service life of the Minuteman III ICBM weapon system. According to Air Force officials, Minuteman III is experiencing challenges related to aging facilities, aging infrastructure, and parts obsolescence. Aging facilities and infrastructure continue to affect the weapon system. According to Air Force officials, most of the real property installed equipment in use today is the original infrastructure that was fielded with the Minuteman I weapon system in 1960, achieving operational capability in 1962, and only slight modifications have been made over the years. Additionally, challenges with critical subsystems also exist, and while there are short-term mitigation strategies for each subsystem, there are no long-term replacements planned for the Minuteman III weapon system except by the fielding of its replacement program: the Ground Based Strategic Deterrent. Examples of facilities and infrastructure challenges include corrosion, water intrusion, collapsed conduits, misaligned doors, and bulging walls. According to Air Force officials, even attempting to replace small items can be difficult, because multiple subsystems must be replaced to support the modification. Diminishing manufacturing sources, material shortages, and obsolescence issues are additional contributing factors, because they cause difficulties in maintaining a credible supply chain for Minuteman III parts. Additionally, officials said that depot maintenance, interim maintenance, and organizational maintenance have all been affected by parts obsolescence, diminishing manufacturing sources, and material shortages, as has NC3 equipment. The Minuteman III weapon system is facing continued asset attrition. According to Air Force officials, as a result of the expected attrition of current field assets, the Minuteman III weapon system will be unable to meet full mission requirements after 2026 should full deployment be required. The Air Force expends four Minuteman III ICBMs per year on testing. According to the officials, continued asset attrition is also affecting the Minuteman III retirement schedule. Additionally, the Air Force Minuteman III program has experienced personnel challenges. According to Air Force officials, the Air Force has a backlog for top secret clearances for missile wing personnel, including maintainers, and it can take up to 2 years of a missileer’s 5-year commitment for a top secret clearance to come through. The officials told us a missileer can complete training with an interim top secret clearance but cannot be certified under the Personnel Reliability Program and therefore cannot be assigned to a two-person alert team. This makes it a challenge for missileers with interim clearances to keep up with their peers. Additionally, since commanders cannot assign them to alert duty, it puts additional burden on those missileers who are cleared to perform more alert-duty assignments. According to Air Force officials, they have also identified challenges associated with scheduling maintenance activities, including the need to balance longer working days with the additional risks that maintainers face as a result of these longer days. Officials also said that as launch control centers, launch facilities, and other elements of the Minuteman III weapon system are dispersed over large areas that make up the missile fields, maintainers may need to travel several hours from their base to arrive at the location of the maintenance activity. These increased travel times have resulted in extended workdays for maintainers and security forces or the need to split maintenance jobs between two shifts, which results in decreasing the number of personnel available to work at other locations. To mitigate challenges associated with the Minuteman III weapon system—including limitations in the availability of parts—the Air Force has broadened the definition of the Minuteman III weapon system, which is a process the Air Force calls demarcation. It was broadened to include some additional facilities related to the Minuteman III weapon system, and programmed depot maintenance was instituted for it. According to Air Force officials, demarcation centralized parts funding and inventory management for all of the essential parts of the Minuteman III and integrated the entire weapon system into the standard Air Force supply process. Additionally, according to the officials, the ICBM System Directorate has established a Weapon System Supply Chain Management office to oversee the commodity and organic support required to meet the daily needs of the warfighter and to sustain Minuteman Ill throughout Ground Based Strategic Deterrent deployment. The officials said the Weapon System Supply Chain Management office conducts predictive forecasting of the demand for parts through predictive data analysis, which tracks the potential demand for parts as well as parts supportability as an ongoing analysis process. Additionally, the Weapon System Supply Chain Management office does this through an analysis tool that draws on information from multiple supply databases to identify rising request levels in maintenance data systems and mission-capable conditions reported from the field. This tool uses data to identify parts that will be needed. Additionally, Air Force Global Strike Command conducted an end-to-end review of Minuteman III weapon system maintenance to determine whether ICBM maintenance organizations are organized, trained, and equipped to meet the current and future needs of the weapon system. The review noted that a questionable manpower standard, aging resources and equipment, and organizational inefficiencies have reduced the effectiveness of maintenance and the health of the Minuteman III. Subject-matter experts from various Air Force organizations and the Navy assessed maintenance and provided recommendations on methods, training, resources (supply and equipment), infrastructure, manpower, support, culture, and leadership. For example, the review observed that parts and equipment availability challenges continue to affect the mission. From this observation the review offered several recommendations, including that the Air Force Nuclear Weapons Center set aside all parts for weapon system testing so that they are available when the tests occur, every 5 years. This is intended to ensure that the parts that are set aside are not used at the missile wings. The review also recommended a number of efforts to improve the management of maintenance schedules, including increased coordination and planning of maintenance schedules in advance. According to Air Force officials, this allows maintenance commanders to make informed decisions, in advance, regarding when longer working days are appropriate. A number of service-life extension programs are under way to sustain the Minuteman III until the Ground Based Strategic Deterrent arrives. Additionally, ICBM programmed depot maintenance was introduced in 2014 and transformed processes for ICBM weapon system sustainment into a standardized, integrated planning and support model that performs maintenance to refurbish portions of the weapon system. According to Air Force officials, the idea was to have the Minuteman III weapon system undergoing depot maintenance in ways similar to the periodic depot maintenance that aircraft undergo. However, the depot team would have to conduct portions of the maintenance in the missile fields instead of bringing the weapon system to a depot. This new programmed depot maintenance takes individual Minuteman III launch facilities offline to conduct major maintenance. Air Force Nuclear Weapons Center works with the Defense Logistics Agency to procure parts as part of programmed depot maintenance planning. According to Air Force officials, the plan is to have 57 launch facilities go through the programmed depot maintenance process each year, with a plan to refurbish all launch facilities over an 8-year period. Additionally, the current programmed depot maintenance efforts are implementing a standard set of maintenance efforts across all facilities, but some additional issues are also being addressed on a case-by-case basis. To track the health of the Minuteman III, the Air Force Nuclear Weapon Center assigns predictive health measures to the systems. These predictive health measures estimate when there will be a specific maintenance activity needed for each weapon system part―for example, when a part will likely fail and need to be replaced—based on assessments of historic data and engineering analysis. It emphasizes ICBM sustainment through reliability-centered maintenance, which allows for the continuous evaluation of system performance. Additionally, the predictive health measures, based on data from Air Force maintenance data-collection systems, are analyzed monthly for all launch facilities and launch control centers across the three missile wings. According to Air Force officials, predictive health measures enable the Air Force to identify early indications of when systems may need additional maintenance as well as to analyze health trends to identify issues―such as parts failures―across all of the Minuteman III force. Additionally, the use of predictive health measures and reliability-centered maintenance allows the Air Force to better plan for when maintenance activities, and related resources, will be needed to address issues prior to when they arise. According to Air Force officials, Air Force Global Strike Command also collects and reports on metrics monthly, based on Integrated Maintenance Data System write-ups and predictive health metrics. Officials told us that the Integrated Maintenance Data System is a difficult system to learn and no formal training on the system is available. The data quality in the Integrated Maintenance Data System is highly dependent on the individual expertise of whoever enters it. The B-2 Spirit is a multirole, dual-capable heavy bomber. The B-2 is the only U.S. aircraft that combines a long-range capability, a large payload, and stealth into a single platform, giving it the ability to project air power globally. The B-2 became operational in 1997, and the current B-2 operational fleet consists of a total of 20 aircraft. The 509th Bomb Wing, located at Whiteman Air Force Base, Missouri, is the sole operational unit for the B-2. The 509th Bomb Wing usually maintains 15 operationally available B-2s. At any one time, there are two aircraft undergoing sustainment and modernization upgrades, two in programmed depot maintenance, and one designated as a test aircraft. The Air Force plans to sustain the B-2 into the 2030s (see fig. 7). The B-2 will eventually be replaced by the B-21, which will assume the penetrating strike role of the B-2. The B-21 is expected to become operational in the mid-2020s, but no replacement schedule for the B-2 has been identified. The B-2 is undergoing multiple modernization programs, while maintaining existing capabilities through form, fit, and function replacements for components that are obsolete or no longer supportable. B-2 modernization efforts are ongoing for communications, navigation, defensive management, weapons, and the airframe. Because the B-2 is aging and the fleet is small, parts obsolescence is a challenge. A unique sustainment aspect of the B-2 is the focus on managing its low-observable stealth capability. The B-2 Low Observable Integrated Product Team manages the Low Observable Signature and Supportability Modifications portfolio of projects, which is aimed at maintaining the stealth capability of the B-2 by monitoring, maintaining, and enhancing the radar cross section (or “signature”) of the aircraft. In addition to specific efforts to sustain the low-observable stealth capability, every other sustainment and modernization activity for the B-2 must be assessed early in the planning stages for any effects on this capability. According to Air Force officials, in addition to maintaining readiness for its nuclear mission, the B-2 platform is also in high demand to support conventional bomber missions. However, the Air Force has a limited number of aircraft to meet this demand. Consequently, the Air Force’s B-2 Division, along with Air Force Global Strike Command and the 509th Bomb Wing, must carefully manage the timing of maintenance activities, aircraft modifications, programmed depot maintenance, assignment of a flight test aircraft, and the flying-hour program. This requires an intricate schedule of availability of aircraft for each effort, while trying to maintain overall operational availability for the B-2 fleet. According to Air Force officials, small-fleet dynamics have led to high costs, diminishing vendor and parts availability, and readiness concerns. Various initiatives are under way to improve the availability of B-2s. A cumulative increase of one additional aircraft available for operations is anticipated by fiscal year 2022. Several of these initiatives are directly related to improving sustainment of the B-2 and maintenance processes and procedures. Examples of sustainment and maintenance-related initiatives include the following: The B-2 Programmed Depot Maintenance Process Improvement initiative is a collaborative effort between the B-2 program office and Northrop Grumman to increase capacity during the depot maintenance process in order to incorporate modifications during depot maintenance. This initiative is expected to result in reduced downtime at the 509th Bomb Wing by allowing modifications that would normally occur at the wing―making an aircraft unavailable for operations―to occur during planned depot maintenance. The B-2 program office increased the interval between programmed depot maintenance periods from 7 years to 9 years. The original B-2 programmed depot maintenance interval of 7 years was driven by the expected life of low-observable coatings. According to B-2 program officials, they have since determined that the expected life of these coatings is 9 years. Additionally, the Air Force’s B-2 Division established the B-2 Obsolescence Integrated Product Team in 2018 to address management oversight of obsolescence. The team convenes monthly to develop a strategic plan to enhance processes, communications, and consolidation of obsolescence issues affecting B-2 modernization and sustainment. A list of obsolete parts, currently totaling over 100, as well as planned mitigation strategies, is consolidated and reviewed quarterly. The integrated product team is also developing a Diminishing Manufacturing and Materiel Shortages Management Plan to define the structure, process, management, and oversight of obsolescence for the life cycle of the B-2. Further, according to Air Force documentation, for each B-2 sustainment and modernization program, the government and prime contractor establish a joint Obsolescence Working Group that is responsible for reviewing the program’s strategy to mitigate diminishing manufacturing and materiel shortages. The B-52 Stratofortress is a dual-capable heavy bomber used to meet the United States’ airborne strategic nuclear deterrence and global precision attack mission and objectives. The B-52 began operations in 1952. Eight models were produced, with a total production quantity of 742. The final version of the B-52, the “H” model, was the last model produced and became operational in 1961. The current B-52 operational fleet consists of a total of 76 aircraft, 46 of which are designated as nuclear capable. B- 52 operational units consist of the 2nd Bomb Wing, located at Barksdale Air Force Base, Louisiana, and the 5th Bomb Wing, located at Minot Air Force Base, North Dakota. The B-52 originally had a planned service life of approximately 20 years. However, the Air Force now plans to sustain the B-52 until at least 2050 (see fig. 8). An eventual replacement for the B-52 has not yet been identified. The B-52 is undergoing several modernization programs planned for completion in the 2020s. The B-52 Commercial Engine Replacement Program will replace the aging TF33-PW-103 engine with new commercial-off-the-shelf engines capable of meeting the needs of the B- 52 platform to keep the B-52 viable until 2050 and beyond. The engine replacement program was scheduled to begin in fiscal year 2019 and to be completed in fiscal year 2023. Additional modernization programs include installation of a Global Positioning System Interface Unit and a radar modernization program. According to B-52 maintainers, the biggest maintenance limitation they are experiencing is with the engine. In 2017, an engine fan disk failure on one of eight engines on a B-52 caused the engine to detach from the aircraft while in flight. The Air Force has identified the resulting fan disk inspection and replacement as a serious risk due to the time it will take to complete and expects the inspection, removal, and replacement to have an effect on the fleet into the 2020s. Further, the current TF33 engines are unsupportable beyond 2030. According to Air Force officials, the engine replacement program is expected to negatively affect aircraft availability rates until it is completed in 2023. Air Force officials also expressed concern that, because the new commercial engines have many digital components, their installation could increase the B-52’s cybersecurity risk. At 60 years old, the B-52 is experiencing structural issues typical of aging aircraft. The extension of the B-52’s service life into the 2050s likely imposes additional unforeseen sustainment and modernization challenges. The aging airframe has required increased depot-level maintenance to correct, for example, problems related to stress corrosion and cracking on the airframe. Further, industry is no longer able to support these aging systems, and the systems have experienced declining performance and system failure. According to Air Force officials, it is difficult to maintain suppliers who will produce the necessary parts for such an old airframe. According to officials at both B-52 wings, a security-clearance backlog limits the number of trained and available B-52 maintainers. Both B-52 wings also have shortages of experienced maintainers. Additionally, the demands of the B-52’s conventional mission create challenges to ensuring that they are available for their nuclear mission. The B-52 has been used in operations against the Islamic State in Syria. According to officials at both B-52 wings, the conventional mission is the day-to-day focus of most B-52 operators and maintainers. These officials said that it is sometimes challenging to shift their collective mindset to focus on the nuclear mission. Further, the B-52 requires different configurations for its conventional and nuclear missions. According to B-52 maintenance officials, the time it takes to change the configurations affects how quickly the aircraft can be ready for a nuclear mission. An official from one B-52 operations group expressed concern that if the B-52 continues to be used heavily in conventional operations, it will begin to experience airframe and personnel problems similar to those that have affected the B-1, which has been used extensively in recent conventional bombing operations. The B-52 engine replacement program is expected to allow the engines to be sustained until the 2050s, when the B-52 is expected to retire. In addition, the modern engines being installed will increase the B-52’s range by approximately 30 percent, significantly decrease maintenance costs and downtime, provide the additional electrical power required for follow-on systems, and decrease the B-52’s dependency on refueling tankers for both conventional and nuclear long-range strike sorties because it will be able to fly longer without being refueled. The B-52 program office is leading a B-52 Aircraft Availability Improvement Plan, which is an enterprise-wide effort to increase the number of B-52s available to operational units. According to officials, the program office is leading an initiative to reduce the number of aircraft that are at the depot at any given time from 11 to 9. This would increase the availability of aircraft to meet operational requirements. This effort is in the early implementation stages, and the program office has not yet evaluated the results. The B-52 program office mitigates parts obsolescence issues through active vendor management, selection of vendors who use an open systems approach, use of predictive database tools to identify diminishing manufacturing and materiel shortages, and leveraging industry and government reporting systems that track diminishing manufacturing and materiel shortages. The AGM-86B ALCM is a long-range self-guided missile with a nuclear warhead that is carried by the B-52H Stratofortress bomber. ALCM complements the B-52 heavy bomber in its strategic mission; its primary missions are strategic attack, interdiction, and suppression of enemy air defenses. It is designed to be carried on the internal B-52 common strategic rotary launcher or externally on pylons located underneath each wing (see fig. 9). The ALCM air vehicle is powered by a low-thrust turbofan engine and flies at subsonic speeds. After release from the carrier aircraft, the ALCM proceeds autonomously to its target. ALCM became operational in 1982 and, according to Air Force officials, had an original planned service life of 10 years; it is on average 25 years beyond its planned service life (see fig. 10). Additionally, ALCM has experienced aging issues with multiple subsystems. For example, the officials told us the Bomber Weapons Integration Equipment, pylons, launcher, common support equipment, ALCM-peculiar support equipment, and automated test equipment all have aging and supportability issues that require assessment and actions that must be taken going forward. Air Force officials stated that because of ALCM’s age, diminishing manufacturing sources and material shortage issues occasionally arise that have required requalification of a product line or qualifying a new source. Additionally, they said that ALCM maintenance and analysis trends have highlighted that electrical components and bearings are wearing out. According to Air Force officials, the ALCM fleet, made up of approximately 535 missiles in active inventory as of May 2019, is affected by attrition resulting from testing. The ALCM is operationally tested with six force development evaluations and two functional ground tests each year. According to Air Force officials, the testing employs ALCM fleet inventory missiles that are consumed during live launch and destructive testing, thereby reducing the fleet by eight missiles per year. The officials noted that the fleet would be sustainable longer if the decision was made to stop testing. However, this would mean that fewer data—collected during the annual tests—would be available to predict the life of the missile, and the Air Force would lose full confidence that it could execute ALCM’s mission. According to Air Force officials, the ALCM will be sustained through 2030. Service-life extension programs have been implemented to sustain the weapon system, and maintenance is performed every 6 years to exchange the missile’s engine. In order to extend the ALCM’s service life until a replacement system is fielded, service-life extension programs were developed through surveillance, studies, and analysis programs that identified numerous components for replacement as a result of aging and obsolescence issues. Officials said these programs address replacement of aged brittle components, bearings, and circuitry and electronic components within navigation and guidance systems. According to Air Force officials, maintainers are being proactive in identifying parts on the ALCM system that will experience issues in the future. Additionally, continued monitoring through flight tests and aging surveillance programs will enable them to identify new aging issues, which may drive additional service-life extension efforts. To mitigate challenges that arise, there is ongoing coordination between the ALCM and Long-Range Stand Off program offices to develop plans to retire ALCMs as Long-Range Stand Off production is executed through full operational capability and complete deployment. To mitigate challenges with support equipment, supportability trades are being conducted for the launcher and pylon service-life extension, and a gap analysis is being conducted to identify components, processes, and procedures that need to be modified to ensure service life through 2030. According to Air Force officials, maintainers are looking for ways to be proactive in maintaining support equipment and identifying future issues before parts break, as they are doing for the missile itself. Through the Automatic Test Systems program office, the Electronic System Test Set is also encountering aging and supportability issues that are being addressed through multiyear technical insertion projects. Additionally, predicting new effects of aging on service life grows increasingly challenging as 2030 approaches. The Ohio-class SSBNs constitute the sea-based leg of the strategic triad. Each SSBN is capable of carrying and launching 20 D-5 Trident submarine-launched ballistic missiles, which can deliver multiple nuclear warheads. The first Ohio-class SSBN, the USS Ohio, entered service in 1981. The last Ohio-class SSBN, the USS Louisiana, entered service in 1997. The Navy maintains a fleet of 14 Ohio-class SSBNs. Eight of the SSBNs are deployed in the Pacific Ocean, homeported in Bangor, Washington, and six are deployed in the Atlantic, homeported in Kings Bay, Georgia. According to a DOD Inspector General report, in a 1998 memorandum from the Commander of the Naval Sea Systems Command to the Chief of Naval Operations, the Navy documented its decision to extend the original 30-year service life of the Ohio-class SSBNs to 42 years. The report noted that this decision was supported by a Navy- directed study led by the manufacturer of the Ohio-class, General Dynamics Electric Boat Division, which determined that extending the service life of the Ohio-class SSBNs to 42 years was technically feasible. Subsequently, in a 2017 memorandum from the Commander of the Naval Sea Systems Command to the Program Executive Office for Submarines, the Commander stated that extensions beyond 2042 were not technically feasible. However, Navy officials said that they are beginning to consider options in case the replacement program, the Columbia-class SSBN, is delayed. As we previously reported, Navy officials noted that the service has never operated a nuclear-powered submarine for as long as 42 years. The Navy plans to replace the 14 Ohio-class SSBNs with 12 Columbia- class SSBNs. The first of the Ohio-class SSBNs is scheduled to be retired from active service in 2027. The remaining Ohio-class SSBNs will be retired at a rate of one per year, with the last one exiting service in 2040 (see fig. 11). According to Navy officials, they do not have a contingency plan in case the Columbia-class SSBN acquisition dates are delayed. However, they said that the fact that 14 Ohio-class SSBNs are being replaced by 12 Columbia-class SSBNs provides some extra time for replacement in case Columbia is delayed. Specifically, there will be an estimated 2 years between when the last Columbia-class SSBN is delivered and the last Ohio-class SSBN is retired. Navy officials also said that they are trying to gather the necessary data to lay the ground work now to be able to make engineering decisions in 10 years about the feasibility of sustaining the Ohio-class SSBNs in the event that the Columbia-class is delayed. The Navy is experiencing challenges in sustaining the Ohio-class SSBN through its planned 42-year service life. According to Navy officials, since the Ohio will be in service longer than expected, the Navy is encountering parts that need replacement that were not originally intended to be replaced. There is no industrial base of suppliers to support the replacement of some of these parts. In addition, the overall amount of maintenance required for the SSBNs increases as they age. According to Navy officials, both of these issues contribute to diminishing manufacturing sources and material shortages for the Ohio-class SSBNs. According to May 2019 congressional testimony by the Director of the Navy’s Strategic Systems Programs, the D-5 Trident submarine-launched ballistic missile has also been deployed for longer than its original planned service life. Specifically, it has been deployed for over 25 years, and the Navy now plans to operate the D-5 for over 50 years total. It has undergone service-life extension programs and is operating on new rocket motors. However, according to the Director’s testimony, this will be more than double the historical service life of any previous sea-based strategic deterrent system. Engineered refueling overhauls—major maintenance periods that occur once during an SSBN’s life—have been completed for all except the last two Ohio-class SSBNs to enter service, the USS Wyoming and the USS Louisiana. The USS Wyoming is currently undergoing its overhaul and is scheduled to complete it in July 2020. The USS Louisiana was scheduled to begin its overhaul in September 2019 and complete it in April 2022. According to Navy officials, in the past SSBNs completing refueling overhauls have cannibalized parts from SSBNs that are beginning to be overhauled. The final Ohio-class SSBN to undergo an overhaul, the USS Louisiana, will not have that option, because there will be no other SSBNs from which to cannibalize parts. However, these officials noted that they have not encountered any insurmountable issues thus far in planning the Louisiana’s overhaul. The DOD Inspector General reported in June 2018 that the Navy did not have a contingency plan in the event that the Columbia-class is delivered late. The Navy has identified a number of efforts under way to ensure that it reduces risks in both the maintenance of the current Ohio-class SSBN and the acquisition schedule of the Columbia-class SSBN. However, as we reported in December 2017 and again in March 2019, the Columbia-class program is facing more risks than its predecessors from its aggressive and concurrent schedule as a result of the continued and pressing need for it to meet the Navy nuclear deterrent requirements. The first Ohio-class SSBN is scheduled to be retired in 2027, and another is to follow each year until 2040. The first Columbia-class SSBN is scheduled to enter service in fiscal year 2031, and another is to follow each year thereafter. We have previously reported that the Navy also plans to increase investment in its SSBN maintenance facilities, equipment, and workforce to improve the execution of SSBN maintenance. According to Navy officials, they have several strategies to combat diminishing manufacturing sources and material shortages. For example, the Ohio program office has made “life of type” purchases for some parts for which the industrial base cannot meet the demand. In other words, according to program officials, the program office purchases in one contract enough of that part to last for the entire life of the SSBN—a large enough order to make it worth the time and cost for a manufacturer to produce the parts. According to the officials, another solution is to retrofit the pieces being used to build the Columbia-class SSBNs to support the needs for the Ohio-class SSBNs. For example, the Navigation Process Unit was retrofitted from the Columbia to use on the Ohio. This allows the Navy to purchase these components from manufacturers who will already be making them for the Columbia. The Navy has initiated major modernizations on a number of systems on the Ohio to upgrade those systems with new capabilities. According to Navy officials, modernization efforts are being planned for navigation, radio, and electronic communications systems, among others. The Navy has also initiated a program to refurbish and extend the service lives of D- 5 Trident submarine-launched ballistic missiles to about 2040. As Columbia-class SSBNs begin to replace Ohio-class SSBNs, refurbished D-5s carried by retiring Ohio-class SSBNs will be transferred to new Columbia-class SSBNs. Columbia-class SSBNs will continue to be armed with these refurbished D-5s until about 2040, at which time the D-5s are to be replaced by a successor submarine-launched ballistic missile. According to Navy officials, maintaining one strategic weapon system configuration during the transition to Columbia is beneficial from a cost, performance, and risk-reduction standpoint. In 2018, the DOD Office of Inspector General reported that the Secretary of the Navy and the Chief of Naval Operations have formally designated strategic nuclear deterrence as the Navy’s top priority. According to the report, as a result, the Navy has reduced the time required for engineered refueling overhauls of SSBNs, increased workforce size at shipyards, accelerated and improved shipyard workforce training, and improved SSBN maintenance procedures and schedules. However, while the Navy was able to reduce the time required for its last two engineered refueling overhauls, it has not hit the target of 27 months since 2010. In addition, according to officials the Navy has created two working groups—the SSBN/Guided Missile Nuclear Submarine Working Group and the Trident Coordination Group—to monitor and mitigate Ohio-class sustainment and maintenance challenges. Joseph W. Kirschbaum, (202) 512-9971 or kirschbaumj@gao.gov In addition to the contact named above, key contributors to this report were Penney Harwell Caramia, Assistant Director; R. Scott Fletcher; Jonathan Gill; Susannah Hawthorne; Brent Helt; Joanne Landesman; Amie Lesser; K. Ryan Lester; Ned Malone; Gabrielle Matuzsan; and Michael Shaughnessy. Nuclear Weapons Sustainment: Fiscal Year 2018 Nuclear Forces Budget Estimates. GAO-19-127R. Washington, D.C.: November 2, 2018. Defense Nuclear Enterprise: DOD Continues to Address Challenges but Needs to Better Define Roles and Responsibilities and Approaches to Collaboration. GAO-19-29. Washington, D.C.: November 1, 2018. Defense Nuclear Enterprise: Processes to Monitor Progress on Implementing Recommendations and Managing Risks Could Be Improved. GAO-18-144. Washington, D.C.: October 5, 2017. Nuclear Weapons: DOD Assessed the Need for Each Leg of the Strategic Triad and Considered Other Reductions to Nuclear Force. GAO-16-740. Washington, D.C.: September 22, 2016. Defense Nuclear Enterprise: DOD Has Established Processes for Implementing and Tracking Recommendations to Improve Leadership, Morale, and Operations. GAO-16-597R. Washington, D.C.: July 14, 2016. Nuclear Weapons Council: Enhancing Interagency Collaboration Could Help with Implementation of Expanded Responsibilities. GAO-15-446. Washington, D.C.: May 21, 2015.", "summary": "In 2014, the Secretary of Defense directed two reviews of DOD's nuclear enterprise. These reviews made recommendations to address problems with leadership, organization, investment, morale, policy, and procedures, as well as other shortcomings that adversely affected the nuclear deterrence mission. In 2015, DOD conducted a review focused on NC3 systems, which resulted in additional recommendations to improve NC3. The National Defense Authorization Act for Fiscal Year 2017 includes a provision for GAO to review DOD's processes for addressing these recommendations. This report addresses the extent to which DOD has made progress in (1) the implementation and tracking of the recommendations from the 2014 and 2015 nuclear enterprise reviews and (2) addressing sustainment and maintenance-related challenges and planning for the continued sustainment and maintenance of existing defense nuclear enterprise systems. GAO reviewed documents and interviewed DOD officials. This is a public version of a classified report that GAO issued in October 2019. Information that DOD deemed classified has been omitted. The Department of Defense (DOD) continues to make progress in implementing recommendations to improve the nuclear enterprise. These recommendations stemmed from DOD's 2014 internal and independent nuclear enterprise reviews, a U.S. Strategic Command 2014 memorandum, and an internal DOD 2015 report on nuclear command, control, and communications (NC3). Since GAO last reported—in November 2018—an additional five of the 247 sub-recommendations from the 2014 reviews have been closed; 91 remain open. In that time, DOD has also closed two more of the 13 recommendations from the 2015 review; six remain open. However, the key tracking tools DOD uses to provide visibility on the status of the recommendations do not provide current and complete information. For example, for those items that are behind schedule, many of the expected completion dates have not been updated to reflect when the items are now expected to be completed. The current DOD guidance for tracking the recommendations' status does not include a specific requirement to keep the information current in the tracking tools. Until DOD addresses these issues, it will not have a complete and accurate picture of when tasks are expected to be finished, whether progress is being made, whether efforts have stalled, or if there are other challenges. Ensuring that there is current and complete information regarding enduring recommendations would also help inform DOD's effort to monitor the health of the defense nuclear enterprise. DOD and the military services are experiencing challenges related to sustainment and maintenance of nuclear weapon systems and have ongoing and planned initiatives intended to mitigate these challenges. All of the systems we reviewed have been operational since before 1998, making these systems at least 22 years old (see figure). The age of the systems has resulted in maintenance and supply issues. For example, the Ohio -class submarine has experienced the failure of parts that were not originally intended to be replaced. DOD and the services have ongoing and planned efforts to mitigate these challenges, such as improving maintenance processes and sources of supply. GAO is making two recommendations for DOD to update guidance to require DOD components to keep information on recommendations current and complete. In written comments on the classified report, DOD concurred with these recommendations.", "document_type": "gao"}
{"report": "Drugs sold in the United States—including active pharmaceutical ingredients (APIs) and finished dosage forms—are manufactured throughout the world. According to FDA, as of August 2019 about 70 percent of establishments manufacturing APIs and more than 50 percent of establishments manufacturing finished drugs for the U.S. market were located overseas. As of March 2019, FDA data showed that India and China had the most manufacturing establishments shipping drugs to the United States, with about 40 percent of all foreign establishments in these two countries. (See fig. 1.) FDA is responsible for overseeing the safety and effectiveness of all drugs marketed in the United States, regardless of where they are manufactured. Drugs manufactured overseas must meet the same statutory and regulatory requirements as those manufactured in the United States. FDA’s Center for Drug Evaluation and Research (CDER) establishes standards for the safety, quality, and effectiveness of, and manufacturing processes for, over-the-counter and prescription drugs. CDER requests that FDA’s Office of Regulatory Affairs (ORA) inspect both domestic and foreign establishments to ensure that drugs are produced in conformance with applicable laws of the United States, including current good manufacturing practice (CGMP) regulations. FDA investigators generally conduct three main types of drug manufacturing establishment inspections: preapproval inspections, surveillance inspections, and for-cause inspections, as described in table 1. At times, FDA may conduct an inspection that combines both preapproval and surveillance inspection components in a single visit to an establishment. FDA uses multiple databases to select foreign and domestic establishments for surveillance inspections, including its registration database and inspection database. Because the establishments are continuously changing as they begin, stop, or resume marketing products in the United States, CDER creates a monthly catalog of establishments. The establishments in the catalog are prioritized for inspection twice each year. In our 2008 report we found that, because of inaccurate information in FDA’s databases, the agency did not know how many foreign drug establishments were subject to inspection. For example, some establishments included in FDA’s registration database may have gone out of business and did not inform FDA that they had done so, or they did not actually manufacture drugs for the U.S. market. In our report, we noted that some foreign establishments may register because, in foreign markets, registration may erroneously convey an “approval” or endorsement by FDA, when in fact the establishment may never have been inspected by FDA. We recommended that FDA take steps to improve the accuracy of this registration information. In our 2010 and 2016 reports we found that FDA had taken steps to improve the accuracy and completeness of information in its catalog of drug establishments subject to inspection, such as using contractors to conduct site visits to verify the existence of registered foreign establishments and confirm that they manufacture the products that are recorded in U.S. import records. To prioritize establishments for surveillance inspections, CDER applies a risk-based site selection model to its catalog of establishments to identify those establishments (both domestic and foreign) that, based on the characteristics of the drugs being manufactured, pose the greatest potential public health risk should they experience a manufacturing defect. This model analyzes several factors, including inherent product risk, establishment type, inspection history, and time since last inspection, to develop a list of establishments that FDA considers to be a priority for inspection. Through this process, CDER develops a ranked list of foreign and domestic establishments selected for inspection that is submitted to ORA. To be efficient with its resources, ORA staff may shift the order of establishments to be inspected on CDER’s prioritized list based on geographic proximity to other planned inspection trips, according to FDA officials. Investigators from ORA and, as needed, ORA laboratory analysts with certain expertise are responsible for inspecting drug manufacturing establishments. FDA primarily relies on three groups of investigators to conduct foreign inspections: ORA investigators based in the United States, who primarily conduct domestic drug establishment inspections but may sometimes conduct foreign inspections. Members of ORA’s dedicated foreign drug cadre, a group of domestically based investigators, who exclusively conduct foreign inspections. Investigators assigned to and living in the countries where FDA has foreign offices, who include both staff based in the foreign offices full time and those on temporary duty assignment to the foreign offices. FDA began opening offices around the world in 2008 to obtain better information on the increasing number of products coming into the United States from overseas, to build relationships with foreign stakeholders, and to perform inspections. FDA full-time foreign office staff are posted overseas for 2-year assignments. FDA staff can also be assigned to the foreign offices on temporary duty assignments for up to 120 days. In fiscal year 2019, there were full-time and temporary duty drug investigators assigned to FDA foreign offices in China and India. FDA’s process for determining whether a foreign establishment complies with CGMPs involves both CDER and ORA. During an inspection, ORA investigators are responsible for identifying any significant objectionable conditions and practices and reporting these to the establishment’s management. Investigators suggest that the establishment respond to FDA in writing concerning all actions taken to address the issues identified during the inspection. Once ORA investigators complete an inspection, they are responsible for preparing an establishment inspection report to document their inspection findings. Inspection reports describe the manufacturing operations observed during the inspection and any conditions that may violate U.S. statutes and regulations. Based on their inspection findings, ORA investigators make an initial recommendation regarding whether regulatory actions are needed to address identified deficiencies using one of three classifications: no action indicated (NAI); voluntary action indicated (VAI); or official action indicated (OAI). Inspection reports and initial classification recommendations for regulatory action are to be reviewed within ORA. For inspections classified as OAI—where ORA identified serious deficiencies—such inspection reports and classification recommendations are to be reviewed within CDER. CDER is to review the ORA recommendations and determine whether regulatory action is necessary. CDER also is to review inspection reports and initial classification recommendations for all for-cause inspections, regardless of whether regulatory action is recommended by ORA. According to FDA policy, inspections classified as OAI may result in regulatory action, such as the issuance of a warning letter. FDA issues warning letters to those establishments manufacturing drugs for the U.S. market that are in violation of applicable U.S. laws and regulations and may be subject to enforcement action if the violations are not promptly and adequately corrected. In addition, warning letters may notify foreign establishments that FDA may refuse entry of their drugs at the border or recommend disapproval of any new drug applications listing the establishment until sufficient corrections are made. FDA may take other regulatory actions if it identifies serious deficiencies during the inspection of a foreign establishment. For example, FDA may issue an import alert, which instructs FDA staff that they may detain drugs manufactured by the violative establishment that have been offered for entry into the United States. In addition, FDA may conduct regulatory meetings with the violative establishment. Regulatory meetings may be held in a variety of situations, such as a follow-up to the issuance of a warning letter to emphasize the significance of the deficiencies or to communicate documented deficiencies that do not warrant the issuance of a warning letter. In December 2019, we found that from fiscal year 2012 through fiscal year 2016, the number of FDA foreign drug manufacturing establishment inspections increased but then began to decline after fiscal year 2016. In fiscal year 2015, the total number of foreign inspections surpassed the number of domestic inspections for the first time. However, from fiscal year 2016 through 2018, both foreign and domestic inspections decreased—by about 10 percent and 13 percent, respectively. FDA officials attributed this decrease to vacancies in the number of investigators available to conduct inspections (which we discuss later in this testimony statement) and to inaccurate data used to select establishments for inspection in fiscal years 2017 and 2018. Despite steps taken to improve the accuracy and completeness of FDA data on foreign establishments, in December 2019, we found that the data challenges we identified in our 2008 report continue to make it difficult for FDA to accurately identify establishments subject to inspection. Specifically, since 2017, FDA had pursued an initiative to inspect approximately 1,000 foreign establishments that lacked an inspection history. As of November 2019, officials said all of these establishments had either been inspected or were determined not to be subject to inspection because it was determined they did not actually manufacture drugs for the U.S. market, or had not recently shipped drugs to the United States. However, officials told us that this effort contributed to the decline in the number of foreign inspections conducted because of how data inaccuracies affected the process for selecting establishments for inspection. Specifically, after selecting uninspected foreign establishments for inspection, FDA determined that a sizeable percentage of these establishments were not actually subject to inspection (e.g., about 40 percent of those assigned to the China Office in fiscal years 2017 and 2018). These foreign establishments were thus removed from the list for inspection for the given year. FDA officials told us that the next highest priority establishments identified through the risk- based model to replace those establishments were domestic establishments. As a result, the number of foreign establishments actually inspected decreased. As part of our ongoing work, we plan to examine the accuracy and completeness of information FDA maintains about foreign establishments and the application of its risk-based site selection process. We further found that FDA continued to conduct the largest number of foreign inspections in India and China, with inspections in these two countries representing about 40 percent of all foreign drug inspections from fiscal year 2016 through 2018. (See table 2.) In addition to India and China, the rest of the countries in which FDA most frequently conducted inspections has generally been the same since our 2008 report. Since we last reported on this issue, FDA announced in March 2020 that, due to COVID-19, it was postponing most inspections of foreign manufacturing establishments. Only inspections deemed mission-critical would still be considered on a case-by-case basis. According to the announcement, while the pandemic has added new complexities, FDA has other tools to ensure the safety of the U.S. drug supply. For example, FDA announced that it was evaluating additional ways to conduct its inspectional work that would not jeopardize public safety and would protect both the establishments and the FDA staff. Such ways, according to FDA, could include reviewing the compliance histories of establishments, using information shared by foreign regulatory partners, and evaluating establishment records in lieu of an onsite inspection. In addition, the FDA Commissioner’s May 11, 2020 press statement stated that while FDA’s regulatory oversight is vital to the long-term health of America, product safety and quality are ultimately the establishment’s responsibility. Most firms, according to FDA, strive to reliably provide quality products and maintain the integrity of the supply chain. However, the lack of foreign inspections removes a critical source of information about the quality of drugs manufactured for the U.S. market. It is not clear when FDA will resume regular inspections. The agency originally announced the postponement would last through April 2020. However, on May 11, 2020, it stated that the postponement would continue. According to FDA, the agency continues to closely monitor the global situation. FDA stated that it remains in contact with its foreign regulatory counterparts and would work with the Centers for Disease Control and Prevention to develop a process that would govern how and where to return to on-site facility inspections as conditions improve. In December 2019, we found that each year from fiscal year 2012 through 2018 at least 50 percent of FDA’s foreign inspections were surveillance inspections. In contrast to preapproval inspections, surveillance inspections are used to ensure drugs already on the market are manufactured in compliance with FDA regulations. In recent years, the proportion of foreign surveillance inspections has increased. As figure 2 shows, in fiscal year 2012, 56 percent of foreign inspections were surveillance-only inspections; in contrast, from fiscal year 2016 through 2018, about 70 percent of foreign inspections were surveillance-only, which was comparable to the percentage for domestic inspections during that period. This is a significant increase from the 13 percent of foreign inspections that were surveillance-only when we made our 2008 recommendation that FDA inspect foreign establishments at a comparable frequency to their domestic counterparts (85 percent of which were surveillance-only at that time). In our December 2019 testimony, we also reported that FDA implemented changes to its foreign drug inspection program since our 2008 report that may have contributed to the increase in surveillance inspections. Prior to 2012, FDA was required to inspect domestic establishments that manufacture drugs marketed in the United States every 2 years, but there was no similar requirement for foreign establishments. As a result, and as we reported in 2008, foreign inspections were often preapproval inspections driven by pending applications for new drugs. FDA thus conducted relatively few surveillance-only inspections to monitor the ongoing compliance of establishments manufacturing drugs that were already on the market, with just 13 percent of foreign inspections conducted for surveillance purposes at the time of our 2008 report. However, in 2012, the Food and Drug Administration Safety and Innovation Act eliminated the 2-year requirement for domestic inspections, directing FDA to inspect both domestic and foreign establishments on a risk-based schedule determined by an establishment’s known safety risks, which was consistent with our 2008 recommendation. In December 2019, we found that from fiscal year 2012 through 2018, FDA identified deficiencies in approximately 64 percent of foreign drug manufacturing establishment inspections (3,742 of 5,844 inspections). This includes deficiencies necessitating a classification of VAI, or the more serious OAI, as described in the text box. Based on their inspection findings, FDA investigators make an initial recommendation regarding the classification of each inspection: No action indicated (NAI) means that insignificant or no deficiencies were identified during the inspection. Voluntary action indicated (VAI) means that deficiencies were identified during the inspection, but the agency is not prepared to take regulatory action, so any corrective actions are left to the establishment to take voluntarily. Official action indicated (OAI) means that serious deficiencies were found that warrant regulatory action. About 59 percent of domestic inspections (3,702 out of 6,291) identified deficiencies during this time period. (See fig. 3.) This proportion is similar to what we found when we last looked at this issue in 2008, when FDA identified deficiencies in about 62 percent of foreign inspections and 51 percent of domestic inspections from fiscal years 2002 through 2006. Our December 2019 analysis showed that serious deficiencies identified during foreign drug inspections classified as OAI—which represented 8 percent of inspections from fiscal year 2012 through 2018—include CGMP violations such as those related to production and process controls, equipment, records and reports, and buildings and facilities. For example: Failure to maintain the sanitation of the buildings used in the manufacturing processing, packing, or holding of a drug product (21 C.F.R. § 211.56(a) (2019)). At an establishment in India producing finished drug products, the investigator reported observing a live moth floating in raw material used in the drug production, and that the facility staff continued to manufacture the drug products using the raw material contaminated by the moth, despite the investigator pointing out its presence. Failure to perform operations relating to the manufacture, processing, and packing of penicillin in facilities separate from those used for other drug products (21 C.F.R. § 211.42 (d) (2019)). At an establishment in Turkey that manufactured penicillin and other drugs, the investigator reported that the manufacturer had detected penicillin outside the penicillin manufacturing area of the establishment multiple times. According to FDA, penicillin contamination of other drugs presents great risk to patient safety, including potential anaphylaxis (even at extremely low levels of exposure) and death. Some investigators who conduct foreign inspections expressed concern with instances in which ORA or CDER reviewers reclassified the investigator’s initial inspection classification recommendations of OAI to the less serious classification of VAI. In December 2019, we found that FDA’s foreign inspection workforce had staff vacancies, which FDA officials said contributed to the recent decline in inspections. As previously mentioned, FDA used multiple types of staff resources to conduct foreign drug inspections—including ORA investigators based in the United States, members of ORA’s dedicated foreign drug cadre based in the United States, and investigators assigned to FDA’s foreign offices. However, we found that each of these groups had current vacancies. At the time of our December testimony, FDA officials told us that the agency was trying to fill vacancies in each of these groups, but the lower staff numbers may limit FDA’s ability to conduct more foreign inspections. ORA investigators based in the United States. This group of investigators conducted the majority of foreign inspections; about 76 percent of foreign inspections in fiscal year 2018 involved an ORA investigator based in the United States who conducts both foreign and domestic inspections. FDA officials said that the more experienced investigators from this group are expected to conduct three to six foreign inspections per year, and investigators hired using generic drug user fees are expected to inspect nine to 12 foreign establishments per year. As of June 2019, there were 190 investigators eligible to conduct foreign drug inspections, but officials said that as of November 2019, the agency had an additional 58 vacancies in this group. At the time of our December 2019 testimony, officials said that the agency was in the process of hiring 26 ORA investigators based in the United States to fill these vacancies, with 32 vacancies remaining. FDA officials attributed the vacancies to multiple factors: investigator retirements, investigator movement to other parts of FDA, and the need to hire to additional investigator positions using generic drug user fees. Officials also said that a reorganization within ORA led to a reduced number of investigators who conduct drug manufacturing establishment inspections. While FDA had recently filled several of the vacancies, officials told us that new investigators are not typically used for foreign inspections until they have been with the agency for 2 to 3 years. ORA dedicated foreign drug cadre. About 15 percent of foreign inspections in fiscal year 2018 involved an investigator from ORA’s dedicated foreign drug cadre—a group of ORA investigators based in the United States who exclusively conduct foreign inspections. FDA officials said that members of the cadre are expected to conduct 16 to 18 foreign inspections each year. According to FDA, the cadre had 20 investigators in 2012 and 15 investigators in 2016. However, the cadre had only 12 investigators as of November 2019, out of 20 available slots. At the time of our December 2019 testimony, FDA officials told us that the agency was attempting to fill these positions from the current ORA investigator pool, but officials were not confident that all 20 slots would be filled. Investigators assigned to FDA’s foreign offices. Approximately 7 percent of foreign inspections in fiscal year 2018 involved investigators from FDA’s foreign offices. The investigators conducting these inspections were those based in the China and India foreign offices—the countries where most drug inspections occur—and also included those investigators on temporary duty assignment to these offices. According to FDA officials, these investigators are expected to conduct 15 foreign inspections each year. We have noted high vacancy rates for these foreign offices in past reports. While these vacancy rates have decreased over time, vacancies persist. As of November 2019, FDA’s China office had three of 10 drug investigator positions vacant (a 30 percent vacancy rate), while FDA’s India office had two of six drug investigator positions vacant (a 33 percent vacancy rate). In our December 2019 testimony, we reported that FDA had taken steps to address vacancies in the foreign offices but continued to face challenges. In our 2010 report, we recommended that FDA develop a strategic workforce plan to help recruit and retain foreign office staff. FDA agreed with our recommendation and released such a plan in March 2016, but the long-standing vacancies in the foreign offices raise questions about its implementation. FDA officials told us that one challenge in recruiting investigators for the foreign offices is that well- qualified investigators for those positions need foreign inspection experience. For example, an official in FDA’s India office told us that foreign inspections can be challenging, and the India office does not have the resources to develop or train new investigators. Therefore, it is important to recruit investigators who have experience conducting foreign inspections, and such investigators are recruited from ORA. Thus, vacancies in the other two groups of investigators can influence the number of staff available to apply for positions in the foreign offices. Further, according to FDA officials, after employees have accepted an in- country position, the agency can experience significant delays before they are staffed in the office due to delays in processing assignments. For example, an official in FDA’s India office said that investigators need to complete a week-long security training program and must obtain the security clearance needed to work at the U.S. Embassy, which is where FDA’s foreign office is located. However, the official told us that there are limited availabilities for that training, and background checks for security clearances can take time. According to this official, FDA investigators did not usually receive first priority for the training. FDA estimated that it can take as little as 1 month to over 2 years for an investigator to clear background and medical checks and arrive at a foreign office. For example, an investigator in FDA’s China office told us that as a result of these requirements and other issues, it took nearly 2 years for the investigator to arrive at the office after FDA had accepted the investigator’s application. According to FDA’s own strategic workforce plan for the foreign offices, these types of delays have resulted in staff changing their decision after accepting a position in the foreign offices. In December 2019, we found that FDA continues to face unique challenges when inspecting foreign drug establishments that raise questions about whether these inspections are equivalent to domestic inspections. Specifically, based on our interviews with drug investigators in the dedicated foreign drug cadre and in FDA’s foreign offices in China and India, we identified four challenge areas related to conducting foreign inspections, which are described below. Of the four challenge areas identified, three areas—preannouncing inspections, language barriers, and lack of flexibility—were also raised in our 2008 report. Preannouncing Inspections. As we reported in 2008, the amount of notice FDA generally gives to foreign drug establishments in advance of an inspection is different than for domestic establishments. Drug establishment inspections performed in the United States are almost always unannounced, whereas foreign establishments generally receive advance notice of an FDA inspection. According to FDA officials, FDA is not required to preannounce foreign inspections. However, they said the agency generally does so to avoid wasting agency resources, obtain the establishment’s assistance to make travel arrangements, and ensure the safety of investigators when traveling in country. In our December 2019 testimony, we found that FDA does conduct some unannounced foreign inspections, particularly if the investigators conducting the inspection are based in FDA’s foreign offices. However, FDA officials told us that FDA does not have data on the frequency with which foreign drug inspections are unannounced, nor the extent to which the amount of notice provided to foreign establishments varies. According to FDA officials, this is because FDA does not have a data field in its database to systematically track this information. However, the officials estimated that the agency generally gives 12 weeks of notice to establishments that investigators are coming when investigators are traveling from the United States. While investigators in FDA’s China and India offices do conduct unannounced or short-notice inspections, these staff do not perform most of the inspections in these countries. (See table 3.) Our work indicated that preannouncing foreign inspections can create challenges and raises questions about the equivalence to domestic inspections. Of the 18 investigators we interviewed, 14 said that there are downsides to preannouncing foreign inspections, particularly that providing advance notice gives foreign establishments the opportunity to fix problems before the investigator arrives. For example, when an inspection is preannounced, it gives establishments time to clean up their facility and update or generate new operating procedures ahead of the inspection. However, establishments are expected to be in a constant state of compliance and always ready for an FDA inspection, and several investigators told us seeing the true day-to-day operating environment for an establishment is more likely during an unannounced inspection. Of the 18 investigators we interviewed for our December 2019 testimony, 12 said that unannounced inspections are generally preferable to preannounced inspections. One investigator told us that, although they believed the best way to ensure industry compliance to CGMPs was for establishments to not know when FDA is coming for an inspection, there was no data that would allow the agency to evaluate whether unannounced inspections were better than preannounced inspections. In addition, some investigators told us that it was still possible to identify serious deficiencies during preannounced inspections. For example, investigators could still identify issues by looking at the firm’s electronic records, including time-stamped data relating to the creation, modification, or deletion of a record. Three investigators also told us that in some cases there could be benefits to announcing inspections in advance. For example, for preapproval inspections, announcing the inspection in advance gives the establishment time to organize the documentation and staff needed to conduct the inspection. Language Barriers. Work for our December 2019 testimony indicated that language barriers—which we first reported as a challenge to conducting foreign inspections in our 2008 report—can add time to inspections and raise questions about the accuracy of information FDA investigators collect and thus about the equivalence to domestic inspections. FDA generally does not send translators on inspections in foreign countries. Rather, investigators rely on the drug establishment to provide translation services, which can be an English-speaking employee of the establishment being inspected, an external translator hired by the establishment, or an English-speaking consultant hired by the establishment. Of the 18 investigators that we interviewed, 14 said that language barriers can be a challenge to conducting foreign inspections and were especially challenging in parts of Asia, including China and Japan. Seven investigators told us this issue was less of a challenge for inspections conducted in other foreign countries, including India and countries in Europe, because workers at establishments in these countries were more likely to speak English, and documentation was also more likely to be in English. Investigators told us that compared to domestic inspections, it can be more challenging and take longer to complete typical inspection- related activities, such as reviewing documentation or interviewing employees, if the investigator needed to rely on translation. Fourteen of the 18 investigators we interviewed said that there can be concerns related to relying on establishment staff and independent translators. Specifically, 11 investigators told us there can be uncertainties regarding the accuracy of the information being translated, particularly when investigators rely on the translation provided by an employee of the establishment being inspected. For instance, one investigator said that there was more risk of conflict of interest if the establishment used its own employees to translate. Another investigator said that they went to a drug establishment in China that told FDA it had English-speaking employees to translate the inspection, but that was not the case, and the investigator had to use an application on their phone to translate the interviews. In addition, the firm representative providing the translation may be someone who does not have the technical language needed, which can make it harder to communicate with firm staff and facilitate the inspection. One investigator told us that the independent translators hired by firms were sometimes consultants and, in those instances, it can seem like the consultants are coaching the firm during the inspection. FDA officials told us that when they conduct unannounced for-cause inspections in China, investigators bring locally employed staff who work in FDA’s China office to act as translators. The investigators we interviewed said that in such instances, they valued knowing that the translation they were getting was accurate. However, FDA does not have the resources to provide locally employed staff on every inspection, according to an FDA official. Lack of Flexibility. Work for our December 2019 testimony indicated that, as we first reported in 2008, the overseas travel schedule can present unique challenges for FDA’s domestically based investigators— including both ORA investigators and members of the dedicated foreign dug cadre—who conduct the majority of foreign inspections. Eight of the 12 dedicated foreign drug cadre investigators that we interviewed for our December 2019 testimony told us that there is little flexibility to extend foreign inspections conducted by domestically based investigators, because the inspections they conduct on an overseas trip are scheduled back-to-back in 3-week trips that may involve three different countries. This raises questions about their equivalence to domestic inspections. For instance, extending one inspection would limit the amount of time the investigator has to complete their other scheduled inspections, some investigators told us. In addition, eight investigators told us that domestically based staff are generally unable to extend the total amount of time spent on an overseas trip—one investigator told us that an investigator would have to find something really bad to justify an extension. In contrast, FDA officials told us that inspections conducted by in-country investigators in China or India, and domestic inspections in the United States, are generally scheduled one at a time and can thus more easily be extended if the investigator needs additional time to pursue potential deficiencies. However, in-country investigators are not involved in the majority of inspections conducted in China or India. Three investigators from the dedicated foreign drug cadre told us that when they travel overseas, they adjust their inspection approach to help ensure they finish foreign inspections on time. For example, one investigator told us that an investigator may start the inspection in an area of the establishment that was noted as having issues during the last inspection. However, one investigator said that sometimes it is not possible to cover everything in depth during a foreign inspection. Another investigator told us that they focus on identifying the most serious issues during a foreign inspection, and that less serious issues can be identified in the establishment inspection report for reference in the next inspection. Five investigators also noted that they work long hours during their inspection to ensure they can complete the needed work. While FDA may assign more than one investigator to an inspection to complete needed work, one investigator said that FDA does not usually assign more than one person to an inspection because investigators are expected to have the experience to conduct inspections by themselves. FDA data show that from fiscal years 2012 through 2018, the majority of both foreign and domestic inspections were conducted by one person— 77 percent and 66 percent, respectively. Post-Inspection Classification Process. According to FDA officials, starting in fiscal year 2018, FDA implemented a new post-inspection classification process: when an ORA investigator recommends an OAI classification following an inspection, ORA compliance is required to send that inspection report to CDER for review within 45 calendar days from the inspection closeout. Among other things, the process was intended to help ensure FDA can communicate inspection results to domestic and foreign establishments within 90 days of the inspection closeout, as committed to under the Generic Drug User Fee Amendments of 2017(GDUFA II). FDA officials told us that the changes also required an additional ORA review for foreign inspection reports to align that process with the process for domestic inspection reports. Although the 45-day reporting time frame for potential OAI classifications is a requirement for both domestic and foreign inspections, adding the additional level of review within ORA effectively shortened the amount of time investigators have to document findings for foreign inspections. Our work indicated that the post-inspection reporting time frames can create challenges for domestic investigators who conduct foreign inspections and raise questions about the equivalence to domestic inspections. Eight of the 18 investigators we interviewed for our December 2019 testimony said shortening the time for completing reports and adding a level of review has made it more challenging to meet reporting requirements, especially if serious deficiencies are identified during the inspection. Investigators told us that for a potential OAI inspection, they now need to send the inspection report to their supervisor for endorsement within 10 days of the closeout of a foreign inspection, regardless of when the investigator’s next inspection is scheduled for, or whether the investigator has to travel from overseas back to the United States after the inspection. For example, if a domestic investigator finds serious deficiencies on the first inspection of an overseas trip—thus indicating an initial OAI classification—the investigator needs to write and send the related inspection report to the ORA supervisor for endorsement before returning home from the 3-week overseas trip to meet the required time frame. One investigator told us that, as a result of the time pressures, post-inspection reports may be less thorough, and that some inspection observations could be better supported if investigators had more time to write the reports. In conclusion, foreign manufacturing establishments continue to be a critical source of drugs for millions of Americans, and FDA inspections are a key tool to ensure the quality of these drugs. Over the years since we first examined this issue, FDA has made significant changes to adapt to the globalization of the pharmaceutical supply chain and has greatly increased the number of inspections it conducts of foreign establishments. However, we found in December 2019 that the agency faced many of the same challenges overseeing foreign establishments that we identified over the last two decades. These included inspector vacancies and unique challenges when inspecting foreign drug establishments that raised questions about the equivalence of those inspections to domestic inspections. Since then, the outbreak of COVID- 19 has added a layer of complexity. It also further highlights the global nature of our pharmaceutical supply chain. Chairman Grassley, Ranking Member Wyden, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Mary Denigan-Macauley, Director, Health Care at (202) 512-7114 or DeniganMacauleyM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are William Hadley (Assistant Director); Derry Henrick (Analyst-in- Charge); Katherine L. Amoroso; George Bogart; Zhi Boon; Rebecca Hendrickson; John Lalomio; Gail-Lynn Michel; Laurie Pachter; and Vikki Porter. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The outbreak of COVID-19 has called greater attention to the United States' reliance on foreign drug manufacturers and further highlighted the importance of ensuring a safe pharmaceutical supply chain. Much of the manufacturing of drugs for treating COVID-19 occurs overseas, which is also true of the majority of other drugs marketed in the United States. While the volume of drugs manufactured overseas for the U.S. market is not fully known, FDA reports that about 70 percent of establishments manufacturing active ingredients and more than 50 percent of establishments manufacturing finished drugs for the U.S. market were located overseas, as of August 2019. FDA is responsible for overseeing the safety and effectiveness of all drugs marketed in the United States, regardless of where they are produced, and conducts inspections of both foreign and domestic drug manufacturing establishments. GAO has had long-standing concerns about FDA's ability to oversee the increasingly global pharmaceutical supply chain, an issue highlighted in GAO's High Risk Series since 2009. In particular: GAO recommended in 2008 ( GAO-08-970 ) that FDA increase the number of inspections of foreign drug establishments. GAO found in 2010 ( GAO-10-961 ) that FDA continued to conduct relatively few foreign inspections than domestic inspections. GAO found in 2016 ( GAO-17-143 ) that FDA was conducting more of these foreign drug inspections, and GAO closed its 2008 recommendation to conduct more foreign inspections. However, GAO also reported that FDA may have never inspected many foreign establishments manufacturing drugs for the U.S. market. In addition, in the summer of 2018, FDA began announcing recalls of blood pressure medications manufactured overseas that were tainted with a potential carcinogen, raising further questions about FDA’s oversight of foreign-manufactured drugs. This statement is largely based on GAO’s December 2019 testimony ( GAO-20-262T ) and discusses 1. the number of foreign inspections FDA has conducted, 2. inspection staffing levels, and 3. challenges unique to foreign inspections. For that testimony, GAO examined FDA data from fiscal years 2012 through 2018 and interviewed investigators from FDA’s 2019 cadre of investigators (who are based in the United States but exclusively conduct foreign drug inspections) and from FDA’s foreign offices in China and India. In December 2019, GAO found that a growing number of foreign drug manufacturing inspections conducted by the Food and Drug Administration (FDA) were in China and India (43 percent in 2018), where most establishments that manufacture drugs for the United States were located. In fiscal year 2015, FDA, for the first time, conducted more foreign inspections than domestic inspections. However, from fiscal year 2016 through 2018, both foreign and domestic inspections decreased—by about 10 percent and 13 percent, respectively. FDA officials attributed the decline, in part, to vacancies among investigators available to conduct inspections. In March 2020, FDA announced that, due to Coronavirus Disease 2019 (COVID-19), it was postponing almost all inspections of foreign manufacturing establishments. While FDA has indicated it has other tools to ensure the safety of the U.S. drug supply, the lack of foreign inspections removes a critical source of information about the quality of drugs manufactured for the U.S. market. GAO also found that FDA had vacancies among each of the groups of investigators who conduct foreign inspections. FDA had 190 investigators in the United States who conduct the majority of foreign inspections, but an additional 58 positions were vacant. At the time of GAO's December 2019 testimony, FDA was in the process filling 26 of these vacancies, with 32 remaining. However, according to FDA officials, it could be 2 to 3 years before new staff are experienced enough to conduct foreign inspections. FDA also faced persistent vacancies among investigators in its foreign offices. GAO further found in December 2019 that FDA investigators identified persistent challenges conducting foreign inspections, raising questions about the equivalence of foreign to domestic inspections. Specifically, GAO found: While FDA inspections performed in the United States were almost always unannounced, FDA's practice of preannouncing foreign inspections up to 12 weeks in advance may have given manufacturers the opportunity to fix problems ahead of the inspection. Investigators from FDA's China and India offices had conducted some unannounced inspections, but these staff do not perform most of the inspections in these countries (27 percent and 10 percent, respectively). FDA was not generally providing translators on foreign inspections. Rather, FDA continued to rely on translators provided by the foreign establishments being inspected, which investigators said raised questions about the accuracy of information FDA investigators collected. For example, one investigator said there was more risk of conflict of interest if the establishment used its own employees to translate. In addition, the establishment representative providing the translation may be someone who does not have the technical language needed, which can make it harder to communicate with establishment staff and facilitate the inspection. The overseas travel schedule can present challenges for FDA's domestically based investigators, who conduct the majority of foreign inspections. Domestically based investigators told us there is little flexibility for them to extend foreign inspections during an overseas trip. The inspections they conduct on an overseas trip are scheduled back-to-back in 3-week trips and may involve three different countries. Therefore, extending one inspection would limit the amount of time the investigator has to complete their other scheduled inspections. FDA officials said that inspections conducted by investigators based in China or India (and domestic inspections in the United States) are generally scheduled one at a time and can thus more easily be extended if the investigator needs additional time to pursue potential deficiencies. However, these in-country investigators are not involved in the majority of FDA inspections conducted in China or India.", "document_type": "gao"}
{"report": "We previously found that there are more than 80 federal programs that provide aid to people with low incomes, which are administered by several federal agencies as well as state and local providers. We reported that the low-income programs were created at various times, to serve different populations, and in response to different policy issues. We also found that many of these programs provide assistance such as cash aid, food, shelter, and health care for those who have limited means or are disadvantaged in other ways, while other programs are designed to help low-income people move toward self-sufficiency through education, training, and employment services. In their current strategic plans, some agencies include goals related to supporting individuals or families to help them move towards self- sufficiency. For example, HHS has a strategic objective to “encourage self-sufficiency and personal responsibility, and eliminate barriers to economic opportunity.” Officials from HHS’s Administration for Children and Families (ACF) said they are currently updating ACF’s strategic plan and it will likely include a vision of ending multigenerational poverty through primary prevention by using a whole family—or two-generation— approach that proactively connects families to services before they are in crisis. Two-generation approaches are different from individual low-income programs because these approaches simultaneously address multiple areas, such as child and family economic supports, education, employment, health, well-being, and social capital, according to HHS. These approaches are based on findings that connect the well-being of parents to their children’s social, emotional, physical, and economic well- being. For example, research indicates that parents’ improved economic security is linked to improvements in children’s home environment, greater parental engagement in their children’s schooling, and stronger parenting skills, which may lead to improved child outcomes. Similarly, children’s well-being directly affects their parents’ ability to succeed in both school and the workplace. For example, if parents participate in a workforce training program, but cannot access safe and affordable child care, they may not be able to accept or keep the job for which they trained. Research developed by an ACF-sponsored project states that two-generation approaches are hypothesized to result in parents experiencing stronger labor force attachment or increased earnings, children improving their school readiness and academic achievement, and families increasing family functioning, community connectedness, goal-directed behavior, and executive functioning, among other potential outcomes. This approach is not a new idea. For example, Head Start programs, which started in 1965, provide early education services to low-income children while offering support to families, such as services that promote housing stability, continued education, and financial security. In addition, federal programs have supported past efforts to improve service coordination for low-income families, such as the use of one-stop centers that deliver workforce, education, and other support services at a single location. These previous efforts, however, often prioritized one generation over the other, according to researchers. Researchers also found that these previous efforts tended to not provide the intensity or duration of services needed to create change for low-income families. More recent two-generation approaches are intentionally meant to provide services for parents, children, and families in innovative ways by equally addressing the needs of children and parents using quality programs and interventions. For example, ACF is using a human- centered design approach to work across its programs to help families achieve economic independence. Human-centered design aims to create solutions from the point-of-view of families that are in need and the states that serve them and to design systems and service delivery to fit families instead of the other way around, according to ACF. These newer two- generation approaches, which aim to be higher quality and more intensive than previous efforts, are still being tested. ACF and others are currently evaluating the effectiveness of these two-generation approaches. Ten federal programs were most commonly cited by selected state and local entities as being used to serve whole families and reduce poverty (see table 1). These programs are administered by USDA, HHS, and DOL. Federal officials have reported that the 10 programs have characteristics—including the target populations, purposes, and services provided—that allow state and local entities to address multiple aspects of child and parent well-being, and these have implications for their two- generation approaches. (See appendix I for the target population, purpose, and services provided by the 10 federal programs.) Specifically: Target Populations. Consistent with two-generation approaches, many of the federal programs target low-income, needy, or at-risk families. However, we previously reported that eligibility requirements for some low-income programs vary significantly with regard to who may obtain benefits and services, how income is counted, and the maximum income applicants may have. As a result, state and local officials told us that some families who they would like to engage in two-generation approaches are not eligible for some of the federally- funded programs they use for these approaches. Purpose. The purposes of Temporary Assistance for Needy Families (TANF) and the Community Services Block Grant (CSBG) are broad and can be used to support families in multiple ways. For example, CSBG’s purposes are, among other things, to reduce poverty, revitalize low-income communities, and empower low-income individuals and families to become fully self-sufficient. In addition, according to HHS, these two broad programs allow state and local agencies to cover costs that other programs do not allow, such as salaries for staff to design the two-generation approach. Among the 23 state and local entities, officials from the three entities that reported using CSBG and seven of the 11 that reported using TANF said that they used the programs in combination with other federal programs that have more limited purposes. The other programs can be used to provide more specific supports, such as developing child care programs, increasing employment and earnings, alleviating hunger, or improving maternal and child health. Benefits or Services Provided. A range of benefits and services for both children and parents are available across the 10 federal programs, such as child care, food assistance, and job skills training. To create their two-generation approach, state and local entities can, to the extent permitted by law, combine services from multiple programs to provide a coordinated approach to addressing the needs of the entire family. In fact, according to HHS, combining services from multiple programs is common in two-generation approaches because single programs tend to cover only one type of service or may be for parents or children instead of both. In addition to the 10 most commonly cited federal programs, state and local entities reported using over 40 other federal programs to support their two-generation approaches (see appendix II). For example, one entity has a housing complex for single parents and their children. The families receive HUD Section 8 Project-Based Rental Assistance, and the entity also provides support services such as child care, parenting classes, and financial counseling. Two entities reported using Medicaid— one of the nation’s largest sources of funding for medical and other health-related services for low-income individuals. There may be additional federal programs beyond those cited that could be used for two-generation approaches that selected state and local entities did not report using. For example, HHS officials told us that the Social Services Block Grant could be used for two-generation approaches because it is one of the most flexible sources of social services funding. HUD officials said the Family Self-Sufficiency Program provides case management services and could be used for two-generation approaches. However, none of the 23 selected state and local entities reported using either program. We found that selected entities leveraged the 10 most commonly cited federal programs in different ways to meet the unique needs of their individual communities. Officials from some entities reported using only one of the 10 programs, while one entity reported using as many as seven of the programs. In addition, all of the 23 selected entities reported using other resources, including state, local, and/or philanthropic funds, to create their two-generation approaches (see fig. 1). Some state and local officials said these additional resources were needed to provide flexibility in meeting the needs of families. For example, officials from one agency told us they used additional resources to provide services to families with incomes that exceeded the eligibility limits for federal programs, noting that some with higher incomes were still in need of assistance. To create their unique two-generation poverty reduction approaches and address the needs of their communities, selected entities reported the significance of involving leadership, changing policies, expanding services, modifying service delivery practices, and/or serving specific populations. Specifically: Involving Leadership. In some states, the governor or state legislature encouraged state agencies to adopt two-generation approaches and/or staff were hired to lead the state’s efforts in implementing two-generation approaches, according to officials. For example, officials told us that Maryland’s governor and Connecticut’s state legislature created commissions with membership from state agencies, local organizations, and the public. The commissions made recommendations related to mitigating multigenerational poverty and developing local programs to pilot two-generation approaches. In addition, Colorado, Connecticut, Maryland, and Minnesota officials reported hiring a two-generation approach program manager within state government to coordinate the state’s efforts across agencies and programs to implement such approaches. Changing Policies. Officials from some states said they changed policies to better support families as part of their two-generation approaches, such as modifying eligibility requirements for certain services. For example, officials from a Colorado state agency said they changed policies across a range of programs to be more family friendly, such as providing transitional food assistance to families no longer eligible for TANF. Officials from a Georgia state agency said they made a number of policy changes, such as increasing the income eligibility threshold for child care subsidies to help families retain this care as their income increases. Expanding Services. Some officials also reported providing additional services to families that they had not provided in the past. For example, a Colorado state agency is piloting an expansion of its maternal and child home visiting program that adds employment, education, and child care to the supports the program already provides in order to improve family economic self-sufficiency. A Colorado local department now provides short-term housing with support services, in addition to its existing emergency shelters, for some families experiencing homelessness. Modifying Service Delivery Practices. Selected state and local entities sometimes changed the methods they used to deliver services to families, according to officials. For example, families served by a local agency in Maryland complete an intake form and a strengths and needs assessment. A trained coach then helps the family complete a pathway plan with family goals and action steps. A Minnesota local department is implementing new tools to assess the health, nutrition, education, and employment needs of families and connect them to supports across the department. A Connecticut state agency is employing a family centered coaching model in its Jobs First Employment Services Program. The coaching goes beyond traditional job search assistance by identifying the needs of children and parents and providing financial literacy training. Serving Specific Populations. Two-generation approaches by selected non-governmental organizations generally provided services to more specific populations, such as single parents or the families of children in certain schools. For example, a non-governmental organization in Minnesota serves single mothers and their young children by providing housing, on-site early childhood education, and weekly life skills training while mothers earn post-secondary credentials. Another non-governmental organization in Minnesota serves families of children in Minneapolis schools and assists families in accessing a variety of services, such as housing stabilization, health, career, and financial counseling. A non-governmental organization in Connecticut provides support services for the parents of children attending its preschool, including helping parents become certified child care workers and obtain full-time employment in a preschool program. Officials from 14 of 23 state and local entities reported challenges related to sharing data across low-income programs. We found that some state and local entities have data sharing practices in place, while others are in the midst of designing or creating related systems. Sharing data across systems and programs serving low-income families could enhance state and local two-generation approaches. For example, officials in one county said that if more robust data sharing occurred across agencies and systems, they would have access to information that would help them make decisions based on the needs of the families they serve. Additionally, officials from one non-governmental organization said they wanted to share data with other relevant organizations in order to be able to measure the impact of their two-generation approach. We previously found that states and localities used data sharing to improve case management by helping caseworkers obtain client information more quickly and make more informed decisions. Yet, state and local officials said that data sharing is difficult due to issues with linking data across low-income programs and concerns about how to protect participant privacy. Specifically, officials from five state and local entities reported issues related to linking data, including that a lack of common data fields across low-income programs made it difficult for the entities to share data. For example, officials at one state agency said they wanted to link mental health and substance abuse data systems. However, the officials have spent 6 years creating matching fields across these systems to allow the data to be shared, and the process is not yet complete. In addition, officials from seven entities said concerns about protecting participant privacy contributed to their data sharing challenges. For example, an official at a state agency noted it can be difficult to balance protecting an individual’s privacy while sharing enough data to be helpful to entities using two-generation approaches. Federal agencies have taken steps to assist state and local entities interested in data sharing by providing related resources and guidance (see fig. 2). Although this information may not be specific to two- generation approaches, federal officials reported that it could be useful to entities utilizing these approaches. Officials from 11 of 23 state and local entities reported a lack of information on two-generation approaches to be challenging. Specifically, they wanted more examples of successful two-generation approaches, opportunities to learn from peers, and information on federal funding sources that can be used to implement these approaches. For example, officials at one state agency said they had difficulties learning about federal funding sources that do not directly relate to the agency, but could be used to support two-generation approaches. In addition, officials at a non-governmental organization said they would benefit from federally sponsored peer learning so that they could gain knowledge from states and localities to build into their two-generation work. We found that HHS has developed information memorandums and policy statements on two-generation approaches that address the topics desired by state and local entities. HHS also has hosted webinars and communities of learning to assist state and local entities that were interested in adopting two-generation approaches. For some of these efforts, HHS partnered with other federal agencies. Specifically, within HHS/ACF, at least seven offices have developed information related to two-generation approaches (see fig. 3 for examples). This information discusses a range of topics, from funding flexibility to options for building service models, that could assist state and local agencies in creating two- generation approaches. According to federal internal control standards, managers should externally communicate the information needed to achieve their organizational goals. To help to ensure effective external communication, managers may want to consider whether the information is readily available to the intended audience when needed. While HHS has created resources relevant to two-generation approaches, it has not made this information readily available to all entities using two-generation approaches. HHS officials said that they disseminate guidance and other resources through existing program or office-specific mechanisms, such as separate email lists and websites for each office. Other federal agencies in our review also created information relevant to two- generation approaches that they distributed through existing program mechanisms. For example, DOL officials said information related to two- generation approaches is woven into technical assistance as relevant to various grant programs. Similarly, Education officials said they published guidance on its website and sent it to state educational agencies and other stakeholders through program specific email lists. Given these current approaches to distribution, state and local entities using two-generation approaches may not have access to or be aware of all relevant resources if these resources are only available to recipients of certain federal programs or entities in contact with certain federal offices. Without access to all pertinent information, state and local agencies may be unaware of the breadth of information available on two-generation approaches and related topics and may be unable to use it to address challenges they face while designing and implementing such approaches. HHS recently identified an interagency effort that officials said should address challenges faced by state and local entities, including their desire for information on two-generation approaches. The interagency Council on Economic Mobility was recently established and is led by HHS with participation from USDA, Education, HUD, DOL, Department of the Treasury, Social Security Administration, Office of Management and Budget, Council of Economic Advisers, and Domestic Policy Council. Its tentative mission is to “create an accountable and effective structure for federal interagency collaboration encouraging economic mobility and to use federal levers and tools to promote family-sustaining careers and economic mobility for low-income Americans.” Since the Council on Economic Mobility was only recently established, it is too early to determine whether it will make information readily available across federal programs that is relevant to state and local entities using two-generation approaches. Poverty negatively affects many aspects of a family’s life, including a child’s education and a parent’s ability to participate in the labor force. Previous attempts to provide better service coordination for low-income families lacked the intensity and quality of services needed for parents and children to create effective change, according to researchers. Some state and local entities are attempting to use two-generation approaches to help families move towards economic self-sufficiency and alleviate the impact of poverty on children, adults, and families through quality programs that address the needs of both generations. State and local officials that we interviewed have implemented a variety of two-generation approaches, but some said they do not have sufficient information on these approaches to most effectively serve families. HHS and other agencies have taken steps to address these challenges, including providing webinars, information memorandums, and other assistance. However, most of these resources are shared through individual program and office mechanisms, potentially making them difficult to access by those not directly connected to certain programs and offices. Moving forward, readily available information and assistance could more effectively help state and local officials learn how to better serve families and help break the cycle of multi-generational poverty. We are making the following recommendation to HHS: The Secretary of HHS, in consultation with the Council on Economic Mobility, should make information that would assist state and local entities in developing and implementing two-generation poverty reduction approaches readily available across federal programs and offices. (Recommendation 1) We provided a draft of this report to USDA, Education, HHS, HUD, and DOL for comment. We received written comments from HHS, which are reproduced in Appendix III. HHS concurred with our recommendation. The agency stated that through its coordination of the Council on Economic Mobility it will promote poverty reduction approaches that aim to provide more integrated, person-centered service delivery. In addition, HHS stated that it will work to identify opportunities for collaboration, promising practices, and successful models that promote economic mobility and will develop strategies for promoting them, such as through technical assistance. HHS also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretaries of the Departments of Agriculture, Education, Health and Human Services, Housing and Urban Development, and Labor, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Appendix I: Characteristics of Federal Programs Most Commonly Cited by Selected Entities Implementing Two-Generation Approaches To develop child care programs that best suit the needs of children and parents in each state, to empower working parents to make their own decisions on the child care that best suits their family’s needs, to provide consumer education to help parents make informed decisions, to provide child care to parents trying to achieve independence from public assistance, and to help states implement their child care regulatory standards. To enforce the support obligations owed by noncustodial parents for the support of their children through locating noncustodial parents, establishing paternity, obtaining child support, and assuring that assistance in obtaining support or order modifications will be available to all who request such assistance. Community Services Block Grant (CSBG) To reduce poverty, revitalize low-income communities, and empower low-income individuals and families in rural and urban areas to become fully self-sufficient. Noncustodial parent location, paternity establishment, establishment of child support orders, review and modification of child support orders, collection of child support payments, distribution of child support payments, and establishment and enforcement of medical support. A wide range of locally determined services and strategies may be supported to help low-income individuals and families become self- sufficient; address the needs of youth in low-income communities; and effectively use and coordinate with related programs. Comprehensive child development services, including educational, dental, medical, nutritional, and social services to children and their families. Services may be center based, home-based, family child care, or a combination, and may be full- or part-day or full- or part- year. To provide two-generation child development, family engagement, and family support services to pregnant women and young children from birth to age 5 and their families. The purpose of the program is to promote children’s school readiness by enhancing social and cognitive development and by providing educational, health, nutritional, social and other services for children and families. To improve maternal and child health, prevent child abuse and neglect, encourage positive parenting, and promote child development and school readiness. Regular home visits and support services from a nurse, social worker, or other professional. Families are provided services that are tailored to their specific needs, such as teaching parenting skills, promoting early learning in the home, or conducting screenings and providing referrals to address caregiver depression, substance abuse, and family violence. Program Supplemental Nutrition Assistance Program (SNAP) To alleviate hunger and malnutrition and permit low-income households to obtain a more nutritious diet by increasing their food purchasing power. Temporary Assistance for Needy Families (TANF) To accomplish one or more of the following: (1) provide assistance to needy families so that children may be cared for in their own homes or in the homes of relatives; (2) end the dependence of needy parents on government benefits by promoting job preparation, work, and marriage; (3) prevent and reduce the incidence of out-of-wedlock pregnancies; and (4) encourage the formation and maintenance of two-parent families. To provide a combination of education and training services that help job seekers obtain employment and advance in the labor market, to emphasize the alignment and integration of Workforce Innovation and Opportunity Act (WIOA) programs, to emphasize that employers are also customers of the workforce system, and to involve employers in helping the system provide the skilled workers they need. To provide a combination of education and training services that help job seekers obtain employment and advance in the labor market, to emphasize the alignment and integration of WIOA programs, to emphasize that employers are also customers of the workforce system, and to involve employers in helping the system provide the skilled workers they need. To provide a combination of education and training services that help job seekers obtain employment and advance in the labor market, to emphasize the alignment and integration of WIOA programs, to emphasize that employers are also customers of the workforce system, and to involve employers in helping the system provide the skilled workers they need. Benefit or service provided Benefits are provided through an electronic benefit transfer card to purchase food from authorized retailers. Allotments are determined on the basis of the thrifty food plan. TANF-funded services include: cash assistance (benefit levels and eligibility criteria defined by individual states); noncash services, including child care, work activities, work supports, and some child welfare services; and various other social services directed toward the statutory goals of family formation and reduced non-marital pregnancies. Employment services, including job searches and placement assistance, and referrals to employers. Training and services, such as occupational skills training, career counseling, and job searches. Educational supports, occupational skills training, counseling, and paid and unpaid work experiences. In addition to the 10 most commonly cited federal programs, state and local entities reported using over 40 other federal programs to support their two-generation approaches to poverty reduction. These programs are administered by the Corporation for National and Community Service and the Departments of Agriculture, Education, Health and Human Services, Housing and Urban Development, Justice, Labor, Transportation, and Treasury. See tables 3 through 11 below. In addition to the contact named above, the following individuals made key contributions to this report: Rachel Frisk and Danielle Giese (Assistant Directors), Andrea Dawson (Analyst-in-Charge), Gretel Clarke, Kelsey Kreider, and Kelly Snow. Also contributing to this report were Alex Galuten, Melissa Jaynes, Joy Solmonson, Almeta Spencer, Curtia Taylor, and Walter Vance.", "summary": "In 2018, nearly one in six children in the United States lived in families with incomes below the federal poverty thresholds, or about $26,000 annually for a family of four. Research has shown that poverty is associated with negative outcomes for the entire family. State and local entities are currently using two-generation, or whole family, approaches to reduce poverty and move families towards economic self-sufficiency. Senate Committee Report 115-150 included a provision for GAO to review two-generation approaches. GAO examined (1) the primary federal programs that support two-generation approaches and how these programs were leveraged by selected state and local entities, and (2) the challenges selected state and local entities faced implementing two-generation approaches and steps federal agencies have taken to address those challenges. GAO reviewed relevant federal, state, and local agency documentation; and interviewed officials from five federal agencies, and from 23 state and local entities in five states. States were selected to achieve variation in approaches used and percentage of families with children in poverty, among other factors. To reduce poverty through a two-generation approach, which involves working simultaneously with adults and children in a family, selected state and local entities most commonly reported leveraging resources from 10 federal programs. Among the 10 programs were the Department of Health and Human Services' (HHS) Temporary Assistance for Needy Families and Head Start; the Department of Agriculture's Supplemental Nutrition Assistance Program; and three Department of Labor Workforce Innovation and Opportunity Act core programs. Some of these entities also reported using state, local, and/or philanthropic resources to enhance their flexibility to provide services. State and local officials told GAO that difficulties with data sharing and limited information on successful two-generation approaches made it challenging to implement them, and some federal agencies have taken steps to address these challenges. State and local officials said that data sharing is difficult due to various concerns, including protecting participant privacy. Multiple federal agencies have resources on data sharing that may be useful to entities implementing two-generation approaches. State and local officials also said they wanted more examples of successful two-generation approaches and information on federal funding to implement them. To help address this challenge, various federal offices provided information and technical assistance, but the information is distributed via separate email lists and websites, thereby limiting cross-programmatic access and availability. HHS officials said the interagency Council on Economic Mobility—led by HHS—may help address information sharing. Given its recent establishment, related efforts are yet to be seen. Without readily available information, state and local entities may lack useful resources when designing programs to serve families. GAO recommends that HHS, in consultation with the Council on Economic Mobility, make information on two-generation approaches readily available. HHS agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "The purpose of STEP is to help small businesses develop their export capacity. Congress initially established STEP in the Small Business Jobs Act of 2010, and later reauthorized the program through fiscal year 2020 and renamed it the State Trade Expansion Program in TFTEA. According to SBA officials, the goals of the program include increasing the number of small businesses exporting, the number of small businesses exploring significant new trade opportunities, and the value of exports for small businesses already engaged in international trade. SBA implements STEP and has other key roles in efforts to promote U.S. exports, including providing training, counseling, and export financing for small businesses. Within SBA, OIT has responsibility for managing export promotion programs, including STEP. OGM has responsibility for administering grants across the agency according to SBA’s standard operating procedure for grants management. This responsibility includes oversight of financial and compliance-related aspects of issuing awards, and recording and tracking relevant information in SBA’s grants management system. According to OIT officials, each state that receives a STEP grant submits quarterly reports to OIT that provide information on the amount of the grant expended and progress made toward the performance targets. Program managers within OIT then review the quarterly reports and provide feedback to each state, including with respect to progress toward performance targets. OIT officials and states that we interviewed told us that if a state reports not meeting its performance targets in a particular quarter, OIT program managers then work with the state to establish an action plan to meet targets in future quarters. At the end of each grant performance period, OIT program managers work with OGM to finalize and close the state’s grant file. This procedure includes saving information on final performance data, the total amount expended, and the use rate, which is determined by dividing the amount expended by the amount awarded. Beginning in fiscal year 2011 and through fiscal year 2018, Congress has appropriated a total of $139.4 million for STEP, and SBA has awarded about $139.1 million, or almost 99.8 percent of the appropriated total. Table 1 shows the amounts appropriated and awarded, and the number of awards for each grant cycle from fiscal years 2011 to 2018. Every state government conducts some export promotion activities. State trade offices, which conduct these activities, can be housed in various state entities, including governors’ offices, state departments of commerce, universities, world trade centers, and state departments of economic development. As we have previously reported, state trade offices often offer export promotion services similar to those offered by certain federal agencies; in addition to SBA, these include the Department of Commerce and the Export-Import Bank. State trade offices often have both domestic and international staff; domestic staff are typically state employees. According to data from the 2017 SIDO survey, state trade offices had a median size of six employees, with a minimum of one and a maximum of 18. SBA awards STEP funds annually to state governments through a competitive application process. According to SBA, the annual STEP cycle begins with the funding opportunity announcement that SBA posts on www.grants.gov. This announcement, which usually occurs in the spring, indicates that the grant application is open and provides official information (e.g., objectives, deadlines, eligibility, and reporting requirements) about STEP. Once the announcement is posted, eligible states and territories may apply for a STEP grant during the application period. When a state trade office applies for a STEP grant, its application outlines any intended activities and establishes performance targets within each of the activities for the fiscal year or period of the grant. For example, the performance targets detailed in the application can include the state’s estimate of the number of businesses that will apply for and receive funding to attend various international trade show exhibitions. Currently, all 50 states, the District of Columbia, the Commonwealth of Puerto Rico, the U.S. Virgin Islands, Guam, the Commonwealth of the Northern Mariana Islands, and American Samoa are eligible to apply for STEP grants. Independent technical experts and OIT program managers score states’ applications, generally during late spring and summer. OIT then selects grant recipients and notifies states of their award status in September. If a state receives a STEP grant, its trade office provides the funds to local small businesses through an application process. The funds are intended to support the businesses’ export activities. Figure 1 shows the process for awarding and distributing STEP grant funds. Once small businesses receive the STEP funding, they can use the money for a variety of export-related purposes. These purposes are outlined in TFTEA. Figure 2 shows the allowable uses of STEP grant funds. In fiscal years 2015-2017, SBA awarded a total of about $54.1 million; 40 to 44 states received a STEP grant each year. Over this time period, the median grant amount was $373,000, with a minimum of about $115,000 and a maximum of $900,000. Figure 3 shows the total award amounts by state for fiscal years 2015-2017. The SIDO survey, conducted annually, also asks member states about the STEP grant and how important it is to their export promotion activities. In recent years, most states responding to this question have indicated that the grant plays a key role in supporting such activities, even though the grant does not typically constitute a majority of the budget. In the three surveys conducted between 2015 and 2017, more than 80 percent of responding states, on average, said that the grant was “extremely” or “very” important each year. In these same surveys, about 60 percent of responding states, on average, said that the grant constituted less than half of their budget. SBA’s STEP grants management processes do not provide reasonable assurance that STEP grant recipients meet some TFTEA requirements before the grant is closed out. OIT does have a process in place to comply with the “proportion of amounts” clause of STEP’s authorizing statute, which caps at 40 percent the amount of grant funds distributed to the 10 states with the largest numbers of eligible small businesses. However, we found that OIT does not have processes sufficient to ensure that states met TFTEA’s total and cash match requirements. TFTEA contains specific requirements for STEP that SBA is responsible for meeting. These requirements include the following: Proportional distribution requirement. SBA must distribute grant funds in a way that caps the amount of grant funds distributed to the 10 states with the largest numbers of eligible small businesses at 40 percent of the total amount awarded each year. This requirement ensures that states with fewer eligible small businesses receive funding, and is known as the “proportion of amounts” clause in the law. Total match requirement. States must provide a 25 percent or 35 percent non-federal total match to the federal grant amount. Cash match requirement. A state’s match cannot be less than 50 percent cash. OIT has established a process for ensuring compliance with the TFTEA requirement outlined in the “proportion of amounts” section of the law. As discussed above, TFTEA requires that OIT determine the 10 states with the highest percentage of eligible small businesses using the most recent data from the Department of Commerce. OIT officials told us they review data from Commerce’s Census Bureau that show the number of exporting small and medium-sized businesses in each state, and then use these data to determine the top 10 states. According to OIT officials, they use the most recent data available, with an approximately 2- to 3-year lag in the data. For example, to assess the top 10 states for the fiscal year 2017 cycle, OIT used data from 2014. Based on these data, the 10 states that SBA identified for the fiscal year 2017 cycle received about 32 percent of the total amount appropriated—which was below the 40 percent threshold and therefore in compliance with the proportional distribution requirement. OIT officials told us that they planned to use available 2016 Census data to determine the top 10 states for the fiscal year 2018 award cycle and then, after receiving applications, determine award amounts that would comply with this requirement. TFTEA requires that states receiving a STEP grant provide matching funds. The total match amount is typically 25 percent of the combined state-federal total amount; as noted above, in a limited number of cases, the state’s total match is 35 percent of this amount. Within either a 25 percent or 35 percent match amount, at least half of the total match must be provided in the form of cash. Matching share requirements are often intended to ensure local interest and involvement through financial participation, and may also serve to hold down federal costs. If SBA determines that a state is not providing sufficient matching funds, it can withhold future reimbursement for expenses incurred under the grant. Figure 4 illustrates the STEP funding proportions described above. OIT’s process for reviewing the quarterly reporting that states provide on STEP grants does not effectively document whether each state has met the total match requirement outlined in TFTEA. To determine whether each state is meeting the total match requirement, OIT program managers monitor state spending over the grant period through quarterly reporting that they require of the state grant recipients. At the end of each grant period, OIT officials told us they review the information collected through the quarterly reporting to determine whether the state met the total match requirement based on the amount of federal dollars expended. According to OIT data and officials, most states provide a greater match than is required; for example, according to OIT calculations, 75 percent of fiscal year 2015 states receiving the grant provided more matching funds than required. However, we identified four instances where, according to OIT’s documentation, one state reported an insufficient total match in fiscal year 2015 and three states reported an insufficient total match in fiscal year 2016. OIT’s documentation showed that these four states failed to meet the required total matching funds by about $76,000 combined over these 2 years of the program. SBA told us they nevertheless closed these grants. OIT officials provided several explanations for their actions. First, OIT officials told us that of these four states, two submitted additional information after the grant had closed, indicating that the states had met the matching requirement. OIT officials stated that they did not verify the accuracy of the total match information before grant closure because of OIT staff error. With respect to the other two states, OIT initially stated that it was working with OGM to verify that the total match requirement had not been met, and how best to recover the funds. Subsequently, OIT reported OGM’s determination that one state had in fact met the match requirement, but that the other had not. In the case of the state that did not meet the requirement, OGM determined that SBA had overpaid federal funds to that state by about $19,600. However, after contacting the state and looking into the matter further, OGM conducted a review of quarterly reporting documentation for this state, and determined that the state had in fact exceeded its required match by about $3,800. Though all four of the states initially identified were eventually determined to have met the total match requirement, SBA did not have an adequate process in place to ensure documentation of a full match before grant closeout. OIT officials stated in July 2018 that, as a result of our review, they planned to implement a new quarterly process to focus on match information specifically, which would ensure documentation of whether a state meets its total match requirement before the grant is closed at the end of each fiscal year program cycle. However, officials were unsure what this process would entail. Standards for Internal Control in the Federal Government states that management should design control activities. By designing and executing appropriate control activities, management helps fulfill its responsibilities and address identified risks in the internal control system. This responsibility applies to the entire process or life cycle of a transaction or event from its initiation and authorization through its final classification in summary records. In addition, management should design control activities so that all transactions are completely and accurately recorded. Such control activities can be preventive for agencies, meaning that the activities prevent an agency from failing to achieve an objective or address a risk. Without a process for effectively documenting that the total match requirement has been met and reviewing this documentation before grant closeout, SBA does not have reasonable assurance that states have complied with TFTEA’s total match requirement, and risks overpayment of federal funds. OIT’s processes do not provide reasonable assurance that states have complied with the TFTEA cash match requirement. As previously noted, TFTEA requires that states provide at least half of the total match requirement in the form of cash. TFTEA allows for the remaining half to be any mixture of cash, in-kind contributions, and indirect costs. OIT collects information about the types of expended matching funds, including the proportion provided in cash; however, OIT does not have a process in place to use this information to address risks to the program. As part of their reporting, states submit on a quarterly basis a detailed expenditure worksheet that contains information on the types of expended matching funds, including cash and other types of allowed contributions. OIT documents show that while proposed cash match amounts are recorded, OIT does not track or analyze states’ expended cash matching funds during or at the close of the grant cycle. OIT officials told us that this information is included in the states’ quarterly detailed expenditure worksheets, and therefore can be reviewed for compliance on a case-by-case basis. However, OIT program officials told us that they do not regularly analyze this information to determine what proportion of the total match the cash portion constitutes. As such, SBA cannot consistently determine whether states are meeting the TFTEA cash match requirement, and risk closing out grants for which states have not met the cash match requirement. OIT does not have a process to monitor whether states are misusing federal funds to offset the cash match requirement. The Uniform Guidance defines matching funds as the portion of project costs not paid by federal funds. Matching funds must be accepted when they are not included as contributions for any other federal award, meaning that federal funds cannot generally be used to meet the state match requirement. The program’s authorizing legislation does not define “cash,” and neither does the Uniform Guidance. OIT considers the salaries of state trade office staff who work on administering the grant to be a form of cash and, according to OIT officials, most states use state staff salaries as their total match, including the required cash portion. OIT does not have a process for ensuring that states reporting staff salaries as their required cash match are not also using grant funds from STEP to pay for portions of these same salaries. In our discussions with officials from 12 states that received STEP grants in fiscal year 2015, 2 states reported using the grant to pay for portions of state staff salaries. Both of these states told us that they also reported staff salaries to OIT as their cash matching funds. Using part of the grant to pay for staff salaries in this way could have the effect of reducing the match below the thresholds mandated by TFTEA. When we asked OIT officials what process they had in place to determine whether states were using staff salaries paid for with STEP funds as part of their match amount, OIT officials told us that they were not aware that STEP grantees had engaged in this practice, and therefore did not monitor for it. In order to determine whether this was happening, officials stated that they would need to inspect each state’s grant files on a case-by-case basis. In previous years, OIT has hired a contractor to select samples of and examine individual state grant files, and this contractor worked with states as needed to improve reporting. OIT officials told us that the last grant year reviewed in this way was fiscal year 2015, and they expect to be able to conduct some examinations for the closed fiscal year 2016 grants. SBA’s grants management standard operating procedure states that the agency should monitor grantees for compliance with the terms and conditions of the awards, which includes compliance with applicable federal law. Further, according to Standards for Internal Control in the Federal Government, management should design and execute control activities, and use quality information to achieve the entity’s objectives. Management should process reliable data into quality information to make informed decisions and evaluate the entity’s performance in achieving key objectives and addressing risks. Without processes to review whether states are meeting the cash match requirement, OIT is not implementing its responsibilities under SBA’s standard operating procedure because it cannot consistently determine whether states are meeting this requirement. Without making such a determination, SBA does not have reasonable assurance that states are contributing to the program as required by STEP’s authorizing statute. According to agency officials, OIT made some changes to the program in response to feedback from states, including addressing some types of challenges that states say affect their ability to use all their grant funds. However, officials of some states said that continuing challenges with the program impeded their ability to use all awarded grant funds within the permitted time period. While the challenges they described cover a variety of topics, most relate to compressed program timelines, administrative burden, or poor communication with and within OIT. OIT officials told us that they made some changes to STEP for the fiscal year 2017 program cycle in response to informal feedback from states, including changes to address concerns about use of funds and the administrative burden of the application. Of the 12 states that we interviewed, officials from 11 agreed that SBA’s changes would improve the program. Changes included: Extending funds usage period to 2 years. OIT officials told us that, beginning with the fiscal year 2017 cycle, they converted the program from a 1-year award to a 2-year award. This change allows an additional 4 quarters to conduct program activities, which, in turn, may help enable states to use the full amount of their grant funding and achieve performance targets. Some state officials that we interviewed said that this change improves the program. Eliminating travel preauthorization requirement. OIT officials also told us that, during the fiscal year 2017 cycle, they eliminated a requirement that any travel funded by STEP grants be approved at least 30 days in advance of each trip. Instead, states now report all travel to OIT as part of STEP’s quarterly reporting. According to some officials from the states that we interviewed, this change reduced the administrative burden on state trade office staff and allows greater flexibility to use grant funds when opportunities that require travel arise with limited notice. Reducing page length of technical proposal. For the fiscal year 2017 cycle, OIT reduced the length of the application’s required technical proposal by nearly half, from 18 pages to 10 pages. According to some state officials that we interviewed, this change helped to streamline the program’s application paperwork. State officials that we interviewed described a variety of ongoing concerns with STEP, including some challenges that reduced their ability to use all of their grant funds. We developed a nongeneralizable sample of the 12 states that did not use 25 percent or more of their grant funds in fiscal year 2015, and interviewed officials from those states in order to gain insight into their experiences with the program, including the challenges that they faced using the full award amount. These 12 low- use states represent almost 70 percent of funds that remained unused during that cycle. As shown in figure 5, we grouped the most commonly reported challenges into the following categories: (1) timing of the application and award processes, (2) administrative burden, and (3) communication. OIT’s recent changes to STEP could help increase future use rates; however, the effect is yet unknown because the changes were introduced for the fiscal year 2017 cycle. Further, nearly all of the concerns expressed by the 12 low-use states relate to aspects of the program outside the changes made by OIT. Our analysis of data from the program’s fiscal year 2015 and 2016 cycles found nearly 20 percent of grant funds unused each year, despite OIT officials stating that they seek 100 percent use of grant funds, as described below: 2015. Across all 40 recipient states, combined grant use was 81 percent, leaving 19 percent, or nearly $3.4 million, unused. This included one state that left 77 percent, or over $432,000, of its funds unused that year. 2016. Across 41 of the 43 recipient states, combined grant use was 82 percent, leaving 18 percent, or nearly $3.2 million, unused. This included one state that left nearly 95 percent, or nearly $184,000, of its funds unused that year. In addition, although officials from several states told us that they have made changes to their STEP grant applications or activities to increase their ability to use grant funds, use problems have persisted. Our interviews, conducted in March and April 2018, indicated that states continue to face obstacles using the full award amount. Officials we interviewed from each of the 12 low-use states cited challenges related to the timeline of the application and award processes. These challenges were a variable application period, a short application window, a short application rewrite period, and award announcements occurring close to the start of the grant period. The exact timing of the STEP application period varies from year to year, which officials from some states that we spoke with cited as a challenge to applying for the grant. In fiscal years 2015, 2016, and 2017, SBA opened the STEP application process at different points between February and May. Officials from five low-use states reported that they had difficulty planning staff resources for completing the application because of the variable time frames. For example, one state official told us that not knowing when the application period would open was one of his office’s biggest planning challenges because spring is a busy time for trade activities and coincides with when his state hosts its largest annual trade event. At some point during this busy season, the application period typically opens. The state official said that, as a result of competing priorities for his office during this season, he might have only 2 weeks to complete the STEP application within the allotted time. He said that if the grant application periods were opened at the same time each year, he would be better able to plan for it. Officials from five low-use states told us that the window for completing the initial STEP application was insufficient because, for instance, of the amount of work the application requires and competing demands on their staff. For fiscal years 2015, 2016, and 2017, SBA announced application windows of between 30 and 42 days. For fiscal year 2016, SBA later extended the window to 50 days. Some state officials said that the variable and short application window creates challenges to writing quality applications. Without quality applications, it may be more difficult for states to develop good plans for using grant funds, which can facilitate the full use of funds. We discussed states’ concerns about when the grant applications opened, and how long they remained open, with SBA officials. These officials said that they were unable to open the application at the same time and for the same length of time each year because of factors beyond their control, such as the federal budgeting process and the involvement of other offices within SBA. In response to states’ concerns regarding the length of time that the STEP application is open, we observed that under OMB’s Uniform Guidance the federal awarding agency must generally make all funding opportunities available for application for at least 60 calendar days; however, the guidance does allow agency officials to make a determination to allow for as few as 30 days. Officials from three of the 12 low-use states told us that the window for rewriting applications is insufficient to adequately consider and implement the changes needed, given that states must rewrite their technical proposals, including updating all supporting financial information as well as proposed performance targets within that time. Once states’ applications have been scored and recipients selected, OIT may require certain states to rewrite their applications to request smaller amounts of federal funds. SBA officials told us that in the fiscal year 2015, 2016, and 2017 cycles, states were given 21 days to rewrite their proposals, but that the rewrite period has been as short as 48 hours. OIT officials told us that they reduce states’ grant requests each year because SBA receives applications for more grant funds than are available. Such rewrites require reducing or removing intended activities and establishing new performance targets within each of the remaining activities. State officials told us that the window for rewriting the grant impacts their ability to write their program proposals, which serve as the basis for states’ performance metrics and measurement of outcomes. SBA announces final STEP award amounts in September, just prior to the beginning of the fiscal year. Officials from nine of the 12 low-use states told us that because the award notifications occur so close to the beginning of the fiscal year, using funds during the first quarter is difficult. For instance, most of these officials said that they cannot plan activities until they know whether they will receive an award and, if so, the amount. Furthermore, officials from four states attributed their low grant use to this issue. For example, officials from one state told us that September award announcements do not allow them enough time to recruit companies to participate and use funds in the first quarter of the federal fiscal year, forcing them to compress their activities into the remaining quarters of the program cycle. One state official referred to the first quarter as a “lost” quarter. Officials from two states reported reducing or eliminating programmatic activities in the first quarter to avoid pressure caused by OIT’s award timeline. OIT officials told us that they notify states of their awards before the start of each fiscal year and in compliance with federal government policy. Figure 6 compares the timelines for the application process, notification of awards, recipients’ grant use period, and closeout activities for the fiscal year 2015, 2016, and 2017 program cycles, as well as associated challenges to grant use that states reported. Challenges related to administrative burden were cited by officials from all 12 of the states that we interviewed. These included challenges related to completing the application, the process for moving funds from one use to another (known as “repurposing”), and the required reporting on the grant. Officials from eight low-use states told us that the STEP application requirements are unrealistic or burdensome because, for instance, the level of detail required about performance targets conflicts with the reality of promoting exports in a fluid international business environment. As discussed above, OIT requires states’ STEP applications to detail their projected use of grant funds. For example, when submitting their applications, states project which trade shows they will attend and the number of small businesses they will take to these trade shows. Further required details include projecting the costs and number of companies that will attend events in particular foreign locations, for example. In the past, the application required performance targets that were based on estimates of business interest and export opportunities up to 18 months in advance. Today, with the aforementioned transition to 2-year awards, STEP applications must project such activities and performance targets up to 30 months in advance of their execution. In the event of differences between planned and actual performance during the course of the program cycle, OIT requires states to explain the differences and their plans for aligning their future performance with the targets established in their applications. Officials from eight low-use states said they attributed their low use of grant funds to challenges with program rules or regulations. Difficulty in repurposing funds was the most common example that they cited. Some officials said that OIT’s difficult repurposing process limits states’ ability to move funds from one purpose to another when participating small businesses’ plans change or don’t align with the original program proposal, leaving funds unused. Several state officials described difficulty adapting to changing business plans or opportunities. For instance, one official said that when unanticipated opportunities appear, such as follow- up trade missions, OIT’s restrictive repurposing process limits states’ ability to move funds from one purpose to another. Officials from two states reported applying for smaller grant amounts than in previous years in order to have a more manageable amount to spend, thereby avoiding the need to repurpose funds. In order for states to use STEP funds in ways that differ from the plans in their approved program applications, SBA requires that states request agency permission to repurpose the funds. According to the Uniform Guidance, the federal awarding agency may restrict grant recipients’ repurposing of funds in excess of 10 percent of the total grant amount, and determine the level of detail required for requests to repurpose funds. OIT officials told us that in cases of repurposing more than 10 percent of total funds, the documentation required is determined on a case-by-case basis depending on the amount of funds involved and the degree of difference between the original approved use and the proposed new use. These OIT officials said that in some cases, states may only need to submit a written request via email; in other cases, states may be required to revise and resubmit their STEP applications. To compensate for the difficulty in repurposing funds, officials from two states told us that in subsequent years they had proposed more general programs that allowed for greater flexibility, such as by increasing the use of stipends, which provide small businesses with a predetermined amount of funds for a range of allowed activities. When a state applies for a STEP grant using this approach, the state is typically less specific in its performance targets, such as which trade shows will be attended and by how many small businesses. One state official said that this approach can increase use rates by providing states with more flexibility in distributing the funds. OIT officials agreed that where a state is less prescriptive in its application performance targets it is easier for the state to repurpose funds, such as when federal trade missions are cancelled or small businesses express interest in activities that were not originally proposed. However, the OIT officials noted that the lack of specificity may result in lower grant application scores. Officials from 11 of the 12 low-use states described challenges related to burdensome or changing reporting requirements, such as the number of forms required for quarterly reporting, or the level of detail required on certain forms, as well as challenges related to changes that SBA makes to reporting requirements during the grant period. Two state officials told us that, because they often have few people working on the grant, complying with these reporting requirements takes undue amounts of time, and thus have the effect of reducing use. For example, one state official described having to divide the cost of shared taxis and hotel rooms for a trade show, reporting the per-person cost for each company that was part of the state’s delegation. Officials from two states also pointed out that the reporting requirements for STEP were much more detailed and burdensome than grants they administered from other federal agencies, such as the Department of Commerce. In addition, officials from six states expressed concern that OIT occasionally makes changes to program requirements, such as reporting requirements, after the grants have been awarded and the grant cycle has begun. Some state officials said that these mid-year changes increase the administrative burden on their limited staff. During our discussion of states’ concerns with OIT, officials said that they are limited in their ability to address certain concerns described by the states. For example, OIT officials told us that the quantity and type of forms and level of detail required in states’ reporting are imposed by federal guidelines, determined by OGM, or based on agency leadership’s expectations. In addition, OIT officials said that sometimes requirements are changed outside of their office and are beyond their control. The officials stated that, for some changes, they are not in a position to wait until the following program cycle for implementation. They said that they do, however, postpone less urgent changes until the following program cycle rather than making them mid-cycle. Communication between OIT and states was a frequently cited area of concern in our interviews with officials from the 12 low-use states. Officials from nine states raised concerns related to the quality of communication with OIT. For instance, some state officials described issues such as sometimes waiting weeks or months for responses to emails that they had sent to OIT, resulting in administrative delays or preventing states from executing some activities as planned. In one example, state officials said that SBA recently waited months to notify the state of a problem opening its emailed quarterly report file, causing the state to wait months to receive its STEP grant funds. The officials said that such delays bring their program to a halt. Communication within OIT was also cited as an issue that hindered states’ fund use. Officials from 10 states noted that program rules, regulations, or requirements are inconsistently communicated. Further, several state officials described witnessing OIT program managers disagree regarding the interpretation of program rules during discussions with state representatives at a SIDO conference. When we discussed states’ concerns with OIT officials, they told us that they maintain open lines of communication with the states and that STEP program managers are required to retain logs of their communication with the states. The officials described making an effort to listen to states’ concerns, adding that they had modified the program in certain ways as a result of state input, as noted above. States can provide feedback about STEP to their respective OIT program managers, who then discuss states’ comments during weekly meetings with OIT management and other program managers. In addition, OIT officials said that they make themselves available for informal conversations with states at SIDO’s annual conference. According to OIT officials, communication with the states is usually channeled through each program manager, even when the content is pertinent to all grant recipients and could be communicated from one source. At the time of our review, OIT had not assessed and fully addressed the risk posed by some states’ low use of funds. OIT officials told us that while they informally collect feedback from states, there is no systematic process to collect states’ perspectives on challenges with the program, including obstacles to their ability to use funds. In addition to the goals of the program outlined earlier, OIT officials told us that one program performance metric is the use rate for STEP funds. Officials said that they seek 100 percent use for each state that receives an award, as well as for the program as a whole. Standards for Internal Control in the Federal Government specify that agency leadership should define program objectives clearly to enable the identification of risks and define risk tolerances in order to meet the goals of the program’s authorizing legislation. These standards for internal control include assessing the risks facing the agency as it seeks to achieve its objectives, with the assessment providing the basis for developing appropriate responses to risks from external and internal sources. Therefore, agency management should set its risk tolerance with regard to STEP at a level that appropriately mitigates risk while enabling the achievement of program objectives. Without assessing and addressing this risk to the program, OIT may continue to fall short of 100 percent grant fund use. Low grant fund use could negatively affect OIT’s ability to achieve program goals in supporting state export promotion activities. In addition, OIT has no systematic process to share best practices with sufficient detail that states struggling to use their STEP funds might apply those practices to improve their own programs. TFTEA requires SBA to publish an annual report regarding STEP, including the best practices of those states that achieve the highest returns on investment and significant progress in helping eligible small businesses. While 12 states did not use 25 percent or more of their grant funds in the fiscal year 2015 cycle, 19 states used all or almost all of their funds, as shown in appendix I. SBA publishes high-level information on what it deems to be notable state activities in its annual report to Congress. OIT officials told us that, when possible, they share best practices with states that may have difficulty accessing external markets. However, OIT officials told us that they do not formally facilitate the sharing of best practices among the states, saying that best practices for promoting exports in one state might not be transferable to another state because each state is unique in terms of the characteristics of its economy. According to the Uniform Guidance, grant recipients’ performance should be measured in a way that helps the federal awarding agency and other non-federal entities to improve program outcomes, share lessons learned, and spread the adoption of promising practices. Further, under federal standards for internal control, management should externally communicate the necessary quality information so that external parties, such as grant recipients, can help to achieve the entity’s objectives. We have also previously reported on the importance of collecting and sharing best practices, as well as the processes for doing so. By sharing detailed information with all participating states about the approaches that some grant recipients are using to successfully achieve STEP’s goals, SBA could encourage all grant recipients to improve the effectiveness of their state STEP programs, including increasing fund use rates in pursuit of OIT’s stated aim of 100 percent grant fund use. The STEP program has provided about $139 million of federal support to assist small businesses in finding export opportunities, and Congress has authorized STEP through 2020. SBA has a process in place to ensure compliance with the program’s legal requirement to cap the total grant amount to the 10 states with the largest number of eligible small businesses at 40 percent—thereby ensuring that states with fewer small businesses benefit from the program. However, SBA has not taken some necessary steps to manage the program’s total and cash matching requirements according to applicable law or federal internal control standards. SBA does not document that states are meeting the total match requirement, and has not developed a process to determine whether states are meeting the cash match requirement. As a result, SBA does not have reasonable assurance that the states are meeting these requirements. Matching requirements directly engage states, augment federal funds, and ensure further support to small businesses that export. As such, meeting the matching requirements is a key aspect of the program’s success. Although not every state has problems using the full federal award amount, about a quarter of the states do, which may hinder the program’s ability to fully achieve its goals of increasing the number of small businesses exporting, increasing the number of small businesses exploring significant new trade opportunities, and increasing the value of exports for small businesses already engaged in international trade. While SBA has taken some steps to improve the program based on feedback from states, it could do more in this regard, including finding ways to assess and address the specific concerns raised by states that have experienced difficulty using grant funds. SBA could also take further steps to collect and disseminate best practices among states to strengthen their ability to fully use grant funds. Higher grant fund use could enhance SBA’s ability to assist as many exporting small businesses as possible, leading to a fuller realization of the program’s goals. We are making the following four recommendations to SBA: The SBA Administrator should establish a process that ensures documentation of states’ compliance with the total match requirement before grant closeout. (Recommendation 1) The SBA Administrator should develop a process to determine states’ compliance with the cash match requirement. (Recommendation 2) The SBA Administrator should assess the risk to achieving program goals posed by some states’ low grant fund use rates. Assessing this risk could include examining the challenges that states reported related to the program’s application and award processes, administrative burden, and communication. (Recommendation 3) The SBA Administrator should enhance collection and sharing of best practices among states that receive STEP grant funds. (Recommendation 4) We provided a draft of this report to SBA for review and comment. In written comments (reproduced in appendix II), SBA generally agreed with our findings and concurred with our recommendations. SBA observed that the states we did not interview may have had different experiences with the program than the states in our sample. As we note in the report, our sample is nongeneralizable, and so the experiences these states reported to us may not be common to all states receiving the grant. As we stated in our report, we selected the 12 states that had the lowest grant use rates in fiscal year 2015 in order to understand the challenges they faced. We are sending copies of this report to the Administrator of SBA and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8612 or gianopoulosk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. The objectives of this report were to examine the extent to which (1) the Small Business Administration’s (SBA) State Trade Expansion Program (STEP) grants management process provides reasonable assurance of compliance with selected requirements of applicable law, and (2) SBA has taken steps to address challenges states report in using grant funds to achieve program goals. To address these objectives, we did the following: Legal and regulatory review. We reviewed the Small Business Jobs Act of 2010 and the Trade Facilitation and Trade Enforcement Act of 2015 (TFTEA), the statutes that established and reauthorized STEP, respectively. We focused on SBA’s compliance with the proportional distribution, total match, and cash match requirements because these requirements are consistent across both of the program’s laws, and to avoid duplication with ongoing Office of Inspector General (OIG) work on the program. In addition, we reviewed the Office of Management and Budget’s (OMB) federal grant guidance, Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards (Uniform Guidance), and Standards for Internal Control in the Federal Government to identify relevant guidance and practices in managing grants to non-federal entities and in designing and executing effective control processes. We also reviewed these documents and previous GAO work for information about the collecting and sharing of best practices. Program data review. We analyzed data on award amounts, matching funds required, and matching funds provided for the fiscal years 2015 and 2016 program cycles, as well as information about the sources of matching funds in selected cases for the fiscal year 2015 program cycle. We focused on these program cycles because these were the most recent years for which the most complete information was available. We also reviewed the data the Office of International Trade (OIT) used to comply with the legal requirement for proportional distribution for the fiscal year 2017 cycle. We assessed the reliability of these data by interviewing OIT officials about the sources of the data, how the data are stored and maintained, and by tracing data from relevant sample documents back to their sources. We found these data sufficiently reliable for the purpose of understanding OIT’s processes for complying with the proportional distribution and matching requirements. Program management review. We reviewed SBA’s standard operating procedure for managing grants and cooperative agreements in order to understand the agency’s requirements for this. We reviewed relevant documentation, including sample grant files, application forms, application scoring forms, grant reporting forms. To examine the relationship that states’ past use rates have on the award process, we reviewed OIT’s forms for scoring the grant applications in fiscal years 2017 and 2018. We interviewed officials from OIT and OGM to understand how SBA monitors STEP grants, including the steps they take to comply with the proportional distribution, total match, and cash match requirements. We examined SBA’s calculations of the total match amounts required for states, and we identified 4 instances in which SBA’s documentation showed an insufficient match in fiscal years 2015 and 2016. We spoke to OIT officials to gain insight into why the scoring procedures for the grants changed for the fiscal year 2018 program cycle. In addition, we reviewed TFTEA, previous GAO work, and the most recent STEP best practices reports for information relating to SBA’s communication to states about best practices in applying for and managing STEP grants. We interviewed officials from OIT to learn about steps they had taken to address concerns raised by states that participate in the program and to facilitate the sharing of best practices among states receiving the grant. Review of grant use rates. We analyzed SBA data on award amounts and amounts used for the fiscal years 2015 and 2016 cycles. We assessed these data by interviewing OIT officials about the sources of the data and how the data are stored and maintained. In discussing the fiscal year 2016 cycle with OIT officials, we learned that South Dakota and Texas had been granted extensions and therefore had not yet completed reporting on their use of these grants. As a result, we dropped these states from our calculation of the fiscal year 2016 cycle use rate. With the exclusions of South Dakota and Texas, we found these data sufficiently reliable for the purpose of calculating use rates for STEP for the fiscal years 2015 and 2016 cycles. In comparing the available grant use data from these years, we found the following: (1) some states that were in the low-use (less than 75 percent utilization) category in fiscal year 2015 were also in this category in fiscal year 2016, (2) some states that were in the low-use category in fiscal year 2015 were not in this category in fiscal year 2016, and (3) some states that were not in the low-use category or did not participate in STEP in fiscal year 2015 were in this category in fiscal year 2016. In our interviews, we asked officials from states in the low-use category in the fiscal year 2015 cycle about their experiences in subsequent years, including whether they intended to apply for the grant in the fiscal year 2018 cycle. Tables 2 and 3 below display data on fund use across participating states in fiscal years 2015-2016, including the percentage of federal award funds unused, amount of federal award funds unused, and percentage of total federal funds unused for each year. We used the data in table 2 to identify our population of low-use states. The 12 states that we interviewed used less than 75 percent of their award funds in the fiscal year 2015 cycle. They represent almost 70 percent of funds that remained unused during that cycle, representing a large proportion of the total unused funds that year. In table 2, the first three rows show data on the 12 states included in our sample. We used the data in table 3 to determine whether our population of low-use states achieved different use rates the following year. Interviews with low-use states. We conducted semi-structured interviews with officials from the 12 states that did not use at least 25 percent of their federal award in the fiscal year 2015 grant cycle. In these interviews, we discussed the states’ practices and reporting with respect to the total and cash match requirements, and inquired about the practice of using the federal award to offset state staff salaries while reporting these salaries as a cash match. These 12 interviews do not constitute a generalizable sample of STEP grantees, because we selected these states on the basis of their low use of grant funds in order to understand challenges faced by those states. As such, the practices reported by states we interviewed may not be common to all states receiving STEP grants. State challenges. In our semi-structured interviews with officials from the 12 states that did not use 25 percent or more of their award in fiscal year 2015, we gathered information about continuing challenges in fully using the grant funds. These states were: Maine We conducted these interviews in March and April 2018. However, our 12 interviews represent a nongeneralizable sample of the population of states that have received the 301 awards made through the STEP grant cycles since fiscal year 2011. As such, challenges reported by these states may not be common to all states receiving this grant. We asked officials from these 12 states questions about their experiences participating in the program, about challenges they had experienced, and about their views on how the program could be improved. To describe the themes that emerged from these 12 interviews with respect to challenges in fully using the funds, we identified categories based on an analysis of the responses that we received. Two GAO analysts independently coded the content of these interviews according to these categories. We conducted further analysis of the results of our coding to identify the three major groupings of challenges that we present in this report: (1) the timing of the application and award periods; (2) administrative burden; and (3) communication. The coders had an initial agreement rate of about 90 percent. Disagreements were resolved through discussion between the coders and, occasionally, through arbitration by a knowledgeable third party. SIDO survey data review. We analyzed 2015, 2016, and 2017 survey data provided by State International Development Organizations (SIDO), a national group that supports state trade offices. We reviewed data from their annual member survey conducted in those years. This survey asks SIDO member states about, among other things, top advocacy priorities, the number of staff in each state’s international trade office, and the location of these trade offices within the state government. The survey also asks states to describe the importance of STEP to each state, and provide the estimated proportion of each state’s export promotion budget that the grant constitutes. To assess these data, we interviewed a SIDO official about the organization’s methods for developing the survey each year, as well as their processes for collecting and storing the data, and reviewed the response rates in each year. We reviewed the survey instrument and data, and conducted testing for missing data, obvious errors, and outliers, and determined that these data were sufficiently reliable for the descriptive purposes for which they are used in this report. However, we noted that the number of respondents by year varied. According to SIDO 36, 38, and 41 states fully or substantially completed the survey in 2015, 2016, and 2017, respectively. We are presenting the results as general proportions or rounded percentages. We did not independently audit the survey results. We conducted this performance audit from September 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Adam Cowles (Assistant Director), Cristina Ruggiero (Analyst in Charge), Jesse Elrod, and Peter Kramer made key contributions to this report. The team also benefitted from contributions made by Martin de Alteriis, Mark Dowling, John Hussey, Jeff Isaacs, Christopher Keblitis, and Kimberly McGatlin.", "summary": "Congress established STEP in 2010 to increase small business exports. Through STEP, SBA has awarded about $139 million in grants to state trade offices, which in turn facilitate small business export activities, including participation in trade missions and attendance at trade shows. Congress reauthorized STEP in 2016. GAO was asked to review SBA's management of the program. This report examines the extent to which (1) SBA's STEP grants management process provides reasonable assurance of compliance with selected requirements of applicable law, and (2) SBA has taken steps to address challenges states report in using grant funds to achieve program goals. GAO reviewed the program's authorizing legislation and federal and agency guidance on grants management, analyzed SBA program data, and interviewed SBA officials. GAO also conducted semi-structured interviews with a non-generalizable sample of 12 of the 40 states that received STEP grants in fiscal year 2015, the most recent year for which complete data were available. GAO selected these states on the basis of their low grant fund use rates. The Small Business Administration's (SBA) management of the State Trade Expansion Program (STEP) does not provide reasonable assurance of compliance with some legal requirements. Specifically, the Trade Facilitation and Trade Enforcement Act of 2015 (TFTEA) requirements for STEP include: Proportional distribution requirement. SBA's Office of International Trade (OIT) must distribute grant funds so that the total amount awarded to the 10 states with the highest percentage of eligible small businesses does not exceed 40 percent of the program's appropriation that year. Total match requirement. States must provide a 25 or 35 percent non-federal match to the federal grant amount. Cash match requirement. A state's match cannot be less than 50 percent cash. GAO found that, while OIT has a process to meet the distribution requirement, it does not have a process for documenting that states have met the total match requirement before grant closeout, and does not have a process to determine whether states are meeting the cash match requirement. Without such processes, SBA cannot be reasonably assured that states are contributing per the law's requirements. GAO found that, while OIT has made changes to STEP in response to states' feedback, officials from states with low grant use described ongoing challenges with the program that affect their ability to fully use funds. These challenges include compressed application and award timelines, administrative burden, and poor communication. SBA has not adequately assessed risks to the program, including the risk to achieving program goals posed by some states' low grant fund use rates. Without such an assessment, OIT's ability to support U.S. exporters may be diminished. Further, SBA has not effectively facilitated sharing best practices among states. By doing this, SBA could help states make full use of funds to achieve the program's goals. GAO is recommending that SBA develop processes to ensure compliance with legal grant matching fund requirements, take steps to assess risks to program goals from low grant fund use rates, and enhance the sharing of best practices among states receiving the grant. SBA concurred with all of the recommendations.", "document_type": "gao"}
{"report": "The decennial census produces data vital to the nation. The data are used to apportion the seats of the U.S. House of Representatives; realign the boundaries of the legislative districts of each state; allocate billions of dollars each year in federal financial assistance; and provide a social, demographic, and economic profile of the nation’s people to guide policy decisions at each level of government. Furthermore, businesses, nonprofit organizations, universities, and others regularly rely on census data to support their work. Given the importance of the decennial census to the nation, it is important for the Bureau to manage risks that could jeopardize a complete, accurate, and cost-effective enumeration. To assist federal government leaders in managing such complex and inherently risky missions across their organizations, in prior work we developed an ERM framework that, among other things, identifies essential elements for federal ERM and good practices that illustrate those essential elements. Notably, these elements and practices apply at all levels of an organization and across all functions—such as those related to managing risks to the 2020 Census. Furthermore, Office of Management and Budget (OMB) Circulars No. A-11 and A-123 require federal agencies to implement ERM to ensure their managers are effectively managing risks that could affect the achievement of agency strategic objectives. As discussed in our ERM Framework, ERM is a decision-making tool that allows leadership to view risks as an interrelated portfolio rather than addressing risks only within silos. Fundamental to ERM is the development of risk mitigation and contingency plans. Mitigation plans detail how an agency will reduce the likelihood of a risk event and its impacts, should it occur. Contingency plans identify how an agency will reduce or recover from the impact of a risk after it has been realized. Among other things, these plans provide the roadmap for implementing the agency’s selected risk response and the vehicle for monitoring, communicating, and reporting on the success of that response. In developing these plans, it is important that agencies keep in mind the interaction of risks and risk responses, as the response to one risk may affect the response to another or create a new risk entirely. We also developed a Fraud Risk Framework to provide a comprehensive set of leading practices that serves as a guide for agency managers developing and enhancing efforts to combat fraud in a strategic, risk- based manner. The framework is designed to focus on preventive activities, which generally offer the most cost-efficient use of resources since they enable managers to avoid a costly and inefficient pay-and- chase model of recovering funds from fraudulent transactions after payments have been made. Consistent with our ERM framework, the Bureau developed a decennial risk management plan which, among other things, requires that it identify risks to the 2020 Census at the portfolio and program levels. Portfolio risks are those that could jeopardize the success of the 2020 Census as a whole, and they typically span several years with many potential risk events over the period. Program risks are narrower—they could jeopardize the success of an individual program, including the 35 operations that support the 2020 Census as well as the 2018 End-to-End Test. As of December 2018, the Bureau had identified 360 active risks to the 2020 Census—meaning the risk event could still occur and adversely impact the census. Of these, 30 were at the portfolio level and 330 were at the program level. As shown in figure 1, the greatest number of active program risks was to the Systems Engineering and Integration operation which manages the Bureau’s delivery of an IT “System of Systems” to meet 2020 Census business and capability requirements. For example, the Bureau’s description of one of the risks to this operation indicated that if certain key system test plans and schedules are not clearly communicated among and collaborated on by relevant Bureau teams, then the 2020 Census systems are at risk of not meeting performance, cost, and schedule goals and objectives. The Bureau’s decennial risk management plan requires that it classify risks by priority level. These classifications are intended to highlight the most critical risks and identify where to allocate additional resources. Figure 2 shows how the Bureau had classified the 360 active risks as of December 2018. To determine risk priority, the Bureau’s decennial risk management plan requires that it assign each risk numerical ratings for likelihood of occurrence and potential impact. When multiplied, the result is a numerical priority rating, which the Bureau divides into three classifications for high priority, medium priority, and low priority (see figure 3). According to the Bureau’s decennial risk management plan, all portfolio- level risks must be mitigated to reduce the likelihood of the risk event and its impacts, should it occur. In contrast, when a program-level risk is identified, risk owners—the individuals assigned to manage each risk— are to select from the following risk responses. Mitigate. This may be an appropriate response where there are actions or techniques that will reduce the likelihood of the risk event and its impact, should it occur. Watch. This may be an appropriate response where a trigger event can be identified far enough in advance so that mitigation activities can be delayed until then. Accept. This may be an appropriate response where the probability and potential impact of the risk is so low that mitigation actions do not appear necessary or the impact can be absorbed if the risk occurs. As of December 2018, the Bureau planned to mitigate 67 percent of the active risks it had identified (see table 1). Notably, this signifies that the Bureau determined there were actions it could take or techniques it could employ to reduce the likelihood of the majority of risks to the enumeration or their impact, should they occur. The Bureau’s decennial risk management plan sets out the following requirements for developing mitigation and contingency plans: Mitigation plans are required for all active portfolio risks and for all active program risks with a mitigate risk response. Contingency plans are required for all active portfolio risks with a high- or medium-priority rating, and a moderate or higher likelihood of occurrence. Contingency plans are also required for active program risks with a high- or medium-priority rating, a moderate or higher likelihood of occurrence, and a risk response of mitigate or accept. Of the 360 active risks to the census as of December 2018, 242 (67 percent) met the Bureau’s criteria for requiring a mitigation plan (see table 2). According to the Bureau’s risk registers, 232 of these risks (96 percent) had a mitigation plan. In addition, 146 of the active risks (41 percent) met the Bureau’s criteria for requiring a contingency plan. According to the Bureau’s risk registers, 102 of these risks (70 percent) had a contingency plan. Our prior reporting similarly found that earlier in the decennial cycle, the Bureau did not have mitigation and contingency plans for all risks that required them. In November 2012, we found that the Bureau had mitigation and contingency plans for each of the portfolio risks it had identified at the time, but none for the program risks. We reported that such plans were needed to help the Bureau fully manage associated risks, and we recommended that the Bureau develop risk mitigation and contingency plans for all program risks. In April 2014, the Bureau provided us with program-level risk registers that contained both risk mitigation and contingency plans where appropriate, and we closed the recommendation as implemented. However, as of December 2018, the Bureau is missing required mitigation and contingency plan for both portfolio and program risks. Example of 2020 Census Risk Without Required Contingency Plan In July 2016, the Bureau added a risk titled, Major Disasters, to its portfolio risk register. The Bureau’s description of the risk stated that if a major disaster—such as an earthquake—occurs during final preparations for or implementation of the 2020 Census, then census operations may not be executed as planned, leading to increased costs, schedule delays, or lower quality data. Leading up to the 2010 Census, Hurricane Katrina devastated the coastal communities of Louisiana, Mississippi, and Alabama; a few weeks later, Hurricane Rita cut across Texas and Louisiana. Damage was widespread. Among other things, in the aftermath of Katrina, the Red Cross estimated that nearly 525,000 people were displaced and their homes were declared uninhabitable. If a major disaster, such as a hurricane, occurs leading up to or during the 2020 Census, having a contingency plan would help ensure that housing units and their residents are accurately counted, particularly when hundreds of thousands of people— temporarily or permanently—may migrate to other areas of the country. As of December 2018, however, the Bureau had neither a draft nor approved contingency plan for this risk, although it required one since first added to the risk register nearly 2.5 years earlier. According to the Bureau, though not documented in a contingency plan, it is taking actions to respond if this risk is realized. However, if such actions are reflected in disparate documents or no documents at all, then decision makers are left without a comprehensive picture of how the Bureau is managing this risk to the 2020 Census. Some of the risks that were missing required plans had been added to the risk registers in recent months, but others had been added more than 3 years earlier. Specifically, the 10 risks without mitigation plans were added from June to December 2018, and the 44 risks without contingency plans were added from June 2015 to December 2018. The one portfolio risk without a required mitigation plan was added in December 2018, and the five portfolio risks without required contingency plans were added in July 2015, July 2016, October 2017, August 2018, and December 2018, respectively. In some instances, a risk may not meet the Bureau’s criteria for requiring a mitigation or contingency plan when first added to the risk register. However, we found that all 10 risks without required mitigation plans and 37 of the 44 risks without required contingency plans met the Bureau’s criteria for requiring such plans within a month of being added to the register (of the 37 risks without a required contingency plan, five were at the portfolio level and 32 were at the program level). The Bureau’s decennial risk management plan states that mitigation and contingency plans should be developed as soon as possible after risks requiring such plans are added to the risk registers, but it does not include a clear time frame for doing so. According to the Bureau’s 2020 Census Portfolio Risk and Issue Process Manager—responsible for developing, maintaining, and administering the risk management process for both portfolio and program risks to the 2020 Census—no time frame is included because risk owners are aware of their responsibility and a specific time frame would not speed up the process given competing demands on their time. However, the official said the Bureau would consider adding a specific time frame when it updates the decennial risk management plan in 2019. Standards for Internal Control in the Federal Government (Standards for Internal Control) states that management should define objectives in specific terms—including the time frames for achievement—so that they are understood at all levels of the entity. In addition, OMB Circular No. A-123 states that effective risk management is systematic, structured, and timely. Without setting a clear time frame for developing mitigation and contingency plans, some risks may go without them for extended periods, potentially leaving the 2020 Census open to the impact of unmanaged risks. The Bureau’s decennial risk management plan requires that both portfolio and program risk registers include the word “draft” or “approved” alongside each contingency plan. As of December 2018, this status showed that 41 percent of contingency plans in the Bureau’s risk registers were still in draft form and had not been approved by management (29 percent at the portfolio level and 42 percent at the program level). Specifically, management had approved 60 of the 102 contingency plans (five at the portfolio level and 55 at the program level) but not the remaining 42 (two at the portfolio level and 40 at the program level). On the other hand, the Bureau’s decennial risk management plan includes no requirements for indicating the status of either portfolio or program risk mitigation plans in the risk registers. Our review of the risk registers found that some of the portfolio risk mitigation plans included the word “draft” alongside the plan, but none included any indication of whether the plan had been approved by management. In addition, none of the program risk mitigation plans indicated whether the plan was in draft or had been approved by management, but we found that at least some appeared to be in draft. For example, one program risk mitigation plan stated that the Risk Review Board had recommended contacting three individuals for next steps; however, the plan did not appear finalized because it did not discuss any next steps and it is not clear that further action had been taken. Although the Bureau had mitigation plans in place for 96 percent of risks that required them, without a clear indication of the status of these plans in the risk registers, we were unable to determine how many had been approved by management. According to Bureau officials, the risk registers are Bureau management’s primary source of information regarding risks to the census. Standards for Internal Control states that management should use quality information from reliable sources and clearly document internal controls to achieve the entity’s objectives and respond to risks. Including a clear indication of the status of both mitigation and contingency plans in the risk registers would help to support Bureau officials’ management of risks to the census; in addition, it would help to ensure that those plans are finalized and that the census is not left open to unmanaged risks. Of the 42 contingency plans awaiting approval, many had been added to the risk registers in recent months, but others had been added more than 4 years earlier. Specifically, the two portfolio risks were added in September 2014 and August 2017, and the 40 program risks were added from October 2015 to December 2018. Moreover, we found that both of the portfolio risks and 34 of the 40 program risks without finalized contingency plans met the Bureau’s criteria for requiring such a plan within a month of being added to the register. The Bureau’s decennial risk management plan requires risk owners to present mitigation and contingency plans to management for approval as soon as possible after risks requiring such plans are added to the risk registers. However, as with development of the mitigation and contingency plans, the Bureau’s decennial risk management plan does not include a clear time frame for doing so because, according to the Bureau’s 2020 Census Portfolio Risk and Issue Process Manager, a specific time frame would not speed up the process given competing demands on risk owners’ time. As previously noted, Standards for Internal Control states that management should define objectives in specific terms—including the time frames for achievement—so that they are understood at all levels of the entity. In addition, OMB Circular No. A- 123 states that effective risk management is systematic, structured, and timely. Without setting a clear time frame for approving draft mitigation and contingency plans, some risks may not be finalized. Mitigation and contingency plans assist agencies in managing and communicating to agency stakeholders the status of risks. We reviewed the mitigation and contingency plans for six portfolio-level risks to the 2020 Census which the Bureau identified as among the “major concerns that could affect the design or successful implementation of the 2020 Census” (see table 3). We found that the Bureau’s mitigation and contingency plans for these risks did not consistently include key information needed to manage them. These six risks, if not properly managed, could adversely affect the cost and quality of the 2020 Census. According to the Bureau’s decennial risk management plan, for each portfolio-level risk the risk owner must develop mitigation and contingency plans using the Bureau’s mitigation and contingency plan templates (see appendixes III and IV for the Bureau’s templates). Those templates require, among other things, that the Bureau specify key activities for reducing the likelihood of the risk and its impacts. We found that the Bureau’s decennial risk management plan generally aligns with our ERM framework which is designed to help agencies, among other actions, identify, assess, monitor, and communicate risks. However, we also found some instances where the Bureau’s risk management plan did not require mitigation and contingency plans to include certain key attributes we identified, which we discuss below. See figure 4 for a list of key attributes that we used when reviewing mitigation and contingency plans. As indicated in the attribute descriptions, six of the seven attributes are applicable to mitigation plans. Clearly defined trigger events do not apply to mitigation plans because they signal when a risk has been realized and contingency activities must begin. Each of the seven attributes are applicable to contingency plans, although two attributes—activity start and completion dates and activity implementation status—are only applicable if the risk has been realized. As of December 2018, the results of our review of the Bureau’s mitigation and contingency plans for the six portfolio-level risks we selected were in most cases mixed: some mitigation and contingency plans aligned with a particular key attribute, while others did not (see table 4). For two attributes—activity start and completion dates and activity implementation status—we found the Bureau generally included the relevant information across the six selected mitigation plans, which should help ensure that activities are carried out in a timely manner and that agency officials and stakeholders are informed and assured that the risks are being effectively managed. On the other hand, none of the mitigation or contingency plans included a monitoring plan, which would help the Bureau to track whether plans are working as intended. We found that where attributes are required but not consistently implemented, the gap stems from the Bureau not always holding risk owners accountable for fulfilling all of their risk management responsibilities, such as keeping plans up to date. Bureau officials responsible for overseeing risk management for the 2020 Census stated that they encourage risk owners to complete all of their risk management responsibilities; however, risk owners do not always do so because they have competing demands on their time. Therefore, the officials said they are generally satisfied if the risk owners have completed at least some of their risk management responsibilities. However, they also agreed that risk management should be among the Bureau’s top priorities and that risk owners should fulfill all of their risk management responsibilities. Bureau officials also stated that the Bureau is managing risks to the census, even if not always reflected in the mitigation and contingency plans. We acknowledge that the Bureau is taking actions to manage risks to the 2020 Census beyond those reflected in its mitigation and contingency plans. However, if these actions are reflected in disparate documents or are not documented at all, then Bureau officials, program managers, and other decision makers are left without an integrated and comprehensive picture of how the Bureau is managing risks to the 2020 Census. Consequently, the Bureau’s risk management efforts are neither clear nor transparent, which may create challenges for decision makers’ ability to quickly and accurately identify essential information to set priorities, allocate resources, and restructure their efforts, as needed, to ensure an accurate and cost-effective enumeration. In addition, where mitigation and contingency plans are not clearly documented and only certain individuals know about them, there is potential for the loss of organizational knowledge, particularly as key personnel change roles or leave the agency altogether. Below we provide examples of gaps, by attribute, in the Bureau’s mitigation and contingency plans for the six risks we reviewed. For each portfolio and program risk mitigation and contingency plan, the Bureau’s decennial risk management plan requires risk owners to enter a description of the plan in the relevant risk register. However, our review of risk register entries for both mitigation and contingency plans across all active risks as of December 2018 found they were missing some key attributes, including monitoring plans, activity start and completion dates for most activities, the implementation status for some activities, individuals responsible for activity completion, and clearly defined trigger events. In some instances, the missing attributes were a result of the Bureau not requiring them in the risk register descriptions. In other instances, where the Bureau’s decennial risk management plan does require the attribute in the risk register descriptions, the gap was due to the Bureau not holding risk owners accountable for them. Some of the attributes missing from the registers were included in the separate mitigation and contingency plans. However, at the program level there are no separate mitigation plans, making the risk registers the only source of information for program-level mitigation activities. According to Bureau officials, after the 2020 Census they plan to require separate mitigation plans for program risks as well. At the same time, Bureau officials noted that they primarily rely on the risk registers to monitor risks to the census and usually do not refer to the separate mitigation and contingency plans. Standards for Internal Control states that management should use quality information from reliable sources that is appropriate, current, complete, accurate, accessible, and provided on a timely basis to achieve the entity’s objectives. Similarly, OMB Circular No. A-123 states that effective risk management is based on the best available information. Because the risk registers are Bureau management’s primary source of information regarding risks to the census—and currently their only source of information on program-level risk mitigation—including this information in the risk registers would help to support Bureau officials’ ability to manage risks to the 2020 Census. The Bureau has designed an approach for managing fraud risk for responses to the 2020 Census. We found that the approach generally aligns with leading practices in the commit, assess, and design and implement components of the Fraud Risk Framework. Specifically, the Bureau demonstrated commitment to combating fraud by creating a dedicated entity to lead antifraud efforts for the 2020 Census, conducted a fraud risk assessment, and developed a risk response plan, among other actions, consistent with leading practices from the selected components. However, the Bureau has not yet determined the program’s fraud risk tolerance or outlined plans for referring potential fraud to the Department of Commerce Office of Inspector General (OIG) to investigate. Bureau officials described plans and milestones to address these steps but not for updating the antifraud strategy to include them. Standards for Internal Control states that management should clearly document internal controls to achieve the entity’s objectives and respond to risks. In addition, management should use quality information that is current and complete. Updating the antifraud strategy to include the Bureau’s fraud risk tolerance and plan for OIG referral will help to ensure that the strategy is current, complete, and conforms to leading practices. Appendix IV presents additional details of our review of applicable leading practices. Managers of federal programs maintain the primary responsibility for enhancing program integrity and managing fraud risks. Those who are effective at managing their fraud risks collect and analyze data, identify fraud trends, and use the information to improve fraud risk management activities. Implementing effective fraud risk management processes is important to help ensure that federal programs fulfill their intended purpose, funds are spent effectively, and assets are safeguarded. The Fraud Risk Framework provides a comprehensive set of leading practices that serve as a guide for agency managers developing and enhancing efforts to combat fraud in a strategic, risk-based manner. The Fraud Risk Framework is also aligned with Principle 8 (“Assess Fraud Risk”) of Standards for Internal Control. It is designed to focus on preventive activities, which generally offer the most cost-efficient use of resources. The leading practices in the Fraud Risk Framework are organized into four components—commit, assess, design and implement, and evaluate and adapt—as depicted in figure 5. The commit component of the Fraud Risk Framework calls for an agency to commit to combating fraud by creating an organizational culture and structure conducive to fraud risk management. This component includes demonstrating a senior-level commitment to integrity and combating fraud, and establishing a dedicated entity to lead fraud risk management activities. The Bureau has taken steps that align with all applicable leading practices in this component, according to our review. Specifically, senior- level commitment to combating fraud helps create an organizational culture to combat fraud. The Bureau showed this commitment by creating an antifraud group, made up of multiple operational divisions within the Bureau—the Decennial Census Management Division, Decennial Information Technology Division, and Decennial Contracts Execution Office—and staff from the Bureau’s technical integration contractor. Staff from these divisions make up the Self-Response Quality Assurance (SRQA) group with the primary purpose of identifying and responding to potentially fraudulent responses received in the 2020 Census. SRQA members were assigned roles and responsibilities to combat fraud in the 2020 Census. According to the framework, antifraud entities should understand the program and its operations; have defined responsibilities and the necessary authority across the program; and have a direct reporting line to senior-level managers within the agency. We found that SRQA met these leading practices through our interviews with knowledgeable officials who discussed the Bureau’s strategy for managing fraud risk for the 2020 Census, and our review of documentation such as the fraud risk assessment, which listed roles and responsibilities for staff from the divisions in the antifraud group and the technical integration contractor. The group also directly reports to senior-level managers within the agency through weekly status reports that include milestones, activities, and challenges. According to the Fraud Risk Framework, the antifraud entity, among other things, serves as the repository of knowledge on fraud risks and controls; manages the fraud risk-assessment process; leads or assists with trainings and other fraud-awareness activities; and coordinates antifraud initiatives across the program. The Bureau staffed the antifraud entity with members knowledgeable of the program and tasked them with managing the fraud risk assessment process. Also, the members facilitated communication with management and among stakeholders on fraud- related issues through weekly status reports. According to SRQA officials, issues and concerns are escalated to senior-level managers on an as- needed basis so they can be coordinated across the program. The assess component of the Fraud Risk Framework calls for federal managers to plan regular fraud risk assessments and to assess risks to determine a fraud risk profile. This includes assessing the likelihood and effect of fraud risks and determining a risk tolerance. Risk tolerance is the acceptable level of variation in performance relative to the achievement of objectives. In the context of fraud risk management, if the objective is to mitigate fraud risks—in general, to have a low level of fraud—the risk tolerance reflects managers’ willingness to accept a higher level of fraud risks. Risk tolerance can be either qualitative or quantitative, but regardless of the approach, Standards for Internal Control states that managers should consider defining risk tolerances that are specific and measurable. The first part of the fraud risk assessment process includes leading practices on tailoring the assessment to the program; planning to conduct assessments both at regular intervals and when there are changes to the program or operating environment; identifying specific tools, methods, and sources for gathering information about fraud risks; and involving relevant stakeholders in the assessment process. The Bureau has met all the leading practices in the first part of the assess component, according to our review. Specifically, the Bureau tailored the fraud risk assessment to the 2020 Census as this is the first time an internet-response option will be available for a decennial census in the United States. To identify specific tools, methods, and sources for gathering information about fraud risks, the Bureau met with relevant stakeholders, along with subject- matter experts, and conducted focus groups to develop various fraud scenarios that became a key part of the assessment. The Bureau also involved relevant stakeholders in the assessment process by outlining their roles and responsibilities for the 2020 Census. For example, the Decennial Census Management Division serves as the fraud lead and oversees managing risks such as operational implementation, methodology, and workload demands with support from the other operational divisions in the antifraud group. According to the Fraud Risk Framework while the timing can vary, effective antifraud entities plan to conduct fraud risk assessments at regular intervals and when there are changes to the program or operating environment, as fraud risk assessments are iterative and not meant to be onetime exercises. The Bureau’s assessment takes this into account by acknowledging that risk assessment is an ongoing process. The assessment also states that the SRQA team will continue to evaluate and develop modeling techniques to train against existing fraud scenarios, and SRQA welcomes input from all stakeholders to ensure the Bureau identifies fraud risks, and works to implement controls and mitigation plans throughout the 2020 Census. The second part of the fraud risk assessment process includes identifying inherent fraud risks affecting the program; assessing the likelihood and effect of inherent fraud risks; determining a fraud risk tolerance; examining the suitability of existing fraud controls and prioritizing residual fraud risks; and documenting the program’s fraud risk profile (see figure 6). The Bureau met three out of these five leading practices, including identifying inherent fraud risk; assigning numeric rankings for likelihood and impact of various fraud scenarios; and documenting the 2020 Census fraud risk profile, which outlines the strengths and weaknesses of the program. We concluded that one leading practice, examining the suitability of existing fraud controls and prioritizing residual fraud risks, was not applicable since the fraud detection system is new to the 2020 Census and changes the way the Bureau will detect different fraud scenarios. As a result, all fraud risks for the 2020 Census are residual risks. In reviewing the remaining leading practice in the fraud assessment processes, we found that after identifying inherent fraud risk and assigning numeric rankings for likelihood and impact of various fraud scenarios, the Bureau did not take the next step to determine a fraud risk tolerance. Some of the steps the Bureau took to develop a risk response plan are similar to steps for developing a fraud risk tolerance. Specifically, the Bureau developed a process that classifies self-responses into risk categories of low, medium, or high. Bureau officials stated that they plan to use the classification to determine appropriate follow-up steps based on risk scores generated by its Fraud Detection Analytics Model that was develop by SRQA for the 2020 Census. However, the Bureau did not define thresholds for the low-, medium-, and high-risk categories. These thresholds, if defined, would meet the intent of a fraud risk tolerance by indicating the acceptable level of variation in self-responses. SRQA officials stated that they are developing these thresholds, and therefore its fraud risk tolerance, and plan to have them completed in August 2019. This includes reviewing available information collected through the 2018 End-to-End Test, running simulations, defining thresholds, and then evaluating the results to make adjustments. Responses will receive a score, but until the Bureau defines fraud risk tolerance thresholds for the low-, medium-, and high-risk categories, it cannot effectively implement its antifraud strategy to allocate responses for follow-up or inclusion. This may also affect the Bureau’s ability to evaluate and adapt its antifraud strategy if initial benchmarks are not in place to use for monitoring, with subsequent adjustments potentially requiring additional time and resources. While officials described steps and time frames to develop a fraud risk tolerance, they did not do so for updating the antifraud strategy to include the tolerance. Updating the antifraud strategy to include the Bureau’s fraud risk tolerance will help to ensure that the strategy is current, complete, and conforms to leading practices. The design and implement component of the Fraud Risk Framework calls for federal managers to design and implement a strategy with specific control activities to mitigate assessed fraud risks and collaborate to help ensure effective implementation. This includes determining risk responses and documenting an antifraud strategy; designing and implementing specific control activities; developing a plan outlining how the program will respond to identified instances of fraud; and establishing collaborative relationships and creating incentives to help ensure effective implementation of the antifraud strategy. For determining risk responses and documenting an antifraud strategy, the framework states that managers should (a) use the fraud risk profile to help decide how to allocate resources to respond to residual fraud risks; (b) develop, document, and communicate an antifraud strategy to employees and stakeholders that describes the program’s activities for preventing, detecting, and responding to fraud, as well as monitoring and evaluation; (c) establish roles and responsibilities of those involved in fraud risk management activities, such as the antifraud entity and external parties responsible for fraud controls, and communicate the role of the Office of Inspector General (OIG) to investigate potential fraud; (d) create timelines for implementing fraud risk management activities, as appropriate, including monitoring and evaluations; (e) demonstrate links to the highest internal and external residual fraud risks outlined in the fraud risk profile; and (f) link antifraud efforts to other risk management activities, if any. The Bureau developed and documented an antifraud strategy (the fraud risk assessment and the risk response plan) and communicated it to applicable employees. Bureau officials provided final versions of the antifraud strategy in October 2018 and stated that all stakeholders were provided with excerpts applicable to their area. The antifraud strategy outlines the beginning and end dates for fraud detection operations, and links to the highest residual fraud risks. The risk response includes links to other risk management activities such as a security layer that is designed, created, and maintained by the technical integration contractor security group in coordination with the Office of Information Security and Decennial Information Technology Division. According to the risk response plan, this group protects the fraud detection system and its associated systems from outside attacks such as hacks and distributed denial of service attacks. However, we found that the Bureau’s approach to managing fraud risk did not fully align with two leading practices in this component. First, until the Bureau defines its fraud risk tolerances, such as defining low-, medium-, or high-risk thresholds, it will not be able to effectively allocate resources to respond to residual fraud risks consistent with the Fraud Risk Framework’s leading practices. Second, the Bureau did not initially coordinate with the Department of Commerce (Commerce) OIG about its antifraud strategy, which is not consistent with the leading practices. Such lack of coordination could have precluded the OIG from determining if potentially fraudulent activities should be investigated. After discussing the results of our review with the Bureau, the Bureau contacted and met with the Commerce OIG in February 2019. Based on the Bureau’s notes from this meeting, the Bureau is on track to addressing the leading practice regarding coordination. The framework states that to design and implement specific control activities to prevent and detect fraud, managers should (a) focus on fraud prevention over detection; (b) consider the benefits and costs of control activities to address identified residual risks; and (c) design and implement the control activities such as data-analytics to prevent and detect fraud. The 2020 Census antifraud control activities focus on detecting potentially fraudulent responses. The Bureaus plans to use a combination of data analytics and follow up to review response data before they are added to the Bureau’s overall Census counts. The Bureau’s efforts for the 2020 Census also focus on minimizing costs. Specifically, if the Bureau’s fraud detection can minimize the amount of cases that require manual investigation or work by field operations staff to collect the information again, it can reduce the cost and workload to the Bureau. The framework states the antifraud strategy should also ensure that responses to identified instances of fraud are prompt and consistent. In addition, effective managers of fraud risks are to refer instances of potential fraud to the OIG or other appropriate parties, such as law- enforcement entities or the Department of Justice, for further investigation. The Bureau’s plan describes its process for scoring responses using its Fraud Detection Analytics Model and then sorting responses into a low-, medium-, or high-risk category. The plan also outlines risk responses that depend on the risk category. For example, medium-risk responses are reviewed internally and could be incorporated into the census count or sent for additional follow up. However, the Bureau’s antifraud strategy does not call for instances of potential fraud to be referred to the Commerce OIG. Specifically, the Bureau’s fraud risk assessment and risk response plan do not mention the Commerce OIG. Bureau officials stated that the Commerce OIG did not participate in the development of these documents. In February 2019, after we discussed the results of our review with the Bureau, the Bureau met with the Commerce OIG to discuss potential referrals. As a result, the Bureau agreed to develop and share with the Commerce OIG a plan that outlines a potential referral process by summer 2019. Managers who effectively manage fraud risks collaborate and communicate with stakeholders to share information on fraud schemes and the lessons learned from fraud control activities. The framework describes collaborative relationships as including other offices within the agency; federal, state, and local agencies; private-sector partners; law- enforcement entities; and entities responsible for control activities. In addition, managers should collaborate and communicate with the OIG to improve their understanding of fraud risks and align their efforts to address fraud. The Bureau collaborated internally with groups such as the Security Operations Center that maintain the security layer that protects Bureau systems and the nonresponse follow-up groups that visit households to collect information again. The Bureau also provided contractors with guidance by finalizing the antifraud strategy and incentives by entering into an agreement with the technical integrator contractor, which allows the Bureau to exercise an option to continue the contract for another year. However, the Bureau did not begin to collaborate and communicate with the Commerce OIG to improve its understanding of fraud risks and align efforts to address fraud until after we discussed the results of our review with the Bureau. Bureau officials viewed the primary purpose of the fraud detection system as a way to improve data reliability, according to interviews. As a result, in 2018, the Bureau changed the name of the operation from Fraud Detection to SRQA. According to Bureau officials, the change better reflects the operation’s focus on detecting potential falsification in decennial census response data and referring suspected responses to a field resolution operation to collect the data again. Bureau officials initially stated that SRQA would not conduct investigations that lead to the kind of law enforcement activities traditionally associated with fraud detection. As mentioned above, the Bureau met with the Commerce OIG in February 2019 to discuss the potential for referrals and, according to the Bureau, initiate a process for doing so. However, officials did not discuss steps and a time frame for updating the antifraud strategy to include this process. Doing so will help to ensure that the strategy is current, complete, and conforms to leading practices. Adequately addressing risks to the census is critical for ensuring a cost- effective and high-quality enumeration. The Bureau has taken important steps to address risks to the 2020 Census, but with less than a year until Census Day, the Bureau has not developed mitigation and contingency plans for all risks that require them. In addition, the Bureau does not have clear time frames for developing and obtaining management approval of mitigation and contingency plans, and some risks have gone without required plans for months and years. Moreover, the status of some plans is unclear and not all plans have received management approval. Some of the plans the Bureau has developed are missing key attributes we identified for helping to ensure the plans contain the information needed to manage risks. For example, none of the Bureau’s plans described how the Bureau will monitor the risk response, so the Bureau may not be able to track whether the plans are working as intended. These issues have arisen in some instances because the Bureau’s decennial risk management plan does not require mitigation and contingency plans to have each of the seven key attributes we identified; in other instances, the issues have arisen because Bureau officials do not always hold risk owners accountable for fulfilling all their risk management responsibilities. Consistently documenting risk management activities would support management’s ability to more quickly make informed decisions in response to risks confronting the 2020 Census. It would also help protect the Bureau from losing institutional knowledge in the event risk owners change roles or leave the agency. The Bureau’s fraud risk strategy generally aligned with our Fraud Risk Framework, including developing response plans and collaborating internally to address risks. However, the Bureau has not yet determined the program’s fraud risk tolerance or outlined a plan for referring potential fraud to the Commerce OIG to investigate, but plans to do so later this year. Setting a tolerance would help the Bureau monitor risks, and referring potential fraud to the Commerce OIG would allow it to determine if further investigation is appropriate. In addition to taking these actions, updating the antifraud strategy to include the Bureau’s fraud risk tolerance and plan for OIG referral will help to ensure that the strategy is current, complete, and conforms to leading practices. We are making the following seven recommendations to the Department of Commerce and the Census Bureau: The Secretary of Commerce should ensure that the Director of the Census Bureau develops and obtains management approval of mitigation and contingency plans for all risks that require them. (Recommendation 1) The Secretary of Commerce should ensure that the Director of the Census Bureau updates the Bureau’s decennial risk management plan to include clear time frames for developing and obtaining management approval of mitigation and contingency plans. (Recommendation 2) The Secretary of Commerce should ensure that the Director of the Census Bureau updates the Bureau’s decennial risk management plan to require that portfolio and program risk registers include a clear indication of the status of mitigation plans. (Recommendation 3) The Secretary of Commerce should ensure that the Director of the Census Bureau updates the Bureau’s decennial risk management plan to require that risk mitigation and contingency plans, including the risk register descriptions and separate plans, have the seven key attributes for helping to ensure they contain the information needed to manage risk. (Recommendation 4) The Secretary of Commerce should ensure that the Director of the Census Bureau holds risk owners accountable for carrying out their risk management responsibilities. (Recommendation 5) The Secretary of Commerce should ensure that the Director of the Census Bureau updates the Bureau’s antifraud strategy to include a fraud risk tolerance prior to beginning the 2020 Census and adjust as needed. (Recommendation 6) The Secretary of Commerce should ensure that the Director of the Census Bureau updates the Bureau’s antifraud strategy to include the Bureau’s plans for referring instances of potential fraud to the Department of Commerce Office of Inspector General for further investigation. (Recommendation 7) We provided a draft of this report to the Secretary of Commerce. In its written comments, reproduced in appendix V, the Department of Commerce agreed with our findings and recommendations and said it would develop an action plan to address them. The Census Bureau also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Commerce, the Director of the U.S. Census Bureau, and the appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report please contact Robert Goldenkoff at (202) 512-2757 or goldenkoffr@gao.gov or Rebecca Shea at (202) 512-6722 or shear@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. The objectives of this study were to examine (1) what risks to the 2020 Census the Census Bureau (Bureau) has identified, (2) the risks for which the Bureau has mitigation and contingency plans, (3) the extent to which the Bureau’s mitigation and contingency plans included information needed to manage risk, and (4) the extent to which the Bureau’s approach to managing fraud risks to the 2020 Census aligns with leading practices outlined in our Fraud Risk Framework. To answer the first three objectives, we reviewed Bureau documentation regarding its approach to managing risks facing the 2020 Census, including its decennial risk management plan, operational plan, governance management plan, Risk Review Board meeting minutes and agendas, and guidance and training documents. In addition, we interviewed Bureau officials responsible for overseeing risk management for the 2020 Census. To describe what risks to the 2020 Census the Bureau has identified and the risks for which the Bureau has mitigation and contingency plans, we also reviewed the Bureau’s portfolio- and program-level decennial risk registers. To assess the extent to which the Bureau’s mitigation and contingency plans included information needed to manage risk, we selected a nongeneralizable sample of six risks from the Bureau’s risk registers based on factors such as likelihood of occurrence and potential impact (see table 3). To select these risks, we began with the 12 risks identified by the Bureau in its 2020 Census Operational Plan as the “major concerns that could affect the design or successful implementation of the 2020 Census.” Next, we sorted the risks by numerical priority rating as of June 2018, a Bureau-assigned figure calculated by multiplying numerical scores for likelihood of occurrence and potential impact (see figure 3). We then selected the six risks with the highest priority ratings. For each selected risk, we reviewed relevant Bureau documentation—including risk mitigation and contingency plans—and we conducted semistructured interviews with the Bureau officials responsible for managing the risk. In addition, drawing principally from our Enterprise Risk Management (ERM) framework as well as secondary sources, we identified seven key attributes for risk mitigation and contingency plans to help ensure they contain the information needed to manage risks (see figure 4). Specifically, we reviewed our ERM framework and other relevant prior work on risk management, as well as commonly used risk management publications from sources including the Office of Management and Budget, the Project Management Institute, and the Chief Financial Officers Council and Performance Improvement Council. We analyzed these publications to identify portions relevant to risk mitigation and contingency planning. Next, we synthesized the information and derived attributes that appeared most important for effective risk mitigation and contingency plans. We assessed the attributes against the essential elements laid out in our ERM framework and found that each attribute aligned with one or more of the elements. Six of the seven attributes—all but clearly defined trigger events—are applicable to mitigation plans. Each of the seven attributes are applicable to contingency plans, although two attributes—activity start and completion dates and activity implementation status—are only applicable if the risk has been realized. We assessed the risk mitigation and contingency plans entered in the Bureau’s risk registers as of December 2018, as well as the separate mitigation and contingency plans for the six selected risks, against the seven key attributes. To evaluate the extent to which the Bureau’s approach to managing fraud risks to the 2020 Census aligns with leading practices outlined in our Fraud Risk Framework, we reviewed Bureau documentation related to the 2020 Census antifraud strategy. This strategy includes a fraud risk assessment that identifies and evaluates scenarios in which fraudulent activity could impact the 2020 Census results. It also includes a concept of operations that uses the fraud risk assessment to develop risk responses and its fraud detection systems. In addition, we interviewed Bureau officials responsible for antifraud efforts for the 2020 Census. We evaluated the information gathered based on the commit, assess, and design and implement components of our Fraud Risk Framework. Our assessment was limited to a review of the presence or absence of leading practices from the framework, not whether they were sufficient. We also did not review the leading practices for the “evaluate and adapt” component of the framework. This component focuses on evaluating outcomes using a risk-based approach and then adapting activities established in the other components to improve fraud risk management. Because the census is not scheduled to start until 2020, the Bureau will not be able to implement leading practices such as: monitoring and evaluating the effectiveness of preventive activities; measuring outcomes, in addition to outputs, of fraud risk management or using the results of monitoring and evaluations to improve the design and implementation of fraud risk management activities. We conducted this performance audit from May 2018 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: U.S. Census Bureau Operations Supporting the 2020 Census Purpose Define and implement program management policies, processes, and the control functions for planning and implementing the 2020 Census to ensure an efficient and well- managed program. Manage the delivery of an Information Technology (IT) “System of Systems” to meet 2020 Census business and capability requirements. Ensure all 2020 Census operations and systems adhere to laws, policies, and regulations that ensure appropriate systems and data security, and protect respondent and employee privacy and confidentiality. Identify and finalize content and design of questionnaires and other associated nonquestionnaire materials. Ensure consistency across data collection modes and operations. Provide optimal design and content of the questionnaires to encourage high response rates. Assess and support language needs of non-English speaking populations. Determine the number of non-English languages and level of support for the 2020 Census. Optimize the non-English content of questionnaires and associated nonquestionnaire materials across data collection modes and operations. Ensure cultural relevancy and meaningful translation of 2020 Census questionnaires and associated nonquestionnaire materials. Provide the geographic foundation to support 2020 Census data collection and tabulation activities within the Master Address File/Topologically Integrated Geographic Encoding and Referencing System. This system serves as the national repository for all spatial, geographic, and residential address data needed for census and survey data collection, data tabulation, data dissemination, geocoding services, and map production. Provide an opportunity for tribal, federal, state, and local governments to review and improve the address lists and maps used to conduct the 2020 Census as required by Public Law 103-430. Deliver a complete and accurate address list and spatial database for enumeration and determining the type and address characteristics for each living quarter. Print and distribute internet invitation letters, reminder cards or letters or both, questionnaire mailing packages, and materials for other special operations, as required. Other materials required to support field operations are handled in the Decennial Logistics Management operation. Capture and convert data from the 2020 Census paper questionnaires, including mail receipt, document preparation, scanning, optical character and mark recognition, data delivery, checkout, and form destruction. Communicate the importance of participating in the 2020 Census to the entire population of the 50 states, the District of Columbia, and Puerto Rico to support field recruitment efforts, engage and motivate people to self-respond (preferably via the internet), raise and keep awareness high throughout the entire 2020 Census to encourage response, and effectively support dissemination of Census data to stakeholders and the public. Internet Self-Response Maximize online response to the 2020 Census via contact strategies and improved access for respondents. Collect response data via the internet to reduce paper and nonresponse follow-up. Purpose Make it easy for people to respond anytime and anywhere to increase self-response rates by providing response options that do not require a unique Census ID. Maximize real-time matching of non-ID respondent addresses to the census living quarters address inventory, assigning nonmatching addresses to census blocks. Update the address and feature data and enumerate respondents in person. Designated to occur in areas where the initial visit requires enumerating while updating the address frame, particularly in remote geographic areas that have unique challenges associated with accessibility. Update the address and feature data and leave a choice questionnaire package at every housing unit identified to allow the household to self-respond. Designed to occur in areas where the majority of housing units do not have a city-style address to receive mail. Enumerate people living or staying in group quarters and provide an opportunity for people experiencing homelessness and receiving service at service-based locations, such as soup kitchens, to be counted in the census. Enumerate individuals in occupied units at transitory locations who do not have a usual home elsewhere, such as recreational vehicle parks, campgrounds, racetracks, circuses, carnivals, marinas, hotels, and motels. Provide questionnaire assistance for respondents by answering questions about specific items on the census form or other frequently asked questions about the 2020 Census, and provide an option for respondents to complete a census interview over the telephone. Also provide outbound calling support of nonresponse follow-up reinterview and coverage improvement. Determine housing unit status for nonresponding addresses that do not self-respond to the 2020 Census and enumerate households that are determined to have a housing unit status of occupied. Create and distribute the initial 2020 Census enumeration universe, assign the specific enumeration strategy for each living quarter based on case status and associated paradata, create and distribute workload files required for enumeration operations, track case enumeration status, run postdata collection processing actions in preparation for producing the final 2020 Census results, and check for fraudulent returns. Obtain counts by home state of U.S. military and federal civilian employees stationed or deployed overseas and their dependents living with them. Prepare and deliver the 2020 Census population counts to the President of the United States for congressional apportionment, tabulate and disseminate 2020 Census data products for use by the states for redistricting, and tabulate and disseminate 2020 Census data for use by the public. Provide to each state the legally required Public Law 94-171 redistricting data tabulations by the mandated deadline of 1 year from Census Day (April 1, 2021). Enhance the accuracy of the 2020 Census through remediating potential gaps in coverage by implementing an efficient and equitable process to identify and correct missing or geographically misallocated large group quarters and their population, and positioning remaining count issues for a smooth transition to the Count Question Resolution Operation. Provide a mechanism for governmental units to challenge their official 2020 Census results. Purpose Coordinate storage of the materials and data and provide 2020 Census records deemed permanent, including files containing individual responses, to the National Archives and Records Administration and to the National Processing Center to use as source materials to conduct the Age Search Service. Also store data to cover in-house needs. Island Areas Censuses Enumerate all residents of American Samoa, the Commonwealth of the Northern Mariana Islands, Guam, and the U.S. Virgin Islands; process and tabulate the collected data; and disseminate data products to the public. Develop the survey design and sample for the Post-Enumeration Survey of the 2020 Census and produce estimates of census coverage based on the Post-Enumeration Survey. Identify matches, nonmatches, and discrepancies between the 2020 Census and the Post-Enumeration Survey for both housing units and people in the same areas. Both computer and clerical components of matching are conducted. Collect person and housing unit information (independent from the 2020 Census operations) for the sample of housing units in the Post-Enumeration Survey to help understand census coverage and to detect erroneous enumerations. Document how well the 2020 Census was conducted, and analyze, interpret, and synthesize the effectiveness of census components and their impact on data quality or coverage or both. Measure the success of critical 2020 Census operations. Formulate and execute an experimentation program to support early planning and inform the transition and design of the 2030 Census and produce an independent assessment of population and housing unit coverage. Support 2020 Census field operations for decennial staff (i.e., headquarters, PDC, Regional Census Center, Area Census Office, Island Areas Censuses, remote workers, and listers/enumerators.) Provide the administrative infrastructure for data collection operations covering the 50 states, the District of Columbia, and Puerto Rico. Coordinate space acquisition and lease management for the regional census centers, area census offices, and the Puerto Rico area office; and provide logistics management support services (e.g., kit assembly, supplies to field staff). Provide the IT-related Infrastructure support to the 2020 Census, including enterprise systems and applications, 2020 Census-specific applications, Field IT infrastructure, mobile computing, and cloud computing. For the 2020 Census, the Census Bureau (Bureau) is trying to increase participation and reduce costs by offering more self-response options to households. This includes self-responses received via internet, phone, or mail. In 2018, the Self-Response Quality Assurance group finalized its antifraud strategy that includes a fraud risk assessment and risk response plan that focuses specifically on these responses. We developed a data collection instrument to structure our review of the antifraud strategy as it related to the commit, assess, and design and implement components of our Fraud Risk Framework. Our assessment was limited to a review of the presence or absence of leading practices from the framework, not whether they were sufficient. We also did not assess the Bureau’s approach against leading practices in the “evaluate and adapt” component of the framework because the Bureau will not be able to implement practices in this component until the 2020 Census begins. The following table summarizes our comparison of the Bureau’s antifraud strategy to leading practices in the fraud risk framework. In addition to the contacts named above, Lisa Pearson and Philip Reiff (Assistant Directors), Emmy Rhine Paule and Ariel Vega (Analysts-in- Charge), Carole Cimitile, Ann Czapiewski, Robert Gebhart, Maria McMullen, Ty Mitchell, James Murphy, Carl Ramirez, Kayla Robinson, Kate Sharkey, Andrea Starosciak, Michael Steinberg, Umesh Thakkar, and Jon Ticehurst made significant contributions to this report.", "summary": "With less than 1 year until Census Day, many risks remain. For example, the Bureau has had challenges developing critical information technology systems, and new innovations—such as the ability to respond via the internet—have raised questions about potential security and fraud risks. Fundamental to risk management is the development of risk mitigation and contingency plans to reduce the likelihood of risks and their impacts, should they occur. GAO was asked to review the Bureau's management of risks to the 2020 Census. This report examines (1) what risks the Bureau has identified, (2) the risks for which the Bureau has mitigation and contingency plans, (3) the extent to which the plans included information needed to manage risk, and (4) the extent to which the Bureau's fraud risk approach aligns with leading practices in GAO's Fraud Risk Framework. GAO interviewed officials, assessed selected mitigation and contingency plans against key attributes, and assessed the Bureau's approach to managing fraud risk against GAO's Fraud Risk Framework. As of December 2018, the Census Bureau (Bureau) had identified 360 active risks to the 2020 Census. Of these, 242 required a mitigation plan and 232 had one; 146 required a contingency plan and 102 had one (see table). Mitigation plans detail how an agency will reduce the likelihood of a risk event and its impacts, if it occurs. Contingency plans identify how an agency will reduce or recover from the impact of a risk after it has been realized. Bureau guidance states that these plans should be developed as soon as possible after a risk is added to the risk register, but it does not establish clear time frames for doing so. Consequently, some risks may go without required plans for extended periods. GAO reviewed the mitigation and contingency plans in detail for six risks which the Bureau identified as among the major concerns that could affect the 2020 Census. These included cybersecurity incidents and integration of the 52 systems and 35 operations supporting the census. GAO found that the plans did not consistently include key information needed to manage the risk. For example, three of the mitigation plans and five of the contingency plans did not include all key activities. Among these was the Bureau's cybersecurity mitigation plan. During an August 2018 public meeting, the Bureau's Chief Information Officer discussed key strategies for mitigating cybersecurity risks to the census—such as reliance on other federal agencies to help resolve threats—not all of which were included in the mitigation plan. GAO found that gaps stemmed from either requirements missing from the Bureau's decennial risk management plan, or that risk owners were not fulfilling all of their risk management responsibilities. Bureau officials said that risk owners are aware of these responsibilities but do not always fulfill them given competing demands. Bureau officials also said that they are managing risks to the census, even if not always reflected in their mitigation and contingency plans. However, if such actions are reflected in disparate documents or are not documented at all, then decision makers are left without an integrated and comprehensive picture of how the Bureau is managing risks to the census. The Bureau has designed an approach for managing fraud risk to the 2020 Census that generally aligns with leading practices in the commit, assess, and design and implement components of GAO's Fraud Risk Framework. However, the Bureau has not yet determined the program's fraud risk tolerance or outlined plans for referring potential fraud to the Department of Commerce Office of Inspector General (OIG) to investigate. Bureau officials described plans to take these actions later this year, but not for updating the antifraud strategy. Updating this strategy to include the Bureau's fraud risk tolerance and OIG referral plan will help ensure the strategy is current, complete, and conforms to leading practices. GAO is making seven recommendations, including that the Bureau set clear time frames for developing mitigation and contingency plans, require that mitigation and contingency plans include all key attributes, hold risk owners accountable for carrying out their risk management responsibilities, and update its antifraud strategy to include a fraud risk tolerance and OIG referral plan. The Department of Commerce agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "We reported in September 2017 that, while estimates of the economic effects of climate change are imprecise due to modeling and information limitations, they can convey useful insight into broad themes about potential damages in the United States. We reported that, according to the two national-scale studies available at the time that examined the economic effects of climate change across U.S. sectors, potential economic effects could be significant and these effects will likely increase over time for most of the sectors analyzed. For example, for 2020 through 2039, one of the studies estimated from $4 billion to $6 billion in annual coastal property damages from sea level rise and more frequent and intense storms. In addition, the national-scale studies we reviewed and several experts we interviewed for the September 2017 report suggested that potential economic effects could be unevenly distributed across sectors and regions. For example, one of the studies estimated that the Southeast, Midwest, and Great Plains regions will likely experience greater combined economic effects than other regions, largely because of coastal property damage in the Southeast and changes in crop yields in the Midwest and Great Plains (see figure 1). This is consistent with the findings of the Fourth National Climate Assessment. For example, according to that assessment, the continued increase in the frequency and extent of high-tide flooding due to sea level rise threatens America’s trillion-dollar coastal property market and public infrastructure sector. As we reported in September 2017, information on the potential economic effects of climate change could help federal decision makers better manage climate risks, according to leading practices for climate risk management, economic analysis we reviewed, and the views of several experts we interviewed. For example, such information could inform decision makers about significant potential damages in different U.S. sectors or regions. According to several experts and our prior work, this information could help federal decision makers identify significant climate priorities as an initial step toward managing climate risks. Such a first step is consistent with leading practices for climate risk management and federal standards for internal control. For example, leading practices from the National Academies call for climate change risk management efforts that focus on where immediate attention is needed. As noted in our September 2017 report, according to a 2010 National Academies report, other literature we reviewed, and several experts we interviewed, to make informed choices, decision makers need more comprehensive information on economic effects to better understand the potential costs of climate change to society and begin to develop an understanding of the benefits and costs of different options for managing climate risks. The federal government faces fiscal exposure from climate change risks in a number of areas, and this exposure will likely increase over time, as we concluded in September 2017. In the March 2019 update to our High-Risk List, we summarized our previous work that identified several of these areas across the federal government, including programs related to the following: Disaster aid. The rising number of natural disasters and increasing reliance on federal assistance are a key source of federal fiscal exposure, and this exposure will likely continue to rise. Since 2005, federal funding for disaster assistance is at least $450 billion. In September 2018, we reported that four hurricane and wildfire disasters in 2017 created an unprecedented demand for federal disaster resources and that hurricanes Harvey, Irma, and Maria ranked among the top five costliest hurricanes on record. Subsequently, the fall of 2018 brought additional catastrophic disasters such as Hurricanes Florence and Michael and devastating California wildfires, with further needs for federal disaster assistance. Disaster costs are projected to increase as certain extreme weather events become more frequent and intense due to climate change—as observed and projected by USGCRP. In July 2015, we reported that the federal government does not adequately plan for disaster resilience and that most federal funding for hazard mitigation is available after a disaster. In addition, our prior work found that the Federal Emergency Management Agency’s (FEMA) indicator for determining whether to recommend that a jurisdiction receive disaster assistance—which was set in 1986—is artificially low because it does not accurately reflect the ability of state and local governments to respond to disasters. Without an accurate assessment of a jurisdiction’s capability to respond to a disaster without federal assistance, we found that FEMA runs the risk of recommending that the President award federal assistance to jurisdictions that have the capability to respond and recover on their own. Federal insurance for property and crops. The National Flood Insurance Program (NFIP) and the Federal Crop Insurance Corporation are sources of federal fiscal exposure due, in part, to the vulnerability of the insured property and crops to climate change. These programs provide coverage where private markets for insurance do not exist, typically because the risk associated with the property or crops is too great to privately insure at a cost that buyers are willing to accept. From 2013 to 2017, losses paid under NFIP and the federal crop insurance program totaled $51.3 billion. Federal flood and crop insurance programs were not designed to generate sufficient funds to fully cover all losses and expenses, which means the programs need budget authority from Congress to operate. The NFIP, for example, was about $21 billion in debt to the Treasury as of April 2019. Further, the Congressional Budget Office estimated in May 2019 that federal crop insurance would cost the federal government an average of about $8 billion annually from 2019 through 2029. Operation and management of federal property and lands. The federal government owns and operates hundreds of thousands of facilities and manages millions of acres of land that could be affected by a changing climate and represent a significant federal fiscal exposure. For example, the Department of Defense (DOD) owns and operates domestic and overseas infrastructure with an estimated replacement value of about $1 trillion. In September 2018, Hurricane Florence damaged Camp Lejeune and other Marine Corps facilities in North Carolina, resulting in a preliminary Marine Corps repair estimate of $3.6 billion. One month later, Hurricane Michael devastated Tyndall Air Force Base in Florida, resulting in a preliminary Air Force repair estimate of $3 billion and upwards of 5 years to complete the work. In addition, we recently reported that the federal government manages about 650 million acres of land in the United States that could be vulnerable to climate change, including the possibility of more frequent and severe droughts and wildfires. Appropriations for federal wildland fire management activities have increased considerably since the 1990s, as we and the Congressional Research Service have reported. Although the federal government faces fiscal exposure from climate change across the nation, it does not have certain information needed by policymakers to help understand the budgetary impacts of such exposure. We have previously reported that the federal budget generally does not account for disaster assistance provided by Congress—which can reach tens of billions of dollars for some disasters—or the long-term impacts of climate change on existing federal infrastructure and programs. For Example, as we reported in April 2018, the Office of Management and Budget’s (OMB) climate change funding reports we reviewed did not include funding information on federal programs with significant fiscal exposures to climate change identified by OMB and others—such as domestic disaster assistance, flood insurance, and crop insurance. A more complete understanding of climate change fiscal exposures can help policymakers anticipate changes in future spending and enhance control and oversight over federal resources, as we reported in October 2013. For budget decisions for federal programs with fiscal exposure to climate change, we found in the April 2018 report that information that could help provide a more complete understanding would include: (1) costs to repair, replace, and improve the weather- related resilience of federally-funded property and resources; (2) costs for federal flood and crop insurance programs; and (3) costs for disaster assistance programs, among other identified areas of fiscal exposure to climate change. To help policymakers better understand the trade-offs when making spending decisions, we recommended in the April 2018 report that OMB provide information on fiscal exposures related to climate change in conjunction with future reports on climate change funding. Although the federal government faces fiscal exposure to climate change, its investments in resilience to climate change impacts have been limited. One way to reduce federal fiscal exposure is to enhance resilience by reducing or eliminating long-term risk to people and property from natural hazards. For example, in September 2018 we reported that elevating homes and strengthened building codes in Texas and Florida prevented greater damages during the 2017 hurricane season. In addition, one company participating in a 2014 forum we held on preparing for climate- related risks noted that for every dollar it invested in resilience efforts, the company could prevent $5 in potential losses. Finally, a 2018 interim report by the National Institute of Building Sciences examined a sample of federal grants for hazard mitigation. The report estimated approximate benefits to society (i.e., homeowners, communities, etc.) in excess of costs for several types of resilience projects through the protection of lives and property, and prevention of other losses. For example, while precise benefits are uncertain, the report estimated that for every grant dollar the federal government spent on resilience projects, over time, society could accrue benefits amounting to the following: About $3 on average from projects addressing fire at the wildland urban interface, with most benefits (69 percent) coming from the protection of property (i.e., avoiding property losses). About $5 on average from projects to address hurricane and tornado force winds, with most benefits (89 percent) coming from the protection of lives. This includes avoiding deaths, nonfatal injuries, and causes of post-traumatic stress. About $7 on average from projects that buy out buildings prone to riverine flooding, with most benefits (65 percent) coming from the protection of property. The interim report also estimated that society could accrue benefits amounting to about $11 on average for every dollar invested in designing new buildings to meet the 2018 International Building Code and the 2018 International Residential Code—the model building codes developed by the International Code Council—with most benefits (46 percent) coming from the protection of property. We reported in October 2009 that the federal government’s activities to build resilience to climate change were carried out in an ad hoc manner and were not well coordinated across federal agencies. Federal agencies have included some of these activities within existing programs and operations—a concept known as mainstreaming. For example, the Fourth National Climate Assessment reported that the U.S. military integrates climate risks into its analysis, plans and programs, with particular attention paid to climate effects on force readiness, military bases, and training ranges. However, according to the Fourth National Climate Assessment, while a significant portion of climate risk can be addressed by mainstreaming, the practice may reduce the visibility of climate resilience relative to dedicated, stand-alone approaches and may prove insufficient to address the full range of climate risks. In addition, as we reported in March 2019, the Disaster Recovery Reform Act of 2018 (DRRA) was enacted in October 2018, which could improve state and local resilience to disasters. DRRA, among other things, allows the President to set aside, with respect to each major disaster, a percentage of the estimated aggregate amount of certain grants to use for pre-disaster hazard mitigation and makes federal assistance available to state and local governments for building code administration and enforcement. However, it is too early to tell what impact the implementation of the act will have on state and local resilience. The federal government has made some limited investments in resilience and DRRA could enable additional improvements at the state and local level. However, we reported in September 2017 that the federal government had not undertaken strategic government-wide planning to manage significant climate risks before they become fiscal exposures. We also reported in July 2015 that the federal government had no comprehensive strategic approach for identifying, prioritizing, and implementing investments for disaster resilience. As an initial step in managing climate risks, most of the experts we interviewed for the September 2017 report told us that federal decision makers should prioritize risk management efforts on significant climate risks that create the greatest fiscal exposure. However, as we reported in our March 2019 High-Risk List, the federal government had not made measurable progress since 2017 to reduce fiscal exposure in several key areas that we have identified. The High-Risk List identified Limiting the Federal Government’s Fiscal Exposure by Better Managing Climate Change Risks as an area needing significant attention because the federal government has regressed in progress toward one of our criterion for removal from the list. As we reported in March 2019, the federal government could reduce its fiscal exposure to climate change by focusing and coordinating federal efforts. However, the federal government is currently not well organized to address the fiscal exposure presented by climate change, partly because of the inherently complicated and crosscutting nature of the issue. We have made a total of 62 recommendations related to limiting the federal government’s fiscal exposure to climate change over the years, 12 of which have been made since February 2017. As of December 2018, 25 of these recommendations remained open. In describing what needs to be done to reduce federal fiscal exposure to climate change, our March 2019 High-Risk report discusses many of the open recommendations. Implementing these recommendations could help reduce federal fiscal exposure. Several of them, including those highlighted below, identify key government-wide efforts needed to help plan for and manage climate risks and direct federal efforts toward common goals, such as improving resilience: Develop a national strategic plan: In May 2011, we recommended that appropriate entities within the Executive Office of the President (EOP), including OMB, work with agencies and interagency coordinating bodies to establish federal strategic climate change priorities that reflect the full range of climate-related federal activities, including roles and responsibilities of key federal entities. Use economic information to identify and respond to significant climate risks: In September 2017, we recommended that the appropriate entities within EOP use information on the potential economic effects of climate change to help identify significant climate risks facing the federal government and craft appropriate federal responses. Such federal responses could include establishing a strategy to identify, prioritize, and guide federal investments to enhance resilience against future disasters. Provide decision makers with the best available climate information: In November 2015, we reported that federal efforts to provide information about climate change impacts did not fully meet the climate information needs of federal, state, local, and private sector decision makers, which hindered their efforts to plan for climate change risks. We reported that these decision makers would benefit from a national climate information system that would develop and update authoritative climate observations and projections specifically for use in decision-making. As a result, we recommended that EOP (1) designate a federal entity to develop and periodically update a set of authoritative climate observations and projections for use in federal decision-making, which other decision makers could also access; and (2) designate a federal entity to create a national climate information system with defined roles for federal agencies and nonfederal entities with existing statutory authority. Consider climate information in design standards: In November 2016, we reported that design standards, building codes, and voluntary certifications established by standards-developing organizations play a role in ensuring the resilience of infrastructure to the effects of natural disasters. However, we reported that these organizations faced challenges to using forward-looking climate information that could help enhance the resilience of infrastructure. As a result, we recommended in the November 2016 report that the Department of Commerce, acting through the National Institute of Standards and Technology—which is responsible for coordinating federal participation in standards organizations—convene federal agencies for an ongoing government-wide effort to provide the best available forward-looking climate information to standards-developing organizations for their consideration in the development of design standards, building codes, and voluntary certifications. In conclusion, the effects of climate change have already and will continue to pose risks that can create fiscal exposure across the federal government and this exposure will continue to increase. The federal government does not generally account for such fiscal exposure to programs in the budget process nor has it undertaken strategic efforts to manage significant climate risks that could reduce the need for far more costly steps in the decades to come. To reduce its fiscal exposure, the federal government needs a cohesive strategic approach with strong leadership and the authority to manage risks across the entire range of related federal activities. The federal government could make further progress toward reducing fiscal exposure by implementing the recommendations we have made. Chairman Yarmuth, Ranking Member Womack, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact me at (202) 512-3841or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are J. Alfredo Gómez (Director), Joseph Dean Thompson (Assistant Director), Anne Hobson (Analyst in Charge), Celia Mendive, Kiki Theodoropoulos, Reed Van Beveren, and Michelle R. Wong. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Since 2005, federal funding for disaster assistance is at least $450 billion, including approximately $19.1 billion in supplemental appropriations signed into law on June 6, 2019. In 2018 alone, there were 14 separate billion-dollar weather and climate disaster events across the United States, with a total cost of at least $91 billion, according to the National Oceanic and Atmospheric Administration. The U.S. Global Change Research Program projects that disaster costs will likely increase as certain extreme weather events become more frequent and intense due to climate change. The costs of recent weather disasters have illustrated the need for planning for climate change risks and investing in resilience. Resilience is the ability to prepare and plan for, absorb, recover from, and more successfully adapt to adverse events, according to the National Academies of Science, Engineering, and Medicine. Investing in resilience can reduce the need for far more costly steps in the decades to come. Since February 2013, GAO has included Limiting the Federal Government's Fiscal Exposure by Better Managing Climate Change Risks on its list of federal program areas at high risk of vulnerabilities to fraud, waste, abuse, and mismanagement or most in need of transformation. GAO updates this list every 2 years. In March 2019, GAO reported that the federal government had not made measurable progress since 2017 to reduce fiscal exposure to climate change. This testimony—based on reports GAO issued from October 2009 to March 2019—discusses (1) what is known about the potential economic effects of climate change in the United States and the extent to which this information could help federal decision makers manage climate risks across the federal government, (2) the potential impacts of climate change on the federal budget, (3) the extent to which the federal government has invested in resilience, and (4) how the federal government could reduce fiscal exposure to the effects of climate change. GAO has made 62 recommendations related to the Limiting the Federal Government’s Fiscal Exposure by Better Managing Climate Change Risks high-risk area. As of December 2018, 25 of those recommendations remained open. The estimated economic effects of climate change, while imprecise, can convey useful insight about potential damages in the United States. In September 2017, GAO reported that the potential economic effects of climate change could be significant and unevenly distributed across sectors and regions (see figure). This is consistent with the recent findings of the U.S. Global Change Research Program's Fourth National Climate Assessment, which concluded, among other things, that the continued increase in the frequency and extent of high-tide flooding due to sea level rise threatens America's trillion-dollar coastal infrastructure. Information about the potential economic effects of climate change could inform decision makers about significant potential damages in different U.S. sectors or regions. According to prior GAO work, this information could help decision makers identify significant climate risks as an initial step toward managing them. The federal government faces fiscal exposure from climate change risks in several areas, including: Disaster aid: due to the rising number of natural disasters and increasing reliance on federal assistance. GAO has previously reported that the federal government does not adequately plan for disaster resilience. GAO has also reported that, due to an artifically low indicator for determining a jursidiction's ability to respond to disasters that was set in 1986, the Federal Emergency Management Agency risks recommending federal assistance for juridisctions that could recover on their own. Federal insurance for property and crops: due, in part, to the vulnerability of insured property and crops to climate change impacts. Federal flood and crop insurance programs were not designed to generate sufficient funds to fully cover all losses and expenses. The flood insurance program, for example, was about $21 billion in debt to the Treasury as of April 2019. Further, the Congressional Budget Office estimated in May 2019 that federal crop insurance would cost the federal government an average of about $8 billion annually from 2019 through 2029. Operation and management of federal property and lands: due to the hundreds of thousands of federal facilities and millions of acres of land that could be affected by a changing climate and more frequent extreme events. For example, in 2018, Hurricane Michael devastated Tyndall Air Force Base in Florida, with a preliminary repair estimate of $3 billion. The federal budget, however, does not generally account for disaster assistance provided by Congress or the long-term impacts of climate change on existing federal infrastructure and programs. GAO has reported that more complete information about fiscal exposure could help policymakers better understand the trade-offs when making spending decisions. Further, federal investments in resilience to reduce fiscal exposures have been limited. As GAO has reported, enhancing resilience can reduce fiscal exposure by reducing or eliminating long-term risk to people and property from natural hazards. For example, a 2018 interim report by the National Institute of Building Sciences estimated approximate benefits to society in excess of costs for several types of resilience projects. While precise benefits are uncertain, the report estimated that for every dollar invested in designing new buildings to particular design standards, society could accrue benefits amounting to about $11 on average. The federal government has invested in individual agency efforts that could help build resilience within existing programs or projects. For example, the National Climate Assessment reported that the U.S. military integrates climate risks into its analysis, plans, and programs. In additon, as GAO reported in March 2019, the Disaster Recovery Reform Act of 2018 could improve resilience by allowing the President to set aside a portion of certain grants for pre-disaster mitigation. However, the federal government has not undertaken strategic government-wide planning to manage climate risks. GAO's March 2019 High-Risk report identified a number of recommendations GAO has made related to fiscal exposure to climate change. The federal government could reduce its fiscal exposure by implementing these recommendations. Among GAO's key government-wide recommendations are: Entities within the Executive Office of the President (EOP) should work with partners to establish federal strategic climate change priorities that reflect the full range of climate-related federal activities; Entities within EOP should use information on potential economic effects from climate change to help identify significant climate risks and craft appropriate federal responses; Entities within EOP should designate a federal entity to develop and update a set of authoritative climate observations and projections for use in federal decision making, and create a national climate information system with defined roles for federal agencies and certain nonfederal entities; and The Department of Commerce should convene federal agencies to provide the best-available forward-looking climate information to organizations that develop design standards and building codes to enhance infrastructure resilience.", "document_type": "gao"}
{"report": "In the Telecommunications Act of 1996 (the 1996 Act), Congress specified that consumers in “rural, insular, and high-cost areas” should have access to telecommunication rates and services that are “reasonably comparable” to consumers in urban areas. The 1996 Act altered the federal mechanism for funding universal service by requiring telecommunications carriers and other entities providing interstate telecommunications service to contribute to the USF, unless exempted by FCC. The carriers generally pass these costs on to customers, sometimes in the form of a line item on customers’ telephone bills. USF provides financial support to carriers through four different programs, each targeting a particular group of telecommunications carriers or consumers. The high-cost program provides support to both wireline and wireless carriers that provide telecommunications services in areas that carriers would otherwise not serve and where there is no competition from other providers. These are typically rural or remote areas where the customer base is relatively small and the cost of installing infrastructure is high. The high-cost program has been the largest USF program based on disbursements and has been particularly important to rural areas. High- cost support is intended to offset the carriers’ higher costs, thereby allowing them to provide services and rates that are reasonably comparable to those that consumers in lower-cost—generally urban— areas receive. In 2009, Congress required FCC to develop a broadband plan to ensure that every American has access to broadband capability, including a detailed plan for providing this service at affordable rates. In response, an FCC task force issued the National Broadband Plan in 2010, which recommended reforming USF so it could support both telephone and broadband service. FCC’s USF Transformation Order of 2011 emerged in response to this recommendation and provided USF support to carriers for broadband capable networks. The order required carriers that receive support to meet broadband-speed and quality-deployment requirements. Through the USF Transformation Order, FCC adopted a framework to transition high-cost carriers from traditional cost-accounting support to incentive-based support mechanisms, using forward-looking broadband cost models and competitive bidding. FCC’s forward-looking cost models use historical data to project the future financial needs of carriers providing telecommunications services. According to FCC, rate-of-return carriers receive about $2.5 billion in annual support from the high-cost program to support service deployments in these carriers’ 1,078 rate-of- return service areas, which FCC refers to as “study areas.” FCC has allowed rate-of-return carriers to choose, on a voluntarily basis, one the following mechanisms to receive USF support: Traditional cost-accounting support mechanism. This method retroactively provides support to carriers for costs already incurred, based on cost studies, including financial statements these companies provide each year. At the time of our review, according to FCC officials, FCC guaranteed these companies recovery of eligible deployment costs, plus a return of 10.25 percent on regulated investment costs. According to FCC’s OIG officials we interviewed, many carriers contract with telecommunications accountants to navigate the complicated process of determining which costs are reimbursable by the high-cost program and file the associated documentation with USAC. According to FCC, as of September 2019, there were approximately 437 study areas served by rate-of- return carriers receiving support through this mechanism. Model-based support mechanism. This method is aimed at providing a level of support to carriers based on modeled forward- looking costs and revenues of an efficient carrier to serve an area with voice and broadband Internet. According to FCC officials, in developing the model, FCC: had experts peer-review the model’s methodology; demonstrated how different inputs affect model support and sought stakeholder feedback on the reasonableness of how these inputs affected support levels; publicly released the model’s methodology; and used historical deployment cost and revenue data to develop the model’s inputs and assumptions. As of September 2019, FCC officials told us that rate-of-return companies serving 641 study areas were receiving support through this mechanism (or almost 60 percent of all 1,078 rate-of-return carriers’ study areas). FCC determines overall policy and issues the regulations that govern the high-cost program, while FCC’s Wireline Competition Bureau in particular implements FCC’s policies and programs regarding rate-of-return carriers. State governments play a role in implementing the federal high- cost program, as do a not-for-profit corporation (USAC) and an association (NECA). As shown in table 1, FCC, USAC, and NECA have responsibilities for the high-cost program to ensure payments to rate-of- return carriers are made properly. FCC has the following audit and oversight procedures for the high-cost program: Carrier self-certification. Carriers submit cost and line count data directly to NECA. Carrier self-certification is the primary tool for ensuring that carriers use high-cost program support consistent with program rules. USAC uses these data to qualify carriers for the program and also to calculate the amount of support carriers are eligible to receive. Carrier audits. Audits of carriers receiving high-cost program support are the primary tool used to oversee carrier activities, and audits may be conducted by USAC, state regulators, or FCC’s OIG. USAC primarily relies on assessments from the Payment Quality Assurance Program and Beneficiary and Contributor Audit Program that occur after disbursements have been made to detect improper payments, which may include fraud. Carrier data validation process. All cost data that the carriers submit to NECA for purposes of high-cost support are subject to several electronic validations, which focus on ensuring that all required data are reported and that the data ranges are consistent with information reported in previous years. In addition, NECA compares the reported cost data with financial records supporting carriers’ audited financial statements to identify any discrepancies and to require corrections when discrepancies are discovered. Carriers’ broadband deployment verification. Since 2018, USAC has performed carrier broadband deployment verifications by obtaining broadband location data to monitor whether a carrier’s broadband deployment meets FCC requirements. Carriers receive verification reports from USAC that reflect the results of the verification process. Whistleblower process. USAC maintains a whistleblower log that is shared with FCC. Through whistleblower complaints, USAC may identify instances of potentially fraudulent activity. FCC has identified three rate-of-return carriers that received at least $34 million in improper payments from the high-cost program in prior years. Two such cases were described above. In the third case, a rate-of-return carrier self-reported to NECA and USAC what it represented to be the costs and revenues of providing its telecommunications service; as discussed previously, NECA and USAC rely upon the accuracy and completeness of the carrier’s reporting to calculate the carrier’s support. An FCC OIG investigation later revealed that the carrier had manipulated FCC’s accounting rules by including the costs of a nonregulated, commercial mobile radio service in the information it submitted to NECA, thus inflating the amount of high-cost program support the carrier received. FCC eventually determined that the carrier owed the federal government almost $7 million in support overpayments received between 2005 and 2010. A petition for reconsideration is pending. As there is a finite amount of funding for the high-cost program, compensating carriers for improper, ineligible, and inflated costs they claim means less program funds are available for deploying service to the areas the program was designed to serve. Federal internal control standards, along with GAO’s fraud risk framework, OMB guidance, and the Fraud Reduction and Data Analytics Act of 2015 have placed an increased focus on the need for federal program managers to take a strategic approach to managing improper payments and risks, including the risk of fraud. GAO’s fraud risk framework provides comprehensive guidance for conducting fraud-risk assessments and using the results to develop a robust fraud risk management strategy. This framework also describes overarching concepts and leading practices for establishing an organizational structure and culture that are conducive to fraud risk management, designing and implementing controls to prevent and detect potential fraud, and monitoring and evaluating fraud risk management activities. The leading practices in the fraud risk framework are organized into four components—commit, assess, design and implement, and evaluate and adapt—as depicted in figure 1. FCC, in various orders, has adopted several funding and other reforms specific to rate-of-return carriers. As described below, the reforms we reviewed were designed to (1) control the carrier and high-cost program expenditures, (2) incentivize efficient broadband deployment, and (3) ensure carriers’ compliance with the high-cost program. Prior to the 2011 USF Transformation Order, rate-of-return carriers primarily received high-cost support based on their actual costs. Under the old rules, carriers faced no FCC-imposed limits and, according to FCC, had no incentive to be more efficient. FCC adopted the reforms described in figure 2 to control the program’s expenditures. As shown in figure 2 above, FCC’s reform effort related to eliminating support to areas with competition has been ongoing since 2011. According to FCC officials, FCC relied on its broadband deployment data to identify competitively served areas, but we have previously reported that FCC’s broadband deployment data are not always accurate. In August 2017, FCC initiated a proceeding to review the Form 477—the principal tool FCC uses to gather data on communications services, including broadband services—to help inform its policy making. According to FCC, a goal of this proceeding was to enable FCC to collect better and more accurate information on the Form 477. In August 2019, FCC adopted an order based on the proceeding that, among other things, established requirements for collecting geospatial broadband-coverage maps from internet service providers. According to the order, FCC will require the service providers to submit granular maps of the areas where they have broadband-capable networks; FCC intends that these broadband-deployment maps will enable FCC to precisely target scarce universal service dollars to where broadband service is lacking. According to FCC, one of the USF’s core principles since 2011 has been to ensure that support is provided in the most efficient manner possible, recognizing that ultimately American consumers contribute to programs like the high-cost program. FCC adopted the reforms described in figure 3 to advance its long-standing objective of adopting incentive-based policies to spur additional broadband deployment, while preserving additional funding in the high-cost program for other reforms. According to FCC, the prior cases of carriers’ abuses of USF support for unrelated purposes prompted FCC to issue more specific rules for compliance and reporting obligations. Accordingly, FCC adopted reforms described in figure 4 to improve accountability and transparency of the high-cost program. Of the reforms we reviewed, one reform in particular—the development of a model-based support mechanism—shows promise in reducing fraud risk, according to stakeholders from federal and state government, industry, and accounting firms we contacted. Stakeholders said the model-based support mechanism is less prone to fraud risks and is a more efficient support mechanism than traditional cost-accounting support. In particular, unlike the traditional cost-accounting-support mechanism, model-based support does not rely on carrier-submitted data to determine support amounts. Instead, the model uses, among other things, a combination of historical cost data and other data, such as expected customer revenue, to determine support amounts. Since there are no data provided by carriers in the process of determining support amounts, there is no means by which carriers can provide falsified information to fraudulently receive excess support. The carriers involved in the previously described improper payments cases were receiving support from the traditional cost-accounting support mechanism. On the other hand, stakeholders told us FCC’s traditional cost-accounting support mechanism is complex and difficult to audit, and that such weaknesses make it prone to fraud risks. For example, USAC officials told us it is time consuming to detect inflated costs associated with carriers’ affiliate company transactions. The traditional cost-accounting support mechanism also requires that carriers separate costs based upon the type of service with which the cost was associated. According to FCC’s OIG officials and representatives from accounting firms we contacted, determining whether a carrier has overly attributed costs to eligible services is difficult. For instance, determining if labor costs are properly being allocated between eligible and ineligible services requires looking at each employee’s timesheet. According to USAC, it also faces challenges auditing traditional cost-accounting support payments due to limited expertise and capacity to address the complexity of the audits. USAC officials noted that this issue has been exacerbated by audit staff turnover. According to USAC officials and some stakeholders we contacted, auditing carriers receiving traditional cost-accounting support is also difficult due to the extensive documentation requirements for this type of support, requirements that often entails hundreds of pages of financial information per carrier. USAC officials told us that a single audit can take over 1,000 hours to complete, and USAC officials told us they only completed 10 audits of carriers that received support on a traditional cost-accounting basis in fiscal year 2018. As previously noted, FCC allows carriers to choose which funding mechanism is best suited for their company. FCC officials told us they developed the model-based funding mechanism in consultation with carriers and industry stakeholders. However, according to FCC officials, the model’s use is not mandatory because some carriers do not believe that the model would accurately reflect their specific costs. FCC officials said the agency does not have plans to assess the accuracy of the model’s cost estimates or require carriers to receive model-based support. FCC officials told us they did not have plans to assess the model. FCC officials told us they had not planned to do so because in May 2019 FCC had just made available model-based support to the remaining legacy carriers, and FCC was still in the process of evaluating next steps. Planning for and conducting such an assessment would enable FCC to demonstrate the validity of the model and its reliability in accounting for the costs of broadband deployment. Federal internal- control standards state that management should use quality information to make informed decisions and evaluate program performance in achieving key objectives. Furthermore, according to FCC’s strategic plan, FCC must ensure its USF programs, including those for the high- cost program, are well managed, efficient, and fiscally responsible, and the National Broadband Plan says that FCC should move rate-of-return carriers to incentive-based regulation mechanisms, such as model-based support. Yet because a substantial number of rate-of-return study areas— 437—continue to receive traditional cost-accounting support, and the carriers that provide service in these areas cannot be effectively audited, significant fraud risks remain for the high-cost program. By assessing the model, FCC would have greater assurance that it is producing reliable cost estimates and be better positioned to determine whether to make its use mandatory. Managers of federal programs are responsible for managing fraud risks. Implementing effective fraud risk-management processes is important to help ensure that federal programs fulfill their intended purpose and funds are spent effectively. GAO’s fraud risk framework is aligned with federal internal-control standards related to assessing fraud risk. It focuses on preventive activities, which generally offer the most cost-efficient use of resources since they enable managers to avoid a costly and inefficient “pay-and-chase” model, which refers to the practice of detecting fraudulent transactions and recovering funds after fraudulent payments have been made. As discussed previously, our fraud risk framework consists of four components—commit, assess, design and implement, and evaluate and adapt—each of which includes overarching concepts and leading practices for carrying them out. We found that FCC has implemented some policies and procedures related to managing fraud risk for the high-cost program. For example, according to a memorandum of understanding between FCC and USAC, FCC requires USAC to alert, as appropriate, FCC’s OIG and Enforcement Bureau about potential instances of fraud. However, as detailed in appendix II, FCC’s efforts do not fully align with some elements of the fraud risk framework. In particular, we found deficiencies in FCC’s efforts related to the following three overarching concepts and one high-level component: creating a structure with a dedicated entity to manage fraud risk activities (overarching concept within the commit component); planning regular fraud-risk assessments tailored to the program and assessing these risks to determine the program’s fraud risk profile (two overarching concepts within the assess component); and designing and implementing an antifraud strategy for the program (the design and implement component). Creating a structure with a dedicated entity to lead fraud risk- management activities. Leading practices for managing fraud risk include demonstrating management’s commitment to combating fraud and designating an entity to design and oversee fraud risk-management activities. According to GAO’s fraud risk framework, an entity should lead these activities by serving as the repository of knowledge on fraud risks and controls, managing the fraud-risk assessment process, leading fraud- awareness activities, and coordinating antifraud initiatives. According to FCC officials, FCC has steering committees for each of the four USF programs, including the high-cost program. According to FCC officials, the steering committees allow in-depth discussions about each program, including on operational issues such as current spending levels and information technology systems, as well as improper payments and other issues. However, fraud risk is but one of many responsibilities of these steering committees, and they do not fill the role of a dedicated fraud risk- management entity, as called for by the fraud risk framework. In August 2019, FCC officially launched a Fraud Division—comprising existing FCC staff who investigate and prosecute fraud—within its Enforcement Bureau. However, FCC told us the scope of the new division’s operations is limited to investigations, so the Fraud Division does not fill the role of a dedicated fraud risk-management entity. Planning regular fraud-risk assessments tailored to the program and determining the fraud risk profile. An effective antifraud entity tailors the approach for carrying out regular fraud-risk assessments of its programs. According to GAO’s fraud risk framework, the approach should, among other things: fully consider the specific fraud risks the agency or program faces, analyze the potential likelihood and effects of fraud schemes, and document prioritized fraud risks. According to FCC officials, FCC has annually worked with USAC high- cost program staff to identify and assess some risks facing the high-cost program, some of which are fraud risks, but has not planned regular fraud-risk assessments that are tailored to the high-cost program in accordance with GAO’s fraud risk framework. FCC officials also told us that they adopted a tool originally developed by another agency that was used to evaluate risks facing that agency’s loan and grant programs, not just fraud risks. Using that tool as a model, FCC created a risk assessment document that included fraud risk as one of nine categories of risks across the high-cost program. Based on our discussions with FCC officials, however, the document does not constitute a fraud-risk assessment that takes into account changes to the program or operating environment. Furthermore, the risk assessment document does not constitute a fully tailored risk assessment because it does not identify and assess the fraud risks stakeholders we interviewed described as inherent to the high-cost program, detailed below. Risk caused by the complexity of the high-cost program’s cost- allocation rules. Officials from three out of four accounting firms, FCC’s OIG, and a state utility commission we contacted singled out the specific fraud risk caused by what they described as confusing and subjective rules governing the process carriers use to separate eligible and ineligible costs. Risks related to oversight challenges. Stakeholders identified several oversight challenges as significant in that they could contribute to fraud risks for the program, such as: financial mismanagement within carriers that allowed companies to submit potentially fraudulent information to USAC and NECA, and that a telecommunications accountant told us contributed to previous instances of alleged fraud; USAC’s audit personnel challenges that were due to attrition and limited resources and expertise and that were identified by officials from FCC, USAC, FCC’s OIG, and an accounting firm; and deficiencies identified by FCC’s OIG in NECA’s internal controls over payments to carriers, data validation, and the appropriateness of NECA’s role validating carriers’ cost information. In addition, FCC’s OIG officials told us of oversight challenges related to carriers’ reporting, including that it is difficult for USAC to detect when carriers improperly report rates billed for services provided by an affiliate of the company or report incorrect labor rates. Furthermore, we found FCC had not identified and assessed risks to determine the fraud risk profile for the high-cost program, as called for in the fraud risk framework. A fraud risk profile is the summation of key findings and conclusions from a fraud-risk assessment, including the analysis of the types of internal and external fraud risks, their perceived likelihood and effects, managers’ risk tolerance, and the prioritization of risks. FCC officials said they consider the risk of fraud to be low in the high-cost program, and FCC includes fraud risk in its current risk assessment process. Since FCC believes the fraud risk is low for the high-cost program, FCC has not deemed it necessary to conduct a separate fraud-risk assessment of the program. For example, FCC provided us with documentation related to its Enterprise Risk Management activities that identifies risks USAC faces to achieving its corporate objectives. However, while FCC considers fraud risks as part of these activities, the document does not specify the fraud risk tolerance for the program or constitute a fraud risk profile. Without conducting regular fraud-risk assessments to gauge the likelihood and effects of the inherent fraud risks described above, and potentially others, FCC cannot determine or document the program’s fraud risk profile. Furthermore, FCC has no assurance that it has fully considered important fraud risks, determined its tolerance for risks that could be lower priorities, or made sound decisions on how to allocate resources to respond to fraud risks. Not doing so could result in FCC compensating carriers for improper, ineligible, and inflated costs, such as in the previously discussed cases of identified fraud. By regularly assessing fraud risks to determine a fraud risk profile, FCC could better determine the extent to which it has designed and implemented adequate fraud-prevention controls. Designing and implementing an antifraud strategy for the program. Managers who effectively manage fraud risk develop and document an antifraud strategy that describes the program’s activities for preventing, detecting, and responding to the fraud risks identified during the fraud-risk assessment. FCC and USAC have established mechanisms to enhance the oversight of USF programs, mechanisms that can also help mitigate fraud risks for the high-cost program, including: In fiscal year 2016, USAC implemented a risk-based selection method for conducting audits to identify the entities with the greatest risk. USAC forwards potential fraud, waste, and abuse cases to FCC. FCC’s OIG established a hotline that can be used to report potential fraud, and USAC established a Whistleblower Alert mechanism to inform USAC of possible instances of universal service support being misapplied or mismanaged, or when carriers are potentially violating laws, rules, or regulations. USAC shares this information with FCC. FCC and USAC formed a working group tasked with developing a data-analytics tool designed to share USAC high-cost program data with FCC. FCC’s documentation of the tool states that once developed, the tool will help FCC’s Enforcement Bureau in its fraud detection activities across all USF programs. FCC officials described the development of the tool as technically challenging and said there is no established timeline for implementing the tool. FCC officials said that FCC has regular, informal interactions concerning fraud risk with USAC and, to a lesser degree, with NECA and that FCC has confidence that USAC’s improved audit processes are identifying issues appropriately. However, FCC has not specifically designed or implemented an overall strategy to mitigate fraud risks across the high-cost program. An FCC official said FCC believes its existing antifraud activities are adequate. The FCC official said FCC considers the risk of fraud in the high-cost program to be low because USAC audits have revealed that carriers’ financial reporting errors occur at a low rate and therefore do not indicate that a large amount of fraud is occurring. The official said FCC’s fraud risk-management practices are based on federal internal-control standards and are woven into existing FCC mechanisms. Given that FCC has not conducted a fraud-risk assessment that is tailored to the high-cost program and that the deceptive nature of fraud makes it difficult to measure in a reliable way, it is unclear how FCC officials reached the conclusion that the program’s risk of fraud is low. Also, in the absence of an antifraud strategy, FCC has little assurance that it has the specific control activities to prevent and detect high-cost program fraud and that the types of misconduct that previously occurred are not widespread. The improper payment activity discussed previously was caught after USAC provided support to these carriers, and it was not prevented or detected through any strategic fraud risk-management effort on FCC’s part. Furthermore, the FCC’s OIG has expressed significant concerns about such abuses by rate-of-return carriers and about the effectiveness of USAC’s auditing processes in helping prevent improper payments to these carriers. As noted above, USAC’s high-cost program audits can take over 1,000 hours and USAC faces human capital challenges that have diminished its audit capacity. In addition, while NECA’s data validations and reviews of high-cost support provide opportunities to identify input and data-reporting errors, they do not specifically address whether or not the data provided by carriers are reasonable or if the money carriers spent addresses the intended purposes of the high-cost program. Designing and implementing an antifraud strategy that conforms to leading practices would help FCC effectively manage and respond to the fraud risks identified during the fraud-risk assessments. Given the continuing importance of deploying telecommunications services in difficult-to-serve areas, effective oversight for rate-of-return carriers is an important component for helping ensure that the high-cost program’s finite funds are used properly to meet the intent of the program. Overall, the traditional cost-accounting mechanism that FCC uses to provide support to a substantial number of rate-of-return carriers is complex, prone to fraud risks, and presents auditing challenges that FCC has not fully addressed. By following leading practices from GAO’s fraud risk framework, FCC could better ensure that it is addressing and strategically targeting the most significant fraud risks facing the high-cost program. Furthermore, FCC’s model-based support mechanism has now been in use by some rate-of-return carriers for several years and stakeholders agree that it is less prone to fraud risks. However, FCC has not assessed the extent to which the model is producing reliable cost estimates. By conducting such an assessment, FCC would have greater assurance that it is producing reliable cost estimates and be better positioned to determine whether to make its use mandatory. We are making the following five recommendations to FCC: The Chairman of FCC should ensure that FCC’s Office of Managing Director follows the leading practices in GAO’s fraud risk framework related to a dedicated entity’s management of its antifraud activities, such as serving as the repository of knowledge on fraud risks and coordinating antifraud initiatives. (Recommendation 1) The Chairman of FCC should plan regular fraud-risk assessments tailored to the high-cost program and assess these risks to determine the program’s fraud risk profile, as provided in GAO’s fraud risk framework. (Recommendation 2) The Chairman of FCC should design and implement an antifraud strategy for the high-cost program with specific control activities, based upon the results of fraud-risk assessments and a corresponding fraud risk profile, as provided in GAO’s fraud risk framework. (Recommendation 3) The Chairman of FCC should assess the model-based support mechanism to determine the extent to which it produces reliable cost estimates. (Recommendation 4) The Chairman of FCC should consider whether to make use of the model-based support mechanism mandatory depending on the results of the assessment. (Recommendation 5) We provided a draft of this report to FCC for review and comment. In FCC’s written comments, reprinted in appendix III, FCC described actions it would take to implement our recommendations. FCC also provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Chairman of the FCC, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. To help managers combat fraud and preserve integrity in government agencies and programs, we identified leading practices for managing fraud risks and organized them into a conceptual framework called the Fraud Risk Management Framework (the framework). As described in the background section of this report and depicted visually in figure 1, the framework encompasses control activities to prevent, detect, and respond to fraud, with an emphasis on prevention, to help managers achieve the objective of mitigating fraud risks. The second of four framework components—Assess—calls for specific actions managers should take to achieve the objective of mitigating fraud risks. Specifically, managers should plan regular fraud-risk assessments and assess these risks to determine a fraud risk profile. Figure 5 illustrates the key elements of the fraud-risk assessment process that lead to the creation of a program’s fraud risk profile. We developed a data collection instrument to structure our assessment of the Federal Communications Commission’s (FCC) antifraud efforts for the high-cost program related to the commit, assess, and design and implement components of GAO’s fraud risk framework. For our assessment, we determined the extent to which FCC had implemented the leading practices within each component, as summarized in table 2. We did not assess FCC’s approach against leading practices in the “evaluate and adapt” component of the framework because we determined that FCC had not adopted fraud risk management leading practices within the first three components, and therefore it was premature for us to assess whether FCC was evaluating and adapting its use of these practices. In addition to the individual named above, Sally Moino (Assistant Director); Sean Standley (Analyst in Charge); Philip Farah; Camilo Flores; Mark Goldstein; Gary Guggolz; Hannah Hubbard; Josh Ormond; Ben Licht; Rebecca Shea; Andrew Stavisky; and Michelle Weathers made key contributions to this report.", "summary": "The Universal Service Fund's high-cost program provides financial support to telecommunications carriers in areas where the cost to provide broadband is high. Through this program, FCC provides about $2.5 billion in annual support payments to rate-of-return carriers. The manner in which FCC currently provides the support payments to some of these carriers is prone to fraud risks. A prior case involved a rate-of-return carrier that received at least $27 million in improper payments from the program. GAO was asked to review funding reforms and fraud controls FCC has implemented for rate-of-return carriers. This report examines the extent to which FCC: (1) has implemented funding reforms specific to rate-of-return carriers, and (2) is managing fraud risks for the high-cost program in accordance with leading practices. GAO reviewed FCC's and USAC's procedures, relevant regulations, and guidance, and assessed these documents against applicable criteria, including federal internal-control standards, FCC's strategic plan, and GAO's fraud risk framework. GAO interviewed FCC and USAC officials, in addition to industry and other stakeholders representing a variety of non-generalizable viewpoints. The Federal Communications Commission (FCC) has implemented several funding reforms for small, rural telecommunications carriers—referred to as “rate-of-return carriers”—receiving high-cost program support. These reforms are aimed at controlling the program's expenditures and incentivizing efficient broadband deployment. According to FCC's strategic plan, FCC must ensure the high-cost program is well managed, efficient, and fiscally responsible. One of the reforms that GAO reviewed established a funding mechanism for the carriers whereby FCC determines the level of financial support to provide the carriers based on cost and revenue estimates produced by a model. Stakeholders told GAO that this model-based funding mechanism is less prone to fraud risks than the traditional cost-accounting funding mechanism, which reimburses carriers for their reported costs. However, FCC did not make use of this reform mandatory and a substantial number of rate-of-return carriers continue to receive support from the traditional funding mechanism. FCC officials said they developed the model-based funding mechanism in consultation with industry stakeholders. However, FCC officials said they did not have plans to assess the accuracy of cost estimates from the model, which has been in use for several years, or require carriers to receive model-based support as a way to reduce fraud risks. By assessing the model, FCC would have greater assurance that it is producing reliable cost estimates and be better positioned to determine whether to make its use mandatory. FCC has some policies and processes in place to manage fraud risks for the high-cost program. For example, the Universal Service Administrative Company (USAC)—the not-for-profit corporation that administers the program—reviews and audits rate-of-return support payments and forwards potential fraud cases to FCC's Office of Inspector General and Enforcement Bureau for further investigation. FCC is also developing a data-analytics tool to help detect fraud, and in August 2019 launched a new Fraud Division to focus on investigating fraud in the Universal Service Fund's programs. However, FCC's efforts do not fully align with some elements of GAO's fraud risk framework, including: designing and implementing an antifraud strategy for the program. Without regular fraud-risk assessments of the high-cost program, FCC has no assurance that it has fully considered important fraud risks, determined its tolerance for risks that could be lower priorities, or made sound decisions on how to allocate resources to respond to fraud risks. Not doing so could result in FCC compensating carriers for improper, ineligible, or inflated costs. Furthermore, in the absence of an antifraud strategy, FCC has little assurance that it can prevent or detect the types of documented rate-of-return carrier misconduct that have previously occurred. Designing and implementing an antifraud strategy that conforms to leading practices would help FCC effectively manage and respond to the fraud risks identified during the fraud-risk assessments. GAO is making five recommendations, including that FCC should assess the model-based support mechanism and consider making its use mandatory, and implement an antifraud strategy for the high-cost program. FCC stated it would take steps to implement these recommendations.", "document_type": "gao"}
{"report": "Drugs sold in the United States—including active pharmaceutical ingredients and finished dosage forms—are manufactured throughout the world. According to a May 2019 FDA report, in fiscal year 2018 about 40 percent of establishments manufacturing drugs for the U.S. market were located domestically and more than 60 percent of establishments manufacturing for the U.S. market were located overseas. As of March 2019, FDA data show that India and China had the most manufacturing establishments shipping drugs to the United States, with about 40 percent of all foreign establishments in these two countries. (See fig. 1.) Drugs manufactured overseas must meet the same statutory and regulatory requirements as those manufactured in the United States. FDA’s Center for Drug Evaluation and Research (CDER) establishes standards for the safety, quality, and effectiveness of, and manufacturing processes for, over-the-counter and prescription drugs. CDER requests that FDA’s Office of Regulatory Affairs (ORA) inspect both domestic and foreign establishments to ensure that drugs are produced in conformance with applicable laws of the United States, including current good manufacturing practice (CGMP) regulations. FDA investigators generally conduct three main types of drug manufacturing establishment inspections: preapproval inspections, surveillance inspections, and for-cause inspections, as described in table 1. At times, FDA may conduct an inspection that combines both preapproval and surveillance inspection components in a single visit to an establishment. FDA uses multiple databases to select foreign and domestic establishments for surveillance inspections, including its registration database and inspection database. Because the establishments are continuously changing as they begin, stop, or resume marketing products in the United States, CDER creates an establishment catalog monthly. The catalog is prioritized for inspection twice each year. In our 2008 report we found that, because of inaccurate information in FDA’s databases, the agency did not know how many foreign drug establishments were subject to inspection. For example, some establishments included in FDA’s registration database may have gone out of business and did not inform FDA that they had done so or did not actually manufacture drugs for the U.S. market. In our report, we noted that some foreign establishments may register because, in foreign markets, registration may erroneously convey an “approval” or endorsement by FDA, when in fact the establishment may never have been inspected by FDA. We recommended that FDA take steps to improve the accuracy of this registration information. In our 2010 and 2016 reports we found that FDA had taken steps to improve the accuracy and completeness of information in its catalog of drug establishments subject to inspection, such as using contractors to conduct site visits to verify the existence of registered foreign establishments and confirm that they manufacture the products that are recorded in U.S. import records. To prioritize establishments for surveillance inspections, CDER applies a risk-based site selection model to its catalog of establishments to identify those establishments (both domestic and foreign) that, based on the characteristics of the drugs being manufactured, pose the greatest potential public health risk should they experience a manufacturing defect. This model analyzes several factors, including inherent product risk, establishment type, inspection history, and time since last inspection, to develop a list of establishments that FDA considers to be a priority for inspection. Through this process, CDER develops a ranked list of foreign and domestic establishments selected for inspection that is submitted to ORA. To be efficient with its resources, according to FDA officials, ORA staff may shift the order of establishments to be inspected on CDER’s prioritized list based on geographic proximity to other planned inspection trips. Investigators from ORA and, as needed, ORA laboratory analysts with certain expertise are responsible for inspecting drug manufacturing establishments. FDA primarily relies on three groups of investigators to conduct foreign inspections: ORA investigators based in the United States, who primarily conduct domestic drug establishment inspections but may sometimes conduct foreign inspections. Members of ORA’s dedicated foreign drug cadre, a group of domestically based investigators, who exclusively conduct foreign inspections. Investigators assigned to and living in the countries where FDA has foreign offices, including staff based in the foreign offices full time and those on temporary duty assignment to the foreign offices. FDA began opening offices around the world in 2008 to obtain better information on the increasing number of products coming into the United States from overseas, to build relationships with foreign stakeholders, and to perform inspections. FDA full-time foreign office staff are posted overseas for 2-year assignments. FDA staff can also be assigned to the foreign offices on temporary duty assignments for up to 120 days. In fiscal year 2019, there were full-time and temporary duty drug investigators assigned to FDA foreign offices in China and India. FDA’s process for determining whether a foreign establishment complies with CGMPs involves both CDER and ORA. During an inspection, ORA investigators are responsible for identifying any significant objectionable conditions and practices and reporting these to the establishment’s management. Investigators suggest that the establishment respond to FDA in writing concerning all actions taken to address the issues identified during the inspection. Once ORA investigators complete an inspection, they are responsible for preparing an establishment inspection report to document their inspection findings. Inspection reports describe the manufacturing operations observed during the inspection and any conditions that may violate U.S. statutes and regulations. Based on their inspection findings, ORA investigators make an initial recommendation regarding whether regulatory actions are needed to address identified deficiencies using one of three classifications: no action indicated (NAI); voluntary action indicated (VAI); or official action indicated (OAI). Inspection reports and initial classification recommendations for regulatory action are to be reviewed within ORA. For inspections classified as OAI—where ORA identified serious deficiencies—such inspection reports and classification recommendations are to be reviewed within CDER. CDER is to review the ORA recommendations and determine whether regulatory action is necessary. CDER also is to review inspection reports and initial classification recommendations for all for-cause inspections, regardless of whether regulatory action is recommended by ORA. According to FDA policy, inspections classified as OAI may result in regulatory action, such as the issuance of a warning letter. FDA issues warning letters to those establishments manufacturing drugs for the U.S. market that are in violation of applicable U.S. laws and regulations and may be subject to enforcement action if the violations are not promptly and adequately corrected. In addition, warning letters may notify foreign establishments that FDA may refuse entry of their drugs at the border or recommend disapproval of any new drug applications listing the establishment until sufficient corrections are made. FDA may take other regulatory actions if it identifies serious deficiencies during the inspection of a foreign establishment. For example, FDA may issue an import alert, which instructs FDA staff that they may detain drugs manufactured by the violative establishment that have been offered for entry into the United States. In addition, FDA may conduct regulatory meetings with the violative establishment. Regulatory meetings may be held in a variety of situations, such as a follow-up to the issuance of a warning letter to emphasize the significance of the deficiencies or to communicate documented deficiencies that do not warrant the issuance of a warning letter. Our preliminary analysis of FDA data shows that from fiscal year 2012 through fiscal year 2016, the number of FDA foreign drug manufacturing establishment inspections increased but then began to decline after fiscal year 2016 (see fig. 2). In fiscal year 2015, the total number of foreign inspections surpassed the number of domestic inspections. From fiscal year 2016 to 2018, both foreign and domestic inspections decreased—by about 10 percent and 13 percent, respectively. FDA officials attributed this decrease to vacancies in the number of investigators available to conduct inspections (which we discuss later in this testimony statement) and to inaccurate data used to select establishments for inspection in fiscal years 2017 and 2018. Despite steps taken to improve the accuracy and completeness of FDA data on foreign establishments, data challenges we identified in our 2008 report continue to make it difficult for FDA to accurately identify establishments subject to inspection. Specifically, since 2017, FDA has pursued an initiative to inspect approximately 1,000 foreign establishments that lacked an inspection history and, as of November 2019, officials said all of these establishments had either been inspected or were determined to not be subject to inspection. However, officials told us that this effort contributed to the decline in the number of foreign inspections conducted because of how data inaccuracies affected the process for selecting establishments for inspection. Specifically, after selecting uninspected foreign establishments for inspection, FDA determined that a sizeable percentage of these establishments were not actually subject to inspection (e.g., about 40 percent of those assigned to the China Office in fiscal years 2017 and 2018). These foreign establishments were thus removed from the list for inspection for the given year. FDA officials told us that the next highest priority establishments identified through the risk- based model to replace those establishments were domestic. As a result, the number of foreign establishments actually inspected decreased. As part of our ongoing work, we plan to examine the accuracy and completeness of information FDA maintains about foreign establishments and the application of its risk-based site selection process. FDA continues to conduct the largest number of foreign inspections in India and China, with inspections in these two countries representing about 40 percent of all foreign drug inspections from fiscal year 2016 (when we last reported on this issue) through 2018. (See table 2.) In addition to India and China, the rest of the countries in which FDA most frequently conducted inspections has generally been the same since our 2008 report. Our preliminary analysis of FDA data shows that each year from fiscal year 2012 through 2018, at least 50 percent of FDA’s foreign inspections were surveillance inspections. In contrast to preapproval inspections, surveillance inspections are used to ensure drugs already on the market are manufactured in compliance with FDA regulations. In recent years, the proportion of foreign surveillance inspections has increased. As figure 3 shows, in fiscal year 2012, 56 percent of foreign inspections were surveillance-only inspections; in contrast, from fiscal year 2016 through 2018, about 70 percent of foreign inspections were surveillance-only, which was comparable to the percentage for domestic inspections during that period. This is a significant increase from the 13 percent of foreign inspections that were surveillance-only when we made our 2008 recommendation that FDA inspect foreign establishments at a comparable frequency to their domestic counterparts (85 percent of which were surveillance-only at that time). FDA has implemented changes to its foreign drug inspection program since our 2008 report that may have contributed to the increase in surveillance inspections. Prior to 2012, FDA was required to inspect domestic establishments that manufacture drugs marketed in the United States every 2 years, but there was no similar requirement for foreign establishments. As a result, and as we reported in 2008, foreign inspections were often preapproval inspections driven by pending applications for new drugs. FDA thus conducted relatively few surveillance-only inspections to monitor the ongoing compliance of establishments manufacturing drugs that were already on the market, with just 13 percent of foreign inspections conducted for surveillance purposes at the time of our 2008 report. However, in 2012, the Food and Drug Administration Safety and Innovation Act eliminated the 2-year requirement for domestic inspections, directing FDA to inspect both domestic and foreign establishments on a risk-based schedule determined by an establishment’s known safety risks, which was consistent with our 2008 recommendation. Our preliminary analysis of FDA data shows that from fiscal year 2012 through 2018, FDA identified deficiencies in approximately 64 percent of foreign drug manufacturing establishment inspections (3,742 of 5,844 inspections). This includes deficiencies necessitating a classification of VAI or the more serious OAI, as described in the text box. Inspection Classifications Based on their inspection findings, FDA investigators make an initial recommendation regarding the classification of each inspection: No action indicated (NAI) means that insignificant or no deficiencies were identified during the inspection. Voluntary action indicated (VAI) means that deficiencies were identified during the inspection, but the agency is not prepared to take regulatory action, so any corrective actions are left to the establishment to take voluntarily. Official action indicated (OAI) means that serious deficiencies were found that warrant regulatory action. About 59 percent of domestic inspections (3,702 out of 6,291) identified deficiencies during this time period. (See fig. 4.) This proportion is similar to what we found when we last looked at this issue in 2008, when FDA identified deficiencies in about 62 percent of foreign inspections and 51 percent of domestic inspections from fiscal years 2002 through 2006. Our preliminary analysis showed that serious deficiencies identified during foreign drug inspections classified as OAI—which represented 8 percent of inspections from fiscal year 2012 through 2018—include CGMP violations such as those related to production and process controls, equipment, records and reports, and buildings and facilities. For example: Failure to maintain the sanitation of the buildings used in the manufacturing processing, packing, or holding of a drug product (21 C.F.R. § 211.56(a) (2019)). At an establishment in India producing finished drug products, the investigator reported observing a live moth floating in raw material used in the drug production, and that the facility staff continued to manufacture the drug products using the raw material contaminated by the moth, despite the investigator pointing out its presence. Failure to perform operations relating to the manufacture, processing, and packing of penicillin in facilities separate from those used for other drug products (21 C.F.R. § 211.42 (d) (2019)). At an establishment in Turkey that manufactured penicillin and other drugs, the investigator reported that the manufacturer had detected penicillin outside the penicillin manufacturing area of the establishment multiple times. According to FDA, penicillin contamination of other drugs presents great risk to patient safety, including potential anaphylaxis (even at extremely low levels of exposure) and death. The identification of serious deficiencies is not unique to foreign inspections. For example, at a domestic establishment producing finished drug products, the investigator observed brown stains, white residues, and brown stagnant water in manufacturing equipment. Some investigators who conduct foreign inspections expressed concern with instances in which ORA or CDER reviewers reclassify the investigator’s initial inspection classification recommendations of OAI to the less serious classification of VAI. We plan to examine this issue as part of our ongoing work. Our ongoing work showed FDA’s foreign inspection workforce has staff vacancies, which FDA officials said contributed to the recent decline in inspections. As previously mentioned, FDA uses multiple types of staff resources to conduct foreign drug inspections—including ORA investigators based in the United States, members of ORA’s dedicated foreign drug cadre based in the United States, and investigators assigned to FDA’s foreign offices. However, each of these groups has current vacancies. According to FDA officials, the agency is trying to fill vacancies in each of these groups, but the lower staff numbers may limit FDA’s ability to conduct more foreign inspections. ORA investigators based in the United States. This group of investigators conducts the majority of foreign inspections; about 76 percent of foreign inspections in fiscal year 2018 involved an ORA investigator based in the United States who conducts both foreign and domestic inspections. FDA officials said that the more experienced investigators from this group are expected to conduct three to six foreign inspections per year, and investigators hired using generic drug user fees are expected to inspect nine to 12 foreign establishments per year. As of June 2019, there were 190 investigators eligible to conduct foreign drug inspections, but officials said that as of November 2019, the agency had an additional 58 vacancies in this group. Officials said that the agency was in the process of hiring 26 ORA investigators based in the United States to fill these vacancies, with 32 vacancies remaining. FDA officials attributed the vacancies to multiple factors: investigator retirements, investigator movement to other parts of FDA, and the need to hire to additional investigator positions using generic drug user fees. Officials also said that a reorganization within ORA led to a reduced number of investigators who conduct drug manufacturing establishment inspections. While FDA recently filled several of the vacancies, officials told us that new investigators are not typically used for foreign inspections until they have been with the agency for 2 to 3 years. ORA dedicated foreign drug cadre. About 15 percent of foreign inspections in fiscal year 2018 involved an investigator from ORA’s dedicated foreign drug cadre—a group of ORA investigators based in the United States who exclusively conduct foreign inspections. FDA officials said that members of the cadre are expected to conduct 16 to 18 foreign inspections each year. According to FDA, the cadre had 20 investigators in 2012 and 15 investigators in 2016. However, the cadre had only 12 investigators as of November 2019, out of 20 available slots. According to FDA officials, the agency is attempting to fill these positions from the current ORA investigator pool, but officials are not confident that all 20 slots will be filled. Investigators assigned to FDA’s foreign offices. Approximately 7 percent of foreign inspections in fiscal year 2018 involved investigators from FDA’s foreign offices. The investigators conducting these inspections are those based in the China and India foreign offices—the countries where most drug inspections occur— and also include those on temporary duty assignment to these offices. According to FDA officials, these investigators are expected to conduct 15 foreign inspections each year. We have noted high vacancy rates for these foreign offices in past reports. While these vacancy rates have decreased over time, vacancies persist. As of November 2019, FDA’s China office had three of 10 drug investigator positions vacant (a 30 percent vacancy rate), while FDA’s India office had two of six drug investigator positions vacant (a 33 percent vacancy rate). FDA has taken steps to address vacancies in the foreign offices, but continues to face challenges. In our 2010 report, we recommended that FDA develop a strategic workforce plan to help recruit and retain foreign office staff. FDA released such a plan in March 2016, but the long- standing vacancies in the foreign offices raise questions about its implementation. FDA officials told us that one challenge in recruiting investigators for the foreign offices is that well-qualified investigators for those positions need foreign inspection experience. For example, an official in FDA’s India office told us that foreign inspections can be challenging and the India office does not have the resources to develop or train new investigators. Therefore, it is important to recruit investigators who have experience conducting foreign inspections, and such investigators are recruited from ORA. Thus, vacancies in the other two groups of investigators can influence the number of staff available to apply for positions in the foreign offices. Further, according to FDA officials, after employees have accepted an in-country position, the agency can experience significant delays before they are staffed in the office due to delays in processing assignments. For example, an official in FDA’s India office said that investigators need to complete a week-long security training program and must obtain the security clearance needed to work at the U.S. Embassy, which is where FDA’s foreign office is located. However, the official told us that there are limited availabilities for that training and background checks for security clearances can take time. According to this official, FDA investigators do not usually receive first priority for the training. FDA estimates that it can take as little as 1 year to over 2 years to clear background and medical checks and arrive at a foreign office. For example, an investigator in FDA’s China office told us that as a result of these requirements and other issues, it took nearly 2 years for the investigator to arrive at the office after FDA had accepted the investigator’s application. According to FDA’s own strategic workforce plan for the foreign offices, these types of delays have resulted in staff changing their decision after accepting a position in the foreign offices. Our preliminary analysis indicates that FDA continues to face unique challenges when inspecting foreign drug establishments—as compared to domestic establishments—that raise questions about the equivalence of these inspections. Specifically, based on our interviews with drug investigators in the dedicated foreign drug cadre and FDA’s foreign offices in China and India, we identified four challenge areas related to conducting foreign inspections, which are described below. Of the four challenge areas identified, three areas—preannouncing inspections, language barriers, and lack of flexibility—were also raised in our 2008 report. Preannouncing Inspections. As we reported in 2008, the amount of notice FDA generally gives to foreign drug establishments in advance of an inspection is different than for domestic establishments. Domestic drug establishment inspections are almost always unannounced, whereas foreign establishments generally receive advance notice of an FDA inspection. According to FDA officials, FDA is not required to preannounce foreign inspections. However, they said the agency generally does so to avoid wasting agency resources, obtain the establishment’s assistance to make travel arrangements, and ensure the safety of investigators when traveling in country. FDA does conduct some unannounced foreign inspections, particularly if the investigators conducting the inspection are based in FDA’s foreign offices. However, FDA officials told us that FDA does not have data on the frequency with which foreign drug inspections are unannounced, nor the extent to which the amount of notice provided to foreign establishments varies. According to FDA officials, this is because FDA does not have a data field in its database to systematically track this information. However, the officials estimated that the agency generally gives 12 weeks of notice to establishments that investigators are coming when investigators are traveling from the United States. While investigators in FDA’s China and India offices do conduct unannounced or short-notice inspections, these staff do not perform most of the inspections in these countries. (See table 3). Our preliminary work indicates that preannouncing foreign inspections can create challenges and raises questions about the equivalence to domestic inspections. Of the 18 investigators we interviewed, 14 said that there are downsides to preannouncing foreign inspections, particularly that providing advance notice gives foreign establishments the opportunity to fix problems before the investigator arrives. For example, when an inspection is preannounced, it gives establishments time to clean up their facility and update or generate new operating procedures. However, establishments are expected to be in a constant state of compliance and always ready for an FDA inspection, and several investigators told us seeing the true day-to-day operating environment for an establishment is more likely during an unannounced inspection. Of the 18 investigators we interviewed, 12 said that unannounced inspections are generally preferable to preannounced inspections. One investigator told us that, although they believe the best way to ensure industry compliance to CGMPs is for establishments to not know when FDA is coming for an inspection, there is no data that would allow the agency to evaluate whether unannounced inspections are better than preannounced inspections. In addition, some investigators told us that it is still possible to identify serious deficiencies during preannounced inspections. For example, investigators can still identify issues by looking at the firm’s electronic records, including time-stamped data relating to the creation, modification, or deletion of a record. Three investigators also told us that in some cases there can be benefits to announcing inspections in advance. For example, for preapproval inspections, announcing the inspection in advance gives the establishment time to organize the documentation and staff needed to conduct the inspection. Language Barriers. Our preliminary work indicates that language barriers—which we first reported as a challenge to conducting foreign inspections in our 2008 report—can add time to inspections and raise questions about the accuracy of information FDA investigators collect and thus about the equivalence to domestic inspections. FDA generally does not send translators on inspections in foreign countries. Rather, investigators rely on the drug establishment to provide translation services, which can be an English-speaking employee of the establishment being inspected, an external translator hired by the establishment, or an English-speaking consultant hired by the establishment. Of the 18 investigators that we interviewed, 14 said that language barriers can be a challenge to conducting foreign inspections and were especially challenging in parts of Asia, including China and Japan. Seven investigators told us this is less of a challenge for inspections conducted in other foreign countries, including India and countries in Europe, because workers at establishments in these countries are more likely to speak English, and documentation is also more likely to be in English. Investigators told us that compared to domestic inspections, it can be more challenging and take longer to complete typical inspection-related activities, such as reviewing documentation or interviewing employees, if the investigator needs to rely on translation. Fourteen of the 18 investigators we interviewed said that there can be concerns related to relying on establishment staff and independent translators. Specifically, 11 investigators told us there can be uncertainties regarding the accuracy of the information being translated, particularly when investigators rely on the translation provided by an employee of the establishment being inspected. For instance, one investigator said that there is more risk of conflict of interest if the establishment uses its own employees to translate. Another investigator said that they went to a drug establishment in China that told FDA it had English-speaking employees to translate the inspection, but that this was not the case, and the investigator had to use an application on their phone to translate the interviews. In addition, the firm representative providing the translation may be someone that does not have the technical language needed, which can make it harder to communicate with firm staff and facilitate the inspection. One investigator told us that the independent translators hired by firms are sometimes consultants and, in those instances, it can seem like the consultants are coaching the firm during the inspection. FDA officials told us that when they conduct unannounced for-cause inspections in China, investigators bring locally employed staff who work in FDA’s China office to act as translators. The investigators we interviewed said that in such instances, they valued knowing that the translation they were getting was accurate. However, FDA does not have the resources to provide locally employed staff on every inspection, according to an FDA official. We will continue to examine this issue with FDA as part of our ongoing work. Lack of Flexibility. Our preliminary work indicates that, as we first reported in 2008, the overseas travel schedule can present unique challenges for FDA’s domestically based investigators—including both ORA investigators and members of the dedicated foreign dug cadre— who conduct the majority of foreign inspections. Eight of the 12 dedicated foreign drug cadre investigators that we interviewed told us that there is little flexibility to extend foreign inspections conducted by domestically based investigators because the inspections they conduct on an overseas trip are scheduled back-to-back in 3-week trips that may involve three different countries. This raises questions about their equivalence to domestic inspections. For instance, extending one inspection would limit the amount of time the investigator has to complete their other scheduled inspections, some investigators told us. In addition, eight investigators told us that domestically based staff are generally unable to extend the total amount of time spent on an overseas trip—one investigator told us that an investigator would have to find something really bad to justify an extension. In contrast, FDA officials told us that inspections conducted by in-country investigators in China or India, and domestic inspections in the United States, are generally scheduled one at a time and can thus more easily be extended if the investigator needs additional time to pursue potential deficiencies. However, in-country investigators are not involved in the majority of inspections conducted in China or India. Three investigators from the dedicated foreign drug cadre told us that when they travel overseas, they adjust their inspection approach to help ensure they finish foreign inspections on time. For example, one investigator told us an investigator may start the inspection in an area of the establishment that was noted as having issues during the last inspection. However, one investigator said that sometimes it is not possible to cover everything in depth during a foreign inspection. Another investigator told us that they focus on identifying the most serious issues during a foreign inspection, and that less serious issues can be identified in the establishment inspection report for reference in the next inspection. Five investigators also noted that they work long hours during their inspection to ensure they can complete the needed work. While FDA may assign more than one investigator to an inspection to complete needed work, one investigator said that FDA does not usually assign more than one person to an inspection because investigators are expected to have the experience to conduct inspections by themselves. From fiscal years 2012 to 2018, the majority of both foreign and domestic inspections were conducted by one person—77 percent and 66 percent, respectively. Post-Inspection Classification Process. According to FDA officials, starting in fiscal year 2018, FDA implemented a new post-inspection classification process: when an ORA investigator recommends an OAI classification following an inspection, ORA compliance is required to send that inspection report to CDER for review within 45 calendar days from the inspection closeout. Among other things, the process was intended to help ensure FDA can communicate inspection results to domestic and foreign establishments within 90 days of the inspection closeout, as committed to under the Generic Drug User Fee Amendments of 2017(GDUFA II). FDA officials told us that the changes also required an additional ORA review for foreign inspection reports to align that process with the process for domestic inspection reports. Although the 45-day reporting time frame for potential OAI classifications is a requirement for both domestic and foreign inspections, adding the additional level of review within ORA effectively shortened the amount of time investigators have to document findings for foreign inspections. Our preliminary work indicates that the post-inspection reporting time frames can create challenges for domestic investigators that conduct foreign inspections and raise questions about the equivalence to domestic inspections. Eight of the 18 investigators that we interviewed said shortening the time for completing reports and adding a level of review has made it more challenging to meet reporting requirements, especially if serious deficiencies are identified during the inspection. Investigators told us that for a potential OAI inspection, they now need to send the inspection report to their supervisor for endorsement within 10 days of the closeout of a foreign inspection, regardless of when the investigator’s next inspection is scheduled for, or whether the investigator has to travel from overseas back to the United States after the inspection. For example, if a domestic investigator finds serious deficiencies on the first inspection, thus indicating an initial OAI classification, the investigator needs to write and send the related inspection report to the ORA supervisor for endorsement before returning home from the 3-week overseas trip to meet the required time frame. One investigator told us that, as a result of the time pressures, post-inspection reports may be less thorough, and that some inspection observations could be better supported if investigators had more time to write the reports. In conclusion, foreign manufacturing establishments continue to be a critical source of drugs for millions of Americans, and FDA inspections are a key tool to ensure the quality of these drugs. Over the years since we first examined this issue, FDA has made significant changes to adapt to the globalization of the drug supply chain and has greatly increased the number of inspections that it conducts of foreign establishments. Notably, it has markedly increased the percentage of foreign inspections conducted to monitor drugs already on the market, which we previously noted were vital to FDA oversight of foreign establishments. However, the agency continues to be faced with many of the same challenges in the oversight of foreign establishments that we identified in our 2008 report. Our preliminary work has identified inspection decreases, related in part to FDA challenges filling investigator vacancies. We have also identified a variety of unique challenges that investigators face in foreign inspections. As we continue to conduct our work, we will further examine the cumulative effect of these challenges that raise questions about FDA’s ability to conduct equivalent inspections in foreign establishments. We will examine the extent to which FDA has assessed its oversight of drugs manufactured overseas and the steps it is taking to mitigate any risks, and make recommendations as appropriate. Chair DeGette, Ranking Member Guthrie, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Mary Denigan-Macauley, Director, Health Care at (202) 512-7114 or DeniganMacauleyM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are William Hadley (Assistant Director); John Lalomio (Analyst-in- Charge); Katherine L. Amoroso; George Bogart; Zhi Boon; Derry Henrick; Laurie Pachter; and Vikki Porter. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "More than 60 percent of establishments manufacturing drugs for the U.S. market were located overseas in fiscal year 2018. FDA has estimated that about 40 percent of finished drugs and 80 percent of active drug ingredients are manufactured overseas. FDA is responsible for overseeing the safety and effectiveness of all drugs marketed in the United States, regardless of where they are produced and conducts inspections of both foreign and domestic drug manufacturing establishments. GAO has had long-standing concerns about FDA's ability to oversee the increasingly global supply chain, an issue highlighted in GAO's High Risk Series for the last 10 years. GAO recommended in 2008 (GAO-08-970) that FDA increase the number of inspections of foreign drug establishments. GAO found in 2010 (GAO-10-961) and 2016 (GAO-17-143) that FDA was conducting more of these foreign drug inspections, but GAO also reported that FDA may have never inspected many establishments manufacturing drugs for the U.S. market. This statement is based on ongoing work and provides preliminary GAO observations on 1) the number of foreign inspections FDA has conducted, 2) inspection staffing levels, and 3) challenges unique to foreign inspections. For this work, GAO examined FDA data, visited FDA foreign offices in China and India, and interviewed drug investigators based in these offices and in the United States. GAO's preliminary analysis of Food and Drug Administration (FDA) data shows that from fiscal year 2012 through 2016, the number of foreign drug manufacturing establishment inspections increased. From fiscal year 2016 through 2018, both foreign and domestic inspections decreased—by about 10 percent and 13 percent, respectively. Howevever, the total number of foreign inspections surpassed the number of domestic inspections in 2015, and a growing percentage of FDA's foreign inspections (43 percent in 2018) were conducted in China and India, where most establishments that ship drugs to the United States were located. FDA officials attributed the decline, in part, to vacancies among investigators available to conduct inspections. GAO previously noted the vital role that inspections play in FDA's oversight of foreign establishments. FDA has vacancies among each of the groups of investigators who conduct foreign inspections. FDA had 190 investigators in the United States who conduct the majority of foreign inspections, but an additional 58 positions were vacant. FDA was in the process of filling 26 of these vacancies, with 32 remaining. However, according to FDA officials, it could be 2 to 3 years before new staff are experienced enough to conduct foreign inspections. FDA also faces persistent vacancies among investigators in its foreign offices. FDA investigators identified persistent challenges conducting foreign inspections, raising questions about the equivalence of foreign to domestic inspections. For example, while domestic inspections are almost always unannounced, FDA's practice of preannouncing foreign inspections up to 12 weeks in advance may give manufacturers the opportunity to fix problems. Investigators from FDA's China and India offices do conduct some unannounced inspections, but they are involved in a small percentage of inspections in these countries (27 percent and 10 percent, respectively). Further, FDA continues to rely on translators provided by the foreign establishments being inspected, which investigators said can raise questions about the accuracy of information FDA investigators collect. GAO will continue to assess these issues as part of ongoing work, and make recommendations as appropriate.", "document_type": "gao"}
{"report": "A total of 43 states operated at least part of their Medicaid programs under demonstrations, as of November 2018. State demonstrations can vary in size and scope, and many are comprehensive in nature, affecting multiple aspects of a state’s Medicaid program. Nationally, federal spending under demonstrations represented over 30 percent of all federal Medicaid spending in fiscal year 2016. (See app. II.) Demonstrations are typically approved by CMS for an initial 5-year period, but some states have operated portions of their Medicaid programs under a demonstration for decades. This can be achieved by a state requesting approval by CMS for one or more 3- to 5-year extensions of an existing demonstration (referred to as an extension). States often make changes to their demonstrations, either through the extension process or by requesting to amend a demonstration during the approval period (referred to as an amendment). From January 2017 through May 2018, CMS approved applications for a new demonstration, extension, amendment, or a combination of these in 23 states. (See fig. 1.) Each demonstration is governed by special terms and conditions (STCs), which reflect the agreement reached between CMS and the state, and describe the parameters of the authority granted to the state. For example, the STCs may define for what populations and services funds can be spent under the demonstration, as well as specify various state reporting requirements. The STCs also include a spending limit for the demonstration that is meant to ensure the demonstration is budget neutral to the federal government; that is, the federal government should spend no more under a state’s demonstration than it would have spent without the demonstration. Requirements for new demonstrations and extensions. As required under PPACA, HHS issued regulations in 2012 to address transparency in the approval of applications for new demonstrations and extensions. The regulations include requirements for states to seek public input on their proposals prior to submitting an application to CMS, requirements for information states must include in their public notices and applications, and procedures that CMS would follow upon receiving the application. CMS reviews the submitted application to check for compliance with these regulations, before seeking additional public input through a 30-day comment period at the federal level. (See fig. 2.) The regulations also provide CMS discretion to engage in additional transparency activities on a case-by-case basis. Requirements for amendments. The 2012 regulations do not apply to states seeking to amend existing demonstrations. Instead, the transparency requirements for amendments are set by guidance HHS issued in 1994 and in the individual STCs that govern each demonstration. The requirements from the guidance and STCs include, for example, that the state seek public input prior to submitting its application and provide in its application an explanation of its process for notifying the public and a detailed description of what is being amended, including the impact on beneficiaries. CMS’s regulations also include monitoring and evaluation requirements to ensure that the outcomes of demonstrations are transparent. Monitoring. States must perform periodic reviews of the implementation of their demonstrations, and the STCs typically require states to report those outcomes to CMS periodically. The regulations also require states to conduct a public forum within 6 months after the implementation date of the demonstration, and annually thereafter, to solicit public comments on the progress of the demonstration project and summarize issues raised in monitoring reports submitted to CMS. The regulations require that states submit the annual monitoring reports to CMS. Evaluation. States are required to conduct evaluations to assess whether their demonstrations are achieving the state’s goals and objectives. After a demonstration is approved, states are required to submit an evaluation design to CMS for review and approval. The evaluation design must discuss the hypotheses that will be tested, the data that will be used, and other items outlined in the STCs. In the event that a state wishes to extend its demonstration, the state’s extension application must include, among other things, a report presenting the evaluation’s findings to date, referred to as an interim evaluation report. States are also required to submit final evaluation reports at the demonstration’s end. All evaluation designs and reports are to be made public. Transparency Requirements for New Demonstrations and Extensions The transparency requirements for new Medicaid section 1115 demonstrations and extensions of existing demonstrations are established in federal regulations at 42 C.F.R. part 431, subpart G. We found that CMS has developed procedures for assessing states’ applications for new demonstrations and extensions against the transparency requirements established in 2012 (see sidebar). Specifically, CMS’s procedures involve reviewing incoming applications for new demonstrations or extensions against detailed checklists the agency designed to align with transparency requirements in the regulations. (CMS refers to these as completeness checks.) For example, the checklist for new demonstrations includes checks for whether the application included a description of the demonstration; any proposed changes to the benefits, delivery system, or eligibility requirements; information on the public hearing(s) and public comment process the state conducted; and a summary of the issues raised in the state public comment process. (See fig. 3.) We found that CMS completed checklist reviews for each of the 11 applications for new demonstrations or extensions that CMS approved from January 2017 through May 2018. CMS has also developed and implemented procedures for seeking public input at the federal level and making that input publicly available. This includes CMS sending email notifications to individuals who have registered on the agency’s website when demonstration applications are open for public comment; posting the application on the website where the public can post comments during the 30-day comment period; and maintaining the public comments on the website, which are maintained indefinitely, according to CMS officials. We found that CMS conducted a federal comment period for all 11 of the new and extension applications in our review period. In addition to storing the federal public comments, CMS’s website contains a record of key decisions and documents for each demonstration (referred to as the administrative record). The administrative record includes states’ applications, as well as CMS’s approvals, denials, and decisions about the completeness of applications—a requirement under the 2012 regulations. CMS officials told us that they include additional documents as standard practice, though they are not required to be posted, such as a fact sheet on the demonstration and other official communication between the agency and the state, to support transparency. CMS first launched this section of its website in December 2011 with an aim to improve access to Medicaid program information, including information on demonstrations, and redesigned the website in 2013 to improve functionality. The administrative record provides a history, dating as far back as 2011, of what a state has tested, how the approach has evolved over time, and what has been learned from the approach. We identified several areas of weakness in CMS’s policies or procedures for ensuring transparency in approvals of new demonstrations and extensions of existing demonstrations. These weaknesses related to the transparency of major changes made to pending applications, the transparency of changes to approved spending limits, and inconsistency in CMS’s review of applications for compliance with transparency requirements for new demonstrations and extensions. CMS did not apply a consistent approach to ensuring transparency in two states that made major changes to their demonstration applications mid- review. Indiana and Kentucky submitted changes to pending applications, the first for an extension and the latter for a new demonstration that had substantial potential effects for some beneficiaries. Indiana’s changes included adding new eligibility requirements for some beneficiaries, and Kentucky’s changes included accelerating the effective dates of new requirements to maintain eligibility (see sidebar). CMS did not require either state to solicit public input, though both states opted to hold a public comment period on the proposed changes concurrent with CMS’s review. Further, CMS reviewed Indiana’s proposed changes against limited transparency requirements but did not do so for Kentucky. Indiana submitted a final version of its application summarizing public input and the state’s response, while Kentucky did not. Thus, the extent to which these comments were considered at the state and federal levels was not transparent to the public. Figure 4 shows a timeline of the events surrounding Indiana’s and Kentucky’s requests to make changes to their pending demonstration applications. Kentucky: In July 2017, Kentucky submitted changes to its pending application for a new demonstration, including: replacing a provision for a year-long phase-in of a proposed 20-hour per week work and community engagement requirement for beneficiaries with a 3- month phase-in of the requirement; and disenrolled for 6 months for failing to timely report changes in income or other circumstances affecting eligibility. CMS approved a significant increase in the spending limit for a portion of Florida’s demonstration—which appeared to reflect a change in the agency’s position on the allowable use of the funds—without making transparent the basis for this decision. Specifically, CMS increased the spending limit for a pool of funds for payments to offset providers’ uncompensated care costs by close to $1 billion in 2017 after having reduced the limit 2 years earlier. In its approval letter, CMS provided limited information on the basis for this change. CMS stated that the limit was based on the state’s most recent data on uncompensated care costs, but did not disclose a significant change in its methodology for setting these limits. In unpublished correspondence to Congress, CMS indicated that the calculation of the spending limit was broadly consistent with previous policy with one significant change. Specifically, the letter indicated that whether the state had opted to expand Medicaid coverage to low-income, childless adults as provided for under PPACA would no longer factor into the limit, thus allowing CMS to include uncompensated care costs for this population in setting the limit. This change led to increasing the state’s spending limit to $1.5 billion annually. (See text box.) Moreover, CMS noted plans to apply this change across all states going forward. CMS officials, however, did not indicate that they had publicly communicated this policy change to all states. In past reports, we have recommended that HHS make public the basis for demonstration approvals including the basis for elements used to set spending limits, and in 2008, we raised the issue as a matter for Congress to consider. CMS has taken a number of steps in the last several years to update and make public its policies for setting spending limits, but has not yet taken action to make public the basis of spending limits. CMS Decision to Increase Spending Limit for a Funding Pool in Florida’s Medicaid Section 1115 Demonstration In letters to Florida, CMS wrote: April 14, 2015: “…over time, CMS has had a number of concerns about the LIP , including its lack of transparency, encouragement toward overreliance on supplemental payments, and distribution of funds based on providers’ access to local revenue instead of service to Medicaid patients. Last year, CMS made clear that LIP would not continue in its current form….We will approach review of a LIP proposal from Florida based on several key principles. First, coverage rather than uncompensated care pools is the best way to secure affordable access to health care for low-income individuals, and uncompensated care pool funding should not pay for costs that would be covered in a Medicaid expansion.…” October 15, 2015: “…Pursuant to our June 23, 2015 agreement in principle, establishing the size, duration, and distribution methodology for the Low Income Pool (LIP)…The total computable dollar limit in demonstration year 10 (2015-2016) will be $1 billion. In demonstration year 11 (2016-2017) the total computable dollar limit will be $607,825,452…” August 3, 2017: “For the extension, CMS has established an amount for the low-income pool’s (LIP) uncompensated care pool to be approximately $1.5 billion annually, based on the most recent available data on hospitals’ charity care costs.” Finally, we observed some inconsistencies in CMS’s reviews of states’ applications for their compliance with the transparency requirements for new demonstrations and extensions. Expected changes in enrollment were not always included in state public notices. In two of the four applications for new demonstrations and extensions for which we conducted in-depth reviews (Florida’s extension and Washington’s new demonstration), estimates for the expected increase or decrease in enrollment were not included in the state’s public notice documents as required. CMS officials told us that they are revising procedures to resolve such inconsistencies, including making additions to written standard operating procedures. Evaluation information was not always included in state applications. Although states seeking extensions are required to submit an interim evaluation report, Florida only included a statement in its application that it had recently executed an evaluation contract and had no findings to report. According to CMS, Florida’s evaluation design was not approved until weeks before the extension application was due. Despite not having information on whether Florida’s demonstration was meeting its goals, CMS officials considered the state’s application complete, stating that Florida had met the intent of the regulation by providing its findings to date. In 2018, we reported that there were limitations in state evaluations of demonstrations, in part, due to how CMS sets requirements for evaluations, and we made a recommendation to improve CMS’s procedures. In line with our recommendation, CMS has since developed an enhanced set of STCs that specify when evaluation reports are due, and reported in November 2018 that it is in the process of developing protocols to ensure that these requirements are consistently included in the STCs. Transparency Requirements for Amendments Applications for amendments to Medicaid section 1115 demonstrations are subject to requirements for seeking public input outlined in guidance the Department of Health and Human Services (HHS) issued in 1994 and those included in the special terms and conditions governing each demonstration. CMS applies limited transparency requirements to states’ applications to amend existing demonstrations, despite the fact that states may propose significant changes to demonstrations through amendments (see sidebar). CMS does not place limits on what changes can be made through amendments. From January 2017 through May 2018, CMS approved 21 amendments in 17 states, and we found that at least 17 amendment applications were pending CMS approval as of January 2019. These 17 states made a wide range of changes to their demonstrations through amendments. For example, one state amended its demonstration to cover dental services for adults with disabilities, while other amendments included such changes as requiring beneficiaries to work or participate in community engagement activities as a condition of maintaining Medicaid eligibility, as was done through amendments in Arkansas and New Hampshire during the period we reviewed. As it does with applications for new demonstrations and extensions, CMS reviews amendment applications by using a checklist, conducting a federal public comment period, and posting state demonstration documentation on the CMS website. However, the transparency requirements for amendments applied during the checklist review are more limited than the requirements for new demonstrations and extensions. (See fig. 5.) The transparency requirements for amendment applications are more limited than those for new demonstrations and extensions in the following key areas, potentially limiting CMS’s ability to ensure public transparency for the approvals of amendments. No requirement to hold a state public comment process or provide CMS a summary of public input received. For amendments, states have a range of options for seeking public input and, unlike for new and extension applications, states are not required to submit a summary of the public input received on their applications and how they responded. Instead, in amendment applications, states are only required to describe the process the state used to solicit public input. Among the three amendment approvals for which we conducted an in-depth review, California did not hold a formal public comment period and did not provide CMS information on any public input it received, neither of which is required under the transparency requirements for amendments. No requirement to include expected changes in enrollment and costs. In contrast with requirements for new demonstrations and extensions, CMS does not require states to include in amendment applications the expected increase or decrease in enrollment, and the amendment applications we reviewed included limited or no information on changes to enrollment. (See text box.) CMS also does not require information on expected changes in costs for all amendments, and we found variation in the information included in amendment applications, including limited or no information on costs. No minimum requirements for information to be included in the public notice. Unlike the transparency requirements for new demonstrations and extensions, there are no requirements specifying what information states must include in their public notices for amendments. For example, Arkansas did not include in its public notice information on the changes to enrollment estimated from any of the amendment provisions. In addition to the differences in the transparency requirements for amendment applications, we identified inconsistencies in how CMS applied the transparency requirements for amendment applications across states, particularly the requirements related to describing changes to the demonstration evaluations. For amendments, as is similar to extensions, states are required to describe how the demonstration’s evaluation will be revised to incorporate amendment provisions. The following are examples of the inconsistencies: CMS determined that Massachusetts’s amendment application, which proposed to waive non-emergency medical transport, eliminate provisional eligibility for most populations, and cover former foster care youth, was determined to be incomplete (that is, not in compliance with the transparency requirements), partially due to the state not submitting a revised evaluation design plan. In contrast, CMS determined that Arkansas’s application, which did not include a revised evaluation design plan, was complete. Arkansas noted in its application that the state planned to revise its evaluation to test two additional hypotheses. However, the added hypotheses did not address, for example, the waiver for retroactive eligibility proposed in the application. Among the 18 other demonstration amendment applications CMS approved during our review period, there was variation in the information the states included about the changes to the demonstration’s evaluation hypotheses or design. For example: Iowa submitted an amendment application, which CMS determined to be in compliance with transparency requirements, to waive retroactive eligibility for all beneficiaries, and said that it was not changing the evaluation design based on the amendment provisions. In at least two other states’ amendments—Florida and Utah—the applications, which CMS determined to be in compliance with transparency requirements, did not include any information on the changes to the evaluation due to the amendment or indicated that the state would be making changes at a later date. The potential effects of policy changes states make through amendments can be comparable to effects of new demonstrations and extensions. CMS has considered taking further steps to ensure transparency for amendments, but has not done so. Specifically, in both its response to comments in the 2012 final rule and a subsequent letter to state Medicaid directors in 2012, CMS indicated that the agency intended to evaluate the types of amendments submitted by states and issue further guidance on how the notice and comment provisions would be applied to amendments that have a significant impact. However, CMS did not issue such guidance and officials told us that they had no plans to do so. CMS officials told us that including standard requirements in demonstration STCs for submitting amendment applications helps improve the transparency of amendments. However, the standard requirements that CMS has included do not ensure that states provide information to the public or CMS on the effect of an amendment on enrollment and costs, key pieces of information for amendments that have and may continue to have a significant impact. According to federal standards for internal controls related to risk assessment, federal agencies should identify and manage risks related to achieving agency objectives. Without a policy with robust transparency requirements for amendment applications with significant impacts, there is the potential that states and CMS will fail to receive meaningful public input on the amendment and thereby lack complete understanding of the impact. As a result, CMS may not be positioned to mitigate any potential risks in the demonstrations being amended or when other states request to test similar policies in the future. CMS reviewed state descriptions of issues raised during the state public input process and the state’s response as part of its application review. Applications for six of the seven approvals for which we conducted in- depth reviews summarized themes from the comments that were received and included the states’ responses to these comments. State responses included laying out changes the state made to the proposal in response to the comments, clarifying certain aspects of the proposed demonstration, or providing justification for not making a change. However, the level of detail in state summaries of their responses to these comments varied considerably. For example: Washington application for new demonstration. Washington provided an extensive summary of the comments received, categorized by themes, along with the state’s responses to each of them. One commenter suggested a 1-year implementation period to ensure that sufficient planning and preparations were undertaken before the new demonstration officially went into effect. The state agreed that this was “essential to assure operational readiness and critical success of this demonstration,” and revised its proposal to include a 9-month implementation period. Florida application to extend demonstration. In contrast to Washington, Florida’s application to extend its managed care program provided a long list of state comments and nearly all were addressed with a standard response from the state that the comments were taken into consideration, but no changes to the existing STCs were being requested. These included concerns about access and choice under current pharmacy networks, and other access issues such as difficulties in obtaining referrals to specialists. Florida officials told us that they addressed stakeholder concerns through the state public comment process, which includes public forums, and that they are not necessarily required to provide any additional explanation in the state’s application to CMS. Florida officials also stated that some demonstration elements stem from state legislation, which limits their ability to make changes in response to comments. According to CMS officials, historically, the agency has not requested the full set of comments that are submitted to the states. None of the states that we reviewed attached the actual comments that were received to the application—only summaries—though some posted them on their website or had them available upon request, according to state officials. CMS officials told us the agency has not consistently requested that states provide the actual comments received; however, in the last year and a half the agency has been making more of an effort to request the full set of comments instead of solely relying on the summaries provided in the applications. Officials said they anticipate that this will be the agency’s standard practice going forward. In making demonstration decisions, CMS reviews and summarizes all input received during the federal comment period. CMS created a summary of comments received for all seven of the demonstrations we reviewed. Officials said that these summaries are used to brief CMS leadership as part of the decision-making process. We found that the level of detail in the summaries we reviewed varied, ranging from bulleted lists identifying and detailing common themes in support, opposition, or both, to a few brief sentences covering all comments. This variation often reflected the differences in the number of comments received and the significance of the concerns raised. For example: CMS received about 100 comments on Washington’s application during the federal comment period that were predominantly supportive of the new demonstration, and CMS’s summary was a brief overview. In contrast, CMS received thousands of comments on Kentucky’s application for each of the three federal comment periods held for that new demonstration, with many opposed to or concerned with certain aspects of the application. CMS’s summaries of comments received on the Kentucky application provided an overview of the issues raised, along with counts of how many fell within different themes and how many were in support, opposition, or unrelated to what was being proposed. CMS officials explained that there are unique circumstances surrounding each demonstration—a comment period with many concerns raised or conflicting viewpoints will necessitate a longer and more detailed summary than one that has broad support and few, if any, areas of disagreement. We found instances where CMS approved changes to certain aspects of the demonstrations that were in line with concerns raised by the comments. Among the seven demonstrations we reviewed in-depth, CMS received comments for four demonstrations that included concerns about the state’s proposal: Arkansas, Indiana, Kentucky, and Massachusetts. For Arkansas and Kentucky, CMS either approved more limited changes than what the state initially proposed or required that certain beneficiary protections be in place. Arkansas: In its amendment application, Arkansas requested a waiver of the requirement to provide retroactive eligibility, among other things. Commenters were concerned that the state’s proposal to eliminate retroactive eligibility would result in gaps in coverage, adverse health outcomes, and medical debt for members. In CMS’s approval, the agency acknowledged these concerns and allowed the state to reduce the period for retroactive eligibility from 90 days to 30 days but not eliminate it completely. (See fig. 6.) Kentucky: In Kentucky, some commenters were concerned about the state’s proposal to implement a work and community engagement requirement as a condition of Medicaid eligibility, noting that individuals need to be healthy to work or look for a job. CMS said in its January 2018 approval that Kentucky was exempting medically frail individuals from this requirement, but CMS would also be requiring that the state add certain protections for vulnerable individuals, including maintaining a system that identifies, validates, and provides reasonable accommodations. We also found there were instances where CMS approved certain aspects of the demonstrations despite concerns raised by the comments. CMS’s rationale for those decisions varied across demonstrations. For example, CMS noted in one instance that sufficient controls were planned to address the concerns raised, and in another instance noted that the potential benefits of the demonstrations outweighed the risks. The following are examples of when CMS approved aspects of states’ demonstrations without changes. Arkansas: In Arkansas, some commenters were opposed to the enforcement mechanism for the state’s proposal to institute a work and community engagement requirement as a condition of maintaining eligibility. The state proposed to disenroll beneficiaries who fail to fulfill these requirements for any 3 months during a calendar year and lock them out from coverage until the start of the next calendar year. CMS approved this proposal and provided an explanation of the circumstances under which it would happen, underscoring that individuals have three opportunities (each of the months they fail to fulfill the requirements) to rectify the situation or seek an exemption before they would ultimately lose coverage. CMS indicated in the approval letter to Arkansas that it believed the health benefits of community engagement outweigh the health risks with respect to those who fail to respond. Indiana: In Indiana, some commenters were opposed to the state’s proposal to institute a work and community engagement requirement as a condition of maintaining eligibility. They argued, in part, that beneficiaries who are able to work are already doing so and the requirement is unnecessarily burdensome. CMS responded that employment is positively correlated with health outcomes and imposing these requirements serves the purposes of the Medicaid statute. (See fig. 7.) In an effort to improve transparency around its approvals, CMS began providing a high-level summary and response to public comments in the demonstration approval letters beginning in January 2018. Agency officials said this will be their standard practice going forward. Our review of the approval letters sent between January 1, 2018, and July 31, 2018, confirmed that CMS included a discussion of some of the issues that were raised in 10 of 11 letters. For example, the approval letters explained the decision to reduce the period of retroactive eligibility in Arkansas instead of eliminating it completely, as well as the decision to approve Indiana’s proposal to implement work and community engagement requirements. However, the approval letters do not respond to every concern raised. For example, a number of commenters were concerned with a request in Arkansas’s amendment application to no longer offer presumptive eligibility, but CMS did not respond to these concerns in the approval letter. CMS officials told us that the agency is striking a balance between transparency and processing applications in a timely manner. Among the four demonstration approvals for which we conducted in-depth reviews and where public comments raised concerns—approvals for Arkansas, Indiana, Kentucky, and Massachusetts—we observed instances where CMS added specific monitoring requirements to the STCs that aligned with these concerns and other cases where the agency did not. For example: The STCs required Arkansas to submit a monitoring plan for its work and community engagement requirement in order to monitor eligibility operations and the impact on beneficiaries reapplying for coverage after being disenrolled for noncompliance. In contrast, CMS did not require a monitoring plan for the Indiana and Kentucky demonstrations, which also included work and community engagement requirements where the public raised concerns about the effects on beneficiaries. This remains the case for Indiana; however, CMS’s new approval of Kentucky’s demonstration in November 2018 included additional monitoring requirements. Specifically, the November 2018 STCs required Kentucky to submit a monitoring protocol that includes measures for monitoring enrollment, disenrollment, and eligibility suspension, among other things. In other cases where public comments raised concerns about the impact of demonstrations on beneficiaries, including changes in eligibility requirements (e.g., retroactive eligibility), we did not observe specific monitoring requirements included in the STCs. Though CMS did not provide any specific examples, officials told us that they consider public input when making decisions about monitoring requirements. Officials also said they were developing monitoring metrics and tools that they plan to use consistently going forward for states implementing work and community engagement requirements. As of January 2019, officials said these materials were in draft form and under review. Regarding evaluations, the extent to which CMS considered concerns raised through public comments for the four demonstration approvals was also not always clear, including how input informed the evaluation requirements in the STCs. For example, commenters on the applications submitted by Indiana and Kentucky raised concerns about aspects of the work and community engagement requirements proposed by each state, such as the requirements for reporting work or other activities and the circumstances under which beneficiaries would lose coverage. In the STCs for Indiana, CMS did not include specific hypotheses that the state would be required to test related to its work and community engagement policies. Instead, CMS noted that the state’s goals should inform the evaluation, subject to CMS approval. For example, Indiana’s goals included determining whether implementing work and community engagement requirements will lead to sustainable employment and improved health outcomes among beneficiaries. In the STCs for Kentucky’s initial approval in January 2018, CMS included the same language as in Indiana—that the goals should inform the state’s evaluation. However, in the STCs approved for Kentucky in November 2018, CMS added some broad guidance for Kentucky’s draft evaluation design. Specifically, CMS included a variety of hypotheses that the state must evaluate, such as the effect of work requirements on enrollment and the impact of the demonstration on uncompensated care costs. When approving evaluation designs, the extent to which CMS considers areas of risk identified through public input is also unclear at this time. As of January 2019, evaluation designs for the Arkansas and Indiana demonstrations were under review at CMS and Kentucky had not yet submitted one. Regarding Arkansas’s evaluation design, CMS sent a letter to the state providing comments and feedback that seem to align with some of the concerns raised about the demonstration through public input. Specifically, the November 2018 letter from CMS raised concerns with the state’s “broadly defined” expected outcomes of the demonstration, which included culture of work and personal life stability. CMS recommended that the state revise the design to include a list of quantifiable outcomes and measures that capture the important features, such as increased employment (e.g., hours worked, wages) and improved health (e.g., health care utilization). For Massachusetts, the one demonstration with an approved evaluation design, the extent to which CMS considered public input during approval was unclear. For example, with regard to Massachusetts’s proposal to discontinue provisional eligibility for most adults, commenters raised concerns about the potential effects on beneficiaries’ timely access to coverage and care; however, the evaluation design did not include plans to examine the effects of the policy on beneficiaries. Though CMS did not provide specific examples of how public input had informed evaluation designs, CMS officials said requirements for evaluations have been evolving as they have gained experience in understanding the public’s concerns. Officials also said they were developing robust evaluation guidance that they plan to use consistently going forward for states implementing work and community engagement requirements. As of January 2019, officials said this guidance was in draft form and under review. While CMS has long recognized the importance of public input in the demonstration approval process, the agency has developed more robust procedures for ensuring transparency since the beginning of 2012. Despite this progress, CMS’s approach to ensuring transparency when states propose major changes to their demonstrations has significant gaps. The lack of policies for ensuring transparency when states make major changes to pending applications and limited transparency requirements applied for amendments—which are being used by some states to make major changes to their demonstrations—puts CMS’s goal of transparency at risk. These gaps may leave the agency and the public without key information to fully understand the potential impact of the changes being proposed, including on beneficiaries and costs. These risks take on increased importance given that CMS is encouraging states to use the flexibility provided under demonstrations to test changes to their Medicaid programs that could have significant effects for beneficiaries and other stakeholders. We are making the following two recommendations to CMS: The Administrator of CMS should develop and communicate a policy that defines when changes to a pending section 1115 demonstration application are considered major and should prompt a new review of the application against the transparency requirements applicable to the pending application. (Recommendation 1) The Administrator of CMS should develop and communicate a policy whereby applications for section 1115 demonstration amendments that may have significant impact are subject to transparency requirements comparable to those for new demonstrations and extensions. (Recommendation 2) We provided a draft of this report to HHS for review and comment. HHS concurred with both recommendations. HHS’s comments are reproduced in appendix III. Regarding our first recommendation concerning when states submit major changes to pending demonstration applications, HHS stated that it will develop (1) standards for determining when such changes are so substantial that it would be appropriate for HHS to solicit additional public input, and (2) a process for informing states and the public about the additional comment period. These steps appear to formalize the approach CMS has already been taking as demonstrated by the agency’s response to the changes submitted by Indiana and Kentucky to their applications. Our recommendation, however, requires additional actions. In particular, we recommended that CMS develop and communicate a policy that includes standards for when changes are substantial enough to warrant a new review of the application against the transparency requirements. The transparency requirements, among other things, call for states to provide for public notice and input at the state level before they submit their applications. As such, holding an additional federal comment period would not be sufficient to meet our concerns. Regarding our second recommendation—concerning transparency requirements for amendment applications that may have significant impacts—HHS said that it has implemented enhanced processes to improve transparency and will review its current processes and develop additional policies and processes, as needed, to enhance the transparency of such applications. However, the enhanced processes HHS referred to do not apply to amendments. Thus, HHS’s planned review of its policies alone would not be sufficient to meet our concerns. HHS’s efforts should also result in actions to develop and communicate a policy that ensures amendments with significant impacts meet transparency requirements comparable to those for other applications, namely new demonstrations and extensions. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Health and Human Services, the appropriate congressional committees, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-7144 or yocomc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix IV. Total Medicaid expenditures (dollars in millions) Total Medicaid expenditures (dollars in millions) In addition to the contact named above, Susan Barnidge (Assistant Director), Linda McIver (Analyst-in-Charge), Michael Moran, and Jessica L. Preston made key contributions to this report. Also contributing were Drew Long, Vikki Porter, and Emily Wilson.", "summary": "Section 1115 demonstrations are a significant component of Medicaid spending and affect the care of millions of beneficiaries. The Patient Protection and Affordable Care Act required the Department of Health and Human Services (HHS) to establish procedures to ensure transparency in approvals of new demonstrations and extensions to existing demonstrations. The act did not address amendments, which are subject to long-standing guidance on public input. GAO was asked to examine the transparency of demonstration approvals. Among other things, this report examines CMS's transparency policies and procedures for new demonstrations and extensions, and amendments to existing demonstrations. To review a variety of approval types across a large number of states, GAO examined all approvals of new demonstrations and extensions of and amendments to existing demonstrations granted from January 2017 through May 2018. GAO also conducted in-depth reviews of one approval in each of seven states, selected to include at least two approvals of each type. GAO reviewed demonstration documentation for these states, and interviewed state and federal Medicaid officials. GAO also assessed CMS's procedures against federal internal control standards. Medicaid demonstrations allow states flexibility to test new approaches for providing coverage and delivering Medicaid services. Since 2012, the Centers for Medicare & Medicaid Services (CMS), which oversees demonstrations, has developed procedures to improve the transparency of the approval process. For example, CMS reviews demonstration applications (including for new demonstrations, extensions, and amendments to existing demonstrations) for their compliance with applicable transparency requirements, including that states seek public input on their applications. However, GAO found weaknesses in CMS's policies for ensuring transparency. Changes to pending applications for new demonstrations or extensions. CMS lacks policies for ensuring transparency when states submit major changes to pending applications. For two of the four approvals of new demonstrations or extensions GAO reviewed in-depth, states submitted changes to their applications that could have significant effects on beneficiaries (such as disenrollment or other penalties) without first obtaining public comment on these changes at the state level. Amendments to existing demonstrations. CMS's transparency requirements for amendments are limited. For example, CMS does not require amendment applications to include how the changes may affect beneficiary enrollment or report on concerns raised in state public comments. However, states have proposed major changes—such as work and community engagement requirements—through amendments, raising concerns that major changes to states' demonstrations are being approved without a complete understanding of their impact. CMS should develop policies for ensuring transparency when states (1) submit major changes to pending demonstration applications and (2) propose amendments to existing demonstrations. HHS concurred with these recommendations.", "document_type": "gao"}
{"report": "Water scarcity occurs when the demand for water in a given area approaches or exceeds available water supplies. In April 2016, we reported that drinkable water has traditionally been assumed to be reliable, cheap, and abundant. However, with parts of the United States—especially the Southwest—facing recurring drought and persistent water scarcity, that view has been challenged. Water is also not always available when and where it is needed, in the amount or quality desired, or in a cost-effective manner. In times of water scarcity, there are often competing demands for water—such as irrigation, power production, municipal water supplies, and supporting aquatic life. As we reported in May 2014, state water managers expect freshwater shortages to continue into the future. According to the United States Global Change Research Program’s Fourth National Climate Assessment, significant changes in water availability are evident across the country and are expected to persist in the future due to changes in precipitation and rising temperatures. For example, droughts occurring from deficits in precipitation, soil moisture, and snow runoff will likely occur more frequently. Further, since a warmer atmosphere holds more water, when rain does fall high-intensity events can occur more frequently. These sudden downpours will increase the mobility of pollutants, such as sediments and nutrients, and of algae, which can reduce the quality and quantity of available drinking water. The assessment noted that in some regions of the United States, the supplies of water are already stressed by increasing consumption, and continued warming will add to this stress, adversely affecting the availability of water in parts of the United States and increasing the risk of water scarcity. The military departments rely on water at installations to conduct and support their missions. For example, according to military department officials, water is necessary to operate missions such as rocket launches for cooling and for noise and fire suppression (see sidebar), to maintain temperatures to properly store equipment such as parachutes, and for firefighting training (see fig. 1). Rocket Launch at Vandenberg Air Force Base, California According to Vandenberg Air Force Base officials, water is used in multiple ways during rocket launch activities. For example, water is necessary for noise and vibration suppression, heat reduction, and fire suppression as needed. The officials stated that between 60,000 to 100,000 gallons of water are needed for each launch. In 2018, there were nine launches. With an anticipated increase in launches in the future, they expect the demand for water to increase as well. OSD officially reorganized its acquisition organization on January 31, 2018, in response to Section 901of the National Defense Authorization Act for Fiscal Year 2017. Under the reorganization, responsibilities of the former Under Secretary of Defense for Acquisition, Technology and Logistics were divided between two new offices—the Under Secretary of Defense for Research and Engineering and the Under Secretary of Defense for Acquisition and Sustainment. According to DOD, responsibilities for energy, installations, and environment were transferred from the Office of the Undersecretary of Defense for Acquisition, Technology and Logistics to the newly created Office of the Under Secretary of Defense for Acquisition and Sustainment in 2018. According to an OSD official, within this office, responsibilities for water management at military installations are delegated to two deputy assistant secretaries under the Office of the Assistant Secretary of Defense for Sustainment—the Office of the Deputy Assistant Secretary of Defense for Environment, who is responsible for water resources management in general, and the Office of the Deputy Assistant Secretary of Defense for Energy, who is responsible for overseeing planning for water at the installation level. Each of the military departments has designated an office or multiple offices with responsibilities for water policy and implementing programs to support that policy at installations. Specifically: Air Force: The Assistant Secretary of the Air Force for Installations, Environment, and Energy is responsible for procedures to manage the Air Force’s water consumption, throughput, and requirements, in alignment with policies and strategic direction. Within this office, the Deputy Assistant Secretary of the Air Force for Environment, Safety and Infrastructure provides strategic direction, policy, and oversight for water management. Navy: The Office of the Assistant Secretary of the Navy for Energy, Installations, and Environment is responsible for establishing policy and overseeing water resource management. This office, along with the Office of the Chief of Naval Operations Shore Readiness Division, and the Commander, Navy Installations Command, makes policy, guidance, and many major investment decisions related to installations’ water departments. Within the Department of the Navy, the Marine Corps also has its own offices responsible for water policy. Specifically, the Deputy Commandant for Installations and Logistics is responsible for establishing energy and water management policy for Marine Corps installations in accordance with the Commandant’s direction. The Commander, Marine Corps Installations Command, is responsible for water management, such as overseeing program planning and execution, and serving as the Marine Corps Installations Energy Program Manager. Army: The Assistant Secretary of the Army for Installations, Energy, and Environment establishes policy, provides strategic direction, and supervises all matters pertaining to energy and environmental programs, among other responsibilities. Within this office, the Deputy Assistant Secretary of the Army for Energy and Sustainability provides strategic leadership, policy guidance, program oversight, and outreach for energy, water, and sustainability throughout the Army enterprise. OSD-level entities and the three military departments conducted six assessments between April 2017 and January 2019 that, despite having varied focus areas, all included at least one component focused on vulnerability to water scarcity. The Office of the Under Secretary of Defense for Acquisition and Sustainment conducted the most recently reported (January 2019) OSD-level assessment, in response to a congressional reporting requirement. OSD-level entities in place before OSD’s 2018 reorganization conducted the other two assessments, reporting their results in January 2018 and July 2018—also responses to congressional reporting requirements. The Air Force’s, Navy’s, and Army’s three assessments span different time frames, encompass different scopes, and respond to different internal reporting requirements. The Air Force reported its results in November 2018; the Navy’s assessment conducted by CNA reported its results in December 2017; and the Army reported in April 2017. Table 1 provides a summary of these assessments, including responsible offices and focus areas. We found that DOD does not have assurance that it is using accurate and reliable information regarding which installations are at risk for water scarcity. When we compared the results of the OSD assessments and the military department assessments, we found that they varied markedly, raising questions about their quality and about which source of information DOD is using to determine which installations are vulnerable to water scarcity. An OSD official told us that the OSD assessments constitute the best DOD information available on installations at risk of water scarcity, but we found that the assessments do not align with leading practices for identifying and analyzing water scarcity—practices that contribute to a reliable assessment of water availability. In contrast, we found that the military department assessments do align with these leading practices, but OSD officials disagree as to whether these assessments can and should be used to identify installations at risk of water scarcity across the defense enterprise. As a result, DOD cannot be assured that it is using reliable information for water resource management. The three OSD assessments and the three military department assessments varied markedly in their results regarding which installations are vulnerable to water scarcity. Collectively, the six assessments identified a total of 102 individual installations at risk of water scarcity, as shown in figure 2. Only one installation, Vandenberg Air Force Base in California, was identified in all three OSD assessments and the applicable military department (Air Force) assessment. Of the 102 individual installations identified in the six assessments as vulnerable to water scarcity, 42 (41 percent) were included in multiple assessments. OSD identified more installations for each military department as at risk than did the military departments themselves. Specifically, across its three assessments, OSD identified 95 installations as being at risk—48 Air Force installations, 29 Navy or Marine Corps installations, and 18 Army installations. The military departments collectively identified a total of 27 installations as being at risk—14 Air Force installations, nine Navy or Marine Corps installations, and four Army installations. Below is a more detailed description of the installations identified as being at risk of water scarcity in the six assessments, by the military departments. Air Force: Of the 48 Air Force installations identified across the OSD assessments, only three—Kirtland Air Force Base, New Mexico; McConnell Air Force Base, Kansas; and Vandenberg Air Force Base, California—appeared in all of them. In addition, as noted above, only one Air Force installation was identified both in all three OSD assessments and the Air Force assessment—Vandenberg Air Force Base, California. Of the 14 Air Force installations identified within the Air Force assessment, 13 appeared in at least one of the OSD assessments. Navy: Of the 29 Navy or Marine Corps installations identified across the OSD assessments, three installations—Marine Corps Air Station Yuma, Arizona; Naval Base Coronado, California; and Naval Weapons Station Seal Beach, California—appeared in at least two of the OSD assessments. Of the nine Navy installations, including the Marine Corps installations identified within the Navy assessment, four appeared in at least one of the OSD assessments. Army: Of the 18 total Army installations identified across the OSD assessments, only one—White Sands Missile Range, New Mexico— appeared in all three. However, the Army’s assessment did not identify that installation as being at risk. In addition, one of the OSD assessments—the climate vulnerability survey—identified more than three times as many Army installations as being at risk as the Army’s own assessment. Of the four Army installations identified within the Army assessment, three appeared in at least one of the OSD assessments. Given the different scopes of these assessments, it is understandable that they would produce different results. However, the substantial differences in results raise questions about whether the assessments that produced them were methodologically sound and about which source of information DOD is using to identify installations at risk of water scarcity— information needed for water resource management. Although an OSD official told us that the OSD assessments constitute the best DOD information available on installations at risk of water scarcity, we found that they did not incorporate four of five leading practices for identifying and analyzing water scarcity. Specifically, our analysis shows that, in conducting their assessments, OSD officials did not always (1) identify current water availability, (2) identify future water availability, (3) take into account all sources of water, or (4) precisely identify locations, as shown in table 2. Below is a detailed comparison of each OSD assessment against the five leading practices. OSD’s climate vulnerability survey. Of the three OSD assessments, the climate vulnerability survey reflects the most (3 out of 5) leading practices. Specifically, we found that the methodology used in the climate vulnerability survey followed the leading practice for identifying current water availability. The survey collected and analyzed drought-related information in a timely and systematic manner by having a question about current drought conditions on its web-based self-reporting survey. did not follow the leading practice for identifying future water availability. The survey focused only on current and past water availability. did not follow the leading practice for taking into account all sources of water. The survey did not account for all sources of water (e.g., precipitation, soil moisture, streamflow, groundwater levels, reservoir and lake levels, and snowpack) because it did not include a question about the sources of the water. followed the leading practice for precisely identifying locations. The survey went directly to all DOD installations and inquired about drought conditions at sites owned or managed by the installation, in addition to the installation itself. This enabled DOD to know the precise location of installations and their associated sites relative to identified drought-prone areas of the state or region and vulnerable economic sectors, individuals, or environments. followed the leading practice for comprehensively including all locations. The survey was completed for all primary installations and associated sites worldwide. OSD’s energy report and climate change report. OSD used the U.S. Drought Monitor map to conduct its assessments for both OSD’s energy report and climate change report. According to an OSD official, use of the U.S. Drought Monitor map constitutes DOD’s best approach for identifying military installations vulnerable to water scarcity. However, we determined that, in doing so, OSD did not follow four of the five leading practices. Specifically, using the U.S. Drought Monitor Map to produce the energy report and climate change report, OSD did not follow the leading practice for identifying current water availability and did not follow the leading practice for identifying future water availability. According to the cofounder of the U.S. Drought Monitor, the conditions reflected on the U.S. Drought Monitor maps are retrospective—weekly assessments of drought conditions based on how much, if any, precipitation occurred from 1 week to several years before the day the map was issued. This is problematic because drought conditions can change from month to month (see fig. 3), and the months chosen may not be representative of the annual drought condition. An OSD official stated that OSD used data from the U.S. Drought Monitor map as of April 2018 for the energy report and only the summer months of 2018 for the climate change report, which is unlikely to reflect current water availability for an entire year. According to the cofounder of the U.S. Drought Monitor, the U.S. Drought Monitor maps also do not show projections of future water scarcity, which would be necessary to fully assess an installation’s vulnerability to water scarcity. did not follow the leading practice for taking into account all sources of water. According to the cofounder of the U.S. Drought Monitor, U.S. Drought Monitor maps do not take into account all sources of water that might be available to a specific installation. The U.S. Drought Monitor maps do not fully assess the availability of water from groundwater sources (e.g., aquifers) or nonlocal sources (e.g., reservoir water delivered by canals). did not follow the leading practice for precisely identifying locations. According to the co-founder of the U.S. Drought Monitor, U.S. Drought Monitor maps only display regional drought conditions, not drought information applicable to precise locations. For this reason, the Drought Monitor Portal warns that the large-scale maps generated should not supersede locally provided information about water availability conditions. Therefore, OSD may have inaccurately identified installations as being at risk of water scarcity. followed the leading practice for comprehensively including all locations. Since the energy report used a map of all installations within the contiguous U.S. to conduct its analysis, and the climate change report included all 79 mission-assurance locations within its scope, these assessments constituted a comprehensive approach. The information we collected from installations identified by OSD as being at risk of water scarcity also indicates weaknesses in OSD’s approach. Of the 17 installations that were identified in OSD’s assessments as being at risk of water scarcity and that we contacted or visited, officials from 12 stated that they did not anticipate water scarcity affecting their future mission-related activities, disagreeing with the conclusions of OSD’s assessments. For example: Officials at Naval Weapons Station Seal Beach, California, told us the installation does not expect water scarcity to affect its mission-related activities because none of its water-using facilities (i.e., administrative facilities) on the installation are particularly water-intensive. They stated the installation’s water is provided by the City of Seal Beach, which in turn is supplied by a larger water company. According to the officials, there are proposed plans to construct a nearby desalination plant, which would prevent water scarcity issues. Officials at Moody Air Force Base, Georgia, stated that the installation is not vulnerable to water scarcity now or over the next 20 years because the base has its own water-treatment plant with wells that draw water from the Floridan aquifer, which spans an area of 100,000 square miles in the southeastern United States, underlying the entire state of Florida and parts of Alabama, Georgia, Mississippi, and South Carolina. According to the officials, use of the aquifer is unconstrained; in addition, Moody Air Force Base holds water permits that create a 64 percent surplus capacity of daily water availability to support current or new mission growth. Officials at Fort Bragg, North Carolina, stated that the installation is in the Southeast region of the United States, which is not known as a region with water scarcity issues. They stated that the region’s primary threats, from a water scarcity perspective, are pollution and population growth. In addition, the officials said that the two public utilities from which it purchases its water are not expected to hit a critical demand for water until the year 2060 or later. When we informed an OSD official of the results of our analysis, the official stated that OSD did not have any concerns about the information it provided to the Congress in its three assessments. Specifically, the official said the climate vulnerability survey might have had different responses depending on the perspective of the responder, but it provided useful qualitative data. The official also maintained that the U.S. Drought Monitor was the best source of information, and is a resource produced by the federal government. However, as outlined above, while the drought monitor is a useful source of information, it is not intended to be used in the manner in which DOD has employed it. Unlike the OSD level assessments, we found that the assessments produced by the military departments are aligned with all five leading practices (see table 3). Below are detailed examples of how the military department assessments were compared against the five leading practices. Specifically, we found that the military department assessments: followed the leading practice for identifying current water availability. For example, the Navy contacted installation staff directly and analyzed water use and billing data directly from departmental water- system databases to assess the extent to which the Navy was facing water-related challenges (which included water availability and quality). followed the leading practice for identifying future water availability. For example, the Air Force assessment considered future water availability by considering long-term effects from climate change, future water restrictions, and changes in water access rights. In addition, the Navy assessment considered future water availability by considering sea-level rise, water rights, diminishing groundwater supplies, and emerging water pollutants. followed the leading practice for taking into account all sources of water. For example, the Army assessment considered alternate water sources by requiring installations to identify and enumerate their potable sources of water as a measure of redundancy. followed the leading practice for precisely identifying locations. For example, the Navy assessment used geospatial data on hazards to water as well as data published by Naval Facilities Command. This enabled the Navy to precisely identify installation and site locations for water and sewer infrastructure, including pumps, storage, sewer lines, and water-treatment plants relative to those hazards. followed the leading practice for comprehensively including all locations. According to service officials and an agency document, the scope of each military department assessment included all respective installations within each military department. Installations we contacted that were identified in the military department assessments as being at risk of water scarcity generally agreed with the assessments. Of the seven installations that were identified in military department assessments as being at risk of water scarcity and that we contacted or visited, officials from six (86 percent) agreed that they anticipated water scarcity may affect their future mission activities or otherwise noted risks of water scarcity that could affect their installations. For example: Officials at Mountain Home Air Force Base, Idaho, stated that water use on the installation was significantly curtailed in 2017 and 2018 (and was anticipated to be curtailed in 2019) due to the inability to produce sufficient quantities of water to meet demand. Officials from F. E. Warren Air Force Base, Wyoming, stated that drought is a continual threat to the area. The officials stated that if the area does not receive adequate precipitation or snowmelt, the city may place a water restriction for the installation. Officials from Marine Corps Air Station Yuma, Arizona, stated that future mission activities could be impacted by water scarcity, especially as the population of the installation continues to grow with the arrival of additional air squadrons. As noted earlier in this report, the Office of the Assistant Secretary of Defense for Sustainment is responsible for water management at all military installations. Individuals from this office with whom we spoke agreed that having accurate information about water scarcity data across DOD is important to help fulfill these responsibilities and inform senior decision-making, including budget development, resourcing, and risk management. However, these officials disagree about whether it would be feasible to rely on the military department assessments, which we found align with leading practices, to identify installations at risk of water scarcity across DOD. According to one OSD official, the military department assessments should not be used to consider water scarcity across DOD as a whole because their methodologies differed and therefore are not comparable to one another. The assessments do not reflect a coordinated, department- wide assessment. For example, the Air Force assessment reported vulnerability to water scarcity as four distinct qualitative ratings, each combining likelihood and severity, without any numerical data. The Army’s assessment, in contrast, reported vulnerability using 34 distinct numerical scores for each installation, averaged into four distinct categories. While both assessments were aligned with leading practices, this OSD official believes that the differences in their specific approaches and subsequent results make it difficult to compare vulnerability to water scarcity across military departments. According to another OSD official, it would be appropriate for DOD to rely on the results of the military department assessments because responsibilities for prioritizing projects and for allocating funds to those projects lie with the military departments. As such, there is not a concern that the departments assessed vulnerability differently. According to this official, were the department to issue a new DOD-wide report on water scarcity, it would simply be a “rollup” of the military department assessments, with an update of current status. According to Standards for Internal Control in the Federal Government, management should use quality information—information that is, among other things, appropriate, current, complete, and accurate—to achieve the entity’s objectives. In identifying information requirements, management should consider the expectations of both internal and external users, as well as the entity’s objectives and related risks. Because the OSD-level assessments do not align with leading practices for identifying and analyzing water availability, OSD lacks assurance that it has quality information and risks potentially using or providing to Congress unreliable information. Further, while the military department assessments are aligned with leading practices, the Office of the Assistant Secretary of Defense for Sustainment has not determined whether they are sufficient for meeting its policy-making and oversight objectives and whether the risk presented by combining results from assessments that used varying methodologies is an acceptable level of risk. Until this question is resolved, the department will not have assurance that it is using accurate and reliable information to assess water scarcity. DOD’s installations rely on billions of gallons of water to operate and conduct their missions, but critical installations are at risk of water scarcity, and the risks are only projected to increase. The substantial differences in results of DOD’s assessments to identify installations at risk of water scarcity raise questions about whether the assessments were methodologically sound and about which source of information OSD is using for water resource management. OSD’s approach to assessing installations at risk of water scarcity did not consistently apply leading practices for identifying current and future water availability, taking into account all sources of water, and precisely identifying locations—yet an OSD official told us that the OSD assessments constitute the best DOD information available on installations at risk of water scarcity. In contrast, the military departments did apply all leading practices in their assessments on installations at risk of water scarcity; however, OSD officials were not in agreement as to whether these assessments could be used at a departmental level. By assessing and documenting whether OSD should conduct a coordinated, department-wide assessment aligned with leading practices or should rely on the military department assessments for identifying and analyzing water availability, OSD would have greater assurance that it has the information that it needs to manage water scarcity across the department and that Congress needs to better understand the threat of water scarcity to DOD’s mission. The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment (1) assesses whether DOD should conduct a coordinated, department-wide assessment aligned with leading practices for identifying and analyzing water availability or rely on military department assessments to determine which DOD installations are at risk of water scarcity and (2) documents this decision. (Recommendation 1) We provided a draft of this report for review and comment to DOD. In written comments, DOD concurred with our recommendation. DOD comments are reprinted in their entirety in appendix III. DOD also provided technical comments, which we incorporated as appropriated. We are sending copies of this report to the appropriate congressional addressees; the Secretary of Defense; and the Secretaries of the Air Force, the Navy, and the Army. In addition, this report will be available at no charge on the GAO website at www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. In this report, we evaluate the extent to which the Department of Defense (DOD) has assurance that it is using reliable information to identify installations at risk of water scarcity. We reviewed statutes and congressional committee reports that directed DOD to conduct assessments for climate-related purposes, including for identifying installations at risk of water scarcity. We also analyzed information contained in the six DOD assessments conducted from April 2017 through January 2019 that identify installations at risk of water scarcity—three Office of the Secretary of Defense (OSD) assessments and three military department assessments—to determine the extent to which the assessments identified the same or different installations. Specifically, we analyzed the following DOD assessments: two OSD assessments that focused on climate-related risks to installations: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, Department of Defense Climate- Related Risk to DOD Infrastructure Initial Vulnerability Assessment Survey (SLVAS) Report (January 2018). We analyzed information on military installations vulnerable to drought in this assessment. Office of the Under Secretary of Defense for Acquisition and Sustainment, Report on Effects of a Changing Climate to the Department of Defense (January 2019). We analyzed information on military installations vulnerable to drought in this assessment. one OSD assessment that focused on installation energy performance, which included an appendix with information on military installations vulnerable to water scarcity: Office of the Assistant Secretary of Defense for Energy, Installations, and Environment, Department of Defense Annual Energy Management and Resilience Report (AEMRR) Fiscal Year 2017 (July 2018). We analyzed the information on military installations vulnerable to water scarcity in this assessment. three military department assessments that contained information on water-related risks: U.S. Air Force, Summary Information on Installations with Water Hazards (Provided November 2018). We analyzed information on military installations with catastrophic and critical water hazards in this assessment. U.S. Navy, including the Marine Corps, CNA, Assessing Water Risk at DON Installations—Identifying Hazards and Water Management Challenges (December 2017). We analyzed information on military installations with water availability risk in this assessment. U.S. Army, FY17 Installation Status Report (Mission Capacity) Water Data Analysis (April 2017). We analyzed information on military installations with minor and severe potable water risk. In analyzing these six assessments, we focused on active-duty military installations in the contiguous United States at risk of water scarcity. Further, to discuss the methodologies used in the six assessments, we interviewed officials who were knowledgeable about the various assessments: officials from the OSD’s Office of the Assistant Secretary of Defense for Sustainment, each of the military departments with responsibilities for water management at military installations, CNA, which completed the Department of the Navy’s assessment, and the University of Nebraska–Lincoln’s National Drought Mitigation Center, which hosts the U.S. Drought Monitor map that shows parts of the United States in drought. We compared the methodologies used to develop OSD’s three assessments and the military departments’ three assessments with five leading practices for identifying and analyzing risks of water scarcity. We derived the five leading practices from the Department of Energy’s and the United States Environmental Protection Agency’s compilation of 14 water efficiency best management practices, and principles published in the University of Nebraska–Lincoln’s National Drought Mitigation Center’s 10-Step Drought Planning Process. These leading practices are: (1) identify current water availability, (2) identify future water availability, (3) take into account all sources of water, (4) precisely identify locations, and (5) comprehensively include all locations. According to the 10-Step Drought Planning Process, data and information derived from these leading practices contribute to a reliable assessment of water availability. We discussed these five leading practices we identified with officials from the Office of the Assistant Secretary of Defense for Sustainment and the military departments and gained their agreement about using these practices for determining installations at risk of water scarcity. We then determined whether, in their respective methodologies, OSD’s and the military departments’ assessments had followed each of these five leading practices. Specifically, we considered the “identify current water availability” leading practice as “followed” if OSD’s and the military departments’ assessment was annually reporting water use or status of water supply, and the leading practice as “not followed” if the assessment was not annually reporting water use or status of water supply; “identify future water availability” leading practice as “followed” if OSD’s and the military departments’ assessment noted whether climate change was a factor in their assessment or considered future water availability from non-climate-change-related factors and the leading practice as “not followed” if the assessment did not note whether climate-change was a factor in their assessment or consider future water availability from non-climate-change-related factors; “take into account all sources of water” leading practice as “followed” if OSD’s and the military departments’ assessment noted consideration of alternate water sources (such as groundwater, purchase agreements, additional reservoirs, etc.) and the leading practice as “not followed” if the assessment did not note consideration of alternate water sources (such as groundwater, purchase agreements, additional reservoirs, etc.); “precisely identify locations” leading practice as “followed” if OSD’s and the military departments’ assessment noted the specific location of the installation they were reviewing and provided data specifically from that installation, and the leading practice as “not followed” if the assessment did not note the specific location of the installation they were reviewing and provide data specifically from that installation; and “comprehensively include all locations” leading practice as “followed” if OSD’s and the military departments’ assessment considered all the locations at potential risk of water scarcity within the scope of their assessment, and the leading practice as “not followed” if the assessment did not consider all the locations at potential risk of water scarcity within the scope of their assessment. Specifically, for OSD’s Department of Defense Climate-Related Risk to DOD Infrastructure Initial Vulnerability Assessment Survey (SLVAS) Report and its Department of Defense Annual Energy Management and Resilience Report (AEMRR) Fiscal Year 2017, the scope of the assessments included all DOD installations; for OSD’s Report on Effects of a Changing Climate to the Department of Defense, the scope of the assessment included 79 mission-assurance priority installations; and for the military department assessments, the scope included all respective installations within each military department. To obtain information about water scarcity at individual installations, we selected a nongeneralizable sample of active-duty military installations in the contiguous United States. To develop this sample, we included installations that were identified by DOD assessments as having water- related vulnerabilities and by military department officials in interviews as having ongoing pilot studies or issues related to water scarcity. We also included installations that had (1) historically experienced water scarcity (prior to 2014); (2) recently experienced water scarcity (from 2014 to 2019); and (3) are projected to experience severe water scarcity (over the next 20 years or longer). From these criteria, we selected a nongeneralizable sample of 17 installations that were identified in OSD’s three assessments that reflected diversity in military service, mission, and water scarcity (see table 4). We visited five of these installations in person and contacted the remaining 12 installations by email. We selected the five installations to visit because three installations (Naval Air Facility El Centro, California; Marine Corps Air Station Yuma, Arizona; and Luke Air Force Base, Arizona) provided diversity among military services and were in close proximity to each other, which allowed us to visit multiple locations in one trip; one installation (Vandenberg Air Force Base, California) had been identified in all three OSD assessments and the applicable military department assessment as being at risk of water scarcity; and one installation (Fort Bragg, North Carolina) provided geographic diversity and inclusion of at least one installation per military service in our sample. For the remaining 12 installations, we developed and sent by email a list of similar questions and document requests that we used during our site visits. We received responses from all 12 installations. Results from our nongeneralizable sample cannot be used to make inferences about all DOD installations. However, the information from these installations provides valuable insights about how water is being used by these installations for their mission-related activities and whether water scarcity had affected or was expected to affect their mission-related activities. To determine the extent to which DOD has assurance it is using accurate and reliable information about installations at risk of water scarcity to manage water resources across the department, we compared the information DOD has from the various assessments with Standards for Internal Control in the Federal Government on using quality information to achieve agency objectives. We conducted this performance audit from September 2018 to November 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 5 provides a list of the 102 individual active-duty military installations in the contiguous United States that were identified in at least one of six DOD assessments—three Office of the Secretary of Defense assessments and three military department assessments—as being at risk of water scarcity. In addition to the contact named above, Brian Lepore (Director), Jodie Sandel (Assistant Director), Barbara Wooten (Analyst-In-Charge), Tracy Barnes, Chaneé Gaskin, Gina Hoover, Mae Jones, Mary Jo LaCasse, Amie Lesser, Shahrzad Nikoo, Paulina Reaves, and Edward Rice made key contributions to this report.", "summary": "DOD reported in January 2019 that critical installations are at risk of water scarcity—that is, of not having sufficient water available to meet their mission needs. According to military department officials, installations depend on water for activities such as training, weapons testing, fire suppression, and sanitation. In its 2018 Fourth National Climate Assessment , the U.S. Global Change Research Program reported that warming temperatures will continue to cause worsening droughts and the decline of surface water quality. Senate Report 115-262 included a provision for GAO to review DOD's identified or potential effects of water scarcity. For this report, GAO evaluated the extent to which DOD has assurance that it is using reliable information to identify installations at risk of water scarcity. GAO analyzed DOD's six assessments conducted from April 2017 through January 2019 to identify installations at risk of water scarcity and compared the assessments with five leading practices for identifying and analyzing water scarcity. GAO also interviewed officials from OSD and the military departments and contacted a nongeneralizable sample of 17 installations identified in OSD's assessments to reflect diversity in military service, mission, and water scarcity. GAO found that the Department of Defense (DOD) does not have assurance that it is using reliable information regarding which installations are at risk for water scarcity. When comparing the results of six Office of the Secretary of Defense (OSD) and military department assessments on installations vulnerable to water scarcity, GAO found that they varied markedly, raising questions about their quality and about which source of information DOD is using to determine which installations are vulnerable to water scarcity (see figure). An OSD official stated that the three OSD-produced assessments provided the best information available on which installations are at risk of water scarcity. However, GAO found that these assessments did not reflect four of five leading practices for identifying and analyzing water scarcity—practices that contribute to a reliable assessment of water availability. Specifically, OSD did not always (1) identify current water availability, (2) identify future water availability, (3) take into account all sources of water, or (4) precisely identify locations. Further, although GAO found that the three military department assessments aligned with all leading practices, OSD officials disagreed as to whether these assessments can and should be used to identify installations at risk of water scarcity across the defense enterprise. Until OSD resolves the question as to whether it should conduct a department-wide assessment of installations that aligns with leading practices or whether it should rely on the military department assessments, the department will not have assurance that it is using reliable information to assess water scarcity. GAO recommends that the Office of the Secretary of Defense assess whether it should conduct a coordinated, department-wide assessment aligned with leading practices or rely on military department assessments to determine which DOD installations are at risk of water scarcity. DOD concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "Funding for university STEM research. The five federal agencies included in our analysis provide billions of dollars annually for university research in STEM fields, with HHS-NIH providing more than the other four agencies combined. Figure 1 details the total amount of basic and applied STEM research funding provided to universities by each agency in fiscal year 2018, these data are preliminary and the most current data available during our review. Sexual harassment. While sexual harassment is not a defined term in Title IX, it can constitute sex discrimination under Title IX in some circumstances if, among other things, the harassment is “so severe, pervasive, and objectively offensive that it effectively bars the victim’s access to an educational opportunity or benefit.” Some federal agencies and NASEM define sexual harassment based on specific behaviors. Specifically, according to NASEM, sexual harassment encompasses three types of behavior: Sexual coercion: Favorable treatment conditioned on sexual activity. Unwanted sexual attention: Verbal or physical sexual advances that are unwelcome, including sexual assault. Gender harassment: Sexist hostility and crude behavior. The most common form of sexual harassment is gender harassment, which generally involves behavior that conveys hostility, objectification, exclusion, or second-class status about a person’s gender. According to the Consultant Report on the University of Texas System Campus Climate Survey for NASEM, female medical and engineering students enrolled in the University of Texas System were more likely to experience sexual harassment by faculty or staff compared with students enrolled in other majors. Title IX enforcement responsibilities. Title IX of the Education Amendments of 1972 is the primary federal law that addresses sex discrimination under education programs or activities receiving federal financial assistance, including federally funded grant programs at educational institutions, such as universities. Under Title IX, federal agencies that award grants to educational institutions, including universities, have enforcement responsibilities to ensure such institutions do not discriminate based on sex. Enforcement responsibilities fall under four main areas: 1. Issuing regulations. Title IX requires that agencies promulgate regulations to provide guidance on Title IX enforcement to recipients of federal financial assistance who administer education programs or activities. 2. Obtaining assurance from university grantees that they are in compliance with Title IX. Most of the agencies we reviewed require grantees to submit an “assurance of compliance” form as part of their grant application or award to attest compliance with anti-discrimination laws, including Title IX. 3. Conducting periodic compliance reviews of funding recipients. Grant funding agencies are required to conduct periodic Title IX compliance reviews of university grantees. A Title IX compliance review is an agency’s assessment of whether a grantee is complying with the law. According to DOJ’s Title IX legal manual, federal agencies have broad discretion in determining which grantees to review for compliance. Federal agencies may conduct these reviews on-site at a university (grantee) or via a desk audit. In both types of review, agency officials, among other things, review documentation that indicates compliance, such as the universities’ policies and procedures for receiving, investigating, and resolving Title IX complaints. During on-site reviews, officials interview staff, faculty, and students about their awareness of Title IX and any issues of potential sex discrimination that they have encountered. 4. Investigating written complaints of sex discrimination against recipients in a timely way. Federal agencies are required to establish and publish procedures for the prompt processing and disposition of complaints. An individual alleging discrimination on the basis of sex by a university can file a discrimination complaint with multiple entities, including the university or one of the federal agencies that provides funding to the university, which could include Education or another funding agency. In 2015, we reported on six federal agencies’ grant making to women in STEM research, including their Title IX compliance activities. We found that the Departments of Defense, and Health and Human Services were not conducting required Title IX compliance reviews at universities they funded and recommended that they periodically do so. In response to our recommendation, HHS conducted three Title IX compliance reviews in 2018 and according to officials, the agency initiated additional Title IX compliance reviews in 2019 and 2020. Education and DOJ also have responsibilities for administering Title IX. Education plays a key role in ensuring compliance with Title IX as it provides funding to most universities in the United States. DOJ’s Civil Rights Division is responsible for enforcing federal statutes prohibiting discrimination of protected classes, including Title IX. Under Executive Order 12250, DOJ also has the responsibility for playing a leadership role in coordinating the “consistent and effective implementation” of several civil rights laws, including Title IX. In 2015, we reported that DOJ had no formal information-sharing process for federal agencies to exchange best practices on Title IX compliance activities, and we recommended that it establish such a process. In response to our recommendation, DOJ reconstituted the Quarterly Title IX STEM discussion group in February 2016 to facilitate information sharing across the six major STEM grant-making federal agencies. Recipients of federal assistance—in this case, university grantees—also have Title IX compliance responsibilities. Specifically, universities are responsible for ensuring Title IX compliance, designating an employee to coordinate compliance (e.g., a Title IX coordinator), establishing procedures to promptly and equitably resolve student and employee complaints of sex discrimination made against the university, and publishing a notice stating that they do not discriminate on the basis of sex. Figure 2 outlines the various compliance activities required under Title IX and the entity responsible for carrying out each activity. Offices and their responsibilities for Title IX and grant management. Among the federal agencies we reviewed, different offices handle various aspects of Title IX and grant compliance activities. Generally, each agency’s civil rights or diversity office conducts Title IX compliance reviews, develops policies and procedures for grantees, and investigates allegations and complaints involving university researchers supported by their agency’s federal STEM grants. All five agencies (DOE, HHS, NASA, NSF, and USDA-NIFA) primarily address Title IX complaints, including sexual harassment complaints, through their civil rights or diversity offices. However, these offices are responsible for more than just addressing complaints and preventing sexual harassment at grantees, including universities; these offices oversee a number of civil rights, diversity, and inclusion efforts for the entire agency. Moreover, most of these offices also address internal employee sexual harassment complaints and other discrimination issues. The agency office that awards grants generally creates and modifies grant terms and conditions for universities receiving funding from the agency. Table 1 outlines each agency’s Title IX and grant management responsibilities by office. All five agencies conducted periodic Title IX compliance reviews, as required by federal laws and regulations, from fiscal years 2015 through 2019, and three completed joint compliance reviews. Two agencies publicized promising practices from Title IX compliance reviews on their websites and did so to assist all grantees with Title IX compliance. The other three agencies have not clearly publicized such practices from their Title IX reviews on their websites. The five agencies we reviewed conducted periodic Title IX compliance reviews, as required by federal laws and regulations. From fiscal year 2015 through 2019, DOE, HHS, NASA, NSF, and USDA-NIFA officials reported that their agencies met the requirement for conducting periodic reviews. During this period, the agencies conducted between 4 and 11 Title IX compliance reviews among hundreds of grantees. No agency completed more than three reviews in a fiscal year. Two agencies— DOE and NASA—have requirements to conduct a minimum of two Title IX compliance reviews annually. DOE and NASA meet their statutory requirements by starting at least two Title IX compliance reviews each year, according to officials. HHS, NSF, and USDA do not have an annual minimum requirement and are not required to have one, according to officials. Agencies conducted visits to universities to assess compliance and developed written compliance reports. In the compliance report, agencies can recommend a grantee take action to improve existing compliance efforts to prevent sex discrimination and may highlight promising practices by grantees. For example, NASA recommended in a written compliance report that a grantee provide more targeted Title IX training geared toward STEM students and faculty, noting that such training should focus on subtle forms of gender bias that pervade STEM programs as well as on more egregious examples of sexual harassment. Similarly, HHS made recommendations in three of its compliance reviews for grantees to notify students and faculty of their right to file a Title IX complaint with the HHS Office for Civil Rights. University grantees are not required to implement the agency’s recommendations, but they must take corrective actions to resolve findings of Title IX noncompliance, according to DOE, HHS-OCR, NASA, and NSF officials. Agencies are required by law to seek voluntary compliance for Title IX violations. If an agency finds that a grantee has violated Title IX (noncompliance), it first seeks to establish voluntary compliance through a resolution agreement—an agreement with the agency and grantee outlining corrective actions for the grantee. If the agency is unable to achieve voluntary compliance in a Title IX case, it may initiate proceedings to suspend or terminate federal funding, or refer the case to DOJ for possible litigation. According to officials, the five agencies we reviewed have not suspended or terminated funding to enforce Title IX, including sexual harassment. Instead, according to agency officials, their reviews have found that most grantees are in compliance with Title IX from fiscal year 2015 through 2019, except for one grantee, where the agency worked with the grantee to achieve voluntary compliance. To leverage limited resources, three of the five agencies—DOE, NASA, and NSF—conducted joint Title IX compliance reviews. These reviews occur when two agencies providing funding to the same grantee jointly assess whether the grantee is complying with the law. DOE and NSF conducted three joint compliance reviews in fiscal years 2015 and 2016, while NASA and NSF conducted a joint review in fiscal year 2019. These joint reviews helped agencies leverage resources. NASA and NSF publicized on their websites a list of promising practices identified as part of their compliance reviews to assist grantees with Title IX compliance. Promising practices—grantee actions that have the potential to advance equal opportunities, diversity, and inclusiveness for program participants regardless of sex—may be considered, adopted, and replicated by other grantees, according to NASA and NSF officials. Some actions may go beyond meeting Title IX compliance requirements. NASA identifies promising practices to provide grantees with information and examples on practices they may wish to consider replicating to help enhance or supplement their equal opportunity efforts, according to officials. For example, NASA noted a promising practice in which a grantee presents campus training sessions on Title IX at which participants develop bystander behavior skills, discuss consent and sexual respect, and learn how to encourage and support reporting of sexual misconduct. In addition, this university grantee facilitates the workshop using clickers to allow real-time, anonymous audience response, enabling the facilitators to measure learning progress and see attitudinal shifts in real time. According to NASA, since the inception of its Title IX compliance program, the agency has followed a philosophy of providing meaningful technical assistance to universities, including identifying and reporting on promising practices of the universities that the agency reviews. NASA officials told us this approach mitigates the fact that the agency only has the resources to conduct compliance reviews at a few of its hundreds of grantees annually. NSF takes a similar approach. For example, NSF’s webpage for promising practices has a link to a university’s complaint resolution flow chart as an example for others to draw on. According to NSF officials, the agency values opportunities to learn about practices that have the potential to make significant and meaningful impacts on grantees’ efforts to create and maintain research environments that are safe and free from sexual and gender-based harassment. Moreover, according to officials, NSF grantees have expressed gratitude to the agency for sharing what other universities are doing that is working well. In contrast, while DOE, HHS, and USDA identified promising practices in some of their Title IX compliance reviews, they have not clearly publicized a list of these practices to the broader grantee community. DOE has posted reports of Title IX compliance reviews, but no list of promising practices. As a result, grantees who want to learn from other universities would need to review individual compliance reports and search for promising practices. DOE does plan to develop a publication that identifies promising practices and lessons learned from its Title IX compliance reviews in fiscal year 2020, according to officials. The agency did not provide any plans or timeframes because officials stated that DOE’s Office of Civil Rights is determining the best approach for this project. USDA-NIFA is planning to create mechanisms to publicize best practices, according to officials, but it has not yet done so. According to USDA officials, the agency is discussing and determining the best promising practices from compliance reviews to publicize; however, recent staff changes have delayed this effort. As a result, USDA did not provide further details about how and when it will publicize promising practices. In October 2019, HHS’s Office for Civil Rights (HHS-OCR) updated its Title IX webpage to include a section dedicated to sexual harassment, including links to resources, guidance, and effective practices (also called promising practices) from other agencies, as well as a written resolution agreement between HHS and a university grantee that resolved findings of sex-based harassment. While a dedicated webpage for sexual harassment is a positive step, HHS’s webpage includes promising practices from other federal agencies—Equal Employment Opportunity Commission and NSF—but not HHS. HHS- OCR officials told us that promising practices are similar across federal agencies. However, HHS Title IX compliance reviews cover grantees that may be different from other federal agencies, such as medical colleges, and these grantees may face unique challenges in complying with Title IX. For example, according to the 2018 NASEM report, women students, trainees, and faculty in academic medical centers experience sexual harassment by patients and patients’ families, in addition to the harassment they experience from colleagues and those in leadership positions. HHS-OCR officials told us that the resolution agreement lists corrective actions that may be considered promising practices. However, a grantee who wanted to learn about these practices would need to know that they exist in the agreement and then review the document to find them. The agency has already identified potential promising practices in some its completed Title IX reviews. Therefore, publishing a separate list of these practices and corrective actions from resolution agreements on its website would require few resources and could benefit grantees. According to Standards for Internal Control in the Federal Government, management should use quality information to achieve its objectives and externally communicate such information to achieve objectives. The vast majority of grantees are reviewed for Title IX compliance infrequently by the five agencies and therefore receive little to no information on such compliance from these agencies. Moreover, while grantees can access completed Title IX reviews on some agencies’ websites, this endeavor would still require grantees to review the written reports in detail to uncover any promising practices. Without clearly publicizing promising practices to the broader grantee community, such as a stand-alone list of practices, DOE, HHS-OCR, and USDA are missing an opportunity to provide quality information to grantees about how best to ensure compliance with Title IX requirements and reduce the likelihood of sexual harassment. The five agencies we reviewed received Title IX complaints, but varied in their efforts to address sexual harassment allegations, including: 1) finalizing procedures for processing Title IX complaints, 2) communicating complete information about the complaint process to grantees, and 3) addressing allegations outside of the Title IX process. Four agencies received three or fewer formal Title IX complaints total from fiscal year 2015 through 2019. Two of the five agencies do not have written procedures for the prompt processing and disposition of Title IX complaints—including allegations of sexual harassment—as required by federal regulations. According to agency officials, all five agencies use their websites as the primary means of communicating Title IX complaint information to grantees and individuals at universities; however, one of the five agencies’ websites does not provide clear guidance for grantees on the basics of the complaint filing processes—such as who can file. Additionally, two agencies have gone beyond the formal Title IX complaint process and also review sex discrimination concerns—including sexual harassment—as a means of improving agency Title IX oversight of university grantees. Title IX affords individuals the ability to file formal complaints of Title IX violations directly to the federal agency providing funding for the program. According to agency officials, the five agencies generally define formal complaints as those that: Are submitted in writing; Are filed within 180 days of the incident—or if ongoing, within 180 days of the last incident—to be considered timely; Provide the name and contact information of the person who is Provide a general description of the person or people injured by the alleged discriminatory act(s) (names of those injured are not required); and Provide a description of the alleged discriminatory act(s) in sufficient detail to enable the agency to understand what occurred, when it occurred, and the basis for the alleged discrimination (sex discrimination in the case of Title IX). All five agencies accept formal Title IX complaints in multiple ways— including at minimum through email and postal mail. From fiscal year 2015 through 2019, four agencies received three or fewer formal complaints (see table 2). Agency officials provided several reasons why they believe agencies receive few formal Title IX complaints: Complaints are more commonly filed with the university or with Education and are rarely directly reported to the agency; Individuals may be unaware of their right to file complaints directly with the agency or how to file such a complaint; and Individuals may fear retaliation or a negative impact on their scientific career (see sidebar). Retaliation At a state university, a graduate student reported her advisor for sexual harassment. The university substantiated her claim and the professor left the university. According to the victim, fellow students upset at the impact of the professor’s departure on their own research and academic careers, retaliated against her. Student-centered retaliation included taking her lab equipment without permission, ostracizing her from social events, and withholding critical information and resources necessary for her research. This retaliation caused her to move her workspace and lose progress on her own work. Officials from DOE, NASA, HHS-NIH, and NSF stated they usually learn about instances of sexual harassment from other sources (e.g. media reports) and rarely from voluntary reporting from universities or other federal agencies. Title IX officials at two universities we interviewed agreed with agency officials about why few formal complaints are filed with agencies. For example, one Title IX official stated that concerns about retaliation for filing a complaint are amplified when there is an agency involved due to concerns over risk to the funding. Officials from NSF, which received the most formal Title IX complaints of the five agencies from fiscal year 2015 through 2019, stated that complaints to their agency have increased in recent years. They could not state definitively the reason for the recent increases, but said it may stem from the increased publicity of sexual harassment cases in STEM—including a Twitter movement known as #MeTooSTEM—along with NSF’s revised grant terms and conditions. DOJ’s regulations provide that federal agencies must establish and publish procedures for the prompt processing and disposition of complaints. While all five agencies specify general requirements for a formal complaint, two do not have clear or updated written procedures for processing and disposing of formal Title IX complaints. Specifically: While DOE’s agency regulations stipulate that the agency will investigate allegations of discrimination under Title IX, agency officials stated that DOE does not currently have written Title IX complaint procedures. In November 2019, the agency provided a preliminary draft outline of its procedures, but officials stated that the agency does not have a timeline for when they may be finished. This is because the agency is devoting its resources to investigating current complaints, according to DOE officials. The website for USDA’s Assistant Secretary for Civil Rights—the office handling complaints across the agency—contains a summary of procedures used to process and investigate discrimination complaints, but a USDA official stated that the procedures need more clarity with regard to the university and research environment. In November 2019, USDA officials highlighted a 1999 USDA Departmental Regulation that addresses processing administrative complaints of discrimination filed against any program or activity receiving financial assistance from USDA. Officials stated that this regulation was revised in fiscal year 2019 and is currently under review for approval. The Departmental Regulations as they stand are outdated, referencing out-of-date organizational responsibilities and department names. Despite the absence of formal complaint procedures for DOE and outdated procedures for USDA, both agencies have evaluated formal Title IX complaints to determine if they meet the necessary criteria for investigation. Specifically, according to agency officials, USDA evaluated and investigated a formal Title IX complaint in fiscal year 2017, and DOE is currently evaluating a complaint to determine if it meets the criteria for a formal complaint. However, without clear and specific guidance for the processing and disposition of complaints, DOE is not complying with DOJ’s regulations—which require federal agencies to establish and publish complaint procedures—and may not be able to consistently and efficiently handle formal Title IX complaints. Moreover, under Standards for Internal Control in the Federal Government, management should implement control activities through policies. For example, management should periodically review procedures for continued relevance and effectiveness in achieving the entity’s objectives or addressing related risks. Without updated complaint procedures, USDA does not have procedures that are aligned with the current structure and oversight responsibilities of the agency. In contrast, NASA, NSF, and HHS-OCR have developed written procedures for how the agency receives, investigates, and resolves formal Title IX complaints. As seen in appendix I, the formal complaint process is extensive and complex, with evaluative benchmarks to be met prior to investigation. According to officials, all five agencies use websites as the main mechanism for communicating information about Title IX complaints to individuals at university grantees. However, USDA’s website is not clear about who can file a Title IX complaint. NASA, HHS-OCR, NSF, and DOE each have a website intended to provide information about filing complaints specifically to individuals involved with agency-funded grants. On these websites, the agencies state that they accept Title IX complaints or sex discrimination complaints, among other types. In contrast, USDA communicates information on its complaints process via a general discrimination website that is not specific to Title IX complaints or to individuals on agency-funded grants, making it unclear who can file Title IX complaints. Specifically, USDA provides information about complaint resolution through its Office of Assistant Secretary for Civil Rights website, but grantees wishing to file a formal complaint would need to do so as a “customer,” a term that is not defined on the agency’s website and that individuals on agency-funded grants may not recognize as including them. A USDA official acknowledged that such individuals may not realize that they can file through the website. USDA officials told us that the agency would consider revising its website to make clear that individuals on USDA grants can file a formal discrimination complaint with the agency. If USDA does not revise its website, the lack of clarity about who is a customer that can submit a complaint may inhibit its ability to obtain information necessary for Title IX oversight. Under Standards for Internal Control in the Federal Government, management should externally communicate the necessary quality information to achieve the entity’s objectives. For example, management may communicate and receive information through established reporting lines, such as websites, from external parties that can help the agency achieve its objectives, such as oversight of Title IX. The lack of clear communication of quality information may reduce the effectiveness of USDA’s Title IX enforcement. Unlike USDA, both HHS-OCR and DOE recently took action to improve website clarity on who can file a Title IX complaint. Specifically: HHS-OCR’s website provides information on filing formal complaints for multiple forms of discrimination—such as race, age, and sex discrimination—and allows formal complaints from all who feel they have been discriminated against by a program or activity that receives funding from any part of HHS. In part due to issues raised during the course of our study, HHS-OCR published several new or updated websites in October 2019—including a Title IX page with university- based examples of entities covered under Title IX and a website on sex-based harassment outlining definitions and examples of what constitutes sex discrimination under Title IX. Before this update, it was not clear if individuals working on HHS-funded grants could file formal Title IX complaints via HHS-OCR’s website. In October 2019, DOE updated its Title IX website to include clear information on the multiple ways individuals can file a formal Title IX complaint with the agency, after we pointed out that this information was missing from DOE’s website, according to officials. The updated website specifies that individuals can notify the agency of a Title IX complaint in person, by email, fax, or mail. Prior to this update, the website only provided information on how to mail the agency a Title IX complaint. In addition to investigating Title IX complaints as required by Title IX, two agencies—HHS-NIH and NSF—go beyond this requirement by also reviewing concerns of sex discrimination—including sexual harassment— and publicly communicating the option for individuals to notify the agency of such concerns outside of the formal Title IX complaint process. “Concerns” are generally defined as information from individuals seeking to inform or notify the agency that sex discrimination has occurred or is occurring, but information is not intended to be a formal Title IX complaint. For example, HHS-NIH established a website, email, and online portal specifically for concerns of sexual harassment, publicly communicating this effort not only on the website, but also in public presentations and official statements. NIH officials stated the agency began reviewing concerns to provide clear channels of communication to NIH. HHS-NIH also developed internal guidance, which is still evolving, for agency staff on how to process concerns from individuals at university grantees through coordination with the grantee. While formal Title IX complaint investigations are agency-led, investigations of sex discrimination concerns, including sexual harassment, filed with HHS-NIH are university-led, with HHS-NIH assessing the university grantee’s response to the allegation to ensure appropriate action is taken to ensure a safe research environment (see appendix II for more details). NSF also publicly communicates the option to notify the agency of concerns of sex discrimination via their Awardee Civil Rights website. In addition to providing details on who should file a formal complaint and how, NSF also provides information on how to notify the agency of concerns and what is done with this information. For example, NSF has initiated a Title IX compliance review for fiscal year 2020 based in part on information contained in a concern it received, according to officials. In fiscal year 2019, HHS-NIH received 93 concerns of sex discrimination and NSF received 47, according to officials from each respective agency (see table 3). In contrast, the remaining agencies—NASA, USDA, DOE, and HHS- OCR—do not publicly communicate the option to notify the agency of concerns of sex discrimination or sexual harassment. Although these agencies stated that they do review all information received—including information from those seeking to notify the agency of concerns—the review is primarily to determine if the information provided meets the agency’s criteria for a formal complaint. Those complaints meeting the criteria for a formal complaint are processed by the agency following the legally required Title IX complaint process. According to officials, the agencies may use the information from concerns to help select a site for a Title IX compliance review. As shown in table 3 above, officials from DOE, USDA, and NASA stated their respective agencies received no concerns of sex discrimination in fiscal year 2019, and HHS-OCR does not track concerns—referred to as communications—that the agency cannot investigate under Title IX, according to HHS-OCR officials. DOE officials stated that the agency received its first sex discrimination concern in fiscal year 2020 and therefore DOE was not aware individuals were looking to notify the agency of concerns. While these agencies accept concerns, they have received few or no concerns and have not publicly communicated that individuals may send concerns to them. The 2018 NASEM report, agency officials, and stakeholders we interviewed all noted the importance of informal ways for individuals to report concerns outside of formal complaint processes. The NASEM report states that formal reporting procedures can re-victimize targets of harassment, and informal procedures—including the acceptance of anonymous complaints—may let them bring concerns forward without fear of retaliation. A stakeholder we interviewed pointed out the arduous nature of agencies’ formal complaint processes, and multiple stakeholders highlighted the difficulty of meeting the federal standard for a Title IX violation. All five agencies agreed that informal information— such as concerns—is helpful in providing the agency with additional information about the research environment on campus. Of the 140 total concerns of sex discrimination received in fiscal year 2019 by the agencies in our study, all were filed with either NSF or NIH. A comparison of the number of concerns and formal complaints received by the agencies shows that the five agencies as a whole received more than three times as many concerns in 1 year as they did formal complaints in 5 years. Title IX specifies federal agencies’ Title IX oversight responsibilities— including enforcing Title IX compliance at the universities they fund. Under Standards for Internal Control in the Federal Government, management should externally communicate the necessary quality information to achieve the entity’s objectives. For example, management may communicate and receive information through established reporting lines from external parties—in this case, through formal complaints and concerns—which can help the agency achieve its objectives, such as oversight of Title IX. By publicly communicating to individuals that they may notify the agency of a concern of sex discrimination outside of the formal Title IX complaint process, NASA, USDA, DOE, and HHS-OCR could receive additional information necessary for appropriate Title IX oversight. According to NSF officials, concerns not only reveal potential issues with the climate at an awardee university, they also aid in Title IX oversight by alerting the agency to possible Title IX violations a university may need to notify the agency of under the grant terms and conditions. In addition to reviewing concerns of sex discrimination from individuals at university grantees, NSF receives notifications directly from university grantees. In 2018, NSF modified its grant terms and conditions to require university grantees to notify the agency if there is a finding of sexual harassment against a principal investigator (PI) or co-PI on an NSF-funded grant, or if administrative action was taken against a PI or co-PI due to an allegation or complaint of sexual harassment. In 2019, NSF established written procedures to review these notifications from university grantees to determine if the university handled the matter adequately and appropriately, and if further action is needed by NSF. NSF received 13 notifications from university grantees through the new grant terms and conditions in fiscal year 2019, according to agency officials. NASA, similar to NSF, proposed changes to its grant terms and conditions. NASA published its notice of the proposed change in July 2019. However, according to NASA officials, the Office of Science and Technology Policy requested that NASA consult with the National Science and Technology Council’s joint committee’s subcommittees— Safe and Inclusive Research Environments Subcommittee and Coordinating Administrative Requirements for Research Subcommittee— prior to moving forward with finalizing the terms and conditions. NASA consulted with the Office of Science and Technology Policy in December 2019 and received concurrence to move forward with finalizing the change to its terms and conditions, according to NASA officials. On March 10, 2020, NASA published a final notice of its new terms and conditions for grants. Upon implementation, the new terms and conditions requires, among other things, grantees to report to NASA any findings or determinations of sexual harassment, other forms of harassment, or sexual assault regarding a NASA funded PI or co-PI. The reporting requirement will be applied to all new NASA awards and funding amendments to existing awards made on or after the effective date—30 days from the date of the publicized notice. HHS differs from the other four agencies in that formal complaints and concerns are handled by two different components, which do not communicate with each other regarding information on sexual harassment, according to officials from both HHS-OCR and HHS-NIH. HHS-OCR—the enforcement authority of the agency—has the authority to conduct Title IX compliance reviews and investigate formal Title IX complaints. However, as previously mentioned, HHS-NIH—the grant- making component—has independently developed its own avenue for receiving concerns of sex discrimination, including sexual harassment. The Standards for Internal Control in the Federal Government state that management should internally communicate the necessary quality information to achieve its objectives. This communication includes conveying information down and across reporting lines to allow staff to perform key roles in achieving objectives and addressing risks. There are no formal procedures within HHS for communicating information across the agency components regarding Title IX complaints, concerns of sex discrimination including sexual harassment, or Title IX compliance, according to officials from both HHS-NIH and the HHS-OCR. An official from HHS-OCR stated that the department already shares broad information about findings of Title IX violations and completed Title IX compliance reviews via a listserv to HHS employees and stakeholders who subscribe, but HHS-NIH officials stated they were not aware of this information. HHS-NIH officials also stated they do not share information with HHS-OCR regarding concerns of sex discrimination, including sexual harassment, received by HHS-NIH or actions taken in response to these concerns. According to HHS-NIH officials, grantees are expected to provide safe and healthy working conditions—a term and condition of the grant—and therefore if harassment threatens the research environment, this is a potential violation of grant terms and conditions and officials stated that HHS-NIH has the authority to handle it. Establishing procedures for communicating grantee sexual harassment findings could improve efforts by both HHS-NIH and HHS-OCR to prevent sexual harassment at universities. For example, HHS-OCR could use HHS-NIH data to aid in selecting grantees for Title IX compliance reviews. Additionally, HHS-NIH could use HHS-OCR’s compliance review findings to inform oversight of NIH grants—including modifying university grantees’ grant terms and conditions if there were findings of non- compliance. Officials from HHS-OCR agreed that information on concerns of sex discrimination, including sexual harassment, from HHS-NIH would be helpful. HHS-NIH officials also agreed that information sharing may be useful for cross agency awareness, but HHS-NIH officials asserted that a formal agreement would be necessary to ensure privacy when sharing information. HHS-NIH officials did not provide any further details on what should be included in a formal agreement for sharing information on sex discrimination concerns, including sexual harassment. All five agencies have taken additional steps beyond Title IX compliance requirements to address sexual harassment by university grantees. As we reported in June 2019, all five agencies have developed and communicated grantee sexual harassment prevention policies, with some providing more detailed guidance than others. All five agencies have also established grantee sexual harassment prevention efforts beyond those required by Title IX, to varying degrees. For example, as we noted above, HHS-NIH launched a website to receive concerns of sex discrimination including sexual harassment, and NSF and NASA have modified grant terms and conditions that require universities to report findings of sexual harassment. All of the agencies we reviewed established and communicated their sexual harassment prevention efforts to grantees within the last 3 fiscal years, and most of them have continued to update and communicate them since we last reported on their efforts in June 2019 (see sidebar for an agency example and appendix III for more information on agencies’ efforts). Agencies have taken steps to create goals for and evaluate some of their individual grantee sexual harassment prevention efforts. However, four of the five agencies have not created goals for all prevention efforts. In addition, none of the five agencies have a plan designed to assess progress toward achieving those goals, including methods to regularly monitor and evaluate their various grantee sexual harassment prevention efforts together—both those that are required by Title IX and those that go beyond these requirements. USDA created a poster for grantees that describes how the agency prohibits discrimination in all forms, including discrimination on the basis of sex. The poster also provides information on how to file a discrimination complaint with USDA by phone, mail, fax or email. USDA requires all grantees to prominently display the poster in all offices where there is a USDA presence and where it may be read by customers. USDA also requires that the poster be a specific size. NSF and USDA-NIFA do not have goals for all of their grantee sexual harassment prevention efforts, according to officials. In addition, NASA and DOE have or are in the process of establishing goals for some prevention efforts related to Title IX requirements, while HHS-NIH has created goals for all of their grantee sexual harassment prevention efforts. NASA and DOE have goals or plan to establish goals for sexual harassment prevention efforts required by Title IX—such as compliance reviews—but they lack goals that include all other sexual harassment prevention efforts for university grantees. For example, according to NASA officials, NASA’s strategic plan has goals for equal opportunity and diversity and inclusion for the NASA workforce and grantees, and it includes a goal for the agency to promote equal opportunity for grantees and to encourage them to use best practices identified by NASA. To measure progress toward this goal, NASA officials told us that the agency plans to establish a timeline to track the percentage of Title IX compliance activities completed by grantees. However, NASA has not established goals for its other grantee sexual harassment prevention initiatives. In addition, DOE officials told us that they are in the process of establishing a goal for the number of Title IX compliance reviews they conduct each year, but DOE does not have goals or a plan for evaluating other DOE grantee sexual harassment prevention initiatives. In contrast, HHS-NIH’s Working Group of the Advisory Committee to the NIH Director has created goals for HHS-NIH’s various grantee sexual harassment prevention efforts and steps to achieve them. These goals include assessing the current state of sexual harassment allegation investigation, reporting, remediation, and disciplinary procedures at NIH- funded organizations; advising grantees on oversight, accountability, and reporting measures that will encourage a reduction in, and prevention of sexual harassment; and developing strategies for encouraging research on anti-harassment policies, procedures and training, and measures and evaluations of their effectiveness. HHS-NIH developed recommendations for the steps needed to achieve these goals, including immediate, actionable efforts and longer-term efforts to change the culture within NIH and at universities to end sexual harassment. HHS-NIH officials published a final report and recommendations in December 2019. The report recommended that HHS-NIH establish a hotline and a web- based form for reporting sexual harassment and inappropriate behavior by any principal investigator or key personnel funded by HHS-NIH, and that HHS-NIH also conduct an analysis of the prevalence and antecedents of sexual and gender harassment in order to develop interventions that address goal-specified gaps, among others. In addition to most of the agencies not having goals for all of their grantee sexual harassment prevention efforts, none of the five agencies have a plan to measure progress toward achieving those goals, including methods to regularly monitor and evaluate them all. Some of the agencies have taken steps toward conducting evaluations of some—but not all—of their grantee sexual harassment prevention efforts: Evaluations of policies and procedures. Three agencies—NASA, NSF, and DOE—have evaluated or are beginning to evaluate some of their sexual harassment policies and procedures for university grantees. NASA officials said they conduct evaluations every five years for all of their agency’s civil rights compliance and complaints procedures, including their Title IX compliance review procedures. NSF is also developing an evaluation plan for its new sexual harassment reporting requirements and how they have affected grantees. NSF officials said that they have an evaluation team in place, which has outlined an approach for evaluating the new grant terms and conditions and has begun gathering information from universities. In addition, DOE officials told us that they are currently reviewing other agencies’ policies and using them as a benchmark as they draft their own grantee policies. However, agencies have not periodically evaluated all of their own sexual harassment policies and procedures related to university grantees. Agencies provided examples of evaluations of grantee or employee prevention policies, rather than an evaluation of their own policies created for university grantees. It is unclear why agencies have not yet established methods to evaluate all of their sexual harassment prevention efforts for university grantees, and we recognize the challenge in doing so. Yet agencies have found ways to evaluate the policies of other entities. For example, officials from HHS-OCR told us that they use information from past compliance reviews to improve their compliance review and resolution requirements for grantees. However, compliance reviews are an evaluation of the university grantee’s sexual harassment prevention policies and procedures, not HHS’s. In addition, when asked whether HHS-NIH evaluates its grantee sexual harassment policies, HHS-NIH officials did not give any examples of evaluations of their own policies created for university grantees. Instead, they gave an example of a climate survey they administered to their employees for work-life climate and harassment. Title IX officials from two universities and one university system all said none of the five agencies had asked for their opinions on how effective the agencies’ sexual harassment prevention policies for grantees are. Nor had they requested suggestions for improvement, even though at least one of the five agencies had been in direct contact with two of these officials for a recent compliance review. Evaluations of communication mechanisms. None of the five agencies periodically evaluate the mechanisms for how they communicate their sexual harassment prevention policies and procedures to individuals at universities receiving federal grants. Instead, agencies rely on general efforts to evaluate their website or are developing plans for such an evaluation. As a result, the agencies do not know the extent to which their various communication mechanisms are working and whether students, researchers, faculty, and university officials are getting the information they need from these mechanisms. For example, as previously stated, all five agencies use their website as the main mechanism to communicate information about Title IX complaint procedures to individuals at universities receiving federal grants. While agencies have taken steps to add more information to their websites for individuals at universities receiving federal grants, we found some of the agencies’ websites difficult to navigate. Even when key content existed, it was sometimes spread across multiple sections of the website or buried in supplemental materials, or in one case, associated with an incorrect destination page. For example, NSF officials stated that they prefer formal Title IX complaints be filed via their online complaint portal. However, this tool is not linked to the tab of the website discussing how to file complaints. Additionally, HHS-OCR’s newly-developed Title IX and sex-based harassment websites are not referenced or linked to the information on laws and regulations enforced by HHS-OCR or the complaints page. Rather, from the HHS-OCR home page, one would have to know to click on the “Special Topics” link to find links to the two new websites or find the link to the “Sex-based Harassment” page embedded within the new Title IX website. All of the agencies acknowledged issues with their websites. For example, NSF officials acknowledged that their agency’s website may not be user-friendly to individuals at universities—such as students—and is in the process of revising the website to increase ease of use. None of the five agencies have periodically evaluated this key communication tool at this time. NASA officials said that they evaluate their website for grantees, but these evaluations are not systematic and have not specifically focused on their sexual harassment prevention efforts for grantees. HHS-NIH officials said that they receive some feedback on the agency’s webpages, and the agency keeps track of website user satisfaction to improve their quality in general, but this effort is not specific to evaluating how HHS-NIH communicates information on sexual harassment prevention to grantees. NSF officials said they are planning to conduct an evaluation on the effectiveness of their communication efforts with their grantee community and will include actions that result from that evaluation in NSF’s corrective action plan. As previously mentioned, HHS-OCR and DOE recently took action to improve website clarity on who can file a Title IX complaint, in part due to issues raised during the course of our study. Evaluating the effectiveness of their communication mechanisms is important, as agencies may not be clearly communicating their sexual harassment prevention policies and procedures to their intended audiences. Nor can they be sure these policies and procedures are reaching the right university officials. For example, Title IX officials from two universities and one university system said that they had not received any information from the five agencies on their sexual harassment prevention policies for grantees. Two Title IX officials stated that, even if this information is already provided to the university departments or offices conducting scientific research, it should also be given to the university’s Title IX office, with one official noting they are the part of the university responsible for overseeing compliance with sexual harassment policies and procedures under Title IX. Title IX university officials also told us that the federal agencies providing their grants had never provided them with information on agencies’ policies and procedures for how individuals at their institutions could file sexual harassment complaints. One Title IX university official described how they would not know how to tell someone to proceed if they wanted to file a complaint with the agency funding their research project. Officials from all five agencies acknowledged the value of evaluating their grantee sexual harassment prevention efforts and noted that they may be able to conduct such evaluations in the future. In addition, four of the agencies have a general goal to prevent sexual harassment by their university grantees, and all have recognized the need to move beyond their current grantee sexual harassment policies and procedures. As we reported in June 2019, their completed or planned actions include modifying current department- level or agency-wide policy statements to include more specific definitions and examples of sexual harassment and strengthening requirements for their university grantees to report on findings of sexual harassment. The 2018 NASEM report also noted that, while it is clear that the agencies are concerned about sexual harassment in STEM, it is not yet apparent whether and how actions such as their new policy statements will translate into meaningful action. Standards for Internal Control in the Federal Government state that management should define objectives clearly to enable the identification of risks and define risk tolerances; for example, in defining objectives, management may clearly define what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement. Standards for Internal Control in the Federal Government also state that federal agencies should establish and operate monitoring activities to evaluate results, select and periodically evaluate methods of communication, and define objectives in specific and measurable terms. We recognize that the agencies’ civil rights or diversity offices are responsible for enforcing and overseeing a number of other civil rights laws, Executive Orders, regulations, and policy directives for grantees and their own employees, and that sexual harassment prevention efforts for grantees are just some of their many discrimination prevention initiatives. We also recognize that most agencies’ grantee sexual harassment prevention efforts are new, and in some cases, still under development. However, establishing goals and an overall plan to assess progress toward achieving those goals—including methods to regularly monitor and evaluate sexual harassment prevention policies and communication mechanisms— would better position the agencies to effectively coordinate and integrate such activities. It would also help them holistically evaluate all their efforts—both those that are required by Title IX and those that go beyond these requirements—to help grantees prevent sexual harassment at universities, determine whether their policies and procedures are reaching the populations they intend to receive the information, and allow them to more effectively target outreach if they find that there are deficiencies. DOJ’s Quarterly Title IX STEM discussion group provides a forum for the five agencies to collaborate and share information on Title IX compliance. While DOJ has implemented four of the six relevant leading practices on federal interagency collaboration, it has not fully implemented two key practices—agreeing on agency roles and responsibilities and developing mechanisms for monitoring, evaluating, and reporting collaborative efforts. Outside of the discussion group, the five agencies have taken collaborative steps to address the culture and climate for women in STEM. DOJ’s Quarterly Title IX STEM discussion group facilitates collaboration and shares best practices on Title IX enforcement across the five agencies. Collaboration can be broadly defined as any joint activity that is intended to produce more public value than could be produced when organizations act separately. Since February 2016, after reconstituting the Quarterly Title IX STEM discussion group, DOJ has held quarterly group meetings to share information on Title IX enforcement. According to DOJ officials, the agencies discuss several topics, including: Strategies for conducting Title IX compliance reviews, including joint Strategies for investigating Title IX complaints General discussion of Title IX complaints, including sexual Title IX court cases and case history Officials at the five agencies agreed the group is useful to coordinate and share information on Title IX, for example, by avoiding duplication in compliance reviews and complaint investigations. Indeed, when multiple agencies received the same Title IX complaint, the three agencies collaborated with DOJ to determine which one would handle the complaint, according to officials. DOJ also provides technical assistance and training on Title IX enforcement, according to officials. Some agency officials identified steps that could potentially improve collaboration within the group, including: Clarifying and documenting the group’s purpose, scope, and roles and responsibilities to ease transition of new agency staff and leadership Incorporating more specific topics related to sexual harassment in meeting agendas Involving all federal agencies that fund STEM research at universities. Although agencies are not required to follow leading practices for interagency collaboration, doing so can enhance and sustain such collaboration, thereby improving performance and results. DOJ officials told us the agency has adopted leading practices for interagency collaboration as part of the group. Based on information from DOJ, we found the agency’s actions were consistent with four out of six of the relevant leading practices we have identified for collaborating across agencies. We also found that DOJ could take additional steps to fully adopt the remaining two leading practices, as shown in table 4. Without implementing the two interagency collaboration leading practices, DOJ is missing an opportunity to enhance and sustain collaboration among the five agencies we reviewed as they continue to address the problem of sexual harassment at universities. All five agencies reported taking collaborative steps with universities and federal agencies to address the culture and climate for women in STEM. For example, in 2019, HHS-NIH established a working group with university experts to collaboratively assess the current state of procedures for sexual harassment allegation investigation, reporting, remediation, and discipline at NIH-funded organizations. In December 2019, the working group made several recommendations. For example, it recommended that NIH require each principal investigator and key personnel on an NIH grant attest that they have not been found to have violated their institution’s code of professional conduct, including having a finding of sexual harassment, for a determined period of time. The working group also recommended that NIH create a parallel process to treat professional misconduct, including sexual harassment, as seriously as research misconduct. In addition, DOE, NASA, NSF, and USDA collaborated with universities at conferences and meetings. According to DOE officials, the agency attends the annual conference of university Title IX administrators, where participants discuss issues related to Title IX, sexual harassment, and sexual assault. In 2016, NASA held a conference to help universities address sexual harassment and share best practices to increase participation of underrepresented populations in STEM education (see sidebar). According to NASA officials, attendees included university presidents, deans, and provosts, as well as NASA leaders. The agency is planning another conference in 2020, according to NASA officials. NSF presented information on Title IX and their policies and procedures at numerous conferences and meetings in 2018 and 2019, according to officials. USDA-NIFA served on the planning committee and participated in a conference with public land grant universities to discuss diversity and inclusion in 2018. According to officials, USDA is considering participation in future events. Such efforts to go beyond compliance reviews and to address the larger culture and climate of STEM research are consistent with the 2018 NASEM report, which states that “adherence to legal requirements is necessary but not sufficient to drive the change needed to address sexual harassment.” increase participation in NASA business and grant opportunities, and education programs. address important issues related to America’s research environment. The joint committee also established the Safe and Inclusive Research Environments Subcommittee, in which DOE, HHS (including NIH), NASA, and NSF participate, along with other federal agencies and offices. USDA-NIFA is a member of the joint committee but does not participate in the Safe and Inclusive Research Environments Subcommittee; instead, USDA’s Agricultural Research Service participates in the subcommittee. The goals and planned actions of the joint committee and subcommittee have not been determined yet, according to Office of Science and Technology Policy officials. Office of Science and Technology Policy officials told us in December 2019 that its work plan is complete, but there are no plans to release it publicly since it is a deliberative document. The subcommittee is broadly focused on preventing harassment in research environments. Sexual harassment in higher education is degrading and illegal. In 2017 alone, the media covered over 97 allegations of sexual harassment at institutions of higher education with some of the most high-profile cases occurring in the fields of science, engineering and medicine, according to the National Academies of Sciences, Engineering, and Medicine. Federal agencies, in connection with the billions of dollars in research funding they provide to universities and other institutions each year, are required to enforce Title IX—prohibiting sex discrimination, including sexual harassment—at these universities. As part of their enforcement responsibilities, all five agencies have conducted the required Title IX compliance reviews, but three agencies—DOE, HHS-OCR, and USDA— are missing an opportunity to share promising practices from their Title IX compliance reviews with the broader grantee community. Given that these agencies conduct compliance reviews at only a handful of the hundreds of grantees they fund in any given year, the vast majority of grantees receive little to no information on Title IX compliance reviews from these agencies. Another tool federal agencies can use to address sexual harassment is the prompt processing and disposition of Title IX complaints from students and employees. Although all five selected agencies received Title IX complaints, DOE and USDA have not finalized and published complaint procedures, as required by DOJ’s regulations. Furthermore, USDA does not provide clear information about the complaint process on its website—its primary means of communicating information to individuals and grantees. As a result, the agency may be missing the opportunity to better serve individuals seeking relief from sexual harassment at universities. Federal agencies can also review information from individuals seeking to notify the agency of a concern related to sex discrimination—including sexual harassment—in an informal manner outside of the Title IX complaint process. However, only HHS-NIH and NSF communicate the option to submit concerns, and only HHS-NIH has a written process for reviewing such concerns. In a single year, these concerns outnumbered formal Title IX complaints received by all the agencies over 5 years. The 2018 NASEM report, agency officials, and stakeholders we interviewed noted the importance of informal ways for individuals to report concerns outside of formal complaint processes, which can protect an individual from retaliation, alert agencies to possible Title IX violations, and help agencies select sites for Title IX compliance reviews. Two HHS components—NIH and OCR—do not share sexual harassment complaint information with each other. This poses the risk that HHS-NIH will be unaware of situations in which HHS-OCR finds non-compliance with Title IX and may approve a STEM research grant for that university. It also raises the possibility that NIH will receive concerns about a university that may warrant a Title IX compliance review, but the Office for Civil Rights may be unaware of these concerns. Establishing clear goals and an overall plan can help agencies assess progress and manage change, including, in this case, the creation of new sexual harassment prevention efforts for grantees. Although all five agencies have established a variety of prevention efforts, they have done so without a plan, and without methods to evaluate their policies and how they communicate them. As a result, agencies do not have a way to measure progress toward preventing sexual harassment at their university grantees, including how or whether these efforts are helping university grantees and individuals who have been subject to harassment. Finally, interagency coordination can help improve the results of agency activities. DOJ has not fully adopted two key interagency collaboration leading practices for its interagency working group. Without doing so, the agency is missing an opportunity to enhance and sustain collaboration among agencies as they continue to address the problem of sexual harassment at universities. We are making 17 recommendations, including four to DOE, one to DOJ, four to HHS, two to NASA, one to NSF, and five to USDA. Specifically: The Secretary of the Department of Agriculture should direct the Assistant Secretary for Civil Rights to publicize promising practices for Title IX compliance on its websites for their university grantees. (Recommendation 1) The Secretary of Energy should direct the Director of the Office of Economic Impact and Diversity to publicize promising practices for Title IX compliance on its websites for their university grantees. (Recommendation 2) The Secretary of the Department of Health and Human Services should direct the Director for the Office for Civil Rights to publicize a stand-alone list of promising practices for Title IX compliance on its websites for their university grantees. (Recommendation 3) The Secretary of Energy should direct the Director of the Office of Economic Impact and Diversity to finalize and publish Title IX complaint procedures, consistent with DOJ’s regulations. (Recommendation 4) The Secretary of the Department of Agriculture should direct the Assistant Secretary for Civil Rights to finalize and publish revised Title IX complaint procedures. (Recommendation 5) The Secretary of the Department of Agriculture should direct the Assistant Secretary for Civil Rights to clarify on its website that individuals on USDA-funded grants can file Title IX complaints through the Assistant Secretary for Civil Rights—including clarifying who is considered “customers.” (Recommendation 6) The Secretary of the Department of Health and Human Services should direct the Director for the Office for Civil Rights to assess the feasibility of receiving and reviewing concerns of sex discrimination— including sexual harassment—and communicating to individuals on agency-funded grants the option to notify the agency of these concerns, outside of the Title IX complaint process. (Recommendation 7) The Secretary of Energy should direct the Director of the Office of Economic Impact and Diversity to assess the feasibility of receiving and reviewing concerns of sex discrimination—including sexual harassment—and communicating to individuals on agency-funded grants the option to notify the agency of these concerns, outside of the Title IX complaint process. (Recommendation 8) The Administrator of NASA should assess the feasibility of receiving and reviewing concerns of sex discrimination—including sexual harassment—and communicating to individuals on agency-funded grants the option to notify the agency of these concerns, outside of the Title IX complaint process. (Recommendation 9) The Secretary of the Department of Agriculture should direct the Assistant Secretary for Civil Rights to assess the feasibility of receiving and reviewing concerns of sex discrimination—including sexual harassment—and communicating to individuals on agency- funded grants the option to notify the agency of these concerns, outside of the Title IX complaint process. (Recommendation 10) The Secretary of the Department of Health and Human Services should direct the Director for the Office for Civil Rights and the Director of NIH to develop and implement formal procedures for sharing relevant information about Title IX (compliance reviews, violations, and complaints) and sex discrimination concerns, including sexual harassment. For example, HHS components should internally share information regarding findings of Title IX violations, concerns of sex discrimination, including sexual harassment, and Title IX compliance review reports. (Recommendation 11) The Secretary of Energy should establish goals and an overall plan to assess all of the agency’s sexual harassment prevention efforts for their university grantees, including methods to regularly monitor and evaluate its sexual harassment prevention policies and communication mechanisms (e.g. Title IX or sex discrimination websites). (Recommendation 12) The Secretary of the Department of Health and Human Services should establish goals and an overall plan to assess all of the agency’s sexual harassment prevention efforts for their university grantees, including methods to regularly monitor and evaluate its sexual harassment prevention policies and communication mechanisms (e.g. Title IX or sex discrimination websites). (Recommendation 13) The Administrator of NASA should establish goals and an overall plan to assess all of the agency’s sexual harassment prevention efforts for their university grantees, including methods to regularly monitor and evaluate its sexual harassment prevention policies and communication mechanisms (e.g. Title IX or sex discrimination websites). (Recommendation 14) The Director of NSF should establish goals and an overall plan to assess all of the agency’s sexual harassment prevention efforts for their university grantees, including methods to regularly monitor and evaluate its sexual harassment prevention policies and communication mechanisms (e.g. Title IX or sex discrimination websites). (Recommendation 15) The Secretary of the Department of Agriculture should establish goals and an overall plan to assess all of the agency’s sexual harassment prevention efforts for their university grantees, including methods to regularly monitor and evaluate its sexual harassment prevention policies and communication mechanisms (e.g. Title IX or sex discrimination websites). (Recommendation 16) In consultation with DOE, HHS, NASA, NSF, and USDA, the Assistant Attorney General for the Department of Justice should direct the responsible Civil Rights Division sections to fully adopt two federal interagency leading practices—agree on agency’s roles and responsibilities and develop mechanisms to monitor, evaluate, and report results of collaborative efforts, for its Quarterly Title IX STEM discussion group. (Recommendation 17) We provided a draft this report to DOE, DOJ, Education, HHS, NASA, NSF, the Office of Science and Technology Policy, and USDA for review and comment. We received written comments from the Departments of Agriculture, Energy, Health and Human Services, Justice, as well as NASA and NSF that are reprinted in appendixes IV through IX, and summarized below. Education did not have comments on the draft report, but it provided technical comments, which we incorporated as appropriate. The Office of Science and Technology Policy stated that it did not have comments on the draft report. All six of the agencies and departments to which we made recommendations stated that they agreed with the recommendations and most provided technical comments, which we incorporated as appropriate. The agencies’ comments are summarized below: In the Department of Agriculture's written comments, reproduced in appendix IV, the department agreed with all five recommendations. USDA outlined actions for improving the complaint process and communication with university grantees. For example, the department stated that its regulation for processing complaints is currently in the clearance process for publication. In addition, USDA will reach out to other agencies within the quarterly Title IX STEM discussion group to assess best practices for monitoring and evaluating sexual harassment prevention policies and communication mechanisms. In DOE's written comments, reproduced in appendix V, the department agreed with all four recommendations. DOE plans to publicize a promising practice guide on its website, publish complaint procedures, and evaluate the feasibility of receiving and reviewing concerns of sex discrimination, including sexual harassment. The department estimates completion by the end of calendar year 2020. DOE will establish goals for prevention efforts and an overall plan by the end of January 2021 and August 2021, respectively. In HHS's comments, reproduced in appendix VI, the department agreed with all four recommendations. In response to one recommendation, HHS stated that HHS-OCR and HHS-NIH would review the current procedure, and develop and implement, as necessary, formal procedures for sharing relevant information about Title IX and sexual harassment concerns. However, the department also noted that it did not \"share GAO's supposition that coordination of Title IX enforcement between HHS-OCR and HHS-NIH raises privacy concerns.\" Therefore, we removed this portion of our recommendation. As stated in the report, it was HHS-NIH officials who asserted that a formal agreement is needed to ensure privacy when sharing information, in particular sexual harassment concerns. For another recommendation, the department stated that HHS-NIH has established goals and will develop a plan to assess progress towards achieving these goals, and that HHS-OCR will also develop a plan for its Title IX enforcement and outreach efforts. In DOJ's written comments, reproduced in appendix VII, the department generally agreed with our recommendation. DOJ stated that the department is prepared to delineate the agencies' roles and responsibilities within the interagency group (quarterly Title IX STEM discussion group), as recommended. DOJ also plans to develop an enhanced process for evaluating, monitoring, and reporting on the group's collaborations in enforcing Title IX that is achievable within its current resource allocation, or if more resources become available. In NASA's comments, reproduced in appendix VIII, the agency concurred with our two recommendations. NASA stated that the agency plans to assess the feasibility of receiving and reviewing concerns of sex discrimination and harassment and estimates completion by September 20, 2020. Through the interagency process, NASA will also adopt the policies and procedures developed by the National Science and Technology Council, overseen by the Office of Science and Technology Policy. In NSF's written comments, reproduced in appendix IX, the agency agreed with our recommendation for goals and an overall plan to assess sexual harassment prevention efforts. NSF is embarking on an assessment process to improve its policies and practices continually in order to achieve the goal of safe and inclusive research environments. We are sending copies of this report to the appropriate congressional committees; the Secretaries of Agriculture, Education, Energy, and Health and Human Services; the Directors of the National Science Foundation and the Office of Science and Technology Policy; the Administrator of the National Aeronautics and Space Administration; the Attorney General for the Department of Justice; and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6888 or neumannj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix X. While Department of Justice (DOJ) regulations require federal agencies to establish and publish complaint procedures, according to the DOJ Title IX Legal Manual, agency regulations with respect to procedures for the investigation of complaints of discriminatory practices are typically brief and lack details as to the manner or timetable for such an inquiry. The National Science Foundation (NSF) and the National Aeronautics and Space Administration (NASA) have developed complaint manuals in addition to the agencies’ Title IX regulations. According to NSF and NASA officials, both agencies follow the same general processes as those published in DOJ guidance or the Department of Education’s Investigative Manual for the prompt processing and disposition of complaints. Figure 3 is a visualization of the general Title IX complaint process described in NASA’s and NSF’s complaint manuals. Formal complaints can conclude in one of four ways: 1) dismissed for a variety of reasons—such as untimeliness or lack of information; 2) referred to another agency based on jurisdiction or authority; 3) resolved through a voluntary resolution process; or 4) resolved via an investigation and formal finding—either supporting the allegation or not. While the formal complaint may be alleging a discriminatory act against an individual, agency investigations focus on the university grantee’s compliance or non-compliance with Title IX. According to NASA and NSF officials, if there is a finding of non-compliance with Title IX, the onus is on the university grantee to take actions to come into compliance—which may include disciplinary action against the harasser. In recent years, the National Institutes of Health within the Department of Health and Human Services (HHS-NIH) has publicly addressed the agency’s efforts to prevent sexual harassment in science and elevate the seriousness with which the agency takes this issue through action. While already receiving information of sex discrimination concerns, including sexual harassment, from relationships built with institutions—including universities, in March 2019, HHS-NIH launched an email address to receive concerns about sexual harassment directly from individuals involved in HHS-NIH funded projects at universities. Shortly after, in June 2019, HHS-NIH also created an online portal in response to user feedback requesting a method of anonymous reporting. HHS-NIH developed preliminary internal guidance for staff regarding the process for handling concerns (see figure 4). According to officials, in order to review a concern, HHS-NIH needs basic information about the allegation, including: First and last name of the person who may have committed Institution that employs that person Brief description of the incident HHS-NIH notifies universities of the concern and may request details on the allegation and the university’s response to the allegations, according to officials. As part of this process, HHS-NIH assesses the university grantee’s response to ensure it is taking appropriate actions to ensure a safe research environment—altering the grant terms and conditions if needed to remove or replace key grant personnel. For example, in 2018, HHS-NIH followed up on sexual harassment-related concerns at more than 20 universities. According to a 2019 HHS-NIH Director Statement, this follow-up resulted in the replacement of 14 principal investigators named on NIH grant awards, disciplinary actions taken by awardee universities against 21 principal investigators—including termination of employment—and removal of 14 individuals from peer review. According to HHS-NIH officials, in fiscal year 2019 HHS-NIH received 93 concerns regarding sexual harassment. HHS-NIH does not just review allegations against personnel already funded by HHS-NIH, but also assesses if the allegations are against applicants for HHS-NIH funding. If a principal investigator or co-principal investigator listed on an application for an HHS-NIH grant is named in an allegation, HHS-NIH works with the institution to gather more information about the allegation in the context of HHS-NIH funded research. While the institution is conducting an internal investigation into the allegations, they may request to change the principal investigator or remove a co-principal investigator listed on the application. This may be a temporary or permanent action depending on the circumstances and the institution’s findings. This appendix contains a summary of the five agencies’ sexual harassment prevention efforts for university grantees or individuals at universities receiving federal grants as of December 2019. This summary indicates the implementation status for each agency’s efforts, and whether they were complete, in progress or partially implemented, or not reported. These efforts are grouped in three categories: 1) activities required by Title IX, 2) activities beyond those required by Title IX, and 3) evaluation activities (see figures 5, 6, and 7, respectively). John Neumann, (202) 512-6888 or neumannj@gao.gov In addition to the individual named above, Mark Gaffigan (Managing Director), Melissa Emrey-Arras (Director), Robert Marek (Assistant Director), Michelle St. Pierre (Assistant Director), Nkenge Gibson (Analyst-in-Charge), Nora Adkins, Caitlin Cusati, Cindy Gilbert, Kristy Kennedy, Anika McMillon, Kristen Pinnock, Amanda Postiglione, Janay Sam, and Benjamin Shouse made key contributions to this report.", "summary": "Sexual harassment is degrading and illegal. Studies show it has a negative effect on the ability of women to engage in research at the same level as men. Title IX prohibits sexual harassment and other forms of sex discrimination in education programs that receive federal funding, and federal agencies are required to enforce the law at universities they fund. In fiscal year 2018, the most recent year for which data were available during GAO's review, U.S. universities were awarded about $27 billion in federal grants for STEM research. GAO was asked to review federal efforts to help prevent sexual harassment at universities that receive such grants. This report examines, among other things, (1) how selected federal agencies receive, investigate, and resolve Title IX complaints; (2) the extent to which selected agencies have established an overall plan for their sexual harassment prevention efforts for university grantees, including for communicating and evaluating these efforts and (3) the extent to which selected agencies collaborate on efforts to prevent sexual harassment at universities they fund for STEM research. GAO reviewed agencies' relevant regulations and documentation and interviewed agency officials. The five agencies GAO reviewed provided approximately 80 percent of federal science, technology, engineering, and mathematics (STEM) research grants since fiscal year 2015. From fiscal year 2015 through 2019, four of the five agencies received few complaints—including sexual harassment—under Title IX from individuals at universities. Inconsistent with federal regulations implementing Title IX, two of the agencies—the Departments of Energy (DOE) and Agriculture (USDA)—lack finalized procedures for complaints and thus cannot ensure they are consistently handling complaints. Sex-discrimination concerns—including sexual harassment—can also be raised by individuals outside of the Title IX complaint process (see table). However, only two agencies—the National Science Foundation (NSF) and Department of Health and Human Services (HHS)—publicly communicate the option to notify them of concerns. The other three—DOE, the National Aeronautics and Space Administration (NASA), and USDA—received no concerns in fiscal year 2019 and may be missing opportunities to obtain information for Title IX oversight. All five agencies have established grantee sexual harassment prevention efforts beyond those required by Title IX. However, none of the agencies have goals and plans for all of their efforts, and thus they lack clear ways to evaluate how well these efforts are working and to identify any needed improvements. They may also be missing opportunities to coordinate and integrate prevention activities. Additionally, the Department of Justice (DOJ) reconstituted an interagency discussion group on Title IX in 2016, where all five agencies share information about their activities. However, DOJ has not fully adopted two leading practices for collaboration: agreeing on agency roles and responsibilities and developing mechanisms to monitor, evaluate, and report collaborative efforts. Officials at one agency said clarifying agencies' roles and responsibilities would improve the group. Adopting leading practices would help enhance and sustain collaboration. GAO is making 17 recommendations to the five agencies funding STEM research and DOJ, including to finalize and publish complaint procedures, establish goals and an overall plan for prevention efforts, and fully adopt two collaboration leading practices. The agencies agreed with GAO's recommendations and identified actions they plan to take to address them.", "document_type": "gao"}
{"report": "DOD has reported that more than a decade of conflict, budget uncertainty, and reductions in force structure have degraded military readiness. In response, DOD has made rebuilding the readiness of the military forces a top priority. The 2018 National Defense Strategy states that the central challenge to U.S. prosperity and security is the reemergence of long-term, strategic competition with China and Russia. Further, the strategy stresses that restoring and retaining readiness for large-scale combat is critical to success in this emerging security environment. Nevertheless, DOD reported that readiness of the total military force remains low and has remained so since 2013. In June 2017, we found that Army readiness goals and timelines for rebuilding readiness are not clear for all portions of the force, especially for the reserve component, although the Army is making progress in these areas. Across the department, DOD has made progress in developing a plan to rebuild the readiness of the military force, with the military services providing regular input on the status of their readiness recovery efforts. In August 2018, we reported that the Office of the Secretary of Defense developed a Readiness Recovery Framework that the department is using to guide the services’ efforts, and plans and to regularly assess, validate, and monitor readiness recovery. The Office of the Secretary of Defense and the services have recently revised readiness goals and accompanying recovery strategies, metrics, and milestones to align with the 2018 National Defense Strategy and Defense Planning Guidance. According to The Army Strategy, the Army projects that it will reach its readiness goals by 2022, at which point its priority is expected to shift to modernization. We have ongoing work assessing DOD’s progress in achieving its overall readiness goals in each of five warfighting domains: ground, sea, air, space, and cyberspace. The number one stated goal of Army leadership is readiness, including recovering the readiness lost from years of sustained conflict while preparing for potential large-scale combat operations against a global competitor such as Russia or China. These efforts are occurring in a challenging context that requires DOD to make difficult decisions regarding how best to address continuing operational demands while preparing for future challenges. An important aspect of this, across all of the military services, is determining an appropriate balance between maintaining and upgrading legacy weapon systems currently in operational use, and modernizing to ensure the ability to outpace advancing competitors. Our work has shown that the Army has improved ground force readiness in recent years; however, the Army has also identified capability shortcomings in its weapon systems and platforms that have yet to be addressed through its modernization efforts. In an effort to achieve higher, more consistent levels of readiness over longer time periods, the Army is implementing a redesigned way to generate forces called the sustainable readiness concept. A key part of the concept includes determining readiness objectives by unit type, which are developed by comparing the numbers of key unit types against planned and potential warfighting demands. In addition, since 2014 the Army has invested significantly in stocks of warfighting equipment that are being stored in Europe, and has begun deploying armored formations to the continent on a continuous basis for training and exercises to enhance its readiness against potential Russian aggression. As the Army works to rebuild and sustain higher readiness of its current force, the service is moving to update its doctrine, equipment, and formations to conduct operations in a more complex warfighting environment. The Army believes that it must be able to operate not only on land against potential adversaries, but also have the capability to act against them in other domains, namely air, sea, cyber, and space. The new Army Operating Concept, published in December 2018, describes how the Army would operate in a “multi-domain” environment. It identifies readiness as being key to deterring aggression from potential adversaries and, should conflict occur, addresses how Army forces would operate in multiple domains to penetrate anti-access and area denial systems. To support this concept, the Army’s modernization strategy aims to build the next generation of weapon systems and platforms that are more agile, lethal, resilient, and sustainable on the future battlefield. We have ongoing work reviewing the Army’s efforts to develop its multi-domain operations concept and to field capabilities to support such operations. The Army is growing slightly from a previously-planned size of 980,000 uniformed personnel to just over 1 million personnel. The Army is also adjusting its force structure to address increasing operational risks as it prepares for potential combat operations against a major adversary. However, our work shows that the Army faces challenges in filling and maintaining key skills in a number of areas, and in managing the time Army personnel spend away from their home station. In 2016, we reported that the Army was planning to reduce its end strength from a high of about 1.11 million uniformed personnel in fiscal year 2011 to an end strength of 980,000 by fiscal year 2018. The Army stated that at this level it could execute the National Defense Strategy, but at significant risk. Army leadership testified in March 2015 that if there were further end strength reductions, the Army would not be able to execute the defense strategic guidance. We reported in 2016 that the Army needed to assess the risks associated with the planned reductions and better document its force-planning process. The Army concurred with both of our recommendations, changed the way it assessed risk, and made adjustments to its force structure based on these assessments. After our 2016 report, Congress partly reversed these planned reductions by authorizing end-strength increases in fiscal years 2017 through 2019. The principal increase occurred in 2017, when Congress authorized an end strength of 1.018 million uniformed personnel, or 28,000 more than the Army had planned for that year. The Army’s authorized end strength since 2011, including planned end strength in 2017 and 2018, are summarized in figure 1. Additionally, as we found since our 2016 report was published, the Army has added or plans to add capacity, including converting two infantry brigades into armored brigades and activating two new Mobile Short Range Air Defense battalions by fiscal year 2022, to better prepare the force for large-scale combat against major adversaries. Also, to support combat forces during a conflict, the Army is activating additional combat sustainment formations that are responsible for supply, distribution, and transportation. Our ongoing work has found that over the next few years the Army is building or plans to build several new cyber and electronic warfare units to operate at various levels within the force to make the Army more effective in contested environments. According to the Chief of Staff of the Army, in a January 2019 speech, the Army has used its end strength increases to increase the manning of combat units. The goal of Army leadership is to fill operational units to 100 percent by the end of fiscal year 2019, and 105 percent by the end of fiscal year 2020. However, in preparing this statement we found that, in three of the past four years, the Army has fallen short of meeting its overall end strength authorizations. Army officials told us that these differences from the authorized end strength fall under the Secretary of Defense’s authority to reduce the end strengths by a certain amount. Moreover, these officials added that in 2015 and 2016, the Army was drawing down end strength and planning further reductions. However, the Army fell short of its end strength authorization by 0.38 percent in 2017, and fell short again by 2.56 percent in 2018. The percentage differences between authorized and actual end strength for the total Army, from 2015 through 2018, are summarized in figure 2. As we prepared this statement, Army officials told us that the primary reason why it has struggled to meet its authorized end strength is because it has had difficulty meeting recruiting goals, which have negatively affected the Army’s ability to expand the force. For example, Army officials told us the Army was short of its goal for 2018 by 6,500 new recruits for the regular Army. Army officials told us that the Army does not expect to be able to achieve its authorized end strength for fiscal year 2019. Looking ahead, the Army is considering revisions to its expansion plans and now expects to reach a new end strength goal by 2025. In addition to challenges in meeting authorized end strength, our past and ongoing work indicates that the Army faces challenges in filling and maintaining key skills in a number of areas, and in managing the time Army personnel spend away from their home station. Both of these challenges can negatively affect readiness. For example: Accelerated activation of Security Force Assistance Brigades led to manning challenges. In December 2018, we reported that the Army’s decision to deploy the first security force assistance brigade 8 months earlier than planned posed challenges to manning the unit. The Army currently plans to activate up to six of these brigades (one in the U.S. Army National Guard) by the end of fiscal year 2019. The Army views the Security Force Assistance Brigades to be critical to restoring the readiness of its combat forces. Prior to their formation, the Army met security force assistance missions by, among other things, pulling senior leaders and other personnel with specific ranks and skills from active-duty brigades, which compromised their readiness for large-scale combat. The Army has had difficulty filling new cyber and electronic warfare units. During our ongoing work, we have found that the Army has had difficulty filling new formations with personnel to conduct operations in the cyber domain, including electronic warfare. In October 2018, the Army activated part of a Multi-Domain Task Force, which is focused on intelligence, information, cyber, electronic warfare, and space missions and is being used in major exercises in the Pacific region. However, Army headquarters officials told us that the Army activated the unit as a pilot, or a test, unit and with an accelerated timeline to learn how the new formation should be structured, equipped, and trained. Based on our ongoing work, filling the unit with personnel with the right skills has been a slow process. Near the end of January 2019 the unit was staffed at 50 percent, and the Army projects it will reach 75 percent by August 2019, according to Army headquarters officials. The officials added that many of the shortages are in senior level and cyber positions. Meanwhile, Army documentation obtained during our ongoing work shows that the service is considering options for creating more task forces for other regions. Additionally, there are plans for new cyber and electronic warfare force structure supporting Brigade Combat Teams. Army officials stated that these will be fielded in an accelerated manner as well, adding that filling these units could be challenging because cyber personnel are in high demand. Army headquarters officials said they are exploring options to address the challenges. Army depots have had difficulty filling and maintaining critical skills in their workforces. For our December 2018 report, officials told us that Army depots experienced consistent challenges in hiring critical personnel. Also, we reported that workload fluctuations usually resulted in too little workload to maintain proficiency in certain skills. For example, we reported that a hiring freeze at Corpus Christi Army Depot in 2017 caused shortages of civilian flight test pilots, who are responsible for test flights before returning aircraft to service after maintenance. The Army, however, had not assessed how effective the depots have been at hiring, training, and retaining the critical skills of their workforce. We recommended that the Army do this, as personnel challenges such as these have affected depots’ ability to meet mission requirements and created maintenance delays for some equipment. The Army concurred with our recommendation and stated that it would assess the effectiveness of the depots’ hiring, training, and retention programs to ensure Army requirements are met and critical skills are maintained. The Army has had difficulty manning ballistic missile defense units. As we reported in October 2017, the Army’s Patriot and Terminal High Altitude Area Defense (THAAD) ballistic missile defense forces have been in high demand for many years. Army officials told us at the time that with reductions in end strength, the Army in 2016 stopped its practice of assigning extra personnel to these units to ensure operational requirements would be met. Army officials stated that the high aptitude standards and specialized nature of operating Patriot and THAAD systems reduced the number of eligible recruits. Officials also stated that enlistment shortfalls could have long-term effects on these forces’ operations and career development. Since we issued our report, Army officials told us that fewer-than-expected new recruits had advanced into Patriot and THAAD career fields in 2018, but the Army was forecasting improvements. High personnel tempos can negatively affect personnel. In 2018, we reported that the pace of operations has had a negative effect on Army readiness, including Brigade Combat Teams and Combat Aviation Brigades. We also reported that managing personnel tempo—the amount of time that individual service members spend away from home on official duties—had been a persistent challenge for the Army. In 2015, the Army issued a regulation identifying a personnel tempo threshold for its service members, but officials told us that the threshold is not enforced and stated the regulation was published only to emphasize that personnel tempo data was a priority. We found that personnel tempo data collected by DOD was incomplete. However, we estimated from the data that at least 41 percent of Army service members who were away from their home station in fiscal year 2016 were away for more than 7 months. Because time away from home can stress the force, we recommended that DOD or the Army take steps to clarify and follow personnel tempo guidance on thresholds, and also take steps to emphasize the collection of complete and reliable personnel tempo data to allow monitoring. DOD concurred with both recommendations. The Army is in the process of updating and developing new concepts and equipment to deal with a future environment that will be increasingly lethal, competitive, complex, and dynamic. The Army anticipates that it will have to contend with a resurgent Russia and a rising China, as well as regional challenges from North Korea and Iran. According to the Army, these adversaries have improved their military capabilities, in particular their ability to prevent U.S. forces from massing close to the potential battlefield, thereby eroding advantages that the Army has enjoyed for decades. Once deployed, the Army stated it expects that its forces will be constantly under surveillance and potentially under attack. To counter the adversaries’ threats, the Army is focusing on updating warfighting concepts and modernizing the force. In December 2018, the Army published a new Army Operating Concept that is specifically designed to deter and defeat China and Russia, and addresses large- scale ground combat. The concept emphasizes that the Army must demonstrate its readiness to conduct multi-domain operations—such as ground, air, and cyber—as a key part of deterring adversaries from escalation. To support its readiness for future missions in this complex environment, the Army has begun to update or upgrade multiple weapon systems. In April 2018, the Army published its Army Modernization Strategy, which identified six priorities that are key to operationalizing multi-domain operations, including long-range precision fires and next generation combat vehicles, as shown in table 1. All six of these priorities involve modernizing equipment and/or acquiring new equipment with improved capabilities. The Army has identified the need to make changes to how it develops and acquires new weapons systems. To that end, the Army established the Army Futures Command to provide unity of command, accountability, and modernization at the speed and scale required to prevail in future conflicts. Our prior work has found that the Army has faced challenges with managing maintenance efforts and developing requirements for future weapon systems. Some of the challenges include the following: The Army lacks an implementation plan to guide its retrograde and reset activities, which could lead to inconsistent reset efforts. As we reported in May 2016, officials from different Army entities disagreed about which documents constituted their guidance for implementing retrograde and reset, suggesting that there was confusion about the Army’s strategies for these activities. We recommended that the Army develop an implementation plan for its retrograde and reset efforts. In August 2018, however, we reported that the Army did not have plans to act on this recommendation. According to one official, this was because guidance and plans are adjusted based on the unique circumstances of each situation. Given the Army’s drawdown of equipment used during operations in Iraq and Afghanistan is coming to a close, we continue to believe that an implementation plan for retrograde and reset of equipment used during any future operations would help ensure that the Army more consistently and effectively budgets for and distributes resources. The Army has not comprehensively assessed the causes of reset maintenance delays for Patriot equipment, which can limit unit training time. In June 2018, we reported that of seven Patriot battalions undergoing reset in fiscal years 2014 through 2017, only one received all of its equipment back from depot maintenance within the Army’s policy of 180 days, as shown in figure 3. Since delays in returning equipment to units can reduce units’ training time, we recommended that the Army analyze the various factors affecting reset delays—such as equipment arriving late to the depot, supply chain delays, and worker errors—to identify their relative importance and inform corrective actions. The Army concurred with our recommendation, stating that it will identify and address factors that may affect reset timeliness. The Army’s near-term modernization efforts face management challenges. In September 2018, we reported that the Army had not established processes for evaluating its modernization efforts against its overarching objective of outpacing rapidly advancing competitors, such as Russia or China. Also, we found that the Army had not fully estimated the costs of its near-term modernization efforts. Further, we found that the Army’s April 2018 modernization strategy report set near-term goals for closing critical capability gaps and a longer term, overarching objective of being able to decisively defeat major adversaries. The strategy also identified the cost of key modernization investments through fiscal year 2023, but did not discuss tens of billions in already-programmed modernization-related investments, or describe how the funding would support upgrades for existing weapon systems. Moreover, the strategy did not disclose the extent to which the Army had relied on Overseas Contingency Operations (OCO) appropriations for upgrading weapon systems. Army officials told us at the time that the Army had been preparing to analyze its efforts to address specific warfighting capability gaps, but had not decided on an overall evaluation approach. Additionally, officials told us that the Army planned to reflect its analysis of near-term modernization investments in the fiscal year 2020 budget submission. We recommended that the Army (1) develop a plan to finalize the processes for evaluating how its near-term investments contribute to the Army’s ability to decisively defeat a major adversary, and (2) finalize its cost analysis of near-term investments and report those costs to Congress. The Army concurred with our recommendations. The Army has been unable to ensure that requirements for new warfighting capabilities are feasible. In June 2017, we reported that the Army had prioritized combat readiness and rebuilding force structure over resourcing its requirements development process to meet future readiness needs. We reported that even though the Army made some improvements in this area, officials were unable to ensure requirements for major defense acquisition programs were well-informed and feasible because of workforce constraints. For example, we found that the Army’s requirements development workforce declined by 22 percent from 2008 to 2017, with some requirements development centers reporting more significant reductions. In that report, we recommended that the Army assess the resources necessary for the requirements development process and determine whether shortfalls can be addressed given other funding priorities. The Army concurred with our recommendation. In 2018, Army officials told us that the Army plans to implement this recommendation once Army Futures Command is fully operational and key Army development entities are reorganized under its command. The Army has not fully applied leading practices for technology development in its modernization efforts. We reported in January 2019 that while the Army has generally applied leading practices identified by GAO to its modernization efforts, it may be beginning weapon systems development at a lower level of maturity than what leading practices recommend. As we concluded in that report, establishing Army Futures Command creates unique opportunities for the Army to improve its modernization efforts. However, proceeding into weapon systems development before technology is sufficiently mature raises the risk that the resulting systems could experience cost increases, delivery delays, or failure to deliver desired capabilities. The Army concurred with our four recommendations to apply leading practices and lessons learned as it moves forward with its modernization efforts. In its response to our January 2019 report, the Army stated that it would conduct operational technology demonstrations and was exploring a train-the-trainer program, among other actions. Our prior work has shown that the Army has made progress in preparing the force for large-scale combat operations by increasing training exercises and reducing mandatory training requirements. It also has addressed past issues we reported on, including making better use of virtual training devices and accounting for the training needs of supporting units in its Pacific Pathways exercises. Moreover, our prior and ongoing work has shown that the Army faces implementation challenges in training new units that the Army plans to field on shortened schedules. Army units are receiving more frequent training for large-scale combat. Our prior work has shown that the Army has made progress in preparing the force for large-scale combat by increasing training exercises. After a decade of focusing its training on counterinsurgency operations, the Army assessed that opportunities to train thousands of company commanders, field-grade officers, and battalion commanders on tasks related to large-scale combat were lost. However, in August 2016, we reported that the Army increased the number of brigades that had completed a decisive-action exercise from one brigade combat team in fiscal year 2011 to 14 brigade combat teams in fiscal year 2015, while at the same time decreasing training for counterinsurgency. We noted in a September 2016 report that a key part of the Army’s plan to rebuild readiness was to ensure that soldiers have repeated training experience on their core competencies. Since we completed our work, the Army is funding up to 26 brigade combat teams to go through a decisive-action training event at its combat training centers in fiscal year 2019. Mandatory training and directed tasks have been reduced. In August 2016, we also reported that the Army had determined that mandatory training requirements and directed tasks were too numerous and were creating challenges for commanders in balancing their units’ training time with these other requirements. Additionally, we identified steps the Army had taken to make these requirements less burdensome. We reported, for example, that the Army had delegated authority to two-star commanders to exempt units, as needed, from certain mandatory training. We reported that the Army had begun to lock in a unit’s planned training six weeks in advance, in an effort to protect units from external tasks that could affect training schedules of brigades and their subordinate units. The early setting of training schedules was intended to prevent an external task from interfering with that training. We did not make any recommendations related to reducing mandatory training; however; since we completed our work, the Secretary of the Army has directed the elimination of numerous individual training requirements, such as eliminating certain requirements to train in avoiding accidents, and other administrative tasks, such as maintaining a physical reference library of corrosion prevention and control publications. The Army is making better use of virtual devices to train and prepare units. In the same 2016 report, we identified a number of challenges the Army faced in using virtual training devices to help units prepare for major conflict. Using such devices is important because of the challenges of training for combat in a live environment, such as limited range availability and resource constraints. We reported that the Army had taken some steps to improve the integration of virtual training devices into its operational training. However, our work identified several factors that limited the Army’s ability to conduct training with virtual training devices, including outdated virtual training policies, a lack of guidance for analyzing the effectiveness of virtual training devices, and the need to better integrate devices in training strategies. As of January 2019, the Army has implemented two of the three recommendations we made in our report. Specifically, the Army published a training analysis best- practices guide, analyzed virtual training devices’ effectiveness, and analyzed regular Army formations’ readiness training models, among other steps to implement these recommendations. Additionally, the Army further plans to modify its policy on virtual training devices in 2021, which would require that training developers consider the amount of time available to train with or expected usage rates of new virtual training devices. Further, in preparing this statement, Army officials told us that the Army has used acquisition authorities provided by Congress to prototype new technologies to replace existing simulators. It is investing in these prototypes based on the usage rates of the older training equipment, and at the same time involving operational forces in the prototyping for their feedback and to help inform requirements. The Army is taking some steps to improve its Pacific Pathways initiative. In November 2016, we reported on an initiative, known as Pacific Pathways, intended to strengthen relationships with allies and build readiness by combining certain exercises with partner nations. The Army began the Pacific Pathways initiative—which deploys a battalion-size task force to the Asia-Pacific region to conduct multiple exercises over 90 days—as a way of building the readiness of its participating units. We found that the size and complexity of the operations under Pacific Pathways created potentially unique training opportunities for supporting units—such as transportation units—to exercise the capabilities they would be required to provide in a contingency. However, we found that the Army could improve its approach by fully synchronizing Army plans, stakeholders, and objectives into the exercises. The Army has implemented two of the recommendations that we made in our report to modify processes and guidance so that stakeholders are integrated into the planning, and also to seek and incorporate the training objectives of supporting units. U.S. Army Pacific officials have stated that they do not plan to implement the recommendation to perform a cost-benefit analysis of Pacific Pathways because it is not required. Our prior and ongoing work has identified some challenges that the Army faces in training personnel in particular specialties, especially as it stands up new units on shortened schedules. These include: A lack of training facilities and airspace creates challenges for UAS pilot training and further steps could be taken to enhance pilot candidate selection. In January 2017, we reported that the Army’s UAS pilot training strategy did not account for some challenges the Army faced, such as a lack of adequate training facilities and limited available airspace. The Army used flexibilities to overcome some of these challenges, but at the time of our report it was too early to tell whether these flexibilities would be enough to overcome training shortfalls. In addition, we found that the way the Army assessed whether service members were good candidates for UAS pilot training could have been improved. For example, we reported that the Army used only 3 of the 78 identified competencies that an Army-Air Force research team identified as “moderately,” “highly,” or “extremely important” for UAS pilots. We made recommendations on these issues, and DOD partially concurred, stating that although the actions we recommended were prudent or already an integral part of workforce management, additional Army guidance would be unnecessary. Fielding and deploying new types of units can pose challenges to training. The accelerated pace at which the Army is creating new units can pose challenges to training and readiness. As previously discussed, the Army is activating new units to sustain readiness and to operate in a more complex environment. However, the Army’s approach can pose training challenges, and negatively affect readiness. Also, our ongoing work indicates that the Army is fielding new cyber units at an accelerated pace, resulting in the units not having either fully trained personnel or the equipment to conduct training, according to Army officials. For example, the Army is planning to add uniformed personnel who specialize in cyber operations to its combat units and as part of newly established Multi- Domain Task Forces, but there is not yet a clear understanding of the tasks they will have to perform or an updated training strategy to support them, according to Army officials. Army officials stated that this will affect the readiness of the units to perform their missions, but they are taking steps to clarify and update these issues. - - - - - In sum, while the Army has made progress in rebuilding readiness, it continues to face challenges meeting its goals. Moreover, the Army will need to balance the readiness of its existing force with plans to grow and modernize. We have made 44 recommendations that the Army has generally concurred with; the Army has implemented 7 of them, and taken actions to begin implementing many others. These recommendations provide a partial roadmap to address important readiness challenges, and implementing our recommendations to improve the management of personnel, equipment maintenance, and training would help the Army meet current threats and assist it as it refocuses on readiness for large- scale combat operations. In addition, sustained management attention and continued congressional oversight will be needed to ensure that the Army demonstrates progress in addressing its personnel, equipment, and training challenges. Chairman Sullivan, Ranking Member Kaine, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff have questions about this testimony, please contact John H. Pendleton, Director, Defense Capabilities and Management at (202) 512-3489 or pendletonj@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Kevin O’Neill (Assistant Director), Matthew Spiers (Analyst In Charge), Steven Bagley, Rebecca Beale, Cynthia Grant, Kris Keener, Alberto Leff, Amie Lesser, Jon R. Ludwigson, Shahrzad Nikoo, Marcus Oliver, Richard Powelson, James A. Reynolds, Cary Russell, Michael Silver, Matthew Ullengren, Nicole Volchko, Erik Wilkins-McKee, Matthew Young, and Delia Zee. Over the past 4 years, we issued several reports related to Army readiness that are cited in this statement. Table 2 summarizes the status of key GAO recommendations related to Army and DOD components in coordination with the Army since 2016, which include a total of 44 recommendations. The Department of Defense has implemented 7 of these recommendations to date. For each of the reports, the specific recommendations and their implementation status are summarized in tables 3 through 19. Army Modernization: Steps Needed to Ensure Army Futures Command Fully Applies Leading Practices, GAO-19-132. Washington, D.C.: January 23, 2019. DOD Depot Workforce: Services Need to Assess the Effectiveness of Their Initiative to Maintain Critical Skills [Reissued with revisions on December 26, 2018.], GAO-19-51. Washington, D.C.: December 14, 2018. Navy and Marine Corps: Rebuilding Ship, Submarine, and Aviation Readiness Will Require Time and Sustained Management Attention, GAO-19-225T. Washington, D.C.: December 12, 2018. Air Force Readiness: Actions Needed to Rebuild Readiness and Prepare for the Future, GAO-19-120T. Washington, D.C.: October 10, 2018. Army Modernization: Actions Needed to Measure Progress and to Fully Identify Near-Term Costs, GAO-18-604SU. Washington, D.C.: September 28, 2018. Military Readiness: Analysis of Maintenance Delays Needed to Improve Availability of Patriot Equipment for Training, GAO-18-447. Washington, D.C.: June 20, 2018. Military Readiness: Clear Policy and Reliable Data Would Help DOD Better Manage Service Members’ Time Away from Home, GAO-18-253. Washington, D.C.: April 25, 2018. European Reassurance Initiative: DOD Needs to Prioritize Posture Initiatives and Plan for and Report their Future Cost, GAO-18-128. Washington, D.C.: December 8, 2017. Military Readiness: Personnel Shortfalls and Persistent Operational Demands Strain Army Missile Defense Units and Personnel, GAO-18-168SU. Washington, D.C.: October 5, 2017. Army Weapon Systems Requirements: Need to Address Workforce Shortfalls to Make Necessary Improvements, GAO-17-568. Washington, D.C.: June 22, 2017. Supply Chain Management: DOD Could More Efficiently Use Its Distribution Centers, GAO-17-449. Washington, D.C.: June 21, 2017. Army Readiness: Progress Made Implementing New Concept, but Actions Needed to Improve Results, GAO-17-458SU. Washington, D.C.: June 8, 2017. Unmanned Aerial Systems: Air Force and Army Should Improve Strategic Human Capital Planning for Pilot Workforces, GAO-17-53. Washington, D.C.: January 31, 2017. Comprehensive Assessment and Planning Needed to Capture Benefits Relative to Costs and Enhance Value for Participating Units [Reissued on November 30, 2016], GAO-17-126. Washington, D.C.: November 14, 2016. Military Readiness: DOD’s Readiness Rebuilding Efforts May Be at Risk without a Comprehensive Plan, GAO-16-841. Washington, D.C.: September 7, 2016. Patriot Modernization: Oversight Mechanism Needed to Track Progress and Provide Accountability, GAO-16-488. Washington, D.C.: August 25, 2016. Army Training: Efforts to Adjust Training Requirements Should Consider the Use of Virtual Training Devices, GAO-16-636. Washington, D.C.: August 16, 2016. Military Readiness: DOD Needs to Incorporate Elements of a Strategic Management Planning Framework into Retrograde and Reset Guidance, GAO-16-414. Washington, D.C.: May 13, 2016. Army Planning: Comprehensive Risk Assessment Needed for Planned Changes to the Army’s Force Structure, GAO-16-327. Washington, D.C.: April 13, 2016. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The 2018 National Defense Strategy emphasizes that restoring and retaining readiness across the entire spectrum of conflict is critical to success in the emerging security environment. The top priority for Army leadership is readiness. The Army has undertaken a variety of efforts since 2016 to prepare for potential large-scale combat operations against major adversaries. This statement provides information on the Army's progress and challenges in readiness rebuilding in the areas of (1) force structure and personnel, (2) equipment repair and modernization, and (3) training for potential large-scale conflict. Also, GAO summarizes recommendations to address these challenges and actions taken by the Army to address them. This statement is based on previously published GAO work since 2016. This prior work related to, among other things, Army readiness, skills shortages, equipment maintenance and modernization, acquisition, training, force structure. GAO also updated information and incorporated preliminary observations from ongoing work related to warfighting concepts. In GAO's prior and ongoing work, GAO found that the Army has made progress in rebuilding readiness and projects that it will reach its readiness goals by 2022. While the Army continues to make progress, it faces challenges in staffing its evolving force structure, repairing and modernizing its equipment, and training its forces for potential large-scale conflicts (see table). Looking to the future, the Army plans to grow its forces, provide them with modernized equipment, and train units to conduct large-scale, decisive-action operations. All of these efforts are underway as the Army contemplates the implications of future warfare—which it reports is likely to require operations in multiple domains, especially cyber. As a result, it is important for the Army to balance its efforts to rebuild and sustain the operational readiness of its existing force with its preparations for future threats. GAO has made 44 recommendations in prior unclassified work described in this statement. DOD and the Army have generally concurred with them, have implemented seven, and have actions underway to address others. Continued attention to these recommendations can assist and guide the Army moving forward as it seeks to rebuild the readiness of its force and transforms for the future.", "document_type": "gao"}
{"report": "GSA’s existing government-wide telecommunications program is called Networx. As part of this program, in 2007 GSA awarded two sets of Networx contracts—called Networx Universal and Networx Enterprise— which had an estimated combined value of $20 billion. These contracts provide similar services, such as voice and data services, wireless services, video and audio conferencing, as well as mobile and fixed satellite services. One differing characteristic between the contracts is that Networx Enterprise contracts have a focus on internet-based services. The Networx Enterprise contracts also require telecommunications services to be available in a smaller geographic area than Networx Universal. Networx Universal contracts were set to expire in March 2017 and Networx Enterprise contracts were set to expire in May 2017; however, GSA has twice extended these Networx contracts. According to GSA officials, the most recent extension, which GSA announced in November 2018, is to include one base year and two 1-year options, plus an additional option for the number of months required for the contracts to reach May 31, 2023. If the extension is executed and all options are exercised, the Networx contracts will expire in May 2023. In addition, GSA provides telecommunications services through programs called Washington Interagency Telecommunications System 3 and Regional Local Service Agreements: Washington Interagency Telecommunications System 3: These contracts support a variety of telecommunications services available to all federal agencies in Washington, D.C., and surrounding Maryland and Virginia counties. For example, among other things, these contracts provide data and voice services, as well as cloud services. These contracts were set to expire on or before May 2020. As of December 2019, GSA planned to extend these contracts. GSA officials stated that the extension is to include one base year and two 1-year options, plus an additional option for the number of months required for the contracts to reach May 31, 2023. If the extension is executed and all options are exercised, the contracts will expire in May 2023. Regional Local Service Agreements: These contracts provide local telecommunications services in every state and major city in the United States. According to GSA officials, the expiration dates for these contracts ranged from October 2019 through March 2023. As of December 2019, GSA was in the process of extending these contracts. Specifically, GSA officials reported that certain contracts had already been extended to May 2023, and the officials planned to extend the remaining contracts through May 2023, as well. According to data provided by GSA officials, in fiscal year 2019, federal agencies spent approximately $2.5 billion on services acquired through Networx, Washington Interagency Telecommunications System 3, and Regional Local Service Agreements contracts. About $2 billion of this spending was on services acquired through Networx alone. GSA’s EIS program is the replacement for the agency’s Networx, Washington Interagency Telecommunications System 3, and Regional Local Service Agreements telecommunications contracts. GSA intends for EIS to address federal agencies’ global telecommunications and IT infrastructure requirements. GSA plans for EIS to provide agencies with traditional and emerging services to meet current and future requirements by, among other things: simplifying the government’s process of acquiring IT and telecommunications products and services; providing cost savings to each agency through aggregated volume buying and pricing (with generally lower costs for services on EIS compared to the costs for similar services on Networx), and spending visibility; and providing updated and expanded security services to meet current and future government cybersecurity requirements. In addition, GSA has identified several benefits that EIS is expected to provide to the agencies that participate in its telecommunications programs. These projected benefits include streamlined contract administration, a possible 15-year period of performance, simplified pricing, and enhanced management and operations support. On August 1, 2017, GSA announced that it had awarded EIS contracts to 10 vendors. These contracts have a combined value of up to $50 billion and are for a possible period of up to 15 years (one 5-year base period and two 5-year option periods). According to GSA’s plans as of November 2019, the transition to EIS is expected to be completed by May 2023, when the current Networx, Washington Interagency Telecommunications System 3, and Regional Local Service Agreements telecommunications contracts are expected to expire (if all contract options are exercised, as discussed earlier). To help ensure that agencies’ services are fully transitioned to EIS before the current contracts expire, GSA issued guidance that identified several critical milestones that agencies should meet. These milestones include: (1) releasing all planned fair opportunity solicitations to EIS vendors by March 31, 2019; (2) issuing all planned task orders by September 30, 2019; and (3) achieving 100 percent transition of services by September 30, 2022. Figure 1 provides a timeline of the planned transition to EIS, including GSA’s critical milestones, as of November 2019. Central to the successful transition from GSA’s current telecommunications services contracts to EIS are transition planning and execution activities that involve GSA, federal agencies, the incumbent telecommunications contractors, and EIS contractors. GSA serves as the facilitator for all transition management activities. The agency is using contractors to assist in tracking transition activities, in order to avoid delays and other problems that can arise throughout the process. In particular, GSA’s primary responsibility is to provide program management for the current telecommunications programs (Networx, Washington Interagency Telecommunications System 3, and Regional Local Service Agreements) and EIS. As part of this, GSA is responsible for conducting government-wide strategy and project management; providing tailored assistance to agencies for transition planning and help with contractor selection and ordering; tracking and reporting the use of metrics that convey the relative complexity and transition progress; and providing customer support, training, and self-help tools and templates. GSA developed two contracting vehicles to provide transition assistance to agencies: (1) a Transition Coordination Center vehicle that includes assistance with inventory validation, transition planning, and solicitation development; and (2) a Transition Ordering Assistance vehicle that addresses tasks including requirements development and source selection assistance, and proposal evaluation. The Coordination Center vehicle was put in place in January 2016 and the Ordering Assistance vehicle was initially awarded in September 2016, but was not finalized until March 2017, after the conclusion of a bid protest. Agencies have principal responsibility for the transition. They are responsible for coordinating transition efforts with the incumbent contractors and EIS contractors to ensure that existing telecommunications services are disconnected and that new services are ordered under EIS. According to GSA, agencies’ responsibilities under EIS include, among other things: identifying key personnel, chiefly a Senior Transition Sponsor, Lead Transition Manager, and Transition Ordering Contracting Officer; engaging expertise from Chief Information Officers, Chief Acquisition Officers, and Chief Financial Officers to build an integrated transition team of telecommunications managers, acquisition experts, and financial staff; analyzing and confirming the accuracy of the inventory of active services that must be transitioned; developing a transition plan that describes technological goals, a transition schedule that includes GSA’s major transition milestones (e.g., releasing all fair opportunity solicitations by March 31, 2019, and issuing all task orders by September 30, 2019), strategy for issuing task orders on EIS, and any constraints or risks; preparing solicitations for task orders; placing task and service orders; and reviewing, accepting or rejecting, and paying for services. At the agencies we reviewed, the staff responsible for the transition were part of their agencies’ offices that were headed by the Chief Information Officers. Finally, the incumbent and EIS contractors are responsible for disconnecting existing services under the current contracts and installing new services that agencies order under EIS. They are also to collaborate with GSA and agencies to (1) share transition planning and execution best practices and (2) help resolve issues. We have previously reported on efforts by GSA and agencies to transition from one telecommunications program to another. In a June 2006 report, we identified a range of transition planning practices that can help agencies reduce the risk of experiencing adverse effects of moving from one broad telecommunications contract to another. These planning practices were to: (1) develop an accurate inventory of telecommunications assets and services, (2) perform a strategic analysis of telecommunications requirements, (3) develop a structured transition management approach, (4) identify the resources needed for the transition, and (5) develop a transition plan. Since that June 2006 report, we have reported multiple times on the extent to which selected agencies were following the transition planning practices. We have generally found that the selected agencies in our reviews had not fully implemented some of the key activities of the practices. For example, in 2008, we noted that one agency was not planning to clearly define all key transition roles and responsibilities and another agency was not planning to identify local and regional points of contact. In addition, in 2017, we reported that, among other things, the five agencies we selected had yet to fully apply most of the five planning practices. In each of our reports we made recommendations to the selected agencies focused on addressing the gaps in transition planning. All five agencies in our 2017 review undertook efforts to address our recommendations, but had not yet fully implemented them as of November 2019. Based on our ongoing work, our preliminary results indicated that the 19 selected agencies have varied plans for transitioning from their current telecommunications contracts to EIS program contracts. As of October 2019, these agencies were also in different stages of their EIS transitions. All of the selected agencies reported that they plan to fully transition their telecommunications services to EIS before the current contracts are set to expire in May 2023. However, over half of the selected agencies did not plan to complete the transition by GSA’s September 30, 2022, milestone. In addition, the majority of selected agencies did not meet GSA’s two critical EIS transition milestones in 2019—to (1) release all fair opportunity solicitations by March 31, 2019, and (2) issue all task orders by September 30, 2019. As of November 2019, the 19 selected agencies had various plans for completing their transitions to EIS. Specifically, Eight of the selected agencies reported that they planned to finish their transitions to EIS by GSA’s September 30, 2022, milestone. The 11 remaining agencies did not plan to complete their transitions by that date. Officials from the 11 agencies that did not plan to finish their transitions to EIS by GSA’s September 30, 2022, milestone reported that they planned to complete the transitions before the current telecommunications contracts are set to expire in May 2023. However, four of these 11 agencies planned to complete their transitions in May 2023, just before the current telecommunications contracts are set to expire. In addition, the planned scope and amount of effort that is expected to be required to fully transition to EIS varied among the selected agencies. Specifically, agencies varied in the scope of their planned efforts related to two of GSA’s critical transition milestones—to release EIS fair opportunity solicitations and issue EIS task orders: One selected agency planned to release 54 EIS fair opportunity solicitations. The eighteen other selected agencies planned to release between one and six solicitations. Six agencies planned to issue more than five task orders. The other thirteen agencies planned to issue between one and five EIS task orders. Further, the selected agencies had different plans for the types of transitions that they would implement. Specifically, Four of the selected agencies planned to implement primarily a like- for-like transition of their services. The remaining 15 agencies planned to conduct a combination of a like-for-like transition and upgrading or transforming services. As of October 2019, the 19 selected agencies were in different stages of their EIS transitions. Eighteen of the agencies were in the acquisition planning and/or acquisition decision phases, during which the agencies release fair opportunity solicitations for vendor proposals and issue task orders to selected vendors, respectively. GSA established two critical milestones for agencies to complete these acquisition activities: (1) release all fair opportunity solicitations by March 31, 2019, and (2) issue all task orders by September 30, 2019. Regarding the first milestone—to release all EIS fair opportunity solicitations by March 31, 2019: Five of the 19 selected agencies reported that they released all of their solicitations by this date. The 14 remaining selected agencies reported that they did not release all of their solicitations by this date. Officials from each of the five agencies that met GSA’s milestone to finish releasing all of their planned EIS solicitations by March 31, 2019, reported that their agencies released either one or two solicitations. We asked officials from the 14 selected agencies that did not release all of their planned EIS solicitations by March 31, 2019, to identify the key factors that contributed to their agencies’ delays in releasing these solicitations. In response, agency officials cited numerous key factors for the delays, including the complexity of their telecommunications requirements, changes to the agency’s or GSA’s contracting strategy, and insufficient staff availability. Figure 2 identifies the key factors that contributed to delays in releasing all EIS solicitations by GSA’s March 31, 2019, milestone, as identified by agency officials. In addition, regarding GSA’s second milestone—to issue all EIS task orders by September 30, 2019—one of the selected agencies reported that it issued all of its task orders by this date. The 18 other agencies reported that they did not issue all of their EIS task orders by this date. We asked officials from the 18 agencies that did not issue all of their EIS task orders by September 30, 2019, to identify the key factors that contributed to their agencies’ delays in issuing these task orders. In response, agency officials cited 19 key factors that led to the delays. Nine of the identified factors were the same factors that officials cited for their agencies’ delays in releasing EIS solicitations, including the complexity of requirements and having insufficient staff available. The officials also identified 10 other factors unique to their delays in issuing EIS task orders. For example, officials from two agencies reported that the EIS vendors needed clarification on the agencies’ requests for proposals. In addition, officials from three agencies reported that they needed clarification from the EIS vendors on the proposals that the agencies received. Figure 3 identifies the key factors that contributed to delays in issuing all EIS task orders by GSA’s September 30, 2019, milestone, as identified by agency officials. Several of the identified factors, such as the partial government shutdown and the need for vendors to receive authorities to operate, have subsequently been resolved. For other factors, agencies can leverage GSA’s available EIS training and customer support to help minimize delays in meeting GSA’s transition milestones. Nevertheless, given that the majority of the selected agencies did not meet these transition milestones in 2019, it will be important for agencies to meet the remaining transition milestones to ensure that they complete the transition before the current telecommunications contracts expire in May 2023. In a June 2006 report, we identified five transition planning practices that can help agencies reduce the risk of experiencing adverse effects of moving from one broad telecommunications contract to another. Implementing these transition planning practices represents a comprehensive and rigorous management approach that can help agencies make the most of the opportunity for change that such a major telecommunications transition provides. Each of the five transition planning practices that we identified consists of various activities that should be implemented to fully address the planning practices. Table 1 identifies the five established transition planning practices and their associated activities. Based on our ongoing work, our preliminary results indicated that all five selected agencies had taken steps to implement the five established transition planning practices. However, consistent with our prior reviews of selected agencies’ efforts to implement these planning practices, none of the selected agencies had fully implemented any of the practices. The five selected agencies had all partially implemented the first established transition planning practice—to develop an accurate inventory of telecommunications assets and services. In particular, all of the selected agencies had partially implemented the two activities associated with this practice. Table 2 summarizes the extent to which the selected agencies had implemented the transition practice to develop an accurate inventory of telecommunications services. Identify a complete telecommunications inventory at every site, facility, and component. The five selected agencies had all partially implemented this activity. While all of these agencies had developed inventories of their telecommunications assets and services, none of the inventories were complete. Establish a documented process for updating and maintaining the inventories. All five selected agencies partially implemented this activity by taking steps to document their inventory update and maintenance processes. However, none of the agencies had fully documented these processes. Agency officials cited various reasons for partially implementing this first planning practice. For example, officials from three of the selected agencies—all of whom were responsible for their agencies’ transitions to EIS—stated that they did not track all of the assets and services ordered by the agencies. The officials added that they were not responsible for maintaining inventories of all of their agencies’ assets and services. Officials from another agency attributed their agency’s lack of a complete telecommunications inventory and associated maintenance procedures to the agency’s decentralized structure. Specifically, the officials stated that the agency’s components were responsible for managing the services that are unique to them. However, the officials stated that the agency did not have a policy that required its components to maintain an inventory of telecommunications assets and services that they acquired independently. Without complete and accurate telecommunications inventories, the selected agencies may be unable to avoid unnecessary transition delays related to an inability to plan for services not identified in the inventory. In addition, without documented processes for maintaining their telecommunications inventories, the agencies may not be able to consistently and accurately incorporate into these inventories any changes made during and after the transition (e.g., adding new services or removing disconnected services), thus hindering their ability to ensure that they are billed appropriately by the vendor. All of the selected agencies had partially implemented the second established transition planning practice—to perform a strategic analysis of telecommunications requirements. In particular, at least four agencies had partially, but not fully, implemented two of the four activities associated with this practice. For the other two activities, at least one agency had partially implemented each activity. Table 3 summarizes the extent to which the selected agencies had conducted strategic analyses of their telecommunications requirements. Identify current and future telecommunications needs using an inventory of existing services. All of the selected agencies had partially implemented this activity by identifying certain current and future telecommunications needs. However, as discussed earlier, none of the agencies had a complete inventory of current services. As a result, the agencies could not use such an inventory to fully identify their needs. Identify areas for optimization or sharing of telecommunications and IT resources. Three agencies had fully implemented this activity by completing strategic analyses to identify areas for optimization or sharing of telecommunications resources. The two remaining agencies had partially implemented this activity. For example, one of the two agencies had developed a draft analysis to justify the potential optimization and sharing across the agency of a telecommunications service for how hardware devices connect to the internet, but it had not yet finalized this analysis. Officials from the other agency had identified potential areas for the sharing of resources across the agency. However, the agency did not provide a documented analysis to justify the sharing of those resources. Evaluate the costs and benefits of any new technology and alternative options. Four agencies had fully implemented this activity by evaluating the costs and benefits of various technologies and alternative options for telecommunications services that they could implement as part of the transition. The one remaining agency had partially implemented this activity by evaluating the costs and benefits of upgrading one service by which hardware devices connect to the internet. While two of this agency’s components had also analyzed the costs and benefits of implementing another type of service for connecting to networks, the agency’s remaining components had not conducted such analyses. Determine that identified telecommunications needs and opportunities are aligned with the agency’s mission, long-term IT plans, and enterprise architecture plans. One agency had fully implemented this activity by determining that its telecommunications needs aligned with its mission and plans. The four remaining agencies partially implemented this activity and did not demonstrate that they had fully aligned their telecommunications needs with their agency’s mission, long-term IT plans, or enterprise architecture plans. Agency officials cited several reasons for not fully implementing the activities associated with this practice. For example, officials from one agency explained that they had not conducted and documented an analysis to identify areas for the sharing of telecommunications resources because they did not believe that there were any additional agency telecommunications resources that could be shared. The officials attributed this to the agency’s security requirements and regulations, and noted that services on the agency’s classified network may not be shared with services on its unclassified network. However, the agency did not provide documentation that demonstrated that it had determined that there were no additional resources that could be shared on its unclassified network. Officials from another agency stated that they thought their telecommunications needs were aligned with the agency’s long-term IT plans. However, the officials did not provide documentation demonstrating this alignment. Agencies that do not use complete inventories of their current telecommunications services to identify their future needs are likely not fully identifying these needs. They may also miss out on opportunities to optimize or share services by consolidating them on EIS. In addition, without aligning their telecommunications needs and opportunities with their missions and plans, agencies risk missing opportunities to use the new contract to address their highest priorities, or may make decisions that are not aligned with their long-term goals. All of the selected agencies had partially implemented the third transition planning practice—to develop a structured management approach for the telecommunications transition. Specifically, four of the agencies had partially, but not fully, implemented two of the three activities associated with this practice. Three agencies had partially implemented the other activity. Table 4 summarizes the extent to which the selected agencies had established a structured management approach. Establish a transition management team and clearly define responsibilities for key transition roles. One agency had fully implemented this activity by establishing a transition management team and defining all key transition responsibilities for the planning and execution phases of the transition, including for project, asset, human capital, and information security management; and contract and legal expertise. The remaining four agencies had partially implemented this activity by establishing transition management teams, but none of these agencies had defined all key roles and responsibilities for their transitions. Develop transition communications plans in order to facilitate information sharing during transition planning and execution. Two agencies had fully implemented this activity by developing transition communications plans and identifying all key parties that need to be involved during the agency’s transition effort. The remaining three agencies partially implemented this activity by identifying stakeholders responsible for communicating transition information to other stakeholders. However, these three agencies had not identified the key local and regional agency transition officials responsible for disseminating information about the transition to employees and working with the vendor to facilitate transition activities. Use established project, configuration, and change management processes in the agency’s transition planning efforts. One agency had fully implemented this activity by demonstrating the use of all established management processes called for in the activity. The four remaining agencies had partially implemented this activity by demonstrating the use of project management processes for their transitions, such as tracking transition costs and developing schedules and risk logs. However, one of these four agencies did not demonstrate that it was applying approved cost and schedule management processes to its transition. The three remaining agencies did not demonstrate that they were applying established configuration management processes to their transitions, and two of the three also did not demonstrate that they had implemented change management processes for their transitions. Officials from the selected agencies generally attributed their lack of full implementation of this practice to the fact that, at the time of our review, the agencies were early in their transition planning processes. For example, officials from one agency stated that they had not defined a role or responsibilities related to human capital management because their human capital needs for the transition will depend on the vendors selected (at the time of our review, the agency had not yet selected all of the vendors for its EIS task orders). Officials from another agency also explained that they planned to work with their selected EIS vendors to implement all of the key management processes for the transition. While the selected agencies were early in their transition planning processes at the time of our review, the limited time remaining to complete the transition makes it critical that agencies conduct early planning with the information that is available. Agencies that do not define all key roles and related responsibilities for their transition management teams risk extending their transition period as they attempt to assign appropriate personnel and update them on transition progress and issues. Further, without identifying all of the key officials that need to be involved with the transition, including the local and regional agency points of contact, agencies may lack the information that is necessary for comprehensive understanding, accountability, and shared expectations among all those with transition responsibilities. All of the selected agencies had partially implemented the fourth established transition planning practice—to identify their transition resource needs. In particular, all of the agencies had partially implemented three of the four activities associated with this practice. For the remaining activity, four of the agencies had partially implemented it and one agency had fully implemented it. Table 5 summarizes the extent to which the selected agencies had identified their transition resource needs. Identify the level of funding needed to support transition planning. One of the selected agencies had fully implemented this activity by identifying the costs needed to support its transition management team and all years of its transition planning efforts. The four other agencies partially implemented this activity by developing partial cost estimates for the transition, but none of these estimates were complete. Identify the organizational need for investments and justify resource requests. The five selected agencies had all partially implemented this activity by identifying the need for investments, including funding to obtain GSA transition assistance; however, none of the agencies had fully justified their resource requests for the transition. For example, four agencies had not justified their resource requests related to transition program management staff and one agency lacked justification for its requests for hardware and software upgrades. Identify human capital needs for the entire transition effort. All of the selected agencies had partially implemented this activity by identifying the need for certain staff to work on the transition, including government and contractor staff. However, none of the agencies had conducted and documented analyses of their human capital needs, to determine the total number of staff required to support their entire transition efforts. Identify and require training for the transition. All of the agencies had partially implemented this activity by identifying training needed by certain transition management staff. However, four agencies had not conducted and documented analyses to identify all of the training needed for their transitions, including training for staff carrying out the transition or operating and maintaining new equipment or services. The final agency had developed a draft analysis to identify training needed by staff carrying out the transition, but it had not finalized this analysis. Officials from the selected agencies generally explained that they were too early in their transition efforts to identify all of the funding, human capital, and training needed for their transitions. However, there is limited time remaining to complete the transition before the current telecommunications contracts expire. If the agencies do not conduct early planning to identify and justify all of their resources needed for the transition, they may underestimate the complexity and demands of their transition efforts. In addition, without using a rigorous management approach to analyze and document the total number of staff required to support the transition and to identify all of the required training for transition staff, agencies risk having insufficient staff available or may experience gaps in staff competencies. Such gaps may lead to delays and unexpected costs as the agencies try to quickly address the lack of resources during the transition’s limited time frame. All of the selected agencies had partially implemented the fifth established transition planning practice—to develop transition plans. Specifically, all of the agencies had fully implemented one of the three activities associated with this practice and partially implemented another of the activities. For the remaining activity, three agencies had fully implemented it and two had partially implemented it. Table 6 summarizes the extent to which the selected agencies had developed transition plans. Identify agency-specific transition objectives and measures of success. Three agencies had fully implemented this activity by identifying transition objectives and associated measures of success that were based on the transition objectives. The remaining two agencies had partially implemented this activity by identifying transition objectives and measures of success. However, their measures were unable to be used to assess transition progress. In particular, the identified measures could be used to determine success at the completion of the transition (e.g., all planned services have been transitioned to EIS). However, the measures did not enable the agencies to compare expected performance with actual results in order to track progress during the course of the transition (e.g., identifying the expected number of services that would be moved to EIS during each year of the transition). Identify risks that could affect transition success, including information security risks, and evaluate the importance of these risks relative to the agency’s mission critical systems and continuity of operations plans. All of the selected agencies had fully implemented this activity. Specifically, each of the agencies had identified transition risks and evaluated the importance of those risks relative to the agencies’ mission critical priorities. Clearly define transition preparation tasks and develop a time line that takes into account the agency’s mission critical systems, contingency plans, and identified risks. All of the selected agencies partially implemented this activity by developing time lines with clearly defined transition preparation tasks. However, none of these time lines accounted for all key priorities identified in the activity. Officials from all of the selected agencies generally said that they had not yet developed complete transition time lines because they were focused on activities associated with the acquisition planning phase of the transition, including developing their EIS solicitations. Officials from all of the agencies said that they planned to develop complete transition time lines after they issue their EIS task orders. While agencies’ lack of issued EIS task orders contributed to delays in developing complete transition plans, the limited time remaining to complete the transition makes it critical that agencies conduct early planning with the information that is available. In summary, although the 19 selected agencies reported that they plan to fully transition to EIS before the current telecommunications contracts expire in May 2023, over half of the agencies do not plan to complete the transition by GSA’s September 30, 2022, milestone to do so. By waiting until close to the end of the current contracts to finish the transition, these agencies are at risk of experiencing disruptions in service if any issues arise that result in transition delays, such as inadequate human capital resources or the need to transition previously unidentified services. Agencies also face a financial risk. During the last transition, significant delays in moving to Networx—which offered generally lower rates than its predecessor—led to hundreds of millions of dollars in missed savings. Should agencies experience similar delays in the current transition, the missed savings could also be significant. In addition, five agencies we reviewed had taken steps to prepare for the transition of their telecommunications services to EIS contracts. However, these agencies’ lack of full implementation of established planning practices increases the risk that they will experience adverse effects— such as schedule delays or cost increases—while transitioning to the new contracts. Several agencies stated that they intend to implement the planning practices after they have issued their EIS task orders. However, limited time remains to complete the transition before the current telecommunications contracts expire. Further, inadequate project planning was a key factor that contributed to delays during the prior transition to Networx. Thus, it is critical for agencies to apply a rigorous management approach from the start of the current transition using the information that is currently available, even though changes may be necessary as conditions evolve. Agencies that do not fully adopt the comprehensive approach captured in these planning practices may not make the most of the opportunity for change, and the potential to save costs, that such a major telecommunications transition provides. Accordingly, our draft report contains 25 planned recommendations to the five selected agencies. By implementing our recommendations, the agencies should be better positioned to reduce their risk of experiencing the types of delays that occurred during previous transition efforts. Because of the generally lower rates available on EIS, significant delays would lead to agencies being unable to take advantage of readily available cost savings. Chairman Connolly, Ranking Member Meadows, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have. If you have any questions concerning this statement, please contact Carol C. Harris, Director, Information Technology Acquisition Management Issues, at (202) 512-4456 or HarrisCC@gao.gov. Other individuals who made key contributions include James R. Sweetman, Jr. (Assistant Director); Emily Kuhn (Analyst-in-Charge); Christopher Businsky; Rebecca Eyler; and Javier Irizarry. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "GSA is responsible for contracts that provide telecommunications services for federal agencies. In preparation for the expiration of current telecommunications programs, GSA has developed a successor program, called EIS. GSA and agencies now must carry out the task of successfully transitioning to EIS. Previous contract transitions experienced significant delays. Those delays during the last transition resulted in hundreds of millions of dollars in missed savings. GAO was asked to summarize its draft report currently out for comment at selected agencies. The draft discusses (1) selected agencies' plans for, and status in, transitioning to EIS; and (2) the extent to which selected agencies were implementing established transition planning practices. In preparing the report on which this testimony is based, GAO administered a survey to 19 selected agencies that spent at least $10 million on telecommunications in fiscal year 2018 regarding their plans for and status in transitioning to EIS. GAO also selected five of these agencies for further review based on, among other things, agency size and structure. For these agencies, GAO evaluated documentation to determine the extent to which they had implemented five planning practices identified in a previous GAO report. GAO's preliminary results show that, as of October 2019, the 19 selected agencies reviewed were in different stages of transitioning from their soon-to-be-expiring telecommunications contracts to the new Enterprise Infrastructure Solutions (EIS) program, which has generally lower rates for services. All of these agencies reported that they plan to fully transition to EIS program contracts before the current contracts expire in May 2023. However, 11 agencies did not plan to fully transition by the General Services Administration's (GSA) September 30, 2022, milestone. The majority of the selected agencies also did not meet GSA's milestones for completing critical contracting actions in 2019 (see table). While transitioning to EIS is a complex undertaking, delays in making this transition will cause agencies to miss out on potential cost savings that would result from the generally lower rates for services on the EIS program contracts. Dates GAO's preliminary results indicate that five of the 19 agencies that were selected for further review had partially implemented established planning practices that can help agencies successfully transition their telecommunications services to new contracts. These practices are to: (1) develop an accurate inventory of telecommunications services, (2) perform a strategic analysis of telecommunications requirements, (3) develop a structured transition management approach, (4) identify the resources needed for the transition, and (5) develop a transition plan. The agencies provided several reasons for partially implementing the practices. For example, transition officials at three agencies said that they were not responsible for tracking all of the telecommunications services in use at their agencies; as such, they were unable to provide complete telecommunications inventories. The agencies also planned to implement certain practices after they issue their EIS task orders. However, the limited time remaining to complete the transition makes it critical that agencies conduct early planning with the information available and fully implement these transition planning practices to reduce the risk that the agencies experience the types of delays and missed savings that occurred in previous transitions. GAO's draft report contains 25 recommendations to the five agencies to fully implement the transition planning practices.", "document_type": "gao"}
{"report": "Our March 2019 report identified a number of areas in which HUD needs to improve its physical inspection process and its oversight of inspectors, which could help better ensure the health and safety of households that live in HUD-assisted properties. These areas include conducting a comprehensive review of the inspection process; incorporating sampling error as part of determining inspection frequency and enforcement actions; tracking whether inspections are conducted by their expected date; enhancing the process and practices related to selecting, training, and evaluating inspectors; and ensuring that new quality control policies and procedures are implemented. We found that REAC had not conducted a comprehensive review of its inspection process since 2001, although new risks to its process have emerged since then. For example, REAC staff have raised concerns that some property owners have taken advantage of the scoring system and others have misrepresented the conditions of their properties. Specifically, because more points are deducted for deficiencies on the property site than for deficiencies in a dwelling unit, some property owners prioritize site repairs over unit repairs. Additionally, some property owners attempt to cover up, rather than address, deficiencies—such as by using mulch on a building exterior to hide erosion. REAC staff also have raised concerns about property owners employing current or former REAC contract inspectors to help prepare for an inspection, sometimes by guiding owners to repair just enough to pass inspection rather than comprehensively addressing deficiencies. REAC also continues to find that some contract inspectors conduct inspections that do not meet REAC’s quality standards. Furthermore, REAC fundamentally changed the entities that conduct inspections. In 1998, REAC employed a few large inspection companies to conduct the inspections. However, in 2005, REAC introduced the reverse auction program and opened up the inspection process to a larger number of small businesses, which resulted in a change in the composition of inspectors. We found that without a comprehensive review, REAC cannot determine if it has been meeting the goal of producing inspections that are reliable, replicable, and reasonable. We recommended that REAC conduct a comprehensive review of the physical inspection process, and HUD agreed with this recommendation. In November 2019, HUD officials told us that they recently completed a comprehensive review of the physical inspection process. In supporting documentation, HUD stated that the current model was insufficient for evaluating HUD-assisted housing when compared to modern expectations of housing quality, and that there is now a need to focus more on health and safety of residents and less on asset preservation and condition and appearance items. We have been assessing HUD’s recent review to determine whether it is has fully addressed our recommendation. We also found that REAC may not be identifying all properties in need of more frequent inspections or enforcement actions because it does not consider sampling errors of the inspection scores. For large properties, REAC inspects a statistical sample of the property’s units and buildings rather than all of them. The results for the sample are then used to estimate a score that represents the condition of the entire property. HUD takes enforcement action for multifamily properties with a score below 60. However, sampling introduces a degree of uncertainty, called sampling error, which statisticians commonly express as a range associated with numerical results. For example, for a property that scored 62 on its physical inspection, due to sampling error, the range associated with this score could be between 56 on the lower bound and 68 on the upper bound. REAC would consider this a passing score that requires an annual inspection and no enforcement action, although the lower bound fell below 60. REAC previously calculated sampling errors but ceased doing so in 2013, according to REAC officials, in part because of a lack of resources and also because they believed there was no need to calculate them. Based on our analysis of REAC inspection data, HUD could have taken enforcement actions against more properties if REAC had taken sampling errors in inspection scores into account. For example, from fiscal years 2002 through 2013, about 4.3 percent of inspections of multifamily and public housing properties had an inspection score of 60 or slightly above 60 but had a lower bound score under 60. Without considering sampling errors when determining whether enforcement action is needed, REAC will not identify some properties that may require more frequent inspections or enforcement actions. We recommended in our March 2019 report that REAC resume calculating the sampling error associated with the physical inspection score for each property, identify what changes may be needed for HUD to use sampling error results, and consider those results when determining whether more frequent inspections or enforcement actions would be needed. HUD neither agreed nor disagreed with this recommendation. However, since our report was issued, HUD said that by September 30, 2020, REAC planned to include the standard error calculations in the next version of its scoring software for physical inspections. REAC officials also stated that a task team concluded that the use of sampling error likely would have no impact on any individual enforcement action. However, REAC’s statement appears to contradict its own policies because inspection scores alone are used to determine whether some properties are referred for potential enforcement actions. We will continue to monitor REAC’s actions regarding this recommendation, including how it uses sampling error results to make decisions about properties. In our March 2019 report, we also found that REAC lacked formal mechanisms to assess the effectiveness of its training program for contractors hired to inspect properties (contract inspectors) and for HUD employees responsible for monitoring and overseeing contract inspectors (quality assurance inspectors). Unlike professional inspection organizations, REAC does not have continuing education requirements. Formal mechanisms to assess the effectiveness of its training program could help REAC ensure that its program supports the development needs of inspectors. Furthermore, requiring continuing education could help REAC ensure that inspectors are current on any changes in REAC’s policies or industry standards. We also found weaknesses in REAC’s process for evaluating the performance of inspectors, which could hinder its ability to ensure the quality of inspections. We made a number of recommendations related to the selection, training, and performance evaluation of inspectors. Specifically, we recommended that HUD take the following actions: Follow through on REAC’s plan to create a process to verify candidate qualifications for contract inspectors—for example, by calling references and requesting documentation from candidates that supports their completion of 250 residential or commercial inspections. Develop a process to evaluate the effectiveness of REAC’s training program—for example, by reviewing the results of tests or soliciting participant feedback. Revise training for quality assurance inspectors to better reflect their job duties. Develop continuing education requirements for contract and quality assurance inspectors. Review performance standards for quality assurance inspectors and revise them to better reflect the skills and supporting behaviors that quality assurance inspectors need to effectively contribute to REAC’s mission. HUD agreed with these recommendations, and we have been evaluating actions it has taken in response to them since our report was issued. For example, in November 2019, HUD officials said that they were moving toward a model of contracting with larger firms to conduct physical inspections of properties. In this model, HUD plans to put the first level of responsibility on the contractor to do its own due diligence on inspector candidates, and the contractor would be required to review 25 verifiable prior inspections completed by each inspector candidate. A REAC official then would be expected to select a sample of the candidate’s inspections to review. In response to our recommendation about revising training for quality assurance inspectors, REAC said that it recently began requiring a minimum of 8 hours of continuing education annually for all quality assurance staff. As of November 2019, REAC had not yet provided us with information about the subject matter of that training. Since our report was issued, REAC also developed continuing education requirements for contract and quality assurance inspectors, which it said will be required beginning in January 2020. In addition, REAC has developed updated performance standards for quality assurance inspectors, which REAC officials said were under review. REAC considers the new standards to be more aligned with the job responsibilities of quality assurance inspectors. We also found that REAC did not always meet its schedule for inspecting multifamily properties or track progress toward meeting scheduling requirements. REAC did not meet its schedule for about 20 percent of multifamily property inspections from calendar years 2013 through 2017. On average, REAC conducted inspections for these properties about 6 months past the targeted date. REAC staff told us that there may be legitimate reasons for not conducting an inspection according to the targeted date. For example, the Office of Multifamily Housing, which oversees the performance of properties that receive project-based assistance, can delay an inspection for reasons such as natural disasters or major rehabilitation to the property. However, REAC maintains limited data on the reasons why inspections have been rescheduled or cancelled. In addition, these data are not readily available to understand retrospectively why an inspection did not occur on schedule. REAC also does not track its progress toward meeting its requirement for inspecting multifamily properties within prescribed time frames. REAC’s inability to adhere to the inspection schedule could hinder the Office of Multifamily Housing’s ability to monitor the physical condition of properties on a timely basis and take enforcement actions when warranted. Furthermore, the lack of a mechanism to track REAC’s progress toward meeting its requirement for inspecting multifamily properties hinders its ability to determine what factors have contributed to delays in conducting the inspections. In our March 2019 report, we recommended that REAC track on a routine basis whether it conducts inspections of multifamily housing properties in accordance with federal guidelines for scheduling, as well as coordinate with the Office of Multifamily Housing to minimize the number of properties that can cancel or reschedule their physical inspections. HUD partially agreed with this recommendation. Since our report was issued, REAC officials told us that REAC developed an electronic spreadsheet to better track information about its inspections, and they expect information technology enhancements that would automate the tracking of information about these inspections to be deployed by September 1, 2020. HUD’s Office of Multifamily Housing also issued a memorandum in March 2019 that provides guidance on when a field office may approve an owner’s request to delay an inspection. We will continue to monitor HUD’s actions related to this recommendation. In our March 2019 report, we found that REAC had yet to implement policies and procedures for its Quality Control group, which was formed in 2017. REAC created the Quality Control group to standardize quality assurance inspector reviews by conducting more frequent oversight and looking for trends across all quality assurance inspectors, according to a Quality Control official. In November 2018, Quality Control developed a mission statement that says that the primary goal of the group is to improve the consistency of inspections. Also in November 2018, Quality Control developed procedures for reviewing quality assurance inspectors, which include processes for conducting field reviews of completed inspections, criteria for acceptable inspections, and processes for providing feedback. An official from the group told us both its mission and procedures have not been implemented, in part because Quality Control staff repeatedly have been occupied with other special projects. Without finalizing and implementing its policies and procedures for reviewing quality assurance inspectors, Quality Control may not be able to provide consistent reviews of quality assurance inspectors, which could affect the quality of inspections and the feedback and coaching that quality assurance inspectors provide to contract inspectors. We recommended that REAC ensure that Quality Control’s policies and procedures for overseeing quality assurance inspectors are implemented, and HUD agreed with this recommendation. Since our report was issued, REAC has begun to implement this recommendation by clarifying in writing the roles, responsibilities, and objectives of the Quality Control group, including how the group plans to support changes in REAC’s inspection program. In determining the status of our recommendation, we will look for evidence that the group has been consistently implementing its policies and procedures. In addition, our March 2019 report made several other recommendations regarding the physical inspection process and oversight of inspectors. These recommendations addressed documenting the sampling methodology for the inspection process, designing and implementing an evaluation plan for assessing the effectiveness of REAC’s pilot program for staffing inspections in hard- to-staff geographic areas, implementing internal HUD recommendations, implementing a plan for meeting management targets for reviews by quality assurance inspectors, and reporting to Congress on why the agency has not complied with a Consolidated Appropriations Act requirement. HUD generally agreed with these recommendations. While HUD has taken some steps, it had not fully addressed them as of November 2019. We have been assessing the actions HUD has taken and will continue to monitor HUD’s progress toward implementing these recommendations. HUD has been undertaking significant changes to the REAC physical inspection program. In a Federal Register notice published on August 21, 2019, HUD said it was soliciting comments on a proposed voluntary demonstration of a new physical inspection process, called the “National Standards for the Physical Inspection of Real Estate.” According to HUD officials, the new inspection model is intended to address issues of inspections not always identifying health and safety conditions and properties with poor unit conditions passing inspections, among other things. HUD officials have said that a transition to the new model may take 2 years or more. HUD also has been taking steps to replace its reverse auction program with a program in which large contractors will be responsible for conducting physical inspections. We will continue to monitor HUD’s actions regarding the recommendations, as well as HUD’s activities more broadly related to implementing a new inspection model. Full implementation of the recommendations, even as the inspection program undergoes changes, can help REAC to ensure that properties are decent, safe, sanitary, and in good repair. Our June 2018 report identified a number of areas in which HUD needs to improve its efforts to identify and address lead paint hazards and protect children in low-income housing from lifelong health problems. Among other issues, we identified shortcomings in compliance monitoring and enforcement, inspection standards, and performance assessment and reporting. Our June 2018 report noted that HUD began taking steps in 2016 to monitor how PHAs comply with lead paint regulations. These steps included tracking the status of lead inspection reports for public housing properties and PHA-reported information about cases of children with elevated blood lead levels living in voucher and public housing units. However, we also identified several limitations with HUD’s monitoring efforts. For example, HUD relies in part on PHAs self-certifying their compliance with lead paint regulations, but investigations found that some PHA officials may have falsely certified that they were in compliance. Also, on-site compliance reviews performed by HUD staff can be used to determine if PHAs are in compliance with these regulations, but HUD performs a limited number of these reviews annually. In fiscal year 2017, HUD conducted these reviews at less than 2 percent of the roughly 4,000 PHAs. Finally, HUD does not have data readily available on the physical condition of the roughly 2.5 million voucher units or these units’ compliance with lead paint regulations because the individual PHAs keep these data. These limitations in HUD’s monitoring suggest that HUD may not be fully aware of the extent to which children may live in unsafe units. As a result, we recommended that HUD establish a plan to mitigate and address risks in its lead paint compliance monitoring processes. These actions could further strengthen HUD’s oversight and keep PHAs accountable for ensuring that housing units are lead-safe. HUD agreed with the recommendation. As of November 2019, HUD officials told us the agency had taken steps to implement the recommendation, including requiring PHAs to submit appropriate documentation regarding public housing units’ compliance with lead paint regulations and updating an internal checklist for on-site compliance reviews that HUD staff conduct. We will continue to monitor HUD’s progress in response to our recommendation. Our 2018 report also found that HUD did not have detailed procedures to address PHA noncompliance with lead paint regulations or to determine when enforcement decisions might be needed. HUD staff stated that they address PHA noncompliance through ongoing communication and technical assistance. However, HUD has not documented specific actions staff should perform when deficiencies are identified. Furthermore, in response to our requests for information on enforcement actions taken, HUD was able to provide information on only one enforcement action, which dated from 2013. As a result, we recommended that HUD develop and document procedures to ensure staff take consistent and timely steps to address issues of PHA noncompliance with lead paint regulations. HUD generally agreed with the recommendation. As of November 2019, HUD officials told us procedures were in draft form and under internal review and were not expected to be finalized until spring 2020. HUD officials noted that the draft procedures could help HUD staff decide when an enforcement action might be appropriate, including determining how long PHAs have to resolve noncompliance. We also found that HUD’s Lead Safe Housing Rule requires a stricter lead inspection standard for public housing than for voucher units. For public housing, inspectors must conduct a risk assessment that includes testing paint chips and dust for the presence of lead paint. For voucher units, inspectors conduct a visual assessment that includes looking for deteriorated paint or visible surface dust but does not include any testing of paint chips or samples. As a result of the different inspection standards in the two programs, children living in voucher units may receive less protection from lead paint hazards than children living in public housing. According to agency officials, HUD does not have the statutory authority to require the more stringent inspection in the voucher program. In our June 2018 report, we recommended that HUD request authority from Congress to use the stricter lead inspection standard in the voucher program as indicated by analysis of health effects for children, the impact on landlord participation in the program, and other relevant factors. In August 2018, HUD officials told us that they planned to convene a working group to design and conduct a statistically rigorous study on the impact of risk assessments to help decide whether to support statutory change for greater flexibility in strengthening inspection standards for pre- 1978 units under the voucher program. Such an analysis could be useful in evaluating the potential benefits and risks of a change in the voucher program, and we will continue to monitor the progress made by the working group. As of November 2019, HUD officials told us they were working on a demonstration proposal to test an alternative inspection standard in the voucher program. The officials noted that details of the demonstration proposal were not currently available. Separately, we have ongoing work reviewing possible changes in the inspection standard for the voucher program. This work started in September 2019 and will include an in-depth review of the impact a change in the inspection standard may have on the cost and length of time of inspections, as well as the impact on landlords and families participating in the voucher program. Our June 2018 report also identified weaknesses in HUD’s performance assessment of and reporting on its lead-safety efforts. We found that HUD had taken limited steps to measure, evaluate, and report on the performance of its programmatic efforts to ensure that housing is lead- safe. First, HUD lacked comprehensive goals and performance measures for its lead-reduction efforts. We found that HUD did not track the number of housing units in the voucher or public housing programs that were lead-safe. At the time of our report, HUD officials told us that the agency did not have systems to count the number of housing units made lead- safe in these two programs. HUD had begun discussing whether existing databases could be used to count lead-safe housing units but did not provided us with details at that time. Second, HUD had not formalized plans and did not have a time frame for evaluating the effectiveness of its lead paint regulations. Third, it had not complied with annual statutory reporting requirements and last reported on its lead efforts in 1997. We noted that by improving its measurement of whether its housing is lead- safe and evaluating and reporting on its efforts, HUD will be better positioned to inform Congress and the public about its progress toward ensuring that housing is lead-safe for residents. As a result of these findings, we recommended that HUD develop performance goals and measures, including a measure to track its efforts to ensure that housing units in its rental assistance programs were lead- safe. Additionally, we recommended that HUD finalize plans for evaluating the effectiveness of its lead paint regulations. Finally, we recommended that HUD complete statutory reporting requirements and make the reports publicly available. HUD generally agreed with these recommendations. In August 2018, HUD told us that it would use existing data systems to begin to establish a baseline for reporting lead-safe housing units in its rental assistance programs. As of November 2019, HUD officials told us they still were exploring whether current data systems could be used to count the number of lead-safe housing units in HUD’s rental assistance programs. According to HUD officials, for public housing, HUD has made progress in counting housing units that have been made lead-safe using funds from the Lead-Based Paint Capital Fund Program. However, officials told us data will not be available until spring 2020. To evaluate the effectiveness of lead paint regulations, in November 2019 HUD officials told us they planned to use data from the forthcoming update to the American Healthy Homes Survey to better estimate the prevalence of lead paint hazards in federally assisted housing. However, officials told us the findings from the updated survey likely would not be available until summer 2020. With respect to complying with statutory reporting requirements, in November 2019, HUD officials told us they planned to issue a report to Congress on the agency’s lead efforts in early 2020. We will continue to monitor HUD’s efforts to implement these recommendations. In summary, it is essential to strengthen HUD’s oversight and keep PHAs accountable for ensuring that housing units are lead-safe because children continue to test positive for lead while living in HUD-assisted housing. As of November 2019, HUD officials told us they continue to learn of confirmed cases of children testing positive for lead while living in HUD-assisted housing because PHAs are required to record the cases in a HUD database. We maintain that improvements to the areas noted in this statement today will help HUD better protect children from lifelong health problems. Chairman Clay, Ranking Member Stivers, and Members of the Subcommittee, this concludes my statement for the record. If you or your staff have any questions about this statement, please contact Daniel Garcia-Diaz, Director, Financial Markets and Community Investment, at (202) 512-8678 or garciadiazd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this statement are Beth Faraguna and Andy Pauline (Assistant Directors), Cory Marzullo (Analyst in Charge), Rachel Batkins, Carl Barden, Charlene Calhoon, Rudy Chatlos, Jeff Harner, Jill Lacey, Lisa Moore, Marc Molino, José Peña, Rhonda Rose, Jessica Sandler, Jennifer Schwartz, Tyler Spunaugle, and Nina Thomas-Diggs. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "As of the end of 2018, roughly 4.4 million low-income households were served by HUD's three largest rental assistance programs. HUD has responsibilities for ensuring that housing units provided under these programs are decent, safe, sanitary, and in good repair, as well as for identifying and addressing lead paint hazards in these units. GAO issued reports in March 2019 ( GAO-19-254 ) on HUD's physical inspections of HUD-assisted properties and in June 2018 on lead paint hazards in the public housing and voucher programs ( GAO-18-394 ). This statement is based on these two reports and discusses prior GAO findings on (1) REAC inspections and inspector oversight and (2) lead paint hazards. For the March 2019 report, GAO reviewed HUD documents and data related to REAC's physical inspection process. For the June 2018 report, GAO reviewed HUD documents and information related to its compliance efforts, performance measures, and reporting. In March 2019, GAO made 14 recommendations to HUD to improve the physical inspections process and oversight of inspectors. In June 2018, GAO made six recommendations to HUD to improve compliance monitoring processes, inspection standards, and performance assessment and reporting on lead reduction efforts in federally assisted properties. HUD generally agreed with these recommendations. As of November 2019, HUD officials had identified planned steps to implement most of these recommendations but had not fully addressed them. The Department of Housing and Urban Development (HUD) plays an important role in providing decent and safe housing for households receiving federal rental assistance. However, HUD needs to improve its physical inspection program and its efforts to identify and address lead paint hazards in federally assisted housing. To that end, GAO made 20 recommendations on these issues in its March 2019 and June 2018 reports. Physical inspections of properties. HUD's Real Estate Assessment Center (REAC) is responsible for conducting physical inspections of HUD-assisted properties. Despite longstanding processes to inspect properties and take action against owners who do not address physical deficiencies, HUD continues to find some properties in poor physical condition and with life-threatening health and safety issues. In a March 2019 report, GAO identified a number of areas in which HUD needed to improve its physical inspection process and oversight of inspectors, which could help ensure the health and safety of those who live in HUD-assisted properties. For example, REAC had not conducted a comprehensive review of its inspection process since 2001, although new risks to the process have emerged since then. A comprehensive review could help REAC identify risks and ensure it meets the goal of producing reliable inspections. In addition, REAC uses contractors to inspect properties; these contract inspectors are trained and overseen by HUD staff known as quality assurance inspectors. However, GAO found REAC lacked formal mechanisms to assess the effectiveness of its training program for contractor inspectors and for HUD employees responsible for monitoring and overseeing contract inspectors. And, unlike professional inspection organizations, REAC does not have continuing education requirements. Formal mechanisms to assess the effectiveness of its training program and requirements for continuing education could help REAC ensure its program supports development needs of inspectors and that inspectors are current on any changes in policy or industry standards. Lead paint hazards. GAO also identified a number of areas in which HUD could improve its efforts to identify and address lead paint hazards to protect children from lifelong health problems. Lead paint hazards (such as dust containing lead and chips from deteriorated lead-based paint) are the most common source of lead exposure for U.S. children. In a June 2018 report, GAO identified shortcomings in HUD's compliance monitoring and enforcement, inspection standards, and performance assessment and reporting for lead-reduction efforts. For example, HUD's monitoring efforts relied in part on public housing agencies to self-certify compliance with lead paint regulations. Additionally, the lead inspection standard for the voucher program is less strict than that for the public housing program. As a result, children living in voucher units may receive less protection from lead paint hazards than children living in public housing. Furthermore, GAO found that HUD did not track the number of lead-safe housing units in the voucher or public housing programs. Therefore, HUD may not be fully aware of the extent to which children have been living in unsafe units.", "document_type": "gao"}
{"report": "Several State and DHS components have roles and responsibilities in the E-2 adjudication process, as shown in Table 1. Depending on which agency (State or USCIS) is conducting the E-2 adjudication, as well as the foreign national’s role in relation to the E-2 business, foreign nationals are described using various terms, as shown in table 2. Both the business and foreign national seeking E-2 status must meet specific eligibility requirements, as shown in table 3. The E-2 eligibility requirements for nationals of treaty countries and their qualified family members (i.e., dependents) are defined in the INA, as amended, as well as in federal regulation. Foreign nationals seeking E-2 status must provide evidence and supporting documentation to State’s consular officers or USCIS’s immigration officers showing that they and their related business meet these requirements. There are two pathways for an individual seeking E-2 status: (1) applying for an E-2 visa through State at a post abroad, and then being inspected and admitted at a U.S. port of entry by CBP, or (2) filing with USCIS to extend, or change to E-2 status if already in the United States in E-2 or other nonimmigrant status, as shown in figure 1. Prior to the expiration of the 2-year period typical for E-2 nonimmigrants, a foreign national seeking to remain in E-2 status must either petition USCIS for an E-2 extension; or depart the country, reapply for an E-2 visa with State at a U.S. embassy or consulate, and seek entry at a U.S. port of entry. However, if the E-2 visa is still valid after having departed, the foreign national may present that visa to apply for admission again at a U.S. port of entry. If applying through State, consular officers are responsible for adjudicating E-2 visa applications at one of State’s 220 posts. Although all posts can adjudicate E-2 visas, approximately 140 posts adjudicated at least one E-2 visa in fiscal year 2018. Taken together, State and USCIS adjudicated an annual volume of E-2 visa applications or petitions of more than 50,000 from fiscal years 2014 through 2018. State accounted for over 80 percent of these adjudications. About 90 percent of State’s E-2 visa applications were issued, and about 83 percent of USCIS’s E-2 petitions were approved. See appendix III for additional State and USCIS data on the characteristics of foreign nationals seeking E-2 status, including annual statistics, the relatively low number of E-2 nonimmigrants who remain in the United States beyond the conclusion of their authorized period of stay (i.e., overstay), and other post-adjudication outcomes. The volume of State’s E-2 visa adjudications increased from fiscal years 2014 through 2017, and decreased slightly in fiscal year 2018, as shown in figure 2. During this time period, State consular officers adjudicated an average of about 45,000 E-2 visas per year. Also during this time period, 44 percent of adjudications were for dependents, and a combined 53 percent were for principals, including 14 percent for the investor, 20 percent for managers, and 19 percent for essential employees. From fiscal years 2014 through 2017, the average E-2 visa refusal rate— that is, the number of refused visas divided by the total number of visas adjudicated during that time period—was about 8 percent, which is generally lower than for other types of nonimmigrant visas (see sidebar). We do not present the fiscal year 2018 refusal rate in figure 3 because that rate is subject to change until the end of fiscal year 2019. Specifically, an application adjudicated in fiscal year 2018 may require the applicant to submit additional information to demonstrate eligibility for an E-2 visa. In such cases, the application is refused under INA § 221(g). The applicant has one year after the date of refusal to overcome the refusal by, for example, providing missing or supplemental information. After one year, the applicant must reapply. As of November 2018, 8,184 of the 11,255 refusals in fiscal year 2018 were refused under INA § 221(g). Depending on the extent to which applicants refused in fiscal year 2018 under INA § 221(g) are able to overcome their refusals, State officials stated that they expected the fiscal year 2018 refusal rate to be similar to prior fiscal years. In addition to analyzing State data on adjudications and refusals, we also analyzed data to identify trends in refusal rates by applicant type, refusal reasons, nationality of applicants, and business sectors, and level of investment, as described below. Refusal Rates by Applicant Type. Our analysis showed that for fiscal years 2014 through 2018, average refusal rates were highest for investors (24 percent), followed by dependents (12 percent), managers (9 percent), and essential employees (6 percent). Figure 4 shows the refusal rates by fiscal year for each applicant type, and appendix III includes additional information on refusal rates for fiscal year 2018. According to State officials, refusal rates may be higher for investors because such applicants are typically the first in their company applying for an E-2 visa; if denied, then future E-2 applicants (e.g., manager or essential employee) would need to wait until such investor is approved or find another individual or business investor to form the basis for their E-2 employment status. Refusal Reasons. Our analysis showed that approximately 10 percent of E-2 visa adjudications from fiscal years 2014 through 2017 were refused. The majority of E-2 visa refusals for fiscal years 2014 through 2017 (75 percent) were because the applicant did not meet eligibility requirements. The next largest reason for refusal (22 percent) was INA § 221(g) for inadequate documentation. Few E-2 visa applicants are refused for other reasons, such as prior immigration violations, fraud, or terrorist activities. For example, in total, less than 4 percent of all E-2 visa adjudications during this time period were refused for other reasons, such as security or criminal-related ineligibilities, fraud or misrepresentation, and immigration violations, among others. Nationality. Our analysis showed that about 80 percent of E-2 visa adjudications from fiscal years 2014 through 2018 were for nationals from nine countries: five European countries (Germany, France, United Kingdom, Italy, and Spain), two Asian countries (Japan and South Korea), and two North American countries (Canada and Mexico). Japan was the largest country of nationality, with 29 percent, followed by Germany (10 percent), Canada (7 percent), and France (7 percent). Figure 5 shows the top ten countries by percentage of E-2 visa adjudications from fiscal years 2014 through 2018. Business Sectors. To obtain information on additional characteristics of E-2 visa principal applicants (i.e., investor, manager, and essential employee), such as their business sector and investment amounts, we reviewed a generalizable sample of 120 fiscal year 2018 E-2 visa applications. Based on our analysis, we estimate that about three- fourths of principal E-2 visa applicants were associated with 4 business sectors: manufacturing (44 percent), food services (13 percent), retail (11 percent), and professional services (10 percent). Figure 6 includes examples of the businesses we found within each of these sectors. Investment. Based on information reported by fiscal year 2018 principal applicants in our generalizable sample of issued visas, we estimate 64 percent of applications were for principal applicants associated with investments reportedly over $10 million, as shown in figure 7. Of these, 30 of 40 applications were for those in the manufacturing sector, particularly for the automotive sector, such as large automobile manufacturers. From fiscal year 2014 through 2018, USCIS adjudicated an average of about 9,400 E-2 petitions per year. During this time period, USCIS adjudicated petitions to extend E-2 status for an average of about 5,900 beneficiaries per year, about 60 percent of which were for E-2 dependents (i.e., an E-2 principal’s spouse or children). Also during the same time period, USCIS adjudicated petitions for an average of about 3,500 beneficiaries per year who were seeking to change to E-2 status from another nonimmigrant category. Of these, about 47 percent of which were E-2 principal beneficiaries (i.e., investors, managers, and essential employees). Figure 8 shows the number of petitions to extend or change to E-2 status from fiscal years 2014 through 2018. The average denial rate for E-2 petitions for fiscal years 2014 through 2018 was about 17 percent. Denial rates were higher for petitions to change status from another nonimmigrant category to E-2 (27 percent) than for petitions to extend E-2 status (11 percent), as shown in figure 9. Further, the denial rate for both extension and change of status petitions increased from fiscal years 2014 through 2017, but fell by several points in 2018. In addition to analyzing USCIS data on adjudications and denials, we also analyzed data to identify trends in country of birth, prior status, date of last U.S. entry, reasons for denial, business sectors, and level of investment, as described below. Country of Birth. Our analysis showed that the top countries of birth for individuals seeking to extend their E-2 status from fiscal years 2014 through 2018 were South Korea, Mexico, and Japan, and the top countries of birth for those seeking to change to E-2 status from another nonimmigrant category were South Korea, Pakistan, and Turkey, as shown in table 4. Although there are similarities with the top countries of nationality for State E-2 visas (see previous figure 5), there are some differences as well. For example, both Pakistan and Thailand are among the top countries of birth for petitioning with USCIS to extend or change to E-2 status, but are not among the top countries of nationality for State E-2 visas. Prior status. Our analysis showed that individuals seeking to change to E-2 status from another nonimmigrant category from fiscal years 2014 through 2018 were most often changing status from a tourist, business, or student visa, as shown in figure 10. For example, more than half (53 percent) of all petitions to change to E-2 status were for beneficiaries that were tourists (B-2) or business visitors (B-1). In addition, about 4 percent of beneficiaries were seeking to change status within the E-2 classification. For example, a child or spouse of an E-2 investor may later work at the company as a manager and therefore would need to petition to change from dependent to principal E-2 status as a manager. Date of last entry into the United States. On the basis of our review of a generalizable sample of petitions of E-2 principals (i.e., investors, managers, and essential employees), we estimate that one third of principal beneficiaries had been in the United States since 2014 or earlier at the time they sought to change to or extend E-2 status in 2018, some as long as 18 years, as shown in figure 11. Such beneficiaries may have changed status from other kinds of nonimmigrant status, or may have requested to extend their E-2 status multiple times. There is no limit on the number of times a foreign national may request to extend their E-2 status. Reason for denial. On the basis of our review of a generalizable sample of fiscal year 2018 denied petitions for E-2 principals, we estimate that the top reasons petitions were denied included (1) the enterprise was not real and operating, and (2) the investment was not substantial, as shown in table 5. Of the denied petitions in fiscal year 2018, about one-third were either withdrawn by petitioner or abandoned, meaning that the petitioner did not respond to USCIS requests for additional evidence. Business Sectors. On the basis of our review, we estimate that the majority of E-2 principal beneficiaries were associated with 4 business sectors, as shown in figure 12: food services (38 percent), retail (18 percent), manufacturing (9 percent), and professional services (13 percent). Comparing our two generalizable samples, a smaller percentage of USCIS’s E-2 principal beneficiaries were associated with manufacturing (44 versus 9 percent) and more with food services (13 versus 38 percent) than State’s E-2 principal visa applicants. Investment. We estimate that about two-thirds of the approved petitions were for principal beneficiaries associated with investments of $200,000 or less, as shown in figure 13. We found that about 30 percent of USCIS’s E-2 principal beneficiaries were associated with investment amounts of $100,000 or less and 7 percent were associated with investments over $10 million. State and USCIS have agency-specific guidance, procedures, and training intended to ensure E-2 applicants and petitioners, respectively, meet E-2 eligibility requirements. However, officials from both agencies identified challenges in the E-2 adjudication process. Some of State’s posts have developed E-2 company registration programs to help streamline the E-2 adjudication process, but there are no minimum standards for these programs, which may result in different processing of companies and applicants across posts. Further, State and USCIS require that consular and immigration officers retain certain documentation for all E-2 applications and petitions; however, during our case file review of E-2 applications and petitions adjudicated in fiscal year 2018, we found that State did not consistently retain all required documents. State and USCIS have guidance and resources to help officers adjudicate E-2 applications and petitions. Both agencies have similar high-level procedures for adjudicating E-2 applications and petitions, but there are some key differences in how each agency implements these procedures based on their specific roles and responsibilities. Further, both agencies provide their staff with some training on E-2 eligibility requirements. Guidance and resources. State and USCIS have guidance and resources available to staff who adjudicate E-2 visas and petitions to help ensure that applicants and petitioners meet E-2 eligibility requirements. Although the guidance documents have some minor differences, they are based on the same eligibility requirements. For example, the main guidance documents for State and USCIS—State’s Foreign Affairs Manual (FAM) and USCIS’s national E-visa standard operating procedures—both include the same eligibility criteria and provide additional explanation on each of the eligibility requirements. State also provides supplementary resources for consular officers on its intranet, such as E-2 adjudication best practices, an adjudication guide, and case studies. State and USCIS both provide headquarters-based legal advisors and attorneys with whom officers can consult for case-specific guidance. For example, a State consular officer at one post we visited told us that he requested such assistance for an application from an investor whose company had a particularly complex ownership structure that made it difficult to determine if at least 50 percent of the company was owned by nationals of a treaty country. Adjudication procedures. State and USCIS high-level procedures for adjudicating E-2 applications and petitions are generally similar, but there are some key differences based on their specific roles and responsibilities. As shown in figure 14, both agencies require foreign nationals to submit an E-2 application or petition, and pay any relevant fees. Additionally, both agencies vet individuals by conducting security checks and reviewing submitted information to ensure that all E-2 eligibility requirements are met. There are four key differences in State and USCIS procedures for adjudicating E-2 visa applications and petitions: Interviews. State requires in-person interviews of most E-2 applicants. According to USCIS officials, USCIS does not conduct interviews of beneficiaries and petitioners because they do not have the resources or facilities to do so. In any case, USCIS’s process for adjudicating nonimmigrant visa petitions for foreign nationals who have already been lawfully admitted into the United States, in E-2 or other nonimmigrant status does not include an interview requirement. Locally Employed Staff (LES) and E-2 Visa Adjudication Consular officers and managers stated that LES play an important role in E-2 visa processing and adjudication. LES are employees hired under the local compensation plan at a U.S. post overseas. LES include foreign service nationals, U.S. citizens residing abroad, third country nationals, and eligible family members of State employees. LES can provide the institutional knowledge and expertise in E-2 visa issues, as consular officers rotate posts every 2 years but LES do not rotate. Consular managers at 4 of the 14 posts we interviewed or visited stated that their post specifically hired LES to work on E-2 visas because of their specialized knowledge and backgrounds in business or law. For example, a consular officer may consult with LES on an application to better understand the legal relationship between two companies, as some LES have a background or developed expertise in financial law. Locally Employed Staff (LES) initial processing and prescreening. In addition to consular officers, State employs local residents in its host country to help with consular services (see sidebar). For example, at some posts State’s LES prescreen visa applications before consular officers adjudicate the application. Procedures for LES varied at the posts we interviewed and visited. For example, LES at some posts provide administrative help and processing—such as scanning application documents, checking applications for completeness, and scheduling interviews. LES at other posts provide additional analytical support—such as by summarizing applications, completing eligibility checklists, and maintaining databases on previously issued E-2 visas. Regardless of the kind of help LES may provide at post, only consular officers adjudicate E-2 visa applications and make decisions on whether or not the visa is issued. The number of LES supporting E-2 visa applications at the 14 posts we visited or interviewed ranged from one part-time position to five full-time LES. Consular managers and officers at all four of the posts we visited described the role of LES in processing E-2 visas as critical (see sidebar). Although USCIS’ California Service Center has staff who assist with processing petitions, such as by organizing folders with the petition materials, immigration officers generally perform the analytical tasks themselves. Staffing model. Depending on E-2 visa application volume, staffing considerations, and workload arrangements, the number of consular officers adjudicating E-2 visas at the 14 posts abroad we interviewed ranged from one to six per post. Further, on the basis of our observations and interviews with consular officials at 14 posts, we found that State’s posts have generally developed three different staffing models for adjudicating E-2 visa applications, as shown in table 6. Consular managers stated that the kind of model used at a post may depend on E-2 visa volume, as well as other factors. For example, a consular manager at a post we visited explained that the specialist model worked well at his post because it had a relatively low volume of E-2 adjudications each year, which meant that a single officer could focus on such visas. In contrast, a consular manager at a post we visited that was staffed with a hybrid of generalists and specialists had higher E-2 visa volume and stated that their model allowed them to balance efficiency and specialization. For USCIS, a specialized office of five immigration officers review and adjudicate all E-petitions (including E-1 and E-2) at one location –USCIS’ California Service Center, as of July 2018. Training. State and USCIS provide training to their respective E-2 processing and adjudication staff on E-2 eligibility requirements. State’s consular officers assigned to adjudicate E-2 visas receive the majority of their adjudication training at post, with a brief introduction to E-2 visas during a mandatory 6-week Foreign Service Institute training course taken prior to serving as a consular officer overseas. According to Foreign Service Institute officials, the course provides consular officers with an overview of the various visa classes they may adjudicate, but focuses on visas that all consular officers will address at post. Because E-2 visas are not adjudicated at every post, and consular officers typically cannot specialize in only one particular classification like USCIS counterparts who have a dedicated E-2 unit, the course does not concentrate on that visa classification. Instead, State relies on the individual posts to provide training to prepare consular officers to adjudicate E-2 visas on an “as needed” basis. On the basis of our interviews and observations, we found that E-2 training programs for consular officers at post generally consist of three components. First, consular managers and senior consular officers at post provide the consular officer who will be adjudicating E-2 visa applications for the first time with an overview of the E-2 eligibility requirements along with any supplementary E-2 training resources, such as illustrative examples of challenging E-2 visa cases the post has previously adjudicated. Second, new consular officers are to observe senior consular officers adjudicate E-2 visas for 1 to 3 weeks, which helps the new officer to learn how the requirements are applied. Finally, new officers adjudicate E-2 visas under the supervision of a senior consular officer with experience adjudicating E-2 visa applications, with 100 percent of their adjudications reviewed by consular managers until management determines that the new officer is proficient. As needed, supervisors will meet with new officers to discuss specific adjudications, including whether the officer properly documented their decision. State’s E-2 training for LES is entirely at post. According to consular managers and LES, LES training generally consists of a review of eligibility requirements and supervision. First, new LES assigned to E-2 visa processing and prescreening receive an overview of the E-2 eligibility requirements from a senior LES. According to LES we interviewed, the overview of the eligibility requirements helps them to identify the types of documents E-2 applicants typically submit to establish E-2 eligibility. Second, new LES are observed by senior LES until management determines that the LES is proficient at processing and prescreening. As noted above, USCIS has staff dedicated to E-2 petitions and USCIS provides training to new E-2 immigration officers that include the same basic components as State, such as a review of eligibility requirements and job shadowing. First, immigration officers who will work on E-2 adjudications receive 3 weeks of classroom training during which they review the E-2 eligibility requirements. The classroom training is followed by a 1-week practicum session where USCIS immigration officers apply the classroom training to sample E-2 petitions. Specifically, immigration officers explained to us that during the practicum they are given example cases to which they are to apply their classroom training. After each officer has adjudicated the example case, they discuss how each applied the various E-2 eligibility requirements and reconcile any differences with the assistance of the immigration supervisor facilitating the training. Second, after the 4 weeks of training, USCIS immigration officers begin to adjudicate E-2 petitions under the guidance of an E-2 immigration supervisor. Third, new E-2 immigration officers have 100 percent of their cases reviewed by their supervisor until they are deemed proficient. State’s consular officers and LES, as well as USCIS officials, stated that given the complexity of adjudicating E-2 applications and petitions, and the level of documentation and time required, the E-2 adjudication process can present challenges with respect to the analysis of the E-2 eligibility requirements. Consular officers and LES we spoke with stated that additional training on E-2 eligibility requirements would be beneficial. USCIS officials said that while E-2 petitions can be challenging to adjudicate, additional training was not necessary. Consular officers we spoke with noted that E-2 visa adjudications are particularly complicated and resource-intensive, involving potentially complex business issues, and often requiring more documentation and time to adjudicate than is typically needed to adjudicate other visas. Specifically, consular officers at 10 of 14 posts we interviewed stated that E-2 visas are among the most difficult nonimmigrant visas to adjudicate because of the amount of supporting documentation that is required to demonstrate that both the business and applicant meet all eligibility requirements, as well as the time required to prescreen and adjudicate the application package. For example, E-2 application packages can include 200 pages or more of supporting documentation, and include a range of detailed business and financial documents (see sidebar). Further, consular officers told us that it can take between 45 minutes to 4 hours to review a single E-2 application with its supporting documents. Consular officers explained that, in contrast, other nonimmigrant visa categories do not require the same amount of time or number of documents to adjudicate. For example, business and tourism nonimmigrant visas typically take less than 10 minutes to adjudicate and do not require that any documentation be submitted by the applicant prior to the adjudication. Consular officers at the 14 posts we visited or interviewed identified challenges with respect to the analysis of the E-2 eligibility requirements. Table 7 provides examples of some of these challenges, as identified by consular officers at the 14 posts. Substantial investment requirement: No prescribed minimum amount of capital, although it must be substantial in proportion to the cost of the business. Sufficient to ensure the investor’s financial commitment to the successful operation of the business. Large enough to ensure the likelihood of success of the business. Determining substantial investment. Consular officers at 10 of 14 posts indicated that it can be challenging to determine substantiality of capital investment amounts. According to the FAM, there is no set amount of capital which is considered substantial; instead, various factors must be considered to ensure there is a large enough investment to support the business. Consular officers noted that it can be difficult to determine how much capital is needed to support the many types of businesses that consular officers see in E-2 applications, which can range from small restaurants to technology start-ups to large automobile manufacturers. For example, a consular officer may be presented with an application for an investor seeking an E-2 visa to open a business that the consular officer has never seen before in an E-2 visa application, such as an airport internet café that rents hourly sleeping pods to travelers on long layovers. The consular officer may be initially unfamiliar with what is considered to be a more unique type of business, and may not know immediately how much investment would be sufficient to ensure the successful operation of the business. In such cases, the officer might gather additional information from the applicant on similar businesses, which the officer could use to inform their determination as to the amount of capital that would be needed to support successful operation of the business in the United States. Real and operating business requirement: The business is a real and active commercial or entrepreneurial undertaking that produces goods (i.e. commodities) or services for profit, and meets applicable legal requirements for doing business in the particular jurisdiction of the United States. Determining real and operating business. Consular officers at 7 of 14 posts indicated that it can be challenging to determine whether the business is real and operating. Consular officers explained that particularly difficult issues may arise for new businesses, which may not be operational yet at the time of the interview. Consular officers stated that it can be very clear when a business is not yet operating, but that additional analysis is required for newly-formed businesses that do not yet have customers or revenue but may have taken other actions to start the business. Consular officers at one post explained it is sometimes very clear that a business is not operating because, for example, the business has not yet made any contracts with clients, does not have a website advertising its services, and has no evidence of any expenses made on behalf of the business. As for newly-formed businesses, consular officers at another post we visited provided a hypothetical example of a restaurant whose owner had a lease for the restaurant space, bought equipment, and hired employees, but had not opened to customers yet because it was waiting for the chef to receive an E-2 visa as an essential employee. The officers indicated that in such a hypothetical scenario in which a business’s qualification as an E-2 business depends on E-2 visa issuance of a key worker, it may not be immediately clear without further analysis, whether such business would be considered real and operating. Manager requirement: The individual is an employee in an executive or supervisory position. Determining manager qualifications. Consular officers at 6 of 14 posts indicated that it can be challenging to determine whether a prospective manager had or will have sufficient executive or supervisory duties to meet the E-2 managerial requirement. Consular officers provided a hypothetical example in which a consular officer may interview an applicant seeking an E-2 visa to become a manager at a restaurant, but the applicant may not have any prior management experience nor will she have any subordinates in the restaurant. Such a situation may pose challenges to the consular officer to determine if the applicant would be eligible for an E-2 visa as a manager. Officers noted that the FAM requirements did not specifically state that the applicant must have prior experience or subordinates to qualify as a manager. In such situations, consular officers said they might request additional information from the applicant about the restaurant, her skills and experience, and the nature of her managerial role in the business. Essential employee requirement: The individual is employed in a lesser capacity than a manager, but possesses special qualifications (i.e. skills and/or aptitudes) essential to the business’ successful or efficient operations in the United States. Determining essential employee qualifications. Consular officers at 6 of 14 posts indicated that it can be challenging to determine whether a prospective essential employee has special qualifications (i.e. essential skills or aptitudes). Consular officers noted that they can ask questions and obtain information about the applicant’s specialized skills, but that often further research is needed to determine if those skills are essential to the business’ operations in the United States. For example, an officer at one post we interviewed provided a hypothetical example of a pet groomer seeking an E-2 visa as an essential employee for a pet grooming service. Although one might be skeptical that pet grooming is a specialized skill and that such an employee would be considered essential, in such a situation, the officer noted that he would likely conduct further research. In doing so, he might determine that the applicant is a well-known expert who specializes in grooming certain breeds of exotic or show animals, and that the grooming service is planning to target that type of animal. Other requirements. Consular officers told us that some of the other E-2 eligibility requirements are not particularly challenging. For example, consular officers at all 14 posts told us that it is relatively straightforward to determine if the applicant has a clear intent to depart the United States upon termination of E-2 status because applicants typically provide an affidavit attesting to their nonimmigrant intent. Further, consular officers stated that it is easy to determine if the applicant is an eligible dependent because consular officers are familiar with local identity information (e.g., birth and marriage certificates) and there are no nationality requirements for dependents. In addition to potential challenges with respect to the analysis of the eligibility requirements, consular officers at 4 of 14 posts also identified challenges in understanding business and financial documents that are provided in support of an E-2 application. For example, at one post we visited, a consular officer explained the challenges he faced in understanding U.S. tax documentation and the differences between various types of corporations. Further, consular managers at two posts stated that officers without prior knowledge in basic business concepts can find E-2 visa adjudication challenging when they first arrive at post. A manager from a third post stated that the complexity of some E-2 visa cases requires knowledge of business and finance acquired through substantial experience or education. More than marginal business requirement: The investment must be made in a business that has the capacity to generate more than enough income to provide a minimal living for the treaty investor or employee and family, or has the present or future capacity (generally within five years) to make a significant economic contribution. Although LES do not adjudicate visas, LES at 6 of 14 posts also indicated that they had encountered challenges with respect to the analysis of the E-2 eligibility requirements. For example, LES at one post indicated that it can be challenging to determine whether a company is more than marginal (see sidebar) because the size, type or investment sector of each E-2 company presents unique facts and circumstances. LES at one post told us that they needed additional examples of how applicants can meet the various criteria, which would help the LES flag potential areas of concern for the consular officer. Further, LES also expressed challenges in understanding some business and financial aspects of prescreening. For example, LES at two posts stated that determining the nationality of large companies can be difficult because they need to trace back ownership to the original, parent company, and that corporate structures can be very complicated. Given the complexity of adjudicating E-2 visas, the majority of consular officers and consular managers we spoke with stated that additional training and resources would be beneficial, such as online training, conferences to share best practices, or documents clarifying eligibility requirements. Specifically, consular officers at 9 of 14 posts and consular managers at 8 of the 14 posts stated that additional E-2 training or resources would be beneficial to consular officers. For example, a consular manager at one post noted that the additional resources provided on State’s intranet, such as the adjudication guide and case studies, have already helped to improve clarity on the eligibility requirements, but more resources and training are needed. Further, consular managers at 4 posts stated that additional training related to tax and business concepts would be useful. For example, one manager stated that additional training on how to read and analyze U.S. tax returns could be helpful to accurately evaluate a company’s overall financial health and make a determination that a business meets the requirement to be “more than marginal.” Further, LES at all 14 posts in our review also stated that additional training or resources would help them perform their responsibilities. For example, LES at one post we visited stated that additional training and resources that clarify the eligibility standards would allow them to better prepare application packages for the consular officers to adjudicate. Further, consular managers at 9 of the 14 posts in our review also stated that additional training and guidance for LES would be helpful. For example, one consular manager suggested that State develop an online training course for both E-2 adjudicating officers and LES that reviews common business documents. Another manager stated that a training or workshop would provide opportunities to LES and E-2 adjudicating officers to learn best practices from other posts that adjudicate E-2 visas. Although State provides guidance and training on adjudicating E-2 visas, consular officers, managers, and LES identified challenges in the E-2 adjudication process, such as ensuring adjudicators adequately understand supporting financial and business documents. Many of these officials indicated that given the complexity of E-2 adjudications, additional training and resources would help them in making E-2 eligibility determinations. State officials noted that eligibility requirements are broadly defined so as to cover various business types and investment amounts. According to the Standards for Internal Control in the Federal Government, management establishes expectations of competence for key roles to help the entity achieve its objectives, which requires that staff have the relevant knowledge, skills, and abilities, needed to carry out their responsibility. Such knowledge, skills, and abilities can be obtained by on-the-job training, formal training, and other training resources, which should be available to all staff performing such roles, regardless of their post. Providing additional E-2 training or related resources would help better ensure that all consular officers and LES prescreening and adjudicating these visas have the necessary knowledge, skills, and abilities to carry out their responsibilities effectively. Such training or other resources should cover topics that include information on E-2 eligibility requirements and how to understand business- and tax-related documents. USCIS Immigration Officers Identified Challenges in Adjudicating Petitions and Noted Ways in Which They Address Them USCIS immigration officers we spoke with communicated challenges with respect to the analysis of E-2 eligibility requirements, but explained that they are able to overcome these challenges with local resources. For example, USCIS immigration officers indicated that it is sometimes challenging to determine whether a prospective “essential employee” has requisite special qualifications, or a business is “more than marginal.” For example, immigration officers indicated that determining if an employee is considered essential depends on the relevant facts and circumstances. Further, immigration officers noted that the non-marginality eligibility requirement can be difficult to determine in some cases because the officer may have to project how successful the business will be in the future. However, the immigration officers explained that their colocation with all of the other immigration officers who adjudicate E-2 petitions helps to mitigate the challenges because the officers can coordinate with each other to determine how USCIS has typically adjudicated such cases. Generally, the USCIS immigration officers stated that additional training or resources for E-2 adjudication was not needed. As of April 2019, 7 of the top 10 E-2 adjudicating posts worldwide have implemented E-2 company registration programs. An E-2 company registration program is a process by which posts assess companies against applicable E-2 eligibility requirements. Companies that meet eligibility requirements are placed on an approved or registered companies list. Companies on the registered list do not have to be reassessed for eligibility each time one of their employees seeks an E-2 visa, which creates processing efficiencies for these posts. Consular managers stated that E-2 company registration programs are intended to give consular officers reasonable assurance that a company meets the minimum E-2 business and investment eligibility requirements, allowing the adjudicating officer to focus the majority of their effort on evaluating the applicant ‘s E-2 eligibility. In fact, we found that at posts with E-2 company registration programs, the consular officer may not need to collect or review any supporting documentation related to the company prior to adjudicating the visa. In contrast, E-2 adjudicating posts without an E-2 company registration program would assess both the company and the applicant against the E-2 eligibility criteria each time they review and adjudicate an E-2 visa application. While State has identified E-2 company registration programs as a potential best practice, these programs are not mentioned in the FAM and State has not developed guidance or minimum standards for how these programs should be implemented. Instead, State has permitted posts to develop and implement their own registration programs, which has led to variation in how the programs are implemented depending on post- specific factors. Specifically, we found that posts with E-2 company registration programs varied in three ways: Registration criteria: Three of the 7 posts with E-2 registration programs require all companies to register, while the remaining 4 posts established criteria so that only certain companies can register, such as large companies or companies with multiple E-2 visa issuances. For example, at one post, only companies with more than 500 employees in the United States are allowed to register. At posts that require all companies to register, the number of registered companies ranged from approximately 2,200 to 4,000. At posts that allow only certain companies to register, the number of registered companies ranged from about 100 to 200. Documentation requirements: Employees of E-2 registered companies seeking to obtain an E-2 visa provide different types of documentation during their E-2 adjudication, depending on the requirements of the post. For example, at two posts, applicants of registered E-2 companies must provide their resume and a company letter that outlines the applicant’s specific role within the company, and do not need to provide any other supporting documentation regarding the company or underlying investment. At these posts, consular officers review their E-2 company registration database to ensure that the company in question is registered with the post’s E-2 company registration program. Revetting policy: Two of 7 posts with E-2 company registration programs vet registered companies annually while the remaining five posts vet companies every 5 years. Consular managers added that if changes, such as changes in ownership, occur without the post knowing it, prospective applicants may no longer be eligible for the visa. However, according to consular managers, companies on the list are required to contact their post sooner than the 5- or 1- year renewal period if there are any changes in the company that would impact visa eligibility for company investors or employees. Although such programs may allow posts to more efficiently adjudicate E- 2 visas, the variation in these programs may result in different processing of companies and applicants across posts, as well as acceptance of varying levels of risk by posts. The more time a post allows companies before reassessing the company’s eligibility for registration, the more risk that post is assuming, as the companies may no longer meet the eligibility requirements and continue to send or keep employees in the United States on E-2 visas for which they are not eligible. According to Standards for Internal Control in the Federal Government, management should design and implement policies and procedures that enforce management’s directives to achieve the entity’s objectives and address related risks. However, State’s Bureau of Consular Affairs has not provided posts with minimum standards governing the implementation of E-2 company registration programs, and thus, it is unclear whether the variations among these programs are consistent with the agency’s requirements and objectives. Establishing minimum standards for posts that choose to implement such programs would better ensure that all posts’ E-2 visa adjudication processes are aligned with State’s policies, objectives, and risk tolerance. State and USCIS require certain information and documents be retained for all E-2 applications and petitions; however, during our file review of State and USCIS E-2 adjudications, we identified that some required documents were missing from State files; USCIS was able to provide copies of all the documents required to be retained for each file we reviewed. State. State’s FAM includes requirements related to the collection of E-2 visa application information for all E-2 principals (i.e. investors, managers, and essential employees). Principal investors provide their information when they complete their application online, which is automatically uploaded to State’s consular database system. However, managers and essential employees provide some information by completing a paper form DS-156E, and the FAM requires officials to scan the forms each applicant’s record. On the basis of our file review, we estimate that about 20 percent of fiscal year 2018 E-2 application files for managers and essential employees were missing required documentation, either in part or in full. Specifically, 14 percent of E-2 applications were missing the entire DS- 156E, and 8 percent (6 of 80) were missing pages of the DS-156E. According the Standards for Internal Control in the Federal Government, management performs ongoing monitoring of the design and operating effectiveness of the internal control system as part of the normal course of operations. Ongoing monitoring includes regular management and supervisory activities. According to State officials, the responsibility for ensuring that document retention is consistent with standards rests with posts, and consular managers are responsible for ensuring compliance. State officials noted that the Bureau of Consular Affairs does not have an ongoing monitoring process in place to ensure that posts are complying with the FAM requirement. Developing a process to ensure that posts are retaining all required E-2 visa documentation by monitoring implementation of the requirement could better position State to be able to access applicant information, should it be needed for law enforcement, anti-fraud, or security purposes later. USCIS. According to USCIS officials, USCIS requires the I-129 petition, supporting documentation, and decision letters for refused petitions to be retained for all petitioners. As part of our review of petition files, we requested 124 randomly selected fiscal year 2018 petition files for investors, managers, and essential employees. In response, USCIS was able to provide us with all of the required elements for each of the petition files. State has resources to help combat nonimmigrant visa fraud, including for E-2 visas. State officials said that the resources available and the steps they take if E-2 fraud is suspected are similar for all types of visa fraud. If a consular officer reviewing an E-2 visa application suspects fraud— either during prescreening or after the interview—the officer is to make a fraud referral to the post’s fraud prevention manager or to diplomatic security officials. According to State officials, not every case with potential fraud concerns will be referred for additional investigation. If a consular officer does not find the applicant to be qualified or overcome immigrant intent, officers may refuse the case without additional fraud assessments. Fraud prevention managers, who are part of State’s Bureau of Consular Affairs, investigate fraud cases and provide information on fraud trends to consular officers. At some posts, State’s Bureau of Diplomatic Security’s ARSO-Is specialize in criminal investigations of visa fraud and coordinate with local law enforcement. Both fraud prevention managers and ARSO-Is are to conduct additional research to determine if fraud exists, such as through open source searches, interviews, and coordination with other U.S. and local government entities. State officials we spoke with stated that they take fraud in all visa fraud categories seriously, but generally consider E-2 visa fraud to be lower risk relative to other visa categories because they believe the large amount of complex paperwork required for the visa would discourage malicious actors. For example, consular officers at 12 of the 14 posts we interviewed stated that E-2 visas were a low fraud risk. Similarly, consular managers at 10 of the 14 posts stated that E-2 visa fraud was generally not a concern at their post. State headquarters officials attributed the low fraud risk to the large amount of paperwork that is required, which includes complex financial documents and U.S. government produced tax forms. For example, State headquarters officials indicated that, given the documentation burden for both the applicant and the company, the E-2 nonimmigrant classification may be less susceptible to fraud than other nonimmigrant classifications. According to State’s E-2 fraud data, the number of E-2 fraud referrals has decreased since fiscal year 2015, but the number of confirmed fraud cases was consistent from fiscal years 2014 through 2018, as shown in figure 15. There was an initial increase in referrals from fiscal year 2014 to 2015, which State officials attributed to consular staff more consistently making such requests through the official system of record rather than by email. From fiscal years 2015 through 2018 the number of E-2 visa fraud referrals decreased each year, from 664 in fiscal year 2015 to 280 in fiscal year 2018. Throughout this time period, the number of confirmed fraud cases stayed about the same, ranging from 39 to 59 cases per year. Although consular officials at 12 of the 14 posts considered E-2 visas to be low fraud risk, consular officers also identified country-specific E-2 fraud trends and indicators that they monitored at their post, as appropriate, such as the type of business, the location of the business, or the nationality of the applicant. Some of the posts in our review have taken additional actions to address E-2 fraud, such as additional fraud reviews and conducting validation studies: Additional fraud review: Consular managers at one post told us that the post has devoted additional resources to ensure that all E-2 visa applications undergo an additional fraud review, given that E-2 visas can have a relatively long validity period than most nonimmigrant visas. At this post, all E-2 visa applications are sent to the fraud prevention manager and the ARSO-I, both of whom conduct additional research and look for fraud indicators. Validation study: Validation studies determine the extent to which foreign nationals who were issued visas later overstayed or misused their visa, and can be conducted by post officials for any visa classification. One post in our review conducted a validation study that focused on E-2 visas that post had issued to foreign nationals associated with food service companies (e.g., restaurants) to determine how many remained in business and how many E-2 visa holders continued to travel or stay in the United States after the business failed. According to this 2016 validation study, the post had concluded that almost one-quarter of food service companies in its study had failed within about three years, and nearly half of E-2 visa holders for those companies did not depart after the company had failed or continued to travel to the United States on their E-2 visa. According to the post’s fraud team, the study showed that even prospective E-2 visa enterprises that meet the applicable requirements at the time of application can become unqualified over time, and that adjudicators should take long-term viability into account when determining the marginality of a business. The post’s fraud team also stated that other posts may wish to consider standardized follow- ups for approved E-2 enterprises and routine confirmations of vetted E companies as the E-2 visa category continues to grow in popularity. USCIS officials stated they consider E-2 fraud to be a significant issue and take several steps to identify fraud, including fraud referrals, fraud assessment technology, and site visits. First, according to USCIS officials, immigration officers reviewing the E-2 petition look for anomalies and other indicators of fraud and send a fraud referral for any potential fraud cases by forwarding the case to the service center’s fraud detection office. Immigration officers in the fraud detection office then are to conduct further research, such as reviewing open sources (e.g., company website) or may request a site visit to the business. Second, USCIS uses a fraud assessment technology on all petitions to determine if an E-2 company exists and is financially viable. Specifically, the Validation Instrument for Business Enterprises (VIBE) is a technology that helps immigration officers to determine if a business is operating, financially strong and viable, has good credit, and has not been involved in past fraud. According to USCIS officials, VIBE reviews existing business-related information on an enterprise, such as an office supply store account or utility bills, to determine if it is real and operating. Finally, immigration officers may request site visits based on their review of the application or VIBE results. During such site visits, immigration officers visit the business location to determine if the business is performing as stated in the petition and in compliance with the E-2 visa eligibility requirements. The results of the site visit are sent back to the originating location for adjudication. According to USCIS officials, if a larger conspiracy is uncovered, such as fraud involving multiple beneficiaries, the immigration officer may make a referral to U.S. Immigration and Customs Enforcement for further criminal investigation and potential prosecution, but added that this is very rare. USCIS immigration officers made 252 requests for site visits based on VIBE results from fiscal year 2014 through 2018 for E-2s. Of these site visits, USCIS determined there was confirmed fraud for 25 percent (63), as shown in figure 16. Of the 63 confirmed fraud cases, 42 enterprises were not located at the site provided in the petition and 14 enterprises had provided fraudulent documents or otherwise mispresented the facts. For example, in one case, the beneficiary paid a dental laboratory to assign her in a fictitious position of office manager so that she could obtain E-2 status, but the beneficiary had never worked there. In another example, an investor seeking E-2 status in May 2015 submitted a petition based on a discount store that had gone out of business in January 2013. According to USCIS officials, when fraud is confirmed, the immigration officer will deny the petition, review any pending or previously approved petitions from the petitioner, and fraud finding will be entered into VIBE, which affects the applicant’s ability to obtain future immigration benefits, including visa application or petition approvals from the United States government. State consular officers can also request that USCIS conduct site visits to help in its adjudication of E-2 visa applications, but USCIS data indicate that such requests are rare. According to USCIS, the agency received 10 external site visit requests from State from fiscal years 2014 through 2018. Of the 10 requests, USCIS conducted site visits to seven businesses and found one incidence of fraud involving a restaurant. According to State officials, site visits are considered to be resource intensive for the USCIS and can take several weeks or months to complete. The officials added that if a consular officer determines that an applicant is unqualified for the visa, it would not be considered an effective use of the post’s resources to conduct additional investigations or request a U.S.-based site visit from USCIS. Based on the results of the site visits and other factors, USCIS officials stated that they have prioritized E-2 fraud, and initiated a site visit pilot program in February 2018 to better determine the extent to which fraud exists. This pilot program focuses on businesses associated with individuals approved for an E-2 status extension and certain eligibility criteria. According to USCIS officials in July 2019, the most commonly encountered fraud or noncompliance issues thus far have involved enterprises that were not operational, not engaged in any business activities, or were not operating as stated in the petition. USCIS plans to continue the E-2 pilot into fiscal year 2020 and to share the results with State. State’s and USCIS’s respective roles in the E-2 process, along with a current lack of coordination on E-2 anti-fraud efforts, may contribute to the differences in the way the agencies view and prioritize the risks of E-2 fraud. Drawing on the results of its site visit pilot project, USCIS has said it views E-2 fraud as a significant issue and plans to prioritize efforts to combat E-2 fraud moving forward. While State has taken some steps to examine and combat E-2 visa fraud, officials we spoke with at posts and at headquarters told us that E-2 fraud is rare and generally low risk. The E-2 validation study that one post conducted, noted earlier, also provided evidence that E-2 fraud occurred, at least in that business sector from that particular country. While it is possible that additional validation studies across different posts and business sectors would uncover fraud trends, State officials noted that validation studies are resource intensive, and that E-2 visas represent only a small fraction of the total visas they adjudicate each year. Therefore, State officials stated that such studies are likely to be focused on more common visa types, such as tourist and business visitor visas. Although some factors may explain why USCIS and State view the risk of E-2 fraud differently, both agencies encounter foreign nationals seeking the E-2 status in the United States. Officials from both agencies stated that USCIS may be more likely to uncover fraud than State because USCIS processes E-2 status extensions for individuals already in the United States. E-2 principals (i.e., investors, managers, and essential employees) would have had up to 2 years to try to run, manage, or work for their business, with the intention to depart at the conclusion of their authorized period of stay. If they failed, gave up, or ended employment, but still sought an E-2 status extension, any materially false representations made as to their eligibility could be considered fraudulent. Officials from both agencies suggested that State may be adjudicating visas for more new businesses, which may qualify at the time of initial adjudication but could ultimately fail. However, during our observations and file reviews, we found that USCIS also adjudicates petitions for new businesses for beneficiaries seeking to change to E-2 status, and State also adjudicates E-2 visa applications for existing businesses that have previously been associated with E-2 visa holders. Further, neither State nor USCIS collect data that track the number of new businesses seeking E-2 status for their employees. As such, we cannot verify the accuracy of this reason for explaining why or if USCIS is more likely to encounter fraud among individuals seeking E-2 status than State. Both State and USCIS collect information that could potentially be useful to each other’s activities to identify and address E-2 fraud, but the agencies do not have a mechanism for regular coordination on fraud. For example, as previously noted, consular officers adjudicating E-2 visas overseas learn to identify country-based fraud trends as well as trends specific to E-2 visas. USCIS immigration officers can identify similar trends, and the results of USCIS’s site visits may further identify potential fraud trends that would be useful for State consular officers. However, interagency coordination is ad hoc, generally among headquarters officials only, and relatively rare. For example, both State and USCIS officials stated that the main formal mechanism of coordination on all E-2 visa issues is a quarterly teleconference. However, such meetings were cancelled 7 out of 8 times in fiscal years 2017 and 2018 because officials did not identify agenda topics to discuss, according to State and USCIS officials. Further, such meetings have not included discussions of E-2 fraud issues. State officials stated that they share country fraud summaries with USCIS. However, these fraud summaries do not focus on E-2 visas, but fraud trends more generally. According to A Framework for Managing Fraud Risks in Federal Programs, agencies should establish collaborative relationships with stakeholders to share information on fraud risks and emerging fraud schemes, as well as lessons learned related to fraud control activities. Managers can collaborate and communicate through a variety of means, including task forces, working groups, or communities of practice. Although State and USCIS have some informal mechanisms in place to share fraud-related information, such as emails among headquarters officials and by sharing high-level country fraud reports, formal information sharing mechanisms have not been regularly operating. Although the two entities view the risk of E-2 fraud visa differently, both State’s and USCIS’ E-2 antifraud efforts would benefit from ensuring that they regularly share information on fraud risks. Doing so will help both entities to better identify emerging fraud trends, prevent foreign nationals from fraudulently obtaining E-2 status, and identify areas for potential collaboration and resource sharing. The E-2 nonimmigrant classification helps to facilitate foreign investment in the United States, which contributes to the U.S. economy each year. State and USCIS share the responsibility for adjudicating thousands of E- 2 visa applications and petitions annually for foreign nationals seeking E- 2 status. Both State and USCIS officials stated that given the complexity of adjudicating E-2 applications and petitions, and the level of documentation and time required, the E-2 adjudication process can present challenges with respect to the analysis of E-2 eligibility requirements. State consular officers, managers, and LES noted that additional training and resources are needed to help them better understand the eligibility requirements and supporting financial and business documents. Enhancing E-2 training and providing additional resources such as documents clarifying E-2 eligibility requirements would help better ensure that consular officers and LES prescreening and adjudicating these visas have the necessary knowledge, skills, and abilities to carry out their responsibilities effectively across posts worldwide. Additionally, some overseas State posts have developed E-2 company registration programs to more efficiently process and adjudicate E-2 visa applications. Although there are benefits to such programs, the variation in the standards of these programs may result in different processing of companies and applicants across posts, as well as acceptance of varying levels of risk by posts. Establishing guidance or minimum standards for posts that choose to implement such programs would better ensure that all posts’ E-2 visa adjudication processes are consistent with State’s policies, objectives, and risk tolerance. Further, State and USCIS require certain information and documents be retained for all E-2 applications and petitions; however, during our file review of State and USCIS E-2 adjudications, we identified that some required documents were missing from State files. Ensuring that posts retain all required E-2 documentation would better position State to be able access applicant information, which could be needed for law enforcement, anti-fraud, or security purposes later. Finally, although State and USCIS collect information that could potentially be useful to each other’s activities to address E-2 fraud, coordination between State and USCIS on E-2 fraud has been ad hoc, generally among headquarters officials only, and relatively rare. Developing regular coordination mechanisms would help both entities to better identify emerging fraud trends and prevent foreign nationals from fraudulently obtaining E-2 status. We are making the following five recommendations to State and USCIS: The Assistant Secretary of State for Consular Affairs should provide additional training or related resources to consular officers and locally employed staff on adjudicating E-2 visas, to cover topics that include the E-2 eligibility requirements and understanding business- and tax- related documents. (Recommendation 1) The Assistant Secretary of State for Consular Affairs should develop minimum standards for E-2 company registration programs, such as standards for how often companies are to be re-vetted. (Recommendation 2) The Assistant Secretary of State for Consular Affairs should develop and implement a process to ensure that posts maintain required E-2 visa application documentation. (Recommendation 3) The Secretary of State, in coordination with the Director of USCIS, should establish regular coordination mechanisms to share information on E-2 fraud risks. (Recommendation 4) The Director of USCIS, in coordination with the Secretary of State, should establish regular coordination mechanisms to share information on E-2 fraud risks. (Recommendation 5) We provided a draft of this report to State and DHS for their review and comment. State and DHS provided written comments, which are reproduced in appendices IV and V, respectively. Both State and DHS concurred with our recommendations. State and DHS also provided technical comments, which we incorporated as appropriate. State concurred with all four recommendations addressed to it in the report (recommendations 1, 2, 3, and 4), and described actions it plans to take in response. To address recommendation 1, State plans to increase the frequency and specificity of E-2 content through webinars, workshops, and guidance, and by developing subject matter experts domestically who can provide consultative services on an as-needed basis for business and tax-related documents. To address recommendation 2, State plans to require a minimum 5-year mandatory review of companies registered at any post using a company registration program. To address recommendation 3, State plans to reinforce its E-2 visa documentation retention policy in regular policy guidance to consular managers. To address recommendation 4, State plans to hold regular, high-level coordination meetings with USCIS to include coordination on E visa adjudication standards. DHS concurred with recommendation 5, and stated that the department plans to share the results of its site visits during quarterly coordination meetings with State. These actions, if effectively implemented, should address the intent of our recommendations. We are sending copies of the report to the Acting Secretary of Homeland Security, Secretary of State, and appropriate congressional committees. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Rebecca Gambler at (202) 512-8777 or gamblerr@gao.gov or Jason Bair at (202) 512-6881 or bairj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. This report reviews the Department of State’s (State) and Homeland Security’s (DHS) U.S. Citizenship and Immigration Services’ (USCIS) oversight and implementation of E-2 adjudications. Specifically, this report examines (1) the outcomes and characteristics of foreign nationals who have sought or received E-2 status during fiscal years 2014 through 2018, (2) State’s and USCIS’s policies and procedures to ensure that individuals meet E-2 eligibility requirements, and (3) State’s and USCIS’s efforts to assess and address potential fraud in the E-2 adjudication process. To determine the outcomes and characteristics of foreign nationals who have sought or received E-2 status, we analyzed data from State’s Bureau of Consular Affairs and USCIS on E-2 visa applications and petitions adjudicated from fiscal years 2014 through 2018. For example, the data we analyzed included E-2 role (e.g., investor, manager, essential employee, and dependents), adjudication outcome (i.e., issued or refused), and nationality, among other data points. To assess the reliability of the E-2 data, we interviewed State and USCIS officials that maintain the data and checked the data for missing information, outliers, and obvious errors, among other actions. For example, we identified and removed duplicate entries in State’s data. On the basis of these steps, we determined that the data were sufficiently reliable for the purposes of our reporting objectives, including providing summary statistics on E-2 adjudications, outcomes, and the characteristics of those seeking E-2 status. To obtain additional data points, such as types of business and investment amount, we analyzed generalizable stratified random samples of E-2 visa applications and petitions adjudicated in fiscal year 2018. Specifically, we reviewed 124 E-2 petitions from USCIS and 120 State applications for E-2 investors, managers, and essential employees. The documents in our file review included, for example, State’s DS-160 online nonimmigrant visa application and DS-156E supplemental application, USCIS’s I-129 petition for nonimmigrant workers, and supporting documents, when available. To collect information from the applications and petitions, we created a data collection instrument and established standard procedures to ensure that we accurately collected the information from the original forms. We chose sample sizes to achieve precision levels for a percentage estimate of plus or minus 10 percentage points for important sub-populations, such as denied petitions and role (e.g., investor, manager, and essential employee). As a result, all percentage estimates presented in this report have a precision of plus or minus 10 percentage points or fewer, unless otherwise noted. Further, we classified the types of businesses in the applications and petitions using the North American Industry Classification System by conducting a content analysis of the business description field in the applications and petitions to group related business types into larger groups, such a food service and manufacturing. Further, we also collected and analyzed data and information from USCIS and U.S. Customs and Border Protection on post E-2 adjudication outcomes, including changing status from E-2 to another nonimmigrant category, adjusting from E-2 status to lawful permanent residency, and E- 2 nonimmigrants who remain in the United States beyond the expiration of their authorized period of stay, known as overstays. We present the results of this analysis in Appendix III. To assess the reliability of these data, we interviewed officials that maintain the data and checked the data for missing information, outliers, and obvious errors, among other actions. On the basis of these steps, we determined that the data were sufficiently reliable for the purpose of providing summary statistics on E-2 post adjudication outcomes. To assess State and USCIS policies and procedures to ensure that individuals meet E-2 eligibility requirements, we reviewed relevant State and USCIS guidance documentation, including State’s Foreign Affairs Manual and USCIS’s E-2 standard operating procedures. We also reviewed relevant provisions of the Immigration and Nationality Act and implementing regulations, which set forth the E-2 eligibility requirements. We interviewed officials from State’s Bureau of Consular Affairs and Foreign Service Institute, and USCIS on their respective agencies’ E-2 processes and procedures, as well as training provided to State’s consular officers and USCIS’s immigration officers. Further, we assessed State’s and USCIS’s policies and procedures to ensure that individuals meet E-2 eligibility requirements against control environment, control activities, and monitoring internal control standards in Standards for Internal Control in the Federal Government, as well as documentation retention requirements in agency guidance. We conducted site visits to State and USCIS locations that adjudicate E-2 visas and petitions, respectively. For State, we conducted site visits to four posts abroad—London, United Kingdom; Seoul, South Korea; Tokyo, Japan; and Toronto, Canada from October through December 2018. For our site visits, we selected posts that (1) were among the 10 highest E-2 adjudicating posts by volume in fiscal year 2017, (2) had different staffing models for processing E-2 visa adjudications, such as posts that had a single officer specializing in E-2 visas or posts that had all consular officers adjudicate E-2 visas, and (3) were geographically dispersed. During these visits, we observed the prescreening and adjudication of E-2 applications and used a data collection instrument to collect information on the cases we observed, such as adjudication outcome and other non- personally identifiable information about the case. We interviewed consular officers and managers, locally employed staff (LES), fraud prevention managers, and the assistant regional security officer- investigators (ARSO-I), where available, about topics such as E-2 visa adjudication policies, procedures, resources and training available at post. Our observations from these site visits provided useful insights into State’s E-2 adjudication procedures, but are not generalizable to all posts that adjudicate E-2 visas. For USCIS, in November 2018, we visited the California Service Center in Laguna Niguel, California—which is the only USCIS service center that adjudicates E-2 petitions—to observe E-2 petition adjudications and interview USCIS officials. In addition to our site visits, we conducted telephonic interviews with consular officers and LES who are responsible for prescreening and adjudicating E-2 visa applications at the remaining six of the top 10 posts in terms of E-2 annual adjudications, as well as four randomly selected low-volume posts. The 4 low-volume posts were selected at random from a list of posts that had adjudicated at least 100 E-2 visa applications in fiscal year 2017. We collected copies of post-specific standard operating procedures and local E-2 visa adjudication tools (e.g., checklists), as available, from the 14 posts we visited or interviewed. Further, we reviewed written responses from the consular managers responsible for supervising E-2 visa adjudications at these 14 posts to a set of questions regarding E-2 adjudication processes and procedures, challenges, E-2 company registration programs, and E-2 training. To determine the efforts that State and USCIS take to assess and address E-2 fraud, we reviewed relevant State and USCIS standard operating procedures and guidance. We interviewed headquarters officials from State and USCIS, such as State’s Office of Fraud Prevention Program and USCIS’s Fraud Detection and National Security Directorate, on how both agencies identify and address potential E-2 fraud and what, if any, coordination or information sharing occur between State and USCIS. During our 4 site visits abroad, we interviewed officials, such as fraud prevention managers and ARSO-Is, on anti-fraud efforts for E-2 visas at their posts, including potential fraud trends. Similarly, we interviewed immigration officers at USCIS’s California Service Center on their anti-fraud efforts for E-2 petitions. We obtained data from State and USCIS on fraud referrals—that is, cases sent to fraud experts for additional research and review—and the results of fraud site visits from fiscal year 2014 through 2018. To assess the reliability of these data, we interviewed State and USCIS officials that maintain the data and checked the data for missing information, outliers, and obvious errors, among other actions. On the basis of these steps, we determined that the data were sufficiently reliable for the purposes of our reporting objectives, including providing summary statistics on fraud referrals and the results of fraud site visits. Further, we assessed State’s and USCIS’s anti-fraud efforts against best practices found in A Framework for Managing Fraud Risks in Federal Programs. We conducted this performance audit from July 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Immigration and Nationality Act requires the existence of a qualifying treaty of commerce and navigation between the United States and a foreign state in order for E-2 visa classification to be accorded to nationals of that foreign state. According to Department of State guidance, such qualifying treaties may include treaties of friendship, commerce and navigation, and bilateral investment treaties. As of June 2019, nationals of the 82 countries listed in Table 7 may be accorded E-2 status pursuant to a qualifying treaty, or pursuant to legislation enacted to extend that same privilege. This appendix presents various statistics on adjudications by State for E-2 visas as well as those by U.S. Citizenship and Immigration Services (USCIS) for E-2 petitions for fiscal years 2014 through 2018. We present these data broken out by fiscal year, outcome (e.g., issued or refused), type (e.g., investor, manager, essential employee, dependent), country of nationality or birth, reason for refusal, and prior nonimmigrant status, if available. Further, we also provide statistics on some post-adjudication outcomes—that is, data on characteristics of those who obtained E-2 status. These outcomes include changes to another nonimmigrant status or lawful permanent residency, or the extent to which E-2 status holders remained in the United States beyond their authorized period of stay, known as overstaying. For the purposes of this appendix, there are four potential roles for foreign nationals seeking E-2 status. First, a foreign national who has committed funds to a U.S. enterprise and is in a position to develop and direct the operations of the enterprise in which he or she has invested substantial capital is known as an investor. Second, a foreign national employee in an executive or supervisory position is known as a manager. Third, a foreign national employee, in a lesser capacity than a manager, but having special qualifications essential to successful or efficient business operations, is known as an essential employee. Finally, the spouse or qualifying child of an investor, manager, or essential employee is known as a dependent. State consular officers will adjudicate the visa application as either issued or refused. A foreign national seeking E-2 status as an investor, manager, or essential employee is known as a principal, and a spouse or qualifying child of a principal is known as a dependent. Foreign nationals seeking E- 2 status through USCIS use different forms based on whether they are a principal or a dependent. USCIS immigration officers will generally adjudicate the petition as either approved or denied. Change of Status From E-2 to Another Nonimmigrant Category. From fiscal years 2014 through 2018, about 5,000 foreign nationals sought to change from E-2 status to another nonimmigrant status. As shown in figure 17 and table 16, most of these requests were to change to academic student status (F-1, 31 percent), temporary workers in specialty occupation status (H-1B, 10 percent), tourist status (B-2, 9 percent), and intracompany transferee executive or manager status (L-1A, 7 percent), as well as dependents of these statuses. Further, about 11 percent of these foreign nationals were requesting to change from one role within E- 2 status to another. As previously noted, this could include, for example, a spouse of an E-2 investor later seeking to work at the company as a manager. Adjusting from E-2 Status to Lawful Permanent Resident. From fiscal years 2014 through 2018, over 22,000 foreign nationals changed from E- 2 status to lawful permanent residents. The large majority of these (73.1 percent) were employment-based (i.e., sponsored by a U.S. employer), as shown in figure 18 and table 17. Overstays. According to DHS data, a relatively low percentage of foreign nationals with E-2 status—obtained either through an E-2 visa from State or an approval to change to, or extend, their E-2 status from USCIS— overstayed their authorized period of admission compared to other nonimmigrant statuses. From fiscal years 2016 through 2018, DHS reported that the total overstay rate decreased slightly from 1.5 percent to 1.2 percent. Similarly, the overstay rate for E-2 status for the same years decreased from 0.8 percent from 0.6 percent, as shown in table 18. As we previously reported, U.S. Customs and Border Protection (CBP) implemented system changes in 2015 that allowed CBP to identify E-2 overstays, along with other nonimmigrant categories beginning in fiscal year 2016. DHS officials stated that the process to track E-2 visa overstays is the same as with other visa categories. They noted that specific visa categories are not prioritized; CBP and U.S. Immigration and Customs Enforcement focus on those overstays where the individual is identified as a national security or public safety risk. In addition to the individuals named above, Adam Hoffman (Assistant Director), Kim Frankena (Assistant Director), Erin O’Brien (Analyst-in- Charge), Juan Pablo Avila-Tournut, Kristen E. Farole, James Ashley, Caitlin Cusati, Eric Hauswirth, Amanda Miller, Sasan J. “Jon” Najmi, Adam Vogt, and K. Nicole Willems made significant contributions to this report.", "summary": "Foreign nationals from 82 countries may obtain E-2 nonimmigrant investor status in the United States. The E-2 nonimmigrant classification allows an eligible foreign national to be temporarily admitted to the United States to direct the operations of a business in which they have invested a substantial amount of capital, or to work in an approved position (e.g., manager or essential employee). To obtain E-2 status, a foreign national can apply through State for an E-2 visa abroad, or if already in the United States, by petitioning USCIS to extend or change to E-2 status. GAO was asked to review State's and USCIS' E-2 adjudication process. This report addresses: (1) outcomes and characteristics of foreign nationals who sought or received E-2 status from fiscal years 2014 through 2018, (2) policies and procedures for ensuring that individuals meet E-2 eligibility requirements, and (3) efforts to assess and address potential E-2 fraud. GAO analyzed State and USCIS data on E-2 adjudications, generalizable samples of E-2 visa applications and petitions, and relevant documents. GAO interviewed officials at 14 State posts abroad, selected based on E-2 application volume and other factors, and observed E-2 adjudications at four of these posts and USCIS's California Service Center. The Department of State (State) and U.S. Citizenship and Immigration Services (USCIS) annually adjudicated about 54,000 visa applications or petitions from fiscal years 2014 through 2018 for foreign nationals seeking E-2 nonimmigrant status, over 80 percent of which were approved. About eighty percent of E-2 adjudications were for State visa applications, and the remaining 20 percent were for USCIS petitions to extend or change to E-2 status. Generally, about half of the foreign nationals seeking E-2 status were investors, managers, or essential employees of an E-2 business, and the other half were their spouses or children. State and USCIS have guidance, procedures, and training intended to help consular and immigration officers ensure foreign nationals meet E-2 eligibility requirements; however, officials GAO interviewed from both agencies identified challenges in the E-2 adjudication process. State. Consular officers noted that E-2 visa adjudications are complicated and resource-intensive, often requiring more documentation and time to complete than other visas. For example, the requirement that the investment in the business be substantial does not prescribe a minimum capital amount. Rather, the investment must be large enough to support the likely success of the business, among other criteria. Consular officers at 10 of 14 posts GAO interviewed indicated that determining the investment's substantiality is difficult for newly encountered business types. Providing additional E-2 training or related resources would help ensure that consular officers and locally employed staff have the necessary knowledge and abilities to carry out their responsibilities. USCIS. Officials identified similar challenges with respect to E-2 adjudications. However, officials stated that colocating immigration officers who adjudicate E-2 petitions helps to mitigate the challenges because the officers can communicate with each other on how USCIS has typically adjudicated such cases. State and USCIS have resources to address E-2 fraud, which includes submitting falsified documents or making false statements material to the adjudication; however, coordination on E-2 anti-fraud efforts is limited. State has anti-fraud efforts in place for all nonimmigrant visa types, but State officials stated that they consider E-2 visa fraud to be lower risk compared to other visas because the large amount of complex paperwork required for the E-2 visa discourages malicious actors. USCIS officials consider E-2 fraud to be a significant issue and have taken steps to identify fraud, such as using fraud assessment technology to determine if a business is financially viable and conducting site visits if fraud is suspected. Both State and USCIS collect information that could be useful to each other's anti-fraud efforts, but interagency coordination on E-2 fraud issues is ad hoc and relatively rare. For example, the main formal mechanism of coordination on E-2 visa issues—a quarterly teleconference—was cancelled 7 out of 8 times in fiscal years 2017 and 2018. Coordinating regularly on fraud issues, which is a best practice from GAO's Fraud Risk Framework, will help both entities to better identify emerging E-2 fraud trends and areas for potential resource sharing. GAO is making five recommendations, including that State provide more E-2 training or resources to consular officers, and that State and USCIS establish a regular coordination mechanism to share information on E-2 fraud risks. State and USCIS concurred with all five recommendations.", "document_type": "gao"}
{"report": "For the T-2017 contracts, DHA consolidated its TRICARE regions from three regions (North, South, and West) to two regions (East and West). Humana Government Business is the managed care support contractor for the East Region, and Health Net Federal Services is the managed care support contractor for the West Region. Health care delivery under the T-2017 contracts began on January 1, 2018. DHA expects the costs of the two contracts to total approximately $58 billion over a 5-year performance period, which is scheduled to end on December 31, 2022. The primary responsibilities of the managed care support contractors include the following: developing civilian provider networks, which include hospitals and processing referrals and authorizations for beneficiaries to receive processing health care claims; providing comprehensive, readily accessible customer services for beneficiaries and providers; and establishing and maintaining a medical management program that includes requirements in the TRICARE Operations Manual. In addition, DHA officials told us that they have begun their planning activities for the fifth generation of TRICARE contracts, referred to as the T-5 contracts. If DHA exercises all option years for the T-2017 contracts, health care delivery under the T-5 contracts is expected to begin in 2023. DHA’s acquisition process for the T-2017 contracts consisted of four steps: (1) planning the acquisition, (2) issuing the Request for Proposals (RFP) and soliciting responses, (3) awarding the contracts, and (4) post award activities (see figure 1). 1. Acquisition planning. DHA defined the contract requirements—the work to be performed by the contractor—and developed an acquisition plan to meet those requirements. The T-2017 program manager and contracting officer developed key acquisition documents—including the T-2017 Acquisition Strategy and the Acquisition Plan—and conducted market research. The T-2017 Acquisition Strategy provides a high-level description of the milestones in the acquisition process and how those milestones will be achieved. The T-2017 Acquisition Plan outlines the specific actions necessary to execute the approach outlined in the approved acquisition strategy. 2. Request for proposals. DHA issued an RFP that documented the requirements for T-2017—including the contract type, significant contract dates, pricing arrangements, and the criteria to be used to assess offerors’ proposals. 3. Award. DHA established a source selection team to evaluate the proposals received in response to the RFP. The source selection authority selects the winning proposals using a best value tradeoff process after considering reports written by other members of the source selection team. 4. Post-Award Activities. DHA provides a 12-month transition period between its outgoing and incoming contractors to ensure that its incoming contractors are prepared for their new responsibilities. The transition period for the T-2017 contracts began on January 1, 2017, and ended on December 31, 2017. The incoming contractors assumed full responsibility for health care delivery on January 1, 2018. The NDAA 2017 required a number of changes to the TRICARE program through its contracts. Specifically, section 705(a) of the NDAA 2017 required DOD to develop and implement value-based incentive programs in its contracts to help improve the quality of health care services provided to eligible TRICARE beneficiaries by rewarding civilian providers with additional payments for improved performance based on certain metrics. In addition, section 705(c) of the NDAA 2017 directed the department to develop and implement a strategy—by January 1, 2018— for its TRICARE contracts that includes 13 specific elements, such as telehealth services and beneficiary referrals, among others (see table 1). The act required DOD to modify its TRICARE contracts to ensure consistency with the required strategy providing for the 13 elements. DHA made selective changes between the T-3 and T-2017 contracts and acquisition strategy. According to DHA officials, the contracts are generally the same, and changes were made to clarify or streamline TRICARE requirements and administrative processes. The T-2017 Acquisition Strategy states that the T-2017 performance work statement, which identifies the TRICARE requirements to be implemented by the contractors, is essentially unchanged from the T-3 contracts. DHA officials explained that their leadership prioritized the continuation of beneficiary services during the T-2017 planning process over making significant changes to contract requirements that could potentially be disruptive. We found that some of the changes that were made to the T- 2017 contracts are consistent with specific provisions and themes we identified in prior NDAA legislation. Although the NDAA 2017 was enacted after the T-2017 contracts had been awarded, some of the contract changes for T-2017 may be consistent with specific provisions outlined in section 705(c)(1), such as provisions related to improving access to care, health outcomes, health care quality, beneficiaries’ experience, as well as lowering health care costs. However, DHA officials stated that because health care delivery under the T-2017 contracts began in 2018, it is too early to measure any benefits from these changes. These contract changes include (1) the consolidation of contract regions, (2) the combining of administrative costs, and (3) the introduction of new contract incentives. 1. Consolidation of contract regions. While DHA awarded the T-3 contracts for three regions (West, South, and North), it consolidated two of the regions (North and South) for the T-2017 contracts (see figure 2). By eliminating the additional regional contract, DHA anticipates a savings of approximately $25 million a year in overhead and management costs. In addition, beneficiaries are less likely to have a disruption in care when moving. For example, beneficiaries who moved between the former North and South regions would now stay enrolled with the same contractor in the larger East region. 2. Combined administrative costs. For T-2017, DHA combined all administrative costs in one contract line item in order to lower total cost of care. For example, under the T-3 contracts, DHA reimbursed the contractors for processing individual claims with a higher rate for paper claims and a lower rate for electronic claims. Without a difference in costs for T-2017, contractors are incentivized to lower their costs and prioritize electronic claims, which DHA officials say are more efficient. 3. Contract incentives. DHA incorporated incentives into the T-2017 contracts to encourage contractors to negotiate reimbursement rate discounts with network providers in order to reduce health care costs. The T-2017 contracts state that the contractor must meet a required discount rate on care provided by network providers. If this discount rate is not met, DOD will offset the discount deficit amount from the next payment due to the contractor. DHA expects that negative incentive will reduce health care costs and result in government savings. We also found examples of changes to the contract or acquisition process for T-2017 that are consistent with selected acquisition themes we identified in prior NDAA legislation. These acquisition themes are 1) leveraging commercial best practices, 2) promoting competition, and 3) focusing on value. We previously reported that the identified acquisition themes can reduce costs and increase value for the government. 1. Leveraging commercial best practices: T-2017 required contractors to increase utilization of commercial best practices, including the use of automation technology to process referrals and authorizations, episodes of care, and procedure diagnosis coding. As we have previously reported, federal agencies can leverage commercial best practices to lower costs and maximize the value of the services they buy. According to DHA officials, adapting automation technology already in use in the health care industry should improve the quality of services, beneficiary satisfaction, and result in cost savings to the government. In addition, officials from one of the current TRICARE contractors stated that the T-2017 RFP was structured to incentivize contractors to innovate and bring best practices from their industry experience in both the commercial sector and other government programs, such as Medicare. For example, the T-2017 contract included a new requirement for contractors to use industry best practices when collecting health care data, in order to identify and reduce gaps in care and enhance quality of care for beneficiaries. 2. Promoting competition: We found that DHA made an effort to promote competition for the T-2017 RFP. Competitive contracts can result in cost savings for the federal government and promote accountability for results. In the acquisition planning phase, DHA identified an increased number of interested contractors through market research, from eight for T-3 to 22 for T-2017. In addition, DHA officials stated that they took steps during the acquisition planning process to ensure that the incumbent contractors did not have a significant advantage over prospective contractors. For example, the T-2017 contracting officer was assigned early in the planning process and did not participate in management of the T-3 contracts or in interactions with the incumbent contractors. DHA officials stated that they expected greater contractor interest in the East region because the larger beneficiary population of that region would result in a more valuable contract. However, DHA received a total of seven proposals each for T-3 and T-2017, including one new company that participated in T-2017 but had not previously submitted a proposal for T-3. 3. Focusing on value: We found that DHA’s approach for T-2017 emphasizes value and quality, not just lower costs. Specifically, DHA focused on the value of improving health care by considering the total cost of care over time, rather than the cost of individual health care. This is reflected in the T-2017 Acquisition Strategy, which prioritizes quality and delivery of health care above lowest cost. For example, the T-2017 contracts required additional preventive screenings and diseases covered under a chronic care program to achieve improved quality of care despite the cost of screenings. We have previously reported that these preventive health services are determined to be cost-effective when they improve the benefit (e.g., health outcomes) in a less costly way than a given alternative care option. Some preventive services may also result in cost savings, where the cost of implementing the service is less than the expected future costs to treat a disease or condition. As of October 2019, DHA had implemented two of the three value-based pilots described in its January 2018 report to Congress, which outlines the department’s plans for addressing the NDAA 2017’s requirement for developing value-based incentive programs. Specifically, in this report, DHA described its intent to implement three value-based pilots in response to section 705(a) of the NDAA 2017—(1) the Performance- Based Maternity Payments Pilot, (2) the Medication Adherence Pilot, and (3) the High-Value Primary Care Provider Pilot—through modifications to its TRICARE contracts over the next 6 to 18 months. 1. Performance-Based Maternity Payments Pilot. DHA modified its T- 2017 contracts to begin implementing the Performance-Based Maternity Payments Pilot in April 2018. This pilot was designed to provide both non-financial and financial incentives to hospitals that achieve and maintain excellence in maternity care quality. The first phase of this pilot focused on non-financial incentives by promoting greater transparency about the quality of maternity care delivered by hospitals in the TRICARE network. Specifically, DHA implemented a “steerage model” that identifies higher-performing hospitals in the managed care support contractors’ provider directories using specific visual prompts in order to encourage beneficiaries to seek care from those institutions. The second phase of the pilot began in October 2018 and incorporated performance-based payments, or financial incentives, for network hospitals that achieve a certain level of performance on specified maternity care quality measures. The anticipated end date for the pilot is March 2021. 2. Medication Adherence Pilot. DHA modified its TRICARE pharmacy contracts to begin implementing the Medication Adherence Pilot in February 2018. This pilot is designed to incentivize beneficiaries’ adherence to medication regimens by reducing or eliminating copayments for two medications (one for diabetes and another for cardiovascular-related illnesses). 3. High-Value Primary Care Provider Pilot. As of October 2019, DHA officials told us they were still assessing the feasibility of implementing the High-Value Primary Care Provider Pilot, which would provide financial incentives (such as additional payments or reduced network discounts) to primary care providers who exceed certain quality thresholds, as well as financial incentives (such as reduced co-shares and copayments) for beneficiaries who use these providers. DHA officials said other value-based efforts are being planned to address section 705(a) of the NDAA 2017, such as value-based pilots and demonstrations that aim to incentivize providers to provide quality care— including hospital, home health, and episode-based bundled payments pilots, among others. DHA has reported that these projects will offer DHA the opportunity to test value-based payment models and incorporate innovative ideas and solutions into its TRICARE contracts. As of January 2020, we found that DOD had partially implemented six of the 13 elements required by sections 705(c)(5) and (c)(6) in the NDAA 2017, in its T-2017 contracts. DHA leadership explained that they had decided that the department would separately address each of the 13 elements through modifications to the TRICARE contracts rather than developing a single strategy that would address all of the elements. According to DHA officials, some of the 13 elements would be implemented through modifications to the T-2017 contracts while other elements would be addressed in the T-5 contracts as certain elements would require more time to develop. Section 705(c)(5): This section includes nine elements that focus on various aspects of health care delivery. We found that DHA had partially implemented six of the nine elements—including provider networks, medical management, telehealth services, beneficiary enrollment, value-based methodologies, and prevention and wellness incentives (see table 2). Although DHA officials generally described their approach for addressing the three other elements, they were not able to provide documentation, such as implementation plans, with specific time frames or actions needed to fully implement each of them. Specifically, when asked about time frames for complete implementation, DHA officials told us that many of the elements should be addressed through the T-5 contracts. DHA officials also told us the department’s approach to addressing these elements—such as provider networks—will be informed by ongoing and future value- based pilots and demonstrations; however, data from these pilots and demonstrations are not expected to be available until they have concluded. Section 705(c)(6): This section included four required elements that focus on the delivery of health care in rural, remote, and isolated areas. DHA has not implemented any of these requirements. DHA officials told us they are considering requirements for T-5 that will address the four elements, but did not provide documentation with specific time frames and actions needed to fully implement each of them (see table 3). Without plans that include specific time frames and actions needed, it is unclear exactly how and when DHA will fully implement all 13 elements into its TRICARE contracts. As we have previously reported, sound planning calls for results-oriented organizations to develop plans that (1) provide tools to ensure accountability, such as time frames, and (2) identify specific activities to obtain desired results, among other things. Developing and implementing plans with time frames and actions needed can help to ensure that DHA fully implements all 13 required elements, which is particularly important since it is in the process of developing its T-5 contracts. The NDAA 2017 required DHA to make numerous changes to its TRICARE program—some of which impact its T-2017 managed care support contracts. In particular, the act required DHA to modify these contracts to ensure consistency with 13 specific elements related to improving health care delivery, such as with provider network flexibility, increased use of telehealth services, and prevention and wellness incentives, among others. While DHA has taken steps to begin implementing some of these elements in its current T-2017 contracts, it has not developed implementation plans with time frames and specific actions needed to guide its efforts, which could help ensure that DHA successfully implements all of the required elements. Until these elements are fully implemented, the department may not achieve the TRICARE program improvements Congress intended related to access to care, health outcomes, quality of care, beneficiaries’ experience, and cost efficiency. We are making the following recommendation to DHA: The Director of DHA should develop and implement plans with timeframes and specific actions needed for all 13 required elements to be reflected in the TRICARE contracts. (Recommendation 1) We provided a draft of this report to DOD for comment. In its written comments, reproduced in appendix I, DOD generally agreed with our findings and concurred with our recommendation. The department reiterated its plans to address each of the elements required by sections 705(c)(5) and (c)(6) in the NDAA 2017 as part of its T-5 contracts. DOD also provided technical comments, which we incorporated as appropriate. In addition, DOD provided updated information on the status of its efforts to address certain elements required by section 705(c)(5). As a result of this information, we updated the status of the following two elements from “not implemented” to “partially implemented” in our overall assessment for the following reasons: 1) Provider Networks: The department provided evidence that the Accountable Care Organization demonstration was implemented on January 1, 2020, and that beneficiaries were enrolled in the program. 2) Medical Management: The department provided evidence that it awarded a contract for the TRICARE Select Patient Navigator Pilot on December 27, 2019, and that the contractor began work on January 1, 2020. The department also provided updates on the status of two additional elements—Financial Incentives and Medical and Lifestyle Incentives. However, while we updated the department’s plans for these elements in the report, we determined that their status should remain “not implemented” in our overall assessment for the following reasons: 1) Financial Incentives: The department provided evidence that it plans to provide financial incentives to Kaiser Permanente providers on an annual basis under the Accountable Care Organization demonstration. These incentives are expected to begin in 2021. 2) Medical and Lifestyle Incentives: According to department officials, these incentives for beneficiaries may be provided by Kaiser Permanente on an annual basis under the Accountable Care Organization demonstration, at no cost to the government. These officials told us they were unsure whether and how such incentives may be more broadly applied to the TRICARE program. We are sending copies of this report to the Department of Defense, appropriate congressional committees, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Sharon Silas, Director, Health Care at (202) 512-7114 or silass@gao.gov or William T. Woods, Director, Contracting and National Security Acquisitions at (202) 512-4841 or woodsw@gao.gov. Contact points for our Office of Congressional Relations and Office of Public Affairs can be found on the last page of this report. Other major contributors to this report are listed in appendix II. In addition to the contact named above, Bonnie Anderson, Assistant Director; La Sherri Bush, Analyst-in-Charge; LaKendra Beard, Jacquelyn Hamilton, Jessica Karnis, Miranda Riemer, and Lauren Wright made contributions to this report. Also contributing were Sam Amrhein and Vikki Porter.", "summary": "In fiscal year 2018, DOD provided health care services to more than 9 million eligible beneficiaries through TRICARE, its regionally structured health care program. In each of its two regions (East and West), DOD uses contractors to manage health care delivery through civilian providers. The NDAA 2017 required a number of changes to the TRICARE program through its contracts. Specifically, it required DOD to implement a strategy with 13 specific elements—related to provider networks, telehealth services, and referrals, among other areas—for its contracts. The NDAA 2017 and the accompanying Senate Report 114-255 included provisions for GAO to examine DOD's managed care support contract acquisition process and requirements. This report (1) describes changes DOD made to its TRICARE contracts and acquisition process between its T-3 and T-2017 contracts and (2) examines the extent to which DOD implemented the 13 elements as required by the NDAA 2017, among other things. GAO reviewed and analyzed relevant federal statutes, T-3 and T-2017 planning and contracting documents, and interviewed DOD officials and TRICARE contractors. The Department of Defense (DOD) made selective changes to its TRICARE managed care support contracts and acquisition process from the third generation of contracts (T-3) to the fourth generation (T-2017) of contracts. According to DOD officials, the contracts are generally the same, and changes were made to clarify or streamline TRICARE requirements and administrative processes. Officials told GAO they prioritized the continuation of beneficiary services, rather than implement significant contract changes that could potentially be disruptive. Some of the T-2017 changes include a reduction from three to two contract regions and a different method for paying the contractors. GAO found that DOD has partially implemented six of the 13 elements required by the National Defense Authorization Act for Fiscal Year 2017 (NDAA 2017), in its T-2017 contracts. DOD leadership explained that they decided to implement each of the 13 elements separately rather than by developing a single strategy that addressed all of the elements. DOD officials explained that some of the 13 elements will be implemented through modifications to the T-2017 contracts, while others will be addressed in the fifth generation of managed care support contracts (T-5), which are expected to be awarded in 2021. While DOD has taken steps to begin implementing some of the required elements, GAO found that DOD lacks plans with specific time frames and actions needed to fully implement all of the elements. As a result, it is unclear exactly how and when all 13 elements will be implemented. GAO recommends that DOD develop and implement plans with time frames and specific actions needed for all 13 required elements in the TRICARE contracts. DOD concurred with GAO's recommendation and noted its plans to address each of the required elements in the T-5 contracts.", "document_type": "gao"}
{"report": "Historically, unmanned aircraft have been known by many names including: “drones,” remotely piloted vehicles, unmanned aerial vehicles, and models. Today, the term UAS is generally used to emphasize the fact that separate system components are required to support airborne operations without a pilot onboard the aircraft. Recreational users have flown UAS—largely model aircraft—for years with minimal FAA interaction. Increasingly though, more technically advanced UAS are being used in a variety of ways by different types of users. Certain industries are interested in expanding the allowable uses for UAS, such as expanding use of UAS in controlled airspace. Expanding allowable uses would likely require more FAA involvement and regulatory action. UAS operators generally fall into the following categories: Recreational users operate UAS primarily for recreational or educational purposes, such as operating UAS to take photographs or video for personal use. To operate UAS recreationally, a user must obtain a certificate of registration from the FAA. The certification constitutes registration for all unmanned aircraft owned by the individual and operated recreationally. Commercial users operate UAS in connection with a business. Examples of commercial uses include: selling photos or videos taken from UAS (such as wedding or real estate photography); conducting mapping or land surveys; or conducting factory or equipment inspections. Commercial users must register each UAS used for commercial purposes with the FAA. Public safety/government users operate UAS in a variety of ways to support key activities of their mission. For example, firefighters use UAS to help put out fires and the Department of the Interior uses UAS to survey national parks. Public safety and government users must either register each UAS or receive an FAA certificate of authorization to function as a public aircraft operator. FAA is the primary agency responsible for facilitating the safe integration of UAS into the national airspace. All airspace is regulated, and FAA’s rules regarding access to the airspace apply to the entire national- airspace system, from the ground up, though there are different rules for different types of airspace. As UAS increasingly enter and operate within the national airspace system—a complex network of airports, aircraft, air- traffic-control facilities, employees, and pilots—it is FAA’s responsibility to plan for and oversee the integration of UAS into both low-altitude airspace (below 400 feet) and, eventually, higher altitude airspace that will be shared with other aircraft. According to FAA’s Fiscal Year 2019 Implementation Plan, the ultimate goal of integration is for UAS to operate harmoniously with manned aircraft, in the same airspace, while ensuring the safety of people and property both in the air and on the ground. Within FAA’s Office of Aviation Safety, the UAS Integration Office is responsible for facilitating the safe, efficient, and timely integration of UAS into the national airspace system; aligning UAS international activities with foreign civil-aviation authorities; supporting standards and policy development related to UAS projects; and providing strategic planning and support for continuous UAS research and development. The Office was established in fiscal year 2017 and, in fiscal year 2018, had 39 full- time equivalent employees. Other offices within FAA coordinate with the UAS Integration Office on UAS-related activities. For example, FAA’s Office of Rulemaking (also under the Office of Aviation Safety) oversees the rulemaking process, including issuing notices of proposed rulemaking and administering the public comment process, in addition to providing general rule information on published regulatory documents. Other offices are also involved in the development of proposed rules, certification of aircraft, compliance and enforcement, and other activities related to UAS integration according to their subject-matter expertise. For example, the Flight Standards Service is responsible for setting standards for unmanned aircraft, and the Aircraft Certification Service is responsible for certifying new UAS designs and approving UAS for advanced operations. Additionally, the Air Traffic Organization is responsible for providing data and information to facilitate the operation of approved UAS near airports. Figure 1 shows FAA offices that are involved in UAS integration efforts. FAA’s activities are primarily funded through revenues to the Airport and Airway Trust Fund (Trust Fund), which is funded through a variety of excise taxes paid by users of manned aircraft as well as interest revenue accrued on the balance of the Trust Fund. These excise taxes are levied on the purchase of airline tickets and aviation fuel, as well as the shipment of cargo, though, as we have previously found, they are generally not closely linked to FAA’s costs for the services received. Trust fund revenues are available to FAA subject to appropriations. In addition to these revenues, a portion of FAA’s funding is often appropriated from general revenues. The Trust Fund provides funding for FAA’s three capital accounts: 1. the Facilities and Equipment account, which funds technological improvements to the air-traffic-control system, including the modernization of the air-traffic-control system called the Next Generation Air Transportation System (NextGen); 2. the Research, Engineering, and Development account, which funds research on issues related to aviation safety, mobility, and NextGen technologies; and 3. the Airport Improvement Program, which provides grants for airport planning and development. The Trust Fund also provides much of the funding for FAA’s Operations account, which funds the operation of the air traffic control system and the UAS Integration Office, among other activities. In general, a user fee is related to some voluntary transaction or request for government goods or services above and beyond what is normally available to the public, such as entrance into national parks, a request that a public agency permit an applicant to practice law or run a broadcast station, or the purchase of maps or other government publications. User fees are normally related to the cost of the goods or services provided. User fees’ designs can vary widely. We have previously reported that the way user fees are set and collected can affect the extent to which the goals of implementing user fees—equity, efficiency, revenue adequacy, and minimal administrative burden—are achieved. In 2017 the Drone Advisory Committee (DAC)—an industry stakeholder group established by FAA to provide advice on key UAS integration issues—created Task Group 3 to make recommendations related to funding the integration of UAS into the national airspace system. The group completed an interim report on short-term funding options in July 2017 and a final report on longer-term funding options in March 2018. The final report identifies various funding mechanisms for further study and recommends that industry, the FAA, and Congress work together to identify long-term funding sources for FAA’s UAS activities. In 2019, the FAA reconvened the DAC and plans to continue to form task groups to study emerging issues in the UAS industry, though no new task groups have been formed related to UAS funding. FAA has leveraged its existing regulatory and oversight framework for UAS integration, with the goal of allowing UAS operators to achieve increasingly complex operational capabilities. For example, FAA is applying existing regulations and standards developed for manned aviation to allow for more complex UAS operations. FAA has also initiated rulemaking efforts to allow operations of small UAS at night and over people in certain conditions and has identified additional areas for potential future UAS integration activities. For some capabilities, FAA has also identified a need for research and development, including for systems that would enable UAS to detect and avoid other aircraft and hazards. To help address these needs, FAA has established programs to draw on private industry’s resources and state and local governments, including the provision of air navigation services. Longer term, however, the extent of activities needed to carry out FAA’s statutory role in the operation, oversight, and enforcement of established rules and systems related to UAS is still unclear. According to FAA officials, just as the ultimate vision for UAS integration is for manned and unmanned aircraft to operate in the same airspace, FAA’s overarching strategy is to integrate UAS into its existing regulatory structure. This strategy is based in an incremental, risk-based approach to developing rules, policies, and procedures for UAS and leverages standards and regulations established for manned aviation as well as existing FAA resources such as rulemaking, flight standards, and aircraft certification personnel. To organize and track UAS integration activities across the agency, FAA has published internal annual-implementation plans for fiscal years 2017– 2019. FAA adjusts the plans annually to reflect changes in policy. These plans describe the range of objectives related to expanding the use and types of UAS in the national airspace that FAA has identified and its plan for achieving these objectives. For instance, the implementation plan includes identification of the steps needed to achieve each operational capability, including development of related regulations, policies, and standards. In recent years, FAA has implemented regulations that allow for routine UAS operations of gradually increasing risk and complexity. To date, FAA has established requirements for aircraft and operator registration as well as regulations to allow for limited operations of small UAS, including the June 2016 Small UAS rulemaking (commonly called Part 107), which established conditions under which small UAS operators are allowed to routinely fly for largely commercial purposes (see fig. 2). Additionally, for those operations not allowed under established regulations, FAA may grant waivers on a case-by-case basis. According to FAA, nearly 14,000 requests for waivers were received as of December 2018, with just over 2,000 of those requests approved. The Flight Standards Service has issued waivers for some UAS operators— including commercial and government users—to operate beyond-visual- line-of-sight or at night for purposes including inspection of hurricane damage and aerial photography. As FAA develops and implements regulations for more and more complex operations, fewer types of operations will require these waivers. Since issuance of the Part 107 rule, FAA has continued its efforts to increasingly allow for routine operations (that is, operations within established regulations that do not require waivers) of more types of UAS (including large UAS) under more conditions, as well as more complex UAS operations. Figure 3 illustrates some of the ongoing and potential future operational capabilities included in FAA’s phased approach for UAS integration, which are detailed below. FAA’s current efforts to allow for more complex UAS operations include the following ongoing rulemaking efforts: Operation of small unmanned aircraft systems over people: FAA issued a proposed rule in February 2019 to expand the operations permitted under the Part 107 rulemaking to allow operations over people and at night in certain conditions. Safe and secure operations of UAS advance notice of proposed rulemaking: FAA released this advance notice in February 2019 to seek public comment on whether FAA should promulgate new rulemaking related to, for instance, additional operating and performance requirements for UAS. Remote identification of unmanned aircraft systems (UAS): Both FAA and stakeholders have identified the ability for FAA, law enforcement agencies, and other UAS users to remotely identify UAS while in flight as foundational to most other rules and system development. FAA currently expects to issue a proposed rulemaking on remote identification in December 2019. With respect to the operation of UAS over people rulemaking, FAA expressly stated that it does not intend to finalize proposed rules in that area until it has issued a final rule on remote identification. In its internal Fiscal Year 2019 Implementation Plan, FAA identified a variety of new types of operations that could be enabled in the next few years. Examples include: Beyond visual-line-of-sight operations: Future integration efforts in this area could allow for low-altitude UAS operations beyond-visual- line-of-sight, such as infrastructure and agricultural inspections primarily below 400 feet. Small-cargo delivery operations: Future integration efforts in this area could allow for delivery of small cargo by networks of small UAS flying at low altitudes in rural and urban areas predominantly below 400 feet. Currently, FAA certifies some UAS operators to enable them to conduct cargo delivery operations under existing air carrier certification regulations. Urban air-mobility passenger operations: Future integration efforts in this area could allow for on-demand, highly automated, passenger air transportation services within and around a metropolitan environment with no pilot physically in the cockpit of the aircraft. These operations are expected to use UAS weighing thousands of pounds that would fly at higher altitudes (500-5,000 feet). UAS operators are currently developing UAS for future passenger transport operations both in the United States and abroad. Large cargo and inspection operations: Future integration efforts in this area could allow for cargo and inspection operations using significantly larger UAS (up to tens of thousands of pounds) operating in controlled airspace at higher altitudes. These UAS are expected to operate similarly to large commercial manned aircraft. These larger UAS may allow the transportation of larger volumes of cargo or execution of inspections over a longer range. Currently, FAA has approved—on a case-by-case basis—limited experimental operation of large UAS to conduct inspections by waiver. FAA’s annual UAS implementation plans reflect the ever-changing nature of the UAS industry, the regulatory environment, and concerns identified by stakeholders from within and outside of government related to public safety and national security. According to FAA, as UAS technology and the industry continue to evolve, additional operational capabilities and associated integration needs will be identified. FAA expects efforts to allow increasingly complex operations to build on lessons learned and technology improvements gained from preceding integration efforts. Until new regulations can be issued for these operations, FAA plans to extend and adjust existing safety standards and requirements—originally designed for manned aircraft—to UAS through waivers and exemptions. For example, in April 2019, FAA awarded the first air carrier certification to a UAS delivery company, Wing Aviation. This certification—under existing regulations for manned air carriers—allows the company to begin commercial package delivery in Blacksburg, Virginia. As discussed in its internal implementation plan, FAA has identified research and development needed to inform the safe expansion of UAS operational capabilities. According to FAA officials, this research focuses on the assessment of risks that UAS integration poses to the national airspace as well as the characteristics required for technology and systems to sufficiently mitigate these risks to achieve the safe implementation of more complex UAS operations. Such systems and technology would enable, for example, detection and avoidance of other aircraft and hazards, reliable navigation capability, and reliable data linkage between the UAS aircraft and the operator for controlling the flight. To that end, FAA coordinates UAS-related research activities being conducted by FAA, other government agencies, and FAA’s partners in industry and academia. For example, FAA has coordinated with NASA to develop a traffic management concept for UAS. Additionally, FAA has implemented two programs—the Test Sites program and Integration Pilot Program—to leverage private industry resources and state and local governments to conduct research and development activities needed to achieve full UAS integration. Test Sites Program: FAA authorized seven test site locations between 2013 and 2016 as directed by statute, at which industry stakeholders can test UAS technologies to further UAS integration. According to a test site participant, these sites have been used, for example, to test technologies such as vertical take-off and landing technology for large UAS, which may be relevant for large-cargo and passenger operations. Integration Pilot Program: This pilot was established in 2017 to enable testing of UAS integration technologies in state, local, and tribal jurisdictions. Through this program, for example, the North Carolina Department of Transportation has partnered with private industry to provide UAS medical-package delivery services (such as the transport of medical test samples). The program’s objectives include: testing and evaluating models of state, local, and tribal government involvement in the development and enforcement of federal UAS regulations, encouraging the development and testing of new UAS concepts of informing further FAA regulation of UAS. As these research efforts make headway, FAA plans to leverage the results to develop a system to provide UAS traffic management services. As stated in FAA’s Fiscal Year 2019 Implementation Plan, on any given day, 60,000 commercial aircraft fly through the national airspace into the 30 biggest airports in the United States and—given current trends—the same number of UAS flights could originate from just one delivery fulfilment center in a major city in a single day. According to FAA, in order to fully integrate commercial UAS into the national airspace, a traffic- management ecosystem complementary to—but separate from—FAA’s air-traffic-management system for manned aviation will likely be needed to control access and flight operations in low-altitude airspace. FAA has identified capabilities required for low-altitude UAS air navigation. One system—the Low Altitude Authorization and Notification Capability (LAANC)—has been implemented, while a UAS traffic management system is still under development. According to FAA and stakeholders we spoke to, LAANC was the first step towards a UAS traffic management system. LAANC: Through 2017 and 2018, FAA established technical and regulatory requirements for private partners to provide LAANC services, which enable UAS to access controlled airspace near approved airports. After deploying a system prototype in November 2017, FAA launched LAANC in April 2018 and then expanded the program to include additional partners in October 2018. Under LAANC, FAA provides data on temporary flight restrictions, notices, and airspace maps of participating facilities through a UAS data exchange. Private companies that have been approved by FAA to provide UAS air navigation services (called UAS service suppliers) develop and maintain—with private funding—automated applications or portals. Approved service suppliers provide differing services, with varying infrastructure and associated costs to provide the service. For example, some suppliers provide LAANC services to UAS operators among the general public, while others process applications for airspace access only for certain UAS operators. Prior to operating in controlled airspace near airports, UAS operators use these applications or portals to apply for airspace authorizations. These requests are checked against the data provided through the UAS data exchange, and if approved, UAS operators receive authorization to fly in the area—within minutes, in some cases. LAANC services were previously available only to commercial operators, but in July 2019, LAANC access was extended to recreational operators. UAS traffic management capability: In 2013, NASA began developing a concept of operations for a UAS traffic management system, which is the proposed system for providing UAS air navigation services in low-altitude airspace. As envisioned by FAA, these services will be separate, but complementary, to those provided by the Air Traffic Control system used for manned aviation. FAA established a pilot program in 2017 to develop and demonstrate early versions of UAS traffic management operations. Much like LAANC, the component applications and infrastructure supporting the traffic management system would be almost entirely developed, owned, and operated by private UAS service suppliers; only the Flight Information Management System—a data exchange gateway—is planned to be built and operated by FAA. The current UAS Traffic Management Concept of Operations envisions that UAS operators will share the timing and destination of a planned flight through a UAS service supplier. FAA envisions that these service suppliers will provide near real-time advisories to affected UAS operators regarding traffic (aircraft in the area), weather and winds, and other hazards pertinent to low-altitude flight (such as cranes or power-line construction or local UAS restrictions). Figure 4 illustrates the UAS traffic management system as outlined in the concept of operations. FAA has not identified an implementation date for the traffic management system. Rather, FAA proposes a “spiral development,” in which low complexity operations would be implemented first, with higher complexity operations and requirements built in incrementally. FAA intends to allow each new development to gradually mature the UAS traffic management system to ultimately support the full range of UAS operations at low altitude. Among other FAA activities, remote identification rules will be key to implementation of traffic management capabilities. Once FAA has developed the foundational UAS rules and systems such that operational capabilities of UAS integration have been substantially achieved, the specific nature of FAA’s role in the operation, oversight, and enforcement of established rules and systems depends on the nature of the established regulations and systems. FAA’s mission to ensure the safety of the national airspace, however, makes it clear that FAA will continue to play a role in each of these areas, given its responsibility for maintaining the safety of the national airspace. For example, FAA will need to continue conducting oversight to ensure compliance with established regulations, policies, and standards to maintain the safety of the national airspace, but the precise nature of the oversight needed in the future will depend on the regulations and systems established. We recently found that local law enforcement agencies may be unclear about their role in UAS enforcement and that most FAA inspectors and local law enforcement agencies GAO met with said that officers may not know how to respond to UAS incidents or what information to share with FAA. Similarly, a recent industry task force commissioned to address the issue of unauthorized UAS near airports found that the role of state and local law enforcement in addressing that threat is unclear, and recommended that federal agencies clearly define related roles, responsibilities, and authorities. As such, FAA’s activities related to enforcement for UAS will likely evolve as UAS become more integrated in the national airspace. Further, according to our interviews with stakeholders, facilities designated for the take-off and landing of UAS for the transport of passengers and cargo as well as other infrastructure to support UAS air navigation services may be needed. FAA’s role in operating or overseeing this infrastructure will likely hinge on the nature of the infrastructure. For example, while FAA’s Office of Airports has responsibility for airport safety and inspections as well as establishing standards for airport design, construction, and operation, the extent to which this type of oversight will be needed for infrastructure to facilitate drone operations is not yet known. FAA receives annual appropriations in four accounts, and since 2016, conference reports accompanying appropriations have directed FAA to allocate some funding from these accounts specifically for UAS-related activities. Table 1 depicts appropriations FAA has allocated to UAS- related activities from these four accounts since 2016 at the direction of Congress. FAA allocates portions of its appropriations for the UAS Integration Office and some other UAS-specific activities based on congressional direction, but FAA may obligate funding that has not specifically been allocated for UAS activities to support UAS activities as well. The vast majority of FAA’s appropriation comes from the Airport and Airway Trust Fund (which is funded through revenues of taxes and fees on manned aviation airline tickets, aviation fuel, and cargo shipments), including all of the appropriations for the facilities and equipment; research, engineering, and development; and grants-in-aid for airports accounts. In fiscal year 2018, about 92 percent of FAA’s approximately $17 billion in total funding was appropriated from the Trust Fund. The remainder of FAA’s funding is appropriated from general revenues. For fiscal year 2018, in accordance with congressional direction, FAA allocated a total of $104.8 million for UAS-related activities and, according to FAA financial data, obligated approximately $69.7 million for these activities. Table 2 provides an overview of the UAS-related activities for which FAA determined it had obligated funds in fiscal year 2018; a more detailed list of UAS-related activities for which FAA identified fiscal year 2018 obligations is provided in appendix 2. Individual activities may be funded through more than one account, depending on their scope. According to officials, and as discussed below, FAA staff outside of the Office of Aviation Safety and Air Traffic Organization may not consistently track their UAS-related obligations. As such, the obligation amounts identified in table 2 may be incomplete and may not represent FAA’s total fiscal year 2018 UAS costs. Within the categories above, specific examples of activities funded in fiscal year 2018 include: About $3.7 million from both the Operations ($2.07 million) and Facilities and Equipment ($1.65 million) accounts for the development of LAANC systems and requirements. Of the about $33 million obligated by the Office of Aviation Safety in fiscal year 2018 for UAS-related activities, about $28 million was obligated by the UAS Integration Office and $166,000 by the Office of Rulemaking. $4.5 million obligated under facilities and equipment for the development of a UAS traffic management system and the associated Flight Information Management System. Since 2017, FAA has been tracking costs associated with many of its UAS activities including time spent by staff as well as other costs, as shown in table 2. A December 2017 internal memorandum instructed FAA offices to track UAS-related activities and costs using project codes. According to FAA officials, the codes are used to identify travel, procurement, time and attendance, and costs related to special events, among other UAS-related activities. The effort was intended to address the administration’s and Congress’ interest in greater cost visibility. According to FAA officials, the project codes to track UAS costs have been implemented in the Office of Aviation Safety—including the UAS Integration Office, Flight Standards Service, and Office of Rulemaking— and staff within the Air Traffic Organization (not including air traffic controllers). According to FAA officials and as demonstrated by the obligations shown in Table 2, the Office of Aviation Safety and the Air Traffic Organization represent the majority of UAS costs for fiscal year 2018 within the Operations account. In addition, according to FAA, because Conference Reports have outlined how activities in the Research, Engineering and Development and Facilities and Equipment accounts should be funded by line item, FAA is able to track these costs without using the project code method. While FAA has started tracking UAS-related costs for some offices, FAA does not know the extent to which UAS costs are tracked throughout the agency, resulting in data that may be incomplete. Many—if not all—FAA offices are doing work related to both manned aviation and UAS, but FAA officials stated that they do not know or plan to assess the extent to which staff in other offices—such as the Office of the Chief Counsel—that spend time on both UAS-activities and other responsibilities are using the project codes to track their UAS-activities. FAA officials stated that, because the bulk of the UAS-related work is being conducted within the Office of Aviation Safety and the Air Traffic Organization, it is not a priority to try to identify the time spent by other offices working on UAS-related activities, which they believe would be time consuming. However, with no way to assess the extent to which the project codes have been implemented, FAA is unable to tell whether it has met the intent of using the codes: greater visibility into UAS-related costs. For instance, FAA does not currently have visibility via the project codes into time spent on UAS activities outside of the Office of Aviation Safety and the Air Traffic Organization. According to OMB instructions to agencies on financial-reporting requirements and standards for federal financial accounting, agencies should report the full cost of each program—to include both direct and indirect costs and the costs of identifiable supporting services provided by other offices within the agency. Further, federal standards for internal control note that agencies should use quality information—that is, data that are complete and accurate—to achieve objectives, make informed decisions, and manage risks. With no assurance that the project codes are resulting in information that is complete, FAA risks making decisions based on information that is unreliable for the purpose of understanding the full costs of its UAS activities. Efforts to track costs need not be overly complex: federal financial-accounting standards note that agencies should consider the precision desired and needed in cost information and the practicality of data collection and processing, among other considerations, when designing cost-accounting processes. For example, FAA could build on its existing project codes for UAS-related activities by monitoring the extent to which the project codes have been used agency- wide. Alternatively, FAA may identify other methods of accounting for UAS-related costs, if there are some costs not easily tracked using the project codes. Further, indirect costs associated with FAA management and facilities could be assigned to the UAS mission based on more complete information on the direct costs identified through use of the project codes. Additionally, as discussed below, many of FAA’s future costs related to UAS are unknown. Ensuring the project code information is complete and accurate now could put FAA in a better position to identify those costs as they are realized in the future. Further, federal standards for internal control state that management should identify, analyze, and respond to significant changes that could affect an agency’s ability to report reliable information and achieve objectives—such as a change in mission that influences costs. Without reliable information on FAA’s UAS-related costs, the administration and FAA may be less equipped to make informed policy decisions regarding resources needed as UAS become further integrated into the national airspace and as UAS oversight becomes an increasing part of FAA’s mission. Because the UAS industry, as well as key systems and technological developments, continue to evolve, it is too early to know what costs related to UAS that FAA is likely to incur in the future . This holds true for future operational costs as well as the costs to develop future systems and regulations and indirect costs. According to FAA and stakeholders we spoke to, in addition to costs to continue regulatory activities and safety oversight, FAA’s future costs will depend on the extent of FAA’s involvement in the everyday operation and oversight of systems, such as those related to UAS traffic management, and the extent to which FAA becomes a provider of UAS-related services. Examples of how FAA’s costs could evolve and possibly expand in each of these areas include: Regulatory development costs: Current costs for activities such as the development of new UAS regulations by the UAS Integration Office could change as UAS become more integrated into the national airspace. As previously discussed, the industry is changing rapidly and new uses for UAS are being developed, uses that will require additional FAA regulation and oversight. FAA cannot know the extent to which additional rulemaking activities will be required for UAS technologies and uses that the industry has not yet contemplated or developed. Costs to develop regulations involve input from offices across FAA, such as the Office of the Chief Counsel, where FAA officials are unsure if staff are consistently using the project codes to track their costs for UAS-related activities. As such, FAA may not have visibility into the extent to which these UAS-related costs may change over time. Safety oversight costs: As part of its safety mandate, FAA is responsible for enforcing compliance with established regulations for both manned aircraft and UAS. Several offices within FAA have a role in UAS compliance and enforcement, including the Flight Standards Service and the Office of Security and Hazardous Materials Safety. As we have recently reported, while FAA has sole responsibility for enforcement of UAS regulations, the agency has focused on engaging and educating law enforcement and public safety agencies at all levels—federal, state, and local—and, to a lesser extent, conducting surveillance to ensure compliance with UAS regulations. While local law enforcement agencies may often be in the best position to deter or respond to UAS incidents, they may not have information on how to respond or what information to share with FAA. According to FAA officials, the Office of Security and Hazardous Materials Safety is one of the offices in which FAA officials do not know if staff are tracking their activities and costs related to UAS through use of the project codes discussed above. Given the uncertainty about the division of responsibilities between federal, state, and local law enforcement, it is unclear how costs for safety oversight and enforcement will evolve and possibly expand in the future. Provision and oversight of UAS services and facilities: FAA will eventually incur costs related to providing air navigation and other services to UAS operators, oversight of UAS service providers, and potential infrastructure, but the extent of FAA’s eventual role in the provision of these services and related oversight is not yet understood, in part, because the industry is still evolving and it is unclear what FAA services will be provided in the future. Some stakeholders believe that an increased industry role in providing air navigation services could keep FAA’s costs for these activities relatively low. For example, the UAS Traffic Management Concept of Operations envisions that leveraging private entities to provide a variety of UAS traffic management services will reduce the infrastructure and manpower burden on FAA and, thus, reduce associated costs. FAA envisions that the Flight Information Management System—a system through which FAA can provide directives and enable information exchange between UAS service suppliers, UAS operators, and FAA—is the component of the UAS traffic management system that FAA will build and manage. FAA has not yet estimated the costs of developing or implementing this system because, according to FAA officials, the agency is still many steps away from developing the core infrastructure and regulatory requirements. As UAS integration progresses and as more UAS are operating in the same airspace as manned aircraft, additional solutions may be needed to manage UAS traffic at higher altitudes, which will also incur costs. For instance, FAA anticipates that air traffic controllers will have a role in de- conflicting manned aircraft and unmanned aircraft and could provide air-traffic-control services to UAS in controlled airspace. FAA officials stated it will be necessary to collect data on the direct and indirect costs of UAS for air-traffic-control services in the future. According to FAA, a new air-traffic-control-cost-allocation study is underway, but FAA does not currently have the information on UAS operations that would be necessary to assign air traffic control costs to UAS users. Beyond system development, once traffic management systems are designed and operational, FAA will incur costs related to its role in overseeing providers of UAS traffic management services as well as operating and maintaining the Flight Information Management System. FAA currently provides UAS operators with services related to registration, aircraft certification, and waivers for operation that fall outside existing regulations, but those services may change depending on future rulemaking. When it becomes clearer what services FAA will likely provide and how it will provide those services, FAA will be better positioned to estimate its costs to inform its budget requests and plan for the future, as it has done for systems that have already been implemented. For example, FAA has estimated future costs associated with the LAANC program, which was implemented in 2018. FAA anticipates obligating approximately $35.64 million from the facilities and equipment account and $26.6 million from the operations account to further develop and operate the LAANC system from fiscal years 2019 through 2023, as shown in table 3. Indirect Costs: In addition to direct costs related to rulemaking, oversight, and provision of services, FAA will continue to incur indirect costs such as those associated with the operation and maintenance of FAA facilities and systems. FAA officials said they do not plan to conduct analysis through which they could allocate indirect costs for UAS, because FAA’s appropriations and funding structure do not require them to track costs in this way. However, as previously discussed, OMB instructions to agencies on financial-reporting requirements state that agencies should report the full cost of each program including indirect costs. As discussed, FAA’s efforts to track costs related to UAS activities may result in incomplete data, and as the UAS industry evolves and becomes more integrated, tracking costs may become even more complex. Generally, FAA officials stated that differentiating between costs related to UAS and manned aviation will not be necessary as UAS become further integrated into the national airspace and FAA’s mission because the agency does not track costs in this way for any other mission areas. However, as discussed later in this report, there is widespread consensus among manned and unmanned aviation industry stakeholders that UAS costs should be borne by the UAS industry rather than the manned aviation industry, and policy makers may opt to recover these costs through user fees or some other mechanism in the future. As discussed below, should FAA and Congress decide that certain fee mechanisms should be pursued, a reliable accounting of total program cost—including indirect costs—is important to setting effective fees, as our prior work related to designing user fees has shown. In the tasking statement to the Drone Advisory Committee’s Task Group 3, FAA asked the committee to recommend options for funding the activities and services required by both government and industry to safely integrate UAS into the national airspace system. The Task Group concluded in its final report that the aviation industry, FAA, and Congress should coordinate to identify one or more revenue streams that are separate and segregated from the Airport and Airway Trust Fund to help fund FAA’s UAS-related activities. The Task Group also identified five different fee mechanisms through which FAA could recover some of the costs of its activities from UAS users, a topic we discuss in this section. The extent to which costs are recovered from UAS users and the methods by which costs are recovered are policy decisions for the administration and Congress. Since 2015, FAA has used one fee mechanism—a $5 registration fee, the same as the fee to register a manned aircraft—to recover some of the costs associated with administering the UAS registration requirement. Most of FAA’s UAS- related costs are in areas unrelated to UAS registration. As such, policy makers may, at some point, consider additional ways to recover the costs of UAS activities, including implementing user fees for additional services and activities, subject to congressional authority to implement fees and use resulting revenue. Our prior work on designing user fees, combined with policies established by the Office of Management and Budget, can provide a framework for designing user fees that reduce the burden on taxpayers to finance FAA’s UAS activities, which benefit specific users. The goals of establishing user fees—efficiency, equity, revenue adequacy, and reducing administrative burden—can be in conflict with each other and necessitate trade-offs depending on policy priorities. Table 4 describes these goals. Our prior work illustrates that four key design elements—namely how fees are (1) set, (2) collected, (3) used, and (4) reviewed—require careful consideration and planning to achieve the desired goals. Based on the prospective nature of user fees to recover FAA’s UAS-related costs, we will focus on how user fees are set and collected. It is important to note that given the tradeoffs involved in establishing user fees, different users and stakeholders may have varying perspectives and opinions on what would be an appropriate fee structure. As these are policy decisions, this report does not recommend any specific fee mechanism. Instead, the considerations and examples we present are intended to inform decision- making by laying out issues to take into account when designing user fees. As discussed in our User Fee Design Guide, determining how UAS user fees should be set and collected involves a number of steps. These steps include: identifying the costs associated with each activity and which costs should be recovered, identifying the beneficiaries of each activity, determining how to set fees for various types of beneficiaries, determining how fees should be collected, and determining when it is appropriate to begin collecting fees. OMB instructions on designing user fees state that user fees should be sufficient to recover the full cost of providing each service or resource, including indirect costs, except to the extent that agencies determine that exceptions should be made. Identifying the full costs of providing a UAS service or resource—such as providing access to maps and air-traffic management services like LAANC—could enable policy makers to determine, consistent with their policy goals, which of those costs should be recovered through user fees. Identify the costs of each activity: Our prior work has found that, to set fees so that total collections cover the intended share of program costs, a reliable accounting of total program cost is important. As previously discussed, while the costs of some current regulatory and operational activities related to UAS are known, some current and most future costs are unclear. Recognizing that generating and maintaining reliable cost data can be expensive, OMB instructions note that program cost should be determined or estimated from the best available records of the agency. Accordingly, policy makers could opt to implement fees to recover the estimated costs of each activity as regulations, services, and systems are established, and adjust fees periodically based on actual costs. Determine which costs to recover: The next step is to determine the extent of the costs for each activity that should be recovered through user fees based on policy goals. For example, as discussed, many of FAA’s current costs relate to the “setup” or integration of UAS into the national airspace, including the costs to develop and promulgate UAS operational rules. Policy makers may or may not decide to recover these current costs from future users. For example, policy makers may decide not to recover these costs based on the idea that the goal of promulgating UAS-related regulations may be related to the general safety of the airspace, rather than providing benefits to specific users. Additionally, some stakeholders we interviewed stated that the costs of startup activities (like rulemaking) and safety oversight activities (like enforcing existing regulations) should not be recovered through user fees because these activities are core government functions. Rather, these stakeholders advocated funding such activities through appropriations from general revenues. However, as we have discussed in prior work, fees have frequently been used to support agencies’ regulatory programs. For example, fees assessed by financial regulatory agencies and the Nuclear Regulatory Commission on their respective regulated industries are used to support those agencies’ regulatory activities. Our prior work has found that the extent to which a program is funded by user fees should generally be guided by who primarily benefits from the program, though the extent to which a program benefits specific users or the general public is not usually clear cut. The beneficiaries of FAA’s UAS-related activities will include both direct users (UAS operators) as well as indirect beneficiaries such as the general public. Direct beneficiaries will accrue benefits from their use of UAS, whether recreational, governmental, or commercial. In contrast, indirect beneficiaries would benefit from maintaining a safe national airspace system and preventing disruption of commercial flights and other manned aviation. Policy makers may decide that, to account for benefits to those who don’t directly engage in UAS activities, a percentage of FAA’s UAS- related costs should be funded with general revenues. For instance, as the Congressional Research Service has reported, there has been general acceptance that appropriations to the FAA from general revenues account for the public benefits of FAA’s regulation of the national airspace. Additionally, while the manned aviation industry will benefit from regulations and oversight that reduce the potential for disruption in the airspace caused by UAS, UAS operators benefit from the regulation and safety oversight of the manned aviation industry as well. Policy makers may choose to account for these benefits in any number of different ways, depending on the perceived extent of the benefit enjoyed by each group. Direct beneficiaries—including recreational, commercial, and governmental UAS operators—will benefit in different ways based on both the type of user and the type of use or activities they engage in. For example, recreational users may experience the joy and excitement of flying UAS, but are not authorized to accrue any economic benefits. In contrast, commercial users are operating UAS with the explicit goal of earning revenue or benefiting business interests in some other way as a result of their UAS operations. As outlined in our prior work, policy makers may set fees for different types of users and activities based on a variety of factors including (1) costs imposed on the system by each user or type of use, (2) the extent of benefits received by different types of users, (3) the ability of each user to pay, and (4) identified policy goals. Figure 5 presents a simplified, hypothetical example of setting fees for various activities and users. The following examples illustrate how these various factors could play out: Considering costs imposed: Policy makers may set fees to recover the costs imposed by UAS users requiring air navigation services—for example, those operating in controlled airspace (such as around airports) or in high traffic areas. Policy makers may set fees to account for the different costs imposed by providing different UAS users access to air traffic services, such as charging per flight for air navigation services or basing the fee on distance traveled in controlled airspace. Policy makers may decide that recreational UAS users should pay lower fees than commercial users because they may generally impose fewer costs on FAA. Considering Benefits Received: Policy makers may set fees for some services that account for the extent of the benefit received, such as charging for air navigation services based on value of cargo or number of passengers transported. Considering ability to pay: Policy makers may decide to allocate a larger share of FAA’s UAS-related indirect costs to commercial users, based on their ability to pay and the monetary benefits they receive. Considering policy goals: Policy makers may decide that public safety agencies (government users), such as local police departments, should be exempt from fees or pay reduced fees because their use of UAS may provide a public benefit. Policy makers may seek to increase safety by reducing or eliminating fees for certain services in order to reduce the probability that users may not comply with requirements to avoid paying an associated fee. This determination would require balancing the potential revenue associated with the fee against (1) the potential costs of ensuring compliance with operational requirements and fees through enforcement activities and (2) the safety risks associated with the portion of operators who may try to avoid fees through not complying with operational requirements. Most stakeholders we spoke to agreed that UAS users should pay a fee when they receive a service from FAA but that fees should be related to the costs incurred by use. In discussing whether distinctions should be made in setting fees based on factors like commercial or recreational status, cargo or passenger flights, size and weight of the aircraft, and intended use of airspace, most stakeholders agreed that fees should be charged based on these distinctions only insofar as they are associated with different costs imposed on UAS-related systems or FAA. Based on the evolving nature of the industry, it is unclear whether distinctions like those above would be related to differences in costs imposed on FAA. Some other countries have implemented user fees to recover the costs associated with UAS-integration and air navigation services, though integration is still in progress. For example, Transport Canada (the Canadian agency responsible for developing transportation regulations, policies, and programs) has established a regulatory structure requiring UAS pilot certification and UAS registration. It set fees to recover Transport Canada’s costs for administering those requirements: $5 Canadian dollars (CAD) for registration (similar to FAA’s registration requirement), $10 CAD for a basic pilot certification, and $25 CAD for certification to perform advanced operations, such as flying in controlled airspace. NAV CANADA (Canada’s private, non-profit air navigation service provider) is in the process of establishing a LAANC-like service through a third-party but has not yet determined whether or how NAV CANADA may seek to recover these costs. In another example, officials told us that the Swiss Federal Office of Civil Aviation is required to recover its costs, so their general philosophy will be to charge a fee whenever costs are incurred. The regulatory structure is still under development, but the office currently charges UAS users for the time required to issue waivers for UAS operations. For example: For certain operations, such as those within visual-line-of-sight and not over people, no authorization is required, and thus no fee is required. For advanced operations—such as those beyond visual line of sight or over people—fees are charged based on the time required to conduct analysis and risk assessment up to a maximum of 5000 Swiss Francs. Policy makers can identify opportunities to collect fees based on the characteristics and requirements of relevant aviation navigation and other systems as these systems are developed. OMB instructions to agencies related to user fees state that fees should be collected prior to or at the time a service is provided unless agencies are legally authorized to collect fees after the service has been provided. Our prior work has found that collecting fees at the time a service is provided may reduce the administrative burden. Here, for example, the UAS traffic management system may include points in the process when users are required to obtain an FAA authorization or notify FAA or UAS traffic-management service providers of operation requirements. Those points may provide an opportunity for fee collection. Similarly, as FAA does for current UAS registration fees, online systems for other services could provide an opportunity for FAA to collect fees associated with those activities. Alternatively, fees could be collected through a third party to reduce the administrative burden on FAA. For example, if UAS passengers are subject to fees, flight operators could collect those fees on behalf of FAA, as occurs with current passenger excise taxes for manned aviation. Similarly, UAS service suppliers could collect fees from UAS operators on behalf of FAA for air navigation services. Decisions about when to implement user fees depend on both practical and policy considerations. For example, user fees could be put in place as soon as FAA implements each UAS-related regulation, service, or system—that is, once FAA’s costs related to a given activity can be estimated and beneficiaries identified. Alternatively, policy makers may decide not to implement user fees, or to implement some fees but not others, for a period of time in order to allow the nascent UAS industry to develop and to increase commercial viability. FAA’s tasking statement for Task Group 3 noted that one option is to consider the UAS industry an “infant industry” in need of special protections, in which case FAA could need to ask Congress for additional appropriations from the general fund to support UAS-related activities in the interim. Our prior work notes that while it may advance a particular policy goal to, for example, waive fees for a nascent industry for a period of time, such provisions might create unfair competitive advantages among users or industries. In discussing what level of system development should be achieved prior to imposing fees, stakeholders we spoke to had a wide range of divergent opinions, including the following: Some fees, like the existing registration fee, can be imposed now—as users are receiving value and FAA is incurring costs—and adjusted as the industry develops. Designing fees for UAS should take place only after the infrastructure and regulatory environments have been established. FAA and other policy makers should start considering user fees and an accompanying cost accounting and allocation system as soon as possible, but implementation should wait until a UAS traffic management system has been implemented. Fees for FAA services should be implemented when commercial operations over people and beyond-visual-line-of-sight are routine (that is, when advanced, revenue-generating UAS operations are being conducted without need for a waiver). The Drone Advisory Committee’s Task Group 3 concluded that funding for integration efforts would be shared across government and industry and that user fee mechanisms should be considered to recover FAA’s costs related to a range of activities including rulemaking, development of policies and standards, and research and development. While the task group did not make a specific recommendation on a particular fee mechanism, its final report identified five possible fee mechanisms with the intention of providing policy makers with ideas: Filing and licensing fees: Similar to the already-implemented UAS registration fee, FAA could impose fees to recover the costs of other FAA services such as reviewing applications for waivers and certifications. Point-of-sale-tax: Legislation could be passed to impose a federal tax on UAS and ensure that the proceeds are used to offset the costs of FAA’s UAS-related activities. Business use fee or tax: A business use or transaction tax could be imposed on the purchase of a UAS-related service: Commercial businesses that use UAS on behalf of a customer or as part of their customer service could be responsible for a “pay as you go” model fee for use of the airspace, which would be added to the invoice. This concept could include, for example, fees for passengers using urban air-mobility services or fees for the transport of cargo by UAS, similar to the existing excise taxes that fund the Airport and Airway Trust Fund for manned aviation. Airspace access fee: FAA could recover some or all of the costs associated with UAS traffic management services by requiring that UAS operators filing flight plans or other requests to operate UAS pay a fee to FAA. For instance, the report proposes that operators could remit a fee online when they request access to airspace near airports using LAANC. Auction or lease of airspace: FAA could recover costs or receive revenue for use of a public resource (navigable airspace) by conducting auctions to grant a license to UAS traffic management service suppliers, similar to granting radio spectrum licenses, which have been used or proposed to address overcrowding of spectrum and have resulted in significant revenue. Stakeholders noted that there is not currently a problem with capacity of the national airspace with respect to the operation of UAS and that there is no need for auctions of airspace on the basis of scarcity. According to FAA, each of these options would generally require additional authority from Congress to enable FAA to collect and use revenue. The Task Group 3 report and most stakeholders we spoke to (many of whom participated in the Task Group) agreed that the fee mechanisms identified generally covered the range of potential options and stated that it is too early to know which fee mechanisms would be appropriate to recover the costs of any one activity. Nonetheless, stakeholders described their overall impressions of how each mechanism could work, including the following considerations: If fees are burdensome for casual users, fees could lead to non- compliance with requirements. Fees that rely on self-reporting by users might be difficult to enforce or might create a disincentive for users to operate within the system (that is, operators might find ways to operate without FAA’s knowledge to avoid paying a fee), an outcome that could decrease compliance with rules meant to increase safety. A point-of-sale tax (generally a percentage of the cost of the products) on UAS would not necessarily be in proportion to the cost of services or benefits being provided by FAA and might be complicated to implement and administer. For example, stakeholders noted that a point-of-sale tax would not apply to home-built or second-hand UAS users and the tax would not be linked to actual use of the UAS (that is, the UAS activities that might impose costs on FAA). FAA officials told us that they have not yet identified or studied potential UAS fee mechanisms or analyzed the findings included in the Task Group 3 report because they have been waiting for the results of our work to inform their decision-making and planning. OMB instructions to agencies related to user fees establish that—to increase efficiency of resource allocation and reduce burden on taxpayers—agencies should recover costs when special benefits are delivered to specific users and that agencies must review all agency programs on a biennial basis to determine whether fees should be assessed. Similarly, federal internal control standards note that management should identify, analyze, and respond to significant change—such as increasing costs related to a change in mission like the integration of UAS to the national airspace— using a forward-looking process. Given the evolving nature of the UAS industry, it is unclear how UAS users and associated government activities and services fit into FAA’s existing funding structure. As the balance of FAA’s activities gradually shifts to include increased focus on UAS-related activities, those activities continue to be funded by a combination of manned aviation users (through revenue to the Airport and Airway Trust Fund) and taxpayers (through general revenues). The revenues to the Airport and Airway Trust Fund are from taxes on airline tickets, cargo, and fuel, but are not closely linked to the costs to FAA of providing specific services. In 2007, FAA and the administration proposed a new funding system that would rely more on cost-based fees for specific manned aviation activities. This proposal, however, was never implemented. We previously testified regarding this proposal, noting that such fees could allow FAA to better identify funding options that link revenues and costs and improve transparency by showing how much is being spent on specific FAA activities, but that achieving these goals would depend on the soundness of FAA’s cost allocation methodology and extent to which revenues are linked to costs. The provision in the FAA Reauthorization Act of 2018 for GAO to conduct this review, FAA’s tasking statement for Task Group 3, and statements made by Task Group 3 in its final report suggest an interest among Congress, FAA, and industry stakeholders, respectively, in considering user fees as an option for recovering the costs of FAA’s UAS activities. Implementation of cost-based user fees for UAS would be different from FAA’s longstanding funding structure for manned aviation, but may not necessitate a change in that existing structure for areas of FAA’s mission other than UAS. Indeed, the Task Group 3 report expresses a consensus that options for UAS funding should not be constrained by the current traditional aviation funding structure, and any recommended funding structure should not alter the current structure of funding for traditional, manned aviation. As UAS integration continues to evolve, FAA may identify ways that the current aviation funding structure can be adjusted to recover costs related to UAS operations. For instance, FAA officials noted that, once large UAS cargo and passenger operations have been established, those operations could become subject to the same excise taxes on fuel, cargo, and passengers as are manned operations. As we have discussed, fees to recover FAA’s costs for its UAS activities need not be assessed on a program-wide basis. That is, fees to recover the costs of individual UAS activities can be implemented separately either as new rules or systems are developed or as FAA reviews its activities and identifies areas in which services to UAS users are incurring costs that could be recovered. Further, fees based on costs to FAA estimated as each rule or system is developed can be periodically adjusted as needed. As explained in our User Fee Design Guide, periodic reviews of user fees can help ensure that Congress, stakeholders, and agencies have complete information about changing program costs and that fees remain aligned with program costs. As UAS integration continues, ongoing conversations between Congress, FAA, and stakeholders may provide additional insight into how fees can be implemented to accomplish goals. To date, FAA has not incorporated steps into its existing UAS planning efforts to identify potential fee mechanisms. Considering potential user fees as part of these efforts—such as FAA’s annual UAS implementation planning—could better position FAA to design effective user fees should policy makers task FAA with implementing them. For instance, collecting information on costs and beneficiaries as new UAS-related services are developed and implemented could ensure that data needed to design effective user fees are available. Similarly, considering ways to collect revenue—such as through third parties or online systems—as services and systems are being developed or adapted for UAS users, could facilitate future implementation of fees. As an example of the type of planning that may be needed, FAA officials said that identifying the costs of UAS traffic management services for the purpose of setting fees would involve (1) tracking which UAS are using the national airspace and (2) tracking and categorizing the type of operations conducted. Incorporating a means of collecting these data during the planning and development of traffic management systems would be useful to future fee-design considerations in this area. This is not to say that cost recovery considerations should drive the development of regulations or systems at the expense of mission goals. Rather, such planning would offer opportunities for FAA to examine systems, policies, and regulations that have been designed to accomplish the goals of UAS integration in order to assess (1) how each system, policy, or regulation will affect FAA’s costs; (2) the need for additional resources; and (3) potential options for collecting revenue. FAA is tasked with managing the integration of UAS into the national airspace within the context of many competing priorities and limited resources. Without a process to ensure information on UAS-related costs is complete for either current or future efforts, neither FAA, nor the administration, or Congress have reliable information about the total costs of FAA’s UAS-related activities and therefore may lack the information needed to effectively prioritize resources. Further, this information could inform the design of effective user fees, should policy makers decide that such fees are appropriate. FAA’s UAS integration-planning efforts offer an opportunity for FAA to build the collection of relevant data, and consideration of user fee options, into ongoing activities. We are making the following two recommendations to the FAA: The Administrator of the Federal Aviation Administration should develop and implement a process to ensure that information on UAS-related costs is complete and reliable as capabilities and related regulations evolve. (Recommendation 1) The Administrator of the Federal Aviation Administration, as part of UAS integration-planning efforts, should use available guidance on effective fee design to incorporate steps that will inform future fee-design considerations. For example, FAA may choose to incorporate these additional steps into its annual UAS implementation plan so that—as existing activities are adapted for UAS users or new regulations, services, or systems are introduced—costs and fee design options are considered. (Recommendation 2) We provided a draft of this report to the Department of Transportation (DOT) for comment. In its comments, reproduced in appendix III, DOT agreed that there are likely opportunities to better track and recover UAS- related costs and concurred with our recommendations. We will be sending copies of this report to appropriate congressional committees and the Secretary of Transportation. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or KrauseH@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this report are listed in appendix IV. Heather Krause, (202) 512-2834 or KrauseH@gao.gov. In addition to the contact named above, the following individuals made important contributions to this report: David Sausville, Assistant Director; Katie Hamer, Analyst-In-Charge; Alexandra Jeszeck; Amy Abramowitz; Camilo Flores; Richard Hung; Delwen Jones; Heather Keister; Hannah Laufe; Susan Murphy; Joshua Ormond; Pamela Snedden; and Elizabeth Wood.", "summary": "UAS have the potential to provide significant social and economic benefits in the United States. FAA is tasked with safely integrating UAS into the national airspace. As the UAS sector grows, so do demands on FAA's staffing and other resources to develop, oversee, and enforce rules and systems needed to safely integrate UAS into the national airspace. The FAA Reauthorization Act of 2018 provides for GAO to review issues related to establishing fee mechanisms for FAA to recover its costs related to UAS. This report discusses, among other things, 1) FAA efforts to track the costs of current and planned activities related to UAS and 2) key considerations and options for designing user fee mechanisms that could recover FAA's costs. GAO reviewed FAA documents and financial data for fiscal years 2017 through 2019 and industry reports on drone integration funding. GAO interviewed a non-generalizable sample of 22 UAS industry stakeholders, selected based on participation in FAA advisory groups or prior GAO knowledge to achieve a range of perspectives. GAO reviewed its guidance on designing effective fee mechanisms and OMB instructions to agencies about implementing user fees. The Federal Aviation Administration (FAA) has undertaken actions to integrate unmanned aircraft systems (UAS or “drones”) into the national airspace and has developed plans to allow for increasingly complex operations, including operations over people and beyond visual-line-of-sight and—eventually—passenger operations (see figure). However, FAA efforts to track related costs may result in incomplete information. FAA established a means of tracking the costs associated with some UAS-activities in certain offices, but many, if not all, FAA offices are doing work related to both manned aviation and UAS. FAA officials stated that they do not know or plan to assess the extent to which staff who split their time between UAS-activities and other responsibilities are tracking those costs. Furthermore, FAA's future costs to conduct oversight and provide air navigation services are largely unknown due to the changing nature of the industry and its early stage of development. Ensuring that information on UAS-related costs is complete and reliable now could put FAA in a better position to identify those costs as they evolve and possibly expand in the future. The extent to which FAA should recover costs for its UAS-related activities, and what fees are appropriate, are policy decisions for the administration and Congress. Accordingly, this report does not recommend any specific fee mechanism. Nonetheless, planning and consideration of policy goals, using available guidance on user fee design, could better position FAA to inform future decision-making on these issues as it proceeds with UAS integration. Since 2015, FAA has collected a registration fee from UAS operators, but most of FAA's UAS costs are not related to registration or covered by this fee. A stakeholder group established by FAA identified potential fee mechanisms and concluded in 2018 that the aviation industry, FAA, and Congress should identify revenue streams to help fund FAA's UAS activities. Further, GAO guidance and Office of Management and Budget instructions provide a framework, including information requirements, for designing effective user fees. FAA officials said that they have not considered user fee mechanisms as part of their planning because they have been awaiting this report to inform their decision-making. By using available guidance as part of its planning, FAA could incorporate steps, such as identifying costs and beneficiaries, which would benefit future fee design considerations. GAO is recommending that FAA (1) implement a process to ensure UAS-related cost information is complete and (2) use available guidance on effective fee design to incorporate steps, as part of UAS integration planning, that will inform future fee design considerations. FAA concurred with the recommendations.", "document_type": "gao"}
{"report": "Together, Executive Order 12898 and the 2011 MOU include eight areas that agencies’ environmental justice efforts should address, as appropriate, including promoting enforcement of all health and environmental statutes in areas with minority populations and low-income populations and ensuring public participation. Executive Order 12898 did not create new authorities or programs to carry out federal environmental justice efforts. As a result, federal environmental justice efforts seek to use existing federal laws, programs, and funding to address environmental and health problems that disproportionately burden minority and low-income communities, such as exposure to environmental pollutants. Such existing laws include the following: Environmental laws. Several environmental laws regulate pollutants in the air, water, or soil and generally require a regulated facility to obtain permits from EPA or a state. For example, under the Clean Air Act, EPA, along with state and local government units and other entities, regulates air emissions of various substances that harm human health. These laws also authorize the issuance of administrative orders, among other things, to require cleanup of contamination. NEPA. Under NEPA, federal agencies must evaluate the environmental impacts of their proposed major federal actions using an environmental assessment or a more detailed environmental impact statement, with some exceptions. Civil Rights Act of 1964. Title VI of the Civil Rights Act of 1964, as amended, prohibits discrimination based on race, color, or national origin in programs or activities that receive federal financial assistance. To carry out and enforce the provisions of the act, federal agencies have developed programs to receive and investigate allegations of discriminatory actions taken by recipients of federal funding. Most working group member agencies reported planning and implementing some actions to identify and address environmental justice issues. Some examples of key activities include the following: EPA mapping tool. In 2015, EPA released its Environmental Justice Mapping and Screening Tool (EJSCREEN), a web-based mapping tool that includes environmental and demographic data at a local level. Users can identify potential exposure to environmental pollutants and related health risks across different communities. Officials from the Department of Justice told us they regularly use EJSCREEN to help determine whether cases involve environmental justice issues. Incorporating environmental justice in NEPA analyses. At least 13 agencies provided examples of efforts to consider environmental justice in their NEPA analyses. At the Department of the Interior (DOI), departmental policy requires all bureaus to include consideration of environmental justice in the NEPA process, and some bureaus have developed their own guidance for doing so. For example, DOI’s 2015 National Park Service NEPA Handbook requires that the agency’s environmental analyses discuss and evaluate the impact of proposals on minority and low-income populations and communities. The Department of Homeland Security also issued an agency-wide directive on NEPA implementation in 2014, and the accompanying 2014 NEPA instruction manual included public involvement requirements for populations with environmental justice issues. Data initiative and reports on chemical exposure. At the Department of Health and Human Services (HHS), the Centers for Disease Control and Prevention (CDC) built a National Environmental Public Health Tracking Network, which brings together health and environmental data from national, state, and city sources. The CDC also developed a National Report on Human Exposure to Environmental Chemicals—a series of reports that uses biomonitoring to assess the U.S. population’s exposure to environmental chemicals. As we reported in September 2019, for fiscal years 2015 through 2018, 11 of the 16 member agencies of the working group reported supporting environmental justice efforts through existing related program funding and staffing resources (i.e., resources not specifically dedicated to environmental justice, such as for civil rights or environmental programs). EPA and the Department of Energy (DOE) dedicated resources specifically for environmental justice efforts in their budgets. In fiscal year 2018, EPA provided about $6.7 million and DOE provided about $1.6 million. Agencies’ progress in identifying and addressing environmental justice issues related to their missions is difficult to gauge because most of the agencies do not have updated strategic plans and have not reported annually on their progress or developed methods to assess progress. As we reported in September 2019, 14 of the 16 agencies issued environmental justice strategic plans after they signed the 2011 MOU agreeing to develop or update such plans. Of the 14 agencies that issued their plans, 12 established strategic goals in these plans. Six of the 14 agencies further updated their plans in 2016 or 2017, and another agency published updated priority areas on its website. The Department of Defense (DOD), which issued a plan in 1995, has not updated it since, and the Small Business Administration (SBA) has never issued a plan. DOD officials said that the agency has not prioritized environmental justice efforts. SBA officials said the agency is uncertain whether it has a role in implementing environmental justice and they were reviewing whether SBA should continue its membership in the working group. The 2011 MOU directs agencies to update their strategic plans periodically, and leading practices for strategic planning suggest that strategic plans should be updated every 4 years. We have previously reported that strategic planning serves as the starting point and foundation for defining what an agency seeks to accomplish, identifying the strategies it will use to achieve desired results, and then determining how well it succeeds in achieving goals and objectives. In our September 2019 report, we recommended that eight agencies update their environmental justice strategic plans. Four agencies agreed, three did not state if they agreed or disagreed, and one disagreed. Education stated that it does not believe this is the most appropriate course of action for the department or an efficient use of resources, but we continue to believe they should implement the recommendation. As we reported in September 2019, 12 of the 16 agencies developed environmental justice strategic plans with strategic goals, but most of the agencies have not shown clear progress toward achieving these goals and the purpose of the executive order. It is difficult to gauge the agencies’ progress for three primary reasons: 1. The agencies have not comprehensively assessed how environmental justice fits with their overall missions. Seven of the 14 agencies that developed environmental justice strategic plans assessed and discussed how their environmental justice efforts aligned with their overall missions after 2011. However, the other seven agencies did not clearly show how their efforts aligned with their missions. We recommended that EPA, as chair of the working group, should develop guidance for the agencies on what they should include in their environmental justice strategic plans. EPA agreed with this recommendation. 2. The agencies have not consistently issued annual progress reports. Fourteen agencies issued at least one progress report after 2011, but most have not issued such reports every year, as they agreed to do in the 2011 MOU. The departments of Homeland Security and Justice issued progress reports every year from 2012 through 2017. The General Services Administration issued progress reports every year through 2015 and then issued one progress report covering fiscal years 2016 through 2018. Several other agencies consistently reported in the first few years after 2011 but then stopped issuing reports. DOD and SBA have not issued any progress reports. We have found that annual program performance reports can provide essential information needed to assess federal agencies’ performance and hold agencies accountable for achieving results. We recommended that 11 agencies report on their progress annually. Five of the agencies agreed with this recommendation, one partially agreed, three did not state if they agreed or disagreed, and two said they did not agree. Education stated that it does not believe this is the most appropriate course of action for the department or an efficient use of resources, and DOD stated that it did not see a tangible benefit to additional reporting. We continue to believe that they should implement the recommendation. 3. Most agencies have not established methods for assessing progress toward goals. The agencies’ progress reports generally describe the environmental justice activities they conducted but do not include any methods to assess progress (e.g., performance measures). For the 14 agencies that issued at least one progress report since 2011, we reviewed the most recent report and found that each report contained information on activities that agency undertook over the previous year. However, our analysis showed that most of the agencies had not established a method that would allow them to assess their progress toward their environmental justice goals, such as tracking performance measures or milestones. Of the 16 agencies that signed the 2011 MOU, four—the Departments of Agriculture (USDA), Health and Human Services (HHS), and DOI and EPA—have established performance measures or milestones for their environmental justice efforts. Of these four, HHS and EPA have reported on their progress toward achieving their performance measures or milestones. The other 12 agencies have not established any performance measures or milestones. The executive order directs the working group to provide guidance to agencies in developing their environmental justice strategies. However, the working group has not provided such guidance on methods to assess and report on environmental justice progress, according to EPA officials. According to these officials, EPA is still pursuing its own agency-wide performance measures. We recommended that EPA, as chair of the working group, develop guidance or create a committee of the working group to develop guidance on methods the agencies could use to assess progress toward their environmental goals. EPA agreed with this recommendation. We found that the interagency working group has coordinated to some extent but does not have a strategic focus or full participation by all the federal agencies. Executive Order 12898 directed the working group to coordinate in seven functions, including to assist in coordinating data collection and examine existing data and studies on environmental justice. In 2016, the working group released its Framework for Collaboration, which describes how it planned to provide guidance, leadership, and support to federal agencies in carrying out environmental justice efforts. The working group has collaborated to develop and issue guidance on several topics, participated in a variety of public meetings to provide information and opportunities for communities to discuss environmental justice issues, and coordinated ways in which the 16 member agencies and the Council on Environmental Quality (CEQ) could assist communities. For example, the working group created nine committees, including on Native American and Indigenous Peoples, Rural Communities, and Climate Change, based on the seven functions in the executive order and on public input, to help carry out its environmental justice responsibilities under the executive order. Officials from 13 member agencies agreed to either chair or become members of one or more committees. Through these committees, among other things, the working group has released a number of documents to help guide federal efforts: A compendium on publicly available federal resources to assist communities impacted by goods movement activities, released in 2017. Guidance to help federal agencies incorporate environmental justice during their NEPA reviews, issued in March 2016, and guidance to communities about NEPA methods, issued in March 2019. A web page, which USDA compiled and launched in fiscal year 2017 with input and vetting from the Rural Communities committee, that provides links to community tools, funding opportunities, educational and training assistance, and case studies to support rural communities, according to USDA officials. However, we found that the working group’s organizational documents— the 2011 MOU, the working group’s 2011 charter, and the 2016-2018 Framework for Collaboration—do not provide strategic goals with clear direction for the committees to carry out the functions as laid out in the executive order. In September 2012, based on a government-wide study, we reported that collaborative mechanisms such as working groups benefit from clear goals to establish organizational outcomes and accountability. We reported that participants may not have the same overall interests or may even have conflicting interests, but by establishing a goal based on common interests, a collaborative group can shape its own vision and define its purpose. The working group has developed some documents with agreed-upon goals, which is beneficial to collaboration, but none of these documents address all seven functions of the executive order. In our September 2019 report, we compared the functions of the executive order to documented working group roles and responsibilities and found that coordinated data collection and examination of research and studies on environmental justice are not included in these documents or committee purposes and have not been a focus of the interagency working group since at least 2011. EPA officials said some agencies, such as HHS and EPA, have done work in environmental justice data collection and research. EPA officials told us that the 2011 MOU, committee groups, and Framework for Collaboration reflect the current priorities of the working group, based on public input. The officials were unsure whether a coordinated effort in the data collection, research, and studies areas was needed, but they said such an effort could be useful. They said that the most useful role of the working group in research might be as a forum for sharing information and providing training opportunities. In our September 2019 report, we recommended that EPA, as chair of the working group and in consultation with the working group, should clearly establish in its organizational documents strategic goals for the federal government’s efforts to carry out the 1994 executive order. EPA disagreed with this recommendation because it believes that the recommendation should be combined with a different recommendation we made about updating the MOU. We believe that EPA misunderstood our recommendation and therefore did not combine it with our other recommendation. We also found that member agencies’ participation in working group activities has been mixed. In the 2011 MOU, the 16 signing agencies and CEQ agreed to participate as members of the working group, such as by chairing, co-chairing, or participating in committees. Eleven of the 16 agencies have not chaired or co-chaired one of the working group’s committees, and four have not participated in any. Our government-wide work has shown that it is important to ensure the relevant participants have been included in a collaborative effort. EPA officials said it is difficult to characterize what specific opportunities are missed because of an agency’s lack of representation. However, they said that nonparticipation limits the working group’s ability to fulfill its mandates in a strategic, methodical way across the entire federal government. EPA officials also said that the limiting factor in the working group’s efforts to address the executive order has always been the will of leadership across the federal government to make clear, measurable commitments to those priorities and ensure adequate resources. We recommended that EPA, as chair of the working group and in consultation with the other working group members, update the 2011 MOU and renew the agencies’ commitments to participate in the interagency collaborative effort and the working group. EPA disagreed and said this recommendation could be combined with the recommendation to provide strategic direction for the working group. We continue to believe that EPA needs to update the MOU to address the matter of participation by the members who signed it but do not participate. In conclusion, incorporating environmental justice into federal agencies’ policies, programs, and activities is a long-term and wide-ranging effort. Federal agencies, led by EPA, have made some headway in developing tools and coordinated policies and have identified others that they need to pursue. Strategic planning and reporting, with meaningful measures, and collaboration across all agencies can help them make and track progress. Chairman Tonko, Ranking Member Shimkus, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Alfredo Gómez, Director, Natural Resources and Environment, at (202) 512-3841or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contact named above, Susan Iott (Assistant Director), Allen Chan (Analyst in Charge), and Elise Vaughan Winfrey made key contributions to the testimony. Other staff who made contributions to this testimony or the report cited in the testimony were Peter Beck, Tara Congdon, Hannah Dodd, Juan Garay, Cindy Gilbert, Rich Johnson, Matthew Levie, Ben Licht, Cynthia Norris, Amber Sinclair, and Kiki Theodoropoulos. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Environmental justice seeks to address the disproportionately high distribution of health and environmental risks among low-income and minority communities by seeking their fair treatment and meaningful involvement in environmental policy. In 1994, Executive Order 12898 directed 11 federal agencies to incorporate environmental justice into their programs, policies, and activities. The executive order also directed the agencies to each establish an environmental justice strategy and created a working group of federal agencies, chaired by EPA, to coordinate federal environmental justice efforts. In 2011, these 11 agencies and five additional federal agencies signed a MOU agreeing to participate in federal efforts in this area as members of the Interagency Working Group on Environmental Justice and to issue annual progress reports on their efforts. This statement summarizes GAO’s findings from its September 2019 report on federal environmental justice efforts (GAO-19-543). Specifically, it focuses on (1) actions the working group agencies have taken to address environmental justice issues related to their missions, (2) the agencies’ progress in identifying and addressing environmental justice issues related to their missions, and (3) interagency working group efforts to help agencies coordinate federal environmental justice efforts under the executive order. To perform this work, GAO reviewed agency environmental justice plans, reports, and funding data; interviewed agency officials; and compared working group collaboration to leading collaborative practices. As GAO reported in September 2019, most of the 16 member agencies of the Interagency Working Group on Environmental Justice reported planning and implementing some actions to identify and address environmental justice issues, such as creating data tools, developing policies or guidance, and building community capacity through small grants and training. For example, the Environmental Protection Agency (EPA) created a mapping tool that can help identify low-income and minority communities exposed to health or environmental risks. Most of the agencies supported their efforts with funds and staff from related programs, but EPA and the Department of Energy provided funds (totaling $8.3 million in fiscal year 2018) and staff specifically for environmental justice. Agencies’ progress in identifying and addressing environmental justice issues related to their missions is difficult to gauge. Most of the agencies do not have updated strategic plans and have not reported annually on their progress or developed methods to assess progress, as they agreed to do by signing a 2011 memorandum of understanding (MOU). Of the 16 agencies that signed the MOU, 14 have issued strategic plans. However, although the MOU directs the agencies to update their strategic plans periodically, only six of these 14 agencies have done so since 2011. Furthermore, most of these 14 agencies have not consistently issued annual progress reports. In September 2019, GAO recommended that nine agencies develop or update their strategic plans and that 11 develop annual progress reports. Eight agencies agreed and one partially agreed, one agreed with one recommendation but disagreed with another, one disagreed, and three did not state if they agreed or disagreed. GAO also found that while four agencies, including EPA, have established performance measures or milestones for assessing progress toward goals, the other 12 have not done so. Agency officials said guidance from the working group on how to do so would be helpful. The 1994 executive order directs the working group to provide guidance to agencies in developing their environmental justice strategies, but the group has not provided specific guidance on what agencies should include in their strategic plans or on methods to assess and report on environmental justice progress. In September 2019, GAO recommended EPA develop such guidance or create a working group committee to do so, and EPA agreed. The interagency working group has coordinated to some extent but does not have a strategic approach, and member agencies are not fully participating. Specifically, the group’s organizational documents do not provide strategic goals with clear direction for the committees. Furthermore, 11 of the 16 signatory agencies have not chaired or co-chaired one of the committees, and four have not participated in any. In September 2019, GAO recommended EPA update the 2011 MOU and clearly establish strategic goals for federal efforts to carry out the executive order. EPA disagreed, but GAO continues to believe these actions are necessary. GAO made 24 recommendations in its September 2019 report, including that agencies update their environmental justice strategic plans and report on their progress annually. GAO recommended that EPA, as chair of the working group, develop guidance on assessing progress and what agencies should include in their strategic plans; coordinate with working group members to develop strategic goals for the group; and update the group’s memorandum of understanding. Of the 15 agencies with recommendations, eight agreed. Other agencies partially agreed, disagreed, or had no comment. GAO continues to support its recommendations.", "document_type": "gao"}
{"report": "In September 2003, the United States and other nations endorsed IAEA’s Code of Conduct, which established basic principles and guidance to promote the safe and secure use of radioactive material. The Code of Conduct applies to category 1, 2, and 3 quantities of radioactive material—all of which are potentially dangerous to human health and could, if not properly controlled, cause permanent injuries or death to a person who handled or was otherwise in contact with them. IAEA’s system considers radioactive material dangerous when gathered in close proximity to people in sufficient quantity and for a sufficient time to cause direct human health effects. NRC, working with the Department of Energy, developed a list of 16 radionuclides of concern that, if gathered in category 1 or 2 quantities, pose the greatest risk of being used by terrorists to make an RDD. Of these 16 radionuclides of concern, 4 are most prevalent in the U.S. economy: americium-241, cobalt-60, cesium- 137, and iridium-192. Since the terrorist attacks in September 2001, concerns have grown that terrorists could obtain and use radioactive material and build an RDD. The risk of an RDD is determined by the function of three components: threat, vulnerability, and consequence. Threat is generally defined as entities or actions with the potential to cause harm—including terrorist attacks. According to NRC officials, there is a general credible threat of malevolent use of radioactive materials. The second component of RDD risk, vulnerability, includes physical features or operational attributes that render an asset open to exploitation, including gaps in security measures such as gates, locks, perimeter fences, and computer networks. Finally, the third component of RDD risk, consequence, includes the effects of terrorist attacks or natural disasters that result in losses to public health and safety and the economy. Taken together, the three components make up a “risk triplet,” which is shown in figure 1. The consequences of detonating an RDD would depend on the quantity and type of radioactive material used and the size and characteristics of the area in which the material was dispersed. An example of the consequence of an RDD occurred in 1987 when two people in Goiânia, Brazil, found an abandoned medical machine containing 1,400 curies of cesium-137. The individuals, who were unaware of the nature of the radioactive material, extracted it from the machine and distributed the material to several families, causing 20 people to be hospitalized and four deaths. The very high internal and external contamination was caused by the way they handled the cesium-137, including rubbing their skin with the material and eating with contaminated hands. In addition, 112,000 people in the surrounding area were monitored for exposure to radiation, of which 249 were found to be internally or externally contaminated. The accident also contaminated 85 houses and required the demolition of homes and other buildings, generating 3,500 cubic meters of radioactive waste. This example shows the range of consequences from the dispersal of radioactive material, from fatalities to socioeconomic effects. Potential consequences of an RDD are outlined in figure 2. Depending on the size and radioactivity of an RDD, the affected population could be evacuated and possibly relocated. EPA’s Protective Action Guide (PAG) presents radiation dose guidelines that are used by federal agencies to protect people from unhealthy levels of radiation. According to the PAG, evacuation is recommended when there is enough radiation to reach 5.0 rem over the first 4 days. The PAG also outlines actions that can be taken in response to projected radiation dose rates, including evacuation, shelter in place, relocation, and avoidance of drinking water or food supplies. The PAG does not consider a specific geographic area, such as a square kilometer, when recommending evacuation. Domestically, the Atomic Energy Act of 1954, as amended, gives NRC primary responsibility for regulating most domestic industrial, medical, and research uses of radioactive material to protect public health and safety, among other things. NRC is composed of five Commissioners (the Commission) appointed by the President and confirmed by the Senate for 5-year terms. One of the Commissioners is designated by the President to be the Chairman and official spokesperson of the Commission. According to NRC’s website, the Commission formulates policies, develops regulations governing nuclear reactor and nuclear and radioactive material safety, issues orders to licensees, and adjudicates legal matters. Issues before the Commission are decided by majority vote, and the Commission directs subsequent actions be implemented by NRC staff. When establishing security requirements for radioactive material, since 2004, NRC has assessed the risks of such material based on the potential of that material to cause prompt fatalities and the deterministic health effects from its radiation; it has not used socioeconomic consequences as a basis for establishing regulations related to the security of radioactive material. Moreover, in response to the recommendations we made in 2016 that NRC should better track category 3 quantities of radioactive material, NRC staff assessed whether they should require additional security measures for category 3 radioactive material and determined that such material did not merit additional security measures. Since 2004, NRC has assessed the risks of radioactive material based on the potential of that material to cause prompt fatalities and deterministic health effects from radiation. NRC on several occasions reassessed and repeatedly reaffirmed its use of the occurrence of prompt fatalities and deterministic health effects as its primary criteria for measuring the consequences of an RDD, including when developing its decision-making framework in 2004, reviewing its regulatory framework after the Fukushima nuclear disaster in 2011, and in its 2014 response to recommendations from the Radiation Source Protection and Security Task Force (the Task Force). NRC first considered prompt fatalities from radiation as criteria for measuring consequences in November 2004 when developing its decision-making framework for evaluating vulnerabilities for theft of radioactive material. Specifically, in 2004 NRC staff recommended that the Commission approve a decision-making framework that assessed risk based on prompt fatalities from radiation. In the Commission Paper, NRC staff stated that the framework would employ the consequence criteria of preventing prompt fatalities from radiation exposure, but they also recognized that including additional consequence criteria, such as land contamination, might be warranted. They also pointed out that DHS’s Risk Analysis and Management for Criteria Asset Protection framework used criteria including economic, environmental, and loss of output of production capability, among other things. In January 2005, the Commission approved using prompt fatalities from radiation for measuring consequence. In its decision, the Commission also said that NRC staff should not independently develop criteria and standards for other consequences, such as land contamination and economic impacts. After the Fukushima nuclear disaster in 2011, NRC staff again considered broadening the criteria for assessing risk to include socioeconomic impacts. Specifically, in an August 2012 analysis presented to NRC commissioners in response to Fukushima that addressed whether NRC’s regulatory framework should be modified to consider economic consequences, NRC staff noted that NRC’s existing requirements have the effect of minimizing economic consequences by preventing or mitigating events that could lead to a radioactive release. The analysis prepared by NRC staff recommended improving guidance for estimating offsite economic costs based on up-to-date data. In March 2013, the Commission approved the staff’s recommendation to provide enhanced guidance but found that socioeconomic consequences should not be considered. at 2 rem for the first year. socioeconomic consequences to the Commission in January 2014, reiterating the staff’s view that Part 37 provides adequate security protection against a significant RDD. The NRC staff also concluded that the current protection and security framework and posture adequately protects against contamination and resulting economic consequences. In 2016, NRC established the Category 3 Source Security and Accountability Working Group (the Working Group) in response to our 2016 recommendations to NRC to better track dangerous quantities of radioactive material. This group issued a report in 2017 assessing whether NRC should require additional security measures for category 3 material and determined that such material did not meet the threshold of prompt fatalities and deterministic health effects set by NRC, and therefore, did not require additional security measures. As part of its analysis, the Working Group stated that a category 2 quantity of a certain radioactive material would not be sufficient to achieve an RDD of consequence that would cause deterministic health effects. NRC officials also told us that there is not enough of this same radioactive material in the United States to create an RDD of consequence even if all of it was used in an RDD. The Working Group also concluded that there is no evidence of adversarial interest in acquiring category 3 quantities of material by theft, that security weaknesses at facilities that contain category 3 quantities of radioactive material had not increased since first evaluated by NRC, and the consequences of an RDD using category 3 material are not significant enough to require additional security measures. Based on the findings of the working group’s report, NRC staff recommended that the Commission not amend regulations to require license verification of category 3 radioactive material or impose security requirements to prevent the aggregating of category 3 material to a category 2 quantity. The report did recommend that the Commission approve the pursuit of rulemaking to require safety and security equipment be in place before granting a license for an unknown entity and clarify license verification methods for transfers involving quantities of radioactive material below the category 2 threshold. The experts we convened with assistance from the National Academies generally agreed that NRC’s assessment of risk does not include the all relevant criteria for establishing security requirements. The experts at our meeting generally agreed that prompt fatalities from radiation and deterministic health effects are not the only relevant criteria for determining the consequences of an RDD, which recent studies we reviewed support. These experts and studies generally agreed that socioeconomic effects and fatalities from subsequent evacuations are relevant criteria for assessing the consequences of an RDD. The experts at our meeting generally agreed that using prompt fatalities and deterministic health effects from radiation as the basis for analyzing consequence have limited value to NRC as criteria for determining the consequences of an RDD, as they are unlikely to occur in the event of an RDD. Experts generally said at our meeting expressed the opinion that NRC is not focusing on all relevant criteria for assessing consequence. For example, one expert from the regulatory community said that prompt fatalities are an unlikely consequence of an RDD. Another expert affiliated with users of radioactive material noted that deterministic health effects from an RDD are limited. Finally, a security expert said that it would be difficult to kill large numbers of people with an RDD, and therefore prompt fatalities are not a good measure of consequence. Another expert pointed out that NRC’s current criteria would be unlikely to support regulating category 1 and 2 materials since an RDD with these materials is unlikely to cause prompt fatalities. He added that this creates a disconnect where category 3 material is ignored, but NRC regulates category 1 and 2 material even though category 1 and 2 materials do not meet NRC’s criteria of causing prompt fatalities and deterministic health effects. Recent studies from Sandia also show that prompt fatalities and deterministic health effects are unlikely to result from an RDD. Specifically, Sandia completed two studies in 2017 and 2018 that modeled an RDD blast and evaluated the potential consequences in New York City. The 2017 study modeled the potential consequences of a category 1 quantity of radioactive material detonated in an RDD and estimated that there would likely be no prompt fatalities from radiation. The 2018 study undertook the same analysis with a category 3 quantity of radioactive material and estimated that it would also produce no prompt fatalities from radiation. The experts who participated in our meeting discussed what type of consequences should be considered and generally agreed that socioeconomic effects and fatalities from subsequent evacuations, rather than prompt fatalities and deterministic health effects, are relevant criteria for NRC to consider when assessing the consequences of an RDD, which recent studies we reviewed support. For example, one expert said that while deterministic health effects from an RDD are limited, socioeconomic impacts are significant. Another expert said that the main point of a terrorist detonating an RDD is to create economic effects, not deterministic health effects. This expert added that the dispersal of radioactive material would result in low-level radiation scattered across an area, leading to socioeconomic consequences. A participating expert from the regulatory community said that it is difficult to quantify socioeconomic effects. Furthermore, the expert said that any model used to determine regulation by predicting consequence must be reproducible. The federal government has recently taken steps to better understand the socioeconomic costs associated with an RDD. For example, Sandia studies completed in 2017 and 2018 estimated socioeconomic costs for RDDs with category 1 and category 3 quantities of radioactive material. The 2017 study that modeled a category 1 quantity of radioactive material estimated that the socioeconomic impact on the national gross domestic product would be approximately $30 billion. The 2018 study, which substituted a category 3 quantity of radioactive material, estimated the socioeconomic impact on gross domestic product at $24 billion. One expert noted that the estimates may be understated. Specifically, the 2017 Sandia study took into account that the facades of some buildings in New York City could be replaced, which would aid cleanup and reduce socioeconomic costs. However, one expert who attended our meeting said that New York City may be a best-case example of an urban target because the city has solid response plans and modern buildings with facades that can be removed more easily than those in other cities. This expert said these factors likely lead to an optimistic calculation of socioeconomic consequence in the study, due to the preparation and resilience posture of New York City, creating a best-case scenario regarding cleanup that may not accurately quantify costs in other cities. In this expert’s view, the federal government may also be underestimating the economic consequences of an RDD by not accounting for the potential that local cleanup standards may be more stringent than the federal government standards assumed in the study. The expert said that locals will always want to clean up to a higher standard than federal government guidance recommends, largely due to a desire to protect economic assets such as trade, brand, and image. In addition to socioeconomic concerns, experts who attended our meeting generally noted that an assessment of the consequence of an RDD should consider fatalities resulting from the evacuation of homes and business. For example, one expert from our meeting said that there were few deaths from radiation during the incident at the Fukushima nuclear complex in 2011, but there were many deaths from the evacuation. Another expert agreed and said that there is evidence from Chernobyl and Fukushima linking health effects to evacuations and that, therefore, fatalities from evacuations should be included on the list of consequences from an RDD. A third expert said that panic cannot be underestimated in the event of an RDD, and the consequences of evacuation and relocation would exceed prompt fatalities and deterministic health effects. Finally, one expert said that many people outside of the evacuation area will also choose to relocate after an RDD rather than wait for direction from the government, which could increase the number of evacuees and lead to additional fatalities. The 2017 and 2018 Sandia studies support these concerns, estimating that these evacuations could cause hundreds to thousands of deaths and that fatalities during evacuations are similar for RDDs using category 1 and category 3 quantities of the same material. Specifically, the 2017 Sandia study examined the number of fatalities that occurred during the evacuation from the disaster at the Fukushima nuclear complex. Using that event as a baseline, the Sandia study estimated that approximately 1,500 people could die from the evacuation associated with the detonation of an RDD containing a category 1 quantity of radioactive material in New York City. The 2018 Sandia study of a detonation of an RDD containing a category 3 quantity of radioactive material estimated that approximately 800 people could die from the evacuation. NRC does not consider socioeconomic consequences or fatalities from evacuations when assessing the consequence of an RDD. Agency officials told us that, under the authority of the Atomic Energy Act, NRC staff has discretion to consider other criteria, including socioeconomic effects, if so directed by the Commission. However, NRC staff told us that they do not currently consider socioeconomic consequences as criteria because they have been specifically directed not to do so by the Commission. In discussions with agency officials, it is unclear why the Commission has directed the NRC staff not to consider other criteria for evaluating the impact of an RDD. NRC’s own guidance states that RDDs would cause few deaths from radiation but result in significant socioeconomic impacts. Specifically, NRC guidance issued in May 2014 states: “RDDs are considered weapons of mass disruption; few deaths would occur due to the radioactive nature of the event; however, significant social and socioeconomic impacts could result from public panic, decontamination costs, and the denial of access to infrastructure and property for extended periods of time.” NRC’s decision to not consider other criteria has limited its assessments of risk presented by the use of radioactive material in an RDD. By considering socioeconomic impacts and fatalities resulting from evacuations in its criteria, NRC would have better assurance that it was considering the more likely and more significant consequences of an RDD when establishing its security requirements for this material. In 2016, NRC evaluated the effectiveness of Part 37, as required by Public Law 113-235, and concluded that the rule is effective for ensuring category 1 and 2 radioactive materials are secure from theft or diversion. However, experts who attended our meeting stated, and recent studies support, that if category 3 quantities of radioactive materials were used in an RDD, the consequences could be comparable to a category 1 or 2 quantity of the same material, which are protected from theft by additional security measures. In addition, experts who participated in our meeting generally said that NRC’s current requirements permit collocation at the same facility of multiple category 3 quantities of americium-241 that in total reach or surpass the threshold for a category 2 quantity without the enhanced security required for category 1 and 2 materials. Furthermore, experts generally agreed that there are security weaknesses in the current trustworthiness and reliability process to protect against an insider threat. In December 2016, NRC issued a report evaluating the effectiveness of Part 37, as required by Public Law 113-235. NRC’s evaluation included an analysis of events and inspection findings related to the security of category 1 and 2 materials, including an analysis of 189 violations issued to NRC State licensees from March 2014 through March 2016. The report found that almost all of the violations were related to conducting background investigations, controlling access to radioactive material, and physical security measures. The violations mainly occurred when licensees had not yet implemented Part 37 or failed to fully document how their security program complied with Part 37. The report noted that there were no Severity Level I or Severity Level II violations. The NRC report looked at the theft of six category 2 quantities of radioactive material since the introduction of the Increased Controls security requirements in 2003 and concluded that carelessness or human error, rather than any gaps in the requirements of Part 37, contributed to the thefts. As we reported in 2014, the thefts included industrial radiography cameras with category 2 quantities of iridium-192 sources stolen from radiography trucks parked outside a company facility, in hotel parking lots, and at a gas station. NRC concluded that in all the events, carelessness or human error contributed to the thefts and had the licensees followed existing regulatory requirements, the thefts could have been prevented. NRC’s 2016 report concluded that better outreach and communication would help improve compliance with Part 37. NRC’s report also documents NRC staff’s determination that the requirements in Part 37 are effective in preventing the theft or diversion of category 1 and 2 quantities of radioactive material. NRC determined that potential rule clarifications and guidance initiatives could help to enhance the clarity and effectiveness of the rule, ensure better understanding of security expectations, and allow for more complete and adequate implementation. NRC’s overall assessment is that Part 37 provides reasonable assurance for the security of category 1 and 2 quantities of radioactive material by protecting the material from theft or diversion. In conducting its evaluation, NRC examined past security incidents and inspection reports, but its report did not review the security requirements for category 3, 4, or 5 quantities of radioactive material because NRC does not consider these categories to be a significant risk. NRC chose to define high-risk radioactive material as only category 1 and 2. NRC does not further elaborate why it took this approach. NRC’s reliance on prompt fatalities and deterministic health effects and its exclusion of socioeconomic consequences and deaths from evacuations as criteria for determining the consequences of an RDD, as discussed earlier, has resulted in security requirements that do not include all high- risk quantities of some radioactive materials. Experts who participated in our meeting generally agreed that some category 3 quantities of radioactive material should be considered high risk based on their potential consequences if used in an RDD. For example, one international expert pointed out that IAEA guidance includes security measures for category 3 quantities of material and expressed surprise that U.S. guidelines do not include additional security measures for category 3 quantities. Another expert suggested that NRC include category 3 quantities of radioactive material in the National Source Tracking System, which would allow for license verification during purchases. In this expert’s opinion, the main vulnerability for category 3 quantities of radioactive material is that they can be purchased with a license that has not been verified as legitimate by the NRC or an agreement state. The experts also generally said that some category 3 radioactive material should be considered high risk and should be subject to additional security measures. For example, one expert suggested that some types of category 3 radioactive material may need additional oversight. However, this expert said that NRC should consider a more nuanced approach to increasing the security for some, but not all, quantities of category 3 radioactive material. Another expert agreed and said that the ability to disperse material is a primary factor in determining if something is high risk. For this reason, this expert said, category 3 quantities of some types of radioactive material should be considered high risk, and there may be need for an additional category of materials that falls below category 2 but that includes the most dangerous high-risk materials in category 3 quantities. An expert who attended our meeting stated that certain radioactive materials pose a unique decontamination challenge because those materials bind to materials like asphalt and concrete, making decontamination difficult and expensive. One expert said that the consequences listed in the 2018 Sandia report were enough to justify requiring additional security measures for category 3 quantities of certain radioactive materials. As shown in table 1, the 2018 Sandia study found that a category 3 quantity of radioactive material could result in socioeconomic consequences and fatalities from evacuations similar to an RDD with a category 1 quantity of radioactive material. The experts also generally said that there could be long-term socioeconomic consequences unique to the risk posed by an RDD that used a category 3 quantity of radioactive material, and certain radioactive materials in smaller quantities should be considered high risk. As we described earlier, NRC reported in a 2017 Threat, Consequence, and Vulnerability Assessment that even if several hundred category 3 quantities of a certain radioactive material were used in an RDD, it would not create an RDD of consequence. In our discussions with NRC staff, they expanded on this point and stated that there may not be enough of this material in the United States to build an RDD of consequence. However, new research from Sandia found that a category 3 quantity of the same material could trigger an evacuation and result in significant socioeconomic consequences. According to an expert from the regulatory community, while the Commission has considered requiring additional security measures for category 3 quantities of material, NRC staff recommended against doing so because the costs of providing additional security would outweigh the benefits. For example, one expert who attended our meeting said that the choice is between the difference in costs of absolute security and adequate security, and the cost/benefit analysis does not support including category 3 quantities of radioactive materials in Part 37. The expert pointed out that there have been relatively few thefts of category 3 sources in the United States and suggested that providing additional security should be weighed against the low likelihood that the radioactive materials would be stolen. While there were differing views in our expert meeting between the regulatory community and other experts, the experts generally agreed, and the Sandia studies support, that the consequences of category 3 quantities of certain types of material could be significant. By requiring additional security measures for these high-risk quantities of category 3 material, and assessing whether other category 3 radioactive materials should also be safeguarded with additional security measures, NRC could have better assurance that its requirements are sufficient to help ensure all high-risk radioactive material is protected from theft and use in an RDD. NRC’s 2016 report looked at the risks posed by the collocation of category 3 quantities of material and insider threats. NRC concluded that rule clarifications and additional guidance could help enhance clarity and effectives of the rule, but its report does not fully address how these risks should be managed. For example, experts who participated in our meeting generally agreed that weaknesses continue to exist in how Part 37 regulates the collocation of multiple category 3 quantities of americium-241 at a single facility. Specifically, NRC requirements permit collocation of multiple category 3 quantities of material that in total reach a category 1 or 2 quantity of material, without applying Part 37. Experts told us that well logging companies, which use americium-241 to inspect wells for oil and natural gas, are storing multiple category 3 quantities, each just below the threshold for category 2, of americium-241 at the same facility; thus, the total quantity does not trigger additional security requirements under Part 37. Figure 3 shows a well logging storage facility containing multiple category 3 quantities of americium-241. Experts at our meeting generally said that collocation of multiple quantities of category 3 americium-241 at well logging facilities creates specific security weaknesses that should be addressed. For example, one expert who attended our meeting from the regulatory community said that NRC has no formal definition for collocation, but Part 37 considers it acceptable to store multiple category 3 quantities of radioactive material in separate, locked containers that, together, add up to a category 2 quantity. Another expert pointed out that when NNSA evaluates threats to materials, it totals up the quantity of materials located at the same facility to determine the total amount of material at risk. A third expert noted that licensees are required to inventory category 3 quantities of material only twice per year. Furthermore, the experts pointed out that these types of facilities are not subject to stricter security requirements, and therefore, do not undertake trustworthiness and reliability evaluations for their employees with unescorted access to radioactive material. By requiring that all licensees implement additional security measures when they collocate multiple quantities of category 3 americium-241—that in total reach a category 1 or 2 quantity—at a single facility, NRC could have better assurance that the material is protected from theft and use in an RDD. Furthermore, experts who participated in our meeting generally agreed that there continue to be security weaknesses in the current trustworthiness and reliability process for securing radioactive material from theft and use in an RDD. For example, one expert from the licensee community who attended our meeting said that NRC’s Part 37 does not go far enough in ensuring the trustworthiness and reliability of individuals given unescorted access. Specifically, the expert said that, based on the Part 37 requirements, licensees make all trustworthiness and reliability determinations for granting unescorted access to employees, which leads to inconsistencies across licensees. The experts generally said that NRC should give licensees more guidance on acceptable criteria for granting unescorted access, which is consistent with recommendations included in past GAO reports. NRC is currently in the process of making revisions to its trustworthiness and reliability guidance. Experts who attended our meeting said that licensees face challenges in making trustworthiness and reliability determinations, including the fear of being sued if they deny employment to an individual with a criminal record, difficulty conducting background investigations for foreign nationals, and the potential for individuals to be radicalized more quickly than the current trustworthiness and reliability process protects against. One expert from the regulatory community who attended our meeting said that trustworthiness and reliability decisions are a “judgment call,” and when an applicant has a criminal record or has committed a felony, a company may not want to give them unescorted access to radioactive material. However, the expert added that denial of unescorted access without backup from NRC guidance may leave the company open to lawsuits. In addition, another expert who attended our meeting said institutions that often employ foreign nationals as researchers, such as hospitals, struggle with verifying limited background information for these individuals. Finally, an expert who attended our meeting said that perception of trustworthiness and reliability has recently changed, and there is now greater concern that people can be radicalized quickly, rendering background investigations insufficient to identify potential issues with an employee’s trustworthiness and reliability during their employment. The expert told the group that there is evidence that individuals can be radicalized in a matter of months. The expert said that current trustworthiness and reliability procedures should take into account that people’s beliefs can change rapidly. Radioactive material is used in thousands of locations throughout the United States for medical, industrial, and research purposes. On several occasions over the past 20 years, NRC has examined and revised the security requirements for these materials in order to prevent terrorists from acquiring radioactive material and constructing an RDD, or “dirty bomb.” When assessing the risk posed by an RDD, NRC has repeatedly looked at different criteria for measuring consequences and chose to base its decisions primarily on preventing prompt fatalities and deterministic health effects from radiation. However, the experts who participated in our meeting generally agreed, and Sandia studies support, that socioeconomic effects and fatalities from subsequent evacuations are relevant criteria for assessing the consequences of an RDD. NRC’s decision to not consider other criteria to assess the consequence of an RDD has resulted in security requirements that do not address the full risks presented by the danger that category 3 quantities of some radioactive material could be used in an RDD to cause significant socioeconomic consequences comparable to what could be caused by category 2 or category 1 quantities of material. By considering socioeconomic impacts and fatalities resulting from evacuations in its criteria, NRC would have better assurance that it was considering more likely and more significant consequences of an RDD when establishing its security requirements for this material. Furthermore, Part 37 requires enhanced security measures for categories 1 and 2 quantities of radioactive material and does not require additional security for category 3, 4, and 5 quantities of material beyond existing safety requirements. Although NRC chose to limit its 2016 evaluation of Part 37 to only category 1 and 2 quantities of material, experts who participated in our meeting generally said that they consider certain category 3 quantities of radioactive material high risk based on their potential consequences if used in an RDD, and data from recent studies support this determination. By requiring additional security measures for these high-risk quantities of category 3 material, and assessing whether other category 3 radioactive materials should be safeguarded with additional security measures, NRC can have better assurance that its requirements are sufficient to help ensure all high-risk radioactive material are protected from theft and use in an RDD. In addition, NRC’s 2016 report looked at the risk posed by the collocation of category 3 quantities of material and concluded that rule clarifications and additional guidance could help enhance the clarity and effectiveness of the rule. However, the report does not fully address how this risk should be resolved. Current NRC security requirements permit the collocation of multiple category 3 quantities of material that in total reach a category 2 quantity of material or higher, without triggering additional security requirements under Part 37. By requiring that all licensees implement additional security measures when they collocate multiple quantities of category 3 americium-241—that in total reach a category 1 or 2 quantity—at a single facility, NRC could have better assurance that the material is protected from theft and use in an RDD. We are making the following three recommendations to the Nuclear Regulatory Commission: The Chairman of NRC should direct NRC staff to consider socioeconomic consequences and fatalities from evacuations in the criteria for determining what security measures should be required for radioactive materials that could be used in an RDD. (Recommendation 1) The Chairman of NRC should require additional security measures for high-risk quantities of certain category 3 radioactive material, and assess whether other category 3 materials should also be safeguarded with additional security measures. (Recommendation 2) The Chairman of NRC should require all licensees to implement additional security measures when they have multiple quantities of category 3 americium-241 at a single facility that in total reach a category 1 or 2 quantity of material. (Recommendation 3) We provided a draft of this report to the Chairman of NRC, the Administrator of NNSA, the Secretary of the Department of Homeland Security, and the Attorney General of the United States. NRC provided written comments on the draft report, which are presented in appendix III. In addition, NRC provided technical comments, which we incorporated as appropriate. NNSA, DHS, and FBI did not provide written comments. NRC disagreed with two of our recommendations and neither agreed nor disagreed with an additional recommendation. Specifically, it disagreed with our recommendations that it (1) consider socioeconomic consequences and fatalities from evacuations when determining security measures for radioactive materials; and (2) require licensees to implement additional security measures when they have multiple quantities of category 3 americium-241 at a single facility that in total reach a category 1 or 2 quantity. NRC stated that it is considering an additional recommendation that it require additional security measures for high-risk quantities of category 3 materials. Regarding the first recommendation with which NRC disagreed, the agency stated that its current regulatory requirements provide for the safe and secure use of radioactive materials, and that we only focused on potential consequences of an RDD without consideration of the two other elements of risk—threat and vulnerability. We disagree. NRC agrees that a general threat exists, and this report, in combination with our previous reports, demonstrate that there are vulnerabilities in current NRC security requirements and that the potential consequences of misusing these materials could be significant. Furthermore, the report discusses new evidence related to the consequence of an RDD that NRC has not yet considered. For the second recommendation with which it disagreed, NRC stated that it has already considered the issue of aggregation of radioactive material and has taken or is in the process of taking actions to clarify relevant guidance and procedures. Again, we disagree. We acknowledge that NRC is taking action to better educate licensees on how to comply with requirements related to aggregation. However, these actions do not address the issue of licensees taking advantage of NRC’s security requirements which permit the storing of multiple category 3 quantities that are just below the threshold for category 2 at the same facility. Finally, for the NRC recommendation to consider additional security measures for high-risk quantities of category 3 materials, the agency said that it has been considering our recommendation in connection with its response to the recommendations in GAO-16-330. However, after we issued GAO-16-330, NRC staff subsequently recommended that the NRC Commission not implement the recommendations from that report. NRC stated that our report and recommendations lack important context in that we did not consider all aspects of risk—threat, vulnerability, and consequences. We disagree. First, as the report states, NRC agrees that a general threat exists for the theft and misuse of radiological materials. Second, the report also states that we have addressed vulnerability in several past GAO reports that provide examples of how the controls that NRC and others have put in place to prevent the theft or misuse of these materials are not always implemented correctly. In fact, we found gaps in these controls each time we reviewed the security of radioactive materials. These gaps in controls create vulnerabilities. Having discussed threat and vulnerability, this report adds important new information concerning the consequences of an RDD. In this regard, both the Sandia studies and the results from our National Academy of Sciences expert meeting show that prompt fatalities from radiation are unlikely to occur if an RDD is detonated, while the same event could result in tens of billions of dollars in economic damage and potentially hundreds to thousands of deaths from evacuations. NRC also stated that our evidence was insufficient for recommending regulatory and policy changes. Specifically, they said that the Sandia studies (1) were based on scenarios that were not probable, (2) did not credit existing protective measures to prevent an RDD, and (3) were not subjected to a formal review and endorsement process. In addition, they said that the views expressed by experts who attended our National Academy of Sciences meeting resulted in conclusions that were not fully supported. We disagree with these characterizations of the studies and our expert meeting. Specifically, the Sandia studies did not attempt to assess existing security measures for radioactive material or the probability or likelihood of an RDD. These Sandia studies examined the consequences of an RDD and represent the most recent research on RDD consequences from an independent and reliable source. In addition, NRC’s claim that the Sandia studies were conducted without a formal review and endorsement process is misleading. Specifically, according to NNSA officials, Sandia and NNSA officials met with officials from NRC, DHS, and EPA, among others, to discuss and gather input on the assumptions to be used in the 2017 Sandia study. During this meeting, according to NNSA officials, NRC staff provided input on key assumptions and subsequently provided data to help support the Sandia study. In addition, NNSA and Sandia briefed their interagency partners, including NRC, about the findings in the study before publishing and received generally positive feedback on their results. Furthermore, we partnered with the National Academies to identify and select a broad range of experts in the field of radioactive material security, including federal agency and agreement state officials; academics; representatives of nonprofit organizations, licensees, and industry; international regulators; and national laboratory specialists. For additional information on how we developed, held, and analyzed data from our National Academy of Sciences expert meeting, please refer to appendix 2. NRC’s comments also state that GAO does not account for the work of the 2014 Radiation Source Protection and Security Task Force (the Task Force), which considered economic consequences related to an RDD. However, as noted in our report, NRC’s response to the Task Force’s recommendations said that NRC staff would need additional direction from the Commission to consider examining alternative consequences. In addition, in 2014, NRC staff recommended that NRC not consider changing the policy to include consideration of socioeconomic consequences to the Commission, reiterating the staff’s view that Part 37 provides adequate security protection against a significant RDD. Today, NRC staff still does not have direction from the Commission to consider socioeconomic effects when setting security requirements. We think that needs to change in order for NRC to conduct a complete analysis of the consequences of an RDD. Finally, NRC stated that a significant gap related to the security of category 3 sources has not been identified. We disagree. As noted in the report, requirements for the security of category 3 quantities of radioactive materials are significantly less stringent than those required for category 1 and 2 quantities of material. Nevertheless, our report shows that the use of category 3 quantities of certain radioactive materials in an RDD may have comparable socioeconomic consequences. Furthermore, previous GAO reports have repeatedly shown that gaps exist related to the security of category 3 and higher radioactive material. We are sending copies of this report to the appropriate congressional committees, the Chairman of the U.S. Nuclear Regulatory Commission, the Secretary of Energy, the Secretary of Homeland Security, the Administrator of the Environmental Protection Agency, and the Attorney General of the United States, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our offices of Congressional Relations and Public Affairs are on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. We focused our review primarily on the Nuclear Regulatory Commission (NRC) because it is the principal federal agency with responsibility for licensing the commercial use of and regulating the security of radioactive materials in the United States. Additionally, Public Law 113-235 specifically directs us to review NRC’s security requirements for radioactive material. We also interviewed officials at various agencies that play a role in radioactive material security, including the National Nuclear Security Administration (NNSA), the Department of Homeland Security (DHS), the Environmental Protection Agency (EPA), and the Federal Bureau of Investigation (FBI). We interviewed officials at NNSA because NNSA’s Office of Radiological Security provides upgrades and enhancements to NRC licensees and removes and disposes of disused radioactive material. We also spoke to DHS officials because DHS is the primary federal agency for implementing domestic nuclear detection efforts for a managed and coordinated response to radioactive and nuclear threats. Additionally, we interviewed officials at EPA, because the agency developed the Protective Action Guide (PAG) manual, which contains radiation dose guidelines that would trigger public safety measures. Finally, we interviewed the FBI, which offered us information on the potential threat related to radioactive material security. In addition to federal agencies, we were contacted by and spoke to a working group that represents the commercial radioactive source industry and received a briefing from a company, which is also a member of the working group that utilizes large panoramic irradiators. We received a series of risk briefings from federal agencies to collect information on current risks related to radioactive material security. NRC officials provided us with information about how the agency evaluates risks associated with radioactive material, including the threat, vulnerability, and consequence of an adversary acquiring and using radioactive material in a radioactive dispersal device (RDD). DHS officials provided us with a risk briefing on current threats to radioactive material and potential consequences of an RDD attack. Specifically, those officials briefed us on historical terrorist interest in using radioactive materials in attacks. NNSA officials and Sandia National Laboratory experts in radioactive security and consequence modeling briefed us on potential economic consequences from an RDD, which they based on an economic impact study completed by Sandia in 2018. Finally, FBI officials gave us a threat briefing focused on current radioactive material security threats, including interest by adversaries in conducting an RDD attack. These briefings were held at a classified level. In order to fulfill the Public Law 113-235 requirement to work with an independent group of experts, we partnered with the National Academies of Sciences to convene a group of experts on radioactive material security on July 26 and 27, 2018. We determined that this method offered the best means of gathering a balanced group of leading experts in the field of radioactive security to discuss issues in a moderated setting. In addition, this method allowed us to implement a structured and systematic approach when gathering evidence. Specifically, our methodology for the meeting included selecting a broad range of experts to participate in the meeting, administering a written questionnaire to the experts before the meeting, designing specific scenarios used during the moderated discussion, and performing a thematic analysis upon completion of the meeting. To describe how NRC assesses risk when establishing security requirements for high-risk radioactive materials and how it chose to primarily consider prompt fatalities as criteria for measuring consequences of an RDD, we reviewed NRC documents addressing how NRC evaluates an RDD, NRC’s study evaluating the effectiveness of Part 37 in response to Public Law 113-235, and NRC’s analysis of the risks posed by high-risk radioactive materials. Specifically, we reviewed NRC Commission Papers and NRC responses to actions taken by the Radiation Source Protection and Security Task Force. We also conducted interviews with agency officials at NRC, NNSA, DHS, EPA, and FBI, as well as academics, agreement state officials, and security managers from industry about the risks associated with different categories of radioactive materials and how NRC regulates these materials. We selected interviewees based on their expertise, but the results of these interviews are not generalizable. To examine the extent to which radioactive security experts agreed that NRC’s assessment of risk includes all relevant criteria for establishing security requirements, we partnered with the National Academies to identify and select a broad range of experts in the field of radioactive material security, including federal agency and agreement state officials; academics; representatives of nonprofit organizations, licensees, and industry; international regulators; and national laboratory specialists. In choosing the group of 18 experts, we specifically chose individuals with a diversity of backgrounds on topics. This ensured a balanced range of opinions and specific expertise on given topics but did not represent a generalizable sample of experts on a specific topic. For example, some individuals had specific expertise in certain topics and could provide a more insightful perspective than others in the group. In advance of the meeting, we developed a written questionnaire to obtain the experts’ views on the key threats, vulnerabilities and consequences of materials regulated under Part 37 and those not regulated under Part 37. We administered the questionnaire to the experts via email and obtained completed questionnaires from all of them. We analyzed their responses to focus the topics we discussed during our two-day meeting. During the meeting, we introduced the threat of malevolent use of radioactive material, and we asked the experts to focus their discussion the potential consequences of an RDD, the vulnerabilities of radiological materials under current security requirements and whether current security requirements were sufficient given these consequences and vulnerabilities. We asked them to discuss the reasons to account for the consequence in the regulation of radioactive material, the reasons not to account for the consequence, and whether the consequence should be accounted for. In addition, we asked the experts the reasons various radionuclides should be considered high risk, the reasons these radionuclides should not be considered high risk, and on balance, whether these radionuclides should be considered high risk. Our meeting agenda and moderator guide also included detailed scenarios designed to probe issues related to radioactive security and provide clear parameters within which the experts could make observations. In particular, we presented the experts with four scenarios of differing quantities of radioactive material used for particular medical and industrial purposes and stored under particular circumstances. The four scenarios presented different types of radioactive materials stored under particular circumstances. For these scenarios, we asked experts about the primary vulnerabilities of these materials in terms of access, monitoring and detection and response, and given the consequences and vulnerabilities, whether the Part 37 security requirements were sufficient. We based these scenarios on situations we observed during our prior work on the security of radioactive material. For each of these scenarios, we asked experts to assess two key elements of the risk triplet—vulnerability to being used and consequences if used. For example, we included a presentation on threat associated with radioactive materials. Additionally, for each scenario we moderated discussions on scenarios focused on vulnerabilities of category 1 and category 3 radioactive materials or scenarios focused on the consequences of category 1 and category 3 RDDs. Furthermore, a GAO methodologist and a National Academies of Sciences official guided discussions, following the structured moderator guide to ensure the discussions addressed all topics. The moderators ensured that experts from all sides had the opportunity to voice their opinions, but time constraints and the nature of an expert meeting may have limited some experts from contributing. Because of this structure, we had no expectation of reaching outright consensus on any specific topic. After the expert meeting, we conducted a structured and systematic thematic analysis of the information gathered to better understand the potential vulnerabilities of radioactive materials to theft and the consequences of an RDD using various radioactive materials. We also worked with GAO methodologists to sort the content in the meeting transcript, identify themes from the sorted information for additional analysis, and evaluate the credibility of expert statements. GAO internally reviewed our analysis for completeness and accuracy and it was found to be sufficient for our purposes. The meeting transcript write-up allowed us to focus on strengths and weaknesses in current security requirements, how the federal government evaluates the consequences of an RDD, what materials should be considered high risk, and whether additional security measures are necessary for these materials. Experts did not speak on every topic, did not have the same level of expertise on every topic, and the meeting format was not designed to quantify experts’ comments. Therefore, we do not report the number of the 18 experts who agreed or disagreed with various statements. Instead, through our thematic analysis, we determined that during the expert meeting experts generally made two types of statements on topics with varying degrees of agreement or corroboration, which we refer to as either “strong evidence” or “evidence of varying viewpoints.” GAO, Government Auditing Standards, 2018 Revision, GAO-18-568G (Washington, D.C.: July 2018). According to government auditing standards, testimonial evidence obtained from an individual who is not biased and has direct knowledge about the area is generally more reliable than testimonial evidence obtained from an individual who is biased or has indirect or partial knowledge about the area. In addition to the individual named above, Ned Woodward (Assistant Director), Jeffrey Barron (Analyst in Charge), Kevin Bray, Mark Braza, Kendall Childers, Tara Congdon, Gabrielle Matuzsan, Amanda Miller, Danny Royer, and Kiki Theodoropoulos made key contributions to this report.", "summary": "NRC is responsible for regulating the security of radioactive material in the U.S. Failure to secure this material could result in an RDD causing socioeconomic damage. The Consolidated and Further Continuing Appropriations Act, 2015 (Public Law 113-235) includes a provision for GAO to review NRC's security requirements for high-risk radioactive material. This report examines, among other things, (1) the extent to which radioactive security experts agreed that NRC's assessment of risk includes all relevant criteria, and (2) NRC's 2016 evaluation of its security requirements for high-risk radioactive material. GAO reviewed NRC policies and procedures, worked with the National Academies of Sciences to convene a meeting with 18 experts in radioactive security, and reviewed 3 recent Sandia studies. GAO used the views of security experts to define high risk, and they generally agreed that high risk includes both larger and some smaller quantities of radioactive materials. The 18 experts at a meeting GAO convened with the National Academies of Sciences generally agreed that the Nuclear Regulatory Commission (NRC) assessment of risks of radioactive material does not include all relevant criteria. NRC limits its criteria to prompt fatalities and deterministic health effects from radiation, which, according to the experts and recent studies, are unlikely to result from a radiological dispersal device (RDD). Two studies from Sandia National Laboratories (Sandia) measuring consequences of RDDs, released in 2017 and 2018, found that there would be no immediate fatalities from radiation. The experts at the meeting generally agreed that socioeconomic effects (e.g., relocations and clean-up costs) and fatalities that could result from evacuations are the most relevant criteria for evaluating the risks of radioactive material. The two Sandia studies found that a large RDD could cause about $30 billion in damage and 1,500 fatalities from the evacuation, and a considerably smaller RDD could cause $24 billion in damage and 800 fatalities from the evacuation. By considering socioeconomic impacts and fatalities resulting from evacuations in its criteria, NRC would have better assurance it was considering the more likely and more significant consequences of an RDD. NRC's 2016 report evaluating its security requirements for high-risk radioactive material, required by Public Law 113-235, considered only the security of larger quantities of such material and not smaller quantities. Experts who attended GAO's meeting stated, and two 2018 Sandia studies agree, that if smaller quantities of certain radioactive material were used in an RDD, the impacts would be comparable to an RDD with a considerably larger amount of such material. For example, a 2018 study from Sandia found that malicious use of certain radioactive materials in smaller quantities could cause significant socioeconomic consequences. By requiring additional security measures for these smaller quantities of high-risk material, NRC can have better assurance that its security requirements are sufficient to secure all high-risk radioactive material from theft and use in an RDD. Example of a Radiological Dispersal Device (RDD) GAO is making three recommendations to NRC, including that it consider socioeconomic consequences and fatalities from evacuations as criteria for determining security measures and require additional security measures for smaller quantities of high-risk material. NRC generally disagreed with the recommendations, stating that GAO's evidence does not provide a sufficient basis for recommended changes. GAO continues to believe these recommendations are important.", "document_type": "gao"}
{"report": "We have previously reported on the extent to which FEMA programs encourage disaster resilience during recovery efforts and our prior and ongoing work also highlight opportunities to improve disaster resilience nationwide. Specifically, we reported on (1) federal efforts to strengthen disaster resilience, (2) FEMA’s efforts to promote hazard mitigation through the Public Assistance program, and (3) crafting appropriate federal responses to the effects of climate change. First, in July 2015, we found that states and localities experienced challenges when trying to use federal funds to maximize resilient rebuilding in the wake of a disaster. In particular, they had difficulty navigating multiple federal grant programs and applying federal resources toward their most salient risks because of the fragmented and reactive nature of the funding. In our 2015 report, we recommended that the Mitigation Framework Leadership Group—an interagency body chaired by FEMA—create a National Mitigation Investment Strategy to help federal, state, and local officials plan for and prioritize disaster resilience efforts. In August 2019, FEMA took action to fully implement our recommendation by publishing this strategy. Second, in November 2017, we found that FEMA had taken some actions to better promote hazard mitigation as part of its Public Assistance grant program, which provides grant funding for cost-effective hazard mitigation measures to reduce or eliminate the long-term risk to people and property from future disasters and their effects. However, we also reported that more consistent planning for, and more specific performance measures related to, hazard mitigation could help ensure that mitigation is incorporated into recovery efforts. We recommended, among other things, that FEMA (1) standardize planning efforts for hazard mitigation after a disaster and (2) develop performance measures for the Public Assistance grant program to better align with FEMA’s strategic goal for hazard mitigation in the recovery process. The Department of Homeland Security (DHS) concurred with our recommendations, and officials reported taking steps to increase coordination across its Public Assistance, mitigation, and field operations to ensure hazard mitigation efforts are standardized and integrated into the recovery process. Additionally, FEMA officials reported taking actions to begin developing disaster-specific mitigation performance measures. However, FEMA has yet to finalize these actions, such as by proposing performance measures to FEMA senior leadership. As such, we are continuing to monitor FEMA’s efforts to address these recommendations. Third, in September 2017, we reported that the methods used to estimate the potential economic effects of climate change in the United States— using linked climate science and economics models—could inform decision makers about significant potential damages in different U.S. sectors or regions, despite the limitations. For example, for 2020 through 2039, one study estimated between $4 billion and $6 billion in annual coastal property damages from sea level rise and more frequent and intense storms. We found that the federal government has not undertaken strategic government-wide planning on the potential economic effects of climate change to identify significant risks and craft appropriate federal responses. As a result, we recommended the Executive Office of the President, among others, should use information on the potential economic effects of climate change to help identify significant climate risks facing the federal government and craft appropriate federal responses, such as establishing a strategy to identify, prioritize, and guide federal investments to enhance resilience against future disasters. However, as of June 2019, officials had not yet taken action to address this recommendation. FEMA and other federal agencies provide multiple forms of disaster recovery assistance after a major disaster has been declared, including through FEMA’s Public Assistance and Individual Assistance programs, HUD’s Community Development Block Grant Disaster Recovery (CDBG- DR) program, and other efforts. Through these programs, the federal government obligates billions of dollars to state, tribal, territorial, and local governments, certain nonprofit organizations, and individuals that have suffered injury or damages from major disaster or emergency incidents, such as hurricanes, tornados, or wildfires. In September 2016, we reported that, from fiscal years 2005 through 2014, FEMA obligated almost $46 billion for the Public Assistance program and over $25 billion for the Individual Assistance program. According to FEMA’s September 2019 Disaster Relief Fund report, total projected obligations through fiscal year 2019 for the Public Assistance and Individual Assistance programs since August 1, 2017, are approximately $19 billion and $9 billion, respectively. Further, in March 2019, we reported that in response to the 2017 disasters, HUD had awarded approximately $32.9 billion in CDBG- DR funds to four grantees as of February 2019—$19.9 billion to Puerto Rico, $9.8 billion to Texas, $1.9 billion to the USVI, and $1.3 billion to Florida. As of September 2019, much of these awarded funds had been allocated to the grantees via Federal Register notices with the exception of Puerto Rico. HUD had not allocated the remaining $10.2 billion it awarded to Puerto Rico as of September 10, 2019, due to recent concerns about the territory’s governance and financial management challenges. Given the high cost of these programs, it is imperative that FEMA and HUD continue to make progress on the challenges we have identified in our prior and ongoing work regarding recovery efforts. FEMA’s Public Assistance program provides grants to state, tribal, territorial, and local governments, as well as certain types of private nonprofit organizations, for debris removal; emergency protective measures; and the repair, replacement, or restoration of disaster- damaged, publicly owned facilities. It is a complex and multistep program administered through a partnership among FEMA, state, and local officials. Prior to implementing the Public Assistance program, FEMA determines a state, territorial or tribal government’s eligibility for the program using primarily the per capita damage indicator. In our September 2018 report on federal response and recovery efforts for the 2017 hurricanes and wildfires, we reported on FEMA’s implementation of the Public Assistance program, which has recently undergone significant changes as a result of federal legislation and agency initiatives. Specifically, we reported on FEMA’s use of its redesigned delivery model for providing grants under the Public Assistance program, as well as the alternative procedures for administering or receiving such grant funds that FEMA allows states, territories, and local governments to use for their recovery. Our prior and ongoing work highlight both progress and challenges with FEMA’s Public Assistance program, including the agency’s methodology for determining program eligibility, the redesigned delivery model, and the program’s alternative procedures. In September 2012, we found that FEMA primarily relied on a single criterion, the per capita damage indicator, to determine a jurisdiction’s eligibility for Public Assistance funding. However, because FEMA’s per capita indicator, set at $1 in 1986, does not reflect the rise in (1) per capita personal income since it was created in 1986 or (2) inflation from 1986 to 1999, the indicator is artificially low. Our analysis of actual and projected obligations for 508 disaster declarations in which Public Assistance was awarded during fiscal years 2004 through 2011 showed that fewer disasters would have met either the personal income-adjusted or the inflation-adjusted Public Assistance per capita indicators for the years in which the disaster was declared. Thus, had the indicator been adjusted annually since 1986 for personal income or inflation, fewer jurisdictions would have met the eligibility criteria that FEMA primarily used to determine whether federal assistance should be provided, which would have likely resulted in fewer federal disaster declarations and lower federal costs. We recommended, among other things, that FEMA develop and implement a methodology that more comprehensively assesses a jurisdiction’s capacity to respond to and recover from a disaster without federal assistance, including fiscal capacity and consideration of response and recovery capabilities. DHS concurred with our recommendation and, in January 2016, FEMA was considering establishing a disaster deductible, which would have required a predetermined level of financial or other commitment before FEMA would have provided assistance under the Public Assistance program. In September 2019, FEMA told us that it was considering options for alternative methodologies for, among other things, assessing a jurisdiction’s independent capacity to respond to and recover from disasters. In addition, the DRRA includes a provision directing the FEMA Administrator to initiate rulemaking to update the factors considered when evaluating requests for major disaster declarations. According to FEMA documentation, as of September 2019, the agency was working to implement this provision through rulemaking proposals, including increasing the per capita indicator to account for inflation. Until FEMA implements a new methodology, the agency will not have an accurate assessment of a jurisdiction’s capabilities and runs the risk of recommending that the President award Public Assistance to jurisdictions that have the capacity to respond and recover on their own. In November 2017, we reported that FEMA redesigned its delivery model for providing grants under the Public Assistance program. As part of the redesign effort, FEMA developed a new, web-based case management system to address past challenges, such as difficulties in sharing grant documentation among FEMA, state, and local officials and tracking the status of Public Assistance projects. Both FEMA and state officials involved in testing the redesigned delivery model stated that the new case management system’s capabilities could lead to greater transparency and efficiencies in the program. However, we found that FEMA had not fully addressed two key information technology management controls that are necessary to ensure systems work effectively and meet user needs. We recommended, among other things, that FEMA (1) establish controls for tracking the development of system requirements, and (2) establish system testing criteria, roles and responsibilities, and the sequence and schedule for integration of other relevant systems. DHS concurred with these recommendations and, as of October 2019, has fully implemented both. FEMA’s original intention was to implement the redesigned delivery model for all future disasters beginning in January 2018. However, in September 2017, FEMA expedited full implementation of the redesigned model shortly after Hurricane Harvey made landfall. In September 2018, we reported that local officials continued to experience challenges with using the new Public Assistance web-based, case management system following the 2017 disasters, such as not having sufficient guidance on how to use the new system and delays with FEMA’s processing of their projects. Our prior and ongoing work highlight the challenges with implementing the Public Assistance program, including the alternative procedures, in Puerto Rico and the USVI. In particular, our work has identified challenges related to (1) developing fixed-cost estimates and (2) implementing flexibilities provided by the Bipartisan Budget Act of 2018. This Act allows FEMA, Puerto Rico, and the USVI to repair and rebuild critical services infrastructure—such as medical and education facilities— so it meets industry standards without regard to pre-disaster condition (see fig. 1). Unlike in the standard Public Assistance program where FEMA will fund the actual cost of a project, the Public Assistance alternative procedures allow awards for permanent work projects to be made on the basis of fixed-cost estimates to provide financial incentives for the timely and cost- effective completion of work. FEMA officials in Puerto Rico and the USVI stated that the development of a “cost factor” for use in the fixed-cost estimating process had slowed the pace of FEMA obligations for permanent work projects. Specifically, these factors are intended to ensure that the costs associated with implementing projects in Puerto Rico and the USVI are sufficiently captured when developing the fixed- cost estimates for alternative procedures projects. Since incorporating the cost factor into the fixed-cost estimating process will increase the amount of funding obligated for any given permanent work project, FEMA officials explained that Puerto Rico and the USVI had an incentive to delay the obligation of individual projects until this factor was finalized. For example, FEMA officials in the USVI told us in May 2019 that obligations for permanent work projects in the territory had been mostly on hold since October 2018 while an independent contractor worked to develop the USVI-specific cost factor. FEMA officials told us that USVI officials disagreed with the initial USVI- specific cost factors the independent contractor proposed. USVI officials contended that the cost factors were insufficient in accurately capturing the unique circumstances that influence construction costs in the territory, such as the limited availability of local resources and the need to import materials and labor. In May 2019, the contractor proposed a new cost factor, which FEMA approved on an interim basis pending further analysis. In July 2019, FEMA officials told us that while additional analyses are required to ensure its final process for developing fixed-cost estimates in the USVI accurately captures construction costs, using this interim cost factor in the meantime allows FEMA and USVI officials to move forward with the development and final approval of alternative procedures projects. In August 2019, a senior USVI official told us the territory plans to begin using the interim cost factor, where appropriate, to keep projects progressing forward. However, this official stated that the USVI questioned whether the interim cost factor did, in fact, sufficiently capture the actual costs of construction in the USVI. Given the uncertainty around these fixed-cost estimates, USVI officials told us the territory will need to balance the potential flexibilities provided by the alternative procedures program with the financial risk posed by cost overruns when deciding whether to use the alternative procedures or the standard Public Assistance program for any given permanent work project. Specifically, these officials stated that the USVI plans to pursue alternative procedures projects that do not include high levels of complexity or uncertainty to reduce the risk of cost overruns, especially given its already difficult financial situation. In addition, according to FEMA guidance, the Puerto Rico-specific cost factor was developed by a third-party center of excellence comprising personnel selected by FEMA and Puerto Rico. Through our ongoing work we learned that FEMA convened a panel of FEMA engineers to assess the cost factor methodologies proposed by the center of excellence. In July 2019, FEMA approved the use of a cost factor designed to account for location-specific construction costs in Puerto Rico to ensure that fixed-cost estimates for alternative procedures projects are accurate. This cost factor consists of cost indices to apply to urban, rural, and insular (the islands of Vieques and Culebra) areas of Puerto Rico. According to FEMA officials, these cost indices will compile location- specific construction costs for each of these three areas. We are currently assessing FEMA’s process for developing cost estimates for projects under both the standard and alternative procedures programs, and plan to report our results in early 2020. As of September 2019, FEMA officials told us the agency had obligated funding for 14 alternative procedures projects in Puerto Rico out of approximately 9,000 projects FEMA and Puerto Rico are working to develop for inclusion in the program. According to FEMA guidance, Puerto Rico must use the alternative procedures for all large permanent work projects and its deadline for finalizing the fixed-cost estimates for these projects was October 11, 2019. However, on October 8, 2019, Puerto Rico requested that FEMA extend this deadline. In response, FEMA acknowledged that Puerto Rico and FEMA have significant work remaining to develop and finalize the fixed-cost estimates for alternative procedures projects. As a result, FEMA authorized all parties to continue developing these projects while FEMA works to establish a new deadline for finalizing fixed-cost estimates in Puerto Rico. Unlike Puerto Rico, the USVI has the flexibility to pursue either the alternative procedures or the standard procedures on a project-by-project basis. As of September 2019, FEMA had obligated funding for two alternative procedures projects in the USVI. As the USVI’s deadline for finalizing these projects is in March 2020, it is too early to gauge the extent to which the alternative procedures will play a role in the USVI’s long-term recovery strategy. In addition, our preliminary observations indicate that FEMA, Puerto Rico, and USVI officials have reported challenges with the implementation of the flexibilities authorized by section 20601 of the Bipartisan Budget Act. This section of the Act allows for the provision of assistance under the Public Assistance alternative procedures to restore disaster-damaged facilities or systems that provide critical services to an industry standard without regard to pre-disaster condition. Officials from Puerto Rico’s central government stated that they disagreed with FEMA’s interpretation of the types of damages covered by section 20601 of the Bipartisan Budget Act of 2018. In response, FEMA officials in Puerto Rico stated they held several briefings with Puerto Rico’s central recovery office to explain FEMA’s interpretation of the section. Further, FEMA officials in the USVI told us that initially, they had difficulty obtaining clarification from FEMA headquarters regarding how to implement key components of section 20601 of the Act. In June 2019, the Additional Supplemental Appropriations for Disaster Relief Act of 2019 was signed into law and provides additional direction to FEMA regarding the implementation of section 20601. Among other things, this legislation includes a provision directing FEMA to change its process for determining whether a disaster- damaged facility is eligible for repair or replacement. FEMA evaluated this and other provisions of the Act and, in September 2019, issued an updated policy to provide clear guidance moving forward, according to agency officials. We will continue to evaluate these identified challenges and any efforts to address them, as well as other aspects of recovery efforts in the USVI and Puerto Rico, and plan to report our findings in November 2019 and January 2020, respectively. The Individual Assistance program provides financial and direct assistance to disaster victims for expenses and needs that cannot be met through other means, such as insurance. In May 2019, we reported on FEMA’s efforts to provide disaster assistance under the Individual Assistance program to older adults and people with disabilities following the 2017 hurricanes. We found that aspects of the application process for FEMA assistance were challenging for older individuals and those with disabilities. Further, according to stakeholders and FEMA officials, disability-related questions in the Individual Assistance registration materials were confusing and easily misinterpreted. While FEMA had made some efforts to help registrants interpret the questions, we recommended, among other things, that FEMA (1) implement new registration-intake questions that improve FEMA’s ability to identify and address survivors’ disability-related needs, and (2) improve communication of registrants’ disability-related information across FEMA programs. DHS concurred with the first recommendation, and officials reported that in May 2019 the agency updated the questions to directly ask individuals if they have a disability. The agency has taken actions to fully implement this recommendation and, according to FEMA’s analysis of applications for assistance following recent disasters—which used the updated questions—the percentage of registrants who reported having a disability increased. However, DHS did not concur with the second recommendation, noting that it lacks specific funding to augment its legacy data systems. FEMA officials stated that they began a long-term data management improvement initiative in April 2017, which they expect will ease efforts to share and flag specific disability-related data. While we acknowledge FEMA’s concerns about changing legacy systems when it has existing plans to replace those systems, we continue to believe there are other cost-effective ways that are likely to improve communication of registrants’ disability-related information prior to implementing the system upgrades. For example, FEMA could revise its guidance to remind program officials to review the survivor case file notes to identify whether there is a record of any disability-related needs. We also have work underway to assess FEMA’s Individuals and Households Program, a component program of Individual Assistance. Through this program, as of April 2019, FEMA had awarded roughly $4.7 billion in assistance to almost 1.8 million individuals and households for federally-declared disasters occurring in 2017 and 2018. Specifically, we are analyzing Individuals and Households Program expenditures and registration data for recent years; reviewing FEMA’s processes, policies, and procedures for making eligibility and award determinations; and examining survivors’ reported experiences with this program, including any challenges, for major disaster declarations occurring in recent years. We plan to report our findings in early 2020. FEMA’s Individuals and Households Program provides individuals with financial assistance, such as grants to help repair or replace damaged homes, and temporary direct housing assistance, such as recreational vehicles. HUD CDBG-DR grants provide funding that disaster-affected communities may use to address unmet needs for housing, infrastructure, and economic revitalization. In March 2019, we reported on the status of CDBG-DR grants following the 2017 disasters, plans for monitoring the program, and challenges HUD and grantees faced in administering these grants. We found that HUD lacked adequate guidance for staff reviewing key information, such as the quality of grantees’ financial processes and procedures and assessments of grantees’ capacity and unmet needs. Further, we found HUD had not completed monitoring or workforce plans that identify key risk factors and critical skills and competencies required for program implementation, among other things. In addition, we found that Congress has not established permanent statutory authority for CDBG-DR but rather has used supplemental appropriation legislation to authorize HUD to establish requirements via Federal Register notices. Without such permanent statutory authority, HUD must customize grantee requirements for each disaster. The ad hoc nature of CDBG-DR has created challenges for CDBG-DR grantees, such as lags in accessing funding and coordinating these funds with other disaster recovery programs. For example, it took 154 days (or 5 months) for HUD to issue the requisite Federal Register notice after the first appropriation for the 2017 hurricanes. According to HUD officials, they delayed issuance of the first notice for the 2017 hurricanes because they expected a second appropriation and wanted to allocate those funds in the same notice. However, because the second appropriation took longer than HUD expected, the first notice allocated only the first appropriation. We recommended that Congress consider permanently authorizing a disaster assistance program to address unmet needs in a timely manner. In addition, we made five recommendations to HUD. Specifically, we made two recommendations to HUD regarding developing additional guidance for staff to use when reviewing grantees’ planning documentation. HUD partially agreed with these two recommendations, stating that some of this guidance was already in place. Because HUD acknowledged that providing additional guidance would improve its review process, we revised these two recommendations accordingly to reflect the need for additional guidance. We also made three additional recommendations to HUD, including that the agency should develop a monitoring plan for grants and conduct workforce training. HUD generally agreed with these recommendations and indicated it planned to develop monitoring strategies. HUD also stated that it had developed a staffing plan, but we noted the agency still needed to conduct workforce planning to determine if the number of staff the agency planned to hire was sufficient. We are continuing to monitor HUD’s efforts to address these recommendations. In addition to those described above, we reported on challenges FEMA faced in (1) providing mass care to disaster survivors, (2) assisting jurisdictions affected by wildfires, and (3) supporting electricity grid recovery efforts in Puerto Rico. In September 2019, we reported on FEMA’s and the American Red Cross’ efforts to coordinate mass care—which includes sheltering, feeding, and distributing emergency supplies—following the 2017 hurricanes. We found that some needs related to mass care were unmet. For example, local officials in Texas said flooded roads prevented trucks from delivering supplies. Further, mass care providers encountered challenges in part because state and local agreements with voluntary organizations that help to provide mass care to disaster survivors did not always clearly detail what services these organizations were capable of providing. Among other things, we also found that while state, territorial, and local grantees of federal disaster preparedness grants are required to regularly submit information on their capabilities to FEMA, the mass care information some grantees provided to FEMA was not specific enough to aid its response in 2017. Moreover, as FEMA does not require grantees to specify the organizations providing mass care services in their capabilities assessments, grantees and FEMA may not be collecting reliable information on capabilities. As a result of our findings in this report, we made one recommendation to DHS, four recommendations to FEMA, and one recommendation to the American Red Cross. Specifically, among other things, we recommended that FEMA should emphasize the importance of defining roles and responsibilities in its guidance to grantees in states and localities and require them to solicit key capabilities information from mass care providers. DHS concurred with four recommendations, but did not concur with our recommendation requiring grantees to solicit key information from organizations providing mass care services and to specify these organizations in capability assessments. Specifically, DHS and FEMA stated that requiring grantees to include this information is not the most effective approach and would increase their burden. We modified our recommendation to address this concern and continue to believe that grantees should make an effort to include mass care providers in assessing capabilities. We will continue to monitor FEMA’s progress in fully addressing these recommendations. Further, in October 2019, we reported on the assistance FEMA provided to jurisdictions in response to major disaster declarations stemming from wildfires from 2015 through 2018 (see fig. 2). We found that FEMA helped state and local officials obtain and coordinate federal resources to provide for the needs of wildfire survivors and provided more than $2.4 billion in federal assistance. However, state and county officials also described challenges in responding to wildfire disasters. For example, onerous documentation requirements for FEMA’s Public Assistance grant program, a shortage of temporary housing for survivors, and the unique challenge of removing wildfire debris led to over-excavation on some homeowners’ lots and lengthened the rebuilding process. We also found that while FEMA had developed an after-action report identifying lessons learned from the October and December 2017 wildfires, the agency could still benefit from a more comprehensive assessment of its operations to determine if any changes are needed to better respond to the threat posed by increased wildfire activity. We recommended that FEMA assess operations to identify any additional updates to its management controls—such as policies, procedures, or training—that could enhance future response and recovery from large- scale and severe wildfires. DHS agreed with our recommendation and described a number of ongoing and planned actions it would take to address it, including supporting states’ efforts to house disaster survivors, developing guidance for housing grants authorized by the DRRA, and taking steps to identify areas of innovation in response to wildfire disasters. DHS anticipates that these efforts will be put into effect by December 2020 and we will continue to monitor DHS and FEMA’s progress in addressing this recommendation. In October 2019, we reported on federal efforts to support electricity grid recovery in Puerto Rico. We found that FEMA and other federal agencies can support long-term electricity grid recovery efforts and incorporate resilience through three primary roles—providing funding and technical assistance and coordinating among local and federal agencies. However, we found that zero permanent, long-term grid recovery projects in Puerto Rico had received funding as of July 2019 as Puerto Rico was still establishing priorities for permanent work. Further, we found that certain challenges are hindering progress on electricity grid recovery efforts in Puerto Rico, including uncertainty about the kinds of projects that may be eligible for federal funding, local capacity constraints, uncertainty about available federal funding, and the need for coordination among local and federal stakeholders. As a result of our findings, we made three recommendations to FEMA and one recommendation to HUD. Specifically, we recommended that FEMA should provide clear written policies, guidance, or regulations to clarify its plans for implementing new authorities provided by the Bipartisan Budget Act of 2018 and take steps to enhance coordination among local and federal entities. DHS concurred with these recommendations and stated it is working to address them. In addition, we recommended that HUD establish timeframes and a plan for publication of the grant process and requirements specifically for CDBG- DR funding for improvements to Puerto Rico’s electricity grid. In its response to this recommendation, HUD stated that it is closely working with its federal partners on the requirements for this funding in Puerto Rico, but did not specifically state whether it would establish the timeframes and a plan for publication of the grant process and requirements as we recommended. We continue to believe that this action is needed since without this information, local entities will continue to be uncertain regarding what is eligible for CDBG-DR funding. We will continue to monitor FEMA’s and HUD’s progress in addressing these recommendations. FEMA’s experiences during the 2017 disasters highlight the importance of continuing to make progress on addressing the long-standing workforce management challenges we have previously reported on and continue to observe in our ongoing work. In particular, our work has identified challenges related to (1) recruiting, maintaining, and deploying a trained workforce, (2) the Incident Management Assistance Team program, (3) Public Assistance program staffing, (4) contracting workforce shortages, (5) assistance to older adults and people with disabilities, and (6) workforce capacity and training. Recruiting, maintaining, and deploying a trained workforce. In September 2018, we reported that the 2017 disasters—hurricanes Harvey, Irma, and Maria, as well as the California wildfires—resulted in unprecedented FEMA workforce management challenges, including recruiting, maintaining, and deploying a sufficient and adequately-trained FEMA disaster workforce. FEMA’s available workforce was overwhelmed by the response needs caused by the sequential and overlapping timing of the three hurricanes. For example, at the height of FEMA workforce deployments in October 2017, 54 percent of staff were serving in a capacity in which they did not hold the title of “Qualified”— according to FEMA’s qualification system standards—a past challenge we identified. FEMA officials noted that staff shortages and lack of trained personnel with program expertise led to complications in its response efforts, particularly after Hurricane Maria. FEMA’s Incident Management Assistance Team program. In February 2016, we reported on, among other things, FEMA’s efforts to implement, assess, and improve its Incident Management Assistance Team program. We found that while FEMA used some leading practices in managing the program, it lacked a standardized plan to ensure that all national and regional Incident Management Assistance Team members received required training. Further, we found that the program had experienced high attrition since its implementation in fiscal year 2013. We recommended, among other things, that FEMA develop (1) a plan to ensure that Incident Management Assistance Teams receive required training, and (2) a workforce strategy for retaining Incident Management Assistance Team staff. DHS concurred with the recommendations. FEMA implemented our first recommendation by developing an Incident Management Assistance Team Training and Readiness Manual and providing a training schedule for fiscal year 2017. In response to the second recommendation, FEMA officials stated in July 2018 that they plan to develop policies that will provide guidance on a new workforce structure, incentives for Incident Management Assistance Team personnel, and pay-for-performance and all other human resource actions. We are continuing to monitor FEMA’s efforts to address this recommendation. Public Assistance program staffing. In November and December 2017, we reported on staffing challenges in FEMA’s Public Assistance program. In November 2017, we reported on FEMA’s efforts to address past workforce management challenges through its redesigned Public Assistance delivery model. As part of the redesign effort, FEMA created consolidated resource centers to standardize and centralize Public Assistance staff responsible for managing grant applications, and new specialized positions to ensure more consistent guidance to applicants. However, we found that FEMA had not assessed the workforce needed to fully implement the redesigned model, such as the number of staff needed to fill certain new positions, or to achieve staffing goals. Further, in December 2017, we reported on FEMA’s management of its Public Assistance appeals process, including that FEMA increased staffing levels for the appeals process from 2015 to 2017. However, we found that FEMA continued to face a number of workforce challenges, such as staff vacancies, turnover, and delays in training, which contributed to processing delays. Based on our findings from our November and December 2017 reports, we recommended, among other things, that FEMA (1) complete workforce staffing assessments that identify the appropriate number of staff needed to implement the redesigned Public Assistance delivery model, and (2) document steps for hiring, training, and retaining key appeals staff, and address staff transitions resulting from deployments to disasters. DHS concurred with our recommendations to address workforce management challenges in the Public Assistance program and have reported taking some actions in response. For example, to address the first recommendation, FEMA officials have developed preliminary models and estimates of staffing needs across various programs, including Public Assistance. However, as of October 2019, the agency has not yet taken actions to implement this recommendation. To address the second recommendation, FEMA has collected information on the amount of time regional appeals analysts spend on appeals, and the inventory and timeliness of different types of appeals. FEMA officials stated in September 2018 that they plan to assess this information to prepare a detailed regional workforce plan. In May 2019, FEMA sent us additional information and documentation involving its analysis of appeal inventory and timeliness. As of October 2019, we are evaluating the information provided by FEMA to determine if they have addressed this recommendation. Contracting workforce shortages. In April 2019, we reported on the federal government’s contracting efforts for preparedness, response, and recovery efforts related to the 2017 hurricanes and California wildfires. We found, among other things, that contracting workforce shortages continue to be a challenge for disaster response and recovery. Further, although FEMA’s 2017 after-action report recommended increasing contract support capacities, it did not provide a specific plan to do so. We also found that while FEMA evaluated its contracting workforce needs in a 2014 workforce analysis, it did not specifically consider contracting workforce needs in the regional offices or address Disaster Acquisition Response Team employees. In our April 2019 report, we recommended, among other things, that FEMA assess its workforce needs—including staffing levels, mission needs, and skill gaps—for contracting staff, to include regional offices and Disaster Acquisition Response Teams, and develop a plan, including timelines, to address any gaps. DHS concurred with this recommendation and estimates that it plans to implement it in the fall of 2019. Assistance to older adults and people with disabilities. In our May 2019 report on FEMA disaster assistance to older adults and people with disabilities following the 2017 hurricanes, we found that FEMA began implementing a new approach to assist individuals with disabilities in June 2018, which shifted the responsibility for directly assisting individuals with disabilities from Disability Integration Advisors—which are staff FEMA deploys specifically to identify and recommend actions needed to support survivors with disabilities—to all FEMA staff. To implement this new approach, FEMA planned to train all of the agency’s deployable staff and staff in programmatic offices on disability issues during response and recovery deployments. According to FEMA, a number of Disability Integration Advisors would also deploy to advise FEMA leadership in the field during disaster response and recovery. We found that while FEMA has taken some initial steps to provide training on the changes, it had not established a plan for delivering comprehensive disability-related training to all staff who will be directly interacting with individuals with disabilities. We recommended, among other things, that FEMA develop a plan for delivering training to FEMA staff that promotes competency in disability awareness and includes milestones and performance measures, and outlines how performance will be monitored. DHS concurred with this recommendation and reported plans to update FEMA’s position task books for all deployable staff with information that promotes competency in disability awareness. In July 2019, officials told us FEMA plans to hire new staff to focus on integrating the disability competency FEMA-wide and work with FEMA’s training components to ensure that disability- related training is consistent with the content of the position task books. We will continue to monitor FEMA’s efforts to address our recommendation. FEMA’s workforce capacity and training. In addition to our prior work on FEMA’s workforce management challenges related to specific programs and functions, we are continuing to evaluate FEMA’s workforce capacity and training efforts during the 2017 and 2018 disaster seasons. Our preliminary observations indicate that there were challenges in FEMA’s ability to deploy staff with the right kinds of skills and training at the right time to best meet the needs of various disaster events. For example, according to FEMA field leadership we interviewed, for some of the functions FEMA performs in the field, FEMA had too few staff with the right technical skills to perform their missions—such as inspections of damaged properties—efficiently and effectively. For other functions, these managers also reported that they had too many staff in the early stages of the disaster, which created challenges with assigning duties and providing on-the-job training. For example, some managers reported that they were allocated more staff than needed in the initial phases of the disaster, but many lacked experience and were without someone to provide direction and mentoring to ensure they used their time efficiently and gained competence more quickly. In focus group discussions and interviews with field managers, FEMA officials told us that difficulties deploying the right mix of staff with the right skills led to challenges such as delays in making purchases to support FEMA operations, problems with properly registering applicants for FEMA programs, or poor communication with nonfederal partners. Nonetheless, FEMA staff have noted that, despite any suboptimal circumstances during disaster response, they aimed to and have been able to find a way to deliver the mission. As part of this ongoing work, FEMA field leadership and managers also reported challenges using agency systems to ensure the availability of the right staff with the right skills in the right place and time. FEMA uses a system called the Deployment Tracking System to, among other things, help identify staff available to be deployed and activate and track deployments. To help gauge the experience level and training needs of its staff, the agency established the FEMA Qualification System (FQS), which is a set of processes and criteria to monitor staff experience in competently performing tasks and completing training that correspond to their job titles. According to the FQS guidance, staff who have been able to demonstrate proficient performance of all the relevant tasks and complete required training receive the designation “qualified,” and are expected to be ready and able to competently fulfill their responsibilities. Those who have not, receive the designation “trainee,” and can be expected to need additional guidance and on-the-job training. FQS designations feed into the Deployment Tracking System as one key variable in how the tracking system deploys staff. Among other challenges with FEMA’s Deployment Tracking System and Qualification System, FEMA managers and staff in the field told us in focus group discussions that an employee’s recorded qualification status was not a reliable indicator of the level at which deployed personnel would be capable of performing specific duties and responsibilities or their general proficiency in their positions, making it more difficult for managers to know the specialized skills or experience of staff and effectively build teams. We are continuing to assess these and other reported workforce challenges and plan to report our findings in spring 2020. Thank you, Chairwoman Titus, Ranking Member Meadows, and Members of the Subcommittee. This concludes my prepared statement. I would be happy to respond to any questions you may have at this time. If you or your staff have any questions concerning this statement, please contact Christopher P. Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement were Joel Aldape (Assistant Director), Bryan Bourgault (Analyst-in-Charge), Amanda Miller, and Heidi Nielson. In addition, Aditi Archer, Anthony Bova, Janice Ceperich, James Cook, Adam Couvillion, Lorraine Ettaro, Eric Hauswirth, Tracey King, Caryn Kuebler, Amy Moran Lowe, Amanda Parker, Sara Pelton, Josephine Perez, Amanda Prichard, Paige Smith, and Johanna Wong made contributions to this statement. Key contributors to the previous work discussed in this statement are listed in each of the cited reports. Highway Emergency Relief: Federal Highway Administration Should Enhance Accountability over Project Decisions. GAO-20-32 (Washington, D.C.: October 17, 2019). Wildfire Disasters: FEMA Could Take Additional Actions to Address Unique Response and Recovery Challenges. GAO-20-5 (Washington, D.C.: October 9, 2019). Puerto Rico Electricity Grid Recovery: Better Information and Enhanced Coordination Is Needed to Address Challenges. GAO-20-141 (Washington, D.C.: October 8, 2019). Disaster Response: HHS Should Address Deficiencies Highlighted by Recent Hurricanes in the U.S. Virgin Islands and Puerto Rico. GAO-19-592 (Washington, D.C.: September 20, 2019). Disaster Response: FEMA and the American Red Cross Need to Ensure Key Mass Care Organizations are Included in Coordination and Planning. GAO-19-526 (Washington, D.C.: September 19, 2019). Disaster Response: Federal Assistance and Selected States and Territory Efforts to Identify Deaths from 2017 Hurricanes. GAO-19-486 (Washington, D.C.: September 13, 2019). Emergency Management: FEMA’s Disaster Recovery Efforts in Puerto Rico and the U.S. Virgin Islands. GAO-19-662T (Washington, D.C.: July 11, 2019). 2017 Disaster Relief Oversight: Strategy Needed to Ensure Agencies’ Internal Control Plans Provide Sufficient Information. GAO-19-479 (Washington, D.C.: June 28, 2019). Emergency Management: FEMA Has Made Progress, but Challenges and Future Risks Highlight Imperative for Further Improvements GAO-19-617T (Washington, D.C.: June 25, 2019). Emergency Management: FEMA Has Made Progress, but Challenges and Future Risks Highlight the Imperative for Further Improvements GAO-19-594T (Washington, D.C.: June 12, 2019). Disaster Assistance: FEMA Action Needed to Better Support Individuals Who Are Older or Have Disabilities. GAO-19-318 (Washington, D.C.: May 14, 2019). Disaster Contracting: Actions Needed to Improve the Use of Post- Disaster Contracts to Support Response and Recovery. GAO-19-281 (Washington, D.C.: April 24, 2019). 2017 Hurricane Season: Federal Support for Electricity Grid Restoration in the U.S. Virgin Islands and Puerto Rico. GAO-19-296 (Washington, D.C.: April 18, 2019). FEMA Grants Modernization: Improvements Needed to Strengthen Program Management and Cybersecurity. GAO-19-164 (Washington, D.C.: April 9, 2019). Disaster Recovery: Better Monitoring of Block Grant Funds Is Needed. GAO-19-232 (Washington, D.C.: March 25, 2019). Puerto Rico Hurricanes: Status of FEMA Funding, Oversight, and Recovery Challenges. GAO-19-256 (Washington, D.C.: March 14, 2019). Huracanes de Puerto Rico: Estado de Financiamiento de FEMA, Supervisión y Desafíos de Recuperación. GAO-19-331 (Washington, D.C.: March 14, 2019). High-Risk Series: Substantial Efforts Needed to Achieve Greater Progress on High-Risk Areas. GAO-19-157SP (Washington, D.C.: March 6, 2019). U.S. Virgin Islands Recovery: Status of FEMA Public Assistance Funding and Implementation. GAO-19-253 (Washington, D.C.: February 25, 2019). 2017 Disaster Contracting: Action Needed to Better Ensure More Effective Use and Management of Advance Contracts. GAO-19-93 (Washington, D.C.: December 6, 2018). Continuity of Operations: Actions Needed to Strengthen FEMA’s Oversight and Coordination of Executive Branch Readiness. GAO-19-18SU (Washington, D.C.: November 26, 2018). Homeland Security Grant Program: Additional Actions Could Further Enhance FEMA’s Risk-Based Grant Assessment Model. GAO-18-354 (Washington, D.C.: September 6, 2018). 2017 Hurricanes and Wildfires: Initial Observations on the Federal Response and Key Recovery Challenges. GAO-18-472 (Washington, D.C.: September 4, 2018). Federal Disaster Assistance: Individual Assistance Requests Often Granted but FEMA Could Better Document Factors Considered. GAO-18-366 (Washington, D.C.: May 31, 2018). 2017 Disaster Contracting: Observations on Federal Contracting for Response and Recovery Efforts. GAO-18-335 (Washington, D.C.: February 28, 2018). Disaster Recovery: Additional Actions Would Improve Data Quality and Timeliness of FEMA’s Public Assistance Appeals Processing. GAO-18-143 (Washington, D.C.: December 15, 2017). Disaster Assistance: Opportunities to Enhance Implementation of the Redesigned Public Assistance Grant Program. GAO-18-30 (Washington, D.C.: November 8, 2017). Climate Change: Information on Potential Economic Effects Could Help Guide Federal Efforts to Reduce Fiscal Exposure. GAO-17-720 (Washington, D.C.: September 28, 2017). Federal Disaster Assistance: Federal Departments and Agencies Obligated at Least $277.6 Billion during Fiscal Years 2005 through 2014. GAO-16-797 (Washington, D.C.: September 22, 2016). Disaster Recovery: FEMA Needs to Assess Its Effectiveness in Implementing the National Disaster Recovery Framework. GAO-16-476 (Washington, D.C.: May 26, 2016). Disaster Response: FEMA Has Made Progress Implementing Key Programs, but Opportunities for Improvement Exist. GAO-16-87 (Washington, D.C.: February 5, 2016). Hurricane Sandy: An Investment Strategy Could Help the Federal Government Enhance National Resilience for Future Disasters. GAO-15-515 (Washington, D.C.: July 30, 2015). Budgeting for Disasters: Approaches to Budgeting for Disasters in Selected States. GAO-15-424 (Washington, D.C.: March 26, 2015). High-Risk Series: An Update. GAO-15-290 (Washington, D.C.: February 11, 2015). Emergency Preparedness: Opportunities Exist to Strengthen Interagency Assessments and Accountability for Closing Capability Gaps. GAO-15-20 (Washington, D.C.: December 4, 2014). Fiscal Exposures: Improving Cost Recognition in the Federal Budget. GAO-14-28 (Washington, D.C.: October 29, 2013). Federal Disaster Assistance: Improved Criteria Needed to Assess a Jurisdiction’s Capability to Respond and Recover on Its Own. GAO-12-838 (Washington, D.C.: September 12, 2012). Opportunities to Reduce Potential Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue. GAO-11-318SP (Washington, D.C.: March 1, 2011). 1. Review of U.S. Virgin Islands recovery planning and progress; 2. Puerto Rico disaster recovery planning and progress; 3. Drinking water and wastewater utility resilience; 4. Disaster and climate change impacts on Superfund sites; 5. FEMA Public Assistance program fraud risk management efforts; 6. Wildland fire collaboration on fuel reduction efforts; 7. Preparedness challenges and lessons learned from the 2017 8. FEMA workforce management and challenges; 9. Small Business Administration response to 2017 disasters; 10. Development of the GAO disaster resilience framework; 11. FEMA Individuals and Households Program operations and 12. National Flood Insurance Program post-flood enforcement; 13. Emergency alerting capabilities and progress; 14. National Flood Insurance Program buyouts and property acquisitions; 15. Economic costs of large-scale natural disasters and impacts on 16. Community Development Block Grants – disaster recovery; 17. Disaster Housing Assistance Program; 18. Contracting workforce and purchase card use for disaster response 19. Power grid and water projects; 20. National Earthquake Hazards Reduction Program (NEHRP); and 21. Disaster resilience and hazard mitigation. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Recent hurricanes, wildfires, and flooding have highlighted the challenges the federal government faces in responding effectively to natural disasters. The 2017 and 2018 hurricanes and wildfires affected millions of individuals and caused billions of dollars in damages. In March 2019, the Midwest experienced historic flooding that affected millions of acres of agriculture and damaged infrastructure. Since 2005, federal funding for disaster assistance is at least $450 billion. Increasing reliance on federal help to address natural disasters is a key source of federal fiscal exposure, particularly as certain extreme weather events become more frequent and intense. This statement discusses, among other things, FEMA's and other federal agencies' progress and challenges related to disaster resilience, recovery programs, and workforce management. This statement is based on GAO reports issued from September 2012 through October 2019, and also includes preliminary observations from ongoing GAO reviews. GAO examined federal laws and documents; interviewed agency officials; and visited disaster damaged areas in California, Florida, South Carolina, North Carolina, Puerto Rico, Texas, and the U.S. Virgin Islands, where GAO also interviewed federal and local officials. GAO's issued and ongoing work has identified progress and challenges in the Federal Emergency Management Agency's (FEMA) and other federal agencies' disaster recovery efforts, as discussed below. Disaster resilience. GAO found that federal and local efforts to improve resilience can reduce the effects and costs of future disasters. FEMA has made progress in this area, but in November 2017, GAO found that more consistent planning could help ensure that rebuilding efforts incorporate hazard mitigation, which would increase the resilience of infrastructure during future disasters. GAO recommended that FEMA take steps to consistently integrate hazard mitigation into its recovery process. FEMA is working to address these recommendations. Managing long-term recovery. GAO's work has shown that federal recovery programs are complicated and can be slow to provide assistance. For example, in October 2019, GAO reported that local officials described onerous documentation requirements in FEMA's Public Assistance program and the unique challenge of removing debris following the 2017 wildfires. GAO recommended that FEMA assess its operations to identify actions to enhance future recovery from severe wildfires. In March 2019, GAO reported that the ad hoc nature of disaster recovery block grants from the Department of Housing and Urban Development delayed the availability of funding. GAO recommended, among other things, that Congress consider permanently authorizing this grant program to meet the needs of disaster survivors in a timely manner. FEMA workforce management. GAO has previously reported on long-standing workforce management challenges, such as ensuring an adequately-staffed and trained workforce to provide effective assistance. For example, GAO reported in September 2018 that the 2017 disasters overwhelmed FEMA's workforce and a lack of trained staff with program expertise led to complications in its response efforts, particularly after Hurricane Maria. While FEMA has taken actions to address several of GAO's workforce management-related recommendations since 2016, a number of recommendations have not yet been implemented. GAO is currently reviewing FEMA's workforce management efforts and lessons learned from the 2017 disasters and will report its findings early next year. GAO has made numerous recommendations in prior reports designed to address the challenges discussed in this statement. Federal agencies have taken steps to address these recommendations and GAO is monitoring agencies' ongoing efforts.", "document_type": "gao"}
{"report": "Our simulations suggest that the sector will likely continue to face a difference between revenues and expenditures during the next 50 years, as measured by its operating balance. We simulated the state and local government sector’s operating balance—a measure of the sector’s ability to cover its current expenditures out of current revenues—to understand the sector’s long-term fiscal outlook based on historical revenue patterns and other assumptions. Because a great majority of states and many local governments are required to balance or nearly balance their operating budgets, the operating balance illustrates the magnitude of fiscal pressures they face. Expenditures and revenues are both simulated to increase as a percentage of gross domestic product (GDP) during the simulation period. However, expenditures are generally expected to grow at a faster rate than revenues, resulting in a declining operating balance (see figure 1). One way of measuring the long-term fiscal challenges faced by the state and local government sector is through an indicator known as the “fiscal gap.” The fiscal gap is an estimate of annual changes in expenditures and in revenues our simulations suggest would be needed to maintain the operating balance equal to zero during the 50-year simulation period. The sector could close the fiscal gap through an increase in revenues, a reduction in expenditures, or a combination of the two of sufficient magnitude. Our simulations suggest that the fiscal gap is about 3.6 percent of GDP over the next 50 years. The sector will need to take actions in annually reducing its expenditures or raising revenues, to achieve fiscal balance. Assuming no change in simulated expenditures, the sector would need to take actions equivalent to increasing its total revenues by 4.2 percent each year to achieve fiscal balance. Alternatively, assuming no change in its simulated revenues, the sector would need to take actions equivalent to decreasing its noninterest expenditures by an amount equal to 3.2 percent of its total expenditures each year. Total expenditure reductions required by the sector are 20.7 percent each year, which includes interest payments on debt that are simulated to be 17.4 percent of annual spending. To eliminate the fiscal gap, the sector would most likely take actions that include a combination of expenditure reductions and revenue increases. Our simulations suggest that growth in the sector’s overall expenditures is largely driven by health care expenditures. Medicaid will likely constitute a growing expenditure for state and local governments. In 2018, Medicaid spending was 2.9 percent of GDP compared to 0.85 percent of GDP for other kinds of health care spending such as non-Medicaid social benefit payments and employee health benefit contributions. At the end of our simulations, Medicaid is simulated to be 4.6 percent of GDP and the other kinds of health care spending are 1.3 percent of GDP. After 2029, Medicaid spending in our simulations is derived from Centers for Medicare & Medicaid Services’ (CMS) projections. On average, Medicaid expenditures are expected to rise by 1 percentage point more than GDP each year over the simulation period. Breaking this down, Medicaid expenditures per capita are expected to increase, on average, about 0.6 percent faster than GDP per capita—referred to as excess cost growth. Excess Cost Growth The extent to which health care costs per capita outpace gross domestic product (GDP) growth per person. As shown in figure 4, health care expenditures are simulated to increase from about 3.94 percent of GDP in 2019 to 5.9 percent of GDP in 2068. In comparison, nonhealth, noninterest expenditures, which include all other operational expenditures other than debt interest payments, will decrease as a share of GDP by 2.74 percentage points over the simulation period. Per capita, national health expenditures, which make up part of the health care expenditures in the figure below, are expected to grow on average 0.8 percent faster than GDP each year during the simulation period, according to CMS. Employee compensation is the largest expenditure for the state and local government sector. It declines from 6.8 percent of GDP in 2018 to 6.1 percent of GDP in 2068. All spending components, including employee compensation, are simulated to increase in actual dollar amounts during our simulation period. Of the spending components included in employee compensation, only health benefits for employees and retirees increase as a share of employee compensation. In contrast, wages and salaries, pension contributions, and other forms of compensation decrease as a share of employee compensation (see figure 5). These percentages reflect a simulated decrease in state and local government employees’ compensation as a share of GDP. Our simulations suggest that spending on health benefits for state and local government employees and retirees is likely to rise, on average, by 0.9 percentage points more than GDP each year. Similar to the growth in Medicaid spending, growth in spending for these health benefits is due to an increase in the simulated number of employees and retirees enrolled as well as an increase in the simulated amount of health benefits for each employee and retiree. According to our simulations, if employee and retiree health benefits follow trends in overall national health spending, they will likely make up an increasingly larger share of total employee compensation going forward. Our simulations suggest that annual contributions to state and local government employee pension plans will need to remain at their historical 10-year average of 12.9 percent of wages and salaries for state and local governments to meet their long-term pension obligations. Prior to the last decade, from 1999 to 2008 the state and local government sector averaged about an 8 percent contribution rate, which was lower than what our current simulations show is necessary for meeting pension obligations. State and local government contributions to employee pension plans are simulated to decline as a share of GDP, as are wages and salaries of state and local government employees. Our simulations suggest that federal grants will increase slightly as a share of GDP. The largest grant receipts are for Medicaid which will likely grow more quickly than other types of federal grants making up an increasing share of revenues in the future (see figure 6). The increase in Medicaid expenditures simulated during this period will likely put increasing pressure on both federal and state governments. As a matching formula grant program, the simulated increase in federal Medicaid grants implies an expected increase in Medicaid expenditures that will be shared by state governments. Federal investment grants (i.e., grants intended to finance capital infrastructure investments) and other federal grants unrelated to Medicaid (i.e., grants intended to finance education, social services, housing, and community investment) are simulated to decline as a share of GDP. Further, our simulations suggest that if historical relationships between state and local governments’ tax revenues and tax bases persist, total tax revenues for the state and local government sector will increase from 8.7 percent of GDP in 2019 to 9.1 percent of GDP by the end of the simulation period. As shown in figure 7, the different components of total tax revenues are simulated to remain fairly consistent or slightly increase. The simulations suggest that personal income tax revenues will increase as a share of GDP by about 0.5 percentage points during the simulation period. Sales tax is expected to decrease by approximately 0.2 percentage points and property taxes are simulated to slightly increase as a share of GDP through 2068 from 2.73 percent to 2.86 percent. Sensitivity Analysis An analysis using alternative assumptions of one variable to determine the uncertainty, or sensitivity, of another variable. Several factors, or key model variables, could affect the state and local government sector’s long-term fiscal outlook, including economic growth, health care excess cost growth, and the rate of return on pension assets. To see how the outlook changes in response to them, we developed sensitivity analyses—simulations that use alternative assumptions about their growth. For each of these key variables we use a baseline assumption, a higher-than-baseline assumption, and a lower-than- baseline assumption. We determined that these alternative assumptions highlighted the operating balance’s sensitivity to changes, shifting the future fiscal outcomes for the sector. Future trends in GDP growth could affect the state and local government sector’s fiscal outlook. In our simulations, GDP growth is based on the most recent data from the Congressional Budget Office (CBO) and the Board of Trustees of the Federal Old-Age and Survivors Insurance and Federal Disability Insurance Trust Funds (OASDI Trustees) which project real GDP (adjusted for inflation) to grow by 1.9 percent per year on average from 2018 through 2029, and by 2 percent per year on average after 2029. Using these projections, our simulations suggest that maintaining current policies would cause the sector’s operating balance to become increasingly negative. Using the OASDI Trustees’ alternative assumptions of real GDP growth at a faster rate—2.7 percent—suggests that the operating balance, while remaining negative, would have an improved outlook compared to the baseline. While growth in revenue and health care spending is largely tied to GDP in our simulations, spending for other components is tied to inflation and population growth and grows more slowly than GDP. As such, increases in GDP growth improve the sector’s outlook. Our simulations, using the OASDI Trustees’ alternative assumptions also show that if GDP were to grow at a slower rate—1.4 percent—the difference between revenues and expenditures would expand, resulting in an increasingly negative operating balance (see figure 8). Excess cost growth in health care is another key determinant of the sector’s fiscal balance. In our simulations Medicaid spending per capita grows about 1.8 percent faster than GDP per capita on average for the period from 2020 through 2029. Medicaid spending per capita grows about 0.6 percent faster on average from the period from 2030 through 2068. Other health expenditures per capita grow about 0.8 percent faster than GDP per capita for the period from 2019 through 2068. Using these projections, our simulations suggest that maintaining current policies will cause the sector’s expenditures to exceed its revenues and this difference will become increasingly negative during the next several decades. The simulations developed assuming zero excess cost growth in Medicaid and national health expenditures suggest that spending would be lower as a share of GDP. The difference between revenues and expenditures would be significantly less negative than the baseline simulations around the middle of the simulation period before stabilizing, but remain negative over the simulation period. In the scenario using the alternative projections from CMS where excess cost growth rises faster— 0.6 percent on average for Medicaid for the period from 2030 through 2068 and 0.9 percent for national health expenditures for the period between 2019 through 2068—our simulations show that the difference between revenues and expenditures would persist for the remainder of the simulation period (see figure 9). The rate of return on pension assets could also affect the state and local government sector’s fiscal outlook. Based on an inflation-adjusted rate of return on pension assets of 5 percent, our simulations suggest that state and local governments would need to make pension contributions equivalent to about 13 percent of employees’ wages and salaries to meet their long-term pension obligations. The simulations we developed using a higher rate of return—7.5 percent—suggest that pension contributions would be about 3 percent of state and local government employees’ wages and salaries to meet pension obligations. Under this scenario, spending would need to be a lower share of GDP and the sector’s outlook would improve. The difference between revenues and expenditures would briefly narrow early on before becoming increasingly negative through the remainder of the simulation period. Alternatively, we estimated that if the rate of return on pension assets is relatively low—2.5 percent—required pension contributions would need to be about 24 percent of state and local government employees’ wages and salaries. Under this scenario our simulations show that spending would be a higher share of GDP and the sector’s outlook would worsen as the sector’s negative operating balance would continue to grow larger (see figure 10). This report was prepared under the direction of Michelle A. Sager, Director, Strategic Issues, who can be reached at 202-512-6806 or sagerm@gao.gov, and Oliver M. Richard, Director, Center for Economics, who can be reached at 202-512-8424 or richardo@gao.gov if there are any questions. GAO staff who made key contributions are listed in appendix II. This update of the state and local government fiscal model used aggregate data on the state and local government sector and national data on other variables from the following sources: the Agency for Healthcare Research and Quality, Bloomberg, the Board of Governors of the Federal Reserve System, the Board of Trustees of the Federal Old- Age, Survivors, and Disability Insurance Program (OASDI Trustees), BEA, the Bureau of Labor Statistics, the Census Bureau, the Centers for Medicare & Medicaid Services (CMS), the Congressional Budget Office (CBO), and the Social Security Administration. These data sources are generally the same data sources we used for our prior update. We used annual observations on historical data through 2018 where available. This report updates GAO’s state and local fiscal model to simulate the fiscal outlook for the state and local government sector. This includes identifying the factors that are likely to contribute to the state and local sector’s fiscal imbalance. The level of receipts and expenditures for the state and local government sector as a whole in future years is based on current and historical spending and revenue patterns. We used Table 3.3 of the National Income and Product Accounts (NIPA)—State and Local Government Current Receipts and Expenditures—prepared by BEA as an organizing framework for developing our model, and we simulated state and local government receipts and expenditures using methods similar to those we have used in prior updates. Our simulations of real U.S. gross domestic product (GDP) were consistent with the growth path developed by CBO for the period from 2019 through 2029 and by the OASDI Trustees for the period thereafter. Our simulations of U.S. population was consistent with the growth path developed by the OASDI Trustees, and our simulations of excess cost growth for national health expenditures and for Medicaid were consistent with CMS projections, all for the entire simulation period. Our simulations of other variables, such as the GDP price index, personal income, and 3-month U.S. Department of the Treasury (Treasury) rates, were consistent with the growth paths for these variables developed by CBO for as much of the simulation period as possible. Otherwise, we developed our own assumptions about the likely future growth path of the variables in our model. In general, we assumed that current policies remain in place and that all levels of government continue to provide services at current per capita levels. A detailed description of the model is in appendix I of GAO, State and Local Governments’ Fiscal Outlook: 2018 Update, GAO-19-208SP (Washington, D.C.: December 2018). We describe below where we updated equations or added equations to the model. Otherwise our approach is the same as the approach we used in that update. We simulated the future growth paths of the following types of state and local government revenues: current tax receipts, contributions to government social insurance, income on financial assets, current transfer receipts, the surplus from government enterprises, and capital transfer receipts. We also simulated the future growth path of state and local government long-term debt issuance. We updated some of the equations we used to simulate tax receipts(see table 1). We also added equations to simulate current transfers from the rest of the world to state and local governments, disaster-related insurance benefits to state and local governments, and other capital transfers to state and local governments, which we had not included in prior updates. The equations we used to simulate the other types of receipts are the same as the equations we used in GAO-19-208SP. We simulated the future growth paths of the following types of state and local government expenditures: consumption expenditures, current transfer payments, interest paid on outstanding state and local government debt, subsidies, capital outlays, and consumption of fixed assets (depreciation). We also simulated the future growth path of the state and local government sector’s net social insurance fund balance. We updated some of the equations we used to simulate the interest paid on outstanding state and local government debt (see table 1 above). We also added equations to simulate current transfer payments to the rest of the world, which we had not included in prior updates. Otherwise, the approach we used to simulate expenditures is the same as the approach we used in GAO-19-208SP. Our main indicator of the sector’s fiscal balance is its operating balance net of funds for capital expenditures (henceforth, operating balance), which is a measure of the sector’s ability to cover its current expenditures out of current receipts. Operating balance is defined as total receipts minus (1) capital outlays not financed by long-term debt issuance, (2) current expenditures less depreciation, (3) current surplus of state and local government enterprises, and (4) net social insurance fund balance. We also estimated the annual changes in spending and in receipts that our simulations suggest would be needed to maintain the operating balance equal to zero during the 50-year simulation period, which we refer to as the “fiscal gap.” As discussed above, our baseline simulations assume that current policies remain in place and that all levels of government continue to provide services at current per capita levels. We then simulated the change in total expenditures needed to maintain the operating balance equal to zero. To estimate the annual change in spending needed to maintain balance we calculated the present value of that change as a percentage of the present value of baseline total expenditures and as a percentage of the present value of U.S. GDP, all for a 50-year period. We also calculated the interest and non-interest expenditure components of the change in total expenditures needed to maintain balance. We used a similar approach to estimate the annual change in total receipts needed to maintain balance. We assessed the sensitivity of our baseline results to alternative projections of real U.S. GDP growth, health care excess cost growth, and the real rate of return on state and local government pension fund assets. Following the same approach we used in GAO-19-208SP, for each of these variables, we selected an alternative projection associated with faster growth or rate of return and one associated with slower growth or rate of return. Real U.S. GDP. For our baseline simulations, we used CBO projections of real GDP for the period from 2019 through 2029 and the OASDI Trustees’ intermediate projections of real U.S. GDP growth for the years thereafter. For our sensitivity analysis, we used the OASDI Trustees’ high-cost and low-cost projections. Health care excess cost growth. For our baseline simulations, we used CMS’s baseline projection of national health expenditures excess cost growth and we estimated Medicaid excess cost growth based on CMS’s baseline projections. For our sensitivity analysis, we used CMS’s alternative projection of national health expenditures excess cost growth and we estimated Medicaid excess cost growth based on CMS’s alternative projections. As another alternative, we simulated the model assuming both zero excess cost growth for national health expenditures and Medicaid. Our simulations used CBO’s projection of federal spending on Medicaid, CHIP, and exchange subsidies as a fraction of GDP to simulate certain variables related to state and local government spending on Medicaid and other health spending. This projection incorporates excess cost growth for the period from 2019 through 2029 but assumes zero excess cost growth starting in 2030, so we could only vary Medicaid excess cost growth in the alternative simulations for 2030 and later. Real rate of return on state and local government pension assets. For our baseline simulations, we assumed a 5 percent real rate of return on state and local government pension assets. For our sensitivity analysis, we used 2.5 percent and 7.5 percent. Table 2 shows the growth rates or rates of return associated with the baseline and alternative projections of each variable for the simulation period. We simulated the model changing either real U.S. GDP growth, health care excess cost growth, or the real rate of return on pension assets, leaving the other variables fixed at their baseline values. Thus, our sensitivity analysis is in the spirit of a partial equilibrium comparative statics analysis that sheds light on how each of the individual variables may affect the state and local government sector’s fiscal outlook. However, these variables are likely to be correlated, so future changes in one would likely be associated with changes in others. Our approach has a number of limitations and the results should be interpreted with caution. First, the state and local fiscal model is not designed for certain types of analyses. The simulations are not intended to provide precise predictions. Even though we know that these governments regularly make changes in tax laws and expenditures, the model essentially holds current policies in place and analyzes the fiscal future for the sector as if those policies were maintained because it would be highly speculative to make any assumptions about future policy adjustments. In addition, fiscal outcomes related to the sector’s financial position and solvency may not reflect all aspects of the sector’s “health.” Other indicators include economic indicators that go beyond the sector’s financial position to include economic growth, income, or distributional equity, as well as indicators of the quality of services provided by the sector, including education, health care, infrastructure, and other public goods and services. Finally, our unit of analysis is the state and local government sector as a whole, so our results provide an assessment of the sector’s fiscal outlook. However, individual state and local governments likely exhibit significant heterogeneity in their expenditure and revenue patterns and their fiscal outlooks will likely differ from the sector as a whole. Nevertheless, it is informative to assess the overall fiscal outlook because doing so reveals the outlook for state and local governments as a sector. In addition, aggregate data on the sector is available on a more timely basis than data for individual state and local governments, allowing for a better assessment of the sector’s current fiscal outlook. Our results for the sector also provide a baseline from which to view the experiences of individual state and local governments. Finally, assessing the fiscal outlook of the sector as a whole can help mitigate the tendency to extrapolate from the most visible, but potentially not representative, experiences of individual states or localities. In addition to the contact named above, Peter Del Toro, Courtney LaFountain, Melissa Wolf (Assistant Directors), Silvia Symber (Analyst-in- Charge), Shelby Clark, Amalia Konstas, Dylan Stagner, Frank Todisco, Walter Vance, and Alicia White made significant contributions to this report. State and Local Governments’ Fiscal Outlook: December 2018 Update, GAO-19-208SP. Washington, D.C.: Dec. 12, 2018. State and Local Governments’ Fiscal Outlook: December 2016 Update, GAO-17-213SP. Washington, D.C.: Dec. 8, 2016. State and Local Governments’ Fiscal Outlook: December 2015 Update, GAO-16-260SP. Washington, D.C.: Dec. 16, 2015. State and Local Governments’ Fiscal Outlook: December 2014 Update, GAO-15-224SP. Washington, D.C.: Dec. 17, 2014. State and Local Governments’ Fiscal Outlook: April 2013 Update, GAO-13-546SP. Washington, D.C.: Apr. 29, 2013. State and Local Governments’ Fiscal Outlook: April 2012 Update, GAO-12-523SP. Washington, D.C.: Apr. 5, 2012. State and Local Government Pension Plans: Economic Downturn Spurs Efforts to Address Costs and Sustainability, GAO-12-322. Washington, D.C.: Mar. 2, 2012. State and Local Governments’ Fiscal Outlook: April 2011 Update, GAO-11-495SP. Washington, D.C.: Apr. 6, 2011. State and Local Governments: Knowledge of Past Recessions Can Inform Future Federal Fiscal Assistance, GAO-11-401. Washington, D.C.: Mar. 31, 2011. State and Local Governments: Fiscal Pressures Could Have Implications for Future Delivery of Intergovernmental Programs, GAO-10-899. Washington, D.C.: July 30, 2010. State and Local Governments’ Fiscal Outlook: March 2010 Update, GAO-10-358. Washington, D.C.: Mar. 2, 2010. Update of State and Local Government Fiscal Pressures, GAO-09-320R. Washington, D.C.: Jan. 26, 2009. State and Local Fiscal Challenges: Rising Health Care Costs Drive Long- term and Immediate Pressures, GAO-09-210T. Washington, D.C.: Nov. 19, 2008. State and Local Governments: Growing Fiscal Challenges Will Emerge during the Next 10 Years, GAO-08-317. Washington, D.C.: Jan. 22, 2008. Our Nation’s Long-Term Fiscal Challenge: State and Local Governments Will Likely Face Persistent Fiscal Challenges in the Next Decade, GAO-07-1113CG. Washington, D.C.: July 18, 2007. State and Local Governments: Persistent Fiscal Challenges Will Likely Emerge within the Next Decade, GAO-07-1080SP. Washington, D.C.: July 18, 2007.", "summary": "Fiscal sustainability presents a national challenge shared by all levels of government. Since 2007, GAO has published simulations of long-term fiscal trends in the state and local government sector, which have consistently shown that the sector faces long-term fiscal pressures. While a great majority of states have requirements related to balancing their budgets, deficits can arise for reasons including planned annual revenues are not generated at the expected rate, demand for services exceeds planned expenditures, or both, resulting in a near-term operating deficit. This report updates GAO's state and local fiscal model to simulate the fiscal outlook for the state and local government sector. This includes identifying the components of state and local expenditures likely to contribute to the sector's fiscal pressures as well as the effects of revenue changes on the sector's outlook. GAO's model uses the Bureau of Economic Analysis's National Income and Product Accounts as the primary data source and presents the results in the aggregate for the state and local government sector as a whole. The model shows the expected level of receipts and expenditures for the sector until 2068, based on historical spending and revenue patterns. In addition, the model assumes that the current set of policies in place across state and local governments remains constant to show a simulated long-term outlook. Because the model covers the sector in the aggregate, the fiscal outcomes for individual states and localities cannot be identified. GAO's simulations suggest that state and local governments will likely face an increasing difference between expenditures and revenues during the next 50 years as reflected by the operating balance—a measure of the sector's ability to cover its current expenditures out of its current revenues. While both expenditures and revenues are projected to increase as a percentage of United States' gross domestic product (GDP), a difference between the two is projected to persist because expenditures are expected to grow faster than revenues throughout the simulation period. The sector would need to make changes to avoid fiscal imbalance and assure that revenues are at least equal to expenditures. GAO's simulations suggest that growth in the sector's overall expenditures is largely driven by health care, with states' share of Medicaid spending as the primary driver. These expenditures are projected to grow more than GDP each year. Employee compensation, the largest share of operating expenditures, decreases as a share of GDP during the simulation period. Health benefits are the only component of employee compensation that increase as a percentage of GDP. Revenues from federal grants to states and localities are also expected to increase during the simulation period, in part because of Medicaid grants to states. GAO also conducts sensitivity analyses to see how the sector's outlook changes when using alternative assumptions of key model variables – economic growth, health care excess cost growth, and the real rate of return on pension assets. Using these alternative assumptions highlights the operating balance's sensitivity to changes and possible shifts in the future fiscal outcomes for the sector.", "document_type": "gao"}
{"report": "Private student loans are not guaranteed by the federal government. Generally, private lenders underwrite the loans based on the borrower’s credit history and ability to repay, and they often require a cosigner. Private student loans generally carry a market interest rate, which can be a variable rate that is higher than that of federal student loans. As of September 30, 2018, five banks held almost half of all private student loan balances. Other private student loan lenders include credit unions and nonbanks: Credit unions originate private student loans either directly or indirectly through a third party. Nonbanks include both for-profit nonbank lenders and nonbank state lenders. For-profit nonbank lenders can originate, service, refinance, and purchase loans. Nonbank state lenders promote affordable access to education by generally offering low, fixed-rate interest rates and low or no origination fees on student loans. As of September 2018, outstanding private student loan balances made up about 8 percent of the $1.56 trillion in total outstanding student loans (see fig. 1). The volume of new private student loans originated has fluctuated, representing about 25 percent of all student loans originated in academic year 2007–2008, 7 percent in 2010–2011 (after the financial crisis), and 11 percent in 2017–2018. FCRA, the primary federal statute that governs consumer reporting, is designed to promote the accuracy, fairness, and privacy of information in the files of CRAs. FCRA, and its implementing regulation, Regulation V, govern the compilation, maintenance, furnishing, use, and disclosure of consumer report information for credit, insurance, employment, and other eligibility decisions made about consumers. The consumer reporting market includes the following entities: CRAs assemble or evaluate consumer credit information or other consumer information for the purpose of producing consumer reports (commonly known as credit reports). Equifax, Experian, and TransUnion are the three nationwide CRAs. Data furnishers report information about consumers’ financial behavior, such as repayment histories, to CRAs. Data furnishers include credit providers (such as private student loan lenders), utilities, and debt collection agencies. Credit report users include banks, employers, and others that use credit reports to make decisions on an individual’s eligibility for products and services such as credit, employment, housing, and insurance. FCRA imposes duties on data furnishers with respect to the accuracy of the data they furnish. Data furnishers are required to, among other things, refrain from providing CRAs with information they know or have reasonable cause to believe is inaccurate and develop reasonable written policies and procedures regarding the accuracy of the information they furnish. The Act entitles financial institutions that choose to offer a private student loan rehabilitation program that meets the Act’s requirements a safe harbor from potential inaccurate information claims under FCRA related to the removal of the private student loan default from a credit report. To assist data furnishers in complying with their responsibilities under FCRA, the credit reporting industry has adopted a standard electronic data-reporting format called the Metro 2® Format. This format includes standards on how and what information furnishers should report to CRAs on private student loans. The information that private student loan lenders furnish to CRAs on their borrowers includes consumer identification; account number; date of last payment; account status, such as in deferment, current, or delinquent (including how many days past due); and, if appropriate, information indicating defaults. An account becomes delinquent on the day after the due date of a payment when the borrower fails to make a full payment. Private student loan lenders’ policies and terms of loan contracts generally determine when a private student loan is in default. While private student loan lenders may differ in their definitions of what constitutes a default, federal banking regulator policy states that closed- end retail loans (which include private student loans) that become past due 120 cumulative days from the contractual due date should be classified as a loss and “charged off.” Private student loan lenders can indicate that a loan is in default and they do not anticipate being able to recover losses on it by reporting to CRAs one of a number of Metro 2® Format status codes. Participation in a private student loan rehabilitation program entitles borrowers who successfully complete the program to request that the indicator of a student loan default be removed from their credit report, but the delinquencies leading up to the default would remain on the credit report. Figure 2 shows an example of credit reporting for a borrower who defaults on a private student loan and completes a rehabilitation program. A credit score is a measure that credit providers use to predict financial behaviors and is typically computed using information from consumer credit reports. Credit scores can help predict the likelihood that a borrower may default on a loan, file an insurance claim, overdraw a bank account, or not pay a utility bill. FICO and VantageScore are the two firms that develop credit score models with nationwide coverage. FICO develops credit score models for distribution by each of the three nationwide CRAs, whereas VantageScore’s models are developed across the three CRAs resulting in a single consistent algorithm to assess risk. FICO and VantageScore each have their own proprietary statistical credit score models that choose which consumer information to include in calculations and how to weigh that information. The three nationwide CRAs also develop credit score models derived from their own data. There are different types of credit scores, including generic, industry- specific, and custom. Generic scores are based on a representative sample of all individuals in a CRA’s records, and the information used to predict repayment is limited to the information in consumer credit records. Generic scores are designed to predict the likelihood of a borrower not paying as agreed in the future on any type of credit obligation. Both FICO and VantageScore develop generic credit scores. FICO and VantageScore generic scores generally use a range from 300 to 850, with higher numbers representing lower credit risk. For example, VantageScore classifies borrowers in the following categories: subprime (those with a VantageScore of 300–600), near prime (601–660), prime (661–780), and super prime (781–850). A prime borrower is someone who is considered a low-risk borrower and likely to make loan payments on time and repay the loan in full, whereas a subprime borrower has a tarnished or limited credit history. FICO and VantageScore generic scores generally use similar elements in determining a borrower’s credit score, including a borrower’s payment history, the amounts owed on credit accounts, the length of credit history and types of credit, and the number of recently opened credit accounts and credit inquiries. FICO has developed industry-specific scores for the mortgage, automobile finance, and credit card industries. These scores are designed to predict the likelihood of not paying as agreed in the future on these specific types of credit. In addition, credit providers sometimes use custom credit scores instead of, or in addition to, generic credit scores. Credit providers derive custom scores from credit reports and other information, such as account history, from the lender’s own portfolio. The scores can be developed internally by credit providers or with the assistance of external parties such as FICO or the three nationwide CRAs. CFPB has supervisory authority over certain private student loan lenders, including banks and credit unions with over $10 billion in assets and all nonbanks, for compliance with Federal consumer financial laws. CFPB also has supervisory authority over the largest CRAs and many of the entities that furnish information about consumers’ financial behavior to CRAs. To assess compliance with Federal consumer financial laws, CFPB conducts compliance examinations. According to CFPB, because of its mission and statutory requirement regarding nonbank supervision, it prioritizes its examinations by focusing on risks to consumers rather than risks to institutions. Given the large number, size, and complexity of the entities under its authority, CFPB prioritizes its examinations by focusing on individual product lines rather than all of an institution’s products and services. CFPB also has enforcement authority under FCRA regarding certain banks, credit unions, and nonbanks and broad authority to promulgate rules to carry out the purposes of FCRA. The prudential regulators—FDIC, Federal Reserve, NCUA, and OCC— oversee all banks and most credit unions that offer private student loans. Their oversight includes routine safety and soundness examinations for all regulated entities. These examinations may include a review of operations, including policies, procedures, and practices, to ensure that private student loans are not posing a risk to the entities’ safety and soundness. Prudential regulators also have supervisory authority for FCRA compliance for banks and certain credit unions with $10 billion or less in assets. As of January 2019, none of the five banks with the largest private student loan portfolios that we contacted offered rehabilitation programs for defaulted private student loans. In addition, officials from the federal banking regulators told us that as of March 2019, no banks had submitted applications to have rehabilitation programs approved. Representatives from three of the five banks we contacted told us they had decided not to offer a rehabilitation program, and the other two had not yet made a final determination. Representatives from these five banks provided several reasons they were not offering rehabilitation programs for private student loans. Low delinquency and default rates. All five banks’ representatives stated that they had low default rates for private student loans, so the demand for these programs would be low for each bank. Availability of predefault payment programs. Representatives of all five banks said they already offer alternative payment programs, such as forbearance, to help prevent defaults, and two of them explicitly noted this as a reason that a rehabilitation program was unnecessary. Operational uncertainties. Most of the banks’ representatives were not sure how they would operationalize rehabilitation programs. One bank’s representatives said that they sell defaulted loans to debt purchasers and that it would be difficult to offer rehabilitation programs for loans that had been sold. Representatives of two other banks said that the banks’ systems are not able to change the status of a loan once it has defaulted, so they were not certain how their systems would track rehabilitated loans. Another bank’s representatives said that they did not know how rehabilitated loans would be included for accounting purposes in developing their financial statements. Reduced borrower incentives to avoid default. Representatives from two banks said they believed the option to rehabilitate a defaulted loan might reduce borrowers’ incentives to avoid default or to enter a repayment program before default. Risk of compliance violations. One bank representative said a rehabilitation program could put the bank at risk for violations of unfair and deceptive acts and practices if borrowers misunderstood or misinterpreted how much the program would improve their credit scores. Representatives from this bank and another explained that they did not know how much the program would improve credit scores, limiting their ability to describe the program’s benefit to borrowers. Representatives from three of these banks and other organizations, however, noted that there could be advantages for banks to offer private student loan rehabilitation programs. Representatives from the banks said these programs could help banks recover some nonperforming debt, and one of these representatives stated the program could be marketed to borrowers as a benefit offered by the bank. A representative of a consumer advocacy group said a rehabilitation program could improve a bank’s reputation by distinguishing the bank from peer institutions that do not offer rehabilitation for private student loans. Because NCUA is not one of the federal banking regulators by statutory definition, officials said the Act does not require credit unions to seek approval from the agency before offering a rehabilitation program. NCUA officials told us examiners would likely review private student loan rehabilitation programs for the credit unions that choose to offer them as part of normal safety and soundness examinations. The two credit unions we spoke with—which are among the largest credit union providers of private student loans—told us they do not plan to offer rehabilitation programs. One of these credit unions cited reasons similar to those offered by banks, including a low private student loan default rate that suggested there would be a lack of demand for a rehabilitation program. The other credit union explained that it was worried about the effect of removing defaults from credit reports on its ability to make sound lending decisions. NCUA officials also noted that as of January 2019, they had not received any inquiries from credit unions about these programs. OCC, FDIC, and the Federal Reserve have issued information regarding the availability of private student loan rehabilitation programs to their regulated entities, including how they would review applications. In doing so, the agencies informally coordinated to ensure that the statements issued would contain similar information on rehabilitation programs. The three agencies’ statements explained that their regulated entities must receive written approval to begin a program and that the relevant agency would provide feedback or notify them of its decision within 120 days of receiving a written application. The agencies will review the proposed program to ensure that it requires borrowers to make a minimum number of consecutive, on-time, monthly payments that demonstrate renewed ability and willingness to repay the loan. Uncertainty exists regarding two issues with private student loan rehabilitation programs. First, some nonbank private student loan lenders are not certain that they have the authority to implement these programs. Second, the Act does not explain what constitutes a “default” for the purposes of removing information from credit reports. With regard to nonbank state lenders, uncertainty exists about their authority under FCRA to offer private student loan rehabilitation programs that include removing information from credit reports. As discussed previously, for financial institutions such as banks and credit unions, the Act provides an explicit safe harbor to request removal of a private student loan default from a borrower’s credit report and remain in compliance with FCRA. However, the Act does not specify that for-profit nonbank lenders and nonbank state lenders have this same authority. Representatives of the five nonbank state lenders we spoke with had different interpretations of their authority to offer rehabilitation programs. At least two nonbank state lenders currently offer rehabilitation programs, and their representatives told us they believed they have the authority to do so. Another nonbank state lender told us its state has legislation pending to implement such a program. In contrast, representatives of two other nonbank state lenders told us they were interested in offering a rehabilitation program but did not think that they had the authority to do so. In addition, representatives from a trade association that represents nonbank state lenders noted that confusion exists among some of their members and they are seeking a way to obtain explicit authority for nonbank lenders to offer rehabilitation programs for their private student loans. Two trade associations that represent nonbank state lenders also told us that some of their members would be interested in offering these programs if it was made explicit that they were allowed to do so. CFPB officials told us the agency has not made any determination on whether it plans to clarify for nonbanks—including for-profit nonbank lenders and nonbank state lenders—if they have the authority under FCRA to have private student loan defaults removed from credit reports for borrowers who have completed a rehabilitation program. CFPB officials said that the agency does not approve or prevent its regulated entities from offering any type of program or product. Unlike for the federal banking regulators, the Act did not require CFPB to approve rehabilitation programs offered by the entities it regulates. However, CFPB does have general FCRA rulemaking authority. It generally also has FCRA enforcement and supervisory responsibilities over its regulated entities, which includes certain entities that originate private student loans. This authority allows the agency to provide written clarification of provisions or define terms as needed. As a result, CFPB could play a role in clarifying for nonbanks whether they are authorized under FCRA to offer private student loan rehabilitation programs. Federal internal control standards state that management should externally communicate the necessary quality information to achieve the entity’s objectives. Without clarification from CFPB on nonbanks’ authority to offer private student loan rehabilitation programs that allow them to delete information from the borrower’s credit report, there will continue to be a lack of clarity on this issue among these entities. Providing such clarity could—depending on CFPB’s interpretation—result in additional lenders offering rehabilitation programs that would allow more borrowers the opportunity to participate, or it could help ensure that only those entities CFPB has interpreted as being eligible to offer programs are doing so. Statutory changes made to FCRA by the Act do not explain what information on a consumer’s credit report constitutes a private student loan “default” that may be removed when a borrower successfully completes a rehabilitation program. According to the three nationwide CRAs and a credit reporting trade association, the term “default” is not used in credit reporting for private student loans. As discussed previously, private student loan lenders use one of a number of Metro 2® Format status codes to indicate that a loan is in default (i.e., they do not anticipate being able to recover losses on the loan). Representatives of the CRAs and a credit reporting trade association said that private student loan lenders will need to make their own interpretation of what information constitutes a default for the purposes of removing information from a credit report following successful completion of a private student loan rehabilitation program. The statements issued by FDIC, the Federal Reserve, and OCC on rehabilitation programs do not explain what information constitutes a private student loan “default” that may be removed from borrowers’ credit reports upon successful completion of a rehabilitation program. Officials from FDIC, the Federal Reserve, and OCC explained that they do not have the authority to interpret what constitutes a private student loan default on credit reports because the responsibilities for interpreting FCRA fall under CFPB. CFPB officials told us they are monitoring the issue but have not yet determined if there is a need to address it. Given CFPB’s rulemaking authority for FCRA, it could clarify the term “default” for private student loan lenders. In doing so, CFPB could obtain insight from the prudential regulators and relevant industry groups on how private student loan lenders currently report private student loan defaults on credit reports and on how to develop a consistent standard for what information may be removed. According to federal internal control standards, management should externally communicate the necessary quality information to achieve objectives. This can include obtaining quality information from external parties, such as other regulators and relevant industry groups. Without clarification from CFPB, there may be differences among private student loan lenders in what information they determine constitutes a “default” and may be removed from a credit report. Variations in lenders’ interpretations could have different effects on borrowers’ credit scores and credit records, resulting in different treatment of borrowers by credit providers. This could affect borrowers’ access to credit or the terms of credit offered, such as interest rates or the size of down payments required on a variety of consumer loans. In addition, as mentioned previously, the credit reporting industry follows a standard reporting format to help ensure the most accurate credit reporting information possible. Without clarification on what information may be removed from credit reports following successful completion of rehabilitation programs, differences in lenders’ interpretation could introduce inconsistencies in credit reporting data that may affect their accuracy. Rehabilitation programs for private student loans are expected to pose minimal additional risk to banks’ and credit unions’ safety and soundness. Prudential regulators require that banks and credit unions underwrite student loans to mitigate risks and ensure sound lending practices, and OCC guidance specifies that underwriting practices should minimize the occurrence of defaults and the need for repayment assistance. Lenders generally use underwriting criteria based on borrowers’ credit information to recognize and account for risks associated with private student loans. According to officials from OCC, FDIC, and the Federal Reserve and representatives from the major bank and credit union private student loan lenders we spoke with, lenders participating in private student loan rehabilitation programs would face minimal additional risks for several reasons: Loans are already classified as a loss. Loans entering a rehabilitation program are likely to be 120 days past due and to have been charged off, and thus they would have already been classified as a loss by banks and credit unions. OCC officials told us a program to rehabilitate these loans would, therefore, pose no additional risks to the safety and soundness of institutions that offer them. Default rates are low, and loans typically use cosigners. Representatives from the five major banks and two credit unions told us that private student loans generally perform well and have low rates of delinquencies and defaults. Aggregate data on the majority of outstanding loan balances show that the default rate for private student loans was below 3 percent from the second quarter of 2014 through the third quarter of 2018. Lenders also generally require borrowers of private student loans to have cosigners—someone who is liable to make payments on the loan should the student borrower default—which helps reduce the risk of the loan not being repaid. Since the academic year 2010–2011, the rate of undergraduate private student loan borrowers with cosigners has exceeded 90 percent. Private student loan portfolios are generally small. Private student loans make up a small portion of the overall loan portfolios for most of the banks and credit unions we spoke with. For four of the five major banks with the largest portfolios of private student loans, these constituted between about 2 percent to 11 percent of their total loan portfolio in 2017. The fifth bank’s entire portfolio was education financing, with private student loans accounting for about 93 percent of its 2017 portfolio. For the two credit unions we contacted, private student loans constituted about 2 percent and 6 percent of their total assets in 2018. Private student loan rehabilitation programs may create certain operational costs for banks or credit unions that offer them. However, no representatives of the five banks and two credit unions with whom we spoke were able to provide a cost estimate since none had yet designed or implemented such a program. Representatives from four banks and one credit union we spoke with said that potential costs to implement a rehabilitation program would be associated with information technology systems, designing and developing new systems to manage the program, increased human resource needs, additional communications with borrowers, credit reporting, compliance, monitoring, risk management, and any related legal fees. In addition, like any other type of consumer loan, banks and credit unions could face potential risks with private student loan rehabilitation programs, including operational, compliance, or reputational risks. For example, a representative of one bank cited operational risks such as those that could stem from errors in credit reporting or inadequate collection practices for rehabilitated private student loans. One concern about removing information from credit reports—as authorized in connection with the Act’s loan rehabilitation programs—is that it could degrade the quality of the credit information that credit providers use to assess the creditworthiness of potential borrowers. However, the removal of defaults from credit reports resulting from loan rehabilitation programs is unlikely to affect financial institutions’ ability to make sound lending decisions, according to prudential regulator officials and representatives from three private student lenders and three other credit providers with whom we spoke. OCC and FDIC officials and representatives from two of these private student lenders noted that because rehabilitation programs leave the delinquencies leading up to the default on borrowers’ credit reports, lenders would still be able to adequately assess borrower risk. In addition, representatives from one automobile lender and one mortgage lender said that over time, the methods they use to assess creditworthiness would be able to detect whether rehabilitated private student loans were affecting their ability to identify risk patterns in credit information and they could adjust the methods accordingly. Representatives from the Federal Reserve provided three additional reasons why they expected that rehabilitation programs would have little effect on banks’ and credit unions’ lending decisions. First, under the statutory requirement for private student loan rehabilitation, removal of a default from a borrower’s credit report can only occur once per loan. A single default removal would be unlikely to distort the accuracy of credit reporting in general. Second, they said that borrowers who have successfully completed a rehabilitation program by making consecutive on-time payments have demonstrated a proven repayment record, and therefore they likely represent a better credit risk. Finally, because participation in the private student loan rehabilitation program is expected to be low, its effect on the soundness of financial institutions’ lending decisions is expected to be minimal. The effects of private student loan rehabilitation programs on most borrowers’ access to credit would likely be minimal. A simulation conducted by VantageScore found that removing a student loan default increased a borrower’s credit score by 8 points, on average. An 8 point rise in a borrower’s credit score within VantageScore’s range of 300 to 850 represents only a very small improvement to that borrower’s creditworthiness. Therefore, most borrowers who successfully completed a private student loan rehabilitation program would likely see minimal improvement in their access to credit, particularly for credit where the decision-making is based solely on generic credit scores. Factors Credit Providers Consider Prior to Lending Credit providers assess a borrower’s creditworthiness based on several factors, including the following: Generic credit scores: Credit providers can rely solely on generic credit scores, such as those developed by Fair Isaac Corporation and VantageScore Solutions, LLC, to make lending decisions. Credit providers generally do not provide credit to borrowers whose scores do not meet a minimum threshold. Industry-specific credit scores: Certain types of credit providers, such as mortgage lenders, automobile loan lenders, and credit card issuers, may use industry-specific credit scores rather than generic credit scores to make lending decisions. This is because these scores may help them better predict lending risks specific to their industry. Internal credit reviews: Credit providers can customize methods unique to their institution that review different aspects of borrowers’ credit information, such as debt-to-income ratios, employment history, and borrowers’ existing relationships with the institution. Credit providers may also develop custom credit scores that are tailored to their specific needs and include factors they have deemed important in predicting risks of nonpayment. Credit providers incorporate their own internal data in these scores as well as information contained in borrowers’ credit reports. The effect of a rehabilitation program on credit scores will likely be somewhat greater for borrowers with lower credit scores, and smaller for borrowers with higher credit scores. For example, the VantageScore simulation suggests that borrowers in the subprime range (with scores of 300–600) could see score increases of 11 points, on average, while borrowers in the prime (661–780) and super prime (781–850) ranges could see increases of less than 1 point, on average (see fig. 3). The effect of removing a default from a credit report varies among borrowers because a credit score is influenced by other information in a borrower’s credit report, such as other outstanding derogatory credit markers, the length of time since the default, and other types of outstanding loans. Reasons that removing a student loan default may improve a borrower’s credit score and access to credit only minimally include the following: Delinquencies remain in the credit report. A key reason that removing a student loan default has a small effect on a credit score, according to VantageScore officials, is that the delinquencies leading to that default remain in the credit report for borrowers who successfully complete rehabilitation programs. Adding a delinquency in the simulation decreased a credit score by 61 points, on average. Thus, the simulation suggests that the increase in a credit score from removing a student loan default is not as substantial as the decrease from adding the initial delinquency. Credit scoring treats student loans differently. Some credit score models place less emphasis on student loans than on other types of consumer loans in predicting the risk of nonpayment. One credit scoring firm and two CRAs we spoke with said that student loans have a lower weight than other types of consumer loans in their generic credit scoring algorithms. They explained that there are fewer student loans than other types of consumer loans in the sample they use to develop the score, and student debt has proved to be less important statistically at predicting credit risk in their models. Student loans also may have less weight in predicting defaults in industry- specific or custom models of scores. A representative of one credit scoring firm said the algorithm for an industry-specific credit score that predicts the risk of nonpayment on a credit card may place less emphasis on a student loan than the algorithm for a generic credit score that is meant to predict risk more broadly. Further, CRA officials we spoke with said that because their custom credit scoring models are specific to clients’ needs, the models may not include student loans as a predictor of default at all, or they may place greater emphasis on student loans, depending on the clients’ needs. Borrowers in default typically already have poor credit. Borrowers who complete a rehabilitation program have a high likelihood of having other derogatory credit items in their credit report, in addition to the student loan delinquencies that led to the default, according to a study conducted by a research organization, several CRAs, and one credit provider with whom we spoke. The VantageScore simulation also showed that borrowers who had at least one student loan delinquency or default in their credit profile had an average of five derogatory credit items in their profile. Because student loan defaults and student loan delinquencies are both negative credit events that affect credit providers’ credit assessment methods, the removal of one student loan default from a borrower’s credit report likely will not make a large difference in how credit providers evaluate the borrower. Consumer advocates and academic studies cited potential benefits of rehabilitation programs apart from their effect on credit scores and access to credit: Borrowers defaulting on private student loans issued by nonbank state lenders could have wage garnishments stopped after successfully completing a rehabilitation program. Rehabilitation would stop debt collection efforts against a private student loan borrower. Participating in a loan modification program for one loan may help borrowers better meet their other loan obligations, according to studies we reviewed. For example, one study found that participation in mortgage modification programs was associated with lower delinquency rates on nonmortgage loans. However, programs may also have some disadvantages or pose challenges to borrowers, according to representatives from consumer advocacy groups and academic sources: A rehabilitation program may restart the statute of limitations on loan collections, according to representatives of consumer advocacy groups. Borrowers who redefault following entry into a rehabilitation program near the end of the statute of limitations on their debt could have collection efforts extended on these loans. Programs may extend adverse credit reporting. Generally, negative credit information stays on consumer reports for 7 or 10 years; therefore, depending on when a borrower enters into a rehabilitation program, a payment on the loan might prolong the adverse credit reporting for that account. The lack of income-driven repayment programs offered to borrowers in the private student loan market means that borrowers who complete rehabilitation programs may have a high likelihood of redefaulting on their loans. Because removing adverse information from credit reports does not change a borrower’s underlying creditworthiness, improved credit scores and access to credit may cause borrowers to borrow too much relative to their ability and willingness to pay. For example, one study found that for consumers who had filed for bankruptcy, their FICO scores and credit lines increased within the first year after the bankruptcy was removed from their credit report. However, the study found the initial credit score increase had disappeared by about 18 months after the bankruptcy was removed and that debt and delinquency were higher than expected, increasing the probability of a future default. Private student loan rehabilitation programs can provide an opportunity for private student loan borrowers to help repair their credit reports. However, some nonbank state lenders have different interpretations of whether FCRA authorizes them to offer such programs. During our review, CFPB had not determined if it would clarify these uncertainties for nonbank state lenders and other nonbank private student loan lenders. Providing such clarity could—depending on CFPB’s interpretation—result in additional lenders offering rehabilitation (allowing more borrowers the opportunity to participate), or help to ensure that only entities deemed eligible by CFPB to offer programs are doing so. In addition, the Act does not explain what information on a consumer’s credit report constitutes a private student loan “default” that may be removed following the successful completion of a private student loan rehabilitation program. Without clarification from CFPB—after consulting with the prudential regulators and relevant industry groups—on what information in a credit report constitutes a private student loan default that may be removed, lenders may be inconsistent in the credit report information they remove. As a result, variations in lenders’ interpretations could have different effects on borrowers’ credit scores and credit records, which could affect how they are treated by credit providers and could also result in inconsistencies that affect the accuracy of credit reporting data. We are making the following two recommendations to CFPB: The Director of CFPB should provide written clarification to nonbank private student loan lenders on their authorities under FCRA to offer private student loan rehabilitation programs that include removing information from credit reports. (Recommendation 1) The Director of CFPB, after consulting with the prudential regulators and relevant industry groups, should provide written clarification on what information in a consumer’s credit report constitutes a private student loan reported “default” that may be removed after successful completion of a private student loan rehabilitation program. (Recommendation 2) We provided a draft copy of this report to CFPB, the Department of Education, FDIC, the Federal Reserve, the Federal Trade Commission, NCUA, OCC, and the Department of the Treasury for review and comment. We also provided FICO and VantageScore excerpts of the draft report for review and comment. CFPB and NCUA provided written comments, which have been reproduced in appendixes II and III, respectively. FDIC, the Federal Trade Commission, OCC, and the Department of the Treasury provided technical comments on the draft report, which we have incorporated, as appropriate. The Department of Education and the Federal Reserve did not provide any comments on the draft of this report. FICO and VantageScore provided technical comments, which we have incorporated, as appropriate. In its written response, CFPB stated that it does not plan to act on our first recommendation to provide written clarification to nonbank private student loan lenders on their authorities under FCRA to offer private student loan rehabilitation programs. CFPB stated—and we agree—that the Act does not regulate the authority of private student loan lenders that are not included in FCRA’s definition of a “financial institution,” nor direct financial institutions that are not supervised by a federal banking agency to seek CFPB’s approval concerning the terms and conditions of rehabilitation programs. However, CFPB’s written response does not discuss the authority of private student loan lenders that potentially fall outside FCRA’s definition of a financial institution to offer rehabilitation programs that include removing information from credit reports. As we discuss in the report, uncertainty exists among nonbank private student loan lenders regarding their authority to implement such programs. We maintain that although the Act does not require CFPB to act on this issue, CFPB could play a role in clarifying whether FCRA authorizes nonbanks to offer rehabilitation programs that enable the lender to obtain legal protection for removal of default information from a credit report. CFPB intervention is warranted given the lack of clarity in the private student lending industry and is consistent with CFPB’s supervisory authority over nonbank financial institutions and its FCRA enforcement and rulemaking authorities. We do not suggest that CFPB play a role in approving rehabilitation programs. As we note in the report, clarification of nonbanks’ authorities could result in additional lenders offering rehabilitation programs and providing more consistent opportunities for private student loan borrowers, or it could help ensure that only those entities authorized to offer programs are doing so. With respect to our second recommendation on providing written clarification on what information in a consumer’s credit report constitutes a private student loan reported default that may be removed after successful completion of a private student loan rehabilitation program, CFPB’s letter states that such clarification is premature because of ongoing work by the Consumer Data Industry Association. The letter states that after that work is completed, CFPB will consult with the relevant regulators and other interested parties to determine if additional guidance or clarification is needed. As we stated in the report, we are aware of the work of the Consumer Data Industry Association to update the credit reporting guidelines for private student loans. We maintain that this work presents a good opportunity for CFPB to participate in these discussions and to work in conjunction with the industry and other relevant regulators to help alleviate any contradiction between what CFPB would determine in isolation from any determination made by industry. Further, such participation would allow CFPB to weigh in on legal and policy issues from the start, potentially avoiding any need for future rulemaking. In addition, CFPB’s involvement in this determination and issuance of clarification would help ensure more consistent treatment among borrowers participating in private student loan rehabilitation programs, as well as consistency in credit reporting information. NCUA’s written response stated that federal credit unions were authorized to offer rehabilitation programs for private student loan borrowers prior to the Act and that federal credit unions are not required to obtain review and approval from NCUA to offer such programs. The letter notes, however, that the Act requires federal credit unions that offer such programs to remove private student loan defaults from consumer credit reports if borrowers successfully complete a rehabilitation program. NCUA noted that even though removal of the default may result in a relatively small credit score increase, this can benefit credit union members. NCUA stated that it stands ready to assist CFPB in implementing the report’s two recommendations. We are sending copies of this report to CFPB, the Department of Education, FDIC, the Federal Reserve, the Federal Trade Commission, NCUA, OCC, the Department of the Treasury, the appropriate congressional committees and members, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or cackleya@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Our objectives were to examine (1) the factors affecting financial institutions’ participation in private student loan rehabilitation programs, (2) the risks that these programs may pose to financial institutions, and (3) the effects that these programs may have on student loan borrowers’ access to future credit. To examine the factors that affect financial institutions’ participation in private student loan rehabilitation programs and how the federal banking regulators are implementing the Economic Growth, Regulatory Relief, and Consumer Protection Act’s (the Act) provisions on private student loan rehabilitation programs, we reviewed the statements issued by the three regulators tasked with approving the loan rehabilitation programs of their regulated entities—the Board of Governors of the Federal Reserve System (Federal Reserve), Federal Deposit Insurance Corporation (FDIC), and Office of the Comptroller of the Currency (OCC)—as well as OCC’s examiner guidance. We also interviewed officials from these regulators about their time frames for issuing statements, what topics the statements cover, and how they coordinated in issuing the statements. We reviewed the legal authorities of the Consumer Financial Protection Bureau (CFPB) and National Credit Union Administration (NCUA)—which oversee nonbank private student loan lenders and most credit unions that issue private student loans, respectively—concerning private student loan rehabilitation programs and the legislative history of the Act’s provisions on the programs. Finally, we interviewed officials from NCUA and CFPB about their authorities related to implementing the Act’s provisions on private student loan rehabilitation programs and whether they planned to take any actions related to the provisions. In addition, we interviewed representatives from a nongeneralizable sample of 15 private student loan lenders: the five largest bank lenders, two of the largest credit union lenders, and eight nonbank financial institutions (nonbank). The eight nonbank lenders included three for-profit nonbank lenders and five nonprofit state-affiliated lenders (nonbank state lenders). We asked these lenders about their decisions to offer private student loan rehabilitation programs, risks and costs associated with the programs, and the effects that such programs could have on their lending decisions. We identified the five largest bank lenders by reviewing data from MeasureOne—a private data analytics company that studies the private student loan market—and discussions with officials from the Federal Reserve, FDIC, OCC, and CFPB. We assessed the reliability of data from MeasureOne through discussions with representatives from the company on the methodology used to develop its estimates and its internal controls. We determined that this data source was sufficiently reliable for selecting a sample of private student lenders to interview about participation in rehabilitation programs. We reviewed these five banks’ 2017 10-K reports (annual financial filings with the Securities and Exchange Commission) to verify the size of their student loan portfolios. We selected the two credit unions to interview by reviewing 2018 NCUA data on credit unions’ portfolios to identify two credit unions that were among the largest credit union private student loan lenders. To select the for-profit nonbank lenders, we used suggestions from officials at CFPB, OCC, and the Department of Education, as well as reports from private sources that contained information on nonbank private student loan lenders. We selected nonbank state lenders based on information that indicated they were operating or interested in offering rehabilitation programs. Sources of this information included the Education Finance Council’s 2018–2019 NonProfit & State-Based Education Loan Handbook, an interview with the Education Finance Council, and information received from a 2013 CFPB Request for Information Regarding an Initiative to Promote Student Loan Affordability. Because this sample is nongeneralizable, our results cannot be generalized to all private student loan lenders. To examine the risks, if any, that private student loan rehabilitation programs pose to financial institutions, we reviewed bank and credit union regulator policies and guidance on private student lending. We also analyzed data on delinquency and default rates of private student loans. To do this, we reviewed industry data from MeasureOne and the 2017 10- K filings for the five banks whose representatives we interviewed. We assessed the reliability of MeasureOne’s performance data through discussions with representatives from the company on the methodology it uses to develop these metrics and its internal controls. We determined that this data source was sufficiently reliable for assessing the performance of banks’ portfolios of private student loans. For these five banks, we also used the 10-K filings to estimate the volume of the portion of their portfolios that was composed of private student loans. We also compared private student loan default rates to default rates of other types of consumer loans, including mortgages, credit cards, and automobile loans. To do this, we used data from FDIC’s Statistics on Depository Institutions database to analyze indicators of asset quality for mortgages, credit cards, and automobile loans from 2013 through 2017. We assessed the reliability of FDIC’s Statistics on Depository Institutions database by reviewing related documentation and conducting testing for missing data, outliers, or any obvious errors. We determined that this data source was sufficiently reliable for assessing the performance and risk of banks’ portfolios of private student loans and other types of consumer loans. We also interviewed officials from the Federal Reserve, FDIC, NCUA, and OCC about the types of costs and risks that could be associated with private student loan rehabilitation programs. In addition, we interviewed representatives of our nongeneralizable sample of 15 private student loan lenders about the potential risks and costs of offering rehabilitation programs. To assess potential risks of private student loan rehabilitation programs for other types of financial institutions, we interviewed a nongeneralizable sample of seven credit providers about how these programs could affect their ability to make sound lending decisions. We focused on financial institutions that offer mortgage loans, automobile loans, and credit cards. According to data from the 2016 Survey of Consumer Finances, these are the most common types of debt consumers hold. We selected a nongeneralizable sample of banks and nonbank financial institutions that provide these types of credit. We selected the bank credit providers using data from FDIC’s Statistics on Depository Institutions by identifying the mortgage and automobile loan lenders and credit card issuers that were among the largest holders of assets in these lending categories as of the fourth quarter 2017. To identify nonbank financial institution lenders, we reviewed an industry report to identify some of the larger nonbank mortgage lenders, and we reviewed a list prepared by CFPB of larger industry participants in the automobile finance market industry. We judgmentally selected the final sample of these credit providers based on their size and, to the extent applicable, their federal regulator to obtain a diversity of opinions. We determined that industry reports, CFPB’s list of larger industry participants, and 10-K filings were sufficiently reliable for selecting a sample of nonbank financial institutions to interview about risks posed by rehabilitation programs. Because this sample is nongeneralizable, our results cannot be generalized to all credit providers. We also interviewed representatives of four industry groups and two trade associations that work with these credit providers and student loan borrowers on the types of risks and costs that rehabilitation programs could create for lenders. To examine the effects that private student loan rehabilitation programs may have on student loan borrowers’ access to future credit, we conducted a literature search for studies that empirically analyzed the effects on credit scores and access to credit of adverse credit events, such as foreclosures or bankruptcies; loan modifications, broadly defined; and removal of accurate but adverse information from credit reports, such as a bankruptcy. We identified these studies through our initial background search, targeted searches of the EconLit database, and a search of the Federal Reserve Bank of New York Center for Microeconomic Data publications, and through bibliographies of studies we reviewed. We also asked VantageScore Solutions, LLC (VantageScore)—a credit scoring firm—to conduct a quantitative analysis simulating the effect of adding a student loan delinquency to and removing a student loan default from a borrower’s credit profile on its VantageScore 3.0 credit score. The analysis was conducted using a sample of VantageScore’s data that it obtained from the three nationwide CRAs and that represents actual credit profiles of borrowers. VantageScore analyzed data for borrowers with at least one outstanding student loan with a balance greater than $0. Table 1 contains the results of the simulation and information on the number and characteristics of borrowers whose credit profiles were analyzed. The results of the simulation are specific to changes in the VantageScore 3.0 credit score. The simulated results represent averages for borrowers whose credit profiles were analyzed and are meant to be illustrative. Additionally, because this was a simulation, it is unlikely that any one borrower’s credit profile exactly matches the average profiles used in the simulations. The results of the VantageScore analysis only apply to VantageScore 3.0 credit scores in the 2014–2016, 2015–2017, and 2016–2018 cohorts of borrowers and may not be generalized to other VantageScore credit scores, to Fair Isaac Corporation (FICO) credit scores, or for different cohorts in different years. While we present only the results of the most recent cohort (2016–2018) in our report, VantageScore simulated the analysis across three cohorts to determine whether the results varied substantially over time. The results for all three cohorts were similar. Through reviewing documentation and conducting interviews, we determined that the data used by VantageScore to conduct this analysis were sufficiently reliable for simulating the effects of derogatory credit marks on borrowers’ credit scores. FICO declined our request to develop a similar analysis. To examine how a rehabilitation program may affect borrowers’ future access to credit, we interviewed officials from CFPB, the Department of Education, FDIC, the Federal Reserve, Federal Trade Commission, NCUA, OCC, and the Department of the Treasury. We also interviewed representatives of the four consumer reporting agencies that collect and report information on student loans (Equifax, Experian, Innovis, and TransUnion) and the two credit scoring firms that develop credit score models with nationwide coverage (FICO and VantageScore). We also interviewed representatives from the 15 private student loan lenders and seven credit providers described above, as well as banking, credit reporting, and student loan lending and servicing industry groups and consumer advocacy organizations. We conducted this performance audit from July 2018 to May 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Jill Naamane (Assistant Director), Christine McGinty (Analyst-in-Charge), Jill Lacey, Courtney LaFountain, Jon D. Menaster, Tovah Rom, Jessica Sandler, Eric Schwab, and Aisha Shafi made key contributions to this report. Also contributing to this report were Melissa Emrey-Arras, Debra Prescott, and Jena Sinkfield.", "summary": "The Economic Growth, Regulatory Relief, and Consumer Protection Act enabled lenders to offer a rehabilitation program to private student loan borrowers who have a reported default on their credit report. The lender may remove the reported default from credit reports if the borrower meets certain conditions. Congress included a provision in statute for GAO to review the implementation and effects of these programs. This report examines (1) the factors affecting financial institutions' participation in private student loan rehabilitation programs, (2) the risks the programs may pose to financial institutions, and (3) the effects the programs may have on student loan borrowers' access to credit. GAO reviewed applicable statutes and agency guidance. GAO also asked a credit scoring firm to simulate the effect on borrowers' credit scores of removing student loan defaults. GAO also interviewed representatives of regulators, some of the largest private student loan lenders, other credit providers, credit bureaus, credit scoring firms, and industry and consumer advocacy organizations. The five largest banks that provide private student loans—student loans that are not guaranteed by the federal government—told GAO that they do not offer private student loan rehabilitation programs because few private student loan borrowers are in default, and because they already offer existing repayment programs to assist distressed borrowers. (Loan rehabilitation programs described in the Economic Growth, Regulatory Relief, and Consumer Protection Act (the Act) enable financial institutions to remove reported defaults from credit reports after borrowers make a number of consecutive, on-time payments.) Some nonbank private student loan lenders offer rehabilitation programs, but others do not, because they believe the Act does not authorize them to do so. Clarification of this matter by the Consumer Financial Protection Bureau (CFPB)—which oversees credit reporting and nonbank lenders—could enable more borrowers to participate in these programs or ensure that only eligible entities offer them. Private student loan rehabilitation programs are expected to pose minimal additional risks to financial institutions. Private student loans compose a small portion of most banks' portfolios and have consistently low default rates. Banks mitigate credit risks by requiring cosigners for almost all private student loans. Rehabilitation programs are also unlikely to affect financial institutions' ability to make sound lending decisions, in part because the programs leave some derogatory credit information—such as delinquencies leading to the default—in the credit reports. Borrowers completing private student loan rehabilitation programs would likely experience minimal improvement in their access to credit. Removing a student loan default from a credit profile would increase the borrower's credit score by only about 8 points, on average, according to a simulation that a credit scoring firm conducted for GAO. The effect of removing the default was greater for borrowers with lower credit scores and smaller for borrowers with higher credit scores (see figure). Reasons that removing a student loan default could have little effect on a credit score include that the delinquencies leading to that default—which also negatively affect credit scores—remain in the credit report and borrowers in default may already have poor credit. GAO is making two recommendations, including that CFPB provide written clarification to nonbank private student loan lenders on their authority to offer private student loan rehabilitation programs. CFPB does not plan to take action on this recommendation and stated that it was premature to take action on the second recommendation. GAO maintains that both recommendations are valid, as discussed in this report.", "document_type": "gao"}
{"report": "Federal agencies are dependent on information systems and electronic data to process, maintain, and report essential information. Virtually all federal operations are supported by computer systems and electronic data, and agencies would find it difficult, if not impossible, to carry out their missions and account for their resources without these information assets. Federal agencies exchange personally identifiable and other sensitive information with state agencies in the implementation of key federal and state programs. The security of systems and data involved in this exchange of information is vital to public confidence and the nation’s safety, prosperity, and well-being. Since federal agencies face computerized (cyber) threats that continue to grow in number and sophistication, it is imperative that such information is protected. In recognition of this growing threat, we designated information security as a government-wide high-risk area in 1997. We further expanded this area in 2015 to include protecting the privacy of personally identifiable information. Several federal laws and policies establish requirements for protecting federal systems and managing cybersecurity risks. Specifically, FISMA is intended to provide a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support federal operations and assets, as well as the effective oversight of information security risks. The act requires each agency to develop, document, and implement an agency-wide information security program to provide risk-based protections for the information and information systems that support the operations and assets of the agency, including those provided or managed by another entity. FISMA also assigns government-wide responsibilities to key agencies. For example, OMB is responsible for developing and overseeing implementation of policies, principles, standards, and guidelines on information security in federal agencies, except with regard to national security systems. NIST is also responsible for developing standards for categorizing information and information systems, security requirements for information and systems, and guidelines for detection and handling of security incidents. For example, NIST Special Publication 800-53 provides guidance to agencies on the selection and implementation of information security and privacy controls for systems. Further, OMB Circular A-130, Managing Information as a Strategic Resource, establishes minimum requirements for federal information security programs and assigns federal agency responsibilities for the security of information and information systems. It requires agencies to implement a risk management framework to guide and inform the categorization of federal information and information systems; the selection, implementation, and assessment of security and privacy controls; the authorization of information systems and common controls; and the continuous monitoring of information systems. Circular A-130 also requires federal agencies to provide oversight of nonfederal entities—such as state agencies—that use or operate federal information systems, as well as nonfederal entities’ information systems that collect or maintain federal information. In doing so, federal agencies are to ensure that security and privacy controls for such information systems are effectively implemented and comply with NIST standards and guidelines and agency requirements. Federal agencies may share data with one or more individual component agencies within a state, such as agencies that execute a state’s tax administration, law enforcement, or human services functions. The state’s responsibility for protecting data shared by federal agencies may reside within an individual state agency or it may be a shared responsibility with the state’s chief information officer and CISO. For example, a state CISO may help to manage the protections over centralized information technology (IT) resources that store, process, and transmit federal data for multiple component agencies within the state. To protect federal data that are shared with state agencies in the implementation of key federal and state programs, federal agencies have developed cybersecurity requirements for state agencies to follow when accessing, storing, and transmitting federal data. Federal agencies are to obtain assurance that state agencies’ security and privacy controls are effectively implemented through independent evaluations. These evaluations include tests and assessments of the effectiveness of state agencies’ information security policies, procedures, and practices. Such assessments are important inputs to decisions by federal officials to authorize or reauthorize a state agency’s use of information systems that create, collect, use, process, store, maintain, disseminate, disclose, and dispose of federal information. To protect federal data that are shared with state agencies, each of the federal agencies in our review have established their own policies that articulate cybersecurity requirements, as well as related compliance assessment programs, based in part on guidance from NIST. Table 1 identifies the types of data that the four selected federal agencies share with state agencies and the cybersecurity policies that they have established to protect that data. The selected federal agencies’ had a significant number of variances in the cybersecurity requirements that they had established for protecting data exchanged with state agencies. Specifically, our review identified hundreds of instances in which the four agencies either had (1) included a requirement in its cybersecurity policy that was not a requirement of the other three agencies (unique requirement); (2) established a requirement with specific, organization-defined technical thresholds that differed from at least one of the other three agencies for a related control (conflicting parameters); or (3) did not fully address in its requirements the guidelines from NIST for associated controls and control enhancements (did not fully address NIST guidelines). Table 2 summarizes the total number of requirements that each agency had included in its security policy and the extent to which the four agencies’ requirements varied from each other and from the NIST guidance. Collectively, the four selected federal agencies’ policies included 86 unique cybersecurity requirements for state agencies with which they exchange data. Specifically, CMS’s policy included 54 requirements that the other three agencies did not include. FBI’s CJIS’s policy included 24 unique requirements, IRS’s policy included five unique requirements, and SSA’s policy included three unique requirements. For example, CMS’s security policy included a requirement that state agencies review their organization-wide information security program plan annually; however, the other three agencies did not have such a requirement in their security policies. As another example, IRS had a requirement for state agencies to employ automated mechanisms to alert security personnel of inappropriate activities, while the other agencies did not have this requirement. Because each agency is addressing different legal requirements and risk management approaches for protecting information shared with states, certain requirements that are unique to an agency may be necessary. Nevertheless, agencies need to ensure that such requirements are necessary by documenting their decisions during the control selection process. Table 3 provides examples of the unique requirements that each agency included in its cybersecurity policies. In total, the four federal agencies had identified 390 requirements for state agencies in their policies, where the parameters conflicted with at least one of the other federal agencies. Across the four agencies, CMS had the largest number of requirements that had conflicting parameters, with 139 such requirements. This was followed by IRS with 131, FBI’s CJIS with 72 requirements, and SSA with 48 requirements with conflicting parameters. For example, each of the selected agencies identified a different time frame for the retention of audit logs related to audited events. As another example, CMS required state agencies to annually review and update their access control policies, whereas IRS required this review every 3 years. FBI’s CJIS and SSA did not have this requirement in their policies. Table 4 provides additional examples of cybersecurity requirements for state agencies that the four federal agencies identified in their policies, where the parameters conflicted with those of at least one other of the federal agencies. The four selected federal agencies did not always fully address guidelines in NIST Special Publication 800-53 (Revision 4) when establishing cybersecurity requirements for related controls, leading to additional differences among the four agencies’ cybersecurity policies. In total, the four agencies did not fully address guidelines from NIST in 141 instances. FBI’s CJIS had the most variances, with 63 requirements that did not fully address NIST guidelines, followed by SSA with 30 variances, CMS with 26 variances, and IRS with 22 variances. For example, FBI’s CJIS’s requirement did not identify the time period to retain individual training records, as called for by NIST guidance. In addition, SSA did not define the frequency of how often agencies should assess the security controls in the information system and its environment of operation. Table 5 provides examples of the cybersecurity requirements for state agencies in which selected federal agencies did not fully address NIST guidelines. The perspectives of state CISOs who responded to our survey reflected the variation we found among the selected federal agencies’ cybersecurity requirements. The majority (at least 29 out of 50) of the state CISOs that responded to our survey question regarding the ways in which federal cybersecurity requirements vary and the extent of the variation reported moderate to very great variation in the selected federal agencies’ cybersecurity requirements. Specifically, of the 50 state CISOs that responded to this question, 34 reported that the federal agencies had moderate to very great variation with respect to unique requirements, 38 reported that the federal agencies had moderate to very great variation due to conflicting parameters that were established, and 29 reported that the federal agencies had moderate to very great variation with respect to addressing NIST guidelines for security controls and control enhancements. Figure 1 represents state CISOs’ perspectives on the extent of variation among selected federal cybersecurity requirements. State agency officials that must comply with multiple federal agencies’ cybersecurity requirements (and related compliance assessments) viewed variances as problematic and burdensome. For example, in responding to a survey question about challenges or impacts that state officials experienced regarding federal requirements and assessment processes, an official from one state agency explained that addressing variances in cybersecurity requirements reduced the ability of state officials to focus on their primary mission of securing data across their state enterprise. In response to the same survey question, another state official said that addressing the variances in federal agencies’ cybersecurity requirements increased the complexity of automating the state’s monitoring and reporting processes. In addition, the same state official commented that staff were burdened by reports and reviews to ensure that the full range of federal agencies’ requirements were met. In responding to our survey, 46 state CISOs reported the extent to which they had experienced a very great, great, moderate, slight, or no increase in calendar time; staff hours; and costs of acquiring additional materials, software, and equipment to address variances in selected federal agencies’ cybersecurity requirements. The majority (at least 34 out of 46) of the state CISOs that responded to this question in our survey reported moderate to very great increases in these types of impacts. Figure 2 represents the extent of impacts that state CISOs reported as a result of variances in selected federal cybersecurity requirements. OMB Circular A-130 requires federal agencies to coordinate with nonfederal entities, such as state agencies, as well as other federal agencies as appropriate, to ensure that security and privacy requirements pertaining to these nonfederal entities are consistent to the greatest extent possible. In addition, GAO and NIST have identified practices that can help federal agencies limit potential variation in security control selection and requirements, such as coordinating to develop compatible policies, procedures, and other means to operate across agency boundaries. For example, according to NIST, agencies can establish a tailored set of baseline requirements to satisfy common security objectives. In addition, by applying practices recommended by GAO for enhancing and sustaining coordination and collaboration, federal agencies could work towards establishing shared requirements with consistent terminology and parameters. However, the four selected federal agencies have not ensured that their cybersecurity requirements for state agencies are consistent to the maximum extent possible through coordination with each other. Officials from IRS, FBI, and SSA acknowledged that they had not coordinated with other federal agencies in establishing their current cybersecurity requirements for state agencies. The agencies had not coordinated, in part, because they have prioritized addressing agency-specific responsibilities from relevant laws and agency policies as well as the needs of relevant communities of interest. CMS officials stated that the agency coordinated with other federal agencies in 2015 when CMS originally established requirements for its security policy, the Minimum Acceptable Risk Standards for Exchanges Document Suite 2.0, Volume III: Minimum Acceptable Risk Standards for Exchanges. CMS officials noted that the agency added controls that IRS and SSA deemed essential to protecting data for which these agencies were responsible. Nevertheless, we found variances between CMS’s requirements and those established by IRS and SSA. Further, CMS last updated its security policy in September 2015 and IRS, SSA, and FBI’s CJIS have each since updated their policies. In addition to the insufficient coordination, the selected federal agencies identified two additional explanations for variances in their cybersecurity requirements for state agencies: (1) agencies’ determination that selected requirements were necessary and therefore, that resulting variances are warranted and (2) agencies’ requirements review processes that resulted in deviations from NIST guidance. Each of the selected agencies noted that they determined the unique controls and competing parameters in their requirements were necessary and warranted. For example, SSA noted that it has been conducting data exchanges with states since the late 1970s, predating NIST Special Publication 800-53. According to SSA officials, the agency’s security requirements retained certain legacy language that state agencies were already familiar with to reduce disruption to them. IRS officials also noted that their security controls incorporate disclosure restrictions from the Internal Revenue Code and internal IRS directives. Agency processes for reviewing their cybersecurity requirements have resulted in deviations from NIST guidance. For example, FBI’s CJIS officials stated that they started with NIST terminology when developing their policy. However, CJIS’s Advisory Policy Board— which recommends the final CJIS policy to the FBI Director— suggested modifications to the wording of requirements during subsequent reviews. As another example, CMS noted that during the review process for its requirements, in certain instances it deviated from NIST guidance to use terminology that would be more familiar to state agency users. Federal agencies may have legitimate reasons for having variances in their cybersecurity requirements. For instance, agencies may need to apply different information security controls, a greater number of controls, or more stringent technical parameters to protect data for which they are responsible in a manner consistent with various security requirements originating in federal laws, executive orders, directives, policies, regulations, standards, or guidelines as well as the agency’s risk assessments. However, according to NIST, organizations should document the relevant decisions taken during the control selection process, and provide a sound rationale for those decisions that is based on agency mission and business needs. Both FBI’s CJIS and IRS had documented the agency’s rationale for unique requirements. SSA stated that their controls were developed before NIST standards were created and they have mapped their current controls to NIST. However, SSA was unable to produce this documentation. CMS officials noted that the rationale for the requirements identified in the agency’s Minimum Acceptable Risk Standards for Exchanges security policy was documented in CMS’s Acceptable Risk Standards. However, the Acceptable Risk Standards did not include all requirements that were included in CMS’s security policy. For example, CMS’s requirements for organizations to review and re-evaluate privileges at least quarterly and for the information system to allocate resources by priority and/or quota were included in the security policy without a defined rationale and were also not included in CMS’s Acceptable Risk Standards. While agencies have identified various reasons for not coordinating on their cybersecurity requirements for state agencies, OMB has not taken steps to evaluate whether agencies are coordinating. OMB officials acknowledged that they could encourage additional coordination among the selected agencies, but said that it is ultimately up to the agencies to set their requirements and determine how best to assess states’ compliance with those requirements. However, without OMB’s involvement and encouragement that federal agencies collaborate to make their cybersecurity requirements for state agencies consistent to the greatest extent possible, federal agencies are less likely to prioritize such efforts. The selected federal agencies will soon have an opportunity to harmonize to the extent possible their requirements as they revisit and potentially update their existing security policies based on anticipated changes in NIST guidance. Until these agencies coordinate, where feasible, to address the variances in their cybersecurity requirements, officials from state agencies may continue to experience cost, time, and other burdens resulting from these variances. Further, without documentation of the rationale for having requirements that are unique or parameters that conflict in comparison to other agencies, it will be more difficult for these agencies to achieve consistent requirements. As previously discussed, OMB Circular A-130 requires federal agencies to assess whether state agencies have implemented effective security and privacy controls on information systems that create, collect, use, process, store, maintain, disseminate, disclose, or dispose of federal information. The circular also encourages federal agencies to coordinate on their approaches to authorizing the use of such systems whenever practicable. For example, the circular notes that multiple agencies are encouraged to jointly plan and execute tasks in NIST’s Risk Management Framework, which includes conducting security assessments. According to the circular, agencies can also leverage information generated by another agency based on the need to use the same information resources (e.g., information system or services provided by the system) by choosing to accept some or all of the information in an existing authorization package, including completed security assessments. As previously stated, NIST and GAO have recommended practices that federal agencies can implement to help with their coordination on cybersecurity assessments, such as assessments of state agencies’ compliance with federal cybersecurity requirements. Those practices fall in two broad areas: (1) coordination with state agencies when assessing states’ cybersecurity and (2) coordination with other federal agencies on the assessment of state agencies’ cybersecurity. In addition, based on the guidance from NIST that pertained to coordination on assessments of cybersecurity and practices recommended by GAO for enhancing coordination among federal agencies, four supporting activities are common to each of these two areas of federal agencies’ coordination on cybersecurity assessments: assessment schedules and time frames; meeting and document requests; security test plans—including testing techniques, location, and tools; and the use of findings from prior assessments. With regard to coordinating with state agencies when assessing their cybersecurity, two of the selected federal agencies—CMS and IRS—had policies that addressed all four of the activities supporting this area of coordination. The two other agencies—FBI’s CJIS and SSA—had policies that addressed some, but not all, of the supporting activities for such coordination. With regard to coordinating with other federal agencies on the assessment of state agencies’ cybersecurity, none of the four federal agencies had policies that addressed the activities supporting this area of coordination. Table 6 summarizes the extent to which selected agencies established policies for coordinating with state agencies and other federal agencies when assessing cybersecurity. See appendix II for details on the extent to which selected agencies addressed individual activities supporting the two areas of coordination. Each of the selected federal agencies addressed at least three of the four activities for coordinating with state agencies when assessing cybersecurity. CMS and IRS fully established policies for coordinating with state agencies by addressing all of the activities supporting such coordination. However, FBI’s CJIS and SSA partially established policies for coordinating with state agencies by addressing some—but not all—of the supporting activities. Specifically, FBI’s CJIS and SSA fully addressed three of the activities: coordinating (1) assessment schedules and time frames, (2) meeting and document requests, and (3) security test plans. For example, FBI’s CJIS policy included instructions for providing the date and time of assessment along with a schedule for the assessment process. Further, the policy stated that assessors should lay out the meetings that need to occur and documentation that state agencies need to provide CJIS, including specifics about the state’s network. SSA’s policy laid out each step of the assessment process, including the anticipated time frames. Further, SSA’s policy identified certain meetings that should be held during the process and documentation to be provided before the assessment. However, FBI’s CJIS and SSA did not fully establish policies for coordinating with state agencies because they did not address the activity associated with coordinating the use of findings from prior assessments. Specifically, while these two agencies’ policies addressed using findings from prior assessments conducted by their individual agency, their policies did not address whether or how assessors should use findings from other security assessments conducted within the state. Officials from FBI stated that in practice they consider findings from independent security assessments conducted within a state, but had not documented this practice in their assessment policies due to the limited instances in which this information is available. Officials from SSA believed that their policy addressed how its assessors were to consider findings from other security assessments that are conducted within a state. However, based on our review of SSA’s policy, this information was not yet addressed. None of the four agencies established policies for coordinating with other federal agencies when assessing state agencies’ cybersecurity. Officials from the four selected agencies reported that this is because their priority is to assess compliance with their own security requirements and they are not comfortable relying solely on other federal agencies’ assessments. Officials from each of the selected agencies provided additional perspectives on coordination with other federal agencies. Specifically: CMS officials stated that while they do not coordinate with other federal agencies in conducting compliance assessments, they did coordinate with other federal agencies when establishing their cybersecurity requirements. In addition, CMS officials stated that they do not conduct assessments of compliance with their security policy and that states engage contractors to perform the assessments. Therefore, CMS officials believed that the agency does not have a need to coordinate with other federal agencies. However, CMS did not include, where feasible, additional and detailed guidance to the state that it could use to inform its assessment contractors about coordination with other federal agencies. CMS guidance to the states could encourage additional coordination with other federal agencies such as planning the assessment, leveraging related efforts by other federal agencies, and sharing the state’s documentation and findings with other federal agencies, as appropriate. By not doing so, CMS is not maximizing coordination with other federal agencies to the greatest extent practicable. FBI’s CJIS officials stated that they schedule their security assessments 6 months ahead of time, but would be willing to reschedule the assessment if the state was unavailable due to another assessment being conducted. In addition, CJIS officials noted that while they test for security controls that other federal agencies are testing, they are not assessing the same information as other agencies because the FBI specifically requires criminal justice data to be logically separated from other data. Further, CJIS officials stated their assessment results and audit findings cannot be shared and that other federal agencies would need to refer to a state’s criminal justice agency for such information. IRS officials stated that they previously attempted to review assessment findings from other agencies, but since IRS was not looking at the same systems, the findings were not helpful. IRS officials stated that they would be willing to review recent assessments conducted by other federal agencies to see if information can be leveraged. SSA officials noted that it is their practice to reschedule an assessment if another federal agency has an assessment scheduled around the same time, but acknowledged that this was not in their policies. Further, according to SSA officials, they do not currently examine or consider findings from independent security assessments conducted within a state. While agencies cited various reasons for not coordinating when assessing state agencies’ cybersecurity, taking steps to coordinate, such as leveraging other agencies’ assessments or conducting joint assessments whenever practicable, would be consistent with practices encouraged by OMB. However, OMB has not taken steps to ensure that they do so. OMB officials noted that they believed several of the agencies had begun to coordinate on their own and acknowledged that they could take additional steps to encourage and promote coordination among the agencies. OMB officials further noted that it is ultimately the responsibility of the agencies to determine how they conduct their assessments. Nevertheless, federal agencies may be placing unnecessary burdens on state officials’ time and resources in responding to similar requests and inquiries. Several state CISOs told us that they have identified various instances in which multiple federal agencies’ lack of coordination resulted in requests for similar documentation and interviews with IT officials. For example, according to three state CISOs, the selected federal agencies have asked them to address similar questions regarding physical security controls, network configurations, and password policies in separate interviews. Three state CISOs also noted that they have provided to multiple federal agencies documentation—such as network diagrams and incident response policies—related to the same IT environment and have facilitated multiple federal assessments of the same physical environment. State CISOs identified additional opportunities for further coordination among federal agencies and impacts in dealing with federal cybersecurity assessments. For instance, in response to our survey, 16 states’ officials commented that the four federal agencies in our review could leverage additional opportunities to coordinate on their assessments within their states, particularly where the states had a consolidated data center or other centrally managed IT infrastructure. Further, four state CISOs noted that federal agencies could potentially leverage security compliance assessments and internal audits performed at the state or local level because they included reviews of controls from NIST Special Publication 800-53. In addition, 11 states mentioned “duplication” in their response to a survey question about challenges or impacts related to federal cybersecurity requirements and assessment processes, while two states mentioned “overlap,” and one state mentioned “fragmentation.” For example: One state identified that assessors from different federal agencies generally ask for the same items from the state, requiring state agency officials to reproduce the same response. Another state identified that multiple federal agencies have been assessing the same state agencies with different scope, tools, and documentation requests. In another example, a state concluded that federal assessors’ interpretation of many technical controls was inconsistent and varied from one federal agency to another and across audit cycles. The state noted that there were opportunities for the federal government to streamline how each agency applied different interpretations. State CISOs also identified impacts on their time and costs from responding to federal agencies’ assessments. Seventeen respondents reported impacts to their time and six reported cost impacts. Further, in responding to questions in our survey and an in-depth interview, state CISOs provided additional insights regarding impacts. For example: One state mentioned that, due to the varying requirements from the selected federal agencies, the state is required to stand up multiple virtual and physical environments. In doing so, the state is required to purchase additional software and hardware to maintain such environments. Another state explained that staff manage various state agencies’ data in one central location and spend a considerable amount of time responding to each of the four selected federal agencies’ assessments. Twenty-four states estimated that the four selected federal agencies conducted at least 188 assessments between calendar years 2016 and 2018 and that the states’ best estimates of the total expenditures associated with those assessments ranged from $43.8 million to $67 million. Of 164 instances where states reported an average time spent on assessments by one of the four selected agencies between calendar years 2016 and 2018, in 97 instances the average time expenditure per assessment was reported to be 301 staff hours or more, and in 67 instances it was less than 301 staff hours. Additionally, there were 34 instances in which the state did not know what its average staff hour expenditure was for a particular agency’s assessment or said that it was not applicable to the state. Figure 3 represents the responses from 50 state CISOs on the average state staff hours expended per assessment from across the four selected federal agencies as reported by state CISOs. While state agencies could benefit from additional coordination among federal agencies in conducting their security assessments, increasing coordination may also save the federal government money. For instance, federal agencies may be able to reduce the number of assessments or the scope of the assessment conducted by each agency, the amount of time multiple federal agencies must spend reviewing state systems, and contractor services acquired to assist in performing assessments. The selected federal agencies reported spending close to $45 million in fiscal years 2016 through 2018 on assessments of state agencies’ cybersecurity. Figure 4, an interactive figure, provides the selected federal agencies’ reported spending for fiscal years 2016 through 2018 for assessing state compliance with cybersecurity requirements. (See appendix III for the cost breakdown of selected federal agencies’ reported spending). Until FBI’s CJIS and SSA fully develop policies for coordinating with state agencies and all of the selected agencies develop policies for coordinating with other federal agencies when assessing state agencies’ cybersecurity, as appropriate, they run the risk of spending more than necessary to assess the security of state systems and networks. Further, federal agencies may be placing unnecessary burdens on state officials’ time and resources in responding to overlapping or duplicative requests and inquiries, retesting controls that have already been evaluated, or reporting similar findings multiple times throughout a state. In addition, until OMB takes steps to ensure agencies coordinate on assessments of state agencies’ cybersecurity, it will not have reasonable assurance federal agencies are leveraging compatible assessments where practicable. Given that the federal government exchanges personally identifiable and other sensitive information with state agencies, it is critical to have effective coordination across the federal and state agencies to protect this information. While the selected federal agencies have taken steps to secure information exchanged between federal and state agencies, they have not coordinated with each other in establishing cybersecurity requirements for state agencies. The selected agencies’ insufficient coordination has contributed to variances in the agencies’ control selection, terminology, and technical parameters across hundreds of cybersecurity requirements imposed on states. Further, OMB requires agencies to coordinate to ensure consistency among cybersecurity requirements for state entities, but it has not ensured that agencies have done so. While federal agencies may have legitimate reasons for having variances in their cybersecurity requirements, states’ compliance with multiple federal agencies’ cybersecurity requirements has resulted in increased costs. Coordinating to address variances in federal agencies’ cybersecurity requirements could help to significantly reduce these costs. The selected agencies will soon have an opportunity to coordinate on any planned updates of their security policies that affect state agencies when reviewing their security policies against expected revisions in NIST guidance. Accordingly, it is important that OMB ensures that selected federal agencies coordinate with state agencies and each other to establish cybersecurity requirements that are consistent to the greatest extent possible. Selected federal agencies had partially established policies to coordinate with state agencies when assessing their cybersecurity, but did not have policies for coordinating with other federal agencies. Federal agencies have not been coordinating with each other on assessments of state agencies’ cybersecurity, in part, because this has not been a priority for them. Further, federal agencies have been less likely to coordinate in their assessments of state agencies’ cybersecurity without additional involvement from OMB. The lack of coordination among federal agencies has been a concern among state CISOs who described instances of duplication and overlap in their cybersecurity assessments. As with the cybersecurity requirements, coordinating with both state and federal agencies when assessing state agencies’ cybersecurity may help to minimize additional cost and time impacts on state agencies, and reduce federal resources associated with implementing state-based cybersecurity assessments. Until OMB takes steps to ensure federal agencies coordinate on assessments of state agencies’ cybersecurity, it will not have reasonable assurance federal agencies are leveraging compatible assessments to the greatest extent possible. We are making a total of 12 recommendations, including two to OMB, two to CMS, three to FBI, two to IRS, and three to SSA. The Director of OMB should ensure that CMS, FBI, IRS, and SSA are collaborating on their cybersecurity requirements pertaining to state agencies to the greatest extent possible and direct further coordination where needed. (Recommendation 1) The Director of OMB should take steps to ensure that CMS, FBI, IRS, and SSA coordinate, where feasible, on assessments of state agencies’ cybersecurity, which may include steps such as leveraging other agencies’ security assessments or conducting assessments jointly. (Recommendation 2) The Administrator of CMS should, in collaboration with OMB, solicit input from FBI, IRS, SSA, and state agency stakeholders on revisions to its security policy to ensure that cybersecurity requirements for state agencies are consistent with other federal agencies and NIST guidance to the greatest extent possible and document CMS’s rationale for maintaining any requirements variances. (Recommendation 3) The Administrator of CMS should revise its assessment policies to maximize coordination with other federal agencies to the greatest extent practicable. (Recommendation 4) The FBI Director should, in collaboration with OMB, solicit input from CMS, IRS, SSA, and state agency stakeholders on revisions to its security policy to ensure that cybersecurity requirements for state agencies are consistent with other federal agencies and NIST guidance to the greatest extent possible. (Recommendation 5) The FBI Director should fully develop policies for coordinating with state agencies on the use of prior findings from relevant cybersecurity assessments conducted by other organizations. (Recommendation 6) The FBI Director should revise its assessment policies to maximize coordination with other federal agencies to the greatest extent practicable. (Recommendation 7) The IRS Commissioner should, in collaboration with OMB, solicit input from CMS, FBI, SSA, and state agency stakeholders on revisions to its security policy to ensure that cybersecurity requirements for state agencies are consistent with other federal agencies and NIST guidance to the greatest extent possible. (Recommendation 8) The IRS Commissioner should revise its assessment policies to maximize coordination with other federal agencies to the greatest extent practicable. (Recommendation 9) The Commissioner of SSA should, in collaboration with OMB, solicit input from CMS, FBI, IRS, and state agency stakeholders on revisions to its security policy to ensure that cybersecurity requirements for state agencies are consistent with other federal agencies and NIST guidance to the greatest extent possible and document the SSA’s rationale for maintaining any requirements variances. (Recommendation 10) The Commissioner of SSA should fully develop policies for coordinating with state agencies on the use of prior findings from relevant cybersecurity assessments conducted by other organizations. (Recommendation 11) The Commissioner of SSA should revise its assessment policies to maximize coordination with other federal agencies to the greatest extent practicable. (Recommendation 12) We provided a draft of this report to OMB and the four other selected federal agencies for their review and comment. In response, three of the agencies (Department of Health and Human Services, FBI, and SSA) stated that they agreed with the recommendations; and one agency (IRS) stated that it partially agreed with one recommendation and disagreed with one recommendation. OMB did not provide comments on our report. The following three agencies agreed with the recommendations. The Department of Health and Human Services provided written comments, in which it agreed with our recommendations and identified steps it said CMS had taken or intends to take to address them. For example, the department stated that CMS intends to solicit input from the other federal agencies identified in this report and from state agency stakeholders when making updates to its MARS-E security policy and when updating its assessment guidance to states on how to maximize coordination with other federal entities. The department noted that CMS had developed and implemented its suite of guidance and requirements, known as MARS-E, based on the Patient Protection and Affordable Care Act, FISMA, and NIST. According to the department, variances in security requirements are to be expected because of the flexibility that NIST allows in its guidance. The department added that CMS tailored some of the controls to allow flexibilities for states while keeping the overall intent of the NIST guidance. The department stated that it collaborated with federal agencies, including FBI's CJIS, in developing MARS-E and during subsequent updates of that security policy. However, CMS did not provide us with documentation as evidence of its collaboration with FBI's CJIS on the development of MARS-E. In addition, as noted in this report, CMS had not collaborated with the other agencies included in our review after the development of the most recent version of MARS-E. It is important that federal agencies collaborate to address variances in their cybersecurity requirements; doing so could help to significantly reduce state agencies’ costs in complying with multiple federal agencies’ requirements. The department's comments are reprinted in appendix IV. The department also provided technical comments, which we incorporated as appropriate. In written comments, FBI's CJIS agreed with our three recommendations to the agency. Among other things, the agency stated that it would, to the greatest extent possible, collaborate with OMB and solicit input from the other federal agencies identified in this report, as well as from state agency stakeholders, on revisions to its security policy. With regard to our recommendation that FBI’s CJIS develop policies for coordinating with state agencies on the use of prior findings, the agency stated that it had implemented this recommendation and updated its security policy to include coordinating with state agencies on the use of prior findings from relevant cybersecurity assessments conducted by other organizations. However, the agency did not provide documentation showing that it had updated the security policy. As a result, we did not change our assessment of this practice. We will continue to monitor the agency’s progress in implementing the recommendation. The agency's comments are reprinted in appendix V. The agency also provided technical comments, which we incorporated as appropriate. In its written comments, SSA stated that it agreed with our recommendations. SSA's comments are reprinted in appendix VI. The agency also provided technical comments, which we incorporated as appropriate. One agency partially agreed with one recommendation and disagreed with one recommendation. Specifically, IRS partially agreed with our recommendation to, in collaboration with OMB, solicit input from the four federal agencies identified in this report and state agency stakeholders on revisions to its security policy. Specifically, the agency agreed to participate in collaborative working sessions with OMB and interested stakeholders to discuss the impact of inconsistent standards and the extent to which the standards might be harmonized. However, IRS stated that it must follow Treasury Directives and internal standards for systems that process tax data and, as a result, its ability to harmonize requirements may be limited. As noted in this report, federal agencies may have legitimate reasons for variances in their cybersecurity requirements, such as applying different information security controls and more stringent technical parameters to protect data for which the agencies are responsible in a manner consistent with various security requirements originating in federal laws, directives, and regulations. Nevertheless, we continue to believe that it is important for all of the agencies in our review to identify opportunities where requirements can be streamlined or made more consistent while still achieving each agency's desired security outcomes because doing so may reduce potential burdens on state agencies, as discussed in this report. Thus, we maintain that our recommendation is still warranted. IRS disagreed with our recommendation to revise its assessment policies to maximize coordination with other federal agencies to the greatest extent possible. Specifically, IRS stated that it has sole statutory oversight authority and enforces requirements for agencies subject to Internal Revenue Code § 6103. As such, IRS cannot solely rely on an assessment conducted by another agency. However, as noted in this report, OMB encourages federal agencies to coordinate on their assessments whenever practicable. Doing so would not necessarily require IRS to solely rely on another agency’s assessment nor conflict with its authority to conduct statutory oversight because IRS could leverage and share relevant information and artifacts with other federal agencies while continuing to conduct its own required assessments and oversight. Further, as discussed in this report, state chief information officers identified a number of areas where federal agencies requested similar information through documentation requests and interviews, such as network configurations, password policies, and incident response policies. Leveraging and sharing relevant information that is collected by federal agencies could help those agencies, including IRS, reduce some of their data collection needs while also helping to minimize burdens on state officials’ time and resources. We acknowledge that complete alignment of assessment policies may not be feasible in light of unique statutory responsibilities and requirements; however, agency coordination and simplification of certain assessment logistics may be possible and could result in gained efficiencies from the perspective of the federal government. Thus, we maintain that our recommendation is still warranted. IRS's comments are reprinted in appendix VII. We are sending copies of this report to the appropriate congressional requesters, the Director of OMB, the Administrator of CMS, the Assistant Attorney General for Administration for the Department of Justice, the FBI Director, the IRS Commissioner, and the Commissioner of SSA. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6240 or at dsouzav@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIlI. We administered a survey to the offices of the Chief Information Officer and Chief Information Security Officer (CISO) in the 50 states, District of Columbia, American Samoa, Guam, Puerto Rico, and the U.S. Virgin Islands. To administer the survey, we emailed each state a fillable PDF questionnaire. We fielded the survey from February 19, 2019, through April 24, 2019. We received usable survey responses from 50 of the 55 states and territories, for a response rate of 91 percent. In developing, administering, and analyzing the survey, we took steps to minimize the five types of errors that may affect survey results— population coverage, sampling, measurement, nonresponse, and data processing. Our results are not subject to either of the first two types of errors—population coverage (under- or over-coverage) error of the study population or sampling error—because we defined all states and five territories as our study population, and sent each a questionnaire. To minimize the third type of error (measurement error), we pretested the questionnaire with CISOs (or their delegates) in four states that varied over two characteristics related to our questions: whether or not the state took a “federated” or “consolidated” management approach to data center and other information technology (IT) infrastructure, and the relative size of the state’s IT budget. Using cognitive interviewing techniques, such as nondirective probing of answers and asking respondents to think aloud when formulating answers, we determined whether (1) the questions were clear and unambiguous, (2) terminology was used correctly, (3) the questionnaire did not place an undue burden on state officials, (4) the information could feasibly be obtained, and (5) the survey was comprehensive and unbiased. Based on the pretests and interviews with external subject matter experts on questionnaire subjects, we modified the questionnaire. During the survey, we also followed up by email or phone with some respondents to clarify unclear answers and edit them if necessary. Additionally, after the survey, our in-depth interviews with four responding states confirmed their answers to selected questions, or resulted in edits to those answers. To minimize the potential for the fourth type of error (nonresponse error), we emailed or called states that did not respond to the initial notice multiple times to encourage survey participation or provide technical assistance, as appropriate. Also, the follow up contacts made to clarify answers resulted in obtaining some answers to questions that had been left blank in returned questionnaires. While the four states and one territory not returning questionnaires may have differed to an unknown extent in what their answers would have been, compared to the aggregate answers of those who did respond, the overall impact on our results from only five missing members of the population is unlikely to be material. To minimize the possibility for the fifth type of error (data processing error), all data entry, edits, and analysis were verified by a second, independent analyst on the engagement team. To further understand the states’ experiences with and views of selected federal agencies’ cybersecurity assessments, we conducted in-depth interviews with four states. In selecting the four states for in-depth interviews, we considered responses from 44 states that had submitted surveys prior to April 11, 2019. From these states, we analyzed responses to survey questions 4, 7, 9, 10, 11, 12, 13, 14, 15, 16, and 17, and identified whether states’ responses reflected a generally favorable opinion or a generally unfavorable opinion of federal cybersecurity requirements and assessments. Based on this information, we selected two states to interview that had a generally favorable opinion and two states that had a generally unfavorable opinion toward federal cybersecurity assessments and requirements. In selecting states to interview from states that had favorable and unfavorable opinions, we chose to interview states that provided different responses about increases in costs and/or coordination with federal and nonfederal agencies. We sent an email to each of the four states to ask for their participation and conducted follow up interviews with officials from the offices of the state CIO and state CISO, state audit entities, and mission agencies from four states. Our interview questions concerned topics such as challenges states may have faced in complying with federal cybersecurity requirements, the impacts federal requirements and assessments may have had on states, the efficiency and effectiveness of assessments performed by each federal agency, and the nature and extent of any duplication in federal agencies’ cybersecurity requirements. Although the results of these in-depth interviews are not generalizable to all of the states and territories that responded to our survey, they provide richer insight into some of the information we collected through our survey, such as the reasons for certain questionnaire responses or the sources of variation in states’ perspectives. The following identifies the survey questionnaire that we administered and the aggregated results from the responses are below under each question. Not all state CISOs who completed the survey responded to all questions, and some questions were not discussed in the body of our report. These questions ask about the federal agency cybersecurity requirements that set standards in any of the related general security control categories, and your experiences with those applicable to your state. 1. For how long has the current CISO of your state been in that role? (check one box) 2. Please provide some background on your state’s governance model for cybersecurity. Specifically, how is the responsibility for managing the following aspects of cybersecurity primarily assigned within your state? (check the one box in each row which best represents your answer) 3. Is your state currently required to meet any security requirements by any of the following federal agencies in order to obtain and use federal data? Federal Bureau of Investigation (FBI) (Criminal Justice Information Services (CJIS) Security Policy CJISD-ITS-DOC-08140-5.7, Version 5.7) Centers for Medicare & Medicaid Services (CMS) (Minimum Acceptable Risk Standards for Exchanges, Version 2.0) Internal Revenue Service (IRS) (IRS Publication 1075, Tax Information Security Guidelines For Federal, State, and Local Agencies, September 2016) Social Security Administration (SSA) (Electronic Information Exchange Security Requirements and Procedures for State and Local Agencies Exchanging Electronic Information, Version 8.0) 4. Federal security requirements applicable to states may vary in a number of ways. Considering as a whole all of the federal agencies’ requirements that your state is currently required to meet, how much do you think they vary from each other in each of the following ways? 5. Consider again all the applicable federal cybersecurity requirements required of your state. Do one or more federal agencies have any requirements that most vary from other agencies? Within each of the following families of security controls, check all boxes that apply to tell us in what ways requirements vary, and which agency(s) vary the most from others. (If “Other(s)” varying agencies selected, list in Question 6.) NIST Control Family Access Control (AC) Awareness and Training (AT) Audit and Accountability (AU) Security Assessment and Authorization (CA) Configuration and Management (CM) Contingency Planning (CP) Identification and Authentication (IA) Incident Response (IR) Media Protection (MP) Physical and Environmental Protection (PE) Planning (PL) Personnel Security (PS) Risk Assessment (RA) System and Services Acquisition (SA) System and Communications Protection (SC) System and Information Integrity (SI) Program management (PM) 6. If you indicated above that any other federal agencies have requirements that most vary from others, what are those other agencies and the control categories and way(s) they vary? (Narrative answers not displayed) 7. If you identified any variation in the requirements of multiple Federal agencies in question 5 above, what is your overall estimation of the degree of that variation in each of the following families of controls? Families of controls (Based on NIST 800-53) Access control (AC) Awareness and training (AT) Audit and accountability (AU) Security assessment and authorization (CA) Configuration management (CM) Contingency planning (CP) Identification and authentication (IA) Incident response (IR) Maintenance (MA) Media protection (MP) Physical and environmental protection (PE) Planning (PL) Personnel security (PS) Risk assessment (RA) System and services acquisition (SA) System and communications protection (SC) System and information integrity (SI) 8. Do you have any comments on or explanations of your answers to the question above that would help us appropriately interpret those answers? (itemize your comments by the row letters above, to the extent possible, in the box below) (Narrative answers not displayed) 9. Has your state taken any of the following actions specifically to address variation(s) across agency requirements? Increased coordination with NASCIO and other non-federal agencies outside your state Increased coordination with other agencies within your state Any other action(s) 10. Have the variations increased any of the following types of costs and/or challenges? The following questions ask about assessments performed by federal agencies on your state on its compliance with the federal cybersecurity requirements covered above. For the purposes of this survey, an “assessment” includes only the activities in the period between the date the state is notified of the assessment and the date the federal agency or entity carrying out the assessment (e.g., contractor) completes its on-site work. 11. Approximately how many assessments did each of the following federal agencies perform on your state’s efforts to comply with its requirements in calendar years 2016-2018? (When counting assessments performed by one federal agency on more than one state mission agency or operational entity at the same time, please count each assessment individually.) Any other federal agency(s) 12. Considering up to the last 3 assessments a federal agency performed in 2016-2018, approximately how long in calendar time was taken per assessment, on average, to perform? Any other federal agency(s) 13. Considering up to the last 3 assessments a federal agency performed in 2016-2018, approximately how many of your state’s staff hours were expended per assessment, on average, to comply? Any other federal agency(s) 14. And considering up to the last 3 assessments a federal agency performed in 2016-2018, what is your best estimate of the range of cost in dollars (including staff hour labor, travel, materials, and contract costs) your state expended per assessment, on average, to comply? Estimated lower end of dollar cost (mean value) $77,103 (17 responses) Estimated upper end of dollar cost (mean value) Don’t know 28 (17 responses) $623,650 19 responses) $840,472 (19 responses) $211,574 (21 responses) $418,238 (21 responses) $33,822 (16 responses) $61,719 (16 responses) 15. Considering all the federal assessments performed on your state’s implementation of requirements in 2016-2018, how would you rate those assessments, overall, on the following factors? 16. In summary, how would you rate the efficiency of assessments performed by each federal agency on your state’s implementation of requirements? Any other agency(s) 17. In summary, how would you rate the effectiveness of assessments performed by federal agencies on your state’s implementation of requirements? Any other agency(s) 18. Considering the issues covered in this questionnaire, what challenges or impacts, if any, has your state experienced regarding the federal requirements and assessment processes? (list and describe up to 5) (Narrative answers not displayed) 19. Do you have any additional explanations of your answers or comments on any of the issues in this questionnaire? (Narrative answers not displayed) 20. Who is the person primarily responsible for completing this questionnaire whom we can contact in case we need to clarify a response? If the state CISO did not complete this questionnaire, we recommend that the CISO review these answers. The tables below identify the extent to which each of the four selected federal agencies established policies that addressed individual activities supporting two areas of coordination: (1) coordination with state agencies when assessing states’ cybersecurity and (2) coordination with other federal agencies on the assessment of state agencies’ cybersecurity. In addition to the individual named above, Josh Leiling (assistant director), Lori Martinez (analyst in charge), Gerard Aflague, Joseph Andrews, David Blanding, Chris Businsky, Rebecca Eyler, Torrey Hardee, Andrea Harvey, Keith Kim, Monica Perez-Nelson, and Carl Ramirez made significant contributions to this report.", "summary": "To protect data that are shared with state government agencies, federal agencies have established cybersecurity requirements and related compliance assessment programs. Specifically, they have numerous cybersecurity requirements for states to follow when accessing, storing, and transmitting federal data. GAO was asked to evaluate federal agencies' cybersecurity requirements and related assessment programs for state agencies. The objectives were to determine the extent to which (1) selected federal agencies' cybersecurity requirements for state agencies varied with each other and federal guidance, and (2) federal agencies had policies for coordinating their assessments of state agencies' cybersecurity. GAO reviewed four federal agencies that shared data with states and had assessment programs: CMS, FBI, IRS, and SSA. GAO compared, among other things, each agency's cybersecurity requirements to federal guidance and to other selected agencies' requirements; and reviewed federal agencies' policies for conducting assessments. In addition, GAO examined OMB's efforts to foster coordination among federal agencies. GAO also surveyed and received responses from chief information security officers in 50 out of 55 U.S. states, territories, and the District of Columbia to obtain their perspectives. Although the Centers for Medicare and Medicaid Services (CMS), Federal Bureau of Investigation (FBI), Internal Revenue Service (IRS), and Social Security Administration (SSA) each established requirements to secure data that states receive, these requirements often had conflicting parameters. Such parameters involve agencies defining specific values like the number of consecutive unsuccessful logon attempts prior to locking out the user. Among the four federal agencies, the percentage of total requirements with conflicting parameters ranged from 49 percent to 79 percent. Regarding variance with National Institute of Standards and Technology guidance, GAO found that the extent to which the four agencies did not fully address guidance varied from 9 percent to 53 percent of total requirements. The variances were due in part to the federal agencies' insufficient coordination in establishing requirements. Although the Office of Management and Budget's (OMB) Circular A-130 requires agencies to coordinate, OMB has not ensured that agencies have done so. Further, while federal agencies' variance among requirements may be justified in some cases because of particular agency mission needs, the resulting impact on states is significant, according to state chief information security officers (see figure). The four federal agencies that GAO reviewed either fully or partially had policies for coordinating assessments with states, but none of them had policies for coordinating assessments with each other. State chief information security officers that GAO surveyed reinforced the need to coordinate assessments by identifying impacts on state agencies' costs, including multiple federal agencies that requested the same documentation. Coordinating with state and federal agencies when assessing state agencies' cybersecurity may help to minimize states' cost and time impacts and reduce associated federal costs. Federal agencies reported spending about $45 million for fiscal years 2016 through 2018 on assessments of state agencies' cybersecurity. GAO is making 12 recommendations to the four selected agencies and to OMB. Three agencies agreed with the recommendations and one agency (IRS) partially agreed or disagreed with them. OMB did not provide comments. GAO continues to believe all recommendations are warranted.", "document_type": "gao"}
{"report": "The federal government plans to invest over $90 billion for IT in fiscal year 2019. Nevertheless, we have previously reported that investments in federal IT too often resulted in failed projects that incurred cost overruns and schedule slippages, while contributing little to the desired mission- related outcomes. For example: The United States Coast Guard (Coast Guard) decided to terminate its Integrated Health Information System project in 2015. As reported by the agency in August 2017, the Coast Guard spent approximately $60 million over 7 years on this project, which resulted in no equipment or software that could be used for future efforts. The Department of Veterans Affairs’ Financial and Logistics Integrated Technology Enterprise program was intended to be delivered by 2014 at a total estimated cost of $609 million, but was terminated in October 2011. The Department of Defense’s Expeditionary Combat Support System was canceled in December 2012 after spending more than a billion dollars and failing to deploy within 5 years of initially obligating funds. The Department of Homeland Security’s (DHS) Secure Border Initiative Network program was ended in January 2011, after the department obligated more than $1 billion for the program. Our past work has found that these and other failed IT projects often suffered from a lack of disciplined and effective management, such as project planning, requirements definition, and program oversight and governance. In many instances, agencies had not consistently applied best practices that are critical to successfully acquiring IT. Federal IT projects have also failed due to a lack of oversight and governance. Executive-level governance and oversight across the government has often been ineffective, specifically from CIOs. For example, we have reported that some CIOs’ roles were limited because they did not have the authority to review and approve the entire agency IT portfolio. In addition to failures when acquiring IT, our cybersecurity work at federal agencies continues to highlight information security deficiencies. The following examples describe the types of risks we have found at federal agencies. In September 2018, we reported that the Department of Education’s Office of Federal Student Aid exercised minimal oversight of lenders’ protection of student data and lacked assurance that appropriate risk- based safeguards were being effectively implemented, tested, and monitored. In August 2017, we issued a report stating that, since the 2015 data breaches, the Office of Personnel Management (OPM) had taken actions to prevent, mitigate, and respond to data breaches involving sensitive personal and background investigation information. However, we noted that the agency had not fully implemented recommendations that DHS’s United States Computer Emergency Readiness Team made to OPM to help the agency improve its overall security posture and improve its ability to protect its systems and information from security breaches. We reported in July 2017 that information security at the Internal Revenue Service had weaknesses that limited its effectiveness in protecting the confidentiality, integrity, and availability of financial and sensitive taxpayer data. An underlying reason for these weaknesses was that the Internal Revenue Service had not effectively implemented elements of its information security program. We reported in August 2016 that the information security of the Food and Drug Administration had significant weaknesses that jeopardized the confidentiality, integrity, and availability of its information systems and industry and public health data. In May 2016, we found that the National Aeronautics and Space Administration, the Nuclear Regulatory Commission, OPM, and the Department of Veteran Affairs did not always control access to selected high-impact systems, patch known software vulnerabilities, or plan for contingencies. An underlying reason for these weaknesses was that the agencies had not fully implemented key elements of their information security programs. Congress and the President have enacted various key pieces of reform legislation to address IT management issues. These include the federal IT acquisition reform legislation commonly referred to as the Federal Information Technology Acquisition Reform Act (FITARA). This legislation was intended to improve covered agencies’ acquisitions of IT and enable Congress to monitor agencies’ progress and hold them accountable for reducing duplication and achieving cost savings. The law includes specific requirements related to seven areas: Agency CIO authority enhancements. CIOs at covered agencies have the authority to, among other things, (1) approve the IT budget requests of their respective agencies and (2) review and approve IT contracts. Federal data center consolidation initiative (FDCCI). Agencies covered by FITARA are required, among other things, to provide a strategy for consolidating and optimizing their data centers and issue quarterly updates on the progress made. Enhanced transparency and improved risk management. The Office of Management and Budget (OMB) and covered agencies are to make detailed information on federal IT investments publicly available, and agency CIOs are to categorize their investments by level of risk. Portfolio review. Covered agencies are to annually review IT investment portfolios in order to, among other things, increase efficiency and effectiveness and identify potential waste and duplication. Expansion of training and use of IT acquisition cadres. Covered agencies are to update their acquisition human capital plans to support timely and effective IT acquisitions. In doing so, the law calls for agencies to consider, among other things, establishing IT acquisition cadres (i.e., multi-functional groups of professionals to acquire and manage complex programs), or developing agreements with other agencies that have such cadres. Government-wide software purchasing program. The General Services Administration is to develop a strategic sourcing initiative to enhance government-wide acquisition and management of software. In doing so, the law requires that, to the maximum extent practicable, the General Services Administration should allow for the purchase of a software license agreement that is available for use by all executive branch agencies as a single user. Maximizing the benefit of the Federal Strategic Sourcing Initiative. Federal agencies are required to compare their purchases of services and supplies to what is offered under the Federal Strategic Sourcing Initiative. In June 2015, OMB released guidance describing how agencies are to implement FITARA. This guidance was intended to, among other things: assist agencies in aligning their IT resources with statutory requirements; establish government-wide IT management controls to meet the law’s requirements, while providing agencies with flexibility to adapt to unique agency processes and requirements; strengthen the relationship between agency CIOs and bureau CIOs; and strengthen CIO accountability for IT costs, schedules, performance, and security. The guidance identifies a number of actions that agencies are to take to establish a basic set of roles and responsibilities (referred to as the common baseline) for CIOs and other senior agency officials and, thus, to implement the authorities described in the law. For example, agencies are to conduct a self-assessment and submit a plan describing the changes they intend to make to ensure that common baseline responsibilities are implemented. In addition, in August 2016, OMB released guidance intended to, among other things, define a framework for achieving the data center consolidation and optimization requirements of FITARA. The guidance directed agencies to develop a data center consolidation and optimization strategic plan that defined the agency’s data center strategy for fiscal years 2016, 2017, and 2018. This strategy was to include, among other things, a statement from the agency CIO indicating whether the agency had complied with all data center reporting requirements in FITARA. Further, the guidance states that OMB is to maintain a public dashboard to display consolidation-related costs savings and optimization performance information for the agencies. In June 2019, OMB issued Memorandum M-19-19, which updated the data center optimization initiative and redefined a data center as a purpose-built, physically separate, dedicated space that meets certain criteria. It also revised the priorities for consolidating and optimizing federal data centers. Specifically, OMB directed agencies to report on spaces designed to be data centers (i.e., tiered data centers) as part of their inventories and to focus efforts on data centers that host business applications, rather than special purpose data centers. According to OMB’s August 2019 quarterly reporting instructions, non-tiered data centers may be flagged for removal in one reporting period and removed in the next unless OMB provides a written denial within a specified time frame. In addition, OMB described criteria for designating certain data centers as mission critical facilities, and would therefore not be taken into consideration when setting new agency-specific closure targets. Those mission critical designations are also assumed to be granted unless OMB specifically overturns them. Regarding cost savings, OMB’s new memorandum, M-19-19, noted that agency-specific targets would be set in collaboration with each agency and appropriately aligned to that agency’s mission and budget. OMB’s new memorandum also replaced the previous optimization metrics with new measures that focus on reporting the numbers of agencies’ virtualized hosts, underutilized servers, data centers with advanced energy metering, and the percentage of time that data centers were expected to be available to provide services. In contrast to the previous guidance, the new memorandum does not specify government-wide performance targets for the optimization metrics. Instead, OMB worked with agencies to establish agency-specific targets. In addition, the guidance describes how agencies could apply for an optimization performance exemption for data centers where typical optimization activities (consolidation of data collection, storage, and processing to a central location) are technically possible but increase the response time for systems beyond a reasonable limit. Congress has recognized the importance of agencies’ continued implementation of FITARA provisions, and has taken legislative action to extend selected provisions beyond their original dates of expiration. Specifically, Congress and the President enacted laws to: remove the expiration dates for the enhanced transparency and improved risk management provisions, which were set to expire in 2019; remove the expiration date for portfolio review, which was set to expire in 2019; and extend the expiration date for FDCCI from 2018 to 2020. In addition, Congress and the President enacted a law to authorize the availability of funding mechanisms to help further agencies’ efforts to modernize IT. The law, known as the Modernizing Government Technology (MGT) Act, authorizes agencies to establish working capital funds for use in transitioning away from legacy IT systems, as well as for addressing evolving threats to information security. The law also creates the Technology Modernization Fund within the Department of the Treasury, from which agencies can “borrow” money to retire and replace legacy systems, as well as to acquire or develop systems. Further, in February 2018, OMB issued guidance for agencies on implementing the MGT Act. The guidance was intended to provide agencies additional information regarding the Technology Modernization Fund, as well as the administration and funding of the related IT working capital funds. Specifically, the guidance encouraged agencies to begin submitting initial project proposals for modernization on February 27, 2018. In addition, in accordance with the MGT Act, the guidance provided details regarding a Technology Modernization Board, which is to consist of (1) the Federal CIO; (2) a senior IT official from the General Services Administration; (3) a member of DHS’s National Protection and Program Directorate; and (4) four federal employees with technical expertise in IT development, financial management, cybersecurity and privacy, and acquisition that were appointed by the Director of OMB. Congress and the President enacted the Federal Information Security Modernization Act of 2014 (FISMA) to improve federal cybersecurity and clarify government-wide responsibilities. The act highlights the increasing sophistication of cybersecurity attacks, promotes the use of automated security tools with the ability to continuously monitor and diagnose the security posture of federal agencies, and provides for improved oversight of federal agencies’ information security programs. To this end, the act clarifies and assigns specific responsibilities to entities such as OMB, DHS, and the federal agencies. Table 1 describes a selection of the OMB, DHS, and agency responsibilities. Beyond the implementation of FITARA, FISMA, and related actions, the administration has also initiated other efforts intended to improve federal IT and the nation’s cybersecurity. Specifically, in March 2017, the administration established the Office of American Innovation, which has a mission to, among other things, make recommendations to the President on policies and plans aimed at improving federal government operations and services. In doing so, the office is to consult with both OMB and the Office of Science and Technology Policy on policies and plans intended to improve government operations and services, improve the quality of life for Americans, and spur job creation. In May 2017, the Administration also established the American Technology Council, which has a goal of helping to transform and modernize federal agency IT and how the federal government uses and delivers digital services. The President is the chairman of this council, and the Federal CIO and the United States Digital Service Administrator are among the members. In addition, in May 2017, the President signed Executive Order 13800, Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure. This executive order outlined actions to enhance cybersecurity across federal agencies and critical infrastructure to improve the nation’s cyber posture and capabilities against cybersecurity threats. Among other things, the order tasked the Director of the American Technology Council to coordinate a report to the President from the Secretary of DHS, the Director of OMB, and the Administrator of the General Services Administration, in consultation with the Secretary of Commerce, regarding the modernization of federal IT. In response, the Report to the President on Federal IT Modernization was issued in December 2017 and outlined the current and envisioned state of federal IT. The report focused on modernization efforts to improve the security posture of federal IT. Further, it recognized that agencies have attempted to modernize systems but have been stymied by a variety of factors, including resource prioritization, ability to procure services quickly, and technical issues. The report provided multiple recommendations intended to address these issues through the modernization and consolidation of networks and the use of shared services to enable future network architectures. Further, in March 2018, the Administration issued the President’s Management Agenda, which laid out a long-term vision for modernizing the federal government. The agenda identified three related drivers of transformation—IT modernization; data, accountability, and transparency; and the workforce of the future—that are intended to push change across the federal government. The Administration also established 14 related Cross-Agency Priority goals, many of which have elements that involve IT. In particular, the Cross-Agency Priority goal on IT modernization stated that modern IT must function as the backbone of how government serves the public in the digital age. This goal established three priorities that are to guide the Administration’s efforts to modernize federal IT: (1) enhancing mission effectiveness by improving the quality and efficiency of critical services, including the increased utilization of cloud-based solutions; (2) reducing cybersecurity risks to the federal mission by leveraging current commercial capabilities and implementing cutting edge cybersecurity capabilities; and (3) building a modern IT workforce by recruiting, reskilling, and retaining professionals able to help drive modernization with up-to-date technology. On May 15, 2018, the President signed Executive Order 13833: Enhancing the Effectiveness of Agency Chief Information Officers. Among other things, this executive order was intended to better position agencies to modernize their IT systems, execute IT programs more efficiently, and reduce cybersecurity risks. The order pertains to 22 of the 24 Chief Financial Officers (CFO) Act agencies; the Department of Defense and the Nuclear Regulatory Commission are exempt. For the covered agencies, the executive order strengthened the role of agency CIOs by, among other things, requiring them to report directly to their agency head; serve as their agency head’s primary IT strategic advisor; and have a significant role in all management, governance, and oversight processes related to IT. In addition, one of the cybersecurity requirements directed agencies to ensure that the CIO works closely with an integrated team of senior executives, including those with expertise in IT, security, and privacy, to implement appropriate risk management measures. In the March 2019 update to our high-risk series, we reported that agencies still needed to complete significant work related to the management of IT acquisitions and operations. As government-wide spending on IT increases every year, the need for appropriate stewardship of that investment increases as well. However, we pointed out that OMB and federal agencies have not made significant progress since 2017 in taking the steps needed to improve how these financial resources are budgeted and realized. To address this issue, we highlighted the need for OMB and federal agencies to further implement the requirements of federal IT acquisition reforms, including the enhancement of CIO authority. Our update to the IT acquisitions and operations high-risk area also stressed that OMB and agencies needed to continue to implement our prior recommendations in order to improve their ability to effectively and efficiently invest in IT. Specifically, since fiscal year 2010, we have made 1,320 recommendations and one matter for Congressional consideration to address shortcomings in IT acquisitions and operations. As stated in our 2019 high-risk update, OMB and agencies should demonstrate government-wide progress by, among other things, implementing at least 80 percent of our recommendations related to managing IT acquisitions and operations. As of November 2019, OMB and agencies had fully implemented 807 (or 61 percent) of their 1,320 recommendations. In addition, the matter for Congressional consideration had also been implemented. Figure 1 summarizes the progress that OMB and agencies have made in addressing our recommendations compared to the 80 percent target. Overall, federal agencies would be better positioned to realize billions in cost savings and additional management improvements if they address these recommendations, including those aimed at implementing CIO responsibilities, reviewing IT acquisitions, improving data center consolidation, and managing software licenses. In all, various laws, such as FITARA and related guidance, assign 35 IT management responsibilities to CIOs in six key areas. These areas are: leadership and accountability, budgeting, information security, investment management, workforce, and strategic planning. In August 2018, we reported that none of the 24 agencies we reviewed had policies that fully addressed the role of their CIO, as called for by federal laws and guidance. In this regard, a majority of the agencies had fully or substantially addressed the role of their CIOs for the area of leadership and accountability. In addition, a majority of the agencies had substantially or partially addressed the role of their CIOs for two areas: information security and IT budgeting. However, most agencies had partially or minimally addressed the role of their CIOs for two areas: investment management and strategic planning. Further, the majority of the agencies minimally addressed or did not address the role of their CIOs for the remaining area: IT workforce. Figure 2 depicts the extent to which the 24 agencies had policies that addressed the role of their CIOs for the six areas. Notwithstanding the shortfalls in agencies’ policies addressing the roles of their CIOs, most agency officials stated that their CIOs are implementing the responsibilities even if the agencies do not have policies requiring implementation. Nevertheless, in their responses to our survey, the CIOs of the 24 selected agencies acknowledged that they were not always very effective in implementing the six IT management areas. Specifically, at least 10 of the CIOs indicated that they were less than very effective for each of the six areas of responsibility. We believe that until agencies fully address the role of CIOs in their policies, they will be limited in addressing longstanding IT management challenges. Figure 3 depicts the extent to which the CIOs reported their effectiveness in implementing the six areas of responsibility. Beyond the actions of the agencies, however, shortcomings in agencies’ policies were also partially attributable to two weaknesses in OMB’s guidance. First, the guidance did not comprehensively address all CIO responsibilities, such as those related to assessing the extent to which personnel meet IT management knowledge and skill requirements and ensuring that personnel are held accountable for complying with the information security program. Correspondingly, the majority of the agencies’ policies did not fully address nearly all of the responsibilities that were not included in OMB’s guidance. Second, OMB’s guidance did not ensure that CIOs had a significant role in (1) IT planning, programming, and budgeting decisions; and (2) execution decisions and the management, governance, and oversight processes related to IT, as required by federal law and guidance. In the absence of comprehensive guidance, CIOs would not be positioned to effectively acquire, maintain, and secure their IT systems. In response to the survey conducted for our August 2018 report, the 24 agency CIOs also identified a number of factors that enabled and challenged their ability to effectively manage IT. Specifically, most agency CIOs cited five factors as being enablers to effectively carrying out their responsibilities: (1) NIST guidance, (2) the CIO’s position within the agency hierarchy, (3) OMB guidance, (4) coordination with the Chief Acquisition Officer (CAO), and (5) legal authority. Further, the CIOs cited three factors as major challenges to their ability to effectively carry out responsibilities: (1) processes for hiring, recruiting, and retaining IT personnel; (2) financial resources; and (3) the availability of personnel/staff resources. As shown in figure 4, the five enabling factors were identified by at least half of the 24 CIOs and the three factors cited as major challenges were identified by at least half of the CIOs. Although OMB issued guidance aimed at addressing the three factors identified by a majority of the CIOs as major challenges, the guidance did not fully do so. Further, regarding the financial resources challenge, OMB recently required agencies to provide data on CIO authority over IT spending; however, its guidance did not provide a complete definition of that authority. In the absence of such guidance, agencies created varying definitions of CIO authority. Until OMB updates its guidance to include a complete definition of the authority that CIOs are to have over IT spending, it will be difficult for OMB to identify any deficiencies in this area and to help agencies make any needed improvements. In order to address challenges in implementing CIO responsibilities, we made three recommendations to OMB and one recommendation to each of the selected 24 federal agencies for each of the six IT management areas. Most agencies agreed with or had no comments on the recommendations. However, as of November 2019, none of the 27 recommendations had been implemented. We will continue to monitor the implementation of these recommendations. FITARA includes a provision to enhance covered agency CIOs’ authority through, among other things, requiring agency heads to ensure that CIOs review and approve IT contracts. OMB’s FITARA implementation guidance expanded upon this aspect of the legislation in a number of ways. Specifically, according to the guidance: CIOs may review and approve IT acquisition strategies and plans, rather than individual IT contracts; CIOs can designate other agency officials to act as their representatives, but the CIOs must retain accountability; CAOs are responsible for ensuring that all IT contract actions are consistent with CIO-approved acquisition strategies and plans; and CAOs are to indicate to the CIOs when acquisition strategies and acquisition plans include IT. In January 2018, we reported that most of the CIOs at 22 selected agencies were not adequately involved in reviewing billions of dollars of IT acquisitions. For instance, most of the 22 agencies did not identify all of their IT contracts. In this regard, the agencies identified 78,249 IT- related contracts, to which they obligated $14.7 billion in fiscal year 2016. However, we identified 31,493 additional IT contracts with combined obligations totaling $4.5 billion, raising the total amount obligated to IT contracts by these agencies in fiscal year 2016 to at least $19.2 billion. Figure 5 reflects the obligations that the 22 selected agencies reported to us relative to the obligations we identified. The percentage of additional IT contract obligations we identified varied among the selected agencies. For example, the Department of State did not identify 1 percent of its IT contract obligations. Conversely, eight agencies did not identify over 40 percent of their IT contract obligations. Many of the selected agencies that did not identify these IT contract obligations also did not follow OMB guidance. Specifically, 14 of the 22 agencies did not involve the acquisition office in their process to identify IT acquisitions for CIO review, as required by OMB. In addition, seven agencies did not establish guidance to aid officials in recognizing IT. We concluded that, until these agencies involve the acquisition office in their IT acquisition identification processes and establish supporting guidance, they cannot ensure that they will identify all such acquisitions. Without proper identification of IT acquisitions, these agencies and their CIOs cannot effectively provide oversight of the acquisitions. In addition to not identifying all IT contracts, 14 of the 22 selected agencies did not fully satisfy OMB’s requirement that the CIO review and approve IT acquisition plans or strategies. Further, only 11 of 96 randomly selected IT contracts at 10 of the 22 agencies were CIO-reviewed and approved as required by OMB’s guidance. The 85 contracts that were not reviewed had a total possible value of approximately $23.8 billion. Until agencies ensure that CIOs are able to review and approve all IT acquisitions, CIOs will continue to have limited visibility and input into their agencies’ planned IT expenditures and will not be able to effectively use the increased authority that FITARA’s contract approval provision is intended to provide. Further, agencies will likely miss an opportunity to strengthen their CIOs’ authority and the oversight of acquisitions. As a result, agencies may award IT contracts that are duplicative, wasteful, or poorly conceived. As a result of these findings, we made 39 recommendations in our January 2018 report. Among these, we recommended that agencies ensure that their acquisition offices are involved in identifying IT acquisitions and issuing related guidance, and that IT acquisitions are reviewed in accordance with OMB guidance. OMB and the majority of the agencies generally agreed with or did not comment on the recommendations. As of November 2019, 23 of the 39 recommendations had been implemented. We will continue to monitor the implementation of the remaining recommendations. Data center consolidation efforts are key to implementing FITARA. Specifically, OMB established the FDCCI in February 2010 to improve the efficiency, performance, and environmental footprint of federal data center activities. The enactment of FITARA in 2014 codified and expanded the initiative. In addition, OMB’s August 2016 memorandum that established the Data Center Optimization Initiative (DCOI) included guidance on how to implement the data center consolidation and optimization provisions of FITARA. Among other things, the guidance required agencies to consolidate inefficient infrastructure, optimize existing facilities, improve their security posture, and achieve cost savings. According to the 24 agencies covered by the initiative, data center consolidation and optimization efforts had resulted in approximately $4.7 billion in cost savings through August 2018. Even so, additional work remains to fully carry out the initiative. Specifically, in a series of reports that we issued from July 2011 through April 2019, we noted that, while data center consolidation could potentially save the federal government billions of dollars, weaknesses existed in several areas, including agencies’ data center consolidation plans, data center optimization, and OMB’s tracking and reporting on related cost savings. In April 2019, we reported that agencies continued to report mixed progress toward achieving OMB’s goals for closing data centers and realizing the associated savings by September 2018. Specifically, as of August 2018, over half of the agencies reported that they had met, or planned to meet, all of their OMB-assigned closure goals for tiered data centers by the deadline. Six agencies reported that they did not plan to meet their goals for tiered data centers. In addition, as of August 2018, 11 agencies reported that they had already met the goal for closing 60 percent of their non-tiered centers, three agencies reported that they planned to meet the goal by the end of fiscal year 2018, and nine agencies reported that they did not plan to meet the goal by the end of fiscal year 2018. In all, the 24 agencies reported a total of 6,250 data center closures as of August 2018, which represented about half of the total reported number of federal data centers. In addition, the agencies reported 1,009 planned closures by the end of fiscal year 2018, with an additional 191 closures planned through fiscal year 2023, for a total of 1,200 further closures. Further, in August 2018, 22 agencies reported that they had achieved $1.94 billion in cost savings for fiscal years 2016 through 2018, while two agencies reported that they had not achieved any savings. In addition to that amount, 21 agencies identified an additional $0.42 billion in planned savings through fiscal year 2018—for a total of $2.36 billion in planned cost savings from fiscal years 2016 through 2018. Nevertheless, this total is about $0.37 billion less than OMB’s goal of $2.7 billion for overall DCOI savings. From July 2011 through April 2019, we made a total of 196 recommendations to OMB and 24 agencies to improve the execution and oversight of the initiative. Most agencies and OMB agreed with our recommendations or had no comments. As of November 2019, 121 of these 196 recommendations had been implemented. We also have ongoing work to review and verify the quality and completeness of federal data center inventories and strategies for consolidation submitted by the agencies covered by the law. We expect to issue the report related to this work in early 2020. In our 2015 high-risk report’s discussion of IT acquisitions and operations, we identified the management of software licenses as a focus area, in part because of the potential for cost savings. Federal agencies engage in thousands of software licensing agreements annually. The objective of software license management is to manage, control, and protect an organization’s software assets. Effective management of these licenses can help avoid purchasing too many licenses, which can result in unused software, as well as too few licenses, which can result in noncompliance with license terms and cause the imposition of additional fees. As part of its PortfolioStat initiative, OMB has developed a policy that addresses software licenses. This policy requires agencies to conduct an annual, agency-wide IT portfolio review to, among other things, reduce commodity IT spending. Such areas of spending could include software licenses. In May 2014, we reported on federal agencies’ management of software licenses and determined that better management was needed to achieve significant savings government-wide. Of the 24 selected agencies we reviewed, only two had comprehensive policies that included the establishment of clear roles and central oversight authority for managing enterprise software license agreements, among other things. Of the remaining 22 agencies, 18 had policies that were not comprehensive, and four had not developed any policies. Further, we found that only two of the 24 selected agencies had established comprehensive software license inventories, a leading practice that would help them to adequately manage their software licenses. The inadequate implementation of this and other leading practices in software license management was partially due to weaknesses in agencies’ policies. As a result, we concluded that agencies’ oversight of software license spending was limited or lacking, thus, potentially leading to missed savings. However, the potential savings could be significant considering that, in fiscal year 2012, one major federal agency reported saving approximately $181 million by consolidating its enterprise license agreements, even when its oversight process was ad hoc. Accordingly, we recommended that OMB issue a directive to help guide agencies in managing software licenses. We also made 135 recommendations to the 24 agencies to improve their policies and practices for managing licenses. Among other things, we recommended that the agencies (1) regularly track and maintain a comprehensive inventory of software licenses and (2) analyze the inventory to identify opportunities to reduce costs and better inform investment decision making. Most agencies generally agreed with the recommendations or had no comments. As of November 2019, all but 19 of the 135 recommendations had been implemented. In particular, for our recommendations on maintaining and analyzing a comprehensive inventory of software licenses, agencies had fully implemented 42 out of 48 recommendations. Table 2 reflects the extent to which the 24 agencies implemented the recommendations in these two areas. Safeguarding federal computer systems has been a longstanding concern. This year marks the 22nd anniversary of GAO’s first designation of information security as a government-wide high-risk area in 1997. We expanded this high-risk area to include safeguarding the systems supporting our nation’s critical infrastructure in 2003, protecting the privacy of personally identifiable information in 2015, and establishing a comprehensive cybersecurity strategy and performing effective oversight in 2018. Most recently, we identified federal information security as a government-wide high-risk area in our March 2019 high-risk update. As we have previously noted, in order to strengthen the federal government’s cybersecurity posture, agencies should fully implement the information security programs required by FISMA. In this regard, FISMA provides a framework for ensuring the effectiveness of information security controls for federal information resources. The law requires each agency to develop, document, and implement an agency-wide information security program. Such a program should include risk assessments; the development and implementation of policies and procedures to cost- effectively reduce risks; plans for providing adequate information security for networks, facilities, and systems; security awareness and specialized training; the testing and evaluation of the effectiveness of controls; the planning, implementation, evaluation, and documentation of remedial actions to address information security deficiencies; procedures for detecting, reporting, and responding to security incidents; and plans and procedures to ensure continuity of operations. Since fiscal year 2010, we have made 3,323 recommendations to agencies aimed at addressing the four cybersecurity challenges. These recommendations have identified actions for agencies to take to strengthen technical security controls over their computer networks and systems. They also have included recommendations for agencies to fully implement aspects of their information security programs, as mandated by FISMA. Nevertheless, many agencies continue to be challenged in safeguarding their information systems and information, in part, because many of these recommendations have not been implemented. Of the 3,323 recommendations made since 2010, 2,511 (or 76 percent) had been implemented as of November 2019, leaving 812 recommendations (or 24 percent) not implemented. In order to determine the effectiveness of the agencies’ information security programs and practices, FISMA requires federal agencies’ inspectors general to conduct annual independent evaluations. The agencies are to report the results of these evaluations to OMB, and OMB is to summarize the results in annual reports to Congress. In these evaluations, the inspectors general are to frame the scope of their analyses, identify key findings, and detail recommendations to address the findings. The evaluations also are to capture maturity model ratings for their respective agencies. Toward this end, in fiscal year 2017, the inspector general community, in partnership with OMB and DHS, finalized a 3-year effort to create a maturity model for FISMA metrics. The maturity model aligns with the five function areas in the NIST Framework for Improving Critical Infrastructure Cybersecurity (Cybersecurity Framework): identify, protect, detect, respond, and recover. This alignment is intended to help promote consistent and comparable metrics and criteria and provide agencies with a meaningful independent assessment of their information security programs. The maturity model is designed to summarize the status of agencies’ information security programs on a five-level capability maturity scale. The five maturity levels are defined as follows: Level 1 (Ad hoc): Policies, procedures, and strategy are not formalized; activities are performed in an ad-hoc, reactive manner. Level 2 (Defined): Policies, procedures, and strategy are formalized and documented but not consistently implemented. Level 3 (Consistently Implemented): Policies, procedures, and strategy are consistently implemented, but quantitative and qualitative effectiveness measures are lacking. Level 4 (Managed and Measurable): Quantitative and qualitative measures on the effectiveness of policies, procedures, and strategy are collected across the organizations and used to assess them and make necessary changes. Level 5 (Optimized): Policies, procedures, and strategy are fully institutionalized, repeatable, self-generating, consistently implemented and regularly updated based on a changing threat and technology landscape and business/mission needs. According to this maturity model, Level 4 (managed and measurable) represents an effective level of security. Therefore, if an inspector general rates an agency’s information security program at Level 4 or Level 5, then that agency is considered to have an effective information security program. For fiscal year 2018, most of the 23 civilian CFO Act agencies’ inspectors general reported that their agencies were at Level 2 (defined) for the detect function; Level 3 (consistently implemented) for the identify, protect, and recover functions; and at Level 4 (managed and measurable) for the respond function. Table 3 shows the individual maturity ratings for each covered agency. In its efforts toward strengthening the federal government’s cybersecurity, OMB also requires agencies to submit related cybersecurity metrics as part of its Cross-Agency Priority goals. In particular, OMB developed a goal so that federal agencies will be able to build and maintain more modern, secure, and resilient IT. A key part of this goal is to reduce cybersecurity risks to the federal mission through three strategies: manage asset security, protect networks and data, and limit personnel access. The key targets supporting each of these strategies correspond to areas within the FISMA metrics. Table 4 outlines the strategies, their associated targets, and the 23 civilian CFO Act agencies’ progress in meeting those targets, as of June 2019. In conclusion, by addressing the high-risk areas on improving the management of IT acquisitions and operations and ensuring the cybersecurity of the nation, the government has the opportunity to both save billions of dollars and advance the efficiency and effectiveness of government services. Most agencies have taken steps to execute key IT management and cybersecurity requirements and initiatives, including implementing CIO responsibilities, requiring CIO reviews of IT acquisitions, realizing data center consolidation cost savings, managing software assets, and complying with FISMA requirements. The agencies have also continued to address the recommendations that we have made over the past several years. Nevertheless, further efforts by OMB and federal agencies to implement our previous recommendations would better position them to improve the management and security of federal IT. To help ensure that these efforts succeed, we will continue to monitor agencies’ efforts toward implementing the recommendations. Chairman Connolly, Ranking Member Meadows, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staff have any questions about this testimony, please contact Carol C. Harris, Director of Information Technology Acquisition Management Issues, at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Kevin Walsh (Assistant Director), Jessica Waselkow (Assistant Director), Chris Businsky, and Rebecca Eyler. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The federal government plans to spend over $90 billion in fiscal year 2019 on IT. Even so, IT investments have too often failed or contributed little to mission-related outcomes. Further, increasingly sophisticated threats and frequent cyber incidents underscore the need for effective information security. To focus attention on these concerns, GAO has included both the management of IT acquisitions and operations and cybersecurity on its high-risk list. For this statement, GAO summarized its key related reports and assessed agencies' progress in implementing the reports' recommendations. Specifically, GAO reviewed the implementation of recommendations on (1) CIO responsibilities, (2) IT acquisition review requirements, (3) data center consolidation, (4) the management of software licenses, and (5) cybersecurity. Federal agencies and the Office of Management and Budget (OMB) have taken steps to improve the management of information technology (IT) acquisitions and operations and ensure the nation's cybersecurity through a series of initiatives. As of November 2019, federal agencies had fully implemented 61 percent of the 1,320 IT management-related recommendations that GAO has made to them since fiscal year 2010. Likewise, agencies had implemented 76 percent of the 3,323 security-related recommendations that GAO has made since fiscal year 2010. Significant actions remain to be completed to build on this progress. Chief Information Officer (CIO) responsibilities . Laws such as the Federal Information Technology Acquisition Reform Act (FITARA) and related guidance assign 35 key responsibilities to agency CIOs to help address longstanding IT management challenges. In August 2018, GAO reported that none of the 24 selected agencies had established policies that fully addressed the role of their CIO. GAO recommended that OMB and the 24 agencies take actions to improve the effectiveness of CIOs' implementation of their responsibilities. Although most agencies agreed or did not comment, none of the 27 recommendations have yet been implemented. CIO IT acquisition review . According to FITARA, covered agencies' CIOs are required to review and approve IT contracts. Nevertheless, in January 2018, GAO reported that most of the CIOs at 22 covered agencies were not adequately involved in reviewing billions of dollars of IT acquisitions. Consequently, GAO made 39 recommendations to improve CIO oversight for these acquisitions. Since then, 23 of the recommendations have been implemented. Consolidating data centers . OMB launched an initiative in 2010 to reduce data centers. In August 2018, 22 agencies reported that they had achieved $1.94 billion in cost savings for fiscal years 2016 through 2018, while two agencies reported that they had not achieved any savings. GAO has made 196 recommendations to OMB and agencies to improve the reporting of related cost savings and to achieve optimization targets. As of November 2019, 121 of the recommendations have been implemented. Managing software licenses . Effective management of software licenses can help avoid purchasing too many licenses that result in unused software. In May 2014, GAO reported that better management of licenses was needed to achieve savings, and made 135 recommendations to improve such management. As of November 2019, all but 19 of the recommendations had been implemented. Ensuring the nation's cybersecurity . While the government has acted to protect federal information systems, GAO has consistently identified shortcomings in the federal government's approach to cybersecurity. The 3,323 recommendations that GAO made to agencies since 2010 have been aimed at addressing cybersecurity challenges. These recommendations have identified actions for agencies to take to fully implement aspects of their information security programs and strengthen technical security controls over their computer networks and systems. As of November 2019, 76 percent of the recommendations had been implemented. Since fiscal year 2010, GAO has made about 1,300 recommendations to OMB and agencies to address shortcomings in IT acquisitions and operations, as well as approximately 3,300 recommendations to agencies to improve the security of federal systems. These recommendations addressed, among other things, implementation of CIO responsibilities, oversight of the data center consolidation initiative, management of software licenses, and the efficacy of security programs. Implementation of these recommendations is essential to strengthening federal agencies' acquisitions, operations, and cybersecurity efforts.", "document_type": "gao"}
{"report": "In September 2018, we found the Coast Guard did not have a sound business case when it established the acquisition baselines for its polar icebreaker program in March 2018 due to risks in four main areas— design, technology, cost, and schedule. Our prior work has found that successful acquisition programs start with solid, executable business cases before setting program baselines and committing resources. A sound business case requires balance between the concept selected to satisfy operator requirements and the resources—design knowledge, technologies, funding, and time—needed to transform the concept into a product, which in this case is a ship with polar icebreaking capabilities. Without a sound business case, acquisition programs are at risk of breaching the cost, schedule, and performance baselines set when the program was initiated—in other words, experiencing cost growth, schedule delays, and reduced capabilities. At the heart of a business case is a knowledge-based approach. We have found that successful shipbuilding programs build on attaining critical levels of knowledge at key points in the shipbuilding process before significant investments are made (see figure 1). We provide additional information below on each of the four main risks that affect the soundness of the polar icebreaker program’s business case. The Coast Guard expressed a commitment to having a stable design for the polar icebreaker program prior to the start of lead ship construction, but it set the program’s baselines before conducting a preliminary design review—a systems engineering event that is intended to verify that the contractor’s design meets the requirement of the ship specifications and is producible. Shipbuilding best practices we identified in 2009 found that design stability on a ship is achieved upon completion of the basic and functional designs. The basic design includes fixing the ship steel structure; routing all major distributive systems, including electricity, water, and other utilities; and ensuring the ship will meet the performance specifications. The functional design includes further iteration of the basic design, such as providing information on the exact position of piping and other outfitting in each block, and completing a 3D product model. At this point of design stability, the shipbuilder has a clear understanding of the ship structure as well as how every system is set up and routed throughout the ship. Consistent with our best practices, prior to the start of construction on the lead ship, the Coast Guard plans to require the shipbuilder to complete basic and functional designs, develop a 3D model output, and provide at least 6 months of production information to support the start of construction. Although the Coast Guard plans to have a stable design prior to ship construction, it set the program’s acquisition program baselines prior to gaining knowledge on the feasibility of the selected shipbuilder’s design. Program baselines inform DHS’s and the Coast Guard’s decisions to commit resources. Our best practices for knowledge-based acquisitions state that before program baselines are set, programs should hold key systems engineering events, such as a preliminary design review, to help ensure that requirements are defined and feasible and that the proposed design can be met within cost, schedule, and other system constraints. The Coast Guard has yet to conduct a preliminary design review for the program because DHS’s current acquisition policy does not require programs to do so until after setting program baselines. However, in April 2017, we found that DHS’s sequencing of the preliminary design review is not consistent with our acquisition best practices, which state that programs should pursue a knowledge-based acquisition approach that ensures program needs are matched with available resources—such as technical and engineering knowledge, time, and funding—prior to setting baselines. As a result, we recommended that DHS update its acquisition policy to require key technical reviews, including the preliminary design review, to be conducted prior to approving programs’ baselines. DHS concurred with this recommendation and stated that it planned to initiate a study to assess how to better align its processes for technical reviews and acquisition decisions. Upon completion of the study, DHS plans to update its acquisition policies, as appropriate. As of June 2018, DHS indicated that it had completed its study and was in the process of updating its acquisition policies. GAO will review the policies once complete to determine if the updates meet the intent of this recommendation. By setting the polar icebreaker program’s baselines prior to gaining knowledge on the shipbuilder’s design, the Coast Guard has established cost, schedule, and performance baselines without a stable or mature design. Although completing the preliminary design review after setting program baselines is consistent with DHS policy, this puts the Coast Guard at risk of breaching its established baselines and having to revise them later in the acquisition process, after a contract has been signed and significant resources have been committed to the program. At that point, the program will be well underway and it will be too late for decision makers to make appropriate tradeoff decisions between requirements and resources without causing disruptions to the program. The Coast Guard intends to use what it refers to as “state-of-the-market” or “proven” technologies for the polar icebreaker program, but it has not yet conducted a technology readiness assessment to determine the maturity of key technologies prior to setting program baselines. This approach is inconsistent with our best practices for technology readiness. A technology readiness assessment is a systematic, evidence-based process that evaluates the maturity of critical technologies—hardware and software technologies critical to the fulfillment of the key objectives of an acquisition program. According to our best practices, a technology readiness assessment should be conducted prior to program initiation. At the time of our earlier review, Coast Guard officials told us the polar icebreaker program does not have any critical technologies and thus, does not need to conduct a technology readiness assessment. From design studies and industry engagement, Coast Guard officials determined that the key technologies required for the polar icebreakers, such as the integrated power plant and azimuthing propulsors, are available commercially and do not need to be developed. Figure 2 provides additional information on the risks for these key technologies, as well as design risks for an icebreaker’s hull form. Coast Guard officials stated that the integrated power plant is the standard power plant used on domestic and foreign icebreakers. Coast Guard officials told us that similarly, market survey data on azimuthing propulsors show that ice-qualified azimuthing propulsors in the power range required have been used on foreign icebreakers. However, according to our best practices, critical technologies are not just technologies that are new or novel. Technologies used on prior systems can also become critical if they are being used in a different form, fit, or function. Based on our analysis of available Coast Guard information, we believe the polar icebreaker program’s planned integrated power plant and azimuthing propulsors should be considered critical technologies given their criticality in meeting key performance parameters, how the technologies are being reapplied to a different operational environment from prior uses of the technologies, and the extent to which they pose major cost risks. By not conducting a technology readiness assessment and identifying, assessing, and maturing its critical technologies prior to setting the program’s program baselines, the Coast Guard is potentially underrepresenting technical risk and understating its cost, schedule, and performance risks. We found that the Navy’s lifecycle cost estimate used to inform the polar icebreaker program’s $9.827 billion cost baseline substantially adheres to most of our cost estimating best practices; however, the estimate is not fully reliable. The cost estimate is not fully reliable because it only partially met best practices for being credible. Highlights from our assessment of the polar icebreaker program’s lifecycle cost estimate are detailed below: Comprehensive: substantially met. The estimate includes government and contractor costs over the full lifecycle of all three ships and documents detailed ground rules and assumptions, such as the learning curve used to capture expected labor efficiencies for follow-on ships. However, the costs for disposal of the three ships were not at a level of detail to ensure that all costs were considered and not all assumptions, particularly regarding operating and support costs, were varied to reflect the impact on cost should these assumptions change. Well-documented: substantially met. The estimate’s documentation mostly captured the source data used as well as the primary methods, calculations, results, rationales, and assumptions used to generate each cost element. However, the documentation alone did not provide enough information for someone unfamiliar with the cost estimate to replicate what was done and arrive at the same results. Accurate: substantially met. The estimate was properly adjusted for inflation, and we did not find any mathematical errors in the estimate calculations we inspected. Officials stated that labor and material cost data from recent, analogous programs were used in the estimate. While the documentation does not discuss the reliability, age, or relevance of the cost data, Navy officials provided us with additional information regarding those data characteristics. Credible: partially met. The Navy only modeled cost variation in the detail design and construction portion of the program and excluded from its analyses any risk impacts related to the remainder of the acquisition, operating and support, and disposal phases, which altogether comprise about 75 percent of the lifecycle cost. Without performing a sensitivity analysis on the entire life cycle cost of the three ships, it is not possible for the Navy to identify key elements affecting the overall cost estimate. Further, without performing a risk and uncertainty analysis on the entire life cycle cost of the three ships, it is not possible for the Navy to determine a level of confidence associated with the overall cost estimate. By not quantifying important risks, the Navy may have underestimated the range of possible costs for about three-quarters of the entire program. The estimate provides an overly optimistic assessment of the program’s vulnerability to cost growth should risks be realized or current assumptions change. This, in turn, may underestimate the lifecycle cost of the program. The Coast Guard’s planned delivery dates of 2023, 2025, and 2026 for the three ships were not informed by a realistic assessment of shipbuilding activities, but rather were primarily driven by the potential gap in icebreaking capabilities once the Polar Star reaches the end of its service life (see figure 3). The Polar Star’s service life is estimated to end between fiscal years 2020 and 2023. This creates a potential heavy polar icebreaker capability gap of about 3 years, if the Polar Star’s service life were to end in 2020 and the lead polar icebreaker were to be delivered by the end of fiscal year 2023 as planned. If the lead ship is delivered later than planned in this scenario, the potential gap could be more than 3 years. The Coast Guard is planning to recapitalize the Polar Star’s key systems starting in 2020 to extend the service life of the ship until the planned delivery of the second polar icebreaker (see figure 4). Further, we compared the program’s planned construction schedule to the construction schedules of delivered lead ships for major Coast Guard and Navy shipbuilding programs active in the last 10 years as well as the Healy, the Coast Guard’s only medium polar icebreaker. We found that the polar icebreaker’s lead ship construction cycle time of 2.5 to 3 years is optimistic, as only 3 of the 10 ships in our analysis were constructed in 3 years or less. Further, as another point of comparison, the Healy was constructed in just under 4.5 years. An unrealistic schedule puts the Coast Guard at risk of not delivering the icebreakers when promised and the potential gap in icebreaking capabilities could widen. Just as importantly, our prior work on shipbuilding programs has shown that establishing optimistic program schedules based on insufficient knowledge can create pressure for programs to make sacrifices elsewhere, which can lead to work being performed concurrently, costly rework, and further delays. To address the risks we identified and establish a sound business case, we made a number of recommendations in our September 2018 report to DHS, Coast Guard, and the Navy, including: Conducting a technology readiness assessment in accordance with best practices, identifying critical technologies, and developing a plan to mature any technologies not designated to be mature before detail design of the lead ship begins; Updating the program’s cost estimate in accordance with best practices before the contract option for construction of the lead ship is awarded; Developing a program schedule in accordance with best practices to set realistic schedule goals for all three ships before the contract option for construction of the lead ship is awarded; and Updating the program’s acquisition program baselines prior to authorizing lead ship construction, after completion of the preliminary design review, and after it has gained the requisite knowledge on its technologies, cost, and schedule. DHS concurred with all of our recommendations and identified actions it planned to take to address them. For example, earlier this month, the Coast Guard indicated that it has identified a preliminary list of potential critical technologies and is in the process of developing a technology readiness assessment plan. The Coast Guard also plans to update the program’s cost estimate within 8 months of the contract award and update the program schedule within 3 months of the contract award. Of the $9.827 billion estimated for the lifecycle costs of the polar icebreaker program, about $3 billion is for acquisition costs. From 2013 through 2018, the polar icebreaker program has received $360 million in funding—$60 million in Coast Guard appropriations and $300 million in Navy appropriations. In addition, according to Coast Guard officials, in fiscal year 2017, Coast Guard reprogrammed $30 million in fiscal year 2016 appropriations for the polar icebreaker program from another program (see figure 5). According to Coast Guard and Navy officials, the Navy plans to use the $300 million in Navy appropriations in fiscal year 2019 to fund the advanced planning, design, engineering, and long lead time materials for the first polar icebreaker. As part of the polar icebreaker program’s acquisition strategy and reflected in the March 2018 request for proposals, the Navy plans to establish options for the subsequent detail design and construction of each of the three ships. The request for proposals specified that the options will be priced as fixed-price incentive type (see table 1). The Navy did not request any funding in fiscal year 2019 for the polar icebreaker program, while Coast Guard requested $30 million. Subsequently, after discretionary budget caps were relaxed by Congress, the administration’s fiscal year 2019 budget addendum requested an additional $720 million in fiscal year 2019 Coast Guard appropriations for the program. As the program prepares to award a contract in fiscal year 2019 worth billions of dollars if all the options are exercised, it is unclear to what extent the program will be funded using Coast Guard or Navy appropriations or how much total funding will be provided. In conclusion, as the Coast Guard embarks on the acquisition of its new polar icebreakers to address capability gaps in the Arctic and Antarctic regions, it faces a number of key acquisition and funding risks. DHS, the Coast Guard, and the Navy must gain key acquisition knowledge before committing significant resources to the program while Congress faces key funding and tradeoff considerations. To put the polar icebreaker program in a position to succeed, Congress and the agencies must remain committed to establishing and executing a sound business case for the program. Chairman Mast, Ranking Member Garamendi, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions. If you or your staff have any questions about this statement, please contact Marie A. Mak, (202) 512-4841 or makm@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony include Rick Cederholm, Assistant Director; Peter Anderson; Kurt Gurka; Claire Li; and Roxanna Sun. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "To maintain heavy polar icebreaking capability, the Coast Guard, in collaboration with the Navy, plans to acquire up to three new heavy polar icebreakers. The Navy plans to award a contract in 2019 for the polar icebreaker program. GAO has found that before committing resources, successful acquisition programs begin with sound business cases, which include plans for a stable design, mature technologies, a reliable cost estimate, and a realistic schedule. This statement addresses, among other things, the key acquisition risks facing the polar icebreaker program. This statement is primarily based on GAO's April 2018 and September 2018 reports examining the Coast Guard's polar icebreaker acquisition, and also draws from GAO's extensive body of published work examining the Coast Guard's and the Navy's shipbuilding efforts. In its prior work, GAO analyzed Coast Guard and Navy guidance, data, and documentation, and interviewed Coast Guard and Navy officials. The Coast Guard—a component of the Department of Homeland Security (DHS)—did not have a sound business case in March 2018, when it established the cost, schedule, and performance baselines for its heavy polar icebreaker acquisition program, because of risks in four key areas: Design. The Coast Guard set program baselines before conducting a preliminary design review, which puts the program at risk of having an unstable design, thereby increasing the program's cost and schedule risks. While setting baselines without a preliminary design review is consistent with DHS's current acquisition policy, it is inconsistent with acquisition best practices. Based on a prior GAO recommendation, DHS is currently evaluating its policy to better align technical reviews and acquisition decisions. Technology. The Coast Guard intends to use proven technologies for the program, but did not conduct a technology readiness assessment to determine the maturity of key technologies prior to setting baselines. Coast Guard officials indicated such an assessment was not necessary because the technologies the program plans to employ have been proven on other icebreaker ships. However, according to best practices, such technologies can still pose risks when applied to a different program or operational environment, as in this case. Without such an assessment, the program's technical risk is underrepresented. Cost. The lifecycle cost estimate that informed the program's $9.8 billion cost baseline was not fully reliable because it only partially met GAO's best practices for being credible. It did not quantify the range of possible costs over the entire life of the program. As a result, the cost estimate may underestimate the total funding needed for the program. However, the estimate substantially met GAO's best practices for being comprehensive, well-documented, and accurate. Schedule. The Coast Guard's planned delivery dates were not informed by a realistic assessment of shipbuilding activities, but rather driven by the potential gap in icebreaking capabilities once the Coast Guard's only operating heavy polar icebreaker—the Polar Star —reaches the end of its service life (see figure). GAO's analysis of selected lead ships for other shipbuilding programs found the icebreaker program's estimated construction time of 3 years is optimistic. As a result, the Coast Guard is at risk of not delivering the icebreakers when promised and the potential gap in icebreaking capabilities could widen. In September 2018, GAO recommended, among other things, that the polar icebreaker program update program baselines following a preliminary design review, conduct a technology readiness assessment, re-evaluate its cost estimate, and develop a schedule according to best practices. DHS concurred with all of GAO's recommendations and identified actions it plans to take to address them.", "document_type": "gao"}
{"report": "Since 9/11, the Coast Guard has made a series of organizational changes to realign its functions. First, from 2004 through 2006, under an effort known as “Sectorization,” the Coast Guard revised its field structure by consolidating field activities under individual commands, known as sectors. The Coast Guard’s 37 sectors report to nine districts, and each district reports to one of two area commands. District commanders are responsible for regional operations and execute operations and missions within their area of responsibility. Sector commanders are responsible for local operations within each district. Each of the Coast Guard area commands, districts, and sectors is responsible for managing its assets and accomplishing missions within its geographic area of responsibility and, for the purposes of this report, are referred to as field units. Figure 1 shows the Coast Guard’s field structure. In June 2006, the Coast Guard implemented another organizational change effort known as modernization. The goal of modernization was to realign its mission planning and mission support functions, among other things. According to Coast Guard documents, the effort was intended to address challenges the Coast Guard faced in aligning its operations with Coast Guard-wide priorities, and delivering mission support in a more cost effective manner. It was also intended to realign the Coast Guard’s operations and policies across multiple headquarters program offices. For example, the Coast Guard has six operational mission programs overseeing its statutory missions, and before modernization the leadership of each of them developed separate action plans and policies to execute their missions, while independently making resource decisions. Through modernization, the Coast Guard also sought to improve delivery of mission support services throughout the field, particularly with respect to maintenance of the Coast Guard’s assets, including its vessels, aircraft, and shore infrastructure. The Coast Guard uses three analytical tools to determine its workforce requirements: manpower requirements determinations, the Sector Staffing Model, and the Activity-Based Staffing Model for boat stations. Manpower requirements determinations, which begin with a manpower requirements analysis (MRA), are the Coast Guard’s preferred tool for determining the number of personnel and mix of skills its units require to meet mission needs, according to Coast Guard documents. The analysis identifies both the number of personnel required, and their necessary competencies, while also taking into account the effect of existing, new, or modified requirements on Coast Guard’s workforce. The Coast Guard considers the manpower requirements determination process to be its preferred method to determine workforce requirements for its assets and field units. The Coast Guard’s other two analytical tools—the Sector Staffing Model and Activity-Based Staffing Model—use historic levels of activity to determine workforce requirements. The Sector Staffing Model assesses workforce requirements for shore force units, while the Activity-Based Staffing Model assesses boat stations. For comparison, while activity models may identify the workforce needed based on the activities previously conducted by a unit, determinations identify the workforce needed to conduct the activities required by a unit to accomplish its planned mission, based on documented requirements. For this reason, the Coast Guard considers activity models to be less reliable for determining workforce needs than manpower requirements. Table 1 summarizes these three Coast Guard analytical tools for determining workforce requirements. In a 2018 report to Congress, the Coast Guard stated that under the modernization effort, it realigned its operations and mission support functions to address deficiencies that affected its ability to fulfill missions. Between 2009 and 2015, the effort focused on establishing headquarters organizations and business processes to manage operations and mission support. Central to the effort was the Coast Guard’s establishment of three new headquarters organizations. Deputy Commandant for Operations. Created to manage operational strategy and policy. The Deputy Commandant for Operations is responsible for the strategic management of the Coast Guard’s mission programs. This includes assessing and monitoring the performance of the Coast Guard’s missions and developing Coast Guard-wide strategy and operational policy. The Deputy Commandant for Operations also provides support for issues that affect multiple Coast Guard missions, such as managing intelligence activities, coordinating interaction with external stakeholders, and identifying new and emerging issues that threaten operations, such as cyberattacks. According to the Coast Guard’s 2018 report to Congress, consolidating these functions under a single organization has enhanced operational effectiveness and efficiency and aligned national priorities with Coast Guard-wide planning efforts. In 2019, the Coast Guard placed its reserve component under the Deputy Commandant for Operations to better incorporate the Coast Guard’s reserves into its plans for meeting mission needs. Deputy Commandant for Mission Support. Created to manage mission support delivery and business processes. The Deputy Commandant for Mission Support is responsible for managing mission support policy, strategy, planning, and resourcing to meet mission needs for human resources, engineering and logistics, information systems, and acquisitions. At the field level, through the Director of Operational Logistics, this organization assists with maintenance of assets and logistics planning through a network of bases. The Director of Operational Logistics manages Coast Guard bases which deliver operations level support to specific assets and oversees the functions of each Coast Guard base. In addition, the Deputy Commandant for Mission Support organization manages Coast Guard Logistic and Service Centers. Each logistic or service center exercises authority over its functions and the delivery of mission support to the Coast Guard’s fleet of aircraft and vessels. For example, the Aviation Logistics Center, located in Elizabeth City, North Carolina, is the lead entity for ensuring aviation asset services, such as maintenance and supply, for Coast Guard’s aircraft, while the Surface Forces Logistics Center, in Baltimore, Maryland, is responsible for ensuring these services for its vessels. Coast Guard officials told us that the modernized mission support structure enabled the Coast Guard to standardize delivery of products and service. For example, they told us that this structure helped them ensure that the materials and parts provided remained consistent across the Coast Guard’s field units. Force Readiness Command (FORCECOM). Created as an organization within the Deputy Commandant for Mission Support to prepare the Coast Guard workforce to properly perform and execute missions. FORCECOM is responsible for overseeing Coast Guard’s training plans and policies. This includes developing and delivering training courses, and conducting performance and compliance assessments of units, to determine whether each mission has the necessary equipment and personnel skills to ensure operational readiness. Figure 2 provides an overview of the Coast Guard’s modernized organizational structure and the responsibilities of the headquarters organizations known as the Deputy Commandant for Operations and Deputy Commandant for Mission Support. In 2018, the Coast Guard reported to Congress that while it completed its primary organizational changes, it continued to modernize its business processes. For example, it reported that it continued making improvements to its risk management process, organizational structure, and mission support functions, including human resources utilization and asset acquisition. In October 2019, Coast Guard officials told us that some of these adjustments continue in smaller, incremental efforts within the Deputy Commandant offices and individual Coast Guard programs. For example, Coast Guard officials from the Office of Mission Support Integration within the Deputy Commandant for Mission Support told us that efforts to modernize its mission support functions were ongoing. Officials told us that they were centralizing management of certain support delivery functions. Officials told us that centralization would help to ensure consistency in how functions are performed across the organization, as well as provide access to timely and complete information about the status of assets, personnel, and equipment. They told us that the Deputy Commandant for Mission Support had largely centralized such functions for one directorate—engineering and logistics—and expected to apply them for another directorate responsible for information systems in fiscal year 2020. According to officials, the Coast Guard has faced difficulty applying these same mission business practices to human resources since these practices focus on a specific capability and are geared more towards assets, such as vessels and aircraft, rather than personnel. Specifically, while information about the status of asset availability is generally static, there are more variables to determining the Coast Guard’s human resources needs. For example, in addition to identifying the size of the workforce necessary to perform missions, the Coast Guard must also consider how to retain personnel and develop a workforce that can adapt to changes such as addressing emerging threats like cyber-attacks. In 1995, the Coast Guard integrated the reserve and active duty workforce at the field level; however, the component did not have headquarters representation. In 2006, the Coast Guard issued a modernization goal to optimize the use of the reserve component by ensuring the workforce had the necessary training and support. In 2018, the Coast Guard chartered a project team to evaluate the state of the reserve component’s governance. The team found that the structure under the Deputy Commandant for Mission Support did not take into account the difference between the reserves workforce and Coast Guard programs. In 2019, the Coast Guard established a new reserve component organization under the Deputy Commandant for Operations. Changes stemming from modernization continued with the Coast Guard’s reorganization of its reserve component (see sidebar). In June 2019, the Coast Guard moved its reserve component from the Deputy Commandant for Mission Support to the Deputy Commandant for Operations. Coast Guard officials stated that the change was meant to address longstanding issues, such as not incorporating the reserve component into Coast Guard-wide policymaking. Coast Guard officials stated that when the reserve component was under the mission support organization, it was not strategically managed to align with Coast Guard- wide mission needs. For example, when reserve components were dispatched, there was no plan to support all of the operational needs of the mission, such as by providing additional equipment needed by the reserve workforce. Figure 3 provides a timeline of key actions the Coast Guard took from 2004 through 2019 to modernize its organizational structure. The Coast Guard has not consistently applied selected key reform practices to its modernization effort. Specifically, the Coast Guard did not apply or partially applied 5 of 7 selected key practices. We have previously reported that an agency must closely and carefully manage organizational reforms, since fully implementing major transformations can span several years. This is particularly important when the transformations include several major changes to the organization. The Coast Guard’s 2018 report to Congress on its modernization effort acknowledged that the risk of complications increases significantly with large-scale reorganization efforts, such as modernization, and noted that such changes require formal processes to look for complications as they arise and to fully assess their impact on the organization, including its workforce. To this end, we assessed the Coast Guard’s implementation of its modernization effort against selected key reform practices in three subcategories—Leadership focus and attention; Managing and monitoring; and Strategic workforce planning—and found the Coast Guard did not consistently apply these practices. Additionally, we assessed the extent to which the Coast Guard’s reorganization of its reserve component applied key reform practices under the Leadership focus and attention, Managing and monitoring and Strategic workforce planning subcategories. Figure 4 shows our assessment of the extent to which the Coast Guard’s actions to implement the modernization effort applied selected key reform practices. We found that the Coast Guard generally applied two key practices under this subcategory, including identifying a case for change and dedicating a team to lead the initial implementation effort, and it partially applied the key practice of holding leadership accountable for its success. Identify case for change. The Coast Guard generally applied this key practice because it identified a case for change to continue to drive the need for the modernization effort. Our prior work shows that key elements of successful initiatives are the demonstrated commitment of top leaders and accountability for change. Further, top leadership involvement and clear lines of accountability for making improvements are critical to overcoming organizations’ natural resistance to change. According to Coast Guard documents, in 2006, when the modernization effort started, Coast Guard leadership promoted the changes outlined in the Coast Guard’s 10 modernization initiatives through internal memos and action plans. Coast Guard documentation highlighted the benefits of the change and identified the next steps to be taken in order to complete the change. Additionally, commandants issued their strategic priorities highlighting plans for the modernization effort. More recently, the Coast Guard’s 2018 report to Congress reiterated the importance of the modernization effort, noting that the challenges that initially drove the need for organizational changes continue to challenge the Coast Guard. Dedicated implementation team. The Coast Guard generally applied this key practice because it established a team to implement its modernization changes. In 2007, Coast Guard created the Strategic Transformation Team to coordinate the early implementation of the modernization effort. According to Coast Guard officials from the Office of Resource, Organizational Analysis, and Workforce Management, as the effort moved from the planning stages to implementation, the team consolidated the goals in the Coast Guard’s 10 modernization initiatives into five main reorganization efforts. The team was responsible for ensuring that the implementation of these five efforts was consistent with the initial goals of modernization. This included facilitating the use of the Coast Guard’s existing organizational review and approval processes for organizational changes and leading the measurement processes for ensuring that the goals of modernization were met. Hold leaders accountable. The Coast Guard partially applied this key practice because it initially established an office to oversee its modernization but did not continue these efforts to ensure leadership accountability for modernization implementation. In 2009, the Coast Guard created a permanent oversight office under the Office of the Vice Commandant to transition the coordination responsibilities of the Strategic Transformation Team to monitor implementation of the modernization effort. The office was given an expanded role of managing change efforts across the Coast Guard, including overseeing the development of metrics related to organizational change efforts to ensure that these changes achieved goals. However, in 2015 the Coast Guard disestablished this oversight office and did not specify any office responsible for ensuring organizational change efforts met intended goals. According to Coast Guard officials from the Office of Resource, Organizational Analysis, and Workforce Management, the Coast Guard redistributed some of the oversight office’s responsibilities among other offices within the established headquarters organizations. The officials told us they did so since they determined the initial goals of modernization—to create the new headquarters organizations—had been met and oversight was no longer needed. These officials stated that the individual headquarters organizations could manage any necessary planning moving forward for their specific organization. As such, the Coast Guard’s shifting leadership priorities affected what parts of the modernization effort were implemented and, in some cases, resulted in years spent working towards a change that was later terminated. For example, in 2012, the Commandant stated that the original modernization initiative to establish a single operations command to manage field operations was not near completion, taking up institutional energy, and impacting operations. As a result, he decided to discontinue the effort and retain the two area field command structure. However, according to Coast Guard officials from the Office of Resource, Organizational Analysis, and Workforce Management, planning for the effort was close to completion, and ending it led to the reassignment of staff. During this time, the Coast Guard also reduced FORCECOM’s role from managing and measuring the overall readiness capabilities of the service to focusing on workforce training, and moved the organization under the Deputy Commandant for Mission Support. At that point, the Coast Guard had already prepared and issued a business plan for FORCECOM outlining the initial primary mission, goals and metrics for evaluating effectiveness. We also assessed the Coast Guard’s application of key reform practices against its reorganization of the reserve component and found, similar to our determinations of the modernization effort, it partially applied key practices under Leadership focus and attention. For example, while the Coast Guard identified key leadership and stakeholders currently responsible for implementing the effort, it could not demonstrate that there is a process to ensure leaders are held accountable for this implementation. The Coast Guard did not apply the two key practices of tracking implementation progress or collecting data to measure progress of the effort, and partially applied the other key practice of measuring employee satisfaction with the modernization effort. We have previously found that organizational transformations must be carefully and closely managed in order to monitor progress towards achieving intended goals, since fully implementing major transformations can span several years. This is particularly important for the modernization effort which the Coast Guard reported in 2018 had fundamentally altered how it conducts business across the organization, for every mission and at every level. Managing and monitoring organizational reforms includes applying key practices such as tracking and measuring progress and developing mechanisms to seek and monitor employee satisfaction with changes resulting from reforms. Track implementation progress. The Coast Guard did not apply this key practice because it did not track its progress in implementing the modernization effort on an ongoing basis. Officials told us that during the early stages of modernization, the Coast Guard developed implementation plans and engaged in a significant planning effort to finalize the organizational realignment. These plans provided a method to track the Coast Guard’s progress as they implemented each phase of modernization; however, as the effort matured, the Coast Guard determined that the effort did not require the same amount of planning as initial implementation. In 2009, during the early stages of modernization, the Coast Guard reported that it had efforts planned or underway to monitor the implementation progress of the modernization effort, including developing implementation plans, goals, and performance metrics. As the modernization effort matured and the Deputy Commandant for Mission Support and Deputy Commandant for Operations were created, Coast Guard officials determined that they did not need the same amount of planning, and the Coast Guard stopped updating its implementation plans. Additionally, for the reorganization of the reserve component, the Coast Guard has minimally applied practices under the Managing and monitoring category. In particular, the Coast Guard did not track implementation progress of the reorganization. For example, the Coast Guard established the new reserve component without finalized plans or milestones and metrics against which it could track implementation progress. Collect data to measure progress. The Coast Guard did not apply this key practice because it did not collect data to measure the extent to which the modernization effort achieved its goals. In 2009, the Coast Guard reported that it had plans underway to identify existing metrics and gather data that would enable evaluation of the performance and effectiveness of its modernized processes and facilitate continued improvements. This was to include indicators that could be applied across the modernization efforts’ multiple goals and priorities such as quality, timeliness, cost, and outcomes. At the time, the Coast Guard reported that this would take approximately 6 months to 1 year to complete. However, according to officials from the Office of Resource, Organizational Analysis, and Workforce Management, its plans were discontinued due to the disestablishment of the oversight office and changing leadership priorities. Further, they stated that the Coast Guard no longer felt the need to monitor the effort since it determined the initial goals had been achieved with the establishment of the new headquarters organizations. In 2018, the Coast Guard reported to Congress that changes to mission support systems and business processes were significant changes and demonstrated the success of the modernization effort by developing a more effective and efficient organization. However, while officials from multiple offices told us that these changes resulted in better data and greater efficiency, the Coast Guard could not identify metrics or a data collection system that could demonstrate that the Coast Guard’s implementation of the modernization effort had improved effectiveness or efficiency. Moreover, in our review of the Coast Guard’s organizational change process, we found no metrics, time frames, or milestones to track whether, and to what extent, its organizational changes were achieving the goals of the effort. Similarly, for the reorganization of the reserve component, the Coast Guard did not collect data to measure progress. For example, the Coast Guard established no milestones or metrics against which to measure the reserve components’ progress in achieving its intended goal of improved mission performance. Measure employee satisfaction. The Coast Guard partially applied this key practice because it sought employee feedback during the early stages of the modernization effort, but did not continue to measure employee satisfaction with the effort. During the initial implementation of modernization, the Coast Guard used a combination of informal and formal mechanisms to seek employee satisfaction. For example, according to a 2009 National Academy of Public Administration report, the Commandant reached out to personnel through informal means, such as social media, to communicate and obtain real time feedback from staff affected by the organizational changes. Formally, the Coast Guard obtained anecdotal information through surveys of staff through the Organizational Assessment Survey and the Office of Personnel Management’s Federal Employee Viewpoint Survey; however, these methods do not include specific questions related to the impact of organizational change efforts. Specifically for modernization, beyond efforts during the early stages of modernization, there has been no sustained Coast Guard-wide effort to monitor the impact of the change on employees. According to a senior Coast Guard official from the Office of Resource, Organizational Analysis, and Workforce Management, the Coast Guard is not required to conduct such assessments as changes are implemented. Specifically, the document governing the Coast Guard’s organizational change process does not specify measuring employee satisfaction as part of the organizational change request process. Additionally, though the Coast Guard currently has formal mechanisms in place that would enable it to seek employee satisfaction, our review of recent surveys found that these instruments do not include questions specific to the impact of organizational change efforts; nor do they capture employee perspective in a timely manner. We found that the Coast Guard partially applied the key practice of assessing effects of modernization on its workforce by engaging in some activities that assess its impact on its current and future workforce and planning to determine whether needed resources and capacity were in place. We have previously reported that people are at the heart of any serious reform effort because people define the organization’s culture, drive its performance, and embody its knowledge base. This is echoed in the Coast Guard’s large-scale enterprise-wide change management guidance, which stresses the need for a formal, structured approach to manage the people side of change to increase likelihood of success. One of the goals of the modernization effort was to create a Coast Guard- wide human resources strategy to better support mission execution. The Commandant reiterated this commitment in September 2018 testimony to Congress by stating that the Coast Guard’s strategic plan would incorporate its 2016 Human Capital Strategy, a 10-year plan to ensure that the Coast Guard develops the workforce necessary to meet mission demands. In addition, the Coast Guard has taken steps to build a Force Planning Construct model to inform leadership on the forces and capabilities needed to execute its steady state and contingency operations. In its April 2018 Manpower Requirements Plan to Congress, the Coast Guard stated that it envisioned using the model to assess future workforce needs. According to developers of the model, the foundation of the tool was the completion of manpower requirements determinations for all 158 Coast Guard unit types. However, the Coast Guard has completed such determinations for a small fraction of its workforce, as we discuss later in this report. Finally, for the reorganization of the reserve component, we found that the Coast Guard had minimally applied the key practice under Strategic workforce planning. In particular, officials from the new reserve component told us that even though the reserve force is not covered by existing workforce planning tools, the Coast Guard continued to proceed with reorganizing the reserve force structure. For each of the key reform practices that were not fully implemented, we found that the Coast Guard’s organizational change request process and associated guidance documents did not require such practices to be followed, nor did they require tracking implementation of changes, collecting data to measure progress, or assessing employee satisfaction. By not fully implementing each of these key practices, the Coast Guard may miss opportunities to demonstrate that its investment in the modernization effort meets its ultimate goals to enhance efficiency and effectiveness and to improve the overall performance of the Coast Guard. Systematically tracking progress of organizational change efforts and measuring their effects, including employee satisfaction, would better position the Coast Guard to identify challenges, if any, to meeting the goals of the organizational change in a timely manner. Further, the Coast Guard noted that metrics used to show the effect on its efficiency, mission effectiveness, and operations may be used to measure and influence future modernization efforts. The Coast Guard’s manpower requirements determination process is its preferred method for determining workforce needs because it identifies the workforce needed to conduct required mission activities; however, since it began implementing the process in 2003, the Coast Guard has completed it for only 6 percent of its workforce. Further, for those positions with which the Coast Guard has used the manpower requirements determination process, it has not consistently done so in accordance with Coast Guard guidance. For example, while required by Coast Guard guidance, the Coast Guard has not tracked the number of MRAs and manpower requirements determinations completed. In its April 2018 Manpower Requirements Plan to Congress, the Coast Guard set a goal for using the manpower requirements determination process to identify staffing needs for all positions in all units, but does not have information on the resources it would need to do so. The Coast Guard has completed workforce assessments for a small portion of its 58,000 personnel across its 158 unit types. From calendar years 2014 through 2019, the Coast Guard used its three analytical tools—manpower requirements determinations, the Sector Staffing Model, and the Activity-Based Staffing Model—to complete workforce assessments for approximately 21 percent of its 58,000 position workforce. According to Coast Guard guidance, manpower requirements determinations are to be updated every 5 years. However, the Coast Guard completed the manpower requirements determination process, its primary workforce analysis tool, for only about 2 percent of positions during this 5-year span. In 2019, the Coast Guard used the Sector Staffing Model to assess workforce requirements for about 9 percent of positions. Finally, in 2019 the Coast Guard used the Activity-Based Staffing Model for boat stations to assess workforce requirements for about 9 percent of positions, according to officials. According to its 2016 Human Capital Strategy, the manpower requirements determination process is the Coast Guard’s primary tool for defining the human capital its units require to meet mission needs. To this end, the Coast Guard’s goal is to use this process to establish manpower requirements for all positions in all units. Coast Guard guidance for implementing the manpower requirements determination process includes three key steps as noted in the service’s 2015 Staffing Logic and Manpower Requirements Manual. MRA. The manpower requirements determination process begins with programs or Coast Guard leadership, such as the Commandant or Vice Commandant, requesting an MRA, which is a comprehensive review of workforce needs as determined from a wide range of factors. These factors include regulations, training, and competencies needed to effectively perform each mission. The MRA assesses the information necessary to adjust personnel, resources, mission, or risk, depending on availability of resources. Officials from the manpower requirements determination program, contractors, or in some cases, other Coast Guard programs, may conduct MRAs. After Action Report. MRA requesters are to submit an after action report within 6 months after the MRA is completed. The after action report is to outline actions to be taken based on an MRA. These actions could include adding resources, adjusting requirements, or assuming additional risk. Manpower Requirements Determination. The process is to conclude with a manpower requirements determination. The determination identifies the number and type of positions a unit type requires to meet mission-based capability requirements. In developing the determination, stakeholders are to review MRA results and develop the determination, while documenting any changes from the initial MRA. These stakeholders typically include representatives from the program assessed in the MRA and experts from around the Coast Guard in areas such as personnel assignments, workforce forecasting, training availability and capacity, and resource oversight, among others. The manpower requirements determination program then submits the determination to be signed by the Assistant Commandant for Human Resources. This signed memorandum, known as the determination, formalizes the final manpower requirement. Figure 5 summarizes the Coast Guard’s manpower requirements determination process, according to Coast Guard guidance. We found that the Coast Guard has not ensured that all three key steps of the manpower requirements determination process are completed since it began implementing it in 2003. Since 2003, the Coast Guard conducted MRAs for 28 percent of its workforce. However, the Coast Guard completed manpower requirements determinations for only 6 percent of its workforce. Moreover, we found that this trend continued with MRAs that the Coast Guard completed within the past 5 years. For example, according to our analysis of Coast Guard documentation, from calendar years 2014 through 2019, the Coast Guard conducted MRAs for 13 percent of its workforce, but completed determinations for 2 percent. Further, Coast Guard officials reported they did not have documentation of having conducted after action reports for any MRAs. Figure 6 shows the share of the Coast Guard’s workforce that is supported by the manpower requirements determination process. The top row shows the share of workforce supported by this process since its inception in 2003. The bottom row shows the workforce supported by up to date MRAs and determinations—completed between 2014 and 2019— according to guidance. The Coast Guard’s 2018 Manpower Requirements Plan to Congress states that the Coast Guard’s goal is to have updated manpower requirements determinations for all authorized positions in all units. When it reaches that goal, the manpower requirements determination process will allow the Coast Guard to know which units are the most understaffed, and to make service-wide decisions based on where the most urgent needs are. Only when determinations have been completed for its entire workforce can Coast Guard leadership allocate personnel in the most effective and efficient manner. Notably, Coast Guard documents emphasize the importance of an enterprise-wide approach to track and manage resources because it enables leadership to compare needs and make informed trade-offs across programs. In 2019, officials in the manpower requirements determination program told us that MRAs were to be updated every 5 years. Officials stated that this is a best practice that aligns with the Department of Homeland Security’s workforce strategy. Pacific Area Command officials we spoke with also told us that they view the guidance as requiring that MRAs should not be older than 5 years. Additionally, the Coast Guard’s 2015 Staffing Logic and Manpower Requirements Manual states that the Manpower Requirements Determination Program Division Chief is responsible for ensuring that each unit type has undergone an MRA within the past 5 years. Nevertheless, in November 2019, Coast Guard officials in the manpower requirements determination program told us that they view it as a goal to update MRAs every 5 years, not a requirement. We found that the Coast Guard does not have current guidance explaining the process steps for Coast Guard officials to follow to systematically execute the manpower requirements determination process. Coast Guard officials told us they were using a combination of two documents to guide its manpower requirements determination process, and neither document was both current and comprehensive in terms of detailing the steps to follow. For example, the 2015 Staffing Logic and Manpower Requirements Manual contains individual process step requirements, but has been rescinded. In contrast, the 2018 Manpower Requirements Manual provides current policy, but does not include guidance on process steps that program officials are to follow. In its 2018 manual, the Coast Guard rescinded the 2015 manual without replacing or affirming its process steps. Officials stated that analysts in the manpower requirements determination program use the rescinded 2015 guidance in executing the process because they have no other guidance to follow. Officials in the manpower requirements determination program provided several reasons for why the program has not consistently ensured that all steps are completed. First, officials told us that completing a determination for each MRA had not always been a priority for the Coast Guard. Officials said that in some cases manpower requirements determinations were not completed due to disagreement among stakeholders about how to apply the results of the MRA. Officials said, for example, that while an MRA may find that a program is significantly understaffed, some stakeholders may argue against including the full scale of the shortfall in the determination due to limited resources and competing needs. Second, officials stated that some determinations were not completed because some programs requesting MRAs were not interested in obtaining the final determination upon receiving the MRA. Specifically, they explained that sometimes the program that requested to initiate the manpower requirements determination process is most interested in the staffing data contained in the MRA, rather than the final determination, which formalizes the trade-offs and results proposed in the MRA. Officials in the manpower requirements determination program told us that both the 2015 and 2018 manpower requirements determination guidance did not identify circumstances when a manpower requirements determination was not required to be completed for an MRA. Further, program officials told us that they were not aware that the process guidance they reported using required after-action reports. Coast Guard officials also stated that having the process guidance in a rescinded document had made their ability to implement and oversee the process a challenge due to the possibility of officials applying the guidance inconsistently. They further said they recognized the manpower requirements determination process was not clear and needed to be revised, and that doing so may help ensure officials consistently implement the process. In June 2019, officials said they planned to issue updated guidance, but had not established a timeframe for doing so. By issuing updated guidance for conducting manpower requirements determinations that outlines required process steps, and any circumstances in which the process steps do not need to be performed, the Coast Guard can better ensure that those responsible for implementing the process do so consistently. In addition to requiring MRAs to be conducted every 5 years, the rescinded 2015 Coast Guard guidance, which officials reported using to execute the manpower requirements determination process, states that the manpower requirements determination program is to maintain and update a master list of MRAs conducted to enable the program to track and organize its workload. However, the Coast Guard has not tracked the extent to which it has assessed Coast Guard unit types through the manpower requirements determination process, as required in the 2015 process guidance, which officials report is still in use. For example, in March 2019 Coast Guard officials stated that they did not maintain a list of MRAs or manpower requirements determinations completed since the program began in 2003, and they were not aware that maintaining a list was a requirement. Officials prepared a list to respond to our request, and in April 2019, provided us with a list of MRAs and determinations the Coast Guard had completed since 2003. However, we found that the list was not accurate. The Coast Guard’s list underrepresented the number of MRAs completed by almost half. Specifically, it showed the Coast Guard had completed MRAs for 34 unit types since 2003, whereas our review of Coast Guard documents found that the Coast Guard had completed MRAs for 63 unit types during this span. We also found that the Coast Guard had not accurately reported to Congress about its progress in assessing workforce requirements. While the list the Coast Guard compiled for us underrepresented the number of MRAs completed, the information it provided to Congress in its April 2018 report overrepresented the extent to which it has assessed its workforce needs. Specifically, in April 2018 the Coast Guard reported to Congress that it had recently analyzed workforce needs for 54 percent of its workforce using the manpower requirements determination process and its activity models. However, more than half of the MRAs it had completed had not been updated in the past 5 years, as Coast Guard guidance requires. We found that less than half of the Coast Guard’s reported figure—21 percent of its workforce—is supported by a workforce analysis that has been updated in the last 5 years. The Coast Guard’s manpower requirements plan does not have time frames or milestones outlining how it plans to reach its manpower requirements determination goal of completing MRAs and determinations for its entire workforce. Coast Guard officials stated that their April 2018 Manpower Requirements Plan to Congress lays out their goal with respect to conducting manpower requirements determinations. However, this plan does not include time frames or milestones for completing determinations for all unit types, nor does it signal that the Coast Guard will track MRAs and determinations it has completed. Coast Guard officials stated that they were using a multi-year program to prioritize manpower studies and complete them as resources allowed. When asked for further information about this plan, officials stated that there was no specific document outlining the plan; rather, the intent of the 2018 Manpower Requirements Plan was to indicate their manpower analysis goal involves a multi-year journey. By updating its manpower requirements plan to include time frames and milestones for completing MRAs and determinations for all positions in all units, the Coast Guard can track progress toward its goal and make necessary adjustments in its planning, as needed. The Coast Guard has reported on the importance of tracking and completing manpower requirements determinations to justify its resource allocation decisions. For example, its 2018 Manpower Requirements Manual states that methods to determine workforce requirements have historically varied from program to program. This variability prevented the Coast Guard from compiling reliable workforce data and comparing workforce needs across the Coast Guard. According to the 2018 manual, manpower requirements determinations enable key decision-makers to effectively manage workforce needs because they provide the data needed to objectively predict future manpower requirements and compare staffing needs across the entire workforce. By tracking and documenting the extent to which it has completed MRAs and determinations for its workforce, the Coast Guard will be better positioned to know which unit types have a defensible basis for the number and type of personnel needed to meet mission demands and to prioritize which MRAs to conduct. The Coast Guard has not determined the resources—both staff and funding—it needs to meet its goal for its manpower requirements determination program to complete determinations for all units. Program officials told us that they have used the manpower requirements determination process for a limited share of its workforce because of resource limitations. Coast Guard documents show that it has been almost 10 years since the Coast Guard last performed an MRA for the manpower requirements determination program to determine its own workforce needs. The 2010 analysis found that the program would require at least 30 full-time equivalent positions to accomplish the Coast Guard’s goal of completing about 25 MRAs each year, which would enable it to assess the Coast Guard’s 158 unit types roughly every 5 years. As of January 2020, the program had six analysts dedicated to conducting manpower analyses and, according to officials, may only be able to produce one MRA each year. Program officials estimated that the cost of conducting an MRA may vary widely, from $170,000 to more than $5 million for more complex unit types. Nevertheless, program officials told us they generally did not track information on the costs of conducting MRAs. According to officials, the manpower requirements determination program cannot track all such costs because cost data is spread across different program offices. For example, officials stated that for contracted MRAs, contracting fees are easier to identify, but the manpower requirements determination program does not have access to other major costs, such as travel by officials conducting the analysis. While the manpower requirements determination program oversees the MRA process, and is tasked with ensuring manpower requirements determinations are completed for every unit in the Coast Guard, officials said that generally the program that is the subject of the MRA provides funding for the study, and only that program maintains access to travel costs associated with the MRA. They said the manpower requirements determination program does not request cost information from the programs requesting MRAs. Additionally, the manpower requirements determination program does not collect cost information from programs that conduct their own MRAs. The Coast Guard has increasingly used contractors to complete MRAs. While the Coast Guard has not tracked the costs of conducting MRAs, Coast Guard analysis has shown that having MRAs completed by contractors is more costly than completing them in-house. Program officials said they have increasingly used contractors because of staffing limitations. For example, from calendar years 2010 through 2019, contractors completed nearly half of the Coast Guard’s 54 MRAs. Figure 7 shows the MRAs and manpower requirements determinations completed by the Coast Guard and contractors from 2003 through 2019. Coast Guard guidance states that in a resource constrained environment, leaders need to make risk-based decisions to prioritize tasks and optimally allocate resources to execute its missions. In addition, our work in the area of strategic human capital management has shown that reassessing resource requirements helps organizations to achieve their missions and match resources to their needs. Developing information on the resources needed for staffing and funding the manpower requirements determination program to achieve its manpower goal would better position the Coast Guard to make informed trade-off decisions and allocate its limited resources to those units most in need of manpower requirements determinations. The Coast Guard’s roles and responsibilities have grown over the past two decades following the terrorist attacks of 9/11. Among other things, increased national security roles, first response duties during natural disasters, and compliance duties for ensuring the safety of increased commercial maritime activity have underscored the importance of the Coast Guard’s multiple missions. Organizational changes it made through the modernization effort were intended to realign operations and support functions. To that end, the creation of headquarters organizations achieved modernization’s initial goals. However, the Coast Guard continues to change as a result of modernization, and it has placed less effort on ensuring achievement of the longer-term goals of creating a more efficient and effective organization. Establishing a process for tracking and measuring the effectiveness of the organizational changes brought on by modernization, including measuring employee satisfaction, would better position Coast Guard to understand whether its goals have been achieved. The Coast Guard reported to Congress in April 2018 that it faced challenges in meeting its daily mission demands because it was operating below the workforce necessary to meet its mission demands. However, the service does not have a complete picture of the workforce necessary to meet its mission demands or whether its existing mix of personnel is efficiently and effectively allocated across units. The Coast Guard considers its manpower requirements determination process instrumental in determining the workforce needed to perform its duties, and the foundation of models the Coast Guard uses to determine workforce size in times of contingency or heightened security. Updated guidance for its staff tasked with conducting such assessments would enable the Coast Guard to better ensure that the process is fully implemented. Further, as of January 2020, the Coast Guard had updated analyses for a small fraction of its workforce, and had not updated its Manpower Requirements Plan with time frames and milestones for achieving its goal of assessing its entire workforce. Additionally, it does not have information on the extent to which analyses have been completed over the years or the resources it needs to complete assessments for its entire workforce. By tracking and updating the completion of MRAs and determinations, updating its plan to complete manpower requirements determinations, and obtaining information on the resources needed to implement such a plan, the Coast Guard will better ensure that it has the right number of people with the right set of skills to meet its mission demands. In this way, the Coast Guard will be better positioned to inform Congress of its workforce and associated resource needs. We are making the following six recommendations to the Coast Guard: The Commandant of the Coast Guard should establish a systematic mechanism to track implementation and measure the Coast Guard’s progress in achieving organizational change goals. (Recommendation 1) The Commandant of the Coast Guard should establish a mechanism to periodically seek and monitor employee satisfaction with organizational change efforts. (Recommendation 2) The Commandant of the Coast Guard should update its Manpower Requirements Manual with guidance for how to execute its manpower requirements determination process, and take steps to ensure the process is implemented. (Recommendation 3) The Commandant of the Coast Guard should track and document the extent to which it has completed manpower requirements analyses and determinations for each unit type. (Recommendation 4) The Commandant of the Coast Guard should update its April 2018 Manpower Requirements Plan to include time frames and milestones for completing manpower requirements analyses and determinations for all positions and units. (Recommendation 5) The Commandant of the Coast Guard should determine the resources its manpower requirements determination program needs, both staff and funding, to achieve its goal of completing manpower requirements determinations for all positions and units. (Recommendation 6) We provided a draft of this report to DHS for review and comment. DHS provided comments, reproduced in appendix V. DHS concurred with our six recommendations and described actions planned to address them. DHS also provided technical comments, which we incorporated into the report, as appropriate. With regard to our first recommendation, DHS stated that the Coast Guard’s Office of Resources, Organizational Analysis, and Workforce Management will update the Coast Guard Organizational Manual to establish policy requiring that requests to change organizational structure include a plan, and establish a mechanism to track implementation and measure progress in achieving organizational change goals. The Coast Guard estimated completing the effort by December 31, 2020. With regard to our second recommendation, DHS stated that Coast Guard leadership agrees that mechanisms to periodically seek and monitor employee satisfaction with organizational change efforts are valuable. DHS stated that the Coast Guard already conducts periodic surveys and each of these instruments provide opportunities for the workforce to provide feedback, including on organizational issues, and that it seems preferable for survey owners to add questions to existing surveys, as opposed to implementing new survey instruments. DHS requested GAO consider the recommendation as implemented because such feedback mechanisms were already in place, and therefore establishing new mechanisms was unnecessary. As we note in our report, it is important that the Coast Guard identify challenges, if any, to meeting the goals of organizational change in a timely manner. We found the Coast Guard’s current surveys do not capture employee perspectives as organizational changes are implemented. In determining whether to close this recommendation, we will review Coast Guard documentation demonstrating that the Coast Guard has modified its existing surveys with added questions that monitor employee satisfaction with organizational changes, and that it has plans for implementing the surveys in a timely manner. With regard to our third recommendation, DHS stated that the Coast Guard’s Office of Human Resources Strategy and Capability is developing a Tactics, Techniques and Procedures document to provide guidance for executing the manpower requirements determination process. The document will provide additional guidance on the overall MRD process, including explicit directions for the collection and analysis of manpower data, and will establish Coast Guard enterprise standards for key factors and allowances used when conducting manpower analysis. The Coast Guard estimated completing the effort by September 30, 2020. With regard to our fourth recommendation, DHS stated that in December 2019 the Coast Guard’s Office of Human Resources Strategy and Capability initiated the process to document and track manpower requirements in the Coast Guard’s system of record. The Coast Guard estimated completing the effort by December 31, 2020. With regard to our fifth recommendation, DHS stated that the Coast Guard’s Assistant Commandant for Human Resources Directorate would update its Manpower Requirements Plan during the next periodic report submitted to Congress, due in fiscal year 2022. The Coast Guard estimated completing the effort by March 31, 2022. With regard to our sixth recommendation, DHS stated that the Coast Guard’s Office of Human Resources Strategy and Capability will review its September 2010 MRA, revalidate the inputs, and update the findings of the MRA to reflect the current needs of the manpower requirements determination program. The Coast Guard estimated completing the effort by September 30, 2020. We are sending copies of this report to the appropriate congressional requesters, the Secretary of the Department of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or AndersonN@gao.gov. GAO staff who made key contributions to this report are listed in appendix VI. This appendix provides additional information on our objectives, scope and methodology. This report examines (1) how the Coast Guard modernized its organization and the extent to which it has applied key reform practices to its organizational change efforts and (2) the extent to which the Coast Guard has assessed its workforce needs. To address our first objective we analyzed Coast Guard documents related to the modernization effort. The documents included policies and guidance regarding how the effort was to be implemented, as well as descriptions of the status of these efforts. To evaluate the extent to which the Coast Guard applied key reform practices and considerations for evaluating organizational change efforts we assessed Coast Guard policies and procedures related to Coast Guard operations against the key practices we outlined in our June 2018 report on government reorganization. We collected and analyzed documentation related to Coast Guard’s actions taken to implement organizational change efforts such as the modernization effort and the integration of the Coast Guard’s reserve component into the headquarters governance structure. We assessed these reports, data and documents against selected criteria for key practices and considerations for agency reorganization identified in our June 2018 report. We selected relevant key practices by examining each of the potential four categories and 12 subcategories identified in our June 2018 report to determine the extent to which the practices under each applied to the Coast Guard’s modernization and reserve component integration efforts. The four categories are Goals and outcomes,” “Process for developing reforms,” “Implementing the reforms,” and “Strategically managing the federal workforce.” We deemed two subcategories under the category of “Implementing the reform”, “Leadership focus and attention” and” Managing and monitoring” and one subcategory “Strategic workforce planning” under the “Strategic planning for the federal workforce” category as relevant criteria for assessing the Coast Guard’s modernization efforts. We deemed the remaining nine subcategories not relevant to the Coast Guard’s modernization efforts since modernization was implemented in 2006 and retrospective analysis of these criteria would not result in the agency being able to make changes. For the three subcategories included in our assessment, we determined seven key practices from these subcategories that were most relevant to the Coast Guard’s modernization efforts and applied those practices to our assessment. We reviewed Coast Guard documentation and then made qualitative determinations about the extent to which the Coast Guard’s implementation of its modernization efforts addressed these criteria. A second analyst independently reviewed and validated each determination. We evaluated the Coast Guard’s actions against key reform practices to determine if they were generally, partially, or not at all applied. Generally applied. Agency documentation demonstrated that Coast Guard officials substantially applied applicable key practices. Partially applied. Agency documentation demonstrated that Coast Guard officials applied some key practices but not to a significant degree. Not at all applied. Agency documentation did not demonstrate that Coast Guard officials applied key practices. We deemed the following seven subcategories under the four categories as relevant criteria for assessing the Coast Guard’s reserve component Integration efforts: “Establishing goals and outcomes,” “Involving employees and key stakeholders,” “Using data and evidence,” “Addressing high risk and Longstanding management challenges,” “Leadership focus and attention,” “Managing and monitoring,” and “Strategic workforce planning.” We determined that the remaining three subcategories were not relevant to the Coast Guard’s reserve component integration efforts because we deemed the key practice more applicable to a government-wide effort or determined that it was too early to consider as the reserve integration effort was in its initial implementation stage. For the seven subcategories included in our assessment, we determined 19 key practices from these subcategories were most relevant to the Coast Guard’s reserve component integration efforts and applied those practices to our assessment. We reviewed Coast Guard documentation and made qualitative determinations about the extent to which the Coast Guard’s reserve component Integration actions addressed these criteria. A second analyst independently reviewed and validated each determination. We assessed the Coast Guard’s actions using the modernization effort scale: (1) Generally applied; (2) Partially applied; (3) Not applied; and (4) Minimally applied. Minimally applied. Agency documentation demonstrated that Coast Guard officials applied a limited number of key practices with significant gaps associated with each key practice. Our determinations are preliminary observations of the effort because Coast Guard’s reserve component organizational effort was in its nascent stages during our review. This presented several challenges in determining the point at which Coast Guard actions justify a rating of generally applied and partially applied. We applied the following decision rules to resolve these discrepancies: If one practice of the subcategory was rated partially applied, then we concluded that the subcategory as a whole partially applied. If one practice of the subcategory was rated generally applied, but one or more other key practices as either partially applied or not at all applied, then we concluded that the subcategory as a whole partially applied. If one practice of the subcategory was rated partially applied, but one or more other key practices rated either minimally applied or not at all applied, then we concluded that the subcategory as a whole minimally applied. We interviewed cognizant officials at Coast Guard headquarters, and field units, including the Atlantic and Pacific Area commands, and two Coast Guard districts and two Coast Guard sectors collocated with them. We interviewed officials from the two area commands because of their role in implementing organizational changes, and the districts and sectors for their perspectives on the Coast Guard workforce assessment process. Headquarters and field officials interviewed were responsible for the overall management of their organization in addition to officials responsible for facilitating the implementation of organizational change efforts. We reviewed prior GAO reports on organizational realignment, Coast Guard organizational changes, and high-risk issues in the federal government. In addition, we reviewed other reports evaluating long- standing agency management challenges. Finally, we reviewed documents and information on these organizational change efforts and compared them against Coast Guard guidance on organizational changes. To address our second objective, we analyzed Coast Guard documents related to management tools the Coast Guard has developed to determine its workforce requirements and identify personnel needs. Documentation included guidance and analysis related to developing workforce staffing needs, and strategies that set out the Coast Guard’s stated human capital principles. As with the first objective, we interviewed cognizant officials at Coast Guard headquarters, its Atlantic and Pacific area commands, and the two Coast Guard districts and two sectors collocated with them. Headquarters officials we interviewed were responsible for the development of manpower requirements and overseeing implementation of workforce assessments for Coast Guard units. We also reviewed prior GAO reports on workforce planning and Coast Guard personnel issues. Finally, we reviewed documents and information on these efforts to assess workforce requirements, and compared them against Coast Guard guidance on organizational changes for conducting the manpower requirements determination process, and our prior work related to strategic human capital management. To assess the extent to which the Coast Guard has supported its workforce with manpower requirements analyses and determinations, we analyzed all manpower requirements analysis (MRA) and determination documents the Coast Guard completed from 2003, when it began implementing the manpower requirements determination process, through calendar year 2019, the last full year of data available at the time of our review. Specifically, we requested the entire collection of MRA and determination documents from the Coast Guard. We then requested the number of positions that make up each unit type with a completed MRA. We assessed the reliability of the Coast Guard’s data through electronic testing, reviewing documentation, and interviewing Coast Guard headquarters and field unit officials regarding how these data were collected and used. We determined that these data were sufficiently reliable for determining the number of positions within each type of Coast Guard unit. With this information, for every MRA and determination completed, we calculated the number of positions in the Coast Guard’s workforce supported by available data. We conducted this performance audit from December 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In 2006, the Commandant of the Coast Guard issued 10 Commandant Intent Action Orders intended to address elements of the Coast Guard’s command and control structure, mission support systems, and business processes that were identified as detracting from mission execution. Table 2 provides an overview of the issues that drove the Coast Guard’s modernization effort, the intended goals for the effort as outlined in the 2006 Commandant Intent Action Orders, and examples of key actions the Coast Guard has taken to address the goals. The Coast Guard’s reserve component is its only workforce dedicated to respond to contingency operations such as natural and manmade disasters. The Coast Guard found that demand for reserve forces to augment its active duty workforce had grown as the service was called to respond to more contingencies. In 2006, under its organizational modernization effort, the Coast Guard issued a goal to optimize the use of the reserve component by ensuring the reserve workforce had the necessary training and support. The Coast Guard shifted governance of the reserve component under the Deputy Commandant for Mission Support; however under this structure, the Coast Guard did not take into account the difference between the reserves workforce being considered a program as opposed to a distinct component of the United States military. As such, in 2018, the Coast Guard chartered a project team to evaluate the state of the reserve component’s governance and develop alternate options to better integrate the reserves into the Deputy Commandant for Operations. In 2019, the Coast Guard integrated its reserve component into its Deputy Commandant for Operations governance structure. Officials told us that the goals for the new reserve component organization are to provide headquarters decision-makers enhanced visibility of operational readiness, competencies assigned and attained, and to use predictive modeling to look 2 or 3 years ahead to anticipate readiness posture and administrative readiness. They noted that achieving these goals relies on better data collection and developing metrics that can capture Coast Guard-wide information. Officials told us that, as of June 2019, the new organization was at initial operational capacity using existing staff. Table 3 provides our assessment of the extent to which the Coast Guard’s actions to reorganize its reserve component governance structure had applied key reform practices and examples of actions and deficiencies. From calendar years 2014 through 2019, the Coast Guard implemented the manpower requirements determination process for 30 of its 158 unit types. The Coast Guard completed manpower requirements analyses for 30 unit types. Of these 30 manpower requirements analyses, the Coast Guard completed required manpower requirements determinations— establishing a manpower requirement—for only four of these 30 unit types. Table 4 shows the most recent manpower requirements analyses, and corresponding determinations, completed from calendar years 2014 through 2019. In addition to the above contacts, Jason Berman (Assistant Director), Jennifer Kamara (Analyst-in Charge), Ben Atwater, Susan Czachor, Elizabeth Dretsch, Eric Hauswirth, Tracey King, Daniel Kuhn, and Kevin Reeves made key contributions to this report.", "summary": "The Coast Guard is a multi-mission maritime military service responsible for maritime safety, security, and environmental protection, among other things. Since 2006 the Coast Guard has implemented organizational changes to improve its effectiveness and efficiency. During this time, the Coast Guard also created a workforce assessments process to determine the number of personnel and skills required to meet mission needs. In April 2018, the Coast Guard reported to Congress that it was operating below the workforce necessary to meet its mission needs. GAO was asked to review the status of the Coast Guard's modernization and workforce assessment efforts. Among other things, this report examines the extent to which the Coast Guard (1) applied key practices for agency reorganization and (2) has assessed its workforce needs. GAO analyzed Coast Guard documents used to plan and implement its modernization effort against GAO key practices for agency reorganization. GAO also analyzed Coast Guard workforce assessments and data from 2003 through 2019. GAO also reviewed policy and planning documents and interviewed Coast Guard officials. The U.S. Coast Guard (Coast Guard) realigned its mission planning and mission support functions through an effort known as “modernization,” but did not consistently apply key practices for agency reorganization in implementing the effort. Of seven key practices, the Coast Guard did not apply two and partially applied three. For example, the Coast Guard did not measure its progress in achieving the goal of modernization, as key practices recommend. Coast Guard documents for organizational change and associated guidance do not require such practices to be followed. By ensuring such practices are implemented, the Coast Guard will be better positioned to determine the extent to which its investments meet modernization's goal of improving effectiveness and efficiency. Although the Coast Guard has informed Congress that it needs to increase its workforce, it has assessed a small portion of its workforce needs. Its preferred tool for assessing workforce needs is its manpower requirements determination process, which includes manpower requirements analyses (MRA) and is completed with a manpower requirements determination (MRD). Coast Guard guidance states that MRAs are to be updated every 5 years, and according to its April 2018 Manpower Requirements Plan, the Coast Guard's goal is to complete MRDs for all of its 58,000 personnel and 158 unit types. However, the Coast Guard had completed MRAs for 13 percent of its workforce and MRDs for 2 percent over the past 5 calendar years (see figure). The Coast Guard's plan does not include time frames and milestones for how it will achieve its workforce assessment goal, and information on the resources it needs to complete MRDs for all positions and units has not been updated in 10 years. By updating its plan to complete manpower requirements determinations and obtaining information on the resources needed to achieve its workforce assessment goal, the Coast Guard will be better positioned to ensure that it has the right number of people with requisite skills in the right units to meet its mission demands and to inform Congress of its manpower needs. GAO is making six recommendations, including that the Coast Guard measure progress in achieving the goal of modernization, update a plan with time frames and milestones for completing its workforce assessment goal, and obtain information on the resources needed to meet its goal. DHS concurred with our recommendations.", "document_type": "gao"}
{"report": "The federal government is the largest real property owner in the United States with a vast inventory costing billions of dollars annually to operate and maintain. Federally owned buildings include courthouses, offices, warehouses, hospitals, housing, data centers, and laboratories. GSA acts as the federal government’s landlord and is responsible for designing, constructing, and managing federal buildings that are occupied by federal agencies and the judiciary. Each year, GSA spends hundreds of millions of dollars on major construction projects, which include both new construction and repairs and alterations (R&A) to existing federal buildings. R&A projects can range from building system replacements and security upgrades to full building renovations. GSA manages its major construction projects through its central office in Washington, D.C., and its 11 regional offices. GSA’s central office establishes programming, design, and construction standards and guidance, and provides technical assistance, as needed, to the regional offices that are responsible for project implementation. To obtain authorization for projects above a defined threshold, GSA must submit to certain congressional committees a project prospectus that, among other items, describes the project and provides its estimated cost. Upon approving a project’s prospectus, Congress provides funding, either through an appropriation from the Federal Buildings Fund or appropriating funding to an agency. GSA posts approved project prospectuses on GSA’s public website. In general, GSA develops and implements projects through a sequential process that includes the following steps: Identification. Federal agencies submit a facility or space need to GSA; GSA prepares a feasibility analysis to determine the best way to fulfill the need, which could be through new construction, an R&A project, or a lease. Some R&A projects—limited to building system replacements—may be by identified by GSA based on building age and condition, and not originate from agencies’ space needs. Initiation. GSA assigns a project manager to define the project’s scope, develop cost and schedule estimates, and draft a project management plan (PMP). If a prospectus has not been previously submitted, GSA submits a prospectus to certain congressional committees for authorization. Planning. GSA’s project manager updates the PMP; the project’s baseline scope, schedule, and budget are finalized. Execution. For authorized and funded projects, GSA awards contracts for design and construction; the project’s baseline scope, schedule, and budget are revised, as needed, based on awarded contracts; GSA’s project manager monitors design and construction progress and manages changes to the project’s scope, cost, or schedule. Close-out. GSA’s project manager completes construction close-out activities and turns the project over for tenants’ use. GSA project managers perform key steps in the process that include overseeing contractors, monitoring and reporting on the progress of projects, managing changes to the project, and coordinating with tenant agencies. Additionally, GSA project managers are responsible for ensuring that “commissioning” is performed during the project. “Commissioning” generally requires that an independent commissioning agent oversee the construction contractor’s testing of installed building components to determine if they are performing as designed. According to GSA data, GSA substantially completed 36 major construction projects in the 5-year period from fiscal year 2014 through fiscal year 2018. The total cost of those 36 projects was approximately $3.2 billion. Listed below are some characteristics of those projects. Cost: Project costs ranged between $21 million and $343 million, with an average cost of about $89.3 million. Schedule: Project durations ranged between about 12 months and 79 months, with an average of about 43 months. Project Type: R&A projects made up the majority of projects (64 percent), with an average cost of about $74.2 million and an average duration of about 47 months. New construction projects accounted for 36 percent, with an average cost of about $116 million and an average duration of about 35 months. On average, R&A projects cost about $42 million less than new construction projects but took about 13 months longer to complete. See figure 1 for summary information on the cost and duration of these projects, by project type. Location: The National Capital Region (GSA Region 11) had the most projects with nine (25 percent), and all but one of the 11 GSA Regions had at least one project. Project Delivery Method: GSA utilized four delivery methods for 35 of the 36 projects in our 5-year time frame. Construction Manager as Constructor, whereby GSA contracts separately with a design firm and a construction contractor. The construction contractor is involved early-on to consult on the design as it is being developed; upon the design’s completion, GSA negotiates with the construction contractor on a price to undertake the construction. GSA used this method for 12 of the 36 projects (average cost of about $99.8 million). Design-Bid-Build, whereby GSA contracts with a design firm to develop a project’s design. After the design is completed, GSA contracts separately with a construction contractor. GSA used this method for 11 of the 36 projects (average cost of about $81.3 million). Design/Build-Bridging, whereby GSA contracts with a construction contractor to finish a partially completed design— termed a “bridging design”—begun by a separately contracted design firm. GSA used this method for 8 of the 36 projects (average cost of about $77.4 million). Design/Build, whereby GSA contracts with a contractor to provide both design and construction services under a single contract. GSA used this method for 4 of the 36 projects (average cost of about $120.4 million). See appendix I for more detailed information on each of the 36 projects. According to GSA officials and GSA’s internal construction-cost study prepared for GSA by the National Institute of Building Sciences (NIBS) in March 2016, several factors can result in higher costs for GSA’s construction projects compared to other similar private sector construction projects. For example, cost models in the 2016 NIBS study indicate that R&A projects cost roughly 15 to 25 percent more than R&A projects for a comparable Class A private sector building. Although the study was based on construction of R&A projects, both GSA and NIBS officials agreed that these same factors can contribute to similar cost premiums for GSA’s new construction projects compared to private sector projects. However, the NIBS staff who conducted the study told us that GSA’s more recent adoption of performance-based design standards, as compared to previously prescriptive standards, likely lowers the federal construction cost’s premium relative to private sector projects but some premium still exists. The performance-based design standards, for example, provide contractors greater latitude in selecting construction materials, which can have cost implications. According to the GSA’s internal construction-cost study, the factors that contribute to higher estimated costs for GSA construction projects when compared to similar private sector projects primarily include design and procurement requirements specific to federal projects that private sector counterparts may not have to comply with. Those requirements are specified in GSA’s design standards, as well as federal statutes and guidelines. Table 1 provides illustrative examples of factors cited by the study and GSA officials. In addition to the factors identified in the GSA’s internal construction-cost study, GSA officials said that meeting other statutory requirements, for example, the Buy American Act and the Federal Information Security Modernization Act of 2014 (FISMA), can contribute to higher costs for federal projects compared to private sector projects. GSA officials said that the cost of making information technology systems FISMA-compliant leads to federal projects costing more than private sector projects. FISMA-compliant systems, among other uses, are needed to enable the sharing of design and construction documents among GSA and contractor staff and the installation of control systems that are integral to the operation of building systems. GSA uses three principal tools—(1) project management plans (PMP), (2) peer reviews, and (3) “earned value management” (EVM)—to monitor its construction projects, including cost and schedule performance. The PMP is the overarching tool GSA and its contractors use to guide projects’ implementation. According to GSA policy, a PMP primarily defines the parameters of a project, to include scope, schedule, cost, implementation strategy, and risks, among other items. GSA policy also indicates that the PMP—which is an industry recognized tool—is to be updated during a project’s execution and reflect notable changes affecting the project’s scope, cost, and schedule. The PMP is to also establish stakeholder roles and responsibilities, project goals, and tenant expectations. In all of the five case-study projects we reviewed, we found the associated PMPs generally: outlined the project’s scope, cost, and schedule information; identified GSA’s project stakeholders—such as GSA’s project manager and GSA’s contracting officer—and representatives for the tenant agencies that the project will benefit; and identified potential risks posed to the delivery of the project. Four of the five PMPs included a “revision history” table that demonstrated that GSA generally used and updated the PMPs over the course of the projects’ execution. The fifth project’s PMP was developed prior to GSA’s 2012 update to its PMP standard format, which then required the use of a revision history log. More information pertaining to our case-study projects, including some information from the GSA PMPs we reviewed can be found in appendix II. The second tool GSA utilizes to monitor its construction projects is peer reviews. GSA policy requires that external peer reviews be conducted on projects with a construction cost over $25 million. Per GSA guidance, these on-site peer reviews typically occur twice during construction— when projects are about 15 percent and 60 percent complete. External peers—typically, construction industry experts who were not involved with the project—assess whether a project is progressing as planned and identify for GSA managers and project stakeholders any issues they observe that may affect its timely completion or cost. In general, peers also assess stakeholders’ working relationships and make recommendations for improvement or identify opportunities for greater consistency in the performance of GSA’s construction program or greater efficiency among project stakeholders. We found that four of our case-study projects utilized external peer reviews during construction, as required. For example, one peer review report included the following observations: The project team showed great progress toward completing the project on time, and potentially ahead of schedule; the implementation of the recommendations made during the initial external peer review resolved potential unknowns and cost issues that would have put the project at high financial risk; the safety record was exceptional; tenants were better informed; and security issues had been streamlined, allowing the contractor to staff the project in a timely manner. Most of the GSA’s project managers and construction contractors we interviewed for these four case study projects said they generally believed the external peer reviews were fair and added value. Our fifth case-study project did not utilize an external peer review because it was not required at the time GSA awarded the construction contract. The third tool GSA uses is EVM, which is an industry-recognized project management tool and is required for major federal acquisitions, such as construction projects, to help project managers monitor cost and schedule during project execution. According to the Office of Management and Budget’s (OMB) guidance and GAO’s cost-estimating guide, EVM measures the value of work accomplished in a given period and compares it with the planned value of work scheduled for that period and the actual cost of work accomplished in that period. The differences between the estimated and actual costs and schedule are used to determine, for example, whether less or more work had been completed than had been planned. By tracking these differences, EVM can provide warning signs of impending cost overruns or schedule delays and provide estimates of anticipated costs at completion. Consistent with our previous findings related to GSA’s use of EVM, we found that GSA continues to use EVM to assess its construction project delivery performance on two dimensions—on-schedule and on-budget: On schedule: GSA considers a construction project to be on- schedule if its construction duration is within 10 percent of the planned duration, from the construction start date to the substantial completion date (i.e., GSA considers a project to be substantially complete on the date the project space is suitable for tenant occupancy; however, the project’s cost could change prior to the actual contract close-out). On budget: GSA considers a construction project to be on budget if its actual cost is within the planned construction cost (as measured by the construction contract’s value at award or the contract value as adjusted based on post-award contract modifications) and the additional 7 to 10 percent construction contingency. According to GSA guidance, a project’s construction contingency is intended to cover unforeseen conditions and design deficiencies; it does not apply to additional scope. According to GSA officials, GSA’s central office uses EVM to conduct monthly performance reviews of GSA’s major construction projects. At these reviews, GSA’s central office considers certain proposed project changes forwarded for approval by GSA regional offices. We have previously reported that federal construction projects typically involve some degree of change as the project progresses and that contract changes, made through contract modifications, can occur for a variety of reasons, including design errors and unforeseen site conditions. In addition, GSA officials said that funding delays, tenant-caused delays, and site acquisition issues can also be factors that cause project delays. According to GSA guidance, while GSA regional offices have some latitude to make contract changes, the regional offices and their project managers must get central office approval if a proposed change is anticipated to exceed the approved contract cost, construction contingency, or schedule contingency. If such a change is approved, GSA will then revise—commonly referred to as “rebaseline”—either the construction contract cost, the planned schedule duration, or both. GSA will then use that new value to measure and report on the project’s budget and schedule performance. According to GSA officials and summary data on its rebaselining decisions, the majority of GSA’s major construction projects within our 5- year scope were rebaselined, within its policy, to account for changes to projects’ costs and schedules. Specifically, GSA officials told us they rebaselined 25 of the 36 projects (about 70 percent). Of those projects, 18 (50 percent) were driven, at least in part, by tenant-requested changes, which GSA officials said were the most prevalent reasons for rebaselining a project. According to GSA policy, if a tenant agency requests a project change that falls outside the original scope, the project manager is to ensure that the tenant agency provides all the associated design-related requirements and funding necessary to perform this additional scope. For example, for one of our case study projects, the tenant provided $17.7 million in additional funding as part of the final phase of its headquarters building’s multi-year modernization. The tenant’s funds paid for, among other things, the tenant-requested change to convert part of the multi-story library into offices to increase the building’s space efficiency and allow more staff to move into the building. Based on our review of GSA’s internal data, we found that four of our five case-study projects were rebaselined; GSA rebaselined the cost of two projects, the schedule of one project, and both the cost and schedule of one project. For example, concerning costs, GSA rebaselined one project to account for a $2.7 million increase to the contract—initially awarded for $21.8 million—upon realizing that the tenant’s plan to increase the number of occupants in the building required another stairwell be added for fire safety purposes. With regard to schedule, GSA rebaselined one project, as previously discussed, to address a tenant-requested change to convert parts of the library into offices; this change extended the schedule by about 1 year. Given GSA’s methodology that allows for rebaselining and GSA’s cost and schedule contingencies, GSA’s EVM performance data showed that all five case-study projects were completed on budget and on schedule, if not early. See appendix II for a summary of the cost and schedule performance of our five case-study projects. Federal agencies should report pertinent and reliable information to the Congress, so that Congress can adequately assess agencies’ progress in meeting established performance goals, ensure accountability for results, and understand how individual programs and activities fit within a broader portfolio of federal efforts to aid in federal funding decisions. GSA has publicly reported high-level information on its construction project performance in its Annual Performance Reports, which GSA provides to Congress and publishes on GSA’s website. For example, GSA’s fiscal year 2014 through 2018 Annual Performance Reports show that GSA met or exceeded its stated performance targets for project delivery (see fig. 2). Over this period (fiscal year 2014 through 2018), GSA took steps to improve the content and usefulness of its annual reports. For example, in fiscal year 2014, GSA included R&A projects in its performance measure to fully encompass all GSA capital construction projects. Prior to fiscal year 2014, GSA’s performance measure was calculated solely on the performance of GSA’s new construction projects. Also, starting in fiscal year 2017, GSA included additional summary-level information in its reports that identified the total number of projects and total contract value of both completed and ongoing projects that fiscal year. In fiscal year 2018, as shown in figure 2, GSA again revised its performance measure to reflect both the budget and schedule performance of projects. Prior to fiscal year 2018, GSA’s performance measure reflected only projects’ schedule performance. Further, in its fiscal year 2018 report, GSA listed the specific costs of its seven largest projects completed on-schedule and on-budget of the 24 projects completed that year. While GSA has taken some actions to improve the usefulness of its external reporting, neither GSA’s Annual Performance Reports nor its public prospectus website provide information on the extent to which projects have been rebaselined or the final costs of projects. Standards for Internal Control in the Federal Government state that agencies should provide necessary quality information to external stakeholders so that the external parties can help the agency achieve its mission and address related risks. As noted above, GSA regularly rebaselines projects, within policy, to account for changes to projects that affect construction contract costs and schedules due to a variety of reasons. GSA officials told us that they manage total project costs to be within the original prospectus estimate provided to Congress adjusted, as applicable, by funds it receives for tenant-requested changes; the officials do not believe that it is critical to report final costs or if projects have been rebaselined. However, we have found that simply measuring and reporting performance based on the most recent baseline may obscure how projects have performed over their entire construction time frame. Being more transparent about which projects or how many projects were rebaselined, as well as reporting cost and schedule growth from original baselines, can provide stakeholders with a more accurate view of project performance and enhance accountability. Reporting on such cost information, for example, would allow GSA to communicate to Congress actual construction costs at a project’s completion that may be different than the estimated costs on the prospectus approved by Congress at the project’s initiation which likely did not account for items to be funded by tenants. Without that information, it is not possible for Congress to know how projects performed against approved estimated costs and whether final project costs are consistently above, below, or meeting estimated costs. Having this information could benefit Congress in its oversight role and in making future funding decisions. Key Challenges Identified during Commissioning of Case Study Projects Issues with State-of-the-Art Building Systems State-of-the-art building systems and the automation systems that monitor and control them were not optimally operating for at least two of our case-study projects at substantial completion. For example, stakeholders for one project reported that it was very challenging to get all the integrated systems to work properly, in part, because the design was very technologically advanced. One GSA official said the biggest challenge was coordinating the operations sequence of the various building systems to function as the design team intended. As such, it took well over a year after the building was completed to resolve these issues. Limited Capabilities of Building Contractors to Maintain Complex Systems In three of the five case-study projects, stakeholders said maintenance service contractors were either not prepared to assume or had not yet been contracted to provide for the higher technical maintenance and operation responsibilities for all the building systems. For example, one construction contractor said there seemed to be a knowledge gap between the technical capabilities needed to effectively manage the more advanced building systems and the skills possessed by the existing maintenance contractor. A GSA official said that GSA plans to solicit a new contract for the building’s maintenance. agencies, and others. The Guide identifies its primary audience to be: GSA’s project managers, their construction management agents who help GSA manage the project, and the commissioning agent who oversees the commissioning process. The Guide’s secondary audience includes the many other stakeholders in the commissioning process, including tenant agencies. According to the Guide, the commissioning process is intended to assist in preparing maintenance personnel to operate and maintain any newly installed building systems. We found that GSA conducted commissioning largely in alignment with the Guide on our five case-study projects based on our review of project documentation and interviews with GSA’s project managers, facilities managers, and contractors. Further, we identified two key challenges in regard to state-of-the-art building systems’ and building contractors’ capabilities. See sidebar for additional information on the two challenges. While GSA generally conducted commissioning according to its Guide on the five case-study projects we reviewed, we found that the 2005 Guide is outdated. For example, the Guide references dated industry practices and some outdated external guidance, both of which were in existence at the time the Guide was developed. Specifically, it references the 2003 Leadership in Energy and Environmental Design (LEED), Green Building Rating System, Version 2.1; however, the LEED rating system for projects since 2016 was Version 4.0, and Version 4.1 was recently issued in 2019. We also found disconnects between the 2005 Guide and GSA’s current design standards or industry practices. For example: While the Guide states that GSA buildings should be LEED certified and strive for a Silver certification, GSA now requires buildings to achieve a higher certification, LEED Gold. The Guide states that GSA “strongly recommends” that GSA regions—and agencies to which GSA has delegated the operations of federal buildings—recommission buildings every 3 to 5 years. The current LEED standards call for “periodic commissioning requirements, ongoing commissioning tasks, and continuous tasks for critical facilities.” In general, over the past decade, federal statutes, guidance, executive orders, and changes to industry building certifications have moved the federal government and the industry toward more real- time, continuous monitoring and commissioning in cases where advanced building-automation systems, energy information- management systems, and advanced meters (e.g., electrical, water, gas, temperature, and light meters) have been installed. The continuous data provided by these systems can help building owners make real-time adjustments to optimize building operations. However, the Guide does not mention continuous monitoring-based commissioning as a possible option to, or in addition to, recommissioning buildings. Standards for Internal Control in the Federal Government state that management should periodically review policies, procedures, and related control activities for continued relevance and effectiveness in achieving the entity’s objectives or addressing related risks. Those standards also indicate that if there is a significant change in an entity’s process, management should review this process in a timely manner after the change to determine that the control activities are designed and implemented appropriately. Without updated guidance, GSA’s commissioning activities may be limited in their effectiveness in assuring building systems are operating optimally. Two of the five GSA contractors we interviewed expressed frustration that the commissioning process on their projects did not run smoothly. GSA’s external peer reviews for those same two projects also found that the roles of the various stakeholders in the commissioning process were not clear. In addition, three stakeholders on one of those projects said that some stakeholders—especially GSA’s contracted design team—were not fully involved during part of building’s commissioning. In light of our review, GSA is planning to evaluate its commissioning guidance to determine an appropriate update. GSA officials stated that this update may result in revising the existing commissioning guide or replacing it with industry-recognized guidance. However, GSA is still in the process of identifying the scope of the update, including a timeline and resources required to do so. According to OMB guidance, Post Occupancy Evaluations (POE) are tools to evaluate the effectiveness of an agency’s overall capital acquisition process. The primary objectives of a POE include (1) identifying how accurately a project meets its objectives, expected benefits, and strategic goals of the agency and (2) ensuring the continual improvement of an agency’s capital-programming process based on lessons learned. The guidance also states that agencies should have a documented methodology for conducting POEs to ensure that each asset is evaluated consistently. The guidance identifies 17 factors to be considered for evaluation in conducting a POE, such as a project’s performance, compliance with design standards, maintenance issues and building workforce competences, use of advanced building technologies, tenant satisfaction, and cost savings. The guidance also notes that a POE should generally be conducted 12 months after the project has been occupied to allow time for the tenant to evaluate the building’s performance and the delivery of the project. However, the guidance allows agencies some flexibility in the timing of a POE to meet their unique needs if 12 months is not the optimal timing to conduct the evaluation. We found GSA did not conduct any POEs on its completed major construction projects in the 4-year period from 2014 to 2017, as called for by OMB guidance. In fiscal year 2018, GSA contracted with the National Institute of Building Sciences (NIBS) to conduct six POEs and seven additional POEs in fiscal year 2019. GSA officials told us that while they understand the value POEs can provide, they are only able to conduct them when funding is available. They explained that POEs are funded through general program funding (not project funding based on the approved prospectus) within GSA’s Office of Facilities Management, and the available resources to conduct such efforts are limited given other GSA portfolio-wide maintenance and operations priorities. GSA acknowledged that it did not have a specific policy for conducting POEs or selecting completed projects for POEs. Instead, GSA officials said when selecting which buildings should undergo a POE, they ensure there is a representation of different building types (i.e., federal buildings, U.S. courthouses, and land ports of entry) and a mix of new and R&A projects. Because GSA does not have a policy for POEs, NIBS developed a general methodology, which it used for conducting each of those POEs. While GSA tries to ensure there is a mix of projects represented when selecting POEs, it is not clear that its selection factors help ensure GSA makes the best use of its limited resources. To balance OMB’s guidance to agencies that POEs should be conducted on agencies’ completed capital-construction projects, and given its resource constraints, GSA could benefit from a more strategic approach to select the projects for POEs. For example, GSA could use a risk-based approach to select for POEs (e.g., more expensive projects or those that include the integration of advanced, state-of-the-art building systems) to help improve the design and construction of future projects. Such an approach is consistent with the Standards for Internal Control in the Federal Government, which states that management should design control activities to achieve objectives and respond to risks and implement those control activities through policies. Control activities could include establishing criteria for selecting projects for POEs and formalizing it through policy. GSA officials also noted that GSA has conducted multi-building studies— which share some similarities with individual building POEs—that GSA officials broadly consider to be POEs. However, while the studies assessed some of the factors described in OMB guidance (e.g., project performance, maintenance, or advanced technology use), none of them comprehensively reviewed the 36 projects in our 5-year time frame. Accordingly, while these broader studies can provide some useful information to GSA, they are limited in their ability to provide GSA with timely information that meets the POE goal stated in OMB’s guidance: “to evaluate the overall effectiveness of the agency’s capital planning and acquisition process” and to “solicit customer feedback and incorporate that feedback into improvements to the performance and delivery of the capital investment process.” OMB guidance states that agencies should establish mechanisms to use lessons learned from POEs to minimize risks of repeating past mistakes on future projects. Along these lines, NIBS produced a summary report for GSA of the six 2018 POEs it conducted; the report identified design, construction, commissioning, and operational maintenance issues and lessons learned. From these lessons learned, NIBS also offered some recommendations to GSA. For example, NIBS said that GSA should establish a POE review committee to examine GSA’s building designs to highlight and offer solutions to previously identified problems in other buildings and develop and distribute a checklist describing the identified problems to teams that are responsible for designing new buildings. GSA developed an operational guide to synopsize the lessons learned from the NIBS report and expects that future building projects will benefit through its efforts to incorporate these lessons in the design of future projects. Further, NIBS reported that improvements to future projects in response to the issues identified in the six 2018 POE projects would result in reductions to GSA’s future operational costs. However, it is unclear whether the extent of these issues and lessons learned are unique to the 2018 POE projects reviewed by NIBS, or may be occurring across more of GSA’s construction projects. According to NIBS officials, they have observed some recurring project issues among the six POEs conducted in fiscal year 2018 and two of the seven conducted in fiscal year 2019. GSA officials said that they plan to implement lessons learned from these POEs into GSA’s design standards by the end of 2019 and expect to later update these design standards based on future POEs. According to GSA officials, they made NIBS’s individual POE reports and the 2018 POE summary report available to their project managers through a shared folder on GSA’s internal intranet site, which can be accessed by over 120 staff. In addition, one GSA project manager told us that GSA periodically holds knowledge-sharing webinars with its project managers where lessons learned from specific projects may be presented. This official indicated that the knowledge-sharing presentations are heavy on photos and that there is no real prescribed format or requirements for content. Accordingly, the presentations are an informal way for project teams to share project knowledge across GSA’s regions. Further, this official said the lessons-learned presentations from those webinars are also posted for a period of time on GSA’s internal website. However, communicating information via such means provides ad-hoc benefits to only the select individuals who know about the availability of the reports or webinars, and choose to access them. This approach may not effectively expand the broader knowledge base of the organization or best position GSA to, as OMB guidance indicates, ensure continual improvement of an agency’s capital-programming process based on lessons learned. Standards for Internal Control in the Federal Government also indicate that management should communicate necessary quality information to all relevant internal stakeholders to achieve the entity’s objectives. Without a sustained effort to consistently conduct POEs on its completed projects, GSA may miss opportunities to gather valuable tenant feedback and to identify marked successes or notable problems, including any issues that are recurring. Such information could inform future improvements to GSA’s major construction projects and increase tenant satisfaction. Further, such information may also help identify the need to change or update some of GSA’s policies, standards, guidance, or practices, such as those recommended by NIBS or other project stakeholders. However, even if GSA undertakes a more systematic approach to conducting POEs, the benefits of doing so can only fully materialize if GSA takes steps to effectively communicate POE lessons learned to all staff who may be at risk of repeating previously identified project mistakes. GSA annually spends hundreds of millions of dollars on major construction projects to provide tenant agencies with new buildings and modernized spaces that help support agencies’ missions and enable the effective delivery of government services. GSA has improved its public reporting on major construction projects to depict project schedule and budget performance over time. However, GSA’s public reporting does not include information about the extent to which projects’ schedule or costs were rebaselined, or on projects’ final costs, which may differ from GSA’s estimates in the initial prospectuses approved by Congress. Providing the additional information on projects’ schedule and cost rebaselining, and projects’ final costs could further benefit Congress in its oversight role and improve public knowledge about the full costs of major federal construction projects. In addition, given the significant fiscal exposure for the government to maintain these buildings for the long term, having updated guidance on commissioning would enable GSA to better ensure that completed projects are meeting GSA’s design standards. Finally, given resource constraints, identifying and communicating information about when and how POEs are to be conducted could help GSA maximize opportunities to capture lessons learned from completed projects. Knowledge gained from POEs could also ensure tenant agencies are satisfied with completed projects and improve the design and construction of major projects in the future. We are making the following three recommendations to GSA: The Administrator of the GSA should report for Congress and the public— for example, on GSA’s prospectus website—the extent to which completed projects’ construction costs and schedules were rebaselined and final construction costs, to include any additional funding tenant agencies may have provided to GSA for changes. (Recommendation 1) The Administrator of the GSA should update its 2005 Commissioning Guide—or replace it with appropriate industry-recognized standards and guidance—to be consistent with GSA’s current design standards and industry practices. (Recommendation 2) The Administrator of the GSA should identify and communicate—such as through policy, guidance, or other appropriate mechanism—(a) when and how Post Occupancy Evaluations should be conducted for completed projects considering resource constraints and (b) how recommendations or lessons learned from those evaluations are effectively communicated to future project teams. (Recommendation 3) We provided a draft of this report to GSA for review and comment. In written comments, reproduced in appendix III, GSA stated that it partially concurred with recommendation 1 and concurred with recommendations 2 and 3, and provided related comments. In response to recommendation 1, GSA agreed to publish key information that would be helpful, such as GSA’s total construction costs at project completion. However, GSA said it would be misleading to publish information on additional funds provided to GSA from tenant agencies— that lead to contract changes and rebaselining—because these funds come from different appropriations. GSA believes this would not accurately reflect how GSA managed its original budget and schedule. However, we believe that reporting total project costs in a way that clearly identifies both GSA and tenant agency costs is possible, and would not be misleading. We continue to believe that such additional transparency in reporting can benefit Congress in its oversight role and improve public knowledge about the full costs of major federal construction projects. Related to recommendation two that GSA concurred with, the agency noted that it has other commissioning documents and processes outside of its Building Commissioning Guide (Guide) that it uses to ensure building systems are operating optimally. We believe GSA’s use of other documents and processes is a good practice in light of the outdated nature of its current Guide, which serves as a key document in its commissioning process. Nevertheless, we continue to believe that it is important for GSA to update its outdated Guide, or replace it with appropriate industry-recognized standards and guidance to be consistent with GSA’s current design standards and industry practices, as we recommended. Finally, regarding recommendation three, after a discussion with GSA officials during the comment period, we modified the wording of the recommendation to recognize the range of administrative tools (e.g., policy, guidance, or other appropriate mechanism) that GSA could use to identify when and how Post Occupancy Evaluations (POEs) should be conducted and how lessons learned from those evaluations are communicated. As we noted in the report, under its current process, GSA selects the number of facilities evaluated as its annual budget allows based on several selection factors. We continue to believe that GSA could benefit from a more formalized and strategic approach to identifying and communicating when and how POEs should be conducted to make best use of its limited resources. GSA also mentioned its Design Guide for Operational Excellence as a tool to communicate lessons learned from POEs. We agree that such a guide is a good example of how POEs can be used to inform the design of future projects. However, because the guide was based on a limited number of POEs from 2018, we believe that there is more GSA can do to maximize opportunities to communicate lessons learned to future project teams. The draft report had included a fourth recommendation for the Administrator of the GSA to improve the transparency of what is being measured and reported in GSA’s Annual Performance Reports, including noting any key limitations, such as comparing results from year to year if the measure changed. While GSA was reviewing the draft, the agency provided clarifications on the structure and content of its annual reports that mitigated our concerns about the transparency of the information being presented. As a result, we made changes to the body of the report and removed that recommendation from our final report. GSA also provided technical and clarifying comments, which we incorporated, where appropriate. We are sending copies of this report to the appropriate congressional committees and the Administrator of the General Services Administration. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or rectanusl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Appendix I: General Services Administration’s (GSA) Completed Major Construction Projects, Fiscal Years 2014 to 2018 Case study project. This appendix contains information on five General Services Administration (GSA) case-study projects that we included in our review. We judgmentally selected these five major construction projects that were substantially completed between fiscal years 2014 through 2018 representing diversity in project type, geographic area, building-type, and range in cost and scope. Although not generalizable to all GSA major construction projects, information gathered from our case studies provides illustrative examples of GSA’s monitoring and construction efforts. For each case study, GSA provided us with extensive project documentation. We reviewed this documentation to obtain key information such as on contract award amounts and modifications that resulted in changes to the project’s original budget or schedule. The contract modifications we discuss for each project are examples of modifications that added cost or credit to the final contract value or that changed the delivery schedule; however, these modifications do not necessarily include all the modifications to the construction contract. In addition, we interviewed relevant stakeholders, such as GSA project managers, contractors, and facility managers who were involved with the projects. All information in the case study narratives is attributable to GSA based on our review of project documentation and interviews with GSA project officials and stakeholders. The Charles F. Prevedel building was constructed in 1990. For fiscal year 2014, GSA proposed alterations and renovations to the building’s interior and upgrades to the building’s systems such that the Veterans’ Benefits Administration could consolidate into the building. The building was nearly two-thirds vacant at the time, as two federal tenants had moved out of the building. The Veterans’ Benefits Administration had been dispersed in both a nearby federal building and leased space. GSA estimated that the Veterans’ Benefits Administration’s move into the consolidated space would save $3.3 million annually in lease costs. The building is five stories above-grade and two-stories below-grade. The project scope included renovating the building’s central atrium; reconfiguring and increasing the building’s useable space; replacing obsolete heating, ventilation, and air-conditioning (HVAC) systems; and, installing an energy-management control system to automate the HVAC and lighting systems and reduce energy consumption. The HVAC upgrades also included replacing and relocating the outdoor air intakes on the roof in order to meet current security requirements. Minor seismic upgrades were also implemented. The design/build-bridging construction contract was awarded in January 2015 for $21.8 million. The construction contract cost was rebaselined to $25.4 million, in part, to provide an additional stairwell to meet life-safety egress requirements as required by GSA’s design guide. GSA reported that change required GSA’s Public Buildings Commissioner to approve an overall project budget escalation of $2.7 million in June 2015. GSA reported the final construction cost was $25 million (roughly a 14.5 percent increase from the initial construction contract award). Construction of the repair and alteration project started in May 2015 and was substantially completed after a year and a half in November 2016, approximately 2 months earlier than originally projected. Figure 3 shows before and after views of the building’s main lobby and newly installed stairwell. Figure 4 shows views of meeting and training room spaces renovated during the project. Location (GSA Region): Bangor, Maine (Region 1) Original Construction Completion Year: 1967 Project Type: Repair and Alteration Project Delivery Method: Construction Manager as Constructor (CMc) The 3-story Margaret Chase-Smith Federal Building and Courthouse was built in 1967 and had not had a major renovation since its construction. The project was funded by the American Recovery and Reinvestment Act of 2009 (Recovery Act). GSA proposed the project be funded to recapture the vacant space in the building, which in part increased to approximately 33 percent after the U.S. Postal Service vacated. The proposed project would renovate and provide alterations to the building that would expand space for its existing tenants—including the U.S. Courts and the Social Security Administration, among others—and provide space for new tenant agencies. GSA officials reported that in order to get the project started quickly using Recovery Act funds, GSA made the decision to deliver the project under the Construction Manager as Constructor (CMc) delivery method. Under CMc method, the contractor was brought in to advise on the design as it was being completed. In addition to space renovations and alterations, the project repaired and replaced HVAC systems, improved energy efficiency, and provided exterior structural improvements including the replacement of windows. New secure elevators were also added to improve court safety. Other components of the project included repairs and replacements of electrical systems, hazardous materials mitigation, elevator improvements, upgrades to the fire protection system, installing sprinklers, and correcting code deficiencies including bringing the building into compliance with accessibility standards. The CMc construction contract was initially awarded in March 2010 for $33.9 million. In September 2010 (6 months later), two contract modifications totaling roughly $4.6 million were issued to increase the contract price to reflect changes made in completing the design. GSA and the contractor reported that the baseline construction contract—after the design was completed—was $38.5 million. While GSA had provided some funding allowances within the initial construction contract to address some project requirements that were not yet fully designed— such as the building’s entry pavilion—another $1.9 million contract modification was issued in March 2011 (a year after the initial contract award), in part, to increase the funding allowances for the front entry pavilion and to provide additional glass that was to be installed in the lobby area. The entry pavilion was added to improve the security screening process and adhere to the U.S. Marshalls Service and U.S. Courts screening station requirements. That $1.9 million cost modification also addressed increased requirements associated with the geothermal heating system and below grade wells. Also, the contract costs increased, in part, due to tenant-requested changes. For example, an $802,000 contract modification was issued, in part, for requested millwork (e.g., judge’s bench and cabinet work) and the Court’s audiovisual equipment, telecommunications, and data-related requirements. GSA reported the final construction cost was approximately $41.3 million (about a 7.5 percent increase above the $38.5 baseline). Construction of the repair and alteration project started in October 2010 and was substantially completed approximately one month early in November 2013. Figure 5 shows the exterior of the building including its new entry pavilion. Figure 6 shows an exterior side view of the new entry pavilion and an interior view of the lobby. As part of the Recovery Act, the Social Security Administration received an appropriation to construct a new National Support Center to replace an older data center whose systems were approaching the end of their useful lives. The new National Support Center provides a state-of-the-art data center, added reliability, and the ability to expand to meet future needs. For example, the data center’s flexible, scalable design allows for a smooth transition to future information technology upgrades and new, emerging technology. The new 300,000 gross square foot data center complexbuilt on a 63 acre siteincludes the data center, warehouse, and office building; the facility was built to accommodate 200 employees. The constructed facility—supporting 24 hours a day, 7 day a week operations—is Leadership in Energy and Environmental Design (LEED) Gold Certified, even though data centers traditionally rank among the largest power users in modern facilities. GSA’s estimated construction cost for the project was adjusted down in August 2012 from $334 million to $262 million. GSA awarded the design- build construction contract in January 2012 for $191.6 million. The project’s construction contract cost was later rebaselined to $207.4 million due in part to the Social Security Administration requesting GSA have the contractor provide operations and maintenance transition services for 6 months. That contract change was made in March 2014—approximately 4 months before substantial completion—for roughly $2.1 million. GSA reported to us that the final construction cost was $208.1 million (roughly an 8.5 percent increase from the base contract award). Because the construction cost was well below GSA’s original construction estimate of $334 million, GSA reported to us the remaining project funds were returned to the Social Security Administration in accordance with the Recovery Act appropriation. GSA issued a notice to proceed (i.e., contract start date) to the design-build contractor in January 2012 and the project was substantially completed on-schedule roughly two and a half years later in July 2014. Figure 7 shows an exterior view of the main entrance to the data center. Figure 8 shows an interior view of the data center’s server space prior to occupancy. Figure 9 shows an exterior view of the on-site solar panel array with the data center in the background. The Department of the Interior (Interior) headquarters building— occupying two city blocks—was initially completed in 1936; upgrades to the building’s systems were required to extend the useful life of the building, support Interior’s operations, and meet current building codes and standards. In 2000, GSA began the construction of its multi-year, six- phase modernization plan, where each of the building’s six wings was to be modernized during one of the six phases. Phase 6 (Wing 1)—the final phase of the building’s modernization— included upgrading the mechanical and electrical systems, replacing the lights and ceiling systems, installing fire safety upgrades and emergency egress stairs, upgrading restrooms, improving accessibility, and restoring historic spaces to include the auditorium, library, and the Undersecretary’s and Secretary’s suites. In 2001, GSA originally negotiated with the contractor the costs to execute Phase 6, which was structured as a contract option. The option could be exercised at GSA’s discretion upon receiving funding but allowed for future, economic price escalation, for inflation. The contract price in 2001 for the Phase 6 scope was approximately $19.3 million. Because appropriated funding was not received until fiscal year 2014, that earlier contract pricing was contractually updated by GSA in 2014 to roughly $38 million; however, that figure included roughly $4.5 million in additional scope that GSA added into the project. The additional scope included, among other items, that the Phase 6 space was to be certified under the Leadership in Energy and Environmental Design criteria and that lessons learned from the earlier completed phases—implemented over nearly 15 years—would be incorporated into the Phase 6 project. Additionally, Interior asked GSA that parts of the library be converted into office spaces to increase the building’s space efficiency and allow Interior to move more personnel into the building. That contract change, for about $6.2 million, was made in May 2016 and also resulted in the schedule’s being rebaselined, adding about one year to the project’s duration. GSA reported that Interior provided $17.7 million in additional funding, inclusive of the costs for converting the library space. GSA reported that the construction contract cost for Phase 6 was $51.7 million (about a 36 percent increase above the 2014 adjusted, base contract cost of $38 million). Phase 6’s construction started in May 2014 and was completed approximately 3 years later in June 2017. Figure 10 shows the exterior of the Department of Interior headquarters building with its six wings. Figure 11 shows interior view of historic spaces that were restored during Phase 6. The primary driver for the project was to address the long term housing needs of the United States Courts and related agencies. The District Court required additional space that the adjacent existing John A. Campbell Courthouse could not provide, and GSA determined that a new courthouse was necessary to accommodate the Courts’ projected 10 to 30 year space needs. The Campbell Courthouse renovation followed the new courthouse construction to allow for the relocation of the Bankruptcy and Probation Courts from leased space, and allow for the full Court family to be co-located between the two adjacent buildings. The new courthouse building, adjacent to the existing Campbell Courthouse, was designed to provide 155,600 gross square feet of space, including parking. The building houses six courtrooms, nine judges’ chambers, the United States Marshalls Service, 38 below-grade parking spaces, and the capability to expand and accommodate eight additional courtrooms in the future. In fiscal year 2010, the new construction project received partial funding in an appropriation in the amount of $50 million, for construction. However, the project was not awarded at that time. The U.S. Courts and GSA had to revisit the long-term space needs for the U.S. Courts, which was later done as part of GSA’s 2013 feasibility study. In fiscal year 2014, an additional $69.5 million was appropriated for a new approach that would involve repairs and alterations to the existing Campbell Courthouse, as well as the construction of a new federal courthouse (which was to be smaller than originally designed), adjacent to the Campbell Courthouse. GSA fiscal year 2014 documentation for the new courthouse project estimated the total design cost at $8.5 million and the total construction cost at $71.1 million, which excluded any prior funding spent on site acquisition costs and the project’s earlier design. In April 2015, GSA awarded a single design-build contract for both the design and construction of the new courthouse and for the repairs and alteration of the existing Campbell Courthouse. GSA baselined the construction cost for the new courthouseexclusive of the costs for the Campbell Courthouse alterationsat $70 million. GSA data showed that the final construction cost for the new courthouse was $72.6 million (an increase of about 4 percent over the baseline cost of $70 million; roughly 9 percent less than the $79.6 million total estimated costs for both the design and construction). Construction started in Spring 2016 and was completed in just over 2 years, in June 2018. The schedule was rebaselined by roughly a month for severe weather delays during the construction. Figure 12 shows the exterior of the new U.S Courthouse and two interior spaces. In addition to the contact named above, Mike Armes (Assistant Director); Catherine Kim (Analyst-in-Charge); John Bauckman; Delwen Jones; Timothy Kinoshita; Ying Long; Malika Rice; Rachel Stoiko; and Crystal Wesco made key contributions to this report.", "summary": "As the federal government's landlord, GSA spends hundreds of millions of dollars to construct or modernize federal buildings. By delivering these major construction projects, GSA supports tenant agencies' missions and facilitates the delivery of government services. GAO was asked to review GSA's major construction projects. This report: (1) identifies costs of these projects in the last 5 years and factors that contribute to those costs; (2) examines how GSA monitors and publicly communicates cost and schedule information; and (3) assesses GSA's efforts to confirm that projects meet GSA's requirements and that tenants are satisfied with completed projects. GAO analyzed GSA's performance data from fiscal years 2014 to 2018 for 36 projects with a minimum cost each of $20 million (i.e., a major construction project); selected five case-study projects representing diversity in project type, geographic area, building type, and range in cost and scope; reviewed applicable GSA policies, procedures, guidance, and reports; and interviewed GSA officials and project stakeholders. In fiscal years 2014 through 2018, the General Services Administration (GSA) completed 36 major construction projects—projects with a minimum cost of $20 million to construct new buildings or modernize existing buildings—with a total cost of $3.2 billion. According to a GSA consultant, factors specific to federal construction projects may result in GSA's projects costing roughly 15 to 25 percent more than comparable private sector projects. For example, GSA uses more durable but more expensive materials to achieve a longer building service life compared to private owners who may plan for a shorter service life. GSA's Annual Performance Reports to Congress do not indicate how much GSA “rebaselined” projects' schedules and costs. Rebaslining reestablishes the point at which GSA measures on-schedule and on-budget performance. In accordance with agency policy, GSA rebaselined 25 of 36 projects GAO reviewed to account for issues such as design changes and tenant-funded requests. For example, GSA rebaselined one of its modernization projects for a $2.7 million increase to the construction contract initially awarded for $21.8 million. The increase resulted from a design change to add a stairwell for fire safety purposes to accomodate the tenant's plan to increase the building's occupants (see figure). After GSA rebaselines a project, costs may differ from the project estimates approved by Congress. Because GSA does not report the extent that it has rebaselined projects or projects' final costs, Congress lacks information about GSA's performance: such as whether final costs are consistently above, below, or meeting estimated costs. Reporting such information could benefit Congress' ability to carry out its oversight role and improve transparency about the full costs of major federal construction projects. GSA assesses whether projects meet requirements and tenants' needs but does not fully capture or share lessons learned. For example, GSA uses “commissioning”—testing installed building systems—to validate that the buildings' systems function as designed. However, because GSA's 2005 commissioning guide references outdated guidance, the effectiveness of its activities may be limited in assuring buildings are operating optimally. GSA also uses post occupany evaluations (POE) to assess projects' performance and tenants' satisfaction. However, in the last 5 years, GSA has not regularly conducted POEs, due in part to resource constraints, and lacks a policy for selecting projects for POEs and communicating findings from completed POEs. As a result, GSA may be missing opportunities to fully utilize POEs to gather tenants' feedback and inform the design and construction of future projects. GAO is recommending that GSA (1) report the extent projects were rebaselined and their final costs; (2) update GSA's commissioning guidance; and (3) identify and communicate when and how to conduct POEs and share lessons learned. GSA concurred with two recommendations and partially concurred with the other, which GAO believes should be fully implemented as discussed in the report.", "document_type": "gao"}
{"report": "In April 2016, IRS released its most recent tax gap estimate, stating that taxpayers should have paid an average of about $2.5 trillion dollars per year in federal taxes for tax years 2008 to 2010. Of this amount, IRS estimated that taxpayers voluntarily and timely paid about 81.7 percent, or $2.04 trillion, leaving $458 billion in unpaid taxes per year, as shown in figure 1. The tax gap estimate is an aggregate estimate of the five types of taxes that IRS administers—individual income, corporation income, employment, estate, and excise taxes. For each tax type, IRS attempts to estimate the tax gap based on three types of noncompliance: (1) underreporting of tax liabilities on timely-filed tax returns; (2) underpayment of taxes due from timely-filed returns; and (3) nonfiling, when a taxpayer fails to file a required tax return altogether or on time. Underreporting of tax liabilities accounted for most of the tax gap estimate for tax years 2008 to 2010, making up 84 percent of the entire estimated gross tax gap, as shown in figure 2. Individual income taxes made up the largest portion ($264 billion) of underreporting. Underreporting of business income accounted for nearly half ($125 billion) of that amount, as shown in table 1. Business income underreporting includes income from sole proprietors, which accounted for the largest share ($78 billion) of individual income tax underreporting. IRS uses various approaches to estimate the different components of the tax gap. A primary source of information IRS uses is its National Research Program (NRP) study of individual tax returns. Through NRP, IRS examines a stratified, random sample of tax returns, and uses statistical modeling to produce estimates of noncompliance for the population of individual income tax return filers. Other areas of the tax gap are estimated using payment data or other statistical models. In 2016, IRS completed examinations for an NRP study on employment tax returns filed from tax years 2008 to 2010. IRS employees reported that they plan to start analyzing the results by June 2019. However, IRS has not provided plans for how it will use the results to update the current state of the employment tax gap estimate, as we previously recommended. The tax gap includes unintentional errors as well as intentional evasion, such as intentionally underreporting income, intentionally overreporting expenses, and engaging in abusive tax shelters or frivolous tax schemes. As we have previously reported, completely closing the tax gap is not feasible, as it would entail more intrusive enforcement and more burdensome recordkeeping or reporting than the public is willing to accept, and more resources than IRS is able to commit. However, even modest reductions would yield significant financial benefits and help improve the government’s fiscal position. Tax noncompliance, even when unintentional, could discourage compliant taxpayers and undermines the integrity of the tax system and the public’s confidence in it. For example, consider two groups of taxpayers with similar tax situations—those who pay the full amount of tax due and those who do not. Those who do not pay taxes are not meeting their obligation to fund government services, which, in effect, shifts the fiscal burden to those who do pay. Further, IRS devotes resources to attempt to collect taxes due from noncompliant taxpayers—resources that could be used for other purposes. In addition, noncompliance can create an unfair competitive advantage among businesses because those that do not pay tax debts are avoiding costs that tax-compliant businesses are incurring. For example, our past investigations identified instances in which federal contractors with tax debts won awards based on price differentials over tax compliant contractors. Our past work has found that three important factors contributing to the tax gap are the extent to which income is reported to IRS by third parties, IRS’s resource trade-offs, and tax code complexity. As we have previously reported, the extent to which individual income tax taxpayers accurately report their income is closely aligned with the amount of income that third parties report to them and to IRS. For example, according to 2008–2010 IRS data, taxpayers misreported more than half of the types of income for which there is little or no third-party information reporting, such as business income (see figure 3). In contrast, when employers both withheld taxes from, and reported information on, wages and salaries to employees and IRS (through Form W-2, Wage and Tax Statement), taxpayers misreported on only 1 percent of such income. Similarly, taxpayers misreported less than 10 percent of investment income that banks and other financial institutions reported to account holders and IRS (through Forms 1099). For items subject to substantial third-party information reporting, IRS is able to use automated processes to address noncompliance. The automated underreporter program, through which IRS matches amounts reported on tax returns with amounts reported on information returns submitted by third parties, is one such process. This computer matching program allows IRS to identify discrepancies between tax returns and information returns, and propose automatic changes to taxpayers. For items with little to no third-party information reporting, IRS must rely on more resource-intensive methods, such as correspondence or face-to- face examinations, to address noncompliance. While these examinations may be started by reviewing specific tax return line items, they may also be expanded to cover other areas of the tax returns if there are indications of misreporting in areas of the return not previously identified. However, it is harder for IRS to detect noncompliance in areas with little third-party information reporting. IRS’s budget declined by about $2.6 billion (18.8 percent) from fiscal years 2011 through 2019, and IRS’s budget for fiscal year 2019 is less than its fiscal year 2000 budget, after adjusting for inflation (see figure 4). Since fiscal year 2011, IRS staffing has fallen from 95,544 full-time equivalent employees to an estimated 75,676 in fiscal year 2019, a 20.8 percent reduction. At the same time, IRS faces increasing responsibilities, such as implementing relevant aspects of Public Law 115-97, which included significant changes to corporate and individual tax law. IRS also faces ever-evolving and significant challenges protecting taxpayer information, preventing identity theft and fraud, and modernizing an aging technology infrastructure. We previously reported that available staff has been a key factor in IRS decisions to scale back a number of program activities, such as examining tax returns, according to IRS officials. Our analysis of IRS data shows the rate of individual returns audited has declined between fiscal years 2011 and 2018 (see figure 5). Reducing examinations can reduce revenues collected through such enforcement action, and may indirectly reduce voluntary compliance. The federal tax system contains complex rules that may be necessary to appropriately target tax policy goals, such as providing benefits to specific groups of taxpayers. However, this complexity imposes a wide range of recordkeeping, planning, computing, and filing requirements upon taxpayers. For example, taxpayers who receive income from rents, self- employment, and other sources may be required to make complicated calculations and keep detailed records. This complexity can lead to errors and underpaid or overpaid taxes. Complexity, and the lack of transparency that it can create, can also exacerbate doubts about the tax system’s integrity. Tax expenditures—tax credits, deductions, exclusions, exemptions, deferrals, and preferential tax rates estimated by the Department of the Treasury to reduce tax revenue by about $1.38 trillion in fiscal year 2018—can add to tax code complexity. In part, this is because taxpayers must learn about, determine their eligibility for, and choose between tax expenditures that may have similar purposes. For example, as we reported in 2012, about 14 percent of filers in 2009 (1.5 million of almost 11 million eligible returns) did not claim an education credit or deduction for which they appeared eligible. The complexity involved with tax expenditures may be acceptable if they achieve their intended purposes. However, in many cases, their effectiveness is questionable or unknown. With some exceptions, tax expenditures generally are not subject to reauthorization and the annual congressional budget processes. We have recommended greater scrutiny of tax expenditures since 1994, as periodic reviews could help determine how well specific tax expenditures achieve their goals, and how their benefits and costs (including complexity) compare to those of other programs with similar goals. Such actions would help facilitate oversight and accountability of tax expenditures more in line with the performance management and reporting requirements of other federal programs. Paid tax return preparers and tax software developers help taxpayers navigate the complexities of the tax code. However, some paid preparers may introduce their own mistakes. For example, in a limited study in 2014, we found that seven of 19 preparers who completed returns for our undercover investigators made errors with substantial tax consequences while, only two preparers calculated the correct refund amount. Likewise, using NRP data, which are statistically representative, we estimated that 60 percent of returns prepared by preparers contained errors. IRS’s overall approach to reducing the tax gap consists of improving services to taxpayers, and enhancing enforcement of the tax laws. In spite of these efforts, the percentage at which taxpayers pay their taxes voluntarily and on time has remained relatively constant over the past three decades. Our past work has demonstrated that no single approach will fully and cost effectively address noncompliance since the problem has multiple causes and spans different types of taxes and taxpayers. In light of these challenges, we have made numerous recommendations to IRS that have not yet been implemented, as well as matters for congressional consideration. For example, in our most recent high-risk update, we highlighted various actions IRS should take to improve enforcement of tax laws and reduce the tax gap. Strategy for using compliance data. Developing and documenting a strategy that outlines how IRS will use data to update compliance strategies could help address the tax gap. For example, a strategy that outlines how IRS plans to use NRP data to update compliance programs and approaches would help IRS determine resource trade- offs and more fully leverage the investment it makes in compliance research, while providing Congress with a better understanding of the merits of the research it is being asked to fund. Voluntary compliance goal. A long-term, quantitative goal for improving voluntary compliance may provide IRS with a concrete target the agency can use in fulfilling its mission. Without a quantitative goal, it will be more difficult for IRS to determine the success of its strategies, adjust its approach when necessary, and remain focused on results, especially since factors that affect compliance change over time. Analyzing employment tax NRP study results. Developing and documenting plans to assess its NRP employment tax study results would help IRS (1) identify areas of noncompliance, (2) devise actions to address such noncompliance, and (3) update its employment tax gap estimate. Without completed analysis of the NRP employment tax study results, IRS risks using outdated data to make decisions about compliance and areas of the tax gap to pursue. Leveraging the Return Review Program. IRS’s Return Review Program (RRP) is a tool to detect and select potentially fraudulent returns to prevent the issuance of invalid refunds. Evaluating the costs and benefits of expanding RRP to analyze individual returns not claiming refunds could support other enforcement activities by streamlining the detection and treatment of other types of noncompliance and fraud. Given that the tax gap has been a persistent issue, reducing it will also require targeted legislative actions, such as those we highlighted in our 2019 high-risk update. Additional third-party information reporting. Expanding third-party information reporting to IRS could increase voluntary tax compliance. For example, reporting could be required for certain payments that rental real estate owners make to service providers, such as contractors who perform repairs on their rental properties, and for payments that businesses make to corporations for services. Enhanced electronic filing. Requiring additional taxpayers to electronically file tax and information returns could help IRS improve compliance in a resource-efficient way. For example, expanding the mandate for corporations to electronically file their tax returns could help IRS reduce return processing costs, select the most productive tax returns to examine, and examine fewer compliant taxpayers. Math error authority. Providing IRS with authority—with appropriate safeguards—to correct math errors and to correct errors in cases where information provided by a taxpayer does not match information in government databases, among other things, could help IRS correct errors and avoid burdensome audits and taxpayer penalties. Paid preparer regulation. Providing IRS with the authority to regulate paid tax return preparers could improve the accuracy of the tax returns they prepare. Chairman Neal, Ranking Member Brady, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact James R. McTigue, Jr. at (202) 512-9110 or mctiguej@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this testimony include Jeff Arkin, Assistant Director; Robyn Trotter, Analyst-in-Charge; A.J. Stephens; and Alicia White. Other staff who made key contributions to the reports cited in the testimony are identified in the source products. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The tax gap—the difference between tax amounts that taxpayers should have paid and what they actually paid voluntarily and on time—has been a persistent problem for decades. The tax gap estimate is an aggregate estimate of the five types of taxes that IRS administers—individual income, corporation income, employment, estate, and excise taxes. For each tax type, IRS attempts to estimate the tax gap based on three types of noncompliance: (1) underreporting of tax liabilities on timely filed tax returns; (2) underpayment of taxes due from timely filed returns; and (3) nonfiling, when a taxpayer fails to file a required tax return on time or altogether. This testimony discusses factors contributing to the tax gap and strategies to reduce it. This testimony is based on prior GAO reports on the tax gap and enforcement of tax laws, including those with open recommendations or matters for congressional consideration that could help reduce the tax gap. Enforcement of tax laws has been on GAO's High Risk List since its inception in 1990, and GAO has made various recommendations to IRS and suggestions to Congress to reduce the tax gap that have resulted in improvements. For example, GAO recommended that IRS consider comparing individuals' tax returns with the information educational institutions report to verify taxpayers' education tax benefits claims and suggested that Congress require brokers to report to both taxpayers and IRS the adjusted cost of the securities sold by taxpayers. These actions resulted in billions of dollars in additional revenue. The Internal Revenue Service's (IRS) latest tax gap estimate (2016) found that taxpayers voluntarily and timely paid about 81.7 percent of owed taxes for tax years 2008-2010, leaving an annual gross tax gap of $458 billion. IRS estimated a net tax gap—after late payments and enforcement actions—of $406 billion. GAO's work has found that three important factors contribute to the tax gap. Limited third party information reporting. The extent to which individual taxpayers accurately report their income is closely aligned with whether third parties (e.g., employers) report income (e.g., wages) to them and to IRS. IRS resource tradeoffs. IRS's budget and staffing levels have fallen over the past decade, and IRS faces increasing responsibilities, such as implementing Public Law 115-97—commonly known as the Tax Cuts and Jobs Act—which involved significant changes to tax law. Tax code complexity. The federal tax system contains complex rules that may be necessary to appropriately target tax policy goals; however, this can engender errors and lead to underpaid taxes. GAO's work has demonstrated that no single approach will fully and cost-effectively address noncompliance since the problem has multiple causes and spans different types of taxes and taxpayers. In light of these challenges, GAO has made numerous recommendations to IRS—some of which have not yet been implemented—such as developing and documenting a strategy that outlines how IRS will use data to update compliance approaches to help address the tax gap. Reducing the tax gap will also require targeted legislative actions. For example, expanding third-party information reporting could increase voluntary compliance and providing IRS with the authority to regulate paid tax return preparers could improve the accuracy of the tax returns they prepare.", "document_type": "gao"}
{"report": "Within DHS, ICE is responsible for immigration enforcement and removal operations. This entails, among other duties, identifying, arresting, and detaining foreign nationals for the administrative purpose of facilitating their appearance during removal proceedings, and for processing, and preparing them for removal from the United States, among other things. As such, ICE manages the nation’s immigration detention system, which houses foreign nationals detained while their immigration cases are pending or after being ordered removed from the country. ICE generally has broad discretion in determining whether to detain removable foreign nationals or release them under various conditions, unless the law specifies that detention is mandatory. Additionally, foreign nationals arriving at the U.S. border or a port of entry without valid entry documents and placed into expedited removal proceedings are required to be detained while awaiting an inadmissibility determination and, as applicable, any subsequent credible fear decision. Except in cases where detention is mandatory, ICE may release an individual pending the outcome of removal proceedings and has various release options for doing so, including the Alternatives to Detention program. While foreign nationals are detained, ICE is responsible for providing accommodations and medical care to individuals in detention with special needs or vulnerabilities, such as those who are pregnant. ICE’s December 2017 memo, Identification and Monitoring of Pregnant Detainees, sets forth policy and procedures to ensure pregnant detainees in ICE custody are identified, monitored, tracked, and housed in an appropriate facility. CBP is a component within DHS and the lead federal agency charged with a dual mission of facilitating the flow of legitimate travel and trade at our nation’s borders while also keeping terrorists and their weapons, criminals and their contraband, and inadmissible foreign nationals out of the country. CBP temporarily holds individuals to complete general processing and determine the appropriate course of action, such as transferring them to a court, jail, prison, or another agency; relocating them into ICE detention facilities; removing them from the country; or releasing them—as CBP has discretion to release individuals with a notice to appear in court. Within CBP, individuals, including pregnant women, could be held by Border Patrol or OFO. ICE detains individuals in both under-72-hour and over-72-hour detention facilities. Detention facilities may be for male only, female only, or both; and some are specifically reserved for family units (also known as family residential centers). ICE uses various types of detention facilities to hold detainees for more than 72-hours. These include ICE owned and operated detention facilities, also known as service processing centers, as well as facilities that ICE oversees but the day-to-day operations are generally run by another entity, as follows: contract detention facilities owned and operated by a private company under direct ICE contract that exclusively houses ICE detainees, facilities owned by state or local government or private entity, operating under an intergovernmental service agreement (IGSA), that exclusively houses ICE detainees or houses ICE detainees and other confined populations, and facilities owned by state or local government or private entity, operating under an intergovernmental agreement (IGA), or contract, with U.S. Marshals Service (USMS), that exclusively houses ICE detainees or houses ICE detainees and other confined populations. ICE detention facilities are generally required to adhere to one of four sets of detention standards. The detention standards vary depending on the contract or agreement. As we have previously reported, ICE’s detention standards are based on the American Correctional Association’s expected practices and have been updated when ICE identified issues of heightened concern or gaps in agency procedures. Some detention facilities used by ICE are not obligated to adhere to ICE’s detention standards—because, for example, ICE is a rider on the contract and the facility may be held to other standards. Further, on-site medical care may be directly provided by ICE Health Service Corps (IHSC) or other entities at these detention facilities. IHSC provides direct on-site medical services in 20 ICE facilities authorized to house detainees for over 72 hours. In addition to any applicable detention standards, IHSC staff must also adhere to IHSC policies. At detention facilities that are not staffed with IHSC personnel (non-IHSC facilities), medical care is provided onsite by local government staff or private contractors and overseen by IHSC. ICE inspects “authorized” detention facilities against detention standards and any applicable IHSC policies. Table 1 details information on each of the detention standards, the number of authorized facilities contractually obligated to each standard, the percent of the average daily population at each, and the presence of IHSC staff. CBP operates all of its short-term holding facilities and hold rooms, and does not utilize contract services for the management of individuals in CBP custody. In October 2015, CBP issued its first nationwide standards, which govern CBP’s interaction with detained individuals. The standards include requirements regarding transport, escort, detention, and search provisions, as well as care for “at-risk individuals”, which includes pregnant women. Given that CBP short-term facilities are intended to hold individuals for no more than 72 hours, CBP historically did not have on-site medical professionals at most of its facilities. However, as a result of surges in unaccompanied minors and families crossing the border, CBP issued a directive in January 2019 titled Interim Enhanced Medical Efforts (January 2019). According to the directive, enhanced medical services were needed to address growing public health concerns and mitigate risk to, and improve care for, individuals in CBP custody along the southwest border. The January 2019 directive was superseded by a December 2019 directive, Enhanced Medical Support Efforts, which also calls for medical support to mitigate risk to, and sustain enhanced medical efforts for persons in CBP custody along the southwest border. A related memo issued by the CBP Commissioner, titled CBP’s Expansion of Existing Medical Services Contracts and Expedited Deployment of Additional Contracted Medical Services Personnel to the Southwest Border, called for the expansion of CBP’s medical services contract to numerous Border Patrol facilities and OFO ports of entry along the southwest border. This effort is discussed later in our report. Number of pregnant women detentions. From calendar year 2016 through 2018, ICE had over 4,600 detentions of pregnant women. The number of detentions decreased from 1,380 in calendar year 2016 to 1,160 in 2017, and then increased to 2,098 in calendar year 2018 (see figure 1). Of the more than 4,600 detentions of pregnant women from calendar year 2016 through 2018, 32 percent involved pregnant women who were expedited removal cases and were subject to mandatory detention, including those that awaited a credible fear determination. Of the remaining detentions, 49 percent involved pregnant women who were deemed inadmissible and were either awaiting their hearing or an adjudication by an immigration judge, 11 percent involved pregnant women who had a final order of removal, and the remaining detentions (8 percent) involved various other immigration-related circumstances, such as those for which ICE was unable to obtain travel documents. Further, as we reported in December 2019, detentions of non-criminal pregnant women accounted for most of the total detentions of pregnant women each year (ranging from 91 to 97 percent). Length of detention. From calendar years 2016 through 2018, 68 percent of ICE detentions of pregnant women were for 7 days or less, 22 percent for 8 to 30 days, and 10 percent for more than 30 days, as shown in table 2. According to ICE officials, individual circumstances of each case dictate how long they detain a pregnant woman. For example, ICE may determine not to release a pregnant woman from ICE custody if her case is adjudicated quickly, she is ordered removed, and she is cleared to travel by a medical professional. Pregnancy outcomes. Our analysis of ICE data shows that from January 2015 through July 2019, 58 pregnant women in ICE custody experienced a miscarriage, two had an abortion, and one gave birth. Of those, 37 miscarriages and one birth involved women detained at IHSC-staffed facilities at the time of the outcome. Some of these women were in our study population of over 4,600 detentions from calendar years 2016 through 2018, but some were pregnant women detained in 2019. Detention facility. Our analyses of ICE data found that of the over 4,600 detentions of pregnant women, 78 percent of detentions of pregnant women were initially detained at an IHSC-staffed facility. See appendix II for more details on these data. According to ICE officials, pregnant women may first learn about their pregnancy when a test is performed during their intake into a detention facility. These over 4,600 detentions of pregnant women resulted in approximately 50,300 detention days with more than 66 percent of total detention days spent at IHSC-staffed facilities (see App. II). Some facilities may have a large number of detention days associated with the intake of pregnant women, but may not detain women for a long period of time before releasing or transferring them. For example, at a facility that had one of the largest number of detention days for pregnant women, officials stated that they generally release women once the pregnancy is confirmed. Further, according to ICE officials, ICE will try to transfer pregnant women from their initial detention facility to an IHSC- staffed detention facility or a family residential center—if she is part of a family unit—to ensure they are provided the appropriate accommodations and care. For example, ICE may transfer a pregnant woman awaiting a credible fear determination, as these cases may take longer to process and result in longer detention stays. However, an IHSC official also stated that ICE may detain pregnant women at non-IHSC facilities if ICE believes that the facility can provide the appropriate level of care. Nearly 70 percent of pregnant women’s detention days were spent at an IHSC- staffed facility or a family residential center. Contract detention facilities— both IHSC-staffed and non-IHSC—had the highest average number of days for the detention of pregnant women, as shown in table 3. Gestation of pregnancy. Of the 1,450 detentions of pregnant women for which gestation data were available, 49 percent were for women in their first trimester and 41 percent were for women in their second trimester at the time of intake. Ten percent were for women in their third trimester at the time of intake. Of the detentions involving pregnant women in their third trimester, 75 percent were released within one week or less, 9 percent between 8 and 15 days, and the remaining 16 percent between 16 and 90 days. According to ICE officials, ICE does not detain pregnant woman in their third trimester or a pregnant woman who is unlikely to be removed. However, officials stated that there are instances when it takes ICE time to gather information prior to making a custody determination— such as when it needs to collect criminal conviction data to making a custody determination—which could result in detained pregnant women who are nearing or in their third trimester. This is consistent with what ICE officials told us during our visits to facilities in all four locations—that they generally do not detain pregnant women in their third trimester. However, some explained, that pregnant women in their third trimester may be detained if, for example, they are subject to mandatory detention. Number of pregnant women. Because of CBP facilities’ short-term nature and limited on-site medical care, CBP does not routinely conduct pregnancy tests of women in their custody, and as such, has limited data on pregnancy. However, ICE data provide insight into CBP encounters with pregnant women. Specifically, our analysis of ICE data from calendar years 2016 through 2018 indicated that nearly 4,400 of ICE’s over 4,600 detentions of pregnant women resulted from CBP arrests. In addition, OFO and Border Patrol collected some data on women in their custody who reported being pregnant. OFO reported holding over 3,900 pregnant women from March 2018 through September 2019 at its ports of entry. At the two sectors where Border Patrol is required to collect such data, Border Patrol reported holding over 750 pregnant women in its facilities from March 2017 through March 2019. As shown in table 4, most of these women reported being in their second or third trimester. These women may have been transferred to ICE and may also be included in the count of pregnant women detained by ICE. In accordance with its January 2019 directive, Interim Enhanced Medical Efforts (January 2019), CBP developed a standardized health interview form that can be used by Border Patrol and OFO. The form includes a question about pregnancy and nursing which could allow for additional data on the number of women in CBP custody that report being pregnant. In December 2019, CBP officials told us that they distributed the form to its field locations. Pregnancy Outcomes. In addition, we reviewed CBP significant incident reports to determine if any pregnant woman encountered or held by CBP had experienced a birth, stillbirth, or miscarriage during calendar year 2015 through February 2019. Our analysis of CBP reports during this time frame found that pregnant women encountered or apprehended by CBP experienced 43 births, three miscarriages, and six stillbirths after being taken to the hospital by CBP. In some of these incidents, Border Patrol agents encountered pregnant women in the field and took them directly to the hospital. In these cases, the pregnant woman was not in a Border Patrol facility directly prior to being taken to the hospital. ICE has policies and detention standards that address a variety of pregnancy-related topics regarding the care of pregnant women, such as pregnancy testing requirements, the use of restraints, and prenatal care. However, we identified certain facility types that did not address all pregnancy-related topics in their policies or detention standards as of December 2019, which ICE is taking actions to address. Appendix III details ICE’s policies and detention standards related to the care of pregnant women in detention. For the purpose of our analysis, the facility type is based on contractually obligated detention standards and the presence of IHSC staff, as these factors dictate which detention standards the facility type is required to adhere to and whether IHSC policies apply. Specifically, we identified 16 topics related to the care of pregnant women and found that in most facility types, ICE had at least one policy or detention standard that addressed many of these topics. Further, we found that if the facility type had policies or detention standards in place regarding a specific topic on the care of pregnant women, at least one of the policies or detention standards generally aligned with recommended guidance from professional associations, NGOs, and federal agencies, (see app. IV for our summary of recommended guidance and associated examples). In addition, we found that from calendar years 2016 through 2018, 64 percent of the detentions of pregnant women were initially detained at the two facility types that had the most policies or detention standards related to each of the pregnancy topics, as of December 2019. Table 5 shows whether policies or detention standards at the various facility types addressed each of the 16 topics, as well as the associated number of detentions of pregnant women—based on the facility in which they were first detained and number of detention days from calendar years 2016 through 2018. ICE is taking numerous actions to address these gaps in its policies and detention standards. For example, according to ICE officials, ICE has updated, or is in the process of updating, its policies and detention standards, and these updates will address many of the gaps that we identified for the pregnancy-related topics. Specifically, ICE revised its 2000 NDS in December 2019 and the 2007 Family Residential Standards are under revision and will be sent to management for review in February 2020. According to IHSC officials, the revised standards will address all of the gaps we identified for 2007 Family Residential Standards and 2000 NDS facility types. Further, IHSC officials stated that they are revising IHSC’s Women’s Health Directive and guidance on care for chronic conditions to include required and recommended vaccines for pregnant women and HIV care, respectively—which will address these gaps at IHSC-staffed facilities. Finally, according to ICE officials, facility types operating under the 2008 PBNDS will be modified to either the 2019 NDS 2019 or 2011 PBNDS. In addition to these updates, in accordance with ICE’s December 2017 memo on Identification and Monitoring of Pregnant Detainees, ICE is to ensure pregnant detainees receive appropriate medical care, and ensure detention facilities are aware of their obligations regarding directives and detention standards that apply to pregnant detainees, among other things. ICE has mechanisms for maintaining oversight of pregnant detainees, as required by policy. Specifically, ICE collects data to monitor the condition of pregnant women in its custody, and according to ICE officials, ensures that the facility can accommodate the woman. In addition, IHSC conducts weekly reviews that focus on high-risk pregnancies, pregnancies in the third trimester, and recent miscarriages. According to an IHSC official, ICE inspections can contribute to IHSC’s understanding of the care of pregnant women at a given facility. Further, although ICE officials stated that it does not have training dedicated to the care of pregnant women in ICE detention specifically, its basic training includes instruction on pregnant detainees. This training is in addition to the professional qualifications of medical staff onsite. CBP has some policies and standards regarding the care of pregnant women held in their short-term facilities. Specifically, CBP has national standards on the transport, escort, detention, and search of detainees, with specific requirements for pregnant women. For example, these standards state that barring exigent circumstances, CBP must not use restraints on pregnant detainees unless they have demonstrated or threatened violent behavior, have a history of criminal or violent activity or an articulable likelihood of escape exists. Further, Border Patrol and OFO have policies that address nutrition and special accommodations for pregnant women. See appendix V for more details on CBP policies related to pregnant women. Although these policies and national standards do not cover the full range of the 16 pregnancy-related care topics we identified, CBP facilities are designed for holding individuals for no more than 72 hours; therefore, CBP’s facilities are not equipped to provide long-term care. Specifically, CBP does not routinely conduct pregnancy testing and historically it did not have on-site medical care at all its facilities. For the policies and standards that CBP does have in place regarding pregnant women, we found that they generally aligned with the recommended guidance from expert and professional organizations. In addition to policies that direct the care of pregnant women, although CBP does not have training dedicated to the care of pregnant women specifically, CBP provides initial and annual refresher training on its national standards for the transport, escort, detention, and search of detainees, which includes requirements for pregnant women. ICE uses various inspections for accessing facilities’ compliance with policies and detention standards—the frequency and focus of which vary. Some inspections also include pregnancy-related performance measures, such as a measure assessing whether a pregnancy test was performed at intake. We reviewed results from the five ICE inspections that address compliance with pregnancy-related policies and detention standards from 2015 through June 2019. These inspections vary in their scope and targeted facility types (see app. I for more details on each of these inspections). These inspections—along with available medical data—offer insight into the care of pregnant women. Two inspections include pregnancy-related performance measures, and compliance with these measures ranged from 53 to 100 percent, with most indicating 79 percent or more compliance. Specifically, one inspection of 129 ICE detention facilities—that included inspections of both IHSC-staffed and non-IHSC facilities—found that compliance was 91 percent or more for each of the six performance measures from December 2016 through March 2019, as shown below. Pregnancy testing performed at intake: 93 percent Pregnancy testing performed prior to x-rays or initiating medication: 100 percent Obstetrician-gynecologist (OB-GYN) consult ordered within 7 days of pregnancy confirmation: 98 percent Patient seen by OB-GYN within 30 days of pregnancy confirmation: Prenatal vitamins prescribed: 100 percent Screened for HIV, sexually transmitted infections, and viral hepatitis: Instances of non-compliance—which were 9 percent or less for each measure—occurred at 16 detention facilities subject to a range of detention standards. Three of these facilities were IHSC-staffed facilities, and 13 were non-IHSC facilities. IHSC documentation indicates that corrective actions are to be implemented to help address inspection findings. See appendix VI for details on the number of records reviewed during the inspections, and the compliance rates. Our analysis of available medical data and interviews with pregnant detainees showed similar findings regarding pregnancy testing at intake. Specifically, from calendar year 2016 through 2018, 92 percent of women in ICE detention facilities received a pregnancy test either the same day as intake to the facility or the next day. This could include women who arrived at a detention facility in the evening and are tested the next day. Of the remaining, 3 percent were tested within 2 to 3 days of intake, 4 percent were tested between 4 days and 2 weeks, and 2 percent were tested after 2 weeks of being detained. According to the 10 pregnant women we interviewed who were detained at 3 ICE detention facilities we visited, all 10 stated that they received a pregnancy test when they arrived at the facility or within the same day. For the second inspection that included performance measures related to the care of pregnant women at IHSC-staffed facilities, overall compliance was 79 percent or more for most of the nine performance measures from fiscal years 2015 through 2018. The following shows the minimum level of overall compliance for all facilities during this timeframe. OB-GYN consult ordered and documented within 7 days of pregnancy Patient seen by OB-GYN within 30 days: 92 percent Prenatal vitamins prescribed: 95 percent Detainee education documented at each encounter: 79 percent Records reviewed by provider after OB appointment: 79 percent Proper diet ordered: 86 percent Appropriate labs ordered if not obtained from OB-GYN: 79 percent Pregnant patient screened for HIV, sexually transmitted infections, and viral hepatitis: 81 percent Hepatitis B vaccine offered: 53 percent However, for one measure—whether the Hepatitis B vaccine was offered—compliance was 53 percent. ICE officials stated that this performance measure reflects recommended practices but is not specifically required by policy or detention standards. According to ICE officials, any issues identified during IHSC inspections are handled locally at the field level through facilities’ quality improvement processes, which includes developing corrective action plans. See appendix VI for the average annual compliance for each measure from fiscal years 2015 through 2018. Our analysis of available medical data for IHSC-staffed facilities and interviews with pregnant detainees and NGOs provides additional perspectives regarding these issues on the care of pregnant women. Specifically, our analysis of ICE data showed 422 detentions in which a pregnant woman was in an IHSC-staffed facility at some point received at least one referral to an OB or OB-GYN between calendar year 2016 and 2018. Based on ICE’s performance measures, pregnant women are to receive an OB-GYN referral within 7 days of pregnancy confirmation— although available data showed that most pregnant women were being released from detention within 7 days. In addition, our analysis of ICE data showed that detentions in which a pregnant woman was in an IHSC- staffed facility at some point were assigned certain special needs, such as a special diet (1,245), lower bunk (113), no heavy lifting (87), and limitations on the use of restraints (316). In addition, all 7 of the pregnant women we spoke with in IHSC-staffed detention facilities said that they received appropriate accommodations, such as a lower bunk and blankets. Similarly, 6 of the 7 pregnant women we spoke with at IHSC-staffed facilities said that they were provided proper nutrition and snacks. The other pregnant woman did not discuss the adequacy of the nutrition she was provided. In addition, both of these two inspections provided insights into OB-GYN referrals and prenatal vitamins that were generally similar to the information we obtained from pregnant detainees at the locations we visited. Specifically, the above inspections indicated 75 to 98 percent compliance on performance measures related to access to OB-GYN care. Eight of the 10 pregnant women we spoke with in ICE detention did not express concerns about access to OB-GYN when asked about the sufficiency of medical care. However, two stated that they would like more timely access to an OB-GYN, and they did not know when their appointments would occur. In addition, representatives from three NGOs stated that they heard concerns about pregnant women not having access to OB-GYN care or prenatal vitamins. Further, the above inspections indicated 95 to 100 percent compliance on performance measures related to prescribing prenatal vitamins, and all 10 of the pregnant women we spoke with in ICE detention said that they were provided prenatal vitamins. Although they did not have specific performance measures, three additional inspections identified 19 findings related to the care of pregnant women. All of the findings occurred at non-IHSC facilities. Three of the 19 findings indicated that medical care was not provided or offered. For example, one pregnant woman was not offered a mental health assessment after reporting that she had a miscarriage at a prior facility. Seven included a recommendation to provide additional medical care, such as pregnancy testing. Four indicated insufficient documentation, such as medical records that were not transferred between facilities, or no documentation that pregnancy testing had occurred. Five indicated that a required policy did not exist or did not specify the required standards of care. All but one of the facilities inspected took corrective actions to address the findings. For example, one inspection found that the facility’s initial health assessment form did not address pregnancy testing. In response, the facility updated its intake screening form to include pregnancy testing. ICE determined that the facility that did not implement corrective actions to address deficiencies identified during the inspection would not be used for the detention of ICE detainees. See appendix VI for additional information on each deficiency, recommendation, and corrective action. Additionally, our review of available data and interviews with pregnant detainees and officials at the locations we visited provided insight into issues related to segregation and the use of restraints—generally finding that these were rarely used. Specifically, our review of ICE data identified two pregnant women who were initially detained from 2015 through 2018, and segregated at some point during their detention—one for 8 days and one for over 4 months. In both cases, ICE reported the reason for the segregation was that the detainee was a threat to the facility’s security. Further, all 10 of the pregnant women we interviewed stated that they had not been segregated, and all the detention officials we interviewed at the four locations we visited stated that they were not aware of any instances of pregnant women being segregated. Similarly, none of the 10 pregnant detainees reported being placed in restraints, and the officials we interviewed at the four locations generally stated that pregnant women are not to be restrained except in extreme circumstances, such as risk of violence or escape—which is consistent with ICE policies and standards. One official said that he was aware of an incident where a pregnant woman was restrained when she attempted to harm herself and her child. In addition, officials from five local organizations or coalitions we spoke with stated that they had not heard concerns about instances of the use of restraints or segregation. CBP generally relies on offsite care for pregnant women, and as a result, has limited available information on care CBP provided to pregnant women. However, they have efforts underway to enhance its medical support at selected facilities. As previously discussed, CBP facilities are designed for short-term care, and CBP does not routinely administer pregnancy tests and generally did not have on-site medical personnel. According to CBP officials, they typically refer individuals to local medical providers in their area, as appropriate and for all emergent or serious issues—including concerns presented by pregnant women. In addition, if CBP needed to provide a pregnancy test to a woman in its custody, it would take the woman to an offsite medical provider. Our analyses of available data indicate that CBP took pregnant women for a hospital visit or admission at least 168 times from 2015 through 2018. See table 6 for additional information. Ninety-nine percent of these hospital trips involved Border Patrol, while the remaining 4 percent involved OFO. Although CBP generally relies on offsite care for pregnant women, CBP established some on-site medical care and has efforts underway to enhance its medical support at additional Border Patrol facilities and OFO ports of entry. Specifically, one port of entry and three Border Patrol facilities established on-site medical care in 2013 and 2015, respectively. CBP officials at one of these locations told us that they developed on-site medical care based on the volume of crossings, as well as the operational costs for transporting individuals to offsite medical facilities and performing hospital watches. Subsequently, CBP’s January 2019 memo regarding enhanced medical efforts at CBP facilities included efforts to expand medical support. According to a senior CBP official, the agency had staffed more than 40 Border Patrol facilities and OFO ports of entry along the southwest border with on-site contracted medical care, as of January 2020. According to CBP officials, contracted medical staff provide enhanced medical support through initial health intake interviews, medical assessments, diagnosis, treatment, referral, and follow up for persons in custody, including pregnant women. CBP officials stated that they will continue to rely on offsite care to provide emergency or advanced care. DHS has various processes to obtain and address the hundreds of medical care complaints it receives annually. Specifically, an individual can file a complaint directly to facilities, ICE, CBP, and other DHS entities, including the Office of Inspector General and Office for Civil Rights and Civil Liberties (CRCL). We identified 107 unique complaints that detainees, family members, NGOs, or other parties submitted to various entities from January 2015 through April 2019—54 that involved ICE’s care of pregnant women, 50 that involved CBP, and 3 that involved both. As shown in figure 2, some of these complaints were under investigation as of August 2019, and some were substantiated; however, in most cases there was not enough information for the investigating agency to determine if proper care had been provided, among other things. Regarding the complaints against ICE, the most common type was that ICE allegedly did not provide medical care, or that the medical care was not quality or timely. See appendix VIII for additional information about the number and types of complaints submitted. Eleven of the 54 complaints against ICE remained open as part of an on- going investigation, while the remaining 43 were closed. Of the 43 complaints that were closed: An investigation substantiated one complaint that prenatal vitamins had not been provided at an IHSC-staffed facility. In response, ICE reported taking actions to address the complaint. Investigations partially substantiated one complaint regarding delays in medical care being provided. According to ICE, the delays had resulted from the time required to get medication approved. In response to the complaint, ICE reported coordinating with the facility to address the issues identified. Investigations found that 18 complaints were unsubstantiated. For example, ICE’s review of medical records found that appropriate care had been provided. For the remaining 23 closed complaints, the complaint was not substantiated or unsubstantiated for a variety of reasons. For 11 complaints, the investigating agency determined that it did not have enough information to conduct an investigation, or the agency investigated the complaint but did not have enough information to establish whether the complaint was substantiated or unsubstantiated. For example, the allegation did not contain detailed biographical information, medical records did not contain enough information, or the detainee had been released and the agency could not follow-up. For the remaining 12 complaints, agency documentation did not clearly specify whether the complaint was substantiated or unsubstantiated. Regarding complaints against CBP, the most common type was that pregnant women had allegedly been physically, verbally, or otherwise mistreated. See appendix VIII for additional information about the number and types of complaints submitted. Of the 50 complaints against CBP, four remained open as part of an on- going investigation, while the remaining 46 were closed. Of the 46 complaints that were closed: An investigation substantiated one complaint that a Border Patrol agent violated social media policy by posting a picture and information about a pregnant woman in custody. In response, CBP reported that the employee was suspended for two days. Investigations found that five complaints were unsubstantiated, and one was partially unsubstantiated. For example, an investigation included a review of video footage at a port of entry, among other things, and found that excessive force had not been used. Eight complaints described an event that occurred, such as a miscarriage, but the complaint did not allege that mistreatment or improper care occurred. For the remaining 31 closed complaints, the complaint was not substantiated or unsubstantiated—for a variety of reasons. For 10 complaints, the investigating agency determined that it did not have enough information to conduct an investigation, or the agency investigated the complaint but did not have enough information to establish whether the complaint was substantiated or unsubstantiated. For the remaining 21 complaints, agency documentation did not clearly specify whether the complaint was substantiated or unsubstantiated. With regard to the three complaints that involved allegations against both ICE and CBP, one remained open as part of an on-going investigation, while the other two complaints were found to be unsubstantiated. We provided a draft of this report to DHS for review and comment. DHS provided comments, which are reproduced in appendix IX. DHS also provided technical comments, which we incorporated as appropriate. In addition, we provided relevant excerpts of the report to American College of Obstetricians and Gynecologists, American Correctional Association, and National Commission on Correctional Health Care for review. Officials from these entities provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees and the Acting Secretary of the Department of Homeland Security. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or goodwing@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix X. This appendix provides additional details on selected methodologies used to address our questions. Specifically, this includes information on our analyses of U.S. Immigration and Customs Enforcement (ICE) data and inspection findings and Department of Homeland Security (DHS) complaints used to address these questions: 1. What do available data indicate about pregnant women detained or held in DHS facilities? 2. What policies and standards does DHS have to address the care of pregnant women, and to what extent are they applicable across all facilities? 3. What is known about the care provided to pregnant women in DHS facilities? To address our first and third objectives, and provide context for our second objective, we reviewed data sources that ICE uses to track pregnant women in detention from calendar years 2016 through 2018 and matched these data with various ICE databases. We selected these years since ICE first collected data on all pregnant women beginning in June 2015, and 2018 was the last full year of available data for our audit. Specifically, we matched ICE Health Service Corps (IHSC) records for pregnant women detained during calendar years 2016 through 2018 with individual-level detention dataset the ICE Integrated Decision Support (IIDS) database to determine the total number of detentions of pregnant women, as well as the length of detention, facility location, case category status, arresting agency, gestation of pregnancy, when the pregnancy test was conducted, and whether there is an associated criminal conviction (criminality). To conduct our analyses, we matched pregnancy data to the IIDS detention data using alien number and excluded additional records we were unable to match. Because individuals may have multiple detentions, we compared the admission or book-in date from each data source with the book-in dates from the IIDS detention data, and excluded additional records with dates more than 30 days apart. ICE collected data for 1,437 pregnant detainees in 2016; 1,170 in 2017; and 2,126 in 2018. We excluded 60 of the unique pregnant detainee records for 2016; 20 for 2017; and 32 for 2018 because we were unable to match these records to the IIDS individual-level detention data using alien number and book-in date combinations. According to ICE officials, this may be due to data entry errors. As a result, our analyses are based on over 4,600 detainee records we were able to match: 1,377 for 2016; 1,150 for 2017; and 2,094 for 2018. In general, this was our study population, unless otherwise noted in the report. We also merged the detention data with data from ICE’s weekly facility list report, as of February 2019, to determine who owned and operated the facility, whether it was staffed by IHSC officials, and in what state the facility was located. Further, we merged additional IHSC data with our study population to determine the number of obstetrician-gynecologist referrals and numbers that were assigned certain special needs, such as a special diet, lower bunk, no heavy lifting, and limitations on the use of restraints. We also obtained and analyzed data from ICE’s Segregation Review Management System to determine if any of the pregnant women had been segregated. Finally, we analyzed ICE IHSC data on pregnancy outcomes—abortions, births, stillbirths, and miscarriages. These women who experienced such outcomes while detained may include the same women reported in our study population of more than 4,600 pregnant women detentions from calendar years 2016 through 2018, as well as pregnant women detained in calendar year 2015 and January through June 2019. We did not merge the outcome data with our other data sets, but were able to confirm that most of the outcomes were associated with alien numbers from the over 4,600 detentions in our study population. We assessed the reliability of the data used in each of our analyses by analyzing available documentation, such as related data dictionaries; interviewing ICE officials knowledgeable about the data; conducting electronic tests to identify missing data, anomalies, or potentially erroneous values; and following up with officials, as appropriate. We determined the data were sufficiently reliable for describing general information on pregnant women detained by ICE, as well as the care provided to them. To address our third objective, we analyzed reports and data from five ICE inspections that address compliance with pregnancy-related policies and detention standards from 2015 through July 2019—the most recent information available at the time of our review. We selected these inspections because they review some aspect of the care provided to pregnant women. Table 7 provides additional information on these inspections. As noted in the table, two of these inspections contained pregnancy- related performance measures. The remaining three inspections assess compliance and identified findings related to the care of pregnant women, but did not have specific performance measures. For the three inspections that did not contain performance measures, we categorized the nature of each finding, such as a recommendation to provide additional medical care. We developed these categories based on a content analysis of the inspection findings, which involved one analyst categorizing the finding and a second person verifying the categories. If there were differences in analyses, these were reconciled through discussion between the two analysts and a final determination of the appropriate category was made. We also analyzed ICE documentation on corrective actions that facilities reported taking to address inspection findings, and used ICE facility data to determine who provided medical care at these facilities. To determine the scope and any limitations of inspection reports and data, we spoke with agency officials responsible for managing these inspections and the data systems used for documenting results. We also reviewed relevant documentation, such as data dictionaries and inspection worksheets. We determined that these data were sufficiently reliable for our purposes of describing the results of inspections regarding the care of pregnant women in ICE custody. We reviewed and categorized complaints that detainees, family members, non-governmental organizations, or other parties submitted to various entities from January 2015 through April 2019—the latest available complaints at the time of our review—regarding ICE and CBP’s care of pregnant women. Specifically, we reviewed complaint data from DHS’s Office for Civil Rights and Civil Liberties (CRCL), DHS’s Office of Inspector General, and IHSC. We selected these complaint systems because, according to DHS officials, they contained relevant information on the care of pregnant women, could be queried in an electronic format, and minimized duplicate complaints across systems. We categorized each complaint based on a content analysis of the complaint narrative, which involved one analyst categorizing the complaint and a second person verifying the category. If there were differences in analyses, these were reconciled through discussion between the two analysts and a final determination of the appropriate category was made. We developed categories for 10 pregnancy outcomes, including births or miscarriages at a DHS facility or hospital, as well as 20 categories to describe the nature of the concerns, including physical mistreatment, use of restraints, or medical care not provided. The total number of concerns identified in our analysis exceeds the number of unique complaints filed because each unique complaint may identify more than one area of concern. We also used ICE facility data to determine, for example, who provides medical care at the facilities where the alleged events occurred. In addition, we analyzed agency documentation on the extent to which complaints could be substantiated, and any corrective actions that agencies and facilities reported taking to address complaints. To determine the scope and any limitations of the complaint information we received, we spoke with agency officials responsible for managing these complaint processes and the data systems used for documenting results. We also reviewed relevant documentation, such as user manuals for complaint systems. This appendix provides additional details on our analyses of U.S. Immigration and Customs Enforcement (ICE) data from calendar years 2016 through 2018 on (a) where pregnant women were initially detained and (b) facilities that had the largest number of detention days involving pregnant women. In particular, these analyses describe whether the facility has ICE Health Service Corps (IHSC) staff and who owns and operates the facility, based on contracts or agreements. Initial detention facility. Our analyses of ICE data found that of the over 4,600 detentions of pregnant women, in regards to IHSC presence, almost 78 percent of detentions of pregnant women were initially detained at an IHSC-staffed facility. Further, 51 percent were at service processing centers that are owned and primarily operated by ICE, all of which were also staffed by IHSC, as shown in table 8. According to ICE officials, many pregnant women first learn about their pregnancy when a test is performed during their intake into a detention facility. Although pregnant women were initially detained in various facility types—based on IHSC presence and who owns and operates the facility, most occurred in eight specific detention facilities located in three states. Specifically, of ICE’s over 4,600 pregnant women detentions from calendar year 2016 through 2018, 86 percent were initially detained in one of eight of these detention facilities—with one facility having 45 percent of the intakes of pregnant women. Facilities with the most number of detention days. For these over 4,600 detentions of pregnant women, ICE detained them for a total of almost 50,300 days from calendar year 2016 through 2018. Our analyses of ICE data found that of the 50,300 detention days of pregnant women, in regards to IHSC presence, 66 percent of these days were at an IHSC-staffed facility. Further, over half were at intergovernmental service agreement facilities—including family residential centers, as shown in table 9. Some facilities may have a large number of detention days associated with the intake of pregnant women, but these facilities may not detain women for a long period of time before releasing or transferring them. For example, at a facility that had one of the largest number of detention days for pregnant women, officials stated that they generally release women once the pregnancy is confirmed. Although pregnant women spent their detention days in various facility types—based on IHSC presence and who owns and operates the facility, most occurred in 19 specific detention facilities located in seven states. Specifically, of those days that pregnant women were detained by ICE, 89 percent of these days were in one of these 19 detention facilities. U.S. Immigration and Customs Enforcement (ICE) detention facilities and staff are subject to a variety of policies, including ICE-wide policy directives and memoranda, ICE Health Service Corps (IHSC) policies, and detention standards, as of December 2019. We categorized and summarized these policies and standards, as shown below. ICE-wide policies are directed at ICE staff and officers, and not to contractors or facility staff. The following ICE policies address pregnant detainees and ICE supervision of pregnant detainees: ICE Directive 11032.3: Identification and Monitoring of Pregnant Detainees (2017) ICE Directive 11065.1: Review of the Use of Segregation for ICE Detainees (2013) ICE Directive 11002.1: Parole of Arriving Aliens found to Have a Credible Fear of Persecution or Torture (2010) ICE Memorandum: Use of GPS Monitoring Devices on Persons who are Pregnant or Diagnosed with a Severe Medical Condition (2009) ICE ERO Policy 11155.1: Use of Restraints (2012) Enforcement and Removal Operations National Detainee Handbook (2016) These ICE-wide policies do not apply to contract or facility staff unless ICE modified the facility’s contract or if these are already included in the facility’s detention standards to which they are obligated. However, the National Detainee Handbook is a resource for detainees at detention facilities operating under ICE detention standards, excluding family residential centers. We categorized these policies and summarized them accordingly. Intake health screening inquiries about pregnancy. The policy refers to ICE’s responsibility to monitor detention facilities and ensure they meet national detention standard requirements to provide all newly admitted detainees an initial medical screening including pregnancy screening. ICE Directive 11032.3: Identification and Monitoring of Pregnant Detainees (2017) Provision of prenatal care. ICE supervisory staff have responsibilities to ensure that pregnant detainees receive appropriate medical care, including transfer to a different facility if necessary. ICE medical staff also have a responsibility to monitor the condition of pregnant detainees and communicate any concerns to supervisory staff. ICE Directive 11032.3: Identification and Monitoring of Pregnant Detainees (2017) Enforcement and Removal Operations National Detainee Handbook (2016) Segregation of pregnant women. ICE has a responsibility to monitor the use of segregation at detention facilities to ensure that they are adhering to detention standards. ICE Directive 11065.1: Review of the Use of Segregation for ICE Detainees (2013) Use of restraints on pregnant women. Officers should take reasonable precautions to avoid causing discomfort when transporting a restrained detainee. At processing sites or non-ICE detention facilities, ICE personnel shall follow local policies and procedures. ICE ERO Policy 11155.1: Use of Restraints (2012) Record keeping on pregnant women actions. ICE supervisors should ensure that ICE staff and contracted medical staff have processes to notify them of the arrival of a pregnant woman to a detention facility and ensure staff and facilities are aware of their obligations regarding pregnant detainees. IHSC staff are responsible for monitoring the condition of pregnant women while detained, as well as maintaining their medical records. Any instance of segregation of a pregnant woman must be documented in writing. ICE Directive 11032.3: Identification and Monitoring of Pregnant Detainees (2017) ICE Directive 11065.1: Review of the Use of Segregation for ICE Detainees (2013) IHSC policies are directed specifically toward IHSC staff at detention facilities where IHSC provides medical services. The following IHSC policies address pregnant detainees: ICE Directive 11772.2: Women’s Health Services (2017) ICE Directive 11741.4: Health Assessment (2016) ICE Directive 11742.2: Pre-Screening (2015) ICE Directive 11744.2: Intake Screening and Intake Reviews (2016) We categorized these policies and summarized them accordingly. Intake health screening inquiries about pregnancy. Intake screening includes pregnancy testing of women 10 to 56 years of age as well as questioning of pregnancy status. ICE Directive 11772.2: Women’s Health Services (2017) ICE Directive 11742.2: Pre-Screening (2015) ICE Directive 11744.2: Intake Screening and Intake Reviews (2016) Pregnancy testing at intake. Intake screening includes pregnancy testing of women 10 to 56 years of age and inquiry of reproductive health including previous pregnancies. ICE Directive 11772.2: Women’s Health Services (2017) ICE Directive 11741.4: Health Assessment (2016) ICE Directive 11744.2: Intake Screening and Intake Reviews (2016) Access to abortion. In the event of a threat to a woman’s life from carrying a pregnancy to term, or else in cases of rape or incest, ICE must bear the cost of a detainee’s decision to terminate a pregnancy; otherwise the woman must bear the cost. ICE should offer medical resources to support effective recovery and follow-up care. ICE Directive 11772.2: Women’s Health Services (2017) Provision of prenatal care. Pregnant women should be seen by medical providers at least once a month while detained. They should also be referred to an obstetric specialist, and their medical records shared with the specialist to facilitate care. ICE Directive 11772.2: Women’s Health Services (2017) ICE Directive 11741.4: Health Assessment (2016) ICE Directive 11744.2: Intake Screening and Intake Reviews (2016) Provision of postnatal care. A postpartum detainee must receive postnatal care from a medical provider, in consultation with an obstetric specialist, at least once a month. ICE Directive 11772.2: Women’s Health Services (2017) Mental health services and counseling for pregnant women. Any female detainee who gave birth, miscarried, or terminated a pregnancy within the last 30 days must receive a mental health evaluation, with the evaluation to occur no later than 72 hours after initial referral. ICE Directive 11772.2: Women’s Health Services (2017) Care for pregnant women with substance use disorder. Chemically dependent pregnant women are considered high-risk and should be referred to an obstetrician or other appropriate medical provider as soon as they are identified. ICE Directive 11772.2: Women’s Health Service (2017) ICE Directive 11744.2: Intake Screening and Intake Reviews (2016) Use of restraints on pregnant women. Pregnant detainees or those in postdelivery recuperation should not be restrained except in extraordinary circumstances that are documented by a supervisor or directed by a medical authority, whether in an ICE detention facility, in transport, or at a medical facility. Detainees in active labor or delivery can never be restrained. Even if restraints are used, a pregnant woman should never be restrained face down or on her back, or restrained with a belt that constricts the abdomen or pelvis. ICE Directive 11772.2: Women’s Health Service (2017) Record keeping on pregnant women actions. Intake screenings and assessments including pregnancy test results must be documented, as are risk factors for high risk pregnancies. Any use of restraints or request for abortion services must be documented. ICE supervisory staff must be notified within 72 hours of the arrival at a detention facility of a pregnant woman. ICE Directive 11772.2: Women’s Health Service (2017) ICE Directive 11741.4: Health Assessment (2016) ICE Directive 11744.2: Intake Screening and Intake Reviews (2016) Entities that have a contract or agreement with ICE to hold immigration detainees are generally contractually obligated to one of four sets of detention standards. These standards address a range of our pregnancy- related categories of care and vary by standard. 2000 ICE National Detention Standards (NDS) 2007 ICE Family Residential Standards (FRS) 2008 ICE Performance-Based National Detention Standards 2008 (2008 PBNDS) 2011 ICE Performance-Based National Detention Standards 2011 (2011 PBNDS) We categorized these standards and summarized them accordingly. The 2011 PBNDS standards received revision in 2016. Whether a 2011 PBNDS facility is contractually required to adhere to the 2016 revision is dependent upon the contract language negotiated in each agreement. Where appropriate, the summaries below note changes to policy as a result of those revisions. Intake health screening inquiries about pregnancy. 2008 PBNDS: Initial screening should be done within 12 hours of arrival and should inquire about the possibility of pregnancy. 2011 PBNDS: Initial screening should be done within 12 hours of arrival and should inquire about the possibility of pregnancy. In the 2016 revisions, the evaluation also includes a pregnancy test for women aged 18 to 56. Pregnancy testing at intake. 2008 PBNDS: Initial screening should be done within 12 hours of arrival and should inquire about the possibility of pregnancy. 2011 PBNDS: In the 2016 revisions, initial screening includes pregnancy testing of women 18 to 56. Access to abortion. 2011 PBNDS: If the life of the mother is endangered by carrying the fetus to term, or in the case of rape or incest, ICE will assume the costs to terminate the pregnancy. ICE shall arrange the transportation for the medical appointment, and to counseling services if requested in all cases, including those where rape, incest, or risk to life do not apply. Every facility, either directly or via contractor, must provide female detainees with access to counseling for pregnancy planning if the detainee wishes to receive an abortion. Provision of prenatal care. FRS: Female residents will have access to pregnancy management services including routine prenatal care 2008 PBNDS: Female detainees will have access to pregnancy management services including routine prenatal care 2011 PBNDS: Pregnant detainees will have access to pregnancy management services including routine prenatal care. They will also receive access to a specialist and receive a health assessment. The 2016 revisions note those actions should occur as soon as appropriate or within two working days. The 2016 revisions also give the medical provider authority to identify pregnant detainees’ special needs such as diet or housing requirements and inform all necessary staff and authorities. Provision of postnatal care. FRS: Female residents will have access to pregnancy management services including postpartum follow-up care. 2008 PBNDS: Female detainees will have access to pregnancy management services including postpartum follow-up care. 2011 PBNDS: Pregnant detainees will have access to pregnancy management services including postpartum follow-up care. After giving birth, receiving an abortion or miscarrying, mental health assessments should also be offered. Provision of perinatal/labor care. 2011 PBNDS: Pregnant detainees will have access to specialized care including labor and delivery. Mental health services and counseling for pregnant women. FRS: Pregnant females will have access to pregnancy management services that include counseling and assistance. 2008 PBNDS: Pregnant females will have access to pregnancy management services that include counseling and assistance. 2011 PBNDS: Pregnant detainees will have access to care including counseling and assistance. Detainees can also request transportation to religious, medical and social counseling when considering termination of a pregnancy. In 2016 revisions, intake screening should include education to female detainees about mental health services related to pregnancy and women’s health. Care for pregnant women with substance use disorder. 2008 PBNDS: Female detainees will have access to pregnancy management services that include addiction management. 2011 PBNDS: In 2016 revisions, all chemically dependent pregnant detainees are to be considered high risk and referred to an obstetrician or other provider capable of addressing their needs immediately. HIV care for pregnant women. 2011 PBNDS: Medical personnel shall provide all detainees diagnosed with HIV/AIDS medical care consistent with national recommendations and guidelines disseminated through the U.S. Department of Health and Human Services, the Center for Disease Control, and the Infectious Diseases Society of America. Prenatal vitamins. 2011 PBNDS: Pregnant detainees will have access to prenatal care including prenatal vitamins. Nutrition for pregnant women. NDS: Physicians may order snacks or supplemental feedings to increase protein or calories for reasons including pregnancy. In hold rooms, pregnant women should have regular access to snacks, milk, and juice. FRS: Physicians may order snacks or supplemental feedings to increase protein or calories for reasons including pregnancy. Pregnant women will have access to pregnancy management services that include nutrition. 2008 PBNDS: Physicians may order snacks or supplemental feedings to increase protein or calories for reasons including pregnancy. In hold rooms, pregnant women should have regular access to snacks, milk, and juice. Pregnant women will have access to pregnancy management services that include nutrition. 2011 PBNDS: Physicians may order snacks or supplemental feedings to increase protein or calories for reasons including pregnancy. In hold rooms, pregnant women should have regular access to snacks, milk, and juice. Pregnant women will have access to pregnancy management services that include nutrition. Special consideration is given to pregnant women when providing meals and snacks during transportation. In the 2016 revisions, the medical provider is responsible for identifying special needs of pregnant detainees, including diet, and notifying all necessary staff. Special accommodations for pregnant women. 2008 PBNDS: In hold rooms, pregnant women will have access to temperature appropriate clothing and blankets and may, depending on facility, have access to bunks, cots, or beds, normally not kept in hold rooms. 2011 PBNDS: In hold rooms, pregnant women will have access to temperature appropriate clothing and blankets and may, depending on facility, have access to bunks, cots, or beds, normally not kept in hold rooms. Pregnant detainees should also have access to lactation services in the facility. In the 2016 revisions, the medical provider is responsible for identifying special needs of pregnant detainees and notifying all necessary staff. Segregation of pregnant women. 2011 PBNDS: In the 2016 revisions, it is stated that women who are pregnant, post-partum, recently had a miscarriage, or recently had a terminated pregnancy should as a general matter not be placed in a Special Management Unit. In very rare situations, a woman who is pregnant, postpartum, recently had a miscarriage, or recently had a terminated pregnancy may be placed in a Special Management Unit as a response to behavior that poses a serious and immediate risk of physical harm, or if the detainee has requested to be placed in protective custody administrative segregation and there are no more appropriate alternatives available. Also in the 2016 revisions, a facility administrator must notify the appropriate field office director in writing as soon as possible, but no later than 72 hours any time a pregnant woman or one who recently had a miscarriage is placed in segregation. In all cases, in the 2016 revisions, this decision must be approved by a representative of the detention facility administration, in consultation with a medical professional, and must be reviewed every 48 hours. Use of restraints on pregnant women. NDS: Pregnant detainees should be given special consideration if restrained as a result of a physical encounter. A medical professional should be consulted immediately in the aftermath, and the detainee examined. Pregnant detainees should be restrained in such a way as to avoid harming the fetus such as not restraining face down. FRS: Medical staff will advise on the necessary precautions to take when restraining a pregnant detainee and restraint should be done only when other methods have been tried or are impracticable. 2008 PBNDS: Medical staff will advise on the necessary precautions to take when restraining a pregnant detainee. Pregnant detainees should be restrained in such a way as to avoid harming the fetus such as not restraining face down. 2011 PBNDS: A pregnant detainee is not to be restrained except in truly extraordinary circumstances. Even then, it must be documented by a supervisor and directed by a medical authority. Women in active labor or delivery can never be restrained, and if restrained, the detainee should never be face down, on her back, or restrained with a belt that constricts the area of pregnancy. Record keeping on pregnant women actions. NDS: The medical provider of a facility will notify the ICE officer in charge whenever a pregnant detainee is identified and any use of force or application of restraints on a detainee should be followed by a medical examination, and its results documented. FRS: The medical provider of a facility will notify the ICE facility administrator whenever a pregnant detainee is identified. A treatment plan should be developed for any detainee requiring close medical supervision, and approved by the appropriate physician or other medical provider. 2011 PBNDS: When a detainee is pregnant, an alert is notified in their medical record and the facility administrator will receive notice. If a detainee is transferred, it is the administrator’s responsibility to inform ICE of the medical alert. Any use of restraints requires documented approval, including in the detainee’s detention and medical files and guidance from the on-site medical authority. A request to terminate a pregnancy must be documented in the medical file and signed by the detainee. In the 2016 revisions, ICE supervisory staff must be informed within 72 hours when a pregnant detainee is identified. Numerous professional associations, non-governmental organizations, and federal agencies have issued guidance on care to be provided to pregnant women. Specifically, we reviewed the following guidance: American Civil Liberties Union: Worse than Second-Class: Solitary Confinement of Women in the United States (2014) American College of Obstetricians and Gynecologists: Committee Opinion: Health Care for Pregnant and Postpartum Incarcerated Women and Adolescent Females (2016) Guidelines for Perinatal Care, Eighth Edition (2017) American Correctional Association Performance-Based Standards and Expected Practices for Adult Correctional Institution, 5th Edition Joint Public Correctional Policy on the Treatment of Opioid Use Disorders for Justice Involved Individuals (2018) Joint Statement on the Federal Role in Restricting the Use of Restraints on Incarcerated Women and Girls during Pregnancy, Labor, and Postpartum Recovery National Commission on Correctional Health Care (NCCHC): Position Statement: Restraint of Pregnant Inmates (2015) Position Statement on Solitary Confinement (Isolation) (2016) Position Statement on Breastfeeding in Correctional Settings (2018) Standards for Health Services in Jails (2018) Sufrin C., Pregnancy and Postpartum Care in Correctional Settings, National Commission on Correctional Health Care, Clinical Resources Series. (2018) National Women’s Law Center: Women Behind Bars: A state-by-state report card and analysis of federal policies on conditions of confinement for pregnant and parenting women and the effect on their children (2010) United Nations Rules for the Treatment of Women Prisoners and Non- custodial Measures for Women Offenders (the Bangkok Rules) (2010) U.S. Department of Homeland Security (DHS): Report of the DHS Advisory Committee on Family Residential Centers (2016) U.S. Department of Justice, Bureau of Justice Assistance: Best Practices in the Use of Restraints with Pregnant Women and Girls Under Correctional Custody (2014) U.S Department of Justice Report and Recommendations Concerning the Use of Restrictive Housing (2016) Because the specificity of the guidance varies across entities, we summarized the recommended guidance for our report purposes. For example, guidance on nutrition may range from calling for additional meals for pregnant women to more specifically outlining extra caloric and dietary needs. Our summary statement for each of the pregnancy-related topics is included below, along with examples from relevant recommended guidance. Intake health screening inquiries about pregnancy. Summary of recommended guidance: The sources that have guidance generally agree that intake health screenings should include inquiry regarding pregnancy and related conditions. Example: “Screening is performed on all inmates upon arrival at the intake facility…The receiving screening form…inquires as to the inmate’s…possible, current, or recent pregnancy…” – NCCHC Standards for Health Services in Jails (2018) Pregnancy testing at intake. Summary of recommended guidance: Sources that have guidance generally agree that pregnancy testing should be conducted on newly detained women of childbearing age, but some provide additional guidance on when this should be done, and this may vary. Example: “All women at risk for pregnancy should be offered a pregnancy test within 48 hours of admission…A simple approach would be to offer pregnancy testing to all women under the age of 55.” – Pregnancy and Postpartum Care in Correctional Settings (2018) Example: “…medical providers should continue to offer pregnancy tests to every female of child-bearing age who is newly detained…” – Report of the DHS Advisory Committee on Family Residential Centers (2016) Access to abortion. Summary of recommended guidance: Sources that have guidance generally agree abortion services should be offered to detained pregnant women, with one source providing additional details, including swift facilitation of a woman’s choice of termination and non- interference of outside bodies in the decision. Example: “Pregnancy termination is generally to be performed as safely and as early in pregnancy as possible…Termination of pregnancy should not depend on whether or not the specific procedure is available on site. Each woman will decide what option to choose…this decision is to be made without undue interference by outside bodies, including governmental bodies.” – Report of the DHS Advisory Committee on Family Residential Centers (2016) Provision of prenatal care. Summary of recommended guidance: Sources that have guidance generally agree that some form of prenatal care should be provided to detained pregnant women, but differ on the level of specificity for the standard of care, from stating simply that prenatal care be provided to specifying requirements including regularly scheduled obstetric care and access to 24-hour emergency care. Example: “Incarcerated women who wish to continue their pregnancies should have access to readily available and regularly scheduled obstetric care, beginning in early pregnancy and continuing through the postpartum period. Incarcerated pregnant women also should have access to unscheduled or emergency obstetric visits on a 24-hour basis.” – American College of Obstetricians and Gynecologists Committee Opinion: Health Care for Pregnant and Postpartum Incarcerated Women and Adolescent Females (2016) Example: “Prenatal care in correctional facilities must reflect national standards, including visit frequency with a qualified prenatal care provider, screening and diagnostic tests, and referrals for complications.” – Pregnancy and Postpartum Care in Correctional Settings (2018) Provision of postnatal care. Summary of recommended guidance: Sources that have guidance generally agree that the provision of postnatal care be provided to women who give birth. However, they vary in their specifics. For example, some specifically state that lactation service or postnatal birth control should be provided. One source also recommends specific forms of accommodation to aid postnatal recovery. Example: “…appropriate accommodations should be made, such as allowing women to rest when needed…Discharge instructions from the hospital, which may include postpartum blood pressure monitoring or diabetes screening, should be adhered to.” – Pregnancy and Postpartum Care in Correctional Settings (2018) Example: “Allow immediately postpartum women to breastfeed their babies and have lactation support services from the hospital.” – NCCHC Position Statement on Breastfeeding in Correctional Settings (2018) Provision of perinatal/labor care. Summary of recommended guidance: Sources that have guidance generally agree a pregnant woman should be transported to a hospital if there are signs of labor. Some sources state that detention staff be trained in emergency delivery in the event of a delivery occurring in the facility, away from professional care. Example: “Due to the time necessary to arrange transport to a nearby hospital, there is a low threshold to send pregnant inmates out for evaluation of a labor when signs or symptoms of labor or ruptured membranes are present… Any facility that houses pregnant women should have an emergency delivery kit available on-site, and health staff should be trained in its use in the event that a delivery occurs in the facility.” – Pregnancy and Postpartum Care in Correctional Settings (2018) Example: “Having a preexisting arrangement to have the babies of incarcerated women delivered at a local hospital reduces confusion and uncertainty when a woman goes into labor.” – National Women’s Law Center Women Behind Bars: A state-by-state report card and analysis of federal policies on conditions of confinement for pregnant and parenting women and the effect on their children (2010) Mental health services and counseling for pregnant women. Summary of recommended guidance: Sources that have guidance generally agree that pregnant and postpartum women should have access to mental health/counseling services. Example: “Pregnant inmates are given comprehensive counseling and care in accordance with national standards and their expressed desires regarding their pregnancy.” – NCCHC Standards for Health Services in Jails (2018) Care for pregnant women with substance use disorder. Summary of recommended guidance: Sources that have guidance generally agree that addicted pregnant women should have access to screening and specialized addiction-treatment programs. Example: “Screening for drug and alcohol use is a first step and is followed with referral to treatment. For women who report opiate use, the standard of care is not to detoxify from opiates during pregnancy due to the fetal risks of withdrawal. Rather the standard of care is to provide…methadone or buprenorphine…” – Pregnancy and Postpartum Care in Correctional Settings (2018) Example: “The standard of care for pregnant women with [opioid use disorder] is and should therefore be offered/continued for all pregnant detainees and incarcerated individuals.” – Joint Public Correctional Policy on the Treatment of Opioid Use (2018) HIV care for pregnant women. Summary of recommended guidance: Sources that have guidance generally agree that pregnant women should have access to testing and treatment of HIV for the benefit of both the mother and child. Example: “The Centers for Disease Control and Prevention recommends universal opt-out HIV screening for pregnant women; with early detection, prevention of mother-to-child transmission can be accomplished…” – Pregnancy and Postpartum Care in Correctional Settings (2018) Vaccinations for pregnant women. Summary of recommended guidance: Sources that have guidance generally agree that vaccines recommended for pregnant women be provided to detainees in accordance with accepted medical guidelines. Example: “Current recommendations are that all pregnant women should be vaccinated with the flu vaccine during flu season and tetanus, diphtheria, and pertussis during the third trimester, regardless of whether they were vaccinated outside of pregnancy.” – NCCHC Standards for Health Services in Jails (2018) Example: “Vaccines related to pregnancy should be offered pursuant to CDC guidelines…” – Report of the DHS Advisory Committee on Family Residential Centers (2016) Prenatal vitamins. Summary of recommended guidance: The sources that have guidance generally agree that prenatal vitamins should be provided to pregnant women, and some sources state that prenatal vitamins should be provided to breastfeeding women. Example: “Pregnant women must also receive prenatal vitamins that contain, among other essential vitamins and minerals, 400mcg to 800mcg of folic acid... Women with documented anemia (hemoglobin<11) should receive additional iron supplementation.” – Pregnancy and Postpartum Care in Correctional Settings (2018) Example: “Appropriate nutrition and prenatal vitamins should be given to lactating women…” – NCCHC Standards for Health Services in Jails (2018) Nutrition for pregnant women. Summary of recommended guidance: Sources that have guidance generally recommend special nutrition regimens for pregnant women, with varying degrees of specificity, ranging from recommending the use of supplements broadly to specifying required nutrients such as folic acid and calcium and extra calories in the form of additional meals, larger meals, or food between meals, and in some cases specifying that these requirements also apply for postpartum women. Example: “Pregnant and postpartum women have additional nutritional needs and should be counseled on the importance of adequate nutrition. Diets provided by correctional institutions should be specialized to the women’s needs and be rich in whole grains, calcium, and fruits and vegetables. In the second and third trimesters, women require an additional 300 calories per day…” – Pregnancy and Postpartum Care in Correctional Settings (2018) Special accommodations for pregnant women. Summary of recommended guidance: Sources that have guidance generally agree that accommodations should be provided to pregnant women. Some sources specify accommodations such as appropriate programming and hygiene for pregnant women and nursing mothers, appropriately adjusted work assignments and exercise, and bottom bunks. Example: “Activity for pregnant women must take into account the physical constraints of being in a correctional facility. All pregnant women must have a bottom bunk so that they do not risk falling from a top bunk. Certain work assignments may be inappropriate…Work assignments should be adjusted accordingly. In the absence of medical or obstetric complications, 30 minutes or more of moderate exercise a day on most, if not all, days of the week is recommended.” – Pregnancy and Postpartum Care in Correctional Settings (2018) Segregation of Pregnant Women. Summary of recommended guidance: Sources that have guidance generally agree that pregnant women should not be placed in segregation, though some suggests this could be necessary in certain cases. Example: “Women who are pregnant, who are postpartum, who recently had a miscarriage, or who recently had a terminated pregnancy should not be placed in restrictive housing…In very rare situations, a woman who is pregnant, is postpartum, recently had a miscarriage, or recently had a terminated pregnancy may be placed in restrictive housing as a temporary response to behavior that poses a serious and immediate risk of physical harm…” – U.S Department of Justice Report and Recommendations Concerning the Use of Restrictive Housing (2017) Use of Restraints on Pregnant Women. Summary of recommended guidance: Sources that have guidance generally agree that restraints generally should not be used on a pregnant woman, except when necessary. Some sources indicate that if restraints are necessary, it should be well documented and require approval and assessment from a senior official and/or medical professional. Some sources specify the types of restraints that should never be used including abdominal restraints, handcuffs behind the back, and leg and ankle restraints. Example: “Restraint of pregnant inmates during labor and delivery should not be used. The application of restraints during all other pre- and postpartum periods should be restricted as much as possible and, when used, done so with consultation from medical staff and in the least restrictive means possible. All uses of restraints in pregnant inmates must be documented and reviewed.” – NCCHC Position Statement: Restraint of Pregnant Inmates (2015) Example: “Policies and procedures on the use of restraints on pregnant women and girls under correctional custody should be developed collaboratively by correctional leaders and medical staff who have knowledge about the potential health risks…The use of restraints on pregnant women and girls under correctional custody should be limited to absolute necessity.” - U.S. Department of Justice, Bureau of Justice Assistance: Best Practices in the Use of Restraints with Pregnant Women and Girls Under Correctional Custody (2014) Record Keeping on Pregnant Women Actions. Summary of recommended guidance: Sources that have guidance generally agree that accurate records of detention regarding pregnant women should be kept, with varying levels of specificity ranging from noting that records should be kept for incidents of restraint to specifying how documentation is kept and reviewed. One source notes that medical records should also be easily accessible for offsite care providers. Example: “If detention continues ICE should ensure…reporting of detention to ICE Headquarters and continued review of the need to detain.” – Report of the DHS Advisory Committee on Family Residential Centers (2016) Example: “Obstetrician-gynecologists and other obstetric care providers of antepartum care should be able to either primarily provide or easily refer to others to provide a wide array of services. These services include… imely transmittal of prenatal records to the site of the woman’s planned delivery so that her records are readily accessible at the time of delivery.” – American College of Obstetricians and Gynecologists Guidelines for Perinatal Care, Eighth Edition (2017) U.S. Customs and Border Protection (CBP) and its components, Border Patrol and the Office of Field Operations (OFO), have several policies and standards that address the care and treatment of pregnant women in their custody. Specifically, these include the following: CBP: National Standards on Transport, Escort, Detention, and Search (2015) OFO: Personal Search Handbook (2004) OFO: Directive: Secure Detention, Transport and Escort Procedures at Ports of Entry, CBP Directive No. 3340-030B (2008) Border Patrol: U.S. Border Patrol Policy: Hold Rooms and Short Term Custody (2008) Summaries of these policies and standards are provided below, along with the titles of the policies or standards on which each summary is based. Processing and holding. Officers and agents will consider pregnancy when expediting processing of vulnerable detained persons and when placing detained persons with others in hold rooms and holding facilities. Secure Detention, Transport and Escort Procedures at Ports of Entry (2008) and U.S. Border Patrol Policy: Hold Rooms and Short Term Custody (2008) Mental health services and counseling for pregnant women. If an agent or officer observes signs of mental illness, it should be reported to a supervisor and appropriate medical care be provided or sought, including calling emergency services in the event of an emergency. Transport, Escort, Detention, and Search (2015) Nutrition for pregnant women. Pregnant detainees should be offered a meal every six hours they are in detention and have access to snacks, milk, or juice at all times. Transport, Escort, Detention, and Search (2015); Secure Detention, Transport and Escort Procedures at Ports of Entry (2008); and U.S. Border Patrol Policy: Hold Rooms and Short Term Custody (2008) Special accommodations for pregnant women. Reasonable accommodations should be made for pregnant women, including placement in the least restrictive appropriate setting. If circumstances permit, pregnant women should not be placed in hold rooms or other secure areas, but instead in an open area under supervision. Transport, Escort, Detention, and Search (2015); Secure Detention, Transport and Escort Procedures at Ports of Entry (2008); and U.S. Border Patrol Policy: Hold Rooms and Short Term Custody (2008) Use of restraints on pregnant women. Officers and agents should not use restraints on pregnant women unless they demonstrate or threaten violence, have a criminal and/or violent history, or there is an articulable escape risk. Even if restraints are used, pregnant detainees are not to be restrained face-down, on their backs, or with a belt that constricts the area of her pregnancy. Pregnant women can never be restrained while in active labor or delivery. All use of restraints must be documented. Transport, Escort, Detention, and Search (2015) Record keeping on pregnant women actions. All physical interactions with pregnant women must be recorded after they occur. Any medical emergency must be recorded as soon as practical after emergency services have been contacted. Further, Border Patrol agents must create a booking record for persons detained and the record must include a medical annotation for conditions requiring care, including pregnancy. Transport, Escort, Detention, and Search (2015) and U.S. Border Patrol Policy: Hold Rooms and Short Term Custody (2008) U.S. Immigration and Customs Enforcement (ICE) uses various inspections for accessing facilities’ compliance with policies and detention standards—the frequency and focus of which vary. Some inspections also include pregnancy-related performance measures, such as a measure assessing whether a pregnancy test was performed at intake. We analyzed reports and data from five ICE inspections that address compliance with pregnancy-related policies and detention standards from 2015 through June 2019—the most recent data available at the time of our review. We selected these inspections because they review some aspect of the care provided to pregnant women. These inspections address compliance at ICE detention facilities where on-site medical care is provided by both ICE Health Service Corps (IHSC) as well as other entities (non-IHSC facilities). We reviewed results from IHSC’s inspections of IHSC-staffed and non- IHSC facilities, which includes pregnancy-related performance measures. We found that instances of non-compliance occurred at 16 facilities subject to a range of detention standards. Three of these facilities were IHSC-staffed, and 13 were non-IHSC. Table 10 shows results from December 2016 through March 2019. We reviewed information on pregnancy-related performance measures reported by facilities staffed by IHSC. Table 11 shows results from fiscal years 2015 through 2018. Although the table shows average annual compliance across all IHSC- staffed facilities, variation exists between facilities, and over time. For example, in fiscal year 2018, one facility improved its performance on the measure of whether prenatal vitamins were prescribed from 33 percent compliance in the first quarter to 100 percent compliance in the second quarter. In addition, in fiscal year 2018, facilities’ compliance with each measure ranged as follows: Obstetrician-gynecologist consult ordered is documented within 7 business days of identification: 50 to 100 percent (average 80 percent) Obstetrician-gynecologist scheduled appointment time documented within 7 business days of identification: 15 to 100 percent (average 75 percent) Detainee education documented at each encounter: 0 to 100 percent (average 79 percent) Records reviewed by provider after obstetrician appointment: 0 to 100 percent (average 79 percent) Appropriate labs ordered if not obtained from obstetrician- gynecologist: 50 to 100 percent (average 79 percent) Three additional ICE inspections identified 19 findings at 13 facilities related to the care of pregnant women. All of the findings occurred at non- IHSC facilities. Table 12 provides additional information on the findings and corrective actions that facilities reported taking. We interviewed ten pregnant women who were detained at three of the four U.S. Immigration and Customs Enforcement (ICE) facilities we visited, including facilities staffed by ICE Health Service Corps (IHSC- staffed) and non-IHSC facilities. We interviewed an additional four pregnant women at a local shelter in Texas which provides temporary accommodations to those in need of housing after their release from DHS custody. These four women may not have known which agency they had been detained or held by prior to entering the shelter. As a result, their perspectives are listed separately in the table below from the 10 women with whom we spoke at ICE detention facilities. Table 13 summarizes the perspectives of these 14 pregnant women. Although these interviews are not generalizable and may not be indicative of the care provided at all detention facilities, they provided us with perspectives on the care provided to pregnant women. We did not independently verify statements made by these 14 women we interviewed. We analyzed and categorized complaints that detainees, family members, non-governmental organizations, or other parties submitted to various entities from January 2015 through April 2019 regarding U.S. Immigration and Customs Enforcement’s (ICE) and U.S. Customs and Border Protection’s (CBP) care of pregnant women. Specifically, we reviewed complaints from Department of Homeland Security’s (DHS) Office for Civil Rights and Civil Liberties (CRCL), DHS’s Office of Inspector General, and ICE Health Service Corps (IHSC). We identified a total of 107 complaints—54 regarding ICE, 50 regarding CBP, and three regarding both ICE and CBP. We identified 54 unique complaints submitted from January 2015 through April 2019 regarding ICE’s care of pregnant women. Each of the 54 complaints may identify more than one area of concern, and as such we identified 104 concerns. The most common concern was that ICE allegedly did not provide medical care or the medical care was not quality or timely. As previously described in this report, the investigating agency determined that one complaint was substantiated and one complaint was partially substantiated. The remaining complaints were either still open as part of an on-going investigation, unsubstantiated by the investigating agency, or the complaint was not substantiated or unsubstantiated for a variety of reasons. Table 14 provides additional information on the number and types of concerns identified in the 54 complaints regarding ICE’s care of pregnant women. We identified 50 unique complaints submitted from January 2015 through April 2019 regarding CBP’s care of pregnant women. Each of the 50 complaints may identify more than one area of concern, and as such we identified 81 concerns. The most common concern was that pregnant women had allegedly been physically, verbally, or otherwise mistreated. As previously described in this report, the investigating agency determined that one complaint was substantiated. The remaining complaints were either still open as part of an on-going investigation, unsubstantiated or partially unsubstantiated by the investigating agency, the complaint was not substantiated or unsubstantiated for a variety of reasons, or the complaint described an event that occurred, such as a miscarriage, but did not allege that mistreatment or improper care occurred. Table 15 provides additional information on the number and types of issues identified in the 50 complaints regarding CBP’s care of pregnant women. In addition to the contact named above, Dawn Locke (Assistant Director), Tracey Cross (Analyst-in-Charge), Hiwotte Amare, David Bieler, Christine Davis, Elizabeth Dretsch, Kelsey Griffiths, Eric Hauswirth, Sasan J. “Jon” Najmi, Sean Sannwaldt, and Adam Vogt made key contributions to this report.", "summary": "In December 2017, the Department of Homeland Security (DHS) updated its policy on pregnant women, removing language that stated that pregnant women would generally not be detained except in extraordinary circumstances or as mandated by law. Within DHS, CBP temporarily holds individuals in its facilities and processes them for further action, such as release or transfer to ICE. ICE manages the nation's immigration detention system. ICE utilizes various facility types to detain individuals, such as those owned and operated by ICE and contract facilities. GAO was asked to review issues related to the care of pregnant women in DHS facilities. This report examines (1) what available data indicate about pregnant women detained or held in DHS facilities, (2) DHS policies and standards that address the care of pregnant women, and (3) what is known about the care provided to pregnant women in DHS facilities. GAO analyzed available DHS data and documents from calendar years 2015 through 2019, including detention data, inspection reports and data, and complaints; reviewed policies related to the care of pregnant women; and interviewed agency officials and three national non-governmental organizations. GAO also interviewed a non-generalizable sample of 14 pregnant women detained or released by DHS and five non-governmental organizations in four field locations that had the greatest number of detentions of pregnant women, among other things. GAO's analyses of U.S. Immigration and Customs Enforcement (ICE) and U.S. Customs and Border Protection (CBP) data on pregnant women found: ICE detained pregnant women over 4,600 times from calendar year 2016 through 2018, with more than 90 percent resulting from CBP arrests. Sixty-eight percent of these detentions were for 1 week or less, while 10 percent were for more than 30 days. Seventy-eight percent of these initial detentions occurred at facilities staffed with ICE medical personnel. ICE has policies and detention standards that address a variety of topics regarding the care of pregnant women, such as pregnancy testing requirements, for which non-governmental organizations, professional associations, and federal agencies have issued recommended guidance. However, some facility types—which vary based on who owns, operates, and provides medical care at the facility—did not address all these pregnancy-related topics in their policies and standards, such as prenatal vitamins, as of December 2019. ICE has plans to address the gaps GAO identified in these facility types, including updating some of its policies and detention standards in February 2020. In regards to CBP, its facilities are designed for holding individuals for no more than 72 hours, and therefore are not equipped to provide long-term care. Nonetheless, CBP has some policies and standards regarding pregnant women for its short-term facilities, including those related to nutrition and the circumstances in which restraints could be used. GAO's analyses of inspections and complaint mechanisms offered the following insights into the care provided to pregnant women: ICE inspections found 79 percent or greater compliance with most of its pregnancy-related performance measures. For example, inspections found 91 percent of pregnant woman were seen by an obstetrician-gynecologist within 30 days of pregnancy confirmation, from December 2016 through March 2019. According to ICE officials and agency documentation, ICE has processes in place to address non-compliance. Additional inspections identified pregnancy-related issues at 13 facilities from January 2015 through July 2019. The facilities or ICE have taken actions to address the issues. CBP generally relies on offsite care for pregnant women, and as a result has limited information on care CBP provided. However, CBP has efforts underway to enhance medical support at selected facilities. Over 100 complaints were filed about ICE's and CBP's care of pregnant women from January 2015 through April 2019. Of these complaints, 3 were substantiated or partially substantiated, and 24 were unsubstantiated or partially unsubstantiated. In most cases there was not enough information for the investigating agency to determine whether proper care had been provided.", "document_type": "gao"}
{"report": "As sales of cigarettes generally decreased over the past 10 years, combined sales of roll-your-own tobacco, pipe tobacco, small cigars, and large cigars have increased as a percentage of the total market. Figure 1 shows a sample of these smoking tobacco products. As shown in figure 2, while the cigarette share of the smoking tobacco market has decreased, cigarette sales continue to dominate the market for smoking tobacco products. Cigarette sales fell from 350.3 billion cigarettes in fiscal year 2008 to 236.9 billion cigarettes in fiscal year 2018, and its percentage of the smoking tobacco market declined from 93.5 percent to 87.3 percent. During this same period, the combined sales of roll-your-own tobacco, pipe tobacco, small cigars, and large cigars increased from the equivalent of 24.5 billion sticks in fiscal year 2008 to 34.6 billion sticks in fiscal year 2018, an increase from 6.5 percent to 12.8 percent of the total market for smoking tobacco products. Although electronic cigarettes are growing in popularity among U.S. youth according to the FDA, they are not included in the sales data on smoking tobacco products represented in figure 2. Electronic cigarettes are not currently taxed under the Internal Revenue Code as a tobacco product. Accordingly, corresponding data on electronic cigarettes sales are not available. Federal excise tax rates on different tobacco products are calculated in different ways. Cigarettes and small cigars are taxed on a per unit basis— the number of sticks. Roll-your-own and pipe tobacco are taxed by weight. Before CHIPRA, the federal excise tax rate on cigarettes was higher than the rates on roll-your-own tobacco, pipe tobacco, and small cigars. In 2009, Congress passed CHIPRA and significantly raised the tax rates on these four products, equalizing the rates for cigarettes, roll-your- own tobacco, and small cigars. CHIPRA also increased the tax rate for pipe tobacco, among other products, but not to the level of the other three products mentioned. Table 1 shows the increases in federal excise tax rates under CHIPRA for these four products. As shown in figure 3, CHIPRA equalized—on a comparable per stick basis—federal excise tax rates for cigarettes, roll-your-own tobacco, and small cigars but not for pipe tobacco. As a result, of the three cigarette products shown previously in figure 1, the cigarette made with pipe tobacco (marked as number 2) is taxed at a much lower rate than either the factory-made cigarette (number 3) or the cigarette made with roll- your-own tobacco (number 1). CHIPRA also increased the federal excise tax rate on large cigars. Large cigars are unique among tobacco products in that the tax rate is ad valorem—calculated as a percentage of the manufacturer’s or importer’s sale price—up to a maximum tax (currently $402.60) per thousand sticks. CHIPRA increased the ad valorem rate for large cigars from 20.72 percent to 52.75 percent of the manufacturer’s or importer’s sale price, up to a maximum of $402.60 per thousand sticks (see table 2). To reduce federal excise taxes, manufacturers of inexpensive small cigars have an incentive to modify their product to qualify for the lower- taxed large cigar category by adding weight. For example, manufacturers of cigars with a sale price of $50 per thousand would pay $26.38 per thousand in federal excise taxes if the cigar qualified as large cigars compared to $50.33 per thousand if they qualified as small cigars. Consequently, a manufacturer of small cigars would experience a tax savings of $23.95 per thousand if it changed the product to qualify as a large cigar. In figure 1, although the small cigar (marked as number 4) and the large cigar (number 5) are similar in appearance, they are likely taxed at significantly different rates, depending on the price of the large cigar. Domestic manufacturers and importers of tobacco products must obtain a permit from TTB before engaging in business. TTB collects federal excise taxes on domestic tobacco products when these products leave manufacturing facilities. CBP, within the Department of Homeland Security, collects the federal excise taxes on imported tobacco products after those products are released from Customs custody. Tobacco products—including roll-your-own tobacco, pipe tobacco, small cigars, and large cigars—are broadly defined in the Internal Revenue Code (see table 3). Roll-your-own tobacco and pipe tobacco are defined by such factors as the use for which the product is suited and how the product is offered for sale, as indicated by its appearance, type, packaging, and labeling. These definitions do not specify any physical characteristics that would differentiate pipe tobacco from roll-your-own tobacco, and TTB faces challenges in distinguishing these two products for tax collection purposes. We reported in 2014 that according to government officials, representatives of nongovernmental organizations, and industry, the new pipe tobacco products introduced after CHIPRA had minimal, if any, differences from roll-your-own tobacco products. We further reported in 2014 that TTB took rulemaking actions intended to more clearly differentiate the two products. As of May 2019, TTB was still finalizing its regulatory approach for distinguishing between the two products. According to TTB officials, TTB continues to face the challenges inherent in identifying specific physical characteristics that clearly distinguish pipe tobacco from roll-your-own tobacco. TTB officials have discussed the complexity of administering the federal excise tax on large cigars because it is calculated as a percentage of the manufacturer’s or importer’s sale price, up to a maximum tax per thousand sticks. We reported in 2014 that TTB’s efforts to monitor and enforce tax payments on large cigars became more complex after CHIPRA as more manufacturers and importers determined their tax liability based on the sale price per stick rather than simply paying the set maximum tax. In addition, we reported that according to TTB officials some large cigar manufacturers and importers began to restructure their market transactions to lower the sale price for large cigars and obtain tax savings based on a lower ad valorem rate. According to TTB officials, some manufacturers and importers, for example, were “layering” sales transactions by including an additional transaction at a low price before the sale to the wholesaler or distributor and using this low initial price to calculate the tax. According to TTB officials, such transactions are conducted with an intermediary that may have a special contract arrangement with the manufacturer or importer. The intermediary may then add a large markup to the subsequent sale price to the wholesaler or distributor. This added transaction effectively lowers the manufacturer’s or importer’s sale price and thus reduces the taxes collected. TTB officials stated that these types of transactions have continued since 2014, and that taking enforcement actions to counter them is challenging and resource intensive due to their complexity. TTB officials also noted that these activities can range from legal tax avoidance to illegal tax evasion, requiring a case-specific analysis of each transaction. Large tax disparities among similar tobacco products created opportunities for tax avoidance and led to immediate market shifts to the lower-taxed products. Specifically, since CHIPRA took effect in 2009, pipe tobacco consumption increased significantly—steeply at first and then leveling off. Over the same period, roll-your-own tobacco consumption fell sharply and then more gradually declined. Similarly, large cigar consumption rose sharply after CHIPRA took effect, while sales of small cigars dramatically decreased and now make up very little of the combined market share for cigars. Following CHIPRA’s passage, pipe tobacco sales rose steeply, peaking in July 2013 and leveling off since then (see fig. 4). Pipe tobacco sales grew from 5.2 million pounds in fiscal year 2008, the fiscal year before CHIPRA came into effect, to 40.7 million pounds in fiscal year 2018. Pipe tobacco sales reached a high in fiscal year 2013, with consumption exceeding 42.4 million pounds and spiking in July 2013 for a monthly high of over 4.9 million pounds. After this spike, the pipe tobacco market leveled off with monthly sales fluctuating from 2.8 million to 4.1 million pounds. Despite this leveling off, pipe tobacco’s share of the combined roll-your- own and pipe tobacco market continued to increase, reaching approximately 95 percent in fiscal year 2018, which is the highest it had been since CHIPRA took effect. Figure 4 also shows that as pipe tobacco sales increased significantly after the passage of CHIPRA, roll-your-own tobacco experienced an immediate drop in sales. Annual sales of roll-your-own tobacco dropped from 17.0 million pounds in fiscal year 2009 to 6.4 million pounds in fiscal year 2010, before declining further to 2.2 million pounds in fiscal year 2018. The lowest annual sales for roll-your-own tobacco since CHIPRA occurred in fiscal year 2018. Over the 11 fiscal years from 2008 through 2018, roll-your-own tobacco’s share of the combined roll-your-own and pipe tobacco market decreased from approximately 78 percent to approximately 5 percent. Figure 5 shows that the overall combined sales of pipe tobacco and roll- your-own tobacco were higher after CHIPRA than before CHIPRA. However, the growth rate declined from 0.69 percent before CHIPRA to 0.33 percent after CHIPRA took effect. In April 2012, we reported that the rise in pipe tobacco sales after CHIPRA coincided with the growing availability of commercial roll-your- own machines that enabled customers to produce a carton of roll-your- own cigarettes with pipe tobacco in less than 10 minutes. Not only were customers able to save money through lower taxes on pipe tobacco, but the commercial roll-your-own machines also provided significant time savings compared with rolling cigarettes by hand. The market shift from roll-your-own to pipe tobacco has persisted in recent years despite a change in the legal status of businesses making commercial roll-your-own machines available to consumers, resulting in these machines being less readily available. Following the growth in the availability of commercial roll-your-own machines, Congress passed a law in July 2012 that included a provision adding “any person who for commercial purposes makes available for consumer use…a machine capable of making cigarettes, cigars, or other tobacco products” to the definition of “manufacturer of tobacco products” for tax purposes. As a result, businesses meeting this definition faced increased tax liability and regulatory requirements. According to TTB officials and industry observers, the number of businesses making commercial roll-your-own machines available to customers declined after the 2012 law’s passage. Nevertheless, combined annual sales of pipe tobacco and roll-your-own tobacco generally have not decreased since the 2012 law was passed. Besides its lower federal excise tax, which creates financial incentives, pipe tobacco has other advantages over roll-your-own tobacco that may also contribute to its sustaining an overwhelming share of the combined roll-your-own and pipe tobacco market. For example, according to the Food and Drug Administration (FDA), pipe tobacco is not covered by the Federal Food, Drug, and Cosmetic Act restriction, such as the ban on flavor additives, imposed on roll-your-own tobacco and cigarettes. Also, according to FDA, pipe tobacco does not currently have the warning label requirements that are imposed on roll-your-own tobacco and cigarettes. Finally, while makers of roll-your-own tobacco are required to make payments under the Tobacco Master Settlement Agreement, makers of pipe tobacco do not make these payments. This increases the incentive for roll-your-own tobacco users to switch to the cheaper pipe tobacco. After CHIPRA, sales of lower-taxed large cigars rose sharply, while sales of small cigars plunged (see fig. 6). From fiscal year 2008 through fiscal year 2018, annual sales of large cigars increased from 5.8 billion sticks to 13.1 billion sticks. This increase included a significant spike in demand immediately after CHIPRA’s passage in 2009. The increase in annual sales then largely leveled off after fiscal year 2010, with sales ranging between 11.9 and 13.2 billion large cigars. As a share of the combined market for small and large cigars, large cigar sales have continued to expand. Large cigar sales increased from approximately 50 percent of the combined market in fiscal year 2008 (before CHIPRA) to approximately 92 percent in fiscal year 2010 and reached approximately 97 percent by the end of fiscal year 2018. Figure 6 also shows that just as large cigar sales increased immediately following CHIPRA, sales of small cigars declined substantially. Annual small cigar sales dropped from 3.6 billion to 1.0 billion sticks between fiscal years 2009 and 2010, and declined further to 0.4 billion sticks by fiscal year 2018. Over the 10-year period between 2008 and 2018, the market share held by small cigars decreased from a high of approximately 50 percent of the combined small and large cigar market in 2008 to approximately 3 percent in fiscal year 2018. Figure 7 shows that the overall combined sales of small and large cigars were higher after CHIPRA than before CHIPRA, although the growth rate for small and large cigars leveled off after CHIPRA took effect in 2009. The growth rate before CHIPRA was 0.78 percent and the growth rate after CHIPRA was 0.03 percent. The makeup of large cigar sales also changed after CHIPRA, with imports replacing domestic cigars as the main contributor to the large cigar market (see fig. 8). When CHIPRA took effect in April 2009, domestic large cigars made up 93.5 percent of the large cigar market. After CHIPRA, the large cigar market began to shift in favor of imports and, by February 2017, imported large cigars consistently became the majority product in the large cigar market. As of September 2018, imported cigars made up 65.6 percent of the large cigar market compared to 93.5 percent held by domestic large cigars in April 2009. Market shifts to avoid increased tobacco taxes following CHIPRA have continued to reduce federal revenue. We estimate that federal revenue losses due to market shifts from roll-your-own to pipe tobacco and from small to large cigars range from approximately $2.5 to $3.9 billion from April 2009 through September 2018, depending on assumptions about how consumers would respond to a tax increase. In contrast, total tax revenue collected for smoking tobacco products, including cigarettes, amounted to about $138 billion over the same time period. We previously reported in 2014 on the estimated federal revenue losses resulting from these market shifts, reporting that estimated federal revenue losses due to the market shifts from roll-your-own tobacco to pipe tobacco and from small to large cigars ranged from approximately $2.6 billion to $3.7 billion from April 2009 through February 2014. Estimated tax revenue losses in the combined roll-your-own and pipe tobacco markets. TTB and CBP collected approximately $2.0 billion in federal excise tax revenue from domestic and imported roll- your-own and pipe tobacco from April 2009 through September 2018. We estimate that during the same period the market shift from roll- your-own to pipe tobacco reduced federal excise tax revenue by an amount ranging from $499 million to $1.2 billion (see fig. 9). Estimated tax revenue losses in the combined small and large cigar markets. TTB and CBP collected about $7.2 billion in federal excise tax revenue from domestic and imported small and large cigars from April 2009 through September 2018. We estimate that during the same period the market shift from small to large cigars reduced federal excise tax revenue by an amount ranging from $2.0 billion to $2.7 billion (see fig. 10). Federal revenue would likely increase if Congress were to equalize the tax rate for pipe tobacco with the rates currently in effect for roll-your-own tobacco and cigarettes. We estimate that federal revenue would increase by a total of approximately $1.3 billion from fiscal year 2019 through fiscal year 2023 if the pipe tobacco tax rate were equalized to the higher rate for roll-your-own tobacco and cigarettes. While equalizing federal excise taxes on small and large cigars should raise revenue based on past experience, the specific revenue effect is unknown because the data needed for conducting that analysis are not available. See appendix 1 for information on our methodology for estimating the effect on tobacco tax revenue if Congress were to eliminate current tax disparities among similar tobacco products and our assumptions about price sensitivity and other factors. We estimate that under current tax rates TTB and CBP would collect approximately $825 million in federal excise tax revenue from domestic and imported roll-your-own and pipe tobacco from October 2018 through September 2023. If Congress were to increase the federal excise tax rate on pipe tobacco of $2.83 per pound to the higher roll-your-own tobacco rate of $24.78 per pound, we estimate that $1.3 billion in additional federal revenue would be collected for these two products for the same time period (see fig. 11). The revenue effect if Congress were to equalize federal excise tax rates on small and large cigars is unknown because data for conducting this analysis are not available. Unlike roll-your-own and pipe tobacco, which are each taxed by weight, the tax rate on large cigars is based on an ad valorem rate and the tax rate on small cigars is based on number of sticks. Legislative proposals in the 115th and 116th Congress for changing the federal excise tax on large cigars have included replacing the ad valorem rate with a rate based on weight, together with a minimum tax per cigar. Shifting from an ad valorem tax to one based on weight could effectively equalize small and large cigar tax rates and address challenges that TTB currently faces in administering the large cigar tax; however, developing a reliable estimate of the revenue effect of such a change is not possible because the data needed on large cigars to conduct this analysis are not available. Specifically, data are not available on (1) large cigar weights or (2) the distribution of large cigars for which the federal excise tax now being paid is above or below the current rate for small cigars. These data on large cigars are not collected by TTB because such data are not needed to administer and collect large cigar taxes under the current tax structure. In the absence of these data, it is not possible to reliably calculate the potential effect on tax revenue of a counterfactual scenario for equalizing small and large cigar federal excise taxes. See appendix I for more information on the additional data needed for developing an estimate of the revenue effect of equalizing the federal excise tax rate on small and large cigars. As previously discussed, the number of imported large cigars has increased in recent years and the ratio of imported to domestic large cigars in the U.S. market has shifted toward imports. As part of this trend, there has also been an increase in the proportion of imported large cigars that are taxed at a lower rate than the small cigar tax rate of 5.03 cents per stick. From fiscal years 2013 through 2018, 72 percent of imported large cigars were taxed at a rate less than 5.03 cents per stick. As a result of this increase in inexpensive imported large cigars, annual large cigar revenue has begun to decline. Large cigar revenue has declined from a monthly average of $71.5 million over the period from April 2009 to December 2012 to a monthly average of $52.9 million over the period from January 2013 through September 2018. Large cigars account for approximately 95 percent of combined small and large cigar revenue. Figure 12 shows actual combined small and large cigar federal excise tax revenue from fiscal year 2008 through fiscal year 2018. The combined average monthly federal revenue for small and large cigars increased significantly after CHIPRA went into effect in 2009, from $21.3 million in fiscal year 2008 to $72.8 million in fiscal year 2010, and remains above the pre-CHIPRA level (see fig. 12). Based on this experience, if Congress were to equalize federal excise taxes through a tax increase for large cigars, revenue should increase. However, the magnitude of the revenue effect of equalizing taxes on small and large cigars is unknown because the data for conducting this analysis are not available. We provided a draft of this report for comments to the Departments of the Treasury, Homeland Security, and Labor. The Department of the Treasury generally concurred with the report’s findings and provided technical comments, which we have addressed as appropriate. The Department of Homeland Security also provided technical comments, which we have addressed as appropriate. The Department of Labor did not provide comments on the report. We are sending copies of this report to the appropriate congressional committees and the Secretary of the Treasury, the Secretary of Homeland Security, the Secretary of Labor, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Our objectives were to examine (1) market shifts among smoking tobacco products since the Children’s Health Insurance Program Reauthorization Act (CHIPRA) of 2009 went into effect, (2) the estimated effects on federal revenue if the market shifts following CHIPRA had not occurred, and (3) what is known about the effects on revenue if Congress were to eliminate current tax disparities between smoking tobacco products. Our analysis focuses on roll-your-own tobacco, pipe tobacco, small cigars, and large cigars. It covers sales and federal excise tax payments for these products from October 2001 through September 2018. To address the objectives in this study, we reviewed documents and interviewed agency officials from the Department of the Treasury’s Alcohol and Tobacco Tax and Trade Bureau (TTB), the Department of Homeland Security’s U.S. Customs and Border Protection (CBP), and the Department of Labor’s Bureau of Labor Statistics (BLS). We also interviewed representatives from other organizations working on tobacco and taxation issues to obtain background information on markets, industry, and consumption practices and trends for tobacco products. For objective one, we identified market shifts among smoking tobacco products by analyzing TTB domestic removals data and CBP imports data to identify sales trends across the different domestic and imported tobacco products before and after CHIPRA took effect. For objectives two and three, we estimated the federal revenue effects of differences in federal excise tax rates for tobacco products by analyzing TTB’s and CBP’s revenue data and BLS price data for smoking tobacco products. We estimated what the effect on tax revenue collection would have been if the sales trends for roll-your-own and pipe tobacco and for small and large cigars had not been affected by substitution between the products but had been affected by the increase in price due to the tax—in other words, if the market shifts resulting from the substitution of higher-taxed products with lower-taxed products had not occurred. In this report, we refer to this estimated effect on federal tax revenue collection as revenue losses. In addition, we analyzed what is known about the effects on federal revenue if Congress were to eliminate current tax disparities between smoking tobacco products. We assumed that the pipe tobacco federal excise tax was increased and equalized to the level of the roll- your-own tobacco tax as of October 1, 2018, and we calculated the cumulative revenue differential for five fiscal years through September 2023. We assessed the reliability of the data for these objectives by performing data checks for inconsistency errors and completeness and by interviewing relevant officials. We determined that the data used in this report were sufficiently reliable for our purposes. Our estimate of federal revenue losses resulting from differences in federal excise tax rates among smoking tobacco products includes combined tax revenue losses for the roll-your-own and pipe tobacco markets as well as the small and large cigar markets. Our analysis takes into account the expected fall in quantity demanded due to the price increases resulting from the higher federal excise tax rates that CHIPRA imposed on these smoking tobacco products, holding other variables constant. To calculate the range of federal revenue losses, we included high and low estimates based on assumptions about the effect of a price increase on projected sales. Economic theory shows that when the price of a product increases, the demand for the product will adjust downward, decreasing at an estimated rate based on demand for the product, i.e., price elasticity. On the basis of our prior work estimating revenue losses from tobacco taxes and a literature review, we determined that the price elasticity for the smoking tobacco products ranges from -0.6 to -0.3, respectively, for the low and high revenue estimates. Our projections also take into account the historic sales trends for these products, the sales trend of cigarettes after CHIPRA and the tax component of the price. We developed our revenue loss estimate by comparing the actual tobacco tax revenues collected by TTB with a counterfactual scenario. The counterfactual model draws from a model used by Dr. Frank Chaloupka, an economist and a leading scholar who has investigated the effect of prices and taxes on tobacco consumption in numerous publications. In particular, we based our methodology on Dr. Chaloupka’s model calculating the effect of raising cigarette taxes in the State of Illinois. This methodology projects the effect of a future tax increase based on the historic sales trend, the amount of the tax, and the price elasticity of demand. Under this model, when a tax increase is enacted, demand for the product is expected to decline based on the price elasticity and the effect on prices. Following this initial decline, demand for the product is expected to continue at the rate of its historic sales trend. We updated this model by assuming that tobacco products that incur a tax increase to match the tax rate on cigarettes will follow the cigarette sales trend after CHIPRA rather than the product’s historic trend. For example, the roll-your-own tax rate increased under CHIPRA to match the rate on cigarettes because it was viewed as a substitute for cigarettes. Projecting the pre-CHIPRA sales trend forward based on historical data could provide a misleading result as it includes the additional consumption from substitution. Under our assumption, the pre- CHIPRA sales trend is adjusted downward based on the actual sales trend for cigarettes, which has generally declined in recent years. The BLS price data used in our analysis are a subset of the data used for calculating the Consumer Price Index for tobacco products. The BLS data contain retail price information collected each month throughout the United States. These price data only include excise taxes from federal, state and local governments and exclude shipping, handling, sales tax, and fuel surcharges. Because the BLS data are at the retail level, there is an expected markup in addition to the charges mentioned above. To simplify the model, we assumed that the markup remains constant after CHIPRA was passed. We calculated an average price for the year before CHIPRA was enacted, and we calculated the post-CHIPRA price by adding the new tax to the pre-CHIPRA price. Therefore, we estimated only the effect of CHIPRA on taxes. We calculated large cigar revenues and developed a revenue loss estimate for large cigars using assumptions based on available data. As discussed earlier in the report, small cigars are currently taxed at $50.33 per thousand sticks, while, large cigars are taxed at 52.75 percent of the manufacturer’s sale price, up to a cap of $402.60 per thousand sticks. TTB collects revenue data for all cigars, but does not collect separate revenue data for small and large cigars. We calculated large cigar revenues by subtracting small cigar revenue from total cigar revenue. We calculated small cigar revenues by multiplying the number of sticks reported to TTB in each month by the tax rate. After calculating large cigar revenue, we estimated the average tax paid per cigar by dividing the large cigar revenue by the number of sticks for each month and calculating the average price. From March 2007 through March 2009, the average large cigar tax collected was 4.2 cents per stick. CHIPRA raised this cap from 4.9 cents to approximately 40 cents per stick. We calculated that the average taxable price for large cigars before CHIPRA was 20.12 cents. Since the tax is based on the price rate, the percentage change in price due to taxation is based on the percentage change of the price, plus tax, before and after CHIPRA. To calculate the potential effect on federal tax revenue from raising the tax rate for pipe tobacco to match the roll-your-own tax rate, we followed the model discussed above, but we adjusted the pipe tobacco tax to the roll-your-own rate of $24.78 per pound. The model assumes that taxes would have been equalized as of October 1, 2018, and calculates the cumulative revenue differential for 5 fiscal years through September 2023. The model takes into account the additional reduction in consumption due to the tax increase and estimates potential revenue differentials. A price elasticity of -0.8 is assumed to provide a conservative scenario. Our model assumes that there are no other smoking tobacco products that are close substitutes, an assumption we also made in our previous models; the higher elasticity of -0.8 accounts for a drop in consumption altogether. The magnitude is based on a literature review and interviews with the Joint Committee on Taxation. After the drop in demand due to the tax increase, demand is projected linearly using the most recent 5-year historic trend. The projection of actual sales is calculated by applying the same historic trend to the actual sales of roll-your-own and pipe tobacco. Actual revenue is calculated by multiplying the tax rate to the projected sales. An analysis projecting the impact of equal tax rates for small and large cigars requires a different set of assumptions. The reliability of any such model would be questionable, particularly for large cigars because the tax rate on them is calculated as a percentage of the price. Compared with determining the tax on all other tobacco products, according to TTB, determining the tax on large cigars is extremely complex. We concluded that modeling hypothetical consumption trends for smoking tobacco products after equalizing tax rates on small and large cigars would require a complex set of assumptions not sufficiently grounded in reliable data. These assumptions include the price distribution of large cigars since CHIPRA was enacted and assumptions about the proportion of the large cigar market captured by imported large cigars if large cigars were taxed similarly to small cigars. Rather than calculating a tax revenue estimate using assumptions not grounded in reliable data, we present actual cigar revenue and show how the large cigar market has changed from domestic cigars to cheaper imported cigars over time. While it is possible to develop a tax equalization model based only on applying a minimum tax rate per large cigar of 5.03 cents per stick—to ensure large cigars are not taxed below the small cigar tax rate of 5.03 cents per stick—this approach would not produce a reliable estimate of the full revenue effect of legislative proposals to equalize small and large cigar taxes. Applying only a minimum tax would have the effect of underestimating the federal excise tax collected from more expensive cigars because this would reduce the revenue estimates on large cigars that are currently taxed at between 5.03 cents per stick and the maximum rate of 40 cents per stick. In addition, the distribution of domestic large cigar sales that are taxed below the small cigar tax rate is unknown because TTB data on domestic large cigar sales are collected by manufacturers and reported monthly as a quantity aggregate. Without incorporating this information on the distribution of large cigars paying above and below the small cigar tax rate of 5.03 cents per cigar, an estimate of the revenue effects of equalizing small and large cigars would understate the potential revenue that could have been collected from large cigars. We conducted this performance audit from September 2018 to June 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Christine Broderick (Assistant Director), Jeremy Latimer (Analyst-in-Charge), Pedro Almoguera, David Dayton, Mark Dowling, Christopher Keblitis, and Ethan Kennedy made key contributions to this report.", "summary": "In 2009, CHIPRA increased and equalized federal excise tax rates for cigarettes, roll-your-own tobacco, and small cigars but did not equalize tax rates for pipe tobacco and large cigars—products that can be cigarette substitutes. GAO reported in 2012 and 2014 on the estimated federal revenue losses due to the market shifts from roll-your-own to pipe tobacco and from small to large cigars. This report updates GAO's prior products by examining (1) the market shifts among smoking tobacco products since CHIPRA, (2) the estimated effects on federal revenue if the market shifts had not occurred, and (3) what is known about the revenue effects if Congress were to eliminate current tax disparities between smoking tobacco products. GAO analyzed data from the Department of the Treasury and U.S. Customs and Border Protection to identify sales trends for domestic and imported smoking tobacco products, to estimate the effect on tax collection if market substitutions had not occurred, and to model the effects of equalizing tax rates for smoking tobacco products. Large federal excise tax disparities among similar tobacco products after enactment of the Children's Health Insurance Program Reauthorization Act (CHIPRA) of 2009 led to immediate market shifts (see figure). Specifically, CHIPRA created tax disparities between roll-your-own and pipe tobacco and between small and large cigars, creating opportunities for tax avoidance and leading manufacturers and consumers to shift to the lower-taxed products. Following the market shifts after CHIPRA, the lower-taxed products have sustained their dominant position in their respective markets. Market shifts to avoid increased tobacco taxes following CHIPRA have continued to reduce federal revenue. GAO estimates that federal revenue losses due to market shifts from roll-your-own to pipe tobacco and from small to large cigars range from a total of about $2.5 to $3.9 billion from April 2009 through September 2018, depending on assumptions about how consumers would respond to a tax increase. Federal revenue would likely increase if Congress were to equalize the tax rate for pipe tobacco with the rates currently in effect for roll-your-own tobacco and cigarettes. GAO estimates that federal revenue would increase by a total of approximately $1.3 billion from fiscal year 2019 through fiscal year 2023 if the pipe tobacco tax rate were equalized with the higher rate for roll-your-own tobacco and cigarettes. While equalizing federal excise taxes on small and large cigars should raise revenue based on past experience, the specific revenue effect is unknown because data for conducting this analysis are not available. These data are not collected by the Department of the Treasury because such data are not needed to administer and collect large cigar taxes under the current tax structure. In its 2012 report, GAO recommended Congress consider equalizing tax rates on roll-your-own and pipe tobacco and consider options for reducing tax avoidance due to tax differentials between small and large cigars. Treasury generally agreed with GAO's conclusions and observations. As of May 2019, Congress had not passed legislation to reduce or eliminate tax differentials between smoking tobacco products. Treasury also generally agreed with this report's findings.", "document_type": "gao"}
{"report": "After Border Patrol agents or OFO officers apprehend noncitizen family units, they are to interview each individual, using interpreters if needed, and collect personal information such as their names, countries of nationality, and age. Agents and officers also collect biometric information, such as photographs and fingerprints, from certain individuals, including those in family units. Border Patrol agents and OFO officers use fingerprints to run records checks against federal government databases to determine whether individuals have any previous immigration or criminal history. Agents and officers are to enter information about the individuals in the appropriate automated data system as soon as possible, in accordance with CBP policy. According to Border Patrol and OFO officials, if noncitizens are determined to be ineligible for admission into the United States, agents and officers must determine whether to place them, including those arriving in family units, into full or expedited immigration removal proceedings, consistent with the Immigration and Nationality Act. In full removal proceedings, individuals have the opportunity to present evidence to an immigration judge to challenge their removal from the country and apply for various forms of relief or protection, including asylum. In expedited removal proceedings, the government can order individuals removed without further hearing before an immigration judge unless they express the intent to apply for asylum or a fear of persecution or torture if returned to their home country. Most arriving family units are eligible to be placed into expedited removal proceedings, with certain exceptions, according to Border Patrol and OFO officials. A 2015 CBP policy requires CBP’s agents and officers to record such decisions for each family unit member in agency data systems. Further, Border Patrol agents and OFO officers print copies of the information they enter into data systems to create a paper file, known as an “A-file,” for each family unit member they apprehend. One of the key required DHS forms in the A-file is Form I-213, Record of Deportable/Inadmissible Alien (Form I-213). Among other things, this form captures biographic information and includes a narrative section for agents and officers to capture details about the circumstances of the apprehension. According to Border Patrol and OFO headquarters officials, each family unit member’s A-file is reviewed and approved by a supervisor. If CBP or ICE determines that a family separation is warranted, agents or officers process the child or children as UAC, according to Border Patrol, OFO, and ICE officials. ICE’s Office of Enforcement and Removal Operations is generally responsible for transferring these children, including those separated from a parent, as appropriate, to ORR. Under the Trafficking Victims Protection Reauthorization Act of 2008, children must be transferred to ORR within 72 hours after determining that they are UAC, except in exceptional circumstances. Table 1 provides additional details about DHS and HHS roles in processing family units. DHS officials told us that CBP typically holds family units together for a limited time before transferring them together to ICE, in accordance with CBP policy. During that time, agents and officers decide on a case-by- case basis whether to place each family unit in expedited or full immigration proceedings, according to Border Patrol and OFO officials. Individuals, including family unit members, placed in expedited removal proceedings and who express a fear of persecution or torture are generally subject to mandatory detention under the Immigration and Nationality Act pending a final credible fear determination. As a result, Border Patrol and OFO officials stated that its agents and officers typically determine whether ICE has space in its family residential centers before processing family units into expedited removal proceedings. From June 2014 through October 2019, ICE, during various periods, operated four family residential centers in Texas, Pennsylvania, and New Mexico for family units who may be subject to removal while they await the resolution of their immigration cases or who have been ordered removed from the United States. As of October 2019, ICE maintains three family residential centers—in Dilley, Texas; Karnes City, Texas; and Leesport, Pennsylvania—with a cumulative capacity of 3,326 beds. For information about these facilities, see table 2. CBP has historically separated children apprehended in family units from their parent(s) in specific circumstances, such as if the parental relationship could not be confirmed, if there was reason to believe the adult was participating in human trafficking, or if the parent was otherwise a threat to the safety of the child. As we reported in October 2018, ORR officials began observing an increase in the percentage of children in its care who were separated from their parents beginning in 2017. ORR officials stated they saw a continued increase in separated children in their care in the first few months of calendar year 2018. In April 2018, the U.S. Attorney General directed federal prosecutors to implement a zero-tolerance policy along the southwest border for immigration offenses and to accept all improper entry cases referred for prosecution to the extent practicable. According to DHS officials, after the Attorney General’s April 2018 memo, CBP began referring a greater number of adults apprehended at the border to the Department of Justice for criminal prosecution, including parents who were apprehended with minor children. CBP generally then separated the family unit, and after processing the children as UAC, CBP transferred them to ORR custody. According to CBP headquarters officials, the goal of the zero tolerance policy was to deliver a consequence to those crossing the border illegally by charging and convicting them of a crime, specifically a criminal conviction for improper entry, which is generally a misdemeanor. This could then lead to escalating criminal consequences for subsequent apprehensions, since noncitizens—in this case, adults in family units— entering the United States illegally for a second time could be charged with illegal reentry after removal from the United States, a felony offense. On June 20, 2018, the President issued an executive order directing that alien families generally be detained together. On June 26, 2018, a federal judge ruled in the Ms. L. v. ICE case, which was filed by the American Civil Liberties Union on behalf of certain parents (referred to as class members) who had been separated from their children. The June 2018 court order stated that certain separated parents must be reunited with their minor children, barring certain disqualifying criteria. On June 27, 2018, the CBP Commissioner issued a policy memorandum to provide direction on complying with the court order, to include potential reasons why a family separation may still be warranted. Figure 1 describes key actions since the Attorney General’s April 2018 memo that have influenced how DHS determines when family separations are warranted. On July 10, 2018, the court approved reunification procedures for the class members covered by the June 2018 court order. At that time the approved class included those adult parents separated from their children by DHS whose children were in ORR custody as of June 26, 2018, barring certain disqualifying criteria. Subsequently, on March 8, 2019, the court ordered an expansion of the class members to include all adult parents, subject to the same disqualifying criteria, who entered the United States at or between designated ports of entry on or after July 1, 2017, and were separated from their children by DHS. As of January 15, 2020, the government provided to the plaintiffs 11 lists identifying a total of 1,556 children of potential expanded class members. This brought the total number of possible separated children of potential class members to 4,370. CBP data indicate that the number of CBP apprehensions of family unit members was greater in the first two quarters of fiscal year 2019 than in all of fiscal year 2018. In addition, apprehensions of family unit members increased from approximately 22 percent of all southwest border apprehensions in fiscal year 2016 to approximately 51 percent of all such apprehensions in the second quarter of fiscal year 2019. The data also indicate that the majority of CBP apprehensions of family unit members were Central American nationals and the majority of apprehensions of children in family units were for children under the age of 12. Further, the data indicate that CBP placed family unit members in full removal proceedings before immigration courts at an increasing rate, and most were released into the United States to await their immigration court proceedings. Finally, CBP data indicate that CBP separated at least 2,700 children from their parents from April 2018 through March 2019. CBP data indicate that the number of apprehensions of family unit members along the southwest border increased from about 120,400 apprehensions in fiscal year 2016 to about 160,400 apprehensions in fiscal year 2018. Further, CBP apprehensions of family unit members reached about 213,400 during the first two quarters of fiscal year 2019 alone—approximately a 33 percent increase over the entire previous fiscal year. Cumulatively, along the southwest border, CBP apprehensions of family unit members reached about 599,000 apprehensions from fiscal year 2016 through the second quarter of fiscal year 2019 (see fig. 2). As shown in figure 3, CBP data indicate that apprehensions of family unit members grew from about 22 percent of total southwest border apprehensions in fiscal year 2016 to about 51 percent of such apprehensions during the first two quarters of fiscal year 2019. CBP data indicate that, during this period, OFO apprehensions of family unit members at U.S. ports of entry accounted for approximately 24 percent of all such CBP apprehensions. Border Patrol apprehensions of family unit members between ports of entry accounted for approximately 76 percent of all such CBP apprehensions. About 63 percent of CBP’s total family unit member apprehensions occurred in just three Border Patrol sectors in Texas and Arizona (see fig. 4). CBP data indicate that most apprehensions of family unit members from fiscal year 2016 through the second quarter of fiscal year 2019 were of Central American nationals and that the majority of children in family units were under the age of 12. Figure 5 shows that from fiscal year 2016 through the second quarter of fiscal year 2019, the vast majority of these apprehensions—about 82 percent—were nationals of Guatemala, Honduras, or El Salvador. Additionally, about 10 percent of apprehensions of family unit members were of Mexican nationals and approximately 7 percent were nationals of other countries. From fiscal year 2016 through the first two quarters of fiscal year 2019, CBP apprehensions of children in family units totaled approximately 327,600. About 72 percent of these apprehensions were of children under the age of 12 when apprehended by CBP, and about 32 percent were under age 5 (see table 3). Border Patrol also maintains information in its data system that allowed us to analyze the composition of family units, that is, whether the family unit was headed by a male or female and how many children were in the family unit. Most family units apprehended by Border Patrol—about 85 percent—consisted of a single parent travelling with a single child. Most family units were led by a single female in fiscal year 2016; however, the number of households led by single males increased and, for the first two quarters of fiscal year 2019, accounted for almost half of the family units Border Patrol apprehended. Appendix II contains additional information about the composition of family units, including the immigration history of adult family members. From fiscal year 2016 through the first two quarters of fiscal year 2019, CBP placed an increasing percentage of family unit members into full removal proceedings. Specifically, CBP data indicate that around 46 percent of all apprehensions of family unit members in fiscal year 2016 resulted in the family unit members receiving Notices to Appear before an immigration court, which initiate full removal proceedings; around 88 percent received Notices to Appear during the first two quarters of fiscal year 2019. Conversely, CBP data indicate that CBP placed a decreasing percentage of all apprehensions of family unit members into expedited removal proceedings during this period. Specifically, the percentage declined from about 42 percent of all apprehensions of family unit members in fiscal year 2016 to about 6 percent during the first two quarters of fiscal year 2019. CBP officials stated that, since the volume of family units apprehended at the border increased in 2018, they have placed fewer family unit members into expedited removal proceedings, for which detention is generally mandatory, due to limited space for family units in ICE’s family residential centers. Department of Homeland Security’s (DHS) Migrant Protection Protocols In January 2019, DHS introduced the Migrant Protection Protocols, also referred to as the “Remain in Mexico” program, at selected ports of entry and, as of March 2019, within certain Border Patrol sectors. Under this policy, CBP issues eligible individuals, including family unit members, Notices to Appear before an immigration court, thereby initiating full removal proceedings. After CBP agents and officers complete processing duties, DHS officials stated that CBP returns the individuals to Mexico to await their court proceedings, rather than releasing them into the interior of the United States. According to CBP officials, through the end of fiscal year 2019, CBP processed approximately 44,200 individuals—among which about 30,100 individuals, or 68 percent, were family unit members—using the Migrant Protection Protocols. While ICE generally has the authority to detain individuals for the duration of their full removal proceedings, CBP and ICE officials stated that ICE faces constraints that typically prevents it from doing so for family units. Specifically, the limited amount of space at family residential centers is reserved for those family units placed in expedited removal. Therefore, according to ICE and CBP officials, with few exceptions, during the period of our review, family units placed into full removal proceedings were released into the United States to await their court proceedings. According to ICE officials, even if there was more detention space for family units, there are other constraints that would prevent ICE from detaining family unit members (placed into full removal proceedings at any point) for the duration of their court proceedings. Specifically, children may generally only be held in federal immigration detention for 20 days pursuant to the Flores Agreement. Due to the duration of full removal proceedings, most full removal proceedings take longer than 20 days. According to Border Patrol and OFO data, CBP separated at least 2,700 children from April 19, 2018, through the second quarter of fiscal year 2019. As we discuss later in this report, CBP may have separated additional children from their parents during this period and not recorded this information in its data systems. As a result, we are reporting approximate, rounded figures on family separations. Specifically: Border Patrol updated its data system to track family unit separations on April 19, 2018, and issued written guidance to its agents about these changes on May 7, 2018, and August 2, 2018. From April 19, 2018, through March 31, 2019, Border Patrol data indicate that agents separated at least 2,670 children. OFO updated its data system to track family unit separations on June 26, 2018, and issued guidance on these changes to its officers on June 29, 2018. From June 30, 2018, through March 31, 2019, OFO data indicate officers separated at least 30 children. As shown in table 4, CBP data indicate that the number of family unit separations was highest between April 19, 2018 and June 27, 2018, due to DHS’s response to the U.S. Attorney General’s April 2018 zero tolerance policy (see table 4). CBP data also indicate that a small percentage of all children that arrived in family units—fewer than 2 percent—were separated from their parents during these time frames. Appendix II provides additional information about the characteristics of family units separated by CBP. Border Patrol and OFO data indicate that the reasons for these family unit separations varied. Regarding Border Patrol, as of April 19, 2018, agents were able to record a family separation and select from options to explain the reason for it in Border Patrol’s automated data system. Border Patrol data indicate that the reasons that 97 percent of the adults and children separated from April 19 through June 27, 2018, were because agents referred the parent to the Department of Justice for criminal prosecution on charges for criminal history or other reasons, or due to a prior immigration violation(s) and a removal order. Table 5 shows the reasons for family separations indicated in Border Patrol data from April 19, 2018 through March 31, 2019. Regarding OFO, as of June 30, 2018, officers were to record the reason for any family unit separation with the child’s record in OFO’s automated data system. From June 30, 2018 to March 31, 2019, OFO data indicate that about 50 percent of adults and children were separated due to the criminal history of the adult or a child safety concern. Table 6 shows the reasons for family separations indicated in OFO data from June 30, 2018 through March 31, 2019. Since 2015, Border Patrol and OFO have issued policies and updated procedures regarding the information to be collected about family units and family separations, increasing the amount of data collected for family units. For example, Border Patrol and OFO have updated their data systems to better track the number of individuals apprehended as part of family units and to record when and why family separations occur. Specifically, Border Patrol updated its data system in October 2015 to track whether individuals were apprehended as members of a family unit and again in 2018 to track family separations. On October 2, 2015, the Chief of the Border Patrol issued policy guidance requiring agents to process family units together in its data system with a unique identifier called a “family unit number,” which links the records of parents and children apprehended together. Border Patrol updated its system on April 19, 2018 and on August 2, 2018, to track the number of separated adults and children and the reasons for the separations, and issued guidance to its agents about these updates. New Border Patrol agents also receive mandatory training on, among other topics, recording information into agency data systems, including procedures specific to family units. OFO updated its data system to track whether children under the age of 18 arrived as part of a family unit and whether they were separated from a parent (or other family member) with whom they arrived. OFO headquarters officials stated the updates were made during fall 2015. On June 29, 2018, OFO issued a policy memorandum that, among other things, required officers to track family separations in OFO’s data system, and announced system updates to allow officers to select a separation reason. This and subsequent data system updates allowed officers to identify which separations were temporary (in which the family was reunited while still in OFO custody), and which were permanent (resulting in OFO referring a child to ORR), according to OFO officials. All OFO officers hired as of March 2011 receive mandatory training on certain processing procedures, including recording information into agency data systems. As of October 2019, OFO’s data system does not have the capability— such as by using a family unit number—to link the records of noncitizen parents and children apprehended together and thus cannot determine the total number of adults involved in family separations. OFO is implementing a new data system across all ports of entry that includes a function to link the records of parents and children in family units using a unique identifier. According to OFO officials, OFO began developing the new data system in August 2017 and, as of October 2019, has implemented it at 90 ports of entry, none of which are land ports of entry along the southwest border. In June 2019, OFO officials stated they planned to train OFO officers on the new data system at land ports of entry along the southwest border in late summer 2019, but as of October 2019 that timeline had been delayed due to the high volume of family units apprehended that summer. According to OFO headquarters officials, they expected to deploy the new system to locations along the southwest border on an ongoing basis as conditions allow. It is too soon to determine whether the new data system will enable OFO to link children apprehended at ports of entry to their parents and allow for OFO to track the total number of family members separated in its aggregated data. It is also too soon to determine whether the new system will provide OFO officers with more readily available information that could help reunify separated family units, if necessary. Since October 2015, some Border Patrol and OFO documents have included inconsistent guidance on how agents are to define a family unit for processing purposes. CBP’s 2015 policy defines a family unit to include one or more non-U.S. citizen juvenile(s) accompanied by his or her parent(s) or legal guardian(s), which Border Patrol agents confirmed is the agency’s official definition that should guide how its agents process family units. However, as shown in table 7, certain Border Patrol policy documents since October 2015 have also stated that “all members of the apprehended family unit must be non-criminal and/or non-delinquent and have no history of violence or substance abuse.” As a result, individuals in family units that Border Patrol considered criminal, delinquent, or to have a history of violence or substance abuse may not have been included in Border Patrol’s aggregated data on apprehended family units and family separations (once the agency began tracking separations in April 2018), because agents did not define and process them as family units. We raised these inconsistencies to Border Patrol headquarters officials in April 2019, and they acknowledged that certain policy and training documents contained inaccurate definitions and guidance, which could have led some agents to process certain parents and children separately, without a family unit number to link their records. Specifically, they stated that the language requiring that “all members of the family unit must be non-criminal and/or non-delinquent, and have no history of violence or substance abuse” should not be included in Border Patrol’s definition of a family unit. In addition, officials noted that any guidance directing agents to process a family unit separately in the data system, as a single adult and UAC rather than linked together with a family unit number, due to a planned prosecution referral is inconsistent with Border Patrol’s processing procedures. They stated this was an oversight and not an intentional change to the agency’s official definition as indicated in CBP’s 2015 policy. The Border Patrol headquarters officials were unsure of how often the inconsistent definitions and guidance may have led agents to incorrectly process family units. On the basis of our analysis of Border Patrol and ORR data, we found evidence that agents processed some family units separately, as single adults and UAC, without a family unit number or record of their separation. Specifically, for children apprehended from June 28, 2018 through March 31, 2019, we compared ORR numbers on UAC involved in family separations to Border Patrol apprehension data on separated children. During that period, ORR records indicated that DHS separated 396 children, while Border Patrol apprehensions data indicated that it separated 180 children. Border Patrol headquarters officials confirmed that the discrepancy we identified between Border Patrol data and ORR records may be attributable, in part, to the agents processing family units incorrectly and separately, without assigning them a family unit number. To better understand the discrepancy between the ORR and Border Patrol data, we selected a random, nongeneralizable sample of 40 ORR records for UAC involved in family separations from June 28, 2019 through March 31, 2019, and found matches for each of the children in Border Patrol apprehensions data. In 14 of the 40 selected ORR records, Border Patrol data indicated the agent had not recorded the child as a member of a family unit linked to a parent’s record with a family unit number. Thus, Border Patrol agents had not recorded the subsequent separation when agents referred the children to ORR as UAC. A Border Patrol headquarters official stated that it is also likely that some agents were processing family units separately, rather than linking them with a family unit number, from May to June 2018 when agents were referring parents for criminal prosecution in response to the April 2018 zero- tolerance policy. The official stated that agents may not have realized that assigning a family unit number was necessary to track the separation in the Border Patrol data. During the course of our audit, we discussed this issue with Border Patrol and, as a result, Border Patrol issued new guidance to its sectors in April 2019 with an updated definition of family units consistent with CBP policy. According to Border Patrol officials, Border Patrol also removed previous policy documents, with the incorrect definitions and guidance, from a website accessible to all Border Patrol agents. However, as of late November 2019, Border Patrol training materials still direct agents to process a parent and child separately, without a family unit number, if a family member has a history of criminality, delinquency, violence, or substance abuse, or if Border Patrol plans to prosecute the parent. This definition and guidance, inconsistent with CBP policy, has been included in training provided to all new agents at Border Patrol’s basic training program since at least October 2017. According to officials from the Border Patrol Academy, which is responsible for updating training materials in coordination with program officials, they plan to update the training materials in 2020. In the meantime, since September 2019, the Border Patrol Academy has been providing trainees with a handout that includes a definition of family units consistent with CBP policy. Regarding OFO, we also found that since 2012, training materials for new officers have included a definition of a family that is inconsistent with CBP and OFO policy. Specifically, OFO training materials issued in January 2012—and in use as of November 2019—define a “family group” as “a juvenile who is accompanied by closely related adults (parent, grandparent, brother, sister, or legal guardian)” and considers the juvenile to be UAC if “the juvenile is accompanied by relative(s) not closely related.” The training document does not include a definition for “family unit.” However, other key OFO policy documents issued subsequently define family units in a way that is consistent with CBP policy—namely, a February 2016 memo on processing family units in OFO’s data system and a June 2018 memo on tracking family separations. We raised the discrepancy in OFO’s training materials with OFO headquarters officials in June 2019. OFO officials were unsure whether this definition had led any officers to incorrectly process adults and children as family units when they did not meet CBP’s definition of a family. OFO headquarters officials stated the training materials were inconsistent with CBP and OFO policy, and officials from CBP’s Office of Training and Development stated they updated the training materials and provided them to OFO in late November 2019. However, as of December 2019, CBP had not provided us with the updated materials to verify that the revisions are consistent with CBP policy. Standards for Internal Control in the Federal Government states that management should design control activities, including by providing the right training tools to achieve operational success. In addition, in GAO’s Guide for Strategic Training and Development Efforts, we have reported that senior managers need to continually observe and assess how changes, such as in policies or practices, may affect the agency’s training needs. This is one way, among others, to help ensure that the agency has a framework to achieve its mission. Border Patrol and OFO officials acknowledged the need to update training materials with definitions and guidance, consistent with CBP policy; they explained that they had not yet done so due to the considerable time and coordination it requires. Issuing updated training materials that reflect CBP policy would help CBP ensure that Border Patrol agents and OFO officers are processing family units appropriately and tracking all separations. CBP has policies and procedures for assessing the validity of family units, but does not have written guidance to help ensure that these cases are well documented, as required by CBP policy. CBP has policies and procedures for assessing the validity of family unit relationships. During processing, Border Patrol and OFO officials said that it is standard practice for agents and officers to assess whether (1) adults and children apprehended together meet CBP’s definition of a family unit and (2) whether agents and officers deem the claimed family relationships to be potentially invalid. A CBP policy issued on June 27, 2018, states that “fraudulent claims of family relationships” should be processed under “current CBP policies and procedures.” In practice, this means that agents and officers are to consider the validity of family relationships on a case-by-case basis with the information they have available at that time, according to Border Patrol and OFO headquarters officials. For example, these officials stated that agents and officers review any available documentation, such as birth certificates, presented by individuals; monitor interactions between adults and children to assess whether interactions are typical of that of a parent and child; and generally use their law enforcement training, such as interviewing skills, to help assess the validity of family relationships. Border Patrol and OFO officials noted that, in some instances, individuals have admitted to falsely posing as a family, while other times agents and officers have to make an assessment based on the totality of the information available to them. In accordance with CBP policy, Border Patrol and OFO are to generally hold individuals no longer than 72 hours, so Border Patrol and OFO officials stated they must assess the validity of the family units based on available information during the time they have individuals in custody. According to Border Patrol and OFO officials and documents, they have observed cases in which (1) a family unit claims a child is under 18 years of age, but agents suspect the child is older, and thus they do not meet CBP’s definition of a family unit, or (2) the adult claims to be the parent, but Border Patrol has concerns that the adult is another family relation, such as an aunt or older sibling, or the adult is not related to the child at all. In June 2019, the Acting Secretary of Homeland Security testified that CBP identified “almost 4,800 migrants this year” in family units that CBP agents and officers determined to be “fraudulent” in nature. In cases when Border Patrol agents or OFO officers, with approval from their respective supervisors, assess that the relationship of a family unit may not be valid, the child is to be processed as a UAC and transferred to ORR. Specifically, according to Border Patrol and OFO officials and documents, agents and officers are to indicate in their data systems that the adult and child were separated and the reason why, and then refer the child to ORR as a UAC. For Border Patrol, this process involves removing the family unit number linking their records. Border Patrol and OFO do not consider these cases to be family separations, since CBP assessed that the individuals may not be part of a valid family unit. According to CBP’s June 2018 policy, if a child arrives with an adult claiming to be the child’s parent, a supervisory-level OFO or Border Patrol official must give approval before an agent or officer transfers a child to ORR as a UAC. According to Border Patrol and OFO headquarters officials, if an adult wishes to appeal CBP’s assessment, the adult may raise the issue with ICE officers when transferred to an ICE detention facility. Border Patrol headquarters officials told us that its agents generally explain to the adults that Border Patrol is processing them separately from the children they arrived with due to concerns about the validity of the family relationship. OFO headquarters officials told us that OFO does not generally notify adults when processing the adults and children in potentially invalid family units because they stated they do not want to jeopardize the safety of the child if they suspect fraud, smuggling, or trafficking. According to Border Patrol and OFO officials, they may separate adults and children who they are concerned might not be valid family units to ensure the safety of the child—for example, if agents and officers cannot be certain that the child has not been a victim of trafficking by the accompanying adult. Further, Border Patrol, OFO, and ICE officials stated that ICE and ORR are better positioned to further investigate these cases if an adult refutes CBP’s assessment that the family unit was invalid, because CBP must generally hold individuals for a short period. In addition, ICE and ORR are the agencies most involved in reunifying family units, when appropriate. CBP began tracking the number of potentially invalid family units in 2018. On April 19 and June 29, 2018, Border Patrol and OFO, respectively, issued guidance about updates to agency data systems and issued guidance to enable agents and officers to record potentially invalid family units. That is, if the appropriate Border Patrol and OFO managers give approval, agents and officers separate potentially invalid family units, and record the separation and the reason for it in agency data systems, according to Border Patrol and OFO officials and documents. More specifically: Border Patrol agents are to delete the family unit number from the parents’ and children’s records, and indicate the reason from options that include “child is over the age of 18,” “no family relationship,” or “no family relationship–prosecuted.” OFO officers are to indicate that a child was separated from the adult with whom they arrived, and are to indicate the reason as “fraudulent relationship.” Border Patrol and OFO officials noted observing cases of potentially invalid family units, and Border Patrol data indicate an increase in the number since Border Patrol began tracking the cases in April 2018. Specifically, during our fall 2018 visits to ports of entry and border stations in Texas and California, Border Patrol and OFO officials stated they have observed suspected or confirmed cases of adults falsely claiming to be a child’s parent, including occasional instances of seeing the same child apprehended multiple times, but with different adults claiming to be their parents. From April 19, 2018 through March 31, 2019, CBP data indicate that CBP referred at least 921 children to ORR (918 by Border Patrol and 3 by OFO) due to CBP’s concerns that the family relationships were potentially invalid. During the same period, Border Patrol data also indicated that 2,245 adults were processed separately from the children with whom they were apprehended due to concerns about the validity of the family relationships. By comparison, Border Patrol data indicated that agents processed 256,743 adults and children in valid family units during this period. However, from July 1, 2018 through March 31, 2019, the number of individuals Border Patrol assessed as part of potentially invalid family units grew at a faster rate than the number of individuals apprehended in valid family units. Specifically, the average monthly increase in adults and children Border Patrol assessed to be in potentially invalid family units rose by about 70 percent per month, on average, during this period. Meanwhile, Border Patrol data indicate the rate of increase for adults and children in valid family units was about 53 percent per month, on average. However, some of the family units that CBP assessed to be potentially invalid are subsequently found to be valid, according to ORR and ICE officials. According to ORR officials and records of UAC involved in family separations from June 28, 2018 through June 28, 2019, ORR was aware of only 46 cases in which CBP referred a child to ORR care because CBP had assessed the family unit to be invalid. In at least 10 of those cases, the family was later determined to be valid and the child reunited with his or her separated parent, according to ORR officials, as of June 2019. Anecdotally, ICE headquarters officials stated that there are occasionally cases in which CBP referred a child to ORR because agents or officers assessed the family to be an invalid family unit, but ICE or ORR later determined the family was valid and eligible to be reunified. For example, ORR’s records on family separations included instances in which the validity of family relationships was determined through DNA testing. ICE officials stated that its officers are able to conduct additional research about the validity of family relationships, as needed, once an adult has been transferred to its custody and the child to ORR. However, ICE does not track how often potentially invalid family units are later assessed to be valid and reunited, and, therefore, could not provide an exact number of how often this has occurred. DHS and ICE officials have tracked the outcomes of some deployments of ICE officers to help CBP assess the validity of family relationships. Specifically, the Acting Secretary of Homeland Security testified before the Committee on Oversight and Government Reform in the House of Representatives on July 18, 2019 that CBP agents and officers referred 2,475 family unit members they suspected had invalid family relationships to go through an additional assessment by ICE officers who, among other training and skills, have specialized forensic interviewing skills. The ICE officers assessed 352 of the 2,475 individuals—approximately 14 percent—to be invalid family members. Additionally, an ICE official also testified before the Senate Committee on Homeland Security and Governmental Affairs on June 26, 2019, and described a May 2019 pilot that involved voluntary rapid DNA testing for some individuals. According to this official, 16 of the 84 family units tested, around 19 percent, proved not to be the parent of the child with whom they arrived. According to ICE officials, those family units it determined to have valid family relationships while still in CBP custody, based on the available evidence, remained together as a family unit and were not separated. On June 27, 2018, the CBP Commissioner issued a policy memorandum requiring that “fraudulent claims of parental or legal guardianship relationship” should be “well-documented to support such claims”; however, CBP does not have guidance to clarify how agents and officers are to fulfill that requirement. Border Patrol and OFO headquarters officials indicated that taking the aforementioned steps to record information in agency data systems meets the CBP policy requirement for documentation. In addition, according to Border Patrol and OFO officials, agents and officers also record details of the apprehension on the Form I- 213, which is required for all individuals. However, neither Border Patrol nor OFO has guidance about whether or what details about a family unit being assessed as potentially invalid should be included on the Form I- 213. Learning about the details of these cases and why CBP made its assessment is important to ICE officials in the event an adult refutes the assessment and ICE must take additional steps to determine the validity of the family relationship. ICE officials can view the information CBP agents and officers record on the Form I-213, since ICE officers can access the form in a database it shares with CBP. However, the headquarters official responsible for coordinating ICE’s family and juvenile programs stated that the level of detail included in the forms varies by location and the narrative often does not include details about the reason why CBP considered a family unit potentially invalid. ORR officials also stated this information would be helpful to ORR because it may be relevant to the decisions ORR staff make for UAC, such as selecting sponsors. ORR intake staff told us that the documents they receive from CBP accompanying UAC referrals typically do not contain narrative information about agents’ or officers’ concerns about potentially invalid family relationships. While CBP tracks cases on potentially invalid family units in its data systems, this tracking does not (1) document the circumstances to support the assessment of invalidity or (2) provide complete and timely information for ORR and ICE to help them to fulfill their responsibilities, including to review cases in which CBP initially determined a family to be invalid but further investigation is needed. Rather, CBP’s data systems only track the assessment of invalidity by allowing agents to select that as a reason from a set of options, but do not track the circumstances to support that assessment. However, CBP policy directs that these cases be “well-documented to support such claims.” Standards for Internal Control in the Federal Government state that management should use quality information to achieve the entity’s objectives. In doing so, management should identify the information needed to achieve objectives and address risks, and should consider the expectations of both internal and external users. Providing guidance on what narrative information Border Patrol agents and OFO officers are to document on a child’s and the accompanying adult’s Forms I-213 about potentially invalid family units could help better ensure that the events are well-documented to support such assessments, in accordance with CBP policy. Further, this could help ensure that ICE and ORR officials have relevant details they need to make decisions for adults and children in their custody, including reuniting valid family units, where appropriate, before adults are removed from the United States. CBP, Border Patrol, and OFO have developed policies and procedures for those agents and officers responsible for recording and approving family separations; however, Border Patrol and OFO do not have sufficient controls to ensure (1) Border Patrol agents are accurately and consistently recording family separations in their data systems, (2) Border Patrol and OFO’s data systems accurately capture separation reasons that are consistent with CBP policy, and (3) required forms include sufficient details about the circumstances of the separations. Regarding policies and procedure for family separations, according to CBP’s June 2018 policy, a Border Patrol watch commander, or equivalent position, must approve every family separation. Border Patrol and OFO officials told us that higher-level officials, such as Border Patrol sector chiefs or Port Directors, are often involved in decisions to separate family units. A 2015 CBP policy requires that agents and officers record family separations in agency data systems. Further, after updating data systems to track family separations in 2018, as previously described, Border Patrol and OFO issued written guidance to agents and officers with specific instructions on how to record family separations in its data systems. For example, Border Patrol issued guidance about how to record family separations in its data system in May 2018, August 2018, and April 2019. In addition, Standards for Internal Control in the Federal Government state that management should use quality information to achieve the entity’s objectives, and identify the information needs to address risks. In doing so, managers should also consider the expectations of both internal and external users when collecting information. Further, changes in conditions affecting the entity and its environment often require managers to revise the internal control system, on a timely basis to maintain effectiveness. Our analysis of Border Patrol and ORR data indicates that Border Patrol agents have not accurately and consistently recorded family separations in the data systems. Specifically, we reviewed a random, nongeneralizable sample of 40 ORR records for UAC involved in family separations between June 28, 2018, and March 31, 2019 and found matches for all 40 of the children in Border Patrol apprehensions data. Among the 40 records, we identified cases in which agents had not documented the family separation in Border Patrol’s data system, as required by CBP and Border Patrol policy. Specifically, Border Patrol data indicated the agent had not processed the separation in the Border Patrol data system for 10 of the 40 UAC involved in family separations. That is, in these cases, Border Patrol agents processed the parents and children together with a family unit number, but did not take the necessary steps in the system to separate them and document the reason why the separation occurred. We shared the results of our analysis with Border Patrol officials, and these officials acknowledged that the discrepancy between Border Patrol and ORR data on family separations may be attributable, in part, to human error—that agents had not correctly recorded family separations in Border Patrol’s data system. However, the officials were unsure of the extent of the problem. Thus, it is unclear whether Border Patrol has accurate records of all separated parents and children in its automated data system. Border Patrol officials stated that data entry errors may have grown with increased processing demands and strained resources faced by Border Patrol as the volume of family units apprehended increased in fiscal years 2018 and 2019. However, as mentioned, federal internal control standards provide that changes in conditions—such as increased processing demands agents faced during periods of increased apprehensions of family units—often require managers to revise the internal control system. Developing and implementing additional controls to ensure that Border Patrol agents accurately record family separations in the data system, consistent with CBP and Border Patrol policies, would better enable Border Patrol to maintain complete and accurate information on all family separations. For example, an additional control could be to require Border Patrol or OFO managers conducting supervisory review of each apprehension to check that family separations have been accurately recorded in the data system. CBP, Border Patrol, and OFO have policies and procedures in place for those agents and officers responsible for approving family separations and recording the reasons in agency data systems. On June 27, 2018, the CBP Commissioner issued a memorandum to the Chief of the Border Patrol and to the Executive Assistant Commissioner of OFO to provide direction on complying with the June 26, 2018, federal court order in Ms. L. v. ICE that generally prohibits the government from separating parents from their children, to include potential reasons that may warrant continued family separations. Specifically, the memorandum states that separations may occur only for the following reasons: (1) the parent has criminal convictions for violent misdemeanors or felonies, (2) CBP plans to refer the parent for a felony prosecution, (3) the parent poses a danger to the child, or (4) the parent has a communicable disease. On June 29, 2018, OFO issued a policy memorandum reiterating the potential separation reasons included in CBP’s June 27, 2018 policy memorandum. According to Border Patrol headquarters officials, Border Patrol did not issue any further implementing guidance. Border Patrol and OFO officials stated that agents and officers are to use all available information to determine whether a family separation is warranted. Such information may include available birth certificates, personal observations of the family unit’s behavior, results of background checks for criminal and immigration history, and results from available medical assessments. In some instances, Border Patrol and OFO officials stated that agents and officers may not always have complete information, such as when a database indicates a parent’s arrest but does not indicate whether he or she was convicted of the charge, but that agents and officers are to weigh the totality of the circumstances. For situations in which agents and officers are unsure whether to separate a family, CBP’s policy states that agents and officers should contact their local Office of Chief Counsel for guidance. Although Border Patrol and OFO data systems allow agents and officers to select among options to indicate the reason for a family separation, the reasons available in the systems do not fully align with CBP policy. For example, Border Patrol’s data system does not include an option that indicates the parent poses a danger to the child—one of the reasons articulated in the Commissioner’s June 2018 memorandum. Table 8 shows how the separation reasons available in Border Patrol and OFO data systems compared with the potential separation reasons established in CBP’s June 2018 family separations policy. Border Patrol and OFO headquarters officials stated they were unsure why the separation reasons available in the data systems do not fully align with CBP policy on family separations, but stated that the data system reasons have an implicit link to CBP policy. They stated that Border Patrol and OFO officials review and approve each family separation to ensure it meets CBP policy. In addition, OFO headquarters officials stated that it issued guidance in June 2018 that reiterated CBP’s policy on potential reasons for family separations. However, as illustrated in table 8, it is sometimes not clear how separation reasons in Border Patrol’s and OFO’s data systems align with CBP policy. For example, Border Patrol’s option for “family member prosecuted for other reasons” does not provide enough information to determine whether Border Patrol is referring a parent for the prosecution of a felony, as required by CBP policy. Both Border Patrol and OFO have previously changed separation reasons in agency data systems, and in June 2019 Border Patrol officials stated they continue to analyze the need for updates. As of October 2019, these officials stated that Border Patrol and OFO do not have any current plans to update the separation reasons in their data system. CBP officials who conduct supervisory review of files and approve family separations rely, in part, on the information agents and officers record in Border Patrol and OFO data systems, in conducting reviews and sharing information, according to Border Patrol and OFO officials. Updating Border Patrol’s and OFO’s data systems to ensure that options for separation reasons clearly align with CBP policy could help ensure that CBP makes decisions about family separations in accordance with CBP policy and that data CBP collects reflects that. CBP’s policies related to family separations do not include written requirements that agents and officers record a description of the family separation. However, Border Patrol and OFO officials stated that they expect agents and officers to record the circumstances surrounding family separations on a narrative section of each family member’s Form I-213. Yet we found that Border Patrol agents are not consistently recording detailed information about family separations on the Form I-213—the official record of the apprehension. Specifically, we analyzed a nongeneralizable sample of Forms I-213 for family units whom Border Patrol separated and found that, for most of the family separation cases, one or more of the selected forms had missing or inconsistent information in the narrative descriptions. Specifically, we reviewed a sample of Forms I-213 for 23 family separation cases, involving 27 children and 25 parents. These separations occurred across each of the Border’s Patrol’s nine southwest border sectors between June 28, 2018 and March 30, 2019. In particular, we assessed (1) whether the forms included a reason for the separation, (2) whether the descriptions of the cases provided enough information to determine whether or not the reason met CBP policy, and (3) whether the information recorded for each separation case was consistent across the parents’ and children’s forms. On the basis of our review of the forms, we found there was missing or inconsistent information on one or more of the family members’ forms for 18 of the 23 separated family units. Specifically, we found for three of the 23 family separations, there was no indication that a separation had occurred on one or more of the family members’ forms; for 20 of the 23 family separations, all of the family members’ forms included some indication of a family separation; Seven of the 25 parents’ forms and seven of the 27 children’s forms did not contain a narrative description explaining why the separation occurred; and 17 of the 25 parents’ forms included sufficient narrative information to determine whether the separation met CBP policy; 12 of the 27 children’s forms included enough information to make that determination. In addition, even among those forms with sufficient information to determine whether the reason met CBP policy, we found inconsistencies. For example: Three parents’ and four children’s forms included information that implied that the parent could potentially present a danger to the child, but the actual separation reason noted on the form was something different, such as the parents’ criminal history. For example, the criminal history information provided on one parent’s Form I-213 included information about an arrest for kidnapping, but did not include evidence that the arrest resulted in a conviction, making it difficult to determine whether the separation aligned with CBP policy and, in particular, what reason the separation would fall under. For nine of the 23 family separations, the separation reason was listed as the parent’s criminal history on one of the family member’s forms, but there was missing or inconsistent information on the other family members’ forms. For example, in one instance, the father’s Form I- 213 indicated he had been convicted of sexually assaulting a 12-year- old child, but there was no separation reason and no information about the parent’s criminal history provided on the child’s form. According to ICE officials responsible for monitoring family separations, and reunifying family units where necessary, the narrative information on the Form I-213 is ICE officers’ primary source of information about the circumstances of a family separation. ICE officers need detailed information, according to officials, to help conduct additional research to confirm whether a separation was warranted or respond to requests for information from ORR. In addition, ORR officials told us that they would benefit from CBP recording certain information on a child’s Form I-213— such as the type and timing of a parent’s criminal conviction or whether the parent may pose a danger to the child—and sharing that information, to better inform ORR’s decisions about where and with whom to place UAC when they leave ORR custody. However, CBP has not issued guidance on what descriptive details surrounding family separations agents and officers are to record on the Form I-213, based on our review of CBP documents. In addition, Border Patrol officials stated that they do not have written guidance for agents about what information should be captured on the Form I-213. Conversely, OFO issued guidance stating that the Form I-213 must be annotated with the reason for the family separation, the name of the approving manager, and, at a minimum, the biographical information and “A-number”—a unique identifier for noncitizens apprehended by CBP—of the parent(s) and children. Border Patrol and OFO headquarters officials acknowledged that the level of detail documented on the Form I- 213 about separations may vary by agent or officer, and rely on their supervisory review process to ensure that family separations are consistent with CBP policy. Border Patrol headquarters officials attributed missing separation reasons or inconsistent information about the circumstances of the family separations on the Forms I-213 to multiple factors. Specifically, they acknowledged that Border Patrol has not issued guidance specifying what descriptive details agents should include on the forms, and does not have, for example, specific information that supervisors check for during their review of each individual’s file. In addition, Border Patrol headquarters officials noted that there have been great demands placed on Border Patrol agents to expedite processing during periods of high numbers of family units apprehended and crowding at Border Patrol facilities. However, as noted previously, federal internal control standards state that changes in conditions affecting the entity and its environment— like an increase of family units apprehended along the southwest border—often require management to change the entity’s internal control system, as existing controls may not be effective for meeting objectives or addressing risks under changed conditions. As of October 2019, Border Patrol and OFO had no plans to (1) implement additional controls to ensure that reasons for family separations are included on individuals’ Form I-213 or (2) issue guidance to agents and officers about what descriptive information about family separations they should record on the forms. Developing and implementing additional controls to ensure that Border Patrol agents and OFO officers include a reason for the family separations on the parent’s and child’s Forms I-213 could help CBP ensure its agents and officers are separating family units in accordance with CBP policy. For example, an additional control could be to require the Border Patrol or OFO manager reviewing the information recorded on the Form I-213 to check that certain information, such as the specific separation reason with relevant details, has been included. In addition, without additional guidance on what specific details Border Patrol agents and OFO officers are to include in the narrative information about the family separation events on the parent’s and child’s Forms I-213, ICE and ORR do not have complete or consistent information to use in determining when it may be necessary to reunify family units in accordance with the Ms. L. v. ICE court order. ICE has procedures for processing family units whom CBP apprehended and for releasing family units from ICE custody (see fig. 6). According to ICE field office officials, upon referral by CBP, ICE officers generally review the family unit’s files to ensure that CBP agents and officers completed the forms sufficiently and, if not, ICE officers can return the case to CBP. For example, ICE officers typically ensure that the appropriate family unit member signed his or her copies of paperwork provided by CBP. Additionally, according to ICE field office officials, ICE officers have the discretion to decline the transfer of a family unit that they determine is not suitable for detention in a family residential center or for release. When ICE accepts CBP’s referral of a family unit and receives custody from CBP, ICE officers are to enter information about each family unit member in ICE’s data system, both for family units that ICE plans to detain and those it plans to release. ICE’s data system pulls some information from CBP’s data systems. For example, ICE officers can find basic biographic information about individual family unit members apprehended by Border Patrol by searching for an individual by his or her “A-number,” a unique identifier. In addition, ICE officers are to enter new information, such as the location(s) where officers detained or released the individual family unit members and the documents officers served to them, among other things. For information about the family unit members that ICE detained at its family residential centers, see appendix II. Family units placed into expedited removal by CBP and detained in ICE family residential centers—who express an intention to apply for asylum, a fear of persecution or torture, or a fear of return—undergo screenings conducted by an asylum officer. These screenings occur during detention and are to determine whether one or more family unit members have a credible fear of persecution or torture. The outcome of the screening (and review by an immigration judge, if requested after the screening) determines whether ICE will remove the family unit from the United States or release the family unit into the interior of the country to pursue immigration relief or protection in full immigration proceedings. Additionally, as stated previously, children may generally only be held in federal immigration detention for 20 days pursuant to the Flores Agreement. Thus, if members of the family unit do not receive a credible fear determination within 20 days, ICE generally releases the family unit into the interior of the United States with a notice to appear before an immigration court, which initiates full immigration proceedings. From fiscal year 2015 through fiscal year 2018, ICE data indicate that 99 percent of family unit members who were detained in one of ICE’s family residential centers were subsequently released by ICE into the interior of the United States. For additional information about the outcomes for family unit members detained in ICE family residential centers, see appendix II. According to ICE headquarters and field office officials, while a family unit is at a family residential center, ICE officers typically assist family units with their post-release plans by asking heads-of-household to identify contacts in the United States, such as relatives, that the family unit can stay with after leaving ICE custody. These contacts pay for the family unit’s travel expenses if the family cannot purchase bus tickets, for example, and ICE officers help coordinate these plans and typically drive the family unit to the bus station upon release, according to ICE officials. For family units who are not placed in a family residential center, ICE’s procedures for assisting them with their post-release plans have varied based on local conditions. ICE headquarters and field office officials explained that, prior to October 2018, when the volume of family units arriving at the southwest border began to increase significantly, ICE officers sometimes coordinated post-release plans for family units that did not stay at a family residential center. However, officials stated ICE has not had the resources to help family units with post-release plans since that time and instead has generally relied on nongovernmental organizations for this assistance. When ICE releases family units from its custody to await immigration court proceedings, ICE officers generally enroll the family unit’s head-of- household in its Alternatives to Detention program. The program uses technology, such as ankle monitoring devices, to track the movement of the adult family unit members. ICE field office officials stated that the availability of ankle monitoring devices and the volume of family units arriving at the southwest border can impact whether or not ICE enrolls a family head-of-household in its Alternatives to Detention program. In addition to ankle monitoring devices, most family units are also released on orders that require heads-of-household to report telephonically or in- person to ICE officers once they reach their destination in the United States. ICE officials stated the level of continued supervision by ICE officers is at the discretion of the ICE officer in charge of the family unit’s case and may also be dependent on a variety of factors, such as whether the family unit entered the United States at or between ports of entry, whether the family unit received a positive credible fear determination, and the head-of-household’s prior criminal and immigration record. ICE relies on a manual process to track family unit separations that occur in ICE custody, but does not systematically record this information in its data system. ICE officers are to report all separations that occur in ICE custody to the headquarters office responsible for coordinating family and juvenile programs. ICE headquarters officials in that office compile the information received from the field offices to populate a spreadsheet, which they use to track all separations that occur in ICE custody. In addition, according to ICE officials at headquarters, officers are to include narrative information about the separation and the approving official’s name in a comments field in the parent’s and children’s records in the data system. According to ICE officials, the narrative information in the comment field is not searchable within ICE’s data system and ICE does not have a mechanism, such as a drop-down menu, to systematically record a family unit separation or the reasons for any separations that occur in ICE custody. Thus, ICE cannot pull data from its system to track such separations. ICE headquarters officials stated that these methods are not an efficient and effective means to have readily available data on family separations that occurred in ICE custody. According to ICE policy for detained parents, detained parents maintain their parental rights during removal proceedings. In particular, if ICE is removing a parent from the United States, field office directors or their designees are to accommodate, to the extent practicable, the detained parent’s efforts to make arrangements for his or her minor child or children, including for the children to be removed with the parent. As such, before removing an adult from the United States, ICE officers are to check the individual’s paper A-file, and specifically the individual’s Form I- 213, for any indication the adult arrived with a child, according to ICE headquarters officials. In addition, according to ICE officials, ICE officers are to review the individual’s record in ICE’s data system where ICE officers would be alerted to whether the individual had ever been separated from a child. Given the limitations in ICE’s data system, officers would need to know to review the narrative information in the comments field within the individual’s records to determine whether he or she had been separated from a child in ICE custody; however, none of ICE’s guidance documents explain that officers are to look for such information in the narrative comments field. Further, ICE officials told us that officers are not required to check the spreadsheet maintained at ICE headquarters or contact headquarters officials prior to removing adults from the United States. As of November 2019, ICE headquarters officials stated they are working with the ICE data unit to create a new module that would enhance ICE’s ability to link and track family units in its data system, including capturing information on families that ICE separates. According to ICE officials, ICE has established a project team for this effort and hopes to deploy the updates in the fourth quarter of fiscal year 2020. However, ICE did not provide documentation with details, such as a project plan with time frames for deploying these system updates, to verify these plans. Standards for Internal Control in the Federal Government states that management designs the entity’s information system and related control activities to achieve objectives and respond to risks. Further, management designs the entity’s information system and the use of information technology by considering the defined information requirements for each of the entity’s operational processes. Given that ICE did not provide documentation with details about planned changes to ICE’s data system, it is too early to determine whether and when ICE’s planned system enhancements will include a mechanism that allows ICE officers to systematically track family separations that occur in ICE custody. Without a mechanism in its data system to systematically track the family units it separates, ICE is unable to ensure that separated parents who are subject to removal are able to make arrangements for their minor child or children (including being removed with them), as provided in ICE policy . DHS and HHS have developed interagency agreements for the transfer and placement of UAC between the two departments; however, information sharing gaps remain. In 2015, we reported that the interagency process to refer UAC from DHS to HHS was inefficient and vulnerable to errors because it relied on emails and manual data entry. In addition, each DHS component (Border Patrol, OFO, and ICE) submitted referrals for UAC to HHS’s ORR in a different way. To increase the efficiency and improve the accuracy of the interagency referral and placement process for UAC, we recommended the Secretaries of Homeland Security and Health and Human Services jointly develop and implement a documented interagency process with clearly defined roles and responsibilities for all agencies involved in the referral and placement of UAC in HHS shelters. DHS and HHS concurred with our recommendation. Since our 2015 report, DHS and HHS developed two documents to guide interagency procedures related to the processing of UAC. Specifically, in April 2018, HHS and DHS established a memorandum of agreement regarding information sharing for UAC. In addition, on July 31, 2018, DHS and HHS issued a Joint Concept of Operations to memorialize interagency policies, procedures, and guidelines related to the processing of UAC. According to the April 2018 memorandum of agreement, among other things ICE and CBP are to provide ORR with information at the time of the referral and documents when the child is transferred to ORR, including whether the child was traveling with other individuals and the Form I-213, so that ORR can make informed decisions for the child. Specifically, once a child has been transferred to ORR, the agency begins the process of identifying a potential sponsor for the child and, when a potential sponsor is identified, ORR requests information about that sponsor. At this step, according to the memorandum of agreement, ICE is to conduct a screening of the potential sponsor that includes, at a minimum, a biographic criminal check of national databases, a check for warrants of arrest, and an immigration status check. DHS is to provide HHS with information necessary to conduct suitability assessments for sponsors, including that which HHS would not otherwise have access. In addition, to the extent permitted by law, and consistent with policy, DHS is to report to ORR the results of any investigations it conducts that are relevant to ORR’s determinations concerning the care and placement of UAC. According to the July 2018 Joint Concept of Operations, ICE or CBP should use ORR’s data system to refer UAC to ORR whenever feasible. If ORR’s data system is not available, DHS may email ORR a referral form along with any supporting documentation. DHS is also to provide ORR with specific documents, including the Form I-213, when the child is transferred to ORR. In the event a child is separated from a parent or legal guardian, CBP or ICE is to enter this information into ORR’s data system, according to the Joint Concept of Operations. CBP or ICE is also to include contact information for parents, legal guardians, or adult relatives, as this information can assist in ORR’s reunification process, if needed. ORR is to contact the child’s family to, among other things, determine whether the child has a potential sponsor who resides in the United States, and to facilitate visitation and contact with family members, regardless of their immigration status. Finally, DHS is to preserve the unity of families during repatriation, according to the Joint Concept of Operations. The memorandum of agreement and Joint Concept of Operations state the roles and responsibilities of DHS and HHS and their components and describe some of the information to be shared between the agencies regarding the placement of UAC, among other things. However, DHS and HHS officials’ statements indicate that, in practice, they have not resolved long-standing differences in opinion about whether and how agencies are to share information, and what type of information is needed to inform decisions about the care and placement of UAC, including those processed as UAC after separation from a parent. We found that DHS has not consistently provided information and documents to ORR as specified in the memorandum of understanding and the Joint Concept of Operations. Further, ORR officials identified additional information they believe ORR needs from DHS at the time of referral (or soon thereafter) to inform their decisions about placing children with sponsors and reunifying separated families, when necessary. With regard to information sharing expectations established in the interagency agreements, as of September 2019, we found that certain documents were not being shared or mechanisms for sharing information were not being used consistently. For example, Border Patrol has taken steps since our 2015 report to improve its referral process, so that Border Patrol’s referral information is uploaded directly into ORR’s data system, in keeping with Joint Concept of Operations requirements. However, the referral screens in Border Patrol and ORR data systems do not fully align, which has required ORR headquarters staff to manually enter some required information into the ORR data system. That is, Border Patrol’s referral screen does not include many of the fields—areas to input specific information—included in ORR’s referral screen. Border Patrol and ORR officials offered different perspectives on why the information on the referral screens in the data systems do not align. Specifically, ORR officials stated that Border Patrol has not updated its referral screen to match updates that ORR has made. For example, in July 2018, ORR added a checkbox in its data system for DHS agencies to indicate whether a UAC had been separated from a parent, as necessary. Border Patrol took steps in October 2018 to similarly update its referral screen, so the indication of a family separation would be automatically uploaded to ORR’s data system with the referral. However, additional steps must be taken by ORR for its data system to upload the information, according to Border Patrol officials. Meanwhile, if ORR staff see some indication of a family separation in the Border Patrol referral form, such as in a narrative text field, ORR staff will typically add that information to the records in their data system manually. Border Patrol has not taken additional steps to update other parts of its system’s referral screen to align with ORR’s data system because ORR’s data system does not comply with DHS security standards, according to Border Patrol officials. ORR officials said they had not been made aware of any security concerns. However, concern about system security standards is a long-standing issue that we noted in our 2015 report. As of October 2019, Border Patrol and ORR did not have any plans to collaborate further to improve automated referrals for UAC. Further, as of October 2019, ORR officials told us that ICE and OFO officials are not consistently accessing the ORR data system to submit a referral for a UAC. Specifically, ICE and OFO officers in certain locations use ORR’s data system to submit a referral infrequently and instead use a form, which ORR last updated in 2013, that they attach to emails to refer UAC. However, ORR officials stated their expectation is that email referrals are to be used only occasionally, such as if DHS officials encounter technical problems using ORR’s data system. ICE and OFO stated that their officers only rarely make referrals to ORR and sometimes face constraints that prohibit them from using ORR’s data system to submit the referral. For example, ICE officials stated that officers generally use ORR’s data system for referrals, but that, on some occasions, the officer’s password to access ORR’s data system has expired due to infrequent use, and they must email the referral. In addition, OFO and ICE officials stated that their officers who have access to the ORR data system to make referrals are not always available, so in those instances, other officers must email a referral form to ORR. OFO and ICE headquarters officials were unsure how often their officers used email to send ORR referrals, rather than directly accessing ORR’s data system. ORR officials also stated that even when ICE and OFO use ORR’s data system to submit the referral, consistent with the Joint Concept of Operations, the officers are not consistently marking the separations checkbox in ORR’s data system for those children involved in family separations. As a result of these challenges, ORR officials said they must often manually enter referral information from ICE or OFO into the ORR data system, including any indication of a family separation or that the child was apprehended with an adult. ORR officials also stated that DHS—CBP and ICE—is not routinely submitting the child’s Form I-213 to ORR, as specified by both interagency agreements. Border Patrol and OFO headquarters officials stated they have concerns about sharing sensitive information, including in referral forms or on the Form I-213, with ORR headquarters or contracted shelter staff because they are not law enforcement officers. ORR headquarters officials stated that they have worked with other federal partners to ensure that only ORR officials with the proper authorization receive sensitive materials. These officials said they are interested in working with DHS to set up a similar process so ORR can receive the information it needs to make decisions for UAC. For example, ORR headquarters officials stated that they would explore options for updating DHS and HHS data systems so the child’s Form I-213 could be shared directly between data systems. This would help ensure that only ORR staff who have the proper authorization will have access to them, according to HHS officials. In addition, DHS and HHS provided different perspectives on the expected information sharing procedures included in the interagency agreements. For example, ORR headquarters officials stated they interpret existing interagency agreements to apply to information sharing on all UAC, regardless of whether they were apprehended alone or with an adult. By contrast, Border Patrol headquarters officials stated that the interagency agreements apply to UAC involved in family separations, but not those children referred to ORR after Border Patrol assessed a family relationship to be invalid. In addition, ICE headquarters officials stated that the interagency agreements were drafted to reflect the circumstances of children apprehended alone, not separated children or those CBP assesses to have invalid family relationships. ICE officials also stated they no longer believe the April 2018 memorandum of agreement is valid for any UAC, because it was developed to address a process ORR no longer requires. ORR identified additional information sharing needs—some not covered by existing interagency agreements—to inform decisions regarding the care and placement of UAC. Specifically, this information includes details about the circumstances of family separations, and information about adults who were apprehended with children (who subsequently were designated as UAC). ORR officials stated that ORR and ICE require this information, collected by DHS, to (1) assess potential sponsors for placement of UAC and (2) to reunify eligible separated families. Assessing Potential Sponsors. ORR officials stated that ORR needs additional information about parents and other adults accompanying a child (who is later designated as a UAC) at the time of apprehension to assess all potential sponsors with whom UAC will be placed as they await immigration proceedings in the United States. However, the Joint Concept of Operations contains limited details about what information should be shared between DHS and HHS about relevant adults. For example, the agreement states that ICE and CBP will provide ORR with contact information for parents, legal guardians, or adult relatives. However, the agreement does not, for example, require DHS to share the details of an adult’s criminal history information to ORR. In addition, Border Patrol headquarters officials stated that agents typically would not alert ORR to any concerns about invalid family relationships, as they do not believe that information is relevant. ORR officials stated they need detailed information about an accompanying adult to assess whether they could potentially pose a danger to the child, and this is not addressed in the Joint Concept of Operations. However, ORR officials stated that this information is often not included in DHS’s referrals for UAC, and ORR sometimes learns about an accompanying adult from a child after placement in an ORR shelter. Reunifying Eligible Separated Family Units. To ensure compliance with the federal court injunction in the Ms. L. v. ICE litigation, ORR officials stated that they need to know enough details about (1) family separations or (2) situations in which CBP had concerns a family relationship was invalid, to determine whether there are any family units potentially eligible for reunification. If DHS and HHS determine that a parent will be reunified with a child, ORR is responsible for (1) verifying the validity of the family relationship and (2) determining whether the parent is fit or poses a danger to the child, according to ORR officials. For family unit reunifications, ORR has relied, in part, on the determinations made by DHS when the family was separated, according to these officials. However, ORR officials stated the information DHS provides about family separations is generally limited or provided inconsistently, often without enough detail for ORR to assess whether the family unit may be eligible for reunification. For example, the referral might state a family separation is due to the parent’s criminal history, but ORR must follow up with ICE to specify the charge, determine whether the adult was convicted, or learn the date of the event. In addition, ORR may conduct family reunifications in accordance with ORR policies and procedures in other situations. For example, there have been cases in which families were separated, but DHS later dropped criminal charges against a parent it planned to prosecute, or a parent has completed a hospitalization that required the parent to be separated from his or her child. According to ICE policy, when ICE is removing a parent from the United States, that parent has the right to determine whether a minor child will be removed with him or her. ORR officials stated that, according to ORR policies and procedures, if the child is to be removed with the parent, ORR must assess whether (1) the family relationship is valid and (2) whether the parent presents a danger to the child. However, ORR officials stated that if this information was not provided at the time of referral, they must reach out to ICE officials to collect it. Further, ORR headquarters officials stated that ICE has removed adults from the United States who wished to be removed with their child or children in ORR custody, before ORR could complete its assessment. However, neither ICE nor ORR could determine exactly how often that had occurred or in exactly what time frame these removals had occurred. DHS and HHS officials provided different perspectives on these information sharing challenges not covered within existing interagency agreements. ORR takes additional steps to collect information from ICE and CBP that ORR is not routinely receiving at the time of referral. This can extend the time that children spend in ORR custody, according to ORR officials. If ORR staff conducting intake duties have questions about UAC and any accompanying adults, ORR headquarters officials told us they typically first contact the local CBP officials who processed the apprehension. In April and August 2019, ORR officials said that some Border Patrol sectors are more responsive than others and that limited and inconsistent information sharing by DHS about separated children has led to delays in placement and release decisions for UAC. ORR staff also reach out to ICE’s field office juvenile coordinators or ICE headquarters officials responsible for juvenile and family management. For example, ORR and ICE headquarters coordinate on a weekly basis via email to assess whether family separations are in compliance with federal court orders in the ongoing Ms. L. v. ICE litigation. Specifically, since February 2019, ORR and ICE have shared a spreadsheet tracking UAC who may have been involved in a family separation, according to ORR and ICE headquarters officials. Further, ICE officials said they gather additional information, such as more details about the reason for a family separation from the Form I-213 or by reaching out to CBP officials. They provide some of this information to ORR, as ICE officials noted that they recognize ORR needs such information to assist in its decision-making for UAC. ICE headquarters officials noted that they have found ways to provide more detailed information to ORR without sharing sensitive law enforcement information. It is through this vetting process that ICE and ORR assess potential family separations to reach a confirmed number of cases and the reasons for them, according to ICE and ORR officials. ORR headquarters officials stated that, from their perspective, it would be more efficient if CBP or ICE provided this information directly into ORR’s data system at the time of referral, where possible, rather than sharing a spreadsheet via email. Specifically, ORR headquarters officials stated that they have experienced delays in releasing a child to a sponsor due to missing information about a parent or the inability to notify a parent in ICE detention about sponsorship decisions. By contrast, Border Patrol and OFO headquarters officials noted concerns about sharing sensitive information with ORR, particularly for adults apprehended with UAC. Border Patrol officials stated, for example, that Border Patrol does not share sensitive law enforcement information with a third party such as ORR. According to ICE headquarters officials, sometimes ICE officers conduct additional research after a child is referred to ORR, such as if CBP was unable to collect certain information before making a separation decision. ICE officials stated that, for their purposes, the current information sharing procedures in place are sufficient, but noted that ICE has added staff resources to keep up with the demands of current information sharing procedures. Specifically, until May 2019, there was one ICE headquarters official, 2in the juvenile and family management unit, responding to all of ORR’s requests, and that ICE added another staff person to assist in responding to ORR’s requests. As of October 2019, there were no plans to discuss further these information sharing concerns, according to ORR, CBP, and ICE officials. Leading practices of high-performing organizations include fostering collaboration both within and across organizational boundaries to achieve results. Further, agencies should work together to establish a shared purpose and goals; develop joint strategies or approaches that complement one another; and ensure the compatibility of policies, procedures, and other means to operate across agency boundaries. We have previously reported that written agreements, such as a memorandum of understanding or interagency agreements, can help facilitate collaboration by articulating roles and responsibilities, among other things. These types of written agreements are most effective when they are regularly updated and monitored, as we reported in 2012. While issuing the April 2018 memorandum of agreement and July 2018 Joint Concept of Operations were important steps toward addressing the weaknesses we identified in our 2015 report, additional actions are needed to fully address our recommendation and increase the efficiency and improve the accuracy of the interagency referral and placement process for all UAC. In addition, further DHS and HHS collaboration about information sharing methods and ways to enhance interagency agreements would better position ORR to make informed and timely decisions for UAC, including those separated from adults with whom they were apprehended. As the number of CBP apprehensions of family units has risen markedly in recent years, DHS has developed policies and procedures for processing family units. For example, since 2015 CBP has introduced policies and procedures for collecting information about family units, which has increased the data it collects, including on family separations. However, DHS continues to face challenges in ensuring that it accurately and consistently tracks information about family units, including those it separates. Specifically, CBP training includes definitions of and guidance for processing family units that are inconsistent with CBP policy. Issuing updated training materials with correct definitions of and guidance for processing family units would help CBP ensure that its agents and officers are accurately tracking family units and, where applicable, family separations. In addition, CBP has policies and procedures related to concerns about the validity of a family unit, but it does not have written requirements about what information on these cases Border Patrol agents and OFO officers are to record. Without additional guidance about what details CBP agents and officers are to record on the required Form I-213, these cases will not be well documented, as required by CBP policy. Further, ICE and ORR officials do not have sufficient information to make decisions for the adults and children involved, including determining when reuniting valid family units is necessary. CBP has developed policies and procedures related to family separations, but additional controls would help Border Patrol and OFO ensure that information about these cases is accurately and consistently captured. By developing and implementing additional controls for tracking family separations—such as requiring checks during supervisory review that separations were documented properly—Border Patrol could better ensure it has accurate information about these cases, consistent with CBP and Border Patrol policies. Further, some of the options for separation reasons in Border Patrol’s and OFO’s data systems do not fully align with CBP policy. Without updating the reasons agents and officers have available to select from, CBP is not well positioned to determine whether its officials are separating family units for reasons consistent with CBP policy. In addition, during our review of selected Forms I-213 for a sample of separated family units, we found that agents did not always include the reason for the separation or include a detailed description of the circumstances of the case. Developing and implementing additional controls to check that Border Patrol agents document family separations and why they occurred on family unit members’ Forms I-213 could help Border Patrol ensure its agents are separating family units in accordance with CBP policy. Additionally, without additional guidance on what specific information about the circumstances of the family separations Border Patrol agents and OFO officers are to include on the parent’s and child’s Forms I-213, ICE and ORR do not have sufficient information to determine, among other things, when family reunifications are required. During our review of ICE’s policies and procedures for processing family units, we found that it does not systematically track the family units it separates in its data system. By updating its data system to do so, ICE would be better able to ensure that separated parents, who are subject to removal, are able to make arrangements for their minor child or children, including being removed with them, consistent with ICE policy. While DHS and HHS have developed written interagency agreements related to the transfer and care of UAC, as we recommended in 2015, we found that information sharing gaps between the two agencies remain. As such, continuing their efforts to address our prior recommendation to jointly develop and implement a documented interagency process for all agencies involved in the referral and placement of UAC could help DHS and HHS increase the efficiency and improve the accuracy of these processes for UAC. Moreover, additional DHS and HHS collaboration about information sharing would help provide ORR with additional information, including about accompanying adults, to make informed and timely decisions for UAC. We are making a total of nine recommendations, including six to CBP and one each to ICE, DHS, and HHS. Specifically: The CBP Commissioner should issue updated Border Patrol and OFO training materials that reflect the correct definition of a family unit and guidance for recording that information. (Recommendation 1) The CBP Commissioner should provide written guidance to Border Patrol agents and OFO officers about what narrative information should be recorded on the child’s and the accompanying adult’s Forms I-213 to document cases in which CBP determines that a parent–child relationship may be invalid. (Recommendation 2) The CBP Commissioner should develop and implement additional controls to ensure that Border Patrol agents accurately record family unit separations in its data system. (Recommendation 3) The CBP Commissioner should update Border Patrol’s and OFO’s data systems to ensure data captured on family unit separation reasons clearly align with CBP policy. (Recommendation 4) The CBP Commissioner should develop and implement additional controls to ensure that Border Patrol agents include a narrative description of a family unit separation on the parent’s / legal guardian’s and child’s Forms I-213, including the reason for the separation. (Recommendation 5) The CBP Commissioner should provide guidance to Border Patrol agents and OFO officers on the narrative information they are to include about family unit separation events on the parent’s / legal guardian’s and child’s Forms I-213. (Recommendation 6) The ICE Director should develop and implement a mechanism to systematically track in its data system the family units ICE separates. (Recommendation 7) The Secretary of Homeland Security, jointly with the Secretary of Health and Human Services, should collaborate to address information sharing gaps identified in this report to ensure that ORR receives information needed to make decisions for UAC, including those apprehended with an adult. (Recommendation 8) The Secretary of Health and Human Services, jointly with the Secretary of Homeland Security, should collaborate to address information sharing gaps identified in this report to ensure that ORR receives information needed to make decisions for UAC, including those apprehended with an adult. (Recommendation 9) We provided a draft of this report to DHS and HHS for review and comment. DHS and HHS provided formal, written comments, which are reproduced in full in appendixes III and IV, respectively. DHS and HHS also provided technical comments on our draft report, which we incorporated, as appropriate. DHS concurred with our recommendations and described actions planned or underway to address them. For example, in response to several of our recommendations that CBP provide additional or revised guidance and training to agents and officers, DHS stated that Border Patrol issued a memo in January 2020 to clarify what information agents are to record for family unit members, potentially invalid family units, and subsequent separations, if applicable. DHS also described planned updates to OFO data systems to automatically record certain information in family unit members’ Form I-213, such as the names and identifying information of all family members apprehended together. Regarding our recommendation that CBP should update Border Patrol’s and OFO’s data systems to ensure the options for family separation reasons clearly align with CBP policy, DHS provided documentation of guidance that OFO and Border Patrol issued about data system updates. DHS requested that we consider the recommendation implemented. We will review the information and documents DHS provided to assess the extent to which CBP fully addressed this recommendation. Regarding our recommendation that ICE develop and implement a mechanism to track its separations in its data system, DHS stated that ICE has efforts underway to enable ICE officers to track separations and reunifications in its data system throughout ICE’s immigration enforcement process. DHS and HHS also both concurred with our recommendations that the agencies collaborate to address information sharing gaps identified in this report, and described plans to coordinate and reach agreement on information sharing practices. We will review the agencies’ actions and planned efforts, including any documentation provided by DHS and HHS, and the extent to which they address each of our nine recommendations. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from its issue date. At that time, we will send copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, and the Secretary of Health and Human Services. In addition, the report is available at no charge on the GAO website at https://gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or gamblerr@gao.gov. Key contributors to this report are listed in appendix V. The objectives of this report were to examine (1) what U.S. Customs and Border Protection (CBP) data indicate about the numbers and characteristics of family units who have been apprehended along the southwest border, (2) the extent to which CBP has developed and implemented policies and procedures for processing family units apprehended along the southwest border, (3) the extent to which U.S. Immigration and Customs Enforcement (ICE) has developed and implemented policies and procedures for processing family units apprehended along the southwest border, and (4) how the Department of Homeland Security (DHS) shares information with the Department of Health and Human Services (HHS) about unaccompanied alien children (UAC), including those children who initially arrived with and were separated from their parents or other adults. To address these objectives and to observe agents and officers processing families, we conducted site visits at U.S. Border Patrol stations and Office of Field Operations (OFO) ports of entry in Arizona, California, and Texas, from July 2018 to October 2018. We also visited ICE family detention facilities, known as family residential centers, in Dilley and Karnes City, Texas in February 2019. Specifically, in Tucson, Arizona we visited Border Patrol’s Tucson sector headquarters and OFO’s Tucson Field Office headquarters and the Nogales port of entry. In the San Diego, California region, we visited Border Patrol’s San Diego sector headquarters and Imperial Beach station and the San Ysidro port of entry. In the Rio Grande Valley, Texas region, we visited CBP’s Central Processing Center, Border Patrol’s McAllen station, and the Hidalgo and Brownsville ports of entry. In the San Antonio, Texas region, we visited ICE’s San Antonio field office headquarters, South Texas Family Residential Center, and Karnes County Residential Center. During these site visits, we interviewed Border Patrol, OFO, and ICE officials, observed agents and officers processing families, and toured CBP and ICE facilities, among other activities. To select these locations, we reviewed CBP data on Border Patrol and OFO apprehensions along the southwest border, including family unit apprehensions, and identified specific locations that had the greatest increase in the number of apprehensions of individuals from fiscal year 2016 to 2017. We also considered the geographical proximity of multiple CBP and ICE facilities to maximize observations. Our observations during site visits are not generalizable to all Border Patrol, OFO, and ICE operations along the southwest border, but provided us the opportunity to learn more about how policies and procedures for processing families are conducted and how CBP and ICE coordinate their efforts. In addition, to address all of our objectives, we interviewed DHS and HHS officials. Specifically, we met with DHS officials from CBP’s Office of the Commissioner and Office of Chief Counsel; Border Patrol’s Law Enforcement Operations Directorate and Strategic Planning and Analysis Directorate; OFO’s Admissibility and Passenger Programs office; ICE’s Enforcement and Removal Operations (including the Juvenile Family and Residential Management Unit, Field Operations, Alternatives to Detention, and Law Enforcement Systems and Analysis) and ICE’s Office of the Principal Legal Advisor. We also interviewed HHS officials from the offices of the Assistant Secretary for Preparedness and Response and Office of Refugee Resettlement (ORR). To address our first objective and describe what CBP data indicate about the numbers and characteristics of family units who have been apprehended along the southwest border, we reviewed record-level apprehensions data from CBP’s Border Patrol and OFO for individuals determined to be inadmissible or potentially subject to removal. We collected data for fiscal year 2016 through the second quarter of fiscal year 2019 because Border Patrol and OFO began to systematically collect data on individuals apprehended as part of a family unit in fiscal year 2016. The second quarter of fiscal year 2019 was the most current data available at the time of our review. We used “number of apprehensions” rather than the “number of individuals or family unit members” as the unit of analysis we reported because an individual may have been apprehended multiple times in the same year. The data we report on apprehensions of family unit members include individuals in family units CBP later separated (for reasons other than concerns about validity of the family relationship) from April 19, 2018, when Border Patrol and OFO began collecting data on family separations, through the first two quarters of fiscal year 2019. The record-level data we analyzed are current as of the date Border Patrol or OFO provided it to us. Specifically, Border Patrol data for fiscal years 2016 through 2018 are current as of January 2019; Border Patrol data for the first two quarters of fiscal year 2019 and selected fields for all fiscal years are current as of April 2019. OFO data for fiscal years 2016 through 2018 are current as of February 2019; OFO data for the first two quarters of fiscal year 2019 are current as of June 2019. We grouped the ages of apprehended children in family units (e.g. ages 0–4, 5–11, and 12–17) according to key agency and court documents. While most of our analysis was conducted on the apprehensions of individuals in family units, we were also able to analyze the composition of family units (i.e., as a group rather than individuals) apprehended by Border Patrol. Specifically, Border Patrol uses a “family unit number” to link the records of adult(s) and children processed as a family unit. As a result, we analyzed whether the family unit was headed by an adult male or adult female and how many children were in the family unit. We could not conduct a similar analysis for the family units apprehended by OFO, because OFO does not assign family units unique identifying numbers to link family members in its data system. As a result, we were unable to report on the composition of family units that OFO encountered. As part of our analysis of CBP data, we determined the number of family unit members Border Patrol and OFO data indicated as separated from April 19, 2018 through March 31, 2019. We selected this time frame because Border Patrol began to systematically collect data on family separations in its data systems on April 19, 2018, and the second quarter of fiscal year 2019 was the most current data available at the time of our review. Our analysis of the reasons for family separations is based on the data recorded by agents and officers in Border Patrol’s and OFO’s data systems. During the period of our review, Border Patrol’s and OFO’s data systems included options for agents and officers to choose from to explain the reason for the separation, including, for example, “family member prosecuted – criminal history” and “family member prosecuted – other reasons.” These reasons, and the numbers of separations for each reason, reflect CBP data and may not match the information about separations (including numbers of, reasons for, and timeframes of separations) that DHS reported to a federal court in response to related litigation, such as Ms. L. v. ICE. According to court filings, the information provided in response to that litigation was based on a manual review of multiple federal datasets and reflect categories as required by the litigation. We excluded family separations indicated in CBP data as temporary from our analysis. We also reported separately on the number of adults and children who were apprehended together, but whom CBP assessed to have potentially invalid family relationships and thus processed separately, as CBP does not consider these family separations. To assess the reliability of CBP data, we completed a number of steps, including (1) performing electronic testing for obvious errors in accuracy and completeness, such as running logic tests; (2) reviewing existing information about the data and the systems that produced them, such as relevant training materials for Border Patrol agents and OFO officers who use agency data systems; and (3) discussing data entry issues and data limitations with Border Patrol and OFO officials. We also received demonstrations on the data systems from Border Patrol and OFO officials at headquarters. The limitations and determinations of reliability for the Border Patrol and OFO data are discussed in more detail below. Border Patrol data. We identified a small number of Border Patrol apprehension records that had the same date of apprehension and unique identifier, known as the “A-number.” It is possible that these apprehension records represented one apprehended individual that Border Patrol agents processed as two apprehensions. These records constituted less than 1 percent of the almost 2.4 million apprehension records we analyzed. We included these apprehension records in our analysis because Border Patrol considers them unique apprehensions and because their small number does not materially affect our analysis. We did not include a small number of records (less than 1 percent of apprehensions of family unit members) that had a family unit number but did not meet CBP’s definition of a family unit in our analysis of records of family unit members. For example, a small number of family unit member records did not include a date of birth, so we could not determine whether the individual was an adult or child (i.e., under or over the age of 18 years). For our analysis of the reasons for family separations, we found a small number (18) of Border Patrol records that included more than one separation reason, so we could not distinguish which reason led to a permanent family separation. Thus, we excluded these records from our analysis of the reasons for family separations. According to Border Patrol headquarters officials and documents, in situations in which only one of the adults in a two-parent family was separated, the child or children would remain with the other adult as an intact family unit (and the child would not be designated a UAC and transferred to the custody of ORR). As such, in these situations, we included the separated adults in our reported numbers of separated family unit members, but did not include associated remaining family units in our analysis of separated family units. We found 18 records for family units that included one adult and one child, with one of the family unit members separated. According to Border Patrol’s procedures, in the event a family separation occurs, both family unit members are to be processed in the data system as “separated.” We included these records in the number of family unit members, but did not include them in our analysis of separated family unit members, as it was unclear from the records whether or not the family unit was separated. We identified data reliability issues with Border Patrol’s data on family separations, as described in our report. When reporting these data, we rounded down to the nearest increment of five, and described relevant data using modifiers such as “at least” because of possible missing information. This enabled us to report on the Border Patrol data that we determined were sufficiently reliable for our purposes. OFO data. For the OFO data, we excluded approximately 11 percent of all apprehension records (including single adults, UAC, and parents and children that arrived as part of a family unit) from our analyses because we could not confirm an A-number, for those apprehensions. Among the apprehension records missing an A-number, 44 percent were cases in which OFO officers paroled the individuals and, according to OFO officials, officers are not required to assign an A-number to these individuals. In addition, 47 percent of the records with a missing A- number were cases that involved individuals withdrawing their applications for admission into the United States, in which OFO officers have discretion whether or not to assign an A-number. According to OFO officials, additional records with missing A-numbers may be due to human error during data entry or problems with the data system saving this information in the database that OFO used to pull the data. Finally, we collapsed 153,025 apprehension records into 71,986 apprehension records because we determined that they were duplicate records for the same individual and the same apprehension, based on factors such as A- number, birth date, and date and time of apprehension. As a result, we determined that we could not present precise figures for analyses that include OFO data and instead provided approximations throughout the report. We rounded all data and figures on OFO apprehensions, including where OFO’s data inform CBP-data and figures, down to the hundreds place. As an exception, for the much-smaller number of OFO family separations, as compared with total apprehensions, we rounded the figures by increments of five, and described relevant data using modifiers such as “at least” because of possible missing information. This enabled us to report on the OFO data that we determined were sufficiently reliable for our purposes. With the previously described modifications, we determined that the Border Patrol and OFO data were sufficiently reliable to generally describe the number and demographic characteristics of family units apprehended by CBP along the southwest border. To address the second objective, on the extent to which CBP has developed and implemented policies and procedures for processing family units—including how CBP defines family units, assesses the validity of family relationships, and determines whether family separations are warranted—we reviewed CBP, Border Patrol, and OFO policy documents, training materials, and other guidance documents in effect from October 2015 through December 2019. For example, we reviewed CBP’s 2015 National Standards on Transport, Escort, Detention, and Search policy, as well as Border Patrol’s data system processing guidance and Border Patrol and OFO policies and procedures on how agents are to record family separations in agency data systems, among other documents. We compared CBP, Border Patrol and OFO policies and procedures to Standards for Internal Control in the Federal Government related to identifying, analyzing, and responding to change; designing control activities to achieve objectives and identify risks; and using quality information to achieve objectives. We also compared Border Patrol definitions for family units, and processes and guidance for tracking family units, invalid family units, and family unit separations against CBP and Border Patrol policy. To evaluate how Border Patrol recorded information for family units apprehended from June 28, 2018 through March 31, 2019, we also selected a sample of ORR records for UAC involved in family separations and compared them to Border Patrol apprehensions data for the same children. Specifically, we selected a small, random, nongeneralizable sample of 40 ORR records for UAC involved in family separations. We then matched all 40 selected records to Border Patrol apprehensions data, using unique identifiers. Our findings are not generalizable due to the size of our sample, so we cannot use our findings to assess the magnitude of the issues we identified in Border Patrol data. We limited the records from which we selected our sample to those ORR records that included an A-number, a unique identifier, for the adult separated from the child in ORR custody, since Border Patrol tracks its separation reasons in the adult’s records. Finally, we compared this information with CBP’s October 2015 National Standards on Transport, Escort, Detention, and Search policy, which states that family separations must be documented in the appropriate data systems. We also assessed information against federal internal control standards, which call for management to identify and use quality information to achieve the entity’s objectives and address risks, among other control activities. To describe how Border Patrol agents document the reasons for and circumstances of each family separation case, we reviewed a nongeneralizable sample of the DHS Form I-213, Record of Deportable/Inadmissible Alien (Form I-213), which is a form that agents are required to complete for each individual CBP apprehends. Specifically, Border Patrol provided us with Forms I-213 for the adults and children involved in the three most recent instances of family separation from June 28, 2018 through March 30, 2019, in each of Border Patrol’s nine sectors along the southwest border. Two of the sectors only had one family separation during that period, so we reviewed the forms for a total of 23 family separations. We reviewed a sample of Forms I-213 prepared by Border Patrol agents, as Border Patrol separated approximately 95 percent of the family separations indicated in CBP data during the period we reviewed. We did not review a sample of Forms I-213 prepared by OFO officers, given the relatively smaller number of families separated by OFO. In addition, we reviewed a sample of forms for cases of family separations only, and did not review forms for cases in which Border Patrol determined the family relationship was invalid because Border Patrol officials told us that they do not record information about assessments of invalid family relationships on the Form I-213. Finally, we compared this information with a 2015 CBP policy that states that family separations must be documented in the appropriate data systems; a June 2018 CBP policy that includes potential reasons to warrant family separations; and federal internal control standards, which call for management to identify and use quality information to achieve the entity’s objectives and address risks, among other control activities. To address the third objective, and examine the extent to which ICE has developed and implemented policies and procedures for processing families apprehended along the southwest border, we reviewed ICE policy documents, training materials, and other guidance documents. For example, we reviewed ICE’s Juvenile and Family Residential Management Unit Field Office Juvenile Coordinator Handbook, ICE’s Family Residential Standards, ICE’s data system training manual, and ICE’s detained parent policy. We compared ICE’s processes against ICE policies and procedures and federal internal control standards, which call for management to design the entity’s information system and related control activities to achieve objectives and respond to risks. ICE data. To report on family members apprehended by CBP and detained in ICE family residential centers, we reviewed ICE detention data from June 2014, when ICE opened its first family residential center on the southwest border, through fiscal year 2018, the most current data available at the time of our review. The data for all fiscal years is current as of May 2019, when ICE provided us with record-level data to analyze. To assess the reliability of ICE’s data, we completed a number of data reliability steps, including (1) performing electronic testing for obvious errors in accuracy and completeness, such as running logic tests; (2) reviewing existing information about the data and the systems that produced them, such as relevant training materials for the ICE officers who use them; and (3) discussing data entry issues and data limitations with ICE officials. We also received demonstrations on ICE’s data system from officials at headquarters. We determined that the data were sufficiently reliable to describe the numbers and demographic characteristics of family members who were apprehended by CBP and detained by ICE at one of its family detention facilities. Additionally, we collected and reviewed data on the families whom ICE separated from July 2018 through September 2019. We selected this time frame because July 2018 is when ICE began to require its field offices to report all instance of family separations to headquarters, which tracks the information on a spreadsheet, and September 30, 2019, the end of the fiscal year. We reported the total number of family separations from the spreadsheet, but could not independently verify the number of separations in ICE’s spreadsheet because ICE does not track family separations systematically in its data system. As a result, we reported the total number of family separations, according to ICE, for context to demonstrate that most family separations occur when family units are in CBP custody. To describe how DHS shares information with HHS about UAC, including those involved in family separations, we reviewed DHS and HHS interagency agreements, including the April 2018 information sharing memorandum of agreement and July 2018 Joint Concept of Operations. Additionally, we interviewed DHS and HHS officials at headquarters and DHS officials at locations along the southwest border. We compared the information we gathered with DHS and HHS interagency agreements, which provide expectations for interagency information sharing and procedures for the care and custody of UAC. We also compared DHS and HHS information sharing practices to leading practices for collaboration among federal agencies. We conducted this performance audit from July 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix provides additional information about apprehensions of noncitizen family units by CBP’s U.S. Border Patrol and Office of Field Operations (OFO) at or between U.S. ports of entry from fiscal year 2016 through the second quarter of fiscal year 2019. It also provides additional information about family unit members who were apprehended by CBP and subsequently detained by U.S. Immigration and Customs Enforcement (ICE) at a family residential center at some point from fiscal year 2015 through fiscal year 2018. The following tables contain information on the demographics of CBP apprehensions of noncitizen family units and family unit members and the processing decisions that CBP agents and officers made for them. CBP data indicate that Border Patrol was responsible for the majority of the overall number of family unit member apprehensions by CBP from fiscal year 2016 through the second quarter of fiscal year 2019 (see table 9). CBP data indicate that family unit member apprehensions grew as a percentage of total CBP apprehensions from fiscal year 2016 through the second quarter of fiscal year 2019 (see table 10). For example, CBP data indicate that apprehensions of family unit members grew from about 22 percent of total southwest border apprehensions in fiscal year 2016 to about 51 percent of such apprehensions during the first two quarters of fiscal year 2019. CBP data indicate that most apprehensions of family unit members from fiscal year 2016 through the second quarter of fiscal year 2019 were nationals of Central American countries (see table 11). CBP data indicate that the majority of apprehensions of adult family unit members by CBP were females, while the majority of children were male (see table 12). Border Patrol’s data system collects information about the family units it apprehends. Border Patrol’s data indicate that family units that agents apprehended were generally headed by females, although the number of family units headed by males and two-parent family units increased from fiscal year 2016 through the first two quarters of fiscal year 2019 (see table 13). Border Patrol’s data indicate that most Border Patrol apprehensions of family unit members occurred in just three sectors (Rio Grande Valley, Texas; El Paso, Texas; and Yuma, Arizona) from fiscal year 2016 through the second quarter of fiscal year 2019 (see table 14). OFO data indicate that most OFO apprehensions of family unit members occurred in just four ports of entry (San Ysidro, California; El Paso, Texas; Hidalgo, Texas; and Nogales, Arizona) from fiscal year 2016 through the second quarter of fiscal year 2019 (see table 15). CBP data indicate that the majority of apprehensions of family unit members resulted in the family unit members being released into the interior of the United States with a notice to appear before an immigration court, which became increasingly common from fiscal year 2016 through the second quarter of fiscal year 2019 (see table 16). The following tables contain information on family units that CBP separated at the border. CBP data indicate that the majority of children that CBP separated from their parents from April 19, 2018 through March 31, 2019 were male (see table 17). CBP data indicate that CBP separated children that ranged in age from less than 1 year old to 17 years old from their parents from April 19, 2018 through March 31, 2019, and the majority of separated children were age 12 and over (see table 18). CBP data indicate that the majority of children that CBP separated from April 19, 2018, through March 31, 2019, were nationals from Central American countries and that more than half were Guatemalan nationals (see table 19). Border Patrol data indicate that the majority of family units that Border Patrol separated from April 19, 2018 through March 31, 2019 were headed by males who were apprehended with a single child (see table 20). Border Patrol data indicate that most adults that were separated from their children by Border Patrol from April 19, 2018, through March 31, 2019, had not been previously apprehended by CBP (see table 21). The following tables and figures contain information about the noncitizen family unit members apprehended by CBP and detained by ICE at ICE’s family residential centers from fiscal year 2015 through fiscal year 2018. ICE data indicate that from fiscal year 2015 through fiscal year 2018, ICE detained 139,098 family unit members at its family residential centers (see table 22). ICE data indicate that most child family unit members (ages 0 to 17) detained in ICE detention facilities were under the age of 13 (see table 23). ICE data indicate that the majority of adults detained at ICE’s family residential centers were female, and the gender of children detained was relatively equal between male and female (see fig. 7). ICE data indicate that the majority of family unit members detained at ICE’s family residential centers were from El Salvador, Guatemala, and Honduras, as well as Mexico (see fig. 8). ICE data indicate that the vast majority of family unit members who were detained in one of ICE’s family residential centers were subsequently released by ICE into the interior of the United States (see table 24). In addition to the contact named above, Kathryn Bernet (Assistant Director), Leslie Sarapu (Analyst in Charge), Hiwotte Amare, James Ashley, Kathleen Donovan, Michele Fejfar, Cynthia Grant, Michael Harmond, Eric Hauswirth, Stephanie Heiken, Jan Montgomery, Heidi Nielson, Kevin Reeves, and Jonathan Still made key contributions to this report.", "summary": "In fiscal year 2019, CBP reported apprehending more than 527,000 noncitizen family unit members at or between U.S. ports of entry along the southwest border—a 227 percent increase over fiscal year 2018. In April 2018, the U.S. Attorney General issued a memo on criminal prosecutions of immigration offenses, which DHS officials said led to an increase in family separations. GAO was asked to review issues related to DHS's processing of family units. This report examines (1) CBP data on apprehended family unit members; the extent to which (2) CBP and (3) ICE developed and implemented policies and procedures for processing family units; and (4) how DHS and HHS share information about UAC. GAO analyzed record-level DHS and HHS data and documents; interviewed DHS and HHS officials; and visited DHS locations in California and Texas where CBP apprehensions of family units increased in 2017. Data from the Department of Homeland Security's (DHS) U.S. Customs and Border Protection (CBP) indicate that apprehensions of family unit members (noncitizen children under 18 and their parents or legal guardians) grew from about 22 percent of total southwest border apprehensions in fiscal year 2016 to about 51 percent of such apprehensions during the first two quarters of fiscal year 2019—the most current data available. During this period, CBP data indicated that most apprehensions of family units—about 76 percent—occurred between ports of entry by the U.S. Border Patrol (Border Patrol). With regard to family separations, from April 2018 through March 2019, CBP data indicate it separated at least 2,700 children from their parents, processing them as unaccompanied alien children (UAC) and transferring them to the Department of Health and Human Services (HHS). CBP developed some policies and procedures for processing family units but does not have sufficient controls to ensure effective implementation. For example, CBP policy requires that Border Patrol agents and officers track apprehended family unit members and, if applicable, subsequent family separations in agency data systems. GAO's analysis of Border Patrol documents and data indicates that its agents have not accurately and consistently recorded family units and separations. Specifically, GAO examined a nongeneralizable sample of 40 HHS records for children involved in family separations between June 2018 and March 2019 and matched them to Border Patrol apprehensions data for these children. GAO found Border Patrol did not initially record 14 of the 40 children as a member of a family unit (linked to a parent's record) per Border Patrol policy, and thus did not record their subsequent family separation. GAO found an additional 10 children among the 40 whose family separations were not documented in Border Patrol's data system as required by CBP policy during this period. Border Patrol officials were unsure of the extent of these problems, and stated that, among other things, data-entry errors may have arisen due to demands on agents as the number of family unit apprehensions increased. Thus, it is unclear the extent to which Border Patrol has accurate records of separated family unit members in its data system. Further, Border Patrol agents inconsistently recorded information about the reasons for and circumstances surrounding family separations on required forms. Developing and implementing additional controls would help Border Patrol maintain complete and accurate information on all family separations. DHS's U.S. Immigration and Customs Enforcement (ICE) is, among other things, responsible for detaining and removing those family units apprehended by CBP. ICE officers are to determine whether to accept or deny a referral of a family unit from CBP for detention in one of ICE's family residential centers, release family unit members into the interior of the United States, or remove family unit members (who are subject to final orders of removal) from the United States. ICE has procedures for processing and releasing family units from ICE custody. However, with regard to family unit separations, ICE relies on a manual process to track separations that occur in ICE custody (generally at one of ICE's family residential centers) and does not systematically record this information in its data system. Without a mechanism to do so, ICE does not have reasonable assurance that parents whom ICE separated from their children and are subject to removal are able to make arrangements for their children, including being removed with them, as provided in ICE's policy for detained parents. In 2018, DHS and HHS developed written interagency agreements regarding UAC. However, DHS and HHS officials stated they have not resolved long-standing differences in opinion about how and what information agencies are to share related to the care and placement of those children, including those referred to HHS after a family separation. GAO found that DHS has not consistently provided information and documents to HHS as specified in interagency agreements. HHS officials also identified additional information they need from DHS, about those adults apprehended with children and later separated, to inform their decisions about placing children with sponsors and reunifying separated families, when necessary. Increased collaboration between DHS and HHS about information sharing would better position HHS to make informed and timely decisions for UAC. GAO is making eight recommendations to DHS and one to HHS. Among them, CBP should develop and implement additional controls to ensure that Border Patrol agents accurately record family unit separations in data systems. GAO also recommends that ICE systematically track in its data system the family units ICE separates. Further, DHS and HHS should collaborate about information sharing for UAC. DHS and HHS concurred with the recommendations.", "document_type": "gao"}
{"report": "Interior has oversight responsibility for the development of federal oil and gas resources located under more than 260 million surface onshore acres, 700 million subsurface onshore acres, and 1.7 billion offshore acres in the waters of the Outer Continental Shelf. In this capacity, Interior is authorized to lease federal oil and gas resources and to collect the royalties associated with their production. BOEM has leasing authority in offshore waters, including the U.S. Gulf of Mexico. BOEM schedules lease sales on a 5-year planning basis. In January 2017, the Secretary of the Interior finalized BOEM’s 2017-2022 Outer Continental Shelf Oil and Gas Leasing Proposed Final Program, which included information for 10 planned lease sales in the Gulf of Mexico. BOEM has traditionally held two lease sales per year in the Gulf of Mexico region—one for the Central Planning Area and one for the Western Planning Area. However, beginning with Lease Sale 249 in August 2017, BOEM transitioned to offering all available tracts in the Gulf of Mexico at each of its twice-yearly lease sales. OCSLA, as amended, directs BOEM to establish minimum bid levels, rental fees, royalty rates, and other related fees to assure receipt of fair market value to the U.S. government for lands leased on the Outer Continental Shelf and the rights conveyed by the federal government. OCSLA directs BOEM to manage the leasing program in a manner that considers economic, social, and environmental value, including the potential impact of oil and gas exploration on other resource values of the Outer Continental Shelf. Subject to the requirement to assure receipt of fair market value, BOEM has the authority to change certain lease terms within the oil and gas fiscal system. Specifically, BOEM has broad authority to change bid terms for offshore leases, including the royalty rate, the bonus bid structure, minimum bid amounts, lease duration, and rental terms within parameters defined in OCSLA. Prior to each lease sale, BOEM publishes a Final Notice of Sale that contains the specific conditions and terms applicable to any leases sold at the lease sale, including rental rates, minimum bid amounts, and royalty rates, each of which may vary by water depth. In some cases, lease terms have been defined in law. For example, in 1995, Congress passed the Outer Continental Shelf Deep Water Royalty Relief Act, which waived or reduced the amount of royalties that companies would otherwise be obligated to pay on the initial volumes of production from certain deep water tracts leased from 1996 through 2000. In implementing the act for leases sold in 1996, 1997, and 2000, BOEM specified that royalty relief would be applicable only if oil and gas prices were below certain levels, known as “price thresholds,” with the intention of protecting the government’s royalty interests if oil and gas prices increased significantly. BOEM did not include these same price thresholds for leases it issued in 1998 and 1999. Figures 1 and 2 below show federal revenue from offshore oil and gas leases from 2006 through 2018. Annually and in aggregate, royalties constitute a majority of revenue from offshore oil and gas leases, followed by bonus bids. Industry develops oil and gas resources on federal lands within the context of broader energy markets. Conditions in those markets— including commodity prices, competition, and technological developments—can change rapidly. For example, the price of oil on the open market has been volatile, ranging from about $39 to $136 per barrel (in 2018 dollars) over the last decade. In addition, companies must weigh potential offshore oil and gas investments against other potential oil and gas investment options domestically and overseas. For example, some companies have expanded the sphere of their development activities to waters off Mexico, areas which now compete for investment against the remaining oil resources in the Gulf of Mexico. Furthermore, technological innovations—such as developments in seismic imaging and in drilling technology—have affected where companies are able to locate and develop resources in subsea areas. According to bureau documentation, BOEM is to evaluate the adequacy of bids in two phases of analysis—economic viability assessments and tract valuations—that incorporate departmental economic and geologic models. BOEM’s bid evaluations are intended to ensure that the bureau awards leases only when the associated bid amount represents at least fair market value to the federal government. According to bureau documentation, after each lease sale, BOEM evaluates the economic viability of tracts receiving bids to determine if they require additional analysis before BOEM decides whether to accept or reject the bids. To make these initial assessments, BOEM first develops thresholds of the minimum quantity of oil or gas that must be present to generate revenue that would offset exploration and development costs—known as the “break-even threshold”—at the given water depth, among other factors. Then, for each tract that receives a bid, BOEM estimates a range of how much oil or gas may be on the tract— known as the tract’s “resource potential”—using geological and geophysical data. This process incorporates collecting and analyzing the most recently available seismic exploration and well data and any information gathered from drilling in that geographical area. BOEM is then to categorize tracts as viable or nonviable by comparing the bureau’s estimated resource potential against the relevant break-even threshold. Nonviable tracts are those for which BOEM’s resource estimates are below the break-even threshold, meaning they are not likely to have enough oil and gas to be profitably explored, developed, and produced. For tracts that BOEM concludes are nonviable, BOEM accepts the highest bid received as long as that bid is higher than the minimum acceptable bid amount. Conversely, viable tracts are those that exceed BOEM’s economic viability threshold and that BOEM considers as having the potential to be profitably explored, developed, and produced. BOEM subjects these tracts to further economic analysis in its next phase, tract valuation. According to bureau documentation, for tracts determined to be economically viable, BOEM is then to conduct a more detailed economic analysis to determine if the high bids represent fair market value. Specifically, BOEM develops an acceptable bid threshold by modeling the likely monetary value of production from a tract less the costs to explore and develop it, including industry profit and payments to the government. BOEM’s Fair Market Value Review Committee oversees the development of tract-specific parameters—production potential, probability of geologic success, economic projections, and development costs and timeframes— that the bureau uses in its proprietary discounted cash flow analysis model. A discounted cash flow analysis is a valuation method used to estimate the present value of an investment—in this case a tract of land— based on estimated future cash flows. As inputs to its model, BOEM uses the oil and gas resource estimates it developed in its economic viability assessments to estimate how much oil and gas could be extracted from each tract, and it analyzes seismic and well data to determine the likelihood of discovering oil and gas. BOEM also develops economic projections for future oil and gas prices as well as projections for exploration and development costs and time frames for each tract, based on historical cost data, drilling equipment, technological innovation, and other factors. BOEM inputs these parameters into its proprietary discounted cash flow model to generate a distribution of potential tract values. BOEM uses the average of these potential values as representative of the present value of the tract. BOEM also develops an estimate of each tract’s value at the next scheduled lease sale—known as the delayed value. The delayed value for the next sale is computed as the present value associated with the delay in leasing under the projected economic, engineering, and geological conditions—for example, by accounting for depletion of resources due to extraction from a nearby tract that shares access to the reservoir. Based on its valuations, the bureau establishes acceptable bid thresholds for the tracts. The acceptable bid threshold for each tract is the higher of: (1) the lesser of the present value and the delayed value or (2) the minimum bid per acre in instances in which BOEM’s present and delayed valuations are below the minimum bid per acre. If the high bid exceeds the acceptable bid threshold, BOEM concludes that the bid represents fair market value and accepts it and awards a lease. Conversely, if the high bid does not exceed the acceptable bid threshold, BOEM rejects the bid as inadequate and the tract is made available for lease at the next lease sale. According to our empirical analysis of BOEM data and interviews with BOEM officials and industry representatives, changes in the price of oil and changes in royalty rates drive changes in the amount industry bids for offshore oil and gas leases. Specifically, the current and expected future price of oil are key factors determining bonus bid amounts, in the context of industry’s assessment of the expected presence of hydrocarbon reserves for a given tract, the likelihood of success in developing those reserves, and the uncertainties in geological and seismic information. Specifically, our econometric model suggests a strong positive correspondence between higher oil prices and higher bonus bids; that is, when oil prices are higher, bonus bids tend be higher and, conversely, when oil prices are lower, bonus bids tend to be lower. For example, from 2006 through 2008, oil prices rapidly rose to historic highs. This period corresponded with an increase in average bonus bids in deep water from an average of about $275 per acre in 2006 to an average of about $800 per acre in 2008. Figure 3 shows the relationship between oil prices and per acre average bonus bids. The results of our analysis are consistent with input from BOEM officials and industry representatives who told us that the price of oil is a key factor in industry bidding decisions. Specifically, these officials and representatives explained that they use the current price of oil as a baseline for expectations regarding future prices of oil—that is, the price at which industry can sell the oil it produces. Therefore, high current oil prices lead to higher projections of future oil prices, thereby driving up bids. Likewise, they told us that industry bidding activity increases in high- price environments because production from existing wells provides financial resources companies can use to invest in acquiring additional leases. Moreover, according to these officials and representatives, higher oil prices make some tracts economically viable to develop that had been viewed as unprofitable at lower prices. According to our analysis of BOEM data, changes in federal royalty rates also drive changes in the amount industry bids on offshore leases. Our econometric model indicates that increases in royalty rates lead to decreased bonus bids and, conversely, decreases in royalty rates lead to increased bonus bids. According to our model, during the royalty relief period from 1996 through 2000, when royalty rates were effectively zero, bonus bids increased between 34 percent and 60 percent over what bonus bids would have been expected to be had the royalty rate remained at the pre-1996 rate of 12.5 percent. Specifically, we found that industry bid approximately 34 percent higher for leases sold in 1996, 1997, and 2000, when leases contained no royalty obligation until oil prices rose above a certain threshold. Similarly, industry bid approximately 60 percent higher for leases sold in 1998 and 1999, when leases carried no royalties for the life of the lease. However, changes in oil prices can work to counter the effect of royalty rate changes on bonus bids. For example, between 2006 and 2008, royalty rates in water depths greater than 400 meters increased from 12.5 percent to 18.75 percent. Based on our model, this royalty rate increase would have a significant downward effect on bonus bids. However, the rapid increase in oil prices during this period resulted in the net effect of an increase in bonus bids for these tracts by more than 150 percent. Our findings are consistent with the views of BOEM officials and industry representatives, who told us that lower royalty rates increase industry bidding because lower royalties result in higher industry tract valuations. Specifically, the smaller financial commitments to the government associated with lower royalty rates increases the projected value of any oil or gas produced. BOEM officials and industry representatives told us that, in turn, the increased projected value of these tracts would lead to increases in the dollar value of individual bids as well as the number of bids submitted. For example, they cited the royalty relief period of 1996 through 2000 as responsible for a significant increase in bidding activity during that time. However, while decreases in royalty rates lead to higher bonus bids, they may still lead to lower overall federal offshore oil and gas revenues. Specifically, our model estimates and BOEM data show that eliminating royalties for tracts leased between 1996 and 2000 would have increased overall bonus bids for those tracts by at most about $1.98 billion over what they would have been had royalty rates remained at their pre-1996 rate of 12.5 percent. However, forgone royalty revenue was more than nine times greater. Specifically, Interior data show approximately $18.0 billion in forgone royalty payments on these leases through the end of 2018. Because most of these leases are still in production, this estimate does not represent the final total of forgone royalty payments. BOEM regularly assesses potential changes to fiscal terms in annual and supplementary lease sale-specific analyses. Additionally, BOEM has advertised its development of a progressive, priced-based royalty system for 6 years but has made little demonstrable progress toward developing this system. Based on our review of planning documents for lease sales held from March 2016 through August 2018, BOEM regularly assesses potential changes to fiscal terms in annual and supplementary lease sale-specific analyses. BOEM’s annual analyses consider various factors that can affect the fiscal system, and its lease sale-specific analyses build on those factors to inform fiscal terms for individual sales. BOEM conducts an annual analysis of various factors affecting the offshore fiscal system that informs its development of fiscal term options for all lease sales to be held in the subsequent year. According to our review of BOEM documentation and interviews with bureau officials, factors BOEM considers include the following: Resource potential. BOEM estimates the likely amount of undiscovered recoverable oil and gas resources remaining in the region based on the bureau’s most recent national assessment. Market conditions. BOEM assesses trends in oil and gas prices as well as forecasts from the Department of Energy’s Energy Information Administration, the World Bank, and the Office of Management and Budget. BOEM uses these assessments to estimate, under existing fiscal terms, results for the lease sales covered by the analysis— including the amount of bonus bids collected and the number of tracts sold—as well as resulting production and net economic value under various price scenarios. Leasing, drilling, development, and production activity. BOEM reviews industry activity over the previous several years, including leases purchased, companies participating in lease sales, exploration and development drilling, new facility installations, and oil and gas production trends. Industry news. BOEM considers industry perception of its fiscal terms by evaluating industry estimates of break-even thresholds (oil and gas market prices at which production from a given area is cost- effective at current costs of production) and announcements of new discoveries, projects, and production. International considerations. BOEM reviews the fiscal terms of international jurisdictions to assess how they compare with the U.S. system. Within this context, BOEM considers potential changes to its fiscal terms by estimating their effects on outcomes including leasing activity, production, and revenue at various oil and gas prices. For example, in its annual analysis for its August 2017 and March 2018 lease sales, BOEM analyzed the potential effect of five royalty rate changes from the 18.75 percent rate that had been in place since 2008. Two of the potential changes were targeted to specific types of production or water depths and three would apply to all production. For the targeted changes, BOEM considered (1) a lower natural gas royalty and (2) a lower shallow water royalty—both at the statutory minimum of 12.5 percent. The other potential changes were to lower royalty rates on all production to (1) 12.5 percent, (2) 15 percent, and (3) 16.67 percent. For each of these scenarios, BOEM modeled effects on overall production and revenue at various market prices. Based on our review of BOEM lease planning documents, BOEM conducts additional lease sale-specific analysis before finalizing the fiscal terms for each sale. For example, BOEM considered changes to each of the fiscal terms for its August 2018 lease sale—minimum bid, rental rates, and royalty rate—but recommended that they not change from the previous sale. Specifically: Minimum bid. BOEM evaluated lowering the minimum bid for tracts in water depths of greater than 400 meters to account for the effects of decreases in (1) oil prices since BOEM raised the minimum bid to $100 in 2011 and (2) corporate tax rates per the Tax Cuts and Jobs Act of 2017. BOEM found that, because of these changes, a $100 per acre minimum bid in 2018 was roughly equivalent to a $170 per acre minimum bid in 2011 and that maintaining the $100 per acre minimum bid in 2018 could reduce the number of tracts sold. However, BOEM assessed that industry has recently shown a preference for holding less acreage, evidenced by relinquishments and bidding on fewer blocks. Therefore, BOEM determined that lowering the minimum bid might not have the desired effect of increasing tracts leased; instead, it could lead to the same number of blocks being sold but with lower total bonus bid revenue. Rental rate. BOEM evaluated adjusting the rental rate to account for inflation since the last adjustment in 2009. It also evaluated increasing the rental rate in water depths greater than 400 meters to $20 per acre to provide additional financial incentive to explore leases. However, BOEM did not recommend this option since it reported that it expected the effects to be minor. Royalty rate. BOEM evaluated the effect of lowering the royalty rate to 12.5 percent for two scenarios: (1) tracts with water depths between 200 and 400 meters and (2) all tracts. BOEM recommended leaving the royalty rate at 18.75 percent for all tracts deeper than 200 meters. In doing so, the bureau cited little effect for lowering the rate for tracts with water depths between 200 to 400 meters—it projected less than a 0.1 percent increase in production and less than 0.1 percent decrease in revenue. BOEM also cited more substantial projected drops in overall revenue of 17 to 19 percent, paired with modest increases in production (1 to 2 percent increase in oil production and 2 to 5 percent increase in gas production) for lowering the royalty rate for all tracts. BOEM also found that these losses to the federal government could be even more substantial if oil prices rise in the future. BOEM officials told us that, in general, they prefer to make minor iterative changes to fiscal terms in order to better gauge their effects—that is, they find it easier to measure the effects of a change to one term at a time rather than the effects of reconfiguring multiple terms—as well as provide predictability for industry. In keeping with this approach, BOEM has made one change to its royalty rate since 2012 (see table 1 for details on the recent history of lease terms in the Gulf of Mexico). Specifically, in advance of its August 2017 lease sale, BOEM announced a reduction in royalty rate for tracts with water depths of less than 200 meters from 18.75 percent to the statutory minimum of 12.5 percent. According to BOEM documentation, the driving factor for this decision was that shallow water in the Gulf of Mexico has been largely explored, leaving generally marginal tracts that either are largely depleted of resources or more gas prone. In turn, the goal in reducing the royalty rate was to incentivize additional industry interest in these more marginal shallow water tracts. BOEM has publicized the development of a progressive royalty system since 2013 but has made little demonstrable headway toward developing such a system. Specifically, in its budget justifications for fiscal years 2014 through 2017, BOEM stated it was developing a package of legislative and administrative proposals to, among other things, improve the return to the federal government from the sale of these federal resources. Among these proposed reforms was a price-based tiered royalty rate to replace the fixed royalty rate structure that BOEM has used since 1983. Under a price-based royalty system, the royalty rate would depend on prevailing commodity prices, with lower prices having lower royalty rates. According to BOEM documents, the current flat-rate royalty system is regressive—that is, a fixed rate that does not adapt to market conditions or the relative success of a lease—but a price-based royalty would share more revenue risk with the lessee and reduce the regressive nature of the system. A more progressive system would provide an increased incentive to lessees to develop resources during times of low oil and gas prices through lower royalty rates, while also ensuring that the federal government receives a greater return for offshore resources when prices are high. BOEM officials we interviewed told us that this type of adaptive system could be more efficient and provide higher returns relative to the existing fixed-rate system. That is, if properly designed, a priced-based system could increase return to the federal government in high-price environments while incentivizing continued industry investment when prices are low. According to BOEM documentation, a progressive, price-based royalty rate could have the additional benefit of “future-proofing” the royalty system because it would adjust the rate for whatever prices prevail in the future and provide a stable, predictable market for industry. We reported in September 2008 that the regressive nature of the offshore fiscal system, among other factors, caused it to be unstable over time and added risk to oil and gas investments that may reduce the total amount industry is willing to pay for the rights to explore and develop federal leases. BOEM officials told us such a system that automatically adjusts could reduce the need for frequent revisiting and continual annual and lease sale-specific evaluations because it would automatically adapt to certain market conditions. According to these officials, a stable, long-lived system would also reduce political pressure to restructure it or rely on legislation—such as the Deep Water Royalty Relief Act—in the future. Additionally, long-term stability in the royalty system could benefit industry, according to a 2007 study. Specifically, industry may consider fiscal system stability more important than the attractiveness of fiscal terms, as the appeal of low government revenue—incorporating bids, rents, and royalties—is limited if there is a high probability the terms will change. In the context of the offshore fiscal system, this means that some companies might prefer a flexible rate that lowers their royalty obligations in low-price environments so long as BOEM clearly defines the specific market conditions under which royalty rates would increase or decrease. BOEM has continued to publicize its efforts to develop a price-based royalty system—though it did not complete them—as follows: July 2017: BOEM announced in a “note to stakeholders” that it was continuing to analyze a price-based royalty system and would subsequently engage stakeholders on this concept; however, it did not do so. January 2018: BOEM released the 2019-2024 National Outer Continental Shelf Leasing Draft Proposed Program, which states that the bureau was studying a priced-based royalty structure as an alternative to the existing fixed royalty rate. February 2018: BOEM’s memorandum documenting lease term decisions for its March 2018 lease sale stated that the bureau was evaluating a potential future option for a price-based mechanism that would lower royalty rates at current oil prices while increasing rates above the current 18.75 percent royalty rate as price conditions warrant. Spring 2018: In the Lease Term Reassessment Report covering its August 2018 and March 2019 lease sales, BOEM indicated that the statutory floor of 12.5 percent might not be low enough to encourage new exploration and development, particularly for smaller fields for which a lower royalty would have a reduced financial benefit and effect on early cost recovery than for larger fields. As a result, BOEM was considering incorporating into its price-based royalty the suspension of royalty collection for a certain initial volume of oil or gas produced to effectively lower the royalty rate below the statutory minimum and incentivize the development of smaller, marginal fields. However, BOEM has demonstrated little tangible progress in the 6 years since it began publicizing the development of a more progressive royalty system. BOEM officials told us that the general concept for a price-based royalty is robust, but the bureau has not determined optimal parameters for sharing risk when prices are low in return for a higher return when prices are higher. BOEM drafted a Federal Register notice and accompanying procedures for implementing a price-based royalty system that the bureau intended to publish to obtain public comment. These draft procedures include different permutations of royalty rates and price thresholds. However, BOEM officials told us that feedback from within the bureau included enough concerns about workability that the draft notice and procedures were not published and the draft no longer reflects bureau leadership’s position on the issue. According to BOEM officials, the main challenges to a price-based system are determining optimal rates and price thresholds for escalating royalties and quantifying the benefits to the government at lower price levels when government revenue would be lower than under the current regressive system. BOEM officials also cited additional challenges, including establishing price inflation parameters and developing mechanisms for assessing and collecting royalty payments on a sliding scale. After the development of the draft Federal Register notice and procedures, according to BOEM officials, they continued to work on a price-based royalty model. However, they did not provide us documentation of any progress made. BOEM officials told us that the concept is too immature to consider testing implementation on a pilot project basis and that there is not a time frame for when any decisions will be made, including whether to proceed with developing the system. According to federal standards for internal control, agency management should define objectives clearly to enable the identification of risks and define risk tolerances. This involves clearly defining what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement. Developing a documented plan for assessing whether and how to implement a progressive royalty structure that defines these aspects would help position BOEM to better understand (1) the potential benefits such a structure could offer in terms of improving fair return to the taxpayer while fostering diligent offshore oil and gas development and (2) how to implement such a structure if it elects to do so. BOEM’s tract valuation process might not fully assure receipt of fair market value, according to our analysis of BOEM tract valuation data and documentation. BOEM’s valuations for tracts were generally low relative to industry bids, largely due to the cumulative effect of three aspects of its bid valuation process: (1) the bureau forecasts conservatively to account for uncertainties, (2) the bureau forecasts unreasonably high levels of depreciation, and (3) BOEM selectively further lowers many valuations from its model to justify accepting bids it otherwise would reject. In addition, BOEM conducts limited self-evaluations of its tract valuation process and does not have a systematic mechanism to address deficiencies, such as those described above. BOEM’s valuations for tracts it determined to be economically viable were generally low relative to industry bids. Specifically, from March 2000 through June 2018, BOEM’s acceptable bid threshold for the 2,035 tracts on which it conducted valuations was, on average, about one-third of industry’s high bid (about $2.26 million to $6.43 million, respectively, as shown in table 2). BOEM accepted the high bid on about 85 percent of tracts it determined to be viable (1,721 of 2,035), for a total of about $12.8 billion in bid revenue, and it rejected about 15 percent of high bids (314 of 2,035) totaling about $287 million. BOEM’s bid rejections generally resulted in higher bids for the same tracts in subsequent lease sales, significantly increasing bid revenue for these tracts and indicating that industry viewed those tracts as more valuable than the original rejected bid. Specifically, for the 314 bids worth about $287 million that BOEM rejected, BOEM subsequently accepted bids for almost 70 percent of the tracts (161 of 236) for about $667 million—more than twice (about 230 percent) the aggregate rejected value for those tracts. BOEM’s acceptable bid thresholds were generally low relative to industry bids due to three compounding aspects of its valuation process: (1) BOEM conservatively forecasts the key parameters used in its valuation model, (2) BOEM forecasts unreasonably high levels of depreciation between lease sales, which further lowers acceptable bid thresholds, and (3) BOEM alters many valuations—valuations that are already low due to the two preceding aspects of its process—downward further in order to justify accepting bids. BOEM officials told us that they forecast conservatively to account for uncertainties, which systemically lowers its tract valuations. Specifically, they told us that they face significant uncertainties associated with the key parameters that contribute to BOEM’s valuations: resource potential, probability of geologic success, price of oil and gas, and cost and scheduling estimates. They told us that they forecast each of these parameters conservatively—that is, being cautious against overestimating any factor that might unreasonably inflate the bureau’s valuation—so as to not reject bids that might represent fair market value. BOEM’s conservative approach is evidenced by its reluctance to reject bids of significant value. Specifically, from March 2000 through June 2018, BOEM rejected three bids of more than $5 million dollars—the highest was for approximately $11.2 million—while accepting 570 bids of more than $5 million. BOEM officials told us that this conservative approach represents fair market value because the objective of the bureau’s tract valuation process is to lease tracts and collect associated revenues except when BOEM determines a tract is worth significantly more than the highest bid received. That is, they told us that the bureau is more inclined to accept bids and collect revenue—and facilitate exploration and development via the award of leases—rather than reject bids. Moreover, they told us that this approach also provides the bureau with greater justification for rejecting the bids when it does so, which they said can drive up subsequent bids for the same tracts. BOEM forecasts unreasonably high levels of depreciation as compared to the government’s recommended discount rate, which further depresses acceptable bid thresholds that were already based on conservative forecasting. As discussed previously, BOEM’s acceptable bid threshold is generally determined by the lesser of BOEM’s present valuation and its delayed valuation. For the 1,412 tracts with a positive present valuation assessed from March 2000 through June 2018, BOEM forecast a median loss in value on these tracts would be about 23 percent (about $494,000) by the time of the next sale opportunity for those tracts. BOEM officials told us that expected lower future values are generally due to discounting the eventual collection of revenue. Specifically, BOEM officials explained that the bureau’s model considers the delayed collection of revenue—bonus bids and royalties—when developing its delayed values. However, because tracts that received a rejected bid would be available for sale during the next year—or, more recently, 6 months on average—the period of discounting is very short. Discounting seems an unreasonable explanation of BOEM’s forecasted depreciation rates for two additional reasons. First, BOEM’s forecasted depreciation rates do not align with industry bidding patterns for tracts that were leased more than once—where the lease for a tract either expired or the leaseholder relinquished it and the tract was therefore available at a subsequent lease sale. Specifically, for the 61 tracts that were leased more than once from March 2000 through June 2018, bids actually increased slightly over time (bids increased at a real average annual rate of 0.2 percent, or about $6,700). Second, since oil prices are generally forecast to rise, the underlying oil and gas resource values would be expected to increase over time rather than decrease, suggesting a smaller difference between present and delayed values should be observed than is reflected in BOEM’s tract valuations. Additionally, BOEM’s forecasted depreciation has increased even though tracts are now available twice as frequently. Until August 2017, BOEM held annual lease sales for each of two lease areas so that tracts were available once per year. On average during this time, BOEM forecast that the median loss in value for tracts with positive present valuations would be approximately 23 percent (about $481,000) of their value in the year between lease sales (see table 3). BOEM has since made tracts available twice per year. Having less time between lease sales should decrease the amount of forecasted depreciation, as there is less time for discounting. Yet the average difference between present and delayed value increased for biannual lease sales to about 27 percent (or about $1.03 million per tract) for tracts with a positive present valuation. BOEM’s depreciation for biannual lease sales is equivalent to an annual rate of approximately 47 percent (or about $1.78 million annually per tract), which is nearly seven times the Office of Management and Budget’s annual recommended discount rate of 7 percent. That BOEM’s forecasted depreciation has increased since moving to biannual lease sales is also at odds with the concept of how discounting should affect tract valuations, as shorter periods of time are generally associated with lower depreciation than longer periods of time. Under federal standards for internal control, management should use quality information to achieve the entity’s objective. Yet, according to our analysis of BOEM data, the bureau’s unreasonably large forecasts of depreciation have increasingly been the deciding factor in decisions to accept bids. Cumulatively, BOEM’s high forecasted level of depreciation resulted in the bureau accepting 205 bids for about $672 million that it would have rejected if its present valuations had been used as the acceptable bid threshold. Based on the return BOEM has realized on rejected bids, had BOEM rejected these 205 bids, it might have subsequently collected more than $873 million in additional bid revenue for these tracts, which would represent an increase in overall bid revenue of about 6.8 percent for tracts BOEM determined to be viable. BOEM officials told us that they were unaware that their model forecasts such high rates of depreciation and that the issue warrants further examination. However, BOEM officials did not indicate they had any plans to conduct such an examination. Though BOEM is not required to follow government auditing standards, these standards highlight that it can be beneficial to consult an independent third party to assess issues that are highly technical as a safeguard to eliminate threats to independence or reduce them to an acceptable level. As BOEM developed and has used its delayed valuations for at least 20 years, outside perspectives and expertise could be beneficial. Enlisting an independent third party to examine the extent to which the bureau’s use of delayed valuations assures receipt of fair market value, and making changes—such as terminating the use of delayed valuations as acceptable bid threshold criteria or amending its model’s assumptions to develop more justifiable depreciation rates—as appropriate, would help BOEM mitigate risks of continuing to accept bids based on poor information on tracts’ future values. Our analysis of BOEM data as well as BOEM testimony indicate that the bureau changed its forecasting parameters, thereby lowering many valuations and acceptable bid thresholds—which were already systematically low due to its conservative forecasting and excessive depreciation—in order to justify accepting bids. BOEM officials told us that when bids are slightly below the bureau’s initial valuations—and therefore would be rejected per BOEM’s procedures for ensuring receipt of fair market value—BOEM reviews and adjusts its forecasting parameters then reruns its model in order to produce new valuations, which they told us can—and which the data indicate generally do—result in lower valuations that justify accepting the bids. BOEM officials told us that they would rather accept bids offered by industry—as well as any associated rental and royalty revenue—than reject them and potentially never recoup the forgone bid revenue. We observed BOEM’s bias, or statistical anomalies, indicating BOEM lowered a portion of its valuations in order to accept bids in our analysis of BOEM tract valuation data from March 2000 through June 2018. Specifically, we found that BOEM never valued a tract as being worth slightly more than industry’s high bid (that is, instances in which BOEM’s valuation is between 100 and 125 percent of the high bid). In contrast, BOEM valued tracts at slightly less than the industry high bid (that is, instances in which the high bid is between 100 and 125 percent of BOEM’s valuation) about 10 percent of the time (117 of the 1,198 bids subjected to valuation for which the acceptable bid threshold was above the minimum bid level). This anomalous absence of any instances in which BOEM valued tracts slightly more than industry is consistent with BOEM officials’ statements that the bureau further lowered its initial valuations when these valuations were only slightly higher than bids. BOEM officials suggested that any pattern of adjusting valuations would be limited to lower-value bids whereby smaller dollar-value changes would represent greater percentage changes. However, the data do not support this, as we found that BOEM’s bias toward lowering valuations does not appear to be limited to those slightly above industry’s high bid, but is nearly systematic for valuations up to double industry’s high bid across all bid levels. Figures 4 and 5 show the distribution of BOEM’s valuations compared with industry’s high bids, with green data points reflecting accepted bids and blue data points reflecting rejected bids. In particular, the middle two bars in figure 4 and the areas between the dotted lines on figure 5 represent instances in which the relationship between BOEM’s valuation and industry’s high bid—and vice versa— were relatively close (that is, BOEM’s valuation was up to double industry’s bid for rejected bids, and industry’s bid was up to double BOEM’s valuation for accepted bids). Within this range, BOEM’s tendency to lower bid valuations to justify acceptance is clear due to the relative abundance of acceptances (359) and the relative scarcity of rejections (27)—a pattern of more than 13 acceptances for every rejection that is anomalous within the data. This disparity would be even greater if we had included in our analysis the 802 bids BOEM accepted because its valuations were below the minimum bid level which is then used as the acceptable bid threshold. BOEM officials told us that they occasionally change valuations to address the uncertainty inherent in the factors that comprise BOEM’s tract valuation process, though doing so in order to justify bid acceptance is inconsistent with BOEM’s fair market value procedures. Specifically, officials told us that the point valuation developed by its discounted cash flow model is not representative of the broadness of the distribution of potential values—though it does represent the average of the distribution. Additionally, these officials told us that the process is iterative—the bureau adjusts its forecasts multiple times before deciding on final valuations. Furthermore, officials said that valuations that are above, but near, the high bids are subject to more iterations. Moreover, BOEM officials told us that all forecasting parameters and valuations, including those that are revisited more frequently, are evaluated and approved through its Fair Market Value Review Committee, which is broadly responsible for ensuring consistency in the application of the bureau’s tract valuation process. Adjusting valuations comports with what BOEM officials told us is their conservative approach and promotes accepting bids unless the bureau has a high level of certainty that the tract is worth more than the high bid. However, BOEM officials told us they were not aware that their adjustments had effectively reduced the acceptable bid thresholds of nearly all valuations that were initially up to double industry’s high bid. Given that BOEM already starts with a conservative approach to valuation, which is compounded by its model generally forecasting high levels of depreciation, this practice of introducing more conservative assumptions in cases when initial valuations are above bids is not consistent with the bureau’s fair market value procedures prescribed in federal regulations, BOEM’s Bureau Manual for ensuring fair market value, and in BOEM’s bid adequacy procedures. These procedures call for BOEM to use the outputs of its discounted cash flow model as the thresholds for determining whether to accept bids. In situations where BOEM determines that its valuation results are not consistent with programmatic goals, BOEM’s procedures allow for the bureau to develop alternative bid evaluation protocols for a given lease sale, but BOEM has not done so. BOEM’s procedures do not explicitly allow for valuations to be adjusted based on how close they are to industry bids, nor is there an allowance for adjusting valuations in an ad hoc fashion for uncertainty. The practice of adjusting valuations this way undermines receipt of fair market value by holding industry to a lower and potentially inconsistent standard for purchasing leasing rights than those outlined in BOEM’s valuation procedures. The practice of lowering valuations also results in the potential loss of hundreds of millions of dollars in revenue. We do not know how many accepted bids would have been rejected based on their initial valuations because BOEM’s data do not indicate which valuations were further lowered. However, if BOEM had rejected 26 percent of the bids that were up to double its valuations—which appears reasonable to interpolate based on the distribution of the other bid-to-valuation relationships—the bureau potentially could have subsequently collected approximately $567 million additional dollars in bid revenue for tracts it determined to be viable (an increase of about 3.9 percent). Without taking steps to ensure that BOEM’s bid valuation process is not biased toward adjusting valuations downward based on their proximity to bids, the bureau risks continuing to undermine the receipt of fair market value for the sale of public resources. BOEM conducts evaluations of some aspects of its tract valuation process but does not comprehensively evaluate the accuracy of its forecasting, the assumptions of its model, and their combined effect on assuring receipt of fair market value. Specifically, since 2004, BOEM has routinely conducted “lookback studies,” self-evaluations to identify opportunities to refine or improve BOEM tract evaluations and decisions. In these lookback studies, BOEM evaluates its performance by comparing the quantity of discovered hydrocarbon resources with BOEM’s pre-drill estimates of resource potential. However, the scope of BOEM’s lookback studies is limited, which reduces the studies’ effectiveness in helping the bureau improve its valuation process. We identified four main limitations, based on our review of the studies and interviews with BOEM officials, as follows: Resource discoveries are not updated. The lookback studies are not necessarily representative of the total resources on a tract because BOEM compares the forecast against the results of only the first exploratory well and does not update its studies with the results of further exploration. Therefore, BOEM officials told us, the studies are a snapshot in time and are not representative of the total resource that may ultimately be discovered and developed on a tract. Consequently, the studies provide limited insight regarding the total quantity of the resource discovered relative to pre-drill forecasts and identify the causes of any significant differences. BOEM does not assess certain factors. BOEM does not formally assess other forecasted factors that are important in its valuations, such as likelihood of success or cost and schedule estimates, or the underlying assumptions and workings of its discounted cash flow model. BOEM officials told us that the bureau periodically updates its cost and schedule estimates based on available data and that it makes adjustments to its model, but that these processes are generally ad hoc and not consistently documented. As previously discussed, BOEM’s model has produced unreasonably high projected levels of depreciation between lease sales—suggesting that BOEM could modify the model or its assumptions to be more consistent and accurate. For example, BOEM has not assessed how depreciation rates implied in its delayed valuations compare with actual depreciation observed in tracts that have been leased multiple times. BOEM does not systematically use the studies to improve processes. BOEM’s lookback studies do not include a systematic process for identifying and documenting steps the bureau plans to take to improve the bid valuation process. BOEM does not use these studies’ findings to systematically inform or document changes to policies, procedures, or processes related to BOEM’s tract evaluations. For example, BOEM officials told us that the lookback database and the studies are used as training aids, the data are not comprehensive, the studies are used as spot checks and to provide lessons learned, and these studies are not a comprehensive effort to assess BOEM’s valuation process (as BOEM conducts no such comprehensive effort). In its written comments on this report, Interior indicated that BOEM uses the results of its lookback studies to improve aspects of its valuation process. However, Interior did not provide documentation to support this claim. Data do not reflect initial valuations. BOEM’s ability to measure the accuracy of its tract valuation process—both its forecasting and the performance of its model—is hindered because some of its data do not reflect the bureau’s initial valuations but rather the adjusted valuations it used to justify bid acceptance. Specifically, BOEM is unable to observe the effect on revenues and sales bids when its initial valuations—which were already low due to conservative forecasting and generally high depreciation—indicated that bids should be rejected when bids are only slightly less than BOEM’s valuation. By altering the valuations to justify acceptance, BOEM is unable to assess how industry would have responded to those rejections in subsequent lease sales. What we observed indicates that BOEM bid rejections for tracts it values as less than double the high bid lead to almost the same average return in future sales as do rejections in which BOEM’s valuation is many multiples of the bid. By taking steps to ensure that BOEM’s bid valuation process is not biased toward adjusting valuations downward based on their proximity to bids, BOEM could better evaluate how its valuations relate to actual outcomes, which would better inform the bureau as to the validity of its forecasting, modeling assumptions, and the extent to which it is assuring receipt of fair market value. According to standards for internal control in the federal government, management should establish and operate monitoring activities to monitor the internal control system and evaluate the results as well as remediate identified internal control deficiencies on a timely basis. Without implementing a systematic process for comprehensively evaluating its tract valuations, such as by expanding the scope of the bureau’s lookback studies effort and remediating any identified deficiencies, the bureau does not have reasonable assurance that its tract valuation process is working as intended, and that opportunities to refine or improve the bureau’s valuation process are identified and pursued to better assure the receipt of fair market value for the federal government for offshore oil and gas leases. Such a systematic process could provide BOEM a better understanding of how well the bureau is able to forecast key factors against actual results. BOEM has policies and practices intended to ensure the bureau receives fair market value for the hundreds of millions of dollars of offshore oil and gas leases sold each year. This includes a process to assess fiscal terms in advance of lease sales that has informed periodic changes to fiscal terms over the years. However, we found that BOEM has made limited progress in considering more fundamental changes. The bureau has publicized the development of a progressive royalty structure since 2013 that may better share the risks and rewards of offshore energy activities, but has made limited headway in developing one despite significant potential benefits of such a system. The bureau has not defined what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement. Developing a documented plan for determining whether and how to develop a progressive royalty structure that defines these aspects would help position BOEM to better understand (1) the potential benefits such a structure offers in terms of improving fair return to the taxpayer while fostering diligent offshore oil and gas development and (2) how to implement such a structure if it elects to do so. After lease sales, BOEM has repeatedly rejected industry bids when they were lower than the bureau’s assessments of a tract’s value, generating significant additional revenue at subsequent lease sales. However, BOEM’s valuation process might not fully assure receipt of fair market value for sale of offshore oil and gas leases because it systematically reduces the thresholds for accepting bids even though rejecting them could lead to significantly increased revenue. We found that BOEM does so by using a conservative approach to estimating tract values, forecasting unreasonably high levels of depreciation in its delayed valuations, and further lowering valuations in order to justify accepting bids it otherwise would have rejected. Enlisting an independent third party to examine the tradeoffs and benefits of the bureau’s continued use of delayed valuations, and making changes—such as terminating the use of delayed valuations as acceptable bid threshold criteria or amending its model’s assumptions to develop more justifiable depreciation rates—as appropriate, would help BOEM mitigate risks of continuing to accept bids based on poor information on tracts’ future values. Furthermore, BOEM generally lowers its valuations and thereby accepts bids as long as the bid is at least half of BOEM’s initial valuation, which is inconsistent with bureau procedures for ensuring receipt of fair market value. Without taking steps to ensure that BOEM’s bid valuation process is not biased toward adjusting valuations downward based on their proximity to bids, the bureau risks continuing to undermine the receipt of fair market value for the sale of public resources. Cumulatively, we calculate that taking these steps could result in BOEM collecting approximately 10.7 percent more in bid revenue for offshore tracts it determines to be economically viable, which would reflect hundreds of millions of dollars in additional bid revenue over the next decade. BOEM’s ability to assure receipt of fair market value is further hindered because it does not systematically assess its own performance and take steps to improve it. For example, BOEM does not (1) assess how its forecasts of key factors (e.g., reserves discovered, likelihood of success, and oil prices) compared to actual results, (2) assess the assumptions and accuracy of its discounted cash flow model results, such as how well the model accounts for depreciation, and (3) collect information about deviations between BOEM’s initial and final valuations that could provide management insights into the frequency and implication of lowering valuations. Without implementing a systematic process for comprehensively evaluating its tract valuations, such as by expanding the scope of the bureau’s lookback studies effort and remediating any identified deficiencies, the bureau does not have reasonable assurance that its tract valuation process is working as intended, and that opportunities to refine or improve the bureau’s valuation process are identified and pursued to better assure the receipt of fair market value for the federal government for offshore oil and gas leases. We are making the following four recommendations to BOEM: The BOEM director should develop a documented plan for determining whether and how to develop a progressive royalty structure that clearly defines what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement. (Recommendation 1) The BOEM director should enlist an independent third party to examine the extent to which the bureau’s use of delayed valuations assures the receipt of fair market value, and make changes—such as terminating the use of delayed valuations or amending its model’s assumptions—as appropriate. (Recommendation 2) The BOEM director should take steps to ensure that BOEM’s bid valuation process is not biased toward adjusting valuations downward based on their proximity to bids. (Recommendation 3) The BOEM director should implement a systematic process for comprehensively evaluating its tract valuations, such as by expanding the scope of the bureau’s “lookback studies” effort, and remediating any identified deficiencies. (Recommendation 4) We provided a draft of this report to Interior for its review and comment. In its written comments, reproduced in appendix II, Interior agreed with one recommendation, partially agreed with two, and disagreed with one, as discussed below. Interior also stated that it is concerned that certain aspects of the draft report do not paint a representative picture of BOEM’s valuation process and efforts to ensure receipt of fair market value. Regarding the recommendation that BOEM should develop a documented plan for determining whether and how to develop a progressive royalty structure, the agency agreed and indicated that BOEM will develop such a plan. Specifically, the agency stated that BOEM would develop a plan to identify the theoretical and practical benefits and drawbacks of a progressive royalty structure based on existing research and prepare materials for management to determine whether implementation of a price-based royalty would be beneficial. Regarding the recommendation that BOEM enlist an independent third party to examine the extent to which the bureau’s use of delayed valuations assures the receipt of fair market value, the agency disagreed. The agency stated it did not agree with our characterization of BOEM’s delayed valuations and stated that BOEM believes it is neither necessary nor cost effective to enlist an independent third party. However, BOEM agreed to (1) examine its delayed value calculation, particularly as it relates to the impact of biannual lease sales, (2) develop a plan to perform a comprehensive internal review of delayed value calculations and make appropriate changes, and (3) institute a peer-review process for all potential changes. These actions may address some of the deficiencies we identified, but our concerns regarding BOEM’s use of delayed valuations are not limited to the move to biannual lease sales and the agency has not provided any reasonable explanations for its high levels of forecasted depreciation. BOEM forecast a median depreciation of about 23 percent. This implies we should observe significant declines in the actual value of tracts over long periods of time, which is impossible to reconcile with actual trends in bonus bids. The real average bonus bid per acre in 2018 was about the same as it was thirty years earlier in 1988. Alternatively, such a high forecast of depreciation implies either a long time frame between lease sales or a high discount rate. But the time between lease sales has been one year or 6 months, on average, and in our view, and the Office of Management and Budget’s annual recommended discount rate of 7 percent would be more appropriate. Recognizing that Interior’s view differs from ours in this regard, we continue to believe that enlisting an independent third party to examine all aspects of the bureau’s use of delayed valuations—not just proposed changes to address the move to biannual lease sales—would better assure the receipt of fair market value. Regarding the recommendation that BOEM take steps to ensure that its bid valuation process is not biased toward adjusting valuations downward based on their proximity to bids, Interior partially agreed. Specifically, Interior stated it agreed with the recommendation, but did not agree with our characterization of BOEM’s bid valuation process. Interior stated that the apparent anomaly—the lack of instances when BOEM valued tracts up to double industry’s bid—is skewed because a very large percentage of the data set comprise relatively low bids, and BOEM-generated valuations relative to the bids are constrained by the minimum bid amount. That is, Interior stated that the minimum bid level created an artificial floor for BOEM’s acceptable bid threshold even in instances in which BOEM’s valuation is substantially lower, resulting in more bids being up to double BOEM’s valuation than would be the case if BOEM’s acceptable bid thresholds were not constrained by the minimum bid amount. However, as discussed above, we removed all valuations for which BOEM used the minimum bid level as its acceptable bid threshold from our analysis (that is, we did not include instances when BOEM’s actual valuation was below the minimum bid level). Had we included these valuations, the asymmetry in the relationship between bids representing 100 to 200 percent of BOEM’s acceptable bid threshold (acceptances) and BOEM’s acceptable bid threshold representing 100 to 200 percent of industry’s bid (rejections) would have nearly doubled (see figure 6). Moreover, even though we did not include these instances, the minimum bid level only affects the distribution of instances when BOEM’s valuation was less than industry’s high bid. As such, it does not explain why there are so few instances when BOEM valued tracts slightly more than industry. We continue to believe that taking steps to ensure that its bid valuation process is not biased toward adjusting valuations downward based on their proximity to bids would be beneficial and will monitor BOEM’s efforts as part of our regular recommendation follow-up. Regarding the recommendation that BOEM implement a systematic process for comprehensively evaluating its tract valuations, such as by expanding the scope of the bureau’s lookback studies effort, and remediating any identified deficiencies, Interior partially agreed. Specifically, Interior stated it agreed with the recommendation, but did not agree with our characterization of BOEM’s bid tract evaluation process and review procedures. The agency identified two areas where they did not agree with our characterization. First, the agency stated that our statement that “resource discoveries are not updated” is inaccurate. According to its comments, BOEM develops independent estimates of recoverable oil and gas contained within discovered fields by conducting field reserve studies. However, any updated estimates are not reflected in the lookback studies, which represent BOEM’s formal mechanism for self-evaluation. For the lookback studies, as noted above, BOEM compares their forecast against the results of only the first exploratory well and does not update its studies with the results of further exploration. Second, the agency stated that we were incorrect to state that BOEM does not use the studies to improve processes because it uses its lookback studies to improve its valuations. However, BOEM did not provide documentation to support this claim. We continue to believe that implementing a systematic process for comprehensively evaluating its tract valuations would be beneficial and will monitor BOEM’s efforts as part of our regular recommendation follow-up. In addition, Interior stated in its letter that it appeared that we did not account for industry assumptions regarding the applicability of price thresholds in comparing estimated increased bonus bid revenue and forgone royalties for leases subject to deep water royalty relief sold from 1996 through 2000. However, as stated above, we based our econometric modeling on the lease terms provided in Interior’s final notice of sale documents for those leases, which reflect the expectations for royalty relief that industry bid on at the time of sale. In addition, as described in appendix I, we used several alternative model specifications to test the sensitivity of our results to the possibility that industry had different understandings of royalty relief than those contained in the sale documents. Our results are robust across these alternative specifications. As noted above, leases sold in 1996, 1997, and 2000 included provisions for royalty relief subject to price thresholds (that is, lease terms indicated that royalties would only be owed if the price of oil exceeded certain thresholds). Leases sold in 1998 and 1999 did not contain price thresholds (that is, lease terms indicated that no royalties would be owed regardless of the price of oil). As evidenced by our econometric modeling results, during the 1996 through 2000 period, we observed higher bidding when no price threshold provisions were included in lease terms, suggesting that industry accounted for the expectation of no royalties when developing bids. As noted above, in 2007, a federal court ruled that Interior’s attempt to collect royalties through the application of price thresholds on production under leases subject to the 1996 through 2000 royalty suspension was unlawful. In its comments, Interior stated that industry bidding would have been different had companies known at the time of sale that the price thresholds would not apply, and as a result, the net amount of forgone revenue—the difference between collected bonus bids and forgone royalties—would have been lower. To account for this, we adjusted our calculation of estimated additional bonus bid revenues so that it is more comparable to BOEM estimated foregone revenues. This adjustment increased our estimate of additional bonus bid revenues to $1.98 billion (an increase of approximately $530 million), which is still subsumed by the $18 billion in foregone royalties collected through the end of 2018. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of the Interior, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. We developed an econometric model to analyze the effect of royalty rates and other key variables on bonus bids for offshore leases between 1985 and 2018. Specifically, we analyzed how changes in royalty rates affected the winning bids for offshore leases. Our analysis used data from 23,081 individual lease sales in the period from 1985 to 2018. Our model analyzes the winning bid for each lease auctioned by the Bureau of Ocean Energy Management (BOEM). We used the log of the inflation-adjusted winning bid per acre for this dependent variable: Where 𝑌𝑌𝑖𝑖𝑖𝑖 represents the inflation-adjusted (real) value of the winning bid per acre and 𝑦𝑦𝑖𝑖𝑖𝑖 is its log value. 𝑦𝑦𝑖𝑖𝑖𝑖=log (𝑌𝑌𝑖𝑖𝑖𝑖) . Our key set of explanatory variables was a set of indicator (dummy) variables that captured the different levels of royalty rates that pertained during our study period. We ran three alternative model specifications to capture the effects of royalty rates and royalty relief that occurred between 1996 and 2000. Each specification contained: A dummy variable for the 16.67 percent royalty rate. A dummy variable for the 18.75 percent royalty rate. In addition to a 16.67 and a 18.75 royalty rate dummy, model 1 included two additional dummy variables: one dummy variable for royalty relief that occurred in 1996, 1997 and 2000, which allowed a 0 percent royalty rate until the oil price reached a specified threshold; and a second dummy variable for royalty relief that occurred in 1998 and 1999, which allowed a 0 percent royalty rate in perpetuity. In addition to a 16.67 and a 18.75 royalty rate dummy, model 2 included five additional dummy variables for each year from 1996 to 2000. In addition to a 16.67 and an 18.75 royalty rate dummy, model 3 included a single dummy variable for the period 1996 to 2000. The omitted royalty rate dummy variable category was a rate of 12.5 percent. The estimates of the parameters for the other royalty rate dummies show the effect relative to this 12.5 percent royalty rate. Our model controlled for variables that were expected to be related to potential lease production and profitability. These variables included a dummy variable for whether the lease was determined by BOEM to be viable or nonviable; a set of dummy variables for different values of the number of bids, that is, 1 bidder, 2 bidders, 3 bidders, and so on; and a variable for the amount of oil production in the area (protraction area) of the lease’s location at the time of the lease auction. We also controlled for various administrative factors. We used a dummy variable to indicate when the winning bid was too low and was rejected; the value of the minimum bid allowed for the auction; and a set of dummies that captured the use of different royalty suspension provisions, variation in rents charged and different amounts of deep gas relief. To control for effects that vary over time, we included a set of time dummy variables for each date of sale. These dummies account for effects that vary over time but are fixed for any given date, such as technology changes and oil and gas market conditions including the price of oil and gas. Our objective was to control for as many time-varying factors as possible. Attempting to include separate effects of, for example, oil prices and exploration costs, would create problems of leaving out important effects that are difficult to measure or for which there are no data. Finally, we included a set of fixed-effect dummies for each protraction area-block combination that account for locational effects not measured by our other explanatory variables. These fixed effects assist in controlling for unobserved heterogeneity. The regression analysis employed an unbalanced panel model using data for offshore BOEM lease auction sales between 1985 and 2018 as follows: 𝑦𝑦𝑖𝑖𝑖𝑖𝑖𝑖=�𝑐𝑐𝑚𝑚𝐶𝐶𝑚𝑚 𝑦𝑦𝑖𝑖𝑖𝑖𝑖𝑖 is the dependent variable; namely, the log of the real winning auction bid for lease j at location (protraction area-block combination) i for sale date t. 𝑐𝑐𝑚𝑚 is a fixed effect parameter for its associated dummy variable 𝐶𝐶𝑚𝑚, 𝑔𝑔𝑖𝑖 is a fixed effect parameter for its associate dummy variable 𝐺𝐺𝑖𝑖 for for winning bidder, company m. 𝑓𝑓𝑖𝑖 is a fixed effect parameter for dummy variable 𝐹𝐹𝑖𝑖, for year t. location (protraction area-block number combination) i. 𝑋𝑋𝑖𝑖𝑖𝑖𝑖𝑖𝑘𝑘 is the kth characteristic associated with lease j at location i for discussed above and 𝛼𝛼𝑘𝑘 is the parameter associated with each of sale date t. There is one of these for each of the control variables these variables. 𝜀𝜀𝑖𝑖𝑖𝑖 are the error terms. We used xtreg in STATA to estimate our model. Our standard errors are heteroscedasticity-robust and are adjusted for clustering at the protraction area-block combination level. In some cases, our model showed that leases sold when royalty rates were lower had significantly higher winning bids. While not all the royalty rate dummy variables were statistically significant, those dummy variables that measured the largest differences compared to the omitted 12.5 percent royalty rate were statistically significant. Specifically, In model 1, the royalty exemption for 1996, 1997, and 2000, when producers expected zero royalties until oil prices rose above a given threshold, corresponded with an increase in bonus bids of about 34 percent. Similarly, the zero-in-perpetuity royalty rate relief for 1998 and 1999 corresponded with an increase in bonus bids of about 60 percent. In model 2, the royalty exemption dummy variables for each individual year, 1996 to 2000, when producers expected royalty relief, were all significant. The results for these parameter estimates range translate into about 19 to 64 percent increase in real bonus bids. In model 3, the royalty exemption dummy variable for 1996 to 2000 combined, when producers expected royalty relief, was significant. The result for this parameter estimate translates into about a 40 percent increase in real bonus bids. We tested for the restriction on the dummy variable parameters (all parameters equal) implied in model 3 versus model 2. Our test rejected equal parameters in favor of the specification in model 2. The 18.75 percent royalty rate dummy parameter was statistically significant and negative in all three models, which is to be expected since the base (comparison) case is 12.5 percent. The result for this effect translates to a drop in bonus bids of about 28 percent in all three models. However, the 16.67 percent dummy variable was not statistically significant in any of the models. We used a set of time-fixed effects for each sale date and, therefore, we could not separate out the individual effects of time-varying variables such as oil prices. These dummies show the effect on bonus bids of conditions pertaining on that particular sale date, where a larger positive value translates to higher bonus bids and a smaller or negative value translates to lower bonus bids. Figure 7 compares oil prices and the values of the sale date dummy variables over time and suggests a correspondence between higher oil prices and the size of these dummy variable estimates. This suggests that higher oil prices are likely to result in higher bonus bids. The set of dummy variables for the number of bids produced parameter estimates that were statistically significant and for the most part were of the expected size and sign. These suggest that greater interest (more bids) is associated with higher bonus bids (the exception was the slight deviation from this pattern for the 8 bids dummy). The number of bids may not represent market concentration because anyone is permitted to bid on a given lease, so potentially there are a large number of bidders. This set of dummies is more likely to represent perceived quality of the lease on the part of bidding firms. Other key factors were either significant with the expected direction of effect or else not statistically significant. Oil production in the protraction area at the time of the auction was positive and significant. Rejected bids were associated with smaller highest bids. Joint winning bids were associated with higher bonus bids. Leases designated as viable by Interior were associated with higher bids. Our model contains no explicit consideration of market concentration effects. Our use of the number of bidders in the model may capture some market concentration effects but possible endogeneity issues that may arise with the use of such measures are not addressed due to lack of reasonable instruments. Our model does not explicitly isolate the impact of oil prices because we needed to include time-fixed effects (dummies). However, we are able to evaluate the effect of oil prices indirectly by observing the correspondence between the estimated values of the time dummies and oil prices. Our tests for joint significance of the time dummies rejected the null hypothesis of non-significance in all cases. Our results showed a significant effect of royalty rates of 18.75 percent relative to 12.5 percent. However, our results did not show a significant effect of royalty rates of 16.67 percent relative to 12.5 percent, which may be due to a lack of statistical power and that relatively modest differences in royalty rates have only a small impact of bonus bids. The model has limited controls for geological conditions at the lease location. We control for location imperfectly using fixed effects for protraction area-block combinations, and by including the amount of oil production on the date of the lease sale in that protraction area. Our use of these protraction area-block fixed effects does not allow us to control for water depth explicitly. Our analysis used data from 1985 to 2018. Earlier data were available, beginning in 1983, but initial tests of our model suggested the 1983 and 1984 data were not well captured by the model’s specification. BOEM’s system of using competitive bidding for leases began in 1983 and there may have been an initial period during which market operators learned how to bid efficiently under the new system. Our model includes a control for the minimum bid but we did not account for any censoring effects that may have arisen from setting this threshold. Ideally, we would have liked to establish whether there were different responses of bonus bids to the control variables in deep versus shallow water. However, separate models for deep and shallow water leases produced mostly non-significant effects for royalty rates, which suggested that splitting the sample in this way resulted in insufficient statistical power to estimate these effects. In addition to the individual named above, Quindi Franco, Assistant Director; Matthew Tabbert, Analyst-in-Charge; Natalie Block; Tara Congdon; William Gerard; Cindy Gilbert; Michael Kendix; Michael Krafve; and Dan Royer made significant contributions to this report.", "summary": "Production of oil and natural gas from offshore leases is a significant source of federal revenue, totaling almost $90 billion from 2006 through 2018. BOEM is required to seek a fair return from offshore leasing and production activities in federal waters. Companies generally pay (1) bids for leases for the right to develop tracts, (2) rents on leased but undeveloped tracts, and (3) royalties on revenues from the sale of oil and gas produced from leases. BOEM holds auctions to award leases to the company offering the highest bid so long as the bureau determines the bid represents fair market value. GAO was asked to examine issues related to offshore federal oil and gas leasing. This report, among other objectives, (1) describes the effect of oil prices and royalty rates on industry bids for leases and (2) examines the extent to which BOEM's valuation process assures receipt of fair market value. GAO reviewed laws, policies, and regulations; interviewed BOEM officials; and developed an empirical model using BOEM data to analyze the effect of royalty rates and other factors on industry bidding. GAO's analysis indicates that changes in the price of oil and in royalty rates drive changes in the amount companies in the offshore oil and gas industry bid for leases (the amount paid upfront at auction for the right to explore and develop offshore tracts of land). Specifically, between May 1985 and June 2018, peaks in industry bidding coincided with higher oil prices. Additionally, when the Department of the Interior's (Interior) Bureau of Ocean Energy Management (BOEM) offered leases at lower royalty rates, industry bid somewhat higher amounts per acre. For example, certain leases were sold from 1996 through 2000 with no royalties on initial volumes of production, which GAO estimates resulted in BOEM collecting, at most, nearly $2 billion in additional bid revenue. However, bureau estimates indicate these leases resulted in about $18 billion in foregone royalties through 2018. BOEM's valuation process might not fully assure receipt of fair market value, based on GAO's analysis of BOEM data. BOEM develops valuations for offshore tracts it assesses to be economically viable—assessments of their fair market value—and awards leases so long as the bid is greater than or equal to BOEM's valuation. BOEM's valuations for tracts were generally low relative to industry bids because, according to BOEM officials, they conservatively forecast to account for inherent uncertainties in, among other things, the quantity of oil and gas present as well as exploration and development costs. In addition, GAO identified two ways BOEM's valuation process results in lowering its already conservative valuations that might not fully assure receipt of fair market value: Unreasonably high depreciation . BOEM forecast that tracts would lose a median of 23 percent of their value in between sales, leading the bureau to accept lower bids because it determined the tracts might be worth even less in the future. Bureau officials told GAO that lower future values are generally due to BOEM discounting the delayed collection of revenue. However, BOEM's forecasted depreciation increased even though tracts are now available twice as frequently as they were prior to August 2017, reducing the time for discounting. Officials said they were unaware of the high rates and the issue warrants further examination. Enlisting a third party to examine the extent to which the bureau's use of delayed valuations assures the receipt of fair market value, and making changes as appropriate, would help BOEM mitigate risks of continuing to accept bids based on poor information on tracts' future values. Lowered valuations . BOEM officials told GAO that they lower some initial valuations that are “slightly above” industry's bids and which would therefore be rejected per procedures to assure fair market value. Officials said they prefer to accept bids unless there is high certainty that the bids are inadequate. However, GAO identified bias, or statistical anomalies, where BOEM lowered many valuations that were initially higher than industry's bids. Specifically, from March 2000 through June 2018, BOEM rejected 27 bids for tracts that it ultimately valued at up to double industry's bid whereas it accepted 359 bids in which industry's bid was up to double BOEM's valuation. Tracts for rejected bids are, on average, subsequently sold for more than twice the initial rejected amount, suggesting that BOEM could be forgoing hundreds of millions of dollars in bid revenue by accepting bids that are too low. GAO is making four recommendations, including that BOEM (1) enlist an independent third party to examine whether the use of delayed valuations assures the receipt of fair market value and (2) take steps to ensure its bid valuation process is not biased toward lowering valuations. Interior disagreed with the first and partially agreed with the second, disagreeing with GAO's characterization of BOEM's process. GAO maintains the recommendations are valid as discussed in the report.", "document_type": "gao"}
{"report": "There are various statutes, regulations, and agency policies that set forth how DHS components are to make decisions about, or process, the family members they encounter. For the purposes of this report, we use the following key terms and definitions. Family. Federal immigration law does not specifically define the term “family” for the purposes of identifying family relationships that are to be documented at apprehension. DHS components and other federal agencies use the term “family” for individuals with a variety of relationships such as step-, half-, foster, or adoptive family members. Some family relationships, including parent-child, may be claimed upon apprehension, but CBP may determine that the relationship is invalid. For example, CBP may determine that (1) those claiming a familial relationship are not related or (2) their relationship does not meet the relevant component or agency’s operational definition of family. For the purposes of this report, “family” refers generally to noncitizens with claimed familial relationships. Unaccompanied alien child (UAC). The Homeland Security Act of 2002 defines a UAC as a child under the age of 18, who has no lawful immigration status in the United States and who has no parent or legal guardian present in the United States, or if present, no parent or legal guardian available to provide care and physical custody for that child. Family unit. Federal immigration law does not specifically define the term “family unit.” However, CBP and ICE policy and guidance documents generally define a family unit as the inverse of a UAC. In other words, a family unit includes a noncitizen child under the age of 18, who has no lawful immigration status in the United States, accompanied by a noncitizen parent or legal guardian who is able to provide care and physical custody. For the purposes of this report, “family unit” refers to this specific subset of family, as previously defined. Dependent. For a number of immigration benefit applications, including asylum, a spouse or child may be included as dependents on a principal’s application and derive lawful immigration status from the principal applicant if the applicant is granted relief. Similarly, consistent with regulation, USCIS policy is to include a spouse or child in a principal applicant’s positive credible fear determination if they arrived concurrently and the spouse or child wants to be included. In this context, “child” is generally defined in federal immigration law as an unmarried biological or legally adopted child under age 21. For the purposes of this report, we refer to principal applicants’ spouses and unmarried children under age 21 as “dependents.” Family members who are apprehended together may encounter multiple federal agencies and components during their immigration proceedings, including DHS components, HHS’s ORR, and the Department of Justice’s Executive Office for Immigration Review (EOIR), as shown in figure 1. CBP documents the circumstances of noncitizens’ apprehension. After Border Patrol agents or OFO officers apprehend noncitizens, including families, they are to interview each individual, using interpreters if needed, and collect personal information such as their names, countries of nationality, and age. Agents and officers also collect biometric information, such as photographs and fingerprints, from certain individuals. Border Patrol agents and OFO officers use fingerprints to run records checks against federal government databases to determine if individuals have any previous immigration or criminal history. Agents and officers are to enter information about the individuals in the appropriate automated data system as soon as possible, in accordance with CBP policy. Border Patrol agents and OFO officers print copies of the information they enter into their data systems to create a paper file, known as an “A-file,” for each noncitizen they apprehend. One of the key required DHS forms in the A-file is Form I-213, Record of Deportable/ Inadmissible Alien. Among other things, this form captures biographic information and includes a narrative section for agents and officers to document the circumstances of the apprehension. According to CBP policy, Border Patrol agents and OFO officers are to determine the validity of family relationships among individuals they apprehend. To do so, for example, they are to review any available documentation, such as birth certificates; monitor interactions between adults and children; and use their law enforcement training, such as interview skills, to help assess the validity of family relationships. After making decisions about the validity of familial relationships, agents and officers are to decide whether and how family members will be detained together while in CBP custody. According to CBP’s 2015 National Standards on Transport, Escort, Detention, and Search, CBP “will maintain family unity to the greatest extent operationally feasible, absent a legal requirement or articulable safety or security concern that requires separation.” According to CBP officials, if individuals are determined to be ineligible for admission into the United States, agents and officers must decide how to process them, which may include placing them into full or expedited immigration removal proceedings, consistent with the Immigration and Nationality Act. In full removal proceedings, individuals have the opportunity to present evidence to an immigration judge to challenge their removal from the United States and apply for various forms of relief or protection, including asylum. In expedited removal proceedings, the government can order individuals removed from the United States without further hearings before an immigration judge unless they indicate an intention to apply for asylum, a fear of persecution or torture, or a fear of return to their home country. Most arriving noncitizens are eligible to be placed into expedited removal proceedings, with certain exceptions, according to Border Patrol and OFO officials. Individuals placed in expedited removal proceedings and who express a fear of persecution or torture are generally subject to mandatory detention under the Immigration and Nationality Act pending a final determination of credible fear of persecution. Regarding family units, in particular, Border Patrol and OFO officials stated that Border Patrol agents and OFO officers typically determine whether ICE has available detention space in one of its family residential centers before placing family units into expedited removal proceedings. ICE and ORR detain or shelter noncitizens and share information about UAC. ICE, among other things, is responsible for detaining and removing noncitizens, including families, who are in the United States in violation of U.S. immigration law and subject to removal. ICE officers are to determine whether to detain, release, or remove such individuals based on a variety of factors, including statutory requirements, medical considerations, and the availability of detention space. ICE detains adults over age 18 in detention facilities that are segregated by gender. For family units placed in expedited removal, ICE officers have the authority to accept or deny a CBP referral for detention in one of ICE’s family residential centers—a decision that ICE officials stated is largely dependent upon available detention space. As of October 2019, ICE operated three family residential centers, with different population characteristics in each center: South Texas Family Residential Center (Dilley, TX), which has a maximum capacity of 2,400 beds for female adults and their male or female children. Karnes County Residential Center (Karnes, TX), which has a maximum capacity of 830 beds for male adults and their male children. Berks County Residential Center (Leesport, PA), which has a maximum capacity of 96 beds for male or female adults and their male or female children. When an individual is transferred from CBP to ICE custody, ICE officers are to enter information about that person in ICE’s data system. The paper A-file is also transferred from CBP to ICE and, according to ICE officials, ICE officers generally review the A-file upon transfer to ensure that it is sufficiently complete. ICE’s data system automatically pulls some information, such as basic biographic information, from CBP’s data systems. ICE officers are to enter new information into ICE’s data system, such as the location(s) where officers detained or released the individual and the documents officers served to the individual, among other things. If CBP or ICE officials determine that a child or children under the age of 18 and without lawful status in the United States arrived in the country without an accompanying parent or legal guardian, the child is classified as a UAC and is to be transferred to ORR custody. Additionally, if DHS determines that a child should be separated from their accompanying parent or parents, DHS then considers the child to be a UAC and transfers him or her to the custody of ORR. ORR provides interim care for UAC at its shelters and identifies qualified sponsors in the United States to take custody of the child while the child waits for his or her full immigration proceedings. CBP’s data systems can share some information about UAC automatically with ORR, including biographic information such as name, date of birth, and alien number; and information about related UAC, such as siblings, who were apprehended together. To assess the suitability of potential sponsors, ORR staff collects information from potential sponsors, which may include parents or other family members, to establish and identify their relationship to the child. For example, ORR screening of potential sponsors includes various background checks. According to ORR officials, they are required to attempt to contact a child’s parent, regardless of the parent’s location, any time they place a child with a sponsor. According to ORR officials, ORR is also responsible for coordinating reunification of separated family units if DHS and HHS determine it is appropriate, or if the adult is later determined by a federal court to be a class member in the ongoing Ms. L v. ICE litigation, related to family separations. ORR officials said that they rely on ICE to gather additional information, such as detailed information from an adult or UAC’s Form I-213, when that information is not available or shared at the time a UAC is transferred to ORR custody. USCIS and EOIR consider claims of relief from removal from the United States. USCIS screens individuals in expedited removal—most of whom are in ICE detention facilities—for credible fear if they indicate an intention to apply for asylum, a fear of persecution or torture, or a fear of returning to their home country. In this screening, an asylum officer is to review certain documentation from CBP and ICE; perform background checks using various automated databases; interview the individual to obtain more details on his or her fear claim, overall credibility, and the nature of any relationships with family members with whom he or she was apprehended; and determine whether there are any dependents who could potentially be included in the individual’s fear determination. The regulation governing the credible fear process allows dependents— specifically a spouse or unmarried child under the age of 21—of a principal applicant to be included in the applicant’s credible fear determination, if the dependent (1) arrived in the United States concurrently with the principal applicant and (2) desires to be included in the principal applicant’s determination. For cases in which USCIS concludes the screening with a positive determination, USCIS is to issue a Notice to Appear, thereby placing the individual into full removal proceedings before an immigration judge. Consistent with regulation, if a principal applicant receives a positive credible fear determination, it is USCIS policy that his or her dependents may be included in the positive determination—and be placed into full removal proceedings—if the dependent arrived concurrently with the principal applicant and wants to be included in the principal’s credible fear determination. For cases in which the asylum officer concludes the screening with a negative determination, USCIS is to refer the individual to ICE for removal from the United States, unless he or she requests a review of the negative determination by an immigration judge. Those in full removal proceedings who apply for asylum before an immigration judge may include a spouse and/or unmarried children under age 21 in their asylum application. If the judge grants asylum to the principal applicant, his or her dependents may also be granted asylum. In 2010, Public Law 111-139 included a provision for us to identify and report annually on programs, agencies, offices, and initiatives—either within departments or government-wide—with duplicative goals and activities. In our annual reports to Congress from 2011 through 2019 in fulfillment of this provision, we described areas in which we found evidence of fragmentation, overlap, and duplication among federal programs, including those managed by DHS. To supplement these reports, we developed a guide to identify options to reduce or better manage the negative effects of fragmentation, overlap, and duplication, and evaluate the potential trade-offs and unintended consequences of these options. In this report, we use the following definitions: Fragmentation occurs when more than one agency (or more than one organization within an agency) is involved in the same broad area of national interest and opportunities exist to improve service delivery. Overlap occurs when multiple programs have similar goals, engage in similar activities or strategies to achieve those goals, or target similar beneficiaries. Overlap may result from statutory or other limitations beyond the agency’s control. Duplication occurs when two or more agencies or programs are engaged in the same activities or provide the same services to the same beneficiaries. DHS has not identified the information about family members apprehended together that its components collectively need or communicated that information to relevant components across the department. Based on our analysis of agency documentation and interviews with agency officials, we determined that CBP, USCIS, and ICE require different information about family members who are apprehended together and each component collects such information that is relevant to its respective operational needs. Specifically, CBP, as the apprehending agency at the border, needs information about family members apprehended together for the purposes of, among other things, informing how family members are to be detained while in CBP custody. In addition, USCIS needs information on family members to identify individuals who may be eligible dependents for credible fear screening purposes. ICE needs information on family members to assist USCIS in identifying eligible dependents and to assist ORR in identifying individuals who may be eligible sponsors for UAC based on their family relationship. While each DHS component has identified the information needed to meet its own specific requirements regarding family members, DHS has not identified information needs regarding family members across its components, resulting in a lack of shared understanding of all components’ needs and fragmented information collection. For example, the information that CBP collects about family members is not aligned with the information that other components, or agencies that might subsequently encounter these family members, need to identify eligible dependents for credible fear purposes or suitable sponsors for UAC. CBP. Regarding family units, CBP (including Border Patrol and OFO) generally collects information about members of family units—including parents and their children under age 18—who are apprehended together. CBP components assign a unique identifier to a family unit that allows members’ records to be linked. CBP components use the information they collect about members of family units to inform how they are to be detained while in CBP custody and to determine how their immigration proceedings are to proceed. In addition, CBP may collect information about certain other relationships among family members apprehended together because CBP and its components—Border Patrol and OFO— have policies that allow certain family members who are not defined as family units to be detained together while in CBP custody. For example, with regard to Border Patrol, family groups composed exclusively of children under the age of 18—such as siblings or a parent under age 18 and his or her child—may be held together in CBP custody, according to Border Patrol guidance. As another example, family members who Border Patrol or OFO agents or officers determine need to be detained together, such as a parent and their child over age 18 with significant medical needs, may also be held together in CBP custody. Border Patrol and OFO have developed processes to collect information about the relationships between family members who are to be detained together, including Border Patrol assigning them a “family group” number in Border Patrol’s data system and OFO documenting the relationship between a juvenile accompanied by a non-parent family member, to facilitate their detention together while in CBP custody. However, CBP generally does not collect information about certain family members— such as spouses or children age 18 to 21—because CBP does not have a need to collect such information if, for example, those family members will not be detained together. Other components may require this information, as described below. USCIS. USCIS requires information about family members for credible fear screening and asylum eligibility purposes, consistent with immigration law. Based on our analysis of agency documentation and interviews with agency officials, this differs from the information that CBP collects about family members for its operational purposes. Specifically, spouses and unmarried children under age 21 may be included in their spouse or parent’s credible fear screening if the family members arrived in the United States together. At the credible fear screening interview, USCIS is to document the name, country of nationality, and alien number, if known, for the spouse and name, date of birth, country of nationality, and alien number, if known, for the child or children of all individuals being screened for credible fear. In addition, consistent with regulation, it is USCIS policy to include any dependents who arrived concurrently with the principal applicant, such as a spouse or unmarried child under the age 21, on a principal applicant’s positive credible fear determination if the dependent wants to be included. This results in both the principal applicant and any dependents being issued a Notice to Appear for full removal proceedings. In addition, USCIS’s training on screening families for credible fear states that families do not need to be detained together to be included in a positive determination. In other words, a principal applicant in a credible fear screening may be detained at one of ICE’s family residential centers and his or her dependent spouse or child between the ages of 18 and 21 may be detained separately at an adult detention facility. Specifically, since ICE’s adult detention facilities are segregated by gender, a female might be detained in a separate adult detention facility from her male spouse. If a parent or spouse receives a positive credible fear screening, his or her dependent’s case could be linked and both family members could receive a notice to appear in immigration court for full immigration proceedings. According to USCIS headquarters officials, USCIS relies on information obtained during the credible fear screening interview to identify family members because the information that USCIS receives from CBP about the circumstances of an apprehension generally does not include details about spouses or children age 18-21. Further, USCIS officials said that family members over age 18 who are apprehended together may be detained in separate ICE facilities and referred to USCIS for fear screenings at different times, which makes it difficult for USCIS and ICE to locate such family members. In addition, USCIS officials said that ICE is often not aware of the family relationship between family members if they are detained separately. Specifically, although ICE is responsible for detaining noncitizens who express fear of returning to their home country before they are screened for such fear by USCIS, ICE officials responsible for detention management told us that (1) they are often not aware of family relationships between family members detained separately and (2) they treat anyone over age 18 as an adult and do not consider that a child age 18 to 21 or a spouse could be a dependent on a credible fear claim. ICE. In addition to assisting USCIS in identifying eligible dependents for credible fear screening purposes, ICE assists ORR in identifying qualified sponsors for UAC. According to ORR, qualified sponsors include, among others, and in order of preference: parent or legal guardian; an immediate relative who previously served as a primary caretaker of the child; an immediate relative who did not previously serve as a primary caretaker of the child; and other distant relatives or unrelated adults with a pre- established relationship with the child. When a child apprehended by CBP is classified as a UAC and transferred to ORR’s custody, CBP is to provide ORR with information about family members with whom the UAC was apprehended. However, officials from ORR told us that they sometimes receive UAC referrals—either through an automated system or via email—from CBP with no information about family members with whom the child was apprehended, but subsequently learn from the child that the child was apprehended with a family member. According to ICE and ORR officials, when ORR has questions about potential sponsors for a child in their care, they coordinate with officials from ICE’s juvenile and family management program to obtain additional information about the circumstances of the child’s apprehension or family members with whom a child was traveling. ICE officials stated that CBP generally provides the information on family members traveling with UAC to ORR, if CBP is aware of such information; however, according to ICE officials, children may not share all relevant details about their family members with CBP agents and officers when they are apprehended, and they may be more comfortable sharing such details once they are in ORR custody. ICE officials said that they can search their data systems, including law enforcement records, for information about the circumstances of a child’s apprehension, which ORR uses when evaluating potential sponsors for the child. ORR cannot access such law enforcement records. For example, ICE can use Border Patrol’s “event” unique identifier to search for information about adults who Border Patrol apprehended at the same time as a child, and can use this information to attempt to identify if there are family relationships between an adult and unaccompanied child. ORR officials said that the lack of family member information they receive from CBP or ICE, or delays in receiving such information, can delay the release of a child from a shelter to a qualified sponsor. Our previous work on collaboration has shown that establishing compatible policies, procedures, and other means to operate across agency boundaries can enhance and sustain collaborative efforts and help ensure that fragmented efforts are being managed effectively. Further, leading practices of high-performing organizations include fostering collaboration both within and across organizational boundaries to achieve results. Moreover, federal programs contributing to the same or similar results should collaborate to ensure that program efforts are mutually reinforcing, and should clarify roles and responsibilities for their joint and individual efforts. Our interviews and analysis indicate that the information each DHS component collects about family members meets its own information needs, but does not consider the information needs of other components that might encounter those family members. Officials from CBP and ICE confirmed that they collect information about family members to meet their own operational needs. For example, CBP may not collect information about spouses apprehended together because CBP does not need such information for its operational purposes. Further, Border Patrol and OFO officials we spoke with told us that CBP components collect all relevant information needed for their operational purposes but that CBP is not responsible for collecting information that USCIS needs to identify eligible dependents, including spouses and children age 18 to 21. Without identifying and communicating department- wide information needs with respect to family members who have been apprehended together, DHS does not have reasonable assurance that its components are identifying all individuals who may be eligible for relief from removal from the United States based on their family relationships or that ICE can provide ORR with the information it needs to help evaluate the suitability of potential sponsors for UAC. CBP’s Border Patrol and OFO document the circumstances under which family members are apprehended at or between U.S. ports of entry and, as a result, are in the best position to collect information about their family relationships. However, our analysis of DHS documentation and interviews with officials indicate that CBP does not routinely collect all of the information about family members that is needed to (1) identify eligible dependents as part of the credible fear screening process and (2) evaluate family members for sponsorship placement for UAC. Further, Border Patrol agents and OFO officers do not routinely document that information on the record of apprehension. CBP’s Border Patrol agents and OFO officers are to document the circumstances of an apprehension using the required Form I-213, Record of Deportable/ Inadmissible Alien (record of apprehension). The record of apprehension is a key form in the paper A-file and is the official record of an apprehension. Among other things, the record of apprehension captures biographic information about the apprehended individual and includes a narrative section for agents and officers to document details about the circumstances of the apprehension. Border Patrol and OFO’s guidance indicates that the record of apprehension may be used as evidence in immigration or criminal courts and that omissions or mistakes on the form may have negative consequences. According to Border Patrol officials, the information captured on the record of apprehension varies and there is no requirement that it include information about family members apprehended together. However, USCIS, ICE, and ORR officials told us that they rely on the record of apprehension for such family information. As discussed below, since CBP does not routinely collect sufficient information about family members apprehended together or document such information on the record of apprehension, there are gaps in the information available to other DHS components about family members apprehended together. Information to identify eligible dependents as part of the credible fear screening process. CBP does not routinely collect sufficient information about relationships between family members apprehended together for USCIS and ICE to later identify if such individuals are eligible dependents as part of the credible fear screening process. As previously discussed, consistent with regulation, it is USCIS policy to include any dependents on a principal applicant’s positive credible fear determination if the dependents arrived concurrently with the principal applicant and want to be included on the principal applicant’s credible fear determination. However, CBP does not routinely collect information about relationships between all parents, children, and spouses apprehended together at the time of their apprehension or share that information with USCIS. Specifically, CBP does not require its agents and officers to collect information about or to document the relationships between certain family members apprehended together, such as spouses and children age 18 to 21. As a result, USCIS’s ability to identify eligible dependents is limited. Asylum officers are to ask all individuals they screen for credible fear if they arrived in the United States with other family members. Asylum officers told us that, when CBP does not collect information about potentially eligible dependents—especially spouses and children age 18 to 21—they face challenges in identifying and locating such dependents. Asylum officers also told us that when CBP agents and officers do not collect and document information about relationships at the time family members are apprehended, asylum officers must rely on the information that the applicant provides in the credible fear screening interview, rather than using the screening interview to corroborate family information already collected by CBP at the time of the apprehension. In addition, a USCIS official told us that it can be beneficial for USCIS to have information about relationships between all parents, children, and spouses who are apprehended together for other processes—such as if one family member placed into expedited removal proceedings is subject to the reasonable fear process—because information in one family member’s claim can impact other family members’ ability to meet the threshold for a positive fear determination. Border Patrol, OFO, and ICE officials stated that, due to the volume of apprehensions at the southwest border, Border Patrol and OFO collect information to meet CBP’s operational needs, but that the level of detail documented on the record of apprehension may vary. Specifically, according to one ICE official responsible for detention at a family residential center and an ICE headquarters official, information about family relationships, including that of spouses, is not consistently documented in the information ICE receives from CBP and shares with USCIS. Since USCIS does not receive consistent information about family members from CBP, USCIS officers must rely on the credible fear screening interview to identify potential eligible dependents. When asylum officers identify eligible dependents during the credible fear screening interview, officers attempt to locate these dependents to link them to their parent’s or spouse’s case. However, according to USCIS and ICE officials, it can be difficult to locate such dependents if they are not detained together. Specifically, because CBP officers and agents do not routinely collect information about the relationships between spouses or parents and children age 18 to 21 or document such information on the record of apprehension at the time they are apprehended, USCIS and ICE do not have the information about those family relationships that they need to locate and identify eligible dependents. Additionally, individuals may not know certain information—such as the alien number of their spouse or child—that would help USCIS or ICE locate them. ICE officials told us that they assist USCIS officials in locating spouses and children age 18 to 21 for the purposes of making them dependents on a spouse or parent’s credible fear application on a case by case basis, but that tracking down such dependents can be difficult. Further, ICE and USCIS officials told us that because they do not have sufficient information about eligible dependents, it is possible that ICE could remove an eligible dependent from the United States while their spouse or parents’ credible fear claim was pending, or after their spouse or parent received a positive credible fear determination. Information to assist ORR in making placement decisions for children transferred to its custody. CBP does not collect all information about family members at the time of apprehension that is needed to assist ORR in making placement decisions for UAC transferred to its custody, according to ICE and ORR headquarters officials. When CBP refers a child for placement at an ORR shelter, CBP is to share some information with ORR, including the name, age, and alien number of the child, as well as information about any family members with whom the child was apprehended. ORR officials stated they use this information to assist in making placement decisions for the child. However, ORR officials stated that the information CBP provides when the child is referred may not include information about family members with whom the child was apprehended. Further, according to ORR officials, they do not typically receive the child’s Form I-213—which documents the circumstances of the child’s apprehension—from CBP. ORR officials said that they sometimes receive UAC referrals from CBP without any information about other family members and they may subsequently learn from the child that he or she was apprehended with a family member. Additionally, if ORR officials have questions about a child in their custody, officials from ICE’s Juvenile and Family Residential Management Unit told us that they are the liaison between DHS and ORR. ICE officials told us that the level of detail that CBP agents and officers collect for UAC apprehended with family members varies. According to an ICE official in ICE’s Juvenile and Family Residential Management Unit, the more information that CBP agents and officers provide about the circumstances of a child’s apprehension, the better equipped ICE is to answer ORR’s questions about familial relationships and potential suitable sponsors for a particular child, as well as to investigate potentially fraudulent familial relationships or circumstances in which an adult apprehended with a child might not be a suitable sponsor. According to ORR officials, they also rely on ICE to provide information about the suitability of reunifying a parent and child where ORR determines that a UAC was separated from their parent or legal guardian with whom they arrived. As we reported in February 2020, DHS and HHS have developed interagency agreements for the transfer and placement of UAC between the two departments; however, information sharing gaps remain. Specifically, ORR headquarters officials stated that they have experienced delays in releasing a child to a sponsor due to missing information about a parent or the inability to notify a parent in ICE detention about sponsorship decisions. We recommended that DHS and HHS should collaborate to address information sharing gaps to ensure that ORR receives information needed to make decisions for UAC, including those apprehended with an adult. DHS and HHS concurred with the recommendations. Border Patrol and OFO developed their own requirements for what information they collect, if any, about family members apprehended together based on their operational needs. However, because CBP agents and officers collect information and document the circumstances of apprehensions when families first arrive in the United States, they are best positioned to identify those family members who were apprehended together and the relationships among them. Additionally, the information that CBP agents and officers collect may impact how family members are subsequently identified or processed by other federal agencies. CBP officials said that their components collect limited information about family members apprehended together because they do not have an operational need for such information and because collecting it is time intensive in an environment where agents and officers are managing a large volume of apprehensions. However, because CBP does not routinely collect sufficient information about family relationships at the time of apprehension, or document that information on the record of apprehension, DHS components do not have information necessary to identify potentially eligible dependents for credible fear purposes and ICE does not have sufficient information to assist ORR in making suitable sponsorship determinations. Further, while we recognize that the collection of additional information on family members can be time intensive for CBP, as the apprehending agency, CBP is best positioned to collect and document information on family members apprehended together. In addition, ICE, USCIS, and ORR may expend resources themselves trying to identify family relationships for their own operational purposes. As previously noted, our prior work on collaboration has shown that establishing compatible policies, procedures, and other means to operate across agency boundaries can enhance and sustain collaborative efforts and help ensure that fragmented efforts are being managed effectively. In October 2019, CBP officials acknowledged that it could be helpful to consider other agencies’ information needs when collecting information about apprehended families. Collecting information about the relationships between family members apprehended together and documenting that information on the Form I-213 could help address fragmentation among DHS components and improve the information available to other agencies, such as ORR, to ensure that relevant information is available to support decisions on individuals’ administrative immigration or other proceedings. DHS does not have a mechanism to link the records of family members apprehended together across its components. Specifically, CBP’s data systems can assign unique family identifiers to link records of certain family members together, as appropriate, upon apprehension. CBP uses these unique identifiers to facilitate the detention of family members together in CBP custody. They also provide a mechanism for CBP to search for and identify family members that share a unique identifier. However, those identifiers are not readily accessible and usable to USCIS and ICE, which also have operational needs to identify and review records of family members apprehended together. Further, USCIS and ICE’s data systems do not assign unique family identifiers. Because DHS’s data systems do not have shared family identifiers to link family members, DHS components may not have access to all the information about family members they need to make effective and efficient operational decisions. CBP’s data systems assign unique family identifiers. Regarding family units, CBP components have guidance on how Border Patrol agents and OFO officers are to enter information on family units in their respective data systems. CBP’s data systems assign a unique identifier to each family unit and link their records, and agents and officers are to collect the following information about family units: Border Patrol guidance indicates that agents are to process adult parents and their children under age 18 who are apprehended together as members of a family unit, and the data system assigns each family unit a unique family unit identifier. This identifier links the records of the family unit members together, and allows agents to search for family unit members using that number. OFO is deploying a new data system and, as of October 2019, OFO officials said that they planned for the new system to be deployed along the southwest border on an ongoing basis as conditions allow. OFO documentation on the new system indicates, and OFO officials told us, that the new system will allow OFO officers to assign a unique family identifier to members of a family unit and will allow officers to document the familial relationship between members of family units. Border Patrol’s data system can also assign a unique family group identifier to family members whom agents determine should be detained together for Border Patrol’s operational purposes. According to Border Patrol guidance and officials, family group numbers may be used to link family members during Border Patrol detention. Further, these numbers may be documented on the record of apprehension and may be shared with ORR to, for example, link the records of two related UAC when Border Patrol transfers them to ORR custody. However, Border Patrol agents have discretion to determine whether family members apprehended together are to be assigned a unique family group identifier, according to agency documentation and our interviews with agency officials. CBP components do not have a mechanism to share their unique family unit or family group identifiers with ICE or USCIS in a way that is readily accessible and usable. CBP’s data systems share limited information on apprehended family members with ICE’s data system. When ICE receives custody of a family unit from CBP, ICE officers create a record for each family member in ICE’s data system. ICE’s data system pulls some information about each family member automatically from CBP’s data systems. For example, ICE officers can find basic biographic information about individual family members apprehended by Border Patrol by searching using the individual’s alien number, a DHS unique identifier assigned to individuals. In addition, ICE identified a need for more information to help identify family units in ICE custody and developed a mechanism to receive that information from CBP. As of August 2018, ICE’s data system displays a family unit “banner” in the data records of those noncitizens CBP processed as a member of a family unit. This banner flags for ICE officers that the individual was identified by CBP as a family unit member, and ICE’s data system displays the Border Patrol or OFO unique family unit identifier. ICE’s family unit banner was a positive development and allows ICE to identify individuals in its custody that CBP processed as a member of a family unit. However, the family unit banner does not provide ICE all the information it needs to identify family members, according to ICE officials. Specifically, ICE can see that a particular individual was processed by CBP as a member of a family unit, but ICE cannot use the system to identify other members of that person’s family because ICE’s data system does not link or display alien numbers for individuals who share a family unit identifier. According to Border Patrol officials, because ICE and Border Patrol’s data systems are both housed within ICE’s Enforcement Integrated Database repository, ICE should have access to the family unit information collected by Border Patrol. However, ICE officials stated that ICE cannot use the information on family units that CBP’s data system shares with ICE’s data system to, for example, search for family unit members using Border Patrol’s unique family unit identifier. According to ICE officials, ICE officers must use a time consuming and manual process to research potential family associations or identify family unit members using the information CBP provides to ICE. Further, ICE’s data system cannot link the records of family unit members in its custody, although these family unit members are generally detained together in one of ICE’s family residential centers. According to ICE guidance and ICE officials, ICE’s data system only displays family unit information as entered by CBP and such information is not available for individuals identified as members of a family unit after entering ICE custody. As of November 2019, ICE headquarters officials stated that they are working with the ICE data unit to create a new module that would enhance ICE’s ability to link and track family units in its data system, including expanding ICE’s use of existing family unit information as entered by CBP. According to ICE officials, ICE has established a project team for this effort and hopes to deploy the updates in the fourth quarter of fiscal year 2020. However, ICE did not provide any documentation on this effort, such as a project plan with time frames for deploying these system updates, to verify these plans. Although ICE has taken steps to identify individuals in its custody that CBP documented as members of a family unit, ICE does not have a mechanism to link the records of family unit members together. In addition, ICE does not have a mechanism, such as a unique family group identifier, to link the records of other family members apprehended together. ICE needs information about these other family members to (1) assist USCIS in identifying eligible dependents for credible fear screening purposes and (2) assist ORR in identifying family members with whom a UAC was apprehended and assessing whether they might be suitable sponsors. According to ICE officials, ICE uses a manual process to identify family members apprehended together. Without a mechanism, such as a shared unique identifier, that ICE can use to access information CBP gathered about family members apprehended together, ICE cannot ensure that it has the information it needs to identify eligible dependents, or to answer ORR’s questions about UAC with the best available information. As of November 2019, ICE is enhancing its data system’s ability to link and track family unit members. However, it is too early to know if ICE’s planned system enhancements will include a mechanism that will allow ICE officers to identify family members apprehended together. CBP and ICE’s data systems do not share information on apprehended and detained family members with USCIS’s data system. USCIS’s data system does not receive information about family members (parents, spouses, and children) from CBP or ICE in an automated manner. According to USCIS officials, because CBP’s and ICE’s data systems do not have a mechanism—such as a linked unique family identifier—to share information about potential dependents with USCIS’s data system automatically, the credible fear interview may be the only way for USCIS to determine that an individual being screened for credible fear was apprehended with other family members, especially if any members of the family are detained separately. For family members detained separately, according to USCIS officials, USCIS asylum officers attempt to locate spouses and children age 18 to 21 when they are made aware of such family relationships as part of the credible fear screening process. However, due to limitations in data sharing between CBP, ICE, and USCIS, USCIS may not be able to locate such spouses and children age 18 to 21 in some circumstances. In particular, USCIS officials told us that, if the spouse or child did not make his or her own claim of credible fear while in CBP or ICE custody, USCIS asylum officers use a time consuming and manual process to attempt to identify family members apprehended together, using data that ICE makes available to USCIS. ICE officials told us that they assist USCIS officials in locating spouses and children age 18 to 21 for the purposes of making them dependents on a spouse or parent’s credible fear application on a case by case basis, but that tracking down such dependents can be difficult. USCIS has developed a mechanism to link family members in its own data system, but this linkage is for USCIS’s purposes and is unrelated to the unique family unit or family group identifier assigned by CBP components at the time family members are apprehended or to the “family unit” banner that ICE’s data system displays for certain family units. Additionally, USCIS’s data system does not assign a unique identifier to family members whose cases are linked for credible fear screening purposes and USCIS does not have access to CBP’s family identifiers. A shared family member unique identifier could allow USCIS, CBP, and ICE access to more complete information about family members who were apprehended together and could give USCIS and ICE, in particular, greater assurance that they have complete information about family members apprehended together that they require for their operational needs. Our previous work on collaboration has shown that identifying and addressing needs by leveraging resources, such as information technology resources, can enhance and sustain collaborative efforts, and help ensure that fragmented efforts are being managed effectively. Border Patrol, OFO, ICE, and USCIS data systems were developed to meet each component’s operational needs, leading to (1) data system integration limitations and (2) variation in the type of information that each component collects or requires. Components have implemented ways to share some information across their data systems—such as ICE’s “family unit” banner for members of family units processed by Border Patrol and USCIS’s ability to access some information in ICE’s data system to attempt to identify eligible dependents of individuals who have received a positive credible fear determination—but such information sharing is limited, and the components do not have a unique shared identifier to identify family members apprehended together. Moreover, DHS and its components have not considered options to share information on family members across components in an automated manner, as each component has been focused on its own operational needs for such information. Evaluating options for developing a shared unique family member identifier across CBP, ICE, and USCIS that would allow each component access to certain information about family members apprehended together would help bridge the information gaps about family relationships between components caused by DHS’s fragmented data systems. Further, it would give DHS greater assurance that its components can identify family members who were apprehended together, even after they leave CBP custody. It would also mitigate the risk that, lacking such information, DHS could remove individuals from the United States who may have been eligible for relief based on their family relationship. Although CBP’s apprehensions of family members have increased significantly in recent years, DHS has not taken steps to better manage fragmentation, including identifying, collecting, documenting, and sharing the information its components collectively need about family members apprehended together. The information each DHS component collects about family members apprehended together meets its own information needs. However, it does not consider the information needs of other components that might encounter those family members. Border Patrol and OFO officials we spoke with told us that CBP components collect all relevant information needed for their operational purposes but that CBP is not responsible for collecting information that USCIS needs to identify eligible dependents, including spouses and children age 18 to 21. Without identifying information needs with respect to family members who have been apprehended together—and without communicating that information department-wide to relevant components—DHS does not have reasonable assurance that its components are identifying all individuals who may be eligible for relief from removal from the United States based on their family relationships. In addition, as the component that apprehends individuals arriving at the border, CBP is best positioned to document the circumstances of an apprehension, including by collecting and documenting information about family members who arrive in the United States together. Collecting information about the relationships between family members apprehended together and documenting that information on the Form I- 213, the record of apprehension, would improve management of fragmentation among DHS components and improve the information available to other agencies, such as ORR, to ensure that relevant information is available to support decisions on individuals’ administrative immigration or other proceedings. Lastly, DHS components’ data systems were developed to meet each component’s operational needs, leading to data system integration limitations and variation in the type of information that each component collects or requires. Components have implemented ways to share some information across their data systems, but such information sharing is limited. Evaluating options for developing a shared unique family member identifier across CBP, ICE, and USCIS that would allow each component access to certain information about family members apprehended together would help bridge the information gaps about family relationships between components caused by DHS’s fragmented data systems. We are making the following four recommendations to DHS: The Secretary of Homeland Security should identify the information about family members apprehended together that its components collectively need to process those family members and communicate that information to its components. (Recommendation 1) The Secretary of Homeland Security should ensure that, at the time of apprehension, CBP collects the information that DHS components collectively need to process family members apprehended together. (Recommendation 2) The Secretary of Homeland Security should ensure that CBP documents the information that DHS components collectively need to process family members apprehended together on the Form I-213. (Recommendation 3) The Secretary of Homeland Security should evaluate options for developing a unique identifier shared across DHS components’ data systems to link family members apprehended together. (Recommendation 4) We provided a draft of this report to DHS and HHS for their review and comment. DHS provided formal, written comments, which are reproduced in full in appendix I. DHS and HHS also provided technical comments on our draft report, which we incorporated, as appropriate. DHS concurred with our recommendations and described actions planned or underway to address them. For example, in response to our recommendation that DHS identify the information its components need about family members apprehended together, DHS stated that the DHS Office of Immigration Statistics within the DHS Office of Strategy, Policy, and Plans will work with CBP, ICE, USCIS, and interagency partners to establish a comprehensive set of information to collect on family members apprehended at the border. Further, in response to our recommendations that DHS collect and document the information its components collectively need about family members apprehended at the border, DHS stated that after DHS identifies the information about families apprehended together that its components collectively need, CBP will work with DHS’s policy office to ensure all required information is collected at the time of apprehension on the Form I-213. In addition, Border Patrol and OFO will issue guidance to their agents and officers to ensure they document the information about family members apprehended together that DHS components collectively need. Regarding our recommendation that DHS evaluate options for developing a unique identifier shared across DHS components’ data systems to link family members apprehended together, DHS stated that its policy office will work with CBP, ICE, and USCIS to develop a unique shared identifier linking family members apprehended together. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Acting Secretary of Homeland Security, and the Secretary of Health and Human Services. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or gamblerr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. In addition to the contact named above, Kathryn Bernet (Assistant Director), Mary Pitts (Analyst in Charge), Carissa Bryant, Miranda Cohen, Michael Harmond, Stephanie Heiken, Leslie Sarapu, Jessica Walker, Dominick Dale, Eric Hauswirth, Jan Montgomery, Heidi Nielson, and Michele Fejfar made key contributions to this work.", "summary": "In fiscal year 2019, CBP reported apprehending more than 527,000 noncitizen family unit members (children under 18 and their parents or legal guardians) at or between U.S. ports of entry along the southwest border—a 227 percent increase over fiscal year 2018. GAO was asked to review issues related to families—including family units—arriving at the southwest border. This report examines the extent to which DHS has identified, collected, documented, and shared information its components need to inform processes for family members apprehended at the border. GAO analyzed DHS documents; interviewed DHS officials; and visited DHS locations in Arizona, California and Texas, where CBP apprehensions of family units increased in 2017. GAO compared the information gathered with leading practices in collaboration to evaluate DHS components' processes for apprehended family members. The Department of Homeland Security's (DHS) processes to identify, collect, document, and share information about family members apprehended at the southwest border are fragmented. DHS's U.S. Customs and Border Protection (CBP) apprehends family members and determines how information about each individual—and his or her relationship to other family members—will be collected and documented. Other DHS components, such as U.S. Immigration and Customs Enforcement (ICE), use information collected at the time of apprehension to inform how those who are members of a family, including children, will proceed through immigration proceedings. Family members apprehended at the border and placed into expedited removal that indicate an intention to apply for asylum, or a fear of persecution or torture or fear of return to their home country, are referred to DHS's U.S. Citizenship and Immigration Services (USCIS) for a credible fear screening. However, DHS has not identified the information its components collectively need about apprehended family members. Each DHS component collects information to meet its own operational needs, and does not consider the information needs of other components. For example, the information about family members that CBP needs differs from the information about family members that USCIS needs. CBP officials told us they would not generally identify spouses and children age 18 to 21 apprehended with a parent as family members, although USCIS's definition of a dependent for credible fear screening purposes includes spouses and unmarried children under age 21. CBP collects information about certain family members for its operational purposes, but does not collect and document information at the time of apprehension that other DHS components may later need. Specifically, CBP collects and documents information about parents and their children under age 18 who are apprehended together. However, consistent with regulation, USCIS policy is to include any dependents who arrived concurrently with the principal applicant, such as a spouse or unmarried child under age 21, on a principal applicant's positive credible fear determination if the dependent wants to be included. According to USCIS and ICE officials, it can be difficult to identify spouses and children age 18 to 21 because CBP does not regularly document such family relationships. DHS does not have a mechanism to link the records of family members apprehended together across its components that need this information. As a result, DHS components may not have access to all the information about family members they need to make effective operational decisions. Because DHS has not identified the information all of its components collectively need to process family members apprehended at the border, collected and documented that information at the time of apprehension, and evaluated options to share that information across components, consistent with leading practices in collaboration, DHS risks removing individuals from the United States who may have been eligible for relief or protection based on their family relationship. GAO is making four recommendations to DHS, including that DHS identify the information its components collectively need to process family members apprehended together, collect and document that information at the time of apprehension, and evaluate options for developing a unique identifier shared across DHS's data systems to link family members apprehended together. DHS concurred with the recommendations.", "document_type": "gao"}
{"report": "Intergenerational economic mobility describes how people’s incomes in adulthood compare with their parents’ incomes in the past or at similar ages. Several measures are used to assess the degree of economic mobility, but fundamentally, a society exhibits more economic mobility when incomes are less related to parents’ income. By contrast, where economic mobility is lacking, individuals are more likely to remain at the economic position of their upbringing. Economists traditionally measure economic mobility in three ways: Absolute economic mobility - whether people make more money (in inflation-adjusted dollars) than their parents did at a similar age (see fig. 1). For example, in 1970, 92 percent of 30-year-olds made more money in inflation-adjusted terms than their parents did at similar ages, implying an absolute economic mobility rate of 92 percent. Relative economic mobility - whether people are at a higher income percentile compared to their parents’ income percentile in the past. For example, according to one estimate, there was an 8 percent chance that a person born in the United States from 1980-1982 to parents in the bottom 20 percent of the income distribution would move to the top 20 percent of the income distribution for their birth cohort by the time he or she was approximately 30 years old. Intergenerational income elasticity (IGE) - the strength of the relationship between a person’s income and their parents’ income. The higher the number, between zero and one, the greater the relationship between parental income and children’s adult income (see fig. 2). For example, if IGE is zero, there is complete mobility between generations; parents’ income does not influence their children’s future income at all. If IGE is 1, there is no mobility between generations, as everyone stays at the same income level in which they were born. IGE measures the “persistence of advantage” from one generation to the next at all points along the economic ladder and therefore captures how much inequality is passed down through generations. A single standard measure of intergenerational economic mobility does not exist, and some researchers use more than one. Each of the three measures provides some insight into the level of opportunity available for people to better their economic circumstances relative to the circumstances of their birth. Many factors may be related to the level of economic opportunity available to an individual, including but not limited to overall macroeconomic conditions (e.g., economic growth), education, race, gender, geography (the region, commuting zone, county, or neighborhood in which a person lives), health care, and neighborhood characteristics. Millennials have a number of unique characteristics that distinguish them from previous generations. According to data from SCF, Millennials are a more diverse group than previous generations—40 percent of Millennial households are headed by someone who belongs to a racial or ethnic minority group. Millennials are also the most educated generation to date in terms of college degree attainment (see fig. 3). An estimated 62 percent of Millennial households had someone with at least an associate’s degree in 2016. Not only did Millennial households have more college degrees overall, a greater percentage of Millennial households in 2016 had advanced degrees, including master’s, doctorate, and professional degrees, compared to previous generations at similar ages. Meanwhile, only 44 percent of Millennials 25-34 years old were married or living with a partner and had children in 2016, while 54 percent of Baby Boomers were partnered and had children by age 34. The 20 studies that we reviewed indicate that economic mobility has remained flat or declined in the United States over the last 40 years; none of the studies we reviewed found that economic mobility has increased (see text box). Additionally, estimates of intergenerational income elasticity (IGE) suggest that economic status persists across generations, particularly for the lowest and highest income groups. Studies identified parental income, race, and geography as key determinants of one’s economic mobility. These findings could have future implications for Millennials. money than their parents at the same age declined between 1970 and 2010 (see fig. 4). One study attributes this decline to an unequal distribution of economic growth, noting it has primarily benefited the highest earners. It remains to be seen if this downward trend will continue for the Millennial generation. The research we reviewed indicates that economic mobility varies by race. The findings on economic mobility and race suggest that not all groups of Millennials may experience the same levels of economic opportunity. Blacks experience less upward intergenerational mobility than whites. In particular, black men are less likely to be upwardly mobile and more likely to be downwardly mobile than white men, even with similar levels of education. Meanwhile, children of low-income white families have had higher rates of upward mobility over time than black children with similar socioeconomic characteristics. Some minority groups have higher economic mobility than others. One study that examined additional racial groups found high earnings among children of low-income Asian households, and found that Asians are likely to remain at income levels comparable to or above-white Americans, though these findings are largely driven by first-generation immigrants. Additionally, Hispanic Americans are moving up the income distribution across generations, although their overall economic mobility is somewhat lower than whites. Meanwhile, American Indians are more likely than whites to be downwardly mobile, even those in the wealthiest 1 percent. The research we reviewed indicates that the region, state, commuting zone, county, and most especially, the neighborhood in which one grows up affects economic mobility and future earnings, but these effects vary by demographic and income groups. Economic mobility varies by location. One study found that areas within the United States offer disparate opportunities, with some localities supporting higher rates of economic mobility than others (see fig. 5). In particular, counties in the southeastern United States were found to have lower levels of economic mobility than counties in the rural Midwest. Another study found that a child’s neighborhood has a statistically significant effect on life chances, and that growing up in a low-income, metropolitan neighborhood has a strong negative effect on future earnings. Conversely, growing up in an affluent neighborhood can have almost as large an impact on future earnings as completing a bachelor’s degree. Specific neighborhood characteristics drive differing rates of economic mobility. Several researchers linked economic mobility to certain area and neighborhood characteristics, including rates of poverty, racial segregation, economic inequality, the proportion of single-parent households, and school quality. Researchers identified racial segregation as a neighborhood characteristic broadly associated with lower mobility. One study found that economic segregation is also negatively associated with economic mobility. One study identified three neighborhood characteristics that are correlated with a weaker relationship between race and mobility: low poverty rates, a high percentage of low-income black fathers present, and low levels of racial bias among whites. According to this study, neighborhoods with these characteristics had higher mobility for black boys and a relatively small black-white mobility gap. The effects of geography on future earnings vary by race, socioeconomic status, and gender. The effects of race and neighborhood characteristics on economic mobility are related and hard to disentangle. For example, one study found that black boys have lower incomes in adulthood than white boys who grow up in the same neighborhood in 99 percent of Census tracts, even when accounting for income. This highlights the effect of race on economic mobility when children face the same neighborhood conditions. Conversely, the same study also found that 4.2 percent of black children grow up in neighborhoods with the characteristics associated with higher levels of mobility, compared to 62.5 percent of white children. This is in line with another study that found that neighborhoods can amplify racial inequality across generations. Another study notes that Hispanic and black children tend to live in neighborhoods with low mobility for those of their racial group, whereas white children tend to live in neighborhoods with higher mobility rates for whites. Neighborhood effects can also vary by socioeconomic status and gender. Regarding socioeconomic status, one study found that place may matter less for children from higher-income families, as they may be better able to insulate themselves from the effects of local conditions (e.g., by switching to private schools if public schools are weak.) Regarding gender, the same study finds that neighborhood matters more for boys than girls. Across studies, common themes emerged that suggest Millennials might not have the same level of economic mobility enjoyed by their parents’ generation. While the studies in our review varied in their estimates of key measures of economic mobility and its determinants, the studies were consistent in their findings that absolute economic mobility is declining, relative mobility is flat or declining, and economic status is somewhat rigid from one generation to the next. Moreover, the studies that examined drivers of mobility found that a child’s race and neighborhood have a significant effect on their economic mobility as adults. This is particularly relevant for Millennials because of their racial and ethnic diversity. It is not clear whether Millennials’ diversity and higher levels of education will lead to a reversal of these trends, or whether these trends will continue into the future. If economic mobility is flat or falling, knowing how a cohort is doing at the beginning of its members’ working lives sheds light on the potential challenges that lie ahead as the cohort ages and moves toward retirement. We analyzed data from the Survey of Consumer Finances (SCF) to provide a snapshot of how Millennials are faring economically as young adults. We compared the financial circumstances of Millennial households in 2016 to Generation X households in 2001 and Baby Boomer households in 1989; in each year, we estimated measures of financial well-being for households in which the head of household, or any spouse or partner, was 25-34 years old. We found that incomes across the three generations have remained relatively flat, which is consistent with our review of economic mobility studies. We also found that Millennials have lower net worth, which we define as assets minus debt. With respect to assets, we found that Millennials are saving for retirement, but the accumulation of wealth through homeownership has decreased as fewer Millennials are buying homes. In terms of debt, Millennials hold large amounts of student debt compared to previous generations, but are also more likely to be college educated. Millennial households in 2016 had similar average real incomes compared to previous generations at similar ages, according to our analysis of SCF data (see fig. 6). Our analysis showed that median incomes were also similar across young adult households in the Millennial and Baby Boomer generations and that Millennial households had slightly lower median incomes than Generation X households (see fig. 7). We also examined average and median incomes among households with college degrees and found similar results. These findings suggest that, on average, real income levels have been stagnant for young adult households across these three generations. As described in figure 3, Millennial households are more likely to be college-educated compared to previous generations. While college graduates generally have higher incomes than non-college graduates, the income of degree holders has remained flat over time. A recent study from the Federal Reserve Board of St. Louis found that the college income premium, the increase in earnings for college graduates compared to non-college graduates, does exist. According to this study, in the first quarter of 2018, college graduates received weekly wages that were 80 percent higher than high school graduates. However, college graduates in recent years have not made higher incomes than college graduates in the past, as they have had relatively flat inflation-adjusted wages since 2001. Overall, Millennial households in 2016 had significantly lower average and median net worth, defined as assets minus debt, than Generation X households at similar ages in 2001, according to our analysis of SCF data (see figs. 8 and 9). This may be explained by lower homeownership rates than previous generations, as well as larger amounts of student debt. Median net worth was much lower for Millennial households in the bottom 50 percent of the net worth distribution compared to previous generations. While median net worth for the lowest net worth quartile of Baby Boomers and Generation X was around zero, it was substantially negative for Millennials in the lowest quartile, indicating that debt was greater than assets (see fig. 10). The median net worth of those Millennial households in the highest 25 percent was also significantly lower than the median net worth of those at the top in previous generations. We analyzed both average and median net worth to examine how net worth was concentrated among young households. Our analysis showed that estimates of median net worth were much lower than estimates of average net worth across all three generations, suggesting that net worth was unevenly distributed among these households and that a relatively small number of households held a substantial percentage of total net worth. As a part of our analysis of net worth, we examined specific types of assets and debt, including homeownership, retirement resources, and student loans, and found the following: Millennials had lower rates of homeownership compared to previous generations. Our analysis of SCF data showed that a significantly lower percentage of Millennial households in 2016 were homeowners compared to previous generations in 2001 and 1989 (see fig. 11). We estimated that about 43 percent of Millennial households owned homes, compared to 51 percent of Generation X households and 49 percent of Baby Boomers. As a result of lower rates of homeownership, Millennial households had less mortgage debt, but also less home equity, compared to households in other generations at similar ages. Home equity has historically been an important source of retirement security as people age. It is unclear whether Millennial households will reach similar rates of homeownership as previous generations, but it is possible they may be more likely to buy homes at older ages compared to previous generations. Millennials were as likely to have retirement resources as previous generations. A similar percentage of Millennials had retirement resources in 2016 (either defined benefit pensions or retirement accounts, such as an IRA, 401(k), or other account-type pension), compared to Baby Boomers in 1989 and Generation X in 2001 (see fig. 12). Millennials have a similar average value of retirement accounts as Generation X (see fig. 13). This may be due, in part, to auto- enrollment policies, which create default retirement savings accounts for workers, and are relatively new. Millennials have a higher average value of defined contribution retirement accounts compared to Baby Boomers, likely because of the shift over time in the retirement system from defined benefit pensions to account-type pensions, such as 401(k)s. Student loans were the key source of debt that distinguished Millennials from previous generations. We found that Millennial households were significantly more likely to have student loans than previous generations at similar ages (see fig. 14). We measured the potential burden of student loan debt by estimating student loan-to-income ratios and found that this measure was significantly higher for Millennial households in 2016 compared to previous generations when they were young. On average, Millennial households in 2016 had a student loan-to-income ratio that exceeded 100 percent compared to ratios of under 50 percent in previous generations (see fig. 15). While the student loan-to-income ratio has increased over time for households of all incomes, it has most greatly affected lower-income households. For example, while we estimated that the average student loan-to-income ratio was about 100 percent for young households in the bottom income quartile in 2001, we estimated it was significantly higher for young households in the bottom income quartile in 2016 (see fig. 16). These findings suggest that, on average, it could take Millennials several more years’ worth of total income to pay back total household student loan debt (without interest). Although Millennial households have more student debt than previous generations, they may also benefit from federal student loan repayment plans and forgiveness programs. Households that qualify for these programs may not have to repay their student debt in full, though to date about half of student loans are still under standard repayment plans and few potentially qualified borrowers have been granted forgiveness (see textbox). Income-Driven Repayment (IDR) plans, available through the Department of Education for federal student loans, generally base student loan payment amounts on a borrower’s income and extend repayment periods from the standard 10 years to up to 25 years with any remaining balance forgiven at the end of the period. Some borrowers may qualify for very low payments and these payments count toward loan forgiveness at the end of the repayment period. As of September 2018, almost half ($414 billion) of the $859 billion in outstanding Direct Loans were being repaid by student loan borrowers using IDRs. The long-term effects of higher educational attainment, along with higher education loans, on Millennial households is unclear. It is possible that those with advanced degrees may be better situated over time to repay their student loans. However, while an estimated 18 percent of Millennial households in 2016 had advanced degrees (master’s degree or above), an estimated 45 percent had student loans, indicating that many Millennial households with student loans did not have an advanced degree. In addition, while the college income premium is real, high levels of student debt may affect the ability to accumulate wealth, which may be why average net worth levels have decreased for college graduates. (PSLF) program forgives federal student loan balances for eligible borrowers who have made 10 years of qualifying payments while in certain public service jobs. As of March 2019, the Department of Education reported that 1,089,846 borrowers had an approved Employment Certification Form, the first step in potentially qualifying for PSLF. However, 99 percent of applicants were denied PSLF, highlighting the confusion with respect to applying and ultimately getting debt relief from these programs. The Millennial generation is different from previous generations on several measures of financial well-being, so there is uncertainty about how they will do financially as they age. On one hand, they have higher levels of educational attainment, and college graduates earn substantially more than non-college graduates. On the other hand, despite Millennials completing college degrees at higher rates than previous generations, average and median income are not higher for Millennials overall, which is consistent with flat intergenerational economic mobility and persistence of economic status across generations. Millennials also have less home equity than past generations because they are buying homes at lower rates. Given relatively stagnant average income across generations, it is not clear whether Millennials will begin earning more and buying homes later in life or whether lower homeownership rates will persist over time. Millennials are saving for retirement at rates comparable to Generation X, and saving early in life should benefit Millennials in the long run. Yet, they have significantly higher levels of student loan debt than past generations. Some Millennials may ultimately qualify for programs that help them lower their federal student loan debt, but it remains to be seen how these factors will affect Millennials’ financial circumstances in the long run, including in retirement. We provided a draft of this report for review and comment to the Departments of Labor (DOL) and the Treasury and to the Social Security Administration (SSA). We received technical comments from DOL, which we incorporated as appropriate. Treasury and SSA provided no comments. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretaries of Labor and the Treasury as well as the Administrator of the Social Security Administration. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or jeszeckc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Our objectives were to examine (1) what is known about intergenerational income mobility, and (2) how the financial circumstances of Millennials compare to previous generations. In order to determine what is known about intergenerational income mobility (which we use interchangeably with “economic mobility”) in the United States, we conducted a literature review of relevant, recent economic studies. We identified the majority of the studies we reviewed through systematic searches of databases such as ProQuest, Scopus, and EBSCO using search terms such as “economic mobility,” “income mobility,” “intergenerational income mobility,” or “intergenerational income elasticity.” We searched for scholarly and peer-reviewed publications, working papers, government reports, and think tank reports. We also reviewed studies recommended during expert interviews as well as some included in the bibliographies of key studies on the topic of economic mobility. We used four criteria to target our literature search. In order to be included, studies had to: (1) include original estimates of at least one of three measures of intergenerational economic mobility: absolute economic mobility, relative economic mobility, and intergenerational income elasticity; (2) focus on the United States; (3) be published in the past 5 years (2014-2019), or 2 years if a working paper (2017-2019); and (4) be published in a U.S.-based publication. We then reviewed over 280 abstracts and further evaluated approximately 90 potentially appropriate studies, eliminating ones that did not meet our four criteria. A technical review of each study by at least two GAO economists included an assessment of key findings about economic mobility, methodology, data, assumptions, and limitations. Twenty studies met our four criteria and, based on our technical review, had sufficient methodological rigor for the purpose of providing information on economic mobility. Researchers attempting to estimate the degree of economic mobility in the United States face challenges in acquiring datasets with precise income measurements and that track incomes across generations with sufficient sample sizes. Potential reasons why researchers produce different estimates of economic mobility measures include: Differences in Datasets and Their Respective Limitations. Different datasets may not equally represent every segment of the population. For example, studies making use of the Panel Study of Income Dynamics (PSID) are not generalizable to populations not included in large numbers when the PSID began, such as recent immigrants and institutionalized populations. In addition, some studies rely on data that are not fully representative of the entire income distribution, either because they do not include a sufficient sample of households with very high income or, conversely, households with very low or zero earnings. Some datasets do not capture individuals who are not working or not filing taxes during the period of analysis. For instance, in one study making use of tax data, the authors noted that if parents never file a tax return, they cannot be linked to their child. In that study, parents of approximately 5 percent of children were not identified. In some cases, the data capture a limited age range, which leaves open the possibility of somewhat different results among different age ranges. In addition to different sampling strategies, datasets also capture different variables for each individual or household observed. Even the most comprehensive datasets currently available may lack the data to completely account for factors that may influence mobility, such as changes in family structure over time or detailed individual demographic characteristics for both parent and child households. Differences in Treatment or Construction of Variables. Estimates of intergenerational income mobility can be affected by choices the researcher makes, such as selecting a price deflator to inflation-adjust parents’ incomes; selecting the ages at which children and parents will be compared, accounting for changing trends in household size and composition; determining the value of non-cash benefits (e.g., employer-sponsored health insurance); and determining work-related costs associated with dual-earner households (e.g., child care). Some studies impute earnings for non-tax filers, and different methods of imputation may lead to slightly different results; in other studies, those with no reported income or observations with other missing variables (e.g., demographic characteristics) may simply be dropped from the dataset. How “parent” and “child” are defined may also differ across datasets (e.g., a parent could be the first adult to claim a child on their tax return, or could be an adult male living with a minor child in a household). Additionally, some studies required the researchers to construct datasets that matched parents and children at different points in time. Each researcher makes choices about how to handle the data, which can lead to different estimates. While we did not perform checks on these constructed data, the studies in our review generally included descriptions of the data and methodologies used as well as the difficulties and limitations associated with dataset construction, which we evaluated in our technical review. Differences in Choice of Economic Mobility Measure and Model Specification. Each measure of economic mobility provides a slightly different lens on mobility and has different interpretations. Absolute economic mobility, which compares the inflation-adjusted income of parents and children at similar ages, tends to reflect trends in overall economic growth and distribution of that growth. For instance, 92 percent of 30-year-olds in 1970 made more in inflation adjusted terms than their parents did at that age, while about half of children born in the 1980’s grew up to make more money than their parents by age 30. The difference may largely have been due to higher economic growth and a more equitable distribution of that growth along the income distribution from 1940-1970, whereas growth was slower and distributed differently between 1970 and the present. IGE offers a different metric with different limitations. Studies that estimate IGE regress log child income on log parent income. This conveniently yields a coefficient that can be interpreted as “the percent change in child income given a 1 percent change in parent income.” However, such estimates tend to be unstable because the relationship is non- linear and sensitive to the treatment of children with zero or very small incomes (because the log of zero is mathematically undefined). IGE is very sensitive to assumptions about the income of those with missing income data and typically does not include households with zero earnings, and so excludes some households with no income. Additionally, elasticities are sensitive to changes in cross-sectional income distributions (like during recessions). If children’s income distribution becomes more unequal, then the elasticity will become larger, all else equal. Despite these limitations, based on our technical review, all of the studies summarized in the report are of sufficient methodological rigor for the purpose of providing information on economic mobility. The authors of the studies we reviewed were generally aware of and transparent regarding the limitations of the datasets they worked with, and carried out analyses to test their results for robustness to different assumptions. Although there were differences in study datasets and methodologies, common themes emerge from the body of literature we reviewed. For example: None of the studies we reviewed found economic mobility to be increasing—all found it to be either flat or declining. While there was variation among studies regarding the exact degree to which parental income influences individuals’ income as adults, all studies we reviewed that examined parental income found it to be an important determinant of economic mobility. None of the studies that examined race found blacks to have higher mobility than whites. The studies we reviewed that examine geography agree that different locations have different economic mobility and that part of this variation is connected to the characteristics of a given place (such as school quality or level of segregation), not just to the characteristics of people who choose to live there. In other words, while the studies varied in their point estimates of various measures of economic mobility and its determinants, there was broad consensus among the studies regarding the sign (positive versus negative) and interpretation of the estimates. Additionally, these studies represent an advance in the data and analysis capabilities relative to past studies that examined economic mobility, largely because improved computing power has enabled more complex analyses of large datasets comprised of millions of records. See table 1 for the list of studies included in our review. After considering possible datasets, we chose the Survey of Consumer Finances (SCF) for this analysis because the data are appropriate for estimating measures of income and wealth across generations, including asset and debt categories of interest like homeownership and student debt. The SCF is a triennial survey of U.S. households sponsored by the Board of Governors of the Federal Reserve System in cooperation with the Department of the Treasury. Every 3 years, the SCF interviews a different sample of households and aims to be representative of households across economic strata, including the top of the wealth distribution. The SCF provides information on household balance sheets, including detailed information on assets and debts, as well as pensions, labor force participation, and demographic characteristics at the time of interview. We compared the financial circumstances of young households across 3 years of the SCF, as each year was representative of a generation (or birth cohort) when someone in the household (either the head of household or a spouse or partner) was 25-34 years old, following similar previous GAO work. Our analysis of SCF data allowed us to make intergenerational comparisons, but not to follow the same individuals over time, so we were not be able to compare children to their parents using these data. While our analysis allowed us to make comparisons, it did not allow us to make statements as to why Millennials are different or similar to other generations. Moreover, our data analysis focused on relatively older Millennials whose experiences may be different than those born later in the generation, especially due to the timing of the Great Recession. The SCF dataset is based on self-reported data and as a result, the data are subject to nonsampling error, including the ability to get information about all sample cases; difficulties of definition; differences in the interpretation of questions; and errors made in collecting, recording, coding, and processing data. Also, demographic analyses using these data may be limited based on the sample size needed to produce reliable estimates. Lastly, we cannot make predictions about the future financial circumstances of Millennials based on this snapshot in time. There are also limitations with the SCF with respect to making comparisons by gender. In a household headed by a single person, the head is taken to be the single core individual. However, in households headed by a central couple who is of mixed sex, the head is taken to be the male in the household. This assumption makes it difficult to make reliable comparisons by gender. Finally, the SCF generally asks questions of household heads and their spouses (and not others living in the household), so it likely underemphasizes young adults who were still living with their parents, which is more prevalent for the Millennial generation. Thus, there may be some selection bias in the SCF with respect to relatively more financially well-off Millennials. For the data used in our analysis, we reviewed documentation and tested the data for anomalies. We determined that these data were sufficiently reliable for the purposes of this report. We defined young households in each generation as those in which the household head or any spouse or partner was 25-34 years old. We compared Millennial households in 2016 to Generation X households in 2001 and Baby Boomer households in 1989. Baby Boomers were born from 1946 to 1964 and were 25-43 years old in 1989, so we used the 1989 SCF for Baby Boomer households when they were young adults. Generation X individuals were born from 1965 to 1981 and were 20- 36 years old in 2001, so we used the 2001 SCF for Generation X households when they were young adults. Millennials were born from 1982 to 2000 and were 16-34 years old in 2016, so we used the 2016 SCF for Millennial households when they were young adults. We used the SCF’s measures of income, net worth, assets, and debt from the summary extract data as measures of financial circumstances. We defined household income as the sum of income across all sources. Income includes a family’s cash income, before taxes, for the full calendar year preceding the survey. The components of income are wages, self-employment and business income, taxable and tax- exempt interest, dividends, realized capital gains, benefits from social safety net programs, pensions and withdrawals from retirement accounts, Social Security, alimony and other support payments, and miscellaneous sources of income for all members of the primary economic unit in the household. We defined household net worth as assets minus debt. Assets include financial assets, including liquid assets in bank accounts, certificates of deposit, money market accounts, stocks and bonds, cash value of life insurance, retirement accounts, and other financial assets. Assets also include nonfinancial assets, such as the value of vehicles, primary residences, other residential property, businesses, and other nonfinancial assets. Debt includes mortgages, home equity loans, credit card balances, education loans, vehicle loans, other installment loans, and other debt, including loans against pensions or life insurance. Households could have financial resources outside of net worth, including future income from defined benefit plans or Social Security; however, we did not attempt to estimate the actuarial present value of these financial resources in our net worth calculation given the long time horizon to retirement and the amount of uncertainty associated with such a measurement. In addition, in our professional judgment, the inclusion of these financial resources would not have altered our finding that Millennials have lower net worth compared to previous generations; the inclusion of these financial resources would likely have widened the gap further between Millennials and previous generations because previous generations had greater access to DB plans than the Millennial generation. We estimated means and medians for variables of interest, both overall and by quartile. We estimated the standard errors and constructed the confidence intervals taking into account the dual-frame sample design in order to estimate the sampling variance for these estimates. One part of the design is a standard, multistage area-probability design, while the second part is a special over-sample of relatively wealthy households. This is done in order to accurately capture financial information about the population at large as well as characteristics specific to the relatively wealthy. The two parts of the sample are adjusted for sample nonresponse and combined using weights to make estimates from the survey data nationally representative of households overall. Unless otherwise indicated, estimates in this report are statistically significant at the p<.05 level, and the error bars in the figures represent the 95 percent confidence intervals for the estimates. We conducted this performance audit from November 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Michael J. Collins (Assistant Director), Jessica K. Rider (Analyst-In-Charge), Jessica Mausner, Kathleen McQueeney, and Layla Y. Moughari made key contributions to this report. Also contributing to this report were James Bennett, Alicia Cackley, Pin-En Annie Chou, Justin Dunleavy, Sarah C. Gilliland, Gina M. Hoover, Susan J. Irving, Dan Luo, Sheila R. McCoy, John W. Mingus Jr., Corinna Nicolaou, Oliver M. Richard, Vernette G. Shaw, Joseph Silvestri, Almeta Spencer, Frank Todisco, and Adam Wendel.", "summary": "The idea that individuals should have the opportunity to economically advance beyond the circumstances of their birth is a familiar element of the American Dream. In an economically mobile society, it is possible for individuals to improve their economic circumstances through effort, education, investment, and talent. In addition to opportunities through the private, public, and nonprofit sectors, the federal government also promotes economic mobility through many efforts, including supporting education, job training, business incentives and development, and child health and well-being programs. However, a recent survey indicates that over approximately the last two decades fewer people report being satisfied with the opportunity to get ahead by working hard. According to recent studies, the Millennial generation, who comprise the largest portion of the American workforce, report feeling overwhelmed by their financial situation and concerned about their future financial security. GAO was asked to review trends in economic mobility and Millennials' economic situation compared to previous generations. This report examines (1) what is known about intergenerational income mobility, and (2) how the financial circumstances of Millennials compare to previous generations. To perform this work GAO conducted an extensive literature review and analyzed data from the nationally representative Survey of Consumer Finances. Recent research indicates that, across three key measures, economic mobility in the United States is limited. Specifically, the Millennial generation (those born between 1982 and 2000) might not have the same opportunity as previous generations had to fare better economically than their parents. According to studies GAO reviewed, the share of people making more money than their parents at the same age (absolute mobility) has declined over the last 40 years, and the chances of moving up the income distribution (relative mobility) have been flat over time. Using a third measure of economic mobility (intergenerational income elasticity), researchers have found that income in adulthood is linked to how much a person's parents made, and that between one-third and two-thirds of economic status is passed down from parents to children. This is especially true of the lowest and highest income groups. Researchers also identified race and geography as key determinants of an individual's economic mobility. Millennials have different financial circumstances than Generation X (born 1965-1981) and Baby Boomers (born 1946-1964), and in light of flat or declining economic mobility, there is uncertainty about how they will fare financially as they age. A snapshot of data that allowed GAO to compare Millennials aged 25-34 to the previous two generations at similar ages showed that Millennial households were more likely than other generations to be college educated; however, incomes have remained flat across the three generations, implying that Millennials have not yet benefited from the potential additional lifetime income earned by college graduates. Millennial households had significantly lower median and average net worth than Generation X households at similar ages (see figure), especially among those with low net worth. Median net worth for the lowest quartile of Baby Boomers and Generation X was around zero, but it was substantially negative for Millennials, indicating that debt was greater than assets for the median low net worth Millennial household. Regarding assets, a significantly lower percentage of Millennials owned homes compared to previous generations at similar ages, but had retirement resources at rates comparable to Generation X and Baby Boomers. Finally, Millennials were more likely to have student loan debt that exceeded their annual income. It remains to be seen how these factors will affect Millennials' financial circumstances in the long run, including retirement.", "document_type": "gao"}
{"report": "In fiscal year 2018, VHA data show that 518,570 veterans received any treatment (specialty or non-specialty services) from VHA’s health care systems for a diagnosed SUD, a 9.5 percent increase from fiscal year 2016 (see figure 1). Because these data include non-specialty services, the data do not indicate the extent to which the veteran received SUD services. For example, a provider briefly discussing the SUD of a veteran in long-term recovery during a primary care visit would be included in SUD treatment data. VHA data show that the majority of veterans who received any treatment from VHA’s health care systems for a diagnosed SUD had an alcohol use disorder. Veterans received any treatment from VHA for a diagnosed SUD at a higher rate than the general population. Data from the 2017 National Survey on Drug Use and Health indicate that 1.5 percent of individuals aged 18 or older nationwide received any SUD treatment in the past year. In comparison, 8 percent of veterans getting health care provided or purchased by VHA received any treatment for a diagnosed SUD in fiscal year 2017, including individuals who received specialty SUD services as well as individuals who received non-specialty services in, for example, primary care or general mental health clinics. VHA’s health care systems provide specialty SUD services in three settings increasing in intensity (see figure 2): Outpatient services. Individual and group therapy, either in person or via telehealth, among other services. VHA also offers intensive outpatient programs, which provide services for 3 or more hours per day, 3 days a week at a minimum. Residential rehabilitation treatment programs. Medically monitored, high-intensity care in a 24-hour supervised environment specifically dedicated to treating SUDs. These programs may also provide social services for community reintegration and treatment for other medical conditions during a veteran’s stay. Inpatient services. Acute in-hospital care, which may include detoxification services. For veterans with opioid use disorder—a subset of SUDs—VHA’s health care systems provide medication-assisted treatment. Medication- assisted treatment combines behavioral therapy and the use of certain medications, including methadone and buprenorphine. Medication- assisted treatment has proven to be clinically effective in reducing the need for inpatient detoxification services for individuals with opioid use disorder, according to SAMHSA. Methadone. This medication suppresses withdrawal symptoms during detoxification. It also controls the craving for opioids in maintenance therapy, which is ongoing therapy meant to prevent relapse and increase treatment retention. Methadone is a controlled substance and, when used to treat opioid use disorder, may generally be administered or dispensed only within a certified opioid treatment program to help prevent diversion. Buprenorphine. This medication eliminates opioid withdrawal symptoms, including drug cravings, and it may do so without producing the euphoria or dangerous side effects of other opioids. It can be used for detoxification and maintenance therapy. Buprenorphine is also a controlled substance, and when used to treat opioid use disorder, may be administered or dispensed within an opioid treatment program, or prescribed or dispensed by a qualifying provider who has received a waiver to do so. Providers who receive this waiver are limited in the number of patients they may treat for opioid use disorder. In addition to medication-assisted treatment, VHA has initiatives aimed at preventing opioid-related overdose deaths. For example, VHA’s Opioid Overdose Education and Naloxone Distribution program includes education and training regarding opioid overdose prevention as well as naloxone distribution. Naloxone is a medication that can reverse opioid overdoses. Veterans may receive services from community providers via local contracts or community care. For local contracts, individual VA medical centers establish contracts with local community providers. For example, a VA medical center may develop a contract with a community residential rehabilitation treatment program provider to set aside a number of beds specifically for veterans. For community care, veterans may be eligible if, for example, VHA does not offer the care or service the veteran requires or VHA cannot provide the care or services consistent with its access standards. In general, community care services must be authorized in advance of when veterans access the care. Prior to June 6, 2019, eligible veterans could receive community care via one of multiple VHA community care programs. In 2018, the VA MISSION Act required VA to implement a permanent community care program that consolidated several community care programs. On June 6, 2019, the consolidated community care program, the Veterans Community Care Program, went into effect. Among the 518,570 veterans who received SUD services in fiscal year 2018, VHA provided specialty SUD services to 152,482 veterans in fiscal year 2018. This number has increased slightly but remained relatively unchanged since fiscal year 2014, as shown in table 1 below. These veterans received care in VHA’s health care systems—that is, in VA medical centers or in one of the medical centers’ affiliated outpatient clinics and other medical facilities. During the same time period, VHA expenditures for these specialty SUD services increased from $552 million in fiscal year 2014 to $601 million in fiscal year 2018. Total specialty SUD expenditures per capita increased from $3,691 to $3,941 from fiscal years 2014 through 2018. Adjusted for inflation, however, per capita expenditures remained relatively unchanged between fiscal years 2014 and 2018. Our analysis of VHA data shows that veterans received specialty SUD services from VHA’s health care systems in multiple settings from fiscal years 2014 through 2018, with most veterans receiving these services in outpatient settings. Veterans may receive specialty SUD services across multiple settings within a year. Below, we provide information on utilization and expenditures for specialty SUD services in outpatient and residential treatment programs and for medication-assisted treatment for veterans with opioid use disorder. In fiscal year 2018, nearly all veterans who received specialty SUD services from VHA’s health care systems received this care in outpatient settings at some point during the year. Of those veterans who received outpatient specialty SUD services, 17 percent received intensive outpatient specialty SUD services, with little change from previous years. Expenditures for outpatient specialty SUD services increased from fiscal years 2014 through 2018, as shown in table 2 below. During this time period, outpatient specialty SUD expenditures per capita increased from $2,176 to $2,348. Adjusted for inflation, per capita expenditures grew 1.5 percent between fiscal years 2014 and 2018. In addition, we found little change in the number of full-time employee equivalents that actively provided outpatient specialty SUD services from fiscal years 2015 through 2018. VHA did not provide specialty outpatient wait-time data because, according to VHA officials, the data do not reliably capture veterans’ wait times to receive SUD services in outpatient settings. VHA officials explained that veterans may receive non-specialty SUD services in various outpatient settings, including primary care and general mental health clinics. Therefore, developing a wait-time measure for specialty SUD services would not accurately capture whether veterans are waiting for SUD services not previously provided or services that would continue ongoing treatment begun in a primary care or general mental health clinic. As a result, we did not analyze outpatient wait-time data. In prior work, we have made recommendations to VHA on ways it can improve its outpatient wait-time data (see sidebar). As of fiscal year 2018, VHA had residential rehabilitation treatment programs available for veterans with complex and long-term mental health needs at 113 facilities, and 67 of these programs were dedicated to SUD treatment. The number of residential rehabilitation treatment programs dedicated to SUD treatment increased from fiscal years 2014 through 2018, as did the number of beds available. Figure 3 shows the location of all 67 residential rehabilitation treatment programs specifically dedicated to SUDs with the corresponding number of beds in fiscal year 2018. See appendix III for more information on residential rehabilitation treatment programs dedicated to SUD treatment. The number of veterans participating in VHA’s specialty SUD residential rehabilitation treatment programs (that is, those dedicated to SUD treatment) remained relatively stable from fiscal years 2014 through 2018, as shown in table 3. Of the veterans who received specialty SUD services in fiscal year 2018, approximately 10 percent participated in one of VHA’s 67 residential rehabilitation treatment programs dedicated to SUD treatment, similar to previous years. Meanwhile, expenditures for VHA’s residential rehabilitation treatment programs dedicated to SUD decreased from fiscal years 2014 through 2016, but increased in fiscal years 2017 and 2018. Similarly, specialty SUD residential expenditures per capita decreased from $15,386 in fiscal year 2014 to $12,526 in fiscal year 2016 and increased again to $16,031 in fiscal year 2018. After adjusting for inflation, specialty SUD residential expenditures per capita in 2018 were about 2 percent less than what they were in 2014. From fiscal years 2014 to 2018, veterans’ average length of stay for VHA’s specialty residential rehabilitation treatment programs specifically dedicated to SUD generally decreased, while wait times varied across programs. Across VHA’s residential rehabilitation treatment programs dedicated to SUD treatment, veterans’ average length of stay generally decreased from fiscal years 2014 to 2018, from nearly 40 days to nearly 36 days. VHA officials said that average length of stay may have decreased as a result of multiple factors, such as programs with longer lengths of stay adjusting their treatment approaches. The median wait times to enter residential rehabilitation treatment programs dedicated to SUD treatment varied considerably, ranging from 0 days to 56 days across the programs in fiscal year 2018, although not all residential rehabilitation treatment programs had sufficient—and therefore reliable— data on wait times. Specifically, out of the 67 residential rehabilitation treatment programs dedicated to SUD, VHA officials identified 12 that did not have sufficient wait-time data, which we excluded from our analysis. VHA officials noted that some specialty residential rehabilitation treatment programs do not have sufficient wait-time data because the facilities do not consistently code whether a patient’s visit included a screening for admission to the program. As such, VHA cannot tell when patients were initially screened for admission. In fiscal year 2019, officials implemented changes to address the lack of reliable data from some facilities. However, it is too early to tell if the new changes will address the data reliability issues in wait-time data for residential rehabilitation treatment programs. VHA health care systems offer veterans medication-assisted treatment for opioid use disorder in a variety of settings, including outpatient specialty SUD settings and residential rehabilitation treatment programs dedicated to SUD treatment, as well as in non-specialty settings, such as primary care and general mental health clinics. Our analysis of VHA data shows the number and proportion of veterans with an opioid use disorder who received medication-assisted treatment from VHA’s health care systems has risen in recent years, as shown in table 4. In fiscal year 2018, 23,798 veterans received medication-assisted treatment, which was 33.6 percent of veterans diagnosed with an opioid use disorder. Veterans with an opioid use disorder may receive medication-assisted treatment through VHA at a lower rate than individuals who received care through private insurance. According to a study by the Department of Health and Human Services, 50.6 percent of individuals diagnosed with an opioid use disorder and enrolled in private insurance received medication-assisted treatment in 2014 to 2015. Some veterans may also have private insurance and may have received their medication-assisted treatment through that private insurance. In fiscal year 2018, 9,132 (38 percent) of the veterans who received medication-assisted treatment received their care at one of VHA’s 33 opioid treatment programs, which is the only setting where methadone can be administered to treat opioid use disorder. Expenditures for these opioid treatment programs increased from $35.9 million in fiscal year 2014 to $39.1 million in fiscal year 2018. In fiscal year 2018, VHA had 2,036 providers with a waiver to prescribe buprenorphine, a 17.6 percent increase from fiscal year 2017. According to VHA officials, VHA has encouraged its providers—including those who are not specialists in treating SUDs, such as primary care providers—to obtain the waiver required to prescribe buprenorphine to treat opioid use disorder. In fiscal year 2018, there were about 29 VHA providers with a waiver to prescribe buprenorphine for every 1,000 veterans with opioid use disorder, a 14 percent increase from fiscal year 2017. VHA’s naloxone kit distribution increased exponentially from 646 in fiscal year 2014 to 97,531 kits in fiscal year 2018. A total of 204,557 naloxone kits have been distributed through fiscal year 2018. VHA health care systems distributed naloxone kits to VA staff, including VA first responders and VA police officers, and veterans with opioid use disorder. Factors contributing to the increase may include: In 2014, VHA implemented the Opioid Overdose Education and Naloxone Distribution initiative to decrease opioid-related overdose deaths among veterans, with one of its key components focused on encouraging naloxone kit distribution. Since the program’s implementation, all VHA health care systems dispense naloxone kits. The Comprehensive Addiction and Recovery Act of 2016 directed VHA to maximize the availability of naloxone to veterans and to ensure that veterans who are considered at risk for opioid overdose have access to naloxone and training on its proper administration. Veterans Health Administration (VHA) Community Care Wait Times GAO has a body of work highlighting challenges VHA has with the reliability of its wait-time data. See below for recent reports about this issue. We have highlighted the importance of reliable community care wait-time data in a testimony regarding VHA’s efforts to address our previous recommendations on these issues. See GAO, Veterans Health Care: Opportunities Remain to Improve Appointment Scheduling within VA and through Community Care, GAO-19-687T (Washington, D.C.: July 24, 2019). We have designated our past recommendations related to community care wait-time data as priorities for the agency. See GAO, Priority Open Recommendations: Department of Veterans Affairs, GAO-19- 358SP (Washington, D.C.: Mar. 28, 2019). We have previously made recommendations to VHA to capture the necessary information and improve the reliability of wait-time data for community care. These recommendations remain outstanding as of October 2019. See GAO, Veterans Choice Program: Improvements Needed to Address Access- Related Challenges as VA Plans Consolidation of its Community Care Programs, GAO-18-281 (Washington, D.C.: June 4, 2018). Through its community care programs, VHA purchased SUD services (specialty and non-specialty) for 20,873 veterans in fiscal year 2018, a significant increase since fiscal year 2014 (see table 5). VHA officials noted that veterans can receive community care in addition to, or instead of, care at a VHA facility; therefore, the number of veterans served through community care cannot be combined with the number who received services within VHA to provide an overall number of veterans receiving care. Expenditures for these SUD services purchased by VHA also increased over time, from nearly $6 million in fiscal year 2014 to over $80 million in fiscal year 2018. Between fiscal years 2014 and 2018, on a per capita basis, SUD services purchased by VHA increased from $3,021 to $3,852. Per capita expenditures adjusted for inflation also increased during this time period. These increases coincided with the establishment of the Veterans Choice Program in early fiscal year 2015, which expanded eligibility for community care. Wait-time data for SUD services purchased through community care were not available because of data reliability issues, VHA officials told us. See sidebar for more information on our previous recommendations to VHA regarding community care wait-time data. While VHA is able to report on the overall number of veterans receiving SUD services through community care, data limitations prevent VHA officials from reliably determining whether veterans received this care in residential or outpatient settings. These issues are as follows: Residential rehabilitation treatment programs. VHA uses billing codes on paid claims to track the settings in which veterans receive community care; however, according to agency officials, there is no specific billing code for a residential setting. VHA officials told us that community residential rehabilitation treatment programs may record treatment provided using inpatient or outpatient billing codes—or a combination of the two—in submitting claims to VHA. As a result, VHA is unable to use claims data to reliably identify veterans who received residential rehabilitation treatment through community care. Outpatient settings. Because some residential care data are coded using outpatient billing codes, outpatient data may contain residential services counted as outpatient services. As a result, VHA is unable to reliably identify veterans who received SUD services in community care outpatient settings. Currently, VHA is taking steps to address these coding issues. VHA officials told us they are developing a payment code that will bundle together common residential program services, which will allow VHA to identify veterans receiving residential rehabilitation treatment for SUDs through community care. Officials explained that using this code for residential SUD services will allow VHA to better distinguish between residential and outpatient community care because residential care will no longer need to be identified using outpatient codes. In contrast to its community care programs, VHA does not centrally track SUD services provided via local contracts. Rather, the individual medical centers that established the contracts with local community providers are responsible for tracking and documenting SUD services provided to veterans. In fiscal year 2019, VHA began conducting market assessments, a broader agency initiative to better understand the supply and demand of all services at all VA medical centers, including both what is available within VHA as well as what is available in the local communities. We reviewed one of the data collection instruments the agency is using as a part of this work and found that it should allow VHA to identify, among other things, the number of community residential rehabilitation treatment beds contracted by individual medical centers to serve veterans with SUDs, as well as the number of veterans who received SUD services through local contracts or community care for SUDs. Agency officials said that they expect the market assessments to be completed in 2020. Although overall use of SUD services was similar among veterans in rural and urban areas, VHA data show the utilization rates of some specialty SUD services differed. The literature and agency documents we reviewed and VHA officials consistently cited several issues, such as recruiting SUD providers and accessing necessary prescriptions for SUDs, which affect the use of services by veterans with SUDs in rural areas. According to agency documents and officials, VHA is taking steps to address these issues. Overall, veterans’ use of SUD services was similar in rural areas compared to urban areas, but use of some specialty services differed. Our analysis of VHA data shows that across VHA’s 140 health care systems, there was relatively little difference in the overall utilization of SUD services (specialty and non-specialty) in rural and urban areas from fiscal years 2016 through 2018. In fiscal year 2018, for example, 7.5 percent of veterans in rural areas received any SUD services compared with 8.8 percent of veterans in urban areas. However, VHA data also show there were some types of specialty services, such as intensive outpatient specialty services, residential rehabilitation treatment programs, and medication-assisted treatment for opioid use disorder, that rural veterans with SUDs tended to use more or less of than their urban counterparts. Among veterans receiving specialty SUD services across all 140 VHA health care systems, those veterans in rural locations used intensive outpatient specialty SUD services at a slightly higher rate (19 percent) than veterans in urban locations (17 percent) in fiscal year 2018. While veterans’ utilization of these specialty SUD services has decreased in both rural and urban locations in recent years, the decreases have been larger in rural areas. In rural locations, the percentage of veterans using intensive outpatient specialty SUD services decreased from 25 percent in fiscal year 2015 to 19 percent in fiscal year 2018. In comparison, in urban areas, the percentage of veterans using these services decreased from 18 percent to 17 percent during this same time period. Officials from VHA health care systems in three urban locations and two rural locations we spoke with indicated that they offered intensive outpatient specialty SUD services in conjunction with either residential or outpatient services. According to officials from the rural VHA health care system that did not offer this service, the location did not have sufficient staff to provide the additional hours of intensive outpatient specialty SUD treatment each week. Veterans in rural locations using specialty SUD services participated in residential rehabilitation treatment programs dedicated to SUD treatment at a higher rate (17 percent) than veterans using these services in urban locations (10 percent) across all 140 VHA health care systems in fiscal year 2018. From fiscal years 2014 through 2018, there was a slight increase in the percentage of rural veterans using specialty SUD services who participated in residential rehabilitation treatment programs dedicated to SUD treatment, from 13 percent to 17 percent. VHA officials told us rural communities often face difficulties with transportation that may make residential programs more feasible than accessing intensive outpatient specialty SUD services, which are at least 3 days per week, at VHA health care systems. All six of the VHA health care systems we interviewed offered residential rehabilitation treatment programs. VHA reported the agency is currently conducting market assessments that may help determine gaps in services for veterans with SUDs, including residential rehabilitation treatment, once the assessments are complete. Across all 140 VHA health care systems, veterans with an opioid use disorder received medication-assisted treatment (in specialty and non- specialty settings) at a higher rate in urban locations (34 percent) than in rural locations (27 percent) in fiscal year 2018. We also found differences in the availability of medication-assisted treatment services between rural and urban areas: Methadone. The only setting in which methadone may be used to treat an opioid use disorder is an opioid treatment program. All of VHA’s opioid treatment programs are located in urban areas. Only one of the six selected VHA health care systems in our review had an opioid treatment program. Officials from the other five VHA health care systems we spoke with told us they typically referred out to community providers if a veteran needed methadone. Regional VHA officials indicated that some locations, especially rural ones, may not have the number of veterans with opioid use disorder needed to justify the resources required to run an opioid treatment program. Buprenorphine. The number of waivered providers per 1,000 veterans with opioid use disorder was slightly higher in rural areas (29.9 providers) than in urban areas (28.7 providers) in fiscal year 2018. Non-specialist rural providers, such as primary care providers, may feel a greater responsibility to obtain a waiver because there are fewer specialists for them to refer their patients to, according to VHA health care system officials. Despite the similar rates of waivered providers in rural and urban areas, as previously mentioned, rural veterans with opioid use disorder use medication-assisted treatment at a lower rate. VHA requires that all rural and urban health care systems offer the same range of SUD services (specialty or non-specialty). However, rural areas have historically faced difficulties delivering all types of health care, including SUD services, according to literature, agency documents, and VHA health care system officials we spoke with. VHA is taking steps to address several issues that affect the delivery of health care services generally, and SUD services in particular, in rural areas. Officials from three of the six VHA health care systems we interviewed noted a shortage of SUD specialists in their area, including addiction therapists and providers with a waiver to prescribe buprenorphine. According to one study and agency documents we reviewed, veterans may reside in mental health professional shortage areas at a higher rate than the general population, therefore they may have less access to providers qualified to offer medication-assisted treatment or other mental health treatment. One study found that efforts to improve access for veterans in rural areas by purchasing care from community providers may have limited effect, because these areas are relatively underserved generally. Officials from two of the three VHA health care systems in rural areas we selected expressed difficulty hiring and retaining providers to provide SUD services. Because of the shortages, recruiting and retaining providers to deliver care to rural veterans is critical. Based on the literature reviewed and half of VHA health care system officials interviewed, rural communities struggle with recruiting and retaining providers, including SUD providers. Some rural areas report provider shortages with ongoing, long-term vacancies. To respond to these provider shortages and hiring and retaining challenges, VHA has implemented new initiatives and practices to increase the supply of rural health professionals. A VHA official noted that these efforts include rural health training and education initiatives to provide rural health experience to health professions trainees, including those who provide SUD services. The agency also plans to use expanded recruitment tools, like greater access to an education debt reduction program, improved flexibility for bonuses for recruitment, relocation, and retention, as well as piloting a scholarship program authorized under the VA MISSION Act of 2018 to hire mental health professionals. However, recruiting health professionals in rural areas, including mental health providers and social workers, remains an issue for VHA and the community at large, and VHA officials noted that data are not yet available to understand the long-term effect of the newly trained providers on the availability of SUD services. Officials from two VHA health care systems we interviewed noted that providing services, such as medication-assisted treatment, through telehealth technology is difficult, especially when the SUD service requires monitoring for medication compliance. However, a VHA official told us the use of telehealth services overall has grown exponentially at VHA’s health care systems and goes beyond traditional video conference capabilities to include advanced technology that can be attached to computers or videoconference equipment like an exam camera to allow for an interactive examination. The official added that the provision of SUD services using telehealth can be supported by medical personnel located at the closest VA facility to complete necessary tests, such as urine screening, when the service is provided at a VHA location. VHA officials from one health care system we spoke with and literature noted that providing medication-assisted treatment via telehealth technologies requires a cultural change within the profession. Officials from one VHA health care system we spoke with told us that delivering medication-assisted treatment using technology is risky. For example, buprenorphine is a controlled substance with a risk of misuse. This official added that many providers may not be open to the idea of delivering this level of treatment using telehealth. One study we reviewed confirmed that acceptance within the profession appears to be the main barrier to the successful implementation of telehealth services. However, VHA’s budget and strategic plan show continued support for the use of telehealth for SUD treatment. Studies have shown that telephone services, a type of telehealth service, potentially have the same outcomes as in-person services. Officials from all six VHA health care systems we selected mentioned they had mental health telehealth services available to facilitate the delivery of care to veterans in both urban and rural areas for SUD services. To ensure adequate access to care, VHA has multiple telehealth initiatives underway. For example, between fiscal years 2017 and 2019, VHA allocated $28.5 million for mental health telehealth hubs at 11 sites. In another instance, VHA allocated more than $750,000 for rural facilities in fiscal years 2018 and 2019 toward a nationwide initiative to improve participation in a program that establishes video connections in the homes of rural veterans to receive mental health treatment, including for SUDs, with psychotherapy and psychopharmacology. While VHA has initiatives underway, the success of these efforts is contingent on rural areas having broadband and internet connectivity, which remains a challenge, according to agency documents and officials. VHA’s Clinical Practice Guidelines for SUDs recommends methadone and buprenorphine, among others drugs, to treat opioid use disorder. However, accessing these drugs in rural areas can be challenging, according to literature we reviewed and VHA officials we spoke with. For example, one national study found that opioid treatment programs providing methadone are generally absent from the treatment options in rural areas. Within VHA, all of the opioid treatment programs are in urban areas. In addition, in rural areas generally, a small percentage of providers nationwide have received waivers to prescribe buprenorphine. VHA officials told us they are steadily expanding the availability of medication-assisted treatment for veterans with opioid use disorder. VHA had an interdisciplinary team of VA staff from a single facility within each region receive training on implementing medication-assisted treatment for opioid use disorder. These teams were responsible for spreading information to other facilities. Thus far, VHA reported it has trained over 300 providers using this model. In a separate initiative, a VHA official reported that its Office of Rural Health provided over $300,000 in fiscal year 2019 for a pilot program that trains primary care and mental health providers in the Iowa City VHA health care system on how to provide medication-assisted treatment for opioid use disorder. The availability of transportation is vital for veterans receiving medication- assisted treatment due to the necessity for frequent travel to the VHA health care systems for treatment. When using methadone for opioid use disorder treatment, the medication generally needs to be administered through an opioid treatment program at a specific location on a daily basis. In addition, during the initial stages of buprenorphine treatment, patients must also come into a facility frequently. Veterans living in rural areas who need this level of care may have to travel long distances every day to receive this medication. Distance and lack of transportation impede access to care, including SUD services, for rural veterans. Specifically, the literature we reviewed noted distance, time, and access to transportation as barriers to care. Veterans may lack access to transportation or are no longer able to drive because of age, health status, or driving restrictions. Some rely on family, friends or vans available through community service organizations; however, they may have other difficulties like reaching pick-up locations or the organization not having vans that are wheelchair-equipped. Officials from all six VHA health care systems we selected noted the lack of transportation as a barrier to accessing SUD services. Officials from two rural locations of the six selected VHA health care systems mentioned that volunteers, including a local veteran service organization, assist with getting veterans from their homes to their appointments; however, they added that these services operate on an abbreviated schedule and veterans are sometimes subjected to riding in the vehicle for long periods of time (2 hours each way). Over the last 10 years, a VHA official told us that the agency has allocated between $10 and $12.9 million for its Veterans Transportation Service for new vehicles, drivers, and mobility managers to assist with rural transportation needs. The VA MISSION Act of 2018 includes provisions that specifically address the need to improve veterans’ access to health care in areas with shortages of health care providers, including those providing SUD and mental health services. Based on this legislation, in June 2019, VHA published a plan organized in three areas: increasing personnel, using technology to connect veterans to care through public and private partnerships, and expanding VHA’s infrastructure through the building or acquiring of space to address the problem of underserved facilities. For example, VHA has a pilot program with 11 Walmart sites and 15-20 additional sites planned with Philips Healthcare, the Veterans of Foreign Wars, and the American Legion to enable veterans who lack the necessary technology in their home and live far from a VHA facility to receive remote health care at a convenient location. VHA’s plan indicates that while all VHA health care systems can use any of the strategies covered under this legislation, they will provide specific additional technical assistance for underserved facilities, monitor the effectiveness of these strategies, and share the findings of this work throughout the broader VHA system. We provided a draft of this report to VA for review and comment. VA provided written comments, which are reprinted in appendix IV, and technical comments, which we incorporated as appropriate. VA’s comments note that the agency generally reports obligations and that the agency is unable to confirm some of our financial data. However, the data provided by VA during the course of this engagement were regarding expenditures, and thus we report them as such. VA’s comments also provide information on additional efforts to expand mental health telehealth and ways the agency recruits providers in rural areas. We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at deniganmacauleym@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Figure 4, an interactive graphic, shows the location and rurality of the Veterans Health Administration’s health care systems, as well as information on veterans treated by these health care systems. For an accessible version of the data used in this map, see https://www.gao.gov/products/GAO-20-35. To describe any differences between veterans’ use of substance use disorder (SUD) services in rural and urban areas and the issues affecting access to those services in rural areas, we selected six Veterans Health Administration (VHA) health care systems and interviewed officials regarding their SUD services and issues serving veterans with SUDs. Because opioid use disorders may pose a greater risk to veterans than the general population, we selected the six VHA health care systems from among those with the highest percentages of veterans with an opioid use disorder diagnosis in fiscal year 2018. We also selected these six health care systems to achieve variation in representation among VHA’s five geographic regions and to include both urban and rural locations. See table 6. The Veterans Health Administration had 67 residential rehabilitation treatment programs dedicated to substance use disorder treatment in fiscal year 2018. See table 7. Mary Denigan-Macauley, (202) 512-7114 or deniganmacauleym@gao.gov. In addition to the contact named above, Lori Achman, Assistant Director; Hannah Marston Minter and Carolina Morgan, Analysts-in-Charge; Sam Amrhein; Amy Andresen; Shaunessye D. Curry; and John Tamariz made key contributions to this report. Also contributing were Giselle Hicks, Diona Martyn, Ethiene Salgado-Rodriguez, and Emily Wilson Schwark. Veterans Health Care: Opportunities Remain to Improve Appointment Scheduling within VA and through Community Care, GAO-19-687T. Washington, D.C.: July 24, 2019. VA Health Care: Estimating Resources Needed to Provide Community Care, GAO-19-478. Washington, D.C.: June 12, 2019. Drug Policy: Assessing Treatment Expansion Efforts and Drug Control Strategies and Programs, GAO-19-535T. Washington, D.C.: May 9, 2019. Priority Open Recommendations: Department of Veterans Affairs, GAO-19-358SP. Washington, D.C.: March 28, 2019. Behavioral Health: Research on Health Care Costs of Untreated Conditions is Limited, GAO-19-274. Washington, D.C.: Feb. 28, 2019. Veterans Choice Program: Improvements Needed to Address Access- Related Challenges as VA Plans Consolidation of its Community Care Programs, GAO-18-281. Washington, D.C.: June 4, 2018. VA Health Care: Progress Made Towards Improving Opioid Safety, but Further Efforts to Assess Progress and Reduce Risk Are Needed, GAO-18-380. Washington, D.C.: May 29, 2018. Opioid Use Disorders: HHS Needs Measures to Assess the Effectiveness of Efforts to Expand Access to Medication-Assisted Treatment, GAO-18-44. Washington, D.C.: October 31, 2017. Opioid Addiction: Laws, Regulations, and Other Factors Can Affect Medication-Assisted Treatment Access, GAO-16-833. Washington, D.C.: September 27, 2016. VA Health Care: Reliability of Reported Outpatient Medical Appointment Wait Times and Scheduling Oversight Need Improvement, GAO-13-130. Washington, D.C.: December 21, 2012.", "summary": "Substance use and illicit drug use are a growing problem in the United States. SUDs occur when the recurrent use of alcohol or drugs causes significant impairment, such as health problems. The veteran population has been particularly at risk. Veterans are 1.5 times more likely to die from opioid overdose than the general population, according to VA and Centers for Disease Control and Prevention data. Furthermore, veterans live in rural areas at a higher rate than the general population, which may affect their ability to access SUD services. VA is the largest integrated health care system in the United States, providing care to about 6.2 million veterans. VA provides SUD services through outpatient, inpatient, and residential care settings and offers various treatment options, including individual and group therapy, medication-assisted treatment, and naloxone kits to reverse overdoses. Senate Report 115-130 included a provision for GAO to study VA's capabilities to treat veterans with SUDs. This report describes (1) trends in the number of and expenditures for veterans receiving SUD services, including specialty SUD services; and (2) any differences between veterans' use of SUD services in rural and urban areas, and the issues affecting access to those services in rural areas. GAO reviewed VA policies and data from fiscal years 2014 through 2018. GAO also interviewed officials from six VA health care systems, selected for their high percentage of veterans with an opioid use disorder and to achieve variation in geography and locations VA has designated as urban and rural. VA provided technical comments, which GAO incorporated as appropriate. The Department of Veterans Affairs (VA) treated 518,570 veterans diagnosed with a substance use disorder (SUD) in fiscal year 2018, a 9.5 percent increase since fiscal year 2016. Of these, 152,482 veterans received specialty SUD services in fiscal year 2018, a number that has remained relatively unchanged since fiscal year 2014. Specialty SUD services are those provided through a clinic or program dedicated to SUD treatment. Expenditures for VA's specialty SUD services increased from about $552 million in fiscal year 2014 to more than $600 million in fiscal year 2018. In the same year, VA expended about $80 million to purchase SUD services from non-VA community providers for more than 20,000 veterans, an increase since fiscal year 2014. The number receiving this care from non-VA providers may include veterans who also received services in VA facilities. Note: Specialty SUD services are those provided through a clinic or program dedicated to substance use disorder treatment. SUD services include services provided by any type of provider. VA data show that overall there was little difference in the percentage of veterans using SUD services, including specialty services, in rural and urban areas in fiscal year 2018. However, there were differences for some specific services. For example, in rural areas, 27 percent of veterans with an opioid use disorder received medication-assisted treatment—an approach that combines behavioral therapy and the use of medications—compared to 34 percent in urban areas. In providing SUD services in rural areas, VA faces issues similar to those faced by the general population, including lack of transportation. The agency is taking steps to address these issues, such as using local service organizations to transport veterans for treatment.", "document_type": "gao"}
{"report": "The Navy bases the majority of its surface ships at homeports in the United States, and five regional maintenance centers manage their maintenance. At the time of our review, the Navy had 38 surface ships assigned to overseas homeports, as illustrated in figure 1. A homeport is where a ship is based and primarily managed and maintained. The Navy assigns all newly commissioned ships entering the fleet to a U.S. homeport, and the Navy may change a ship’s homeport throughout its service life. The Navy may move a ship to an overseas homeport to respond to strategic needs or to relieve another ship returning to a U.S. homeport. We found in May 2015 that basing ships at overseas homeports provides considerable additional time in strategic areas of operation and other benefits ranging from increased opportunities for collaboration with partners and allies to faster response time for emerging crises. However, we also found that the Navy’s high pace of operations for its overseas-homeported ships affected the material condition of these ships, and that they had experienced a worsening trend in overall ship readiness when compared to U.S.- homeported ships over the preceding 5 years. We also reported that the Navy generally intended ships to be homeported overseas for about 7 to 10 years, according to officials, but that some ships in Japan had been based there for longer than 10 years. In 2018 Congress instituted a 10- year cap on the length of time certain U.S. Navy ships may be based at overseas homeports. A number of organizations and commands within the Navy share responsibilities for setting maintenance policies and planning, scheduling, and executing ship maintenance, from the offices of the Secretary of the Navy and Chief of Naval Operations, to fleet commanders and ships’ crews. Key organizations include: Type Commanders. The Navy’s type commanders for surface ships— Commander, Naval Surface Force, U.S. Pacific Fleet, and Commander, Naval Surface Force, U.S. Atlantic Fleet—are responsible for maintaining, training, and ensuring the readiness of the surface ships assigned to each fleet. Naval Sea Systems Command (NAVSEA). NAVSEA, among other things, maintains surface ships to meet fleet requirements within defined cost and schedule parameters. These offices perform contract administration, program management, and planning for future maintenance periods informed by the historical maintenance needs of Navy ships. For example, the following NAVSEA organizations have certain responsibilities for overseas ship maintenance: NAVSEA’s Deputy Commander for Ship Maintenance and Modernization (NAVSEA 21). This office provides life-cycle management for surface ships and manages critical modernization, maintenance, training, and inactivation programs. NAVSEA’s Surface Maintenance Engineering Planning Program (SURFMEPP). SURFMEPP provides life-cycle management of maintenance requirements for surface ships, including providing centralized class maintenance and modernization planning and management of maintenance strategies. Commander, Navy Regional Maintenance Center (CNRMC). This office oversees the regional maintenance centers in the United States, as well as the Forward Deployed Regional Maintenance Center (FDRMC) headquarters in Italy, and its detachments in Rota, Spain, and Manama, Bahrain, that manage the maintenance for the U.S. Navy ships homeported there. NAVSEA’s Logistics, Maintenance, and Industrial Operations (NAVSEA 04). This office manages and oversees the naval shipyards and the Ship Repair Facility and Japan Regional Maintenance Center (SRF-JRMC) in Yokosuka, Japan, and its detachment in Sasebo, Japan. Surface Team One. This body of stakeholders from across the Navy’s surface ship maintenance, modernization, and sustainment organizations collaborates for the purpose of setting and developing surface ship maintenance and modernization priorities, conducting analyses, and improving surface ship maintenance performance. A Senior Flag Oversight Council comprised of Commander, Naval Surface Force Pacific, and Commander, NAVSEA, provides strategic vision and directs Surface Team One’s efforts, which may include knowledge-sharing networks, working groups, or deep-dive studies and business case analyses. The level of complexity of ship repair, maintenance, and modernization can affect the length of a maintenance period, which can range from a few weeks to 6 months or longer. The types of maintenance periods include the following: Chief of Naval Operations (CNO) maintenance. CNO maintenance periods are scheduled to accomplish industrial, depot-level maintenance and modernization—work that cannot be conducted by ship’s crews or goes beyond fleet capabilities. These depot-level maintenance periods can last 6 months or longer and the Navy generally schedules them every 2 to 3 years throughout a ship’s service life. This can include major repair, overhaul, or complete rebuilding of systems needed for ships to reach their expected service life, and involve complex structural, mechanical, and electrical repairs. For example, in certain types of depot-level maintenance, ships are taken out of the water and put into a dry dock to perform maintenance on below-water parts of the ship (see fig. 2 for a photo of a dry dock at SRF-JRMC in Yokosuka, Japan). To inform the planning of the work package for this maintenance period, Navy officials or contractor representatives typically perform one or more “ship checks” to assess the material condition of the ship in advance of the maintenance period. Continuous maintenance. Continuous maintenance periods are to conduct maintenance outside of the longer CNO maintenance periods that can be done in short periods typically scheduled to be 2 to 6 weeks in duration. According to Navy officials, the schedules of these periods can vary, and commanders can adjust, postpone, or cancel them based on operational demands. Voyage repair. Voyage repair maintenance periods are solely to accomplish corrective maintenance of a mission- or safety-essential nature necessary for a ship to deploy or continue its deployment. For example, ships based in the United States that are deployed overseas on a temporary basis schedule mid-deployment voyage repair to ensure they can continue their deployment. The process for planning surface ship depot-level maintenance periods (i.e., CNO maintenance periods), whether the ship is based overseas or in the United States, is contained in the Navy’s Joint Fleet Maintenance Manual. In general, the Navy begins planning for a ship’s depot-level maintenance period 720 days—or roughly 2 years—before the planned start of the maintenance period. During this time, a variety of organizations within the Navy plan what will be repaired, how long it will take, where the work will be done, as well as select the contractors to perform the work, among other things. This process also includes activities to close out the maintenance period once it is complete, which overlap with the start of the planning cycle for the next maintenance period. For example, certain milestones serve both planning and closeout purposes—such as the Life-cycle Planning Conference Meeting, which is to both closeout a ship’s completed maintenance period and to begin planning for the next one by reviewing the maintenance requirements, deferred work, and planned schedules (see figure 3). NAVSEA 21, including SURFMEPP, is generally responsible for the advanced planning of maintenance periods, which includes setting the baseline requirements and early estimates of how long maintenance might take. In general, regional maintenance centers have overall responsibilities for meeting milestones approximately a year prior to the start of maintenance through execution and closeout, as illustrated in figure 3 above. Overseas, the responsible regional maintenance centers are the SRF-JRMC at the homeport in Yokosuka, Japan, and its detachment at the homeport in Sasebo, Japan, and the FDRMC detachments at the homeports in Rota, Spain, and Manama, Bahrain. Naval Supply Systems Command’s Fleet Logistics Centers offices overseas are responsible for soliciting and awarding maintenance contracts, for ships based overseas, among other things. The Navy has developed different maintenance capacity and approaches to maintain the 38 surface ships based in Japan, Spain, and Bahrain. The Navy maintains these ships through a mix of Navy, host government, and contractor industrial base facilities and resources that are different at each location. The Navy has tailored the maintenance approaches it uses at each homeport considering the available Navy and contractor capacity, as well as the number and type of ships, according to Navy documents and officials. Table 1 provides an overview of the Navy and contractor industrial base capacity for depot-level maintenance of surface ships based at the four main overseas homeports. U.S. Naval Ship Repair Facility and Japan Regional Maintenance Center (SRF-JRMC), Yokosuka, Japan. The Navy’s largest overseas maintenance facility, SRF-JRMC is located in Yokosuka and is responsible for the maintenance of 12 surface ships homeported there— including the most destroyers and the only cruisers based outside of the United States. According to Navy officials, SRF-JRMC in Yokosuka operates as a public shipyard would in the United States, with three on- base dry docks that fit all sizes of ships based there, as well as other smaller dry docks. SRF-JRMC employs a Japanese workforce that conducts the majority of the maintenance workload through a cost- sharing agreement between the United States and Japan. For example, in fiscal year 2018, SRF-JRMC directly conducted about two-thirds of the total ship maintenance workload, with about one-third conducted by local contractors, according to SRF-JRMC workload reporting documentation. For the contracted work, SRF-JRMC relies on one main contractor, Sumitomo Heavy Industries, for ship maintenance in Yokosuka, though additional smaller contractors are also used. Most contracted work also takes place at Navy facilities on base, according to SRF-JRMC officials. Ships in Yokosuka are able to receive deeper, more complex maintenance than other ships based overseas because of the Navy maintenance capacity at SRF-JRMC, according to NAVSEA officials. SRF-JRMC in Yokosuka also conducts detailed planning for maintenance periods that other regional maintenance centers do not, according to NAVSEA officials. Specifically, it plans all the individual maintenance and repair tasks to be conducted in each maintenance period, while other U.S. and overseas maintenance centers can rely on the contractors to plan the work they do. For additional information on SRF-JRMC in Yokosuka, Japan, see appendix III. SRF-JRMC Detachment, Sasebo, Japan. The Navy also operates its own shipyard with a Japanese workforce at the SRF-JRMC detachment in Sasebo, though it primarily relies on the local contractor base to conduct maintenance work. In fiscal year 2018, the SRF-JRMC detachment directly conducted about one-third of the total maintenance workload, with nearly two-thirds performed by contractors according to SRF-JRMC workload reporting documentation. For the contracted work, the Navy relies on about 14 smaller contractors, and while the SRF- JRMC detachment coordinates the work of the multiple contractors that may contribute to a single maintenance period, the contractors directly plan and manage their portion of the work, according to Navy officials. The SRF-JRMC detachment in Sasebo includes two Navy dry docks, though only one is used for depot-level maintenance periods. As a result, dry-dock maintenance and modernization can be conducted on ships based in Sasebo, but it is generally limited to the four MCM and two LSD ships. The other amphibious ships based in Sasebo receive depot-level maintenance that has been planned from about 2 to as long as nearly 9 months, but this does not include dry-dock maintenance. A unique maintenance consideration in Sasebo is the deployment schedule of the amphibious ships based there. These ships typically deploy three at a time with U.S. Marines based in Okinawa on board. As a result, there are times when all ships are in port and require maintenance, so the detachment tries to stagger the work with the MCMs and closely coordinate with contractors there in an effort to manage workload, according to SRF-JRMC officials. For additional information on the SRF- JRMC detachment in Sasebo, Japan, see appendix IV. Forward Deployed Regional Maintenance Center (FDRMC) Detachment, Rota, Spain. The FDRMC detachment and four destroyers are based in Rota, Spain, where a single state-owned contractor, Navantia, performs all depot-level maintenance on the ships. Beginning in 2014, the Navy deployed four destroyers to Spain to support the U.S. ballistic missile defense mission to the North Atlantic Treaty Organization. The Navy designed the maintenance approach for these ships with the understanding that they would not require access to Navy- or contractor- operated dry docks during the time they are based in Spain, according to Navy officials. The Navy initially expected these destroyers to be in Spain for about 6 years and to receive maintenance every 2 years. However, in 2015 the Navy updated its maintenance strategy for these ships to provide shorter, but more frequent maintenance periods to support a longer time basing them in Spain. Under the updated approach, the Navy plans for each destroyer to receive six maintenance periods during a roughly 8-year time period based in Spain. For additional information on the FDRMC detachment in Rota, Spain, see appendix V. FDRMC Detachment, Manama, Bahrain. The FDRMC detachment in Bahrain is responsible for the depot-level maintenance of the 10 patrol coastal and 4 mine countermeasures ships based there—the most ships based at an overseas homeport. While the Navy does not operate any dry docks or depot-level repair facilities in Bahrain, it relies on two main contractors, Bahrain Ship Repairing and Engineering Company and Arab Shipbuilding and Repair Yard, to conduct ship maintenance in Bahrain. The ships in Bahrain receive depot-level maintenance at contractor facilities there. Both contractors in Bahrain have dry docks or similar capacity to fit the MCMs and PCs based there, as well as some larger Navy ships. A unique capacity consideration for ships visiting Bahrain, according to officials there, is that the Navy does not have dedicated pier space for ships when they come into port. As a result, the Navy must rely on contractor space for maintenance, and on other pier space when visiting ships are at the homeport—which they share with others, such as commercial cruise lines. For additional information on FDRMC detachment in Manama, Bahrain, see appendix VI. In addition to the depot-level maintenance periods for the surface ships we reviewed, the Navy maintenance centers in Japan, Spain, and Bahrain, also support additional maintenance functions, such as voyage repairs or technical assistance for visiting U.S. ships; coordinating intermediate-level maintenance that may be conducted there; and providing additional maintenance support to overseas ships outside of scheduled depot-level periods. The Navy did not complete the majority of the maintenance periods performed on ships based overseas on time during fiscal years 2014 through 2018. Navy officials identified a variety of factors that contribute to delays, such as the discovery of additional work requirements after maintenance has begun or staff shortages affecting management and oversight of maintenance. The Navy collects information on overseas maintenance at individual homeports, but its analysis of factors contributing to the delays is generally focused on the planning and execution of individual maintenance periods. The Navy underestimated the time needed to complete maintenance for 50 of the 71 maintenance periods—about 70 percent—started during fiscal years 2014 through 2018. Specifically, 21 maintenance periods ended early or on time and 50 maintenance periods ran beyond their planned schedules, as illustrated in figure 4. More than half of the maintenance periods that were completed late—29 of 50—went 31 or more days beyond the Navy’s planned schedule. As a result, from 2014 through 2018 there were 29 times when ships based overseas were unavailable for operational requirements, certain training, or other purposes for 31 or more unplanned days. During this time period, the Navy completed more maintenance periods a month or more later than planned than it completed early or on time. As a result of maintenance schedules not being completed on time, all four overseas Navy homeports with surface ships we analyzed— Yokosuka, Japan; Sasebo, Japan; Rota, Spain; and Manama, Bahrain— experienced a total of 3,475 days ships were in maintenance beyond their expected durations—referred to in this report as days of maintenance delay. As illustrated in figure 5, Manama, Bahrain, experienced the most days of maintenance delay during fiscal years 2014 through 2018, while Rota, Spain, experienced the least. We also analyzed delays at overseas homeports by calculating the days of delay experienced as a percentage of its total workload in terms of total days of maintenance conducted. Using this analysis, we found that ships in Bahrain experienced the highest rate of delay at 34 percent while ships based in Rota, Spain, experienced only a 2.2 percent rate of delay (as illustrated in figure 6). Taking workload into account illustrates some difference in the rate at which each of these homeports experiences ship maintenance delays. For example, ships in Sasebo and Yokosuka experienced a similar total number of days ship maintenance was delayed—1,001 days and 994 days over the 5-year time period, respectively. However, when port workload is taken into account, Sasebo’s rate of delay is higher. Specifically, ships based in Sasebo experienced a maintenance delay rate of 31.2 percent compared with 18.5 percent of the time for the surface ships in Yokosuka. According to Navy maintenance center officials and crewmembers from the ships we visited, and our analysis of Navy information, a number of interrelated factors and issues contribute to maintenance delays for the surface ships based overseas including: Discovery of additional, unplanned work after maintenance is underway. According to maintenance officials in Bahrain and Japan, the discovery of the need for additional maintenance and repair work after the work planned for the maintenance period has been finalized is a key driver of maintenance delays. This additional work can be in the form of growth in the magnitude of planned work, or identification of the need for new work that was not previously planned. For example, maintenance officials in Japan attributed maintenance delays they experienced on ships at both Yokosuka and Sasebo during fiscal years 2016 through 2018 to this growth in planned work or new work. Similarly, officials in Bahrain said that growth and new work is one of many contributing factors to maintenance delays for the aging MCMs and PCs based there. For example, officials from Commander, Naval Surface Squadron Five that track their ships’ depot maintenance identified that additional work to stern tubes on the USS Squall, which is homeported in Bahrain, resulted in the ship’s maintenance schedule being extended by 137 days. Navy officials also stated that the reason growth and new work is such a key driver of delays is that it can add further delays beyond that needed to complete the repair, due to time required for additional contract actions and ordering parts that are needed to conduct the added work. A number of factors can cause or further exacerbate growth and new work, according to Navy officials. For example, the Navy has made efforts to catch up on backlogs of deferred maintenance and improve the health and condition of the ship, so the Navy may decide to extend the maintenance period to ensure all identified maintenance has been completed rather than deferring it to a subsequent maintenance period. Additionally, officials pointed to ships’ complex propulsion, communication, and weapons systems that have complicated maintenance and modernization requirements that cannot always be fully anticipated. Missing or late maintenance planning milestones. The Joint Fleet Maintenance Manual emphasizes the importance of meeting planning milestones to identify, estimate, and schedule the work to be done in the maintenance period. These milestones include steps to guide advanced planning of initial maintenance requirements and schedules, and to further refine and develop the work, cost, and schedule estimates for each maintenance period. For example, these milestones include assessments of the ship’s condition and other ship checks to identify and validate planned work intended to minimize growth and new work; to identify and mitigate risks to planned schedules; and to provide deadlines for developing and awarding contracts to do the work. Adherence to these planning milestones becomes more critical as the planned start of the maintenance period approaches to ensure work can be contracted and begun on time. The final contract is awarded about 2 months prior to work beginning, and the Navy finalizes the planned duration and schedule of the maintenance period about a month before maintenance is scheduled to begin. According to Navy officials, missing or late planning milestones can contribute to maintenance delays. For example, NAVSEA and overseas maintenance officials emphasized that getting on board a ship at various points in the planning process to assess the ship’s condition and validate planned work is critical to developing accurate work scope, cost estimates, and schedules—otherwise, growth and new work or other issues can emerge once maintenance is underway. According to the Joint Fleet Maintenance Manual, ship checks are needed to inform specific planning milestones, to validate planned work, and should be done as early in the planning process as possible. The Navy requires this validation to ensure needed maintenance work is sufficiently defined, problems are accurately diagnosed, and feasible resolutions are recommended. However, even though ship condition assessments are important milestones to limit growth and new work, NAVSEA officials part of Surface Team One said that these assessments and other checks are regularly postponed, which can prevent work from being identified with sufficient time to plan for it. Similarly, maintenance officials in Japan said that, due to the operational tempo in Yokosuka and Sasebo, ships are often not available for required ship checks until the ship arrives in port at the start of its maintenance period. Though officials could not provide the frequency that such milestones are missed, they said missing assessments and other milestones can contribute to schedule delays and result in maintenance periods exceeding planned resources. For example, the Naval Inspector General found that the shortage of personnel at the FDRMC and Fleet Logistics Center in Bahrain resulted in contracting milestones being routinely missed for ships based there, and once these ships were in maintenance, the growth in work to be completed grew by an average of $830,000 for maintenance periods in fiscal years 2017 and 2018. Shortages of experienced and skilled personnel for planning, management, and oversight. According to NAVSEA and overseas maintenance center officials, shortages of U.S. personnel that perform maintenance planning, contracting, and oversight roles, particularly staff with critical skills and experience, can affect ship maintenance and contribute to delays. For example: Personnel shortages hinder staffing of project teams. FDRMC Bahrain officials said that due to personnel shortages, they are often unable to assign staff to the project teams until the maintenance period starts. According to the Joint Fleet Maintenance Manual, a project team is assigned to manage an individual maintenance period, and is composed of personnel with specific skills and responsibilities. Additionally, according to CNRMC Instruction 4790.4B, the project team is responsible for key maintenance planning and execution activities and related milestones from as early as a year before the maintenance begins. CNRMC Instruction 4790.4B also states that such maintenance planning milestones are to aid in developing project plans, identifying and mitigating risks, and tracking progress of planning. Project teams are also responsible for overseeing contracted maintenance work and ensuring it meets quality standards. For example, prior to the start of the maintenance period, project teams are responsible for identifying and mitigating risks to completing maintenance within the planned schedule and budget. However, officials in Bahrain stated that as a result of persistent staffing shortages, they have been unable to staff these project teams until the maintenance period begins, and have also been unable to provide sufficient oversight of the contractors’ performance during the maintenance period, which has resulted in maintenance delays. Shortages of personnel with relevant experience affect management and oversight of maintenance. Officials in Japan and Bahrain stated that insufficient numbers of personnel with ship maintenance experience can negatively affect maintenance timeliness. For example, the Fleet Logistics Center in Bahrain—which manages the contracting process for ships based there—had only eight of 18 authorized U.S. civilian contracting-related positions filled, as of March 2019, according to officials. Additionally, of the filled positions, only one contracting officer had prior experience with ship maintenance contracting, according to Fleet Logistics Center officials. Officials in Japan said that experience levels of U.S. civilians at SRF-JRMC have decreased as a result of high turnover in recent years with the average amount of work experience for U.S. civilians managing ship maintenance in Sasebo declining from over 5 years in 2014 to 3 years in 2017. Staff shortages on ships affect crews’ ability to conduct maintenance. Navy officials also emphasized the importance of ship crews in identifying and providing needed maintenance work, but noted that ship crew shortages negatively affect on-board ship maintenance. This can increase the amount of work required during depot-level maintenance periods. In May 2017, we reported that reduced crew sizes contributed to maintenance being deferred and increased maintenance costs, and Navy officials and ships’ crews we spoke to in Japan and Bahrain stated that ships there continue to experience manning shortages. For example, from September 2018 through February 2019, nearly 30 personnel from Bahrain-based Navy organizations were temporarily assigned to ships based in Bahrain to fill manning shortages, according to Navy officials and information, including for maintenance-specific positions. According to maintenance officials overseas and in the United States, other factors also can add to the complexity of maintenance planning and contribute to delays including the length of time it takes to obtain spare parts overseas, availability of obsolete parts, and other challenges associated with maintaining aging ships, such as the MCMs and PCs, which are at or beyond their original service lives. The Navy uses a number of mechanisms to monitor the planning and execution of individual maintenance periods to track progress and mitigate possible risks. According to Navy documentation and officials, these mechanisms include: Individual homeports identify technical reasons for delays on individual maintenance periods. Maintenance centers overseas and in the United States monitor the planning and progress of individual maintenance periods. SRF-JRMC officials in Yokosuka, Japan, monitor ongoing and recently completed maintenance periods and may identify technical causes for ship delays. For example, new work was identified on the main reduction gear of the USS Barry that was not in the planned work package and led to delays, according to officials. Additionally, Commander, Naval Surface Squadron Five in Bahrain tracks instances of growth and new work during the depot- level maintenance periods for the PCs and MCMs based there, including tracking the specific number of delayed days attributed to certain issues. NAVSEA conducts regular meetings to report status of upcoming and ongoing maintenance. NAVSEA collects information on and monitors the progress of individual maintenance periods, including at overseas homeports, through a variety of regular meetings and briefings. For example, NAVSEA 04 and CNRMC each conduct biweekly meetings with their respective maintenance centers to monitor advanced planning of upcoming maintenance periods and the progress of ongoing maintenance periods for the ships under their responsibilities, according to officials. Information shared during these briefings can include tracking whether certain planning milestones are met and identifying risks to the on-time completion of individual ships’ maintenance periods. This information is then compiled into monthly briefings to the NAVSEA commander providing a snapshot of upcoming and ongoing maintenance periods and seeking approval for adjustments, according to officials. Collecting and sharing lessons learned throughout the planning process. According to the Navy’s maintenance manual and related guidance, the collection and sharing of lessons learned from individual maintenance periods is to be part of certain planning milestones, including to inform the maintenance schedule and work estimates. For example, CNRMC Instruction 4790.4B directs that maintenance completion conferences with relevant stakeholders are to provide a detailed review of the maintenance period, including lessons learned that can be used to plan future maintenance periods, such as to revise specific work items. According to CNRMC and NAVSEA 04 officials, lessons learned are collected at the end of each maintenance period and can be shared with other project teams. The Joint Fleet Maintenance Manual also states that while the lessons learned process is owned by the type commanders—for surface ships, these are Commander, Naval Surface Force, U.S. Pacific Fleet for ships in Japan and the western United States, or Commander, Naval Surface Force, U.S. Atlantic Fleet, for ships in Spain, Bahrain, and the eastern United States—the lessons learned process is part of the Surface Team One structure. However, Surface Team One officials noted that each of the milestones that include them is led by other Navy organizations, and its role in the lessons learned process is managed by a part-time contracted position. CNRMC tracks overall days of maintenance delay by fiscal year. CNRMC tracks and monitors the overall number of days individual ship maintenance periods are delayed and can perform analysis of overall delays, such as the number of days experienced by ship class and fiscal year. Additionally, CNRMC analysis has also identified specific ships that experience the longest delays, though it did not regularly include maintenance periods in Japan until 2018, according to officials. CNRMC tracks costs associated with growth and new work for individual maintenance periods. CNRMC tracks the costs associated with growth and new work discovered during maintenance periods by the regional maintenance centers it manages, including at overseas detachments in Bahrain and Spain. The costs that are tracked do not include information on any related delays, however, and do not include these costs for the ships in Japan. Other recent Navy efforts have begun to examine issues related to delays. According to Navy officials, several Navy entities are beginning efforts to improve the execution of surface ship maintenance. For example, in fiscal year 2019 the Navy began a broad effort to improve Navy surface ship, submarine, and aviation readiness, as well as public shipyards. This effort, called Performance to Plan, designates Commander, Naval Surface Forces, and Commander, NAVSEA, to improve performance of ship maintenance in private and public shipyards. According to Navy officials, the effort to improve surface ship maintenance consists of a pilot program examining how to better execute maintenance periods for destroyers, improve forecasts of maintenance period duration and assessments of ship condition, planning for growth and new work, and adherence to planning milestones. However, officials said this effort is still in the early stage and does not specifically assess maintenance delays for ships based overseas. NAVSEA’s SURFMEPP and Surface Team One also have recently begun related efforts. For example, SURFMEPP officials said they recently began an effort to examine and correct causes of growth and new work by analyzing changes to contracts or work items that result in more than $100,000 of additional cost. However, while officials said in July 2019 that this effort has been underway for about 9 months, they could not provide additional information on how it relates to delays. According to NAVSEA officials that co-chair Surface Team One, it has begun an effort to improve how adherence to key planning milestones is tracked across surface ship maintenance periods. To support this effort, in October 2018 the Navy updated the Joint Fleet Maintenance Manual to include additional requirements for meeting maintenance milestones and to document any changes, including reasons for those changes. However, according to officials, these efforts are in their early stages, and the Navy has not used the information to analyze maintenance delays for overseas ships. Although a number of different Navy entities conduct a variety of activities through which information on maintenance delays is collected and analyzed, these efforts are limited as the existing analysis is not comprehensive and systematic in nature. Specifically, the Navy has not positioned itself well to address the factors contributing to the maintenance delays because it has not (1) designated an individual entity responsible for conducting a single, comprehensive systematic analysis of overseas surface ship maintenance delays; and (2) developed a plan based on that analysis to address these delays. First, this is in part because the responsibilities for managing surface ship maintenance overseas is shared among NAVSEA 21, CNRMC, and NAVSEA 04, which use somewhat different processes for their work, according to officials. For example, NAVSEA 04 has responsibility for the maintenance of aircraft carriers and submarines at naval shipyards, while CNRMC focuses on surface ships. In addition, until SRF-JRMC was brought under control of NAVSEA in October 2018, CNRMC was not regularly including maintenance periods in Japan as part of its tracking and monitoring of days of maintenance delay. According to officials, an operating instruction to align roles, responsibilities, and processes for surface ship maintenance in Japan between CNRMC and NAVSEA 04 is being developed, but as of September 2019, this instruction had not yet been finalized. Further, CNRMC and NAVSEA 04 officials pointed to NAVSEA 21 or Surface Team One as more appropriate entities to conduct a comprehensive systematic analysis of ship maintenance delays given their broad, enterprise-wide roles for managing and improving surface ship maintenance. Surface Team One officials said that it could be an appropriate entity to conduct such analysis, and according to its charter, one of the entity’s purposes is to measure performance of the planning and execution of surface ship maintenance periods and to manage and improve schedule, cost, and quality. However, officials said they have not conducted such a systematic analysis of maintenance period performance or developed a comprehensive plan to address them, in part due to inconsistent organizational leadership and personnel turnover. According to officials, since its founding in 2009, Surface Team One has been re-chartered twice and is in the process of further reorganizing under a fourth version of its charter. Part of the reason for this reorganization, according to officials, is to resource and structure Surface Team One to conduct more systematic, enterprise-wide analyses of issues affecting surface ship maintenance, for which they hope to develop a plan by the end of 2019. However, officials said these efforts did not specifically include analysis of maintenance delays for ships based overseas. Additionally, while Navy officials said that Performance to Plan efforts could help inform overseas maintenance delays, this effort is in the early stages of a pilot effort looking only at destroyer maintenance, and does not specifically analyze maintenance delays for ships based overseas. Second, as a result of there being no single, comprehensive analysis of overseas surface ship maintenance delays, there is no plan for the Navy to improve the timeliness of its maintenance in a holistic way. Instead, individual organizations and maintenance centers have identified improvements for individual ships’ maintenance or have undertaken efforts to address certain contributing factors to delays. While these efforts are important, given the interrelated challenges related to maintenance across the Navy, and that the Navy is dependent upon synchronized and timely maintenance to provide ships for operations to meet national security needs, the Navy would benefit from a plan of action that was comprehensive in nature. Standards for Internal Control in the Federal Government state that management should assign responsibility to achieve objectives and remediate deficiencies; compare actual performance against planned performance; and evaluate deficiencies on both an individual basis and in the aggregate. Further, OMB Circular No. A-123, Management’s Responsibility for Enterprise Risk Management and Internal Control, emphasizes that when developing corrective actions, agencies should perform a root-cause analysis of the deficiency and ensure that subsequent strategies and plans address the root of the problem and not just the symptoms. Additionally, our past work on results-oriented management cites a number of key practices that can strengthen the use of performance information for process improvements. These practices include aligning agency-wide goals and measures, and building analytic capacity to use the information. Our past work has further shown this information should then be incorporated into improvement plans that include identifying analytically based goals; results-oriented metrics to measure progress; required resources, risks, and stakeholders to achieve those goals; and regularly reporting on progress. While several different Navy entities have a variety of efforts underway related to issues associated with ship maintenance delays, without designating an entity to conduct a comprehensive, systematic analysis to identify and understand the underlying causes maintenance periods grow beyond planned schedules, the Navy risks continuing to underestimate maintenance needs and the time and resources required to address them. Further, without conducting such an analysis to understand the underlying, interrelated causes of these delays, and incorporating this analysis into a comprehensive results-oriented plan to address them, the Navy cannot effectively target corrective actions to improve timely completion of ship maintenance to ensure ships are available for the critical training crews need and operations to support U.S. military and national security goals. The Navy is in the process of updating the maintenance approach for cruisers, destroyers, and amphibious ships based in Japan, but it has not assessed and mitigated risks that several challenges may pose to its successful implementation. Additionally, the Navy has not included assessments of overseas maintenance requirements in its long-range plans to support fleet growth to 355 ships. The Navy has developed a new maintenance approach for the cruisers and destroyers in Yokosuka and the amphibious ships in Sasebo based on the approach developed for destroyers in Spain. Specifically, the Navy developed a new maintenance approach for the four destroyers it began to deploy to Rota, Spain, in 2014 and 2015 that includes generally shorter, but more frequent, maintenance periods. According to maintenance center and other Navy officials, the Navy developed this approach to avoid conducting dry-dock maintenance overseas so that the Navy could maximize the time the ships were available for operations. According to officials, the Navy tailored this approach to the specific ships, mission, and maintenance resources available in Rota. For example, the four destroyers in Rota conduct patrols two ships at a time with predictable patrol schedules. With such specific operational and maintenance schedules officials said there is little margin for changes, and adjustments or delays could affect the ships’ operational availability to support their ballistic missile defense mission. Under this approach, the Navy completed the majority of its maintenance on these four ships during fiscal years 2014 through 2018 on time—with only 20 total days of maintenance delay, equating to a relatively low overall delay rate of 2.2 percent. Navy officials said that the new maintenance approach in Rota has been successful because the Navy: selected four ships with a high degree of commonality; for example, the ships were of similar age, systems, and equipment configuration, which helped facilitate planning for and conducting maintenance; ensured the ships received all needed maintenance and modernization before being sent to Spain, and arrived from the United States in good condition, which reduced the likelihood that they would require unexpected maintenance while overseas; designed the maintenance center and its staffing around the maintenance approach for the four destroyers; and coordinated with the contractor in Spain to ensure it had sufficient workforce and resources, including capacity to surge resources if additional work is discovered so that it can be completed on time. Based on the performance of the maintenance approach for destroyers in Spain, officials stated that the Navy began to develop a similar approach in 2016 for its ships in Japan. NAVSEA officials identified that shorter, more frequent maintenance could help ensure that its ships based in Japan received the maintenance they need, while also meeting their high operational demands. The Navy finalized a new maintenance approach for cruisers and destroyers in Yokosuka in December 2018, and was in the process of finalizing the maintenance concept for the amphibious ships in Sasebo, according to NAVSEA officials in June 2019. For example, like in Spain, the Navy has adjusted the schedules for the planned periods in Yokosuka to be shorter, but more frequent. Planning documents show that under the new approach for the destroyers in Yokosuka, the Navy plans to provide them with eight maintenance periods over approximately 8 years overseas before rotating the ships back to the United States. Previously, the Navy planned for destroyers in Japan to receive eight maintenance periods over an estimated span of over 16 years overseas under the prior approach. Under the new approach, the surface ships in Japan are expected to receive all required maintenance, including completing most or all backlogged maintenance according to officials, in the United States before relocating the ships to Japan. Additionally, while officials expect ships in Yokosuka to receive some dry-dock maintenance during their rotation overseas, the amphibious ships in Sasebo generally will not—similar to the arrangement for destroyers in Spain. As a result, the new maintenance approach expects that ships in Sasebo will accrue maintenance backlogs that must be resolved upon return to the United States. The Navy has decided to apply its new maintenance approach for cruisers, destroyers, and amphibious ships in Japan and in 2018 began initial implementation on certain ships already based there, but a number of challenges may pose risks to successful implementation of the strategy. Based on information from planning documents and officials, successful implementation relies on several planning assumptions that may be optimistic when compared to actual experience maintaining surface ships overseas and in the United States. Specifically, the new approach in Japan assumes that: Ships will receive robust, deep maintenance and modernization in the U.S. and meet their life-cycle health requirements prior to overseas assignment. Ships will receive and complete planned maintenance on time while overseas to maximize operational availability. Ships will rotate back to receive full maintenance in the United States after no longer than 9 years of overseas assignment. However, Navy officials and our analysis identified several challenges: (1) U.S. industrial base maintenance capacity limitations, (2) maintenance delays in the United States and overseas, (3) the ability of the overseas contractor industrial base to support future workload in Japan, and (4) differences in the operating environments between Spain and Japan. These challenges, which are discussed below in more detail, could pose risks to the successful implementation of the new maintenance approach. U.S. industrial base maintenance capacity limitations. Implementing the new maintenance approach in Japan assumes that the ships identified for deployment will receive all required maintenance and modernization in the United States prior to being based overseas. However, the Navy has been challenged to do this in the past due to limited domestic maintenance capacity. For example, the Navy deferred maintenance assessments of the condition of the USS Barry and USS Milius that were to take place in the United States before moving the ships to Japan. As a result, Navy officials said these assessments had to be done in Japan. Additionally, upon arriving in Japan in November 2017, the USS Barry had to begin immediate unscheduled maintenance to correct various issues, and as of our visit in February 2019, was still undergoing maintenance. According to U.S. Pacific Fleet and maintenance center officials, in fiscal year 2014 the USS Curtis Wilbur received modernization in Japan due to lack of capacity in the United States. Further, Navy planning documents identified U.S. commercial dry-dock capacity shortfalls that may hinder the Navy’s ability to support the future maintenance workload in the United States. For example, the Navy’s Long-Range Plan for the Maintenance and Modernization of Naval Vessels for Fiscal Year 2020 identified limited U.S. dry-dock capacity in the United States as posing a significant challenge to maintenance of U.S.-homeported ships and that this situation reduces the margin for schedule changes. According to the Navy’s analysis, demand for surface ship maintenance in the United States will exceed available maintenance resources for fiscal years 2019 through 2026. During this time, the Navy will be rotating ships based in the United States to exchange with those currently based in Japan and Spain. Navy officials said the capacity shortfall in the United States negatively affects ship condition and maintenance of ships sent to Japan. However, the Navy’s analysis does not account for the need to perform deep maintenance and modernization on ships in the United States before and after sending them to overseas homeports, as required by the new maintenance approach for ships bound for Japan, as well as Spain. Maintenance delays in the United States and overseas. Maintenance delays at both U.S. and overseas homeports may also affect the Navy’s implementation of its new maintenance approach. Successful implementation of the new approach depends in part on ships receiving all required maintenance on time prior to moving overseas, as well as receiving timely maintenance during their time based abroad. Our analysis of Navy surface ship maintenance periods that started in fiscal years 2014 through 2018 found that about 60 percent of maintenance periods in the United States ran 31 or more days beyond schedule. Additionally, our analysis shows that ships homeported at both U.S. and overseas locations experienced an average rate of delay of about 25 percent (see fig. 7). Additionally, rates of delay in Sasebo, where the Navy plans to implement one of its new maintenance approaches, exceed 30 percent. According to Navy officials, the new maintenance approach for ships in Japan is intended to provide more frequent maintenance periods, in an effort to improve ship maintenance and to maximize ships’ availability for operations. However, the approach also relies on most of these maintenance periods being shorter—and being completed on time. Given the Navy’s history of persistent maintenance delays in Japan, this could be a challenge. Further, Navy officials said that maintenance delays experienced in the United States could also affect the maintenance that ships bound for and returning from overseas homeports may receive, and pose a risk that maintenance will be deferred to overseas homeports. Challenges with overseas contractor industrial base meeting future workload in Japan. Navy maintenance officials in Spain said that successful implementation of the new maintenance approach there relied on sufficient contractor capacity overseas, and that the Navy involved the contractor in the development of the maintenance approach to ensure they could implement it. In contrast, Navy officials in Japan stated that current contractor capacity may not meet expected future workload. For example, Navy documentation shows that contractors performed almost two-thirds of ship maintenance in Sasebo in fiscal year 2018. Additionally, the documentation shows that maintenance planned for fiscal year 2020 is expected to increase beyond existing Navy and contractor capacity. Maintenance in Sasebo relies on a number of smaller contractors, and these contractors have experienced challenges planning for the unpredictable maintenance workload there, according to officials. Specifically, the amphibious ships based in Sasebo typically deploy as a group of three. As a result, Navy officials said the workload in Sasebo can be uneven. When all three ships return to port, they require maintenance at the same time. The Navy found that contractors have difficulty planning for this uneven workload, among other issues, which can deter contractors from bidding on work. For example, in fiscal year 2015, the Navy found that they were unable to award over 25 percent of work planned for contractors in Sasebo because no contractor bid on the work. The Navy plans to add a fifth amphibious ship in Sasebo in fiscal year 2020, in part to provide a more stable workload there, according to officials. The Navy expects the additional ship will also result in a forecasted increase in overall maintenance workload there. Navy officials also expressed concerns about the continuity of the existing industrial base in Yokosuka to be able to support future Navy needs. According to Navy documentation, in fiscal year 2018, about one-third of ship maintenance in Yokosuka was conducted by contractors, and, according to officials, the Navy relied on one main contractor to conduct much of this work. However, Navy maintenance center officials in Japan stated they have concerns about the continuity of the contractor to support this workload. The Navy has begun efforts to consider conducting maintenance at contractor facilities outside the ships’ homeports of Yokosuka and Sasebo. Specifically, the Navy has begun to consider using contractor facilities located outside the Yokosuka area, as far as 2 hours away from where the ships are currently based. For example, Navy officials told us that they conducted market research and outreach to potential contractors, and have awarded a small contract for a short continuous maintenance period to a new contractor about 30 minutes outside the Yokosuka area. However, maintenance and contracting officials stated these efforts face their own challenges. For example, conducting weeks or months of maintenance on a ship as far as 2 hours outside a ship’s homeport—where crews and families live—could require additional travel, housing, and other costs. Additionally, maintenance and contracting officials in Yokosuka stated that the substantial regulatory, legal, and Navy requirements that private companies must adopt to contract with the U.S. government might serve as disincentives for prospective Japanese contractors, and developing these contractors will take time. Differences in the operating environments in Japan and Spain. According to NAVSEA officials, the decision to apply the approach in Japan was based on its timely performance in Spain, but the ships, missions, and operating environment in Yokosuka and Sasebo differ substantially from the environment in Spain. For example: Greater diversity and number of ships in Japan. Navy officials told us that the four destroyers sent to Rota in 2014 and 2015 were specifically chosen with similar age, configuration, and condition, which made it easier to sustain the maintenance approach, since issues and lessons from one ship could be easily applied to the next. The ships in Japan in fiscal year 2019 consisted of a more diverse set of ships—eight destroyers and three cruisers in Yokosuka, and various classes of amphibious ships in Sasebo. According to officials, these ships are of different configurations and capabilities. Greater workload and staffing challenges in Japan. Navy officials have attributed the persistent maintenance delays experienced in Japan to insufficient U.S. maintenance prior to deployment, insufficient estimation of the maintenance work package, missed planning milestones, staffing challenges, and other causes, that are not currently being experienced in Rota. Less predictability in operational tempo in Japan. According to Navy officials in Rota, the four ships based there have the same mission, regular and predictable patrol schedules, and do not go above Navy deployment limits. Additionally, officials said the patrol schedules allow for some additional maintenance to be conducted when ships are in port, if needed. As a result, Navy officials in Rota said that they are able to meet key maintenance planning milestones such as conducting ship checks and other assessments. In Japan, however, Navy officials and operational commanders described operational tempo as more unpredictable, and that ships can be unavailable due to the operational demands of the varied missions with different timeframes for ships in Seventh Fleet’s area of responsibility. For example, according to Seventh Fleet officials, the cruisers and destroyers in Yokosuka are expected to serve a number of different missions, including conducting patrols around Japan or Guam; providing ship presence in the East China Sea; or escorting the carrier as part of the strike group. Additionally, according to Navy officials, operational tempo in Japan continues to be high, and in 2015 we reported that to meet increasing demands overseas, the Navy has extended deployments and increased operational tempo. Standards for Internal Control in the Federal Government state that it is a key responsibility of management to analyze and respond to identified changes and related risks, and to monitor program effectiveness. These standards also note that changing conditions often result in new risks or changes to existing risks that need to be assessed. Additionally, the April 2011 DOD Product Support Business Case Analysis Guidebook further states that each risk should be reviewed and assessed, and that effective mitigation plans may involve making tradeoffs in capabilities, schedule, and performance. However, NAVSEA officials said the Navy has not assessed the risks posed by these challenges to implementing its new maintenance approach in Japan. Instead, officials based the decision to implement the approach in Japan on the performance of the approach in Rota, Spain. Without a full assessment of the risks these challenges may pose to successful implementation of its new maintenance approach, and without identifying ways to mitigate any risks posed by these challenges, the Navy cannot ensure its overseas homeported ships complete all required maintenance as planned in support of fleet readiness needs. The Navy’s timeline for growing the fleet from 290 total ships (as of September 2019) to 355 ships shows that the largest increase will be in the number of surface ships. Specifically, the Navy plans to increase the number of surface ships in the fleet by a total of 48 ships in the next 15 years, or by 2034. However, the Navy’s long-range plans to grow its fleet do not consider the maintenance these ships will require while based or traveling overseas. The Navy’s Report to Congress on the Annual Long- Range Plan for Maintenance and Modernization of Naval Vessels for Fiscal Year 2020, which is intended to assess the maintenance and modernization requirements for the fleet as it grows, only assesses maintenance provided by private and public shipyards in the United States, not overseas. It does not identify or assess the maintenance requirements needed overseas—including those provided by Navy facilities or the contractor industrial base. Moreover, it does not identify overseas requirements, such as any expected changes in the number of ships based there or growth in the number of ships visiting overseas locations from the United States. For example: As the number of ships in the overall fleet grows, NAVSEA officials said they expect the number of ships based overseas to grow proportionally, and the number of U.S.-based ships conducting operations and exercises overseas to increase, thereby increasing overseas maintenance requirements. However, the expected increase in the fleet and associated maintenance requirements for ships based and visiting overseas were not included in the recent long-range plans. For example, according to officials, the Navy plans to base an additional amphibious ship in Sasebo, Japan, by fiscal year 2020, and the Navy is examining a possible increase to the number of destroyers in Rota, Spain. According to maintenance center officials in Rota, increasing the number of ships based in Rota would require additional planning to meet the Navy’s needs, such as negotiating with the Government of Spain to request additional capacity, such as pier space, for such future requirements. The Navy projects the number and type of ships based in Japan and Bahrain to change in the next few years. Specifically, the Navy plans to decommission the mine countermeasures (MCM) ships currently homeported in Japan and Bahrain by 2023 and replace them with littoral combat ships to perform the mine countermeasures missions. However, maintenance center officials in Bahrain stated that as of March 2019, plans for the overseas maintenance of littoral combat ships remained uncertain, even though officials expect the initial deployments of littoral combat ships to Bahrain to begin as early as 2020. Additionally, the Navy has not developed deployment timelines and overseas maintenance requirements for littoral combat ships in the Middle East and Western Pacific areas of operation, even though the USS Montgomery arrived in Singapore to begin its overseas rotational deployment in July 2019. According to Navy officials, the Navy expects long-term deployments of littoral combat ships to both areas of operation as the MCMs are decommissioned. Ships based in the United States also rely on voyage repair at overseas shipyards while conducting missions or patrols. For example, according to the Navy’s annual report to Congress listing all repairs and maintenance performed on Navy ships, in fiscal year 2018, the maintenance center in Bahrain conducted voyage repairs for the USS Monterey and USS Arleigh Burke, both based in Norfolk, Virginia, and the USS The Sullivans, based in Mayport, Florida. Additionally, voyage repairs were conducted in Japan for visiting Navy ships and submarines based in Washington and Hawaii. Standards for Internal Control in the Federal Government state that it is a key responsibility of management to consider changes within the environment and other factors, and analyze and respond to identified changes and related risks through methods such as strategic planning and other assessments. These standards also note that conditions affecting the organization and its environment continually change, and management can anticipate and plan for significant changes by using a forward-looking process. NAVSEA officials said that when planning for future growth, they have focused on analyzing U.S. industrial base issues and potential mitigations to increase capacity for U.S.-based ship maintenance as demand grows beyond existing dry docks and pier space. Officials said the Navy did not analyze overseas maintenance requirements or projected growth overseas to include in the long-range plan. According to NAVSEA officials, future iterations of long-term maintenance planning are to include analysis of the Navy’s overseas maintenance capacity, which Navy officials said could begin in March 2020. As the Navy continues its long-term maintenance planning, it will be important for the Navy to conduct and include analysis of anticipated overseas maintenance requirements given that substantial growth of surface ships is expected through 2034—including destroyers and amphibious ships, two types of ships currently based overseas. Without analyzing maintenance needs and requirements for ships based overseas, including any projected growth or other force changes, in its long-range plans, the Navy cannot ensure it is sufficiently planning for the total needs—and resulting readiness and health—of the future fleet. The Navy bases and maintains 38 surface ships—such as destroyers, cruisers, and amphibious ships, among others—at homeports outside of the United States. The 2018 National Defense Strategy has prioritized military readiness, which depends in part on ships completing maintenance on time, to ensure that the United States is positioned to respond to events quickly all over the world. Ship maintenance is a complex process involving numerous Navy and private industry stakeholders that devote substantial time and effort to ensure that ships receive the maintenance they need. Yet we have previously reported on the persistent delays and other challenges the Navy faces in completing maintenance on time both for ships in the United States and overseas. While a number of entities in the Navy have different efforts underway to examine individual ship maintenance issues, a comprehensive, systematic understanding of the underlying and interrelated causes for these delays is essential to implementing corrective actions to ensure these strategically based ships are able and ready for operations when needed. The Navy has also taken steps to adjust its maintenance strategies to improve ship maintenance while balancing the high operational demands for ships based in Japan. Additionally, the Navy has begun planning to grow the fleet, but the expected increase in the fleet and associated maintenance requirements for ships based and visiting overseas were not included in the recent long-range plans. Also, the Navy’s plans to implement updated maintenance strategies overseas, as well as to grow the total fleet, were developed without accounting for risks that challenges may pose to these strategies, as well as analysis of the necessary overseas maintenance requirements to sustain the Navy’s strategically important ships homeported or visiting overseas locations. Ensuring the Navy’s maintenance plans and capacity for the total fleet align with its plans for substantial future fleet growth will enhance the Navy’s ability to conduct timely maintenance of its overseas surface fleet, which, in turn, is essential to the Navy achieving its strategic goals. We are making a total of five recommendations to DOD. The Secretary of the Navy should assign responsibility to an entity to conduct a single, comprehensive systematic analysis of overseas surface ship maintenance delays. (Recommendation 1) The Secretary of the Navy should ensure the designated entity conducts a comprehensive, systematic analysis to identify the underlying, interrelated causes of overseas surface ship maintenance delays. (Recommendation 2) The Secretary of the Navy should use the results of the analysis to develop a plan to address surface ship maintenance delays overseas. Such a plan should incorporate results-oriented elements, including analytically based goals, identification of risks to achieving those goals, identification of required resources and stakeholders, metrics to measure progress, and regular reporting on progress. (Recommendation 3) The Secretary of the Navy should ensure that Naval Sea Systems Command assesses and mitigates risks posed by any challenges, such as persistent delays and capacity limitations, to successful implementation of its new maintenance approach in Japan. (Recommendation 4) The Secretary of the Navy should ensure that Naval Sea Systems Command conducts analysis to include overseas maintenance requirements as part of its long-term maintenance plan to support the planned growth and readiness of the fleet. (Recommendation 5) We provided a draft of this report to DOD for review and comment. In written comments provided by the Navy (reproduced in appendix VII), DOD concurred with our recommendations. The Navy also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of the Navy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have questions about this report, please contact me at maurerd@gao.gov or (202) 512-9627. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. This report (1) describes existing maintenance capacity and approaches the Navy uses for surface ships based overseas, (2) assesses the extent to which the Navy completed maintenance periods as scheduled in fiscal years 2014 through 2018 and analyzes factors contributing to any delays, and (3) evaluates the extent to which the Navy has assessed any challenges facing future overseas ship maintenance efforts. The scope of this review includes the regularly scheduled depot-level maintenance of surface ships based overseas, the maintenance of which is generally the responsibility of Naval Sea Systems Command (NAVSEA). These ships comprised 38 of the 40 ships based overseas during the time period we analyzed, and consisted of the following ship classes: guided-missile cruisers (CG 47 class), guided-missile destroyers (DDG 51 class), mine countermeasures ships (MCM 1 class), patrol coastal ships (PC 1 class), amphibious assault ships (LHD 1 class), amphibious transport dock ships (LPD 17 class), dock landing ships (LSD 41 class), and an amphibious command ship (LCC 19 class). These ships were based overseas at homeports located in Japan, Spain, and Bahrain as of the end of fiscal year 2018. For objective one, to describe existing capacity and maintenance approaches the Navy uses for the regularly scheduled depot-level maintenance periods for the 38 surface ships based overseas during the time of our review, we reviewed Navy documents and information on the Navy’s overseas maintenance centers’ physical capacity and authorized workforce, local contractor industrial base and capacity, and other Navy organizations and commands responsible for planning, managing, and overseeing the maintenance of these ships. To examine physical capacity, we analyzed Navy information on U.S. and contractor facilities and equipment such as dry docks and information on future planning or improvements. We reviewed NAVSEA information and data on ship maintenance periods, as well as information and documentation on historic and forecasted workloads at each homeport, including the number and type of ships that have received maintenance at those shipyards. We also reviewed Navy maintenance plans and guidance that document Navy maintenance approaches and organizations at overseas homeports. We conducted site visits to three overseas homeports— Yokosuka and Sasebo, Japan, and Manama, Bahrain—where the Navy bases a majority of the surface ships overseas. We observed the physical capacity and operations of the maintenance centers and shipyards there, as well as the Forward Deployed Regional Maintenance Center (FDRMC) headquarters in Naples, Italy. We interviewed cognizant officials at Navy commands, numbered fleets, and maintenance centers, including officials at all the overseas maintenance centers responsible for ships based overseas: the U.S. Naval Ship Repair Facility and Japan Regional Maintenance Center (SRF-JRMC) in Yokosuka, Japan, and its detachment in Sasebo, and the FDRMC headquarters in Naples, Italy, as well as its two detachments—in Rota, Spain, and Manama, Bahrain. For objective two, to assess the extent to which maintenance schedules are completed as planned, we analyzed Navy data on regularly scheduled, depot-level maintenance periods for surface ships—including those maintained at overseas homeports and in the United States. NAVSEA collects and manages data on these maintenance periods— known as Chief of Naval Operations maintenance availabilities—for surface ships, submarines, and aircraft carriers. We obtained the data on surface ship depot-level maintenance periods used by NAVSEA’s Surface Maintenance Engineering Planning Program and the Commander, Navy Regional Maintenance Center (SURFMEPP). We used Navy data to identify depot-level maintenance periods conducted at each homeport starting in fiscal years 2014 through 2018 and to assess the extent to which maintenance schedules for ships based overseas were executed as planned from fiscal year 2014 through 2018, and the delays experienced. To assess the reliability of this data, we interviewed cognizant NAVSEA officials to understand system operating procedures, organizational roles and responsibilities, and any data limitations. NAVSEA provided information based on our questions regarding data reliability, including an overview of the data, data-collection processes and procedures, data quality controls, and overall perceptions of data quality. NAVSEA also provided documentation of how the systems are structured and what written procedures are in place to help ensure that the appropriate information is collected and properly categorized. We interviewed officials from SURFMEPP and CNRMC to obtain further clarification on data reliability, discuss how the data were collected and reported, and explain how we planned to use the data. Some of these data were used in prior reports, and their reliability had previously been assessed. In addition, we also assessed the reliability of the data by checking: (1) for missing data entries, (2) for duplicate records, and (3) to ensure the data was formatted consistently. We determined that the data were sufficiently reliable for the purposes of summarizing surface ship maintenance periods and related information at homeports both overseas and in the United States, including reporting on the duration of maintenance periods and the number of days of maintenance delays. Because maintenance periods may cross over one or more fiscal years, to be able to report on days ships spent in maintenance periods from fiscal years 2014 through 2018, we analyzed data on maintenance periods that began in fiscal years 2012 through 2018 for all surface ships included in the data, including those based at overseas and U.S. homeports. Specifically, we used the dates of the planned and actual durations of the maintenance periods in our data set to determine the total number of days ships spent in maintenance in fiscal years 2014 through 2018 and by how many days the maintenance periods were extended beyond their planned number of days—which the Navy refers to “days of maintenance delay.” To determine the total number of days ships spent in maintenance in each fiscal year, we allocated the number of days spent in maintenance periods according to the fiscal year in which the maintenance days occurred. After we calculated the number of days each maintenance period went beyond the planned duration, we allocated these days of maintenance delay to the fiscal years in which they occurred. To compare ship maintenance delays experienced at different homeports while accounting for the varying workload at each, we calculated days of maintenance delay as a percentage of the total number of days ships spent in maintenance periods each location, which resulted in a rate of delay that we could compare across homeports. In addition, we analyzed the number of maintenance periods that were completed on or ahead of time or were completed later than planned, and we examined these maintenance durations by the fiscal year in which the maintenance periods started. We interviewed officials to understand the reasons they identified for delays. We reviewed the actions the Navy has taken to identify, evaluate, and resolve these delays, including information in Navy policies, guidance, and documentation on the planning, management, and oversight of overseas ship maintenance. This information included the Joint Fleet Maintenance Manual and related Navy instructions, documents establishing maintenance requirements. We also reviewed Navy guidance and documentation on the planning and execution of maintenance for ships based overseas and in the United States, including documentation such as status briefings, planning documents, and lessons learned information identifying certain reasons for maintenance delays of individual maintenance periods. We interviewed cognizant Navy officials responsible for planning, managing, and conducting oversight for surface ship maintenance in the United States and overseas to understand how they produce and use this information to improve maintenance planning and execution. We compared this information to standards for planning, scheduling, and monitoring events to correct deficiencies and identify process improvements, including Standards for Internal Control in the Federal Government, which includes principles pertaining to oversight responsibility, evaluating issues, and remediating deficiencies; our Schedule Assessment Guide; and OMB Circular No. A-123, Management’s Responsibility for Enterprise Risk Management and Internal Control, which includes guidance on conducting a root-cause analysis when developing corrective actions. We also compared this information with our past work identifying best practices for results- oriented performance management and planning. For objective three, to assess the extent to which the Navy has assessed and mitigated challenges that may affect overseas ship maintenance efforts, including new maintenance approaches and future maintenance requirements as the Navy seeks to grow the fleet, we analyzed Navy documentation, NAVSEA data, and available information documenting challenges that affect maintenance overseas, as well in the United States. We also analyzed Navy efforts to address these challenges, as well as Navy plans for future fleet growth and maintenance workload, including the long-range plans for shipbuilding and maintenance as the Navy seeks to grow its fleet, as well as other studies and analyses pertaining to these plans. We interviewed cognizant Navy officials who plan, execute, and oversee overseas shipyards and maintenance, as well as operational commanders, to obtain their perspectives on issues and challenges associated with execution of ship maintenance. We compared this information to government standards for planning and monitoring events to assess changes in risk, correct deficiencies, and identify process improvements, including Standards for Internal Control in the Federal Government and DOD Product Support Business Case Analysis Guidebook. Logistics, Maintenance, and Industrial Operations (NAVSEA 04) Deputy Commander for Ship Maintenance and Modernization (NAVSEA 21) Commander, Navy Regional Maintenance Center (CNRMC) Surface Maintenance Engineering Planning Program (SURFMEPP) U.S. Naval Ship Repair Facility and Japan Regional Maintenance Center (Yokosuka, Japan, and detachment in Sasebo, Japan) Forward Deployed Regional Maintenance Center in Naples, Italy, and its detachments in Rota, Spain, and Manama, Bahrain Naval Supply Systems Command Fleet Logistics Centers in Yokosuka, Japan, and Manama, Bahrain U.S. Naval Forces Central Command U.S. Naval Forces Europe-Africa Human Resources Office for Commander Navy Region Europe, Africa, Southwest Asia (CNREURAFSWA) We conducted this performance audit from August 2018 to February 2020 in accordance with generally accepted government auditing standards. Those standards require we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. History and Mission The Ship Repair Facility and Japan Regional Maintenance Center (SRF-JRMC) was originally created in 1947 as the Ship Repair Department, and in 1951 became the Ship Repair Facility. In 2004 it became the combined SRF-JRMC. Headquartered in Yokosuka, SRF- JRMC provides oversight and support to its detachment in Sasebo and is responsible for the maintenance for ships based in Yokosuka. SRF-JRMC also provides technical assistance and voyage repairs for Navy ships visiting Japan. U.S. Navy Surface Ship Maintenance Snapshot Surface Ships Maintained in Yokosuka as of Fiscal Year 2018 8 Guided-missile destroyers (DDG) 3 Guided-missile cruisers (CG) 1 Amphibious command ship (LCC) Fiscal Year 2018 Authorized SRF-JRMC Workforce U.S. Civilians: 272 U.S. Navy: 108 Japanese nationals: 2,341 (paid for by the Government of Japan) In 2018, the Navy finalized a new maintenance approach for cruisers and destroyers based in Yokosuka. This approach relies on deep maintenance in the United States prior to ships moving to Japan, and increases the frequency of maintenance periods while ships are in Japan. The Navy has begun efforts to identify additional private companies to conduct ship maintenance to meet future planned workload, according to Navy officials. History and Mission The Ship Repair Facility and Japan Regional Maintenance Center (SRF-JRMC) detachment in Sasebo was originally designated as the Sasebo Office to the Ship Repair Facility in Yokosuka in 1976, and made a detachment in 1984. The SRF-JRMC detachment is responsible for supporting the maintenance for the eight surface ships based in Sasebo, and can provide technical assistance and other maintenance to ships in and visiting Japan. U.S. Navy Surface Ship Maintenance Snapshot Surface Ships Maintained in Sasebo as of Fiscal Year 2018 1 Amphibious Assault Ship (LHD) 1 Amphibious Transport Dock (LPD) 2 Dock Landing Ships (LSD) 4 Mine Countermeasures Ships (MCM) Fiscal Year 2018 Authorized SRF-JRMC Workforce U.S. Civilians: 65 U.S. Navy: 40 Japanese nationals: 450 (paid for by the Government of Japan) U.S. Navy Surface Ship Maintenance Snapshot Surface Ships Maintained in Rota as of Fiscal Year 2018 4 Guided-missile destroyers (DDG) Fiscal Year 2018 Authorized FDRMC Workforce U.S. Civilians: 73 U.S. Navy: 8 Future Considerations The Navy plans to rotate the four current ships back to the United States beginning in 2020 through 2022. FDRMC officials said the next set of ships will not be as standardized as the first four. Additionally, the Senate Armed Services Committee has directed the Navy to assess the feasibility of increasing the number of guided-missile destroyers based in Rota from four to six. FDRMC officials said increasing the number of ships would require additional staff and physical infrastructure that would need to be negotiated with the Spanish government. History and Mission The Forward Deployed Regional Maintenance Center detachment in Bahrain (FDRMC Detachment Bahrain) was established in June 2014. FDRMC Detachment Bahrain manages the maintenance of ships based there, and can provide fleet technical assistance and coordinate voyage repairs for other ships in the U.S. Fifth Fleet area of operations including Military Sealift Command ships and visiting U.S. Navy ships. FDRMC Detachment Bahrain manages the maintenance for the most homeported ships of all overseas locations. U.S. Navy Surface Ship Maintenance Snapshot Surface Ships Maintained in Manama as of Fiscal Year 2018 10 Patrol Coastal Ships (PC) 4 Mine Countermeasures Ships (MCM) Fiscal Year 2018 Authorized FDRMC Workforce U.S. Civilians: 87 U.S. Navy: 29 Foreign nationals: 14 Future Considerations Beginning in fiscal year 2020, the Navy will decommission U.S.-based MCMs to provide spare parts to MCMs overseas. The Navy plans to decommission the MCMs and PCs in Bahrain in fiscal years 2023 and 2026, respectively. The Navy plans to replace the MCM mission with littoral combat ships but has not finalized plans for their deployment or maintenance, according to Navy officials. In addition the contact named above, the following staff members made key contributions to this report: Suzanne Wren (Assistant Director), Sally Williamson (Analyst in Charge), David Ballard, Martin De Alteriis, Alexandra Gonzalez, Amie Lesser, Shahrzad Nikoo, Carol Petersen, Clarice Ransom, Rachel Schultz, and Samuel Woo. Navy Maintenance: Persistent and Substantial Ship and Submarine Maintenance Delays Hinder Efforts to Rebuild Readiness. GAO-20-257T. Washington, D.C.: December 4, 2019. Naval Shipyards: Key Actions Remain to Improve Infrastructure to Better Support Navy Operations. GAO-20-64. Washington, D.C.: November 25, 2019. Navy Readiness: Actions Needed to Evaluate the Effectiveness of Changes to Surface Warfare Officer Training. GAO-20-154. Washington, D.C.: November 14, 2019. Military Depots: Actions Needed to Improve Poor Conditions of Facilities and Equipment that Affect Maintenance Timeliness and Efficiency. GAO- 19-242. Washington, D.C.: April 29, 2019. Military Personnel: Strategy Needed to Improve Retention for Experienced Air Force Aircraft Maintainers. GAO-19-160. Washington, D.C.: February 5, 2019. DOD Depot Workforce: Services Need to Assess the Effectiveness of Their Initiatives to Maintain Critical Skills. GAO-19-51. Washington, D.C.: December 14, 2018. Navy and Marine Corps: Rebuilding Ship, Submarine, and Aviation Readiness Will Require Time and Sustained Management Attention. GAO-19-225T. Washington, D.C.: December 12, 2018. Navy Readiness: Actions Needed to Address Costly Maintenance Delays Facing the Attack Submarine Fleet. GAO-19-229. Washington, D.C.: November 19, 2018. Military Readiness: Analysis of Maintenance Delays Needed to Improve Availability of Patriot Equipment for Training. GAO-18-447. Washington, D.C.: June 20, 2018. Navy Shipbuilding: Past Performance Provides Valuable Lessons for Future Investments. GAO-18-238SP. Washington, D.C.: June 6, 2018. Weapon Systems Annual Assessment: Knowledge Gaps Pose Risks to Sustaining Recent Positive Trends. GAO-18-360SP. Washington, D.C.: April 25, 2018. Military Readiness: Clear Policy and Reliable Data Would Help DOD Better Manage Service Members’ Time Away from Home. GAO-18-253. Washington, D.C.: April 25, 2018. Navy Readiness: Actions Needed to Address Persistent Maintenance, Training, and Other Challenges Affecting the Fleet. GAO-17-809T. Washington, D.C.: September 19, 2017. Naval Shipyards: Actions Needed to Improve Poor Conditions that Affect Operations. GAO-17-548. Washington, D.C.: September 12, 2017. Navy Readiness: Actions Needed to Address Persistent Maintenance, Training, and Other Challenges Facing the Fleet. GAO-17-798T. Washington, D.C.: September 7, 2017. Navy Shipbuilding: Policy Changes Needed to Improve the Post-Delivery Process and Ship Quality. GAO-17-418. Washington, D.C.: July 13, 2017. Department of Defense: Actions Needed to Address Five Key Mission Challenges. GAO-17-369. Washington, D.C.: June 13, 2017. Navy Force Structure: Actions Needed to Ensure Proper Size and Composition of Ship Crews. GAO-17-413. Washington, D.C.: May 18, 2017. Navy Ship Maintenance: Action Needed to Maximize New Contracting Strategy’s Potential Benefits. GAO-17-54. Washington, D.C.: November 21, 2016. Military Readiness: Progress and Challenges in Implementing the Navy’s Optimized Fleet Response Plan. GAO-16-466R. Washington, D.C.: May 2, 2016. Defensed Civilian Compensation: DOD and OPM Could Improve the Consistency of DOD’s Eligibility Determinations for Living Quarters Allowances. GAO-15-511. Washington, D.C.: June 16, 2015. Navy Force Structure: Sustainable Plan and Comprehensive Assessment Needed to Mitigate Long-Term Risks to Ships Assigned to Overseas Homeports. GAO-15-329. Washington, D.C.: May 29, 2015.", "summary": "To meet operational demands, the Navy has doubled the number of ships based overseas since 2006. Navy ships based abroad represent about 14 percent of the total fleet and are there to provide presence, deter threats, quickly respond to crises, and build partnerships. Effective and timely maintenance is essential to meet strategic objectives, fulfill operational requirements, and ensure ships reach their expected service lives. House Report 115-676 included a provision that GAO assess maintenance for ships based overseas. This report: (1) describes existing maintenance capacity and approaches the Navy uses for surface ships based overseas, (2) assesses the extent to which the Navy completed maintenance periods as scheduled in fiscal years 2014 through 2018 and analyzes factors contributing to any delays, and (3) evaluates the extent to which the Navy has assessed any challenges facing future overseas maintenance efforts. To address these objectives, GAO analyzed Navy policies and maintenance data from fiscal years 2012 through 2018, and interviewed officials, including from Naval Sea Systems Command and overseas fleets and maintenance centers. The Navy maintains the 38 surface ships based in Japan, Spain, and Bahrain through a mix of Navy-operated facilities and private contractors. The Navy uses different maintenance approaches at each location depending on the number and type of ships based there and the Navy and private contractor industrial base available to provide maintenance support. For example, to support the 12 surface ships based in Yokosuka, Japan, the Navy uses both private contractors and its Ship Repair Facility and Japan Regional Maintenance Center, which is subsidized by the government of Japan. In Rota, Spain, the Navy relies on one Spanish contractor to maintain the four ships based at that location. Maintenance on surface ships based overseas took longer than planned for 50 of the 71 maintenance periods—or about 70 percent—started during fiscal years 2014 through 2018. More than half of these maintenance delays lasted a month or longer, which reduced the ships' availability for training and operations. Various factors contribute to delays, such as discovery that unanticipated additional repairs are needed, missed planning milestones, or shortages of key staff. However, the Navy's efforts to understand delays often solely focus on individual maintenance periods and result in steps to improve specific issues related to maintenance timeliness. The Navy has not conducted a comprehensive analysis of maintenance delays to systematically identify and address their root causes. Without such an analysis, the Navy cannot effectively target corrective actions, and risks continuing to underestimate maintenance needs and the time and resources required to address them. The Navy has developed a new maintenance approach for ships in Japan, but has not assessed the risks associated with this approach or analyzed the overseas maintenance requirements for a growing fleet. The new maintenance approach calls for ships to obtain all required maintenance in the United States before and after going overseas, among other things. The Navy decided to implement this approach in Japan based on use of the approach in Spain—where ships have experienced few maintenance delays. However, the Navy has not assessed the risks posed by differences between the operating environments in Spain and Japan, or by shortfalls in maintenance capacity at U.S. facilities. The Navy also plans to replace aging ships in Bahrain as it grows the fleet to 355 ships, but it did not analyze or include overseas maintenance requirements in its long-range plan. Without assessing the risks challenges may pose to the success of its new maintenance approach in Japan or analyzing the requirements of a growing fleet, the Navy could be hindered in its ability to ensure these ships are ready and available for operations. GAO is making five recommendations, including that the Navy comprehensively analyze and address maintenance delays, and assess the risks and analyze requirements of future overseas maintenance efforts. The Navy concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "The Direct Loan program provides financial assistance to students and their parents to help pay for postsecondary education. Under the Direct Loan program, Education issues several types of student loans (see sidebar). Current William D. Ford Federal Direct Loan Types Subsidized Stafford Loans: Available to undergraduate students with financial need (generally the difference between their cost of attendance and a measure of their ability to pay, known as expected family contribution). Borrowers are not responsible for paying interest on these loans while in school and during certain periods of deferment, an option that allows eligible borrowers to temporarily postpone loan payments. Unsubsidized Stafford Loans: Available to undergraduate and graduate school students irrespective of financial need. Borrowers must pay all interest on these loans. PLUS Loans: Available to graduate student borrowers and parents of dependent undergraduates. Borrowers must pay all interest on these loans. Consolidation Loans: Available to eligible borrowers wanting to combine multiple federal student loans (including those listed above) into one loan. Repayment periods are extended up to a maximum of 30 years, thereby lowering monthly payments. Interest rates for these loans are tied to the Department of the Treasury’s 10-year note rate and can vary by loan type. In addition, there are limits on the annual and aggregate amounts that can be borrowed for certain loan types. After a prospective borrower applies for and is awarded a Direct Loan, Education disburses it through the borrower’s school. Once the loan is disbursed, it is assigned to one of nine loan servicers under contract with Education. These loan servicers are responsible for such activities as communicating with borrowers about the status of their loans, providing information about and enrolling borrowers in repayment plans, and processing payments. Once borrowers leave school, they are responsible for making payments directly to their assigned loan servicer. A variety of repayment plans are available to eligible Direct Loan borrowers, including Standard, Graduated, Extended, and several IDR plans. Borrowers are automatically enrolled in the Standard plan if they do not choose another option, and generally make fixed monthly payments over a period of 10 years. IDR plans can ease repayment burden by setting monthly loan payments based on a borrower’s income and family size and extending the repayment period up to 20 or 25 years, depending on the plan. Unlike Standard, Graduated, and Extended repayment plans, IDR plans offer loan forgiveness at the end of the repayment period and monthly payments may be as low as $0 for some borrowers. There are a variety of IDR plans, and these plans have differences in eligibility requirements, how monthly payment amounts are calculated, and repayment periods before potential loan forgiveness (see table 1). To participate in an IDR plan, borrowers must submit an application to their loan servicer that, among other things, includes information about their income, marital status, and family size (see table 2). According to Education, Education’s loan servicers review the information borrowers submit on their IDR applications to determine if borrowers are eligible for IDR plans. If the servicer determines that a borrower is eligible, it enrolls the borrower in an IDR plan and calculates the borrower’s monthly payment amount. To continue making monthly payment amounts based on income and family size, IDR borrowers must annually submit the IDR application form certifying their income and family size, which servicers then use to update monthly payment amounts. If a borrower’s income changes significantly prior to the borrower’s annual recertification date, the borrower can use the same application form to request a recalculation of the monthly payment amount (see fig. 1). However, borrowers are not required to report any such changes before their annual recertification date. If IDR borrowers do not have any discretionary income, their scheduled monthly payment amount is zero dollars (meaning they will not have to make a monthly loan payment until their discretionary income is high enough to warrant one). Scheduled monthly payments of zero dollars count as qualifying payments towards eventual loan forgiveness at the end of the 20- to 25-year repayment period. Borrowers who make monthly payments on IDR plans that are much lower than they would be under the Standard 10-year repayment plan for a long period of time may end up paying less than their original loan amount because their remaining loan balances may be forgiven. However, some borrowers on IDR plans will fully repay their loans before qualifying for forgiveness. Extending the repayment period may also result in some borrowers paying more interest over the life of the loan than they would under the 10-year Standard repayment plan. Fraud in federal programs occurs when individuals or entities intentionally misrepresent themselves in order to benefit from the programs. Fraud poses a significant threat to the integrity of federal programs and erodes public trust in government. Federal programs are at risk for fraud when individuals have both the opportunity and incentive to commit fraud. Although the occurrence of one or more cases of fraud indicates there is a fraud risk, a fraud risk can exist even if fraud has not yet been identified or occurred. Proactive fraud risk management is meant to facilitate a program’s mission and strategic goals by ensuring that taxpayer dollars and government services serve their intended purposes. In July 2015, GAO issued the Fraud Risk Framework, which provides a comprehensive set of components and leading practices that serve as a guide for agency managers to use when developing efforts to combat fraud in a strategic, risk-based way. The Framework recommends that to effectively manage fraud risks, managers should design and implement specific control activities to prevent and detect potential fraud, such as data analytics. After issuance of the Fraud Risk Framework, the Fraud Reduction and Data Analytics Act of 2015 was enacted to improve federal agency controls and procedures to assess and mitigate fraud risks, and to improve agencies’ development and use of data analytics for the purpose of identifying, preventing, and responding to fraud. The act requires agencies to establish financial and administrative controls that incorporate the Fraud Risk Framework’s leading practices. We previously reported that Education identified itself as subject to the act. Error also poses a risk to the integrity of federal programs. According to federal internal control standards, to maintain an effective internal control system, managers should use quality information to achieve agency objectives. This includes obtaining information from reliable sources that is reasonably free from errors and communicating it externally to achieve objectives. Our analysis of Education’s IDR plan data and HHS’s NDNH wage data for borrowers who reported zero income found that about 95,100 approved IDR plans (11 percent of all IDR plans we analyzed) were held by borrowers who may have had sufficient wages to warrant a monthly student loan payment. These plans were held by about 76,200 unique borrowers who owed nearly $4 billion in outstanding Direct Loans as of September 2017. According to our analysis, 34 percent of these plans were held by borrowers who had estimated annual wages of $45,000 or more, including some with estimated annual wages of $100,000 or more (see fig. 2). Our results from matching the Education and HHS data indicate the possibility that some borrowers misrepresented or erroneously reported their income, highlighting the risk of potential fraud and errors in IDR plans. Borrowers may have a financial incentive to commit fraud to reduce their monthly payment amount and, by extension, possibly increase the amount of loan debt forgiven at the end of their repayment periods. However, we cannot determine whether fraud occurred through data matching alone. Where appropriate, we are referring these results to Education for further investigation. Among the 76,200 borrowers in our data matching results, it is possible that some accurately reported zero income even though they had wages reported in NDNH in the same quarter in which their IDR application was approved. For example, a borrower may have earned wages at the start or end of a quarter, but was not earning wages at the time of submitting the IDR application. Conversely, our analysis cannot identify borrowers who may have earned additional taxable income that is not part of NDNH data, but should be included on IDR applications, such as income for individuals who are self- employed or receiving alimony. Regarding the potential for error, officials from Education and all four loan servicers we spoke with stated that it is possible that borrowers could incorrectly report that they had no taxable income. Officials from Education said, for example, that borrowers may misunderstand the question about taxable income on the IDR application, and one loan servicer, echoing this perspective, stated that some borrowers may mistakenly think that some of their income is nontaxable when it is in fact taxable. To examine how borrowers’ failure to report their income could affect the amount repaid to Education over the course of a year, we used Education’s online repayment estimator to illustrate how much hypothetical borrowers with different annual adjusted gross incomes would expect to pay under each IDR plan (see fig. 3). If a borrower at one of these income levels instead reported zero income on the IDR application, Education could lose thousands of dollars per borrower each year in student loan payments. Such a situation could also potentially increase the ultimate cost to the federal government and taxpayers for loan forgiveness because scheduled monthly payments of zero dollars count toward the borrower’s 20- or 25-year repayment period. To examine the extent to which Education’s IDR plan data on family size may indicate potential fraud or error, we analyzed the family sizes for about 5 million IDR plans approved between January 1, 2016 and September 30, 2017. Of these plans, over 2.1 million (43 percent) were approved with a family size of one, meaning only the borrower was included (see fig. 4). In addition, over 2.6 million plans (52 percent) were approved with family sizes of two to five. At the high end of the spectrum, about 40,900 of the plans we analyzed (about 1 percent) were approved with family sizes of nine or more (see fig. 5). We consider IDR plans with family sizes of nine or more atypical or outliers because they comprise the top 1 percent of all family sizes in Education’s data. Of these plans, almost 1,200 had family sizes of 16 or more, including two plans held by different borrowers that were approved with a family size of 93. In total, the 40,900 plans approved with family sizes of nine or more corresponded to about 35,200 unique borrowers who owed almost $2.1 billion in outstanding Direct Loan debt as of September 2017. While IDR plans with family sizes of nine or more were atypical in our data and could indicate fraud or error, IDR plans with smaller or more typical family sizes could also pose problems. Borrowers may have a financial incentive to commit fraud because larger family sizes reported on the IDR application can reduce borrowers’ discretionary income and, by extension, their monthly payment amounts. Regarding the potential for error, officials from Education and all four loan servicers we spoke with said borrowers or loan servicers may inadvertently make mistakes related to family size. For example, officials from Education and one loan servicer said borrowers sometimes report inaccurate family sizes if they are confused about who to count as a member of their family. Officials from this loan servicer told us that a borrower initially applied for an IDR plan claiming a family size of five— himself and four other family members who were not his spouse or children. They said that during a subsequent phone call with loan servicer staff about the borrower’s loan, the borrower volunteered that the other members of his family did not live with him, meaning that for IDR purposes, he had a family size of one. It is unclear whether this borrower may have misrepresented his family size to receive a lower monthly payment or did not understand the definition and reported it in error. In regards to loan servicer error, Education officials said that servicers may make mistakes when entering family sizes from paper applications into their computer systems or when determining the total family size because borrowers provide information on family members in up to three places on the application. To examine the effect of family size on monthly payment amounts in IDR plans, we used Education’s online repayment estimator to illustrate how much hypothetical borrowers with the same income but different family sizes would be expected to pay each month under certain IDR plans. We found that a hypothetical borrower with a family size of one and an adjusted gross income of $40,000 who enrolls in one of three IDR plans that base monthly payment amounts on 10 percent of discretionary income would have a monthly payment amount of $182 (see fig. 6). If this borrower instead reported a family size of two people, the monthly payment amount would decrease by $54, to $128. For each additional person, the monthly payment would decrease by $54. At a family size of five people, the borrower would have no monthly payment. Education does not have procedures to verify borrower reports of zero income nor, for the most part, procedures to verify borrower reports of family size; although there are approaches it could use to do so. Because income and family size are the basis for calculating borrowers’ monthly payment amounts for IDR plans, it is important that this information is accurate on IDR applications. While Education instructs loan servicers to review tax returns and other documentation of taxable income that borrowers are required to provide, as previously discussed, borrowers are not required to provide documentation to support self-attestations of zero income or their family size on IDR applications. Officials from Education and all four loan servicers we spoke with said that servicers are generally instructed to take these self-attestations at face value. However, Education has limited, voluntary procedures for reviewing family size information submitted by borrowers. In 2016, Education implemented a voluntary procedure for loan servicers to contact borrowers who report changes in family size of four or more from one year to the next in order to verify the accuracy of the most recently reported family size. Education officials told us that servicers are not contractually required to follow this procedure. In addition, this procedure is not applicable to student loan borrowers when they initially apply for IDR plans. In October 2018, Education officials told us they began to follow up with loan servicers about family sizes of 20 or more in IDR program data to ensure these data match the family size information in the loan servicer systems from which they originated. Officials said that this process is to ensure that family size data were accurately transferred from servicers to Education. Borrowers are not contacted for verification of the information itself. Officials from Education and three of the four loan servicers we spoke with acknowledged that IDR plans are at risk for fraud or error because verification is generally not performed on borrower reports of zero income and borrower reports of family size. Officials from Education and two of the loan servicers also said that certain program requirements discourage borrowers from providing false information. For example, borrowers are required to sign the IDR form to certify that all provided information is true, complete, and correct, and the form warns borrowers that false statements or misrepresentations are subject to penalties including fines, imprisonment, or both. However, the extent to which this requirement may serve as a deterrent is unknown because Education has not assessed the risk of fraudulent reports on IDR applications. Moreover, Education officials told us that they were not aware of any IDR borrowers being investigated or facing penalties for providing false information on the IDR application. Officials from one loan servicer also said that borrowers may be deterred from falsely claiming zero income or misrepresenting their family size because they assume that servicers, acting on behalf of the government, can check the information on IDR applications. However, it is also possible that borrowers would assume that this self-reported information would not be routinely verified because the only documentation requirements discussed on the application relate to verifying taxable income. Education officials also said that the risk of borrowers providing inaccurate information on IDR applications must be balanced against the impact of adding verification procedures. They said additional procedures could make the already complex IDR application process more burdensome for borrowers to navigate and result in longer application processing times. While it is important to make IDR plans accessible to borrowers who could benefit from them, it is also important that Education design internal control activities to achieve program objectives and respond to risks, including addressing the risk of fraud and error in borrower self-reported information. GAO’s Fraud Risk Framework describes the importance of developing procedures for preventing, detecting, and responding to the risk of fraud in government programs. The risk of fraud exists when there is opportunity and incentive to commit it. The lack of verification of borrower reports of zero income and limited verification of borrower reports of family size on IDR applications creates the opportunity for borrowers to commit fraud. Because lower income and larger family sizes can reduce borrowers’ monthly payment amounts and, by extension, possibly increase the amount of loan debt forgiven at the end of their repayment periods, there is also an incentive for some borrowers to commit fraud. In regard to error, federal internal control standards state that agencies should obtain information from reliable sources that are reasonably free from error. Education officials and all four loan servicers told us that borrower-reports of family size or zero income can be susceptible to error if, for example, borrowers misunderstand the definitions of these items on IDR applications. Addressing the risk of fraud and error would also help to minimize the costs associated with IDR plans that are passed on to the government and taxpayers. As more borrowers enter IDR plans, the costs of these plans—including loan forgiveness—increase for the government and taxpayers. Using data underlying the President’s fiscal year 2017 budget request, GAO previously reported that Education estimated Direct Loans repaid with IDR plans would cost the federal government about $74 billon over their repayment periods. In its fiscal year 2015-2019 strategic plan for Federal Student Aid, Education acknowledged that as IDR plans continue to grow in popularity, the cost of loan forgiveness could be a major issue for the federal government. Education can minimize the costs associated with IDR plans by ensuring payment amounts are based on accurate income and family size information. Education has not fully leveraged available approaches to help detect and prevent fraud or error in IDR plans. Federal internal control standards call for agency management officials to identify, analyze, and respond to risks related to achieving program objectives, such as the risk of using potentially fraudulent or erroneous information about borrowers to calculate monthly payment amounts for student loans. Approaches, such as using data analytic practices and follow-up procedures, can help identify and address these risks. Two data analytic practices that can help identify such risks with respect to IDR plans are (1) anomaly detection to identify atypical or unusual information about borrowers and (2) data matching with outside data sources to verify information that borrowers provide. These practices, which can be used on their own or together, can help prevent fraud from occurring and detect potential fraud or error that may have occurred. Because data analytics alone may not be sufficient to determine whether fraud or error has occurred, follow-up procedures can then be used in the investigation and verification to make such determinations. A leading practice in data analytics in GAO’s Fraud Risk Framework is conducting data mining to identify suspicious activity or transactions, including anomalies, outliers, and other red flags in the data. Similar to our family size analysis, borrower-reported family sizes above a certain threshold on IDR applications could be flagged in loan servicers’ and Education’s data systems for further verification. Anomaly detection is used to a limited extent to identify errors in family size on IDR plans by one loan servicer and by Education. According to officials at Education and all four loan servicers we spoke with, anomaly detection is not used to systematically identify potentially fraudulent reports of family size. Anomaly detection can also identify deviations from expected patterns in data over time. Because IDR borrowers are required to fill out applications annually, it would be possible to develop automated queries to look for unusual patterns in borrower-reported income and family size from one year to the next. Officials from Education and servicers described several patterns across applications that could indicate potential fraud, specifically large swings in income from one year to the next, reporting zero income for multiple years, and having a large family size, but relatively low income. Another leading practice for data analytics in GAO’s Fraud Risk Framework is conducting data matching to verify key information, including self-reported data and information necessary to determine eligibility. The results of our analysis illustrate the usefulness of this technique to identify potential inconsistencies in the income information on IDR plans. Education does not have authority to access wage data from HHS’s NDNH or income data from the Internal Revenue Service (IRS) for the purpose of verifying IDR borrowers’ income information through data matching. However, private data sources are also available for data matching. We reported in 2016 on the benefits of government agencies using private data to address the risk of fraud. Moreover, some state agencies (such as those administering the Supplemental Nutrition Assistance Programs) use a private, commercial verification service known as The Work Number® to help determine eligibility for government assistance. We reported in 2016 that 45 states used income information from The Work Number to help determine eligibility for food assistance benefits under the Supplemental Nutrition Assistance Program. Education may also be able to draw on follow-up procedures it has in place for verifying information submitted by students and their families when applying for federal student aid using the Free Application for Federal Student Aid (FAFSA). Education uses a process called “verification” to help identify and correct erroneous or missing information on the application to aid the department’s efforts to reduce improper payments of federal student aid. Each award year, a portion of FAFSA applications are selected for verification, and schools are required to work with the selected applicants to obtain documentation and confirm the accuracy of information provided on these applications. When selecting FAFSAs for verification, Education aims to select those applications with the highest statistical probability of error and the greatest impact of such error on award amounts. FAFSA applicants who are selected to verify their income for the 2018-2019 or 2019-2020 award years may provide a signed copy of their prior years’ tax returns. FAFSA applicants may also obtain documentation from the IRS through the IRS Data Retrieval Tool, an IRS tax return transcript, or an IRS Verification of Non-filing Letter. FAFSA applicants selected to verify their household size must provide a signed statement that provides the name, age, and relationship to the student of each person in the household. For IDR plans, Education could implement follow-up procedures for IDR applications it identifies as at risk for fraud or error and seek additional documentation from borrowers. For example, to verify reports of no income, borrowers could be asked to provide an IRS Verification of Non- filing Letter, documentation that the borrower recently lost a job, or documentation that shows income the borrower receives is nontaxable, such as public assistance benefits. To verify family size, as is the case with FAFSA verification, borrowers could be asked to provide a signed statement with the names, ages, and relationship to the borrower of each family member. Another option might be to request that borrowers provide documentation showing that family members (other than the borrower’s spouse and children) receive mail at the borrower’s address as well as documentation of the financial support provided by the borrower. Such follow-up procedures would be consistent with federal internal control standards advising managers to design control activities to achieve program objectives and respond to risks. While Income-Driven Repayment plans can help borrowers with limited incomes afford their monthly student loan payments, these plans can also result in high costs to the federal government and taxpayers. To minimize these costs, it is important that Education accurately determine monthly payment amounts under its IDR plans. Because these determinations are based on income and family size information that borrowers self-report, there is risk for potential fraud or error. Our data matching analysis showed, for example, that tens of thousands of borrowers who were not making monthly loan payments because they reported zero income on IDR applications may have had enough income to do so. Where appropriate, we are referring these borrowers to Education for further investigation. In addition, an increase in family size can cause a borrower’s payments to decrease, creating a potential incentive for fraud, and our analysis found atypically large family sizes that are generally not verified by Education. The results of our analyses highlight the risk for fraud or error, as well as weaknesses in Education’s procedures. In turn, the weaknesses we identified raise questions about the strength of Education’s institutional oversight of a major program involving hundreds of billions of dollars. The fact that, cumulatively, the borrowers and their plans we reviewed owed over $6 billion in loans helps illustrate the risk of potential financial loss for the government from fraud or error absent comprehensive oversight. It is important for Education to take steps to obtain data to verify borrower reports of zero income and to implement other data analytic practices and follow-up procedures for verifying borrower-reported information. Such actions would help ensure that (1) IDR payment amounts are based on information that accurately represents a borrower’s situation and is free from fraud and error; and (2) the federal government’s fiscal exposure to IDR loans is safeguarded from the risk of loss. Implementing data analytic practices and follow-up procedures to review and verify borrower reports of zero income could help deter borrowers from inaccurately reporting zero income and detect those who have done so, either fraudulently or in error. Similarly, implementing practices and procedures to review and verify reported family sizes could further stem potential fraud or error. Without such changes, IDR plans will remain vulnerable to fraud and error, potentially raising program costs for the federal government and taxpayers. We are making the following three recommendations to Education’s Federal Student Aid office: The Chief Operating Officer of Federal Student Aid should obtain data in order to verify income information for borrowers reporting zero income on IDR applications. For example, Education could pursue access to federal data sources or obtain access to an appropriate private data source. (Recommendation 1) The Chief Operating Officer of Federal Student Aid should implement data analytic practices, such as data matching, and follow-up procedures to review and verify that borrowers reporting zero income on IDR applications do not have sources of taxable income at the time of their application. (Recommendation 2) The Chief Operating Officer of Federal Student Aid should implement data analytic practices, such as data mining, and follow-up procedures to review and verify family size entries in IDR borrower applications. For example, Education could review and verify all borrower reports of family size or a subset identified as being most susceptible to fraud or error. (Recommendation 3) We provided a draft of this report to the Departments of Education (Education) and Health and Human Services (HHS) for review and comment. HHS provided technical comments, which we incorporated as appropriate. We also provided relevant report sections to the Social Security Administration and the four loan servicers included in our review for technical comments. Loan servicers provided technical comments, which we addressed as appropriate. Education generally agreed with our recommendations, stating that it plans to implement significant additional verification policies to ensure that borrowers who participate in IDR plans do not misrepresent their income or family size to the department. While Education agreed with our recommendation to obtain data in order to verify income for borrowers reporting zero income, it suggested that GAO may wish to convert this recommendation to a Matter for Congressional Consideration to provide Education with access to IRS data. In its response, Education stated that the President’s fiscal year 2020 budget request includes a proposal that Congress pass legislation allowing the IRS to disclose tax return information directly to the department for the purpose of administering certain federal student financial aid programs. According to Education, such legislation, if enacted, would allow borrowers to more easily certify their income on an annual basis to maintain enrollment in IDR plans, and allow the department to use the information to mitigate improper payments to borrowers as a result of misreported income data. Education also stated that in the meantime, it would explore whether commercially available data are sufficient in terms of scope, reliability, and cost effectiveness. Given that there are existing actions Education can take to implement our recommendation, we believe our recommendation is appropriate. Moreover, we believe that Education is best positioned to determine whether the proposal, if enacted, would address our recommendation, or if it would need to be expanded or modified in order to do so. Regarding our second recommendation, Education stated that it would develop data analytic practices to verify borrower reports of zero income contingent upon the enactment of legislation providing the department with access to federal income data. However, implementing our recommendation does not necessarily require Education to wait for such legislation. Our draft report describes data analytic practices, such as anomaly detection, which Education could implement using its own data to identify deviations from expected patterns in data over time. Education also stated that it plans to develop additional follow-up procedures to verify borrower reports of zero income, such as requiring borrowers to substantiate reports of zero income with appropriate documentation. In addition, Education described plans to formalize procedures to make referrals to Education’s Office of Inspector General or the Department of Justice for suspected cases of IDR fraud. We encourage Education to combine its follow-up procedures with data analytic practices to satisfy the recommendation. Education agreed with our third recommendation to implement data analytic practices and follow-up procedures to verify family size, noting that this information could be subject to misrepresentation or erroneous reporting by borrowers. Education stated that it would review various data points that can be used to select IDR applications and certifications for additional review prior to approval, such as providing more scrutiny when borrowers report unusual increases in family size from one year to the next. Education also stated that it plans to formalize additional procedures to require certain borrowers to substantiate their family size. For example, Education will consider requiring IDR applicants to provide statements listing each household member and how they are related to the borrower. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to relevant congressional committees, the Secretary of Education, the Chief Operating Officer of Federal Student Aid, and other relevant parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact us at (617) 788-0534 or emreyarrasm@gao.gov or (202) 512-6722 or bagdoyans@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff that made key contributions to the report are listed in appendix III. This report examines (1) whether there are indicators of potential fraud or error in income and family size information provided by borrowers seeking to repay their loans with Income-Driven Repayment (IDR) plans and (2) the extent to which the Department of Education (Education) verifies this information. To address these questions, we reviewed relevant IDR policies and procedures from Education and its four largest loan servicers—Navient, Nelnet, Great Lakes Educational Loan Services, Inc., and the Pennsylvania Higher Education Assistance Agency. We selected these loan servicers because, at the time of our analysis, together they serviced 96 percent of the outstanding balance of loans being repaid with IDR plans as of September 2017. We also interviewed Education officials from Federal Student Aid, the office responsible for developing policies and procedures for administering IDR plans and overseeing how loan servicers carry them out, as well as the officials from the selected loan servicers. Additionally, we reviewed relevant federal laws and regulations and Education’s procedures for verifying information on the Free Application for Federal Student Aid. We assessed Education’s procedures against federal standards for internal control for developing sufficient control activities, risk assessment, and information and communication. We also assessed Education’s procedures against the leading practices for data analytics activities in GAO’s Framework for Managing Fraud Risks in Federal Programs. To determine whether there were indicators of potential fraud or error in borrowers’ income and family size information on IDR plans, we obtained data from Education’s Enterprise Data Warehouse and Analytics (EDWA) database on borrowers with William D. Ford Federal Direct Loans (Direct Loans) and IDR plans approved between January 1, 2016 and September 30, 2017, the most recent data available at the time of our analysis. EDWA is a centralized data warehouse that contains administrative data reported by loan servicers on IDR borrowers and their loans. Some borrowers had multiple approved IDR plans in the data we analyzed. We also obtained national quarterly wage data from the U.S. Department of Health and Human Services’ (HHS) National Directory of New Hires (NDNH) for the same time period. NDNH is a national repository of information reported by employers, states, and federal agencies. The NDNH is maintained and used by HHS for the federal child support enforcement program, which assists states in locating parents and enforcing child support orders. In addition to information on newly hired employees, NDNH contains (1) data on quarterly wages for existing employees, collected and reported by state workforce agencies and federal agencies; and (2) data on all individuals who apply for or received unemployment compensation, as maintained and reported by state workforce agencies. For our analysis of borrower-reported incomes, we matched approximately 656,600 Education borrowers to NDNH quarterly wage data to determine if any borrowers who reported zero income on their IDR applications had wages reported in the same quarter in which their IDR plans were approved. We took additional steps to further review and refine these matches and provide reasonable assurance that the NDNH wage data were associated with the correct borrower by comparing (1) the borrower’s state of residence as reported in the Education data to the state agency submitting the NDNH wage data and (2) the borrower’s name as reported in the Education data to the employee name reported in the NDNH data. For the refined matches, we then estimated whether the borrowers may have had sufficient annual wages based on wages reported in NDNH to potentially warrant monthly student loan payments greater than zero dollars on their associated IDR plan. Specifically, we aggregated all NDNH wages reported for the borrower in the quarter in which their IDR plan was approved to determine a total quarterly wage amount. We then multiplied the total quarterly wage amount by four—the number of quarters in a calendar year—to generate an estimate of annual wages for the borrower. Our approach was based on the methodology Education instructs loan servicers to use to calculate annual wages when borrowers provide an alternative to a tax return to document their income on IDR applications. This methodology may understate or overstate income given that borrowers may not have earned the same amount in each of the four quarters. Our estimates of annual wages are based on the wages reported in NDNH for each borrower and do not take into account any pre-tax deductions that may apply when determining IDR payments. Our estimates of annual wages also do not include borrowers’ spousal income or any other taxable income for the borrower that is not included in the NDNH quarterly wage data—such as unemployment compensation received or unearned income such as alimony. We did not independently verify the wages reported in NDNH or the actual total annual income earned by borrowers identified in our match, as this was outside the scope of our review. Using the estimated annual wage, we then determined whether a borrower would have had a monthly payment greater than zero by using Education’s IDR plan repayment calculations for each IDR plan. To calculate the monthly payment, we used (1) the estimated annual wage from our NDNH data analysis; (2) the family size reported on the borrower’s approved IDR plan; (3) the borrower’s approved IDR plan type; and (4) the relevant percentage of the HHS poverty guideline amount for the borrower’s family size, state of residence, IDR plan approval year, and IDR plan type. For borrowers on Income-Based Repayment, New Income-Based Repayment, Pay As You Earn, and Revised Pay As You Earn plans, we rounded all calculated monthly payments that were less than $5 down to zero, in accordance with Education’s repayment calculations. We then identified which borrowers had calculated payments that were greater than zero. We did not determine the actual repayment amount borrowers may have had, as this was outside the scope of our review. Finally, for borrowers for whom we had calculated a payment greater than zero, we determined the total outstanding Direct Loan balance (principal and accrued interest) as of September 2017, based on EDWA data. For our analysis of borrower-reported family sizes, we analyzed the overall distribution of family sizes reported on approximately 5 million approved IDR plans. We reviewed the percentile distribution for family size on all IDR plans in our analysis and identified those in the top 1 percent of the data—in this case, IDR plans that had a reported family size of nine or more. We defined these IDR plans as having atypical family sizes for the Education data. We did not independently verify the family size reported on the IDR plans. For the borrowers with family sizes of nine or more, we determined the total outstanding Direct Loan balance (principal and accrued interest) as of September 2017. To examine the effects of borrowers inaccurately reporting income and family size on loan payment amounts, we analyzed the estimated monthly loan payment amounts for various hypothetical repayment scenarios from Education’s online repayment estimator as of January 2019, which used the 2018 HHS poverty guidelines. To examine the effect of various family sizes on loan payment amounts, we assumed a hypothetical borrower lived in the continental United States; had an adjusted gross income of $40,000; an outstanding Direct Loan balance of $30,000 (close to the average outstanding Direct Loan balance of $33,600 as of September 2018); and an interest rate of 5.1 percent (the Direct Loan 2018-2019 interest rate for an undergraduate borrower). To examine the effect of various incomes on monthly payment amounts, we assumed hypothetical borrowers had adjusted gross incomes based on estimated annual wages common in our data matching analysis ($30,000, $45,000, and $60,000), a family size of one (meaning just the borrower), and lived in the continental United States. For this analysis, we also assumed hypothetical borrowers had an interest rate of 5.1 percent and an outstanding Direct Loan balance of $50,000, which we selected to be high enough to qualify these hypothetical borrowers for all IDR plans at each of the selected income levels. To assess the reliability of the EDWA data, we reviewed documents related to the database and Education loan data generally; interviewed knowledgeable Education officials; performed electronic testing to determine the validity of specific data elements that we used to perform our work; compared the data we received to published Education data on the number of IDR borrowers and amount of their outstanding loans; and compared borrowers’ personal information to the Social Security Administration’s Enumeration Verification System to identify borrowers whose information may not have been accurate. As part of our reliability assessment of the EDWA data, we selected a nongeneralizable sample of 16 borrowers and their IDR plan and loan information from the EDWA data to compare against four selected loan servicers’ records. Specifically, we stratified borrowers into two groups based on common and potentially outlying incomes and family sizes in the EDWA data. We then randomly selected two borrowers from each stratum for each of the four selected loan servicers (a total of four borrowers per loan servicer). We reviewed all IDR plan data in our scope for each selected borrower, including the plan type, family size, income, and total monthly payment. We did not review original documents, such as the IDR applications or documentation of income. We discussed the results of our review with knowledgeable Education and loan servicer officials to gain additional understanding of each selected borrower’s IDR plan information as well as any differences between EDWA and loan servicer data. We originally obtained EDWA data on approximately 6.5 million IDR plans approved between January 1, 2016 and September 30, 2017 that were held by almost 4.8 million Direct Loan borrowers. Based on data reliability issues we identified during our review, we had to limit the scope of our analysis to a subset of EDWA data that we determined were sufficiently reliable for our purposes. Education officials disclosed issues that impacted the IDR plan data reported to Education by one of its loan servicers. Specifically, Education and the loan servicer had identified instances where the loan servicer’s internal data were changed for valid reasons but the changes were not reported to Education correctly. As a result, we excluded data reported by this servicer from all analyses in our report. We also identified issues with monthly payment amounts for some borrowers in the EDWA data. Accordingly, we limited our borrower- reported income analysis to borrowers who reported zero income and had a scheduled monthly payment of zero dollars. Ultimately, we analyzed about 878,500 IDR plans held by about 656,600 borrowers for our income analysis and approximately 5 million IDR plans held by 3.5 million borrowers for our family size analysis. Consequently, our overall income and family size analyses results may be understated and are not generalizable to all IDR plans and borrowers. Consistent with our report scope, our analyses of borrower-reported income focused on identifying indications of potential fraud or error; however, our analyses do not show that fraud or error occurred. It is not possible to determine whether fraud or error occurred through data matching alone. As previously discussed, our estimates of annual wages are based on the NDNH quarterly wage data, and do not take into account any deductions that may be applicable for determining adjusted gross income, which is used to determine IDR plan payment amounts. As a result, our estimates could overstate borrowers’ incomes for IDR plan purposes. Additionally, wages are reported in NDNH quarterly, so we are not able to determine when in a quarter a borrower earned wages. For example, a borrower may have earned wages at the start or end of a quarter, but was not earning wages at the time of submitting the IDR application. Because borrowers are only required to certify their income annually, such a scenario would not constitute fraud or error even though it would result in a match in our analysis. In addition, our use of Education’s methodology to annualize wages based on quarterly wages may understate or overstate income if a borrower did not earn wages at the same level over the entire year. We are also not able to identify additional taxable income that is not reported to NDNH but should have been included on borrowers’ IDR applications, which could understate borrowers’ incomes. Consequently, our analysis may overstate or understate the number of borrowers who reported no income on their IDR application yet may have had sufficient wages to warrant a monthly student loan payment. To assess the reliability of the NDNH data, we reviewed documents related to the database, interviewed knowledgeable HHS officials, and performed electronic testing to determine the validity of specific data elements in the NDNH data that we used to perform our work. On the basis of our own reliability assessment results, we determined that the NDNH data were sufficiently reliable for the purposes of this report. We conducted this performance audit from June 2017 to June 2019, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contacts named above, Debra Prescott and Philip Reiff (Assistant Directors), Nancy Cosentino and Mariana Calderón (Analysts- in-Charge), Sarah Cornetto, Jeffrey G. Miller, and Rachel Stoiko made key contributions to this report. Additional assistance was provided by Susan Aschoff, David Ballard, Deborah Bland, Benjamin Bolitzer, Melinda Cordero, Vijay D’Souza, Kevin Daly, Angie Jacobs, Candace Silva-Martin, Sheila R. McCoy, Maria McMullen, Kevin Metcalfe, John Mingus, Drew Nelson, Mimi Nguyen, Matt Valenta, and Ariel Vega.", "summary": "As of September 2018, almost half of the $859 billion in outstanding federal Direct Loans was being repaid by borrowers using IDR plans. Prior GAO work found that while these plans may ease the burden of student loan debt, they can carry high costs for the federal government. This report examines (1) whether there are indicators of potential fraud or error in income and family size information provided by borrowers on IDR plans and (2) the extent to which Education verifies this information. GAO obtained Education data on borrowers with IDR plans approved from January 1, 2016 through September 30, 2017, the most recent data available, and assessed the risk for fraud or error in IDR plans for Direct Loans by (1) matching Education IDR plan data for a subset of borrowers who reported zero income with wage data from NDNH for the same time period and (2) analyzing Education IDR plan data on borrowers' family sizes. In addition, GAO reviewed relevant IDR policies and procedures from Education and interviewed officials from Education. GAO identified indicators of potential fraud or error in income and family size information for borrowers with approved Income-Driven Repayment (IDR) plans. IDR plans base monthly payments on a borrower's income and family size, extend repayment periods from the standard 10 years to up to 25 years, and forgive remaining balances at the end of that period. Zero income. About 95,100 IDR plans were held by borrowers who reported zero income yet potentially earned enough wages to make monthly student loan payments. This analysis is based on wage data from the National Directory of New Hires (NDNH), a federal dataset that contains quarterly wage data for newly hired and existing employees. According to GAO's analysis, 34 percent of these plans were held by borrowers who had estimated annual wages of $45,000 or more, including some with estimated annual wages of $100,000 or more. Borrowers with these 95,100 IDR plans owed nearly $4 billion in outstanding Direct Loans as of September 2017. Family size. About 40,900 IDR plans were approved based on family sizes of nine or more, which were atypical for IDR plans. Almost 1,200 of these 40,900 plans were approved based on family sizes of 16 or more, including two plans for different borrowers that were approved using a family size of 93. Borrowers with atypical family sizes of nine or more owed almost $2.1 billion in outstanding Direct Loans as of September 2017. These results indicate some borrowers may have misrepresented or erroneously reported their income or family size. Because income and family size are used to determine IDR monthly payments, fraud or errors in this information can result in the Department of Education (Education) losing thousands of dollars of loan repayments per borrower each year and potentially increasing the ultimate cost of loan forgiveness. Where appropriate, GAO is referring these results to Education for further investigation. Weaknesses in Education's processes to verify borrowers' income and family size information limit its ability to detect potential fraud or error in IDR plans. While borrowers applying for IDR plans must provide proof of taxable income, such as tax returns or pay stubs, Education generally accepts borrower reports of zero income and borrower reports of family size without verifying the information. Although Education does not currently have access to federal sources of data to verify borrower reports of zero income, the department could pursue such access or obtain private data sources for this purpose. In addition, Education has not systematically implemented other data analytic practices, such as using data it already has to detect anomalies in income and family size that may indicate potential fraud or error. Although data matching and analytic practices may not be sufficient to detect fraud or error, combining them with follow-up procedures to verify information on IDR applications could help Education reduce the risk of using fraudulent or erroneous information to calculate monthly loan payments, and better protect the federal investment in student loans. GAO recommends that Education (1) obtain data to verify income information for borrowers who report zero income on IDR plan applications, (2) implement data analytic practices and follow-up procedures to verify borrower reports of zero income, and (3) implement data analytic practices and follow-up procedures to verify borrowers' family size. Education generally agreed with our recommendations.", "document_type": "gao"}
{"report": "Lebanon is a small, religiously diverse country bordering the Mediterranean Sea (see fig. 1). Religious tensions among Lebanon’s Maronite Christians, Sunni Muslims, Shiite Muslims, and others, have for many years contributed to conflicts within Lebanon as well as with neighboring countries. According to State, Lebanon’s political system is characterized by sectarian divisions and pressures from external and internal forces that limit its ability to function. Upon gaining independence from France in 1943, Lebanese leaders adopted a power-sharing agreement, in which each of the country’s officially recognized religious groups were to be represented in the government according to their share of the population based on the 1932 census. This unwritten agreement established a status quo in which the president must be a Maronite Christian (the largest single denomination in 1932), the prime minister a Sunni Muslim, and the speaker of parliament a Shia Muslim. Tensions over the balance of power among these groups have provoked conflict. During the Lebanese Civil War from 1975 to 1990, both Syrian and Israeli forces occupied the country. In the midst of the civil war and Israel’s occupation of southern Lebanon, Hizballah emerged in Lebanon as a powerful Islamic militant group. In 2000, Israeli forces withdrew from southern Lebanon. In 2005, owing to pressure from the international community, Syrian forces withdrew from Lebanon following the assassination of Lebanon’s prime minister. Parliamentary elections in that year led to a member of Hizballah holding a cabinet position for the first time, and at least one member of Hizballah has held a cabinet position ever since. Instability arising from the Syrian civil war that began in 2011 has also exacerbated sectarian conflict and created new challenges within Lebanon. In particular, that war has caused an influx of over 1.3 million Syrian refugees into Lebanon, a country with a population of only 4.5 million. The Syrian civil war has also increased the risk of terrorist incidents in Lebanon, as foreign terrorist fighters have crossed Lebanese borders going to and from the conflict. Since 2013, the United States’ primary goal in providing security assistance to Lebanon has been to strengthen its state institutions to allow them to exert sovereign authority and enhance security. Since at least 2015, the primary objectives supporting this goal have focused on 1) building the capacity of Lebanese security forces to exert sovereign authority over Lebanese territory, including at the border and by maintaining internal security; and 2) enhancing the capacity of Lebanese security forces to respond to terrorist and criminal threats. Through both these objectives, the U.S. also seeks to delegitimize and marginalize Hizballah by helping to support legitimate state institutions. To achieve these objectives, a number of agencies and offices within State and DOD provide support to the LAF, which is generally responsible for providing border security, counterterrorism, and national defense, and to the ISF, or national police force, which is generally responsible for maintaining law and order within Lebanon. See table 1. U.S. support for Lebanese forces has included a variety of assistance, including training, equipment, and sustainment, as shown in figure 2. State and DOD reported that they obligated nearly $1.5 billion in security assistance funding for Lebanon in fiscal years 2013 through 2018. The largest security assistance programs were State’s Foreign Military Financing program, which provides grants and loans to foreign governments for the acquisition of U.S. defense equipment, services, and training, and DOD’s Global Train and Equip program, which funds training and equipment for foreign military forces to conduct counterterrorism operations and enhance maritime and border security. These two programs collectively accounted for nearly 80 percent of assistance. State provided about 56 percent of the overall funding and DOD contributed 44 percent, as shown in figure 3. DOD and State are required to conduct end-use monitoring (EUM) for some of the equipment provided to Lebanon. In 1996, Congress amended the Arms Export Control Act to require the President to establish a program for monitoring the end-use of defense articles and defense services sold, leased, or exported under the act, including through Foreign Military Sales, Foreign Military Financing, or the Foreign Assistance Act of 1961. The law requires that the program be designed to provide reasonable assurances that recipients are complying with restrictions imposed by the U.S. government on the use, transfer, and security of defense articles and defense services, and that such articles and services are being used for the purposes for which they are provided. DOD’s Defense Security Cooperation Agency (DSCA) is responsible for EUM for Foreign Military Sales. The President is also required to take all reasonable steps to ensure that equipment made available to foreign countries for international narcotics control under the Foreign Assistance Act are used only in ways consistent with the purposes for which such equipment was made available. State’s INL implements this requirement through its End-Use Monitoring Program. DSCA administers the Golden Sentry program, which DOD uses to comply with requirements related to the end-use of defense articles and services transferred to foreign governments. DOD officials at the Office of Defense Cooperation-Beirut conduct the EUM activities established and overseen by DSCA. DSCA’s policy manual for EUM, the Security Assistance Management Manual, and the associated standard operating procedures for Beirut require DOD officials to, among other things, conduct two levels of monitoring: routine EUM and enhanced EUM. Routine EUM: DOD conducts routine EUM for defense articles and services that do not have any unique conditions associated with their transfer. In conducting routine EUM, DOD personnel are required to observe and report any potential misuse or unapproved transfer of U.S.-origin defense articles. Routine EUM is to be conducted in conjunction with other required security-related duties, using any readily available information. For example, U.S. officials might observe how a host country’s military uses U.S. equipment when they visit a military installation on other business. DOD policy states that routine EUM must be documented at least quarterly. DOD policy does not require inventories and physical security checks as part of routine EUM. Enhanced EUM: DOD conducts enhanced EUM for defense services, technologies, or articles specifically identified as sensitive. Lebanon has five types of sensitive defense articles that require enhanced EUM—night vision devices, sniper rifles, light attack aircraft, unmanned aerial vehicles, and Hellfire missiles. DOD policy requires serial number inventories for defense articles needing enhanced EUM within 90 days of delivery of the articles and thereafter within one year of the last inventory performed. In addition, the purchase agreements authorizing the sale of an item may contain specialized notes directing the purchaser to adhere to certain physical security and accountability requirements. In addition to enhanced and routine EUM, DSCA is required to conduct periodic Compliance Assessment Visits to evaluate the Office of Defense Cooperation in Beirut’s compliance with DOD’s EUM policy and the Lebanese government’s compliance with physical security and accountability requirements. According to State and DOD assessments, reports, and interviews with State and DOD officials, the LAF’s border security and counterterrorism capabilities have demonstrated some notable improvements from 2013 to 2018. For example, a 2013 DOD assessment noted that the Lebanese government lacked effective control over its sovereign territory and indicated the LAF leadership was reluctant to engage aggressively in counterterrorism operations. By 2018, however, U.S. agencies reported that, following the expansion of LAF Land Border Regiments, Lebanon had established control of a large part of its borders for the first time in its history. In addition, U.S. agencies reported that the LAF had enhanced its capacity in counterterrorism and counter-narcoterrorism, resulting in more operations. In 2017, for example, the LAF undertook a successful operation to expel ISIS elements along the border with Syria, making Lebanon, DOD officials noted, the only country in the region to successfully expel ISIS from its territory without the involvement of U.S. ground forces. Similarly, State reported improvements in the ISF’s capabilities. For example, INL reported that its ISF training program has become increasingly specialized because of the force’s improved capabilities. According to State reporting, from 2008 through 2012, INL focused its training for the ISF on basic skills. As the ISF became more capable, however, INL reported that the ISF assumed responsibility for all basic training, allowing INL to focus its resources on providing specialized courses. Some examples of these specialized courses include advanced technical radio training and advanced interview and interrogation training. INL also reported that providing equipment and facilities to the ISF helped further to enhance ISF capabilities. For instance, INL reported that the ISF uses the academy INL constructed for it in 2015 in Aramoun for advanced forensics training. In addition, the ISF improved its overall investigative capacity and counterterrorism capabilities since 2013, as shown in a 2017 assessment of State’s Antiterrorism Assistance program. U.S. officials stated that the quality of the working relationship between the U.S. and Lebanon is an important component of success, and Lebanese officials said that U.S. assistance is critical to achieving their mission. U.S. officials noted that the LAF and ISF have been some of the most committed U.S. partners in the region. The LAF and ISF officials we met with also said that U.S. assistance enhances their capabilities and allows them to do their jobs more effectively. One ISF unit, for example, stated that buses purchased with U.S. assistance allow it to transport large numbers of personnel to mission locations. In addition, one LAF unit noted that U.S.-provided armored personnel carriers form the backbone of the LAF’s armored brigades. Despite reported progress, U.S. agencies indicated that some challenges remain for the ISF and the LAF. While the ISF’s capabilities have improved since 2013, U.S. officials said it continues to be more capable in and around Beirut than in other parts of the country. As a result, the LAF often provides internal security to supplement the ISF outside of the capital. Additionally, the ISF needs to improve its internal coordination of cybercrime cases and analyses of digital evidence, according to a 2017 assessment of State’s Antiterrorism Assistance program. For example, the ISF units handling digital investigations and processing, the assessment noted, were fractured and divided, resulting in overwhelming workloads for some units and underutilization of others. DOD assessments also noted that the LAF continues to have some capability gaps, including an ongoing need for equipment and challenges with operating and maintaining U.S.-provided equipment. For example, LAF personnel have expressed concerns about the complexity and sustainability of some U.S. systems, such as the M2 Bradley Fighting Vehicles and A-29 light attack aircraft. Additionally, while U.S. officials stressed they have no desire for direct confrontation between Lebanese security forces and Hizballah, U.S. agencies report that Hizballah’s presence within Lebanon remains a challenge for both the ISF and LAF. In 2018, for example, State reported that Hizballah was the most capable terrorist organization in Lebanon and that it continued to exert control over some areas of the country. In addition to periodically assessing long-term performance, State’s Foreign Affairs Manual and internal guidance outline a number of good practices for ICS management. First, the Foreign Affairs Manual says all missions, such as Embassy Beirut, should have an ICS with a hierarchy of goals, objectives, sub-objectives, and, as needed, key activities. Second, missions must assess progress against ICS strategic objectives at least annually. Third, State internal guidance says it is a good practice for missions to establish ICS performance indicators with targets to show the expected change over the course of each period of performance. Fourth, it is also a good practice for missions to practice regular, ongoing data collection against key performance indicators to gauge the direct and near-term effects of activities. The 2018 ICS for Lebanon includes a hierarchy of goals, objectives, and sub-objectives, in line with the guidance in State’s Foreign Affairs Manual. For instance, the Lebanon ICS has objectives with sub- objectives that include activities outlining how to accomplish those objectives. The 2018 ICS contains 19 security-related activities with corresponding performance indicators for State and DOD activities, such as training Lebanese security forces in counterterrorism or border security operations. The hierarchy included in the 2018 ICS represents an improvement from the previous ICS, developed in 2015, which included information on goals, objectives, and sub-objectives, but did not outline specific activities or performance indicators. State guidance notes the benefit of such a hierarchy is that it shows the individuals who work on such activities how their actions contribute to achieving mission objectives. According to State officials, Embassy Beirut conducted an annual review of the ICS in October 2019. The goals of the annual review, according to State officials, were to assess progress against the ICS objectives and to remove or add goals, objectives, and key activities as needed. In July 2019, Embassy Beirut officials told us that they planned to conduct a review of the ICS approximately one year after its approval, which was in August 2018. However, State officials told us that leadership turnover in the summer of 2019 resulted in Embassy Beirut delaying the review until October 2019. Embassy Beirut, however, has not established targets for all of the 19 security-related performance indicators in its 2018 ICS. The Foreign Affairs Manual emphasizes that having targets to indicate the expected change over the course of each period of performance is a good practice. Several of Embassy Beirut’s security-related ICS indicators lack such targets, making it difficult for State to use the indicators to assess progress because it cannot compare the actual results of activities to the expected results. For example, several of the embassy’s security-related performance indicators deal with the number of people trained or improvements in specific capabilities of the security forces. Because the embassy has not established targets for these particular indicators, State cannot quantify the results it expects to achieve or determine how the actual results compare to those expectations. State officials noted that some bureaus have established performance indicators that are the same as, or similar to, security-related performance indicators in the ICS and some of those indicators have targets. For example, INL officials noted that INL has a Country Plan for Lebanon that has performance indicators and targets similar to some of the security-related performance indicators found in the ICS. However, many of the security-related activities included in the ICS are implemented by more than one agency or bureau. Therefore, the performance indicators for these activities would require targets that account for all the implementers. Additionally, Embassy Beirut did not have complete performance data for its security-related ICS performance indicators. State’s Foreign Affairs Manual emphasizes that regular, ongoing data collection against performance indicators to gauge the direct and near-term effects of activities is a good practice. Federal standards for internal control also state that agencies should use quality information that is, among other things, complete. Information is complete if it includes relevant data needed by decision makers to assess performance or to allocate resources. When we requested information on progress made toward the security-related indicators in the 2018 ICS, Embassy Beirut provided incomplete data for 11 of the 15 security-related indicators we analyzed. Data for six of these 11 were incomplete because the indicator called for quantitative data that were not included. For example, three of the six quantitative indicators called for data on the number or percentage of people trained. Embassy Beirut provided information that stated training had occurred, but did not quantify the number or percentage of people trained, as called for by the indicators. Data for the other five of these 11 indicators were incomplete because the indicators called for qualitative data that were not included. For example, three of the five qualitative indicators dealt with improving the capacity or capabilities of Lebanese units, but the information Embassy Beirut provided did not include a description of whether or how Lebanese units improved in those areas. Embassy Beirut provided complete data for four of the 15 indicators we analyzed, as shown in table 2. For three of the four indicators, Embassy Beirut provided the quantitative data called for by the indicator. For the remaining indicator, which dealt with number of personnel trained and the completion of facility upgrades, the embassy provided data on the number of personnel trained and a description of the status of the upgrades. According to Embassy Beirut officials, individual programs have targets and collect performance data associated with the security-related ICS performance indicators, but the Embassy did not have such information consolidated in a centralized document covering the time period we reviewed. Officials further noted that the ICS contains performance indicators, but not specific targets, as the ICS was not a vehicle for establishing specific targets when it was drafted in 2018. Additionally, State officials at headquarters stated that they do collect performance data related to some of the Lebanon ICS security-related indicators, but they did not provide evidence that this data is available to or used by Embassy Beirut as part of its ICS review. To review targets and indicators as part of the annual ICS review, Embassy Beirut officials said they planned to use evaluations and assessments of programs conducted by State and DOD headquarters entities or third parties. However, these assessments and evaluations cannot provide complete data on Embassy Beirut’s security-related performance indicators because not all of the security assistance programs in Lebanon have conducted them. In addition, these assessments and evaluations do not take place annually, which limits Embassy Beirut’s ability to use them on an ongoing basis to monitor strategic activities. Without setting targets and collecting complete data on performance indicators, Embassy Beirut will be limited in its ability to monitor its progress toward achieving the expected results of its security-related activities. State documents indicate that sound program design and performance management serve as the basis for efficient and effective use of department resources to achieve strategic objectives. If Embassy Beirut does not address the gaps in its performance information, it will be limited in its ability to ensure the intended alignment of policy, planning, resources, and programs through its annual reviews of the ICS. State and DOD’s two primary safeguards to limit the risk that U.S. security assistance to Lebanon will benefit terrorist organizations are: 1) reviewing Lebanese security organizations for ties to terrorist organizations and 2) vetting individual recipients of assistance. For the first safeguard, State examines Lebanese security organizations for associations with foreign terrorist organizations (FTO) prior to providing support. Annual State, Foreign Operations, and Related Appropriations acts for fiscal years 2013 through 2018 included provisions to restrict funding for the ISF or the LAF if they are controlled by a U.S.-designated FTO. According to State officials, under these provisions, State regularly evaluates the LAF and ISF to determine if they have strong individual or organizational connections or alignment of purpose with Hizballah or any other FTO. State officials said they have determined that both the LAF and ISF are independent institutions that Hizballah does not control. State officials added that some longstanding divisions exist between Hizballah and the ISF, in particular. For example, one State official noted that Hizballah has assassinated ISF leaders in the past. Furthermore, members of the ISF are not allowed to be members of any political party, according to State officials. Second, State and DOD vet members of the Lebanese security forces who will receive U.S. assistance, such as training, for ties to terrorism. State and DOD vet by checking the names and other biographic or biometric information of potential recipients of assistance against information about known or suspected terrorists and their supporters. State and DOD officials conduct name-check vetting using one or more of three methods: In-country screening: State officials said they review a variety of sources in Lebanon to screen all potential recipients of State and DOD-funded training. Consular Affairs officials use State’s Independent Namecheck application to vet all potential trainees in country. This application allows overseas posts to screen names of individuals through State’s Consular Lookout and Support System (CLASS) database. CLASS contains records from numerous U.S. agencies on persons with immigration violations and terrorism connections, among other potential visa ineligibilities. In addition, officials said they may examine other sources, including local law enforcement or U.S. intelligence community sources. Terrorist Screening Center: State INL sends the names of potential ISF trainees to the Terrorist Screening Center, a multi-agency center administered by the Federal Bureau of Investigation, for further vetting. INL officials noted that this step does not result in many more exclusions beyond the initial in-country screening, but it serves as an additional check to ensure INL funding does not benefit FTOs. Nonimmigrant visa vetting: Any potential trainees who apply to come to the U.S. for training undergo vetting for a nonimmigrant visa, which includes interagency counterterrorism checks. According to DOD officials, a majority of their LAF trainees receive training in the United States. Some trainees under State programs also receive training in the United States. Officials said they believe these vetting procedures provide sufficient assurances that LAF and ISF trainees are not members of an FTO. They also stated they receive a relatively small number of “hits,” or indications that screening uncovered derogatory information. In these cases, officials said they remove the individual from the training roster and screen a substitute applicant instead. According to State officials, INL is in the process of moving its namecheck vetting from the Terrorist Screening Center to an internal State office. From 2012 to 2017, State piloted a counterterrorism vetting program for five countries, including Lebanon, through the Risk Analysis and Management (RAM) team in State’s Bureau of Administration. Vetting for Lebanon conducted through the pilot focused primarily on vetting contractors and grantees that would potentially implement U.S. assistance programs, including a security assistance program in 2015. RAM officials said that they resumed vetting in February 2019 for some programs in Lebanon, as determined by programming offices based on program-specific risk assessments that identify risks that can be mitigated through namecheck vetting. These officials said all the screening they conducted for Lebanon during the initial pilot phase was for programs determined to be of low or medium risk and, as of November 2019, they had not found derogatory information for any of the screened individuals. INL conducted annual inspections of equipment it provided to the ISF, as required by State policy. According to INL’s annual EUM reports, from 2013 through 2018, INL annually inspected 100 percent of the equipment valued at over $2,500 and defense articles regardless of value provided to the ISF, either by on-site inspection or host government verification. During our visit to an ISF site in Beirut, Lebanon, we found that all 16 items included in our random, non-generalizable sample were either physically present or accounted for through documentation. We observed 12 of the 16 items, such as police motorcycles and buses. The ISF provided documentation showing that the remaining four items, all trucks, were unavailable for inspection because the ISF had deployed them on missions. Figure 4 shows police motorcycles provided to the ISF that were inventoried by serial number. To provide reasonable assurance that recipients comply with U.S. government restrictions on the use and security of defense articles, DOD’s EUM standards require the Office of Defense Cooperation in Beirut to conduct enhanced EUM for designated sensitive defense articles, such as night vision devices provided to the LAF. U.S. officials must conduct an initial inventory of equipment requiring enhanced EUM within 90 days of delivery and must visually inventory 100 percent of enhanced EUM-designated defense articles within one year of the last inventory, or within 90 days of an acceptable reason for missing an inspection (such as the item was deployed), and enter inventory information into DOD’s SCIP database. DOD officials accounted for all of the 2,991 items subject to enhanced EUM from 2013 through 2018 at least once, according to our analysis of SCIP data. DOD officials in Beirut said they conducted serial number inventories of all items requiring enhanced EUM from 2013 through 2018, as required by DOD’s EUM program. During our visit to Lebanon in April 2019, DOD officials in Beirut physically observed nearly 100 percent (270 of 271) of the defense articles requiring enhanced EUM at the three LAF locations we visited. Only one of the 271 items was unavailable for inspection and the LAF provided documentation showing it was out for repair. Figure 5 shows night vision devices provided to the LAF that were inventoried by serial number. DOD reporting, including a 2017 DSCA Compliance Assessment Visit and U.S. Central Command Inspector General reports, indicates that the LAF has generally complied with DOD requirements to account for and secure equipment and conduct compliance checks of all required equipment. DOD officials said the LAF is transparent about the location of the equipment and goes out of its way to ensure DOD officials are able to account for it. The officials also said the LAF is rigorous about safeguarding all required equipment and consistently meets standards equivalent or similar to U.S. standards for equipment accountability. While our analysis showed that DOD generally accounted for items requiring enhanced EUM, we also found that DOD did not always conduct inspections consistent with its timeliness standards. If DOD does not inspect an item within the timeframes required by its standards, DOD considers the inspection delinquent. Our analysis of the duration between inspections from 2014 through 2018 showed delinquencies in each year and, in total, 32 percent (4,533) of the 14,287 recorded observations we analyzed for timeliness were delinquent. We found that 86 percent of the 2,874 items we analyzed for timeliness had at least one delinquent inspection during the 6 years we reviewed, and 61 percent had two or three delinquent inspections. While inspections were often delinquent, we found that the length of time items remained delinquent was not extensive, with the average length of each delinquency lasting 2.6 months. Only 1 percent of recorded observations showed a delinquency of 6 months or longer. Figure 6 shows the duration of delinquencies for those inspections that were delinquent. Officials from both the Office of Defense Cooperation in Beirut and DSCA stated that the method that DSCA uses to determine inspection due dates for annual inspection plans impedes the Office of Defense Cooperation’s ability to meet DOD’s timeliness standards. DSCA assigns due dates for items based on a general category code instead of an individual item’s serial number, which according to DOD officials, does not allow the Office of Defense Cooperation-Beirut to plan inspections in a way that meets DOD’s timeliness standards. For example, one type of night vision device represents 61 percent of the 2,991 items requiring enhanced EUM. Because these items all have the same general category code, DSCA designates all of them as due for inspection on the same day, regardless of when DOD officials last inspected each individual item. As a result, the inspection due dates DSCA establishes may be inconsistent with DOD’s guidance, which complicates planning and could result in some items having nearly 2 years between inspections before DCSA flags them as delinquent. Addressing how DSCA determines inspection due dates for items requiring enhanced EUM is important for ensuring the Office of Defense Cooperation has the information it needs to meet DOD’s timeliness standards for equipment accountability. According to DOD officials, as of April 2019, equipment on order for the LAF would double the number of items subject to enhanced EUM inspections. This increase underscores the importance of providing the Office of Defense Cooperation-Beirut accurate inspection due dates for the equipment provided to Lebanon. By not assigning inspection due dates consistent with DOD standards, DSCA hinders the Office of Defense Cooperation’s ability to plan effectively. It also increases the likelihood DOD will experience continued delays in conducting the required checks that ensure the proper safeguarding and usage of sensitive defense articles. Recognizing Lebanon’s importance to the security and stability of the Middle East, U.S. agencies invested nearly $1.5 billion in security assistance to the country from fiscal years 2013 through 2018. However, the prominent role of Hizballah in the Lebanese government complicates the U.S. relationship with Lebanon and heightens the importance of ensuring strong management controls over U.S. assistance. U.S. agencies report that the LAF and ISF have improved in their capabilities to secure Lebanon’s border and to combat terrorist activity. Embassy Beirut has also taken a number of steps to track progress toward meeting U.S. security-related objectives. Gaps in the embassy’s performance information, however, limit its ability to monitor the ongoing progress of specific activities and to make informed decisions about where to allocate resources and attention. State and DOD did conduct end-use checks of all required items and their reporting indicates the LAF and ISF have generally taken appropriate steps to safeguard equipment. DOD, however, did not meet its timeliness standards for nearly one-third of all observations of sensitive equipment from 2013 through 2018. DSCA does not assign inspection due dates in a way that is consistent with DOD standards, which may limit DOD’s ability to fully ensure items requiring enhanced end-use monitoring are safeguarded and used as intended in a timely manner. We are making a total of three recommendations, including two to State and one to DOD: The Secretary of State should direct the Department’s relevant bureaus to work with Embassy Beirut to establish, as appropriate, and consolidate targets for each of the security-related performance indicators. (Recommendation 1) The Secretary of State should direct the Department’s relevant bureaus to work with Embassy Beirut to collect and review performance data for key security-related performance indicators. (Recommendation 2) The Secretary of Defense should direct DSCA to revise the inspection due dates it establishes for items requiring enhanced EUM for the Office of Defense Cooperation in Beirut to align with DOD’s standards for EUM by considering the date of last inspection. (Recommendation 3) We provided a draft of this report to the Departments of State and Defense for comment. In its comments, reproduced in appendix II, State concurred with the recommendations that Embassy Beirut 1) establish, as appropriate, and consolidate targets for; and 2) collect and review performance data for its security-related performance indicators. State also provided technical comments, which we incorporated as appropriate. We requested comments on a draft of this product from DOD. The Director for Egypt, Israel, and the Levant in the Office of the Secretary of Defense for Policy provided us with the Department’s comments in an email stating that DOD concurs with the recommendation that DOD direct DSCA to revise the inspection due dates it establishes for items requiring enhanced EUM for the Office of Defense Cooperation in Beirut to align with DOD’s standards for EUM by considering the date of last inspection. The Director noted that the current SCIP EUM software complicates annual inventory planning and reporting and that DSCA’s EUM personnel have documented a software modification requirement that would allow them to implement the recommendation. Additionally, she stated that final design and budget decisions for fiscal year 2021 are not yet complete and the magnitude of this software modification is a major task that is core to EUM programming. We are sending copies of this report to the appropriate congressional committees and the Secretaries of State and Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff has any questions about this report please contact me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. For fiscal years 2013 through 2018, we (1) examined to what extent the Department of State (State) and the Department of Defense (DOD) assessed the progress of their efforts to meet strategic objectives related to security for Lebanon; (2) described what safeguards State and DOD have put in place to limit the risk of U.S. security assistance provided to Lebanon benefitting terrorist organizations; and (3) analyzed to what extent State and DOD conducted end-use monitoring (EUM) checks of equipment provided to Lebanese security forces. To determine to what extent State and DOD assessed the progress of their efforts, we reviewed agency documentation and interviewed State, DOD, and Lebanese government officials. We reviewed Embassy Beirut’s Integrated Country Strategies (ICS), for fiscal years 2015 through 2018 to determine agencies’ strategic objectives for security assistance. We compiled information from State and DOD assessments and performance reporting on security assistance programs operating in Lebanon from fiscal years 2013 through 2018, including assessments, evaluations, and surveys. While we did not independently evaluate the quality of these documents, we did review their methodologies and determined that the approaches taken generally appeared reasonable. We also reviewed State and DOD performance reporting, such as program annual reports and Embassy Beirut’s Performance Plans and Reports for fiscal years 2013 through 2018. We then reviewed the compiled evidence to determine what this reporting showed about to what extent agencies had made progress toward their strategic objectives from fiscal years 2013 through 2018. We reviewed Embassy Beirut’s process for monitoring progress on its 2018 ICS—including what information the embassy compiles and how it determines whether programs are achieving their intended results. We reviewed State’s Foreign Affairs Manual and federal standards for internal control to identify key practices for ICS management. We compared the 2018 ICS to these key practices and requested information on Embassy Beirut’s assessment of progress on 19 security-related activities and indicators included in its 2018 Lebanon ICS. Embassy Beirut provided information for each of the activities and indicators as of May 2019. We reviewed the information provided by Embassy Beirut to determine if it was complete. Four of the 19 performance indicators covered activities for which performance data was not yet available. We did not include these four indicators in our analysis. For the remaining 15 indicators, we determined that Embassy Beirut provided complete data if it included relevant data needed by decision makers to assess performance. According to federal internal control standards, relevant data have a logical connection with identified information requirements. For example, if the information required for an indicator was quantitative in nature (such as the number or percentage of people trained), then we considered the information provided to be relevant if it included quantitative data that directly addressed the indicator. To describe what safeguards U.S. agencies have put in place to prevent U.S. security assistance from benefitting terrorist organizations, we reviewed legislative requirements, State policy guidance, and agency documentation. We reviewed annual appropriations acts from fiscal years 2013 through 2018 to determine what, if any, restrictions were placed on funding for Lebanon to ensure assistance did not benefit terrorist organizations. We reviewed State policy guidance on counterterrorism vetting and interviewed State and DOD officials in Washington, D.C. and Beirut, Lebanon about the steps they take to prevent assistance from benefitting U.S.-designated foreign terrorist organizations, including Hizballah. We reviewed agency documentation, including a risk assessment, an interagency memo, and State memos requesting the release of funding. We reviewed what safeguards State and DOD use to limit the risk of U.S. security assistance benefitting terrorist organizations, but did not analyze how the agencies made determinations when applying these safeguards. To evaluate to what extent State and DOD conducted EUM checks of equipment provided to Lebanese security forces, we reviewed agency documentation and data and interviewed State and DOD officials in Washington, DC and Beirut, Lebanon. We also reviewed State and DOD EUM standards to determine what requirements the agencies established for their respective programs and conducted site visits in Lebanon. To evaluate to what extent State conducted EUM checks in accordance with its standards, we reviewed State’s Bureau of International Narcotics and Law Enforcement Affairs (INL) annual end-use monitoring reports for 2013 through 2018 and analyzed INL EUM data. We also interviewed State officials in Washington, D.C. and Beirut, Lebanon about their processes for conducting and recording EUM. Because INL officials told us its EUM annual reports are the agency’s official documents for tracking adherence to EUM requirements, we used the reports in our analysis of State’s compliance with its standards. We interviewed INL officials about any identified discrepancies within the annual reports or between the annual reports and INL’s EUM data and determined that the reports were sufficiently reliable for our purposes. In Beirut, Lebanon, we visited two Lebanese Internal Security Forces (ISF) sites to observe the ISF’s processes for safeguarding and inventorying equipment. At the ISF’s Mobile Forces site, we reviewed a random, nongeneralizable sample of 16 items requiring EUM—all of which were vehicles, including buses, motorcycles, and trucks. We reviewed the serial numbers of items that were available on-site and reviewed ISF documentation accounting for those items that were not immediately available. To evaluate to what extent DOD had conducted EUM checks in accordance with its standards, we analyzed data from DOD’s Security Cooperation Information Portal (SCIP) database for 2013 through 2018 and interviewed officials from DOD’s Defense Security Cooperation Agency (DSCA) in Washington, D.C. and the Office of Defense Cooperation in Beirut, Lebanon. To analyze SCIP data, we compared observations recorded in the database against DOD’s standards. DOD’s Security Assistance Management Manual standards for EUM state that Security Cooperation Offices, like the Office of Defense Cooperation in Beirut, must visually inventory 100 percent of in-country enhanced EUM- designated defense articles within one year from the last inventory performed, except for those enhanced EUM-designated defense articles not available for observation (such as deployed or returned to the United States for repair), or as stipulated otherwise in the SCIP-EUM database or by separate policy memo. According to DOD’s standards, enhanced EUM-designated items not available for inventory during their annual inventory cycle due to deployment, returned to the United States for repair, or other legitimate reason, must be inventoried within 90 days after returning from deployment or repair. Each observation in the SCIP database represented a single inspection or attempted inspection of an item and includes, among other things, the item’s serial number, equipment category type, location, status, and date observed. Because we analyzed multiple years of data, DOD recorded more than one observation for almost all items. To evaluate to what extent DOD met its standards, we used the following parameters in our analysis: We determined that an observation met the requirement for being inventoried within one year from the last inventory performed if it occurred within 12 months of the last observation of the same item. Using a standard of 12 months between visits provides a small amount of leeway to account for the fact that security conditions, Lebanese Armed Forces (LAF) scheduling, or other factors (such as the 365th day falling on a holiday or weekend) outside of the Office of Defense Cooperation’s control could impact the exact date on which inventories were scheduled. If, for example, an item was inspected in February 2017 and again in February 2018, our analysis would consider it timely regardless of the actual date of inspection. We considered items that were unavailable for inspection due to deployment, repair, and security conditions to be accounted for because they were unavailable for legitimate reasons. If the Office of Defense Cooperation recorded an observation showing that an item was unavailable for legitimate reasons within 12 months of the last observation, we considered that observation to be timely. Because the SCIP dataset we analyzed does not include the date an item was returned from deployment or repair, we determined that a reinspection was timely if it was conducted within 4 months of the observation indicating the item was unavailable for inspection. We used a 4-month standard by examining the average and median length of time for a reinspection, which were 3.7 months and 3 months, respectively. The 4-month standard provides some time for an item to be returned before triggering the 90-day reinspection requirement. After we applied the 4-month standard, 367 observations, or about 3 percent of all observations we analyzed for timeliness, were still considered delinquent because they had not been reinspected within 4 months. On average, the items that were considered delinquent under this standard were reinspected about 8 months after they were considered delinquent, or about 12 months after the last attempted visit, indicating, on average, that these items were not inspected again until the next annual cycle. Because we do not know the date on which an item was returned, however, our analysis may slightly over-count delinquencies resulting from an item being unavailable for inspection. Our analysis only examined the time between recorded observations. Therefore, it did not count any items that were delinquent as of the end of 2018 if no observation had been recorded. The SCIP dataset includes no observation for 609 items in 2018. Of these, 117 were disposed of, lost, or expended in combat prior to 2018, 476 were observed in 2017 but delinquent as of the end of 2018, and 16 items were last observed before 2017. Additionally, due to data limitations, we did not analyze whether the first observation for each item was timely. Because our data set started in 2013, we did not have data on the date of the last observation for items delivered prior to 2013. DOD’s standards also state that DOD officials must first inspect items requiring enhanced end-use monitoring within 90 days of the item’s delivery. However, the SCIP data we analyzed only included the date the item was entered into SCIP, rather than the item’s delivery date, so we could not analyze whether the first inspection for items delivered after 2013 was timely. We did not include these observations in our analysis of timeliness. We conducted logical tests of the SCIP data, interviewed knowledgeable DOD officials about the database, and discussed our analysis with DSCA and ODC officials. We determined the data were sufficiently reliable for our purposes. We traveled to Tripoli, Lebanon and visited three LAF facilities to observe DOD procedures for conducting end-use monitoring and to see how the LAF safeguarded the equipment provided to them. We observed DOD’s enhanced end-use monitoring process for the 271 items in these three locations. We conducted this performance audit from October 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Elizabeth Field, (202) 512-2775 or fielde1@gao.gov. In addition to the contact named above, Biza Repko (Assistant Director), Kara Marshall (Analyst-in-Charge), Adam Brooks, Lisa G. Shibata, Aldo Salerno, Neil Doherty, Martin de Alteriis, and Ashley Alley made key contributions to this report.", "summary": "Since 2013, State and DOD have obligated nearly $1.5 billion in assistance to support Lebanese security forces. U.S. support for Lebanon is complicated due to the prominent role in the country of Hizballah, an Iranian-backed terrorist organization, which retains considerable influence as a major political party and a militia. The U.S. support includes equipment and training to build the capacity of Lebanese security forces. The equipment provided is subject to end-use monitoring requirements, which seek to ensure items are properly accounted for in Lebanon's inventory. GAO was asked to review U.S. security assistance provided to Lebanon since 2013. For fiscal years 2013 through 2018, this report (1) examines to what extent State and DOD assessed progress toward achieving strategic objectives related to security; (2) describes safeguards to limit the risk of U.S. assistance benefitting terrorist organizations; and (3) evaluates State and DOD end-use monitoring checks of equipment provided to Lebanese security forces. GAO analyzed State and DOD reports, documents, and data; and interviewed officials in Washington, D.C. and Beirut, Lebanon. The Departments of State (State) and Defense (DOD) reported progress in meeting security objectives in Lebanon, but gaps in performance information limit their ability to fully assess the results of security-related activities. State and DOD report improvements in Lebanese security forces' capabilities in key areas, such as border security. As part of monitoring such improvements and assessing the performance of security activities in Lebanon, State created related indicators but has not established targets for all of these indicators. Furthermore, State's data were incomplete for 11 of the 15 indicators GAO analyzed. For example, performance data for three indicators did not identify the number or percentage of people who received security training, as called for by the indicator. Without addressing these gaps, State has limited ability to determine to what extent it is achieving the intended results of its security-related activities in Lebanon. State and DOD use two primary safeguards to limit the risk of terrorist organizations benefitting from U.S. assistance to Lebanon. First, State routinely reviews the leadership of the Lebanese military and police forces and has determined they are not controlled by a foreign terrorist organization. Second, State and DOD vet potential trainees to ensure they do not have known or suspected ties to terrorism. Consistent with end-use monitoring requirements, State and DOD conducted required inventory checks of equipment provided to Lebanese security forces, but DOD did not meet its timeliness standards for nearly one-third of its observations. According to DOD officials, the method DOD uses to determine when it should complete annual inspections does not consider the date of the equipment's last inspection, which results in some inspections taking longer than prescribed by DOD's timeliness standards. Without conducting checks in a timely manner, DOD cannot fully ensure the equipment is properly accounted for and safeguarded. GAO is making three recommendations: 1) State should establish, as appropriate, and consolidate targets for its security-related performance indicators; 2) State should collect complete performance data for security-related indicators; and 3) DOD should revise its approach for determining when end-use monitoring inspections are to be completed to consider the date of last inspection. State and DOD concurred with these recommendations.", "document_type": "gao"}
{"report": "SEC has five Commissioners who oversee its operations and provide final approval over staff interpretation of federal securities laws, proposals for new or amended rules to govern securities markets, and enforcement activities. Headed by the SEC Chairman, the Commissioners oversee five divisions, 24 offices, and 11 regional offices. As shown in figure 1, SEC has designated four offices and five divisions as mission-critical (i.e., primarily responsible for implementing SEC’s mission). Table 1 outlines the roles and responsibilities of these mission-critical offices and divisions. The mission-critical offices and divisions are supported by other offices, such as the Office of Human Resources and the Office of Financial Management. SEC’s Office of Human Resources provides overall responsibility for the strategic management of SEC’s personnel management and assesses compliance with federal regulations for areas such as recruitment, retention, leadership and staff development, and performance management. In addition, certain divisions have internal human resource coordinators that coordinate between the Office of Human Resources and their respective division heads. The Office of Human Resources reports to SEC’s Office of the Chief Operating Officer, which in turn reports to the Office of the Chairman. The Office of Financial Management administers the financial management and budget functions of SEC. The Office assists the Chief Operating Officer in formulating budget and authorization requests, monitors the use of agency resources, and develops, oversees, and maintains SEC financial systems. To carry out its mission, SEC employs staff with a range of skills and backgrounds throughout the United States. As of September 2019, SEC employed 4,369 staff. Of these, approximately 69 percent were designated as mission-critical, and the remaining 31 percent were other professional, technical, administrative, and clerical staff. As shown in figure 2, the largest mission-critical occupational category is attorneys, who make up over 50 percent of all mission-critical employees. In addition, over 40 percent of all mission-critical employees work in one of SEC’s 11 regional offices. The regional offices are responsible for investigating and litigating potential violations of securities laws. The regional offices also have enforcement and examination staff to inspect regulated entities. SEC staff are represented by the National Treasury Employees Union (which we refer to in this report as the SEC employees’ union). To help SEC attract and retain qualified employees, in 2002 Congress enacted the Investor and Capital Markets Fee Relief Act (Pay Parity Act), which allowed SEC to implement a new compensation system with higher pay scales, comparable to those of other federal financial regulators. To stay within its annual appropriation, SEC imposed a hiring freeze beginning on October 1, 2016, and lifted it on April 1, 2019. During the hiring freeze, SEC permitted some exceptions on a case-by-case basis to fill positions that it determined to be critical to meeting key agency objectives and maintaining critical programs. Based on SEC’s budget justification documents, from October 1, 2016, through September 30, 2018, SEC lost a net total of 476 positions agency-wide, including 363 positions across its mission-critical offices and divisions. Figure 3 shows the staffing levels in SEC’s mission-critical offices and divisions during fiscal years 2016, 2017, and 2018. The results of our 2019 survey of mission-critical nonexecutive SEC employees indicate that most employees had positive views on some aspects of SEC’s personnel management and organizational culture, such as the skills of their direct supervisors and colleagues. Our survey results also indicate that employees had concerns related to SEC’s performance management system, perceptions of a risk-averse culture, and perceptions of favoritism in hiring and promotions. Employees had mixed views in other areas, such as morale, communication, and training. Finally, employees’ responses to key questions on organizational culture in our 2019 survey generally remained consistent with the results from our 2016 survey. See appendix III for a comparison of our 2016 and 2019 survey results for selected questions. Based on the results of our survey of mission-critical nonexecutive employees, we estimate that more than 75 percent of employees had favorable views of their direct supervisors in areas such as their skills and expertise, how they share information, and their willingness to listen to differing approaches (see fig. 4). In addition, we estimate that 70 percent of employees agreed that supervisors and managers in their division or office tolerate honest mistakes as learning experiences, and 68 percent agreed that supervisors and managers in their division or office are genuinely interested in the opinions of their staff. Similarly, in OPM’s 2018 Federal Employee Viewpoint Survey (hereafter referred to as OPM’s 2018 survey), SEC employees expressed positive views about their supervisors. In that survey, more than 80 percent of SEC employees agreed that they have trust and confidence in their supervisor (83 percent) and that their supervisor listens to what they have to say (88 percent) and treats them with respect (90 percent). Our survey results also indicate that most employees had positive views about the people SEC hires. As shown in figure 5, we estimate that 79 percent of employees agreed that their division or office is able to attract talented and qualified employees. We also estimate that 75 percent agreed that SEC management usually hires employees who are a good fit for SEC’s mission. In addition, in OPM’s 2018 survey, an estimated 90 percent of all employees agreed that SEC’s workforce has the job- relevant knowledge and skills necessary to accomplish the organization’s goals. For OPM’s 2018 survey of SEC employees, employees responded positively to questions related to their satisfaction with SEC as a place to work. Based on that survey, SEC’s overall score on OPM’s Global Satisfaction Index—which measures employee satisfaction with job, pay, and their organization—was 82 percent, while the government-wide score was 64 percent. In addition, SEC’s score on OPM’s Employee Engagement Index—which measures employees’ perceptions of leadership, interpersonal relationships between workers and supervisors, and employees’ feelings of motivation and competency related to their roles in the workplace—was 78 percent (compared to 68 percent government-wide). Moreover, from OPM’s 2013 survey to the 2018 survey, SEC’s scores improved in both of these categories by more than 15 percentage points, indicating that employees’ views are improving over time. More than 40 percent of employees expressed dissatisfaction with key aspects of SEC’s performance management system. As discussed later in this report, at the time of our survey, SEC employees covered by the union’s bargaining unit were rated under a pilot performance management system in which they received an initial four-tier rating, which was converted into a final two-tier rating of acceptable or unacceptable. Our survey results indicated areas of dissatisfaction with this system, as shown in figure 6. For example, based on our survey, we estimate that 48 percent of employees disagreed that the performance management system created meaningful distinctions in performance among employees. Similarly, in OPM’s 2018 survey, employees also expressed concerns about various aspects of performance management. For example, an estimated 33 percent of employees disagreed that their work unit takes steps to deal with poor performers, and 35 percent disagreed that differences in performance are recognized in a meaningful way. Our survey indicated that more than 40 percent of SEC employees continued to have concerns about excessive risk aversion—the condition in which the agency’s ability to function effectively is hindered by the fear of taking on risk. We estimate that 47 percent of nonsupervisors and 48 percent of supervisors agreed that the fear of public scandal has made SEC overly cautious and risk averse. These results were similar to our 2016 survey (46 percent of nonsupervisors and 49 percent of supervisors agreed), which were an improvement from the results of our 2013 survey. In addition, as shown in figure 7, about 40 percent of SEC employees agreed that the fear of being wrong makes senior officers in their division or office reluctant to take a stand on important issues. As we reported in 2013, changes to organizational culture, including reducing excessive risk aversion, require sustained efforts by senior management. Responses to other questions on our survey suggest that managers support the types of activities that may help reduce excessive risk aversion. For example, an estimated 60 percent of employees agreed that innovative ideas are encouraged in their division or office. Also, as noted above, we estimate that 70 percent of employees agreed that their supervisors and managers tolerate honest mistakes as learning experiences. Our survey results suggest that a quarter of employees had concerns about favoritism in SEC’s hiring process, and more than a third had such concerns about its promotion process. With respect to hiring, we estimate that 25 percent of employees agreed that hiring is sometimes based more on personal connections than on substantive experience and qualifications. With respect to promotions, as shown in figure 8, we estimate that 35 percent of nonsupervisory staff disagreed that promotion to management is based more on substantive experience than on favoritism and that favoritism is not an issue in promotions. A lack of clarity in the hiring and promotion processes may have contributed to employees’ perceptions related to favoritism. Based on our survey results, an estimated 50 percent of employees disagreed that the criteria for rewarding and promoting staff are clearly defined. Later in this report we discuss the steps SEC has taken to improve its promotion and hiring policies. While OPM’s 2018 survey results indicated that SEC employees largely had positive views about SEC as a place to work, the results of our 2019 survey of mission-critical nonexecutive employees indicate that the recent hiring freeze may have negatively impacted their views on morale. Based on our survey, we estimate that 37 percent of employees disagreed that morale is generally high most of the time, as shown in figure 9. In addition, based on our survey, we estimate that 63 percent of employees believed the recent hiring freeze had a negative impact on morale, including 31 percent who believed the negative effect was large. Over 60 SEC employees provided written survey comments related to morale. Some employees who provided written comments cited other concerns that had a negative impact on morale. For example, some employees stated that low pay increases and the lack of merit pay have contributed to low morale among high-performing employees. Some employees also noted that the 2019 government shutdown had a negative impact on morale by implying that federal employees’ work is not valuable. Most employees expressed positive views on whether cross-divisional communication is encouraged, but employees in some offices and divisions had concerns about communication within their division or office. Specifically, an estimated 66 percent of employees agreed that communication with other divisions and offices on work-related matters is encouraged. These survey results are generally consistent with SEC’s results on OPM’s 2018 survey, in which an estimated 73 percent of employees agreed that managers support collaboration across work units to accomplish work objectives, and an estimated 69 percent agreed that managers promote communication among different work units. However, in our survey, we found that some employees had more negative views about communication within divisions and offices. For example, we estimate that 34 percent of employees disagreed that information and knowledge are openly shared at all levels within their division or office, and 27 percent of employees disagreed that SEC management ensures employees are included in the flow of relevant information. As shown in figure 10, these figures were highest for employees in the Division of Corporation Finance and the Office of Information Technology. Most SEC employees expressed positive views on SEC’s commitment to training and the extent to which their training provided the skills and experience to meet SEC’s needs (see fig. 11). However, our survey results indicated heightened concerns about the number of training opportunities with outside instructors in some divisions and offices. While we estimate that 76 percent of employees reported that there were opportunities to participate in training that provided the latest industry-specific knowledge with outside instructors, we estimate that more than 30 percent of employees in several offices and divisions indicated that the number of such opportunities was less than adequate (see fig. 12). These concerns were highest in the Office of Information Technology, where more than half of the staff viewed such training opportunities as less than adequate. We administered a separate survey to 80 SEC senior officers in mission- critical offices and divisions, and 50 provided responses. Respondents generally had favorable views on issues such as hiring and retaining talent, communication, training, and morale. For example, 90 percent of senior officers we surveyed said their division or office is able to attract talented and qualified employees and that information is adequately shared across groups in their division or office. In addition, 82 percent agreed that morale is generally high most of the time. However, similar to nonexecutive employees, senior officers expressed concern about SEC’s performance management system. For example, 70 percent disagreed that current performance incentives were effective tools to motivate employees to perform well, and 50 percent disagreed that SEC’s performance management system provides consistent standards for rewarding performance. Since 2013, SEC has twice redesigned its performance management system without periodically validating it, as we recommended in 2013. Validating the system typically refers to obtaining staff input and general agreement on the competencies, rating procedures, and other aspects of the system. In our 2013 report, we found that SEC’s performance management system reflected many elements of OPM’s guidance but that implementation of the system could be improved. Also, consistent with best practices, we recommended that SEC conduct periodic validations, with staff input, of the performance management system and make changes as appropriate based on these validations. SEC agreed with our recommendation. In fiscal year 2016, SEC began to pilot a new performance management system with a four-tier rating scale. According to SEC officials, the four- tier rating system for non-bargaining-unit employees was fully implemented in 2017 and continued as a pilot in fiscal years 2017, 2018, and 2019 for bargaining unit employees. However, SEC did not validate this system. In our 2016 report, we reiterated the importance of our 2013 recommendation and emphasized that SEC should only make changes to its performance management system based on validations and staff feedback. Despite plans to survey all employees to validate the agency’s pilot performance management system and obtain employee feedback in fiscal years 2017 and 2018, SEC officials said they have been unable to do so, in part because they could not reach agreement with the SEC employee union on the planned survey questions. SEC and the union agreed in November 2018 that SEC will implement another new performance management system, including a new incentive bonus program, in 2020. Because SEC did not validate the four-tier system it was piloting, it missed an opportunity to obtain employee input to inform the design of the new system. Under the new system, all SEC employees will be evaluated on a two-tier rating scale: “accomplished performer” and “unacceptable.” In addition, SEC plans to implement a new incentive bonus program that will provide opportunities for high- performing employees to earn a bonus of up to $10,000 once per fiscal year. According to SEC officials, SEC plans to work with OPM to validate the new performance management system by surveying staff on the new system at the conclusion of the 2020 appraisal period, after which OPM will submit a final assessment of the program with any recommended actions for SEC. These plans are consistent with our 2013 recommendation that SEC should conduct periodic validations of its performance management system. However, until SEC completes its planned activities, this recommendation remains unaddressed. The negative views expressed by many employees in our survey underscore the need for SEC to validate its performance management system. As discussed earlier, more than 40 percent of employees were dissatisfied with key aspects of SEC’s performance management system, such as the extent to which the performance management system created meaningful distinctions in performance among employees. In addition, based on our survey, we estimate that 30 percent of SEC employees disagreed that SEC’s performance management system uses relevant criteria to evaluate their performance. Validating the new performance management system with staff input should help SEC better ensure that it is achieving its goals and identify any changes needed to address employee dissatisfaction with performance management. In prior work, we reported that effective performance management requires that the organization’s leadership make meaningful distinctions between acceptable and outstanding performance of individuals and appropriately reward those who perform at the highest level. In addition, our prior work on strategies federal agencies can use to manage performance-oriented pay systems has shown the need for agencies to build in safeguards to enhance transparency and ensure the fairness of pay decisions. One such safeguard is to include multiple levels of review of performance ratings and pay decisions to ensure consistency and fairness in the process and the resulting decisions. Another safeguard is to publish aggregate data on the results of the performance cycle, which allows employees to compare results across various groups within the agency while protecting the confidentiality of individual ratings and pay decisions. SEC has not yet developed mechanisms for transparency and fairness for its new performance incentive bonus program. Under the program, a supervisor may nominate an employee who demonstrates exceptional performance according to certain criteria to receive a bonus payment of up to $10,000 once per fiscal year. SEC officials told us that specific policies and procedures for the bonus program were still being developed at the time of our review, but they could not provide details on how they planned to ensure transparency and fairness in implementing the program. Moreover, as of November 2019, SEC had not provided detailed policies and procedures, nor had it established a date by which such policies and procedures would be finalized, despite its goal of implementing the new program in January 2020. Developing and implementing adequate safeguards could increase employees’ confidence in the new performance incentive bonus program. Without adequate safeguards to enhance transparency and better ensure fairness, employee dissatisfaction with performance management may persist and could undermine the credibility of the new bonus program. SEC has taken action to fully implement the two recommendations from our 2013 report related to developing and implementing a comprehensive workforce and succession planning process that is consistent with OPM guidance. In our 2016 report, we found that SEC had developed a workforce and succession plan in response to these recommendations. However, we identified weaknesses with this plan, such as the lack of a comprehensive skills gap analysis to help ensure that employees across all occupations have the skills necessary to fulfill SEC’s mission. Since our 2016 review, SEC completed a more comprehensive skills gap analysis and began to implement new workforce and succession planning processes that address other weaknesses we had identified. In fiscal year 2019, SEC developed and began to implement a new workforce planning strategy that outlined new processes for workforce and succession planning. SEC’s previous process focused on creating a consolidated workforce plan in a single document that focused on five divisions and two offices, accounting for 67 percent of SEC employees. SEC officials told us that the new process is more dynamic and responsive because it provides more workforce data to officials in the divisions and offices. Specifically, SEC developed various human capital dashboards that provide the Office of Human Resources and agency leaders with up-to-date data on the state of the agency’s workforce, such as data on hiring, attrition, skill gaps, and other workforce demographics. Key components of SEC’s new workforce and succession planning processes address weaknesses identified in our prior work: Skills gap analysis. Our 2016 review found that SEC’s workforce plan lacked a comprehensive skills gap analysis covering all SEC occupations. In 2018, SEC conducted an agency-wide competency survey to identify skills gaps by position in each division and office. SEC incorporated the results of this survey into one of its human capital dashboards that allows users to interact with the data directly. Specifically, SEC’s Workforce Competency Dashboard provides competency data (including gaps) across offices and divisions, allowing users to explore critical skill gaps by competency. According to SEC’s workforce planning strategy, divisions and offices can use the data to address skill gaps through activities such as training, hiring, and knowledge sharing. For example, to address an identified gap in written communication and critical thinking for newly hired investigative attorneys, the Division of Enforcement and the Office of Human Resources developed interview questions to better screen for these skills during the hiring process. Human capital reviews. We also found in 2016 that SEC’s workforce plan was not clearly linked to its budget formulation and did not inform decision-making about the structure of the workforce. Under its new workforce planning process, SEC links its workforce planning to its budget through annual human capital reviews in which divisions and offices work with the Office of Human Resources to identify workforce needs and priorities to directly inform their operating plans and budget requests. These human capital reviews include discussions about the capacity and capability of the organization to meet current mission needs and whether areas of the workforce need to be reshaped to meet SEC’s mission. SEC officials told us that under SEC’s previous workforce planning process, these reviews were conducted concurrently with budget meetings, whereas under its new process these meetings are conducted prior to the budget meetings. This change allows divisions and offices to use the information from the review meetings to prepare for their budget meetings. In addition, the human capital review meetings are informed by data maintained in SEC’s new Workforce Supply Dashboard, which provides information on the composition and demographics of SEC divisions and offices and allows users to view data on hiring, attrition, and other workforce indicators. For example, through this process, SEC recently determined that it had an excess of certain positions, such as clerks and assistants responsible for data processing and management. This determination led SEC to request permission from OPM for a targeted early retirement authority and incentives for individuals in such positions. New succession planning processes. In 2016, we found that SEC’s succession planning lacked information on workforce attrition and a fair and accurate process for identifying future leaders. Under SEC’s new succession planning process, the Office of Human Resources tracks senior-level turnover to determine the level of attrition at senior leadership levels and to determine whether SEC is filling these positions internally or externally. In addition, the Office of Human Resources created a standardized template that managers in each division and office use to identify key leadership positions and candidate pools. According to SEC, this more standardized approach offers an extra level of precision and rigor to identify the specific leadership strengths and risks across the largest divisions and offices. In addition, since our 2016 report, SEC has improved processes for analyzing its talent pool for new leaders. In 2017, the Office of Human Resources surveyed employees to gauge their interest and intent in progressing to higher levels of management responsibility, including to the senior officer ranks. SEC is also developing a centralized program to screen and select a cohort of high-potential leaders who will be certified and available to fill senior officer positions as they become vacant. SEC officials said they anticipate the program will be launched in the second half of fiscal year 2020. The processes and tools described above are still new, and SEC is continuing to integrate and develop them fully. For example, 2019 was the first year SEC used its new workforce planning process, and SEC officials told us that senior officers are still learning how they can best use new tools, such as the new human capital dashboards. One SEC official told us that SEC is still refining this new approach and plans to consider additional enhancements to the dashboards, such as including more forward-looking data to inform discussions of future workforce needs. Although SEC continues to enhance its new process and practices, the actions it has taken fully implement our two 2013 recommendations. SEC has taken steps to improve certain practices related to hiring and promotions. For example, in 2016, we found that SEC had not identified skills gaps among its hiring specialists and that these staff received limited training. As a result, SEC lacked assurance that its hiring specialists had the necessary skills to hire and promote the most qualified applicants. We recommended that SEC develop and implement training for hiring specialists that is informed by a skills gap analysis. In response to our recommendation, SEC’s Talent Acquisition Group partnered with SEC’s training group to conduct a competency gap assessment for each of the Talent Acquisition Group’s five primary jobs. Based on the results of this competency assessment, in 2018 SEC developed and prioritized a 2-year training plan for hiring specialists to address the identified skills gaps and to better enable SEC to recruit, develop, and retain competent staff. This skills gap analysis and the new training curriculum for hiring specialists fully address our 2016 recommendation. SEC also made changes to policies for promotion announcements to improve perceptions of fairness and transparency. For example, since 2016, a promotion opportunity can be limited to applicants within a single division or office only if that division or office has at least 15 eligible candidates. If there are fewer than 15, the announcement must be opened more broadly to candidates in SEC beyond that particular office or division. In addition, SEC now requires that promotion announcements be open for a minimum of 10 business days. SEC has fully addressed recommendations we made in 2013 and 2016 to improve intra-agency communication and collaboration: Incentives for staff to communicate and collaborate. In 2013, we found that SEC had made efforts to improve communication and collaboration but had not fully addressed barriers to an environment of open communication. We recommended that the SEC Chief Operating Officer identify and implement incentives for all staff to support an environment of open communication and collaboration. We determined that this recommendation had been fully implemented in November 2017. Among other steps, in 2016 SEC revised its performance expectations for supervisors to encourage communication and collaboration and proactively share relevant information. Best practices for communication and collaboration. In 2013, we recommended that SEC explore communication and collaboration best practices and implement those that could benefit SEC. SEC has taken action to fully implement this recommendation. Specifically, SEC’s Office of the Chief Operating Officer engaged a third-party management consultant team to complete a study of best practices for communication and collaboration, which was completed in 2018. For the study, the consultants developed a framework of best practices recognized in the public and private sectors and assessed SEC’s practices against the framework. The consultants found that each of the best practices in its framework was met by at least one of SEC’s activities, tools, technologies, or initiatives. The report included eight recommendations to help address barriers to cross-division communication and collaboration, among other goals. In response to these recommendations, as of May 2019, SEC had taken action on six recommendations and developed planned actions for the remaining two. For example, to facilitate staff-to- staff communication and collaboration, SEC officials updated the intranet sites of each mission-critical office and division with main contact telephone numbers and staff directories. In addition, SEC plans to pilot an electronic communication tool for project execution among teams collaborating across divisions and offices that will provide more functionality than SEC’s current application. Cross-divisional committees and working groups. In 2016, we noted that the lack of a central position or office with authority over the daily operations of all divisions and offices made it difficult to address challenges related to communication and collaboration. We recommended that SEC enhance or expand the responsibilities and authority of the Chief Operating Officer or another official or office to help ensure that improvements to communication and collaboration across SEC were made. While SEC disagreed with this recommendation, it has taken actions that meet the intent of our recommendation. First, SEC created cross-divisional committees and working groups that help to enhance intra-agency communication and collaboration. For example, in 2018, SEC created an Operations Steering Committee, which consists of senior operational leaders throughout the agency who meet on a regular basis to discuss and collaborate on cross-agency operational issues, including those related to human capital. SEC also created other formal intra-agency committees and working groups, including an Information Technology Capital Planning Committee, an Emerging Risk Group, and a Data Management Board. Second, between 2009 and 2018, SEC established Managing Executive positions in the Office of the Chairman and in eight of its nine mission- critical offices and divisions. Managing Executives are responsible for working closely with one another, including serving together on intra- agency working groups, to facilitate effective internal collaboration on operations issues, including personnel management. The Managing Executive in the Office of the Chairman, established in 2017, acts as a liaison between the Chairman’s office and the various committees and working groups. According to an agency official, having a Managing Executive position in the Office of the Chairman helps ensure that someone from the Chairman’s office has the time to devote to operational issues. SEC has taken a number of actions since 2016 to strengthen its personnel management. It has implemented a more comprehensive approach to workforce planning and improved intra-agency communication and collaboration through new working groups and implementation of best practices. OPM’s 2018 employee survey also suggests that employee satisfaction at SEC has improved. Despite this progress, SEC has yet to validate its performance management system since we recommended it do so in 2013. Without such validation, SEC may lack information that could help it identify changes needed to address employee dissatisfaction and ensure its system achieves its goals. We therefore reiterate our 2013 recommendation that SEC conduct periodic validations, with staff input, of the performance management system and make changes as appropriate based on these validations. Consistent with our recommendation, SEC officials stated they plan to work with OPM to validate the new performance management system. However, until SEC completes its validation of the new system, which it plans to do at the conclusion of the 2020 appraisal period, this recommendation remains unaddressed. Finally, a key feature of SEC’s new performance management system will be a performance incentive bonus program through which SEC supervisors will be able to nominate individual employees for a bonus of up to $10,000 once per fiscal year. Our prior work on performance management has highlighted the importance of safeguards that can help ensure that agencies’ performance management systems—and particularly the systems affecting pay—are fair and transparent. At the time of our review, SEC was in the process of designing the performance incentive bonus program and did not provide us with detailed policies or procedures. As SEC works to finalize procedures for this bonus program, incorporating safeguards such as multiple levels of review of performance ratings and pay decisions can help to promote employee confidence in the integrity of the program. The Chair of the Securities and Exchange Commission should direct the Chief Operating Officer to develop and implement safeguards to better ensure transparency and fairness in SEC’s new performance incentive bonus program. Such safeguards could include multiple levels of review of performance ratings and pay decisions and publishing aggregate data on the results of the performance cycle that allow employees to compare results across various groups within the agency while protecting the confidentiality of individual ratings and pay decisions. (Recommendation 1) We provided SEC a draft of this report for its review and comment. SEC provided written comments that are reprinted in appendix VI. SEC also provided technical comments that we incorporated, as appropriate. In its written comments, SEC stated that it concurred with, and plans to implement, our recommendation to develop and implement safeguards to better ensure transparency and fairness in its new performance incentive bonus program. SEC stated that it appreciated our suggested practices, and that it will conduct research to consider additional safeguards. SEC also highlighted its implementation of eight of nine of our previous recommendations related to personnel management. SEC noted its progress in the areas of workforce planning and intra-agency communication and recognized that further work remains to be done. With respect to our 2013 recommendation that it conduct periodic validations of the performance management system, which SEC has not yet implemented, SEC stated that it expects to obtain feedback from employees and managers at the conclusion of the 2020 performance cycle to identify further improvements, and that it is committed to conducting periodic evaluations of its system in the future. We will continue to monitor SEC’s progress toward implementing this recommendation. We are sending copies of this report to the appropriate congressional committees, the Chairman of the Securities and Exchange Commission, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at 202-512-8678 or clementsm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. Table 2 provides the status of recommendations we made to the Securities and Exchange Commission in 2013 and 2016. This report examines (1) employees’ views on the Securities and Exchange Commission’s (SEC) personnel management and organizational culture, (2) SEC’s efforts to implement a performance management system, (3) SEC’s implementation of a workforce planning process, and (4) SEC’s steps to strengthen communication and collaboration within and across its divisions and offices. To examine employees’ views on SEC’s personnel management and organizational culture, we conducted two surveys of SEC staff, performed a content analysis of open-ended responses to our surveys, and conducted individual interviews. Surveys. To obtain employees’ views on SEC’s personnel management and organizational culture, we implemented two web-based surveys from March 2019 to May 2019. We administered the first survey to a stratified random sample of 877 nonexecutive employees in mission-critical occupations in mission-critical offices and divisions. We administered the second survey to all 80 senior officers in mission-critical offices and divisions. To determine our sample of nonexecutive employees, we stratified the population of mission-critical SEC employees into sampling strata by office and division to help mitigate the risk that a particular part of SEC could be over- or underrepresented by the respondents to our survey. We stratified the Division of Enforcement and the Office of Compliance Inspections and Examinations into two further categories (“headquarters” and “regional office”) because this division and office have a majority of their staff located in one of SEC’s 11 regional offices. Table 3 shows the total number of employees and the number of employees selected in our sample for each of the strata. Due to their small employee counts, we combined the Offices of the Chief Accountant and Credit Ratings into one stratum for the purpose of selecting the sample. Prior to selecting the sample, we sorted the sample frame by supervisory status within each stratum. We then selected the sample via systematic random sampling within each stratum. Our initial sample size allocation was designed to achieve a stratum-level margin of error no greater than plus or minus 8 percentage points at the 95 percent level of confidence. Based upon our prior surveys on SEC’s personnel management, we assumed a response rate of 70 percent to determine the sample size for the mission-critical employees. Because some employees left SEC between the time we obtained a list of SEC employees and the launch of the survey, the final sample size decreased from 884 to 877. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval. This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. We provide confidence intervals along with each sample estimate in the report. All survey results presented in the body of this report are generalizable to the estimated population of 2,907 in-scope mission-critical employees at SEC as of September 30, 2018. For our survey of nonexecutive employees in the mission-critical offices and divisions, 563 nonsupervisors and supervisors responded to our survey, for a response rate of 64 percent. For our survey of all mission- critical senior officers, 50 responded to our survey, for a response rate of 63 percent. For the nonexecutive survey, we carried out a statistical nonresponse bias analysis using available administrative data and determined that the results are generalizable to SEC’s mission-critical employees. We do not attempt to extrapolate the findings of our senior officer survey to those who chose not to participate. Each GAO survey of SEC staff included questions on personnel management issues related to (1) recruitment, training, staff development, and resources; (2) communication among and within divisions and offices; (3) leadership and management; (4) performance management and promotions; and (5) organizational culture and climate. The separate survey of all mission-critical SEC senior officers (those at the SO-1, SO-2, and SO-3 pay grades) covered the same topic areas but omitted questions not relevant to senior officers and included additional questions specifically relevant to senior officers. Our surveys included both multiple-choice and open-ended questions. We analyzed the results of our 2019 survey of supervisory and nonsupervisory staff and senior officers, and we compared the results to results of similar surveys we conducted in 2013 and 2016. In addition, we reviewed the Office of Personnel Management’s (OPM) 2018 Federal Employee Viewpoint Survey results to obtain additional perspectives from SEC staff on issues related to the agency’s personnel management and to compare SEC’s results to government-wide responses. To minimize certain types of errors, commonly referred to as nonsampling errors, and enhance data quality, we employed recognized survey design practices in the development of the questionnaires and the collection, processing, and analysis of the survey data. To develop our survey questions, we drew on prior GAO SEC personnel management surveys. For both of our 2019 surveys, we took steps to ensure that survey questions from 2016 were still relevant and to determine if new issues warranted new questions. To do this, we reviewed information from individual interviews with current and former employees, met with five mission-critical employees to pretest the nonexecutive survey, and met with two senior officers to obtain their feedback on the senior officer survey. As a result of these meetings, for example, we added three questions related to the impact of SEC’s hiring freeze on personnel management. In addition, a GAO survey expert reviewed and provided feedback on our survey instrument. To reduce nonresponse, another source of nonsampling error, we sent multiple emails encouraging SEC employees to complete the surveys, and we made telephone calls to nonrespondents to encourage participation and troubleshoot any logistical issues in accessing the questionnaire. We also had respondents complete questionnaires online to eliminate errors associated with manual data entry. On the basis of our application of these practices and follow-up procedures, we determined that the survey data were of sufficient quality for the purpose of obtaining employees’ views on SEC’s personnel management and organizational culture. Content analysis. To analyze the information we obtained from the open-ended survey responses, we conducted a content analysis on the 633 responses to the six open-ended survey questions from the survey of the mission-critical offices and divisions. Five staff members developed coding categories based on our researchable objectives, information collected during our individual interviews, and the findings from our December 2016 report. Coding categories were as follows: (1) workforce management, (2) communication, (3) management, (4) promotions, (5) performance management, and (6) risk aversion. For each of the responses to the six open-ended questions, a GAO analyst categorized the response into the respective coding categories. A second GAO analyst reviewed the coding, and any disagreements in the coding were resolved through discussion or with a third analyst. Individual interviews. We interviewed 51 nonsupervisory and supervisory employees—in person at SEC headquarters and by telephone for those in headquarters and regional offices—in November and December 2018 to obtain their views on personnel management at SEC. Using information provided by SEC, we sent 577 letters to all employees who separated from SEC between March 2016 and November 2018, offering them an opportunity to schedule a meeting with us. We interviewed 15 of these former SEC employees by phone in January and February 2019. We asked certain questions of every person we interviewed related to (1) what personnel management practices were working well, (2) what challenges existed in personnel management, and (3) what initiatives, if any, SEC had taken to address these challenges. To maintain the confidentiality of individual responses, we did not record individual names in our transcripts. Instead, we collected and analyzed the information by division and rank only, and we aggregated our findings so that no individual comments could be identified. GAO analysts summarized themes that emerged from these individual interviews and used them to identify key issues related to SEC’s personnel management and inform the design of our surveys. To obtain information on SEC’s efforts related to performance management, workforce planning, and communication and collaboration, we reviewed relevant SEC documents and interviewed SEC officials in the Office of Human Resources and other divisions and offices. We reviewed changes SEC made to its personnel management practices since our 2016 review, including steps taken to address our recommendations in these areas. We interviewed SEC staff from the Office of Human Resources about the status of SEC’s efforts to pilot and implement a performance management system, including the status of SEC’s efforts to address our 2013 recommendation that SEC conduct periodic validations of its performance management system and make changes, as appropriate, based on these validations. We also reviewed documents describing changes to SEC’s performance management system. At the time of our review, SEC had plans to implement a new performance management system, including a new incentive bonus program, in January 2020 but had not yet completed detailed policies and procedures to implement this new system. However, we compared the system’s key features with criteria identified in prior GAO work, including work on strategies federal agencies can use for fair and transparent performance management. In addition, we reviewed the SEC Office of Inspector General’s 2018 report that described progress and challenges in the agency’s performance management efforts. To examine SEC’s workforce and succession planning practices, we obtained and reviewed a copy of SEC’s fiscal year 2019–2022 Workforce and Succession Planning Strategy, which outlines new approaches to workforce and succession planning that SEC began to implement in fiscal year 2019. We also obtained and reviewed documentation of SEC’s implementation of key steps in its workforce and succession planning processes, such as the survey instrument used to identify skill gaps for all SEC occupations, slide presentations of SEC divisions’ operating plans and budget requests that are informed by human capital review meetings, examples of action plans SEC divisions and offices developed to address identified skill gaps, SEC’s Succession Planning Tool Kit, and relevant training plans for SEC divisions. In addition, we attended an SEC-led demonstration of the agency’s new human capital dashboards, which are interactive software tools that provide the Office of Human Resources and agency leaders with up-to- date data on the state of the agency’s workforce, such as data on hiring, attrition, skill gaps, and other workforce demographics. We also interviewed staff from SEC’s Office of Human Resources and senior leaders from different SEC divisions. We compared SEC’s workforce planning process against key principles for effective workforce planning, and we assessed SEC’s efforts to strengthen its workforce and succession planning efforts to determine the extent to which they addressed our 2013 recommendations related to developing a more comprehensive approach to workforce and succession planning. This assessment included reviewing the extent to which key components of SEC’s workforce and succession planning processes aligned with OPM standards on workforce and succession planning. In addition, we reviewed the changes SEC made to its hiring and promotion policies since our last review, including the steps SEC took to address our 2016 recommendation related to developing and implementing training for hiring specialists that is informed by a skill gap analysis. To examine steps SEC has taken to strengthen intra-agency communication and collaboration, we assessed SEC’s efforts to address prior recommendations in this area. Specifically, we reviewed a report by a third-party vendor on communication and collaboration practices at the agency and met with the vendor’s program manager. We also obtained and reviewed documentation of SEC’s actions to implement recommendations included in the vendor’s report. In addition, we reviewed documentation related to SEC’s cross-divisional committees and working groups, including the charter of SEC’s Operations Steering Committee, a cross-agency group chaired by the Chief Operating Officer whose purpose is to facilitate predecisional communications on significant cross-agency operational issues. To obtain information on the effectiveness of SEC’s efforts to enhance communication and collaboration, we also met with senior leaders from SEC’s largest offices and divisions, as well as selected members of SEC’s Operations Steering Committee. We assessed the reliability of all of the data we used during this review and determined they were sufficiently reliable for the purposes of selecting our survey sample; developing summary tables on staffing ratios and turnover; and describing trends and views on personnel management practices at SEC. We used SEC data extracted from the Department of the Interior’s Federal Personnel/Payroll System to construct the sample frames for our two surveys and develop summary tables in our appendixes. To determine the reliability of these data, we reviewed related documentation, tested the data for missing data and errors, and obtained written responses from SEC employees about data quality and control. To assess the reliability of the Federal Employee Viewpoint Survey data, we reviewed technical documentation of the survey and conducted routine data checks. We conducted this performance audit from August 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Figure 13 below shows the results of eight questions related to personnel management and organizational culture from our 2013, 2016, and 2019 surveys of Securities and Exchange Commission (SEC) employees in mission-critical occupations in mission-critical divisions and offices. However, there are important limitations in comparing the results of our 2019 survey to the previous surveys. First, while the results of our 2019 survey were generalizable to all mission-critical nonexecutive employees, the results of our 2013 and 2016 surveys were not. Second, while we present the results for mission-critical employees for each year, for our 2019 survey, we changed the definition of mission-critical to reflect changes SEC had made to its mission-critical designations. The divisions, offices, and occupational categories largely remained the same across the 3 survey years with the following exceptions: for our 2019 survey, the Offices of Information Technology, Credit Ratings, and the Chief Accountant were added to the category of mission-critical offices and divisions. In addition, financial analysts were removed and information technology specialists were added to our list of mission-critical occupations. Third, while we administered the 2019 survey to a representative sample of mission-critical employees, we administered our 2013 and 2016 surveys to all mission-critical employees. As such, we present our 2019 results as estimated percentages with bands representing the range of results within a 95 percent confidence interval. Finally, when comparing our 2019 results on these eight questions to the 2016 survey results, we found that employees’ views on these questions were generally within the confidence intervals of the 2019 results. In these cases, we cannot conclude whether the changes are statistically significant. Overall, employees’ views on whether there is an atmosphere of trust improved since our 2016 survey. Nonsupervisory employees’ views on whether the criteria for promotion are clearly defined and whether information is adequately shared across groups in their division or office also improved. However, for the remaining survey questions, we could not conclude whether employees’ views improved or worsened because changes in employees’ views were within the confidence intervals or were only seen on either the “agree” or “disagree” side of the survey scale, not both. Section 962 of the Dodd-Frank Wall Street Reform and Consumer Protection Act included a provision for us to review whether there is an “excessive number of low-level, mid-level, or senior-level managers” at the Securities and Exchange Commission (SEC). We did not identify any standards that have been established for evaluating excessive numbers of supervisors. Therefore, we are reporting on the ratio of SEC employees at the various levels for fiscal years 2008 through 2018 in mission-critical offices and divisions. Table 4 illustrates the ratio of nonsupervisors to supervisors at SEC. Table 5 illustrates the ratio of nonsupervisors to senior officers, and table 6 illustrates the ratio of supervisors to senior officers. Section 962 of the Dodd-Frank Wall Street Reform and Consumer Protection Act included a provision for us to review turnover rates within Securities and Exchange Commission (SEC) subunits. While staff turnover rates could be used to identify potential areas for improvement and further develop current supervisors, turnover may not be a good indicator of poor supervision for several reasons. For example, staff may leave to pursue opportunities with a different employer or a different career path, or for personal reasons. Tables 7 and 8 show the percentage of staff who left SEC from fiscal years 2008 through 2018 from headquarters and the 11 regional offices, respectively. Table 9 shows the total number of staff who left SEC during the same period. In addition to the contact above, John Fisher (Assistant Director), Charlene J. Lindsay (Analyst-in-Charge), Grzegorz (Greg) Borecki, Carl Barden, Pamela Davidson, Jill Lacey, Marc Molino, Kirsten Noethen, Shannon Smith, Jennifer Schwartz, Benjamin Wiener, and Jason Wildhagen made key contributions to this report.", "summary": "The Dodd-Frank Wall Street Reform and Consumer Protection Act contains a provision for GAO to report triennially on SEC's personnel management. GAO's first two reports ( GAO-13-621 and GAO-17-65 ) identified a number of challenges and included nine recommendations. This report examines (1) employees' views on SEC's personnel management, (2) SEC's performance management system, (3) SEC's steps to improve its workforce planning processes, and (4) SEC's efforts to improve communication and collaboration. GAO surveyed a representative sample of nonexecutive SEC employees in key occupations and all senior officers in nine key divisions and offices (with response rates of 64 and 63 percent, respectively). The results of the nonexecutive employee survey are generalizable to SEC's mission-critical employees. GAO also followed up on prior recommendations, reviewed SEC documents and personnel management practices, analyzed SEC workforce data, and interviewed SEC officials. Securities and Exchange Commission (SEC) employees in the five divisions and four offices GAO surveyed expressed positive views on some aspects of SEC's personnel management but reported concerns in other areas. For example, employees GAO surveyed generally had positive views on their direct supervisors and colleagues—81 percent of nonexecutive employees agreed that their direct supervisors had the skills and expertise to be effective managers. However, more than one-third of employees expressed concerns in areas such as performance management and favoritism. For example, 48 percent of nonexecutives disagreed that the performance management system in place at the time of GAO's review created meaningful distinctions in performance. SEC has implemented eight of GAO's nine recommendations related to personnel management. However, SEC has not yet implemented a 2013 GAO recommendation to validate its performance management system—that is, to obtain staff input and agreement on the competencies, rating procedures, and other key aspects of the system. SEC plans to implement a new system in 2020, and validating this system would help ensure that it achieves its goals and identify changes needed to address employee dissatisfaction with performance management. In addition, a key feature of SEC's new performance management system will be a bonus program through which supervisors can nominate high-performing employees for a bonus of up to $10,000 once per fiscal year. However, SEC has not yet developed mechanisms for transparency and fairness for this new bonus program. GAO has previously highlighted the need for safeguards to better ensure fairness and transparency in performance management, particularly around systems affecting pay. Incorporating safeguards into the new bonus program—such as including multiple levels of review and publishing aggregate data on award decisions—would promote transparency and could increase employee confidence in the program. Since GAO's most recent review in 2016, SEC has taken actions to implement a more comprehensive workforce planning process and strengthen intra-agency communication and collaboration. For example, SEC conducted a comprehensive analysis to identify skills gaps in its workforce. It also improved the link between its budget formulation process and annual meetings in which the Office of Human Resources consults with each division and office on its workforce needs and priorities. Additionally, to strengthen communication and collaboration, SEC commissioned a study to identify relevant best practices and created formal mechanisms, such as working groups, to enhance collaboration across divisions and offices. For example, in 2018, SEC created its Operations Steering Committee through which senior operational leaders throughout the agency periodically meet to coordinate on cross-agency operational issues, including those related to human capital. SEC should develop and implement safeguards to better ensure transparency and fairness in its new incentive bonus program. SEC agreed with this recommendation. GAO also reiterates its recommendation in GAO-13-621 that SEC conduct periodic validations (with staff input) of the performance management system and make changes, as appropriate, based on these validations. SEC stated that it expects to take action on this recommendation at the end of the 2020 performance cycle.", "document_type": "gao"}
{"report": "Federal agency IT systems provide essential services that are critical to the health, economy, and defense of the nation. However, federal agencies increasingly rely on aging legacy systems that can be costly to maintain. As we previously reported in May 2016, our review of federal legacy systems found that 26 federal agencies reported spending almost $61 billion on operations and maintenance costs in fiscal year 2015. In addition, many of the government’s IT investments used hardware parts that were unsupported and outdated software languages, such as the common business oriented language (COBOL). In some cases, this lack of vendor support created security vulnerabilities and additional costs because these known vulnerabilities were either technically difficult or prohibitively expensive to address. Congress enacted the MGT Act in December 2017 and established the TMF to help agencies improve, retire, or replace existing systems. Congress appropriates money to the TMF, which is used to fund projects approved by the board. As of August 2019, Congress had appropriated $125 million to the TMF—$100 million was appropriated in fiscal year 2018 and $25 million in fiscal year 2019. The MGT Act assigns specific responsibilities to OMB, GSA, and the Technology Modernization Board for the fund’s administration and also assigns responsibilities to federal agencies that received awarded funds. Among other things, OMB. The act requires the Director of OMB to issue guidance on the administration of the fund and report the status of the awarded projects on a public website. The information reported is to include a description of the project, project status (including any schedule delay and cost overruns), financial expenditure data related to the project, and the extent to which the project is using commercial products and services. GSA. The act designates the Administrator of General Services with responsibility for administering the fund. This includes, among other things: (1) providing direct technical support in the form of personnel services and other services; (2) assisting the Technology Modernization Board with the evaluation, prioritization, and development of agency modernization proposals; and (3) performing regular project oversight and monitoring of approved agency modernization projects. In March 2018, GSA established a TMF Program Management Office within the agency to manage these functions. An executive director leads the office and reports to the Office of the Deputy Administrator within GSA. The act requires the Administrator of General Services, in consultation with the Director of OMB, to establish administrative fees at levels sufficient to ensure the solvency of the fund in order to help offset GSA’s operating expenses for these functions. Agencies pay fees if they receive funding for a project. Technology Modernization Board. The board has responsibility for providing input to the Director of OMB for the development of processes for agencies to submit proposals, making recommendations to the Administrator of GSA to help agencies refine their submitted proposals, and reviewing and prioritizing submitted proposals. The board also is responsible for recommending the funding of modernization projects to the Administrator of GSA, and for monitoring the progress and performance of approved projects. In addition, the board is tasked with monitoring the operating costs of the fund. As part of its oversight of awarded projects, the board requires each project to present a quarterly update and report on the status of milestones achieved in order to ensure the project is on schedule. Other federal agencies. The act stated that any agency that submits an IT-related project proposal and receives TMF funding must repay the transferred amount as well as pay an administrative fee. After the board approves a project proposal, the respective agency is required to sign an interagency agreement with the TMF Program Management Office that specifies the terms of the TMF funding repayment, the administrative fee, and the repayment schedule before initial funds are disbursed and the project begins. Figure 1 provides an overview of key TMF activities that OMB, GSA, and the Technology Modernization Board have undertaken to meet the responsibilities outlined in the MGT Act. These include the establishment of TMF administrative processes and the Technology Modernization Board’s project award announcements, among other activities. These activities are also discussed in greater detail following the figure. In February 2018, OMB issued guidance on the implementation of the MGT Act that included instructions for agencies on submitting applications for TMF funding. Agencies were allowed to begin submitting initial application proposals on February 27, 2018. The guidance included an initial application template that agencies were required to complete. As part of the template, agencies were required to provide an estimate of the TMF funding request and the agency’s method used for cost estimation. Subsequently, in March 2018, OMB issued funding guidelines for projects receiving awards. The guidelines stated that project proposals must include a reliable estimate of any project-related cost savings or avoidance relative to pre-modernization activities using the templates provided. In addition, the guidelines stated that estimates must undergo appropriate due diligence and concurrence from the requesting agency’s Office of the Chief Financial Officer prior to submission to the board, in consultation with OMB’s Resource Management Office and GSA’s TMF Program Management Office. Further, the guidelines stated that the agency’s estimation process would be subject to GAO review, pursuant to the act. For agencies receiving a TMF award, the guidelines stated that agencies were required to repay all transferred funds as well as an administrative fee, which was determined based on the amount of awarded funding. As part of the process, agencies were required to establish a written agreement with GSA that set forth the terms for repaying the transferred funds and the administrative fee. Agencies were required to start making payments one year after the initial amount of award funding was transferred and complete all payments within five years, unless otherwise approved by OMB. While the guidelines noted that reimbursement was not contingent upon the achievement of project-related savings, agencies could use the project’s generated cost savings to repay the award. The TMF application process occurs in two phases, each of which requires agencies to submit specific documents. During Phase 1, agencies are required to submit an initial project proposal providing preliminary information about the project, its purpose, and its anticipated benefits. Within this documentation, agencies must confirm that funding for this project has never explicitly been denied or restricted by Congress or OMB, in accordance with the MGT Act. Also during this phase, the Technology Modernization Board evaluates proposals and makes recommendations for project funding based on how well the project documentation demonstrates a strong execution strategy, technical approach, and includes a strong team with a demonstrated history of successful modernization efforts. The board encourages agencies to consider the adoption of commercial technology solutions in their proposals and present a strong technical approach and acquisition strategy to implement those solutions. In addition, agencies are encouraged to provide information on the potential impact of the modernization effort on the agency’s mission, feasibility, opportunity enablement (e.g. cost savings), and common solutions. If the board approves the Phase 1 initial project proposal, the project team will move on to Phase 2. In Phase 2, the agency must submit a financial plan showing a cost estimate and estimated savings from the implementation of the proposed project. Agencies must provide a more comprehensive project description than that provided in Phase 1, including discrete milestones, funding schedule, project plan, and financial plan. These documents must be approved by the agencies’ chief financial officer and CIO. Phase 2 proposals must also address any other areas identified by the board in the initial project review. Further, the agency proposal team must also prepare an in-person presentation for the board. OMB’s Resource Management Office reviews the proposal documentation to ensure that the proposed project aligns with the requesting agency’s mission. The office’s review is intended to ensure that the proposal does not duplicate funding provided through existing appropriations, or previously has been expressly denied funding or restricted by Congress. The review includes an assessment of the proposed project’s information on the reimbursement of the awarded funds, the project’s planned schedule, and out-year budget impacts. OMB also reported that the agency sends information on the proposed projects to Congressional appropriation committees for their review prior to the Technology Modernization Board’s approval of a project. Agencies with projects that the board recommends for TMF funding are required to sign an interagency agreement outlining the repayment terms. In addition, projects receive incremental funding contingent on the successful execution of milestones outlined in the written agreement for the transfer of funds. Figure 2 describes the steps in both phases of the TMF proposal process. As of August 2019, the Technology Modernization Board had awarded $89.36 million to seven projects. Table 1 lists the projects that have received funding (in alphabetical order by agency), descriptions of the projects, and when the TMF funding awards were announced. For more details on each of the awarded projects, see appendix II. OMB Circular A-11 directs agencies to follow the guidelines outlined in its appendix on cost estimating for all IT investments and acquisitions within the federal government. Since OMB first introduced its cost estimate appendix in 2006, as noted in the circular, the cost estimating appendix has been based on the GAO Cost Estimating and Assessment Guide. The appendix outlines a number of major steps in the cost estimating process and references the practices in GAO’s cost guide. Specifically, these steps include preparing a high-level work breakdown structure, defining ground rules and assumptions, developing the data by collecting information on the cost drivers, developing the estimate using various risk factors, performing a sensitivity analysis, documenting the estimate, and updating it on a regular basis. According to the GAO guidance, a cost estimate is considered reliable if it meets four characteristics and the specific set of best practices associated with each characteristic. Those characteristics are: Comprehensive: An estimate should include all life cycle costs (from the program’s inception and design through operations and maintenance), reflect the current schedule, and have enough detail to ensure that cost elements are not omitted or double counted. Specifically, the cost estimate should be based on a product-oriented work breakdown structure that allows a program to track cost and schedule by defined deliverables, such as hardware or software components. In addition, all cost-influencing ground rules and assumptions should be detailed in the estimate’s documentation. Well-documented: An estimate should be thoroughly documented, describe how it was developed; and include source data, clearly detailed calculations and results, and explanations of why particular estimating methods and references were chosen. Data should be traced to their source documents. Accurate: An estimate should be based on historical data or actual experiences on other comparable programs and an assessment of most likely costs, and be adjusted properly for inflation. In addition, the estimate should be updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it should always reflect the current status. Credible: An estimate should discuss any limitations of the analysis because of uncertainty surrounding data or assumptions. In addition, the estimate should incorporate the results of a sensitivity analysis (that examine the effects of changing assumptions on the estimate), and risk and uncertainty analysis (that identifies all of the potential project risks and assesses how these might affect the cost estimate). The estimate’s results should be cross-checked, and an independent cost estimate should be conducted to see whether other estimation methods produce similar results. If any of the characteristics is not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high- quality estimate and cannot be considered reliable. Federal agencies are generally required to use full and open competition to award contracts for the procurement of goods and services (including commercial IT products), with certain exceptions. The Competition in Contracting Act of 1984 requires agencies to obtain full and open competition through the use of competitive procedures in their procurement activities unless otherwise authorized by law. Using competitive procedures to award contracts means that all prospective contractors that meet certain criteria are permitted to submit proposals. While the Competition in Contracting Act generally requires federal agencies to award contracts using full and open competition, agencies are allowed to award contracts noncompetitively under certain circumstances. Generally, these awards must be supported by written justifications that address the specific exception to full and open competition that is being used in the procurement. An example of an allowable exception to full and open competition includes circumstances when the contractor is the only source and no other supplies or services will satisfy agency requirements. Federal agencies have the option to use a variety of contract types when purchasing IT products and services, including government-wide acquisition contracts, IT Schedule 70 contracts, and blanket purchase agreements. These contracts and agreements allow agencies to establish a group of prequalified contractors to compete for future orders under streamlined ordering procedures once agencies determine their specific needs. Agencies can then issue orders on these contracts and agreements, obligating funds and authorizing work to be performed. Agencies are required to publicly report their contract transactions in the FPDS-NG database. This contract transaction data includes information on the type of award made, the amount of the award, and whether competitive procedures were used. Specifically, agencies are required to identify the extent to which the contract was competed and what solicitation procedures were used. In addition, if an agency awards task orders on an existing contract, then the agency is required to identify whether competitive procedures were used. Further, if the contract did not use competitive procedures, then the agency is required to report the reason that the contract was not competed. As of August 31, 2019, GSA’s TMF Program Management Office had obligated about $1.2 million in operating costs for activities related to the establishment and oversight of the fund. While the office intended to assess administrative fees to fully recover its operating expenses, the actual amounts collected as of August 2019 had been less than planned. This was due to factors such as the office’s formulation of fee rates based on appropriations levels that were higher than what was ultimately received, along with changes to several projects’ scope and milestones. Further, cost savings have yet to be realized. Officials from the seven TMF-funded projects reported that they expect to begin realizing cost savings from their projects starting in fiscal year 2020 or later. According to the MGT Act, the TMF Program Management Office may obligate funds to cover its operating expenses out of the appropriations received for the fund (totaling $125 million as of August 2019) in order to provide support to the Technology Modernization Board in meeting its responsibilities. To help offset TMF operating expenses, the act required that the GSA administrator, in consultation with the OMB director, to establish administrative fees at levels sufficient to ensure the solvency of the fund (so that obligations or transfers of funds to awarded projects never exceed the amount available in the fund for these obligations or award transfers). Subsequent OMB guidance, issued in March 2018, required TMF- awarded projects to pay an administrative fee on awarded funds, beginning the first year after the initial incremental amount of award funding was transferred to the agency. The TMF Program Management Office issued further guidance in June 2018 that established administrative fee rates based on a percentage of the amount transferred to an agency project and the payment period. During the time of our review, the office’s current administrative rate was for the period from July 2018 through September 2019. The fee rates were set in June 2018 with the intent to operate the fund as a full cost recovery model, meaning that the Program Management Office planned to fully recover all operating expenses through administrative fee collection by fiscal year 2029 if the office’s assumptions regarding appropriation levels and project selections were met. The office’s reported intention is to help preserve the capital of the fund, which would maximize the amount of appropriations available for award. Table 2 outlines the rates for TMF administrative fees based on the number of years to repay the awarded funds and the percentage of the transferred amount, for the period of July 2018 through September 2019. The TMF Program Management Office sets new rates annually after review from the Technology Modernization Board and approval by GSA’s Deputy Administrator; these rates go into effect in October of each year. As of August 31, 2019, the TMF Program Management Office had obligated about $1.2 million to cover its operating expenses and had begun to collect administrative fees from agency projects, consistent with the MGT Act. Specifically, from March 2018 (when the office began operations) through August 31, 2019, the office obligated approximately $409,000 in fiscal year 2018 and $797,000 for the first 11 months of fiscal year 2019. During the same period, the office collected $33,165 in administrative fees as of August 31, 2019. Based on this amount, the fund was able to only offset approximately 3 percent of its obligated operating costs as of August 31, 2019. The TMF Program Management Office’s administrative fee collection has been limited due to a number of factors that have affected the amounts scheduled to be collected: (1) no fees were collected in the first year of operation; (2) projects chose longer periods to make payments; (3) projects make payments based on funds transferred; (4) fee rates were determined based on assumptions regarding appropriations that were not met; and (5) project changes may affect fee collection. No fees were collected during the first year of operation. OMB’s funding guidelines allowed agencies to start paying administrative fees one year after a project received an award. Since the Technology Modernization Board began awarding funding in June 2018 (within fiscal year 2018), no projects were required to start paying administrative fees until fiscal year 2019, which deferred the start of the TMF Program Management Office’s fee collection by one year. Projects chose longer periods to make payments. When the TMF Program Management Office set administrative fee rates, agencies receiving awards were allowed to determine what rate they would pay according to how many years they planned to make payments. The office reported that a lower administrative fee rate was offered to projects that chose to repay awarded funds over a shorter period (3 years) rather than 5 years. All seven projects that have been awarded funding as of August 31, 2019, chose the longer repayment period of 5 years with a 3 percent rate. The Executive Director of the TMF Program Management Office reported that the office offered a lower administrative rate with the intent of making repaid funds available more quickly to be awarded to new projects. In doing so, the Technology Modernization Board expected to be able to make additional awards, which would increase the collection of administrative fees. Further, according to the Executive Director, the office did not expect that the agencies’ selection of a 5-year repayment term instead of a 3-year term to significantly affect the performance of the fund. However, as the Executive Director noted, these longer repayment terms do affect the collection of administrative fee payments because a longer repayment term means that these funds are not as readily available to award to new projects and generate new fees. Projects make payments based on funds transferred. Agencies receiving awards were only required to make administrative fee payments based on the amount of the award funding that was transferred, rather than based on the full awarded amount. As such, this reduced the amount of fees that the TMF Program Management Office could collect in the initial years that agencies made fee payments. As of August 31, 2019, the Technology Modernization Board had authorized the transfer of $37.65 million (of the $89.36 million awarded) to the seven projects. Based on the amounts transferred, the office is scheduled to collect $1.2 million in administrative fees through 2025 from the seven projects. Table 3 shows the current scheduled administrative fee payments that will be collected from the seven projects based on the amount of awarded funding that the projects had received as of August 31, 2019. Going forward, as the seven projects receive all of the remaining awarded funds, the projects are planning to pay a total of $2.68 million in administrative fees through 2025. However, the Technology Modernization Board had not made awards to any additional projects as of August 2019, and, as a result, the office will not likely be able to collect any additional fees from new projects until at least fiscal year 2021. Any newly awarded projects would be eligible to delay paying administrative fees until 1 year after the initial award date in accordance with the funding guidelines. Fee rates were determined based on assumptions regarding appropriations that were not met. The TMF Program Management Office set its current administrative fee rates in June 2018 based on the assumption that the fund would receive higher levels of appropriations than what was ultimately received. In doing so, the office projected that it would transfer more funds to projects, which would result in larger administrative rates over the initial years of the fund. Specifically, GSA requested $438 million in its fiscal year 2018 and 2019 budget requests for the TMF, but actually received $125 million in appropriations. Table 4 lists the amounts that GSA requested in its budget requests and the amounts appropriated for fiscal years 2018 through 2020. In making its June 2018 assumptions about the appropriations, the office projected that it would distribute larger amounts of funds in the first 2 years of operation and collect more administrative fees through fiscal year 2025. However, the office’s projected collection of administrative fees is less than what was scheduled as of the end of August 2019. In particular, while the office exceeded its projections for distributing funds in fiscal year 2018 ($1.93 million more than projected), the office had not yet met its projection of distributing $75 million in fiscal year 2019—specifically, as of August 31, 2019, the office had distributed only $25.71 million to awarded projects. Consequently, these lower levels of distributed funds decreased the amount of administrative fees scheduled to be collected. Table 5 shows the TMF Program Management Office’s projections for fund distribution for fiscal years 2018 through 2019 and its projected fee collection, compared to the current scheduled distributions and administrative fee collection for fiscal years 2018 through 2025, as of August 31, 2019. Going forward, the office had projected that it would distribute $75 million in fiscal year 2020. However, based on our analysis, only approximately $35.6 million was available in the fund as of August 31, 2019, to award to new projects. The Executive Director of the TMF Program Management Office stated that the office had to make assumptions about the TMF appropriation levels in order to develop the rate model. In doing so, all of the underlying assumptions and parameters related to determining the administrative fee rates and ensuring the fund operated at full cost recovery were reviewed by GSA’s Office of the Chief Financial Officer and Office of General Counsel, OMB, and the Technology Modernization Board before the GSA Deputy Administrator approved the fee rates in June 2018. In addition, the Executive Director noted that, at the time the rate model was developed, the office did not yet have information on the fiscal year 2019 appropriations and made the assumption that the fund would receive the same level of appropriations as in fiscal year 2018 ($100 million). However, based on the wide gap between the budget requests and what funds were ultimately appropriated in fiscal years 2018 and 2019, these assumptions regarding fund appropriation levels did not materialize and impacted the amount of fees that could be collected from awarded projects in fiscal year 2019. Four projects’ changes will affect fee collection. As of August 31, 2019, officials responsible for the management of four of the seven TMF- funded projects reported that they were planning to make significant changes to their projects’ approved scope or scheduled milestones. Officials from two projects reported that they had received approval for these scope changes from the Technology Modernization Board (in June 2019 and August 2019, respectively) and are currently waiting on approval for the repayment schedule changes as of August 31, 2019. Officials from the other two projects reported in August 2019 that they planned to present their changes to the board for approval. Based on our analysis, these changes are expected to affect the four projects’ administrative fee repayment schedules and reduce two projects’ administrative fee collection by $369,117. Table 6 lists the changes to the four TMF-funded projects as of August 31, 2019, as reported by the agencies; the status of the Technology Modernization Board’s approval of the changes; and the potential impacts these changes are expected to have on administrative fee collection. In addition, more details on the changes reported by the four projects are included in appendix II. The Executive Director of the TMF Program Management Office stated that the four projects’ reduction or delay in administrative fee payments should not affect administrative fee collection. The Executive Director explained that the return of prior awarded funds will allow the Technology Modernization Board to have more funds available to award to new projects, which would generate new administrative fees. However, these proposed changes to the four projects’ scope and schedule likely will affect upcoming administrative fee collection because additional time will be needed to review new project proposals. In addition, the agencies may delay administrative fee payments for one year after award issuance. As a result of the five factors that we identified that had impacted administrative fee collection as of August 2019, there is likely to be a period of time between when the office’s current administrative fee collection occurs and when the office can recover its operating expenses from this collection. Specifically, based on our analysis, it will take the TMF Program Management Office at least 5 years (until 2024) to recover the operating costs expended as of August 31, 2019, (over $1.2 million) with the current collection of administrative fees. In addition, once the two projects’ proposed scope and schedule changes are approved by the Technology Modernization Board (decreasing fees collected by $369,117), it is likely that the office will take longer than 5 years to recover these operating costs. Further, it is not clear when the TMF Program Management Office will recover future operating expenses incurred in fiscal year 2020 and beyond. Moreover, these factors will most likely continue to be a challenge for OMB and the office going forward if newly awarded projects choose longer repayment periods or more awarded projects make changes that affect fee collection. Consequently, OMB and the TMF Program Management Office are not currently on track to operate the fund at full cost recovery, as intended. The Executive Director of the TMF Program Management Office stated that the office had reduced its fiscal year 2019 operating expenses by almost 50 percent from the original planned operating level (in the fiscal year 2019 President’s Budget). In particular, the Executive Director reported that the office used temporary staff internally to deliver administrative and support activities, such as website updates and the preparation of meeting agendas and minutes, rather than rely on contractors. The office added that, using internal temporary employees had provided the office with the flexibility to scale operations up and down as appropriate. As of August 2019, the office was not pursuing a staff increase. Further, the Executive Director stated that, as of August 2019, the office was reassessing the assumptions for the administrative fee rate model for the upcoming year, including assumptions for fiscal year 2020 appropriations. The Executive Director added that the office would like to have more information on its fiscal year 2020 appropriations in order to help determine the new rate. These assumptions would be used to develop a new rate model that is expected to go into effect on October 1, 2019, for fiscal year 2020. As for the office’s ability to manage the fund at full cost recovery, the Executive Director stated that all of the assumptions would have needed to be met in order to ensure the TMF operated with full cost recovery. The Executive Director added that the office still intends to pursue full cost recovery going forward, but noted that this may change if the new set of assumptions is not met. Further, the Executive Director reported that four project proposals were in draft stages or pending a Technology Modernization Board determination as of August 2019. Since the fund was established in December 2017, OMB, the Technology Modernization Board, and the TMF Program Management Office have provided oversight of the fund’s awarded projects by requiring the respective agencies to provide quarterly updates on the status of project milestones and transferring additional funds only when milestones were reached. However, the board had not made a corresponding effort to ensure that the TMF Program Management Office’s operating costs and administrative fee collection remained on track to achieve full cost recovery as intended. In addition, the office’s plan to take 12 years—from the start of operations in fiscal year 2018 until fiscal year 2029—to fully recover its operating costs hinders GSA’s ability to maximize the amount of appropriations available for award due to the length of time necessary to recover its costs. As a result, as of August 2019, OMB and the TMF Program Management Office were not on track to recover all operating expenses related to fund administration and oversight, thereby leaving less of the fund’s capital available for project awards. The TMF Program Management Office’s authorized collection of administrative fees is intended to allow the office to offset expenses, which maximizes the amount of funding that can be awarded to projects. However, given the lower-than-expected collection of these administrative fees and the office’s lengthy time frame for recovering all costs, it may be prudent to review those fees and determine whether their rates are set appropriately. Unless OMB and the TMF Program Management Office take steps to develop a plan that outlines the actions needed to fully recover TMF operating expenses with administrative fee collection in a timely manner, there will be fewer funds available to award to projects that are intended to improve the efficiency and effectiveness of government IT systems. The MGT Act established the TMF to help improve, retire, or replace federal IT systems with more efficient and effective systems that would cost less money to operate and maintain. As part of its selection criteria, the Technology Modernization Board stated that the agency would need to clearly demonstrate in its proposal how the proposed project would generate cost savings or how the modernization of the system would dramatically improve the quality of service provided. In addition, OMB’s funding guidelines stated that the project proposal must include a reliable estimate of any project-related cost savings or avoidance using the templates provided. Agencies were required to identify what year their project would start to realize cost savings in the TMF application after receiving an award (the earliest year savings could begin to be realized was fiscal year 2019). Further, the guidelines stated that the agency’s estimation process would be subject to GAO review, pursuant to the act. As of August 31, 2019, officials responsible for project management for each of the seven TMF-funded projects reported that their projects had not yet begun to realize cost savings because either the project was still currently being implemented or the project had experienced changes to prior projections. Specifically, officials for four of the seven projects reported that their projects were currently meeting targeted milestones for implementation and would begin to realize cost savings starting in fiscal year 2020 or later as planned. Officials for the other three projects reported that they had recently made changes to the projects’ scope and scheduled milestones that delayed when the projects would begin to realize savings. For more details on the changes reported by these three projects, see appendix II. Table 7 shows the year that each of the seven TMF-funded projects report that they would begin to realize cost savings. One of the three projects that experienced changes, Agriculture’s Infrastructure Optimization project, had originally planned to begin realizing cost savings starting in fiscal year 2020; however, project scope and milestone changes delayed the expected date for realization of this savings. Officials from Agriculture’s Infrastructure Optimization project reported in August 2019 that the new time frame for realizing cost savings remained to be determined. In addition, Energy’s Enterprise Cloud Email project had originally intended to begin realizing cost savings in 2021, but changes to the project’s scope have delayed the realization of savings until 2024. The third project, GSA’s NewPay, had originally planned to begin realizing savings in 2024, but changes to the project’s technological implementation have delayed the realization of savings. In particular, officials from GSA’s NewPay project reported that the project still anticipates realizing cost savings, but the date for these savings remains to be determined. Congress established the MGT Act and the TMF to help agencies transform their legacy IT systems to be more cost effective and efficient. As the awarded projects complete implementation efforts, it will be critical for agencies to realize cost savings from these modernization efforts in order to help ensure the fund is successful. OMB’s Circular A-11 directs agencies to follow the guidelines outlined in its appendix on cost estimating for all IT investments and acquisitions within the federal government. Since 2006, as noted in the circular, the cost estimating appendix has been based on the GAO Cost Estimating and Assessment Guide. As discussed earlier, the appendix outlines a number of major steps in the cost estimating process and references the practices in GAO’s cost guide. According to GAO’s guidance, a reliable estimate should meet the criteria for four characteristics and the specific set of best practices associated with each of the characteristics. The four characteristics and the specific best practices, among others, are: comprehensive – the estimate should include all life cycle costs, a work breakdown structure, and ground rules and assumptions; well-documented – the estimate documentation should describe how the source data were used, the calculations that were performed and their results, and the estimating methodology used; accurate – the estimate should be based on historical data or actual experiences on other comparable programs and be updated regularly to reflect changes in the program; and credible – the estimate should incorporate the results of sensitivity, and risk and uncertainty analyses. According to the GAO guidance, if the overall assessment rating for each of the four characteristics is not fully or substantially met, then the cost estimate cannot be considered reliable. Based on our analysis of the cost estimates for the seven TMF-funded projects, the reported savings estimates that were derived from those estimates cannot be considered reliable. Officials responsible for developing the cost estimates for each of the projects did not incorporate all of the best practices for a reliable cost estimate, as defined in the GAO guidance and OMB Circular A-11. Table 8 describes the four GAO cost estimating characteristics, key practices associated with each characteristic (and the major steps in OMB Circular A-11), and the results of our analysis of the seven TMF- funded projects’ cost estimates. In addition, appendix III provides more details on our individual assessments of the seven projects’ cost estimates. In assessing the reliability of the projects’ cost estimates, we found that the TMF Program Management Office did not provide written guidance for developing the cost estimates in a manner consistent with federal requirements outlined in Circular A-11 or our best practices. Specifically, the only guidance that the Technology Modernization Board provided on the TMF website was the instruction to submit a project cost estimate using a template developed by the Program Management Office, and approved by OMB and the Technology Modernization Board. While the template provided a means to report costs for the proposed projects, the template did not require agencies to follow any of the best practices outlined in GAO’s Cost Estimating and Assessment Guide, and which is referenced by Circular A-11. Further, there were no written instructions for the template regarding the data elements or the fields required to be completed. Agency officials responsible for developing the cost estimate for each of the seven projects all confirmed that they were instructed to use the project cost estimate template to report their projects’ cost and savings estimates. In addition, these officials acknowledged that they did not follow their own internal cost estimate development processes or GAO best practices when developing their estimates. The Executive Director of the TMF Program Management Office stated that the project teams were expected to follow their own internal investment management process for developing the cost estimates. Additionally, the agencies’ chief financial officers and CIOs were required to review and approve the project proposal applications, including the completed cost estimate templates, prior to the agencies’ submissions to the Technology Modernization Board. Further, the Executive Director acknowledged that written guidance had not been developed for completing the project proposal documentation. Instead, the Executive Director stated that the office had held meetings, as requested by each project team, to provide assistance on how to complete the cost estimate template. The Executive Director stated that these meetings enabled the project teams to ask targeted questions on how to complete the template for their individual projects, which enabled the office to provide specific assistance on completing the template for each project. Staff in OMB’s Office of E-Government and Information Technology stated that agencies are required to follow the requirements outlined in Circular A-11 regarding the development of a cost estimate for all IT investments. In addition, the staff noted that each proposal is required to be approved by the agency’s Chief Financial Officer and CIO before being submitted to the Technology Modernization Board. The staff added that the information regarding the guidance for completing the proposal documentation and cost estimates is available on the TMF website. However, our review of the documentation provided on the TMF website did not identify any guidance regarding the development of the cost estimate as part of the proposal—except a statement requiring the completion of the provided template. The website also did not include any guidance instructing the agencies to follow the requirements outlined in Circular A-11, which references GAO’s cost estimating guidance. As noted in GAO’s cost estimating guide, reliable cost estimates can provide management the data necessary to make informed investment decisions, measure program progress, proactively correct course when warranted, and ensure overall accountability for results. Having a realistic estimate of projected costs also helps to ensure that projected cost savings are reliable. Building such quality into a cost estimate is addressed by the steps described in Circular A-11 (that references the practices outlined in GAO’s cost guide). Regardless of whether or not agencies were told to do so, it is an agency’s responsibility to follow these steps. Ensuring agencies understand the requirements they are supposed to follow when developing a cost estimate for their TMF proposal is critical to the success of the proposal process. If OMB and GSA do not clarify the requirement that agencies follow Circular A-11’s cost estimating process (that references GAO’s cost estimating guidance discussed in this report), agencies are at risk of continuing to provide unreliable cost information in their proposals to the Technology Modernization Board. Further, absent detailed guidance from the TMF Program Management Office on how to complete the cost estimate template, including information on the data elements and the fields required to be completed, agencies are at risk of providing incomplete or insufficient information in their project proposals. As a result, the board may not have sufficiently reliable project cost and savings information with which to make decisions on potential awards and whether these projects offer appropriate value for the investment being requested. The MGT Act requires the Administrator of GSA to ensure that the use of commercial off-the-shelf products and services are incorporated to the greatest extent practicable in agency projects awarded funding through the TMF. As required under the Competition in Contracting Act of 1984, all procurements, with certain exceptions, must be competed as full and open so that any qualified entity can submit an offer. Agencies are also required to publicly report their contract transactions in the Federal Procurement Data System-Next Generation (FPDS-NG), including information on the type of award made and whether competitive procedures were used. In addition, if an agency issues task orders on an existing contract, then the agency is required to identify whether competitive procedures were used. Further, if the contract did not use competitive procedures, then the agency is required to report the reason that the contract was not competed. As of August 31, 2019, six of the seven TMF-funded projects had awarded 23 contracts or task orders for work on the projects. Agency officials responsible for management of the six funded projects reported that 22 of the 23 awards used full and open competitive procedures, which we confirmed using acquisition data from FPDS-NG. HUD officials reported that the remaining award was based on a sole source contract that was not competed and an exception was documented. One project had not yet made an award. Table 9 lists the seven TMF-funded projects and the agencies’ reported use of full and open competitive procedures in FPDS-NG for the related awards, as of August 31, 2019. In making the 22 awards, agency officials responsible for the management of the six funded projects reported that they had relied on existing IT service contracts and blanket purchase agreements, or had established new blanket purchase agreements for these projects. Specifically, 11 awards were based on task orders issued on existing contracts. 9 awards were based on orders from existing blanket purchase agreements. 2 awards were made on new blanket purchase agreements. In making these awards using existing contracts and blanket purchase agreements that had followed full and open competitive procedures, the agencies complied with the requirements for using competitive procedures. In those cases where the agencies used existing blanket purchase agreements, these orders were coded as competitive based on data reported in FPDS-NG. For the one award where competitive procedures were not used, HUD completed a justification and approval for other than full and open competition, indicating that only one responsible source and no other supplies or services would satisfy the agency’s requirements. HUD officials stated that they chose a sole source contract because they wanted to retain the expertise of the existing contractors and maintain cohesion between the different phases of project work. For the project that had not yet made an award, officials responsible for the management of Agriculture’s Infrastructure Optimization project reported that, due to a change in the scope of the project made in June 2019, no contracts had been awarded yet for work on the project. The officials reported that they anticipated making an award by the end of December 2019 and that the contract is to be awarded using competitive procedures. Agencies’ continued adherence to federal acquisition requirements for full and open competition should help ensure that their TMF-funded investments deliver the intended services to benefit both the agencies and the public. Since March 2018, when GSA established the TMF Program Management Office to administer fund operations, the office has obligated about $1.2 million to cover its expenses from managing the fund but has collected limited administrative fees to offset its expenses. As a result, the Technology Modernization Board has fewer funds than anticipated available to award to new projects. Going forward, OMB and the TMF Program Management Office are likely to face ongoing challenges in collecting administrative fees due to the factors that we have identified that affect fee collection and the office’s lengthy time frame for recovering all costs. While OMB and the TMF Program Management Office are not currently on track to recover all operating expenses in a timely manner, Program Management Office officials have expressed the intent to revisit their fee structure, in part to address the lower than anticipated amount of fiscal year 2019 appropriations. Because of the number of factors that are likely to affect fee collection, it will be critical that OMB and the TMF Program Management Office take steps to develop a plan that outlines the actions needed to fully recover TMF operating expenses with administrative fee collection in a timely manner in order to maximize the funds available for awards. By creating a new funding mechanism to help modernize federal IT systems, Congress intended that funds would be used to improve, retire, or replace existing federal IT systems to improve efficiency and effectiveness of these systems. However, since none of the seven TMF- funded projects’ cost savings estimates can be considered reliable, it is not clear whether the projects receiving funding to date will save the government as much money as was estimated. An important aspect to the success of the TMF will be clarifying the established requirement that agencies follow Circular A-11’s cost estimating process (that references GAO’s cost estimating guidance discussed in this report) in order to help ensure that the reliability of estimated savings for awarded projects is improved. We are making five recommendations: two to OMB and three to GSA. Specifically: The Director of OMB should develop and implement a plan with GSA that outlines the actions needed to fully recover the TMF Program Management Office’s operating expenses with administrative fee collection in a timely manner. (Recommendation 1) The Director of OMB should work with GSA to clarify the requirement in the TMF guidance that agencies follow the cost estimating process outlined in Circular A-11 (that references GAO’s cost estimating guidance discussed in this report), when developing the proposal cost estimate. (Recommendation 2) The Administrator of General Services should develop and implement a plan with OMB that outlines the actions needed to fully recover the TMF Program Management Office’s operating expenses with administrative fee collection in a timely manner. (Recommendation 3) The Administrator of General Services should work with OMB to clarify the requirement in the TMF guidance that agencies follow the cost estimating process outlined in Circular A-11 (that references GAO’s cost estimating guidance discussed in this report), when developing the proposal cost estimate. (Recommendation 4) The Administrator of General Services should develop detailed guidance for completing the Technology Modernization Fund project cost estimate template, including information on the data elements and the fields required to be completed, in order to help ensure the accuracy and completeness of the provided information. (Recommendation 5) We provided a draft of this report to OMB and the five agencies for their review and comment. In response, of the two agencies to which we made recommendations, GSA stated that it agreed with one recommendation and partially agreed with the remaining two recommendations; and OMB did not state whether it agreed or disagreed with the recommendations. In addition, of the four agencies to which we did not make recommendations, one agency stated that it concurred with information presented in the report, two other agencies stated that they had no comments on the report, and a fourth agency did not state whether it had comments on the report. Further, four agencies provided technical comments on the report, which we incorporated as appropriate. The following discusses the comments received from each agency to which we made recommendations. GSA provided written comments in which it agreed with our recommendation to develop detailed guidance for completing the TMF project cost estimate template. Additionally, the agency partially agreed with our recommendation to develop and implement a plan with OMB that outlines the actions needed to fully recover TMF operating costs with administrative fee collection, stating the agency had concerns with our discussion of this topic in the report. Among the concerns was that we clearly did not acknowledge that GSA is on track to meet the requirement codified in the statute to maintain the solvency of the fund. However, our report did not make a conclusion that the fund was insolvent, or that the fund was on track to being insolvent. Rather, we discussed the factors that have affected administrative fee collection to date. In our discussion, we noted that as a result of these factors, it will take the TMF Program Management Office at least 5 years (until 2024) to recover the operating expenses expended as of August 31, 2019 (over $1.2 million) with the current collection of administrative fees. Consequently, as of August 2019, OMB and the TMF Program Management Office were not on track to recovering all operating expenses in a timely manner, thereby hindering GSA’s ability to maximize the amount of appropriations available for award. As such, we continue to believe our assessment is accurate. GSA also had concerns that we did not state that the TMF Program Management Office’s goal of full cost recovery for operating expenses was over the lifetime of the fund. In our report, we discuss that the TMF Program Management Office planned to fully recover all operating expenses through administrative fee collection by fiscal year 2029. In doing so, we noted that the office’s plan to take 12 years to fully recover its costs hinders GSA’s ability to maximize the amount of appropriations available for award due to the length of time necessary to recover its costs. Therefore, we believe that we have sufficiently discussed the time frame GSA plans to take to fully recover its costs. Further, GSA stated that our discussion of the TMF Program Management Office’s operating costs would be improved if we noted the large percentage of fund administrative costs was devoted to salaries for a limited number of staff. In determining the cost of administering the TMF, we analyzed the costs of establishing and overseeing the TMF and evaluated the collection of administrative fees from projects awarded funding, consistent with the MGT Act. In doing so, we noted the steps taken by the TMF Program Management Office to reduce its operating expenses, including reducing costs by 50 percent for fiscal year 2019, and not pursuing a staff increase in fiscal year 2019. We did not analyze any individual operating expenses and therefore, have no basis to comment on current salary expenses and whether they could or could not be reduced. As such, we believe that we appropriately discuss the costs of establishing and overseeing the TMF and the relationship of those costs to the goal of fully recovering all operating expenses. Accordingly, we believe our recommendation to develop and implement a plan to fully recover office operating expenses with administrative fee collection is still warranted. The agency also partially agreed with our second recommendation to work with OMB to clarify the requirement in TMF guidance that agencies follow the federal cost estimating guidance discussed in this report. GSA stated that the agency does not set cost estimating policy requirements for agencies, as that is the responsibility of OMB and agency CIOs. In our report, we discuss the MGT Act’s requirement that the Administrator of GSA, in consultation with the CIO Council and with the approval of the Director of OMB, administer the TMF. Because the GSA Administrator has been designated responsibility for administering the fund, the agency has a role in clarifying what guidance agencies should follow when developing their cost estimates for the TMF proposal application. Further, we acknowledge GSA’s statement that the agency will commit to working with OMB and the Technology Modernization Board to identify necessary updates to the cost estimating guidance as a positive step towards addressing our recommendation. Consequently, we believe our recommendation for GSA to work with OMB to clarify the requirement in TMF guidance that agencies follow Circular A-11’s cost estimating process (that references GAO’s cost estimating guidance discussed in this report), when developing the proposal cost estimate, is still appropriate. GSA’s comments are reprinted in appendix IV. OMB provided written comments in which the agency did not state whether it agreed or disagreed with our recommendations; however, OMB stated that the agency remains concerned with the facts, characterizations, and opinions in the draft report. The agency further stated that the draft report contains many key assumptions and recommendations that are misleading and paints an incomplete picture of the TMF. OMB then stated that while we met with the agency twice during the course of the audit, we engaged with GSA multiple times in contrast. According to OMB, many of the questions we posed to GSA would have been better answered by OMB, whose authorities in the budget, apportionment, and approval process for TMF proposals could have enabled us to state items in the report with greater accuracy. In addition, the agency stated that many of its corrections and suggestions offered in its review of the statement of facts were rejected by us, although the agency offered no examples to support its comments. We disagree with OMB’s statements regarding our audit methodology for several reasons. First, in meetings with staff from OMB’s Office of E-Government and Information Technology, we obtained information from the staff in all of the areas noted by OMB in its letter. In our report, we discuss OMB’s role in the fund’s administration and the approval process for TMF proposals, as well as OMB’s guidance in these areas. Further, we made ourselves available to engage with OMB throughout the course of the audit. For example, we arranged a meeting with the Federal CIO and her staff to discuss the administration of the TMF and to present our preliminary observations, but the meeting was cancelled by the Federal CIO’s office due to scheduling constraints and not rescheduled. Second, we incorporated many of OMB’s comments on the statement of facts related to OMB’s role in fund administration and the approval process into our draft report. For example, although we had included information in the statement of facts regarding the requirement that agency CIOs and chief financial officers approve TMF proposals prior to submittal to the Technology Modernization Board, OMB requested that we include this information in other sections throughout the report. OMB also requested that we include language in the report to ensure that it was understood that TMF projects began after an interagency agreement was signed between the TMF Program Management Office and the agency and not when TMF awards were announced. We incorporated these changes into the background and other relevant report sections. However, in cases where OMB asked us to incorporate the entirety of language from the MGT Act—rather than summarizing the law’s key requirements—we chose not to do so for the purposes of conciseness. In addition, OMB also requested that we update the status information for the TMF awarded projects in our report to be closer to the report’s issuance. However, as we had told OMB staff during our review, we intended to report project information as of August 31, 2019, based on our audit methodology and reporting timeframes. Consequently, we believe that we have accurately characterized the facts related to OMB’s role in TMF administration and sufficiently incorporated OMB’s relevant comments into our report. OMB also disagreed with our characterization of the TMF repayment process and the assumptions about potential insolvency of the fund. As noted above in our response to GSA’s comments, our report did not make a conclusion that the fund was insolvent, or that the fund was on track to being insolvent. Rather, our report discusses the factors affecting administrative fee collection and the impact these ongoing challenges have on the TMF Program Management Office’s ability to pursue a full cost recovery model and recover all costs by fiscal year 2029, as GSA intended. In addition, we acknowledged the Program Management Office’s efforts to reduce its operating costs in fiscal year 2019 (to under $1 million). OMB also stated that the primary shortcoming has been the fact that the TMF has been underfunded by Congress, leading to slower than anticipated project volume. In our report, among the factors that we discussed as affecting TMF fee collection, we noted that the initial TMF fee rates were determined in June 2018 based on assumptions regarding appropriations that were not met. We also noted the impact that these assumptions had on the TMF Program Management Office’s projected collection of administrative fees in the first two years of operation and for fiscal year 2020. Specifically, we noted that the office projected it would distribute $75 million in fiscal year 2020 but had only approximately $35.6 million available in the fund as of August 31, 2019. We concluded that OMB and the TMF Program Management Office were not on track to recovering all operating expenses in a timely manner, thereby leaving less of the fund’s capital available for project awards. At no point did we assert the fund was insolvent, or was in danger of becoming so. As such, we continue to believe our assessment of the fund’s ongoing fee recovery is accurate and that our recommendation for OMB and GSA to work together to develop and implement a plan to use administrative fee collection to fully recover operating expenses is still warranted. OMB also challenged our analysis of agency projects’ cost estimates using our Cost Estimating and Assessment Guide because, according to the agency, we had asserted that federal agencies must follow the cost guide when developing cost estimates for federal projects. OMB stated that all projects, including those submitted for consideration, must follow OMB Circular A-11, not the GAO guide. Since OMB first introduced its cost estimating appendix to Circular A- 11 in 2006, the circular has stated that the appendix is based on the GAO cost estimating guide. Specifically, the circular stated that the appendix is based on GAO’s “guide to their auditors on how to evaluate an agency's cost estimating process, and the reliability and validity of the data used to develop the cost estimates. Following these guidelines will help agencies to meet most cost estimating requirements.” Further, we reported that OMB’s Circular A-11 cost estimating appendix outlined a number of major steps in the cost estimating process, and referenced the practices outlined in GAO’s cost guide. As our report states, OMB Circular A-11 directs agencies to follow the guidance outlined in the appendix on cost estimating for all IT investments and acquisitions within the federal government, and as mentioned above, is based on GAO’s cost estimating guidance. We noted that OMB’s guidance referenced GAO’s cost guide; however, we did not assert that agencies were required to follow GAO’s cost guide independent of Circular A-11. Further, our analysis of the cost estimates for the seven projects found that none of the projects incorporated all of the best practices for a reliable estimate cost estimate, as defined in either OMB Circular A-11 or GAO guidance. We noted that the TMF’s website did not include any guidance instructing agencies to follow the requirements outlined in Circular A-11; however, we stated that, regardless of whether or not agencies were told to do so, it was an agency’s responsibility to follow these steps. Further, we noted that ensuring agencies understand the requirements they are supposed to follow when developing a cost estimate for the TMF proposal process is critical to the success of the proposal process. Accordingly, we continue to believe our assessment of the seven projects’ cost estimates is accurate and based on appropriate and generally-accepted criteria, and that our recommendations to OMB and GSA in this area are still warranted. However, in the interest of ensuring that our recommendations are explicit about clarifying which requirements agencies are to follow when developing cost estimates, we have modified the language of our related recommendations to more directly address Circular A-11. OMB also noted the additional requirements—beyond those found in Circular A-11—imposed on agency submissions by the Technology Modernization Board, including authoritative signoff by the agency chief information officer and chief financial officer for schedule and repayment documentation. The agency further asserted that the characteristics of the TMF, including the ability to incrementally fund projects and to adjust project scope and timing of project transfers, means that projects funded by the TMF are more likely to succeed. We agree that agencies’ executive review of submissions to the board is an integral part of ensuring the quality of those submissions. Such reviews, coupled with more clear direction to agencies on what federal guidance they are required to follow, as discussed above, will further strengthen the quality of the supporting documentation submitted to the board. Further, OMB also stated that the board takes seriously its responsibilities to make sure approved projects meet the requirements of the MGT Act, the guiding principles established by the board, and to ensure that projects repay all required amounts while successfully delivering smarter, more secure commercial capabilities to improve citizen services. In addition, OMB stated that the board requires that all approved projects have requirements to provide information, best practices, playbooks, and other supporting documentation. OMB also stated that the board has managed the TMF both in alignment with industry-wide best practices for iterative, agile financing for technology projects, and has been judicious and discerning in how it invests TMF funds. We agree with the importance of ensuring approved projects meet the requirements of the MGT Act. In our report, we acknowledged OMB, the Technology Modernization Board, and the TMF Program Management Office’s efforts to provide oversight of the fund’s awarded projects. However, our report also identified ongoing challenges with the TMF Program Management Office’s fee collection, including the office’s plan to take 12 years to fully recover its operating costs—a plan that was reviewed by the Technology Modernization Board and OMB—that will hinder GSA’s ability to maximize the funds available for awards. We also agree that it is important that all approved projects have requirements in place related to providing information and supporting documentation. In our report, we discussed that OMB’s funding guidelines required projects to include a reliable estimate of project- related savings. However, as we also noted, none of the seven projects’ reported savings estimates were reliable because they did not incorporate all of the best practices for a reliable cost estimate as defined in OMB Circular A-11 and GAO’s cost estimating guide. Therefore, it was not certain whether the projects that we reviewed would save the government as much money as was estimated. While it is important that the board have requirements in place, it is equally vital that agencies clearly understand the requirements they are supposed to follow—and that these requirements are clearly articulated on the TMF website—for the proposal process to be successful. As such, we continue to believe our recommendations to OMB and GSA are appropriate. OMB’s comments are reprinted in appendix V. In addition to the aforementioned comments, the four agencies to which we did not make recommendations provided the following responses. In an email received on November 22, 2019, a Director of Strategic Planning, Policy, Egovernment and Audits in the Office of the CIO at Agriculture stated that the agency concurred with the information presented in the report. In an email received on November 7, 2019, an audit coordinator in Energy’s Office of the CIO did not state whether the agency had comments on the report and provided technical comments, which we incorporated as appropriate. In written comments provided on November 19, 2019, the department stated that it had no comments to provide on the written report. HUD’s comments are reprinted in appendix VI. In an email received on November 6, 2019, an economist in Labor’s Office of the Assistant Secretary for Policy stated that the agency had no comments on the report. We are sending copies of this report to the appropriate congressional committees; the Director of the Office of Management and Budget; the Secretaries of the Departments of Agriculture, Energy, HUD, and Labor; the Administrator of GSA; and other interested parties. This report will also be available at no charge on our website at http://www.gao.gov. If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII. Our objectives were to: (1) determine the costs of establishing and overseeing the Technology Modernization Fund (TMF), as compared to the savings realized by projects that have received awards; (2) assess the extent to which cost savings estimates for awarded projects are reliable; and (3) determine the extent to which agencies have used full and open competition for any acquisitions related to the awarded projects. The scope of our review included the Office of Management and Budget (OMB) and the General Services Administration (GSA) TMF Program Management Office, the two organizations responsible for TMF administration, as well as the five agencies that had received the seven awards from the fund as of August 2019—the Department of Agriculture (Agriculture), Department of Energy (Energy), Department of Housing and Urban Development (HUD), Department of Labor (Labor), and GSA. For our first objective, we obtained and analyzed financial data from GSA related to actual and planned operating costs for establishing and overseeing the TMF for fiscal years 2018 through 2025 (fiscal year 2018 was the first year that the TMF was in operation). To ensure the accuracy and completeness of GSA’s financial data on the operating costs for TMF administration, we obtained information from officials within GSA’s Office of the Deputy Administrator on the controls in place for ensuring the reliability of the financial data. We also reviewed GAO, GSA Office of Inspector General, and GSA reports that discussed the results of prior reviews of internal controls for GSA financial systems. Based on discussions with agency officials and our reviews of these prior reports, we did not identify any specific findings that would affect our reporting of these data. In addition, we reviewed GSA-provided data for obvious errors and inconsistencies and identified no significant errors related to the accuracy or completeness of the data. Based on these steps, we determined that these data were sufficiently reliable for us to be able to report accurately on GSA’s operating costs for TMF administration. We also obtained and analyzed agency documentation from, and interviewed officials within, GSA’s TMF Program Management Office regarding the fund’s actual and planned operating expenses as of August 31, 2019. We assessed the collection of administrative fees used to ensure the solvency of the fund during the period from June 2018 (when projects first began to receive awards) through August 31, 2019. In addition, we interviewed staff in OMB’s Office of E-Government and Information Technology regarding OMB guidance and its administrative responsibilities for the fund. Further, we obtained and analyzed TMF project proposal documentation and signed interagency agreements and interviewed officials in charge of the TMF-funded projects within the Office of the CIO and other appropriate offices at each of the five agencies to determine the scheduled repayment transfers, administrative fee payments, and whether awarded projects had realized cost savings for fiscal year 2019. (Fiscal year 2019 was the first fiscal year that awarded projects could have realized cost savings as a result of receiving TMF funding.) In doing so, we confirmed that none of the seven projects had begun to realize cost savings; therefore, it was premature to compare the projects’ realized savings to TMF administrative costs. For the second objective, we analyzed TMF project proposals, including cost estimates and supporting documentation, from the five agencies that received the seven awards. In addition, we interviewed the agencies’ project officials responsible for developing the overall TMF cost savings estimate and associated cost estimates regarding their estimation processes. We compared each TMF-funded project team’s estimating methodologies and documentation to the best practices of a reliable cost estimate discussed in GAO’s Cost Estimating and Assessment Guide. Our analysis enabled us to determine whether each project’s cost estimate, used to determine the project’s cost savings estimate, was comprehensive, well-documented, accurate, and credible. The GAO Cost Estimating and Assessment Guide considers an estimate to be comprehensive if its level of detail ensures that all pertinent costs are included and no costs are double-counted or omitted; well- documented if the estimate can be easily repeated or updated and can be traced to original sources through auditing; accurate if it is not overly conservative, is based on an assessment of the most likely costs, and is adjusted properly for inflation; and credible if the estimate has been cross-checked with an independent cost estimate and a level of uncertainty associated with the estimate has been identified and quantified. For each characteristic, our analysis had five possible assessment categories: Not met. The estimate provided no evidence that satisfies any of the characteristic’s set of best practices. Minimally met. The estimate provided evidence that satisfies a small portion of the characteristic’s set of best practices. Partially met. The estimate provided evidence that satisfies about half of the characteristic’s set of best practices. Substantially met. The estimate provided evidence that satisfies a large portion of the characteristic’s set of best practices. Met. The estimate provided complete evidence that satisfies the characteristic’s entire set of best practices. A cost estimate is considered reliable if the overall assessment for each of the four characteristics are met or substantially met. We presented the results of our initial analysis of each TMF project cost estimate to its respective agency in July 2019. We asked the agencies to verify the information presented in the analysis and provide any updates or additional supporting documentation, as appropriate. Each of the agencies provided updated information, which we incorporated into this analysis, as appropriate. In addition, we interviewed staff in the Office of E-Government and Information Technology, as well as officials from the TMF Program Management Office, about the process for the review and approval of TMF-funded project cost savings estimates and cost estimate documentation. Because the Technology Modernization Board required agency project teams to use a template to submit the project cost savings estimates and because we learned from project officials at each of the five agencies that they did not rely on data from agency financial systems when completing the template, we took additional steps to assess the reliability of the data in the completed templates. First, we interviewed officials in the TMF Program Management Office responsible for developing the template in order to understand the purpose of each template data field and what information was required to be completed. We took this step because there were no written instructions for the template regarding the data elements or the fields required to be completed. We also interviewed officials in the Office of the CIO and other appropriate offices at each agency, who were in charge of completing the TMF cost estimate template. We discussed with these officials how the template was filled out and what sources of data were used. Because project teams did not rely on data from agency financial systems when completing the spreadsheet template, we reviewed agency responses and other supporting documentation to determine how the estimated costs and savings were derived and whether there were any qualifications of the provided data. This included whether certain costs were excluded from the program cost estimate, how up-to-date the data were, or whether there were other qualifications of the provided data. We followed up with agency officials regarding these qualifications as appropriate. Further, we reviewed the completed templates to identify missing data, or other errors, and consulted with our cost estimation specialists about these issues, as appropriate. Based on our assessment of each project’s cost estimate (used to derive the cost savings estimate) and the other measures we took to assess the reliability of the data included in the completed templates, we determined that the cost savings data for all seven TMF projects were not sufficiently reliable; thus, we did not include the estimated savings amounts in our report. In addition, we discuss the data’s shortcomings in the report. To accomplish the third objective, we obtained and analyzed contract documentation for each of the seven awarded projects. We also interviewed officials in charge of the TMF-funded projects within the Office of the CIO and other appropriate offices at each of the five agencies about acquisitions related to the awarded projects. Using the agency provided contract information, we obtained and analyzed data from the Federal Procurement Data System-Next Generation (FPDS- NG)—the government’s procurement database—for the period of June through August 2019. We assessed whether each awarded acquisition used full and open competition in accordance with the Competition in Contracting Act of 1984 and the federal acquisition regulation. To ensure the accuracy and completeness of the awarded projects’ contract information related to the use of full and open competition, we searched FPDS-NG data to confirm that all contracts and task orders related to the projects had been provided. We then presented the results of our analysis to officials in charge of project acquisitions at each agency and asked these officials to verify the completeness and accuracy of the FPDS-NG data and provide any updates, as appropriate. Officials in charge of all of the awarded projects confirmed the contract information related to the use of full and open competition and provided additional contract acquisition data, as appropriate. Based on these steps, we determined that these data were sufficiently reliable to report on the TMF-funded project acquisitions’ use of full and open competition. We conducted this performance audit from March 2019 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. As of August 31, 2019, seven projects had been awarded funding from the Technology Modernization Fund (TMF). Once an award had been made, TMF funds were distributed to project teams incrementally based on each project’s performance against the milestones established in the project’s written agreement. These seven projects had received incremental funding of approximately $37.65 million and, of that amount, had obligated $18.05 million towards project implementation. The following description of each of the seven projects includes an overview of the awarded project, funding transfer, and project status information as of August 31, 2019, and how the project intends to repay the funds awarded. The Department of Agriculture’s (Agriculture) Farmers.Gov Portal project is intended to help update and modernize conservation financial assistance and payment operations within the department’s Farm Service Agency and National Resources Conservation Service. These two agencies provide financial and technical assistance to farmers and ranchers through related conservation programs. While separately authorized and appropriated, the programs share common customers and also share interconnected systems. The project is intended to work to reengineer related financial assistance business processes at these agencies and update the agencies’ legacy systems so that the systems can be properly connected with the department’s common financial system. Due to changes to the project’s schedule, an official responsible for the management of the Farmers.Gov Portal project reported that the agency plans to delay requesting the remaining balance of $6 million in awarded funds from the Technology Modernization Board until fiscal year 2020. Figure 3 provides a summary of the Farmers.Gov Portal project. Officials from the Farmers.Gov Portal project reported that the department intends to repay the TMF funds awarded using annual appropriations from each of the two agencies involved in the project. Agriculture’s Infrastructure Optimization project, managed by the Office of the Chief Information Officer (CIO), was originally intended to migrate 10 applications within the department to cloud services by the end of fiscal year 2019. However, officials responsible for the management of the project reported that they began working with the TMF Program Management Office to make changes to the project’s scope in June 2019, changing which applications would be migrated and reducing the number of applications to be migrated to one. Officials reported that the project now intends to migrate the Farm Production and Conservation’s Emergency Watershed Protection Program to cloud services but has not yet determined when the project will be completed. The program helps landowners, operators, and individuals to implement emergency measures after a natural disaster in order to help relieve imminent hazards to their life or property. Due to the change in scope for the project, officials responsible for the management of the Infrastructure Optimization project reported that planned to request a total of $500,000 for the project from the Technology Modernization Board ($4.5 million less than the original award amount). As a result of this change in scope, officials reported that the repayment period, administrative fee, and the time frames for repaying the transferred amount and associated fee, was being reevaluated by the agency. Project officials reported in August 2019 that they planned to present their revised project plan to the Technology Modernization Board for consideration and approval. If approved by the board, the project would likely reduce its administrative fee from $150,000 to $15,000. Figure 4 provides a summary of the Infrastructure Optimization project. Officials from the Infrastructure Optimization project reported that the department originally intended to repay the TMF awarded funds by using the planned cost savings and avoidances accrued from not having to pay the costs for the maintenance of these 10 applications. In fiscal year 2018, the department reported spending approximately $4 million to cover labor costs for maintaining these 10 on-premise applications. However, project officials reported that, with the change in scope to the project, the details for how they will repay the awarded funding are currently under reevaluation. The Department of Energy’s (Energy) Enterprise Cloud Email project, managed by the Office of the CIO, was originally intended to complete the consolidation, upgrade, and migration of 26 of the department’s on- premises email systems to cloud email software as a service by fiscal year 2021. However, the department made changes to the project’s scope in February 2019, reducing the number of mailboxes that would be migrated from approximately 47,080 to 24,531. Officials responsible for the management of the Enterprise Cloud Email project within Energy’s Office of the CIO reported that the department was able to migrate 22,549 mailboxes to cloud services using department funds prior to receiving TMF-awarded funds. Due to the change in scope for the project, officials from the Enterprise Cloud Email project reported that they planned to request a total of $7.41 million in funding for the project from the Technology Modernization Board ($7.80 million less than the original award amount). As a result of this change in scope, officials reported that the repayment period, administrative fee, and the time frames for repaying the transferred amount and associated fee, will change from what was originally approved by the Technology Modernization Board. Project officials reported in August 2019 that they intended to present their revised plan to the Technology Modernization Board for consideration and approval. If approved by the board, the project would reduce its administrative fee from $456,510 to $222,406 and would complete the fund repayment in 2024 rather than 2025. Figure 5 provides a summary of the Enterprise Cloud Email project. Officials from the Enterprise Cloud Email project reported that the department intends to repay the TMF funds awarded by using the planned cost savings and avoidances accrued from future operations and maintenance costs for these email systems. In fiscal year 2018, the department reported spending approximately $4.78 million to cover operations and maintenance costs for the 26 on-premise email systems originally in scope for the project. However, the department could not provide an update on the operations and maintenance costs for the current email systems that are to be migrated using TMF funds. The Department of Housing and Urban Development’s (HUD) Unisys Migration project managed by the Office of the Chief Technology Officer was originally intended to migrate five of the department’s most critical business systems from an on-premise mainframe database to cloud computing services by the end of fiscal year 2020. These systems help manage the Federal Housing Administration’s mortgage insurance program as well as over one hundred HUD grant, subsidy, and loan programs managed through the Office of the Chief Financial Officer. Due to delays in awarding contracts for the project, a HUD official reported that the department had submitted a request to the Technology Modernization Board in August 2019 for the project to be rebaselined. The official reported that the project planned to delay requesting the next disbursement of $5 million from fiscal year 2019 to fiscal year 2020 and the project is now intended to be completed by March 2021. Figure 6 provides a summary of the Unisys Migration project. Officials from the Unisys Migration project reported that the department intends to repay the TMF funds awarded by using the planned cost savings accrued from reducing the department’s overall operations and maintenance costs for these systems. In fiscal year 2018, the department reported spending approximately $11.6 million in operations and maintenance contract costs for maintaining these five legacy systems. The Department of Labor’s (Labor) Visa Application Transformation project, managed by the Office of the CIO, is intended to replace a paper- based labor certification process for certain types of work visas with an E-Certification process. The new system is intended to enable the department to issue a labor certification securely and electronically to employer applicants, similar to an electronic boarding pass issued by airlines. In addition, this project is expected to streamline and improve data accessibility and reporting capabilities by creating a data hub at Labor. This hub is expected to allow the department to securely transmit these labor certifications and other necessary documentation to the Department of Homeland Security’s U.S. Citizenship and Immigration Service, with an eventual linkage to the Department of Agriculture and the Department of State. Figure 7 provides a summary of the Visa Application Transformation project. Officials responsible for the management of the Visa Application Transformation project within the Office of the CIO reported that the department intends to repay the TMF funds awarded by using the planned cost savings accrued from eliminating the costs of procuring security paper and printers for printing the certifications as well as reduced costs for contractor and federal employee support of the paper process. In fiscal year 2019, the department reported spending approximately $1.9 million on these costs for the paper-based process. The General Services Administration’s (GSA) Application Modernization project, managed within the Office of the Chief Technology Officer, is intended to modernize 11 applications currently using proprietary vendor technology by converting them to use open source technologies. GSA currently has 88 applications that are in need of modernization and intends to use the lessons learned and new capabilities as a repeatable process that will be used for future migrations of other proprietary applications to open source technologies. Figure 8 provides a summary of the Application Modernization project. Officials responsible for managing the Application Modernization project reported that it intends to repay the TMF funds awarded through: (1) its existing working capital fund and (2) the planned cost savings and avoidances accrued from reducing operations and maintenance costs, and eliminating hardware and operating system software costs for these proprietary applications. In fiscal year 2018, the agency reported spending approximately $23.9 million to cover these costs. The NewPay project, managed within GSA’s Office of the CIO, is intended to modernize GSA’s payroll system for its 21,000 users and replace it with a cloud-based software as a service solution. This is expected to lay the foundation for modernizing federal legacy payroll systems to a cloud-based solution for approximately 2.1 million federal civilian employees. Currently, four federal agencies (Agriculture, Department of Defense, Department of the Interior, and GSA) serve as payroll providers for federal civilian employees. NewPay also is intended to encompass time and attendance solutions which are intended to be implemented in later project phases. Project officials reported that they originally planned to complete the migration to NewPay and shut down GSA’s legacy systems by 2023 and consolidate all other government legacy provider payroll operations into NewPay. However, officials reported that the strategy for transitioning other legacy payroll providers to NewPay was revised in mid-summer 2019. Going forward, GSA and the other federal payroll providers plan to focus on completing the migration of all systems to NewPay prior to transitioning and consolidating payroll operations within GSA. Project officials reported that GSA is working with OMB and the other agency payroll providers to identify funding available for these efforts so that a new schedule can be developed. Figure 9 provides a summary of the NewPay project. Officials responsible for managing the NewPay project within the Office of the CIO reported that the agency intends to repay the TMF funds awarded through subscriptions and fees that federal agencies are to pay to utilize the software as a service solution and through fees NewPay intends to collect for serving as a payroll operations provider. In fiscal year 2018, the four federal agency payroll providers spent approximately $300 million providing payroll services for approximately 2.1 million federal civilian employees. Agencies submitting full project proposals to the Technology Modernization Board during phase II of the proposal process for the Technology Modernization Fund (TMF) were required to submit information on the project’s cost estimate and cost savings estimate using a spreadsheet template (known as appendix B). We compared each TMF-funded project team’s estimating methodologies and documentation to the best practices of a reliable cost estimate discussed in the GAO Cost Estimating and Assessment Guide. According to GAO’s guidance, a reliable estimate should meet four characteristics and the specific set of best practices associated with each of the characteristics. Those four characteristics are: Comprehensive: An estimate should include all life cycle costs (from the program’s inception and design through operations and maintenance), reflect the current schedule, and have enough detail to ensure that cost elements are not omitted or double counted. Specifically, the cost estimate should be based on a product-oriented work breakdown structure that allows a program to track cost and schedule by defined deliverables, such as hardware or software components. In addition, all cost-influencing ground rules and assumptions should be detailed in the estimate’s documentation. Well-documented: An estimate should be thoroughly documented; describe how it was developed; and include source data, clearly detailed calculations and results, and explanations of why particular estimating methods and references were chosen. Data should be traced to their source documents. Accurate: An estimate should be based on historical data or actual experiences on other comparable programs and an assessment of most likely costs, and be adjusted properly for inflation. In addition, the estimate should be updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it should always reflect the current status. Credible: An estimate should discuss any limitations of the analysis because of uncertainty surrounding data or assumptions. In addition, the estimate should incorporate the results of a sensitivity analysis (that examine the effects of changing assumptions on the estimate), and risk and uncertainty analysis (that identifies all of the potential project risks and assesses how these might affect the cost estimate). The estimate’s results should be cross-checked, and an independent cost estimate should be conducted to see whether other estimation methods produce similar results. In assessing each project’s estimate against the components of the four characteristics, we assigned one of five assessment categories: Not met. The estimate provided no evidence that satisfies any of the characteristic’s set of best practices. Minimally met. The estimate provided evidence that satisfies a small portion of the characteristic’s set of best practices. Partially met. The estimate provided evidence that satisfies about half of the characteristic’s set of best practices. Substantially met. The estimate provided evidence that satisfies a large portion of the characteristic’s set of best practices. Met. The estimate provided complete evidence that satisfies the characteristic’s entire set of best practices. A cost estimate is considered reliable if the overall assessment ratings for each of the four characteristics are met or substantially met. The following discusses in detail our assessment of the seven TMF awarded projects’ cost estimates. Table 10 includes our detailed assessment of the Department of Agriculture’s (Agriculture) Farmers.Gov Portal project. Based on the overall assessment ratings for each of the four characteristics, Agriculture’s project cost estimate is not considered reliable. Table 11 below includes our detailed assessment of Agriculture’s Infrastructure Optimization project. Based on the overall assessment ratings for each of the four characteristics, Agriculture’s project cost estimate is not considered reliable. Table 12 includes our detailed assessment of the Department of Energy’s (Energy) Enterprise Cloud Email project. Based on the overall assessment ratings for each of the four characteristics, Energy’s project cost estimate is not considered reliable. Table 13 includes our detailed assessment of the Department of Housing and Urban Development’s (HUD) Unisys Migration project. Based on the overall assessment ratings for each of the four characteristics, HUD’s project cost estimate is not considered reliable. Table 14 includes our detailed assessment of the Department of Labor’s (Labor) Visa Application Transformation project. Based on the overall assessment ratings for each of the four characteristics, Labor’s project cost estimate is not considered reliable. Table 15 includes our detailed assessment of the General Services Administration’s (GSA) Application Modernization project. Based on the overall assessment ratings for each of the four characteristics, GSA’s project cost estimate is not considered reliable. Table 16 includes our detailed assessment of GSA’s NewPay project. Based on the overall assessment ratings for each of the four characteristics, GSA’s project cost estimate is not considered reliable. In addition to the individual named above, the following staff made key contributions to this report: Dave Hinchman (Assistant Director), Jason Lee (Assistant Director), Jessica Waselkow (Assistant Director), Chris Businsky, Jennifer Echard, Emile Ettedgui, Valerie Hopkins (Analyst in Charge), Anna Irvine, Julia Kennon, Sandra Kerr, James MacAulay, Priscilla Smith, and Mary Weiland.", "summary": "In December 2017, the MGT Act was enacted, which established the TMF. OMB, the Technology Modernization Board, and GSA oversee the TMF. The board is responsible for approval of agency project proposals focused on replacing aging IT systems. Agencies receive incremental award funding and are required to repay the funds transferred and an administrative fee within five years. Agencies may use the project's generated cost savings to repay the award. GSA can use TMF appropriations to cover its operating expenses, and is required to collect administrative fees from awarded projects to offset these expenses. GSA's fee rate was established with the intent to fully recover its costs. As of August 2019, Congress had appropriated $125 million to the TMF. The act included a provision for GAO to report biannually on the TMF. For its first TMF report, among other things, GAO analyzed the TMF's operating costs and assessed the reliability of selected projects' cost savings estimates. To do so, GAO reviewed OMB and GSA's administrative fund processes, and GSA financial data on TMF operating costs. GAO also analyzed TMF project proposal and supporting cost estimate documentation from selected agencies. As of August 2019, the Technology Modernization Board had made seven Technology Modernization Fund (TMF) awards to five agencies, totaling about $89 million, and had transferred $37.65 million of this funding to the projects (see table). In addition, pursuant to the Modernizing Government Technology (MGT) Act, the General Services Administration (GSA) had obligated about $1.2 million to cover TMF operating expenses, but had recovered only about 3 percent of those expenses through fee payments. The seven projects are expected to make $1.2 million in scheduled fee payments by the end of fiscal year 2025; as of August, three projects have made fee payments totaling $33,165. Based on the current schedule, GSA will not fully recover these expenses until fiscal year 2025 at the earliest. GSA had collected fewer fees than planned to offset costs due to several factors. For example, the seven projects paid fees based on the amounts transferred, rather the total funds awarded, thereby reducing fee collections in the initial years. Two projects also proposed scope changes that are expected to reduce funding required and, thus, reduce total fees. Such factors raise doubts on whether GSA will be able to fully recover future operating expenses. Although GSA acknowledged this issue, the agency has not yet developed a plan outlining the actions needed to fully recover its TMF operating costs in a timely manner. The Office of Management and Budget's (OMB) funding guidelines require projects to include a reliable estimate of any project-related savings. However, the seven projects' reported savings estimates derived from cost estimates are not reliable. None of the projects incorporated all of the best practices for a reliable cost estimate, as defined in GAO and OMB guidance. Without clarifying the requirement that agencies follow Circular A-11's cost estimating process (that references GAO's cost estimating guidance discussed in this report), agencies are at risk of continuing to provide unreliable cost information in their proposals. GAO is making five recommendations—two to OMB and three to GSA, including developing a plan to fully recover operating costs and clarifying that agencies should follow required cost guidance. OMB raised a number of concerns that GAO addresses in the report. GSA agreed with one recommendation and partially agreed with the other two. GAO continues to believe all of its recommendations are appropriate.", "document_type": "gao"}
{"report": "NMB is headed by a three-member board, with each member appointed by the President and confirmed by the Senate for a term of 3 years. The board members provide overall leadership and strategic direction for NMB, and retain responsibility for key functions such as releasing parties from the mediation of major disputes if no agreement can be reached. In May 2018, NMB reorganized various agency components to improve its management and oversight of agency operations. This resulted in the creation of three mission areas and three mission support areas. The Offices of Fiscal Services and Information Services were newly created as a result of the delegation order (see fig. 1). In June 2019, NMB hired a Chief Financial Officer (CFO), who serves as the Director of the Office of Fiscal Services. The CFO has authority over NMB’s budget, accounting, and financial auditing functions. In January 2019, NMB hired a Chief Information Officer (CIO), who serves as the Director of the Office of Information Services. The CIO has authority over NMB’s information technology and related systems, including its electronic record keeping functions. All offices, along with NMB’s Designated Agency Ethics Official, report directly to the Board. Previously, the Offices of Administration, Mediation, and Arbitration reported to a Chief of Staff, a position that was eliminated in 2018. NMB’s overall mission is to provide for the independence of air and rail carriers and employees in matters of self-organization, help prevent interruption to commerce conducted through the operation of those carriers, administer adjustment boards, as well as develop complementary strategies to resolve disputes. NMB has three program areas to fulfill its mission: Representation. Rail or air carrier employees select unions for the purposes of collective bargaining through secret-ballot elections conducted by NMB. NMB is charged with resolving any questions concerning representation of a specific craft or class through the agency’s Office of Legal Affairs, and has sole jurisdiction to decide these disputes. Mediation and Alternative Dispute Resolution. The RLA provides for mediation to help resolve disputes between management and labor during collective bargaining negotiations. When rail or air carriers and unions cannot reach agreement on the terms of a new or revised collective bargaining agreement – such as working conditions or rates of pay – either party can apply for NMB’s mediation services to resolve their differences. Additionally, NMB may impose mediation if it finds that resolving the dispute is in the public’s interest. NMB also offers grievance mediation to parties as an alternative way to resolve disputes filed for grievance arbitration. Although mediation is voluntary, it is a less expensive approach to resolving grievances, using NMB’s existing mediation staff rather than outsourcing—and paying—external arbitrators. Arbitration. The RLA also offers grievance arbitration to help resolve disagreements between carriers and unions over how to interpret and apply provisions of existing collective bargaining agreements. NMB does not directly provide arbitration services, but rather maintains a list of registered arbitrators from which the parties can select someone to review and decide their case. In the airline industry, the parties pay the costs of arbitration. In the railroad industry, however, consistent with the requirements of the RLA, NMB pays the fee and travel expenses of the arbitrator. The Office of Management and Budget (OMB) and the Office of Personnel Management (OPM) have key oversight responsibilities for all federal agencies, including NMB. OMB is responsible for the oversight of NMB’s management and information technology. OPM is the central personnel management agency of the federal government charged with administering and enforcing civil service laws, regulations, and rules. OPM annually administers surveys to federal employees across the government, including NMB, to solicit their views on their agencies including agency leadership, collaboration, and other issues. OPM also offers various services to agencies to evaluate organizational climate. Federal law does not establish an Inspector General (IG) for NMB. However, the agency signed a Memorandum of Understanding (MOU) in 2018 with the National Labor Relations Board’s (NLRB) Office of Inspector General to provide independent audit and investigative oversight. In the MOU, the NLRB IG agreed to (1) operate a hotline for employees to anonymously submit information—via email or telephone messages—regarding fraud, waste, and abuse involving the NMB’s programs and operations, and (2) take action to address complaints, such as inform the appropriate law enforcement agency or the NMB Chairman or Board Members, as appropriate. FedRAMP is a government-wide program that provides authorizations for use of cloud services. As an executive agency that uses a cloud service approved through FedRAMP, NMB is subject to related requirements. Through a December 2011 memorandum, OMB established requirements for executive agencies to use FedRAMP when conducting security authorizations for agency use of cloud services. In addition, the FedRAMP Program Management Office issued guidance in 2017 that specifies authorization requirements, including that an agency should document the authorization of the agency system supported by a cloud service approved through FedRAMP and the related cloud service used by the agency. GAO has issued three prior reports on NMB and collectively had 13 recommendations. NMB had previously implemented six of those recommendations, and seven remained in our current review. We issued our first report in December 2013 with seven recommendations in key management areas, including strategic planning, performance measurement, and workforce planning. We also suggested that Congress consider authorizing an IG at an appropriate federal agency to provide independent audit and investigative oversight at NMB. We issued a second report in February 2016, which found that NMB needed to take additional actions to implement the seven recommendations from our December 2013 report. We also made one additional recommendation related to procurement. We issued our third report in March 2018, which found that NMB had taken action to implement four of the recommendations from our December 2013 report and the recommendation from our February 2016 report. However, additional actions were needed to close the remaining three recommendations. We also made five additional recommendations related to the backlog of arbitration cases, outside employment, organizational climate, and NMB’s travel and telework policies. NMB implemented a recommendation from GAO’s 2018 report to create and monitor requests for outside employment, but has not taken action to fully implement the remaining six recommendations from GAO’s past reviews (see table 1). By not fully implementing these recommendations, NMB remains at risk in several areas key to its mission, including information privacy and security and organizational climate, among others. GAO 2018 Recommendation: Develop and implement policies for approval and monitoring of employee requests for outside employment and other outside activities to prevent violations of ethics rules, consistent with Office of Government Ethics standards of conduct and federal internal control standards. Since our 2018 review, we found that NMB has developed and implemented policies for approving employee requests for outside employment and the agency monitors these requests. We reported in 2018 that NMB did not have a policy for approving and monitoring employee requests for outside employment consistent with the Office of Government Ethics (OGE) standards of conduct and federal internal controls. NMB also did not systematically track or monitor when managers or board members approved such activities for an employee. We recommended that establishing an outside employment policy and a system to monitor activities would help to prevent violations of ethics rules. In our current review, we found that NMB has implemented our recommendation. NMB worked with OGE to develop a policy on outside employment that details how employees should submit outside employment requests, consistent with OGE standards. NMB has incorporated the policy into annual and new employee ethics training. Once NMB approves an outside employment request, the agency monitors outside employment through employees’ annual financial disclosure forms. GAO 2018 Recommendation: Develop and execute a plan to address the rail arbitration case backlog. Since our 2018 review, we found that NMB has used several strategies to reduce its backlog by 57 percent; however, without a plan establishing specific goals and timeframes, it is difficult to track the agency’s progress against specific measures of success. We reported in 2018 that NMB’s rail grievance arbitration case backlog had more than tripled since 2011, and that NMB did not have a specific plan and related processes to address it. However, identifying and assessing the risks associated with the backlog and developing a plan to effectively manage it are key to implementing effective risk management. In our current review, we found that NMB has implemented several initiatives to reduce the rail grievance arbitration case backlog, including removing older cases, using lead cases—cases that have the same parties and similar fact patterns, allowing a decision from one case to settle others—and promoting an “Ambassador Program” to move cases from grievance arbitration to grievance mediation. NMB officials credit these strategies with reducing the backlog from a height of 8,550 cases at the end of fiscal year 2017 by 4,852 cases—about 57 percent—to 3,698 cases as of the end of fiscal year 2019 (see table 2). 1. Removing older cases. NMB officials said that NMB has removed older arbitration cases that were filed, but had not yet been moved forward to arbitration. Specifically, officials explained that, in late summer 2018, NMB removed 400 cases from the backlog that were 3 years or older. NMB officials said that the agency subsequently removed 1,025 cases that were 2 years or older. NMB officials told us that parties may choose to re-file a removed case. NMB has not received objections from unions and carriers regarding the removal of older cases. 2. Using lead cases: For lead cases, NMB and the parties agree that the decision for one case will be used to settle other cases with similar fact patterns. For example, officials said that a similar fact pattern would be cases that had the same union and carrier and dealt with the same underlying issue. In fiscal year 2017, NMB used the decisions for nine lead cases to settle 4,240 additional claims. In fiscal year 2018, NMB used the decisions for four lead cases to settle 600 additional claims. 3. Promoting the Ambassador Program. NMB’s Ambassador Program involves NMB reaching out to parties to encourage them to voluntarily move cases from arbitration to grievance mediation. NMB has assigned experienced mediators to carriers and unions as “ambassadors.” Unions that have disputes with a carrier can raise the issue through the ambassador in hopes of avoiding the formal arbitration process; in that way, the Ambassador Program may proactively decrease the number of arbitration cases filed. NMB is interested in using the Ambassador Program to resolve multiple claims regarding the same issue, policy, or employment action. NMB officials said in fiscal year 2018, NMB had seven cases in the Ambassador Program and closed six cases. NMB officials said in fiscal year 2019, NMB had four cases in the Ambassador Program; none are closed to date. NMB officials said that the Ambassador Program and the lead case program are related, in that many of the cases moved through the Ambassador Program are lead cases. For example, NMB reported that in fiscal year 2018, one grievance mediation case was used to settle 300 claims. In fiscal year 2017, NMB heard five cases in the Ambassador Program, and the decisions on these cases were applied to 1,951 remaining claims to resolve them. In addition, NMB officials told us a small number of railway carriers and unions file the largest percentage of the grievance arbitration cases (see fig. 2). In fiscal year 2019, four railway carriers represented 72 percent of the backlog, and four railway unions represented 87 percent of the backlog. The Office of Arbitration seeks to coordinate with the organizations with the most arbitration cases to help them move toward mediation or other techniques to decrease the arbitration backlog. Another method NMB reported using to reduce the backlog is to direct otherwise unobligated funding at the end of the fiscal year to fund arbitration cases, in addition to the amount of funds it had initially budgeted for arbitration. Specifically, NMB officials said that the agency allocated at least $1 million in additional funds in fiscal years 2017, 2018, and 2019 for arbitration cases at the end of each fiscal year, which allowed NMB to fund arbitration for approximately 4,200 more cases overall, closing nearly all of those cases. Officials said that these additional funds came from unfilled full-time equivalent staff position salaries and contracts that NMB did not award. Officials said they do not anticipate having similar amounts of funding available for arbitration in the future, once NMB hires staff and awards the contracts. While NMB has implemented various strategies to reduce the rail arbitration case backlog, it has not developed a plan to link the strategies to specific goals or timeframes. GAO’s Standards for Internal Control in the Federal Government state that management should define objectives in specific and measurable terms. Further, federal agencies are required to develop annual performance plans that measure performance to reinforce the connection between long-term strategic goals and day-to- day activities of its managers and staff. NMB’s 2018 Annual Performance and Accountability Report does not link NMB’s efforts to reduce the backlog to specific and measurable objectives to assess their effectiveness. By developing specific and measurable objectives to reduce the overall backlog or any component thereof, NMB and Congress would be able to more adequately assess NMB’s progress in reducing the backlog relative to its goals. GAO 2018 Recommendation: Complete and take actions on the organizational climate assessment and survey results as a means to address employee concerns. Since our 2018 review, we found NMB has completed an organizational climate assessment but has not taken actions to address the results of that assessment. We reported in 2018 that surveyed NMB employees expressed concerns about the organizational climate at NMB. In addition, NMB’s strategic plan called for an organizational climate assessment to be conducted by the end of calendar year 2015 and every 3 years thereafter. However, at the time of our 2018 report, NMB had not conducted such an assessment. In addition, NMB officials said that they did not take action in response to survey results, which had a 59 percent response rate, because they believed the negative responses were attributable to a few employees. GAO recommended that NMB conduct an organizational climate assessment and develop actions to address the results of that assessment. In our current review, we found that NMB conducted an organizational climate assessment and has taken some actions to address the elements identified in the assessment, but must take additional actions to address employee concerns. NMB worked with OPM to conduct an organizational climate assessment in April 2019. The assessment had a response rate of 95 percent. Several NMB officials said the agency achieved a higher response rate than prior surveys because the Board held an all staff meeting to emphasize the importance of taking the assessment. In May 2019, NMB received the results of the organizational climate assessment from OPM. NMB identified a lack of communication across departmental staff as an issue. To address this, NMB directed regular interdepartmental updates, where each quarter a department is given an opportunity to present the activities within that department. NMB officials said that NMB held its first interdepartmental update in October 2019, with the Office of Legal Affairs presenting. The next interdepartmental update is scheduled for February 2020. NMB has identified some additional potential actions to address issues raised by the organizational climate assessment, including directing NMB’s CFO to rewrite the travel policy and to work with OPM to identify recommended training for supervisors, among others. However, these potential actions are not finalized and are generally unlinked to timeframes for implementation. By not taking these actions, NMB employees may be less engaged, which may lead to absenteeism or turnover. GAO 2018 Recommendation: Revise NMB’s Travel Policy and develop appropriate internal controls to ensure compliance with federal regulations. Since our 2018 review, we found that NMB has not revised its travel policy to be consistent with the Federal Travel Regulation (FTR) issued by the General Services Administration. We reported in 2018 that NMB’s travel policy was, in some respects, not consistent with the FTR. NMB management had also granted NMB staff exceptions to the agency travel policy that were not consistent with the FTR. For example, the FTR requires employees to rent the least expensive car available, but a former NMB management official approved the use of a luxury rental car in some cases. Our 2018 report found that without greater oversight of employee travel expenses, NMB may be incurring unnecessary additional expenses for employee travel. In our current review, we found that NMB has not revised its travel policy to be consistent with the FTR. However, NMB’s Office of Fiscal Services plans to rewrite portions of the travel policy, including clarifying roles and responsibilities of NMB employees and adding a Frequently Asked Questions portion to the policy. NMB officials said the revised policy is expected to be completed in 2020, and will be reviewed by the CFO in consultation with the Office of Legal Affairs prior to its publication. It is unclear the extent to which these changes will make NMB’s travel policy consistent with the FTR. In addition, NMB has taken steps to strengthen its internal controls related to travel, including: 1. Replacing the Chief of Staff role in travel policy. In August 2018, NMB replaced references to the eliminated Chief of Staff position in its travel policy to make the Board the decision making body for travel- related issues. This clarification strengthened internal controls because no one individual is singularly responsible for approval. 2. Updating NMB’s travel charge card program. In 2019, NMB transitioned to a new travel charge card program run by the General Services Administration. Both NMB and the Department of Treasury’s Bureau of Fiscal Services, which provides accounting services to NMB, routinely monitor the program, including monitoring each employee’s use of the travel card to ensure only appropriate official government-related expenses are being charged to the card. The CFO receives reports from this new program. 3. Issuing an interim procedure. Separately, NMB has established an interim procedure for disputed claims that sets timeframes for when vouchers must be approved to avoid delays in returning vouchers to travelers. The interim procedure requires travelers to cite the specific regulatory authority to support their disputed claim. The NMB Board is determining whether this procedure should be established officially in the travel policy. While NMB has made these initial efforts to strengthen internal controls related to travel, such as increasing oversight from the Board, NMB has not revised its travel policy to be consistent with the FTR. For example, NMB has not updated its policy to clarify the use of personal credit cards as discussed in our 2018 review. Without an updated policy consistent with the FTR, NMB may be incurring needless additional expenses for employee travel. Since our 2018 review, we found that NMB has not yet revised its telework policy, but the agency has collected telework agreements and provided training for teleworking employees. We reported in 2018 that NMB’s telework policy is not consistent with the requirements of the Telework Enhancement Act of 2010, which requires employees to take telework training and have signed telework agreements prior to beginning telework, and NMB did not consistently enforce its policy. NMB’s telework policy, effective October 2015, did not mention employee telework training nor did management require employees to complete training before entering into a telework agreement, as required by federal law. In addition, management allowed employees to telework without a written telework agreement, even though this requirement is specified in NMB’s telework policy. NMB agreed to review its policy and make any revisions determined to be necessary. In our current review, we found that NMB now tracks telework training and agreements to ensure that teleworking employees have telework agreements and completed telework training prior to engaging in telework. However, NMB has not updated its telework policy to be consistent with the requirements of the Telework Enhancement Act of 2010, instead determining after reviewing its policy that a revision was unnecessary. Despite this determination, the telework policy, last updated in October 2015, does not reflect the current structure of NMB: for example, it includes responsibilities for the Chief of Staff, a position that no longer exists. Further, the policy does not mention employee telework training. Until NMB updates its policy, it will continue to be outdated regarding official responsibilities and inconsistent with relevant law. GAO 2013 Recommendation: Establish a privacy program that includes conducting privacy impact assessments and issuing system of record notices for systems that contain personally identifiable information. Since our 2018 review, we found that NMB has not always followed key information privacy practices to protect personal information federal agencies collect. In our 2018 review, we found that NMB did not establish a privacy program that included practices such as conducting privacy impact assessments and issuing system of records notices for systems that contain personally identifiable information. For example, in our 2018 review, we found that while NMB designated a privacy officer, the agency did not conduct privacy impact assessments for its systems and those of third-party providers containing the agency’s personally identifiable information. In our current review, we found that, of the four key information privacy practices described in our 2013 report, NMB is still following one, partially following two, and minimally following one practice. For example, NMB documented a privacy impact assessment dated July 2018. However, the assessment did not specify whether a system of records notice would be developed as required by OMB. For additional details on the extent to which NMB is following key information privacy practices, see appendix II. GAO 2013 Recommendation: Develop and fully implement key components of an information security program in accordance with the Federal Information Security Management Act of 2002. Since our 2018 review, we found that NMB continues to only partially follow the eight key information security practices in accordance with the Federal Information Security Management Act (FISMA). These practices include developing and implementing risk-based policies and procedures to ensure compliance with applicable standards and guidance, including system configuration requirements. For example, in our 2018 review, we found that, while NMB had its information security policy documented in its April 2016 Information Program Plan, which included risk assessment requirements, NMB had not developed agency- wide policies and procedures on the oversight of its third-party providers that support the operations and assets of the agency, as required by FISMA. In our current review, we found that, while NMB has created a policy to conduct periodic risk assessments of cyber threats and vulnerabilities, the agency did not provide risk assessment documentation of its enterprise network for fiscal year 2019. NMB officials said that the agency had not fully addressed information security practices due to a lack of resources. NMB officials stated the agency plans to address several of these practices with the targeted completion expected in fiscal year 2020. As a step to further focus on information technology challenges, NMB established the Office of Information Services and, as noted earlier, hired a CIO. While hiring a CIO does not directly address the practices described above, NMB officials said that these actions, along with hiring more staff and making key acquisitions through contracts, will enable NMB to fully follow the practices in the future. For additional details on the extent to which NMB is following key information security practices, including NMB’s recent engagement of contractors, see appendix II. In addition to the gaps in key information security practices discussed above, we found in our current review that NMB has not fully implemented federal requirements related to authorizing the cloud service approved through FedRAMP that the agency uses. OMB defines an authorization to operate as an official management decision where a federal official or officials authorize the operation of information system(s) and accept the risk to agency operations and assets, individuals, and other organizations based on the implementation of security and privacy controls. OMB requires agencies to use FedRAMP processes when granting authorizations to operate for their use of cloud services. The FedRAMP Program Management Office published guidance in 2017 to describe the process by which agencies can reuse existing authorizations. According to the FedRAMP guidance, agencies should document the authorization of 1) the agency system supported by the cloud service; and 2) the cloud service used by the agency. Additionally, the agency should provide a copy of its authorization letter for the cloud service to the FedRAMP Program Management Office so that the office can verify the agency’s use of the service and keep agencies informed of any changes to a provider’s authorization status. These steps ensure that federal agencies have made a determination of whether the cloud service provider’s risk posture is acceptable for use at that agency. According to NMB, the agency is using a cloud service that was approved through FedRAMP to support the agency’s enterprise network. NMB had documented the authorization of its enterprise network, but NMB had not documented its authorization of the cloud service to demonstrate that it had accepted the risk of using the service. In addition, NMB had not provided the authorization letter for the cloud service to the FedRAMP Program Management Office. NMB officials stated that the agency’s internal information security guidance did not include procedures to address FedRAMP requirements because the officials were unaware of those requirements. Without taking these steps, the FedRAMP Program Management Office may not be able to inform NMB, in a timely manner, if its cloud service provider has experienced a security incident. NMB has taken steps to improve its agency management and oversight, such as reorganizing some agency mission areas and filling key staff positions; however, it lacks effective internal controls to manage and oversee its annual appropriation and ensure that its audit policy is consistently followed. As a result, the agency did not use funding the Board said is needed to accomplish NMB goals. NMB had about $4 million in unobligated appropriations in expired accounts in the U.S. Treasury and unavailable to NMB for new obligations from fiscal years 2016 through 2019. In addition, NMB has not taken corrective actions to address management deficiencies identified during audits. NMB has not established effective internal controls to assist the agency in managing and overseeing its annual appropriations. NMB has had significant unobligated balances remaining for the last 4 fiscal years, even though officials said they could not accomplish some of the agency’s goals – such as hiring staff and information technology initiatives – due to a lack of financial resources (see table 3). For example, from fiscal years 2016 through 2019, NMB had unobligated balances ranging between approximately $600,000 to over $2 million. These are the remaining funds from its appropriations received each year from fiscal year 2016 through 2019. In total, over 8 percent of NMB’s appropriations for the last 4 fiscal years went unobligated. NMB officials noted that hiring challenges and uncertainty regarding the agency’s final appropriation as a result of continuing resolutions— legislation that continues to fund federal agencies until final agency appropriations for a fiscal year are made—kept the agency from obligating funds during those fiscal years to achieve its goals. For example, NMB officials said that the Board did not pursue certain planned hiring, as well as other contract actions and travel, because of uncertainty about the amount of final appropriations that would be available. GAO has reported that continuing resolutions present challenges for federal agencies, and that agencies may not have enough time to spend funding on high-priority needs such as hiring. However, given the frequency of continuing resolutions, it is even more important for NMB to develop an effective plan to use its appropriations to accomplish agency goals. During our review, we found that NMB struggled to plan effectively for contingencies such as funding under continuing resolutions, although NMB’s budget request and appropriations were generally consistent for several years. Additionally, NMB officials told us they lacked an effective process to reliably forecast the amount of funding the agency would have remaining at the end of a fiscal year, and we found NMB did not plan effectively to allow the agency to obligate its fiscal year appropriations. NMB officials said the agency waited until the end of the third quarter to assign unobligated funds to other priorities in order to allow for the option of retaining temporary services during periods of high demand. Although NMB was able to reassign at least $1 million to arbitration work in each of the fourth quarters in 2017, 2018, and 2019, there was insufficient time to use other available funding in additional areas of need. The Board has taken steps to improve its budget execution process. In particular, the Board has implemented new bi-weekly budget reviews with the CFO meant to help NMB better forecast the agency’s available funds, including more reliably predicting the amount of unobligated funds and how to use those funds to meet agency goals. However, these changes have not been incorporated into a formal, written process to help NMB manage its appropriations more effectively to achieve agency goals. One goal under NMB’s Strategic Plan is to provide timely, efficient, and responsible stewardship of agency fiscal resources. Federal internal control standards state that internal controls comprise the plans used to fulfill the goals of the agency, and we have reported that maintaining written policies and procedures can help ensure that adequate internal controls are in place. Further, those standards state that management should obtain reliable financial data on a timely basis to enable effective monitoring. Until NMB establishes and documents an effective plan to manage its appropriations, as well as timely, reliable financial data, it may miss opportunities to achieve its objectives as efficiently and effectively as possible. NMB lacks effective internal controls to ensure that it consistently addresses deficiencies identified from financial and other audits. For example, NMB did not follow its own requirements to create corrective action plans to address findings of financial audits or GAO recommendations. Under agency policy, those corrective action plans should detail major steps for NMB to take, estimated completion dates, and other related information. Although NMB provided its financial auditors and GAO with general plans to address findings and recommendations, those plans have not always included major steps or estimated completion dates, and NMB has not always followed through with the steps it agreed to take. For instance, NMB’s financial auditor noted a deficiency in NMB’s internal controls related to financial reporting in 2017, and noted a similar deficiency in 2018 because NMB still had not addressed the problem sufficiently. Effective remediation of internal control deficiencies, like those found by GAO and other audits, is essential to achieving the objectives of the Federal Managers’ Financial Integrity Act, as amended (FMFIA). Unless NMB follows its own policy and federal guidance on corrective action plans, it may not do what is needed to address the risks associated with any deficiency. Likewise, NMB did not follow its policy to circulate draft financial audit findings and provide a draft response to the Board. When NMB received notice of a 2018 draft management letter from its independent financial auditors, the letter was not circulated for over 5 months nor was the Board provided with any draft response to the findings. Moreover, although NMB’s Board was notified of the letter’s existence in November 2018, the Board did not ask for the letter prior to May 2019, and said instead that they relied on the official in charge of the audit to follow procedure. Federal internal control standards state that management should obtain relevant data, including compliance data, in a timely manner so that they can be used for monitoring, but NMB officials and the Board did not obtain such information, putting the agency at risk for missed opportunities to identify and address audit deficiencies. Additionally, NMB has not effectively monitored the sufficiency of its internal controls as required under FMFIA. NMB has also not conducted its planned fiscal year 2017 internal controls review of its Office of Mediation or its fiscal year 2018 internal controls review of its Office of Legal Affairs in order to complete its annual review and report under FMFIA. Monitoring the effectiveness of internal controls provides the basis for an agency’s annual assessment and report of internal control, as required by FMFIA. NMB officials said the agency had not completed those reviews in a timely manner due to the timing of multiple audits occurring at NMB. NMB recently scheduled those reviews for 2020. Without monitoring its internal controls, NMB may not identify and be able to address significant management problems that can impede the agency’s ability to achieve its goals. Although NMB has identified and taken steps to address some of these audit and internal control deficiencies, it has not established an effective process to consistently monitor adherence to its audit policy and federal standards, evaluate the results, and remediate any deficiencies. For example, NMB has revised its audit policy to assign responsibility for audits and related follow-up to the CFO, who is tasked with helping NMB develop appropriate corrective action plans. Additionally, the Board said it addressed the issue of not circulating the audit management letter with the responsible official and changed the protocols for circulating letters for audit findings to include the Board in addition to the CFO. However, these actions, by themselves, do not establish the monitoring activities required by NMB’s audit policy and federal internal control standards. Under NMB’s new audit policy, the Board has responsibility to provide top-level oversight of NMB’s management activities related to audit coordination and follow-up; federal internal control standards require management to establish and operate monitoring activities to monitor the internal control system, evaluate the results, and remediate identified deficiencies on a timely basis. Further, FMFIA requires regular evaluation of the sufficiency of an agency’s internal controls. The failure of NMB to conduct the necessary reviews to support its annual assertion under FMFIA hampers the agency’s ability to identify risks in its internal controls and to correct any associated material weaknesses, as well as deprives Congress of information necessary to oversee the agency. Further, by not following its own policies and federal internal control standards, NMB may miss opportunities to improve its ability to achieve objectives, address audit deficiencies, and improve management oversight. NMB has fully implemented one of the seven recommendations still open from prior GAO reports: creating standards on outside employment, which will help prevent employee violations of ethics rules. However, while making varying degrees of progress on the others, NMB still has more work to implement all six remaining recommendations. NMB has decreased its backlog of rail arbitration cases, but it has no specific goals against which to measure its progress toward reducing the backlog and ensuring NMB and Congress can adequately assess NMB’s resolution of disputes. Likewise, while the Board’s implementation of the climate assessment illustrated that it recognizes the need to understand employee concerns regarding communication across teams, agency travel, and training for management, among other things, it has not fully executed plans to address those concerns in order to benefit from that assessment. Finally, while NMB has improved certain aspects of how it implements its travel and telework policies, it has not sufficiently changed the policies themselves to ensure that NMB policies are consistent with the Federal Travel Regulation and the Telework Enhancement Act of 2010, respectively. Moreover, NMB established the Office of Information Services and hired a new CIO to assist NMB in addressing information security and privacy recommendations, but NMB still must change its underlying information policies and procedures, including updating its information privacy policy to reflect the current structure of NMB and perform a review of its system security plans. Additionally, until NMB complies with the recent FedRAMP requirements, its data may be at greater risk in the event of a security incident. Without fully implementing the remaining six recommendations and addressing the recent FedRAMP requirements, NMB is missing opportunities to mitigate information security risks and improve its own management and performance. Moreover, NMB faces challenges in managing and overseeing its annual appropriation and audit policy as a result of ineffective internal controls. Specifically, as a result of ineffective internal controls for managing and overseeing its annual appropriation, NMB has forgone several million dollars in funding that could have been used to accomplish agency goals. While continuing resolutions can make it difficult for agencies to achieve hiring and other goals, until NMB develops a written plan to document NMB’s process for reviewing and monitoring the agency’s annual appropriation to effectively manage its budgetary resources and spending, NMB will likely continue to miss opportunities to accomplish its goals. Similarly, until NMB establishes a specific process for the Board to monitor and evaluate NMB’s adherence to audit protocols, NMB will not be well positioned to address audit recommendations from its financial auditors and GAO, hindering efforts to improve its operations. While NMB officials have told us that they did not have the resources for certain changes that we recommended, such as information security and privacy improvements, they had more resources than they actually used, as evidenced by unused appropriations. Given the range of management issues that have remained unaddressed over the past 6 years, NMB should ensure their available resources are used effectively. We are making the following four recommendations to the National Mediation Board (NMB): 1. The Chairman of the NMB should document NMB’s authorizations for its use of cloud services approved through FedRAMP and submit the authorizations to the FedRAMP Program Management Office. (Recommendation 1) 2. The Chairman of the NMB should update NMB’s security policies and procedures to include FedRAMP’s authorization requirements. (Recommendation 2) 3. The Chairman of the NMB should develop a written plan to document NMB’s process for reviewing and monitoring the agency’s annual appropriation to ensure that funds are used effectively. (Recommendation 3) 4. The Chairman of the NMB should establish a process for the Board to effectively monitor and evaluate NMB’s adherence to audit protocols and implementation of actions to address audit recommendations. (Recommendation 4) We provided a draft of this report to the National Mediation Board (NMB) for review and comment. The agency provided written comments, which are reproduced in their entirety in appendix III. NMB also provided technical comments, which we incorporated as appropriate. NMB agreed with our four recommendations, and stated that it would take actions to address them. With regard to our first two recommendations concerning the Federal Risk and Authorization Management Program authorizations, NMB stated that it plans to complete the required actions by the end of fiscal year 2020. While NMB stated that it would take actions to address our third and fourth recommendations, concerning improvements to better monitor its annual appropriations and adhere to audit protocols to implement audit recommendations, respectively, NMB did not provide a timeframe for when these actions would be completed. NMB also said that it is taking actions to fully implement the remaining recommendations from our prior reports concerning its rail arbitration case backlog, organizational climate assessment, travel and telework policies, and information privacy and security. We are sending copies of this report to the appropriate congressional committees, NMB, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or nguyentt@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. NMB appointed a senior agency official for privacy in April 2019 and documented the assignment through a memorandum. Partially following NMB established a privacy policy dated October 2017 that includes procedures for protecting sensitive information, including personally identifiable information. However, the policy reflects outdated roles and responsibilities. For example, the policy reflects the role of chief of staff that no longer exists in the agency. An NMB official stated the agency engaged a technical writer (contractor) to update the policy by the end of fiscal year 2020. Partially following The NMB documented a privacy impact assessment dated July 2018. assessments for systems containing personally identifiable information 4. However, the assessment did not specify whether a system of records notice would be developed as required by the Office of Management and Budget (OMB). An NMB official stated the agency engaged an information system security officer (contractor) to address this practice by the end of fiscal year 2020. NMB did not issue a system of records notice for its enterprise network and did not provide any documentation that this notice was not required in the agency’s privacy impact assessment. An NMB official stated the agency engaged an information system security officer (contractor) to address this practice by the end of fiscal year 2020. Partially following NMB developed an Information Program Plan dated April 2016 that states the agency annually conduct a risk analysis. NMB had assessments of its enterprise network conducted on May 2016 and November 2017. NMB also completed an information system risk assessment dated October 2017 that identifies and describes threats. However, NMB did not provide any assessment documentation for its network in fiscal year 2019. An NMB official stated the agency engaged a security assessor (contractor) to address this practice by the end of fiscal year 2020. Partially following NMB has developed an information security policy by documenting its based policies and procedures to ensure compliance with applicable standards and guidance including system configuration requirements existing April 2016 Information Program Plan. While the policy includes risk assessment requirements, it does not reflect oversight of NMB third- party providers. An NMB official stated that the agency engaged a technical writer (contractor) to address this practice by the end of fiscal year 2020. Partially following NMB’s current system security plan for its enterprise network has been in that cover networks, facilities, and systems or groups of systems, as appropriate place since March 2016. However, the plan does not include full implementation details on operational controls or a rationale on why controls are not applicable as recommended in National Institute of Standards and Technology guidance. An NMB official stated that the agency engaged an information system security officer (contractor) to address this practice by the end of fiscal year 2020. Extent NMB is following Partially following NMB has security awareness training guidelines signed April 2016 that specify agency employees and contractors will receive annual security awareness training. An NMB official stated that security awareness training is to be conducted each fiscal year. However, an NMB official stated the agency did not provide security awareness training in fiscal year 2018. NMB provided that training in fiscal year 2019, and an NMB official said the agency engaged an information system security officer (contractor) to address this practice by the end of fiscal year 2020. In May 2016, the NMB’s enterprise network was independently tested by the Department of the Treasury’s Bureau of Fiscal Service Division of Security Services. In addition, an NMB official documented a security assessment for the network signed November 2017. However, NMB did not provide us with any additional documentation to show the enterprise network was assessed in fiscal year 2019. According to an NMB official, the agency engaged a security assessor (contractor) to address this practice by the end of fiscal year 2020. Program Plan dated April 2016. In addition, the agency documented a plan of actions for its enterprise network dated January 2018. However, the plan of actions did not fully meet OMB requirements such as planned completion dates and changes to milestones, among other things. An NMB official stated that the agency engaged an information system security officer (contractor) to address this practice by the end of fiscal year 2020. Partially following NMB’s security-incident procedures dated June 2016 include information procedures for detecting, reporting, and responding to incidents on handling cyber incidents. However, the procedure did not include required actions specified by the Federal Information Security Modernization Act of 2014, such as notifying the federal information security incident center, law enforcement agencies, and relevant offices of inspector general and general counsel. An NMB official stated the agency engaged a technical writer (contractor) to address this practice by the end of fiscal year 2020. Partially following NMB documented a continuity of operations plan policy dated March 2016. However, the agency has not documented a contingency plan for its enterprise network. An NMB official stated the agency engaged an information system security officer (contractor) to address this practice by the end of fiscal year 2020. covered by information in a system of records, the category of records that are maintained about the individuals, and how the information is shared and routinely used by the agency. In addition to the contact named above, Mary Crenshaw (Assistant Director), Andrew Nelson (Analyst-In-Charge), Cindy Brown Barnes, Larry Crosland, Mikey Erb, Chelsa Gurkin, John Lack, and Dana Pon made significant contributions to this report. Also contributing to this report were Shirley Abel, Amy Anderson, Bill Anderson, J. Howard Arp, Gary Bianchi, Rachael Chamberlin, Vijay D’Souza, Robert Graves, Carol Henn, Janice Latimer, Barbara Lewis, Benjamin Licht, Jessica Orr, Monica Perez- Nelson, James Rebbe, Constance Satchell, Monica Savoy, Almeta Spencer, Sabrina Streagle, Barbara Steel, Amy Sweet, Curtia Taylor, Candice Wright, and Paul Wright.", "summary": "NMB was established under the Railway Labor Act to facilitate labor relations for airline and railway carriers by mediating and arbitrating labor disputes and overseeing union elections. The FAA Modernization and Reform Act of 2012 included a provision for GAO to evaluate NMB programs and activities every 2 years. GAO's previous reports, issued in December 2013, February 2016, and March 2018, included 13 recommendations for NMB based on assessments of policies and processes in several management and program areas. NMB had implemented six of those recommendations previously, leaving seven for our review. This fourth report examines the (1) extent to which NMB has taken actions to fully implement GAO's remaining recommendations, and (2) other challenges NMB faces in key management areas and in overseeing its operations. GAO reviewed relevant federal laws, regulations, and NMB documents, such as its travel and telework policies; examined arbitration caseload data and the results of NMB's 2019 Organizational Climate Assessment; and interviewed NMB officials. The National Mediation Board (NMB), which facilitates labor relations for airline and railway carriers, has implemented one of GAO's seven recommendations remaining from past reports (see table). Specifically, NMB has developed a policy to prevent violations of ethics rules regarding outside employment and monitors compliance with that policy. NMB has not yet fully implemented the other six recommendations. For example, NMB has developed some strategies to reduce its arbitration case backlog, but lacks a plan with goals and time frames to complete that work. Similarly, NMB has completed an organizational climate assessment, but still must take additional actions to address employee concerns. By not fully implementing these and other recommendations, NMB remains at risk of not fulfilling its mission in several key areas, including information security and organizational climate. In this review, GAO found that, in addition to the six unimplemented recommendations, NMB lacks internal controls to effectively manage and oversee its appropriations and consistently follow its audit policies. NMB officials said the agency needed its full funding to address various agency priorities, such as hiring information technology specialists, but NMB did not use all of its funding for fiscal years 2016 through 2019, leaving a total of more than $4 million unobligated from those years; those funds are not available to NMB for new obligations. Officials said that hiring challenges and uncertainty concerning the agency's final appropriations made managing its budget resources difficult. NMB has a new process to monitor its budget resources, but has not documented that process. Without documenting that process, NMB may not be certain it uses its funding effectively to achieve its hiring and other goals. Additionally, NMB has not consistently followed its audit policy to address deficiencies identified in financial and other audits. For example, NMB did not create specific corrective action plans to address findings from financial or GAO audits. The NMB Board said it relied on senior managers to follow procedures, but the Board is ultimately responsible for ensuring that its managers implement the internal control system. Without a process to effectively oversee and evaluate its adherence to internal controls and its own audit policies, NMB may miss opportunities to achieve objectives, address audit deficiencies, and improve management oversight. GAO is making four recommendations, including that NMB document its process for reviewing and monitoring the agency's annual appropriations to ensure effective use of funds, and establish a process for the Board to effectively monitor and evaluate NMB's adherence to audit policies. NMB agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "By the end of fiscal year 2019, the federal debt held by the public had climbed to 79 percent of GDP. By comparison, such debt has averaged 46 percent of GDP annually since 1946. If current trends continue, debt as a share of GDP will exceed the historic high 1946 level of 106 percent of GDP within 11 to 14 years. In 2050, it will be nearly twice that level and about four times its post-World War II average. Figure 1 shows that in GAO, CBO, and 2019 Financial Report projections, debt held by the public as a share of GDP grows substantially over time. Under GAO, CBO, and the 2019 Financial Report projections, spending for the major health and retirement programs grows more rapidly than GDP in coming decades. This is a consequence of both an aging population and projected continued increases in health care costs per beneficiary. Medicare spending is expected to exceed $1 trillion per year by fiscal year 2026, and Social Security spending already exceeds $1 trillion per year. However, according to the projections, these spending categories will eventually be overtaken by spending on net interest, which primarily consists of interest costs on the federal government’s debt held by the public. In recent years, persistently low interest rates have resulted in lower interest costs for the government than previously projected. Despite these low interest rates, spending on net interest grew from $263 billion in 2017 to $376 billion in 2019. That $376 billion is 8.4 percent of total federal spending, which exceeded combined spending on agriculture, transportation, and veterans’ benefits and services. Going forward, both interest rates and the debt are projected to grow, which means spending on net interest is projected to grow faster than any other component of the budget. In 2032, spending on net interest is projected to exceed $1 trillion annually. Over the past 50 years, net interest costs have averaged 2 percent of GDP but these costs are projected to increase to 7.2 percent by 2049. As figure 2 shows, we project that as a share of GDP, net interest spending will exceed Medicare spending in 2041, Social Security spending in 2044, and total Discretionary spending in 2049. Interest costs will also depend in part on the outstanding mix of Treasury securities. The Department of the Treasury issues securities in a wide range of maturities to appeal to a broad range of investors to support its goal of borrowing at the lowest cost over time. Treasury refinances maturing debt by issuing new debt in its place at the prevailing interest rate. At the end of fiscal year 2019, 61 percent of the outstanding amount of marketable Treasury securities held by the public (about $9.9 trillion) was scheduled to mature in the next 4 years. If interest rates are higher, Treasury will have to refinance these securities at the higher interest rates, adding to the interest costs of the growing federal debt. Impending financial challenges for major programs and fiscal risks are both straining the federal budget and contributing to the growing debt. Sustaining key programs will require changes (see fig. 3). The President’s Budget, CBO, and the Chair of the Board of Governors of the Federal Reserve System all make it clear that rising federal debt could have long-term consequences for the economy. For example it could: constrain Congress’s ability to support the economy or address other national priorities, restrain private investment and thereby reduce productivity and overall growth, and erode confidence in the U.S. dollar. In addition, it may increase the risk of a fiscal crisis, in which investors would lose confidence in the U.S. government’s financial position, and interest rates on Treasury securities would increase abruptly. To change the long-term fiscal path, policymakers will need to consider policy changes to the entire range of federal activities, both revenue (including tax expenditures) and spending (entitlement programs, other mandatory spending, and discretionary spending). As Congress considers changes in revenue and spending policies to improve the federal government’s long-term fiscal path, it will also need to consider other approaches for managing the level of debt. As currently structured, the debt limit is a legal limit on the total amount of federal debt that can be outstanding at one time. The debt limit does not restrict Congress’s ability to pass spending and revenue legislation that affects the level of debt, nor does it otherwise constrain fiscal policy. Without legislation to suspend or raise the debt limit, Treasury cannot continue issuing debt to finance the decisions already enacted by Congress and the President. We have reported on the negative impacts of uncertainty around the debt limit which include (1) increased Treasury borrowing costs, (2) decreased demand for Treasury securities, and (3) constrained Treasury cash management. We have reported numerous times that the full faith and credit of the United States must be preserved. We have also recommended that Congress consider other approaches to the current debt limit to avoid seriously disrupting the Treasury market and increasing borrowing costs and to allow it to better manage the federal government’s level of debt. A number of bills have been introduced in this Congress to address this issue. The Senate Budget Committee’s proposal to reform the Congressional budget process would automatically adjust the debt limit to conform to levels established in the budget resolution. In contrast to the debt limit, fiscal rules can support efforts to achieve fiscal sustainability by imposing numerical limits or targets on the budget to guide fiscal policy. Fiscal rules are intended to influence decisions about spending and revenue as they are made. The Senate Budget Committee’s proposal to reform the Congressional budget process is an example of one such approach. This legislation would specify target ratios for debt as a share of GDP and track legislation against that target. As Congress continues to consider options, two key points should be emphasized. An agreed-upon goal can help policymakers justify and frame their choices. With that in mind, a fiscal target that establishes a common goal for controlling the size of the federal debt relative to the economy—as well as well-designed rules that put the federal government on a path to achieve that target—could form part of a long-term fiscal plan to put the government on a sustainable fiscal path. The longer action is delayed, the greater and more drastic the changes will have to be, placing an additional burden on future generations. While changes in spending and revenue to ensure long-term fiscal sustainability require legislative actions to alter fiscal policies, executive agencies can also take actions to contribute toward a sustainable fiscal future. Although executive actions alone cannot put the U.S. government on a sustainable fiscal path, it is important for agencies to act as stewards of federal resources. These actions include reducing improper payments, which agencies estimate totaled $175 billion in fiscal year 2019; addressing the $381 billion annual net tax gap; better managing fragmentation, overlap, and duplication across the federal government; and improving information on federal programs and fiscal operations to aid agency decision-making. Chairman Enzi, Ranking Member Sanders, and Members of the Committee, this completes our prepared statement. We would be pleased to respond to any questions that you may have. For further information on this testimony, please contact Susan J. Irving, Senior Advisor to the Comptroller General, Debt and Fiscal Issues, who may be reached at (202) 512-6806 or IrvingS@gao.gov; Robert F. Dacey, Chief Accountant, who may be reached at (202) 512-3406 or daceyr@gao.gov; or Dawn B. Simpson, Director, Financial Management and Assurance, who may be reached at (202) 512-3406 or simpsondb@gao.gov. Contact points for our Congressional Relations and Public Affairs offices may be found on the last page of this statement. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "This testimony summarizes information contained in GAO's March 2020 report, entitled The Nation’s Fiscal Health: Action Is Needed to Address the Federal Government’s Fiscal Future ( GAO-20-403SP ). Long-term fiscal projections by GAO, the Congressional Budget Office (CBO), and in the 2019 Financial Report of the U.S. Government (2019 Financial Report) all show that, absent policy changes, the federal government continues to face an unsustainable long-term fiscal path. Although the assumptions in each of these projections vary somewhat, all result in the same conclusion: over the long term, the imbalance between spending and revenue that is built into current law and policy will lead to (1) deficits exceeding $1 trillion each year beginning in fiscal year 2020 and (2) both the annual deficit and the cumulative total debt held by the public continuing to grow as shares of gross domestic product (GDP). This situation—in which debt grows faster than GDP—means the current federal fiscal path is unsustainable. To change the long-term fiscal path, policymakers will need to consider policy changes to the entire range of federal activities, both revenue (including tax expenditures) and spending (entitlement programs, other mandatory spending, and discretionary spending). As Congress considers changes in revenue and spending policies to improve the federal government’s long-term fiscal path, it will also need to consider other approaches for managing the level of debt.", "document_type": "gao"}
{"report": "Congress first enacted Section 653(a) in the Foreign Assistance Act of 1971. According to Section 653(a), “not later than thirty days after the enactment of any law appropriating funds to carry out any provision of this Act (other than section 451 or 637) or the Arms Export Control Act, the President shall notify the Congress of each foreign country and international organization to which the United States Government intends to provide any portion of the funds under such law and of the amount of funds under that law, by category of assistance, that the United States Government intends to provide to each.” To provide Congress with the mandated data within the mandated time frame, State and USAID officials review the annual appropriations act and the accompanying joint explanatory statement to identify the congressional instructions contained within them. Although State has the delegated authority to approve the programming of foreign assistance funds and is charged with submitting the Section 653(a) report to Congress, State and USAID have shared responsibilities regarding the administration of certain foreign assistance accounts. Throughout this report we refer to the congressional instructions in the annual appropriations acts and those allocation tables within the joint explanatory statements that are incorporated by reference into the act as “requirements,” and we refer to Congress’s instructions to the agencies presented as additional language in the joint explanatory statement as “directives.” Congress funds foreign assistance by appropriating funds to 16 accounts, each of which has a distinct purpose and specific legal requirements, such as the number of years the funds are available for obligation. Table 1 provides a summary of these 16 accounts. These accounts are generally administered individually by State or USAID, or jointly by both agencies. In addition, the period of availability for obligation for these accounts ranges from 1 to 5 years, or in some cases, until funds are expended. The annual appropriations acts have hundreds of specific instructions— both requirements and directives—attached to many of the foreign assistance accounts that State and USAID address in the Section 653(a) report. According to State officials, the annual appropriations acts have become more detailed since the addition of the Section 653(a) mandates in the Foreign Assistance Act of 1971. For example, State officials said that when the Section 653(a) reporting mandate began the annual appropriations act contained fewer requirements and directives. The Foreign Assistance and Related Programs Appropriations Act, 1971, appropriated $2.2 billion in foreign assistance and was 18 pages in length. By contrast, the relevant portion of the annual appropriations act for fiscal year 2018 appropriated $33.7 billion in foreign assistance and was 138 pages in length. In addition, the accompanying joint explanatory statement was 9 pages in 1971 and 31 pages in 2018. As shown in table 2 below, during the 4 fiscal years covered by our review, the total number of requirements and directives addressed in the Section 653(a) reports has varied depending on congressional instructions within the annual appropriations acts and corresponding joint explanatory statements. For example, fiscal year 2016 had 1,056 total requirements and directives, while fiscal year 2018 had 657. The total number of requirements and directives has also varied by account. For instance, the Economic Support Fund, which was appropriated roughly $3.9 billion in fiscal year 2018, had 107 requirements and directives that instructed agencies how to allocate about $2.9 billion. Other accounts appropriated funds in fiscal year 2018 had fewer requirements and directives. For example, the International Disaster Assistance account, which was appropriated about $4 billion in fiscal year 2018, had three requirements and directives. The requirements and directives also vary in their specificity. For instance, of the $8.6 billion appropriated for Global Health Programs in fiscal year 2018, the joint explanatory statement required that $829.5 million be allocated toward maternal and child health. State and USAID were also required to make funds allocated for the Global Health Programs available in specific amounts, such as making $755 million available for activities addressing malaria in fiscal year 2018. State and USAID also balance the requirements and directives with administration priorities. For example, officials from OMB said that they review the Section 653(a) report to ensure that allocations for certain countries— such as Israel and Jordan—are consistent with the administration’s financial commitments to those countries. The text box below provides examples of the types of requirements and directives found in the fiscal year 2018 appropriations act. Foreign Assistance Requirements and Directives Congress includes a variety of instructions to the agencies managing foreign assistance funds in statutes, such as the annual appropriations acts, and in legislative history, such as the joint explanatory statements. These instructions come in two broad categories. Requirements: Congress’s instructions to agencies as contained in the annual appropriations act, including mandatory and non-mandatory spending, and tables within the joint explanatory statement that are required by statute. Directives: Congress’s instructions to agencies presented as additional language in the joint explanatory statement that are not required by statute. Examples Mandatory requirement detailed in law: “Of the funds appropriated by this Act, not less than $400,000,000 shall be made available for water supply and sanitation projects pursuant to the Senator Paul Simon Water for the Poor Act of 2005 (Public Law 109-121), of which not less than $145,000,000 shall be for programs in sub-Saharan Africa, and of which not less than $15,000,000 shall be made available to support initiatives by local communities in developing countries to build and maintain safe latrines.” Mandatory requirement referenced in law but detailed in the joint explanatory statement: Law: “For necessary expenses to carry out the provisions of chapter 4 of part II of the Foreign Assistance Act of 1961, $1,816,731,000, to remain available until September 30, 2019.” Joint explanatory statement: “Funds for certain programs under this heading are allocated according to the following table:” Ambassador-at-Large for Global Women’s Issues Conflict and Stabilization Operations Disability Programs Disability Programs Family Planning/Reproductive Health (U.S. Agency for International Development) House Democracy Partnership Organization of American States Polio Reconciliation Programs Trade Capacity Building 1,900,000 9,000,000 7,500,000 12,000,000 10,000,000 Nonmandatory requirement detailed in law: “Of the funds appropriated by this Act under the heading ‘Economic Support Fund,’ up to $112,500,000 may be made available for assistance for Egypt, of which not less than $35,000,000 should be made available for higher education programs including not less than $10,000,000 for scholarships for Egyptian students with high financial need to attend not-for-profit institutions of higher education.” Nonmandatory directive detailed in the joint explanatory statement: “The State Department Secretary and U.S. Agency for International Development Administrator are directed to provide no assistance to the central Government of the People’s Republic of China under Global Health Programs, Development Assistance, and Economic Support Fund, except for assistance to detect, prevent, and treat infectious diseases.” In addition, requirements and directives can be specific to a country or organization, specific to a sector, or be cross-cutting such that they may be applicable across multiple countries and accounts—as the examples above demonstrate. State submitted Section 653(a) reports that provided mandated data notifications on how foreign assistance funds are allocated by country and account; however, State did not submit the reports within the mandated time frame during fiscal years 2015 through 2018. During the years covered in our review, State’s Section 653(a) reports provided information to Congress on the tens of billions of dollars for foreign assistance accounts specified in the annual appropriations act. In addition to detailing the category of assistance by account, the Section 653(a) reports further delineated funding by the countries and international organizations to which the foreign assistance was directed. State also included supplemental spreadsheets with the reports that outlined how the various requirements and directives in the annual appropriations act were to be addressed. In fiscal years 2015 through 2018, State submitted Section 653(a) reports an average of 169 days after the enactment of the annual appropriations act (or 139 days late). During those 4 fiscal years, State submitted Section 653(a) reports from 80 to 230 days past the 30-day mandated time frame for reporting, as shown in figure 1. During the 4 fiscal years covered by our review, State took the longest amount of time to submit the Section 653(a) report in fiscal year 2015. State officials explained that in fiscal year 2015 and prior years it generally took them longer to submit the report because they first submitted a draft Section 653(a) report to the House and Senate appropriations committees. According to State officials, the appropriations committees’ majority and minority staff then engaged in negotiations with each other and with State on the draft to reach agreement on the final allocation of funds. In addition, the fiscal year 2015 appropriations act allowed State to propose deviations from the requirements in the joint explanatory statement. Thus, the agencies submitted the draft report to the appropriations committees with allocations that, in some cases, varied from the levels Congress included in the tables in the joint explanatory statement. State submitted the draft Section 653(a) report for the fiscal year 2015 appropriations act in April 2015 with the proposed deviations and engaged in a 5-month negotiation process to finalize the allocation of funds in September 2015. According to State, Congress changed the Section 653(a) reporting requirements in the fiscal year 2016 appropriations act to forestall the months-long negotiation process with the appropriations committees that had occurred in prior years. The fiscal year 2016 appropriations act authorized State and USAID to deviate in their allocations by up to 5 percent from the mandated amounts in the tables of the joint explanatory statement. This change allowed State to submit the report in a more timely fashion than in fiscal year 2015. By specifying how much leeway State and USAID were allowed in their allocations, the agencies were able to develop their plans without submitting a draft Section 653(a) report and seeking further agreement from the appropriations committees. As a result, officials submitted the 2016 report in 110 days, compared with 260 days in 2015. We found that delays in submitting the Section 653(a) report were primarily attributable to State’s complex process to address appropriation requirements and directives while also reflecting administration priorities, as well as to data collection weaknesses. Nevertheless, State has not reviewed its process to identify and address such issues and other potential inefficiencies. Absent such a review, State is not in a position to improve its process to meet the 30-day mandate. In fiscal year 2018, State officials noted that reaching agreement on priorities within the new administration and staff vacancies also adversely affected the timeliness of the Section 653(a) report submission. State has a multistep process to provide the mandated Section 653(a) report. Pre-appropriation preparatory work. According to State documentation, the process for responding to Section 653(a) mandates begins with State and USAID developing notional allocation estimates in a spreadsheet before the upcoming fiscal year’s annual appropriations act is passed. Allocation analysis and development. After the act is passed, USAID and State review their allocation estimates against the requirements and directives in the act and adjust their spreadsheet containing allocation estimates as necessary, taking into consideration policy direction from State and USAID leadership. According to State and USAID officials, detailed congressional instructions for particular accounts can limit the agencies’ ability to allocate funds according to the administration’s priorities and to consider country-specific foreign assistance needs. For example, Congress appropriated $876 million in fiscal year 2018 for the Nonproliferation, Anti-Terrorism, Demining, and Related Programs account and included 40 associated requirements and directives. State and USAID officials added that in order to satisfy all the requirements and directives they sometimes have to allocate appropriated amounts to address more than one requirement or directive. For instance, in the fiscal year 2018 Section 653(a) report, some of the funds allocated to meet an appropriation requirement for conventional weapons destruction were also designated as an allocation to satisfy a different requirement for humanitarian demining. Further, some amount of those funds satisfied a more specific requirement for humanitarian demining in Laos. State officials noted that by allocating appropriated amounts to more than one requirement or directive, they have greater flexibility to address administration priorities, while also meeting congressional instructions. Allocation negotiation, review, and agreement. State and USAID ensure that input from all the various parties is taken into consideration when further developing allocations. About 200 State and USAID bureaus and overseas posts review the allocations and propose changes in their copies of the spreadsheet that are then returned to State’s Office of U.S. Foreign Assistance Resources. According to State and USAID officials, they consider the proposed changes in light of emerging issues in selected foreign countries that may lead to the redirection of or changes to the proposed allocation of funds. State and USAID also review the proposed changes with agency leadership. OMB review and Section 653(a) report finalization and transmission. Once State and USAID agree on changes to the allocations, State submits the Section 653(a) report to OMB to be reviewed against the policy direction of the Executive Office of the President. State officials indicated that OMB feedback must be resolved before finalizing allocations. Concurrent with OMB’s review, State begins the process of finalizing allocation levels. Once State and USAID’s allocations are complete, State provides final allocation levels to bureaus and overseas posts and submits the Section 653(a) report to the relevant appropriations subcommittees. Given its complexity, State’s process is not designed to meet the mandated 30-day time frame. For example, in fiscal year 2018, State planned to complete the Section 653(a) process in 85 days. Figure 2 below outlines the stages of the Section 653(a) report development process and the targeted number of days for each stage during fiscal year 2018. The data developed for the Section 653(a) report plays a critical role in the obligation of tens of billions of dollars in foreign assistance funds appropriated annually. According to State and USAID officials, the agencies are constrained from obligating funds until the report is completed because a number of pre-obligation requirements are based on allocations in the Section 653(a) report. While the submission of the Section 653(a) report does not legally affect State’s ability to obligate foreign assistance funds, according to State and USAID officials, consultations, spend plans, and congressional notifications cannot be completed until allocation amounts are finalized through the Section 653(a) process. As a result, State and USAID officials said the amount of time it takes to submit the Section 653(a) report affects the obligation of funds. State officials indicated that their process for collecting appropriations- related feedback and information from various offices, bureaus, and overseas posts necessitates significant staff time to correct data entry errors. Throughout the Section 653(a) process, State officials use a spreadsheet to consolidate information. For example, after State develops its initial allocations in the spreadsheet, it sends the spreadsheet out to about 200 bureaus and overseas posts to review and make appeals related to account and country allocations, which State and USAID then take into consideration as they continue to modify the allocations. According to State officials, reviewing suggested changes to allocations from about 200 bureaus and overseas posts is time consuming. This process is further complicated when they sometimes find mistakes in the returned spreadsheets, such as incorrect formulas and currency formats. Occasionally, returned spreadsheets also include additional data columns that were not in the original documents. Such discrepancies make it difficult to merge and process all of the suggested changes and identify how the changes interact with the various requirements and directives. State officials said that these discrepancies occur because they do not have controls in place to prevent modification of the spreadsheet. For example, the formulas and format of the spreadsheet can be manipulated by the various individuals reviewing the document. In addition, the spreadsheet does not automatically verify that the changes proposed by the bureaus and overseas posts comply with the requirements and directives. Instead, officials have to individually compare the changes with the requirements and directives and ensure that they are in compliance. State officials indicated that it takes them time and resources to discover and correct the errors, merge all of the spreadsheets, and ensure compliance, which contributes to delays in developing the Section 653(a) report. According to State’s Foreign Affairs Manual, State must maintain effective systems of management control that are designed to provide reasonable assurance regarding the prevention of or prompt detection of errors and irregularities. State officials indicated that while they do correct errors and validate the data in the Section 653(a) report for accuracy before final submission, it takes time and resources to do so, which adds to the total amount of time it takes to produce the report. Given the individual account and country allocations, and number of stakeholders involved in providing feedback, State officials acknowledged that their spreadsheet-based system is inadequate for the complexity of the task. State officials said that their existing data information system—the Foreign Assistance Coordination and Tracking System Info Next Generation—could potentially be modified to automate the distribution and collection of appropriations-related feedback from their offices and overseas posts, as well as to ensure that the changes comply with the annual appropriations act’s requirements and directives. Currently, State uses this system during the last phase of the Section 653(a) process to input the final allocations and share the Section 653(a) report with bureaus and overseas posts and the appropriations committees. While State officials said that they are exploring options to improve this process, they have not yet decided how to address weaknesses in their data collection system. State officials said that the Section 653(a) process that they developed is necessary to address congressional instructions and administration priorities and because they use the allocations in the report as a basis for spend plans required for the obligation of funds. Federal standards for internal control state that management should set objectives to meet the requirements of applicable laws and regulations. State officials noted that it might be possible to meet the 30-day mandate but that doing so would be inefficient because the subsequent report would need major revisions before finalizing allocations. According to officials, that alternative process, while it would meet the 30-day mandate, would be likely to further delay the development of spend plans and obligation of funds. State officials told us that they have informally suggested to the congressional appropriations committees that the mandated time frame for delivering the report should be extended, but they said they have not formally requested that Congress amend the 30-day reporting mandate. State officials said that they would also need to engage in conversations with authorizing committees responsible for making changes to the reporting mandate in the Foreign Assistance Act. GAO’s guidance on business process reengineering states that agencies should model their processes to identify problem areas and non–value- added activities that need to be changed or eliminated, such as excessive reviews. State officials said that, while they have made adjustments to improve their Section 653(a) process, they have not conducted a systematic review of their process since it changed in fiscal year 2016. Such a review could identify changes to expedite the completion and submission of the mandated report. Given that State’s process is not designed to meet the Section 653(a) 30-day reporting mandate, absent changes to its processes or Section 653(a), State is unlikely to meet the 30-day reporting mandate in the future. According to State officials, reaching agreement on administration priorities affected the timeliness of their Section 653(a) report in fiscal year 2018. The current Secretary of State and USAID Administrator both had their first experience with the Section 653(a) process during fiscal year 2018, which led to more detailed review within both agencies than in previous years, according to State officials. In addition, State officials said that USAID recommended unanticipated and significant changes to the proposed allocations before OMB’s review. USAID officials said that significant changes were necessary since USAID disagreed with the allocations State proposed for USAID’s appropriations within the Global Health Programs, Development Assistance, and the Economic Support Fund accounts. In fiscal year 2018, it took State and USAID 110 days to complete the allocation negotiation, review, and agreement step of the Section 653(a) process. According to State and USAID officials, they used 46 of the 110 days to reach agreement on the changes that USAID proposed. OMB officials noted that they needed to resolve policy issues concerning the administration’s foreign assistance priorities, which also contributed to delays. Once State sent the report to OMB in August 2018, OMB officials said that they approved the report after 36 days. In previous years, OMB officials explained that they usually approved the report within 15 days. However, they said that they were working to resolve a policy issue with other offices in the Executive Office of the President, and were therefore delayed in approving the fiscal year 2018 report. In total, it took State and USAID 189 days to produce the Section 653(a) report in fiscal year 2018. In fiscal year 2018, State planned to complete the Section 653(a) process in about 85 days after the enactment of the appropriations act—which exceeds the 30-day reporting, as shown in figure 3. In 2018, staffing gaps in State’s Office of U.S. Foreign Assistance Resources also affected the development of the Section 653(a) report. State’s Office of U.S. Foreign Assistance Resources is staffed by State and USAID-funded personnel and provides supervision and direction of State and USAID’s foreign assistance funding and programs. While the development of the Section 653(a) report is a critical task for the Office of U.S. Foreign Assistance Resources, the office and its staff are also responsible for developing a U.S. foreign assistance strategy, annual country-specific assistance operational plans, consolidated strategic and program plans, and operational budgets. State’s Office of U.S. Foreign Assistance includes two subordinate offices involved in developing the Section 653(a) report, both of which had vacancies in 2018. Within the Office of U.S. Foreign Assistance, the Resources and Appropriations office has primary responsibility for reviewing and identifying the Section 653(a) requirements and directives, but in 2018, five of 13 full-time equivalent positions, or 38 percent, were vacant. In addition, 10 of 41 full-time equivalent positions within the Regional and Global Affairs office, or 24 percent, were vacant. This office provides geographic and functional expertise to help develop and adjudicate allocations for the report. According to State officials, these vacant positions affected the timeliness of the Section 653(a) report in 2018 because the staff in both these offices assist with developing the report throughout the Section 653(a) process. As previously shown in figure 3, most of the delays in the fiscal year 2018 process occurred during the allocation negotiation, review, and agreement phase—which relies heavily on officials from the offices experiencing staffing gaps. According to State officials, the staff shortfall affecting the development of the fiscal year 2018 Section 653(a) report was due to the State hiring freeze that affected the entire agency, as well as vacancies among USAID personnel assigned to State’s Office of U.S. Foreign Assistance Resources. State’s hiring freeze took effect in January 2017 and was lifted in May 2018. In addition, State officials said that USAID has not filled USAID-funded vacancies within State’s Office of U.S. Foreign Assistance Resources. In fiscal year 2018, nine of the 13 full-time equivalent positions in State’s Resources and Appropriations office were funded by USAID, of which four were vacant, and 21 of the 41 full-time positions in Regional and Global Affairs office were funded by USAID, of which six were vacant. Our 2019 High-Risk Series report calls for agencies to design and implement action plans for closing skills gaps, which can include when an agency has an insufficient number of people to complete its work. The report states that the action plan should define the root cause of all skills gaps within an agency and provide suggested corrective measures, including steps necessary to implement solutions. State officials said that they have received permission to fill the vacant State positions, and USAID has provided permission to advertise two vacant USAID positions within State’s Office of U.S. Foreign Assistance Resources. Thus, that office is requesting additional State full-time equivalent positions. Despite these efforts, State and USAID officials said that they do not have an action plan to address the vacancies. Without a plan to fill these vacancies, a lack of staff resources will likely continue to impact the timeliness of the Section 653(a) reports. Congress appropriates tens of billions of dollars for foreign assistance annually and mandates the President to report to Congress on how the U.S. government will allocate funds for foreign countries, by category of assistance, within 30 days of the enactment of the annual appropriations act. State and USAID have developed a complex process to balance how their allocations will meet the detailed requirements and directives within the annual appropriations acts, the administration’s priorities, and country-specific foreign assistance needs. However, State has been unable to meet the mandated time frame for submitting the Section 653(a) report for various reasons. Most importantly, State’s process for completing the various phases of the Section 653(a) process is not designed to meet the mandated 30-day deadline. Moreover, State officials have not systematically reviewed their process since it changed in fiscal year 2016, to identify areas that can be streamlined or eliminated to expedite the completion and submission of the report. Additionally, State’s system for collecting input on foreign assistance allocations from its various offices, bureaus, and overseas posts is prone to data entry errors that take extra time to correct, contributing to delays in submitting the Section 653(a) report. Further, State’s two offices primarily responsible for managing the Section 653(a) process had a substantial number of positions vacant in 2018 but did not have a formal plan to address the resulting skills gaps. Absent addressing these challenges, State and USAID will likely continue to be in violation of their legal mandate for submitting Section 653(a) reports to Congress within 30 days after the annual appropriations act is enacted. We are making a total of three recommendations to State. The Secretary of State should ensure that the Director of State’s Office of U.S. Foreign Assistance Resources conducts a review of the Section 653(a) process to identify process steps that can be streamlined or eliminated and determine the time frame needed to prepare the annual Section 653(a) report. If State determines that the time frame exceeds 30 days, the office should coordinate with other appropriate officials to submit a legislative proposal to Congress to extend the mandated time frame for submitting Section 653(a) reports. (Recommendation 1) The Secretary of State should ensure that the Director of State’s Office of U.S. Foreign Assistance Resources improves the data collection from the many sources contributing to the Section 653(a) reports, such as by enhancing their data information systems. (Recommendation 2) The Secretary of State should develop a plan to address vacancies within State’s Office of U.S. Foreign Assistance Resources, consulting with the USAID Administrator as appropriate. (Recommendation 3) We provided a draft of this report to State, USAID, and OMB for review and comment. State and USAID provided written comments about the draft, which are reprinted in appendix II and appendix III, respectively. State also provided technical comments about the draft report, which we incorporated as appropriate. OMB did not provide comments on the draft report. State concurred with our three recommendations. USAID concurred with our first two recommendations; however, USAID’s written comments indicate that they do not believe staffing shortages at State were responsible for the chronic delays in the submission of the Section 653(a) report. While we do not report that staffing gaps were the primary reason for State not meeting reporting deadlines, we did find them to be a contributing factor to the delays in fiscal year 2018. State officials indicated that staffing gaps in their Office of U.S. Foreign Assistance Resources affected the development of the Section 653(a) report, contributing to delays. We are sending copies of this report to the appropriate congressional committees, the Secretary of State, the Administrator of USAID, the Acting Director of OMB, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6881 or bairj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines (1) the extent to which the Department of State (State) met the data notification and timeliness mandates under Section 653(a) of the Foreign Assistance Act of 1961 (Foreign Assistance Act) for fiscal years 2015 through 2018, and (2) the factors that affected State’s ability to address Section 653(a) mandates for fiscal years 2015 through 2018. To examine the extent to which State has met the data notification and timeliness mandates under Section 653(a) of the Foreign Assistance Act, we reviewed State’s Section 653(a) reports for fiscal years 2015 through 2018 to assess whether they documented the amounts of U.S. foreign assistance to be provided to each foreign country and international organization, as well as the amounts provided by category of assistance. To determine the timeliness associated with the development and submission of State’s Section 653(a) reports for fiscal years 2015 through 2018, we also reviewed documentation to identify when the Department of State, Foreign Operations, and Related Programs Appropriations Acts were enacted, the mandated submission dates, and compared those dates with the dates that State submitted the reports to Congress. We used this information to generate a figure that shows the actual submission time frames for Section 653(a) reports during those years compared with the 30-day reporting mandate. We also interviewed officials from State, the U.S. Agency for International Development (USAID), and the Office of Management and Budget (OMB) to better understand how the agencies address the Section 653(a) mandates. To examine the factors that affected State’s ability to address Section 653(a) mandates for fiscal years 2015 through 2018, we reviewed State and USAID documents. We also interviewed State, USAID, and OMB officials to get their views on what factors, if any, affected the timeliness of the Section 653(a) reports. For those factors that we identified, we requested and analyzed additional information as described below. In reviewing State’s Section 653(a) process, we analyzed State and USAID guidance documents and reports developed to address Section 653(a) mandates. We reviewed State’s analyses that identified the requirements and directives in the annual appropriations acts and joint explanatory statements for fiscal years 2015 through 2018. These requirements and directives outline how the agencies should allocate the funding for programs and for countries and international organizations. In addition, we reviewed State’s and USAID’s guidance documents that outlined the Section 653(a) process. Based on this information, we summarized State’s process and developed a figure that shows the major steps of State’s process, as well as the amount of time that each step lasted during the development of the fiscal year 2018 Section 653(a) report. We assessed State’s process against federal standards for internal control, which state that management sets objectives to meet the requirements of applicable laws and regulations. We also assessed the process against GAO’s guidance on business process reengineering, which outlines best practices on how agencies should model their processes. To examine the quality of the data State collects during development of the Section 653(a) reports, we reviewed whether State’s analyses followed State’s Foreign Affairs Manual requirement that State must maintain effective systems of management control programs designed to provide reasonable assurance regarding the prevention of or prompt detection of errors and irregularities. We analyzed State’s reports on the requirements and directives in the annual appropriations acts from fiscal year 2015 through 2018. In addition, we validated a judgmental sample of the requirements and directives that State identified to ensure that they were in the applicable appropriations act, joint explanatory statement, and reports from the appropriations committees in the Senate and House of Representatives. Although we identified an error in the appropriated amount recorded for the fiscal year 2016 International Narcotics Control and Law Enforcement account, we did not find errors specific to the requirements and directives State identified. Therefore, we concluded that the analyses were sufficiently reliable for our purpose, and we used State’s analyses to determine the total number of requirements and directives. Moreover, we reviewed the fiscal year 2015 through 2018 appropriations acts to identify the amounts appropriated for the accounts included in the corresponding Section 653(a) reports. We also identified the purpose and the time frame during which the appropriations for each account were available for obligation in the fiscal year 2018 appropriations act. To examine the issue of staff vacancies in fiscal year 2018, we received data from State and USAID on staff vacancies in key offices involved in the development and submission of the Section 653(a) report. In addition, we interviewed State and USAID officials about vacancies and whether they had developed plans to address the vacancies. We assessed whether State had designed and implemented action plans for closing skills gaps, which could include gaps caused by having an insufficient number of people to complete its work—as described in our 2019 High- Risk Series report. We conducted this performance audit from December 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate, evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Thomas Costa (Assistant Director), Mason Thorpe Calhoun (Analyst in Charge), Katya E. Rodriguez, Ashley Alley, Faisal Amin, David Dayton, Neil Doherty, Justin Fisher, and Melissa Wolf made key contributions to this report.", "summary": "State and USAID were responsible for managing $33.7 billion in foreign assistance funds in fiscal year 2018. Section 653(a) of the Foreign Assistance Act of 1961 mandates the President to report to Congress, on an annual basis, funding allocations by foreign country and category of assistance within 30 days of Congress appropriating certain funds. State, in coordination with USAID, makes decisions on how to allocate the funds, taking into consideration congressional instructions, the administration's priorities, and country-specific foreign assistance needs. GAO was asked to review State and USAID's process to respond to Section 653(a). This report examines (1) the extent to which State met the mandates under Section 653(a) for fiscal years 2015 through 2018 and (2) factors that affected State's ability to address the mandates. GAO reviewed annual appropriations acts and Section 653(a) reports submitted during fiscal years 2015–2018, and met with State, USAID, and Office of Management and Budget officials in Washington, D.C. The Department of State (State), through its Section 653(a) report, has provided Congress with information on the allocation of U.S. foreign assistance funds to foreign countries and international organizations by category of assistance as mandated, but the reports were not submitted within the mandated time frame. Specifically, in fiscal years 2015 through 2018, State submitted Section 653(a) reports from 80 to 230 days past the 30-day mandate, as shown in the figure. Multiple factors contributed to delays in submitting the Section 653(a) report. First, State has developed a multistep process for responding to hundreds of congressional instructions each year, while also reflecting administration priorities, which is not designed to meet the mandated time frame. This process involves coordination with the U.S. Agency for International Development (USAID), about 200 bureaus and overseas posts, and the Office of Management and Budget. Even though State's process is complex and does not meet the mandated time frame, State has not systematically reviewed its process since it revised the process in fiscal year 2016. State officials said that the process is necessary to address congressional instructions and administration priorities and because they use the allocations in the report as a basis for spend plans required to obligate funds. Second, a key part of State's process, involving data collection, has weaknesses that lead to discrepancies and hinder efficiency. According to federal internal control standards, agency data systems should provide quality data that is free from errors. However, State's mechanism for collecting information is a spreadsheet-based system susceptible to human errors, and State does not have appropriate controls in place to ensure data consistency. Third, in fiscal year 2018, staffing gaps also affected the development of the Section 653(a) report. State's two offices primarily responsible for managing the Section 653(a) process had 15 of 54 full-time equivalent positions vacant, which contributed to delays in submitting the Section 653(a) report, according to State officials. GAO has identified the filling of staffing gaps as a high-risk area that agencies should address. Unless State and USAID take steps to address these factors, they will continue to face challenges meeting their Section 653(a) requirements within the currently mandated time frame. GAO is making three recommendations to State: (1) conduct a systematic review of the Section 653(a) process to identify inefficiencies and determine the amount of time needed to prepare the Section 653(a) report, and if it exceeds 30 days, request that Congress extend the mandated time frame; (2) improve data collection; and (3) develop a plan to address staff vacancies, in consultation with USAID as appropriate. State concurred with these recommendations.", "document_type": "gao"}
{"report": "Congress passed TRIA in 2002 to address some of the challenges the insurance industry and businesses faced after the September 11 terrorist attacks. For example, after the attacks, insurers left the market, excluded terrorism risk coverage from policies, or steeply increased premiums. The Real Estate Roundtable reported in 2002 that nearly $16 billion of real estate projects in 17 states were stalled or cancelled because of the lack of coverage for terrorism risk (because many businesses are required to have coverage for terrorism risk as a condition for a mortgage loan). The purpose of TRIA is to (1) protect consumers by addressing market disruptions and ensuring the continued widespread availability and affordability of commercial property/casualty insurance for terrorism risk; and (2) allow for a transitional period for private markets to stabilize, resume pricing of such insurance, and build capacity to absorb any future losses, while preserving state insurance regulation and consumer protections. By law, an insurer’s coverage for terrorism losses must not differ materially from the terms, amounts, and other coverage limitations applicable to losses arising from other events. For example, an insurer offering $100 million in commercial property coverage also must offer $100 million in commercial property coverage for certified acts of terrorism. Insurers may charge a separate premium to cover their terrorism risk. TRIA requires insurers to make terrorism coverage on certain lines of property/casualty insurance (such as coverage for fire, workers compensation, and liability) available to commercial policyholders (such as businesses), although TRIA does not require commercial policyholders to buy it. The federal government does not collect an up-front charge from insurers for the government’s coverage of terrorism risk under TRIA. In a 2019 report, we noted that the federal government has multiple programs that can provide compensation to specific third parties if they suffer certain losses from future adverse events, and the federal government may not always charge premiums for accepting this risk of loss. However, under TRIA, the government must recoup at least some of its losses following a certified act of terrorism, as discussed below. TRIA has not caused financial liabilities to the federal government, but it could require large, previously unbudgeted expenditures by the federal government if an event occurred. For insurers to start submitting claims and receiving payments to cover terrorism losses, Treasury must first certify an event as an act of terrorism under TRIA. Certification requires the Secretary of the Treasury to evaluate the event based on two criteria: 1. Did the event meet the nonmonetary definition established under TRIA? Defining an event as an act of terrorism includes determining whether it was “committed by an individual or individuals as part of an effort to coerce the civilian population of the United States or to influence the policy or affect the conduct of the United States Government by coercion.” It also includes determining whether it was a “violent act or an act that is dangerous” to human life, property, or infrastructure, and whether it resulted in damage within the United States or certain areas outside the United States. As part of this determination, the Secretary of the Treasury must consult with the Attorney General and Secretary of the Department of Homeland Security before certifying an event. 2. Did the event cause at least $5 million in insurance losses in TRIA-eligible lines? TRIA prohibits the Secretary of the Treasury from certifying acts of terrorism unless insurance losses exceed this threshold. In 2004 Treasury issued regulations to implement TRIA’s procedures for filing insurer claims for payment of the federal share of compensation for insured losses. Within 7 days after certification of an act of terrorism, a Treasury contractor is to activate a web-based system for receiving claims from insurers and responding to insurers that seek assistance. The Terrorism Risk Insurance Program provides for shared public and private compensation for insured losses resulting from certified acts of terrorism. Under the current program, if an event were to be certified as an act of terrorism and insured losses exceeded $200 million, an individual insurer that experienced losses first would have to satisfy a deductible before receiving federal coverage. An insurer’s deductible under TRIA is 20 percent of its previous year’s direct earned premiums in TRIA-eligible lines. After the insurer pays its deductible, the federal government would reimburse the insurer for 80 percent of its additional losses and the insurer would be responsible for the remaining 20 percent. Annual coverage for losses is capped––neither private insurers nor the federal government cover aggregate industry insured losses in excess of $100 billion. After an act of terrorism is certified and once claims are paid, TRIA requires Treasury to recoup part of the federal share of losses in some instances. Under this provision, when insurers’ uncompensated insured losses are less than a certain amount (up to $41 billion for 2020), Treasury must impose policyholder premium surcharges on commercial property/casualty insurance policies until total industry payments reach 140 percent of any mandatory recoupment amount. When the amount of federal assistance exceeds this mandatory recoupment amount, TRIA allows for discretionary recoupment. Prior TRIA reauthorizations decreased federal responsibility for losses and increased private-sector responsibility for losses, but the 2019 reauthorization of TRIA made few changes to the program. For instance, the 2015 reauthorization required incremental decreases in the federal share of losses over 5 years (to 2020). The 2019 reauthorization extended the program to December 31, 2027 and proportionately adjusted the dates by which the Secretary must recoup policyholder surcharges to the new reauthorized time frame, but it did not change the federal share of losses. TRIA covers insured losses in eligible lines that result from a certified act of terrorism (see table 1). Many lines of commercial property/casualty insurance are eligible for TRIA, such as workers’ compensation, fire, and commercial multiple peril (multiperil) lines. States generally require that workers’ compensation insurance covers terrorism risk and do not permit exclusions, including for terrorism, according to Treasury. Workers’ compensation covers an employer’s liability for medical care and physical rehabilitation of injured workers and helps to replace these workers’ lost wages. TRIA also excludes certain lines (such as personal property and casualty insurance and health and life insurance). Terrorism coverage typically is embedded in all-risk property policies but also may be available in stand-alone policies, according to Treasury: Embedded. Most policyholders have terrorism risk insurance coverage embedded in a policy that covers other risks. Embedded policies are subject to TRIA’s “make available” requirements. In the event of a certified act of terrorism, policyholders would be covered if they have not declined terrorism coverage. Stand-alone. Stand-alone terrorism policies provide coverage only for terrorism risks. Insurers may provide stand-alone terrorism coverage through “certified” policies that are subject to TRIA terms and conditions and provide coverage only in the event of a certified act of terrorism. Alternatively, insurers may provide terrorism coverage through “noncertified” policies that do not meet TRIA terms and conditions. Such noncertified policies cover terrorism-related losses regardless of whether Treasury certifies an event, but losses paid by insurers would not be eligible for reimbursement under TRIA. Nonconventional terrorism risks generally include nuclear, biological, chemical, or radiological (NBCR) weapons, as well as cyber risks. Predicting losses associated with nonconventional risks can be particularly challenging because of the difficulty in predicting terrorists’ intentions and the potentially catastrophic losses that could result. TRIA is silent on NBCR and cyber risks, but Treasury has clarified how these nonconventional risks are covered under TRIA. In 2004, Treasury issued an interpretive letter clarifying that the act’s definition of insured loss does not exclude losses resulting from nuclear, biological, or chemical attacks, and does not preclude Treasury from certifying a terrorist attack involving such weapons. According to Treasury’s interpretive letter, the program covers insured losses from NBCR events resulting from a certified act of terrorism. However, for TRIA provisions to apply, insurers must provide coverage for those perils. Most insurers are not required to provide NBCR coverage and generally have attempted to limit their exposure to NBCR risks by largely excluding NBCR events from property and casualty coverage. In December 2016, Treasury issued guidance clarifying that, to the extent that insurers write cyber insurance in TRIA-eligible lines, the TRIA provisions apply. We further discuss Treasury’s guidance on cyber risk later in this report. TRIA authorizes Treasury to administer the Terrorism Risk Insurance Program. The Secretary of the Treasury administers the program with the assistance of Treasury’s Federal Insurance Office, according to Treasury officials. TRIA requires Treasury to conduct a biennial study of the effectiveness of the program. The 2015 TRIA reauthorization added a requirement that insurers submit information to Treasury about the coverage they write for terrorism risk, including the lines of insurance with exposure to such risk, the premiums earned on such coverage, and the participation rate for such coverage. The 2019 reauthorization added a requirement that Treasury report on the availability and affordability of terrorism risk insurance, including an analysis specifically for places of worship. Since 2016, Treasury has completed annual assessments of the program, including a report on the effectiveness of the program in June 2018. Treasury’s reports focused specifically on small insurers in June 2017 and June 2019. Treasury conducts an annual data call to collect information for the required studies and for purposes of analysis and program administration. Participation in the data call is mandatory for all insurers that write commercial property and casualty policies in lines of insurance subject to TRIA, subject to two exceptions. Treasury collects data separately for the following four groups of insurers: Small insurers have both a policyholder surplus and prior-year TRIA- eligible direct earned premium of less than five times the program trigger. Nonsmall insurers have policyholder surplus or the specified premiums above the small threshold and are not classified as captive or alien surplus line insurers. Captive insurers are special-purpose insurance companies set up by commercial businesses to self-insure risks arising from the owners’ business activities. Alien surplus lines insurers are foreign insurers that are qualified to do business in the United States through a process administered by NAIC. The market for terrorism risk insurance has been stable in recent years, with coverage both available and generally affordable. According to our reviews of policy language, reports from and interviews with Treasury, researchers, insurers, and other industry stakeholders, the expiration of TRIA and the absence of an alternative backstop to terrorism risk insurance would cause disruptions to the market. Reports from Treasury and an industry risk-management firm generally suggest there has been a stable market for terrorism risk insurance in recent years, with the coverage available and generally affordable in the United States. According to Treasury’s reports analyzing industry data, the majority of commercial policyholders in the United States purchase terrorism risk insurance, and at a relatively small percentage of total premiums. The market for terrorism risk insurance in the United States continues to remain competitive for most buyers according to 2018 and 2019 reports by Marsh, an insurance risk-management firm. Marsh attributed the competitive market for buyers to a steady decline in the frequency of global terrorist incidents and minimal insurance claims. Since all insurers must offer terrorism risk insurance, the availability of such coverage can be measured in terms of take-up rates—the rates at which policyholders select terrorism risk insurance. These rates have remained stable in recent years, according to Treasury. However, take-up rates vary by line of insurance, industry sector of the policyholder, geographic location, and type of insurer writing the policies. Terrorism risk coverage is considered available when insurers offer coverage for losses resulting from a terrorism event, and take-up rates are an indication of how insurers are complying with TRIA’s “make available” requirement, according to Treasury. Treasury found take-up rates by insurer category ranged from 62 to 78 percent in its 2018 report, depending on how the rates were measured. According to Marsh’s 2019 report, the take-up rate for terrorism coverage embedded in policies that cover other risks has been around 60 percent for the past several years. Lines of insurance. According to Treasury’s 2018 report, take-up rates across lines of insurance ranged from 43 percent in the products liability line to 83 percent in the boiler and machinery line in 2017, as measured by direct earned premium (see fig. 1). The take-up rate for cyber insurance coverage is in the middle of the range, relative to other lines of coverage. Specifically, the take-up rate in 2018 for terrorism risk insurance under cyber policies (by TRIA-eligible direct earned premium) was 69 percent for stand-alone policies, up from 50 percent in 2017, as reported by Treasury. For coverage that is part of an embedded policy, the 2018 rate was 63 percent, up from 54 percent in the prior year. Industry sectors. Take-up rates across the industry sectors of the policyholders varied widely and ranged from 7 percent in the information sector to 76 percent in the finance and insurance sector in 2017, according to Treasury’s 2018 report (see fig. 2). Marsh found in its 2019 report that commercial policyholders in the education, media, financial, and real estate sectors were the most frequent buyers of terrorism risk insurance in 2018. Geographic location. Take-up rates varied by location, with the highest rates in the Northeast. In Treasury’s 2018 report, the rates ranged from 50 to 75 percent by state (see fig. 3). In its 2018 report, Marsh noted that the Northeast had both the highest rate of purchase and the most expensive coverage, and said that these trends were because of the presence of major metropolitan areas (such as New York and Boston) that have high-value targets for terrorism. According to Treasury’s 2018 report, premiums associated with terrorism coverage have remained relatively consistent in recent years and are a small part of overall premiums for embedded policies. According to that report, about 80 percent of the market (as measured by terrorism risk direct earned premium) comprises embedded policies and 20 percent stand-alone policies, and the price for each varies. Premiums for terrorism risk insurance embedded in a property/casualty policy are priced at a relatively small percentage of the total premium charged for the policy and typically range from 2.5 to 3.0 percent when a charge is made. In about 30 percent of policies, insurers do not charge for providing terrorism risk coverage. Stand-alone policies vary significantly in terms of cost because of differences in the relative size or nature of exposures covered under each policy, whether the policy was certified, and the type of insurer providing the coverage, according to Treasury’s data. Premiums also varied across lines covered and insurer types, with the most premium collected for workers’ compensation. According to Treasury’s 2018 report, about 36 percent of the total premium collected in TRIA-eligible insurance lines was for workers’ compensation. In stand- alone cyber policies an average 6.2 percent of the total premium was allocated to terrorism risk. See table 2 for more information on how premiums vary across lines of coverage. Trends for small and captive insurers in many instances are different from trends for nonsmall insurers. Small insurers. Total market share for small insurers within TRIA-eligible lines of coverage declined, relative to nonsmall insurers, over the past decade. The small insurer market share, as measured by direct earned premium, fell from 18.6 percent in 2009 to 12.6 percent in 2018. (Despite that overall decline, there was an increase from 2016 to 2018 as more insurers were defined as small because of the increased dollar amount of the program trigger). In addition, take-up rates tended to be lower for policies written by small insurers, compared to nonsmall insurers, both within most individual lines and across the overall market. Small insurers generally charged less premium for terrorism risk insurance overall than nonsmall insurers, although they may charge proportionally higher premiums in some lines of insurance, such as commercial multiple peril (liability). According to Treasury’s 2019 report, small insurers allocated a lower percentage of direct earned premium for terrorism risk than nonsmall insurers. Furthermore, small insurers also were more likely to offer terrorism risk insurance for free. In addition, small insurers earned a higher percentage of their total program direct earned premium in commercial multiple peril and workers’ compensation lines than did nonsmall insurers. The workers’ compensation market is subject to very high loss amounts with no defined limits of liability and significant potential aggregation risks. Captive insurers. Like small insurers, captive insurers often have premiums that are small, relative to other insurer categories. However, captive insurers generally can offer broader coverage than commercial policies, according to Marsh’s 2019 report. The report states that a captive insurer often offers policies that cost less than policies from commercial insurers, which also often restrict coverage for NBCR or cyber events. In addition, according to Treasury a highly concentrated event affecting only captive insurers (or small insurers) carries a higher likelihood that the affected insurers’ losses would not meet the program trigger, and therefore would not be reimbursed under the program. In this case, captive insurers could incur significant losses. There could be significant disruptions to the insurance market if no federal terrorism risk insurance program existed, according to our reviews of policy language, reports from and interviews with Treasury, researchers, insurers, and other industry stakeholders. As Marsh noted in its 2019 report, TRIA’s federal backstop remains crucial to the continued stability of the terrorism risk insurance market. In its 2018 report, Treasury concluded that TRIA had made the coverage available and affordable, supporting a relatively stable market over the past decade. According to NAIC, TRIA helps foster the existence of a broader market for risks that otherwise would be either largely uninsured or borne by taxpayers. In the absence of a loss-sharing program, insurers likely would limit coverage, exit certain markets, or attempt to increase capacity, according to our review of reports from the federal government, researchers, industry entities, and interviews with industry stakeholders. For example: Limiting coverage. Most insurers begin the process to limit their coverage more than a year before any TRIA expiration by filing conditional exclusions, which, in effect, limits terrorism risk coverage in the event TRIA expired. According to one industry association, insurers have filed conditional exclusions before each of TRIA’s reauthorizations, although they are not commonly used for policies more than a year away from a potential expiration of the law. Our analysis of several policy endorsements filed with conditional exclusions suggests that, in the event of TRIA’s expiration, insurers likely would limit the total losses associated with an attack, and exclude certain types of terrorist attacks. We reviewed a nongeneralizable sample of conditional exclusions provided by the Insurance Services Office, which representatives say are widely used in the industry, and several selected conditional exclusions from individual insurers. These policies suggest that insurers filing conditional exclusions cap coverage for losses associated with an attack at $25 million, and entirely exclude losses caused by NBCR weapons. One policyholder association said that TRIA’s potential expiration and the need to file conditional exclusions results in a chaotic process, with insurers needing to file exclusions in each state in which they operated. Exiting markets. In the absence of a loss-sharing program, some insurers likely would exit certain markets, no longer offering terrorism coverage in specific geographic locations or lines of insurance, according to federal and industry reports and interviews with stakeholders. Small and midsize insurers in particular may withdraw from providing terrorism risk coverage entirely, according to one industry association. Furthermore, insurers providing NBCR or workers’ compensation coverage may decide to limit the policy terms or stop providing coverage, because of the risk of increased losses and potential exposures, according to Treasury. In addition, workers’ compensation risks are greater in large, metropolitan, more densely populated areas, and there are higher aggregation risks for insurers in large metropolitan areas, particularly for events involving NBCR weapons. Small insurers tend to operate on a regional basis in a smaller number of states than nonsmall insurers, and thus have a significant presence in individual local markets, according to Treasury. Options for increasing capacity. Insurers told us that they also likely would increase their premiums and purchase additional reinsurance for terrorism coverage in the absence of a program, although their ability to do so may be limited. One insurer said that premiums likely would go up significantly, although rate increases are subject to state limits. According to another insurer, reinsurance coverage for terrorism risk likely would become more limited, and be provided at notably higher rates. Insurers that are public companies may be able to increase capital through the stock market to build loss-absorbing capacity to help mitigate their increased loss exposures if TRIA expired. However, mutual insurers are not owned by shareholders and therefore cannot raise capital through the sale of shares; instead, they would have to rely on other ways of building capital. Several industry stakeholders pointed to particular challenges for certain insurers and lines of coverage if TRIA expired and Congress did not establish another loss-sharing program. Small insurers. Small insurers may be particularly vulnerable, facing ratings downgrades or otherwise being forced to exit the market for terrorism risk coverage, according to industry stakeholders. In May 2019, AM Best, a credit rating agency that focuses on the insurance industry, said insurers that did not limit exposure to terrorism risk losses before TRIA’s potential expiration in 2020 could face negative ratings pressure. AM Best identified 30 insurers (of about 230 with significant terrorism risk exposure) that failed stress tests, but said in October 2019 that implementation of plans established by these insurers would mitigate concerns about insolvency in the event TRIA expired and a terrorist attack occurred. The 30 insurers generally were small or midsize insurers. Captive insurers. Captives (entities that businesses set up to self- insure) generally require private reinsurance to insure against terrorism risk, and it is unclear if there would be sufficient capacity in the reinsurance market to obtain this coverage without TRIA. Captives tend to insure against a broader range of risks, including NBCR and cyber risks, when that coverage is unavailable or unaffordable in the market. One industry association representing captive insurers noted that captive insurance likely would become a more common way to insure against terrorism risk without a federal loss-sharing program. However, it warned that captive insurers may lack the capacity to ramp up operations quickly enough or secure the necessary reinsurance to fully absorb the risk of increased losses. NBCR coverage. Coverage for terrorism attacks involving NBCR weapons, which is already limited, would be further limited without a federal loss-sharing program, according to industry stakeholders. One industry association of insurance agents said that insurers’ capacity to absorb losses from such an attack would be a challenge without a backstop, as it was during the aftermath of the September 11 attacks, when there was very little capital devoted to coverage for terrorism risk. The representatives said this capacity would be even more limited for an NBCR attack, as losses could be significantly greater and few insurers offer NBCR coverage. Workers’ compensation coverage. The cost of coverage for workers’ compensation likely would increase significantly and availability likely would decrease without a federal loss-sharing program, according to researchers. Insurers have less flexibility to control terrorism exposure in workers’ compensation coverage, relative to other TRIA-eligible lines, according to Treasury. As noted earlier, state laws require employers to have the coverage and prohibit insurers from excluding terrorism risk, including NBCR risks, from workers’ compensation policies, according to Treasury. Insurers might respond to the absence of a federal loss-sharing program by not providing workers’ compensation coverage to employers, particularly those near high-risk targets in major metropolitan areas, according to a 2014 RAND Corporation policy brief issued before TRIA’s 2014 expiration. The brief added that this would force high-risk employers in these areas to obtain the required coverage from the residual market (state-run insurers or mechanisms of last resort), in which premiums are higher. In addition, the absence of a loss-sharing program could disrupt policyholders and the greater economy by stalling new building projects. Some stakeholders noted concerns that new building projects might be stalled if the law expired, similar to concerns in the weeks and months following the September 11 terrorist attacks. At that time, policymakers were concerned that the reduction in coverage by insurers uncertain of future losses would render commercial developers in high-risk areas unable to finance their projects, according to a report by the Congressional Budget Office. An insurance industry association told us businesses might find it difficult to obtain terrorism risk insurance, particularly for high-value projects in cities considered high-risk, such as New York and Washington, D.C. Treasury has a process to certify acts of terrorism. However, industry stakeholders said Treasury does not publicly communicate information about the process and the lack of timely information might negatively affect the speed with which insurers respond to policyholder claims. Additionally, Treasury is to consult with DOJ and DHS but DHS’s understanding of its role during the certification process appears inconsistent with Treasury’s purpose, and no agreements document these roles. Treasury also has a process to pay insurer claims and has issued guidance concerning how cyber insurance is treated under TRIA. Treasury has established a process for certifying an event as an act of terrorism that provides the Secretary a flexible time period for gathering information after an event. Before insurers may submit claims under TRIA, the Secretary must certify an event as an act of terrorism. Congress directed Treasury to study the certification process in the 2015 reauthorization of TRIA, including the establishment of a “reasonable timeline” for a certification determination. In response, Treasury sought and received public comments on the process. Treasury issued its conclusions in an October 2015 report. According to this report, seven of the nine comments received recommended Treasury adopt a timeline governing the certification decision. But Treasury concluded the certification process must provide the Secretary with flexibility to gather information after an event, and thus a “rigid” timeline for certification would not be appropriate. Instead, Treasury concluded that “enhanced public communication” about the status of the Secretary’s assessment of an act may address commenters’ concerns. Treasury established an interim final rule for the certification process in December 2016. Treasury’s process for certification decisions includes an internal review phase and a public review phase before Treasury can make a determination (see fig. 4). Internal review phase. During this phase, Treasury establishes and convenes a certification management team and prepares a brief for the Secretary, according to interviews with agency officials and our review of Treasury documents. Treasury may conclude the internal review of an event without progressing to the public review phase. Public review phase. The public phase of the certification process includes communication requirements set by Treasury’s certification regulations. TRIA regulations direct that within 30 days of the Secretary commencing review of an event, Treasury must publish a notice in the Federal Register informing the public that an act is under review for certification. Treasury also may publish a notice that it is not reviewing an act for certification. The regulation does not establish a timeline by which the Secretary must begin reviewing an event, which leaves the timeline for certification flexible. Treasury’s public announcement that an event is under review begins a series of requirements for public notification and consultation with other agencies, according to TRIA regulations. As of March 2020, Treasury has not conducted the public review phase of its certification process. When the Secretary of the Treasury’s review concludes that an act satisfies the elements of certification, the Secretary then is to consult with the Attorney General and the Secretary of Homeland Security within 30 days, or as soon as practicable. According to our review of Treasury documents, this Secretary-level consultation is to occur immediately before Treasury issues a certification decision. According to interviews with officials and Treasury documents, Treasury engages with staff in specific offices in DHS and DOJ much earlier in the process, during the internal review phase. Coordination with officials in these offices continues throughout both phases of the certification process. For example, Treasury documents state it may hold conferences with DHS and DOJ to discuss factors relevant to making a recommendation to certify an event. No later than 5 business days after the certification determination, Treasury must publish a statement in the Federal Register notifying the public. By contrast, the UK’s terrorism risk insurance program publicly communicates clear timelines by which government entities must certify potential events. The UK Treasury has 21 days to certify an event once the program administrator requests a formal review. This deadline was extended from 10 days in 2015 to allow the police enough time to determine if an event met the definition of terrorism, according to UK Treasury officials. This timeline was chosen to balance providing time for certification with ensuring that businesses would see claims paid quickly. Regular communication with industry stakeholders after an event maintains confidence in the certification process, they said. Treasury’s procedures for certifying an event do not include public communication of its internal review phase. Steps Treasury is to take during this internal review stage include establishing and convening a certification management team and preparing a brief for the Secretary, according to interviews with agency officials and our review of Treasury documents. To date, Treasury has not communicated to industry stakeholders whether it was reviewing events as possible acts of terrorism. Treasury officials told us that after events have occurred, they have looked into the circumstances and the amount of insurance losses caused. These considerations did not progress past the internal review phase of the certification process, which meant Treasury did not publicly communicate that it was reviewing these events for certification. For example, Treasury conducted internal reviews after the Boston Marathon bombing in 2013, but Treasury did not publicly communicate that it was looking into the event or that it had decided not to formally review the event for certification. Treasury ultimately did not certify the event because insured losses from the bombing on TRIA-eligible lines of insurance totaled $2.1 million, which was under the $5 million certification threshold, according to Massachusetts state insurance officials. In interviews and formal public comments on Treasury’s proposed certification rule, some industry stakeholders said the Boston Marathon bombing raised questions about the certification process because they viewed the event as a clear terrorist attack. It was unclear to some industry stakeholders if the event was not certified because it did not reach the monetary loss threshold for certification, which was unknown at the time, or because it did not meet TRIA’s nonmonetary requirement for establishing intent. Insurers and industry stakeholders told us they were uncertain about the length of time Treasury would take after future events to communicate that it was considering certification. All five insurers we interviewed said they would like improved communication from Treasury after an event like the Boston Marathon bombing. Treasury officials said that in response to the Boston Marathon bombing, they documented procedures for certification. However, these procedures do not include steps to communicate publicly during the internal review phase, according to our review of Treasury documents. If a future event analogous to the Boston Marathon bombing were to occur, under Treasury’s current procedures it would not communicate the status of its internal review publicly, and public communication would not occur if it chose to conclude its review before the public review phase began. Implication of Certification of an Act of Terrorism for Terrorism Risk Insurance Act (TRIA) Coverage TRIA is designed to share losses from a certified act of terrorism between insurers and the government. For insurers to receive support from this federal backstop, they must offer insurance for “acts of terrorism” defined in a manner consistent with the law, which requires certification by the Secretary of the Treasury. A certification determination affects policyholders differently, depending on if they purchased or declined terrorism coverage. Specifically, insurers would pay claims from policyholders that purchased terrorism coverage in the event of a certified act of terrorism, whereas insurers would not pay claims from policyholders that declined terrorism coverage. Insurers could face uncertainty about whether to pay claims on both policy types, however, if the Secretary of the Treasury does not make a certification determination. This is because the definition of an act of terrorism in insurance policies for both policy types is often linked to certification. Industry stakeholders and insurers we interviewed said they need to know whether Treasury is considering certifying an event to help provide certainty in paying policyholder claims and receiving reinsurance payments (see sidebar). Policyholder claims. Industry stakeholders and four of five insurers we interviewed said Treasury’s lack of communication about an event’s potential certification can lead to uncertainty about whether to pay claims on policies—both those that include and exclude terrorism coverage. Delays in paying claims while waiting for communication about certification put them at risk of violating their agreements with policyholders and state laws, they said. Insurance policies typically have timeline requirements for the insurer to investigate and pay claims, and some state laws require insurers to pay claims by a certain date, according to NAIC. Treasury officials said state requirements to pay claims by a certain date may receive extensions under state regulation when uncertainty requires that a claim investigation continue. One insurer with which we met said that a statement from Treasury when it was considering an event would help them determine whether to pay claims or not. Reinsurance. Industry stakeholders said uncertainty would delay reinsurance coverage. If insurers delayed paying policyholder claims because of uncertainty about certification of a terrorist attack, reinsurers also might delay payments to insurers. Reinsurance payments are often triggered by the insurer’s payment of a claim to the policyholder. Additionally, some reinsurance contracts may define terrorism specifically as a Treasury-certified act of terrorism, and may be contingent on Treasury making a certification determination. The goals of TRIA are to foster market stability and to protect consumers by addressing market disruptions. In addition, according to federal standards for internal control, management should externally communicate the necessary quality information to achieve the entity’s objectives, including communicating with external parties. Treasury officials said they have not chosen to set a deadline for public communication after a potential terrorist event because they need flexibility to collect accurate information about events whose circumstances can vary widely. In the preamble to its interim final rule on certification, Treasury concluded that public communication about the certification process provides the public with necessary information while avoiding the problems Treasury raised with establishing a strict timeline. However, Treasury’s internal review phase includes no public communication. Additionally, Treasury may conclude its review of an event without progressing to the public review phase and therefore may not issue any public communications on the event. Without public communication about when it is considering certification, Treasury risks contributing to market uncertainty rather than stability after an attack. TRIA requires cabinet-level consultation with DOJ and DHS in the public review phase of the certification process, but Treasury officials also conduct staff-level consultations. Treasury officials consult with DOJ’s National Security Division and DHS’s Support Anti-terrorism by Fostering Effective Technologies (SAFETY) Act office during the internal review phase of the certification process and have identified a single point of contact in each office (see sidebar). Consultation Agencies The Department of the Treasury consults with two other federal offices in the Department of Homeland Security (DHS) and the Department of Justice (DOJ), respectively, that have the following responsibilities: The Support Anti-terrorism by Fostering Effective Technologies (SAFETY) Act Office in DHS provides liability protections to manufacturers and sellers of specified anti-terrorism technologies. The Office of SAFETY Act Implementation reviews if an attack meets the SAFETY Act definition of an act of terrorism and if terrorists use such technology in the course of an attack, according to DHS officials. The Secretary of Homeland Security then determines whether an act has met the size and intent definitions of the SAFETY Act. DOJ’s National Security Division also makes recommendations for the International Terrorism Victim Expense Reimbursement Program, which provides funds to compensate victims of international terrorism occurring outside the United States. The Assistant Attorney General for National Security, in consultation with the National Counterterrorism Center, then determines whether to certify an event for the program, according to DOJ officials. determine whether an event meets TRIA’s definition of an act of terrorism. Such information might include things like who claimed responsibility for the event or evidence of the motivation for the attack. Officials said they provide this information upon request within 24 hours after an event. DOJ officials said the process they use to review events for TRIA purposes is similar to that used for DOJ’s International Terrorism Victim Expense Reimbursement Program. In contrast, DHS officials said their office does not provide information about an event to Treasury for purposes of certification, and that they believed DOJ would have the majority of this information. They said DHS informs Treasury about whether the event is being reviewed for the purposes of the SAFETY Act and whether terrorists used SAFETY Act-qualified technology (see sidebar). DHS officials said this is the information Treasury has requested from them and they consult with Treasury because many applicants for SAFETY Act designations have insurance policies backed by TRIA. Treasury officials stated that that they expect these two DHS and DOJ offices to serve as a single point of contact and coordinate with other relevant offices in their agencies as needed. DOJ officials confirmed they see this as their role, and said they would work with other offices in DOJ, including the Federal Bureau of Investigation, to consult with Treasury on certifying an act of terrorism. However, DHS officials said they do not see this as their role. The Secretary of the Treasury must consider, along with monetary requirements, the nature and motivation behind a potential terrorist attack to determine if it meets TRIA’s definition of an act of terrorism, according to TRIA regulations. Coordination among Treasury, DOJ, and DHS allows the Secretary access to critical and timely information relevant to certification, according to Treasury. In addition, according to federal internal control standards, management should use quality information to achieve the entity’s objectives, which includes identifying information requirements and obtaining relevant data from reliable sources in a timely manner. The standards also state that agencies should use methods such as written documentation to internally and externally communicate the information needed to achieve their objectives. In addition, our 2009 report on disaster planning provides an example of the benefits of clearly defined roles among federal agencies. We reported that defining the roles and responsibilities of stakeholders prior to a disaster could help foster collaboration, and that effective recovery plans should identify specific roles and responsibilities among various stakeholders. However, Treasury has not documented DOJ’s and DHS’s roles in certification consultations and instead relies on informal relationships with agency staff. This may contribute to the different perspectives DHS officials had on their role in the process. Treasury officials said although they do not have a written agreement, each agency understands its obligation to consult with Treasury in light of TRIA’s provisions requiring it. Although each agency told us it understood the certification process, DHS officials and Treasury differed in their understanding of DHS’s role in certification. A documented agreement among the agencies would provide procedures on roles and information sharing to which to refer during the potentially chaotic aftermath of a terrorist attack. As agency staff change over time, documenting these roles and information sharing among Treasury, DOJ, and DHS could help ensure continuity of operations if future events occurred. Furthermore, a written agreement would help Treasury access quality information and help ensure a smooth and timely process for certifying events under TRIA. Treasury has a process for fulfilling claims that uses a web-based system developed and operated by a contractor. Once the Secretary certifies an act of terrorism, Treasury is to issue a task order to the contractor, which is to make the claims website operational within 7 business days, according to its contract. The claims process begins for insurers when their total insured losses exceed 50 percent of their deductible within a calendar year, at which point insurers must submit a form notifying Treasury. An insurer may claim the federal share of compensation when its total insured losses exceed its deductible for a calendar year, according to TRIA regulations. The responsibilities of Treasury’s contractor include reviewing and testing the web-based claims system; activating and providing ongoing operation of the claims system; receiving and reviewing insurers’ required documents for completeness and accuracy; obtaining information from insurers as needed and answering questions by email and telephone; and recommending Treasury pay claims. Treasury’s contractor has developed operating guidelines that detail work flows and controls for how it will begin processing claims. The operating guidelines include a plan to transfer existing staff from other responsibilities to operate the claims process, as needed. According to the contractor, staff responsible for processing claims in the event of a certified terrorist attack participate in an annual training session. Treasury’s contractor also built quality checks within its web-based system to automatically review submissions. Moreover, Treasury’s contractor has tested the web-based claims system. The contractor said it completed more than 40 rounds of readiness testing since 2004. The contractor must conduct readiness testing at least three times a year and test contingency plans and disaster recovery procedures at least annually, according to the contract. In addition, Treasury’s contractor developed a demonstration website that is publicly available (see fig. 5). Of the five insurers GAO interviewed, one said it used the demonstration website, two said they had not, and two were unsure if anyone in the company had used the website. The contractor said they previously have invited insurers to participate in testing. The website outlines the general claims process and includes the forms insurers would submit in the event of a certified terrorist attack. Most industry stakeholders who were familiar with the claims process told us they found it to be clear. Those stakeholders who were unfamiliar with the process said they had no concerns about it at present. Of the five insurers we interviewed, three said the only concern they had regarding the claims process is how quickly Treasury would certify an event and pay insurers’ claims. One insurer said the claims process was clear, and one said it was unable to comment because it had not tested the process. In December 2016 Treasury issued guidance clarifying that, to the extent that insurers write cyber insurance under an embedded or stand-alone policy in TRIA-eligible lines, the TRIA provisions apply. In our May 2014 report, we found insurers were uncertain about whether TRIA covered risks from a cyberterrorism attack, and recommended that Treasury clarify whether losses that may result from cyberterrorism were covered under TRIA. Treasury’s 2016 guidance included three elements: 1. Treasury considers cyber policies that are reported under the “cyber liability” line for state regulatory purposes to be “property and casualty” insurance under TRIA, and therefore eligible for payment of the federal share of compensation in the event of a certified terrorist attack. 2. Policies only would be eligible if insurers made the same required disclosures to policyholders about the program as other TRIA-eligible lines. 3. Treasury requires insurers to provide disclosures and offers that comply with TRIA and the program regulations on any new or renewal policies reported under the cyber line. Industry stakeholders said that Treasury’s guidance about cyber insurance coverage under TRIA was clear. Some industry stakeholders said that there was some initial confusion about the guidance because it indicated the NAIC created a new line for cyber liability on the property/casualty annual statement, although this was not the case. According to NAIC representatives, changes were made to how insurance products were coded for rate-filing purposes, and these changes did not affect the lines of business reported on the property/casualty annual statement state page. Treasury officials said there may have been some ambiguity in how they communicated the 2016 guidance. NAIC representatives said despite this initial confusion, the industry understood the guidance. Industry stakeholders said that questions remain about what type of cyberattack Treasury would certify as an act of terrorism. TRIA’s definition of an act of terrorism requires an act “to have been committed by an individual or individuals as part of an effort to coerce the civilian population of the United States or to influence the policy or affect the conduct of the United States government by coercion.” However, according to industry stakeholders and industry analysts, the nature of a cyberattack means that tracing and attributing the event to an individual is difficult. Additionally, generally the Secretary of the Treasury may not certify an act if it is committed as part of a war declared by Congress. The Advisory Committee on Risk-Sharing Mechanisms, which provides recommendations to the Federal Insurance Office about risk sharing for terrorism losses, has been researching issues related to cyberterrorism insurance. According to this advisory committee, this group will provide Treasury with recommendations regarding this and other issues in spring 2020. Since shortly after the attacks of September 11, 2001, the Terrorism Risk Insurance Program has helped to ensure stability in the market for terrorism risk insurance, with the coverage generally available and affordable. However, insurers and policyholders are not aware of whether, and through what process, Treasury considers certifying an event as a terrorism event. Without public communication about when it is considering certification, Treasury risks contributing to market uncertainty rather than stability after an attack. The purpose of Treasury’s required consultation with DHS and DOJ in certifying an event is to provide Treasury the necessary law enforcement, intelligence, and homeland security information within the two agencies’ authorities and jurisdictions. However, DHS’s understanding of its role in the internal review phase of the certification process appears to differ from this stated purpose. Treasury has established and maintained informal connections with both agencies, but it has not documented these roles. By documenting agreements between Treasury and the two consulting agencies, Treasury can obtain quality information to help ensure a smooth and timely certification process. We are making the following three recommendations to Treasury: The Director of the Federal Insurance Office should publicly communicate information about when it is considering certifying an event as an act of terrorism under TRIA. (Recommendation 1) The Director of the Federal Insurance Office should document an agreement with DHS about DHS’s role, and how the agencies share information, during the process of certifying an event as an act of terrorism under TRIA. (Recommendation 2) The Director of the Federal Insurance Office should document an agreement with DOJ about DOJ’s role, and how the agencies share information, during the process of certifying an event as an act of terrorism under TRIA. (Recommendation 3) We provided a draft of this report to Treasury, DOJ, DHS, and NAIC for review and comment. DOJ and NAIC did not have any comments. Treasury provided written comments through the Federal Insurance Office, which are reproduced in appendix II and discussed below. Treasury and DHS provided technical comments, which we incorporated as appropriate and discuss below. We also solicited and received technical comments from the UK Treasury and incorporated them as appropriate. In its written comments, Treasury agreed with our three recommendations and described how it would address them. In response to our first recommendation, Treasury stated that it will consider potential changes to the certification process in conjunction with the results of the review by the Advisory Committee on Risk-Sharing Mechanisms of certification procedures (due in spring 2020). In response to our second and third recommendations, Treasury said that it will further coordinate with DOJ and DHS on their respective roles and evaluate any additional steps to clarify their roles in investigating potential events. In technical comments, DHS questioned our characterization of its role during the certification process. DHS reiterated that it would provide Treasury with information on how DHS handles an incident in relation to the DHS SAFETY Act process, and not information regarding any possible investigation of a terrorist event. DHS stated that this is the information Treasury requested from the office for potential events in the past. However, we found that Treasury has not documented the type of information it expects from each agency during its internal review phase and maintain that information related to the DHS SAFETY Act process is inconsistent with Treasury’s purpose for consultation—to obtain law enforcement and intelligence information. We maintain that documenting the information Treasury expects from each agency would ensure that Treasury obtains the information it needs to make a certification decision. We are sending copies of this report to the Secretary of the Treasury, the Acting Secretary of Homeland Security, the Attorney General, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or garciadiazd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In this report, we use “TRIA” to refer to the Terrorism Risk Insurance Act of 2002 and its subsequent reauthorizations. The objectives of our report were to examine (1) the current market for terrorism risk insurance and TRIA’s role in the market; and (2) the Department of the Treasury’s (Treasury) certification and claims processes, and industry stakeholders’ views on these processes, including guidance on cyber risk coverage. To address these objectives, we reviewed the Terrorism Risk Insurance Act of 2002; Terrorism Risk Insurance Extension Act of 2005; the Terrorism Risk Insurance Program Reauthorization Acts of 2007, 2015, and 2019; and implementing regulations, and congressional records. We also reviewed prior GAO work on this topic. We interviewed officials from the Treasury, National Association of Insurance Commissioners (NAIC), and Congressional Research Service and reviewed relevant reports from these entities. We also interviewed and reviewed reports from an academic researcher and several industry participants to obtain information for all our objectives, including insurers, representatives from insurance trade associations (representing insurers, reinsurers, mutual insurers, and captive insurers), risk modeling firms, and a rating agency. Specifically, we obtained information from five insurers. In all interviews, we asked participants about the potential effects of TRIA’s expiration on terrorism risk coverage, the effect of changes to the program from 2015 to 2020, and their views on Treasury’s certification and claims process, and guidance on coverage for cyberterrorism. We initially contacted eight insurers—four from among the largest U.S. commercial property and casualty insurers in TRIA-eligible lines of business (according to SNL Financial) and four smaller insurers previously recommended by insurance brokers and trade associations during prior GAO work. Five of these eight insurers, all of whom provided terrorism coverage to businesses, responded to our request and agreed to meet with us. Among these five insurers, two were large, two were small, and one was a captive insurer; two provided workers’ compensation and one provided cyber risk coverage. We determined that the information we obtained from these five insurers was sufficient for the purposes of obtaining a range of views of the market, but it is not generalizable to the practices of other insurers not included. To describe the current status of the market for terrorism risk insurance and how the market might be affected if TRIA were to expire, we reviewed annual Treasury reports on the program from 2017, 2018, and 2019, as well as reports from Marsh, an insurance risk-management firm, and other industry stakeholders. We reviewed these reports for information on affordability and availability of terrorism risk insurance, including data on take-up rates, premiums, geographic coverage, and trends over time. We also reviewed language in insurance policies that excluded some terrorism coverage in the event that TRIA was not reauthorized. To assess Treasury’s certification and claims processes, we reviewed documentation on the certification process, including Treasury’s internal policies and websites. We interviewed agency officials and the contractor responsible for operating the claims process after a certified terrorist attack, and we reviewed Treasury’s contract with this operator and the contractor’s internal policies. We also interviewed officials from the Departments of Homeland Security and Justice regarding their role in consulting with the Secretary of the Treasury on certification decisions. We reviewed relevant documents from the Organisation for Economic Co- operation and Development and relevant industry reports from four foreign countries with terrorism risk insurance programs: Australia, Belgium, Israel, and the United Kingdom (UK). We selected these countries because their terrorism risk insurance programs require certification by a government entity to pay claims. We interviewed the terrorism risk insurance pool operator and the certification entity for the UK because this program includes a short (21-day) timeline for certifying terrorist events. Additionally, we interviewed and reviewed documentation from a U.S. company that provides loss estimates, primarily to the insurance-linked securities market, which investors use to determine if a catastrophe bond has been triggered by an event. We compared Treasury’s certification and consultation process against criteria in federal internal control standards on management communication. To determine how cyber terrorism is covered under TRIA and in commercial policies, we reviewed Treasury guidance. We also met with Treasury officials and representatives of the Insurance Services Office, a property/casualty insurance industry association that develops standardized policy language, and reviewed its standard policies for cyber insurance. We also reviewed Treasury reports on cyberterrorism coverage, including data on take-up rates and direct earned premiums for cyberterrorism risks. We conducted this performance audit from April 2019 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Jill Naamane (Assistant Director), Nathan Gottfried (Analyst in Charge), Anna Blasco, William R. Chatlos, Giselle Cubillos-Moraga, Kaitlan Doying, Karen Jarzynka-Hernandez, May Lee, Barbara Roesmann, Jessica Sandler, Jena Sinkfield, and Rachel Whitaker made significant contributions to this report. Terrorism Risk Insurance: Market Challenges May Exist for Current Structure and Alternative Approaches. GAO-17-62. Washington, D.C.: January 12, 2017. Terrorism Risk Insurance: Comparison of Selected Programs in the United States and Foreign Countries. GAO-16-316. Washington, D.C.: April 12, 2016. Terrorism Insurance: Treasury Needs to Collect and Analyze Data to Better Understand Fiscal Exposure and Clarify Guidance. GAO-14-445. Washington, D.C.: May 22, 2014. Terrorism Insurance: Status of Coverage Availability for Attacks Involving Nuclear, Biological, Chemical, or Radiological Weapons. GAO-09-39. Washington, D.C.: December 12, 2008. Terrorism Insurance: Status of Efforts by Policyholders to Obtain Coverage. GAO-08-1057. Washington, D.C.: September 15, 2008. Terrorism Insurance: Implementation of the Terrorism Risk Insurance Act of 2002. GAO-04-307. Washington, D.C.: April 23, 2004.", "summary": "TRIA created a federal program to help ensure the availability and affordability of terrorism risk insurance. Insurers must make terrorism risk coverage available to commercial policyholders. The federal government and insurers share losses on such policies resulting from a certified act of terrorism causing at least $5 million of insurance losses. Annual coverage for losses by insurers (who have met their insurer deductible) and the government is limited to $100 billion. The program is set to expire December 31, 2027. GAO was asked to review TRIA. This report examines (1) the current market for terrorism risk insurance and the program's role in the market, and (2) Treasury's processes to certify acts of terrorism and fulfill claims. GAO reviewed Treasury reports and related industry studies, Treasury's guidance and procedures for the program, and insurance policy language. GAO also interviewed Treasury officials and industry stakeholders, including a nongeneralizable sample of insurers of different sizes providing various types of insurance. With the support of a program established under the Terrorism Risk Insurance Act (TRIA) in which the federal government and insurers would share losses in the event of a certified act of terrorism, terrorism risk insurance is generally available and affordable in the United States. For example, the majority of commercial policyholders generally purchased terrorism risk insurance in recent years, according to Department of the Treasury (Treasury) data. The insurance market would be significantly disrupted without a loss-sharing program such as that established under TRIA. Specifically, insurers generally would not have to offer terrorism risk coverage and likely would charge higher premiums in the absence of a loss-sharing arrangement and cap on losses, according to GAO's review of policies and interviews with industry stakeholders, including insurers and insurer associations. Without access to affordable coverage, new building ventures could be delayed and employers could struggle to find affordable workers' compensation coverage. Treasury has processes for certifying terrorist events and fulfilling claims under the program, but a lack of communication about aspects of Treasury's certification process could pose challenges for insurers. Some industry stakeholders, such as insurers and representatives of insurer associations, raised issues about Treasury communications on certification. They cited confusion over why the 2013 Boston Marathon bombing was not certified when they clearly viewed it as a terrorist attack. These industry stakeholders also expressed concern that Treasury never communicated whether it was reviewing the event for certification or its reasons for not certifying it. Most insurers GAO interviewed said such lack of communication by Treasury again could lead to uncertainty about whether to pay claims, putting them at risk of violating state laws and their policyholder agreements. TRIA regulations on certifying acts of terrorism include some public notification requirements but do not require Treasury to communicate when it is considering reviewing an event for certification. One purpose of TRIA is to stabilize the insurance market after a terrorist attack. Public communication of when Treasury is considering an event for certification would reduce uncertainty about which claims insurers should pay and lessen potential disruptions to the market after an attack. One step in determining when to certify an event is Treasury's consultation with offices in the Department of Homeland Security (DHS) and Department of Justice (DOJ) to obtain law enforcement, intelligence, and homeland security information. However, GAO found that DHS had a different understanding of its role in this staff consultation process, and Treasury had not documented agreements with either agency. By documenting agreements between Treasury and the two consulting agencies, Treasury can better ensure a smooth and timely certification process. Once an event is certified as an act of terrorism, Treasury has a process for fulfilling claims that uses a web-based system developed and operated by a contractor. As of February 2020, the system had not yet been used because Treasury had not certified any acts of terrorism or paid claims under the program. GAO is making three recommendations, including that Treasury publicly communicate when it is considering reviewing an event for TRIA certification and document agreements with both DHS and DOJ on the agencies' roles in the process. Treasury agreed with the recommendations.", "document_type": "gao"}
{"report": "According to the President’s budget, the federal government planned to invest more than $96 billion for IT in fiscal year 2018—the largest amount ever budgeted. Despite such large IT expenditures, we have previously reported that investments in federal IT too often resulted in failed projects that incurred cost overruns and schedule slippages, while contributing little to the desired mission-related outcomes. For example: The tri-agency National Polar-orbiting Operational Environmental Satellite System was disbanded in February 2010 at the direction of the White House’s Office of Science and Technology Policy after the program spent 16 years and almost $5 billion. The Department of Homeland Security’s (DHS) Secure Border Initiative Network program was ended in January 2011, after the department obligated more than $1 billion for the program. The Department of Veterans Affairs’ Financial and Logistics Integrated Technology Enterprise program was intended to be delivered by 2014 at a total estimated cost of $609 million, but was terminated in October 2011. The Department of Defense’s Expeditionary Combat Support System was canceled in December 2012 after spending more than a billion dollars and failing to deploy within 5 years of initially obligating funds. The United States Coast Guard (Coast Guard) decided to terminate its Integrated Health Information System project in 2015. As reported by the agency in August 2017, the Coast Guard spent approximately $60 million over 7 years on this project, which resulted in no equipment or software that could be used for future efforts. Our past work has found that these and other failed IT projects often suffered from a lack of disciplined and effective management, such as project planning, requirements definition, and program oversight and governance. In many instances, agencies had not consistently applied best practices that are critical to successfully acquiring IT. Such projects have also failed due to a lack of oversight and governance. Executive-level governance and oversight across the government has often been ineffective, specifically from CIOs. For example, we have reported that some CIOs’ roles were limited because they did not have the authority to review and approve the entire agency IT portfolio. In addition to failures when acquiring IT, security deficiencies can threaten systems. As we previously reported, in order to counter security threats, the 23 civilian Chief Financial Officers (CFO) Act agencies spent a combined total of approximately $4 billion on IT security-related activities in fiscal year 2016. Even so, our cybersecurity work at federal agencies continues to highlight information security deficiencies. The following examples describe the types of risks we have found at federal agencies. In September 2018, we reported that the Department of Education’s Office of Federal Student Aid exercises minimal oversight of lenders’ protection of student data and lacks assurance that appropriate risk- based safeguards are being effectively implemented, tested, and monitored. In August 2017, we reported that, since the 2015 data breaches, the Office of Personnel Management (OPM) had taken actions to prevent, mitigate, and respond to data breaches involving sensitive personal and background investigation information. However, we noted that the agency had not fully implemented recommendations made to OPM by DHS’s United States Computer Emergency Readiness Team to help the agency improve its overall security posture and improve its ability to protect its systems and information from security breaches. In July 2017, we reported that information security at the Internal Revenue Service had weaknesses that limited its effectiveness in protecting the confidentiality, integrity, and availability of financial and sensitive taxpayer data. An underlying reason for these weaknesses was that the Internal Revenue Service had not effectively implemented elements of its information security program. In May 2016, we reported that the National Aeronautics and Space Administration, the Nuclear Regulatory Commission, OPM, and the Department of Veteran Affairs did not always control access to selected high-impact systems, patch known software vulnerabilities, and plan for contingencies. An underlying reason for these weaknesses was that the agencies had not fully implemented key elements of their information security programs. In August 2016, we reported that the information security of the Food and Drug Administration had significant weaknesses that jeopardized the confidentiality, integrity, and availability of its information systems and industry and public health data. Congress and the President have enacted various key pieces of reform legislation to address IT management issues. These include the federal IT acquisition reform legislation commonly referred to as the Federal Information Technology Acquisition Reform Act (FITARA). This legislation was intended to improve covered agencies’ acquisitions of IT and enable Congress to monitor agencies’ progress and hold them accountable for reducing duplication and achieving cost savings. The law includes specific requirements related to seven areas: Agency CIO authority enhancements. CIOs at covered agencies have the authority to, among other things, (1) approve the IT budget requests of their respective agencies and (2) review and approve IT contracts. Federal data center consolidation initiative (FDCCI). Agencies covered by FITARA are required, among other things, to provide a strategy for consolidating and optimizing their data centers and issue quarterly updates on the progress made. Enhanced transparency and improved risk management. The Office of Management and Budget (OMB) and covered agencies are to make detailed information on federal IT investments publicly available, and agency CIOs are to categorize their investments by level of risk. Portfolio review. Covered agencies are to annually review IT investment portfolios in order to, among other things, increase efficiency and effectiveness and identify potential waste and duplication. Expansion of training and use of IT acquisition cadres. Covered agencies are to update their acquisition human capital plans to support timely and effective IT acquisitions. In doing so, the law calls for agencies to consider, among other things, establishing IT acquisition cadres (i.e., multi-functional groups of professionals to acquire and manage complex programs), or developing agreements with other agencies that have such cadres. Government-wide software purchasing program. The General Services Administration is to develop a strategic sourcing initiative to enhance government-wide acquisition and management of software. In doing so, the law requires that, to the maximum extent practicable, the General Services Administration should allow for the purchase of a software license agreement that is available for use by all executive branch agencies as a single user. Maximizing the benefit of the Federal Strategic Sourcing Initiative. Federal agencies are required to compare their purchases of services and supplies to what is offered under the Federal Strategic Sourcing Initiative. In June 2015, OMB released guidance describing how agencies are to implement FITARA. This guidance is intended to, among other things: assist agencies in aligning their IT resources with statutory requirements; establish government-wide IT management controls to meet the law’s requirements, while providing agencies with flexibility to adapt to unique agency processes and requirements; strengthen the relationship between agency CIOs and bureau CIOs; and strengthen CIO accountability for IT costs, schedules, performance, and security. The guidance identifies a number of actions that agencies are to take to establish a basic set of roles and responsibilities (referred to as the common baseline) for CIOs and other senior agency officials and, thus, to implement the authorities described in the law. For example, agencies are to conduct a self-assessment and submit a plan describing the changes they intend to make to ensure that common baseline responsibilities are implemented. In addition, in August 2016, OMB released guidance intended to, among other things, define a framework for achieving the data center consolidation and optimization requirements of FITARA. The guidance directed agencies to develop a data center consolidation and optimization strategic plan that defined the agency’s data center strategy for fiscal years 2016, 2017, and 2018. This strategy was to include, among other things, a statement from the agency CIO indicating whether the agency had complied with all data center reporting requirements in FITARA. Further, the guidance states that OMB is to maintain a public dashboard to display consolidation-related costs savings and optimization performance information for the agencies. Congress has recognized the importance of agencies’ continued implementation of FITARA provisions, and has taken legislative action to extend selected provisions beyond their original dates of expiration. Specifically, Congress and the President enacted laws to: remove the expiration dates for the enhanced transparency and improved risk management provisions, which were set to expire in 2019; remove the expiration date for portfolio review, which was set to expire in 2019; and extend the expiration date for FDCCI from 2018 to 2020. In addition, Congress and the President enacted a law to authorize the availability of funding mechanisms to help further agencies’ efforts to modernize IT. The law, known as the Modernizing Government Technology (MGT) Act, authorizes agencies to establish working capital funds for use in transitioning from legacy IT systems, as well as for addressing evolving threats to information security. The law also creates the Technology Modernization Fund, within the Department of the Treasury, from which agencies can “borrow” money to retire and replace legacy systems, as well as acquire or develop systems. Further, in February 2018, OMB issued guidance for agencies on implementing the MGT Act. The guidance was intended to provide agencies additional information regarding the Technology Modernization Fund, and the administration and funding of the related IT working capital funds. Specifically, the guidance encouraged agencies to begin submitting initial project proposals for modernization on February 27, 2018. In addition, in accordance with the MGT Act, the guidance provides details regarding a Technology Modernization Board, which is to consist of (1) the Federal CIO; (2) a senior IT official from the General Services Administration; (3) a member of DHS’s National Protection and Program Directorate; and (4) four federal employees with technical expertise in IT development, financial management, cybersecurity and privacy, and acquisition, appointed by the Director of OMB. Congress and the President enacted the Federal Information Security Modernization Act of 2014 (FISMA) to improve federal cybersecurity and clarify government-wide responsibilities. The act addresses the increasing sophistication of cybersecurity attacks, promotes the use of automated security tools with the ability to continuously monitor and diagnose the security posture of federal agencies, and provides for improved oversight of federal agencies’ information security programs. Toward this end, the act clarifies and assigns specific responsibilities to entities such as OMB, DHS, and the federal agencies. Table 1 describes a selection of the OMB, DHS, and agency responsibilities. Beyond the implementation of FITARA, FISMA, and related actions, the administration has also initiated other efforts intended to improve federal IT. Specifically, in March 2017, the administration established the Office of American Innovation, which has a mission to, among other things, make recommendations to the President on policies and plans aimed at improving federal government operations and services. In doing so, the office is to consult with both OMB and the Office of Science and Technology Policy on policies and plans intended to improve government operations and services, improve the quality of life for Americans, and spur job creation. In May 2017, the Administration also established the American Technology Council, which has a goal of helping to transform and modernize federal agency IT and how the federal government uses and delivers digital services. The President is the chairman of this council, and the Federal CIO and the United States Digital Service Administrator are among the members. In addition, on May 11, 2017, the President signed Executive Order 13800, Strengthening the Cybersecurity of Federal Networks and Critical Infrastructure. This executive order outlined actions to enhance cybersecurity across federal agencies and critical infrastructure to improve the nation’s cyber posture and capabilities against cybersecurity threats. Among other things, the order tasked the Director of the American Technology Council to coordinate a report to the President from the Secretary of DHS, the Director of OMB, and the Administrator of the General Services Administration, in consultation with the Secretary of Commerce, regarding the modernization of federal IT. As a result, the Report to the President on Federal IT Modernization was issued on December 13, 2017, and outlined the current and envisioned state of federal IT. The report focused on modernization efforts to improve the security posture of federal IT and recognized that agencies have attempted to modernize systems but have been stymied by a variety of factors, including resource prioritization, ability to procure services quickly, and technical issues. The report provided multiple recommendations intended to address these issues through the modernization and consolidation of networks and the use of shared services to enable future network architectures. Further, in March 2018, the Administration issued the President’s Management Agenda, which lays out a long-term vision for modernizing the federal government. The agenda identifies three related drivers of transformation—IT modernization; data, accountability, and transparency; and the workforce of the future—that are intended to push change across the federal government. The Administration also established 14 related Cross-Agency Priority goals, many of which have elements that involve IT. In particular, the Cross-Agency Priority goal on IT modernization states that modern IT must function as the backbone of how government serves the public in the digital age. This goal establishes three priorities that are to guide the Administration’s efforts to modernize federal IT: (1) enhancing mission effectiveness by improving the quality and efficiency of critical services, including the increased utilization of cloud-based solutions; (2) reducing cybersecurity risks to the federal mission by leveraging current commercial capabilities and implementing cutting edge cybersecurity capabilities; and (3) building a modern IT workforce by recruiting, reskilling, and retaining professionals able to help drive modernization with up-to-date technology. More recently, on May 15, 2018, the President signed Executive Order 13833, Enhancing the Effectiveness of Agency Chief Information Officers. Among other things, this executive order is intended to better position agencies to modernize their IT systems, execute IT programs more efficiently, and reduce cybersecurity risks. The order pertains to 22 of the 24 CFO Act agencies: the Department of Defense and the Nuclear Regulatory Commission are exempt. For the covered agencies, the executive order strengthens the role of agency CIOs by, among other things, requiring them to report directly to their agency head; serve as their agency head’s primary IT strategic advisor; and have a significant role in all management, governance, and oversight processes related to IT. In addition, one of the cybersecurity requirements directs agencies to ensure that the CIO works closely with an integrated team of senior executives, including those with expertise in IT, security, and privacy, to implement appropriate risk management measures. In the February 2017 update to our high-risk series, we reported that agencies still needed to complete significant work related to the management of IT acquisitions and operations. We stressed that OMB and federal agencies should continue to expeditiously implement FITARA and OMB’s related guidance, which includes enhancing CIO authority, consolidating data centers, and acquiring and managing software licenses. Our update to this high-risk area also stressed that OMB and agencies needed to continue to implement our prior recommendations in order to improve their ability to effectively and efficiently invest in IT. Specifically, since fiscal year 2010, we have made 1,242 recommendations to OMB and federal agencies to address shortcomings in IT acquisitions and operations. As stated in the update, OMB and agencies should demonstrate government-wide progress in the management of IT investments by, among other things, implementing at least 80 percent of our recommendations related to managing IT acquisitions and operations. As of November 2018, OMB and agencies had fully implemented 732 (or about 59 percent) of the 1,242 recommendations. Figure 1 summarizes the progress that OMB and agencies have made in addressing our recommendations compared to the 80 percent target. Overall, federal agencies would be better positioned to realize billions in cost savings and additional management improvements if they address these recommendations, including those aimed at implementing CIO responsibilities, reviewing IT acquisitions; improving data center consolidation; and managing software licenses. In all, the various laws, such as FITARA, and related guidance assign 35 IT management responsibilities to CIOs in six key areas. These areas are: leadership and accountability, budgeting, information security, investment management, workforce, and strategic planning. In August 2018, we reported that none of the 24 agencies we reviewed had policies that fully addressed the role of their CIO, as called for by federal laws and guidance. In this regard, a majority of the agencies had fully or substantially addressed the role of their CIOs for the area of leadership and accountability. In addition, a majority of the agencies had substantially or partially addressed the role of their CIOs for two areas: information security and IT budgeting. However, most agencies partially or minimally addressed the role of their CIOs for two areas: investment management and strategic planning. Further, the majority of the agencies minimally addressed or did not address the role of their CIOs for the remaining area: IT workforce. Figure 2 depicts the extent to which the 24 agencies addressed the role of their CIOs for the six areas. Despite the shortfalls in agencies’ policies addressing the roles of their CIOs, most agency officials stated that their CIOs are implementing the responsibilities even if the agencies do not have policies requiring implementation. Nevertheless, the CIOs of the 24 selected agencies acknowledged in responses to a survey that we administered that they were not always very effective in implementing the six IT management areas. Specifically, at least 10 of the CIOs indicated that they were less than very effective for each of the six areas of responsibility. We believe that until agencies fully address the role of CIOs in their policies, agencies will be limited in addressing longstanding IT management challenges. Figure 3 depicts the extent to which the CIOs reported their effectiveness in implementing the six areas of responsibility. Beyond the actions of the agencies, however, shortcomings in agencies’ policies also are partially attributable to two weaknesses in OMB’s guidance. First, the guidance does not comprehensively address all CIO responsibilities, such as those related to assessing the extent to which personnel meet IT management knowledge and skill requirements, and ensuring that personnel are held accountable for complying with the information security program. Correspondingly, the majority of the agencies’ policies did not fully address nearly all of the responsibilities that were not included in OMB’s guidance. Second, OMB’s guidance does not ensure that CIOs have a significant role in (1) IT planning, programming, and budgeting decisions; and (2) execution decisions and the management, governance, and oversight processes related to IT, as required by federal law and guidance. In the absence of comprehensive guidance, CIOs will not be positioned to effectively acquire, maintain, and secure their IT systems. In response to the survey conducted for our August 2018 report, the 24 agency CIOs also identified a number of factors that enabled and challenged their ability to effectively manage IT. Specifically, most agency CIOs cited five factors as being enablers to effectively carry out their responsibilities: (1) NIST guidance, (2) the CIO’s position in the agency hierarchy, (3) OMB guidance, (4) coordination with the Chief Acquisition Officer (CAO), and (5) legal authority. Further, three factors were cited by CIOs as major factors that have challenged their ability to effectively carry out responsibilities: (1) processes for hiring, recruiting, and retaining IT personnel; (2) financial resources; and (3) the availability of personnel/staff resources. As shown in figure 4, the five enabling factors were identified by at least half of the 24 CIOs and the three factors cited as major challenges were identified by at least half of the CIOs. Although OMB has issued guidance aimed at addressing the three factors that were identified by at least half of the CIOs as major challenges, the guidance does not fully address those challenges. Further, regarding the financial resources challenge, OMB recently required agencies to provide data on CIO authority over IT spending; however, its guidance does not provide a complete definition of the authority. In the absence of such guidance, agencies have created varying definitions of CIO authority. Until OMB updates its guidance to include a complete definition of the authority that CIOs are to have over IT spending, it will be difficult for OMB to identify any deficiencies in this area and to help agencies make any needed improvements. In order to address challenges in implementing CIO responsibilities, we made three recommendations to OMB and one recommendation to each of the selected 24 federal agencies to improve the effectiveness of CIOs’ implementation of their responsibilities for each of the six IT management areas. Most agencies agreed with or had no comments on the recommendations. As of November 2018, all 27 of the recommendations had not been implemented. We will continue to monitor the implementation of these recommendations. FITARA includes a provision to enhance covered agency CIOs’ authority through, among other things, requiring agency heads to ensure that CIOs review and approve IT contracts. OMB’s FITARA implementation guidance expanded upon this aspect of the legislation in a number of ways. Specifically, according to the guidance: CIOs may review and approve IT acquisition strategies and plans, rather than individual IT contracts; CIOs can designate other agency officials to act as their representatives, but the CIOs must retain accountability; CAOs are responsible for ensuring that all IT contract actions are consistent with CIO-approved acquisition strategies and plans; and CAOs are to indicate to the CIOs when planned acquisition strategies and acquisition plans include IT. In January 2018, we reported that most of the CIOs at 22 selected agencies were not adequately involved in reviewing billions of dollars of IT acquisitions. For instance, most of the 22 agencies did not identify all of their IT contracts. In this regard, the agencies identified 78,249 IT- related contracts, to which they obligated $14.7 billion in fiscal year 2016. However, we identified 31,493 additional contracts with $4.5 billion obligated, raising the total amount obligated to IT contracts by these agencies in fiscal year 2016 to at least $19.2 billion. Figure 5 reflects the obligations that the 22 selected agencies reported to us relative to the obligations we identified. The percentage of additional IT contract obligations we identified varied among the selected agencies. For example, the Department of State did not identify 1 percent of its IT contract obligations. Conversely, eight agencies did not identify over 40 percent of their IT contract obligations. Many of the selected agencies that did not identify these IT contract obligations also did not follow OMB guidance. Specifically, 14 of the 22 agencies did not involve the acquisition office in their process to identify IT acquisitions for CIO review, as required by OMB. In addition, 7 agencies did not establish guidance to aid officials in recognizing IT. We concluded that until these agencies involve the acquisitions office in their IT acquisition identification processes and establish supporting guidance, they cannot ensure that they will identify all such acquisitions. Without proper identification of IT acquisitions, these agencies and CIOs cannot effectively provide oversight of these acquisitions. In addition to not identifying all IT contracts, 14 of the 22 selected agencies did not fully satisfy OMB’s requirement that the CIO review and approve IT acquisition plans or strategies. Further, only 11 of 96 randomly selected IT contracts at 10 agencies that we evaluated were CIO- reviewed and approved as required by OMB’s guidance. The 85 contracts not reviewed had a total possible value of approximately $23.8 billion. Until agencies ensure that CIOs are able to review and approve all IT acquisitions, CIOs will continue to have limited visibility and input into their agencies’ planned IT expenditures and will not be able to effectively use the increased authority that FITARA’s contract approval provision is intended to provide. Further, agencies will likely miss an opportunity to strengthen their CIOs’ authority and the oversight of acquisitions. As a result, agencies may award IT contracts that are duplicative, wasteful, or poorly conceived. As a result of these findings, we made 39 recommendations in our January 2018 report. Among these, we recommended that agencies ensure that their acquisition offices are involved in identifying IT acquisitions and issuing related guidance, and that IT acquisitions are reviewed in accordance with OMB guidance. OMB and the majority of the agencies generally agreed with or did not comment on the recommendations. As of November 2018, 27 of the recommendations had not been implemented. In our February 2017 high-risk update, we stressed that OMB and agencies needed to demonstrate additional progress on achieving data center consolidation savings in order to improve the management of IT acquisitions and operations. Further, data center consolidation efforts are key to implementing FITARA. Specifically, OMB established the FDCCI in February 2010 to improve the efficiency, performance, and environmental footprint of federal data center activities. The enactment of FITARA in 2014 codified and expanded the initiative. In addition, in August 2016, OMB issued a memorandum which established the Data Center Optimization Initiative (DCOI) and included guidance on how to implement the data center consolidation and optimization provisions of FITARA. Among other things, the guidance required agencies to consolidate inefficient infrastructure, optimize existing facilities, improve their security posture, and achieve cost savings. According to agencies, data center consolidation and optimization efforts have resulted in approximately $4.5 billion in cost savings through 2018. However, additional work remains to fully carry out the initiative. Specifically, in a series of reports that we issued from July 2011 through May 2018, we noted that, while data center consolidation could potentially save the federal government billions of dollars, weaknesses existed in several areas, including agencies’ data center consolidation plans, data center optimization, and OMB’s tracking and reporting on related cost savings. In these reports, we made a total of 160 recommendations to OMB and 24 agencies to improve the execution and oversight of the initiative. Most agencies and OMB agreed with our recommendations or had no comments. As of November 2018, 47 of these 160 recommendations remained unimplemented. In addition, in a draft report on data center optimization that we have provided to the agencies for comment and plan to issue in early 2019, our preliminary results indicate that agencies continued to report mixed progress toward achieving OMB’s goals for closing data centers and realizing the associated savings by September 2018. Specifically, as of August 2018, over half of the agencies reported that they had met, or planned to meet, all of their OMB-assigned closure goals for tiered data centers by the deadline. However, 6 agencies reported that they did not plan to meet their goals for tiered data centers. In addition, as of August 2018, 11 agencies reported that they had already met the goal for closing 60 percent of their non-tiered centers, 3 agencies reported that they planned to meet the goal by the end of fiscal year 2018, and 9 agencies reported that they did not plan to meet the goal by the end of fiscal year 2018. In all, the 24 agencies reported a total of 6,250 data center closures as of August 2018, which represented about half of the total reported number of federal data centers. In addition, the agencies reported 1,009 planned closures by the end of fiscal year 2018, with an additional 191 closures planned through fiscal year 2023, for a total of 1,200 further closures. Further, in August 2018, 22 agencies reported that they had achieved $1.94 billion in cost savings for fiscal years 2016 through 2018, while 2 agencies reported that they had not achieved any savings. In addition to that amount, 21 agencies identified a further $0.42 billion in planned savings through fiscal year 2018—for a total of $2.36 billion in planned cost savings from fiscal years 2016 through 2018. Nevertheless, this total is about $0.38 billion less than OMB’s goal of $2.74 billion for overall DCOI savings. In our 2015 high-risk report’s discussion of IT acquisitions and operations, we identified the management of software licenses as an area of concern, in part because of the potential for cost savings. Federal agencies engage in thousands of software licensing agreements annually. The objective of software license management is to manage, control, and protect an organization’s software assets. Effective management of these licenses can help avoid purchasing too many licenses, which can result in unused software, as well as too few licenses, which can result in noncompliance with license terms and cause the imposition of additional fees. As part of its PortfolioStat initiative, OMB has developed a policy that addresses software licenses. This policy requires agencies to conduct an annual, agency-wide IT portfolio review to, among other things, reduce commodity IT spending. Such areas of spending could include software licenses. In May 2014, we reported on federal agencies’ management of software licenses and determined that better management was needed to achieve significant savings government-wide. Of the 24 selected agencies we reviewed, only 2 had comprehensive policies that included the establishment of clear roles and central oversight authority for managing enterprise software license agreements, among other things. Of the remaining 22 agencies, 18 had policies that were not comprehensive, and 4 had not developed any policies. Further, we found that only 2 of the 24 selected agencies had established comprehensive software license inventories, a leading practice that would help them to adequately manage their software licenses. The inadequate implementation of this and other leading practices in software license management was partially due to weaknesses in agencies’ policies. As a result, we concluded that agencies’ oversight of software license spending was limited or lacking, thus potentially leading to missed savings. However, the potential savings could be significant considering that, in fiscal year 2012, 1 major federal agency reported saving approximately $181 million by consolidating its enterprise license agreements, even when its oversight process was ad hoc. Accordingly, we recommended that OMB issue a directive to help guide agencies in managing software licenses. We also made 135 recommendations to the 24 agencies to improve their policies and practices for managing licenses. Among other things, we recommended that the agencies regularly track and maintain a comprehensive inventory of software licenses and analyze the inventory to identify opportunities to reduce costs and better inform investment decision making. Most agencies generally agreed with the recommendations or had no comments. As of December 2018, 27 of the 135 recommendations had not been implemented. Table 2 reflects the extent to which the 24 agencies implemented the recommendations in these two areas. Since information security was added to the high-risk list in 1997, we have consistently identified shortcomings in the federal government’s approach to cybersecurity. In particular, in a September 2018 report, we identified four major cybersecurity challenges: (1) establishing a comprehensive cybersecurity strategy and performing effective oversight, (2) securing federal systems and information, (3) protecting cyber critical infrastructure, and (4) protecting privacy and sensitive data. To address these challenges, we identified 10 critical actions that the federal government and other entities need to take. For example, in order to address the challenge of securing federal systems and information, we identified 3 actions that the agencies should take: (1) improve implementation of government-wide cybersecurity initiatives, (2) address weaknesses in federal information security programs, and (3) enhance the federal response to cyber incidents. Figure 6 depicts the 10 critical actions to address the four major cybersecurity challenges. As we have previously noted, in order to strengthen the federal government’s cybersecurity posture, agencies should fully implement the information security programs required by FISMA. In this regard, FISMA provides a framework for ensuring the effectiveness of information security controls for federal information resources. The law requires each agency to develop, document, and implement an agency-wide information security program. Such a program should include risk assessments; the development and implementation of policies and procedures to cost- effectively reduce risks; plans for providing adequate information security for networks, facilities, and systems; security awareness and specialized training; the testing and evaluation of the effectiveness of controls; the planning, implementation, evaluation, and documentation of remedial actions to address information security deficiencies; procedures for detecting, reporting, and responding to security incidents; and plans and procedures to ensure continuity of operations. Since fiscal year 2010, we have made over 3,000 recommendations to agencies aimed at addressing the four cybersecurity challenges. These recommendations have identified actions for agencies to take to strengthen technical security controls over their computer networks and systems. They also have included recommendations for agencies to fully implement aspects of their information security programs, as mandated by FISMA. Nevertheless, many agencies continue to be challenged in safeguarding their information systems and information, in part, because many of these recommendations have not been implemented. Of the roughly 3,000 recommendations made since 2010, 73 percent had been implemented as of November 2018; leaving 688 recommendations unimplemented. In order to determine the effectiveness of the agencies’ information security programs and practices, FISMA requires federal agencies’ inspectors general to conduct annual independent evaluations. The agencies are to report the results of these evaluations to OMB, and OMB is to summarize the results in annual reports to Congress. In these evaluations, the inspectors general are to frame the scope of their analyses, identify key findings, and detail recommendations to address the findings. The evaluations also are to capture maturity model ratings for their respective agencies. Toward this end, in fiscal year 2017, the inspector general community, in partnership with OMB and DHS, finalized a 3-year effort to create a maturity model for FISMA metrics. The maturity model aligns with the five function areas in the NIST Framework for Improving Critical Infrastructure Cybersecurity (Cybersecurity Framework): identify, protect, detect, respond, and recover. This alignment is intended to help promote consistent and comparable metrics and criteria and provide agencies with a meaningful independent assessment of their information security programs. The maturity model is designed to summarize the status of agencies’ information security programs on a five-level capability maturity scale. The five maturity levels are defined as follows: Level 1 Ad-hoc: Policies, procedures, and strategy are not formalized; activities are performed in an ad-hoc, reactive manner. Level 2 Defined: Policies, procedures, and strategy are formalized and documented but not consistently implemented. Level 3 Consistently Implemented: Policies, procedures, and strategy are consistently implemented, but quantitative and qualitative effectiveness measures are lacking. Level 4 Managed and Measurable: Quantitative and qualitative measures on the effectiveness of policies, procedures, and strategy are collected across the organizations and used to assess them and make necessary changes. Level 5 Optimized: Policies, procedures, and strategy are fully institutionalized, repeatable, self-generating, consistently implemented and regularly updated based on a changing threat and technology landscape and business/mission needs. In March 2018, OMB issued its annual FISMA report to Congress, which showed the combined results of the inspectors general’s fiscal year 2017 evaluations. Based on data from 76 agency inspector general and independent auditor assessments, OMB determined that the government-wide median maturity model ratings across the five NIST Cybersecurity Framework areas did not exceed a level 3 (consistently implemented). Table 3 shows the inspectors general’s median ratings for each of the NIST Cybersecurity Framework areas. In its efforts toward strengthening the federal government’s cybersecurity, OMB also requires agencies to submit related cybersecurity metrics as part of its Cross-Agency Priority goals. In particular, OMB developed the IT modernization goal so that federal agencies will be able to build and maintain more modern, secure, and resilient IT. A key part of this goal is to reduce cybersecurity risks to the federal mission through three strategies: manage asset security, protect networks and data, and limit personnel access. The key targets supporting each of these strategies correspond to areas within the FISMA metrics. Table 4 outlines the strategies, their associated targets, and the 23 civilian CFO Act agencies’ progress in meeting those targets, as of June 2018. In conclusion, FITARA and FISMA present opportunities for the federal government to address the high-risk areas on improving the management of IT acquisitions and operations and ensuring the security of federal IT, thereby saving billions of dollars. Most agencies have taken steps to execute key IT management and cybersecurity initiatives, including implementing CIO responsibilities, requiring CIO reviews of IT acquisitions, realizing data center consolidation cost savings, managing software assets, and complying with FISMA requirements. The agencies have also continued to address the recommendations that we have made over the past several years. However, further efforts by OMB and federal agencies to implement our previous recommendations would better position them to improve the management and security of federal IT. To help ensure that these efforts succeed, we will continue to monitor agencies’ efforts toward implementing the recommendations. Chairmen Meadows and Hurd, Ranking Members Connolly and Kelly, and Members of the Subcommittees, this completes my prepared statement. I would be pleased to respond to any questions that you may have. If you or your staff have any questions about this testimony, please contact Carol C. Harris, Director, Information Technology, at (202) 512- 4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Kevin Walsh (Assistant Director), Chris Businsky, Rebecca Eyler, Meredith Raymond, and Jessica Waselkow (Analyst in Charge). This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The federal government planned to invest more than $96 billion in IT in fiscal year 2018. However, IT investments have often failed or contributed little to mission-related outcomes. Further, increasingly sophisticated threats and frequent cyber incidents underscore the need for effective information security. As a result, GAO added two areas to its high-risk list: cybersecurity in 1997 and the management of IT acquisitions and operations in 2015. This statement summarizes federal agencies' progress in improving the management, and ensuring the security, of federal IT. It is primarily based on GAO's reports issued between February 1997 and August 2018 (and an ongoing review) on (1) CIO responsibilities, (2) agency CIOs' involvement in approving IT contracts, (3) data center consolidation efforts, (4) the management of software licenses, and (5) compliance with cybersecurity requirements. The Office of Management and Budget (OMB) and federal agencies have taken steps to improve the management of information technology (IT) acquisitions and operations and ensure federal cybersecurity through a series of initiatives. As of November 2018, agencies had fully implemented about 59 percent of the 1,242 IT management-related recommendations that GAO has made since fiscal year 2010. Likewise, agencies had implemented about 73 percent of the approximately 3,000 security-related recommendations that GAO has made since 2010. Even with this progress, significant actions remain to be completed. Chief Information Officer (CIO) responsibilities . Laws such as the Federal Information Technology Acquisition Reform Act (FITARA) and related guidance assigned 35 key IT management responsibilities to CIOs to help address longstanding challenges. However, in August 2018, GAO reported that none of the 24 selected agencies had policies that fully addressed the role of their CIO, as called for by laws and guidance. GAO recommended that OMB and each of the 24 agencies take actions to improve the effectiveness of CIOs' implementation of their responsibilities. As of November 2018, none of the 27 recommendations had been implemented. IT contract approval . According to FITARA, covered agencies' CIOs are required to review and approve IT contracts. Nevertheless, in January 2018, GAO reported that most of the CIOs at 22 covered agencies were not adequately involved in reviewing billions of dollars of IT acquisitions. Consequently, GAO made 39 recommendations to improve CIO oversight over these acquisitions. As of November 2018, 27 of the recommendations had not been addressed. Consolidating data centers . OMB launched an initiative in 2010 to reduce data centers. According to agencies, data center consolidation and optimization efforts have resulted in approximately $4.5 billion in cost savings through 2018. Even so, additional work remains. GAO has made 160 recommendations to OMB and agencies to improve the reporting of related cost savings and to achieve optimization targets. However, as of November 2018, 47 of the recommendations had not been fully addressed. Managing software licenses . Effective management of software licenses can help avoid purchasing too many licenses that result in unused software. In May 2014, GAO reported that better management of licenses was needed to achieve savings, and made 135 recommendations to improve such management. As of December 2018, 27 of the recommendations had not been implemented. Improving the security of federal IT systems . While the government has acted to protect federal information systems, agencies need to improve security programs, cyber capabilities, and the protection of personally identifiable information. The approximately 3,000 recommendations that GAO has made to agencies since 2010 were aimed at improving the security of federal systems and information. Specifically, these recommendations identified actions for agencies to take to strengthen their information security programs and technical controls over their computer networks and systems. As of November 2018, 688 of the security-related recommendations had not been implemented. Since fiscal year 2010, GAO has made 1,242 recommendations to OMB and agencies to address shortcomings in IT acquisitions and operations. Since fiscal year 2010, GAO also has made over 3,000 recommendations to federal agencies to improve the security of federal systems. These recommendations include those to improve the implementation of CIO responsibilities, the oversight of the data center consolidation initiative, software license management efforts, and the strength of security programs and technical controls. Most agencies agreed with the recommendations, and GAO will continue to monitor their implementation.", "document_type": "gao"}
{"report": "Marine debris originates from multiple sources and types of materials, entering the marine environment in a variety of ways, as shown in figure 1. Most plastics do not biodegrade, that is, decay naturally and become absorbed by the environment. Instead, plastics slowly break down into smaller and smaller fragments, eventually becoming what are known as microplastics. Microplastics are very small pieces of plastic that are generally less than 5 millimeters in size (about the size of a sesame seed). The formation of microplastics occurs when plastic debris is exposed to sunlight and the plastic begins to weather and fragment. Microplastics have been found in the stomachs of numerous aquatic organisms including insects, worms, fish, and clams, according to a 2018 study. A study from 2011 showed that once animals ingest microplastics, they can be stored in tissues and cells, providing a possible pathway for the accumulation of contaminants and potentially harming the animals. pots, and other recreational or commercial fishing equipment that has been lost, neglected, or discarded in the marine environment. According to the Global Ghost Gear Initiative, at least 640,000 tons of derelict fishing gear enters the ocean each year, a weight equivalent to two Empire State Buildings. Derelict fishing gear may entrap sea life, adversely affect marine habitats, present hazards to navigation, and cause other harmful effects (see fig. 2). For example, according to a 2015 NOAA report, derelict fishing gear threatens a variety of fish, turtles, seabirds, whales, and seals, and may be especially problematic for endangered and protected marine species. Abandoned and derelict vessels. Abandoned and derelict vessels are vessels without identified ownership, in significant disrepair, or both. There are thousands of such vessels in ports, waterways, and estuaries around the United States that have been left to deteriorate by the owner or operator or are the result of a catastrophic weather event, according to NOAA documents. Abandoned and derelict vessels can impede marine transportation by blocking navigable waterways, and, if not visible or well-marked, could pose collision risks to vessel operators. These vessels may also become sources of pollution since they may contain fuel oil or other hazardous materials that can leak into the water as the vessels deteriorate, impacting the local community, marine life, and nearby habitat. Marine debris has garnered increasing interest from the international community. In September 2015, the United Nations General Assembly unanimously adopted an agenda with a set of global sustainable development goals through 2030. One of the goals (goal 14) calls for conservation and sustainable use of the oceans, seas, and marine resources, and includes a target for prevention and significant reduction of marine pollution of all kinds, including marine debris, by 2025. In June 2018, five members of the Group of Seven and the European Union endorsed the Group’s Ocean Plastics Charter, which committed them to accelerating implementation of the Group of Seven Leaders’ Action Plan to Combat Marine Litter, previously agreed to in 2015. The United States and Japan were the two members of the Group of Seven that did not endorse the charter. Also, in May 2019, the parties to the Basel Convention on the Control of Transboundary Movements of Hazardous Waste and Their Disposal adopted a decision that would, beginning January 1, 2021, require parties to take appropriate measures to ensure that certain plastic waste is reduced to a minimum, taking into account social, technological and economic aspects, among other things. The Marine Debris Act governs the activities of the interagency committee. For example, it required the interagency committee to issue a report to Congress that included recommendations to reduce marine debris domestically and internationally. In 2008, the committee submitted an interagency recommendation report that contained 25 recommendations intended to guide the federal government’s strategies for addressing marine debris (see appendix II for a list of the 25 recommendations). The recommendations were categorized by an overarching topic, such as education and outreach or cleanup. Within each category, the committee then identified specific recommendations. For example, within the education and outreach category, the committee specified three recommendations: Demonstrate leadership by distributing educational materials to personnel on the sources and impacts of marine debris as well as methods for prevention with the goal of reducing the federal contribution to marine debris. Support public awareness campaigns by providing technical expertise and educational materials and by encouraging private sector participation, when appropriate. Engage and partner with state, local, tribal and nongovernmental entities to support coordinated events, such as Earth Day, the International Coastal Cleanup, and other activities that have relevance to marine debris. The act also requires the interagency committee to submit biennial reports to Congress that evaluate progress in meeting the purposes of the Marine Debris Act. Specifically, these biennial reports are to include: the status of implementation of any recommendations and strategies of the committee and analysis of their effectiveness, and estimated federal and nonfederal funding provided for marine debris and recommendations for priority funding needs. Starting in 2010, the interagency committee has issued five biennial reports to Congress, issuing its most recent report in March 2019. The Marine Debris Act designates six federal agencies as interagency committee members. The six agencies are NOAA, EPA, U.S. Coast Guard, U.S. Navy, Department of State, and Department of the Interior. The act also specifies that the committee shall include senior officials from other federal agencies that have an interest in ocean issues or water pollution prevention and control as the Secretary of Commerce determines appropriate. The act designates the senior official from NOAA to serve as the chair. The interagency committee coordinates primarily through quarterly meetings where agencies share information about their individual activities related to addressing marine debris. Such activities range from education and outreach to research and technology development and are generally driven by the missions and authorities of the agencies. However, we found that NOAA has not established a process to determine the committee’s membership. In addition, the Marine Debris Act requires the interagency committee to include a “senior official” from member agencies, but NOAA has not determined the level of official it would consider senior. The interagency committee coordinates primarily through quarterly meetings where federal agencies share information about their individual marine debris-related activities. According to its charter, which was last revised in 2014, the committee is responsible for sharing information, assessing and implementing best management practices, and coordinating interagency responses to marine debris. The charter states that the interagency committee will ensure the coordination of federal agency marine debris activities nationally and internationally as well as recommend research priorities, monitoring techniques, educational programs, and regulatory action. The charter also states that the interagency committee will work to consider the interests of nongovernmental organizations, industry, state governments, Indian tribes, and other nations, as appropriate. NOAA officials said the main focus of the interagency committee has been to serve as an information-sharing body. The officials said they also seek opportunities to collaborate on individual projects, but the committee does not otherwise collaborate on activities, beyond compiling statutorily required biennial reports. NOAA officials explained that individual agencies each have a unique set of authorities and missions that largely determine their role and involvement in marine debris-related issues. For example, under its Marine Debris Program, NOAA conducts a variety of education, outreach, research, and other activities to identify sources of and address marine debris. In recent years, congressional committee reports accompanying NOAA’s annual appropriations have directed the agency to spend a certain amount of its appropriations on its marine debris program. Specifically, these reports directed NOAA to spend $7 million in fiscal year 2018 and $7.5 million in fiscal year 2019 for its Marine Debris Program. The program is also authorized to award grants to, and enter into cooperative agreements and contracts with, eligible entities to identify the sources of, prevent, reduce, and remove marine debris. In contrast, officials from other agencies on the interagency committee said their agencies have not received such direction or specific appropriations to address marine debris. Rather, the activities these agencies have conducted generally tie to their authority or agency mission. For example, EPA officials said they have relied on voluntary partnerships with states, industry, and other sources and leveraged existing funds from related programs, such as the agency’s stormwater and water quality programs, to support its Trash Free Waters Program. This is a program that encourages collaborative actions by public and private stakeholders to prevent trash from entering water. EPA officials said they also support a number of other activities related to education, outreach, and research, and these activities are a high priority for the agency, but EPA does not have a line item in its budget dedicated to marine debris activities. The interagency committee’s biennial reports describe general types of activities individual agencies reported conducting—often in coordination with nonfederal partners such as nongovernmental organizations, industry, states, Indian tribes, and other nations—to address marine debris, which include activities in the following categories: (1) education and outreach; (2) legislation, regulation, and policy; (3) cleanup; (4) research and technology development; and (5) coordination (see table 1 for descriptions of types of activities in each category; see app. III for specific examples of activities carried out by agencies). To help agencies share information, NOAA chairs quarterly meetings where agencies are invited to discuss their individual activities. In reviewing meeting minutes, we found that the meetings were generally well-attended by representatives from multiple agencies. During the meetings, officials discussed marine debris issues and some provided updates on their agencies’ activities. For example, at the April 2019 meeting, officials discussed ways in which different agencies may be meeting the sense of Congress on international engagement in the Save our Seas Act of 2018. At the May 2018 meeting, officials from NOAA and U.S. Coast Guard gave presentations on their agencies’ emergency response authorities and efforts. NOAA officials described their actions in response to Hurricanes Harvey, Irma, and Maria in 2017, which included coordinating debris removal activities across federal and state agencies, such as EPA and Florida State’s Department of Environmental Protection. U.S. Coast Guard officials also presented information on their marine debris removal activities in response to Hurricanes Irma and Maria. These activities included coordinating with multiple federal, state, and local agencies and contractors to remove or mitigate potential environmental impacts from 2,366 damaged or derelict vessels in Florida and the Florida Keys after Hurricane Irma and 377 vessels in Puerto Rico and the Island of Vieques after Hurricane Maria, according to U.S. Coast Guard officials. The interagency committee has also used its quarterly meetings to identify opportunities for collaboration among federal agencies and with nonfederal partners, according to NOAA officials. For example, during committee meetings in early 2018, NOAA, the National Park Service, and the Department of State identified an opportunity to collaborate with the German government to bring the Ocean Plastics Lab to the United States. This Lab is an international traveling exhibition that explains the role of science in helping to understand and address plastic pollution in the ocean. NOAA officials said that to collaborate on this effort, officials from three federal agencies served on a steering committee, leveraged volunteers, promoted the Ocean Plastics Lab through outreach efforts to the public and helped staff the exhibits while they were on display in Washington, D.C., during the summer of 2018. We found that NOAA has not established a process to determine interagency committee membership. The Marine Debris Act designates six federal agencies as members of the committee, and also specifies that committee members shall include senior officials from other federal agencies that have interests in ocean issues or water pollution prevention as the Secretary of Commerce determines appropriate. The committee’s 2014 charter lists five agencies as members in addition to the six identified in the act, for a total of 11 member agencies. The charter also states that the committee consists of representatives from “any other federal agency that has an interest in ocean issues and water pollution prevention and control,” but does not specify the process for documenting membership or how the Secretary of Commerce, or a delegate of the Secretary, will determine that such membership is appropriate, as required by the act. Various information sources, such as the committee’s biennial reports and minutes from quarterly meetings, have provided differing lists of committee member agencies. For example, the committee’s March 2019 biennial report and NOAA’s website as of July 2019 listed the 11 agencies identified in its charter as members. But, various meeting minutes from meetings held in fiscal year 2019 listed up to 13 members. One agency, the U.S. Agency for International Development (USAID), has regularly attended the committee’s quarterly meetings since early 2018 when USAID officials said they were invited to participate on the committee. USAID officials said that their understanding is that USAID is a member of the interagency committee and that this is especially important to recognize given their significant international development assistance related to marine debris over the last few years. However, USAID is not listed as a member on NOAA’s website and the agency’s marine debris-related activities are not included in the committee’s 2019 biennial report. As a result, some agencies may not be included in the required biennial reports on the committee members’ marine debris activities. In April 2019, NOAA officials told us that USAID was a contributing member to the interagency committee. The officials said that “official” member agencies are those six agencies designated by the Marine Debris Act and that they consider other participating agencies as “contributing” members. They said it has been the practice of the interagency committee to enable participation and coordination with other agencies, including those who may not be designated as official members. We found that NOAA does not have a documented process for determining membership on the interagency committee. NOAA officials were unable to locate records from 2006 or earlier documenting the addition of contributing agencies to the committee or the Secretary, or a delegate of the Secretary, making a determination of the appropriateness of such agencies being members. NOAA officials stated the need for the agency to establish a documented process to determine the appropriateness of federal agencies being committee members. The officials said they have started working with NOAA’s General Counsel to formalize and document the committee’s membership process, and that the process will include a step for the Secretary of Commerce, or a delegate of the Secretary to determine the appropriateness of additional agencies being members. However, NOAA officials did not have an estimated time frame for developing such a process. Our past work on interagency collaboration has identified the importance of ensuring that relevant participants have been included in the collaborative effort. By establishing a time frame for developing a documented membership process, NOAA and the interagency committee can benefit from capturing all members’ activities, and ensuring it provides Congress a complete picture of marine debris efforts across the federal government. In addition, the Marine Debris Act requires the interagency committee to include a “senior official” from member agencies, but NOAA has not determined the level of official it would consider senior. The interagency committee’s charter states that the committee will be composed of “federal agency managers and technical experts,” but does not define what is meant by senior official. NOAA officials said that the level of engagement from agency officials has varied over time and often depends on the specific officials participating. The officials said they have had difficulty in the past getting some member agency officials to engage during quarterly meetings and often those that do participate are not decision makers. Specifically, for some agencies, participating officials may not represent the entire agency, but rather a program within the agency, and they may not have decision-making authority, according to NOAA officials. As a result, the officials may not be able to commit agency resources, or they may be uncertain what activities their agency may be able to commit to. NOAA officials said that it may be helpful to specify the level of official needed to represent the agencies on the interagency committee. The officials said that they have been discussing potential revisions to the interagency committee’s charter, and within that broader discussion they are looking into whether the charter should specify what level of official is needed. However, NOAA officials did not have an estimated time frame for revising its charter or determining what those revisions may entail. Our past work on interagency collaboration has identified the importance of ensuring that participants have full knowledge of the relevant resources in the agency, including the ability to commit resources for their agency. By clarifying what is meant by “senior official” such as through revisions to its charter, NOAA would have greater assurance that it has the full engagement of member agency officials who can speak for their agency and commit to activities. While the interagency committee’s biennial reports provide information on marine debris-related activities of individual agencies, our review found that they do not contain certain required elements. As previously noted, the Marine Debris Act requires the biennial reports to include (1) the status of implementation of any recommendations and strategies of the committee and analysis of their effectiveness, and (2) estimated federal and nonfederal funding provided for marine debris and recommendations for priority funding needs. However, we found that the biennial reports did not include an analysis of the effectiveness of the recommendations implemented or recommendations for priority funding needs. The five biennial reports the interagency committee issued from 2010 to 2019 lay out the committee’s 2008 recommendations along with a description of activities taken by individual member agencies related to those recommendations. Specifically, each biennial report references the 25 recommendations the committee first adopted in its 2008 interagency recommendation report, organized into categories (see app. II). The reports then provide a description of activities taken by individual member agencies that fell within the recommendation categories for each preceding 2-year period. However, we found that the five biennial reports do not include an analysis of the effectiveness of the implementation of the committee’s recommendations and strategies as required by the Marine Debris Act. Some of the descriptions of agencies’ activities include information on the number of people reached through education or outreach efforts or other quantitative information related to specific activities, but the reports do not include an analysis of the effectiveness of those activities. NOAA and EPA officials confirmed that the interagency committee did not include an analysis of effectiveness in its biennial reports, stating that undertaking such an effort is beyond the scope of the information-sharing focus of the interagency committee. NOAA officials said that they have attempted to bring member agencies together to discuss how the committee could analyze the effectiveness of its collective efforts, but this has been a challenge because each member has its own priorities and legal authority related to addressing marine debris. Activities to implement the committee’s 25 recommendations occur at each individual agency, rather than at the committee level, according to the officials. As such, NOAA officials said each member agency may evaluate the effectiveness of its individual activities and pointed to measures NOAA has in place to evaluate its Marine Debris Program. For example, NOAA estimates the amount of debris removed annually and the number of students it reaches through education and outreach efforts. EPA officials said that determining a baseline and quantifying the results of specific marine debris efforts to determine effectiveness is challenging, as is the case for other broad, nonpoint sources of pollution. For example, trash enters water bodies through innumerable water and sewer system outfalls, so EPA may focus on strategies to change people’s behavior to minimize trash from entering the systems (see fig. 3). But unlike measuring emissions from a smokestack, it is difficult to determine a baseline and then measure and demonstrate progress in terms of trash reduction exiting through the system outfalls. EPA officials said they recognize the need to measure the effectiveness of their efforts related to marine debris—especially as addressing marine debris has become a high priority for the agency—but measuring progress has yet to be determined across all of its various offices and programs that carry out marine debris-related activities. Within the Trash Free Waters program specifically, EPA officials said they take steps to evaluate the effectiveness of the program through a variety of means, such as seeking feedback from stakeholders. Our past work has shown that collaborative entities—including those addressing complex, cross-cutting issues—can better demonstrate progress and identify areas for improvement if they develop a means to monitor, evaluate, and report the results of their collective efforts. Developing such a means would help the interagency committee ensure that its member agencies are using their authorities and aligning their priorities in the most effective manner possible. Moreover, developing and implementing a process to analyze the effectiveness of the interagency committee’s recommendations and strategies, and reporting the results in its biennial reports as required by the Marine Debris Act would better position the committee to determine the extent to which its efforts are making a difference in addressing the complex facets of marine debris. The five biennial reports include some estimates of funding for marine debris-related activities, but do not identify recommendations for priority funding needs as required by the Marine Debris Act. Specifically, we found that the reports included estimates for some member agencies’ spending related to their marine debris-related activities and estimated nonfederal spending for certain activities. The reports also state that several member agencies conduct activities within multiple programs, offices, and projects indirectly related to marine debris efforts. These agencies do not receive annual appropriations specifically for marine debris activities but instead receive appropriations to fulfill their missions or implement programs, making it difficult to estimate exact spending related to marine debris, according to the reports. The 2019 biennial report states that the interagency committee’s recommendations for priority funding needs are reflected in the President’s budget request and operating plan for each member agency in any given fiscal year. NOAA officials said that it would be difficult to identify and communicate priority funding needs outside of these documents, particularly given the complications associated with estimating each agency’s individual spending. For example, an EPA official said that EPA’s efforts to address marine debris are decentralized and the agency does not receive an appropriation specifically for marine debris-related activities, making it difficult to determine how much the agency spends—or may need to spend—on marine debris. Moreover, NOAA and EPA officials said that because the interagency committee serves primarily as an information-sharing body and each member agency operates independently in identifying resource needs, the interagency committee has not needed to develop a process to identify recommendations for priority funding needs. However, the Marine Debris Act requires the interagency committee to include recommendations for priority funding needs in its biennial reports, and without a process to identify such recommendations, the interagency committee cannot meet that requirement. Our past work on leading collaborative practices has shown the importance of identifying and leveraging resources, such as funding, in collaborative efforts. By developing a process to identify recommendations for priority funding needs in its biennial reports, the interagency committee could provide Congress with required information about priority funding needs across the federal government to address marine debris. The 14 experts we interviewed with expertise in marine debris-related issues suggested a range of actions that the federal government could take to most effectively address various types of marine debris. Their suggestions included increasing or improving actions already being taken by some federal agencies as well as taking new actions. The experts stressed that there is not one solution to the growing, multi-dimensional problem of marine debris. Rather, they said that a multitude of actions involving federal agencies and nonfederal partners—such as international, state and local governments, Indian tribes, industry, and environmental groups—will need to be taken to address the issue. Experts as well as agency officials we interviewed indicated that there would be a number of factors to consider in evaluating the suggested actions. Some of these factors are overarching, applying to most or all of the actions; others relate to specific actions. For example, several experts and agency officials said that competing priorities and limited resources would be important factors to consider related to all of the suggested actions. Several agency officials also said that their agencies may not have the authority to take some of the actions suggested by the experts, and therefore new legislation would need to be enacted before they could take those actions. Additionally, some actions could result in impacts or costs to particular industries, underserved communities, or consumer groups, and understanding and identifying ways to mitigate such impacts would be important. Moreover, several agency officials said some actions, such as those related to waste management, may be better suited for local or state governments and that those entities would be better- equipped to deal with particular aspects of marine debris. The following are examples of actions the experts suggested that the federal government could take. We organized the actions into the following five categories, which generally correspond to the categories laid out in the interagency committee’s reports: (1) education and outreach, (2) establishment of federal requirements or incentives, (3) cleanup, (4) research and technology development, and (5) coordination. Seven of the 14 experts suggested actions to educate or conduct outreach to the public or specific consumer or industry groups or international governments about ways to prevent, reduce, mitigate, or clean up waste that can become marine debris. A few experts emphasized that education and outreach efforts should be focused on ways to prevent trash from entering the marine environment. Examples of education and outreach actions suggested include: Domestic education and outreach. Five experts suggested different types of education or outreach campaigns the federal government could undertake to target certain domestic groups, such as consumers. One expert suggested that the federal government develop a national campaign to educate the public about marine debris. Such a campaign would develop a single message that various entities, including federal agencies and nonfederal stakeholders, could include in advertisements, social media, and other public awareness efforts. The expert pointed to similar state-led campaigns, such as “Nobody Trashes Tennessee,” a litter campaign developed by Tennessee’s Department of Transportation. This state campaign features celebrities, such as athletes and musicians, in advertisements and involves selling stickers, hats, and other items to help spread the message. However, the expert said that securing collaboration and agreement on a single message across federal agencies and nonfederal stakeholders could pose a challenge and that a national campaign would need a long-term commitment from all parties to be successful. NOAA officials said that national campaigns can be expensive and demonstrating results from such efforts can be difficult, especially when they are broad in nature. As a result, these officials said that NOAA’s Marine Debris Program targets its education and outreach efforts to a specific audience for a particular type of behavior change or type of debris, such as educating and training high school students to lead “Zero Litter Campaigns” in their schools and communities. International outreach. Two experts suggested actions the federal government could take to conduct outreach internationally to promote programs, policies, or technologies that can reduce marine debris. For example, one expert suggested the federal government conduct outreach to government officials in countries that have limited waste management infrastructure to demonstrate effective waste management technologies. The expert said that the federal government could partner with private sector companies to demonstrate waste-to-energy technologies, such as gasification and pyrolysis that can convert plastic waste to fuel. According to the expert, demonstrating such technologies would provide information on its benefits, including reducing sources of waste and creating a source of energy to either use or sell. Several agency officials we interviewed agreed that international outreach efforts are critical to successfully addressing marine debris and that emphasis should be placed on assisting countries with improving their waste management practices. However, these officials said there are many factors to consider with regard to waste-to-energy technologies. For instance, State Department officials said such technologies may not be supported by civil organizations because of environmental concerns. Waste-to-energy technologies could also entail high upfront capital investments, and waste-to-energy facilities should adhere to strict environmental standards with monitoring and enforcement to help ensure the technology is not causing negative effects, according to agency officials. As a result, they said it may not be practical for some countries to adopt such technologies. In addition, USAID officials said that promoting waste-to-energy technology presupposes that waste is already being collected in sufficient quantity and quality to serve as a fuel for such technology, but that in some countries waste is openly dumped or burned and therefore sufficient waste may not be available. They cautioned that waste-to-energy technologies can be a part of a response to address marine debris abroad, but would not be sufficient alone. Microfibers are a widespread type of microplastic; they have been found on the shorelines of six continents and in oceans, rivers, soils, table salt, and public drinking water, according to scientific studies. Microfibers enter the marine environment through various pathways. For example, microfibers are shed from synthetic clothing and other materials made of polyester and nylon. These microfibers pass through to waterways because washing machines and wastewater treatment plants typically do not have processes sufficiently refined to remove the fibers. Little is known about other potential sources of microfibers, such as carpet manufacturing; the rate of generation, such as how quickly materials break down and shed microfibers; and any health impacts to humans or wildlife. federal requirements for manufacturers to design certain products to minimize the chances of material becoming marine debris. For example, two experts suggested the federal government develop design standards for washing machine manufacturers to ensure filters are designed to prevent microfibers from entering wastewater systems and then the marine environment. Three experts suggested the federal government develop design standards to require or incentivize manufacturers to use specific amounts of post-consumer material in developing certain products. For example, one expert recommended requiring the manufacturers of plastic beverage bottles to produce bottles using at least a minimum amount of recycled plastic. According to the expert, this would increase the demand for recycled plastic as a raw material, which in turn would reduce the likelihood that such plastic would end up as waste. The expert said that requiring the use of recycled plastic would likely impose increased costs on manufacturers because virgin plastic—the raw material typically used in producing plastic beverage bottles—is currently less expensive than recycled plastic. Such increases would likely be short term, however, because the increased demand would decrease the price after more of the recycled material is used, according to another expert. Some federal agency officials said that establishing such proposed federal design standards could be difficult due to limited existing statutory authorities. Requirements for fishing gear. Three experts suggested the federal government establish requirements to mitigate the impact of lost or derelict fishing gear in federal waters. For example, one expert suggested requiring the use of modified fishing gear, such as crab traps with biodegradable escape mechanisms that allow entrapped marine life to escape if the trap is lost or abandoned (see fig. 4). Requiring the use of fishing gear with biodegradable escape mechanisms would likely impose increased costs to the fishing industry, according to the expert, but those costs could be minimized if the federal government offered a subsidy to help purchase required gear. NOAA officials said that it would be challenging to require the use of certain types of fishing gear in part because of the cost to the federal government in ensuring implementation of the requirement. On the other hand, NOAA officials said they promote innovation and voluntary use of certain types of fishing gear through various efforts such as their Fishing for Energy program. Restrictions on single-use plastics. Four experts suggested that the federal government establish restrictions on the manufacturing or sale of certain single-use plastics. For example, the federal government could establish restrictions on the manufacturing and distribution of plastic bags in the form of thickness or material composition requirements, or production volume limits. Two of these experts also said that the federal government could review existing local, state, and international efforts to restrict single-use plastics to identify best practices so that these types of actions could potentially be scaled appropriately at the federal level. According to the United Nations Environmental Programme, 127 countries and two states have placed various types of restrictions on the retail distribution of plastic bags as of 2018. One expert pointed to research that shows that plastic bags are one of the most abundant forms of marine debris and suggested that banning them would therefore significantly reduce the amount of debris entering the marine environment. NOAA officials agreed that restricting the sale of single-use plastic bags could help address the marine debris problem, but said that identifying an agency with sufficient legal authority to be responsible for implementing and enforcing any restriction would be important and could be a challenge at the federal level. NOAA and EPA officials said that it would be important to carefully determine and assess trade-offs or other potential impacts before considering these types of restrictions. Single-use plastics are any plastic items— such as plastic soda or water bottles—that are intended for use only once before they are thrown away or recycled as defined by the United Nations Environment Programme. Single-use plastics can have environmental impacts when they are left in the marine environment. For example, single-use plastics may be ingested by hundreds of species of marine wildlife, such as turtles and dolphins, who mistake them for food, potentially blocking their airways and stomachs, according to a 2018 report by the United Nations Environment Programme. Incentives for waste management. Four experts suggested actions the federal government could take to provide incentives to local governments to help them improve their waste management and recycling programs. The experts said that waste and water management is typically the responsibility of local governments, but that given the scope and scale of the marine debris problem, the federal government could use its resources to provide incentives to help local governments make improvements. For example, the federal government could provide grants or subsidies to help local governments implement best management practices, such as using trash traps to help remove debris from waterways and prevent it from becoming marine debris. In addition, the experts said that the federal government could provide local governments with resources to help purchase bins with lids to help prevent inadvertent loss of waste or to pay for infrastructure such as trucks and recycling facilities to improve the collection and recycling of waste. According to one expert, transporting materials from consumers to the appropriate waste management or recycling facilities is a significant barrier to achieving better waste management. EPA officials agreed with the importance of local waste management efforts. The officials emphasized that it is the agency’s mission, in part, to address management of waste to prevent trash, and management of water that carries the trash to the marine environment. The officials said that this is particularly critical for addressing marine debris since an estimated 80 percent of aquatic trash originates from land-based sources. The officials said the agency has provided some funding to local governments to implement mechanisms to capture trash before it enters waterways or to remove trash from water. They added that there is no one size fits all approach, however, to working with local governments. Rather, different localities may have differing needs—such as for funding, information, or technical assistance—and EPA tries to create a climate where localities can identify and best address those needs, according to the officials. Five of the 14 experts suggested the federal government support marine debris cleanup and removal activities by providing resources to organizations that coordinate cleanup projects (see fig. 5). Several agency officials said that preventing waste from entering the marine environment should be the primary focus of addressing marine debris, but cleaning up existing marine debris continues to be a critical part of the multi-faceted response to the problem, especially after severe weather events such as hurricanes. According to one expert, debris deposited into the marine environment around the Florida Keys after Hurricane Irma in 2017 included construction debris from demolished buildings, household items such as refrigerators and televisions, cars, and boats, among other types of debris. The expert suggested the federal government provide funding and technical assistance to state and local governments to help locate such debris. According to the expert, after a severe weather event, the distribution of debris can vary greatly with ocean and wind currents, and the debris can extend for miles into the ocean. As a result, the expert suggested that the federal government assist with conducting aerial flyovers to locate major concentrations of debris. The flyovers would employ mapping technology, such as global positioning system equipment and cameras, to locate and map the debris for removal. NOAA officials agreed with the importance of cleanup activities, particularly after severe weather events. In 2018, NOAA provided $18 million to states for the detection, removal, and disposal of debris after the 2017 hurricanes. Ten of the 14 experts suggested actions related to research or technology development. A few experts commended federal research efforts related to marine debris to date but stressed that additional research is needed in multiple areas. Examples of research and technology development actions suggested by experts include: Research on sources, pathways, and location of marine debris. Five experts suggested the federal government support research on identifying and understanding the various sources, pathways, and location of marine debris. For example, one expert suggested that the federal government conduct a national study to identify where waste is generated, through which types of major pathways it enters the marine environment (such as rivers or stormwater), and where the waste ends up. This study could include a focus on specific pathways, such as where illegal dumping occurs, which has not been researched at the national level, according to the expert. The expert said that federal agencies and others could use the results of such a study to help target education for the public, policy makers, and law enforcement officials on how to prevent and properly dispose of the types of waste that most commonly end up as marine debris. NOAA officials said that illegal dumping tends to be localized, so it may be difficult to carry out research on a national scale, but agreed with the need to better understand sources and types of marine debris since many factors contribute to the problem. Research on effects of marine debris. Four experts suggested the federal government support research to determine the effects of debris on wildlife and the marine environment as well as on human health. For example, one expert suggested that the federal government conduct or fund research to determine the effects of microplastics on human health to help the federal government and other stakeholders identify the most appropriate solutions. EPA officials said that this type of research is one among many competing areas related to marine debris research their agency has targeted. Development of technology to address marine debris. Five experts suggested actions that the federal government could take to develop new technology to help address marine debris. For example, one expert suggested that the federal government fund the development of new technology to recycle hard-to-recycle plastic materials so that these materials are less likely to end up as waste and become marine debris. The expert said that, in particular, plastic materials such as packaging used to preserve food products are not readily recyclable because the technology to recycle these types of plastics is not available or is not economically viable. EPA officials said that even when there is technology to recycle these types of plastics, food contamination is a problem that may prevent them from being recycled. In addition, an increased capacity for recycling may not result in a behavior change on the part of the consumer, which is another factor to consider in evaluating whether to pursue this type of action, according to the officials. Nine experts suggested that the federal government coordinate with local, state, federal, and international governments and other nonfederal partners to address marine debris. Experts emphasized that because marine debris is a complex issue with domestic and international impacts, it requires contributions from and coordination across these many groups. Examples of coordination suggested by experts include: Coordination with stakeholders on management of fishing gear. Two experts suggested the federal government coordinate to identify ways to prevent fishing gear from becoming a source of marine debris and causing harm to fish and other marine species. One expert suggested the federal government coordinate with stakeholders to identify and implement best management practices for responsible management and use of fishing gear. Specifically, the expert suggested that the federal government coordinate with state agencies, gear designers and manufacturers, fishermen, and other stakeholders to adopt best practices in particular locations such as in the Chesapeake Bay or Puget Sound where there are extensive commercial or recreational fisheries. The expert said it would be important to work with industry stakeholders to avoid the best practices being perceived as unnecessary government intervention. In addition, one of the experts said that adoption of best practices could incur additional costs for activities such as replacing gear, which could be minimized through government subsidies or other incentives. NOAA officials said these types of coordination activities align with current efforts within their Marine Debris Program. For example, in 2016 NOAA partnered with California State University and other stakeholders to encourage the adoption of best practices to prevent the loss of gear used to catch spiny lobster in the Channel Islands in California. Coordination with international governments. Four experts suggested the federal government increase its coordination internationally such as through developing international agreements and participating in multinational forums. For example, one expert suggested that the United States and other countries enter into an international agreement to prevent further release of plastic into the ocean. Under such an agreement, each country would set a target to reduce the amount of plastic released into the ocean, develop strategies and approaches to meet that target, and measure and report on progress in meeting the target. The expert said that taking actions to meet the target would incur costs and that securing commitments from countries could be difficult. However, the expert said that allowing countries the flexibility to develop their own strategies for meeting their targets could help overcome these difficulties. State Department officials said that in addition to coordination with international governments, coordination is needed with other key stakeholders such as waste management and marine debris experts, local leaders, private-sector industry and retail entities, and nongovernmental organizations. This is in part because so much of the international marine debris problem stems from waste management issues at the local level. In some countries, as in the United States, the government may not have the authority to work on waste management at the local level and as a result, understanding this complexity is an important factor to consider in coordinating internationally, according to the officials. USAID officials agreed that coordination with international stakeholders beyond international governments is needed and said that given the local nature of waste management issues that contribute to the international marine debris problem, stakeholders such as local and municipal governments are also important and should be a major focus for coordination and capacity building. Marine debris is a global, multi-faceted problem and multiple federal agencies, along with nonfederal stakeholders such as nongovernmental organizations, industry, states, Indian tribes, and others, have important roles to play in addressing the problem. The interagency committee’s sharing of information about its members’ activities is a good first step to ensure the agencies are aware of their respective marine debris-related efforts. NOAA, as chair of the committee, has recognized the need to develop a documented membership process, but has not established a time frame for doing so. By establishing a time frame for developing a documented membership process, NOAA and the interagency committee can benefit from capturing all members’ activities, and ensuring it provides Congress a complete picture of marine debris efforts across the federal government. NOAA also recognizes that it may be helpful to specify the level of the official needed to represent the agencies through revisions to its charter, but has not determined what those revisions may entail. By clarifying what is meant by “senior official” such as through revisions to its charter, NOAA would have greater assurance that it has the full engagement of member agency officials who can speak for their agency and commit to activities. The interagency committee’s biennial reports provide information on the committee’s recommendations and individual agencies’ activities to implement those recommendations, but the reports do not include an analysis of the effectiveness of the committee’s recommendations and strategies as required by the Marine Debris Act. By developing and implementing a process to analyze the effectiveness of the interagency committee’s recommendations and strategies, and reporting the results in its biennial reports as required, the interagency committee would be in a better position to determine the extent to which its efforts are making a difference in addressing the complex facets of marine debris. Additionally, the interagency committee has not identified required recommendations for priority funding needs. By developing a process to identify recommendations for priority funding needs and including such recommendations in its biennial reports, the interagency committee could provide the Congress with required information about priority funding needs across the federal government to address marine debris. We are making a total of four recommendations, including two recommendations to the NOAA Administrator and two recommendations to the chair of the interagency committee, specifically: The NOAA Administrator, in coordination with interagency committee member agencies, should establish a time frame for documenting the committee’s membership process. (Recommendation 1) The NOAA Administrator, in coordination with interagency committee member agencies, should clarify what is meant by “senior official” in the Marine Debris Act, such as through revisions to its charter. (Recommendation 2) The chair of the interagency committee, in coordination with member agencies, should develop and implement a process to analyze the effectiveness of the interagency committee’s recommendations and strategies, and include the results in its biennial reports. (Recommendation 3) The chair of the interagency committee, in coordination with member agencies, should develop a process to identify recommendations for priority funding needs to address marine debris, and include such recommendations in its biennial reports. (Recommendation 4) We provided the Departments of Commerce, Defense, Homeland Security, Interior, Justice, and State; EPA; the Marine Mammal Commission; and USAID a draft of this report for their review and comment. The Department of Commerce and USAID provided written comments, which are reprinted in appendixes IV and V respectively, and discussed below. We also received technical comments from the Departments of Commerce, Homeland Security, the Interior, and State; EPA; the Marine Mammal Commission; and USAID, which we incorporated into the report as appropriate. The Departments of Defense and Justice indicated that they had no comments. In written comments from the Department of Commerce, Commerce and NOAA agreed with our four recommendations. Regarding our first two recommendations, NOAA stated that its Administrator will establish a time frame for documenting the interagency committee’s membership process and, in coordination with the interagency committee, will define the term “senior official” through revisions to its charter so that the term can be consistently applied across all federal agency structures. In forming its definition of “senior official,” NOAA indicated that it would consider seniority requirements of similarly situated advisory committees, along with related factors such as the ability to make decisions on behalf of an agency. Regarding our third recommendation on developing and implementing a process to analyze the effectiveness of the interagency committee’s recommendations and strategies, NOAA stated that it agreed with this recommendation to the extent it can be implemented with available budgetary resources. It indicated that the interagency committee lacks the existing resources to require and routinely evaluate the effectiveness of agency activities. Instead, individual agencies are expected to work toward implementing the interagency committee’s 2008 recommendations in accordance with each agency’s legal and programmatic authorities, mission priorities, and resource limitations. Nevertheless, NOAA stated that to the extent possible it will work with interagency committee members to identify common or easily translatable metrics for evaluating the effectiveness of its 2008 recommendations and include these in the next biennial report to Congress. Regarding our fourth recommendation, NOAA stated that it agreed with our recommendation, but noted that it does not have the authority to control the implementation of a process for identifying priority funding needs of other member agencies. It stated that the interagency committee’s recommendations for priority funding needs are already reflected in the President’s annual budget request and operating plan for each member agency. However, NOAA stated that to the extent possible, it will work with interagency committee members to develop a process for identifying priority areas, which can be reflected in each agency’s respective budgeting process and shared in the committee’s biennial reports. We agree that NOAA does not have the authority to control the implementation of a process for identifying priority funding needs of other member agencies. However, as chair of the committee, NOAA can coordinate with member agencies to develop a process that each individual member agency—under its individual authority and budgetary processes—can use to identify recommendations for priority funding needs to address marine debris. We believe that coordinating such information and providing it in the committee’s biennial reports could provide Congress with required information about priority funding needs across the federal government to address marine debris. In addition, in written comments from USAID, the agency said it is committed to addressing the challenge of marine debris through its programs and in collaboration with interagency committee partners. USAID stated that it has significant opportunities to play an important role in the international response to address marine debris and, as the lead federal agency on foreign assistance, has several programs that target mismanaged municipal waste in the developing world. For example, USAID stated that the agency’s Municipal-Waste Recycling Program has helped reduce land-based sources of ocean plastic waste in four of the top five contributing countries—Indonesia, the Philippines, Sri Lanka, and Vietnam—by providing small grants and technical assistance to a variety of local actors in towns and cities. USAID also stated that it greatly appreciates the work of its interagency committee partners in addressing marine debris and looks forward to continued collaboration with them. We are sending copies of this report to the appropriate congressional committees; the Secretaries of Commerce, Defense, Homeland Security, Interior, Justice, and State; the Administrators of EPA and USAID; and the Commissioners of the Marine Mammal Commission. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. This report examines (1) how the interagency committee coordinates among federal agencies and the process for determining membership and agency representation, (2) the extent to which the interagency committee’s biennial reports contain required elements, and (3) experts’ suggestions on actions the federal government could take to most effectively address marine debris. To examine how the interagency committee has coordinated among federal agencies and the process for determining membership and agency representation, we reviewed the Marine Debris, Research, Prevention, and Reduction Act, as amended (Marine Debris Act), and interagency committee documents, including the committee’s 2008 report with recommendations, charter, and five biennial reports to Congress issued as of March 2019. Specifically, we reviewed meeting minutes from the interagency committee’s quarterly meetings from November 2012 through April 2019, to understand the topics and activities the committee has coordinated on and the federal agencies that have participated. We attended five of the interagency committee’s quarterly meetings (in May, September, and December of 2018, and April and July of 2019) to directly observe committee coordination among agencies during these meetings. We also reviewed documents from committee member agencies and interviewed and reviewed written responses from those agencies to obtain information on their coordination efforts. Agencies we included were those agencies designated as members in the Marine Debris Act as well as additional agencies identified as members in the committee’s charter (see table 2). In addition, we interviewed officials and reviewed documents from the National Science Foundation, Office of the U.S. Trade Representative, and the U.S. Agency for International Development, based on suggestions from interagency committee officials. From the committee’s 2008 report with recommendations, the five biennial reports, and other member agency documents, we summarized activities conducted by member agencies. For reporting purposes, we selected examples from the 2016 and 2019 biennial reports (those most recently available) of activities the agencies have taken to illustrate interagency committee member efforts to address marine debris, to reflect a range of activities across categories of activities and member agencies. In addition, we compared information we received about the interagency committee’s coordination to leading practices we identified in our past work on implementing interagency collaborative mechanisms. To examine the extent to which the interagency committee’s biennial reports contain required elements, we compared information contained in the committee’s five biennial reports to the statutory reporting requirements in the Marine Debris Act. Specifically, two analysts independently reviewed each of the five biennial reports to evaluate information the reports included about (1) the status of implementation of any recommendations and strategies of the committee, (2) analysis of the recommendations and strategies’ effectiveness, (3) estimated federal and nonfederal funding provided for marine debris, and (4) recommendations for priority funding needs. The analysts then compared and summarized the results of their analyses. We also interviewed and reviewed written responses from National Oceanic and Atmospheric Administration (NOAA) officials (in the agency’s capacity as chair of the interagency committee) and officials from other members of the committee about steps to develop the biennial reports, including the reports’ required elements. In addition, we compared information from the reports and the information we received from the officials to leading practices we identified in our past work on implementing interagency collaborative mechanisms. To obtain suggestions on actions the federal government could take to most effectively address marine debris, we conducted structured interviews with a nongeneralizable sample of 14 experts with expertise in marine debris-related issues. We selected the experts from a list of individuals we identified through interviews with agency officials and through a snowball approach, in which we reviewed relevant literature on marine debris, such as articles the experts authored, to identify other key experts and asked experts to identify other experts for including in this review. We also identified experts through our participation in key marine debris events, such as presenting at the Sixth International Marine Debris Conference. We considered factors such as the individual’s experience with different types of debris (e.g., abandoned fishing gear or consumer debris) or association with various sectors (e.g., academia or industry). Experts selected included: (1) academics with expertise in areas such as sources, prevalence, and transport of plastic marine debris; (2) officials representing the plastic manufacturing, food and beverage, and commercial fishing industries; (3) officials from nonprofit organizations with expertise in marine debris removal from coastal areas, litter prevention, and recycling management systems and strategies; and (4) state and local government officials from the District of Columbia, Florida, and Washington with expertise in local litter prevention efforts, derelict vessels, and lost and derelict fishing gear. We asked the 14 experts to suggest up to 5 to 10 actions the federal government could take to most effectively address different types of marine debris. We defined the term “actions” to mean any policy, program, effort, or intervention that could be taken by the federal government to prevent, remove, or dispose of marine debris. Actions could include new actions that the federal government may not have implemented or actions the federal government may already have taken. We did not limit experts’ suggestions to actions that agencies currently have authority to implement. We do not take a position on the merits of, the necessary legal authority for, or the most appropriate entity for the actions suggested by the 14 experts. Prior to the interview, we provided experts with background information about our review, the interview methodology, and definitions for key terms to ensure that terminology was used consistently throughout all the interviews. We also reviewed this information with each expert at the start of the interview. For each action, we asked that the expert identify: Name of action; Type(s) of debris: (Select any or all of the following types of marine debris that may be affected by the action: consumer-based, abandoned fishing gear, derelict vessels, and/or miscellaneous. If miscellaneous is selected, please explain); Describe this action: (Briefly describe this action and how it will address (i.e. prevent, remove, or dispose) marine debris and if it is currently being implemented by the federal agencies); Federal agency(ies) (Please briefly describe the federal agency(ies) that have implemented or could play a role in implementing the action); Nonfederal partners: (Please briefly describe the nonfederal partners the federal agencies may need to coordinate with when implementing the action (such as international, state and local governments, nonprofit groups, industry, and/or researchers); Advantages: (Briefly describe the advantages of the federal agencies implementing the action in terms of the ability of this action to address marine debris, the cost of the action, and the technical and administrative feasibility of implementing the action, or any other advantage that you believe may affect implementation); Disadvantages: (Briefly describe the disadvantages of the federal agencies implementing the action in terms of the ability of this action to address marine debris, the cost of the action, and the technical and administrative feasibility of implementing the action, or any other disadvantage that you believe may affect implementation); Challenges: (Describe any factors that may hinder this action from being successfully implemented by the federal agencies and how these factors may be overcome); Examples: (In instances where the federal agencies have previously implemented the action, please provide examples of how it helped address marine debris. If other entities that are not federal agencies have successfully implemented the action, please provide examples of how the action helped address marine debris); Authorities: (Briefly describe what legal authorities these actions would be implemented under. If new authorities are needed, please describe them); and Support: (Provide any studies, reports, or research you are basing your responses on). We conducted the interviews via teleconference between July 2018 and November 2018. The experts suggested over 70 actions that we organized into five categories based on common themes. Specifically, two analysts independently reviewed each expert’s description for individual actions and identified an appropriate category using decision rules the team developed. The analysts then discussed and compared their decisions. For actions the analysts categorized differently, they reviewed the decision rules together and came to agreement on the best category for a particular action. For reporting purposes, we selected several actions within each of the broader categories to provide illustrative examples of the types of actions experts suggested. Our selection of actions was based on a variety of factors, including our analysis of the number experts that suggested similar types of actions, the detail provided by the experts, and the availability of supporting information, such as instances where an action had been taken by state or local governments. Actions suggested by the 14 experts cannot be generalized to actions that might be suggested by other experts but provide examples of actions federal agencies could take to address marine debris. We also obtained written and oral responses to questions we asked of agency officials regarding factors their agencies would need to consider in potentially implementing any of the actions identified by the 14 experts. In addition, to corroborate statements from experts and agency officials and provide additional context on marine debris, we reviewed scientific studies and documents from international organizations, such as the United Nations; academic institutions and nonprofit organizations such as the Ocean Conservancy; and federal and state agencies to understand what is known about the types, sources, and effects of marine debris. We identified these studies and documents through various means, such as recommendations from experts and agency officials and authorship by experts. We also interviewed individuals from academia, environmental groups, and industry actively working on marine debris issues and attended the Sixth International Marine Debris Conference held in San Diego, California, in March 2018, to gain an understanding of areas of emphasis in the marine debris community. We conducted this performance audit from October 2017 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Table 3 lists the 25 recommendations contained in the Interagency Marine Debris Coordinating Committee’s 2008 report entitled Interagency Report on Marine Debris Sources, Impacts, Strategies, and Recommendations. According to this report, these recommendations are intended to guide the federal government’s strategies with respect to addressing problems of persistent marine debris. Each of the five biennial reports the committee issued subsequent to its initial 2008 report reference the 25 recommendations; the committee has not revisited the recommendations to determine the extent to which any adjustments may be warranted. The following are examples of activities members of the Interagency Marine Debris Coordinating Committee (interagency committee) reported conducting—often in coordination with nonfederal partners such as nongovernmental organizations, industry, state governments, Indian tribes, and other nations—to address marine debris based on information from the committee’s 2016 and 2019 biennial reports and agency documents and interviews. These examples include activities from the categories outlined in the biennial reports: (1) education and outreach; (2) legislation, regulation, and policy; (3) cleanup; (4) research and technology development; and (5) coordination. The examples discussed below do not represent all activities conducted by member agencies, but rather illustrate the nature and type of activities the agencies reported conducting. In addition, the examples include activities from agencies that were identified in the interagency committee’s 2014 charter and were included in the committee’s most recent biennial reports. Nine of the 11 member agencies reported conducting activities to support education and outreach related to addressing marine debris, such as developing and distributing educational materials, supporting public awareness campaigns, or partnering with or funding state, local, tribal, or nongovernmental education efforts. For example: Online public education. The Trash Free Waters Program—a program established in the spring of 2013 by the Environmental Protection Agency (EPA) to encourage collaborative actions by public and private stakeholders to prevent trash from entering water— provides information to the public, including online information about actions that can be taken to reduce trash from entering waterways. For example, in 2017, the program produced a series of eight webinars with experts on microplastics with the goal of promoting increased knowledge of the sources, distribution, and impacts of plastics and microplastics in the environment. Additional topics included research on global waste management and mismanagement of plastics, potential replacements for plastic products, and ways to improve the design of materials and products to minimize their environmental impacts. Grants for public awareness projects. The National Oceanic and Atmospheric Administration’s (NOAA) Marine Debris Program awards grants to eligible entities to, among other things, develop projects to educate the public about various aspects of preventing marine debris. For example, in 2014, NOAA awarded one grant to Virginia State’s Department of Environmental Quality to develop and implement a social marketing approach to reduce balloon debris. Balloons can end up in streams, rivers, and the oceans where marine animals can ingest the balloons or become entangled by their attachments, causing injury or death. This project aimed to help educate the public about the importance of refraining from releasing balloons in parks or outside schools, churches, wedding venues, or other events where balloons may be common. Sea Partners Program. Through its Sea Partners Program established in 1994, the U.S. Coast Guard Auxiliary conducts education and outreach to waterway users such as boaters, fishermen, marina operators, marine industry, and the general public with information on protecting the marine environment. For example, its Sayreville, New Jersey unit reaches an annual average audience of about 10,000 people, according to a program document, including youth groups, primary and secondary education science classes, senior citizen groups, and others. Topics presented include an introduction to marine pollution and oil spills and environmental pollution and recreational boating. Nine member agencies reported conducting activities to identify noncompliance or help ensure compliance with laws and regulations and develop or encourage policies and programs to implement practices that address specific types of marine debris. For example: Notice for offshore oil and gas operators. In November 2018, the Bureau of Safety and Environmental Enforcement renewed a notice for offshore oil and gas lessees and operators in the Gulf of Mexico that clarifies and provides more detail about marine trash and debris awareness training. Specifically, the notice stated that all offshore employees and contractors active in offshore operations are to complete marine debris awareness training annually. The notice further specifies that lessees and operators are to provide the bureau with an annual report that describes their training process and certifies that the training process was followed. Criminal enforcement of environmental laws. The Department of Justice prosecuted two shipping companies in 2017 for, among other things, falsifying records regarding disposal of garbage from a ship, in violation of the Act to Prevent Pollution from Ships. Specifically, the ship’s crew was instructed to throw plastic garbage bags filled with metal and incinerator ash overboard without recording the incidents in the ship’s record book. The companies pled guilty and were, among other things, sentenced to pay a $1.5 million fine and make a $400,000 community service payment. Policies for financing waste management infrastructure in Asia. The Department of State helped convene a meeting in Japan in 2016, under the Asia-Pacific Economic Cooperation framework, to discuss policy changes needed to overcome barriers to financing waste management infrastructure in the Asia-Pacific region to prevent and reduce debris from entering the marine environment. The meeting brought together government officials from the economic cooperation, representatives from industry, international financial institutions, and experts. Ministers of the economic cooperation endorsed nine recommendations developed at the meeting. State Department officials said they have continued to work with Asian governments, industry, and nongovernmental organizations to encourage policy changes and spur financial support for increasing waste management infrastructure and addressing land-based sources of plastic and in Asian countries. For example, at a 2017 meeting on waste management, State Department officials informed Asia-Pacific Economic Cooperation officials of the social and economic impacts of marine debris resulting from mismanaged waste in the region. Officials also said they used the meeting to connect economic cooperation officials with private sector stakeholders to encourage policy changes intended to enable private investment in waste management. Eight of the 11 member agencies reported conducting a variety of activities to support the removal and disposal of marine debris, often in partnership with others, such as state governments. For example: Debris removal grants. In 2016 and 2017, NOAA’s Marine Debris Program awarded $2.4 million in grants to 25 entities such as state and tribal governments in 17 coastal states and U.S. territories for projects including community cleanups, crab trap recovery, and derelict vessel removal. For example, in September 2017, the program awarded a grant to the Makah Indian Tribe to remove three sunken vessels from the Makah Marina within the Makah Tribe Indian Reservation on Washington’s Olympic Peninsula. National Park cleanup. National Park Service staff conducted coastal cleanups across the various regions of the National Park System during 2016 and 2017. For example, in fiscal year 2017, park officials from Biscayne National Park, located off the coast of Southern Florida and comprised mostly of water, partnered with the Coastal Cleanup Corporation, a nonprofit organization, to organize 252 volunteers in removing 14,000 pounds of debris from the park. Maintaining navigation channels. The U.S. Army Corps of Engineers has authority to remove accumulated snags, obstructions, and other debris located in or adjacent to federally-maintained navigation channels. The Corps’ operations and maintenance appropriation is available to pay for the removal of obstructions to navigation, and the Corps is sometimes directed to use this appropriation for drift removal. For instance, in fiscal year 2018, the explanatory statement accompanying the Corps’ annual appropriation directed the Corps to use about $9.9 million of its appropriation for drift removal in New York Harbor. Debris the Corps removes typically consists of lumber, trees and branches, large waste items like tires, and large plastic items, according to Corps’ officials. Five of the 11 member agencies reported coordinating activities to conduct or sponsor research to monitor, understand the sources of, prevent, mitigate, or reduce the effects of marine debris or to support developing new technologies such as using more sustainable or recyclable types of materials. For example: Research grants. Since 2006, NOAA’s Marine Debris Program has supported at least two marine debris research projects that address questions such as monitoring marine debris, identifying fishing gear improvements and alternatives, or better understanding the environmental or economic impacts of marine debris. For example, in 2016, NOAA awarded a contract to a private research and consulting firm to conduct an economic study on how marine debris affects the economies of tourism-dependent coastal communities around the United States. The purpose of the project was to evaluate changes in tourism spending based on changes in the amount of marine debris to help prioritize areas of the United States where future prevention and removal efforts may be needed. NOAA officials said they expect the final report to be issued by the end of 2019. Microplastics workshop. In June 2017, EPA hosted a Microplastics Experts Workshop that convened experts from academia and other federal agencies, including NOAA, the U.S. Geological Survey, and the Food and Drug Administration, to identify microplastics research needs. The effort resulted in a 2018 report that identified four main areas where additional research is needed: (1) standardization of research methods, (2) debris sources and fate, (3) ecological risk assessment, and (4) human health risk assessment. EPA is using the report to consider how the agency can best address these high- priority microplastics research needs as it develops the agency’s larger environmental research agenda, according to EPA officials. Development of new fishing gear. In 2016, the Marine Mammal Commission awarded a grant to the New England Aquarium to test a ropeless fishing gear prototype intended to prevent whale entanglements in fishing gear. According to a document from the Commission, entanglement in fishing gear is the number one direct cause of marine mammal injury and death, including the endangered Northern Atlantic right whale. The Commission has used the results of this effort to emphasize the potential for ropeless gear to reduce and prevent entanglement in meetings with lobster and crab fishermen on the east and west coasts. Seven of the 11 member agencies reported conducting a variety of activities to foster coordination among member agencies and with nonfederal partners, such as international, state, and local government agencies. For example: Global Partnership on Marine Litter. In 2012, the United Nations launched the Global Partnership on Marine Litter, a voluntary network of international governments, nongovernmental organizations, academia, private sector companies, and others with the goal of protecting human health and the global environment primarily by reducing and managing marine debris. Interagency committee members, including NOAA and EPA, are partners to the global partnership. For example, from 2012 through 2017, the NOAA Marine Debris Program Director served as the Steering Committee chair of the global partnership. EPA has coordinated with the global partnership in Latin American and Caribbean countries to help develop a regional strategy for addressing marine debris in those regions and through in-person meetings and with other global partnership staff and NOAA colleagues through the steering committee. Sister Cities initiative. In 2015, the State Department announced the creation of a “Sister Cities” initiative with China to share best practices related to waste management and preventing marine debris. As part of the initiative, in November 2016, a Chinese delegation, comprised of central government officials and officials from Weihai and Xiamen, visited Chicago, New York City, and San Francisco to study U.S. practices in addressing marine debris. In November–December 2017, a U.S. delegation comprised of U.S. government officials and a New York City official, visited Xiamen, Weihai, and Beijing to learn about Chinese waste management practices. The partner city relationships were formalized with a memorandum of understanding between San Francisco and Xiamen in July 2016, and New York and Weihai in December 2017 to work together to address marine debris. State emergency response guides and regional action plans. NOAA’s Marine Debris Program has coordinated with coastal managers, nongovernmental organizations, industry, academia, and other groups to develop state marine debris emergency response guides. For example, in 2016 and 2017, NOAA coordinated with Florida, Georgia, Mississippi, North Carolina, and South Carolina to develop individual guides for those states. According to NOAA officials, federal, state, and local officials used the Florida response guide during the 2017 and 2018 hurricane seasons to inform responding agencies which agency has jurisdiction and to better coordinate marine debris removal efforts after an event. In addition, NOAA coordinated efforts to develop, enhance, and implement regional action plans for the Great Lakes, the Gulf of Maine, the Gulf of Mexico, the Mid-Atlantic, the Southeast, California, Florida, Hawaii, Oregon, and Washington regions. The purpose of the action plans is to bring stakeholders together to prevent and reduce marine debris throughout the United States, according to NOAA documents. For example, NOAA officials said that under the Hawaii action plan, several federal agencies and nongovernmental organizations worked together to purchase and maintain bins to collect used fishing line for recycling. In addition to the contact named above, Alyssa M. Hundrup (Assistant Director), Mark Braza, Jeanette Soares, Jason Trentacoste, and Lisa Vojta made key contributions to this report. Eric Charles; Kim Frankena; Ellen Fried; Karen Howard; Edward J. Rice, PhD.; Dan C. Royer; Anne Stevens; and Sarah Veale also contributed to the report.", "summary": "Marine debris—waste such as discarded plastic and abandoned fishing gear and vessels in the ocean—is a global problem that poses economic and environmental challenges. The Marine Debris Act, enacted in 2006, requires the committee to coordinate a program of marine debris research and activities among federal agencies. The act also requires the committee to submit biennial reports to Congress that include certain elements such as an analysis of the effectiveness of the committee's recommendations. GAO was asked to review federal efforts to address marine debris. This report examines (1) how the committee coordinates among federal agencies and the process for determining membership, (2) the extent to which the committee's biennial reports contain required elements, and (3) experts' suggestions on actions the federal government could take to most effectively address marine debris. GAO examined the Marine Debris Act and committee reports, compared committee practices with leading collaboration practices, interviewed federal agency officials, and interviewed a nongeneralizable sample of 14 marine debris experts selected to reflect various sectors and experiences with different types of marine debris. The Marine Debris Research, Prevention, and Reduction Act, as amended, (Marine Debris Act) designated six agencies as members of the Interagency Marine Debris Coordinating Committee and specifies that members shall include senior officials from certain other agencies as the Secretary of Commerce determines appropriate. Within Commerce, the National Oceanic and Atmospheric Administration (NOAA) serves as the committee chair. The committee coordinates through sharing information about members' activities to address marine debris, but GAO found that NOAA has not established a process for determining committee membership for agencies not specifically designated in the act. As a result, such agencies may not be included in the biennial reports required by the act which discuss committee members' marine debris activities. NOAA officials said they plan to develop a membership process but have not established a time frame to do so. By establishing a time frame, the committee can more fully benefit from capturing all members' activities. The committee's biennial reports provide information on members' activities such as education and cleanup, but they do not contain some information required by the Marine Debris Act. Specifically, the reports do not include (1) an analysis of the effectiveness of the committee's recommendations and strategies to address marine debris and (2) recommendations for priority funding needs. Our past work has shown that collaborative entities can better demonstrate progress if they develop a way to monitor and report the results of their collective efforts and identify and leverage resources. By doing so, the committee would be in a better position to know the extent to which it is effectively addressing marine debris and provide Congress with required information about priority funding needs. Experts suggested a range of actions—from research to cleanup—the federal government could take to most effectively address marine debris. They stressed that there is not one solution to the growing problem (see figure). Committee officials noted factors to consider, such as cost, when evaluating these actions. GAO is making four recommendations, including that NOAA establish a time frame for documenting membership and the committee develop processes to analyze the effectiveness of its efforts and identify priority funding. The agency agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Black lung benefits include both cash assistance and medical benefits. Maximum cash assistance payments ranged from about $660 to $1,320 per month in 2018, depending on a beneficiary’s number of dependents. Miners receiving cash assistance are also eligible for medical benefits that cover the treatment of their black lung-related conditions, which may include hospital and nursing care, rehabilitation services, and drug and equipment charges, according to DOL documentation. DOL estimates that the average annual cost for medical treatment in fiscal year 2018 was approximately $9,667 per miner. There were about 25,600 total beneficiaries (primary and dependents) receiving black lung benefits during fiscal year 2018 (see fig. 1). The number of beneficiaries has decreased over time as a result of declining coal mining employment and an aging beneficiary population, according to DOL officials. Black lung beneficiaries could increase in the near term due to the increased occurrence of black lung disease and its most severe form, progressive massive fibrosis, particularly among Appalachian coal miners, according to National Institute for Occupational Safety and Health (NIOSH) officials. Black lung claims are processed by the Division of Coal Mine Workers’ Compensation in the Office of Workers’ Compensation Programs (OWCP) within DOL. Contested claims are adjudicated by DOL’s Office of Administrative Law Judges (OALJ), which issues decisions that can be appealed to DOL’s Benefits Review Board (BRB). Claimants and mine operators may further appeal these agency decisions to the federal courts. If an award is contested, claimants can receive interim benefits, which are generally paid from the Trust Fund according to DOL officials. Final awards are either funded by mine operators—who are identified as the responsible employers of claimants—or the Trust Fund, when responsible employers cannot be identified or do not pay. In fiscal year 2018, black lung claims had an approval rate of about 34 percent, according to DOL data. In 2009, we reported on the benefits adjudication process and made several recommendations for DOL that could improve miners’ ability to pursue claims. An April 2015 DOL Inspector General (IG) report followed up on DOL’s progress on our recommendations and found continuing problems and raised new concerns about the black lung claims and appeals process. For instance, the IG reported that OALJ needed to address staff shortages, improve communication between its headquarters and district offices, and upgrade the training provided to judges and law clerks. To further expedite claim adjudication, the IG recommended, among other things, that OALJ begin hearing more cases remotely using video or telephone hearings to reduce judges’ travel costs and time. In fiscal year 2018, OWCP reported that it took about 335 days on average to issue a decision on a claim. This is an increase from the average of 235 days that OWCP had reported to the DOL IG for fiscal year 2014. Trust Fund revenue is primarily obtained from mine operators through the coal tax. The current coal tax rates, which took effect in 2019, are $0.50 per ton of underground-mined coal and $0.25 per ton of surface-mined coal, up to 2 percent of the sales price. Coal tax revenue is collected from mine operators by Treasury’s Internal Revenue Service and then transferred to the Trust Fund where it is then used by DOL to pay black lung benefits and the costs of administering the program. Trust Fund expenditures include, among other things, black lung benefit payments, certain administrative costs incurred by DOL and Treasury to administer the black lung benefits program, and debt repayments. When necessary for the Trust Fund to make relevant expenditures under federal law, the Trust Fund borrows from the Treasury’s general fund. When this occurs, the federal government is essentially borrowing from itself—and hence from the taxpayer—to fund its benefit payments and other expenditures. As we reported in 2018, Trust Fund expenditures have consistently exceeded revenue. The Trust Fund borrowed from Treasury’s general fund almost every year since 1979, its first complete fiscal year. We noted in our 2018 report that Trust Fund borrowing would continue to increase through 2050 due, in part, to the planned coal tax rate decrease of about 55 percent that took effect in 2019 and declining coal production. We simulated the effects of the tax rate decrease on Trust Fund finances through 2050, and reported the results of a moderate case set of assumptions related to future coal production and prices and the number of new black lung beneficiaries. These simulations were not predictions of what will happen, but rather models of what could happen given certain assumptions. Our moderate case simulation suggested that Trust Fund revenue may decrease from about $485 million in fiscal year 2018 to about $298 million in fiscal year 2019, due, in part, to the approximate 55 percent decrease in the coal tax rate. Our simulation, which incorporated EIA data on future expected coal production, also showed that annual Trust Fund revenue would likely continue to decrease beyond fiscal year 2019 due, in part, to declining coal production. Domestic coal production declined from about 1.2 billion tons in 2008 to about 775 million tons in 2017, according to EIA. Based on these projections, our moderate simulation showed that Trust Fund annual revenue may continue to decrease from about $298 million in fiscal year 2019 to about $197 million in fiscal year 2050. Future simulated Trust Fund revenue would likely be insufficient to cover combined black lung benefit payments and administrative costs, according to our moderate case simulation. Specifically, revenue may not be sufficient to cover beneficiary payments and administrative costs from fiscal years 2020 through 2050 (see fig. 2). For instance, in fiscal year 2029, simulated benefit payments and administrative costs would likely exceed simulated revenue by about $99 million. These annual deficits could decrease over time to about $4 million by fiscal year 2050 due, in part, to the assumed continued net decline in total black lung beneficiaries. If Trust Fund spending on benefit payments and administrative costs continues to exceed revenues each year, then the Trust Fund would need to continue borrowing from Treasury’s general fund to cover those costs, as well as borrowing to cover debt repayment. Our moderate simulation suggested that the Trust Fund’s outstanding debt could increase from about $4.2 billion in fiscal year 2019 to about $15.4 billion in fiscal year 2050 (see fig. 3). While our moderate case simulated a $15.4 billion Trust Fund debt in 2050, the amount could vary from about $6 billion to about $27 billion depending, in part, on future coal production and the number of new beneficiaries. Even if the Congress were to completely eliminate black lung benefits as of fiscal year 2019, the Trust Fund’s outstanding debt in fiscal year 2050 could still exceed $6.4 billion, according to our simulation. Eliminating black lung benefits, however, would generally mean that coal tax revenue would be collected solely to fund the repayment of Trust Fund debt. As we reported in 2018, other options such as adjusting the coal tax and forgiving interest or debt, could also reduce future borrowing and improve the Trust Fund’s financial position (see GAO-18-351). Federal law generally requires that coal operators secure their black lung benefit liability. Operators can purchase commercial insurance for this purpose or may self-insure if they meet certain DOL conditions. For example, self-insurers must obtain collateral in the form of an indemnity bond, deposit or trust, or letter of credit in an amount deemed necessary and sufficient by DOL to secure their liability. DOL officials said that the collateral they required from the five self- insured operators that filed for bankruptcy between 2014 and 2016 was inadequate to cover their benefit liabilities. For example, the collateral DOL required from Alpha Natural Resources was about 6 percent of its estimated benefit liability. As a result, approximately $185 million of estimated benefit liability was transferred to the Trust Fund, according to DOL data. We reviewed DOL documentation related to the five operator bankruptcies. Table 1 shows the bankrupt operators; the amount of collateral each operator had at the time of bankruptcy; estimated benefit liability at the time of bankruptcy; and estimated benefit liability and number of beneficiaries that transferred to the Trust Fund, if applicable. Overall, three of these bankruptcies affected the Trust Fund, and two did not according to DOL. DOL officials told us that the bankruptcies of Arch Coal and Peabody Energy did not affect the Trust Fund because their benefit liabilities were assumed by the reorganized companies after emerging from bankruptcy. As of June 2019, there are 22 operators that are self-insured and actively mining coal, according to DOL officials. To ensure that the collateral they required from these operators was adequate to protect the Trust Fund, DOL officials said that they periodically reauthorized them which entailed, among other things, reviewing their most recent audited financial statements and claims information. DOL officials said that they prepared memos documenting these reviews and communicated with coal operators about whether their financial circumstances warranted increasing or decreasing their collateral. Table 2 provides information on the 22 self-insured operators including the date of each operator’s most recent DOL reauthorization; the amount of DOL required collateral; and the operator’s most recent estimated black lung benefit liability. Should any of these operators file for bankruptcy, they could also affect the Trust Fund because the amount of an operators’ benefit liability that is not covered by collateral could also become the responsibility of the Trust Fund. Preliminary analysis from our ongoing work indicates that DOL did not regularly monitor self-insured operators. Agency regulations state that DOL may adjust the amount of collateral required from self-insured operators when experience or changed conditions warrant. We reviewed DOL’s most recent reauthorization memos for each of the 22 operators. While some of these operators had been reauthorized more recently, we found that others had not been reauthorized by DOL in decades. One operator in particular had not been reauthorized by DOL since 1988. Additionally, for most of these operators, DOL either did not have estimates of their benefit liabilities, or the estimates were out of date (see table 2). Beginning in summer 2015, DOL officials said that they stopped permitting any new coal mine operators to self-insure as the agency worked with auditors, economists, and actuaries to develop new procedures for self-insurance. At the same time, DOL generally stopped reauthorizing the 22 self-insured operators. Earlier this year, two of these operators have filed for bankruptcy—Westmoreland Coal Company and Cloud Peak Energy—according to DOL officials. Additionally, due to deteriorating financial conditions, DOL recommended revoking another operator’s self-insurance authority (Murray Energy). However, Murray appealed this decision and DOL postponed responding to the appeal until their new self-insurance procedures are implemented, according to DOL officials. DOL’s new self-insurance procedures are currently being reviewed by OMB, and DOL officials said they did not know when they would likely be implemented. Until such procedures are implemented, DOL cannot ensure that the collateral it has required from self-insured operators is adequate to protect the Trust Fund should these operators become insolvent. Chairwoman Adams, Ranking Member Byrne, and Members of the Subcommittee, this concludes my prepared statement. I would be happy to respond to any questions you may have at this time. If you or your staff has any questions concerning this testimony, please contact me at (202) 512-7215. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contact named above, Blake Ainsworth (Assistant Director), Justin Dunleavy (Analyst in Charge), Angeline Bickner, Alex Galuten, Courtney LaFountain, Rosemary Torres Lerma, Kate van Gelder, Catherine Roark, and Almeta Spencer made key contributions to the testimony. Other staff who made key contributions to the reports cited in the testimony are identified in the source products. Black Lung Benefits Program: Options to Improve Trust Fund Finances, GAO-18-351 (Washington D.C: May 30, 2018). Mine Safety: Basis for Proposed Exposure Limit on Respirable Coal Mine Dust and Possible Approaches for Lowering Dust Levels, GAO-14-345 (Washington, D.C.: April 9, 2014). Black Lung Benefits Program: Administrative and Structural Changes Could Improve Miners’ Ability to Pursue Claims, GAO-10-7 (Washington, D.C.: October 30, 2009). Federal Compensation Programs: Perspectives on Four Programs for Individuals Injured by Exposure to Harmful Substances, GAO-08-628T (Washington, D.C.: April 1, 2008). Mine Safety: Additional Guidance and Oversight of Mines’ Emergency Response Plans Would Improve the Safety of Underground Coal Miners, GAO-08-424 (Washington, D.C.: April 8, 2008). Mine Safety: Better Oversight and Coordination by MSHA and Other Federal Agencies Could Improve Safety for Underground Coal Miners, GAO-07-622 (Washington, D.C.: May 16, 2007). This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Since 2009, GAO has produced a body of work on the Black Lung Benefits Program. In 2018, for instance, GAO reported that the Trust Fund, which pays benefits to certain coal miners, faced financial challenges due, in part, to the coal tax rate decrease that took effect in 2019 and declining coal production. Trust Fund finances could be further strained by coal mine operator bankruptcies, as they can lead to benefit liabilities being transferred to the Trust Fund. This testimony describes Trust Fund finances through 2050 and provides preliminary observations from ongoing work for this committee regarding the Department of Labor's (DOL) oversight of coal mine operator insurance. To describe Trust Fund finances, in its 2018 report GAO developed simulations through 2050 based on various assumptions related to future coal production and the number of future black lung beneficiaries. To develop preliminary observations from its ongoing work, GAO analyzed DOL documentation and data on black lung beneficiaries and coal mine operators. GAO also reviewed relevant federal laws, regulations, policies, and guidance and interviewed DOL officials, insurance carriers, and coal mine operators, among others. GAO reported in 2018 that Black Lung Disability Trust Fund (Trust Fund) expenditures have consistently exceeded revenue. The Trust Fund borrowed from the Department of the Treasury's (Treasury) general fund and hence from the taxpayer almost every year since 1979, its first complete fiscal year, causing debt and interest to accumulate. Federal law does not limit the amount the Trust Fund may borrow as needed to cover its expenditures. Trust Fund revenue will be further limited by the coal tax rate decrease of about 55 percent that took effect in 2019, and declining coal production, according to GAO's simulation. Specifically, Trust Fund revenue may not be sufficient to cover beneficiary payments and administrative costs, from fiscal years 2020 through 2050. Therefore, the Trust Fund could need to continue borrowing to cover its expenditures—including the repayment of past debt and interest—and the Trust Fund's simulated outstanding debt could exceed $15 billion by 2050 (see figure). However, as GAO reported in 2018, various options, such as adjusting the coal tax and forgiving debt, could improve the Trust Fund's financial position. GAO's preliminary observations indicate that Trust Fund finances will be further strained by coal operator bankruptcies. Since 2014, an estimated black lung benefit liability of over $310 million has been transferred to the Trust Fund from insolvent self-insured coal mine operators, according to DOL data. Federal law generally requires that operators secure their black lung benefit liability. To do so, operators can self-insure if they meet certain DOL conditions. As of June 2019, there are 22 operators that are self-insured and actively mining coal, according to DOL officials. GAO's preliminary analysis indicates that DOL did not regularly review these operators so that it could adjust collateral as needed to protect the Trust Fund. As a result, the amount of collateral DOL required from some of these operators is tens of millions less than their most recent estimated black lung benefit liability. GAO will be considering recommendations, as appropriate, when ongoing work is finished.", "document_type": "gao"}
{"report": "DHS generally uses the information sources that EEOC guidance recommends to help identify potential barriers. As directed by EEOC guidance, DHS analyzes its workforce data to help identify triggers or indicators of potential EEO barriers by comparing the racial, national origin, gender, and disability profiles of its total workforce, and for various occupational categories to relevant civilian labor workforce data. In addition to analyzing workforce data, in each of the fiscal years 2014 through 2017, DHS utilized the U.S. Office of Personnel Management’s Federal Employee Viewpoint Survey and DHS’s employee exit survey results to help identify and address barriers. To further help identify barriers, EEOC guidance states that agencies must solicit input from agency employee and advocacy groups, and union officials. During our small group discussions, DHS employee groups told us that through the MD-715 report development process, they helped identify and address triggers and barriers. For example, Special Emphasis Program Managers we spoke with told us that DHS components conduct climate surveys to obtain input from employees on workforce practices every 1 or 2 years. Further, several DHS components’ MD-715 reports referenced soliciting employee input, such as obtaining Disability Employment Program Managers’ input via quarterly disability employment advisory council meetings where they share best practices and discuss issues and topics including barriers. DHS reports some improvements in employee engagement and representation of minorities and women. DHS’s employee engagement scores in the Federal Employee Viewpoint Survey increased from 54 percent in 2014 to 62 percent in 2019. In addition, our review of DHS’s workforce data from fiscal years 2014 through 2017 showed that every minority group as well as individuals with disabilities and individuals with targeted disabilities had been trending in a positive direction since fiscal year 2014. Further, DHS officials told us that minority representation was up 3 percent and female representation was up 2 percent from 2015 to February 2019. According to EEOC, one important tool in examining the fairness and inclusiveness of an agency’s recruitment efforts is applicant flow data. EEOC guidance states that having department-wide applicant flow data could aid in analyzing differences in selection rates among different groups for a particular job. In July 2017, EEOC informed DHS that the agency’s applicant flow data were incomplete, which makes it difficult to pinpoint barriers. DHS has reported challenges in collecting department- wide data because the department does not have a consolidated applicant flow data system. According to DHS, four of its components use one system (USA Staffing), while five other components use a different system (Monster Government Solutions). CRCL officials told us that DHS is developing a new system to integrate applicant flow data department-wide. However, the officials could not give us a time frame for when the system is expected to be completed. In its fiscal year 2018 MD-715 report, DHS reported that it continues to work towards developing a central repository for all applicant flow data. As a work-around, DHS officials said that it obtains these data directly from each component that uses Monster Government Solutions. In its fiscal year 2018 MD-715 report, DHS reported that it used applicant flow data to complete analyses, but it also reported a number of limitations, including that data were not available. In February 2020, CRCL officials told us that they plan to report complete applicant flow data in DHS’s fiscal year 2019 MD-715 report. DHS does not have complete performance metrics or mechanisms for tracking progress towards eliminating its identified EEO barriers, such as workplace satisfaction of white females or the retention rate of women in law enforcement positions. According to CRCL officials, they are not required to establish performance metrics or mechanisms for tracking progress towards eliminating barriers beyond what is included in the department-wide MD-715 report. However, Standards for Internal Control in the Federal Government states that management should establish specific and measureable objectives, and ways to assess progress including performance metrics and milestones. Further, EEOC guidance states that agencies are not prevented from establishing additional practices that exceed its requirements. Implementing performance metrics could help DHS assess its progress in eliminating EEO barriers. Accordingly, our July 2019 report included a recommendation that the Secretary of Homeland Security should develop performance metrics for the department’s EEO program including a mechanism for tracking progress towards eliminating barriers. DHS concurred with the recommendation and stated that it would implement it by April 30, 2020. In February 2020, CRCL officials told us they are working with DHS’s Management Directorate to develop a potential overarching performance metric that, if approved, would be implemented beginning in fiscal year 2021. Our analysis of DHS’s MD-715 reports found that the department-wide EEO program did not meet about a quarter of the compliance measures for a model EEO program for each fiscal year from 2014 through 2017. For example, in each of the fiscal years 2015 through 2018, DHS reported that senior managers at DHS components did not successfully implement EEO action plans and incorporate EEO action plan objectives into agency strategic plans. In addition, our analysis of components’ MD- 715 reports showed that component EEO programs did not meet 9 percent of the compliance measures for a model EEO program from fiscal years 2014 through 2017. DHS components did not have action plans to address nearly half (179 out of 369) of the deficiencies self-reported by all components from fiscal years 2014 through 2017. For example, in fiscal year 2017, four DHS components did not have action plans to ensure that their EEO directors report directly to their agency heads. EEOC guidance requires that for each deficient measure, agencies are to develop an action plan for correcting the deficiency. CRCL officials told us that DHS and its components’ MD-715 reports met EEOC requirements for action plans for fiscal years 2014 through 2017 by providing explanations for, or briefly stating plans to address, the majority of their deficiencies rather than developing action plans identifying how each deficiency would be addressed. Developing policies and procedures to help ensure components’ EEO programs have action plans for addressing deficiencies could help DHS components better comply with EEOC requirements. DHS and its components lack adequate staffing to address EEO program deficiencies, in part, because CRCL and component EEO officials told us that they do not have formal staffing models to assess appropriate staffing of their EEO program sections. CRCL officials said that each component EEO program section is unique with its own assessments and measures by the leaders in charge of their funding and staffing resources. However, EEOC MD-715 guidance states that an agency must provide its EEO program with sufficient budget and staffing to be able to successfully implement various activities. Developing and utilizing formal staffing models—a tool to determine the number of staff required—for their EEO programs could help DHS and its components to identify, request, and obtain the staff they need. Thus, in our recently issued report, we recommended that (1) DHS component EEO Directors, in consultation with the Deputy Officer for EEO and Diversity, should develop policies and procedures to help ensure that their component EEO programs have action plans for addressing deficiencies in their MD-715 reports, and (2) the Deputy Officer for EEO and Diversity should develop a formal staffing model for its EEO program. DHS concurred with the recommendations and stated that it would implement them by April 30, 2020. In February 2020, CRCL officials told us that they are developing policies and procedures for components to consider. They also told us that they are collaborating with the DHS Management Directorate to develop a formal staffing model for DHS’s department-wide EEO program. In addition, we recommended that DHS component EEO Directors, in collaboration with the Deputy Officer for EEO and Diversity, develop component formal staffing models. DHS concurred with the recommendation and stated that it would implement it by July 31, 2020. In February 2020, CRCL officials told us that the DHS Management Directorate plans to work with components to develop formal staffing models for their individual EEO programs after the agency develops a formal staffing model for the department-wide EEO program. DHS has plans to address the nine areas of noncompliance in its EEO program identified by EEOC. For example, in its July 2017 review of DHS compliance with EEOC requirements, EEOC identified that DHS did not provide complete demographic data on new hires and promotions in its fiscal year 2016 report to EEOC. In April 2019, DHS officials told us that the department plans to report the data by collecting complete data from DHS components in fiscal year 2019. In its fiscal year 2018 MD-715 report, which DHS sent to EEOC in July 2019, DHS stated that it had collected and analyzed demographic data on new hires and promotions. DHS’s EEO and human capital offices assist and support DHS components in identifying and addressing EEO barriers. For example, CRCL meets with each component to obtain updates on their EEO efforts and provide verbal feedback as they develop their MD-715 reports. DHS components told us that they are generally satisfied with CRCL’s collaboration practices to identify and address EEO barriers. For example, all nine components required to submit MD-715 reports told us that CRCL regularly meets with them and provides guidance on identifying and addressing barriers. From fiscal years 2014 through 2017, EEOC found areas of noncompliance in DHS and its components’ EEO programs. We found that DHS components had not responded timely and completely to areas of noncompliance identified in EEOC feedback letters. According to CRCL officials, CRCL does not have policies and procedures to ensure that components have addressed EEOC’s feedback letters in a complete and timely manner. However, EEOC MD-715 guidance states that an agency’s EEO Director ultimately is responsible for ensuring equal opportunity throughout the entire agency. In addition, Standards for Internal Control in the Federal Government states that management should implement control activities through policies. Developing policies and procedures for responding completely and timely to EEOC’s feedback letters may help the department comply with EEOC guidance. CRCL officials said they lack authority to ensure components’ compliance with EEOC requirements. Standards for Internal Control in the Federal Government states that an effective management practice includes periodically evaluating the agency’s organizational structure to ensure that it meets its objectives. DHS has not taken steps—in consultation with EEOC and other agencies as relevant—to analyze options to address EEO program management weaknesses. Specifically, it has not analyzed alternatives for granting additional authorities to the Deputy Officer for EEO and Diversity to ensure DHS components comply with MD-715 guidance, or assessed benefits and trade-offs of each alternative. Without addressing these issues, DHS may not be effectively positioned to manage its EEO program. In our report, we recommended that the (1) Deputy Officer for EEO and Diversity develop policies and procedures for responding in a complete and timely manner to EEOC’s feedback letters, and (2) the Secretary of Homeland Security—in consultation with CRCL and EEOC, and other agencies and components, as relevant—analyze options for granting additional authorities to the Deputy Officer for EEO and Diversity to ensure DHS components comply with MD-715 guidance, including the authority of the Deputy Officer for EEO and Diversity to certify components’ MD-715 reports. DHS concurred with the recommendations and stated that it plans to implement them by April 30, 2020. In February 2020, CRCL officials told us they are developing policies and procedures for responding in a complete and timely manner to EEOC’s feedback letters. They also told us that a cross-component working group, with input from EEOC subject- matter experts, is developing a report benchmarking best practices at similar federal agencies that it expects to complete by the end of March 2020. In conclusion, as the third largest U.S. government department, the challenges DHS has faced to fully implement effective EEO programs may result in widespread negative consequences such as (1) monetary expenses borne by the agency in connection with workplace disputes and (2) decreased morale and productivity resulting from ineffective and inefficient use of human capital resources. We found areas for improvement in DHS and its components’ EEO programs that could help ensure success and compliance with MD-715. The commitment of DHS’s leadership is essential to successfully addressing these issues. By focusing leadership attention on developing performance metrics, policies and procedures, and staffing models, DHS and its components can help improve their EEO programs by making progress towards eliminating barriers, obtaining sufficient staffing, and addressing areas of noncompliance. Madam Chairwoman Torres Small, Ranking Member Crenshaw, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions you may have at this time. If you or your staff have any questions about this testimony, please contact Yvonne D. Jones at (202) 512-6806 or jonesy@gao.gov, or Christopher P. Currie at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this testimony include Clifton G. Douglas, Jr. (Assistant Director), Luis E. Rodriguez (Analyst-in-Charge), Andrew Howard, Kate Lenane, Steven Putansu, and Rachel Whitaker. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "EEOC's Management Directive 715 requires that, to attract and retain top talent, federal agencies are to identify EEO barriers in their workforces and deficiencies in their EEO programs, execute plans to address them, and report annually to EEOC. GAO reported in 2009 on DHS's opportunities to address barriers to EEO in its workforce and in 2019 on DHS's challenges to ensuring EEO in its workforce. GAO was asked to testify on the steps DHS has taken to (1) identify and address barriers to EEO in its workforce, (2) identify and address EEO program deficiencies, (3) address areas of noncompliance in its EEO program identified by EEOC, and (4) oversee and support components' EEO programs. To do so, GAO summarized the findings discussed in its July 2019 report on DHS's EEO efforts and reported on DHS's actions taken to address recommendations. To obtain updates on actions taken by DHS, GAO reviewed relevant documentation and interviewed DHS EEO officials. The Department of Homeland Security (DHS) uses multiple information sources to identify potential barriers to equal employment opportunity (EEO), but lacks performance metrics for tracking its progress towards eliminating identified barriers. DHS generally uses the information sources that the Equal Employment Opportunity Commission (EEOC) guidance recommends, such as employee survey results, to help identify potential barriers. While DHS reports some improvements in employee engagement and representation of minorities and women from fiscal years 2014 through 2018, it does not have complete performance metrics, such as the retention rate of women in law enforcement positions. Using performance metrics could help DHS assess its progress in eliminating barriers. DHS and its components have identified various deficiencies in their EEO programs, but lack policies and procedures for developing action plans and formal staffing models to address deficiencies. For example, in each of the fiscal years 2015 through 2018, DHS reported that senior managers at DHS components did not successfully implement EEO action plans and incorporate EEO action plan objectives into agency strategic plans. Further, DHS components lacked action plans to address nearly half (179 out of 369) of the deficiencies self-reported by all components from fiscal years 2014 through 2017. For example, in fiscal year 2017, four DHS components did not have action plans to ensure that their EEO directors report directly to their agency heads, as required by EEOC guidance. Developing policies and procedures to help ensure components' EEO programs have action plans for addressing deficiencies could help DHS components better comply with EEOC requirements. In addition, developing and using formal staffing models—a tool to determine the number of staff required—for their EEO programs could help DHS and its components to identify, request, and obtain the staff they need. For example, DHS and its components reported that staffing challenges contributed to some of their program deficiencies, and acknowledged they did not have formal staffing models for their EEO programs. DHS has plans to address nine areas of noncompliance in its EEO program identified by EEOC. In its July 2017 review of DHS compliance with EEOC requirements, EEOC found that DHS did not provide complete demographic data on new hires and promotions in its fiscal year 2016 report to EEOC. DHS reported to EEOC that it had collected and analyzed such demographic data beginning in fiscal year 2019. DHS's EEO and human capital offices assist and support DHS components in identifying and addressing EEO barriers. However, DHS's EEO office lacks policies and procedures to ensure components respond timely and completely to areas of noncompliance identified in EEOC feedback letters. Additionally, DHS EEO officials said they lack authority to ensure components' compliance with EEOC requirements. Without addressing these issues, DHS may not be effectively positioned to manage its EEO program. In its July 2019 report, GAO recommended that DHS take six actions, including develop performance metrics for the department's EEO program; develop DHS and component formal staffing models; and analyze options for granting additional authorities to the most senior official for EEO and Diversity. DHS concurred with the six recommendations and described actions the department plans to take to address them.", "document_type": "gao"}
{"report": "Since the early 1980s, the Air Force has been working to modernize and consolidate its space command and control systems and improve its space situational awareness. Effective command and control systems are important because DOD space capabilities are globally distributed and operated from geographically diverse locations. With new threats against space assets, the ability to quickly respond or take action can mean the difference between mission success and failure. Space situational awareness is the current and predictive knowledge and characterization of space objects and the operational environment upon which space operations depend. Good space situational awareness data are the foundation of command and control systems because the data are critical for planning, operating, and protecting space assets and informing government and military operations. The Air Force’s last three space command and control programs over more than three decades have ended significantly over budget and schedule, and key capabilities have gone undelivered. Those programs were the Cheyenne Mountain Upgrade, the Combatant Commanders’ Integrated Command and Control System, and the Joint Space Operations Center Mission System. Some capabilities were deferred from one program to the next, making the true cost growth in each program significantly higher when compared to original program content. This deferral was due in part to the complicated nature of the planned work. Enabling a single system to command and control numerous assets in space and on the ground at multiple levels of information classification is a technically challenging task. In addition, as discussed below, we found that the Air Force made optimistic cost and schedule estimates for these programs, and thus did not assign adequate resources to their development. Begun in 1981, the Cheyenne Mountain Upgrade was intended to modernize systems that provide critical strategic surveillance and attack warning and assessment information. We issued 11 reports on the Cheyenne Mountain Upgrade program between 1988 and 1994. In 1991, we found that the program planned to complete only a portion of its requirements in an attempt to stay within budget and schedule constraints. We also found that the Air Force had adopted a strategy of deferring some requirements on the optimistic assumption that these requirements could be achieved during later stages of system development. We concluded that while such deferrals may have permitted the Air Force to meet revised short-term goals, they also masked the magnitude of problems the program experienced as it moved forward. We also found that DOD had not formally evaluated the performance risks related to deferring requirements and concluded that the strategy of deferral significantly raised the risk that system development would be more costly and take longer. DOD declared the program operational in 1998; however, some critical capabilities were not delivered. At that time, the program was nearly $1 billion over budget and 11 years late. That same year, DOD determined that some of the program’s components were not well integrated and would be unresponsive to future mission needs. DOD initiated the Combatant Commanders’ Integrated Command and Control System program in 2000 to modernize and integrate the Cheyenne Mountain Upgrade computer systems and to replace a space situational awareness data computer system called the Space Defense Operations Center (SPADOC). At that time, the SPADOC system was significantly overtaxed and in need of replacement by a system that could handle larger volumes of data. In 2006, we found that Combatant Commanders’ Integrated Command and Control System program costs had increased by approximately $240 million, 51 percent over initial estimates, and the program was at least 3 years behind schedule. In addition, we found that that some capabilities had been deferred indefinitely, resulting in increased risks to performing future operations. Further, we found that the Air Force did not effectively assess the appropriateness of the program’s requirements prior to initiating the program, leading to significant additions, deletions, and modifications to the program’s initial requirements. Consequently—similar to what transpired within the Cheyenne Mountain Upgrade program—significant amounts of work were deferred to address the cost increases associated with requirements changes. Ultimately, the Combatant Commanders’ Integrated Command and Control System program was not able to successfully replace SPADOC. Started in 2009, the Joint Space Operations Center Mission System (JMS) was the Air Force’s most recent effort to meet command and control capability and space situational awareness data needs and replace the SPADOC system. JMS was a software-intensive system and was supposed to be delivered in three increments. Increment 1 was to provide the foundational structure for the overall program. Increment 2 was to deliver numerous operational capabilities to users, including replacing SPADOC by the end of fiscal year 2014 with the ability to automatically determine if objects in space were likely to collide (called conjunction assessments), which was a key performance parameter for the program. Increment 3 was to provide additional command and control capabilities and the ability to incorporate data from highly classified special access programs. Of the three planned increments, Increment 1 is the only one that is fully operational today. JMS Increment 2 encountered significant challenges during development, and in 2016 the program experienced a critical change because of significant schedule delays and cost increases. Specifically, JMS Increment 2 planned to delay delivery by more than 1 year, in turn increasing total program costs by over 25 percent. According to the August 2016 JMS Critical Change Report, which the program office submitted to Congress in September 2016 as a result of the critical change, several issues contributed to Increment 2’s challenges. These included an overly aggressive schedule, inadequate staffing, underestimating the amount of work required to integrate various pieces of the system that were developed by different groups, and numerous concurrent development efforts. An independent program assessment team comprised of military, intelligence, and contractor staff determined that the JMS program had underestimated the complexity of developing the system. Further, the program reported that its organizational structure proved problematic. For example, the program reported that program- related contracts were awarded and administered outside the program office, which limited program flexibility and support and hampered effective oversight. As a result of the critical change, the program re- estimated its costs, established new schedule goals, and deferred a number of capabilities and requirements to Increment 3. Even after these changes, JMS Increment 2 was not successful at delivering its planned capabilities. Air Force operational testing in 2018 revealed significant issues with JMS Increment 2 performance. The Air Force’s test team determined that Increment 2 was not suitable for operations, as it was unable to provide conjunction assessments or maintain the catalog of space objects, another key performance parameter. In the wake of these findings and the numerous issues found in testing, the Air Force stopped further development on JMS Increment 2. When development ended, JMS was almost 3 years behind schedule and $139 million (42 percent) over budget. Air Force leadership placed the JMS Increment 2 program in sustainment and transferred three of the 12 planned Increment 2 capabilities into operations; the remaining nine capabilities were to be used for planning and analytic purposes only, as they were not reliable enough for operational use. Key requirements from Increment 2, including automated conjunction assessments and the ability to maintain a high-accuracy space catalog, as well as all of the requirements from Increment 3, were deferred to a subsequent effort, called the Space C2 program. Because JMS was unable to replace SPADOC, the system is still in use today. Since 2000, the Air Force has been addressing unique space surveillance requirements for follow-on systems to SPADOC. Air Force officials we spoke with stated that the system’s ability to continue operations is a growing concern. While work is underway to move SPADOC onto a more modernized platform and infrastructure, the Air Force has not established a schedule for that effort. In the meantime, Air Force officials told us that large amounts of data are going unprocessed as the volume of available sensor data today is greater than ever before—and is expected to increase exponentially in the next year as new DOD sensors come online. The Space C2 program is the Air Force’s latest software-intensive program to develop capabilities to anticipate and respond to emerging threats in space and ensure the uninterrupted availability of capabilities to the warfighter. SPADOC is expected to be retired as Space C2 capabilities become operational. The Air Force expects to spend between $72 million and $108 million per year on the Space C2 program, which is managed by the Air Force’s Space and Missile Systems Center, through fiscal year 2024. While it is still early in the planning and development stages, the Air Force’s Space C2 program office expects to deliver a consolidated space command and control system over the next few years using a new system design. The program also plans to use a modernized, iterative software development process called Agile development to more quickly and responsively provide capability to users. According to Air Force officials, this development approach is relatively new to DOD programs. Therefore, the Space C2 program and DOD officials are working to determine the appropriate level of detail needed for the program’s planning documents as well as the best way to provide oversight of a non-traditional development approach. The Space C2 program is intended to consolidate operational level command and control capabilities for DOD space assets into an integrated system, allowing operators and decision makers to have a single point of access to command and control space assets around the globe in a timely manner. A consolidated space command and control capability will: allow operators to comprehensively identify and monitor threats to identify possible courses of action to mitigate or eliminate threats, communicate courses of action to decision makers, and direct action to respond to threats. A consolidated space command and control capability is necessary, according to Air Force and DOD officials we met with, because the space domain has transitioned from a benign environment to one that—like ground, sea, and air domains—is contested by foreign adversaries. According to these officials, DOD needs the ability to respond to the increased threats to U.S. space assets in near real-time. Consequently, the Air Force is planning for Space C2 program capabilities to be significantly more automated than in the past, requiring high-quality software development and architecture planning. As shown in figure 1, the Space C2 program itself will consist of multiple layers. Program officials explained that the foundational layer is the computing infrastructure, which must be secure from vulnerabilities and have adequate processing power to accommodate the complexity of the system. On this infrastructure will run the software platform, which forms the backbone of the operating system. The Space C2 program plans to procure the platform commercially. The software platform will contain standards that developers will need to comply with to create applications that will work on the platform. Some applications may be targeted to a broad number of users, and some may be more niche capabilities for a particular group of users. Space C2 program officials told us they believe this structure will allow them to be flexible in meeting multiple user needs more responsively than has been possible in past DOD programs. Users include, for example, space system operators responsible for predicting and avoiding space object collisions, and other operators responsible for responding to conflicts in space. The program also expects applications from a variety of developers, both commercial and government, to run on the platform, thus presenting opportunities for companies that do not regularly do business with DOD to participate in the program. The work being done for the Space C2 program is spread out among multiple Air Force groups. For example, the Air Force Research Laboratory has been developing applications for the Space C2 program both internally and with commercial partners since 2016. The Laboratory is also working on some battlespace awareness capabilities that may eventually run on the Space C2 program’s platform. Additionally, officials from the Air Force Rapid Capabilities Office stated that they have been working on common interface standards for applications, and this work will feed into the Space C2 program. As the Enterprise Manager, the Space C2 program manager is responsible for integrating all of the development work selected for use in the Space C2 program, irrespective of its origin. A principal component of the Space C2 program is a data repository that will be populated with data from a wide variety of commercial, civil, military, and intelligence space sensors. Eventually the program plans for operators using the Space C2 program’s platform and applications to be able to retrieve data from the data repository. The data will be electronically tagged with its appropriate classification level and will be accessible to users according to their individual security clearances. The overall design of the Space C2 program is for data to be gathered from sensors, placed into the data repository, and then be available for various applications to process and provide timely information to space operators and commanders on threats to space assets and anomalies in the space environment. Operators and commanders will then be able to promptly direct actions, such as tasking sensors to collect additional data or respond to threats. Figure 2 shows the proposed construct of the Space C2 program, including the various actions that can be taken in response to the data collected by the sensors. The Space C2 program is planning to use an approach new to DOD in terms of software development, known as Agile. Agile development is a flexible, iterative way of developing software that delivers working capabilities to users earlier than the traditional, incremental DOD software development processes, known as the waterfall approach. Agile practices integrate planning, design, development, and testing into an iterative life cycle to deliver software early and often, such as every 60-90 days. The frequent iterations of Agile development are intended to effectively measure progress, reduce technical and programmatic risk, and be responsive to feedback from stakeholders and users. This is different from the way DOD has developed software in the past, in which requirements were solidified in advance of development and the software was delivered as a single completed program at the end of the development cycle—with no continual involvement or feedback from users or ability to modify requirements. Traditional software development mirrored the development of a hardware system. We have previously reported on past DOD software programs that experienced challenges due, in part, to that traditional development approach. The differences between the two approaches are illustrated in figure 3. The Space C2 program is one of the first DOD software-intensive programs to move away from the traditional approach and into the more modernized Agile development methodology. Program officials told us that many of the problems with JMS’s development stemmed from its more traditional approach, and that with the Space C2 program they wanted to avoid circumstances that did not lead to program success. Considering that past software development problems were caused, at least in part, by the traditional method of software development, utilizing a different approach could be a positive step. However, the current DOD acquisition instruction does not include guidance for Agile software programs. According to DOD officials, new software guidance is in development, and this guidance is expected to offer pathways for developing Agile programs. DOD has also developed a draft template to assist Agile programs with developing their acquisition strategies, though the template and associated software guidance are in the early stages of development. In the meantime, however, Space C2 program officials confirmed that they are currently operating without specific software acquisition guidance. Space C2 officials also clarified that while official Agile software acquisition guidance has not yet been formally published, the program office has been actively engaged with the Office of the Under Secretary of Defense for Acquisition and Sustainment on refining draft policy and guidance. The program office noted that its program activities over the past year have been informed by and are consistent with this draft guidance. The Space C2 program has submitted preliminary planning documents to the Under Secretary of Defense for Acquisition and Sustainment for approval. While officials in the Under Secretary’s office expect these documents to be modified and expanded upon in late 2019, the Under Secretary gave the program approval to begin its development under an Agile process, signifying her support for using alternative approaches. In addition, Air Force officials told us that the Commander of Air Force Space Command has requested frequent briefings on the program’s development process, and while he does not have approval authority over the program, he is monitoring the program closely. Plans show that the program is conducting 90-day development iterations with the goal of providing working software at the end of each cycle. As of August 2019, the program had completed three program development iterations, and reported delivering capabilities which included: expanding the commercial data available in the data repository; tasking various sensors; and providing a tool for visualization and analytics. The Air Force noted that these capabilities were deployed in a relatively short time; however, most capabilities delivered so far are considered to be available for use “at your own risk,” since they have not yet been fully approved for use in operations. Though the Air Force has not yet published a time frame for certifying these capabilities for operational use, the new development approach is underway and delivering some early capabilities. DOD officials noted that the foundational elements of the Space C2 system, including the infrastructure and software platform, should be completed prior to significant application development; however, at this early stage of the program, the schedule indicating the time frame in which these elements will be completed appears to be still in development. For government programs, some level of insight and oversight is essential when using public funds to develop a system. According to DOD officials, DOD is embracing Agile development because software can be delivered quickly and can be more responsive to user needs. However, according to GAO’s upcoming guide for assessing Agile development programs, known as the Agile Assessment Guide, sound engineering principles are still beneficial when employing this approach. For example, continuous attention to technical excellence and good design requires the developers to consider security requirements throughout development. This is particularly true with complex programs that process sensitive data with complex security requirements. In past work, we have found that teams overlooking security requirements may end up developing systems that do not comply with current federal requirements (for example cybersecurity requirements for information technology programs), resulting in the software not becoming operational until these components are addressed. In addition, the Agile Assessment Guide notes that transitioning to Agile software development can be challenging because Agile methods require organizations to do more than implement new tools, practices, or processes. Agile requires a re-evaluation of existing organizational structures, planning practices, business and program governance, and business measures, in addition to technical practices and tools. However, Agile does not mean eliminating the need for documentation, planning, oversight, architecture, risk analysis, or baseline schedule, for example. Leading practices for Agile software development—as described in GAO’s upcoming Agile Assessment Guide—state that, among other things, programs should have the following characteristics: a product owner who manages the requirements prioritization, communicates operational concepts, and provides continual feedback to the development team; staff who are appropriately trained in Agile methods; management that has established an Agile supportive environment; a program strategy that reflects the mission, architectural, safety- critical components, and dependencies; organization’s acquisition policy and guidance that require the contract type and the acquisition strategy to be aligned to support Agile implementation; an architecture that is planned upfront to enable flexibility and to provide support to Agile methods; and mission goals that drive the prioritization of the most advantageous requirements (e.g., security and privacy) that are well understood and reviewed throughout development. Recognizing the need to change traditional processes to accommodate more iterative software development, both the Air Force and Under Secretary of Defense for Acquisition and Sustainment have created software advisor positions. The Air Force Chief Software Officer and the Special Assistant for Software Acquisition are working to improve and modernize the way DOD acquires software. In addition, DOD is looking into how to use industry practices to modernize the way it develops software. For example, the Office of the Secretary of Defense has a Development Security Operations (DevSecOps) pathfinder program for software, which helps programs define and develop a technical digital roadmap and leverages industry and Office of the Secretary of Defense expertise in developing appropriate infrastructure for software programs. The DevSecOps concept emphasizes rapid prototyping, security, and continuous integration and delivery of software products. In a May 2019 Acquisition Decision Memorandum, the Under Secretary of Defense for Acquisition and Sustainment directed the Space C2 program to become a pathfinder program. This is a positive step, because it should increase input into the program’s acquisition planning by the Office of the Secretary of Defense software development experts. The Office of the Secretary of Defense has other groups that draw on private-sector software development expertise to help DOD programs, including the Defense Digital Service and the Defense Innovation Board. These groups’ missions include improving DOD’s technology and innovation, and the groups can be valuable DOD resources for helping the Space C2 program develop its plans and Agile processes. The Defense Innovation Board conducted a review of some of the Space C2 program’s software acquisition plans in December 2018. According to the Office of the Secretary of Defense officials we spoke with, this informal review was beneficial and resulted in real-time feedback on the approach the program was taking, as well as suggestions for areas to focus on. In the May 2019 memorandum, the Under Secretary of Defense for Acquisition and Sustainment noted that in October 2019 she will determine if an independent technical assessment of the Space C2 program is necessary. Considering the stated benefits of the prior Defense Innovation Board review of the Space C2 program, as well as the fact that using Agile processes for a DOD program is relatively new and includes many unknowns, independent reviews could help ensure the program is on a successful path. As the Office of the Secretary of Defense and the Air Force have made an effort to increase in-house Agile software development expertise, programs like the Space C2 program—especially in light of its early stage of development—could benefit from periodic attention from the experts at its disposal, including input from independent, external reviews to help ensure the necessary software development steps are taken to set programs up for success. DOD programs following traditional acquisition processes conduct internal reviews at major milestones, and GAO best practices for knowledge-based acquisitions also include conducting independent program reviews at these milestones. The draft GAO Agile Assessment Guide notes that while traditional DOD program milestone reviews are not used for Agile programs, Agile programs rely on other review methods such as stakeholder demonstrations and retrospective program reviews during each iteration of work. In addition, the GAO Schedule Assessment Guide, which identifies best practices for managing a program’s schedule, states that programs should conduct periodic reevaluations of risks, and that an independent view in this is valuable. Such reviews offer greater objectivity, as the reviewers are not responsible for the activities being evaluated, and programs benefit from the wide variety of expertise and experience represented by the external review team. In addition, in many cases, having these external reviews periodically can prove useful. The Space C2 program faces a number of management, technical, and workforce challenges. Some of these challenges may ultimately be overcome by time and experience, and the Air Force has efforts underway to mitigate others in the near-term. But it is too early to determine whether these efforts will be sufficient to achieve program success. The Space C2 program faces several management challenges. The Air Force has been working on developing various parts of the Space C2 program since 2016, but as previously noted, the program is working from a draft acquisition strategy and does not yet have an overall program architecture. These plans are important for providing direction for a program and facilitating effective oversight by establishing a business case for the effort. A business case establishes that the program is necessary and that it can be developed with the resources available, and typically includes: a requirements document, an acquisition strategy, sound cost estimates based on independent assessments, and a realistic assessment of risks, including those relating to technology and schedule. In addition, according to Air Force officials, the Space C2 Enterprise Manager has management responsibility—but not authority—over multiple development efforts included in the Space C2 enterprise. For example, technology maturation and risk reduction activities are divided across three program offices, managed by two program executive officers, and reliant upon multiple sources of information. This division of work is being done in part because the various organizations have areas of expertise that the program was hoping to leverage. However, such distribution of activities among many organizations can result in synchronization and coordination challenges. JMS’s development was hampered by similarly-split responsibilities for development contracts for various efforts. Because space is becoming an increasingly contested domain, DOD has noted that its ability to effectively respond to space threats has increased the importance of focused leadership in national security space, to include Space C2. See table 1 for additional details of management challenges facing the Space C2 program. According to officials from the Space C2 program and the Office of the Secretary of Defense, the Space C2 program was allowed to begin development work without an acquisition strategy, due to the program’s urgency. In May 2019, the Under Secretary of Defense for Acquisition and Sustainment tasked the Space C2 program office with revising its preliminary acquisition strategy to be consistent with DOD’s draft template for software acquisition. DOD’s draft template contains specific elements for ongoing planning and evaluation that are to be included in DOD software acquisition strategies moving forward, including acquisition and contracting approach; program management structure, including authorities and oversight plans for platform and infrastructure development; requirements management and development approach, and plans for prioritization; risk management plans, including how the program will identify and mitigate risks; metrics for measuring quality of software, and how those results will be shared with external stakeholders; manpower assessment identifying program workforce needs and state of expertise in Agile methods; requirements for reporting program progress to decision makers; and yearly funding levels. We have also noted these factors in our previous reports that identify the need to develop a sound, executable business case at the outset of a program, and the importance of using knowledge-based decision making in DOD acquisition programs. In addition, our work on best practices for knowledge-based acquisitions has emphasized that the success of any effort to develop a new product hinges on having the right knowledge at the right time, and that a better opportunity exists to meet program goals when the knowledge is available early. However, given that DOD’s draft template is still subject to change, including these elements in the finalized acquisition strategy would help position the program for success. The Space C2 program also faces significant technical challenges, as described in table 2. For example, the program is planning to meet previously deferred requirements that proved too complex for prior programs to achieve. It also plans to address new and emerging threats to space assets, for which requirements are not yet defined. In addition, the program plans to use an Agile software development approach, the processes of which DOD has yet to show proficiency in applying, as discussed above. Integration of the multiple types of software planned for Space C2 is also likely to present considerable technical challenges. Further, cybersecurity is a growing concern for DOD space programs, including Space C2. In addition to the management and technical risks we identified, limited availability of staff with expertise in Agile software development poses a challenge to the Space C2 program and to DOD in general. The Space C2 program manager stated that the program is undertaking an effort that is fast-paced in nature and needs to be rapidly fielded, and she expressed confidence in her staff’s abilities to meet the development demands. However, various DOD officials told us that a lack of qualified software developers within DOD, and within the Space C2 program, is an issue. Agile software development methods are different from the traditional approaches used by DOD, and according to DOD officials, proficiency in Agile methods requires specific training. Software developers with this training are in high demand in the private sector, and according to DOD officials, sufficient numbers may not be immediately available for the Space C2 program. One industry best practice for software development states that to be successful, programs should ensure that each development team has immediate access to people with specialized skills including contracting, architecture, database administration, development, quality assurance, operations (if applicable), information security, risk analysis, and business systems analysis. As early as March 2009, DOD acknowledged it had a top priority to establish a cadre of trained information technology professionals, and that the lack thereof was a significant impediment to successful implementation of any future software development process. Furthermore, a 2018 Defense Science Board report highlights the lack of Agile software expertise in DOD, citing no modern software expertise in program offices or the broader acquisition workforce. Moreover, the report states that DOD defense prime contractors need to build their own internal competencies in modern software methodologies. Similarly, we found in March 2019 that DOD faces several challenges related to hiring, assigning, and retaining qualified personnel to work on space acquisition programs, similar to the challenges it faces more generally with the acquisition workforce. We also noted that DOD is taking steps to address these challenges where possible. In May 2019, the DOD’s Defense Innovation Board issued a congressionally mandated study on software acquisition and practices. The report stated that numerous past studies have recognized the deficiencies in software acquisition and practices within DOD. The report also noted the importance of digital talent and stated that DOD’s current personnel processes and culture will not allow its military and civilian software capabilities to grow fast or deep enough to meet its mission needs. In addition, the report stated that new mechanisms are needed for attracting, educating, retaining, and promoting digital talent and for supporting the workforce to follow modern practices, including developing software in close coordination with users. Finally, the report emphasized that the military services and Office of the Secretary of Defense will need to create new paths for digital talent (especially internal DOD talent) by establishing software development as a high-visibility, high-priority career track and increasing the level of understanding of modern software within the acquisition workforce. This is the case for all DOD space programs, including Space C2. DOD’s ability to command and control U.S. space assets, as well as anticipate and respond to the threats these assets face, is critical. However, over more than three decades, DOD’s efforts to improve its space command and control capabilities—commensurate with the space threats that have continued to grow in frequency and type—have been fraught with development problems. The Air Force has again undertaken a program to meet the nation’s ongoing and future consolidated command and control needs, while trying to overcome past problems with a modern software development process. The Space C2 program is making a concerted effort to learn from past software development mistakes while forging a new path for Agile development. Though DOD is taking steps to ensure that the Space C2 program has a comprehensive approach in place for managing, identifying, and mitigating challenges associated with this approach, key program plans and agency-wide guidance are still in draft form, leaving uncertainty as to how program development and oversight will ultimately proceed. Finalizing a robust acquisition strategy containing the key elements for ongoing planning and evaluation would position the program for success. Striking the right balance between trying new development methods and working within DOD’s knowledge-based framework will be essential for meeting cost, schedule, and performance goals. Periodic assessments of the program’s approach to developing software, done by independent software development experts, could not only help ensure the reviews are balanced, but would also help ensure the Space C2 program effectively addresses the challenges it faces and is situated for success. Such reviews would also help the Space C2 program to identify potential roadblocks, and ultimately, potential solutions. Effectively addressing the challenges facing the Space C2 program will help ensure that needed space command and control capabilities are no longer deferred, but actually delivered. We are making two recommendations to the Department of Defense. The Under Secretary of Defense for Acquisition and Sustainment should ensure that the Air Force’s finalized Space C2 program’s acquisition strategy includes, at a minimum, the following elements: acquisition and contracting approach; program management structure, including authorities and oversight plans for platform and infrastructure development; requirements management and development approach, and plans for prioritization; risk management plans, including how the program will identify and mitigate risks; metrics for measuring quality of software, and how those results will be shared with external stakeholders; manpower assessment identifying program workforce needs and state of expertise in Agile methods; requirements for reporting program progress to decision makers; and yearly funding levels. (Recommendation 1) The Under Secretary of Defense for Acquisition and Sustainment should ensure that the Air Force’s Space C2 program conducts periodic independent reviews to assess the program’s approach to developing software and provide, as needed, advice to the program and recommendations for improving the program’s development and progress. Participants could include, but are not limited to, officials from the Defense Innovation Board, the Defense Digital Service, the office of the Air Force Chief Software Advisor, and the Under Secretary of Defense for Acquisition and Sustainment’s Special Assistant for Software Acquisition. (Recommendation 2) We provided a draft of this product to the Department of Defense for comment. In its comments, reproduced in appendix II, DOD concurred with our recommendations. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of the Air Force, and the Under Secretary of Defense for Acquisition and Sustainment. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or ChaplainC@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. The House Armed Services Committee report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2018 contained a provision for us to review the Department of Defense’s (DOD) efforts to develop space command and control capabilities. This report (1) assesses the status of and plans for ongoing Air Force efforts to develop advanced command and control capabilities for space, and (2) identifies challenges the Air Force faces in developing these capabilities. To assess the status of and plans for ongoing Air Force efforts to develop advanced command and control capabilities for space, we analyzed Air Force Space Command and Control (C2) Program Increment Demonstration and Planning Retrospective reports for the first three increments and examined acquisition strategies for relevant programs, including acquisition strategies and addenda for Joint Space Operations center (JSPOC) Mission System (JMS) Increments 1 and 2. We also examined the Air Force’s draft acquisition strategy for Space C2 and DOD’s draft acquisition strategy for Major Agile Software Programs; reviewed Space C2 document mapping planned capabilities to the specific requirements that will be met by program deliveries; and analyzed status updates from the Space C2 program and the Combined Space Operations Center and program update briefings prepared for congressional staff by the JMS and Space C2 programs and the National Space Defense Center. In addition, we analyzed Space C2 program plans in conjunction with interim DOD guidance for Agile Software Acquisition, the Joint Chiefs of Staff Cyber Survivability Endorsement Implementation Guide, the Office of the Secretary of Defense guidance on cybersecurity operational test and evaluation procedures in acquisition programs and DOD Enterprise Development, Security and Operations (DevSecOps) processes; and examined the Principal DOD Space Advisor’s Capabilities Based Assessment which included issues relating to Space C2. We also reviewed Air Force Broad Agency Announcements and Requests for Information for Space Battle Management Command and Control and Space Situational Awareness capability development. In addition, we obtained information from 12 of the 16 companies with whom the Air Force is working to obtain their perspectives of the Air Force’s approach to developing Space C2 capabilities. To identify challenges the Air Force faces as it develops advanced command and control capabilities for space, we analyzed the JMS Critical Change Certification; examined Joint Requirements Oversight Council memoranda pertaining to the JMS critical change management and certification; reviewed the Air Force’s Space and Missile Systems Center evaluation of commercial capability gaps and capabilities; reviewed the JMS Program Manager briefing on lessons learned; and examined the DOD test and evaluation report on JMS Increment 2 (Service Pack 9). We also reviewed a selected chapter of GAO’s draft Agile Assessment Guide (Version 13), which is intended to establish a consistent framework based on best practices that can be used across the federal government for developing, implementing, managing, and evaluating agencies’ information technology investments that rely on Agile methods. To develop this guide, GAO worked closely with Agile experts in the public and private sector; some chapters of the guide are considered more mature because they have been reviewed by the expert panel. We reviewed this chapter to ensure that our expectations for how the Air Force should apply best practices for development of software capabilities for space command and control are appropriate for an Agile program and are consistent with the draft guidance that is under development, and we compared Space C2 program plans to the practices outlined in the guide. Additionally, since Agile development programs may use different terminology to describe their software development processes, the Agile terms used in this report are specific to the Space C2 program. We also compared Air Force development plans with interim and established DOD guidelines for software development, and GAO best practices for knowledge-based decision-making in weapons system development. We also reviewed prior GAO reports on the Cheyenne Mountain Upgrade, the Combatant Commanders’ Integrated Command and Control System, software acquisition, and cybersecurity. Additionally, we interviewed DOD officials from the Office of the Under Secretary of Defense for Acquisition and Sustainment; Joint Chiefs of Staff, Force Structure, Resources, and Assessment Directorate; U.S. Strategic Command; Air Force Combined Space Operations Center; Defense Advanced Research Projects Agency; Missile Defense Agency; Office of the former Principal DOD Space Advisor; Air Force Space Command; Air Force Research Laboratory; Defense Digital Service; Office of Cost Assessment and Program Evaluation; Air Force Rapid Capabilities Office; National Space Defense Center; and Air Force Space and Missile Systems Center. Finally, we interviewed officials from commercial companies that are known in the space community to have potential input into the development of space command and control capabilities to understand how the Space C2 program plans to integrate commercial capabilities into the program. We conducted this performance audit from January 2018 to October 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Rich Horiuchi, Assistant Director, Emily Bond, Claire Buck, Maricela Cherveny, Burns Eckert, Laura Hook, and Roxanna Sun made key contributions to this report. Assistance was also provided by Pamela Davidson, Kurt Gurka, Jennifer Leotta, Harold Podell, Marc Schwartz, James Tallon, Eric Winter, and Alyssa Weir.", "summary": "Since the early 1980s, the Air Force has been working to modernize and consolidate its space command and control systems into a single comprehensive platform. The past three programs to attempt this have ended up significantly behind schedule and over budget. They also left key capabilities undelivered, meeting the easier requirements first and deferring more difficult work to subsequent programs. At the same time, the need for a consolidated space command and control capability has been growing. The House Armed Services Committee report accompanying a bill for the National Defense Authorization Act for Fiscal Year 2018 contained a provision for GAO to review DOD's newest efforts to develop space command and control capabilities. This report describes the status of these efforts and identifies challenges the Air Force faces in bringing them to fruition. To conduct this work, GAO analyzed acquisition and strategy documentation, management directives, and lessons learned; and compared Air Force development plans with leading industry practices for software development, DOD guidelines, and best practices included in a draft GAO guide for assessing Agile software development programs. Given emerging and evolving threats in the space domain, as well as significant development problems in similar prior efforts, the Air Force is prioritizing the Space Command and Control (C2) program. Early prototype work on the program's software began in 2016. As of mid-2019, the program had delivered some initial capabilities; however, the capabilities delivered so far are not approved for use in operations. Because the program is still early in development, it has not yet established a time frame for certifying these capabilities for operational use. Further, the foundational elements of the program, including the infrastructure and software platform, are still being conceptualized. All Space C2 program capabilities will be significantly more automated than past development efforts and are being designed to allow operators to identify and monitor threats to U.S. space assets, identify courses of action to mitigate or eliminate those threats, communicate these actions to decision makers, and direct actions in response. To develop Space C2's technologically complex software, the Air Force is following a modernized, iterative process called Agile development—a relatively new approach for Department of Defense (DOD) programs (see figure). The Space C2 program is facing a number of challenges and unknowns, from management issues to technical complexity. Additionally, DOD officials have not yet determined what level of detail is appropriate for acquisition planning documentation for Agile software programs. They are also not certain about the best way to provide oversight of these programs but are considering using assessments by external experts. These knowledge gaps run counter to DOD and industry best practices for acquisition and put the program at risk of not meeting mission objectives. Additionally, software integration and cybersecurity challenges exist, further complicating program development. The Air Force has efforts underway to mitigate some of these challenges in the near term, but until the program develops a comprehensive acquisition strategy to more formally plan the program, it is too early to determine whether these efforts will help to ensure long-term program success. GAO is making two recommendations, including that DOD should ensure the Air Force develops a comprehensive acquisition strategy for the Space C2 program. DOD concurred with the recommendations.", "document_type": "gao"}
{"report": "Under the National Response Framework, the Department of Homeland Security is the federal department with primary responsibility for coordinating disaster response, and within the department, FEMA has lead responsibility. Due to the massive response needed after Hurricanes Irma and Maria in the U.S. Virgin Islands and Puerto Rico, FEMA utilized the National Response Framework to activate all 14 ESFs, including ESF#8. The National Response Framework designates state, local, tribal, and territorial agencies as primarily responsible for response activities in their jurisdictions, including those related to public health and medical services. However, when effective disaster response is beyond the capabilities of the state, territorial, or tribal government and affected local governments, such as was the case for Hurricanes Irma and Maria, those governments can request federal assistance. The federal response for a specific ESF is designed to supplement the state, local, tribal, or territorial resources that respond to a disaster or other emergency. However, due to the physical destruction of the two hurricanes in the U.S. Virgin Islands and Puerto Rico, the territorial government agencies that were tasked with coordinating resources to respond to such disasters were largely incapacitated. This resulted in an unprecedented federal role in the response to these disasters. As the lead agency for an ESF#8 response, ASPR is responsible for coordinating the ESF#8 response core capabilities outlined in the National Response Framework. These core capabilities include assessment of public health and medical needs, patient evacuation, patient care, the provision of medical equipment and supplies, and public health communication, among others. ASPR coordinates these core capabilities through two main roles defined in the National Response Framework—the coordinator and the primary agency. As the coordinator, ASPR oversees and coordinates the preparedness activities for ESF#8 support agencies, nongovernmental organizations, and the private sector. For example, ASPR must maintain contact with support agencies through conference calls, training, and other activities, prior to events; monitor the ESF’s progress in being able to meet the outlined core capabilities; as well as coordinate planning and preparedness efforts with nongovernmental organizations and the private sector. As the primary agency, ASPR has significant authorities, roles, resources, and capabilities to fulfill during an ESF#8 response. Its responsibilities include notifying and requesting assistance from support agencies and coordinating resources, as well as working with all types of organizations, such as ESF#8 support agencies, territory officials, and other stakeholders to maximize the use of all available resources. As part of a response, ASPR may activate the National Disaster Medical System (NDMS)—an interagency partnership among HHS, DOD, VA, and the Department of Homeland Security to supplement health and medical systems and response capabilities during a public health emergency. Under NDMS, ASPR and its partner agencies provide medical response (by deploying medical personnel teams, for example), evacuate patients, and provide medical care in NDMS medical facilities when requested by state, local, tribal, and territorial governments or other federal agencies. For example, as part of NDMS, DOD and FEMA may provide transportation to evacuate seriously ill or injured inpatients. DOD and VA may operate and staff NDMS Federal Coordinating Centers, which are activated during an emergency to receive, triage, stage, track, and transport patients affected by a disaster or national emergency to a participating NDMS medical facility capable of providing the required care to manage the patient’s condition. After an ESF#8 response, ASPR evaluates HHS’s disaster response activities through an after-action review. According to the Department of Homeland Security’s Homeland Security Exercise and Evaluation Program guidance, which ASPR follows, this review should include collecting feedback about the response activities to identify strengths and areas for improvement, and developing corrective actions to address identified areas for improvement. This information is then documented in an after-action report and corrective action improvement plan. The populations in the U.S. Virgin Islands and Puerto Rico are older than the general U.S. population. Estimates indicate that the total population in the U.S. Virgin Islands in 2018 was approximately 107,000 and about 18 percent (or about 19,000 individuals) were age 65 or older. Estimates for Puerto Rico indicate that the total population in 2018 was approximately 3.3 million and about 20 percent (or about 666,000 individuals) were age 65 or older. In comparison, almost 16 percent of the general population in the 50 states and the District of Columbia, totaling approximately 329.3 million, were age 65 or older in 2018. To serve these populations, the U.S. Virgin Islands has two hospitals, one on St. Thomas and one on St. Croix, each with a capacity of 150 beds. Puerto Rico has 68 hospitals scattered throughout the island. The capacity of beds ranges from less than 10 to 515, with a total of almost 10,000 hospital beds to serve the territory. The 2017 Atlantic Hurricane season was one of the most active seasons in U.S. history, causing widespread damage and destruction to significant populations in the continental United States and the territories. In particular, two hurricanes—Irma and Maria—struck in quick succession and devastated the U.S. Virgin Islands and Puerto Rico. Hurricane Irma – a category 5 storm passed by the U.S. Virgin Islands—St. Thomas and St. John—on September 6, and continued by Puerto Rico. In the U.S. Virgin Islands, the storm caused high storm surge, flooding, extensive damage to buildings and infrastructure, and widespread power outages. It became one of the strongest Atlantic hurricanes on record. Hurricane Maria – a category 5 storm passed by the U.S. Virgin Islands—St. Croix—on September 20, and made landfall in Puerto Rico as a category 4 storm. Hurricane Maria compounded the damage caused by Hurricane Irma in the U.S. Virgin Islands, and devastated Puerto Rico. Heavy flooding and high winds led to the catastrophic damage to Puerto Rico’s power grid, as well as severe damage to the water, communications, transportation, and health care infrastructure. The majority of Puerto Rico’s power grid was down for nearly two months following Hurricane Maria, with outages continuing through 2018. Figure 1 depicts the paths of Hurricanes Irma and Maria. Figure 2 contains photographs of damage sustained in the U.S. Virgin Islands. Figure 3 contains photographs of damage sustained in Puerto Rico. At the same time ASPR was responding to the catastrophic hurricanes in the U.S. Virgin Islands and Puerto Rico, the agency was also responding, or had recently responded, to hurricanes in other areas. Specifically, ASPR led the ESF#8 response to Hurricane Harvey, a category 4 hurricane that made landfall in Texas on August 25, 2017. Further, in addition to responding to the effects of Hurricane Irma on the U.S. Virgin Islands, ASPR was leading the response to that hurricane in Florida. Also, while ASPR was still responding to Hurricanes Irma and Maria, Hurricane Nate, a category 1 hurricane, hit Louisiana and Mississippi on October 7 and 8, 2017, respectively. While not as severe as the prior hurricanes, Hurricane Nate resulted in wind damage, flooding, and storm surge, and required a public health and medical services response. (See figure 4 for a timeline of the 2017 hurricanes requiring ASPR to lead an ESF#8 response.) ASPR and support agencies evacuated critical care and dialysis patients and deployed medical staff and temporary medical facilities as part of the response to Hurricanes Irma and Maria. These activities centered on saving lives and preventing human suffering. During the response to Hurricanes Irma and Maria, ASPR led the NDMS evacuation of critical care and dialysis patients. According to ASPR officials, Hurricane Irma damaged critical health care infrastructure and created a deteriorating situation in St. Thomas that necessitated life-saving evacuations to Puerto Rico, particularly as St. Croix’s health care facilities could not support the needs of both islands. Specifically, after Hurricane Irma damaged the only hospital on St. Thomas, ASPR prioritized evacuating critical care patients to Puerto Rico. Once ASPR officials further determined that St. Thomas did not have the capacity to treat dialysis patients, ASPR also coordinated the movement of dialysis patients to Puerto Rico. This was the first time ASPR had coordinated the evacuation of such patients during an ESF#8 response. ASPR used HHS’s Centers for Medicare and Medicaid Services’ data to locate dialysis patients on St. Thomas who were unable to be reached by local authorities for evacuation. As the threat of Hurricane Maria making landfall in Puerto Rico became evident, ASPR began moving U.S. Virgin Islands patients previously evacuated to Puerto Rico to the continental United States, according to ASPR and Department of Interior documentation. See figure 5 for a timeline of patient evacuations conducted through NDMS from the U.S. Virgin Islands and Puerto Rico after Hurricanes Irma and Maria. ASPR worked with other agencies to evacuate NDMS patients. Specifically, ASPR relied on DOD to provide transportation because HHS did not have its own transportation capabilities. For example, DOD provided personnel and transportation to conduct aeromedical evacuations of patients from the U.S. Virgin Islands to Puerto Rico and the continental United States. In addition, DOD operated a Federal Coordinating Center in the continental United States, and VA operated Federal Coordinating Centers in Puerto Rico and the continental United States to receive evacuated patients and place them into NDMS medical facilities. For example, the day after Hurricane Irma passed the U.S. Virgin Islands, ASPR requested that VA operate the San Juan Federal Coordinating Center to begin receiving evacuated U.S. Virgin Islands patients. See figure 6 for a photograph of NDMS evacuation of U.S. Virgin Islands dialysis patients to the continental United States. During the response to Hurricanes Irma and Maria, ASPR and some of its ESF#8 support agencies—DOD and VA—deployed medical staff and temporary medical facilities to respond to the public health and medical needs in the U.S. Virgin Islands and Puerto Rico. Using these medical assets, ASPR and its support agencies served almost 16,000 patients in Puerto Rico and almost 2,000 patients in the U.S. Virgin Islands over the course of about four weeks after Hurricane Maria, according to ASPR reports. Examples of ASPR medical staff and facilities include, but are not limited to, the following: Disaster Medical Assistance Teams. ASPR placed Disaster Medical Assistance Teams in front of the major hospitals in the U.S. Virgin Islands and Puerto Rico to triage patients and to relieve the hospitals’ emergency departments by treating patients with acute care needs during the response to Hurricanes Irma and Maria. Disaster Medical Assistance Teams comprise about 35 medically trained personnel and equipment. In addition, Disaster Medical Assistance Teams were sometimes divided into six-person teams—known as Health Medical Taskforce Teams—that are more agile, according to ASPR officials. These smaller teams supported response operations in the U.S. Virgin Islands and Puerto Rico by traveling into hard–to-reach places to provide acute medical care, stabilize patients, and call for the transport of patients, when needed. According to ASPR officials, ASPR deployed a Disaster Medical Assistance Team to Puerto Rico prior to Hurricane Maria making landfall and then divided it into smaller teams to provide medical care around San Juan, Puerto Rico. According to these officials, HHS was one of the few federal agencies to have operational personnel available immediately post landfall. See figure 7 for photographs of Disaster Medical Assistance Teams setting up and providing services in Puerto Rico. Federal Medical Stations. ASPR placed Federal Medical Stations in tents in front of hospitals in Puerto Rico after Hurricane Maria made landfall to assist with relieving the hospitals’ emergency departments. Federal Medical Stations are to have a 3-day supply of medical and pharmaceutical resources to sustain up to 250 stable, primary, or chronic care patients. Because the entire island of Puerto Rico was affected by Hurricane Maria, ASPR implemented a “hub and spoke” strategy for the first time—a system to deliver medical care over affected areas’ population centers—according to ASPR officials. Under this strategy, ASPR designated San Juan’s Centro Medico hospital as the “hub” of activity with six “spokes” delivering care to the island’s population centers, and placed Federal Medical Stations in tents in front of each hospital, including the “hub.” USNS Comfort Deployed to Puerto Rico to Respond to Hurricane Maria The USNS Comfort is a seagoing medical treatment facility that had more than 850 medical and support staff embarked as part of the public health and medical services response to Hurricane Maria in Puerto Rico, according to Department of Defense (DOD) officials. DOD officials stated that approximately 2,000 patients in Puerto Rico were provided care on the USNS Comfort during the course of its 45-day relief mission that began in early October 2017. The USNS Comfort’s primary mission is to provide an afloat, mobile, medical–surgical facility to the U.S. military that is flexible, capable, and uniquely adaptable to support expeditionary warfare. The ship’s secondary mission is to provide full hospital services to support U.S. disaster relief and humanitarian operations worldwide. Medical Companies provided trauma, medical, and surgical care to populations in Puerto Rico after Hurricane Maria. Among other medical facilities, DOD also provided a Combat Support Hospital to Puerto Rico 3 weeks following Hurricane Maria—which consisted of 44 beds with emergency medical technicians; an operating room, laboratory, pharmacy, and X-ray machine; and primary care and intensive care capabilities. DOD also sent the USNS Comfort—a hospital ship maintained by the U.S. Navy that served as a mobile, floating hospital—to help relieve the hospitals in Puerto Rico. VA medical staff. VA deployed medical personnel through its Disaster Emergency Medical Personnel System—VA’s main deployment program for clinical and non-clinical staff to an emergency or disaster—to assist ASPR with staffing the Federal Medical Stations. According to VA officials, these personnel worked side by side with other federal personnel, such as Disaster Medical Assistance Teams, to provide medical assistance. Our review identified several key deficiencies with ASPR’s leadership of the federal public health and medical services response to Hurricanes Irma and Maria in the U.S. Virgin Islands and Puerto Rico that could adversely affect future large-scale responses unless they are addressed. Limited ASPR presence in the U.S. Virgin Islands. As the primary agency, ASPR is responsible for coordinating the ESF#8 response, including coordinating with support agencies and officials at operations centers. Further, FEMA’s ESF#8 statement of work for ASPR states that HHS should provide appropriate personnel at emergency operations centers near disaster sites to lead an ESF#8 response. HHS officials maintained that the Department is not required to address all capabilities in the ESF#8 statement of work, as the actual response provided by HHS depends on other factors, such as resource availability. Emergency Operations Center An emergency operations center is a physical location where responders, including federal and state/territory responders, as well as nongovernmental responders, can meet to coordinate information and resources to support incident management (on-scene operations) during a response. According to Department of Homeland Security documentation, decision makers gather at emergency operations centers to ensure they receive the most current information, which allows for improved communication and decision-making during a response. During the initial weeks after the hurricanes, ASPR liaison officers were not always stationed at the emergency operations centers in St. Thomas and St. Croix. Instead, the liaisons rotated between the emergency operations center, hospital, and airport on each island to manage patient evacuations, or stayed at the hospital, according to ASPR officials. This led to confusion with regard to the ESF#8 response status on the ground, according to FEMA, DOD, and territory health officials. For example, FEMA officials stated that when they needed information on patients’ health needs and evacuation status, they had to spend time trying to locate an ASPR liaison officer to obtain it. The FEMA officials then had to relay this information to DOD, territory health officials, and hospital representatives who were making numerous requests for this information to FEMA in ASPR’s absence at the centers. FEMA officials stated that relaying medical information was outside their areas of expertise as were other activities they conducted in ASPR’s absence, such as addressing public health issues at shelters. One FEMA official stated that he had to read handwritten notes from the hospital that contained patient information, such as vitals and prescription needs, and provide this information to other responders. Without a medical background, he did not know the meaning of a lot of the medical terms used. Furthermore, these FEMA officials stated that given communication systems were down on the islands, having a reliable, physical presence at the emergency operations centers in St. Thomas and St. Croix became even more critical. After a few weeks into the response, ASPR liaison officers were stationed at emergency operations centers, according to ASPR officials, but the officers generally rotated about every 2 weeks with limited time to hand off information and were often not from Region II. This limited ASPR’s leadership of the response and put undue resource strain on other responders, according to FEMA and territory health officials. For example, according to FEMA and U.S. Virgin Islands health officials, the liaison officer would not necessarily understand the big picture, the tasks to be done, or the players involved. Thus, FEMA and territory health officials would have to take time to bring the ASPR liaison officer up to speed on the pressing public health and medical services issues, and shortly thereafter the officer would leave to be replaced by someone else, who would also need to be brought up to speed. ASPR officials provided two different reasons for the staffing challenges encountered at the emergency operations centers in the U.S. Virgin Islands. First, some ASPR officials cited personnel resource constraints. Specifically, these officials stated that ASPR personnel had already been deployed multiple times, given the prior hurricane (Hurricane Harvey) and concurrent events that ASPR was responding to in multiple locations. As a result, officials said there was not enough time to educate rotating officials on issues faced in the U.S. Virgin Islands and deployments were shorter than ideal. Second, other ASPR officials stated that a lack of transportation from Puerto Rico to the U.S. Virgin Islands may have resulted in minimal overlap of liaison officers. According to these officials, they had to request such transportation from FEMA, and FEMA did not always prioritize their needs, since it was also managing transportation needs from other ESFs. However, FEMA officials contested this statement and stated there was ample opportunity for ASPR liaison officers to get to the U.S. Virgin Islands. In retrospect, ASPR officials acknowledged that staffing emergency operations centers, as well as other strategic locations is ideal. ASPR documentation after the response states that the officers’ presence at emergency operations centers is important because they need to be working at the operational and tactical levels on the ground. In addition to staffing emergency operations centers, ASPR officials agreed with statements from FEMA and DOD officials who told us that the ideal scenario would be to have at least one other liaison officer (if not more) to support the lead liaison officer at all strategic locations. The officials noted that the number of liaison officers may vary depending on the response needs. In the case of patient evacuations, for example, this would include having a liaison officer at the airport and one at the hospital, in addition to the lead at the emergency operations center. In contrast, DOD officials stated that after Hurricane Irma, one ASPR liaison was on St. Croix trying to manage all the ESF#8 activities, including patient evacuations and hospital assessments, which was too much for one person. In May 2019, ASPR officials told us they have a long-term goal of creating an incident response team that will comprise 17 full-time response personnel. If implemented, this strategy may allow ASPR to provide more liaisons on the ground during a response and address the staffing deficiency we identified. However, ASPR officials did not provide us with a draft strategy or a timeline for the creation of such a team. Until ASPR develops a response personnel strategy to ensure it has sufficient liaison officers available to consistently lead a response from emergency operations centers and other strategic locations, the agency risks repeating the challenge encountered in the U.S. Virgin Islands—notably, a situation with inadequate liaison officer presence to effectively lead a response on the ground. Delay in tracking evacuated patients. Tracking NDMS evacuated patients and ensuring their care is a critical component of the public health and medical services response. The ESF#8 Annex of the National Response Framework states that patients should be tracked from their point of entry into NDMS. However, our review found that ASPR did not track patients evacuated through NDMS from the U.S. Virgin Islands to Puerto Rico immediately after Hurricane Irma. This occurred because of delays in getting HHS tracking personnel to the territories, according to VA documentation, as well as ASPR, DOD, VA, FEMA, and U.S. Virgin Islands Department of Health officials. Specifically, HHS teams that track patients were not deployed to the region until about 5 days after patients were already being evacuated through NDMS. These teams are (1) Joint Patient Assessment and Tracking System (JPATS) teams, which enter patient information into JPATS—ASPR’s tracking system—and (2) service access teams, which track and monitor the status of evacuated patients, including facilitating movement to home or other final destination after being discharged from care. As a result of the delayed deployment of the tracking teams, ASPR officials did not initially know the locations of some NDMS evacuated patients in Puerto Rico. For example, once in Puerto Rico, the service access teams had to drive around the territory looking for evacuees, according to ASPR officials. ASPR officials explained that there was a delay in tracking patients after Hurricane Irma because it takes time for JPATS and service access teams to deploy to a region. ASPR officials told us that they did not pre-deploy the tracking teams before the hurricane, because the U.S. Virgin Islands officials did not request ASPR’s help with patient evacuations until after Hurricane Irma hit. ASPR officials also stated that at the time of the hurricanes, the agency had no policy for tracking patients from the start of NDMS evacuations; however, since the hurricanes, the agency has developed a federal patient movement framework that may help prevent future delays in patient tracking. This framework describes the pre-deployment of JPATS and service access teams, which would allow for tracking to start at the beginning of NDMS evacuations. ASPR officials told us this is the optimal solution. However, during an event such as a hurricane, sufficient notice for pre-deployment is not always possible. One option identified in ASPR’s federal patient movement framework is for FEMA to track patients initially and share these data with ASPR and for DOD to provide patient movement manifests to ASPR so that the data can be manually entered into JPATS once deployed, which will contain the overall dataset for patient tracking. By working with DOD and FEMA, ASPR may be able to consistently track patients from the start of evacuations even when there is a deployment delay in HHS’s own tracking capabilities. While ASPR’s development of the framework is an important step forward to address delays in patient tracking, ASPR has not exercised the framework with its NDMS partners to ensure it is sufficient and reliable. For example, given the potential need to manually enter information into JPATS, there could still be a delay in HHS knowing where patients are located and being able to inform family members. An exercise of the framework could help determine if this is indeed a concern that needs to be addressed. We have previously reported that exercises are a key tool for testing and evaluating preparedness. ASPR officials told us that exercising the framework prior to the next hurricane season had been discussed, but as of May 2019, nothing had been scheduled. Without a framework that has been exercised with the other agencies involved in federal patient movement and tracking, ASPR risks delays in patient tracking when conducting future NDMS patient evacuations. Final status of one-fourth of evacuated patients not readily available. The ESF#8 Annex of the National Response Framework states that NDMS evacuated patients should be tracked to their final disposition. Further, federal internal controls standards stress the importance of information controls to ensure quality information is used to achieve objectives, which includes information that is complete and accurate. However, we found that of the approximately 800 NDMS patient evacuations during the response to Hurricanes Irma and Maria, the agency could not readily provide us with the final status of approximately 200 of these patients. ASPR officials stated they did not have information indicating the final status of the 200 evacuated patients, because case workers are not required to report this information to ASPR. ASPR officials explained that the case workers on the service access teams deployed during the response are responsible for keeping track of patients’ final status. However, we found that without conducting a review of files in which the case workers recorded patients’ final status, ASPR officials could not determine if the patients were appropriately discharged and returned back to the U.S. Virgin Islands, left the system against medical advice, or were otherwise unaccounted for. Additionally, as of June 2019, ASPR did not provide documentation indicating the steps the agency takes to ensure the data held by case workers are accurate. Until ASPR has controls in place to ensure that data on NDMS evacuated patients are complete and accurate, the agency cannot ensure it is sufficiently tracking all NDMS evacuated patients and risks losing track of patients when conducting future patient evacuation efforts. Limited focus on chronic and primary care needs in isolated locations. As the coordinator, ASPR is responsible for ensuring that appropriate planning and preparedness activities are undertaken. This includes planning for the care of elderly and chronically ill patients in isolated areas. Our review found that at the time of the hurricanes, ASPR Region II’s response plans for the U.S. Virgin Islands and Puerto Rico—known as Incident Response Plans—did not account for the need for chronic and primary care in isolated communities. This type of care was greatly needed, given that many people, especially the elderly, could not easily access hospitals, according to officials from ASPR, DOD, the Puerto Rico Department of Health, and three stakeholders we interviewed. Consistent with the views of these officials, the HHS Deputy Inspector General reported that during Hurricane Maria, hundreds of patients across Puerto Rico sought access to urgent care, primary care, and pharmacy services at community-based health care centers, known as Federally Qualified Health Centers, because they could not travel to hospitals for treatment. Further, we reported in May 2019 and heard from two stakeholders that because of the widespread power outages and infrastructure damage in both territories, the chronically ill often did not have access to electricity to power their medical devices—such as ventilators—and gasoline to run generators was scarce. ASPR’s initial response activities—which generally focused on supporting the hospitals and patients with acute care needs—were based on response plans with assumptions that did not hold true given the unprecedented level of destruction in the areas. Specifically, according to ASPR officials, the agency focused its response planning on managing the surge of patients at hospitals, assuming that individuals would make their way to hospitals, and projecting that smaller communities could care for one another until further needs assessments could be conducted. For example, ASPR Region II and Puerto Rico health officials assumed in their planning that patients in the harder to reach areas, such as the mountainous areas, would make their way to the coast where hospital care was available, according to ASPR officials. ASPR officials also stated that preparedness planning for an immediate response is generally focused on managing the surge of patients at hospitals, with the assumption that after about a week into the response, assessments would be conducted to determine other needs, such as chronic care needs. However, ASPR officials told us that in retrospect, the planning and the assumptions used for planning for the U.S. Virgin Islands and Puerto Rico were not adequate given the unprecedented level of destruction in the areas, which affected communications and transportation. FEMA officials also said that given how difficult it was to assess the situation in Puerto Rico after Hurricane Maria, having prior knowledge of the situation on the ground that could affect the response (such as the general public health and medical needs in the territories during non-disaster times) was a lesson learned that applies to them, as well as ASPR. ASPR has taken steps to better account for the need for chronic and primary care in isolated communities in future public health and medical services responses. However, these efforts have not been finalized or incorporated into ASPR Region II Incident Response Plans for the territories, which according to a lead HHS Region II official, are internal agency plans that serve as a playbook for HHS officials during an ESF#8 response in these territories. Specifically, ASPR is working with the Puerto Rico Department of Health officials to map the locations of health care facilities in Puerto Rico—such as clinics, Federally Qualified Health Centers, urgent care centers, and hospitals—including their bed, generator, communication, and surge capacities. This is the first time all such information has been brought together, and ASPR continues to work on this effort as it helps the territory recover, according to agency documentation. ASPR officials also told us that moving forward they would like to involve Federally Qualified Health Centers in planning and response activities, including involving them in the provision of primary care during responses. We agree that these are important steps that ASPR can take to address this deficiency. However, until ASPR Region II Incident Response Plans for the territories include the provision of chronic and primary care in isolated communities, there is a risk that disaster survivors will not receive needed care. For example, this could include the incorporation of Federally Qualified Health Centers or other local health clinics into these plans. Misalignment of support agencies’ capabilities to response needs. As the coordinator, ASPR is responsible for ensuring that appropriate planning and preparedness activities are undertaken, including monitoring the progress in meeting the ESF#8 core capabilities. Further, FEMA guidance issued in June 2015 states that each ESF coordinator should maintain a capabilities inventory for the ESF. However, our review found that ASPR did not have a sufficient understanding of ESF#8 support agencies’ capabilities prior to the hurricanes. Consequently, ASPR’s resource needs for the response in the U.S. Virgin Islands and Puerto Rico were not always aligned with the resources its support agencies—DOD, VA, and FEMA—could provide. According to ASPR documentation and DOD officials, this resulted in some deployed resources not being properly and efficiently utilized. As an example of the misalignment of resources, DOD officials told us that, through FEMA, ASPR requested that DOD provide stand-alone medical assistance teams (i.e., teams of medical personnel and equipment, similar to ASPR’s Disaster Medical Assistance Teams) to deliver medical care to the hurricane survivors in the U.S. Virgin Islands and Puerto Rico. However, since DOD does not have stand-alone teams, it deployed Area Support Medical Companies, which included facilities, equipment, and supply packages. These teams are equipped to serve the military population—those approximately 18-60 years of age, wounded, and requiring trauma and medical-surgical care. However, trauma and medical surgical care was not the primary need in the islands, which, in general, have an older population with chronic and primary care needs. ASPR documentation also shows that ASPR had trouble defining how FEMA and DOD assets fit into the overarching ESF#8 response. For example, ASPR documentation states that it took the agency nearly a week to fully realize that the two Area Support Medical Companies provided by DOD were not equivalent to the five stand-alone medical assistance teams that HHS had requested. According to DOD officials, the misalignment of resources during the response was troublesome as the Department’s involvement in the ESF#8 response activities affected patient care for military health beneficiaries and potentially increased overseas contingency response risks for the Department. In another example, during the response, there were conflicting expectations about VA personnel’s role in supporting the Federal Medical Stations, with VA responders thinking they would run shelter operations and ASPR believing the VA staff would support medical operations, according to ASPR documentation. According to ASPR officials, the agency had never anticipated needing— and therefore did not plan for—certain ESF#8 agency support, such as teams similar to ASPR’s Disaster Medical Assistance Teams. ASPR’s role in a response has traditionally been to support states or territories; however, because of the catastrophic nature of the hurricanes, ASPR effectively led the territories in the response as opposed to playing a supporting role. ASPR’s response system was not designed to handle that large of a role, according to officials. Since the hurricanes, ASPR has taken steps to understand the resources available from its support agencies, but ASPR officials agreed that it is an activity that the agency needs to continue to undertake. Specifically, ASPR officials stated that the agency is currently working with its NDMS partners (FEMA, DOD, and VA) to develop memorandums of agreement that outline the roles and responsibilities of each organization; however, the discussions are in the preliminary stages as ASPR continues to collaborate with each organization to understand their resource gaps and capabilities. Continuing to understand each ESF#8 support agency’s potential capabilities and its limitations—knowing that the actual capacity of these capabilities may fluctuate—is important, as evidenced by the misalignment that occurred during the response. Until ASPR can better identify the capabilities and limitations of support agencies to meet ESF#8 core capabilities, ASPR cannot, as the coordinator, determine whether the ESF is prepared for future disasters. Reliance on DOD support. As the coordinator, ASPR is responsible for ensuring that appropriate planning and preparedness activities are undertaken. This includes planning for a scenario in which DOD assistance is unavailable. We have previously reported that DOD provided much of the ESF#8 support during the initial response to Hurricanes Irma and Maria, which may not always be available in future responses. DOD’s support included providing the core capabilities of patient care (through the provision of Area Support Medical Companies, among other medical facilities) and patient evacuations (through the provision of personnel and transportation to conduct aeromedical evacuations), as mentioned above. We found that ASPR does not have a response strategy that will account for the core capabilities needed to be filled by itself or other support agencies in a large or long-term ESF#8 response if DOD were unable to assist. For example, DOD’s 2017 hurricane after-action report included reliance on DOD as a concern and recommended that HHS and FEMA establish contracts with the commercial sector to ensure the federal government has other options available for larger ESF#8 responses should DOD not have the needed capability or available capacity. Similarly, in September 2018, we reported that ESF lead agencies’ (including ASPR for ESF#8) dependence on DOD capabilities was a challenge for DOD during the response to Hurricanes Irma and Maria. We reported that the increased reliance may create vulnerability, if in the future, DOD capabilities are needed to conduct its primary mission—to defend the nation from threats—at the same time its support is needed for a domestic disaster response. ASPR told us that it does not have a contingency plan for a response in DOD’s absence, because for large-scale events, such as Hurricanes Irma and Maria, ASPR has to rely on DOD, given ASPR’s own resource constraints. ASPR officials stated that, in general, ASPR’s resource response capacity—personnel and supplies—can support a response to two simultaneous events that occur in different areas in the Continental United States for 30 days. Beyond that, ASPR has to rely on other agencies, including DOD, which occurred with Hurricanes Irma and Maria. However, ASPR officials did state that the agency has recently taken some steps to reduce its reliance on DOD. Specifically, in September 2018, ASPR entered into a contract with a private company to provide medical personnel teams similar to Disaster Medical Assistance Teams that can be utilized to supplement ASPR response personnel, especially if DOD resources are not available. Similarly, to assist with future patient evacuations, in October 2018, the agency entered into contracts with private companies for commercial air ambulance transport. In addition, ASPR officials told us that through ASPR’s participation in the Whole of Government Logistics Council, the agency has begun to further discuss air transport options during major disasters with other agencies including FEMA, DOD, and VA. However, ASPR officials also stated there is a need to hold discussions with all agencies involved in the ESFs to prioritize and coordinate air transportation during a response in the event that DOD is not available. While these are important steps to potentially minimize reliance on DOD, ASPR’s own capacity constraints make it all the more important for ASPR to a develop response strategy that includes other support agencies in the event that DOD support is unavailable. For example, such a strategy could involve conducting an exercise to simulate a large-scale ESF#8 response without DOD capabilities. Until ASPR develops a strategy demonstrating how ESF#8 core capabilities can be provided through HHS and its support agencies without DOD’s assistance, it risks being unprepared to respond to a large-scale disaster or multiple disasters if they occur when DOD’s capabilities are limited due to other events, such as military missions. ASPR completed a draft after-action report in February 2018 after several months of collecting feedback from HHS staff on the strengths and areas for improvement in the agency’s 2017 ESF#8 response activities; however, the draft is missing the perspectives of key parties involved in the response. Not collecting the perspectives of key parties involved in the response is inconsistent with federal standards for information and communication, which state that management needs access to relevant information from external parties to help achieve objectives and address related risks. Further, the Standard for Program Management states that program managers should actively engage key stakeholders throughout the life cycle of a program, which would include any evaluation activities. Specifically, when collecting feedback, ASPR did not reach out directly to support agencies, territorial governments in the U.S. Virgin Islands and Puerto Rico, or other stakeholders intimately involved in the response. Instead, ASPR gathered observations through facilitated discussions, or “hotwashes,” with HHS personnel stationed at key response sites in headquarters and the field, such as personnel stationed at the HHS Secretary’s Operations Center and those stationed at medical sites in Puerto Rico. In addition, ASPR distributed an electronic feedback link to all personnel involved in the HHS ESF#8 response, both in the field and headquarters. ASPR officials stated they did not obtain feedback directly from outside parties, such as support agencies or territorial governments, during the after-action review because the review was focused on internal aspects of the HHS response. Instead, the officials said that FEMA—as the overall lead for the federal response—typically writes the overall after-action report for the whole federal government, and those perspectives would be captured there. However, FEMA’s after-action report was focused only on its response activities for the 2017 hurricanes and did not include any strengths or areas for improvement related to ESF#8. Because ASPR did not obtain feedback from its ESF#8 support agencies and other partners, its draft after-action report dated February 2018 has key gaps in its assessment. For example, three of the deficiencies we identified based on our review of documentation and interviews with agency and territory officials—the delay in tracking evacuated patients, the final status of some evacuated patients not readily available, and the reliance on DOD support—were not included in ASPR’s draft after-action report. This indicates that key perspectives, and related lessons learned, were missing from ASPR’s after-action review. Similarly, FEMA officials said that during the course of soliciting feedback on its own response actions, FEMA’s provider of NDMS medical evacuation transportation for Hurricanes Irma and Maria said that if ASPR had reached out, it would have identified challenges with the NDMS patient evacuations conducted. In particular, the provider told FEMA that patients were evacuated to an airport in the continental United States with limited hours of availability, and if patients had to be evacuated outside of those hours, they were sent to other airports with inadequate medical care, so the patients needed to be transported again as a result. Without an after-action report that includes the perspectives of all key parties—including ESF#8 support agencies—ASPR management is likely to lack the necessary information to comprehensively identify all strengths and areas for improvement of its ESF#8 response. The catastrophic destruction encountered as a result of Hurricanes Irma and Maria proved overwhelming to the U.S. Virgin Islands and Puerto Rican governments and resulted in a large federal disaster response, complicated by losses of power, communication, transportation, and health care infrastructure in the territories. ASPR and its support agencies, such as DOD, undertook numerous actions to address the public health and medical needs in the territories—including evacuating critical care and dialysis patients from the U.S. Virgin Islands and Puerto Rico. Nevertheless, key deficiencies with ASPR’s leadership of the response resulted in confusion and resource strain among responders from support agencies and territory health departments at emergency operations centers in the U.S. Virgin Islands. The deficiencies also resulted in service access teams having to search for evacuated patients, ASPR’s inability to readily and reliably identify the final status of all evacuated patients, and disaster survivors in isolated areas potentially not receiving needed health care. ASPR’s leadership also led to an inefficient use of federal resources. Many of the deficiencies were a function of ASPR policy and its preparedness planning, and as such, they could be repeated unless ASPR addresses them. Additionally, the agency remains unprepared to respond to future large-scale disasters if DOD is unavailable. Further, the likelihood that deficiencies will recur in future responses increases, because ASPR did not include feedback from the support agencies involved in the response in its after-action report. We are making the following seven recommendations to the Assistant Secretary for Preparedness and Response: ASPR should develop a response personnel strategy to ensure, at a minimum, a lead ASPR liaison officer is consistently at the local emergency operations center(s) during an ESF#8 response and another liaison, if not more, is at strategic location(s) in the area. (Recommendation 1) As ASPR finalizes its federal patient movement framework, the agency should exercise the framework with its NDMS partners to ensure that patients evacuated through NDMS will be consistently tracked from the start of their evacuation. (Recommendation 2) ASPR should put controls in place to ensure data on all NDMS evacuated patients are complete and accurate. (Recommendation 3) ASPR Region II should revise its Incident Response Plans for the territories to include strategies for providing chronic and primary care in isolated communities. These strategies could include the incorporation of Federally Qualified Health Centers and other local health clinics as part of a response. (Recommendation 4) ASPR should work with support agencies to develop and finalize memorandums of agreement that include information on the capabilities and limitations of these agencies to meet ESF#8 core capabilities. (Recommendation 5) ASPR should develop a strategy demonstrating how it ESF#8 core capabilities can be provided through HHS and ESF#8 support agencies if DOD’s capacity to respond is limited. (Recommendation 6) ASPR should take steps to ensure the perspectives of key external parties are incorporated in the development of HHS’s after-action reviews, following future ESF#8 activations. (Recommendation 7) We provided a draft of this report for advance review and comment to HHS, DOD, the Department of Homeland Security, VA, and the governments of the U.S. Virgin Islands and Puerto Rico. HHS and VA provided written comments, which we have reprinted in appendixes I and II, respectively. HHS concurred with five of our seven recommendations and stated that it had, or was in the process of, taking action. While we made no recommendations to VA, in its comments VA stated that it looks forward with working with HHS on matters we have presented in this report. HHS and DOD provided technical comments, which we incorporated as appropriate. U.S. Virgin Islands and Puerto Rican government officials stated they had no comments on the draft report. HHS did not concur with a recommendation in the draft report directing ASPR to develop and finalize ESF#8 response plans for the territories that include strategies for providing chronic and primary care in isolated communities. In its comments, HHS stated that while ASPR has federal plans in place that guide federal response, each state and locality is responsible for developing its own individual plans. We modified the language in our report and our recommendation to clarify we are referring to ASPR Region II’s Incident Response Plans for the U.S. Virgin Islands and Puerto Rico. According to a lead ASPR Region II official, these plans are internal agency plans that serve as a playbook for HHS officials during an ESF#8 response in these territories. However, as we reported, these plans do not account for the provision of chronic and primary care in isolated communities. Accordingly, we believe our recommendation is warranted. HHS also did not concur with a recommendation in the draft report that ASPR work with support agencies to develop an inventory to identify the capabilities and limitations of support agencies to meet ESF#8 core capabilities. According to HHS, such an inventory will be out of date immediately after development due to world events and changes in investments, technologies, and priorities. Instead, HHS proposed the continued use of interagency liaison officers at the HHS emergency operations center, as they can provide real-time updates on available resources during a response. We agree that HHS should continue this practice in future responses. However, as is evidenced by the misalignment that we identify in our report, this action was not adequate during the response to Hurricanes Irma and Maria in the U.S. Virgin Islands and Puerto Rico. Further, as we reported, ASPR officials acknowledged that more needs to be done to better understand the resources available from its support agencies. To clarify the intent of our recommendation—that is, that ASPR take steps to ensure it has a sufficient understanding of each ESF#8 support agency’s potential capabilities and its limitations—we modified language in our report and the recommendation. Specifically, we modified our recommendation to direct ASPR to include information on the capabilities of these agencies as it works to develop and finalize memorandums of agreement with support agencies. The memorandums of agreement that ASPR is beginning to draft with support agencies provide an opportunity to begin to address this issue. As we have reported, taking such action is needed to help ensure that future ESF#8 responses are more efficiently and effectively coordinated. We are sending copies of this report to the appropriate congressional committees, the Secretaries of the Health and Human Services, Defense, Homeland Security, Veterans Affairs, and Interior, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or DeniganMacauleyM@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Mary Denigan-Macauley, (202) 512-7114 or DeniganMacauleyM@gao.gov. In addition to the contact named above, Kelly DeMots (Assistant Director), Deirdre Gleeson Brown (Analyst-in-Charge), Kenisha Cantrell, Justin Cubilo, and Rebecca Hendrickson made key contributions to this report. Also contributing were Sam Amrhein, Kaitlin Farquharson, and Vikki Porter. Disaster Response: FEMA and the American Red Cross Need to Ensure Key Mass Care Organizations are Included in Coordination and Planning. GAO-19-526. Washington, D.C.: September 19, 2019. Disaster Response: Federal Assistance and Selected States and Territory Efforts to Identify Deaths from 2017 Hurricanes. GAO-19-486. Washington, D.C.: September 13, 2019. Disaster Assistance: FEMA Action Needed to Better Support Individuals Who Are Older or Have Disabilities. GAO-19-318. Washington, D.C.: May 14, 2019. 2017 Disaster Contracting: Actions Needed to Improve the Use of Post- Disaster Contracts to Support Response and Recovery. GAO-19-281. Washington, D.C.: April 24, 2019. 2017 Hurricane Season: Federal Support for Electricity Grid Restoration in the U.S. Virgin Islands and Puerto Rico. GAO-19-296. Washington, D.C.: April 18, 2019. Disaster Recovery: Better Monitoring of Block Grant Funds is Needed. GAO-19-232. Washington, D.C.: March 25, 2019. Puerto Rico Hurricanes: Status of FEMA Funding, Oversight, and Recovery Challenges. GAO-19-256. Washington, D.C.: March 14, 2019. U.S. Virgin Islands Recovery: Status of FEMA Public Assistance Funding and Implementation. GAO-19-253. Washington, D.C.: February 25, 2019. 2017 Disaster Contracting: Action Needed to Better Ensure More Effective Use and Management of Advance Contracts. GAO-19-93. Washington, D.C.: December 6, 2018. Homeland Security Grant Program: Additional Actions Could Further Enhance FEMA’s Risk-Based Grant Assessment Model. GAO-18-354. Washington, D.C.: September 6, 2018. 2017 Hurricanes and Wildfires: Initial Observations on the Federal Response and Key Recovery Challenges. GAO-18-472. Washington, D.C.: September 4, 2018. Federal Disaster Assistance: Individual Assistance Requests Often Granted but FEMA Could Better Document Factors Considered. GAO-18-366. Washington, D.C.: May 31, 2018. 2017 Disaster Contracting: Observations on Federal Contracting for Response and Recovery Efforts. GAO-18-335. Washington, D.C.: February 28, 2018. Disaster Assistance: Opportunities to Enhance Implementation of the Redesigned Public Assistance Grant Program. GAO-18-30. Washington, D.C.: November 8, 2017.", "summary": "Hurricanes Irma and Maria hit the U.S. Virgin Islands and Puerto Rico within two weeks of each other in September 2017, causing catastrophic damage. HHS is responsible for leading the federal public health and medical services response during a disaster, such as these hurricanes. As part of its lead federal role during these hurricanes, HHS called upon support agencies, including the Departments of Defense, Homeland Security, and Veterans Affairs, to assist with the public health and medical services response. GAO was asked to review the federal public health and medical services response to Hurricanes Irma and Maria in the U.S. Virgin Islands and Puerto Rico. This report examines HHS's actions and leadership of this response, among other things. GAO reviewed documentation on the preparedness for, and response to, the hurricanes. It also interviewed federal and territory officials and interviewed or received written responses from eight nonfederal stakeholders involved in the response, such as nongovernmental organizations. GAO identified these stakeholders through research and referrals. The catastrophic destruction encountered as a result of Hurricanes Irma and Maria proved overwhelming to the U.S. Virgin Islands and Puerto Rican governments and resulted in a large federal disaster response, complicated by losses of power, communication, and health care infrastructure. The Department of Health and Human Services (HHS) led the federal public health and medical services response and undertook numerous actions to address the needs in the territories—including evacuating critical care and dialysis patients from the U.S. Virgin Islands and Puerto Rico and providing medical personnel and facilities. However, GAO identified several shortcomings in HHS's leadership. While the scale, location, and timing of these storms complicated response efforts, the deficiencies GAO identified were in many cases a function of preparedness policies, or lack thereof. As a result, they could adversely affect future large-scale responses unless addressed. For example, as the lead agency, HHS is responsible for ensuring that appropriate planning activities are undertaken, including monitoring the federal ability to provide core public health and medical services response capabilities. However, GAO found that HHS did not have a full understanding of the capabilities and limitations of its support agencies, including the Departments of Defense, Homeland Security, and Veterans Affairs. Consequently, HHS's needs were not always aligned with the resources that its support agencies could provide, resulting in some deployed resources not being properly and efficiently utilized. For example, HHS requested Department of Defense medical teams, but these teams specialized in trauma and surgical care, not the chronic and primary care needed. HHS lacked plans for the territories that accounted for the chronic and primary care needs in isolated communities. This care was greatly needed, given that many, especially the elderly, could not easily access hospitals. GAO is making seven recommendations, including that HHS develop agreements with support agencies that include response capability and limitation information, and develop response plans for providing care in isolated communities. HHS disagreed with two of the seven citing, among other things, territory responsibility for plans. GAO clarified the intent of the two recommendations and believes that all seven are warranted.", "document_type": "gao"}
{"report": "FAA, within the U.S. Department of Transportation, provides air traffic services for the continental United States (domestic airspace) and over parts of the Atlantic, Pacific, and Arctic oceans (oceanic airspace). More than 24 million square miles of oceanic airspace are under U.S. control. This airspace is divided into flight information regions (flight regions): Anchorage Arctic, Anchorage Oceanic, New York Oceanic, and Oakland Oceanic. Air traffic service (ATS) route is a specified route designed to channel the flow of traffic as necessary for the provision of air traffic services. ATS routes are defined by predetermined geographical positions— waypoints. For example, ATS route G344 is published by FAA and is defined by waypoints. Organized Track System is a series of ATS routes. For example, A590, R591, and G344, along with other ATS routes, comprise the North Pacific Route System. In areas with high flight volume, such as between California and Hawaii, FAA publishes air traffic service (ATS) routes that allow air traffic controllers to handle large volumes of traffic. A set of ATS routes—an organized track system—functions as a freeway in the sky, with routes serving as lanes (see sidebar). ATS routes may be “fixed” or “flexible.” A fixed route does not change; whereas a flexible route changes daily depending on weather patterns, such as prevailing winds. As detailed in industry reports, multiple factors—including weather conditions, congestion, and airspace restrictions—affect whether aircraft operators plan to fly on ATS routes published by FAA or on routes they determine to be the most efficient for that flight (i.e., user-preferred routes). Figure 1 shows U.S. oceanic airspace and the location of various organized track systems. To fly through U.S. oceanic airspace, aircraft operators (e.g., airlines) file a flight plan, which includes the departure and arrival airports and the planned route (i.e., the path the aircraft plans to take to get to its destination). Air traffic control may clear the flight plan as filed—with no changes—and/or makes changes to an aircraft’s planned route during the flight. To manage air traffic, air traffic controllers must be able to monitor an aircraft’s position as it flies along its planned route. As we have previously reported, in domestic airspace, radar and ground-based Automatic Dependent Surveillance-Broadcast (ADS-B) technology provides this surveillance information. Radar is a ground-based system that provides information on an aircraft’s position to air traffic control facilities. Ground- based ADS-B uses equipment installed in aircraft (transmitters) to broadcast an aircraft’s position, altitude, and other information to ground stations, which transmit the data to air traffic control facilities. Surveillance information from radar and ADS-B is nearly instantaneous— allowing domestic air traffic controllers to effectively “see” where an aircraft is at all times. FAA manages radar and ground-based ADS-B infrastructure, in some cases through contracts. Through its contract with the provider of ADS-B services, FAA also pays for the cost of transmitting ADS-B messages from aircraft to air traffic control in domestic airspace. Future Air Navigation System (FANS) Equipage in U.S. Oceanic Airspace By 2020, FAA estimates that about 80 percent of aircraft flying in U.S. airspace above the Atlantic Ocean will be equipped with FANS as will 84 percent of aircraft flying in U.S. airspace above the Pacific Ocean. However, FANS equipage varies within these airspaces. In the New York flight region, specifically along the West Atlantic Route System, FAA estimates that by 2020 the FANS equipage rate will be 66 percent—lower than other sectors of Atlantic airspace. Similarly, in the Oakland flight region, along the Central East Pacific Route System, FAA estimates that by 2020 the FANS equipage rate will be 75 percent—lower than found in other sectors of Pacific airspace. controllers receive reports on an aircraft’s position from a radio operator who receives verbal updates from pilots using a high frequency radio or automatically through a technology called Future Air Navigation System (FANS): High frequency radio allows pilots to speak with a third-party radio operator and share surveillance information via spoken position reports at mandatory reporting points. The radio operator then relays position reports as a data message to air traffic controllers. FANS includes a communication system—Controller Pilot Data Link Communications (CPDLC)—and a surveillance system—Automatic Dependent Surveillance-Contract (ADS-C). CPDLC allows pilots and air traffic controllers to communicate directly by exchanging text- based messages. Through ADS-C, air traffic control can request position reports and specify their frequency as well as the information they should include. As we have previously reported, position reports sent through ADS-C can transmit at defined time intervals, when specific events occur such as pilot deviation from the planned route, or at the request of air traffic control. ADS-C reports sent at a defined time interval are called periodic reports—in U.S. oceanic airspace these are typically sent every 10 to 14 minutes. As detailed in an industry report, aircraft operators pay to use the satellite communication networks required to transmit communication and surveillance information to air traffic control in oceanic airspace. In addition, aircraft operators are responsible for the cost of equipping their aircraft with communication, navigation, and surveillance equipment. To help them manage oceanic airspace, U.S. air traffic controllers use a computer system called Advanced Technologies and Oceanic Procedures (ATOP). ATOP is a flight data processing system that controllers use at their workstations. It provides oceanic air traffic controllers with several automated tools to assist in maintaining aircraft at safe distances from one another, coordinate with air traffic controllers in other flight regions, and facilitate controller-pilot communication through CPDLC, among other things. ATOP incorporates information from aircraft flight plans and position reports allowing controllers to monitor an aircraft’s progress, ensure it is following the route cleared by air traffic control, and to continually check for any potential conflicts between aircraft flying through their area of control, i.e., aircraft that could get too close to one another. Separation standards—the minimum distances required between aircraft—help ensure that aircraft do not collide with one another. As illustrated in figure 2, separation standards dictate the minimum required longitudinal, lateral, and vertical distance between aircraft. The International Civil Aviation Organization (ICAO) publishes minimum separation standards for oceanic airspace. Using ICAO separation standards as the minimum, FAA sets the separation standards and aircraft requirements that are used in U.S. oceanic airspace. Currently, the minimum distance that must be maintained between aircraft in U.S. oceanic airspace is 30 nautical miles lateral and 30 nautical miles longitudinal. To be eligible for this U.S. oceanic minimum separation standard, an aircraft must be equipped with FANS, in addition to meeting other communication, navigation, and surveillance requirements. For aircraft without FANS, the minimum distance required between aircraft is larger, at least 50 nautical miles lateral and approximately 80 nautical miles longitudinal. While requiring more distance between aircraft helps ensure safety, it means less airspace capacity and may result in fewer direct and fuel- efficient routes. To maintain the required separation distance between aircraft, air traffic control may instruct an aircraft—either before or during flight—to fly at an altitude or along a route that is not the most efficient for that aircraft in terms of flight time or fuel usage. For example, aircraft spaced 50 nautical miles apart laterally and longitudinally are less likely to be able to fly at a fuel-efficient altitude (e.g., 38,000 feet) as fewer aircraft will fly at that altitude, especially in congested airspace. In contrast, when aircraft are spaced 30 nautical miles apart laterally and longitudinally, more aircraft can fly at fuel-efficient altitudes. FAA may adopt ICAO’s minimum separation standards for the oceanic airspace it manages or it can adopt standards that require aircraft to fly farther apart than ICAO’s minimum standards. For example, ICAO published the minimum separation standard for 30 nautical miles longitudinal in 2002. FAA began applying these minimum separation standards in the Oakland Oceanic flight region in 2007, in the Anchorage Oceanic flight region in 2012, and in the New York Oceanic flight region in 2013. In 2016, ICAO published a new minimum separation standard, which allows a minimum lateral distance of 23 nautical miles. FAA has not yet adopted the 23 nautical mile lateral standard. Since 2012, ICAO has worked to develop new minimum separation standards for oceanic airspace that require even less distance between properly equipped aircraft. These new minimum separation standards are based on improved surveillance capabilities, with aircraft using space-based ADS-B potentially eligible to use one set of reduced minimum separation standards (19 nautical miles lateral and 17 nautical miles longitudinal) and aircraft using enhanced ADS-C potentially eligible to use a different set of minimum separation standards (23 nautical miles lateral and 20 nautical miles longitudinal). These new minimum separation standards are undergoing review with final approval expected in 2020. FAA’s Advanced Surveillance Enhanced Procedural Separation (ASEPS) program, which is part of FAA’s Air Traffic Organization, was tasked with examining how to increase the efficiency and capacity of operations in U.S. oceanic airspace using enhanced surveillance technologies. In fiscal years 2015 through 2018, congressional committees directed FAA to accelerate its evaluation of space-based ADS-B and provided funding for that purpose. In response, the ASEPS program, among other things, evaluated and compared the costs and benefits of two technologies that could improve surveillance capabilities in U.S. oceanic airspace— enhanced ADS-C and space-based ADS-B. Following are descriptions of how these enhanced surveillance technologies work: Enhanced ADS-C. Uses the same ADS-C technology already installed on FANS-equipped aircraft, but ATOP would request that automatic position reports be sent more frequently to air traffic control. Aircraft equipped with ADS-C and transmitting position reports every 3.2 minutes would be eligible for ICAO’s proposed minimum separation standard of 20 nautical miles longitudinal. ICAO’s 23 nautical miles lateral separation standard, published in 2016, does not require more frequent ADS-C position reports. Space-based ADS-B. Uses low-earth orbiting satellites to capture automatic reports broadcast by ADS-B transmitters installed on aircraft, which will be required for aircraft flying at certain altitudes in domestic U.S. airspace by 2020. ADS-B messages are to be received by air traffic control about every 8 seconds. Aircraft equipped with ADS-B transmitters using the space-based ADS-B system and also equipped with required communication and navigation technologies, would meet the eligibility requirements for ICAO’s proposed minimum separation standards of 19 nautical miles lateral and 17 nautical miles longitudinal. As shown in figure 3, enhanced ADS-C and space-based ADS-B use similar transmission networks but relay different information at different time intervals to air traffic control. To compare these options, FAA prepared a business case analysis that estimated the costs to the agency and aircraft operators, identified safety benefits from enhanced surveillance, and identified and calculated the value of operational efficiency benefits from using reduced minimum separation standards enabled by enhanced ADS-C and space-based ADS-B. For more detail on the costs and benefits included in FAA’s business case analysis, see appendix IV. FAA used this business case analysis to inform its decision on which enhanced surveillance technology to use to support new minimum separation standards. FAA is implementing new minimum separation standards supported by enhanced ADS-C in U.S. oceanic airspace. FAA does not plan to use space-based ADS-B in U.S. oceanic airspace; instead, the agency intends to study how to use space-based ADS-B in other U.S. airspace over the next 5 years. According to FAA, this approach is driven by its analysis of the costs and benefits of each enhanced surveillance technology and the safety and operational challenges of using space- based ADS-B in U.S. oceanic airspace. According to FAA officials and based on project status reports, FAA is implementing new minimum separation standards in U.S. oceanic airspace that are supported by enhanced ADS-C. The agency plans to apply these standards in all sectors of U.S. oceanic airspace by 2022, as shown in figure 4. Specifically, FAA will begin operational use of the 23 nautical mile lateral separation standard in U.S. oceanic airspace in 2021 and the 20 nautical mile longitudinal separation standard in 2022. In April 2019, FAA executives approved a schedule and funding for the implementation of these new minimum separation standards (i.e., 23 nautical miles lateral and 20 nautical miles longitudinal) in U.S. oceanic airspace using enhanced ADS-C. To implement these new standards, FAA officials are upgrading ATOP and working through a review process required to change minimum separation standards in U.S. oceanic airspace. This review process involves 18 milestones, including safety assessments, coordinating with industry and international participants, and developing procedures and training materials for pilots and air traffic controllers. According to FAA officials, the costs and benefits of pursuing this approach—using enhanced ADS-C to support the adoption of new minimum separation standards, i.e., 23 nautical miles lateral and 20 nautical miles longitudinal—drove this decision. Specifically, FAA found that the benefits to airspace users of using enhanced ADS-C to enable new minimum separation standards, such as improved access to fuel- efficient altitudes, outweighed, by 2 to 1, the total costs, including FAA’s costs to upgrade ATOP and the aircraft operators’ data costs due to more ADS-C position reports. In addition, FAA officials said that although new minimum separation standards can provide benefits to airspace users overall, the current minimum separation standards support safe operations for current and anticipated levels of air traffic in U.S. oceanic airspace. Officials noted that the benefits to airspace users of new minimum standards are contingent on the communication, navigation, and surveillance capabilities of aircraft in an airspace and the frequency of disruptive weather patterns. According to FAA officials and air traffic controllers we spoke with, the current minimum separation standards (i.e., 30 nautical miles lateral and longitudinal) are rarely used as the density of aircraft traffic in U.S. oceanic airspace does not require such close spacing. In areas of U.S. oceanic airspace with higher traffic volumes, such as along the West Atlantic Route System and the Central East Pacific Route System, the number of aircraft without FANS and the frequency of disruptive weather patterns often prevent air traffic controllers from applying current minimum separation standards. Officials noted that they are also implementing the new minimum separation standards to harmonize with adjacent air navigation service providers. FAA’s ability to implement these new minimum separation standards (i.e., 23 nautical miles lateral and 20 nautical miles longitudinal) in their documented time frames depends on the success of planned ATOP upgrades. For example, FAA officials and air traffic controllers we spoke to told us that there is a current limitation in ATOP that under certain circumstances, air traffic controllers cannot rely on the system to ensure that minimum longitudinal separation distances are maintained. As a result, air traffic controllers cannot grant aircraft flying at the current minimum longitudinal separation distance their requests to deviate from their planned route for reasons such as avoiding disruptive weather or turbulence. Representatives of the union that represents FAA air traffic controllers told us this limitation must be resolved before new separation standards (i.e., 23 nautical miles lateral and 20 nautical miles longitudinal) can be safely applied. FAA officials told us that they have developed an ATOP software upgrade that could resolve this issue; the upgrade is scheduled to occur in 2021. However, if this upgrade does not resolve the issue or it takes longer to resolve than planned, implementation of the new minimum separation standards could be delayed. According to FAA officials, the cost of space-based ADS-B was a major factor in their decision not to use this technology in U.S. oceanic airspace. FAA’s initial business case analysis found that the costs of using space- based ADS-B to enable reduced separation outweighed the benefits. Specifically, the estimated subscription costs to access the data collected by space-based ADS-B and needed upgrades to ATOP outweighed the estimated benefits to airspace users by 6 to 1. As mentioned above, according to FAA officials, current minimum separation standards allow safe operations for current and anticipated levels of air traffic in U.S. oceanic airspace. Therefore, without a positive business case (i.e., benefits are larger than the costs), FAA officials decided they could not pursue this enhanced surveillance option for U.S. oceanic airspace. FAA officials we interviewed also had safety concerns about using space- based ADS-B to manage reduced separation in U.S. oceanic airspace at this time. Specifically, FAA officials told us the operational considerations for most of the U.S. oceanic airspace were not reflected in the data used by ICAO to model the safety of these standards—air traffic control response times and rates of approved and unapproved aircraft weather deviations. For example, the ICAO panel responsible for analyzing the safety of the proposed minimum separation standards enabled by space- based ADS-B used data from the North Atlantic on the number of times aircraft deviate without authorization from their expected flight plan due to weather conditions. According to FAA officials, other oceanic regions— especially in U.S. oceanic airspace—experience a higher frequency of these deviations. As a result, FAA officials do not plan to use the new minimum separation standards enabled by space-based ADS-B (i.e., 19 nautical miles lateral and 17 nautical miles longitudinal) until FAA can further address how to implement these standards in U.S. oceanic airspace. FAA officials we interviewed also had operational concerns about using space-based ADS-B with ATOP to manage separation between aircraft in U.S. oceanic airspace. Specifically, FAA officials told us that ATOP is designed to use information in ADS-C position reports—i.e., an aircraft’s current location, the next waypoint the aircraft will pass and at what time, and the subsequent waypoint the aircraft will pass—to determine potential conflicts in aircraft flight paths. Without this information, ATOP would not receive the data it uses to detect conflicts within the next 2 hours of a flight, according to FAA officials. ADS-B messages do not include this information and therefore, space-based ADS-B would not replace ADS-C in U.S. oceanic airspace. Due to these cost, safety, and operational concerns with using space- based ADS-B to enable reduced separation, the ASEPS program deferred a decision, originally scheduled for September 2018, on whether to invest in using space-based ADS-B in U.S. oceanic airspace. FAA officials said that while they have not yet found a positive business case for using space-based ADS-B in U.S. oceanic airspace, they will further study space-based ADS-B in U.S. offshore and oceanic airspace. According to FAA officials, they expect further study to identify additional benefits and resolve operational challenges to using space-based ADS-B. FAA officials and documents indicate that the agency has near-term, medium-term, and long-term plans with goals, milestones, and time frames to evaluate how to use space-based ADS-B in U.S. airspace over the next 5 or more years. These plans include an operational evaluation and other studies to assess the uses and benefits of space-based ADS-B in U.S. airspace. FAA officials told us they expect to use findings from the near-term operational evaluation to inform medium-term and long-term plans. According to FAA officials and documentation, the ASEPS program intends to conduct an operational evaluation of space-based ADS-B in U.S. offshore airspace managed by controllers based in Miami, as shown in figure 5. FAA officials told us that this operational evaluation will assess space- based ADS-B with the computer system used by domestic air traffic controllers—the En Route Automation Modernization (ERAM) system. The operational evaluation will also focus on how to use space-based ADS-B in the heavily travelled airspace between the U.S. East Coast and islands in the Caribbean and assess potential benefits. As detailed by FAA officials, a radar that is located on Grand Turk Island provides critical data to U.S. air traffic controllers and enables the use of domestic separation standards of 5 nautical miles in this airspace. When this radar is out of service, which happens on a regular basis, aircraft traversing the airspace between Florida and Puerto Rico must be spaced using oceanic separation standards (e.g., separation distances of 30 nautical miles or greater). According to an industry report and FAA officials, this situation leads to re-routes and delays, which negatively affect airline operations. Using space-based ADS-B as a back-up surveillance system would ensure that even when the Grand Turk radar fails, U.S. air traffic control can continue to manage air traffic using domestic separation standards. In 2021, once the operational evaluation is complete, the ASEPS program expects to make recommendations to FAA executives on how to use space-based ADS-B in the Miami oceanic flight region, in addition to other areas. FAA officials also said that this evaluation will allow the agency to test space-based ADS-B in an operational environment and that the findings can inform its medium-term and long-term plans for using space- based ADS-B. The use of space-based ADS-B in this airspace could also result in more direct routes between the U.S. East Coast and islands in the Caribbean. According to FAA officials and documentation, the ASEPS program expects to study additional potential benefits of space-based ADS-B over the next 3 to 5 years. These medium-term initiatives are expected to: Analyze the use of space-based ADS-B for contingency operations in U.S. airspace. This study would define where space- based ADS-B can be used to provide surveillance capabilities when ground-based infrastructure (e.g., radar) is unavailable, such as after a hurricane. As part of this plan, the ASEPS program would also identify upgrades that would be needed for air traffic control computer systems to support using space-based ADS-B. Analyze operational challenges in U.S. oceanic airspace and potential solutions. This study of U.S. oceanic airspace would include a data-driven analysis of the use and constraints on the use of user-preferred routes by aircraft in U.S. oceanic airspace. In addition to providing information on potential inefficiencies in oceanic airspace operations, the analysis will cover how to mitigate potential safety hazards related to the use of space-based ADS-B in the oceanic environment and the requirements for future upgrades to ATOP to support the use of space-based ADS-B. According to FAA officials, both medium-term initiatives would result in recommendations for consideration by FAA executives in 2021. Using space-based ADS-B for contingency operations could lead to updated air traffic control procedures and computer upgrades; however, this would depend on the results of the analysis and the approval of FAA executives. The analysis of user-preferred routes in oceanic airspace could lead to recommendations on how to optimize route systems and how to use space-based ADS-B to support the use of user-preferred routes. According to FAA officials and documentation, using space-based ADS-B to enable the use of new minimum separation standards in U.S. oceanic airspace will be reviewed and evaluated over the next 5 or more years. This long-term initiative will use information learned through the near-term and medium-term plans. As part of this initiative, the ASEPS program intends to investigate options for enhanced communication technologies and encourage industry development of these technologies. As with the medium-term initiatives, the ASEPS program expects to make recommendations to FAA executives on how to proceed with this plan in 2021. Based on the results of this initiative, program officials told us they could start preparing for an investment decision on using space-based ADS-B in oceanic airspace to enable the use of new minimum separation standards in 2025 or later. Most (11 of 14) of the selected airlines we interviewed and surveyed support FAA’s approach to enhance surveillance capabilities in U.S. oceanic airspace by pursuing enhanced ADS-C and adopting new minimum oceanic separation standards of 23 nautical miles lateral and 20 nautical miles longitudinal in the near term. Most (12 of 14) also support continuing to evaluate how to use space-based ADS-B in oceanic airspace. Of those selected airlines that did not support FAA’s approach, the reasons included concern that using enhanced surveillance technologies will increase operator costs with no clear benefits and that FAA is prioritizing enhanced ADS-C over space-based ADS-B despite the safety and technological advances the latter would enable. While most selected airlines (12 of 14) were satisfied or very satisfied with how FAA manages the safety of U.S. oceanic airspace, most noted the need to improve operational efficiency in this airspace. Specifically, many selected airlines (10 of 14) reported experiencing operational inefficiencies, including not being able to fly at fuel-efficient altitudes. Many of these airlines (9 of 10) view adopting new minimum separation standards as a way to address these inefficiencies. Other aviation stakeholders, including the unions representing FAA air traffic controllers and commercial airline pilots, also see the need to enhance surveillance and adopt new minimum separation standards to ensure that U.S. oceanic airspace remains efficient as international air traffic grows. Selected airlines identified several benefits they would expect to see from the implementation of new minimum oceanic separation standards, including improved access to fuel-efficient altitudes, redesigned organized track systems, and improved access to user-preferred routes. Most selected airlines (12 of 14) we surveyed view improved access to fuel-efficient altitudes as a benefit of reduced separation standards. Aircraft flying in controlled airspace cannot change altitudes (e.g., move from 36,000 feet to 38,000 feet) without air traffic control approval. With reduced minimum separation standards, air traffic control could grant more altitude change requests, allowing aircraft to more consistently fly at fuel-efficient altitudes. For example, representatives from one airline told us that an aircraft’s ability to climb and descend as needed provides both safety and operational benefits. Other airline representatives also told us that the ability to fly at fuel-efficient altitudes results in savings on fuel costs. Many selected airlines (9 of 14) think FAA should make changes to organized track systems once new minimum separation standards are adopted. These changes include reducing lateral separation between routes or removing the systems entirely to enable aircraft to fly user- preferred routes all the time. Reduce lateral separation between the routes in organized track systems. Currently, all organized track systems in U.S. oceanic airspace have routes spaced at least 50 nautical miles apart laterally. Several selected airlines (3) told us that they would expect FAA to take advantage of new reduced minimum separation standards by spacing routes more closely together. For example, representatives from one airline suggested spacing the routes in the West Atlantic Route System 30 nautical miles apart laterally—thus increasing the number of routes from 10 to 19 and significantly increasing airspace capacity. In a report prepared by the NextGen Advisory Committee’s Enhanced Surveillance Task Group at the request of FAA, there was also support for taking advantage of new minimum separation standards enabled by enhanced surveillance to reduce the lateral separation between routes in the Central East Pacific Route System. Remove all organized track systems. Several selected airlines (5 of 14) also viewed the adoption of new minimum separation standards as a step toward the removal of all organized track systems. Removing all organized track system routes would, by definition, mean aircraft operators could fly user-preferred routes optimized according to their preferences, such as fuel use and flight time. Air navigation service providers in Canada and the United Kingdom, which are responsible for managing the North Atlantic Organized Track System, told us that the use of space-based ADS-B and the proposed separation standards it supports (i.e., 19 nautical miles lateral and 17 nautical miles longitudinal), may lead to the end of published ATS routes for the North Atlantic Organized Track System. Many selected airlines indicated that current separation standards inhibit their ability to fly user-preferred routes (10 of 14) as well as their ability to fly the most efficient user-preferred routes (11 of 14). Many selected airlines (9 of 14) view more access to user-preferred routes or the ability to fly more efficient user-preferred routes as an expected benefit of new minimum separation standards. Several selected airlines (3 of 14) also told us that they no longer request to fly user-preferred routes in the airspace covered by the Central East Pacific Route System or along the West Atlantic Route System because these requests are denied or they are re-routed during the flight. Selected airlines also noted the importance of understanding the costs, benefits, and timelines associated with the implementation of enhanced surveillance technologies in making their own investment decisions. Specifically, most selected airlines (11 of 14) told us that their decision to use an enhanced surveillance technology is contingent upon how much it will cost them to implement the technology—which can involve equipping aircraft and potentially paying subscription costs for the service— compared to the benefits airlines receive from the technology. For example, representatives from one airline told us that they are interested in the benefits of space-based ADS-B and enhanced ADS-C, but before paying for new or additional surveillance services, they would need to have evidence that the benefits of these services would outweigh the costs. Specifically, the representatives would like to know to what extent enhanced surveillance, if at all, would result in the actual use of new minimum separation standards and the likelihood they would be able to fly the flight plan they filed. With this information, the airline representatives said the airline could determine whether they could realize cost savings or additional revenue, such as through adding flights to their schedules. Representatives from another airline told us they would like to know what FAA’s plan is for enhancing surveillance and enabling new minimum separation standards and to have assurance that FAA will stick to this plan. According to FAA officials and documents, the agency’s approach addresses some of the efficiency benefits expected by airspace users. Improved access to fuel-efficient altitudes. FAA officials and air traffic controllers we spoke to expect the adoption of new minimum separation standards to offer efficiency benefits to airspace users through more consistent access to fuel-efficient altitudes. In a business case analysis, FAA estimated that this benefit would result in over $280 million in cost-savings for aircraft operators. According to air traffic controllers we spoke to, with new minimum separation standards they would be able to more frequently grant aircraft requests to access these altitudes. Redesign of organized track systems. When considering changes to organized track systems, FAA officials said they must balance benefits to airspace users with workload demands that would be placed on air traffic controllers. FAA officials told us they are currently redesigning the North Pacific Route System to take advantage of the 23 nautical mile lateral separation standard by reducing the lateral separation between tracks. According to FAA officials, this redesign, which is planned to be complete by 2021, could offer benefits to aircraft operators flying between Japan and Alaska, such as allowing air traffic to move more efficiently and with fewer restrictions on user- preferred routes. FAA officials told us that redesigning the North Pacific Route System is possible because of high FANS-equipage rates (over 95 percent) and the absence of disruptive weather patterns. However, according to FAA officials, they do not plan any changes to other organized track systems, such as the Central East Pacific Route System and the West Atlantic Route System, at this time because of aircraft equipage rates and weather patterns. In such areas, moving the routes closer together would prevent air traffic controllers from approving aircraft requests to deviate due to bad weather. Access to user-preferred routes. FAA officials differ with selected airline representatives on whether reduced separation standards would lead to increased access to user-preferred routes. According to FAA officials and documents, improved access to user-preferred routes requires an increase in aircraft equipped with FANS, not changes to the airspace. FAA officials also said that airlines can fly user-preferred routes in the Central East Pacific Route System and the West Atlantic Route System but also acknowledged that air traffic controllers often cannot grant access to user-preferred routes in these airspaces because of the volume of air traffic or disruptive weather patterns. Given the differing perspectives and limited data on user- preferred routes, in April 2019, FAA decided to engage a third-party research company to study the use of and access to user-preferred routes in U.S. oceanic airspace, to be completed in late 2021. Based on this study, FAA may investigate changes to U.S. airspace to address problems identified. FAA identified venues to share and coordinate their enhanced surveillance plans, timelines, and expectations with aviation industry stakeholders. As previously noted, FAA’s process for implementing changes to separation standards requires the agency to coordinate with and brief domestic and international aviation industry stakeholders. FAA officials also pointed to other venues where they plan to share information on these plans with airlines, including formal and informal working groups. Given the relatively early stages of the implementation of the 23 nautical mile lateral and 20 nautical mile longitudinal separation standards enabled by enhanced ADS-C, FAA has not yet completed this industry outreach. The agency plans to coordinate with the aviation industry on the implementation of these separation standards by January 2021. Several selected airlines and other aviation stakeholders—representing pilots, commercial airlines, business aircraft operators, and general aviation—noted the importance of FAA taking advantage of technology advancements and benefits that space-based ADS-B can offer. For example, several (5) selected airlines view space-based ADS-B as a major advancement in oceanic surveillance. Representatives from one airline told us that FAA risks losing its position as a global leader if it does not move forward with space-based ADS-B and the reduced separation standards it enables. According to FAA officials, the agency is a leading air navigation service provider as demonstrated by its use of advanced computer systems to apply minimum separation standards when possible, its role in developing ICAO’s new minimum separation standards, and its plans to move forward with space-based ADS-B in a manner that best fits U.S. oceanic airspace needs. FAA officials also pointed to other air navigation service providers, such as the Japan Civil Aviation Bureau, that are not currently planning to use space-based ADS-B. Several selected airlines and other aviation stakeholders representing commercial and business airlines expressed concern that by not adopting enhanced surveillance and the minimum separation standards it enables, aircraft transitioning into and out of U.S. oceanic airspace could experience delays. Representatives of the Canadian and United Kingdom air navigation service providers, which began using space-based ADS-B and the new minimum separation standards it enables in 2019, told us that different separation standards between their oceanic airspace and U.S. oceanic airspace could lead to delays for aircraft as air traffic increases. Specifically, as air traffic grows and air traffic controllers apply separation distances closer to the minimum standards, those flight regions with lower minimum standards will have to space out aircraft crossing into flight regions with higher minimum separation standards prior to an aircraft crossing a flight region boundary. This situation could lead to delays crossing flight region boundaries and less access to efficient routes across oceanic airspace. FAA views other factors, such as the low volume of air traffic in some airspaces, the frequency of disruptive weather patterns, and the relatively low percentage of aircraft equipped with FANS in high volume airspaces, to contribute more to the operational efficiency of the oceanic airspace than the use of minimum standards. As previously noted, according to FAA officials and air traffic controllers, the current minimum separation standards for U.S. oceanic airspace (30 nautical miles lateral and longitudinal) are rarely used because of these factors. In addition, FAA officials told us that the difference between the separation standards FAA plans to adopt in U.S. oceanic airspace with enhanced ADS-C (23 nautical miles lateral and 20 nautical miles longitudinal) and the separation standards enabled by space-based ADS-B (19 nautical miles lateral and 17 nautical miles longitudinal) is unlikely to result in delays even as air traffic increases. Other air navigation service providers in the Atlantic and Pacific Oceans are still assessing the costs and benefits of space-based ADS-B. For example, the Portuguese air navigation service provider told us they are still considering whether to use space-based ADS-B. In the Pacific Ocean, the Japanese air navigation service provider has not decided whether to use space-based ADS-B and therefore will not be adopting the minimum separation standards (19 nautical miles lateral and 17 nautical miles longitudinal) enabled by this technology. While the Japanese plan to adopt the 23 nautical mile lateral separation standard supported by enhanced ADS-C, they do not plan to adopt the 20 nautical mile longitudinal separation standard at this time. We provided a draft of this report to the Department of Transportation (DOT) for review and comment. DOT responded by email and provided technical clarifications, which we incorporated into the report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Transportation, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2834 or krauseh@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. This report examines (1) the Federal Aviation Administration’s (FAA) approach to enhancing surveillance capabilities to improve safety and efficiency in U.S. oceanic airspace and (2) selected aviation stakeholders’ perspectives on FAA’s approach to enhancing surveillance. To address both of our objectives, we reviewed FAA and other aviation stakeholders’ documents on the management and organization of U.S. oceanic airspace; the functionality and use of communication, navigation, and surveillance equipment in aircraft flying in U.S. oceanic airspace; and descriptions of the enhanced surveillance technologies that were being considered by FAA—space-based Automatic Dependent Surveillance- Broadcast (ADS-B) and enhanced Automatic Dependent Surveillance- Contract (ADS-C). Specifically, to understand how U.S. air traffic controllers manage oceanic airspace and the procedures aircraft operators must follow, we reviewed FAA Advisory Circulars on Oceanic and Remote Continental Airspace Operations (91-70B) and Data Link Communications (90-117) and FAA Order JO 7110.65X: Air Traffic Control. We also reviewed a NextGen Advisory Committee report, Enhanced Surveillance Capabilities in FAA Controlled Oceanic Airspace: Operational Need and Added Benefits, that was prepared at the request of FAA on this topic, to understand the industry perspective on the need for enhanced surveillance in U.S. oceanic airspace and the costs and benefits of using space-based ADS-B. To understand how space-based ADS-B and enhanced ADS-C would function, we interviewed representatives from Aireon, which offers the space-based ADS-B service, and Inmarsat, which provides the primary satellite communication network used by the providers of ADS-C services. We also interviewed other aviation industry stakeholders, including trade associations representing aircraft operators and unions representing pilots, including Airlines for America, International Air Transport Association, National Air Carrier Association, National Business Aviation Association, Aircraft Owners and Pilots Association, Coalition of Airline Pilots Associations, and Air Line Pilots Association. These organizations were selected based on several factors: their inclusion in prior GAO reports, their role in the aviation industry, and recommendations from other industry stakeholders or FAA. To examine FAA’s approach to enhancing surveillance capabilities in U.S. oceanic airspace, we reviewed FAA documents and interviewed FAA officials. The documents we reviewed included those related to FAA’s plans to modernize management of oceanic airspace, specifically The Future of the National Airspace System (June 2016) and National Airspace System Capital Investment Plan FY2018-2022. We also reviewed FAA’s policy guidance on acquisitions and investment documents related to the Advanced Surveillance Enhanced Procedural Separation (ASEPS) program’s planned investment decision on enhanced surveillance. These internal FAA documents included the ASEPS Concept of Operations, the Initial and Final Business Case Analyses, the Final Investment Decision Benefits Basis of Estimate, and a Safety Risk Management Assessment of space-based ADS-B and enhanced ADS-C. In reviewing the business case analysis, we did not independently evaluate the methodology or data sources used. We interviewed FAA officials and program managers that are working on different elements of FAA’s efforts to enhance surveillance in U.S. oceanic airspace. Within the Air Traffic Organization, we interviewed officials from several offices, including the ASEPS program, which managed the evaluation of surveillance technologies; the Oceanic/Offshore Standards and Procedures Branch, which oversees air traffic operations in oceanic airspace such as facilitating changes to air traffic procedures and systems to enable the use of new technologies and new standards; and the Advanced Technologies and Oceanic Procedures Program Office, which oversees changes to the air traffic control computer system used to manage oceanic air traffic. We also interviewed FAA officials with the Flight Standards Service, which works to improve flight operations, standardization, and aviation safety across U.S. and international airspace systems. In addition, we interviewed the contractor who prepared FAA’s business case analyses. We interviewed FAA air traffic controllers at the Anchorage, New York, and Oakland air route traffic control centers, which are responsible for managing the flight information regions that comprise U.S. oceanic airspace. In addition, we conducted site visits to the New York and Oakland air route traffic control centers, where we observed air traffic controllers providing oceanic air traffic services. We also interviewed representatives from the National Air Traffic Controllers Association, which is the union representing FAA air traffic controllers. We also interviewed or received written responses from representatives of the air navigation service providers for oceanic airspace adjacent or close to U.S. oceanic airspace—Canada, Japan, Portugal, and the United Kingdom—to understand their plans to enhance surveillance capabilities. To obtain selected aviation stakeholders’ perspectives on FAA’s approach to enhancing surveillance in U.S. oceanic airspace, we selected 10 U.S. and foreign commercial airlines using FAA data from fiscal year 2016 on the annual number of flights by airline in U.S. oceanic flight information regions–Anchorage Arctic and Oceanic, Oakland Oceanic, and New York Oceanic. Specifically, we selected the five airlines in each U.S. oceanic flight information region with the most annual flights. Some airlines were in the top five in more than one flight information region. All 10 airlines selected using this method were passenger airlines. We selected an additional passenger airline because it planned to begin service in U.S. oceanic airspace. We selected three large cargo airlines, based on tons of cargo transported, to ensure that the cargo airlines’ perspective was represented. Of the 14 airlines we selected, we conducted semi-structured interviews with or received written responses to our questions from 13. To obtain additional information from airline operators, we conducted a follow-up survey of the 14 selected airlines. The survey included questions on perceptions of the safety of FAA’s management of U.S. oceanic airspace, operational inefficiencies experienced by airlines in U.S. oceanic airspace, effect of current separation standards on airlines’ use of user-preferred routes, airlines’ expectations of the benefits of reduced separation standards, and airlines’ support for FAA’s planned approach to enhance surveillance in oceanic airspace. We developed the survey based on our objectives and included topics not covered in our initial interviews. We pre-tested our survey with representatives of three of the 14 selected airlines. We conducted the survey between December 2018 and January 2019, and all 14 selected airlines completed the survey. For the complete list of airlines we interviewed and/or surveyed, see table 1. In this report, we use the following conventions in reference to information obtained from the 14 selected airlines: “several” is three to seven, “many” is eight to 10, and “most” is 11 to 13. We conducted this performance audit from March 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The International Civil Aviation Organization (ICAO) publishes minimum separation standards and related eligibility requirements for oceanic airspace. Air navigation service providers, such as the Federal Aviation Administration (FAA), may adopt these standards or apply standards that are more conservative (e.g., require greater distances between aircraft). Table 1 lists selected ICAO current and proposed minimum separation standards for oceanic airspace that rely on either Automatic Dependent Surveillance-Contract (ADS-C) or space-based Automatic Dependent Surveillance-Broadcast (ADS-B). The lateral and longitudinal separation standards commonly applied by U.S. air traffic controllers in U.S. oceanic airspace—the Anchorage Arctic, Anchorage Oceanic, New York Oceanic, and Oakland Oceanic flight regions—are shown in table 2. Aircraft meeting these communication, navigation, and surveillance equipment and performance requirements are eligible for the separation standards detailed above. However, the actual standards applied by U.S. air traffic controllers depend on several factors, including the number of similarly eligible aircraft and air traffic volume. For example, while an aircraft may be eligible to use the 30 nautical mile lateral separation standard, nearby aircraft may not. When aircraft with differing communication, navigation, and/or surveillance capabilities are flying near one another, air traffic controllers must apply the larger separation standard based on the aircraft with the fewest capabilities. Air traffic controllers consider not just an aircraft’s current location but also where it is going when applying separation standards. Therefore, as aircraft approach the boundaries of U.S. oceanic airspace, U.S. air traffic controllers also consider the separation standards and eligibility requirements of the neighboring flight region. Based on our interviews, U.S. air traffic controllers hand off aircraft to their foreign counterparts (and vice versa) so that aircraft enter a new flight region in conformance with that flight region’s standards. For example, air traffic controllers managing aircraft in the Anchorage Oceanic flight region do not typically space aircraft heading towards Russian oceanic airspace (the Magadan Oceanic Flight Information Region) at the minimum separation—even if they are eligible. According to these air traffic controllers, any benefits that aircraft would gain from flying at the minimum separation distance in U.S. airspace would be lost when entering Russian airspace, where the separation standards are 10 minutes longitudinal (approximately 80 nautical miles). Therefore, aircraft must be spaced at least 10 minutes apart longitudinally upon entering Russian airspace. As shown in tables 1 and 2 above, FAA uses the 30 nautical mile longitudinal standard but does not use the 23 nautical mile lateral standard. According to interviews with FAA officials and FAA documentation, FAA plans to adopt and start using the 23 nautical mile lateral standard in U.S. oceanic airspace in 2021 and the 20 nautical mile longitudinal standard in this airspace in 2022. According to FAA officials, the agency does not plan to adopt the other ICAO proposed minimum standards (i.e., 19 or 15 nautical miles lateral and 17 or 14 nautical miles longitudinal) that depend on the use of space-based ADS-B at this time. The Federal Aviation Administration (FAA) Acquisition Management System (AMS) policy outlines a process for evaluating potential investments. This process includes the following milestones: 1. definition of the concept and requirements of a program; 2. investment analysis readiness decision; 3. initial investment decision (business case analysis to determine the 4. final investment decision (final business case and implementation 5. solution implementation (program implementation). FAA’s corporate-level acquisition decision-making body—the Joint Resources Council (JRC) —approves or disapproves at each AMS milestone. If the JRC approves the final investment decision, this commits FAA to funding the program segment and moving forward with the investment plan. From January 2014 to April 2019, FAA’s Advanced Surveillance Enhanced Procedural Separation (ASEPS) program—tasked with evaluating and comparing the costs and benefits of enhanced Automatic Dependent Surveillance–Contract (ADS-C) and space-based Automatic Dependent Surveillance–Broadcast (ADS-B)—progressed through the following steps in the AMS process to prepare for a final investment decision on enhancing surveillance and enabling new minimum separation standards in U.S. oceanic airspace. January 2014 (investment analysis readiness decision). JRC approved FAA to begin further analysis of options, including enhanced ADS-C and space-based ADS-B to support the adoption of reduced separation standards in U.S. oceanic airspace. As part of this analysis, FAA took the following actions. July 2015. JRC recommended that the ASEPS program continue evaluating the space-based ADS-B option to accommodate user (i.e., airline) preference. July 2016. FAA tasked the NextGen Advisory Committee with evaluating (1) the need and benefit of enhanced surveillance capabilities, including associated costs, funding mechanisms and funding models and (2) evaluate the business case, including insight regarding several operational factors impacting potential benefits from an investment. FAA requested input from the NextGen Advisory Committee to better understand industry’s assessment of (1) the quantified benefit that industry expects the investment will deliver and (2) how much industry would be willing to pay if it was responsible for the investment. However, according to FAA officials, the report did not address the quantified benefit industry expects the investment will deliver, determine how much industry would be willing to pay if it was responsible for the investment, or conduct an overall assessment of whether the investment is cost beneficial to industry. The report cited not having sufficient information, such as expected benefits and costs, to conduct an analysis of how much industry would be willing to invest. October 2017 (initial investment decision). ASEPS Program presented the initial business case analysis comparing the two enhanced surveillance options, enhanced ADS-C and space-based ADS-B, to the JRC. Given the negative return on investing in space- based ADS-B, the JRC directed the ASEPS program to evaluate the costs and benefits of space-based ADS-B within sub-sectors of U.S. oceanic flight regions, such as Oakland flight region north and New York east. March 2018. JRC directed the ASEPS Program to proceed with both enhanced surveillance options—enhanced ADS-C and space-based ADS-B—to a final investment decision, which was planned for September 2018. June 2018. ASEPS Program proposed a strategic shift, which involved delaying the final investment decision on enhanced ADS- C and deferring a final investment decision on space-based ADS- B to allow additional testing on how to use space-based ADS-B in oceanic and domestic airspace. Drivers of this shift in approach included the results of the business case analysis. September 2018 (strategy decision). JRC approved the ASEPS program’s strategic shift. The ASEPS Program asked the JRC to approve their plan to delay a final investment decision on enhanced ADS-C and to defer a final investment decision on space-based ADS-B. The JRC approved the ASEPS program’s proposal to merge the ASEPS enhanced ADS-C investment with a planned final investment decision on upgrades to the Advanced Technology and Oceanic Procedures (ATOP) system. The JRC also approved the ASEPS program’s proposal to continue studying space-based ADS-B through an operational evaluation in U.S. offshore airspace and longer-term studies concerning using space-based ADS-B for contingency operations and future use in U.S. oceanic airspace. April 2019 (final investment decision). JRC approved a final investment decision on the ASEPS Program’s plan to use enhanced ADS-C to enable new minimum separation standards in U.S. oceanic airspace. The ATOP program management office asked the JRC to approve investments in large-scale ATOP enhancements that include system changes that will enable the implementation of new minimum separation standards (i.e., 23 nautical miles lateral and 20 nautical miles longitudinal) with the use of enhanced ADS-C. As part of its acquisition process (outlined in app. III), the Federal Aviation Administration (FAA) contracted with a third-party to prepare a business case analysis for the Advanced Surveillance Enhanced Procedural Separation (ASEPS) program. This analysis estimated the costs to the agency and aircraft operators, identified safety benefits from enhanced surveillance, and identified and calculated the value of efficiency benefits from applying new minimum separation standards enabled by two technologies: enhanced Automatic Dependent Surveillance-Contract (ADS-C) and space-based Automatic Dependent Surveillance-Broadcast (ADS-B). The analysis described below was developed for FAA’s initial and final investment decision on the program: ASEPS Initial Business Case (August 2017). This business case analysis compared the costs and benefits of space-based ADS-B and enhanced ADS-C to a baseline scenario. ASEPS Final Business Case (August 2018). This business case analysis compared the costs and benefits of enhanced ADS-C to a baseline scenario. No final business case analysis was prepared for space-based ADS-B since FAA deferred a final investment decision on the use of space-based ADS-B. This appendix discusses the costs and benefits that were included in these business case analyses based on our review of FAA’s business case documentation and interviews with FAA officials. In the initial business case, a baseline scenario and two alternative scenarios were used to evaluate the costs and benefits of using enhanced ADS-C and space-based ADS-B as compared to not using these enhanced surveillance options: baseline with no change in current minimum separation standards of 30 nautical miles lateral and 30 nautical miles longitudinal, use enhanced ADS-C with minimum separation standards of 23 nautical miles lateral and 23 nautical miles longitudinal, and use space-based ADS-B with minimum separation standards of 15 nautical miles lateral and 15 nautical miles longitudinal. In the final business case analysis, only a baseline scenario and the enhanced ADS-C scenario were included. In the business case analysis, costs and benefits were modelled between 2020 and 2040 in the Atlantic and Pacific Oceans. To model these scenarios, researchers used projections on flight demand and aircraft equipage with the technology required to use these enhanced surveillance services: Future Air Navigation System (FANS) or ADS-B and FANS. In order to use enhanced ADS-C and space-based ADS-B to enable new minimum separation standards, FAA and airspace users will need to make certain investments. Based on our review of FAA’s business case documentation, we found that certain costs were factored into the business case analysis, including: upgrades to the Advanced Technologies and Oceanic Procedures additional ADS-C message traffic, and subscription fee for space-based ADS-B service The final business case analysis focused on enhanced ADS-C and included only those costs to FAA and users related to use of this service. The business case analysis focused on the costs of these enhanced surveillance services and did not include the cost of equipping aircraft with FANS and/or ADS-B equipment, which are required to use these enhanced surveillance technologies. According to FAA officials, these costs were not included because aircraft operators are equipping their aircraft for other reasons. Specifically, FAA regulations requiring ADS-B equipment for aircraft flying through U.S. domestic airspace by 2020 means most aircraft flying in U.S. oceanic airspace will be ADS-B equipped. In addition, mandates from other air navigation service providers requiring FANS will compel most aircraft crossing into non-U.S. oceanic airspace to equip with FANS. The business case considered the costs FAA would incur using the data from these enhanced surveillance technologies, including upgrades to ATOP software. The business case analysis also considered the costs airspace users would face in using these enhanced surveillance technologies. In the business case analysis, FAA assumed that aircraft operators would continue to pay for ADS-C services. Since enhanced ADS-C would involve more messages per flight hour than currently sent via ADS-C, FAA estimated that aircraft operators would see an increase in messaging costs per flight hour, according to our review of FAA documentation. FAA also made assumptions about how much a subscription fee for space-based ADS-B will cost. As a new service that FAA has not yet contracted for, the actual cost of space-based ADS-B subscription fees are not known. However, initial estimates of the cost per flight hour for space-based ADS-B are much greater than the estimated cost per flight hour of additional ADS-C messages, according to FAA. FAA’s business case analysis considered safety benefits and efficiency benefits. As detailed in the analysis, the size of these benefits depends on the participation of aircraft in each enhanced surveillance service (i.e., enhanced ADS-C and space-based ADS-B). The benefits presented in the business case represent the maximum benefit pool. Specifically, the analysis assumes that all properly equipped aircraft will use space-based ADS-B or enhanced ADS-C services. The business case analysis discussed safety benefits offered by improved surveillance, such as increased air traffic controller situational awareness and improved detection and resolution of aircraft on conflicting flight paths. According to oceanic air traffic controllers we interviewed at the three air route traffic control centers responsible for U.S. oceanic airspace, enhancing surveillance capabilities offers safety benefits, such as improved situational awareness and search and rescue capabilities. Enhanced ADS-C and space-based ADS-B both offer these safety benefits. However, space-based ADS-B also provides information to air traffic controllers to reduce the risk of a vertical collision between aircraft. This safety benefit was monetized by FAA. Enhanced surveillance can enable a reduction in the minimum required distance applied between aircraft, with potential efficiency benefits for airspace users. The three efficiency benefits included in FAA’s business case analysis that were monetized are: Improved accommodation of altitude requests. According to FAA’s analysis, a primary benefit of reduced separation standards is that aircraft will be more likely to fly at a fuel-efficient altitude. In oceanic airspace, aircraft must make a request to air traffic control to change their altitude. Despite the immensity of oceanic airspace, there is competition for the most fuel-efficient altitudes at certain times of day. For example, according to oceanic air traffic controllers in Oakland, the majority of the air traffic they handle is flights between Hawaii and the U.S. west coast, with most aircraft departing at the same time. Air traffic controllers we spoke with agreed that with enhanced surveillance and reduced separation standards, they should be able to grant more altitude requests and allow more aircraft to fly at optimal altitudes. Reduced need for aircraft to carry extra fuel. According to FAA’s analysis, aircraft operators typically carry more fuel on an aircraft than needed to fly their planned route. Aircraft carry extra fuel to hedge against the possibility that its actual flight path will be less fuel-efficient than its planned flight path. The cost of carrying extra fuel (i.e., the cost to carry) comes from the added weight of carrying extra fuel, weight that causes an aircraft to use more fuel and that reduces an aircraft’s ability to carry revenue-generating cargo. This benefit flows from the improved accommodation of altitude requests, discussed above. More efficient arrivals and departures at Pacific island airports. According to FAA’s analysis, some Pacific island airports do not have radar surveillance and require U.S. oceanic air traffic controllers in the Oakland air route traffic control center to manage aircraft arrivals and departures. As a result, oceanic separation standards are applied as aircraft arrive and depart these islands’ airports. FAA’s analysis shows that reducing oceanic separation minimums will allow air traffic controllers to allow more frequent arrivals and departures from these airports. According to this analysis, the benefit of more frequent arrivals and departures is measured in terms of the costs to aircraft operators (an aircraft’s direct operating costs) and the cost to passengers (a passenger’s value of time). FAA’s business case analysis also includes efficiency benefits of reduced separation that were not monetized, including emissions savings and improved air traffic control accommodation of aircraft requests for descents, routing changes, and speed changes. FAA policy does not currently allow programs to value carbon dioxide emissions avoided for investment decisions. Another efficiency benefit of reduced separation— giving air traffic controllers more flexibility to grant deviations from planned flight paths due to disruptive weather—was quantified and monetized, but not factored into the benefit calculation. To implement new separation standards in U.S. oceanic airspace, the Federal Aviation Administration (FAA) has a set of 18 critical milestones that it follows: 1. Determine the operational need. 2. Evaluate the benefits. 3. Establish an operational concept. 4. Assess the impact on air traffic control. 5. Conduct a safety assessment and record it with the appropriate safety risk management documentation. 6. Determine requirements. 7. Conduct a feasibility and economic analysis. 8. Establish requirements for aircraft and operator approval. 9. Conduct rulemaking. 10. Coordinate with industry and international participants. 11. Coordinate with air traffic control representatives and pilot groups. 12. Complete regional documentation. 13. Acquire approval for aircraft and operators. 14. Develop pilot and air traffic control procedures. 15. Design pilot and air traffic control training materials. 16. Confirm that the system works. 17. Employ the separation standard. 18. Monitor the performance of the system in accordance with safety risk management practices. In addition to the contact named above, Jonathan Carver (Assistant Director), Sarah Arnett (Analyst-in-Charge), Amy Abramowitz, Melissa Bodeau, Samuel Gaffigan; David Hooper, Richard Hung, Amanda Miller, Malika Rice, and Pamela Vines made key contributions to this report.", "summary": "Recent developments in surveillance technologies, which provide an aircraft's location to air traffic controllers, have the potential to improve air traffic operations over the oceans. FAA has explored how to improve surveillance capabilities in U.S. oceanic airspace to take advantage of new international separation standards that could lead to the more efficient use of this airspace. GAO was asked to review planned improvements to aircraft surveillance. This report examines: (1) FAA's approach to enhancing surveillance capabilities to improve safety and efficiency in U.S. oceanic airspace and (2) selected aviation stakeholders' perspectives on FAA's approach. GAO reviewed documents related to FAA's planned investment in enhanced oceanic surveillance and interviewed FAA officials working on this effort. Interviews included those with the Air Traffic Organization and air traffic controllers who manage U.S. oceanic airspace. GAO surveyed representatives of 14 commercial airlines, including 11 U.S. and foreign passenger airlines, which were selected based on factors such as flight volume; and 3 U.S. cargo airlines, which were selected based on tons of cargo shipped. GAO also interviewed other aviation stakeholders, including trade associations, unions representing pilots, and foreign air navigation service providers that manage airspace adjacent to U.S. oceanic airspace. The Federal Aviation Administration (FAA) evaluated two aircraft surveillance technologies that would allow aircraft to safely fly in closer proximity while in oceanic airspace. Based on its evaluation, FAA committed to using one in the near term and to continue to study another for future use. Specifically, in April 2019, FAA committed to implement by 2022 new international standards that allow reduced distances between aircraft, called minimum separation standards. These reduced distances would be enabled by a surveillance technology known as enhanced Automatic Dependent Surveillance-Contract (ADS-C). FAA also decided to continue studying the use of another enhanced surveillance technology known as space-based Automatic Dependent Surveillance-Broadcast (ADS-B)—to further improve surveillance in U.S. airspace. Both technologies offer increased frequency in reporting of an aircraft's location, which enhances safety, and can support new minimum separation standards. FAA decided to proceed with enhanced ADS-C in the near term because the efficiency benefits to airspace users exceeded the costs of more frequent location reporting and air traffic control system upgrades by 2 to 1. In contrast, FAA determined that the costs of using space-based ADS-B in U.S. oceanic airspace outweigh the efficiency benefits by 6 to 1. FAA officials added that operational challenges to using space-based ADS-B to manage air traffic in U.S. oceanic airspace have not yet been resolved. FAA plans to continue studying potential uses for space-based ADS-B in U.S. airspace to determine if benefits can outweigh the costs (see figure). GAO found that most selected airlines (11 of 14) support FAA's overall approach to enhance oceanic surveillance. Selected airlines also said they expect the new minimum separation standards to improve access to more direct and fuel-efficient routes. FAA is taking steps to provide these benefits by restructuring routes in one area of U.S. oceanic airspace and by applying new minimum standards to give aircraft better access to fuel-efficient altitudes. According to FAA officials, however, additional benefits, such as redesigning other U.S. oceanic airspace, expected by selected airlines are limited by (1) relatively low rates of aircraft equipage with the technology that enables reduced separation and (2) the frequency of disruptive weather patterns in parts of U.S. oceanic airspace.", "document_type": "gao"}
{"report": "Ensuring that there is adequate time to complete acquisition planning activities and identifying the contracting workforce required to execute mission needs can help agencies establish a strong foundation for successful acquisition outcomes. However, our prior work identified challenges FEMA faced in its acquisition and workforce planning efforts for disaster contracting. The Federal Acquisition Regulation (FAR) requires agencies to perform acquisition planning activities for all acquisitions to ensure that the government meets its needs in the most effective, economical, and timely manner possible. In our December 2018 report, we found that FEMA had guidance in place establishing timeframes for certain FEMA acquisitions following the completion of the acquisition package. Further, FEMA implemented an acquisition tracking tool in 2016—the 5-Year Master Acquisition Planning Schedule (MAPS)— which monitors the status of and provides acquisition planning timeframes for certain high value and mission-critical acquisitions, including advance contracts, regardless of dollar value. However, we found that FEMA had not established timeframes or released guidance for the pre-solicitation phase of the acquisition planning process, when program officials identify a need and develop key acquisition package documents (see figure 1). Not adhering to suggested timeframes can place a burden on contracting officers and increase the likelihood of not awarding a contract on schedule. This, in turn, may create a need for FEMA to non-competitively extend the existing contract—this extension may be considered a bridge contract. Given the lack of a government-wide definition, we defined bridge contracts in our prior work as: extensions to an existing contract beyond its period of performance (including base and options) and new, short-term contracts awarded on a sole-source basis to an incumbent contractor to avoid a lapse in service caused by a delay in awarding a follow-on contract. FEMA officials acknowledged that the use of non- competitive bridge contracts is not an ideal practice as they cannot ensure the government is paying what it should for goods and services. However, in December 2018, we found that FEMA used bridge contracts for at least 10 of its advance contracts used in response to the 2017 disasters—with some of these contracts lasting for several years. To decrease dependence on bridge contracts, FEMA established MAPS to help track and monitor the status of acquisition planning timeframes for certain acquisitions. However, most of the program office and contracting officials we spoke with during our December 2018 review had limited familiarity with the tool. In our December 2018 report, we recommended that FEMA update and implement existing guidance to identify acquisition planning timeframes and considerations across the entire acquisition planning process and clearly communicate the purpose and use of its acquisition planning tool to relevant personnel. DHS concurred, but in its response to our report stated it believed existing outreach and training on MAPS had resolved these challenges. We acknowledged FEMA’s training in our report, but noted that not all relevant staff we spoke with were familiar with MAPS, and that there was no formal guidance on the timeframes for the entirety of the acquisition planning process. Given these issues, we continue to believe FEMA needs to take additional steps to implement our recommendation. Without planning and guidance on its use of advance contracts, FEMA lacks reasonable assurance that it is maximizing their use to the extent practicable and cost-effective to quickly provide goods and services following a disaster. PKEMRA requires the FEMA Administrator to develop a contracting strategy that maximizes the use of advance contracts to the extent practical and cost effective, and FEMA contracting officials told us that advance contracts should be used before awarding new contracts. However, in December 2018, we found that FEMA’s advance contract strategy and guidance did not clearly identify the objectives of advance contracts or whether and how they should be prioritized for use in relation to new post-disaster contracts. For example, we reported that FEMA’s lack of an updated strategy and guidance contributed to confusion and challenges with the use of advance contracts for tarps, used to cover small areas of roof damage. Although FEMA had awarded advance contracts to provide tarps, a subsequent modification to these contracts limited the ability to use them for immediate disaster response needs—one of FEMA’s stated purposes. Furthermore, we found that FEMA awarded vendors new post-disaster contracts for tarps before using its existing advance contracts. According to FEMA officials at that time, neither of the post-disaster contract vendors was able to provide the required tarps when needed. We concluded that the timing and use of the existing tarp advance contracts raised questions about the ability of contracting officers to use these contracts to provide tarps immediately following disasters. Additionally, we concluded that an updated advance contracting strategy could have enabled FEMA to more quickly provide the needed tarps to survivors, considering the additional time and staff resources needed to award new post-disaster contracts. In our December 2018 report, we recommended that FEMA update its strategy to clearly define the objectives of advance contracts, how they contribute to FEMA’s disaster response operations, and whether and how they should be prioritized in relation to making new, post-disaster contract awards. We also recommended FEMA update its guidance accordingly. DHS concurred with these two recommendations and identified actions it plans to take to address them. Our prior work also showed that FEMA’s ability to adequately plan for and manage its disaster contracts is further complicated by persistent acquisition workforce challenges, including attrition and staffing shortages. In April 2019, we found that FEMA had identified workforce shortages as a continuing challenge for disaster response and recovery. But FEMA had not assessed its contracting workforce—including regional contracting workforce needs—since at least 2014. We recommended FEMA assess its workforce needs to address these shortcomings and develop a plan, including timelines. DHS agreed, identified steps FEMA has taken and plans to take to address the recommendation, and estimated addressing the recommendation by September 2019. Our prior reports found that FEMA experienced challenges coordinating with state, local, and federal partners over disaster preparation and response efforts. Coordination is critical to ensuring that states and localities have their own tools in place to facilitate disaster response, and that contracting needs are clearly communicated and considered among federal agencies. Yet FEMA faced continued challenges and inconsistencies in its coordination with states and localities over the use of advance contracts. In January 2017, FEMA updated guidance to include requirements for coordination with state and local governments on the use of federal advance contracts. This update was in response to our September 2015 finding that there were inconsistencies in whether and how staff in FEMA’s regional offices performed state and local outreach on advance contracting efforts. However, in December 2018, we reported on similar inconsistencies in state and local outreach. We found that FEMA’s guidance did not specify how often or what types of advance contract information should be shared with states and localities, or instruct FEMA contracting officers to encourage states and localities to establish their own advance contracts for the types of goods and services needed during a disaster. As a result, we found that while some FEMA regional officials regularly performed outreach with states and localities to assist them with establishing advance contracts for goods and services commonly needed during a disaster—like security, transportation, and office supplies—other FEMA regional officials did so less frequently. According to regional officials, coordinating more frequently with states and localities allows them to avoid overlap between state and federal contracting efforts, and helps FEMA officials know what resources the states have in place before a disaster occurs and how long states are capable of providing those resources following a disaster. We recommended in our December 2018 report that FEMA update its guidance to provide specific direction for contracting officers to perform outreach to states and localities on the use and establishment of advance contracts. DHS concurred and stated it would update guidance and continue efforts to establish resources for state and local governments on advance contracts. Information on FEMA’s advance contracts can be used to facilitate state and local coordination over the use and establishment of advance contracts. However, our work showed that this information was inconsistent and could further hinder FEMA’s information sharing and coordination efforts. In December 2018, we reviewed FEMA’s advance contract list and other resources FEMA contracting officials said they used to identify advance contracts—like biannual training documentation—and found differences in the advance contracts identified. For example, we reported that FEMA officials told us that the advance contract list available to contracting officers is updated on a monthly basis. However, our analysis found that 58 advance contracts identified on the June 2018 advance contract list had not been included in contracting officers’ May 2018 training documentation. The missing contracts included those for telecommunications services, generators, and manufactured housing units. Recognizing some of the shortcomings in communicating with state and local governments following the 2017 disasters, FEMA stated it would develop a toolkit to provide states and localities with recommendations for advance contracts, emergency acquisition guidance, and solicitation templates. However, at the time of our December 2018 review, FEMA officials were uncertain what information they would share with states and localities on advance contracts, and said they did not plan to provide the complete list of the advance contracts FEMA has in place to avoid being overly prescriptive. Yet without a centralized and up-to-date resource on advance contracts, FEMA contracting officers and their state and local counterparts may not be able to effectively communicate about advance contracts and use them to respond to future disasters. Given FEMA’s recent emphasis on the importance of states and localities having the capability to provide their own life-saving goods and services in the immediate aftermath of a disaster, we concluded that clearly communicating consistent and up-to-date information on the availability and limitations of federal advance contracts is imperative to informing state and local disaster response efforts. In our December 2018 report, we recommended that FEMA identify a single centralized resource listing its advance contracts and ensure that resource is updated regularly. Further, we recommended that FEMA should communicate information on advance contracts using that resource to states and localities to inform their advance contracting efforts. DHS concurred with these two recommendations and identified some steps it planned to take, but also stated it believes the existing advance contract list satisfies our recommendation for a single centralized resource. However, as our report noted, we found inconsistencies in this list that FEMA needs to address for advance contract information to be complete and up-to-date for the contracting officers who rely on it. In addition to challenges coordinating with state and local governments, we identified coordination and planning concerns between FEMA and other federal agencies. As the federal disaster coordinator, FEMA obtains requirements from states and localities. It then tasks the appropriate federal agencies with specific missions, based on their emergency support functions. Agencies assigned to specific missions are then responsible for fulfilling requirements, and may use contracts to do so. However, we reported in April 2019 that some federal agencies experienced challenges coordinating with FEMA and state and local partners. For example, USACE officials reported that, during their debris removal mission following the California wildfires, local officials believed that the soil removed would be replaced. However, this was not part of the mission assignment from USACE to FEMA. In these instances, agency officials told us they relied on FEMA to communicate information on their mission assignments to be able to administer contracts. According to a FEMA official during our April 2019 review, coordination and planning concerns related to mission assignments—like contracting considerations—should be worked out in advance between FEMA and agencies such as USACE. However, we found that FEMA policy and guidance lack details on how that coordination should take place. Further, a FEMA official told us that contracting considerations are not necessarily built into mission assignments. We recommended in April 2019 that FEMA revise its mission assignment policy and guidance to better incorporate consideration of contracting needs and ensure clear communication of coordination responsibilities related to contracting. DHS concurred and plans to develop tools and training within the next year to provide the necessary guidance. Limited transparency into disaster contracting obligations further complicates the challenges noted above. We found in April 2019 that the full extent of disaster contracting—for both advance and post-disaster contracts—related to the 2017 disasters was and continues to be unknown. This was due to changes in the criteria for establishing and closing a national interest action (NIA) code—a mechanism for government-wide tracking of emergency or contingency-related contracting—in FPDS-NG, and DHS’s inconsistent implementation of the updated criteria for closing codes. Specifically, the codes for Harvey and Irma closed on June 30, 2018, less than a full year after the hurricanes hit. The code for Maria is valid through June 15, 2019, about 21 months after that hurricane made landfall. This is in contrast to prior hurricanes, for which codes sometimes remained open more than 5 years after the disaster, with the code for Hurricane Katrina being open for 13 years after the disaster. The ability to identify disaster contracting for the 2018 hurricanes was similarly limited as the NIA codes for Hurricanes Florence and Michael expired on March 15, 2019 and April 12, 2019, respectively, about 6 months after those storms made landfall. Based on a memorandum of agreement, the General Services Administration (GSA), DHS, and the Department of Defense (DOD) are jointly responsible for determining when a NIA code should be established and closed. DHS delegated its role, on behalf of civilian agencies for disaster or emergency events, to its Office of the Chief Procurement Officer. The agreement outlines criteria DHS should consider in making determinations to establish and close a NIA code. For our April 2019 review, we identified changes in these criteria between June 2012 and June 2018. For example, the updated agreement does not include the national interest and visibility of an event as criteria for extending a NIA code, allowing a NIA code to expire regardless of the high visibility of the event and information needs of key users. DHS officials reported several rationales to support their decision to close the NIA codes for the 2017 hurricanes, but these were inconsistent with the criteria in the agreement and did not consider key user needs or fully explain the decisions to close the codes. Once a NIA code in FPDS-NG is closed, there is no other publicly available, government-wide system available to comprehensively track contract obligations for specific events. Our April 2019 report demonstrated the magnitude of contract dollars that are no longer easily trackable once a NIA code is closed. For example, using the description field in FPDS-NG, we found that between July 1 and September 30, 2018—after the NIA codes were closed—agencies obligated at least $259 million on contracts for Hurricanes Harvey and Irma. However, not all agencies put event-specific information in the description field, and we found for the 2017 hurricanes only 35 percent of contract obligations linked to a NIA code included this information. Moreover, as we have previously reported, and illustrate in figure 2, it can take years to fully account for federal contract obligations related to response and recovery after a hurricane. In our April 2019 report, we made two recommendations, including that GSA, in coordination with DOD and DHS, assess whether the criteria in the current NIA code agreement meets the long-term needs for high visibility events and account for the needs of users, such as FEMA, other agencies, and Congress; and in the interim, DHS, in coordination with DOD and GSA, should keep the existing NIA codes for disasters open, reopen the NIA codes for Hurricanes Harvey, Irma, Florence, and Michael, and request that agencies retroactively update applicable contract actions to reflect these codes, to the extent practicable. GSA and DOD indicated they would work jointly with DHS to assess the criteria in the agreement within the year. DHS did not comment on that recommendation. Given the high visibility and national interest in these events, assessing the criteria, keeping NIA codes open, and reopening closed codes for the recent disasters to the extent practicable would help ensure visibility over federal disaster contracts. In conclusion, given the circumstances surrounding the 2017 disasters, and the importance of preparedness for future disasters, it is critical to ensure that FEMA is well-positioned to respond through its use of contracts. Our work has shown that without effective planning on the use of contracts, FEMA may face challenges in quickly providing critical goods and services to survivors following a disaster. Further, without effective coordination, FEMA cannot ensure that local, state, and federal partners have the tools they need to assist in disaster response. Moreover, not tracking certain information on a government-wide basis in FPDS-NG may result in key users lacking the information necessary to provide oversight of FEMA’s and other agencies’ disaster contract actions. Implementing our recommendations to update its planning guidance and advance contract strategy; assess acquisition workforce needs; improve coordination with state, local, and federal partners; and improve tracking of disaster contracting actions will help FEMA overcome key challenges it faces in contracting during a disaster, and improve future response efforts. Chairman Payne, Chairwoman Torres Small, Ranking Members King and Crenshaw, and members of the subcommittees, this concludes my statement. I would be pleased to respond to any questions. If you or your staff have any questions about this statement, please contact me at (202) 512-4841 or makm@gao.gov. Contacts for our Office of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this statement include Janet McKelvey (Assistant Director); Caryn Kuebler and Meghan Perez (Analysts in Charge); Emily Bond; Erin Butkowski; Suellen Foth; Julia Kennon; Sylvia Schatz; Lindsay Taylor; and Robin Wilson. Key contributors for the previous work on which this statement is based are listed in the products cited. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "According to FEMA—a component within DHS—the 2017 disasters affected 47 million people, or about 15 percent of the nation's population. Federal contracts have played a key role in responding to these disasters and in long-term community recovery. So far, FEMA has obligated billions of dollars on these contracts. This testimony is based primarily on GAO's recent reports on disaster contracting—specifically advance contracting and post-disaster contracts related to the 2017 disasters—which detail much of FEMA's disaster contracting activities. It addresses key challenges FEMA faced contracting for goods and services in response to these disasters. To conduct this work, GAO analyzed data from the Federal Procurement Data System-Next Generation through June 30, 2018, the latest and most complete data available for the 2017 disasters. GAO also analyzed FEMA guidance and documentation and interviewed FEMA officials to discuss the use of contracts to respond to the 2017 disasters. Following Hurricanes Harvey, Irma, and Maria, and the 2017 California wildfires, federal agencies entered into disaster-related contracts worth about $9.5 billion, according to data as of June 30, 2018—the latest and most complete data at the time of GAO's review (see figure). The Federal Emergency Management Agency (FEMA) obligated about $2.9 billion of this total through advance contracts, which it establishes prior to a disaster to rapidly mobilize resources. FEMA obligated an additional $1.6 billion through post-disaster contracts, which are established after disasters hit. In its December 2018 and April 2019 reports, GAO made 10 recommendations to strengthen FEMA's ability to address challenges GAO identified in how FEMA plans, coordinates, and tracks its contracts: Planning: FEMA has an outdated strategy and unclear guidance on how contracting officers should use advance contracts and has not fully assessed its contracting workforce needs. Effectively planning its contract use is critical to FEMA quickly providing critical goods and services. Coordination: FEMA did not fully coordinate with states and localities on certain contracts and encountered communication and coordination challenges with other federal agencies. Effective coordination helps FEMA ensure stakeholders have the tools needed to facilitate their disaster response efforts. Tracking: The full extent of 2017 disaster contracting activities, for FEMA and other agencies, is unknown. GAO found that codes used to track obligations for these disasters in a federal procurement data system were closed without full consideration of user needs or due to inconsistent implementation of criteria established by the Department of Homeland Security (DHS) and other agencies, limiting visibility over federal disaster contracts. GAO has made a total of 19 recommendations—most of which were to FEMA—related to contracting activities in response to the 2017 disasters. Ten of these are described in this statement. DHS concurred with most of these recommendations, and has some actions underway, but it has not fully implemented them. Attention to these recommendations can assist FEMA as it uses contracts to respond to future disasters.", "document_type": "gao"}
{"report": "Medicaid expenditures are financed jointly by the federal government and the states. In order to receive federal matching funds for Medicaid expenditures, states must adhere to a broad set of federal requirements and administer their programs consistent with individual state plans approved by CMS. These plans are agreements between a state and the federal government that describe how states will administer their Medicaid programs, including how the state will administer Medicaid third- party liability procedures. When states make changes to their Medicaid programs or policies, including when necessary to comply with a change in federal law, they must submit a state plan amendment to CMS. CMS reviews and approves state Medicaid plans and state plan amendments. The federal government matches each state’s Medicaid expenditures for services according to a statutory formula called the Federal Medical Assistance Percentage. This formula provides for a match that is no lower than 50 percent of a state’s Medicaid expenditures and no higher than 83 percent. States can receive a 90 percent federal match for the costs associated with the development of each state’s Medicaid Management Information System (MMIS), a claims processing and retrieval system supporting the administration of the state’s Medicaid program. States also receive a 75 percent match for the costs associated with ongoing MMIS maintenance and operations. States use their MMIS systems to process provider claims, including claims for prenatal care services, pediatric preventive services, and services provided to CSE beneficiaries. The Medicaid program is administered at the state level and overseen at the federal level by CMS, which, among other things, ensures that funds are used appropriately and beneficiaries have access to covered services. Medicaid allows significant flexibility for states to design and implement their programs. Within broad federal parameters, states have discretion in, among other things, setting Medicaid eligibility standards and provider payment rates; determining the amount, scope, and duration of covered benefits; and developing their own administrative structures. States may also decide how Medicaid-covered services provided to beneficiaries will be delivered. For example, states may pay health care providers for each service they provide—fee-for-service—or contract with MCOs to provide a specific set of Medicaid-covered services to beneficiaries and pay them a set amount per beneficiary, typically per month. While most states use both delivery systems, the percentage of beneficiaries served through MCOs has grown in recent years, and represented nearly 70 percent of all Medicaid beneficiaries in 2016. Medicaid beneficiaries across various eligibility categories may have access to private health insurance or other sources of third-party coverage. For example, some adult beneficiaries may be covered by employer-sponsored private health insurance even though they also qualify for Medicaid. Children, similarly, may be eligible for Medicaid, while also being covered as a dependent on a parent’s private health plan. As such, federal law requires states to perform various activities to ensure that Medicaid is the payer of last resort, including taking all reasonable measures to identify Medicaid beneficiaries’ other potential sources of health coverage and their legal liability. Specifically, states must ensure that the following steps, among others, are taken. 1. Coverage identification. To identify beneficiaries with third-party health coverage, states are required to request coverage information from potential Medicaid beneficiaries at the time the agency makes any determination or redetermination of eligibility. States are also required to obtain and use information pertaining to third-party liability, for example, by conducting data matches with state wage information agencies, Social Security Administration wage and earning files, state motor vehicle accident report files, or state workers’ compensation files. 2. Coverage verification. When other health coverage is identified, states often verify the information, including the services covered through the other insurance and the dates of eligibility. 3. Cost avoidance payment procedures. As a general rule, federal law requires states to apply cost avoidance payment procedures to claims for most Medicaid items and services. Under cost avoidance procedures, the state must reject claims for which a third party is or is probably liable, and the agency instructs the provider to collect from the third party. Once the provider determines the amount of the third party’s liability, the provider submits a claim to the state Medicaid agency for any remaining balance, up to the maximum amount allowed under the state’s payment schedule. States are then required to make timely payment to the provider, generally within 30 days from the date the claim for the balance is filed. 4. Pay-and-chase payment procedures. The Consolidated Omnibus Budget Reconciliation Act of 1985 made an exception to cost avoidance procedures for three types of services: prenatal care services, pediatric preventive services, and services provided to CSE beneficiaries. It required states to pay such claims without regard to the liability of the third party, a procedure CMS calls “pay and chase.” Under the pay-and-chase payment procedure, the state Medicaid agency is generally required to make a timely payment to the provider within 30 days, and then the state, instead of the provider, will seek to recover payment from any potentially liable third parties within 60 days. According to CMS, cost avoidance does not apply to these claims because there is a risk some providers might not participate in Medicaid to avoid dealing with the administrative burden of cost avoidance. (See fig. 1.) The Bipartisan Budget Act of 2018 amended various sections of the Medicaid third-party liability statute, including the required processes states must follow when paying claims with probable third-party liability for the following three types of services: Prenatal care services. The Bipartisan Budget Act of 2018 eliminated, effective February 2018, the statutory exception for prenatal care services that had required states to apply pay-and- chase procedures to such claims. Thus, under the amended statute, states must apply cost avoidance procedures to claims for prenatal care services when it is apparent that a third party is or may be liable at the time the claim is filed. Additionally, to the extent states had opted under CMS regulations to apply pay-and-chase procedures to claims for labor, delivery, or postpartum care services—which CMS calls “pregnancy-related services”—states must now apply cost avoidance procedures to those as well. Pediatric preventive services. Beginning in October 2019, under federal law as amended by the Bipartisan Budget Act of 2018, states are no longer required to pay claims for pediatric preventive services immediately. While states will still have the option to apply pay-and- chase procedures to these claims, a state may instead choose—if it determines doing so is cost-effective and will not adversely affect access to care—to require the provider to first submit the claim to the third party and wait 90 days for payment by the third party before seeking Medicaid payment. For purposes of this report, we refer to such a 90-day period as a “wait-and-see period.” Services provided to CSE beneficiaries. Beginning in October 2019, states must make payment for a CSE beneficiary’s claim if the third party has not paid the provider’s claim within a 100-day wait-and- see period. However, the state may instead choose—if the state determines doing so is cost-effective and necessary to ensure access to care—to make payment within 30 days. (See table 1.) Once the third-party liability changes in the Bipartisan Budget Act of 2018 are fully implemented, states will have authority to require providers to wait longer to receive Medicaid payments in certain circumstances. For example, Prenatal care services claims, which were previously paid within 30 days under pay-and-chase procedures, are now subject to cost avoidance. This could potentially result in providers waiting indefinitely to receive payment, depending on whether the provider is able to resolve the third-party liability (i.e., submit a claim for payment to the third party and determine the amount of the third-party liability), which must occur before the state may make payment under cost avoidance. Pediatric preventive services claims, which are generally paid within 30 days under pay-and-chase procedures, could be subject to a 90-day wait-and-see period beginning in October 2019 if a state decides to implement one. This could result in providers waiting 120 days to receive payment (90 days to wait and see if the liable third party pays, and then another 30 days for the state to make timely payment on any remaining balance). Claims for services for CSE beneficiaries, which are currently subject to pay-and-chase procedures or a 30-day wait-and-see period at state option, may be subject to either a 30-day or 100-day wait-and- see period beginning in October 2019, depending on which option the state chooses. This could result in providers waiting 130 days to receive payment (up to 100 days to wait and see if the liable third party pays, and then another 30 days for the state to make timely payment on any remaining balance). Officials from four of the nine selected states we reviewed reported having implemented the required third-party liability changes for prenatal care services. The changes were required to be implemented in February 2018. For the third-party liability changes affecting pediatric preventive services and services provided to CSE beneficiaries, which are due to take effect October 2019, Medicaid officials from six of the nine selected states noted that they were in the very early stages of planning how they might implement the changes. Officials from four of the nine selected states we reviewed stated that their state Medicaid agency had implemented the mandated third-party liability changes for prenatal care services, which required states to implement cost avoidance payment procedures for claims for these services beginning in February 2018. Officials from three of the four states that have implemented the third-party liability changes for prenatal care services told us that changing from pay-and-chase to cost avoidance procedures involved identifying all the applicable service codes for prenatal care and making the necessary changes in their systems to ensure that any new claims were subject to cost avoidance procedures. They said it also involved communicating the need for such changes to the MCOs in their state. State Medicaid officials from the remaining five states generally noted that they were discussing the changes internally, researching how to implement the changes in their MMIS, assessing the likely impact of these changes on MMIS, or waiting for additional guidance from CMS. For example: Officials from several states noted that they were undertaking activities, such as identifying the prenatal care codes in their data systems that would need to be switched to cost avoidance payment procedures, or researching the best way to implement these changes. Officials from one state said they were in the process of assessing what the likely impact of these changes on beneficiaries and providers would be, and would only subject claims for prenatal care services to cost avoidance if they determined that doing so was the best course of action. Officials from one state indicated that they were waiting to determine whether it was more cost effective to implement these changes in their legacy MMIS, or wait and implement the changes in the new MMIS they are planning to roll out in the future. State Medicaid officials also described other efforts that they would need to undertake as they implemented these changes to third-party liability. These included staff retraining and communicating the changes to providers in their states. Beyond state Medicaid programs, officials from the five Medicaid MCOs we interviewed all stated that their organizations had not yet implemented the prenatal care third-party liability changes. The MCO officials stated that they were waiting for additional instructions on how to implement the third-party liability changes or for revised contract language from their state Medicaid agencies. Officials from one of the MCOs noted they were not aware of the third-party liability changes until we reached out to them for an interview. Officials from two MCOs we interviewed generally agreed that the third-party liability changes for prenatal care services would require changes to claims processing systems and internal processes, but would not be significant. Several MCO officials noted that these changes would likely result in some cost-savings to MCOs in the future. Medicaid officials from six of the nine selected states noted that they were in the very early stages of planning whether—or how—they would implement the wait-and-see periods for pediatric preventive services and services to CSE beneficiaries. For example, some Medicaid officials from these six states described how they were assessing what changes would need to be made to their MMIS, deciding whether to implement the wait-and-see periods, or exploring how to assess the potential impact of these changes. Some Medicaid officials also expressed uncertainty regarding how such changes would affect Medicaid beneficiaries or the amount of effort required by their agency to implement the third-party liability changes. Officials from one state noted that they had begun discussions about implementing the third-party liability changes for both pediatric preventive services and services for CSE beneficiaries in June 2018, and were in the process of identifying the necessary system changes needed to implement third-party liability changes by the October 2019 effective date. Officials from two of these states stated that they do not believe their state will implement the wait-and-see periods for pediatric preventive services or CSE beneficiaries when the changes go into effect. Officials from the remaining three states noted at the time of our interviews they had not yet developed plans for assessing implementation of these changes. Table 2 summarizes the status of selected states’ implementation of the third-party liability changes. For pediatric preventive services, state Medicaid officials generally noted that the third-party liability changes would involve identifying the relevant codes and making changes to their MMIS to ensure those claims were subject to a wait-and-see period, if implemented. Several state Medicaid officials characterized this effort as “significant” or “difficult.” For services delivered to CSE beneficiaries, officials from several state Medicaid agencies speculated that making the third-party liability changes to their MMIS would necessitate having some sort of indicator in their system to identify which claims were for the CSE beneficiaries and, therefore, should be subject to a wait-and-see period, if implemented. Some state Medicaid agency officials said that this would require obtaining the information from another state agency responsible for administering CSE agreements. Several of the state Medicaid officials we interviewed expressed concerns regarding how to implement the wait-and-see periods for pediatric preventive services and services for CSE beneficiaries. Specifically, these officials noted that—within their MMIS—it is not possible to capture on a Medicaid claim when a provider has billed a third party, waited a specified amount of time, and not received payment. As a result, officials from one state noted that additional guidance from CMS on how to implement and track provider billing of third parties—including wait-and-see periods and providers’ collection of payment—would be necessary before moving forward with implementing the third-party liability changes. Officials from two states said that the administrative burden associated with these changes would possibly make them not cost-effective to implement. However, MCO officials we interviewed generally acknowledged that while these changes would require changes to their claims processing systems and internal processes, they were not significant and could potentially result in some cost-savings to their MCO in the future. The third-party liability change affecting all Medicaid services provided to CSE beneficiaries was a particular concern for officials from three state Medicaid agencies and three MCOs. Specifically, these officials said there is currently no way to identify CSE beneficiaries in their MMIS or claims processing systems, which could potentially make this change difficult, if not impossible, to implement. Officials from one state described how setting up a system to receive this information would involve a significant effort, potentially necessitating new hardware and system modifications, as well as a data sharing agreement with the state entity maintaining the CSE information. Officials from one MCO noted that the third-party liability changes affecting CSE beneficiaries was a particular concern, because those changes would potentially require additional administrative work and changes to their processes in order for providers in their network to track down insurance information from a non-custodial parent. After enactment of the Bipartisan Budget Act of 2018 in February 2018, CMS issued guidance in the form of an informational bulletin to states on June 1, 2018, to facilitate states’ implementation of the key provisions of the Bipartisan Budget Act of 2018 related to third-party liability in Medicaid. However, CMS’s June informational bulletin is missing some key information and contains information that is inconsistent with the federal law. This is inconsistent with CMS’s responsibility for ensuring states’ compliance with federal requirements. In particular, Pregnancy-related claims. Under federal law, states must apply standard cost avoidance procedures to all non-pediatric claims, including claims for prenatal services beginning in February 2018. However, CMS guidance indicates that a state need not apply cost avoidance procedures to claims for labor and delivery services if those claims can be differentiated from prenatal services. The guidance also provides that, effective October 1, 2019, states will have 90 days to pay claims related to labor, delivery, and postpartum care claims. As a result, CMS’s guidance is inconsistent with federal law, which requires states to reject any such claim under cost avoidance procedures until the third-party liability is resolved, regardless of how many days that might take. Pediatric preventive claims. Under federal law, states must generally apply pay-and-chase procedures to pediatric preventive services. However, beginning in October 2019, states are permitted to implement a 90-day wait-and-see period before making payment for these services if the state determines that it would be cost-effective and would not adversely affect access to care to do so. However, CMS guidance simply provides that states will have 90 days to pay such claims, suggesting that states need not make the cost- effectiveness or access determinations required by statute. CSE beneficiary claims. Under federal law, beginning in October 2019, for claims for services to CSE beneficiaries, states may choose to make payment within 30 days (as opposed to implementing a 100- day wait-and-see period), if the state determines doing so is cost- effective and necessary to ensure access to care. If the state does not make such a determination, the statute would require the state to avoid making payment for such services for up to 100 days to allow third parties to make payment first. However, CMS guidance does not identify this as an option for states. Instead, CMS guidance simply provides states with the option of implementing the wait-and-see period, omitting the option for states to make payment within 30 days. CMS officials told us that the Bipartisan Budget Act of 2018 did not change state responsibilities related to cost-effectiveness and access to care, and CMS does not intend to issue additional guidance on this issue. However, prior to enactment of the Bipartisan Budget Act of 2018 in February 2018, federal third-party liability law did not authorize states to apply cost avoidance procedures to preventive pediatric claims or pediatric services provided to CSE beneficiaries. Furthermore, other CMS guidance documents, such as the third-party liability handbook and CMS regulations on third-party liability, are out of date and not a reliable source of information for states to use in implementing the new federal third-party liability requirements. In particular, the third-party liability handbook was last revised in 2016 and does not reflect the Bipartisan Budget Act of 2018 changes. Additionally, CMS regulations implementing federal requirements for state payment of claims for prenatal care, labor and delivery services, postpartum care, preventive pediatric services, and services to CSE beneficiaries were last amended in 1997 and, accordingly, do not reflect current statutory requirements, including the Bipartisan Budget Act of 2018 requirement to cost avoid prenatal and other non-pediatric claims beginning February 2018. CMS officials told us the agency is in the process of updating its third- party liability handbook and anticipates issuing the updated document in September 2019. Agency officials also told us they plan to revise the agency’s regulations regarding pay-and-chase and release the revised regulations in early 2020. However, federal law requires state Medicaid plans to provide for proper third-party liability procedures, which states often carry out through references to federal regulation, according to CMS officials. Without updated third-party liability guidance that is timely, complete, and consistent with federal law, states may lack the necessary information to update their state Medicaid plans so that they comply with these requirements. CMS has not taken steps to determine the extent to which state Medicaid agencies are meeting the third-party liability requirements, and therefore CMS officials were unaware of whether states were meeting the new requirements. In particular, CMS officials did not know the extent to which the selected states in our review had implemented the required third-party liability changes. In our interviews with nine selected state Medicaid agencies conducted between November 2018 and March 2019, we learned that five states continued to apply pay-and-chase procedures to prenatal care claims, despite the federal requirement to implement cost avoidance since February 2018. During our interviews, we also learned that CMS had not monitored state Medicaid agencies’ third-party liability approaches prior to the Bipartisan Budget Act of 2018. For example, officials from one of the selected states told us that they had been using cost avoidance for most claims for pediatric preventive care, rather than applying pay-and-chase procedures, as required by law. We also learned from an official from another selected state that the state had been applying cost avoidance procedures to claims for prenatal care services well in advance of the enactment of the Bipartisan Budget Act of 2018, despite the federal requirement to apply pay-and-chase procedures to such claims from 1986 to 2018. CMS’s failure to monitor the implementation of the third-party liability changes in the Bipartisan Budget Act of 2018 is inconsistent with the agency’s responsibilities for oversight of the Medicaid program, including ensuring that federal funds are appropriately spent. We have previously recommended that, given the significant federal Medicaid outlays, the federal government has a vested financial interest in further increasing states’ third-party liability cost savings, and that CMS should play a more active leadership role in monitoring, understanding, supporting, and promoting state third-party liability efforts. However, CMS officials stated that they expect states to comply with current law for Medicaid third-party liability, and that they do not verify whether states have implemented the required third-party liability changes unless the agency is made aware of non-compliance. When asked how CMS ensures that states apply pay-and-chase procedures required under federal law, such as for pediatric preventive claims, CMS officials stated that it is the agency’s expectation that states comply with current law. According to agency officials, if a state has difficulty complying and reaches out to CMS for technical assistance, the agency will work with that state to come into compliance. CMS officials told us that CMS plans to review all state Medicaid plans and provide technical assistance for any necessary action only after the agency has updated its regulations related to third-party liability. As of May 2019, CMS anticipated that it would release updated regulations in early 2020. Because CMS has not monitored states’ compliance with federal third- party liability requirements, the agency does not know whether states have applied the federally required third-party liability procedures to certain Medicaid claims as required by federal law. In the case of prenatal care services claims, the failure to implement cost avoidance payment procedures could result in unnecessary Medicaid expenditures, to the extent that Medicaid pays providers for services for which a third party is liable. To the extent that states are not properly applying pay-and-chase procedures to pediatric preventive service claims, children’s access to such services could be impacted. According to most of the stakeholders we interviewed, Medicaid providers—especially prenatal care and rural providers—could face increased administrative requirements or delays in payments for services as a result of the third-party liability payment changes to the three service categories in the Bipartisan Budget Act of 2018. Several stakeholders agreed that the tasks associated with identifying sources of third-party liability and attempting to collect from third parties would shift from state Medicaid agencies to providers as a result of the payment changes, although opinions differed on the extent to which this shift would affect providers. Several stakeholders said that the third-party liability changes could increase administrative requirements for providers, because obtaining accurate information on third-party liability sources for Medicaid beneficiaries and resubmitting claims that result from incorrect or outdated third-party liability information can be resource intensive and time consuming. One provider and officials from one state Medicaid agency noted that providers may lack the administrative resources or claims-processing expertise to deal with these changes. Officials from one state Medicaid agency, two state provider associations, and an organization advocating for Medicaid beneficiaries also noted that providers may encounter Medicaid beneficiaries who may be unaware or may not disclose that they have other insurance policies; for example, children who are covered under multiple insurance policies by custodial and non-custodial parents or experience insurance transitions following birth. These issues may increase the amount of time and resources providers spend on processing and resubmitting claims. Other stakeholders were less certain that the added requirements would cause difficulties for providers. Officials from one state Medicaid agency and one MCO said that the payment changes would not be difficult to implement, because providers were familiar with billing third parties for medical services for other beneficiaries. Officials from four state Medicaid agencies and two MCOs noted that providers may prefer to submit claims to commercial insurers, because these insurers pay at a higher rate compared with state Medicaid programs. Several stakeholders we interviewed agreed that providers could wait longer periods of time for payment as they track down third-party insurers or wait up to 100 days for potential payment from these insurers before seeking payment from the state Medicaid agency. According to one provider and officials from two provider associations and one MCO, these delays could put providers at risk of not receiving payments for services or not having enough cash on hand to sustain operations. Additionally, officials from three provider associations noted that payment delays would affect pediatric providers in particular, because the majority of the services that pediatricians provide are preventive care—which would be affected by the third-party liability changes. According to several stakeholders we interviewed, smaller or independent providers and those located in rural areas could be more affected by the third-party liability changes compared with providers affiliated with managed care systems or those located in urban areas. Officials from one state Medicaid agency, two provider associations, and one MCO noted that smaller or rural-based providers generally have fewer staff and resources to deal with the larger volume of administrative paperwork and delays in payment for services that could result from the payment changes. Most of the stakeholders we interviewed said that providers might be less willing to serve Medicaid beneficiaries due to the administrative and payment issues, potentially reducing access to care or delaying services for children and pregnant women. However, some other stakeholders said that the third-party liability changes would have little to no effect on Medicaid beneficiaries. Officials from one state Medicaid agency and one MCO noted that third-party liability payment practices for other Medicaid populations and services have been in place for many years, and providers would already be familiar with processing claims with third-party liability. Several stakeholders said that providers may opt to reduce or eliminate the number of Medicaid beneficiaries they serve, because of actual or perceived increase in administrative requirements or payment delays. Officials from three state provider associations speculated that the potential for additional delays in payment for services could be the “final straw” in providers’ decision to stop serving Medicaid beneficiaries. Other stakeholders, including a Medicaid expert, one provider, and officials from one state provider association noted that providers may decide to see fewer Medicaid beneficiaries, but are unlikely to stop seeing them entirely, because some providers are reluctant to deny care to these beneficiaries. Payment delays could also lead to delays in beneficiaries receiving time- sensitive services, such as immunizations, as well as reduced access to specialists, such as midwives or mental health professionals, according to several stakeholders. Officials from one national provider association and an organization advocating for Medicaid beneficiaries noted that providers may seek to identify sources of third-party liability before providing services to beneficiaries. In addition, officials from one state Medicaid agency, a state provider association, and an organization advocating for Medicaid beneficiaries expressed concern that the third-party liability changes had the potential to reduce access to care for populations, such as children and pregnant women, that already faced challenges in accessing adequate, timely, or quality health care. Medicaid officials we interviewed from seven of the nine selected states said that their agencies will—or could—use existing methods to assess the effects of the third-party liability changes on provider availability and beneficiary access to prenatal care services, pediatric preventive services, and services for CSE beneficiaries. Officials from the remaining two states did not discuss or provide information on how they could assess the effects of the changes. Medicaid officials provided examples of existing methods that could be used to assess the effects of payment changes: Tracking beneficiary access by comparing a set of access-to-care measures for a state’s Medicaid population with its non-Medicaid, commercially insured population, as well as carrying out customer satisfaction surveys with Medicaid beneficiaries, Using a third-party liability hotline to track patient issues and conducting secret shopper calls to monitor if providers are accepting new patients, Contracting with a state university to evaluate Medicaid beneficiary access for prenatal and pediatric services. In addition, one state has an independent health advocacy agency that monitors and seeks to resolve provider availability and beneficiary access issues on behalf of the state’s Medicaid population. However, one state Medicaid official and a Medicaid expert agreed that measuring any possible effects of the third-party liability changes—such as a decline in provider availability or beneficiary access—would be difficult without baseline data. According to officials from two state Medicaid agencies and a Medicaid expert, many other factors could potentially affect provider availability and beneficiary access, making it difficult or impossible to pinpoint if a decline in provider availability or beneficiary utilization of services was the result of the third-party liability changes or something else—such as changes in the managed care market or levels of private coverage among beneficiaries. We found other evidence suggesting that it might be challenging for some states to assess the effects of the third-party liability changes. Specifically, Medicaid officials from eight of the nine selected states did not readily identify the number of beneficiaries in their state that had third- party liability and would be affected by the changes. Moreover, officials from two states noted that obtaining this data would require a “significant” effort. Officials from five states shared information on the number of children, pregnant women, or births covered by Medicaid in their state, but did not specify how many of these beneficiaries had other insurance coverage. Seven of the nine selected states had no data readily available on CSE beneficiaries who were also covered by Medicaid. In several cases, officials noted that their MMIS or other data systems had no way to track whether a child was a CSE beneficiary. Omissions and inaccuracies in CMS’s guidance to states on third-party liability changes from the Bipartisan Budget Act of 2018 have the potential to adversely affect the extent to which Medicaid expenditures are being used to pay for services for which a third party is liable, as well as states’ compliance with federal requirements. Furthermore, CMS has not assessed whether state Medicaid agencies are complying with federal third-party liability requirements, under which states must change how they pay claims for certain services as a result of the Bipartisan Budget Act of 2018 and subsequently enacted legislation. In the absence of CMS overseeing states’ compliance, the agency cannot ensure that federal funds are being spent properly and that states are complying with current federal statute. We are making the following two recommendations to CMS: The Administrator of CMS should ensure the agency’s Medicaid third- party liability guidance is consistent with federal law related to the requirement for states to apply cost avoidance procedures to claims for labor, delivery, and postpartum care services, the requirement for states to make payments without regard to potential third-party liability for pediatric preventive services unless the state has made a determination related to cost-effectiveness and access to care that warrants cost avoidance for 90 days, and state flexibility to make payments without regard to potential third- party liability for pediatric services provided to child support enforcement beneficiaries. (Recommendation 1) The Administrator of CMS should determine the extent to which state Medicaid programs are meeting federal third-party liability requirements and take actions to ensure compliance as appropriate. Such actions can include ensuring that state plans reflect the law. (Recommendation 2) We provided a draft of this report to the Department of Health and Human Services for comment. In its written comments, which are reprinted in appendix I, HHS concurred with our recommendations and indicated a commitment to providing states with accurate guidance on the third-party liability changes in the Bipartisan Budget Act of 2018. The agency noted that it is in the process of updating its guidance and third-party liability handbook to reflect the changes and ensure that such guidance is consistent with federal law. The agency also noted that it will determine the extent to which state Medicaid programs are meeting federal third- party liability requirements and will take actions to ensure compliance. We are sending copies of this report to the appropriate congressional committees, the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or yocomc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs can be found on the last page of this report. Major contributors to this report are listed in appendix II. In addition to the contact named above, Tom Conahan (Assistant Director), Andrea E. Richardson (Analyst-in-Charge), Marybeth Acac, Drew Long, Corinne Quinones, Jennifer Rudisill, and Ethiene Salgado- Rodriguez made key contributions to this report.", "summary": "The Medicaid program is typically the payer of last resort. The Bipartisan Budget Act of 2018 changed the Medicaid third-party liability payment requirements for prenatal care services, pediatric preventive services, and services provided to CSE beneficiaries. Before the act, in the case of these three services, states were generally required to pay providers for services delivered to Medicaid beneficiaries and then obtain any payments from liable third parties. The Bipartisan Budget Act of 2018 also included a provision for GAO to study the potential effects of these changes. In this report, GAO (1) describes the status of selected states' implementation of Medicaid third-party liability changes; (2) evaluates CMS's implementation and oversight of the Medicaid third-party liability changes; and (3) describes stakeholders' views of the possible effects of these changes on providers and beneficiaries. GAO conducted interviews with state Medicaid agencies and provider associations in nine selected states, which were selected by taking into consideration Medicaid spending and stakeholder recommendations, among other factors. GAO also conducted interviews with national experts in Medicaid, national organizations representing beneficiaries and providers, and officials from CMS. Medicaid officials in the nine selected states GAO reviewed described being in various stages of implementing third-party liability changes as required by law. These changes affect whether health care providers must seek payment from a liable third party, such as private insurance, before the state Medicaid agency pays for services. The changes apply to prenatal care services, pediatric preventive services, and services for children subject to child support enforcement (CSE beneficiaries). At the time of GAO's review, Officials from four of the nine selected states reported having fully implemented the changes for prenatal care services, which were required to be implemented starting in February 2018. Officials from the remaining five states were discussing the changes internally, researching how to implement the changes in their Medicaid payment systems, or waiting for additional guidance from the Centers for Medicare & Medicaid Services (CMS), the federal agency responsible for overseeing states' Medicaid programs. None of the nine states had implemented the changes to pediatric preventive services and services for CSE beneficiaries, which must be implemented starting in October 2019. Officials from six states told GAO that they were in the early stages of exploring how they would make the changes, while the remaining three states had not developed such plans. GAO found that guidance issued by CMS in June 2018 to assist states in implementing the third-party liability changes contains information inconsistent with the law. For example, CMS's guidance incorrectly informs states that providers do not need to seek third-party payments before the state pays for some prenatal services. In addition, CMS has not determined the extent to which states are meeting third-party liability requirements. CMS officials stated that they expect states to comply with current law for Medicaid third-party liability and that they do not verify whether states have implemented the required third-party liability changes unless the agency is made aware of non-compliance. However, this approach is inconsistent with CMS's Medicaid oversight responsibilities, including its responsibility to ensure federal funds are appropriately spent. Medicaid experts and other stakeholders told GAO that the third-party liability changes could affect some health care providers in ways that could result in decreased beneficiary access to care, because some providers might be less willing to see Medicaid patients. According to stakeholders, this could occur for two primary reasons. 1. The changes may increase administrative requirements for providers by requiring them to identify sources of coverage, obtain insurance information, and submit claims to third-party insurers before submitting them to Medicaid. 2. The changes may result in providers waiting longer to receive Medicaid payment for certain services to the extent that states require providers to seek third-party payments before paying the providers' claims. GAO is recommending that CMS (1) ensure that its guidance to states on third-party liability requirements reflects current law, and (2) determine the extent to which state Medicaid programs are meeting federal third-party liability requirements. The Department of Health and Human Services concurred with these recommendations.", "document_type": "gao"}
{"report": "The internet is a worldwide network of networks comprised of backbone networks, servers, and routers. Network addresses are used to help send information from one internet-connected device, such as a computer, to another by routing the information to its final destination. The protocol that enables the administration of these addresses is the IP. IP addresses provide a numerical description of the location of networked devices such as computers, routers, and smartphones. These numerical descriptions allow devices to be distinguished from each other over the internet. In some ways, an IP address is like a physical street address. For example, in the physical world, if you want to send a letter from one location to another, the contents of the letter must be placed in an envelope that lists both the sender’s and the recipient’s addresses. Similarly, if data is transmitted across the internet from one device to another, IP addresses must be placed in an IP header with sender and recipient information. In addition to containing the addresses of the sender and the receiver, the header also contains a series of fields that provide information about what is being transmitted. Figure 1 provides a simplified illustration of this concept. The Internet Engineering Task Force, the principal body engaged in the development of internet standards, developed IPv6 in the 1990s to address IPv4’s limited address space, among other things. Although IPv6 has been available for over 20 years, IPv4, the older IP, is still more widely deployed. Nevertheless, IPv6 deployment is rising worldwide amidst the exhaustion of available IPv4 addresses. However, IPv6 is not backwards compatible with IPv4, which means that organizations such as DOD have to change their network infrastructure and systems in order to deploy IPv6. DOD relies on its current IPv4 networks to fulfill its mission to provide the military forces needed to deter war and ensure our nation’s security. According to DOD officials leading the transition to IPv6, the department utilizes IPv4 for enterprise-wide and mission partner wired and wireless communications, including infrastructure, technologies, and devices supporting large-scale globally dispersed command and control systems, naval vessels, aircraft, satellites, and ground operations. Figure 2 presents a simplified depiction of the department’s use of IPv4. DOD’s mission requires considerable IP address space. The department currently has 300,149,760 IPv4 addresses—more than any other organization in the world. These approximately 300 million addresses are divided into blocks; each block contains a series of multiple addresses. DOD has 13 particularly large blocks of addresses, each containing 16,777,216 addresses. Also included in the approximately 300 million addresses are 59,767,040 IPv4 addresses that currently are unused. These addresses are reserved for future use by DOD and its components. DOD has stated that it expects to exhaust its reserve of unused IPv4 addresses by 2030. According to the officials leading the IPv6 transition effort, the department expects to have to support IPv4 after it exhausts its IPv4 address space in 2030 due to mission system modernization and replacement timelines, as well as new emerging technologies that may require IPv4 resources while the department transitions to IPv6. According to prior government reports and DOD documentation, IPv6’s improved functionality would benefit the department in many ways. In addition to the general benefits of eliminating IPv4 address space limitations, enhancing mobility features, and integrating IP security, IPv6 has the potential to enhance DOD battlefield operations, improve decision-making with the increased reliance on the Internet of Things (IoT), support U.S. global technological competitiveness, and enhance mission partner interoperability. IPv6 eliminates address space limitations. As previously mentioned, IPv6 dramatically increases the amount of possible address space from 4.3 billion addresses in IPv4 to approximately 340 undecillion (3.4 × 10In contrast to IPv4, the massive address space available in IPv6 will allow almost any device to be assigned a unique IP address. This change fosters greater end-to-end communication abilities between devices with unique IP addresses and can better support the delivery of data-rich content such as voice and video. Enhanced mobility improves connectivity. IPv6 improves mobility features by allowing each device to have a unique IP address independent of its current network or connection point to the internet. This enables mobile IPv6 users to move from network to network while keeping the same unique IP address. The ability to maintain a constant IP address while switching networks is cited as a key factor for the success of a number of evolving capabilities, such as telephone technologies, laptop computers, and internet connected automobiles. Added IP security helps to protect data. IP security—a means of authenticating the sender and encrypting the transmitted data—is better integrated into IPv6 than it is in IPv4. This improved integration, which helps make IP security easier to implement, can help support broader data protection efforts. This extra security is accomplished through the use of two header extensions that can be used together or separately to improve the authentication and confidentiality of data being sent via the internet. These headers serve to provide the receiver with a greater assurance of the sender’s identity and provide encryption protection for the transmitted data. Enhanced capabilities could improve DOD battlefield operations. According to the 2014 DOD IG report on DOD’s IPv6 transition efforts, use of IPv6 could provide DOD with several potential benefits related to battlefield operations, such as improved communication, warfighter mobility, situational awareness, and quality of service. IPv6 auto- configuration capabilities provide secure ad hoc networking and mobility, as well as improved end-to-end security and simplified network management capabilities. This could potentially enable individuals and entire units to disconnect from military base networks, travel into theater, and quickly establish communications. Additionally, IPv6 capabilities could allow warfighters and commanders to improve situational awareness and mission execution during deployment and battle operations, allowing units to securely move from one wireless network to another. Increased use of the IoT may improve decision-making. The increased address space available with IPv6 enables the increased connectivity necessitated by the proliferation of the IoT. DOD’s IoT could include any object for which remote communication, data collection, or control might be useful, such as vehicles, appliances, medical devices, electric grids, transportation infrastructure, manufacturing equipment, or building systems. According to DOD’s Digital Modernization Strategy from July 2019, IoT could be significant to the department’s decision-making processes because the assortment of connected objects that comprise IoT could enable technology to gain the ability to sense, predict, and respond to DOD’s needs. The department could use computers to track, count, and analyze data from these objects in order to reduce waste, loss, and cost. DOD managers could also know whether these objects were running well or needed replacing, repairing, or recalling. IPv6 could support U.S. global technological competitiveness. According to a DOD presentation, the transition to IPv6 could allow the department to remain technologically competitive globally. Interest in IPv6 is gaining momentum around the world, particularly in parts of the world that have limited IPv4 address space to meet their industry and consumer communications needs. Regions that have limited IPv4 address space, such as Asia and Europe, have undertaken efforts to develop, test, and implement IPv6. For example, China has been aggressive at deploying IPv6. Japan has set up an IPv6 Promotion Council, using tax incentives to encourage research and adoption of IPv6 by its private sector. Europe has a task force that has the dual mandate of initiating country and regional IPv6 task forces across European states and seeking global cooperation around the world. Transitioning to IPv6 may preserve mission partner interoperability. According to an August 2017 report from DOD’s CIO, deploying IPv6 capabilities is essential to preserve interoperability with mission partners in the private sector and in other countries and to assure future access to technology. Since the pools of unassigned IPv4 addresses are exhausted, DOD’s mission partners may increasingly rely on IPv6 addresses in the future, furthering the department’s need to increase its IPv6 capabilities. Along with the potential benefits, the National Institute of Standards and Technology (NIST) has indicated that transitioning to IPv6 also could present challenges for organizations such as DOD. These challenges include the complexity added by dual IPv4 and IPv6 operations and the immaturity of IPv6 security products and processes. Complexity added by dual IPv4/IPv6 operations. DOD plans to deploy IPv6 while still supporting IPv4 for legacy applications, services, and clients. This will result in a dual protocol environment and increased complexity. With two protocols, DOD would have to ensure the proper functioning of two separate, but interrelated, networks instead of only one network. Further, hackers and other online adversaries could exploit either the department’s IPv4 or IPv6 network connections when launching cyberattacks. Immaturity of IPv6 security processes. While IPv6 could offer enhanced security, NIST states that its deployment could also lead to new challenges with respect to the types of threats facing an organization such as DOD. For example, organizations in the process of transitioning to IPv6 may lack robust IPv6 security controls and may have security staff members who lack an overall understanding of IPv6. This could allow attackers to exploit IPv6 assets or leverage IPv6 access to exploit IPv4 assets. While general security concepts are the same for both IPv4 and IPv6 protocols, it may take time and effort for transitioning organizations such as DOD to acquire the level of operational experience and practical deployment solutions that have been developed for IPv4 over the years. In August 2005, OMB issued a memorandum to federal CIOs specifying a series of IPv6 transition planning and implementation requirements and associated due dates for federal agencies to enable the use of IPv6. Specific to planning for the transition, the memorandum required agencies to assign an official to lead and coordinate IPv6 transition planning efforts, complete an inventory of IP-compliant devices and technologies, and complete an impact analysis comprised of a cost estimate and a risk analysis by specific due dates during fiscal year 2006. Table 1 lists the transition planning requirements and due dates defined in the OMB memorandum. Aware of the limitations of IPv4, DOD first began planning for the implementation of IPv6 in June 2003. At that time, the department’s CIO issued the memorandum, “Internet Protocol Version 6 (IPv6).” According to this memorandum, the department’s initial goal was to complete its transition to IPv6 by fiscal year 2008. Within a month of the issuance of DOD’s June 2003 memorandum, the department designated the Defense Research and Engineering Network (DREN) as the first DOD IPv6 pilot network. DREN, being a research and development network, is not connected to DOD’s operational networks, such as the department’s nonclassified IP router network, which transmits nonclassified operations traffic. According to a DOD report about the DREN pilot, the entire DREN wide-area network was routinely supporting end-to-end IPv6 traffic by July 2005. According to the department, DREN remains DOD’s only IPv6-enabled network. Further, DOD’s IG reported that the department had undertaken an initial transition effort that temporarily satisfied OMB’s August 2005 implementation requirement to demonstrate IPv6 on its infrastructure by the end of June 2008. Specifically, DOD was able to demonstrate IPv6 within the department’s Defense Information Systems Network. However, according to the report, the department’s IPv6 transition manager said DOD disabled IPv6 functionality due to a lack of trained personnel and potential security risks. We received additional information about why DOD disabled the IPv6 functionality, but we are not including it in the report due to the information being marked as for official use only. DOD began its next effort to plan and implement IPv6 in response to OMB’s subsequent 2010 guidance. In this guidance, OMB gave agencies—including DOD—requirements intended to further their transitions to IPv6. Two of these requirements stated that agencies were to: upgrade their public-facing IT servers and services (e.g., web and email) to IPv6 by the end of September 2012; and upgrade internal client applications that communicate with public internet servers and supporting enterprise networks to IPv6 by the end of September 2014. According to DOD, the department originally planned to meet the 2010 OMB requirements; however, it decided not to complete the upgrades due to security concerns. Again, we received additional information about the department’s security concerns; however, we are not including those details in this report because they were marked as for official use only. OMB’s IPv6 transition guidance requires federal agencies, such as DOD, to perform specific tasks as part of their IPv6 planning efforts. These tasks include: (1) assigning an official to lead and coordinate agency planning, (2) completing an inventory of existing IP-compliant devices and technologies, (3) developing a cost estimate (as part of an impact analysis), and (4) developing a risk analysis (as part of an impact analysis). Although these requirements were originally due in 2005 or 2006, they are still applicable; OMB has not replaced or rescinded them. Assigning an official to lead and coordinate agency planning can help agencies better manage their transition to IPv6. Specifically, a senior- level focal point to lead IPv6 transition efforts can provide assurance that the program is based on a coherent strategy and is well coordinated. A lead official may also help an agency avoid duplicative, overlapping, and fragmented efforts, which can result in avoidable additional costs. According to NIST, having an inventory of IP-compliant assets is crucial to IPv6 transition planning because it helps determine transition requirements and give an agency a clear understanding of the IP capabilities of the devices on the network. Specifically, an inventory helps determine which assets will transition to IPv6, the order in which assets will transition, the transition methods selected, and the security controls that would need to be implemented. Further, cost estimates are critical to decision-makers not only because they help establish budgets, but also because they are integral to determining and communicating a realistic view of likely cost and schedule outcomes that could be used to plan the work necessary to develop, produce, install, and support a program. As we have previously reported, without the ability to generate reliable cost estimates, agencies are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls. In addition, a risk analysis enables an agency to assess the significance of potential threats to the success of its transition to IPv6, as well as assess how those threats could be mitigated or avoided. Further, without a risk analysis to help the agency understand the potential threats and obstacles facing the IPv6 transition initiative, agencies run the risk of creating goals and plans that are too optimistic. According to DOD, the department began its most recent effort to transition to IPv6 in April 2017. As part of this effort, in November 2019, the department released its IPv6 implementation strategy, which articulates DOD’s overarching vision for its IPv6 transition initiative: to provide secure and reliable IPv6 services that enable innovation for competitive advantage. According to the strategy, DOD’s goals for the effort include implementing an interim solution of using both IPv4 and IPv6 and then planning for an eventual IPv6-only environment. As of March 2020, DOD had not yet completed three of the four selected transition planning requirements. Specifically, the department had completed the requirement of appointing an agency lead for its IPv6 transition efforts. However, it had not yet completed the three other requirements: complete an inventory, develop a cost estimate, and develop a risk analysis. Table 2 summarizes the status of DOD’s completion of the four selected OMB IPv6 transition planning requirements. DOD completed OMB’s requirement to have an official lead the IPv6 planning efforts by first assigning its Joint Information Environment Executive Committee responsibility for overseeing the department’s transition to IPv6 in August 2017. The committee, in turn, established the DOD IPv6 Working Group in November 2017 to coordinate and manage department-wide IPv6 planning, implementation, and testing. An official in the Office of the CIO serves as the chair of the working group, and, according to the group’s charter, is responsible for leading meetings, soliciting and prioritizing issue topics for review, and overseeing the coordination of working group activities supporting the department-wide transition to IPv6. However, DOD did not complete the requirement to develop an inventory of existing IP-compliant devices and technologies. In November 2005, DOD provided a memorandum to OMB that indicated that the department would not complete the inventory of IP-compliant devices and technologies; instead, DOD would continue following its existing transition plan, which did not require an inventory. Further, the DOD officials leading the IPv6 transition effort informed us that the department had not developed such an inventory and that it still does not plan to do so. The officials said that conducting a task of this size would be impractical given DOD’s size and the number of IP-compliant devices in the department. The officials leading the IPv6 transition also said that DOD has been mitigating the risk of not having an inventory by ensuring that the department has only been acquiring IPv6-capable IT devices since December 2009. However, while only acquiring IPv6-capable devices and applications could help the transition move forward, it would not be as complete as an inventory, given that an inventory would include technology purchased before December 2009. DOD also did not complete a cost estimate or a risk analysis. According to the DOD officials leading the IPv6 transition effort, the initiative was not a top priority until the CIO released the “Internet Protocol Version 6 Implementation Direction and Guidance” memorandum in February 2019. This memorandum, which was developed to guide the department’s IPv6 transition planning and implementation efforts, gave the transition initiative greater attention in the department. Prior to the issuance of the memorandum, the officials stated that they did not have sufficient resources to conduct the cost estimate and did not have enough understanding to complete the risk analysis. DOD officials leading the IPv6 transition effort explained that the department plans to develop these requirements by the end of May 2020; however, we have not received any documentation confirming this deadline. Until DOD develops an inventory of IP-compliant technologies and devices, a cost estimate, and a risk analysis, the department’s IPv6 transition initiative may have an increased likelihood of cost overruns, schedule delays, and security vulnerabilities. Specifically, not having an inventory of IP-compliant devices and technologies may lead to the department developing plans without being aware of all the system and infrastructure requirements necessary to successfully transition a large organization such as DOD to IPv6. Further, without a cost estimate, DOD may be making decisions without the benefit of relevant information on the initiative’s potential cost and schedule outcomes, thereby introducing unnecessary risk into the implementation process. Finally, by moving forward without a risk analysis, DOD increases the probability that it is not proactively managing potential threats that could disrupt the transition or introduce new IT security vulnerabilities. DOD’s February 2019 memorandum lists 35 required activities for transitioning to IPv6 that DOD’s various components or offices, such as the Office of the CIO, are to complete or work on during fiscal years 2019 through 2021. Of these 35 activities, 18 were to be completed prior to March 2020. However, DOD had not completed most of the required activities. Specifically, of the 18 activities that were to be completed by March 2020, the department had completed six and had not completed 12. In addition, the department had completed one of eight other activities that did not have specific due dates. Table 3 outlines the department’s seven IPv6 transition activities that had been completed as of March 2020. (See appendix I for a full list of DOD’s transition activities and their completion status.) One notable required activity that DOD completed was to develop a CIO- approved strategy to implement its transition to IPv6. DOD’s strategy outlines, among other things, the overall goals for the IPv6 transition initiative. These goals include implementing a network that is both IPv4 and IPv6 capable; planning for an IPv6-only environment; and establishing and optimizing training for IPv6. In addition, DOD’s Defense Information Systems Agency leveraged online training providers to offer on demand IPv6 training courses for network engineers and cybersecurity personnel. In addition to basic familiarization training for those new to IPv6, select training courses are labeled as being at the advanced level. However, the department had not completed 12 of 18 activities that were due prior to March 2020. Notably, DOD missed its original September 2019 deadline to enable IPv6 on all commercially hosted public facing unrestricted services. According to the department officials leading the IPv6 transition, DOD expects to be able to complete this task by the end of July 2021. The department also missed its original June 2019 deadline for identifying the public facing unrestricted services hosted by DOD. The officials leading the IPv6 transition initiative stated that DOD currently plans to complete this activity by May 2020. Other activities that were past due in March 2020 include: developing supplemental guidance for the acquisition of IPv6-capable products, updating and maintaining IPv6 standards and implementation profiles, and determining DOD’s cybersecurity architecture and posture impacts, among others. DOD officials leading the IPv6 transition effort stated that the department had not yet completed its required activities because the original time frames that the department had established were unrealistic. Although the activities were initially thought to have been reasonable, DOD adjusted the activities’ due dates after the department began executing the tasks and realized that it would take a large amount of work to accomplish their goals and complete the activities. One contributing factor for the unrealistic due dates is that DOD developed this list of required activities without the benefit of an inventory of IP-compliant devices and technologies, a cost estimate, or a risk analysis. Without completing these basic planning requirements, DOD significantly reduced the probability that it could have developed a realistic transition schedule. Addressing requirements would supply DOD with sources of meaningful information that would enable the department to develop realistic, detailed, and informed transition plans and time frames. While DOD’s current IPv6 transition effort is showing progress, the department has not completed most of OMB’s planning requirements. Notable signs of progress include the appointment of an official to lead the initiative and the development of an overarching strategy document that outlines the transition’s scope and goals. Nevertheless, DOD had not completed an inventory of IP-compliant technologies, a cost estimate, or a risk analysis before moving ahead with developing its February 2019 guidance and working against the guidance’s list of transition activities. The lack of an inventory is problematic due to the role that it should play in developing transition requirements. In addition, without a cost estimate to guide decision-makers, DOD’s current IPv6 transition plans could be based on unrealistic assumptions about costs and resource demands. Further, by moving forward without a risk analysis, DOD increases the probability that it is not proactively managing potential threats that could either disrupt the transition or introduce new IT security vulnerabilities. Completing these longstanding planning requirements would enable DOD to develop realistic plans with accurate transition requirements and proactive risk mitigation strategies, among other things. We are making three recommendations to DOD. The Secretary of Defense should direct the DOD CIO to complete a department-wide inventory of existing IP-compliant devices and technologies to help with planning efforts and requirements development for the transition to IPv6. (Recommendation 1) The Secretary of Defense should direct the DOD CIO to develop a cost estimate as described in OMB memorandum M-05-22 for the department’s transition to IPv6. (Recommendation 2) The Secretary of Defense should direct the DOD CIO to develop a risk analysis as described in OMB memorandum M-05-22 for the department’s transition to IPv6. (Recommendation 3) We provided a draft of this report to DOD and OMB for review and comment. In response, DOD agreed with two recommendations and disagreed with one recommendation that we made to the department. OMB did not state whether it agreed or disagreed with the report’s findings. In written comments, DOD stated that it agreed with our recommendations to develop a cost estimate and risk analysis for the department’s transition to IPv6 (Recommendations 2 and 3). The department said that it plans to complete both the cost estimate and the risk analysis by the end of May 2020. However, DOD stated that it did not agree with our recommendation to complete a department-wide inventory of existing IP-compliant devices and technologies (Recommendation 1). Specifically, the department referred to the draft IPv6 guidance that OMB developed in March 2020, stating that the draft guidance will rescind OMB’s fiscal year 2005 IPv6 guidance, which includes the inventory requirement. DOD also said that creating such an inventory would be impractical given the department’s size. It added that it has been mitigating the risk of not having an inventory by only acquiring IPv6-capable devices since December 2009. We acknowledge that OMB’s March 2020 draft IPv6 guidance, once finalized, would rescind its fiscal year 2005 IPv6 guidance. However, the draft guidance focuses on completing the operational deployments of IPv6, not on the initial key transition step of completing an inventory, as required in the 2005 guidance. The draft guidance also requests information on agencies’ completion of certain milestones using the percentage of IP-enabled devices that are IPv6-only as the metric. DOD’s completed inventory would be essential to accurately responding to OMB’s draft requirement. In addition, NIST’s current IPv6 transition guidance cites an inventory of IP devices as a key step in transitioning to IPv6 since such information would help identify requirements for transitioning, including which assets would transition and what security controls would be needed. As DOD has acknowledged, however, it has not yet completed an inventory. Accordingly, we believe that our recommendation that DOD complete a department-wide inventory of its existing IP-compliant devices and technologies is warranted. In addition, DOD provided a technical comment, which we incorporated as appropriate. DOD’s comments are reprinted in appendix II. In comments provided via email on May 8, 2020, an Associate General Counsel in OMB’s Office of General Counsel expressed OMB’s appreciation for the opportunity to review and comment on our draft report. The official did not state whether OMB agreed or disagreed with the report’s findings. OMB also provided a technical comment, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of DOD, and the Acting Director of OMB. In addition, the report is available at no change on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6240 or dsouzav@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In February 2019, the Department of Defense’s (DOD) Chief Information Officer (CIO) released “Internet Protocol Version 6 Implementation Direction and Guidance,” a memorandum containing guidance and a list of required implementation and planning activities for the department’s transition to Internet Protocol version 6 (IPv6). For each activity, the memorandum included information such as the component or office responsible for completing the work and a description of the work to be completed. Out of 35 total activities, seven were completed and 12 were past due as of March 2020. One key contributing factor behind the activities’ unrealistic deadlines was that the department developed this list of required activities without having completed key planning efforts, such as an inventory of IP-compliant devices and technologies, a cost estimate, or a risk analysis. Table 4 shows the status of the completion of the required transition activities as called for in the department’s guidance. In addition to the contact named above, Larry Crosland (Assistant Director), Meredith Raymond (Analyst in Charge), Amy Apostol, Chris Businsky, West Coile, Kristi Dorsey, Vernetta Marquis, and Evan Rapson made key contributions to this report.", "summary": "An internet protocol provides the addressing mechanism that defines how and where information moves across interconnected networks. Increased use of the internet has exhausted available IPv4 address space, spurring the adoption of its successor protocol, IPv6. OMB has required that agencies plan for transitioning from IPv4 to IPv6. Senate and House reports accompanying the 2020 National Defense Authorization Act included provisions for GAO to review DOD's IPv6 transition planning efforts. This report (1) identifies past DOD attempts to transition to IPv6, (2) examines the extent to which DOD has completed OMB's planning requirements for its current transition effort, and (3) identifies DOD's progress in completing its own IPv6 transition activities. To do so, GAO assessed DOD's IPv6 transition plans and documentation against OMB's requirements, reviewed DOD's planned IPv6 transition activities, and interviewed agency officials. The Department of Defense's (DOD) current initiative to transition to Internet Protocol version 6 (IPv6), which began in April 2017, follows at least two prior attempts to implement IPv6 that were halted by DOD. In one effort that began in approximately 2003, DOD initially did make progress implementing IPv6 on its systems, but then the department ended the effort due to security risks and a lack of personnel trained in IPv6. DOD initiated another attempt in response to 2010 OMB guidance. However, this initiative was terminated shortly thereafter, again due to security concerns. For its current initiative, DOD has not completed three of four longstanding OMB requirements (see table). Without an inventory, a cost estimate, or a risk analysis, DOD's plans have a high degree of uncertainty about the magnitude of work involved, the level of resources required, and the extent and nature of threats, including cybersecurity risks. In February 2019, DOD released its own IPv6 planning and implementation guidance that listed 35 required transition activities, 18 of which were due to be completed before March 2020. DOD completed six of the 18 activities as of March 2020. DOD officials acknowledged that the department's transition time frames were optimistic; they added that they had thought that the activities' deadlines were reasonable until they started performing the work. Without an inventory, a cost estimate, or a risk analysis, DOD significantly reduced the probability that it could have developed a realistic transition schedule. Addressing these basic planning requirements would supply DOD with needed information that would enable the department to develop realistic, detailed, and informed transition plans and time frames. GAO is making three recommendations to DOD to develop an inventory of IP compliant devices, an estimate of the IPv6 transition costs, and an analysis of IPv6 transition risk. DOD agreed with the recommendations to develop a cost estimate and risk analysis, but disagreed with the recommendation to develop an inventory of IP-compliant devices. Nevertheless, GAO believes the recommendation to develop an inventory is warranted.", "document_type": "gao"}
{"report": "As shown in table 1 the cost of counting the nation’s population has been escalating with each decade. The 2010 Census was the most expensive in U.S. history at about $12.3 billion, and was about 31 percent more costly than the $9.4 billion 2000 Census (in 2020 dollars). According to the Bureau, the total cost of the 2020 Census in October 2015 was estimated at $12.3 billion and in October 2017 that cost estimate grew to approximately $15.6 billion, approximately a $3 billion increase. Additionally, Bureau officials told us that while the estimated cost of the census had increased to $15.6 billion, it was nevertheless managing the 2020 Census to a lower cost of $14.1 billion. Bureau officials explained that the $14.1 billion includes all program costs and contingency funds to cover risks and general estimating uncertainty. The remaining $1.5 billion estimated cost is additional contingency for “unknown unknowns”—that is, low probability events that could cause massive disruptions—and several what-if scenarios such as an increase in the wage rate or additional supervisors needed to manage field operations. Moreover, as shown in figure 1, the average cost for counting a housing unit increased from about $16 in 1970 to around $92 in 2010 (in 2020 constant dollars). At the same time, the return of census questionnaires by mail (the primary mode of data collection) declined over this period from 78 percent in 1970 to 63 percent in 2010. Declining mail response rates has led to higher costs because the Bureau sends temporary workers to each non-responding household to obtain census data. Achieving a complete and accurate census has become an increasingly daunting task, in part, because the population is growing larger, more diverse, and more reluctant to participate in the enumeration. In many ways, the Bureau has had to invest substantially more resources each decade to conduct the enumeration. In addition to these external societal challenges that make achieving a complete count a daunting task, the Bureau also faces a number of internal management challenges that affect its capacity and readiness to conduct a cost-effective enumeration. Some of these issues—such as acquiring and developing IT systems and preparing reliable cost estimates—are long-standing in nature. At the same time, as the Bureau looks toward 2020, it also faces newly emerging and evolving uncertainties. For example, on March 26, 2018, the Secretary of Commerce announced his decision to add a question to the decennial census on citizenship status. On January 15, 2019, the U.S. District Court for the Southern District of New York ruled on one of a number of legal challenges to the Secretary’s decision. That ruling is being appealed, thus, leaving the use of the question uncertain. The U.S. Supreme Court is scheduled to begin hearing arguments in April 2019 regarding the addition of the citizenship question to the census form. In our prior work we have noted the risks associated with late changes of any nature to the design of the census if the Bureau is unable to fully test those changes under operational conditions. The Bureau also faced budgetary uncertainties that, according to the Bureau, led to the curtailment of testing in 2017 and 2018. However, the Consolidated Appropriations Act, 2018 appropriated for the Periodic Censuses and Programs account $2.544 billion, which more than doubles the Bureau’s request in the President’s Fiscal Year 2018 Budget of $1.251 billion. According to the explanatory statement accompanying the act, the appropriation, which is available through fiscal year 2020, is provided to ensure the Bureau has the necessary resources to immediately address any issues discovered during operational testing, and to provide a smoother transition between fiscal year 2018 and fiscal year 2019. The availability of those resources enabled the Bureau to continue preparations for the 2020 Census during the 35 days when appropriations lapsed for the Bureau. Moreover, the Consolidated Appropriations Act, 2019 appropriated for the Periodic Censuses and Programs account $3.551 billion. According to Bureau officials, this level of funding for fiscal year 2019 is sufficient to carry out 2020 Census activities as planned. Importantly, the census is conducted against a backdrop of immutable deadlines. In order to meet legally mandated reporting requirements, census activities need to take place at specific times and in the proper sequence. Thus, it is absolutely critical for the Bureau to stay on schedule. Figure 2 shows some dates for selected decennial events. The Bureau has begun to open its area census offices (ACO) for the 2020 Census. It has signed leases for all 248 ACOs, of which 39 of the offices will be open for the address canvassing operation set to begin in August 2019 where staff verifies the location of selected housing units. The remaining 209 offices will begin opening this fall. In 2010 the Bureau opened 494 census offices. The Bureau has been able to reduce its infrastructure because it is relying on automation to assign work and to record payroll. Therefore there is less paper—field assignments, maps, and daily payroll forms—to manually process. For the 2020 Census, the Bureau is refining its recruiting and hiring goals, but tentatively plans to recruit approximately 2.24 million applicants and hire nearly 500,000 temporary field staff from that applicant pool for two key operations: address canvassing, and nonresponse follow-up, where they visit households that do not return census forms to collect data in person. In 2010 the Bureau recruited 3.8 million applicants and hired 628,000 temporary workers to conduct the address canvassing and nonresponse follow-up field operations. According to Bureau officials, it has reduced the number of temporary staff it needs to hire because automation has made field operations more efficient and there is less paper. As of April 15, 2019, for its early operations efforts which includes hiring listers for address canvassing, the Bureau has processed approximately 264,000 applicants which represent 128.4 percent of its 205,000 recruiting goal. The Bureau is also in the process of hiring approximately 1,500 partnership specialists needed by June 2019 to help increase awareness and participation in the 2020 Census in minority communities and hard-to-reach populations. As of April 17, 2019, the Bureau has hired 467 partnership specialists, and another 329 applicants are waiting to have their background checks completed. Moreover, Bureau officials also stated that the current economic environment (i.e., the low unemployment rate compared to the economic environment of the 2010 Census) has not yet impacted their ability to recruit staff. The Bureau will continue to monitor the impact of low unemployment on its ability to recruit and hire at the local and regional levels. For the 2020 Census, the Bureau is significantly changing how it intends to conduct the census, in part by re-engineering key census-taking methods and infrastructure, and making use of new IT applications and systems. For example, the Bureau plans to offer an option for households to respond to the survey via the internet and enable field-based enumerators to use applications on mobile devices to collect survey data from households. To do this, the Bureau plans to utilize 52 new and legacy IT systems, and the infrastructure supporting them, to conduct the 2020 Census. A majority of these 52 systems have been tested during operational tests in 2017 and 2018. For example, the Bureau conducted its 2018 End-to- End test, which included 44 of the 52 systems and was intended to test all key systems and operations in a census-like environment to ensure readiness for the 2020 Census. Nevertheless, additional IT development and testing work needs to take place before the 2020 Census. Specifically, officials from the Bureau’s Decennial Directorate said they expect that the systems will need to undergo further development and testing due to, among other things, the need to add functionality that was not part of the End-to-End test, scale system performance to support the number of respondents expected during the 2020 Census, and address system defects identified during the 2018 End-to-End test. To prepare the systems and technology for the 2020 Census, the Bureau is also relying on significant contractor support. For example, it is relying on contractors to develop a number of systems and components of the IT infrastructure, including the IT platform that is intended to be used to collect data from households responding via the internet and telephone, and for non-response follow-up activities. Contractors are also deploying the IT and telecommunications hardware in the field offices and providing device-as-a-service capabilities by procuring the mobile devices and cellular service to be used for non-response follow-up. In addition to the development of technology, the Bureau is relying on a technical integration contractor to integrate all of the key systems and infrastructure. The contractor’s work is expected to include, among other things, evaluating the systems and infrastructure and acquiring the infrastructure (e.g., cloud or data center) to meet the Bureau’s scalability and performance needs; integrating all of the systems; and assisting with technical, performance and scalability, and operational testing activities. In February 2017, we added the 2020 Decennial Census as a high-risk area needing attention from Congress and the executive branch. This was due to significant risks related to, among other things, innovations never before used in prior enumerations, the acquisition and development of IT systems, and expected escalating costs. Among other things, we reported that the commitment of top leadership was needed to ensure the Bureau’s management, culture, and business practices align with a cost-effective enumeration. We also stressed that the Bureau needed to rigorously test census-taking activities; ensure that scheduling adheres to best practices; improve its ability to manage, develop, and secure its IT systems; and have better oversight and control over its cost estimation process. Our experience has shown that the key elements needed to make progress toward being removed from the High-Risk List are top-level attention by the administration and agency leaders grounded in the five criteria for removal, as well as any needed congressional action. The five criteria for removal that we identified in November 2000 are as follows: Leadership Commitment. The agency has demonstrated strong commitment and top leadership support. Capacity. The agency has the capacity (i.e., people and resources) to resolve the risk(s). Action Plan. A corrective action plan exists that defines the root causes and solutions, and that provides for substantially completing corrective measures, including steps necessary to implement solutions we recommended. Monitoring. A program has been instituted to monitor and independently validate the effectiveness and sustainability of corrective measures. Demonstrated Progress. The agency has demonstrated progress in implementing corrective measures and in resolving the high-risk area. These five criteria form a road map for efforts to improve, and ultimately address, high-risk issues. Addressing some of the criteria leads to progress, while satisfying all of the criteria is central to removal from the list. As we reported in the March 2019 high-risk report, the Bureau’s efforts to address the risks and challenges for the 2020 Census had fully met one of the five criteria for removal from the High-Risk List—leadership commitment—and partially met the other four, as shown in figure 3. Additional details about the status of the Bureau’s efforts to address this high-risk area are discussed later in this statement. The 2020 Census is on our list of high-risk programs because, among other things, (1) innovations never before used in prior enumerations are not expected to be fully tested, (2) the Bureau continues to face challenges in implementing IT systems, (3) the Bureau faces significant cybersecurity risks to its systems and data, and (4) the Bureau’s cost estimate for the 2020 Census was unreliable. If not sufficiently addressed, these risks could adversely impact the cost and quality of the enumeration. Moreover, the risks are compounded by other factors that contribute to the challenge of conducting a successful census, such as the nation’s increasingly diverse population and concerns over personal privacy. The basic design of the enumeration—mail out and mail back of the census questionnaire with in-person follow-up for non-respondents—has been in use since 1970. However, a lesson learned from the 2010 Census and earlier enumerations is that this traditional design is no longer capable of cost-effectively counting the population. In response to its own assessments, our recommendations, and studies by other organizations, the Bureau has fundamentally re-examined its approach for conducting the 2020 Census. Specifically, its plan for 2020 includes four broad innovation areas: re-engineering field operations, using administrative records, verifying addresses in-office, and developing an internet self-response option (see table 2). If they function as planned, the Bureau initially estimated that these innovations could result in savings of over $5 billion (in 2020 constant dollars) when compared to its estimates of the cost for conducting the census with traditional methods. However, in June 2016, we reported that the Bureau’s initial life-cycle cost estimate developed in October 2015 was not reliable and did not adequately account for risk. As discussed earlier in this statement, the Bureau has updated its estimate from $12.3 billion and now estimates a life-cycle cost of $15.6 billion, which would result in a smaller potential savings from the innovative design than the Bureau originally estimated. According to the Bureau, the goal of the cost estimate increase was to ensure quality was fully addressed. While the planned innovations could help control costs, they also introduce new risks, in part, because they include new procedures and technology that have not been used extensively in earlier decennials, if at all. Our prior work has shown the importance of the Bureau conducting a robust testing program, including the 2018 End-to-End test. Rigorous testing is a critical risk mitigation strategy because it provides information on the feasibility and performance of individual census-taking activities, their potential for achieving desired results, and the extent to which they are able to function together under full operational conditions. To address some of these challenges we have made numerous recommendations aimed at improving reengineered field operations, using administrative records, verifying the accuracy of the address list, and securing census responses via the internet. The Bureau has held a series of operational tests since 2012, but according to the Bureau, it scaled back its most recent field tests because of funding uncertainties. For example, the Bureau canceled the field components of the 2017 Census Test including non-response follow-up, a key census operation. In November 2016, we reported that the cancelation of the 2017 Census Test was a lost opportunity to test, refine, and integrate operations and systems, and that it put more pressure on the 2018 End-to-End test to demonstrate that enumeration activities will function under census-like conditions as needed for 2020. However, in May 2017, the Bureau scaled back the operational scope of the 2018 End-to-End test and, of the three planned test sites, only the Rhode Island site would fully implement the 2018 End-to-End test. The Washington and West Virginia sites would test just one field operation. In addition, due to budgetary concerns, the Bureau decided to remove three coverage measurement operations (and the technology that supports them) from the scope of the test. However, removal of the coverage measurement operations did not affect testing of the delivery of apportionment or redistricting data. Without sufficient testing, operational problems can go undiscovered and the opportunity to improve operations will be lost, in part because the 2018 End-to-End test was the last opportunity to demonstrate census technology and procedures across a range of geographic locations, housing types, and demographic groups under decennial-like conditions prior to the 2020 Census. To manage risk to the census, the Bureau has developed hundreds of mitigation and contingency plans. To maximize readiness for the 2020 Census, it will also be important for the Bureau to prioritize among its mitigation and contingency strategies those that will deliver the most cost-effective outcomes for the census. We reported on the 2018 End-to-End test in December 2018 and noted that the Bureau had made progress addressing prior test implementation issues but still faced challenges. As the Bureau studies the results of its testing to inform the 2020 Census, it will be important that it addresses key program management issues that arose during implementation of the test. Namely, by not aligning the skills, responsibilities, and information flows for the first-line supervisors during field data collection, the Bureau limited its role in support of enumerators within the re-engineered field operation. The Bureau also lacked mid-operation training or guidance, which, if implemented in a targeted, localized manner, could have further helped enumerators navigate procedural modifications and any commonly encountered problems when enumerating. It will be important for the Bureau to prioritize its mitigation strategies for these implementation issues so that it can maximize readiness for the 2020 Census. We have previously reported that the Bureau faces challenges in managing and overseeing IT programs, systems, and contractors supporting the 2020 Census. Specifically, we have noted challenges in the Bureau’s efforts to manage, among other things, the schedules and contracts for its systems. As a result of these challenges, the Bureau is at risk of being unable to fully implement the systems necessary to support the 2020 Census and conduct a cost-effective enumeration. To help improve its implementation of IT for the 2020 Census, the Bureau recently revised its systems development and testing schedule. Specifically, in October 2018, the Bureau organized the development and testing schedule for its 52 systems into 16 operational deliveries. Each of the 16 operational deliveries has milestone dates for, among other things, development, performance and scalability testing, and system deployment. According to Bureau officials in the Decennial Directorate, the schedule was revised, in part, due to schedule management challenges experienced, and lessons learned, while completing development and testing during the 2018 End-to-End test. The Bureau has made initial progress in executing work against its revised schedule. For example, the Bureau completed development for the systems in the first operational delivery—for 2020 Census early operations preparations—in July 2018, and deployed these systems into production in October 2018. However, our current work has determined that the Bureau is at risk of not meeting several near-term systems testing milestones. As of April 2019, six systems that are expected to be used in a total of two operational deliveries are at risk of not meeting milestone dates which would signal that the systems have completed development and are ready for testing. These six systems are needed for, among other things, field assignment management and worker performance tracking during address canvassing, data collection for operations, business and support automation, and customer support during self-response. According to Bureau documentation, these systems were at risk due, in part, to the lack of finalized system requirements and specifications. Figure 4 presents an overview of the status for all 16 operational deliveries, as of April 2019. The at-risk systems previously discussed add uncertainty to a highly compressed time frame over the next 4 months. Importantly, between April and August 2019, the Bureau is expected to begin integration testing for the systems in seven operational deliveries, including internet self- response and non-response follow-up. Officials from the Bureau’s integration contractor noted concern that the current schedule leaves little room for any delays in completing the remaining development and testing activities. In addition to managing the compressed testing time frames, the Bureau also has to quickly finalize plans related to its IT infrastructure. For example, in March 2019, the Bureau’s technical integration contractor stated that it needed the Bureau to obtain approval from federal partners for its Trusted Internet Connection or finalize alternative plans in order to complete performance and scalability testing in a timely manner. As of mid-April 2019, the Bureau stated that it was still awaiting final approval. Given that these plans may impact systems being tested this summer or deployed into production for the address canvassing operation in August 2019, it is important that the Bureau quickly addresses this matter. Our past reporting noted that the Bureau faced significant challenges in managing its schedule for system development and testing that occurred in 2017 and 2018. We reported that while the Bureau had continued to make progress in developing and testing IT systems for the 2020 Census, it had experienced delays in developing systems to support the 2018 End-to-End test. These delays compressed the time available for system and integration testing and for security assessments. In addition, several systems experienced problems during the test. We noted then, and reaffirm now, that continued schedule management challenges may compress the time available for the remaining system and integration testing and increase the risk that systems may not function or be as secure as intended. The Bureau has acknowledged that it faces risks to the implementation of its systems and technology. As of March 2019, the Bureau had identified about 330 active risks for the 2020 Census program, through its risk management process, including 20 high risks that may have substantial technical and schedule impacts if realized. Taken together, these risks represent a cross-section of issues, such as the effects of late changes to technical requirements, the need to ensure adequate time for system development and performance and scalability testing, contracting issues, privacy risks, and skilled staffing shortages. Going forward, it will be important that the Bureau effectively manages these risks to better ensure that it meets near-term milestones for system development and testing, and is ready for the major operations of the 2020 Census. The risks to IT systems supporting the federal government and its functions, including conducting the 2020 Census, are increasing as security threats continue to evolve and become more sophisticated. These risks include insider threats from witting or unwitting employees, escalating and emerging threats from around the globe, and the emergence of new and more destructive attacks. Underscoring the importance of this issue, we have designated information security as a government-wide high-risk area since 1997 and, in our most recent biennial report to Congress, ensuring the cybersecurity of the nation was one of nine high-risk areas that we reported needing especially focused executive and congressional attention. Our prior and ongoing work has identified significant challenges that the Bureau faces in securing systems and data for the 2020 Census. Specifically, the Bureau has faced challenges related to completing security assessments, addressing security weaknesses, resolving cybersecurity recommendations from DHS, and addressing numerous other cybersecurity concerns (such as phishing). Federal law specifies requirements for protecting federal information and information systems, such as those systems to be used in the 2020 Census. Specifically, the Federal Information Security Management Act of 2002 and the Federal Information Security Modernization Act of 2014 (FISMA) require executive branch agencies to develop, document, and implement an agency-wide program to provide security for the information and information systems that support operations and assets of the agency. In accordance with FISMA, National Institute of Standards and Technology (NIST) guidance, and Office of Management and Budget (OMB) guidance, the Bureau’s Office of the Chief Information Officer (CIO) established a risk management framework. This framework requires system developers to ensure that each of the Bureau’s systems undergoes a full security assessment, and that system developers remediate critical deficiencies. According to the Bureau’s risk management framework, the systems expected to be used to conduct the 2020 Census will need to have complete security documentation (such as system security plans) and an approved authorization to operate prior to their use. Currently, according to the Bureau’s Office of the CIO: Fourteen of the 52 systems have authorization to operate, and will not need to be reauthorized before they are used in the 2020 Census Thirty-two of the 52 systems have authorization to operate, and may need to be reauthorized before they are used in the 2020 Census Six of the 52 systems do not have authorization to operate, and will need to be authorized before they are used in the 2020 Census. Figure 5 summarizes the authorization to operate status for the systems being used in the 2020 Census, as reported by the Bureau in April 2019. As we have previously reported, while large-scale technological changes (such as internet self-response) increase the likelihood of efficiency and effectiveness gains, they also introduce many cybersecurity challenges. The 2020 Census also involves collecting personally identifiable information (PII) on over a hundred million households across the country, which further increases the need to properly secure these systems. Thus, it will be important that the Bureau provides adequate time to perform these security assessments, completes them in a timely manner, and ensures that risks are at an acceptable level before the systems are deployed. We have ongoing work examining how the Bureau plans to address both internal and external cyber threats, including its efforts to complete system security assessments and resolve identified weaknesses. FISMA requires that agency-wide information security programs include a process for planning, implementing, evaluating, and documenting remedial actions (i.e., corrective actions) to address any deficiencies in the information security policies, procedures, and practices of the agency. Agencies must establish procedures to reasonably ensure that all information security control weaknesses, regardless of how or by whom they are identified, are addressed through the agency’s remediation processes. For each identified control weakness, the agency is required to develop and implement a plan of actions and milestones (POA&M) based on findings from security control assessments, security impact analyses, continuous monitoring of activities, audit reports, and other sources. Additionally, the Bureau’s framework requires that security assessment findings that need to be remediated are to be tracked as POA&Ms. These POA&Ms are expected to provide a description of the vulnerabilities identified during the security assessment that resulted from a control weakness. As of March 2019, the Bureau had over 500 open POA&Ms to remediate for issues identified during security assessment activities, including ongoing continuous monitoring. Of these open POA&Ms, 247 (or about 48 percent) were considered “high-risk” or “very high-risk.” While the Bureau established POA&Ms for addressing these identified security control weaknesses, it did not always complete remedial actions in accordance with its established deadlines. For example, of the 247 open “high-risk” or “very high-risk” POA&Ms we reviewed through March 2019, the Bureau identified 115 as being delayed. Further, 70 of the 115 had missed their scheduled completion dates by 60 or more days. In addition, the number of open “high-risk” or “very high-risk” POA&Ms that the Bureau identified as delayed has substantially increased since June 2018, as shown in figure 6. According to the Bureau, these POA&Ms were identified as delayed due to technical challenges or resource constraints to remediate and close them. However, without resolving identified vulnerabilities in a timely manner, the Bureau faces an increased risk, as continuing opportunities exist for unauthorized individuals to exploit these weaknesses and gain access to sensitive information and systems. The Bureau is working with federal and industry partners, including the Department of Homeland Security, to support the 2020 Census cybersecurity efforts. Specifically, the Bureau is working with DHS to ensure a scalable and secure network connection for the 2020 Census respondents (e.g., virtual Trusted Internet Connection with the cloud), improve its cybersecurity posture (e.g., improve risk management processes and procedures), and to strengthen its response to potential cyber threats (e.g., federal cyber incident coordination). Federal law describes practices for strengthening cybersecurity by documenting or tracking corrective actions. As previously mentioned, FISMA requires executive branch agencies to establish a process for planning, implementing, evaluating, and documenting remedial actions to address any deficiencies in their information security policies, procedures, and practices. GAO’s internal control standards also state that agencies should establish effective internal control monitoring that includes a process to promptly resolve the findings of audits and other reviews. Specifically, agencies should document and complete corrective actions to remediate identified deficiencies on a timely basis. This would include correcting identified deficiencies or demonstrating that the findings and recommendations do not warrant agency action. Since January 2017, DHS has been providing cybersecurity assistance (including issuing recommendations) to the Bureau in preparation for the 2020 Census, and the Bureau has reported making progress in addressing those recommendations. Specifically, DHS has been providing cybersecurity assistance to the Bureau in five areas: management coordination and executive support, including a cybersecurity threat intelligence and information sharing enhancement through, among other things, a DHS cyber threat briefing to the Bureau’s leadership; network and infrastructure security and resilience, including National Cybersecurity Protection System (also called EINSTEIN) support; incident response and management readiness through a Federal Incident Response Evaluation assessment; and risk management and vulnerability assessments on specific targets provided by the Bureau. In the last 2 years, as a result of these activities, DHS has provided 17 recommendations for the Bureau to strengthen its cybersecurity efforts. Among other things, the recommendations pertained to strengthening incident management capabilities, penetration testing and web application assessments of select systems, and phishing assessments to gain access to sensitive PII. Due to the sensitive nature of the recommendations, we are not identifying the specific recommendations or specific findings associated with them in this statement. As of February 2019, the Bureau had fully completed actions to address three recommendations, needed to further improve on actions taken for one recommendation it indicated had been completed, and needed to complete actions in progress for the remaining 13 recommendations (as summarized in table 3). However, the Bureau had not established a formal process for documenting, tracking, and completing corrective actions for all the recommendations provided by DHS. To the Bureau’s credit, it had incorporated the corrective actions associated with the three completed recommendations into its formal process used for tracking POA&Ms, which includes identifying remediation activities, resources required, milestones, and completion dates. The Bureau did not incorporate the remaining 14 recommendations into the POA&M process. Instead, in November 2018, the Bureau created an informal document to track the 17 DHS recommendations, but this document does not consistently include details such as the resources required, expected completion date, or whether the recommendations do not warrant agency action. Until the Bureau implements a formal process for tracking and implementing appropriate corrective actions to remediate identified cybersecurity weaknesses from DHS, and addresses the identified deficiencies, it faces an increased likelihood that these weaknesses will go uncorrected and may be exploited to cause harm to agency’s 2020 Census IT systems and gain access to sensitive respondent data. Implementing a formal process would also help to ensure that DHS’s efforts result in improvements to the Bureau’s cybersecurity posture. The Bureau faces other significant cybersecurity challenges in addition to those previously discussed. More specifically, we previously reported that the extensive use of IT systems to support the 2020 Census redesign may help increase efficiency, but that this redesign introduces critical cybersecurity challenges. These challenges include those related to the following: Phishing. We have previously reported that advanced persistent threats may be targeted against social media web sites used by the federal government. In addition, attackers may use social media to collect information and launch attacks against federal information systems through social engineering, such as phishing. Phishing is a digital form of social engineering that uses authentic-looking, but fake, emails, websites, or instant messages to get users to download malware, open malicious attachments, or open links that direct them to a website that requests information or executes malicious code. Phishing attacks could target respondents, as well as Bureau employees and contractors. The 2020 Census will be the first one in which respondents will be heavily encouraged to respond via the internet. This will likely increase the risk that cyber criminals will use phishing in an attempt to steal personal information. Disinformation from social media. We previously reported that one of the Bureau’s key innovations for the 2020 Census is the large-scale implementation of an internet self-response option. The Bureau is encouraging the public to use the internet self-response option through expanded use of social media. However, the public perception of the Bureau’s ability to adequately safeguard the privacy and confidentiality of the 2020 Census internet self-responses could be influenced by disinformation spread through social media. According to the Bureau, if a substantial segment of the public is not convinced that the Bureau can safeguard public response data against data breaches and unauthorized use, then response rates may be lower than projected, leading to an increase in cases for follow-up and subsequent cost increases. Ensuring that individuals gain only limited and appropriate access to 2020 Census data. The Bureau plans to enable a public- facing website and Bureau-issued mobile devices to collect PII (e.g., name, address, and date of birth) from the nation’s entire population— estimated to be over 300 million. In addition, the Bureau is planning to obtain and store administrative records containing PII from other government agencies to help augment information that enumerators did not collect. The number of reported security incidents involving PII at federal agencies has increased dramatically in recent years. Because of these challenges, we have recommended, among other things, that federal agencies improve their response to information security incidents and data breaches involving PII, and consistently develop and implement privacy policies and procedures. Accordingly, it will be important for the Bureau to ensure that only respondents and Bureau officials are able to gain access to this information, and enumerators and other employees only have access to the information needed to perform their jobs. Ensuring adequate control in a cloud environment. The Bureau has decided to use cloud solutions as a key component of the 2020 Census IT infrastructure. We have previously reported that cloud computing has both positive and negative information security implications and, thus, federal agencies should develop service-level agreements with cloud providers. These agreements should specify, among other things, the security performance requirements— including data reliability, preservation, privacy, and access rights— that the service provider is to meet. Without these safeguards, computer systems and networks, as well as the critical operations and key infrastructures they support, may be lost; information—including sensitive personal information—may be compromised; and the agency’s operations could be disrupted. Ensuring contingency and incident response plans are in place to encompass all of the IT systems to be used to support the 2020 Census. Because of the brief time frame for collecting data during the 2020 Census, it is especially important that systems are available for respondents to ensure a high response rate. Contingency planning and incident response help ensure that, if normal operations are interrupted, network managers will be able to detect, mitigate, and recover from a service disruption while preserving access to vital information. Implementing important security controls, including policies, procedures, and techniques for contingency planning and incident response, helps to ensure the confidentiality, integrity, and availability of information and systems, even during disruptions of service. Without contingency and incident response plans, system availability might be impacted and result in a lower response rate. The Bureau’s CIO has acknowledged these cybersecurity challenges and is working to address them, according to Bureau documentation. In addition, we have ongoing work looking at many of these challenges, including the Bureau’s plans to protect PII, use a cloud-based infrastructure, and recover from security incidents and other disasters. Since 2015, the Bureau has made progress in improving its ability to develop a reliable cost estimate. We have reported on the reliability of the $12.3 billion life-cycle cost estimate released in October 2015 and the $15.6 billion revised cost estimate released in October 2017. In 2016 we reported that the October 2015 version of the Bureau’s life-cycle cost estimate for the 2020 Census was not reliable. Specifically, we found that the 2020 Census life-cycle cost estimate partially met two of the characteristics of a reliable cost estimate (comprehensive and accurate) and minimally met the other two (well-documented and credible). We recommended that the Bureau take specific steps to ensure its cost estimate meets the characteristics of a high-quality estimate. The Bureau agreed and has taken action to improve the reliability of the cost estimate. In August 2018 we reported that while improvements had been made, the Bureau’s October 2017 cost estimate for the 2020 Census did not fully reflect all the characteristics of a reliable estimate. (See figure 7.) In order for a cost estimate to be deemed reliable as described in GAO’s Cost Estimating and Assessment Guide and thus, to effectively inform 2020 Census annual budgetary figures, the cost estimate must meet or substantially meet the following four characteristics: Well-Documented. Cost estimates are considered valid if they are well-documented to the point they can be easily repeated or updated and can be traced to original sources through auditing, according to best practices. Accurate. Accurate estimates are unbiased and contain few mathematical mistakes. Credible. Credible cost estimates must clearly identify limitations due to uncertainty or bias surrounding the data or assumptions, according to best practices. Comprehensive. To be comprehensive an estimate should have enough detail to ensure that cost elements are neither omitted nor double-counted, and all cost-influencing assumptions are detailed in the estimate’s documentation, among other things, according to best practices. The 2017 cost estimate only partially met the characteristic of being well- documented. In general, some documentation was missing, inconsistent, or difficult to understand. Specifically, we found that source data did not always support the information described in the basis of estimate document or could not be found in the files provided for two of the Bureau’s largest field operations: Address Canvassing and Non- Response Follow-Up. We also found that some of the cost elements did not trace clearly to supporting spreadsheets and assumption documents. Failure to document an estimate in enough detail makes it more difficult to replicate calculations, or to detect possible errors in the estimate; reduces transparency of the estimation process; and can undermine the ability to use the information to improve future cost estimates or even to reconcile the estimate with another independent cost estimate. The Bureau told us it would continue to make improvements to ensure the estimate is well- documented. The 2017 life-cycle cost estimate includes significantly higher costs than those included in the 2015 estimate. The largest increases occurred in the Response, Managerial Contingency, and Census/Survey Engineering categories. For example, increased costs of $1.3 billion in the response category (costs related to collecting, maintaining, and processing survey response data) were in part due to reduced assumptions for self- response rates, leading to increases in the amount of data collected in the field, which is more costly to the Bureau. Contingency allocations increased overall from $1.35 billion in 2015 to $2.6 billion in 2017, as the Bureau gained a greater understanding of risks facing the 2020 Census. Increases of $838 million in the Census/Survey Engineering category were due mainly to the cost of an IT contract for integrating decennial survey systems that was not included in the 2015 cost estimate. Bureau officials attribute a decrease of $551 million in estimated costs for Program Management to changes in the categorization of costs associated with risks. Specifically, in the 2017 version of the estimate, estimated costs related to program risks were allocated to their corresponding work breakdown structure (WBS) element. Figure 8 shows the change in cost by WBS category for 2015 and 2017. More generally, factors that contributed to cost fluctuations between the 2015 and 2017 cost estimates include: Changes in assumptions. Among other changes, a decrease in the assumed rate for self-response from 63.5 percent in 2015 to 60.5 percent in 2017 increased the cost of collecting responses from nonresponding housing units. Improved ability to anticipate and quantify risk. In general, contingency allocations designed to address the effects of potential risks increased overall from $1.3 billion in 2015 to $2.6 billion in 2017. An overall increase in IT costs. IT cost increases, totaling $1.59 billion, represented almost 50 percent of the total cost increase from 2015 to 2017. More defined contract requirements. Bureau documents described an overall improvement in the Bureau’s ability to define and specify contract requirements. This resulted in updated estimates for several contracts, including for the Census Questionnaire Assistance contract. However, while the Bureau has been able to better quantify risk; in August 2018 we also reported that the Secretary of Commerce included a contingency amount of about $1.2 billion in the 2017 cost estimate to account for what the Bureau refers to as “unknown unknowns.” According to Bureau documentation these include such risks as natural disasters or cyber attacks. The Bureau provides a description of how the risk contingency for “unknown unknowns” is calculated; however, this description does not clearly link calculated amounts to the risks themselves. Thus, only $14.4 billion of the Bureau’s $15.6 billion cost estimate has justification. According to Bureau officials, the cost estimate remains at $15.6 billion, but they are managing the 2020 Census at a lower level of funding—$14.1 billion and, at this time, do not plan to request funding for the $1.2 billion contingency fund for unknown unknowns or $369 million in funding for selected discrete program risks for what-if scenarios such as an increase in the wage rate or additional supervisors needed to manage field operations. Instead of requesting funding for these contingencies upfront the Bureau plans to work with OMB and Commerce to request additional funds, if the need arises. According to Bureau officials they anticipate that the remaining $1.1 billion in contingency funding included in the $14.1 billion will be sufficient to carry out the 2020 Census. In June 2016 we recommended the Bureau improve control over how risk and uncertainty are accounted for. This prior recommendation remains valid given the life-cycle cost estimate still includes the $1.2 billion unjustified contingency fund for “unknown unknowns”. Moreover, given the cost growth between 2015 and 2017 it will be important for the Bureau to monitor cost in real-time, as well as, document, explain and review variances between planned and actual cost. In August 2018 we reported that the Bureau had not been tracking variances between estimated life-cycle costs and actual expenses. Tools to track variance enable management to measure progress against planned outcomes and will help inform the 2030 Census cost estimate. Bureau officials stated that they already have systems in place that can be adapted for tracking estimated and actual costs. We will continue to monitor the status of the tracking system. According to Bureau officials it plans to release an updated version of the 2020 Census life-cycle estimate in the spring of 2019. To ensure that future updates to the life-cycle cost estimate reflect best practices, it will be important for the Bureau to implement our recommendation related to the cost estimate. The difficulties facing the Bureau’s preparation for the decennial census in such areas as planning and testing; managing and overseeing IT programs, systems, and contractors supporting the enumeration; developing reliable cost estimates; prioritizing decisions; managing schedules; and other challenges, are symptomatic of deeper organizational issues. Following the 2010 Census, a key lesson learned for 2020 that we identified was ensuring that the Bureau’s organizational culture and structure, as well as its approach to strategic planning, human capital management, internal collaboration, knowledge sharing, capital decision- making, risk and change management, and other internal functions are aligned toward delivering more cost-effective outcomes. The Bureau has made improvements over the last decade, and continued progress will depend in part on sustaining efforts to strengthen risk management activities, enhancing systems testing, bringing in experienced personnel to key positions, implementing our recommendations, and meeting regularly with officials from its parent agency, Commerce. Going forward, we have reported that the key elements needed to make progress in high-risk areas are top-level attention by the administration and agency officials to (1) leadership commitment, (2) ensuring capacity, (3) developing a corrective action plan, (4) regular monitoring, and (5) demonstrated progress. Although important steps have been taken in at least some of these areas, overall, far more work is needed. We discuss three of five areas below. The Secretary of Commerce has successfully demonstrated leadership commitment. For example, the Bureau and Commerce have strengthened this area with executive-level oversight of the 2020 Census by holding regular meetings on the status of IT systems and other risk areas. In addition, in 2017 Commerce designated a team to assist senior Bureau management with cost estimation challenges. Moreover, on January 2, 2019, a new Director of the Census Bureau took office, a position that had been vacant since June 2017. With regard to capacity, the Bureau has improved the cost estimation process of the decennial when it established guidance including: roles and responsibilities for oversight and approval of cost estimation processes, procedures requiring a detailed description of the steps taken to produce a high-quality cost estimate, and a process for updating the cost estimate and associated documents over the life of a project. However, the Bureau continues to experience skills gaps in the government program management office overseeing the $886 million contract for integrating the IT systems needed to conduct the 2020 Census. Specifically, as of February 2019, 15 of 44 positions in this office were vacant. For the monitoring element, we found to track performance of decennial census operations, the Bureau relied on reports to track progress against pre-set goals for a test conducted in 2018. According to the Bureau, these same reports will be used in 2020 to track progress. However, the Bureau’s schedule for developing IT systems during the 2018 End-to-End test experienced delays that compressed the time available for system testing, integration testing, and security assessments. These schedule delays contributed to systems experiencing problems after deployment, as well as cybersecurity challenges. In the months ahead, we will continue to monitor the Bureau’s progress in addressing each of the five elements essential for reducing the risk to a cost-effective enumeration. Over the past several years we have issued numerous reports that underscored the fact that, if the Bureau was to successfully meet its cost savings goal for the 2020 Census, the agency needed to take significant actions to improve its research, testing, planning, scheduling, cost estimation, system development, and IT security practices. As of April 2019, we have made 97 recommendations related to the 2020 Census. The Bureau has implemented 72 of these recommendations, 24 remain open, and one recommendation was closed as not implemented. Of the 24 open recommendations, 11 were directed at improving the implementation of the innovations for the 2020 Census. Commerce generally agreed with our recommendations and is taking steps to implement them. Moreover, in April 2018 we designated 15 recommendations as “priority.” Priority recommendations are those recommendations that we believe warrant priority attention from heads of key departments and agencies. Eight of these 15 priority recommendations have been closed as implemented over the past year. On July 19, 2018, in response to our April 2018 letter calling his attention to our priority recommendations, the Commerce Secretary concurred that there was still much work to be done, and that the number of our priority recommendations concerning the 2020 Census was reflective of Commerce’s focus on ensuring a successful census in 2020. On April 23, 2019, we sent an updated priority recommendation letter to the Commerce Secretary that included five new recommendations from our recent work and also reflected the department’s progress on implementing past recommendations. We believe that attention to these recommendations is essential for a cost-effective enumeration. The recommendations included implementing reliable cost estimation and scheduling practices in order to establish better control over program costs, as well as taking steps to better position the Bureau to develop an internet response option for the 2020 Census. In addition to our recommendations, to better position the Bureau for a more cost-effective enumeration, on March 18, 2019, we met with OMB, Commerce, and Bureau officials to discuss the Bureau’s progress in reducing the risks facing the census. We also meet regularly with Bureau officials and managers to discuss the progress and status of open recommendations related to the 2020 Census, which has resulted in Bureau actions in recent months leading to closure of some recommendations. We are encouraged by this commitment by Commerce and the Bureau in addressing our recommendations. Implementing our recommendations in a complete and timely manner is important because it could improve the management of the 2020 Census and help to mitigate continued risks. In conclusion, while the Bureau has made progress in revamping its approach to the census, it faces considerable challenges and uncertainties in implementing key cost-saving innovations and ensuring they function under operational conditions; managing the development and testing of its IT systems; ensuring the cybersecurity of its systems and data; and developing a quality cost estimate for the 2020 Census and preventing further cost increases. For these reasons, the 2020 Census is a GAO high-risk area. Regarding cybersecurity, the Bureau’s involvement of DHS to improve its cybersecurity posture, including cyber threat briefings and vulnerability assessments, is a positive step forward. However, the Bureau’s corrective actions to address its high-risk and very high-risk security weaknesses are frequently delayed—often for months—which increases the risk that these weaknesses could be exploited to cause harm to the agency’s systems. In addition, the Bureau’s process for addressing DHS’s cybersecurity recommendations has shortcomings, which increases the risk that the underlying deficiencies identified by DHS may be exploited to gain access to the Bureau’s systems and sensitive data. Going forward, continued management attention and oversight will be vital for ensuring that risks are managed, preparations stay on track, and the Bureau is held accountable for implementing the enumeration, as planned. Without timely and appropriate actions, the challenges previously discussed could adversely affect the cost, accuracy, schedule, and security of the enumeration. We will continue to assess the Bureau’s efforts and look forward to keeping Congress informed of the Bureau’s progress. We are making the following two recommendations to Commerce: The Secretary of Commerce should direct the Director of the Census Bureau to direct the Census Bureau’s CIO to take steps to ensure that identified corrective actions for cybersecurity weaknesses are implemented within prescribed time frames. (Recommendation 1) The Secretary of Commerce should direct the Director of the Census Bureau to direct the Bureau’s CIO to implement a formal process for tracking and executing appropriate corrective actions to remediate cybersecurity weaknesses identified by DHS, and expeditiously address the identified deficiencies. (Recommendation 2) Chairman Serrano, Ranking Member Aderholt, and Members of the Subcommittee, this completes our prepared statement. We would be pleased to respond to any questions that you may have. If you have any questions about this statement, please contact Robert Goldenkoff at (202) 512-2757 or by email at goldenkoffr@gao.gov or Nick Marinos at (202) 512-9342 or by email at marinosn@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other key contributors to this testimony include Jon Ticehurst (Assistant Director); Ty Mitchell (Assistant Director); Lisa Pearson (Assistant Director); Andrea Starosciak (Analyst in Charge); Christopher Businsky, Rebecca Eyler, Scott Pettis, Lindsey Pilver; Kate Sharkey; Kevin R. Smith; Umesh Thakkar; and Tim Wexler. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "The Bureau, a component of the Department of Commerce (Commerce), is responsible for conducting a complete and accurate decennial census of the U.S. population. The decennial census is mandated by the Constitution and provides vital data for the nation. A complete count of the nation's population is an enormous undertaking as the Bureau seeks to control the cost of the census, implement operational innovations, and use new and modified IT systems. In recent years, GAO has identified challenges that raise serious concerns about the Bureau's ability to conduct a cost-effective count. For these reasons, GAO added the 2020 Census to its High-Risk list in February 2017. GAO was asked to testify about the reasons the 2020 Census remains on the High-Risk List and the steps the Bureau needs to take to mitigate risks to a successful census. To do so, GAO summarized its prior work regarding the Bureau's planning efforts for the 2020 Census. GAO also included preliminary observations from its ongoing work examining the IT systems readiness and cybersecurity for the 2020 Census. This information is related to, among other things, the Bureau's progress in developing and testing key systems and the status of cybersecurity risks. The 2020 Decennial Census is on GAO's list of high-risk programs primarily because the Census Bureau (Bureau) (1) is using innovations that are not expected to be fully tested, (2) continues to face challenges in implementing information technology (IT) systems, and (3) faces significant cybersecurity risks to its systems and data. Although the Bureau has taken initial steps to address risk, additional actions are needed as these risks could adversely impact the cost, quality, schedule, and security of the enumeration. Innovations : The Bureau is planning several innovations for the 2020 Census, including allowing the public to respond using the internet. These innovations show promise for controlling costs, but they also introduce new risks, in part, because they have not been used extensively, if at all, in earlier enumerations. As a result, testing is essential to ensure that key IT systems and operations will function as planned. However, citing budgetary uncertainties, the Bureau scaled back operational tests in 2017 and 2018, missing an opportunity to fully demonstrate that the innovations and IT systems will function as intended during the 2020 Census. To manage risk to the census, the Bureau has developed hundreds of mitigation and contingency plans. To maximize readiness for the 2020 Census, it will also be important for the Bureau to prioritize among its mitigation and contingency strategies those that will deliver the most cost-effective outcomes for the census. Implementing IT systems : The Bureau plans to rely heavily on IT for the 2020 Census, including a total of 52 new and legacy IT systems and the infrastructure supporting them. To help improve its implementation of IT, in October 2018, the Bureau revised its systems development and testing schedule to reflect, among other things, lessons learned during its 2018 operational test. However, GAO's ongoing work has determined that the Bureau is at risk of not meeting near-term IT system development and testing schedule milestones for two upcoming 2020 Census operational deliveries, including address canvassing (i.e., verification of the location of selected housing units). These schedule management challenges may compress the time available for the remaining system development and testing, and increase the risk that systems will not function as intended. It will be important that the Bureau effectively manages IT implementation risk to ensure that it meets near-term milestones for system development and testing, and that it is ready for the major operations of the 2020 Census. Cybersecurity : The Bureau has established a risk management framework that requires it to conduct a full security assessment for each system expected to be used for the 2020 Census and, if deficiencies are identified, to determine the corrective actions needed to remediate those deficiencies. As of March 2019, the Bureau had over 500 corrective actions from its security assessments that needed to be addressed, including nearly 250 that were considered “high-risk” or “very high-risk.” However, of these 250 corrective actions, the Bureau identified 115 as being delayed. Further, 70 of the 115 were delayed by 60 or more days. According to the Bureau, these corrective actions were delayed due to technical challenges or resource constraints. Resolving identified vulnerabilities within the Bureau's established time frames can help reduce the risk that unauthorized individuals may exploit weaknesses to gain access to sensitive information and systems. To its credit, the Bureau is also working with the Department of Homeland Security (DHS) to support its 2020 Census cybersecurity efforts. For example, DHS is helping the Bureau ensure a scalable and secure network connection for the 2020 Census respondents and to strengthen its response to potential cyber threats. During the last 2 years, as a result of these activities, the Bureau has received 17 recommendations from DHS to improve its cybersecurity posture. However, the Bureau lacks a formal process for tracking and completing corrective actions for these recommendations which would help to ensure that DHS's efforts result in improvements to the Bureau's cybersecurity posture. In addition to addressing risks which could affect innovations and the security of the enumeration, the Bureau has the opportunity to improve its cost estimating process for the 2020 Census, and ultimately the reliability of the estimate itself, by reflecting best practices. In October 2017, the 2020 Census life-cycle cost estimate was updated and is now projected to be $15.6 billion, a more than $3 billion (27 percent) increase over its earlier estimate. GAO reported in August 2018 that although the Bureau had taken steps to improve its cost estimation process for 2020, it needed to implement a system to track and report variances between actual and estimated cost elements. According to Bureau officials, they plan to release an updated version of the 2020 Census life-cycle estimate in the spring of 2019. To ensure that future updates to the life-cycle cost estimate reflect best practices, it will be important for the Bureau to implement GAO's recommendation related to the cost estimate. Over the past decade, GAO has made 97 recommendations specific to the 2020 Census to help address these risks and other concerns. Commerce has generally agreed with these recommendations and has taken action to address many of them. However, as of April 2019, 24 of the recommendations had not been fully implemented. Of the 24 open recommendations, 11 were directed at improving the implementation of the innovations for the 2020 Census. To ensure a cost-effective enumeration, it will be important for the Bureau to address these recommendations. GAO is making two recommendations to the Bureau to (1) better ensure that cybersecurity weaknesses are addressed within prescribed time frames, and (2) improve its process for addressing cybersecurity weaknesses identified by DHS.", "document_type": "gao"}
{"report": "AFSOC is the Air Force component of U.S. Special Operations Command and is responsible for providing Air Force capabilities and forces to support special operations activities. Special operations are operations requiring unique modes of employment, tactical techniques, equipment, and training often conducted in hostile, denied, or politically sensitive environments. Demand for AFSOC capabilities, including those provided by the ARC, is identified as part of the Department of Defense’s (DOD) Global Force Management process for assigning and allocating forces to meet global requirements. This process allows the Secretary of Defense to strategically manage forces—including the military services, conventional forces, and special operations forces—to support strategic guidance and meet combatant commander requirements. As part of this process, the Joint Staff validates requirements for forces. U.S. Special Operations Command, as the joint force provider, is responsible for identifying and recommending forces to support special operations requirements. U.S. Special Operations Command coordinates with its service component commands, including AFSOC, to determine which capabilities and specific units are best suited to meet validated requirements for special operations capabilities. After receiving these requirements, AFSOC considers its available options to provide the capabilities needed. This consideration includes reviewing active duty and reserve component units that provide specific sets of capabilities, such as intelligence, surveillance, and reconnaissance; personnel recovery; and radio and television broadcasting for psychological operations. If AFSOC, in conjunction with Headquarters Air Force, determines that the best solution to meet a requirement is to use capabilities from the ARC, it can rely on either volunteerism or involuntary recall to active duty—referred to as involuntary mobilization—to activate the needed forces. These two types of activation are described below. Volunteerism. The Secretary of the Air Force is authorized to activate ARC personnel on active duty with the consent of those individuals; however, the consent of the state governor is required for the voluntary activation of ANG personnel. According to Joint Publication 4-05, Joint Mobilization Planning (Feb. 21, 2014), volunteerism is important because it enables a service to fill required positions with reserve component personnel without its counting against the statutory limits related to involuntary mobilization. However, the guidance also states that volunteerism should be used judiciously, because excessive use of volunteers removes personnel from reserve component units, which could result in a reduction of the unit’s readiness in the event of unit mobilization. Another factor that mobilization planners must take into account is dwell time policy in relation to deployments. Furthermore, the Air Force has established specific goals for managing the operational tempo of its forces, and planners need to consider this factor as well. Involuntary Mobilization. Any unit or individual of a reserve component may be ordered to active duty under multiple mobilization statutory authorities under Title 10 of the U.S. Code that vary regarding the number of personnel who can be mobilized, the duration of the mobilization, and the approval authority. For example, section 12304 of Title 10, U.S. Code, provides authority to the President to involuntarily activate up to 200,000 members of the selected reserve for up to 365 days to augment active forces for an operational mission or in response to certain emergencies. AFSOC is required to follow Air Force guidance for accessing ARC units and personnel. The Air Force guidance implements DOD Instruction 1235.12, Accessing the Reserve Components (RC), which establishes the overarching policies and procedures for accessing the reserve components for all military departments. When AFSOC officials determine that ARC capabilities are the appropriate option for a given special operation requirement, their access to the reserve component is governed by Air Force Instruction 10-301, Managing Operational Utilization Requirements of the Air Reserve Component Forces. This instruction outlines roles and responsibilities for managing requirements for reserve component capabilities accessed through both involuntary mobilizations and volunteerism. Among other things, it establishes that AFSOC use the reserve component in a cyclical or periodic manner that provides predictability to ARC individuals, to the individual’s employer, and to the combatant command receiving the capabilities. The process for accessing the reserve component through involuntary mobilization is further outlined in Air Force Instruction 10-402, Mobilization Planning. This guidance implements and expands on the specific timelines for particular milestones during the mobilization process established in DOD Instruction 1235.12, such as the identification of the types of capabilities required and of the unit responsible for providing them. These timelines vary, depending on whether the requirement for capabilities is known well ahead of mobilization—a rotational, or preplanned, requirement—or, conversely, is emergent. Rotational or preplanned requirements: AFSOC must provide the reserve component with a request for particular capabilities at least 330 days prior to the mobilization, to allow ANG or AFR officials to identify the specific individuals who are available to support the request. Air Force guidance communicates the time frames in which reserve component personnel are to receive their mobilization orders. Specifically, AFSOC is required to submit requests to mobilize the ARC to Air Force headquarters to provide the Secretary of the Air Force enough time to approve the request; and then to communicate with ANG and AFR in sufficient time to provide personnel with their mobilization orders at least 180 days prior the start date of rotational or preplanned requirements. Emergent requirements: AFSOC is required to submit requests so that personnel receive notification at least 120 days prior to the mobilization date. In comparison, there are no specific time frames in the guidance for accessing the reserve component through volunteerism. The guidance generally discusses volunteerism as an approach that allows for ARC personnel to quickly respond to requests for forces. AFSOC officials told us that they have observed an increase in requests from ARC units to use involuntary mobilizations rather than rely on the use of volunteerism, and that they anticipate this trend to continue, since involuntary mobilizations afford more predictability than do voluntary deployments. As such, involuntary mobilizations help personnel manage the frequency of time spent away from home and maximize their access to military medical and retirement benefits. Specifically: Managing time away from home: Air Force guidance limits the frequency of involuntary mobilizations for an individual to a standard of five periods of time spent at home for every one period spent involuntarily mobilized. For example, an individual involuntarily mobilized for 90 days would not be available to AFSOC for involuntary mobilization for another 450 days after the individual’s return. This provides ARC personnel with some assurance that they will not deploy again for a specific window of time, unless they volunteer to do so. We have previously reported on challenges faced by DOD in setting policies to establish thresholds and track the total time individual servicemembers may be away from home, including for exercises, training, and deployment. We found that, with the exception of the Navy and U.S. Special Operations Command, the services either were not enforcing or had not established specific and measurable thresholds in their policies. Additionally, we found that DOD lacked reliable data for tracking the total time individual servicemembers spent away from home. We recommended that DOD clarify its policy to include specific and measurable department-wide thresholds and take steps to emphasize the collection of complete and reliable data. DOD concurred with our recommendation. Medical and retirement benefits: Involuntary mobilization can also maximize the window during which personnel receive medical and retirement benefits. All ARC personnel are eligible for benefits up to 180 days prior to their involuntary mobilization or voluntary deployment. However, to receive these benefits the individual must also have been issued mobilization orders identifying the mobilization date or, for a volunteer, the deployment date. As previously discussed, Air Force guidance identifies notification time frames designed to provide ARC personnel involuntarily mobilized to support rotational or preplanned requirements with their orders at least 180 days prior to the mobilization start date. This time frame allows personnel to receive these benefits for the entire time they are potentially eligible. By contrast, personnel who are involuntarily mobilized for emergent requirements are supposed to receive their orders with at least 120 days’ notice, and, according to AFSOC officials, volunteers can receive as little as one week’s notice. As a result, personnel may prefer involuntary mobilization, as it generally results in their receiving military medical and retirement benefits for more time than they would have received them if they had volunteered to deploy. AFSOC has mobilization processes that follow Air Force guidance, but it faces difficulties in implementing these processes. Specifically, we found AFSOC faces challenges in (1) consistently providing ARC units and personnel with timely notifications regarding anticipated demand for their capabilities; (2) coordinating with ANG and AFR commands on potential requirements for ARC capabilities; and (3) sharing reliable information about mission requirements and resources with ARC units and personnel. The notifications that AFSOC gives ARC units or personnel of anticipated demand for their capabilities generally do not meet the notification time frames associated with involuntary mobilizations for non-emergent requirements, thereby impeding ARC units’ ability to prepare for deployments. Officials at three of the four reserve component units we spoke with told us that AFSOC routinely provides units with limited notice of requirements for capabilities, even though they predominately support preplanned requirements that are known to AFSOC well in advance of their execution. Therefore, the officials stated, AFSOC should have sufficient time to identify and communicate the requirement for ARC capabilities to reserve component units to enable them to meet required time frames (for example, no less than 180 days in the case of non- emergent requirements). However, according to these officials, they routinely receive 90 or fewer days’ notice of when they are expected to provide capabilities for a given requirement. Due to this truncated time frame, the requirement must either be staffed using volunteers or receive approval from the Secretary of Defense to involuntarily mobilize reserve component personnel with limited notice. Receiving limited notification can create challenges for the ARC unit providing the capabilities for AFSOC requirements. For example, officials at one unit we spoke with stated that they requested that AFSOC provide at least 9 months’ notice prior to a mobilization to ensure that personnel received adequate training, because the unit provides a range of specialized capabilities. However, officials stated that what they generally received was 60 to 90 days’ notice, and that within this time frame the unit faced challenges in obtaining access to the equipment needed to train personnel for specific missions. Officials at another unit we spoke with stated that since 2015 they had received 60 or fewer days’ notice for their support of AFSOC requirements, one of which was an involuntary mobilization supporting a non-emergent requirement. An official explained that while AFSOC’s communication of requirements and planning of involuntary mobilizations has improved over time, the unit expects that orders for its next mobilization will be provided with fewer than 180 days’ notice. The official explained that in addition to limiting ARC personnel’s access to medical and retirement benefits, the abbreviated time frames make it difficult for them to coordinate their absences with their civilian employers. AFSOC officials acknowledged that they have been late to notify units in the past and identified this as an area in which they are working to improve. The officials explained that in some instances the late notification is a result of factors outside of AFSOC’s control, such as instances in which the Secretary of Defense’s process for approving requirements is delayed. We identified concerns regarding AFSOC’s practice of communicating directly with reserve component units, rather than formally coordinating with ANG and AFR commands, to develop potential requests for ARC unit capabilities. For example, AFR officials stated that geographic proximity to AFSOC frequently results in one unit’s receiving informal requests from AFSOC for its capabilities. That unit provides remotely piloted aircraft capabilities, which do not require personnel to deploy overseas. Officials explained that AFSOC will contact that unit directly to request capabilities to supplement the active duty personnel completing the same mission, but commonly AFSOC will provide only a few days’ notice prior to the requirement. According to these officials, personnel generally respond to these requests by volunteering with limited advance notice. AFSOC officials stated that communicating informally with the units to determine the availability of their personnel and capabilities enables AFSOC to expedite the identification of personnel potentially available to meet a requirement. However, headquarters officials for both ANG and AFR—who are responsible for identifying the specific personnel available to meet a requirement—stated that these indirect communications impede their ability to strategically manage and appropriately resource units. For example, headquarters AFR officials identified an instance in which changes to a unit’s anticipated contribution to a mission were arranged with the unit, but not with officials at their higher headquarters at the AFR. The requirement was originally for the AFR unit to supplement an active duty unit already providing the capability for AFSOC, but was expanded to require the AFR unit to have sole responsibility for providing part of the capability. The absence of direct communication and formal coordination between AFR headquarters and AFSOC during this expansion led to differing expectations regarding the number of AFR personnel needed to provide the capability required. AFR officials stated that as a result of limited transparency into future requirements for that unit, AFR headquarters did not request the appropriate level of funding for the unit, thereby limiting the resources available to support the requirement. AFSOC officials acknowledged that their use of informal communication with units instead of coordinating with ANG and AFR headquarters is not an ideal approach and could be improved. We identified concerns regarding the frequency with which AFSOC has changed the information it has communicated to ARC units about anticipated requirements, thereby creating unpredictability and impeding those units’ ability to train for and ultimately provide the capabilities needed to execute those requirements. While requirements may change subject to combatant command needs, AFSOC’s availability to proactively coordinate with both the combatant command and the ARC has been limited. AFSOC officials stated that, due to their limited capacity to manage involuntary mobilizations, they are often dedicating time only to those mobilizations that require urgent attention, as opposed to refining the details of the requirement and coordinating with the units in advance of the mobilization. ARC officials stated that the unpredictability resulting from the changes that occur can introduce challenges to the units’ ability to execute requirements. For example, officials at one unit stated that the location of a previous requirement changed at least three times in the 60 days preceding its involuntary mobilization. Officials explained that changes to the location of the requirement meant that the capabilities required by AFSOC also changed, because the unit provides intelligence, surveillance, and reconnaissance capabilities that need to be supported by specific communications equipment. Depending on the location, this equipment may already be in place, or it may be that the unit must bring it with them. In a different instance, the same unit arrived at a location to provide its intelligence, surveillance, and reconnaissance capabilities and found that the location lacked the communications equipment the unit needed to effectively use its capabilities. Further, ARC officials explained that changes regarding what capabilities are needed can create training challenges unique to the reserve component. ARC unit officials explained that while reserve component personnel maintain a standard level of readiness at all times, deployments may require them to train to a specific skill set to meet a mission requirement. For example, special tactics squadrons supporting AFSOC requirements can support three different mission sets, each of which may require specialized training to prepare for a specific mission, according to unit officials. Given the nature of the reserve component, these personnel have to complete this training during the limited windows of time in which they are called in from their full-time civilian jobs. As a result, the ARC has limited flexibility in responding to changes in training requirements. AFSOC officials acknowledged that the ARC can face challenges in meeting training requirements and that the advanced planning associated with involuntary mobilizations can help ensure that units have enough time to meet training requirements. Other Air Force entities that provide ARC capabilities to meet Air Force requirements through mobilization have established alternative approaches to initiating, planning, and coordinating their respective requirements for reserve component capabilities. Specifically, officials from Air Combat Command and Air Mobility Command, which are Air Force components similar to AFSOC regarding mobilization of ARC units, described entities established within their operations departments to coordinate with the ARC when implementing the involuntary mobilization process. These entities each consist of four to five individuals who are tasked on a full-time basis with ensuring that the reserve components are utilized in a predictable manner. The efforts of these entities include coordinating with the ARC to create plans that cover at least 2 years of anticipated rotational and preplanned requirements. While Air Combat Command and Air Mobility Command officials stated that they are responsible for coordinating a larger number of mobilizations than AFSOC coordinates, they noted that all three follow the same Air Force guidance with regard to the involuntary mobilization process. Officials from an ARC personnel recovery unit that supports Air Combat Command missions highlighted the benefits of the predictability that comes from Air Combat Command’s planning efforts. According to those officials, anticipated mobilizations are communicated to them in a schedule that covers a span of 5 years. More than a year before the unit is scheduled to involuntarily mobilize, Air Combat Command communicates the specifics of the requirement for the mission. The officials stated that, in their experience, these details rarely change once they have been communicated to the unit. By contrast, as previously discussed, we spoke with officials from an ARC special tactics squadron that provides AFSOC with capabilities similar to those of the personnel recovery unit described above, who stated that they regularly receive only 60-90 days’ notice prior to being deployed. They stated that they face difficulties in adequately training personnel to provide capabilities within these time frames. AFSOC officials stated that this issue is driven in part by the fact that units coordinate directly with requesting commands to fill their desired requirements. According to AFSOC officials, AFSOC does not have a headquarters entity dedicated to managing the planning, coordination, and execution of reserve component capabilities because, until recently, AFSOC did not use its reserve components to support ongoing missions to the extent that they do today. As a result, it was not considered necessary to have an organizational entity dedicated to managing involuntary mobilizations. Instead, AFSOC assigned the roles and responsibilities associated with initiating, planning, and coordinating ARC mobilizations within its overall process for managing AFSOC’s assignment and allocation of forces. AFSOC and ARC officials stated that under this process, a single individual at AFSOC is responsible for managing the involuntary mobilizations as a secondary duty. AFSOC officials stated that, given the scope of other assigned responsibilities, this individual focuses on managing involuntary mobilizations about half of one day in a work week. According to the officials, having a limited staff dedicated to initiating, planning, and coordinating involuntary mobilizations results in AFSOC’s responding to issues as they become urgent and impedes its ability to utilize the ARC in a predictable and stable manner. AFSOC officials also stated that the shift to using the ARC to support AFSOC’s steady state requirements, along with the increasing use of involuntary mobilizations to access ARC capabilities, have exposed the limitations of their capacity to manage involuntary mobilizations. These officials added that creating a more robust organizational capacity to manage the involuntary mobilization of the reserve component could counteract some of the challenges they have experienced in providing timely notification to ARC units, directly coordinating with ANG and AFR commands, and identifying and communicating reliable information about requirements to ARC units. AFSOC officials attribute the challenges faced in implementing AFSOC’s involuntary mobilization processes to the absence of adequate capacity to manage involuntary mobilizations. Specifically, they acknowledged that with additional capacity they would be better positioned to undertake the efforts needed to (1) provide more timely notification to ARC units, (2) coordinate with ANG and AFR commands, and (3) increase communication with the commands generating requirements. While officials acknowledged that some last-minute changes are unavoidable, they told us that having more personnel dedicated to AFSOC’s mobilization process could potentially lead to having more timely notifications or better indications of imminent changes. Further, some factors that can affect the involuntary mobilization process fall outside of AFSOC’s control, such as delays in the decision making process at the Secretary of Defense level and changes in combatant commander requirements. Although AFSOC cannot control all factors that affect involuntary mobilization of the ARC, increasing its capacity to manage involuntary mobilizations would improve its ability to anticipate and proactively address the challenges introduced by external factors. AFSOC officials stated that in recognition of this need, the command’s operations center has submitted multiple requests for additional resources to the headquarters in order to create a more robust organizational capacity to manage the involuntary mobilization of the reserve component. For example, the request submitted in January 2019 stated that AFSOC currently does not provide the support and guidance that ARC units need to properly execute the involuntary mobilization process. The request sought one additional full-time position dedicated to managing involuntary mobilizations and coordinating involuntary mobilizations with the ARC. Although AFSOC officials told us that the request was validated by AFSOC leadership, the validation of a request does not ensure that it will receive funding. After competing against other funding requests from other Air Force components, the Financial Management Board did not fund the position in fiscal years 2018 or 2019 because those other requests received higher priority. As an alternative to the full-time position requested by the operations center, AFSOC officials identified ongoing efforts to coordinate with the ANG that would result in the ANG’s allocating personnel to fill a temporary position at AFSOC. The individual in this position would be responsible for supporting the mobilization process. AFSOC officials stated that such an arrangement would help address the capacity challenges they currently face, but also noted that it would be a short-term solution, and highlighted that the individual filling the position would need to be familiar with AFSOC, ANG, and AFR processes to execute his or her duties. In addition, AFSOC officials could consider realigning existing capacity within the command to directly address the limited capacity to manage involuntary mobilizations. However, AFSOC officials emphasized that the command as a whole currently operates with limited capacity. In the absence of the Air Force developing additional AFSOC organizational capacity dedicated to the planning, coordination, and execution of involuntary mobilizations, AFSOC will continue to be impeded in its ability to manage involuntary mobilizations in accordance with Air Force guidance, including providing the notice required to access the ARC through involuntary mobilization in support of preplanned or rotational requirements. Additionally, at its current capacity AFSOC will likely face increasing challenges in providing timely notification to ARC units, coordinating with ANG and AFR commands, and enhancing communication with the commands generating requirements to help solidify mission specifics, as the number of involuntary mobilizations quadruple by 2021, as estimated by AFSOC officials. As a result, units may not be fully prepared to support requirements or able to effectively conduct their mission once in theater. Further, AFSOC will continue to be impeded in coordinating with ANG and AFR commands in a manner that enables the ARC to strategically manage and resource units in support of AFSOC’s requirements. While the Air Force’s force-generation model provides the ARC with a 24- month picture of the units it anticipates will be used to meet potential Air Force-wide deployments, the ARC does not have a comparable model with information on which ARC units could be used to support AFSOC requirements for special operations activities. According to officials, the ARC does not have a force-generation model for two reasons. First, while the Air Force model works for Air Force-wide requirements, it does not apply to special operations-specific requirements because they are unique to the Air Force’s special operations component, AFSOC. According to AFSOC officials, their command deploys units and personnel differently from typical Air Force units in order to maximize the number of requirements they can support with a smaller force. Second, the ARC has historically supported special operations activities using volunteerism, which is much more flexible than involuntary mobilization and requires less upfront planning or notification. As a result, ARC officials did not feel the need to develop a force-generation model for special operations requirements. ANG and AFR officials told us that ARC units will sometimes keep a unit-level schedule of their potential deployments, but that information is not available in a consolidated or consistent format. AFSOC officials added that any force-generation model for special operations should consider the limited capacity of some special operations capabilities. Officials stated that some capabilities in the ARC are limited to one unit, which results in AFSOC deploying parts of units rather than the whole unit to cover more requirements. ANG and AFR officials agreed that a force-generation model regarding future deployments could help identify which ARC units would be susceptible to deploy during a given period of time, which would be beneficial for planning ARC deployments. Consolidated information on potential unit deployments would provide units with advanced notification, making it easier to accomplish deployment preparation activities and helping ARC personnel make arrangements for their potential deployments. For example, ANG officials told us that advanced notification to units can give the ARC more time to incorporate needed training into drill training. Furthermore, unit personnel would also have more time to make arrangements with civilian employers or in their personal lives, making their transition to active duty easier and making it more likely that they will view mobilizations favorably in the future. Additionally, these officials stated that, with such a model, ANG and AFR could more easily identify and communicate which ARC units would be available for mobilization to support special operations activities. AFSOC officials stated that, in turn, this could provide AFSOC with more certainty that it would have access to ARC forces when needed. According to ANG and AFR officials, AFSOC officials have expressed some concerns about whether their command will have access to ARC forces. Specifically, since a substantial part of the total Air Force capability resides in the ARC, AFSOC officials are not certain that the capacity of ARC units supporting special operations will be able to meet future requirements. The officials added that by identifying units or individuals susceptible for deployment in advance, AFSOC would have more confidence in the ARC’s ability to support the command’s requirements. According to Air Force guidance, a predictable force-generation model is used to ensure proper force readiness and rapid responses to emerging crises. Specifically, Air Force Instruction 10-401, Air Force Operations Planning and Execution, calls for the Air Force and its components, including the ANG and AFR, to manage the deployment of its forces in order to meet global requirements while maintaining the highest possible level of overall readiness. The instruction calls for the Air Force to accomplish this task by establishing a force-generation model that can be used to manage the rhythm of force deployments to meet global combatant command requirements. The intent of the force-generation model is to establish a predictable, standardized pattern to ensure that forces are properly organized, trained, equipped, and ready to sustain capabilities while rapidly responding to emerging crises. ANG officials told us that they have taken some initial steps to create a force-generation model and consolidate the various unit-level schedules of ARC forces supporting special operations activities. Specifically, the ANG advisor to AFSOC was developing a consolidated schedule of ARC units intended for use by AFSOC to identify ANG units that could mobilize to support AFSOC requirements. However, according to AFSOC officials, the ANG advisor was expected to retire soon, and we found that ANG headquarters officials were not aware of this effort, and there were no plans to institutionalize it. AFR officials were likewise not aware of any similar effort to consolidate schedules for their units’ different capabilities to support special operations activities. Without having a method for providing consolidated information on reserve component units that are available for deployment, the ARC will not have the information it needs to successfully plan its deployments, or easily identify and communicate to AFSOC which of its units are or will be available for mobilization. Furthermore, AFSOC officials may continue to have concerns that they will not have access to high demand ARC capabilities to deploy under a mobilization. According to officials, although ANG and AFR units have a general understanding as to how many volunteers they have supporting special operations requirements at the unit level, the ANG and AFR lack a mechanism for tracking volunteer deployment rates across the ARC. Specifically, information on reserve components’ volunteer deployments is not available in a form that facilitates tracking in order to understand rates of volunteering or the contributions made by the ARC in supporting special operations activities, according to officials. The Air Force requires the ANG and AFR to track key data to ensure proper management of ARC utilization and mission execution. Specifically, Air Force Instruction 10-301, Managing Operational Utilization Requirements of the Air Reserve Component Forces, calls for the Air Force to identify full mission requirements for ARC utilization by collecting, tracking, and organizing relevant data and prioritizing requirements. It also states that these data are intended to aid in allocating funding, matching units to requirements, executing requirements, assessing each step of the process, and forecasting future requirements. Additionally, Standards for Internal Control in the Federal Government establishes that management should obtain relevant data from reliable internal and external sources in a timely manner to facilitate effective monitoring. ANG and AFR officials told us that voluntary deployments are more difficult to track than are involuntary mobilizations. Specifically, the statutory requirements for involuntarily mobilizing ARC units or personnel make tracking them simpler. For example, according to officials the Secretary of Defense is required to approve or be notified of involuntary mobilizations, and ANG and ARC units receive specific orders, all of which are tracked closely. Voluntary deployments, however, do not have the same approval requirements. Nevertheless, ANG and AFR officials told us that ARC units may have some information, as detailed below, on the numbers of volunteer deployments, although this information provides only a partial picture of volunteerism. Travel System Data: Officials from a reserve component unit we visited reported that some of the information on voluntary deployments could be compiled from travel systems used to send ARC units and personnel overseas. However, these officials added that matching travel records to the volunteer status of individuals could be time-consuming, because the travel systems are not designed to perform this function. Furthermore, unit officials told us that this travel information would be incomplete even if it were compiled, because it would not include units and individuals supporting operational requirements from their home stations—that is, not traveling outside their normal locations. For example, according to unit officials, personnel supporting remote piloted aircraft would not be included in the information collected from the travel systems because they do no travel outside their normal duty stations to carry out their missions. Without travel orders, the system would not show these types of deployments. Unit officials told us that there could be several cases like this one in which the information compiled from the travel system or other sources could be incomplete. Man-Day Estimates: An AFSOC official told us that the system used to track military personnel appropriation man-days could be another source used to track volunteerism among ANG and AFR units. According to this official, AFSOC uses a data system to transfer man- days to the volunteering ARC unit. This official stated that the system used to make these transfers may contain the information needed to track volunteerism, but acknowledged that no one at AFSOC was using the system for this purpose. Furthermore, ANG and AFR officials confirmed that the data system is not currently used for tracking rates of volunteerism among ARC units. According to AFR officials, tracking volunteerism would allow them to more easily document the ARC’s contributions to support special operations and evaluate whether ARC forces were being effectively utilized. Specifically, ANG and AFR officials expressed concerns that different rates of individual volunteerism within and across ARC units may result in a misleading picture of overall unit utilization. In some cases, incomplete data on volunteerism can result in overstating unit contributions. For example, the unit-level figures regarding deployments are actually averages of all the individuals in the unit. Officials expressed concerns that as a result of using averages, units may appear to be more highly utilized than they actually are, due to the high rates at which some individuals from the unit volunteer to deploy. According to ANG and AFR officials, some ARC personnel volunteer at high rates because they prefer the additional income or benefits from these deployments, while other personnel from the same units may prefer to deploy less often. ARC officials expressed concerns that this disparity may not be immediately visible to ARC and AFSOC leadership. AFSOC officials told us that they share some of these concerns. Other officials expressed concerns that without good information on volunteerism rates, the ANG and AFR could not effectively manage operational tempo goals. To measure operational tempo, DOD has established policies relating to how long military personnel are deployed versus at home (referred to as dwell time, or dwell). For example, ARC personnel who deploy for 7 months and are in dwell for 14 months would have a deployment-to-dwell ratio of 1:2. For ARC units, DOD also tracks the mobilization-to-dwell ratio, which is the ratio of how long ARC personnel are involuntarily mobilized versus not mobilized. DOD guidance establishes that the mobilization-to-dwell ratio for ARC units should be 1:5. According to ANG and AFR officials, ARC voluntary deployments are not factored into the dwell calculations for either ratio, making it more difficult to ensure that deployments do not fatigue ARC forces. Additionally, Special Operations Command policy specifies that ARC units supporting special operations should maintain the same deployment cycle as active duty units, which as a goal should be no less than a 1:2 deployment-to- dwell ratio. ANG, AFR, and AFSOC officials agreed that tracking volunteer deployment rates more comprehensively and consistently would provide greater perspective on how ARC units are utilized and help them more effectively manage their operational tempo goals. Officials stated that an additional consequence of having incomplete data on volunteerism is that the overall contributions of the ARC can be understated, because the full range of support that ARC units and personnel are providing is not being documented. For example, a report used by the Air Force to track force contributions from its components, including the ARC, shows that the AFR contributed forces to support special operations activities for about 6 months of an approximately 4- year period. However, according to AFR officials, the command’s contribution to support special operations activities was much higher than what is documented in the report. The officials stated that AFR also provided volunteer support to AFSOC over the entire period but that its contributions are not fully reflected in the report, because volunteers supporting an AFSOC-assigned mission are counted among the contributions made by other active duty forces, rather than by AFR. Without complete information on volunteer deployment rates among reserve component forces, the ANG and AFR may face difficulties in ensuring the effective utilization of their forces to support special operations activities, documenting force contributions from the ARC, and managing operational tempo and deployment-to-dwell goals. Further, the ARC will not have the information it needs to ensure effective management of its force utilization and mission execution. Specifically, it will not be able to determine whether units are being fully utilized, because of the distorted or incomplete volunteerism information. With a substantial part of the total Air Force capability residing in the ARC, AFSOC relies on mobilized ARC forces to support its operations. Furthermore, AFSOC’s increasing use of ARC as an operational reserve has highlighted the importance of the ARC’s and AFSOC’s planning and information-sharing efforts. However, AFSOC’s implementation of its mobilization process impedes its ability to provide the ARC with timely notification of mobilizations, coordinate with ANG and AFR commands, and share reliable information about requirements with the ARC. Without resolving AFSOC’s organizational capacity challenge in managing AFSOC requirements for reserve capabilities, AFSOC’s implementation of this process is unlikely to improve. AFSOC’s use of the ARC is also affected by the unavailability of complete information regarding both the units available to mobilize and voluntary deployment rates. Specifically, while the ARC is able to identify the units anticipated to be available to support non-special operations requirements, it does not have a method for communicating consolidated information on the availability of units for special operations requirements. Without such a method, AFSOC and the ARC do not have easily accessible information about the current and future availability of ARC units to support special operations requirements. In addition, voluntary deployments are a key piece of the ARC’s support of AFSOC requirements. However, the ARC has not developed a mechanism for tracking the rate at which they occur. Without tracking volunteer deployment rates, the ARC is limited in its ability both to ensure that its forces are effectively utilized and to communicate the level of contribution made by ARC volunteers in support of special operations requirements. We are making three recommendations to DOD: The Secretary of the Air Force, in coordination with ANG and AFR, should ensure that AFSOC has the organizational capacity to effectively initiate, coordinate, and execute ARC mobilizations, to include ensuring timely and reliable notification of requirements to those units. (Recommendation 1) The Secretary of the Air Force should ensure that the ANG and AFR develop a method for providing AFSOC with consolidated information regarding units available for immediate and future mobilizations to support special operations activities, such as the Air Force provides to its units with its force-generation model. (Recommendation 2) The Secretary of the Air Force should ensure that the ANG and AFR develop a mechanism for tracking volunteer deployments to better manage ARC force utilization. (Recommendation 3) In written comments on a draft of this report, DOD concurred with one recommendation and partially concurred with two recommendations. DOD’s comments are restated below and reprinted in appendix I. DOD also provided technical comments, which we incorporated where appropriate. DOD concurred with the first recommendation that the Secretary of the Air Force, in coordination with ANG and AFR, should ensure that AFSOC has the organizational capacity to effectively initiate, coordinate, and execute ARC mobilizations, to include ensuring timely and reliable notification of requirements to those units. In its response, DOD stated that the Air Force continues to balance manning requirements across the spectrum of operations. DOD also stated that fully manning AFSOC for this staff function would be helpful, whether additional manpower is programmed or AFSOC mitigates internally by reallocating manpower. We believe that fully manning AFSOC for this staff function, if fully implemented, would meet the intent of the recommendation. In its comments on this recommendation, DOD also stated that the ARC has a process in place to provide timely notification to ANG and AFR units once requirements are known. The department added that the ANG implemented the Agile ARC Mobilization Process on June 1, 2019, which streamlined policy and procedural chokepoints and improved notification timelines by an average of 60 days. We note that, while improvements in the notification timelines would be beneficial, it is too soon to understand the long-term effect of the implementation of this process. DOD partially concurred with the second recommendation that the Secretary of the Air Force should ensure that the ANG and AFR develop a method for providing AFSOC with consolidated information regarding units available for immediate and future mobilizations to support special operations activities, such as the Air Force provides to its units with its force-generation model. In its comments, DOD stated that the AFR currently provides AFSOC with information on units available, using Reserve Component Periods, and that the AFR will assess whether re- posturing in multiple Reserve Component Periods will provide a portion of capability with greater flexibility. We agree that this is a reasonable approach. However, as we noted in our report, consolidated information on reserve component units that are available for deployment could provide ARC units with advanced notification, making it easier to accomplish deployment preparation activities and help ARC personnel make arrangements for their potential deployments. Additionally, DOD stated that current information technology initiatives with the Air Force Integrated Personnel and Pay System will eventually provide the Air Force with functionality allowing a single, integrated system of software suites. According to the department, Air Force Integrated Personnel and Pay System will support a rapid and accurate information flow from the first identification of a requirement through the processing and delivering of orders, allowing the Air Force to start pay and benefits in an auditable manner. However, DOD did not identify a timeline for when that system would be available. We believe that improvements in the flow of information regarding ARC unit availability are necessary and would help to ensure that the ARC can successfully plan deployments, or easily identify and communicate to AFSOC which of its units are or will be available for mobilization. We believe that if this planned system is implemented as described, it would meet the intent of the recommendation. DOD partially concurred with the third recommendation that the Secretary of the Air Force should ensure that the ANG and AFR develop a mechanism for tracking volunteer deployments to better manage ARC force utilization. In its response, DOD stated that tracking volunteer deployments requires timely information from AFSOC to properly identify the requirements, establish expeditionary ARC units, and document the transaction when ARC members are activated. Further, it stated that in the short term, the AFR will work with AFSOC on further developing use of the Air Force Consolidated Planning Schedule to better define requirements. While coordination with AFSOC could help improve the tracking process, we believe that the ANG and AFR also need to develop a mechanism for tracking volunteer deployments to better manage ARC force utilization. Additionally, DOD noted that the planned information technology initiative, which it described in its response to our second recommendation, could also have benefits for tracking voluntary deployments. We believe that if the planned system is able to fully track voluntary deployments, it would meet the intent of the recommendation. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; the Under Secretary for Personnel and Readiness; the Chief of the National Guard Bureau; and the Commanders of Special Operations Command, Air Force Special Operations Command, and Air Force Reserve Command. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-5431, or russellc@gao.gov. Contact points for our respective offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. In addition to the contact named above, individuals who made key contributions to this report include Jim Reynolds, Assistant Director; Adam Anguiano, Tracy Barnes, Adrianne Cline, Shylene Mata, Walter Vance, and Cheryl Weissman.", "summary": "Over the past decade the Air Force has increasingly relied on the ARC to meet operational requirements. The ARC is composed of two entities—the Air National Guard (ANG) and the Air Force Reserve (AFR)—which together comprise a substantial part of the total Air Force capability. AFSOC relies on either volunteerism or involuntary mobilization to activate ARC units. House Report 115-676, accompanying a bill for the National Defense Authorization Act for Fiscal Year 2019, contains a provision for GAO to assess ANG and AFR involuntary mobilization plans to support special operations. GAO evaluated the extent to which (1) AFSOC's mobilization process provides the ARC with timely and reliable forecasts of planned utilization of units and personnel; and (2) the ARC identifies and communicates information to AFSOC on the units and individuals available for mobilization or on voluntary deployments. The Air Force Special Operation Command's (AFSOC) mobilization process does not fully support Air Reserve Component (ARC) needs for timely and reliable information. While AFSOC has established mobilization processes in line with Air Force guidance, the command faces difficulties, as follows: consistently providing ARC units and personnel with timely notifications regarding anticipated demand for their capabilities; coordinating with ARC commands on potential requirements for ARC capabilities; and sharing reliable information about mission requirements and resources with ARC units and personnel. According to AFSOC officials, these difficulties stem from AFSOC's limited organizational capacity to conduct the planning, coordination, and execution of involuntary mobilizations (that is, ARC units or personnel ordered to active duty). Other Air Force entities that provide ARC capabilities to meet Air Force-wide requirements have established the capacity within their operations departments to coordinate with the ARC when implementing the involuntary mobilization process. AFSOC officials stated that because AFSOC did not, until recently, regularly use involuntary mobilizations to access the ARC, it was not considered necessary to have an organizational entity dedicated to managing involuntary mobilizations. AFSOC officials stated that the command's operations center has submitted requests to its headquarters for additional resources toward creating such organizational capacity, but the requests were not funded in fiscal years 2018 or 2019, as other requests received higher priority. According to officials, AFSOC is currently exploring possible short-term solutions. In the absence of the organizational capacity to conduct the planning, coordination, and execution of involuntary mobilizations, AFSOC will continue to be impeded in providing the notice required to access the ARC in support of requirements. The ARC does not provide AFSOC with complete information regarding which of its units could be used to support AFSOC requirements for special operations activities. The Air Force uses a model that captures and organizes Air Force-wide requirements, but the model does not include special operations requirements, and AFSOC is expected to develop its own processes for its unique requirements. According to AFSOC and ARC officials, the ARC has not developed a method for capturing and organizing special operations requirements because it has historically supported special operations activities using volunteerism, which is more flexible and requires less up-front planning. Consolidated information on potential unit deployments would provide units with advanced notification, facilitating deployment preparation activities and helping personnel make arrangements with civilian employers or in their personal lives. Without a method to provide consolidated information on reserve component units available for deployment, the ARC will not have the information it needs to successfully plan its deployments, or to easily identify which of its units will be available for mobilization. GAO is making three recommendations, including that the Air Force should ensure that AFSOC has the organizational capacity to effectively initiate, coordinate, and execute ARC mobilizations; and should develop a method for providing AFSOC with consolidated information regarding units available for mobilizations. DOD concurred with one of these recommendations and partially concurred with two, stating that some information is being shared and a planned initiative could improve the information flow. GAO believes this initiative, if implemented, could address the intent of its recommendations.", "document_type": "gao"}
{"report": "While U.S. airlines’ business practices were largely deregulated following the Airline Deregulation Act of 1978, a number of consumer protections are in place at the federal level. For example, some consumer protections are required by federal statute, such as the Air Carrier Access Act of 1986 (ACAA), as amended, which prohibits airlines from discriminating against individuals based on a disability. Federal statutes have also authorized DOT to regulate certain areas affecting passengers. For example, DOT has the authority to stop airlines from engaging in unfair or deceptive practices, or unfair methods of competition, and promulgates consumer protection regulations under its statutory authorities. Under these authorities, DOT issued three final rules on Enhancing Airline Passenger Protections from 2009 through 2016. These rules have addressed long tarmac delays, increased compensation amounts for passengers who are involuntarily denied boarding, and required certain airlines to post information about their fees and on-time performance on their websites. In 2018, we found that airlines’ operational performance—as measured by DOT data on denied boardings; mishandled baggage; and late, cancelled, or diverted flights—generally improved from 2008 through 2017, the most recent data available at the time of our review. While rates of voluntary and involuntary denied boardings and mishandled baggage generally declined, airlines’ on-time performance stayed about the same (fig. 1). For example, over the 10-year period of our review, the lowest rate of involuntary denied boardings occurred in 2017. Specifically, in 2017, airlines involuntarily denied boarding to about .003 percent of all passengers (or about 23,000 of more than 680 million passengers)—a slight decrease from prior years. Our more recent work on airlines’ denied boarding practices found that even fewer passengers were denied boarding involuntarily in 2018. Rates of mishandled baggage also generally declined in recent years. For example, in 2017 airlines posted a rate of 2.5 mishandled bags per 1,000 passengers (a rate of .25 percent of mishandled bags per passenger enplanement), compared to a rate of 5.25 mishandled bags per 1,000 passengers in 2008. In 2019, we identified a number of factors that can cause airlines’ operational issues. For example, passengers might be denied boarding when airlines overbook their flights (i.e., intentionally sell more seats than are available on a flight) or have to substitute smaller aircraft than what was originally scheduled due to maintenance issues. We also found that outages associated with airline IT systems—which are used for flight and crew planning, passenger reservations or check-in, or for providing flight information to the Federal Aviation Administration—can cause flight delays and cancellations. While we found some outages caused minimal issues, the impact of others was more substantial. For instance, in 2016, an outage in one airline’s system that is used to check in and board passengers resulted in the cancellation of 2,300 flights over 3 days. While airlines’ operational performance generally improved, we found in 2018 that the number of passenger complaints reported to DOT, relative to passenger boardings, generally increased from 2008 through 2017 for 12 selected airlines, peaking in 2015 and declining somewhat in later years. Specifically, in that work we found that the rate of passenger complaints reported to DOT, relative to passenger boardings, increased about 10 percent, from about 1.1 complaints per 100,000 passengers in 2008 to 1.2 complaints per 100,000 passengers in 2017. Complaints about operational issues discussed above—which make up three of DOT’s 15 complaint categories—accounted for about half of all complaints for the 12 selected airlines from 2008 through 2017. More specifically, in 2018 we found: Flight problems generally accounted for an average of about 33 percent of all complaints. This category includes complaints related to delays, cancellations, and missed connections, among other things. From 2008 through 2017, the rate of complaints in this category generally increased. Baggage issues generally accounted for an average of about 15 percent of total complaints. Complaints were largely related to lost, delayed, or damaged bags. The rate of baggage complaints generally decreased over our time period. Denied boardings generally accounted for an average of about 4 percent of total complaints. Complaints were related to airlines’ failure to solicit volunteers or providing compensation below the required amount. Rates of complaints about denied boardings generally stayed constant over our time period. Two of the remaining 12 complaint categories tracked by DOT accounted for about a quarter of passenger complaints. One category related to reservations, ticketing, and boarding, and the other related to customer service—such as airline staff having a poor attitude or refusing to provide assistance, and unsatisfactory seat assignments. Each of these categories generally accounted for an average of about 13 percent of all complaints over the 10-year period. Our previous work identified actions taken by airlines or DOT in response to such operational issues. DOT’s actions are primarily related to establishing regulations about operational issues. For example, while DOT does not prohibit airlines from overbooking flights, it has set compensation amounts for passengers denied boarding involuntarily. DOT has also issued regulations related to returning mishandled baggage within 24 hours, tarmac delays, and prohibiting chronically delayed flights. Examples of airlines’ actions are listed below. Reducing denied boardings. In 2019, we reported that selected airlines have taken a range of actions, aimed at reducing involuntary denied boardings. Some of these actions also provide additional incentives for passengers to volunteer to be denied boarding. Actions include reducing or eliminating overbookings; improving software to better predict passenger no-shows; requesting volunteers earlier (e.g., at check-in instead of at the gate); increasing compensation for volunteers; and conducting reverse auctions to solicit volunteers. Less mishandled baggage. As we reported in 2018, representatives from almost all airlines we interviewed reported investing resources to improve baggage-handling efforts and minimize the effects to passengers whose bags are lost or delayed. Among other actions, airline representatives told us they upgraded baggage technology; modernized the claims process, so passengers could complete forms on-line; and instituted replacement baggage programs, where passengers can get a replacement bag at the airport. One airline also invested several million dollars to use radio frequency identification technology to track bags, as well as allowing passengers to track their baggage via an application on their smartphone. Efforts to minimize flight disruptions. In 2018, we also reported that selected airlines had taken numerous actions to improve on-time performance or mitigate challenges for passengers associated with flight delays and cancellations. For example, one airline began tracking flights that were “at-risk” of meeting DOT’s definition of a chronically delayed flight, so it could, among other things, swap crews or substitute aircraft and avoid these types of delays. Other airlines told us they use technology, such as text-messaging updates, to communicate with passengers during delays and cancellations or increased the number of circumstances for which passengers are compensated during delays and cancellations. Our prior work has shown that passengers may be affected to varying degrees by airline operational issues, and that incidents can be costly and disruptive for some passengers. Airlines are required by DOT regulations to provide compensation or certain amenities to inconvenienced passengers under certain circumstances. For example, some passengers who are denied boarding involuntarily are entitled to compensation, with the amount varying based on certain factors. Airlines are also required by DOT’s interpretation of the statutory prohibition on unfair and deceptive practices to provide refunds for canceled and significantly delayed flights, if a passenger chooses to cancel his or her trip. Beyond those requirements, DOT officials previously told us that airlines are not obligated to provide accommodations for flight disruptions, such as cancellations and delays, unless specified in an airline’s contract of carriage, although as mentioned above, some voluntarily choose to do so in certain situations. This may result in significant inconveniences for passengers, who may incur costs for lodging, meals and transportation. However, according to our prior work, available information about the number and magnitude of these effects is largely anecdotal and cannot be quantified. Furthermore, our review of selected airlines’ contracts of carriage in February 2019 showed variation in the types of accommodations airlines provide and circumstances in which they will be provided, when operational issues occur. According to the 2010 U.S. Census, 57 million Americans (roughly 1 in 5) have a disability, and more than half of those 57 million Americans have mobility issues. Furthermore, older Americans are representing an increasing share of the U.S. population. As the population continues to age, the likelihood of this group needing assistance may increase. Without accommodations—such as effective communication of flight information, accessible seats, appropriate boarding assistance, and careful handling and stowage of wheelchairs and other assistive devices—people with accessibility or mobility issues may face challenges when flying, or they may be unable to fly altogether. As previously mentioned, the ACAA prohibits airlines operating in the U.S. from discriminating against individuals on the basis of disability in the provision of air transportation. Under this law, DOT has promulgated regulations requiring that airlines provide passengers with disabilities (1) assistance in enplaning and deplaning; and (2) compensation for lost, damaged, or delayed wheelchairs or other assistive devices. In contrast to all other complaints that passengers submit directly to airlines, DOT regulations require that airlines report annually to DOT the number of all disability-related complaints they received. In our May 2017 report, we provided information showing that disability complaints reported to airlines and DOT generally increased from 2005 through 2015. More recent data shows that passenger complaints reported to U.S. airlines continued to increase (see table 1). In particular, we found that complaints reported to airlines on disability issues increased by about 50 percent from 2010 (19,347) to 2017 (29,662), the most recent year for which data are available. Based on our review, the vast majority of passengers chose to file their disability complaints directly to the airlines. Notably, the number of passenger complaints on disability issues reported to DOT from 2010 through 2019 ranged from 572 to 944 and averaged about 780 complaints per year. Complaints reported to DOT rose in 2019, after peaking in 2015 and declining the three following years. In 2017, the last year data are available for both, complaints reported to airlines and DOT were most commonly related to failure of airline staff to provide assistance, seating accommodation issues, and service animal issues. As we have previously reported, the number of complaints may not fully reflect the inconvenience experienced by passengers or would-be- passengers with accessibility issues. Some may choose not to fly and others may have to take inconvenient or uncomfortable precautionary measures to avoid using the aircraft lavatory. For example, in our recent work examining the accessibility of aircraft lavatories, stakeholders we interviewed told us that some passengers severely limit their food and fluid intake in advance of the flight, risking dehydration; use a catheter; or wear a protective undergarment. Furthermore, because lavatories accessible by the aircraft’s onboard wheelchair are not required on most aircraft (i.e., single-aisle aircraft) and there may not be an expectation that the lavatory be accessible by an onboard wheelchair, passengers may not see grounds to complain or may not take the time to submit a complaint. More generally, in our prior work, we found that complaint data are inherently limited because a substantial portion of dissatisfied individuals do not submit complaints and are therefore not represented in the complaint data. A number of federal statutes also prohibit or have been interpreted by DOT to prohibit airline discrimination against airline passengers. Federal statute also allows airlines to refuse to transport any passenger if the airline determines that the passenger is, or might be, a threat to safety. According to DOT guidance, this determination is made by the pilot in command of the aircraft or certain other specified airline personnel and cannot be arbitrary, but must be based on specific facts and circumstances known at the time. In its guidance, DOT has unequivocally provided that a passenger’s status in a protected class (e.g., race, ancestry, national origin, or religion) cannot be the determinative factor in an airline’s decision to deny boarding or remove a passenger from a flight. Our August 2019 report showed that the total number of passenger complaints reported to DOT against U.S. airlines alleging discrimination generally declined from 2010 through 2015, but began to increase starting in 2016. Moreover, updated data for 2019 show a further increase, with 96 complaints filed (table 2). According to our analysis, from 2010 through 2019, DOT received, on average, 80 discrimination- related complaints a year, most commonly about racial discrimination. Despite the recent increase in the total number of discrimination complaints, they account for a small percentage of total passenger complaints DOT receives, as well as total passenger boardings. For example, in 2019, of the 9,547 complaints DOT received against U.S. airlines, 96 alleged discriminatory treatment. As noted above and previously reported, DOT’s discrimination complaint data does not capture passenger complaints reported directly to airlines. In 2018, we reported that DOT officials estimated that, across all complaint categories, for every passenger complaint they receive, airlines receive about 50. While we have previously requested discrimination complaint data from selected airlines, they have generally declined, citing the proprietary nature of this information. Since 2017, DOT has disaggregated discrimination complaints into sub-categories, such as racial or religious discrimination, and published this data in its Air Travel Consumer Report. We previously identified actions that DOT and airlines have taken that are intended to ensure that no passengers are discriminated against on the basis of disability or other protected class. Our work primarily examined airlines’ efforts to train staff and contractors. However, our work also identified other airline actions (both proactive and reactive) taken to enhance compliance with consumer protections in these areas. For example, one airline developed a wheelchair tracking system in response to a DOT enforcement action to help reduce incidents of lost or mishandled wheelchairs. DOT requires that airlines provide their employees and contractor staff who interact with the traveling public training on the proper and safe operation of equipment used to accommodate passengers with a disability, as well as on boarding and deplaning assistance. While not required, DOT encourages airlines to implement comprehensive non- discrimination trainings to help prevent discrimination. DOT has also developed training materials, available on its aviation consumer protection website, for airline employees and contractor staff. These materials include brochures, digital content, and videos on the rights of passengers with disabilities, as well as tips on providing wheelchair assistance at airports and onboard aircraft. In 2017, DOT also developed guidance for airline personnel on non-discrimination topics. The material included scenarios for recognizing discriminatory behavior and provided examples of how to ask additional questions or conduct additional screening in a non-discriminatory manner. In 2017 we reviewed disability training programs for 12 selected airlines and found that they all had disability-related training requirements for their staff and contractors, with some variations in the content and format. Over the course of that work, each airline demonstrated that it had, as required, initial and recurrent training for its employees, contractors, and complaint resolution officers (CRO). All 12 selected airlines used a mix of training, including classroom-based training, computer-based training, situational scenarios, and hands-on training, such as wheelchair handling and lifting passengers into aisle seats to assist in boarding for specific groups. We also found that these selected airlines generally consulted with disability organizations when developing ACAA training programs. Some airlines also voluntarily implemented quality assurance programs to improve and sustain their disability-training programs’ performance. Another step some airlines have taken, though not required by the ACAA or its implementing regulations, is the creation of a disability board, which serves as a forum for increasing awareness among their workforce about disability issues. In 2019, we reported that representatives from all six U.S. airlines we selected for review told us they provide non-discrimination training to employees, although not all contractor staff receive that training. These representatives told us they provide initial non-discrimination training to newly hired employees who interact with passengers—including, for example, pilots, flight attendants, and customer service representatives— and that most regularly update the training based on current events or changes in policy. Airline representatives provided high-level examples describing the content of their trainings, but with one exception, they declined to provide more specific information, citing the sensitive or business proprietary nature of such materials. We found some similarities and differences in what representatives reported their trainings covered. For example, representatives generally stated that non-discrimination trainings—which were typically embedded in larger training programs and combined in-person and web-based modules—emphasized treating all individuals fairly and without bias, regardless of race, ancestry, or religion, among other things. Most also said trainings covered implicit bias—a term that refers to attitudes or stereotypes about groups of people that unconsciously affect a person’s understanding, actions, and decisions—and half said they have used DOT’s guidance discussed above, with some airline-specific modifications. In our recent work on aircraft lavatories, we found that some U.S. airlines voluntarily installed lavatories accessible by the aircraft’s onboard wheelchair for some of their single-aisle aircraft. However, we found that these aircraft only constituted about 4.5 percent of the eight selected airlines’ combined single-aisle fleet. According to airline representatives, providing lavatories accessible by the aircraft’s onboard wheelchair may reduce the number of revenue generating seats in the aircraft cabin, which can increase airlines’ costs and result in higher fares for consumers. In lieu of lavatories accessible by the aircraft’s onboard wheelchair, airline representatives said they have added certain features—such as assist handles or grab bars, and accessible call buttons or door locks—designed to increase access to certain lavatory functions. DOT has recently issued three notices of proposed rulemaking (NPRM) designed to improve the accessibility of aircraft lavatories, regulate service animals, and clarify DOT’s authority to stop airlines from engaging in unfair or deceptive practices. For example, in January 2020, DOT issued an NPRM to solicit comments on short-term accessibility improvements on single-aisle aircraft through the installation of accessibility features within the lavatory, such as those mentioned above, without changing the size of lavatories. In addition, DOT announced its intention to issue an advance NPRM to address long-term accessibility improvements and to solicit comments and gather information on the costs and benefits of requiring airlines to increase the size of the single- aisle lavatory on new aircraft models to accommodate a wheelchair as well as an assistant. In 2008, DOT noted that accessible lavatories on single-aisle aircraft would benefit passengers with disabilities, but also expressed concerns that revenue loss and other cost impacts could be too great for the airlines. The FAA Reauthorization Act of 2018 included a number of ongoing requirements for DOT in the airline consumer protection area. For example, DOT is responsible for developing leading non-discrimination practices for airlines, in consultation with airlines and other consumer advocates. In addition to our recently published work, we have ongoing work examining airport accessibility for passengers with disabilities, as well as DOT’s enforcement approach to consumer protections. We anticipate issuing reports on the results of this work later this year. Chairman Larsen, Ranking Member Graves, and members of the Subcommittee, this completes my prepared remarks. I look forward to answering any questions you may have. If you or your staff have any questions about this statement, please contact me at (202) 512-2834 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Jonathan Carver, Assistant Director, Geoffrey Hamilton, Delwen Jones, Josh Ormond, Amy Suntoke, Melissa Swearingen, and Elizabeth Wood. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Each year, hundreds of millions of passengers rely on airlines to get them to their destination without incident—including some of the 57 million Americans with a disability. While airlines maintain their performance and service have improved, passengers may still experience a range of inconveniences. A number of consumer protections are in place at the federal level. These protections have addressed long tarmac delays and increased compensation for passengers who are involuntarily denied boarding. Some protections are specific to passengers with disabilities, requiring that airlines provide (1) help enplaning and deplaning, and (2) compensation for lost or damaged wheelchairs. DOT enforces these protections. This statement discusses (1) DOT's data on airline operational performance from 2008 through 2017, and (2) what is known about passenger complaints and airlines' practices related to accessibility and non-discrimination issues. This statement is based on six prior GAO reports issued in the past 3 years. For that work, GAO analyzed relevant DOT data and passenger complaints; reviewed DOT documents and regulations; and interviewed DOT officials and representatives from selected airlines and consumer advocate organizations. For this statement, GAO updated prior analyses on passenger complaints for accessibility and discrimination issues and reviewed recent DOT rulemakings. The Department of Transportation's (DOT) data show that airlines' operational performance—as measured by rates of denied boardings, mishandled baggage, and flight delays—generally improved from 2008 through 2017, the latest available data at the time of GAO's review. Nevertheless, in 2018, GAO found that passenger complaints to DOT across all complaint categories increased about 10 percent from 2008 through 2017 for 12 airlines that GAO selected for review. Complaints about airlines' operational performance accounted for around 50 percent of the total. Passenger disability complaints submitted to airlines—which vastly outnumber such complaints submitted directly to DOT—have steadily increased since 2011. Unlike all other categories of passenger complaints, airlines are required to annually report the number of disability-related complaints they receive to DOT. Passenger disability complaints submitted directly to DOT also increased in 2019, accounting for the second highest level in the past 10 years. Complaints to airlines and DOT in 2017—the most recent year data were available—were most commonly about failure of airline staff to provide assistance, seating accommodation issues, and issues related to service animals. Passenger complaints submitted to DOT related to discrimination also rose in 2019, with 96 complaints filed. From 2010 through 2019, DOT received, on average, 80 complaints a year from passengers alleging discrimination, most commonly about racial discrimination. DOT requires that airlines provide training on accessibility issues and encourages non-discrimination training for its staff. In 2017, GAO found that 12 selected airlines had accessibility-related training requirements for their staff and contractors, with some variations in the content and format. In 2019, GAO reported that representatives from six selected U.S. airlines provide non-discrimination training to employees, although not all contractor staff receive that training. Airlines have taken initial actions in other areas. More recently, in 2020, GAO found that only about 4.5 percent of the eight largest U.S. airlines' fleet of aircraft with single aisles were designed to accommodate airplane onboard wheelchairs.", "document_type": "gao"}
{"report": "As previously mentioned, multiple federal departments and components have responsibilities for combating the flow of illicit drugs into the United States. Figure 1 summarizes the missions and responsibilities of the federal departments and components primarily responsible for combating the trafficking of illicit drugs. In 1989, DOD created several joint task forces, which aimed to bridge the military’s counterdrug efforts with those of civilian, federal law enforcement agencies. These task forces have evolved since then and eventually developed into the present-day iterations of JIATF-South under the U.S. Southern Command and JIATF-West under the U.S. Indo-Pacific Command. JIATF-South and JIATF-West both consist of representatives from DOD, DHS, and DOJ components, among others. Coast Guard admirals currently serve as the Directors of both of the JIATFs. Previously, DOD service components have led JIATF-South; however, while DOD is responsible for detection and monitoring of drug flow, it is precluded from taking law enforcement actions in counterdrug efforts. Task force officials stated that Coast Guard leadership encourages participation from both DOD and DHS because the Coast Guard is both a military and a law enforcement agency. The deputy and vice leadership positions at the JIATFs are held by officers and civilians from DOD, DHS, and DOJ components, which allow the task forces to leverage various experiences and authorities across these components, according to task force officials. In 2014, DHS established three new joint task forces — (1) JTF–East, (2) JTF–West, and (3) JTF–Investigations—as pilot programs to, among other things, address the smuggling of illicit drugs over the southern border and approaches to the United States. Additionally, according to the DHS Southern Border and Approaches Campaign Plan, the JTFs were created to strengthen the unity of effort within DHS toward common goals. The 2017 National Defense Authorization Act subsequently codified these task forces and established new JTF requirements, such as establishing outcome-based and other appropriate performance measures to evaluate the effectiveness of each JTF. In 2017, DHS also created a JTF Coordination Cell to develop JTF performance measures and enhance awareness among DHS components about the role of the JTFs, among other things. The DHS JTFs primarily consist of representatives from CBP, ICE, and the Coast Guard, and a representative from each of these components serves as the Director for each of the three JTFs. The deputy leadership positions of each JTF are held by officers from the other two components. For example, DHS JTF-West’s director is an officer from CBP, and the deputy directors are officers from the Coast Guard and ICE. According to a DHS memorandum, in establishing the JTFs, DHS wanted each JTF to be led and supported by the different DHS components in order to integrate their varied capabilities. For more information on the task forces’ leadership and compositions, see table 1. Each of the five task forces are similarly organized by functional areas and all include areas such as administration and personnel, intelligence, and operations. For example, JIATF-South, JIATF-West, and each of the DHS JTFs have Planning sections, which help guide the task forces’ overarching strategic plans and operations, with input from other sections. One task force—JIATF-West—further tailored its organizational structure to its missions and activities. Specifically, JIATF-West reorganized in January 2016, at the direction of the former U.S. Indo-Pacific Combatant Commander, to operationalize and combine its Intelligence and Operations functional areas into a Counternarcotics Operations Center. According to JIATF-West leadership, the Counternarcotics Operations Center better reflects the nature of its intelligence gathering and sharing activities with other federal law enforcement agencies and foreign countries. JIATF-West officials also stated the task force merged its section that provided support and training of foreign law enforcement agencies into its Planning and Engagement section since that section directs activities related to JIATF-West’s engagement with partner nations. As shown in figure 2, the two DOD task forces (JIATF-South and JIATF- West) and two of the three DHS task forces (JTF-East and JTF-West) have geographical areas of responsibility. In contrast, the third DHS task force (JTF-Investigations) is focused on coordinating investigations and information sharing to support DHS and the other two DHS JTFs. As a result, it does not have a geographical area of responsibility. Example of Joint Interagency Task Force– South (JIATF-South) Activity to Combat Illicit Drug Trafficking When JIATF-South receives information about a potential illicit drug smuggling event, it will use available air and maritime assets allocated to it to detect and monitor the suspect smuggling vessel. Once JIATF-South locates the suspect vessel and has assets in place, JIATF-South turns over control of the assets to the relevant law enforcement agencies (e.g., the Coast Guard, CBP, etc.) to interdict the smuggling vessel and any illicit drugs that may be on board. JIATF-South: Focuses its activities on detecting, monitoring, and supporting the interdiction of bulk cocaine movements being smuggled on noncommercial maritime vessels. According to JIATF- South officials, this focus is partly because the key coca-producing countries are within its area of responsibility, and partly because cocaine is a key source of profit for transnational criminal organizations. JIATF-South is also allocated assets, such as ships and surveillance aircraft, from DOD and DHS components (such as the Coast Guard and CBP Air and Marine Operations), as well as from foreign partners. JIATF-South uses these maritime and air assets, in conjunction with available intelligence, to detect and monitor the trafficking of illicit drugs, such as cocaine, being smuggled north across its area of responsibility. Once JIATF-South detects a smuggling event occurring, it passes this information and control of the assets to law enforcement authorities to interdict the smuggling event. For an example of how this occurs, see the sidebar. Example of Joint Interagency Task Force– West’s (JIATF-West) Capacity Building Efforts to Combat Illicit Drug Trafficking JIATF-West has helped countries in its area of responsibility—such as Vanuatu—build their financial investigative capacity by providing law enforcement training on topics such as bank records analysis, money laundering theory, and accounting. This training is intended to help foreign law enforcement agencies better detect transnational criminal organizations’ transactions, thus making it more difficult for such organizations to operate in their area of responsibility. JIATF-West: Focuses its missions and activities on four priorities: (1) detecting precursor chemicals that can be used to manufacture illicit drugs, such as synthetic opioids; (2) supporting allies and foreign partners in combating illicit drug trafficking in its area of responsibility; (3) monitoring drug flows moving to, from, and through Asia and other countries in the Indo-Pacific region; and (4) detecting the flow of fentanyl and other synthetic opioids, according to JIATF-West documents. According to JIATF-West officials, JIATF-West’s activities primarily consist of intelligence gathering and collaboration with law enforcement partners within foreign countries where precursor chemicals are manufactured or combined to manufacture illicit drugs. JIATF-West also engages in capacity building with law enforcement authorities in foreign countries in the Pacific region, such as the Philippines and Thailand. For an example of JIATF-West’s capacity building efforts, see the sidebar. Unlike JIATF-South, JIATF-West does not have assets, such as ships or aircraft. However, JIATF-West officials stated that even if JIATF- West had assets, it would not alter the focus of its missions and activities because of the threat transnational criminal organizations pose and the nature of the flow of illicit drugs and precursor chemicals in its expansive area of responsibility. For example, JIATF-West officials told us that precursor chemicals are typically shipped in commercial cargo containers. Notably, all precursor chemicals are legal to manufacture and sell for legitimate uses, such as the production of pharmaceutical drugs and pesticides, and it is difficult to determine when such chemicals have been diverted for illicit use. Officials stated that JIATF-West would face legal and logistical challenges if they were to directly disrupt precursor chemicals being diverted, such as if the vessel was state-owned or was in a foreign country’s territorial waters. Thus, even if JIATF-West had assets, JIATF-West officials noted that the legal and logistical challenges would not change how the task force approaches its missions and activities. DHS JTFs: Focus on coordinating with DHS components (e.g., CBP, ICE HSI, Coast Guard) to facilitate awareness about cross- component, cross-geographic homeland security issues. The JTFs have broader missions than countering the flow of illicit drugs. For example, the JTFs also have responsibilities for coordinating migrant interdiction and counter-terrorism activities. Further, given their areas of responsibility, JTF-East primarily focuses on threats along the southern maritime border of the United States and JTF-West primarily focuses on threats along the southwest land border. In contrast, JTF-Investigations focuses on supporting DHS-wide investigations and sharing information to support the other two task forces. Similar to JIATF-West, the JTFs do not have physical assets to support these activities. According to JTF officials, this is partly because the 2017 National Defense Authorization Act requires the JTFs to be cost neutral. Additionally, JTF officials stated the JTFs were not meant to serve a similar function as the DOD combatant commands and, instead, are meant to help with planning and coordinating missions and activities across joint operating areas. Task force officials reported that the task forces effectively coordinated counterdrug missions and activities to minimize duplication of efforts. The extent to which the task forces coordinate varied based on whether they have (1) shared purposes and (2) areas of responsibility with overlapping or shared geographical boundaries. In particular, those task forces that have shared purposes and those task forces that have overlapping areas of responsibility or shared boundaries tended to coordinate with one another more than with the other task forces. We also found that the task forces use a variety of mechanisms to coordinate counterdrug missions and activities, such as the use of working groups and liaison officers, that our prior work has identified as best practices. Officials we met with from each of the task forces stated that they are satisfied with the level of coordination that takes place with other task forces and that the coordination efforts have been effective. Our analysis of their responses found that of the five task forces, JTF-Investigations’ coordination activities were rated as the most effective by the other four task forces. JIATF-South was rated the second highest task force in terms of both the effectiveness of its coordination activities and the number of other task forces with which it coordinated. Figure 3 provides a visual representation of the task force officials’ views on the extent to which the task forces coordinate with one another and the effectiveness of the coordination efforts. of coordination with the other task forces varies based on the extent to which the task forces have shared purposes. Of the five task forces, JTF-Investigations was the one task force that coordinated with all the other task forces, which is consistent with its purpose to enhance DHS investigations, coordinate priorities, and share information with the other joint task forces. As a part of its process in designating cases as a Homeland Criminal Organization Target (HOMECORT), JTF-Investigations conducts a Comprehensive Criminal Network Analysis that identifies links between multiple cases and criminal organizations that can cross geographical and task force boundaries (see sidebar for more information on the HOMECORT process). According to JTF-Investigations officials, this analysis helps identify cases that may be related and helps to coordinate cases across task force jurisdictions to prevent duplication of missions and activities. forces told us that they coordinated more with those task forces with which they had a shared border or joint operating area. For example, JIATF-South shares a joint operating area or a geographical boundary with both JTF-East and JIATF-West and, as a result, officials from these three task forces provided more robust examples of coordination. According to task force documentation, such as operational guidance, and our discussions with task force officials, JIATF-South, JIATF-West, and the DHS JTFs coordinated with each other on missions and activities where they have a shared interest, such as a common illicit drug threat. These coordination activities include information sharing and joint operations, as well as mechanisms, such as the use of working groups and liaison officers, which our prior work has identified as best practices for coordination. According to task force officials, this coordination is intended to enhance counterdrug efforts and avoid duplication of missions and activities. As described earlier, the task forces have different mission focuses that depend on their geographically defined areas of responsibility, which also help the task forces avoid duplication of missions and activities. However, as shown earlier in figure 2, there are some areas of land, sea, and air in which more than one task force may conduct missions and activities (e.g., between JIATF-South and JTF- East). These areas of overlap are called joint operating areas. According to our review of task force documents and discussions with task force officials, within these joint operating areas, the task forces share intelligence information, coordinate missions and activities with one another, and sometimes conduct joint operations. For example, in 2018, JTF-East led and coordinated with JIATF-South on an operation to increase intelligence and targeting capabilities to disrupt illicit drug trafficking organizations operating within their joint operating area in the Caribbean. JTF-East personnel deployed to JIATF-South’s headquarters to facilitate coordination and information sharing. As a result of this joint operation, the law enforcement agencies involved seized over 3,700 pounds of cocaine and apprehended 69 migrants, one smuggler, and the smuggling vessel, according to JTF-East documentation. Officials from the task forces we spoke with reported coordinating most frequently through meetings and working groups, and through liaison officers, as detailed in examples below. JIATFs: In 2018, JIATF-South and JIATF-West officials developed a collaborative process to track and target shipments with potential illicit drugs and precursor chemicals moving between their respective areas of responsibility. For example, JIATF-West analysts traveled to JIATF- South to initiate the process, and officials stated they continue to work with JIATF-South analysts remotely on an ongoing basis on such collaborative efforts. JTFs: The JTF Coordination Cell hosts quarterly “synchronization meetings” with the three DHS JTFs to discuss emerging drug and smuggling trends, ongoing coordination efforts, and investigations. All five task forces utilize liaison officers to enhance coordination with the other task forces and components. For example, in 2018, JIATF-West sent an analyst to JTF-Investigations to coordinate on a HOMECORT case related to drug threats in the Indo-Pacific region. Further, the five task forces coordinate with each other and their participating components through liaison officers that reside at the task forces. For example, JIATF- South officials told us that they coordinate with JTF-East through a Coast Guard liaison at JIATF-South. Liaison officers also provide direct access to their components’ information systems, which task force officials said further aids them in sharing information and coordinating missions and activities. In addition to meetings, working groups, and liaison officers, the task forces utilize other coordination mechanisms, such as memoranda of understanding and agreement, shared databases, and conferences, as detailed below. Memoranda of understanding or agreement: The two JIATFs have nine separate formal memoranda of understanding or agreement with various DHS and DOJ components, such as the Drug Enforcement Administration and ICE, that detail how the task forces and agencies will coordinate with one another and share resources. Shared databases: Each of the five task forces, along with other federal agencies, can submit information, sometimes known as a “critical movement alert” to shared databases, to alert JIATF-South about a potential drug event in its area of responsibility. According to JIATF-South officials we spoke with and our observations, JIATF- South uses these critical movement alerts, along with other intelligence that may exist, to determine whether it will dedicate assets to target a smuggling event, in conjunction with other, relevant law enforcement agencies. Conferences: Each of the five task forces participates in periodic in- person, telephone, or video conferences to coordinate with one another and share information on key issues. For example, JIATF- South officials stated they have ongoing discussions once a quarter via video conference with JIATF-West officials and other federal agencies and task forces to coordinate on illicit drug threats. JIATF-South uses both output-based and outcome-based performance measures to gauge the effectiveness of its counterdrug missions and activities, and it reports the results to the DOD Deputy Assistant Secretary of Defense for Counternarcotics and Global Threats in JIATF- South’s annual Performance Summary Reports. JIATF-South consistently assesses four key performance measures, called interdiction continuum measures, using data from the Consolidated Counterdrug Database to determine the effectiveness of its missions and activities. Specifically, JIATF-South measures (1) total maritime smuggling events, (2) targeted smuggling events, (3) detected smuggling events, and (4) seized or disrupted smuggling events. According to JIATF-South officials, data on these events allow JIATF-South to develop its primary measure to determine the effectiveness of its counterdrug missions and activities: the percentage of smuggling events JIATF-South detected that it handed off to law enforcement agencies that resulted in disrupted or seized illicit drugs. These measures and the results for fiscal years 2014 through 2018 are shown in table 2. From fiscal years 2014 through 2018, the rate at which JIATF-South successfully detected and handed off smuggling events for interdiction was generally 70 percent or higher. While JIATF-South officials acknowledged they have not met the target set by DOD, they noted there are many factors that influence the effectiveness of JIATF-South’s counterdrug missions and activities in any given year that are outside of its span of control. For example, drug trafficking organizations may adapt their tactics in response to JIATF-South’s activities to make it more difficult for the task force to target and detect their movements. This could include changing their trafficking routes or altering the size or type of smuggling conveyances the drug trafficking organizations use to transport the illicit drugs. In September 2014, JIATF-West set up an Assessments Branch to provide an annual assessment of the task force’s counterdrug efforts that was intended to inform leadership about whether the task force was undertaking the best activities to achieve its mission and implementing them effectively. According to JIATF-West officials, the nature of JIATF- West’s missions and activities make it inherently more difficult to assess and quantify the effectiveness of its efforts relative to other task forces. For example, unlike JIATF-South, which is annually allocated assets to support its missions and activities and can measure results—such as tons of cocaine seized—JIATF-West’s initiatives and activities are primarily focused on information sharing and helping partner nations improve their counterdrug capabilities, activities for which results may be more difficult to quantify. To develop its annual assessment report, JIATF-West’s Assessments Branch evaluates and assigns scores for each of the approximately 20 counterdrug initiatives and more than 100 corresponding activities it conducts each year. (For an example of a JIATF-West initiative and a corresponding activity and a description of how they were assessed, see the sidebar.) In particular, JIATF-West evaluates its initiatives to determine the progress made toward achieving objectives defined in JIATF-West’s strategic documents, such as its Theater Counternarcotics Campaign Plan. Further, JIATF-West evaluates its activities to determine whether they were executed as planned, including considerations of whether the activities were done with the intended organizations, at the specified locations and times, and whether they met stated objectives. Nevertheless, we identified ways JIATF-West measures its performance that inhibit its ability to demonstrate its overall effectiveness of countering the flow of illicit drugs. Specifically, we found that JIATF-West (1) lacks a vital few performance measures that summarize its overall effectiveness that can be consistently assessed over time and (2) that it does not have established targets for assessing the effectiveness of its numerous missions and activities. JIATF-West has focused its performance measures on assessing its numerous initiatives and activities; however, it has not developed a vital few, comprehensive performance measures that summarize the overall effectiveness of its numerous initiatives and activities in a manner that would convey essential information on its counterdrug activities to decision makers at the DOD command level and above. Such information could help these decision makers better understand the overall effectiveness of JIATF-West’s counterdrug missions and activities in relation to broader U.S. counterdrug efforts. For example, JIATF-West could develop a performance measure that calculates the percentage of leads it provides to foreign partners that result in seizures or apprehensions. Such a measure could demonstrate JIATF-West’s overall effectiveness in supporting allies and foreign partners in combating illicit drug trafficking in its area of responsibility, in keeping with one of its operational priorities. Guidance on performance measures from the DOD Office of the Deputy Assistant Secretary of Defense for Counternarcotics and Global Threats states that agencies should develop a vital few measures, no more than two or three, which convey essential information on counterdrug activities to decision makers. JIATF-West used to assess and report such measures as recently as fiscal year 2015. For example, it reported scores on the effectiveness of its mission and broader lines of effort, such as countering illicit drug and precursor chemical flows in its area of responsibility. However, JIATF-West officials told us they stopped reporting on these performance measures in fiscal year 2015 because, from the task force’s perspective, the measures did not provide meaningful insights into the effectiveness of the task force’s operations. However, such performance measures could provide meaningful information on the effectiveness of JIATF-West’s counterdrug activities to decision makers at the DOD command level and above, even if such summary information might not affect the effectiveness of operations at the task force level. We have previously reported on the importance of developing performance measures that demonstrate broader effectiveness and have also reported it is worthwhile for agencies to develop them to better determine and understand the overall effectiveness of their missions and activities. Further, JIATF-West is also unable to assess the effectiveness of its initiatives and activities over time because it has annually changed the way it measures the effectiveness of more than 100 counterdrug initiatives and activities. Specifically, JIATF-West has adjusted how it assesses the initiatives and activities each year since 2014—for example, by changing the weight scale for scoring its initiatives and activities to give more emphasis to some over others. These changes to the methodology make it difficult to compare results and assess the effectiveness of its activities over time. DOD guidance states that agencies should have measures that are consistent over time to capture trend results. In discussing these issues with JIATF-West officials, they stated that assessing the initiatives and activities provides valuable information on the effectiveness of the task force’s efforts for internal, task force management. They acknowledged that JIATF-West’s performance measures could be improved to allow for assessments of the effectiveness of the task force’s activities over time, but they added that they plan to use the same methodology to calculate the task force’s initiatives and activities scores in the future because this information is still needed internally. Given that JIATF-West’s individual initiatives and activities change year to year, however, it will be difficult for JIATF-West to assess trends in the effectiveness of its initiatives and activities over time. By also establishing a vital few, comprehensive performance measures that can be assessed consistently over time, as appropriate, JIATF-West will be able to better convey trends in the overall effectiveness of its counterdrug missions and activities over time. JIATF-West has not developed specific performance targets (i.e., established acceptable levels of performance or outcomes) for its initiatives and activities as part of its documented assessment methodology, and no such targets appear in any of the task force’s annual assessment reports. When assessing its initiatives and activities, JIATF-West officials told us it aims to achieve the best possible outcome—or the highest possible score—for each of the initiatives and activities it undertakes and assesses. DOD guidance states that targets should be set for each performance measure to establish a minimum level of performance to be accomplished within a given time frame. Additionally, establishing specific performance measure targets that set a minimum level of performance to achieve could better encourage the task force to meet the targets and identify ways to improve, as needed. The DHS JTFs were fully operational in fiscal year 2016 and began assessing their performance and producing performance reports in fiscal year 2017. Since they began reporting on their performance, the measures the JTFs reported changed in fiscal year 2018 and, according to JTF officials, will change again in fiscal year 2019. Specifically, in the fiscal year 2017 performance report, the JTFs reported on activities, such as the amounts of drugs seized, arrests made, and currency seized. However, according to task force officials, the 2017 report’s performance measures did not accurately reflect the strategic-level coordination the JTFs performed. For example, the measures the JTFs reported in fiscal year 2017 focused on drug seizures and arrests made by the DHS components. While the drug seizures and arrests made by the DHS components may have been made possible because of coordination activities of the DHS JTFs, using data on drug seizures and arrests as JTF performance measures resulted in double-counting because the components reported on the same seizures and arrests for their respective counterdrug programs. To address these issues for fiscal year 2018, the JTFs and the DHS Coordination Cell developed a new set of performance measures that were intended to better reflect the JTFs’ coordination activities and contributions. For example, a new JTF performance measure developed for fiscal year 2018 included the number of leads that the JTFs provided to a partner law enforcement agency, DHS component, or foreign government partner for interdiction or investigative action. Table 3 shows the evolution of the JTF performance measures from fiscal year 2017 to fiscal year 2018. The 2017 National Defense Authorization Act requires the Secretary of DHS to establish outcome-based and other appropriate performance measures to evaluate the effectiveness of each joint task force. Although the DHS JTF Coordination Cell and the JTFs developed performance measures in fiscal year 2018 that better reflect the specific missions and activities of the three task forces, these measures are focused on outputs—such as the number of operations conducted in combating transnational criminal organizations—and not outcomes, such as the number or percentage of leads that resulted in seizures of illicit drugs. According to JTF Coordination Cell officials, the fiscal year 2018 JTF performance measures are not outcome-based because it is difficult to quantify and capture the contributions of the JTFs through their roles as coordinators and facilitators of missions and activities that are conducted by DHS components. Table 4 illustrates each of the DHS JTF performance results for fiscal year 2018 under the revised measures. In addition to the changes to the performance measures made from fiscal years 2017 to 2018, JTF Coordination Cell officials told us in October 2018 they plan to further revise their performance measures for fiscal year 2019, as they believe their measures could continue to improve to better reflect the value added by the JTFs and their coordination and information-sharing activities. JTF Coordination Cell officials further stated that they had considered linking the fiscal year 2018 performance measures to relevant strategic-level outcomes in DHS plans. However, they noted that such outcomes—including the number of drug seizures and apprehensions—are already reported by the individual DHS components and they are trying to avoid the double-counting that occurred in the fiscal year 2017 performance report. We acknowledge that the types of coordination activities that the JTFs perform are inherently more difficult to measure, but developing and implementing outcome-based performance measures that reflect the value the JTFs add would better position the JTFs to demonstrate the effectiveness of their coordination efforts. For example, a performance measure that calculates the percentage of leads provided to components, partner law enforcement agencies, or foreign government partners that result in a successful seizure or arrest could help demonstrate the JTFs’ contributions to DHS counterdrug efforts. Further, in designing its outcome-based performance measures that are reflective of their coordination and information sharing activities, establishing a consistent set of performance measures across years, as appropriate, will allow the JTFs to better assess and convey their progress over time. In 2017, 70,237 Americans died from an overdose involving synthetic opioids, heroin, cocaine, and other drugs. The number of annual overdose deaths has nearly doubled over the past decade. Combating the trafficking and availability of illicit drugs in the United States is a government-wide priority that requires a coordinated effort by federal departments and agencies with counterdrug responsibilities. JIATF- South, JIATF-West, and the three DHS JTFs are five task forces that are focused on strengthening interagency counterdrug efforts. While these task forces have worked together to coordinate and avoid duplicative activities, improvements to the performance measures used by four of the five task forces could enable them to better determine the effectiveness of their counterdrug missions and activities. In particular, by developing a vital few, comprehensive measures that are consistent from one year to the next, and establishing specific targets against which it can measure its missions and activities, JIATF-West will be better able to determine the effectiveness of its missions and activities and assess performance trends across years. In addition, by developing outcome-based performance measures that are consistent, the JTFs would be better positioned to demonstrate the effectiveness of their counterdrug efforts over time. We are making a total of three recommendations: two for JIATF-West and one for DHS. The Director of JIATF-West should establish a vital few performance measures that are consistently measured over time. (Recommendation 1) The Director of JIATF-West should establish specific targets that set a minimal level of performance. (Recommendation 2) The Secretary of Homeland Security should develop outcome-based performance measures for the DHS JTFs that are consistent. (Recommendation 3) In May 2019, we provided a copy of this report to DOD, DHS, DOJ, and the Office of National Drug Control Policy (ONDCP) for review and comment. In written comments, which are included in appendix II, DOD stated that it concurred with the two recommendations directed to JIATF- West and noted that JIATF-West plans to conduct an internal evaluation to establish a vital few performance measures to allow it to measure performance over time. Additionally, JIATF-West has identified several areas where it can establish specific targets that set a minimal level of performance to support DOD priorities. In its written comments, which are included in appendix III, DHS stated that it concurred with its recommendation and plans to implement new performance measures in a phased approach. DHS also provided technical comments, which we have incorporated into the report, as appropriate. Additionally, ONDCP provided technical comments, which we have incorporated into the report, as appropriate. DOJ did not have any comments on the draft report. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, the Acting Secretary of Defense, and other interested parties. In addition, the report is available at no charge on the GAO website at www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-3841or AndersonN@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. This appendix provides further details regarding the performance measures and performance targets for the Department of Homeland Security Joint Task Forces. In addition to the contact named above, Christopher Conrad (Assistant Director), Kelsey Hawley (Analyst-in-Charge), and Julia Vieweg made key contributions to this report. Also contributing to the report were Billy Commons, Pamela Davidson, David Dornisch, Eric Hauswirth, and Susan Hsu. Drug Control: DOD Should Improve Its Oversight of the National Guard Counterdrug Program. GAO-19-27. Washington, D.C.: January 17, 2019. Colombia: U.S. Counternarcotics Assistance Achieved Some Positive Results, but State Needs to Review the Overall U.S. Approach. GAO-19-106. Washington, D.C.: December 12, 2018. Illicit Opioids: Office of National Drug Control Policy and Other Agencies Need to Better Assess Strategic Efforts. GAO-18-569T. Washington, D.C.: May 17, 2018. Illicit Opioids: While Greater Attention Given to Combating Synthetic Opioids, Agencies Need to Better Assess their Efforts. GAO-18-205. Washington, D.C.: March 29, 2018. Counternarcotics: Overview of U.S. Efforts in the Western Hemisphere. GAO-18-10. Washington, D.C.: October 13, 2017. Border Security: Additional Actions Could Strengthen DHS Efforts to Address Subterranean, Aerial, and Maritime Smuggling. GAO-17-474. Washington, D.C.: May 1, 2017. Coast Guard: Resources Provided for Drug Interdiction Operations in the Transit Zone, Puerto Rico, and the U.S. Virgin Islands. GAO-14-527. Washington, D.C.: June 16, 2014. Combatting Illicit Drugs: DEA and ICE Interagency Agreement Has Helped to Ensure Better Coordination of Drug Investigations. GAO-11-763. Washington, D.C.: July 28, 2011.", "summary": "The U.S. government has identified illicit drugs, as well as the criminal organizations that traffic them, as significant threats to the United States. In 2017, over 70,000 people died from drug overdoses, according to the Centers for Disease Control and Prevention. DOD and DHS created joint task forces to help facilitate and strengthen interagency efforts in combating the flow of illicit drugs, particularly in the maritime domain. GAO was asked to review the structure of these task forces and their ability to coordinate and conduct missions effectively. Among other objectives, this report (1) assesses the extent to which the task forces coordinate effectively to minimize duplication, and (2) examines how the task forces measure the effectiveness of their missions and activities. GAO reviewed and assessed documentation on the task forces' missions, coordination efforts, and performance assessments and compared them to best practices from prior work, departmental guidance, and federal internal control standards. GAO also met with task force officials to discuss and observe planning and coordination activities. Many federal agencies are involved in efforts to reduce the availability of illicit drugs by countering the flow of such drugs into the United States. Among them are the Department of Defense (DOD), which has lead responsibility for detecting and monitoring illicit drug trafficking into the country, and the Department of Homeland Security (DHS), which is responsible for securing U.S. borders to prevent illegal activity. DOD and DHS lead and operate task forces—Joint Interagency Task Force (JIATF)-South, JIATF-West, and three DHS Joint Task Forces (JTF)—to coordinate and conduct counterdrug missions and activities. Task force officials reported that the task forces coordinated effectively with each other when they had shared purposes and overlapping or shared geographical boundaries (see map). The task forces also used coordination mechanisms that align with best practices, such as working groups and liaison officers, to minimize duplication of their missions and activities. Note: DHS also has JTF-Investigations, which is a functional task force with no geographic area of responsibility. Each of the five task forces GAO reviewed has performance measures, but only JIATF-South uses output (e.g., number of detected smuggling events) and outcome-based measures to assess the effectiveness of its activities. Specifically, JIATF-South developed an outcome-based measure of its overall effectiveness: the percentage of smuggling events it detected and provided to law enforcement that resulted in disrupted or seized illicit drugs. JIATF-West evaluates its numerous initiatives and activities, for instance, by determining if they were executed as planned, but has not established a vital few performance measures that consistently convey the overall effectiveness of its activities. Lastly, the DHS JTFs' performance measures are not outcome-based and do not fully assess the effectiveness of the task forces' activities. Enhancing their measures would better position JIATF-West and the JTFs to demonstrate contributions and convey trends in the overall effectiveness of their activities. GAO is making three recommendations, including that JIATF-West establish a vital few, consistent performance measures for its overall performance; and that DHS develop outcome-based performance measures for the JTFs' activities. DOD and DHS concurred with the three recommendations.", "document_type": "gao"}
{"report": "The 8(a) program is designed to assist small, disadvantaged businesses in competing in the American economy through business development. Over the course of the program, qualified small, disadvantaged businesses can receive business development support from SBA, such as mentoring, procurement assistance, business counseling, training, financial assistance, surety bonding, and other management and technical assistance. One of the key areas of support is eligibility for competitive and sole-source federal contracts that are set aside for 8(a) businesses, which can be an important factor of the financial development for ANC-owned firms. Oversight and monitoring of all firms participating in the 8(a) program are delegated to each of SBA’s 68 district offices nationwide. Of its 68 district offices—staff at the Alaska District Office were assigned and oversaw the majority of all participating ANC-owned firms. ANCs and ANC-owned firms have a unique status in the 8(a) program and can enter into complex business arrangements In terms of their organizational structures, ANCs can be either for-profit or not-for-profit and can own a family of for-profit subsidiary firms, including but not limited to, wholly owned holding companies that often provide administrative support to smaller sister ANC-owned firms. As a condition of the 8(a) program, participating ANC-owned firms must be for-profit. Generally, ANC-owned firms can remain in the 8(a) program for up to 9 years, provided they maintain their eligibility. During the first four “developmental” years, participating firms may be eligible for assistance in program areas including sole-source and competitive 8(a) contract support, and training in business capacity development and strategies to compete successfully for both 8(a) and non-8(a) contracts, among other things. In the last 5 years, firms prepare to transition out of the program, and are required to obtain a certain percentage of non-8(a) revenue to demonstrate their progress in developing into a viable business that is not solely reliant on the 8(a) program. Across three reports on SBA’s 8(a) program, we have found persistent weaknesses in the oversight and monitoring of participating Tribal firms, in particular ANC-owned firms. Specifically, we found that SBA had (1) incomplete information and documentation on ANC-owned firms’ compliance with regulatory requirements; (2) limitations in its ability to track and share key program data needed to enforce revenue rules of Tribal firms, including ANC-owned firms; (3) insufficient staffing in its Alaska District Office to carry out necessary and critical monitoring tasks of ANC-owned firms; and (4) inadequate program guidance for clearly communicating to staff how to interpret new regulations. Incomplete information and documentation on ANC-owned firms and their compliance with regulations: We reported in 2016 that during a 2014 site visit to the Alaska District Office, we noted that incomplete information and documentation limited SBA’s oversight of the regulatory requirements specific to ANC-owned firms we examined. For example, SBA faced significant challenges in providing us with very basic information on ANC-owned firms, such as the total number of firms serviced by the agency. For example, during the course of our review, it took 3 months for SBA to provide us with a list of ANC-owned firms in the 8(a) program, and on three separate occasions SBA officials provided three separate numbers for the total number of ANC-owned firms— ranging from 226 to 636. We noted in our 2016 report that SBA’s inability to account for and make available principal information on all of the ANC- owned firms participating in the program raises concerns about the integrity of the agency’s internal controls and ability to provide effective and sustained oversight. As another example, we reported in 2016 that SBA was unable to provide seven of 30 required agency offer letters for 8(a) contracts that we requested for our review of contracts that may have been follow-on, sole- source contracts. According to the regulation, these required offer letters are critical documents that could have assisted SBA staff in understanding a contract’s acquisition history and any small business that performed this work prior to any subsequent awards. Once an applicant is admitted to the 8(a) program, it may not receive an 8(a) sole-source contract that is also a follow-on contract to an 8(a) contract that was performed “immediately previously” by another 8(a) program participant (or former participant) owned by the same ANC. We found that SBA’s inability to enforce the regulatory prohibition against follow-on, sole- source contracts was directly tied to the quality of the documentation it collected from contracting agencies. While we found that one program official in the Alaska District Office took steps during our 2016 review to ask agencies to specifically report whether contracts are follow-on, sole- source awards in offer letters, we have no evidence supporting that this practice was more broadly adopted by the program as a whole. Ultimately, we recommended and SBA agreed to enhance its internal controls and oversight of ANC-owned firms in the 8(a) program by ensuring that all ANC-owned firm files contain all relevant documents and information and providing additional guidance and training to SBA staff on the enforcement of related policies, among other things. Limitations in tracking and sharing key program data needed to enforce 8(a) revenue rules: In all three reports mentioned in this testimony, we found that SBA faced limitations in tracking information on the primary revenue generators for Tribal firms, including ANC-owned firms, to ensure that multiple firms under one parent ANC are not generating their revenue in the same primary line of business—that is, expressed as and operating under the same North American Industry Classification System (NAICS) code—which SBA’s regulation intends to limit. As discussed later in this testimony, we first identified this issue in our 2006 report, noting that SBA was not effectively tracking ANC-owned firms’ revenue data to ensure that the sister firms were not generating the majority of revenue in the same line of business. We recommended that SBA collect information on the participation of 8(a) ANC-owned firms as part of required overall 8(a) monitoring, to include tracking the primary revenue generators for ANC-owned firms and to ensure that multiple subsidiaries under one ANC are not generating their revenue in the same primary line of business. Then in our 2012 report, we found that SBA had not addressed this limitation and recommended that SBA develop a system that had the capability to track revenues from ANC-owned firms’ primary and secondary lines of business to ensure that ANC-owned firms under the same parent ANC are not generating the majority of their revenue from the same primary line of business. In our 2016 report, we found that SBA still had not developed such a system and thus was not effectively tracking and sharing the type of revenue information needed to ensure 8(a) ANC-owned firms are following the intent of 8(a) revenue rules. For example, we found that without such a system, sister ANC-owned firms owned by the same ANC could circumvent the intent of the prohibition. In particular, one sister ANC-owned firm could generate a greater portion of revenues under its secondary line of business that another sister ANC-owned firm is using as its primary line of business. Although this type of activity is not prohibited, we determined that if such activity is left untracked, a firm’s secondary line of business could effectively become its primary revenue source in the same line of business that its sister firm claims for its primary line of business without actually violating SBA’s regulation. During our 2016 review, we found 5 pairs of ANC-owned firms participating in the 8(a) program from fiscal years 2011 through 2014 that concurrently generated millions of dollars in the same line of business as their sister ANC-owned firm’s primary line of business, while generating less or no revenue under their own primary line of business. As we found then, such activity could, intentionally or not, potentially circumvent the intent of SBA’s prohibition, and as discussed later, we recommended that SBA take action to prevent ANC-owned firms from circumventing this rule. Figure 1 below illustrates one example we reported on in our 2016 report. Insufficient staffing levels in SBA’s Alaska District Office: In our 2006 report, we noted that SBA lacked adequate staffing levels in the Alaska District Office—a district office responsible for the oversight of the majority of ANC-owned firms. Our reports, and a 2008 report issued by the SBA’s Office of the Inspector General, have shown that inadequate staffing was a long-standing challenge and a consistent weakness that directly contributed to SBA’s inability to provide adequate oversight. In our 2012 report, we noted that ANC-owned firms could quickly outgrow the program. It should be noted that we recommended that SBA evaluate its staffing levels in 2006, and in our 2016 report, we found that the staffing challenges persisted. As a result, we found that SBA needed a sustained and comprehensive approach to staffing its Alaska District Office in order to conduct sufficient oversight of ANC-owned firm activities. We were told that frequent staff turnover directly contributed to the limited number of staff in the Alaska District Office with ANC firm expertise—limiting their ability to conduct effective and timely oversight of the ANC-owned firms participating in the program. An SBA official told us at the time that the optimum number of staff for the Alaska District Office was five with no more than 100 assigned 8(a) firm files each; however, that office had 1.5 staff responsible for about 200 files each. We found, based on SBA documentation and observation during our site visit to Alaska that, because of this staffing shortage, supervisory review of contract monitoring activities and annual reviews fell behind, resulting in a backlog of oversight duties related to ANC-owned firms. In 2016, we found that SBA took some short-term actions to address the issues that we identified, such as temporarily redistributing the management of ANC-owned firm files across several other district offices and within the Alaska District Office. As for long-term action, SBA officials provided us with documentation describing the program’s long-term staffing strategy, which included succession planning and managing attrition. For example, SBA planned to hire four additional BOS, and an attorney who understands ANCs. At that time, SBA began implementing its staffing strategy by hiring additional business opportunity specialists for its Alaska District Office. However, we have not evaluated whether the agency implemented the remainder of its strategy for succession planning and managing attrition. Inadequate program guidance: We reported that SBA lacked program guidance that could have assisted the Alaska District Office in improving staff’s knowledge of program rules and monitoring practices. We initially raised our concern about the need for strong guidance in 2006 given the unique status in the 8(a) program and relationships entered into by ANC- owned firms. For our 2012 report, SBA officials told us that it was in the process of updating its program guidance for the program. However, in our 2016 report, we similarly found that staff lacked sufficient guidance and training on key program regulations and internal monitoring practices, and concluded that resulting inconsistent supervisory review of ANC transactions and related documentation increased SBA’s vulnerability to compliance and fraud risks. Several months after we issued our report in 2016, SBA issued updated standard operating procedures on program rules that address the 2011 regulatory changes related to sister ANC-owned firms receiving follow-on, sole-source contracts and sister subsidiaries sharing primary NAICS codes. In addition to updating the guidance, SBA also provided training to its Alaska District Office staff on its 2011 regulations, specifically training on prohibitions against follow-on sole source contracts. SBA officials also told us in 2016 that staff in the Alaska District Office were provided training in supervisory review and other critical file management procedures, which we noted were weaknesses. To address the weaknesses described above, as well as others related to oversight and monitoring, our 2006, 2012, and 2016 reports contained a total of 21 recommendations to SBA. While SBA has fully implemented 15 of these recommendations, SBA has not implemented six recommendations—three of which we highlight in this statement. All six recommendations are important to enhancing SBA’s oversight of ANC-owned firms in the 8(a) program. We have not evaluated the operational effectiveness of SBA’s actions to implement the 15 recommendations, but if effectively implemented, those actions should help SBA improve its oversight and monitoring of ANC-owned firms in the 8(a) program. In response to our recommendations, SBA’s actions included providing training to its staff that emphasized regulations governing the requirement for procuring agencies to specifically state whether a contract is a follow-on contract in their offer letters, which could help reduce the award of a follow-on, sole-source contracts to sister ANC- owned firms; developing and enacting a regulation that gives SBA the authority, under certain circumstances, to change an ANC-owned firm’s primary line of business (expressed as a NAICS code) to the NAICS code that generates the greatest portion of the firm’s revenue; this action is intended to help SBA enforce rules preventing sister ANC-owned firms from operating in the same primary lines of business; and updating and providing written guidance to field staff officials on the enforcement of follow-on sole-source contract regulations. However, to date SBA has not provided us with evidence that it has implemented the three following recommendations, which if implemented as intended, could significantly improve its oversight of the 8(a) program. Absent action on these recommendations, SBA exposes the program to continued noncompliance. Tracking revenue data and other information on 8(a) ANC-owned firms: As previously discussed, SBA’s regulation prohibits ANCs from owning multiple firms that operate under the same primary line of business (expressed as a primary NAICS code). In each of our 2006, 2012, and 2016 reports we identified weaknesses in SBA’s ability to track this information in order to prevent sister ANC-owned firms from violating this rule or circumventing its intent. As a result, in 2006 we recommended that SBA track the primary revenue generators for ANC-owned firms and to ensure that multiple subsidiaries under one ANC are not generating their revenue in the same primary line of business, among other things. Similarly, in 2012 we recommended that, as SBA is developing a tracking system, it should take steps to ensure that the system tracks information on ANC-owned firms, including revenues and other information. In 2006 and 2012, SBA did not indicate whether it agreed with and intended to implement these recommendations. However, during our 2016 audit, SBA informed us that it had plans to address this issue, but could not provide any details. We therefore recommended in 2016 that SBA document this planned method for tracking revenue generated under subsidiaries’ primary and secondary lines of business. SBA agreed to implement this 2016 recommendation. As part of this recommendation, we stated that SBA’s documentation should include milestones and timelines for when and how the method will be implemented. We also recommended that SBA provide the appropriate level of access to and sharing of relevant subsidiary data across district offices, including primary and secondary lines of business and revenue data, once SBA develops a database with the capabilities of collecting and tracking these revenue data. In August 2018, SBA informed us that regulations promulgated in 2016 allow it to change an 8(a) ANC-owned firm’s primary line of business under certain circumstances if the greatest portion of the firm’s revenues evolved from one line of business to another. In our 2016 report, we concluded that the new regulations were a step in the right direction but would be difficult to implement effectively without the proper tracking and visibility of revenue data that we describe above and in our 2016 report. In 2018, SBA officials noted that they were testing an analytics tool that, they said, would allow them to track revenues for ANC-owned firms, as we recommended. SBA’s estimated completion date for the evaluation and implementation of this tool was December 31, 2018, but as of October 2019, SBA has not been able to provide documentation on whether this action has been implemented. We will continue to monitor SBA’s efforts to implement this recommendation. Criteria thresholds for contract modifications: As we reported in 2006, SBA regulation requires that when the contract execution function is delegated to the procuring agencies, these agencies must report to SBA certain 8(a) information, including contract modifications. Further, the agreements between SBA and the procuring agencies that we reviewed in 2006 require that the agencies provide SBA with copies of all 8(a) contract modifications within 15 days of the date of the contract award. However, in our 2006 report, we found that contracting officers were not consistently following these requirements. While some had notified SBA when incorporating additional services into the contract or when modifying the contract ceiling amount, others had not. Hence, we recommended that when revising relevant regulations and policies, the SBA Administrator should revisit the regulation that requires agencies to notify SBA of all contract modifications and consider establishing thresholds for notification. In 2006, SBA disagreed with this recommendation and thus had not revisited this regulatory requirement, but rather reiterated a preexisting requirement to provide all contract modifications, including administrative modifications, to SBA. We determined that this action did not fulfill our recommendation as it does not help to ensure that agencies are going to comply with the regulatory requirement. Small businesses potentially losing contracts to 8(a) ANC-owned firms: In our 2006 report, we found SBA’s oversight had fallen short in that it did not consistently determine whether other small businesses were losing contracting opportunities when large, sole-source contracts were awarded to ANC-owned firms. Further, we found cases where SBA did not take action when incumbent small businesses lost contract opportunities when ANC-owned firms were awarded a large sole-source contract. Hence, we recommended, that when revising relevant regulations and policies, the SBA Administrator should consistently determine whether other small 8(a) businesses are losing contracting opportunities when awarding contracts through the 8(a) program to ANC- owned firms. SBA did not agree with this recommendation, nor did it address the intent of this recommendation by developing a procedure to consistently perform this action. Instead, SBA reported to us that in 2009 it performed a single analysis of a limited set of procurement data from a limited period and concluded the data did not indicate that other small 8(a) firms (e.g., small businesses which are unconditionally owned and controlled by one or more socially and economically disadvantaged individuals, such black-owned and Hispanic-owned firms) were losing contracting opportunities to ANC-owned firms. We continue to believe that without a strategy for consistent monitoring of this issue, SBA is limited in determining the extent to which other small 8(a) businesses are being adversely impacted by contracts awarded to ANC-owned firms. In summary, the findings I have described in my statement today have persisted over time as SBA has struggled to articulate and execute an effective overall monitoring and oversight strategy. Implementing our remaining recommendations could help SBA address its monitoring and oversight control weaknesses in a comprehensive manner. Chairwoman Chu, Ranking Member Spano, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. For further information regarding this testimony, please contact Seto J. Bagdoyan, (202) 512-6722 or bagdoyans@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony are: Latesha Love (Assistant Director), Tatiana Winger (Assistant Director), Flavio Martinez (Analyst in Charge), Carla Craddock, April VanCleef, Tracy Abdo, Marcus Corbin, Colin Fallon, Julia Kennon, Barbara Lewis, Michele Mackin, Maria McMullen, James Murphy, Anna Maria Ortiz, William Shear, and Erin Villas. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Federal obligations under SBA's 8(a) Business Development Program totaled about $10.9 billion in fiscal year 2019, according to federal procurement data reported as of October 7, 2019. SBA's 8(a) program is one of the federal government's primary vehicles for developing socially and economically disadvantaged small businesses, including firms owned by ANCs. One of the key benefits of this program is the ability for ANC-owned firms to receive federal contract awards that have been set aside solely for 8(a) firms. From 2006 through 2016, GAO issued three reports detailing the limitations of SBA's oversight and monitoring of ANC-owned firms participating in the 8(a) program. GAO's testimony discusses the highlights of the aforementioned three reports and the extent to which SBA has addressed the recommendations GAO made in those reports, as of October 2019. GAO examined SBA files and other documents, conducted site visits, and interviewed program officials to perform the work of those reports. In three reports issued between 2006 and 2016, GAO has found persistent weaknesses in the Small Business Administration's (SBA) oversight and monitoring of Tribal 8(a) firms, in particular the Alaska Native Corporations' (ANC) subsidiary firms (ANC-owned firms) that participate in SBA's 8(a) program. Over the course of the program, qualified small, disadvantaged businesses, including ANC-owned firms, can receive federal contract awards that have been set aside solely for such businesses, and business development support from SBA, such as mentoring, financial assistance, and other management and technical assistance. In its three reports, among other things, GAO found that SBA had (1) incomplete information and documentation on ANC-owned firms and their compliance with regulatory requirements; (2) limitations in its ability to track and share key program data needed to enforce its own program; (3) insufficient staffing in its Alaska District Office to carry out necessary and critical monitoring tasks; and (4) inadequate or vague program guidance for clearly communicating to staff how to interpret new regulations. GAO made 21 recommendations to SBA that address weaknesses in SBA's oversight and monitoring of ANC-owned firms participating in the 8(a) program. SBA has taken steps to implement many of those recommendations, including enhancing training for SBA staff that emphasized program rules, and developing and implementing a regulation that helps SBA better enforce rules against ANC-owned firms obtaining contracts for which they were not necessarily eligible. However, SBA has not yet implemented recommendations that, if implemented as intended, could significantly improve its oversight of the 8(a) program. For example, SBA has not yet addressed limitations raised in GAO's 2006 and 2016 reports regarding SBA's tracking of revenue information for ANC-owned firms, which limits SBA's oversight of 8(a) rules prohibiting multiple subsidiaries under one ANC from generating revenue in the same primary line of business—which 8(a) program regulations intend to limit. SBA officials informed GAO of the agency's plans to develop an information system capable of addressing this issue. However, at the time of GAO's 2016 report, SBA could not provide detailed information or plans about this system, and as of today, the agency could not provide documentation that this system is operational. As another example, SBA has not addressed GAO's 2006 recommendation to consistently determine whether other small businesses are losing contracting opportunities when SBA awards contracts through the 8(a) program to ANC-owned firms, as required in regulation—an area where GAO found that SBA had fallen short in its oversight. Instead, in 2009, SBA reported that it performed a single analysis of a limited set of procurement data from a limited period and concluded the data did not indicate that other small 8(a) firms (e.g., black-owned, Hispanic-owned, and others) were losing contracting opportunities to ANC-owned firms. However, SBA's actions did not address the intent of GAO's recommendation to “consistently” perform this oversight. Absent action on these recommendations, the program continues to be at risk of noncompliance. GAO made multiple recommendations in its reports from 2006 through 2016, many of which SBA has taken steps to implement. However, SBA has not addressed key GAO recommendations, including tracking and sharing ANC-related information across SBA regional offices, considering the establishment of criteria thresholds for contract modifications, and developing policies to consistently assess whether other small businesses are losing 8(a) contracts to ANC-owned firms. GAO continues to believe that implementing these recommendations would enhance SBA's oversight and monitoring of firms in the 8(a) program.", "document_type": "gao"}
{"report": "Entities seeking to do business with DOD may have opaque ownership structures that obscure ownership or control by other entities or individuals. Beneficial Owner For the purposes of this report, we define a beneficial owner as the natural person or persons who directly or indirectly own and control, or receive substantial economic benefit from, a company. As the number of layers of ownership increases, ownership information becomes more opaque, as shown in figure 1. This opacity can make it difficult for DOD to determine which entities and individuals ultimately own or control its contractors. In the United States, no centralized information source or national registry maintains company ownership information. In 2014, the National Association of Secretaries of State found that most states collect minimal ownership data. The association reviewed key information collected by the 50 states and the District of Columbia during the entity-formation process and in annual or periodic reports. During both the entity-formation process and in annual or periodic reporting, the association found that very few states collect some form of entity ownership or control information from limited liability companies or corporations. The Securities and Exchange Commission collects some ownership information on publicly traded companies. Any person or group of persons that acquires beneficial ownership of more than 5 percent of a publicly traded company’s registered voting securities must register with the Securities and Exchange Commission. Institutional investment managers regularly disclose their holdings, and company officers, directors, and holders of more than 10 percent of a class of the company’s registered equity securities must file a statement of ownership with the Securities and Exchange Commission. GSA’s SAM is a federal government-wide database for vendor data that is used across all federal agencies. Any entity that wishes to do business with the government must register in SAM to be eligible to receive a contract award, except in specific circumstances outlined in the law and FAR. To increase procurement transparency and traceability, and broaden the government’s ability to implement fraud-detection technologies, the FAR was amended to begin requiring entities that wish to do business with the federal government to provide additional ownership information through the annual registration process in SAM starting on November 1, 2014. The required ownership information includes the “immediate” and “highest” level ownership of an offeror, as shown in figure 2 below. The FAR includes a requirement for ownership to be provided at the entity level. There is no requirement for offerors to report their beneficial owners. The FAR contains several provisions governing the selection of an offeror. Provisions such as price and past performance of the offeror are generally applicable in determining which offeror should win a contract. Additional requirements may apply to certain types of procurements, such as the procurement of national security systems. We outline several of the relevant FAR provisions; however, this does not represent a comprehensive list of all steps required by the FAR in making contract- award decisions. A prospective contractor must affirmatively demonstrate its responsibility, including, when necessary, the responsibility of its proposed subcontractors. Contracting officers must then determine the responsibility of prospective contractors, including whether prospective contractors can perform the terms of a contract. To be determined responsible, a prospective contractor must have adequate financial resources to perform the contract (or the ability to obtain them); be able to comply with the required delivery or performance schedule; have a satisfactory performance, integrity, and ethics record; have the necessary organization, experience, accounting and operational controls, and facilities to carry out the contract (or the ability to obtain them); and be otherwise qualified and eligible to receive an award under applicable laws and regulations. Before awarding a contract over the simplified acquisition threshold (generally $250,000 at the time of our review), a contracting officer must review the prospective contractor’s performance and integrity information available in the Federal Awardee Performance and Integrity Information System (FAPIIS). FAPIIS is a federal government-wide database designed to assist contracting officers with making a responsibility determination by providing integrity and performance information of covered federal agency contractors and grantees. FAPIIS provides a prospective contractor “Report Card” that includes information pertaining to the prospective contractor’s past performance (if applicable), such as any administrative agreements, contract terminations, nonresponsibility determinations, and exclusions, among other things. It also includes the ability to view the company relationship information, which details the ownership information that prospective contractors are required to report in SAM. When making a responsibility determination, the contracting officer must consider all the information available through FAPIIS with regard to the prospective contractor and any immediate owner, predecessor (an entity that the prospective contractor replaced by acquiring assets and carrying out affairs under a new name), or subsidiary identified for that prospective contractor in FAPIIS. The contracting officer must document in the contract file how the information in FAPIIS was considered in any responsibility determination, as well as the action that was taken as a result of the information. DCMA can play a role in supporting contracting officials in making responsibility determinations. For example, DCMA officials stated that they may provide information on a company’s business systems, financial capabilities, and company history, and assess whether the prospective contractor is likely to stay in business for the duration of the contract. When assessing the capacity to perform a contract, DCMA officials stated they examine company assets as a whole, including any parent company, to make a determination. According to officials, DCMA’s goal for identifying the organizational structure is to determine whether the company as a whole has the assets to perform the contract rather than to identify fraud or other risks that may be associated with that company. The level and type of support that DCMA provides to contracting officials depends on the particular needs of contracting officials for any given procurement. Some contracts require contractors to comply with cost- accounting standards and submit disclosures of their cost-accounting practice to show from which specific business units they receive allocations and to which specific business units they pass allocations; however, these disclosures are only required after a contract that is covered by cost-accounting standards is awarded. Contract award decisions are based on evaluation factors and significant subfactors that are tailored to the procurement, at the discretion of procurement officials. At a minimum, these factors must include: price/cost, quality, and past performance. Federal law grants DOD additional authority to use public and nonpublic information to make source-selection decisions when acquiring national security systems. DOD may exclude an offeror if necessary to protect national security by reducing supply-chain risk. Under this authority, DOD does not have to disclose the reason an offeror was excluded, nor can the offeror protest DOD’s decision. The FAR requires contracting officers to purchase supplies and services from responsible sources at fair and reasonable prices. For negotiated contracts, price reasonableness is ordinarily established by adequate competition, such as when there are more than two responsible offerors competing independently. For noncompetitive purchases with only one offeror, the contracting officer must obtain certified cost or pricing data, or data other than certified cost or pricing data, as necessary to establish a fair and reasonable price. Procurements with only one offeror may still be considered competitive if there was a reasonable expectation that two or more responsible and independent offerors would submit offers and the offeror submitted the offer with the expectation of competition. Section 841 of the 2015 National Defense Authorization Act grants DOD and other federal agencies the authority to limit contracts with entities that provide funds to a person or group that actively opposes U.S. or coalition forces involved in a contingency operation in which members of the armed forces are actively engaged in hostilities. It also allows agencies to terminate for default, void, or restrict the award of a contract to any contractor that provides funds received under a federal contract directly or indirectly to entities actively opposing U.S. forces engaged in hostilities. Fraud and “fraud risk” are distinct concepts. Fraud involves obtaining something of value through willful misrepresentation and is challenging to detect because of its deceptive nature. Fraud risk exists when individuals have an opportunity to engage in fraudulent activity, have an incentive or are under pressure to commit fraud, or are able to rationalize committing fraud. When fraud risks can be identified and mitigated, fraud may be less likely to occur. According to federal standards and leading practices, executive-branch agency managers are responsible for managing fraud risks and implementing practices for combating those risks. Federal internal control standards call for agency management officials to assess the internal and external risks their agencies face as they seek to achieve their objectives. The standards state that, as part of this overall assessment, management should consider the potential for fraud when identifying, analyzing, and responding to risks. In July 2015, GAO issued its Fraud Risk Framework, which provides a comprehensive set of key components and leading practices that serve as a guide for agency managers to use when developing efforts to combat fraud in a strategic, risk-based way. The Fraud Risk Framework consists of four components to effectively manage fraud risk: Assess, Design and Implement, Evaluate and Adapt, and Commit. The Assess component calls for federal managers to plan regular fraud risk assessments and to assess risks to determine a fraud risk profile. Identifying fraud risks is one of the steps included in the Fraud Risk Framework for assessing risks to determine a fraud risk profile. The fraud risk profile supports the development of a strategy to mitigate fraud risks. The Fraud Reduction and Data Analytics Act of 2015 (FRDAA), enacted in June 2016, requires the Office of Management and Budget to establish guidelines for federal agencies to create controls to identify and assess fraud risks and to design and implement antifraud control activities. The act further requires the Office of Management and Budget to incorporate the leading practices from the Fraud Risk Framework in the guidelines. In July 2016, the Office of Management and Budget published guidance about enterprise risk management and internal controls in federal executive departments and agencies. Among other things, this guidance affirms that managers should adhere to the leading practices identified in the Fraud Risk Framework. The act also requires federal agencies to submit to Congress a progress report each year for 3 consecutive years on the implementation of the controls established under the Office of Management and Budget guidelines, among other things. Recent GAO work examined federal agencies that are subject to FRDAA, including DOD, and found that 85 percent of those agencies have started planning and 78 percent have started implementing efforts to meet FRDAA requirements; however, the majority of these efforts were characterized as not being mature. Maturity was determined by agency responses to a survey question that asked whether the agency’s status of implementing FRDAA requirements was “not started,” “started but not mature,” or “mature.” The report identified the number and percentage of agencies that fell into each of these status categories, but did not state the level of maturity for any individual agency. Contractors with opaque ownership structures can pose a range of financial and nonfinancial fraud and national security risks to DOD by misrepresenting or concealing company ownership information to commit fraud against the government or to do harm to U.S. national security concerns. We identified multiple types of fraud and national security risks by examining 32 cases for fraud involving DOD contractors that were adjudicated or settled from calendar years 2012 through 2018 and conducting interviews with knowledgeable DOD officials and criminal investigators. There may be additional risks and cases related to contractor ownership that are not identified below. Court cases we identified were investigated by DOD and other entities based on, for example, information from whistleblowers, defective parts received by DOD, lawsuits involving contractors, and U.S. government officials determining they were receiving false contractor information. As discussed later in this report, DOD has not systematically assessed risks posed by contractor ownership; therefore the magnitude and prevalence of the risks we identified are not known. Appendix II of this report contains a complete listing and additional details of the 32 cases we identified. Contractors can use opaque ownership structures for illicit financial gain through a variety of methods, as described below. Concealing relationship with subcontractor to inflate prices. Contractors can subcontract with companies they own or control to inflate prices for financial benefit. For example, in a 2014 federal court case we examined, a contractor and another company with common ownership pled guilty to major fraud against the United States. They agreed to pay $434 million in criminal penalties and to settle a lawsuit in connection with concealing their relationship with a subcontractor that the contractor directed to fraudulently mark up costs on items that the contractor purchased and resold to DOD. Specifically, the contractor purchased goods from a company that its owners created, controlled, and used to make the fraudulent markups appear legitimate. Further highlighting the relationship between the company and the contractor, contractor personnel were also responsible for hiring individuals to work for the subcontractor. The contractor billed the government an artificially high price for the goods from July 2005 through April 2009 and resulted in a loss to DOD of $48 million. Figure 3 below illustrates this scheme to conceal ownership and fraudulently inflate prices. Billing for work not performed. Contractors or subcontractors can bill for work not performed by creating fictitious invoices that add costs to a contract. For example, in four court cases we examined, multiple DOD subcontractors were actually shell companies that did not have the inventory they purported to ultimately provide to the government or perform the work indicated in the contract requirements. According to documents filed in U.S. district court, some of these subcontractors hired other companies to perform work, but created additional invoices that added costs for work the subcontractors did not perform. These additional costs were then passed on to DOD. Disguising conflicts of interest. Contractors or subcontractors can conceal conflicts of interest for financial benefits. We identified a case involving a DOD subcontractor that concealed ownership for illicit financial gain. According to court records, a DOD contractor employee and his spouse formed a company and concealed their interests by not listing their names but listing the names of family members on formation documents. This company became a subcontractor to the company that employed the DOD contractor. The contractor employee, in his official position, wrote letters justifying awards of purchase orders to the subcontractor he owned and approving recommendations that the awards be made to the subcontractor. The co-owner of the subcontractor concealed her involvement by signing contracts using a different name, knowing that the use of her real name could reveal the DOD contractor employee’s ownership of the subcontractor and affect the awards. Creating the appearance of competition on a contract to inflate prices. In our review of 32 cases, we also identified the potential risk of companies creating the appearance of competition by submitting bids from fictitious companies. Specifically, we identified one case that involved a DOD contractor whose executives admitted as part of their plea agreements to creating fictitious, inflated bids that were not from actual businesses to ensure that the contractor’s own bid would be selected by DOD as the supposed lowest. In this instance, the contractor was required to obtain at least two competitive bids for certain services and items and provide the bids to DOD for selection. As part of their plea agreements, the individuals involved with the scheme also admitted that the scheme allowed the contractor to control and inflate the prices charged to DOD without any true, competitive bidding, as required. The contractor also fraudulently inflated invoices that were sent to DOD, and two individuals involved in the scheme admitted they were aware of losses to DOD of at least $34.8 million. Court records state that the scheme took place from 2011 to 2013. In 2017, two contractor executives involved with this scheme were sentenced to prison for 70 and 46 months. Additionally, we identified additional cases involving this contractor and its owner bribing government officials in exchange for the approval of fraudulent invoices, steering contracts, and covering up the contractor’s overcharging practices, which has led to at least 22 individuals pleading guilty. Additionally, DOD officials from Defense Pricing and Contracting and DLA identified the risk of different companies concealing common ownership to create the appearance of competition on a solicitation and attempt to inflate prices. By analyzing a subset of DOD solicitation data, we further examined the risk that contractors could disguise their ownership to create the appearance of competition. We identified potential relationships among the offerors of solicitations that could indicate common ownership. Our analysis of responses to approximately 2,700 solicitations in the Federal Business Opportunities (FBO) website from fiscal years 2015 through 2017 found indications that at least 16 offerors were potentially related to at least one other offeror when bidding on the same solicitation. This analysis shows indications that offerors may not always compete independently and the relationship among offerors is not always readily apparent to contracting officials or disclosed in SAM registration information. Specifically, we identified the following types of potential relationships among offerors. Offerors who shared the same management. We identified two offerors who each submitted bids on the same three solicitations and also shared the same mailing address and point-of-contact address, including suite number. According to the companies’ websites, the owner (who was also the President and Chief Executive Officer) for one offeror was the President and Chief Executive Officer of the other offeror. Further, both companies shared the same management team and neither company had reported any ownership information in SAM. According to DOD contracting officials, no additional information was disclosed to the contracting office for these offerors, nor were they otherwise aware of the potential relationship. Figure 4 below shows an example from one solicitation. Offerors who were potentially related to an entity excluded from doing business with the government. We identified two offerors who were potentially related to a third offeror who was actively excluded from doing business with the government. One of these offerors bid together with the excluded offeror on eight solicitations. Figure 5 below shows an example of one solicitation. In addition, a third potentially related offeror was identified as sharing information with one of these offerors who later bid together on a ninth solicitation. For one of the nine solicitations, one of the offerors potentially related to the excluded company was awarded a contract. According to DOD contracting officials, no additional information was disclosed to the contracting office for these offerors, nor were they otherwise aware of the potential relationship. Offerors who shared other information. We identified 11 offerors who shared other information with at least one other offeror when bidding on the same solicitation. In some instances, these potentially related offerors bid on multiple solicitations. For example, we found two potentially related offerors bid together on three separate solicitations in our FBO data. We further examined these 11 potentially related offerors’ SAM registration information to determine whether they reported shared ownership in SAM, and found one instance in which two of the potentially related offerors self-reported their relationship that one offeror owned the other; the remaining nine offerors did not report any type of shared ownership information in SAM. According to DOD contracting officials, none of the nine offerors disclosed a relationship with another offeror nor was the contracting officer otherwise aware of the potential relationship. While sharing certain information does not definitively confirm they are owned by the same entity, it is an indicator that these offerors are related. Figure 6 below highlights an example in which two offerors bidding on the same solicitation shared information and did not report shared ownership in SAM. Additionally, we identified an instance in which this type of information was also shared between two offerors and a subcontractor for a third offeror, as shown in figure 7 below. The potentially related offerors we identified did not appear to affect the overall competition on these contracts because other, seemingly unrelated offerors also submitted bids. As a result, it is unlikely that they would have affected the price paid by the government in these contracts. However, these potentially related offerors represent a risk that offerors may not always be competing independently and these types of relationships may not always be readily apparent to contracting officers, which is important when evaluating the sufficiency of competition on a solicitation and the independence of its offerors. Further, contractors may not always be forthcoming in reporting their ownership information in SAM, which can affect other areas of the procurement process, including any procedures that rely on the accuracy of this information. Contractors can pose nonfinancial fraud risks to DOD by concealing their ownership structure to bid on and obtain contracts that they are not eligible to receive. These nonfinancial risks may not pose a direct financial cost to DOD, but they can allow ineligible companies to contract with DOD while potentially denying eligible companies from contracting with DOD. As discussed below, these risks can also lead to additional vulnerabilities. In our review of 32 cases, we identified DOD contractors that concealed their ownership information to obtain contracts set aside for particular types of businesses, to obtain contracts only intended for domestic companies, and to circumvent debarment by the government. Set-Aside Contract Eligibility. Contractors with opaque ownership structures can pose the risk that government contracts set aside for small businesses are awarded to ineligible companies. Ineligible contractors could take advantage of Small Business Administration set-aside programs that allow small businesses that are owned by service-disabled veterans, women, minorities, or economically and socially disadvantaged individuals to receive government contracts specifically set aside for these types of businesses. Of the 32 cases we reviewed, we identified 20 cases in which DOD contractors or DOD contractor employees were found guilty, pled guilty, or settled with the government for representing themselves as eligible to receive set-aside contracts. These contractors falsified self-reported information and made false certifications to the government to claim eligibility by using eligible individuals as figurehead owners. In these cases, the figurehead owners did not actually maintain the level of beneficial ownership or control of the contractor required by federal regulations, or the contractors simply used the names of eligible individuals when communicating with the government to bid on and win contracts. For example, we identified one case that involved two DOD contractors participating in a single scheme to misrepresent their common ownership and obtain over $200 million in awards that they were not eligible to receive. One of the contractors that fraudulently obtained set-aside contracts claimed it was owned by a service-disabled veteran; however, that veteran had virtually no involvement with the contractor. The other contractor claimed to be owned by an economically disadvantaged individual who worked full-time for another entity and did not control the contractor. These contractors were not eligible to receive the set-aside contracts because they were not at least 51 percent controlled by the eligible individuals and the eligible individuals did not make long-term decisions for the companies. Rather, the contractors were controlled by an ineligible individual who owned and controlled a separate company that actually performed work on the set-aside contracts. To obtain government contracts set aside for companies owned by economically and socially disadvantaged individuals, the qualifying individuals must also control the majority of the company and make day- to-day decisions. Figure 8 below, which is based on an actual case, illustrates how ineligible contractors can obtain and receive government funds on contracts intended for Service-Disabled Veteran–Owned Small Businesses. Domestic Contractor Eligibility. Contractors with opaque ownership structures can also pose the risk of circumventing eligibility requirements for contracts that are only designated for domestic companies, which can lead to other vulnerabilities that affect warfighter readiness. Of the 32 cases we reviewed, we identified four cases in which individuals created domestic shell companies for foreign manufacturers and bid on contracts designated for domestic companies. In three of the four cases, the individuals behind the shell companies also had ownership interests in the foreign manufacturing companies. Foreign manufacturers received payments from the contracts, despite the contracts only allowing domestic manufacturers to be eligible, and one such manufacturer ultimately supplied DOD with defective and nonconforming parts that led to the grounding of at least 47 fighter aircraft. In multiple instances, another ineligible contractor supplied parts that were unusable due to design flaws and nonconformities. Three of these companies also exported military technical drawings and blueprints to foreign countries in violation of the Arms Export Control Act. Figure 9 below, which is based on an actual case, illustrates a contractor acting as a shell company and misrepresenting foreign manufacturing. Circumventing Debarment. Individuals that have been debarred, or prohibited from conducting business with the federal government, can circumvent their debarment by concealing their ownership in new companies that were created for the sole purpose of continuing to conduct business with the government. Of the 32 cases we reviewed, we identified one conviction of an individual who was debarred from 2013 to 2016 for supplying defective parts to DOD. This individual created three shell companies and concealed his beneficial ownership and control of these companies by omitting his name from communication with DOD and using fictitious names and names of family members as company officials. These three shell companies continued to provide defective and nonconforming parts to DOD, and the debarred individual received approximately $2.8 million in payments from DOD from May 2013 to June 2016. DOD officials we spoke with and published DOD research have identified the risk of contractors disguising company ownership as an enabler to do harm to national security interests. Contractors fraudulently misrepresenting themselves to DOD could actually be operated by adversaries seeking to act against the government’s interests. Foreign- owned contractors can conceal ownership information when registering in SAM, which allows contractors to self-attest ownership information. For example, in addition to the 32 cases we identified through our review, we also identified a bid protest filed with GAO challenging a contract award made to a foreign-owned DOD contractor in fiscal year 2018 that prohibited the participation of foreign firms or domestic companies under foreign ownership, control, or influence. This contractor did not disclose its foreign ownership or control in SAM or to DOD, as required by the FAR and the solicitation. As a result of the bid protest, DOD subsequently terminated the contract later in fiscal year 2018 after confirming the foreign ownership with the contractor. DIA and DLA officials stated that adversarial foreign governments or other malicious entities, such as companies attempting to access sensitive government information, could access sensitive systems to conduct sabotage or surveillance. These entities could infiltrate DOD’s supply chain to introduce components, such as circuit-board chips and routers modified to fail, facilitate state or company espionage, or compromise the integrity of DOD’s information-technology systems. According to CIO officials, adversarial entities could also potentially gain access to sensitive information through their relationship with DOD contractors. For example, DIA officials identified the possibility of foreign or adversarial entities exploiting companies in DOD’s supply chain with financial difficulties, and according to CIO officials, DOD may not always have visibility over foreign entities acquiring a domestic contractor. In 2017, the Office of the Director of National Intelligence released a management background paper discussing supply-chain risks, which stated that the multiple layers and networks of suppliers in this chain can allow foreign adversaries the ability to access the supply chain at multiple points. For example, according to the background paper, a hostile foreign intelligence entity could potentially conceal its presence in government supply chains by operating through multiple front organizations, companies, hackers, and organized crime, making it extremely difficult to discover and counter its actions. The paper also states that adversaries may be able to penetrate the supply chain to access sensitive research and development programs, steal intellectual property and personally identifiable information, insert malware into critical components, and mask foreign ownership, control, or influence of key providers of components and services. Furthermore, in April 2018, the U.S.-China Economic and Security Review Commission issued a report identifying a supply-chain threat to U.S. national security that stems from products produced, manufactured, or assembled by entities that are owned, directed, or subsidized by national governments or entities known to pose a supply- chain or intelligence threat to the United States. DOD officials have also identified an additional risk of contracting with companies that have opaque ownership structures. For example, a 2017 Defense Contract Audit Agency report to Congress described the risk of individuals receiving government contracts or gaining access to government installations who would harm deployed troops. Officials we spoke with from the Joint Staff Logistics Directorate also acknowledged the risk that government funds could be provided to contractors owned by a person or entity that is actively opposing U.S. or coalition forces involved in a contingency operation in which service members are actively engaged in hostilities. These adversaries can potentially use opaque ownership structures to disguise their ownership and contract with the government in areas involved in contingency operations, such as Iraq or Afghanistan, to fund their operations or gain access to military bases. DOD has taken steps that could address some fraud and other risks related to contractor ownership in the procurement process. It has not yet conducted a department-wide assessment of these risks or identified them as a risk area for assessment in its development of a fraud risk management program in accordance with federal internal control standards and leading practices, however. As mentioned previously, DOD and other federal agencies revised the FAR in 2014 to collect some contractor ownership information. DOD has also begun to consider contractor ownership to address national security risks, including identifying and using contractor ownership information as part of its supply-chain risk analysis in the procurement of national security systems and critical components, avoiding contracting with the enemy, and determining whether contractor facilities can be cleared to access classified materials. Although DOD has taken these actions, it faces a number of challenges in identifying and verifying contractor ownership. To assist the department and its components in identifying and assessing fraud risks, DOD has also begun a department-wide fraud risk management program. As it develops a fraud risk assessment across the department, DOD has opportunities to systematically assess risks related to contractor ownership as part of this larger effort. This fraud risk assessment, if used to inform the development of a risk-based antifraud strategy, could enhance the effectiveness of managing fraud risks for DOD, including those related to contractor ownership. DOD, GSA, and the National Aeronautics and Space Administration amended the FAR in May 2014 to require prospective contractors to self- report their immediate and highest-level entity owner, but not their beneficial owner, as part of contractors’ annual registration process in SAM. The agencies added the requirement to support the implementation of business tools to help track contractor performance issues across corporations as well as to improve supply-chain transparency and integrity efforts, among other reasons. According to DOD procurement policy officials, the intent is that the ownership information would be made available in FAPIIS for contracting officers to help identify past-performance issues across corporations to aid with responsibility determinations. The FAR requires contracting officers to consider all relevant information available in FAPIIS when making responsibility determinations, but, according to DOD procurement policy officials, there is no requirement to document whether and how ownership information is considered. According to DOD procurement policy officials, contracting officers’ general focus in the responsibility determination process is largely centered on whether the contractor is financially solvent, has the ability to carry out the contract, and has satisfactory past performance. DOD procurement policy officials said that they did not want to be too prescriptive in directing contracting officers on the use of this information, and therefore have not developed policies or procedures or provided training on how to specifically use the ownership information collected. According to these officials, DOD has not historically considered contractor ownership structures in the responsibility determination process, nor has the agency been aware of the extent to which such structures could pose a range of risks. As discussed below, conducting a department-wide assessment of risks posed by contractor ownership—an action that DOD has not yet taken—would be a key first step for the department before developing such policies and procedures. Within DOD, DLA has taken steps that could address some risks posed by contractor ownership. First, according to procurement officials, DLA provides its contracting officials with a “contractor responsibility matrix,” which outlines mandatory, recommended, and optional steps to take when making a responsibility determination for procurements both below and above the simplified acquisition threshold. Among the steps included, DLA requires contracting officials to review contractors’ attestations to ownership or control by a foreign government to determine whether the prospective contractor is qualified and eligible to receive an award. It also recommends contracting officials obtain responsibility information from other sources, including an internet search of the company’s reviews, and its owners and principals. This step is listed as optional for existing contractors. Further, DLA’s contracting officers are required to review the Defense Contractor Review List to identify any past-performance information. The Defense Contractor Review List is an internal tool used by DLA that is designed to monitor fraud, waste, and abuse for commercial entities and military unique items. The system is designed to allow DLA to identify and communicate information on its contractors, such as performance ability, delinquency information, suspension and debarment information, and various types of notes that may be relevant to contract performance or procurement decisions. DLA officials told us the Defense Contractor Review List can be used to communicate information or risks about contractor ownership. The Defense Logistics Acquisition Directive requires DLA contracting officers to review any Special Attention Reason Codes in the Defense Contractor Review List and comply with its associated Special Attention Treatment Codes when making responsibility determinations. The Special Attention Reason Codes describe the basis for being on the list and the Special Attention Treatment Codes provide recommended actions to contracting officers for mitigating risk. According to DLA officials, contractor ownership information is generally not identified in the Defense Contractor Review List. Nevertheless, ownership information may be included in the documentation if, for example, the contracting officer identifies that two or more companies appear to be related or in cases in which there may be suspected collusion. DOD has taken steps in other areas to use contractor ownership information to address risks in specific types of procurements, including those involving national security systems. For example, DOD has taken steps to address national security concerns related to contractor ownership, including conducting threat assessments to identify risks related to supply chains for critical components and national security systems. DOD has also taken steps to identify contractor ownership information to avoid contracting with the enemy, and to address foreign ownership, control, and influence in contracts involving classified information. DOD has outlined policies and procedures in some, but not all, of these areas. As discussed below, conducting a department-wide assessment of risks posed by contractor ownership—an action that DOD has not yet taken—would be a key first step for the department before fully developing such policies and procedures. Steps taken to use ownership information to address supply chain risks. DOD has taken some steps to identify and consider contractor ownership to address supply-chain risks. For example, DIA considers contractor ownership information when conducting threat assessments as part of its supply-chain risk analysis for procurement of national security systems and critical components, according to DIA officials. Specifically, DOD is able to use public and nonpublic intelligence information to exclude sources that present risks of an adversarial foreign government or other malicious entities infiltrating DOD’s supply chain and stealing information or compromising government systems. DIA officials told us that, as part of this supplier- related threat assessment, they identify and consider ownership information along the supply chain, including beneficial-ownership information. The guidelines in Intelligence Community Standard 731-02 state that a supply-chain threat assessment for a procurement item determined to be mission-critical should at a minimum include information on the contractor’s parent company, ultimate parent company, and subsidiaries. However, the guidance does not specify whether this ownership and related company information is to be independently verified or whether it relies on the contractor self-attestations in SAM. According to the guidance, supply-chain threat assessments should also include, at a minimum, information on the contractor’s key management personnel, such as members of the board of directors, officers, general partners, and senior management officials. The guidance does not mention, however, identifying beneficial owners or those who do not have direct control over a contractor but derive substantial economic benefit from it. Steps taken to use ownership information to address legal provisions against contracting with the enemy. Officials from the Joint Staff Logistics Directorate responsible for DOD’s vendor vetting program told us that contractor ownership information, including beneficial ownership, may be identified as part of the intelligence information gathered on vendors by combatant commands to ensure that money is not flowing to contractors owned by a person or entity that is actively opposing U.S. or coalition forces involved in a contingency operation in which service members are actively engaged in hostilities. According to these officials, DOD has not established department-wide policies or procedures to implement reviews of contractor ownership during the process of vetting vendors, but it is something the department is currently developing. These officials stated that a vendor threat-mitigation working group discusses how to close gaps in information sharing among the intelligence, procurement, and operations communities. Officials also noted some challenges. Although contracting officers are responsible for determining the responsibility of vendors and whether vendors can perform the terms of a contract, the information that may be available to contracting officers and the actions that they can take are not always clear. For example, the officials we spoke with mentioned concerns that contracting officers are not always able to access or act on intelligence information. GAO recently completed a review of this program in a classified report. Steps taken to address ownership risks in contracts involving classified information. DOD has taken steps to address risks posed by contractor ownership as part of the Facilities Clearance Process. DOD uses the Facilities Clearance Process to determine whether a contractor is eligible to access classified information. DOD has developed written policies and procedures for how contractor ownership, including foreign ownership, control, and influence, is to be investigated and addressed. As part of this process, Defense Security Service (DSS) guidance instructs its officers to identify key management personnel and to assess the risks they pose for possible foreign ownership, control, or influence. DSS guidelines indicate that key management personnel include company officers, directors, and members of a limited liability company, among others. Some key management personnel, such as members of a limited liability company, may also be the owners. According to DSS officials, beneficial owners who benefit financially but do not partake in active management may be identified as key management personnel as part of the clearance process, depending on various factors including the percentage of ownership. As an example, DSS officials stated that an individual who owns 50 percent of a company would not be able to purport that he or she does not control the company. According to the DSS guidance, if foreign ownership, control, or influence is found, mitigation agreements can be put into place to reduce the risk. DOD officials identified a number of challenges in identifying and verifying contractor ownership, especially if the contractor is actively seeking to misrepresent its ownership. For example, verifying contractor ownership can be challenging because state governments determine the type of information collected during company formation and, as discussed earlier, most states collect minimal ownership information as part of this process. As described earlier, there is no centralized information source or registry on company ownership information in the United States. As a result, contracting officers could face challenges in time-consuming efforts to verify contractor ownership. Further, DOD procurement policy officials stated that workload and resource constraints limit the extent to which they can verify contractor ownership. The nature of ownership information submitted during the SAM registration process also presents challenges to any verification efforts conducted by contracting officers. The ownership information submitted in SAM is self-reported by the prospective contractor, and therefore relies on the contractor to honestly report such information. DOD officials told us that, for most procurements, with the exception of those involving classified work or other national security concerns, this information is not verified. A related limitation involving SAM ownership information is that contractors must provide information on the immediate and highest-level entity owners and are not required to report beneficial-ownership information, that is, on the natural person or persons who own or control, or benefit financially from, the company. Lastly, while the SAM ownership requirement provides some transparency at the prime-contractor level, it does not provide transparency at the subcontracting levels below the prime contractor. Subcontractors are not required to register in SAM and, therefore, are not required to report their ownership. Consequently, DOD generally does not have insight into the ownership of its subcontractors. DOD procurement policy officials noted that this poses particular challenges in identifying fraud and other risks to the supply chain. For example, the contractor itself may not pose a risk; but that does not guarantee that the contractor’s suppliers do not pose fraud or other risks. DOD procurement policy officials told us that it would be helpful to require subcontractors to register in SAM and report their ownership. This requirement would be an additional burden on contractors, however, and would need to be balanced with the potential benefit of being able to identify problem actors. Another challenge involves the use of publicly available ownership information, including commercially available data services, by contracting officers to help identify contractor ownership. Depending on how a company is structured, there may be no publicly available ownership information. Furthermore, DOD procurement policy officials told us that public information, including ownership information, could be inaccurate or outdated and potentially expose the department to bid protests from the contractor. Therefore, any external or supplemental information used that was not part of the contractor’s submission would need to be vetted by the contractor before using it. These officials said that DOD would need to come up with an efficient process to inform the prospective contractor of the additional information and provide due process to allow it the opportunity to refute any information obtained. Additionally, DOD procurement policy officials noted that another difficulty with using a commercial tool to determine ownership is the volume of contracts processed by contracting officials, which amounted to over 570,000 new contracts in fiscal year 2018. For sensitive procurements in which DOD has the authority to use both public and nonpublic information (for example, those involving national security systems or classified work), DSS officials stated that the process of identifying and verifying ownership is lengthy, particularly with complex ownership. In some instances, it has taken DSS 1 to 2 years to resolve issues that have arisen when clearing contractors’ facilities for access to classified materials. In addition, DSS officials mentioned that the many different types of business structures, including new structures that DSS comes across, create challenges for identifying ownership. According to DIA officials, it is significantly easier to identify the beneficial owner of publicly traded companies than privately owned companies. DSS officials also mentioned that it is difficult and resource-intensive to monitor changes to contractor ownership, particularly given that they monitor 13,000 facilities. According to DOD procurement policy officials, DOD would need to determine which contracts require additional research into contractor ownership and which office would be responsible for conducting the research. Officials noted that DOD does not currently have the resources in place to focus on these kinds of activities because contracting officers are already operating in a constrained environment with limited resources, lacking the time, resources, or training they need to conduct in-depth reviews or analysis of the ownership aspects of a particular company. According to these officials, DOD should dedicate staff and funds to resolve this problem, including bringing in people with data- analysis and data-mining skillsets to learn from private-sector companies and organizations that already conduct vendor ownership-related risk assessments and data analytics. DOD procurement policy officials identified that another strategy to address opaque ownership structures would be to require contractors to report additional ownership information, such as beneficial-ownership information, when registering to do business with the federal government in SAM. However, the officials also noted that, previously, both public- sector organizations and private companies have resisted requirements to provide additional ownership information, due in part to the difficulty in defining ownership. Additionally, regulatory trends within government contracting have generally focused on easing the burden to do business with the government. New requirements to provide additional information may be viewed as an additional burden. A selected group of companies that contracted with DOD in the last 5 years provided us with mixed views on the potential burden of providing additional ownership information. Most small-business contractors we contacted told us that an additional beneficial-ownership reporting requirement would pose little to no further burden on them. In contrast, both of the large, publicly traded companies that similarly contracted with DOD expressed concerns about the complexity and difficulty of reporting their beneficial ownership. One large company noted that beneficial ownership would need to be more narrowly defined for it to determine the resulting regulatory burden. DOD has taken steps to conduct a department-wide fraud risk management program designed to identify and assess fraud risks. According to DOD’s Fraud Risk Management Pilot Program Instructions, in 2017 DOD began efforts to design, implement, and operate an internal control system that addresses fraud risks and to comply with requirements established by FRDAA. As mentioned earlier, FRDAA created requirements for agencies to establish financial and administrative controls for managing fraud risks. FRDAA also requires agencies to report their progress identifying risks and vulnerabilities to fraud affecting payroll, beneficiary payments, grants, purchase and travel cards, and large contracts. As part of this implementation process, and to test the development of its fraud risk management program, DOD conducted a fraud risk management pilot program in 2018 by selecting four components to identify fraud risks, assess controls they have in place to mitigate these risks, and develop mitigation plans, as appropriate. According to DOD, the pilot program was designed to assist DOD and its components in the development of a department-wide fraud risk management program by identifying and assessing fraud risks in a manner that is aligned with the leading practices within GAO’s Fraud Risk Framework. To prepare for this pilot program, in 2017, the Office of the Under Secretary of Defense (Comptroller) (OUSD) conducted a survey requesting that 66 DOD components determine the extent and maturity of control activities currently in place related to the prevention, detection, and response to fraud. The survey asked components to provide, among other things, information on any antifraud programs, key fraud risks identified, and processes for identifying, responding to, and monitoring risks. The responses from the 41 responding components were scored to determine their fraud program maturity. According to DOD’s Fraud Risk Management Pilot Program Instructions, the results of this survey were also used to identify potential vulnerabilities from the FRDAA requirements and guide the development of DOD’s pilot program. DOD officials told us that before the recent development of their fraud risk management pilot program, the department did not have a process for assessing fraud risks department-wide. Also, as part of the pilot program, OUSD(C) and the components identified seven fraud schemes that affect large contracts, five of which we discuss above as having the potential to involve risks posed by contractor ownership. Specifically, the pilot program identified fraud schemes involving service-disabled veteran–owned businesses, inflated prices charged by contractors for the services rendered, bid submission with the same two or three offerors on multiple contract opportunities, inclusion of one or more contractors as a subcontractor on the bid rigger’s proposal, and counterfeit parts. As discussed previously in this report, opaque ownership structures can play a role in carrying out these types of fraud schemes. DOD completed the pilot program in 2018, and in March 2019 began expanding the fraud risk management program department- wide by requesting that DOD components identify fraud risk and controls in place to mitigate these risks by July 2019. As with the pilot program, the components were requested to identify and assess fraud risks to meet requirements established by FRDAA and allow DOD to identify fraud risks and vulnerabilities facing the department. While DOD has taken some steps to identify and potentially address fraud and other risks posed by contractor ownership, it has not conducted a department-wide assessment of these risks or selected them as a risk area for assessment in its development of a fraud risk management program. DOD procurement policy officials told us that contractor ownership and financing structures have not historically been considered by the department. DOD procurement policy officials expressed the need for a strategic assessment of contractor ownership risks at the Office of the Secretary of Defense (OSD) level to deal with the wide range of potential threats that exist. Still, getting support at the senior OSD level to consider the risks posed by contractor ownership and dedicate resources to mitigating these risks is a challenge, according to these officials. The challenge exists because senior DOD officials may not be aware of the potential magnitude or frequency of risks posed by contractor ownership issues, including the extent to which risks cross multiple areas throughout the department. Additionally, DOD procurement policy officials told us that contracting officers do not have anyone within the department to contact for assistance in determining ownership during the procurement process and there is no dedicated entity within the department that deals with contractor ownership issues. Federal internal control standards call for agency management officials to assess the internal and external risks their entities face as they seek to achieve their objectives. The standards state that as part of this overall assessment, management should consider the potential for fraud when identifying, analyzing, and responding to risks, including changes to risks, and consider factors such as absent or ineffective controls that provide an opportunity to commit fraud. In a complementary fashion, the Assess component of GAO’s Fraud Risk Framework calls for federal managers to plan regular fraud risk assessments and to identify and assess risks to determine a fraud risk profile, as described in figure 10 below. According to the Fraud Risk Framework, a fraud risk profile documents the findings from a fraud risk assessment and can help agencies decide how to allocate resources to respond to residual fraud risks. The Assess component also indicates that relevant stakeholders, including those with responsibilities for specific control activities and with knowledge of emerging fraud risks, should be involved in the assessment process. This could include a variety of internal and external stakeholders, such as general counsel, contractors, or other external entities with knowledge about emerging fraud risks or responsibilities for specific control activities. For example, the DOD Office of Inspector General and its work on emerging risks involving contractor ownership may inform the fraud risk assessment process and help managers to identify fraud risks. Additionally, an assessment of ownership risks could include relevant DOD officials responsible for assessing and responding to national security risks, such as those responsible for assessing supply- chain risks in national security system procurements, vetting vendors to ensure DOD avoids contracting with the enemy, and determining whether contractor facilities can be cleared to access classified materials. Including relevant stakeholders would allow DOD to leverage the knowledge and experience of such officials and more comprehensively identify risks related to contractor ownership. Further, it would allow DOD to better understand the extent to which risks cross multiple areas throughout the department. At a fundamental level, assessing risks arising from contractor ownership would allow DOD to take a strategic, risk-based approach to identifying and managing these risks. In addition, a risk assessment would help DOD better understand the magnitude and prevalence of these risks, including the effects these risks have from both a fraud and national security perspective, and whether certain types of procurements are more vulnerable to contractor ownership risks. Further, conducting a department-wide assessment of risks posed by contractor ownership would assist the department in its evaluation of whether its existing control activities are sufficient and designed to effectively respond to these risks or whether additional control activities are needed. For example, it would allow DOD to better determine how contractor ownership information should be used and verified, and whether additional ownership information should be collected. In accordance with leading practices, DOD would then be positioned to design and implement specific control activities to prevent and detect contract ownership-related fraud and make informed decisions on how best to use its resources. DOD is the largest contracting agency in the federal government in terms of contract dollars obligated and number of contracts awarded. The scope and scale of this activity makes DOD procurement inherently susceptible to fraud. Our various analyses and discussions with procurement officials from across the department identified risks posed by contractors with opaque ownership that involve various types of procurements. DOD has taken some steps that could address some risks posed by contractor ownership in the procurement process. It has the opportunity to include these risks as part of its department-wide fraud risk assessment at a strategic level. Assessing risks related to contractor ownership, as a fundamental first step, would help DOD better determine whether certain types of procurements are more vulnerable to this type of risk. Further, it would help DOD determine whether additional policies and procedures are needed to articulate how officials should use and verify the ownership information it collects, or to require additional ownership information. We recognize that collecting additional ownership information, including beneficial-ownership information, could pose compliance burdens for contractors; and regulatory trends have generally focused on easing the burden to do business. Additionally, verifying contractor ownership can be challenging and time-consuming. Nevertheless, having a thorough assessment of contractor-ownership risks will better position DOD to make informed decisions on how best to use its resources and help ensure that the department’s fraud risk management program is organized and targeted to manage risks in a prioritized manner. Lastly, involving relevant stakeholders with knowledge of emerging risks could help inform other types of risk assessments across the department, including national security concerns. Doing so will contribute to the effective implementation of leading fraud risk management practices when considering the existing and emerging risks to the department. The Office of the Undersecretary of Defense (Comptroller) (OUSD) should include an assessment of risks related to contractor ownership as part of its ongoing efforts to plan and conduct a department-wide fraud risk assessment. As part of this assessment, consistent with leading practices, DOD should involve relevant stakeholders with knowledge of emerging risks and use this information to help inform other types of risk assessments across the department, including for national security concerns. (Recommendation 1) We provided a draft of the sensitive version of this report to DOD and GSA for comment. In commenting on a draft of the sensitive version of this report, DOD concurred with our recommendation and provided additional written comments outlining current and planned efforts in response to our recommendation. These written comments were deemed sensitive by DOD and have been omitted from this report. In an email, GSA stated that it did not have any comments. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Administrator of GSA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6722 or bagdoyans@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made contributions to this report are listed in appendix III. This report is a public version of a sensitive report that we issued on September 12, 2019, with the objectives to (1) identify types of fraud and other risks, if any, that contractors with opaque ownership could pose to the Department of Defense (DOD) in the procurement process and (2) assess whether DOD has taken steps to address risks posed by contractor ownership in the procurement process. The sensitive report included the results of data analysis we conducted to identify offerors who might disguise their ownership to create the appearance of competition. DOD deemed some of the details from this analysis to be sensitive, which must be protected from public disclosure. This report also omits sensitive information about ongoing investigations, certain internal controls and vulnerabilities, and actions taken to address some of these vulnerabilities. Although the information provided in this report is more limited, it addresses the same overall objectives as the sensitive report and uses the same methodology. To address our first objective, we researched information on closed cases investigated by the Defense Criminal Investigative Organizations or prosecuted by the Department of Justice (DOJ) from calendar years 2012 through 2018. These cases were identified by researching press releases from the websites of the DOJ Office of Public Affairs, Offices of the U.S. Attorney, DOD Office of Inspector General, and Defense Criminal Investigative Organizations. We also researched legal databases and news articles involving DOD contractors to identify federal court cases and federal agency decisions. We reviewed GAO bid-protest decisions to identify cases in which a contractor may have failed to disclose foreign ownership or concealed beneficial-owner information to obtain contracts that they were not eligible to receive. We interviewed investigators from the Defense Criminal Investigative Organizations and DOD contracting offices to supplement our research. For each case identified, we reviewed the associated federal court filings or DOJ press releases to determine the outcome of the case and how contractor ownership was used or concealed to carry out the offense. To identify additional types of risks that may not have been identified through our case-study research, we interviewed officials from the General Services Administration (GSA) and officials from across DOD, including the Office of Inspector General, Defense Criminal Investigative Organizations, Defense Pricing and Contracting, the Office of the Under Secretary of Defense (Comptroller) (OUSD), the Office of the Chief Information Officer, Defense Intelligence Agency (DIA), Defense Security Service (DSS), Defense Logistics Agency (DLA), Defense Contract Management Agency (DCMA), and Defense Contract Audit Agency, and relevant procurement policy officials from the Departments of the Army, Navy, and Air Force. We examined known risks identified through our case-study research and interviews with DOD officials; however, these risks are not necessarily representative of the extent or the types of these risks. There may be additional fraud or other risks and cases related to contractor ownership that are presently undiscovered fraud and are not identified in our report. Additionally, we further examined the risk that contractors could be disguising their ownership to create the appearance of competition on a contract to inflate prices by analyzing bid response data from GSA’s Federal Business Opportunities (FBO) website and registration data in GSA’s System for Award Management (SAM). Specifically, we analyzed responses to approximately 2,700 solicitations submitted for fiscal years 2015 through 2017 to identify indications of potentially related offerors bidding on the same solicitation. We selected this date range because fiscal year 2015 was the first year in which the Federal Acquisition Regulation (FAR) required offerors to report their ownership and fiscal year 2017 was the most-recent complete year of data at the time of our analysis. To identify whether offerors were potentially related, we analyzed information to identify instances in which different offerors shared certain information. Offerors sharing information does not definitively prove that the offerors are related or share ownership; however, it is an indicator that these offerors may not be independent of each other. For offerors we identified as potentially related, we researched company websites and third-party data sources to determine whether we could find other indicators of a relationship. Further, we provided a list of the potentially related offerors we identified to the relevant DOD contracting office to determine whether the offeror disclosed any relationships to other offerors or whether the contracting officer was otherwise aware of the relationship with another offeror. The results of our analysis are limited to the approximately 2,700 solicitations we reviewed and are not generalizable to other DOD solicitations. To assess the reliability of the data used in our analysis, we performed electronic testing to determine the validity of specific data elements in the FBO bid module and other datasets. We also reviewed documentation related to these databases, compared the data to published sources and source documentation maintained in the DOD contracting files, and interviewed GSA officials responsible for these databases. We determined that the data were sufficiently reliable for the purposes of analyzing potential ownership relationships. To address our second objective, we reviewed federal laws, the FAR, DOD regulations, directives, instructions, policies, procedures, and training documents. We also reviewed OUSD(C) fraud assessment templates and preliminary results from DOD’s fraud risk management pilot program. We interviewed procurement policy officials from GSA, Defense Pricing and Contracting, DLA, and the Departments of the Army, Navy, and Air Force as well as officials from the Office of the Chief Information Officer, OUSD(C), DIA, DSS, DCMA, the Defense Contract Audit Agency, the Joint Staff Logistics Directorate, the Defense Industrial Policy office, members of DOD’s Procurement Fraud Working Group, and the Naval Contracting Council to discuss how DOD has addressed risks. We also interviewed officials from the Defense Acquisition University to determine how, if at all, DOD trained contracting officials to consider risks posed by contractor ownership. To assess these efforts, we compared these documents and the information from our interviews to federal internal control standards and the leading practices outlined in GAO’s Framework for Managing Fraud Risks in Federal Programs. To gain the perspectives of contractors on whether a requirement to report beneficial- ownership information when doing business with DOD would impose a burden on companies, we researched and contacted several government contractors’ associations to gain the perspectives of their members. The contractors’ associations we contacted included associations for large, medium, and small businesses working in a variety of industries doing business with the government. We received responses to our inquiries from three associations. To gain their members’ perspectives, officials from the three associations forwarded our inquiries to their members and we received responses from 16 members. These 16 members were from a range of business sizes and industries. The perspectives gained from our queries are limited to the contractors from whom we received a response and are not generalizable to all contractors. We conducted this performance audit from August 2017 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with DOD from September 2019 to November 2019 to prepare this version of the original sensitive report for public release. This public version was also prepared in accordance with these standards. The table below summarizes the information we reviewed involving Department of Defense (DOD) contractors or subcontractors that provided false information about ownership or corporate structure to allegedly commit fraud. We identified cases involving contractors that posed financial and nonfinancial risks to DOD (see app. I for additional details on the methodology used). Financial risks we identified involved DOD contractors using opaque ownership structures to fraudulently inflate prices on DOD contracts. We also identified subcontractors that misrepresented ownership or shared common ownership with a contractor for the purpose of obtaining awards or overcharging the government. Nonfinancial risks we identified involved contractors bidding on and obtaining contracts that they were not eligible to receive, including contracts set aside for small businesses owned by service-disabled veterans or socially and economically disadvantaged individuals. We also identified cases involving ineligible foreign manufacturers creating domestic shell companies to obtain government contracts. As discussed in our report, DOD has not assessed risks posed by contractor ownership; therefore the magnitude and prevalence of these risks are not known. There may be additional risks and cases related to contractor ownership that are not identified below. The 32 cases below were adjudicated or settled from calendar years 2012 through 2018. As shown in the table below, we used public court records and Department of Justice and DOD press releases to identify the type of fraud and calendar years in which the cases were adjudicated or settled, a summary of how the contractor’s ownership was disguised or obfuscated to carry out the fraud schemes, dollar amount awarded or received to the extent available in each case, and the government agencies affected by the fraud. In addition to the contact named above, the following staff members made key contributions to this report: Tonita Gillich (Assistant Director); Tracy Abdo (Analyst-in-Charge); Marissa Esthimer; Colin Fallon; Mollie Lemon; Maria McMullen; Madeline Messick; Dustin Milne; Lauren Ostrander; Daniel Purdy; Daniel Silva; Sabrina Streagle; and Shana Wallace. Others who contributed to this report include Steven Campbell, Suellen Foth, and Pamela Snedden.", "summary": "DOD generally accounts for about two-thirds of federal contracting activity. Some companies doing business with DOD may have an opaque ownership structure that conceals other entities or individuals who own, control, or financially benefit from the company. Opaque ownership could be used to facilitate fraud and other unlawful activity. The House Armed Services Committee report on the National Defense Authorization Act for fiscal year 2018 included a provision for GAO to examine the risks posed by contractors with opaque ownership and DOD's processes for identifying ownership. This report identifies types of fraud and other risks that opaque contractor ownership poses to DOD in the procurement process and assesses whether DOD has taken steps to address those risks. GAO reviewed applicable laws and regulations and interviewed DOD officials, including procurement staff and criminal investigators. GAO researched cases from 2012–2018 where contractors may have concealed or failed to disclose ownership information. GAO compared DOD's efforts to leading practices in GAO's Fraud Risk Framework. This is a public version of a sensitive report that GAO issued in September 2019. Information that DOD deemed sensitive involving ongoing investigations and certain internal controls and vulnerabilities has been omitted. The Department of Defense (DOD) faces several types of financial and nonfinancial fraud and national security risks posed by contractors with opaque ownership. These risks, identified through GAO's review of 32 adjudicated cases, include price inflation through multiple companies owned by the same entity to falsely create the appearance of competition, contractors receiving contracts they were not eligible to receive, and a foreign manufacturer receiving sensitive information or producing faulty equipment through a U.S.-based company. For example, one case involved an ineligible foreign manufacturer that illegally exported sensitive military data and provided defective and nonconforming parts that led to the grounding of at least 47 fighter aircraft, as illustrated below. DOD has taken some steps that could address some risks related to contractor ownership in the procurement process but has not yet assessed these risks across the department. DOD, in coordination with other agencies, revised the Federal Acquisition Regulation in 2014 to require contractors to self-report some ownership information. DOD has taken steps to identify and use ownership information—for example, as part of its supply-chain risk analysis when acquiring critical components. DOD has also begun a department-wide fraud risk management program, but it has neither assessed risks of contractor ownership across the department nor identified risks posed by contractor ownership as a specific area for assessment. Assessing risks arising from contractor ownership would allow DOD to take a strategic approach to identifying and managing these risks, make informed decisions on how to best use its resources, and evaluate its existing control activities to ensure they effectively respond to these risks. GAO recommends that DOD assess risks related to contractor ownership as part of DOD's ongoing efforts to assess fraud risk. DOD should use this information to inform other types of risk assessments, including national security concerns. DOD concurred with GAO's recommendation.", "document_type": "gao"}
{"report": "The naval shipyards were originally designed to build wind- and steam- powered ships and range in age from 111 years to 252 years old (see fig. 1). Over the years, the Navy has adapted them into highly industrialized, large-scale operations that are essential to national defense and fulfill the legal requirement for DOD to maintain a critical logistics capability that is government owned and operated to support an effective and timely response for mobilization, national defense contingency situations, and other emergency requirements. However, as we have reported, the shipyards’ age, residual configuration for the shipbuilding mission, and poor condition reduces their efficiency for their modern-day mission of repairing nuclear-powered ships and submarines. The naval shipyards perform depot-level maintenance, which involves the most comprehensive and time-consuming maintenance work, including ship overhauls, alterations, refits, restorations, nuclear refuelings, and inactivations—activities crucial to supporting Navy readiness. This maintenance is performed during periods designated in the Navy’s Optimized Fleet Response Plan, an operational schedule of maintenance, training, and deployment periods for the entire fleet. It is designed to maximize the fleet’s operational availability to combatant commanders while ensuring adequate time for the training of personnel and maintenance of ships. We reported in 2016 that successful implementation of the Optimized Fleet Response Plan depends, in part, on the shipyards completing maintenance on time and that maintenance delays reduce the time that ships are available for training and operations. As a result, successful implementation of the Optimized Fleet Response Plan is essential to the Navy’s ability to maintain readiness and support operational needs. The Navy developed the congressionally directed Shipyard Infrastructure Optimization Plan (the plan) to mitigate infrastructure deficiencies at the public shipyards. For some infrastructure, the Navy had preexisting planning that it used to outline specific mitigation projects that would address deficiencies. For other aspects of its infrastructure, the plan outlines the Navy’s strategy for developing a more detailed mitigation approach. The plan serves as the Navy’s engineering analysis and strategy for the optimal placement of facilities and major equipment at each public shipyard, including a 20-year investment plan for infrastructure investments needed to improve shipyard performance. The plan proposes mitigations to address limitations with three major facets of the public shipyards’ operations: their dry docks, facilities, and capital equipment (see fig. 2). Navy officials said they integrated previous studies in these three areas to create the plan. For example, Naval Sea Systems Command (NAVSEA)—which is responsible for program management of the shipyards—completed a dry dock study that identified gaps in capacity and configuration, which served as the basis for the dry dock portions of the plan. In addition, the Navy had previously developed capital investment strategies intended to help improve the state of the shipyards’ facilities and equipment, which were also included in the plan. The Navy estimates that the plan could eventually save 328,000 labor-days each year and recover most of the maintenance periods it currently cannot support. Capital planning is the process by which an organization prepares for the acquisition of capital assets such as the facilities and equipment in the Navy’s plan. Congress, the Office of Management and Budget, and we have identified the need for effective capital planning, which can help ensure that capital funds are spent productively. In the overall capital programming process, planning is the first phase, and it drives the remaining phases of budgeting, procurement, and management. For decision makers to conduct effective capital planning, they must have reliable cost estimates. A reliable cost estimate is critical to the success of any program. Such estimates provide the basis for informed investments, realistic budgets, meaningful measurement of progress, proactive course correction, and accountability for results. According to the Office of Management and Budget, cost estimates should be well-documented and updated on a regular basis. Estimates should also encompass life-cycle costs of the program. Without high-quality estimates, agencies are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls. The GAO Cost Estimating and Assessment Guide has identified a number of best practices grouped into four “characteristics” that are the basis of effective program cost estimating and should result in reliable and valid cost estimates that management can use to make informed decisions, as shown in figure 3 and discussed below. Comprehensive: A comprehensive cost estimate includes all costs of the program over its complete life cycle, from the start of the program through design, development, deployment, operation and maintenance, and retirement. It also fully defines the program, reflects the current schedule, and is technically reasonable. Comprehensive cost estimates provide sufficient detail to ensure that cost elements are neither omitted nor double counted. Finally, where information is limited and judgments must be made, the comprehensive cost estimate documents all cost-influencing ground rules and assumptions. Well-documented: A well-documented cost estimate is supported by detailed documentation that describes how it was derived and how the funds will be spent in order to achieve a given objective. Therefore, the documentation includes such things as the source data used, the calculations performed and their results, and the estimating methodology. Moreover, this information is captured in such a way that the data used can be easily replicated and updated. The documentation also discusses the technical baseline and how the data were standardized. Finally, the documentation includes evidence that the cost estimate was reviewed and accepted by management. Accurate: An accurate cost estimate provides results that are unbiased, and is not overly conservative or optimistic. An estimate is accurate when it is based on an assessment of the most likely costs, adjusted properly for inflation, and contains few, if any, minor mistakes. In addition, an accurate cost estimate is updated regularly to reflect significant changes in the program—such as when schedules or other assumptions change—and actual costs, so that it always reflects the program’s current status. During the updating process, differences between planned and actual costs are documented, explained, and reviewed. Among other things, the estimate is grounded in a historical record of cost estimating and actual experiences on comparable programs. Credible: A credible cost estimate discusses any limitations of the analysis resulting from uncertainty or biases surrounding the data or assumptions. Major assumptions should be varied and other outcomes recomputed to determine how sensitive they are to changes in the assumptions. Risk and uncertainty analysis are performed to determine the level of confidence associated with the estimate. Finally, an independent cost estimate is developed by a group outside the organization to determine whether other estimating methods produce similar results. The Shipyard Infrastructure Optimization Plan has identified a number of infrastructure deficiencies at the Navy’s four public shipyards—including deficiencies in dry docks, facilities, and capital equipment—that negatively affect their ability to support the Navy’s current and future force structure. However, the extent to which the Navy’s plan addresses these deficiencies remains to be seen because facility planning has not been completed and the proposed actions are complex and years away from being implemented. The Navy’s plan outlines steps that generally address the critical dry dock deficiencies the Navy has identified, although it does not anticipate completing these steps until 2035. Of the shipyards’ 18 dry docks, the plan states that eight require modernization and recapitalization projects to meet the Navy’s operational needs, including accommodating new classes of ships. If all of the projects are completed as planned, the Navy anticipates that it will be able to recover 67 of the 68 maintenance periods that it currently cannot support through fiscal year 2040. According to the Navy, without these planned dry dock investments, the Navy would lack sufficient capacity for about a third of its planned maintenance periods at the shipyards and would have to defer maintenance for some ships. This could result in ships being unavailable for use until a dry dock is available, effectively reducing the size of the fleet available for operational missions. However, the extent to which the plan’s actions will address the shipyards’ dry dock deficiencies remains to be seen because the initiation and completion of many of these projects is years away. Built in 1919, dry dock 3 is not certified to handle nuclear fuel, which means submarines must be defueled elsewhere before this dock can be used, according to Navy officials. Additionally, because of its shallow depth, the Navy can move Los Angeles–class submarines into or out of the dock only during high tides. Even then, the shipyard has to remove portions of the submarine to decrease weight and over-flood the dock to create enough clearance for the boat. Shipyard officials said dry dock 3 could be modified to create a new multimission dry dock (M2D2) that would provide an additional spot to dry dock an aircraft carrier on the West Coast. This would provide the redundancy necessary to allow the Navy to perform significant seismic upgrades to dry dock 6, which faces significant seismic risks and is the only dry dock on the West Coast capable of accommodating an aircraft carrier. Navy officials said the final decision regarding the location of the M2D2 is pending a formal Environmental Impact Study. insufficient capacity to support the longer Virginia-class submarines with the Virginia Payload Module (see fig. 4), lack of redundancy for a West Coast aircraft carrier, and various other capacity and capability deficiencies that hinder the maintenance process such as small dry docks that require time- consuming workarounds or an inability to handle nuclear fuel (see sidebar). Though the Navy intends to recover most of the missed maintenance periods with these projects, according to Navy officials, the plan was developed using then-current estimates of fleet size and shipbuilding schedules derived from the fiscal year 2017 force structure projection. The Navy has since revised that projection, and the fiscal year 2020 shipbuilding projection increases both the number and accelerates the build rate of the nuclear powered ships supported by the naval shipyards (see fig. 5). Program office representatives told us that the plan, if implemented, will support the higher numbers and accelerated schedule of the Navy’s 2020 shipbuilding plan. Officials also stated that they plan to stay aware of further changes to depot maintenance requirements by attending annual fleet scheduling conferences in the future. These conferences are intended to reach a Navy-wide consensus on, among other things, changes to shipbuilding plans and the schedules for various ships to undergo their dry dock maintenance. Program officials noted that their presence at the conference allows them to update the SIOP in the event that there are additional changes to the shipbuilding schedule in the future. We have previously recommended that the Navy assess the risks to implementing shipyard infrastructure improvements; changes to the shipbuilding schedule are one such risk. Because of our previous recommendation and the Navy’s process for reviewing changes, we are not making an additional recommendation on this matter. It is too soon to determine whether the Navy’s plan will fully address the shipyards’ facility deficiencies as the Navy has not yet completed the complex effort necessary to develop detailed facility optimization plans for each shipyard. Implementing the plan will be a complex, multiyear effort to redesign the workflows at each shipyard and will involve several steps (see fig. 6). As part of the facility optimization effort, the Navy will seek to address several critical facility deficiencies it has identified at the public shipyards that negatively affect the Navy’s ability to complete maintenance on time. These include the average age of shipyard production shop facilities is 76 years, exceeding DOD’s expected average service life of 67 years for facilities; the average condition rating of shipyard production shop facilities is 66, which is considered poor, falling below the Navy standard of 80; and inefficient facility layout at the shipyards that has not been optimized to support the maintenance, repair, and disposal of nuclear-powered Navy ships and submarines. According to the Navy, the shipyards were originally designed to support the construction of ships and submarines and not the maintenance mission for the nuclear fleet that they perform today. Because the shipyards were designed for a different mission, key facilities such as maintenance shops may be located at significant distances from where the majority of work is performed. As a result, it is not uncommon for workers to walk several miles each day because of the inefficient layout of the shipyards, according to shipyard officials. For example, building 155 at Pearl Harbor Naval Shipyard, which is actively involved in submarine maintenance, is about 1/3 mile away from the nearest dry dock. This distance creates additional travel time for both personnel and material, resulting in maintenance inefficiencies. We have noted previously that waterfront locations are often ideally located to support the shipyards’ maintenance mission, but that the challenges of dilapidated structures, historical designations, and other issues can make it difficult for the shipyards to make full use of the locations (see sidebar). Building 6 at Pearl Harbor Naval Shipyard is a former foundry that has not been used since the 1980s. The building’s distinctive 3-tier roof architectural style make it a historic facility and therefore difficult to restore and modernize, according to shipyard officials. Because of its size and close proximity to the waterfront, shipyard officials would like to use the facility to support the maintenance mission, rather than let it sit empty. However, the building has extensive health and safety issues and also requires environmental remediation. The Navy’s plan estimates that the implementation of facility optimization will take at least 20 years and require increased spending for facility construction and modernization over that time. In addition, this will be a highly complex effort to redesign four large operational industrial installations, and the time frame for its completion remains uncertain at this stage. The modeling and simulation of shipyard production facilities began in February 2019 and will not be completed until 2020. According to program office representatives, Pearl Harbor’s “current state” facility model is scheduled to be completed near the end of fiscal year 2019, and the optimal facility model is scheduled to be completed in the 2nd quarter of fiscal year 2020 (see fig. 7). Modeling and simulation at the Norfolk, Portsmouth, and Puget Sound shipyards are not scheduled to be completed until the end of fiscal year 2020. However, some shipyard officials have expressed doubt about this timeline, stating that the modeling and simulations may take more effort to complete. For example, officials from Puget Sound Naval Shipyard told us they have done some degree of industrial modeling and simulation since 2007, but never at this magnitude and with this many variables. Because the modeling and simulation effort is so complicated, officials said it may be necessary to use the model to optimize the most critical parts of the industrial process first before gradually adding others. Shipyard officials also said that running the models will require a highly skilled and interdisciplinary team due to the complexity of the effort. If the simulations are completed as planned, the Navy expects to use them to complete the shipyard Area Development Plans in fiscal year 2021 and a prioritized list of facility development projects by fiscal year 2022. Navy officials said the list would likely inform facility investments for the following 5 years. Navy officials told us that they are suspending work on many facilities’ projects in order to avoid funding projects that do not serve the larger optimization goal, although some critical projects have been allowed to continue because they provide improvements needed to meet immediate operational needs, such as dry dock improvements. However, according to Navy officials, some projects have been deferred until 2022 when the prioritized list of projects to support shipyard optimization is expected to be complete. In addition, specific actions to address other infrastructure deficiencies at the shipyards are not addressed in the current plan, adding additional uncertainty. Navy officials explained that the optimized layout of shipyard facilities, which is still in early development, will drive the future efforts that address deficiencies associated with roads, utilities, sidewalks, and information-technology systems, which are not addressed in the plan. These officials explained that it will likely be several years before they can incorporate specific actions into the plan to address these deficiencies. The Navy plans to mitigate equipment deficiencies at the shipyards through increased funding to replace aged shipyard equipment. Specifically, the Navy’s plan states that funding levels for shipyard capital equipment will need to increase from historical levels to about $150 million annually and be sustained for at least 20 years to bring the average age of shipyard equipment to within industry standards. However, it is not clear whether this will fully address shipyard equipment deficiencies, because the Navy officials stated that they will not be able to create a more detailed goal until after the facility modeling and simulation effort is complete. The Navy’s plan states that most shipyard capital equipment is beyond its effective service life, obsolete, unsupported by the original manufacturers, or at risk of failure. According to the plan, the average age for industrial equipment in the private sector is 7 to 10 years, while the average age of equipment at the four shipyards is 24 years. According to the Navy’s plan, aged equipment can increase the costs of depot maintenance for submarines and aircraft carriers and place schedules at risk. Modernizing the capital equipment at naval shipyards is essential to improving their efficiency, reducing maintenance costs, and supporting fleet readiness, according to the plan. The capital equipment deficiencies identified by the Navy’s plan are consistent with our recent work, which found that the equipment at the shipyards was, on average, past its expected service life (see table 1). However, it is too early to determine whether the Navy’s plan to increase equipment funding will fully address the shipyards’ equipment deficiencies. Navy officials told us that they have not yet established a specific improvement goal for shipyard capital equipment, because developing this metric will not be possible until after the modeling and simulation phase to develop optimized facilities is complete. For example, during the modeling and simulation phase to optimize shipyard operations, the Navy will likely make decisions that will affect the amount and cost of capital equipment, such as concentrating some specialized equipment at certain yards, standardizing equipment items and purchasing them in bulk at lower cost, or purchasing more efficient items that may reduce the quantity needed. Officials stated that they developed a rough order-of-magnitude estimate of the cost to replace aging equipment. The Navy has, in the past, spent about $50 million to $60 million annually to invest in capital equipment at the shipyards. However, the Navy estimates that it will require average annual funding of $150 million over the course of 20 years at a total cost of $3 billion in order to modernize capital equipment to within private industry standards. If this effort is sustained over the 20 years identified in the plan, the capital equipment deficiencies at the shipyards will not be fully addressed until fiscal year 2040. However, this estimate is based off an earlier Navy study that identified a need for annual average funding of $150 million over a longer, 30-year period. According to this earlier equipment study, the 10 additional years of investment would total $1.5 billion. Navy officials have stated that they will attempt to address the shipyards’ equipment deficiencies over the 20-year time frame by taking advantage of different equipment purchasing strategies and gaining efficiencies from the facility optimization effort that will allow the Navy to recapitalize equipment more effectively than was possible with its previous strategies. However, the efficacy of these strategies cannot be assessed until the Navy completes its modeling and simulation phase in fiscal year 2020 and develops more detailed plans for recapitalizing its shipyard equipment. The Navy estimates the Shipyard Infrastructure Optimization Plan will cost about $21 billion to implement; however, the estimate is preliminary and therefore is not complete or reliable. To develop the plan, the Navy first identified deficiencies in three major categories—dry docks, facilities, and equipment—and then developed a cost estimate to understand the resources it would need to mitigate those deficiencies. For the dry dock and equipment portions of the estimate, the Navy was able to build on previous cost estimates that had investigated additional investments in those areas. For the facilities portion of the estimate, the Navy assumed total reconstruction of most current facilities based on current processes, using notional square-footage facility requirements in the absence of a more detailed engineering assessment. The initial estimated cost to implement this plan over 20 years includes $4 billion for improvements to the dry docks, $14 billion for facilities, and $3 billion for capital equipment. Navy officials stated that they wanted to provide Navy leadership and congressional decision makers with a rough order-of-magnitude estimate, not a budget-ready cost estimate. That is why the estimate was released in its 2018 report to Congress, instead of after the Navy completes its shipyard modeling and simulation effort, which they believe will give them a more accurate picture of the necessary investments. For example, the Navy acknowledges that several expected costs are not included in its plan, such as those for utilities, roads, and environmental remediation. Officials with the Navy agree that including these will likely add hundreds of millions of dollars to the plan’s cost. However, they decided that it was not useful to calculate these costs before the facility optimization process was complete, since the facility layout is going to have an effect on the placement of roads and utilities, for example. Navy officials stated that the initial cost estimate was prepared using applicable Navy guidance and that they plan to develop a more detailed cost estimate after the Navy has finished creating digital models of the shipyards and they start prioritizing specific projects, which they estimate will be in fiscal year 2021. We found that the Navy’s initial cost estimate minimally met two characteristics of a reliable cost estimate, partially met one, and did not meet one, as shown in table 2. The GAO Cost Estimating and Assessment Guide identifies four “characteristics” of a reliable cost estimate as well as associated cost estimating best practices as previously discussed. Specifically, we found that the initial cost estimate was not reliable because it was not developed using the following best practices: Program Baseline: The Navy’s plan includes some pieces of a program baseline, such as a schedule and goals, but does not fully establish a common definition of the program from which all cost estimates could be derived. A program baseline for cost and schedule may be established prior to contract award or funding work and allows decision makers to track and report on cost and schedule deviations above certain thresholds from initial estimates throughout the life of the project, to facilitate oversight. Navy officials stated that the plan’s first phase is meant to serve as the program baseline, containing all relevant data to address systemic issues across all four shipyards. However, only the facilities estimate was developed specifically for the plan; the dry dock and equipment estimates came from previous Navy efforts, conducted under different conditions. Without a program baseline, a cost estimate will not be based on a complete program description and will lack specific information regarding technical and program risks. Work breakdown structure: The Navy’s plan includes a broad list of high-level goals, such as timelines for completing major lines of effort, but the estimate does not include a more detailed work breakdown structure. Including a work breakdown structure is an important part of a comprehensive plan. A work breakdown structure deconstructs a program’s end product into successive levels with smaller specific elements until the work is subdivided to a level suitable for management control. This allows program office and shipyard personnel to accurately track and closely monitor the progress of efforts to meet the SIOP’s goals. In addition, including this structure would ensure consistency across the various cost estimating contributors, the shipyards, and NAVSEA, and would ensure that there were no omissions from the analysis and that costs are not double counted. Navy personnel stated that a more detailed work breakdown structure would not be possible until after the modeling and simulation of the shipyards is complete, after fiscal year 2020. Methodology and key assumptions: The Navy’s plan describes assumptions, but not the methodology used to develop the initial cost estimate. For example, the plan states that the size and configuration of existing facilities make it difficult to increase capacity without a significant investment, but does not describe how the Navy intends to address the issue of a larger fleet. Cost estimates are often built around assumptions—such as the rate of inflation or material costs— because they are attempting to predict future costs. However, the plan must include a clearly identified methodology to be considered well- documented according to GAO best practices. Unless methodology and assumptions are clearly documented, it will be impossible to reproduce the estimate, and decision makers will lack information on which costs are concrete and which are best guesses. Inflation: The estimate did not account for inflation, which is an important component of an accurate cost estimate. If an estimate does not include adjustments for inflation, cost overruns can result. Inflation costs on a $21 billion program over 20 years could reach 45 percent or more. Applying inflation is an important step in cost estimating because, in the development of an estimate, cost data must be expressed in like terms. If a mistake is made or the inflation amount is not correct, cost overruns can result. Navy officials noted that they are currently evaluating the use of a covered dry dock model at the shipyards, which could result in significant maintenance efficiencies. A covered dry dock is a dry dock with an area built over it, which allows the shipyard to develop production space that can minimize personnel and materiel movement. However, the Navy is still investigating the cost benefit of the covered dry dock, which means that there could be additional costs or complications not included in the current plan. not include a cost risk or uncertainty analysis. A comprehensive analysis of risk and uncertainty in the estimate is an important component of an accurate cost estimate. Navy officials have identified a number of risks to implementing the plan, such as the costs of complying with historical preservation requirements, environmental remediation, and the acquisition or adaptation of alternative workspace for shipyards to use while facility upgrades are performed. Officials have stated that these factors could add hundreds of millions of dollars more to the total cost of the plan. For example, an official from Norfolk Naval Shipyard said that environmental remediation of certain sites at Norfolk alone could easily cost millions of dollars to execute. Furthermore, the plan excluded certain costs that the Navy will necessarily incur in implementing it, such as those related to utilities or roads. Because cost estimates predict future program costs—sometimes for projects that have never been built before— Navy officials always associate uncertainty with them (see sidebar). Lacking risk and uncertainty analysis, management cannot determine a defensible level of contingency reserves that is necessary to cover increased costs resulting from unexpected design complexity, incomplete requirements, technology uncertainty, and other uncertainties. While the Navy did not initially include mitigation strategies for these risks in the plan, Navy officials have stated that they are involved in a number of efforts to address them. Sensitivity: Our analysis showed that the Navy’s estimate does not include a sensitivity analysis, which evaluates the effect that individual elements or assumptions can have on the estimate. Without a sensitivity analysis, cost estimators and management will not have a full understanding of the implications that changes in ground rules and assumptions can have. Officials have stated that they plan to include a sensitivity analysis in their more detailed cost estimate in 2021. Independent Cost Estimate: Our analysis showed that the Navy’s plan does not include an independent cost estimate. An independent cost estimate provides an evaluation of the quality, accuracy, and reasonableness of a program’s cost estimate by a neutral third-party, with emphasis on specific cost and technical risks. It also helps to identify risks associated with budget shortfalls or excesses. Navy officials noted that an independent cost estimate was likely not feasible at this point, considering that the effort was still in its very early stages. However, the officials stated that given the size and projected cost of the plan, they anticipate they will likely seek out an independent cost estimate in the future. Navy officials said they plan to develop a more detailed cost estimate after the Navy has finished creating digital models of the naval shipyards and identifying their optimized layouts, which they estimate will be in fiscal year 2021. However, even in the context of a preliminary estimate, the best practices associated with the four characteristics are foundational and should be the building blocks upon which any sound program is based. The importance of best practices is only magnified by the size of the program, which means ignoring best practices can have meaningful effects. For example, as we noted previously, not adjusting for inflation is likely to underestimate the cost of the program. Navy officials have expressed openness to the best practices as they prepare the more detailed cost estimate. However, without incorporating these cost estimating best practices that inform Navy decision makers and Congress of the full costs of shipyard optimization, the Navy is at risk of not identifying the resources it needs to fully implement its optimization plan. In addition, without fully accounting for all costs, management will have difficulty successfully planning program resource requirements. The Navy created a management structure—a program management office (referred to as PMS 555)—to oversee the estimated 20-year-long process of optimizing the shipyards in June 2018. Shortly thereafter, in September 2018, the Assistant Secretary of the Navy for Research, Development, and Acquisition stated that, though the shipyard optimization effort did not fit all the characteristics of a formal acquisition program, its size and importance required the Navy to treat it as one. As a result, the newly created program office was designated as the acquisition lead for all efforts related to shipyard optimization. The program office was also required to report on its progress quarterly to an executive oversight council consisting of leadership representatives from a number of Navy organizations. This program office includes representatives from Navy organizations that would necessarily be involved in shipyard construction, including Navy Installations Command and Naval Facilities Engineering Command. Navy officials explained that NAVSEA is managing the implementation of the plan through the program office; Navy Installations Command provides installation support through management of shipyard land and facilities; Naval Facilities Engineering Command provides acquisition and technical expertise for real estate, facilities, and related environmental studies; and the shipyards implement the plan’s activities (see fig. 8). In the year since its creation, the program office told us they have begun taking steps to prioritize shipyard projects and complete the modeling and simulation of the existing shipyards. The office has also developed its internal organizational structure, which includes describing its relationships to essential stakeholders such as Navy Installations Command and Naval Facilities Engineering Command. However, the program office has not yet formally clarified the extent to which it will interact with the shipyards or its expectations of support from the shipyards. For example, officials with the program office have stated that they plan to locate new staff both in Washington, D.C., and at field offices at each of the shipyards. However, neither the program office nor the shipyards yet know where the field offices will fall in the shipyards’ reporting structures—including the chain of command—or precisely what their roles will be. According to program office representatives, the Navy is in the process of developing documentation, including a memorandum of agreement, to formally codify roles and responsibilities for executing the plan among the program office and its field offices, the shipyards, and other Navy organizations to accomplish these tasks, but did not provide an estimated time frame for when these roles and responsibilities would be determined. Officials said their current plan is for field offices to serve as extensions of the program office and that they will help the shipyards to oversee the execution of the plan. Program office representatives intend for shipyard personnel to help define project requirements, collect data, provide input on the digital shipyard models, and communicate the plan among the entire shipyard workforce. However, the development of the memorandum of agreement has been extended. Standards for Internal Control in the Federal Government states that management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. Shipyard officials told us that the current lack of clarity has created concerns and confusion about what their roles should be during implementation. For example, shipyard officials were uncertain in what fiscal years certain positions would be needed for implementing the plan, and shipyard officials were not always involved with planning efforts. In addition, according to NAVSEA officials, the lack of clear roles and responsibilities has hampered several long-term planning efforts, including identifying and tracking performance metrics. According to Navy officials, the program office has since received funding that it intends to use to hire additional staff, and they intend to embed some of those staff members at the shipyards to coordinate with shipyard personnel. However, at present, shipyard personnel have stated that they are generally left to interpret and enact implementation activities, which could lead to inefficient or duplicative efforts. Given the time frames of the plan, even minor delays due to inefficiency could result in projects being postponed and critical ship maintenance being deferred. Establishing clear roles and responsibilities for the shipyards would better position the Navy to effectively implement the plan. The Navy’s four public shipyards are critical for repairing and maintaining the Navy’s nuclear fleet, and the Navy spends millions of dollars annually on facilities and equipment in order to sustain shipyard performance. Inefficient shipyards can lead to longer maintenance times, increased costs, and reduced readiness. Lack of adequate capacity can also result in critical parts of the fleet sitting idle awaiting maintenance, incurring hundreds of millions of dollars in operating and support costs without providing any operational benefit. We note that the shipyards are struggling to meet the Navy’s current needs with inadequate facilities, aging equipment, poorly configured dry docks, and an ineffective management approach for addressing these issues. The Navy is attempting to address these concerns with its Shipyard Infrastructure Optimization Plan. However, the cost estimate for implementing this plan is preliminary and therefore likely under states the costs of what will be a decades-long effort. Because the Navy will be required to request funding from Congress over 20 years in order to implement this plan, the lack of a reliable cost estimate places the effort at risk. By developing a more complete cost estimate, the Navy could reduce the risk that it might request too little funding to achieve its desired outcome. Without high- quality estimates, agencies are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls. In addition, determining the roles and responsibilities of the organizations involved in implementing the plan would enhance the Navy’s ability to successfully complete the effort by ensuring that all stakeholders clearly understand their roles and expectations. We are making the following four recommendations to the Department of Defense: The Secretary of the Navy should ensure that the shipyard optimization program office (PMS 555) include all costs—such as costs for program office activities, utilities, roads, environmental remediation, historical preservation, and alternative workspace—when developing its second, more detailed, cost estimate. (Recommendation 1) The Secretary of the Navy should ensure that the shipyard optimization program office (PMS 555) use cost estimating best practices—as outlined in the GAO Cost Estimating and Assessment Guide—in developing its second cost estimate, including a program baseline, work breakdown structure, a description of the methodology and key assumptions, inflation, fully addressing risk and uncertainty, and a sensitivity analysis. (Recommendation 2) The Secretary of the Navy should ensure that the shipyard optimization program office (PMS 555) obtain an independent cost estimate of the program prior to the start of its project prioritization effort. (Recommendation 3) The Secretary of the Navy should ensure that the shipyard optimization program office (PMS 555), in coordination with relevant stakeholders, establish clear roles and responsibilities for the shipyards involved in the Shipyard Infrastructure Optimization Plan. (Recommendation 4) We provided a draft of this report to DOD for review and comment. In written comments provided by the Navy (reproduced in appendix III), DOD concurred with our recommendations. The Navy also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of the Navy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have questions about this report, please contact me at maurerd@gao.gov or (202) 512-9627. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. The Navy released a 5-year plan for depot maintenance on submarines in December 2018, for fiscal years 2020 through 2024. The workload plan identifies efforts to address shipyard capacity issues across the nuclear- maintenance enterprise. According to the workload plan, the root cause of submarine idle time and associated loss of operational availability is largely that public shipyard capacity is not keeping pace with growing maintenance requirements. As a result, the public shipyards have historically struggled to complete maintenance on time, as shown in table 4. As we have previously reported, maintenance on ships and submarines may be delayed as a result of a number of factors, such as workforce gaps and inexperience, the poor condition of facilities and equipment, parts shortages, changes in planned maintenance work, and weather. The Navy’s workload plan discusses several efforts to improve maintenance performance at the public shipyards. Increasing hiring for the public shipyards. The Navy hired over 20,600 workers during fiscal years 2013 through 2018. After accounting for attrition, these hires increased total end strength from 29,400 to 36,700. Accelerating training for new employees. The effect of significant attrition and hiring resulted in approximately 56 percent of the shipyard production workforce having fewer than 5 years’ experience. The public shipyards implemented new approaches for accelerating training to develop skills in a relatively inexperienced workforce. Accounting for new employees’ lower proficiency and productivity. Shipyard officials have noted that employees with less than 5 years’ experience are generally not as skilled or productive as more experienced personnel. The Navy has established more realistic maintenance planning parameters to account for the lower proficiency and productivity of recently hired, less experienced workers. Improving the definition of workload requirements. Naval Sea Systems Command (NAVSEA) evaluated technical and program maintenance requirements with stakeholders in the maintenance community to identify and address barriers to on-time completion. Among the areas evaluated were time and condition-based maintenance strategies; logistic strategies; work estimating processes; shipyard overtime levels; and technology strategies. Improving material reliability and availability. The Navy is taking actions such as updating class maintenance plans; identifying and tracking frequently needed parts to determine appropriate acquisition strategies; creating an improved material forecasting tool; and moving material closer to the user. Balancing the submarine maintenance workload across the public and private shipyards. The Navy identified two submarines for which maintenance could be outsourced to Electric Boat or Huntington Ingalls over the next 5 years, in addition to the four submarines for which maintenance is currently outsourced. The workload plan contains some optimistic assumptions which may jeopardize achieving the intended benefits. According to the Navy’s workload plan, the Navy’s efforts identified above are intended to eliminate all submarine idle time and fully address the submarine maintenance backlog by fiscal year 2023. However, success of the plan depends on the public and private shipyards and the Navy realizing improvements in their performance that they have not yet demonstrated. For example: On-time completion of submarine maintenance, at both the public and private shipyards. The workload plan states that on-time completion of submarine maintenance, at both the public and private shipyards, is critical to eliminating submarine idle time and the submarine maintenance backlog. However, this assumption may not be realistic in light of recent performance at public and private shipyards. As discussed above, on average the public shipyards have completed maintenance on time only about 26 percent of the time between fiscal years 2007 and 2017. Of the three submarine maintenance periods that were allocated to the private shipyards between fiscal years 2015 and 2018, all three are projected to be completed about a year late, according to Navy reports. Officials with both Electric Boat and Huntington Ingalls have acknowledged the delays, which they attribute to an inexperienced workforce, lack of capital investment, and the submarines being in worse condition than expected. These officials also stated that if the Navy were to provide them with regular submarine repair work, they would expect their repair times to improve as their planning process matures and their workforce gains experience. Timely implementation of the Navy’s Shipyard Infrastructure Optimization Plan. Dry dock projects outlined in the Shipyard Infrastructure Optimization Plan must be completed on schedule, or else the Navy will not have the capacity to conduct some of its anticipated maintenance. This would in turn result in additional idle time and backlog. Some projects, such as the multimission dry dock project in Portsmouth, require the completion of earlier projects in order to proceed. Anything that disrupts the schedule of the earlier project could also affect the schedule of the later project. Given that the Shipyard Infrastructure Optimization Plan describes a 20-year- long effort that, at present, does not have clear organizational roles and responsibilities or a complete accounting of all the costs, it is possible that the gains it is intended to produce will take longer than expected to materialize. In addition to the individual named above, key contributors to this report are Suzanne Wren, (Assistant Director), James Lackey and Cody Raysinger (Analysts-in-Charge), A. Steven Bagley, Anna Irvine, Jennifer Leotta, Amie Lesser, Felicia Lopez, Carol Petersen, Michael Silver, and William Tedrick. Military Depots: Actions Needed to Improve Poor Conditions of Facilities and Equipment That Affect Maintenance Timeliness and Efficiency. GAO-19-242. Washington, D.C.: April 29, 2019. DOD Depot Workforce: Services Need to Assess the Effectiveness of Their Initiatives to Maintain Critical Skills. GAO-19-51. Washington, D.C.: December 14, 2018. Navy and Marine Corps: Rebuilding Ship, Submarine, and Aviation Readiness Will Require Time and Sustained Management Attention. GAO-19-225T. Washington, D.C.: December 12, 2018. Navy Readiness: Actions Needed to Address Costly Maintenance Delays Facing the Attack Submarine Fleet. GAO-19-229. Washington, D.C.: November 19, 2018. Navy Shipbuilding: Past Performance Provides Valuable Lessons for Future Investments. GAO-18-238SP. Washington, D.C.: June 6, 2018. Navy Readiness: Actions Needed to Address Persistent Maintenance, Training, and Other Challenges Affecting the Fleet. GAO-17-809T. Washington, D.C.: September 19, 2017. Naval Shipyards: Actions Needed to Improve Poor Conditions that Affect Operations. GAO-17-548. Washington, D.C.: September 12, 2017. Navy Shipbuilding: Policy Changes Needed to Improve the Post-Delivery Process and Ship Quality. GAO-17-418. Washington, D.C.: July 13, 2017. Department of Defense: Actions Needed to Address Five Key Mission Challenges. GAO-17-369. Washington, D.C.: June 13, 2017. Military Readiness: DOD’s Readiness Rebuilding Efforts May Be at Risk without a Comprehensive Plan. GAO-16-841. Washington, D.C.: September 7, 2016. Defense Inventory: Further Analysis and Enhanced Metrics Could Improve Service Supply and Depot Operations. GAO-16-450. Washington, D.C.: June 9, 2016. Military Readiness: Progress and Challenges in Implementing the Navy’s Optimized Fleet Response Plan. GAO-16-466R. Washington, D.C.: May 2, 2016. Defense Inventory: Actions Needed to Improve the Defense Logistics Agency’s Inventory Management. GAO-14-495. Washington, D.C.: June 19, 2014. DOD’s 2010 Comprehensive Inventory Management Improvement Plan Addressed Statutory Requirements, But Faces Implementation Challenges. GAO-11-240R. Washington, D.C.: January 7, 2011.", "summary": "The poor condition of infrastructure at the Navy's four public shipyards—Norfolk Naval Shipyard, Virginia; Portsmouth Naval Shipyard, Maine; Puget Sound Naval Shipyard, Washington; and Pearl Harbor Naval Shipyard, Hawaii—affects the readiness of the aircraft carrier and submarine fleets they are charged with maintaining. In response to congressional direction to create a plan to address the shipyards' infrastructure deficiencies, the Navy developed the Shipyard Infrastructure Optimization Plan , which the Navy estimates will require $21 billion and 20 years to implement. Senate Report 115-262 accompanying a bill for the National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to review the Shipyard Infrastructure Optimization Plan . GAO evaluated the extent to which the plan (1) addresses deficiencies in the infrastructure needed to support the Navy's projected needs, (2) includes reliable cost estimates to address those deficiencies, and (3) identifies clear roles and responsibilities for implementation. GAO reviewed the Navy's shipyard infrastructure plan and cost estimates; conducted site visits to shipyards selected to provide a variety of operational perspectives; and interviewed Navy and shipyard officials. The Navy's 2018 Shipyard Infrastructure Optimization Plan includes actions to address critical deficiencies at the shipyards, but the extent to which the plan fully addresses those deficiencies remains to be seen as the proposed actions are complex and years away from being implemented. The plan includes steps to address dry dock deficiencies, which the Navy expects willl provide it with the capacity and capability to perform 67 of 68 ship maintenance periods it is currently unable to support through fiscal year 2040. Once area development plans are complete (see figure), the Navy projects it will take at least $21 billion over 20 years to fully implement the plan. The Navy's initial cost estimate for the plan did not use certain best practices in developing the estimate, such as documenting key assumptions, accounting for inflation, and addressing risks that together could add billions to the ultimate cost. Navy officials stated that high-quality cost estimates will not be possible until they complete modeling and simulation in fiscal year 2020 and subsequently identify the most effective shipyard layouts and prioritize projects in fiscal year 2022. However, without fully following best practices in subsequent estimates, the Navy risks requesting inadequate resources to address shipyard deficiencies. The Navy created a program management office in June 2018 to oversee the estimated 20-year-long process of optimizing the shipyards. This program office includes representatives from multiple Navy organizations. However, the office has not formally defined the role of shipyard officials. Navy officials stated that they intend to develop an agreement to address roles and responsibilities, but this has not yet been finalized. Without defining clear shipyard roles and responsibilities, the Navy risks an ineffective implementation of its plan. GAO recommends that the Navy enhance the quality and reliability of its shipyard infrastructure plan by incorporating GAO's cost estimating best practices and determining clear shipyard roles and responsibilities for implementing the plan. The Navy concurred with these recommendations.", "document_type": "gao"}
{"report": "When children are removed from their homes, state or local child welfare agencies are typically responsible for coordinating their placement and provision of services. State or local child welfare agencies may also contract with private child welfare agencies to help them administer child welfare services. Child welfare agencies often place children in a foster home or in group care settings, depending on the child’s needs. Children placed in foster families may live with unrelated foster parents, relatives, or fictive kin (e.g., close family friends who are not relatives). Group care settings— also known as congregate care—typically include group homes and child care facilities that provide 24-hour care and/or treatment for groups of children. For example, these settings may include child care institutions, residential treatment facilities, or maternity homes, according to HHS. Child welfare agencies may provide foster caregivers with a monthly payment (referred to as a foster care maintenance payment) to help cover the costs of a child’s care, as determined by each state. Children generally remain in foster care until a permanent suitable living arrangement can be made, either by addressing the issues that led to the child’s removal and returning the child to his or her family; or through adoption, guardianship, placement with a relative, or another planned permanent living arrangement. In some cases, the child reaches adulthood before leaving foster care, commonly referred to as “aging out” of foster care. Historically, aging out typically occurred at age 18, but some state-funded programs implemented prior to the enactment of the Fostering Connections Act in 2008 allowed youth to remain in care beyond their 18th birthday. Title IV-E of the Social Security Act authorizes federal funding to state child welfare agencies to help cover the costs of operating their foster care programs. ACF administers title IV-E and oversees states’ foster care programs for compliance with IV-E requirements. Title IV-E is the largest single federal source of funding for child welfare programs, comprising about 89 percent of federal child welfare appropriations in fiscal year 2017 (approximately $7 billion of nearly $7.9 billion), according to ACF. Under title IV-E, states can access funding for their foster care programs in a few ways. Title IV-E reimbursements: Title IV-E funds may be used to reimburse states for a portion of expenses to support the daily care and supervision of eligible youth in foster care (such as for food, clothing, and shelter). The eligibility requirements generally limit reimbursements under title IV-E to youth that were removed from homes with very low incomes, among other criteria. However, some states have received waivers from certain title IV-E funding requirements to carry out HHS-approved demonstration projects. According to HHS, under these waivers states receive a capped allocation of title IV-E funds for youth in foster care, and these funds are generally reserved for youth under age 18. Chafee Program funds: The Chafee Program provides funding to states, under title IV-E, for support services that are intended to assist eligible youth in the transition to adulthood. Chafee Program funds are allocated to each state based on the state’s proportion of the nation’s foster care caseload. Through the Chafee Program, states can offer services to youth who have experienced foster care at ages 14 or older, youth who are likely to age out of foster care, and certain older youth who have aged out or left the foster care system. States generally have flexibility in determining the goals, strategies, and other features of their Chafee programs, as long as states design and conduct their programs based on the key purposes outlined in the law. Chafee Program services may include help with education, employment, financial management, housing, and emotional support through mentoring. State and county child welfare agencies may contract with private entities to provide these services. Up to 30 percent of a state’s Chafee Program allotment may also be used for room and board costs for certain eligible youth. According to HHS data, services provided under the Chafee Program may vary by state, including the extent to which states provide room and board financial assistance services. While title IV-E is the primary source of federal funding available to states for child welfare programs and services, states may also use other federal funds, such as title IV-B of the Social Security Act, Temporary Assistance for Needy Families (TANF) and Social Services Block Grant funds, as well as Medicaid. State child welfare agencies also generally combine state and local funds with federal funds to support their programs. According to a state survey funded by the Annie E. Casey Foundation and Casey Family Programs, in state fiscal year 2014, 43 percent of all child welfare expenditures were from federal sources, and 57 percent were from state and local funds. Since the Fostering Connections Act was enacted in 2008, HHS has as of February 2018 approved 26 states and six federally recognized tribes to claim title IV-E funding to extend foster care to youth ages 18 to 21. According to ACF, most of these states (19 of 26) began offering federally funded extended care between 2010 and 2012, with one state beginning to provide extended care as recently as 2018. States have several different options for implementing their extended care programs. For example, in order to be eligible for extended care, title IV- E requires a youth to meet at least one of five employment or education conditions specified in the statute. States can choose to use all or some of these conditions to determine which youth are eligible for their extended care program (referred to in this report as “eligibility criteria”). In addition, states can utilize supervised independent living settings for youth in extended foster care—a placement setting not typically available for federal reimbursement for youth under age 18. (See table 1 and fig. 1). The amendments made by the Fostering Connections Act permit states to use title IV-E funds to place eligible youth age 18 or older in supervised settings in which the youth live independently. According to a research brief sponsored by ACF, supervised independent living settings are unlike other foster care placement options (such as foster homes and group homes) because they are primarily intended for youth in extended care. States may place youth under age 18 in supervised independent living settings, but, as previously noted, they generally may not seek federal reimbursement for foster care maintenance payments for youth under 18 in such settings. According to ACF’s program instructions for implementing the Fostering Connections Act, state child welfare agencies have the discretion to develop a range of supervised independent living settings which can be reasonably interpreted as consistent with the law, including whether such settings need to be licensed and any safety protocols that may be needed. These instructions state that child welfare agencies may determine that—when paired with a supervising agency or supervising caseworker—host homes, college dormitories, shared housing, semi- supervised apartments, or other housing arrangements meet the supervised setting requirement. Youth in such settings may not have onsite caregivers and often have increased responsibilities, such as paying bills, assuming leases, and working with a landlord, according to ACF documentation. However, youth may receive foster care services, such as financial support and case management, to help them become successful adults. Additionally, ACF encouraged child welfare agencies to continue to work with youth who are in supervised independent living settings to ensure that youth form permanent connections with caring adults. This could include exploring options for guardianship, adoption, or living with other caring adults, and helping youth work towards those outcomes. In response to our survey, all 26 extended-care states reported offering a variety of supervised independent living arrangements as a placement option for older youth in care, with most of the options falling largely in two broad categories—transitional living programs and private residences—in addition to a range of other types of supervised independent living settings (see fig. 2). Overall, across 21 of the 26 extended-care states for which we analyzed placement data, about 34 percent of youth in foster care who were ages 18 to 21 were placed in supervised independent living settings in state fiscal year 2017 (see appendix I for additional information). Twenty-three of the 26 extended-care states reported that they offer transitional living programs as a type of supervised independent living arrangement. Generally, officials said these programs provide youth with the opportunity to practice daily independence and may include on-site case management and high levels of support built into regular programming. Supports may include experiential learning activities to build independent living skills, such as grocery shopping and budgeting. Local officials in California told us that transitional living programs are a stepping stone to independence, and they encourage older youth to start in one of these programs before living on their own. In our discussion groups with youth in extended care, youth told us transitional living programs helped them gradually move towards more independence and learn how to pay bills on their own. Officials we spoke with in the five selected states—California, Illinois, Maryland, New York, and Tennessee—described transitional living programs that use single-site or scattered-site models, or both, ranging from a group of youth living in a single family home to youth living in apartments dispersed across a geographical area (see table 2). Local caseworkers may continue to check in with youth in both single-site and scattered-site transitional living programs, but private agencies operating these programs are typically responsible for providing more frequent case management and supervision, according to officials in the selected states (see text box). One youth in a scattered-site program in California told us that he meets with his county case worker every couple of months, but he meets with program staff on a weekly basis and sometimes several times a week. Examples of Single-Site and Scattered-Site Transitional Living Programs in Selected States Single-site program: Officials from a single-site, all-male program in Tennessee described it as a family-like environment. Youth in the program live in cottages and house managers live in private quarters attached to the cottages. Program participants have their own bedrooms, and share bathrooms and common areas such as the kitchen. Youth over age 18 do not require 24-hour supervision, but have 24-hour access and daily interaction with the house managers. Youth typically receive an allowance of $150 per month (reduced to $100 if they are employed) to purchase personal items such as hygiene products. Scattered-site program: Officials from a private agency in California explained that their agency holds master leases on one- and two-bedroom apartments across Los Angeles and Alameda counties for youth who have demonstrated their readiness for more independence. An education and employment specialist or youth advisor meets with youth living in these apartments once a week, but youth can access program staff via phone anytime and are encouraged to attend activities at the agency’s main site. Youth receive a monthly stipend for personal expenses and their rent is paid by the program, but they are encouraged to work to manage additional living expenses. Those who stay in the program until they age out of foster care may remain in the apartment and take over the lease. Officials and representatives of private agencies we spoke with in four of the selected states said that they have transitional living programs specifically for certain populations that need specialized support. For example, Illinois officials described programs in their state that specialize in serving youth with mental illnesses, youth who are developmentally delayed, pregnant or parenting youth, youth who are dually involved with child welfare and justice systems, and youth with problematic sexual behaviors, such as sexual offenders. (See sidebar.) State and local officials we spoke with in the selected states said that youth who are ready to practice living more independently, in many instances with no on-site supervision, may live in private residences. Youth in these settings may choose where they would like to live, in comparison to youth in transitional living programs whose options may be limited to the apartments and facilities offered by the programs. Officials said that youth may live in apartments or private homes that are integrated in the community, and they are often responsible for signing their own lease or rental agreement. According to officials in the selected states, the level of supervision provided to youth living in private residences varies, but is generally less than youth in transitional living programs. Officials we spoke with in all five selected states confirmed that, at a minimum, case workers are expected to have monthly, in-person contact with youth. These visits are typically to confirm if youth are still in their living arrangement, and to determine if youth are experiencing any challenges, such as limited access to resources in their community. However, officials said that some youth may receive more frequent case management and supervision, such as youth who have behavioral or mental health conditions. For example, officials at one private agency in Illinois told us their staff have weekly check-ins with youth who require legal assistance or have mental health needs. State and local officials in two states mentioned that in addition to monthly in-person contact, case workers typically have more frequent communication with youth via phone calls, texting, or video conferencing. If youth are located far away to attend college or pursue other opportunities, caseworkers may use these alternative modes of communication to maintain contact and provide support. For example, a case worker for a private agency in Illinois told us that she video chats with her client twice a week, and communicates via text messaging and email several times a week, to provide support for this college student living in a private residence while studying abroad. Generally, states vary in the types of settings they allow for private residences, according to our survey results. Among the 26 extended-care states, the most common types of settings states reported offering include living in a shared apartment or home with a friend or roommate (25) or living in a private apartment or home (24). Most states reported that they also allow supervised independent living arrangements where a young person lives in a foster family home (20) (see text box). Living Arrangements with Current or Former Foster Parents In response to our survey, most states reported that they allow youth in supervised independent living arrangements to live with a foster parent. Officials in two states mentioned that because these placements are considered supervised independent living settings, youth are eligible to receive a stipend directly from the child welfare agency for rent and living expenses. The youth and their foster parent(s) may establish a rental agreement so the youth can practice paying rent. In three states, officials also said that they provide foster families with information and resources to teach and demonstrate independent living skills, such as cooking and cleaning a home. State and local officials said this arrangement allows youth to remain connected to a supportive adult, while also learning to live independently. In our discussion groups with youth in extended care, one youth told us that living with a foster family can be a good choice when a youth has a good bond with the foster parents, and when the foster parents allow the youth to have more independence. Youth in private residences typically receive a stipend to pay for their rent and other costs, according to our survey results. States reported that stipend amounts ranged from $421.80 to $1,715 per month. However, states reported having different expectations for youths’ responsibilities in covering their living expenses. Officials in some states said they require youth to supplement their stipend with income from their employment to cover living expenses. Officials in other states said they may encourage youth to work because the stipend may not be enough to cover most of their housing and other costs. Officials in one county said that youth may receive up to $850 per month but that amount may be adjusted based on the income they earn while attending school. In another state, officials reported that youth are expected to pay at least $50 towards their rent and, depending on the young person’s budget and financial obligations, the child welfare agency typically pays the difference directly to the landlord. One youth we spoke with in Illinois said that while working and going to school part-time, he built up his savings with help from the stipend that he received from the child welfare agency. States also reported allowing youth to live in a variety of other types of supervised independent living arrangements. According to our survey, other types of supervised independent living arrangements include college dorm rooms (22 states), Job Corps or employment training programs (21), single room occupancies (19), or mental health or substance use treatment facilities (15). Officials we spoke with said that they continue to provide case management and supervision to youth in these placements, and may also support youth with a stipend to assist with living expenses. Additionally, they said that when colleges go on break during the winter or summer, case workers work with youth to find a temporary placement if needed to ensure there is no lapse in housing. Temporary placements may include living with former foster parents, friends, or renting their own apartment. Supervised independent living settings are primarily intended for youth age 18 and older in extended care; however, some states also reported allowing these arrangements for youth under 18. In response to our survey, 19 of 26 states reported allowing youth under age 18 to live in supervised independent living arrangements. State and local officials said that in instances when youth under 18 are placed in supervised independent living arrangements, they are generally placed in college dorms or transitional living programs. In one state we visited, a transitional living program official said that the program accepts youth as young as 15 years old in a very few instances. (See table 3.) Local officials and case workers said that they place youth under age 18 in supervised independent living settings on a case-by-case basis if their case worker determines that the young person is mature and capable enough to manage living with less supervision (see text box). In response to our survey, 17 states reported that when youth under age 18 are placed in supervised independent living settings, they generally receive additional supervision or supportive services compared to youth 18 and older. Two states reported that no additional supervision or supportive services are provided for youth under 18 in independent living arrangements beyond those provided to youth ages 18 to 21. Officials we spoke with in three of the five selected states provided examples of the additional supports they offer to younger youth in these settings. Officials at one private agency in Tennessee told us that if youth under 18 in their transitional living program are unable to ride the bus to high school, program staff will transport them to school. State officials in California said that generally, youth under 18 living in transitional living programs receive more hands-on support. Officials at one private agency in California said the supports provided to youth under 18 in their transitional living program include 24 hour access to on-site program staff and weekly meetings with their case manager. Officials in Maryland told us youth under 18 living on their own receive additional case management support to evaluate how the youth is adjusting to living on their own, and to provide additional guidance on budgeting and household management. While many states allow youth under age 18 to live in supervised independent living settings in certain circumstances, over half of all extended-care states (15 of 26) reported that the typical age that youth enter this placement type is 18. Officials in three of the selected states said that even when the state allows it, these arrangements are not often used for youth under 18. Additionally, officials we spoke with reported that highly independent settings, such as their own apartment, are typically reserved for youth age 18 or older. Most states we surveyed reported considering youths’ readiness, such as life skills, education, and employment status, when placing youth in supervised independent living arrangements. Officials we spoke with in four of the five selected states similarly noted that supervised independent living is most appropriate for youth who have demonstrated their readiness. These officials also noted that they consider the availability of housing options, including for youth with complex needs, which can affect youths’ placement in supervised independent living arrangements. Life skills. In response to our survey, most states reported considering youths’ life skills when placing youth in supervised independent living. Specifically, 19 states reported requiring or recommending that youth participate in an assessment to gauge their mastery of certain life skills, such as the ability to budget finances or schedule medical appointments. For example, state officials in California said they developed an assessment form with questions such as whether youth understand the risks associated with using credit cards, can shop for food and prepare meals, and know how to do laundry. Caseworkers in one California county said that they consider youths’ ability to manage their living costs and their rental responsibilities prior to approving a supervised independent living placement. Education and employment. Most states also reported considering education and employment when placing youth in supervised independent living. Specifically, 16 states reported requiring or recommending that youth have a high school diploma or General Education Development certificate, or be enrolled in secondary or post- secondary school, in order to be placed in supervised independent living. Additionally, 17 states reported requiring or recommending that youth be employed or enrolled in a job training program. For example, in Illinois, youth who want to live in private residences must have a job for at least 45 days prior to being referred for placement, and continue working for at least 15 hours per week while in the private residence. Five states reported considering other indicators of readiness beyond life skills, education, and employment when placing youth in supervised independent living settings. For example, the District of Columbia reported requiring that youth (1) have no pending or unresolved criminal proceedings at the time they apply for supervised independent living, and (2) have a checking account with a minimum balance of $100 and actively participate in the child welfare agency’s financial literacy program. Availability of housing options, including for youth with complex needs. State and local officials in two of the five selected states said that they may place youth in private residences as a default or short-term option when no other living arrangements are available for youth age 18 or older. For example, officials in Tennessee said that transitional living programs and other housing options for youth outside of metropolitan areas are more limited. To ensure that youth in these areas do not experience homelessness, the officials said they may rely on other living arrangements, such as providing youth with stipends to live in private residences. Additionally, officials in New York and California said in certain localities, there may be constraints offering certain supervised independent living settings due to a lack of affordable housing. For example, officials in one San Francisco area county said some youth who want to live independently choose to move to a different part of the state, or even out of state, because their stipend for a private residence is insufficient to afford an apartment there. Officials in all selected states also said that there are shortages of foster parents willing to provide care for youth ages 18 to 21, which can generally affect their options for living arrangements. State and local officials from all five selected states also cited challenges finding housing options that are equipped to serve certain subpopulations of youth with complex needs, such as youth with acute mental health needs, youth that are also involved in the juvenile justice system, and pregnant and parenting youth. For example, local and private agency officials said state child welfare agencies may require parenting youth in supervised independent living settings to be in larger apartments, which may be too costly to afford. In addition, officials in four selected states said private agencies may be less likely to accept youth with acute mental health conditions or youth involved in the juvenile justice system if the agencies do not have the resources to address their needs. In addition to these considerations, officials we spoke with in all five selected states and other stakeholders said they consider the importance of giving youth in extended care greater involvement in decision making as they become adults. Local officials in California said they believe in allowing youth to take risks and experience the challenges of living independently while they have support from the child welfare system. Similarly, local officials in Tennessee said that giving youth choices in their living arrangements and considering their needs and goals is important for keeping them engaged in extended care. Most (24 of 26) extended-care states we surveyed reported using title IV- E funds to support youth in supervised independent living arrangements. Most (19 of 26) states also reported using Chafee Program funds, which primarily support independent living services, to fund supervised independent living arrangements. For example, state officials in Tennessee said that in certain situations when youth need to secure an apartment quickly, officials may use Chafee Program funds to help with the deposit. Officials in Illinois told us that they also use some Chafee Program funds for room and board, particularly for youth who are not title IV-E eligible. In addition to title IV-E and Chafee Program funds, most states reported using a combination of sources, including other federal funds such as those available under title IV-B of the Social Security Act, as well as state and local funds, to pay for supervised independent living arrangements. All states reported using state funds for supervised independent living (see fig. 3). Although most states reported using title IV-E funds to support supervised independent arrangements, in several states, few youth in extended care (regardless of their living arrangement) were eligible for title IV-E funding. In 14 of the 17 extended-care states for which we calculated eligibility rates, we found that the majority of youth ages 18 to 21 in care were not eligible for title IV-E reimbursement in state fiscal year 2017, meaning that the state was responsible for most or all of the cost of their care. We found that six of the states had title IV-E eligibility rates of 30 percent or lower. Two states, Virginia and Hawaii, had eligibility rates of over 70 percent (see fig. 4). State officials we spoke with in three of the five selected states said that ineligibility frequently stems from family income exceeding the income limit for title IV-E funding. Specifically, title IV-E is limited to youth removed from homes that would have qualified for cash assistance under the Aid to Families with Dependent Children (AFDC) program as of July 16, 1996. To receive title IV-E funding, youth must have met eligibility requirements at the time they were initially removed from their home, or at the time of their voluntary placement agreement. ACF has reported that fewer families meet the AFDC income standards over time, thereby reducing the number of all youth who are eligible for title IV-E funding, regardless of their age. In 2018, we reported on declining title IV-E eligibility rates for the entire population of youth in foster care. Officials we spoke with in Illinois and Tennessee said they reconsider a youth’s title IV-E eligibility once the youth turns 18, using the youth’s income at that point in time. Officials from these two states told us they close the original foster care case when a young person turns 18 and then reopen the case when the young person re-enters extended care. Using this approach, officials said they can base the title IV-E eligibility determination on the youth’s income at the time of re-entry into the foster care system, rather than the income of the home from which the youth was removed upon initially entering foster care. Illinois officials said this process helped increase the eligibility of their extended-care population by more than 30 percent since 2012. Officials we spoke with in the remaining three selected states said they do not use this approach. ACF officials said that states can choose to close and re-open cases for youth in extended care, which would allow new eligibility determinations based on a youth’s income. However, ACF officials said they do not monitor states’ choices in this area, or how states’ choices affect their title IV-E eligibility rate. Officials said they focus on helping states identify options and provide examples in ACF program instructions so states can determine what works best with their own policies and procedures. Four states (Arkansas, Indiana, Maryland, and Massachusetts) reported in our survey not claiming title IV-E reimbursements for any youth in extended care in state fiscal year 2017. Officials in two states, Indiana and Maryland, reported not claiming title IV-E funds for youth in extended care at least in part because they use a title IV-E waiver. However, according to ACF, states can claim title IV-E reimbursements for youth over 18 in addition to their waiver funds, so these two states may be able to claim additional title IV-E funds for their extended care population. State officials in Arkansas told us they have not yet established the proper procedures, internal controls, and monitoring mechanisms to allow them to claim title IV-E funds for youth in extended care. Officials said they do not have systems to track when youth may have lost their jobs or dropped out of school, which could affect the youth’s eligibility for extended care. To help youth develop independent living skills and successfully transition out of care, state and local officials in four of the five selected states reported offering targeted training and support for youth ages 18 to 21. For example: Officials in one Maryland county said they offer youth nearing age 21 a 3-week intensive training focused on employment and housing called “Keys to Success.” According to officials, Keys to Success offers experiential learning through cooking demonstrations, budgeting and financial literacy training, group trips to stores to look at furniture or interview-appropriate clothing, and housing fairs. (See fig. 5.) Officials from Youth Villages, a national nonprofit organization, said they offer the organization’s intensive YVLifeSet program to youth in extended care in Tennessee. The program generally lasts six to nine months and pairs youth with a specialist to meet with weekly to help them achieve their goals for independent adulthood. For example, to help youth maintain employment, officials said specialists can help youth build skills such as how to handle conflict with supervisors or coworkers, provide customer service, and understand job expectations, among others. In Illinois, the state’s “Countdown to 21” program is intended to encourage youth to plan for long-term education and vocational goals, and promote their financial stability through financial literacy training. All youth, at age 19, are referred to the financial literacy training, which covers topics such as credit and investing, and officials said most youth complete the training prior to exiting extended care. Officials in one California county said they offer youth ages 18 to 21 an annual public transportation pass through the Youth on the Move program, to ensure they can get to work or school. According to ACF, some extended care states also have a specialized case management system for youth over 18. For example, officials from one county in Maryland told us that all youth in foster care are automatically enrolled in a supervised independent living case management track when they turn 18. This case management system is intended to identify a youth’s areas of need and design a plan to prepare them for living independently, regardless of their current living arrangement. Youth in extended care can also participate in independent living services that are offered more broadly to all older youth in foster care. Officials in all five selected states reported offering services that support youth in extended care in their housing, education, financial literacy, and employment goals, as well as offering health education, mentoring, and training on daily living skills (e.g., grocery shopping and budgeting). Examples of types of services the selected states reported offering to all older youth include: Housing. Officials from Youth Villages described how their organization assists youth in Tennessee to learn to find and maintain stable housing. For example, Youth Villages specialists work with youth to search for affordable housing options, develop a housing budget, complete applications, and address background check issues. Specialists also help youth build and maintain relationships with roommates and landlords. Education. County officials in New York described how a local community college helps youth complete financial aid forms, and conducts college day simulations and resume-building workshops. Financial literacy. County officials in Maryland told us they offer classroom instruction on financial literacy which includes how to understand financial aid, good banking practices, and how to asses loans and grants. Employment. In California, county officials told us about several programs they offer to help youth meet their employment goals, such as a youth worker program in which 16- to 21-year-olds are hired to rotate through different county departments for 18 months. Health. County officials in Maryland told us they offer yoga classes and other stress reduction techniques, as well as outings to local clinics for youth to learn about family planning resources. Mentoring. County officials in California told us they host weekly social events to establish community connections between youth in care, and youth have regular meetings with supportive adults focused on building connections with family and friends. Daily living. Officials at a private agency in Illinois told us the youth in their housing programs have a set schedule to participate in different life skills activities, such as cooking, doing laundry, and other chores. Although officials in all five selected states said they offer a variety of training and supports to help youth in extended care develop independent living skills, officials in these states also said that youth may experience challenges using these skills. For example, officials said youth in private residences may have difficulty covering their living expenses, which can lead to evictions. A New York county official said some youth living on their own may struggle with the lack of structure and the amount of independence in making their own decisions and setting their schedule. For these youth, officials said the child welfare agency will increase the level of case management, and offer additional support or services. To best support the development of independent living skills in youth in extended care, officials we spoke with in all five selected states said they use assessments such as the Casey Life Skills assessment, transition planning, or regular check-ins to determine youth goals, direct youth to services to meet their needs, and to measure their progress. A county official in New York described how private child welfare agencies use the Casey Life Skills assessment every 6 months for youth in care to target services to areas in which the youth needs to build skills. Through its Ready by 21 program, Maryland has yearly independent living benchmarks for youth, starting at age 14. Officials we spoke with said that caseworkers can use these benchmarks to assess a youth’s progress towards living independently, create an individual service plan based on their progress, and direct youth to additional resources as needed (see fig. 6). In addition, title IV-E requires that caseworkers assist youth with developing a transition planning document. According to state and local officials, youth periodically meet with their caseworkers to discuss their progress on their goals, and caseworkers may provide additional guidance and support as needed during these meetings. For example, a caseworker in Tennessee described providing youth funds for driver’s education classes and licenses, prom expenses, and extracurricular activities, based on conversations with youth about their needs. Officials we spoke with in all five states also discussed providing financial incentives to encourage youth to participate in independent living services. For example, according to officials in Illinois, if youth complete the financial literacy course offered by the state child welfare agency, they receive $1,200 when they exit care. County officials in Maryland described participating in the Jim Casey Opportunity Passport program. In this program, officials said, youth who complete financial literacy training are eligible for a matched savings program of up to $3,000 to purchase an asset, such as a car. In our discussion groups with youth in extended care, youth told us that they are responsible for taking the initiative to participate in services, but financial incentives are helpful. Youth told us that as a result of these incentives, as well as other supportive savings programs, some are able to exit care with substantial savings. We provided a draft of this report to HHS for review and comment. In response, HHS provided technical comments, which we incorporated as appropriate, but did not provide general comments on the draft report. As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution until 30 days from its issue date. At that time, we will send copies of this report to interested congressional committees and to the Department of Health and Human Services. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Figure 7 below presents the total number and percentage of youth in extended care by living arrangement (referred to as placement type) during states’ fiscal year 2017. Table 4 and Figure 8 present the number and percentage of youth in extended care by state and placement type during each state’s fiscal year 2017, respectively. Placement types include foster family homes with relatives or non-relatives, group homes or institutions, supervised independent living arrangements, and other types of arrangements. Figure 7, Table 4, and Figure 8 present information for 21 extended-care states. We excluded five of the 26 extended-care states we surveyed because they reported point in time data that did not reflect their entire state fiscal year 2017, or they reported data that were not reliable for the purposes of our analysis. To develop Figure 7, Table 4, and Figure 8, we assessed the information we collected in our survey of extended-care states on youth ages 18 to 21 for whom the state child welfare agency had responsibility for placement, care, or supervision during states’ fiscal year 2017. We administered the survey to state child welfare agencies in the 26 states approved by the Department of Health and Human Services (HHS) to offer federally funded extended care as of February 2018. The survey was conducted between August and October 2018 and we obtained a 100 percent response rate. Specifically, we asked states to provide data on the number of these youth in care by placement type. In addition, we asked states to provide data on the number of youth under age 18 that were placed in supervised independent living or other independent living arrangements. To ensure the quality and reliability of the survey, we pretested the questionnaire with three extended-care states to check (1) the clarity and flow of the questions, (2) the appropriateness of the terminology used, (3) if the information could be easily obtained and whether there were concerns about the reliability of data that would be collected, and (4) if the survey was comprehensive and unbiased. We revised the questionnaire based on the pretests. We reviewed responses to assess if they were consistent and contained all relevant information, and followed up as necessary to determine that states’ responses were complete, reasonable, and sufficiently reliable for the purposes of this report; we excluded data where we had concerns about their reliability. In addition to the contact named above, Sara Schibanoff Kelly (Assistant Director), Aimee Elivert (Analyst-in-Charge), Ada Nwadugbo, and Alexandra Squitieri made key contributions to this report. Also contributing to this report were Lucas Alvarez, Sarah Cornetto, Erik Gartland, Jean McSween, Mimi Nguyen, Jessica Orr, Jerry Sandau, and Almeta Spencer. Foster Care: Additional Actions Could Help HHS Better Support States’ Use of Private Providers to Recruit and Retain Foster Families. GAO-18-376. Washington, D.C.: May 30, 2018. Foster Care: Most Tribes Do Not Anticipate Challenges with Case Goal Changes but HHS Could Further Promote Guardianship Assistance. GAO-16-625. Washington, D.C.: August 8, 2016. Higher Education: Actions Needed to Improve Financial Access to Federal Financial Assistance for Homeless and Foster Youth. GAO-16-343. Washington, D.C.: May 19, 2016. Foster Care: HHS Could Do More to Support States’ Efforts to Keep Children in Family-Based Care. GAO-16-85. Washington, D.C.: October 9, 2015. Foster Care: HHS Needs to Improve Oversight of Fostering Connections Act Implementation. GAO-14-347. Washington, D.C.: May 29, 2014. Child Welfare: States Use Flexible Funds, but Struggle to Meet Service Needs. GAO-13-170. Washington, D.C.: January 30, 2013. Foster Youth: HHS Actions Could Improve Coordination of Services and Monitoring of States Independent Living Programs. GAO-05-25. Washington, D.C.: November 18, 2004.", "summary": "Youth who leave the foster care system at age 18 are often ill-prepared to live on their own and may face challenges as they transition to adulthood, such as difficulties finding stable housing. The Fostering Connections to Success and Increasing Adoptions Act of 2008 allowed states to receive federal reimbursement through title IV-E of the Social Security Act for a portion of the cost of extending care to certain eligible youth up to age 21. The Act also allows youth ages 18 up to 21 to live in a supervised setting in which the individual is living independently. One such setting may be an apartment, with monthly check-ins with a case worker (referred to as supervised independent living arrangements). GAO was asked to review supervised independent living arrangements and services for older youth. Among other things, this report examines (1) the types of supervised independent living arrangements available; (2) factors states reported considering when placing youth in these living arrangements; and (3) how selected states prepare youth to live independently. GAO surveyed state child welfare agencies in the 26 states approved by the Department of Health and Human Services (HHS) to receive federal funding to support their extended foster care programs; interviewed state and local child welfare officials and stakeholders in five states selected for factors such as variation in child welfare administration systems; reviewed relevant federal laws, regulations, and guidance; and interviewed HHS officials. The 26 states that have approval to receive federal funding to support their extended foster care programs for youth ages 18 up to 21 reported providing a range of supervised independent living arrangements. These arrangements include transitional living programs, private residences, and other settings (see figure). Officials we spoke with in five selected states said transitional living programs typically involve private child welfare agencies that lease apartments or facilities for youth, either at a single site or scattered across a geographic area, and offer on-site case management and supports to help youth build independent living skills. For private residences, youth may choose where to live, such as a private or shared apartment. In these cases, youth are typically responsible for their own lease, and may receive minimal supervision compared to youth in transitional living programs. For other settings, states reported options such as college dorms and residential employment training programs. Nineteen states also reported allowing youth under 18 to live in supervised independent living settings in certain instances, such as when they are pregnant, parents, or attending college, although such placements are generally not eligible for federal funding. Factors that most states reported considering when placing youth in supervised independent living arrangements include the youth's life skills—for example, their ability to budget finances and schedule medical appointments—as well as their education and employment status. Officials in selected states also said they consider the availability of housing, which may be limited in certain localities due to a lack of affordable housing options or other factors, and the options available to youth with complex needs, such as those who are pregnant and parents. Officials in four selected states said they help prepare youth in extended foster care to live independently by providing targeted trainings and other supports, such as financial literacy training. In all five selected states, youth can also learn independent living skills through services offered more broadly to all older youth in foster care, officials said, including assistance with housing, education, employment, and daily living skills, such as grocery shopping and budgeting.", "document_type": "gao"}
{"report": "Campus climate surveys on sexual violence are designed to collect information on the incidence and characteristics of sexual violence on college campuses as well as related student attitudes and behaviors. The topics covered by campus climate surveys can vary, depending on the questions included on the survey instrument. For example, these surveys may include questions about incidents of sexual violence, such as the number of incidents of sexual assault, intimate partner violence, or stalking, among other topics. There are two different methods that colleges can use to administer these surveys: In a census survey, all members of a group, such as the student body of a college, are surveyed. This type of survey can be used when the group that is the focus of the survey is small, when substantial resources and time are available to obtain enough responses to the survey, or when there is reason to provide all members of the group the opportunity to participate. In a sample survey, a portion of the group is selected using statistical methods to provide accurate information about the larger group. Administering a survey to a sample of students can reduce the time and resources needed to obtain enough survey responses to produce accurate data. Sample-based surveys are appropriate when it is not practical or desirable to survey every member of a group. With both sample and census-based surveys, collecting data that accurately represents the experiences of respondents requires taking a variety of steps when designing, administering, and analyzing the survey, such as weighting or analyzing the completed responses to ensure that they represent the larger group. A 2014 White House Task Force report recommended conducting these surveys as an initial step in a college’s plan to address campus sexual assault. The report also suggested follow-up actions for colleges to consider, such as providing training for college officials and creating partnerships with community sexual assault support services. In addition, there have been efforts to compare campus climate survey results across colleges. Some states have also enacted laws requiring colleges in their state to administer campus climate surveys. These state laws may vary in the nature of the survey requirements, such as the types of colleges covered by these requirements and whether a particular survey instrument must be used. For example, Louisiana requires public colleges in the state to administer these surveys, while New York requires all colleges located within the state to do so. Additionally, Louisiana requires that schools use a standard survey instrument developed by the state, while New York allows colleges to select their own survey instrument. However, there is currently no federal requirement for colleges to conduct campus climate surveys on sexual violence. Education, Justice, and HHS currently engage in a variety of efforts to address sexual violence on college campuses, including overseeing relevant federal laws and funding prevention and response activities. Education and Justice oversee colleges’ compliance with Title IX of the Education Amendments of 1972 (Title IX), which prohibits discrimination on the basis of sex in any education program or activity that receives federal financial assistance. Title IX prohibits sex discrimination— including sexual harassment and sexual violence—that effectively denies victims equal access to recipients’ educational opportunities or benefits. Under Education’s regulations, colleges receiving federal financial assistance from Education, such as those participating in federal student aid programs, must establish procedures for resolving Title IX complaints, and take steps to ensure that members of the college community are aware of their rights under Title IX. In addition, these colleges must designate at least one employee to coordinate their efforts to comply with and carry out their responsibilities under Title IX. According to Education guidance, the Title IX coordinator is responsible for coordinating the college’s response to all complaints involving possible sex discrimination, including monitoring outcomes, identifying and addressing any patterns, and assessing effects on the campus climate. Education also oversees the Clery Act, which requires colleges that participate in student financial assistance programs under Title IV of the Higher Education Act, as amended, to collect statistics on certain crimes that occur on or near their campuses, including specified sex offenses, publish those statistics in an annual security report, and annually report them to Education. Colleges must also include a policy statement in their annual security reports describing their sexual violence prevention and awareness programs for students and employees. In addition, Justice and HHS have funded grants for campus sexual violence prevention and response efforts. HHS has also developed a technical assistance document for planning and implementing sexual violence prevention strategies on college campuses. Education and Justice also manage key efforts to collect data related to campus sexual violence (see table 1). For example, Education oversees the Campus Safety and Security Survey, which collects information from colleges that participate in student financial aid programs on reported criminal incidents, including specified sex offenses, which occur on or near campuses that the colleges own or control, as required by the Clery Act. Colleges are required to include data on specified crimes that are reported to local police or campus security authorities and that occurred (1) on campus (including the subset of crimes that occurred in on-campus student housing facilities), (2) on public property within or immediately adjacent to campus, and (3) in or on non-campus buildings or property the college owns or controls. The survey collects data on the following offenses related to sexual violence: rape, fondling, incest, statutory rape, domestic violence, dating violence, and stalking. Education publishes the data on a public website. Justice collects data on crimes, including sex crimes, through the Bureau of Justice Statistics’ National Crime Victimization Survey (NCVS). The NCVS captures data on a range of offenses related to sexual violence: completed rape, attempted rape, threatened rape, sexual assault other than rape or attempted rape, unwanted sexual contact with or without force (e.g., grabbing, fondling), verbal threat of sexual assault other than rape, and stalking. The NCVS collects data through in-person interviews and phone calls with a nationally representative sample of households on the frequency, characteristics, and consequences of criminal victimization in the United States. In particular, the NCVS collects information about crimes reported and not reported to the police. Although the NCVS includes certain group residences, such as college residence halls, in its sample of households, the resulting data may not fully represent the sexual victimization experiences of college students residing on campus because the sample is primarily comprised of households. Research has found that individuals living in group residences may be at higher risk of sexual violence. Research has also noted concerns with how the NCVS is administered. Specifically, interviews are conducted in person at respondents’ homes or over the phone. As a result, victims may be less likely to honestly answer sensitive questions, such as those related to sexual violence, as their responses might be overheard by other members of their household or the offender. Stakeholders we interviewed, including survey developers, other researchers, and federal, state, and college officials, considered campus climate surveys a useful tool for learning more about the incidence of campus sexual violence and identifying areas for improvement to address it. However, stakeholders also noted that colleges face a variety of challenges with developing and conducting surveys, such as limited access to needed survey expertise and low response rates, which can affect the reliability of campus climate survey results. Nearly all stakeholders said that campus climate surveys provide an opportunity to learn more about the incidents of sexual violence occurring on individual campuses, such as those that students may not have previously reported to campus authorities or law enforcement. According to Justice officials and one researcher, campus climate surveys, which collect data directly from victims, can help overcome limitations in law enforcement data that rely on victims reporting to authorities (see sidebar). For example, the three campus climate surveys we reviewed are designed to capture information on incidents of sexual violence that students have experienced regardless of whether the incidents were previously reported to campus authorities or law enforcement. Underreporting of Traditional Crime Statistics According to a National Research Council panel, rape and sexual assault are generally underreported to law enforcement, which can affect traditional crime statistics for these incidents. For example, according to a 2014 Department of Justice report, National Crime Victimization Survey (NCVS) data showed the rate of rape and sexual assault for female college students was 6.1 per 1,000 (the 95 percent confidence interval ranges from 5.0 to 7.2 percent) for the period 1995–2013, and an estimated 80 percent of rape and sexual assault incidents went unreported to police (the 95 percent confidence interval ranges from 75 to 85 percent). College students responding to the NCVS who indicated they did not report incidents of rape and sexual assault to police cited a variety of reasons, such as considering the assault to be a personal matter, fear of reprisal, or not considering the victimization important enough to report. Similarly, Clery Act data are based on reports made to campus security authorities and law enforcement. According to one federally funded pilot study, data from campus climate surveys at nine colleges suggested that the majority of rapes are not represented in a college’s Clery numbers. In contrast, Clery Act data collected through Education’s Campus Safety and Security Survey provides information only on incidents that are reported to campus security authorities or law enforcement and that occurred on or near campuses that the colleges own or control. This can result in campus climate surveys identifying a larger number of campus sexual violence incidents than federal Clery Act data. For instance, a pilot study of campus climate surveys at nine colleges found that undergraduate students attending these colleges experienced an estimated 2,380 incidents of rape during the 2014-2015 academic year, of which an estimated 770 occurred on campus. In contrast, Clery Act data documented 40 reported rape incidents for these colleges during the 2014 calendar year. Several stakeholders we spoke with also said that campus climate surveys can provide information on a broader range of sexual violence incidents than federal crime statistics data, such as the National Crime Victimization Survey and Clery Act data, which collect information specifically on criminal offenses. For example, one researcher we spoke with noted that campus climate surveys can collect information about sexual harassment, which is not included in federal Clery Act crime statistics. The three surveys we reviewed collect information on a range of sexual violence incidents, including sexual assault, coerced sexual contact, stalking, intimate partner violence, and sexual harassment. Behavioral Questions Campus climate surveys on sexual violence ask students about a variety of topics that are often sensitive. One challenge of conducting these surveys is that students’ understanding of what behaviors are considered “sexual assault” or “rape” may differ and the words used to describe sexual violence will determine what is measured by the survey, such as incidents of rape. Research has found, for example, that surveys asking directly whether students experienced specific types of sexual violence can produce inaccurate data. To improve the quality of information collected by campus climate surveys, researchers ask students about specific behaviors and events that describe the incident rather than referring to it using a label such as “sexual assault” or “rape.” campus climate survey instruments we reviewed included questions that asked for additional context on incidents of sexual violence reported by students, such as the victim’s relationship to the perpetrator. Researchers we interviewed also noted that campus climate surveys can include behaviorally specific questions to identify conduct that survey respondents might not categorize as sexual violence (see sidebar). Each of the three surveys we reviewed used behaviorally specific questions to describe behaviors that may constitute sexual violence for survey respondents, without using specific terms, such as rape. For example: One survey we reviewed asks, “Since the beginning of the current academic year, has an intimate partner threatened to hurt you and you thought you might really get hurt?” instead of asking whether the respondent has experienced “intimate partner violence.” Another survey we reviewed asks, “How many times have one or more people left you unwanted messages (including text or voice messages)?” instead of asking if the respondent has experienced “stalking.” Seven of the nine researchers we spoke with considered behavioral questions to be a best practice for collecting data on sexual violence, including one that noted the general public may not be aware of the definitions of rape or other types of unwanted sexual contact or behaviors. However, one researcher we spoke with expressed concern that the wording of behavioral questions can be imprecise. Each of the three campus climate surveys we reviewed included questions regarding student knowledge of the administering college’s policies and resources related to preventing and responding to sexual violence on campus. According to nearly all of the stakeholders we interviewed, these data can help colleges identify areas for improvement. In particular, about half of these stakeholders noted that campus climate survey results can help colleges address barriers to reporting. For example, officials from one college we spoke with reported increasing their efforts to educate students about where to go if they experienced sexual assault based on gaps in awareness identified through survey results. Further, information from campus climate surveys also helped one state identify how it could better assist colleges, such as by providing training on intimate partner violence, according to a state official. Several stakeholders reported that campus climate surveys may also help colleges assess their performance on reducing sexual violence. For example, two researchers said that these surveys can help colleges see where improvements were made and where additional action might be needed. Another researcher we spoke with noted that colleges are very interested in using campus climate surveys to establish baseline data and are beginning to understand the usefulness of having data on sexual violence prevalence. While campus climate surveys can provide additional information on campus sexual violence, stakeholders reported that colleges face a variety of challenges with developing and conducting surveys, as well as analyzing the results. Although some survey instruments are free, about half of the stakeholders we interviewed offered that some institutions, particularly smaller colleges, may not have the resources to effectively administer surveys on their own. Stakeholders cited costs associated with hiring contractors or relying on faculty and staff to administer and analyze the results of a survey. For example, one researcher we spoke with said that administering a survey can require having people available to respond to student questions about the survey. Officials from one college we spoke with said they relied on faculty volunteers to analyze survey results over a school break, due to a limited survey budget. About half of the stakeholders also noted that providing incentives to students can help increase survey response rates, yet incentives can also be the most expensive part of a college’s survey budget. For example, one college reported that the $10 incentives offered to students who completed the survey constituted the college’s largest survey expense. Given the potentially high costs of these surveys, officials in one state we spoke with reported that the state provided funding to help its colleges administer surveys, analyze results, prepare reports, and translate the survey results into action. About half of the stakeholders also reported that some colleges may not have technical expertise readily available to conduct a campus climate survey on sexual violence. As previously noted, colleges can administer campus climate surveys to a sample of students (sample approach) or all students at a college (census approach). According to federal guidance, a sample approach can reduce the amount of follow-up needed to encourage survey completion; however, expertise is needed to create a sampling frame that includes all, or nearly all, of a target population, and then to accurately select a sample of that population to survey that still represents the target population. Several stakeholders said colleges may face challenges in creating a representative sample of their students, in particular. For example: One researcher noted that colleges may not collect sufficient demographic data or have adequate funding to create a representative sample of their students. Another researcher observed that for both sample and census surveys, colleges may also lack the expertise to ensure, through statistical methods such as non-response bias analysis or weighting responses, that respondents are representative of the student body. Justice officials noted that properly administering campus climate surveys requires personnel with adequate statistical expertise, as well as support from college administration. Research shows that statistical methods like testing for non-response bias and weighting responses are an important consideration when developing estimates on prevalence since non- response bias can potentially limit the extent to which the results can be generalized to the entire student population. One college that had not conducted a campus climate survey also noted that doing so would be a challenge due to limited expertise with conducting surveys on sensitive topics, such as sexual violence. However, two of the selected colleges that conducted campus climate surveys reported working with a third party to ensure more reliable results. Response rates are a key factor in producing reliable survey results, and most stakeholders reported that obtaining a sufficient number of responses from students can be a challenge. Achieving a sufficient response rate can help ensure that the survey results are representative of the target population, so that the results can be used with confidence to inform decisions. However, our prior work on federal sexual violence data found surveys are subject to variable response rates over time, and different surveys may have different response rates, which may affect the resulting estimates and the validity of the data. The seven selected colleges that conducted surveys reported response rates ranging from less than 10 percent to more than 60 percent. Additionally, officials we interviewed in two of the selected states reported that their survey response rates were not high enough to generalize or draw meaningful conclusions regarding campus sexual violence, as originally intended. Officials in these states said they primarily included the data in required state reports, with limitations noted as needed. Most stakeholders noted that survey design or administration factors can affect response rates. About half of the stakeholders noted that keeping the survey short is critical to ensuring more students complete it, but some topics of interest to the colleges may not be covered as a result. For example, at one college, officials included survey questions about sexual assault and sexual harassment, but did not pursue questions about stalking due to concerns about survey length. One researcher also told us that long and complicated surveys may not work well on smart phones, which is how many students take these surveys. As for survey administration, most stakeholders noted that choices on how to administer the survey can also affect the response rates for surveys. For example, two researchers we spoke with said that it is better to leave surveys open to respondents for a longer time period to increase the response rate. In addition, a researcher and one state official noted that technical issues can interfere with obtaining a high response rate, such as sending survey invitations to university email accounts that students may not check regularly. About half of the stakeholders stated that differences in survey instruments and methodology may make it difficult for colleges to compare their results with the results of other colleges. Variation in questions and definitions. The surveys we reviewed varied in the wording used to ask respondents about their knowledge of institutional policies for reporting sexual violence. According to one researcher, differences in the wording of questions and structure of questionnaires can affect comparability across surveys. Officials in one state also reported that colleges used different definitions of key terms on their campus climate surveys, which made it challenging to reach general conclusions across colleges. Similarly, another researcher stated that differences in the definitions of terms used in colleges’ campus climate surveys make accurate comparisons difficult. Variation in time periods. The surveys we reviewed ask respondents about incidents of sexual violence occurring over different time periods, which may also limit the comparability of survey results across colleges. According to one researcher we spoke with, colleges using different survey instruments should not compare prevalence estimates with results from other surveys that ask about incidents of sexual violence for different time periods. For example, one survey instrument we reviewed asks students about their experiences with sexual violence during the current academic year. In contrast, the two other survey instruments we reviewed ask students about their experiences with sexual violence since first enrolling at college, which covers a longer time period for seniors than first-year students. Time periods may also affect the accuracy of the data collected. One researcher we spoke with, for example, stated that survey questions that cover longer time periods can introduce bias, such as the telescoping effect, whereby respondents recall certain events as being more recent than they actually are. Additionally, longer time periods may yield larger numbers of incidents, since more individuals may experience the behavior over time. To address these comparability challenges, some colleges have used the same survey instruments as other colleges. For example, two colleges included in our review participated in a survey effort among multiple colleges that was designed to allow for comparisons across participating schools. To make these comparisons, a third party administered the survey at participating colleges using a survey instrument with standardized questions and a standardized methodology to enable the measurement of prevalence, and then analyzed the results. In summary, while all stakeholders noted the value of conducting campus climate surveys, about half of them generally cautioned against requiring colleges to administer them in light of the associated challenges previously discussed. Officials at one college that voluntarily conducted a campus climate survey using a one-time grant stated they would have to use funds from faculty and staff salaries if they were required to conduct a survey in the future. Additionally, an official from one college that had not conducted a campus climate survey noted that high turnover in the Title IX coordinator position would make it difficult for the college to sustain a survey effort over time. Further, another college that has not conducted a campus climate survey to examine the incidence of sexual violence noted it would be difficult to design a standard survey instrument that would apply across all colleges, such as those that primarily serve students who take courses online. The seven selected colleges that conducted campus climate surveys used various survey design, administration, and outreach strategies to learn about the incidence of campus sexual violence. Most of these colleges also chose to publicly report some survey results. Choosing a survey instrument. These seven colleges considered several factors when choosing a survey instrument: Rigor. Officials from each of the seven colleges that conducted a campus climate survey said it was important to use a rigorous survey instrument, such as one that survey developers have validated or colleges have widely adopted. One college official explained that using a validated instrument provided assurances that helped secure a timely approval from the college’s institutional review board. Flexibility. Officials from five colleges said they valued the flexibility of using a survey instrument that could be modified based on the specific characteristics and needs of their colleges. For example, officials from one college said that the chosen instrument enabled administrators to use gender-inclusive language and ask questions about incidents of sexual violence from the perspective of the perpetrator in addition to the victim. Comparability. Officials from four colleges noted that comparability was a consideration when selecting a survey instrument, including the potential to compare survey results across colleges that share similar characteristics or at their own colleges over time. However, as previously discussed, stakeholders noted that differences across survey instruments can limit the comparability of survey results. Cost. Officials from four colleges said the cost of conducting campus climate surveys informed their selection of a survey instrument. For example, officials at one college said they used a free, publicly available survey instrument because the college lacked the resources to pay for an instrument. Length. Officials from four colleges identified survey length as another factor they considered. Officials from three of these colleges specifically noted that longer surveys may result in lower response rates. In addition, officials from one of these colleges stated that because longer surveys collect more data, the college would need more time to analyze the results. An official from another college expressed concern that longer surveys with multiple follow-up questions about incidents of sexual violence risk re-traumatizing victims. Modifying the survey instrument. Six of the seven colleges modified their survey instruments to some extent. Officials at five of the six colleges reported adding questions to their survey instruments. For example, two colleges reported adding questions to comply with a state survey requirement, while another college reported adding follow-up questions to collect information on events prior to an incident of sexual violence. Officials at three of the five colleges reported limiting the number of questions they added to keep the survey short. Officials at two of these colleges noted that lengthening the survey could result in fewer students completing it. Officials at one of these colleges cited additional fees that the vendor charged for such modifications as another factor in their decision to limit the number of questions they added. Officials from two colleges also said they modified the language in the survey instruments to reflect the names of specific offices and programs on their campuses. In contrast, officials at one college reported making no changes to their survey instrument because they planned to use the original survey as a baseline against which to compare future survey results. Identifying the survey population. Six of the seven selected colleges that conducted a campus climate survey distributed their surveys via email to all students in the target population (i.e., a census approach), and one worked with a third party to select a representative sample of students to receive the survey. As previously discussed, surveying a sample of students can reduce the amount of follow-up work needed to obtain sufficient responses to provide information about the student body as a whole. However, officials at four of the seven colleges cited other considerations for choosing a census approach. Specifically, officials from three of these colleges said that a census approach provided every student the opportunity to share their experiences and perspectives through the climate survey. Officials from two of these colleges further explained that administering the survey to a sample of students could give the appearance they were excluding students, some of whom might be victims of sexual violence, from participating in the survey. Another college reported using a census approach because it lacked the resources needed to develop a representative survey sample. Determining survey timing and frequency. All seven of the selected colleges that conducted a climate survey administered at least one survey during the spring semester. Officials from three of these colleges said that administering climate surveys in the spring ensures that first-year students have spent time on campus prior to taking the survey. However, officials at four colleges said that competing demands for students’ time, such as other surveys and final exams, are a tradeoff to administering these surveys in the spring. As a result, students may experience “survey fatigue”—that is, they may be less likely to respond to or complete the survey. The seven selected colleges administered surveys with varying frequencies. For example, one college reported administering its survey biennially in accordance with a state requirement, while two others administered their surveys less frequently (e.g., every 4 years) to avoid survey fatigue and low response rates. Protecting confidentiality. Six of the colleges reported taking steps to preserve the confidentiality of survey respondents. For example, officials from five colleges explained that in order to maintain respondents’ confidentiality they had to redirect students who completed the survey to a separate webpage to claim their incentive or enter a drawing. Officials from four colleges reported using a third-party vendor to help protect students’ confidentiality or, at a minimum, signal that the college had no direct role in collecting or storing student responses. For example, to protect students’ confidentiality, officials from three of these colleges said their vendors provided summary data, rather than student-level data, and did not report results with a low number of respondents. Officials from three of the six colleges reported consulting their institutional review boards to help ensure that the colleges protected respondents’ confidentiality. Officials from another college reported limiting how often they administered campus climate surveys to head off potential student concerns that they were being “tracked” during their time on campus. Offering survey incentives. As part of their outreach efforts, six of the seven colleges offered incentives to students who completed surveys, which some research suggests can increase web-based survey participation rates (see fig. 1). For example, one college offered a $20 gift card to survey respondents, which college administrators considered critical to achieving a higher response rate. This comports with a study funded by Justice that found incentives between $20 and $30 appear to help maximize survey participation, whereas a $40 incentive does not clearly offer any additional advantage. To manage the cost of incentives, two colleges offered a limited number of incentives to students via lottery drawings. Officials from one of these colleges said it funded its lottery for five $200 gift cards with proceeds from an on-campus student event. Another college offered a coupon for a free drink at a campus coffee shop to the first 300 survey respondents. An official from the college that did not offer incentives in its most recent climate survey said that incentives would help improve response rates for future surveys. While incentives can help increase survey participation, two colleges noted that offering incentives may require additional precautions to prevent abuse. For example, one college had to put its survey on hold to fix a technical error that enabled a student to collect additional incentives by completing the survey multiple times. Another college with experience offering survey incentives reported that it received calls and emails from other colleges requesting assistance with preventing such abuses. Marketing the survey. Each of the seven selected colleges that conducted a climate survey used email to invite students to respond to the survey and various marketing efforts to encourage survey participation (see fig. 1). Officials from five of the colleges reported following up with email reminders. For example, one college reported adding the incentive dollar amount to the subject lines and another reported varying the gender of those who sent follow-up emails and the timing of them to increase student responses. In addition, five colleges reported using social media to advertise their climate surveys. Recognizing the importance of gaining institutional buy-in, officials at all seven colleges said they engaged college administrators or faculty in their marketing efforts. For example, officials at one college said that deans of its various schools were asked to send emails encouraging students to take the survey. The officials credited this particular email strategy for doubling the college’s survey response rate. Officials from five colleges also reported involving student leaders and influencers in their marketing efforts, such as creating t-shirts for students to wear that included information about the survey; having students publish an op-ed in the campus newspaper promoting the survey; and asking student leaders to share information about the survey with student organizations. Six of the seven selected colleges publicly reported at least some of the results of their surveys. Five colleges, for example, published survey results on their respective websites, and another created a campus poster with an infographic illustrating key survey results. Two of these colleges also presented the results during meetings with different student populations, such as fraternities and sororities and lesbian, gay, bisexual, transgender, and queer/questioning students. Officials from four colleges expressed that they felt a responsibility to be transparent. However, according to an official at one college, a potential drawback to making survey results publicly available is that the results could create or reinforce negative perceptions of a college’s climate regarding campus sexual violence. Finally, officials from the one college that had not publicly disclosed any survey results explained that, due to a lack of resources and in-house expertise, they did not feel sufficiently confident in their analysis of the survey results to publish them. Since the issuance of the White House Task Force to Protect Students from Sexual Assault report in 2014, federal agencies have created and disseminated informational resources for colleges interested in conducting campus climate surveys. For example, Justice’s Bureau of Justice Statistics and Office on Violence Against Women funded the development of a publicly available survey instrument and a validation study from 2014 to 2016, to provide colleges and researchers with access to a free and reliable survey instrument to collect school-level data on campus climate and sexual victimization. In 2017, Justice also collaborated with HHS’s Centers for Disease Control and Prevention to provide funding and project planning assistance for a pilot study to develop and test a campus climate survey for use at two Historically Black Colleges and Universities. According to Justice and HHS officials, this survey instrument was based on the validated Justice survey instrument, with some modifications made to the campus climate questions. In October 2019, Justice officials told us the agency had decided not to proceed with funding for the study due to concerns that modifications to the original validated survey instrument would result in data that are not comparable to data from the validation study. In addition, Justice has developed technical assistance materials for colleges interested in conducting a campus climate survey. For example, from 2016 to 2017 Justice’s Office on Violence Against Women issued documents outlining lessons learned from the Justice survey validation study, talking points to help college administrators and students communicate about climate surveys, and a frequently asked questions sheet on campus climate surveys. These documents covered a range of topics, including the goals of a campus climate survey, best practices for developing survey content, and tips for choosing survey participants and protecting their confidentiality, among others. Justice’s campus climate survey, validation study, and technical assistance documents are publicly available on Justice’s website. Justice’s campus climate survey and validation study are also available through the Center for Changing Our Campus Culture, an online clearinghouse developed and maintained by a nonprofit organization with funding from Justice’s Office on Violence Against Women. The clearinghouse provides resources for colleges on addressing sexual assault, domestic violence, dating violence, and stalking. For example, the clearinghouse includes documents outlining (1) selected research initiatives and resources on campus climate surveys, (2) suggested campus sexual assault policies and procedures, and (3) steps college institutional review boards and administrators can take to oversee research on sexual violence while maintaining participant confidentiality. Most stakeholders we interviewed were aware of federal information and resources available to assist colleges in conducting campus climate surveys. For example, officials at two of the colleges reported using Justice’s survey instrument for their campus climate surveys, with officials from one college noting they selected the instrument because it had been validated as a reliable instrument. An official from another college reported using Justice’s validation study during the survey instrument selection process, to better understand the strengths and weakness of survey instruments and potential sources of bias in the data collected. In addition to the resources provided by Justice, Education has offered information to colleges regarding the prevention of campus sexual violence. For example, Education’s 2015 Title IX Resource Guide encouraged Title IX coordinators to help colleges develop a method, appropriate to their college, for surveying students about the campus climate. Additionally, to address Title IX concerns or complaints, Education may enter into voluntary resolution agreements with colleges. These agreements describe the changes colleges agree to make to ensure their procedures for preventing and responding to sex discrimination comply with the law. According to agency officials, Education may include campus climate surveys as part of these voluntary agreements, on a case-by-case basis. Justice and HHS have also funded campus sexual assault prevention and response grants. For example, Justice’s Office on Violence Against Women provides grant funding to colleges to help improve responses to sexual assault and other types of domestic and sexual violence through its Grants to Reduce Sexual Assault, Domestic Violence, Dating Violence, and Stalking on Campus Program. According to a Justice official, colleges receiving these grants are allowed, with prior approval, to use a small percentage of the grant funds to conduct campus climate surveys for program improvement purposes, but it is not a requirement of the program. Additionally, HHS’s Office on Women’s Health provided funding for the College Sexual Assault Policy and Prevention Initiative from 2016 to 2019 to organizations that partnered with colleges to provide technical assistance and support in developing sexual assault policies and prevention strategies. According to HHS officials, grantees were encouraged to conduct campus climate surveys to establish baseline data for their partner campuses. HHS officials also reported providing grantees with information on different campus climate survey instrument options, including a free, publicly available survey instrument. One college we spoke with reported partnering with one of these HHS grantees to conduct baseline and follow-up campus climate surveys and to develop comprehensive campus prevention strategies. For example, officials from the college and the grantee told us they used funds from the grant to help the college establish memoranda of understanding with community-based organizations, such as the local women’s crisis center, to support students living off-campus who may have experienced sexual violence. We provided a draft of this report to the Departments of Education and Justice for review and comment. The Departments of Education and Justice provided technical comments, which we incorporated as appropriate. We also provided relevant report sections to the Department of Health and Human Services, and to third parties, including survey developers and states included in our review, for technical comments. The Department of Health and Human Services, survey developers, and state officials provided technical comments, which we incorporated as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Education, the Attorney General, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or larink@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. This report examines (1) what key stakeholders view as the strengths and limitations of using campus climate surveys to examine the incidence and characteristics of sexual violence on college campuses, (2) what approaches selected colleges have used to survey their students about the incidence of sexual violence on campus, and (3) what role federal agencies play in helping colleges develop and implement climate surveys. To inform our examination of stakeholders’ views on the strengths and limitations of campus climate surveys, we reviewed three commonly used survey instruments that included questions regarding the incidence of sexual violence, including sexual assaults, coerced sexual contact, stalking, and intimate partner violence. Each of these survey instruments is also available, online or by request, for any college to use for free. For each of the surveys, we reviewed survey questions and methodological reports, and conducted interviews with representatives from the organizations involved in developing them. The Association of American Universities (AAU), an association of 65 research universities, developed its survey instrument in conjunction with the research firm Westat. AAU administered its survey to participating colleges in spring 2015 and spring 2019. The Administrator Research Campus Climate Collaborative (ARC3), an organization of sexual assault researchers, university administrators, and student and legal affairs professionals, developed and tested its campus climate survey from 2014 to 2015. The final survey instrument was made available to colleges in 2015. According to the survey developers, there is no comprehensive list of schools that have conducted the ARC3 survey. The Department of Justice (Justice) survey instrument was initially developed by the White House Task Force to Protect Students from Sexual Assault in 2014, and later refined and tested by Justice in collaboration with RTI International, a research organization. The survey instrument, also known as the Campus Climate Survey Validation Study, is publicly available online. According to Justice officials, there is no comprehensive list of schools that have conducted the Justice survey. Additionally, we reviewed two key federal data sources on campus sexual violence: Clery Act data from Education’s Campus Safety and Security Survey and the National Crime Victimization Survey (NCVS) from Justice. We identified these data sources based on a review of prior GAO work and interviews with Education and Justice officials. We examined documentation for these data sources and interviewed the responsible agency officials to determine the type of data they collect on campus sexual violence, the methods for collecting this information, and their limitations. We determined these data sources were sufficiently reliable for our purposes. To inform all three objectives, we also interviewed a total of 25 stakeholders with relevant expertise, including representatives of four organizations involved in developing the three surveys we reviewed and five additional researchers who have studied campus sexual violence; officials from 10 colleges; officials from four states; and federal officials from Education and Justice. We refer to the representatives of these organizations and entities collectively as “stakeholders” in our report. When discussing stakeholder views, we group them into the following categories: “several” (between four and nine), “about half” (between 10 and 14), “most” (between 15 and 19), and “nearly all” (20 or greater). In instances where we report on the views of specific groups, such as colleges, researchers, or state officials, we refer to the individual group and enumerate the number of group members. During these interviews, we gathered information on issues related to designing and conducting campus climate surveys and analyzing and communicating survey results. We also discussed federal information and resources available to help colleges develop and implement campus climate surveys. Findings from our interviews summarize selected stakeholders’ views regarding campus climate surveys on sexual violence. These findings do not represent the views of all researchers on these topics and do not represent the experiences of all colleges developing or implementing these surveys. To identify researchers with a variety of perspectives, we reviewed research on sexual violence and conducted targeted web searches. We then selected individuals or organizations with experience conducting research on campus sexual violence or developing and administering a campus climate survey on sexual violence. We also spoke with representatives of the organizations responsible for developing the three climate survey instruments we reviewed. We used multiple approaches to identify the 10 selected colleges included in our review since there is no central repository of information on whether colleges have conducted a campus climate survey on sexual violence. Colleges that conducted a campus climate survey. Based on targeted web searches, we identified colleges that had conducted a campus climate survey and then grouped them according to which of the three survey instruments they used. We analyzed data from the Department of Education’s Integrated Postsecondary Education Data System to identify the characteristics of these colleges, including sector (i.e., public, private not-for-profit, and private for-profit), program length (i.e., 2-year and 4-year), size, and geographic location. Colleges that had not conducted a campus climate survey. We also used data from the Integrated Postsecondary Education Data System to help identify colleges that had not conducted campus climate surveys on sexual violence. Specifically, we grouped colleges into categories by sector and program length and randomized the lists within each category. To select specific colleges, we started with the college in each category at the top of the randomized list and conducted targeted web searches in an effort to ensure the college had not publicly reported conducting a campus climate survey. We conducted outreach to the Title IX coordinators at each of the selected colleges via email or telephone to confirm whether or not the college had conducted a campus climate survey on sexual violence. In total, we selected 10 colleges, including seven that have conducted campus climate surveys that examine the incidence of sexual violence on their campuses and three that have not. We selected these colleges to ensure variation in size, sector (i.e., public, private not-for-profit, and private for-profit), program length (i.e., 2-year and 4-year), geographic location, survey instrument used, and whether the college was located in a state that as of January 1, 2017 had a statutory requirement in effect for at least some colleges in their state to conduct a campus climate survey (see table 2 for selected colleges by program length, sector, and use of campus climate survey). We interviewed Title IX coordinators and other knowledgeable officials regarding the selected colleges’ experiences with conducting campus climate surveys and their perspectives on the strengths and limitations of these surveys. As part of our efforts to obtain a variety of perspectives, we also conducted semi-structured interviews with officials from four states regarding the use of campus climate surveys in their states. We selected three states (Louisiana, New York, and Washington) that as of January 1, 2017, had a statutory requirement in effect for at least some colleges in their state to conduct a campus climate survey, and one state (Ohio) that recommended colleges conduct such surveys. To identify states that required or recommended that colleges conduct campus climate surveys, we used several approaches to develop a preliminary list, including consulting with researchers, reviewing annual reports from 2014 to 2019 prepared by the National Conference of State Legislatures on state higher education legislation, and conducting targeted web searches. Based on these reviews, we judgmentally selected four states to ensure a diversity of state experiences with requiring or recommending campus climate surveys. We also confirmed applicable state requirements with state officials. The selected states differed in the nature of the survey requirement or recommendation, such as the types of colleges covered (e.g., public or public and private) and how frequently the survey was required or recommended to be administered. To supplement information gathered from our interviews, we also identified and reviewed studies and reports that examined the design and use of campus climate surveys. We conducted a targeted search of various databases to identify studies on leading survey practices. We selected studies for additional review based on their relevance to our objectives, and, using a standard review instrument, assessed the quality and rigor of each study’s findings and methods. Our report includes information about leading survey design and implementation practices from those studies we found were appropriate through this review process. To examine the role that federal agencies play in helping colleges develop and implement climate surveys, we reviewed Justice and Education resources available to help colleges conduct campus climate surveys. Additionally, we reviewed information on campus sexual violence prevention grants provided by the Department of Health and Human Services. We also reviewed relevant federal laws and regulations, as well as federal guidance and other documentation pertaining to campus sexual violence and campus climate surveys. We conducted this performance audit from July 2018 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Debra Prescott (Assistant Director), Maria Gadel (Analyst-in-Charge), Jonathan Adams, Will Colvin, Caitlin Cusati, Kirsten Lauber, and Erica Vilay made key contributions to this report. Additional assistance was provided by MacKenzie Cooper, Sarah Cornetto, Holly Dye, Monika Gomez, Dana Hopings, Connor Kincaid, Sheila R. McCoy, Mimi Nguyen, and Almeta Spencer.", "summary": "Sexual violence–which can include crimes such as rape and other forms of sexual coercion–is widely acknowledged as a problem on college campuses. Although Education collects some data on sexual violence at colleges that receive federal funding, measuring the prevalence of campus sexual violence has proven difficult, due in part to underreporting of these incidents to law enforcement. While some researchers have used surveys to gather additional information regarding sexual violence on college campuses, estimates from these surveys can vary widely due to factors such as differing methodologies and response rates. This report examines (1) key stakeholders' views on the strengths and limitations of campus climate surveys on sexual violence, (2) approaches selected colleges have taken to survey their students, and (3) the role federal agencies play in helping colleges develop and implement these surveys. GAO reviewed documentation for three widely administered survey instruments, and relevant federal laws, regulations, and guidance. GAO interviewed 25 stakeholders, including researchers; Education and Justice officials; officials in four states that required or recommended campus climate surveys as of January 1, 2017, a date selected to allow time for implementation; and 10 colleges—including seven that conducted campus climate surveys—selected based on program length (2- or 4-year), geographic diversity, and other factors. Campus climate surveys that examine sexual violence occurring on individual college campuses have several strengths and limitations, according to stakeholders GAO interviewed. Strengths. Nearly all stakeholders said colleges can use these surveys to gather more comprehensive information about incidents of campus sexual violence, such as those not previously reported to the colleges or law enforcement. Surveys can also provide information on students' knowledge of the colleges' procedures for reporting incidents, among other topics, which can help colleges identify areas for improvement. Limitations. Most stakeholders said getting students to respond can be challenging. In addition, about half of stakeholders said some colleges may not have the resources to effectively administer these surveys, and results across colleges that use different surveys may not be comparable. The seven selected colleges that conducted surveys reported using various approaches to survey their students about the incidence of campus sexual violence. Each college used one of three widely used surveys, but six modified them to some extent. Six colleges sent the survey to all undergraduates, and one surveyed a representative sample of students. Colleges also reported using multiple outreach strategies to increase participation, including offering incentives, such as gift cards, to students who completed the survey; using social media; and involving student leaders (see figure). Colleges' reported response rates ranged from less than 10 percent to more than 60 percent. The Departments of Justice (Justice), Education (Education), and Health and Human Services (HHS) have created and disseminated informational resources for colleges interested in conducting campus climate surveys. For example, from 2014 to 2017, Justice made funding available for the development of a campus climate survey instrument for public use, and developed technical assistance materials covering various topics, including how to choose survey respondents and protect their confidentiality. In addition, in 2015, Education issued guidance encouraging colleges to develop ways to survey students about the campus climate. Justice and HHS have also funded grant programs that allowed grantees to use some funding to conduct campus climate surveys.", "document_type": "gao"}
{"report": "NMFS and the eight regional fishery management councils are responsible for managing approximately 460 fish stocks in federal waters, as shown in figure 1. NMFS has overall responsibility for collecting data on fish stocks and ocean conditions and for generating scientific information for the conservation, management, and use of marine resources. NMFS carries out this responsibility primarily through its five regional offices and six regional fisheries science centers, which are responsible for collecting and analyzing data to conduct stock assessments. Stock assessments consider information about the past and current status of a managed fish stock, including information on fish biology, abundance, and distribution that can be used to inform management decisions. To the extent possible, stock assessments also predict future trends of stock abundance. NMFS provides the results of its stock assessments and other analyses, as appropriate, to the councils for use in implementing their respective fisheries management responsibilities. In the South Atlantic and Gulf of Mexico regions, NMFS provides support to the councils’ management efforts through its Southeast Regional Office and the Southeast Fisheries Science Center. Under the Magnuson-Stevens Act, the councils are responsible for managing the fisheries in their region. This includes developing fishery management plans, subject to NMFS approval, based on the best scientific information available and through collaboration with a range of stakeholders. The councils convene committees and advisory panels to assist them in developing research priorities and selecting fishery management options, in addition to conducting public meetings. The councils are to comprise members from federal and state agencies, as well as the commercial and recreational fishing sectors (see fig. 2). The councils—supported by council staff such as biologists, economists, and social scientists—are responsible for preparing proposed fishery management plans or plan amendments for NMFS review. These plans or amendments are to identify, among other things, conservation and management measures to be used to manage a fishery, including determining the maximum size of a fish stock’s allowable harvest. This is generally done by developing annual catch limits for each fish stock, that is, the amount of fish that can be harvested in the year. Fishery management plans or amendments also include establishing or revising any allocations between the commercial and recreational sectors for mixed-use fish stocks where the councils determine it may be warranted. For example, councils may allocate a percentage of a fish stock’s annual catch limit between the recreational and commercial fishing sectors. See figure 3 for an overview of the federal fisheries management process. Council staff facilitate the fisheries management process by organizing council meetings, preparing and providing analyses for those meetings, and facilitating input from stakeholders and the public on fisheries management issues, among other things. Stakeholders include participants in the commercial and recreational fishing sectors and related industries, such as fishing associations, seafood dealers and processors, food and travel industry representatives, and conservation groups. Once the councils complete proposed fishery management plans or plan amendments, they are to provide them to NMFS for review. NMFS is responsible for determining if the plans or amendments are consistent with the Magnuson-Stevens Act and other applicable laws, and for issuing and enforcing final regulations to implement approved plans. Tables 1 and 2 highlight the mixed-use fish stocks the South Atlantic and Gulf of Mexico councils manage, respectively. Under the Magnuson-Stevens Act’s national standards for fishery management plans, allocations are to be fair and equitable to all U.S. fishermen; reasonably calculated to promote conservation; and carried out in such manner that no particular individual, corporation, or other entity acquires an excessive share. NMFS guidelines for the national standards further indicate that in making allocations, councils should consider certain factors relevant to the fishery management plan’s objectives. These factors include economic and social consequences of the allocations, food production, consumer interest, dependence on the fishery by present participants and coastal communities, efficiency of various types of gear used in the fishery, transferability of effort to and impact on other fisheries, opportunity for new participants to enter the fishery, and enhancement of opportunities for recreational fishing. In reviewing and approving fishery management plans and amendments, NMFS is responsible for ensuring that the councils’ allocation decisions comply with the Magnuson-Stevens Act’s national standards. In this report, the terms “established” and “revised” allocations refer to allocations established or revised by the councils and subsequently approved by NMFS, unless otherwise stated. Historically, mixed-use fisheries allocations have been based predominantly on data estimating each fishing sector’s past use of the resource, according to NOAA. To collect commercial and recreational data, NMFS works with partners such as coastal states and interstate marine fisheries commissions. In particular, for the commercial fishing sector, NMFS collects data on landings, which include the weight and value of fish stocks sold to seafood dealers using a network of cooperative agreements with states. For recreational fishing, NMFS uses data from its Marine Recreational Information Program, which the agency began implementing in 2008 in place of the Marine Recreational Fisheries Statistics Survey. The Marine Recreational Information Program collects data on private anglers’ fishing effort and catch rates and uses these to estimate total recreational fishing catch. NMFS officials said that the program also collects information to estimate recreational landings. The program collects these data through such methods as mail surveys and shore-side interviews of anglers at public access fishing sites. Recognizing the difficulty in making allocation decisions—in part because allocations may be perceived as unfair by some stakeholders—NMFS commissioned a nationwide study in 2012 to examine allocation issues and gain stakeholders’ perspectives from commercial and recreational fishing sectors. The results of the study showed widespread dissatisfaction with how past allocation decisions were made. The study found little consensus on how to address concerns with allocations. For example, some stakeholders said that some allocations were outdated and that changes over time in human population, seafood demand, and recreational fishing warranted a comprehensive examination of allocations. Other stakeholders expressed concern that a uniform approach to allocation policy could harm fishing sectors, while others noted that it is important for the councils to have the flexibility to make regionally-focused decisions. The study concluded that many stakeholders may continue to view allocations as unbalanced or unfair unless the outcomes align with the positions they seek. The study recommended that NMFS take a number of steps to address allocation issues, including increasing stakeholder engagement in allocation decisions, periodically reviewing allocations, and creating a list of factors to guide allocation decisions. In response to the 2012 study, NMFS issued a fisheries allocation review policy in 2016 and two guidance documents to the councils, intended to help the councils and NMFS review and update allocations. The objective of the NMFS policy was to describe the fisheries allocation review process, which called for using an adaptive management approach. NMFS policy defined fisheries allocation review as the evaluation that leads to the decision of whether or not the development and evaluation of allocation options is warranted, but the allocation review is not, in and of itself, an implicit trigger to consider alternative allocations. Through its policy, NMFS established a multi-step process for reviewing and potentially revising fisheries allocations. Specifically, once an allocation review trigger has been met (as described below), the councils are to complete an allocation review. For this review, NMFS policy does not call for in-depth analyses but calls for a clear articulation of how objectives are or are not being met and a clear rationale and documentation on relevant factors considered. Based on the allocation review, the councils may decide to maintain existing allocations, or proceed to evaluate allocation options for a fishery management plan amendment. When proceeding with this next step, the councils are to undertake formal analyses and follow the fishery management plan amendment process to ultimately recommend that an existing allocation either be retained or revised. To supplement its fisheries allocation review policy, NMFS also issued two guidance documents, as follows: Criteria for initiating fisheries allocation reviews. NMFS guidance recommended that the councils establish criteria for initiating allocation reviews—or allocation review triggers—within 3 years, or as soon as practicable, for all fisheries that have allocations between sectors. The guidance identified three types of potential criteria for allocation review triggers: (1) time-based, which include provisions for periodic allocation reviews at specific time intervals on a regular basis; (2) public interest-based, which provide an opportunity for the public to express interest in allocation reviews; and (3) indicator-based, such as triggers based upon economic or other metrics. Factors to consider when reviewing and making allocation decisions. NMFS guidance outlined four categories of factors for the councils to consider when making allocation decisions, and noted that there may also be other appropriate factors to consider. These factors are not intended to prescribe particular outcomes with respect to allocations, but rather are intended to provide a framework for analysis, according to the guidance. The four categories of factors include: Fishery performance and change factors, to assess the current conditions of a fishery and any changes in those conditions that may indicate a need for updated allocations. Such factors could include historical or current trends in catch or landings, the status of the fish stock (for example, whether it is subject to overfishing, is overfished, or is rebuilding), or changes in the distribution of species within the fishery. Economic factors, to consider the monetary consequences of an allocation, such as by analyzing (1) whether the existing or recommended allocation is the most economically efficient, and (2) the economic impacts of the allocation. Social factors, to assess the consequences of an allocation on individuals and communities, such as whether an allocation may have disproportionate adverse effects on low income or minority groups or could lead to fishing despite unsafe conditions if access to the fishery is restricted to a limited number of days. Ecological factors, to consider the potential ecological impacts of allocations, such as impacts on the habitat or predator-prey dynamics of the fishery or of other fisheries within the ecosystem. Since the Magnuson-Stevens Act was passed in 1976, the South Atlantic and Gulf of Mexico councils have established and revised allocations to varying degrees for the mixed-use fish stocks they manage in their regions. The South Atlantic council has established allocations for almost all of its mixed-use fish stocks and the Gulf of Mexico council has done so for certain stocks. Based on documents from the South Atlantic council, we found that the council has established allocations for 50 of the region’s 51 mixed-use fish stocks. The council first established an allocation for one fish stock—king mackerel—in 1985. From 1987 through 2010, the council set allocations for eight fish stocks. The council then established most allocations, encompassing 40 of its mixed-use fish stocks, in 2011, with allocations generally based on estimates of each fishing sector’s historical landings. The council’s most recently established allocation—for a cobia stock—was in 2014, according to council documents. Appendix I provides additional information on the allocations for the mixed-use fisheries in the South Atlantic council region and the years in which the council established and revised allocations. According to South Atlantic council staff, the council’s approach to revising allocations has been to rely on stakeholder input to inform them of allocations that may need revision but to otherwise leave established allocations in place. For example, council staff noted that the allocation for king mackerel—which distributes a percentage of the annual catch limit to each fishing sector—has not changed since 1985 because it is still effective for both the commercial and recreational fishing sectors. Council staff explained that because neither sector has typically caught the amount of king mackerel they have been allocated, the council has not needed to revise the allocation. As of December 2019, the South Atlantic council had revised allocations for most of their mixed-use fish stocks once, according to council documents, as shown in table 3. The council revised allocations for 30 fish stocks in 2012, based on changes to the source of recreational catch data the council was using in its formulas for calculating allocation percentages. The South Atlantic council has revised few allocations more than once. Specifically, they revised allocations for two fish stocks twice and for one, dolphin, three times. For example, the council first established an allocation for dolphin (also known as mahimahi, dolphinfish, and dorado) in 2003. It established the allocation to maintain the fishery as predominantly recreational and based the allocation on historical landings, according to the council’s fishery management plan (see fig. 4). According to council documents, the council then revised the dolphin allocation three times: in 2011, when initially setting annual catch limits for dolphin, in 2013, based on changes to the source of recreational catch data used to calculate allocation percentages, and in 2015, because the recreational sector had not been catching the amount of fish it was allocated, and the council was concerned that the commercial sector could exceed its allocation in the future. The extent to which the South Atlantic council may have considered other revisions to allocations is unclear. For example, South Atlantic council staff said that their council had deliberated on revising allocations for some fish stocks at council meetings, but they do not have records of the deliberations because the council decided not to make revisions and did not initiate related fishery management plan amendments. South Atlantic council staff explained that they document all allocation revisions through fishery management plan amendments, but they have not otherwise formally documented reviews that did not result in revisions. Council staff said they recognize the need to better document such reviews in the future; however, the council did not identify how it plans to do so, as discussed later in this report. The Gulf of Mexico council established commercial and recreational allocations for nine of the region’s 23 mixed-use fish stocks, according to documents from the council (see app. I for allocations for the mixed-use fisheries in the Gulf of Mexico council region). Council staff said most of the council’s allocations were made based on estimates of each sector’s historical landings. The council has not established allocations for most mixed-use fish stocks in the region because allocations for these stocks have not been warranted, according to council staff. Council staff said the council generally considers establishing allocations when stakeholders identify issues, or if new information such as a stock assessment becomes available and indicates that allocations may be needed to help manage a fish stock. In the absence of such information, the Gulf of Mexico council manages the fish stocks with other methods— for example, with seasonal closures or trip or bag limits, which establish the number of fish that can be legally taken in a specified period. As of December 2019, the Gulf of Mexico council had revised allocations for three mixed-use fish stocks, as shown in table 4. For example, the council revised the allocation for red grouper in 2008 to increase the recreational sector’s allocation after a stock assessment indicated the fishery had recovered from overfishing, according to a council document. In 2008, the council also revised the gag grouper allocation to increase the commercial sector’s allocation. In addition, the Gulf of Mexico council completed a fishery management plan amendment in 2015 that revised the red snapper allocation by increasing the recreational sector’s percentage. However, after the Secretary of Commerce approved the amendment in 2016, a U.S. District Court vacated the amendment in 2017, and the council returned to the initial allocation established for red snapper. Gulf of Mexico council staff said the council has not identified a need to revise allocations for the other mixed-use fish stocks in the region with allocations. For instance, for the deep water grouper and tilefish complexes, council staff said there has been limited competition between the recreational and commercial fishing sectors and the council has not needed to revise the allocations initially established for those fish stocks in 2011. When the Gulf of Mexico council has considered revising allocations, it has done so through fishery management plan amendments, according to council staff. For example, in a 2016 fishery management plan amendment, the council considered revising the allocation for king mackerel because estimates indicated that the recreational sector had not been landing the amount of fish it was allocated. However, the council decided not to revise the allocation, citing the potential for increased recreational fishing for king mackerel in the future. Through our review of agency documents and interviews with NMFS and South Atlantic and Gulf of Mexico council staff, we found that various sources of information may be available to help NMFS and the councils review allocations, but each source presents some challenges to councils for supporting allocation decisions. Councils can use these sources of information to consider the factors NMFS’ 2016 guidance calls for— including fishery performance and change, economic, social, and ecological factors—when reviewing allocations. Five key sources of information that NMFS and the councils identified are trends in catch and landings, stock assessments, economic analyses, social indicators, and ecosystem models. NMFS officials said that the councils would like to incorporate these key sources into their allocation reviews, and use such information in supporting future allocation decisions. However, they said the availability, specificity, or quality of information can present challenges to using some of the information. In particular, they noted that available information other than landings is often sparse and uncertain for many fish stocks. As a result, the officials said it may be difficult for the councils to use such information as the basis for allocation decisions. NMFS is taking some steps to improve the information available, as discussed below. NMFS’ 2016 guidance states that changes in the performance or conditions of a fishery may indicate the need for updated allocations. Fishery performance and change factors include trends in catch or landings. Data on historical and current catch and landings can provide the councils with important information about demand, according to NMFS guidance, including whether a fishing sector may be catching above or below its allocation. Generally, NMFS collects landings data for commercial fisheries from state fisheries agencies, who obtain landings data from monthly reports submitted by seafood dealers on the weight and value of fish sold at the dock. NMFS collects data to estimate recreational catch and landings through survey and interview methods through its Marine Recreational Information Program. However, recreational catch estimates present some limitations. A 2017 National Academies study noted that obtaining reliable data on recreational catch can be challenging because of several attributes of the recreational fishing sector. For example, the greater number of recreational anglers compared with the number of participants in the commercial fishing sector, and the greater number of access and landing points available to recreational anglers, make it difficult to obtain reliable data on the extent of recreational fishing, according to the study. In 2018, the Marine Recreational Information Program updated how NMFS estimates recreational catch based on a change in the survey methodology used to collect data from anglers on the Atlantic and Gulf of Mexico coasts. According to NMFS documents, updated recreational catch estimates for many fish stocks are several times higher than previous estimates because of the change in methodology. However, any implications these updated estimates may have for allocations in the South Atlantic and Gulf of Mexico may not be fully understood until NMFS incorporates the estimates into stock assessments, which were scheduled for completion between 2019 and 2021, according to NMFS documents. Further, in the Gulf of Mexico, states collect recreational catch data through their own programs, which supplement NMFS’ Marine Recreational Information Program data. The states’ programs use different methodologies, however, which Gulf of Mexico council staff said make it difficult to reconcile the states’ recreational fisheries data with NMFS’ data on catch estimates. According to an NMFS document, some of the different methodologies the states use to design surveys have produced different estimates in years when two or more surveys were conducted side by side, making it difficult to determine the best estimates of recreational catch in the Gulf of Mexico. NMFS is taking steps to improve its recreational catch estimates. For instance, in September 2019 NMFS issued procedural guidance to help ensure that survey estimates from the Marine Recreational Information Program are based upon the best scientific information available and to promote nationwide consistency in collecting data and estimating recreational catch. NMFS is also working with Gulf of Mexico states to evaluate the critical assumptions made by each state’s data collection program and to help ensure that the states’ recreational catch estimates are comparable across years and with other states. As part of this effort, NMFS is calibrating recreational catch estimates from Gulf of Mexico states with data from the Marine Recreational Information Program. According to an agency official, NMFS anticipates completing this effort in May 2020. Stock assessments are a key source of information the councils can use to review allocations given the information they provide on the status of fish stocks, according to NMFS documents. Stock assessments can range in complexity from a simple description of historical trends in catch and landings to complex assessment models that incorporate spatial and seasonal analyses in addition to ecosystem or multispecies considerations. Stock assessments are not available for all fish stocks with allocations, however. In the South Atlantic, 32 of the 50 mixed-use fish stocks with allocations do not have stock assessments, according to council staff. Of these fish stocks, NMFS plans to complete stock assessments for three—gray triggerfish, scamp, and white grunt—by 2024, according to South Atlantic council staff. In the Gulf of Mexico, stock assessments are available for the mixed-use fish stocks with allocations, with the exception of the shallow and deep water grouper aggregate complexes. Stock assessments can provide maps of the spatial distributions of fish stocks and may show changes in those distributions over time, according to NMFS officials. Changes in a fish stock’s distribution may lead to allocation disputes, and basing allocations on historical catch may not be appropriate in such situations, according to an NMFS document. NMFS’ 2016 guidance states that the councils may need to update allocations if the distributions of fish stocks change over time for reasons such as climate change or natural fluctuations in abundance. However, NMFS officials noted that few stock assessments incorporate spatial models that would allow forecasts of future spatial distributions. To help improve the availability of such information, NMFS is conducting evaluations that will, among other things, assess changes in the distribution of fish stocks in the Gulf of Mexico and South Atlantic in response to regional climate change impacts. NMFS officials said they anticipate completion of these evaluations in 2020, which will help them forecast future spatial distributions for some fish stocks going forward. In addition, stock assessments are one source of information that the councils can use to assess each fishing sector’s expected ecological impacts, according to NMFS officials. For example, NMFS officials said that stock assessments commonly provide information on each sector’s discards—fish intentionally thrown back. Discards may be caught as bycatch—that is, incidentally to the harvest of the primary fish stock targeted. NMFS’ 2016 guidance states that councils can consider the expected impacts of each fishing sector’s allocation on bycatch and bycatch mortality. However, the availability and certainty of bycatch and discard information can vary, according to NMFS officials. NMFS is taking steps to improve information on bycatch and discards. For instance, beginning in 2020, the for-hire component of the recreational fishing sector is to use an electronic system to report its bycatch and discards in the South Atlantic and Gulf of Mexico, according to NMFS officials. The officials said that the commercial fishing sector will begin using this system by 2023. NMFS officials said that the agency is also developing a model that will, among other things, estimate the number of released fish caught by the recreational fishing sector in the South Atlantic and Gulf of Mexico. The officials said that the first version of the model is focused on gag grouper in the Gulf of Mexico, but that the model could be customized to any fish stock with the necessary data available. As of December 2019, NMFS officials anticipated completion of the model by late 2020 and estimated that the model would be ready to incorporate into stock assessments in fiscal year 2021 or later. Economic analyses can provide information on the economic consequences of allocations, according to NMFS documents. NMFS’ 2016 guidance notes that councils should consider if the current or preferred allocation results in the most economically efficient use of the fishery resource. According to the guidance and NMFS officials, economic efficiency refers to how well scarce resources are used in production and consumption, and is achieved when all resources are allocated to their most valuable productive use. In principle, an allocation is most economically efficient when the net economic benefits to the commercial and recreational fishing sectors in total are maximized. If net economic benefits are not maximized, then modifying the allocation may increase economic efficiency and economic benefits to the nation. NMFS officials said the agency focuses on conducting economic efficiency analyses to help guide allocation reviews. Economic efficiency analyses can help NMFS and the councils analyze whether a proposed change in an allocation would generate greater net economic benefits for society (that is, improve economic efficiency), compared with the current allocation, according to NMFS officials. We found the councils face challenges in using economic efficiency analyses in allocation decisions. According to NMFS officials and the agency’s published research, reliable data for estimating economic values associated with recreational fishing may not be readily available. This is because no market prices for fish caught by private anglers are available and thus, non-market valuation techniques must be used to estimate the marginal value of fish to recreational anglers. For example, a 2014 NMFS study on the economic efficiency of allocations for gag, red, and black grouper found that there are insufficient data on the recreational harvest by grouper species to generate statistically reliable estimates of economic value for each fish stock. In addition, it is difficult to estimate the economic value associated with one fish stock because recreational anglers may be willing to catch other species of fish if fishery managers limit anglers’ access to a particular stock, according to members of both councils’ socioeconomic panels. This transfer of effort from one fish stock to another makes it difficult to determine which fish stock drives the economic value that anglers associate with fishing. Further, a 2014 NMFS study on the economic efficiency of red snapper allocations indicated that a relevant market price that could be used as a benchmark for the recreational estimates is unavailable. The study found that in prior work the agency attempted to use charter fishing trip prices to address this concern, but no current data on charter prices existed to update that analysis. As a result, the study cautioned against comparing estimates of recreational value to that in the commercial sector, which is a key aspect of determining an economically efficient allocation. Moreover, two 2014 NMFS studies found that there are also methodological and data challenges related to obtaining economic information from the commercial fishing sector. For example, the studies raised questions about the quality of some of the price data that were used in developing estimates of economic values for the commercial sector. In addition, the studies’ estimates of the economic value of commercial fishing did not include the potential net value derived from other components of the commercial seafood supply chain, such as the processing, distribution, and sale of the fish to the end consumers, according to the NMFS studies and agency officials (see fig. 5). These NMFS studies noted that data for estimating the values from these other components are not readily available. Council staff and members, socioeconomic panel members, and fishery stakeholders we interviewed noted the importance of including the value of fish to the end consumers when considering the economic value of commercial fishing. To estimate the values of these other components of the commercial seafood supply chain, NMFS would need information about the consumer demand for fish as a function of domestic and international production, as well as information on changes in the price of the fish as they move from the dockside to retail markets, according to a separate NMFS study. NMFS officials said they are taking some steps related to improving economic analyses that the councils could consider in allocation reviews. For example, the agency is developing a manual of best practices for NMFS and council staff responsible for conducting economic analyses. NMFS officials said that they anticipate completing the manual by the end of fiscal year 2020. According to NMFS officials, the manual is intended to help (1) achieve consistency in analyses across the councils and regions, (2) establish an understanding of why economic analyses of allocations are important to fisheries management decisions, as well as their role in complying with various legal requirements and NMFS’ policy, and (3) establish an understanding of the basic concepts and tools used in these analyses and how they are expected to be applied in practice. In addition, NMFS conducted a study on the economics of the for-hire fishing sector in federal waters of the South Atlantic and Gulf of Mexico and completed a report on the study at the end of 2019. Among other things, agency officials said the study provides data sufficient to estimate producer surplus for the for-hire sector. This information could help inform future allocation decisions, according to NMFS officials. NMFS has developed social indicators to characterize community well- being for coastal communities engaged in fishing activities, which the councils could consider in reviewing allocations, according to NMFS officials. NMFS’ 2016 guidance states that the councils could consider individual, local, and regional fishing dependence and engagement, and that such analyses should include potential impacts on commercial, for- hire, private angler, and subsistence fishing, as well as fishing-related industries if data are available. NMFS’ social indicators are numerical measures that describe the well-being of fishing communities in coastal counties across the United States and their level of dependence on commercial and recreational fishing. For example, one indicator describes the vulnerability of fishing communities to disruptive events, such as a change to a fishing sector’s access to a fishery. Communities that are dependent on commercial fishing can be more socially vulnerable than other communities to changes, according to an NMFS document. However, NMFS’ social indicators on communities’ reliance on and engagement in commercial and recreational fishing are not specific to particular fish stocks. NMFS officials said this makes it challenging for councils to incorporate the information into their allocation reviews for specific fish stocks. The officials said that given current resource limitations and limited data available, it would be difficult to generate social indicators that are specific to fish stocks. In some instances, NMFS has some stock-specific information at the community level for the commercial fishing sector. But NMFS officials said that comparable information is not available for the recreational sector at the community level, making it difficult to develop fish stock-specific social indicators. NMFS officials said that the agency continues to work to update and improve social indicators relevant to recreational and commercial fisheries and is exploring other sources to provide better social data for fisheries management decisions. However, NMFS officials did not identify specific steps they plan to take to improve social indicators—such as developing information specific to particular fish stocks—so that the councils could more easily incorporate such information into their allocation reviews. NMFS’ 2016 guidance calls for the councils to consider the potential ecological impacts of allocation alternatives in determining the allocation between different sectors or groups. However, NMFS officials said there are few ecosystem models that incorporate ecological information that could be considered in reviewing allocations, in part because limited quantifiable ecological information is available. They said that it will be difficult to use ecosystem models in allocation decisions until such models are more fully developed. NMFS officials said they are taking some steps to enhance the use of ecological and ecosystem-based information. For instance, they noted that in 2016, NMFS released a policy to, among other things, establish a framework of guiding principles to enhance and accelerate the implementation of ecosystem-based fisheries management. Ecosystem- based fisheries management is a systematic approach to fisheries management in a geographically specified area that: contributes to the resilience and sustainability of the ecosystem; recognizes the physical, biological, economic, and social interactions among the affected fishery- related components of the ecosystem, including humans; and seeks to optimize benefits among a diverse set of societal goals, according to the policy. Among other things, this approach can help communicate the potential consequences of management decisions—including allocations—across fish stocks and improve the understanding of the potential benefits and effectiveness of management decisions, according to the policy. In 2019, NMFS issued plans for implementing ecosystem- based fisheries management in the South Atlantic and Gulf of Mexico. The South Atlantic and Gulf of Mexico councils each established criteria for initiating allocation reviews in response to NMFS’ 2016 guidance, but neither council has developed processes to guide how they will conduct or document their allocation reviews. The Gulf of Mexico council has taken initial steps to develop a process for how it will review allocations, and staff from both councils said they are waiting for our report to inform their next steps on developing processes for conducting allocation reviews in the future. The North Pacific council plans to review The four councils also identified public input as a potential allocation review trigger, but they did not specify what threshold of public interest would trigger a review. The remaining two councils—the Western Pacific and Caribbean—do not have allocations subject to National Marine Fisheries Service (NMFS) policy requiring councils to establish allocation review criteria, according to NMFS officials. the council reviews a fishery performance report. The South Atlantic council’s policy also established time-based triggers as secondary criteria for initiating allocation reviews. Its policy states that the council will review allocations not less than every 7 years if one of the conditions identified in the policy has not already triggered a review. The policy also states that once a review occurs, the next one will be automatically scheduled for 7 years later. In contrast, the Gulf of Mexico council’s April 2019 policy established time-based triggers as its primary criteria for initiating allocation reviews. Specifically, its policy indicates time intervals of 4 to 7 years for reviewing allocations, depending on the particular fish stock, and identifies the planned month and year for beginning each review. The council’s policy also identified public interest as a secondary allocation review trigger but did not specify thresholds for the level or type of public input that would trigger an allocation review. According to the policy, the council is to consider relevant social, economic, and ecological conditions as an intermediate step before determining whether public interest will trigger a review. According to NMFS’ 2016 guidance, periodic review of allocations on a set schedule is in several respects the most simple and straightforward criterion for such a review—it is unambiguous and less vulnerable to political and council dynamics. The guidance also states that time-based triggers for initiating allocation reviews might be most suitable for fisheries where the conflict among sectors or stakeholder groups makes the decision to simply initiate a review so contentious that use of alternative criteria is infeasible. In such a situation, a fixed schedule ensures that periodic reviews occur regardless of political dynamics or specific fishery outcomes, according to the guidance. However, the guidance also indicates that, compared with alternative approaches, adherence to a fixed schedule may be less sensitive to other council priorities and the availability of time and resources to conduct such reviews, which could potentially lead to significant expenditures. Therefore, given the inflexible nature of time-based triggers, the guidance recommends that they be used only in those situations where the benefit of certainty outweighs the costs of inflexibility. The South Atlantic and Gulf of Mexico councils’ policies laid out planned schedules for their respective allocation reviews, which both councils adjusted after issuing their policies. Table 5 shows both councils’ plans for allocation reviews as of December 2019. For example, the Gulf of Mexico council’s policy states that it plans to review the red grouper allocation in 2026. However, in response to the completion of an updated stock assessment for red grouper in July 2019, the council directed its staff in October 2019 to begin work on a fishery management plan amendment to update the red grouper allocation, according to a council document. The stock assessment for red grouper included the Marine Recreational Information Program’s updated estimates for recreational landings. The updated estimates approximately doubled previous estimates of recreational landings, according to a council newsletter. Council staff said that applying these updated estimates to the time series the council had used to establish the red grouper allocation could result in a percentage shift of the allocation to the recreational fishing sector. As a result, the council decided to begin review of the red grouper allocation sooner than the policy’s scheduled 2026 time frame, according to the staff. In addition, we found that the councils’ planned allocation review schedules may affect their workload and other priorities, but it is not clear to what extent. NMFS’ 2016 allocation guidance states that the councils’ allocation review processes should include consideration of current council priorities, other actions under deliberation, and available resources. NMFS officials and council staff expressed concern that the councils’ planned schedules—as identified in their April and July 2019 policies—may negatively affect the workloads and other priorities of NMFS’ social scientists, economists, and data analysts and council staff. For instance, staff from both councils said the planned allocation review schedules will increase their workloads and, depending on the nature and substance of how those reviews are conducted, could take resources away from other council activities and lead them to reprioritize or delay those activities. One council’s staff also noted that the council members have a difficult time keeping up with existing workloads. NMFS officials and council staff said that factors that may affect these types of costs include the complexity of the analyses, the number of NMFS or council staff involved in the process, and the degree of public interest. Fishery management plan amendments that establish or revise allocations can be controversial, and will likely have more public hearings and opportunity for public comment than other types of amendments, according to NMFS officials and council staff. NMFS officials and South Atlantic and Gulf of Mexico council staff said they have not tracked costs of establishing, reviewing, or revising allocations. The councils often make allocation decisions concurrently with other management actions, making it difficult to isolate costs. Further, NMFS officials stated the councils’ accelerated schedules as of December 2019, as shown in Table 5, will exacerbate the concerns. These schedules include starting reviews for 50 allocations in the South Atlantic between 2019 and 2026, assuming no conditions trigger earlier reviews, and reviews for 10 allocations in the Gulf of Mexico between 2019 and 2026. One NMFS official said that any additional workload for economists and social scientists in the Southeast Fisheries Science Center is difficult to anticipate because it will depend on the type of information the councils would like to use for the reviews and whether additional studies may be needed or data collected. Another NMFS official stated that the regional office will shift priorities from less important tasks and gain efficiencies where possible to accommodate the planned allocation reviews. The South Atlantic and Gulf of Mexico councils have not developed processes for how they will conduct or document their allocation reviews to implement NMFS’ 2016 policy and related guidance, although the Gulf of Mexico council has begun taking steps to do so. As noted, NMFS policy calls for a multi-step process for reviewing and potentially revising fisheries allocations. Specifically, once an allocation review trigger has been met, NMFS policy calls for an allocation review, after which the councils may maintain existing allocations or evaluate allocation options through a fishery management plan amendment. NMFS guidance states that the councils should develop a structured and transparent process for conducting allocation reviews, including consideration of current council priorities, other actions under deliberation, and available resources. In April 2019, the Gulf of Mexico council began taking steps to develop an allocation review process, according to council documents. Specifically, the Gulf of Mexico council convened an allocation review workgroup consisting of staff from the council and from NMFS’ Southeast Regional Office and Southeast Fisheries Science Center. The council expects the workgroup to propose draft allocation review procedures, including identifying data sources that would be needed to conduct allocation reviews, according to a council document. The workgroup met in June and July 2019 and discussed these topics and other potential proposals, such as establishing a tiered system for allocation reviews that would involve different levels of analysis for different tiers of reviews, according to council documents. Council staff said the workgroup plans to next meet after the issuance of our report to finalize a proposal for developing an allocation review process for the council to consider. However, the council has not indicated what actions it will take, if any, regarding the workgroup’s proposal; instead, the council will determine its course of action after reviewing this report, according to council staff. The South Atlantic council postponed discussion of defining or documenting its allocation review process until March 2020, according to council staff and members, to review our report before deciding any next steps. At the council’s June 2019 meeting, the council chair questioned the need for developing an allocation review process through policy. For instance, the chair cited concerns that the council may be continuously developing exceptions to such a policy to accommodate fishery-specific issues or other unique circumstances. The chair also stated that aside from establishing criteria for initiating allocation reviews, NMFS’ guidance does not require the councils to take other actions related to developing allocation review processes. NMFS officials said that the agency’s 2016 guidance recommending that the councils develop a structured and transparent process was not intended to require the councils to develop a separate policy or documented process for conducting allocation reviews. NMFS officials said that the agency’s operational guidelines for processes under the Magnuson-Stevens Act and associated regional operating agreements with the councils lay out the key requirements and processes guiding development, review, and implementation of fishery management plans and plan amendments, which would include actions related to allocations. The officials further explained that in developing the 2016 allocation policy, they intended that allocation reviews be conducted through the processes identified in the agency’s operational guidelines and regional operating agreements with the councils, which allow the councils flexibility to factor in their own needs. However, the operational guidelines and regional operating agreements for the South Atlantic and Gulf of Mexico councils apply to the fishery management plan and amendment process overall, and they do not specifically address allocations. The goals of the operational guidelines include promoting a timely, effective, and transparent public process for development and implementation of fishery management measures, and the guidelines note that the regional operating agreements are meant to make council procedures and processes transparent. The guidelines and agreements, however, do not lay out processes the councils are to follow in reviewing allocations apart from developing fishery management plans or plan amendments. As noted in NMFS’ 2016 policy and guidance, the councils may conduct allocation reviews separate from the fishery management plan amendment process. Moreover, the regional operating agreements are not intended to limit or prevent the councils’ use of additional processes in response to specific management needs, according to these documents and the operational guidelines, and the Gulf of Mexico council has taken initial steps in developing an allocation review process as previously described. Based on the framework for internal controls established by the Committee of Sponsoring Organizations of the Treadway Commission, documented policies and processes can be more difficult to circumvent, less costly to an organization if there is turnover in personnel, and increase accountability. The framework also states that when subject to external party review, policies and processes would be expected to be formally documented. Among other things, documented processes— according to the framework—promote consistency; assist in communicating the who, what, when, where, and why of internal control execution; enable proper monitoring; and provide a means to retain organizational knowledge and mitigate the risk of having the knowledge within the minds of a limited number of individuals. The 2012 report commissioned by NMFS to review fisheries allocation issues found that allocation reviews had not been done in a regular, consistent manner and stated that this makes it harder for stakeholders to understand the reviews as well as the process for conducting them. Similarly, stakeholders we interviewed indicated that a clear process for conducting allocation reviews is needed and would increase their confidence in or understanding of the councils’ decisions, regardless of specific outcomes. Other stakeholders stressed the need for predictability and certainty to be able to plan critical business decisions, such as securing loans from local banks or other lenders. Such uncertainty may cause participants in the commercial sector to leave the fishery because they cannot secure loans or meet other business requirements, according to one stakeholder, or it may create instability that could affect the market price of fish, according to another stakeholder. By working with the councils to develop documented allocation review processes, NMFS would have better assurance that the councils carry out their upcoming allocation reviews in a structured and transparent manner, consistent with the agency’s 2016 guidance. Further, it is unclear whether or how the councils plan to document each allocation review, such as the basis for their allocation decisions, whether fishery management plan objectives are being met, and what factors were considered in each review. NMFS’ operational guidelines state that fishery management decisions must be supported by a record providing the basis for the decision. In addition, NMFS’ 2016 policy and guidance call for the councils to clearly articulate in their allocation reviews how fishery management plan objectives are or are not being met, as well as to document their rationale for determining whether any factors are unimportant or not applicable in making an allocation decision. NMFS officials and council staff said that any allocation revisions would be documented through fishery management plan amendments. However, the councils may conduct allocation reviews separate from the fishery management plan amendment process, and it is not clear whether or how the councils will document those reviews. For example, as previously noted, in the past the South Atlantic council has not formally documented the results of allocation reviews that did not lead to fishery management plan amendments that revised the allocations. By working with the councils to specify how they plan to document their allocation reviews, NMFS could help ensure that the councils provide a clear record of the basis for their decisions, whether fishery management plan objectives are being met, and applicable factors considered. Clear records could also help increase transparency and stakeholder understanding of the councils’ decisions, particularly in those instances when reviews are separate from the fishery management plan amendment process. Making allocation decisions between the commercial and recreational fishing sectors can be complex and difficult, and the outcomes of those decisions may have important economic and social implications for stakeholders in each of the sectors. The South Atlantic and Gulf of Mexico councils have taken an important step in developing policies outlining criteria for initiating allocation reviews, in accordance with NMFS guidance. The Gulf of Mexico council has also taken initial steps to define how it will conduct its allocation reviews. However, neither council has developed a process for how they will conduct such reviews. By working with the councils to develop documented allocation review processes, NMFS would have better assurance that the councils carry out their upcoming allocation reviews in a structured and transparent manner, consistent with the agency’s 2016 guidance. Moreover, by working with the councils to also specify how they plan to document their allocation reviews, NMFS could help ensure that the councils provide a clear record of the basis for their decisions, whether fishery management plan objectives are being met, and applicable factors considered. We are making the following two recommendations to the NMFS Assistant Administrator for Fisheries: The NMFS Assistant Administrator for Fisheries should work with the South Atlantic and Gulf of Mexico councils, and other councils as appropriate, to develop documented processes for conducting allocation reviews. (Recommendation 1) The NMFS Assistant Administrator for Fisheries should work with the South Atlantic and Gulf of Mexico councils, and other councils as appropriate, to specify how the councils will document their allocation reviews, including the basis for their allocation decisions, whether fishery management plan objectives are being met, and what factors were considered in the reviews. (Recommendation 2) We provided a draft of this report to the Department of Commerce for review and comment. In written comments (reproduced in app. II), Commerce and NOAA agreed with our recommendations and stated that NOAA’s NMFS will work to implement them to the extent possible. NOAA stated that the report accurately describes the extent to which the councils established and revised allocations for mixed-use fisheries, the key sources of information that may be available for reviewing allocations, and the extent to which the councils have developed processes to help guide such reviews. NOAA also highlighted the delicate balance that councils seek to achieve in deciding what fishery management approaches to implement to comply with the Magnuson-Stevens Act and its 10 national standards. In addition, Commerce and NOAA stated that NMFS does not have the legal authority to direct the councils to take the actions included in our two recommendations, stating that such actions are outside of legal requirements that guide council fishery management actions. In response, we revised the wording of our two recommendations to state that the NMFS Assistant Administrator for Fisheries should “work with,” rather than “direct,” the councils to take the recommended actions. In response to our first recommendation, NOAA stated that it would build on the recommendations in its allocation policy by working with the South Atlantic and Gulf of Mexico councils, and other councils as appropriate, to develop documented processes for conducting allocation reviews. In response to our second recommendation on specifying how the councils will document their allocation reviews, NOAA stated that it will work with the councils on consistent documentation of allocation reviews. NOAA noted that transparency in the allocation process improves with a documented process for conducting allocation reviews, and that consistent documentation of those reviews will create further transparency in the allocation process and could improve stakeholders’ understanding of the councils’ decisions. NOAA also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Commerce, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Tables 6 and 7 provide information on mixed-use fisheries allocations— privileges for catching fish between the commercial and recreational fishing sectors—in the South Atlantic and Gulf of Mexico Fishery Management Council (council) regions, respectively. Not all mixed-use fish stocks in these regions have allocations. In the South Atlantic council region, spiny lobster does not have an allocation. In the Gulf of Mexico council region, 14 of 23 mixed-use fish stocks do not have allocations. Anne-Marie Fennell, (202) 512-3841 or fennella@gao.gov. In addition to the contact named above, Alyssa M. Hundrup (Assistant Director), Krista Breen Anderson (Analyst in Charge), Leo Acosta, Mark Braza, Tim Guinane, Paul Kazemersky, Patricia Moye, Cynthia Norris, Dan C. Royer, Rebecca Sandulli, Kiki Theodoropoulos, and Khristi Wilkins made key contributions to this report.", "summary": "Commercial and recreational marine fisheries—including those in the South Atlantic and Gulf of Mexico—are critical to the nation's economy, contributing approximately $99.5 billion to the U.S. gross domestic product in 2016, according to the Department of Commerce. NMFS and the councils may allocate fishing privileges for mixed-use fisheries in federal waters, but establishing and revising such allocations can be complex, in part because of concerns about equity. The Modernizing Recreational Fisheries Management Act of 2018 includes a provision for GAO to review mixed-use fisheries allocations in the South Atlantic and Gulf of Mexico. For these regions, this report examines (1) the extent to which the councils established or revised mixed-use fisheries allocations, (2) key sources of information that may be available for reviewing allocations, and (3) the extent to which the councils have developed processes to help guide such reviews. GAO reviewed NMFS and council policies and other council documents; analyzed information on allocations established and revised; compared council processes to agency guidance and internal control standards; and interviewed NMFS officials, council members and staff, and 46 stakeholders that reflected various interests. Views from these stakeholders are not generalizable. The South Atlantic and Gulf of Mexico regional fishery management councils, with approval from Department of Commerce's National Marine Fisheries Service (NMFS), established and revised allocations to varying degrees for mixed-use fish stocks—fisheries with a combination of commercial and recreational fishing. Regional councils were created by statute to help manage fisheries in federal waters, including allocating—or distributing—fishing privileges, when warranted. Starting in 1985, the South Atlantic council established allocations, generally a percentage of allowable harvest, for 50 of its 51 mixed-use fish stocks and revised most of those at least once. The Gulf of Mexico council established allocations for nine of its 23 mixed-use fish stocks, revising three of those once. Historically, allocations have been largely based on estimates of the commercial and recreational fishing sectors' past use of the resource, according to NMFS. Key sources of information that may be available to help NMFS and the councils review allocations include trends in catch and landings (the amount of fish caught or brought to shore); fish stock assessments; and economic analyses. Each source presents some challenges in supporting allocation decisions, however. For example, NMFS works with states to estimate recreational catch, which provides information about demand, but faces difficulties generating reliable estimates. This is in part because of attributes of the recreational fishing sector, including the greater number of recreational anglers compared with commercial fishing participants. NMFS issued guidance in 2019 to promote consistency in estimating recreational catch data to help improve the quality of the information. The South Atlantic and Gulf of Mexico councils developed processes for when to initiate fish stock allocation reviews, but not for how to conduct those reviews. A 2012 report for NMFS found that reviews had been done inconsistently, and stakeholders were dissatisfied with allocation decision-making. In response, NMFS developed guidance calling for structured and transparent allocation review processes. Both councils established criteria for initiating reviews, such as time-based triggers, and as of December 2019 they had several reviews underway (see figure). In April 2019, the Gulf of Mexico council began convening a workgroup to propose a draft allocation review process, but has not indicated what actions it will take, if any, in response to a proposal. The South Atlantic council postponed any discussions until March 2020. As of December 2019, neither council had a documented process. Documented processes for conducting allocation reviews would provide NMFS with better assurance that the councils carry out upcoming reviews in a structured and transparent manner. GAO is making two recommendations, including that NMFS work with the councils to develop documented processes for conducting allocation reviews. The agency agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Federal law specifies that the EPSDT benefit covers screening, vision, dental, and hearing services, as well as other Medicaid coverable services that are necessary to correct or ameliorate any conditions discovered through screening. The EPSDT benefit generally entitles beneficiaries to these services regardless of whether such services are covered in a state’s Medicaid state plan and regardless of any restrictions that the state may impose on coverage for adult services. The EPSDT screening component includes a wide range of preventive services, such as comprehensive child health assessments known as well-child screenings and age-appropriate blood lead screenings. Because EPSDT covers any medically necessary service that could be covered for adults in addition to the specified preventive screenings, the EPSDT benefit is generally more comprehensive than the benefits provided for adult beneficiaries. The federal government and states jointly share responsibility for implementing the EPSDT benefit. CMS, as part of its Medicaid oversight responsibilities, approves state Medicaid plans, which describe how the state administers its Medicaid program, including components related to the provision of EPSDT services. CMS also develops and issues general guidance to states about the EPSDT benefit, such as explanations of covered services and strategies for providing those services. Additionally, CMS has developed a goal for EPSDT, which is to assure that beneficiaries get the health care they need when they need it: the right care to the right child at the right time in the right setting. Further, CMS established performance measures, some with associated targets, to guide states’ implementation of EPSDT. For example, CMS set performance measures and performance measure targets as part of its Oral Health Initiative. CMS developed the performance measure targets to carry out statutory requirements, quality improvement efforts, and agency policy. (See table 1 for EPSDT performance measures that have associated targets.) States have flexibility, within federal parameters, to determine how EPSDT services are provided. For example, states are required to ensure that Medicaid-eligible beneficiaries and their families are aware of the EPSDT benefit and have access to required services, but states can choose whether to administer the benefit themselves or to oversee managed care organizations that are contracted to provide the benefit. States may also determine the frequency of screening services and communicate them through periodicity schedules that meet federal requirements. CMS uses various sources of information to oversee the EPSDT benefit, such as the CMS-416, the Child Core Set, and the Medicaid and CHIP Scorecard. States report information about the provision of select ESPDT services to CMS annually through the CMS-416 and measures on the Child Core Set. The CMS-416 provides CMS with basic information about EPSDT services, such as the participant ratio and number of beneficiaries receiving a preventive dental service. It includes the information necessary for CMS to assess states’ performance on the participant ratio and the screening ratio, among other things. The agency then can compare performance on the two ratios with the agency’s ESPDT performance measure targets. The Child Core Set provides CMS with information about the quality of health care provided to Medicaid and CHIP beneficiaries, and supports state efforts to improve health care quality and health outcomes. Child Core Set reporting becomes mandatory on an annual basis beginning with the state reports on fiscal year 2024. As of 2019, the Child Core Set included performance measures related to the provision of EPSDT services, such as well-child visits in the first 15 months of life. Because reporting is currently voluntary, states vary in the number of performance measures they choose to report. In fiscal year 2017, for example, 50 states and the District of Columbia voluntarily reported on at least one of the 27 Child Core Set performance measures, with states reporting a median of 18 Child Core Set performance measures. (See app. II for the information reported in the CMS-416 and Child Core Set.) As shown in table 2, there are both similarities and differences between the CMS-416 and Child Core Set. Since the Child Core Set performance measures include CHIP beneficiaries who may not be entitled to the EPSDT benefit, data from the Child Core Set are not directly comparable with reporting on the CMS- 416. In addition, CMS-416 data cover a longer period of time, as they are available from 1995, while Child Core Set data are available from 2011. CMS officials said that having more years of CMS-416 data helps identify trends in the provision of EPSDT services over a longer period of time than possible with the Child Core Set. On the other hand, CMS officials said it is difficult to compare states’ performance using the CMS- 416, because some performance measures are based on periodicity schedules, which vary state-to-state and over time. In contrast, the Child Core Set allows for more consistency in comparing data across states, because each state is expected to calculate performance measures in the same way. In June 2018, CMS published the first Medicaid and CHIP Scorecard, which includes performance measures about the provision of services to Medicaid and CHIP beneficiaries. The scorecard includes 17 performance measures related to the provision of EPSDT services, six of which are performance measures from the Child Core Set—and one of these six measures is derived from the CMS-416. In January 2019, CMS officials reported that the scorecard will be used to provide increased transparency about state Medicaid program administration and beneficiary health outcomes, and drive health care quality improvement across states. According to CMS officials, CMS envisions that the scorecard will be strengthened as state reporting of data through T-MSIS becomes more timely, accurate, and complete. CMS has been working since 2011 to implement T-MSIS as a replacement for some current reporting to improve and increase states’ reporting of Medicaid and CHIP data. CMS intends for T-MSIS to provide a national data repository to support federal and state Medicaid and CHIP program management, among other things. T-MSIS includes data not previously reported by states and is intended to improve Medicaid and CHIP program efficiency, in part, by allowing states to compare their data with other states’ data. T-MSIS includes data that can measure the provision of EPSDT services. According to CMS officials, T-MSIS also includes aspects designed to improve the accuracy of available state data. For example, states’ T-MSIS submissions undergo approximately 2,800 automated quality checks, which provide states with feedback on data format and consistency. As of January 2019, all 50 states and the District of Columbia were submitting data monthly, according to CMS, but T-MSIS data were not being used to create the CMS-416, Child Core Set, or the scorecard. Agency officials said research-ready files are in development and T-MSIS data are improving in quality over time with historical state resubmissions. According to our analysis of CMS-416 data for fiscal year 2017, millions of Medicaid beneficiaries received recommended EPSDT well-child screenings and preventive dental services. However, nearly as many eligible beneficiaries did not receive the various recommended screenings and services, and few states met CMS’s performance measure targets for EPSDT services. Additionally, while available data show that millions of blood lead screenings were performed, the total number of beneficiaries receiving blood lead screenings is unknown, because the data are incomplete. In fiscal year 2017, 20.2 million (59 percent) of the 34.2 million beneficiaries who should have received at least one recommended well- child screening received that screening, known as the participant ratio, according to our analysis of state-reported CMS-416 data. Additionally, our analysis indicates that the national participant ratio has declined 5 percentage points since fiscal year 2010. Three states met CMS’s participant ratio target of 80 percent in fiscal year 2017, as shown in figure 1. Our analysis also indicates that no more than four states met CMS’s participant ratio target in any one fiscal year from 2010 through 2017. (See app. III, table 6, for participant ratios in each state and nationally from fiscal years 2010 through 2017.) Our analysis also indicates that as beneficiaries age, they tend to receive fewer recommended well-child screenings, which results in lower participant ratios. (See fig. 2 for participant ratios and numbers of beneficiaries receiving and not receiving well-child screenings for each CMS-416 age group in fiscal year 2017.) CMS has issued a guide on serving older eligible beneficiaries, stating that regular preventive care visits can lead to early identification of health issues. CMS officials said the agency included measures focusing on these beneficiaries on the Child Core Set and Medicaid and CHIP Scorecard to recognize the importance of addressing these beneficiaries and to encourage states to focus on this population. CMS officials noted that some states have already taken steps to increase the number of well-child screenings that older eligible beneficiaries receive, for example, by partnering with schools. In fiscal year 2017, 18.3 million (48 percent) of the 38.3 million Medicaid beneficiaries aged 1 to 20 received a preventive dental service, according to our analysis of CMS-416 data. This is an increase from the 42 percent of beneficiaries receiving preventive dental services in 2011—the baseline year for measuring state progress toward CMS’s Oral Health Initiative targets—but less than CMS’s 52 percent national performance measure target. Our analysis also shows that from fiscal years 2011 through 2017, nine states met CMS’s performance measure target of a 10 percentage point increase in each state’s percentage of beneficiaries aged 1 to 20 receiving a preventive dental service. (See fig. 3 and table 8 in app. III for the percentage of beneficiaries aged 1 to 20 that received preventive dental services in each state and nationally from fiscal years 2011 through 2017.) Available data on blood lead screenings in the CMS-416 are incomplete and, as a result, do not provide information necessary to determine how many beneficiaries received the screenings. According to CMS’s November 2016 guidance, CMS-416 data do not accurately represent the number of beneficiaries receiving blood lead screenings. The CMS-416 data capture screenings paid for by Medicaid, but not those performed using funding from other sources, such as the Centers for Disease Control and Prevention. This could under-count the number of screenings performed. In addition, the blood lead screening data reported on the CMS-416 show how many screenings were performed, but do not identify the number of beneficiaries who received a blood lead screening. Our analysis of available CMS-416 data shows that in fiscal year 2017 states reported 2.0 million blood lead screenings for beneficiaries aged 12 through 24 months, and there were 4.6 million beneficiaries aged 12 through 24 months. CMS has regularly taken actions to use both the CMS-416 and the Child Core Set to improve the quality of information about the provision of EPSDT services. These actions have made the data reported about EPSDT services more complete and reliable. For example, CMS collects data annually from states on performance measures for both the CMS- 416 and the Child Core Set. (See table 3.) Additionally, CMS annually reviews the Child Core Set measures to determine whether measures need to be added, deleted, or revised. CMS also regularly provides technical assistance to states about data reliability, such as through its monthly Quality Technical Advisory Group. For example, during one group meeting, states shared challenges with reporting information about developmental screenings on the Child Core Set and suggestions for how to overcome these challenges. These actions are generally consistent with federal internal control standards regarding information and communication, which specify that management should use quality information to achieve the entity’s objectives. While CMS has taken actions to improve the quality of information about EPSDT, and agency officials said they regularly assess whether the information CMS collects on the CMS-416 is appropriate and useful for EPSDT oversight, CMS has not taken action, as needed, based on such assessments. For example, CMS has not added, removed, or amended any performance measures on the CMS-416 since 2010, even though officials acknowledge limitations in these measures. The participant ratio, for example, is dependent, in part, on a state’s chosen periodicity schedule, which means that the measure is not consistently defined across states. The screening ratio reflects the extent to which beneficiaries received the recommended number of well-child screenings during the year, but this information is aggregated and therefore cannot be used to determine whether individual beneficiaries received the recommended number of well-child screenings. Although federal law requires collecting certain information about the provision of EPSDT services, it provides the agency with flexibility to determine the form and manner in which data are collected and to set performance measures. For example, CMS could change the way states are required to calculate the participant ratio or the screening ratio, and could examine ways to do so to address the limitations that the agency has identified and improve the quality of information about the provision of EPSDT services. Because CMS has not taken action, as needed, based on assessments of the appropriateness of its CMS-416 performance measures, the agency cannot be sure that it has the information it needs to oversee state implementation of EPSDT. This is inconsistent with federal internal control standards regarding information and communication, which specify that management should identify information requirements in an iterative and ongoing manner and ensure information remains relevant. We have previously reported that results-oriented organizations set performance goals to define desired program outcomes and develop performance measures that are clearly linked to these performance goals and outcomes. CMS has taken steps to develop, assess, and use CMS-416 information to improve states’ performance in providing EPSDT services. For example, CMS has set performance measure targets for participant and screening ratios reported on the CMS-416, and CMS publishes state-level results of the participant and screening ratios. In addition, after identifying issues with calculating the performance measure and target for the permanent molar sealants, CMS removed them from the Oral Health Initiative. CMS also convenes affinity groups and technical advisory groups to provide assistance to states in improving performance, often centered on specific services, such as dental services. However, CMS and state Medicaid officials told us that CMS does not consistently (1) communicate CMS-416 performance measure targets to states, (2) evaluate state performance against performance measure targets, or (3) provide states with assistance in reaching performance measure targets. While it has not done so across all performance measure targets, CMS did take these actions regarding targets for preventive dental services as part of its Oral Health Initiative. For example, CMS communicated with states about the preventive dental service performance measure target after it developed the Oral Health Initiative; disseminated a national oral health strategy and published a review of eight states identifying innovative approaches in providing preventive dental services; and provided targeted outreach to states with the lowest performance on the preventive dental service performance measure. Improvements in the provision of dental services occurred in many states. For example, in 2013, CMS met with state Medicaid officials in Florida about improving the provision of preventive dental services. Five years later, the percentage of beneficiaries receiving preventive dental services had increased 18 percentage points. CMS has not taken action in other areas. For example: CMS does not communicate the participant and screening ratio targets. Officials from CMS and from each of our 16 selected states told us that CMS does not mention these targets in communications with states, including discussions related to performance improvement. CMS has not evaluated state performance in meeting the participant and screening ratio targets, nor has it provided focused assistance to states to resolve gaps in states’ performance in reaching these targets comparable to the assistance provided for the preventive dental screening performance measure as part of the Oral Health Initiative. CMS did not provide formal written notification to states when in March 2016 the agency informed participants in two meetings that CMS no longer planned to use the target for measuring states’ performance on the permanent molar sealants performance measure. The notification was not provided through an official policy document, such as an agency informational bulletin distributed to all states. Despite removing the target, CMS issued a technical assistance brief in March 2018 that referenced it, which could have led to confusion among state officials. With regard to the Child Core Set, CMS has not established any performance measure targets and agency officials were not able to provide information about plans for setting targets. CMS officials said that the CMS-416 will remain a part of its EPSDT oversight. However, because its information is not standardized across states, CMS plans to increasingly rely on the standardized Child Core Set data to assess and improve states’ performance on the provision of EPSDT services. CMS officials noted that it publishes median, top quartile, and bottom quartile information for each state for all the Child Core Set measures that are publicly reported. Officials further reported in June 2019 that CMS and states use these as performance benchmarks, with an aim of reaching the national median on these measures if not the top quartile. Reporting these data is an important step in ensuring better oversight of EPSDT. However, CMS has not developed fixed targets that explicitly track states’ progress in increasing beneficiaries’ receipt of EPSDT screenings and services. Using a median to assess states’ performance ensures that half the states will not meet this target, regardless of their individual performance. Further, CMS officials have not provided plans or timelines for when the Child Core Set would be used to help states achieve performance measure targets. CMS’s inaction regarding using the CMS-416 and Child Core Set to improve performance on the provision of EPSDT services limits the agency’s oversight and is inconsistent with federal internal control standards for monitoring, and practices of leading organizations. Federal internal control standards specify that management should (1) set performance measure targets in measureable, numeric terms; (2) communicate necessary information to achieve performance targets; (3) evaluate progress toward desired targets; and (4) take action to resolve identified issues. Without regularly using the CMS-416 and Child Core Set to improve the provision of EPSDT services, CMS is unable to identify whether state or federal efforts and policies are increasing the number of beneficiaries receiving EPSDT services. As a result, CMS’s oversight is limited and beneficiaries may not be receiving appropriate EPSDT services when they need them—CMS’s stated goal for EPSDT. (See table 4 for examples of actions CMS has and has not taken regarding using the CMS-416 and Child Core Set for improving the provision of EPSDT services.) CMS is unable to determine whether all eligible EPSDT beneficiaries are receiving blood lead screenings in accordance with CMS policy. As previously noted, CMS-416 data are incomplete, because they only include blood lead screenings paid for by Medicaid, and the form reports the number of screenings performed instead of the number of beneficiaries receiving screenings. State examples of collecting blood lead screening data Nebraska. Medicaid officials said that the state has developed a database with the Nebraska Health Information Initiative containing laboratory testing data. Treating providers and managed care organizations can access the database to determine whether a Medicaid beneficiary has received a blood lead screening. New Jersey. Medicaid officials said that it can be difficult to track blood lead screenings that are performed using funding from sources other than Medicaid; for example, those performed by the state health department. Officials said that they have been building a lead registry to capture data on lead screenings performed, regardless of how they are funded. New Jersey Medicaid officials said they collect data every 6 months on screenings not paid for by Medicaid and enter the data into the state’s blood lead registry. CMS has stated that screenings are important for identifying beneficiaries with elevated blood lead levels at as young an age as possible, because lead exposure can harmfully affect nearly every system of the body and cause developmental delays. According to a presidential task force on environmental health and safety risks to children, co-chaired by HHS, early identification of developmental delays allows providers and communities to intervene earlier to improve health outcomes. The presidential task force issued goals in December 2018 to reduce lead exposure and associated harms, including a goal to identify lead-exposed individuals and improve their health outcomes. Without complete information about blood lead screenings, CMS cannot identify the number of beneficiaries who have not received blood lead screenings. As a result, the agency may be unaware of beneficiaries with unidentified lead exposures. CMS issued guidance in 2016 to states on improving blood lead screening reporting, including correcting reporting errors and partnering with providers to ensure beneficiaries receive blood lead screenings. (See sidebar for examples of efforts states have taken to improve available data about blood lead screenings.) However, as of February 2019, the screening data remained incomplete, according to agency officials. CMS officials also told us they are currently in discussions with the Centers for Disease Control and Prevention about how to capture more complete information about Medicaid beneficiaries who are receiving blood lead screenings through programs funded by that agency. However, as of February 2019, CMS officials had not identified specific actions to gather this data. The lack of data is inconsistent with federal internal control standards, which specify that management should obtain relevant data from reliable sources based on identified information requirements, and use such data for effective monitoring. According to CMS, the results of recent pilot studies indicate that T-MSIS data can be used to replicate some information on the CMS-416 and Child Core Set. CMS officials said that the results also suggest that CMS may eventually be able to use T-MSIS data to produce the CMS- 416 and Child Core Set data, thus eliminating the need for states to report this information themselves separately. As previously noted, CMS intends for T-MSIS to both reduce the number of reports CMS requires states to submit and to provide more information to improve Medicaid oversight. CMS officials said that they were encouraged that the pilot studies to replicate portions of the CMS-416 and Child Core Set generally yielded positive results. For example, CMS was able to use T-MSIS to replicate the total number of Medicaid beneficiaries aged 20 and under eligible for EPSDT from the CMS-416 within 5 percent of state-reported values for eight of nine pilot states—which CMS officials viewed as a positive result. CMS officials noted some concerns with inaccurate state Medicaid eligibility data; for example, multiple dates of birth reported through T-MSIS for the same beneficiary. However, CMS officials believe the accuracy and completeness of T-MSIS data has improved since the pilot studies, which were conducted using data from 2015 and 2016. Regarding the Child Core Set, CMS was able to use T-MSIS to replicate some of the information, such as adolescent well-care visits, but not other information, such as emergency department visits. While CMS found generally positive results from the pilots, the agency has not developed a plan with time frames and interim milestones for when it will use state-reported T-MSIS data to produce the CMS-416 and Child Core Set data sets instead of states separately producing both T- MSIS data and the two data sets. In April 2019, CMS officials said that they were planning additional pilots beginning in fiscal year 2019 to replicate portions of the CMS-416 and the Child Core Set. However, CMS officials were unable to provide planned next steps, including time frames and interim milestones, for using T-MSIS data to replace the CMS-416 and Child Core Set. This is inconsistent with federal internal control standards related to using and communicating quality information to achieve objectives. Without a specific plan with time frames with interim milestones, CMS may miss opportunities to use T-MSIS data to streamline state reporting and better oversee states’ provision of EPSDT services. This limitation is similar to one we reported in December 2017 about the initial steps CMS had taken for using T-MSIS data. We found CMS was limited in using T-MSIS for its broader oversight efforts of state Medicaid programs, in part, due to the absence of an articulated plan and time frames. Under EPSDT, millions of Medicaid’s youngest beneficiaries received well-child screenings and dental services in fiscal year 2017; however, nearly as many of them did not. Further, existing data on blood lead screenings are incomplete and inaccurate, leaving CMS unaware of beneficiaries with unidentified lead exposures that can cause developmental delays. The EPSDT data collected—whether via the CMS- 416, Child Core Set, or T-MSIS—have the potential to improve CMS oversight of beneficiaries’ receipt of necessary services and screenings. However, CMS has not taken sufficient steps to help ensure the appropriateness of its state data collection, evaluations, and assistance; and its plans for new reporting, including time frames and interim milestones, are lacking. We are making the following six recommendations to CMS: The Administrator of CMS should work with states and relevant federal agencies to collect accurate and complete data on blood lead screening for Medicaid beneficiaries in order to ensure that CMS is able to monitor state compliance with its blood lead screening policy, and assist states with planning improvements to address states’ compliance as needed. (Recommendation 1) The Administrator of CMS should regularly assess the appropriateness of performance measures and targets for the EPSDT benefit, and take any necessary actions to ensure their relevance and use, including adding, changing, or removing measures, or targets, and regularly communicating performance measures and targets to states. (Recommendation 2) The Administrator of CMS should conduct regular evaluations of state performance by comparing states’ performance measurement data with CMS’s EPSDT targets to identify gaps in states’ performance and areas for improvement. (Recommendation 3) The Administrator of CMS should assist states with planning needed improvements, including providing focused assistance, to resolve gaps in states’ performance in meeting CMS’s EPSDT targets. (Recommendation 4) The Administrator of CMS should develop a plan with time frames and interim milestones for using T-MSIS data to generate the necessary data from the CMS-416 to improve EPSDT oversight and streamline state reporting. (Recommendation 5) The Administrator of CMS should develop a plan with time frames and interim milestones for using T-MSIS data to generate the necessary data from the Child Core Set to improve EPSDT oversight and streamline state reporting. (Recommendation 6) We provided a draft of this report to HHS for comment, and its comments are reprinted in appendix IV. HHS also provided us with technical comments, which we incorporated in the report as appropriate. Overall, HHS concurred with three recommendations and did not occur with three recommendations. HHS concurred with our first recommendation that CMS should work with states and relevant federal agencies to collect accurate and complete data on blood lead screening for Medicaid beneficiaries and assist states with planning improvements to resolve gaps in states’ performance as needed. However, HHS stated that it would not be possible to obtain complete data on blood lead screenings, because some screenings are not paid for by Medicaid. In our report, we noted some state and CMS efforts to improve available data on blood lead screenings. We continue to believe CMS needs to take additional actions to collect accurate and complete data to oversee whether eligible EPSDT beneficiaries are receiving blood lead screenings in accordance with CMS policy. HHS did not concur with our second recommendation, which stated that CMS should regularly assess the appropriateness of performance measures and targets for the EPSDT benefit, and take any necessary actions to ensure their relevance and use. HHS noted that it assesses the appropriateness of Child Core Set measures annually and may update existing measures based on that assessment, including measures on the CMS-416. We acknowledge CMS’s actions to assess the appropriateness of Child Core Set measures annually and update those measures as appropriate, and we found these actions generally consistent with federal internal control standards regarding information and communication. However, CMS has not taken action, as needed, related to any assessments of the CMS-416 performance measures, even though officials acknowledge limitations in these measures, such as the participant and screening ratios. HHS also stated that it may set targets in key areas as appropriate, and has done so as part of the Oral Health Initiative, but that HHS does not believe it would be productive at this time to set targets for every measure. We are encouraged that HHS agreed that it may set targets in key areas as appropriate. This is consistent with our recommendation for CMS to regularly assess the appropriateness of its targets. Our recommendation does not assume that targets should be set for every measure—rather, that CMS needs to regularly assess the appropriateness of performance measures and targets for the EPSDT benefit and communicate them to states. HHS did not concur with our third recommendation, which stated that CMS should conduct regular evaluations of state performance by comparing states’ performance measurement data with CMS’s EPSDT targets. HHS stated that it offers a wide range of technical assistance on quality improvement to help states address performance goals. HHS commented that it believes this is the most effective method of helping states identify and address areas for potential improvement. We acknowledge that CMS has provided states with technical assistance and individual state snapshots of selected Child Core Set measures over time. However, regular evaluations of states’ performance against appropriate EPSDT targets are necessary to help identify gaps in states’ performance and areas for improvement. HHS noted that states recently received snapshots about their performance on publicly reported Child Core Set measures for the past 5 years, through fiscal year 2017. According to HHS, the snapshots include information about a state’s performance on each measure relative to other states’ performance and highlights significant changes in a state’s performance for each measure. However, these snapshots include descriptions of all states’ performance—using medians, and top and bottom quartiles—which are subject to change over time. Moreover, because the median is the midpoint of all states’ performance, this target ensures that half of states will not meet it, regardless of their individual performance. A fixed target—or targeted improvement goal, such as the one developed as part of the Oral Health Initiative—would provide states with the opportunity to measure performance over prior years’ results, which is a more meaningful measure that all states can strive to achieve. HHS did not concur with our fourth recommendation, which stated that that CMS should assist states with planning needed improvements to resolve gaps in states’ performance in meeting EPSDT targets. HHS stated that it has developed national and state-specific improvement goals for children enrolled in Medicaid with respect to receipt of at least one preventive dental service and provided targeted technical assistance to the lowest performing states. In this report, we noted states’ progress in meeting targets once CMS developed a performance measurement target for preventive dental services, including actions to improve state performance. Developing additional targets on performance measures critical to beneficiaries’ health and well-being could help improve oversight of EPSDT. HHS also described other examples of targeted technical assistance to remedy gaps in states’ performance, which included working with states on improving their performance on certain Child Core Set measures and improving access to EPSDT services by better leveraging schools as settings for care. Such technical assistance could be valuable for CMS to provide to states after identifying gaps in states’ performance relative to EPSDT targets. Doing so would allow CMS to share additional strategies to help states plan and implement needed improvements. HHS concurred with our fifth and sixth recommendations that CMS should develop a plan with time frames and interim milestones for using T-MSIS data to generate the necessary data from the CMS-416 and Child Core Set to improve EPSDT oversight and streamline state reporting. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services, the Administrator of the Centers for Medicare & Medicaid Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Carolyn L. Yocom at (202) 512-7114 or yocomc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Major contributors to this report are listed in appendix V. Selected states used several types of practices to promote and facilitate the delivery of Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) services, according to Medicaid officials in the 16 selected states we interviewed and profiles of these states created by the American Academy of Pediatrics. The practices selected states used included outreach and education, financial incentives, collaboration in EPSDT administration, and EPSDT service delivery initiatives, as shown in figure 4. States annually report information about Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) services to the Centers for Medicare & Medicaid Services (CMS), through the Form CMS-416 and the Child Core Set. The CMS-416 provides basic information about EPSDT for Medicaid beneficiaries aged 20 and under, such as the participant ratio and number of beneficiaries receiving a preventive dental service. The Child Core Set provides CMS with information about the quality of health care provided to Medicaid beneficiaries and individuals aged 18 and under who are covered under the Children’s Health Insurance Program. In fiscal year 2024, annual reporting of the Child Core Set will become mandatory. As of 2019, the Child Core Set included performance measures related to the provision of EPSDT services, such as well-child visits in the first 15 months of life. Because Child Core Set reporting is currently voluntary, states vary in the number of performance measures they choose to report. In fiscal year 2017, for example, 50 states and the District of Columbia voluntarily reported on at least one of the 27 Child Core Set performance measures, with states reporting a median of 18 Child Core Set performance measures. Some information is only reported on the CMS-416 or Child Core Set, while other information— well-child visits, preventive dental services, and dental sealants—is reported on both CMS-416 and Child Core Set. (See table 5 for information reported on the CMS-416, the Child Core Set, or both.) Tables 6 through 8 present annual state-reported data from the Centers for Medicare & Medicaid Services’ (CMS) Form CMS-416 on the provision of selected Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) services by state and nationally. Well-child screenings are presented from fiscal year 2010, the year in which the current reporting template was implemented, through fiscal year 2017, the most recent year for which data were available at the time of our review. Preventive dental services data are presented from fiscal year 2011, the baseline year for measuring states’ progress toward CMS’s Oral Health Initiative targets, through fiscal year 2017, the most recent year for which data are available. Carolyn L. Yocom at (202) 512-7114 or yocomc@gao.gov. In addition to the contact named above, Karen Doran (Assistant Director), Peter Mangano (Analyst-in-Charge), Matthew Green, Erika Huber, Drew Long, Jennifer Rudisill, and Kelly Turner made key contributions to this report. Also contributing were Muriel Brown, Giselle Hicks, Erika Lessien, and Madeline Ross.", "summary": "The EPSDT benefit is key to ensuring that Medicaid beneficiaries aged 20 and under receive periodic screening services, such as well-child screenings, and diagnostic and treatment services, such as physical therapy and eyeglasses, to correct or ameliorate conditions discovered during a screening. GAO was asked to examine the extent to which Medicaid beneficiaries aged 20 and under receive health care services under the EPSDT benefit. Among other things, GAO examined (1) what is known about the provision of EPSDT services based on CMS-required annual state reporting, and (2) CMS oversight of the EPSDT benefit. To do this, GAO analyzed annual state reporting data from fiscal years 2010 through 2017, the most current year data were available; CMS documentation; and federal internal control standards. GAO also interviewed CMS officials and Medicaid officials from 16 states selected, in part, on the variation in number of beneficiaries and geographic diversity. Approximately half of all Medicaid beneficiaries aged 20 and under received screenings and services recommended under the Early and Periodic Screening, Diagnostic, and Treatment (EPSDT) benefit in fiscal year 2017, but nearly as many did not. For example, GAO's analysis of state-reported data found that about 59 percent of all beneficiaries (20.2 million) who should have received at least one recommended well-child screening received one. About 48 percent of beneficiaries aged 1 to 20 (18.3 million) received a preventive dental service in fiscal year 2017. Older beneficiaries tended to have lower rates of screening. Number of Medicaid Beneficiaries Receiving and Not Receiving Well-Child Screenings in Fiscal Year 2017, by Age Group The Centers for Medicare & Medicaid Services (CMS), the agency that oversees Medicaid, including EPSDT, has taken steps to improve the quality of information that states report about the provision of EPSDT services. CMS has also set some EPSDT performance measure targets for states; yet, the agency has not taken other steps to oversee the EPSDT benefit, such as collecting the data necessary to evaluate whether states are complying with CMS's policy for beneficiaries to receive a blood lead screening; taking action, as needed, based on assessments of the appropriateness of some performance measures, such as well-child screening measures; and using state-reported information to regularly evaluate states against CMS's EPSDT targets, or assisting states in planning improvements to meet the targets. Absent these steps, CMS's oversight is limited and beneficiaries may not be receiving appropriate EPSDT services when they need them. GAO is making six recommendations to CMS regarding its oversight of the EPSDT benefit, including collecting appropriate blood lead screening data; taking action, if needed, after assessing the appropriateness of performance measures and targets for EPSDT; and evaluating states' performance in meeting CMS's EPSDT targets. CMS agreed with three recommendations, but disagreed with three others regarding performance measures and targets. GAO maintains that these recommendations are valid, as discussed in this report.", "document_type": "gao"}
{"report": "To be entitled to VR&E services and related benefits, veterans generally must (1) have at least a 20 percent service-connected disability rating from VA and (2) be in need of rehabilitation because of an employment handicap. Entitled veterans may generally receive up to 48 months of vocational rehabilitation services and up to an additional 18 months of employment services, which include counseling, and placement and postplacement services. If a veteran is entitled to receive VR&E services and found to be employable, a counselor is to work with the veteran to identify a suitable employment goal, and to incorporate that goal and the needed services and benefits to achieve it into a vocational rehabilitation and employment plan (hereafter “employment plan”). To develop an employment plan, the counselor and veteran review labor market information for jobs within the veteran’s identified abilities, aptitudes, and interests that will not aggravate his or her service-connected disability or disabilities. After assessing obstacles to employment, they agree on a written employment plan that describes the employment goal and the services needed to achieve it. Common services provided by VR&E are funding for higher education, career counseling, and short-term employment services like job search assistance. Counselors have the authority to approve a wide variety of educational programs and may approve employment plans that have an annual cost of up to $25,000. Within VA’s Veterans Benefits Administration (VBA), the VR&E central office is responsible for overseeing the VR&E program, including training staff and monitoring their work to ensure high performance and consistency. Among other elements, VR&E’s quality assurance efforts entail reviewing a subset of case files on a monthly basis to ensure that the entitlement decisions, development of plans, and delivery of services are performed and documented in accordance with VA regulations, VR&E’s operations manual, and other directives. VR&E services are provided by field staff at 56 regional offices and about 300 satellite locations. The satellite locations include college campuses to help veterans successfully complete their training and find employment, as well as military sites to help servicemembers with disabilities as they begin their transition to veteran status and the civilian workplace. VR&E field positions include (1) VR&E officers who manage the program and its staff in each region; (2) vocational rehabilitation counselors who work directly with veterans to assess their entitlement, develop their employment plans, and manage their progress; and (3) staff to support the administration of the program. As of June 2019, 1,394 field staff members were administering the VR&E program, of which nearly 75 percent (1,026) were counselors. From September 2013 to June 2019, VR&E’s total caseload peaked in fiscal year 2016 with almost 135,000 participants (see fig. 1). Over the same period, the number of counselors changed little until 2019. In 2019, the number increased after VA hired an additional 88 counselors in response to a provision in an appropriations law suggesting that the agency aim to serve 125 veterans or fewer per full-time equivalent counselor. The increase in staffing helped reduce the average caseload of 130-141 cases per counselor during fiscal years 2013 through 2016 to 113 cases in June 2019 (see fig. 2). VR&E counselors consider a set of common factors, including the veteran’s disability or disabilities, interests, and local labor market conditions, when developing and approving veterans’ employment plans. Program regulations require an assessment of some of these factors when the veteran is initially evaluated. VR&E quality review data from fiscal years 2016 through 2018 suggest that counselors generally documented certain plan considerations during the evaluation. For example, in 98 percent of the 1,080 cases VA reviewed for accuracy in fiscal year 2018, counselors documented the veteran’s service needs based on their functional limitations. In 95 percent of cases, counselors documented that they assessed the veteran’s abilities, aptitudes, and interests. Lastly, in nearly 99 percent of cases, counselors documented that the veteran was involved in vocational exploration activities such as career searches and labor market research. During our more focused review of how counselors developed plans for a non-generalizable sample of 34 VR&E case files, we found that counselors generally documented a set of common factors. Consistent with program guidance stipulating that counselors are to consider a veteran’s service needs, abilities, aptitudes, and interests, we identified common consideration factors including one’s functional limitation from disability, prior education, aptitude results, and career interests. Table 1 presents these factors and the number of files in which the factors were documented. Our case file review found that 30 of the 34 counselors also documented the estimated cost of VR&E employment plans. According to testimony from a veteran service organization, many VR&E participants are dissuaded by their counselor from pursuing education at a top tier university because of cost. VA’s VR&E operations manual states that if more than one local training or educational facility will meet a veteran’s needs, counselors must justify their decision to select a school that is more expensive than the least costly one. Counselors are not required to document all of the educational facilities that would serve a veteran’s needs; therefore, we could not determine the extent to which counselors chose the lowest cost facility. Counselors we interviewed in each of the three regional offices we visited said that while mindful of cost, they strive to develop employment plans that best meet the needs of the veteran. For example, counselors at one regional office described a situation in which a higher priced school was chosen because the school offered smaller class sizes that better suited the veteran’s particular mental health conditions. Of the 34 files we reviewed, the annual plan cost exceeded $25,000 in 3 cases. Counselors we interviewed said that they considered the veteran’s career interests but weighed these interests against other factors, such as the veteran’s functional limitations and information about the local labor market. All 34 plans we reviewed aligned with the veteran’s stated career goals, though in some cases the veteran’s goals evolved after talking with the counselor about alternative occupations. In a few instances among these cases, the final plan’s career goal was notably different from the initial goal that the veteran had stated on the program intake form. Table 2 presents examples of how a plan can evolve as a result of career exploration activities and conversations between the veteran and their counselor. Counselors we interviewed described how veterans’ employment plans are individually designed to suit a veteran’s needs and, as a result, may differ from one another even when veterans have similar goals, characteristics, and circumstances. In some instances two veterans may appear to be similar, but may actually differ in some critical respect that results in appropriate variation across plans. A common difference among veterans is the geographical location where they are seeking employment. Counselors described how a veteran may be encouraged to explore an occupation with many job opportunities within a specific region, while a veteran with similar characteristics and interests living in a different area may be dissuaded from pursuing the same occupation for a lack of job opportunities in the area for that occupation. Local labor markets may also drive the need for a certain type of educational credential. For example, counselors said that some veterans will be competitive in certain labor markets with a bachelor’s degree, while others living in a different region with a more educated population may need a master’s degree. Likewise, they said that certain occupations, such as certified public accountants or school teachers, may require different forms of credentialing in different states. Other characteristics of individual veterans may also cause counselors to develop different plans for veterans who appear to have similar circumstances. One counselor we interviewed described a scenario in which one veteran who received a high score on an aptitude test for reading comprehension skills might obtain a certain employment plan while another veteran who received a much lower score would be steered toward a different plan. If the veterans were to compare their final plans, but were unaware of the differences in their aptitude test scores, they could perceive inconsistent treatment. Counselors also described how conversations they have with veterans as they work to develop employment plans can reveal other character traits, such as interpersonal skills, which can lead them to suggest different plans to two otherwise similar veterans. The counselors said that such conversations play an important role towards the development of successful plans. However, counselors we interviewed in each of the three regional offices we visited acknowledged that unintended variation likely occurs across plans developed for similarly-situated veterans. They explained that the reasons for such potential inconsistency can include (1) the prominent role professional judgment plays in the program and the potential for unintended bias, (2) counselors’ different VR&E experience levels, and (3) variations in regional offices’ policies. Judgment and bias. The counseling role is inherently subjective and requires counselors to use their professional judgment in each case. The VR&E operations manual describes counselors’ responsibilities in broad terms, stating that counselors are to guide and assist the veteran in making an informed decision on an appropriate plan based on the veteran’s abilities, aptitudes, and interests. According to counselors we interviewed, professional judgment enables them to develop a plan that is best suited for the veteran’s unique needs, although it also introduces the potential for personal bias and inconsistent plans for veterans with similar circumstances. For example, a counselor we interviewed cited a case in which he saw the need to develop a plan that allowed for a school that was closer to a veteran’s home over other, less costly options because of his sensitivity to the veteran’s childcare responsibilities. Another counselor in the same office may not have seen the need for that accommodation. Further, counselors we interviewed said that some of their colleagues may be more comfortable suggesting that a veteran reconsider his or her career goal given circumstances such as the veteran’s disabling conditions or the local labor market. They explained that while some counselors would be hesitant to make the veteran unhappy, and possibly angry, other counselors would be more inclined to work through the conflict. Counselors said that they try to mitigate inconsistency by asking their fellow counselors to weigh in on these sorts of judgments, either informally, or at periodic information-sharing meetings. Counselor experience. Although all counselors have at least a master’s degree in rehabilitation counseling or a related field, differences in counselors’ levels of VR&E experience may affect their approach to plan development. Counselors at two regional offices noted that the focus of the VR&E program has oscillated between education and employment, with employment being the current primary focus. They said, as a result, a counselor’s general approach to plan development could be influenced by the prevailing focus that existed at the time he or she was hired. Counselors also said that, in general, counselors with more experience will tend to approach plan development differently than a less seasoned counselor because they will apply lessons learned from serving many other veterans. For example, one counselor said that, based on years of prior experience and observation, he has developed a better understanding of which local educational programs offer veterans the best chance for success and which do not. He said while he is able to apply his institutional knowledge and experience to do what is best for veterans, a less experienced counselor may not have the same level of knowledge, which could lead to inconsistent plans for veterans with similar circumstances. According to counselors we interviewed, because of the recent hiring of new counselors to meet caseload targets, differences in VR&E experience among counselors may be more pronounced at this time. Regional office variation. Differences in administrative policies specific to individual regional offices may also contribute to inconsistent plan development. For example, according to program officials, to ensure the soundness of employment plans in the local labor market, some regional offices require management to approve plans involving a master’s degree, while others do not. Counselors in one region told us that requiring management approval might dissuade a counselor from developing a plan focused on a master’s degree because of the time the extra step would require. They acknowledged that this sort of approval policy could cause inconsistency across counselors’ plans and also cause a discrepancy in the number of master’s degree programs being approved at one regional office versus another. In general, the large number of variables involved in the development of employment plans may complicate the ability to determine the extent to which differences among counselors lead to inconsistent plans among veterans. Counselors we interviewed said that given the subjective nature of the program, such inconsistency is likely. However, counselors cautioned against making the plan development process overly structured and formulaic. In their view, a more restrictive approach would eliminate the flexibility that they need to generate plans that suit each veteran’s unique needs. VA trains counselors on developing sound and complete employment plans for veterans. New counselors receive a series of training courses that are developed and deployed through VR&E’s central office, and then receive additional courses and mentorship that are delivered through the regional offices. Course topics for new counselors include understanding vocational impairments, developing a rehabilitation plan, and documenting a narrative of the plan. The formal training emphasizes that plans should be individualized to accommodate the veteran’s rehabilitation needs, abilities, aptitudes, and interests. Collectively, these trainings take up to 80 hours. As of 2019, experienced counselors—those on the job for at least a year—take up to 20 hours of refresher training each year determined according to how they score on an annual assessment. The assessment evaluates counselors’ technical competencies such as knowledge of relevant regulations, vocational assessment and evaluation, and case management. If a counselor scores low on a particular topic, related courses are identified for the counselor to complete. In designing training for counselors, VA followed principles for strategically developing training that are consistent with a related guide for federal managers. For instance, VA obtained and considered input from multiple sources—including field advisory committees, quality assurance reviewers, and internal site visit auditors—to identify needs for counselor training. For example, questions and input from the field about a policy clarification led to a training about veterans’ entitlement to VR&E services. In addition, VA built flexibility into its training curricula for counselors so they could receive training on emerging topics such as implementing new policies throughout the year as needed. VA also has evaluated its training efforts in multiple ways. For example, it has evaluated training courses by surveying counselors to get immediate feedback and by checking with attendees and their supervisors to gauge improvements in skills and knowledge. VA monitors employment plans to ensure that they are complete, but does not check for consistency among counselors for veterans with similar circumstances. Quality reviews occur nationally as well as locally at each regional office. The purpose of the national reviews is to monitor the quality of regional offices’ work such as plan development, whereas the purpose of the local reviews is to help evaluate the performance of individual counselors. Nationally, a centralized quality assurance team monitors the completeness of regional offices’ VR&E entitlement decisions, employment plans, and service delivery by reviewing a randomly-selected subset of case files from each regional office on a monthly basis. Among other criteria, reviewers check whether a veteran’s plan identified goals and objectives, included an employment focus, and incorporated the veteran’s need for various services. Locally, VR&E officers or their designees are to review plans using the same criteria. Officers are supposed to review at least three cases per counselor per quality category (e.g., accuracy of evaluation, planning and rehabilitation services) per quarter. Reviewers do not check for consistency among counselors for similarly-situated veterans, at either the national or local levels. VR&E officials we interviewed identified challenges to completing and monitoring local reviews, but VA is addressing these challenges. According to the VR&E officer in each of the three regional offices we visited, it is difficult to complete local reviews given system limitations and their other job responsibilities such as implementing case management initiatives. They said that it is likely that some officers are not completing the required reviews while others are conducting them with varying degrees of thoroughness. Historically, VA has not identified the specific cases VR&E officers are supposed to review locally. Consequently, VA could not determine if VR&E officers conducted the requisite number of reviews or whether officers were selecting cases for quality review uniformly and fairly. In June 2019, during the course of our review, VA began a pilot in five regional offices to centrally and systematically identify the cases officers are to review to gauge individual counselors’ performance. The new process and system are intended to help officers conduct and track local reviews as well as to help VA monitor the completion of local reviews. VA plans to expand this process to all regional offices in fiscal year 2020. Although VA trains counselors to develop complete employment plans and reviews the completeness of some plans, it does not monitor the consistency of plans among different counselors. The code of professional ethics for rehabilitation counselors calls for counselors to be fair in the treatment of all clients and to provide appropriate services to all. In addition, one of the objectives of VR&E’s central office is to provide training and guidance to ensure high performance and consistency among field staff. Several veteran service organizations have testified at congressional hearings that VR&E is marked by inconsistent treatment of similarly-situated veterans. For example, one testimony cited veterans who allegedly received different plan approvals, such as access to graduate level education, merely on the basis of their counselor. Unlike for VA staff members who work on disability claim decisions, VA does not compare the output of VR&E counselors by, for example, analyzing responses to identical hypothetical cases for training or monitoring purposes. As a result, in addition to missing a training opportunity for counselors about employment plan development, VA does not know the degree to which inconsistency among counselors occurs. For example, the agency does not know the extent to which counselors would agree to a particular veteran’s pursuit of a master’s degree through VR&E. Moreover, VA cannot respond in an informed way—and take mitigating steps if warranted—to criticisms of subjectivity in the program. VR&E officials explained that the agency has not yet conducted such a comparative analysis because of other priorities, but agreed that it could do so particularly through its training efforts. VA uses several training and monitoring practices to help ensure that VR&E counselors develop employment plans that help veterans with disabilities obtain and sustain employment. In approving these plans, VR&E counselors use their judgment and discretion fostered in part by their formal education and professional experience in vocational rehabilitation. While our review of a non-generalizable sample of 34 cases found that counselors generally considered common factors in developing employment plans, counselors we interviewed nevertheless acknowledged that counselors may apply the factors differently because of their varying backgrounds and experience levels. The variability of counselors’ experiences and veterans’ circumstances may make it difficult to determine the full extent of any inconsistency. However, taking steps to examine the prevalence and type of any inconsistency among counselors who, for example, consider the same hypothetical case, would better position VA to mitigate any unfair differences in plans for veterans with similar circumstances. An understanding of how effectively and consistently counselors assist veterans will be even more important in the coming years as VA fully integrates the new counselors hired to decrease the average caseload. The Secretary of VA should ensure that the Director of VR&E assesses the consistency of VR&E plans among counselors and takes mitigating steps if results warrant. For example, as part of its training efforts, VA could have counselors respond to identical hypothetical veteran cases and, if unfair inconsistencies in plans result, the agency could enhance training on plan development. (Recommendation 1) We provided a draft of this report to VA for comment, and its written comments are reproduced as appendix I in this report. VA concurred with our recommendation and said that VBA will develop a consistency study of VR&E plan development. It emphasized that no two veterans are the same. It also provided technical comments which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report will be available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the individual named above, Mark Glickman (Assistant Director), Joel Green (Analyst in Charge), and David Perkins made significant contribution to the report. In addition, Jennifer Cook, Holly Dye, Alex Galuten, Monica Savoy, Mimi Nguyen, Almeta Spencer, Jeff Tessin, Rosemary Torres Lerma, and Sonya Vartivarian made key contributions.", "summary": "VA's VR&E program helps veterans with service-connected disabilities obtain and maintain suitable employment. VR&E participants work with vocational counselors to develop career goals and employment plans. However, some veteran service organizations have questioned the consistency with which participants are treated by counselors in developing these plans. GAO was asked to review how VR&E vocational counselors work with participants to select employment plans, and VA's efforts to ensure high quality and consistency. This report examines (1) the factors that vocational counselors considered when developing VR&E participants' plans and how consistently they applied those factors, and (2) the extent to which VA trains and monitors vocational counselors to ensure a consistent, high-quality approach to helping veterans develop plans. GAO analyzed VR&E quality review data from fiscal years 2016 through 2018; reviewed a random, non-generalizable sample of 34 VR&E case files from 2019; reviewed relevant federal laws, regulations, and VA policy; and interviewed VR&E counselors and other program officials. The Department of Veterans Affairs' (VA) Vocational Rehabilitation and Employment (VR&E) counselors in GAO's review generally considered a set of common factors when developing plans to help veterans with disabilities obtain employment, but counselors explained that inconsistent application of those factors likely occurs. These factors included the veteran's disability, his or her interests, and local labor market conditions. The 34 VR&E plans GAO reviewed showed that counselors' generally considered and documented these factors (see table). Counselors in each of the three regional offices GAO visited said that plans are individualized to suit the veteran's needs and as a result will differ because each veteran's case is unique. Nonetheless, these counselors acknowledged that some veterans with similar circumstances likely receive different types of plans given differences in counselor judgment and experience. VA trains and monitors counselors to develop complete VR&E plans but does not assess the consistency of plans across counselors for veterans with similar circumstances. VA's training for VR&E counselors emphasizes that plans should accommodate each veteran's individual needs, abilities, aptitudes, and interests. In designing training for counselors, VA followed principles identified by GAO for strategically developing training. VA monitors the completeness of VR&E plans through national and regional quality reviews that check, among other elements, whether plans have an employment focus and include needed services. However, these quality reviews do not assess the consistency of plans developed by different counselors. VR&E officials explained that the agency has not yet conducted such an analysis because of other priorities, but agreed that it could do so. One of the objectives of VR&E's central office is to provide training and guidance to help ensure consistency among field staff. Assessing consistency across counselors would better position VA to mitigate any unfair differences in plans for similarly-situated veterans. GAO recommends that VA assess the consistency of VR&E plans among counselors by, for example, comparing counselors' responses to identical hypothetical cases, and take mitigating steps if warranted. VA concurred with the recommendation and planned to develop a consistency study.", "document_type": "gao"}
{"report": "Enacted in December 2015, ESSA’s amendments to Title I, Part A (Title I) of the ESEA included a number of requirements for SEAs and school districts to ensure the educational stability of children in foster care. For the purposes of this report, we refer to these requirements collectively as the “ESSA educational stability provisions.” Specifically, SEAs are required to describe in their Title I state plans the steps they will take to ensure collaboration with the state child welfare agency to ensure the educational stability of children in foster care, including assurances that: Such children enroll or remain in their school of origin, unless a determination is made that it is not in the child’s best interest to attend the school of origin. This decision shall be based on all factors relating to the child’s best interest, including consideration of the appropriateness of the current educational setting and the proximity to the school in which the child is enrolled at the time of placement. When a determination is made that it is not in a child’s best interest to remain in the school of origin, the child is immediately enrolled in a new school, even if the child is unable to produce records normally required for enrollment. The enrolling school shall immediately contact the school last attended by the child to obtain relevant academic and other records. The SEA will designate an employee to serve as a point of contact for child welfare agencies and to oversee implementation of the above provisions. LEAs are required to provide in their Title I LEA plans assurances that they will collaborate with the state or local child welfare agency to: designate a point of contact, if the corresponding child welfare agency notifies the LEA in writing that the child welfare agency has designated a point of contact for the LEA; and develop and implement clear written procedures governing how transportation to maintain children in foster care in their school of origin when in their best interest will be provided, arranged, and funded for the duration of the time in foster care. The ESSA requirements described above were generally required to be implemented by December 10, 2016. In addition, SEAs and school districts are required to publicly report on the academic achievement and graduation rates of youth in foster care on their annual report cards. States and localities also have some flexibility in implementing the ESSA educational stability provisions. For example, ESSA does not prescribe a specific process for determining whether it is in a child’s best interest to remain in their school of origin. In making this determination, state and local agencies have flexibility in determining which factors should be considered when evaluating the appropriateness of a child’s current educational setting, as well as any additional factors that pertain to a child’s best interest. Similarly, school districts and child welfare agencies generally determine the transportation procedures to use, provided they meet the minimum statutory requirements. In addition, SEAs may choose various approaches to help LEAs implement the ESSA educational stability provisions. For example, SEAs may decide to independently, or with their state child welfare agency, issue policies or guidance, disseminate question and answer documents, or hold informational meetings and webinars. Education and HHS collaborated to provide states with joint non- regulatory guidance specific to the ESSA educational stability provisions. In addition to this written guidance, Education provides technical assistance to states, such as through the State Support Network, one of its technical assistance providers. Each state also has a point of contact at Education for questions, according to Education officials. Education’s Office of School Support and Accountability oversees state implementation of Title I, Part A of the ESEA, including the amendments made by ESSA. Education’s oversight of SEAs includes reviewing state Title I plans that describe how states will follow a variety of federal requirements outlined in Title I, and periodic reviews of how each state is implementing Title I. These reviews occur every few years. HHS’s Children’s Bureau oversees state child welfare agencies’ implementation of Title IV-E, including the provisions in the Fostering Connections Act, and also provides related technical assistance. State and local officials reported facing several challenges related to implementing the ESSA educational stability provisions. Specifically, officials reported challenges with (1) turnover among local child welfare and educational agency staff, (2) obtaining school district input during the process for determining whether it is in a youth’s best interest to remain in their school of origin (referred to as best interest determinations), (3) providing and funding transportation, (4) ensuring accurate identification of youth in foster care, and (5) monitoring how school districts implement these provisions. In addition, while we did not ask on our survey about the requirement to immediately enroll youth in a new school if it is determined that remaining in the school of origin is not in their best interest, or about the requirement for the enrolling school to immediately contact the last school attended to obtain relevant records, education and child welfare officials we interviewed said they experienced challenges with immediate enrollment and records transfer for special populations of youth. Turnover among Local Child Welfare and Educational Agency Staff Turnover of local educational and child welfare agency officials was reported as a significant challenge that affects how many states and localities implement the ESSA educational stability provisions, according to our survey and interviews. Specifically, in our survey, 43 of 51 SEAs reported turnover of local child welfare agency points of contact as at least somewhat challenging. A similar number of respondents (39) reported facing challenges with turnover of school district points of contact (see fig. 1). During our discussion group, state child welfare agency officials highlighted turnover of local child welfare agency and school district staff as one of the most significant challenges their states face in ensuring educational stability for youth in foster care. In addition to turnover itself being a challenge, several other challenges reported by SEAs are related to staff turnover, according to officials we spoke with from four state and local educational and child welfare agencies. Specifically: Thirty-two SEA survey respondents identified maintaining an accurate list of school district foster care points of contact for their state as challenging, and officials from four state and local educational and child welfare agencies we spoke with stated turnover makes it difficult to keep these lists updated. One SEA point of contact said that when she sends emails to school district points of contact, she receives numerous responses each time from school district staff saying they are no longer the point of contact. Officials we interviewed at one school district noted that they tried to identify a new point of contact at another school district, but the list on the state website had not been updated. Thirty-eight SEAs reported on our survey that ensuring that school district points of contact are aware of their responsibilities is a challenge. Eight state and local educational and child welfare agency officials we interviewed echoed this observation and cited staff turnover as leading to a lack of awareness of responsibilities or protocols related to the ESSA educational stability provisions. Local staff being unaware of their responsibilities under ESSA can lead to conflicts, according to officials from two state and three local agencies we interviewed, and resolving conflicts between school districts and local child welfare agencies was a challenge reported by three-quarters (38) of SEA survey respondents. For example, officials at one local child welfare agency said they encountered school district officials who did not believe a youth in foster care could attend their current school, since their foster parent lived outside the school district. To resolve the conflict, the school district point of contact discussed the provisions with the school officials. To alleviate challenges related to turnover, SEA points of contact we surveyed and interviewed explained that they regularly provide information to local school district and child welfare agency officials on the ESSA educational stability provisions. To inform and remind local officials about the provisions, a few of these officials said they send emails to school district points of contact or provide training on the provisions at orientation for new staff at child welfare agencies. In all three states we visited, the SEAs and/or state child welfare agencies said they held joint presentations for both school districts and local child welfare agencies, and SEA officials in Georgia said they are considering holding regional collaborative meetings every four to six months. In addition, most SEAs reported on our survey that they work with their state child welfare agencies to provide or develop assistance, guidance, and sample documents or templates to facilitate implementation of the ESSA educational stability provisions at the local level. (See tables 1 and 2 in appendix II for more information on this assistance.) School District Input for Best Interest Determinations On our survey, 34 of 50 SEAs reported that ensuring school districts participate in best interest determinations is a challenge (see fig. 2). Two of five state child welfare agency officials in our discussion group also described challenges related to the lack of collaboration between child welfare agencies and schools on best interest determinations. While ESSA does not prescribe who should be involved in the best interest determination, the joint federal guidance encourages state and local child welfare and educational agencies, including school districts, to develop a process that involves all relevant parties. School district involvement, however, depends on child welfare agencies informing them when a child enters foster care or changes homes. Officials we interviewed at several child welfare agencies indicated they may not include school districts or schools in these determinations due to time constraints. Child welfare officials explained that removing a child from a home and placing them into foster care is a chaotic time and many steps need to be taken to quickly provide the child with a safe environment. During this time, caseworkers may lack the capacity to collaborate with school districts or schools. Child welfare agency officials at two local offices we visited explained that they prioritize a child’s health and safety when placing a child in a new foster home and that they place a greater focus on these issues than on educational stability. Some child welfare agency officials we spoke with said they do not always need school district input to make a best interest determination. For example, officials at two local child welfare agencies said that in some cases, the commute to a child’s current school may be so long that remaining there is clearly not in a child’s best interest. Officials from one state and two local child welfare agencies told us they assume it is in the best interest of the child to remain at their current school. Officials from the state child welfare agency said they do not believe they need to consult with school districts to make that decision. Officials at another local child welfare agency said it would not be helpful to collaborate with school districts on the best interest determination, since the child welfare officials do not believe where the child attends school is the highest priority. However, youth we spoke with in our discussion groups told us that changing schools can create several challenges (see text box). Officials from other state and local child welfare agencies told us they recognize the need to involve school districts and are taking steps to try to include them in best interest determinations. For example, one state child welfare agency we visited includes a line for the school district point of contact’s signature on the state’s best interest determination form; however, we heard from officials at a local child welfare agency that the school district point of contact may not be involved in making the best interest determination, and the form may not be consistently used. Officials at a local child welfare agency told us that they hold best interest determination meetings with the school district by phone because these meetings are faster to schedule than in-person meetings. Rather than speaking with school district staff, officials from four local child welfare agencies said they try to contact school staff that may be close to a child, such as a counselor or teacher, but officials from three of these agencies said they may not do so in every case. Thirty-seven of 50 SEAs reported on our survey that assisting school districts with identifying or arranging transportation is at least somewhat challenging (see fig. 3). To help school districts and local child welfare agencies identify transportation options, SEAs in two states we visited provide guidance or other documents to these agencies that describe potential transportation options. School district and local child welfare agency officials we spoke with reported using different approaches to transport youth, including having foster parents, school district or child welfare staff, or the youth drive to school; rerouting buses; hiring a taxi or other private transportation service; or using public transportation. Sometimes they reported combining these methods to transport youth to their current school. However, eight school district and local child welfare officials noted difficulties with their options, including limited options in rural areas and lack of appropriate transportation for younger youth and those with behavioral issues. For example, an Arizona local child welfare official explained that while they can use taxis to transport youth, they are not approved for use for children age 6 and younger. Foster parents and youth we spoke with shared challenges they have experienced with transportation to the school of origin (see text box). Experiences of Selected Foster Parents and Youth with Transportation to School of Origin Multiple foster parents in two states we visited shared that they were told by child welfare case workers that the foster parent(s) would have to transport children in their care to school for those children to remain in their current school. They told us that sometimes they could not drive the child due to distance or the needs of other youth in their care, and the child transferred to a new school. We also heard that other modes of transportation may be unreliable or cause difficulties for a child’s schedule. For example: One child in foster care in Arizona told us that she missed a week of school because the taxi provided by the child welfare agency failed to pick her up. A child in a foster care group home in Ohio said that despite being placed in a school which was in the same school district as her school of origin, her commute was long—she needed to take two public buses—and she sometimes missed dinner. On our survey, 30 of 50 SEAs reported that helping school districts determine how to fund the additional transportation costsdefined in the joint federal guidance as the difference between what a school district would otherwise spend transporting a student to their assigned school and the cost of transporting a child in foster care to their school of origin is also challenging. Among these 30 SEAs, 12 noted it was very or extremely challenging. Six school district and child welfare agency officials we interviewed also indicated that funding was a concern and some noted that transporting youth to their school of origin can result in extensive additional costs (see text box). Examples of Transportation Costs to Maintain Youth in Foster Care in Their School of Origin Over a school year, officials from a local child welfare agency said it spent $155,000 to transport students in one school district. According to officials at one school district, to transport one student, the school district had to hire a van at an estimated cost of up to $30,000 per year. In one month, another school district reported paying over $4,000 to transport five students. School district and child welfare officials said that they can rely on multiple funding streams—local, state, and/or federal—to cover these additional costs. Districts and local child welfare agencies reported that they sometimes split these costs, depending on their state’s policies. (See fig. 9 in appendix II for state-specific cost-sharing requirements reported in our survey.) For example, in Arizona, one agency transports the child to school and the other transports the child home and each pays for the cost of their one-way trip. To assist localities with funding additional transportation costs, nine SEAs said their state provides funding that partially or fully covers these costs. While educational and child welfare agencies may use federal funding through Title I or Title IV-E for the additional transportation costs, some SEA, school district, and child welfare agency officials we interviewed noted that they do not use these funds. Officials at a few school districts said they use Title I funding for other needs, while some child welfare agency officials explained their agency does not use Title IV-E funds because they did not have state “matching” funding, did not understand how to use the funds to reimburse schools for their costs, or had some youth who are not Title IV-E eligible. Ensuring Accurate Identification of Youth in Foster Care Thirty-two SEA survey respondents reported that ensuring school districts can accurately identify youth in foster care is at least somewhat challenging (see fig. 4). School district officials we spoke with expressed similar concerns. Officials we interviewed in nine of 10 districts stated they are not consistently aware of which students in their district are in foster care, and seven explained that there is no systematic way for school districts to be notified when a child enters or leaves care. Similarly, officials in four local child welfare agencies said they have no systematic way to inform schools when youth in foster care leave care or when their status in foster care changes. Officials from two school districts also stated their data systems have no way to indicate that a student is in foster care, so even if the child welfare agency notifies them of a youth’s status, they may not easily track the information. Officials from two school districts said not knowing the status of youth in foster care in their district impedes their ability to effectively implement the ESSA educational stability provisions. For example, one district official stated they would probably be transporting more youth to their school of origin if they knew which students were in foster care. In addition, two school district officials said that if they do not know which students are in foster care, they cannot provide additional supports that may be available to these youth, such as tutoring, financial assistance, or mental health services. The ability of school districts to accurately identify youth in foster care can also affect the accuracy of state and local report cards. Nine SEAs reported on our survey that they rely exclusively on school districts’ identification of youth in foster care for their state report cards. Of those nine, seven reported that ensuring that school districts accurately identify these youth is a challenge, which may affect the accuracy of the additional report card data required by ESSA. Some states and localities we visited had different ways to inform school districts when a youth’s foster care status changes, but officials noted varying degrees of consistency in notifying the districts of changes. Officials at two state child welfare agencies we visited told us they require the person enrolling the youth in school to present an official document that shows the youth is in state custody; however, they said schools are not informed when a child leaves foster care. One county and one state we visited had electronic data sharing agreements between child welfare and educational agencies for the purposes of updating school district records when a child enters and leaves foster care. Specifically, in that county, once a child enters foster care under the custody of the county child welfare agency, the school district’s database automatically receives pertinent information from the child welfare agency, according to officials. School and child welfare agency officials meet monthly to ensure data accuracy. In Georgia, officials from the state educational agency said they signed a data sharing agreement in spring 2018 with the state child welfare agency to allow information about youth in foster care to be provided to school districts. The previous data sharing agreement prevented the SEA from sharing the data with the school districts, according to officials. In Idaho (a state that participated in our discussion group), state officials said they ensure school districts are aware of youth in foster care by using an automated letter (see text box). Idaho’s iCARE System for Youth in Foster Care When a youth enters foster care or changes placements, Idaho’s iCARE system produces an automated letter that provides an initial communication from a child welfare social worker to the school district, SEA foster care points of contact, and the school principal. When the youth’s school of origin is entered into the system, the letter automatically populates the email addresses of the appropriate school district point of contact, SEA point of contact, and school principal. The letter contains the social worker’s initial best interest determination, and indicates if the student will need transportation to attend their school of origin, which the school district point of contact is responsible for coordinating. The school district point of contact has three days to provide input on the best interest determination when school is in session and 14 days during the summer months. The school district foster care point of contact and the child welfare social worker both must sign off on the plan identified within the electronic letter. Monitoring School Districts’ Efforts to Implement ESSA Educational Stability Provisions Under federal grant regulations, SEAs, which subgrant Title I funds to school districts, are required to conduct regular monitoring and oversight to ensure appropriate implementation of Title I by their school districts, and 43 SEA survey respondents reported that their states used one of the methods asked about in our survey to monitor how school districts implement at least one of the ESSA educational stability provisions. For example, over half (33) of SEAs reported that the Title I plans they receive from school districts include an assurance related to at least one of the ESSA educational stability provisions we asked about on the survey. More than two-thirds (36) of SEAs reported on our survey that effectively monitoring school districts’ implementation of the provisions is a challenge (see fig. 5). In their survey comments, eight SEA points of contact said limited state resources hinder their ability to ensure that the hundreds of school districts in their states properly execute the provisions. Officials we interviewed from all three SEAs in our site visits told us their states incorporate the educational stability provisions into their existing procedures for overseeing implementation of federal education programs. For example, SEA officials in Georgia told us that during one of their state reviews, they look for evidence of local agency collaboration, such as meeting agendas or emails. In Arizona, the SEA point of contact said he examines school district transportation procedures during on-site reviews. These on-site reviews occur for one- sixth of school districts in the state every year. (See table 3 in appendix II for more information on SEA monitoring of school districts.) Ensuring Immediate Enrollment and Obtaining Records While we did not ask on our survey about challenges related to immediate enrollment or obtaining records, seven state or local officials we spoke with noted difficulties with enrolling or obtaining records for students with disabilities who have individualized education programs, or students who previously attended juvenile justice or residential treatment facilities. Officials at a local child welfare agency and two school districts said that if an individualized education program is missing from a child’s records, they cannot know which services or classes a child might need and it may delay the child’s enrollment in the school or require switching classes again. Officials from Georgia’s SEA said they mitigate this challenge by providing school districts the option to share individualized education programs electronically, which enables other school districts that need the records to more easily obtain them. Education has provided technical assistance to states, at times in collaboration with HHS, to help states implement the ESSA educational stability provisions. Education’s technical assistance included written guidance, webinars, and in-person meetings, according to Education officials. Written guidance: Education and HHS jointly issued non-regulatory guidance on June 23, 2016 to help state and local educational agencies meet their obligations related to educational stability for youth in foster care under ESSA. On the same day, Education and HHS also issued a joint letter to chief state school officers and state child welfare directors that provided an overview of the ESSA educational stability provisions. Education sent an additional letter to chief state school officers on December 5, 2016, that provided information about the timelines for implementing the provisions. The letter also requested states to provide Education with their state foster care point of contact. Webinars: Education and HHS hosted several webinars for state educational and child welfare agencies that addressed a number of issues related to implementation of the ESSA educational stability provisions. In late summer 2016, Education and HHS hosted four webinars on the roles and responsibilities of educational and child welfare agency points of contact; best interest determinations and immediate enrollment; transportation; and effective collaboration. These webinars described the related ESSA requirements and featured selected states’ approaches to implementing the provisions. The State Support Network, one of Education’s technical assistance providers, facilitated another series of webinars that were offered in summer 2018 to address areas of implementation that states reported to be particularly problematic. HHS staff also participated in the webinar series, and topics included collaboration with child welfare agencies, data systems, transportation, and roles and responsibilities of points of contact. In-person and other assistance: Education provided additional assistance to state educational agencies through an in-person meeting and continuously provides assistance upon request. Education and HHS jointly held a session on sharing data to support students in foster care during its Combined Federal Programs Meeting for SEA officials in December 2018 in Washington, DC. At this meeting, Education also facilitated a session during which foster care points of contact networked with each other and subject matter experts, shared resources, and discussed outstanding implementation challenges. In addition, Education officials told us that they assign each state a point of contact at Education, and states can request technical assistance at any time through their assigned contact. This contact can work with the appropriate offices within Education to provide information requested by states and can facilitate further technical assistance through the State Support Network. Education officials said they respond to questions from states generally asking about expectations and requirements for the ESSA educational stability provisions. Thirty-seven SEAs reported on our survey that they would like additional federal assistance as they continue to implement the ESSA educational stability provisions. Our survey showed that most SEAs were interested in receiving additional guidance related to transportation cost sharing, transportation funding options, and arranging transportation; data privacy; and state monitoring of school districts’ efforts to implement these provisions, among other topics (see fig. 6). (Also see fig.10 in appendix II for all survey responses on these topics.) With respect to transportation issues, several state officials commented that they would like more information on how other states and localities are arranging and funding transportation. Regarding data privacy, a few other officials commented that they could use more information regarding privacy laws and what information can be shared across agencies. A few SEA officials noted that guidance on how they could monitor school district implementation would be useful. A majority of SEAs reported that opportunities for in-person and virtual meetings with a federal point of contact and their SEA and state child welfare agency counterparts, and a federally supported clearinghouse of information with sample documents from other states, would be moderately to extremely helpful (see fig. 7). (Also see fig. 11 in appendix II for all survey responses on this topic.) State educational and child welfare officials we interviewed explained that in-person and virtual meetings are helpful because they allow them to ask the federal contact questions and share and discuss issues with each other. Similarly, SEA officials in our discussion sessions said they would like federal agencies to organize more collaborative opportunities for SEA points of contact to interact with their peers to help identify best practices they can adapt in their state. Some states suggested Education could adopt methods it uses for other programs, such as the Education for Homeless Children and Youth program, to provide assistance and support to foster care points of contact, such as facilitating regional phone calls and identifying a point of contact specific to foster care at the federal level. According to Education officials, in June 2019 the agency selected a staff person to serve as the federal point of contact to work directly with SEA foster care points of contact, and they told us Education maintains a designated mailbox for all foster care-related correspondence (FosterCare@ed.gov). Education officials informed us that they plan to develop a community of practice for a small group of SEA foster care points of contact who will meet regularly for several months, which may facilitate more peer to peer interaction for a select number of states. Education plans to work with the Legal Center for Foster Care and Education to convene and facilitate the community of practice. According to Education officials, the community of practice will provide networking opportunities for participants to ask questions and obtain answers from their peers, and may include discussions of promising practices at the state and local level, among other areas. Officials said they will solicit interest from all SEAs about the opportunity to participate in the community of practice. However, they will limit the number of participants, depending on the level of interest, to 10 to 12 SEAs to promote discussion and sharing among states. Officials noted that if more states are interested in participating in the community of practice than they can accommodate, they will consider additional ways to support and share information with those additional states. Education officials also noted that they are exploring other types of technical assistance to facilitate more interaction and information exchange among states, such as a web portal where states can upload and share documents. Although Education is planning to develop a community of practice and is exploring other types of technical assistance, it may not have effective methods to reach all SEA points of contact to inform them of this assistance. In the course of our follow up on our survey, we determined that 22 of the current SEA points of contact were missing from Education’s email list. Education primarily disseminates information pertaining to the ESSA educational stability requirements to states through email. Twenty-three SEAs reported on our survey that they were not aware of webinars that Education offered in summer 2018. We discussed the email list with Education officials in June 2019 and they told us they had not conducted outreach to states to update the email list since they initially identified the SEA points of contact in 2016. Rather, officials said the email list was updated on an ad hoc basis, and Education depended on states to inform them when they want someone added to the email list. Subsequent to that discussion, in response to a recommendation included in a draft of this report which Education reviewed, Education officials told us they updated the email list in July and August 2019, and planned to update it quarterly moving forward. Education officials also acknowledged it could be useful to publicize the email list on its website. Education does not maintain information about its technical assistance webinars or other relevant materials in a centralized online location. Information relevant to implementing the ESSA educational stability provisions is located on multiple Education web pages, and the materials from the most recent 2018 webinars, including the recorded session and related sample documents shared by a number of states, are only available on a third party website for which there is no link from Education’s website. In our survey, SEA points of contact reported that they are interested in receiving additional information from other states. Thirty-seven SEAs reported in our survey that a clearinghouse of information with sample documents from other states would be helpful, and 22 of these 37 reported that this would be extremely helpful. One SEA official commented that it would be useful to have a clearinghouse that could be shared with school districts and other relevant parties nationwide. Federal standards for internal control maintain that management should select appropriate methods of communication, such as providing hard copy or electronic documents or conducting face-to- face meetings, and should periodically evaluate the methods of communication in order to communicate quality information on a timely basis. Without creating and maintaining a centralized online location for SEAs to access all related information, Education cannot ensure that all SEAs have access to technical assistance and guidance that could help them implement the ESSA educational stability provisions. Education officials told us that in 2020, they expect to fully implement the monitoring protocols for reviewing how states are implementing the ESSA educational stability provisions. Education officials said they plan to test draft protocols as part of a pilot by fall 2019 to determine necessary revisions and expect the final protocols to be implemented by fall 2020. According to Education officials, once the protocols are implemented, they plan to use a risk assessment approach to determine which states to review each year, and anticipate reviewing approximately nine states each year, depending on staff and resources. As part of their reviews, Education officials told us they plan to visit two school districts in each state under review to assess how the selected states are implementing the ESSA requirements, and to determine whether districts are getting appropriate support from the states. According to the draft monitoring protocols, during its state reviews, Education plans to obtain information on the following areas related to educational stability: SEA collaboration with the child welfare agency, best interest determinations, immediate enrollment, SEA foster care point of contact, and school district points of contact and transportation procedures. Education reviewed states’ plans for implementing Title I, however, Education officials said that the plans contain little information about the ESSA educational stability provisions. To receive Title I funds, states are required to submit state plans to the Secretary of Education, and the Secretary is required to approve the state plans if they meet the requirements in the law. While state plans are required to describe the steps the SEA will take to ensure collaboration with the state child welfare agency to ensure the educational stability of children in foster care, including various assurances, Education did not include specific instructions for information states should include on these provisions in the state plan template it developed for states. Youth in foster care face enormous challenges in their everyday lives and school can offer a stabilizing environment. Maintaining connections with teachers and friends, in addition to remaining in a familiar academic environment, can enhance the chances that a student is academically successful. However, many children in foster care are at higher risk of frequently changing schools, which can affect their academic achievement. ESSA made changes to the Title I program to help improve the educational stability of children in foster care. In the years since ESSA was enacted, SEAs and school districts have taken different approaches to implement its educational stability provisions, including collaborating with their child welfare agency counterparts. Most SEAs we surveyed reported common challenges with staff turnover and assisting districts with arranging transportation, among others, which can affect the successful implementation of the educational stability provisions. In addition, SEA officials are seeking more opportunities to understand how other states and localities have implemented the provisions and learn from their peers. Despite the assistance Education has provided to SEAs on a range of topics, the mechanisms Education uses to inform states of assistance are limited. The email list it uses to notify SEA foster care points of contact had not been systematically updated until July 2019, and resources on educational stability are not housed in one space. Without improvements in areas like these, states will not have access to all of the available resources that can help them improve the educational stability of youth in foster care, and ultimately, their academic success. The Secretary of Education should develop an online clearinghouse of sample documents from states and localities who wish to share them, past webinar recordings and their related documents, and links to other relevant resources that all SEAs can access. (Recommendation 1) We provided a draft of this report to Education and HHS for review and comment. Education provided written comments, which are reproduced in appendix III, as well as technical comments, which we incorporated as appropriate. HHS did not have comments. We also provided relevant excerpts to states we visited and incorporated their technical comments as appropriate. In its written comments, Education agreed with our recommendation to develop an online clearinghouse and noted actions it plans to take to implement it. Specifically, Education said in fall 2019, its Office of Elementary and Secondary Education will restructure its entire website to better organize its information, and create a new web page to house all foster care-related information and resources. Additionally, Education said this office will launch a virtual portal through which SEA foster care points of contact may collaborate and share resources. In addition, in a draft report sent to Education in August 2019, we included a recommendation to Education to update its foster care point of contact email list, and develop a process to update it at regular intervals. Education noted in its comment letter that it had updated its email list and that it will solicit updates to the email list on a quarterly basis, so we subsequently removed this recommendation. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretaries of Education and Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at 617-788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines (1) the challenges states and selected local educational agencies face implementing the requirements of the Every Student Succeeds Act (ESSA) related to educational stability for youth in foster care, and (2) how the Department of Education (Education) provided technical assistance and monitored states and localities to ensure compliance with these requirements, including collaborating with the Department of Health and Human Services (HHS). To address both objectives and obtain national information, we held three discussion groups with officials from state educational agencies and child welfare agencies and conducted a web-based survey of state educational agencies in the 50 states, the District of Columbia, and Puerto Rico. To obtain more in-depth information, we visited three states—Arizona, Georgia, and Ohio—where we interviewed officials from state and local educational agencies and child welfare agencies. We reviewed relevant federal laws and regulations, Education and HHS guidance to states, and other research publications. We also interviewed officials from Education and HHS’s Administration for Children and Families, and other organizations that carry out efforts related to education and child welfare, including the Legal Center for Foster Care and Education and Casey Family programs, regarding the provisions, federal requirements and guidance, and state and local implementation. To learn about actions states have taken to implement the ESSA educational stability provisions and challenges they have encountered, we held three discussion groups, two with state educational agency (SEA) officials, and one with state child welfare agency officials, during a national meeting for SEA foster care points of contact and state child welfare agencies in Greensboro, North Carolina in October 2018. To solicit participants for these groups, we asked the meeting organizers to forward an invitation we drafted to all individuals who registered for the meeting to participate in our discussion groups, and also allowed individuals to sign up once they arrived at the conference. Meeting attendees self-selected to participate in the groups. Each of our discussion groups with SEA officials had seven participants, for a total of 14 state agency officials representing 14 states. Our discussion group of state child welfare agency officials had six participants representing five states. Discussion groups were guided by a GAO moderator using semi- structured interview protocols. These protocols included open-ended questions that encouraged participants to share their thoughts and experiences on implementing the ESSA educational stability provisions, including how they monitored local agencies, and whether any additional federal assistance is needed. To reach group consensus on the top challenges facing states as they implement the provisions, we used a nominal group technique. Officials from each state identified their state’s top three implementation challenges. The group then created a list from those named challenges and officials from each state used stickers to identify their top challenges from the list. Discussion groups are intended to generate in-depth information about the reasons for participants’ attitudes on specific topics and to offer insights into their concerns about and support for an issue. They are not designed to (1) demonstrate the extent of a problem or generalize results to a larger population, (2) develop a consensus to arrive at an agreed- upon plan or make decisions about what actions to take, or (3) provide statistically representative samples or reliable quantitative estimates. For these reasons, and because discussion group participants were self- selected volunteers, the results of our discussion groups are not generalizable. To learn about actions states have taken to implement the ESSA educational stability provisions and challenges they have encountered, we conducted a survey of SEA officials in the 50 states, the District of Columbia, and Puerto Rico. The survey was administered from January to March 2019 and we had a 98 percent response rate. The survey used a self-administered, web-based questionnaire, and state respondents received unique usernames and passwords. Our survey population was foster care points of contact at SEAs. We used multiple sources to create an initial list of points of contact, including a list provided by the Department of Education, SEA website pages related to foster care, and information from knowledgeable experts in the field. We reached out to each point of contact to ask them to confirm they were the foster care point of contact for their state or identify the appropriate point of contact. We instructed respondents to consult with others who were familiar with their state’s implementation of the provisions, if doing so would provide more accurate responses. Our survey included 20 fixed-choice and open-ended questions. We asked how SEAs collaborated with the state child welfare agency, how they assisted local educational and/or child welfare agencies, what challenges they encountered, and what assistance has been and would be helpful from the Department of Education in implementing the provisions. To draft the closed-ended questions and answer choices on the survey, we drew from recommended practices suggested in HHS and Education’s joint non-regulatory guidance to states, information shared during webinars sponsored by HHS and Education, and interviews with stakeholders, including our discussion groups with state educational and child welfare agencies. A draft of the survey questionnaire was reviewed by officials at Education, a knowledgeable stakeholder organization, and an independent GAO survey professional for completeness and accuracy. We made revisions based on their comments. We conducted three pretests—one by phone and two in-person—with SEA foster care points of contact from three different states to check that (1) the questions were clear and unambiguous, (2) terminology was used correctly, (3) the questionnaire did not place an undue burden on agency officials, (4) the information could feasibly be obtained, and (5) the survey was comprehensive and unbiased. To obtain our 98 percent response rate (51 out of 52 SEAs), we made multiple follow-up contacts by email and phone from January to March 2019 with points of contact who had not yet completed the survey. While 51 surveyed officials affirmatively checked “completed” at the end of the web-based survey, not all officials responded to every question or the sub-parts of every question. We conducted additional follow-up with a small number of respondents to verify key responses. Because this was not a sample survey, the survey has no sampling errors. However, the practical difficulties of conducting any survey may introduce errors, commonly referred to as non-sampling errors. For example, unwanted variability can result from differences in how a particular question is interpreted, the sources of information available to respondents, or how data from respondents are processed and analyzed. We tried to minimize these factors through our reviews, pre-tests, and follow-up efforts. In addition, the web-based survey allowed SEA foster care points of contact to enter their responses directly into an electronic instrument, which created an automatic record for each state in a data file. By using the electronic instrument, we eliminated the potential errors associated with a manual data entry process. To learn about actions states and localities have taken to implement the ESSA educational stability provisions and challenges they have encountered, we conducted site visits to three states to obtain information from state and local educational agency officials, state and local child welfare officials, foster parents, and current and former youth in foster care. We selected the three states—Arizona, Georgia, and Ohio—to represent a mix of factors, including type of child welfare agency (state or county administered), number of children in foster care, number of school districts, geographic dispersion, and variety in types of school districts (urban, suburban, rural). In each state we visited an urban, suburban, and rural school district, where we met with the school district officials responsible for implementing the ESSA educational stability provisions, and their primary child welfare agency counterparts. We also met with state educational and child welfare agency officials. We used a semi- structured interview protocol for these meetings. We held discussion groups with a total of 13 youth in foster care or formerly in foster care in three states, and in two states, we held discussion groups with a total of 14 foster parents, to obtain their perspectives on implementation of the provisions and educational stability generally. Although we cannot generalize our findings beyond these states and localities, these visits provided us with illustrative examples of how states and localities are implementing the ESSA educational stability requirements. We conducted this performance audit from June 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Additional Survey Data to the question: “How much of a challenge, if at all, is each of the following items in implementing the ESSA educational stability provisions?” The term “ESSA educational stability provisions” refers to the amendments made by the Every Student Succeeds Act (ESSA) to Title I, Part A of the Elementary and Secondary Education Act of 1965 that are related to the educational stability of youth in foster care. These provisions have been codified at 20 U.S.C. §§ 6311(g)(1)(E), 6311(h)(1)(C), and 6312(c)(5). Best interest determination documents, like meeting documentation templates, questions to consider during the meeting, or sample notices to inform parties of the decision Sample memorandum of understanding/agreement for data sharing between school districts and local child welfare agencies for the purposes of identifying youth in foster care for the report card reporting 17 10 school of origin when in their best interest will be provided, arranged, and funded for the duration of the time in foster care. This includes states that reported that they solely respond when alerted to issues and do not conduct any other systematic monitoring activities. Specifically, nine states reported responding when alerted to issues regarding the provisions on best interest determinations and immediate enrollment, and did not report conducting any other monitoring activities. Similarly, 14 states reported solely responding when alerted to issues regarding new enrolling schools immediately contacting schools of origin to obtain relevant academic and other records, and did not report conducting any other monitoring activities. Finally, seven states reported responding when alerted to issues related to the provision on transportation procedures, and did not report conducting any other monitoring activities or did not know if their state monitors LEAs in other ways. educational stability of youth in foster care. These provisions have been codified at 20 U.S.C. §§ 6311(g)(1)(E), 6311(h)(1)(C), and 6312(c)(5). In addition to the contact named above, the following individuals made important contributions to this report: Elizabeth Morrison (Assistant Director), Kate Blumenreich (Analyst-in-Charge), Aimee Elivert, and Kelsey Kreider. Also contributing to this report were Steven Campbell, William Chatlos, Sarah Cornetto, Holly Dye, Jill Lacey, Jessica Orr, Catherine Roark, and Curtia Taylor.", "summary": "Roughly 270,000 school-aged youth were in foster care at the end of fiscal year 2017. Youth in foster care may change schools frequently, which can negatively affect their academic achievement. ESSA, enacted in 2015, reauthorized the Elementary and Secondary Education Act of 1965 and included provisions to improve educational stability for youth in foster care. These included requiring state educational agencies to ensure youth placed into foster care stay in their current school, unless it is not in their best interest to do so. GAO was asked to review implementation of these provisions. This report examines (1) the challenges SEAs and selected school districts face implementing the ESSA educational stability provisions for youth in foster care, and (2) how Education provides technical assistance and monitors state implementation efforts. GAO surveyed SEA foster care points of contact in the 50 states, District of Columbia, and Puerto Rico and all but one state responded. In addition to interviewing federal officials, GAO interviewed selected state and local educational and child welfare agency officials, and held discussion groups with foster youth and parents, in three states selected by number of youth in foster care, among other factors. GAO also held discussion groups with officials from 14 SEAs and 5 state child welfare agencies, and reviewed relevant federal laws, regulations, guidance, and technical assistance. State educational agencies (SEAs) reported several challenges in implementing the provisions in the Every Student Succeeds Act (ESSA) related to educational stability for youth in foster care. In their responses to GAO's national survey, SEAs reported challenges, including high turnover among local educational and child welfare agency officials, and with identifying and arranging transportation to schools for students (see figure). Turnover of local staff can result in the loss of knowledge and experience needed to implement the provisions, according to SEA and local officials we interviewed. Regarding transportation, ESSA requires school districts to work with child welfare agencies to provide and fund transportation so that youth in foster care can remain in their current school when it is in their best interest. Six school district and child welfare agency officials we interviewed indicated that funding was a concern and some noted that transporting youth to their current school can result in extensive costs. The Department of Education (Education) provided technical assistance in the form of written guidance, webinars, and in-person meetings to help states implement the ESSA educational stability provisions. Education officials said they also plan to monitor state implementation of the provisions. Most SEA officials reported in GAO's survey that they would like additional assistance and more opportunities to interact with other state officials. Education plans to convene a community of practice for several states in which participants will meet regularly for several months, and is exploring other technical assistance efforts. To share information about implementing the ESSA educational stability provisions, Education maintains an email address list of SEA foster care points of contact. GAO found that the list was inaccurate and not regularly updated. Education updated the list in late summer 2019 and plans to do so quarterly. Education also provides information online, but the information is scattered across different web pages. Twenty-two SEA officials reported on GAO's survey that a clearinghouse of information would be extremely helpful. Federal standards for internal control require agencies to externally communicate necessary information in a manner that enables them to achieve their objectives. Without a dedicated web page about implementing the provisions, states may not receive the assistance they need to improve educational stability for youth in foster care. GAO recommends that Education develop an online clearinghouse of resources. Education agreed with GAO's recommendation.", "document_type": "gao"}
{"report": "Corrosion is defined in section 2228 of Title 10, U.S. Code, as the deterioration of a material or its properties due to a reaction of that material with its chemical environment. Corrosion can take varied forms, such as rusting, pitting, galvanic reaction, calcium or other mineral build- up, degradation due to ultraviolet light exposure, and mold, mildew, or other organic decay. Corrosion can be either readily visible or microscopic. To provide leadership on corrosion matters, including the development of policy guidance and oversight, consistent with section 2228, DOD has established an organizational structure that includes the Corrosion Office and Corrosion Executives. The Director of the Corrosion Office is to provide oversight and coordination of corrosion control and prevention efforts for the department. The military departments have each assigned officials to serve as Corrosion Executives. The Corrosion Executives operate within the chain of command of their respective military departments, while also coordinating with the Corrosion Office. Prior to August 2018, the Corrosion Office reported directly to the Under Secretary of Defense for Acquisition, Technology, and Logistics. In 2018 the Corrosion Office was relocated within DOD, after section 901 of the National Defense Authorization Act for Fiscal Year 2017 (Pub. L. No. 114- 328, hereinafter referred to as the Act) required DOD to restructure parts of the Office of the Secretary of Defense. Among other things, the Act eliminated the Under Secretary of Defense for Acquisition, Technology, and Logistics, and it created: The Under Secretary of Defense for Research and Engineering, who, among other things, serves as the principal advisor to the Secretary of Defense on all research, engineering, and technology development activities and programs in DOD. The Under Secretary of Defense for Acquisition and Sustainment, who, among other things, serves as the principal advisor to the Secretary of Defense on acquisition and sustainment in DOD. In addition, the Under Secretary establishes policies on and supervises all elements of DOD relating to acquisition and sustainment. As part of this restructure, effective August 1, 2018, DOD relocated the Corrosion Office within the department’s restructured acquisition and sustainment organization, as shown in figure 1. The Corrosion Office is now located within the department’s Office of the Under Secretary of Defense for Acquisition and Sustainment. Within this office, the Deputy Assistant Secretary of Defense for Materiel Readiness oversees the Corrosion Office. The Deputy Assistant Secretary of Defense for Materiel Readiness is a principal advisor to the Assistant Secretary of Defense for Sustainment; provides integration and oversight of DOD’s maintenance program; and develops policies and procedures for materiel readiness and maintenance support of DOD’s major weapon systems and military equipment. The Deputy Assistant Secretary of Defense for Materiel Readiness stated that he is supportive of the Corrosion Office’s mission and views its move to the Materiel Readiness organization as fitting in with the other areas under his oversight. Officials representing the military departments’ Corrosion Executives also stated that they support DOD’s organizational movement of the Corrosion Office. They stated that they continue to find the Corrosion Office to be helpful in establishing corrosion prevention standards and in providing opportunities for networking and information sharing by means of triannual corrosion forums. In addition, they stated that they have found the Deputy Assistant Secretary of Defense for Materiel Readiness to be supportive of corrosion oversight and prevention in their meetings with him. Since August 1, 2018, the Corrosion Office has had an acting director. According to a Corrosion Office official, the Deputy Assistant Secretary of Defense for Materiel Readiness is involved in the ongoing hiring process for a permanent director. According to a Corrosion Office official, the time frame in which a permanent director is projected to be in place is Spring 2019. Section 2228 of Title 10, U.S. Code, contains provisions regarding the duties and responsibilities of the Director of the Corrosion Office. Specifically, these duties and responsibilities include the following: overseeing and coordinating efforts throughout DOD to prevent and mitigate corrosion of military equipment and infrastructure, and developing and recommending corrosion policy guidance to be issued by the Secretary of Defense; developing and implementing, on behalf of the Secretary of Defense, a long-term strategy to reduce corrosion and the effects of corrosion on military equipment and infrastructure; reviewing corrosion programs and funding levels proposed by the military departments during the annual internal DOD budget review process as those programs and funding proposals relate to programs and funding for the prevention and mitigation of corrosion, and submitting recommendations regarding those programs and proposed funding levels to the Secretary of Defense; providing oversight and coordination of efforts within DOD to prevent or mitigate corrosion during the design, acquisition, and maintenance of military equipment, as well as the design, construction, and maintenance of infrastructure; monitoring DOD acquisition practices to ensure that the use of corrosion prevention technologies and the application of corrosion prevention treatments are fully considered during research and development in the acquisition process; and ensuring that, to the extent determined appropriate for each acquisition program, such technologies and treatments are incorporated into that program, particularly during the engineering and design phases of the acquisition process. The Corrosion Office continues to perform the duties outlined in section 2228, as evidenced below. Specifically, the Corrosion Office is taking the following actions: Developing and recommending corrosion policy guidance. DOD previously developed and issued an instruction that establishes policy, assigns responsibilities, and provides guidance for corrosion prevention and mitigation. The Corrosion Office, via a working group in a working integrated product team, plans to update this DOD instruction. The working group intends for the updated DOD instruction to reflect the Corrosion Office’s movement within DOD’s restructured acquisition and sustainment organization; any statutory changes made to section 2228 since it was last issued; direction from the new acquisition and sustainment leadership; and any changes made to address the findings and recommendations in our 2018 report. Additionally, the Corrosion Office plans to create a new DOD manual on corrosion that, according to Corrosion Office officials, will contain operating procedural details on, among other items, conducting and recording the Corrosion Office’s review and evaluation processes. According to Corrosion Office officials, the Corrosion Office’s target time frame for updating this DOD instruction and creating this new manual is by the end of calendar year 2020. Also, since July 2018 the Corrosion Office has been reviewing other DOD policy guidance to identify relevant documents in which corrosion content should be added or updated. Corrosion Office officials stated that the new director will update existing corrosion prevention and mitigation policy guidance, directives, and instructions in coordination with the military departments’ Corrosion Executives, under the guidance of the Deputy Assistant Secretary of Defense for Materiel Readiness. Developing and implementing a long-term strategy to reduce corrosion and its effects. In 2015 DOD issued a long-term strategy for preventing and mitigating corrosion that calls for implementing DOD- wide standards and improving strategies and processes to prevent, detect, and treat corrosion. According to Corrosion Office officials, there was a planning meeting for the working integrated product teams’ leads and co-leads in mid-March 2019. At this meeting, the team leads and co-leads prepared a draft update to the long-term strategy, which had last been updated in 2015. These officials told us that examples of changes included in the draft update are revised goals, objectives, and metrics. In addition, these officials told us that the draft update was aligned to reflect the DOD sustainment and materiel readiness mission statements and objectives articulated by the Assistant Secretary of Defense for Sustainment and the Deputy Assistant Secretary of Defense for Materiel Readiness. According to Corrosion Office officials, this draft plan is being reviewed internally, and the Corrosion Office’s target time frame is to update it by the end of calendar year 2020. Reviewing corrosion programs and funding levels proposed by the military departments and submitting related recommendations to the Secretary of Defense. As it did prior to the restructure, the Corrosion Office continues to review the military departments’ proposed corrosion-related programs and funding levels during the annual internal DOD budget review process. In addition, it continues to annually submit a report to Congress on corrosion funding with the defense budget materials. As part of this process, the Corrosion Office collected information from the Corrosion Executives on the corrosion control and prevention programs within the respective military departments. The Corrosion Office in Autumn 2018 included the information provided by each Corrosion Executive as appendixes in its annual report on corrosion funding. The fiscal year 2020 report was submitted to Congress on February 15, 2019. Monitoring and ensuring that corrosion prevention and mitigation are incorporated into acquisition and maintenance programs. As we reported in November 2018, Corrosion Office officials told us that they continue to perform the Corrosion Office’s acquisition and maintenance-related duties. For instance, the Corrosion Office continues to review acquisition documentation, such as Systems Engineering Plans, and to maintain information on hundreds of technologies for preventing and mitigating corrosion. In November 2018 we recommended that the Corrosion Office develop a process to maintain documentation of its reviews of corrosion planning for major weapon system programs. Further, we stated that these records, at a minimum, should show what comments were made by the Corrosion Office in its reviews and evaluations, and should track the actions taken to resolve those comments. DOD concurred with this recommendation and stated that the Corrosion Office would develop and maintain such a process. More specifically, Corrosion Office officials stated that they plan to describe this process in a new DOD manual on corrosion. According to Corrosion Office officials, the DOD manual will also include information on considering corrosion during the weapon system program-planning evaluation process. In addition, the Corrosion Office plans to develop an internal data system that these officials told us will track its reviews and evaluations along with the weapon system programs’ responding actions. According to Corrosion Office officials, their target time frame is to create this new manual and internal data system by the end of calendar year 2020. Corrosion Office officials told us that they have not changed the way in which they carry out additional authorities identified in section 2228. For example, the Corrosion Office continues to develop and deliver corrosion training with the Defense Acquisition University. In addition, it continues to interact with industry, trade associations, other government corrosion prevention agencies, academic research and educational institutions, and a scientific organization engaged in corrosion prevention. According to the Deputy Assistant Secretary of Defense for Materiel Readiness, he is working to change some of the ways in which the Corrosion Office operates. Specifically, he is working to increase the following: corrosion advocacy throughout DOD; oversight of the Corrosion Office; the accountability of the military departments and the Corrosion Office to mitigate corrosion; and the transparency of corrosion and its alignment with materiel readiness. One of the efforts made by the Corrosion Office for achieving these objectives is by providing funding for corrosion technology demonstration projects proposed and implemented by the military departments. According to Corrosion Office officials, the Deputy Assistant Secretary of Defense for Materiel Readiness changed the process for awarding fiscal year 2019 funding by obtaining feedback from the military departments’ Corrosion Executives as to which project proposals should receive funds. Officials representing the military departments’ Corrosion Executives confirmed that they were able to provide such feedback. Corrosion Office officials told us that, as of April 2019, they had selected and funded demonstration projects for fiscal year 2019 in part based on the information provided by the military departments. In addition, the Deputy Assistant Secretary of Defense for Materiel Readiness stated that he wanted to have more of an emphasis on funding demonstration projects that would be beneficial to all of the military services. According to Corrosion Office officials, another effort they undertook at the direction of the Deputy Assistant is that of working to make the Corrosion Office more cost-efficient by streamlining the number of professional services and other support contracts it awards. For example, Corrosion Office officials stated that by consolidating five contracts for professional services and reporting on the cost of corrosion into a single contract by a target date of mid-July 2019, they estimate achieving savings of approximately $2 million. In another effort, the Deputy Assistant Secretary of Defense for Materiel Readiness provided written feedback to each of the military departments’ Corrosion Executives in March 2019 on their respective departments’ corrosion control and prevention programs. Specifically, the feedback concerned whether each military department’s calendar year 2018 corrosion report complied with statutory requirements; each department’s strengths and weaknesses related to its corrosion efforts; and recommendations each department had identified for itself to implement. In calendar years 2003 through 2018, we made 35 recommendations to the Corrosion Office in 11 corrosion-related products on topics such as strategic planning, performance management, and mandatory oversight reports. In responding to these products, DOD initially concurred with 16 of those recommendations, partially concurred with eight, and non- concurred with 11. As of March 2019 DOD’s Corrosion Office had taken action or had plans to take action on most of our recommendations. Specifically, out of 35 recommendations, DOD’s Corrosion Office had taken action on 18 recommendations, including sufficient action for us to close those recommendations as implemented; planned to take action to implement 12 additional recommendations. These planned actions include, among other actions, updating existing guidance and developing new policy or processes; and did not plan to take action on the remaining five recommendations. Corrosion Office officials stated that they did not plan to take action on these recommendations for a variety of reasons. For instance these officials stated that the Corrosion Office did not have the authority over the military departments to take the recommended actions. We continue to believe our recommendations are valid. Appendix I summarizes all 35 recommendations and DOD’s response to each recommendation at the time of our report and provides information, as of March 2019, on DOD’s actions or planned actions to address each recommendation. In some instances DOD had taken action or planned to take action on recommendations with which it had not concurred at the time of our report. We provided a draft of this report to DOD for review and comment. DOD concurred with the draft and had no technical comments. We are sending copies of this report to the appropriate congressional committees and to the Acting Secretary of Defense and the Under Secretary of Defense for Acquisition and Sustainment. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or our staff have any questions about this report, please contact me, Diana Maurer, at (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs are listed on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, Marilyn Wasleski (Assistant Director), Dawn Godfrey, Shvetal Khanna, Amie Lesser, Edward Malone, Nathan J. Napolitano, Carter Stevens, and Cheryl Weissman made key contributions to this report.", "summary": "Corrosion negatively affects DOD equipment and infrastructure and can lead to reduced asset availability, deterioration in performance, and increasing weapon system and infrastructure costs. According to a study contracted by DOD, the cost impact of corrosion to DOD in fiscal year 2016 was $20.6 billion. Senate Armed Services Committee Report 115-262 accompanying a bill for the John S. McCain National Defense Authorization Act for Fiscal Year 2019 included a provision for GAO to review aspects of the DOD Corrosion Office. This report examines (1) how the restructuring within the Office of the Secretary of Defense has affected DOD's Corrosion Office, including its performance of its statutory roles and responsibilities; and (2) what actions, if any, DOD has taken or has planned to implement recommendations GAO made from calendar years 2003 through 2018 related to corrosion management. GAO analyzed DOD documents, such as guidance and required reports provided to Congress, and interviewed DOD officials to address these objectives. GAO also assessed DOD's actions against its prior recommendations to determine the extent to which DOD had addressed the recommendations or has actions underway to address those recommendations. The Department of Defense (DOD) relocated the Office of Corrosion Policy and Oversight (Corrosion Office) within the restructured acquisition and sustainment organization in fiscal year 2018. Prior to the restructure, the Corrosion Office reported directly to the Under Secretary of Defense for Acquisition, Technology, and Logistics. As part of the restructure, DOD relocated the Corrosion Office within the Office of the Under Secretary of Defense for Acquisition and Sustainment, where it reports to the Deputy Assistant Secretary of Defense for Materiel Readiness. It continues to perform its statutory roles and responsibilities under the new oversight organization. For instance, it is continuing to develop and recommend corrosion policy guidance; develop and implement a long-term strategy to reduce corrosion; review corrosion programs and funding levels proposed by the military departments, and submit related recommendations to the Secretary of Defense; and monitor and ensure that corrosion prevention and mitigation are incorporated into acquisition and maintenance processes. DOD is also making or planning changes to the operation of the Corrosion Office, specifically planning to increase corrosion advocacy throughout DOD, oversight of the Corrosion Office, corrosion accountability of the military departments, and corrosion transparency and its alignment with materiel readiness. DOD's Corrosion Office has taken or planned actions to implement most recommendations GAO made in calendar years 2003 through 2018 related to corrosion management. Specifically, GAO made 35 recommendations to the Corrosion Office in 11 corrosion-related products on topics such as strategic planning, performance management, and mandatory oversight reports. In comments on these products, DOD concurred with 16 of those recommendations, partially concurred with eight, and non-concurred with 11. As of March 2019, DOD had taken action or planned to take action on most of GAO's prior recommendations (see figure). Specifically, DOD's Corrosion Office had taken action on 18 recommendations. Corrosion Office officials also described to GAO their plans to take action to implement 12 additional recommendations. These planned actions include, among other actions, updating existing guidance and developing new policy or processes. DOD stated that the Corrosion Office does not plan to take action on the remaining five recommendations. GAO continues to believe that its recommendations are valid.", "document_type": "gao"}
{"report": "This section provides information about abandoned hardrock mines, sites, and features; and federal and state agency roles in addressing abandoned hardrock mines. Federal and state agencies generally describe abandoned hardrock mines in terms of mine sites, the individual features that comprise a site, or both. However, these agencies do not all have a common definition for what constitutes an abandoned hardrock mine or mine site, as we found in 2008. The agencies generally agree on what constitutes an individual feature—for example, a feature can be a mine opening (such as a tunnel, pit, or vertical shaft), a structure, or a pile of discarded materials (known as mine tailings or waste rock) that is left behind after ore is crushed and the valuable minerals are extracted. They also generally agree that an abandoned mine site can be comprised of only one feature (e.g., an isolated mine shaft) or many features (e.g., an area with multiple entries, shafts, open pits, mill buildings, and tailings piles). There is no universally agreed-upon average number of features per site. Also, not all federal and state agencies count both sites and features—some agencies only count sites, some only count features, and some count both. The individual features that make up a mine site may pose hazards to physical safety and risks to human health and the environment. Physical safety hazards. Abandoned hardrock mine features that pose physical safety hazards generally present immediate danger of injury or death. Examples of physical safety hazards include unstable mine tunnels that can collapse without warning; unmarked open mine shafts and deep pits that pose a danger to individuals who may inadvertently drive off-road vehicles into them; and deadly concentrations of gases, such as carbon monoxide and methane, present inside some mines that can asphyxiate explorers. To address physical safety hazards, federal and state agencies typically focus on identifying and mitigating the risk from individual features. They may safeguard these features by, for example, filling, capping, or gating the abandoned mine openings with engineered structures. After a dangerous feature is identified, an agency may post a warning sign or erect a fence to temporarily limit access to the feature until the agency can permanently close it. According to a 2008 Interior Inspector General report, physical hazards require the least expertise to identify and evaluate and the least funding to fix or mitigate. Environmental hazards. Mine features can also contribute to degradation of the environment and present short- and long-term risks to human health. For this report, we refer to these collectively as environmental hazards. People may be exposed to these hazards when recreating or living near an abandoned mine. Examples of environmental hazards include a mine tunnel that drains acidic water laden with heavy metals into a waste rock or tailings piles located along the banks or in the middle of streams that release hazardous substances such as arsenic, lead, and mercury into the water; and tailings that have dispersed into a surrounding community’s soil, exposing residents to harmful substances. The extent of environmental hazards at abandoned mines can vary widely, from sites that contain one draining tunnel and a few waste rock piles to sites with extensive underground tunnel networks, many waste rock piles, and miles of dispersed tailings. Some contaminated hardrock mine sites are included on the National Priorities List, which includes some of the most seriously contaminated sites that EPA identifies for long-term cleanup. The work required to address environmental hazards varies depending on the extent, type, and concentration of contaminants. For example, agencies may take one or more of the following actions at a site: remove waste rock or tailings from streams; develop passive water treatment systems that allow water to flow out of mines into treatment ponds; manage the waste on-site or transport it off-site for disposal; or establish active water treatment systems for the most contaminated sites that require continuous long-term monitoring, among other actions. According to EPA documents, sites with environmental hazards can cost hundreds of millions of dollars and take many years to address. For example, as of July 2019, the actual costs at the 25 most expensive mine and mineral processing sites ranged from $50 million to $583 million per site, and EPA had been working on some of the sites for over 20 years. Furthermore, agencies monitor remedies after completion to help ensure that they are achieving the desired results. Figure 1 depicts examples of physical safety and environmental hazards found at abandoned hardrock mine sites and activities that could take place to address them. Land ownership at abandoned mine sites is often complicated. The General Mining Act authorizes miners to patent, or purchase, the land associated with their mining claims—thereby mined land often passed from federal to private ownership. Partly because of this, many abandoned mine sites are a patchwork of federal, private, and other lands, and the ownership boundaries are not always clear. Agencies refer to these sites as mixed ownership sites. The Forest Service, BLM, the Park Service, EPA, and OSMRE, as well as states with abandoned hardrock mines, administer programs that address abandoned hardrock mines. Specifically, these federal and state agencies collect information about abandoned hardrock mine sites and features, and the associated hazards, on land under their jurisdictions. These agencies also safeguard the physical safety hazards and clean up the environmental hazards present at these mines. Agencies inventory and address these mines based on their different authorizing statutes, regulations, and missions. The Forest Service is responsible for managing about 193 million acres of national forests and grasslands throughout the United States. The Forest Service’s Safety and Environmental Restoration program oversees the agency’s work on abandoned hardrock mines. The Forest Service distributed $15.9 million in appropriations to the Safety and Environmental Restoration program in fiscal year 2019. USDA also distributed about $6.9 million in fiscal year 2019 to the Forest Service to address environmental hazards at several abandoned hardrock mines. USDA seeks recovery of cleanup costs under the Comprehensive Environmental Response, Compensation, and Liability Act of 1980, as amended (CERCLA), from responsible parties, such as current and former owners and operators of a contaminated site, to help reimburse costs at such sites. The Forest Service develops and maintains its information about abandoned hardrock mines primarily at its regional, national forest, and district offices. In general, the Forest Service tracks physical safety hazards by feature and environmental hazards by mine site. As of November 2019, the Forest Service did not have a national inventory of abandoned hardrock mine features or sites and the physical safety hazards they may pose. However, information about environmental hazards at abandoned hardrock mine sites is contained in a database maintained by USDA that tracks progress on all hazardous waste cleanup projects funded by the department, including projects at abandoned hardrock mines. Forest Service regional and national forest staff inventory, assess, mitigate, and monitor the physical safety and environmental hazards at abandoned hardrock mine sites on Forest Service-managed land as part of their daily responsibilities. BLM manages 245 million acres of public lands in the United States, located primarily in the western states and Alaska. BLM’s Abandoned Mine Lands program is aimed at protecting public safety and reducing liabilities by eliminating or minimizing physical safety and environmental hazards posed by abandoned mines, among other objectives. BLM’s Hazardous Materials Management program also addresses environmental hazards at all types of contaminated sites, including abandoned hardrock mines. In fiscal year 2019, BLM received a total of $38.5 million in appropriations for these programs. In addition, Interior distributed $2.7 million to BLM in fiscal year 2019 from the Central Hazardous Materials Fund—an Interior account that supports response actions undertaken at contaminated sites pursuant to CERCLA—for work at abandoned hardrock mines. BLM maintains a national inventory of abandoned hardrock mines in its Abandoned Mines and Site Cleanup Module database to help track information about sites, features, and hazards. However, as of 2019, BLM officials said the agency is shifting from tracking information by site, which can be subject to interpretation, to primarily tracking and reporting abandoned mine features. In addition to its abandoned mine database, BLM submits a subset of information about its contaminated abandoned mines to Interior for inclusion on the department’s list of contaminated sites, the Environmental and Disposal Liabilities list. BLM state, district, and field office staff inventory, assess, and mitigate the physical safety and environmental hazards at abandoned hardrock mines on BLM- managed land while conducting their daily work. The Park Service manages more than 85 million acres in 419 park units across the country. The Park Service addresses abandoned hardrock mines on this land through an abandoned mine safety program and an environmental compliance and cleanup program. In fiscal year 2019, the Park Service received $5 million in appropriations to address physical safety hazards on abandoned mineral lands. Interior also distributed $890,000 from the Central Hazardous Materials Fund to the Park Service to address contaminated abandoned hardrock mine sites in fiscal year 2019. The Park Service recovers costs from responsible parties at abandoned mine sites through CERCLA. The Park Service maintains information about abandoned hardrock mine sites and features in its Abandoned Mineral Lands Database. In 2013, the Park Service completed a system-wide inventory and assessment project to identify abandoned mines on lands it manages. In addition, the Park Service submits information to Interior about contaminated abandoned mine sites for inclusion on the Environmental and Disposal Liabilities list. Park Service headquarters and regional offices may assist park units in addressing hazards and preserving cultural resources and wildlife habitat at abandoned hardrock mines on Park Service-managed land. EPA administers the Superfund program, which was established under CERCLA to address the threats that contaminated waste sites pose to human health and the environment. As part of the Superfund program, EPA oversees and conducts investigations and cleanup actions at a variety of hardrock mine and mineral processing sites on private and other nonfederal lands and mixed ownership sites. The Superfund program operates on the principle that polluters are to pay for the cleanups rather than passing on the costs to taxpayers. EPA may compel parties statutorily responsible for contamination at sites to clean them up or to reimburse EPA for its cleanup costs. Responsible parties at abandoned hardrock mines could include current or former owners or operators of a site; persons who arranged for disposal, treatment, or transportation of hazardous substances; or the transporters of hazardous substances. To address contaminated sites—including abandoned mines—that do not have viable responsible parties, EPA uses funding from appropriations to the Superfund program, which were approximately $1.1 billion in fiscal year 2019. EPA maintains information about abandoned hardrock mine and mineral processing sites on nonfederal lands, including tribal lands, and mixed ownership sites in its national database of contaminated sites, the Superfund Enterprise Management System. EPA counts these mines and processing facilities by site and not by individual mine feature. According to EPA officials, many of the mine sites included in the database may contain tens to hundreds of individual features. In addition, EPA does not count sites that pose solely a physical safety hazard since they fall outside of the Superfund program mission. In addition, EPA and authorized states address certain abandoned hardrock mines in accordance with the Clean Water Act. Specifically, EPA and state agencies regulate discharges of pollutants to waters of the United States at abandoned mine sites under the act, such as mine tunnels draining contaminated water that exceeds water quality standards. To comply with the act, an entity operating a cleanup project involving a draining mine tunnel or other concentrated discharge source must obtain a permit, under which the discharge must be treated or managed to meet and maintain applicable water quality standards. OSMRE’s Abandoned Mine Land program primarily focuses on reclaiming and restoring land and water resources degraded by past coal mining, but the program also supports reclamation at abandoned hardrock mines. In accordance with the Surface Mining Control and Reclamation Act of 1977, as amended, OSMRE can provide grants for the reclamation of certain abandoned hardrock mines under limited circumstances—in particular, after a state or Indian tribe certifies that it has cleaned up its abandoned coal mine sites and the Secretary of the Interior approves the certification. Absent such certification, OSMRE can award these grants at the request of a state or Indian tribe where necessary to protect the public health, safety, general welfare, and property from extreme danger of adverse effects from the abandoned hardrock mine, and the Secretary of the Interior grants the request. In fiscal year 2019, OSMRE distributed a total of $310.5 million in grants to states and tribes to address abandoned coal and non-coal mines. OSMRE does not maintain an inventory of abandoned hardrock mines since the Abandoned Mine Land program’s primary objective is to address abandoned coal mines. States that receive grants from OSMRE to address non-coal abandoned mines may maintain their own inventories of abandoned hardrock mines. According to OSMRE budget documents, western states in particular often use OSMRE grants to address physical safety hazards at high-priority abandoned hardrock mines for which there is no other source of federal funding. States identify and address physical safety and environmental hazards at abandoned hardrock mines on state, county, and private lands within their borders, often through state abandoned mine programs. States may also work with federal agencies to identify and address these hazards on federal land. Some state agencies manage or oversee cleanup activities under CERCLA at abandoned hardrock mines. State agencies may receive funds to support their work at abandoned hardrock mines from nonfederal and federal sources, including state-appropriated funds, responsible parties under CERCLA, and cooperative funding agreements or grants from federal agencies. States with abandoned hardrock mines generally maintain databases or inventories that identify the locations of these mines and any associated hazards. As of May 2019, the Forest Service, BLM, the Park Service, and EPA together identified in their databases at least 140,652 abandoned hardrock mine features—of which over 60 percent are known to pose or may pose physical safety or environmental hazards. Officials from 13 western states also identified from their state databases about 246,000 abandoned hardrock mine features on federal and nonfederal lands within their states, including about 126,000 features that pose physical safety or environmental hazards. Some state information overlaps with federal agency information, but the extent of overlap is unknown, according to state officials. Federal and state officials also estimated that there likely are hundreds of thousands of additional abandoned hardrock mine features that they have not yet captured in their databases. The Forest Service, BLM, the Park Service, and EPA identified in their databases at least 140,652 abandoned hardrock mine features, as of May 2019. Of this amount, BLM identified 103,029 features and the Park Service identified 20,675 features. As previously noted, the Forest Service and EPA track abandoned mines by site and not by features associated with the sites; the Forest Service identified 16,375 sites and EPA identified 573 sites. According to agency officials, many abandoned hardrock mine sites contain more than one feature. Since there is no agreed-upon average number of features per site, we counted the minimum of one feature per Forest Service and EPA site for the purpose of this analysis. As a result, the total number of features identified by federal agencies likely is underestimated. Of the 140,652 total features, about 89,000 features are known to pose or may pose a physical safety or environmental hazard, according to information in the federal agencies’ databases. Specifically, agencies confirmed 7,802 features pose a hazard, of which 6,439 pose a physical safety hazard and 1,363 pose an environmental hazard; and identified 81,541 features with an unconfirmed hazard (whereby agency staff had not assessed current conditions in person to confirm the hazard), of which 60,279 may pose a physical safety hazard and 21,262 may pose an environmental hazard. Table 1 shows information about abandoned hardrock mine features that pose or may pose physical safety and environmental hazards, by agency. However, agency officials said there could be approximately 393,000 more abandoned hardrock mine features on federal land that the agencies identified on historic maps but have not captured in their captured in a central database. In addition, BLM officials estimated there are about 380,000 abandoned hardrock mine features on the land BLM manages that are not captured in its abandoned mine database. Park Service officials did not estimate a number of additional abandoned mines that might be in Park Service units; they said they believe their database is relatively comprehensive. Given the Forest Service and BLM estimates of additional features not found in their databases, the total number of estimated and identified abandoned hardrock mine features on lands within Forest Service, BLM, Park Service, and EPA jurisdiction is at least 533,652. Figure 2 depicts federal agency information about the numbers of confirmed and unconfirmed physical safety and environmental hazards on the lands under these agencies’ jurisdictions, in relation to the total estimated abandoned hardrock mine features, as of May 2019. To develop more comprehensive information about the total number of abandoned hardrock mine features on the lands they manage, the Forest Service and BLM are taking steps to improve their databases, including capturing information about abandoned mines that are not currently in a database. Specifically, Forest Service officials told us that they are establishing a centralized geospatial database that will consolidate information about abandoned hardrock mine features with physical safety hazards that is currently maintained in regional and national forest offices. They said they expect the new database will be populated in fiscal year 2020 and that it will provide regional and headquarters managers with better information about the extent of features with physical safety hazards. In addition, BLM officials said that field staff have been identifying and adding new features each year to its database, prioritizing features located close to communities and recreational areas. BLM officials said that they plan to update the database and communicate this information to field staff in fiscal year 2020 to help ensure staff enter information about new features into the database consistently. Officials with the 13 western states that we reviewed identified about 246,000 abandoned hardrock mine features on the federal, state, and private lands within their state borders, as of May 2019. As with the federal agencies, officials with five of the 13 states provided information about total mine sites and not features; as a result, we counted the minimum of one feature for each reported mine site for the purpose of our analysis. Of the 246,000 total features in these states, state officials estimated that about 115,000 features pose a physical safety hazard and about 11,000 features pose environmental hazards. State officials said that many of the features identified in their databases were also likely to be found in the federal agencies’ databases, but the extent of overlap is unknown. Specifically, the state officials’ estimates include abandoned mine features on federal, state, and private land because states may work on abandoned hardrock mines on both federal and nonfederal lands. However, state officials are not always able to quantify the number of mine features on federal land versus private or state land. For example, some states’ inventories are based on information from maps and databases that did not always include details about land ownership boundaries, which are necessary to differentiate on what lands the features are located. In addition, in instances in which the states could identify the features that are on federal land, such as in Utah and Nevada, state officials did not know how many of those features were also captured in federal agency databases. Similar to the federal agencies, officials with the 13 states estimated that the actual number of abandoned hardrock mine features in their states is higher than the information contained in their databases. State officials noted that their inventories are incomplete, in part because they have not conducted comprehensive, on-the-ground work to identify all the abandoned mine features in their states. They primarily focus on addressing the hazards they have already identified. Nevertheless, state officials estimated that the number of abandoned hardrock mine features in the 13 states could total more than 620,000. For example, California officials we interviewed said field staff had identified more than 70,000 individual abandoned mine features in the state as of May 2019. However, based on information from topographic maps, they estimated that 274,000 total mine features exist statewide, with an undetermined number of physical safety and environmental hazards. The states’ estimates of abandoned hardrock mine features reflect the different ways states collect information about abandoned hardrock mines. For example, California and Nevada officials explained that they count each individual abandoned mine feature in their states, whereas Colorado and Utah officials said that they only collect information about potentially hazardous features. Colorado officials estimated that there are 23,000 potentially hazardous abandoned hardrock mine features in the state. However, if the state were to count all of the features in Colorado, including shallow prospecting pits that are unlikely to pose a physical safety hazard, the officials said the total estimate would be hundreds of thousands of mine features. In addition, some states, including Idaho and California, reported numbers of abandoned mine features that included non-hardrock mines, such as sand and gravel pits, because their abandoned mine programs address different types of abandoned mines. Federal agencies spent, on average, about $287 million annually identifying, cleaning up, and monitoring abandoned hardrock mines, for a total of about $2.9 billion, from fiscal years 2008 through 2017. The Forest Service, BLM, the Park Service, EPA, and OSMRE primarily worked in partnership with other federal and state agencies and some nongovernmental stakeholders when addressing these mines, according to federal officials. Officials from the 13 western states we reviewed estimated spending an additional total of about $117 million in nonfederal funds over the 10-year period, or an average of nearly $12 million annually, to address abandoned hardrock mines within their states. Federal agency officials said they estimated it would cost billions more to address abandoned hardrock mines in the future. Federal agencies spent, on average, about $287 million annually, or a total of about $2.9 billion, to identify, clean up, and monitor hazards at abandoned hardrock mines from fiscal years 2008 through 2017. (See fig. 3.) EPA spent 80 percent of the total federal expenditures—about $2.3 billion—to address environmental hazards. Of the $2.9 billion in total federal expenditures, approximately $1 billion was reimbursed by responsible parties. Appendix II contains additional information about Forest Service, BLM, Park Service, EPA, and OSMRE expenditures by state. The agencies used some expenditures to address physical safety hazards but used most to address environmental hazards at abandoned hardrock mines. Physical safety hazards. The Forest Service, BLM, and the Park Service spent a total of over $105 million from fiscal years 2008 through 2017 to address mine features that posed physical safety hazards. According to officials with these agencies, this included filling in holes and installing gates at tunnels and other mine openings to allow bats, tortoises, and other wildlife to continue accessing important habitat. (See fig. 4.) Officials also said that their expenditures include funds provided to state agencies and others through cooperative funding agreements for projects where the state or other entity managed the work at the sites. Environmental hazards. From fiscal years 2008 through 2017, the Forest Service, BLM, the Park Service, and EPA spent a total of about $2.5 billion to address environmental hazards at abandoned hardrock mines. According to agency officials, work at these sites included conducting initial site investigations, designing and implementing remedies to address contamination, operating water treatment facilities, and monitoring completed cleanup actions. The agencies either managed this work themselves or provided funding through cooperative agreements to state agencies or others to manage the work. EPA spent about $2.3 billion at 394 sites, with about 40 percent spent at five sites. Of EPA’s total expenditures, $983 million (43 percent) was reimbursed by responsible parties. In addition, the Forest Service, BLM, and the Park Service spent a total of about $232 million to address various environmental hazards on lands they manage, of which about $40 million was reimbursed by responsible parties. Further, OSMRE reported that 12 states and two Indian tribes spent approximately $190 million in OSMRE grants to address abandoned hardrock mines and other non-coal sites from fiscal years 2008 through 2017. OSMRE officials did not specify how much of the $190 million was spent to address physical safety hazards versus environmental hazards since the agency does not require states and tribes to report such information. Table 2 shows federal agency expenditures by agency and type of hazard. Forest Service, BLM, and Park Service officials we interviewed said they conducted most of their work to address physical safety and environmental hazards at abandoned hardrock mines in collaboration with state agencies, nonfederal stakeholders, and other federal agencies, including EPA. These officials noted that it is important to partner with state agencies and EPA because many of the abandoned mine sites are of mixed ownership and the federal land management agencies generally do not have authority to address mine features on nonfederal lands. Federal agency officials said it is also helpful to pursue partnerships at mixed ownership sites to leverage limited funding. For example, Forest Service and BLM officials told us that they have partnered with Trout Unlimited, a nongovernmental organization focused on conserving freshwater fisheries and their watersheds, on projects to address environmental hazards at mixed ownership abandoned hardrock mine sites in several western states. Examples of projects that federal agencies undertook with partners include: Flat Creek-Iron Mountain Mine and Mill, Montana. Since 2014, the Forest Service has coordinated with EPA and the state of Montana to address contamination from this abandoned mine and mill site on private and Forest Service-managed lands upstream from the town of Superior. Silver, lead, and other hardrock mining operations left mill tailings piles that contaminated soil, groundwater, and surface water in Flat Creek, which flows for 3.5 miles from the mine site through Forest Service and private lands into the town. The local government and individuals also used tailings as fill material in yards, roadways, and other locations, including the high school track. The Forest Service took the lead on the portion of the site on the land it manages, and EPA and the state took the lead on various nonfederal portions of the site. At the state’s request, in 2000, EPA started assessing and cleaning up 79 residential and community properties in Superior; it completed this effort in 2013. In 2017, the state removed mine tailings from the private lands along Flat Creek with EPA oversight. As of November 2019, the Forest Service has been working with Trout Unlimited and the state to remove the mine tailings from the banks of Flat Creek on Forest Service land. Trout Unlimited representatives and Forest Service officials said they are also planning to reconstruct the stream channel and floodplains and restore fisheries habitat in the summer of 2020 after the tailings are removed. Gold Butte National Monument, Nevada. In 2018, BLM and the Nevada Division of Minerals worked with other federal, state, and local agencies to address 40 features that posed physical safety hazards within the historic Gold Butte Mining District in southern Nevada. The abandoned mine features were within the BLM- managed Gold Butte National Monument, which was established in 2016. According to project documents, the anticipated increase in recreation as a result of the monument designation prompted BLM and the state to evaluate the area for potential physical safety hazards. The 40 abandoned mine features included horizontal mine tunnel openings and deep vertical openings. BLM and the Nevada Division of Wildlife conducted cultural and wildlife surveys, respectively, to help determine appropriate closure methods. The state then filled the hazardous openings with foam and rock or installed gates that provide access to bats and desert tortoises. The local county government also contributed to the installation of the bat gates. Officials from the 13 states in our review estimated spending about $117 million in total, or an average of nearly $12 million annually, of nonfederal funds from fiscal years 2008 through 2017 to address physical safety and environmental hazards at abandoned hardrock mines within their states. Spending in three of the 13 states—California, Colorado, and Idaho—represented over 86 percent of the total nonfederal expenditures. Of the approximately $117 million, states spent about $26 million addressing physical safety hazards and about $91 million addressing environmental hazards. (See table 3.) State officials said that the sources of nonfederal funds that the states spent to address abandoned hardrock mines included (1) state-generated funds and (2) funding from settlements with responsible parties. State-generated funds. Officials from eight of the 13 states reported that they expended revenue raised by the state government to work on abandoned hardrock mines. Revenue sources include mine license taxes and royalties on oil and gas, hardrock mines, and other mineral extraction, and other sources such as the state general fund. For example, officials from the California Department of Conservation said the agency spent funds generated by state fees on active gold and silver operations to address physical safety hazards at abandoned mines on public lands. In addition, Colorado officials said they spent funds from a state severance tax collected on oil and gas, coal, metallic minerals, and other mineral production to address physical safety and environmental hazards. Settlements with responsible parties. Officials from five of the 13 states reported that they spent funds received from settlements with responsible parties to either conduct cleanup actions or oversee the responsible parties’ work to address environmental hazards. For example, from fiscal years 2008 through 2017, the state of New Mexico spent over $3.8 million that it had collected from responsible parties at two abandoned hardrock mine sites, according to state documents. Nevada and Washington officials said that their agencies’ expenditures to address environmental hazards during the 10-year period were entirely funded by collections from responsible parties. State officials we interviewed said they spent these nonfederal funds to address abandoned hardrock mines located primarily on private, county, state, or other nonfederal lands, including at mixed ownership sites. Officials from two of the 13 states (Colorado and Nevada) said they also spent state-generated funding to address hazards on federal land. Officials from the Nevada Division of Minerals’ abandoned mine program said that they generally spend about 80 to 90 percent of the program’s nonfederal funding addressing physical safety hazards on federal land. These officials explained that fees from unpatented mining claims on federal land are the division’s main funding source and, therefore, the state spends most of this funding to address hazards on federal land. Officials with the 13 states also told us that, in addition to spending about $117 million in nonfederal funds over the 10 years, states also spent more than $440 million they received from federal agencies, primarily through grants and cooperative agreements, during this period. Officials with seven states reported that they receive significantly more federal funds than nonfederal funds to work on abandoned hardrock mines and that federal funding is critical to addressing hazards at these mines. The Forest Service, BLM, the Park Service, and EPA estimated that their future costs to inventory and address physical safety and environmental hazards at abandoned hardrock mines would be in the billions of dollars. Each agency has generated some information about estimated future costs using a variety of methods and covering a range of activities. Given the level of uncertainty associated with the estimates, they likely understate the amounts that will be needed to comprehensively inventory and address these hazards. The Forest Service and BLM estimated that it could cost over $650 million to finish inventorying abandoned hardrock mines on lands they manage. Specifically, Forest Service information indicated it could cost about $147 million to complete the agency’s inventory, which includes identifying potential environmental hazards at 15,247 sites as well as the locations and conditions at approximately 13,000 sites not currently captured in a database. In addition, BLM officials estimated that it would cost about $510 million to complete the agency’s inventory of abandoned hardrock mines. This estimate includes about $130 million to evaluate approximately 66,000 features identified as posing an unconfirmed physical safety or environmental hazard. It also includes another $380 million to confirm the locations and presence of hazards at the approximately 380,000 additional features that may be on BLM-managed land but are not in its database. The Park Service and EPA did not provide estimates for future inventory work. Park Service officials said they have not estimated costs for additional inventory work because they believe that their inventory is largely comprehensive. EPA officials explained that the agency does not manage lands so they do not work to identify the existence of contaminated abandoned mines. Rather, EPA relies on external sources, such as state agencies and local governments, to alert it of potentially contaminated sites on nonfederal lands that may need attention. BLM and the Park Service estimated it could cost nearly $5 billion to address the physical safety hazards at abandoned hardrock mines on the lands they manage, and the Forest Service has not estimated this amount. Specifically, BLM estimated it could cost about $4.7 billion to fill in, gate, or otherwise address the nearly 65,000 features it has identified with confirmed and unconfirmed physical safety hazards and the estimated 380,000 additional features that are not yet included in the agency’s database. Park Service officials said they estimated that it would cost about $86 million to address the physical safety hazards at the abandoned hardrock mines identified in the agency’s database. These officials said that they plan to revise this estimate once they have better information about the actual costs to close the features where they are currently working. The Forest Service and EPA did not have estimates for addressing physical safety hazards. The Forest Service has not comprehensively estimated these costs, although the individual forests identify priority projects for spending each year, according to agency officials. EPA has not separately estimated costs to address physical safety hazards since those costs are included in its estimates to address environmental hazards. The Forest Service, BLM, the Park Service, and EPA have partly estimated costs to address environmental hazards at abandoned hardrock mines. Agency officials said that they do not have comprehensive estimates, in part because they have not yet selected the cleanup remedy at numerous sites—information they need to develop detailed estimates—nor have they identified all of the contaminated sites that will need to be addressed. The officials explained that a remedy to address an abandoned mine site with one waste rock pile (e.g., removing the pile from a creek and constructing a repository for it) is different from a remedy needed to address a site with perpetually draining mine tunnels, which could include operating and maintaining water treatment systems over the long term. As a result, the costs of cleanup remedies can vary from hundreds of thousands to hundreds of millions of dollars per site. Estimates of future costs to address environmental hazards at abandoned hardrock mines and what the estimates included varied by agency: Forest Service. Forest Service and USDA officials said that they estimated in 2014 that it could cost about $6 billion to address environmental hazards at 6,600 abandoned hardrock mine sites on Forest Service-managed land. This estimate includes costs to assess the extent of contamination, search for responsible parties, design and implement an action to remove a small waste rock or tailings pile, and monitor and maintain each site for 30 years after the cleanup is complete. According to the estimate, costs to maintain the completed sites make up half of the $6 billion in estimated future costs. These officials also said they assumed that all 6,600 sites are relatively simple and not complex with more extensive contamination. In developing this estimate, the Forest Service did not assume that responsible parties would cover any of these costs. BLM. BLM estimated a portion of the costs associated with addressing environmental hazards at abandoned hardrock mines on BLM-managed land, since BLM officials said there are too many unknowns and unique circumstances at each feature to comprehensively estimate total costs. These officials said the agency has estimated costs for some sites with confirmed environmental hazards in accordance with Interior’s environmental liabilities reporting guidance. Specifically, as of June 2019, BLM estimated that future costs to address environmental hazards at 105 abandoned hardrock mine sites on BLM-managed land range from $61 million to about $265 million. Interior and BLM officials explained that these costs do not represent all future costs needed to clean up these sites. Instead, the range includes the future costs that the agency determines are reasonably estimable at the time for these sites. In some cases, these costs are limited to the cost of conducting a study if the agency has not selected a cleanup remedy. As a result, officials said they expect that BLM’s estimate of total future costs will increase once the agency selects the cleanup remedies and estimates their costs. Officials also said they have not estimated future costs for sites where the agency has not determined the type or extent of the contamination or where BLM is not likely to fund the cleanup, for example, because a responsible party may pay for it. Park Service. Similar to BLM, Park Service officials estimated the future costs associated with addressing environmental hazards at 50 contaminated abandoned hardrock mines, based on Interior’s guidance. As of June 2019, the Park Service estimated that these future costs range from $21 million to $35 million, exclusive of any reimbursements from responsible parties. The Park Service did not estimate the future costs to address 19 additional sites that the agency identified as posing environmental hazards because either work at these sites is in the early stages, the agency was unable to estimate costs, or the Park Service is not likely to fund the cleanup, according to Park Service and Interior officials. EPA. EPA officials told us that they do not have a comprehensive estimate of costs to clean up hardrock mines. Specifically, officials said EPA tracks planned obligations to be incurred for sites where the agency anticipates taking action within the next 3 years to help support its budget development process. As of fiscal year 2018, EPA identified about $519 million in planned obligations for 115 hardrock mine or mineral processing sites. EPA officials said the planned obligations do not necessarily reflect the total estimated costs remaining at a site because the agency typically requires its regions to report known planned obligations for 3 years, or longer, if available. According to EPA data, future costs to address hardrock mines likely will exceed these obligations. For example, EPA did not report planned obligations for 423 mine and mineral processing sites where the agency has not completed site assessment work or selected a cleanup remedy. According to EPA officials, they generally do not plan obligations for future cleanup work while conducting an assessment. However, they said that if an assessment reveals a need for a time-sensitive response at a site, the agency may fund it. EPA officials also told us that they expect responsible parties to pay a portion of the future costs associated with these sites, but that amount is unknown. Federal agency officials, state officials from three selected states (Colorado, Montana, and Nevada), and stakeholders cited availability of resources and legal liability concerns as factors that limit efforts to identify, clean up, and monitor hazards at abandoned hardrock mines. Federal and state officials said their backlog of work on abandoned mines is greater than current staff and budget levels. In addition, state agency officials and other stakeholders we interviewed, such as nongovernmental organizations and mining companies, have limited their participation in projects to address environmental hazards at abandoned mines because of concerns about their potential legal liability under CERCLA and the Clean Water Act. All of the officials we interviewed from the Forest Service, BLM, the Park Service, and EPA, as well as from Colorado, Montana, and Nevada, cited availability of resources as a factor that limits their efforts to identify and address the physical safety and environmental hazards at abandoned hardrock mines. Representatives from state associations and nongovernmental organizations we interviewed also cited this factor as limiting federal and state efforts. Federal and state officials said that their backlog of work on these mines far exceeds their current staff and budget levels. For example, BLM officials estimated that with the agency’s current abandoned mine budget and staff resources, it could take up to 500 years to confirm the presence of physical safety or environmental hazards at the approximately 66,000 features in its database and the estimated 380,000 features not yet captured in its database. Officials from Colorado and Montana and representatives from a state association noted that these two states regularly receive reclamation funding from OSMRE to address abandoned coal mines in their states. As a result of having access to such funds, five states, including Montana and Wyoming, as well as three tribes have certified that they have addressed all of their known priority abandoned coal mines. These officials also noted that there is not a similar or consistent source of funding for states to address hazards at abandoned hardrock mines. In Nevada, although state-collected mining fees contribute to addressing safety hazards at abandoned hardrock mines, state officials said they do not have a consistent source of funding to address environmental hazards. As a result, Nevada officials explained that they tend to work primarily on mines where there is a viable responsible party to fund the cleanup. However, one official said that most of the approximately 190 abandoned hardrock mine sites in the state that pose or may pose environmental hazards do not have a viable responsible party. Federal and state agency officials described several steps they have taken to work more efficiently within existing limited resources. For example, federal agency officials said they prioritize proposed projects to address abandoned mines that pose the highest safety and environmental risks. In addition, federal officials explained that they have established several formal mechanisms for national and local collaboration to facilitate leveraging resources. For instance, federal and state officials working in Colorado said they formed a working group in 2010 to jointly identify and prioritize watersheds that have been contaminated by abandoned hardrock mines. The agencies work collaboratively to evaluate the extent of contamination in each watershed, leading to a more holistic approach to addressing contamination, according to EPA and Colorado state officials. Regional Forest Service officials we interviewed who also work outside of Colorado said the group is a national model for collaboration and efficient use of resources. Forest Service, BLM, Park Service, EPA, and state officials also said that they work to leverage federal and state resources by searching for responsible parties to contribute funding to their efforts at abandoned hardrock mines. However, officials told us that identifying such parties is difficult and can be resource intensive given the length of time that has elapsed since the mines were abandoned and the lack of a clear chain of custody and land ownership boundaries at mine sites. All of the state officials and nearly all of the stakeholders from nongovernmental organizations, state associations, and industry we interviewed cited concerns over legal liability—that is, being held legally responsible for addressing environmental contamination—as a factor that limits efforts to address certain abandoned hardrock mine hazards on nonfederal land. Specifically, liability concerns can prevent third parties— entities who offer assistance in addressing environmental hazards that they did not create and are not legally required to clean up—from taking actions to help address such hazards that are on private land and on nonfederal portions of mixed ownership sites. These parties are often referred to as Good Samaritans and may include state agencies, nongovernmental organizations, local governments, private landowners, and mining companies, among others. Federal and state officials and stakeholders we interviewed said that Good Samaritans have avoided taking certain cleanup actions—in particular, addressing mine tunnels that perpetually drain highly contaminated water—at abandoned hardrock mines because they are concerned about potentially being held legally responsible under CERCLA and the Clean Water Act. Specifically, a Good Samaritan undertaking cleanup actions at an abandoned hardrock mine might become a responsible party under CERCLA and thereby would be responsible for the entire cost of cleaning up the site. As a result, representatives from an industry association and a nongovernmental organization told us that while they are interested in addressing contamination on private land in the West, they generally have not done so, in part because of concerns about becoming responsible under CERCLA for cleaning up all of the contamination present at the site. In addition, a Good Samaritan undertaking cleanup actions to address draining mine tunnels may be required to do so in accordance with a discharge permit under the Clean Water Act. Complying with such a permit requires that the cleanup meet and maintain water quality standards, which can be expensive and may require perpetual water treatment. State officials and stakeholders explained that meeting and maintaining such standards at certain mines is difficult because of naturally occurring heavy metals and continual drainage from the mines. They said they are interested in undertaking smaller-scale projects to address mine tunnel drainage that may significantly improve water quality and aquatic habitat but would not fully meet water quality standards. However, Colorado and Montana state officials and various stakeholders said they generally decide not to undertake such projects, even if they could make incremental improvements, because of the risk of being held responsible for meeting and maintaining water quality standards in perpetuity. To encourage nongovernmental organizations, other stakeholders, and states to participate in abandoned hardrock mine projects at mixed ownership sites and on other private land, EPA developed administrative tools aimed at limiting Good Samaritans’ CERCLA and Clean Water Act liability. In 2007, EPA developed guidance for issuing “comfort/status letters” to Good Samaritans willing to perform cleanup work under EPA oversight and for entering into settlement agreements—legally enforceable documents signed by EPA and a Good Samaritan that include a federal covenant not to sue under CERCLA in exchange for cleanup work. In 2012, EPA also issued guidance stating that, as a general matter, the agency would not require a Good Samaritan to obtain a Clean Water Act discharge permit if they successfully complete a cleanup action under a comfort/status letter or settlement agreement with EPA. Good Samaritans have participated in some projects at abandoned hardrock mines using EPA’s administrative tools. As of January 2020, EPA had issued four comfort letters and entered into three settlement agreements, generally to address hazards at sites that did not require a Clean Water Act permit. Some state officials and stakeholders we interviewed said they have not pursued using EPA’s administrative tools because, in part, these tools do not sufficiently alleviate liability under the Clean Water Act. For example, they explained that the tools and guidance provide reassurance that EPA may not sue the Good Samaritan but do not ensure that certain outside parties will not sue to require they meet water quality standards. State officials and stakeholders we spoke with said that they believe that resolving the concerns over CERCLA and Clean Water Act liability may require federal legislation. However, other stakeholders expressed concerns that legislative changes, such as amending CERCLA or the Clean Water Act, could inadvertently result in weakening the existing environmental protections in these and other laws or could limit the ability of outside parties to enforce their provisions. Since 1999, several bills have been introduced that would have responded to liability concerns but as of December 2019, none had been enacted. State officials and stakeholders have been involved in efforts to draft legislation that would address liability concerns, but the interested parties have disagreed about the specific provisions to include. While federal agency officials did not cite liability concerns as a factor that limits their agencies’ efforts to address abandoned hardrock mines on lands under their jurisdictions, Forest Service, BLM, and EPA officials concurred that legal liability concerns deter Good Samaritans from participating in projects with federal agencies at mixed ownership sites. Federal officials explained that, unlike Good Samaritans, the abandoned hardrock mines the federal agencies address are under their jurisdiction and the agencies are already responsible for meeting the requirements of CERCLA and other applicable laws. However, federal agency officials have observed the effects of Good Samaritan legal liability concerns on projects. For example, Forest Service officials in Colorado said that potential partners have expressed interest in addressing contamination on the private land portions of mixed ownership sites but declined once they learned they would be subject to liability under CERCLA. In the absence of legislative changes, EPA officials said they are looking for new ways to encourage Good Samaritan participation in abandoned hardrock mine projects. For example, they are working to update and refine the agency’s administrative tools and identify new solutions to better address Good Samaritans’ concerns. They are also looking to encourage Good Samaritan participation in more projects that would not require a Clean Water Act permit, such as moving mine tailings piles away from streams. We provided a draft of this report to the Department of Agriculture, the Department of the Interior, and EPA for their review and comment. The Forest Service Audit Liaison provided comments by email, stating that the Forest Service generally agreed with the report. USDA and EPA provided technical comments, which we incorporated as appropriate. Interior told us they had no comments. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Secretaries of Agriculture and the Interior, the Administrator of EPA, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or fennella@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report describes (1) what is known about the number of abandoned hardrock mines in the United States; (2) federal and state agency expenditures to address abandoned hardrock mines from fiscal years 2008 through 2017, and what is known about future costs to address these mines; and (3) factors that limit federal and state agencies’ and stakeholders’ efforts to address abandoned hardrock mines. To address these objectives, we reviewed our previous work on abandoned hardrock mines, including a March 2008 report in which we summarized information about the number of abandoned hardrock mines in the United States and the amount of federal spending on these mines from fiscal years 1998 through 2007. We also reviewed federal agency reports to identify the federal agencies that track numbers of abandoned hardrock mines, conduct work to address hazards at these mines, or fund projects to address these hazards. We identified the U.S. Department of Agriculture’s (USDA) Forest Service; the Department of the Interior’s Bureau of Land Management (BLM), National Park Service (Park Service), and Office of Surface Mining Reclamation and Enforcement (OSMRE); and the Environmental Protection Agency (EPA) to include in our review. We reviewed agency documents detailing these agencies’ cleanup efforts and abandoned hardrock mine programs. We also selected 13 western states to include in our review: Alaska, Arizona, California, Colorado, Idaho, Montana, Nevada, New Mexico, Oregon, South Dakota, Utah, Washington, and Wyoming. We selected these states because our March 2008 report and other federal and state agency reports indicated that most of the abandoned hardrock mines are in these states. We conducted two site visits to abandoned hardrock mines in Colorado in February 2019. We selected sites in Colorado because they provided opportunities to observe examples of physical safety and environmental hazards on federal and nonfederal lands. We visited sites with physical safety hazards that BLM and the state had addressed on BLM and county lands. We also visited a National Priorities List site where EPA and the state were addressing environmental hazards on private land. To describe what is known about the number of abandoned hardrock mines in the United States, we obtained and summarized information about abandoned hardrock mine features and sites—including the number of features and sites that pose confirmed and unconfirmed physical safety and environmental hazards—that the Forest Service, BLM, the Park Service, and EPA maintained in databases as of May 2019, the most current at the time of our review. Specifically: the Forest Service provided information about abandoned hardrock mine sites from USDA’s National Environmental Accomplishment Tracking system; BLM provided information about abandoned hardrock mine features from the Abandoned Mines and Site Cleanup Module; the Park Service provided information about abandoned hardrock mine sites and features from the Abandoned Mineral Lands Data Entry and Edit database and from Interior’s Environmental and Disposal Liabilities list; and EPA provided information about hardrock mining and mineral processing sites from its Superfund Enterprise Management System. In addition, we obtained information on the agencies’ estimates of the number of additional abandoned hardrock mine sites or features that are not captured in their databases, where applicable. We assessed the reliability of the agencies’ databases by testing the data for accuracy by cross-referencing with relevant data sets and checking for missing data and errors. We also reviewed agency documents about the databases and our previous related work regarding the use of these data. We also interviewed headquarters officials from each agency and discussed the data and any limitations. We determined that the information in the agencies’ databases about the number of abandoned hardrock mines was sufficiently reliable to summarize in our report. We calculated the agencies’ total number of abandoned hardrock mines in terms of the number of features. According to agency officials, many abandoned hardrock mine sites contain more than one feature, but there is no agreed-upon average number of features per site. Since the Forest Service and EPA reported information only by mine site, we counted the minimum of one feature per site in our calculations. As a result, the total number of features likely is underestimated. Further, we collected information about the number of abandoned hardrock mines in the 13 western states through semi-structured interviews with state officials. For each state, we interviewed officials with the relevant state agencies that address abandoned hardrock mines through, for example, a dedicated abandoned mine program or a broader program focused on addressing environmental hazards. In each interview, we asked the officials to provide information about the numbers of abandoned hardrock mine sites they identified in their state, features that posed a hazard to public health and safety, and features that caused environmental degradation as of the time of our review. We provided the states with a common definition of abandoned mine site and feature. However, officials with five states provided information only for abandoned mine sites and not features. For those states, we counted the minimum of one feature per site to calculate the states’ total number of abandoned hardrock mine features. As a result, the states’ total number of features likely is underestimated. We assessed the reliability of the states’ information by reviewing documents about the data systems, checking for missing data and errors, and discussing the data and their sources with state officials, including any limitations. We determined that the data were sufficiently reliable to describe what the state agencies know about abandoned hardrock mines within their jurisdictions. To describe federal agency expenditures to address abandoned hardrock mines from fiscal years 2008 through 2017, we summarized expenditure information from the Forest Service, BLM, the Park Service, EPA, and OSMRE for this time period, the most recent 10 years of information available at the time of our review. Specifically, we collected information about total expenditures to address abandoned hardrock mines, expenditures to address physical safety hazards, expenditures to address environmental hazards, and expenditures of collections from responsible parties under the Comprehensive Environmental Response, Compensation, and Liability Act of 1980 (CERCLA), as applicable. We assessed the reliability of the agencies’ information by testing the data for accuracy and completeness by checking for missing data and errors. We also reviewed our previous related work regarding the use of the information and interviewed agency officials involved with collecting or analyzing the information. We determined that the information obtained from the agencies was sufficiently reliable for our descriptive purposes. Additional details on agency-specific information we used follows: Forest Service. The Forest Service provided expenditure information for fiscal years 2008 through 2017 for its Abandoned Mine Land and Environmental Compliance and Protection programs from its Foundation Financial Information System. The Forest Service also provided information from this system about expenditures of reimbursements from responsible parties. USDA provided information about the Forest Service’s expenditures from the department’s Hazardous Materials Management Account for fiscal years 2008 through 2017 from the Financial Management Modernization Initiative system. BLM. BLM provided expenditure information from Interior’s Financial Business Management System for fiscal years 2009 through 2017. BLM’s budget office provided expenditure information for fiscal year 2008 since information prior to fiscal year 2009 is not included in Interior’s current financial system. BLM provided information about abandoned hardrock mine expenditures from relevant subactivity codes, including Abandoned Mine Lands, Hazardous Materials Management, American Recovery and Reinvestment Act-Abandoned Mine Land projects, and Central Hazardous Materials Fund, among others. Park Service. The Park Service provided expenditure information from Interior’s Financial Business Management System and the Park Service’s Project Management Information System and Administrative Financial Systems 3 and 4 for fiscal years 2008 through 2017 for its Abandoned Mine Lands program and the Contaminants Cleanup Branch. The Park Service also provided information from Interior’s system about expenditures of reimbursements from responsible parties. OSMRE. OSMRE provided expenditure information from Interior’s Financial Business Management System for fiscal years 2008 through 2017 from its non-coal account, which includes spending for projects to address abandoned hardrock mines, non-hardrock abandoned mines, and other eligible projects. To further narrow the non-coal account expenditures to spending on abandoned hardrock mines, we reviewed information for projects that states completed during the 10-year period and eliminated expenditures that were clearly identified for non-hardrock-related projects. We also compared the expenditure information from OSMRE with expenditure information we obtained during our semi-structured interviews with officials from six state agencies that reported spending OSMRE grants specifically on hardrock abandoned mines—Alaska, Colorado, New Mexico, Montana, Utah, and Wyoming. We determined that Alaska’s and Colorado’s reported expenditures were more specific to abandoned hardrock mines than the information OSMRE provided for those states. As a result, we used Alaska’s and Colorado’s information to report expenditures for those states and used OSMRE’s information to report expenditures for all other states. OSMRE officials agreed with this approach. EPA. EPA provided information about the Superfund program’s expenditures at mine and mineral processing sites from the Integrated Financial Management System for fiscal years 2008 through 2011 and the Compass Financial System for fiscal years 2012 through 2017. EPA provided expenditures from its (1) Superfund appropriation accounts, (2) special accounts through which EPA receives resources from settlements with responsible parties for EPA to conduct site- specific work, and (3) state cost-share accounts, through which states contribute 10 percent of costs for EPA’s Superfund-financed remedial actions. EPA also reported expenditures of funds provided by other federal agencies; we excluded these expenditures from our reporting of EPA’s spending to avoid potential double counting. Further, we obtained information through our semi-structured interviews with officials from the 13 selected states about their expenditures of nonfederal and federal funds at abandoned hardrock mines for state fiscal years 2008 through 2017. We obtained and summarized information on total expenditures to address abandoned hardrock mines, expenditures to address physical safety hazards, and expenditures to address environmental degradation. We also obtained information about the sources of the agencies’ funding, such as collections from responsible parties. The states provided expenditure information by state fiscal year and not federal fiscal year because their financial systems are organized by state fiscal year. We assessed the reliability of the states’ expenditure information by testing for missing data and errors, reviewing documents, and discussing the information and any limitations with state agency officials. Three states were unable to provide expenditure information specific to abandoned hardrock mines for the entire 10-year period. Therefore, we discussed and agreed with each of these states how they could provide information that most closely responded to our request—for example, by providing information for the years that were available—and we are reporting the state agencies’ total expenditures as estimates. We determined that the data were sufficiently reliable to describe an estimate of how much in nonfederal and federal funds the state agencies spent to address abandoned hardrock mines. We are reporting both federal and state agency expenditures in nominal dollars. We are doing so for several reasons, including that there was a relatively low rate of inflation from fiscal year 2008 through 2017 (about 1.5 percent per year, on average); not all states reported annual expenditures that could be adjusted for inflation; and federal and state agencies reported annual expenditures differently, with federal agencies reporting by federal fiscal year and state agencies reporting by state fiscal year. To describe what is known about future costs to address abandoned hardrock mines, we reviewed and summarized documentation of the federal agencies’ most recently available estimates of costs to inventory additional abandoned hardrock mine features and to address physical safety and environmental hazards that have not been addressed. We discussed these estimates, and the assumptions used to create the estimates, with relevant agency officials. We describe the estimates and their underlying assumptions in the report. To identify factors that limit federal and state agencies’ and stakeholders’ efforts to address abandoned hardrock mines, we reviewed relevant agency documents and independent reports that describe limiting factors. We interviewed federal agency officials, state agency officials, and stakeholders. More specifically, we interviewed Forest Service, BLM, Park Service, EPA, OSMRE, and Interior headquarters officials and officials from these agencies’ regional or state-based offices who work in Colorado, Montana, and Nevada. We also interviewed officials with the relevant state agencies that address abandoned hardrock mines in these three states. We selected these states for geographic diversity, higher numbers of abandoned hardrock mines, and variation in the types of hazards posed by abandoned hardrock mines in these states. The sample of states is not generalizable, and the results of our work do not apply to all states where abandoned hardrock mines are located, but provide illustrative examples. In addition, we obtained perspectives from stakeholders that have participated in or expressed interest in participating in projects to address abandoned hardrock mines. We interviewed a sample of stakeholders, selected to provide perspectives from industry associations, nongovernmental organizations, state agency associations, and individuals with long-standing involvement in issues related to addressing abandoned hardrock mines. We identified and selected these stakeholders based on our previous work, including the stakeholders we interviewed for our March 2008 report; a review of relevant literature, including written testimony statements and a summary of proceedings from a 2018 conference on abandoned hardrock mines; interviews with federal and state agency officials; and recommendations from stakeholders. Our sample of stakeholders is not generalizable to all stakeholders involved with abandoned hardrock mines, but provides perspectives on factors that limit efforts to address abandoned hardrock mines. In total, we obtained responses from officials with 13 federal agency offices, including six headquarters offices and seven regional or state- based offices; officials with three states; and representatives of 11 stakeholder organizations, including three state associations that represent states with abandoned mine programs, two nonprofit conservation organizations, two mining industry associations, one mining company, and three individuals with long-standing involvement in abandoned hardrock mine policy. In our discussions, officials and representatives with each entity identified the factors that limit their or others’ efforts to address abandoned hardrock mines. We reviewed the responses and identified the factors that officials and stakeholders in each group (i.e., federal agencies, state agencies, and stakeholders) frequently mentioned. Two factors arose frequently both within and across the groups—we describe these factors in our report. We conducted this performance audit from June 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings based on our audit objectives. Table 4 includes expenditures to address abandoned hardrock mines for the Bureau of Land Management, Environmental Protection Agency, Forest Service, Office of Surface Mining Reclamation and Enforcement, and National Park Service. In addition to the contact named above, Elizabeth Erdmann (Assistant Director), Leslie Kaas Pollock (Analyst-in-Charge), Matthew Elmer, William Gerard, Anne Rhodes-Kline, Sheryl Stein, Sara Sullivan, and Rajneesh Verma made key contributions to this report.", "summary": "The General Mining Act of 1872 allowed individuals to obtain exclusive rights to valuable hardrock mineral deposits on land belonging to the United States. Miners explored, mined, and processed valuable minerals, but many did not reclaim the land after their operations ended. Unsecured mine tunnels, toxic waste piles, and other hazards—known as mine features—are found at abandoned hardrock mines across federal and nonfederal lands. The Forest Service, BLM, National Park Service, EPA, and OSMRE—as well as state agencies—administer programs that identify and address hazardous features at abandoned hardrock mines. Addressing features could include, for example, sealing mine tunnels or treating contaminated water. GAO was asked to provide information about abandoned hardrock mines. This report describes (1) what is known about the number of abandoned hardrock mines in the United States; (2) agency spending to address abandoned hardrock mines from fiscal years 2008 through 2017 and estimated future costs; and (3) factors that limit federal and state agencies' and stakeholders' efforts to address abandoned mines. GAO obtained and summarized information from agency databases about the number of abandoned mines, features, and hazards as of 2019; summarized agency spending data from fiscal years 2008 through 2017, the most currently available; and interviewed federal and state agency officials and stakeholders, selected to provide diverse perspectives. The U.S. Department of Agriculture's Forest Service, the Department of the Interior's Bureau of Land Management (BLM) and National Park Service, and the Environmental Protection Agency (EPA) identified at least 140,000 abandoned hardrock mine features, such as a tunnel, on lands under their jurisdictions. Of these, about 67,000 pose or may pose physical safety hazards—danger of injury or death—and about 22,500 pose or may pose environmental hazards—risks to human health or wildlife from long-term exposure to harmful substances. Agency officials also estimated there could be more than 390,000 abandoned hardrock mine features on federal land they have not captured in their databases, and agencies are developing more comprehensive information about these mines. Forest Service, BLM, National Park Service, EPA, and Interior's Office of Surface Mining Reclamation and Enforcement (OSMRE) spent, on average, about $287 million annually to address physical safety and environmental hazards at abandoned hardrock mines from fiscal years 2008 through 2017, for a total of about $2.9 billion (see figure). Of this total, the agencies spent about 88 percent ($2.5 billion) addressing environmental hazards, and about $1 billion was reimbursed by private parties, such as former mine owners. Federal officials also estimated that it would cost billions more to address these mines in the future. Nearly all of the federal and state agency officials and stakeholders GAO interviewed cited availability of resources and legal liability concerns as factors that limit efforts to address hazards at abandoned hardrock mines. Federal and state officials said their backlog of work is greater than what can be done with available staff and budgets, but they have taken steps to collaborate to help leverage resources. State officials and stakeholders, such as conservation groups, said they want to help address environmental hazards that they did not cause at abandoned hardrock mines. However, they generally do not do so because they are concerned about becoming legally responsible for the entire cost of addressing contamination at an abandoned mine if they attempt partial cleanup. EPA officials said they are considering new ways to encourage volunteer participation, in addition to existing administrative tools.", "document_type": "gao"}
{"report": "Treasury borrows money by issuing Treasury securities to finance the federal deficit (i.e., the difference between current spending and revenues), which includes paying interest on outstanding debt, and refinancing maturing debt. According to Treasury’s Strategic Plan, the primary objective of its debt management strategy is to finance the government’s borrowing needs at the lowest cost over time. Treasury reports that it achieves this objective by issuing marketable debt with a regular and predictable framework— meaning Treasury debt managers provide the market clear and transparent information about planned issuance, and set a standard calendar of auctions of each security type. managing its debt portfolio to mitigate “rollover risk”—the risk that it may have to refinance its debt at higher interest rates; fostering a healthy and liquid secondary market—the marketplace in which Treasury securities are traded; and promoting a broad and diverse investor base. To this end, Treasury issues securities in a wide range of maturities to appeal to a broad range of investors, and in sufficient amounts to promote liquid markets so investors can easily buy and sell Treasury securities. Treasury’s regular and predictable auction framework also provides investors greater certainty and better information to plan their investments. Treasury regularly issues nominal securities that range in maturity from 4 weeks to 30 years, inflation protected securities with 5-, 10-, and 30-year maturities, and floating rate notes (see table 1). A nominal security returns the face value of the security at maturity; an inflation-indexed security repays the principal adjusted for inflation. Floating rate notes pay interest quarterly at a rate that varies with changes in the indexed rate, such as the discount rate on the 13-week Treasury bill. The interest rates associated with the range of maturities of the nominal securities issued by Treasury creates a “yield curve” which represents the relationship between the maturity of an asset and its yield (the interest rate paid by Treasury or cost of borrowing). Each security has different cost and risk features for Treasury. Generally, Treasury must pay a higher interest rate for longer-dated securities to compensate buyers for waiting longer for principal to be repaid and accepting increased risk due to uncertainty about future market conditions. But longer-dated securities offer more certainty for budget planning because they lock in interest rates for the duration of the security. Similarly, as Treasury offers more of any given security, it may have to pay more interest to attract investors. However, if Treasury offers too little of a specific security given changing market demand, it could reduce the security’s liquidity in the secondary market, which would increase the interest cost Treasury must pay to compensate investors for less liquidity. The mix of securities changes regularly as Treasury issues new debt and funding needs change. Figure 1 shows the outstanding marketable debt held by the public by security type between 2005 and 2019. Treasury typically responds to long-term increases in borrowing needs by taking the following steps: Increasing the amount of securities offered at scheduled auctions. In 2018, Treasury increased auction sizes for securities at all maturities as borrowing needs increased. For example, Treasury increased the average size of auctions for floating rate notes by 15 percent (from about $16.2 billion in 2017 to $18.6 billion in 2018) and 3-year notes by 32 percent (from about $25.9 to $34.1 billion). Increasing the frequency of scheduled auctions. For example, in 2003 and 2008, Treasury adjusted the auction calendar to include additional reopenings of 10-year notes. More recently, Treasury added an October 5-year TIPS issue, with the first auction held on October 17, 2019. Introducing new types of securities to offer at its auctions. For example, in 2014, Treasury introduced a 2-year floating rate note. In October 2018, Treasury began auctioning a 2-month bill. According to Treasury officials, the addition of the 2-month bill allowed Treasury to issue more bills without increasing auction sizes for existing bills beyond maximum sizes recommended by market participants. In taking these steps, Treasury announces expected auction sizes each quarter and publicly discusses the changes well in advance. Treasury securities are held by a wide range of investors for a variety of different reasons, including cash and liquidity management, collateral, hedging, speculation, arbitrage, and as long-term “buy and hold” investments. As shown in figure 2, these investors can be grouped into three categories: The Federal Reserve System (Federal Reserve), the U.S. central bank, conducts monetary policy to promote maximum employment, stable prices, and moderate long-term interest rates. As part of this role, the Federal Reserve banks may buy and sell Treasury and other securities in the secondary market and roll over holdings of Treasury securities at auction as a noncompetitive bidder. The Federal Reserve is the largest individual holder of Treasury securities, and as of June 2019, held approximately $2.3 trillion in Treasury securities— or 14 percent of marketable debt held by the public. International investors include both private investors and foreign official institutions, including central banks and government-owned investment funds. As of June 2019, foreign holdings represented 41 percent of marketable debt held by the public; about $6.6 trillion. Most foreign holdings are from official sources (63 percent according to available data), such as foreign central banks. Domestic investors include banks, investment funds, pension funds, insurance companies, state and local governments, and individuals. As of June 2019, domestic investors held 45 percent of marketable debt held by the public; more than $7 trillion. Figure 2 shows the sectors that represent the domestic investor category. The combination of the liquidity, depth, and safety of the Treasury market is unmatched in global markets. These characteristics make Treasury securities a unique and critical asset for a broad range of investors. Market participants and subject matter experts we interviewed and surveyed identified liquidity, depth, and safety as the most important characteristics of Treasury securities. As shown in figure 3, 63 of 67 market participants we surveyed from across 10 domestic sectors reported that liquidity is one of the most important characteristics, followed by depth and safety. Moreover, 55 of the 67 survey respondents cited at least two of these characteristics as the most important. Liquidity, depth, and safety are interrelated characteristics of Treasury securities (see fig. 4). For example, liquidity and depth are both related to the size of the market and the willingness of market participants to buy and sell securities at low cost. In addition, liquidity is enhanced by safety, for example by minimizing the risk that trading could be disrupted by default. Treasury securities are considered one of the safest assets in the world because they are backed by the full faith and credit of the U.S. government. The importance of these characteristics was consistent across sectors, as liquidity, depth, and safety support a variety of business practices and needs. For example, Treasury securities serve as a close substitute to cash for financial institutions and corporate treasurers, are one of the cheapest and one of the most widely used forms of collateral for financial transactions, and are a benchmark for pricing many other financial products, such as corporate bonds, derivatives, and mortgages. In addition, international investors and experts we interviewed said that both foreign official sector and foreign private sector investors value the liquidity, depth, and safety of the Treasury market. For example, foreign central banks value the ability to buy and sell large quantities of securities to assist in managing their exchange rates and, in times of economic stress, provide foreign currency credit to their country’s businesses that borrow or trade in U.S. dollars. Officials from a foreign central bank we spoke with told us that Treasury securities are well suited for their investment needs because of the combination of the large and deep market—which accommodates high-volume transactions—and their safety and liquidity. The combination of liquidity, depth, and safety supports reliable demand for Treasury securities through changing market conditions. A diverse investor base helps to protect Treasury from large swings in interest costs due to shifts in demand from particular sectors. After liquidity, depth, and safety, the fourth most cited characteristic of Treasury securities (25 of 67 survey respondents) was the ability to purchase across the yield curve—that is, purchasing securities of various maturities to match investment needs. In addition to issuing securities at various maturities, Treasury’s strategic plan includes a goal to develop new products to increase the investor base. As previously noted, Treasury began issuing 2-month bills in October 2018. Market participants we surveyed said there is potential demand for (1) a new nominal security; (2) expansion of the floating rate note offerings; and (3) a zero-coupon bond. (For more information on the survey results, see appendix II.) “An increase in global risk (political or economic) will determine flight to quality and higher allocation to Treasuries.” Many investors are willing to accept a lower yield on Treasury securities in exchange for the liquidity, depth, and safety they provide. For example, only 14 of the 67 market participants we surveyed cited the yield of Treasury securities as one of the top three characteristics. Market participants we surveyed and interviewed emphasized that there is no true substitute for Treasury securities because other assets come with additional risks or do not have the liquidity and depth of the Treasury market. As a result, in times of economic uncertainty or stress, investors often move quickly into Treasury securities—known as a “flight to quality”—which increases demand and drives down yields. While a broad and diverse investor base helps promote stability for the Treasury market as a whole, demand for Treasury securities by different types of investors fluctuates over time, reflecting changes in the investment needs of particular sectors. Since the 2007-2009 financial crisis, changes in monetary policy operations, financial regulation, and foreign central bank needs have changed the composition of demand for Treasury securities across different sectors. Figure 5 shows the overall changes in holdings of Treasury securities by the three primary investor groups—domestic investors, international investors, and the Federal Reserve. As part of its response to the 2007-2009 financial crisis, the Federal Reserve substantially increased its purchases of longer-term Treasury securities. In turn, these purchases substantially increased the overall size and duration of the Federal Reserve’s holdings of Treasury securities (see fig. 6). From 2008 to 2014, its holdings of Treasury securities increased by 475 percent; from roughly $480 billion in 2008 to $2.7 trillion in 2014. The average duration of the holdings also increased from 2.7 years in 2007 to a high of 7.8 years in 2013. This substantial shift in the size and composition of the Federal Reserve’s holdings began in late 2008 when the Federal Reserve undertook the first of a series of large-scale asset purchase programs, often referred to as quantitative easing, to better reduce long-term interest rates and improve economic conditions. The Federal Reserve’s purchases of long-dated Treasury securities, and other assets, substantially increased the size of its balance sheet and meaningfully reduced interest rates on long-term Treasury securities.One study estimated that quantitative easing reduced interest rates on 10-year Treasury securities as much as 160 basis points (or 1.6 percentage points) (see sidebar). Federal Funds Rate A market determined interest rate that banks charge each other to borrow reserves overnight. The Federal Reserve needed a new approach to managing short-term interest rates while maintaining a large balance sheet. Therefore, in 2014, the Federal Reserve outlined a new framework it intended to adopt for implementing monetary policy when it began to increase interest rates for the first time since the financial crisis. The new operating framework entails setting two short-term interest rates to manage the federal funds rate (see sidebar). Changes in these rates are intended to influence other short-term interest rates (including rates on Treasury securities), the availability of credit, and the economy as a whole to assist the Federal Reserve in achieving its monetary policy objectives. In response to the improving economy the Federal Reserve, in October 2017, began a process to slowly shrink its balance sheet by limiting the reinvestment of proceeds from maturing securities, intending to return to a smaller balance sheet and lower holdings of Treasury securities. In January 2019, however, the Federal Reserve announced that it intended to continue to operate with its post-crisis framework and would therefore evaluate the appropriate time to stop shrinking its balance sheet. In October 2019, the Federal Reserve announced that it would expand its balance sheet, through purchases of Treasury bills, to satisfy increases in the market’s demand for cash and keep the federal funds rate in its target range. As a result of these announcements, the Federal Reserve will continue to hold a much larger portfolio of Treasury securities and will therefore continue to purchase much larger quantities of Treasury securities on an ongoing basis. If economic and financial conditions warrant, the Federal Reserve has stated that it may again buy specific maturities of Treasury securities in significant amounts to influence prevailing long-term interest rates to improve economic conditions and thereby aid in achieving its monetary policy objectives. The possibility of these purchases during future periods of economic stress could increase current demand for Treasury securities among market participants, even during normal times. This could keep interest rates on Treasury securities somewhat lower than they would be otherwise. The implementation of recent financial regulations and reforms in the wake of the 2007-2009 financial crisis resulted in changes in certain domestic sectors’ holdings of Treasury securities, including money market funds and banking institutions. Money Market Fund A money market fund is a type of mutual fund that is required by law to invest in low-risk securities. Money market funds act as intermediaries between investors seeking highly liquid, safe investments and corporate and government entities that issue short-term debt to fund operations. Money market funds typically invest in short-term, highly liquid securities, such as Treasury bills, and pay dividends that generally reflect short-term interest rates. Money market fund reforms that took effect in 2016 resulted in a significant increase in this sector’s holdings of Treasury securities (see sidebar). This sector experienced significant volatility during the 2007- 2009 financial crisis as large numbers of investors rapidly withdrew from these funds. To address this risk, the Securities and Exchange Commission (SEC) placed a number of restrictions on prime money market funds. Prime funds invest primarily in taxable short-term corporate and bank debt. The SEC regulations exempted government money market funds— which invest only in cash and U.S. government securities, including Treasury securities—from certain requirements because these assets are less risky and more liquid than other investments. Since these exemptions make government funds particularly attractive, many investors replaced prime money market fund investments with government money market fund investments (see fig. 7). Money market funds now represent one of the largest shares of Treasury securities holdings among domestic investors, holding approximately 8 percent (around $743 billion) of the domestic total as of June 2019 (excluding the Federal Reserve). The five money market funds we surveyed all reported that one of the top three ways they use Treasury securities is to comply with regulations. Following the financial crisis, U.S. and international regulators implemented reforms intended to promote a more resilient financial sector, including reforms aimed at the banking sector. Overall, these reforms increased demand from large banking institutions for Treasury securities. The reforms strengthened global capital and liquidity standards to make banking institutions more resilient and better able to lend in the event of an economic shock. For example, through the “Liquidity Coverage Ratio,” large banking institutions are now required to ensure they can cover short-term cash needs by holding a proportionate amount of high-quality liquid assets—cash reserves, Treasury securities, or Ginnie Mae securities. Since Treasury securities are classified as part of the group of most liquid assets, they are attractive for banks looking to meet these requirements. “Changes in bank liquidity regulations steered us to use more Treasuries in recent years.” Overall, bank holdings of Treasury securities increased from less than 1 percent of the sector’s total assets in 2008 (just over $100 billion) to more than 3 percent (over $800 billion) as of June 2019. The five banks we surveyed all reported that one of the top three ways they use Treasury securities is to comply with regulations. Foreign official demand for Treasury securities—which includes foreign governments and central banks as well as government-owned investment funds—has fluctuated based on economic conditions, especially the need for foreign central banks to manage their exchange rates. After the 2007- 2009 financial crisis, foreign governments increased holdings of Treasury securities from $1.5 trillion in 2007 to $4.1 trillion in 2015. In recent years, foreign governments’ accumulation of Treasury securities has slowed substantially. As of December 2018, they held about $4 trillion, or about 25 percent of all marketable Treasury securities. According to market participants and subject matter experts we interviewed, this slowdown does not imply a change in the nature of foreign demand for Treasury securities, but rather is a consequence of foreign central banks’ changing need for foreign reserves—many of which are held in the form of Treasury securities—to assist in managing their currencies. The U.S. dollar is the dominant currency used by foreign central banks in their official foreign exchange reserves, referred to as a reserve currency (see sidebar). As the reserve currency, foreign central banks buy and sell U.S. dollars to influence the value of their currencies to help manage their exchange rates, among other uses. To this end, foreign central banks hold Treasury securities in part because they can be converted to U.S. dollars quickly and in great quantity. Foreign central banks often act to limit the impact of exchange rate fluctuations and maintain the stability of their own currency. For example, a fall in U.S. interest rates tends to reduce the demand for dollars as private investors seek higher yielding assets abroad. In response, foreign central banks buy dollars—often investing those dollars in Treasury securities—and sell their own currency on foreign exchange markets which reduces the demand for—and hence the value of—their own currency relative to the dollar (see fig. 8). Conversely, when U.S. interest rates began increasing in 2015, dollar- denominated assets became more attractive to private investors seeking higher yields, which increased the value of the dollar relative to other currencies. In response to this and other events, experts we spoke with highlighted the role of China in particular— the largest foreign official holder of Treasury securities—in selling Treasury securities during that time period to help stabilize its exchange rate. Because U.S. interest rates are cyclical, foreign central bank interventions will also be cyclical, which implies their demand for Treasury securities will continue, to some extent, to vary over time so long as the U.S. dollar is a dominant reserve currency. Future changes in market conditions or policies—especially to the extent those changes significantly affect the combination of liquidity, depth, and safety of Treasury securities—could raise new and important risks to the Treasury market. Market participants we interviewed and surveyed across various sectors have raised concerns about risks that could affect demand for Treasury securities: risks from a future debt limit impasse, the sustainability of the federal debt, the dollar’s status as the primary reserve currency, and changes in the structure of the market which might affect liquidity, all of which could degrade the unique advantages of the Treasury market. Debt Limit The debt limit is a legal limit on the total amount of federal debt that can be outstanding at one time. (31 U.S.C. §§ 3101, 3101A.) It is not a control on debt but rather an after- the-fact measure that restricts the Department of the Treasury’s authority to borrow to finance the decisions already enacted by Congress and the President. Many market participants from all 10 sectors we surveyed and interviewed identified delays in raising (or suspending) the debt limit as potentially undermining the perceived safety of Treasury securities (see sidebar). During these times, Treasury departs from normal cash and debt management operations and takes extraordinary actions to avoid breaching the limit. Once all of the extraordinary actions are exhausted, Treasury may not issue debt without further action from Congress and could be forced to delay payments until sufficient funds become available. Treasury could eventually be forced to default on legal debt obligations. We previously reported that delays in raising the debt limit can lead to increased borrowing costs and significant disruptions in the Treasury market. For example, there were lengthy impasses over the debt limit in 2011 and 2013. During the 2013 impasse, investors reported taking the unprecedented action of systematically avoiding certain Treasury securities (i.e., those that would mature around the dates when Treasury projected it would exhaust the extraordinary actions available). Consequently, interest rates for these securities increased dramatically and liquidity declined in the secondary market where securities are traded among investors. “Treasury securities are held for liquidity management. It is critical that we have confidence in the timely payment of principal and interest on U.S. Treasury securities. Gamesmanship by political parties that impacts the confidence in timely payment on U.S. Treasury securities simply is not acceptable. We therefore are forced to invest in other forms of liquid securities, or to modify our participation in T-bills to avoid key dates around debt limits.” Overall, 48 of the 67 (72 percent) investors we surveyed reported that they anticipated they would take similar action—such as avoiding purchases of securities that would mature around the affected dates and requiring higher yields for purchasing those securities—to manage potential market disruptions caused by any future debt limit impasses. A default would have devastating effects on U.S. and global economies and the public. It is generally recognized that a default would prevent the government from honoring all of its obligations to pay for such things as program benefits; contractual services and supplies; employees’ salaries and wages and retirement benefits; and principal on maturing securities. Any disruption of these payments would have cascading effects on the economy. A default would call into question the full faith and credit of the U.S. government, and therefore immediately and significantly decrease demand for Treasury securities. Those investors who did purchase Treasury securities would demand a premium in the form of higher interest rates, to compensate for this increased risk. We have reported numerous times that the full faith and credit of the United States must be preserved. We have recommended that Congress consider alternative approaches to the current debt limit to avoid seriously disrupting the Treasury market and increasing borrowing costs. Experts have suggested replacing the debt limit with a fiscal rule imposed on spending and revenue decisions. As previously reported, Congress could consider this change as part of a broader plan to put the government on a more sustainable fiscal path. Some market participants we interviewed and surveyed expressed concern that continued deterioration of the federal government’s fiscal position could negatively affect the safety of Treasury securities. We have reported that the federal government is on an unsustainable fiscal path. Over the last 10 years, debt held by the public has more than doubled; increasing from about $7 trillion in 2009 to $16 trillion in 2019. We, the Office of Management and Budget, and the Congressional Budget Office estimate that federal debt will continue to grow, surpassing its historical high of 106 percent of gross domestic product within 13 to 20 years. Congress and the administration face serious economic, security, and social challenges that require difficult policy choices in the near term in setting national priorities and charting a path forward for economic growth. We have reported that a broad plan is also needed to put the federal government on a sustainable long-term fiscal path and ensure that the United States remains in a strong economic position to meet its security and social needs, as well as to preserve the flexibility to address unforeseen events. In August 2011, one of the major credit rating agencies, Standard & Poor’s, lowered its long-term sovereign credit rating on the U.S. from AAA to AA+, citing the United States’ rising public debt burden and greater policymaking uncertainty. The other major rating agencies have not lowered their rating of U.S. debt but continually monitor fiscal conditions and the political climate. If market participants perceive that the deteriorating fiscal outlook of the federal government could undermine the credit quality of Treasury securities, some investors could seek out alternative investments or demand a risk premium. This could further increase yields and therefore costs to Treasury. In general, larger deficits are likely to increase the yields on Treasury securities that are required by market participants, all else equal. Market participants and subject matter experts we interviewed emphasized the importance of the U.S. dollar’s status as the dominant global reserve currency in supporting demand for Treasury securities. So long as the U.S. dollar remains the dominant reserve currency worldwide, Treasury securities are likely to remain in high demand by foreign central banks and other investors. However, events that undermine the liquidity, safety, or depth of the Treasury market—such as debt limit impasses or concerns about fiscal sustainability—could reduce the share of U.S. dollar assets in foreign central bank reserves. Furthermore, reduced openness of the U.S. economy in global trade or financial markets would reduce the advantages of holding U.S. dollar reserves and could similarly precipitate a shift away from the U.S. dollar toward other currencies. Such a shift would likely reduce foreign official holdings of Treasury securities and could potentially reduce demand from other sectors that use U.S. dollars for global trade and other transactions. Consequently, Treasury’s cost to borrow would likely increase. Secondary market trading in Treasury securities is increasingly conducted on electronic platforms. The resulting changes and innovations have led to a number of benefits for market participants, but could also introduce new risks. For example, the Treasury Market Practices Group reported in 2015 that electronic trading had arguably improved overall liquidity through enhanced order flow and competition, reducing trading costs and allowing market participants to more effectively manage risk. Many market participants we surveyed agreed. For example, a market participant we surveyed reported that increased electronification of the Treasury market made it easier to price, trade, and settle holdings. However, market participants we surveyed and interviewed also told us that there is a potential risk of reduced liquidity and increased volatility in the Treasury secondary market. Market participants attributed these potential risks to a number of different factors related to the changing structure of the market: (1) increased use of automated trading; (2) increased role of principal trading firms; and (3) post-crisis financial reforms. Automated Trading A subset of electronic trading that relies on computer algorithms—advanced mathematical models—to make decisions about the timing, price, and quantity of the market order. High-frequency Trading A subset of automated or algorithmic trading in which the trading opportunities are identified and acted upon algorithmically and executed through technology at high speeds. Market participants we surveyed and interviewed said that automated trading—particularly high-frequency algorithmic trading (see sidebar)— may introduce operational risks that could interfere with market functioning. Automated trading relies on speeds that are beyond manual detection and intervention. Consequently, the Treasury Market Practices Group pointed out that internal controls may not be sufficient to counteract malfunctioning algorithms or algorithms reacting to inaccurate or unexpected data. For example, a malfunctioning algorithm could interfere with market functioning by creating sharp, short-lived spikes in prices as a result of other algorithms responding to an initial incorrect order. “Our Treasury trading desk is about 50 percent smaller than it was a decade ago, and we now have nearly as many traders devoted to algorithmic and electronic market- making as traditional market-making activity.” Market participants also noted that this type of trading may lead to more frequent episodes of volatility, making it more difficult to buy or sell Treasury securities at predictable or stable prices, particularly during periods of market stress. In one notable example, on October 15, 2014— in what has been referred to as a “flash rally”—the Treasury secondary market experienced record-high trading volumes and significant intraday volatility that could not be explained by external policy announcements or other factors. A 2015 interagency report examining the events of that day observed that as the speed of market activity increases, the Treasury market could continue to experience more frequent variations in market liquidity than in the past. Increased Role of Principal Trading Firms Advancements in technology, and the associated growth in high-speed electronic trading, have contributed to a shift in the composition of the types of firms actively trading and making markets in Treasury securities. Market-makers serve a crucial role in financial markets by providing liquidity to facilitate market efficiency and functioning (see sidebar). The 2015 interagency report examining the “flash rally” found that principal trading firms—proprietary trading firms that almost exclusively use automated trading strategies—conducted more than half of the trading activity on certain electronic platforms on the days reviewed. Market participants we spoke with expressed concern that some of the principal trading firms might not continue to provide liquidity in times of stress. According to the 2015 interagency report, principal trading firms tend to buy and sell frequently in small amounts, rarely holding Treasury securities beyond a day, and generally not trading on behalf of clients. Additionally, the extent of these firms’ presence in the Treasury market and the role they play is less well understood in part because they are not required to report their Treasury holdings and other financial information to the SEC that other financial institutions, such as broker-dealers and investment companies, are required to report. These firms’ holdings of Treasury securities are reflected in the Federal Reserve’s “household” category; the largest category of Treasury securities holdings among all domestic investors (excluding the Federal Reserve). As of June 2019, “households” held roughly $2 trillion in Treasury securities, up from $565 billion at the beginning of 2009—a 249 percent increase. According to Treasury, its 2018 market outreach revealed that data on the size of trades (market volume) are not transparent, which may hinder liquidity for certain securities. In September 2019, Treasury announced that the Financial Industry Regulatory Authority, Inc. (FINRA) expects to publicly release aggregate trading volume data for the Treasury secondary market in 2020. At the same time that the number of principal trading firms increased, market participants we surveyed and interviewed told us that broker- dealers are holding a smaller inventory of Treasury securities, which they attributed to certain post-crisis financial reforms that increased the cost of holding a large inventory of securities, including Treasury securities, for broker-dealers that are part of the larger banking institutions. As discussed above, these reforms were introduced to promote a more resilient financial sector. One set of reforms requires that large banking institutions hold a certain amount of high-quality liquid assets, including Treasury securities, to cover short-term cash needs. Another bank capital regulation—the supplementary leverage ratio—requires an institution to hold a supply of capital proportionate to total assets, which includes both low-risk assets (e.g., Treasury securities) and higher-risk assets. Because there are costs for holding capital, these institutions may prefer to reduce the size of their Treasury securities portfolio for the purpose of making markets and instead expand other lines of business that offer higher returns for the same amount of capital under the supplementary leverage ratio. Broker-dealers have traditionally been the predominant market makers for customers, including foreign central banks, mutual funds, hedge funds, pension funds, and insurance companies; buying and selling Treasury securities to meet customer trading needs, which could involve maintaining a large balance sheet to be able to buy and sell in large amounts and across days. According to market participants, broker-dealers’ smaller balance sheets have resulted in reduced liquidity for certain securities and could lead to additional risks during periods of secondary market stress or volatility. A well-functioning secondary market is important to Treasury in part because rates in the secondary market ultimately affect Treasury’s borrowing costs, as investors generally demand similar rates at auction to those in the secondary market. Treasury must regularly make important debt issuance decisions—such as what type of Treasury security to issue and in what quantities—to maintain broad-based demand and support its goal of borrowing at the lowest cost over time. Treasury officials described the steps the Office of Debt Management takes to make decisions about Treasury’s debt issuance strategy (see fig. 9). Treasury officials told us that they rely on three key inputs to help analyze financing options and inform these decisions: (1) market outreach, (2) auction and market metrics, and (3) analytical models. This is consistent with World Bank-IMF guidelines for public debt management. These guidelines highlight the importance of communicating regularly with investors, monitoring market activity, and having a strong analytical framework to inform decisions about the timing and amount of each type of security to issue. However, we found Treasury lacks policies governing some of these key inputs. Specifically, Treasury’s draft policy for bilateral market outreach does not include guidance on systematically selecting and documenting these interactions. Furthermore, Treasury does not have a policy governing important aspects of its analytical modeling, including requiring that analyses are documented and that Treasury staff follow and document appropriate quality assurance steps. Primary Dealers A group of banks and broker-dealers designated by the Federal Reserve Bank of New York (FRBNY) to serve as trading counterparties to the FRBNY in the implementation of monetary policy. They are also required to participate in all Treasury auctions. meets with half of them in person on a rotating basis to obtain estimates on borrowing, issuance, and the federal budget deficit (see sidebar). Treasury also uses the survey and meetings to obtain input on a variety of debt management discussion topics, posed in advance. For example, in April 2018 Treasury officials asked the primary dealers to comment on foreign private and official demand for Treasury securities over the short to intermediate term. Treasury Borrowing Advisory Committee An advisory committee composed of 15 senior officials from broker-dealers, asset managers, banks, and hedge funds. Treasury Borrowing Advisory Committee (TBAC). Treasury and TBAC meet quarterly as part of Treasury’s quarterly refunding process (see sidebar). At these meetings, Treasury officials and the committee members discuss economic forecasts, federal borrowing needs, debt management issues, and market dynamics. For example, in January 2019, Treasury asked TBAC to examine any products or debt management practices that might expand the investor base for Treasury securities, among other things. TBAC also provides Treasury with technical assistance intended to complement Treasury’s internal analyses. For example, in 2016, TBAC members began work to develop a debt issuance model to help guide the committee’s recommendations to Treasury about how to finance the government’s borrowing needs. In November 2017, based on the modeling framework as well as other factors, TBAC recommended that Treasury increase issuance of 2-, 3-, and 5-year notes to meet higher funding needs. Bilateral market outreach. To reach a broader range of investors, Treasury officials and staff also communicate directly—via email, telephone, conferences, and in-person meetings—with other market participants, such as foreign central banks, asset managers, investment banks, life insurance companies, pension funds, hedge funds, principal trading firms, and trading platforms. According to Treasury, staff use this bilateral outreach to discuss new products or distribution channels; assess investor needs; determine the drivers of market demand; and guide market perception about Treasury policy. Treasury officials said they select individuals for bilateral outreach using a combination of qualitative and quantitative information, such as data on specific investors’ participation in the Treasury market. According to Treasury, the bilateral market outreach helps mitigate an over-reliance on a subset of market participants that might not represent the full spectrum of views of Treasury market investors. However, we found that Treasury does not have an official policy to ensure that its bilateral market outreach is conducted or documented in a systematic manner. This is consistent with our reporting from 2010. In May 2010, Treasury officials told us that one of Treasury’s priorities was to improve investor outreach and collect information more systematically. Treasury acquired a customer relationship management tool, but Treasury officials said they only use it to store contact information. Treasury also drafted a policy document in November 2017 for Office of Debt Management staff that specifies the nature, restrictions on, and expectations for bilateral discussions with market contacts, but the policy is not final. While Treasury’s 2017 draft policy includes some guidance on documenting the bilateral outreach, Treasury officials told us they did not systematically produce formal documentation of these meetings. Treasury officials said that one reason Treasury did not have formal documentation of market outreach is because the staff who conduct the outreach also make the policy recommendations. Treasury officials also said direct outreach can sometimes cover market-sensitive information and that confidentiality is important to ensure candid exchange of information. However, the discreet nature of the outreach does not preclude Treasury staff from taking steps to document summary level information that would meet their needs and still maintain confidentiality. For example, Treasury officials and staff are experienced at managing market sensitive information for TBAC and primary dealers and communicating appropriate information to the public. While the level and nature of documentation can vary based on the materiality to decision-making, documentation is a necessary part of an effective internal control system. Documentation provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel. In 2017, Treasury conducted market outreach—through the primary dealers, TBAC, and bilateral discussions with market participants—about demand for a potential Treasury ultra-long bond (50- or 100-year bonds). At that time, Treasury decided not to proceed with introducing ultra-long bonds in part because its analysis indicated that the bond would be too costly to issue relative to other Treasury securities, such as the 30-year bond. In August 2019, Treasury announced that it was conducting broad market outreach to update its understanding of market demand for an ultra-long bond. Federal standards for internal control direct agencies to design and implement control activities—policies, procedures, and mechanisms—to achieve program objectives and respond to risks. A policy governing the selection of individuals for bilateral outreach could help Treasury ensure it is systematically obtaining market views from investors across various sectors. A policy for documenting bilateral outreach would also ensure that the information that Treasury staff obtains is available to help inform future deliberations. Treasury officials said that they are considering updating and finalizing the 2017 draft outreach guidance based on our review. In addition to market outreach, Treasury calculates and monitors metrics that summarize important aspects of the debt portfolio, Treasury auctions, and the secondary market. Treasury officials stated they monitor metrics to understand changing market dynamics and highlighted some of the key metrics they use to inform decisions (see table 2). According to Treasury officials, the percent of debt maturing in a given period is among the better indicators of rollover risk (see sidebar). 2. market access risk—the operational risks inherent in coming back to the market to refinance the debt. As of September 2019, more than half of the $16.3 trillion marketable debt held by the public will mature in the next 3 years; about 27 percent will mature in the next 12 months (see fig. 10). A significant share of that maturing debt will need to be refinanced at prevailing interest rates. Treasury publishes a number of key auction metrics that provide insight into auction demand for Treasury securities as well as which sectors purchase securities at auction (see table 3). Treasury also analyzes more granular data on bidders that are not publicly available. According to Treasury officials, one indicator of demand for Treasury securities at auction is the bid-to-cover ratio. When the ratio is greater than one, buyers submitted bids for more securities than were offered. Figure 11 shows weighted average bid-to-cover ratios for the 4-week bill, 2-year note, and 10-year note from 2000 to 2019. Treasury regularly engages with the Federal Reserve, SEC, and the U.S. Commodity Futures Trading Commission regarding secondary market activity, including significant price movements and their causes, trends in market structure (such as changes in venues, participants, and trade protocols), liquidity conditions, and market functioning. Treasury officials reported that they routinely review data relevant to secondary market activity (see table 4). Figure 12 shows the average daily trading volumes between primary dealers for Treasury bills; this is a measure of liquidity of the market. In the past, Treasury has had limited data on transactions in the secondary market. As a result, it has had limited real-time information on secondary market trading activity, which, as discussed earlier, has changed significantly in recent years, and has experienced abrupt changes in liquidity conditions, such as the October 2014 “flash rally” event. In July 2017, Treasury and other agencies gained access to more granular data on secondary market transactions as reported to the Financial Industry Regulatory Authority, Inc. (FINRA) by its broker-dealer members through the Trade Reporting and Compliance Engine (TRACE). Currently, the TRACE data are available to Treasury, the SEC, the Federal Reserve, and other official entities. According to Treasury officials, analyzing the raw TRACE data can provide insight into pricing in the market, patterns of trading activity, and the timing of trades. Treasury officials stated no other data source offered such detailed and reasonably comprehensive information on secondary market transactions in Treasury securities. However, there are limitations to the TRACE data, and Treasury is continuing to work with FINRA and the SEC to improve the quality of the data. Treasury has made policy recommendations supportive of expanding the scope of TRACE data reporting. Treasury reported that in April 2019, FINRA made enhancements to the Treasury transaction data that are reported through TRACE. For example, FINRA now requires more detailed transaction reporting to better understand the firms that are trading with each other. These identifying data will be available only to Treasury and regulators, such as the SEC and the Federal Reserve. According to Treasury, this will provide them with a better understanding of principal trading firm activity in the Treasury secondary market. Treasury’s analytical models are another source of information for the department’s financing decisions, but Treasury lacks a policy governing important aspects of these activities. According to Treasury officials, they use a number of analytical approaches, from fully specified models to simple illustrative analyses. Some models are more complex, combining information on the debt portfolio along with assumptions about future financing needs, economic conditions, and interest rates. Other models perform relatively simple calculations based on market data. Treasury officials told us they use these analyses to illustrate trade-offs, test potential financing options, and understand long-term dynamics of the Treasury market. These kinds of analytical tools can play an important role in good debt management decisions. According to Treasury officials, the bulk of modeling is completed by the Office of Debt Management’s Quantitative Strategies Group. Treasury officials told us that the group, which was formed in 2011, has two full- time-equivalent employees. Treasury officials provided examples of some internal analysis and modeling they have used in the last few years. Portfolio simulation models of the Treasury debt portfolio. These simulations produce estimates of future costs and risks—among other potential outputs—arising from the debt portfolio and potential issuance strategies. For example, the simulation can produce a cost metric that represents Treasury’s interest cost for a particular issuance strategy. In addition, the simulation can produce a risk metric that represents the amount of debt maturing over various periods (e.g., in 1 year, 3 years, 5 years) given a specific issuance strategy. One use of such a model is to represent an issuance strategy as one cost-risk choice among a range of options associated with alternative issuance strategies (see fig. 13). As assumptions about the economy or financial markets change, or as issue sizes or maturities are adjusted, the cost and risk outcomes change. In August 2018, Treasury officials stated that model output, along with market outreach and analysis of historical auction data, supported Treasury’s decision to increase issuance at all maturities with a focus on the intermediate range of 2, 3, and 5years. Stress testing to examine how the debt portfolio might perform in challenging environments. For example, Treasury staff examined projections of future borrowing needs and interest rates and analyzed how a strategy might perform under different interest-rate assumptions. Calculations to estimate the yields on potential new securities. For example, in 2017, Treasury used several analytical approaches to create a range of potential prices for an ultra-long bond. One approach estimated the additional yield for an ultra-long bond, assuming it would be proportionate to the difference between 30-year and 10-year bond yields. Analytical models can improve decisions, but they also come with risks, including possible adverse consequences of decisions based on models that are incorrect or misused. These risks can be managed through appropriate documentation and quality assurance. In our previous work, we identified the elements of economic analyses that are relevant for federal agency decision-making, including transparency and documentation of the analyses for internal stakeholders. Analyses should be transparent by describing and justifying the analytical choices, assumptions, and data used. Transparency allows internal stakeholders to understand the implications of these analytical choices and their associated risks. Sufficient documentation ensures that analytical choices, data, assumptions, limitations, and uncertainties are clear and available to future model developers and users. Documentation also provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel. Documentation of quantitative analyses and models should be clearly written, with a plain language summary and clearly labeled tables that describe the data used and results, and a conclusion that is consistent with these results. Documentation should also indicate that analyses comply with a robust quality assurance process. The Federal Reserve outlines a quality assurance process intended to verify that models are performing in line with their design objectives and business uses and also identifies potential limitations and assesses their possible impact. The degree of quality assurance required should be commensurate with the level of complexity, risk, and materiality to decision-making. Federal standards for internal control also direct agencies to design and implement control activities—such as documentation and quality assurance—through policies to achieve program objectives and respond to risks. Treasury provided information on its analytical models which included some key elements relevant to the documentation and transparency of Treasury’s analyses, including: Internal Treasury presentations that described the purpose, rationale, and certain analytical choices and results for a portfolio simulation model. Internal presentations detailing results and some analytical choices related to pricing estimates for an ultra-long bond. A code repository that can facilitate replication of some models and examples of code used to operate models. While Treasury’s documentation of its analytical models contained useful information for internal stakeholders, the documentation did not fully characterize the analytical choices, data, assumptions, limitations, and uncertainties associated with the analyses. For example: Treasury’s internal presentations on its portfolio simulation models did not fully justify analytical choices or describe the limitations of the models. Treasury’s internal presentations on pricing estimates for an ultra-long bond contain estimates from six different analytical approaches developed by Treasury but only detail a subset of the assumptions needed to arrive at the estimates. For example, there is no description of the precise structure of the approaches or the necessary sources of uncertainty that would lead to the range of estimates that Treasury presents for each approach. Treasury officials did not have documentation indicating that analytical models had been subject to quality assurance or that quality assurance activities had been commensurate with the level of complexity, risk, and materiality to decision-making. These issues arise in part because Treasury does not have a policy governing important aspects of the Office of Debt Management’s analytical modeling activities, including requiring that analyses are documented and that Treasury staff follow and document appropriate quality assurance steps. Treasury officials told us that they take steps to ensure that analytical work is appropriately reviewed. They stated that the review process is based on the nature of the work, and according to Treasury officials, quality assurance generally entails cross checks among staff and review by office leadership. One model was also shared with external contacts for feedback. Treasury officials emphasized that models are only one input of many into Treasury’s decision-making and explained that their practices are sufficient for the more straightforward analyses that typically inform decisions. However, the analyses that Treasury relies on—both relatively straightforward and more complex—to inform important decisions should be documented and subject to quality assurance to ensure that decision makers receive quality information based on appropriate analytical approaches. Treasury relies on a range of analytical methods, all of which require some degree of technical expertise to develop, implement, and evaluate, despite varying degrees of complexity. A policy requiring appropriate documentation and quality assurance would help Treasury ensure that analytical methods, data, assumptions, limitations, and uncertainties are transparent, appropriate, and available to future model developers and users. U.S. Treasury securities play a vital role in U.S. and global financial markets because of their deep and liquid market and because investors are confident that debt backed by the full faith and credit of the U.S. government will be honored. This combination of characteristics has helped support reliable demand for Treasury securities through ever changing market conditions, which, in turn, has helped minimize Treasury’s borrowing costs. Changing investment needs across different sectors and fluctuations in demand for Treasury securities are a normal part of economic cycles. Treasury and Congress need to be alert to risks that could compromise these key characteristics to preserve Treasury securities’ unique advantages. These risks include changing dynamics of the secondary market, including new participants using high-frequency trading strategies that could reduce liquidity, particularly in times of market stress. Treasury’s recent efforts to coordinate with the SEC and FINRA to obtain detailed information on the secondary Treasury market are an important step. In addition, as we have previously reported, Congress needs to consider taking action to address the unsustainable long-term fiscal path as well as alternative approaches to managing the debt limit that would ensure the continued safety of U.S. Treasury securities. Treasury has a critical role to play through its management of the federal debt portfolio to support its goal to borrow at the lowest cost over time. Treasury must promote strong demand for its securities from a diverse group of investors while making debt issuance decisions that appropriately balance risks and interest costs. Therefore, it is important that Treasury make these decisions based on the best information possible. Consistent with good debt management practices, Treasury uses a range of qualitative and quantitative inputs to inform its decision-making. It does not, however, have policies governing important aspects of two of these inputs: bilateral market outreach and analytical modeling. Until Treasury has designed and implemented policies around these key activities, it cannot be certain that needed information for debt issuance decisions is available, complete, and appropriately reviewed. Moreover, without appropriate documentation of important market outreach or analytical models, Treasury risks losing critical organizational information as staff leave the agency. Given the size and importance of the Treasury market, ensuring the quality of information available to decision-makers is essential to Treasury’s efforts to reduce risk and cost to taxpayers. We are making the following two recommendations to Treasury. The Secretary of the Treasury should finalize the Office of Debt Management’s policy for conducting bilateral market outreach and ensure it includes guidance on selecting market participants and documenting and sharing relevant information throughout the office while safeguarding the confidentiality of discussions. (Recommendation 1) The Secretary of the Treasury should establish a policy for the documentation and quality assurance of the Office of Debt Management’s analytical models. At a minimum, this policy should require (1) appropriate and sufficient documentation of analytical models, and (2) documented quality assurance of analytical models commensurate with the level of complexity, risk, and materiality to decision-making. (Recommendation 2) We provided a draft of this report to Treasury and the Federal Reserve for review and comment. In its comments, reproduced in appendix III, Treasury agreed with our recommendations and said it would work to implement them over the coming months. Treasury and the Federal Reserve also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Treasury, the Federal Reserve, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. For questions about this report, please contact Tranchau (Kris) T. Nguyen at (202) 512-6806 or nguyentt@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. To address both of our objectives, we surveyed market participants regarding (1) factors that affect demand for Treasury securities, (2) experiences interacting with the Department of the Treasury (Treasury), and (3) evolution of the Treasury market. In March 2019, we administered an online survey to 109 institutions. We selected the 10 largest institutions by total assets (or other equivalent financial indicator) in nine sectors that hold Treasury securities and the 15 largest mutual funds and exchange-traded funds by total assets under management (see table 5). We also sent the survey to four market participants we interviewed in September that did not meet our top 10 criterion for its sector. The survey results are not generalizable to all investors in Treasury securities. To define the sectors for our sample, we reviewed data from the Federal Reserve’s Financial Accounts of the United States, (table L.100 to L. 133, first quarter 2018) to identify sectors holding Treasury securities. We excluded some sectors due to challenges in contacting certain entities, such as foreign monetary authorities, other foreign investors, and the household sector. According to the Federal Reserve, the household sector is a residual category and includes individuals holding Treasury securities, hedge funds, and other institutions not required to report to regulatory bodies. We excluded this sector due to the difficulty of identifying, ranking, and contacting individual household investors and other entities. We excluded Government Sponsored Enterprises because these entities are unlikely to provide additional insights into the Treasury market beyond our sample, which includes commercial banks. We excluded federal government retirement funds because the Thrift Savings Plan does not invest in marketable Treasury securities. To identify the organizations within each sector that would receive our web-based survey, we used rankings of the largest organizations in each sector based on total assets or an equivalent financial indicator, such as assets under management or direct premiums written, and selected the 10 largest in each sector. In the case of mutual funds and exchange traded funds, we used information from the Investment Company Institute on total assets under management in Treasury- and government-focused funds to identify the largest 15 in that sector. For the broker-dealer sector, we selected the 10 largest primary dealers. As part of our survey of market participants, we asked respondents to identify products or debt management practices that, if the Department of the Treasury (Treasury) introduced, would increase the respondent’s overall demand for Treasury securities. Results from our related survey questions are presented below. Survey Question: If Treasury were to make the following changes to its offerings, would your overall demand for Treasury securities increase? (see fig. 14). Survey Question: If Treasury were to change its debt management practices in the following ways, would your overall demand for Treasury securities increase? (see fig. 15). In addition to the contact named above, Thomas J. McCabe (Assistant Director), Margaret M. Adams (Analyst-in-Charge), Abigail Brown, Michael Hoffman, Loren Lipsey, Daniel Mahoney, Anna Beth Smith, Andrew J. Stephens, Farrah Stone, and Wade Tanner made significant contributions to this report. Robert Gebhart, Jerome Sandau, Peter Verchinski, and Alicia White also contributed to this report.", "summary": "The Congressional Budget Office projects that federal deficits will reach $1 trillion in 2020 and average $1.2 trillion per year through 2029, further adding to the more than $16 trillion in current debt held by the public. As a result, Treasury will need to issue a substantial amount of debt to finance government operations and refinance maturing debt. To support its goal to borrow at the lowest cost over time, Treasury must maintain strong demand from a diverse group of investors for Treasury securities. GAO prepared this report as part of continuing efforts to assist Congress in identifying and addressing debt management challenges. This report (1) identifies factors that affect demand for Treasury securities and (2) examines how Treasury monitors and analyzes information about the Treasury market to inform its debt issuance strategy. GAO analyzed data on investor holdings of Treasury securities; surveyed a non-generalizable sample of 109 large domestic institutional investors across 10 sectors (67 responded); reviewed Treasury analysis and market research; and interviewed market participants across sectors, experts on foreign investors, and Treasury officials. The large institutional investors GAO surveyed across multiple sectors identified liquidity, depth, and safety as the most important characteristics of Treasury securities. This combination supports reliable demand from different types of investors through changing market conditions. Many investors accept low yields because of these characteristics, keeping the Department of the Treasury's (Treasury) borrowing costs low. Market participants GAO interviewed and surveyed identified risks that could degrade these key characteristics and reduce future demand: Debt limit impasses could force Treasury to delay payments on maturing securities and interest, until sufficient funds are available, compromising the safety of Treasury securities. Unsustainable levels of federal debt could cause investors to demand a risk premium and seek out alternatives to Treasury securities. A reduced role for the U.S. dollar as the dominant reserve currency could diminish the advantages of holding Treasury securities for foreign investors, particularly foreign government investors who hold large amounts of dollar-denominated assets to assist in managing their exchange rates. Changes in the Treasury secondary market where securities are traded— including high-frequency trading and a reduced role for broker-dealers who buy and sell for customers—could increase volatility and reduce liquidity. Treasury regularly makes important issuance decisions—such as what types of securities to issue and in what quantities—to maintain broad-based demand and support its goal of borrowing at the lowest cost over time. Treasury officials said three key inputs support these decisions: market outreach; auction and market metrics (e.g., trading volumes); and analytical models . However, Treasury has not finalized its policy for systematically conducting bilateral market outreach to ensure a thorough understanding of market demand. Treasury also does not have a policy governing important aspects of its analytical modeling, including following and documenting quality assurance steps to ensure that analytical methods are appropriate and available to future model developers and users. Codifying policies governing key information sources would help ensure that Treasury's decisions are based on the best possible information. GAO recommends that Treasury (1) finalize its policy for conducting bilateral market outreach and (2) establish a policy for the documentation and quality assurance of analytical models. Treasury agreed with these recommendations.", "document_type": "gao"}
{"report": "The GDPs were constructed in the 1940s and 1950s and were used to enrich uranium for the U.S. military as well as the nation’s domestic nuclear power industry. The GDPs are located near Oak Ridge, Tennessee; Paducah, Kentucky; and Portsmouth, Ohio (see fig. 1). The GDPs were rendered obsolete due to the emergence of newer, more efficient technologies and the globalization of the uranium enrichment market. All three GDPs eventually ceased uranium enrichment activities, with Paducah being the last to stop enriching by 2013. The GDP sites are similar in many ways. For example, the primary structures at each GDP are large buildings for uranium enrichment processing using the same gaseous diffusion technology. In addition, at each of the sites, these large buildings all housed similar equipment, such as compressors, converters, and other equipment necessary for enriching uranium. EM measures these buildings in acres rather than square feet (see fig. 2). For example, the five uranium enrichment processing buildings that once stood at Oak Ridge measured a total of 114 acres. Each GDP site also consists of hundreds of other similar buildings and facilities used to fabricate, service, repair, and clean machinery as well as additional infrastructure, such as electrical switchyards and cooling towers. Rescission of the USEC Fund The Energy Policy Act of 1992 created the United States Enrichment Corporation (USEC) as a government corporation authorized to, among other things, acquire, market, and enrich uranium. The 1992 Act also established a revolving fund in the U.S. Treasury—the USEC Fund—for carrying out USEC’s purposes. In 1996, Congress enacted the USEC Privatization Act authorizing establishment of a private, for-profit corporation. The act provided that “expenses of privatization” were to be paid from certain accounts, including the USEC Fund. One week before privatization, Congress enacted the “McConnell Act,” which reserved approximately $373 million from certain accounts, including the USEC Fund, for the disposition of depleted uranium stored at government-owned enrichment plants operated by USEC. USEC was privatized on July 28, 1998. After privatization, the USEC Fund balance of $1.2 billion was retained on the books of the Treasury. Since then, the balance of the USEC Fund has grown to an estimated $1.695 billion as of fiscal year 2020. In 2015, we found that the entire balance of the USEC Fund is available for permanent rescission since the two statutorily authorized uses for the USEC Fund have been fulfilled: (1) environmental clean-up expenses pursuant to the “McConnell Act,” and (2) expenses of privatization. In the fiscal year 2017 federal budget, the Administration proposed using the balances of the USEC Fund to carry out purposes authorized to be funded by the Uranium Enrichment Decontamination and Decommissioning Fund. This is not one of the authorized purposes of the USEC Fund. We have previously found that DOE’s effort to utilize USEC Fund monies instead of general fund appropriations to support efforts other than the authorized purpose of the USEC Fund would diminish transparency in budgeting. In May 2019, we highlighted this issue in our annual report on fragmentation, overlap, and duplication. As of September 2019, Congress had not passed legislation to permanently rescind the balance of the USEC Fund, as we suggested in April 2015. Rescission may increase the transparency of federal agencies' budget presentations and help Congress have a clear understanding of how new funding requests relate to funding decisions for existing projects with continuing resource needs. nuclear power reactors and enrichment continued until 1985. The Oak Ridge GDP permanently closed in 1987. Portsmouth. The Portsmouth GDP, a 3,778-acre site located north of Portsmouth, Ohio, operated from 1954 until 2001. The GDP enriched uranium for both commercial reactor fuel and military applications. The Portsmouth GDP includes three uranium enrichment processing buildings, as well as over 300 other buildings and facilities. Management of both Portsmouth and Paducah has changed over time. Specifically, the Energy Policy Act, as amended, established the United States Enrichment Corporation (USEC) as a government corporation to, among other things, provide uranium enrichment services and take over operations of the GDPs in Portsmouth and Paducah beginning in 1993 (see sidebar). By 1998, USEC was privatized under the USEC Privatization Act and became a subsidiary of the newly created USEC, Inc. USEC produced low-enriched uranium for commercial power plants until 2001, when it ceased operations at the Portsmouth GDP. Later that year, the plant was placed on cold standby—a dormant condition that would allow operations to be resumed within 18 to 24 months if needed—and USEC, under contract with DOE, maintained the site. In 2011, USEC returned the Portsmouth GDP to DOE and EM’s contractor initiated deactivation activities of the uranium enrichment processing buildings. Paducah. The Paducah GDP, located on 3,556 acres of land west of Paducah, Kentucky, initially produced enriched uranium for nuclear weapons from 1952 until 1993. From 1993 through 2013, USEC leased and operated the facilities to produce enriched uranium for the commercial nuclear power sector. Similar to the Portsmouth GDP site, management of the Paducah site has changed over time. The Paducah GDP has four uranium enrichment processing buildings as well as more than 500 other buildings and facilities. After shutting down operations in 2013, USEC returned the Paducah GDP to DOE in 2014. Table 1 provides additional detail on the GDPs, including the date when cleanup began, the site size, and the size of the contractor workforce performing the cleanup activities. Cleanup of the GDPs is a complex process that involves multiple, coordinated activities: surveillance and maintenance, D&D, and site remediation. Throughout the cleanup process, EM must conduct surveillance and maintenance activities at the GDPs to ensure public and worker safety. This includes maintaining and repairing site infrastructure, such as buildings and facilities and electrical and water supplies. The D&D process involves the following activities: deactivation, decontamination, decommissioning, and demolition. According to the National Academies and DOE, these cleanup activities are encompassed within the detailed processes described below: Characterization and measurement of the contaminants present. During this process, cleanup workers determine the identities, forms, amounts, and locations of hazardous and radioactive substances. According to DOE, common contaminants found at the GDPs include radioisotopes stemming from the historical enrichment process (e.g., uranium and technetium-99); hazardous chemicals (e.g., trichloroethylene, polychlorinated biphenyls, and beryllium); asbestos, and other hazardous materials typical of industrial facilities. When the GDPs were in operation, workers used volatile organic compounds in large quantities to clean and degrease equipment, which resulted in the release of such compounds, specifically trichloroethylene, into the environment. These compounds contaminated soil, surface water, and groundwater when they were spilled, burned in pits, discharged in holding ponds, or placed in trenches for disposal. Removal of large uranium deposits. During this process cleanup workers remove large deposits of enriched uranium from the process equipment and piping. This step is necessary at some of the uranium processing buildings to reduce the possibility of nuclear criticality—an event in which an assemblage of enriched uranium produces a short- duration (millisecond) burst of heat and radiation. This step is also necessary to resolve security concerns regarding the protection and handling of special nuclear materials. Disassembly and decontamination of equipment and building structural components. Hundreds of large process equipment components, such as converters, compressors, and motors may need to be disassembled and decontaminated. In addition, the floors, walls, and other structural components of buildings that housed such equipment must be decontaminated. Demolition of buildings and facilities. Hundreds of structures— including analytical laboratories, electrical switch yards, and uranium enrichment processing buildings that are many acres in size—must be demolished at the GDP sites. Management or disposal of waste. The D&D process generates significant amounts of waste, including building materials and hazardous and radioactive waste removed from equipment and piping. Waste management activities include treatment, storage, transportation, and disposal of low-level radioactive waste, hazardous waste, mixed radioactive and hazardous waste, and sanitary waste. In addition to surveillance and maintenance activities and the D&D of buildings and facilities, remediation of contaminated soils, surface water, and groundwater is a part of GDP cleanup and is an important aspect of protecting human health and the environment. According to DOE, remediation of contaminated soils, surface water, and groundwater involves assessing the site, including subsurface soils and groundwater contaminated by past GDP operation, and addressing the sources of contamination. According to EM, the Paducah GDP has the most groundwater and soil contamination of the three GDPs, and the Portsmouth GDP has the least amount of contamination. At each GDP site, EM is required to consult and reach agreement with federal and state regulatory agencies in determining cleanup requirements, strategies, and priorities. Federal laws, including the Resource Conservation and Recovery Act of 1976 (RCRA), as amended; the Comprehensive Environmental Response, Compensation, and Liability Act of 1980 (CERCLA), as amended; and cleanup agreements with state regulatory agencies in Kentucky, Ohio, and Tennessee govern cleanup at the three GDPs. RCRA establishes the framework for the management of hazardous and non-hazardous solid waste. CERCLA authorizes the federal government to respond directly to releases or threatened releases of hazardous substances, pollutants, or contaminants that may endanger public health or the environment. CERCLA requires that EPA maintain a National Priorities List that includes some of the most seriously contaminated sites that EPA identifies for long-term cleanup of hazardous substances, pollutants, or contaminants throughout the United States and its territories. Federal sites on this list are required to have an interagency agreement for expeditious completion of all remedial action at the facility. The interagency agreement, termed a Federal Facility Agreement, guides the cleanup process and sets enforceable milestones for priority cleanup activities as agreed to by all the parties to the agreement. The Oak Ridge and Paducah GDPs are both included on EPA’s National Priorities List under CERCLA. As a result, both sites have negotiated tri- party Federal Facility Agreements signed by DOE, EPA, and the relevant state regulator. Under the terms of these agreements, DOE must reach agreement with EPA and Tennessee and Kentucky state regulators to establish cleanup priorities and schedules for work with enforceable milestones subject to the agreements’ dispute resolution procedures. In addition, the agreements state that DOE must consult with these regulators in making budget requests to Congress for the GDPs. The Portsmouth GDP is not listed on EPA’s National Priorities List due to an agreement among regulators and, therefore, does not have a Federal Facility Agreement. Instead, the Ohio regulator is responsible for overseeing cleanup under a State of Ohio Consent Decree under RCRA and an Ohio Environmental Protection Agency Directors Final Findings and Orders for Decontamination and Decommissioning, which guide the cleanup process at Portsmouth. Under Presidential Executive Order 12580, DOE is the lead federal agency for implementation of CERCLA at Portsmouth. According to DOE’s Fiscal Year 2020 Congressional Budget Justification, the Ohio regulator used the CERCLA framework in developing the Orders. According to EPA officials we interviewed, EPA is not involved in regulating the CERCLA or RCRA components of the cleanup at the Portsmouth GDP. Decontamination and Decommissioning Fund: Uranium and Thorium Reimbursements Title X of the Energy Policy Act, as amended, authorizes the Decontamination and Decommissioning (D&D) Fund to reimburse licensees of uranium and thorium processing sites for their portion of D&D activities, reclamation efforts, and other cleanup costs attributable to the uranium and thorium materials they sold to the federal government. These sites became contaminated with radon and other decay products of uranium over time. According to a DOE report, as of 2017, there were ten sites that were continuing remedial activities and where DOE was continuing to provide reimbursements. According to the 2017 DOE report, DOE had at the time issued about $716 million in reimbursement payments since inception of the D&D Fund. The largest recipient is West Chicago Environmental Response Trust, with over $380 million in reimbursement payments through fiscal year 2017. As of fiscal year 2016, DOE estimates that the total remaining payouts to uranium and thorium producers will be approximately $164 million. In 1992, the Energy Policy Act established the D&D Fund to pay for the cleanup of the three GDPs. The act authorized $480 million in annual contributions to the D&D Fund (annually adjusted for inflation) for a period of 15 years—from fiscal years 1993 through 2007. According to the act, of the $480 million in annual contributions originally authorized, up to $150 million was to come from a special assessment collected from domestic utility companies that used the enriched uranium produced by the GDPs for nuclear power generation, and the remainder was authorized to be appropriated by the federal government for a period of 15 years. While domestic utility payments were discontinued in 2007, as prescribed by the 1992 Energy Policy Act, additional sums have continued to be appropriated for the D&D Fund. The act specified that any unused balances in the D&D Fund be invested in Treasury securities and any interest earned be made available to pay for activities covered under the D&D Fund. The act also authorizes reimbursements to uranium and thorium processing site licensees who provided raw materials to the GDPs for their cleanup costs (see sidebar). The Energy Policy Act, as amended, authorizes the D&D Fund to pay for the costs of all D&D and remediation activities at the GDPs. Specifically, according to EM officials, the D&D Fund is used to pay for the following cleanup activities: (1) D&D of inactive facilities either by cleaning up the facilities so they could be reused or by demolishing them; (2) remedial actions such as assessing and treating groundwater or soil contamination; (3) waste management, such as the transport and disposal of hazardous waste; (4) the surveillance and maintenance of the GDPs, such as general repairs to keep the buildings and facilities in a safe condition; (5) uranium and thorium licensee reimbursements; (6) training for contractor personnel who work on D&D activities; and (7) other activities, such as legal costs associated with the GDPs, funding to support site-specific advisory boards at Portsmouth and Paducah, and pension costs of workers involved in uranium enrichment or D&D. According to EM officials, there are additional cleanup-related activities taking place at the GDPs that are not covered by the D&D Fund, which include: (1) security—which provides services to protect nuclear materials, sensitive uranium enrichment technology, equipment, and facilities; (2) operation of the onsite waste disposal facility at Oak Ridge; and (3) conversion of depleted uranium hexafluoride—a byproduct of the enrichment process—into a more stable form, such as uranium oxide, that will require eventual disposal (see sidebar). Depleted uranium hexafluoride—referred to as depleted uranium “tails”—is a byproduct of the uranium enrichment process. The uranium enrichment process involves concentrating uranium-235, which is the isotope of uranium that undergoes fission to release enormous amounts of energy. Natural uranium contains 0.7 percent of the uranium-235 isotope, and tails contain less uranium-235 than natural uranium (i.e., less than 0.7 percent of uranium-235). Tails have historically been considered waste because the enrichment process required to extract the remaining useful quantities of uranium-235 is significant and can be costly. In addition, tails may be dangerous to human health and the environment and can form extremely corrosive and potentially lethal compounds when in contact with water. Therefore, the Department of Energy (DOE) has opted to convert its inventory of tails into a more stable chemical form, such as uranium oxide, that would allow for long-term storage and minimize environmental impacts and costs. The Portsmouth and Paducah gaseous diffusion plants (GDP) each store their inventories of tails in thousands of cylinders, and both GDPs have an onsite conversion facility. As of March 2018, DOE estimated that the combined tails stockpile at the Portsmouth and Paducah GDPs was estimated at 62,000 cylinders. DOE estimates the Portsmouth GDP will complete conversion of its tails inventory by 2034 and Paducah by 2047. Most of the tails inventory at the Oak Ridge GDP (approximately 7,200 cylinders) has been shipped to Portsmouth for conversion. According to DOE officials, the D&D Fund is not used to pay for conversion of the tails. To pay for these additional cleanup-related activities, EM officials reported that EM has used the Defense Environmental Cleanup and the Non-Defense Environmental Cleanup Appropriation Accounts. At Portsmouth, EM has also transferred natural uranium to site contractors in exchange for cleanup services—a practice EM refers to as “barter.” Additional details on this practice are discussed later in the report. As we reported in February 2019, effective program and project management are important to the success of efforts like the EM program. According to PMI, a program is defined as “related projects, subsidiary programs, and program activities managed in a coordinated way to obtain benefits not available from managing them individually.” According to a PMI conference paper, to reach the ultimate goal from a program—to obtain benefits not available from managing the related projects and program activities individually—a structured way of working has to be established. The Program Management Improvement Accountability Act requires the Office of Management and Budget (OMB) to adopt and oversee implementation of government-wide standards, policies, and guidelines for program and project management in executive branch agencies. In June 2018, OMB issued a memorandum on the implementation of this law that includes initial implementation guidance and calls for agencies to generally align their own program management standards to the management practices and principles found in the memorandum. The memorandum states that the act aims to improve program and project management practices within the federal government. The OMB memorandum also states that agencies may use program management leading practices developed by us, other agencies, and external voluntary consensus standard-setting bodies, such as PMI. EM has managed cleanup of the GDPs as three individual sites, rather than as an integrated program, and has not managed the cleanup of the GDPs consistent with relevant program management leading practices. For over a decade, DOE has reported to Congress in its triennial reports that its intent is to manage the GDPs in an integrated manner but has not developed an integrated program management plan, integrated master schedule, and a reliable, integrated, comprehensive life-cycle cost estimate. In addition, EM estimates that cleanup of the Oak Ridge GDP is nearing completion, that Portsmouth will be completed by 2041, and that Paducah will be completed between 2065 and 2070. The Energy Policy Act, as amended, establishes a single, shared D&D Fund to pay for the D&D costs of the GDP sites, such that EM must coordinate and make trade-offs in its use of limited resources among the three GDPs. In addition, since 2007, DOE has stated in its triennial reports to Congress that its intent is to manage the GDPs in an integrated manner. While neither EM nor DOE explicitly refers to the management of the GDP cleanup as a program, DOE’s stated intent is consistent with PMI’s definition of a program—”related projects, subsidiary programs, and program activities managed in a coordinated way to obtain benefits not available from managing them individually.” However, we compared EM’s management of the cleanup of the three GDPs to the three relevant PMI program management leading practices that we examined—those addressing planning, scheduling, and cost estimating—and found that EM is not managing the cleanup of the GDPs consistent with these practices: Planning—Having a program management plan. We found that EM does not have a GDP-wide program management plan. According to PMI, a program management plan formally expresses an organization’s concept, vision, mission, and expected benefits produced by the program; it also defines program-specific goals and objectives. In a 1996 report, the National Academies recommended that DOE develop a GDP-wide program management plan for cleanup of the three GDPs that would help coordinate decisions across the three GDPs. Representatives from the National Academies told us in December 2018 that they continue to believe this recommendation is valid. Furthermore, EPA and state regulators have criticized EM for not having a long-term vision for GDP cleanup. According to EM officials, EM developed site-level plans for each of the three GDPs over time as the GDPs ceased operating and became available for cleanup at different times—Oak Ridge ceased operating in 1987, Portsmouth in 2011, and Paducah in 2013. However, in reviewing what EM officials refer to as GDP program management plans, we found that the documents were created for different purposes and do not contain comparable information. For example, The Oak Ridge plan was created in 2017 as an update of a fiscal year 2014 through 2024 site-level plan for the three EM cleanup sites located at Oak Ridge reservation—the GDP, the Oak Ridge National Laboratory, and the Y-12 National Security Complex. This document presents a high-level picture of cleanup activities. EM officials told us that the Oak Ridge plan is intended to be high- level because cleanup of the Oak Ridge GDP is further along than cleanup of the Portsmouth and Paducah GDPs and because the Oak Ridge plan covers all three cleanup efforts at the Oak Ridge Reservation. EM officials also noted that other specific planning materials on the Oak Ridge GDP could be found in other documentation, but such documentation was not in the plan or in a usable form. The document EM provided as the Portsmouth plan contains a series of PowerPoint presentations for a March 2018 symposium on waste management. The PowerPoint slides were presented by both DOE officials and contractor representatives about different projects at the Paducah and Portsmouth sites. However, the slides contain contradictory information on when the Paducah GDP began deactivation—one slide indicates that deactivation began in 2014, but another shows deactivation will begin in 2035. EM officials at the Paducah GDP provided the 2015 site management plan for the Paducah GDP, which was signed by DOE and the contractor. This plan includes actions taken to date, site prioritization information (i.e., risk prioritization criteria), and key planning assumptions. The Paducah plan is the most comprehensive and detailed. The individual GDP plans differ in their level of detail; do not present comparable information, such as milestones that each GDP is to meet; and do not reference past, ongoing, or planned work at the other GDPs. As a result, they are not useful as plans for decision- making on the three GDPs in an integrated manner. Further, EM does not have a document that contains a concept, vision, mission, and expected benefits from GDP cleanup or that defines program-specific goals and objectives. By developing a GDP-wide program management plan, EM would have a comprehensive and consistent roadmap to achieve GDP cleanup and would be in a better position to leverage resources among the three GDPs. Scheduling—Having a reliable, integrated master schedule. We found that EM does not have an integrated master schedule for cleanup of the GDPs. According to PMI’s Program Management Standard, a program master schedule is the top-level program planning document that defines the individual component schedules and dependencies among program components (individual components and program-level activities) required to achieve the program goals. It should include those component milestones that represent an output to the program or share interdependency with other components. The program master schedule should also include activities that are unique to the program including, but not limited to, activities related to stakeholder engagement, program-level risk mitigation, and program-level reviews. The program master schedule determines the timing of individual components, enables the program manager to determine when benefits will be delivered by the program, and identifies external dependencies of the program. EM officials told us that the agency’s corporate database—the Integrated Planning, Accountability, and Budgeting System (IPABS)— contains the integrated master schedule for all of EM’s cleanup work, including the GDPs. The purpose of IPABS is to provide information on (1) changes to the life-cycle scope, cost, and schedule and (2) performance data such as earned value, performance metrics, and cleanup milestones. While IPABS provides a top-line planned completion date as well as other information, including cleanup milestones negotiated with regulators and performance metrics, it does not provide all of the information needed to build up to that date, including sequences clearly showing how related portions of work depend on one another. Without information such as sequences, it will not be possible for EM to identify the consequences of changes or possible managerial action to respond to them. An integrated master schedule makes it possible to help coordinate cleanup across the GDPs by establishing each GDP site’s schedule and identifying how related portions of work, such as funding profiles and workforce and equipment requirements that tie the sites together, depend on one another. For example, EM officials stated that certain demolition equipment, such as high-reach excavators, are in limited supply and may be shared among the three GDPs. By creating an integrated master schedule, EM would be in a better position to coordinate individual project activities across the three GDPs and thus help achieve program goals. Cost Estimating—Having a reliable, integrated, comprehensive life-cycle cost estimate. We found that EM does not have a reliable, integrated, comprehensive life-cycle cost estimate for cleanup of the GDPs consistent with PMI’s Program Management Standard, which calls for estimating a program’s full life-cycle costs. According to PMI, calculating full life-cycle costs and including transition and sustainment costs results in total cost of ownership. Total cost of ownership is considered to be relative to the expected benefit of one program against another to derive a funding decision. There are numerous estimating techniques to derive program cost estimates. Program cost estimates should also identify any critical assumptions upon which the estimates are made, as these assumptions may prove unfounded in the course of program delivery and require reconsideration of the program business case or revision of the program management plan. Finally, program cost estimation can support or guide cost estimation at the component level. Any prevailing program level cost estimation guidance intended for use at the component level should be documented and communicated to component managers. Instead, EM has, over time, developed separate cost estimates for each of the three GDPs that do not reference historic costs at the other GDPs. EM officials stated that IPABS contains the life-cycle cost estimate for EM’s cleanup work, including the GDPs. However, IPABS only provides a top-line cost estimate. It does not provide details on what information is included in developing that estimate, such as any critical assumptions upon which the estimates are made. Moreover, in February 2019 we reported that certain IPABS data, including expenditure data, were not reliable. By developing an integrated, comprehensive life-cycle cost estimate, EM management, Congress, and stakeholders would have information on total cleanup costs, including underlying costs, enabling more informed decision-making on funding and resource allocations from the shared D&D Fund across the three GDPs. EM officials acknowledged that cleanup work at the GDPs is managed independently by the three sites and not as an integrated program. However, the officials noted that the GDP cleanup work is managed as part of EM’s overall work to clean up radioactive and other hazardous waste that remains at 16 different sites across the nation, which they explained was all managed as one program. Further, according to EM officials, since the cleanup work is part of EM’s overall cleanup program it is able to make decisions at a high-level to support overall funding priorities, reduce the greatest risks, and effectively use taxpayer dollars. However, in February 2019, we reported on EM’s cleanup program and found that EM’s cleanup policy—which governs its cleanup work—does not follow any of the relevant program management leading practices related to a program’s management of scope, cost, schedule performance, and independent review of performance. The benefits of managing the work at the GDPs as a program have long been recognized. In 1996, the National Academies in its report to Congress recognized GDP cleanup as having the characteristics of a program noting that the repetitive and common design of the GDPs would allow for economies of scale in performing D&D. The report recommended that DOE develop a GDP-wide program management plan that integrates the D&D of the facilities and environmental remediation activities, as previously mentioned. According to the National Academies report, coordinating efforts across the GDPs at the complex level would help to ensure that D&D is integrated at the three sites and that resources, including disbursements from the shared D&D Fund, would be used effectively. Moreover, the report noted that delays would lead to substantial expenditures for surveillance and maintenance; deterioration of the facilities would exacerbate these costs; risks to individuals would increase; and the costs for safeguards and security for the sites would continue. In December 2018, representatives from the National Academies told us that they continue to believe that managing the GDPs as an integrated program would benefit cleanup efforts. By taking steps to manage the three GDPs as an integrated program and following relevant program management leading practices (developing a program management plan; an integrated master schedule; and a reliable, integrated, comprehensive life-cycle cost estimate), EM would have more reasonable assurance that it is taking every opportunity to increase the efficiency and effectiveness of its management activities. EM estimates that cleanup of the Oak Ridge GDP is nearing completion, that Portsmouth will be completed by 2041, and that Paducah will be completed between 2065 and 2070. Cleanup of the three GDPs— primarily remediation efforts—began in the late 1980s, and EM estimates that cleanup of the last GDP, Paducah, will be completed by 2070 at the latest. As figure 3 shows, based on DOE’s estimates, cleanup from start to completion will take 33 years at Oak Ridge, 52 years at Portsmouth, and 77 to 82 years at Paducah. Each GDP site still has varying levels of cleanup work remaining, mainly relating to when the site was closed. For example, the majority of cleanup work began at Portsmouth and Paducah after the contractor operating the GDPs—USEC—returned the site to DOE (in 2011 and 2014, respectively). The following provides a brief overview of the work remaining and estimated cleanup completion dates for each of the GDPs. See appendix II for a summary of the cleanup work completed as of June 2019. Oak Ridge. At Oak Ridge, the work remaining includes cleaning up surface and groundwater contamination, remediating soils on approximately 800 acres, and conducting D&D on more than 130 remaining facilities. DOE reported in its 2019 triennial report that it intends to complete cleanup of the Oak Ridge GDP by fiscal year 2022. However, according to EM documentation and officials, EPA officials, and state regulators, EM is unlikely to complete the cleanup by this date. In information provided to us in 2018 and in documentation supporting its cost estimate, EM cited fiscal year 2024 as the completion date for the Oak Ridge cleanup. In addition, in March 2019, EM officials said that all facilities at the Oak Ridge GDP will be demolished by fiscal year 2020 and remediation activities will be completed by fiscal year 2024, stating that the fiscal year 2022 date in the 2019 triennial report is based on outdated data. EPA and Tennessee regulators also told us they do not believe that EM’s current estimated completion date is realistic for the Oak Ridge GDP cleanup based on their understanding of the scope of remaining work, particularly cleanup of groundwater contamination. They said it is more realistic that cleanup of the Oak Ridge GDP will not be completed until the late 2020s and EPA believes cleanup completion could go out as far out as the 2040s, due to the lack of an agreed approach to address contaminated groundwater. The completion date for the Oak Ridge GDP has slipped in the past. Oak Ridge was previously scheduled to be completed in fiscal year 2009 and then in fiscal year 2012. Portsmouth. At Portsmouth, EM must complete D&D for three uranium enrichment processing buildings. Specifically, the first of three processing buildings is undergoing the final stages of deactivation, and the contractor is scheduled to begin demolition in fiscal year 2020. EM has started deactivation procedures at the second of the processing buildings, where EM is scheduled to start demolition in fiscal year 2024. At the third processing building, deactivation has yet to begin, and EM estimates the building will be ready for demolition in fiscal year 2031. In addition, EM must conduct D&D on hundreds of other support buildings and facilities. EM also plans to continue to remediate groundwater plumes at Portsmouth and to complete construction of an onsite waste disposal facility, which is scheduled to be operational by fiscal year 2020. According to the 2019 triennial report, cleanup of the Portsmouth GDP will be completed in 2041 based on scope and funding projections. However, in June 2019, EM officials told us that the Portsmouth cleanup will more likely be completed in 2043. Paducah. At Paducah, EM is focusing its near-term cleanup efforts on D&D of the C-400 building—a building that was used to clean machinery parts and test equipment and has been identified as the primary source of groundwater contamination at the site. After the demolition of this building, EM plans to dig up the slabs underneath the building to remove contaminants that EM believes are the source of the contamination, according to EM officials. According to EPA, EM is also focusing its near-term cleanup efforts on other activities, such as stabilization and deactivation of uranium enrichment and support buildings across the GDP, infrastructure optimization activities (including railroad upgrades for safe waste transport and downsizing the electrical power grid network), and new facility construction. According to an EM document and officials, deactivation of the processing buildings began in 2014, after USEC returned the site to DOE. In addition to the process buildings, EM will also need to conduct D&D on hundreds of other buildings and facilities. In addition, according to EM officials, EM has yet to decide on whether the waste produced from the GDP cleanup will be shipped offsite or if it will construct an onsite waste facility. EM estimates the cleanup of the Paducah GDP will be completed between fiscal years 2065 and 2070. The completion date for the Paducah GDP has slipped in the past. Paducah was previously scheduled to be completed in fiscal year 2040, and then in fiscal year 2047. EM reported it has spent at least $15.5 billion on GDP cleanup as of 2018, including approximately $5.1 billion on the Oak Ridge cleanup, approximately $6.7 billion on the Portsmouth cleanup, and approximately $3.7 billion on the Paducah cleanup. However, EM has limited detailed expenditure information on the cleanup activities carried out at the GDPs. Moreover, EM’s cost estimates for completing cleanup at the three GDPs are not reliable because they do not fully or substantially meet all of the characteristics of a high-quality, reliable cost estimate as described in our Cost Estimating Guide. Efforts to Supplement the Decontamination and Decommissioning Fund: Transfer of Natural Uranium for Cleanup As we reported in September 2011, from 2009 through 2011, the Department of Energy (DOE) used 1,473 metric tons of natural uranium to pay for $194 million in cleanup services performed by a contractor—the United States Enrichment Corporation (USEC)—at the Portsmouth gaseous diffusion plant (GDP). USEC then sold the natural uranium and retained the proceeds. The cleanup services provided by USEC included removing chemical and hazardous material from the GDP. DOE has in the past referred to this practice as “barter.” We found in our September 2011 report that DOE mischaracterized certain transactions with USEC as barters. From December 2009 through March 2011 DOE’s uranium transactions with USEC were sales authorized by the USEC Privatization Act, but they did not comply with federal fiscal law. The USEC Privatization Act requires that before a uranium sale, DOE must determine: the materials are surplus to national security needs; the department is receiving fair market value; and the sales will not adversely affect the domestic uranium mining, conversion, and enrichment industries. We found that DOE met these requirements. Nevertheless, by not depositing the value of the net proceeds from the sales of uranium into the Treasury, we found that DOE violated the miscellaneous receipts statute. This statute requires an official or agent of the government receiving money from any source on the government's behalf to deposit the money into the Treasury. By not depositing an amount equal to the value of the uranium into the Treasury, DOE inappropriately circumvented the power of the purse granted to Congress under the Constitution. DOE disagreed that its actions did not comply with federal fiscal law. We suggested that Congress consider authorizing DOE to, among other things, retain the proceeds of future uranium transactions. Pursuant to direction from Congress, in March 2018, DOE suspended this practice through fiscal year 2019. In its fiscal year 2020 budget request, DOE indicated that it would resume this practice to help pay for cleanup at Portsmouth. a practice EM refers to as “barter.” According to data provided by EM officials in 2018, from December 2009 through March 2018, EM transferred uranium valued at about $1.4 billion. According to an EM official, EM has used this transfer process exclusively at Portsmouth (see sidebar). Among other sources, the Non-Defense Environmental Cleanup Appropriation Account supplied over $1.2 billion in cleanup funding at Portsmouth for activities such as the operation of the depleted uranium hexafluoride conversion facility. Paducah. EM also reports that it has spent about $3.7 billion on the Paducah cleanup as of 2018. Similar to the Oak Ridge and Portsmouth GDPs, the D&D Fund paid for the majority of the cleanup costs at the Paducah GDP—approximately $2.7 billion. The remaining $1 billion in cleanup expenditures were funded by aforementioned appropriation accounts, including $138 million from the Defense Environmental Cleanup Appropriation Account on activities such as security and safeguards. EM tracks annual expenditures for cleanup activities at each GDP site in STARS, according to EM officials. However, EM does not track detailed expenditure information by GDP site on specific cleanup activities—such as remediation, waste management, or surveillance and maintenance—in that system. For example, EM officials provided data from STARS indicating that EM spent about $262 million on D&D at the Oak Ridge GDP in fiscal year 2007, but officials could not provide a breakdown of what specific cleanup activities the funds were used for, such as remediation or waste management. EM headquarters and site officials explained that they do not track detailed expenditure information of GDP cleanup activities in STARS because they are not required to do so. EM has previously provided a detailed breakdown of expenditures. For example, in our July 2004 report, in addition to expenditures on D&D, EM provided expenditures for the following categories: remedial actions, surveillance and maintenance, uranium and thorium reimbursements, waste management, and other activities. In addition, DOE’s 2007 triennial report has an appendix on GDP future costs that provided a similar breakout. However, EM officials could not provide current expenditure information similar to these prior reports. EM site officials told us that EM tracks more detailed expenditure data on certain categories by project, including demolition activities, and that these data were available in various project management systems maintained across the three sites. However, according to these officials, the various project management systems do not consistently track expenditures across the three GDP sites. EM headquarters officials stated that EM tracks more detailed expenditure data centrally in IPABS. However, in February 2019, we reported that the earned value management data in IPABS, which contain the expenditure data, were unreliable. Detailed expenditure data are important for developing reliable cost estimates, according to our Cost Estimating Guide. The Cost Estimating Guide states that it is always better to use actual costs rather than estimates as data sources, since actual costs represent the most accurate data available. EM officials told us that they used expenditure data at Oak Ridge, supplemented by other information, to help develop cost estimates at Portsmouth and Paducah. However, according to EM officials, EM does not track detailed expenditure data consistently across the three GDPs, therefore its ability to develop accurate and informed cost estimates for future work at the three GDP sites is limited. By tracking consistent and detailed expenditure information on cleanup activities across the GDPs, EM management would be better able to develop reliable cost estimates to plan for future work. EM’s cost estimates for cleanup of the three GDPs (about $28-$30 billion, according to DOE’s 2019 triennial report to Congress) are not reliable and likely underestimate the future cleanup costs. EM has developed individual cost estimates for each of the three GDPs over time and has presented those cost estimates in the triennial reports to Congress. EM prepared the latest cost estimate for Oak Ridge in 2013, for Portsmouth in 2014, and for Paducah in 2017. We assessed EM’s cost estimates for the three GDPs individually by comparing them with the best practices identified in our Cost Estimating Guide. The guide outlines best practices for developing a high-quality, reliable cost estimate and identifies four characteristics of such an estimate: comprehensive, well-documented, accurate, and credible (see fig. 5 for a depiction of the four characteristics and some of the best practices that underlie them). A cost estimate is considered reliable if the assessment for each of the four characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and cannot be considered reliable. We found that the Portsmouth and Paducah cost estimates fully or substantially met some of the characteristics of a reliable cost estimate, but none of the three cost estimates fully or substantially met all of the characteristics, so EM’s cost estimates for completing cleanup of the three GDPs are not reliable. Specifically, EM’s cost estimate for Portsmouth fully met the comprehensive characteristic and substantially met the well-documented and accurate characteristics. EM’s cost estimate for Paducah fully met the accurate characteristic and substantially met the comprehensive characteristic. However, in all other instances, the cost estimates partially or minimally met the characteristics, with Oak Ridge obtaining the lowest scores. Figure 6 provides a summary of our assessment of the cost estimates for Oak Ridge, Portsmouth, and Paducah for each characteristic. Appendix III provides additional information on our assessment. We also found that the cost estimates likely underestimate the cleanup costs because of challenges in reaching consensus on cleanup decisions with regulators that we discuss later in this report. In commenting on our assessment of the GDPs’ cost estimates, EM officials stated that they disagreed with our findings. According to EM officials, the cost estimates for the three GDPs have been audited numerous times and contain thousands of pages of support. Officials also questioned how the cost estimate for Oak Ridge scored the lowest of the three sites, when the documentation supporting that cost estimate was prepared by the same contractor that prepared the Paducah cost estimate using the same processes, practices, and procedures. We use the same criteria—our Cost Estimating Guide—to assess cost estimates throughout the federal government, and we follow the same process for assessing cost estimates. As we do for all agencies, we provided EM the opportunity to review the detailed analysis that we prepared as part of our assessment and the opportunity to provide additional documentation that may fill gaps identified in that assessment. While EM had documentation for the Paducah GDP cost estimate, which included a project life-cycle summary schedule and life-cycle baseline work breakdown structure, EM did not include such documentation for the Oak Ridge GDP cost estimate. In addition, many of the documents EM officials provided to support the Oak Ridge cost estimate were more than 5 years older than the cost estimate itself, a point by which EM should have had actual expenditure data rather than proposed data to inform the estimate. Because these documents did not contain actual expenditure data, we determined they were out of date for Oak Ridge’s 2013 cost estimate. We met with EM officials a second time to discuss our assessment of the Oak Ridge GDP cost estimate and reviewed additional documents provided by officials and modified the assessment to reflect that additional information. However, this information did not change our overall assessment. Until EM ensures the site-specific life-cycle cost estimates for the cleanup of each of the GDPs fully incorporate best practices for cost estimation, EM, DOE, regulators, and Congress will not have the information needed to understand the level of resources required to achieve cleanup of the GDPs. Under EM’s current cost estimates, remaining GDP cleanup costs exceed the balance of the D&D Fund by at least $25 billion, and EM faces challenges that could affect cleanup progress and the sufficiency of the fund. According to EPA and state regulatory officials from Kentucky and Tennessee, negotiations with EM regarding various cleanup decisions have strained relations between EM and the regulators and present challenges to the GDP cleanup progress that could affect cleanup progress and put additional demands on the D&D Fund. Finally, EM’s reporting to Congress on the sufficiency of the D&D Fund is based on old data and is not always complete or clear, which presents challenges to Congress’s ability to be fully informed in taking actions to address the sufficiency of the Fund. EM’s estimated costs of about $28 billion to $30 billion to complete cleanup of the GDPs—cited in DOE’s 2019 triennial report—exceed the $2.7 billion balance of the D&D Fund cited in a 2018 document agency officials provided. Most recently, in its 2019 triennial report, DOE stated that, as of September 2016, estimated cleanup costs exceeded the balance of the D&D Fund by $26.6 billion. DOE has therefore estimated that the D&D Fund would be exhausted by fiscal year 2020. Prior triennial reports have made similar estimations. However, according to EM data, this shortage is likely to be billions more. In 2017, EM prepared a revised cost estimate for Paducah, revising Paducah’s life-cycle cost estimate for completing cleanup to $34 billion from $15 to $16 billion in 2016 data. EM did not include this revision or note it in any way in the final 2019 triennial report provided to Congress. Based on this revision, EM’s estimated costs would be about $47 billion to $48 billion to complete cleanup of the GDPs. The sufficiency of the D&D Fund has been a long-standing issue. In July 2004, we reported that based on projected costs and revenues at the time, the D&D Fund would be insufficient to cover the cleanup activities at the three GDPs. To better ensure that the fund would be sufficient to cover the projected costs for authorized activities, we recommended that Congress consider reauthorizing the fund for an additional 3 years—to 2010—and require DOE to reassess the fund’s sufficiency before it expired in 2007 to determine if further extensions would be necessary beyond 2010. In November 2007, the U.S. Senate Committee on Energy and Natural Resources held a hearing on a bill which would have reauthorized the fund and required DOE to continue to assess the fund’s sufficiency. Although the committee did not take further action on that bill, Congress has continued providing appropriations to the D&D Fund. According to EPA and state regulatory officials from Kentucky and Tennessee, negotiations with EM regarding key cleanup decisions have strained relations between EM and the regulators and present challenges to the GDP cleanup progress. If EM is unable to reach agreement with the regulators on its preferred outcomes, there will likely be further delays and increases in GDP cleanup costs. The EPA and state regulatory officials said that their negotiations over pending cleanup decisions have raised concerns regarding EM’s priorities, cleanup remedies, and cost estimates. Because both the Oak Ridge and Paducah GDPs are included on EPA’s National Priorities List, both sites are required to have a Federal Facility Agreement—an agreement that guides the cleanup process and establishes cleanup priorities and schedules with enforceable milestones as agreed to by EM, EPA, and state regulators. Disagreements among the parties at both the Oak Ridge and Paducah GDPs present challenges to EM’s assumptions regarding the acceptance of its preferred cleanup strategy and will likely lead to delays and increases in EM’s estimated cleanup costs if that strategy is not followed. Disagreements over cleanup priorities. EPA and state regulatory officials disagree with EM’s cleanup priorities at Oak Ridge and Paducah. EM officials we interviewed told us their priority is characterizing, decontaminating, and demolishing buildings and facilities. EPA and state regulatory officials said that their priority is soil and groundwater remediation to address contamination. The Tennessee regulatory official said that the state agrees that the D&D of buildings is valuable and beneficial but that those operations must be followed by management and mitigation of soil and groundwater impacts. EPA officials also told us that EM needs to better balance D&D and remediation efforts by conducting more remediation activities. EM officials stated that at the Oak Ridge GDP, EM balances D&D with remediation activities, but they did not provide documentation about these efforts. The Tennessee regulatory officials added that EM has been reluctant to commit to milestones that regulators identify as a priority. In addition, EPA officials and the Kentucky state regulatory official said that EM reprioritizes the cleanup effort every few years. The Kentucky regulator added that this has led to delays in approving the site management plan. These issues have led to disputes, and strained relations at the Paducah GDP. Specifically, per the terms of their Federal Facility Agreement, EM, EPA, and the Kentucky regulator must annually agree to a site management plan that establishes enforceable milestones. However, the parties have not agreed to such a plan since 2015, and in its draft 2018 plan, EM changed its priorities from the 2015 plan by moving a number of enforceable milestones to non-enforceable planning dates. As of February 2019, these and other technical disputes between EM and EPA and state regulatory officials had delayed demolition of the C-400 building—the primary source of groundwater contamination at the Paducah site—by a year and led to cost increases. In commenting on a draft of this report, both DOE and EPA officials stated that disputes associated with the C-400 building demolition were resolved in a memorandum of agreement signed in August 2019. Differences in preferred cleanup remedies at Oak Ridge. The Oak Ridge Federal Facility Agreement requires EM to reach agreement with the regulators on cleanup remedies. According to EM, EPA, and Tennessee regulatory officials we interviewed, EM and the regulators differ in their choice of preferred cleanup remedies at the Oak Ridge GDP, an issue subject to dispute under the Federal Facility Agreement. At Oak Ridge, EM officials we interviewed said that their cost estimate for all of the groundwater cleanup assumes that regulators will agree to a waiver for active cleanup across the site, relying on a cleanup remedy called monitored natural attenuation— allowing natural processes to decrease or “attenuate” concentrations of contaminants in the groundwater and monitoring that progress over time. EM officials acknowledged that they have not reached agreement with regulators on groundwater cleanup remedies. The officials noted that their proposed approach is based on their analysis of what remedies are cost effective, technically practicable, technically feasible, fully protective, and likely to be agreed upon by the state. EM officials also noted that their cost estimates are developed following federal standards that require EM to assume the lowest cost remedy if no remedy is more likely than another. However, DOE’s preferred cleanup remedy may not be accepted by regulators. EPA and Tennessee regulators told us that while they may agree to a waiver for specific areas at Oak Ridge, they would not agree to a “blanket” waiver covering the entire site. They added that they would prefer that EM more actively address contamination, for example, by installing a pump-and-treat system at Oak Ridge. Without the blanket waiver included in their cost estimate, EM officials said that cleanup would likely be delayed by several years, and costs would likely increase by as much as hundreds of millions of dollars. EM officials later said that they are not seeking a blanket waiver and do not believe a blanket waiver will be required for all groundwater remediation requirements, but rather that focused waivers may be necessary for certain areas that cannot be restored by available technology. Notably, in reviewing EM’s most recent cost estimate, we found that the estimate continues to assume a waiver for the entire site. Concerns about EM’s cost estimation assumptions. EPA and the Kentucky and Tennessee state regulatory officials we interviewed told us that EM generally shares information under the terms of the Federal Facility Agreement. However, the officials said they were concerned that the assumptions behind EM’s cost estimates for GDP cleanup are not transparent and that EM has not worked with them to develop the estimates. EPA officials told us that EM does not adequately or transparently include EPA on technical scope and cleanup schedule considerations that underlie EM’s cost estimates. Tennessee regulatory officials added that EM’s cost estimates do not reflect the state’s assumptions about the technical scope and schedules for the remedies for soil and groundwater remediation. In commenting on a draft of this report, DOE officials stated that estimates for the Oak Ridge GDP reflect the technical scope and schedules to accomplish the end state remedies that the Tennessee regulator has agreed to for soil remediation. The officials added that they are working with the regulator on the remedy for groundwater remediation. Similarly, at the Paducah GDP, the Kentucky state regulatory official expressed concern that EM’s cost estimates were unrealistic—especially EM’s assumption that Paducah would receive over $1 billion in funding (in escalated dollars) for most years starting in 2036 and ending in 2050. Total enacted appropriations for Paducah in fiscal year 2019 were about $274 million; EM’s assumption would constitute a significant increase in Paducah’s funding. Without these increased funding levels, Paducah’s cleanup would likely extend beyond the 2065 to 2070 time frame, and EM’s estimates for completion and cleanup costs would likely increase. EM site officials at Oak Ridge disagreed that they have not been transparent with EPA and Tennessee state regulators, emphasizing that they have complied with all Federal Facility Agreement requirements regarding regulator participation in the budget process. At Paducah, the challenges between EM, EPA, and the Kentucky regulator are not new. In April 2004, we reported that EM, EPA, and the Kentucky regulator had difficulty agreeing on an overall cleanup approach as well as on the details of specific projects. Further, we found that over time, these disagreements had undermined trust and damaged the parties’ working relationship. We recommended that EM involve EPA and the Kentucky regulator early in the development of the annual site management plan and specific projects—before submitting formal cleanup proposals for regulatory approval—so that the parties can identify and resolve their concerns and reach consensus on cleanup decisions in a more timely manner. EM stated it believed at the time that it had been successful in fostering constructive relationships with its regulators and through its intent to involve regulators early in the decision-making process. In commenting on a draft of this report, DOE officials stated that every year DOE conducts scoping meetings with EPA and the Kentucky regulator to establish the strategy, planning schedules, and milestones for the annual site management plan prior to it being transmitted to the regulators in November. According to a September 2012 Memorandum on Environmental Collaboration and Conflict Resolution issued by OMB and the Council on Environmental Quality, departments and agencies should “increase the appropriate and effective use of third-party assisted environmental collaboration as well as environmental conflict resolution to resolve problems and conflicts that arise in the context of environmental, public lands, or natural resource issues, including matters related to energy, transportation, and water and land management.” Pursuant to the memorandum’s annual reporting requirement, DOE’s draft annual report from March 2018 presents information on the department’s use of third parties and other collaborative problem-solving approaches in fiscal year 2017. In that report, DOE cites the benefits of integrating third-party facilitation into DOE site and program office projects, including expanded and clearer communication that leads to smoother relationships with the regulators and the public. EM officials told us that they, in conjunction with the regulators, have used outside facilitators to help scope site management plans, work plans, and other project documents over the past few years. They said that they have engaged the services of a facilitator at Paducah on two significant efforts, and in both cases the facilitator added value and was effective. In addition, Tennessee state regulatory officials told us that they have used a mediator with EM at the Oak Ridge GDP site in the past, and they believe the process had a positive result. However, EM is currently not engaging the services of a facilitator at the three GDP sites to help the parties address differences in setting priorities, agreeing on remedies, and ensuring the cost estimates reflect regulator assumptions. By working with an independent, third-party facilitator to help resolve disagreements over cleanup priorities, cleanup remedies, and cost estimation assumptions, EM would be in a better position to achieve stakeholder concurrence on these issues and avoid future cleanup delays. EM’s reporting to Congress on the sufficiency of the D&D Fund is based on old data, incomplete information, and unclear scope, presenting challenges to Congress’s ability to be fully informed in taking actions to address the sufficiency of the fund. The Energy Policy Act, as amended, required the Secretary of Energy to report within 3 years of enactment, and at least once every 3 years thereafter, on the progress of the GDP cleanup effort. DOE has continued to prepare triennial reports on the status of the D&D Fund and GDP cleanup for Congress. However, DOE’s 2019 triennial report is based on outdated information, provides limited information on the challenges EM faces in reaching agreement with EPA and state regulators, and is not clear on the scope of work. These limitations reduce the quality of the information Congress receives for making decisions about allocating resources to the D&D Fund at the same time that Congress will have to address a continued need for resources for GDP cleanup given the fund is estimated to be exhausted by 2020. The 2019 triennial report is based on outdated information. The latest triennial report, issued in May 2019, is based on financial information as of September 2016 and on cost estimates prepared in 2013 (Oak Ridge) and 2014 (Portsmouth and Paducah). In addition, the report does not contain information on an updated cost estimate for the Paducah site. Specifically, for Paducah, the report cites a cost estimate—prepared in 2014—of $15 billion to $16 billion and a completion date of 2047. However, EM prepared a revised cost estimate in 2018 that estimated costs to be $34 billion and estimated completion dates ranging from 2065 to 2070. EM had initially included information from this 2018 estimate in a draft of the 2019 triennial report, but ultimately did not include this information or note it in any way in the final report provided to Congress. EM headquarters officials told us that they did not include the updated 2018 Paducah cost estimate in the final 2019 report because they had already completed an extensive field and headquarters review process of the 2019 triennial report and did not want to repeat that process. The 2019 triennial report does not discuss the challenges EM faces in reaching agreement with EPA and state regulators. The 2019 triennial report has a section on challenges and uncertainties for each GDP. For the Oak Ridge and Paducah GDPs, this section does not discuss the challenges EM faces in reaching agreement with regulators on cleanup remediation decisions. For example, the Oak Ridge challenges and uncertainties section of the 2019 triennial report mentions that some groundwater treatment may be required, but the report does not disclose EM’s assumption in its cost estimate that it will receive a waiver allowing it to avoid active groundwater remediation activities or that this is an area of disagreement with the regulators. Similarly, the report’s discussion of challenges and uncertainties at Paducah mentions that several CERCLA decisions regarding groundwater need to be made, but does not discuss disagreements with the regulators over priorities or the implications of those decisions on cost or schedule. Information in triennial reports is not always clear on scope of work. Some information in the triennial reports has not always been clear. For example, when reporting its cost estimates in its three most recent triennial reports (2010, 2016, and 2019), DOE reports only future costs for Oak Ridge; whereas for Portsmouth and Paducah it reports either total costs (past plus future estimated costs), or future costs, or does not clearly indicate if the cost estimate represents total or future costs. These differences make it difficult to make comparisons among the three GDPs. In addition, in six triennial reports, DOE reported similar estimated future costs for completing the Oak Ridge GDP cleanup—$1.2 billion in the 1998 report; $1.3 billion in 2001; $1.6 billion in 2007; $2.1 billion in 2010; $1.4 billion in 2016; and $950 million in 2019. Estimated costs to complete cleanup would likely be reduced over time as work scope is completed, unless the scope of work is increasing, costs for materials are increasing, or prior estimates were incorrect; however, DOE has not clearly explained the factors contributing to these similar future cost estimates in any of its reports since 2007 (2007, 2010, 2016, 2019). Standards for Internal Control in the Federal Government state that management should externally communicate the necessary quality information to achieve the entity’s objectives. Quality information is appropriate, current, complete, accurate, accessible, and provided on a timely basis. Given that DOE estimates the D&D Fund will be exhausted in 2020, there is an urgency for DOE to communicate current and accurate information on the fund on a timely basis to Congress. By regularly reporting on the status of the D&D Fund and cleanup efforts at the three GDPs with current information that contains details on challenges in reaching agreement with regulators and a clear scope of work, DOE will be able to provide better information for congressional decision-making on the sufficiency of the fund. EM has made progress in cleaning up DOE’s three former GDPs— particularly at Oak Ridge where contractors have demolished all five uranium enrichment processing buildings measuring a combined 114 acres as well as most other supporting buildings and facilities—but future work remains. Although DOE has stated its intent to manage cleanup of the GDPs in an integrated manner, EM is not managing the cleanup as an integrated program, even though cleanup of the GDPs meets the definition of a program as defined by PMI and Congress established a single, shared D&D Fund to pay for the cleanup. By taking steps to manage the three GDPs as an integrated program and following relevant program management leading practices we examined (developing a program management plan, an integrated master schedule, and a reliable, integrated, comprehensive life-cycle cost estimate), EM would have more reasonable assurance that it is taking every opportunity to increase the efficiency and effectiveness of its management activities. Further, EM has limited expenditure data and its cost estimates for completing cleanup are not reliable. Detailed expenditure data are important for developing reliable cost estimates. However, according to EM officials, EM does not track detailed expenditure data consistently across the three GDPs. As a result, EM’s ability to develop accurate and informed cost estimates for future work at the three GDP sites is limited. By tracking consistent and detailed expenditure information on cleanup activities across the three GDPs, EM management will be better able to develop reliable cost estimates to plan for future work. Moreover, EM does not have reliable cost estimates for completing cleanup of the three GDPs. Until EM ensures the site-specific life-cycle cost estimates for the cleanup of each of the GDPs fully incorporate best practices for cost estimation, EM, DOE, regulators, and Congress will not have the information needed to understand the level of resources required to achieve cleanup of the GDPs. According to EPA and state regulatory officials from Kentucky and Tennessee, negotiations with EM regarding various cleanup decisions have strained relations between EM and regulators and present challenges to the GDP cleanup progress that will likely cause further delays and increase GDP cleanup costs if EM is unable to reach agreement on its preferred outcomes. EM officials said they have used third-party facilitators with the regulators in the past but are not currently engaging the services of a facilitator at the three GDP sites. By working with an independent, third-party facilitator to help resolve disagreements over cleanup priorities, cleanup remedies, and cost estimation assumptions, EM would be in a better position to achieve stakeholder concurrence on these issues and avoid future cleanup delays. Finally, DOE’s 2019 triennial report is based on outdated information, provides limited information on the challenges EM faces in reaching agreement with EPA and state regulators, and is not clear on the scope of work, thereby reducing the quality of the information Congress receives about the sufficiency of the fund. Given that DOE estimates the fund will be exhausted in 2020, there is an urgency for the department to communicate current information on the fund on a timely basis to Congress. By regularly reporting on the status of the D&D Fund and cleanup efforts at the three GDPs with current information that contains details on challenges in reaching agreement with regulators and a clear scope of work, DOE will be able to provide better information for congressional decision-making on the sufficiency of the fund. We are making five recommendations to DOE: The Secretary of Energy should direct the Assistant Secretary of the Office of Environmental Management to take steps to manage the three GDPs as an integrated program and follow relevant program management leading practices (developing a GDP-wide program management plan; an integrated master schedule; and a reliable, integrated, comprehensive life-cycle cost estimate.) (Recommendation 1) The Secretary of Energy should direct the Assistant Secretary of the Office of Environmental Management to track consistent and detailed expenditure information on cleanup activities across the three GDPs. (Recommendation 2) The Secretary of Energy should direct the Assistant Secretary of the Office of Environmental Management to ensure the site-specific life- cycle cost estimates for the cleanup of each of the GDPs fully incorporate best practices for cost estimation. (Recommendation 3) The Secretary of Energy should direct the Assistant Secretary of the Office of Environmental Management to work—in conjunction with EPA and Kentucky and Tennessee state regulators—with an independent, third-party facilitator to help resolve disagreements over cleanup priorities, cleanup remedies, and cost estimation assumptions. (Recommendation 4) The Secretary of Energy should regularly report on the status of the D&D Fund and cleanup efforts at the three GDPs with current information that contains details on challenges in reaching agreement with regulators and a clear scope of work. (Recommendation 5) We provided a draft of this report to DOE and EPA for comment. In DOE’s comments, reproduced in appendix IV, the agency generally agreed with our findings and recommendations, and described actions that DOE intends to take in response to our recommendations. Specifically, of our five recommendations, DOE concurred with four and partially concurred with one. DOE also provided technical comments, which we incorporated as appropriate. EPA did not provide written comments but provided technical comments, which we incorporated as appropriate. DOE concurred with our first and second recommendations that the Secretary of Energy should direct the Assistant Secretary of the Office of Environmental Management to (1) take steps to manage the three GDPs as an integrated program and follow relevant program management leading practices and (2) track consistent and detailed expenditure information on cleanup activities across the three GDPs. In its response to the first recommendation, DOE stated that EM will develop a program management master plan, to include site integrated master schedules and life cycle costs for the remaining cleanup at the Portsmouth and Paducah GDPs, and that the plan will incorporate program management leading practices as appropriate. In response to the second recommendation, DOE stated that EM will assess and identify an appropriate mechanism for tracking expenditures for both the Portsmouth and Paducah GDPs, using a standardized approach with an Earned Value Management System reporting on, at a minimum, an annual basis. We appreciate DOE’s commitment to improve cleanup at the Portsmouth and Paducah sites; however, we emphasize that these two recommendations are directed at all three GDPs, including the Oak Ridge GDP. We reported that DOE intends to complete cleanup of the Oak Ridge GDP by fiscal year 2022, but according to EM documentation we reviewed and EM officials we interviewed, as well as EPA officials and state regulators we interviewed, EM is unlikely to complete the cleanup by this date. EPA officials and Tennessee regulators stated that it is more realistic that cleanup of the Oak Ridge GDP will not be completed until the late 2020s, and EPA officials told us that cleanup may not be completed until the 2040s. Given the potential for Oak Ridge cleanup to continue for at least another decade, we continue to believe it is important that DOE include Oak Ridge in its implementation of these two recommendations. DOE partially concurred with our third recommendation that the Secretary of Energy should direct the Assistant Secretary of the Office of Environmental Management to ensure the site-specific life-cycle cost estimates for the cleanup of each of the GDPs fully incorporate best practices for cost estimation. DOE stated that EM will direct the Portsmouth and Paducah sites to review and incorporate practices from our Cost Estimating Guide, as appropriate, into the next revisions of each site’s life-cycle cost baselines. DOE also stated that the remaining scope for the Oak Ridge GDP will become part of the performance baseline for the next Oak Ridge contractor. We appreciate DOE’s commitment to improve cost estimation for the Portsmouth and Paducah GDPs. However, we continue to believe that improving cost estimation for the Oak Ridge GDP is also important, given that cleanup of Oak Ridge may continue for at least another decade, as described above. As such, we continue to believe it is important that DOE include Oak Ridge in implementing this recommendation. DOE concurred with our fourth recommendation that the Secretary of Energy should direct the Assistant Secretary of the Office of Environmental Management to work—in conjunction with EPA, and Kentucky and Tennessee state regulators—with an independent, third- party facilitator to help resolve disagreements over cleanup priorities, cleanup remedies, and cost estimation assumptions. DOE stated that as disagreements over cleanup priorities, remedies, and cost estimation assumptions arise, EM will work with all parties to determine the feasibility and benefits of using a facilitator on a case by case basis to help resolve issues. DOE also concurred with our fifth recommendation that the Secretary of Energy should regularly report on the status of the D&D Fund and cleanup efforts at the three GDPs with current information that contains details on challenges in reaching agreement with regulators and a clear scope of work. DOE management stated that EM will produce its next triennial Uranium Enrichment Decontamination and Decommissioning Fund Report following closeout of fiscal year 2019, and release of the most recent environmental liability estimate associated with the remaining challenges and scope of cleanup at the GDPs. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, the Administrator of EPA, and other interested parties. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made significant contributions to the report are listed in appendix VI. Our report examined: (1) the extent to which the Department of Energy’s (DOE) Office of Environmental Management (EM) has managed cleanup of the three gaseous diffusion plants (GDP) compared with relevant program management leading practices and the status of the cleanup effort; (2) what EM has spent on cleanup at the three GDPs and the extent to which EM’s cost estimates for completing GDP cleanup are reliable; and (3) the extent to which the Decontamination and Decommissioning (D&D) Fund is sufficient to cover EM’s estimated cleanup costs of the GDPs and challenges, if any, that could affect the sufficiency of the D&D Fund. To inform all three objectives, we reviewed the Energy Policy Act of 1992, as amended; DOE triennial reports to Congress on GDP cleanup efforts; and prior reports issued by us, DOE’s Office of Inspector General (both performance audits and financial statement audits on the D&D Fund), and the National Academies of Sciences, Engineering, and Medicine (National Academies). We also interviewed officials from DOE’s Office of Inspector General, the Environmental Protection Agency (EPA), and representatives from of the National Academies, regarding their knowledge of EM’s cleanup progress at the GDPs and any past, ongoing, or future work they have conducted or are planning on the GDP cleanup. We visited all three GDP sites to observe the cleanup work and meet with EM officials responsible for the cleanup, representatives of the DOE contractor responsible for D&D activities, state regulators working with EM on environmental compliance activities (from Kentucky, Ohio, and Tennessee), members of GDP site-specific advisory boards, and representatives of community reuse organizations. During our interviews, we discussed topics including funding for the GDP cleanup, cleanup progress to date, and any challenges facing the cleanup effort. We selected these interviewees because we determined, based on input from EM officials, that they would be the most knowledgeable about GDP cleanup status, funding, and challenges. Following these interviews, we conducted a content analysis of all responses to our interview questions to determine any key challenges that EM faces in completing cleanup of the GDPs. We then grouped, coded, and verified the content in our analysis and performed second-rater review. Through our content analysis, we found that stakeholders primarily cited three key challenges related to EM’s program management; relations between EM, EPA, and state regulators; and transitioning the local communities to cleanup completion. To examine the extent to which EM has managed the cleanup of the GDPs compared with relevant leading practices for program management, and the status of the cleanup effort, we reviewed documents, including site-specific GDP cleanup plans and GDP cleanup progress briefings, as well as reports issued by the National Academies, us, and DOE. We interviewed EM officials and contractor representatives on their past, present, and future plans for cleanup. We also interviewed EPA and state regulatory agency representatives at each of the GDPs regarding their role in the cleanup and interactions with EM. We assessed the information from these reviews and all interviews (content analysis from interview responses) and identified the relevant program management leading practices that aligned with the assessed information. We identified the three program management leading practices by reviewing our prior work and the Project Management Institute’s (PMI) The Standard for Program Management—Fourth Edition. The three leading practices were having (1) a program management plan, (2) an integrated master schedule, and (3) a reliable, integrated, comprehensive life-cycle cost estimate. We compared EM’s management of the GDPs with these leading practices. Specifically, during our interviews with EM, the DOE Office of Inspector General, and EPA officials; Kentucky, Ohio, and Tennessee regulators; representatives of the National Academies; and members of the site- specific advisory board from all three sites, we asked about challenges EM faces in completing cleanup of the three GDP sites. As discussed above, we conducted a content analysis of their responses to our interviews and found that stakeholders primarily cited three key challenges, including EM’s poor program management. Under poor program management, stakeholders cited three sub-challenges: (1) frequent changes in EM’s cleanup priorities and staff turnover, which most closely aligns with the program planning leading practice; (2) lack of integrated schedules across the GDPs, which most closely aligns with the scheduling leading practice; and (3) lack of transparency in EM’s cost estimation processes, which most closely aligns with the program cost estimating leading practice. As a result, we assessed the three leading practices that aligned with those issues: (1) program management plan, (2) integrated master schedule, and (3) integrated comprehensive life- cycle cost estimate. To examine the status of cleanup at the GDPs, we reviewed EM’s documentation of the work completed and the work remaining at each GDP. To examine what EM has spent on cleanup at the three GDP sites, and the extent to which EM’s cost estimates for completing GDP cleanup are reliable, we reviewed historical funding and cleanup expenditure data for all three sites for the period from fiscal year 1994 through 2018 and analyzed EM documentation supporting its cost estimates for each of the three GDPs. The data the sites provided include expenditures from the D&D Fund as well as from other funding sources including: the American Recovery and Reinvestment Act, Uranium Facilities Maintenance and Remediation funds, Environmental Management Waste Management Facility funds, and Technetium-99 cleanup funds. We reviewed financial statement audit reports issued on the D&D Fund for fiscal years 2005 to 2012 and met with relevant headquarters and field staff in financial management, budget, and planning. In addition, we assessed the reliability of the historical funding and expenditure data provided by EM. Specifically, we obtained from EM officials familiar with DOE’s financial management system responses to a series of data reliability questions such as data entry access, quality control procedures, and the accuracy and completeness of the data. During our review of the GDP expenditure data, we identified a number of inconsistencies between the data received from EM site officials and the data reported in DOE’s 2019 triennial report to Congress. EM officials were able to provide satisfactory responses and documentation to address the identified inconsistencies. We therefore found the data to be reliable for our purposes. To examine the reliability of EM’s cost estimates for completing cleanup at the three GDPs, we reviewed EM’s cost estimate documentation, interviewed EM site officials, and compared GDP cost estimates against characteristics of reliable cost estimates contained in our Cost Estimating Guide. Our review included documents that established the basis and assumptions for site contractors’ contributions to the cost estimate, documents that established the contractors’ work breakdown structures, and presentations on contractors’ cost estimating models. We interviewed EM site officials and contractor staff responsible for producing the cost estimates to understand the methods, assumptions, information, and data EM used to produce the estimates. Our cost estimation specialists assessed this information against the best practices for cost estimating found in our Cost Estimating Guide that we developed to establish a consistent methodology that can be used across the federal government to develop, manage, and evaluate capital program cost estimates. We shared our draft assessment for each GDP cost estimate with EM officials and then revised those assessments based on EM’s written comments and additional documentation they provided as appropriate. At EM’s request, we met with Oak Ridge officials a second time to discuss our assessment of the Oak Ridge GDP cost estimate and reviewed additional documents provided by officials, and we reflected that additional information into our assessment of the Oak Ridge cost estimate. To examine the extent to which the D&D Fund is sufficient to cover EM’s estimated cleanup costs of the GDPs and challenges, if any, that could affect the sufficiency of the D&D Fund, we reviewed information on the balance of the D&D Fund and compared it to EM cost estimate information, past reports that describe the balance of the fund, and our prior report on the fund. Despite our findings that the three cost estimates were unreliable, we were able to report on the cost estimates provided in DOE’s 2019 Triennial Report by presenting an “at least” cost estimate. In addition, we interviewed key stakeholders, including officials from EM, the DOE Office of Inspector General, and EPA; regulators from the states of Kentucky, Ohio, and Tennessee; representatives of the National Academies; and members of the site-specific advisory boards and representatives of the community reuse organizations from all three sites, regarding challenges EM faces in completing cleanup of the three GDP sites and challenges that could affect the sufficiency of the D&D Fund. As noted above, we conducted a content analysis of their response and found that stakeholders primarily cited three challenges that could affect cleanup progress and further strain the D&D Fund, including challenges with negotiations with EPA and state regulators. We also reviewed DOE’s triennial reports from 1996 to 2019 and compared information included in each of these triennial reports to determine the extent to which the information provided was presented consistently across reports and consistent with other documentation provided, such as site-specific plans and DOE’s cost estimates. We also interviewed DOE officials about the sufficiency of the D&D Fund and factors affecting the sufficiency of the fund. We conducted this performance audit from April 2018 to December 2019, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. This appendix provides information on cleanup work completed at the Department of Energy’s (DOE) former gaseous diffusion plants (GDP) as of June 2019. DOE’s Office of Environmental Management (EM) is responsible for their cleanup. EM began cleanup at Oak Ridge in 1989 and Decontamination and Decommissioning (D&D) of the uranium enrichment process buildings in 1998. Since that time, EM has characterized the levels and types of contamination for most of the site and conducted D&D on all five uranium enrichment process buildings. EM has also demolished over 390 additional buildings and facilities, including a fire water tower and the Central Neutralization Facility that was used to treat the site’s industrial wastewater. In addition, EM has remediated nearly 1,400 acres of contaminated soils and has used an onsite waste disposal facility to dispose of much of the waste generated from cleanup. Some specific cleanup work EM has completed at Oak Ridge includes: Removed slabs from two uranium enrichment process buildings and completed cleanup of contaminated soils beneath the slab, clearing the way for transition to industrial reuse. Excavated and disposed of approximately 100,000 cubic yards of contaminated materials from a burial ground. Remediated an area considered to be a primary source of organic contamination in area soils and groundwater and treated the resulting approximately 175 cubic meters of contaminated soil. Removed more than 48,000 tons of scrap metal from two scrap yards. EPA and Tennessee state regulators agree that the end use for the site will be a commercial industrial park, and several businesses are already leasing portions of former GDP lands. In addition, more than 3,000 acres of the former GDP lands have been cleared for conservation and recreational use. EM has partnered with the Community Reuse Organization of East Tennessee to attract businesses to operate on the available lands. According to a representative of the Community Reuse Organization of East Tennessee, EM has transferred over 1,000 acres of land and 14 buildings to the reuse organization, who has in turn sold over 300,000 square feet to the private sector. There are 20 private companies operating at the site. EM began cleanup at the Portsmouth GDP in 1989 and D&D of the uranium enrichment process buildings in 2011, after the contractor that operated the site—the Unites States Enrichment Corporation (USEC) returned the buildings to DOE in 2010. As of May 2019, EM is preparing the first of three uranium enrichment process buildings for demolition and is starting to characterize contamination in the second. EM is also conducting ongoing remediation activities and constructing an on-site waste disposal facility, where EM intends to dispose of D&D waste that meets the approved acceptance criteria of the disposal facility. Several site support facilities, including a large electric switchyard, have been demolished. Some specific cleanup work EM has completed at Portsmouth includes: Completed sampling and removal for off-site disposal of all 7,020 uranium enrichment components (converters, compressors, and coolers) from one of the uranium enrichment process buildings. Closed five on-site landfills covering 60 acres. Removed more than 37,000 pounds of trichloroethylene—a solvent for degreasing metal that contaminated the groundwater at the site— through groundwater remediation. EM contractors at Portsmouth told us that they are cleaning up the site for future industrial use. EM began cleanup at the Paducah site in 1988. USEC officially returned the GDP to DOE in 2014 and according to an EM document and officials, deactivation of the uranium processing buildings began that same year. In January 2019, EM reached a milestone—deactivation of the C-400 building—by completing the cleanup of legacy materials in the building. C-400 was a cleaning facility used to clean machinery parts and test equipment and has been identified as the primary source of groundwater contamination at the site. According to EM officials, EM has primarily been using a pump-and-treat method to control the high concentration portion of the groundwater plumes at Paducah. EM officials stated that EM is focusing its cleanup efforts on D&D of the C-400 building and remediation from now until the early 2030’s. According to EM officials, EM is continuing to treat large contamination plumes and demolish inactive facilities. Some specific cleanup work EM has completed at Paducah includes: Demolished and removed 43 inactive facilities including a 210,000 square foot uranium hexafluoride feed plant and a 60,000 square foot metals plant. Treated over four billion gallons of contaminated groundwater from two operating pump-and-treat facilities and, as part of this treatment, removed approximately 3,700 gallons of trichloroethylene. Removed more than 850,000 cubic feet of low-level and mixed low- level legacy wastes and material storage area waste. Resurfaced 74 acres of roofs at the site and rerouted roof drains in order to reduce infiltration of water into the facilities. Officials at Paducah told us that they are cleaning up the site for future industrial use. In addition to the individual named above, Amanda K. Kolling, Assistant Director; Luqman Abdullah; Mark Braza; Jennifer Echard; Emile Ettedgui; Juan C. Garay; Mark Keenan; Jennifer Leotta; Gregory Marchand; Kiki Theodoropoulos; and Lauren Woodard made key contributions to this report. Also contributing to this report were Alexandra Edwards; Keegan Maguigan; Anne Stevens; and Doris Yanger.", "summary": "Cleaning up DOE's former uranium enrichment sites will cost billions of dollars and span decades. These sites, near Oak Ridge, Tennessee; Paducah, Kentucky; and Portsmouth, Ohio, are contaminated with radioactive and hazardous materials. EM is responsible for their cleanup. This report examines (1) the extent to which EM has managed cleanup of the GDPs compared with relevant program management leading practices and the status of the cleanup effort; (2) what EM has spent on cleanup at the GDPs, and the extent to which EM's cost estimates for completing GDP cleanup are reliable; and (3) the extent to which the D&D Fund is sufficient to cover EM's estimated cleanup costs of the GDPs and challenges, if any, that could affect the sufficiency of the fund. GAO reviewed relevant legislation and DOE reports to Congress on GDP cleanup; compared program management to relevant leading practices; assessed EM expenditure and cost estimation documents; and interviewed EM and state regulatory officials at the three GDPs. Since 2007, the Department of Energy (DOE) has stated in reports to Congress that it intends to manage its three former gaseous diffusion plants (GDP) in an integrated manner. Also, a Decontamination and Decommissioning (D&D) Fund was established by law to pay for the cleanup costs of the GDP sites, so that DOE's Office of Environmental Management (EM) must coordinate and make trade-offs in its use of resources among the three GDPs. However, EM has managed the cleanup of the three GDPs as three individual sites. In addition, EM is not following relevant leading practices GAO reviewed for managing the cleanup as a program (having a program management plan; a reliable integrated master schedule; and a reliable, integrated, comprehensive life-cycle cost estimate. By managing the three GDPs as an integrated program and following these program management leading practices, EM would have more reasonable assurance that it is taking every opportunity to increase the efficiency and effectiveness of its management activities. EM has reported spending a total of about $15.5 billion on GDP cleanup as of fiscal year 2018. However, EM's cost estimates for completing cleanup at the three sites are not reliable. GAO assessed EM's cost estimates for the GDPs individually by comparing them with best practices for developing high-quality, reliable cost estimates. EM's cost estimates for completing cleanup of the GDPs do not fully or substantially meet all of the characteristics of a reliable cost estimate Until EM ensures that its site-specific cost estimates fully incorporate best practices for cost estimation, EM, DOE, regulators, and Congress will not have the information needed to understand the level of resources required to achieve cleanup of the three GDPs. Under EM's current cost estimates, remaining GDP cleanup costs exceed the balance of the D&D Fund by at least $25 billion and EM faces challenges that could affect cleanup progress and the sufficiency of the fund. For example, DOE's reporting to Congress on the sufficiency of the D&D Fund is based on old financial data, incomplete information, and unclear scope. These limitations reduce the quality of the information Congress receives for making decisions about the sufficiency of the fund and allocating resources to the fund. For example, DOE reported to Congress on the status of the D&D fund and GDP cleanup in May 2019. The report was based on financial data as of September 2016 and on cost estimates prepared in 2013 for one GDP and in 2014 for the other two. Given that DOE estimates the fund will be exhausted in 2020, there is urgency for DOE to communicate current information on the fund on a timely basis to Congress. By regularly reporting on the status of the D&D Fund and cleanup efforts at the three GDPs with current information that contains details on challenges in reaching agreement with regulators and a clear scope of work, DOE will be able to provide better information for congressional decision-making on the sufficiency of the fund. GAO is making five recommendations, including that DOE (1) manage the cleanup of the three GDPs as an integrated program and follow program management leading practices, (2) ensure cost estimates fully incorporate cost estimating best practices, and (3) report regularly on the status of the D&D Fund and cleanup efforts at the three GDPs. DOE agreed with four of them and partially agreed with one. GAO believes all of the recommendations should be implemented at all three sites.", "document_type": "gao"}
{"report": "MDA is responsible for developing a number of systems, known as elements, with the purpose of defending against ballistic and hypersonic missile attacks. MDA’s mission is to combine these elements into an integrated system-of-systems known as the Ballistic Missile Defense System (BMDS). The goal of the BMDS is to combine the abilities of two or more elements to achieve objectives that would not have been possible for any individual element. These emergent abilities are known as integrated capabilities or BMDS level capabilities. Table 1 provides a brief description of selected BMDS elements. MDA was established in 2002 with exceptional flexibilities to manage the acquisition of the BMDS—developed as a single program—that allow MDA to expedite the fielding of assets and integrated ballistic missile defense capabilities. These flexibilities allow MDA to diverge from DOD’s traditional acquisition life cycle and defer the application of certain acquisition policies and laws designed to facilitate oversight and accountability until a mature capability is ready to be handed over to a military service for production and operation. In addition, MDA has been operating in an environment of tight timeframes for delivering capabilities—beginning with a presidential directive in 2002 to field a limited capability by 2004. This was followed by a presidential announcement in 2009 to begin deploying U.S. missile defense in Europe in 2011 finishing in 2020. This schedule required concurrency among technology, testing and other development activities. More recently, MDA has been directed to develop and deploy defenses against hypersonic and cruise missile threats as soon as technologically able. These schedule pressures compound challenges associated with complex technology, design, and integration associated with the missile defense mission that normally require careful planning, disciplined engineering practices, extensive coordination, and effective management and oversight to be successful. MDA has taken important actions to increase transparency, reduce high- risk approaches in its management of BMDS elements, and test and deliver BMDS capability. Specifically, MDA has improved reporting in its annual progress reports to the Congress and made advances across a broad range of management activities, including the involvement of stakeholders, reducing concurrency, and continued efforts to improve key aspects of testing necessary to demonstrate delivered capability. Increased Transparency: MDA, consistent with several of our recommendations has increased the ability to track progress over time in the BMDS Accountability Report (BAR). This is MDA’s annual report that presents the current estimate of the BMDS programs’ baselines. To increase insight into MDA’s management of the BMDS, MDA implemented significant changes to its key acquisition processes and for the first time developed and reported detailed baselines for each element in the BAR in 2010. As we found in March 2011, MDA’s prior approach limited the ability for DOD and congressional decision makers to measure MDA’s progress on cost, schedule, and testing. While MDA’s changes were positive, over the years, we made additional recommendations to further improve MDA’s reporting. In response to our recommendations, MDA made improvements to the BAR that include providing details on variances to its test plan from year to year and including information on its use of contract actions known as an Undefinitized Contract Actions (UCA) and Unpriced Change Orders (UCO). Improved Stakeholder Outreach: MDA has increased its outreach to DOD stakeholders over the past few years. Our prior work on defense acquisitions has shown that establishing buy-in from decision makers is a key factor in achieving better acquisition outcomes because DOD components provide varying perspectives due to their unique areas of expertise and experience. For example, as we reported in December 2019, MDA has recently increased its interaction with the defense intelligence community. Specifically, MDA engaged the defense intelligence community on an analysis of alternatives the agency completed in February 2017 that assessed future sensor options for the BMDS. In addition, MDA reached out to the defense intelligence community on another analysis of alternatives pertaining to defense against hypersonic missiles. In fact, officials from several DOD organizations we met with over the past two years observed that MDA’s engagement with their organizations was improving. Reducing Concurrency: MDA continues to take steps to reduce concurrency, an issue we have reported on for many years. Concurrency is broadly defined as the overlap of development, testing, and production; coupled with an aggressive testing schedule. MDA’s concurrent development has often left the agency committing to production and fielding before development is complete. This approach has resulted in performance shortfalls, cost increases, and schedule delays. MDA has taken steps to mitigate this risk consistent with our recommendations. For example, as we found in May 2017, MDA took steps to reduce concurrency in the Aegis BMD SM-3 Block IB by adding in tests and delaying the full-rate production decision until the tests were completed. Figure 1 represents a highly concurrent acquisition schedule as compared to an approach based on gaining knowledge before proceeding to the next acquisition phase. Improving BMDS Testing: MDA has improved the accuracy of tools it uses to assess integrated BMDS capabilities. The BMDS is a system of systems that cannot be completely assessed using intercept flight tests because of the system’s scope and complexity, and because of safety constraints. Consequently, MDA, independent DOD testing organizations, and the warfighter must rely heavily on representations of the integrated BMDS called models and simulations in ground testing. This approach is used, rather than live tests, to test the operational performance of the whole BMDS against attacks with more threats represented. Our preliminary observations for fiscal year 2019 are that the number of accredited models and simulations that are needed to assess the integrated performance of the BMDS has steadily risen over the last 3 years. Over the past several years, we have reported on MDA’s progress in delivering assets and capabilities to counter attacks as well as cyber threats. MDA delivered important BMDS capabilities for architectures in the United States as well as those defending U.S. troops and allies in Europe, the Middle East, and the Eastern Pacific. For example: Homeland Defense: In fiscal year 2017 and 2018, MDA delivered a significant integrated capability for defending the United States, including improvements in the ability to discriminate lethal objects in targets, and increased capacity. This was a key achievement in fulfilling a directive from the Secretary of Defense to increase inventory of ground-based interceptors by the end of 2017. Regional BMD: In fiscal year 2016, MDA delivered capabilities for the second phase of its effort in Europe, called European Phased Adaptive Approach (EPAA). This effort required coordinated development of a number of elements and their integration to provide integrated BMDS-level integrated capabilities against short and medium range ballistic missiles. More recently, in fiscal years 2018 and 2019, MDA rapidly delivered capabilities for its effort to meet an urgent regional need. In addition, preliminary observations from our review covering fiscal year 2019 indicate that cybersecurity assessments in fiscal year 2019 informed the network defense posture in U.S. Northern Command and provided data on how to reduce mission risk for these elements operating in a cyber-contested environment. Moreover, the agency is incorporating lessons learned from prior cyber activities, and continues to address issues discovered in prior testing, improving its overall cybersecurity survivability. However, our preliminary observations indicate much remains to be done to ensure cyber resiliency of the BMDS including the completion of cybersecurity testing for capabilities delivered in 2017 and 2018, along with conducting element-level operational cooperative and adversarial assessments. MDA has made efforts to put some programs on a more sound footing and it has taken actions to address the issues I just mentioned. However, MDA can go further to align itself with best practices for acquisitions. Today, I will highlight certain acquisition challenges MDA still faces. Stakeholder involvement: While MDA has increased its outreach to the stakeholders over the past few years, opportunities remain for further engagement on key decisions. For instance, as we found in December 2019, although MDA has been increasing its engagement with the intelligence community, MDA provides the defense intelligence community with limited insight into how the agency uses threat assessments to inform its acquisition decisions. MDA is not required to obtain the defense intelligence community’s input; however, the community is uniquely positioned to assist MDA keep pace with rapidly emerging threats. Moreover, this limited insight has, in part, prevented validation of threat models designed to assess BMDS capabilities. Without validation, any flaws or bias in the threat models may go undetected, which can have significant implications for the performance of MDA’s weapon systems. MDA and the defense intelligence community recently began discussing a more suitable level of involvement in the agency’s acquisition processes and decisions. As we recommended in May 2017 and December 2019, MDA also needs to strengthen its collaboration with other stakeholders, including the warfighting community and independent cost and technical experts. In the early stages of the RKV program, concerns raised about the design—which ultimately was a key reason for the cancellation of the RKV—went unheeded. For example, preliminary observations for our assessment covering fiscal year 2019 showed that MDA and contractors did not adequately address technical risks despite numerous warnings from stakeholders about the performance issues. However, MDA officials indicate they are working with stakeholders more closely as they plan for the Next Generation Interceptor, a new more advanced interceptor. Concurrency: Although MDA has taken steps to reduce concurrency as we have previously recommended, the agency still turns to this practice when experiencing developmental delays or schedule pressures. For example, we reported in June 2019 that delays to construction resulted in MDA’s introduction of increasing levels of concurrency into the delivery schedule for the Aegis Ashore site in Poland. We found that key phases of the delivery process had been shortened from 16.5 months to 6.5 months. While overlapping acquisition activity, in theory, could speed up the construction process, this risky practice ultimately failed to mitigate the effects of problematic construction practices. However, program plans indicate that the site has experienced further delays and will not be ready for operational use until at least 2022—a 4 year delay from the original 2018 delivery date. In addition, the recently canceled Redesigned Kill Vehicle (RKV) program originally sought to avoid concurrency by aligning production decisions with flight testing. However, later—in response to the advancement of the North Korean missile threat—the program accelerated RKV development by concurrently performing development and production and reducing the number of necessary flight tests. This acceleration altered the schedule for the previously aligned flight tests and production decisions. Contracting: Although MDA has flexibilities in managing its acquisition process, it must follow the same contracting regulations that apply to DOD, including the Federal Acquisition Regulation and the Department of Defense Federal Acquisition Regulation Supplement. These regulations allow MDA to use a particular type of contract action called an undefinitized contract action when the negotiation of a definitive contract is not possible in sufficient time to meet the government’s requirements and government interests demand that the contractor be given a binding commitment so that contract performance can begin immediately. These actions authorize contractors to begin work before an agreement on terms, specifications, or price have been agreed upon. In May 2018, we found that the average length of the undefinitized period and the not- to-exceed price of MDA’s undefinitized contract actions had increased over the past 5 years. While MDA policy permits use of undefinitized contracts on a limited basis, we and others have found that they can place unnecessary cost risks on the government. As we reported in June 2019, while MDA improved its performance in timely definitization of these contract actions, the total not-to-exceed value of the undefinitized contract actions MDA initiated in 2018 far exceeded previous years we reviewed. Transparency in test cost estimates: As we reported in May 2017, MDA requests more than $1 billion in funding each fiscal year for the tests outlined in its integrated test schedule based on MDA’s internally developed test cost estimates. However, our analysis found these estimates were inconsistent and lacked documented traceability. A cost estimate is the summation of individual costs using established methods and valid data. Developing and maintaining reliable cost estimates ensures the appropriate amount of funds are needed when requested and for the expressed purpose. We found, however, in May 2017, MDA’s testing budget lacked transparency and could be improved. Specifically, we found that MDA’s annual budget submission did not provide insight into the funding for each specific test. MDA regularly makes changes to its test schedule without reporting the impacts to its costs and funding needs. Without a breakout of MDA’s costs by test in its annual budget submission and BAR, how many times or how much funding has been requested, received, or used for a specific test will continue to be unclear. Therefore, we recommended that MDA break out funding request by test. DOD did not concur with our recommendation and stated that MDA’s current approach for assigning resources prior to the test execution, is adequate. We continue to believe that breaking out funding requests by test will improve transparency into planned versus actual test costs and aid departmental and congressional decision makers as they make difficult choices of where to invest limited resources. MDA also continues to struggle with fully achieving its annual flight testing goals. After MDA revised its approach to developing the annual Integrated Master Test Plan in 2009, in February 2010, we recognized the new test schedule’s potential to address prior issues with shifting testing requirements or test dates, and adding or deleting tests. MDA also focused its testing to collect data necessary to support the development of models and simulations. However, MDA’s test plan has not stabilized. Since it formalized its approach in 2010, MDA has continued to revise its test schedule frequently by adding new tests, and deleting or delaying tests, in some cases, multiple times and further into future fiscal years. As a result, less testing is being conducted prior to delivery than originally planned, which means less data are available to understand BMDS capabilities and limitations. Specifically, preliminary observations from our fiscal year 2019 review show that from fiscal year 2010 through fiscal year 2019, MDA has conducted only 37% of its planned testing as originally scheduled, while the remainder has been either been delayed, deleted or conducted in a later fiscal year, as shown in figure 2. In addition, we reported in June 2019 that European Phased Adaptive Approach (EPAA) Phase 3 testing against intermediate range ballistic missiles (IRBM) had been reduced by 80 percent and MDA no longer planned to conduct a flight test against a raid—a likely tactic in a real- world attack—prior to delivery. The lack of raid flight testing prevented the accreditation of Aegis BMD models for assessment under those circumstances in all fiscal year 2019 ground tests that included Aegis BMD. MDA is currently at a pivotal crossroads, needing to balance its ability to pursue new and advanced efforts while also maintaining its existing portfolio of BMDS elements that have not transferred to the military services as originally planned. The new and advanced efforts, such as hypersonic defense and a Next Generation Interceptor (NGI) for GMD, are research and development-intensive tasks, which carry significant technical risks and financial commitments. If MDA’s elements are not transferred as originally intended, as they move further into production and operations and sustainment these elements will continue to consume a growing portion of the agency’s budget. MDA and military services have taken some actions to prepare for transferring the BMDS elements; however, the actions have not enabled transfer primarily due to a lack of early and frequent coordination, according to officials from the Undersecretary of Defense for Research and Development and Acquisitions and Sustainment. Consequently, there are overarching concerns related to transfer such as who funds the sustainment of the elements which have not been resolved. Congress and the Secretary of Defense have directed multiple reviews to determine how to address these concerns and chart a path forward for MDA. Chairman Cooper, Ranking Member Turner, and members of the Subcommittee, this concludes my prepared statement. I would be happy to respond to any questions you may have at this time. If you or your staff members have any questions about this testimony, please contact Cristina T. Chaplain, Director, Contracting and National Security Acquisitions, at (202) 512-4841 or Chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are LaTonya Miller (Assistant Director), Steven Stern (Analyst in Charge), Matthew Ambrose, Pete Anderson, Helena Johnson, Michael Moran, Wiktor Niewiadomski, Miranda Riemer, Brian Tittle, and Alyssa Weir. Missile Defense: Delivery Delays Provide Opportunity for Increased Testing to Better Understand Capability. GAO-19-387. Washington, D.C.: June 2019. Missile Defense: The Warfighter and Decision Makers Would Benefit from Better Communication about the System’s Capabilities and Limitations. GAO-18-324. Washington, D.C.: May 2018. Missile Defense: Some Progress Delivering Capabilities, but Challenges with Testing Transparency and Requirements Development Need to Be Addressed. GAO-17-381. Washington, D.C.: May 2017. Missile Defense: Ballistic Missile Defense System Testing Delays Affect Delivery of Capabilities. GAO-16-339R. Washington, D.C.: Apr. 2016. Missile Defense: Opportunities Exist to Reduce Acquisition Risk and Improve Reporting on System Capabilities. GAO-15-345. Washington, D.C.: May 2015. Missile Defense: Mixed Progress in Achieving Acquisition Goals and Improving Accountability. GAO-14-351. Washington, D.C.: Apr. 2014. Missile Defense: Opportunity to Refocus on Strengthening Acquisition Management. GAO-13-432. Washington, D.C.: Apr. 2013. Missile Defense: Opportunity Exists to Strengthen Acquisitions by Reducing Concurrency. GAO-12-486. Washington, D.C.: Apr. 2012. Missile Defense: Actions Needed to Improve Transparency and Accountability. GAO-11-372. Washington, D.C.: Mar. 2011. Defense Acquisitions: Missile Defense Transition Provides Opportunity to Strengthen Acquisition Approach. GAO-10-311. Washington, D.C.: Feb. 2010. Defense Acquisitions: Production and Fielding of Missile Defense Components Continue with Less Testing and Validation Than Planned. GAO-09-338. Washington, D.C.: Mar. 2009. Defense Acquisitions: Progress Made in Fielding Missile Defense, but Program is Short of Meeting Goals. GAO-08-448. Washington, D.C.: Mar. 2008. Defense Acquisitions: Missile Defense Acquisition Strategy Generates Results but Delivers Less at a Higher Cost. GAO-07-387. Washington, D.C.: Mar. 2007. Defense Acquisitions: Missile Defense Agency Fields Initial Capability but Falls Short of Original Goals. GAO-06-327. Washington, D.C.: Mar. 2006. Defense Acquisitions: Status of Ballistic Missile Defense Program in 2004. GAO-05-243. Washington, D.C.: Mar. 2005. Missile Defense: Actions Are Needed to Enhance Testing and Accountability. GAO-04-409. Washington, D.C.: Apr. 2004. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "For over half a century, the Department of Defense has funded efforts to defend the United States from ballistic missile attacks. From 2002 to 2020, MDA has received about $174 billion to develop the BMDS and has requested about $9.2 billion for fiscal year 2021. The BMDS consists of diverse and highly complex land-, sea-, and space-based systems and assets located across the globe. This statement summarizes lessons that GAO has identified from its prior reviews of MDA starting in 2004 that can be applied to strengthen the transparency and acquisition practices for developing and fielding missile defense elements. Specifically, this testimony provides information on (1) steps MDA has taken to increase transparency and reduce acquisition risks; and (2) ongoing challenges associated with improving transparency and reducing high risk acquisition practices. In our prior work, GAO reviewed key MDA management documents including annual program reviews, tests plans and budget documents. We also interviewed officials from MDA and from other key DOD offices. The Missile Defense Agency (MDA) has taken important steps in recent years to improve management practices, reduce acquisition risks, and deliver capabilities to defend the United States and its allies from ballistic missile attacks. Specifically, MDA has made advances across a broad range of management activities, such as improving stakeholder outreach, reducing concurrency, (broadly defined as the overlap between product development, testing, and production), improving testing of the Ballistic Missile Defense System (BMDS) and increasing transparency of its progress. MDA has also made progress toward improving homeland and regional defense. However, MDA can go further to align itself with best practices as it faces ongoing challenges associated with improving transparency and reducing high risk acquisition practices. These challenges include: Stakeholder involvement: MDA has improved its outreach to stakeholders, including the intelligence community and other DOD stakeholders, however, opportunities remain, such as obtaining more input from the defense intelligence community. While MDA is not required to do so, the community is uniquely positioned to help keep pace with emerging threats and validate threat models. Concurrency: MDA has taken steps to reduce concurrency, but falls back on this practice when experiencing developmental delays or schedule pressures. The recently canceled Redesigned Kill Vehicle (RKV) initially aligned production decisions with flight testing. However, in response to advancements from North Korea, development and production were performed concurrently and flight testing was reduced, thereby removing the safeguards that had been put into place. Flight test schedule changes: Despite initiating a new approach to developing its flight test schedule in 2009, MDA continues to struggle with execution. Namely, MDA is frequently revising its annual schedule by adding new tests, and deleting or delaying others—sometimes multiple times. Transparency of test cost estimates: MDA regularly makes changes to its test schedule without reporting the impact to its costs and funding needs. We continue to believe that breaking out funding requests by test will improve transparency into planned versus actual test costs and aid departmental and congressional decision makers as they make difficult choices of where to invest limited resources. MDA is at a pivotal crossroads, needing to balance its ability to pursue new and advanced efforts while also maintaining its existing portfolio. Congress and the Secretary of Defense are undertaking multiple reviews to determine how to address these concerns and chart a path forward for MDA. GAO is not making any new recommendations in this statement. GAO has previously recommended that MDA take steps to increase transparency and align its acquisition approach to reduce high-risk practices. MDA concurred with certain recommendations and is taking steps to implement them.", "document_type": "gao"}
{"report": "In 1991, following the collapse of the Soviet Union, the U.S. government authorized the President to establish the Nunn-Lugar Cooperative Threat Reduction (CTR) program to provide nuclear security assistance to Russia and the former Soviet states. At the time, there were significant concerns about Russia’s ability to maintain adequate security over its large numbers of nuclear weapons and vast quantities of weapons-usable nuclear materials. In 1995, DOE established the Material Protection, Control, and Accounting (MPC&A) program to equip Russia and other countries with modern nuclear material security systems and promote effective nuclear material security practices. The CTR umbrella agreement with Russia—which established an overall legal framework under which the United States would provide nuclear security assistance to Russia—expired in June 2013. Joint nuclear security activities in Russia, however, continued under a multilateral agreement and a related bilateral protocol. In December 2014, in response to U.S. sanctions over Russian actions in Ukraine, the Russian government ended nearly all nuclear security cooperation with the United States. Until then, the United States had been gradually transitioning responsibility to Russia for supporting its nuclear material security systems, and it was anticipated that the U.S. MPC&A program would continue to help Russia sustain its nuclear material security systems until January 1, 2018. See figure 1 for a timeline of major events during the period of cooperation. Starting with fiscal year 2015, and with each fiscal year since, language in annual appropriations laws and national defense authorization acts has largely prohibited NNSA from funding new efforts in Russia, including nuclear material security assistance, unless the prohibition is waived by the Secretary of Energy under certain conditions. Russia’s weapons-usable nuclear materials are stored and processed at more than two dozen sites overseen by a number of Russian entities, and the MPC&A program’s focus was on 25 of these sites at the time of our last report in 2010. The Russian State Corporation for Atomic Energy (Rosatom) is the Russian agency that manages much of Russia’s nuclear security enterprise, including seven nuclear weapons complex sites located in closed cities. These sites store and process the nuclear materials used in Russia’s nuclear weapons. Of the other 18 sites, many are overseen by Rosatom, but some are independent of Rosatom or managed by other Russian government entities. These sites often hold HEU and plutonium for research reactors or for other civilian purposes. See figure 2 for the location of the 25 Russian nuclear material sites. Other Russian government organizations with responsibilities in nuclear security include the following: Russian Ministry of Foreign Affairs (MFA). MFA is responsible for overseeing Russian policy and agreements for cooperation with the United States, including cooperation on nuclear security. Russian Federal Service of Environmental, Technological, and Nuclear Supervision (Rostekhnadzor). Rostekhnadzor is the regulator responsible for Russia’s civilian nuclear facilities. Russian Ministry of Industry and Trade (Minpromtorg). Minpromtorg coordinates nuclear material security activities and develops nuclear material security regulations for Russian naval shipbuilding sites, including Sevmash Shipyard, the primary builder of nuclear submarines for the Russian Navy. Russian Ministry of Defense. DOD and NNSA supported Russian efforts to secure Russian Ministry of Defense nuclear warheads and strategic rocket sites. That work is outside the scope of this report. The MPC&A program was the primary NNSA program that worked with Russia to help improve Russia’s ability to secure its nuclear materials and its nuclear warheads. To secure Russia’s nuclear materials, the program consisted of three main efforts: Site-level projects. NNSA managed MPC&A projects at the 25 Russian nuclear material sites to upgrade security systems at those sites. Teams of specialists from across DOE’s national laboratories, referred to as U.S. project teams, identified and carried out MPC&A upgrades on behalf of NNSA. MPC&A includes the following types of security systems, among other things: physical protection systems, such as fences around buildings containing nuclear materials and metal doors protecting rooms where nuclear materials are stored; material control systems, such as seals attached to nuclear material containers to indicate whether material has been stolen from the containers, and badge systems that allow only authorized personnel into areas containing nuclear material; and material accounting systems, such as nuclear measurement equipment and computerized databases to inventory the amount and type of nuclear material contained in specific buildings and to track their location. Material control and material accounting are collectively known as material control and accounting. National-level projects. NNSA managed cross-cutting projects to enhance Russia’s national-level infrastructure to sustain MPC&A systems for nuclear materials, including enhancing Russian nuclear security culture, developing Russian regulations for MPC&A operations, and strengthening Russian inspection and oversight capabilities. Sustainability support for individual sites. NNSA also fostered development of MPC&A sustainability practices and procedures at the Russian nuclear material sites based on seven sustainability elements, such as the presence at the site of an effective MPC&A management structure that plans, implements, tests, and evaluates the site’s MPC&A systems. Based on our review of available NNSA documentation and interviews with project team personnel, we found that NNSA had completed many— but not all—site-level MPC&A projects at the 25 Russian nuclear material sites when cooperation ended in 2014. NNSA also made progress on 11 cross-cutting projects that were intended to improve Russia’s national- level nuclear material security infrastructure. In addition, NNSA made progress on supporting the ability of the 25 Russian sites to sustain nuclear material security efforts. However, at the time cooperation ended, NNSA still had a number of concerns about both the sustainability of nuclear security efforts at the 25 sites and the state of Russia’s national- level nuclear material security infrastructure. Based on our review of available NNSA documentation and interviews with stakeholders, we determined that NNSA completed many MPC&A projects at the 25 Russian nuclear material sites, and stakeholders said that these upgrades significantly improved the state of nuclear material security at the sites. In particular, they told us that during the early years of the MPC&A program, the program completed upgrades focused primarily on the most significant security gaps, and in later years the program became more focused on transitioning the responsibility for sustaining nuclear security efforts to Russia. However, not all work was completed before cooperation ended, and project team members told us that the extent of completion varied by site. For example, project team members estimated that 90 percent of MPC&A projects were completed at one site, but that projects at other sites had lower levels of project completion. NNSA was unable to provide a complete set of documents detailing all projects completed and not completed across the 25 sites because several projects were consolidated into continuing programs and have not yet been closed out. In addition, the available site documentation did not always include detailed information on all projects completed or not completed. As a result we could not quantify how much planned work was completed and not completed when cooperation ended across all 25 sites. However, based on our review of available NNSA documents, we were able to identify many completed projects that included specific types of physical protection measures, material access controls, and material accounting upgrades. Project team members we interviewed and documentation we reviewed also indicated that some projects were not completed when cooperation ended. NNSA documentation identifies a variety of uncompleted projects at specific sites, such as not constructing or upgrading perimeter fencing, not replacing aging physical protection equipment, and not upgrading entry control points with vehicle radiation monitors. For example, at one site there were several kilometers of modernized perimeter fencing, guard towers, and sensors that had not been completely installed by the time cooperation ended, according to NNSA documents and project team members. Project team members told us that the site had plans to complete these projects. However, because Russia ended cooperation, the project team was unable to verify that the equipment was installed or operating appropriately. Similarly, project team members told us about two major efforts at another site that were terminated by Russia when cooperation ended: a $1 million project to relocate the guard force building to reduce the reaction time for protective forces and a $300,000 project to update software for the central alarm station and other security systems. According to project team members, the contracts were agreed to and associated costs obligated by NNSA, but Russia ended cooperation before signing the agreements. In addition, in our 2010 classified report, we found that NNSA faced challenges in implementing MPC&A upgrades against insider and outsider threats at some Russian nuclear material facilities to reduce the risk of material theft. At the time of the 2010 report, NNSA had proposed MPC&A upgrades at certain Russian sites to address these concerns, and GAO found that progress in implementing upgrades at some locations and in some MPC&A technical areas had been limited. For our classified report issued in December 2019, we asked NNSA for an update on the status of these upgrades; in response to our request, NNSA officials told us that due to a lack of cooperation, they had not received additional information from Russian counterparts to determine the status of these upgrades. In addition to site-level MPC&A security projects, NNSA managed 11 cross-cutting projects to support Russia’s national-level nuclear material security infrastructure, such as projects to enhance Russian nuclear security culture, develop Russian regulations for MPC&A operations, and strengthen Russian MPC&A inspection and oversight capabilities. We found that—at the time cooperation ended in 2014—NNSA had made substantial progress on its cross-cutting projects. NNSA reported that work was fully completed or mostly completed on at least 10 of the 11 cross-cutting projects by the time cooperation ended. However, NNSA could not provide complete documentation detailing the level of progress for some of these projects. See table 1 below for a description of these project areas. We found that NNSA had planned to do more work on some national- level projects, but that the end of cooperation in 2014 resulted in some planned work not being completed. For example, in the case of regulations development, project team members told us that the project teams had planned to develop numerous regulations with Rosatom, but these were not completed because of the end of cooperation. As part of its plan to shift to Russia the responsibility for nuclear material security efforts, NNSA supported the adoption of MPC&A sustainability practices and procedures at the individual Russian nuclear material sites based on seven “sustainability elements.” NNSA identified these elements, such as performance testing of systems to evaluate MPC&A effectiveness, as being fundamental to the long-term sustainability of a modern nuclear material security system. See table 2 below for more information about the seven sustainability elements. To determine a site’s ability to sustain its security systems, project teams periodically assessed each site based on the seven elements, and rated sites in each element on a scale from low to high. In our 2010 classified report, we reported the results of these sustainability assessments across the 25 Russian nuclear material sites and found that the MPC&A program had made limited progress and faced challenges in developing effective practices and procedures consistent with the seven elements of sustainability. For our classified report issued in December 2019, we reviewed and reported on the most recent sustainability assessments, largely conducted between 2012 and 2014. We compared the ratings from the most recently completed site sustainability assessments for the same 25 sites to the ratings we reported in 2010. We found that sustainability ratings generally improved, but low scores persisted at many sites and in some sustainability areas. For example, we found that the number of high ratings increased over this period by about half, and the number of low ratings decreased by about half. We believe this indicates general progress in improving sustainability across the sites. Of the seven sustainability elements, the MPC&A organization sustainability element was the element most frequently rated as “high” in the most recent assessment, and it showed the most improvement across the 25 sites. This indicates that the ability of Russian site organizations to plan and coordinate MPC&A operations had improved. We also found in our review of these assessments that NNSA had continuing concerns when cooperation ended about both the sustainability of MPC&A upgrades at individual Russian sites and the state of the national-level nuclear material security infrastructure in Russia. In their final reports after cooperation ended, U.S. project teams documented ongoing concerns with the sustainability of MPC&A upgrades at Russian nuclear material sites. We reviewed the concerns in the 25 final site summary documents and interviewed project team members who provided additional examples of these concerns. Based on our documentation review and interviews with project team members, we identified the six most common areas of concerns, including: (1) the responsiveness of protective forces, (2) performance testing the effectiveness of MPC&A systems, (3) sustainment funding, (4) physical protection systems, (5) nuclear security culture, and (6) access and cooperation at Russian sites. Stakeholders we interviewed highlighted a number of national-level concerns in other areas, such as the state of security equipment. Project team members were concerned that some of the equipment provided in the early years of cooperation had become outdated or obsolete by the time cooperation ended, such as surveillance cameras and monitoring equipment, and would need to be replaced. There is little specific information available about the current state of security at Russian nuclear material sites, though anecdotal evidence suggests that nuclear material security regulations have improved and that Russia funds some nuclear security efforts. We interviewed DOE officials and national laboratory personnel about security risks and threats to Russian nuclear material security. The details of these conversations are classified. However, according to nongovernmental experts we interviewed, the theft of nuclear materials by insiders is currently considered the greatest threat to Russia’s nuclear materials. According to stakeholders, little information is available about site-level security currently at the 25 sites holding Russian nuclear material, including the status of U.S. upgrades funded through the MPC&A program. Stakeholders told us that this is primarily because U.S. personnel no longer have access to the sites to observe security improvements and discuss MPC&A practices with Russian site personnel. According to DOE officials, the ability of U.S. project teams and other personnel to visit Russian nuclear material sites helped provide transparency into the state of Russian security at these facilities, such as the status of radiation portal monitors at entry points within nuclear material storage buildings. Since the end of cooperation, few U.S. personnel have visited Russia’s nuclear material sites, greatly limiting transparency into the status of U.S. security investments and Russian security practices. According to NNSA officials and U.S. project team personnel, NNSA documentation—such as the U.S. project team closeout documents that are referred to above—are based on observations primarily from 2014 or earlier. This documentation provides the most recent direct assessments of security at the site level. These officials stated that while such reports are useful for identifying the state of Russian nuclear material site security at the time cooperation ended, they likely do not provide an accurate picture of the nuclear material security at the 25 sites currently. Regarding national-level efforts in Russia to support nuclear security in the country, stakeholders we interviewed said that information exists in two main areas: development of nuclear security regulations and nuclear security funding. Development of nuclear security regulations. According to stakeholders, Russia has improved its nuclear security regulations in recent years, including since cooperation ended in 2014. Although U.S. efforts to help Rosatom develop modern MPC&A regulations ended in 2014, NNSA has continued work with Rostekhnadzor to improve Russian nuclear material security regulations through a national-level MPC&A sustainability project. Stakeholders said that this project has resulted in Russian nuclear security regulatory improvements. For example, this project provided technical support on 11 regulations, including regulations to improve vulnerability assessments of nuclear sites and nuclear materials in transit. However, stakeholders also noted some limitations. For example, they stated that compliance with regulations at nuclear material sites is mostly unknown. Similarly, the effectiveness of enforcement in cases of noncompliance is unknown, though fines are thought to be negligible. Nuclear security funding. Information on nuclear security funding is limited, according to stakeholders. Some stakeholders we interviewed stated that, based on their experiences and conversations with Russian officials, they believed that Russia was generally providing sufficient funding for nuclear material security at sites. However, others doubted that Russia was providing sufficient resources to replace the funding lost when the U.S. MPC&A program ended. Stakeholders generally agreed that funding for nuclear security likely varies by site. A few stakeholders expressed concern that security at nuclear material sites could be one of the first areas cut during an economic downturn, as nuclear security is not seen to be as significant a priority for site managers as other operations and revenue-generating activities at the sites. We interviewed DOE officials and national laboratory personnel about security risks and threats to Russian nuclear material security. The details of these conversations are classified. However, according to nongovernmental experts we interviewed, the theft of nuclear materials by insiders is currently considered the greatest threat to Russia’s nuclear materials. According to nongovernmental experts we interviewed, Russia’s nuclear security culture generally does not prioritize protection against the threat of nuclear material theft by insiders, a threat that modern nuclear security systems are designed and maintained to prevent. For example, experts said that Russian nuclear material site managers were more likely to devote resources—such as training, manpower, and funding— to measures that protect facilities from outsider threats, and less likely to devote resources to measures that protect facilities against insider threats. Experts told us that while the MPC&A program advanced Russian appreciation of the insider threat during the period of cooperation, they were concerned that—without U.S. influence and training—protection against insider threats would still be insufficient and likely ignored unless the Russian government required such protection, which was not the case when cooperation ended. As a result, according to experts, Russian sites are likely not currently supporting MPC&A systems adequately to counter insider threats. One nongovernmental expert noted that Russian security services have assumed greater control and tightened security in the closed cities that contain the vast majority of Russia’s nuclear materials, and that this may have reduced the near-term threat from insiders. However, according to this expert, over time this reliance on the security services could create vulnerabilities. For example, some Russian sites may rely too heavily on the physical security elements of nuclear security systems—such as guard forces—to protect nuclear materials and may become complacent in modernizing other elements, such as material control and accounting practices to deter and prevent insider theft risks, or measures that can protect against other emerging, nontraditional threats such as drone or cyber risks. According to nongovernmental experts, other factors in the country may also exacerbate the risk of theft posed by both outsiders and insiders to Russia’s nuclear materials. For example, experts said the existence of massive amounts of weapons-usable nuclear materials at many dispersed sites across Russia is the primary factor that makes Russia’s nuclear materials a greater threat than the nuclear materials held in most other countries. In addition, according to experts, persistent corruption and existing terrorist groups near some of the closed cities are other contributing factors that could further increase the risk of theft. According to stakeholders, there could be opportunities to help Russia improve aspects of its nuclear security system that NNSA and others identified as continuing risks. However, stakeholders noted that any future cooperation would likely be limited in scope and would face considerable political challenges. According to stakeholders we interviewed, there could be opportunities for future U.S.-Russia cooperation to address some of the continuing nuclear security risks in Russia. However, stakeholders said that any future cooperation would likely differ dramatically from the donor-recipient model of the past MPC&A program. The Russian government would likely expect to be treated as an equal and would not want to be seen as a recipient of U.S. funds for infrastructure improvements. Therefore, the scope of future cooperation would likely be a limited partnership, would primarily involve training and information sharing rather than directly supporting security upgrades at Russian sites, and would require fewer U.S. resources than the past MPC&A program did. Stakeholders told us that engagement and cooperation are important because of the size of the Russian nuclear complex, the large amounts of Russian nuclear material, and the continuing security concerns in certain areas. Stakeholders told us they believed there would be security benefits to the United States in resuming nuclear security cooperation with Russia in some form. Stakeholders generally identified increased transparency and advancing security best practices as the two main benefits to nuclear security cooperation. Stakeholders we spoke to identified examples of opportunities for cooperation that could support U.S. interests by providing information on the security of Russia’s nuclear materials and by helping Russia improve nuclear material security practices and procedures. These include the following: Exchange of best practices. Stakeholders noted that the United States and Russia could share MPC&A best practices in conferences and workshops. Best practices could cover areas such as performance testing of MPC&A systems, insider threat protection, and material control and accounting. Some stakeholders said that Russian expertise, such as in nuclear forensics, could increase U.S. knowledge and potentially improve U.S. practices in certain areas. Technical exchanges. Stakeholders told us that there could be benefits to both the United States and Russia from reciprocal technical exchanges or meetings of nuclear security experts to review specific, technical MPC&A practices that each country employs. National laboratory personnel noted that past exchanges under the MPC&A program allowed Russian personnel to view MPC&A systems at U.S. facilities, which helped Russian personnel understand the features of modern MPC&A systems, such as insider threat prevention measures. U.S. personnel participated in reciprocal visits to view security measures at sites in Russia, which helped them understand Russian security practices. Stakeholders told us that such technical exchanges could help U.S. personnel better understand the state of Russian nuclear security funding and current Russian practices. Training. Experts and national laboratory personnel noted that training Russian personnel on technical matters—such as how to conduct comprehensive vulnerability assessments—could improve Russian security practices. Conversations on legal agreements. Some stakeholders said that initiating conversations with Russia on the status of existing but suspended legal agreements could provide an opening for other forms of cooperation. For example, a few stakeholders mentioned an existing—but suspended—research and development agreement from 2013 under which future nuclear security cooperation might be pursued if both parties were interested in reactivating the agreement. Cooperation within multilateral organizations. Some stakeholders noted that existing multilateral organizations, such as the International Atomic Energy Agency (IAEA) and the Global Initiative to Combat Nuclear Terrorism, could provide venues for the United States to pursue cooperative opportunities with Russia. For example, Russia and the United States could cooperate on developing recommendations to the IAEA on physical protection measures for nuclear material, which could then be shared with IAEA member states. Other opportunities. The Nuclear Threat Initiative, a U.S. nongovernmental organization (NGO), and the Center for Energy and Security Studies, a Russian NGO, coauthored a report that identified 51 mutually beneficial opportunities to cooperate in nuclear security, nuclear safety, nuclear energy, nuclear science, and nuclear environmental remediation. For example, the report identifies an opportunity for Russian and U.S. experts to establish a joint research and development program to improve nuclear security technologies to address emerging threats to nuclear material storage sites, such as drones. Russia would likely insist that it and the United States be seen as equal partners under any future arrangement or program for cooperation on nuclear security, according to stakeholders. However, U.S. project team personnel told us that Russian nuclear material sites often lack the financial resources to pay travel costs for Russian personnel or to cover costs for venues or workshops necessary for training or the exchange of best practices. Therefore, the level of funding to support any potential future cooperation might be disproportionate between the United States and Russia. Because we were unable to obtain views from Russian officials and Russian nuclear material site representatives, we were unable to establish the extent to which Russia would be willing to pursue any form of nuclear material security cooperation with the United States, regardless of funding sources and requirements. Stakeholders we interviewed were generally pessimistic about cooperation under the current political and diplomatic climate, and they noted that the deterioration of political relations is the most significant challenge to any future cooperation. Stakeholders identified other specific challenges, including the following: Funding prohibition. Some stakeholders said that provisions in recent appropriations acts and National Defense Authorization Acts (NDAA) prohibiting NNSA from funding nuclear security activities in Russia have been obstacles to cooperating on nuclear security matters. In a report submitted to Congress in May 2019, NNSA stated that “the lack of ability to sign new contracts or engage on a modest scale denies NNSA the insights necessary to directly monitor nuclear material security in Russia and the sustainment of past security improvements.” According to U.S. officials and U.S. project team personnel, the prohibition largely prevents U.S. personnel from sharing best practices with and training Russian counterparts, and the existence of the prohibition discourages U.S. and Russian personnel from interacting and maintaining relationships. Although the acts allow the Secretary of Energy to waive the prohibition under certain conditions, no secretary has done so since a prohibition was first included in the fiscal year 2015 appropriations act. In addition, according to NNSA officials we interviewed, the language describing waiver requirements in NDAAs has become more restrictive in recent years. Initially, the Secretary of Energy could waive the prohibition on the basis of a notification to certain congressional committees that the waiver was in the national security interest of the United States, an accompanying justification, and the passage of 15 days. Starting with the fiscal year 2017 NDAA, however, a waiver can only be issued if it is necessary to address an urgent nuclear-related threat in Russia, and any such waiver requires concurrence from the Secretary of Defense and the Secretary of State. Russian conditions on cooperation. Stakeholders we interviewed said that Russia has set conditions on any future nuclear security cooperation. For example, they said that Russia has indicated that it is unwilling to discuss nuclear security cooperation with the United States unless the United States is willing to discuss related areas, such as nuclear energy, nuclear safety, and nuclear science. According to stakeholders, in the past the United States has been unwilling to discuss these other areas as a condition for cooperating on nuclear security. Russian antagonism to U.S. security efforts. Stakeholders noted antagonism at some levels of the Russian government toward U.S. nuclear security efforts. For example, although Russia participates in nuclear security efforts at the IAEA, some stakeholders noted that Russia regularly obstructs U.S. initiatives and recommendations in that organization. As noted above, stakeholders view the general deterioration of political relations between the United States and Russia as the greatest challenge to cooperation, and it is not clear whether Russia is prepared to reengage with the United States on these or other options for rekindling U.S.- Russian nuclear security cooperation. We reached out to the Russian government to request meetings with Russian government officials and representatives of nuclear material sites who could provide Russian perspectives on efforts to secure Russia’s nuclear materials, the status of past U.S. nuclear material security investments, and potential opportunities for cooperation. The Russian government declined our requests to meet with these officials and site representatives. Therefore, without Russian perspectives on the likelihood of possible future cooperation, we were unable to determine whether changes to U.S. policy, such as lifting the funding prohibition, would have any meaningful effect on the status of nuclear security cooperation between the United States and Russia. We provided a draft of the classified version of this report to NNSA for review and comment. NNSA had no comments on the report. We are sending copies of this product to the Senate Armed Services Committee, the NNSA Administrator, and the Secretaries of Defense and State. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. This report (1) examines the extent to which the National Nuclear Security Administration’s (NNSA) planned nuclear material security efforts in Russia were completed when cooperation ended and what nuclear security concerns remained, (2) describes what is known about the current state of nuclear material security in Russia, and (3) describes stakeholder views on potential opportunities for future U.S.-Russian nuclear security cooperation. For all three objectives, we identified and interviewed relevant stakeholders, including U.S. government officials from NNSA, the Department of Energy (DOE), the State Department, and the Department of Defense; experts on Russian nuclear security from academia and nongovernmental organizations (NGO); and knowledgeable personnel at six U.S. national laboratories that supported U.S. nuclear security efforts in Russia, including personnel at Brookhaven National Laboratory, Lawrence Livermore National Laboratory, Los Alamos National Laboratory, Oak Ridge National Laboratory, Pacific Northwest National Laboratory, and Sandia National Laboratory. We identified the stakeholders by contacting government agencies and NGOs with nuclear security expertise and asking them to identify other knowledgeable stakeholders. We reached out to these other knowledgeable stakeholders and interviewed those who responded and were willing to speak with us. To identify nongovernmental experts, we compiled a list of individuals who stakeholders identified as having expertise in the area of nuclear security in Russia. We also worked with a staff librarian to conduct an independent search of published literature to identify nongovernmental experts who had authored multiple publications related to Russian nuclear security. In addition, to ascertain whether an individual should be considered a nongovernmental expert, we considered other information, such as invitations to speak at nuclear security panels, being an editor of nuclear security related journals, and relevant positions in academic and other nongovernmental institutions. We interviewed six nongovernmental experts who fit these criteria. To examine the extent to which NNSA’s planned nuclear material security efforts in Russia were completed when cooperation ended and what nuclear security concerns remained, we reviewed documents prepared by NNSA and the national laboratories for each of the 25 nuclear material sites in Russia where the United States worked previously with Russia to improve security. To identify NNSA sustainability programs at a national level, we reviewed GAO reports and NNSA project documentation. We also reviewed NNSA guidelines that detailed how project teams were to support and assess the ability of Russian sites to sustain their material protection, control, and accounting (MPC&A) systems. We reviewed the NNSA documents that assessed site sustainability and analyzed how site sustainability had changed at sites by the end of cooperation. These documents included project team assessments for each of the 25 sites in seven different sustainability elements. In these assessments, project teams provided ratings from low to high on the extent to which sites were prepared to sustain these areas. We also reviewed NNSA documents and identified concerns that site teams documented about site sustainability. We then analyzed the concerns from the 25 sites and grouped similar concerns into categories. We developed these categories based on the similarity of the concerns, definitions of key nuclear security areas in NNSA documents, and professional judgement. We then identified the six concerns that appeared most frequently, which accounted for about 70 percent of all concerns. To describe what is known about the current state of nuclear security in Russia—in addition to interviews with our stakeholder group—we reviewed U.S. government and open-source documents. Specifically, we reviewed reports from the International Panel on Fissile Materials, the Nuclear Threat Institute, the National Academies of Science, and a national laboratory; articles on Russian nuclear security; and periodic reports on Russian nuclear security published by an expert independent consultant. In addition to general internet searches for published documents relating to Russian nuclear security and the MPC&A program, we conducted literature searches of published materials with assistance from a staff librarian; we excluded from our literature review any search results that were published prior to 2014 or were not related to nuclear material security in Russia. In addition to unclassified interviews with U.S. government officials on Russian nuclear material security, we received classified briefings from DOE. We requested threat and risk information relating to Russian nuclear material security from the Central Intelligence Agency, but we were not provided this information. To describe stakeholder views on potential opportunities for future U.S.- Russia nuclear security cooperation, we interviewed those in our stakeholder group identified above. We also reviewed administration plans and reports, including the National Security Strategy, the National Strategy for Countering Weapons of Mass Destruction Terrorism, and NNSA’s May 2019 Report to Congress describing NNSA’s funding of nuclear security improvements in Russia. To inform our understanding of the prohibition on NNSA’s expenditures on nuclear security in Russia, we reviewed laws since fiscal year 2015 that restricted relevant NNSA funding in some way. In addition, to obtain Russian perspectives on nuclear material security and past U.S. efforts, we requested—through the State Department and the U.S. Embassy in Moscow—interviews with Russian officials at relevant Russian agencies and representatives at five Russian nuclear material sites. However, the Russian government declined our request to meet with these officials and representatives. David Trimble, (202) 512-3841 or trimbled@gao.gov In addition to the contact named above, William Hoehn (Assistant Director), Dave Messman (Analyst in Charge), and Dan Will made key contributions to this report. Antoinette Capaccio, Ellen Fried, Greg Marchand, Dan Royer, and Sara Sullivan also contributed to this report.", "summary": "Russia possesses the world's largest stockpile of weapons-usable nuclear materials, largely left over from the Cold War. These nuclear materials could be used to build a nuclear weapon if acquired by a rogue state or terrorist group. Starting in 1993, and for the next 2 decades, DOE worked with Russia to improve security at dozens of sites that contained these nuclear materials. In 2014, following Russian aggression in Ukraine and U.S. diplomatic responses, Russia ended nearly all nuclear security cooperation with the United States. The Senate report accompanying the Fiscal Year 2019 National Defense Authorization Act includes a provision for GAO to review NNSA's efforts to improve Russian nuclear material security. This report (1) examines the extent to which NNSA had completed its planned nuclear material security efforts when cooperation ended and what nuclear security concerns remained, (2) describes what is known about the current state of nuclear material security in Russia, and (3) describes stakeholder views on opportunities for future U.S.-Russian nuclear security cooperation. To address all three objectives, GAO interviewed U.S. government officials, personnel from DOE's national laboratories, and nongovernmental experts. In this report, GAO refers to all of these groups as stakeholders. GAO also reviewed relevant U.S. government plans, policies, and program documentation. GAO requested the opportunity to interview Russian officials and representatives at nuclear material sites for this review, but the Russian government denied this request. Over more than 2 decades starting in the early 1990s, the Department of Energy (DOE) and its National Nuclear Security Administration (NNSA) completed many of their planned efforts to improve nuclear material security in Russia, according to DOE documentation, U.S. government officials, and nuclear security experts. These efforts, carried out primarily through NNSA's Material Protection, Control, and Accounting (MPC&A) program, included a range of projects to upgrade security at dozens of Russian nuclear material sites, such as the installation of modern perimeter fencing, surveillance cameras, and equipment to track and account for nuclear material. However, not all planned upgrades were completed before cooperation ended in late 2014. NNSA also completed many—but not all—of its planned efforts to help Russia support its national-level security infrastructure, such as by helping improve the security of Russian nuclear materials in transit. In addition, NNSA made some progress in improving each site's ability to sustain its security systems, such as by training Russian site personnel on modern MPC&A practices and procedures. NNSA documentation that GAO reviewed showed that by the time cooperation ended, Russian sites had generally improved their ability to sustain their MPC&A systems, but this documentation showed that concerns remained. According to stakeholders, there is little specific information about the current state of security at Russian nuclear material sites because U.S. personnel no longer have access to sites to observe security systems and discuss MPC&A practices with Russian site personnel. However, stakeholders said there is some information on national-level efforts. Specifically, stakeholders said that Russia has improved regulations for some MPC&A practices, and there are signs that Russian sites receive funding for nuclear material security, though it is unlikely that Russian funding is sufficient to account for the loss of U.S. financial support. Regarding threats to Russia's nuclear material, nongovernmental experts GAO interviewed raised concerns about the risk of insider theft of Russian nuclear materials. Experts stated that it is likely that Russian sites have maintained nuclear material security systems to protect against threats from outsiders, but it is unlikely that sites are adequately protecting against the threat from insiders. Stakeholders said that there may be opportunities for limited future cooperation between the two countries to help improve Russian nuclear material security. Such opportunities could include technical exchanges and training. These opportunities could provide the United States with better information about the risk posed by Russia's nuclear materials and could help address areas of concern, such as by training Russian personnel to help sites better address the insider threat. However, any potential cooperation faces considerable challenges, according to stakeholders, most notably the deterioration of political relations between the two countries. In addition, stakeholders said that cooperation is challenged by current U.S. law, which generally prohibits NNSA from funding nuclear security activities in Russia; by Russian antagonism toward U.S. proposals to improve nuclear material security internationally; and by Russian conditions for cooperation that the United States has not been willing to meet.", "document_type": "gao"}
{"report": "Prior to the enactment of the CFO Act, government reports found that agencies lost billions of dollars through fraud, waste, abuse, and mismanagement. These reports painted the picture of a government unable to properly manage its programs, protect its assets, or provide taxpayers with the effective and economical services they expected. Reported financial management problems included (1) unreliable financial information driven by widespread weaknesses in agency internal controls over financial reporting and obsolete and inefficient agency financial management systems and (2) financial reporting practices that did not accurately disclose the current and probable future cost of operating, permit adequate comparison of actual costs among executive branch agencies, or provide the timely information required for efficient program management. For example, in 1988, we reported on internal control problems such as the Department of Defense being unable to account for hundreds of millions of dollars in advances paid by foreign customers for equipment, weak controls permitting things such as over $50 million in undetected fraudulent insurance claims paid by the Federal Crop Insurance Corporation, millions of dollars in interest penalties because agencies paid 25 percent of their bills late, and over $350 million in lost interest because agencies paid their bills too soon. In 1990, Congress mandated financial management reform through enactment of the CFO Act. The CFO Act was the most comprehensive and far-reaching financial management improvement legislation enacted since the Budget and Accounting Procedures Act of 1950. The CFO Act established a Controller position at the government-wide level and a CFO position for each of the agencies identified in the act (referred to as the CFO Act agencies), provided for long-range planning, and began the process of preparing and independently auditing federal agency financial statements. The act aimed to strengthen internal controls, integration of agency accounting and financial management systems, financial reporting practices, and the financial management workforce. The act also called for systematic performance measurement and cost information. As figure 1 shows, a number of other financial management reforms were subsequently enacted to help improve federal financial management, some of which I will briefly discuss in my statement today. A chronological list of statutes cited in this report and selected additional financial management reforms is included in appendix II. The federal government has made substantial progress toward improving financial management and achieving the purposes of the CFO Act. Table 1 highlights some of the progress that has been made. The centralized leadership structures envisioned by the CFO Act—a Controller position at the government-wide level and a CFO position at each CFO Act agency—have been established. OMB’s Deputy Director for Management and Office of Federal Financial Management, headed by the Controller and Deputy Controller, have led reform efforts by developing and periodically updating guidance and initiatives in areas such as financial management systems, auditing, financial reporting, internal control, and grants management. The CFO Act also required OMB to submit to Congress, annually, a 5- year plan for improving financial management—mirrored in corresponding CFO Act agency plans. Among other things, the plan required a description of the existing financial management structure and changes needed; a strategy for developing adequate, consistent, and timely financial information; proposals for eliminating unneeded systems; identification of workforce needs and actions to ensure that those needs are met; a plan for the audit of financial statements of executive branch agencies; and an estimate of the costs for implementing the plan. The CFO Act also required annual financial management status reports government-wide and for executive branch agencies. From 1992 to 2009, OMB annually prepared comprehensive 5-year government-wide financial management plans. Agency CFOs have significantly contributed to improvements in financial management. According to the survey we issued to CFOs and deputy CFOs, some of these improvements include advising executive leadership on financial management matters and direction for agency financial operations and professional financial management personnel; taking steps to develop and maintain financial management systems; reducing duplicative financial management systems; resolving audit findings; supporting audits of the agency’s financial statements; helping to ensure the quality of financial information, and preparing the agency financial report and other financial reports. In addition, the CFO Council periodically met to advise and coordinate activities and initiatives, including those related to internal controls, financial management systems, and enterprise risk management. OMB stated that the CFO Council is also working on a workforce plan. In addition, the Department of the Treasury (Treasury) made contributions to improving federal financial management. Among other things, Treasury has developed and periodically updated government-wide guidance and tools to support federal financial reporting; issued, in coordination with OMB, the Financial Report of the U.S. Government since fiscal year 1997, which includes the government-wide consolidated financial statements; and developed a long-term vision for improving federal financial management. In 2010, Treasury established the Office of Financial Innovation and Transformation, which identifies and facilitates the implementation of innovative solutions to help agencies become more efficient and transparent, and Treasury also issues an annual message to agency CFOs to set the direction and goals of federal financial management. In 1990, OMB, Treasury, and GAO jointly established the Federal Accounting Standards Advisory Board (FASAB) to develop and promulgate accounting standards and principles for financial reporting in the federal government. In 1999, FASAB was recognized by the American Institute of Certified Public Accountants as the standard setter for generally accepted accounting principles for federal government entities. FASAB has issued 57 statements of federal financial accounting standards (SFFAS) that provide greater transparency and accountability over the federal government’s operations and financial condition, including SFFAS 36, Comprehensive Long-Term Projections for the U.S. Government, which requires the Statement of Long-Term Fiscal Projections as part of the government-wide consolidated financial statements. In addition, OMB, Treasury, and GAO have regularly provided guidance to agencies that improves transparency, consistency, and usefulness of financial reporting. Agencies have significantly improved the quality and timeliness of their financial reporting since the enactment of the CFO Act. As expanded by the Government Management Reform Act of 1994 (GMRA) and the Accountability of Tax Dollars Act of 2002 (ATDA), federal law now requires every CFO Act agency and most other executive agencies to annually prepare audited financial statements no later than March 1—5 months after the end of the federal fiscal year. However, OMB has accelerated this due date for audited financial statements. For the first time, for fiscal year 2005, all CFO Act agencies completed their audited financial statements by November 15, approximately 45 days after the close of the fiscal year, compared to the 60–90 day requirement for public companies filing with the Securities and Exchange Commission. For fiscal year 1996, the first year that all CFO Act agencies were required to prepare audited financial statements, six CFO Act agencies received an unmodified (“clean”) audit opinion on their respective entities’ financial statements, compared with 22 CFO Act agencies that received clean audit opinions for fiscal year 2018. Today, to demonstrate transparency and accountability to Congress and citizens, the CFO Act agencies make their annual performance reports and annual financial reports, which include audited financial statements, available on their websites. In addition, since fiscal year 1997, Treasury, in coordination with OMB, has annually prepared government-wide consolidated financial statements, which are available on Treasury’s website. Substantial benefits have been achieved as a result of the preparation and audit of financial statements, which provide useful and necessary insight into government operations, including federal agency accountability to Congress and citizens, including independent assurance about the reliability of reported financial information; greater confidence to stakeholders (governance officials, taxpayers, consumers, or regulated entities) that federal funds are being properly accounted for and assets are properly safeguarded; an assessment of the reliability and effectiveness of systems and related internal controls, including identifying control deficiencies that could lead to fraud, waste, and abuse; a focus on information security; early warnings of emerging financial management issues; and identification of noncompliance with laws and regulations, which can present challenges to agency operations. Our CFO survey respondents (18 of 23) agreed that preparation and audit of financial statements are greatly or moderately beneficial to federal agencies, noting that the financial audit process helped identify and eliminate material weaknesses in internal control, greatly strengthened internal control processes, and led to more discipline and integrity in federal accounting. Continuation of annual agency financial statement audits is critical to maintaining accountability and sustaining financial management improvements. Also, independent assurance that financial management information included in agency financial statements is fairly stated is an important element of accountability and provides agency management, OMB, Treasury, Congress, and citizens with assurances that the information is reliable and properly accounted for. A key goal of the CFO Act was to improve internal control to reasonably assure that the federal government’s financial management information is reliable, useful, and timely. Compared with 1990, internal control is markedly stronger. The number of material weaknesses in internal control over financial reporting—significant issues that create the potential for inaccurate financial information that would change or influence the judgment of a reasonable financial report user relying on the information—reported as part of financial statement audits has been significantly reduced. For fiscal year 2005, financial statement auditors reported no identified material weaknesses for only seven of 24 CFO Act agencies, based on their financial statement audits; by 2018, that number had doubled to 14. In addition, auditors identified and agencies fixed thousands of internal control problems over the past 3 decades. Further, Treasury and OMB have addressed many of the internal control problems related to the processes used to prepare the U.S. government’s consolidated financial statements. However, some internal control problems are long-standing, complex, and not quickly resolved, such as accounting for transactions between federal agencies. Annual financial statement audits also uncovered the significance of improper payments and prompted legislation to strengthen controls over improper payments. Agencies have made progress in estimating the amount of improper payments and implementing efforts to reduce them, but this remains an area of concern. We have reported improper payments as a material deficiency or weakness since the fiscal year 1997 initial audit of the U.S. government’s consolidated financial statements. For fiscal year 2018, 79 programs across 20 agencies reported estimated improper payments totaling about $151 billion. Since fiscal year 2003— when certain agencies were required to begin reporting estimated improper payments—cumulative improper payment estimates have totaled about $1.5 trillion. The annual financial statement audits, which include an assessment of information systems controls, surfaced widespread information security weaknesses. Since fiscal year 1997, we have reported information security as a material weakness in the audit of the U.S. government’s consolidated financial statements. We have also reported information security as a government-wide high-risk area since 1997. To address information security challenges surfaced by federal agency audits, Congress enacted the Federal Information Security Management Act of 2002 and its successor, the Federal Information Security Modernization Act of 2014. These laws require agencies to develop, document, and implement programs to provide security for the information and information systems that support agency operations and assets. One key purpose of the CFO Act and of the Federal Financial Management Improvement Act of 1996 (FFMIA) that followed was to improve federal agencies’ financial management systems. FFMIA requires CFO Act agencies to maintain financial management systems that substantially comply with (1) federal financial management systems requirements, (2) applicable federal accounting standards, and (3) the U.S. Government Standard General Ledger at the transaction level. Agencies have improved their compliance with FFMIA requirements. For fiscal year 2018, auditors reported that 16 of 24 CFO Act agencies’ financial systems substantially comply with FFMIA’s systems requirements for fiscal year 2018, up from four agencies in fiscal year 1997. Federal agencies have taken steps to implement new financial systems. While progress has been made in modernizing financial management systems, we have previously reported that efforts to modernize financial management systems have often exceeded budgeted cost, resulted in delays in delivery dates, and did not provide the anticipated system functionality and performance. For example, one-half (12 of 24) of the CFOs and deputy CFOs who responded to our survey indicated that they still use old systems and use obsolete software or hardware to perform financial management responsibilities. Some agencies have used migration of financial systems to external providers as part of their system modernization efforts, but others have experienced challenges in using shared services. For example, some CFO Act agencies have had difficulty in finding a provider with sufficient capacity and decided to modernize their financial system internally. Others that have attempted to move their financial system to a shared service provider failed to meet their cost, schedule, and performance goals. The federal government also has taken action aimed at reducing duplicative efforts by increasing agencies’ use of shared services for commonly used computer applications—such as payroll or travel. Over the past 15 years, there have been some notable shared services successes. For example, consolidating payroll services resulted in more than $1 billion in cost savings and cost avoidance over 10 years, according to Office of Personnel Management (OPM) estimates. In April 2019, OMB issued Memorandum M-19-16 on shared services, which among other things described the process and desired outcomes for shared services and established a governance and accountability model for achieving them. To help achieve the CFO Act’s purposes, the federal government established a financial management workforce structure, improving the quality of the federal workforce. Since then, steps have been taken to strengthen the federal financial management workforce, including the following: In 2000, the CFO Council and OPM worked together to align qualifications standards for accounting, auditing, and budget competencies with emerging financial management position requirements. In 2002, Congress and the President enacted legislation to empower OPM to provide agencies with additional authorities and flexibilities to manage the federal workforce and created the chief human capital officer (CHCO) positions and the CHCO Council to advise and assist agency leaders in their human capital efforts. In 2011, OPM and the CHCO Council created a working group that identified critical skills gaps in six government-wide, mission-critical occupations, including that of auditor. In 2017, OPM published a regulation requiring each CFO Act agency to develop a human capital operating plan describing agency-specific skills and competency gaps that are selected for closure and the strategies that will be implemented. While substantial progress has been made, additional attention is needed in several areas to help fully achieve the vision of the CFO Act and, in doing so, improve and modernize federal financial management. Based on the preliminary results from our ongoing review, we have identified several opportunities for enhancements that could help ensure that the CFO Act reaches its full potential. 1. To help ensure uniform responsibility, enhance strategic decision- making, and correct inconsistencies across government, amend agency CFO’s statutory responsibilities to ensure that they include all of the responsibilities necessary to effectively carry out financial management activities. Currently, responsibilities vary across agencies and do not include all key responsibilities that CFOs should possess. 2. To help ensure continuity in agency financial management operations when CFO vacancies occur, establish appropriate statutory responsibilities for deputy CFOs. This would minimize the effects of inevitable turnover in CFO positions. 3. Based on the maturity of federal financial management, extend the reporting frequency of the government-wide and agency-level financial management plans from annually to at least every 4 years (with timing to match the Government Performance and Results Act reporting requirements). In addition to the current government-wide financial management plan requirements, the plans should include actions for improving financial management systems, strengthening the federal financial management workforce, and better linking performance and cost information for decision-making. The government-wide plan should also include key selected financial management performance-based metrics. It is our view that OMB and Treasury should consult with the CFO Council, the Chief Information Officer Council, the Council of the Inspectors General on Integrity and Efficiency, GAO, and other appropriate financial management experts in preparing the government-wide plan. 4. To provide more complete and consistent measurement of the quality of agencies’ financial management, require OMB to develop, in consultation with the CFO Council, key selected performance-based metrics to assess the quality of an agency’s financial management, and changes therein. Examples of potential metrics include the number of internal control deficiencies, the number of internal control deficiencies corrected during the year, and the number of Antideficiency Act violations.The metrics should be included in the government-wide and agency-level financial management plans discussed above and agencies’ performance against the metrics reported in the annual status reports. Also, consider requiring auditor testing and reporting on the reliability of each agency’s reported performance against the metrics. 5. To reasonably assure that key financial management information that an agency uses is reliable, require agency management to (1) identify key financial management information, in addition to financial statements, needed for effective financial management and decision- making and (2) annually assess and report on the effectiveness of internal control over financial reporting and other key financial management information. Also, consider requiring auditor testing and reporting on internal control over financial reporting and other key financial management information. We provided a draft of the progress and opportunities for enhancements to OMB, Treasury, and OPM. OPM provided technical comments. OMB and Treasury generally agreed with enhancements 1 and 2, regarding CFOs’ and deputy CFOs’ statutory responsibilities. OMB generally disagreed with enhancement 3, regarding preparation of government- wide and agency-level financial management plans, stating that developing government-wide plans poses an administrative burden and is no longer relevant in light of the current state of financial management. However, we believe that a complete and integrated government-wide plan could help to ensure continuity in direction and a comprehensive understanding of the status and financial management challenges across government. Eight of the 10 financial experts we interviewed stated that without a government-wide financial management plan, the government lacks a clear strategic direction and agency improvement efforts may not appropriately address government-wide priorities. For enhancement 4, regarding performance metrics for agencies’ financial management, OMB generally disagreed, stating that it would be difficult to develop additional metrics that would apply to all agencies. We recognize the challenges in developing the metrics but continue to believe that a limited number of key metrics can be developed to effectively assess the quality of agencies’ financial management. For enhancement 5, regarding identifying key financial management information and assessing, reporting, and auditing internal control, Treasury generally agreed and OMB generally disagreed, noting that no action is needed and these controls are adequately addressed under existing initiatives and the enterprise risk management program contained in OMB guidance. We believe that a separate assessment is needed to reasonably assure that key agency financial management information used by the agency is reliable. Chairman Enzi, Ranking Member Sanders, and Members of the Committee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Dawn B. Simpson, Director, Financial Management and Assurance, at (202) 512-3406 or simpsondb@gao.gov or Robert F. Dacey, Chief Accountant, at (202) 512-3406 or daceyr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Phyllis Anderson (Assistant Director), LaDonna Towler (Assistant Director), Beryl Davis (Director), David Ballard, Jeremy Choi, Anthony Clark, Patrick Frey, Ryan Guthrie, Isabella Hur, Jason Kelly, Jason Kirwan, Chris Klemmer, Michael LaForge, Jill Lacey, Diana Lee, Christy Ley, Keegan Maguigan, Lisa Motley, Heena Patel, Matthew Valenta, Walter Vance, and William Ye. This testimony highlights some of the most significant achievements in federal government financial management since enactment of the Chief Financial Officers Act of 1990 (CFO Act) and some preliminary observations on how federal financial management can be enhanced. The information in this testimony is based on our ongoing review and analysis of relevant legislation; federal financial management guidance, such as Office of Management and Budget (OMB) circulars; reports on financial management issued by the Government Accountability Office (GAO), agency offices of inspector general, and others; summarization of interviews and a panel discussion with experts in federal financial management; and summarization of results of GAO surveys to federal chief financial officers (CFO), inspectors general (IG), and independent public accountants (IPA). To obtain perspectives of agency personnel on federal financial management, we developed and administered two web-based surveys from May 22, 2019, through August 5, 2019. We administered one survey to 47 individuals from the CFO offices of the CFO Act agencies and included individuals holding the position of CFO, acting CFO, deputy CFO, or equivalent at these agencies as of May 1, 2019. Of the 47 individuals we surveyed, 24 individuals responded, which resulted in a 51 percent response rate. We administered the other survey to 53 individuals holding the position of IG, deputy IG, or counsel to the IG at the CFO Act agencies as of May 1, 2019, and an additional 24 IPAs who have performed financial statement audits for these agencies since fiscal year 2014. Of the 77 individuals we surveyed, 29 individuals responded, which resulted in a 38 percent response rate. Results of both surveys only represent the views of those individuals who responded to the surveys and may not be representative of all individuals from the CFO offices, IG offices, or IPA offices of the CFO Act agencies. In May 2019, we hosted an expert meeting with the theme “CFO Act - Progress and Challenges.” When planning the meeting, we considered experts with a broad array of expertise. We had a total of eight experts participate, representing both the federal and private sectors. They included individuals who had served in auditing capacities and individuals who had represented federal entities being audited. Some experts were currently serving in their roles, and others had retired. Including experts with both present and past experiences helped to ensure an examination and discussion of the history of the CFO Act from its inception to the present. Topics for discussion included progress and challenges since enactment of the CFO Act, the role of the Department of the Treasury (Treasury) and OMB with regard to the act, and suggestions for improvements to financial management processes and systems. The meeting transcript was categorized by key points, including progress, challenges, OMB’s and Treasury’s roles, government-wide plans, financial management systems, shared services, leading practices, and proposed reforms or suggestions for improvements. Budget and Accounting Procedures Act of 1950, ch. 946 §§ 110-118, 64 Stat. 834 (Sept. 12, 1950). Federal Managers’ Financial Integrity Act of 1982, Pub. L. No. 97-255, 96 Stat. 814 (Sept. 8, 1982), codified at 31 U.S.C. § 3512(c), (d). Chief Financial Officers Act of 1990, Pub. L. No. 101-576, 104 Stat. 2838 (Nov. 15, 1990). Government Performance and Results Act of 1993, Pub. L. No. 103-62, 107 Stat. 287 (Aug. 3, 1993). Government Management Reform Act of 1994, Pub. L. No. 103-356, title IV, § 405, 108 Stat. 3410, 3415 (Oct. 13, 1994). Clinger-Cohen Act of 1996, Pub. L. No. 104-106, div. D & E, 110 Stat. 642 (Feb. 10, 1996), codified as amended at 40 U.S.C. § 11101, et seq. Federal Financial Management Improvement Act of 1996, Pub. L. No. 104-208, div. A, § 101(f), title VIII, 110 Stat. 3009-389 (Sept. 30, 1996), codified at 31 U.S.C. § 3512 note. Reports Consolidation Act of 2000, Pub. L. No. 106-531, 114 Stat. 2537 (Nov. 22, 2000), codified as amended at 31 U.S.C. § 3516. Accountability of Tax Dollars Act of 2002, Pub. L. No. 107-289, 116 Stat. 2049 (Nov. 7, 2002). Chief Human Capital Officers Act of 2002, Pub. L. No. 107-296, title XIII, subtitle A, 116 Stat. 2135, 2287 (Nov. 25, 2002). Improper Payments Information Act of 2002, Pub. L. No. 107-300, 116 Stat. 2350 (Nov. 26, 2002), codified as amended at 31 U.S.C. § 3321 note. Federal Information Security Management Act of 2002, Pub. L. No. 107- 347, title III, 116 Stat. 2899, 2946 (Dec. 17, 2002), codified as amended at 44 U.S.C. §§ 3551-3558. Department of Homeland Security Financial Accountability Act, Pub. L. No. 108-330, 118 Stat. 1275 (Oct. 16, 2004). Federal Funding Accountability and Transparency Act of 2006, Pub. L. No. 109-282, 120 Stat. 1186 (Sept. 26, 2006), codified as amended at 31 U.S.C. § 6101 note. Improper Payments Elimination and Recovery Act of 2010, Pub. L. No. 111-204, 124 Stat. 2224 (July 22, 2010), codified as amended at 31 U.S.C. § 3321 note. GPRA Modernization Act of 2010, Pub. L. No. 111-352, 124 Stat. 3866 (Jan. 4, 2011). Improper Payments Elimination and Recovery Improvement Act of 2012, Pub. L. No. 112-248, 126 Stat. 2390 (Jan. 10, 2013), codified as amended at 31 U.S.C. § 3321 note. Digital Accountability and Transparency Act of 2014, Pub. L. No. 113-101, 128 Stat. 1146 (May 9, 2014), codified at 31 U.S.C. § 6101 note. Federal Information Security Modernization Act of 2014, Pub. L. No. 113- 283, (Dec. 18, 2014), codified at 44 U.S.C. §§ 3551-3558. Carl Levin and Howard P. ‘Buck’ McKeon National Defense Authorization Act for Fiscal Year 2015, Pub. L. No. 113-291, div. A, title VIII, subtitle D, 128 Stat. 3292, 3438-3450 (Dec. 19, 2014) (commonly referred to as the Federal Information Technology Acquisition Reform Act). Federal Improper Payments Coordination Act of 2015, Pub. L. No. 114- 109, 129 Stat. 2225 (Dec. 18, 2015). Fraud Reduction and Data Analytics Act of 2015, Pub. L. No. 114-186, 130 Stat. 546 (June 30, 2016). National Defense Authorization Act for Fiscal Year 2018, Pub. L. No. 115- 91, div. A, title X, subtitle G, 131 Stat. 1283, 1586 (Dec. 12, 2017), codified at 40 U.S.C. § 11301 note (commonly referred to as the Modernizing Government Technology Act). Foundations for Evidence-Based Policymaking Act of 2018, Pub. L. No. 115-435, 132 Stat. 5529 (Jan. 14, 2019). The CFO Act provided agency CFOs with broad responsibilities for all financial management activities of their respective agencies, including financial management systems (including financial reporting and internal controls); agency financial management personnel, activities, and operations; preparation of financial statements; and monitoring of budget execution. The specific responsibilities assigned to CFOs vary among agencies and are inconsistent government-wide. We previously reported that CFO Act agencies need to ensure that CFOs possess the necessary authorities within their agencies to achieve change. For instance, because of the interdependency of the budget and accounting functions, some agencies have included both budget formulation and execution functions under the CFO’s authority while others have not. Most financial experts we interviewed agreed and the CFO Council and the Council of the Inspectors General on Integrity and Efficiency (CIGIE) reported that to allow for better strategic decision-making, CFO responsibilities should include budget formulation and execution, planning and performance, risk management and internal controls, financial systems, and accounting. Most experts agreed that standardizing the CFO portfolio across agencies would promote standardized financial management training and education and consistent skill sets across agencies, both at the executive and staff levels. The CFO Council and CIGIE have identified turnover of agency CFOs, even during the same administration, as a significant challenge. They also stated that major financial management improvement initiatives can take years to fully implement and realize, often outlasting the average tenure of a political appointee to a CFO position. With frequent CFO turnover and potentially lengthy intervals between official appointments, long-term planning and leadership continuity can be affected because career deputy CFOs, who frequently serve as acting CFOs during CFO vacancies, do not always have the same breadth of responsibilities as CFOs. Deputy CFOs can be better prepared to act for CFOs when there are vacancies if appropriate responsibilities are established for deputy CFOs. In our survey to CFOs and deputy CFOs, 17 of 24 respondents stated that the deputy CFO position should include all, most, or many of the same responsibilities as the CFO position. Additionally, some respondents to our survey replied that it is important for the deputy CFO to be able to step into the CFO position should there be a vacancy. CIGIE also said that deputy CFOs should be sufficiently empowered with more standard responsibilities to ensure effective succession planning. The CFO Act called for annual comprehensive government-wide 5-year plans for improving federal financial management. It also called for each agency CFO to annually prepare a plan to implement the government- wide plan prepared by the Office of Management and Budget (OMB). Moreover, it required annual government-wide and agency-level status reports. The OMB plans and status reports were to be submitted to Congress to enable comprehensive congressional oversight. Since it issued the 2009 report, OMB has neither prepared nor submitted to Congress the annual 5-year government-wide plans as required by the CFO Act. Instead, OMB stated that it is meeting the intent of the requirement by providing information in the President’s Management Agenda (PMA), in the annual government-wide consolidated financial statements, and in documents placed on Performance.gov and the CFO Council’s website. For the consolidated financial statements, the information is included in a section in the Management’s Discussion and Analysis (MD&A) entitled Financial Management. This section discusses several of the priorities and accomplishments in financial management for the prior and current fiscal years and in some cases discusses goals for the next fiscal year. In addition, according to OMB, financial management elements are being considered in implementing the 2018 PMA. The CFO Council, in coordination with OMB, has identified six financial management cross- agency priorities and is developing detailed plans for each. Two of these plans, results-oriented accountability for grants and getting payments right, have been completed and posted on Performance.gov. The others are being managed by executive steering committees comprising CFO Council–approved members. While the various MD&A Financial Management sections, the PMA, and other OMB documents contain relevant information about improvements in financial management, these documents do not provide a complete and integrated financial management strategy for making continued improvements and for reporting on the administration’s accomplishments in a comprehensive manner. In 2019, OMB proposed eliminating the CFO Act requirement for a separate comprehensive plan, arguing that this change would provide it with flexibility to report information that is most relevant to financial management in a manner that is most efficient. However, having a complete and integrated financial management plan would help to address long-standing, costly, and challenging concerns in financial management in a strategic, comprehensive, efficient, and cost-effective manner. Eight of the 10 financial experts we interviewed stated that without a government-wide financial management plan, the government lacks a clear strategic direction and agency improvement efforts may not appropriately address government-wide priorities. To hold people accountable and facilitate congressional oversight, a complete and integrated financial management plan should include the resources required and measure progress through interim milestones with completion dates. Several experts also stated that they believe that a government-wide plan should be done every few years instead of annually, but that the status report could continue to be prepared annually. A complete and integrated government-wide financial management plan and supporting agency plans, prepared every few years, could help ensure continuity in direction and a more comprehensive understanding of gauging progress toward addressing financial management challenges across government. The CFO Act calls for agencies to (1) develop and maintain integrated accounting and financial management systems that provide for, among other things, systematic measurement of performance and (2) develop and report cost information. While the Government Performance and Results Act of 1993 (GPRA) laid a foundation for results-oriented management, we found that agencies’ reported use of performance data to make decisions has generally not improved. While agencies have made efforts in this direction, opportunity exists to enhance the availability and reliability of performance and cost information, and better link this information for decision-making. One example of this is linking program performance to program cost. A number of agencies have implemented activity-based costing, which creates a cost model of an organization by identifying the activities performed, the resources consumed, and the outputs (products and services) that an organization produces. However, linking cost and performance information for effective decision-making has been challenging. Respondents to our CFO survey noted that agencies face challenges in (1) developing and maintaining an integrated agency accounting and financial management system (19 of 24 respondents), (2) developing and reporting cost information (19 of 24 respondents), and (3) having financial management systems that produce the needed financial data to help address agency performance goals (21 of 24 respondents). Agencies that lack readily available, reliable, and linked performance and cost information may not be able to effectively make financial management decisions that are based on dollars allocated and results achieved and thus may miss opportunities to reduce costs or enhance mission effectiveness. Agencies have limited financial management performance-based metrics (e.g., financial statement audit opinion and number of reported material weaknesses in internal control over financial reporting) to help them assess the quality of their financial management. A broader set of key selected financial management performance-based metrics can provide more complete analysis across the breadth of financial management functions. Examples of potential metrics include the number of internal control deficiencies, the number of internal control deficiencies corrected during the year, and the number of Antideficiency Act violations. Key selected financial management performance-based metrics, including identifying metrics in the government-wide and agency-level plans discussed above and reporting of agency performance against the metrics in the annual status reports, can help ensure that the federal government better manages and uses the resources entrusted to it. Also, auditor testing and reporting on each agency’s reported performance against the metrics can provide assurance that such information is reliable. The CFO Act required CFOs to develop and maintain an integrated agency accounting and financial management system that provides for complete, reliable, consistent, and timely information prepared on a uniform basis and that responds to agency management’s financial information needs. To ensure the reliability of financial information, agencies need effective internal controls. While agencies have made important progress in strengthening internal control, as noted earlier, the federal government faces many internal control problems. The following discusses three areas: assessing internal control over key financial management information, government-wide improper payments, and material weaknesses preventing an opinion on the U.S. government’s consolidated financial statements. Management may not have reasonable assurance that internal control over financial reporting and other key financial management information that the agency uses is reliable. Since fiscal year 1997, agency auditors’ assessments of the effectiveness of internal control over financial reporting have identified long-standing, as well as new, material weaknesses. As a result of new material weaknesses, a number of agencies have not been able to sustain “clean” audit opinions on their financial statements. In addition, continuing material weaknesses have hindered two CFO Act agencies, the Departments of Defense and Housing and Urban Development, and the government as a whole, from achieving clean audit opinions. For fiscal year 2018, auditors of CFO Act agencies reported a total of 41 material weaknesses. One key to strengthening internal control over financial reporting at federal entities has been OMB Circular No. A-123, which carries out OMB’s responsibility to provide guidelines for agencies to follow in evaluating their systems of internal control. In December 2004, OMB issued A-123, Appendix A, Internal Controls over Financial Reporting, which provided a methodology with which agency management could assess, document, and report on internal control over financial reporting. It emphasized management’s responsibility for establishing and maintaining effective internal control over financial reporting. Appendix A required CFO Act agency management to annually assess the adequacy of internal control over financial reporting, provide a report on identified material weaknesses and corrective actions, and provide separate assurance on the effectiveness of the agency’s internal control over financial reporting. The CFO Council subsequently issued the Implementation Guide for Appendix A in 2005. In 2018, OMB reported that since the issuance of OMB Circular No. A- 123’s Appendix A, federal agencies have made substantial progress in improving their internal controls over financial reporting. OMB referred to this as a rigorous process for agencies to separately assess internal control over financial reporting. Beginning in fiscal year 2018, however, OMB no longer requires such a process. On June 6, 2018, OMB issued an updated Appendix A, Management of Reporting and Data Integrity Risk. The revised Appendix A integrates internal control over reporting, along with internal controls over operations and compliance, in an overall assessment of the agency’s internal control. This reporting guidance includes internal control over financial reporting as well as over other financial and nonfinancial information. It also requires that agencies develop and maintain a data quality plan that considers the risks to data quality in federal spending data required by the Digital Accountability and Transparency Act of 2014 (DATA Act) and any controls that would manage such risks in accordance with OMB Circular No. A-123. Further, agency senior accountable officials are required to certify each quarter, among other things, that their data submissions under the DATA Act are valid and reliable. However, the appendix does not require a separate management assessment of internal controls over the reliability of federal spending data. As we previously reported, there are significant data quality problems related to the completeness and accuracy of DATA Act data. In addition, the Federal Financial Management Improvement Act of 1996 (FFMIA) requires CFO Act agencies and their auditors to determine whether agency financial management systems comply substantially with federal financial management systems requirements. However, such systems requirements are focused on preparing agency financial statements and do not generally include system requirements related to other key financial management information (e.g., performance information and cost information) needed for management decision- making. We have expressed concerns about the adequacy of financial management systems requirements contained in the Treasury Financial Manual. In our survey of CFOs and deputy CFOs, most (20 of 24) respondents said that ensuring data quality of financial information was somewhat, very, or extremely challenging. Without (1) identifying all key financial management information needed for effective financial management and decision-making, (2) separately assessing and reporting on the effectiveness of internal control over financial reporting and other key financial management information, and (3) independently assessing such controls, management may lack reasonable assurance of the reliability of such information. Improper payments have consistently been a government-wide issue, despite efforts to reduce them. Since fiscal year 2003, cumulative improper payment estimates have totaled about $1.5 trillion. Although agencies have made progress identifying and reducing improper payments, more work needs to be done to address this government-wide material weakness in internal control. We continue to report, as a government-wide material weakness in internal control, that the federal government is unable to determine the full extent to which improper payments occur and reasonably assure that appropriate actions are taken to reduce them. OMB stopped reporting a government-wide improper payment estimate in fiscal year 2017. According to OMB, it stopped reporting a government-wide estimate because program-by-program improper payment data were more useful. However, we believe that the aggregation of improper payment estimates is essential for transparency as without such the extent and magnitude of the government-wide improper payments is not readily available to key decision makers. As such, we support a key provision in the Payment Integrity Information Act of 2019—a bill which has passed the Senate— to require OMB to report a government-wide improper payment estimate amount. Implementing this provision would be a positive step in determining the overall progress the federal government is making in the improper payment area. The federal government also needs to reasonably assure that agencies take appropriate actions to reduce improper payments. For example, in supplemental appropriations acts providing disaster relief funds in 2017 and 2018, Congress mandated an oversight framework for these funds by requiring federal agencies to submit internal control plans to Congress, based on OMB guidance. However, in June 2019, we reported that OMB lacked a strategy for ensuring that federal agencies provide sufficient, useful plans in a timely manner for oversight of disaster relief funds. As a result, we found that selected agencies did not submit their disaster aid internal control plans timely. The plans also lacked necessary information, such as how the selected agencies plan to meet OMB guidance and federal internal control standards. Such a strategy could help provide Congress some assurance that agencies will establish effective and efficient controls over disaster aid. The federal government also needs to reasonably assure that states, local governments, and nonprofit organizations take appropriate actions to reduce their improper payments of federal funds. For example, OMB recently revised its compliance supplement for Medicaid to enable auditors, as part of the single audit of all federal financial assistance that a state received or administered, to test beneficiaries for eligibility for the program. If this expansion of the compliance supplement is successful for Medicaid, other federal programs that states, local governments, and nonprofit organizations administer may also benefit from such revisions. Since the federal government began preparing consolidated financial statements over 20 years ago, three major impediments have continued to prevent us from rendering an opinion on the federal government’s accrual-based consolidated financial statements over this period. 1. Serious financial management problems at the Department of Defense (DOD) have prevented its financial statements from being auditable. DOD’s strategy for achieving a clean opinion on its financial statements and improving overall financial management has shifted from preparing for audit readiness to undergoing financial statement audits and remediating audit findings. In a positive development, DOD underwent an audit of its entity-wide fiscal year 2018 financial statements, which resulted in a disclaimer of opinion issued by the DOD Office of Inspector General (OIG). The DOD OIG also reported 20 material weaknesses in internal control over financial reporting, contributing to its disclaimer of opinion. DOD has acknowledged that achieving a clean audit opinion will take time. However, it stated that over the next several years, the resolution of audit findings will serve as an objective measure of progress toward that goal. DOD will need to develop and effectively monitor corrective action plans to appropriately address audit findings in a timely manner. Partially in response to our recommendations, DOD recently developed a centralized database for tracking the audit findings, recommendations, and related corrective action plans. 2. While significant progress has been made over the past few years, the federal government continues to be unable to adequately account for intragovernmental activity and balances between federal entities. Federal entities are responsible for properly accounting for and reporting their intragovernmental activity and balances in their entity financial statements. When preparing the consolidated financial statements, intragovernmental activity and balances between federal entities should be in agreement and must be subtracted out, or eliminated, from the financial statements. OMB and the Department of the Treasury (Treasury) have issued guidance directing component entities to reconcile intragovernmental activity and balances with their trading partners and resolve identified differences. In addition, the guidance directs the CFOs of significant component entities to report to Treasury, their respective inspectors general, and GAO on the extent and results of intragovernmental activity and balance reconciliation efforts as of the end of the fiscal year. 3. The federal government has an ineffective process for preparing the consolidated financial statements. Treasury, in coordination with OMB, has implemented several corrective actions during the past few years related to preparing the consolidated financial statements. Corrective actions included improving systems used for compiling the consolidated financial statements, enhancing guidance for collecting data from component entities, and implementing procedures to address certain internal control deficiencies. However, the federal government’s systems, controls, and procedures were not adequate to reasonably assure that the consolidated financial statements are consistent with the underlying audited entity financial statements, properly balanced, and in accordance with U.S. generally accepted accounting principles. Further, significant uncertainties, primarily related to achieving projected reductions in Medicare cost growth, and a material weakness in internal control prevented us from expressing an opinion on the sustainability financial statements. We, in connection with our audits, and agency auditors, in connection with their audits, have identified numerous deficiencies underlying the above weaknesses and have provided recommendations for corrective action. The federal government has made unsuccessful efforts to implement new financial management systems, most notably at DOD, the Internal Revenue Service, the Department of Homeland Security, and the Department of Housing and Urban Development—which have spent billions of dollars on failed systems. We have reported that the executive branch has undertaken numerous initiatives to better manage the more than $90 billion that the federal government annually invests in information technology (IT). However, we reported that federal IT investments too frequently fail or incur cost overruns and schedule slippages, while contributing little to mission-related outcomes. These investments often suffered from a lack of disciplined and effective management, including inadequate project planning, clearly defined requirements, and program oversight and governance. In 2015, we added the government’s management of IT acquisitions and operations to our High-Risk List, where it remains in 2019. In fiscal year 2018, eight of 24 CFO Act agencies’ financial management systems still did not substantially comply with FFMIA’s systems requirements. Moreover, a number of agencies rely on critical legacy systems that use outdated languages, have unsupported hardware and software, and are operating with known security vulnerabilities. We previously reported that some agencies have not established complete modernization plans and face an increased risk of cost overruns, schedule delays, and project failure. In addition, most respondents to our CFO survey (15 of 24) stated that it has been extremely, very, or somewhat challenging to work with financial management systems that are old and use obsolete software or hardware. Efforts to promote greater use of shared services in certain areas, such as human resources and financial management activities, resulted in some cost savings and efficiency gains, but challenges (e.g., implementation weaknesses, project scheduling, and project management and costs) impede widespread adoption. Almost all respondents to our CFO survey (22 of 24) indicated that they currently use or plan to use shared services. Most of those respondents (16 of 24) believed that use of shared services could help reduce costs. As noted above, in April 2019, OMB issued Memorandum M-19-16 on shared services, which among other things described the process and desired outcomes for shared services and established a governance and accountability model for achieving them. Also, OMB stated that, building off of OMB’s and Treasury’s efforts to create a Quality Service Management Office for Financial Management, they are establishing a more centralized approach to standardize, consolidate, and automate agency financial systems. A government-wide plan for improving federal financial management systems, including shared services, that is incorporated into the government-wide and agency-level plans discussed above could help ensure, among other things, that financial management system problems are addressed. Insufficient numbers of staff, inadequate workforce planning, and a lack of training in critical areas create gaps between what the federal government needs and the skills federal employees have. We have made a number of recommendations toward achieving a federal workforce with the necessary skills, including in financial management. In a 2007 testimony, we reported that one key challenge to strong federal financial management is building a financial management workforce for the future. This holds true today. Our CFO survey respondents (14 of 24) noted that CFO Act agencies do not have all of the staff with the professional qualifications, capabilities, and expertise needed to effectively support financial management operations and practices. With rapid changes, such as emerging technologies and growing availability of data, it is critical for the government to identify and strategically plan for the future workforce to achieve effective financial management. A comprehensive, long-term plan to address the challenges in the federal financial management workforce that is incorporated into the government-wide and agency-level plans discussed above could help ensure that agencies are held accountable for a long-term vision of attracting and retaining a workforce that maintains the professional qualifications, capabilities, and expertise that will meet current and future needs. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Prior to the enactment of the CFO Act, government reports found that agencies lost billions of dollars through fraud, waste, abuse, and mismanagement. These reports painted the picture of a government unable to properly manage its programs, protect its assets, or provide taxpayers with the effective and economical services they expected. The CFO Act was enacted to address these problems—calling for comprehensive federal financial management reform. Among other things, the act established CFO positions, provided for long-range planning, and began the process of auditing federal agency financial statements. The act also called for integrating accounting and financial management systems and systematic performance measurement and cost information. This statement is based on preliminary observations from GAO's ongoing review of the federal government's efforts to meet the requirements of the CFO Act. GAO reviewed federal financial management legislation, guidance, and reports. GAO also conducted interviews and a panel discussion with experts in federal financial management, and surveyed federal CFOs, inspectors general, and independent public accountants. The federal government has made significant strides in improving financial management since enactment of the Chief Financial Officers Act of 1990 (CFO Act). Substantial progress has occurred in areas such as improved internal controls, reliable agency financial statements, and establishment of chief financial officer (CFO) positions. To help ensure that the CFO Act achieves its full potential, there are several opportunities for enhancement. Standardize CFO and deputy CFO responsibilities across government. The responsibilities assigned to CFOs vary among agencies. Uniform and effective responsibilities of CFOs would help enhance strategic decision-making and correct inconsistencies across government. In addition, deputy CFOs should have appropriate responsibilities in order to be better prepared to act for CFOs when there are vacancies. Prepare government-wide and agency-level financial management plans. Since 2009, the Office of Management and Budget (OMB) has not prepared the annual 5-year government-wide plans that the CFO Act requires. Instead, OMB has provided information in the President's Management Agenda, the U.S. government's consolidated financial statements, and other documents. A complete and integrated government-wide financial management plan and supporting agency plans, prepared every few years, could help ensure continuity in direction and a more comprehensive understanding of gauging progress toward addressing financial management challenges across government. Better link performance and cost information for decision-making. While agencies have made efforts in this direction, opportunities exist for agencies to better link performance and cost information to effectively make financial management decisions that are based on dollars allocated and results achieved. Develop a broader set of key selected financial management performance-based metrics. Agencies currently have limited performance-based metrics to help them assess the quality of financial management and ensure that the federal government better manages and uses the resources entrusted to it. Rectify internal control issues in certain areas. The federal government faces many internal control problems. For example, assessments continue to identify long-standing, as well as new, material weaknesses. Improper payments continue to be a long-standing internal control issue. And finally, material weaknesses continue to prevent GAO from rendering an opinion on the U.S. government's consolidated financial statements. Improve financial management systems. The federal government has made unsuccessful efforts to implement new financial management systems at several agencies and spent billions of dollars on failed systems. Moreover, in fiscal year 2018, eight of 24 CFO Act agencies' still did not substantially comply with federal systems requirements. Strengthen the federal financial management workforce. With rapid changes, such as emerging technologies, it is critical for the government to identify and strategically plan for the future workforce. GAO obtained comments from OMB, the Department of the Treasury, and the Office of Personnel Management and has incorporated their comments as appropriate. As GAO finalizes its work for issuance next year, it will consider feedback on its work in making recommendations related to the opportunities for enhancement, as appropriate.", "document_type": "gao"}
{"report": "The Corps is one of the world’s largest public engineering, design, and construction management agencies. Located within the Department of Defense, the Corps has both military and civilian responsibilities. Through the civilian Civil Works Program, the Corps plans, constructs, operates, and maintains a wide range of water resources development projects such as navigation and flood risk projects. The Assistant Secretary of the Army for Civil Works, appointed by the President, sets the strategic direction for the program and has principal responsibility for the overall supervision of functions relating to the Army’s Civil Works Program. The Chief of Engineers, a military officer, is responsible for execution of the civil works and military missions. At the Corps level, the Civil Works Program is organized into three tiers: headquarters in Washington, D.C.; eight regional divisions; and 38 local district offices (see fig. 1). Corps headquarters primarily develops policies and guidance to implement the agency’s responsibilities and plans the direction of the organization. The divisions, which were established generally according to watershed boundaries, primarily coordinate the districts’ civil works and military projects and are commanded by military officers. The districts, also commanded by military officers, are to, among other things, plan and implement feasibility studies and the resulting water resources development projects that are approved by the divisions and headquarters. There are several steps in conducting a Corps water resources development project. When a local community perceives a need or experiences a water resources problem that is beyond its ability to solve, it typically contacts the Corps for assistance. These communities and Congress, as well as other entities, play key roles in the process. Figure 2 illustrates the major steps in conducting a Corps water resources development project. As identified above, one of the major steps in initiating a water resources development project is conducting a feasibility study. Feasibility studies further investigate a water resources problem and make recommendations on whether a project is in the federal interest, and if so, how the problem should be addressed. Generally, the cost of a feasibility study is shared between the Corps and a nonfederal sponsor, such as a local port authority or a state agency. In 2012, the Corps began using a new approach to conducting feasibility studies, referred to as SMART Planning. As part of this approach, Corps officials are to use and document a risk-informed approach to decision- making. Specifically, Corps officials are to consider risks at each point in the feasibility study process and balance the probability and consequences associated with those risks with the time and costs needed to avoid or mitigate risks through, for example, collecting additional data or conducting additional analysis. By doing so, they are to conduct only the additional analysis needed to make a decision at that point in the process. At each step, Corps officials are to use an approach that balances the level of detail, data collection, research, and associated risks with what is necessary to deliver the feasibility study, and they are to justify any additional work as the best course forward. The Corps’ feasibility study process consists of four phases (scoping, alternative evaluation and analysis, feasibility-level analysis, and Chief’s report) and a number of key milestones, such as identifying project alternatives for further review (see fig. 3). The complete feasibility study process is to take place within the statutory target time frame of less than 3 years (36 months). The Corps uses SMART Planning to help feasibility studies meet the agency’s 3x3x3 rule. Corps policy allows the Corps to spend more money and take more time on an unusually complex feasibility study if the district leading the study requests and receives an exemption from headquarters or the Assistant Secretary of the Army for Civil Works. However, Corps policy indicates that such exemptions are not routine and are to be granted only after careful consideration and review by division and headquarters officials. In addition, WRRDA 2014, as amended, provides that the Secretary of the Army may make an exception by extending the timeline of a study if the Secretary determines that the study is too complex to comply with the 3x3x3 rule. The Secretary is not to extend the timeline for a feasibility study for a period of more than 10 years, and any feasibility study that is not completed before that date shall no longer be authorized. The act also requires the Secretary to provide written notice to the Senate Committee on Environment and Public Works and the House Committee on Transportation and Infrastructure each time the Corps grants such an exception. The feasibility study process includes work the Corps undertakes to satisfy requirements under the National Environmental Policy Act (NEPA) and other environmental statutes. Under NEPA, federal agencies are to evaluate the potential effects of proposed projects on the environment. When the Corps determines that a water resources development project could have significant environmental effects, it must prepare an EIS. The Corps issues a draft EIS as part of the overall draft feasibility report for public and stakeholder review and issues a final EIS when it issues its final feasibility report. Feasibility studies that require an EIS typically represent larger and more complex studies than those that do not require an EIS. According to a 2013 Congressional Research Service report, Corps feasibility studies that are larger and more complex tend to require additional funding and time when compared to less complex, smaller studies. While the Corps does not publish information on the length of time it takes to complete feasibility studies, our analysis of publicly available data showed that the median time it took the Corps to complete a feasibility study with an EIS was more than 7 years for those studies completed from 2008 through 2018. WRRDA 2014 contains provisions related to, among other things, accelerating the completion of feasibility studies for which an EIS is prepared. These provisions broadly fall into different general categories, which we grouped as follows: Coordination and administration. These provisions are generally process oriented. Among other things, they relate to facilitating the process of coordinating and administering feasibility studies by, for example, encouraging the Corps and other agencies to coordinate early in the feasibility study process and resolve issues expeditiously. Environmental review. These provisions relate to implementing NEPA and other environmental statutes when conducting feasibility studies. For example, the Corps is to establish a program to measure and report on progress made to improve and expedite the planning and environmental review process. Public transparency. These provisions generally require the Corps to, among other things, make information publicly available on how it is implementing the acceleration provisions. The Corps has taken steps to address broad WRRDA 2014 provisions related to facilitating the process of coordinating and administering feasibility studies. For example: Issuance of a joint coordination guide. In September 2015, as a result of the act and previous ongoing coordination efforts, the Corps, NMFS, and FWS worked together to jointly issue a coordination guide for conducting feasibility studies. The guide discusses the feasibility study process in depth and emphasizes the importance of substantive, early engagement among the three agencies to successfully deliver projects and avoid delays later in the process that may result from lingering disagreements among the agencies. Issuance of Corps guidance on WRRDA 2014 acceleration provisions. In March 2018, the Corps issued guidance on how officials should implement the WRRDA 2014 acceleration provisions when conducting feasibility studies. This includes guidance on implementing administrative changes such as deadlines for gathering agency or public comments. It also includes guidance on coordination within the agency as well as with other agencies and stakeholders, such as nonfederal sponsors. For example, WRRDA 2014 provides that the Corps is to make certain information available to other agencies as early as practicable in the environmental review process. The Corps’ March 2018 guidance indicates that Corps officials are to provide information on the (1) environmental and socioeconomic resources located within the physical area associated with a feasibility study, and (2) general locations of the different alternatives under consideration. While the guidance was not issued for almost 4 years after the enactment of WRRDA 2014, several Corps headquarters and district officials said the Corps disseminated information on how to implement the acceleration provisions to the districts in various ways, such as through webinars and working with teams that had initiated feasibility studies subject to the act’s acceleration provisions. Many Corps headquarters, division, and district officials said that many of the act’s coordination and administration provisions are similar to long- standing practices they followed, based on requirements in other laws such as NEPA. For example, according to many Corps headquarters, division, and district officials, the WRRDA 2014 provision to develop a coordinated environmental review process is generally consistent with NEPA and its implementing regulations. According to a Corps headquarters official, the WRRDA 2014 coordination provisions add specificity to the Corps’ existing practices by detailing which agencies to involve in coordination efforts and when to involve them. The Corps also has taken steps to address one of the WRRDA 2014 provisions related to public transparency. Specifically, the Corps is to annually prepare, and make publicly available, a list of feasibility studies subject to the acceleration provisions that do not have adequate funding to make substantial progress toward completion of the study. Corps headquarters and district officials said that in the past the Corps funded several hundred active feasibility studies at any given time. While this allowed for many feasibility studies to remain active and make some progress, it also made less funding available for individual feasibility studies and slowed the progress of some studies, according to several Corps officials. According to a February 2012 Corps policy memo, agency leadership initiated a process to review all active feasibility studies to determine which were the most viable for congressional funding. The Corps re-scoped or deactivated the remainder of the feasibility studies. Many Corps district and headquarters officials told us this allowed for increased funding for and progress to be made on the feasibility studies that remained active. As a result of the Corps’ efforts, headquarters officials said the number of active Corps feasibility studies decreased from 653 in 2012 to 89 at the end of 2018. In addition, they said that because active feasibility studies now have greater levels of funding, the agency has not had to report any active feasibility studies that do not have adequate funding. However, as of May 2019, the Corps has not addressed other WRRDA 2014 provisions related to public transparency and environmental review. These include the following: Status and progress database. By June 2015, the Corps was to establish and maintain an electronic database and, in coordination with other federal and state agencies, issue reporting requirements to make publicly available the status and progress regarding compliance with applicable requirements of NEPA and other required approval or action. Performance measurement. The Corps is to establish a program to measure and report on progress made toward improving and expediting the planning and environmental review process. Environmental review guidance. The Corps is to (1) prepare, in consultation with the Council on Environmental Quality and other federal agencies with jurisdiction over actions or resources that may be impacted by a project, guidance documents that describe the coordinated environmental review processes the Corps intends to use to implement reforms for planning projects, and (2) issue guidance on the use of programmatic approaches for the environmental review process that carries out specified actions and meets specified requirements. In other instances, the Corps has taken some initial steps but has not fully addressed certain WRRDA 2014 provisions. Specifically, not later than 180 days after the act’s enactment, the Corps was to survey the agency’s use of categorical exclusions in projects since 2005, publish a review of that survey, and solicit requests from other federal agencies and project sponsors for new categorical exclusions. By June 2015, the Corps was to propose a new categorical exclusion if it identified a category of activities that merited such action. As of May 2019, the Corps had conducted an internal survey and solicited input through the Federal Register on its procedures for implementing NEPA. However, Corps headquarters officials said they had not published a review of its survey, targeted requests for new categorical exclusions to other federal agencies and nonfederal sponsors, or proposed new exclusions as merited. Appendix II contains a more detailed summary of the WRRDA 2014 acceleration provisions, along with information on Corps actions to address each provision. Corps headquarters officials identified resource constraints as the primary reason for not addressing some public transparency and environmental review provisions. For example, to develop environmental review guidance, Corps headquarters officials told us that they would need to conduct various steps, including drafting guidance, conducting administrative review with other federal agencies, soliciting public comment, and revising the guidance. Headquarters officials also said they were involved in a similar effort with other federal agencies to develop environmental review guidance in a publication called the 2015 Red Book, an effort they characterized as labor intensive. In addition, to establish a database to publicly report on the status of its feasibility studies, Corps headquarters officials said they would need to stand up and maintain a website similar to the Federal Infrastructure Permitting Dashboard for federal infrastructure projects. The Corps is one of many agencies involved in the effort to create and maintain this dashboard, and Corps headquarters officials said the effort was a resource-intensive process. Corps headquarters officials said that while they have not created the database required by WRRDA 2014, relevant information is available through the agency’s annual public reports on active and recently completed feasibility studies’ milestones and schedules. Corps headquarters officials also said the status of feasibility studies is often available on the Corps districts’ websites. However, this information is not easily accessible without knowing which district office is responsible for a given feasibility study. While Corps officials identified resource constraints as the primary reason for not addressing certain WRRDA 2014 provisions, they did not provide specific estimates on the resources that the Corps would need to address these provisions. In addition, the officials said they do not have a plan that addresses how and when they intend to implement the provisions they have yet to address. We have previously reported on leading practices for sound planning and have found that implementation plans that include resource estimates help ensure organizations achieve their goals and objectives. Such a plan would better position the Corps to address the remaining WRRDA 2014 provisions related to environmental review and public transparency. The Corps monitors feasibility studies and has done some review of its acceleration reforms but has not conducted a comprehensive evaluation of the impacts of these reforms. In terms of monitoring, Corps policy states that division and district leaders are responsible for monitoring feasibility studies within their areas of responsibility. According to Corps policy, districts are to prepare a quality control plan for each project to ensure compliance with all technical and policy requirements, and divisions are responsible for quality assurance by ensuring that districts plan, design, and deliver quality projects on schedule and within budget. Corps headquarters officials also said they monitor the progress of feasibility studies during management meetings, during which they discuss the cost and status of feasibility studies as well as the quality of those studies; such meetings are largely led by Corps management or by the Corps’ Planning Advisory Board, which oversees the quality of feasibility studies. In addition to monitoring individual feasibility studies, Corps headquarters officials said they have conducted some broader reviews of how the acceleration reforms are progressing. For example, they conducted a trend analysis in October 2018 and again in April 2019 to identify the reasons why some feasibility studies have received exceptions from the timing and cost requirements of the 3x3x3 rule. These analyses, among other things, identified that some studies were too complex to be completed within 3 years or for less than $3 million, according to Corps officials. Furthermore, based on their experiences with various reform efforts, Corps officials said that they have been making real-time enhancements. For example, based on input from the Corps’ Planning Advisory Board, Corps leadership has called for the agency to clarify its updated approach to risk management, according to Corps officials. These officials said each component within the Corps that is involved in conducting feasibility studies is to issue internal guidance on its risk management approach. However, Corps headquarters officials said the Corps has not conducted a comprehensive evaluation of acceleration reforms to determine what impacts the reforms have had and whether any modifications to those reforms are needed. Corps and other agency officials and stakeholders we interviewed differed in their views of the acceleration reforms’ impacts on the cost, time frames, and quality of feasibility studies: Cost and time frames for completing feasibility studies. Many Corps officials said they agreed with the overall goals of reducing costs and increasing the speed with which feasibility studies are carried out. Some Corps headquarters and district officials said SMART Planning and the 3x3x3 rule are changing the Corps’ culture around the amount of time and cost a feasibility study should take. However, several Corps district and headquarters officials said some Corps staff are experiencing difficulties with the cultural change represented by SMART Planning and the 3x3x3 rule. For example, a Corps district official said that in the past some Corps navigation economists had one year to complete some modeling analyses for feasibility studies, but they now are to complete such work in 90 days due to the constraints of SMART Planning and the 3x3x3 rule, which has been a difficult adjustment. In addition, many Corps headquarters, division, and district officials raised concerns that the cost limitation of $3 million may not be realistic given differences in cost across geographic locations or the loss of spending value over time caused by inflation. Quality of feasibility studies. Several Corps district officials we interviewed said they like the Corps’ new policy of involving other agencies earlier in the process and with more frequency. They said they believe this approach has improved coordination with other agencies—by, for example, inviting the other federal agencies to join the Corps in a formal initiation meeting—which can in turn improve the overall quality of a feasibility study. However, some FWS and NMFS officials said they would like to be more involved and have better communication with the Corps than they currently do, such as throughout the feasibility study process rather than just at the beginning of a study and at the end when their formal review is requested. Similarly, several Corps headquarters, district, and division officials have commended the agency’s new approach to risk management and stated that they aim to provide partner agencies with the information they need to conduct their work on the feasibility study. However, many Corps, FWS, and NMFS officials and nonfederal sponsors we interviewed said they were concerned that this new approach might result in insufficient information for making decisions, which could affect the quality of feasibility studies. For example, for six of the seven studies that we reviewed, officials from FWS and NMFS said it has become more difficult for them to provide meaningful input on the feasibility study alternatives considered because the Corps provides them with less detailed information than in the past. Corps officials and other stakeholders we interviewed also expressed concern about possible impacts of the 3x3x3 rule on the quality of feasibility studies. For example, many Corps headquarters, division, and district officials said that because the 3x3x3 rule puts constraints on costs and time frames, if the scope of a feasibility study is not similarly reduced, it can affect the study’s quality. In addition, nonfederal sponsors for four of the seven studies we examined expressed concerns with the 3x3x3 rule; three of these four nonfederal sponsors said they believe that the Corps is more focused on meeting the cost and schedule timelines than on the needs or quality of the study. Senior Corps headquarters officials said they are confident that the cost and duration of feasibility studies has decreased overall as a result of the acceleration reforms but could not provide us with documentation to support this observation. Specifically, officials said in March 2019 that based on analysis they had recently conducted, most feasibility studies are now being completed within 4 years and at a lower cost than feasibility studies undertaken prior to implementation of the 3x3x3 rule. While these results may not meet the 3x3x3 rule, officials said that these feasibility studies were the first subject to the acceleration reforms and may not depict the likelihood of future feasibility studies meeting the rule. This is, in part, because Corps officials who are working on new feasibility studies have the benefit of the past several years of experience working with the SMART Planning process. Further, Corps officials said that they do not have formal documentation summarizing how the acceleration reforms have affected the quality of their feasibility studies overall, but they monitor individual feasibility studies, as described earlier. According to Corps headquarters officials, the Corps has not conducted a more comprehensive evaluation of the broader impacts of the acceleration reforms because it has only completed a small number of feasibility studies since 2012 under the acceleration reforms, and officials are focused on monitoring their ongoing individual studies. These officials said they see the value in conducting such an evaluation as they complete more studies but that they have not developed formal plans to do so. Effective program evaluation includes an evaluation plan—that is, a plan that takes into account the questions guiding the evaluation, the constraints faced in studying the program, and the information needs of the intended users. Developing an evaluation plan would help position the Corps to conduct a timely and effective review of the impacts of the acceleration reforms overall. The Corps has not maintained complete data on the 10 key milestones in its central data system for more than half of the feasibility studies we reviewed. Specifically, for the 19 feasibility studies we reviewed, we found that: seven studies in the Corps’ central data system included complete data for all 10 key milestones, and twelve studies were missing one or more milestones in the data system. Table 1 provides information on the key milestone data included in the Corps’ central data system for the 19 feasibility studies we reviewed. Many Corps headquarters and division officials said that Corps officials vary in their knowledge of its central data system. Many headquarters, division, and district officials we interviewed also acknowledged that, in general, the milestone information entered into the Corps’ central data system can be inconsistent across different feasibility studies. Corps headquarters officials said agency policy requires district officials conducting feasibility studies to enter data on 10 key milestones for each study into the agency’s central data system. However, while the policy identified the 10 milestones, it only explicitly requires that two of the 10 milestones be entered into the agency’s central data system. Specifically, the policy states that officials are to enter into the Corps’ data system the milestones for (1) feasibility study initiation and (2) posting of the plan for peer and stakeholder review. Corps officials said the intent of the policy is for all 10 key milestones to be entered into the central data system but acknowledged that the policy may not be clear. In part to assist district officials in conducting feasibility studies, Corps headquarters officials created a template, which includes information on nine of the 10 key milestones. In addition, a Corps district official said she was unclear on the agency’s expectations about which milestones to enter into the central data system. Corps headquarters officials said they contact district officials responsible for feasibility studies to obtain up-to- date information and ensure they understand the progress of each feasibility study. While this may help to ensure accuracy and completeness of milestone data on feasibility studies, several Corps district officials said the process of responding to such data calls can be time consuming and take them away from their core responsibilities. Without clarifying its policy to help ensure district officials enter data on all key milestones for feasibility studies into its central data system, the Corps will not have complete data to efficiently monitor the progress of feasibility studies. The Corps has taken steps to address the acceleration provisions in WRRDA 2014, such as those related to coordination. However, it has not fully addressed provisions related to environmental review or public transparency. Corps officials said they do not have a plan that addresses implementation of remaining provisions or the resources that will be required to implement them. An implementation plan that includes resource estimates would better position the Corps to address the remaining provisions in WRRDA 2014. Further, the Corps monitors the progress of feasibility studies and has conducted some reviews of the individual acceleration reforms. However, the agency has not developed an evaluation plan for its acceleration reforms to better understand the reforms’ impacts overall and determine whether any modifications to those reforms are needed. Developing such a plan would enable the Corps to conduct a timely and effective evaluation. Further, without clarifying its policy to ensure district officials enter all key milestone dates for feasibility studies into its central data system, the Corps will continue to lack complete data to efficiently monitor the progress of feasibility studies. We are making the following three recommendations to the Department of Defense: The Secretary of the Army should direct the Assistant Secretary of the Army for Civil Works to develop an implementation plan that includes resource estimates to address the remaining WRRDA 2014 acceleration provisions. (Recommendation 1) The Secretary of the Army should direct the Assistant Secretary of the Army for Civil Works to develop a plan to conduct a comprehensive evaluation of the impacts of the agency’s feasibility study acceleration reforms. (Recommendation 2) The Secretary of the Army should direct the Assistant Secretary of the Army for Civil Works to clarify its policy to help ensure district officials enter data on all key milestones for feasibility studies into its central data system. (Recommendation 3) We provided a draft of this report to the Department of Defense for review and comment. In its written comments, reprinted in appendix III, the Department concurred with our recommendations. The Department commented that we should redirect our recommendations to the Assistant Secretary of the Army for Civil Works rather than to the Chief of Engineers and the Commanding General of the U.S. Army Corps of Engineers, which we did. The Department also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Secretary of the Department of the Interior, the Secretary of Commerce, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or FennellA@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines the extent to which the U.S. Army Corps of Engineers has (1) addressed the feasibility study acceleration provisions under the Water Resources Reform and Development Act of 2014 (WRRDA 2014), (2) reviewed the impact of its feasibility study acceleration reforms, and (3) maintained complete milestone data for feasibility studies in its central data system. To conduct our work, we reviewed the first 19 feasibility studies subject to the WRRDA 2014 feasibility study acceleration provisions, among other things. These feasibility studies included those that (1) were initiated after June 10, 2014, the date WRRDA 2014 was enacted, through August 15, 2018, and (2) for which an environmental impact statement (EIS) is prepared. We chose to review studies through August 15, 2018, because after that date the Corps initiated several feasibility studies using funding in a supplemental appropriation the Corps received in February 2018 to conduct work in response to recent large hurricanes, and Corps officials said they planned to use a somewhat different approach to conducting these studies. For each study, we reviewed Corps guidance on the agency’s process for planning feasibility studies and other related documentation. We examined information from the Corps on the progress and status of the19 feasibility studies. We also reviewed information for each feasibility study on the Corps’ business line or program, the district or division overseeing the study, and information on which studies had received exceptions from the 3x3x3 rule. We also conducted a more in-depth review of seven of these 19 feasibility studies. We selected these seven studies because they represent different types of water resources development projects, were at varying stages of completion, and are geographically dispersed. The seven studies, and the Corps districts leading these studies, are: Coastal Texas Protection and Restoration (Galveston District); Houston Ship Channel Expansion Channel Improvement Project (Galveston District); Matagorda Ship Channel (Galveston District); Gulf Intercoastal Waterway: Brazos River Floodgates and Colorado River Locks Systems (Galveston District); Mississippi River Ship Channel, Gulf to Baton Rouge, Louisiana General Reevaluation Report (New Orleans District); Sacramento River, General Reevaluation Report (Sacramento Port of Long Beach Deep Draft Navigation Improvements (Los Angeles District). For each of these seven studies, we reviewed project management plans and other project documents, such as draft feasibility studies, if available. From August 2018 through November 2018, we visited the four district offices that led these seven studies, including the Corps’ Galveston, Los Angeles, New Orleans, and Sacramento district offices. During these visits, we discussed the status and progress of each of these feasibility studies and the Corps’ coordination with other federal agencies and nonfederal sponsors, among other things. For each study, we interviewed officials from nonfederal sponsors—such as the state or local government associated with individual studies—and from federal partners—including the Fish and Wildlife Service (FWS) and the National Marine Fisheries Service (NMFS). We selected FWS and NMFS because of the important role they play in reviewing environmental aspects of Corps feasibility studies and their role in the 2015 joint publication on coordination. We also interviewed Corps officials at the three divisions overseeing the districts that conducted the feasibility studies we selected. This included officials from the Corps’ South Pacific, Mississippi Valley, and Southwestern divisions. While the seven studies provide illustrative examples, they are not generalizable to all of the Corps’ feasibility studies for which an EIS is prepared. We developed and used four standard sets of semi-structured interview questions for the following groups: the (1) Corps district office officials conducting the seven selected feasibility studies, (2) FWS and NMFS officials working with the Corps on these studies, (3) Corps division officials overseeing each study, and (4) nonfederal sponsors who worked with the Corps on each study. To characterize the views of those we interviewed throughout the report, we defined modifiers to quantify officials’ views as follows: “some” refers to responses from two to four Corps officials and/or stakeholders; “several” refers to responses from five to seven Corps officials and/or stakeholders; and “many” refers to responses from eight or more Corps officials and/or stakeholders. To examine the extent to which the Corps addressed the WRRDA 2014 feasibility study acceleration provisions, we compiled a list of the provisions. We then reviewed the Corps’ documentation related to the implementation of these provisions, including agency guidance and policies. We compared this information with the WRRDA 2014 acceleration provisions. To do this, we created categories for the acceleration provisions and grouped the provisions by category. To examine the extent to which the Corps has reviewed the impact of its acceleration reforms, we reviewed Corps policy, guidance, training, and other documentation on implementation of those reforms. We use the term acceleration reforms to refer to the requirements that new feasibility studies are to be completed in less than 3 years and at a cost of not more than $3 million, the Corps’ risk management of feasibility studies through its new SMART Planning process, and the WRRDA 2014 acceleration provisions. We reviewed documentation from the Corps on the feasibility studies that have received exceptions from the 3x3x3 rule. We interviewed Corps headquarters officials to learn what, if any, (1) new policies were in place to help division and district staff implement the reforms; and (2) review or analysis headquarters officials had completed of the impacts of the reforms on the cost, time frames, or quality of feasibility studies. We also interviewed Corps districts and division officials who were responsible for the seven studies about how the acceleration reforms were working, as well as FWS and NMFS officials and nonfederal sponsors about their views of the impacts of the new processes on their work on these feasibility studies. We compared this information with program evaluation guidance. To examine the extent to which the Corps has maintained complete milestone data for feasibility studies in its central data system, we obtained milestone data from the system for the 19 Corps feasibility studies in our review. We analyzed the milestone data to determine which milestone dates were in the system and then worked with Corps headquarters officials to verify that information. We assessed the reliability of these data by reviewing related documentation and interviewing knowledgeable officials, among other things. We determined that the data were sufficiently reliable for the purpose of understanding which districts and divisions conducted feasibility studies and for understanding the types of milestones that were entered into the central data system. However, as discussed in this report, we determined that the milestone data were not sufficiently reliable for other purposes. We reviewed data for all feasibility studies in our review to determine whether they conformed to Corps expectations on what milestone data should be in the system. We estimated the median time it took the Corps to complete a feasibility study for which an EIS was prepared. To do this, we obtained from the Corps website the names of all feasibility studies completed with a Chief’s Report from July 2008 through June 2018 and the dates they were completed. We verified with Corps headquarters officials that its list of studies with a Chief’s Report was current for that time frame. For each of these feasibility studies, we then found the associated notice of intent to complete an EIS as published in the Federal Register. While the date the Corps filed a notice of intent to complete an EIS is not the initiation date for the feasibility study, we used it as a proxy since Corps headquarters officials said that, in the past, the notice of intent was filed soon after a study was initiated. We calculated the time between the date the notice of intent was filed and the date of the Chief’s report to arrive at an estimate of the amount of time the each feasibility study took to complete. We then calculated the median time it took to complete these feasibility studies. We conducted this performance audit from April 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform our audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: U.S. Army Corps of Engineers Project Acceleration Statutory Provisions and Corps Actions Related to Each Provision provides guidance, independently evaluates, and approves the document before taking subsequent action; and ensures the project sponsor complies with all design and mitigation commitments. In addition, any NEPA documents prepared in this way are to be adopted and used by any federal agency when making any determination to the same extent the agency could adopt or use a document prepared by another federal agency under NEPA. Category name Coordination and Administration Designating Jurisdictional Agencies GAO summary of statutory provision For all federal, state, and local governments and Indian tribes that may have jurisdiction over a project or that may be required to review some aspect of the feasibility study or make a determination on issuing a permit or other decision, the Corps must: identify these agencies as early as practicable, and invite these agencies to participate or coordinate as early as practicable and set a deadline for response. Corps actions related to provision The Corps issued its WRRDA 2014 acceleration guidance in March 2018. as well as the Principles and Guidelines and Planning Guidance Notebook. Any federal agency invited by the Corps will be designated as a cooperating agency unless that agency follows certain specified steps. Plan for Coordinating Input and Completing Environmental Review The Corps, after consultation with and with the concurrence of relevant entities is to establish a plan for coordinating public and agency participation in, and comment on, the environmental review process for each feasibility study or category of studies. As soon as practicable but not later than 45 days after the close of the public comment period on a draft Environmental Impact Statement (EIS), the Corps, after consultation with and with the concurrence of relevant entities, also is to establish, as a part of the coordination plan, a schedule for completing the environmental review process. In doing so, the Corps is to consider certain factors, provide the schedule to relevant entities, and make it available to the public. The Corps issued its WRRDA 2014 acceleration guidance in March 2018. In addition, a Corps official indicated that portions of this provision are implemented under the Corps’ NEPA procedures. The Corps issued its WRRDA 2014 acceleration guidance in March 2018. not more than 60 days for agency or public comment on a draft EIS, and not more than 30 days for agency and public comment on other environmental review documents. GAO summary of statutory provision Issue Identification and Resolution The Corps, the cooperating agencies, and any participating agencies are required to work cooperatively to identify and resolve issues that could delay completion of the environmental review process or result in the denial of any approval required for the project study under applicable laws. Corps actions related to provision The Corps issued its WRRDA 2014 acceleration guidance in March 2018. In addition, a Corps official indicated that portions of this provision are implemented under the Corps’ NEPA procedures and Planning Guidance Notebook. Many Corps district officials said they have used various strategies, such as meetings, to resolve issues with other agencies. The Corps is to make information available to the cooperating and participating agencies as soon as practicable in the environmental review process regarding the environmental and socioeconomic resources located within the project area and the general locations of the alternatives under consideration. Based on information from the Corps, cooperating and participating agencies are to identify as early as practicable any issues of concern regarding the potential environmental or socioeconomic impacts of the project, including any issues that could substantially delay or prevent an agency from granting a permit or other approval that is needed for the project study. On the request of a participating or cooperating agency or project sponsor, the Corps is to convene an issue resolution meeting with the relevant entities to resolve issues that may (1) delay completion of the environmental review process, or (2) result in denial of any approval required for the project study under applicable laws. Such a meeting is to be held not later than 21 days after the Corps receives the request for the meeting unless the Corps determines there is good cause to extend that deadline. Additionally, the Corps may convene an issue resolution meeting at its discretion, regardless of whether such a meeting is requested. If resolution cannot be achieved within 30 days of an issue resolution meeting and the Corps determines that all information necessary to resolve the issue has been obtained, the Corps is to forward the dispute to the heads of the relevant agencies for resolution. Corps actions related to provision The Corps issued its WRRDA 2014 acceleration guidance in March 2018. The Corps must notify the Senate Committee on Environment and Public Works and the House Committee on Transportation and Infrastructure as soon as practicable. The Corps must continue notifications every 60 days thereafter until all decisions have been made by the federal agency. The amount of funds made available to support the office of the head of that federal agency must be reduced by certain specified amounts, subject to certain limitations. The Corps, NMFS, and FWS jointly issued a coordination guide for conducting feasibility studies in September 2015. The Corps also issued its WRRDA 2014 acceleration guidance in March 2018.b In addition, a Corps official indicated that portions of this provision are implemented under the agency’s NEPA procedures and Planning Guidance Notebook,f as well as the Principles and Guidelines.e Upon request by a state or project sponsor, and to the maximum extent practicable and appropriate, as determined by the agencies, the Corps and other federal agencies with relevant jurisdiction in the environmental review process are to provide technical assistance to the state or project sponsor in carrying out early coordination activities. If requested by a state or project sponsor, the Corps, in consultation with other federal agencies with relevant jurisdiction, may establish memoranda of agreement with certain entities to carry out early coordination activities, subject to certain limitations. New Information The Corps is to consider information received after the close of a comment period if the information satisfies the requirements for a supplemental EIS under NEPA regulations. The Corps issued its WRRDA 2014 acceleration guidance in March 2018. Corps actions related to provision The Corps issued its WRRDA 2014 acceleration guidance in March 2018. With respect to the environmental review process for any project study, the Corps is to have the authority and responsibility to (1) take actions as are necessary and proper and within the Corps’ authority to facilitate the expeditious resolution of the environmental review process for the project study, and (2) prepare or ensure that any required EIS or other environmental review document required to be completed under NEPA is completed in accordance with applicable federal law. Publishing Information on Studies with Inadequate Funding to Make Substantial Progress The Corps is to annually prepare and make publicly available a list of feasibility studies that the agency does not have adequate funding to make substantial progress toward the completion of the study. The Corps has undertaken a multi-year effort to focus funding on the feasibility studies the agency determined are the most viable options for Congressional funding and then re-scope or deactivate the remaining studies. The Corps has not taken action as of May 2019. not later than June 10, 2015, establish and maintain an electronic database and, in coordination with other federal and state agencies, issue reporting requirements to make publicly available the status and progress with respect to compliance with applicable requirements of NEPA and other required approval or action; and publish the status and progress of any such required approval or action on a feasibility study. Categorical Exclusions Not later than 180 days after June 10, 2014, the Corps is to: conduct an internal survey on its use of categorical exclusions since 2005, publish a review of the survey that includes a description of certain specified information, and solicit requests from other federal agencies and project sponsors for new categorical exclusions. As of May 2019, the Corps had conducted an internal survey and solicited public input through the Federal Register on its procedures for implementing NEPA. However, Corps headquarters officials said they had not published a review of its survey, targeted requests for new categorical exclusions to other federal agencies and nonfederal sponsors, or proposed new exclusions as merited. If the Corps identifies a category of activities that merits establishing a new categorical exclusion, the agency is also to propose new categorical exclusions by June 10, 2015. Performance Measurement The Corps is to establish a program to measure and report on progress made toward improving and expediting the planning and environmental review process. The Corps has not taken action as of May 2019. GAO summary of statutory provision Guidance on Coordinated Environmental Review The Corps, in consultation with the Council on Environmental Quality and other federal agencies with jurisdiction over actions or resources that may be impacted by a project, is to prepare guidance documents that describe the coordinated environmental review processes that the Corps intends to use to implement the reforms for the planning of projects. Corps actions related to provision The Corps has not taken action as of May 2019. Corps officials said they have reached out to the Council on Environmental Quality several times and are waiting for feedback on preparing this guidance. Guidance on Programmatic Approaches to Environmental Review The Corps is to issue guidance on the use of programmatic approaches to carry out the environmental review process that carries out specified actions and meets specified requirements. The Corps has not taken action as of May 2019. U.S. Army Corps of Engineers, Implementation Guidance for Section 1005 of the Water Resources Reform and Development Act of 2014 (WRRDA 2014), Project Acceleration (Washington, D.C.: March 2018). Pub. L. No. 91-190, 83 Stat 852 (1970) (codified as amended at 42 U.S.C. §§ 4321-4347). In addition to the contact named above, Vondalee R. Hunt (Assistant Director), Candace Carpenter (Analyst in Charge), Matthew Levie, and Rebecca Makar made key contributions to this report. In addition, Michael Armes, Justin Fisher, Gwen Kirby, Patricia Moye, and Kiki Theodoropoulos contributed to the report.", "summary": "Water resources development projects undertaken by the Corps—such as those to reduce the risks from coastal storms—historically have taken years or even decades to complete. To implement these projects, the Corps first conducts a feasibility study, which includes an analysis of the federal interest and the costs, benefits, and environmental impacts of a project; such studies can take several years to complete. WRRDA 2014 requires the Corps to, among other things, conduct activities to accelerate the completion of feasibility studies. The act also includes a provision for GAO to assess acceleration reforms. This report examines the extent to which the Corps has (1) addressed the WRRDA 2014 feasibility study acceleration provisions, (2) reviewed the impact of its feasibility study acceleration reforms, and (3) maintained complete milestone data for its studies. GAO reviewed WRRDA 2014 and Corps documents; reviewed 19 feasibility studies subject to the act's acceleration provisions; analyzed data on key milestones; and interviewed Corps officials and stakeholders. The U.S. Army Corps of Engineers has taken steps to address some feasibility study acceleration provisions under the Water Resources Reform and Development Act of 2014 (WRRDA 2014) but not others. For example, to implement a provision related to coordination, the Corps in September 2015 issued guidance emphasizing the importance of early coordination with other federal agencies to avoid delays later in the process. However, the Corps has not taken steps to address other provisions, such as one that calls for the Corps to establish a database to make publicly available information on the status of feasibility studies, citing resource constraints. The Corps does not have a plan to address these other provisions. A plan that includes resource estimates would better position the Corps to address the remaining acceleration provisions. The Corps regularly monitors feasibility studies and has conducted some reviews of its acceleration reforms, such as an analysis that found that some studies were too complex to complete within the agency's timing and cost requirements—i.e., within 3 years and for less than $3 million. However, the Corps has not comprehensively evaluated the reforms' impacts. Corps officials and stakeholders expressed differing views on the reforms' impacts on the costs, time frames, and quality of feasibility studies. For example, many Corps officials GAO interviewed said the reforms' overall goals to reduce studies' cost and time frames were positive, but others raised concerns, such as that the $3 million cost limitation may not be realistic for different geographic areas. Corps officials said they have not conducted a comprehensive impact review in part because they are focused on monitoring ongoing studies. These officials said they see the value in conducting such a review as they complete more studies, but they have not developed a plan to do so. Developing an evaluation plan would help the Corps conduct a timely and effective review. The Corps has not maintained complete milestone data in its central data system for the 19 feasibility studies GAO reviewed (see figure). For example, 12 studies did not include data for one or more milestones. Corps officials said agency policy requires the entry of information on 10 key milestones in the agency's central data system. However, GAO found that the policy only explicitly requires that two of the key 10 milestones be entered into the agency's central data system. Without clarifying its policy to help ensure officials enter data on all milestones in the central data system, the Corps will not have complete data to efficiently monitor the progress of feasibility studies. GAO is making three recommendations to the Department of Defense to direct the Assistant Secretary of the Army for Civil Works to (1) develop a plan with resource estimates to address the remaining WRRDA 2014 provisions, (2) develop a plan to comprehensively evaluate the impacts of the agency's acceleration reforms, and (3) clarify its policy to help ensure district officials enter data on required milestones for feasibility studies in its central data system. The agency concurred with the recommendations.", "document_type": "gao"}
{"report": "The Marine Corps uses a fleet of 23 helicopters to support the President in the national capital region and when traveling in the continental United States and overseas. These aircraft have been in service for decades. In April 2002, the Navy began development of a replacement helicopter later identified as the VH-71 program. By 2009, schedule delays, performance issues, and a doubling of cost estimates, from $6.5 billion in 2005 to $13 billion in 2009, prompted the Navy to terminate the program. The need for a replacement helicopter remained, and by April 2012, the Office of the Secretary of Defense approved the Navy’s current acquisition approach. The Navy’s approach is based on the modification of an in-production aircraft to replace the legacy aircraft, by incorporating an executive cabin interior and unique mission equipment such as communications and mission systems, and limiting modifications to the aircraft to avoid a costly airworthiness recertification. In May 2014, the Navy awarded a fixed-price incentive (firm target) contract to Sikorsky Aircraft Corporation, a Lockheed Martin Company, for an Engineering and Manufacturing Development (EMD) phase. The contract includes options for production quantities. The VH-92A presidential helicopter is based on Sikorsky’s S-92A commercial helicopter. The fixed- price incentive contract includes a ceiling price of $1.3 billion that limits the maximum amount that the Navy may have to pay the contractor under the contract subject to other contract terms. The VH-92A is expected to provide improved performance, survivability, and communications capabilities, while offering increased passenger capacity when compared to the current helicopters. Sikorsky is taking S-92A aircraft from an active production line (at the Sikorsky plant in Coatesville, Pennsylvania) to a dedicated VH-92A modification facility for subsystem integration at its plant in Stratford, Connecticut. When the aircraft arrives from Coatesville, some components, such as circuit breaker panels, engines, and main and tail rotor blades are removed. After airframe modifications are done, the aircraft is then transferred to the Sikorsky facility in Owego, New York, where integration of the mission communications system, painting, and contractor-led testing, installation of the executive cabin interior, and the delivery of the aircraft will take place. See figure 1 for a depiction of modification of the commercial S-92A aircraft to the VH-92A presidential helicopter. The VH-92A development program includes delivery of two Engineering Development Model (EDM) test aircraft and four System Demonstration Test Article (SDTA) aircraft. The first flight of the first EDM aircraft took place in July 2017 and the second EDM aircraft’s first flight occurred in November 2017. The two EDM aircraft are currently undergoing government-led integrated testing, at Naval Air Station Patuxent River, Maryland, and were used to conduct an operational assessment in March 2019 to support a decision on whether to enter low-rate initial production. The four SDTA aircraft, now in the modification stages, are production representative aircraft being built under the development contract. These aircraft are to be used in the VH-92A’s initial operational test and evaluation, which is planned to begin in March 2020. The results of that testing will be used to inform a decision whether to enter full-rate production in 2021. These SDTA aircraft will be used to determine whether the VH-92A is operationally effective and suitable for its intended use. In July 2018, the Federal Aviation Administration certified the VH-92A EDM-1 aircraft and supporting documentation to allow delivery to the government under the contract. According to the program office, the first EDM VH-92A configured test aircraft arrived at Naval Air Station in Patuxent River, Maryland, to begin government-led performance testing. The program office explained that in December 2018, the contractor provided VH-92A EDM-2, the second development aircraft, to the Navy and it, too, is undergoing government testing. The VH-92A total program acquisition cost estimate has declined from $5.18 billion to $4.95 billion (then-year dollars)—since the program started in April 2014. Contractor officials attribute that the estimated decline in cost is due to stable requirements, a low number of design changes, and streamlined processes and reviews. The program has incurred delays of about 5 months to the start of its operational assessment due to parts shortages and early integration problems during product development. Program officials told us they have adjusted schedule milestones accordingly and now project that the VH-92A is on track to meet its key performance parameters, including providing a fully interoperable mission communications system (MCS) in time for initial operational test and evaluation in 2020. The Navy continues to reduce its acquisition cost estimate for the VH-92A program. The total VH-92A program acquisition cost estimate has decreased $234 million or about 4.5 percent—from $5.18 billion to $4.95 billion (then-year dollars)—since the program started in April 2014. The total program acquisition unit costs have decreased by the same percentage. According to the program office, this decrease is comprised, in part, by reductions of approximately: $36 million for lower than expected inflation rates, $88 million for efficiencies gained during development, and $103 million for revised spare parts cost and equipment production list. A key factor in controlling total program acquisition cost has been performance requirements stability. The Navy has not added any key performance requirements to the fixed-price contract, thereby limiting cost growth. In addition, the Navy and the contractor have been able to limit the number of necessary design changes that require modifications to aircraft. These modifications are now being incorporated into the four production representative aircraft. The Navy is using an existing basic ordering agreement with Sikorsky, separate from the VH-92A contract, for two additional design changes that are not part of the baseline program. These changes are to allow for improved visibility from the aircraft’s forward door and the addition of a fifth multi-functional display in the cockpit (which is identical to the existing four displays) to improve situational awareness. The program office is working with the contractor to determine the best time to make these modifications to the aircraft in order to minimize the effect on the production schedule. The final costs are still being negotiated; however, the program office expects the cost of implementing these two engineering changes to be minimal relative to the program’s total acquisition cost. The Navy and contractor have also taken advantage of other cost saving measures including streamlining some work processes and revised testing approach for some components; they are also sharing secure facilities used in support of the current presidential helicopter. In addition, they eliminated activities deemed redundant to the Federal Aviation Administration VH-92A airworthiness certification and plan to use a streamlined reporting process for the March 2019 operational assessment. According to program officials, the VH-92A has also optimized its live fire test and evaluation program. Overall, Sikorsky reported it had accomplished about 83.3 percent of development work, with the remainder to be completed by October 2020. As of February 2019, the contractor estimates it would have completed nearly all of its activities necessary to demonstrate performance specification compliance per the contract, by February 2019, and the Navy is now more than halfway through its ground and flight testing requirements needed to a support Milestone C, the decision point for entering into low-rate initial production. The program has addressed delays resulting from technical challenges and new discoveries during development by delaying the start dates for the operational assessment, the low-rate initial production decision, and initial operational test and evaluation by 5 months each. The milestone start dates still meet the baseline schedule thresholds. As we found in the past, part shortages and the integration and assembly effort taking longer than planned have all contributed to delays early in the development of the two engineering development model aircraft. The overall effect has been between 3 and 5 months of schedule delays. In addition, some work initially allocated to the contractor’s site will now be completed at the Naval Air Station, Patuxent River, Maryland. This is a result of the contractor’s inability to get some parts when needed to maintain the planned build schedule. According to the program office, the Navy has implemented a number of mitigation strategies to reduce the effect of the schedule slip, including leasing a commercial S-92A for pilot training, reducing the duration of some future activities, adjusting the program’s schedule, and reexamining and optimizing some work processes to maintain the approved program baseline schedule. We also found that the program’s integrated master schedule met the best practices for a reliable schedule compared against best practices criteria in the GAO Schedule Assessment Guide. The success of programs depend, in part, on having an integrated and reliable master schedule that defines when and how long work will occur and how each activity is related to the others. Such a schedule is necessary for government acquisition programs for many reasons. It provides not only a road map for systematic project execution but also the means by which to gauge progress, identify and resolve potential problems, and promote accountability at all levels of the program. An IMS provides a time sequence for the duration of a program’s activities and helps everyone understand both the dates for major milestones and the activities that drive the schedule. A program’s IMS is also a vehicle for developing a time-phased budget baseline. Moreover, it is an essential basis for managing tradeoffs between cost, schedule, and scope. Among other things, scheduling allows program management to decide between possible sequences of activities, determine the flexibility of the schedule according to available resources, predict the consequences of managerial action or inaction on events, and allocate contingency plans to mitigate risks. Our research has identified 10 best practices associated with effective schedule estimating that can be collapsed into 4 general characteristics (comprehensive, well-constructed, credible, and controlled) for sound schedule estimating. Overall, we found the program’s IMS fully met one and substantially met three of the four characteristics for sound schedule estimating. Table 2 provides a comparison of the planned timeframe for key events at development start to the current estimated schedule. The Navy’s operational assessment began in March 2019 and ended about 30 days later; this is nearly 2 months prior to the Milestone C review, which will authorize low-rate initial production. The contractor’s delivery of the first engineering development model aircraft to the government was about a month late. A Developmental Test and Evaluation official stated that this reduced the already short window of time between the end of development testing and start of the operational assessment. A Director, Operational Test and Evaluation official responsible for monitoring the program expressed concern that there is little time to address any new discoveries found during the operational assessment. The program office acknowledged that, while solutions to any newly discovered problems may not be ready to implement at the start of production, it expects to have enough information from government-led integrated testing and the operational assessment to move forward with the Milestone C decision. According to the contractor, by February 2019, its test program for the first two development aircraft will be nearly completed. In addition, as of December 2018, the government completed about 48 percent of its development ground and flight test points to support Milestone C but is slightly behind, as it had planned to complete about 57 percent at this time. Between August and December 2018, the program conducted three major test events—the Navy conducted 14 landings on the White House south lawn to assess approaches, departures, and operations in the landing zone. The Navy also installed MCS version 2.0 on the second EDM aircraft in support of the operational assessment and tested the ability to transport the VH-92A in a cargo plane. Figure 2 shows the status of government testing as of January 2019. While the program has made progress, the VH-92A program continues to face development challenges that could affect Sikorsky’s ability to deliver fully capable aircraft prior to the start of initial operational test and evaluation. Those challenges include issues associated with the aircraft’s start procedures for the propulsion system, landing zone suitability, and the aircraft’s mission communications system interoperability with secure networks. According to the program office, the performance requirements associated with these challenges may not be fully achieved until after the low-rate initial production decision currently planned for June 2019, which may result in a need to retrofit already built aircraft. Below is additional information on each of those performance requirements. VH-92A aircraft start procedures: As we reported last year, the VH- 92A was pursuing technical improvements related to the S-92A propulsion system, which was not meeting a performance requirement. According to program officials, a previously identified solution is no longer being pursued. However, these officials stated that the program is continuing to assess current capabilities and both material and non-material solutions to any potential capability shortfalls. Testing to demonstrate aircraft performance against the requirement will be completed prior to the Milestone C review in June 2019. Design changes, if needed, will be coordinated with program stakeholders. Program risk for this performance requirement has not changed since our April 2018 report on the program. Landing zone suitability: The VH-92A operates in and out of a variety of restrictive and highly visible landing zones. The White House South Lawn is one of the most frequent locations utilized for helicopter operations in support of the President. As we reported last year, the program was not meeting a key system capability requirement to land the aircraft without adversely affecting landing zones (including the White House South Lawn). The program has still not fully met this requirement and its assessment of this risk has increased since our last report. According to program officials, Sikorsky expects to have a solution for this requirement by November 2020. Mission Communications System (MCS): The mission communications system is a subsystem of the VH-92A aircraft that provides on-board and off-board communications services for the pilots, passengers, and crew. Currently, the VH-92A program has experienced problems connecting the MCS to secure networks, presenting a new risk area for the program. According to program officials, the MCS cannot connect to required secure networks due to recent changes in security protocols. Design changes will be needed to permanently correct this problem. For the March 2019 operational assessment, the program plans to connect to existing networks that do not use the new security protocols. This allowed the operational assessment to proceed but will limit the scope of testing. The Navy plans to have a final fix by January 2020 that will then be incorporated into the four production representative helicopters built under the development contract. These changes have caused the Navy to delay the start of the VH-92 initial operational test and evaluation by 3 months, a delay that is still within the approved program baseline threshold, as discussed earlier. We provided a draft of this report to DOD for review and comment. DOD provided technical comments, which were incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Defense and the Secretary of the Navy. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contacts points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix l. In addition to the contract above, Bruce H. Thomas, Assistant Director; Marvin E. Bonner; Bonita J. P. Oden: Peter Anderson, Juana S. Collymore, Danny C. Royer, and Marie Ahearn made key contributions to this report. Presidential Helicopter: VH-92A Program Is Stable and Making Progress While Facing Challenges. GAO-18-359. Washington, D.C.: April 30, 2018. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-17-333SP. Washington, D.C.: March 30, 2017. Presidential Helicopter: Program Progressing Largely as Planned. GAO-16-395. Washington, D.C.: April 14, 2016. Presidential Helicopter Acquisition: Program Established Knowledge- Based Business Case and Entered System Development with Plans for Managing Challenges. GAO-15-392R.Washington, D.C.: April 14, 2015. Presidential Helicopter Acquisition: Update on Program’s Progress toward Development Start. GAO-14-358R. Washington, D.C.: April 10, 2014. Department of Defense’s Waiver of Competitive Prototyping Requirement for the VXX Presidential Helicopter Replacement Program. GAO-13-826R.Washington, D.C.: September 6, 2013. Presidential Helicopter Acquisition: Program Makes Progress in Balancing Requirements, Costs, and Schedule. GAO-13-257. Washington, D.C.: April 9, 2013. Presidential Helicopter Acquisition: Effort Delayed as DOD Adopts New Approach to Balance Requirements, Costs, and Schedule. GAO-12-381R. Washington, D.C.: February 27, 2012. Defense Acquisitions: Application of Lessons Learned and Best Practices in the Presidential Helicopter Program. GAO-11-380R. Washington, D.C.: March 25, 2011.", "summary": "The mission of the presidential helicopter fleet is to provide safe, reliable, and timely transportation in support of the President. The Navy plans to acquire a fleet of 23 VH-92A helicopters to replace the current Marine Corps fleet of VH-3D and VH-60N aircraft. Initial delivery of VH-92A presidential helicopters is scheduled to begin in fiscal year 2020 with production ending in fiscal year 2023. The total cost of this acquisition program was originally estimated at almost $5.2 billion. The National Defense Authorization Act of 2014 included a provision for GAO to report on the VH-92A program annually, until the Navy awards the full-rate production contract. This report discusses (1) the extent to which the program is meeting its cost and schedule goals and (2) challenges facing the program in system development. To determine how the program is progressing, GAO analyzed program documents; and spoke with officials from the program office, the Defense Contract Management Agency, contractors, Director, Operational Test and Evaluation, and Department of Defense, Developmental Test and Evaluation. GAO also assessed the program's integrated master schedule against GAO best practices. Acquisition cost estimates for the Presidential Helicopter Replacement Program (also known as the VH-92A) have declined from $5.18 billion to $4.95 billion, for 23 new helicopters, since the program started in April 2014 (see table), and the program remains within its planned schedule. The contractor attributes this cost decrease to several factors: stable requirements, a low number of design changes, and program efficiencies. The program has delayed some program milestones—for example, its low-rate production decision—by 5 months from its original baseline goal. Although this remains within the approved schedule, the program will have less time than planned between the end of development testing and start of operational assessment. Program officials told GAO they expect to have enough information from both the government-led integrated testing and the operational assessment to inform the low-rate production decision. Continuing development challenges concerning performance requirements may affect whether the program can deliver fully capable aircraft on time in the future. These include: VH-92A start procedures: As we reported last year, the VH-92A was pursuing technical improvements related to Sikorsky's S-92A propulsion system, which has yet to meet a VH-92A performance requirement. Program risk for this performance requirement has not changed since our April 2018 report on the program. Landing zone suitability: As GAO found in 2018, the program has not yet met a key system capability requirement for landing the helicopter without damaging the landing zone—for example, the White House South Lawn. According to program officials, Sikorsky plans to have a solution for this performance requirement by November 2020 . Mission communications system: The VH-92A program has experienced problems connecting the aircraft's communication system to secure networks, due to changes in network security requirements, presenting a new risk area for the program. The Navy anticipates having a fix by January 2020. These changes are expected to be incorporated into the four production representative helicopters being built under the development contract in time for the program's initial operational test and evaluation. GAO is not making any recommendations in this report, but will continue to monitor the potential cost growth and schedule delays as the program responds to challenges meeting capability requirements.", "document_type": "gao"}
{"report": "CBP facilitates trade and travel, and enforces immigration and customs laws at the nation’s 167 land border crossings along the northern and southern border. CBP’s OFO is responsible for inspecting and processing pedestrians, passengers, cargo, and other items at all land border crossings. OFO has 20 Field Offices nationwide with nine that oversee the operations of all 110 land ports of entry—which may consist of one or more land border crossings—within their designated areas of responsibility. CBP OFAM manages CBP’s portfolio of owned and leased real property, including all 167 land border crossings. OFAM is responsible for capital planning at all land border crossings and for prioritizing capital projects across its portfolio based on need. GSA owns 101 (60 percent) of the 167 land border crossings, partially owns three, and leases 19 (11 percent). CBP owns 40 land border crossings (24 percent) and leases one directly from private owners. The National Park Service owns two and U.S. Forest Service owns one land border crossing. For the 101 land border crossings that GSA owns, it has occupancy agreements with CBP, which is the principal user of the facilities. GSA has responsibilities related to capital planning and construction at all 101 GSA-owned land border crossings. Since CBP’s operations depend heavily on the condition and functionality of infrastructure at land border crossings, GSA works closely with OFAM to plan, design, construct, and implement capital infrastructure improvements to accommodate ever-growing trade and travel at land border crossings. GSA-owned and leased land border crossings consist of large, medium, and small crossings along the northern and southern border. Land border crossings owned by other federal agencies—including CBP—tend to be small by comparison and are typically situated in remote locations along the northern border. See appendix I for more information on the nation’s portfolio of land border crossings. Of the 167 land border crossings at which CBP operates, 120 are located along the northern border and 47 are located along the southern border. Land border crossings vary across the northern and southern border, but are generally designed to process some combination of pedestrian, passenger vehicle, and commercial traffic with separate facilities for each mode. Infrastructure and layout at each land border crossing may vary depending on a variety of factors including the modes of traffic CBP processes at that location, traffic volume, local climate, and area-specific threats, among others. Many large land border crossings, including GSA’s Otay Mesa land border crossing in California, are designed to process pedestrians, passenger vehicles, and commercial traffic and are equipped with distinct infrastructure for each mode of traffic. Other land border crossings are designed to process a single mode of traffic, such as San Luis II in Arizona, which processes only commercial trucks. In general, CBP’s inspection process at land border crossings follows a standard sequence that includes separate areas designated for preprimary inspection, primary inspection, and secondary inspection for each mode of traffic and a main building which houses administrative and operational support activities, which we describe below. Preprimary inspection: Upon proceeding to cross the border into the United States, pedestrians and vehicles enter the land border crossing and are directed to preprimary inspection, where initial screening takes place. Depending on availability, CBP may deploy officers with canines to walk among the vehicles in preprimary waiting to reach an inspection booth. Overhead signage may be present to help CBP actively manage traffic by directing travelers to different lanes according to the type of travel documents they have. For example, CBP may use signs to designate specific lanes for travelers with Radio Frequency Identification (RFID) or other machine readable documents (“Ready lanes”) or for trusted travelers. Infrastructure in the pedestrian preprimary area often includes a space for travelers to queue prior to entering primary inspection. Infrastructure in the preprimary area for passenger vehicle and commercial traffic includes lanes for traffic to queue and radiation portal monitors that are designed to detect radiation and help prevent the smuggling of nuclear material into the United States. The passenger vehicle preprimary area also often includes screening technologies, including license plate readers and RFID readers to capture information on vehicles and RFID-ready travel documents such as passport cards and border crossing cards. At some land border crossings, CBP may use RFID readers in the commercial preprimary inspection area to electronically transmit identification, manifest, and other information to CBP officers prior to entering primary inspection. See figure 1 for examples of preprimary infrastructure. Primary inspection: After preprimary inspection, pedestrians enter the primary inspection area, typically located within the main building. Infrastructure for pedestrian primary inspection may include one or more lanes and officer booths where CBP officers review traveler information. Passenger vehicles and commercial traffic enter a primary inspection area where CBP officers verify passenger identification and perform an initial inspection of the vehicle, which may include a visual inspection of vehicles’ exterior and interior. Infrastructure supporting vehicular primary inspection includes one or more lanes and officer booths. Each booth may be equipped with an HVAC system to keep dangerous vehicle emissions and other fumes from entering the workspace and maintain a safe work environment during extreme heat and cold. Primary inspection booths are designed to be bullet and blast resistant to ensure officer safety. See figure 2 for examples of primary inspection infrastructure. Secondary inspection: If a pedestrian, driver, passenger or vehicle gives reason for suspicion or if the CBP officer is unable to complete the inspection at primary inspection for any reason, the officer may refer them to secondary inspection. Infrastructure in the pedestrian secondary inspection area is typically located within the main building and may include a processing area and a separate secure room where CBP officers can perform more thorough inspections for travelers suspected of criminal activity. Infrastructure in the passenger vehicle secondary inspection area may include work areas where CBP officers can search vehicles, vehicle lifts, and non-intrusive inspection x-ray technologies to identify contraband hidden in concealed compartments. Passengers may wait in the pedestrian secondary inspection area while CBP officers inspect vehicles. Infrastructure in the commercial secondary inspection area may include a loading dock where CBP officers can manually examine cargo and use x-ray technologies to identify hidden contraband. In addition, CBP uses canines at some land border crossings to conduct secondary inspections in the pedestrian, passenger, and commercial environments. See figure 3 for examples of secondary inspection infrastructure. Main buildings: Land border crossings may have facilities that support various administrative and operational activities. Infrastructure at CBP’s main buildings may include agricultural labs, commercial facilities, traveler processing areas, holding rooms, staff work areas, and locker rooms, among other infrastructure. See figure 4 for examples of main building infrastructure. Outbound infrastructure: Pedestrians and vehicles leaving the United States at land border crossings exit through the outbound area. Outbound infrastructure in the passenger vehicle, bus, commercial, and pedestrian area typically consists of one or more exit lanes and may also include inspection booths, inspection technologies, a secondary inspection area and support facilities, among others, to process traffic leaving the United States. See figure 5 for examples of outbound infrastructure. Figure 6 depicts a generic layout of a land border crossing with all modes of traffic. Travel: The volume of traffic at land border crossings varies across the northern and southern borders. At the nation’s busiest land border crossing—San Ysidro in California—CBP processed over 32 million entries in 2017. Conversely, at the Whitlash land border crossing in Montana—one of the smaller land border crossings—CBP processed 1,339 entries that same year. In total, CBP processed over 252 million entries in 2017 including 43 million pedestrian entries, 209 million passengers traveling to the United States in over 104 million passenger vehicle entries, 256,000 buses, and nearly 12 million commercial truck crossings. Figure 7 shows the largest northern and southern border U.S. land ports of entry by volume in 2017. Trade: In 2017, CBP processed and inspected nearly $721 billion in traded goods (imports and exports) through U.S. land ports of entry. As shown in figure 8, trade in goods transported via commercial truck through the largest northern and southern border land ports of entry impacted states across the country. Law Enforcement: Land border crossings serve a critical role in enabling CBP’s enforcement of immigration and customs laws. According to CBP, its officers encountered nearly 139,000 inadmissible individuals at land border crossings in fiscal year 2018. According to CBP, the lack of required travel documents, such as a visa, was the most common reason CBP officers determined individuals to be inadmissible. Further, according to the Drug Enforcement Administration, the nation’s land border crossings remain a target for exploitation by transnational criminal organizations. Specifically, the Drug Enforcement Administration’s 2018 National Drug Threat Assessment found that the most common smuggling method used by Mexican transnational criminal organizations involves transporting illicit drugs through U.S. land border crossings in passenger vehicles with concealed compartments or commingled with legitimate goods on tractor trailers. In fiscal year 2018, CBP seized 363,000 pounds of drugs at land border crossings, including approximately 265,000 pounds of marijuana, 70,000 pounds of methamphetamine, 20,000 pounds of cocaine, and 1,400 pounds of fentanyl. As part of its capital planning process, CBP is responsible for identifying land border crossing infrastructure needs and prioritizing capital projects across its portfolio of 167 land border crossings. At CBP-owned land border crossings, CBP generally funds these projects and hires a contractor to plan and execute capital infrastructure projects. At GSA- leased land border crossings, CBP and GSA typically work with the property owner to plan and execute capital projects. The owner of the land border crossing funds these projects, while CBP funds any alterations needed to fulfill its mission. At GSA-owned land border crossings, CBP typically works with GSA to complete a feasibility study and uses this information to prioritize infrastructure projects. According to GSA policy documents, feasibility studies are intended to determine the technical and economic viability of a project, define the project budget and scope, and establish an initial project design. GSA and CBP are to further refine land border crossing capital projects with a program development study, which updates project plans and budgets and provides the necessary information to pursue project funding. Each year, the Office of Management and Budget reviews each project included in GSA’s budget request and Congress authorizes projects and appropriates project funds as part of the federal budget cycle. GSA typically includes CBP’s top priority land border crossing capital infrastructure projects in its annual budget submission. GSA may pursue project funding for design and construction in separate budget requests or in a single appropriation, depending on the contract vehicle used. Once funded, GSA hires one or more contractors to design and execute the project. Figure 9 identifies funding for CBP and GSA-owned land border crossings in fiscal years 2009 through 2019. CBP defines its general land border infrastructure requirements in its Land Port of Entry Design Standards, which describe various infrastructure at land border crossings and detail how this infrastructure should operate. According to CBP, it updates these standards every few years to ensure the standards reflect CBP’s changing mission, including new technologies and infrastructure requirements. CBP officers we spoke with at 16 land border crossings and OFO field offices that oversee land border crossings reported examples of land border crossing infrastructure constraints they face at each stage of the inspection process including preprimary, primary, and secondary inspections. CBP relies on infrastructure to fulfill its mission at land border crossings. Specifically, according to CBP, well-functioning infrastructure is a critical factor in its ability to effectively screen persons and cargo, and facilitate cross-border travel and trade. For example, CBP officials stated that the number of operational inspection lanes is a key variable that affects traffic wait times. These officers also identified land border crossing infrastructure challenges with office space and port security. Examples of infrastructure constraints identified by CBP officers include: Limited space in the preprimary inspection area. According to CBP officers, land border crossings with primary inspection booths located in close proximity to the border line with Mexico have restricted space for CBP to conduct operations in the preprimary area. Figure 10 below shows a photo of restricted space in the preprimary area at a land border crossing on the southern border. Non-functioning screening technology in the preprimary inspection area. CBP officers stated that vehicle inspection technologies may not always function correctly. For example, at a land border crossing on the southern border, license plate readers and radiation portal monitors are inoperable at least once a week during summer months due to overheating, according to CBP officials. Temperatures can exceed 120 degrees Fahrenheit and the technology is exposed to the sun. Figure 11 shows license plate readers and radiation portal monitors in the preprimary area exposed to the sun at a land border crossing on the southern border. Officer inspection booths in the primary inspection area in need of repair. CBP officers stated that officer inspection booths may be inadequately cooled or heated resulting in officers more frequently rotating out of the booths for health and safety reasons. At one land border crossing, officers stated that the booth windows provide limited visibility since the old bullet resistant glazing has deteriorated and clouds officers’ view. At another land border crossing we visited, we observed that the doors on the primary inspection booths do not have working locks. Officers stated that as a result, when the land border crossing closes overnight they are unable to secure the booths or the computer equipment inside. Inadequate holding facilities in the secondary inspection area. Holding facilities at several land border crossings we visited had holding rooms that did not meet current CBP safety requirements, according to CBP officers. Officers at two land border crossings stated that safety concerns included inadequate ventilation. Officers at another land border crossing identified exposed wiring in a holding room as a safety hazard. Other land border crossings we visited did not have holding rooms and officers stated they detain individuals in the lobby of the administration building as a result. Figure 12 shows examples of holding facilities at land border crossings on the northern and southern borders that CBP officials identified as not meeting CBP requirements. Lack of availability of non-intrusive inspection (NII) technology in the secondary inspection area. CBP officers stated that the availability of NII technology improves their ability to conduct inspections. However, NII technology is not always available because it may need maintenance or repair, or CBP may share the technology with multiple land border crossings. Officers stated they may perform manual inspections of vehicles when NII technology is not available, which they noted can be less effective. Inadequate facilities for canine inspection in the secondary inspection area. CBP officers provided examples of limited facilities for inspection canines. For example, officers at one land border crossing stated they do not have a dedicated area to exercise inspection canines. Officers at another land border crossing stated they recently converted a storage closet into a climate-controlled canine kennel within the secondary inspection building. Previously, the CBP officers at this land border crossing kept the canines in running vehicles with air conditioning to keep them cool. Impeded traffic flow within the land border crossing. CBP officers identified challenges with facilitating traffic flow within the land border crossing. For example, the layout at a commercial land border crossing on the southern border impedes the flow of traffic because it requires commercial trucks to make a series of sharp turns as they travel through the border crossing. In addition, commercial traffic referred for secondary inspection must cut across four primary egress lanes to enter and exit the secondary inspection area. According to CBP officers, commercial trucks proceeding toward the border crossing exit may need to stop or reverse direction to create space for the trucks entering or exiting the secondary inspection area which creates delays in processing commercial traffic. Figure 13 shows an aerial view of a land border crossing with a diagram of where CBP officers identified that the land border crossing layout impedes traffic flow. Insufficient capacity to accommodate the volume of traffic. CBP officers stated that the number of travelers can exceed the capacity of the facility. For example, CBP officers stated that insufficient number of inspection lanes can result in lengthy wait times for travelers. Limited administrative space. CBP officers stated that insufficient administrative office space can be a challenge at land border crossings. For example, one land border crossing we visited did not have sufficient space for officer lockers and as a result placed some lockers in the contraband seizure room. Figure 14 shows lockers located in the contraband seizure room at a land border crossing on the northern border due to insufficient administrative space. Port security limitations. CBP officers also described challenges with land border crossing security. For example, officers stated the lack of measures to prevent travelers from exiting the crossing without authorization, such as vehicle barriers and security gates, impedes CBP’s ability to stop drivers from fleeing the land border crossing and entering the United States without inspection. Figure 15 shows exit lanes constructed with temporary barriers to control the flow of traffic leaving the land border crossing and entering the United States. Lack of inspection facilities for outbound traffic. CBP officers at land border crossings without facilities to inspect outbound traffic can face difficulties when inspecting traffic exiting the United States. For example, at one land border crossing without outbound inspection facilities, officials stated they park CBP vehicles in the outbound traffic lanes to slow traffic so that CBP officers can stop and inspect vehicles exiting the United States. CBP collects information on the condition of infrastructure at some land border crossings through contracted Facility Condition Assessments (FCA), but has not assessed conditions at all land border crossings. FCAs are engineering inspections that evaluate the condition of the facility and identify repair and improvement needs. The output of an FCA is a report that describes infrastructure deficiencies at a facility and represents the condition of the land border crossing infrastructure at the time of the FCA. From 2016 through 2018, CBP and GSA assessed the condition of infrastructure at 95 of the 167 land border crossings. As of December 2018, CBP had conducted FCAs at 74 of the 167 land border crossings within the previous three years. In addition, according to CBP officials, in 2016 GSA funded and conducted Building Engineering Reviews at 21 land border crossings in response to conversations between CBP and GSA on how to improve GSA service delivery at land border crossings. CBP officials stated they use facility condition information from the 2016 Building Engineering Reviews because they contain information similar to what CBP collects through an FCA. According to GSA officials, GSA now rarely conducts Building Engineering Reviews because they are costly and their data quickly become obsolete. GSA now uses other tools to assess infrastructure condition and GSA officials were not aware of any reviews at land border crossings since 2016. See table 1 for a breakdown of the land border crossings that CBP and GSA have assessed. According to the assessments, the condition of infrastructure varies across land border crossings. The facility condition index—the ratio of the costs to correct facility infrastructure deficiencies to the total replacement value of the facility—ranges from 0 percent to 69 percent across the 95 FCAs and Building Engineering Reviews conducted between 2016 and 2018. These assessments identified approximately $140 million in estimated infrastructure deficiencies and the average facility condition index is 16 percent. See table 2 for the distribution of facility condition indices across land border crossings by ownership type. See appendix I for facility condition index scores across CBP’s land border crossing portfolio. CBP began conducting FCAs at CBP-owned land border crossings in 2008. OFAM officials stated they set a goal of conducting FCAs at each CBP-owned land border crossing on a three-year cycle, but have not always been able to do so due to resource constraints. Our analysis identified that CBP conducted FCAs at only four of the 40 CBP-owned land border crossings over three years—2016 to 2018—when its goal was to have conducted FCAs at all 40 facilities over this time frame (see table 1 above). CBP also began conducting FCAs at GSA-owned facilities in 2016, and at GSA-leased facilities in 2017. According to CBP officials, they plan to conduct several FCAs at selected GSA-owned facilities each year to obtain information on the condition of infrastructure at these facilities, though there is no required interval at which they must assess these facilities. CBP officials stated they prioritize GSA land border crossings in need of capital investment when selecting which facilities to assess. DHS Directive 119-02-004 “DHS Real Property Facility Condition Assessment” instructs each DHS component—including CBP—to implement and maintain a program to ensure that the condition of real property is assessed every three years and updated each fiscal year through FCAs beginning in fiscal year 2018. The Directive applies to land border crossings owned by CBP and is intended to ensure that CBP collects information on the condition of infrastructure across these facilities. Although CBP has a goal of conducting FCAs at CBP-owned land border crossings every three years, it has not met this goal in recent years as CBP assessed only four of the 40 land border crossings from 2016 through 2018. According to CBP officials, FCAs older than three years may not accurately reflect the current condition of infrastructure at land border crossings. According to OFAM officials, they have not developed a plan to ensure that CBP implements its program consistent with the Directive by conducting FCAs on a three-year cycle going forward due to limited resources to conduct the assessments. Specifically, CBP officials stated that CBP has not been able to fully fund the FCA program due to other competing facility priorities. However, developing a plan that accounts for the new requirements under the Directive could assist CBP in planning funding needs for the FCA program. Further, developing and implementing a plan to ensure CBP executes its FCA program consistent with Directive 119-02-004 would assist CBP in making resource decisions for this program. Implementing its FCA program consistent with DHS Directive 119-02-004 would enable CBP to collect more complete and current information on the condition of infrastructure at land border crossings it owns. CBP and GSA conduct separate assessments of facility conditions at GSA-owned land border crossings; however, they do not routinely share or use the results of each other’s efforts to inform their assessments of facility condition. More specifically, separate from CBP’s process for assessing facility condition, GSA uses its Building Assessment Tool to assess the condition of infrastructure across its entire real property portfolio, including land border crossings. This process is intended to assist GSA in estimating its future costs for repairing and maintaining the buildings in its portfolio. Although the CBP FCA and the GSA Building Assessment Tool both assess elements of facility condition, these assessments have different methodologies, scopes, and purposes. We reviewed a 2018 CBP comparative analysis of the FCAs and Building Assessment Tool processes. CBP’s analysis showed that FCAs are detailed assessments of all building systems that CBP uses at a land border crossing. According to CBP officials, CBP uses FCAs to collect information on the condition of infrastructure at land border crossings and to inform land border crossing capital infrastructure projects. In comparison, GSA’s Building Assessment Tool is a standardized assessment used across GSA’s federal real property portfolio to identify, plan for, and prioritize repair and maintenance needs across GSA properties. As a result, while the two types of assessments may be related in some aspects, officials from each agency stated they could not use the other’s facility assessment in place of their own. GSA officials assessing land border crossing infrastructure condition are not required to consult with CBP officials who operate the port or review any existing CBP FCAs, according to GSA officials. CBP provides GSA with pre-assessment questionnaires prior to conducting FCAs at GSA- owned land border crossings. These questionnaires inquire about available GSA information on facility condition. However, CBP officials stated they do not specifically request GSA Building Assessment Tool data, and as a result, have not generally received these data prior to conducting a FCA. GSA officials stated that CBP FCAs and GSA Building Assessment Tool assessments differ in scope and as a result GSA does not use FCAs in place of their Building Assessment Tool assessments. However, FCAs identify infrastructure needs at land border crossings and the results could provide GSA with an understanding of infrastructure needs identified by CBP at land border crossings. Likewise, GSA’s Building Assessment Tool is used to identify infrastructure in need of repair and could provide CBP with an understanding of infrastructure needs identified by GSA. We have previously identified key practices for collaboration among federal agencies. Specifically, agencies can enhance and sustain their collaborative efforts by identifying and addressing needs by leveraging resources. According to CBP officials, improving information sharing on facility condition could help ensure that both CBP’s and GSA’s assessments are as accurate and complete as possible. Moreover, using each other’s facility condition information could enable CBP and GSA to improve the accuracy and completeness of their respective assessments of facility condition at land border crossings. CBP uses a software system called TRIRIGA to manage its real property asset portfolio, but information in this system is not fully reliable. Among other functions, CBP uses TRIRIGA to track infrastructure needs and the condition of facilities at land border crossings. CBP identifies infrastructure needs through FCAs and records these data in TRIRIGA. CBP also identifies additional infrastructure needs as they arise and records these data in TRIRIGA. For example, an infrastructure need may arise at a building and be recorded in TRIRIGA in the months following a CBP FCA. CBP uses TRIRIGA data to calculate a score reflecting the overall current condition of infrastructure at a land border crossing. CBP uses this score on condition to inform internal planning and prioritization of maintenance and repair projects at the local level, according to CBP officials. In addition, CBP’s goals for facility condition data in TRIRIGA include making facility condition information available in real time, starting with TRIRIGA for responses to data calls and reporting, and using data in the system for more efficient planning and decision making. However, according to CBP officials, land border crossing facility condition data in TRIRIGA have not been consistently reliable because some data on infrastructure needs are duplicative, out of date, or incomplete. Duplicate Data: CBP officials stated that in the past, OFAM officials responsible for entering infrastructure needs into TRIRIGA created duplicate entries in some instances. For example, OFAM officials have identified, and entered into TRIRIGA, infrastructure needs at land border crossings that had already been identified and entered in the past. As a result, TRIRIGA double-counted the costs associated with these duplicate infrastructure needs which impacted the reliability of the calculation of the score on facility condition for the associated land border crossing. According to OFAM officials, they have taken several steps to improve the TRIRIGA data entry processes. During the course of our review, OFAM officials identified internal confusion regarding who had the authority to remove infrastructure needs from TRIRIGA. In response, in April 2019 OFAM developed new guidelines clarifying roles and responsibilities for accurately entering FCA data and removing infrastructure needs from TRIRIGA. OFAM officials stated they expect this process to avoid duplicative data entry in the future. Further, as described earlier, by conducting FCAs for each CBP-owned land border crossing every three years, updating them annually consistent with DHS Directive 119-02-004, and then entering the results into TRIRIGA in accordance with the new guidelines for reviewing existing infrastructure needs and removing them as needed, CBP would be positioned to more frequently review and validate these data in the system on an ongoing basis. Out of Date Data: Officials stated that FCA data for some land border crossings in TRIRIGA originate from as early as 2013, the last time CBP conducted an FCA at those border crossings. As a result, TRIRIGA does not accurately reflect the current condition of these facilities. Historically, CBP has updated TRIRIGA with facility condition information collected through FCAs. As described earlier, DHS Directive 119-02-004 directs CBP to conduct FCAs for each CBP- owned land border crossings every three years and update them annually. By developing and implementing a plan to complete more timely FCAs at CBP-owned land border crossings, CBP will be better positioned to ensure that TRIRIGA is updated to reflect more current condition information. In addition, as CBP continues to conduct FCAs at GSA owned and leased land border crossings, CBP can continue to update TRIRIGA with more current information on facility condition consistent with OFAM’s April 2019 guidance on TRIRIGA data entry. Incomplete Data: Officials stated that because CBP oversees maintenance and repair work at CBP-owned land border crossings, data in TRIRIGA are more reliable for these land border crossings than for GSA-owned land border crossings where GSA is responsible for planning and executing maintenance and repair work. CBP officials said that while they do identify infrastructure needs at GSA- owned land border crossings and enter related information into TRIRIGA, the information on these needs can be incomplete. CBP officials stated that for example, a past CBP FCA may have identified a building roof in need of repair. Following the FCA, CBP would then enter a record of this infrastructure need in TRIRIGA. If GSA repaired the roof during the following year as part of its planned maintenance work, but did not inform CBP headquarters, TRIRIGA would continue to identify a deficient roof at the land border crossing after GSA repaired it. CBP officials stated that GSA may conduct maintenance or repair work to address an infrastructure need without CBP’s knowledge because CBP and GSA did not have a process for GSA to notify CBP of maintenance and repair work the agency conducts at land border crossings. According to OFAM officials, GSA began sharing with OFAM monthly summary-level data on maintenance GSA performs at land border crossings. However, these data do not include the level of detail required to update condition data or close out deficiencies in TRIRIGA. We previously identified key practices for collaboration among federal agencies, including that agencies can enhance and sustain their collaborative efforts by identifying and addressing needs by leveraging resources. Sharing information on GSA maintenance and repair work at GSA-owned land border crossings at the level of detail necessary for CBP to update TRIRIGA would enable CBP to improve the completeness and accuracy of data in the system. As a result, CBP would have access to more complete and accurate data to use when planning and prioritizing infrastructure maintenance activities, improving the availability of real-time facility condition information, and responding to data calls and reporting. For example, more complete and accurate data in TRIRIGA would better position CBP to identify and report to Congress on improvements needed at land ports of entry. Specifically, the 2018 United States Ports of Entry Threat and Operational Review Act requires CBP to submit to Congress a threat and operational analysis that includes, among other elements, an assessment of current and potential threats due to security vulnerabilities and unlawful entry, and improvements needed at ports of entry to enhance travel and trade facilitation and reduce wait times. CBP officials stated they have not yet determined which data they will use to develop this report, but this reporting requirement is one potential example of how more reliable data from TRIRIGA could be used to effectively report on the condition of land border crossing infrastructure. CBP prioritizes prospective land border crossing projects within its annual Five-Year Land Port of Entry Capital Investment Plan (five-year plan). CBP is statutorily required to complete a detailed five-year plan each fiscal year and include it with its annual budget submission to Congress (i.e., President’s budget), which typically occurs in February. Each five- year plan is to cover all federal land border port of entry projects with a yearly update of total projected future funding needs delineated by land port. According to CBP officials, CBP generally completes an initial draft of the five-year plan in November or December each fiscal year and submits it to CBP and GSA leadership, DHS leadership, and the Office and Management and Budget for review and approval. However, our analysis of CBP’s five-year plans for fiscal years 2014 through 2018 identified that CBP completed its five-year plan after the annual budget submission in fiscal year 2016 and 2018 and did not complete a plan at all in fiscal year 2017. Specifically, CBP submitted its fiscal year 2016 five-year plan in July 2016—163 days after CBP’s annual budget submission—and its fiscal year 2018 plan in October 2018—235 days after CBP’s annual budget submission. Table 3 identifies the days between CBP’s submission of its five-year plan and budget to Congress in fiscal years 2014 through 2018. CBP officials stated they completed the five-year plans after the annual budget submission in fiscal years 2016 and 2018, and did not complete a five-year plan for Congress in fiscal year 2017, due to delays in the review and approval process. CBP officials stated the review and approval process may take several months to complete due to revisions at various stages and competing priorities among stakeholders that may slow the process. Officials also said they have little control over how long it takes stakeholders within CBP leadership, DHS, and the Office of Management and Budget to review and approve the five-year plan. Consequently, according to CBP officials, CBP has not attempted to establish time frames for completing the plan. While we acknowledge that setting time frames for completing the plan may not guarantee timeliness, establishing time frames for each stakeholder could help measure and assess progress in reviewing and approving the draft plan. Standards for Internal Control in the Federal Government state that management should define objectives so that they are understood at all levels, including by outlining the time frames for achievement of those objectives. By establishing time frames for stakeholders throughout the five-year plan review and approval process, CBP would be better positioned to identify and address sources of delay and could improve its ability to meet statutory reporting requirements by including its five-year plan with its annual budget submission to Congress. CBP develops a list of roughly eight to twelve priority land border crossing capital projects each year and presents these projects to Congress in the five-year plan, but the agency has not established a consistent methodology in developing this list. CBP’s five year plans note five broad steps CBP follows in developing the list of priority capital projects. These steps are applicable to the entire land border crossing portfolio— regardless of ownership—and include: 1. Strategic Resource Assessment (SRA): According to the five-year plan, CBP conducts SRAs cyclically to compare infrastructure requirements across its portfolio and present a uniform picture of capital investment needs at all land border crossings along the northern and southern borders. 2. Capital Project Scoring: Using data generated during the SRA, CBP scores and ranks each land border crossing by criticality and relative urgency of infrastructure needs. 3. Sensitivity Analysis: CBP then applies a sensitivity analysis and updates its initial ranking based on factors unaccounted for through the SRA, including unique regional conditions, bilateral planning with partners in Canada and Mexico, or interests of other federal, state, or local agencies. 4. Assess Feasibility and Risk: CBP coordinates with project stakeholders—including GSA for all GSA-owned land border crossings—to evaluate the feasibility, risk, and cost associated with project implementation by completing a feasibility study. These studies analyze alternatives and review environmental, cultural, and historic preservation requirements as well as land acquisition requirements and procurement risks. CBP also assesses the likelihood of obtaining funding for the proposed project. 5. Establish a Five-year Capital Investment Plan: After the SRA and the scoring, analysis, and assessment phases, CBP prioritizes land border crossing capital projects and develops a five-year capital investment plan in coordination with GSA. CBP updates the plan annually, taking into account the changing conditions at land border crossings. Although CBP has outlined the five broad steps it uses to prioritize projects, our analysis of CBP’s five-year plans for fiscal years 2014 through 2018 identified that CBP did not follow a consistent methodology across the years or across projects when prioritizing prospective land border crossing projects. For example, in some five-year plans CBP prioritized projects by comparing relative need at land border crossings using more recent SRA data for some land border crossings and older data for other land border crossings. In one such instance in fiscal year 2018, CBP compared relative need using 2015 data for some land border crossings and data dating as far back as 2007 for other land border crossings. Although CBP’s five-year plan states that CBP performs SRAs cyclically, CBP has not established the frequency at which SRAs are to be completed. In 2015, CBP completed a partial SRA update for 36 of 167 land border crossings that it considered high-priority, but has not completed a portfolio-wide SRA since 2007. Our analysis of CBP’s five-year plans for fiscal years 2014 through 2018 also identified that CBP had feasibility studies for some, but not all, projects listed in the five-year plans. Specifically, our analysis identified that CBP had feasibility studies for approximately two thirds (28 of 41) of the projects it prioritized over these years. CBP officials told us that due to the limited shelf-life of feasibility studies (two to three years), CBP and GSA target high-priority land border crossing projects for feasibility studies that are likely to receive funding within the next two to three years. However, of the top five projects CBP ranked as the highest priority in each of its five-year plans in fiscal years 2014 through 2018, CBP completed feasibility studies for approximately half (12 of 20) of these projects. Further, among the 12 projects CBP ranked in the top five in its fiscal years 2014 through 2018 five-year plans that had feasibility studies, 10 of 12 projects had a feasibility study that was more than five years old when CBP prioritized them. In addition, CBP prioritized projects on each of its five-year plans by comparing cost estimates developed through different methodologies. Specifically, CBP prioritized projects using detailed cost estimates developed as part of a feasibility study for some projects and order of magnitude cost estimates for projects that do not have a feasibility study or that had an out-of-date feasibility study. These order of magnitude cost estimates were significantly different from the cost estimates that were later produced for these projects through feasibility studies. For example, CBP’s fiscal year 2015 plan included an order of magnitude cost estimate of $95 million to implement a single project at two separate crossings—San Luis I and II. However, after completing a feasibility study for the project in October 2017, GSA estimated it would cost $289 million—a nearly 300 percent cost increase—to complete the project. CBP outlines the five broad steps it is to take in general to develop a list of priority projects each year and establish an annual five-year plan and these steps are documented at a high level. However, there is not a detailed planning methodology that would help ensure officials consistently and appropriately develop and assess priority projects each year. For example, the five-year plans do not define what minimum steps CBP personnel are to take at each step in the process, such as guidance and procedures on which projects require feasibility studies. The plans also do not include time frames for completing each step, such as establishing expectations for the frequency at which CBP personnel are to update SRA data. As a result, CBP officials told us they rely on informal processes and procedures to complete these steps and prioritize land border crossings in its annual five-year plans. CBP officials acknowledged that they have not issued formal guidance documenting the steps in its prioritization process or establishing procedures and time frames for each step, but stated that they plan to do so going forward. Specifically, officials told us that CBP plans to document its process for prioritizing land border crossing projects to improve transparency, better educate staff on roles and responsibilities, and help ensure CBP consistently applies this process each year. While these would be positive steps, CBP was not able to provide information on specific plans or expected time frames for implementing these steps. Standards for Internal Control in the Federal Government state that management should define objectives so that they are understood at all levels by outlining what is to be achieved, how it will be achieved, and the time frames for achievement. The standards also establish that management should implement control activities through documented policies. To achieve this, management should document policies that establish each unit’s responsibility for achieving the objectives related to an operational process. Establishing and documenting a methodology for CBP’s annual land border crossing capital prioritization process, including procedures and time frames for each step, could help ensure that CBP identifies key activities needed to prioritize projects and that CBP personnel follow a consistent methodology across projects and across years. For example, such a methodology could help CBP identify which projects require feasibility studies in a given fiscal year, and how they are to use information on project feasibility, risk, and cost when prioritizing projects. Further, having time frames for each step could help CBP determine how often to update SRA data across its portfolio for purposes of comparing relative infrastructure needs at land border crossings. Lastly, establishing and documenting a land border crossing prioritization methodology could help CBP ensure it consistently provides Congress with more up-to-date and complete information in its five-year plans. From fiscal years 2014 through 2018, GSA initiated or completed 10 capital infrastructure projects at eight land border crossings. Among these projects, six were complete and four were ongoing as of March 2019. Projects at three of these border crossings—Alexandria Bay, Calexico West, and San Ysidro—consist of multiple phases. GSA manages each phase as a distinct project funded under separate congressional appropriations and executed through separate contracts. Across all 10 projects, the amount of schedule growth against the original schedule baselines ranged from 0 percent growth to 59.2 percent growth, though several of these projects revised their baselines to account for the schedule growth. Half of the projects experienced less than 10 percent schedule growth above their original schedule baselines, and the other half experienced more than 10 percent schedule growth. When accounting for projects for which schedule baselines were revised, among the 10 projects, six have met or are on track to meet schedule baselines. The Alexandria Bay project, which GSA expects to complete in January 2020, is the only project on track to meet its original schedule baseline. GSA revised its schedule baselines during construction for the remaining five projects and all have met or are on track to meet these revised baselines. More specifically, Calexico West, Derby Line, and Nogales West-Mariposa are the three projects that are complete and met revised schedule baselines. San Ysidro phases II and III are the two ongoing projects that are on track to meet their revised schedule baselines as of January 2019. See table 4 below for a breakdown of project schedule performance. Four of GSA’s 10 projects did not meet, or are not expected to meet, their schedule baselines. The Tornillo-Guadalupe project experienced the most schedule growth of the projects we reviewed. GSA completed the Tornillo-Guadalupe project in October 2014, 470 days later than its original baseline in July 2013 and 80 days later than its August 2014 revised baseline. Schedule growth at Tornillo-Guadalupe was primarily due to delays in the construction of corresponding Mexican infrastructure, unstable soil conditions, and contractor performance, according to GSA officials. In addition to Tornillo-Guadalupe, the San Ysidro I and Laredo projects did not meet their schedule baselines, and the Columbus project is not on track to meet its schedule baseline, as of January 2019. Of the four projects that experienced schedule growth against their final schedule baselines, two projects had less than 5 percent growth and two projects had about 10 percent growth. While none of the 10 projects kept costs at or below baselines, eight projects stayed within their 10 percent cost contingency allowance. The Tornillo-Guadalupe and Derby Line projects both exceeded their cost contingency allowance. GSA completed the Tornillo-Guadalupe project in October 2014 at a final construction cost of $59 million—18.7 percent above its cost baseline—due to challenges described above. GSA completed the Derby Line project in November 2018 with a final construction cost of $26.4 million—10.6 percent above its cost baseline— mainly due to CBP-requested changes, according to GSA officials. The total baseline construction cost for all 10 projects, as of January 2019, is $1.03 billion and the combined current contract value is $1.09 billion— which is about $62.9 million (6.1 percent) over baseline budgets. See table 5 below for a breakdown of project cost performance. GSA has completed, or expects to complete, nine out of the 10 projects at full scope. GSA reduced scope for one project—Laredo, TX—due to cost concerns after the construction contract award. During Laredo project construction, GSA removed plans to build a footbridge spanning the passenger vehicle primary lanes and cosmetic finishes to buildings to avoid further cost overruns, according to GSA and CBP officials. See appendix II for detailed descriptions of the ten projects. GSA reported facing challenges planning and designing land border crossing capital projects. These challenges included delays between design and construction and the division of large projects into smaller phases, which GSA officials reported led to higher costs and longer development timelines. Funding Lags. GSA officials reported that funding lags between project design and construction can increase costs and extend construction timelines. GSA has requested separate appropriations for project design and construction using a model known as design-bid-build, which created the potential for funding lags to occur. According to CBP and GSA officials, the process from requesting an infrastructure project to completing the project lasts approximately 7 years. However, GSA experienced funding lags of up to 10 years between design and construction. Figure 16 identifies development timelines from initial planning through construction for our 10 selected land border crossing capital projects. The cost of labor and materials can escalate when funding lags occur between design and construction. For example, after completing design for the Calexico West project, GSA requested construction funding in fiscal year 2010, but did not receive funding until five years later. As a result, estimated construction costs escalated from $78.5 to $90.8 million (16 percent). To keep project cost estimates up-to-date during funding lags, GSA officials explained that GSA typically increases project cost estimates over time to account for inflation, changes in the labor market, and the cost of materials, among other factors. To help address cost escalation, contractors have purchased materials upfront, and GSA has combined projects that would otherwise be constructed separately. To address increasing materials costs for the Alexandria Bay project, the contractor purchased steel upfront in order to avoid future cost increases due to import tariffs, according to GSA officials. The Laredo project faced significant labor and material cost growth due to a boom in the Texas construction market. As a result, GSA decided to combine the two Laredo crossings into one contract to lock in prices and avoid paying higher prices in the future. According to GSA officials, funding lags between design and construction may result in outdated project designs that do not reflect newer CBP infrastructure requirements. In such instances, GSA must invest additional time and resources to update project designs and incorporate new CBP requirements, such as newer inspection technologies or facilities. According to GSA officials, design refreshes can be challenging due to a lack of continuity and staff turnover at the architecture and engineering firms that originally designed the project. In some instances, according to GSA officials, the original firms may not be available or interested in redesigning the project and GSA may need to hire a new firm. For example, GSA spent $3.3 million on design for the Columbus project in fiscal years 2007 and 2009. However, the funding lag between design and construction required a $7.4 million design refresh in fiscal year 2014. In another example, GSA established the Calexico West project’s design concept in fiscal year 2007, but didn’t receive construction funding until fiscal year 2015. According to officials, GSA had to spend approximately $1 million for a design refresh to account for new CBP requirements, which resulted in a longer development timeline. To address risks of funding lags with the design-bid-build approach, GSA has shifted toward using contract vehicles for land border crossing capital projects that combine design and construction into a single appropriation. This approach allows for more precise planning, less risk from delays, and less time for costs to escalate, according to GSA officials. Project Phasing. According to GSA officials, OMB may request that GSA and CBP divide large projects into separate phases when high-cost projects are unlikely to be funded in a single appropriation. For example, of the eight border crossing locations represented across the 10 projects in our review, CBP and GSA broke three projects at three locations into phases to obtain approval: Alexandria Bay, Calexico West, and San Ysidro. However, for similar reasons as those related to funding lags between design and construction, breaking up projects into smaller phases can increase overall costs and add years to project timelines. According to GSA and CBP officials, when appropriations do not align with project schedules, contractors may leave the site after completing a single phase to pursue new work opportunities. Additionally, by the time GSA receives appropriations for latter phases, the contractor must remobilize equipment and labor, costs of labor and material may have increased, and projects may need design refreshes, as described above. For example, after Calexico West phase II remained unfunded two years after phase I was completed, GSA estimated that project costs increased by $27.7 million due to increases in labor and materials and potential redesign work. In another example, GSA officials told us that after originally designing the Alexandria Bay project as a single-phase in 2010, OMB directed GSA to break the project into two phases in 2014 to increase the likelihood of funding. According to GSA officials, redesigning Alexandria Bay as a two- phase project added as much as $16.5 million to total project costs. Construction costs escalated by about $58.4 million from the single-phase estimate in fiscal year 2011 to fiscal year 2017 when phase I construction began, an increase of 36 percent. Further, completing the Alexandria Bay project in two phases added an additional three years to the project timeline. While breaking projects into phases can potentially lead to higher costs, GSA officials told us that doing so can be an effective way to start work on a large capital project when funding for the entire project is not available in a single year and can be cost effective when GSA receives appropriations for each phase in line with its planned schedule. GSA and CBP have reported facing challenges constructing land border crossing projects, including those related to CBP-requested changes, geographical and environmental factors, and inadequate or incomplete infrastructure in neighboring countries. CBP Change Requests. CBP may request modifications to ongoing projects through Reimbursable Work Authorizations to meet changing infrastructure requirements, such as incorporating newer technologies and CBP design standards. These requests range from installing new information technology and security equipment to enhancing office, holding facilities, or public-facing areas of the port. CBP change requests are often necessary because the span between design and construction can last up to 10 years, according to CBP and GSA officials. While CBP typically pays for the cost of these modifications, GSA must incorporate changes into existing project plans, which can result in schedule growth, according to GSA officials. CBP-requested changes led to cost and/or schedule growth at the Calexico West, Columbus, Derby Line, Nogales West-Mariposa, and San Ysidro land border crossing projects, according to GSA officials. In one example, GSA revised the Nogales West- Mariposa project’s schedule baseline from March 2014 to August 2014 to incorporate a $10 million Reimbursable Work Authorization from CBP that added an outbound inspection facility. Environmental and geographical challenges. Environmental and geographical factors including extreme climates, remote locations, and limited space, can create construction challenges, according to CBP and GSA officials. Extreme climates can disrupt construction activities, such as concrete work at land border crossings. CBP officials said that at some southern crossings concrete may crack when it dries too quickly due to extreme heat, requiring contractors to pour concrete in the early morning when temperatures are cooler. However, officials said that because this work typically occurs outside of regular business hours, it often comes at a premium and can increase project costs. Along the northern border, contractors may not be able to do concrete work during the winter months because temperatures can be too cold to pour concrete. At Derby Line, because of delays earlier in construction, work extended into an additional winter season, contributing to cost and schedule growth because contractors were slowed or limited by weather, according to GSA officials. Environmental conditions surrounding construction sites also led to construction challenges, and in turn, cost and schedule growth. The area surrounding the Columbus land border crossing is prone to severe flooding, and major flood events have forced CBP to close the port several times a year, according to GSA officials. Officials also said flooding posed a potential risk of deteriorating port structures. After GSA spent $3.3 million to develop the original design, it spent an additional $7.4 million on a design refresh to incorporate flood protection and update CBP requirements to prepare for construction. In another example, GSA and the contractor discovered unstable soil conditions during the Tornillo- Guadalupe project that resulted in a two month delay and $1.3 million cost increase (about 3 percent of the project budget) to mitigate. GSA officials told us they may also experience challenges accessing labor, materials, and utilities for projects at remote land border crossings. For example, Alexandria Bay’s remote location created logistical challenges for transporting concrete to the site. Because the land border crossing is on an island and only accessible via toll bridge, the contractor determined it was more cost effective to construct a temporary concrete plant onsite. GSA officials also stated the labor market in Alexandria Bay is limited—due in part to its remoteness—and that labor costs were high because the contractor had to temporarily relocate its employees to the area. In another example, officials reported challenges with transporting construction materials to the Tornillo-Guadalupe site due to its remote location, contributing to 2.5 months in schedule growth. Natural features and dense population centers surrounding land border crossings can create challenges for contractors during construction. For example, the Alexandria Bay project—which will triple the crossing’s footprint when complete—required contractors to blast massive rock formations to create more room for facilities. GSA officials stated the rock removal entailed significant coordination with CBP because GSA required CBP to temporarily halt vehicle processing for safety reasons when GSA’s contractor was using dynamite. Officials also told us that snow removal is a challenge at Alexandria Bay because there are limited places to put plowed snow without impeding traffic and interrupting CBP operations. Corresponding international infrastructure. Inadequate or incomplete infrastructure in neighboring countries can lead to project delays. GSA officials explained that because land border crossings on both sides of the border need to connect, capital infrastructure projects in the United States are largely dependent on the readiness of Mexican or Canadian infrastructure. For example, GSA completed the Tornillo-Guadalupe project in October 2014, but delayed opening cargo processing facilities due to Mexico’s delays in completing its new commercial facilities and bridge system required for commercial traffic. As a result, CBP did not begin processing inbound cargo at Tornillo-Guadalupe until March 2016—16 months after it began processing passenger vehicles. Furthermore, after processing 277 trucks in 14 months, CBP suspended commercial inspection operations in May 2017, citing low traffic volumes. CBP officials said that commercial transporters were unwilling to use underdeveloped Mexican infrastructure in the region, leading to low commercial traffic volumes, and in turn, CBP’s decision to suspend commercial operations. Similarly, GSA had to delay work for 3 months on the Calexico West project because Mexico was behind schedule on its infrastructure project, according to GSA officials. To address this issue, GSA slowed work in that area and Mexico accelerated its schedule so that GSA and Mexico could complete their sections simultaneously. CBP is charged with facilitating billions of dollars in trade and travel at the nation’s border, while also preventing terrorists, criminals and other inadmissible individuals from entering the country. Given that CBP relies on infrastructure at land border crossings to fulfill its mission, maintaining the condition of the infrastructure is critical and can also be challenging, as many land border crossings were built more than 70 years ago. By developing and implementing a plan to ensure CBP executes its FCA program to assess the condition of infrastructure at CBP-owned land border crossings consistent with DHS policy, CBP would be able to maintain more complete and current information on its overall infrastructure needs. Also, given that GSA owns many of the land border crossings out of which CBP operates, sharing and using certain relevant information with each other—such as their respective facility assessments and repairs at land border crossings—could help both agencies improve the accuracy and completeness of their respective assessments of facility condition. Additionally, while CBP develops five-year plans to prioritize capital projects at land border crossings, establishing time frames for stakeholders who review and approve the plans would better position CBP to identify and address sources of delay and could improve its ability to complete a plan each year and include it in the budget submission to Congress. Furthermore, by also establishing a methodology for prioritizing its capital projects—including key required procedures and time frames—CBP could better ensure consistency in its approach from year to year. We are making a total of seven recommendations, including five to CBP and two to GSA: The CBP Commissioner, in conjunction with the DHS Office of the Chief Readiness Support Officer, should develop and implement a plan to ensure that CBP executes its FCA program by conducting FCAs at each CBP-owned land border crossing consistent with DHS Directive 119-02- 004. (Recommendation 1) The CBP Commissioner should share FCA reports with GSA and use facility condition information in GSA’s Building Assessment Tool to inform FCAs. (Recommendation 2) The GSA Administrator should share Building Assessment Tool reports with CBP and use facility condition information in CBP’s FCAs to inform its assessments through the Building Assessment Tool. (Recommendation 3) The GSA Administrator, in conjunction with CBP, should share with CBP information on GSA maintenance and repair work at GSA-owned land border crossings at the level of detail necessary to inform CBP’s data in TRIRIGA. (Recommendation 4) The CBP Commissioner should use information on maintenance and repair work conducted by GSA at GSA-owned land border crossings to update facility condition information in TRIRIGA on an ongoing basis. (Recommendation 5) The CBP Commissioner should establish review time frames for stakeholders involved in its Five-year Capital Investment Plan review and approval process. (Recommendation 6) The CBP Commissioner should establish and document a methodology for its annual land border crossing capital prioritization process that includes procedures and time frames for each step. (Recommendation 7) We provided a copy of this report to DHS and GSA for review and comment. DHS and GSA provided comments, which are reproduced in full in appendix III and appendix IV, respectively, and discussed below. DHS also provided technical comments, which we incorporated as appropriate. In its comments, DHS and GSA concurred with our seven recommendations and described actions planned to address them. With respect to our first recommendation that CBP develop and implement a plan to execute FCAs at CBP-owned land border crossings consistent with DHS Directive 119-02-004, DHS stated that CBP intends to develop a plan for completing FCAs at CBP-owned land border crossings consistent with the Directive. With regard to our second recommendation that CBP share FCA reports with GSA and use GSA’s Building Assessment Tool to inform CBP FCAs, DHS stated that CBP plans to provide FCA data to GSA. DHS also stated it has already begun receiving Building Assessment tool reports from GSA and will determine how to best use the information to inform CBP FCAs. With respect to our third recommendation that GSA share Building Assessment Tool reports with CBP and use CBP’s FCAs to inform its assessments, GSA stated it is developing a plan to share Building Assessment Tool information and use FCA information to inform its assessments. With regard to our fourth recommendation that GSA share information on its maintenance and repair work at GSA-owned land border crossings at the level of detail necessary to inform CBP’s data in TRIRIGA, GSA stated it will develop a plan to share information on GSA maintenance and repair work at the level of detail necessary to inform CBP’s data in TRIRIGA. With respect to our fifth recommendation that CBP use information on maintenance and repair work conducted by GSA at land border crossings and update facility condition information in TRIRIGA on an ongoing basis, DHS stated it has already begun receiving data from GSA on corrective maintenance work at land border crossings and that CBP will develop a plan for updating facility condition information in TRIRIGA using the data. With regard to our sixth recommendation that CBP establish time frames for stakeholders involved in its Five-year Capital Investment Plan review and approval process, DHS stated that CBP will establish a policy that outlines time frames for stakeholders involved in the review and approval process. DHS also concurred with our seventh recommendation that CBP establish and document a methodology for its annual land border crossing capital prioritization process that includes procedures and time frames for each step. Specifically, DHS stated that CBP will document the process and procedures, and provide time frames, for each step in the process. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Homeland Security, and the Administrator of the General Services Administration. In addition, this report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at 202-512-8777 or gamblerr@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of our report. GAO staff who made key contributions to this report are listed in appendix V. U.S. Customs and Border Protection (CBP) operates at 167 land border crossings along the northern and southern borders. Of the 167 land border crossings, CBP owns 40. The General Services Administration (GSA) fully owns 101, partially owns three, and leases 19. The National Park Service owns two and the U.S. Forest Service owns one. One land border crossing is privately owned. Further, CBP and GSA have assessed the condition of 95 of the 167 land border crossings along the northern and southern borders and calculated a facility condition index (0- 10% good, 10-20% fair, 20-30% poor, and 30-100% critical) and identified the total cost of infrastructure deficiencies at each crossing. Table 6 identifies land border crossings by name, state, ownership, year constructed, the year last renovated, facility condition index score, and the cost of known infrastructure deficiencies, according to CBP data, and is for informational purposes only. To provide an overview of recent land border crossing capital infrastructure projects, we developed a profile for each project that was active during fiscal years 2014 through 2018. These profiles contain background information on each crossing, along with basic travel, trade, and law enforcement data. Each profile also contains information on how infrastructure constraints affected U.S. Customs and Border Protection (CBP) operations, and how CBP and the General Services Administration (GSA) addressed those constraints through the capital project. Finally, the profiles include an assessment of project cost and schedule performance. We compiled information in the following project profiles from a variety of federal sources. We provide background information on each land border crossing in the “At A Glance” section of each profile. Some land ports of entry contain multiple land border crossings. While each project, and associated project performance data, refers to a single crossing unless otherwise noted, all throughput and trade data in this section is provided at the port-level. Law enforcement data are provided at the port-level, with the exception of arrests, which is provided at the crossing level. Daily CBP officers assigned to the port refers to the daily average for fiscal year 2017. We obtained condition, staffing, and law enforcement data from CBP’s Office of Facilities and Asset Management. Condition information includes the year GSA built each individual crossing and when GSA last modernized it through a major capital project. The number of arrests refers to arrests at land border crossings made by CBP Office of Field Operations officers, and does not include Border Patrol apprehensions. We analyzed data on imports, exports, and trade values from the Department of Transportation’s Bureau of Transportation Statistics (BTS) TransBorder Freight Data. These data are collected by CBP, processed and validated by the U.S. Census Bureau, and analyzed by BTS. Value of trade includes the combined totals of imports and exports for 2017. We also analyzed BTS’s Crossing/Entry Data to determine throughput for pedestrians, passenger vehicles, and cargo trucks. We analyzed project cost and schedule performance data from GSA’s Electronic Project Management system. These data included project cost and schedule baselines, and updated cost and schedule performance data as of January 2019. For multi-phase projects with only one phase included in our scope, phase costs may not equal total project costs when combined because certain project costs, such as site acquisition, cannot be attributed to an individual phase. Under schedule performance, original completion date refers to the project’s baseline substantial completion date. Revised completion date, if applicable, refers to a project’s updated substantial completion as revised by GSA to address project setbacks or delays. For ongoing projects, expected completion date is the date when GSA officials expect to complete the project. For completed projects, the actual completion date is the date the project reached substantial completion. We obtained information on crossing infrastructure constraints and project plans through interviews with GSA and CBP officials and project documents. These officials included GSA headquarters and project management officials, as well as CBP Office of Field Operations field office officials and local CBP officers. “Infrastructure Impacts on CBP Operations” refers to infrastructure constraints that existed prior to GSA’s recent capital project, while “Infrastructure Improvement Plans” describes each project’s scope and performance. To assess the reliability of project performance data from GSA’s Electronic Project Management system, we examined the data for obvious errors, and discussed the data with GSA project management officials. We determined the data to be sufficiently reliable for our purposes. To assess the reliability of trade data, we reviewed documentation and conducted interviews with officials from the U.S. Census Bureau, the original source of the validated data. Specifically, we analyzed procedures by agencies responsible for collecting the statistics, and reliability assessments by those agencies and outside sources. After reviewing data dictionaries and BTS’s quality control measures for analyzing the Census data, and conducting data quality checks, we determined that the trade data, originally collected by Census and released by BTS, are sufficiently reliable for providing contextual information about the value of trade. To assess the reliability of BTS crossing/entry data, we reviewed relevant documentation and procedures for analyzing the data, and met with BTS officials to discuss potential limitations. We determined the data to be sufficiently reliable for the purposes of reporting entry data for pedestrians, passenger vehicles, and trucks. Finally, we found the dates land border crossings were built and last modernized may be inconsistently recorded as provided by CBP’s Office of Facilities and Asset Management, but we provided accurate information in the project profiles. Built in 1974, Alexandria Bay is the seventh busiest commercial border crossing between the United States and Canada, as of 2017. In 2017, U.S. Customs and Border Protection (CBP) processed about 4,100 passengers, 1,600 passenger vehicles, 4 buses, and 600 trucks per day at Alexandria Bay. The majority of people crossing into the United States through Alexandria Bay in passenger vehicles are tourists traveling from the Ottawa, Kingston, Toronto, and Montreal regions, according to General Services Administration (GSA) project documentation. In 2017, GSA began phase I of a capital infrastructure project at Alexandria Bay. Prior to the project, the existing crossing lacked capacity to process growing traffic volumes which led to significant backups. These delays effectively brought cross-border traffic to a standstill, with traffic backups sometimes stretching three miles into Canada. The preprimary area did not provide adequate space for commercial traffic because the bridges connecting the United States and Canada were not designed to support prolonged periods of heavy traffic caused by backups. The commercial inspection facility provided enough space to unload a single commercial truck at a time and CBP’s commercial office space was housed in mobile trailers. GSA’s projected increases in traffic volume and updated CBP security procedures would necessitate an increase in the federal workforce beyond the existing crossing’s capacity. Phase I of this two-phase project will feature a new commercial building and warehouse, new commercial inspection lanes, and a new veterinary services building, among other enhancements. The completed two-phase project will more than double building space and triple the crossing’s footprint. Phase I will include five commercial inspection lanes—some of which will be equipped to process both commercial and passenger vehicles. After phase II, the crossing will feature five more passenger vehicle lanes and five more commercial lanes than the existing facility. An improved traffic pattern throughout the crossing will increase queuing space and allow safe and secure processing of traffic entering from Canada. Total funding for the entire project is $238 million, including $105 million for phase I, and construction began in August 2017. Phase I remains largely on budget and on schedule for completion in January 2020. GSA is expected to begin phase II in January 2020 and complete the project in July 2022. Calexico West, located in downtown Calexico, California, processes pedestrians and passenger vehicles. Inbound commercial and bus traffic are processed at the nearby Calexico East land border crossing, which opened in 1997 after Calexico West ceased commercial operations. Calexico West is the main crossing linking the California Imperial Valley agricultural industry to the Mexican state of Baja California and, according to U.S. Customs and Border Protection (CBP) officials, processes large volumes of farm workers during harvest season. CBP and General Services Administration (GSA) officials reported that the crossing’s facilities were undersized relative to current traffic volumes and obsolete in terms of inspection officer safety and border security. According to GSA, the crossing’s layout was also inefficient, resulting in bottlenecks and long lines for passenger vehicles and pedestrians. Passenger vehicle wait times regularly exceeded 1.5 hours during peak travel times, with outbound traffic often extending 1.5 miles into the United States. Facilities in the main building, including agricultural inspection laboratories, storerooms, holding cells, waiting areas, and officer work areas, were inadequate and undersized. CBP faced challenges finding space to install newer inspection equipment and technologies in the existing facilities, according to CBP officials. Finally, the passenger vehicle secondary inspection area was open to public view, enabling individuals to observe CBP inspections. CBP and GSA officials reported that phase I of this two-phase project reconfigured and expanded the existing crossing to reduce congestion and created five times more building space. Phase I delivered a new main building, 10 of 16 planned inbound vehicle inspection lanes, and five outbound vehicle inspection lanes. It also included a secondary vehicle inspection facility with canine kennels. The new preprimary inspection area is significantly larger, allowing CBP to actively manage traffic and reduce congestion. Further, the larger preprimary inspection area allows CBP officers to safely and effectively patrol this area with canine units, improving the effectiveness of CBP’s inspections. GSA completed the $94.6 million phase I construction in September 2018, about 6.4 percent above its cost baseline and six months later than planned. Delays associated with a corresponding infrastructure project in Mexico and CBP- requested modifications contributed to schedule growth. Phase II received partial funding in February 2019—two years after Phase II was scheduled to begin. Built in 1989, Columbus processes commercial traffic, passenger vehicles, and pedestrians. It is the only 24-hour pedestrian border crossing in New Mexico. Commercial traffic has steadily increased from about 5,700 trucks in 2007 to over 14,100 trucks in 2017. Historically, according to a GSA planning study, commercial traffic spiked in August and September during harvest season because produce is one of Columbus’s primary imports. Pedestrian traffic is higher during the harvest months due to farm workers and the winter when seasonal visitors cross the border. In 2017, the General Services Administration (GSA) began a capital infrastructure project at Columbus. U.S. Customs and Border Protection (CBP) and GSA officials reported that prior to this project, CBP operated from deteriorating facilities that were reaching the end of their useful lives. The volume of commercial trucks and travelers has increased significantly since the crossing opened and is expected to continue to grow. Over the years, GSA added additional facilities that, in turn, impeded traffic flow, caused backups, and threatened officer safety. Prior to the project, CBP could inspect two trucks at a time at the cargo loading dock. CBP also lacked the space to completely offload cargo, limiting inspection effectiveness. The site experienced significant flooding during major rain events that further limited inspection space and further deteriorated infrastructure, according to officials. CBP and GSA officials reported that the project involves a complete demolition of existing facilities and more than triples the crossing’s footprint with donated land. New facilities include a separate commercial processing facility and an expanded main building with new Non-Intrusive Inspection technologies, a hazardous material inspection area, canine kennel, narcotics vault, and site drainage improvements to address flooding. Processing capacity will expand from one pedestrian lane to four, from two passenger vehicle lanes to three, and from zero commercial lanes to one, and will increase usable commercial dock spaces from two to 12. GSA spent $3.3 million on design from 2007 to 2009. It spent another $7.4 million in 2014 on a redesign that incorporated flood protection and new CBP standards. GSA expects to complete the $87 million project in April 2019--about 3 percent above its cost baseline and two months later than planned due to CBP requested changes. Built in 1965, Derby Line I-91 is the busiest land border crossing in Vermont. The crossing processes passenger vehicles, buses, cargo, and pedestrians. There are two border crossings in Derby Line, at I-91 and about a half mile west on Route 5. The I-91 crossing is a large facility located on a major highway whereas the Route 5 crossing is relatively small, located on the village’s Main Street. U.S. Customs and Border Protection (CBP) processed about 3,000 passengers per day in 2017, along with about 1,500 passenger vehicles and 300 trucks. In 2016, General Services Administration (GSA) began a capital infrastructure project at the Derby Line I-91 crossing. CBP and GSA officials reported that CBP substantially increased staffing at the crossing over the years, resulting in overcrowded conditions. The administrative building lacked sufficient office and storage space, had limited secure areas to perform interviews and searches, and lacked a secure holding area. Due to insufficient space and outdated IT systems, the crossing could not accommodate newer inspection technologies. The commercial secondary inspection area was too small to completely offload cargo trucks for inspection and the vehicle lift was inoperative. The facility also lacked sufficient space to inspect buses and luggage. The crossing had poor lighting and inadequate perimeter security, and lacked measures to prevent travelers from exiting the crossing without authorization. Finally, poorly designed inbound primary inspection lanes made it difficult for commercial trucks to navigate through the crossing, at times resulting in long traffic delays, according to officials. CBP and GSA officials reported that the capital project will reduce cross- border travel times and improve the traveler experience. The project expanded the crossing’s footprint from 0.25 to 23 acres and improved traffic flow around the crossing, while adding measures to prevent travelers from exiting the crossing without authorization. Site improvements included new lighting, fire protection, and storm water management systems, among others. The project included a main building, and a commercial secondary inspection facility for CBP to offload and inspect trucks. GSA completed construction in November 2018 about 5 months later than originally planned and 11 percent above its cost baseline. Cost and schedule growth were primarily due to CBP-requested changes and contractor performance. The Laredo Land Port of Entry is made up of four land border crossings, each with its own bridge. In January 2019, the General Services Administration (GSA) completed a capital project at two of these crossings —the Convent Street Bridge (Laredo 1), and the Lincoln-Juarez Bridge (Laredo 2). Laredo 1 and 2 are located in downtown Laredo and process passenger vehicle and pedestrian traffic. The other two crossings–the Colombia Solidarity Bridge (Laredo 3) and the World Trade Bridge (Laredo 4)—primarily process cargo. The city of Laredo owns and maintains these bridges, while GSA owns and maintains the crossings and all property inside the crossing facilities. U.S. Customs and Border Protection (CBP) and GSA officials reported that volume at Laredo 1 and 2 have increased significantly in recent decades. Prior to the capital project, facilities at Laredo 1 did not effectively separate vehicles, bicycles, and pedestrians within the crossing, creating congestion, safety concerns, and pedestrian queues that could extend across the bridge into Mexico. GSA is unable to make extensive alterations or expand Laredo 1 because it is a U.S. Historic Site and is surrounded by businesses and homes. Laredo 2 was unable to efficiently process current traffic volumes. For example, GSA originally designed Laredo 2 to process up to 10 buses per day. However in 2017, Laredo 2 processed approximately 110 buses and 2,000 bus passengers each day. To accommodate these volumes, CBP converted Laredo 2’s passenger vehicle secondary facility to inspect buses and moved secondary vehicle inspections to a temporary facility. CBP and GSA officials reported that the capital project focused on improving efficiency, safety, and security while expanding pedestrian capacity at Laredo 1 and bus capacity at Laredo 2. GSA combined improvements at the two crossings into one estimated $96.6 million project ($33 million for Laredo I and $63.6 million for Laredo II) to save on labor and material costs. At Laredo 1, GSA replaced the main building, expanded pedestrian lanes from eight to 14, and reconfigured vehicle lanes to integrate newer inspection technologies. At Laredo 2, GSA enlarged the main building, built a facility to process passenger vehicle and bus passengers, and expanded bus processing capacity from two to eight lanes. GSA scoped out a footbridge and scaled back aesthetic finishes to control costs. GSA completed Laredo 1 in April 2018 and Laredo 2 in January 2019—about 3 months later than originally planned and 6 percent above cost baseline. Nogales West-Mariposa is one of three land border crossings in Nogales, Arizona and is one of the busiest land border crossings in the United States. It serves as the southern border’s main entry and distribution point for produce entering from Mexico. Nogales West processes about half of the agricultural commodities entering the United States from Mexico and has facilities for pedestrian, passenger vehicle, and commercial traffic. The other crossings in Nogales are the DeConcini (pedestrians and passenger vehicles) and Morley Gate crossings (pedestrians). In 2010, the General Services Administration (GSA) initiated a $180 million capital infrastructure project. U.S. Customs and Border Protection (CBP) and GSA officials reported that facilities and technologies at the original Nogales West-Mariposa land border crossing were outdated. The crossing’s layout was also inefficient resulting in bottlenecks, congestion, and commercial traffic backups that extended for miles into Mexico. GSA subsequently added new facilities to accommodate bus and pedestrian inspections, but did so in a way that further constrained space, impairing traffic movement within the crossing, according to officials. Wait times of up to eight hours resulted in spoilage or reduced shelf-life of perishable goods, resulting in financial losses for businesses. The original crossing also lacked adequate space and CBP repurposed some facilities to accommodate operational needs, including storing evidence in holding areas. CBP and GSA officials reported that the capital project focused on improving operational efficiencies, processing capacity, and security and safety of officers and the traveling public. The project entailed demolishing all existing structures and replacing them with new facilities, including new inspection areas, a main building, and other support facilities. GSA added 13 acres to the crossing’s footprint and expanded processing capacity from three to eight cargo primary lanes, one to five commercial exit lanes, 23 to 56 cargo docks (including six for refrigerated inspection), four to 12 passenger vehicle primary lanes, and eight to 24 passenger vehicle secondary inspection spaces. GSA completed the $180 million project in August 2014 more than 5 months later than originally planned and 5.5 percent above its cost baseline. This was due to CBP-requested changes, design deficiencies, and high site utility costs, among other reasons, according to officials. The project resulted in reduced wait times, but led to higher than expected operational and maintenance expenses. Built in 1932, San Ysidro is the busiest land border crossing in the western hemisphere, with 24/7 operations. San Ysidro processes pedestrians, passenger vehicles, and buses. The crossing does not have any commercial facilities for screening cargo. In 2017, U.S. Customs and Border Protection (CBP) processed about 65,000 northbound vehicle passengers and 23,000 northbound pedestrians each day at San Ysidro. The General Services Administration (GSA) began construction on a three-phase, $741 million project in 2011, with plans to complete all three phases by late 2019. CBP and GSA officials reported that queues and wait times at San Ysidro steadily increased over the years and that existing facilities could no longer accommodate the traffic volume. CBP also reported that outdated infrastructure in the pedestrian primary inspection area created officer safety concerns and that renovations were necessary to provide a safe and secure work environment for CBP staff. For example, CBP officials stated that the design and location of the existing pedestrian primary inspection booths obstructed officers’ view of pedestrians as they entered the primary inspection area. CBP and GSA officials reported that to better accommodate traffic growth and CBP’s requirements, GSA’s capital project is expanding and reconfiguring the crossing. The project entails demolishing existing structures and constructing new primary and secondary passenger vehicle inspection areas, a new main building, and other support structures. The project also includes two pedestrian processing areas—on the east and west sides of the crossing—that connect with transportation centers in Mexico and the United States. Once complete, the crossing will have 34 passenger vehicle lanes with 62 booths, including stacked booths that allow CBP officers to simultaneously inspect two vehicles in most lanes. The crossing will also add a dedicated bus lane and a total of 36 pedestrian primary inspection lanes across its two pedestrian facilities. GSA is building the $741 million project in three stand-alone phases, with expected completion in November 2019. Tornillo-Guadalupe (also known as the Marcelino Serna land border crossing) opened in 2015. Tornillo-Guadalupe replaced the Fabens land border crossing, which dated back to 1938. U.S. Customs and Border Protection (CBP) currently processes passenger vehicles and pedestrians at Tornillo-Guadalupe. Although Tornillo-Guadeloupe has commercial processing facilities, CBP ceased using these facilities in 2017 due to low volumes of commercial traffic. CBP and General Services Administration (GSA) officials reported that the original Fabens land border crossing was unable to process high traffic volumes and that the existing bridge connecting the United States and Mexico was no longer structurally sound enough to support commercial crossings. CBP ceased all commercial operations at Fabens in 2001, limiting CBP to pedestrian and passenger vehicle traffic processing. The number of CBP personnel at the crossing exceeded facility capacity and the limited space hindered CBP’s ability to process, interview, isolate, and detain travelers, according to CBP officials. Further, the existing septic system was not designed for the number of employees at the facility and the original water system was insufficient. CBP had to haul water on site to operate its facilities and provide bottled water for its employees and the public to drink, according to officials. CBP and GSA officials reported that the recent capital project delivered new passenger vehicle and pedestrian inspection facilities along with a new main building. The project also included a dedicated bus inspection area and a parking lot for seized vehicles. Commercial facilities included a new bridge and commercial building, 10 covered secondary inspection docks, two primary inspection lanes with a canopy, a hazardous materials containment area, agriculture lab, and seized narcotics storage. The project also added an additional 109 acres of donated farmland to the original crossing’s 6 acre footprint. GSA completed the $73.5 million construction project in October 2014, about 15 months later than planned and 19 percent above its cost baseline. Unstable soil conditions and contractor performance issues contributed to cost and schedule growth, according to GSA. Delays associated with infrastructure in Mexico delayed the start of cargo processing by 16 months. Despite investing in new commercial processing facilities at the crossing, CBP suspended cargo processing in May 2017 after 14 months, citing low traffic volumes due to underdeveloped infrastructure in Mexico. In addition to the contact named above, Michael Armes (Assistant Director) Kirk Kiester (Assistant Director), Bruce Crise (Analyst in Charge), Lilia Chaidez, Michele Fejfar, Eric Hauswirth, Susan Hsu, Daniel Kuhn, Jeremy Manion, Mara McMillen, Marc Meyer, and Sasan J. “Jon” Najmi made significant contributions to this report.", "summary": "CBP and GSA own, lease, or manage all of the nation's 167 land border crossings. CBP facilitates trade and travel at these crossings and has identified significant capital investment needs at these facilities. GAO was asked to review land border crossing infrastructure. This report examines (1) infrastructure constraints CBP faces and the extent CBP and GSA have information on infrastructure condition, (2) the extent CBP prioritizes capital projects and (3) the extent recent GSA capital projects met cost, schedule, and scope goals and challenges CBP and GSA reported. GAO analyzed land border crossing data and documentation, including CBP and GSA facility assessments, CBP capital investment plans for fiscal years 2014 through 2018, and data for GSA capital infrastructure projects active during those years. GAO also interviewed officials from CBP field offices that oversee all crossings about infrastructure constraints and visited 16 crossings selected based on high traffic volume and border crossings CBP has prioritized for infrastructure improvement. The Department of Homeland Security's (DHS) U.S. Customs and Border Protection (CBP) reported infrastructure constraints at land border crossings including limited inspection capacity, technology challenges, and security limitations. However, CBP does not have complete information on infrastructure conditions at all land border crossings. Specifically, CBP assessed facility conditions at four of the 40 land border crossings it owns from 2016 through 2018. Further, CBP has not developed a plan to ensure it conducts such assessments, consistent with DHS policy which calls for them every three years. Developing and implementing a plan to ensure CBP executes its facility condition assessment program would enable CBP to collect more complete and current infrastructure information. In addition, while CBP and the General Services Administration (GSA) both assess facility conditions at 101 GSA-owned land border crossings, they do not consistently share or use each other's information. Doing so could enable CBP and GSA to improve the accuracy and completeness of their respective assessments. CBP prioritizes land border crossing capital projects in a five-year plan, which by statute is to be submitted with DHS's annual budget request to Congress. In fiscal years 2014 through 2018, CBP submitted two plans on time, submitted two plans more than 100 days after submission of the budget request, and did not submit a plan in one year due to delays in the plan's review and approval process. By establishing timeframes for the review process, CBP would be better positioned to identify and address sources of delay in the review process, and improve its ability to meet statutory reporting requirements by including its five-year plan with its annual budget submission to Congress. The 10 completed or ongoing GSA land border crossing capital projects in fiscal years 2014 through 2018 generally experienced schedule growth ranging from 0 to 59 percent, but stayed within a 10 percent cost contingency allowance. Circumstances contributing to increased project costs or schedule growth include funding lags between project design and construction, and CBP-requested changes during construction to meet evolving mission needs, according to GSA and CBP officials. GAO is making seven recommendations, including that CBP develop a plan to ensure it conducts facility condition assessments consistent with DHS policy; that CBP and GSA share and use each other's information on facility conditions at land border crossings; and that CBP establish review timeframes for its capital investment plan.", "document_type": "gao"}
{"report": "Federal agencies have many programs that provide services and benefits to tribes and their members. For example, the Department of the Interior’s (Interior) Bureau of Indian Affairs (BIA) within the Office of the Assistant Secretary-Indian Affairs (Indian Affairs) administers programs in natural resources management, Indian child welfare, and economic development—among other responsibilities. One key BIA responsibility is to facilitate tribes’ development of energy resources on and beneath tribal lands by reviewing and approving leases, permits, and other documents required when the lands with Indian energy resources are held in trust or restricted status. The Bureau of Indian Education (BIE), also within Indian Affairs at Interior, administers education programs to approximately 41,000 students on or near Indian reservations at 185 schools around the country. The Indian Health Service (IHS) within the Department of Health and Human Services is charged with providing health care to approximately 2.6 million Indians through more than 600 IHS or tribally operated facilities as of 2019. When services are not available at these facilities, IHS may pay for services provided through external providers. In addition, as part of its disease prevention efforts, IHS provides technical and financial assistance to Indian tribes for the cooperative development and construction of drinking water and wastewater systems and support facilities. These and other federal programs may also be administered by tribal governments under a self-determination contract or self-governance compact under the Indian Self-Determination and Education Assistance Act of 1975, as amended. BIA and IHS are responsible for administering self-determination contracts that allow for tribal administration of specific government programs, including negotiating and approving each contract and its associated annual funding agreement and disbursing funds to the tribes. Each federally recognized tribe voluntarily decides whether, and to what extent, to pursue the administration of federal programs. According to a 2017 law journal article, by that year, nearly all tribes had used a self- determination contract or self-governance compact to take over the administration for one or more federal programs. In February 2017 we added federal management of programs that serve tribes and their members to our high-risk list of federal areas that are most vulnerable to fraud, waste, abuse, or mismanagement or that are in need of transformation to address economy, efficiency, or effectiveness challenges. In particular, we found numerous challenges facing BIE and BIA and IHS in administering education and health care services, that put the health and safety of American Indians served by these programs at risk. In addition, we reported that BIA mismanages Indian energy resources held in trust and thereby limits opportunities for tribes and their members to use those resources to create economic benefits and improve the well-being of their communities. Our recommendations identified in the high-risk area do not reflect the performance of programs administered by tribes nor are they directed at any tribally operated programs and activities. In our March 2019 high-risk update, we reported that the three agencies demonstrated progress to partially meet all five criteria for addressing high-risk issues: leadership commitment, capacity, action plan, monitoring, and demonstrated progress. We continue to monitor and report on progress made by the agencies in addressing issues in these three program areas. We have previously reported that constraints in federal agency capacity, funding and budget uncertainty limit effective delivery of some federal programs for tribes and their members managed by Indian Affairs, BIA, BIE, IHS, and in other agencies’ tribal consultation activities as shown in the following examples: High staff vacancies. In November 2016, we found BIA had high vacancy rates at some agency offices and that the agency had not conducted key workforce planning activities to ensure its workforce resources are appropriately deployed. We recommended that BIA establish a documented process for assessing its workforce composition at agency offices taking into account BIA’s mission, goals, and tribal priorities. In response to our recommendation, BIA has taken initial steps to assess its workforce composition; however more work is needed from BIA to establish a process to regularly assess its workforce composition and ensure it meets BIA and tribes’ needs. In February 2017 when we added improving federal management of programs that serve tribes and their members to our high-risk list, we found that high vacancies or declining staff levels across all three designated high-risk areas—education, health care, and Indian energy resources programs. For example, we reported that IHS had over 1,550 vacancies for various health care positions nationwide in 2016, and IHS officials said that the agency’s insufficient workforce was the biggest impediment to providing timely primary care. IHS has made some progress in demonstrating it has the capacity necessary to address the program risks we identified in our reports. For example, among other actions, in January 2019, IHS established an Office of Quality which includes divisions for Enterprise Risk Management and Internal Controls, Quality Assurance, Innovation and Improvement, and Patient Safety and Clinical Risk Management. As of August 2019, the Office of Quality had filled 14 positions. However, there are still key positions in the agency not yet permanently filled, including the Director of the Office of Finance and Accounting and the Deputy Director for Management Operations. In our August 2018 report, we also found that IHS’s overall vacancy rate for clinical care providers was 25 percent. Additionally, in our March 2019 high risk update and testimony, we reported that about 50 percent of all BIE positions had not been filled, according to recent BIE documentation, for a variety of reasons, including difficulty recruiting qualified individuals. Insufficient staff skills or knowledge. We have also identified concerns about existing staff having the right skills and expertise to adequately perform job duties for effective implementation of Indian energy development programs. For instance, in November 2016, we found that BIA had not completed key workforce planning activities, such as an assessment of work skills gaps, that contributed to BIA’s inability to effectively support energy development. We recommended that BIA incorporate effective workforce planning standards by assessing critical skills and competencies needed to fulfill BIA’s responsibilities related to energy development and by identifying potential gaps. Interior agreed with this recommendation and in fiscal year 2019, BIA began identifying the skills and competencies necessary for select Indian energy-related occupations. BIA officials told us that, once complete, agency officials will be able to use the catalog of necessary skills and competencies to identify training needs for existing staff. Additionally, in our March 2019 report on tribal consultation, 47 of 100 tribes that provided comments to federal agencies in 2016 identified insufficient agency officials’ knowledge or training on tribal consultation as a key factor that hinders effective consultation. Several tribal officials we interviewed shared similar concerns, and officials from 9 of 21 agencies we spoke with (43 percent) identified staff knowledge or training as a factor that hinders effective consultation. Inadequate funding. We have previously reported on agency and tribal perspectives on the adequacy of funding and how it impacts federal programs and also examined spending levels of some programs. In May 2018, we reported that federal agencies provided about $370 million to develop, construct, or repair tribal drinking water and wastewater infrastructure projects to address tribes’ needs in fiscal year 2016. This amount is about 11 percent of the more than $3 billion in total existing tribal drinking water and wastewater infrastructure needs that IHS had identified that same year. Further, in January 2019, we found that funding shortfalls— estimated at 60 percent for one BIA program in a 2013 report to Congress— may limit tribal options for administering federal programs using self- determination contracts or self-governance compacts. Many tribal stakeholders told us that they supplement federal funding when there are funding shortfalls. As we have previously reported, when tribes financially supplement the federal program they take over, it diverts funds away from other economic development opportunities and other government functions and services they provide to their communities and citizens. In our March 2019 report on tribal consultation, according to tribal comments we reviewed and interviews with tribal officials we found tribes’ ability to participate in consultations is limited by availability of funding from the tribe, federal agencies, or other sources. Tribes and agencies both identified insufficient resources, including funding to support tribes’ participation in consultation activities, as a key factor hindering effective consultation. Effects of budget uncertainty. Budget uncertainty arises during continuing resolutions—temporary funding periods during which the federal government has not passed a budget—and during government shutdowns. Failure to enact annual appropriations for federal tribal programs in a timely manner may exacerbate the problem of limited resources. For example, in our September 2018 report examining advance appropriations authority for IHS, IHS officials and tribal representatives described several effects of budget uncertainty on their health care programs and operations. Among other things, we reported that effects of budget uncertainty include (1) exacerbated challenges related to recruitment and retention of staff, and (2) additional administrative burden and costs for both IHS and tribes involved in calculating revised allocations and modification of hundreds of tribal contracts each time a new continuing resolution is enacted. IHS officials and tribal representatives said that advance appropriation authority could mitigate the effects of this uncertainty because IHS could use this authority to ensure continuity of health care services during lapses in annual appropriations. In our prior work, we have found a range of management weaknesses related to internal controls at Indian Affairs, BIA, BIE, and IHS that hinder effective delivery of some federal programs for tribes as shown in the following examples: Oversight weaknesses. In March 2016, we found that weaknesses in Indian Affairs oversight led to safety and health deficiencies at BIE school facilities that endangered students. We recommended that Indian Affairs ensure that all BIE schools are annually inspected for safety and health, as required by its policy, and that inspection information is complete and accurate. Indian Affairs has taken steps toward implementing our recommendations. For example, in April 2019 Indian Affairs provided documentation that it had assessed the quality of two fiscal year 2018 BIE safety inspection reports. However, Indian Affairs has not provided us with documentation that it has assessed the quality of BIE safety inspection reports for fiscal year 2019—the first year BIE was responsible for inspecting all of its schools. We believe it is important that the agency demonstrate that BIE is capable of inspecting all schools for safety in fiscal year 2019 and that they produce inspection reports for schools that are complete and accurate. As of November 2019, we have not received further updates from the agency on this recommendation’s status. Additionally, in March 2016, we reported on weaknesses in IHS’s oversight of the timeliness of patient care that led to long wait times at IHS facilities. We found that IHS had not set any agency-wide standards for patient wait times at IHS federally-operated facilities. We recommended that IHS (1) develop and communicate specific agency-wide standards for patient wait times in federally- operated facilities, and (2) monitor patient wait times and ensure corrective actions are taken when standards are not met. IHS agreed with our recommendations and implemented the first recommendation by publishing patient wait time standards as part of its Indian Health Manual website in August 2017. As of March 2019, IHS officials said that the agency was working to implement the second recommendation by developing system-wide monitoring capacity. We will continue to review IHS’s progress. Management weaknesses. In June 2015, we found shortcomings in BIA’s management of permits, and other approvals for energy development have led to lengthy review times and negatively impacted energy development on tribal lands. These lengthy review times have increased energy development costs for tribes, delayed projects, and led to lost revenue, among other impacts. For example, according to a tribal official, BIA took as long as 8 years to review some of its energy-related documents. In the meantime, the tribe estimates it lost $95 million in revenue that it could have earned from tribal permitting fees, oil and gas severance taxes, and royalties. We recommended that BIA develop a documented process to track its review and response times and enhance its data collection efforts. As of November 2019, the agency had taken initial steps toward implementing the recommendation by developing system enhancements to capture key dates during the review and approval process for energy-related documents. However, BIA needs to collect data from its system, develop time frames, and monitor agency performance to fully address these recommendations. In our January 2019 work on tribal self-governance, we reported that Interior’s process does not ensure that funds associated with self-determination contracts and self-governance compacts are disbursed in a timely manner, according to tribal stakeholders. These funding delays can therefore be a factor that hinders tribal use of these agreements. When funds are not disbursed in a timely manner, a tribal stakeholder told us that tribes may have to use funds from their general revenue accounts or seek other sources to cover federal program expenses. According to several tribal stakeholders, when a tribe has to use its own funds for the administration of federal programs—even temporarily—it can adversely affect the tribe in various ways. To help ensure that funds are disbursed in a timely manner, we recommended that Interior establish a process to track and monitor the disbursement of funds associated with self-determination contracts and self-governance compacts. Interior agreed with this recommendation, and as of November 2019, we are following up on its status. Weaknesses in planning. In May 2017, we found that Indian Affairs did not have a comprehensive capital asset plan to guide funding for construction projects to maintain, repair, or replace infrastructure at its 185 BIE schools. Specifically, although Indian Affairs had determined which 10 schools it planned to replace next, it did not have a long-term capital asset plan for the remaining 175 BIE schools. Many of the 175 schools were in poor condition and had safety hazards. We recommended Indian Affairs develop a comprehensive long-term capital asset plan that includes a prioritized list of projects with the greatest need of funding. Indian Affairs agreed with the recommendation. As of October 2018, Indian Affairs provided a list of deferred maintenance projects for 2018 and documentation of processes for prioritizing such projects, among other things, but as of November 2019 had not yet provided documentation that it had completed a comprehensive long-term capital asset plan. In conclusion, the resource constraints and management weaknesses in federal programs that serve tribes limit federal agencies’ effective delivery of programs to Native Americans. In many cases, we have made recommendations to agencies to take steps to address identified issues. While agencies have made some progress addressing recommendations to improve tribal programs identified in our high-risk and other areas, continued work to address these and other issues is needed. Sustained congressional attention to these issues and the relevant factors contributing to the disparities identified in the U.S. Civil Rights Commission’s report will help the federal government makes progress in addressing the needs of Native Americans. Chairman Gallego, Ranking Member Cook and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. For further information regarding this testimony, please contact Anna Maria Ortiz at (202) 512-3841 or ortiza@gao.gov. If you or your staff have any questions about health care issues in this testimony or the related reports, please contact Jessica Farb at (202) 512-7114 or farbj@gao.gov. For questions about education, please contact Melissa Emrey-Arras at (617) 788-0534 or emreyarrasm@gao.gov. For questions about energy resource development, please contact Frank Rusco at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this statement include Lisa Van Arsdale (Assistant Director), Swati Thomas (Analyst-in-Charge), Edward Bodine, Kelly DeMots, Summer Lingard-Smith, Elizabeth Sirois, Jeanette Soares, Kiki Theodoropoulos and Leigh White. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "As Congress affirmed in the Indian Trust Asset Reform Act, the United States has undertaken a unique trust responsibility to protect and support Indian tribes and Indians. Thus, federal agencies have many programs that provide services to tribes. However, in 2018, the U.S. Commission on Civil Rights found that, due to a variety of reasons—including historical discriminatory policies, insufficient resources, and inefficient federal program delivery—Native Americans continue to rank near the bottom of all Americans in terms of health, education, and employment. In February 2017 GAO designated federal management of programs that serve tribes in education, health care and energy as high risk. This designation is neither reflective of the performance of programs administered by tribes nor directed at tribal activities. This testimony, which is based on reports GAO issued from June 2015 through March 2019 primarily related to education, health care, and energy development, provides examples of (1) capacity and funding constraints and budget uncertainty and (2) management weaknesses that limit the effective delivery of federal programs for tribes and their members. GAO previously reported that constraints in federal agency capacity and funding and budget uncertainty limit effective delivery of some federal programs and activities serving tribes. Key federal agencies serving tribes include the Department of the Interior's Bureau of Indian Affairs (BIA) and Bureau of Indian Education (BIE), and the Department of Health and Human Services' Indian Health Service (IHS). For example: High staff vacancies and insufficient staff capacity. In February 2017, GAO reported that IHS had over 1,550 vacancies for health care positions in 2016, and IHS officials said that the agency's insufficient workforce was the biggest impediment to providing timely primary care. In addition, GAO's March 2019 high-risk update reported that about 50 percent of all BIE positions had not been filled, according to recent BIE documentation. Inadequate funding. In January 2019, GAO reported on agency and tribal perspectives on the adequacy of funding and how it impacts federal programs. GAO found that inadequate program funding to meet tribal needs (e.g., BIA estimated a funding shortfall at 60 percent for one program in a 2013 report to Congress) may limit tribal options for administering federal programs using self-determination contracts or self-governance compacts. Many tribal stakeholders told GAO that they supplement federal funding when there are shortfalls, which diverts funding from economic development and services provided to their communities. Effects of budget uncertainty. Budget uncertainty arises during continuing resolutions—temporary funding periods during which the federal government has not passed a budget—and during government shutdowns. In a September 2018 GAO report, IHS officials and tribal representatives described the effects of budget uncertainty on their health care programs and operations. GAO reported that these effects include recruitment and retention of staff challenges and additional administrative burden and cost for both tribes and IHS. In GAO's prior reports and March 2019 high-risk update, GAO found that management weaknesses at some federal agencies limit the effective delivery of some federal programs serving tribes. For example: Oversight weaknesses. In March 2016, GAO found weaknesses in IHS's oversight of timeliness of patient care leading to long wait times at IHS facilities. GAO recommended that IHS develop standards for patient wait times, monitor these wait times, and take corrective action as needed. IHS has established wait times standards and is developing monitoring capacity. Management weaknesses . In June 2015, GAO found shortcomings in BIA's management of energy development permitting processes that led to lengthy reviews and negatively impacted energy development on tribal lands. Among other things, GAO recommended that BIA develop a process to track its review and response times. BIA has taken initial steps to develop system enhancements to capture key dates during the review and approval process for energy development documents. GAO has made more than 50 recommendations related to its high- risk area and more than 40 recommendations for tribal water infrastructure, tribal self-governance and tribal consultation of which 60 recommendations are open. Sustained focus by the respective agencies and Congress on these and other issues are essential to continued progress.", "document_type": "gao"}
{"report": "In March 2017, the President issued an executive order requiring comprehensive reorganization plans for executive branch agencies (see fig. 1). In April 2017, OMB provided guidance to federal agencies for developing their respective reform plans. The government-wide reform plan was to have been based on the agency reform plans, OMB-coordinated crosscutting proposals, and public input. According to OMB’s M-17-22 guidance, OMB, in coordination with the President’s Management Council, was to establish a way to track the progress of the reforms. OMB’s guidance also stated that it would track progress of the reforms by leveraging the federal performance planning and reporting framework originally put into place by the Government Performance Results Act of 1993 (GPRA), and significantly enhanced by the GPRA Modernization Act of 2010 (GPRAMA), through the use of cross-agency priority (CAP) goals, agency priority goals, and Performance.gov. In March 2018, OMB released the President’s Management Agenda (PMA), which provided information on the preliminary status of government reorganization efforts and is connected with these reform efforts. The PMA also identified a set of CAP goals, required under GPRAMA, to target those areas where multiple agencies must collaborate to effect change and report progress in a manner the public can easily track. The PMA gave OPM a key role in fulfilling the administration’s human capital-related goals. Specifically, OPM, along with OMB and the Department of Defense (DOD), were tasked to “align and strategically manage the workforce to efficiently and effectively achieve the federal government’s mission.” The administration is planning to transfer OPM’s background investigations to DOD, policy and workforce strategy functions to the EOP, and all remaining functions to GSA (see fig. 2). These remaining functions include human resource solutions, information technology systems, healthcare and insurance, retirement services, merit system accountability and compliance, and IG functions. The President’s fiscal year 2020 budget proposal, which was issued in March 2019, states that the administration is planning to complete the reorganization of OPM by the end of fiscal year 2020. As such, the budget proposal provided no funds for OPM for fiscal year 2020. According to that budget proposal, “the Administration has been developing plans to execute transfers of OPM functions to GSA and the DOD using a combination of existing legal authority and legislation” since June 2018. The budget proposal also requested $50 million to transfer certain OPM functions to GSA, including an additional $1 million to cover costs associated with merging the OPM IG with the GSA IG. On May 16, 2019, the administration proposed new legislation requesting authority to fully implement its reorganization proposal. As we previously reported, a critical first step in the reform and reorganization process is to define the benefits of the merger, and describe how the future will be both different from and better than the past. As of May 17, 2019, OMB, OPM, and GSA had not fully established outcome-oriented goals and performance measures for, or assessed the costs and benefits of, the administration’s proposal to reorganize OPM (see fig. 3). Specifically, GSA provided one document, a draft Qualitative Business Case and Value Proposition for the GSA/HRS Merger (October 2018), which includes some preliminary goals and measures, such as to improve customer satisfaction. However, this document focuses only on the goals and measures related to the transfer of human resources solutions from OPM to GSA, rather than on the entire reform proposal. In addition, that document explicitly states that it is not a cost-benefit analysis, and OMB staff have told us that they have not conducted a cost-benefit analysis of the reform. In our prior work on organizational mergers and transformations, we have found that establishing a coherent mission and integrated strategic goals to guide the transformation involves adopting leading practices for results-oriented strategic planning and reporting. We have previously reported that organizational transformations should be led by a dedicated team of high-performing leaders within the agency, and GSA has provided some evidence of this leadership focus and attention, but OMB, OPM, and GSA have only partially addressed this key practice (see fig. 4). According to GSA officials and documents we reviewed, the agency designated a member of its Senior Executive Service as the leader of the reorganization within GSA, and has established a Project Management Office with dedicated staff and resources which will take on the responsibility of supporting the transfer of OPM’s functions to GSA. Also, GSA officials told us that OMB leads the reform by, for example, leading meetings under the Six Sigma management approach to manage progress on implementing the reorganization. However, as of May 17, 2019, OMB did not provide documents we requested about the role of these management meetings for the reorganization, and OPM did not provide relevant information or documents. Our past work has also found that leadership should articulate a succinct and compelling reason for the reform, as this helps build morale and commitment to the organizational changes. OMB provided the case for change in several public documents, such as the government-wide reform plan, which primarily state that the administration’s reason for moving OPM’s functions to GSA and the EOP is that these changes would create greater efficiencies and elevate the importance of human resources policy. However, sharing the case for change is only one key factor in successful reforms and reorganizations. As we stated above, illustrating what success looks like is also important, and involves articulating the specific goals and costs and benefits of the reform. Our prior work has shown that it is important for agencies to directly and continuously involve their employees, Congress, and other key stakeholders in the development of any major reforms. OMB and GSA have taken some actions to involve and communicate with Congress, employees, and other key stakeholders, but these initiatives lack documentation (see fig. 5). For example, GSA officials told us that they have met with members of Congress, conducted town hall meetings in which they provided information to and answered questions from GSA officials, and established an email inbox for communication between GSA leaders and employees on the reform. However, as of May 17, 2019, GSA officials had not provided us with documentation of their meetings and communications with employees, and neither OMB nor OPM had provided relevant documents on employee outreach and inclusion. We have previously reported that organizational transformations must be carefully and closely managed by developing an implementation plan with key milestones and deliverables to track and communicate implementation progress, among other actions. However, as of May 17, 2019, OMB, OPM, and GSA had not developed an implementation plan or publicly reported on key milestones (see fig. 6). This is the case despite the fact that the President’s fiscal year 2020 budget states that the reform is underway in fiscal year 2019, and that all remaining portions of the reform would be completed in fiscal year 2020 through legislation. Moreover, these agencies have not ensured transparency of their efforts by publicly reporting on implementation progress. Our prior work has shown that successful reorganizations seek to implement best practices in the systems and processes wherever they may be found, and guard against automatically adopting the approaches used by the largest or acquiring component. The risk is that the new organization may migrate less-than-fully efficient and effective systems and processes merely because those systems and processes are most often used. Accordingly, OPM’s proposed reorganization should address agency management challenges, such as those in our high-risk program, priority open recommendations, or those identified by agency IGs. OMB, OPM, and GSA are aware of our related prior work, including major management challenges, but have not demonstrated how the proposed reorganization will help address these challenges (see fig. 7). Based on a document released by the administration on May 15, 2019 discussing its rationale for the merger of OPM and GSA, the reorganization should better support human capital delivery across the federal government by centralizing the services provided by both agencies, and reducing duplication. The reform plan also acknowledges that federal human capital management remains a high-risk area due to mission-critical skills gaps within the federal workforce. The reform plan further states that OPM does not have the capacity to address the high- risk issues we have identified, and progress would be achieved more efficiently by transferring OPM’s responsibilities to other government entities, including GSA and the EOP. However, as of May 17, 2019, OMB, OPM and GSA had not provided any documentation or analysis to demonstrate how the proposed reorganization would help resolve high- risk issues. The reform plan also draws attention to the OPM security breach that occurred several years ago, and cites it as a reason for moving information technology systems to GSA. We have five open priority recommendations to OPM regarding information security, as we reported to OPM in April 2019. For example, in May 2016, we recommended that OPM update security plans to ensure controls specific to high-impact systems are addressed, provide and track training for individuals with significant security responsibilities, and ensure that security control assessments specific to high-impact systems are comprehensive. To fully implement these recommendations, we reported that OPM needs to complete its ongoing efforts in each of these areas by implementing an automated system for management of security controls and security plans, defining and completing its planned corrective actions on training, and reviewing completed security control assessments. It is unclear whether OMB, OPM, and GSA have fully considered how relevant major management challenges identified by OPM’s and GSA’s IGs may affect the proposed reorganization (see fig. 7). For example, the GSA IG’s 2018 report on management challenges contains a number of findings that call into question GSA’s capacity to take on certain responsibilities the administration proposes transferring to GSA as part of the reorganization. Specifically, the report discusses GSA’s challenges with managing internal controls, prioritizing cybersecurity, and managing human capital. By addressing major management challenges and adopting best practices and processes as part of the reorganization effort, the administration will be better positioned to successfully implement their proposal. As of May 17, 2019, OMB, OPM and GSA had not provided documentation that they had identified specific actions that can be taken administratively versus those that will require legislative action to reorganize OPM. We asked OMB, OPM, and GSA for their views on what legal authority, including appropriations, they are relying on to reorganize OPM, including any additional authority that may be needed. As described earlier in this statement, these agencies have not provided implementation plans or other details on the reorganization. Similarly, they have not provided details on the statutory underpinnings for OPM’s reorganization. To the extent the administration identifies the legal authority it is it relying on to support this proposed reorganization, or the additional legal authority it needs, we will continue to assess it. OPM is statutorily created as “an independent establishment in the executive branch.” In addition, the Director of OPM is vested with certain functions by statute, and the Director (or OPM designee) is required to perform those functions, including executing, administering, and enforcing civil service requirements. While the Director of OPM may delegate selected human capital management functions to other agencies, OPM remains statutorily responsible for certain oversight activities, such as establishing standards that apply to such delegated activities and making written findings, where appropriate, if an agency to which OPM delegated human capital management functions acts contrary to law, rule, regulation, or standard, and requiring that the agency take corrective action, among other activities. OPM has various statutorily required responsibilities related to administering civil service retirement, insurance, health benefits, and life insurance programs, among others. OPM is funded primarily through its revolving fund—which is made up of fees or reimbursements provided by agencies for services OPM provides, such as background investigations and human resources services— transfers from OPM’s Earned Benefits Trust Funds for administrative services, and discretionary appropriations for OPM’s general activities and the Office of IG. To execute certain transfers of functions from OPM to GSA, the administration has acknowledged the need for additional statutory authority, but has also stated that it will rely on existing authority to move certain functions administratively. For example, the Analytical Perspectives accompanying the President’s fiscal year 2020 budget acknowledges that the transfer of OPM functions to GSA will be completed using a combination of existing legal authority and legislation. However, the administration does not identify which functions will require legislation and which OPM functions may be transferred administratively. In particular, OMB’s Deputy Director for Management stated, in July 2018, that many of the administration’s reorganization proposals can be implemented in whole or in part through existing administrative authorities. The conference report accompanying the 2019 Appropriations Act directed OPM to submit a report that included, among other things, the legal authority under which OPM proposed to transfer the human resources solutions function within the OPM revolving fund to GSA. OPM’s report stated that it and GSA, in consultation with OMB, continue to deliberate upon the application and use of administrative authorities to transfer the OPM functions to GSA. In addition, in April 2019, the General Counsel of OPM told us that the agency is unable to provide its legal analysis to us because it was still in progress and the agency was waiting for certain executive branch actions to be finalized. Without this information, we cannot assess the legal authorities the administration is relying on to implement the reorganization of OPM. As Congress and the administration consider whether or how to restructure OPM, regardless of the eventual decision about the organizational arrangement, we believe that it will be important to retain the capacity to execute certain government-wide, strategic human capital functions. These include the capacity to (1) identify trends affecting the future of the federal workforce; (2) effectively collaborate and coordinate with key stakeholders to address these government-wide trends; (3) lead the design of government-wide solutions to shared human capital challenges; and (4) administer and enforce civil service laws and regulations. As noted in our prior work, these functions are desirable and appropriate because they generate broad consistency across federal agencies, which is critical for, among other things, ensuring that each federal employee has certain safeguards and protections regardless of where he or she works. They also produce certain efficiencies and economies of scale that come from central coordination, and help maintain a reasonably level playing field among federal agencies when competing for talent. This is particularly important because we continue to designate strategic human capital management as a high-risk area. While many day-to-day human capital responsibilities have been delegated from OPM to individual agencies over the years, OPM continues to play an important strategic role including in the creation, execution, oversight, and strengthening of human capital policies and programs. For example, OPM’s 2018-2022 strategic goals are to: Transform hiring, pay, and benefits across the federal government to attract and retain the best civilian workforce. Lead the establishment and modernization of human capital information technology and data management systems and solutions. Improve integration and communication of OPM services to federal agencies to meet emerging needs. Optimize agency performance. Moreover, OPM was given a key role in fulfilling the human capital-related goal of the most recent President’s Management Agenda, in which the administration noted its intention to partner with Congress on “overhauling the statutory and regulatory rules that have, over time, created an incomprehensible and unmanageable civil service system.” OPM, along with OMB and DOD was tasked with the goal of aligning and strategically managing the workforce to efficiently and effectively achieve the federal government’s mission. To carry out these government-wide, strategic responsibilities, the following capabilities, whether possessed by OPM or some other entity, will be essential for ensuring cost-effective leadership, management, and oversight of the federal workforce. In our March 2019 report, we noted that such trends as technological advances, an increased reliance on nonfederal partners, and changing demographics and shifting attitudes toward work, are affecting how federal work is done, and consequently the skills and competencies that workers need to accomplish agency missions. Moreover, recent publications by the administration and others have raised concerns about whether the government’s employment policies and practices are still relevant and desirable to the current and future workforce. As far back as 1989, we reported that OPM had not provided the leadership necessary to sustain attention to identifying and resolving critical human resource problems affecting government operations and preparing for the future. Although OPM has made progress in this area and provides a variety of services, its progress has been inconsistent and issues still remain. For example, in 2018, OPM issued its Federal Workforce Priorities Report, which identifies changes in the external environment that will likely affect federal human capital management, including the evolving role of workers, changes in technology, employee health, and shifting generational demographics. OPM has also hosted a series of symposia that provide human capital specialists insight on addressing workforce challenges of the future. While these and other efforts are all important steps in the right direction, more work is needed in other areas. For example, as discussed in our March 2019 report, over the years we have made a number of recommendations to OPM to help agencies better meet their missions in an era of changing technology, demographics, fiscal constraints, and other challenges. OPM agreed with most of these recommendations and has made some progress, but additional actions are needed. They include, for example, identifying existing skills and competencies, assessing gaps in existing and future skills and competencies, and monitoring progress toward closing skills gaps. Moreover, in our March 2019 High Risk report, we noted that OPM needs to fully address the recommendations in our January 2015 report. Our recommendation called on OPM to make more strategic use of government workforce data to build a predictive capacity for identifying and mitigating emerging skill gaps across government. Certain human capital issues, such as addressing mission critical skills gaps, are crosscutting in nature and require the coordinated efforts of multiple stakeholders. However, a key challenge we identified in our May 2014 report on strategies to help agencies meet their missions in an era of highly constrained resources was that the federal human capital community is highly fragmented, with multiple actors both inside and outside of government informing and executing human capital policies and initiatives in ways that are not always aligned with broader, government-wide human capital efforts. Within government, OPM, OMB, the Chief Human Capital Officers (CHCO) Council, and individual agencies create, implement, and oversee human capital initiatives. Those initiatives are shaped, in part, by input provided by labor unions and federal management councils such as the President’s Management Council. The federal chief human capital officers with whom we spoke noted that each of these actors possess its own mission, initiatives, agendas, chain of command, budgets, and oversight. While this is to be expected given their various roles and responsibilities, these same factors can create disincentives to collaborating to achieve common human capital goals. In response to this issue, we recommended in 2014 that OPM work with the CHCO Council to, among other actions, strengthen coordination and leadership on government-wide human capital issues. OPM agreed with our recommendation and issued a final regulation, effective in April 2017, requiring it and agencies take significant steps in identifying, prioritizing, and coordinating efforts to address critical human capital issues. We believe this final regulation represents an important step toward addressing fragmentation within the federal human capital community. Going forward, it will be important for OPM, or another entity, if reorganized, to work with the CHCO Council and other stakeholders to address our open recommendations concerning specific human capital functions. Indeed, many of our open recommendations, including those that require priority attention from OPM, call on OPM to work in conjunction with the CHCO Council. Government-wide or “enterprise” solutions are important because they can integrate the efforts of multiple departments and agencies to address crosscutting human capital challenges more effectively by leveraging agencies’ expertise, experience, technology, and other resources. However, in our 2014 report, we found that while agencies have many common human capital challenges, they tend to address these issues independently without looking to enterprise-wide solutions that could resolve them more effectively. Across government, there are examples of agencies and OPM initiating enterprise solutions to address crosscutting issues, including the consolidation of federal payroll systems into shared-services centers. While these and other actions are important steps in the right direction, the CHCOs we spoke with in 2014 identified certain barriers to greater coordination to address common problems. For example, federal budgeting and account structures reinforce the prevailing tradition of controlling agency resources within a single agency. Moreover, agencies may be reluctant to contribute resources to a government-wide approach because they may not get an equitable return on their investment, or may get a product that does not fit their needs. According to the CHCOs in 2014, two areas that are ripe for greater government-wide collaboration are human resource information technology (HR IT), and strategic workforce planning. Specifically, the CHCOs said agencies could be missing cost-savings opportunities by not coordinating HR IT investments within and across agencies. They noted that agencies are individually procuring identical systems rather than leveraging the purchasing power of multiple agencies to negotiate better prices or services, or use shared service centers. Similarly, several CHCOs we spoke with said agencies are not consistently leveraging lessons learned or collaborating to address difficulties they encounter with workforce planning models. To further agencies’ use of government-wide approaches, we recommended that the Director of OPM, in conjunction with the CHCO Council, should explore the feasibility of expanded use of enterprise solutions to more efficiently and effectively address shared or government-wide human capital challenges. Such actions could include: (1) seeking cost savings and improved functionality through coordinated government-wide human resources information technology planning and acquisition; (2) seeking agency input to ensure OPM’s workforce planning tools provide effective guidance for agencies; and (3) sharing workforce planning lessons learned and successful models across the government. OPM agreed with the recommendation and in September 2018, it reported that in spring 2019, data will be available to indicate whether surveys and tools to address government-wide human capital challenges are meeting their intended goals. In March 2019, OPM told us that it was conducting Human Capital Reviews with relevant agencies. However, to fully implement the recommendation, OPM, or another entity, if reorganized, needs to demonstrate continued progress in addressing government-wide human capital challenges. Broad consistency across federal agencies is important for ensuring that all federal employees have the same safeguards, rights, and protections regardless of where they work. These include, for example, merit principles; protection from prohibited human capital practices; the ability to organize, bargain collectively, and participate through labor organizations; and due process that is fair, fast, and final. OPM is responsible for executing, administering, and enforcing the civil service rules and regulations, and the laws governing the civil service. Additionally, OPM is required to establish and maintain oversight over delegated human capital activities, including delegated competitive examining activities, to ensure agencies are acting in accordance with the merit system principles and the relevant standards established by OPM, such as compliance with applicable laws, rules, regulations, executive orders, and OPM policies. OPM monitors overall implementation and identifies corrective actions when deficiencies are found. OPM conducts this oversight through three primary means: delegated examining unit audits, human resource management evaluations, and special studies. However, in our prior work, we have identified the need for more effective oversight in such areas as agencies’ use of hiring authorities, agencies’ classification programs, the conversion of political appointees to career positions, and the Senior Executive Service performance-based pay system. With respect to agencies’ use of hiring authorities, for example, to help strengthen the government’s ability to compete in the labor market for top talent, and to improve the federal hiring process, we recommended in 2016 that the Director of OPM, in conjunction with the CHCO Council, should determine whether opportunities exist to refine, consolidate, eliminate, or expand agency-specific authorities to other agencies and implement changes where OPM is authorized. OPM agreed with the recommendation and in December 2018, OPM said that it continues to research and examine streamlining opportunities, such as those identified in its July 2018 study on excepted service hiring authorities, as part of the broader initiative to modernize federal hiring practices under the President’s Management Agenda. However, OPM did not provide a time frame for implementation. In its March 2019 Congressional Justification for the fiscal year 2020 budget request, OPM included legislative proposals for new hiring authorities such as highly qualified experts and temporary appointments to help agencies meet critical needs as well as a change to the criteria for granting direct hire authority. While OPM has made some progress in this area, it will be important for the agency to follow through on its planned actions to streamline hiring authorities. To fully implement the recommendation, OPM or another entity, if reorganized, needs to complete these efforts and, as appropriate, develop legislative proposals in consultation with the CHCO Council. Thank you, Chairman Connolly, Ranking Member Meadows, and Members of the Subcommittee. This concludes my testimony. I would be pleased to answer any questions. If you or your staff has any questions concerning this testimony, please contact Triana McNeil at (202) 512-6806 (McNeilT@gao.gov). Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contacts named above, Sarah Veale (Assistant Director), Peter Beck (Analyst-in- Charge), Colenn Berracasa, Robert Goldenkoff, Chelsa Gurkin, Shelby Kain, Steven Putansu, Janet Temko-Blinder, Peter Verchinski, and Alicia White made key contributions to the testimony. Other staff who made contributions to the reports cited in the testimony are identified in the source products. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In June 2018, the administration proposed reorganizing OPM by devolving its responsibilities to other agencies and entities including GSA and the EOP; see the figure for details. OMB's role is to coordinate and oversee the reorganization proposal, with support from OPM and GSA. In June 2018, GAO reported on key practices to assess agency reform efforts. This testimony focuses on preliminary observations from GAO's ongoing work related to the transfer of functions from OPM to GSA and the EOP. Specifically, we evaluated (1) the extent to which OMB, OPM, and GSA have addressed key practices for effective reforms and reorganizations; (2) legal authorities that may affect the reorganization of OPM, and (3) key capacities important for effective strategic human capital management, which need to be in place regardless of how the leadership over federal human capital is organized. For the information in this testimony, as of May 17, 2019, GAO met with OMB staff, GSA officials, OPM's and GSA's Inspectors General staff, and analyzed documentation provided by GSA. GAO also reviewed its prior related work. The Office of Management and Budget (OMB), Office of Personnel Management (OPM), and General Services Administration (GSA) have generally not addressed key practices for agency reform efforts as they have moved forward with their proposal to reorganize OPM. They have not established outcome-oriented goals, developed a cost-benefit analysis or implementation plans, and have not fully involved or communicated their efforts with the Congress, employees, and other key stakeholders. OPM and GSA also have not shown how they will address management challenges that may affect their ability to successfully reorganize the government's central human capital functions. OMB, OPM and GSA have not identified specific actions, as of May 17, 2019, that can be taken administratively versus those that will require legislative action to reorganize OPM. The administration has acknowledged the need for additional statutory authority to execute certain transfers of functions from OPM to GSA and the Executive Offices of the President (EOP), but has also stated that it will rely on existing authority to move certain functions administratively. Without additional information from OMB and agencies, GAO cannot assess the legal authorities the administration is relying on to implement the reorganization. As the Congress and administration consider whether or how to restructure OPM, it will be important to retain the capacity to execute certain government-wide, strategic human capital functions, regardless of the decision made about the organizational arrangement. These capacities include an ability to identify future workforce trends and to effectively collaborate with stakeholders—for the purpose of creating, executing, and overseeing human capital policies and programs, and enforcing civil service laws and regulations. This is particularly important because GAO continues to designate strategic human capital management as a high-risk area.", "document_type": "gao"}
{"report": "As the federal agency with primary responsibility for civil aviation security within the United States, TSA promulgates security requirements, primarily through regulations but also through security directives and other mechanisms, and conducts inspections to ensure that airport operators, air carriers, and other regulated entities are in compliance with these requirements. Additionally, TSA oversees security operations at airports through different types of testing and vulnerability assessments to analyze and improve security, among other activities. As of December 2019, there were approximately 430 commercial airports nationwide. Airport operators, air carriers, and other regulated entities are responsible for implementing security requirements, primarily in accordance with their TSA-approved security programs. These programs generally cover day- to-day operations, including measures that contribute to mitigating insider threats. For example: For most commercial airports, airport operators must ensure there is an adequate law enforcement presence to support operations and prevent unauthorized access to security-restricted areas through, among other measures, employee vetting, the use of personnel identification media, and implementing access control systems. For most air carrier operations, the air carriers must implement measures to ensure the security of aircraft and facilities, such as preventing unauthorized access to aircraft; searching aircraft prior to boarding passengers; randomly searching service personnel, such as caterers, and their property prior to boarding the aircraft; and training employees in security procedures. In accordance with an airport operator’s security program, an air carrier may enter into an agreement with the airport operator to assume exclusive responsibility for specified security measures for all or portions of an airport’s security-restricted areas, including access points. This is known as an exclusive area agreement. The security programs that airport operators and air carriers implement, in accordance with federal regulations, are generally consistent across similarly-situated airports and air carriers. For example, all airports operating under complete security programs generally implement TSA- approved security programs that address the same requirements. However, the details of these programs and their implementation can differ widely based on the individual characteristics of the airport. For example, methods that airport operators use to control access into security-restricted areas vary because of differences in the design and layout of individual airports, but all access controls must meet minimum performance standards in accordance with TSA requirements. Airport operators and air carriers may also choose to implement measures beyond what is required by TSA, but they may choose not to pursue incorporating these additional measures into their security programs, because if incorporated into their security programs, TSA could then hold the regulated entities accountable for implementing such additional measures. By not incorporating the additional measures into their security programs, airport operators and air carriers retain the flexibility to alter such measures without TSA approval. The security measures that airport operators and air carriers implement are generally carried out within, or to prevent access to, security- restricted areas of an airport or aircraft. These areas include: Secured areas. Areas for which security measures, such as access controls, must be carried out to prevent and detect the unauthorized entry, presence, and movement of individuals and ground vehicles. This includes areas where domestic and foreign air carriers enplane and deplane passengers and sort and load baggage, and any adjacent areas not separated by adequate security measures. Security identification display areas (SIDA). Areas for which security measures, such as personnel identification systems, must be carried out to prevent the unauthorized presence and movement of individuals. Air operations areas. Areas for which measures must be carried out to prevent and detect the unauthorized entry, presence, and movement of individuals and ground vehicles. This includes aircraft movement and parking areas, loading ramps, and safety areas for use by TSA-regulated aircraft, and any adjacent areas not separated by adequate security systems, measures, or procedures. Sterile areas. Areas that, in general, provide passengers access to boarding aircraft and to which access is controlled through the screening of passengers and property. Figure 1 illustrates the variety of security-restricted areas of a typical larger airport, such as a category X or I airport, and aviation stakeholders’ primary responsibilities for securing the area. TSA’s Insider Threat Program, which was established in 2013, consists of offices across TSA conducting different portions of the insider threat mission, with TSA’s Law Enforcement/Federal Air Marshal Service office serving as the program lead. The program’s mission is to deter, detect, and mitigate insider threats to the nation’s transportation sector personnel, operations, information, and critical infrastructure. Other TSA offices that have key responsibilities in the Insider Threat Program include TSA’s Security Operations; Enrollment Services and Vetting Programs; Inspection; Intelligence and Analysis; and Policy, Plans, and Engagement, among others. To support inter-office coordination, TSA established the Insider Threat Advisory Group in 2015, which is a multi- office team of experts who review and analyze the program’s activities, identify gaps, and develop mitigation strategies, among other activities. The group is co-chaired by two TSA offices—Law Enforcement/Federal Air Marshal Service and Intelligence and Analysis. TSA’s Insider Threat Unit, which operates within the Law Enforcement/Federal Air Marshal Service office, serves as the focal point for all referrals of potential insider threat incidents. According to TSA, an insider threat includes direct risks to TSA’s security operations, as well as indirect risks that may compromise critical infrastructure or undermine the integrity of the aviation security system. Examples of insider threat events include compromises of airport security (e.g. using access and knowledge to smuggle contraband) and sabotage (e.g. intentionally damaging equipment meant to detect unauthorized access to security-restricted areas). TSA recognizes, however, that some insider threats may arise from complacency or ignorance rather than a malicious intent to cause harm, such as when workers assume a negligent approach to policies, procedures, and potential risks. The Insider Threat Unit receives referrals from a telephone tip line and email address; daily reports from the Transportation Security Operations Center detailing security policy violations, such as aviation workers attempting to bring prohibited items not necessary to their work duties into security-restricted areas of the airport; and internal and external intelligence reports and referrals. After a referral is made, the unit is to coordinate, disseminate, and retain all information when reviewing referrals and conducting investigations into potential insider threats. Specifically, the unit is to coordinate inquiries and investigations with the appropriate lead entities to include TSA offices; federal, state, and local law enforcement and intelligence agencies; and various airport and transit law enforcement authorities. According to one TSA official, many of these referrals do not require additional investigation because they were already appropriately mitigated at the local level. Referrals that meet the unit’s criteria are accepted for further investigation—called acceptances. Criteria include, for example, whether the incident involved a prohibited item, the perpetrator has multiple violations, the perpetrator attempted to circumvent security, or the perpetrator made threatening statements. According to Insider Threat Unit data from fiscal year 2017 through fiscal year 2019, there were an average of 138 referrals and 14 acceptances per month. The majority of referrals accepted for investigation during this time period occurred at category X and I airports (63 and 25 percent, respectively). Referrals where air carrier employees and other aviation workers are the potential insider threat each account for approximately one-third of referrals accepted for investigation. Table 1 discusses examples of insider threat incidents. TSA has ongoing activities that help mitigate insider threats, including long-standing historical efforts and more recent efforts initiated since 2017. For example, TSA initiated operations to randomly search aviation workers at high-risk airports through pat down searches and explosives trace detection. TSA also has plans to enhance its current Insider Threat Program. Airport operators are to implement security measures, primarily in accordance with their TSA-approved security programs, which detail the day-to-day operations of those entities and their responsibilities for controlling access to security-restricted areas, among other responsibilities. Based on our analysis of TSA’s representative sample, some airport operators choose to implement security measures beyond those required by TSA. For example, some airport operators use sophisticated technologies such as fingerprint readers to control access to security-restricted areas, or offer or require training for aviation workers about topics such as insider threats. Similarly, air carriers are to implement security measures in accordance with TSA-approved security programs. For example, air carriers are required to perform regular searches of aircraft. Some air carriers we spoke to said they also choose to implement additional measures not required by TSA to enhance their security posture, such as conducting full employee screening at dedicated checkpoints. Figure 2 provides examples of the variety of security procedures and technologies used by TSA, airport operators, and air carriers at typical category X or I airports to control access to security-restricted areas of airports and help mitigate insider threats. These efforts vary by airport, local needs, and resources available, among other factors. TSA has long-standing, established activities that the agency has conducted that help mitigate insider threats. These efforts directly or indirectly regulate or facilitate security at commercial airports and help mitigate insider threats. Specifically, TSA has programs to increase awareness of insider threats in the aviation community, analyze and disseminate intelligence, vet aviation workers and TSA staff, inspect and assess security at airports, and share information with the aviation community. We have previously reported on these efforts in our work on aviation security and perimeter and access control security at airports. Awareness and training. TSA promotes awareness of insider threats to the aviation community and disseminates materials on how to identify and report insider threats to aviation stakeholders, which they may use on a voluntary basis. Analyze and disseminate intelligence. TSA evaluates intelligence information related to both domestic and international adversaries (such as terrorists) who seek to leverage insiders and target the U.S. transportation system, among other things. TSA regularly disseminates this information to aviation stakeholders through TSA’s intelligence officers at its field offices, for example. There are approximately 80 field intelligence officers stationed throughout the U.S., Puerto Rico, and Guam, who provide information to airport officials and the aviation community on insider tactics and emerging threats, among other things. Vetting aviation workers. TSA facilitates background checks of aviation workers (e.g. baggage handlers and concessionaire employees) applying for unescorted access to security-restricted areas of airports. The background check includes a Security Threat Assessment that is generally made up of three parts: (1) near real- time vetting against terrorism watch lists and other federal databases, (2) verification of the applicant’s lawful presence in the United States, and (3) a fingerprint-based criminal history records check.Additionally, TSA staff, such as transportation security officers, undergo a pre-employment screening, including all parts of the Security Threat Assessment and other security checks, and a background investigation to determine the applicant’s suitability for the position. Depending upon their job duties, TSA staff at airports may be issued credentials for unescorted access to security-restricted areas of an airport. Inspections and assessments. Staff at TSA compliance hubs (field offices) inspect airports and air carriers and test security measures to ensure compliance with federal requirements. To further enhance airport security, TSA also performs comprehensive, targeted, and supplemental inspections and other compliance activities, such as assessments, investigations, and tests. Guidance, policies, and information sharing. TSA issues guidance and policies that, among other things, require airport operators and air carriers to implement or enhance access controls or other security measures, or share best practices on improving security and mitigating insider threats. TSA regularly communicates with aviation stakeholders to discuss security issues and policies. Since the beginning of fiscal year 2017, TSA has implemented a variety of activities to oversee and facilitate insider threat mitigation at commercial airports, either through new activities or by enhancing ongoing efforts. Among other things, TSA has taken steps to further augment vetting of aviation workers, enhance aviation worker screening, test airport security targeted toward identifying insider risks and vulnerabilities, and develop reference tools and guidance. See below for examples of TSA’s insider threat mitigation efforts initiated since the beginning of fiscal year 2017. Social media analysis. TSA augmented the vetting process for aviation workers, described above, in 2018 to include an evaluation of publically available social media information for individuals who match against a federal watch list and are applying for unescorted access to security-restricted areas of an airport. TSA uses information about the individual, including the social media information, to conduct the security threat assessment and determine whether to approve or deny the application. Proposed requirement for Rap Back enrollment. The Federal Bureau of Investigation’s Rap Back Service provides participating entities with ongoing notification of subsequent criminal activity that occurs after an individual’s initial criminal history records check. In 2019, TSA proposed requiring airport operators and air carriers to enroll in Rap Back and to subscribe covered aviation workers. As of December 2019, TSA has not yet imposed this requirement. Physical Screening of Aviation Workers Advanced Threat Local Allocation Strategy (ATLAS). TSA’s ATLAS tool generates a randomized schedule and location of procedures to physically screen aviation workers. The ATLAS tool randomly identifies the type of screening procedure by balancing on- person screenings, such as pat-down searches, and in-property screenings, such as testing for traces of explosives on workers’ property. Federal security directors may tailor the screenings and location based on local intelligence. TSA started using ATLAS in 2018 at high-risk airports to screen aviation workers entering or within security-restricted areas. Covert testing. TSA’s covert testing teams help identify security vulnerabilities in multiple aspects of aviation security (including airport access controls and vulnerabilities to insiders) and may recommend additional measures or procedures be implemented to mitigate these vulnerabilities. As described above, TSA increased the number of covert tests related to airport access controls and insider vulnerabilities in response to provisions of the Aviation Security Act of 2016. Further, in 2019, TSA began a covert test to assess vulnerabilities in TSA’s ATLAS program. Joint Vulnerability Assessment. Joint teams of TSA and Federal Bureau of Investigation officials assess vulnerabilities in multiple aspects of airport security and operations including fuel, cargo, catering, general aviation, terminal area, and law enforcement operations. The assessments are conducted at commercial airports identified as high-risk every three years and on a case-by-case basis at other airports. TSA revised the joint vulnerability assessment process in fiscal year 2017 to identify insider threat vulnerabilities and to suggest options to mitigate them. Insider Threat Mitigation Activity. In addition to the regular airport inspection and assessment duties, starting in fiscal year 2017, TSA required its aviation transportation security inspectors to conduct unannounced tests related to mitigating insider threats every fiscal year. Guidance, Notice, and Information Sharing Fraudulent identification guidance. In fiscal year 2017, TSA developed guidance for airport operators and air carriers on detecting fraudulent identification documents, including methods for detecting fraudulent identification and appropriate responses when discovered. Security directives. TSA updated a security directive in 2018 to mitigate potential insider threats by, among other things, requiring airport operators to post signs at sterile area entry points accessible by credentialed aviation workers. These signs advise individuals that they may be subject to inspection, among other things. Additionally, airport operators are required to conduct random inspections of vehicles when entering secured areas. Information Circulars. TSA issued information circulars in 2018 and 2019 that (1) recommended that airport operators and air carriers with exclusive area agreements conduct a vulnerability assessment of insider risks and develop a risk mitigation plan, and included best practices for the mitigation plan, and (2) described measures to prevent unauthorized access to aircraft and the flight deck. TSA has implemented efforts aimed toward enhancing its Insider Threat Program. TSA established an Executive Steering Committee with members from the program’s key offices to provide executive support and oversight across the multiple offices that compose the program. Also, TSA’s Insider Threat Advisory Group collaborated with the Aviation Security Advisory Committee (ASAC) to review and develop recommendations that would address gaps, redundancies, and vulnerabilities in the program. TSA Insider Threat Executive Steering Committee. TSA established the Steering Committee in October 2018 to be the central oversight body for managing insider risks and coordinating the agency’s mitigation strategies. Its purpose is to facilitate collaboration and decision-making across the program’s multiple offices, advance an integrated agency-wide strategy, and establish consistent executive support for TSA and ASAC efforts, among other things. Its work to date includes reviewing the 2019 ASAC recommendations described above and approving the development of the Insider Threat Roadmap, which is to describe TSA’s strategic vision. TSA Administrator’s Intent initiatives. Several objectives and initiatives from the Administrator’s Intent, published in June 2018, relate to mitigating insider threats. It identifies specific priorities, strategic goals, and objectives that the Administrator plans to accomplish by 2020. For example, one objective is to modernize TSA’s Insider Threat Program by, among other initiatives, expanding the Insider Threat Unit with dedicated staff from several key TSA offices. ASAC Subcommittee on Insider Threats. In 2018, the ASAC established a permanent, joint industry-government Subcommittee with members from TSA and various aviation stakeholders. The purpose of the Subcommittee is to provide a holistic and sustained body to research and make recommendations on risks posed by aviation workers to harm the aviation system. Previously, ASAC convened an industry-only Working Group on Airport Access Control on an as-needed basis. ASAC recommendations. In May 2019, at the request of the TSA Administrator, the ASAC issued a report to help enhance and broaden TSA’s Insider Threat Program through 21 recommendations. The recommendations span six areas of the insider threat concept: 1. threat detection, assessment, and response; 2. aviation worker vetting and evaluation; 3. aviation worker screening and access control; 4. 5. 6. governance and internal controls. TSA concurred with all 21 of the recommendations. As of October 2019, TSA officials reported that the agency had implemented one of the recommendations and created a document that details implementation steps for the remaining 20, progress on those implementation steps, and estimated timeframes for completion. According to TSA officials, previous recommendations made by ASAC have significantly contributed to the establishment and development of the Insider Threat Program, and they anticipate the 2019 report’s recommendations will have a similar positive effect. Further, TSA officials said that the next iteration of the Administrator’s Intent will incorporate these ASAC recommendations to help ensure that their implementation is tracked at the enterprise level. Overall, many airport operators help ensure the security of their facilities, including mitigating insider threats, through their efforts to comply with TSA regulations. However, airport operators may also implement additional measures beyond those required by TSA to improve their security posture. Some examples of voluntary efforts airport operators have reported implementing to help mitigate insider threats include physical screening of aviation workers at access points to SIDAs or secured areas in addition to TSA’s random screening under the ATLAS program, using sophisticated access control technologies such as biometric fingerprint readers, and offering or requiring training for aviation workers on additional security awareness topics. Although TSA requires airport operators to perform random aviation worker screening at sterile area access points, it does not require them to physically screen all aviation workers at all access points to security- restricted areas, at all times. However, some airport operators choose to voluntarily implement screening programs to physically search some or all workers or their property as they enter security-restricted areas. According to our analysis of TSA data collected in July through September 2019 from a representative sample of airports on their current insider threat mitigation measures, seven of 27 category X airports’ officials and 13 of 54 category I airports’ officials reported that when they screen aviation workers passing through an access point, they screen 100 percent of workers, their property, and their vehicles (if the screening operations take place at a vehicle access point). Airport officials from four of 44 sampled category II airports, 10 of 54 sampled category III airports, and one of 58 sampled category IV airports reported that they screen 100 percent of workers when screening operations are underway. At one category X airport we visited, airport officials said they implemented full worker screening, following the lead of one tenant air carrier. According to the officials, the airport has two worker screening checkpoints in the publicly-accessible baggage claim area that are used by all workers entering the security-restricted areas. These checkpoints use X-ray machines, explosives trace detection, and walk-through metal detectors to screen aviation workers and their property and ensure they do not carry items that are otherwise prohibited (e.g. firearms and illicit substances) and not required to perform their work duties beyond the worker checkpoint. Airport officials said these checkpoints are staffed by a dedicated crew of screeners employed by the airport operator, and officials believe having a consistent crew over time makes it easier for screeners to detect if a worker is behaving in an uncharacteristic or suspicious way. At one category I airport we visited, officials said that they established an insider threat program and implemented measures to mitigate insider threats in response to an illegal drug smuggling operation involving aviation workers that occurred at their airport. For example, they partner with TSA and local law enforcement to conduct full worker screening operations two to three times per week at randomly-selected times and locations, which supplements TSA’s ATLAS operations. Officials said during these operations, all arriving workers are funneled to the screening locations, and they are directed to walk through screening equipment that is capable of identifying metallic threats (e.g. guns and knives) and non- metallic threats (e.g. suicide vests and other weapons) both on person and in property. If the machines are not used, airport officials coordinate with TSA to conduct full-body pat-downs of all employees. Airport officials may also use open-and-look bag searches. At the same time, local law enforcement patrols the screening area with canine units to search for drugs and explosives. In general, category X, I, II, and III airports are required to implement measures to control access and prevent unauthorized entry to security- restricted areas of the airport. Airports choose their specific access control system and technology, such as cipher or keyed locks, proximity swipe cards, PIN readers, and biometric (e.g. fingerprint) authentication, provided such technology meets the standards of their TSA-approved security program. Category IV airports—which are typically the smallest commercial airports—are generally not required to identify security- restricted areas within their security programs and thus may not have mechanisms in place to control access to such areas. However, like the larger commercial airports, security programs for category IV airports must provide for adequate law enforcement support, and airport operators at these airports may choose to establish security-restricted areas and implement access control technologies or other measures at their discretion. According to our analysis of TSA data collected in July through September 2019 from a representative sample of airports, officials from most category X, I, and II airports reported that they have systems that use more than one technology to control access to sterile and secured areas of the airport, as shown in figure 3. Among category III airports, officials from 27 of 54 also reported using multiple technologies. Among category IV airports, officials from 37 of 58 reported using some type of access control technology, the most common being locks and keys. Technology at two category X airports we visited is used specifically to prevent workers from “piggybacking,” or attempting to enter security- restricted areas by following close behind another worker without swiping a proximity card or entering a PIN for access. For example, one airport has sensor towers at high-traffic doors from unsecured to secured areas of the airport. The two towers—one on each side of the door—can detect if more than one person crosses the threshold after only a single proximity card swipe and PIN entry. According to airport officials, when this happens, the nearby security cameras will pan toward the door so that security officials who monitor the feeds can view the individuals at the door and respond appropriately. Figure 4, below, shows this technology, as well as the proximity card reader and PIN pad, a separate reader and pad for elevator access, and signs describing security rules. At a second category X airport we visited, locking turnstiles are used to prevent piggybacking. Each worker who wishes to go through the access point must present their proximity badge and provide a fingerprint. Only then will the locked turnstiles unlock to allow that worker through. The turnstiles are on a timer, so if a worker does not go through within a set time, they will have to repeat the process from the beginning. Additionally, if a badge is presented more than one time within a specified time period, an alarm is triggered in the Airport’s Security Operations Center to alert airport security staff of a potential piggybacking incident. Figure 5 shows the card reader, fingerprint reader, and turnstile in use at one access point. Behind the turnstile, a TSA agent conducting ATLAS countermeasures waits for workers to come through. In general, according to TSA requirements, individuals with unescorted access to security-restricted areas of category X, I, II, and III airports must be trained on, among other things, escort procedures and the display and use of identification media. All airport operators across all airport categories must ensure that training for law enforcement personnel addresses the airport’s security program, among other security-related topics. For training offerings beyond what is required by TSA, our analysis of TSA data collected in July through September 2019 from a representative sample of airports showed the majority of airport operators at category X, I, II, and III airports reported that they offered or required training for aviation workers that specifically discusses insider threats, as shown in Table 2. Moreover, although they are not required to do so by TSA, many category IV airports reported they offer or require training on a variety of security- related topics, such as insider threats and reporting suspicious behavior and unusual activity. The six air carriers we spoke with reported they mitigate insider threats via their efforts to comply with federal requirements through their TSA- approved security programs. In general, federal regulations require that air carriers employ a variety of procedures to mitigate security threats. Among others, these measures may include: Preventing unauthorized access to security-restricted areas over which they have primary responsibility, such as aircraft (e.g. by performing regular searches) and areas covered by an exclusive area agreement, as applicable; Submitting applicant biographic information for criminal history records checks prior to issuing air carrier identification media or recommending that airport operators issue access credentials that grants an individual unescorted access to security-restricted areas of the airport; Using personnel identification systems that track information such as identification media expiration dates and appropriate level of access; and Providing training for workers who perform security-related duties or otherwise require access to security-restricted areas. Air carriers may also choose to voluntarily implement additional efforts to improve their security posture. As described above, these may be incorporated into an individual air carrier’s security program, but not necessarily. Air carriers we spoke with have implemented a variety of security measures. For example: To prevent unauthorized access to secured areas included in their exclusive area agreement or within their operations area, all air carriers we spoke to said they secure their facilities by employing at least one form of access control technology. The majority of air carriers (five of six) reported that they secure most access points with proximity card or fob readers, including one air carrier that reported it secures its access doors using additional measures beyond a proximity card swipe, requiring a PIN and a fingerprint as well. The sixth air carrier we spoke to said workers access security-restricted areas using keys or cipher combinations. Prospective air carrier employees may require access media credentials from the airport operator in addition to the air carrier. In some cases, the air carrier will accept the criminal history records check conducted by the airport operator to issue its own credentials, but officials from some air carriers we spoke to said they conduct more rigorous checks before issuing their air carrier credentials. For instance, one air carrier reported that it checks both the applicant’s employment history in addition to their criminal history, and it uses an additional set of disqualifying criteria beyond the regulatory minimum to determine suitability for hire. Some air carriers choose to further enhance their insider threat mitigation efforts. For example, one air carrier has a dedicated insider threat program and, at 16 airports, it implemented a screening program of workers and their belongings at dedicated checkpoints. Another air carrier created a team to monitor the use of the Known Crewmember program, a screening program that provides flight and cabin crews with expedited screening that may include a dedicated screening lane. According to air carrier officials, at its largest hub airport, the team reports on workers from all air carriers who violate the program’s rules to TSA. Some examples of such violations include crewmembers using the dedicated lane for leisure international travel or carrying other individuals’ bags through the Known Crewmember portal or passenger screening checkpoint and into sterile areas of the airport. Although TSA has multiple ongoing efforts to mitigate insider threats at commercial airports carried out by a number of offices, it does not have a strategic plan in place to guide its Insider Threat Program. When the program began in 2013, TSA initially developed a 2014-2016 Insider Threat Action Plan, which described TSA’s vision of an integrated insider threat program at TSA, and it included strategic goals, each with a set of objectives. However, according to TSA officials, TSA did not fully implement this Action Plan, and TSA did not renew or revise the Action Plan after 2016 due to the departure of the key sponsoring senior leader. Further, TSA officials said that the Action Plan does not reflect all the existing activities that TSA’s Insider Threat Program currently encompasses because the program has changed since 2014. TSA is aware of the importance of strategic planning and took steps to strategically plan for other programmatic efforts at the agency. For example, in 2019, TSA revised its National Strategy for Airport Perimeter and Access Control Security. This strategy describes how TSA seeks to secure the perimeter and control access to security-restricted areas of U.S. commercial airports, which is one concern related to insider threats. In 2018, TSA published its Administrator’s Intent to outline how TSA planned to execute its agency-wide strategy in the short term. The Intent includes one strategic objective to modernize elements of TSA’s Insider Threat Program, such as vetting capabilities. Also in 2018, TSA published the Cybersecurity Roadmap 2018, which details the agency’s efforts to protect its information technology infrastructure from adversaries who might seek to cause harm. Each of these documents contains the critical elements of strategic plans that are laid out by the Office of Management and Budget, including strategic goals and objectives. These strategic planning documents contain elements related to insider threats and can be drawn upon to help develop a comprehensive strategic plan that encompasses the myriad of activities across its many offices that compose TSA’s Insider Threat Program. In October 2018, TSA established the Insider Threat Executive Steering Committee in an effort to establish consistent executive-level engagement and support from the agency’s senior management. As described above, TSA’s Insider Threat Program is carried out by multiple, distinct offices at TSA, and TSA officials have indicated that the program could benefit from a more cohesive approach and oversight. During the course of our review, the Steering Committee approved the development of an Insider Risk Roadmap (Roadmap). According to TSA officials, the Roadmap is under development as of January 2020, and when completed, is to describe the future of insider risk mitigation for TSA. TSA officials were uncertain, however, of when the Roadmap would be completed and implemented. Given that TSA did not fully implement its 2014-2016 Insider Threat Action Plan, and it was never renewed or revised, it is important that TSA remain committed to developing and implementing the Roadmap and, as it moves forward in drafting the Roadmap, ensuring that it contains the critical elements of a strategic plan, including strategic goals and objectives. Federal internal control standards establish that management should define the entity’s objectives clearly and in alignment with the entity’s mission and strategic plan. Objectives should specifically identify what is to be achieved, how, by whom, and in what time frame, and should be defined in measurable terms so that performance toward achieving such objectives can be assessed consistently. More specifically, the Office of Management and Budget clarifies that a strategic goal articulates clearly what the agency wants to achieve to advance its mission, while strategic objectives reflect the outcome or impact the agency is trying to achieve and should facilitate prioritization and assessment for planning, management, reporting, and evaluation. For example, mission-focused strategic objectives express specifically the path an agency plans to follow to achieve or make progress on a single strategic goal. Having a strategic plan for its Insider Threat Program would better position TSA to ensure it is effectively coordinating across its multiple offices and leveraging each office’s resources to mitigate insider threats, a threat which has consistently been identified as the second-highest enterprise level risk. A strategic plan, such as the ones included in other examples of TSA roadmaps, would help both to (1) link these individual efforts to the program’s strategic goals and (2) describe how they contribute to the achievement of those goals and the agency’s stated mission. TSA officials agreed that developing and implementing a strategic plan such as the ones associated with other roadmaps would help ensure that (1) its efforts to develop the Insider Threat Roadmap would continue to progress and (2) executive-level support for strategic planning would remain a priority. Individual TSA offices have made progress developing methods to assess their individual office’s efforts, but TSA does not have a comprehensive set of performance goals that can be used to assess progress toward achieving the Insider Threat Program’s stated mission. The National Insider Threat Task Force, established under Executive Order 13587 of October 7, 2011, outlined the minimum standards and basic elements of an insider threat program as well as a Maturity Framework to help Executive Branch departments and agencies, such as TSA, increase the effectiveness of their insider threat programs, among other things. According to the Framework, program senior officials should use metrics to represent progress and better articulate the central role of its insider threat program in achieving the department or agency’s strategic objectives. The Office of Management and Budget specifies that performance goals are statements of the desired performance target to be accomplished within a certain timeframe, and a suite of performance goals should be used to assess progress toward achieving each strategic objective. Federal standards for internal control also state that entities should use performance goals to evaluate their performance in achieving their strategic objectives. Some TSA offices have developed indicators for measuring characteristics of their insider threat activities, but these do not exhibit the characteristics of performance goals as defined by the Office of Management and Budget. For example, TSA’s Security Operations office developed Key Performance Indicators for its ATLAS operations, which are operational indicators for the TSA staff carrying out the countermeasures. These include that teams must screen a percentage of workers who pass through the checkpoint and must meet their assigned screening time allotment. However, operational indicators such as these do not include baselines and timeframes for completion, which are characteristics of performance goals as described by the Office of Management and Budget. Moreover, the Insider Threat Program is without a strategic plan, and as a result, these operational indicators cannot link back to a strategic objective or show progress achieving such an objective, as called for by the Office of Management and Budget guidance. TSA identified the need to develop performance goals to assess its progress and effectiveness in its 2014-2016 Insider Threat Action Plan, which called for “a performance management system monitors and measures effectiveness of insider threat program.” According to officials, such a performance management system was never developed because of the departure of the key senior leader, as described above. Further, in its May 2019 report to the Administrator, ASAC recommended that TSA develop measures that assess the performance of its insider threat efforts. For example, ASAC recommended that TSA commission a comprehensive federally-funded research and development center to assist TSA in evaluating the performance of random or unpredictable aviation worker screening methods to mitigate insider threats. The report indicated that establishing measures of effectiveness and evaluating performance on such measures is “vital to proactive and effective insider threat management.” TSA officials said that the planned Insider Risk Roadmap may include performance goals for the Insider Threat Program, in addition to strategic goals and objectives. However, previous examples of Roadmaps for TSA efforts did not include references to specific, measurable performance goals that can be used to represent progress via targets and timeframes. Moreover, as described above, TSA officials are still drafting the Roadmap and are uncertain when it will be issued. Having documented and clearly defined performance goals that are linked to the program’s overarching strategic goals and objectives would better position TSA to understand the effectiveness of its insider threat efforts. As a result, TSA would be able to reduce the likelihood of expending resources on efforts that are not meeting the program’s stated mission. Focusing on the intended results of TSA’s insider threat efforts can promote strategic and disciplined management decisions that are more likely to be effective because managers are better able to target areas most in need of improvement and to select appropriate levels of investment. TSA could determine the success of its strategies, adjust its approach when necessary, and remain focused on results. Further, agency accountability can be enhanced when both agency management and external stakeholders—such as Congress—can assess an agency’s progress toward meeting its strategic goals. By developing such performance goals, TSA will better position itself to determine the Insider Threat Program’s progress toward achieving its mission of deterring, detecting, and mitigating insider threats to the aviation sector. TSA has consistently identified the insider threat among its highest enterprise-level risks and characterizes it as a significant and complex risk to aviation security. In the last ten years, TSA and aviation stakeholders have faced a consistent threat posed by insiders who used their access privileges and knowledge to commit criminal acts, such as drug smuggling, gun smuggling, theft, and attempted suicide bombing. Having an effective Insider Threat Program is critical to TSA’s ability to mitigate the risk of insiders causing harm to the civil aviation system. Since establishing its Insider Threat Program in 2013, TSA has taken steps to strengthen its efforts to combat the insider threat such as by implementing a program to physically screen aviation workers at high-risk airports. However, responsibility for the Insider Threat Program is spread across multiple offices within TSA and has made it challenging to synchronize and integrate activities across each office’s efforts. As of January 2020, TSA officials said that the Insider Threat Program does not have a strategic plan. However, officials said they are developing a new strategic “roadmap” for the Insider Threat Program but are uncertain when it will be issued. Developing and implementing a strategic plan with strategic goals and objectives will help improve coordination across the program’s multiple offices and prioritize and focus TSA’s efforts to ensure that resources are targeted effectively. Additionally, TSA has also not established performance goals to help assess its overall progress in achieving its Insider Threat mission. With specific performance goals tied to strategic objectives, TSA will have the necessary mechanism to assess the extent to which the program is achieving its objectives and overall mission. TSA has numerous efforts across the agency to address insider threats; and with performance goals, the program could assess progress, identify successes, gaps, and redundancies and prioritize and allocate resources effectively. When dealing with a program designed to keep the aviation system safe from criminal and terrorist acts, agency leaders and policy makers need to know how well the government is doing implementing its objectives. Establishing performance goals will help the agency and Congress assess the progress of the overall insider threat effort, target areas most in need of improvement, and select appropriate levels of investment. We are making the following two recommendations to TSA: The TSA Administrator should develop and implement a strategic plan for its Insider Threat Program that includes strategic goals and objectives. (Recommendation 1) The TSA Administrator should develop performance goals for its Insider Threat Program that assess progress achieving the strategic objectives in the insider threat strategic plan. (Recommendation 2) We provided a draft of this report to the Department of Homeland Security (DHS) for comment. In written comments, which are included in appendix I, DHS concurred with our two recommendations and described steps it plans to take to implement them, including an estimated timeframe for completion. TSA also provided technical comments, which we incorporated as appropriate. In response to our recommendations, DHS’s letter notes that TSA is in the process of drafting the 2020 Insider Threat Roadmap, which will include strategic goals and objectives to guide TSA in its efforts to mitigate insider threats. The letter further explains that the Roadmap will include performance measures to assess TSA’s progress achieving those strategic objectives. If fully implemented, these actions should address the intent of the recommendations. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of the Department of Homeland Security, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8777 or McNeilT@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. In addition to the contact named above, William Russell (Director), Kevin Heinz (Assistant Director), Winchee Lin (Analyst in Charge), Sarah Williamson, Benjamin Crossley, Dominick Dale, Daniel Gaud, Thomas Lombardi, and Amanda Miller made key contributions to this report.", "summary": "Aviation workers using their access privileges to exploit vulnerabilities and potentially cause harm at the nation's airports is known as an “insider threat.” TSA, airport operators, and air carriers share the responsibility to mitigate all insider threats at airports. In October 2019, TSA estimated there are about 1.8 million aviation workers at the nation's airports. GAO was asked to review TSA's and aviation stakeholders' efforts to mitigate insider threats at airports. This report (1) discusses the efforts that TSA, airport operators, and air carriers have taken to help mitigate insider threats at airports and (2) evaluates the extent to which TSA's Insider Threat Program is guided by a strategic plan and has performance goals. GAO reviewed TSA guidance; analyzed TSA data from a questionnaire sent to a representative sample of airport operators; and obtained information from TSA officials, officials from selected larger U.S.-based air carriers, and a nongeneralizable sample of seven airport operators, selected, in part, based on the number of aircraft take-offs and landings. The Transportation Security Administration (TSA), airport operators, and air carriers mitigate insider threats through a variety of efforts. TSA's Insider Threat Program comprises multiple TSA offices with ongoing insider threat mitigation activities, including long-standing requirements addressing access controls and background checks, and compliance inspections. TSA also initiated activities more recently, such as implementing TSA-led, randomized worker screenings in 2018. Airport and air carrier officials implement security measures in accordance with TSA-approved programs and may implement additional measures to further mitigate threats. For example, many airport operators reported using sophisticated access control technologies (e.g. fingerprint readers). Additionally, some air carriers reported conducting more rigorous background checks prior to issuing identification credentials to employees. TSA‘s Insider Threat Program is not guided by a strategic plan with strategic goals and objectives nor does it have performance goals. TSA does not have an updated strategic plan that reflects the Program's current status. TSA officials said that the plan was not updated due to turnover of key senior leadership. As of January 2020, TSA officials said they were developing a roadmap that could serve as a new strategic plan for the Program. However, officials had not finalized the contents and were uncertain when it would be completed and implemented. Developing and implementing a strategic plan will help guide TSA's ongoing efforts and coordinate TSA's agency-wide approach. TSA has not defined performance goals with targets and timeframes to assess progress achieving the Program's mission. Without a strategic plan and performance goals, it is difficult for TSA to determine if its approach is working and progress is being made toward deterring, detecting, and mitigating insider threats to the aviation sector. GAO recommends that TSA develop and implement a strategic plan that has strategic goals and objectives, and develop performance goals to assess progress achieving objectives in the strategic plan. TSA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "NASA’s human spaceflight plans have changed focus three times over the last 15 years. These plans have shifted back and forth between conducting a human lunar landing in order to inform the longer-term goal of human exploration of Mars and a mission that sends astronauts to an asteroid boulder in orbit around the Moon but does not include a lunar landing. Figure 1 highlights key events in NASA’s human spaceflight plans from 2005 to 2019. We have found that NASA has faced challenges developing systems capable of transporting humans to space over the past two decades. These include development efforts under NASA’s prior human spaceflight program—the Constellation program—which was canceled in the face of acquisition problems and funding-related issues. More recently, we have found that NASA has struggled to complete its current human spaceflight programs—Orion, SLS, and Exploration Ground Systems—within their established cost and schedule baselines. Establishing a sound business case to ensure resources align with requirements includes following best practices for product development and creating cost estimates and schedules. NASA’s prior human spaceflight programs highlight challenges created when programs do not establish a sound business case. For example: In 2009, we found that NASA had not developed a solid business case—including firm requirements, mature technologies, a realistic cost estimate, and sufficient funding and time—needed to justify moving the Constellation program forward into the implementation phase. We found that the program had not developed a solid business case because the program had a poorly phased funding plan that increased the risk of funding shortfalls, among other reasons. This resulted in the program not completing planned work to support schedules and milestones, and ultimately the program was canceled. Over the past 5 years, we have issued several reports assessing the progress of NASA’s Orion, SLS, and Exploration Ground Systems programs relative to their agency baseline commitments and on technical challenges facing the programs. In 2018, we found that all three programs have been at risk of cost and schedule growth since NASA approved their baselines, and have since experienced cost growth and schedule delays. This was in part because NASA did not follow best practices for establishing cost and schedule baselines for these programs, including not updating cost and schedule analyses to reflect new risks. As a result, NASA overpromised what it could deliver from a cost and schedule perspective. Further, in 2019 we found that NASA should enhance contract management and oversight to improve SLS and Orion program outcomes. NASA’s past approach in this area has left it ill-positioned to identify early warning signs of impending schedule delays and cost growth or reap the benefits of competition. We have made 20 recommendations in prior reports to strengthen NASA’s acquisition management of SLS, Orion, and Exploration Ground Systems. NASA generally agreed with GAO’s recommendations, and has implemented eight of the recommendations. Further action is needed to fully implement the remaining recommendations. For example, in 2019, we recommended that NASA direct the SLS and Orion programs to reevaluate their strategies for incentivizing contractors and determine whether they could more effectively incentivize contractors to achieve the outcomes intended as part of ongoing and planned contract negotiations. NASA agreed with the intent of this recommendation and stated that the SLS and Orion program offices will reevaluate their strategies for incentivizing contract performance as part of contracting activities including contract restructures, contract baseline adjustments, and new contract actions. We will continue to follow up on the actions the agency is taking to address this recommendation. NASA initiates space flight programs and projects to accomplish its scientific or exploration goals. A NASA program has a dedicated funding profile and defined management structure, and may or may not include several projects. Projects are specific investments under a program that have defined requirements, life-cycle costs, schedules, and their own management structure. NASA uses the term “tightly coupled program” to refer to a program that is composed of multiple projects that work together to complete the program’s mission. NASA policy states that programs and projects shall follow their appropriate life cycle. The life cycle for programs and projects consists of two phases: 1. formulation, which takes a program or project from concept to 2. implementation, which includes building, launching, and operating the system, among other activities. Senior NASA officials must approve programs and projects at milestone reviews, known as key decision points (KDP), before they can enter each new phase. The life cycle for a single program closely resembles the life cycle for a spaceflight project. For example, the SLS program follows the project acquisition life cycle because it is not composed of multiple projects. Figure 2 depicts a notional NASA life cycle for a tightly coupled program and for a project. The formulation phase culminates in a review at KDP I for tightly coupled programs and KDP C for projects. This decision point is also known as confirmation, at which cost and schedule baselines are established and documented in a decision memorandum. The decision memorandum outlines the management agreement and the agency baseline commitment. The management agreement can be viewed as a contract between the agency and the program or project manager. The program or project manager has the authority to manage the program or project within the parameters outlined in the agreement. The agency baseline commitment includes the cost and schedule baselines against which the agency’s performance on a program or project may be measured. To inform the management agreement and the agency baseline commitment, each program and project with a life-cycle cost estimated to be greater than $250 million must also develop a joint cost and schedule confidence level (JCL). A JCL produces a point-in-time estimate that includes, among other things, all cost and schedule elements from the start of formulation through launch, and incorporates and quantifies known risks, assesses the effects of cost and schedule to date on the estimate, and addresses available annual resources. The results of a JCL indicate the probability of a program or project’s success of meeting cost and schedule targets. NASA has initiated multiple programs to help the agency achieve its Artemis III mission and longer-term lunar exploration goals. These programs include a platform in the lunar orbit, a landing system to put humans on the surface of the Moon, and robotic lunar landing services. Gateway. The Gateway program aims to build a sustainable platform in lunar orbit to support human lunar exploration and scientific experiments by NASA and its commercial and international partners. NASA is planning for Gateway to maneuver to different orbits around the Moon, which will allow access to a variety of locations on the lunar surface. The Gateway program is the first program NASA has designated as a tightly coupled program. The program is composed of multiple projects, which are responsible for executing portions of the Gateway mission. Individual teams manage the projects and each project will have its own cost estimate and launch readiness date. Gateway program management is responsible for ensuring the overall integration of all the individual projects. See figure 3 for a description of the three Gateway projects that NASA has initiated. In addition to Gateway, NASA initiated several other programs: Human Landing System. The Human Landing System, or lunar landers, is to provide crew transportation from Gateway to the lunar surface and back and demonstrate capabilities required for deep space missions. NASA anticipates that there will be three stages to the landers—a descent, ascent, and transfer stage—but the number of stages may vary depending on the contractors that NASA selects to develop the system. NASA is planning for the descent stage to serve as a crew and cargo lander; the ascent stage to bring crew back to Gateway from the lunar surface; and the transfer stage to transfer the ascent and descent stages from Gateway orbit to a lower lunar orbit for the landing. Space Suits. NASA plans to update the design of its space suits, which supply life support, including oxygen and water, among other things, to astronauts. The updates include additional protection from extreme temperatures and hazards in the lunar environment, such as dust; increased mobility; and extended service life for lunar surface operations. Commercial Lunar Payload Services. Under Commercial Lunar Payload Services, commercial partners provide end-to-end commercial payload delivery services to the surface of the Moon. The services include integrating payloads onto a robotic lander, launching the lander, and operating the lander and payloads. The payloads include science instruments and technology demonstrations that will characterize the lunar environment and inform the development of future landers and other exploration systems needed for humans to return to the lunar surface. Volatiles Investigating Polar Exploration Rover. NASA plans to develop a robotic lunar rover for long duration operations to investigate volatiles—which include water, carbon dioxide, and other chemicals that boil at low temperatures—at the lunar South Pole that could be used to support sustained human presence on the Lunar surface. NASA plans to utilize landers from the Commercial Lunar Payload Services to deliver the rover to the lunar surface. Orion and SLS. Orion is the crew capsule to transport humans from the Earth to Gateway and beyond. SLS is the vehicle NASA will use to launch the Orion crew capsule and cargo beyond low-Earth orbit, including to Gateway. Figure 4 shows a notional configuration of Gateway, the first integrated Human Landing System, and the Orion crew capsule. In this configuration, the Human Landing System ascent stage, Gateway Logistics and Power and Propulsion Element (PPE), and Orion crew capsule are designed to dock with the Gateway Habitation and Logistics Outpost. The Advanced Exploration Systems organization is responsible for overseeing the Gateway and Human Landing System programs and reports to NASA’s Associate Administrator for Human Exploration and Operations Mission Directorate (HEOMD). Another organization within HEOMD—Exploration Systems Development—is responsible for the development of the Orion crew capsule. The Office of the Chief Engineer and Office of the Chief Financial Officer are responsible for NASA policies and guidance related to the development of these systems. After the March 2019 announcement to accelerate the human lunar landing to 2024, NASA acknowledged that it could not complete all of its original plans under the new time frame. The original plans for a human lunar landing in 2028 included an expanded Gateway and uncrewed demonstrations of components of the Human Landing System. In response to the new direction, NASA decided to execute its lunar plans in two phases. Phase 1 focuses on systems NASA identified to support the Artemis III mission in 2024. Phase 2 builds upon Phase 1 efforts and focuses on establishing a long-term presence on the lunar surface through future Artemis missions, and is not currently the focus of NASA’s efforts (see figure 5). NASA made several changes to its prior lunar plans to increase the speed of developing the systems needed to meet the aggressive timeline for the Artemis III mission. For example: NASA reduced the scope of the Gateway program for Phase 1 by deferring or eliminating components, and changing its configuration. NASA removed a component that an international partner had planned to contribute and deferred work on a habitation component and other potential international contributions to Phase 2. Acknowledging that some elements of Gateway had to be deferred or eliminated for the first phase is a positive step NASA has taken to try to achieve an aggressive schedule. In some cases, NASA changed the acquisition strategy to increase the speed of development work. For example, NASA had planned to build the Habitation and Logistics Outpost in-house, but due to the 2024 acceleration announcement, now plans to award a contract for its development. In addition, NASA changed its plans to acquire the Human Landing System as an integrated system instead of by stage to meet the accelerated timeline. NASA developed a broad agency announcement for the Human Landing System with the goal of awarding contracts by the end of January 2020. NASA released a draft broad agency announcement for the integrated system in July 2019, about 4 months after receiving direction to land humans on the Moon by 2024. Human Landing System program officials raised concerns about the program’s ability to meet the 2024 timeline, but said they are trying to mitigate this risk by incorporating input from prior studies and feedback from industry into the program’s draft broad agency announcement. See table 1 for the status of NASA’s lunar programs, including changes NASA made to prior plans and timelines to meet the 2024 lunar landing goal. NASA is still considering the extent to which competition will be part of its acquisition plans to meet the accelerated 2024 landing. Competition may be a critical tool for achieving the best possible return on investment for taxpayers, and can help improve contractor performance. In addition, in 2014, we found there were competition opportunities for future SLS development work that may promote long-term affordability. We recommended that NASA assess the extent to which the agency could competitively procure development and production of future elements of the SLS to promote affordability. NASA agreed with this recommendation. However, NASA’s progress implementing it has been limited. For example, NASA awarded a sole-source contract for the upper stage engine, which further limits an opportunity for competition for the program. For Gateway Logistics Services and the Human Landing System, NASA officials stated that they are considering awarding multiple initial contracts. If NASA does award multiple contracts, NASA officials stated they would then be able to have the contractors compete for further development of the components and possibly for specific missions. Conversely, NASA does not plan to competitively award a contract for the Gateway Habitation and Logistics Outpost, citing the aggressive Artemis III schedule as a factor for this decision. NASA has identified the components of its lunar architecture—such as Gateway and lunar landers—but it has not fully defined a system architecture or established requirements for its lunar mission. A system architecture, among other things, defines the dependencies and interfaces between the components. The NASA systems engineering handbook states that defining the system architecture early enables NASA to develop components separately from each other while ensuring that they work together effectively to achieve top-level requirements. For example, a system architecture for the Artemis III mission would describe the relationships and interfaces between Gateway and the Human Landing System, ensuring that after the two programs are completed, they will work together properly to execute the mission. Figure 6 is an illustration of how specific program and project requirements flow down from NASA’s strategic goals and objectives. NASA officials told us they started with defining individual program and project requirements, and then plan to define the system architecture in an architecture definition document and the lunar system requirements in six separate HEOMD documents. These documents are in various stages of completion. HEOMD officials said they expect to finalize the overall architecture definition document at the end of 2019. They plan for this document to include a description of the integrated architecture, including the architecture’s components and high-level interfaces required for initial human lunar surface missions. In addition, HEOMD has six other documents that establish requirements for human space exploration missions, among other things. Three of these documents are currently outdated because they do not address lunar landings. HEOMD officials stated that they do not expect the documents to be updated before the end of 2019. NASA officials told us that they did not start with these higher-level architecture and all of the requirements documents because they thought it was important to first establish requirements for individual programs and review what contractors proposed for Gateway and the Human Landing System, and incorporate industry input on what requirements are feasible. The Human Landing System draft request for proposals contained a notional architecture that has three stages, but the agency is open to selecting contractors that do not follow this notional architecture. In our work to develop a framework for assessing and improving enterprise architecture management, we found that a mature architecture should ensure that components of the architecture align their plans with enterprise-level plans. Establishing such alignment is essential to achieving goals and supporting solutions that are appropriately integrated and compatible. NASA’s approach of defining the lunar architecture and associated requirements concurrently with programs setting their own requirements presents the risk of mismatches of requirements across and within programs. Such mismatches increase the risk of technical problems, schedule delays, and cost overruns. For example, the Gateway program is tracking the potential misalignment of requirements as a risk because the PPE project finalized its requirements before the Gateway program finalized corresponding requirements at the program level. PPE officials stated they finalized their requirements first because they had started work under a prior project and, as a result, moved quickly through early development activities. The Gateway program and PPE project officials said that when they reviewed the PPE requirements with Gateway’s requirements, they found two possible gaps. For example, NASA officials explained that there is a difference in the amount of power the PPE contractor is required to deliver for the PPE and Gateway’s requirements for power. The program is working with the PPE project office and contractor to study the potential gaps and determine how to resolve them if needed. The Gateway program officials said they would continue to assess gaps and risks related to requirements alignment for all projects. HEOMD officials agreed that there is a risk of discovering integration challenges across programs. NASA officials have taken action on one strategy to minimize this risk, and are considering two other potential mitigation strategies. To help ensure that the components of the lunar architecture can work together, NASA included international interoperability standards in its requests for proposals for the lunar programs. For example, there are standards for how the components will dock with each other. NASA officials said that including these standards would help mitigate integration challenges. The two other potential mitigation strategies are the following: Establish a Lunar Exploration Control Board. NASA is in the process of establishing a board that would act as an architecture configuration management body. Configuration management is a process used to control changes to top-level requirements. In our prior work on developing and maintaining systems or networks, we found that effective configuration management is a key means for ensuring that additions, deletions, or other changes to a system do not compromise the system’s ability to perform as intended. The board could serve as a body to make decisions that affect multiple lunar programs and ensure that changes to components of the lunar architecture do not affect NASA’s ability to accomplish a successful lunar landing. Hold cross-program synchronization or integration reviews. To help ensure that requirements are aligned across programs, a senior HEOMD official said NASA plans to hold cross-program synchronization or integration reviews. However, the official said NASA has not defined at what level those reviews would occur, when those reviews would occur, or what specific contractor data would be reviewed. Ensuring the Lunar Exploration Control Board is involved in these reviews will help the board in its role as a configuration management body and inform decisions that affect multiple lunar programs. NASA’s system engineering handbook states that activities to integrate systems throughout a system life cycle help to make sure that integrated system functions properly. These activities include conducting analysis to define and understand integration between systems. NASA is moving quickly to develop individual programs and projects that must work together as part of the broader lunar architecture. Delaying decisions about how and when NASA plans to hold synchronization or integration reviews risks discovery of changes late in the acquisition process. As stated in NASA’s system engineering guidance, the later in the development process changes occur, the more expensive they become. NASA has taken positive steps to increase the visibility into the cost and schedule performance of the Gateway program’s projects, but decisions on analyses to support program-level cost and schedule are still pending. In addition, the NASA Administrator has stated that Artemis III may cost between $20 billion to $30 billion, but NASA officials stated that the agency does not plan to establish an official cost estimate. As of October 2019, NASA was still defining its approach for developing cost and schedule estimates for all programs and projects in the lunar architecture, but we found NASA has made some decisions related to the structure of the Gateway program that will provide visibility into cost and schedule performance. In particular, NASA’s decision to structure the Gateway program as a tightly coupled program means that the projects that compose the Gateway—Power and Propulsion, Habitation and Logistics Outpost, and Logistics—are to develop individual project cost and schedule baselines by which performance will be measured. NASA officials stated that they expect this will provide accountability for each project to adhere to its cost and schedule baseline. This structure is a positive step for NASA to improve management of large, complex programs, and could have been beneficial to previous human spaceflight programs. For example, cost and schedule baselines for key hardware elements of the Space Launch System program—such as the core stage—might have provided earlier warning signs of development challenges affecting cost and schedule performance. NASA policy requires tightly coupled programs with a life cycle cost estimate greater than $250 million to conduct a program-level joint cost and schedule confidence level (JCL) to inform an agency baseline commitment. A JCL is a calculation that NASA uses to estimate the probability of success of a program or project meeting its cost and schedule baselines. However, NASA decided to remove the requirement for the Gateway program to establish an agency baseline commitment, and instead, require the program to document its cost and schedule estimates for phase 1 in a program commitment agreement. NASA officials explained that the agency viewed requiring the Gateway program to conduct a JCL to inform cost and schedule baselines as duplicative of analysis the projects are required to conduct to inform their project level baselines. In October 2019, Gateway program officials stated they have reconsidered this direction and now plan to conduct a program-level JCL. However, given that NASA officials previously determined they would not require the Gateway program to establish a baseline that is informed by a program-level JCL, the decision to conduct a JCL is subject to change again. NASA’s commitment to the program’s October 2019 decision to conduct a program-level JCL would enhance oversight and management for Gateway. NASA’s cost estimating handbook states that a JCL can serve as a valuable management tool that helps enforce some best practices of program planning and control, and potentially enhance vital communication to various stakeholders. Having a program-level JCL could help the program identify additional cost and schedule risks associated with integration of, or dependencies across, Gateway components that individual projects may not identify. As a tightly coupled program, Gateway has project schedules that are dependent on one another. For example, PPE provides power to subsequent Gateway components, such as the Habitation and Logistics Outpost, and must be launched and in lunar orbit for the outpost to dock with PPE. A program- level JCL would be able to quantify risk of delay across all dependent activities, regardless of which individual project experiences the delay. It would also provide NASA decision-makers and external stakeholders, such as Congress, with the probability of the program meeting both its cost and schedule commitments to support the Artemis III mission. The Gateway program is also the program in the lunar architecture that is the furthest along in developing a schedule aside from the SLS, Orion, and Exploration Ground Systems programs. The program expects to have an integrated master schedule in late 2019, but in the meantime has developed a high-level notional schedule. We identified two challenges with the Gateway program’s schedule that stem from decisions to meet the program’s rapid pace of development. Program and project technical reviews do not align. The NASA program management handbook states that lower-level technical reviews, such as project preliminary design reviews, are typically conducted prior to the program-level reviews. In addition, GAO’s Schedule Assessment Guide states that lower-level project schedules should be consistent with upper-level program review milestones. This creates consistency between program and project schedules, which enables different teams to work to the same schedule expectations and ensures the proper sequencing of activities. The Gateway program obtained approval from the NASA Associate Administrator to tailor its review schedule. This includes the replacement of traditional reviews with program sync reviews informed by project-level technical reviews. The program has some of the project-level technical reviews for its projects—PPE, Habitation and Logistics Outpost, and Logistics—occurring after equivalent Gateway program-level reviews. The Gateway program-level reviews are referred to as sync reviews, during which information is assessed across all projects. For example, the Logistics project plans to hold its preliminary design review after the Gateway program preliminary design-informed sync review. Figure 7 shows the preliminary Gateway program schedule and identifies reviews that differ from the notional tightly coupled program schedule found in NASA guidance. Without the results of project-level reviews, program officials may have limited information to assess progress at program-level reviews. This opens up the possibility of costly re-designs at later stages of the program life cycle. Gateway program officials said as the program progresses, they plan to assess the risk of holding a project-level review after a program-level review against the risk of delaying a program-level review to hold all the project-level reviews first. Officials added that they are still reviewing their approach for the timing of the reviews. We will continue to follow up through future work on the Gateway program’s risk assessments related to the timing of the technical reviews. Scheduling of key program milestone reviews after 2021 deferred. The Gateway Program does not yet have key milestone reviews—known as key decision points (KDP)—scheduled after 2021 (see figure 7). Currently, the final key decision point scheduled for the program is KDP I in 2021, which evaluates the completeness of the preliminary design, including for projects, and determines the program’s readiness to begin the detailed design phase. However, NASA policy requires the program to conduct two other key decision points that the Gateway program has not yet scheduled. Program officials told us that they want to determine the need for subsequent decision points after the systems have matured further in their development. During the period between 2021 and 2024, the Gateway program plans to launch and assemble its three components—PPE, Habitation and Logistics Outpost, and the first logistics vehicle—and integrate with the Human Landing System and Orion. It may be appropriate not to schedule a KDP III—a decision point that evaluates the readiness of the program, including its projects, for launch and early operations—for the Gateway program since the projects will launch separately and conduct operations on different timelines. However, not having a KDP II—a decision point that evaluates the program’s readiness for assembly, integration, and testing, prior to a system integration review—will limit information available to senior leaders for decision-making. Without scheduling a KDP II, NASA risks not having a formal mechanism to ensure that NASA has identified and sufficiently addressed any integration issues across the three projects. The NASA Administrator made a public statement that the Artemis III mission may cost between $20 billion and $30 billion, but NASA officials told us they do not plan to develop an official cost estimate for the Artemis III mission. A senior HEOMD official said that the agency developed a cost estimate that included costs for the lunar mission to 2028 to support budget submissions. However, the official said this life-cycle cost estimate included costs outside of the Artemis III mission, such as for missions later than Artemis III, and may not include integration and overall management costs. NASA officials told us that it is complicated to separate out costs for each mission and, as a result, do not plan to develop an Artemis III cost estimate. In addition, senior NASA officials stated that many of the programs needed to execute the mission are currently in the early stages of acquisition, and therefore NASA has limited cost information. Meanwhile, NASA requested an additional $1.6 billion in fiscal year 2020 above its initial budget request to support the Artemis III mission. Cost estimates provide management with critical cost-risk information to improve the control of resources in the present and future. GAO’s Cost Estimating and Assessment Guide states that a life-cycle cost estimate enhances decision-making, especially in early planning of an acquisition. Individual program cost estimates would not capture the integration costs across programs. Without an Artemis III cost estimate, NASA will not be able to effectively monitor total mission costs and Congress would have limited insight into mission or program affordability when making decisions about each year’s budget request. Given the breadth of activity and funding required for NASA to achieve a human lunar landing, a number of stakeholders have advocated for NASA to carry out this mission in a different way than NASA is pursuing. For example, one advocate proposed alternative lunar architectures that do not include the use of Orion, SLS, or Gateway, and instead rely on the use of commercial vehicles, and a former NASA associate administrator has promoted increased use of NASA’s current programs, including SLS. Agencies can use the process of assessing alternatives to justify their decisions and demonstrate careful planning. While NASA policy does not require programs to analyze alternatives before starting work, GAO best practices state that analyzing alternatives provides a framework to help ensure that entities consistently and reliably select the alternative that best meets the mission need based on selection criteria, such as safety, cost, or schedule. Similarly, the Department of Defense, an agency that also invests billions of dollars in acquisitions, considers an analysis of alternatives a key input to defining a system’s capabilities and assessing affordability. We previously found that analyzing alternatives is a key element in establishing a sound business case for a new architecture or program. Having a strong business case, including a formal assessment of alternatives, would help NASA effectively communicate its decisions to various stakeholders and facilitate a better understanding of its current lunar plans. NASA officials told us that they arrived at the current architecture and the designs of its lunar programs by conducting numerous studies and analyses over multiple decades. These studies looked at aspects of the various lunar missions NASA has planned over time, including the prior Constellation program and Journey to Mars effort. A HEOMD official responsible for mission directorate analyses said that the studies ranged from quick turn-around analyses to long-term, thorough studies. NASA officials identified 12 studies completed since the conclusion of the Constellation program in 2010 that informed their decision to build Gateway and other aspects of the lunar architecture. The studies varied in focus, ranging from a study on the overall framework for a mission to Mars to a study exclusively on the human lunar landers. We reviewed these 12 studies to determine the extent to which NASA analyzed alternatives to inform its current lunar architecture. We found that some of the studies contained detailed analyses, but had a narrow scope. For example: NASA conducted a study on the design of its human lunar landers that identified several alternative designs for the lander configuration, including two- and three-stage landers. The study provided an analysis on each alternative in order to compare those alternatives, given the physical constraints of SLS and commercial launch vehicles. HEOMD reviewed prior studies on a cislunar habitation facility conducted by internal and external partners that informed an Assessment of Alternatives for the Gateway program. At the time the mission directorate conducted this assessment, the concept was focused on the Journey to Mars effort, and mentioned lunar landers only as a potential secondary mission. The assessment analyzed various alternative configurations that Gateway might use and selected one of them based feasibility and schedule. NASA conducted studies on the best orbit in which to place Gateway. While these studies were robust, they did not more broadly analyze whether Gateway was the best solution to meet the mission need based on selection criteria. The following are examples of topics that NASA could have addressed if they had analyzed alternatives with a broader scope: Assessing commercial alternatives to SLS and Orion for a human landing on the Moon. Each of the studies assumes the use of SLS and the Orion capsule in order to conduct the required mission. A HEOMD official told us that they did not assess commercial alternatives to SLS and Orion because commercial alternatives are not available. If commercial technology to replace SLS and Orion becomes available, the official said NASA can on- ramp those options if SLS and Orion are not delivered on time. Assessing how a more capable SLS could have affected the lunar architecture. NASA did not assess whether refocusing investment on more capable versions of its current programs, including SLS, might affect risk, cost, and schedule for a lunar landing mission. For example, developing a more capable SLS earlier may have enabled NASA to propose a different lunar lander design or to launch components of the architecture in fewer launches. In the study on the design of its human lunar landers, NASA assumed that a more capable version of SLS would not be available until at least 2028, and therefore did not assess using it as a part of its architecture. Further, at the time of the study in 2018, NASA was unsure it would have enough SLS core stages available to utilize them for any components of the architecture other than to transport crew. Identifying alternatives to a lunar landing without using Gateway. All of the studies assumed the use of Gateway or similar capability as opposed to a capability that would take astronauts directly to the lunar surface. A HEOMD official told us that NASA did not assess architectures without Gateway because they planned to utilize SLS and Orion, and NASA did not design the Orion capsule for a direct-to- moon landing. However, a HEOMD official provided us with a quick turn-around analysis that NASA conducted in 2019, after NASA initiated the Gateway program, in response to questions about alternative lunar architectures. This analysis compared a lunar landing from Gateway to a landing without Gateway and found that NASA would have to upgrade the Orion Capsule to have a direct-to-moon landing, which would increase the cost and development time of the program. As a result, the analysis concluded that a lunar landing using Gateway was the superior option. Additionally, officials said Gateway helped develop an architecture that was sustainable and could contribute to a mission to Mars. In addition, only one of the studies focused on a lunar landing mission because NASA completed most of the studies prior to the December 2017 Space Policy Directive-1. NASA officials stated that this is because they were told not to analyze a lunar landing during the previous administration. As a result, none of these studies represents a comprehensive assessment for NASA’s current plans to return to the Moon and are, in total, missing information on potential alternatives. While conducting a formal analysis of alternatives for the lunar architecture is no longer viable given NASA’s schedule, by not having such an analysis NASA is ill-equipped to consider other alternatives as off-ramps if the current lunar architecture plans run into delays. Further, none of the studies contained a life-cycle cost estimate and without this, NASA does not know the costs of its architecture or of potential alternatives. In October 2019, NASA officials stated they had begun to develop an Architecture Campaign Document, which would provide a summary of the studies and analyses that have informed NASA’s lunar architecture. However, this document was still in draft form at the time of our review and officials did not commit to a completion date. Until NASA completes this summary, it will not have a cohesive document outlining the rationale for how it selected its current lunar architecture and lunar programs. Lastly, the practice of formally assessing alternatives is a beneficial practice for future architectures and programs. However, NASA policy and guidance describe an analysis of alternatives as a tool, but does not require officials to analyze alternatives prior to starting work to develop a system architecture or initiating directed missions. NASA may analyze alternatives for an architecture, program, project, or specific design or capability, but conducting a formal analysis of alternatives is optional. Without a requirement to conduct an analysis of alternatives prior to NASA authorizing the initial planning of a program, NASA could miss opportunities to move forward with a more viable architecture or program to meet mission needs in the future. For a new architecture or large programs that require a lot of investment, such as future exploration efforts including Mars, conducting an analysis of alternatives would better position NASA to build a sound business case, justify and document its decisions, and advocate for its plans. Effectively executing the Artemis III mission will require extensive coordination within NASA and its commercial partners, and for each individual program to meet aggressive development time frames. As NASA continues to develop its architecture and program schedules, it will be important that the agency use program management tools and practices to set these new programs up for success. Ensuring that NASA identifies points in time to conduct synchronization reviews, that the role of the proposed Lunar Exploration Control Board in these reviews is understood, and that programs are prepared with the necessary information to make the reviews successful will help NASA mitigate the risk of discovering integration challenges across the lunar programs. The reviews could be a helpful checkpoint on the agency’s progress towards meeting the aggressive timeline of the Artemis III mission. Further, ensuring that the Gateway program has an integrated schedule early on will help the program plan work to meet critical deadlines and avoid unnecessary rework due to the misalignment of requirements or design changes. To date, NASA has provided decision makers with limited cost information to inform decisions on the overall lunar investment. Without an overall cost estimate for the Artemis III mission, NASA is asking Congress to appropriate additional funding to meet a 2024 lunar deadline without having information available on how much it will cost in total to support such plans. Further, NASA senior leadership made a decision that resulted in limiting information regarding the probability of the Gateway program meeting cost and schedule estimates to support the 2024 lunar landing. Requiring the program to conduct a joint cost and schedule confidence level analysis would help to determine whether NASA can meet its lunar goal and whether it has resources to be able to do so. NASA will continue to have many stakeholders interested in its human space exploration plans, which requires NASA to establish a lunar architecture and programs that the agency can defend over time and to demonstrate that it has a solid business case. However, NASA is ill- positioned to explain how it arrived at its current lunar architecture without a comprehensive assessment that documents how NASA decided that its current plans are the best way to meet the agency’s long-term lunar exploration goals. NASA has taken a positive step by planning to create a summary of the studies and analyses that informed its lunar architecture, but has not committed to a date to finalize it. Finally, ensuring that NASA conducts a formal analysis of alternatives for future strategic missions and architectures, including as it further develops its plans for a human mission to Mars, will better position the agency to consistently and reliably select alternatives that best meet the mission need. We are making the following six recommendations to NASA. The NASA Administrator should ensure that the NASA Associate Administrator for Human Exploration and Operations directs the Advanced Exploration Systems division to define and determine a schedule for synchronization reviews, including the role of the proposed Lunar Exploration Control Board, to help ensure that requirements between mission and program levels are reconciled. (Recommendation 1) The NASA Administrator should ensure that the NASA Associate Administrator for Human Exploration and Operations directs the Gateway program to conduct a joint cost and schedule confidence level at the program level for the Artemis III mission. (Recommendation 2) The NASA Administrator should ensure that the NASA Associate Administrator for Human Exploration and Operations directs the Gateway program to update its overall schedule for 2024 to add a KDP II to occur before system integration. (Recommendation 3) The NASA Administrator should ensure that the NASA Associate Administrator for Human Exploration and Operations creates a life-cycle cost estimate for the Artemis III mission. (Recommendation 4) The NASA Administrator should ensure that the NASA Associate Administrator for Human Exploration and Operations directs the Advanced Exploration Systems division to commit to a completion date and finalize a cohesive document outlining the rationale for selecting its current lunar architecture and lunar programs. (Recommendation 5) The NASA Administrator should ensure that the Office of the Chief Engineer determines under what conditions it is appropriate to complete an analysis of alternatives, particularly when there are multiple pathways—including architectures or programs—that NASA could pursue in the future, and document the justification for not completing an analysis. (Recommendation 6) We provided a draft of this report to NASA for comment. In written comments, NASA agreed with our six recommendations. NASA provided estimated dates of completion for all of the recommendations ranging from April 2020 to September 2021. The comments are reprinted in appendix I. NASA also provided technical comments, which have been addressed in the report, as appropriate. We are sending copies of this report to the NASA Administrator and interested congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II. Cristina T. Chaplain, (202) 512-4841 or chaplainc@gao.gov. In addition to the contact named above, Molly Traci, Assistant Director; Katie Bassion; Lorraine Ettaro; Laura Greifner; Anna Irvine, Erin Kennedy; Jason Lee, Assistant Director; Jennifer Leotta, Assistant Director; Ryan Lester; Dennis Mayo; Sylvia Schatz; Roxanna Sun; Jay Tallon, Assistant Director; Alyssa Weir; and Tonya Woodbury made significant contributions to this report.", "summary": "In March 2019, the White House directed NASA to accelerate its plans to return humans to the moon by 4 years, to 2024. To accomplish a lunar landing, NASA is developing programs including a small platform in lunar orbit, known as Gateway, and a lunar lander. NASA plans to use the Space Launch System and Orion crew capsule—two programs with a history of cost growth and schedule delays—to launch and transport crew to Gateway. The House Committee on Appropriations included a provision in its 2018 report for GAO to review NASA's proposed lunar-focused programs, including the Gateway program. GAO's report assesses (1) how NASA updated its lunar plans to support the accelerated 2024 landing timeline; (2) the extent to which NASA has made initial decisions about requirements, cost, and schedule for its lunar mission and programs; and (3) the extent to which NASA analyzed alternatives for its lunar plans, including the Gateway program. GAO analyzed NASA lunar mission and program documents, assessed NASA studies that informed NASA's lunar plans, and interviewed NASA officials. To support accelerated plans to land astronauts on the moon by 2024—four years earlier than planned—the National Aeronautics and Space Administration (NASA) quickly refocused its acquisition plans. In particular, NASA separated its lunar plans into two phases, with the first phase focused on the systems NASA identified to support the new timeline (see figure). One system, Gateway, includes three components—power and propulsion, habitation, and logistics—to form a small platform in lunar orbit. NASA has begun making decisions related to requirements, cost, and schedule for programs, but is behind in taking these steps for the whole lunar mission: NASA risks the discovery of integration challenges and needed changes late in the development process because it established some requirements for individual lunar programs before finalizing requirements for the overall lunar mission. NASA plans to take steps to mitigate this risk, such as by holding reviews to ensure that requirements align across programs, but has not yet defined these reviews or determined when they would occur. NASA has made some decisions that will increase visibility into the costs and schedules for individual lunar programs, but does not plan to develop a cost estimate for the first mission. Cost estimates provide management with critical cost-risk information to improve control of resources. Without a cost estimate for this mission, Congress will not have insight into affordability and NASA will not have insight into monitoring total mission costs. NASA conducted studies to inform its lunar plans, but did not fully assess a range of alternatives to these plans. GAO best practices state that analyzing alternatives provides a framework to help ensure that entities consistently and reliably select the alternative that best meets the mission need and justify agency decisions. Given NASA's schedule, conducting this analysis is no longer viable. Instead, NASA intends to create a summary of the studies that informed its lunar plans. However, it has not committed to a completion date. Without a documented rationale, NASA is ill-positioned to effectively communicate its decisions to stakeholders and facilitate a better understanding of its plans. GAO is making a total of 6 recommendations to NASA, including to define and schedule reviews that align requirements across lunar programs; create a cost estimate for the first lunar mission; and commit to a completion date and finalize a cohesive document outlining the rationale for selecting its current lunar plans. NASA concurred with the recommendations made in this report.", "document_type": "gao"}
{"report": "We reported in August 2018 that the total number of NIV applications that consular officers adjudicated (NIV adjudications) annually peaked at about 13.4 million in fiscal year 2016, which was an increase of approximately 30 percent since fiscal year 2012. In fiscal year 2017 (the most recent data available at the time of our report), NIV adjudications decreased by about 880,000 adjudications, or about 7 percent. Figure 2 shows the number of applications adjudicated each year from fiscal years 2012 through 2017. As shown in figure 2, the percentage of NIVs refused—known as the refusal rate—increased from fiscal years 2012 through 2016, and was about the same in fiscal year 2017 as the previous year. The NIV refusal rate rose from about 14 percent in fiscal year 2012 to about 22 percent in fiscal year 2016, and remained about the same in fiscal year 2017; averaging about 18 percent over the time period. The total number of NIVs issued peaked in fiscal year 2015 at about 10.89 million, before falling in fiscal years 2016 and 2017 to 10.38 million and 9.68 million, respectively. According to State data, while the majority of NIV refusals from fiscal years 2012 through 2017 were a result of consular officers finding the applicants ineligible, a relatively small number of refusals were due to terrorism and other security-related concerns. State data indicate that more than 90 percent of NIVs refused each year from fiscal years 2012 through 2017 were based on the consular officers’ determination that the applicants were ineligible nonimmigrants—in other words, the consular officers believed that the applicant was an intending immigrant seeking to stay permanently in the United States, which would generally violate NIV conditions, or that the applicant otherwise failed to demonstrate eligibility for the particular visa he or she was seeking. For example, an applicant applying for a student visa could be refused as an ineligible nonimmigrant for failure to demonstrate possession of sufficient funds to cover his or her educational expenses, as required. As we reported in August 2018, our analysis of State data indicates that relatively few applicants— approximately 0.05 percent—were refused for terrorism and other security-related reasons from fiscal years 2012 through 2017. As shown in figure 3, in fiscal year 2017, State data indicate that 1,256 refusals (or 0.05 percent) were based on terrorism and other security-related concerns, of which 357 refusals were specifically for terrorism-related reasons. The President issued Executive Order 13769, Protecting the Nation from Foreign Terrorist Entry Into the United States (EO-1), in January 2017. In March 2017, the President revoked and replaced EO-1 with the issuance of Executive Order 13780 (EO-2), which had the same title as EO-1. Among other things, EO-2 suspended entry of certain foreign nationals for a 90-day period, subject to exceptions and waivers. In September 2017, as a result of the reviews undertaken pursuant to EO-2, the President issued Presidential Proclamation 9645, Enhancing Vetting Capabilities and Processes for Detecting Attempted Entry into the United States by Terrorists or Other Public-Safety Threats (Proclamation), which imposes certain conditional restrictions and limitations on the entry of nationals of eight countries—Chad, Iran, Libya, North Korea, Somalia, Syria, Venezuela and Yemen—into the United States for an indefinite period. These restrictions, identified in table 1, are to remain in effect until the Secretaries of Homeland Security and State determine that a country provides sufficient information for the United States to assess adequately whether its nationals pose a security or safety threat. Challenges to both EOs and the Proclamation affected their implementation and, while EO-2’s entry restrictions have expired, the indefinite visa entry restrictions outlined in the Proclamation continued to be fully implemented as of our August 2018 report. We reported in August 2018 that our analysis of State data indicates that out of the nearly 2.8 million NIV applications refused in fiscal year 2017, 1,338 were refused due to visa entry restrictions implemented in accordance with the executive actions. To implement the entry restrictions, in March 2017, State directed its consular officers to continue to accept all NIV applications and determine whether the applicant was otherwise eligible for a visa without regard to the applicable EO or Proclamation. If the applicant was ineligible for the visa on grounds unrelated to the executive action, such as having prior immigration violations, the applicant was to be refused on those grounds. If the applicant was otherwise eligible for the visa, but fell within the scope of the nationality-specific visa restrictions implemented pursuant to the applicable EO or Proclamation and was not eligible for a waiver or exception, the consular officer was to refuse the visa and enter a refusal code into State’s NIV database indicating that the applicant was refused solely due to the executive actions. More than 90 percent of the NIV applications refused in fiscal year 2017 pursuant to an executive action were for tourist and business visitor visas, and more than 5 percent were for students and exchange visitors. As we reported in January 2017, CBP electronically vets all travelers before they board U.S.-bound flights, and continues to do so until they land at a U.S. port of entry. Through these vetting efforts, CBP seeks to identify high-risk travelers from the millions of individuals who travel to the United States each year. As we reported in January 2017, CBP’s vetting and targeting efforts are primarily conducted by its NTC and entail (1) traveler data matching and analysis, (2) rules-based targeting, and (3) recurrent vetting. Specifically: CBP’s primary method of identifying high-risk individuals is through the comparison of travelers’ information (such as name, date of birth, and gender) against records extracted from U.S. government databases, including the Terrorist Screening Database (TSDB)—the U.S. government’s consolidated terrorist watch list. Traveler data matching focuses on identifying known high-risk individuals—that is, individuals who may be inadmissible to the United States under U.S. immigration law or who may otherwise pose a threat to homeland or national security. CBP’s primary tool for vetting and targeting travelers is the Automated Targeting System (ATS), which is a computer-based enforcement and support system that compares traveler information against intelligence and law enforcement data to identify high-risk travelers. Traveler data matching occurs throughout the travel process and, upon a positive or possible match, CBP officers can select these individuals for further vetting, interviewing, and inspection. CBP’s rules-based targeting efforts seek to identify unknown high-risk travelers—that is, travelers for whom U.S. government entities do not have available derogatory information directly linking them to terrorist activities or any other actions that would make them potentially inadmissible to the United States but who may present a threat and thus warrant additional scrutiny. CBP identifies unknown high-risk individuals by comparing their information against a set of targeting rules based on intelligence, law enforcement, and other information. NTC officials stated that these rules have identified potential high-risk travelers, including potential foreign fighters. Rules-based targeting evaluates travelers during the travel process and, in some cases, in advance of the travel process. If a traveler is a rule “hit,” this individual can be selected for further vetting, interviewing, and inspection. CBP supports its traveler data matching and rules-based targeting efforts through the use of recurrent vetting. NTC’s vetting, targeting, and traveler data matching activities in ATS run 24 hours a day and 7 days a week and automatically scan updated traveler information, when available. This process is to ensure that new information that affects a traveler’s admissibility is identified in near real time. Recurrent vetting occurs throughout the travel process and continues until a traveler arrives at a domestic port of entry. For example, after checking into a foreign airport, a traveler may have his or her visa revoked for a security or immigration-related violation. Due to recurrent vetting, CBP would be alerted to this through ATS and could take action, as appropriate. As we reported in January 2017, throughout the travel process, CBP’s predeparture programs use the results of NTC’s efforts to identify and interdict high-risk individuals destined for the United States while they are still overseas; however, we found that CBP had not evaluated the effectiveness of its predeparture programs as a whole, including implementing a system of performance measures and baselines to assess whether the programs are achieving their stated goals. CBP operates three air predeparture programs that are responsible for all U.S.-bound air travelers—(1) Preclearance; (2) the Immigration Advisory Program (IAP) and Joint Security Program (JSP); and (3) the regional carrier liaison groups (RCLG). As we reported in January 2017, CBP data indicated that these programs identified and ultimately interdicted approximately 22,000 high-risk air travelers in fiscal year 2015, the most recent data available at the time of our review. Information on individuals who the NTC identifies through traveler data matching or rules-based targeting, including recurrent vetting, is compiled automatically through ATS into a daily high-priority list (or, traveler referral list). CBP officers at the NTC review the traveler referral list for accuracy and to remove, if possible, any automatically generated matches determined to not be potential high-risk individuals. After this review, CBP officers at the NTC use ATS to send the traveler referral list to officers at each Preclearance, IAP, JSP, and RCLG location, as shown in figure 4. Preclearance. Preclearance locations operate at foreign airports and serve as U.S. ports of entry. Preclearance operations began in 1952 in Toronto to facilitate trade and travel between the United States and Canada. As of March 2018, CBP operated 15 air Preclearance locations in six countries. Through the Preclearance program, uniformed CBP officers at a foreign airport exercise U.S. legal authorities to inspect travelers and luggage and make admissibility determinations prior to an individual boarding a plane to the United States. According to CBP officials, an inspection at a Preclearance location is the same inspection as an individual would undergo at a domestic port of entry, and officers conducting Preclearance inspections exercise the same authority as officers at domestic ports of entry to approve or deny admission into the United States. As a result, travelers arriving at domestic air ports of entry from Preclearance locations do not have to be re-inspected upon entry. According to CBP data, in fiscal year 2015, CBP officers at Preclearance locations determined that 10,648 air travelers were inadmissible out of the approximately 16 million air travelers seeking admission to the United States through a Preclearance location. In addition to requiring that all travelers undergo a primary inspection, CBP officers in these locations also referred almost 290,000 individuals for secondary inspection. Immigration Advisory Program (IAP) and Joint Security Program (JSP). IAP and JSP operated at nine and two foreign airports, respectively, as of January 2017. According to CBP officials, under this program, unarmed, plainclothes CBP officers posted at foreign airports partner with air carriers and host country government officials to help prevent terrorists and other high-risk individuals from boarding U.S.- bound flights by vetting and interviewing them before travel. According to CBP program documentation, CBP established IAP in 2004 to prevent terrorists, high- risk travelers, and improperly documented travelers from boarding airlines destined to the United States. Building on the IAP concept, CBP established JSP in 2009 to partner with host country law enforcement officials to identify high-risk travelers. CBP officers at IAP and JSP locations have the ability to question travelers and review their travel documents. They are to act in an advisory manner to the air carriers and host governments and do not have authority to deny boarding to individuals on U.S.-bound flights or fully inspect travelers or their belongings. IAP and JSP officers are authorized by CBP to make recommendations to airlines as to whether to board or deny boarding (known as a no-board recommendation) to selected travelers based on their likely admissibility status upon arrival to the United States. The final decision to board travelers, however, lies with the carriers. According to CBP data, CBP officers at IAP and JSP locations made 3,925 no-board recommendations in fiscal year 2015 for the approximately 29 million air travelers bound for the United States from such locations. During this same time period, CBP data indicated 1,154 confirmed encounters with individuals in the TSDB, including 106 on the No Fly List. Regional Carrier Liaison Groups (RCLG). RCLGs are located and operate at three domestic airports—Miami International Airport, John F. Kennedy International Airport, and Honolulu International Airport. CBP established RCLGs in 2006 to assist air carriers with questions regarding U.S. admissibility requirements and travel document authenticity. According to CBP officials, RCLGs are responsible for coordinating with air carriers on all actionable referrals from NTC on U.S.-bound travelers departing from an airport without an IAP, JSP, or Preclearance presence. Each RCLG is assigned responsibility for travelers departing out of a specific geographic location. Similar to IAP and JSP, CBP officers in RCLGs also make no-board recommendations, as appropriate, to air carriers. CBP officers at RCLGs do not have authority to make admissibility determinations about U.S.-bound air travelers, and the final decision to board or not board a traveler lies with the carrier. We reported in January 2017 that CBP officers working at the three RCLGs made 7,664 no-board recommendations in fiscal year 2015 for the approximately 59 million travelers bound for the United States from locations within the RCLGs’ spheres of responsibility. During this time period, CBP data indicated that RCLGs also reported 1,634 confirmed encounters with individuals in the TSDB, including 119 on the No Fly List. In January 2017, we reported that CBP had not evaluated the effectiveness of its predeparture programs as a whole, including implementing a system of performance measures and baselines to assess whether the programs were achieving their stated goals. We reported that CBP had taken some initial steps to measure the performance of these programs. Specifically, CBP officials told us that they had collected a large quantity of data and statistics regarding the actions of their predeparture programs and had done so since program inception for all programs. However, due to changes in operational focus, technology updates, and the use of separate data systems at program locations, CBP had not collected consistent data across all of its predeparture programs. As a result, CBP did not have baseline data on which to measure program performance. Therefore, we recommended that CBP develop and implement a system of performance measures and baselines for each program to help ensure that these programs achieve their intended goals. In response, as of March 2018, CBP has developed three performance measures for its predeparture programs. On the basis of our review of CBP documentation, as of December 2018, CBP has collected the fiscal year 2018 data relevant to these measures, used those data to set preliminary targets for fiscal year 2019, and plans to analyze the fiscal year 2019 results and set targets for future fiscal years by October 31, 2019. We will review documentation of CBP’s analysis of the fiscal year 2019 results and future targets, when available, to determine if CBP’s actions address our recommendation. Chairwoman Rice, Chairman Rose, Ranking Members Higgins and Walker, and Members of the Subcommittees, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. For further information regarding this testimony, please contact Rebecca Gambler at (202) 512-8777 or gamblerr@gao.gov. In addition, contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony are Kathryn H. Bernet, Assistant Director; Eric Hauswirth; Thomas Lombardi; Sasan J. “Jon” Najmi; Erin O’Brien; and Natalie Swabb. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "Previous attempted and successful terrorist attacks against the United States have raised questions about the security of the U.S. government's screening and vetting processes for NIVs. State manages the visa adjudication process. DHS seeks to identify and interdict travelers who are potential security threats to the United States, such as foreign fighters and potential terrorists, human traffickers, drug smugglers and otherwise inadmissible persons, at the earliest possible point in time. DHS also has certain responsibilities for strengthening the security of the visa process. In 2017, the President issued executive actions directing agencies to improve visa screening and vetting, and establishing nationality-based visa entry restrictions, which the Supreme Court upheld in June 2018. This statement addresses (1) data and information on NIV adjudications and (2) CBP programs aimed at preventing high-risk travelers from boarding U.S.-bound flights. This statement is based on prior products GAO issued in January 2017 and August 2018, along with selected updates conducted in December 2018 to obtain information from DHS on actions it has taken to address a prior GAO recommendation. In August 2018, GAO reported that the total number of nonimmigrant visa (NIV) applications that Department of State (State) consular officers adjudicated annually increased from fiscal years 2012 through 2016, but decreased in fiscal year 2017 (the most recent data available at the time of GAO's report). NIVs are issued to foreign nationals, such as tourists, business visitors, and students, seeking temporary admission into the United States. The number of adjudications peaked at about 13.4 million in fiscal year 2016, and decreased by about 880,000 adjudications in fiscal year 2017. State refused about 18 percent of adjudicated applications during this time period, of which more than 90 percent were because the applicant did not qualify for the visa sought and 0.05 percent were due to terrorism and security-related concerns. In 2017, two executive orders and a proclamation issued by the President required, among other actions, visa entry restrictions for nationals of certain listed countries of concern. GAO's analysis indicates that, out of the nearly 2.8 million NIV applications refused in fiscal year 2017, 1,338 applications were refused specifically due to visa entry restrictions implemented per the executive actions. In January 2017, GAO reported that the Department of Homeland Security's (DHS) U.S. Customs and Border Protection (CBP) operates predeparture programs to help identify and interdict high-risk travelers before they board U.S.- bound flights. CBP officers inspect all U.S.-bound travelers on those flights that are precleared at the 15 Preclearance locations at foreign airports—which serve as U.S. ports of entry—and, if deemed inadmissible, a traveler will not be permitted to board the aircraft. CBP also operates nine Immigration Advisory Program and two Joint Security Program locations, as well as three Regional Carrier Liaison Groups, through which CBP may recommend that air carriers not permit identified high-risk travelers to board U.S.-bound flights. CBP data showed that it identified and interdicted over 22,000 high-risk air travelers through these programs in fiscal year 2015 (the most recent data available at the time of GAO's report). While CBP tracked some data, such as the number of travelers deemed inadmissible, it had not fully evaluated the overall effectiveness of these programs. GAO recommended that CBP develop a system of performance measures and baselines to better position CBP to assess program performance. As of December 2018, CBP set preliminary performance targets for fiscal year 2019, and plans to set targets for future fiscal years by October 31, 2019. GAO will continue to review CBP's actions to address this recommendation. GAO previously recommended that CBP evaluate the effectiveness of its predeparture programs. DHS agreed with GAO's recommendation and CBP has actions under way to address it.", "document_type": "gao"}
{"report": "VA’s process for deciding veterans’ eligibility for disability compensation begins when a veteran submits a claim to VA. Staff in one of VBA’s 57 regional offices assist the veteran by gathering additional evidence, such as military and medical records, that is needed to evaluate the claim. Based on this evidence, VBA decides whether the veteran is entitled to compensation and, if so, how much. A veteran dissatisfied with the initial claim decision can generally appeal within 1 year from the date of the notification letter sent by VBA. Under the current appeals process (now referred to by VA as the legacy process), an appeal begins with the veteran filing a Notice of Disagreement. VBA then re-examines the case and generally issues a Statement of the Case that represents its decision. A veteran dissatisfied with VBA’s decision can file an appeal with the Board. In filing that appeal, the veteran can indicate whether a Board hearing is desired. Before the Board reviews the appeal, VBA prepares the file and certifies it as ready for Board review. If the veteran requests a hearing to present new evidence or arguments, the Board will hold a hearing by videoconference or at a local VBA regional office. The Board reviews the evidence and either issues a decision to grant or deny the veteran’s appeal or refers the appeal back to VBA for further work. According to VA’s appeals plan, VA intends to implement the Act by February 2019, by replacing the current appeals process with a process offering veterans who are dissatisfied with VBA’s decision on their claim five options. Two of those options afford the veteran an opportunity for an additional review of VBA’s decision within VBA, and the other three options afford them the opportunity to bypass additional VBA review and appeal directly to the Board. Under the new appeals process, the two VBA options will be: 1. Request higher-level review: The veteran asks VBA to review its initial decision based on the same evidence but with a higher-level official reviewing and issuing a new decision. 2. File supplemental claim: The veteran provides additional evidence and files a supplemental claim with VBA for a new decision on the claim. The veteran can also request a VBA hearing. The three Board options will be: 3. Request Board review of existing record: The veteran appeals to the Board and asks it to review only the existing record without a hearing. 4. Request Board review of additional evidence, without a hearing. 5. Request Board review of additional evidence, with a hearing. In November 2017, VA initiated a test of the new VBA higher-level review and supplemental claim options. According to VA’s appeals plan, a purpose of this test—the Rapid Appeals Modernization Program (RAMP)—is to reduce legacy appeals by providing veterans with a chance for early resolution of their claims within VBA’s new process. Participation in RAMP is voluntary, but veterans must withdraw their pending legacy appeal to participate, according to VA’s appeals plan. In our March 2018 report, we found that VA’s November 2017 plan for implementing a new disability appeals process while attending to appeals under way in the current (legacy) process, addressed 17 of 22 elements required by the Act. For the 5 remaining elements, we found that it partially addressed 4 elements related to implementation monitoring, productivity projecting, and workforce planning, and did not address 1 element related to identifying total resources. This element called for delineating the resources needed by VBA and the Board to implement the new appeals process and address legacy appeals. We recommended in March 2018 that VA address all 22 required elements in the Act in VA’s appeals plan to Congress—including delineating resources required for all VBA and Board appeals options— using sensitivity analyses and results from its test, RAMP, where appropriate and needed. Since our March 2018 report, VA has taken some action on each of the five elements that we found were not fully addressed at that time. For example, VA added details related to projecting staff productivity, identifying total resources, as well as determining personnel requirements and productivity projections for processing appeals. For identifying total resources, VA added FTE information for other offices that help implement the appeals process and prepared a model to project resource needs. Although VA now addresses the 1 element related to projecting productivity, it only partially addresses 4 elements related to monitoring implementation, workforce planning, and delineating the total resources. For example, as of November 2018, VA’s plan does not contain metrics for monitoring implementation. Moreover, for total resources, the updated plan does not delineate the total resources required by VBA and the Board, such as the resources necessary for information technology and training. We acknowledge that in some cases delineating total resources could prove challenging, such as delineating information technology resources for the legacy and new appeals processes. We also acknowledge that implementing corrective actions to fully address these 4 elements may be challenging within the next several weeks, but we continue to believe VA has an opportunity to further address these 4 elements as part of certifying the agency’s readiness prior to the full implementation of the new process. In our March 2018 report, we found gaps in VA’s planning for how it will monitor and assess performance of the new appeals process when it is implemented. Specifically, we reported that the plan did not (1) establish timeliness goals for two of the three Board options (i.e., Board review of additional evidence without a hearing and Board review of additional evidence with a hearing); (2) articulate aspects of performance important for managing appeals, such as accuracy of decisions, veteran satisfaction with the process, or cost; (3) explain how the performance of the new appeals process would be compared to that of the legacy process; or (4) explain how the agency would monitor relative workloads of, and resources devoted to, the new and legacy appeals processes. To address these gaps, we recommended that VA clearly articulate in its appeals plan how VA will monitor and assess the new appeals process compared to the legacy process, including specifying a balanced set of goals and measures—such as timeliness goals for all VBA appeals options and Board dockets, and measures of accuracy, veteran satisfaction, and cost—and related baseline data. Articulating a balanced set of goals that cover key aspects of managing appeals is important to avoid promoting skewed behaviors (e.g., favoring timeliness over accuracy) and to fully understanding performance. In its progress reports, VA addressed some but not all aspects of this recommendation (see table 1). VA has made progress in monitoring performance and addressing workload changes in its new and legacy appeals processes, but still lacks a complete set of balanced goals and measures. As we noted in our July 2018 testimony, VA has developed sensitivity models and other analyses to monitor and forecast future VBA and Board workloads, production, and staffing requirements to help VA manage the legacy and new appeals processes. However, VBA and the Board have yet to specify a complete set of balanced goals for monitoring the performance of the new appeals processes. According to the November 2018 progress report, the Board plans to develop timeliness goals after VA fully implements the new appeals process. Until VA fully develops a set of balanced goals and measures, the agency risks not fully understanding how well the reforms are performing. Regarding comparing the performance of the new and legacy appeals processes, VA has previously reported that the agency plans to implement the reporting requirements in section 5 of the Act. This section requires VA to report performance measures related to, among other things, timeliness, productivity, and outcomes, without specifying whether or how VA should compare performance of the new versus legacy processes. In November 2018, VBA and Board officials told us they intend to use timeliness and productivity metrics from section 5 to compare the two processes. However, in its updated plans to date, VA has been reporting average timeliness of decisions made to date under RAMP—VA’s test of the two VBA options—without reporting the average time cases are pending. Moreover, VA has not been reporting timeliness data on both decisions and pending cases according to the month that they entered into RAMP, which present a more balanced indication of performance and trends. In November 2018 VBA and Board officials told us they would consider reporting timeliness using a monthly cohort that reflects when appeals were filed. VBA and Board officials also said they have taken steps to collect, through surveys, comparable information on veterans’ satisfaction with the new and legacy appeals processes. According to VBA and Board officials, they have pre-tested the surveys—which is considered a best practice by survey methodologists—and are coordinating the survey efforts with one another. VBA and Board officials also told us that the agency will report on accuracy and outcomes (grants and denials of claims) in the new process. However, they also stated that these measures would not provide a fair comparison with the legacy process because the Act eliminated several of the requirements formerly required in the legacy appeals administrative processes. Although VA officials said they would develop a plan for comparing the performance of the two appeals processes after the new process is fully implemented, they did not indicate how soon they would do so. Developing such a plan would better position the agency to fully understand whether the new process is an improvement. Our March 2018 report identified elements of a high-quality and reliable implementation schedule that were missing from VA’s master schedule for appeals reform. Specifically, we reported that VA’s high-level master schedule—which the agency included with its November 2017 plan—did not (1) include all key activities; (2) show which activities must finish prior to the start of other activities, or the amount of time an activity could be delayed before the delay affects VA’s estimated implementation date; (3) reflect interim goals and milestones for monitoring implementation; or (4) assign resources for activities. We recommended that VA augment the master schedule for its appeals plan to reflect all activities—such as modifications to information technology systems—as well as assigned responsibilities, interdependencies, start and end dates for key activities for each workgroup, and resources. These steps establish accountability and reduce overall risk of implementation failures. In response to our recommendation, the Board, VBA and other VA administrations made progress over time with developing and integrating underlying plans into the integrated master schedule (IMS) in spring and summer 2018. According to VA officials, VA set a baseline schedule for implementing appeals reform in response to the potential February 2019 implementation date established in the Act. Since November 2017, VA’s plan and progress reports have stated that VA uses an agency-wide governance structure to coordinate implementation, and regularly uses the schedule as a management tool for monitoring progress on appeals reform. For example, the Board’s project manager meets regularly with those responsible for major activities to check progress, including weekly meetings with leadership, and identifies and corrects issues related to schedule execution. In October 2018, VA provided us with lower-level schedules and information that allowed us to conduct a more detailed assessment of VA’s IMS against applicable best practices criteria. The six criteria we assessed lower-level schedules against were: Capturing all activities: schedule should reflect all activities necessary to perform work to accomplish a project’s objective. Sequencing activities: activities should be logically sequenced in the order they are to be carried out so that critical program dates can be met. Assigning resources: schedule should reflect all resources necessary to complete work, verify whether resources will be available, and identify any constraints. Verifying horizontal and vertical traceability: schedule should be rational and logically sequenced, account for interdependencies among activities, and provide a way to evaluate the current status (horizontal traceability). Also, the various levels of a schedule— summary, intermediate, and detailed—should be consistent with one another and enable different teams to work to the same schedule expectations (vertical traceability). Updating the schedule using actual progress and logic: maintain and continually update the schedule to reflect a realistic forecast of start and end dates of activities. Maintaining a baseline schedule: use original configuration of the program plan as a point of comparison for the current plan to manage scope, timeframes, and required resources. We found that, while VA has made progress with providing more detail, its master and underlying schedules only minimally met sound practices for project management. Specifically, as with our March 2018 assessment, we found that the schedule does not contain enough detail to manage the work or provide a realistic representation of the resources and time needed for this project. For example, the schedule did not contain a work breakdown structure that defines the work, activities, and resources necessary to accomplish implementation. Moreover, half of all the remaining activities are missing logic that shows which activities must finish prior to the start of other activities. In addition, the schedule contains an invalid critical path, meaning that the schedule does not present the amount of time that key activities could be delayed before such delays affect VA’s estimated implementation date. Without a valid critical path, management cannot focus on activities that will detrimentally affect the key program milestones and deliveries if they slip. To address our March 2018 recommendation, VA would need to ensure that all activities are accounted for, that scheduled activities appear in the correct order, that resources are properly allocated, that all activities appear on the critical path, and that a schedule risk analysis accounts for all risks. We provide a more detailed explanation of our assessment results in appendix I. In addition, establishing an overly optimistic schedule can reduce capacity for carrying out a project and potentially create pressure to sacrifice the quality of work activities to meet deadlines. Moreover, many of VA’s activities are slated to be concurrently completed just before implementation, posing a significant risk to implementing reform in February. For example, according to VA’s schedule, the agency needs to complete 117 activities after January 1, 2019. Further, other VA efforts to redesign or update key aspects of VA’s disability compensation process—including the Veterans Benefits Management System (VBMS)—were not driven by robust, comprehensive planning and did not achieve their schedule goals. While VA intends to start full implementation in February, we do not know the extent to which the lack of a robust schedule poses risks to successful and smooth implementation. Even if taking corrective actions to address our findings may not be feasible before February, incorporating such lessons learned into future project planning could help VA improve its project scheduling capabilities. In our March 2018 report, we found that VA’s appeals plan could more fully assess key risks related to implementing the new appeals process. In particular, we found that VA’s plan did not include testing of new Board options or clearly define how it would assess the RAMP test of the VBA- only options before implementing them more broadly. Further, we reported that VA’s plan had not comprehensively reflected key risks because the agency had not established a complete and balanced set of goals and measures, which are a necessary pre-condition to effectively assessing risk. We recommended that VA ensure that the appeals plan more fully addresses risk associated with appeals reform by, for example, assessing risks against a balanced set of goals and measures, articulating success criteria and an assessment plan for RAMP, and testing or conducting sensitivity analyses of all five appeals options before fully implementing the new appeals process. In its progress reports, VA took many steps to address our recommendation, although key steps are remaining for VA to better assess risks associated with implementing appeals reform and managing appeals workloads in the legacy process (see table 2). Sound redesign and change management practices both suggest that tests be rigorously monitored and evaluated and that further roll-out occur only after an agency takes any needed corrective action and determines that the new process is achieving previously identified success criteria. Until VA takes these remaining steps, it may not have comprehensively addressed key risks to better position the agency for successful implementation of appeals reform. In conclusion, VA is undertaking an ambitious effort to reform its disability appeals process—while onboarding hundreds of new staff and implementing new technology—that will affect the lives of hundreds of thousands of veterans with disabilities for years to come. Consistent with our prior recommendations, VA has made concrete progress to improve its planning for disability appeals reform while it attends to legacy appeals. Efforts such as resuming sensitivity analysis to monitor workloads and testing VBA and Board appeals options will provide useful information to guide VA through the uncertainty often associated with process change. However, VA has reported it plans to fully implement the new disability appeals process in February 2019 even though it has yet to fully address our recommendations. While fully implementing our recommendations prior to February 2019 may not be feasible, doing so would better position VA to ensure successful implementation. Nevertheless, VA should still work to increase clarity around its plans prior to fully implementing reform. Moreover, many of the principles of sound planning practices that informed our recommendations remain relevant during process change. By continuing to improve its approach to performance measurement, scheduling, and risk management, even after implementation, VA could better ensure that the new process meets veterans’ needs. Chairman Roe, Ranking Member Walz, and Members of the Committee, this concludes my prepared statement. I would be pleased to respond to any questions you may have at this time. For further information about this testimony, please contact Elizabeth H. Curda at (202) 512-7215 or curdae@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony. Other key contributors to this testimony include James Whitcomb (Assistant Director), Juaná Collymore, Michele Grgich, Sara Pelton, and Rachel Pittenger. In addition, key support was provided by Susan Aschoff, Mark Bird, Alex Galuten, Jason Lee, Sheila R. McCoy, Almeta Spencer, and Walter Vance. For this testimony, we assessed the steps that the Department of Veterans Affairs (VA) has taken to address our March 2018 recommendations and what aspects remain unaddressed, including the extent to which VA is using sound practices for scheduling key projects. In summary, we identified several areas where VA’s most recent schedule falls short of sound practices. Further incorporating sound practices into future project planning could help VA improve its project scheduling capabilities. We reviewed VA’s integrated master schedule (IMS) for the appeals reform effort and underlying sub-schedules to assess them against 6 of the 10 best practices, which we determined most relevant to our March 2018 recommendation that VA augment its master schedule for VA’s appeals plan to reflect all activities—such as modifications to information technology systems—as well as assigned responsibilities, interdependencies, start and end dates for key activities for each workgroup, and resources, to establish accountability and reduce the overall risk of implementation failures. Specifically, we analyzed the following related scheduling best practices: (1) Capturing all activities, (2) Sequencing all activities, (3) Assigning resources to all activities, (4) Verifying that the schedule can be traced vertically and horizontally, (5) Updating the schedule using actual progress and logic and (6) Maintaining a baseline schedule. We assessed VA’s lower-level schedules against these 6 best practices by: Checking for specific problems that could hinder the schedule’s ability to respond to changes. For example, we: o Examined if there are any open-ended activities (i.e., activities with no predecessor and/or successors), o Searched for activities with poor logic: For example, Start to Start successor only or Finish to Finish predecessor only which represent dangling logic, or Logic on summary tasks rather than attached to detailed tasks (summary tasks are for organizing the schedule and should not drive the logic). o Looked for activities with constraints which keep the schedule rigid (e.g., start no earlier than, finish no later than, etc.), o Determined if activities were resource loaded—which helps to cost out the schedule—and examine whether resources are over- allocated or not available when needed, o Examined the schedule’s critical path to determine whether or not it was reliable and logical, o Examined schedule float and determined if it was reasonable, and o Examined whether the schedule was baselined, its status cycle, and what deviations there were from the original plan. We also determined if there were any actual start or finish dates recorded in the future and whether there was any broken logic between planned tasks. We also interviewed VA officials responsible for managing the schedule. We scored each scheduling leading practice on a five-point scale: “not met”, “minimally met”, “partially met”, “substantially met” and “fully met.” We determined the characteristic assessment rating by assigning each best practice rating a number and taking the average. Our resulting conclusions based on this assessment are as follows: VA’s project schedule minimally meets the best practice of capturing all activities. The schedule does not have well-defined start and finish milestones and there is not a project work breakdown structure (WBS) or corresponding WBS dictionary to define the work for each WBS element. We were not able to independently verify contractor work or major handoffs and deliverables in the schedule. In addition, there were activities with duplicate names, which could make communication difficult between VA teams, particularly between team members who are responsible for updating and integrating multiple schedules. VA’s project schedule minimally meets the best practice of sequencing activities. There are issues with missing dependencies, dangling activities, summary links, constraints and lags that affect the schedule meeting this best practice. Specifically, of the remaining activities, 55 percent have missing logic, over 12 percent are dangling, 42 percent have date constraints and 4 percent have leads assigned. When activities are not correctly linked, the program cannot use the integrated master schedule (IMS) to identify disconnects or hidden opportunities and cannot otherwise promote efficiency and accuracy or control the program by comparing actual to planned progress. When this happens, the schedule will not allow a sufficient understanding of the program as a whole, and users of the schedule may lack confidence in the dates and the critical path. VA’s project schedule minimally meets the best practice of assigning resources. While the schedule contains ‘Task Owner’ assignments, the Task Owner information has no effect on the durations or forecasted start and finish dates of detailed activities. Information on resource needs and availability in each work period assists the program office in forecasting the likelihood that activities will be completed as scheduled. If the current schedule does not allow insight into the current or projected allocation of resources, then the risk of the program’s slipping is significantly increased. VA’s project schedule minimally meets the best practice of verifying the schedule is traceable horizontally and vertically. There was no evidence in the schedule of hand-offs within the schedule—that is givers and receivers are easily identifiable in the schedule. We were unable to determine the relationship between lower-lever activities in the project schedule and higher-level activities and milestones in the management briefs provided to us. Specifically, we could not map the activities in the briefs to activities in the schedule. This inconsistency also prevented the verification of dates between the project schedule and higher-level management documents, even with documents that were provided from the same month as the October schedule. Products and outcomes were not easily traced through the sequencing of effort in the project schedule. In both cases the schedule did not respond appropriately to “shocks”; that is, greatly increasing the durations of some activities to increase the overall time required to complete the project did not affect the dates of key milestones. The duration increase of each activity did not affect the overall time line because the activity in question had a constraint that would not allow the project to appropriately extend. VA’s project schedule minimally meets the best practice of updating the schedule using progress and logic. Date anomalies, such as planned dates in the past or actual dates in the future, were found. The schedule was not current as of the date delivered to GAO. While officials report that they update the schedule regularly, a schedule narrative document does not accompany the schedule update that would detail changes to the current schedule and describe information such as the status of key milestone dates, changes in network logic, and a description of the current critical path(s). VA’s project schedule minimally meets the best practice of maintaining a baseline schedule. Officials said that the baseline schedule is the basis for performance measurement. But while baseline start and baseline finish dates were provided in the initial schedule, its activities were too high level, obfuscating the calculation of detail variances in subsequent schedules. There is also no evidence of a schedule basis document, which would include a general overview of the purpose of the schedule, other key basis information such as an overview of assumptions, rationale for durations specific to the CMR schedule, and required software settings. There is also no evidence of performance measuring. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "VA's disability compensation program pays cash benefits to veterans with disabilities connected to their military service. In recent years, veterans who appealed VA decisions on their claims have waited an average of 3 years. The subset of appeals resolved by the Board of Veterans Appeals—a separate VA agency that provides a higher level of appeals review—took on average 7 years to resolve. The Veterans Appeals Improvement and Modernization Act of 2017 makes changes to VA's current (legacy) process, giving veterans options to have their claims reviewed by VA or to appeal directly to the Board. The Act requires VA to submit to Congress and GAO a plan for implementing a new appeals process (which VA submitted in November 2017) and periodic progress reports (which VA submitted in February, May, August, and November 2018). The Act also includes a provision for GAO to assess VA's original plan. In March 2018, GAO found that VA could help ensure successful implementation of appeals reform by addressing gaps in planning and made four recommendations, with which VA agreed. This testimony focuses on the steps VA has taken to address GAO's recommendations, what aspects remain unaddressed, and risks these gaps pose for implementation. For this statement, GAO reviewed VA's updated plans, assessed VA's schedules against best practices, interviewed VA officials and reviewed information they provided about steps taken to implement GAO's recommendations. In a March 2018 report, GAO made four recommendations to address planning gaps in the Department of Veterans Affairs' (VA) November 2017 plan for changing its appeals process for disability compensation claims. Since then, VA has updated its appeals reform plan and taken steps to address aspects of these recommendations, but further steps could enhance its readiness for implementation: Address all legally required elements . VA's November 2017 plan did not address one and only partially addressed four of 22 elements required by the Veterans Appeals Improvement and Modernization Act of 2017 (Act); GAO recommended VA fully address all 22. As of November 2018, VA addressed one element related to projecting productivity and took steps to partially address the other four. VA is still missing information the agency needs to certify that it has the resources needed to successfully implement appeals reform. Articulate plans for performance monitoring and assessment . GAO recommended VA clearly articulate how it will monitor and assess the new appeals process relative to the legacy process, including, for example, specifying timeliness goals for the five new appeals options, and measures for decision accuracy in processing appeals. As of November 2018, VA officials stated their intention to use productivity, timeliness, accuracy, and veteran satisfaction metrics to assess the new versus the legacy appeals processes. However, VA has yet to specify a complete set of goals or measures for monitoring and assessing the relative efficacy of the new process or articulate detailed steps and timeframes for establishing them. Augment master schedule . GAO recommended VA augment its master schedule for appeals reform to reflect sound practices for guiding implementation of reform. Although VA's updated schedule reflected progress since VA's original 2017 plan, it still did not fully meet sound practices for project management. For example, the schedule does not appropriately define the work, activities, and resources necessary to accomplish appeals reform implementation. Without following sound practices, it is unclear whether the schedule poses risks to successful implementation of appeals reform. Address risk fully . GAO recommended that VA's plan more fully address risks in implementing a new appeals process by, for example, testing all appeals options prior to full implementation. As of November 2018, VA took many steps to address risks, although opportunities exist to better assess them. For example, although VA has used lessons learned from tests to update the implementation process, it has not fully tested all aspects nor has it developed mitigation strategies for all identified risks, such as veterans appealing to the Board at higher rates than expected. Until VA takes these remaining steps, it may not have sufficiently accounted for key risks in implementing the new process.", "document_type": "gao"}
{"report": "IT systems supporting federal agencies are inherently at risk. These systems are highly complex and dynamic, technologically diverse, and often geographically dispersed. This complexity increases the difficulty in identifying, managing, and protecting the numerous operating systems, applications, and devices comprising federal systems and networks. Compounding these risks, federal systems and networks are often interconnected with other internal and external systems and networks, including the internet, thereby increasing the number of avenues of attack and expanding their potential attack surface. Without proper safeguards, computer systems are vulnerable to individuals and groups with malicious intent who can intrude and use their access to obtain sensitive information, commit fraud and identity theft, disrupt operations, or launch attacks against other computer systems and networks. Cyber-based threats to information systems can come from sources internal and external to the organization. Internal threats include errors or mistakes, as well as fraudulent or malevolent acts by employees or contractors working within the organization. External threats include the ever-growing number of cyber-based attacks that can come from a variety of sources such as individuals, groups, and countries that wish to do harm to an organization’s systems. Yet, IT systems are often riddled with security vulnerabilities—both known and unknown. These vulnerabilities can facilitate security incidents and cyberattacks that disrupt critical operations; lead to inappropriate access to and disclosure, modification, or destruction of sensitive information; and threaten national security, economic well-being, and public health and safety. Until fiscal year 2016, the number of information security incidents reported by federal agencies to DHS’s United States Computer Emergency Readiness Team (US-CERT) had steadily increased each year. From fiscal year 2009 through fiscal year 2015, reported incidents increased from 29,999 to 77,183, an increase of 157 percent. Changes to federal incident reporting guidelines for 2016 contributed to the decrease in reported incidents in fiscal year 2016. Specifically, updated incident reporting guidelines that became effective in fiscal year 2016 no longer required agencies to report non-cyber incidents or incidents categorized as scans, probes, and attempted access. More recently, agencies reported 35,277 incidents in fiscal year 2017 and 31,107 incidents in fiscal year 2018, as reflected in figure 1. According to US-CERT incident report data, the incidents reported in fiscal year 2018 involved several threat vectors. These threat vectors include web-based attacks, phishing attacks, and the loss or theft of computer equipment, among others. Figure 2 provides a breakdown of information security incidents by threat vector in fiscal year 2018. These incidents and others like them can pose a serious challenge to national security, economic well-being, and public health and safety, as shown by two incidents reported in fiscal year 2018: In March 2018, the Department of Justice reported that it had indicted nine Iranians for conducting a massive cybersecurity theft campaign on behalf of the Islamic Revolutionary Guard Corps. According to the department, the Iranians allegedly stole more than 31 terabytes of documents and data from more than 140 American universities, 30 U.S. companies, and five federal government agencies, among other entities. In March 2018, a joint alert from DHS and the Federal Bureau of Investigation stated that, since at least March 2016, Russian government actors had targeted U.S. government entities and critical infrastructure sectors, including the energy, nuclear, water, aviation, and critical manufacturing sectors. Congress enacted FISMA 2014 to provide a comprehensive framework for ensuring the effectiveness of information security controls over information resources that support federal operations and assets and to clarify government-wide responsibilities. The act addresses the increasing sophistication of cybersecurity attacks, promotes the use of automated security tools with the ability to continuously monitor and diagnose the security posture of federal agencies, and provides for improved oversight of federal agencies’ information security programs. FISMA requires agencies to develop, document, and implement an agency-wide information security program to secure federal information systems. These information security programs are to provide risk-based protections for the information and information systems that support the operations and assets of the agency. FISMA requires agencies to comply with OMB policies and procedures, DHS binding operational directives, and NIST federal information standards and guidelines. In addition, FISMA assigns to agency inspectors general responsibility for annually assessing the effectiveness of the information security policies, procedures, and practices of the agency. FISMA directs OMB to oversee agencies’ information security policies and practices. Among other things, FISMA requires OMB to develop and oversee the implementation of policies, principles, standards, and guidelines on information security in federal agencies, except with regard to national security systems. The law also assigns OMB the responsibility of requiring agencies to identify and provide information security protections commensurate with assessments of risk to their information and information systems. In addition, FISMA 2014 clarified and expanded DHS’s responsibilities for government-wide information security. Specifically, the act requires DHS, in consultation with OMB, to administer the implementation of agency information security policies and practices for non-national security information systems by: (1) assisting OMB with carrying out its oversight responsibilities; (2) developing, issuing, and overseeing implementation of binding operational directives; and (3) providing operational and technical assistance. Further, FISMA 2002 assigned to NIST the responsibility for developing standards and guidelines that include minimum information security requirements. FISMA also includes reporting requirements. Specifically, OMB is to report annually, in consultation with DHS, on the effectiveness of agency information security policies and practices, including a summary of major agency information security incidents and an assessment of agency compliance with NIST standards. Further, the law requires agencies to report annually to OMB, DHS, certain congressional committees, and the Comptroller General on the adequacy and effectiveness of their information security policies, procedures, and practices, including a description of each major security incident. In May 2017, the President signed Executive Order 13800, which sets policy for managing cybersecurity risk as an executive branch enterprise. Specifically, the order outlines actions to be taken by federal agencies and critical infrastructure sectors to improve the nation’s cybersecurity posture and capabilities. To this end, the order states that the President will hold executive agency heads accountable for managing agency-wide cybersecurity risk and directs each executive branch agency to use the NIST cybersecurity framework to manage those risks. In addition to requirements set in the executive order, OMB’s reporting metrics that were developed to facilitate agencies’ compliance with FISMA’s reporting requirement are aligned to the core functions outlined in the cybersecurity framework. Consequently, agencies are required to report on the effectiveness of their information security policies and practices according to the cybersecurity framework’s core functions. The NIST cybersecurity framework is based on five core security functions: Identify: Develop an understanding of the organization’s ability to manage cybersecurity risk to systems, people, assets, data, and capabilities. Protect: Develop and implement appropriate safeguards to ensure delivery of critical services. Detect: Develop and implement appropriate activities to identify the occurrence of a cybersecurity event. Respond: Develop and implement appropriate activities to take action regarding a detected cybersecurity incident. Recover: Develop and implement appropriate activities to maintain plans for resilience and to restore capabilities or services that were impaired due to a cybersecurity incident. According to NIST, these five functions should be performed concurrently and continuously to address cybersecurity risk. In addition, when considered together, they provide a high-level, strategic view of the life cycle of an organization’s management of cybersecurity risk. Within the five functions, NIST identifies 23 categories and 108 subcategories of activities and controls for achieving the intent of each function. Appendix II provides a description of the cybersecurity framework categories and subcategories of activities and controls. The Council of Inspectors General for Integrity and Efficiency (CIGIE), in collaboration with OMB, DHS, and other stakeholders, developed a capability maturity model for agency inspectors general to assess and report on the effectiveness of their agencies’ information security programs. As described in table 1, the model identifies five maturity levels with each succeeding level representing a more advanced level of implementation. Using the five-level maturity model described above, the inspectors general are to assign a maturity-level rating for each of the five core security functions based on an assessment of their agencies’ implementation of the activities and controls associated with each function using metrics that CIGIE developed in collaboration with OMB. The inspectors general then consider the maturity level ratings of the core security functions to evaluate the overall effectiveness of their agency’s information security program. OMB instructs inspectors general to rate their agency’s information security program as effective or not effective by applying a rule of simple majority. Specifically, if three or more of the five core security functions are rated effective, the overall information security program is considered to be effective. According to this maturity model, Level 4 (managed and measurable) is the lowest level to represent an effective level of security. Therefore, if an inspector general rates three or more of the agency’s core security functions at Level 4 or Level 5, then the inspector general can consider that agency to have an effective information security program. However, the inspector general has the discretion to have a different conclusion on program effectiveness if he or she deems it appropriate to do so. Similar to the inspector general FISMA reporting metrics, OMB and DHS worked with interagency partners to develop the CIO FISMA metrics, which are intended to be used by the agencies, OMB, and DHS to track agencies’ progress in implementing cybersecurity capabilities. The CIO FISMA reporting metrics are organized around the five core security functions outlined in NIST’s cybersecurity framework. In addition, certain CIO FISMA reporting metrics represent key milestones of the administration’s IT Modernization Cross-Agency Priority (CAP) goal, which includes a cybersecurity initiative. As a result, the CIO reporting metrics allow agency CIOs, OMB and DHS to monitor progress toward meeting key milestones and targets for the CAP goal. The cybersecurity initiative within the IT Modernization CAP goal is designed to reduce cybersecurity risks to the federal government’s information systems by mitigating the impact of risks to federal data, systems, and networks. The initiative consists of three strategies that contain 10 milestones that relate to key areas within the CIO FISMA metrics—information security continuous monitoring; identity, credential, and access management; and advanced network and data protections. In addition, each of the 10 milestones has an expected level of performance, or target, for implementation, as described later in this report. Each year, OMB requires agencies to report how much they spend on information security. In fiscal year 2018, the 23 civilian agencies covered by the CFO Act reported spending between $9 million and almost $1.9 billion on cybersecurity- or IT security-related activities. For these 23 agencies, their total reported security spending accounted for about 14 percent of their IT spending, with percentages for individual agencies ranging from 5 percent to 208 percent, as seen in table 2. Information security reports issued by GAO, inspectors general, and CIOs indicate that information security policies and practices of the agencies we reviewed are ineffective. Specifically, information security evaluation reports that we and agency inspectors general issued during fiscal year 2018 showed that most of the 16 selected agencies did not consistently or effectively implement policies or practices related to the core security functions of the cybersecurity framework. In addition, most of these selected agencies had deficiencies in implementing the eight elements of an information security program, as defined by FISMA. Also, inspectors general reported that most of the 24 CFO Act agencies did not have effective information security programs and were not effectively implementing security controls over financial systems during fiscal year 2018. Further, agency CIOs reported that most of the 23 civilian CFO Act agencies had not met targets for implementing cyber capabilities to reduce risk. FISMA requires agencies and their inspectors general to report on the adequacy and effectiveness of information security policies, procedures, and practices. To facilitate meeting this reporting requirement, CIGIE, in collaboration with OMB and DHS, developed metrics that agency inspectors general are to use to report on eight security domains that align with the five core security functions—Identify, Protect, Detect, Respond, and Recover—of the NIST cybersecurity framework. Table 3 illustrates how the inspector general reporting domains are related to the core security functions. Most of the 16 agencies that we reviewed had deficiencies in implementing policies and practices related to the cybersecurity framework core security functions and related domains during fiscal year 2018. Figure 3 shows the number of agencies with reported deficiencies in each of the framework’s core security functions. The Identify core security function includes the key process of risk management. NIST defines risk management as the process of identifying and assessing risk, and taking steps to reduce those risks to an acceptable level. NIST guidance specifies activities that agencies should implement to effectively identify and manage cybersecurity risks, including: establishing a risk management strategy that includes a determination identifying assets that require protection; assessing risk; and documenting plans of action and milestones (POA&Ms) to mitigate known deficiencies. Fifteen of the 16 selected agencies had deficiencies in activities associated with identifying risks. Figure 4 illustrates the number of selected agencies that had deficiencies in each of the activities. Establishment of a Risk Management Strategy Risk management strategies include strategic-level decisions and considerations for how senior leaders and executives are to manage risk to organizational operations and assets, individuals, other organizations, and the nation. GAO and inspectors general reports identified that 10 of the 16 selected agencies had deficiencies in developing, documenting, or implementing a risk management strategy. Specifically, nine of the 10 agencies had not developed or documented an enterprise-wide risk management strategy or process. Another agency had developed an enterprise risk management strategy but had not implemented it consistently across the agency. Without developing or documenting a risk management strategy, agencies lack clear guidance to help them make informed decisions for managing risk. Further, if agencies do not consistently implement a risk management strategy, they can potentially hinder their efforts to effectively identify and manage risk. FISMA requires agencies to develop and maintain an inventory of major information systems operated by or under the control of the agency to support risk management activities. Further, NIST Special Publication 800-53 states that centralized inventories of hardware, software, and firmware assets should be maintained to ensure proper accountability of those assets. These inventories also should be current, complete, and accurate to ensure proper accountability. Twelve of the 16 selected agencies did not fully identify or account for their major information systems or information technology assets. One agency did not maintain a comprehensive and accurate inventory of information systems and two other agencies did not maintain a current inventory of hardware and software assets. Nine additional agencies maintained neither a comprehensive and accurate inventory of information systems nor a current inventory of software and hardware assets. If agencies do not maintain comprehensive, accurate, or up-to- date inventories of information systems or hardware and software assets, agencies cannot ensure the protection of all assets within their networks. FISMA requires agencies to develop, document, and implement an agency-wide information security program that includes periodic risk assessments. According to NIST, these assessments are to address potential adverse impacts resulting from the operation and use of information systems and the information those systems process, store and transmit. Eight of the 16 selected agencies exhibited deficiencies in conducting risk assessments. Of the eight agencies that had deficiencies, four did not consistently perform risk assessments of their information systems; three did not fully update risk assessments subsequent to system changes; and one did not conduct a risk assessment supporting the agency’s decision to allocate resources to support mission and business processes. Without a sufficient process for conducting periodic risk assessments, agencies cannot determine, or appropriately respond to, risks to the information systems supporting the organization. Documentation of Plans of Action and Milestones FISMA requires agency information security programs to include a process for planning, implementing, evaluating, and documenting remedial action to address deficiencies in information system policies, procedures, and practices. In addition, NIST’s risk management framework states that agencies should implement a consistent process for developing POA&Ms using a prioritized approach to risk mitigation that is guided by a risk assessment. Further, documentation of POA&Ms should also be updated to reflect the current status of the deficiencies and, after remedial actions have been completed, agencies should test the actions to determine if they effectively addressed the deficiencies. Thirteen of the 16 selected agencies had deficiencies in their POA&M processes. Specifically, five agencies did not have an effective process for remediating vulnerabilities in a timely manner; seven other agencies did not adequately document or track the status of POA&Ms; and another agency did not assess the root cause of identified deficiencies to prioritize corrective actions based on the highest areas of risks. Additionally, one of the agencies that did not adequately document POA&Ms also did not have sufficient evidence to conclude that deficiencies were corrected even though the agency validated the remediation of the deficiency through its closure verification process. Without sufficiently documenting POA&Ms, agencies may not sufficiently remediate information security deficiencies in a timely manner, exposing their systems to increased risks that nefarious actors will exploit the deficiencies to gain unauthorized access to information resources. Agencies are to implement appropriate safeguards associated with the following four security domains that align with the Protect core security function: identity and access management; data protection and privacy; and security training. Each of the 16 selected agencies was deficient in developing and implementing appropriate safeguards to protect agency systems and networks. As shown in figure 5, most of the selected agencies had deficiencies in each of the four domains. NIST guidelines specify that agencies are to develop, implement, and maintain a baseline configuration; control changes to system configurations; and securely configure information systems. However, 14 of the selected 16 agencies reported weaknesses in one or more of these configuration management activities. Of the 14 agencies, nine had weaknesses in developing, maintaining, and implementing a baseline configuration for their information systems. For example, four agencies did not develop a baseline configuration for all systems or network devices. In addition, two agencies did not review or approve their baseline configurations. Further, three agencies did not consistently implement their baseline configurations. If agencies do not develop, maintain, or implement a current and comprehensive baseline of information systems and network devices, agencies cannot validate configuration information for accuracy, thereby hindering them from controlling changes made to a system. Eleven agencies did not effectively or consistently control changes to the configuration of their information systems. Properly controlling system changes can help agencies to ensure that changes are formally identified, proposed, reviewed, analyzed for security impact, tested, and approved prior to implementation. However, six of the 11 agencies did not properly approve or test changes before they were implemented; four other agencies did not consistently implement change control activities across their organization or their information systems; and one other agency did not consistently ensure accountability and responsibility for individuals performing configuration management activities. In addition, 12 agencies did not securely configure their information systems. NIST specifies that agencies should apply software patches in a timely manner, use vendor-supported software, apply secure configuration settings, and limit system functionality to least level needed to meet organizational requirements. However, of the 12 agencies that had deficiencies in implementing secure configurations, nine did not implement patches to address vulnerabilities or use up-to-date software that was supported by a vendor. Ten agencies also did not apply secure configuration settings to effectively enable security and facilitate the management of risk, while two agencies did not implement controls for limiting system functionality. As a result, these agencies cannot validate configuration information for their information systems and assets, detect or prevent unauthorized changes to information system resources, or provide a reasonable assurance that systems are configured and operating securely and as intended. Access controls are intended to limit or detect inappropriate access to computer resources to protect them from unauthorized modification, loss, and disclosure. Such controls include logical controls that require users to validate their identity and limit the files and other resources that those validated users can access and the actions they can execute. All 16 agencies that we reviewed had deficiencies in effectively implementing one or more controls associated with the identity and access management domain during fiscal year 2018. Fifteen of the 16 selected agencies did not adequately control user’s access to information systems and the information residing on them. For example, seven agencies did not appropriately authorize or approve system access before access was granted, and eight agencies did not perform user access reviews to ensure that they complied with account management policy. Additionally, 11 of the 16 agencies did not properly identify and validate information system users, which involve enforcing strong passwords and requiring passwords to be changed periodically. In addition, 11 of the 16 agencies had deficiencies in implementing access management to ensure separation of duties, or segregating work responsibilities so that one individual does not control all critical stages of a process. Without adequate access controls, unauthorized individuals, including outside intruders and former employees, can surreptitiously read and copy sensitive data and make undetected changes or deletions for malicious purposes or personal gain. According to NIST guidance on security and privacy controls, agencies should protect data at rest and in transit on their network through implementation of cryptography and other technologies to achieve confidentiality and integrity protections over that data. In addition, NIST’s guidance states that agencies should implement contingency strategies, such as conducting backups of information systems and having alternate processing and storage sites to protect data from loss during an interruption and to resume activities after an interruption. Further, NIST guidance states that agencies should develop privacy policies, procedures, and guidance for safeguarding the collection, access, use, dissemination, and storage of personally identifiable information that supports a privacy program. However, 15 of the 16 selected agencies did not effectively implement controls to protect data and ensure its privacy during fiscal year 2018. Specifically, eight of the 16 agencies did not adequately implement controls for protecting information at rest and four agencies did not adequately implement controls for ensuring the integrity and confidentiality of data in transit. In addition, five of the 16 agencies did not conduct backups of information systems and five agencies did not use alternate processing sites to retrieve backups or resume essential mission/business functions. Further, the inspectors general for 14 of the 16 agencies reported that their respective agency did not effectively document or implement policies and procedures supporting the agency’s privacy program. If agencies do not effectively implement controls to protect data and ensure its privacy, agencies may be hindered in limiting or containing the impact of a potential cybersecurity event. FISMA requires agency information security programs to include security awareness training to inform personnel of information security risks associated with their activities and responsibilities in complying with agency policies and procedures intended to reduce risk. In addition, FISMA requires agencies to provide role-based training to personnel with significant responsibilities for information security. Further, NIST guidance on building an IT security awareness and training program states that an awareness and training program is the means to communicate information that users need to support the mission of the organization, and security requirements across the agency. Most of the selected agencies exhibited deficiencies in implementing a security training program during fiscal year 2018. Only three of the 16 selected agencies effectively implemented elements of a security training program. Of the 13 agencies that had deficiencies, 12 did not ensure that personnel received security awareness training and 10 did not ensure that personnel with significant responsibilities for information security received role-based training, including nine agencies that were deficient in providing both types of training. As a result, these agencies risk having employees or contractors that are ill-prepared to protect systems, and risk inadvertently or intentionally compromising security. Agencies are to develop and implement controls to Detect cyber events and vulnerabilities. FISMA requires agencies to develop, document, and implement an agency-wide information security program that includes periodic testing and evaluation of effectiveness and procedures for detecting security incidents. NIST guidelines define these and other activities as part of information security continuous monitoring, including: defining an information security continuous monitoring strategy and implementing an information security continuous monitoring program in accordance with that strategy; assessing and reporting on the effectiveness of all implemented collecting, correlating, and analyzing security related information obtained through information system auditing. However, as shown in figure 6, agencies exhibited deficiencies in activities associated with information security continuous monitoring. Continuous Monitoring Strategy and Program NIST’s guidance on information security continuous monitoring states that defining an information security continuous monitoring strategy and developing an information security continuous monitoring program are the first two steps in creating, implementing, and maintaining information security continuous monitoring. In addition, agencies should implement the information security continuous monitoring program in accordance with the defined strategy. However, half of the 16 selected agencies did not develop an information security continuous monitoring strategy or program, or implement the information security continuous monitoring program. Specifically, five of the agencies did not fully develop an information security continuous monitoring strategy or program. In addition, while three agencies had developed, or made organizational changes to create a foundation for, an information security continuous monitoring strategy, those agencies did not consistently or effectively implement the strategy. Without a well- designed and implemented information security continuous monitoring strategy, agencies could be hindered in assuring ongoing situational awareness of information security, vulnerabilities, and threats. As stated above, FISMA requires agencies to include periodic testing and evaluation of information security policies, procedures, and practices in agency-wide information security programs. Security control assessments determine the extent to which controls are implemented correctly, operating as intended, and producing the desired outcome with respect to meeting the system requirements. Most agencies assessed the controls implemented on their systems. However, seven agencies did not consistently perform system control assessments to ensure that the controls were operating effectively, or as intended. Further, seven agencies had not completed or implemented other activities in their security assessment and authorization process that assists agencies with ensuring that appropriate controls are implemented on an information system and that the system is authorized to operate. If agencies do not perform consistent testing of information security controls, they cannot determine that implemented controls are appropriately designed or operating effectively. Audit Review, Analysis, and Reporting According to NIST guidance on log management, routine log analysis is beneficial to identifying security incidents, policy violations, fraudulent activity, and operational problems. As a result, log analysis supports information security continuous monitoring capabilities. However, more than half of the 16 selected agencies did not review, analyze, and report auditable events from audit logs. For example, nine agencies did not implement audit log review capabilities on their information systems. Without reviewing, analyzing, and reporting audit logs, agencies limit their ability to identify unauthorized, unusual, or sensitive access activity on their networks. Agencies should have policies and practices in place to Respond to detected incidents. FISMA requires agency information security programs to include procedures for responding to security incidents in order to mitigate risks associated with such incidents before substantial damage is done. According to NIST, incident response involves rapidly detecting incidents, minimizing loss and destruction, mitigating the weaknesses that were exploited, and restoring IT services. An effective incident response process includes, for example: an incident handling capability that incorporates lessons learned from ongoing incident handling activities; the monitoring of incidents through documentation that includes pertinent information necessary for forensics, evaluating incident details, trends, and handling; the timely reporting of incidents with sufficient detail to allow analysis; and an incident response plan. Most of the 16 selected agencies had deficiencies in at least one of the activities associated with incident response processes, as shown in figure 7. According to NIST, agencies should have the ability to detect and analyze security incidents in order to minimize loss and destruction and mitigate the weaknesses that were exploited. In addition, agencies should incorporate lessons learned from an incident to improve existing security controls and practices. Most of the selected agencies did not report deficiencies associated with their incident handling capability, including the ability to analyze and respond to security incidents and incorporate lessons learned. However, seven agencies did not adequately implement capabilities to analyze and respond to security incidents. In addition, one of the seven agencies did not use lessons learned from prior incidents to improve incident handling. Without an effective incident handling capability, agencies have limited ability to detect and analyze security incidents to minimize destruction and mitigate exploited vulnerabilities. According to NIST, agencies should monitor and document security incidents with sufficient detail in order to effectively respond to and mitigate the risks associated with the incident. Doing so enables agencies to analyze security incidents, understand the impact of the incident, and perform analysis to identify trends and indicators of attack. Inspectors general for 12 of the 16 selected agencies did not identify deficiencies related to monitoring detected incidents. However, four agencies did not effectively monitor incidents. For example, one agency did not consistently document incidents detected and another agency had not implemented an automated enterprise tool for monitoring incidents. If agencies do not effectively implement incident monitoring processes, they hinder their ability to adequately analyze and respond to security incidents. FISMA requires agencies to develop, document, and implement an agency-wide information security program that includes procedures for reporting security incidents to US-CERT. In addition, NIST guidance states that agencies should have specific incident reporting requirements for reporting suspected security incidents to an internal incident reporting organization. However, 10 agencies had deficiencies in their implementation of incident reporting. While only two agencies did not clearly define incident reporting requirements, eight agencies did not effectively implement those requirements. For example, these agencies did not consistently categorize incidents or ensure timely reporting of incidents to US-CERT and internal reporting organizations. If agencies do not consistently categorize or report incidents in an accurate and timely manner, they cannot effectively respond to incidents because they may lack effective situational awareness in order to appropriately respond to incidents. Incident response plans are an important element to ensuring that incident response is performed effectively, efficiently, and consistently throughout the agency. Among other things, NIST guidance states that incident response plans should provide a roadmap for implementing an incident response capability, describe metrics for measuring the incident response capability, and be approved. Inspectors general for nine of the selected agencies did not report deficiencies related to incident response plans. However, seven agencies did not fully develop or monitor the effectiveness of their incident response plans. Specifically, five agencies had incident response plans that did not fully define requirements for implementing their incident response capability or were not approved. In addition, the other two agencies did not use performance metrics to verify the effectiveness of their incident response plan. Without an effective and comprehensive incident response plan, agencies cannot implement a coordinated approach to incident response. Agencies should be able to Recover from cyber events. FISMA requires agencies to develop, document, and implement an agency-wide information security program that includes plans and procedures to ensure continuity of operations for information systems that support the operations and assets of the agency. NIST defines contingency planning as a coordinated strategy involving plans, procedures, and technical measures that enable the recovery of information systems, operations, and data after a disruption. Contingency planning is significant to protecting electronically maintained data and an agency’s ability to process and retrieve data during and after a cyber intrusion. According to NIST, agencies should develop and document a comprehensive contingency plan or suite of related plans for restoring capabilities during and after a cyber event. The suite of related plans should include a disaster recovery plan and business impact analysis. However, 11 of the 16 selected agencies did not sufficiently plan for recovering system operations after an interruption. Specifically, these 11 agencies did not consistently develop contingency plans, to include disaster recovery plans, or other associated documentation, such as business impact analyses for all of their information systems. In addition, one agency did not define how the agency is to process and retrieve data during and after an interruption. Without an effective contingency planning process, agencies are exposed to the risk of interruptions to information system operations and disruption to their mission and business processes. Controls associated with the five core security functions are related to elements of agencies’ information security programs. FISMA requires each agency to develop, document, and implement an information security program that includes the following eight elements: 1. periodic assessments of the risk; 2. cost-effective policies and procedures that reduce risk to an acceptable level, ensure that information security is addressed throughout the life cycle of each system, and ensure compliance with applicable requirements; 3. subordinate plans for providing adequate information security for networks, facilities, and systems or groups of information systems, as appropriate; 4. security awareness training and training for personnel with significant responsibilities for information security; 5. periodic testing and evaluation of the effectiveness of security policies, procedures, and practices; 6. a process for planning, implementing, evaluating, and documenting remedial actions to address information security deficiencies; 7. procedures for detecting, reporting, and responding to security 8. plans and procedures to ensure continuity of operations for information systems. As discussed earlier in this report, most of the 16 selected agencies had deficiencies related to implementing the eight elements of an agency- wide information security program. Figure 8 shows the number of selected agencies with deficiencies in implementing the eight elements of an agency-wide information security program. For example, of the 16 selected agencies: Eight agencies did not effectively assess risk; 11 agencies did not have policies to ensure that CIOs carried out their role as it relates to information security; Four agencies developed incomplete system security plans; 13 agencies did not ensure that personnel received security awareness training, or that personnel with security responsibilities received role-based security training; Seven agencies did not consistently perform control assessments to ensure that the controls were operating effectively, or as intended; 13 agencies did not effectively implement their POA&M process to address information security deficiencies; 13 agencies did not adequately detect or respond to incidents; and 11 agencies did not comprehensively develop plans to ensure the continuity of its operations. We and inspectors general have made numerous recommendations aimed at improving information security programs and practices over the years. Until these agencies take action to address deficiencies in implementing the eight elements of an agency-wide information security program, they lack assurance that their information systems and networks are protected from inadvertent or malicious activity. Inspectors general determined that few agencies covered by the CFO Act of 1990 had effective agency-wide information security programs during fiscal year 2018. Further, in agency financial statement audit reports for fiscal year 2018, inspectors general reported that they continued to identify significant deficiencies in information security controls over financial systems. As a result, inspectors general reported material weaknesses or significant deficiencies in internal control over financial reporting for fiscal year 2018. FISMA requires inspectors general to determine the effectiveness of their respective agencies’ information security programs. To do so, OMB instructed inspectors general to provide a maturity rating for agency information security policies, procedures, and practices related to the five core security functions established in the NIST cybersecurity framework, as well as for the agency-wide information security program. For fiscal year 2018, the inspectors general for only six of the 24 CFO Act agencies reported that their agencies had an effective agency-wide information security program. However, the remaining 18 agencies were reported as having ineffective information security programs. When considering each of the five core security functions, most inspectors general reported that their agency was at Level 3 (consistently implemented) for the Identify, Protect, and Recover functions; at Level 2 (defined) for the Detect function; and at Level 4 (managed and measurable) for the Respond function, as shown in figure 9. Agency inspectors general report on the effectiveness of agencies’ information security controls as part of the annual audits of the agencies’ financial statements. The reports resulting from these audits include a description of information security control deficiencies related to the five major control categories defined by the Federal Information System Controls Audit Manual (FISCAM)—security management, access controls, configuration management, segregation of duties, and contingency planning. For fiscal year 2018, inspectors general identified information security control deficiencies related to most of the FISCAM general control categories for most of the 24 CFO Act agencies as shown in figure 10. Overall, inspectors general for the 24 CFO Act agencies continued to report deficiencies in agencies information security practices for fiscal year 2018. Specifically, during that time, 18 inspectors general designated information security as either a material weakness (6) or significant deficiency (12) in internal control over financial reporting systems for their agency. Further, inspectors general at 21 of the 24 agencies cited information security as a major management challenge for their agency for fiscal year 2018. OMB, in its fiscal year CIO reporting metrics, directed CIOs to assess their agencies’ progress toward achieving outcomes that strengthen federal cybersecurity. To do this, CIOs evaluated their agency’s performance in reaching targets for meeting key milestones of the current administration’s IT Modernization Cross-Agency Priority (CAP) goal. This CAP goal includes a cybersecurity initiative to mitigate the impact of risks to federal agencies’ data, systems, and networks by implementing cutting edge cybersecurity capabilities. The CAP goal’s cybersecurity initiative has three strategies that include key milestones with specific implementation targets, most of which are expected to be met by the end of fiscal year 2020. Table 4 shows the key milestones and targets related to the three strategies of the IT Modernization CAP goal’s cybersecurity initiative, as well as how many agencies were meeting the targets for each of the milestones. Overall, only two of the civilian 23 CFO Act agencies met all 10 targets for the cybersecurity initiative of the IT Modernization CAP goal, during fiscal year 2018. Whereas, 10 agencies met seven to nine of the targets and the remaining 11 agencies met six or fewer targets. More specifically, by strategy area, Seven agencies met all four targets for the manage asset security strategy. Eight agencies met all three targets for the limit personnel security strategy. Seven agencies met all three targets for the protect networks and data strategy. OMB, DHS, and NIST have ongoing and planned initiatives to support FISMA’s implementation across the federal government. Specifically, OMB developed and oversaw the implementation of information security policies, procedures, and guidelines over the past 2 years. In addition, DHS oversaw and assisted government efforts that were intended to provide adequate, risk-based, cost-effective cybersecurity. Further, NIST continued to provide guidance to federal agencies to improve information security across the government. However, beyond fiscal year 2016, OMB held CyberStat meetings at significantly fewer agencies. These meetings are intended to help ensure effective implementation of information security policies and practices. In addition, OMB’s guidance to agencies for preparing their fiscal year 2018 FISMA report does not sufficiently address FISMA’s requirement for developing subordinate plans for providing adequate information security for networks, facilities, and information systems. FISMA requires that OMB submit a report to Congress no later than March 1 of each year on the effectiveness of agencies’ information security policies and practices during the preceding year. This report is to include: a summary of incidents described in the agencies’ annual reports; a description of the threshold for reporting major information security a summary of results from the annual IG evaluations of each agency’s information security program and practices; an assessment of each agency’s compliance with NIST information an assessment of agency compliance with OMB data breach notification policies and procedures. As of June 2019, OMB had not issued its annual FISMA report to Congress for fiscal year 2018. OMB officials stated that the lapse in appropriations during the start of 2019 caused a delay in the report’s development and release. The officials declined to provide a time frame for when they expected to issue the report. FISMA requires OMB to develop and oversee the implementation of policies, principles, standards, and guidelines on information security. Since the start of fiscal year 2018, OMB has developed or proposed policies and generally monitored their implementation. Specifically: In May 2019, OMB issued policy to address federal agencies’ implementation of identity, credential, and access management (ICAM). Among other things, the policy requires agencies to (1) implement identity, credential, and access management guidelines, standards, and directives issued by NIST, DHS, and the Office of Personnel Management; and (2) harmonize their enterprise-wide approach to ICAM governance, architecture, and acquisition through activities such as designating an integrated agency-wide ICAM governance structure and establishing solutions for ICAM services that are flexible and scalable. In December 2018, OMB issued a memorandum on the high-value asset (HVA) program that (1) outlined agency expectations for establishing agency governance; (2) required agencies to take action to improve the identification of HVAs; and (3) defined agency reporting, assessment, and remediation requirements for HVAs. In March 2018, OMB reported that agencies’ continued to have challenges in mitigating security vulnerabilities identified across the federal HVA landscape in its fiscal year 2017 FISMA report to Congress. In addition, OMB required agencies to report on the implementation of security controls to protect HVAs during fiscal year 2018. In October 2018, OMB issued new federal information security and privacy management guidance that required agencies to (1) report on the adequacy and effectiveness of their information security programs, (2) submit a current and prioritized list of HVAs through the Homeland Security Information Network, and (3) report major incidents to DHS, OMB, Congress and their agency inspectors general. In addition, the guidance required agencies to ensure that DHS has authorization and the information necessary to monitor and provide technical assistance related to vulnerability scanning. In addition to developing and monitoring the implementation of information security policies, FISMA directs OMB to oversee agencies’ compliance with the act’s requirements to provide information security protections commensurate with the risk and magnitude of the harm resulting from unauthorized access, use, disclosure, modification, or destruction of information or information systems. During fiscal year 2018, OMB issued four reports summarizing government-wide implementation of the information security requirements, as described below: In September 2018, OMB issued an assessment of intrusion detection and prevention capabilities across the federal enterprise. In its assessment, OMB briefly described federal agencies’ implementation of intrusion detection and prevention capabilities through DHS’s EINSTEIN sensor suite. In May 2018, OMB issued its Federal Cybersecurity Risk Determination Report and Action Plan. For this report, OMB evaluated risk management assessment reports for 96 agencies and described actions that it and agencies plan to take to address government-wide cybersecurity gaps. Two major actions discussed in the report are: (1) federal agencies must consolidate their security operations center capabilities and processes, or migrate the security operations center as a service; and (2) OMB, DHS, and other federal agencies are to assist with implementing the cyber threat framework developed by the Office of the Director of National Intelligence. In March 2018, OMB issued its annual FISMA report to Congress for fiscal year 2017, which summarized the performance of 97 agencies in implementing effective information security programs and managing risk, among other things. In December 2017, OMB released its Report to the President on Federal IT Modernization, which outlined a vision and recommendations for the federal government to build a more modern and secure architecture for federal systems. For example, OMB described government-wide initiatives intended to improve the security of federal networks that emphasized perimeter network- based security protections, but had gaps in the application and data- level protections needed to provide complete security. To address these deficiencies, OMB recommended a layered defensive strategy in government-wide programs to provide greater defense-in-depth capabilities that are intended to prevent malicious actors from moving laterally across linked networks to access valuable information. Number of Agencies Scheduled for CyberStat Meetings Significantly Declined Since Fiscal Year 2016 OMB, in coordination with DHS, is responsible for coordinating CyberStat review meetings. As mentioned previously, FISMA requires OMB to oversee agency compliance with requirements to provide information security protections on information and information systems. One means of fulfilling this oversight responsibility is through CyberStat engagements. For these engagements, OMB, in coordination with DHS, intends to engage agency leadership on Administration priorities and perform outreach to ensure that agencies are taking the appropriate actions to strengthen their cybersecurity posture. However, since our September 2017 report on fiscal year 2016 FISMA implementation, the number of agencies that have participated in a CyberStat engagement has significantly declined. In fiscal year 2016, OMB scheduled these engagements with 24 agencies to help develop action items that address information security risk, identify areas for targeted assistance, and track performance at the agencies throughout the year. The number of agencies scheduled to participate in an engagement decreased to five during fiscal year 2017, and decreased further to three during fiscal year 2018. As of May 2019, OMB staff in the Office of the Federal CIO informed us that the agency had not scheduled any agencies to participate in a CyberStat engagement during fiscal year 2019. According to OMB officials in the Office of the Federal CIO, updates to the CyberStat process resulted in extended engagements between DHS, OMB, and the agencies that lasted 4 to 6 weeks or more. Beginning in fiscal year 2017, according to DHS’s CyberStat concept of operations, OMB and DHS took a collaborative approach with the CyberStat process. Specifically, officials from the participating agencies, OMB’s Cyber and National Security Unit, and DHS’s Federal Network Resilience (FNR) division collaborated through these CyberStat engagements to reach a desired performance outcome at the participating agencies. DHS’s CyberStat concept of operations states that the department focuses on agency performance in key federal information security reporting, including agency FISMA reporting, DHS reports of agency compliance with binding operational directives, and reports issued by GAO and agency inspectors general. A DHS official from the department’s FNR division informed us that it uses these information security reports to make recommendations to OMB, who then decides which agencies will be scheduled to participate in a CyberStat engagement. According to OMB, the three agencies that participated in a CyberStat engagement initiated during fiscal year 2018 volunteered to do so after discussing their cybersecurity implementation issues with OMB. However, as discussed earlier in this report, deficiencies reported in agency fiscal year 2018 FISMA reports and information security evaluation reports issued by GAO and inspectors general for fiscal year 2018 indicate that several agencies are in need of OMB and DHS assistance to improve their information security posture. In addition, the three agencies that participated in CyberStat engagements scheduled during fiscal year 2018 saw value in changes resulting from the updated engagement process. For example, officials from the Office of the CIO (OCIO) at one of the three agencies stated that the updated process was more constructive and valuable than the prior CyberStat process that was based more on a compliance checklist. In addition, OCIO officials at all three agencies stated that the process helped improve their agencies’ information security posture and that their collaboration with OMB and DHS was beneficial to assisting with FISMA implementation. By conducting fewer CyberStat engagements with agencies, OMB loses an opportunity to assist agencies with improving their information security posture. Additionally, OMB will limit its ability to oversee specific agency efforts to provide information security protections for federal information and information systems. FISMA includes reporting requirements for OMB, agency CIOs and inspectors general. According to OMB’s FISMA reporting guidance, OMB and DHS collaborate with interagency and inspector general partners to develop the CIO and inspector general metrics, which are intended to facilitate agencies’ compliance with FISMA-related reporting requirements. These entities created separate sets of reporting metrics for agency CIOs and agency inspectors general. However, the inspector general reporting metrics did not specifically address the development and maintenance of system security plans, although subordinate plans, such as system security plans, are a key element of an agency-wide information security program required by FISMA. OMB officials in the Office of the Federal CIO informed us that, while they work in coordination with CIGIE to establish the reporting metrics, CIGIE is ultimately responsible for developing the metrics. According to both the published metrics and OMB’s guidance memorandum, OMB collaborates with DHS and inspector general partners to develop the IG FISMA metrics. According to representatives from CIGIE, the existence of system security plans is addressed in multiple questions within the reporting metrics, which is in alignment with OMB’s focus toward ongoing assessments and authorizations. Nevertheless, our review of the reporting metrics and supplemental evaluation guide did not identify any reference to the development and maintenance of system security plans. The lack of a defined reporting metric for addressing agency system security plans could lead to inconsistent reporting by inspectors general. Until such a metric is developed and reported on, OMB will not have reasonable assurance that inspectors general evaluations appropriately address each of the required elements of an information security program. Under FISMA, DHS, in consultation with OMB, is responsible for carrying out various activities, including developing and overseeing the implementation of binding operational directives and providing operational and technical assistance to agencies. Over the last 2 years, DHS had developed four binding operational directives as of April 2019, as required by FISMA. These directives instructed agencies to: remove and discontinue use of all present and future Kaspersky- branded products; enhance email security by adopting domain-based message authentication, reporting and conformance (DMARC) to prevent email spoofing and web security by ensuring all publicly accessible federal websites provides services through a secure connection; submit a current and prioritized high-value asset list to DHS and if selected, participate in risk and vulnerability assessments; and review and remediate critical and high vulnerabilities on internet- facing systems within 15 and 30 calendar days of initial detection, respectively. We have ongoing work evaluating DHS’s process to develop and oversee the implementation of binding operational directives as part of another engagement. We will report on the results of this evaluation in a separate report. DHS also provided operational and technical assistance to agencies through its Continuous Diagnostics and Mitigation (CDM) and National Cybersecurity Protection System (NCPS) programs. DHS is taking steps to deploy the CDM and NCPS capabilities to all participating federal agencies to enhance detection of cyber vulnerabilities and protection from cyber threats. Continuous Diagnostics and Mitigation program (CDM). The program is to provide federal departments and agencies with commercial off-the- shelf capabilities and tools that identify cybersecurity risks on an ongoing basis, prioritize these risks based upon potential impacts, and enable cybersecurity personnel to mitigate the most significant problems first. In December 2018, we reported that the department was in the process of enhancing the capabilities of federal agencies to automate network monitoring for malicious activity through its CDM program. In our December report, we also recommended that DHS coordinate further with federal agencies to identify training and guidance needs for implementing CDM. DHS plans to complete implementation of our recommendation this fiscal year. In addition, we have an ongoing review to evaluate the extent to which selected agencies have effectively implemented CDM and to identify practices for effective and efficient implementation of the program. We will report on the results of this review separately. National Cybersecurity Protection System (NCPS). The program is one of the tools to aid federal agencies in mitigating information security threats. The system is intended to provide DHS with the capability to provide four cyber-related services to federal agencies: intrusion detection, intrusion prevention, analytics, and information sharing. In January 2016, we made nine recommendations to further improve NCPS capabilities by, among other things, developing metrics that clearly measure the effectiveness of NCPS’s efforts, including the quality, efficiency, and accuracy of actions related to detecting and preventing intrusions, providing analytic services, and sharing cyber-related information. As of June 2019, DHS had implemented six of our nine recommendations and plans to implement the remainder by the end of this fiscal year. According to FISMA, NIST is to develop information security standards and guidelines, in coordination with OMB and DHS. Specifically, NIST’s Computer Security Division is responsible for developing cybersecurity standards, guidelines, tests, and metrics for the protection of federal information systems. NIST has developed information security guidelines for federal agencies. Specifically, in April 2018, NIST issued an update to its cybersecurity framework that it originally issued in February 2014. Although the cybersecurity framework was initially intended for critical infrastructure, Executive Order 13800 requires federal agencies to use the cybersecurity framework to also manage their cybersecurity risk. The revised framework includes a new section on cybersecurity measurement; an expanded explanation of using the framework for cyber supply chain risk management; refinements to authentication, authorization, and identity proofing policies within access controls; and a new section on using the cybersecurity framework to understand and assess an organization’s cybersecurity risk. In May 2017, NIST published draft guidance for agencies to use in implementing the cybersecurity framework. This publication is intended to provide guidance on the use of the framework in conjunction with the current and planned suite of NIST security and privacy risk management publications, such as NIST Special Publication 800-53. According to NIST officials in the agency’s Computer Security Division, the agency is in the process of finalizing the implementation guidance and plans to publish the final version by the end of fiscal year 2019. Further, in December 2018, NIST released the revised Risk Management Framework for Information Systems and Organizations (risk management framework). According to NIST, the update provides an integrated, robust, and flexible methodology to address security and privacy risk management. Among the changes in the updated version is the integration of privacy risk management into the existing information security risk management processes. In addition, the risk management framework includes direct references to the cybersecurity framework, which demonstrates how organizations that implement the risk management framework can also achieve the outcomes of the cybersecurity framework. In April 2019, NIST released revised guidance on vetting the security of mobile applications. According to NIST, the revised publication provides guidance for planning and implementing a mobile application vetting process, developing security requirements for mobile applications, identifying appropriate tools for testing mobile applications, and determining if a mobile application is acceptable for deployment on an organization’s mobile devices. In addition, NIST is currently developing a privacy framework to help improve agencies’ privacy risk management. In April 2019, NIST issued a discussion draft for its privacy framework. According to the discussion draft, NIST will use feedback received on the discussion draft to develop a preliminary draft of the privacy framework, which is intended to assist organizations in identifying, assessing, and responding to privacy risks. Further, the framework is intended to foster the development of innovative approaches to protecting individuals’ privacy and increase trust in systems, products and services. According to NIST officials, the agency continues to engage stakeholders, both nationally and internationally, through roundtable meetings, webinars, and public workshops to solicit stakeholder input to inform development of this framework. NIST’s website states that the agency anticipates publishing the privacy framework in October 2019. Federal agencies continued to have deficiencies in implementing information security programs and practices. Inspectors general reported that 18 of 24 CFO Act agencies did not have effective agency-wide information security programs in fiscal year 2018. In addition, most of the selected agencies had deficiencies in the five core security functions. We and the inspectors general have made thousands of recommendations aimed at improving information security programs and practices over the years. Implementation of these recommendations will assist agencies in strengthening their information security policies and practices. OMB, DHS, and NIST have issued directives and guidance and implemented programs that, to some extent, have improved agencies’ security posture. However, OMB has not issued its report to Congress on the effectiveness of agencies’ information security policies and practices for fiscal year 2018, although the report was due several months ago. Further, while agencies indicated that the collaborative CyberStat engagements with DHS and OMB have aided with their FISMA implementation, the number of these engagements has declined significantly. In addition, the OMB-approved metrics that inspectors general use to evaluate FISMA implementation do not include one of the elements—system security plans—required by FISMA for information security programs. By not including this element, oversight of agencies’ information security programs has been diminished. We are making the following three recommendations to OMB: The Director of OMB should submit the statutorily required report to Congress on the effectiveness of agencies’ information security policies and practices during the preceding year. (Recommendation 1) The Director of OMB should expand its coordination of CyberStat review meetings for those agencies with a demonstrated need for assistance in implementing information security. (Recommendation 2) The Director of OMB should collaborate with CIGIE to ensure that the inspector general reporting metrics include the FISMA-required information security program element for system security plans. (Recommendation 3) We provided a draft of this report to OMB and the 28 selected agencies for review and comment. In response, OMB provided comments orally and via email in which the office, respectively, generally concurred with our first two recommendations and concurred with a revised version of our third recommendation. Specifically, in oral comments, officials in the Office of the Federal Chief Information Officer noted actions that they said OMB plans to take to address our first two recommendations. According to these officials, the office plans to issue its fiscal year 2018 report to Congress on the effectiveness of agencies’ information security policies and practices in the near future. In addition, the office plans to continue to collaborate with DHS to identify information security gaps at agencies and work with agencies to address those gaps in CyberStat meetings or by other means. With regard to our third recommendation, the officials expressed concern with the wording of the recommendation in our draft report, which related to OMB updating the IG metrics. They noted that CIGIE, rather than OMB, is responsible for updating these metrics. Accordingly, we revised the recommendation to emphasize the need for OMB to collaborate with CIGIE. In a subsequent email from our OMB liaison, the office concurred with the revised recommendation. The office emphasized its plans to continue working collaboratively with the inspector general community to assist with improving and evolving the metrics to ensure that the metrics address FISMA requirements. OMB also provided technical comments, which we incorporated, as appropriate. In addition, five of the 28 selected agencies provided written responses regarding the draft report: In its response (reprinted in appendix III), the Department of Housing and Urban Development stated that it had reviewed our draft report and had no comments. In its comments (reprinted in appendix IV), the Department of Veterans Affairs stated that it remains committed to complying with the requirements of FISMA and to safeguarding the department’s systems and data, which support the delivery of care, benefits, and services to veterans. The department also stated that it continues to prioritize efforts to address our prior information security-related recommendations to the department. In its response (reprinted in appendix V), the Environmental Protection Agency stated that it had reviewed our draft report and had no comments. In its comments (reprinted in appendix VI), the Social Security Administration stated that it will continue to improve its cybersecurity safeguards and looks forward to receiving additional guidance to assist the agency with its efforts. In its comments (reprinted in appendix VII), the U.S. Agency for International Development stated that it has developed, documented, and implemented an agency-wide program to provide security for its information and systems, pointing out that its inspector general reported that the agency had an effective program in fiscal year 2018. The agency also cited its commitment to continuing compliance with FISMA’s requirements and to safeguarding its information technology services to facilitate its mission. Further, four of the selected agencies—the Departments of Commerce, Homeland Security, and Transportation, as well as the National Science Foundation—also provided technical comments which we have incorporated in the report, where appropriate. The remaining 19 selected agencies provided emails stating that they had no comments on the report. These agencies were the Departments of Agriculture, Defense, Education, Energy, Health and Human Services, the Interior, Justice, Labor, State, and the Treasury; and the Federal Communications Commission; Federal Retirement Thrift Investment Board; General Services Administration; Merit System Protection Board; National Aeronautics and Space Administration; Nuclear Regulatory Commission; Office of Personnel Management; Presidio Trust; and Small Business Administration. We are sending copies of this report to appropriate congressional committees, the Director of OMB, the heads of the CFO Act agencies and their inspectors general, the heads of four selected non-CFO Act agencies, and other interested congressional parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Gregory C. Wilshusen at (202) 512-6244 or wilshuseng@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VIII. Our objectives were to (1) describe the reported adequacy and effectiveness of selected federal agencies’ information security policies and practices and (2) evaluate the extent to which the Office of Management and Budget (OMB), the Department of Homeland Security (DHS), and the National Institute of Standards and Technology (NIST) have implemented their government-wide Federal Information Security Modernization Act of 2014 (FISMA) requirements. To describe the reported adequacy and effectiveness of federal agencies’ information security policies and practices, we analyzed our, agency, and inspectors general information security-related reports for 16 selected agencies. Our selection of 16 agencies included 12 Chief Financial Officers (CFO) Act of 1990 agencies and four non-CFO Act agencies. To select the 12 CFO Act agencies, we first ranked the 23 civilian CFO Act agencies by the number of information security systems each agency reported operating in fiscal year 2017. We then separated the agencies into large, medium, and small categories based on the number of systems they reported, and selected four agencies from each category using a random number generator. To select the four non-CFO Act agencies, we listed the 73 non-CFO Act agencies reported in OMB’s annual FISMA report to Congress for fiscal year 2017 and then randomly selected four agencies. Although we randomly selected agencies and assured we had CFO Act and non-CFO Act agencies, due to the small number of agencies examined, results based on these agencies do not generalize beyond the agencies reviewed. The 16 agencies were the Departments of the Agriculture, Commerce, Education, Housing and Urban Development, Justice, Labor, State, and the Treasury; the Environmental Protection Agency; Federal Communications Commission; Federal Retirement Thrift Investment Board; Merit Systems Protection Board; National Aeronautics and Space Administration; Presidio Trust; Small Business Administration; and the Social Security Administration. For these agencies, we analyzed, categorized, and summarized weaknesses identified in inspector general and GAO reports using the NIST Framework for Improving Critical Infrastructure Cybersecurity (cybersecurity framework) core security functions and the eight elements of information security programs required by FISMA. In addition, for the 24 agencies covered by the CFO Act, we summarized (1) the inspector general ratings of agency-wide information security programs and (2) the inspector general designation of information security as a significant deficiency or a material weakness for financial reporting systems as reported for fiscal year 2018. For the 23 civilian agencies covered by the CFO Act, we summarized fiscal year 2018 agency Chief Information Officer (CIO) reports of their agency’s progress in meeting targets for implementing cyber capabilities supporting the Administration’s cybersecurity-related Cross-Agency Priority (CAP) goal. To gain insight into how agencies collect, report, and ensure the accuracy and completeness of the FISMA data they report, we analyzed documentation describing and supporting the processes at eight of the 16 selected agencies to ensure the accuracy and completeness of those data. We also interviewed officials at the eight agencies to obtain additional information on the quality controls implemented on the system used for FISMA reporting. The eight agencies selected were the Departments of Education, Justice, Labor, and the Treasury; the Federal Communications Commission; National Aeronautics and Space Administration; Presidio Trust; and the Small Business Administration. These agencies were randomly selected from the list of 16 agencies described above. Based on our assessment, we determined that the data were sufficiently reliable for the purpose of our reporting objectives. To evaluate the extent to which OMB, DHS, and NIST have implemented FISMA requirements, we analyzed the FISMA provisions to identify federal responsibilities for OMB, DHS, and NIST. We evaluated documentation of these agencies’ government-wide responsibilities to determine if the agencies were meeting FISMA requirements, including documentation obtained from their websites. Specifically, for OMB, we collected and reviewed information security-related policies and guidance that it issued since we last reported in September 2017. We also obtained reports issued by OMB to determine the extent to which the agency had overseen the policies and guidelines it issued, as well as other agency efforts for improving information security. In addition, we analyzed fiscal year 2018 inspector general and CIO FISMA reporting metrics to determine if the metrics sufficiently addressed the agency-wide information security program elements required by FISMA. We also interviewed OMB officials to obtain information on any actions they have planned or taken to improve the information security posture of the federal government. Further, we interviewed OMB and DHS officials to understand their process for scheduling CyberStat engagements with senior agency officials. We also interviewed officials at the three agencies that participated in a CyberStat engagement initiated during fiscal year 2018 to understand the benefits and challenges of their collaboration with OMB and DHS. For DHS, we reviewed and summarized a recently issued GAO report describing updates to the department’s Continuous Diagnostic and Mitigation Program and National Cybersecurity Protection System. We also collected and summarized the binding operational directives issued by DHS over the last 2 years. Further, we interviewed DHS officials to obtain information on any actions they have planned or taken to improve the information security posture of the federal government. For NIST, we collected and summarized the standards and guidance issued or updated by the agency since the start of fiscal year 2018. We also interviewed NIST officials and obtained information on draft standards and guidance to describe NIST’s current and planned efforts to help improve the information security posture of the federal government. We conducted this performance audit from December 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The National Institute of Standards and Technology established the cybersecurity framework to provide guidance for cybersecurity activities within the private sector and government agencies at all levels. The cybersecurity framework consists of five core functions: identify, protect, detect, respond, and recover. Within the five functions are 23 categories and 108 subcategories that define discrete outcomes for each function, as described in table 5. In addition to the individual named above, Jeffrey Knott (assistant director), Di’Mond Spencer (analyst-in-charge), Andrew Ahn, Chris Businsky, Fatima Jahan, and Priscilla Smith made key contributions to this report.", "summary": "For 22 years, GAO has designated information security as a government-wide high-risk area. FISMA requires federal agencies to develop, document, and implement information security programs and have independent evaluations of those programs and practices. It also assigns government-wide responsibilities for information security to OMB, DHS, and NIST. FISMA includes a provision for GAO to periodically report to Congress on agencies' implementation of the act. GAO's objectives in this report were to (1) describe the reported adequacy and effectiveness of selected federal agencies' information security policies and practices and (2) evaluate the extent to which OMB, DHS, and NIST have implemented their government-wide FISMA requirements. GAO categorized information security deficiencies as reported by 16 randomly selected agencies and their IGs according to the elements of an information security program; evaluated IG reports for 24 CFO Act agencies; examined OMB, DHS, and NIST documents; and interviewed agency officials. During fiscal year 2018, many federal agencies were often not adequately or effectively implementing their information security policies and practices. For example, most of the 16 agencies GAO selected for review had deficiencies related to implementing the eight elements of an agency-wide information security program required by the Federal Information Security Modernization Act of 2014 (FISMA) (see figure) . Further, inspectors general (IGs) reported that 18 of the 24 Chief Financial Officers (CFO) Act of 1990 agencies did not have effective agency-wide information security programs. GAO and IGs have previously made numerous recommendations to agencies to address such deficiencies, but many of these recommendations remain unimplemented. With certain exceptions, the Office of Management and Budget (OMB), Department of Homeland Security (DHS), and National Institute of Standards and Technology (NIST) were generally implementing their government-wide FISMA requirements, including issuing guidance and implementing programs that are intended to improve agencies' information security. However, OMB has not submitted its required FISMA report to Congress for fiscal year 2018 and has reduced the number of agencies at which it holds CyberStat meetings from 24 in fiscal year 2016 to three in fiscal year 2018—thereby restricting key activities for overseeing agencies' implementation of information security. Also, OMB, in collaboration with the Council of Inspectors General for Integrity and Efficiency (CIGIE), did not include a metric for system security plans, one of the required information security program elements, in its guidance on FISMA reporting. As a result, oversight of agencies' information security programs was diminished. GAO is making three recommendations to OMB to (1) submit its FISMA report to Congress for fiscal year 2018, (2) expand its coordination of CyberStat meetings with agencies, and (3) collaborate with CIGIE to update the inspector general FISMA reporting metrics to include assessing system security plans. OMB generally agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Medication synchronization is a process whereby a pharmacist aligns the refill dates of two or more of a patient’s medications to a single day each month—referred to as the synchronization date. Patients who are interested in medication synchronization must enroll or opt into the service, if offered at their pharmacy. To initiate medication synchronization, the pharmacist selects an anchor medication to which the other medications are synchronized, and dispenses short fills—that is, a quantity of less than a month’s supply—so that the patient has enough medication until the next synchronization date. Figure 1 illustrates the process by which a pharmacist may synchronize three medications for a patient. Before each synchronization date, the pharmacist generally contacts the patient to determine if the patient has had any changes in his or her medications or medical history. The pharmacist then makes any needed adjustments so that the patient can continue to pick up his or her medications on the synchronization date and avoid disruptions in their medication regimen. In 2017, national spending on prescription drugs dispensed by pharmacies totaled over $330 billion. Medicare accounted for over $100 billion and private health plans accounted for over $140 billion of total spending on prescription drugs that year. Medicare provides prescription drug coverage under Part D, a voluntary program in which beneficiaries can elect to enroll. In February 2019, about 45 million or three-fourths of Medicare beneficiaries were enrolled in Part D plans—including stand-alone prescription drug plans and Medicare Advantage prescription drug plans, which combine medical and prescription drug benefits. In comparison, in 2017, about 200 million patients were enrolled in a private health plan that provides prescription drug coverage, among other benefits, according to CMS. Health plans that provide prescription drug coverage interact with both patients and pharmacies. For patients, health plans may vary their benefits with regards to cost-sharing arrangements—such as copayments for medications—and quantity limits for medications covered—such as restricting the dosage or number of refills of a medication provided within a given period of time. For pharmacies, health plans pay pharmacies a share of the medication costs and a dispensing fee for the pharmacies’ administrative costs in preparing and dispensing the medication. Limited information available indicates that the use of medication synchronization has increased, but comprehensive data on its use by pharmacies and patients do not exist. Among the 22 peer-reviewed studies we identified, one study reported that the use of medication synchronization increased among pharmacies and patients. Specifically, the study examined survey data on the use of medication synchronization in retail pharmacies and reported that the number of retail pharmacies using medication synchronization increased from 3,324 in 2013 to 5,534 in 2014, a 66 percent increase. In addition, the study found that the number of patients using medication synchronization at these retail pharmacies increased from 124,608 in 2013 to 438,100 patients in 2014. We did not identify other studies that examined the use of medication synchronization. Officials from all five selected pharmacies that reported using medication synchronization told us that their pharmacies have increased their use of medication synchronization, but they generally could not provide us with data on their patients’ use of medication synchronization over time. The pharmacies included three retail pharmacy chains—two large national chains and one mid-size regional chain—one independent pharmacy, and one mail order pharmacy. Officials from four of these pharmacies told us that they started using medication synchronization within the last 5 years; officials from the fifth pharmacy told us that they started using medication synchronization in 2011. For example, officials from the two large retail pharmacy chains, each with about 10,000 pharmacies nationwide, told us that they first piloted medication synchronization to a small number of pharmacies in either 2015 or 2016. One of these chains now uses medication synchronization at all of its pharmacies, and the other is in the process of doing so. Officials from the mid-size retail pharmacy chain stated that they piloted medication synchronization in 2011 with about 2,500 patients enrolled across 50 pharmacies. They have since expanded it to about 83,000 patients across all their more than 90 pharmacies. Seven other stakeholders, including those representing patients and pharmacies, also told us that the use of medication synchronization has increased in recent years, but generally did not provide data on the increase. In addition, officials from an organization representing pharmacies told us that as of 2018, approximately 80 percent of independent pharmacies offered medication synchronization; however, they could not provide data from prior years. Limited information exists on the effects of medication synchronization, but available studies and stakeholders indicate several potential benefits, primarily for patients. According to CMS officials, CMS does not have data on the effects of medication synchronization, such as patient medication adherence; other stakeholders we interviewed indicated that such national data do not exist. Seventeen of the 22 peer-reviewed studies we identified evaluated the effects of medication synchronization—14 of these studies evaluated effects for patients and the rest for pharmacies and health plans. However, the data reported by these studies are limited in scope and are not generalizable to broader populations. Twelve of 14 peer-reviewed studies evaluating the potential effects of medication synchronization for patients reported two potential benefits— improved medication adherence or improved medical outcomes. Improved medication adherence. Twelve peer-reviewed studies that evaluated the potential effects of medication synchronization on patients’ adherence reported that medication synchronization improved adherence. For example, nine of the 12 studies compared medication adherence among patients using and not using medication synchronization and found that medication adherence was greater among patients using medication synchronization—one of the most recent studies showed adherence was 3 percent higher for those using medication synchronization. Two studies compared medication adherence among patients before and after using medication synchronization and found that adherence improved after synchronization was started—the most recent of these studies showed an improvement of 2 percent in average adherence after a year of enrollment. The last study reported that 56 percent of patients surveyed stated that they would be more adherent to their medications if their refills were synchronized. In addition, eight of these studies evaluated the effects of medication synchronization for patients with different chronic conditions and found differences by type of chronic condition. According to the studies, medication adherence improves as a result of medication synchronization because it simplifies the refilling process. Improved medical outcomes. One peer-reviewed study reported that medication synchronization may also lead to improved medical outcomes for patients. The study found that rates of hospitalization and emergency department visits and rates of outpatient visits were 9 percent and 3 percent lower, respectively, among patients using medication synchronization compared with those who were not. Stakeholders also cited improved medication adherence and medical outcomes as potential benefits for patients, and also identified additional benefits that may result from medication synchronization. Specifically, 14 of the 15 stakeholders representing patient and pharmacy organizations and selected pharmacies we interviewed said that medication synchronization may help improve patients’ medication adherence, and 12 of these stakeholders said that it may improve patients’ medical outcomes. These stakeholders also indicated other potential benefits for patients: Improved convenience. Medication synchronization improves convenience for patients—for example, by reducing the number of trips patients need to make to the pharmacy or making it easier to manage their medications, according to 10 of the 15 stakeholders representing patient and pharmacy organizations and selected pharmacies. Fewer trips to the pharmacy help to minimize the need for transportation arrangements, which is particularly important for older patients, patients who live in rural areas, and patients who lack reliable transportation. Five of the 10 stakeholders added that medication synchronization simplifies patients’ experience with managing their medications—patients no longer need to keep track of multiple refill dates for all their medications. Under medication synchronization, the pharmacists proactively perform this work and send reminders to the patients. Increased interaction between pharmacists and patients. Seven of 15 stakeholders representing patient and pharmacy organizations and selected pharmacies told us that medication synchronization increases the interaction patients have with their pharmacists, which may help patients better manage their medication regimens and improve their overall health. For example, a stakeholder representing pharmacies said that prior to the medication synchronization date, pharmacies generally contact patients to confirm that their medications should be filled; as part of this outreach, they also inquire about any changes in the patients’ medical history or therapy. If such changes are identified, pharmacists follow up with patients, and their physicians if necessary, to ensure that patients receive refills reflecting any necessary medication changes. In addition, according to some stakeholders, if a consultation is also provided on the medication synchronization date, pharmacists have more opportunities to answer patients’ questions about medication use, provide counseling, and offer patients other auxiliary services. For example, some stakeholders told us that pharmacists may provide screenings for blood pressure and diabetes, or recommend immunizations to patients when they pick up their medications. Because pharmacists regularly assess patients’ medical history in preparing for medication synchronization, they can target patients who may be at high risk for medical problems or immunization-preventable diseases. Regarding pharmacies, some studies and stakeholders identified the following potential benefits of medication synchronization. Operational efficiencies. Three peer-reviewed studies reported that medication synchronization can lead to operational efficiencies. For example, one study reported that medication synchronization can help pharmacists better manage inventory and personnel costs and improve workflow. Nine out of 12 stakeholders representing pharmacy organizations and selected pharmacies also said that medication synchronization can lead to improvements in operational efficiencies. For example, officials from one organization representing pharmacies said that pharmacists save time when they can dispense all of a patient’s medications at one time instead of several times throughout the month. Increased marketability of pharmacies to health plans. According to one peer-reviewed study and five of the 12 stakeholders representing pharmacy organizations and selected pharmacies, the extent to which a pharmacy’s medication synchronization program improves patient care, such as by improving medication adherence, may make the pharmacy more desirable to health plans. For example, according to the study and officials from one organization representing pharmacies, health plans may include pharmacies with highly adherent patients in the plans’ preferred pharmacy networks. Pharmacies in preferred pharmacy networks can offer lower medication prices, attracting more customers. Increased revenue. Three peer-reviewed studies and five of the 12 stakeholders representing pharmacy organizations and selected pharmacies reported that, to the extent that medication synchronization can improve patients’ medication adherence, it can also lead to increased pharmacy revenues generally because of an increase in filled prescriptions. For example, one of the three peer- reviewed studies reported that medication synchronization resulted in an average increase in medication adherence of almost 5 percent over the first 6 months of its use. Similarly, an industry study found that medication synchronization leads to an additional 20 fills per patient per year, and may lead to an average of $1,120 of additional revenue per enrolled patient annually. The three peer-reviewed studies and the industry study did not examine the causes of the increase in prescription fills, but their authors generally attributed the increase to the improved adherence of patients using medication synchronization. Two of the three studies reported that medication synchronization can increase pharmacy revenues generally because of an increase in filled prescriptions. In addition, officials from four stakeholders representing pharmacy organizations and selected pharmacies told us that pharmacies that use medication synchronization can leverage these opportunities to speak with patients and offer additional services, such as immunizations; these services can further help increase pharmacy revenue. Regarding health plans, some studies and stakeholders identified the following potential benefits of medication synchronization. Higher Medicare quality performance scores. Three peer-reviewed studies reported that medication synchronization can potentially improve health plans’ Medicare quality performance scores. CMS assesses the quality performance of Part D plans using information on various measures, such as adherence to medications for diabetes, high cholesterol, or hypertension. Specifically, CMS rates the plans’ performance using a star rating system, which gives each plan a score of between one and five stars, with five stars being the highest rating. Medication adherence measures are triple weighted in the calculation of a plan’s overall rating. Plans with the highest star ratings are rewarded with member enrollment incentives, while plans with lower star ratings are penalized. In addition, Medicare Advantage plans with high ratings may also receive financial bonuses from Medicare. To the extent that there are improvements in beneficiaries’ medication adherence as a result of medication synchronization, health plans may experience improved performance ratings and the commensurate financial benefits. However, only one of the four stakeholders representing health plan organizations and selected health plans indicated improved Medicare quality performance scores as a potential benefit of medication synchronization. Reduced medical costs. Medication synchronization may also benefit health plans by reducing their overall medical costs, according to one peer-reviewed study. The study found that medication synchronization can result in significant savings in medical costs for health plans, despite the increase in medication costs to the health plan. Specifically, the study reported that medical savings per additional dollar spent on medications under medication synchronization ranged from approximately $1 to $37, depending on the medication. According to the study, health plans could potentially experience such reduced medical costs as a result of medication synchronization because when patients are adherent to their medications, they may decrease their utilization of healthcare services. However, only one of the four stakeholders representing health plan organizations and selected health plans cited this as a potential benefit. A small number of studies and several stakeholders indicated that there are some potential limitations associated with medication synchronization. For example, Patients. One peer-reviewed study indicated that medication synchronization may not be beneficial for all patients. Similarly, 14 of the 15 stakeholders representing patient and pharmacy organizations and selected pharmacies said that not every patient may want to use medication synchronization. For example, 12 of the 14 stakeholders said that some patients may not be able to afford paying all copayments for their medications at one time each month, which deters them from using medication synchronization. In addition, one stakeholder said that some patients prefer going to the pharmacy regularly or consider trips to the pharmacy as opportunities for social interaction and may not be interested in medication synchronization. Pharmacies. One peer-reviewed study reported that using medication synchronization is time- and labor-intensive for pharmacies. Specifically, the study reported that almost 60 percent of pharmacists surveyed indicated that implementing medication synchronization involves a significant change in a pharmacy’s workflow. Seven of 12 stakeholders representing pharmacy organizations and selected pharmacies said this was because of several challenges. For example, it may be complicated to set up the initial synchronization, determine the best anchor medication and synchronization date, or adjust patients’ medication synchronization because of changes in their medical needs or therapy. In addition, pharmacists may have to conduct extensive follow-up with health plans because health plans may not be consistent in how they process pharmacies’ claims that involve short fills. For example, private health plans may initially deny coverage of short fills; such denials may require the pharmacist to expend additional resources to follow-up with the health plan to obtain approval for the short fill. Health plans. Officials from the two selected health plans told us that they do not require their Part D network pharmacies to use medication synchronization, nor do they compensate pharmacies for providing these services. While all stakeholders representing health plan organizations and selected health plans said that they view medication synchronization as having the potential to improve patients’ medication adherence and health outcomes, two of these stakeholders noted the lack of data explicitly tying medication synchronization to improved patient medication adherence, medical outcomes, and overall medical costs. Our review shows that a CMS regulation and laws related to prescription drug coverage in five selected states may support the use of medication synchronization. For example, CMS and the five selected states allow for reduced patient cost sharing for short fills needed to synchronize their medications. CMS does not have a formal medication synchronization program or policy for Medicare; however, a CMS regulation related to prescription drug benefits may support medication synchronization by reducing beneficiary cost sharing for certain amounts dispensed, according to officials. Specifically, CMS issued a regulation that, starting in 2014, required Medicare Part D plans to establish a daily cost-sharing rate (for example, a prorated copayment) when a beneficiary receives less than a month’s supply of a prescription medication—generally referred to as a short fill. According to CMS, the primary goal of the regulation was to reduce medication cost and waste—such as by allowing beneficiaries to initially receive a short fill of a new medication so that they can assess, in consultation with their providers, the efficacy of the medication and any associated adverse side effects. Because short fills may be needed to initially synchronize multiple medications to the same refill date, the prorated copayment may reduce the financial burden on beneficiaries who require these fills, according to CMS officials. In addition, officials from a selected pharmacy and officials from a technology vendor added that from a value perspective, beneficiaries may be reluctant to enroll in medication synchronization if they had to pay a full copayment for less than a month’s supply of medication. For example, as illustrated in figure 2, to initiate medication synchronization for a beneficiary taking three medications, each with a different refill date, the pharmacist may dispense short fills for two of the three medications. In this case, the pharmacist may dispense 8 days’ supply of one medication and 3 days’ supply of another medication. Prior to the regulation, the beneficiary would have paid $45 in copayments for these two short fills, as compared to $7 with prorated copayments—a difference of $38. The five selected states—Georgia, Illinois, Maine, Texas, and Washington—enacted laws within the last 4 years that may support medication synchronization. Specifically, these laws: Require insurance coverage of short fills. Laws in all five selected states require health plans in their state to provide coverage for medication short fills. These laws may also support medication synchronization by allowing health plans and pharmacies to work around certain plan policies that may impose limits on medication refills. Specifically, officials from a technology vendor told us that some health plans may impose limits on the number of refills that can be dispensed in a month. For example, if a patient is taking five medications and is limited to five refills a month, a short fill would count towards that limit and the patient may not then be able to get all of his or her medications covered by the health plan that month. Such laws allow the health plan and pharmacy to work around these quantity limits so that the patient can receive the needed short fills to synchronize all of his or her medications. Additionally, two states— Maine and Texas—specifically require their health plans to allow pharmacies to override denials related to refilling a prescription too soon. A pharmacy may receive such denials when refilling a prescription after having just filled it—for example, dispensing a short and full refill of a medication too close together. Officials from two selected pharmacies and an organization representing pharmacies told us that such laws also help to reduce the time and resources that pharmacies otherwise would have expended on addressing issues with these drug claims. Require prorated cost sharing for short fills. Like CMS’s regulation, laws in all five selected states require health plans in their state to prorate a patient’s cost sharing, such as a copayment, when the patient receives a short fill of a medication. Officials from four selected pharmacies told us such laws help reduce the financial burden on patients when they first have their medications synchronized. Without such a law, patients would have paid a full copayment for these medications, which may have discouraged or prevented some patients from enrolling in medication synchronization. Prohibit prorated dispensing fees for short fills. Laws in four of the five selected states prohibit health plans in their state from paying pharmacies a prorated dispensing fee for medication short fills. Pharmacies receive a dispensing fee from health plans for each prescription they fill to cover the pharmacies’ administrative costs of preparing and dispensing a fill. The dispensing fee is in addition to the reimbursement pharmacies receive from health plans for the costs of the medications. In states without this law, health plans may prorate the dispensing fee for short fills—that is, pay a lower fee because a smaller quantity of medications (for example, 10 pills rather than 30 pills) is dispensed. Officials from a technology vendor and an organization representing pharmacies told us that ensuring that a health plan pays a full dispensing fee provides an incentive for pharmacies to use medication synchronization. They explained that a pharmacy’s administrative costs of dispensing a medication remains the same, regardless of the quantity dispensed. Require medication synchronization process or policy. Laws in two of the five selected states—Texas and Washington—require health plans in their state to establish a process or policy for providing medication synchronization services. Both states require that, as part of this process or policy, the pharmacist or prescribing physician must ensure that medication synchronization is appropriate or in the best interest of the patient before the process is used. In addition to approval from both the pharmacist and physician, Texas also requires that the health plan and patient approve the medication synchronization plan. Officials from an organization representing pharmacies said that involving all these entities further helps to ensure the appropriateness of medication synchronization for a particular patient. While stakeholders generally told us that these laws have helped to support medication synchronization, they also said that the absence of such laws has not prevented pharmacies from using it in other states. For example, the five selected pharmacies that reported using medication synchronization—including three pharmacy chains—offered these services in at least some states without such laws. Additionally, officials from a selected pharmacy told us that they continue to offer medication synchronization despite receiving a prorated dispensing fee for short fills. We provided a draft of this report to the Department of Health and Human Services (HHS). HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of HHS, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or CosgroveJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Appendix I: Information about Stakeholders Interviewed We interviewed organizations reflecting a range in interests: Six organizations representing pharmacies or pharmacists Three organizations representing patients Two organizations representing health plans One organization representing the pharmaceutical industry 1 We interviewed one organization that specializes in Medicare issues and conducts analysis related to access to and quality of care, among other things. 5 We interviewed selected pharmacies that reported using medication synchronization Two large national retail chains One mid-size regional retail chain One small independent, single-store, retail pharmacy One mail order pharmacy 1 We interviewed a selected large national retail chain pharmacy that reported not using medication synchronization. 2 We interviewed two selected Medicare health plans that offer prescription drug coverage (Part D) that are among the top five Part D plans covering the largest Medicare populations—the combined Medicare Part D enrollment in these two plans totaled almost 14 million—or 31 percent of all Part D beneficiaries, as of August 2018. 2 We interviewed two selected medication synchronization vendors that contract with pharmacies to provide technological support in performing tasks such as identifying patients who would benefit from medication synchronization; determining the anchor medication; and setting up automated reminder to patients, in advance of their prescription refills. We identified these vendors in peer-reviewed studies on medication synchronization or through interviews with stakeholders. 5 We interviewed experts in medication synchronization, identified in peer-reviewed studies on medication synchronization or through interviews with stakeholders. In addition to the contact named above, Tim Bushfield, Assistant Director; Pauline Adams, Analyst-in-Charge; George Bogart; Nina Daoud; Krister Friday; Melissa Trinh-Duong Ostergard; and Vikki Porter made key contributions to this report. Andrews, S. B., T. R. Marcy, B. Osborn, and L. G. Planas. “The Impact of an Appointment‐Based Medication Synchronization Programme on Chronic Medication Adherence in an Adult Community Pharmacy Population.” Journal of Clinical Pharmacy and Therapeutics, vol. 42, no. 4 (2017): pp. 461-466. Barnes, Brenda, Ana L. Hincapie, Heidi Luder, James Kirby, Stacey Frede, and Pamela C. Heaton. “Appointment-Based Models: A Comparison of Three Model Designs in a Large Chain Community Pharmacy Setting.” Journal of the American Pharmacists Association, vol. 58, no. 2 (2018): pp. 156-162. Butler, Kendra T., Janelle F. Ruisinger, Jessica Bates, Emily S. Prohaska, and Brittany L. Melton. “Participant Satisfaction with a Community-Based Medication Synchronization Program.” Journal of the American Pharmacists Association, vol. 55, no. 5 (2015): pp. 534-539. Dao, Nancy, Sun Lee, Micah Hata, and Lord Sarino. “Impact of Appointment-Based Medication Synchronization on Proportion of Days Covered for Chronic Medications.” Pharmacy, vol. 6, no. 2 (2018): pp. 1- 9. DiDonato, Kristen L., Kristin R. Vetter, Yifei Liu, Justin R. May, and D. Matthew Hartwig. “Examining the Effect of a Medication Synchronization or an Education Program on Health Outcomes of Hypertensive Patients in a Community Pharmacy Setting.” INNOVATIONS in Pharmacy, vol. 5, no. 3 (2014): pp. 1-9. Doshi, Jalpa A., Raymond Lim, Pengxiang Li, Peinie P. Young, Victor F. Lawnicki, Joseph J. State, Andrea B. Troxel, and Kevin G. Volpp. “A Synchronized Prescription Refill Program Improved Medication Adherence.” Health Affairs, vol. 35, no. 8 (2016): pp. 1504-1512. Doshi, Jalpa A., Raymond Lim, Pengxiang Li, Peinie P. Young, Victor F. Lawnicki, Andrea B. Troxel, and Kevin G. Volpp. “Synchronized Prescription Refills and Medication Adherence: A Retrospective Claims Analysis.” American Journal of Managed Care, vol. 23, no. 2 (2017): pp. 98-104. Ghassemi, Emily, Jennifer Smith, Laura Owens, Charles Herring, and Melissa Holland. “Relationship Between Medication Synchronization and Antiretroviral Adherence.” Journal of the American Pharmacists Association, vol. 58, no. 4 (2018): pp. S78-S82. Girdish, Charmaine, William Shrank, Sarah Freytag, David Chen, Doug Gebhard, Andrew Bunton, Niteesh Choudhry, and Jennifer Polinski. “The Impact of a Retail Prescription Synchronization Program on Medication Adherence.” Journal of the American Pharmacists Association, vol. 57, no. 5 (2017): pp. 579-584. Hinson, Jessica L., Gretchen K. Garofoli, and Betsy M. Elswick. “The Impact of Medication Synchronization on Quality Care Criteria in an Independent Community Pharmacy.” Journal of the American Pharmacists Association, vol. 57, no. 2 (2017): pp. 236-240. Holdford, David A., and Timothy J. Inocencio. “Adherence and Persistence Associated with an Appointment-Based Medication Synchronization Program.” Journal of the American Pharmacists Association, vol. 53, no. 6 (2013): pp. 576-583. Holdford, David, and Kunal Saxena. “Impact of Appointment-Based Medication Synchronization on Existing Users of Chronic Medications.” Journal of Managed Care & Specialty Pharmacy, vol. 21, no. 8 (2015): pp. 662-669. Krumme, Alexis A., Robert J. Glynn, Sebastian Schneeweiss, Joshua J. Gagne, J. Samantha Dougherty, Gregory Brill, and Niteesh K. Choudhry. “Medication Synchronization Programs Improve Adherence to Cardiovascular Medications and Health Care Use.” Health Affairs, vol. 37, no. 1 (2018): pp. 125-133. Krumme, Alexis A., Danielle L. Isaman, Samuel F. Stolpe, J. Samantha Dougherty, and Niteesh K. Choudhry. “Prevalence, Effectiveness, and Characteristics of Pharmacy-Based Medication Synchronization Programs.” American Journal of Managed Care, vol. 22, no. 3 (2016): pp. 179-186. Luder, Heidi R., Natalie Kunze, Pamela C. Heaton, and Stacey M. Frede. “An Appointment-Based Model to Systematically Assess and Administer Vaccinations.” Journal of the American Pharmacists Association, vol. 58, no. 3 (2018): pp. 290-295. Nguyen, E., and D. M. Sobieraj. “The Impact of Appointment‐Based Medication Synchronization on Medication Taking Behaviour and Health Outcomes: A Systematic Review.” Journal of Clinical Pharmacy and Therapeutics, vol. 42, no. 4 (2017): pp. 404-413. Patterson, Julie, and David Holdford. “Understanding the Dissemination of Appointment-Based Synchronization Models Using the CFIR Framework.” Research in Social and Administrative Pharmacy, vol. 13, no. 5 (2017): pp. 914-921. Patterson, Julie A., David A. Holdford, and Kunal Saxena. “Cost-Benefit of Appointment-Based Medication Synchronization in Community Pharmacies.” American Journal of Managed Care, vol. 22, no. 9 (2016): pp. 587-593. Renfro, Chelsea P., Michael Patti, Jordan M. Ballou, and Stefanie P. Ferreri. “Development of a Medication Synchronization Common Language for Community Pharmacies.” Journal of the American Pharmacists Association, vol. 58, no. 5 (2018): pp. 515-521. Ross, Alexander, Humaira Jami, Heather A. Young, and Richard Katz. “Sync and Swim: The Impact of Medication Consolidation on Adherence in Medicaid Patients.” Journal of Primary Care & Community Health, vol. 4, no. 4 (2013): pp. 240-244. White, Nicole D. “Pharmacy Medication Synchronization Service Works to Improve Medication Adherence.” American Journal of Lifestyle Medicine, vol. 10, no. 6 (2016): pp. 385-387. Witry, Matthew, and Thao Hoang. “Community Pharmacist Attitudes on Medication Synchronization Programs.” INNOVATIONS in Pharmacy, vol. 8, no. 2 (2017): pp. 1-7.", "summary": "Medication adherence—that is, taking medications as prescribed—is important because not doing so increases the risk of hospitalization and can result in avoidable medical costs. According to some pharmacy industry groups, medication synchronization may help improve medication adherence, particularly for patients with multiple chronic conditions. More than 40 percent of Medicare beneficiaries had two or more chronic conditions in 2015. Congress included a provision in the Bipartisan Budget Act of 2018 for GAO to review and report on the use of medication synchronization. In this report, GAO examines (1) what is known about the use and potential effects of medication synchronization and (2) steps CMS and selected states have taken to support its use. GAO identified and reviewed 22 peer-reviewed studies on medication synchronization. In addition, GAO interviewed CMS officials and 30 stakeholders to gather a wide range of perspectives on medication synchronization. Among others, GAO interviewed six selected pharmacies and two selected Medicare health plans. GAO also reviewed CMS regulations as well as medication synchronization laws from five selected states that vary by geographic region. GAO provided a draft of this report to the Department of Health and Human Services, which provided technical comments. GAO incorporated these comments, as appropriate. Medication synchronization is a process whereby a pharmacist aligns the refill dates of two or more of a patient's medications to a single day (see figure below). GAO found that no comprehensive national data exist on the extent to which medication synchronization has been used or its potential effects. However, limited information suggests that the use of medication synchronization has increased in recent years and that it may have benefits. According to a study published in the American Journal of Managed Care that examined survey data on retail pharmacies, the number of pharmacies using medication synchronization increased from 3,324 in 2013 to 5,534 in 2014. Most of the studies that GAO identified found positive effects from medication synchronization, primarily on patients. For example, a 2018 study reported a 3 percent improvement in medication adherence among patients using medication synchronization than those who were not. Several stakeholders also identified potential limitations of using medication synchronization. For example, some patients may not be able to afford paying all the copayments for their medications at one time each month, and some patients prefer the social interaction of multiple trips to the pharmacy each month. The Centers for Medicare & Medicaid Services (CMS) issued a regulation and some states enacted laws that may help support the use of medication synchronization. While CMS does not have a formal medication synchronization policy for Medicare, a CMS regulation allows for reduced beneficiary cost sharing (for example, a lower copayment) when the beneficiary receives less than a month's supply of a medication. Similar laws pertain to private health plans that provide prescription drug coverage for patients in the five states GAO selected—Georgia, Illinois, Maine, Texas, and Washington. Such measures support medication synchronization because initially aligning the refill dates of multiple medications may require one or more of these medications to be refilled with a quantity that is less than a month's supply. Officials from CMS and four of the selected pharmacies said that lowering the copayments for these refills reduces the financial burden on patients when they first have their medications synchronized. They noted that requiring full copayments for a shorter supply may have discouraged or prevented patients from using medication synchronization.", "document_type": "gao"}
{"report": "From January through August 2017, the Navy suffered four significant mishaps at sea that resulted in the death of 17 sailors and hundreds of millions of dollars in damage to Navy surface ships (see fig.1). More recently, the Navy experienced two incidents during which Navy surface ships collided. First, on February 5, 2019, a Ticonderoga-class guided missile cruiser—USS Leyte Gulf (CG 55)—collided with a Navy resupply ship—the USNS Robert E. Peary (T-AKE 5)—while conducting an underway replenishment operation off the coast of Florida. Second, on June 21, 2019, a Freedom-class Littoral Combat Ship—the USS Billings (LCS 15)—struck a merchant ship while leaving a pier in Montreal, Canada. According to Navy officials, these recent incidents did not result in serious damage to the ships or injuries to the crews but they demonstrate the need for continued focus and attention on safe ship driving. As of March 2019, the Navy had approximately 8,400 SWOs—Navy officers whose training and primary duties focus on the operation of Navy ships at sea and the management of various shipboard systems. The Navy expects SWOs to progress over the course of their careers from Division Officers driving ships, to Department Heads participating in combat operations, to Executive Officers managing ship crews, and to Commanding Officers overseeing operations. Figure 2 below outlines a SWO’s career progression and associated shipboard duties. The Commander, Naval Surface Forces, in coordination with Office of the Chief of Naval Operations, Surface Warfare Officers School Command, and Navy Personnel Command, manages and provides ship-driving training to SWOs throughout their careers. Initially, the primary focus of a Division Officer is on leading sailors and developing ship-driving competency, ultimately working toward qualification as an Officer of the Deck and Surface Warfare Officer. Therefore, a large part of initial SWO training focuses on leading a division and developing the ship-driving skills needed to qualify and perform as an Officer of the Deck. This training is provided during a SWO’s initial training—in the Basic Division Officer Course—offered in Norfolk, Virginia, and San Diego, California, and is a mix of classroom and simulator-based training. After completing the Basic Division Officer Course, SWO candidates begin their first at-sea assignment as Division Officers. Division Officers have three primary roles aboard a ship: 1. They support ship-driving operations. New Division Officers gain ship- driving experience in pursuit of the qualification to stand as Officers of the Deck. Once qualified as Officers of the Deck, they lead watch teams in driving ships. 2. They support ship department operations under the supervision of Department Heads, and are responsible for a portion of ship equipment and operations. 3. They lead a division of approximately 12 to 50 enlisted personnel within departments, and are responsible for the administrative and supervisory duties for divisions. In addition to their Division Officer responsibilities, new Division Officers are expected to earn qualification as a SWO by completing required education and meeting experience standards, as well as gaining watchstanding experience and demonstrating proficiency in the execution of their duties, according to Navy officials. These qualifications include Officer of the Deck and Combat Information Center Watch Officer, where an officer assists in observing and analyzing information of importance for combat, among others. Navy officials stated that after an officer completes their qualifications, their Commanding Officer reviews the officer’s ability and experiences, and can grant the candidate SWO qualification. In June 2010, we reviewed Navy policies for surface force training— including initial SWO ship-driving training—and found that the Navy had reduced and altered initial SWO training as an efficiency measure, but lacked performance measures and data necessary to evaluate the impact of changes to training programs. We found that in 2003, the Navy had replaced its 6-month Division Officer course consisting of classroom and simulator training with software-based training where new Division Officers were instead expected to learn SWO skills from computer-based education software while onboard their first ship. The Navy said this change saved about $50 million annually, but we found that the Navy lacked outcome-based performance measures to evaluate the effects of these changes to training on officer performance. We recommended that the Navy develop metrics to measure the effects of training on SWO job performance, knowledge, skills, and abilities. The Navy concurred with this recommendation but did not implement the recommendation for the software training or for subsequent training programs. The Navy has enhanced ship-driving training for SWOs at the early stages of their careers following the 2017 collisions at sea, and by 2021 plans to triple the number of ship-driving training hours when compared with the amount of training SWOs were required to receive prior to the collisions. The Navy’s plans to increase ship-driving proficiency hinge on the completion of two new simulator-based training facilities—the Mariner Skills Training Centers—which are planned to be completed in June 2021 (San Diego, California) and in January 2023 (Norfolk, Virginia). Overall, the Navy plans to invest more than $467 million to develop new ship- driving training courses, build simulator facilities, and deliver the training through fiscal year 2025. Prior to the 2017 ship collisions, SWOs were required to complete 174 hours of ship-driving training during their Division Officer assignment by attending the Basic and Advanced Division Officer training courses. Following the collisions, the Navy increased the amount of required ship- driving training in these two courses to 203 hours. In June 2019, the Navy added a 4-week ship-driving course—the Junior Officer of the Deck course—that focused exclusively on building ship-driving skills. This course added 158 hours of required classroom and simulator ship-driving training. In June 2021, the Navy plans to expand the curriculum of the Junior Officer of the Deck course and rename it the Officer of the Deck Phase I course, and add an additional 3-week Officer of the Deck Phase II course. These two courses will add an additional 185 hours of required ship-driving training for Division Officers in preparation for their first and second at-sea assignments. Once these ship-driving training courses are in place, Division Officers will be required to complete a total of 535 hours of training—triple (a threefold increase in) the number of ship- driving training hours SWOs were required to complete prior to the 2017 collisions (see fig. 3). Below are detailed descriptions of the changes completed and planned to enhance ship-driving training. Basic Division Officer Course. From November 2017 through January 2019, the Surface Warfare Officers School Command changed the Basic Division Officer Course—a 9-week course for new SWO candidates—by increasing the required hours of classroom instruction and simulator training by 12 percent, and broadening the course curriculum. Specifically, prior to the 2017 collisions, SWO candidates were required to spend 113 hours (81 hours of classroom instruction and 32 hours in simulators) in this course to develop their ship-driving skills. After January 2019, however, SWO candidates were required to spend 126 hours (89 hours of classroom instruction and 37 hours in simulators) to develop their ship-driving skills. Regarding added course content, the Surface Warfare Officers School Command added subject matter including additional training on the internationally accepted ship-driving standards that govern ship maneuvers; radar navigation; and the tools used to aid ship-driving. Advanced Division Officer Course. From November 2017 through January 2019, the Surface Warfare Officers School Command changed the Advanced Division Officer Course—a 5-week course for SWOs returning from their first at-sea assignment—to improve ship-driving skills by increasing the required hours of simulator training from 24 to 36 hours. Prior to the 2017 collisions, SWOs were required to spend 61 hours (37 hours in the classroom and 24 hours in simulators) refining their ship- driving skills in this course. As of January 2019, SWOs were required to spend 77 hours (41 hours in the classroom and 36 hours in simulators) developing and honing their ship-driving skills. Surface Warfare Officers School Command officials also added subjects to classroom time to build on the subject matter presented in the Basic Division Officer Course, including more complex ship-driving techniques and advanced radar navigation. Surface Warfare Officers School Command plans to reduce the hours of training in this course once the Officer of the Deck Phase II course comes online in 2021. Junior Officer of the Deck course. In June 2019, Surface Warfare Officers School Command provided this new 4-week course for the first time—the course having been developed after the 2017 collisions and focused predominately on building ship-driving skills. The Junior Officer of the Deck course takes place after SWO candidates complete the Basic Division Officer Course and before they begin their first at-sea assignment. SWOs taking this course are required to complete 158 hours of classroom and simulator training designed to increase their ship-driving skills by exposing them to a variety of scenarios involving different maneuvers, and varying sea and weather conditions. The Navy plans to expand this course into a 6 week ship-driving training course (Officer of the Deck Phase I), scheduled to begin in June 2021. Officer of the Deck Phase I course. According to Commander, Naval Surface Forces documentation, the Junior Officer of the Deck course will expand into the Officer of the Deck Phase I course. Officer of the Deck Phase I is under development and will be 6 weeks long (an additional 2 weeks longer than Junior Officer of the Deck), and will take place after SWO candidates complete the Basic Division Officer Course and before they begin their first at-sea assignment. Officer of the Deck Phase I is intended to build on the Junior Officer of the Deck curriculum by increasing the required number of ship-driving training hours from 158 to 241, and expanding the course content to include instruction on more advanced radar navigation techniques. Surface Warfare Officers School Command and Commander, Naval Surface Forces officials expect the Officer of the Deck Phase I course to begin in June 2021. Officer of the Deck Phase II course. According to Navy documentation, the Officer of the Deck Phase II course that is under development will be 3 weeks long, and will take place after SWOs have completed their first at-sea assignment and before they attend the Advanced Division Officer Course. This course is intended to continue the development of ship- driving skills through an additional 102 hours of required classroom and simulator training. Surface Warfare Officers School Command and Commander, Naval Surface Forces officials stated that Officer of the Deck Phase II course could begin as early as June 2021. Mariner Skills Training Centers. According to Commander, Naval Surface Forces and Surface Forces documentation, Surface Warfare Officers School Command will provide the Officer of the Deck Phase I and Phase II courses at the Mariner Skills Training Centers—new simulator-based facilities expedited after the 2017 collisions. Officials from the Office of the Chief of Naval Operations stated that these facilities—including upgraded simulators, the instructors, classrooms, and the curriculum development for the Officer of the Deck Phase I and Phase II courses—will cost approximately $467.5 million through fiscal year 2025. According to Navy officials, construction on the San Diego, California Mariner Skills Training Center will begin in early fiscal year 2020 and will be complete by June 2021 and Norfolk, Virginia Mariner Skills Training Center will begin in fiscal year 2021 and will be complete in January 2023. The Mariner Skills Training Program is based upon the Littoral Combat Ship ship-driving training program, which according to the Navy, provides a balance of classroom, simulation, and shipboard experience. According to Navy officials, since Littoral Combat Ship SWOs serve in rotating crews and have less opportunity to train aboard their ships, the Navy developed the Littoral Combat Ship Training Facility to support SWOs’ training ashore (see fig. 4). The foundation of the Littoral Combat Ship ship-driving program is repetitive training in sophisticated simulators to build ship-driving proficiency. According to the Navy, the effectiveness of this training has been validated over the last 10 years by the superior ship-driving proficiency of Littoral Combat Ship officers during at-sea operations and assessment performance when compared with non- Littoral Combat Ship officers, in many cases. The Navy has relied on a series of added skill checks throughout a SWO’s career to help validate that SWOs have necessary ship-driving and other skills, but has not developed key processes and assessments to evaluate the overall effectiveness of its existing and planned training programs. The Navy is implementing a series of ten skill checks on ship-driving and other mariner tasks at various career points—for example, before a SWO begins leading a ship department and before the SWO takes command of a ship. The Commander, Naval Surface Forces, issued an instruction in September 2018 detailing ten skill checks to be conducted over the course of a SWO’s career to periodically gauge SWOs’ ship-driving skills. These checks, summarized in appendix IV, are to occur at standardized points in a SWO’s career, either during training or at the beginning or conclusion of certain at-sea assignments. Four of the ten checks were already in place at the time the instruction was issued in September 2018, with a preliminary version of a fifth check also in place. According to Navy documentation, three more of the ten checks had also been implemented as of August 2019, and Navy guidance states that the remaining checks are scheduled to be in place by 2021 or earlier. Navy officials stated they were making these checks more rigorous. For example, according to Navy officials, previously Department Heads were allowed to retake the Command Qualification Assessment ship-handling test as many times as they needed to pass the assessment. According to these officials, in 2018 Surface Warfare Officers School Command allowed only three chances to take the test, leading to five of the 256 Department Heads assessed in 2018 to be disqualified from advancing beyond the role of Department Head. Navy officials report that these skill checks are intended to enhance the development and sustainment of ship-driving proficiency across a SWO’s career and to ensure that the changes in training are resulting in competent SWOs at each level of their careers—essentially that SWOs have the skills required to perform their duties. Surface Warfare Officers School Command will administer checks during SWO training on ship- driving to better evaluate individual proficiency and target remediation for those whose performance presents significant concerns. Ship Commanding Officers will also observe and evaluate SWOs on a series of ship-driving scenarios before the completion of their first Division Officer assignment and later as a Department Head to certify that they are prepared for more advanced ship-driving training and responsibilities. While the planned skill checks are designed to help ensure that SWOs have the skills required to perform their duties, senior Navy officials stated that it could take 16 years or more to know if the planned changes to SWO training were effective in increasing Commanding Officer ship- driving proficiency across the fleet. These officials stated that they intend to closely monitor the implementation of changes to the training; however, we found a number of interrelated challenges that limit the Navy’s ability to determine in the near term if the significant investments it is making to expand and enhance SWO ship-driving training are effective. Specifically and described in detail below, in planning an approach for evaluating its efforts, the Navy has not (1) solicited fleet-wide feedback on the quality of the increased ship-driving training, (2) planned to routinely conduct ship- driving competency assessments, (3) provided standard criteria for qualifying Officer of the Deck candidates, and (4) determined how to analyze and use information from logbooks that SWOs are required to complete. The Navy’s Comprehensive Review of Recent Surface Force Incidents—one of the internal reviews completed after the 2017 mishaps—notes the importance of assessing and monitoring performance so that corrective actions can take place. In addition, federal government internal control standards state that management should use quality information and monitoring activities to ensure the agency’s objectives are achieved. Moreover, our prior work on assessing training efforts in the federal government states that an agency should evaluate the effectiveness of its training and development efforts, to include obtaining feedback, assessing competency, and analyzing relevant data. We found that while the Navy collects feedback from certain groups of SWOs, it did not have a formal fleet-wide process to solicit feedback from SWOs on the quality of the increased amount of ship-driving training or to gauge the health of the SWO community. In group discussions we held as part of our review, SWOs identified challenges that Division Officers experience in applying classroom and simulator training to their duties. According to SWOs in 19 of 24 group discussions with Department Heads and Division Officers, Division Officers have challenges in applying the ship-driving training they receive, due to factors such as differences between training curriculum and actual duties, extended lengths of time elapsed between training and application, varying ship-driving opportunities during Division Officer assignments, and difficulty retaining the large volume of course material. SWOs that participated in our discussion groups and interviews identified positive aspects of ship-driving training, as well as concerns about training material. During five of 12 ship group discussions with Division Officers, those Division Officers that had taken the Basic Division Officer Course identified positive aspects of the training such as valuable practical exercises and simulator time. However, SWOs in all 12 Division Officer group discussions also identified challenges related to this training, such as the information covered in training being too broad, and a lack of connection to actual duties on their ship. More experienced Division Officers in four of 12 Division Officer group discussions identified challenges related to the Advanced Division Officer Course, such as insufficient time in ship-driving simulators, and too much time spent covering material that Division Officers were already expected to learn during their first at-sea assignment. Commanding Officers and Executive Officers in seven of 12 interviews, and Department Heads in four of 12 group discussions likewise identified positive aspects of the Basic Division Officer Course, such as improved knowledge of ship operations for Divisions Officers that recently completed the course. However, Commanding Officers and Executive Officers in three of 12 interviews and Department Heads in seven of 12 group discussions identified challenges with the course, including areas where they had to compensate with on-the-job training for skills they felt should have been addressed in initial training, such as ship-driving proficiency in high-traffic environments. Our prior work on assessing training efforts in the federal government states that an agency should evaluate the effectiveness of its training and development efforts, to include obtaining and analyzing feedback. However, the Navy does not currently have a formal fleet-wide method of soliciting feedback from SWOs to obtain input on the quality of their classroom, simulator, and at-sea training on Division Officer performance and evaluate trends in feedback, and instead uses more limited means to assess training. For example: According to Navy officials, Surface Warfare Officers School Command conducts end-of-class surveys at the end of officer training, but no follow-up is conducted by the command after SWOs have assumed their ship duties or to obtain input from the trainees’ superior officers on the value of the training. The Navy had a survey for Division Officers and Department Heads in the past, but this survey gave little helpful feedback on training and, according to Navy officials, the Navy discontinued the survey after 2015. Surface Warfare Officers School Command assembles a board of officers from the fleet each year to review areas of its training curriculum, but Navy officials stated that participants are invited based on their expertise. As a result, only those selected to serve on the board (not officers across the fleet) have the opportunity to provide feedback. The Navy’s current means to assess training do not allow for the full range of junior and senior officers across the fleet to provide feedback on how well training prepares SWOs for their ship duties. Senior Navy officials acknowledged the value of conducting fleet-wide surveys of SWOs to obtain feedback on how to improve SWO training and gauge the health and morale of the SWO community. SWOs’ experiences in the fleet are diverse, therefore fleet-wide data is of particular value as centralized organizations like Naval Surface Forces, Surface Warfare Officers School Command, and the Office of the Chief of Naval Operations consider costly and consequential training investments. Without a method to regularly collect and analyze information from SWOs across the fleet, such as in a survey, regarding the quality of the increased classroom, simulator, and at-sea training on Division Officer performance, and evaluate trends in feedback received, Navy decision makers lack valuable information that could help them to assess the effects of training on SWO performance. Navy Surface Warfare Officer School Command training experts developed a ship-driving proficiency measurement system and used it in fiscal year 2018 to conduct ship-driving competency assessments. Specifically, from January through March 2018 Surface Warfare Officers School Command conducted “spot check” ship-driving competency assessments of 164 SWOs that had recently qualified as Officers of the Deck during their first at-sea assignment. Each assessment was conducted by three Navy inspectors that were independent of the assessed SWOs’ chain of command. The independent Navy inspectors found concerns in the ship-driving competency levels of more than 80 percent of these SWOs (see fig. 5). Specifically, Surface Warfare Officers School Command found that 29 SWOs (18 percent) had significant competency problems and 108 had some concerns (66 percent). According to Surface Warfare Officers School Command officials, those SWOs who experienced significant problems in their assessments likely should not have been qualified as Officer of the Deck at the time of the assessment because they violated fundamental ship-driving rules, among other issues. Navy guidance to the fleet emphasizes that these assessments performed by independent experts are valuable in supporting impartial results and providing quality information for analysis. According to Navy documentation, the Navy also used the 2018 competency assessments to help validate its new Junior Officer of the Deck and Officer of the Deck training curriculum. Specifically, Surface Warfare Officers School Command used the same Officer of the Deck competency assessment criteria to assess six officers in May 2018 and 12 in July 2018 that completed a pilot version of the Junior Officer of the Deck course. Surface Warfare Officer School Command found that the students with no at-sea experience that had completed a pilot of the new Junior Officer of the Deck training course in some cases outperformed qualified Officers of the Deck that had over a year of at-sea experience. According to Navy officials, the ability to compare ship-driving proficiency among populations and with earlier baselines using these competency assessments was valuable to the Navy in identifying the effects of changes to training, and could also be valuable in the future, as well. However, when we visited Surface Warfare Officers School Command in February 2019, officials told us they did not plan to conduct additional competency assessments until 2020. In meetings with Surface Warfare Officers School Command and senior Navy leaders, we noted that delaying additional assessments could limit visibility over ship driving proficiency trends and that small sample sizes could affect the Navy’s ability to make comparisons over time. In response, the Navy accelerated and expanded additional competency assessments. According to Navy officials, in spring 2019, Surface Warfare Officers School Command began to assess a sample of Division Officers using the Officer of the Deck competency assessment at the beginning of each Advanced Division Officer Course to collect and analyze performance data and refine training curriculum. Further, as of July 2019, the Navy had assessed 38 SWOs from three courses and found that the proficiency level of the SWOs assessed had not improved from the proficiency levels seen in the 2018 assessments. Senior Navy officials we met with as part of this review stated that they recognize the value in implementing periodic ship-driving competency assessments by independent inspectors to identify trends in ship-driving proficiency over time. However, we also found that the Navy has not planned to routinely conduct these assessments in the future. Specifically, in July 2019, Navy officials stated that they do not plan to complete these Officer of the Deck competency assessments beyond 2021 and plan to replace them with a different assessment at the end of the planned Officer of the Deck Phase II course. However, our analysis shows that mid-fiscal year 2024 is the first time Officer of the Deck Phase I course graduates will have completed their first at-sea assignment and be available to have their ship-driving training assessed, resulting in a multi-year gap in planned competency assessments. In order to measure the effectiveness of the full complement of Navy’s new and enhanced ship-driving training, the independent Navy inspectors will need to continue administering the Officer of the Deck competency assessments beyond 2021. In addition, an assessment performed at the end of training, such as the planned Officer of the Deck Phase II assessment, indicates the SWOs’ proficiency after additional training and may give a less accurate indication of prior at-sea proficiency. According to federal government internal control standards, management should use quality information and monitoring activities to ensure the entity’s objectives are achieved. Moreover, our prior work on assessing training efforts in the federal government states that an agency should evaluate the effectiveness of its training and development efforts, to include assessing competency and analyzing relevant data. Without routinely conducting Officer of the Deck competency assessments across the fleet using samples of sufficient size and selection methods, the Navy will be hindered in its ability to gauge fleet-wide ship driving proficiency trends and determine the effectiveness of the changes made to training, and the Navy may not know whether additional changes are needed. We found that the Navy has not provided standard criteria to ship Commanding Officers on fleet-wide ship driving proficiency expectations to inform the qualification of Officer of the Deck candidates. Instead, the Navy has determined that ship Commanding Officers should use their individual judgment in granting this qualification based on a set of required officer experiences, which the Navy refers to as Personnel Qualification Standards. Following the 2017 collisions, Surface Warfare Officers School Command developed proficiency standards to measure and test Officer of the Deck ship-driving proficiency to implement the Officer of the Deck competency assessments described above. The proficiency standards require an Officer of the Deck to demonstrate knowledge of navigation systems, rules of the road, and effective bridge resource management and to demonstrate the ability to successfully navigate high-traffic environments. However, the varying at-sea experiences of officers and subjective nature of some requirements have led to different experiences for SWO candidates working to qualify as Officer of the Deck. SWOs must complete a standard series of requirements in ship-driving and other experience before they are eligible to qualify as Officer of the Deck, with Commanding Officers granting qualification after their assessment of the SWO’s performance and fitness. However, when we held group discussions with SWOs on ships in the fleet, SWOs in nine of 12 group discussions with Division Officers, eight of 12 group discussions with Department Heads, and three of 12 interviews with Commanding Officers and Executive Officers identified significant differences in opportunities, experiences, and assessments that Division Officers experience in earning their qualification as Officers of the Deck during their first Division Officer assignment. For example: In one group discussion, Division Officers reported being qualified as Officers of the Deck without ever having stood watch at sea, with the Commanding Officer granting qualifications based on their classroom and simulator experience alone. In five of 12 group discussions with Division Officers, Division Officers stated that SWOs on ships in maintenance had few opportunities to stand watch on the bridge at sea to build proficiency in difficult ship- driving operations, but still received their qualifications. SWOs in 17 of 24 group discussions stated that some Division Officers get more ship driving experience than others before earning their Officer of the Deck qualifications. For example, Division Officers assigned to ships with more time at sea or fewer Division Officers get more experience to practice ship driving than those on ships with little time at sea or that must divide ship-driving opportunities among numerous Division Officers. Commanding Officers in three of 12 interviews reported that they had to temporarily place their Division Officers on other ships to gain qualifying experience, and had to rely on the judgment of the other ships’ Commanding Officers in determining their qualifications as Officers of the Deck. According to Navy officials, the Navy has not provided Officer of the Deck assessment criteria based on the developed proficiency standards to ship Commanding Officers, out of deference to their judgment in interpreting an officer’s preparedness to drive their ship. Navy officials emphasized the importance of allowing ship Commanding Officers to make their own determination of an officer’s preparedness to drive a ship, due to their knowledge of the ship’s operating conditions. Navy officials also stated that they considered the Officer of the Deck assessment standards to be a resource for use by Surface Warfare Officer Schools Command in assessing training curriculum and had not considered using the standards in the fleet for other purposes. However, the Navy’s 2018 and 2019 Officer of the Deck competency assessments identified significant variance in the ship-driving competency levels of recently qualified Officers of the Deck. Since the Navy has developed fleet-wide standards for assessing Officer of the Deck proficiency, the Navy could use these to provide standard Officer of the Deck assessment criteria in guidance to ship Commanding Officers. Federal government internal control standards state that management should internally communicate the necessary quality information to achieve the entity’s objectives and ensure decisions are made based on consistent standards. A SWO’s assigned ship, Commanding Officer, and operating conditions may change during a career, so a standard set of criteria would help Commanding Officers to determine what is expected of Officers of the Deck elsewhere in the fleet as they determine a junior officer’s qualification. Without providing standard Officer of the Deck assessment criteria and incorporating them into surface fleet guidance to Commanding Officers, the Navy risks creating uncertainty in Officer of the Deck qualification expectations— which can contribute to variations in ship-driving proficiency among SWOs that could jeopardize safe operations at sea. In September 2018, the Commander, Naval Surface Forces, U.S. Pacific Fleet and Commander, Naval Surface Forces Atlantic, began requiring SWOs to document their ship-driving and related experience in a handwritten logbook. The logbook—referred to as the Surface Warfare Mariner Skills Logbook (see fig. 6)—captures an officer’s experience gained during each watch aboard a ship, special evolution (e.g., underway replenishment, flight operations, and sea and anchor duty), and simulator training session. During our ship discussion groups, SWOs at the Division Officer and Department Head levels reported that they had begun filling out their logbooks and having them reviewed as required, but some acknowledged that they are inconsistently filling them out or that they were not entering any information in them. Specifically, SWOs in five of 24 discussion groups reported that logbooks are completed with inconsistent quality or not completed at all. Additionally, SWOs in four of 24 discussion groups reported that they are unaware of any plans to use the logbook information to identify any additional training needs and provide opportunities for SWOs to improve their ship-driving proficiency. Navy Personnel Command officials told us that, as of July 2019, they had received 174 summaries of Surface Warfare Mariner Skills Logbook data from Commanding Officers. Navy officials stated that over time, as they gather these data, they intend to examine the link between ship-driving proficiency and SWO experience. However, officials did not have any specific, measurable plans to analyze and use these data or to assess the completeness of these data. Federal internal control standards state that management should obtain relevant data from reliable sources and process those data into quality information to aid decision-making. Furthermore, Naval Surface Forces guidance states that the surface warfare community should analyze and use logbook data to link SWO experience with ship-driving proficiency. Despite this guidance, the Navy does not yet have a plan that includes specific steps to analyze and use logbook information to link SWO experience with ship-driving proficiency. According to senior Navy officials, while the Surface Warfare Mariner Skills Logbook is still relatively new, developing a plan to use the information would be a logical next step. Without a plan for analyzing and using Surface Warfare Mariner Skills Logbook data, the Navy cannot determine the relationship between SWO experience and ship-driving proficiency or use these data to aid decision-making. SWOs play a critical role in Navy surface fleet readiness, as they are responsible for safely driving ships at sea and successfully leading ships in Navy operations across the world. The Navy is making numerous changes and investments to enhance Surface Warfare Officer ship- driving training following the 2017 collisions at sea—with plans to triple initial training hours and spend nearly half a billion dollars to build simulator capacity to deliver this training. The Navy’s oversight of these efforts is centered on a series of added checks throughout SWOs’ careers to ensure that they have basic ship-driving and other skills. These checks are steps in the right direction but may not provide adequate assessment mechanisms in the near term and might lead to missed opportunities going forward. For example, the Navy is expanding its ship- driving training but is not planning to collect fleet-wide feedback on classroom, simulator, and at-sea training received. In addition, the Navy developed standards for conducting spot checks on ship-driving competency but is planning to stop those checks in 2021, missing an opportunity for an outside assessment and to evaluate how well new and updated training is working. Moreover, ship commanders are expected to qualify SWOs on ship driving but have not been provided standard guidance for how to do this, which can contribute to wide variations in SWO competence. Finally, the Navy has developed detailed logbooks for SWOs to track their experiences but the Navy has not developed a specific plan to analyze and use the logbook data. Without actions to address these challenges, the Navy cannot fully assess in the near term if the significant investments it is making to expand and enhance SWO ship-driving training are effective; further adjustments are necessary; and, ultimately, Navy ships are being operated safely at sea. We are making the following four recommendations to the Department of Navy: We recommend that the Secretary of the Navy ensure that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, develop a method to regularly collect feedback from SWOs across the fleet, such as in a survey, regarding the quality of their classroom, simulator, and at-sea training on Division Officer performance; and evaluates trends in the feedback received for the purpose of improving SWO training. (Recommendation 1) We recommend that the Secretary of the Navy ensure that the Commander, Naval Surface Forces, routinely conduct regular Officer of the Deck competency assessments using samples of sufficient size and using selection methods to gauge the level of fleet-wide ship-driving proficiency trends following the implementation of the planned ship- driving training programs. (Recommendation 2) We recommend that the Secretary of the Navy ensure that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, provide Commanding Officers with standard criteria to inform their evaluation of candidates for their Officer of the Deck qualification and incorporates these criteria into surface fleet guidance. (Recommendation 3) We recommend that the Secretary of the Navy ensure that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, develop a plan to analyze and use Mariner Skills Logbook information to inform decision-making. (Recommendation 4) We provided a draft of this report to DOD for review and comment. In written comments provided by the Navy through DOD (reprinted in their entirety in appendix V), the Navy concurred with all four of our recommendations and identified actions it plans to take to evaluate the effectiveness of changes to SWO training. The Navy also provided additional information and context in its comments and provided technical comments, which we incorporated as appropriate. The Navy concurred with our first recommendation that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, develop a method to regularly collect feedback from SWOs across the fleet, such as in a survey, regarding the quality of their classroom, simulator, and at-sea training on Division Officer performance; and evaluate trends in the feedback received for the purpose of improving SWO training. The Navy stated that it plans to explore additional means of garnering holistic SWO feedback regarding newly-implemented SWO training and assessments as well as gathering additional targeted feedback. However, the Navy stated that the use of performance data will remain the primary focus of surface force training improvement efforts. While using performance data is valuable, it will be important that the Navy follow through to develop a holistic means of collecting feedback, such as in a survey of SWOs across the fleet, on the effectiveness of Division Officer training on SWO performance to ensure a variety of perspectives are considered. In its comments, the Navy noted that the SWOs who participated in our ship visits and discussion groups had not experienced the changes made or planned to SWO training. We agree that our ship visits did not include officers who had experienced the expanded Division Officer training courses, as they were first introduced to the fleet in June 2019, after we had completed the majority of our work. While our discussion groups pre- date the implementation of new SWO training courses, the discussion groups we conducted with over 200 SWOs reinforced the our finding that the Navy needs to develop a method to regularly collect feedback from SWOs across the fleet. Also, the Navy plans to more than triple initial training, so routinely soliciting and analyzing feedback from SWOs on Division Officer training will be needed to determine the effectiveness of the Navy’s investments in these training programs and inform the Navy’s decisions on whether further adjustments are necessary. The Navy acknowledged that only officers participating in the Surface Warfare Officers School Command’s Board of Visitors provide direct feedback on the training curriculum. The Navy noted, however, that all available Surface Warfare units are invited to participate in Surface Warfare Officer School Board of Visitors events and so could provide feedback then. In addition, the Navy noted that Surface Warfare Officer School Command also solicits feedback through visits to fleet concentration areas and through semiannual symposiums of ship Commanding Officers. While such targeted means of collecting feedback may provide valuable information, officers may not be able to participate due to their deployment status, position on shore duty, timing of events during other personal responsibilities, or other factors. We believe that developing a method to regularly collect feedback from SWOs across the fleet would provide decision makers with valuable information that could help them assess the effects of training on SWO performance. The Navy concurred with our second recommendation that the Commander, Naval Surface Forces, routinely conduct regular Officer of the Deck competency assessments using samples of sufficient size and using selection methods to gauge the level of fleet-wide ship-driving proficiency trends following the implementation of the planned ship- driving training programs. The Navy stated that it plans to routinely collect and analyze standardized mariner skills performance data across an officer’s career path. However, the Navy stated it will use training checks, rather than the current Officer of the Deck competency assessment, to evaluate SWO performance after 2020. This presents two problems in meeting the intent of our recommendation. First, the Navy will need to ensure that the training checks are sufficiently rigorous to assess competency. Second, the Navy will not have valid data to compare the effects of training changes on competency if it changes its assessment approach. In 2018, the Navy used the Officer of the Deck competency assessment to establish a baseline of SWO ship-driving proficiency. We found that the 2018 competency assessments showed significant variation in ship- driving proficiency and the 2019 follow up assessments found that competency had not improved. In currently documented plans, the Officer of the Deck Phase II check after a Division Officer’s first assignment will occur at the end of the course. Even if the Navy changes the Officer of the Deck Phase II check to occur at the beginning of training as stated in its comments, performance data from this check cannot be directly compared with the results of the current competency assessment. Differences in assessment content or difficulty, remediation attempts, and the fact that the new check may have career implications for SWOs as a go/no-go assessment may affect proficiency measurements and pass rates. Due to these factors, we believe that a comparison between the current Officer of the Deck competency assessment and the planned Officer of the Deck Phase II check or another standard should not be considered as valid means for demonstrating changes in ship-driving proficiency over time. That is, adopting a new standard may affect the Navy’s ability to determine the impact of training on ship-driving proficiency compared with the 2018 baseline results. The Navy also stated in its comments that the SWO training and assessment continuum is designed to provide training and evaluation at all career milestone levels. The planned system of additional skills checks will provide the Navy with more insight into SWO proficiency levels over the course of an officer’s career and help the Navy to understand the effects of changes to training. As we stated in the report, we believe these checks are significant steps in the right direction but may not provide adequate assessment mechanisms in the near term. The more robust Officer of the Deck competency assessments are necessary to gauge the level of fleet-wide ship-driving proficiency trends following the implementation of the planned ship-driving training programs. Further, the Navy stated in its comments that while numerous means of assessing SWO mariner skills proficiency at various milestone levels are in place, the ultimate SWO career path goal is to develop the most proficient, experienced, and confident Commanding Officers, which occurs approximately 16 years into the SWO career path. While the quality of ship Commanding Officers is a vital component of Navy readiness and capability, the majority of SWOs do not remain in the Navy long enough to advance beyond the position of Division Officer, according to Navy documentation. Similarly few advance to the position of Commanding Officer during their career as a SWO. Since Division Officers constitute over one third of the SWO workforce and by design of the SWO career path do most of the ship-driving, it is of utmost importance to build and evaluate fundamental ship-driving skills for all Division Officers to support excellence in the ship-driving proficiency across the Navy. Finally, the Navy stated in its comments that our report language implies an absence of any Officer of the Deck assessments from 2021 through 2024. We acknowledge that the Navy will conduct assessments of Officers of the Deck during this time period in line with its planned system of ten checks over a SWO’s career. However, without maintaining the current Officer of the Deck competency assessments through at least 2024, the Navy will be unable to demonstrate any proficiency improvement, compared with the 2018 baseline, resulting from its new training programs. Further, the Navy stated in its comments that it is important to clarify that the SWOs who received competency checks in 2019 had not benefitted from the new and expanded ship-driving training courses. Our ship visits did not include officers who had experienced the expanded Division Officer training courses, as they were first introduced to the fleet in June 2019. Nonetheless, it is concerning that SWO competency had not improved in the 2 years since the 2017 collisions despite the fleet-wide attention to improving ship-driving skills. The Navy concurred with our third recommendation that the Commander, Naval Surface Forces, in coordination with Surface Warfare Officers School Command, provide Commanding Officers with standard criteria to inform their evaluation of candidates for their Officer of the Deck qualification and incorporate these criteria into surface fleet guidance. However, the Navy stated that such criteria are already in place. Specifically, the Navy noted that existing Personnel Qualification Standards provide the standard evaluation criteria for the Officer of the Deck qualification. We agree that the Personnel Qualification Standards are in place, but disagree that Qualification Standards provide standard evaluation criteria. Unless the Navy provides additional guidance for Commanding Officers to measure proficiency in addition to the list of required experiences present in the Personnel Qualification Standards, the actions the Navy identified as addressing our recommendation will not meet the intent of our recommendation. The Navy’s Officer of the Deck Personnel Qualification Standards provide a list of required experiences; however, the 2018 and 2019 competency assessments indicate that these existing criteria have not resulted in high levels of proficiency among Officers of the Deck. The Navy’s Personnel Qualification Standards do not require SWOs to demonstrate a standard level of proficiency, but rather that SWOs participate in a required number of ship-driving experiences at a level determined by his or her Commanding Officer. The absence of a common proficiency standard across the Navy may contribute to inconsistency in ship-driving skills among SWOs. Since the Officer of the Deck competency assessment provides a means to measure proficiency, communicating appropriate standards in line with those used in the current assessments as qualification criteria would help ensure a common understanding of proficiency expectations. In comments, the Navy stated that for junior officers whose ships experience maintenance periods, it is an historic surface force-wide practice for Commanding Officers to temporarily assign those officers to similar ships whose operational schedule better support qualification. This practice is understandable and may contribute to SWO career development, but can lead to significant differences in opportunities, experiences, and assessments that SWOs receive during their first Division Officer assignment. For example, as noted in our report, some Commanding Officers stated because of this temporary assignment, they had to rely on the judgment of the other ships’ Commanding Officers to determine their SWOs’ qualifications as Officers of the Deck. The Navy concurred with our fourth recommendation that Commander, Naval Surface Forces, in coordination with Surface Warfare Officer School Command, develop a plan to analyze and use Mariner Skills Logbook information to inform decision-making. The Navy noted that it would comprehensively evaluate performance data relative to Mariner Skills Logbook data in order to refine mariner skill milestone performance and proficiency criteria. If Navy efforts result in a plan that includes specific and measurable steps for analyzing and using Mariner Skills Logbook data, the efforts will meet the intent of our recommendation. In its comments the Navy stated that during the time we conducted our group discussions (i.e. January through April 2019), Mariner Skills Logbooks were still being introduced to the Fleet and recording practices were still being established. While at the time of our discussion groups the Mariner Skills Logbooks were relatively new, in September 2018, the Navy issued an instruction that established guidance for the implementation and use of the logbooks. In addition, all of the SWOs we met with as part of our review had already received their Mariner Skills Logbooks. We are sending copies of this report to congressional committees, the Secretary of Defense, the Secretary of the Navy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3489 or pendletonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Following the four 2017 mishaps at sea, the Navy completed two internal reviews on surface fleet readiness, ultimately compiling 111 recommendations for improvement. The Navy established a Readiness Reform and Oversight Council under the leadership of the Vice Chief of Naval Operations to oversee implementation of these recommendations. The Readiness Reform and Oversight Council reported in February 2019 that it considered 91 of these recommendations to be implemented. We reviewed the recommendations, identified 12 recommendations related to Surface Warfare Officer (SWO) initial ship-driving training, and requested the implementation status of each of these recommendations from the Commander, Naval Surface Forces. The Navy considers a recommendation to be “implemented” when there is a policy in place or action has been taken to address a recommendation. The Navy considers a recommendation to be “transitioned” when the Readiness Reform and Oversight Council no longer maintains regular oversight of a recommendation and has transitioned oversight to another Navy organization. As of August 2019, the Navy considered all 12 of the recommendations related to ship-driving training as implemented with the final recommendation estimated to transition by September 30, 2019. Table 1 lists the 12 recommendations related to SWO initial ship-driving training, and our summary of the Navy’s explanation for why they are considered to be implemented. The John S. McCain National Defense Authorization Act for Fiscal Year 2019 and Senate Armed Services Committee report 115-262 to accompany a bill for the National Defense Authorization Act for Fiscal Year 2019 contained provisions that we review Surface Warfare Officer (SWO) training and career paths. This report (1) describes the changes the Navy has made or planned to SWO ship-driving training since the 2017 collisions and (2) assesses the extent to which the Navy has taken actions to evaluate the effectiveness of those changes. We plan to issue a separate report on SWO career paths in the future. For objective one, we reviewed Navy documentation from Commander, Naval Surface Forces, Surface Warfare Officers School Command, and the Office of the Chief of Naval Operations on the content, purpose, cost, and status of changes made and further changes planned to ship-driving training since the 2017 collisions. We focused our analysis on changes made to SWOs’ ship-driving training at the junior officer level as the Navy prioritizes ship-driving training and ship-driving experience for junior officers, and the Navy has identified actions it is taking to address recommendations from the Navy’s two 2017 internal reviews to ensure safe operations at sea through improvements to junior officer training. We analyzed planned investments from fiscal year 2018 through fiscal year 2025 for the construction of two ship-driving training facilities and the development of three ship-driving training courses, which includes the cost of purchasing new simulators, hiring new instructors, military construction, and course curriculum development. We discussed implementation plans for the 2017 internal reviews’ recommendations with the Commander, Naval Surface Forces; officials from the Surface Warfare Officers School Command; and officials from the Readiness and Reform Oversight Council, a group within the Office of the Chief of Naval Operations established to monitor the implementation of the internal reviews’ recommendations. For objective two, we reviewed Navy documentation and interviewed Navy officials on how they evaluate SWOs throughout their careers, gather and use feedback from SWOs, assess the effectiveness of SWO ship-driving training, and use available data to inform decisions regarding SWO training. Specifically, we reviewed the implementation of the 10 career milestone checks outlined in Naval Surface Forces Instruction 1412.5 Surface Warfare Officer Milestone Mariner Skills Assessments, Evaluations, and Competency Checks that are to be administered during a SWO’s career; Navy’s efforts to collect feedback from the surface fleet on the quality of SWO ship-driving training and the health of the SWO community; Navy’s 2018 Officer of the Deck competency assessment results, criteria, and plans to continue evaluating SWO ship-driving competency; extent to which the Navy had provided standardized criteria for ships’ Commanding Officers to use when evaluating SWO’s for ship-driving qualification; and format of SWO Mariner Skills Logbooks used to track SWO ship- driving experiences, and Navy policies regarding the logbooks. To do this we compared the Navy’s practices with relevant Navy reviews, instructions, and guidance, Standards for Internal Control in the Federal Government, and our prior work on assessing training efforts in the federal government. We assessed the reliability of the results of the Navy’s 2018 Officer of the Deck competency assessments by examining them for missing values, comparing other sources that provide the same types of data to ensure consistency, and interviewing knowledgeable agency officials regarding the assessments’ accuracy and completeness. In addition, we reviewed the Navy’s internal controls for performing the assessments, such as grading criteria and use of independent inspectors to ensure quality and consistency in the information. We determined that the results of the Navy’s 2018 Officer of the Deck competency assessments were sufficiently reliable for the purposes of reporting on the number and percentage of the graded categories. In addition to meeting with Navy offices, we visited 12 surface ships in the Pacific and Atlantic fleets from January through April 2019, selected according to which ships and crews were available at each of the sites we visited. Aboard the ships we held group discussions and interviews with approximately 225 SWOs to discuss their views on the sufficiency and appropriateness of SWO training. Discussion group sizes ranged from two to 20 SWOs. In conducting these group discussions, we held 24 group discussions, with two separate discussions for each of the 12 ships—one with Department Heads and one with Division Officers; interviewed Commanding and Executive Officers aboard each of the 12 ships, where available; and conducted each group discussion without the group’s supervisors or subordinates present. The ship crews we visited were those the Navy identified as available to hold group discussions with us during site visits, and the results of these group discussions are not generalizable to anyone outside these groups. Due to the timing of our work, the interviews and group discussions did not include SWOs that experienced changes made or planned for SWO training beyond April 2019. We asked each group a standard set of questions to obtain their views on the following topics: the sufficiency and appropriateness of SWO training programs in preparing SWOs for their ship responsibilities, including ship driving; the SWO career path, including the potential benefits and drawbacks of more specialized career paths; and any opportunities to improve the SWO community. We conducted an analysis of the discussion group responses to identify common themes and provide illustrative examples in our report. Specifically, we reviewed the responses received during discussion groups, grouped the responses by themes, and counted how many discussion groups and interviews provided similar feedback to our questions. One GAO analyst conducted this analysis, coding the information and entering it into a record of summary, and a different GAO analyst checked the information for accuracy and agreement on themes. Any initial disagreements in the coding were discussed and reconciled by the analysts. The analysts then tallied the responses to determine the extent to which the certain themes were covered during our discussion groups and interviews. We interviewed officials, or where appropriate, obtained documentation at the organizations listed below: Office of the Chief of Naval Operations Director of Surface Warfare (N96) Surface Warfare (N96) Manpower and Training Readiness Reform and Oversight Council Commander, Naval Surface Forces, U.S. Pacific Fleet Littoral Combat Ship Training Facility Navigation, Seamanship, and Ship-handling Training facility USS Ardent (MCM 12) USS Lake Champlain (CG 57) USS New Orleans (LPD 18) USS Paul Hamilton (DDG 60) USS Tulsa (LCS 16) Commander, Naval Surface Forces, Atlantic Navigation, Seamanship, and Ship-handling Training facility USS Bataan (LHD 5) USS Cole (DDG 67) USS Mahan (DDG 72) USS Mesa Verde (LPD 19) USS Oak Hill (LSD 51) USS San Antonio (LPD 17) USS San Jacinto (CG 56) Surface Warfare Officer School Command Basic Division Officer Course facilities—San Diego, California and Surface Warfare Officer (PERS-41) We conducted this performance audit from November 2018 to November 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Commander, Naval Surface Forces, issued an instruction in September 2018 listing ten ship-driving skill checks to be conducted periodically over the course of a Surface Warfare Officer’s (SWO) career. Failure to pass some of the checks can result in required remediation or disqualification from career advancement. Table 2 lists the ten current and planned checks as of August 2019, as well as information on their timing and content as described in the instruction. In addition to the contact named above, Chris Watson (Assistant Director), Tobin McMurdie (Analyst-in-Charge), David Beardwood, Vincent Buquicchio, Mae Jones, Amie Lesser, Shahrzad Nikoo, Michael Silver, and Brandon Voss made key contributions to this report.", "summary": "In 2017, the Navy had four mishaps at sea including two collisions that resulted in the loss of 17 sailors' lives and hundreds of millions of dollars in damage to Navy ships. In the wake of those mishaps, the Navy identified deficiencies in SWO ship-driving training and related experience as contributing factors and has undertaken a number of efforts to improve these areas. Senate Report 115-262, accompanying a bill for the Fiscal Year 2019 National Defense Authorization Act, contained a provision that GAO assess SWO training. This report (1) describes the changes the Navy has made to SWO ship-driving training since the 2017 collisions and (2) assesses the extent to which the Navy has taken actions to evaluate the effectiveness of changes made to SWO ship-driving training. GAO reviewed and analyzed changes made to Navy training and assessment practices and related investments; interviewed cognizant officials; and conducted discussions with SWOs aboard 12 ships. Since 2017, the Navy has made numerous changes and plans additional changes to enhance Surface Warfare Officer (SWO) ship-driving training. The Navy plans for these changes to result in a threefold increase in the number of initial ship-driving training hours for SWOs by 2021, compared with the number of training hours prior to the 2017 collisions (see fig.). The Navy added classroom and simulator time to existing training courses to improve ship-driving skills and is developing two additional simulator-based ship-driving courses planned for 2021. These plans hinge on the completion of two new simulator-based training facilities, scheduled for completion in June 2021 and in January 2023. The Navy has relied on added skill checks conducted throughout a SWO's career to ensure that each SWO has basic ship-driving skills, but has not put key processes and assessments in place to evaluate comprehensively the effectiveness of its changes to ship-driving training. Senior Navy officials stated that it could take 16 years or more to know if the planned changes to SWO training were effective in increasing Commanding Officer ship-driving proficiency across the fleet and stated that they intend to closely monitor the implementation of changes to the training. However, GAO found that in planning an approach for evaluating the changes, the Navy has not: (1) identified a method to solicit fleet-wide feedback on the quality of the increased ship-driving training received by SWOs; (2) planned to routinely conduct ship-driving competency “spot checks” that were instituted after the 2017 collisions despite Navy inspectors having found concerns with more than 80 percent of SWOs' ship-driving skills; (3) provided standard criteria to ship Commanding Officers for qualifying SWOs to drive ships, contributing to significant variance in ship-driving experience and competency levels across the fleet; nor (4) developed a specific plan to analyze and use information from logbooks in which SWOs are to document ship-driving and related experience. Without addressing these challenges, the Navy cannot assess in the near term if the significant investments made to expand and enhance SWO ship-driving training are effective; further adjustments are necessary; and Navy ships are being operated safely at sea. GAO is making four recommendations to the Navy to routinely evaluate SWO training, including that the Navy collect and evaluate fleet-wide feedback on the quality of training; routinely conduct ship-driving competency assessments; provide standard criteria for qualifying ship drivers; and develop a plan to analyze and use logbook information. The Navy concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "The FSM and RMI are independent countries about 3,000 miles southwest of Hawaii. The FSM is a federation of four semiautonomous states—Chuuk, Kosrae, Pohnpei, and Yap—whose population and income vary widely. Chuuk, the largest state by population, has the lowest per-capita gross domestic product (GDP). Overall, the FSM had a 2016 population of approximately 102,000 and a GDP per capita of about $3,200. The RMI’s 2016 population was approximately 54,000 with a GDP per capita of about $3,600. The RMI’s most recent census, in 2011, found that approximately three-quarters of the population lived in Majuro, the nation’s capital, and on the island of Ebeye in the Kwajalein Atoll. Table 1 shows the FSM’s, FSM states’, and RMI’s estimated populations and annual GDP per capita in fiscal year 2016. U.S. relations with the FSM and the RMI began during World War II, when the United States ended Japanese occupation of the region. Starting in 1947, the United States administered the region under a United Nations trusteeship. In 1986, after a period of negotiations, the United States entered into a compact of free association with the FSM and RMI that provided for economic assistance to the two countries, secured U.S. defense rights, and allowed FSM and RMI citizens to migrate to the United States. In 2003, after a period of negotiations, the United States approved separate amended compacts with the FSM and the RMI that went into effect on June 25, 2004, and May 1, 2004, respectively. The amended compacts’ implementing legislation authorized and appropriated direct financial assistance to the FSM and the RMI in fiscal years 2004 through 2023, with the base amounts decreasing in most years. The legislation also provided for partial inflation adjustment of the base amount of compact sector grants and trust fund contributions each year. As the base amount of compact sector grants decreases, the trust fund contributions generally increase by an equivalent amount. Because the annual inflation adjustment is less than full inflation, the value of compact sector grants declines in real terms. Figure 1 shows the amount of compact sector grants and trust fund contributions each fiscal year from 2004 through 2023. The amended compacts and associated fiscal procedures agreements require that compact sector grants support the countries in six core sectors—education, health, infrastructure, environment, private sector development, and public sector capacity building—with priority given to the education and health sectors. These grants are described in section 211(a) of each compact and are referred to as compact sector grants or 211(a) grants. Section 211(b) of the RMI compact further states that the RMI must target a specified amount of grants to Ebeye and other Marshallese communities within Kwajalein Atoll. The RMI military use and operating rights agreement (MUORA) states that the Kwajalein- related funds provided to the RMI in the compacts shall be provided through fiscal year 2023 and thereafter for as long as this agreement remains in effect. The amended compacts and their subsidiary trust fund agreements provided that each trust fund is to be managed by a compact trust fund committee. Each compact trust fund committee includes representatives from both the United States and the respective country, but the terms of the trust fund agreements require the United States to hold the majority of votes. The Director of Interior’s Office of Insular Affairs serves as the chair of each committee. Trust fund committee responsibilities include overseeing fund operation, supervision, and management; investing and distributing the fund’s resources; and concluding agreements with any other contributors and other organizations. As part of this oversight, the committees are to establish an investment and distribution policy. The committees are also to determine fiscal procedures to be used in implementing the trust fund agreements on the basis of the fiscal procedures used for compact grant administration, unless otherwise agreed by the parties to the agreement. The U.S.–FSM and U.S.–RMI trust fund agreements allow for the agreements to be amended in writing at any time, with mutual consent of the governments. However, the U.S. legislation implementing the amended compacts requires that any amendment, change, or termination of all, or any part, of the compact trust fund agreements shall not enter into force until incorporated into an act of Congress. The compact trust fund agreements state that no funds, other than specified trust fund administrative expenses, may be distributed from the funds before October 1, 2023. From fiscal year 2024 onward, the maximum allowed disbursement from each compact trust fund is the amount of the fiscal year 2023 annual grant assistance, as defined by the trust fund agreement, with full inflation adjustment. In addition, the trust fund committees may approve additional amounts for special needs. The RMI compact trust fund agreement excludes Kwajalein-related assistance, defined in section 211(b) of the RMI compact, from the calculation of the allowed disbursement. Although the compact trust fund agreements state the maximum allowable disbursement level, they do not establish or guarantee a minimum disbursement level. Each country’s compact trust fund consists of three interrelated accounts: the “A” account, the “B” account, and the “C” account. The A account is the trust fund’s corpus and contains the initial, and any additional, U.S. and FSM or RMI contributions; contributions from other countries; and investment earnings. No funds, other than specified trust fund administrative expenses, may be disbursed from the A account. The B account is the trust fund’s disbursement account and becomes active in fiscal year 2023. All income earned in 2023 will be deposited in the B account for possible disbursement in 2024. Each subsequent year’s investment income will similarly be deposited in the B account for possible disbursement the following year. If there is no investment income, no funds will be deposited in the B account for possible disbursement the following year. The C account is the trust fund’s buffer account. Through 2022, any annual income exceeding 6 percent of the fund balance is deposited in the C account. The size of the C account is capped at three times the amount of the estimated annual grant assistance in 2023, including estimated inflation. From 2023 onward, if annual income from the A account is less than the previous year’s disbursement, adjusted for inflation, the C account may be tapped to address the shortfall. After 2023, any funds in the B account in excess of the amount approved for disbursement the following fiscal year are to be used to replenish the C account as needed, up to the maximum size of the account. If there are no funds in the C account and no prior-year investment income in the B account, no funds will be available for disbursement to the countries the following year. Figure 2 shows the compact trust fund account structure and associated rules. According to the U.S. trust fund agreements with the FSM and the RMI, contributions from other donors are permitted. In May 2005, Taiwan and the RMI reached an agreement that Taiwan would contribute a total of $40 million to the RMI’s compact trust fund A account between 2004 and 2023. A “D” account may also be established to hold any contributions by the FSM and the RMI governments of revenue or income from unanticipated sources. According to the trust fund agreements, the D account must be a separate account, not mixed with the rest of the trust fund. Only the RMI has a D account, governed in part by the agreement between Taiwan and the RMI. The amended compacts’ implementing legislation incorporates, by reference, related agreements extending programs and services to the FSM and RMI. The programs and services agreement with each country identifies the following programs and services as being available to each country: U.S. postal services, weather services, civil aviation, disaster preparedness and response, and telecommunications. Each programs and services agreement extends for 20 years from the compact’s entry into force. The agreement with the FSM ends on June 24, 2024, and the agreement with the RMI ends on April 30, 2024. The amended compacts’ implementing legislation (Pub. L. No. 108-188) and other U.S. legislation authorize other U.S. grants, programs, and services for the FSM and RMI. Pub. L. No. 108-188 authorizes an annual supplemental education grant (SEG) for the FSM and RMI in fiscal years 2005 through 2023, to be awarded in place of grants formerly awarded to the countries under several U.S. education, health, and labor programs. The FSM and RMI are not eligible for the programs replaced by the SEG during these years. Unlike the compact sector grants, the amended compacts’ implementing legislation authorized the SEG but did not appropriate funds for it. Funding for the SEG is appropriated annually to the U.S. Department of Education (Education) and is transferred to Interior for disbursement. Other provisions of the amended compacts’ implementing legislation, as well as other U.S. law, make the FSM and RMI eligible for a number of additional programs. As of fiscal year 2016, compact sector grants and the SEG, each of which end in 2023, supported a substantial portion of government expenditures in the FSM and RMI. Compact sector grants and the SEG supported about one-third of all FSM government expenditures. The four FSM states relied on these grants to a greater extent than did the FSM national government. In the RMI, compact sector grants and the SEG supported about one-quarter of all government expenditures. The expiration of the compacts’ programs and services agreements in 2024 would also require the FSM and RMI to bear additional costs to provide services currently provided by the United States under the agreements. Compact sector grants, the SEG, and other U.S. grants supported almost half of FSM national and state government expenditures in fiscal year 2016. Compact sector and supplemental education grants that end in 2023 supported approximately one-third of total FSM national and state government expenditures in fiscal year 2016, while other U.S. grants supported an additional 15 percent of total FSM government expenditures (see fig. 3). In fiscal year 2016, compact sector and supplemental education grants that end in 2023 supported a larger proportion of FSM state governments’ expenditures than of the FSM national government’s expenditures. Compact sector grants and the SEG supported 8 percent of national government expenditures but supported 50 percent or more of each state’s government expenditures. Among the FSM states, Chuuk, which has both the largest population and the lowest per-capita income in the FSM, had the highest percentage of expenditures supported by U.S. grants. (See table 2 for a summary of FSM national and state government expenditures supported by compact sector grants and the SEG and by other U.S. grants.) Compact sector and supplemental education grants that end in 2023 supported approximately 25 percent of the RMI’s $123.5 million in government expenditures in fiscal year 2016, while other U.S. grants supported an additional 8 percent. Kwajalein-related compact grants that do not end in 2023 supported an additional 3 percent (see fig. 4). FSM and RMI budgets would be further affected if the countries assumed responsibility for providing programs and services currently provided by the United States. The following describes the status after 2023 of U.S. grants, programs, and services in the FSM and RMI under current law: Compact sector grants are scheduled to end in 2023, but the RMI MUORA extends the time frame of Kwajalein-related compact grants for as long as the MUORA is in effect. The SEG and additional grants identified in the amended compacts’ implementing legislation are scheduled to end in 2023. Also, after fiscal year 2023, the FSM and RMI will no longer be eligible for some programs that the SEG replaced, including Head Start (early childhood education, health, and nutrition services for low-income children and their families). The compact-related programs and services agreements with each country will end in 2024. However, some U.S. agencies, such as the National Weather Service, Federal Aviation Administration, and U.S. Agency for International Development, may continue to provide programs and services similar to those provided in the agreement under other authorities. The FSM and RMI will generally remain eligible for other programs identified in the amended compacts’ implementing legislation. These programs include U.S. Department of Agriculture (USDA) Rural Utilities Service grant and loan programs and U.S. Department of Education Pell grants for higher education and grants under Part B of the Individuals with Disabilities Education Act for children with disabilities. The FSM and RMI will remain eligible for additional programs we identified that have been provided under other current U.S. laws. Examples of these programs include USDA housing assistance programs and multiple public health, medical, and disease control and prevention grants provided by the U.S. Department of Health and Human Services. See appendix I for more information about the status after 2023 of U.S. grants, programs, and services in the FSM and RMI under current law. Our May 2018 projections for the compact trust funds showed that after fiscal year 2023, the funds are unlikely to provide maximum annual disbursements and may provide no disbursements at all in some years. The risk of disbursements below the maximum and the risk of zero disbursements increase over time for both funds. Potential strategies we analyzed in our May 2018 report would reduce or eliminate the risk of the compact trust funds’ experiencing years of zero disbursement. However, all of the potential strategies would require the countries to exchange a near-term reduction in resources for more-predictable and more- sustainable disbursements in the longer term. Our May 2018 projections for the FSM and RMI compact trust funds after 2023 indicated that, given their balance at the end of fiscal year 2017 and current compact trust fund rules—the baseline scenario—the funds will be unable to provide maximum disbursements (equal to the inflation- adjusted amount of annual grant assistance in 2023) in some years and unable to provide any disbursement at all in some years, with the likelihood of zero disbursement in a given year increasing over time. The compact trust funds’ C account—designed as a buffer to protect disbursements from the B account in years when the funds do not earn enough to fund the disbursement—could be exhausted by a series of years with low or negative annual returns. Since current rules do not allow disbursements from the compact trust fund corpus (the A account), exhaustion of the C account would result in zero disbursement in years when fund returns are zero or negative. Thus, there may be no funds available to disburse even if the funds’ A accounts have a balance. As a result of low or zero disbursements, the countries could face economic and fiscal shocks and significant challenges in planning programs and budgets. Since we published our May 2018 report, an additional year of compact trust fund performance data and updated estimates of future inflation have become available; however, the updated information does not alter the conclusions we presented in May 2018. The updated data and inflation estimates change our model’s assumptions about the current compact trust fund balance, size of future U.S. contributions to the FSM and RMI compact trust funds, annual grant assistance in fiscal year 2023, and C account balance—each of which are relevant variables for our analysis. However, the updated variables would result in only slight changes to our 2018 report’s projections of future compact trust fund performance presented in this testimony and do not alter our broader conclusions about future risks to the compact trust funds. FSM compact trust fund projections. In May 2018, our model projected that, given the baseline scenario and a 6 percent net return, the FSM compact trust fund will experience declining disbursements relative to the maximum allowable disbursements and an increasing chance of zero disbursements. (See app. I of GAO-18-415 for a full description of our methodology, and see app. V of GAO-18-415 for the baseline results with alternative net returns.) Projected disbursements. We projected that the FSM compact trust fund will, on average, be able to provide disbursements equal to 82 percent of the maximum allowable disbursement—the inflation- adjusted amount of 2023 annual grant assistance—in its first decade of disbursements. The likely average disbursement falls to 49 percent of the maximum in the next decade and falls further in subsequent decades. In addition, the amount available for disbursement may fluctuate substantially from year to year. Depending on the compact trust fund’s performance in the previous year, disbursements may be higher or lower than the average amount if the balance in the C account is not sufficient to provide additional disbursements. Likelihood of providing zero disbursement. We projected a 41 percent likelihood that the FSM compact trust fund will be unable to disburse any funds in 1 or more years during the first decade of trust fund disbursements. This likelihood increases over time, rising to 92 percent in fiscal years 2054 through 2063. Figure 5 shows our May 2018 projections of the FSM compact trust fund’s average disbursements as a percentage of maximum disbursement and the likelihood of 1 or more years of zero disbursement, given the baseline scenario and a 6 percent net return. We calculated the average disbursement as a percentage of the maximum allowable disbursement by averaging, over each 10-year period and over 10,000 simulated cases, the ratio of simulated disbursement to the maximum inflation-adjusted allowable disbursement in the given period. We calculated the likelihood of zero disbursement by counting cases with 1 or more years of zero disbursement in each of the given periods over 10,000 simulated cases. RMI compact trust fund projections. In May 2018, our model projected that, given the baseline scenario and a 6 percent net return, the RMI compact trust fund will experience declining disbursements relative to the maximum allowable disbursements and an increasing chance of zero disbursements. Projected disbursements. We projected that in its first decade of disbursements, the RMI compact trust fund will, on average, be able to provide disbursements nearly equal to the inflation-adjusted amount of 2023 annual grant assistance as defined by the trust fund agreement—the maximum allowable. However, in each subsequent decade, the projected disbursements as a percentage of the maximum disbursements decline by about 10 percentage points. In addition, from year to year, the amount available to disburse may fluctuate substantially. Depending on the compact trust fund’s performance in the previous year, disbursements may be higher or lower than the average amount if the balance in the C account is not sufficient to provide additional disbursements. Likelihood of providing zero disbursement. We projected a 15 percent likelihood that the RMI compact trust fund will be unable to disburse any funds in 1 or more years during the first decade of trust fund disbursements. This likelihood increases over time, rising to 56 percent in fiscal years 2054 through 2063. Figure 6 shows our May 2018 projections of the RMI compact trust fund’s average disbursements as a percentage of maximum disbursement and its likelihood of 1 or more years of zero disbursement, given the baseline scenario and a 6 percent net return. We calculated the average disbursement as a percentage of the maximum allowable disbursement by averaging, over each 10-year period and over 10,000 simulated cases, the ratio of simulated disbursement to the maximum inflation-adjusted allowable disbursement in the given period. We calculated the likelihood of zero disbursement by counting cases with 1 or more years of zero disbursement in each of the given periods over 10,000 simulated cases. For our May 2018 report, we conducted a series of simulations to determine the likely effects of potential strategies for improving the outlook of the FSM and RMI compact trust funds. For example, we developed and analyzed potential strategies in which annual disbursements are reduced below the maximum allowable additional annual contributions are made to the trust fund prior to the end of fiscal year 2023; and the trust fund agreement disbursement policies are modified to limit the annual disbursement to a fixed percentage of the fund’s moving average balance over the previous 3 years, up to the maximum disbursement amount defined by the current trust fund agreement. All of the potential strategies we analyzed would reduce or eliminate the risk of the compact trust funds experiencing years of zero disbursement. However, some of the potential strategies might require changing the trust fund agreements, and all of the potential strategies would require the countries to exchange a near-term reduction in resources for more- predictable and more-sustainable disbursements in the longer term. (See app. VII of our May 2018 report for detailed results of our analysis.) The compact trust fund committees have not taken the actions we recommended in 2018 to prepare for the 2023 transition to trust fund income. The committees have not yet prepared distribution policies, required by the trust fund agreements, which could assist the countries in planning for the transition to trust fund income. In addition, the committees have not established fiscal procedures for oversight of compact trust fund disbursements as required by the trust fund agreements. Further, the committees have not yet addressed a potential misalignment between the timing of their annual calculation of the amounts available to disburse and the FSM’s and RMI’s budget timelines, potentially complicating each country’s planning and management. The compact trust fund committees have not yet developed, as the compact trust fund agreements require, policies to guide disbursements from the trust funds after fiscal year 2023. Under the agreements, each trust fund committee must develop a distribution policy, with the intent that compact trust fund disbursements will provide an annual source of revenue to the FSM and RMI after the scheduled end of compact grant assistance. The trust fund committees could use distribution policies to address risks to each fund’s sustainability. For example, the committees have the discretion to disburse an amount below the established maximum. Our analysis of potential strategies for improving the funds’ outlook shows that reducing the size of disbursements would improve each compact trust fund’s long-term sustainability. Without a distribution policy that provides information about the size of expected disbursements, the FSM and RMI are hampered in their current and ongoing efforts to plan for the potential reduction in U.S. compact assistance after 2023. The compact trust fund committees have not yet established fiscal procedures for compact trust fund disbursements after fiscal year 2023. Each trust fund agreement requires the respective committee to determine the fiscal procedures to be used in implementing the trust fund agreement. The committees are to base their procedures on the compact fiscal procedures agreements, unless the parties to the trust fund agreement agree to adopt different fiscal procedures. No compact trust fund disbursements are to be made unless the committee has established such trust fund fiscal procedures. Without fiscal procedures in place, the trust fund committees will not be able to provide disbursements and the United States, the FSM, and the RMI will lack clear guidance to ensure oversight for trust fund disbursements. The timing for the compact trust fund committees’ calculation of the amounts available for annual disbursement to the FSM and the RMI after fiscal year 2023 does not align with the countries’ budget and planning timelines. The amounts available for disbursement in a given fiscal year cannot be determined until each fund’s returns have been determined at the end of the prior year. Further, if the disbursement amounts are calculated from audited fund returns as determined by annual audits required by the trust fund agreements, the amounts may not be determined until as late as March 31, 6 months into the fiscal year for which the disbursement is to be provided. However, both the FSM and the RMI government budget cycles are completed before the annual amounts available for disbursement will be known. As a result, the FSM and RMI would have to budget without knowing the amount to be disbursed, complicating their annual budget and planning processes. The compact trust fund committees, chaired by Interior, have continued to discuss potential actions to address the recommendations in our May 2018 report. In May 2018, we made six recommendations to Interior— three parallel recommendations regarding each country’s trust fund. We recommended that the Secretary of the Interior ensure that the Director of the Office of Insular Affairs work with other members of the trust fund committees to develop distribution policies, develop the fiscal procedures required by the compact trust fund address the timing of the calculation of compact trust fund disbursements. Interior concurred with our recommendations and has stated that it plans to implement them before the FSM and RMI transition to trust fund income in 2023. The FSM and RMI also concurred with our recommendations to Interior. According to the Trust Fund Administrator and Interior officials, the distribution policy was discussed at trust fund committee meetings convened since our May 2018 report. At their September 2019 meetings, the FSM and RMI compact trust fund committees did not make any decisions regarding steps to address our recommendations. The FSM’s and RMI’s transition to relying on income from the compact trust funds will likely require significant budgetary choices. However, the lack of trust fund distribution policies as well as the lack of alignment between the trust fund committees’ annual disbursement calculations and the countries’ budget cycles, hampers the countries’ ability to plan for the transition. In addition, without the required fiscal procedures governing trust fund actions after 2023, the trust fund committees will be unable to make disbursements and the United States, the FSM, and the RMI will not have assurance of necessary oversight for trust fund disbursements. However, as of September 2019, Interior had not implemented our recommendations to address these issues. Further, while Interior has continued to discuss possible actions to address our recommendations with the trust fund committees, it targeted implementation of our recommendations for 2023. Chairmen Grijalva and Engel, Ranking Members Bishop and McCaul, and Members of the Committees, this concludes my statement. I would be pleased to respond to any questions you may have. If you or your staff have any questions about this testimony, please contact David Gootnick, Director, International Affairs and Trade, at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Emil Friberg (Assistant Director), Ming Chen, Neil Doherty, Mark Dowling, Reid Lowe, Moon Parks, and Michael Simon. The amended compacts, compact–related agreements, the amended compacts’ implementing legislation, and other U.S. laws provide grants or eligibility for U.S. programs and services for the Federated States of Micronesia (FSM) and Republic of the Marshall Islands (RMI). The amended compacts provide compact sector, Kwajalein-related, and audit grants. Under current law, compact sector and audit grants are each scheduled to end in 2023, but the RMI military use and operating rights agreement (MUORA) extends the time frame of Kwajalein-related compact grants for as long as the agreement is in effect. The amended compacts’ implementing legislation provides additional grants, including authorizing a supplemental education grant (SEG), and identifies several specific U.S. programs as available to the FSM and RMI. Under current law, the additional grants end in 2023, but the statutory authorizations for some programs identified in Pub. L. No. 108-188 provide for the continued eligibility of the FSM and RMI to receive benefits under the programs. However, after fiscal year 2023, the FSM and RMI will no longer be eligible under current U.S. law for some programs that the SEG replaced. The compact-related programs and services agreements with each country identify additional programs and services that the United States makes available to the FSM and RMI. While these agreements will end in 2024, under current law, some U.S. agencies may continue to provide programs and services similar to those provided in the agreement under other authorities. Based on the status of current law, the FSM’s and RMI’s eligibility for other programs we identified that have been provided under other current U.S. laws will not change after fiscal year 2023. Under current law, compact sector grants provided to the FSM and the RMI under section 211(a) of the amended compacts are scheduled to end in 2023. However, the RMI is scheduled to continue to receive $7.2 million, partially inflation adjusted, related to the U.S. military base in Kwajalein Atoll and provided under section 211(b) of its compact. Under the terms of the RMI MUORA, the United States agreed to provide these Kwajalein-related grants for as long as the MUORA is in effect. The MUORA continues until 2066 and may be extended at the discretion of the United States until 2086. The amended RMI compact provides for $18 million, partially inflation adjusted, in annual payments to the RMI government to compensate for impacts from the U.S. Army Garrison– Kwajalein Atoll. These payments will continue for as long as the MUORA is in effect. Annual compact grants of up to $500,000 (not inflation adjusted) to each country to pay for required annual audits of compact grants are scheduled to end in 2023. See table 3 for a summary of compact sector, Kwajalein-related, and audit grants. The supplemental education grant (SEG) authorized by the amended compacts’ implementing legislation is scheduled to end in fiscal year 2023 and, under current law, FSM and RMI eligibility for most programs that the SEG replaced will not resume after fiscal year 2023. Absent changes to current law, the FSM and RMI will not be eligible after fiscal year 2023 for the following programs that the SEG replaced during fiscal years 2005 through 2023: U.S. elementary and secondary education grant programs, adult education and literacy programs, career and technical education programs, job training programs, and Head Start early education programs. However, under other provisions of current law, qualifying individuals in the FSM and RMI will be eligible after fiscal year 2023 for undergraduate education grants and work-study programs that the SEG replaced. See table 4. Although the programs and services agreements with the FSM and RMI will end in fiscal year 2024, current U.S. law enables U.S. agencies to continue providing some programs and services now provided under the agreements. After the agreements end, no current provisions of U.S. law will enable the Federal Emergency Management Agency (FEMA) to provide disaster response funding, enable the Federal Deposit Insurance Corporation to provide deposit insurance, or enable the U.S. Postal Service to provide the services that it currently provides to the FSM and RMI. However, the National Weather Service, the U.S. Department of Transportation’s (DOT) Federal Aviation Administration (FAA), and the U.S. Agency for International Development (USAID) could, under other legal authorities, provide services similar to those they now provide under the programs and services agreements. National Weather Service. The programs and services agreements authorize the National Weather Service to fund the operations of weather stations in the FSM and RMI, which it can continue to fund after the end of the agreements under other authorities, according to Department of Commerce officials. Federal Aviation Administration. The programs and services agreements authorize DOT’s FAA to provide technical assistance in the FSM and RMI, which it can continue to provide after the end of the agreements under other provisions of current U.S. law. However, DOT officials stated that FAA would require new bilateral agreements with the FSM and the RMI in order for the countries to continue to receive the civil aviation safety services that FAA currently provides under the programs and services agreements. The FAA would also seek reimbursement for any technical assistance it provides to the FSM and RMI. With regard to the civil aviation economic services provided under the programs and services agreements, DOT officials stated that, while the FSM and RMI could voluntarily decide to allow U.S. air carriers to continue operations in the FSM and RMI, new bilateral agreements would be needed to assure that result. U.S. Agency for International Development. Following a U.S. presidential disaster declaration, FEMA provides the funding for disaster relief and reconstruction, which is programmed through USAID. Under current law, FEMA funds will no longer be available to the FSM and RMI for this purpose once the agreements end; however, USAID will be able to provide foreign disaster assistance funding to the two countries under the same terms as it provides this assistance to other countries. After the programs and services agreements end, FEMA will be able to support disaster relief efforts only if USAID or the countries request such support on a reimbursable basis. In addition, according to State and Interior officials, telecommunications- related services that the two agencies provide to the FSM and RMI under the programs and services agreements will continue as long as the FSM and RMI provide appropriate authorization for such services. Table 5 shows the status, under current law, of programs and services currently provided to the FSM and the RMI under the programs and services agreements after the agreements end in fiscal year 2024. Although additional grants provided to the FSM and the RMI under the amended compacts’ implementing legislation will end in fiscal year 2023, the countries’ eligibility for programs now provided under that legislation will generally continue under current U.S. law. Grants provided under the amended compacts’ implementing legislation for (1) judicial training in the FSM and the RMI, and (2) agricultural and planting programs on the RMI’s nuclear-affected Enewetak Atoll are scheduled to end. However, under current U.S. law, legal authorities permitting the operation of other programs will remain available to the FSM and RMI after fiscal year 2023. Eligibility under these legal authorities continues either because the amended compacts’ implementing legislation does not specify an ending date or because other provisions in current U.S. law make the FSM and RMI eligible for the program. Programs provided in the amended compacts’ implementing legislation include U.S. Department of Agriculture Rural Utilities Service grant and loan programs; U.S. Department of Education Pell grants for higher education and grants under Part B of the Individuals with Disabilities Education Act for children with disabilities; programs for nuclear-affected areas in the RMI; and additional programs provided by the Departments of Commerce and Labor as well as law enforcement assistance provided by the U.S. Postal Service. See table 6 for a summary of the programs identified in the amended compacts’ implementing legislation and their status as of the end of fiscal year 2023. In addition to being eligible for the programs provided through the compact, its associated agreements, and the amended compacts’ implementing legislation, the FSM and RMI are also eligible for a number of programs under other provisions of current U.S. law. The FSM and RMI have each received funds from the U.S. Department of Agriculture for forestry and rural housing programs, multiple U.S. Department of Health and Human Services public health program grants, U.S. Department of the Interior technical assistance and historic preservation programs, and the DOT FAA airport improvement program, among others. Under current U.S. law, the legal authorities permitting the provision of these programs in the FSM and RMI would not necessarily change after 2023. Table 7 shows the FSM’s and RMI’s eligibility for these additional grants and programs under current law after fiscal year 2023.", "summary": "In 2003, the United States approved amended compacts of free association with the FSM and RMI, providing a total of $3.6 billion in economic assistance in fiscal years 2004 through 2023 and access to several U.S. programs and services. Compact grant funding, overseen by the Department of the Interior (Interior), generally decreases annually. However, the amount of the annual decrease in grants is added to the annual U.S. contributions to the compact trust funds, managed by joint U.S.–FSM and U.S.–RMI trust fund committees and chaired by Interior. Trust fund earnings are intended to provide a source of income after compact grants end in 2023. This testimony summarizes GAO's May 2018 report on compact grants and trust funds (GAO-18-415). In that report, GAO examined (1) the use and role of U.S. funds and programs in the FSM and RMI budgets, (2) projected compact trust fund disbursements, and (3) trust fund committee actions needed to address the 2023 transition to trust fund income. For this testimony, GAO also reviewed key variables for its trust fund model as of June 2019 to determine whether these variables had substantially changed. In addition, GAO reviewed the status of Interior's response to GAO's May 2018 recommendations. The Federated States of Micronesia (FSM) and the Republic of the Marshall Islands (RMI) rely on U.S. grants and programs, including several that are scheduled to end in 2023. In fiscal year 2016, U.S. compact sector grants and supplemental education grants, both scheduled to end in 2023, supported a third of the FSM's expenditures and a quarter of the RMI's. Agreements providing U.S. aviation, disaster relief, postal, weather, and other programs and services are scheduled to end in 2024, but some U.S. agencies may provide programs and services similar to those in the agreements under other authorities. GAO's 2018 report noted that the FSM and RMI compact trust funds face risks and may not provide disbursements in some future years. GAO projected a 41 percent likelihood that the FSM compact trust fund would be unable to provide any disbursement in 1 or more years in fiscal years 2024 through 2033, with the likelihood increasing to 92 percent in 2054 through 2063. GAO projected a 15 percent likelihood that the RMI compact trust fund would be unable to provide any disbursement in 1 or more years in fiscal years 2024 through 2033, with the likelihood increasing to 56 percent in 2054 through 2063. Potential strategies such as reduced trust fund disbursements would reduce or eliminate the risk of years with no disbursement. However, some of these strategies would require changing the trust fund agreements, and all of the strategies would require the countries to exchange a near-term reduction in resources for more-predictable and more-sustainable disbursements in the longer term. Interior has not yet implemented the actions GAO recommended to prepare for the 2023 transition to trust fund income. The trust fund committees have not developed distribution policies, required by the agreements, which could assist the countries in planning for the transition to trust fund income. The committees have not developed the required fiscal procedures for oversight of disbursements or addressed differences between the timing of their annual determinations of the disbursement amounts and the FSM's and RMI's annual budget cycles. In its May 2018 report, GAO made three recommendations to Interior regarding each country's trust fund to address trust fund disbursement risks. Interior concurred with GAO's recommendations and has continued to discuss actions in response at trust fund committee meetings, with implementation targeted for 2023.", "document_type": "gao"}
{"report": "DOD’s policy is to ensure that eligible personnel and their families have access to affordable, quality housing facilities and services consistent with grade and dependent status, and that the housing generally reflects contemporary community living standards. From the inception of MHPI, the military departments were provided with various authorities to obtain private-sector financing and management to repair, renovate, construct, and operate military housing in the United States and its territories. These authorities included the ability to make direct loans to and invest limited amounts of funds in projects for the construction and renovation of housing units for servicemembers and their families. The projects were generally financed through both private-sector financing, such as bank loans and bonds, and funds provided by the military departments. The Army and the Navy generally structured their privatized housing projects as limited liability companies in which the military departments formed partnerships with the developers and invested funds in the partnership. The Air Force generally provided direct loans to the developers. Because privatized housing projects involve budgetary commitments of the federal government, each project was scored at inception by the Office of Management and Budget to determine the amount of funds that needed to be budgeted for that particular project. The military departments have flexibility in how they structure their privatized housing projects, but typically the military departments lease land to developers for a 50-year term and convey existing housing located on the leased land to the developer for the duration of the lease. The developer then becomes responsible for renovating and constructing new housing and for the daily management of the housing units. At the end of fiscal year 2017, 14 private partners were responsible for 79 privatized military family housing projects—34 for the Army, 32 for the Air Force, and 13 for the Navy and the Marine Corps. See appendix II for a list of all of these housing projects. Each privatized housing project is a separate and distinct entity governed by a series of legal agreements that are specific to that project, hereafter referred to as business agreements. These agreements include, among other things, an operating agreement, a property management agreement, and an agreement that describes the management of funds in the projects, including the order in which funds are allocated within the project. However, while each project is distinct, there are some common elements in how projects invest and use funds. Every project takes in revenue, which consists mostly of rent payments. Projects then pay for operating expenses, including administrative costs, day-to-day maintenance, and utilities, among other things. After that, projects generally allocate funds for taxes and insurance, followed by debt payments. Figure 1 shows a typical funding structure for a privatized housing project. In the typical privatized housing project depicted in figure 1, once debt payments are made, funds are allocated to accounts that fund scheduled maintenance, such as repair and replacement of items like roofs, heating and cooling systems, and infrastructure. After that, funds are allocated to a series of management incentive fees, such as the property management fee. Finally, the project divides these remaining funds according to a fixed percentage between accounts that (1) fund major renovations and rebuilds and (2) are provided to the developer. The percentages may vary across agreements, but according to military department documentation, typically, the majority of funds go toward the accounts funding major renovations and rebuilds. Most of the projects’ business agreements also include the option for the private partners to receive performance incentive fees based on achieving the performance metrics established in each individual project’s business agreement. These fees are intended to incentivize private partner performance. The incentive fees can be paid to private partners on an annual or quarterly basis and can be withheld in part or in total if the private partner fails to meet the established metrics. The weight each performance metric and underlying indicator carries toward the incentive fee varies by project, so incentive fees for some projects may be heavily dependent on financial performance, while others may be more heavily weighted toward resident satisfaction. The Deputy Assistant Secretary of Defense for Facilities Management, under the authority, direction, and control of the Assistant Secretary of Defense for Sustainment, is responsible for all matters related to MHPI and is the program manager for all DOD housing, whether DOD-owned, DOD-leased, or privatized. In this capacity, the Deputy Assistant Secretary is to provide both guidance and general procedures related to military housing privatization, as well as required annual reports to Congress on privatized military housing projects. However, it is the responsibility of the military departments to execute and manage the privatized housing projects, including conducting financial management and monitoring their portfolio of projects. Each military department has issued guidance that outlines its responsibilities for privatized housing, such as which offices are responsible for overseeing privatized housing projects. See figure 2 for details on each military department’s roles and responsibilities in the MHPI program. We have previously reported on DOD’s privatized housing program. In 2002, we reported that although military installation officials were participating with developers in making improvement decisions for selected projects, DOD and military department headquarters oversight of those decisions appeared limited. We recommended, among other things, that DOD implement several changes to enhance government protections in the privatization program, such as requiring service headquarters and the OSD to review and approve privatization project reinvestment account expenditures over an established threshold. DOD generally agreed with our recommendations and took steps to implement them. Specifically, DOD revised guidance to establish new rules and thresholds for review and approval of project reinvestment expenditures, among other things. In addition, in 2006, we reported that although DOD and the individual military departments implemented program oversight policies and procedures to monitor the execution and performance of privatized housing projects, opportunities existed for improvement. Specifically, we reported that the value of DOD’s semiannual report to Congress was limited because it lacked a focus on key project performance metrics to help highlight any operational concerns. We also reported that data collected on servicemember satisfaction with housing, important for tracking satisfaction over time, were inconsistent and incomplete because DOD had not issued guidance for the standardized collection and reporting of such information. We recommended, among other things, that DOD streamline its report to Congress to focus on key project performance metrics and to provide guidance to the military departments to ensure the consistent collection and reporting of housing satisfaction information from all servicemembers. DOD generally agreed with our recommendations and took steps to implement them. For example, DOD took steps to streamline its report to Congress and update its guidance directing the services to ensure consistent reporting using a numerical rating system to rank housing satisfaction information. OSD and each of the military departments conduct a range of activities to oversee both the condition of privatized housing and performance of the private partners and have recently implemented initiatives to improve this oversight—such as increasing the frequency of the physical inspection of homes and issuing guidance to ensure consistency in the framework used to measure project performance. However, we found that these oversight efforts remain limited. Specifically, our review showed (1) the scope of oversight of the physical condition of privatized housing has been limited; (2) performance metrics focused on quality of maintenance and resident satisfaction may not accurately reflect private partner performance related to the condition of privatized housing; (3) there is a lack of reliable or consistent data on the condition of privatized housing; and (4) past DOD reports to Congress on resident satisfaction are unreliable due to the inconsistent handling and calculation of the data and therefore may be misleading. The military departments have taken steps to oversee the condition of their privatized military housing inventory and each has issued guidance that outlines their respective oversight roles and responsibilities, but the scope of these oversight efforts has been limited. Military department oversight activities generally fall into two categories—(1) daily oversight of management and operations and (2) periodic reviews of compliance with each project’s business agreements. Daily oversight of management and operations. Each installation has a military housing office that is responsible for conducting daily oversight of a project’s management and operations. Military housing officials told us that activities to monitor the physical condition of housing units generally include reviewing sample work order requests, following up with a sample of residents to check on their experience with recently completed work, and inspecting homes during the change-of-occupancy process. However, the implementation and scope of these activities varies and can be limited. For example, during our site visits conducted from June through August 2019, we identified the following installation- specific practices: The rate of inspections of homes following change-of-occupancy maintenance at the installations we visited varied. For example, at the time of our site visits, military housing office officials at Tinker Air Force Base, Oklahoma, told us that they inspect 100 percent of homes that have completed change-of-occupancy maintenance, while officials from Langley Air Force Base, Virginia, stated that they inspect 10 to 20 percent of these homes. In November 2019, Air Force officials told us that they are moving to a 100-percent inspection policy. Similarly, the Army issued an order in March 2019 directing military housing office officials to inspect 100 percent of homes where change-of-occupancy maintenance has been completed. Officials from Army installations we visited noted that this was an increase from previous practices, and for one installation was a change in practice from conducting inspections only during the move-out process, which occurs prior to change-of-occupancy maintenance. According to Department of Navy officials, the Navy’s business agreements stipulate that Navy and Marine Corps installations have access to all work order information. However, practices for following up on work order records varied among some of the Navy and Marine Corps installations we visited. For example, military housing office officials at Camp Pendleton, California, told us that for one of the two partners that own housing on the base, they had access to only 3 percent of completed work orders from the previous month. For the other partner that owns housing on the base, military housing office officials noted that the partner provided them with nine work orders of varying priority each month to review. One military housing office official added that these were the minimum requirements needed for monthly reporting and that they were working with the private partner to increase their access to work order records. Following a different practice, military housing office officials at Naval Station Norfolk, Virginia, told us that they had access to the private partner’s maintenance record system and would pull reports on homes that had made six or more maintenance calls in a 30-day period. Periodic reviews of compliance with each project’s business agreements. Periodic reviews of compliance with a project’s business agreements are a joint effort between the local military housing office, the private partners, military department installation commands, and other echelons of command. These reviews can include neighborhood tours to view project amenities such as community centers, playgrounds, and pools, all of which are owned, maintained, and operated by the private partner companies, as well as exteriors of housing units. However, similar to the daily oversight activities, these annual reviews have been narrow in the scope of their assessment of the physical condition of the housing units, as interior walk-throughs were, at times, focused on just a few homes at each installation. For example: The Air Force Civil Engineer Center is the primary oversight and governance body for the Air Force’s privatized housing projects. The Air Force oversight process includes periodic compliance reviews of all privatized housing projects. To accomplish this task, the Air Force is to use a compliance checklist to review the private partner’s compliance with a project’s business agreements. In addition to the compliance reviews, guidance states that Air Force Civil Engineer Center officials visit projects annually, and officials told us that they tour a sample of homes and interview private partner representatives, military housing office staff, and residents during these visits. However, according to selected annual site visit reports we reviewed and a discussion with an Air Force official, annual site visit reports typically include only an evaluation of three to four housing units on an installation and can be restricted to empty units or units that have completed change-of-occupancy maintenance, limiting the robustness of the assessment of the installation’s housing units’ physical condition. According to Department of the Navy officials, the Navy and the Marine Corps provide oversight of privatized housing projects through a tool called the monitoring matrix. Officials from the various organizational entities involved with privatized housing—to include the Commander, Naval Installation Command; the Naval Facilities and Engineering Command; and the military housing office—are to use this monitoring matrix to periodically review private partner compliance with a project’s business agreements. The matrix contains a condition assessment component, which includes a tour of privatized housing neighborhoods and a visual inspection of individual privatized housing units. However, similar to the Air Force, according to select assessments we reviewed and a discussion with a military housing office official, the visual inspections are typically focused on two to three homes in each neighborhood on an installation and to homes that have recently undergone change-of-occupancy maintenance. Army guidance calls for the U.S. Army Corps of Engineers to conduct an annual ground lease inspection to review private partner compliance with a project’s business agreements. The guidance also calls for the Army’s program manager to conduct an annual installation visit to each project to evaluate performance and ensure a project’s compliance with the business agreements. The visit is to include a recommended site tour, described in guidance as a brief visual inspection tour of community elements, and a walk-through visual inspection of at least four housing units—two renovated and two recently built—including one unit designated as an accessible home under federal guidelines. However, according to a May 2019 report by the Army Inspector General, these requirements were inconsistently met, and the results did not include a follow-up process and were not communicated to senior commanders. Through the recent housing reviews that they have conducted, each military department’s internal oversight body has recognized that the departments’ oversight guidance has been limited in addressing the condition of privatized homes and provides little clarity to housing officials about their roles and responsibilities in assessing the physical condition of homes. For example, in May 2019, the Department of the Army Inspector General reported that senior commanders and garrison staffs expressed confusion concerning the roles, responsibilities, and authorities regarding privatized housing and that oversight, governance, and synchronization were insufficient to identify current housing challenges. Similarly, an April 2019 report from the Air Force Inspector General noted that ambiguous guidance had resulted in inconsistent action and uneven performance across Air Force housing projects. In addition, a November 2019 report by the Naval Audit Service identified nine separate guidance documents for the oversight of privatized housing and found that personnel at installation and regional levels were unclear on the guidance and requirements for performing oversight of privatized housing. According to military department officials, each department has completed initiatives and is undertaking initiatives to revise guidance and standardize daily oversight activities in an effort to provide consistent oversight across projects and installations and to increase the focus on the physical condition of housing. In addition, the military departments have initiatives to increase staffing levels, improve training for military housing office officials, and ensure that military department housing officials have independent access to work order data to strengthen their oversight activities. Figure 3 outlines examples of completed and ongoing initiatives by military department to improve the oversight of privatized housing. However, each military department is working to implement service- specific initiatives with minimal guidance from OSD on the level of oversight expected as it relates to the condition of privatized housing. OSD guidance as it pertains to the condition of privatized housing is limited compared with the guidance OSD provides for monitoring the condition of military-owned housing. Specifically, OSD guidance is focused on the oversight of the implementation of projects, the construction of new housing units, and project financial monitoring. The guidance stipulates that after privatized housing projects are awarded, monitoring should include descriptions of deal structure and strategies for project monitoring. In contrast, OSD guidance for military-owned housing provides clearly defined objectives to the military departments for oversight, including the physical condition of the homes. For example, the DOD manual for housing management directs the military departments to provide managerial oversight of DOD’s government-owned family housing to ensure that (1) the required inventory is being provided and maintained in good condition, (2) the program is being operated in an effective and cost-efficient manner, and (3) servicemembers and their families have adequate housing choices. Further, the manual provides specific objectives for the condition of DOD’s government-owned family housing, stating that for DOD family housing to be considered adequate overall, it must meet minimum standards for configuration, privacy, condition, health, and safety. It also states that military service condition assessments shall use private-sector housing industry and DOD standards or codes as a basis for assessing inventory adequacy. The manual adds that for DOD government-owned family housing to be considered in adequate condition, the construction cost for all needed repairs and improvements cannot exceed 20 percent of the replacement cost. According to DOD’s housing manual, program assumptions for privatized housing are that privatization allows the military departments to work with the private sector to generate housing built to market standards. While the military departments’ policies provide for some measureable oversight activities, such as requiring a certain number or type of home to be inspected, OSD has not provided guidance to the military departments clearly defining oversight objectives for monitoring the physical condition of privatized housing units. DOD’s housing manual further states that because privatization creates a long-term governmental interest in privatized housing, it is essential that projects be attentively monitored. The 50-year term for the ground leases creates a long-term interest in monitoring the privatized housing assets, to include the physical condition of the housing units. However, unless DOD updates its guidance on the oversight of privatized housing with objectives for overseeing the physical condition of housing units, it cannot be assured that the military departments’ oversight activities will be sustained over time or be sufficiently consistent across projects, raising the risk that private partners may not provide adequate quality housing. Notably, the military departments have entered into privatized housing agreements with some of the same companies, and members of different military services may live at installations managed by military services different than their own. As such, it is important that oversight expectations generally be consistent across the military departments and the projects they manage. Moreover, all military departments have an interest in ensuring that residents feel confident that the private partners will be held to a consistent standard for maintaining the condition of their homes. Participants in 8 of our 15 focus groups stated that they will no longer live in privatized housing following their current experience, and participants in 6 of our 15 focus groups stated that their current experience with privatized housing will affect the future career decisions for their family. One participant stated that he plans to exit the service after 8 years, noting that his decision is largely based on his experience with privatized housing. In addition, in our online tool we asked residents if their experience with privatized housing would impact their future career and housing decisions. For those residents that responded to these questions, the majority said their experience will make them less likely to continue to live in privatized housing in the future. For example, one respondent stated that while living in privatized housing is a benefit to being in the military, living in housing that is subpar and where nothing seems to be getting fixed or at least acknowledged makes the family hesitant to live in privatized housing again. Some residents also indicated that their experience would impact their future career decisions. The military departments each use a range of project-specific performance metrics to monitor private partner performance. However, the indicators underlying the metrics designed to focus on resident satisfaction and on the quality of the maintenance conducted on housing units may not provide meaningful information or reflect the actual condition of the housing units. For example, in April 2019 the Air Force Inspector General reported that the current incentive structure measures many things with precision, but does not measure the right things. Private partner performance is commonly measured through four key metrics—resident satisfaction, maintenance management, project safety, and financial management. To determine how well the private partners are performing under these metrics, military housing office officials told us that they rely on a range of indicators established in the project business agreements. Table 1 provides examples of various indicators that the performance metrics comprise. According to officials from each military department, the performance metrics and their underlying indicators are a key tool that each military department uses to hold private partners accountable for providing quality management of the privatized housing projects. However, we found that the indicators themselves may not reflect how the private partner is performing in terms of providing servicemembers and their families with quality services and housing. For example: Maintenance management: One commonly used indicator of performance in maintenance management measures how often the property manager’s response time to work orders meets required time frames established in the project’s business agreements. While this indicator measures the timeliness of the private partner’s response, it does not measure or take into account the quality of the work that was conducted or whether the resident’s issue was fully addressed. As such, a property manager may fully meet the metric for maintenance management, even if a given repair has not been adequately completed. Residents in 13 of our 15 focus groups noted that they typically have had to submit multiple work order requests before an individual maintenance issue has been fully addressed. For example, a resident who participated in one of our focus groups provided us with a copy of work orders she had submitted related to a single maintenance issue in her home. The first work order was marked completed on time, yet the resident had to submit a work order for the same issue a week later. Further, an official at one Army installation told us that since the incentive fee for the project is awarded on a quarterly basis, judging property managers only on the basis of work orders completed on time for that quarter could mask persistent ongoing housing problems. This is because many smaller work orders get closed out each quarter, while work orders for more complicated issues might stay open over multiple quarters. Some projects include indicators that aim to more directly measure quality, such as the number of work orders placed during the first 5 business days of residency. This type of indicator may more clearly indicate the extent to which change-of-occupancy maintenance was complete on a given home. Resident satisfaction: One example of an indicator of resident satisfaction is whether a project has met target occupancy rates established in the project’s business agreements. An OSD official and private partner representatives told us they use occupancy as an indicator of satisfaction, based on the assumption that residents would move if they are dissatisfied with their home’s condition. However, according to the Army’s Portfolio and Asset Management Handbook, occupancy rates are not a recommended metric to monitor private partner performance because occupancy rates already impact project finances. Our focus groups and the responses we received to our online tool also indicate that this may not be a reliable assumption. Although most residents are not required to live in military housing, residents in each of our 15 focus groups and responses to our online tool indicated a variety of reasons for choosing to live in privatized housing, many of which do not have to do with their satisfaction with the quality or condition of their homes. For example, residents in our focus groups cited other factors influencing their decision to live in privatized housing, such as living in close proximity to military medical or educational services for children or other family members that are part of the military’s Exceptional Family Member Program, a lack of safe and affordable housing in the surrounding community, and access to quality schools. Volunteers that responded to our online tool also cited accessibility to base services, commute time, and safety as reasons for choosing to live in privatized housing. Another commonly used indicator of resident satisfaction is the results of various resident satisfaction surveys, such as maintenance surveys and leasing surveys, as well as the annual satisfaction survey. The military departments and the private partners use these survey tools to gauge resident satisfaction with the maintenance conducted on their homes, service provided by property managers, and amenities provided in their community, among other things. However, residents in 4 out of our 15 focus groups indicated that the surveys they receive related to maintenance performed on their homes do not ask questions about the quality of maintenance work. For example, residents told us that maintenance surveys, which they generally receive after maintenance work is completed on their homes, ask if the maintenance worker was courteous, but not about the quality of the work performed on the home. We reviewed maintenance surveys from 3 of the 10 installations we visited and found that the surveys asked residents to provide feedback on the quality of the work, with questions asking them to rate their satisfaction with the quality of the maintenance work completed. In addition, we reviewed a quarterly Army survey from one of the installations we visited and found that this survey asked residents about their satisfaction with the courteousness and professionalism of the maintenance team and the responsiveness and timeliness of maintenance work, but did not specifically ask about their satisfaction with the quality of the maintenance work completed. We also found that the information used to support the indicators can vary. For example, officials at one Army installation—Fort Huachuca, Arizona—use quarterly resident surveys, the Army’s annual survey, and action plans on Army annual survey results as indicators of resident satisfaction. However, officials at another Army installation—Fort Knox, Kentucky—use residential community office relationship management and point of service surveys. Similarly, we found differences in the information used as indicators of the maintenance management metric. For example, officials at both Hickam Air Force Base, Hawaii, and Davis- Monthan Air Force Base, Arizona, rely on the timeliness and quality of change-of-occupancy maintenance as an indicator of maintenance management. However, officials at Hickam Air Force Base also use work order response and completion times as indicators of the maintenance management metric, whereas officials at Davis-Monthan Air Force Base, Arizona, only use work order response times. Standards for Internal Control in the Federal Government state that management should evaluate performance and hold individuals accountable for their internal control responsibilities. If management establishes incentives, management should recognize that such actions can yield unintended consequences and evaluate incentives so that they align with the entity’s standards of conduct. The standards further state that management should use quality information to achieve the entity’s objectives, including relevant data from reliable sources. In October 2019, OSD, in collaboration with the military departments and private partners, issued new guidance standardizing the performance incentive fee framework across the military departments. The new guidance provides a framework for standardizing the minimum and maximum percentages of the fee that each metric can account for, allowing for some flexibility in the weight each metric will carry for an individual project. Specifically, maintenance management and resident satisfaction can account for between 60 and 90 percent of the fee, project safety can account for between 5 and 15 percent of the fee, and financial performance can account for between 5 and 15 percent of the fee. However, despite DOD’s efforts to ensure more focus on the condition and quality of, and resident satisfaction with, privatized housing through the standardization of metrics across the military departments, the metrics may be misleading if the specific underlying indicators used to determine whether a metric has been reached are not reevaluated on an ongoing basis to ensure they are accurate measures of the private partners’ performance and an accurate reflection of the condition and quality of privatized homes. OSD and military department officials have recognized that the current indicators for measuring performance do not consistently focus on or prioritize the private partners’ performance with maintaining housing units and ensuring resident satisfaction. For example, Army officials told us they are no longer using occupancy rate as an indicator of resident satisfaction and have taken steps to standardize performance indicators across all Army projects, while still allowing for flexibility at the installation level to modify the weight of indicators to provide incentives reflective of the specific needs of the installation. Limitations to the current indicators may hinder the military departments’ ability to accurately determine private partner performance. OSD and the military department officials told us they have not yet reevaluated the specific indicators used to determine whether a private partner has met a specific metric because doing so will require negotiation with each of the private partners for each project. However, without reviewing the specific indicators used to award performance incentives, OSD and the military departments do not have assurance that the information the military departments are using to award these incentives reflects the actual condition of the housing. Maintenance data collected by the private partners are not captured consistently or reliably across projects for use in ongoing monitoring of the condition of privatized housing units over time. The privatized housing projects’ business agreements typically include a requirement for the private partner to maintain a records management system to record, among other things, maintenance work requested and conducted on each housing unit. According to private partner representatives from all 14 companies, each company uses commercial property management software platforms for activities such as initiating maintenance work orders and dispatching maintenance technicians. Some private partner representatives stated that while data from the work order tracking systems are primarily used to prioritize and triage maintenance work, the data were never intended to monitor the overall condition of privatized housing units. Military department officials told us that efforts are underway to monitor work order data from the private partners’ work order tracking systems in an effort to increase the military departments’ oversight and accountability of the private partners for providing quality housing to servicemembers. For example, the Army and the Navy are taking steps to create data dashboards to track installations’ work orders by priority, status, and category. However, while data from these work order tracking systems may be useful for point-in-time assessments of work order volume at a given installation, we found that these data are not captured reliably or consistently for use in ongoing monitoring of the condition of privatized housing units across projects and over time. We received and reviewed data from each of the 14 private partners’ work order tracking systems covering each of the 79 privatized housing projects. Based on our review of these data and discussions with private partner representatives, we found two primary factors that would limit the reliability or consistency of using these data for ongoing monitoring of the condition of privatized housing units over time—(1) inconsistent use of terminology in work order records and (2) differing practices for opening and closing work orders. Data in these work order tracking systems include information such as records of resident requests for service, history of work conducted on specific housing units, change-of-occupancy maintenance performed, and work completed on common areas. Residents may request service for a broad range of issues, such as lost keys, broken appliances, ceiling or wall damage, lack of hot water, or water leaks or floods. According to private partner representatives, work orders can be entered into the system by property management office staff, maintenance technicians, or call center representatives for those companies that use offsite call centers to process resident service request calls. At some installations, residents can also enter work orders into the work order tracking system through online portals or mobile applications. However, we noted cases where work orders were inconsistently entered into the work order tracking systems with respect to two primary factors— (1) how the request is described by the resident or interpreted by the individual entering the data, which can differ for each work order; and (2) the existing range of pre-established service category options in the private partner’s work order tracking system, which differ among the partners. According to private partner representatives, the individual responsible for entering the work order into the system—property management office staff, maintenance technicians, call center representatives, or residents—makes a judgment on how to categorize the work order. These factors create challenges for looking at the data across projects. Private partner representatives from one installation we met with stated that the quality of the work order data is dependent on the data input into the system. In some cases, the data input can be inaccurate or imprecise, depending on the specificity with which a resident describes his or her maintenance issue or how a staff person enters the data into the system. A private partner representative from another installation we visited stated that reporting on data from the work order tracking system can be challenging because individuals across installations inputting data may have a different interpretation of a resident’s reported issue. Private partner representatives from another installation noted that the work order tracking system they used could not be easily updated with a new category if needed, making it more difficult to identify systemic issues. For example, there is one category for all exterior repairs, but no way to break that category down into what the specific repairs are, such as roofs. In the event that there is an issue with several roofs in the same area, the private partner representative said it would be hard to identify the issue because the only option available is to look through the notes section. According to this individual, the regional maintenance technicians, not the work order tracking system, are the best resource for identifying trends or recurring issues. This inconsistent entering of information into the work order tracking systems, which occurs both within and across installations, means that the military departments cannot readily use the data to capture the prevalence of a particular issue, such as mold, among the homes in a reliable manner. For example, if someone wanted to use work order data to track instances of mold, he or she would find that these may be represented in the work order systems under a variety of service categories, such as mold or mildew, plumbing and bath, heating and cooling, or general. To isolate service requests related to mold, one may have to rely on using the service comments for each request, which can vary in their level of detail. In addition, service requests for mold issues may be entered into the work order systems under different priority levels, such as routine, urgent, or emergency. As a result of the variation in the type and amount of information collected in the work order tracking systems, work order data alone cannot be used to determine the validity of a service request, the severity of the problem, or whether the work was completed to a quality standard. Figure 4 shows examples of differences in how a perceived mold issue can be captured in these systems based on our review of the data provided by the private partners. Military department officials found similar limitations when analyzing the work order data. According to some officials, one challenge in using the work order data for oversight is that, while there are good data in the individual records, people report and record things differently. Specifically, a Navy official working with these data told us they have to consider these differences and create unique algorithms to query data for each partner. At some installations we visited, private partners noted changes in practices for opening or closing work orders, limiting the usefulness of data in monitoring the status of work orders over time and thus the condition of privatized housing. For example, according to private partner representatives at one installation we visited, a practice for tracking emergency work orders in the work order tracking system had changed in 2013. Work that comes in under an emergency priority may take several steps to complete: A maintenance technician may first have to stop the emergency, then clean up any resulting damage, before repairing the root cause and completing any finishing work. Prior to 2013, maintenance technicians would open and close new work orders for each step in the process. Under the new practice, the original work order is kept open until completion. Representatives from a different private partner described a similar change in practices, noting that if a work order was closed or recategorized before the work was finished there could be issues for how it is tracked, such as getting dropped out of the system and the work not getting done. A third partner noted the same practice, but added that an emergency work order can be downgraded to urgent or routine status during the time that the work is taking place. As a result, work order data alone may not accurately identify the number of open work orders at any given time, the time it took to address a maintenance issue, or if a maintenance request has been fully completed. Additionally, we identified anomalies in the work order data provided to us from each of the 14 partners. For example, we identified instances of, among other things, duplicate work orders, work orders with completion dates prior to the dates that the resident had submitted the work order, and work orders still listed as in-progress for more than 18 months. According to military department officials, they have increased their efforts to review data from the private partners’ work order tracking systems and have found similar anomalies. For example, a Navy official working with work order data found that a couple of homes had six or seven unique work order records in the system, but each contained identical information in the various data fields. Officials from both the Navy and Air Force have come across work order records that were marked as complete within minutes of being entered into the system or marked as complete with a date prior to the work order being open, which signaled the need for further scrutiny. Each military department has efforts underway to monitor private partner work order data in an effort to increase oversight of the quality of privatized housing. However, because neither OSD nor the military departments have identified minimum data requirements, established consistent terminology or practices for data collection, or developed processes for the military departments to validate the work order data collected by the private partners, data from these work order tracking systems are not reliable for use in the ongoing monitoring of the condition of privatized homes. Further, military department data monitoring efforts are department-specific, even though the departments have entered into privatized housing agreements with some of the same companies. Standards for Internal Control in the Federal Government state that management should use quality information to achieve the entity’s objectives and design information systems and related control activities to achieve objectives and respond to risks. Information, among other things, should be complete and accurate. The standards also state that management should define the identified information requirements at the relevant level and requisite level of specificity for appropriate personnel. Without direction from OSD to establish minimum data requirements and consistent terminology or practices for data collection, as well as a requirement for the military departments to validate the data, the military departments’ ability to use data from the private partners’ work order tracking systems to monitor the condition of privatized homes over time is limited and may vary across projects. DOD has provided periodic reports to Congress on the privatized housing program; however, reported results on resident satisfaction have been unreliable and are misleading due to (1) variances in the data the military departments collect and provide to OSD and (2) OSD’s calculation and presentation of the data. DOD is statutorily required to provide reports to Congress that include, among other things, information about military housing privatization projects’ financial health and performance and the backlog, if any, of maintenance and repairs. These reports have also included information on resident satisfaction based on the results of the annual military department satisfaction surveys. In May 2019, DOD issued its report for fiscal year 2017, which stated that overall resident satisfaction for calendar year 2017 was 87 percent. However, this number is misleading due to issues associated with the collection and calculation of the data DOD used. The military departments provide data on resident satisfaction to OSD for inclusion in DOD’s submission to Congress based on information from the annual resident satisfaction surveys. Specifically, OSD’s instructions to the military departments for the fiscal year 2017 report required the military departments to report the following: The month and year of the most recently completed tenant satisfaction survey. The number of residents surveyed and the total number of tenants who completed the survey during the reporting period. Resident responses to the question that asks: “Would you recommend privatized housing?” Results should indicate how many tenants responded “Yes,” “No,” or “Don’t Know.” However, instead of asking whether residents would recommend privatized housing, the military departments’ annual resident satisfaction survey asked residents the following: “How much do you agree or disagree with the following statement, ‘I would recommend this community to others.’” The difference in the wording between the question asked of residents and the question reported to Congress is notable, as a resident’s satisfaction with his or her community and inclination to recommend it to others may not be reflective of satisfaction with either the privatized housing unit or privatized housing in general. We also found differences in how the military departments interpreted responses to the question they asked. When asked whether they would recommend their community to others, residents were provided the following response categories on a scale of five to zero: (5) strongly agree, (4) agree, (3) neither agree nor disagree, (2) disagree, (1) strongly disagree, and (0) not applicable, no opinion, don’t know, or no answer. However, we found that the ways in which the military departments translated these responses into the “yes,” “no,” or “do not know” categories differed across the military departments, and in the case of the Army differed from year to year. For the fiscal years 2015 through 2017 reports, Navy officials told us they counted responses reported in categories 5 (strongly agree) and 4 (agree) as “yes,” responses in categories 2 (disagree) and 1 (strongly disagree) as “no,” and responses in categories 0 (not applicable, no opinion, don’t know, or no answer) and 3 (neither agree nor disagree) as “don’t know.” For the same time period, Air Force officials told us they counted responses in categories 5 (strongly agree), 4 (agree), and 3—neither agree nor disagree—as “yes,” responses in categories 2 (disagree) and 1 (strongly disagree) as “no,” and responses in category 0 (not applicable, no opinion, don’t know, or no answer) as “don’t know.” If 3 had not been counted as “yes,” the reported resident satisfaction rate would have been lower. For example, for Lackland Air Force Base, Texas, if officials had not counted responses in category 3 as “yes,” the resident satisfaction rate for newly constructed units would have been more than 20 percent lower than what was reported. The Army calculated responses differently for fiscal years 2015, 2016, and 2017. Specifically: For the fiscal year 2017 report, the Army counted responses in categories 5 (strongly agree) and 4 (agree) as “yes,” responses in categories 2 (disagree) and 1 (strongly disagree) as “no,” and responses in categories 0 (not applicable, no opinion, don’t know, or no answer) and 3 (neither agree nor disagree) as “don’t know.” For the fiscal year 2016 report, the Army counted responses in categories 5 (strongly agree) and 4 (agree) as “yes,” responses in categories 2 (disagree), 1 (strongly disagree), and 0 (not applicable, no opinion, don’t know, or no answer) as “no,” and responses in category 3 (neither agree nor disagree) as “don’t know.” For the fiscal year 2015 report, the Army counted responses in categories 5 (strongly agree), 4 (agree), and 3 (neither agree nor disagree) as “yes,” responses in categories 2 (disagree) and 1 (strongly disagree) as “no,” and responses in category 0 (not applicable, no opinion, don’t know, or no answer) as “don’t know.” In addition, we identified errors and inaccuracies in how OSD calculates these data and reports them to Congress. Specifically, we found missing data points and incorrect formulas, among other errors, in OSD’s calculation of the data submitted by the military departments. For example: The formula used by OSD to calculate overall resident satisfaction for fiscal year 2017 did not include data for several projects, including four Army projects—Fort Bragg, North Carolina; Fort Knox, Kentucky; Joint Base Lewis-McChord, Washington; and Presidio of Monterey/Naval Postgraduate School, California. As of September 30, 2017, these four projects accounted for over 18 percent of the Army’s total housing inventory. The formula used by OSD to calculate resident satisfaction by project double counted resident satisfaction data for new and unrenovated homes for Vandenberg Air Force Base, California, by incorrectly using the Vandenberg Air Force Base data for both the Vandenberg and for the Fort Huachuca/Yuma Proving Ground project. As a result, incorrect data were reported for the Fort Huachuca/Yuma Proving Ground project for some categories of homes. OSD did not include resident satisfaction data for New Orleans Naval Complex, Louisiana, in its fiscal year 2017 report to Congress, even though the Navy had included data for this project when submitting its data to OSD. OSD also reported identical resident satisfaction data for Wright-Patterson Air Force Base, Ohio, in fiscal years 2015, 2016, and 2017, despite the fact that Air Force officials noted in their submissions to OSD that the annual resident satisfaction data was from the annual resident satisfaction survey for Wright-Patterson Air Force Base conducted December 2013. Further, Army data provided to OSD had calculation errors that OSD did not reconcile. Specifically, the Army provided OSD the total number of surveys received for a project, as well as the number of surveys broken out by different housing categories. However, we found instances where the sum of the data broken out by different housing categories was not equal to the reported total number of surveys received. For example, when we reviewed data for Fort Rucker, Alabama, the calculated sum of surveys broken out by different housing categories was 1,372, but the Army reported a total of 530 surveys received, a difference of 842 surveys. Further, the presentation of data in OSD’s report to Congress is misleading because OSD did not explain the methodology it used to calculate the overall resident satisfaction percentage or include caveats to explain limitations to the data. Specifically, OSD did not include information on overall response rates to the annual satisfaction survey for each military department, nor did it include response rates by project. Low response rates can create the potential for bias in survey results. For example, in its fiscal year 2017 report, OSD reported that 25 percent of residents living in renovated homes at the MHPI project including Fort Detrick, Maryland/Walter Reed Army Medical Center, Washington, D.C., were satisfied with their housing. However, only four residents provided responses to this question, meaning that just one resident reported being satisfied. In addition, we found that OSD did not include an explanation in the report for why five projects were listed as not applicable. According to OSD officials, this error was a quality control issue that they plan to address, but the officials told us there are no plans for quality control in development at this time. The Fiscal Year 2020 NDAA includes a provision requiring each military installation to use the same satisfaction survey for tenants of military housing—including privatized military housing—the results of which are not to be shared with private partners until reviewed by DOD. The statute also states that DOD’s reports to Congress shall include additional information, such as the results of residence surveys, as well as assessments of maintenance response times, completion of maintenance requests, the dispute resolution process, overall customer service for tenants, and other factors related to the condition of privatized housing. OSD’s report to Congress states that, given DOD’s objective of improving the quality of life for its servicemembers, the degree of satisfaction military families experience in privatized housing is a critical indicator of overall program success and the military departments and private partners use tenant surveys to help assess the quality of privatized housing. Additionally, Standards for Internal Control in the Federal Government state that management should obtain relevant data from reliable internal and external sources in a timely manner based on identified information requirements. Relevant data have a logical connection with, or bearing upon, the identified information requirements. Reliable internal and external sources provide data that are reasonably free from error and bias and faithfully represent what they purport to represent. Management should evaluate both internal and external sources of data for reliability, and obtain data on a timely basis so they can be used for effective monitoring. However, the errors we identified in OSD’s data calculations, as well as the differences in how the military departments translate data provided to OSD, indicate the need for better internal controls, including a process for collecting and calculating resident satisfaction data from the military departments, and explanation of the data collected and reported on resident satisfaction to ensure they are reasonably free from error and bias and represent what they purport to represent. According to an OSD official responsible for preparing the reports to Congress, her office inherited the MHPI report process from its predecessors and had to quickly catch up on reports because DOD was behind on its reporting requirement. However, she noted her office is working with the military departments to review the resident satisfaction survey questions and will be identifying and implementing measures to ensure an accurate and reliable process to compile, calculate, report, and compare MHPI residents’ satisfaction by military department and across DOD. Additionally, for future survey data reporting, OSD officials told us they plan to research the possibility of directly collecting resident survey data from the survey administrator to minimize data transcription errors. Until OSD makes these changes to the data collection and calculation efforts that make up the department’s report to Congress and provides explanations of the data in the reports, OSD will not be able to provide Congress with an accurate picture of resident satisfaction with privatized housing. Military housing offices located at each installation are available to provide resources to servicemembers experiencing challenges with their privatized housing, among other services, but these offices have not always clearly and systematically communicated this role to residents of privatized housing. Military housing office officials noted that servicemembers living in privatized military housing primarily interact with their installation’s military housing office when they first receive orders to move to an installation. The military housing office provides new residents with information on their local housing options, to include referral services for housing options. However, military department guidance calls for the military housing office to provide continued assistance to servicemembers and their families living in privatized housing. For example, each military department has guidance that establishes the role of its housing offices in the resident dispute resolution process—specifically, if servicemembers are experiencing a dispute with the private partner: Army policy states that each installation should have an official tasked with providing support to servicemembers regarding resident issues that cannot be resolved by the private property manager. This individual is also in charge of resolving every resident complaint, and the military housing office, if required, can request mediation by the garrison commander. Air Force policy directs installation commanders to establish regular meetings with the private partners to discuss resident disputes and develop resolutions for residents’ issues. Also, the Air Force business agreements for each project are to establish Management Review Committees, in which the private project owner, Air Force housing office officials, and the Air Force Civil Engineer Center meet quarterly to review and facilitate the resolution of prevalent issues. The Navy announced a standardized two-step resolution process in May 2019 for housing residents who have issues or concerns with their current homes. The first step is to report any issue to the local property manager. If the issue is not resolved in either a timely manner or to quality standards, residents are asked to contact their local Navy housing service center, which directly reports to the installation commanding officer, or the servicemember’s chain of command. Prior to the standardization of this process, Navy guidance established a general responsibility to assist residents in the dispute resolution process and each project’s tenant lease includes specific dispute resolution processes. The Marine Corps has established a three-step dispute resolution process for residents to follow when they are experiencing a dispute with the private partner. Further, Marine Corps policy calls for each of the private partners to establish standard operating procedures that should include complaint resolution procedures. Despite established military department guidance, we found that residents were sometimes confused and lacked awareness of the availability of the military housing office to assist them with issues they were experiencing with privatized housing. For example, residents who participated in our focus groups and responded to our online tool expressed the following concerns: At least one resident in each of our focus groups noted being sometimes confused about the military housing office’s roles and responsibilities with regard to the maintenance of their home. These residents indicated they did not know the military housing office existed or could serve as a resource. Further, some individuals that responded to our online tool indicated that they did not know they could reach out to military housing office officials or their chain of command with issues related to the condition of their home. Residents in at least three of our focus groups indicated they perceived that the military housing office was not working independently of the partner or in the residents’ best interest. For example, residents in at least three focus groups noted that they viewed the military housing office as an extension of the private partner. Other residents noted that they did not know what the military housing office was or what role the office plays in managing privatized housing. In addition, residents we solicited information from through our online tool indicated that they felt they have not had any recourse in resolving issues and disagreements with private partners. For example, one individual who responded to our online tool stated that she was glad she moved off post because she now has legal recourse if the landlord does not meet maintenance requirements. The military department oversight agencies have found that the military departments have not clearly and systematically communicated their roles to residents, and resident confusion and a lack of awareness regarding the role of the military housing offices is an issue. In April 2019, the Air Force Inspector General reported that less than half of the residents interviewed used their military housing office to resolve complaints, and at some installations officials visited many residents did not know the military housing office had an oversight role. Similarly, in May 2019, the Army Inspector General reported to the Secretary of the Army that at 82 percent of Army installations with privatized housing, residents did not know how to escalate issues with either the private partner or the Army housing office. Additionally, the Army Inspector General reported that installation command teams and staff cited multiple circumstances where military housing offices and tenant advocacy roles and responsibilities were unclear. Further, military housing office officials with whom we spoke during our site visits acknowledged the gap in resident awareness regarding the existence and purpose of the military housing office. Officials also noted that at times residents were unaware of the difference between the military housing office and the private partner office due, in part, to their physical co-location and unclear building signage. For example, a military housing office official at Fort Bragg, North Carolina, told us the military housing office was the best kept secret on the installation. Moreover, residents that participated in our four focus groups at Fort Bragg expressed confusion in differentiating the Army military housing office officials from private partner representatives. Similarly, officials at the military housing office at Tinker Air Force Base, Oklahoma, told us that many residents were confused by their office’s role because the private partner office goes by the name “Tinker Housing Office.” Further, we observed that both private partner representatives and some military housing office officials are located in the same building, and signage does not distinctly indicate that the office houses both military officials and private partner representatives. In contrast, the military housing office at Camp Pendleton, California, is intentionally branded as the “Camp Pendleton Joint Housing Office” and signage indicates the office houses officials from both the Marine Corps and the installation’s private partners. See figure 5 for examples of the varying level of detail in military housing office signage. Some military housing office officials told us they have taken steps to improve resident awareness, such as increasing advertising of the military housing office’s role and contact information, using town hall meetings to inform residents of their roles and responsibilities, and rebranding their military housing offices to differentiate them from the private partners. For example, the Army housing office at Fort Sill, Oklahoma, changed its name from the “Residential Communities Initiative Housing Office” to the “Garrison Housing Office” to more clearly denote that the military housing office is not associated with the private partner. In addition, a Marine Corps housing office official provided us with a flyer, which is distributed to residents by the private partner, informing residents of housing office contact information and the service’s three-step dispute resolution process. See figure 6 for a copy of the flyer. According to DOD officials, the military departments generally decreased their staffing and oversight of daily privatized housing operations after the MHPI was enacted, which led to less ongoing resident interaction. For example, Army officials we spoke with in January 2019 told us they typically filled 80 percent of available military housing office positions across their installations. Additionally, officials stated that housing offices were generally staffed with two or three officials responsible for assisting servicemembers with housing needs both on the installation as well as in the local community. Further, the officials told us that the team at Fort Bragg, North Carolina, was decreased from about 15 to 3 positions. According to OSD officials, while housing offices should generally not require the number of personnel that were necessary prior to privatization, reductions following sequestration reduced housing staff below the level necessary to fully perform required privatized housing oversight as it was originally envisioned at the outset of the program. OSD has also recognized that the military departments’ communication with residents about their role as a resource for residents has not been clear or systematic. In February 2019, the Assistant Secretary of Defense for Sustainment testified before Congress that a way forward in addressing resident concerns would require focus in three key areas: communication, engagement, and responsiveness. In support of this, OSD and the military departments are collaborating with each of the private partners on several initiatives aimed at improving the residents’ experience with privatized housing and ensuring a consistent resident experience across installations. These initiatives include: establishing a tenant bill of rights that will clearly define tenants’ rights establishing a resident advocate position that is planned to provide advice, education, and support to the resident and advocate on the resident’s behalf in disputes with private partners; developing a common lease that provides a common framework and language in residential leases across all privatization projects; and developing a standardized formal dispute resolution process to ensure the prompt and fair resolution of disputes that arise between privatized housing landlords and residents. Despite the development of initiatives aimed at improving the resident’s experience with privatized housing and various ad hoc efforts to better brand and advertise the roles and responsibilities of some military housing offices, the military departments have not systematically or clearly communicated these efforts to residents, and military officials we met with acknowledged that there still appears to be a gap in residents’ awareness of the military housing office and its role in the dispute resolution process. Standards for Internal Control in the Federal Government state that management should externally communicate the necessary quality information to achieve the entity’s objectives. Management communicates this externally through reporting lines so that external parties can help the entity achieve its objectives and address related risks. Moving forward, having plans in place to clearly and systematically communicate the difference between the military housing office and the private partners— including the military departments’ roles, responsibilities, and military housing office locations and contact information—will better position the military departments to achieve the intended objectives of the initiatives they are currently developing with OSD. OSD, the military departments, and the private partners have identified and begun collaborating on a series of initiatives aimed at improving residents’ experience with privatized housing. According to an OSD official, a series of initiatives have been identified and are currently in various phases of development and implementation. Tri-service working groups, each chaired by a designated military department and including officials and legal counsel from each military department as well as private partner representatives, are leading efforts to develop and implement the initiatives. In addition, in the Fiscal Year 2020 NDAA, Congress established several requirements aimed at addressing military privatization housing reform. Several of the statutory requirements provide specific provisions that DOD will need to incorporate into its development and implementation of existing MHPI initiatives, as well as additional requirements aimed at improving the oversight of privatized housing. Table 2 outlines key initiatives aimed at improving privatized housing, as well as additional selected requirements mandated by the Fiscal Year 2020 NDAA. In addition to the provisions noted in table 2, the Fiscal Year 2020 NDAA included requirements for increased oversight of the physical condition of privatized housing. Specifically, the legislation required the following: The Secretary of Defense is to designate a Chief Housing Officer to oversee housing units, including the creation and standardization of policies and processes regarding housing units. The Secretary of Defense is required to establish a uniform code of basic standards for privatized military housing and plans to conduct inspections and assessment of the condition of privatized homes. The military departments are required to create a council on privatized military housing for the purposes of maintaining adequate oversight of the military housing program and serving as a mechanism to identify and resolve problems regarding privatized military housing. The head of the installation military housing office is responsible for conducting a physical inspection and approving the habitability of a vacant housing unit for the installation before the landlord managing the housing unit is authorized to offer the housing unit available for occupancy; conducting a physical inspection of the housing unit upon tenant move-out; and initiating contact with a tenant regarding the tenant’s satisfaction with his or her housing unit not later than 15 days after move-in, and again 60 days after move-in. Each installation is required to use the same satisfaction survey for tenants of military housing, including privatized military housing, and results are not to be shared with partners until reviewed by DOD. DOD and private partner representatives have cited several challenges that could affect their ability to implement initiatives aimed at improving MHPI. Specifically, key challenges include the timeliness with which they are able to implement initiatives, a lack of resources needed for implementation, and concerns that implementation could have unintended negative impacts on the financial viability of the privatized housing projects. Timeliness of implementation due to the need to collaborate with and obtain input and agreement from the large number of stakeholders involved in privatized housing. According to DOD officials and private partner representatives, many of the initiatives designed to improve privatized housing require not only agreement between DOD and the private housing partners, but also discussion with and, in some cases, approval by the project bond holders. Because DOD does not have the ability to unilaterally make changes to existing business agreements, this need for stakeholder agreement limits DOD’s control over the implementation timeline of any initiative that requires changes to a project’s business agreement—such as the implementation of a standardized dispute resolution process. Additionally, the private partners noted that the bond holders may be reluctant to agree to changes to the business agreements that could result in higher project costs. The need for more military department staff with targeted expertise. As noted earlier, the military departments had reduced their involvement in daily privatized military housing operations as part of the overall privatization effort. This included reducing staffing levels at the installations, and military housing office officials at over half of the installations we visited stated that reduced staffing levels had impacted their ability to carry out oversight duties, such as work order data analysis and housing inspections. Further, until recent issues surfaced over concerns of the quality of privatized housing, the military departments had distanced themselves from involvement in daily military housing operations. For example, the Army issued a memorandum in 2013, which has since been rescinded, stating that garrison commanders were not to authorize, direct, or permit Army representatives to initiate health and welfare inspections of privatized housing. Each of the military departments has plans to increase the military housing office staffing at each installation to allow for enhanced oversight. In particular, according to military department officials, these positions will focus on quality control and quality assurance of the maintenance of privatized homes. However, improvements to the condition of privatized housing may not be fully realized until DOD establishes a uniform code of basic standards for privatized military housing, as required by the Fiscal Year 2020 NDAA, and these new personnel are trained in these standards. The potential for unintended negative financial impacts on the projects that could outweigh the intended benefits of the initiatives. OSD officials and private partner representatives have expressed concern that some proposed initiatives could result in unintended financial consequences for the housing projects. In particular, private partner representatives noted costs from things such as legal fees associated with the development of a common lease and the various addendums that would be required and the unanticipated costs of hiring outside third party inspections. In particular, some of the private partners noted that the financial impact of unfunded requirements to projects that are already experiencing financial distress could result in even fewer funds available to reinvest in improvements to the current and future physical condition of the homes. Moreover, OSD officials told us they have concerns that some initiatives—such as increased frequency of change-of-occupancy inspections that may result in homes remaining vacant longer than planned and therefore not collecting rent—may unintentionally impact a project’s cash flow. Officials noted that some installations have large-scale housing turn over at the same time and inspections may not be able to be completed in the required time frames. For example, OSD officials said that at Fort Leavenworth, Kansas, the vast majority of homes generally turn over during a 2-week time period. Officials said that in a location like this, new oversight requirements may have a negative impact on residents’ move-in timelines, which could subsequently impact occupancy rates and project cash flow as a result of delays in rent payments. OSD officials also stated that residents’ ability to have their basic allowance housing payments segregated and held in escrow may present financial challenges to both the resident and the project. These officials noted that they did not yet know how the withholding of these payments would be implemented. According to OSD officials, as of January 2020, there are many questions surrounding the implementation of the Fiscal Year 2020 NDAA provisions. Officials told us that they have not yet assessed the impact of increased oversight on the financial viability of the MHPI projects, but stated that as they develop processes to implement each new statutory provision, the financial impact is something that needs to be considered. DOD’s Military Housing Privatization Initiative Performance Evaluation Report for fiscal year 2017 stated that the government’s interests are not always aligned with those of the private sector, and that oversight and engagement are required and expected in a public-private partnership over the long term to ensure success. We have previously reported that the military departments have not defined their risk tolerance levels for privatized housing relative to the program’s objective of providing quality housing that reflects community living. Specifically, we recognized that the Office of Management and Budget guidance on the preparation, submission, and execution of the federal budget suggests that public- private partnerships, such as privatized military housing projects, contain some elements of risk to the government. Standards for Internal Control in the Federal Government state that management should identify, analyze, and respond to risks related to achieving defined program objectives. While DOD is in the process of developing and implementing initiatives to improve privatized military housing, OSD and the military departments have not assessed the risk of the proposed initiatives on the financial viability of the privatized housing projects. According to an OSD official, the intention of privatization was to reduce the government’s role in the management of military housing and put more responsibility on the private partners. As described earlier in this report, the military departments have ramped up their oversight efforts in response to recent concerns about the condition of privatized housing by, for example, revising guidance and hiring additional staff. However, OSD has not assessed the impact of these activities on the financial viability of the MHPI projects. For example, OSD has not determined how increasing the frequency of housing office inspections and residents’ withholding of rent could impact the bottom line of some of its privatized projects. Without assessing risks to the financial viability of the MHPI projects associated with the implementation of these initiatives aimed at improving privatized housing, DOD’s efforts to improve the privatized housing program could be compromised. Further, DOD has a long-term interest in ensuring the financial health of the properties privatized under MHPI. As we have reported, typically the titles to the homes that were conveyed to the private partners and any improvements made to these homes during the duration of the ground leases will automatically revert to the military departments upon expiration or termination of the leases. DOD’s oversight of privatized housing is critical to ensure that residents are being provided with affordable, quality housing that generally reflects contemporary community living standards. In light of recent concerns about the effect of inadequate and poor quality housing on servicemembers and their families, the military departments have recently implemented steps to increase the oversight of the condition of privatized housing. However, OSD has not provided the military departments with specific objectives for this monitoring. The newly established Chief Housing Officer position, intended to standardize guidance and processes for the oversight of privatized housing, provides DOD with an opportunity to ensure that revised guidance provided to the military departments includes objectives for increased oversight. In addition to oversight of the condition of homes, DOD has taken initial steps to standardize performance incentive metrics across the military departments. However, unless efforts are made to ensure that the indicators driving these metrics accurately reflect private partners’ performance in maintaining the condition and quality of privatized homes, DOD’s ability to hold private partners accountable will remain limited. Further, while the military departments continue to increase the access to and use of work order data to monitor and track the condition of privatized housing, without consistent terminology and practices for work order data collection and processes for validating data collected from the private housing partners, the use of these data may not result in reliable findings. Finally, DOD has frequently reported high customer resident satisfaction rates as a key indicator of the success of the privatization initiative. However, the process used to collect and calculate the data used for determining these rates and limitations in its presentation to Congress raise questions about the reliability of DOD’s reports and their usefulness as an indicator of program success. By improving oversight guidance, mechanisms for measuring private partner performance, the reliability of housing data, and reporting on resident satisfaction, DOD can better ensure that MHPI is providing servicemembers with quality housing. Despite a decreased role in the daily management of privatized housing, the military departments still maintain responsibility for providing servicemembers with resources for seeking resolution to any issues identified in their privatized homes. However, without plans in place to communicate military housing office roles, responsibilities, and locations to residents of privatized housing, these individuals may not receive the full benefits of the assistance that the military housing offices provide. In light of the increased focus by DOD and Congress in ensuring that residents are aware of their rights and responsibilities, improved communication with residents about the military housing offices’ roles and responsibilities can help ensure that residents are utilizing the full range of resources available to them if they have issues with privatized housing. As OSD, the military departments, and the private partners take steps to improve the resident experience with privatized military housing and increase the department’s focus on the condition of privatized homes, ensuring that their efforts do not inadvertently harm the financial viability of these projects is key. Without assessing and mitigating the potential risk program improvements may have on the financial viability of the MHPI projects, DOD cannot ensure that these initiatives and the implementation of new statutory requirements will ultimately result in improvements to the condition of privatized housing. We are making a total of 12 recommendations—six to the Office of the Secretary of Defense, two to the Secretary of the Army, two to the Secretary of the Air Force, and two to the Secretary of the Navy: The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments, provide updated guidance for the oversight of privatized military housing, to include oversight objectives for each service to monitor the physical condition of privatized homes over the remaining duration of the ground leases. (Recommendation 1) The Secretary of the Army should take steps, in collaboration with the Army’s private housing partners, to review the indicators underlying the privatized housing project performance metrics to ensure they provide an accurate reflection of the condition and quality of the homes. (Recommendation 2) The Secretary of the Air Force should take steps, in collaboration with the Air Force’s private housing partners, to review the indicators underlying the privatized housing project performance metrics to ensure they provide an accurate reflection of the condition and quality of the homes. (Recommendation 3) The Secretary of the Navy should take steps, in collaboration with the Navy and Marine Corps’ private housing partners, to review the indicators underlying the privatized housing project performance metrics to ensure they provide an accurate reflection of the condition and quality of the homes. (Recommendation 4) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments and private housing partners, establish minimum data requirements and consistent terminology and practices for work order data collection for comparability across installations and projects and to track trends over time. (Recommendation 5) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment require the military departments to establish a process to validate data collected by the private housing partners to better ensure the reliability and validity of work order data and to allow for more effective use of these data for monitoring and tracking purposes. (Recommendation 6) The Secretary of Defense should ensure the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments, develop a process for collecting and calculating resident satisfaction data from the military departments to ensure that the data are compiled and calculated in a standardized and accurate way. (Recommendation 7) The Secretary of Defense should ensure the Assistant Secretary of Defense for Sustainment provides additional explanation of the data collected and reported in future reports to Congress, such as explaining the limitations of available survey data, how resident satisfaction was calculated, and reasons for any missing data, among other things. (Recommendation 8) The Secretary of the Army should develop and implement a plan to clearly and systematically communicate to residents the difference between the military housing office and the private partner. At a minimum, these plans should include the Army housing office’s roles, responsibilities, locations, and contact information and should ensure that all residents are aware that they can directly contact Army housing office officials. (Recommendation 9) The Secretary of the Air Force should develop and implement a plan to clearly and systematically communicate to residents the difference between the military housing office and the private partner. At a minimum, these plans should include the Air Force housing office’s roles, responsibilities, locations, and contact information and should ensure that all residents are aware that they can directly contact Air Force housing office officials. (Recommendation 10) The Secretary of the Navy should develop and implement a plan to clearly and systematically communicate to residents the difference between the military housing office and the private partner. At a minimum, these plans should include the Navy housing office’s roles, responsibilities, locations, and contact information and should ensure that all residents are aware that they can directly contact Navy housing office officials. (Recommendation 11) The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments, assess the risks of proposed initiatives aimed at improving the privatized military housing program on the financial viability of the projects. (Recommendation 12) We provided a draft of this report to DOD for review and comment. In written comments, reprinted in their entirety in appendix III, DOD concurred with 10 of our recommendations and partially concurred with 2, identifying actions it plans to take to address each of them. DOD also provided technical comments, which we incorporated as appropriate. DOD partially concurred with our recommendation that the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments and private housing partners, establish minimum data requirements and consistent terminology and practices for work order collection. The department noted that neither the Assistant Secretary of Defense for Sustainment nor the military departments could mandate changes to existing privatized housing project ground leases or legal agreements. DOD further noted that it cannot unilaterally make changes to the project ground leases and associated legal documents without concurrence from the private partners. However, the department noted that to the maximum extent practical, it would work to establish minimum data requirements and consistent terminology and practices for work order collection. DOD also partially concurred with our recommendation that the Under Secretary of Defense for Sustainment, in collaboration with the military departments, develops a process for collecting and calculating resident satisfaction data because there is no Under Secretary of Defense for Sustainment. Based on the department’s comments, we revised the addressee of this recommendation, directing action to the Assistant Secretary of Defense for Sustainment. However, the department noted that effective with the survey collection effort for Fiscal Year 2021, it would refine the process for collecting and calculating resident satisfaction data from the military departments to ensure that DOD compiles and calculates data in a standardized and accurate way. We are sending copies of this report to the appropriate congressional committees; Senator Catherine Cortez Masto; Senator Mark Warner; Representative Gus Bilirakis; the Secretary of Defense; and the Secretaries of the Departments of the Army, the Navy, and the Air Force. In addition, the report is available at no charge on our website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2775 or FieldE1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV. The Conference Report accompanying a bill for the Fiscal Year 2019 Department of Defense Appropriations Act included a provision for us to review ongoing issues within privatized military housing. This report examines the extent to which the Office of the Secretary of Defense (OSD) and the military departments (1) conduct oversight of privatized military housing for servicemembers and their families, (2) have communicated their roles and responsibilities to servicemembers and their families, and (3) have developed and implemented initiatives to improve privatized housing. We included all privatized housing projects in each military department. For each of our objectives, we reviewed OSD and military department policies and guidance for the implementation of the Military Housing Privatization Initiative (MHPI) program, including guidance on the authority, roles, and responsibilities for oversight and management of privatized housing. We evaluated the extent to which the evidence we collected aligned with OSD policy and stated goals for oversight and management of privatized housing, and whether the evidence adhered to the principles in Standards for Internal Control in the Federal Government. We conducted interviews with officials from the Office of the Assistant Secretary of Defense for Sustainment, Office of Facilities Management; the Office of the Deputy Assistant Secretary of the Army (Installations, Housing and Partnerships); the Army Installation Management Command; the Army Assistant Chief of Staff for Installation Management; the Assistant Secretary of the Air Force for Installations, Environment, and Energy; the Air Force Civil Engineer Center; the Commander, Navy Installations Command; the Commander, Naval Facilities Engineering Command; the Marine Corps Installation Command; and representatives from each of the 14 private partners that are currently responsible for privatized housing projects. We visited a non-generalizable sample of 10 installations selected to represent each of the military departments, six private partners—including the five largest who own the majority of privatized military housing—and geographic and climate diversity. The selected sites in our non- generalizable sample were three Army installations—Fort Bragg, North Carolina; Fort Huachuca, Arizona; and Fort Sill, Oklahoma; two Navy installations—Naval Station Norfolk, Virginia, and Naval Base San Diego, California; two Marine Corps installations—Marine Corps Base Camp Lejeune, North Carolina, and Marine Corps Base Camp Pendleton, California; and three Air Force installations—Davis-Monthan Air Force Base, Arizona; Langley Air Force Base, Virginia; and Tinker Air Force Base, Oklahoma. We reviewed the ground leases and other MHPI project documents for housing projects at each of these locations, and at each installation we met with officials from the installation commander’s office and conducted interviews with officials from both the installation military housing office and representatives from the private partners. To collect input from residents of privatized housing, we facilitated 15 focus groups with a self-selected group of current residents of privatized military housing. During the focus groups, a methodologist led participants through a structured questionnaire, which we pretested with 11 residents of privatized housing prior to the first focus group. To solicit participants for our focus groups, we requested that local military housing office officials email all current residents of privatized housing prior to our visit to inform them of our focus groups. Individuals interested in participating in our focus group sessions were instructed to contact us directly for further information. We had over 70 residents participate in our focus groups. In addition to the 15 focus groups, we conducted an additional five sessions in which fewer than three residents attended. We collected information from these residents, but we did not include their input in our focus group analysis. Comments from focus group participants are not generalizable to all residents of privatized military housing. We also developed and administered a publically available online tool that provided an opportunity for any resident of privatized military housing to voluntarily submit information on their experiences. Participants had the option to remain anonymous and make multiple submissions in order to provide us information on their experience at more than one installation. We developed our tool in conjunction with a survey methodologist to ensure it met our requirements for publically available anonymous data collection instruments, and conducted five pretests of the questions with residents of privatized housing. Our online tool was made available to the public from June 17, 2019, through August 31, 2019. We received a total of 658 responses. In analyzing information provided through the online tool, we took steps to identify responses that did not meet our criteria, including removing 13 responses for reasons such as responses with duplicative usernames or Internet Protocol (IP) addresses that described the same experience or had been started but not fully completed, responses from DOD officials that informed us they had provided responses to test our tool, and responses from residents living on installations outside of the United States. In reporting results from our online tool, we used the following qualifiers in presenting our results— most (to indicate 80 percent or higher); majority (to indicate 51-79 percent); and some (to indicate less than 50 percent). Findings from our focus groups and online tool are not generalizable to all privatized military housing residents. To determine the extent to which DOD conducts oversight of privatized military housing for servicemembers and their families, we conducted the following additional data analysis. Through the steps described in the following bullets, we determined these data to be reliable for the purposes of our findings: To determine the extent to which performance incentive fee metrics assessed the condition of privatized housing, we collected information on the structure of the incentive fees from private partners for 74 privatized housing projects and received confirmation that there are 5 projects that do not have incentive fee plans as part of their business agreements. We reviewed all of the incentive fee plans and identified commonly used metrics and indicators. We met with officials from the military housing offices, the military departments, and private partner representatives to discuss the administration and measurement of the incentive fee structures. To gain an understanding of how performance incentive fees are used, we reviewed documents and guidance from OSD and the military departments that explains the processes for developing and awarding performance incentive metrics and fees. In addition, we obtained information from residents through focus groups and our online tool and spoke with military housing office officials to obtain anecdotal information regarding the extent to which the metrics are adequately measuring the condition of the housing. To assess the extent to which private partner work order data could be used to monitor and track the condition of privatized homes, we collected and reviewed private partner work order data from October 2016 through April 2019 from each of the 79 MHPI projects and discussed these data with the private partners and military department officials. Given that we requested the work order data from the private partners in April and May 2019, we selected the October 2016 through April 2019 time frame to include complete data for fiscal years 2017 and 2018 and the most comprehensive data available at the time for fiscal year 2019. Prior to requesting these data, we contacted representatives from each of the 14 private partner companies to discuss our forthcoming data request and to better understand each company’s data system and potential limitations for using the data. Subsequently, we requested that each partner provide us with data for all work orders across all data elements for each installation under their management. We received data on over 8 million work orders among the 14 private partners. We performed manual testing on initial data files received by each partner to identify issues that would impact the validity and reliability of using these data for ongoing monitoring and tracking of the condition of privatized housing units. In doing so, we identified instances of anomalies in work order data from each of the 14 partners. For 12 of the 14 partners, we found at least one of the following anomalies in the initial work order data files received for the time period requested: (1) duplicate work orders; (2) work orders with completion dates prior to the dates that a resident had submitted the work order; and (3) work orders still listed as in-progress for more than 18 months. We reviewed work order data from at least one installation for each private partner to check for instances of these anomalies. We also held follow-up discussions with 10 of the 14 private partners to discuss anomalies found in the data and potential factors contributing to the presence of these anomalies. In addition to the initial data collected on all of the work orders, we requested a second data run of work orders over the same time period—October 1, 2016, through April 30, 2019—for service requests related to lead- based paint, mold, and pest/rodent/vermin infestation. As part of this request, we asked that partners provide the criteria used for querying the data they provided us. We reviewed these data to determine how requests for data on specific hazards were getting sorted into the work order tracking systems by category and priority among the various partners. To identify differences in terminology and business practices used by the private partners in their work order tracking systems, we requested and reviewed private partner documentation of data definitions and protocols for managing work order data. In addition, we conducted interviews with military department officials to discuss ongoing efforts by the military departments to collect and analyze work order data. To evaluate resident satisfaction data reported in OSD’s reports to Congress on privatized housing, we reviewed the processes for collecting, calculating, and reporting these data for the three most recently issued reports for fiscal years 2015, 2016, and 2017. We reviewed the instructions OSD provided to the military departments outlining how the military departments are to submit resident satisfaction data to OSD. We also reviewed the question the military departments asked on their annual surveys to gauge resident satisfaction. We then requested the survey data the military departments submitted to OSD to be included in the annual report to Congress for fiscal years 2015, 2016, and 2017. We performed data quality checks and identified inaccuracies on a random sample of data reported by OSD. We reviewed how military departments calculated overall resident satisfaction for each privatized housing project. Further, we discussed these data with OSD and the military departments to assess the validity and reliability of using these data for identifying overall tenant satisfaction with the condition of privatized housing. To determine the extent to which the military departments have communicated their respective military housing office roles and responsibilities to residents, we reviewed military department policies and guidance related to their roles and responsibilities for working with residents of privatized housing. During our site visits to 10 installations, we interviewed military department housing office officials and private partner representatives to discuss their specific roles and responsibilities. We asked questions soliciting information about residents understanding of the roles and responsibilities of the military housing office and the dispute resolution process during our 15 focus groups. We also solicited resident feedback in our online tools regarding residents’ experience reporting maintenance issues and working with military housing offices and private partners to get maintenance issues resolved. To determine the extent to which DOD and private partners have developed and implemented initiatives to improve privatized housing, we interviewed OSD and military department officials to discuss ongoing initiatives developed over the course of our audit work aimed at improving MHPI and reviewed relevant guidance. We met with private partner representatives to discuss their involvement in developing these initiatives, as well as to gain an understanding of any challenges or concerns that may impact the implementation of these initiatives. Following the passage of the National Defense Authorization Act for Fiscal Year 2020, we reviewed provisions of the statute designed to improve the condition of privatized housing and evaluated the extent to which these provisions would impact ongoing or planned DOD initiatives or provide new oversight roles and responsibilities for OSD and the military departments. We discussed these provisions with OSD officials and private partner representatives to understand how, if at all, their implementation may impact the privatized housing projects, as well as any potential barriers to implementation in the current legal construct of the program. We conducted this performance audit from November 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In table 3, we provide the complete listing of the Department of Defense’s 79 privatized military housing projects, as of September 30, 2017. This list reflects information that the Office of the Assistant Secretary of Defense for Sustainment provided in its annual report to Congress for the time period of October 1, 2016, through September 30, 2017. The report was provided to Congress in May 2019. The projects can consist of one or multiple installations. Elizabeth A. Field, Director, (202) 512-2775 or FieldE1@gao.gov. In addition to the contact above, the following are key contributors to this report: Kristy Williams (Assistant Director), Tida Barakat Reveley (Analyst in Charge), Austin Barvin, Ronnie Bergman, Vincent Buquicchio, William Carpluk, Juliee Conde-Medina, Mae Jones, Jordan Tibbetts, Kelly Rubin, Monica Savoy, and John Van Schaik. Military Housing: Preliminary Recommendations to Strengthen DOD's Oversight and Monitoring of Privatized Housing. GAO-20-471T. Washington, D.C.: March 3, 2020. Military Housing Privatization: Preliminary Observations on DOD’s Oversight of the Condition of Privatized Military Housing. GAO-20-280T. Washington, D.C: December 3, 2019. Defense Infrastructure: Additional Actions Could Enhance DOD’s Efforts to Identify, Evaluate, and Preserve Historic Properties. GAO-19-335. Washington, D.C.: June 19, 2019. Military Housing Privatization: DOD Should Take Steps to Improve Monitoring, Reporting, and Risk Assessment. GAO-18-218. Washington, D.C.: March 13, 2018. Defense Infrastructure: Army Has a Process to Manage Litigation Costs for the Military Housing Privatization Initiative. GAO-14-327. Washington, D.C.: April 3, 2014. Military Housing: Information on the Privatization of Unaccompanied Personnel Housing. GAO-14-313. Washington, D.C.: March 18, 2014. Military Housing: Enhancements Needed to Housing Allowance Process and Information Sharing among Services. GAO-11-462. Washington, D.C.: May 16, 2011. Military Housing Privatization: DOD Faces New Challenges Due to Significant Growth at Some Installations and Recent Turmoil in the Financial Markets. GAO-09-352. Washington, D.C.: May 15, 2009. Military Housing: Management Issues Require Attention as the Privatization Program Matures. GAO-06-438. Washington, D.C.: April 28, 2006. Military Housing: Further Improvement Needed in Requirements Determination and Program Review. GAO-04-556. Washington, D.C.: May 19, 2004. Military Housing: Better Reporting Needed on the Status of the Privatization Program and the Costs of Its Consultants. GAO-04-111. Washington, D.C.: October 9, 2003. Military Housing: Opportunities That Should Be Explored to Improve Housing and Reduce Costs for Unmarried Junior Servicemembers. GAO-03-602. Washington, D.C.: June 10, 2003. Military Housing: Management Improvements Needed as the Pace of Privatization Quickens. GAO-02-624. Washington, D.C.: June 21, 2002. Military Housing: DOD Needs to Address Long-Standing Requirements Determination Problems. GAO-01-889. Washington, D.C.: August 3, 2001. Military Housing: Continued Concerns in Implementing the Privatization Initiative. GAO/NSIAD-00-71. Washington, D.C.: March 30, 2000. Military Housing: Privatization Off to a Slow Start and Continued Management Attention Needed. GAO/NSIAD-98-178. Washington, D.C.: July 17, 1998.", "summary": "Congress enacted the Military Housing Privatization Initiative in 1996 to improve the quality of housing for servicemembers. DOD is responsible for general oversight of privatized housing projects. However, private-sector developers are responsible for the construction, renovation, maintenance, and repair of about 99 percent of military housing in the United States. Recent reports of hazards, such as mold and pest infestation, have raised questions about DOD's oversight of privatized military housing. Conference Report 115-952 included a provision for GAO to review ongoing issues within privatized housing. This report assesses, among other things, the extent to which OSD and the military departments (1) conduct oversight of privatized housing and (2) have developed and implemented initiatives to improve privatized housing. GAO reviewed policies and guidance; visited a non-generalizable sample of 10 installations; analyzed work order data; and interviewed DOD officials and private partner representatives. The Office of the Secretary of Defense (OSD) and the military departments conduct a range of oversight activities, but some of these activities have been more extensive than others. Specifically, GAO found that: DOD provides reports to Congress on the status of privatized housing, but some data in these reports are unreliable, leading to misleading results. DOD provides periodic reports to Congress on the status of privatized housing, but reported results on resident satisfaction are unreliable due to variances in the data provided to OSD by the military departments and in how OSD has calculated and reported these data. OSD has made progress in developing and implementing a series of initiatives aimed at improving privatized housing. In addition, Congress established several requirements addressing privatization housing reform. However, DOD officials and private partner representatives have identified challenges that could affect implementation of these various initiatives. These include concerns that implementation could have unintended negative impacts on the financial viability of the privatized housing projects. However, DOD has not assessed the risk of the initiatives on project finances. GAO is making 12 recommendations, including that DOD take steps to improve housing condition oversight, performance indicators, maintenance data, and resident satisfaction reporting as well as to assess the risk of the initiatives on project finances. DOD generally concurred with the recommendations and identified actions it plans to take to implement them.", "document_type": "gao"}
{"report": "To support its mission, DOD uses contracts to procure many different types of supplies (such as ships, planes, and munitions) and services (such as management, maintenance, and technical services). The federal acquisition process generally includes three phases: the pre-award phase, which includes acquisition planning and activities such as conducting market research and defining contract terms and conditions prior to soliciting proposals; the award phase, which includes activities such as soliciting offers from prospective contractors, evaluating prospective contractors’ proposals and qualifications, and awarding the contract; and the contract performance phase, which includes monitoring contract performance. Within these phases, contracting officials complete certain activities as provided by applicable federal statutes and the Federal Acquisition Regulation (FAR). These activities differ somewhat based on the unique circumstances of each contract, including, for example, whether a contract is awarded competitively through full and open competition or non-competitively through other than full and open competition, and whether negotiated procedures are used. (See fig.1.) Before awarding a contract in excess of the simplified acquisition threshold (generally $150,000 at the time of our review), the FAR requires contracting officials to review information in the Federal Awardee Performance and Integrity Information System (FAPIIS), which can include descriptions of a prospective contractor’s past safety and health violations. Furthermore, for competitively awarded acquisitions using negotiated procedures and expected to exceed the simplified acquisition threshold, agencies generally must evaluate prospective contractors’ past performance. Contracting officials enter and view performance assessments in the Contractor Performance Assessment Reporting System (CPARS). In evaluating past performance, agencies may review a contractor’s past performance assessments, which can contain information about prior safety incidents and may be used to support award decisions. While the FAR prescribes policies and requirements that apply to executive agencies, there can be wide variation concerning the acquisition practices at individual agencies. For example, USACE often can take advantage of a robust competitive market and frequently uses competitively awarded fixed-price contracts. NAVSEA, by contrast, operates within an industrial base that has far fewer participants that often are uniquely qualified to produce specific classes of ships. As a consequence, many of NAVSEA’s contracts are negotiated on a sole- source or limited competition basis. Under the Occupational Safety and Health Act of 1970 (OSH Act), OSHA sets and directly enforces occupational safety and health standards for the private sector in about half the states. The remaining states have chosen to set and enforce their own occupational safety and health standards for these employers under a state plan approved by OSHA. State standards and their enforcement must be “at least as effective” in providing safe and healthful employment as the federal standards. Most private sector employers, including federal contractors, are covered by the OSH Act and must comply with any applicable state or federal occupational safety and health standards. In addition to the OSH Act, several other federal laws require federal contractors, depending on the type and amount of the contract, to comply with occupational safety and health standards. OSHA and the states have approximately 2,100 compliance officers responsible for enforcing health and safety standards at more than 8 million worksites across the nation, which employ approximately 130 million workers. According to data provided by OSHA officials, in fiscal year 2017, OSHA and the states conducted about 76,000 inspections. A little less than half of these inspections were in the construction industry (about 34,000), and about one-fifth were in the manufacturing industry (about 14,000). OSHA and state occupational safety and health agencies conduct both programmed and unprogrammed inspections. Programmed inspections— which represented about 44 percent of all federal OSHA inspections in fiscal year 2017—are planned based on workplace injury incidence rates, previous citation history, or random selection. Programmed inspections include those conducted under OSHA’s emphasis programs, which focus on a particular safety or health hazard or a specific industry. OSHA’s nine current national emphasis programs include one on shipbreaking, which covers some companies with DOD contracts. In addition, OSHA has regional and local emphasis programs. Unprogrammed inspections— which represented the other 56 percent of all federal OSHA inspections in fiscal year 2017—are unplanned and are conducted in response to reports of imminent danger, fatalities, severe injuries, worker complaints, referrals from other government agencies, and catastrophic events that cause worker deaths and hospitalizations. Before beginning an inspection, OSHA or state compliance officers generally hold a brief opening conference to inform employer and employee representatives of the purpose of the inspection and their rights during the inspection, and provide a copy of the complaint, if applicable. After completing an inspection, if OSHA or state compliance officers determine that the employer has violated any safety or health standards, they may issue a citation, including a deadline for correcting the hazards, and related financial penalties (see fig. 2). If OSHA issues a citation, it is required to do so within 6 months of the occurrence of a violation. After receiving a citation, the employer may request an informal conference with OSHA officials to present evidence or views that they believe would support an adjustment to the citation or penalty, but an informal conference is not required. The employer may also contest the citation. Employers are required to certify that the hazards have been corrected by the deadline and provide supporting documentation. If they do not, OSHA may conduct a follow up inspection, and may issue additional citations and penalties if the hazards were not corrected. When an employer is inspected and OSHA finds violations, various factors might affect the number of violations identified. For example, an inspection with a narrow focus may identify fewer violations than a full inspection of the same worksite. OSHA officials said that construction inspections are often focused on a particular issue, such as protecting workers from falls or securing a trench, and thus may not be as comprehensive as a full inspection of a general industry facility. In addition, the number of violations identified during an inspection could be affected by factors such as company size, industry, and the presence of other safety oversight efforts. For example, OSHA officials said that in the construction industry they routinely cite both a general contractor and a subcontractor for the same violation, but do so to a lesser extent in other industries. Officials also noted that on USACE construction sites, both USACE and contractor representatives conduct safety inspections, which enhances employer compliance with both OSHA standards and the USACE Safety and Health Requirements Manual. DOL has authority under the Contract Work Hours and Safety Standards Act to debar federal contractors in the construction industry from receiving federal contracts if they have committed “repeatedly willful or grossly negligent” violations of OSHA safety and health standards. However, as of October 2018, officials said that DOL had not debarred a construction contractor for this reason in the last 10 years. According to officials, DOL does not have debarment authority for violations of safety and health standards in industries other than construction, although it has debarment authority for other types of labor law violations. Of the 192 companies we selected with DOD manufacturing or construction contracts in fiscal year 2017, we found that a little more than half (106) were inspected by OSHA or state occupational safety and health agencies from fiscal years 2013 to 2017. Of the companies that were inspected, 59 had construction contracts, and 47 had manufacturing contracts in fiscal year 2017. During this 5-year time period, OSHA or state agencies conducted 609 inspections of these 106 companies. Most of these inspections (about 81 percent) were conducted by OSHA. The percentages of programmed and unprogrammed inspections of our selected companies from fiscal years 2013 to 2017 were similar to these percentages for all federal OSHA inspections in fiscal year 2017. (See fig.3.) OSHA’s enforcement policy is designed to focus OSHA’s inspection resources on the most hazardous workplaces. Officials told us that employers, including DOD contractors, may not be inspected if they do not meet OSHA’s criteria for programmed inspections and do not experience a safety or health incident that would lead to an unprogrammed inspection. In addition, officials said employers that participate in OSHA’s Voluntary Protection Programs (VPP) must have high-quality safety and health programs, are exempt from regular programmed inspections, and are only inspected if OSHA is notified of a safety or health incident. According to OSHA officials, of the 86 selected companies that were not inspected by OSHA or state agencies from fiscal years 2013 to 2017, one currently participates in the VPP. Our analysis found that of the 106 selected companies that were inspected during this time period, 83 were cited for at least one safety or health violation of any type, and of those, 52 were cited for serious violations (when there was a substantial probability that death or serious physical harm could result, and the employer knew, or could have known with the exercise of reasonable diligence, of the hazard). Three companies were cited for at least one repeated violation. (See fig. 4.) However, we were unable to determine from the available data whether these safety and health violations occurred during work on a DOD contract because OSHA inspection data do not include that information. The 83 selected companies that were cited for workplace safety or health violations from fiscal years 2013 to 2017 had a total of 405 violations, including 195 serious violations, 7 repeated violations, and 203 violations of other types. These companies were assessed financial penalties totaling about $1.2 million over that time period, including about $742,000 in penalties for serious violations. In fiscal year 2017, the 83 companies previously cited for violations of any type had DOD contracts totaling about $113 billion, and the 52 companies previously cited for serious violations had DOD contracts of $46 billion (as measured by federal obligations). (See table 1.) Furthermore, for some of the selected companies cited for serious violations, the related OSHA inspection data described worker injuries or deaths. As previously noted, 52 of the selected companies were cited for a total of 195 serious violations from fiscal years 2013 to 2017. For some, but not all, of these serious violations, the related inspection data described accidents in which 7 workers died, 20 were hospitalized for severe injuries—including fractures, chemical burns, other burns, and amputations—and 4 had severe injuries that did not require hospitalization. According to the inspection data, the accidents in which 7 workers died included the following: a hydrogen blast in a melting chamber resulted in one worker being pinned under a 20,000 pound lid, another receiving second degree burns, and a third being killed; a barge capsized after a crane tilted over, and one worker drowned; a worker fell 98 feet from an elevator and was killed; a worker sustained a fatal electric shock when replacing jumper wires on a high voltage transmission corner tower, and another worker was injured; an autoclave exploded, striking and killing a worker with extreme a vessel became unmoored due to high winds and struck a pier which then collapsed, pulling two workers underwater, one of whom died. While we could identify some selected companies with DOD contracts in fiscal year 2017 that were previously cited for safety or health violations, the incidence of these violations among all inspected companies with DOD contracts is unknown because data limitations prevent comprehensive matching of federal contracting data with OSHA inspection data. Specifically, the corporate identification numbers used in the federal contracting databases—the Employer Identification Number/Taxpayer Identification Number (EIN/TIN) and the Data Universal Numbering System (DUNS) number—are not well-populated in OSHA’s database because OSHA has not designated them as required fields. OSHA officials are required to enter certain types of data in OSHA’s inspection database—such as the employer’s name and address, the type of inspection, and any violations that were identified during the inspection—and have the option to enter the employer’s EIN/TIN and DUNS number. However, at the time of our review, for manufacturing and construction inspections initiated from fiscal years 2013 to 2017, the EIN/TIN of the inspected company was entered in OSHA’s inspection data for about one-third of all inspections, and the DUNS number was entered for about 8 percent of all inspections (see fig. 5). OSHA has acknowledged that it is difficult to match records across different databases without corporate identification numbers. In addition, OSHA’s website with information about safety and health violations cannot currently be searched by a company’s EIN/TIN or DUNS number. OSHA makes information about violations publicly available on its website, which can be searched by company name and industry code, among other fields. However, when searching OSHA’s website by company name, interested parties may experience challenges obtaining relevant information because company names differ across databases. When we searched the OSHA website by company name as part of selecting USACE and NAVSEA contracts for review, we were unable to determine whether 18 of the 66 company names we searched had been inspected. For example, when we searched the OSHA website using the first word in one company’s name, the search results included 34 inspections, but none of the company names in the search results exactly matched the company name in the federal contracting data. OSHA officials said the EIN/TIN and DUNS numbers are not required fields because employers or their on-site representatives do not always have these numbers. Officials told us that smaller companies, such as small construction companies, are less likely to have these numbers than larger companies. When companies do have corporate identification numbers, officials said that the employer’s on-site representative who interacts with the OSHA compliance officer—such as a foreman—might not know these numbers, and OSHA officials may not have the opportunity to meet with other employer representatives who would be more likely to know these numbers. However, if an employer requests an informal conference with OSHA officials after being cited for a violation, the conference provides an opportunity for OSHA officials to obtain the employer’s corporate identification number from knowledgeable representatives. In addition, OSHA officials said requiring a corporate identification number in OSHA’s inspection database could prevent closing an inspection record and issuing any related citations if they were unable to obtain this number within the required six-month timeframe. Officials added that delays in issuing citations could also lead to delays in addressing workplace hazards, because employers are not required to begin addressing these hazards until they receive a citation. However, OSHA officials noted that if an employer’s EIN/TIN or DUNS number is not available during an inspection, the number can be added to the inspection database at a later time. Collecting corporate identification numbers as part of inspections could benefit both OSHA and users of OSHA’s website. OSHA officials said that the EIN is useful for collecting financial penalties from companies that have been cited for violations. In addition, OSHA officials told us that requiring the EIN/TIN or DUNS number in OSHA’s inspection database would make it easier to search for companies in OSHA’s online inspection data. According to federal internal control standards, management should externally communicate the necessary quality information to achieve the entity’s objectives. Quality information is appropriate, current, complete, accurate, accessible, and provided on a timely basis. Without exploring the feasibility of requiring a corporate identification number in OSHA’s inspection database and enabling OSHA’s website to be searched by that number, contracting officials and other interested parties are likely to experience challenges obtaining accurate information about companies’ safety and health violations. Officials at DOD have multiple opportunities to address contractor safety throughout the acquisition process. For example, during the award phase, officials can consider safety information when they evaluate contractors’ past performance for contracts awarded competitively using negotiated procedures. However, not all contracting officials are aware that relevant contractor safety information is available on the OSHA website. During the contract performance phase, USACE and NAVSEA both take additional steps related to contractor safety and health, including accident prevention and accident reporting. Only USACE, however, has a practice of requiring contracting officials to assess contractor safety performance on construction contracts at the completion of the contract. As a consequence, safety performance information for other contracts across DOD may not be readily accessible to officials when awarding new contracts. The FAR does not specifically require contracting officials to consider information about prospective contractors’ records of safety performance before awarding a contract. Furthermore, DOD, Army, Navy, USACE, and NAVSEA policy and guidance do not specifically direct contracting officials to consider information about prospective contractors’ safety records before awarding contracts, according to officials. However, contracting officials have several opportunities to consider contractor safety and health records and other safety information during the pre- award and award phases of the contracting process. Developing requirements and drafting the solicitation. As part of acquisition planning, contracting and program officials develop requirements. In addition, when drafting a solicitation, the FAR or agency guidance may prescribe the use of certain clauses. For example, the FAR requires that fixed-price construction contracts above the simplified acquisition threshold include a provision related to workplace safety. Specifically, these contracts must include an Accident Prevention clause that requires the contractor to provide appropriate safety barricades, signs, and signals, and comply with OSHA safety and health standards, among other requirements. In addition, for DOD construction fixed-price contracts above the simplified acquisition threshold, this Accident Prevention clause requires contractors to comply with the USACE Safety and Health Requirements Manual. USACE contracting officials also told us that if contracts include work associated with asbestos abatement, lead abatement, or hazardous waste remediation, clauses specific to these areas are also included in the solicitation and resulting contract. NAVSEA also uses clauses as applicable to the specific work performed, for shipbuilding procurements or ship repair, in its contracts, and NAVSEA stated that many of these clauses are related to safety and environmental issues. Further, program officials can include specific requirements for unique or high-risk activities. For example, one of our selected NAVSEA contracts specified that the contractor must ensure that all required safety and emergency devices, such as emergency escape breathing devices, were onboard the ship before the contractor conducted sea trials. Soliciting and evaluating offers from prospective contractors. For contracts awarded competitively using negotiated procedures, contracting officials are required to identify the factors on which they will evaluate prospective contractors’ proposals and their relative weights. Contracting officials can designate safety in the solicitation as among the criteria that they will use to evaluate proposals and require prospective contractors to submit related information. For example, solicitations for two of our selected contracts included aspects of safety in the evaluation of certain factors. Safety also may be considered during the evaluation of contractor past performance. For acquisitions following negotiated procedures that are expected to exceed the simplified acquisition threshold, the FAR generally requires an evaluation of prospective contractors’ past performance, which can include compliance with safety requirements on past contracts. The evaluation must include past performance as an evaluation factor unless the contracting officials document the reason past performance is not an appropriate evaluation factor for the acquisition. In evaluating past performance, the contractor’s performance assessments in CPARS may be reviewed and used by contracting officials to support future award decisions. For example, for one NAVSEA contract, the assessments identified instances when the contractor’s safety program failed to comply with NAVSEA’s safety standards. In noncompetitive acquisitions following negotiated procedures, there is no requirement that there be evaluation criteria that include past performance. In these situations, opportunities for considering safety issues may be limited to the responsibility determination. Determining that contractors meet responsibility standards. Prior to contract award, contracting officials must determine that prospective contractors are “responsible,” which is also known as the responsibility determination. The responsibility determination has several required elements, some of which may include consideration of workplace safety and health. For example, before awarding a contract, contracting officials must: determine that prospective contractors have the necessary organization, experience, accounting and operational controls, and technical skills, or the ability to obtain them, which may include assessing whether contractors have applicable safety programs; and determine that prospective contractors have a satisfactory performance record which for contracts that will be in excess of the simplified acquisition threshold includes reviewing and considering prospective contractors’ performance and integrity information in FAPIIS, which may include information about proceedings related to safety and health violations. For one of our selected contracts, the FAPIIS search result in the contract file described OSHA safety and health violations. While they were not required to do so, none of the responsibility determinations for our six selected contracts contained information about workplace safety. If contracting officials considered safety when making this determination, we did not locate it in the contract files. The information available to contracting officials in the federal contracting databases about contractors’ past performance varies by DOD component. Specifically, USACE has a practice of requiring officials to assess and rate contractors’ performance on construction contracts with respect to safety, among other required factors, in CPARS at the completion of the contract. USACE contracting officials enter a safety performance rating—exceptional, very good, satisfactory, marginal, or unsatisfactory—and provide a supporting written narrative in a specific tab in CPARS. As a result, information on safety performance is summarized in a single location within CPARS, and thus, readily accessible to federal contracting officials, including those at DOD, when a previous USACE construction contractor is considered for future contract awards. For all other DOD contracts, according to officials we interviewed, information on contractors’ safety performance may be included in various places throughout CPARS, but is not required to be summarized as a separate rating in a single location. The contract file documentation we reviewed illustrated these differences. For two of our selected USACE contracts, we found that this safety performance rating was available to officials for their consideration when awarding the contract. Based on our review of the contract file for one of these contracts, source selection officials identified less than satisfactory comments related to the safety performance rating in CPARS. As a result, they considered the rating, as well as the actions explained by the contractor to mitigate the safety issues. These officials determined that the corrective actions were sufficient, according to the documentation we reviewed. In contrast, for one of our selected NAVSEA contracts, we found that the past performance assessments in CPARS contained no information about workplace safety or health—either satisfactory or unsatisfactory. The past performance for the remaining NAVSEA contract file for which we obtained a CPARS report contained information on workplace safety and health—for example, the assessments noted corrective action requests were submitted for safety incidents. According to federal internal control standards, management should use quality information—information that is complete and accessible—to achieve its objectives. Without a safety performance rating for contractors in industries with relatively high rates of occupational injuries, such as manufacturing or ship building and repairing, contracting officials may lack complete, readily accessible information on prospective contractors’ workplace safety performance. As a result, DOD may miss opportunities to address safety and health concerns when awarding contracts in these high-risk industries—for example, by considering whether and how prospective contractors resolved or mitigated violations or safety issues on prior contracts. In addition, DOD contracting officials may not be aware that the OSHA website is a resource for additional information about contractors’ workplace safety and health records. Since CPARS only includes past performance assessments for federal government contracts, contracting officials may not know about OSHA violations committed by companies during work that took place outside of these contracts, or when the company was not a federal contractor. DOD officials told us that they expect contracting officials to use their discretion in evaluating safety performance; however, DOD has not advised its contracting officials that the OSHA website is a resource for additional information on workplace safety records. For one of our selected contracts, the contracting official told us he was not aware of a past OSHA violation when determining contractor responsibility. According to the contracting official, OSHA issued the citation for non-Navy work performed at the contractor’s commercial shipyard. Several contracting officials told us that they would likely only consider violations they deemed relevant, for example, those that occurred at the facility where the contract will be performed. However, without knowing that a past violation occurred, the official we interviewed for our selected contract may not have had the opportunity to consider all of the available information when evaluating the contractor or addressing potential safety issues. Moreover, contracting officials for the six USACE and NAVSEA contracts we reviewed said they have not sought information about contractor safety and health violations from the OSHA website, and several were unaware that the website contained information on violations. Without being made aware that the OSHA website is a resource for additional information, contracting officials may not have the opportunity to utilize all of the available information about prospective contractors’ safety history. As a result, DOD contracting officials may miss opportunities to consider safety and health concerns when they are awarding new contracts. Officials from our two selected components—USACE and NAVSEA—also identified various actions they may take during the contract performance phase related to the workplace safety of their contractors. For example, according to officials, during the contract performance phase, USACE and NAVSEA oversee contractors’ compliance with contract requirements related to workplace safety and health. The steps they take may include ensuring that contractors submit accident prevention plans, when required, and conducting safety inspections, among other actions. Monitoring for OSHA violations. As mentioned above, the FAR requires a clause regarding compliance with safety and health standards to be included in certain federal construction contracts. USACE officials told us that this is monitored as a reportable item while work is being conducted, and that if violations occur during the performance of the contract, the contracting official is to enter information about the violations into CPARS. An Army official also told us that the Army recently implemented a system to track OSHA violations, but that it did not yet contain any data. Accident prevention. The FAR requires compliance with the USACE Safety and Health Requirements Manual for certain DOD construction contracts. According to the manual and USACE officials, USACE does not allow construction to begin until officials have reviewed and accepted the contractor’s accident prevention plan, including changes if necessary. For example, the contract documentation we reviewed for a dredging contract included a memorandum outlining changes that were to be addressed in the accident prevention plan (for example, outlining credentials for the safety officer on the site). According to the manual, USACE requires contractors’ submitted accident prevention plans to be job-specific and include work to be performed by subcontractors. NAVSEA may require contractors to submit an occupational health and safety plan for ship repair work. For example, one of the files we reviewed for a maintenance, modernization, and repair contract for a certain class of ships included a safety plan—a required deliverable under the contract—covering topics such as fall protection, evacuation procedures, and accident notification. Inspections. The USACE manual requires daily safety inspections of contractors’ worksites by both contractor and USACE personnel, and officials told us that USACE procedures require these inspections to be entered in USACE’s Resident Management System. The manual also requires the accident prevention plan or the USACE project safety and occupational health plan to provide for “frequent” safety inspections of the work sites, material, and equipment to ensure compliance with the plan and the USACE manual. For one of the USACE contracts we selected for review, CPARS documentation provided by USACE officials indicated that USACE staff noted repeated issues with safety requirements, including exposed live electrical wiring, lack of adequate lighting, and improper use of extension cords. This CPARS example indicates that the contractor worked to increase safety compliance. Finally, USACE’s manual states that when an employee is deemed to be in imminent danger, contractor or USACE officials must immediately stop the unsafe work being performed. For NAVSEA, officials told us that safety requirements and oversight responsibilities will vary depending on the type of work involved. For new construction, the Navy Supervisor of Shipbuilding, Conversion, and Repair (SUPSHIP) oversees safety. For repair and maintenance, the Regional Maintenance Centers are charged with safety oversight, among other administrative responsibilities. At both organizations, if problems are found, personnel issue corrective action requests. For example, a regional maintenance center staff member issued a corrective action request because the contractor failed to monitor the use of personal protective equipment and a contractor employee fell through a deck opening. As previously noted, the CPARS assessments for one of our selected contracts specifically noted that safety corrective action requests had been issued to that contractor. NAVSEA officials told us that quality assurance staff also have regular meetings with the contractors and monitor workplace safety. Accident reporting. USACE policy is to investigate and report USACE accidents in order to prevent recurrences and to comply with OSHA, DOD, Army, and other requirements. USACE regulation requires contracting officials to inform contractors of their responsibilities for accident reporting and investigation, and ensure all accidents that occur within their area of responsibility are investigated and reported. USACE also collects information about accidents at contractors’ worksites, and disseminates summaries of incidents on a regular basis. For example, one summary described a fall by a contractor employee resulting in stitches and a broken nose. The summary reminds USACE personnel of the importance of protective equipment to prevent this type of incident. USACE officials also told us that they have on-site engineers who would typically address any safety concerns directly with the contractor and inform the contracting official responsible for entering information into CPARS. Navy policy requires significant problems, including severe personnel injuries, to be reported to the NAVSEA Commander through the use of trouble reports. In addition, the SUPSHIP supervisor implements hazard identification and reporting processes and ensures the collection, evaluation and reporting of data for the determination of contractor award fees and past performance data bases. SUPSHIP also assesses the overall effectiveness of contractor safety and health management systems and provides safety program assessments for quarterly reviews. Finally, the SUPSHIP Operations Manual provides that SUPSHIP personnel who are aware of any major or willful contractor violation of federal, state, or local laws and regulations (for example, recurring/major unsafe work practices) will report these violations. Personnel training. According to a USACE official, USACE requires the designated quality control manager for each worksite to take the USACE Construction Quality Management course before being approved to work on a project. The course aims to ensure that construction is performed according to plans and specifications, on time, within budget, and in a safe work environment. In addition, contractor safety managers are required by the USACE manual to be on all project sites for USACE construction contracts and must complete certain OSHA training or equivalent training. This program provides training on the recognition, avoidance, abatement, and prevention of workplace hazards. According to officials, NAVSEA recommends that its personnel working in acquisition of defense systems or maintenance of ships and aircraft undergo acquisition safety training. A NAVSEA contracting official told us that the first draft of this training has been developed and will be required training for NAVSEA contracting personnel. Officials said that the aim of this training is ensure that safety is considered when developing contract requirements. DOD obligates hundreds of billions of dollars each year on contracts, including those for work in high-risk industries such as construction and ship building and repairing, and some companies have received DOD contracts after being cited for serious workplace safety or health violations. Even if these violations did not occur during work on a DOD contract, they could be relevant to decisions about new DOD contracts, for example, when a prospective contractor has not previously received federal contracts or when past performance information does not address workplace safety. However, the incidence of serious safety and health violations among all inspected DOD contractors remains unknown because OSHA does not require a corporate identification number in its inspection data. Furthermore, OSHA’s website currently cannot be searched by a corporate identification number. Without these enhancements to OSHA’s inspection data and website, DOD contracting officials and other interested parties are likely to experience challenges obtaining accurate information about contractors’ workplace safety and health records. In addition, DOD contracting officials may be unaware of OSHA’s website because DOD has not advised them that this resource exists. Despite some data limitations, OSHA’s website currently can be used to obtain information about contractors’ workplace safety and health records in some cases. While DOD contracting officials are not required to consider information about contractors’ workplace safety and health before awarding contracts, they have multiple opportunities to do so. Unless DOD provides information about OSHA’s website to contracting officials, they may remain unaware that this resource exists, and may miss opportunities to consider safety and health concerns when awarding new contracts. Furthermore, some DOD contracting officials may lack readily accessible information on contractors’ past workplace safety performance because DOD does not require a safety performance rating for contracts department-wide. One of the DOD components we selected has a practice of requiring construction contractors’ performance to be rated with respect to workplace safety at the completion of each contract. However, DOD does not require a safety performance rating for other components’ construction contracts or contracts in other industries with similarly high rates of occupational injuries, such as manufacturing. Without exploring the feasibility of requiring a department-wide safety performance rating for all contracts in high-risk industries, DOD may miss opportunities to reduce risks by considering safety concerns when awarding new contracts in these industries. We are making three recommendations, including one to OSHA and two to DOD. Specifically: The Assistant Secretary of Labor for Occupational Safety and Health should explore the feasibility of requiring a corporate identification number in its inspection database and enabling its website to be searched by that number. This should include exploring the following issues: which corporate identification number would be most appropriate to options for obtaining this number from employers; and options for entering this number in its database that would prevent or minimize delays in closing inspection records. (Recommendation 1) The Secretary of Defense should provide information to contracting officials to advise them that the OSHA website is a resource for information about contractors’ workplace safety and health records. (Recommendation 2) The Secretary of Defense should explore the feasibility of requiring a safety performance rating for contracts in industries that have relatively high rates of occupational injuries, such as manufacturing, construction, and ship building and repairing. (Recommendation 3) We provided a draft of this report to DOL and DOD for review and comment. DOL’s Occupational Safety and Health Administration (OSHA) and DOD provided written comments, which are reprinted in appendixes II and III, respectively. With respect to our first recommendation that OSHA explore the feasibility of requiring a corporate identification number in its database and enabling its website to be searched by that number, OSHA did not state whether it agreed with our recommendation. OSHA acknowledged the potential utility of obtaining a unique identifier from each employer and said it will continue to promote the collection of Employer Identification Numbers (EIN) or Tax Identification Numbers (TIN) whenever possible by issuing a revised memorandum to field staff to reinforce the importance of collecting this information. OSHA stated that it does not view EINs as confidential or protected from disclosure. However, OSHA expressed concerns about protecting TINs and Social Security Numbers from disclosure, and noted that it would not be able to make a data field available for public search if it contained either of these numbers. OSHA also raised concerns about the financial cost associated with redesigning the agency’s data system. We encourage OSHA to explore options for addressing these concerns as it further considers how to implement our recommendation. With respect to our second and third recommendations that DOD provide information to contracting officials about the OSHA website and explore the feasibility of requiring a safety performance rating for contracts in high-risk industries, DOD agreed with both recommendations and identified implementation timelines. We are sending copies of this report to the appropriate congressional committees, the Secretary of the Department of Labor, the Secretary of the Department of Defense, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact William T. Woods at (202) 512-4841 or woodsw@gao.gov, or Chelsa Gurkin at (202) 512-7215 or gurkinc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This report examines (1) the incidence of prior serious safety or health violations among selected companies with Department of Defense (DOD) manufacturing and construction contracts, and (2) how DOD and selected DOD components address contractor workplace safety and health during the acquisition process. To describe the incidence of prior serious safety or health violations among selected companies with DOD manufacturing and construction contracts, we matched federal contracting data to the Department of Labor’s (DOL) Occupational Safety and Health Administration (OSHA) inspection data for selected contractors, interviewed OSHA officials, and reviewed relevant OSHA policy. Our data matching process is described below. To describe how DOD and selected DOD components address contractor workplace safety and health during the acquisition process, we selected two military departments (Army and Navy) and two components within these departments (the U.S. Army Corps of Engineers (USACE) and the Naval Sea Systems Command (NAVSEA)). We interviewed officials from DOD and these departments and components, and reviewed relevant DOD and component-level policy and guidance. To provide illustrative examples, we selected a non-generalizable sample of three USACE and three NAVSEA contracts, reviewed relevant contract file documentation, and interviewed knowledgeable contracting officials. Our criteria for selecting contracts and our review of contract file documentation are described below. While this review primarily focused on the award phase of the contracting process, NAVSEA and USACE officials also provided some information on the pre-award and contract performance phases of the contracting process, which we include in this report where relevant. For example, we interviewed USACE and NAVSEA safety officials about safety oversight practices during the contract performance phase. We also obtained examples of safety- related requirements for each of our selected contracts by interviewing contracting officials and reviewing contract documentation. To address both objectives, we reviewed relevant federal laws and regulations. To describe the incidence of prior serious safety or health violations among selected companies with Department of Defense (DOD) manufacturing and construction contracts, we matched federal contracting data to OSHA inspection data. OSHA categorizes a violation as “serious” when there is a substantial probability that death or serious physical harm could result, and the employer knew, or could have known with the exercise of reasonable diligence, of the hazard. Specifically, we matched contracting data from the Federal Procurement Data System- Next Generation (FPDS-NG) and the System for Award Management (SAM) to inspection data from the Occupational Safety and Health Information System (OIS) and Integrated Management Information System (IMIS). We assessed the reliability of the federal contracting data and OSHA inspection data by (1) performing electronic testing of relevant data elements, (2) reviewing existing information about the data and the systems that produced them, and (3) collecting information from federal officials knowledgeable about the data. Based on these reviews, we found the employer identification information in the federal contracting data, obligation amounts in the federal contracting data, and the OSHA inspection data to be sufficiently reliable for our purposes. First, we used FPDS-NG data to select the 100 companies with the largest DOD manufacturing contracts and the 100 companies with the largest DOD construction contracts (as measured by federal obligations) in fiscal year 2017. We focused on the manufacturing and construction industries because they have relatively high rates of occupational injuries, according to data from DOL’s Bureau of Labor Statistics (BLS), and over half of DOD contract obligations in that year were for contracts in these industries, according to FPDS-NG data. Next, we identified duplicate or related companies, and entities that were not private companies with DOD contracts performed within the United States, and narrowed this list of 200 companies to 192 companies. In fiscal year 2017, DOD obligations for contracts with these 192 companies accounted for about 79 percent of DOD’s obligations for contracts in the manufacturing and construction industries and about 46 percent of DOD’s total contract obligations. Then, to determine whether the 192 companies had been inspected by OSHA or state occupational safety and health agencies from fiscal years 2013 to 2017, we used automated matching procedures that compared the Employer Identification Numbers (EINs) and company names entered in the federal contracting databases to those entered in OSHA’s inspection databases. Specifically, we used FPDS-NG to identify the Data Universal Numbering System (DUNS) numbers for our selected companies, and then used SAM to identify the EINs that corresponded to each of those DUNS numbers. Next, we matched those EINs to the EINs in the OSHA inspection data. We considered a company to be a match if the EINs were identical and either (1) the company names were the same or similar, or (2) the company names were different, but we identified a relationship between the two company names, such as a parent company/subsidiary relationship, through an internet search. Using this process, we initially identified 90 selected companies that were inspected by OSHA from fiscal years 2013 to 2017. After completing this matching process, we sent a list of the remaining companies that we were unable to match to OSHA for review. Specifically, we asked OSHA officials to identify whether those companies were inspected from fiscal years 2013 to 2017, and provide inspection numbers for those companies that were inspected. OSHA officials reviewed this list and reported that OSHA had inspected some of the unmatched companies, and provided related inspection numbers. We then added those inspections to our analysis, which brought the total number of selected companies inspected by OSHA during this time period to 106 of 192. Our results are not generalizable to all companies that were awarded DOD manufacturing and construction contracts in fiscal year 2017. That year, about 29,000 companies had DOD manufacturing or construction contracts, and we reviewed a non-generalizable sample of 192 companies. In addition, limitations in the data do not allow a determination of whether the safety and health violations we identified occurred during work on a DOD contract because OSHA data do not include that information. Our counts of violations include only those in citations issued by OSHA or state agencies to our selected contractors as determined by our matching process, and only those that resulted from closed inspections where the violations and penalties are considered final. In addition, our counts of violations exclude any in citations issued only to subcontractors. According to OSHA officials, in certain circumstances, OSHA may cite both a prime contractor and a subcontractor for a violation, but in these cases the data would be recorded under two separate inspection numbers, which may or may not be linked in OSHA’s database. As a result, we did not attempt to identify inspections and violations for subcontractors. Furthermore, our counts of violations might exclude those in citations issued to any of the selected contractors’ subsidiaries or locations not identified by our matching process. We counted inspections and violations at the parent company level. Many of the companies we selected had multiple locations, and some may have had subsidiaries. OSHA inspections take place at the local worksite level. As a result, the number of violations we report reflects the total number of violations we identified across the selected companies’ various locations or subsidiaries that were inspected from fiscal years 2013 to 2017. To the extent that our matching process did not capture every company location or subsidiary, our findings may underestimate the actual number of inspections and violations among our selected companies. To provide examples of how selected DOD components address contractor workplace safety and health during the acquisition process, we selected a non-generalizable sample of three USACE and three NAVSEA contracts. To select these contracts, we first identified the 50 companies with the largest DOD construction contracts and the 50 companies with the largest DOD manufacturing contracts (as measured by federal obligations) in fiscal year 2017. Because our review focused on USACE and NAVSEA, we narrowed this list to USACE construction contractors (45) and NAVSEA shipbuilding or ship repair contractors (18). Next, we searched OSHA’s online inspection data to determine whether these contractors were cited for serious workplace safety or health violations within the last five years (fiscal years 2013 to 2017). Then, we selected the three USACE contractors and the three NAVSEA contractors that had the highest number of serious violations for closed OSHA inspections. We counted serious violations at the parent company level, which may include violations at different company locations. For example, OSHA violations we identified for one selected contractor occurred during both tank manufacturing and ship repair at different locations within the United States. After selecting these six companies, we identified the new USACE and NAVSEA contracts that were awarded to each selected company in fiscal years 2017 and 2016. We selected contracts that were awarded in fiscal year 2017 or 2016 because we expected that documentation for those contracts would be more readily available than for contracts awarded in previous years. Starting with the contracts that were awarded in fiscal year 2017, we selected one contract for each of these six contractors that had the highest total contract value (including base and all options) and provided diversity with respect to the contracting office that awarded the contract and the location where the work was performed. We excluded contracts that were for design or planning, rather than actual construction, shipbuilding, or ship repair. We also considered the proximity of the violation dates to the contract award date, and excluded contracts where all of the violations occurred after the contract was awarded, or immediately before the contract was awarded. In three cases, to satisfy these inclusion and exclusion criteria, it was necessary to select a contract that was awarded in 2016 and/or a contract that had the second highest total value. Each of our six selected contracts had a total contract value that was above the simplified acquisition threshold, which for the timeframe of our sample was generally above $150,000. (See table 2.) Of these six contractors, two had prior OSHA violations that occurred at the same location where the work on the selected contract was performed. However, we were unable to determine from the available data whether these violations occurred during work on a prior DOD contract, because OSHA data do not include that information. For each of the six selected contracts, we reviewed available relevant documentation in the contract file to determine how, if at all, officials considered information about contractors’ workplace safety and health in awarding the contract. These documents—depending on the type of source selection—included the contract solicitation, source selection plan, evaluation of contractor proposals, and the responsibility determination. In addition, we interviewed the contracting officials who awarded each of the six contracts to discuss how and why they considered available information about contractors’ workplace safety and health before awarding the contract, including whether this information was considered as part of the responsibility determination. We also determined whether prior OSHA violations were recorded in the Federal Awardee Performance and Integrity Information System (FAPIIS), to the extent these records were saved in the contract file. In addition, we obtained and reviewed Contractor Performance Assessment Reporting System (CPARS) reports for any information about occupational safety and health performance on past contracts, such as comments on safety practices or accidents, or the presence of safety ratings, as available in the files. The results of this review of contract file documentation cannot be generalized. We conducted this performance audit from February 2018 to February 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contacts named above, Betty Ward-Zukerman (Assistant Director), Caitlin Croake (Analyst-In-Charge), Amy Sweet, Sonja Bensen, and Cathy Roark made key contributions to this report. Also contributing to this report were Marie Ahearn, Blake Ainsworth, Hiwotte Amare, Vincent Balloon, James Bennett, Linda Collins, Sarah Cornetto, Holly Dye, Andrea Evans, Laurier Fish, Suellen Foth, Kristen Jones, Sheila McCoy, Jean McSween, Diana Moldafsky, Stacy Ouellette, Anh Nguyen, Jason Rodriguez, Almeta Spencer, Kelly Turner, Kristin Van Wychen, Alyssa Weir, and Eve Weisberg.", "summary": "DOD is the largest contracting agency in the federal government, obligating about $320 billion for contracts in fiscal year 2017. Some DOD contracts—including some in the manufacturing and construction industries—involve work that can be dangerous, and questions have been raised about working conditions for these workers. The National Defense Authorization Act for Fiscal Year 2018 includes a provision for GAO to review issues related to the safety and health records of DOD contractors. This report examines: (1) the incidence of prior serious safety or health violations among selected companies with DOD manufacturing and construction contracts, and (2) how DOD and selected DOD components address contractor workplace safety and health during the acquisition process. GAO matched federal contracting data for fiscal year 2017 to OSHA inspection data for fiscal years 2013-2017 (most recent available); interviewed officials from OSHA, DOD, selected military departments, and selected DOD components; reviewed documentation from six selected DOD contract files; and reviewed relevant federal laws and regulations, policy, and guidance. Some selected companies with Department of Defense (DOD) manufacturing or construction contracts in fiscal year 2017 were previously cited for serious safety or health violations, according to GAO's analysis of federal data. Of the 192 companies with DOD contracts GAO selected for review, 106 had been inspected by the Department of Labor's (DOL) Occupational Safety and Health Administration (OSHA) or state occupational safety and health agencies during fiscal years 2013 through 2017. These inspections resulted in 83 companies being cited for at least one violation, including 52 with at least one serious violation (see figure). However, available data do not allow a determination of whether these violations occurred during work on a DOD contract because OSHA inspection data do not include that information. The incidence of violations among all inspected companies with DOD contracts cannot be determined because OSHA does not require its staff to obtain and enter a corporate identification number in its inspection data, which is needed to match contracting data to inspection data. As a result, OSHA's data do not consistently include these numbers, and users of OSHA's website cannot use these numbers to search for companies' previous violations. According to federal internal control standards, management should share the quality information necessary to achieve the entity's objectives. Unless OSHA explores the feasibility of requiring a corporate identification number in its inspection data, website users will likely have difficulty obtaining accurate information on individual companies' previous violations. DOD contracting officials have opportunities during the acquisition process to address contractor workplace safety and health. For example, before awarding certain types of contracts, officials may consider workplace safety and health information when they evaluate prospective contractors' performance on past contracts. However, the past performance information that is available for officials to consider varies by DOD component. One component has a practice of requiring construction contractors to be rated on workplace safety at the completion of the contract, but DOD does not require a safety performance rating department-wide. As a result, contracting officials in other components may lack readily accessible information on contractors' past safety performance, and DOD may miss opportunities to consider safety concerns when awarding new contracts, particularly those in high-risk industries with relatively high rates of occupational injuries, such as manufacturing and construction. GAO is making one recommendation to OSHA and two recommendations to DOD to enhance available information on contractor workplace safety. OSHA neither agreed nor disagreed with GAO's recommendation, but planned to take action to address it. DOD agreed with the recommendations.", "document_type": "gao"}
{"report": "In 2012, DOD developed plans to establish 133 CMF teams focused on offensive operations, defensive operations, and DOD network protection. DOD provided budget resources for these teams beginning in fiscal year 2014. It subsequently set goals for reaching initial operational capability and full operational capability. Later in this report we describe how some of the methods used to facilitate these teams’ achievement of full operational capability subsequently affected readiness. Once each CMF team has achieved full operational capability, it is required to certify to its mission at least every 2 years. According to CYBERCOM’s 2017 readiness guidance, in order for each CMF team to achieve the best readiness rating it must certify to its mission every 12 months. According to the DOD Cyber Strategy published in 2015, the first wave of CMF teams will include nearly 6,200 military, civilian, and contractor support personnel from across the military departments and defense components, when they are fully staffed. In February 2017, the commander of CYBERCOM endorsed an Army proposal to present its 21 Reserve component Cyber Protection Teams (11 Army National Guard and 10 Army Reserve) for assignment to U.S. Strategic Command to help address increased mission requirements. These 21 teams represent a second wave of teams, which CYBERCOM has scheduled to achieve full operational capability by September 30, 2024. The second wave of 21 Army Reserve component teams are to include more than 800 personnel once they are fully staffed. The CMF teams are aligned with various DOD organizations, as shown in figure 1. The military service cyber components—Army Cyber Command, Fleet Cyber Command, Marine Corps Forces Cyberspace, and Air Forces Cyber—are CYBERCOM’s service elements and support CYBERCOM in achieving its missions. The personnel on each team represent a variety of specialties, such as intelligence analysts, linguists, and cyber operators and specialists. Figure 2 provides a hypothetical example of how each team might combine personnel from different specialties to carry out its missions. This figure does not show the actual composition of any type of team, but rather provides notional examples of how each team consists of personnel from different specialties who unite to perform cyber missions as part of the CMF. Training personnel for the CMF occurs in four phases and is administered by different entities, as shown in figure 3. Phase one basic training is the initial training performed by the military services that is delivered to any new recruit so that he or she may be assigned a military specialty. As shown in figure 2, CMF personnel draw from a number of different military specialties, including cyber, all-source intelligence, signals intelligence, information technology, and language specialists. Phase one basic training is not necessarily cyber-specific, as it is meant to provide military personnel with the basic skills needed to perform a particular occupation for the service. For example, CMF teams include intelligence professionals who may be assigned to analyze intelligence information that comes from a variety of sources. Training in phases two (foundational), three (collective), and four (sustainment) are focused more directly on the specific skills required to function as a member of the various CMF teams. To establish and train the CMF teams, DOD has assigned components and senior officials with CMF training roles and responsibilities. The key responsibilities for training the CMF are summarized in table 1 below; a more inclusive list is presented in appendix I. As part of the department’s efforts to develop and maintain trained CMF teams, CYBERCOM and the military services have implemented a number of initiatives. Specifically, CYBERCOM established consistent training standards, developed standard operating procedures for readiness reporting, and established and maintained a series of phase two foundational training courses. Further, CYBERCOM and the military services used existing training capabilities to build CMF teams. However, many of the teams that have been built are not yet fully trained and, according to agency officials, have “generally low” readiness levels. In 2012, CYBERCOM established consistent standards for CMF training phases within its responsibility, and the command has continuously updated those standards, as needed, to meet evolving requirements. Specifically, the command has established and updated the standards for phases two (foundational), three (collective), and four (sustainment) of CMF training. These standards apply to all military personnel regardless of service affiliation or active/reserve status. The standards are contained primarily in two documents. First, CYBERCOM issued and has regularly updated the Joint Cyberspace Training and Certification Standards (JCT&CS) to create standardized joint procedures, guidelines, and standards for individual staff and collective training, and to accurately assess CMF teams’ ability to perform their missions. This document was most recently revised in February 2018, to update, among other things, the tasks and abilities associated with CMF work roles based on feedback from experts within the military services and CYBERCOM. Second, CYBERCOM published the CMF Training and Readiness Manual to serve as the primary training and evaluation guidance for DOD cyber professionals. The CMF Training and Readiness Manual has been updated 13 times since it was originally issued in 2013, and it is CYBERCOM’s authoritative guide to building and maintaining cyber training and readiness for its personnel. It provides graduated levels of evaluated training that teams can use in preparing for certification and in being certified. Additionally, it identifies approved training events and the mission-essential tasks, associated standards, and key duties for members of CMF teams. The manual requires each team to recertify every 2 years, or upon recovery from a 50 percent or higher turnover of CMF team personnel. In December 2017, CYBERCOM published standard operating procedures for readiness reporting that CMF teams are to use to assess whether they have the resources and capability to perform their missions. The procedures define CMF readiness reporting guidelines related to personnel, equipment, and training. For example, the document identifies three training metrics that evaluate (1) whether personnel are trained to job qualification standards; (2) whether CMF teams have successfully completed supporting tasks during training exercises, events, or real world operations; and (3) the length of time between formal evaluations. Specifically, the standard operating procedures emphasize that in order to obtain the best training readiness rating, teams must perform an evaluated event or operation at least once every 12 months. CYBERCOM maintains and coordinates a series of CMF courses for phase two foundational training. It develops and administers these course requirements for all of the CMF work roles and requires personnel to complete courses specific to their job responsibilities. All CMF personnel filling a specific mission and role complete the same foundational courses, regardless of military service, employment status—active duty or reserve—or type of CMF team to which they are assigned. For example, all intelligence analysts on CMF teams are to complete the same 14 courses that are specific to their role on the team. CYBERCOM training directorate officials told us they had to make changes to the training progression over time to adapt to the changing threat environment. Accordingly, CYBERCOM has added, modified, or deleted phase two foundational training courses over the past 4 years. For example, in the past 4 years CYBERCOM consolidated four existing courses into a single introductory cyber course that is taken by all-source intelligence analysts who will be part of CMF teams. In November 2017, the command updated the phase two foundational training requirements by removing three courses that were required for a variety of Cyber Protection Team work roles. CYBERCOM also added a new networking course that is a pre-requisite to a course that comes later in the training progression for Cyber and National Mission Team mission commanders. The most recent update also emphasized that Cyber Protection Team personnel must complete the Intermediate Cyber Core Course, the Cyber Protection Team Core Course, and then their specific methodology courses, in that order. According to officials from the service cyber components, the changes CYBERCOM has made to its phase two foundational training progression have been transparent and have addressed evolving threats. However, the changes have also negatively affected training time frames, particularly for the CMF teams composed of National Guard and Reserve personnel. Because National Guard and Reserve teams are scheduled to achieve full operational capability after the active duty teams, they are more likely to be subject to the newer training progressions, which in some cases require a few additional days of courses. Officials from the National Guard told us that this additional training time is more difficult to schedule for National Guard and Reserve personnel because—unlike the active duty personnel who are available to train full time—National Guard and reservist personnel are available to train only one weekend per month and generally for 2 weeks of annual training. Additionally, most of these personnel must coordinate time off from their full-time jobs to take the required phase two foundational training courses. To help address these challenges, CYBERCOM officials told us they use mobile training teams. The Army Cyber School has also used mobile training teams to provide CMF training opportunities to Reserve personnel. The officials from CYBERCOM and the Army told us that the mobile training teams make training more accessible by avoiding the need for the National Guard and Reserve personnel to travel. DOD has used existing training capabilities—including courses, instructors, and facilities—throughout all phases of CMF training. For example: Joint Cyber Analysis Course. The Navy’s Center for Information Warfare Training is the host for the Joint Cyber Analysis Course—a phase one basic training course for personnel designated for cryptologic roles. CYBERCOM recommends this course for many CMF work roles. Cyber and Cryptologic training institutions. CYBERCOM has partnered with the Defense Cyber Investigation Training Academy, the Defense Information Systems Agency, the National Security Agency, and military service schoolhouses to deliver phase two foundational training for the CMF. The Defense Cyber Investigation Training Academy offers almost all of the training courses needed by Cyber Protection Teams, and Army officials said they used the expertise and course materials provided by the Defense Cyber Investigation Training Academy to develop Cyber Protection Team training courses that they offer at the Army Cyber School as well. National Security Agency’s National Cryptologic School provides a majority of the other phase two foundational CMF training courses. According to officials from CYBERCOM and the National Cryptologic School, reliance on existing training capabilities and expertise from the National Security Agency enabled the command to quickly establish CMF capabilities. Operational events. CYBERCOM used both simulated and real-world operational events on networks to support the certification of CMF teams. For example, CYBERCOM officials told us that CYBER KNIGHT is a training event offered periodically by CYBERCOM for CMF teams to exercise national and non-national mission sets. CYBER FLAG and CYBER GUARD, also conducted by CYBERCOM on a periodic basis, utilize a dynamic joint cyber training environment and, according to CYBERCOM officials train all types of CMF teams. In addition to using simulated events through exercises, CYBERCOM and military service officials said that teams were allowed to use real- world operations to meet phase three collective training requirements. The military services and CYBERCOM plan to continue to use existing resources, such as the service school houses, for new and continuous training into the future. For example, as part of their training transition plan, Marine Corps officials reported that they have a contract in place with Navy’s Space and Naval Warfare Systems Command to provide additional training to Marine Corps CMF personnel after they complete the phase two foundational training progression. Additionally, the Army Cyber School, which provides CMF-specific training for the Army, currently trains Marine Corps personnel as well. The Army and Marine Corps have training agreements in place to continue this arrangement. Figure 4 below shows a member of the National Guard participating in a cyber training exercise. We found that many of the CMF teams for which DOD has reported achieving full operational capability actually require further training, for varying reasons. For example, officials from many key organizations across the DOD cyber enterprise told us that the services moved some personnel among teams, reducing the readiness for teams from which personnel were transferred. Officials from the Office of the Under Secretary of Defense for Personnel and Readiness, Joint Staff, and the military services cited other challenges affecting CMF team readiness levels as well, including the long time frames needed to obtain the appropriate clearances for CMF personnel and the high pace of operations for the teams, leaving little time for training. The same officials from across DOD’s cyber enterprise affirmed that, taken together, these actions and circumstances have had a negative effect on CMF team resource readiness levels. In April 2018, the commander of CYBERCOM acknowledged in testimony that “much works remains to be done to make the personnel proficient at their duties and the whole team ready and able to perform whatever missions might be directed.” The CMF teams were not fully trained and had lower readiness levels because CYBERCOM and the military services focused primarily on the teams’ achieving full operational capability by October 1, 2018, rather than on building operational readiness. Building operational readiness requires the teams to simultaneously have the appropriate number of sufficiently trained personnel across the force. According to the CMF Training Transition Plan, CYBERCOM’s senior leadership directed the command to achieve full operational capability, and it designated that effort as a higher priority than operational readiness. CYBERCOM officials told us that they recognized the low readiness of the CMF teams and have identified two actions to address the training deficiencies—and associated effects on readiness—for the CMF teams. First, according to the officials, CYBERCOM has developed a system that assigns unique identifiers to each person in the CMF and allows CYBERCOM to easily track when personnel move from one team to another. Second, in December 2017, CYBERCOM issued its readiness reporting standard operating procedure that establishes new readiness reporting guidelines. CYBERCOM officials stated that these guidelines emphasize readiness over the achievement of interim milestones, such as full operational capability. Given that CYBERCOM recently implemented these efforts to improve the readiness of the CMF teams, and that the quarterly readiness reports indicate improved resource readiness for personnel and training metrics, we are not making recommendations related to this issue. Through our body of work on defense cyber issues, we will continue to monitor DOD’s and CYBERCOM’s efforts to maintain a ready CMF. DOD has taken steps to shift its focus from building a trained CMF to maintaining this force, but it has not taken key actions to ensure that the department is poised to maintain CMF training following this transition. Specifically, the military services have not developed plans that include time frames for validating all phase two foundational training courses, or that comprehensively assess their training requirements. Further, as of June 2018, CYBERCOM had not provided a plan for establishing independent assessors to evaluate and certify the completion of phase three collective training for CMF teams. DOD officials told us that the department is shifting its focus away from building and toward maintaining a trained CMF. For example, the Army is leading the development of a Persistent Cyber Training Environment. The goal of that training environment is to provide on-demand access to scenarios that Army officials told us will enhance the quality, quantity, and standardization of phase three (collective) and phase four (sustainment) training and exercise events. The Persistent Cyber Training Environment is scheduled to provide some operational capability by 2019, and it is expected to continue to evolve to meet training needs. In addition to building a Persistent Cyber Training Environment, the department has developed the CMF Training Transition Plan, which will transfer administration of phase two foundational training from CYBERCOM to the services. Specifically, beginning in October 2018, the military services will assume responsibility for phase two foundational training of CMF personnel, which CYBERCOM has centrally managed since CMF training began in 2013. Officials from the services and CYBERCOM have held quarterly meetings to help guide the implementation of this plan. According to the CMF Training Transition Plan, the transfer is being made in response to a direction in Senate Report 114-49 accompanying a bill for the National Defense Authorization Act for Fiscal Year 2016. The report directed the DOD Principal Cyber Advisor, the Commander, CYBERCOM, and the service secretaries to develop a plan for the military services to complete all required training for the second wave of CMF teams and to maintain individual training capabilities for the existing teams. In January 2017 the Joint Staff and Principal Cyber Advisor published the CMF Training Transition Plan, to transition CMF training to a model that complied with congressional committee direction. The principal goal of this approach is to drive efficiencies and reduce training development and delivery costs. According to the plan, CYBERCOM maintains control of the standards for phase two foundational training, while the Army, Navy, and Air Force are to assume specific joint curriculum lead roles. These roles entail developing joint training plans for the courses under the work roles they are assigned. In addition, the joint curriculum leads (i.e., Army, Navy, and Air Force) are responsible for identifying training gaps and developing learning objectives and courseware based on the CYBERCOM training task list requirements for each of the work roles. For example, under its curriculum lead role, the Army has accepted responsibility for the cyber planner courses. In carrying out this role, the Army developed the Cyber Operations Planners Course and submitted it to CYBERCOM to establish as an approved course for all cyber planners—regardless of service affiliation and of active or reserve duty status—in the CMF. Figure 5 shows the work role categories and responsibilities for which each military service has agreed to be curriculum lead. In November 2017, CYBERCOM directed the military services to develop plans to implement their responsibilities in support of the CMF Training Transition Plan. In accordance with the training transition plan, the military services will assume responsibility for phase two foundational course validation as part of their joint curriculum lead duties. In February 2018, each of the four services provided a plan to CYBERCOM that, at a minimum, highlighted the efforts each service was taking to prepare for its new training transformation responsibilities, including phase two foundational course validation. The purpose of course validation is to determine whether a course adheres to CYBERCOM’s joint training standards as published in the Joint Cyberspace Training and Certification Standards (JCT&CS). CYBERCOM’s draft course validation guidance states that validation involves an examination of both the content of the courses, as well as the instructional methods. The manual states that the content should align with the knowledge, skills, and abilities for the appropriate CYBERCOM work roles and should meet the joint training standard. Further, the manual states that the validation of instructional methods examines how the course is taught and determines whether the methods are appropriate to support desired course outcomes. CYBERCOM’s draft course validation guidance lays out a series of requirements for the validation process, among which are the following: The military service that is submitting the course for validation is responsible for assembling course information, providing back-up data about the course, and securing subject matter experts to review the submission. The military service that is the joint curriculum lead for the course is responsible for reviewing the submissions and offering recommendations for modifications to courses to reflect joint standards. CYBERCOM is responsible for making final determinations of course validity. In this final review, CYBERCOM may hold discussions with key stakeholders, audit the course, review student feedback on the course, or review evaluation data from the course to inform its final validation determination. Our review of the services’ training transition plans found that the Army’s and Air Force’s plans address course validation to some degree, but they do not identify specific time frames for completing course validation. Specifically, the Army’s plan identifies the milestones, dates, and resources for the submission of two of its analyst and planner courses to CYBERCOM for validation, but it does not indicate when the service will submit its Cyber Protection Team Core Training Course for validation. The Air Force’s plan establishes a timeline for developing, finalizing, and distributing course validation guidance, but it does not have time frames or milestones indicating a time for beginning the process of submitting courses to CYBERCOM for validation. Standards for Internal Control in the Federal Government highlights the need to define objectives in specific terms, to include how objectives are to be achieved and time frames for their achievement. For example, the Navy’s plan indicates that the four courses for which it is responsible will be iteratively validated between fiscal years 2019 and 2021. While a 24- month time frame is broad and it may be challenging for CYBERCOM and the other services to know with precision when the Navy will complete its course validation efforts, the plan includes a time frame that CYBERCOM and the services can use for further discussion and planning purposes. The plans submitted by the Army and the Air Force indicate that the course validation time frames for phase two foundational courses are unknown because course validation is still dependent upon CYBERCOM’s review. The Army’s plan includes time frames for submitting to CYBERCOM two of the three courses it is responsible for developing, but one of the courses does not have any time frames. Further, the Air Force plan includes time frames for developing guidance on how to perform course validation that only carry it through September 2018; it does not have time frames for actually carrying out its course validation processes. As the military services assume phase two foundational training responsibilities from CYBERCOM, it is important that they coordinate with CYBERCOM to establish a timeline for course validation, as appropriate. With a clearer idea of which information can appropriately be removed from training courses, the services will be able to make informed decisions to balance the cost-effectiveness of the training with delivering trained cyber personnel to CMF teams more quickly. However, without an established time frame to assess and validate the efficiency and effectiveness of all phase two individual foundational training against established expectations, DOD will not be well positioned to reasonably assure that the phase two foundational training meets the needs of the CMF and its mission. Training plans should be detailed enough to provide insight into the number of people needed to fill specific positions to sustain an organization. As part of the training transition process, CYBERCOM required the military services to submit implementation plans that identify, among other things, training requirements and execution. Also, according to our prior work published in Human Capital: A Guide for Assessing Strategic Training and Development Efforts in the Federal Government, training plans should be designed to determine the skills and competencies a workforce needs to prepare for current, emerging, and future agency needs in pursuit of its missions. These needs include the size of the workforce; its deployment across the organization; and the knowledge, skills, and abilities needed for the agency to pursue its current and future missions. To ensure a strategic workforce planning approach, it is important that agencies consider how hiring, training, and other human capital strategies can be aligned to support mission success. The Army, Navy, and Air Force developed training transition implementation plans to address training requirements and execution to some degree, but the plans do not identify the number of personnel or teams and the specific training activities needed across all phases of training to maintain the CMF. For example, neither the Army nor the Air Force plan identifies the number (average or total) of personnel for each of the work roles described in figure 2 (for example, cyber operators, intelligence analysts, linguists) that the military services need to complete phase two foundational training courses to maintain the appropriate sizing and deployment of personnel across CMF teams. Additionally, the Army and Air Force plans do not identify the number of personnel or teams needed to conduct phase three (collective) and phase four (sustainment) training in future years. In contrast, the Navy’s plan identifies the average number of personnel who would need to take specific phase two foundational courses—including those being developed by other services and CYBERCOM—to maintain its CMF teams. However, the Navy’s plan does not include this same information for phases three and four of training. The Marine Corps did not address training requirements and execution within its implementation plan. According to officials from the Army and the Air Force, the February 2018 documents they provided in response to CYBERCOM’s requirement do not include plans that identify training requirements because submission of that information was not required by CYBERCOM. However, a November 2017 CYBERCOM memorandum clearly directed the military services with joint curriculum lead responsibilities to submit plans that support implementation of the department’s CMF Training Transition Plan, including training requirements execution data. Having a comprehensive plan that identifies the number of personnel or teams needed to accomplish specific training activities would help the services to better manage the number of personnel who need to be rotated into the CMF teams. It would also help the military services coordinate with each other on course offerings by providing situational awareness of the number of personnel from other services who could attend their courses in any given year. For example, the Air Force would know how many Army, Navy, and Marine Corps personnel would attend the courses being offered by the Air Force. Without a plan that comprehensively assesses and identifies the services’ training needs for each type of personnel, DOD cannot reasonably ensure that its training plan will support the transition to a joint training model or be aligned with its stated goal to maintain a trained and ready force. As of June 2018, CYBERCOM had not provided a plan for establishing independent assessors to evaluate and certify the completion of phase three collective training for CMF teams. CYBERCOM’s CMF Training and Readiness Manual explains that evaluations are necessary to assess readiness and provide commanders with a process to determine a team’s proficiency in the tasks it must perform during a mission. Assessors play an important role in this evaluation process by judging the performance of CMF teams using CYBERCOM’s evaluation forms, which establish common evaluation criteria to determine whether the team being evaluated has met the certification standards. CYBERCOM officials told us that to evaluate teams completing phase three certification through CYBERCOM events (approximately 50 percent, according to agency officials), the command provided a joint team of assessors. CYBERCOM and service officials told us that the services provided their own assessors for teams that completed phase three training through their respective service-hosted exercises. In discussions with Army and Air Force officials, they identified two challenges they have experienced with the services providing assessors to evaluate their own teams, which could lead to subjectivity in CMF team evaluations. First, in some instances the assessors have come from within the same chain of command as the CMF team and thus are not truly independent. Standards for Internal Control in the Federal Government discusses the importance of segregation of duties in designing control activities so that incompatible duties are segregated in order to mitigate the risk of management override of internal control. In this case, having an assessor from the same chain of command evaluate a CMF team’s performance in a certification event presents an increased risk of fraud through management override. Second, while the CMF Training and Readiness Manual includes checklists that assessors can use to evaluate team performance, according to service officials, the manual does not provide clear guidance on how to evaluate whether the tasks and performance standards have been sufficiently met by the team. The absence of such information could lead to subjective evaluations as to whether a team met the desired performance standard. According to one service official, these challenges could be addressed if CYBERCOM were to provide an expert who evaluates the training tasks and performance standards—an action that could lead to a more consistent application of evaluation criteria. When we asked officials from CYBERCOM’s training directorate about whether the command could provide more oversight for certification events, the officials acknowledged that, among other tasks, the command is responsible for ensuring that assigned joint cyber forces are trained, certified, and interoperable with other forces. The officials said that to do this, the command will use established training standards and develop a plan to train and certify CMF team evaluators to a set of standardized criteria. Command officials said they believe this will enable the services and CMF teams to have qualified assessors who are trained and certified by CYBERCOM to consistently evaluate the performance of the CMF teams based on joint standards. With this capability, for example, a Navy Cyber Protection Team assessor can be used by an Army Cyber Protection Team to evaluate that team in an operation, exercise, or training event. This training capacity should enhance the interoperability between the services and allow for consistent evaluation of a team’s performance. However, as of June 2018, CYBERCOM had not provided a plan to train and certify assessors from across the services; as such a plan had not yet been developed. Standards for Internal Control in the Federal Government explains that in defining objectives, management should clearly define what is to be achieved, how it will be achieved, and the time frames for achievement. Documenting these objectives in a plan also will help formalize the new process and ensure that the appropriate managerial emphasis is given to the effort. DOD has used similar mechanisms to implement changes to cyber training in the past, such as developing the CMF Training Transition Plan in response to moving phase two foundational training responsibility from CYBERCOM to the military services. Since phase three certification events act as a quality control mechanism for CMF teams, it is important that the events be independently evaluated to ensure that CMF teams are trained to a consistent standard. Without a documented plan to train and certify assessors to evaluate CMF phase three collective training certification events, the CMF teams will not be consistently evaluated as they are operationally certified. CYBERCOM assesses the prior experience of CMF personnel to meet training requirements through a process known as individual training equivalency. This process allows personnel to be exempted from specific training courses by showing that they have already met the learning objectives of the course through their prior experience. CYBERCOM established an Individual Training Equivalency Board consisting of subject matter experts and representatives from CYBERCOM, the National Security Agency, and service cyber components who review the applications and recommend whether equivalency should be granted. The Individual Training Equivalency Board reviewed approximately 700 applications for equivalency from September 2013 through April 2018, and more than three-quarters of those applicants had at least one course exemption approved. According to officials from CYBERCOM’s training directorate, which is responsible for administering the individual equivalency process, there are a number of reasons why requests for course exemptions are not approved. For example, some applicants are denied for administrative reasons, such as not filling out the paperwork correctly. Also, applicants are not eligible to receive exemptions for courses that are not part of their work role requirements, but some personnel try to do so. Officials also said that board members do not deem some applicants’ reported experiences as comparable to the knowledge and skills they would obtain from taking courses for which they seek exemptions. Based on our review CYBERCOM’s memorandums that document the approval or disapproval of approximately 700 individual requests for training exemptions, we observed that applicants typically requested exemptions for multiple courses, with some seeking exemptions for up to 16 courses. Altogether during this period, we found that CYBERCOM granted more than 1,400 equivalencies for approximately 90 different phase two foundational training courses. Certain courses were exempted more often than others. For example, the course for which CYBERCOM most frequently granted individual exemptions was the Joint Advanced Cyber Warfare Course. This 4-week course provides an orientation to CYBERCOM, the global cryptologic platform, the intelligence community, and allies and major partners in the conduct of cyber warfare operations, planning, and analysis of effects. Other courses that were commonly granted training exemptions included 1-week courses related to computer network exploitation, cyber offensive and defensive operations, and understanding network and operating system fundamentals. These courses teach the basic skills associated with performing CMF operations. Additionally, we found that CYBERCOM’s Individual Training Equivalency Board approved approximately 50 exemptions for Intermediate Cyber Core, which is an 8- week course that CYBERCOM training officials described as providing the background and proficiency needed to identify, understand, and navigate the digital environment. The officials said that the course also provides an understanding of network operational methods and offensive and defensive cyber operation principles. CYBERCOM has not established master training task lists for phase two foundational training, a key set of standards the services are to use in preparing course equivalency standards. The task lists correlate to the knowledge, skills, and abilities that the services will use to develop learning objectives and course materials for training. They are also important in informing the services’ ability to make equivalency application determinations because they form the learning objectives of the courses that may be bypassed. To determine whether an applicant’s experience is equivalent to what would be taught in a course; the entity making the decision must know the learning objectives of the course. However, as of May 2018, CYBERCOM officials were unable to provide evidence that the command had developed master training task lists for phase two foundational CMF training courses, as required. The January 2017 CMF Training Transition Plan required CYBERCOM to provide all mission and support team master training task lists for the phase two foundational training courses to the military services no later than March of 2018. Service and CYBERCOM officials said that they are holding monthly meetings to provide updates related to the training standards and other training transition-related information, but as of May 2018, CYBERCOM officials had not confirmed that they had provided the master training task lists to the services. Officials from the services told us that they need these master training task lists to develop clear decision rules as they assume responsibility for making equivalency decisions for phase two foundational training courses. When we interviewed CYBERCOM in February of 2018, officials told us that they were not aware of the requirement established in the CMF Training Transition Plan, but said they would start developing the master training task lists. Establishing clear standards is particularly important at this time, because the services are scheduled to assume responsibility for administration of the individual training equivalency process for Cyber Protection Team phase two foundational training courses in October 2018. Until CYBERCOM establishes and disseminates the master training task lists for phase two foundational CMF courses, the military services are at risk of developing inconsistent decision rules for their training equivalency processes, and the development of such processes could be delayed, resulting in the funding of training that is unnecessary. Developing and maintaining a trained cyber mission force is imperative to DOD’s ability to achieve its missions in the connected world within which it operates. DOD has made progress toward its goals of building and maintaining a trained cyber mission force. As DOD starts to focus on maintaining a ready CMF, addressing gaps in its training plans and structure will help it reach those goals. The Army’s and Air Force’s lack of time frames, like those established by the Navy in its implementation plan, for validating phase two foundational training could contribute to training inefficiency and unnecessarily long time frames for training personnel. Further, the military services, by not clearly identifying the number of personnel they need to train, hinder planning and coordination efforts to ensure that the training infrastructure is sufficient and is used efficiently. In addition, the absence of a plan for CYBERCOM to establish independent assessors for phase three collective training certification events may lead to teams being certified to different standards. Also, not having the master training task lists necessary to establish clear decision rules for granting individual training exemptions for phase two foundational training courses may contribute to inconsistent personnel skill levels and inefficient use of training resources. Focusing on maintaining sustainable readiness, as DOD has already begun to do, and addressing these weaknesses can lead to long-term improvements in the capability and capacity of its CMF. We are making eight recommendations to DOD. The Secretary of Defense should ensure that the Army, in coordination with CYBERCOM and the National Cryptologic School, where appropriate, establish a time frame to validate all of the phase two foundational training courses for which it is responsible. (Recommendation 1) The Secretary of Defense should ensure that the Air Force, in coordination with CYBERCOM and the National Cryptologic School, where appropriate, establish a time frame to validate all of the phase two foundational training courses for which it is responsible. (Recommendation 2) The Secretary of the Army should ensure that Army Cyber Command coordinate with CYBERCOM to develop a plan that comprehensively assesses and identifies specific CMF training requirements for phases two (foundational), three (collective), and four (sustainment), in order to maintain the appropriate sizing and deployment of personnel across the Army’s CMF teams. (Recommendation 3) The Secretary of the Navy should ensure that Fleet Cyber Command coordinate with CYBERCOM to develop a plan that comprehensively assesses and identifies specific CMF training requirements for phases three (collective) and four (sustainment) in order to maintain the appropriate sizing and deployment of personnel across the Navy’s CMF teams. (Recommendation 4) The Secretary of the Air Force should ensure that Air Forces Cyber coordinate with CYBERCOM to develop a plan that comprehensively assesses and identifies specific CMF training requirements for phases two (foundational), three (collective), and four (sustainment), in order to maintain the appropriate sizing and deployment of personnel across the Air Force’s CMF teams. (Recommendation 5) The Commandant of the Marine Corps should ensure that Marine Corps Forces Cyberspace coordinate with CYBERCOM to develop a plan that comprehensively assesses and identifies specific CMF training requirements for phases two (foundational), three (collective), and four (sustainment), in order to maintain the appropriate sizing and deployment of personnel across the Marine Corps’ CMF teams. (Recommendation 6) The Secretary of Defense should ensure that the commander of CYBERCOM develops and documents a plan for establishing independent assessors to evaluate CMF phase three collective training certification events. (Recommendation 7) The Secretary of Defense should ensure that the commander of CYBERCOM establishes and disseminates the master training task lists covered by each phase two foundational training course and convey them to the military services, in accordance with the CMF Training Transition Plan. (Recommendation 8) We provided a draft of the FOUO version of this product to DOD for review and comment and worked with the department to develop this unclassified product. In its comments on the FOUO version of this, reproduced in appendix II, DOD concurred with our recommendations. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to appropriate congressional committees; the Secretary of Defense, the office of the Principal Cyber Advisor, the Office of the Under Secretary of Defense for Personnel and Readiness, the Office of the Deputy Assistant Secretary of Defense for Cyber Policy, the Commander of CYBERCOM, the leadership of each of the service cyber components, and the director of the National Security Agency’s National Cryptologic School. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9971 or kirschbaumj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Based on our review of related statutes, Department of Defense (DOD) instructions and directives, and other guidance, we found that various DOD officials have been assigned a variety of CMF training roles and responsibilities, summarized in table 1 below. In addition to the individual named above, Tommy Baril, Assistant Director; Tracy Barnes; Patricia Farrell Donahue; Ashley Houston; Amie Lesser; Randy Neice; Geo Venegas; and Cheryl Weissman made key contributions to this report.", "summary": "Developing a skilled cyber workforce is imperative to DOD achieving its offensive and defensive missions, and in 2013 it began developing CMF teams to fulfill these missions. CYBERCOM announced that the first wave of 133 such teams achieved full operational capability in May 2018. House Report 115-200 includes a provision for GAO to assess DOD's current and planned state of cyber training. GAO's report examines the extent to which DOD has (1) developed a trained CMF, (2) made plans to maintain a trained CMF, and (3) leveraged other cyber experience to meet training requirements for CMF personnel. To address these objectives, GAO reviewed DOD's cyber training standards, planning documents, and reports on CMF training; and interviewed DOD officials. This is an unclassified version of a For Official Use Only report that GAO previously issued. U.S. Cyber Command (CYBERCOM) has taken a number of steps—such as establishing consistent training standards—to develop its Cyber Mission Force (CMF) teams (see figure). To train CMF teams rapidly, CYBERCOM used existing resources where possible, such as the Navy's Joint Cyber Analysis Course and the National Security Agency's National Cryptologic School. As of November 2018, many of the 133 CMF teams that initially reported achieving full operational capability no longer had the full complement of trained personnel, and therefore did not meet CYBERCOM's readiness standards. This was caused by a number of factors, but CYBERCOM has since implemented new readiness procedures that emphasize readiness rather than achieving interim milestones, such as full operational capability. DOD has begun to shift focus from building to maintaining a trained CMF. The department developed a transition plan for the CMF that transfers foundational (phase two) training responsibility to the services. However, the Army and Air Force do not have time frames for required validation of foundational courses to CYBERCOM standards. Further, services' plans do not include all CMF training requirements, such as the numbers of personnel that need to be trained. Also, CYBERCOM does not have a plan to establish required independent assessors to ensure the consistency of collective (phase three) CMF training. Between 2013 and 2018, CMF personnel made approximately 700 requests for exemptions from training based on their experience, and about 85 percent of those applicants had at least one course exemption approved. However, GAO found that CYBERCOM has not established training task lists for foundational training courses. The services need these task lists to prepare appropriate course equivalency standards. GAO is making eight recommendations, including that the Army and Air Force identify time frames for validating foundational CMF courses; the military services develop CMF training plans with specific personnel requirements; CYBERCOM develop and document a plan establishing independent assessors to evaluate training; and CYBERCOM establish the training tasks covered by foundational training courses and convey them to the services. DOD concurred with the recommendations.", "document_type": "gao"}
{"report": "VA’s mission is to promote the health, welfare, and dignity of all veterans by ensuring that they receive medical care, benefits, social support, and lasting memorials. In providing health care and other benefits to veterans and their dependents, VA relies extensively on IT systems and networks to receive, process, and maintain sensitive data, including veterans’ medical records and other personally identifiable information. Accordingly, effective information security controls based on federal guidance and requirements are essential to ensure that the department’s systems and information are adequately protected from loss, unauthorized disclosure, inadvertent or deliberate misuse, or improper modification, and are available when needed. Implementing an effective information security program and controls is particularly important for VA since it uses IT systems and electronic information to perform essential activities for veterans, such as providing primary and specialized health care services, medical research, disability compensation, educational opportunities, assistance with home ownership, and burial and memorial benefits. The corruption, denial, or delay of these services due to compromised IT systems and electronic information can create undue hardship for veterans and their dependents. The Federal Information Security Modernization Act of 2014 (FISMA) requires the head of each agency to provide information security protections commensurate with the risk and magnitude of harm resulting from unauthorized access, use, disclosure, disruption, modification, or destruction of the information and information systems used by or on behalf of the agency. The act also requires federal agencies to develop, document, and implement an agency-wide information security program to provide security for the information and information systems supporting their operations and assets by implementing policies and procedures intended to cost-effectively reduce risks to an acceptable level. In May 2017, the president signed Executive Order 13800 on strengthening the cybersecurity of federal networks and critical infrastructure. The order sets policy for managing cybersecurity risk and directs each executive branch agency to use the National Institute of Standards and Technology’s (NIST) cybersecurity framework to manage those risks. The NIST cybersecurity framework identifies specific activities and controls for achieving five core security functions: Identify: Develop an understanding of the organization’s ability to manage cybersecurity risk to systems, people, assets, data, and capabilities. Protect: Develop and implement appropriate safeguards to ensure delivery of critical services. Detect: Develop and implement appropriate activities to identify the occurrence of a cybersecurity event. Respond: Develop and implement appropriate activities to take action regarding a detected cybersecurity incident. Recover: Develop and implement appropriate activities to maintain plans for resilience and to restore capabilities or services that were impaired due to a cybersecurity incident. According to NIST, these five functions provide a high-level, strategic view of the life cycle of an organization’s management of cybersecurity risk. In fiscal year 2018, the 23 civilian agencies covered by the Chief Financial Officers Act of 1990 (CFO Act), including VA, reported spending over $6.5 billion on IT security- or cybersecurity-related activities. The 23 civilian agencies individually reported spending between $9 million and almost $1.9 billion on these activities. Collectively, these 23 agencies spent on average about 14 percent of their total IT expenditures on cybersecurity-related activities. VA reported spending about $386 million on cybersecurity, which represented about 8 percent of its total IT expenditures. In fiscal year 2018, federal agencies continued to report large numbers of information security incidents. As we previously noted, federal agencies reported over 30,000 security incidents during each of the last three fiscal years. Specifically, agencies reported a total of 30,899, 35,277, and 31,107 information security incidents in fiscal years 2016, 2017, and 2018, respectively. During those same periods of time, VA reported an average of 2,415 incidents annually, although the number of reported incidents steadily decreased from 2,808 to 1,776, as shown in figure 1. In fiscal year 2018, VA reported 1,776 incidents involving several threat vectors. These threat vectors included web-based attacks, phishing attacks, and the loss or theft of computer equipment, among others. Figure 2 provides a breakdown of information security incidents, by threat vector, reported by VA in fiscal year 2018. Perhaps most concerning of the incidents reported by VA is the relatively large percentage of incidents (41 percent) for which VA identified “Other” as the threat vector. Government-wide, agencies identified approximately 27 percent of their incidents in the “Other” category in fiscal year 2018. A large percentage of these incidents may indicate a lack of agency awareness and ability to investigate and catalog incidents. FISMA requires IGs to determine the effectiveness of their respective agency’s information security programs. To do so, OMB instructed IGs to provide a maturity rating for agency information security policies, procedures, and practices related to the five core security functions— identify, protect, detect, respond, and recover—established in the NIST cybersecurity framework, as well as for the agency-wide information security program. The ratings used to evaluate the effectiveness of agency information security programs are based on a five-level maturity model, as described in table 1. According to this maturity model, Level 4 (managed and measurable) represents an effective level of security. Therefore, if an IG rates the agency’s information security program at Level 4 or Level 5, then that agency is considered to have an effective information security program. VA was one of 18 CFO Act agencies where the IG determined that the agency-wide information security program was not effectively implemented during fiscal year 2018. The VA IG also determined the department’s maturity level for each of the five core security functions: Level 2 (defined) for the Detect function; Level 3 (consistently implemented) for the Identify, Protect, and Level 4 (managed and measurable) for the Respond function. As shown in figure 3, VA’s ratings were generally consistent with the maturity level ratings of other CFO Act agencies. Agency IGs or independent auditors assess the effectiveness of information security controls as part of the annual audits of the agencies’ financial statements. The reports resulting from these audits include a description of information security control deficiencies related to the five major general control categories defined by the Federal Information System Controls Audit Manual (FISCAM): security management controls that provide a framework for ensuring that risks are understood and that effective controls are selected, implemented, and operating as intended; access controls that limit or detect access to computer resources, thereby protecting them against unauthorized modification, loss, and disclosure; configuration management controls that prevent unauthorized changes to information system resources and assure that software is current and known vulnerabilities are patched; segregation of duties controls that prevent an individual from controlling all critical stages of a process by splitting responsibilities between two or more organizational groups; and contingency planning controls that help avoid significant disruptions in computer-dependent operations. For fiscal year 2018, most of the 24 CFO Act agencies had deficiencies in most of the control categories, as illustrated in figure 4. VA’s IG reported deficiencies in each of these categories for the department. As a result of these deficiencies, the IGs at 18 of the 24 CFO Act agencies designated information security as either a material weakness (six agencies, including VA) or significant deficiency (12 agencies) in internal control over financial reporting for their agency. For VA, fiscal year 2018 was the 17th year in a row that the department had reported a material weakness in information security. In addition, IGs at 21 of the 24 agencies, including VA, cited information security as a major management challenge for their agency for fiscal year 2018. The administration has developed key milestones and performance metrics for agency chief information officers (CIO) to use to assess their agency’s progress toward achieving outcomes that strengthen federal cybersecurity. The milestones and metrics have specific implementation targets, most of which are expected to be met by the end of fiscal year 2020. As of fiscal year 2018, most civilian CFO Act agencies, including VA, had reported meeting most of the implementation targets for that year. VA reported meeting six of 10 targets. Table 2 shows the number of agencies meeting their targets as of fiscal year 2018, as well as VA’s status in doing so. In several reports issued since fiscal year 2016, we described deficiencies related to key challenges that VA has faced in safeguarding its information and information systems. The challenges we reported related to effectively implementing information security controls; mitigating known security deficiencies; establishing elements of its cybersecurity risk management program; and identifying critical cybersecurity staffing needs. Our work stresses the need for VA to address these challenges as well as manage IT supply chain risks as it modernizes and secures its information systems. Effectively Implementing Information Security Controls VA has been challenged to effectively implement security controls over its information and information systems. As previously mentioned in this statement, the VA IG reported that the department did not have an effective information security program and has had deficient information security controls over its financial systems. The weaknesses described by the IG are consistent with the control deficiencies we identified during an examination of VA’s high-impact systems that we reported on in 2016. In those reports, we described deficiencies in VA’s implementation of access controls, patch management, and contingency planning. These deficiencies existed, in part, because the department had not effectively implemented key elements of its information security program. Until VA rectifies reported shortcomings in its agency-wide information security program, it will continue to have limited assurance that its sensitive information and information systems are sufficiently safeguarded. Adequately Mitigating Known Security Deficiencies VA has not consistently mitigated known security deficiencies in a timely manner. As mentioned earlier, VA has reported a material weakness in information security for financial reporting purposes for 17 consecutive years. In fiscal year 2016, we recommended 74 actions for the department to take to improve its cybersecurity program and remedy known control deficiencies with selected high-impact systems. However, as of October 2019, over 3 years later, VA had implemented only 32 (or 43 percent) of the 74 recommendations. One of the remaining unimplemented recommendations calls for the department to consistently and comprehensively perform security control assessments. This recommended activity is an important element of a cybersecurity program and helps to provide assurance that controls are operating as intended and to detect controls that are not functioning correctly. VA has also been challenged in assuring that its actions to mitigate vulnerabilities and implement recommended improvements are effective. The department has asserted that it had implemented 39 of the 42 remaining open recommendations from our fiscal year 2016 reports. However, the evidence VA provided was insufficient to demonstrate that it had fully implemented the recommendations. The department subsequently provided additional evidence, which was also insufficient, indicating that its remedial action process was not validating the effectiveness of actions taken to resolve known deficiencies. Until VA adequately mitigates security control deficiencies, the sensitive data maintained on its systems will remain at increased risk of unauthorized modification and disclosure, and the systems will remain at risk of disruption. Fully Establishing Elements of a Cybersecurity Risk Management Program VA has been challenged in managing its cybersecurity risk. In July 2019, we reported that the department had fully met only one of the five foundational practices for establishing a cybersecurity risk management program. Although VA established the role of a cybersecurity risk executive, the department had not fully: developed a cybersecurity risk management strategy that addressed key elements, such as risk tolerance and risk mitigation strategies; documented risk-based policies that required the department to perform agency-wide risk assessments; conducted an agency-wide cybersecurity risk assessment to identify, assess, and manage potential enterprise risks; or established coordination between cybersecurity and enterprise risk management. VA concurred with our four recommendations to address these deficiencies and asserted that it is acting to do so. Nevertheless, until VA fully establishes a cybersecurity risk management program, its ability to convey acceptable limits regarding the selection and implementation of controls within the established organizational risk tolerance will be diminished. Identifying Critical Cybersecurity Staffing Needs VA has been challenged to accurately identify the work roles of its workforce positions that perform IT, cybersecurity, or cyber-related functions—a key step in identifying its critical cybersecurity staffing needs. In March 2019, we reported that the department had likely miscategorized the work roles of many of these positions in its personnel system. Specifically, VA had reported that 3,008 (or 45 percent) of its 6,636 positions in the 2210 IT management occupational series— positions that most likely performed IT, cybersecurity, and cyber-related functions—were not performing these functions. VA concurred with our recommendation to review the work roles for positions in the 2210 IT management occupational series and assign the appropriate work roles, and stated that it had begun to do so. Nevertheless, until VA completely and accurately categorizes the work roles of its workforce positions performing IT, cybersecurity, and cyber- related functions, the reliability of the information needed to improve workforce planning will be diminished and its ability to effectively identify critical staffing needs will be impaired. Managing IT Supply Chain Risks as Part of IT Modernization Programs Assessing and managing supply chain risks are important considerations for agencies, including VA, when operating and modernizing IT systems. In July 2018, we reported that reliance on a global IT supply chain introduces risks to federal information systems. We noted that supply chain threats are present during various phases of a system’s development life cycle and we identified the following threats: Installation of malicious or intentionally harmful hardware or software; Installation of counterfeit hardware or software; Failure or disruption in the production or distribution of critical Reliance on a malicious or unqualified service provider; and Installation of hardware or software that contains unintentional vulnerabilities, such as defects in code that can be exploited. These threats can have a range of impacts, including allowing adversaries to take control of systems or decreasing the availability of materials or services needed to develop systems. Accordingly, agencies such as VA need to take appropriate measures to assess and manage IT supply chain risks as they operate and modernize their information systems. Failure to do so could result in data loss, modification, or exfiltration; loss of system availability; and a persistent negative impact on the agency’s mission. In summary, similar to other federal agencies, VA continues to be challenged in implementing an effective agency-wide program and controls for securing its information and information systems. As VA pursues efforts to modernize and secure its IT systems, it will need to successfully address multiple challenges in order to achieve effective outcomes. Chair Lee, Ranking Member Banks, and Members of the Subcommittee, this completes my written statement. I would be pleased to answer your questions. If you or your staff members have any questions concerning this testimony, please contact me at (202) 512-6244 or wilshuseng@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals who made key contributions to this testimony include Jeffrey Knott (Assistant Director), Di’Mond Spencer (Analyst-in-Charge), Chris Businsky, Nancy Glover, Franklin Jackson, and Daniel Swartz. Also contributing were Melina Asencio, Scott Pettis, and Zsaroq Powe. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "In providing health care and other benefits to veterans and their dependents, VA relies extensively on IT systems and networks to receive, process, and maintain sensitive data, including veterans' medical records and other personally identifiable information. Accordingly, effective security controls based on federal guidance and requirements are essential to ensure that VA's systems and information are adequately protected from loss, unauthorized disclosure, inadvertent or deliberate misuse, or improper modification, and are available when needed. For this testimony, GAO summarized the status of information security across the federal government and particularly at VA. It also discusses the security challenges that VA faces as it modernizes and secures its information systems. To develop this statement, GAO reviewed its prior reports and relevant Office of Management and Budget, IG, and agency reports. , detect , respond , and recover —established by the National Institute of Standards and Technology's cybersecurity framework. VA's ratings were generally consistent with the ratings of other major agencies (see figure) and its information security program was one of 18 agency programs that IGs deemed ineffective. Most major agencies, including VA, had significant security control deficiencies over their financial reporting. For example, for fiscal year 2018, VA's IG reported deficiencies in control areas, such as security management, access control, configuration management, segregation of duties, and contingency planning. Additionally, as of fiscal year 2018, VA reported meeting six of the 10 cybersecurity performance targets set by the administration. VA faces several security challenges as it secures and modernizes its information systems. These challenges pertain to effectively implementing information security controls; mitigating known vulnerabilities; establishing elements of its cybersecurity risk management program; and identifying critical cybersecurity staffing needs. VA also faces the additional challenge of managing IT supply chain risks as the department takes steps to modernize its information systems. In 2016, GAO recommended 74 actions for VA to take to address deficiencies and improve its cybersecurity program. However, as of October 2019, VA had not demonstrated that it had addressed 42 of these recommendations. In 2019, GAO made four additional recommendations to improve the department's cybersecurity risk management program and one recommendation to accurately identify work roles of IT and cybersecurity workforce positions. VA concurred with these recommendations and planned to implement them.", "document_type": "gao"}
{"report": "It is DOD policy that installations, property, and personnel shall be protected and that the authority of a DOD commander to take reasonably necessary and lawful measures to maintain law and order and to protect installation personnel and property includes the individuals’ removal from or denial of access to, an installation when those individuals threaten the orderly administration of the installation. The Under Secretary of Defense for Intelligence develops overall security policy, including requirements for the DOD Physical Security Program, and the secretaries of the military departments and heads of DOD components establish policies and procedures to implement the Under Secretary’s policies. Individuals may seek unescorted, escorted, or trusted traveler access to DOD installations. As previously mentioned, this report focuses on individuals seeking unescorted access. Unescorted installation access requires, with limited exceptions, individuals seeking access to establish their identity, be determined fit for access, and establish an acceptable purpose for their presence on the installation. DOD components’ security forces establish the identity of individuals at authorized installation control points by using identification credentials, specifically a DOD-issued common access card or other credentials listed in DOD guidance. DOD’s Identity Matching Engine for Security and Analysis (IMESA), which is maintained by the Under Secretary of Defense for Personnel and Readiness, helps security forces make current fitness-for-access determinations for installations that have PACS that connect to IMESA. IMESA electronically links PACS to federal government (including DOD’s) and local population databases to verify information contained in individuals’ credentials and to search for derogatory information. IMESA continuously vets individuals for fitness- for-access determinations against these authoritative government databases every 24 hours. If derogatory information is found, IMESA is to send an alert to the PACS so that security forces can take appropriate action if and when those individuals next seek access to installations. Individuals without a common access card or another acceptable credential who seek access to installations with PACS are sent through the installations’ visitor control process where security forces are to (1) authenticate the individuals’ identity, (2) establish an acceptable purpose for their presence on the installations, and (3) make fitness-for-access determinations using any derogatory information from authoritative government databases. These databases could include those accessible through IMESA, where available and as applicable. Figure 1 illustrates the process for gaining unescorted access to installations with PACS that connect to IMESA—both for individuals with and without acceptable credentials. DOD components have fielded the following types of PACS at their domestic installations: DBIDS. DMDC developed DBIDS and it is used by the Air Force, the Navy, the Marine Corps, and DLA to control access to their respective installations. DBIDS consists of hardware and software—specifically, computers, servers, badge printers, and handheld identification devices. DBIDS has the capability to electronically connect to authoritative government databases using IMESA. AIE. The Army developed AIE to control access to its installations. AIE consists of hardware and software—specifically, computers, servers, badge printers, and handheld identification devices. AIE also includes additional hardware such as gate arms and automated pedestals where individuals can scan their own credentials. AIE has the capability to electronically connect to authoritative government databases using IMESA. RAPIDGate. RAPIDGate is a legacy system that according to DMDC officials is no longer being fielded to DOD installations and, according to Army officials, as of October 2018 was in use at only four domestic Army installations. RAPIDGate does not have the capability to electronically connect to authoritative government databases. Deployed by DOD in 2014, IMESA verifies enrolled individuals’ information against (1) DOD’s Defense Enrollment Eligibility Reporting System to determine if the credentials have been revoked; (2) the Federal Bureau of Investigation’s National Crime Information Center’s Wanted Persons file to determine if there are records on the individuals for an outstanding felony warrant; (3) the Federal Bureau of Investigation’s Terrorist Screening Database to determine if the individuals are known or suspected terrorists; and (4) the local population database, according to an OUSD(I) official, to determine if credentials issued by installations have been revoked or have expired. Individuals with enrollable credentials are enrolled in IMESA when their credentials are scanned by PACS for the first time. According to DMDC officials, once individuals are enrolled, IMESA continuously vets them against these authoritative government databases every 24 hours and it takes approximately 2 seconds for each individual’s credential to be vetted through IMESA. Figure 2 illustrates the process of using PACS to electronically connect to IMESA to validate individuals’ identity and continuously vet individuals’ fitness for access to DOD installations. The Under Secretary of Defense for Intelligence is responsible for establishing department-wide physical access control standards, procedures, and guidance, consistent with DOD guidance and applicable laws, to include developing processes for establishing the identity of individuals seeking access to installations. The Under Secretary of Defense for Personnel and Readiness is responsible for designing and maintaining IMESA, and establishing and executing a plan to integrate IMESA with PACS at all DOD installations. DMDC is a center within the Office of the Under Secretary of Defense for Personnel and Readiness that provides identity management services and oversees the fielding and maintenance of DBIDS. DOD components issue their own component- and installation-specific requirements for physical access control. These include physical access barrier requirements such as fences, as well as the use of PACS. Each DOD component has designated a program manager to supervise and oversee its physical security program, to include PACS. According to DOD component guidance and officials: The Army Acquisition Corps, Product Manager for Force Protection Systems, is responsible for the procurement and fielding for the Army’s PACS. The Army Office of the Provost Marshal General develops PACS requirements based on DOD and Army policies for the Army’s physical security program. The Commander Navy Installations Command is responsible for the Navy’s PACS. The Air Force Security Forces Center is responsible for the Air Force’s PACS. The Office of the Deputy Chief of Staff for Logistics, Engineering, and Force Protection, Directorate of Security Forces, is responsible for developing service-wide access control policies. The Commander, Marine Corps Installations Command, is responsible for the Marine Corps’ PACS. The Deputy Commandant, Plans, Policies, and Operations establishes policies, sets requirements, and is responsible for the Marine Corps’ Physical Security Program. DLA Information Operations and Installation Support Security and Emergency Services Staff Directors share responsibility for the DLA’s PACS. Additionally, DOD component installation commanders are responsible for the physical security of their installations, including for the use of PACS. OUSD(I) issued a physical security manual in January 2019 that addresses minimum department-wide standards for access to DOD installations. The manual incorporates and cancels Directive-Type Memorandum 09-012, the interim policy for DOD physical access control that was in effect for about 9 years. The manual directs DOD components to, among other things, implement procedures for all populations to gain access to component installations; field electronic PACS at all DOD installations; and fund the continued operation, maintenance, and enhancement of IMESA with additional government data sources. The manual also states that new electronic PACS and existing electronic PACS undergoing significant upgrades (valued at more than 50 percent of replacement cost) must interface with IMESA. Each DOD component had also issued guidance on installation physical access control standards that pre-date the January 2019 physical security manual. For example, DLA Manual 5200.08 Volume 1 identifies DBIDS as DLA’s PACS and requires certain installation commanders to incorporate and maximize the use of electronic credential authentication. In another example, Army Regulation 190-13 assigns installation commanders responsibility for implementing AIE, when available, and states that deviations from the Army AIE standards and specifications are not authorized without written approval from Army headquarters. DOD component officials said that they will update their guidance to incorporate the DOD installation access control standards contained in OUSD(I)’s 2019 physical security manual. To implement these department-wide access control standards, according to OUSD(I) and DOD component officials, each DOD component has fielded or plans to field PACS that connect to IMESA at all their domestic installations. According to DOD component officials, as of February 2019, the Air Force, the Navy, the Marine Corps, and DLA have fielded DBIDS at all of their domestic installations. Specifically, according to DOD component officials, DBIDS is fielded at: 67 Air Force installations 16 Marine Corps installations According to Army officials, as of February 2019, AIE was fielded at 35 of the Army’s domestic installations. The officials stated that the Army currently plans to field AIE at an additional 60 installations by September 2019, and at all of its remaining domestic installations by the end of fiscal year 2021. However, Army officials told us that, at the direction of the Secretary of the Army, AIE is undergoing additional testing and assessment to inform a comparison with DBIDS. The Secretary of the Army is expected to make a decision sometime in summer 2019 on which PACS to field at remaining Army installations. DMDC plans to enhance IMESA’s capabilities to allow for increased information sharing and vetting, and to expand the type of credentials that DBIDS can scan. Specifically, the Under Secretary of Defense for Intelligence has identified additional authoritative government databases that IMESA will connect with to access derogatory information. For example, the Under Secretary of Defense for Intelligence directed the secretaries of the military departments to develop a plan to vet individuals seeking unescorted access to domestic installations for disqualifying derogatory information in additional files within the National Crime Information Center’s database and the Interstate Identification Index by September 30, 2019. According to an OUSD(I) official, IMESA will be able to access two additional National Crime Information Center files by 2020: the National Sexual Offender Registry File and the Violent Persons File. The official also stated that there are plans to connect IMESA to DOD’s Automated Biometric Identification System by 2020. DMDC plans to expand the types of credentials that DBIDS can scan, to include all credentials listed in DOD’s 2019 physical security manual. For example, according to DMDC officials, scheduled enhancements to DBIDS will enable security forces to scan cards and driver’s licenses compliant with the REAL ID Act of 2005 by the end of fiscal year 2019. Moreover, according to DMDC officials, this enhancement will eliminate the time and expense to annually issue and print hundreds of thousands of temporary DBIDS credentials. The officials also stated that DMDC has plans to enable DBIDS handheld devices to read military veterans’ health identification cards, although no time frame for implementation has been set. Army Office of the Provost Marshal General officials told us that AIE can already scan identification cards and driver’s licenses compliant with the REAL ID Act. This capability allows individuals with these credentials to be vetted and enrolled in IMESA in the access control lane without having to go the visitor control center. According to Army officials, this “in-lane” initial vetting and IMESA enrollment takes approximately 30 seconds by checking the National Crime Information Center database and Interstate Identification Index for criminal history and active warrants. Further, these officials told us that the Army has also identified future enhancements to AIE, such as transitioning to a cloud-based version. The officials told us that a cloud-based version of AIE will allow for quicker and more cost- effective fielding because of fewer installation prerequisites and reduced computer hardware requirements. Army officials are also considering other enhancements, such as self-service kiosks and web-based registration options, to streamline and expedite initial visit registrations. The Air Force and DLA monitor their installations’ use of PACS and the Army, the Navy, and the Marine Corps do not. As a part of our work, we conducted numerous site visits to domestic installations to observe the DOD components’ use of PACS, but details concerning our findings associated with these visits are omitted because the information was deemed sensitive by DOD. Air Force and DLA officials stated they routinely collect data on PACS use and the number of credentials scanned at their installations and provide those data to their leadership. Additionally, the Air Force is using these data to brief installation commanders on the risks associated with not using DBIDS at their installations. Army, Navy and Marine Corps officials stated they do not monitor PACS use at their installations because there is not a requirement to do so. Our review of DOD guidance also found no such requirement. DOD component officials emphasized the importance of installation commanders having discretion to make risk-based decisions regarding access control in general, and in deciding when or when not to use PACS. Nevertheless, OUSD(I), Army, Navy, and Marine Corps officials agreed that monitoring installations’ use of PACS would be beneficial and could be readily accomplished without significant cost using existing technology. For example, Army, Navy, and Marine Corps officials stated that their installations could collect monthly scanning data using existing PACS reporting mechanisms to identify below average use and determine if actions are needed to increase use. One OUSD(I) official further stated that, depending on the extent to which installations are not using PACS, changes to guidance might be warranted to require monitoring of the use of PACS. DOD Instruction 5010.40, Managers’ Internal Control Program Procedures directs the Office of the Secretary of Defense and DOD component heads to implement a comprehensive system of internal controls that provides reasonable assurance that programs are operating as intended and to periodically evaluate the effectiveness of those controls. Furthermore, Standards for Internal Control in the Federal Government for performing monitoring activities states that management should monitor and evaluate the results of its internal control systems by obtaining relevant data on a timely basis, and determine appropriate control actions for any identified deficiencies. Because the Army, the Navy, and the Marine Corps do not monitor the use of PACS and because OUSD(I) does not require that they do so, those military services do not know the extent to which PACS are being used at more than 100 installations. Consequently, the military services do not have the data they need to evaluate the effectiveness of PACS and inform risk-based decisions to safeguard personnel and mission– critical, high-value installation assets. Demonstrating the importance of using PACS that connect to IMESA, we note that, according to DMDC, IMESA has identified more than 42,000 instances of individuals who were granted access to a DOD installation and were subsequently issued a felony warrant. Installation security forces call the DMDC helpdesk for assistance in resolving DBIDS technical issues. According to DMDC officials, this helpdesk handles technical issues for more than 100 DMDC applications and programs, including DBIDS, and is staffed 24 hours a day, 7 days a week. DMDC helpdesk staff classify DBIDS technical issues into one of three tiers, based on complexity and the estimated time to resolve an issue. According to DMDC officials, tier I issues tend to be the least complex and typically take the least time to resolve, whereas tier III issues tend to be the most complex and typically take the longest time to resolve. Tier II issues fall between tier I and tier III issues with respect to complexity and anticipated resolution time. Below are examples of issues that are experienced in each tier: Tier I. Unresponsive computer screens, passwords that need to be reset, and relatively simple network printer issues. Tier II. Handheld device battery charging issues, network synchronization issues, and problems installing fingerprint readers. Tier III. Handheld devices not connecting to servers, locked user accounts, and equipment that needs to be replaced. According to DMDC officials, all calls to the helpdesk are initially handled by a tier I customer service representative. The tier I representative triages the issue using DBIDS reference materials, and if he or she is unable to resolve the issue it is passed to a tier II customer service representative. If the tier II representative is unable to resolve the issue using DBIDS reference materials, then, with a supervisor’s review and approval, the call is transferred to the tier III group. The issue is then assigned to either the tier III hardware group or the tier III software/application group, depending on the nature of the technical issue. According to DMDC officials, the tier III hardware group is located in Ashburn, Virginia, and the tier III software/application group is located at DMDC’s offices in Seaside, California. The Army also has instituted a tiered approach for resolving AIE technical issues through its helpdesk. The AIE helpdesk is also staffed 24 hours a day, 7 days a week. Similar to DBIDS, the Army classifies AIE technical issues into one of three tiers, based on complexity and time to resolve. According to Army officials, all Army installation security forces’ calls to the helpdesk are initially handled by a tier I customer service representative who tries to resolve the issue using AIE reference materials. If the tier I representative is unable to resolve the issue, the issue is passed to a tier II field service representative. The field service representative is expected to contact the installation within 24 hours and attempt to resolve the issue by email or phone. If the field service representative is unable to resolve the issue remotely, the representative will make an in-person service visit to attempt to resolve the issue. If the issue cannot be resolved, then the customer service representative classifies the issue as tier III and transfers the issue to AIE system engineers for resolution. According to Army officials, tier III issues are usually Army-wide issues, such as problems associated with software updates. DMDC has collected data on DBIDS technical issues; however, DMDC has not been able to assess its performance due to a lack of performance measures and associated goals. Table 1 shows the number of DBIDS technical issues and the average time it took to resolve them, by tier, from January 2016 through July 2018. Specific details regarding the number of issues and the resolution time were omitted because the information was deemed sensitive by DOD. The Army collects data on AIE technical issues and has developed performance measures and associated goals to assess AIE performance. Specifically, the AIE Reliability Analytics Model tracks real-time information on operational availability with a goal of 100 percent, the number and age of open helpdesk tickets with a goal of resolving tier II issues within 48 hours, and field service representative performance with a goal of a 100 percent closure rate for tier II issues. According to Army officials, the Army is currently developing specific targets for its tier I and tier III technical issues. The Army has used data on AIE technical issues to improve AIE performance. For example, due to the age and number of tickets, the Army analyzed 646 AIE helpdesk tickets generated from October 2017 through February 2018 and determined that the root causes of the most prevalent technical issues were site server and handheld device failures. As a result of its analysis, the Army implemented an AIE software update and has begun fielding a more reliable brand of handheld device to installation security forces. According to Army officials, AIE operational availability has increased and technical issues are resolved more quickly since the AIE Reliability Analytics Model came online in September 2017. For example, from September 2017 through August 2018, AIE’s operational availability increased from 93 percent to 98 percent and the average ticket age for all tiers decreased by 33 percent. Increased AIE operational availability allows for increased continuous vetting of individuals seeking access to Army installations. Army officials at all levels have access to the model, and the Army Product Manager for Force Protection Systems sends weekly emails to Army leadership highlighting AIE performance achievements and challenges. We have previously reported, that by tracking performance and developing performance measures, agencies can better evaluate whether they are making progress and achieving their goals. Further, to fully address challenges agencies must be able to demonstrate progress achieved through corrective actions, which is possible through the reporting of performance measures. Characteristics of effective performance measures include having baseline or trend data, setting measurable program goals, and establishing time frames for achieving goals. Program goals communicate what results the agency seeks and allow agencies to assess or demonstrate the degree to which those desired results are achieved. Both performance measures and goals give managers crucial information to identify gaps in program performance and plan any needed improvements. Although user agreements between DMDC and the DOD components state that DMDC will provide helpdesk and maintenance support, the agreements do not include performance measures and associated goals regarding DBIDS’ operational availability and the timely resolution of technical issues. DMDC officials acknowledged that performance measures and associated goals would likely reduce the time it takes to resolve DBIDS technical issues, particularly for tier II and tier III issues. However, until DMDC develops performance measures and goals, its ability to systematically address the underlying issues negatively affecting DBIDS’ operational availability is hindered. Although according to DOD officials DOD has fielded or plans to field PACS that connect to IMESA at all domestic installations, only the Air Force and DLA have monitored PACS use at their installations. The Army, the Navy, and the Marine Corps at more than 100 installations have not monitored the use of PACs because, as stated by officials, there is not a requirement to do so. As a result, these components do not have the data necessary to evaluate PACS effectiveness and inform risk-based decisions regarding PACS use to safeguard personnel and mission- critical, high-value installation assets. Further, DOD component and installation officials told us about their dissatisfaction with the time it takes to resolve DBIDS’ technical issues. Although the Army has developed performance measures and associated goals for its helpdesk that have improved the ability to resolve technical issues and overall AIE operational availability, DMDC has not. Without such performance measures and associated goals, DMDC is unable to systematically evaluate how well DBIDS is performing and address underlying issues negatively affecting DBIDS’ operational availability. We are making the following five recommendations to the Department of Defense: The Secretary of Defense should ensure that the Under Secretary of Defense for Intelligence requires that DOD components (including the military departments and DLA) monitor the use of PACS at their installations. (Recommendation 1) The Secretary of the Army should ensure that the Office of Provost Marshal General monitors the use of PACS at Army installations. (Recommendation 2) The Secretary of the Navy should ensure that the Commander, Navy Installations Command, monitors the use of PACS at Navy installations. (Recommendation 3) The Secretary of the Navy, in coordination with the Commandant of the Marine Corps, should ensure that the Commander, Marine Corps Installations Command, monitors the use of PACS at Marine Corps installations. (Recommendation 4) The Secretary of Defense should ensure that the Under Secretary of Defense for Personnel and Readiness develops appropriate performance measures and associated goals for the timely resolution of DBIDS technical issues to facilitate improved PACS performance. (Recommendation 5) We provided a draft of this report to DOD for comment. In its written comments, reproduced in appendix II, DOD concurred with our five recommendations and identified actions that it was taking or planned to take to implement our recommendations. Regarding our second recommendation, DOD concurred with that recommendation to monitor the use of PACS at Army installations, and on the basis of the department’s written comments we modified the recommendation to indicate that the Army Office of the Provost Marshal General is responsible for monitoring the use of PACS at Army installations. DOD also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Under Secretary of Defense for Intelligence. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In this report we (1) describe actions the Department of Defense (DOD) has taken to develop guidance on physical access to domestic installations and to field physical access control systems (PACS) at these installations, (2) evaluate the extent to which DOD components have monitored the use of fielded PACS at these installations, and (3) evaluate the extent to which DOD has implemented an approach for addressing PACS technical issues and assessing associated performance. This report is a public version of a sensitive report that we issued on May 31, 2019. The sensitive report included an objective related to the extent to which security forces at various DOD domestic installations used fielded PACS. DOD deemed a significant portion of the information related to this objective to be sensitive, necessitating protection from public disclosure. This public report omits information related to our observations of PACS use at these installations and the risks associated with not using PACS. As a result of this omission, we updated the wording of the second objective to focus on DOD components’ efforts to monitor the use of fielded PACS at installations. Although the second objective and the information associated with it in this public report is more limited, we relied on the same methodology to support our findings and the excluded information does not impact our recommendations. The first and third objectives in this report are the same as in the sensitive report and use the same methodology as in the sensitive report. DOD deemed some of the detailed information presented in conjunction with the third objective to be sensitive, necessitating protection from public disclosure. As a result, this public report omits specific details regarding the technical issues of PACs. This report focuses on physical access controls at authorized access control points at DOD’s domestic installations that are owned and operated by the Army, the Navy, the Air Force, the Marine Corps, and the Defense Logistics Agency (DLA). We did not consider actions DOD has taken to prevent unauthorized access to its domestic installations by means such as tunneling under or climbing over perimeter barriers. For objective one, we analyzed key Office of the Under Secretary of Defense for Intelligence (OUSD(I)) and DOD component policies outlining physical access control requirements. The key guidance documents we analyzed are listed in table 2. Additionally, we interviewed officials from OUSD(I), the Joint Staff, each of the DOD components, and the U.S. Northern Command to discuss the guidance documents and any efforts to update, revise, or draft new guidance on the use of installation PACS. We also reviewed DOD component documentation and interviewed OUSD(I) and DOD component officials to determine the extent to which PACS was fielded at domestic installations and to identify ongoing efforts to field PACS at additional domestic installations. Finally, we interviewed DOD officials to identify any planned future enhancements to PACS and the Identify Matching Engine for Security and Analysis (IMESA). For our second objective, we focused on individuals seeking unescorted access to DOD domestic installations. We reviewed and analyzed OUSD(I), DOD component, and installation-specific guidance on the use and monitoring of PACS. We conducted site visits to six domestic installations to meet with installation command and security force officials to discuss their experiences using PACS and to observe their use of PACS. We then compared the guidance and our observations with Standards for Internal Control in the Federal Government for monitoring activities, which states that management should obtain data on a timely basis so that they can be used for effective monitoring. Although findings from these six installations are not generalizable to all DOD domestic installations, they are illustrative of how PACS are used, and more generally, how installation access is controlled. In selecting the six installations to visit we considered installation ownership to ensure that we included an installation from each DOD component, geographic proximity among installations, and the type of PACS used by the installation. We also visited an installation where no PACS was installed. We limited our site selection to active-duty installations in the continental United States. Based on this methodology we visited Fort Stewart, Georgia; Moody Air Force Base, Georgia; Naval Station Mayport, Florida; Marine Corps Support Facility Blount Island, Florida; Tobyhanna Army Depot, Pennsylvania; and DLA Distribution Center Susquehanna, Pennsylvania. For our third objective, we reviewed DOD user agreements to determine the support agreement terms, requirements, and responsibilities for addressing PACS technical issues. We analyzed DOD component data on the number and type of Defense Biometric Identification System (DBIDS) helpdesk technical issues reported from January 2016 through July 2018, and compared the data with provisions in the user agreements that discuss the PACS helpdesk. We also compared the steps the Army and DMDC have taken or planned to address helpdesk technical issues with Standards for Internal Control in the Federal Government for developing performance measures, which states that management should establish performance measures and indicators. We interviewed officials from DOD components and the installations we visited to discuss their experiences with PACS helpdesks, and their views on the performance and reliability of PACS. We assessed the reliability of the helpdesk technical issue data by interviewing knowledgeable officials about the data and by testing the raw data to determine the accuracy of the summary data provided by DOD. Additionally, we collected and analyzed the raw data to determine whether calculations were made correctly. We determined that the data were sufficiently reliable for our understanding the number and types of PACS technical issues. To address our three reporting objectives, we met with officials from the DOD organizations listed in table 3. We conducted this performance audit from February 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with DOD from July 2019 to August 2019 to prepare this public version of the original sensitive report. This public version was also prepared in accordance with these standards. In addition to the contact name above, GAO staff who made key contributions on this report include Brian Lepore, Director (retired); Jason Bair, Acting Director; Marc Schwartz, Assistant Director; Shawn Arbogast, Analyst-in-Charge; Jamilah Moon; Richard Hung; Mae Jones; Amie Lesser; Serena Lo; Amber Lopez Roberts; and Carter Stevens.", "summary": "In November 2009, an Army officer killed or wounded 45 people at Fort Hood, Texas; 4 years later in September 2013, a Navy contractor killed or wounded 16 people at the Washington Navy Yard in Washington, D.C. Independent reviews conducted in the aftermath of these shootings identified physical access control weaknesses at DOD installations. The conference report accompanying the National Defense Authorization Act for Fiscal Year 2018 contained a provision for GAO to assess DOD's installation access control efforts. GAO (1) described actions DOD has taken to develop guidance on physical access to domestic installations and to field PACS at these installations, (2) evaluated the extent to which DOD has monitored the use of fielded PACS at these installations, and (3) evaluated the extent to which DOD has implemented an approach for addressing PACS technical issues and assessing associated performance. GAO analyzed DOD guidance on physical access control requirements, and visited installations to discuss with installation command and security force officials their experiences using PACS. This is a public version of a sensitive report that GAO issued in May 2019. Information that DOD deemed sensitive has been omitted. The Department of Defense (DOD) has issued guidance on accessing its domestic installations and strengthening physical access control systems (PACS)—used to scan credentials to authenticate the identity and authorize individuals to access DOD installations. Specifically, DOD has recently issued guidance directing the fielding of PACS and has fielded or plans to field such systems at domestic installations. The Defense Manpower Data Center (DMDC) developed the PACS used by the Air Force, the Navy, the Marine Corps, and the Defense Logistics Agency. The Army developed its own PACS. Both types of PACS electronically connect to DOD's Identity Matching Engine for Security and Analysis (IMESA). IMESA accesses authoritative government databases to determine an individual's fitness for access (i.e., whether an individual is likely a risk to an installation or its occupants), and continually vets this fitness for subsequent visits (see fig.). The Air Force and DLA have monitored their installations' use of PACS, but the Army, the Navy, and the Marine Corps have not. Army, Navy, and Marine Corps installation officials stated that they do not monitor PACS use at their installations because there is no requirement to do so. Because the Army, the Navy, and the Marine Corps do not monitor PACS use and DOD does not require that they do so, those military services do not have the data they need to evaluate the effectiveness of PACS and make informed risk-based decisions to safeguard personnel and mission-critical, high-value installation assets. DOD, Army, Navy, and Marine Corps officials agreed that monitoring installations' use of PACS would be beneficial and could be readily accomplished without significant cost using existing technology. The Army and DMDC have used a tiered approach and established helpdesks to address PACS technical issues. The Army has established performance measures and goals to assess its approach, which has improved the ability to resolve technical issues. DMDC, however, does not have performance measures and goals, and thus lacks the information needed to evaluate its PACS' performance and address issues negatively affecting operational availability. GAO made five recommendations, including that DOD monitor installations' use of PACS and develop appropriate performance measures and goals for resolving technical issues to improve PACS performance. DOD concurred with GAO's recommendations.", "document_type": "gao"}
{"report": "The F-35 Lighting II program is a joint, multinational acquisition program intended to develop and field a family of next-generation strike fighter aircraft for the U.S. Air Force, Navy, and Marine Corps (hereinafter referred to as the services); seven international partners; and four foreign military sales customers (collectively hereinafter referred to as program participants). The program has developed and is delivering three variants of the F-35 aircraft: F-35A conventional takeoff and landing variant for the Air Force. (see fig. 1) F-35B short takeoff and vertical landing variant for the Marine Corps. F-35C carrier-suitable variant for the Navy. The characteristics of the services’ variants are similar in that each is intended to be a multi-role, stealthy strike aircraft, but each service’s variant also has unique operating requirements. For example, the Marine Corps requires that the F-35B be capable of operating from aircraft carriers, amphibious ships, and main and austere operating bases alike, requiring the ability to conduct short take offs and vertical landings. DOD initiated the F-35 program in October 2001. Since then, the Marine Corps and Air Force declared initial operational capability in 2015 and 2016, respectively, while the Navy declared initial operational capability in February 2019. Operational testing of the F-35 aircraft began in December 2018 and is currently scheduled to be completed late 2020. At that time, DOD will make a decision on whether to proceed with plans to begin full-rate production of the aircraft. DOD has, concurrently, been fielding and operating a growing fleet of aircraft as part of low-rate initial production. As of October 2019, more than 435 U.S. and international aircraft had been fielded and were operating from 19 sites worldwide. By 2023, the global F-35 fleet is expected to expand to more than 1,100 aircraft across 43 operational sites. In total, the program participants plan to purchase more than 3,300 F-35 aircraft, with the U.S. services planning to purchase nearly 2,500 of those aircraft. See Figure 2 for a timeline of anticipated worldwide fleet growth in the F-35 program. DOD has two primary contractors for the F-35 program: Lockheed Martin for the overall aircraft system and Pratt & Whitney for the engine. As the prime contractor for the overall aircraft system, Lockheed Martin (hereinafter referred to as the prime contractor) is responsible for managing the F-35 supply chain, depot maintenance, and pilot and maintainer training, as well as for providing engineering and technical support. Currently, DOD is contracting for this support with the prime contractor largely through annual contracts. It plans to transition to multiple-year, fixed-price, performance-based sustainment contracts when the program achieves certain condition-based criteria, including the establishment of critical sustainment capabilities and the government’s ability to collect and more fully assess performance and cost data. In addition, the U.S. Air Force, Navy, and Marine Corps have each established an F-35 integration office or similar construct focused on how the services will operate and afford the F-35, among other things. Figure 3 depicts how these key stakeholders provide support to the F-35 program participants across the three aircraft variants. The Autonomic Logistics Information System (ALIS) is a system of systems that serves as the primary logistics tool to support F-35 operations, mission planning, and sustainment. ALIS is intended to help maintainers manage tasks including aircraft health and diagnostics, supply-chain management, and other maintenance events. ALIS functionality is intended to support many of the F-35 program’s key performance parameters such as: Increase sortie generation rate: Number of aircraft sorties launched in a flight day. Increase mission reliability: The probability that a system will perform mission essential functions for a period of time. Reduce logistics footprint: The size of in-theater logistics support needed to move and sustain a warfighting force. The footprint includes all the necessary support needed to maintain the force such as fuels, parts, support equipment, transportation, and people. According to DOD officials, ALIS is integral to supporting F-35 operations. Figure 4 shows some of the key intended capabilities of ALIS. These capabilities reside in multiple software applications within the system that perform specific functions for maintainers, pilots, supply personnel, and data analysts. Lockheed Martin is the prime contractor for ALIS and has been responsible for developing and managing the capabilities of the system, as well as developing training materials for F-35 pilots, maintainers, and supply personnel. ALIS is co-located with F-35 aircraft both at U.S. military installations and in theater to support missions and assist with maintenance and resource allocation. ALIS consists of the overarching system, the applications housed within it, and the network infrastructure required to provide global integrated and autonomic support of the F-35 fleet. It comprises both hardware and software, and supports the flow of unclassified and classified aircraft-related data. As a system of systems, major components of ALIS consist of: The Autonomic Logistics Operating Unit (ALOU). The ALOU is the central computer unit that all F-35 data are sent through. As part of the unit, the ALOU consists of two servers that process and store classified and unclassified data respectively. There is only one ALOU, and it is owned by the prime contractor. The Central Point of Entry (CPE). The CPE is a server unit configured to provide software and data distribution for a country’s entire F-35 fleet. It is the node between the ALOU and each country’s Standard Operating Units (generally housed at F-35 installations). The CPE consists of two servers that process and store classified and unclassified data respectively. There is typically one operational CPE per country, although the United States has separate CPEs for its operational commands and training sites. The Standard Operating Unit (SOU). The SOU is a server that is intended to provide all ALIS capabilities to support flying, maintenance, and training at F-35 installations. Typically, each F-35 squadron has at least one SOU. It is the node local to each F-35 squadron. There are two types of SOUs: a classified SOU that supports the flow of classified aircraft-related data and an unclassified SOU that supports the flow of unclassified aircraft-related data. The Portable Memory Device (PMD). The PMD is informally referred to as the “brick” that F-35 pilots use to upload information such as mission planning data. F-35 personnel use the PMD to store mission and maintenance data generated during flight which may then be downloaded into the ALIS SOU to support maintenance and mission debrief activities. The Portable Memory Device Reader (PMD Reader). The PMD Reader is a device intended to be used to remove maintenance data, including health-related codes, off of the Portable Memory Device and load into the SOU. The Portable Maintenance Aid (PMA). The PMA is an unclassified ruggedized laptop used by F-35 maintainers and flight-line supervisors to view unclassified technical data, and perform and document maintenance activities. According to the F-35 program office, the purpose of the server construct is to support the exchange of information necessary to support the F-35 sustainment enterprise. As of September 2019, according to program officials, there was one operational ALOU and CPE within the United States. Each F-35 site in the United States has a varying number of SOUs depending on the site’s number of aircraft and squadrons. The SOU was designed to have its components fit into transit cases that can be carried by two personnel, with each case weighing up to 200 pounds. The PMDs, PMD Readers, and PMAs reside at the squadron and support the collection and transfer of unclassified and classified aircraft-related data. Figure 5 shows how unclassified ALIS data are collected and transferred from component to component. As we have previously reported, ALIS has experienced recurring developmental issues and schedule delays. The development of ALIS originated in 2002, a year after the start of the F-35 program. However, the first major ALIS release was not fielded until October 2009, nearly 7 years after initial development began. DOD officials had originally planned for the version of ALIS that would include all of the capabilities required to complete developmental testing of the program to be finalized in 2010. However, this milestone was reached in September 2018, nearly 8 years behind the original schedule. Figure 6 shows the timeline of major ALIS software version releases and other significant ALIS-related milestones. ALIS users from all 5 F-35 locations we visited reported that ALIS has improved in some aspects over the last 5 years. However, these users continue to report significant challenges with ALIS that are affecting the day-to-day operations of the aircraft. DOD is currently unable to assess the overall performance of ALIS because it has not developed performance metrics. Additionally, DOD is unaware of how challenges with ALIS are affecting F-35 fleet-wide readiness. According to pilots, maintainers, supply personnel, and contractors at 5 U.S. F-35 locations, ALIS is generally performing better than it was 5 years ago. Specifically, users at all 5 locations stated that data processing, downloading of information, and screen navigation were generally faster than previous years. According to users at 1 location, in previous releases of ALIS, it could take several minutes to complete a simple function like a screen download. Further, some users also reported minor functionality improvements within certain ALIS applications, such as the Computerized Maintenance Management System, leading to reduced time required to perform actions within those applications. We reported in April 2016 that ALIS users had problems accessing data in ALIS to produce service-specific reports for their squadrons. Users we spoke to at 4 locations for this report stated that they can now access some data within ALIS and can generate reports that they previously could not. For example, users at 1 location said that it was now easier to export aircraft-related maintenance information from ALIS and put it into an external spreadsheet. Additionally, in December 2015, the F-35 program began deploying software “fixes” to address minor defects in ALIS at F-35 locations in between major ALIS software version releases, which users at 1 location said have made improvements to the system. According to the F-35 program office, these software releases, referred to as service packs, have focused on improving user interface-related flaws that were discovered during major releases. Service packs provide users more frequent functionality fixes to the system, preventing them from having to wait, in most cases, over a year for a major ALIS software release. While users at all 5 F-35 locations we visited said that ALIS is performing better than it was 5 years ago, they also stated that the system still posed significant challenges to day-to-day F-35 operations. Specifically, users across the 5 locations we visited stated that seven significant challenges still exist with ALIS, as shown in table 1. Many of the challenges cited above are similar to those we reported in April 2016, including deployability, inefficient issue resolution process, and data inaccuracies. We recommended at that time that DOD develop a plan to prioritize and address ALIS issues. DOD concurred and in 2016 developed a plan that identified key areas for system modernization and sustainment, which included prioritizing issues related to ALIS. While DOD’s development of this plan is a positive step, significant user issues persist today, which are discussed in more detail below. Continued attention on ALIS is needed to make improvements to the system, reduce the burden on its users, and mitigate risks to operations and maintenance. Users at all 5 F-35 locations we visited expressed concern about data integrity issues related to inaccurate or missing data within ALIS. For example, users at all the locations said they have had consistent problems with data related to aircraft parts. Certain F-35 parts have an associated electronic record, which is used to track the remaining time before the part must be replaced, among other things. To be cleared for flight, F-35 policy states that an aircraft must be electronically “complete” in ALIS, meaning that all of the electronic records from each installed F- 35 part must be entered into ALIS. However, users at all 5 of the locations we visited told us that electronic records are frequently incorrect, corrupt, or missing, resulting in ALIS signaling that the aircraft should be grounded, often in cases where maintainers know that the parts have been correctly installed and are safe for flight. Users at 1 location said that within a 6-month period in 2019, they experienced anywhere between 0 and 400 issues per week related to inaccurate or missing electronic records. These same users said that it is common for their squadron leadership to elect to allow an aircraft to fly with over 20 inaccurate or missing electronic records that ALIS signals to ground. According to users at all 5 locations we visited, squadron leadership (e.g., DOD personnel designated by maintenance squadron commanders) may decide to fly an aircraft with inaccurate or missing electronic records, but we found that this practice varies by location and type of part. In June 2019, the Department of Defense Inspector General published a report on missing electronic records on F-35 spare parts. The report found that since 2015, F-35 locations have been consistently receiving spare parts without requisite electronic records. For example, of the 263 spare parts delivered to one location in June 2018, 213 spare parts (81 percent) did not have electronic records. Due in part to the unreliability of the data in ALIS, users at all 5 F-35 locations we visited have been collecting and tracking information outside of the system that should be automatically captured in ALIS. Although not a requirement, users said they need to track information outside of the system because they do not always trust the data that reside in ALIS. Users provided examples of critical aircraft data that they are tracking outside of ALIS—such as aircraft performance data and maintenance inspection deadlines—and said that manually tracking this information is a time-intensive process that pulls maintainers away from completing other aircraft maintenance-related responsibilities. For example, users at 1 location estimated that they spend an average of 5,000 to 10,000 hours per year manually tracking information that should be automatically and accurately captured within ALIS. In addition, there may be risks associated with using information tracked outside of the system of record to make decisions about the safety and operational health of aircraft. For example, users at one location said that there is a danger of overlooking a critical piece of information when key aircraft data used to determine an aircraft’s status must be tracked manually using Excel spreadsheets. Users also said that by continuously ignoring alerts in ALIS caused by missing or inaccurate data, squadrons could be at risk of ignoring an alert for a legitimate aircraft issue. Finally, one commander we spoke with said that while his policy is to generally require maintainers to resolve data issues before releasing an aircraft for flight, in a wartime scenario, his squadron will carry out missions with inaccurate or missing ALIS data and assume the subsequent risk that this may entail. Users at all 5 F-35 locations we visited cited challenges deploying with ALIS to forward locations. Users stated that the required hardware for ALIS is bulky, can be cumbersome to transport, and, when necessary, difficult to store on a ship. For example, the unclassified and classified Standard Operating Unit (SOU) servers that are required for collecting and analyzing aircraft data in ALIS are broken up into a series of transportable cases. These cases each weigh approximately 200 pounds and require at least two people to lift. Users from 1 location told us that they have taken several separate SOU-related cases to support ALIS on deployments. These servers, as shown in figure 7, require dedicated transportation to transport them to forward locations, and heavy-duty equipment to load them on and off of ships. Some users stated that it was challenging to find space on the ship to store these servers since they typically require an entire room to function, as well as specific power and environmental controls. Additionally, users at all 5 locations stated that limited internet connectivity can make deployments challenging. Although SOU servers are critical ALIS hardware components, due to their size, squadrons will not always take them on deployments. In these instances, internet connectivity is important to access critical aircraft data from the forward location and send it back to the squadron’s SOU for processing. However, internet connectivity can be slow or non-existent at these locations. In 2018, we recommended that the F-35 Program Executive Officer should test operating the F-35 disconnected from ALIS for extended periods of time in a variety of scenarios to assess the risks related to operating and sustaining the aircraft. DOD concurred with the recommendation, but as of December 2019, DOD had still not determined how long the aircraft can safely fly without connectivity to ALIS. Finally, users at 2 locations stated that contractor support is critical to supporting deployments. For example, at one location, due to inaccuracies with parts data in ALIS, the prime contractor prefers to match every requisite electronic record with its respective spare part prior to a deployment, which requires significant time and advanced planning. Furthermore, according to users at another location, due to the complexities and functionality issues related to ALIS, contractor support is required on deployments; however, deploying with contractors could become problematic in a combat scenario. Overall, users at all 5 locations said that they have completed deployments using ALIS. However, deployments are challenging and the current deployment preparation process for ALIS inhibits a military service’s ability to deploy on short notice. Users at 4 of 5 F-35 locations we visited stated that ALIS requires more contractor or military personnel support than originally planned. According to the F-35’s Operational Requirements Document—the document that outlines the overall requirements for the F-35 program— ALIS is supposed to help reduce the logistics footprint for the F-35. However, a 2013 DOD-commissioned study on reducing F-35 costs stated that the current ALIS support plan already uses 30 percent more administrators across squadrons and bases than a similarly-scaled IT implementation would normally require. In addition, current ALIS users at these 4 locations are finding that as ALIS becomes more mature, even more personnel are required to support the system’s operations. For example, according to users at 1 Air Force location, the Air Force currently relies on about 8 contractor employees to support each ALIS SOU server, but has determined that this is not sufficient. Users at 2 Air Force locations stated that until the Air Force can train more military personnel to support ALIS-related issues, they will need to increase the number of contractor employees per squadron to support F-35 operations. Further, users from 1 Air Force location said they have had to assign full- time “ALIS Expeditor” responsibilities to military personnel within the squadrons to keep track of ALIS-related issues and pressure the contractor for resolution. Since these roles are not official billets, their resulting responsibilities are adding to the military personnel’s existing, non-ALIS related responsibilities on the flight line. Air Force users from 1 location reported that due to inconsistencies within ALIS, they now have 20 full-time ALIS Expeditors to track ALIS-related issues and help ensure safety of flight for the aircraft. The Marine Corps had originally planned to maintain ALIS using only military personnel; however, as the numbers of aircraft and requisite SOUs increased, users at 1 Marine Corps location said that it was too difficult to develop and retain personnel with ALIS- specific expertise. According to these users, this has resulted in the Marine Corps needing increased numbers of contractor personnel to support its squadron operations. Users at all 5 F-35 locations we visited said that the process for resolving F-35 issues within ALIS remains problematic and inefficient. The Action Request (AR) process requires personnel to use an application within ALIS to submit an AR about any F-35 problem, including those about ALIS itself, to the contractor for triaging and ultimate resolution. In April 2016, we reported that ALIS users thought the AR process did not allow for the effective reporting and resolution of F-35 aircraft and ALIS issues. Specifically, users stated that the process did not provide transparency to all ARs submitted across F-35 locations and placed responsibility for resolving the requests primarily on the contractor. ALIS users at 4 locations stated that this remains the case. Users from 3 locations stated that the overall process would be more efficient if they were able to search ARs submitted by other squadrons across the fleet to determine if a solution to the problem already exists. Without this ability, users must submit an AR for every issue and wait for a response that can sometimes take months. For example, 1 location reported that from October 2018 through September 2019, F-35 aircraft were grounded for 9,262 hours or 9 percent of possible flight hours, due to unresolved ALIS- related ARs attributed mainly to missing and inaccurate electronic parts records. Officials from another location reported that during a 6-month period they had to ground aircraft for 2,200 hours as a result of waiting for contractors to resolve parts-related ARs. Users from a third location stated that more transparency in the AR process could reduce reliance on contractor support, provide a way to address F-35 problems more efficiently, and reduce costs to the program since DOD incurs a fee each time an AR is submitted. Users at all 5 F-35 locations we visited stated that ALIS is not user- friendly or intuitive. While users stated that there have been some limited improvements to ALIS over the past years, as previously discussed, in general, users at all 5 locations described ALIS applications as difficult to navigate. For example, users from 1 location stated that it is more difficult and time-consuming to search for information on parts in ALIS than in legacy logistics systems because the information is located in multiple locations within ALIS. Additionally, users from all 5 locations said that some of the applications within ALIS have very slow processing speeds. According to users at 1 location, in some instances, ALIS’s slow applications require maintainers to work additional hours to complete required maintenance tasks. During a demonstration of ALIS and its Joint Technical Data application at one of the locations we visited, we observed maintainers deal with a slow log-in process, problems filtering and searching for data in an application, and ultimately having the application freeze and kick them out. Figure 8 shows a maintainer using a PMA to work in ALIS. Users at all 5 F-35 locations we visited stated that the training and mission planning applications within ALIS remain immature. Users at all 5 locations said they are not using the Training Management System (TMS), an application designed for pilots and maintainers to track training qualifications and assign personnel to carry out specific tasks, for its intended purpose. Users from 4 locations said that because of the ongoing issues with TMS, they are using legacy systems in its place. For example, one Air Force command released a memorandum in January 2018 allowing some squadrons to use an external legacy system in place of the TMS application due to shortfalls in TMS functionality, which it stated had caused excessive work to execute normal operations and become an unacceptable burden. Marine Corps and Navy users from 2 locations we visited said that they are using other legacy systems to circumvent the TMS application as well. Additionally, pilots at 4 locations stated that the Off-Board Mission Support (OMS) application within ALIS is immature and remains non- intuitive, time consuming, and difficult to navigate. The OMS application is a key application for pilots to conduct mission planning and debriefing. Pilots at 2 locations said that they rely on contractors to help them complete tasks in the application. Users at all 5 F-35 locations we visited stated that training to learn how to use ALIS does not provide adequate knowledge or information to fully prepare users to operate the system. Specifically, users at 3 locations we visited stated that the training for ALIS does not reflect a realistic operational environment. Instead, users at all 5 locations stated that training materials are usually in the form of PowerPoint slides and that knowledge of ALIS and its functionality is primarily obtained at the squadron level through on-the-job-training. In April 2016, we reported that almost every user in the F-35-related focus groups we conducted at that time noted that they did not learn how to operate any ALIS applications until on-the-job training began on the flight line. Users stated that this remains true today. Users at 1 of the locations we visited stated that learning how to use ALIS in this manner has caused people to develop their own unique way of operating the system, which creates an F-35 fleet environment that is using its primary logistics tool in different ways. Although DOD and F-35 program officials agreed that ALIS continues to provide challenges for users and is generally not performing well, DOD still has not determined how it wants the system to perform. For example, officials from the Joint Strike Fighter Integrated Test Force told us that testing for individual ALIS software version releases focuses primarily on whether the new version is performing “better” than the previous version. Specifically, ALIS testers have developed criteria to determine if the newest version of ALIS is functioning more efficiently than the previous version by comparing such tasks as screen download times. However, according to these officials, these tests are not determining if the ALIS system is performing to a specified standard because DOD has not defined this standard. In September 2014, we recommended that DOD develop a performance- measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on the intended behavior of the system in actual operations and (2) tie system performance to user requirements. The DOD Systems Engineering Guide for Systems of Systems states that to fully understand performance of systems of systems (such as ALIS), it is important to have a set of metrics that assess the system’s performance and trace back to user requirements because the system will likely evolve based on incremental changes—similar to ALIS’s incremental fielding. These metrics should measure the intended behavior and performance of the system in actual operations versus the progress of the development of the system, allowing an assessment of system capabilities based on user requirements. After over 5 years, and more than 400 aircraft fielded, DOD has not yet established a performance-measurement process for ALIS. DOD concurred with our 2014 recommendation, and repeated its commitment to develop performance metrics for ALIS after the release of our 2016 report on ALIS risks. In September 2019 program officials told us that DOD remains in the process of developing these metrics and has no set timeline for their completion. Without a performance-measurement process, the F-35 program does not have critical information about ALIS performance across F-35 locations. Such information could help address current and future ALIS performance issues and systematically measure ALIS functionality compared to intended performance. Users at all 5 F-35 locations we visited also stated that problems with ALIS are affecting the overall readiness of the F-35 fleet; however, they were unable to tell us the degree to which this is the case. Overall F-35 fleet-wide performance has been falling short of warfighter requirements—that is, aircraft cannot perform as many missions or fly as often as required. Figure 9 shows F-35 fleet aircraft performance from October 2018 through September 2019. Full mission capability, or the percentage of time during which the aircraft can perform all of its tasked missions, was 31.6 percent across the fleet, as compared with the warfighter minimum target of 60 percent. Mission capability, or the percentage of time during which the aircraft can safely fly and perform at least one tasked mission, was 59.5 percent across the fleet, as compared with the warfighter minimum target of 75 percent. Furthermore, citing less than desirable aircraft performance, in September 2018, the Secretary of Defense directed the military services to achieve and maintain 80 percent mission capability rates for their critical aviation platforms, including the F- 35 fleet, by the end of fiscal year 2019. Two F-35 locations have started tracking information on how ALIS is affecting F-35 aircraft performance at their locations. Officials from one location told us that from October 2018 through September 2019, F-35 aircraft were grounded and thus non-mission capable for 16,221 hours, or 2 percent of possible flight hours, as a direct result of issues with ALIS— such as inaccurate or missing electronic records. However, according to officials at this location, this number does not capture all scenarios in which ALIS is affecting aircraft performance because sometimes squadron commanders make decisions to fly an aircraft when ALIS signals that they should not, in order to fulfill mission requirements. Officials from another location reported that in fiscal year 2018, ALIS- related issues caused the F-35 aircraft to be non-mission capable for 3,246 hours, or .5 percent of possible flight hours; however, as was the case with the previous location, officials said that this number also did not capture all scenarios in which ALIS is affecting aircraft performance. These limited efforts represent squadron-specific initiatives, as no other F-35 location has tracked similar ALIS-related data. Further, the data collected by the two locations only capture non-mission capability rates when ALIS signals to ground the aircraft and makes the aircraft incapable of completing a mission. The data do not account for the workarounds users said they are routinely performing to circumvent a non-functioning aspect of ALIS in order to get an aircraft ready to fly, or the times when squadron leadership decides to fly the aircraft when ALIS signals otherwise. Different factors can play a role in reducing F-35 aircraft readiness. For example, in April 2019, we reported that reduced aircraft performance was due largely to spare parts shortages. This conclusion was drawn from data that had been collected and tracked by both the contractor and DOD across the entire fleet to determine non-mission capability rates due to supply issues. Further, the F-35 program collects data on the degree to which maintenance issues are affecting F-35 mission capability. And, there are ongoing efforts to improve F-35 fleet readiness that are specifically targeted at supply and maintenance issues that are causing the significant mission-capability degradation. However, users and program officials stated that recurring issues with ALIS could also be affecting aircraft performance and noted that data on these issues are not being collected by the contractor or DOD. Although users reported multiple instances when ALIS-related issues grounded aircraft, these issues are being captured and categorized as either supply or maintenance-related issues, thus masking ALIS’s effect on fleet-wide readiness. DOD Instruction 5000.02T, “Operation of the Defense Acquisition System,” states that the program manager will use technical performance measures and metrics to assess program progress. It further states that the analysis of technical performance measures and metrics, in terms of progress against established plans, will provide insight into the technical progress and risk of a program like the F-35. In the case of ALIS, the F-35 program does not have a fleet-wide process for measuring, collecting, and tracking information on how ALIS is affecting the performance of the F-35 aircraft, such as fleet-wide mission capability rates. Without such a process, the F-35 program may be limited in its ability to identify all of the drivers of reduced aircraft performance and appropriate target solutions. Further, as we previously reported, DOD plans to enter into multi-year, performance-based F-35 sustainment contracts with the prime contractor, but may not be well positioned to enter into such contracts because, in part, it does not fully understand the technical characteristics of the aircraft. ALIS may or may not be having a notable effect on mission capability rates for the F-35 fleet. However, without understanding how or the extent to which ALIS is affecting the performance of the aircraft, DOD risks entering into long-term, performance-based logistics contracts without fully understanding all of the factors currently affecting aircraft operations. This could hinder DOD’s ability to effectively negotiate performance-related terms of the contract. Finally, without understanding how ALIS is affecting the performance of the aircraft, DOD risks developing a performance-measurement process for ALIS that is not tied to the overall performance goals of the program. DOD is taking actions to enhance the long-term viability of ALIS. Limited DOD attention on ALIS has resulted in a troubled history with the system. As a result, multiple efforts are currently underway to re-design and attempt to improve ALIS. However, key technical and programmatic uncertainties hinder these efforts. Furthermore, DOD does not have an overarching strategy for the future redesign of ALIS. As originally envisioned, ALIS was intended to be a first-of-its-kind, fully autonomic system that would provide users access to data on a range of capabilities—including operations, maintenance, prognostics, supply chain, customer support services, training, and technical data—in one logistics system to support aircraft operations. According to Joint Strike Fighter Integrated Test Force officials, previous DOD aircraft logistics systems were much simpler, not fully autonomic, and generally included data related to fewer major capabilities. However, the F-35 program office did not clearly specify what it required from ALIS from the warfighter’s perspective beyond the broad capabilities to be included in the system. Air Force officials stated that instead, the F- 35 program office relied on the prime contractor to take the lead in managing the development of the system. For example, the F-35 Operational Requirements Document provides only overarching, high- level requirements for ALIS and does not include specific, user-related requirements or requirements to adapt and modernize the system over time. DOD officials acknowledged that historically, DOD has prioritized other aspects of the F-35 program, such as the development of the airframe, over its logistics system. In addition, DOD’s focus with ALIS development over the last 5 years has largely centered on adding capabilities required to complete developmental testing for the F-35. As issues with the fielded system have arisen, DOD and the prime contractor’s approach has generally been to resolve these issues on a case-by-case basis as available resources allowed, as opposed to making more costly and time-intensive improvements to the system’s underlying design and functionality. DOD contracting officials and prime contractor representatives stated that the need to balance a limited number of software development personnel between efforts to stabilize the current system and add new features has negatively affected the development of ALIS. In a 2017 report, the Air Force Digital Service recommended that the F-35 program office cease adding new capabilities in order to re-evaluate ALIS-related design choices and improve software development processes and procedures. According to the report, many of the issues with ALIS have known root causes that are directly related to software and hardware design choices that are 15 years old. For example, ALIS is made up of siloed applications that each have their own, sometimes conflicting, databases. Further, according to the Air Force Digital Service report, efforts to upgrade ALIS from an out-of-date operating system have not been prioritized by the F- 35 program office. Finally, ALIS hardware is cumbersome, consisting of heavy servers as well as laptops that were originally designed in the mid- 1990s. The current approach to developing ALIS has generally led to scheduling delays and challenges addressing a backlog of ALIS deficiencies. For example, the ALIS version required to complete developmental testing for the F-35 was not released until 2018—8 years after the originally planned release date. F-35 program office officials emphasized that in general, the timeframe for releasing major software updates for ALIS—up to 18 months—has been long. Further, based on data from the prime contractor, as of September 2019, there were about 4,700 open ALIS deficiencies, which are used by the prime contractor to track and manage issues with the system. According to an F-35 program office official, ALIS deficiencies may be identified in the field by F-35 users, in the prime contractor’s testing laboratory, or during DOD-led developmental and operational testing of the F-35 and ALIS. Of these 4,700 deficiencies, about 34 percent were identified in 2017 or earlier and 22 percent were category 1 or category 2 deficiencies. Category 1 deficiencies are considered critical and could jeopardize safety, security, or another requirement; category 2 deficiencies are those that could impede or constrain successful mission accomplishment. As shown in figure 10, the total number of open deficiencies has generally increased over the last 2 years. In addition, the number of open category 1 through category 3 deficiencies, which are considered critical or have an adverse effect on mission accomplishment, generally increased during this period. While the rate at which the prime contractor closed deficiencies during this period increased, the rate of increase was generally lower than the rate at which new deficiencies were identified. Officials from the Joint Strike Fighter Integrated Test Force and Office of the Director of Operational Test and Evaluation expressed concerns about the number and nature of the ALIS-related deficiencies they have identified during developmental and operational testing. For example, F- 35 testers identified a number of deficiencies with the most recent ALIS software version, ALIS 3.5, including eight category 1 deficiencies. ALIS 3.5 is referred to as the “stabilization” release because it was intended to address longstanding issues with ALIS. In addition, F-35 testers stated that since 2016, they have identified a number of cyber-related ALIS deficiencies, most of which remain open today. While officials said that the number of cyber deficiencies is consistent with other DOD weapons systems, they stressed that a vulnerable ALIS is particularly problematic because of how interconnected the system is with the F-35 aircraft and its operations. DOD and the prime contractor have acknowledged ALIS’s troubled history and have established three initiatives to re-design and fix ALIS. At a November 2019 congressional hearing, the F-35 Program Executive Officer stressed that significant additional work is required to improve ALIS functionality and that this work cannot be done in old and outdated ways. Table 2 summarizes the three initiatives, led by the F-35 program office, Air Force, and prime contractor respectively. According to the F-35 program office, the three initiatives are complementary and will eventually be integrated in a final redesign of ALIS. However, we found that DOD lacks clarity on how it will address key technical and programmatic uncertainties about the future of the system (see figure 11). These uncertainties relate to complex aspects of ALIS that will significantly impact the future design of the system and how it will be managed. Further, there are divergent views among officials involved with the various initiatives in terms of how DOD should approach key aspects of the re-design, highlighting the uncertainty that exists about the future of ALIS. DOD has not fully determined what capabilities will be included in the ALIS re-design. After years of focusing on adding new capabilities with each major ALIS software version release, DOD officials agreed that that their current goal is to streamline and simplify ALIS. For example, the Mad Hatter initiative is designing applications based on the minimum capabilities required by maintainers to quickly release an aircraft for flight. Similarly, the ALIS Next initiative is working to optimize functions in ALIS by identifying aspects of the current design that could be slowing down the system—for example, transferring an aircraft’s entire digital history each time the jet is transferred from one SOU to another. However, officials from the Office of the Director of Operational Test and Evaluation indicated that there continues to be uncertainty about the capabilities— both classified and unclassified—that will be included in the re-design. Further, as discussed previously, the F-35 program office has not formally established how it expects ALIS to perform in operations or developed a performance-measurement process for ALIS. Program officials indicated the need for discussions with the services and international partners about aspects of the current system that are not consistently being used and may therefore not be required (such as the Training Management System) through an updated process for establishing ALIS-related requirements. This process, which requires coordination across all military services and international partners, has proven to be challenging in the past. According to a 2017 Air Force Digital Service report, the F-35 program office faces challenges identifying and prioritizing ALIS capabilities across multiple services and international partners, and this has negatively affected the development of the system. DOD is unclear about the extent to which it can adopt a more flexible software development model known as Agile. As we reported in April 2019, the F-35 program as a whole is pursuing a faster and more incremental approach for delivering new aircraft capabilities to the warfighter in order to more flexibly address evolving threats. One approach to software development that helps facilitate such incremental delivery is Agile, which calls for the delivery of software in small, short increments rather than in the typically long, sequential phases of a traditional software development approach. More a philosophy than a methodology, Agile emphasizes early and continuous software delivery, as well as using collaborative teams, and measuring progress with working software. According to some F-35 program office officials, adopting Agile could result in a more secure system because it involves continually testing software for security vulnerabilities. Further, we have previously reported that following an incremental development approach, such as Agile, gives agencies the opportunity to obtain additional feedback from users, which increases the probability that each successive increment will meet user needs. The Mad Hatter initiative is experimenting with an Agile approach and has had some initial successes using this model. For example, in July 2019, we observed a demonstration of a Mad Hatter-developed application that allows the user to quickly and easily search through Joint Technical Data, an application within ALIS that has been reported by some users as being extremely difficult to navigate. However, the Mad Hatter initiative has operated outside of F-35 program office policies and processes and its applications are currently not integrated with the fielded ALIS system. Further, Mad Hatter and F-35 program office officials said that they have faced challenges communicating the value of their approach with one another, and according to a senior Air Force official associated with the Mad Hatter initiative, the F-35 program office has not clarified the role of Mad Hatter representatives in current planning efforts aimed at scaling the results of the Mad Hatter initiative to the entire F-35 enterprise. Separately, as part of its own ALIS initiative, prime contractor officials said that their company recently began taking steps to adopt best practices for delivering new ALIS software using an Agile model. However, these efforts are new, and the F-35 program office has not developed standards for software developed by the prime contractor using this model. DOD officials we spoke with expressed differing views on the extent to which DOD should adopt an Agile software delivery model for ALIS. For example, in a 2018 memorandum establishing the Mad Hatter pilot, a senior Air Force acquisition official stated that the F-35 program should embrace the tenets of this type of model in order to innovate and rapidly deliver useful capability through ALIS. Similarly, Air Force, Office of the Secretary of Defense, and some F-35 program office officials stated that modernizing ALIS will require DOD to adopt industry best practices by making decisions quickly, delivering usable products early and often, and revising plans to reflect experience from completed software iterations. In contrast, Marine Corps and some F-35 program office officials indicated that DOD should carefully consider different commercially-available software tools, as well as DOD-specific constraints, before delivering new ALIS capabilities. For example, F-35 program office officials associated with the ALIS Next initiative stated that they conducted an assessment of the commercial software tools that could be used for new ALIS software development. These officials said that some of the tools that were initially being used by the Mad Hatter initiative to develop applications make software development easier in the short-term but more difficult to switch toolsets and/or contractors in the long-term. Marine Corps and some F-35 program officials also noted that current DOD processes and procedures—such as the software certification and cost-estimating processes—may not be able to support quick software releases. While an Agile software delivery model has been identified as having the potential to improve the way in which the federal government develops and implements IT, we previously reported that this type of model requires significant procedural and organizational changes in order to be implemented successfully. DOD has not made a decision about the extent to which the ALIS re- design will be hosted in the cloud as opposed to onsite servers at the squadron level. In April 2019, we reported that cloud computing allows federal agencies to access on-demand, shared computing resources with the goal of delivering services more quickly and at a lower cost. More specifically, purchasing IT services through a provider enables agencies to avoid paying for all of the computing resources (e.g., hardware, software, networks) that would typically be needed to provide such services. This approach offers federal agencies a means to buy the services faster and possibly at less cost than building, operating, and maintaining these computing resources themselves. However, National Institute of Standards and Technology guidance states that public cloud computing represents a significant shift from the norms of on-site data centers and should therefore be approached carefully with consideration to the sensitivity of data. While the Mad Hatter initiative has embraced hosting ALIS in the cloud, including at the squadron level, ALIS Next is conducting an assessment of the extent to which a cloud-based system is the best option for ALIS. Further, as part of its internal ALIS investment, the prime contractor has designed an alternative model to the current system that includes an onsite server at each F-35 squadron. Office of the Secretary of Defense, Air Force, and F-35 program office officials we talked to agreed that the ALIS re-design will involve migrating some portions of ALIS from onsite servers to the cloud. For example, these officials agreed that DOD should explore options for migrating the ALOU and U.S. CPE to the cloud. However, these officials disagreed about how much of the future system should be cloud-based at the squadron level. For example, Air Force, Office of the Secretary of Defense, and some F-35 program office officials stressed that for day-to- day maintenance at U.S. bases, F-35 squadrons should be able to access ALIS using Wi-Fi, and that the reliance on onsite servers should therefore be minimal and limited to deployed scenarios. According to these officials, DOD can achieve significant cost savings by moving ALIS to the cloud. These officials also indicated that DOD’s hesitation about moving from onsite servers to the cloud is mostly cultural and the result of a lack of understanding about what the cloud is. One senior Office of the Secretary of Defense official with software expertise stated that warfighters should be able to deploy with a minimal amount of ALIS hardware (for example, only a high-powered laptop). In contrast, other F- 35 program office officials told us that the F-35 program office is restricted in the extent to which it can migrate to cloud-based SOUs due to connectivity and security restrictions. Further, at an ALIS Next conference, some partner country representatives expressed concerns about hosting ALIS in the cloud, stating that stringent security requirements would likely prevent their governments from accepting a cloud-based solution for ALIS. DOD does not have a plan for incorporating users early and often in the development of new ALIS software across the F-35 enterprise. Previous GAO reports as well as other DOD studies have found that giving users the opportunity to provide feedback on actual working software early and often in the software development process, and incorporating that feedback in subsequent development, is critical to the success of any software development effort. For example, in March 2019, we reported that obtaining frequent feedback is linked to reducing risk, improving customer commitment, and improving technical staff motivation. Historically, user feedback has not been prioritized in the ALIS software development process. According to users we talked to, working groups do exist that serve as a venue for voicing user-related issues; however, users stated that these working groups meet infrequently and often do not lead to desired changes. Further, prime contractor representatives told us that while they recently began soliciting user feedback as part of their ALIS initiative, the F-35 program office has not contractually required incorporating user feedback in the ALIS software development process. The Mad Hatter initiative is currently incorporating user feedback into new software development for ALIS and has established a process whereby F-35 users and Mad Hatter software developers can communicate directly about the Mad Hatter applications that are in development. As part of this process, Mad Hatter product teams develop simple applications, field the applications to users, and then use feedback from users—obtained by email or videoconferences—to adjust and enhance the applications. Although Mad Hatter’s process for incorporating user feedback aligns with the practice of incorporating feedback early and often, the initiative is being executed at one F-35 installation, with one military service. Further, while the F-35 program office intends to eventually scale the results of Mad Hatter’s experimentation to the rest of the F-35 enterprise, it has not formally outlined how it will institutionalize the initiative’s process for incorporating user feedback across multiple services and international partners. DOD has not determined the roles of DOD and the prime contractor in future ALIS development and management. DOD officials stressed that historically, the department has relied heavily on the prime contractor to develop and manage ALIS. Officials also said that moving forward, DOD will need to play a more active role in the management of ALIS. For example, Air Force, Office of the Secretary of Defense, and F-35 program office officials all said that DOD should serve as the primary owner of the ALIS software system, with the prime contractor and other firms developing applications that will feed into DOD’s software pipeline. However, the F-35 program office has not officially named DOD as the prime ALIS owner, or specified how it will coordinate software development across these multiple entities. Further, while one of the long- term objectives of the Mad Hatter initiative is to build DOD’s capacity to manage and develop new ALIS software itself, Air Force officials involved in this initiative stated that DOD has not yet fully developed this capacity. As the original ALIS developer, prime contractor representatives stated that their company is in the best position to modernize ALIS. F-35 program office officials acknowledged that because the prime contractor plays such a critical role in the development and sustainment of the F-35, it will be necessary for DOD to work closely with the contractor, regardless of the direction DOD decides to take. For example, DOD officials said they have faced challenges obtaining key technical data from the prime contractor that would be required by DOD to lead ALIS software development, such as the underlying source code for current ALIS software, and that they were uncertain about the extent to which they would be able to obtain these data in the future. At a November 2019 congressional hearing, the Under Secretary of Defense for Acquisition and Sustainment stressed that many of the challenges with ALIS stem from the fact that ALIS data are fed back through prime- contractor computers, and there is resulting ambiguity over the ownership of that data. As we previously reported, DOD continues to lack clarity about the technical data it owns and the additional data it would require to maintain flexibility in the sustainment of the F-35. DOD has not agreed on the extent to which the ALIS re-design will incorporate current ALIS software—consisting of 8 million lines of code. As part of the ALIS Next initiative, F-35 program office officials said they intend to review the underlying source code for ALIS to determine which aspects of the current software should be integrated in the re-design. These officials explained that redesigning ALIS software from scratch will take too long and the future ALIS system will therefore need to incorporate, to some extent, current ALIS software. In contrast, a senior Air Force official associated with the Mad Hatter initiative stated that the initiative intends to replace most current ALIS applications with commercial or new custom applications, retaining only those ALIS applications that can be cost-effectively modernized. Further, officials from the Air Force, Office of the Secretary of Defense, and F-35 program office indicated that because most of the ALIS source code has not been updated in years and contains numerous security vulnerabilities, the software should be completely re-designed. DOD is unclear about how it will approach the key technical and programmatic uncertainties surrounding ALIS because the department has not developed a strategy for the future re-design of the system. DOD guidance for program managers states that a sound strategy requires, among other things, a clear articulation of program goals as well as an understanding of the risks or uncertainties and costs associated with achieving those goals. While DOD and the prime contractor have established various initiatives to re-design ALIS, DOD has not developed a strategy for the future of ALIS that clearly identifies and assesses goals, key risks or uncertainties, and associated costs. For example, as discussed previously, DOD lacks clarity about the goals of the re-design, such as the capabilities that will be included in the future system and the extent to which ALIS will be hosted in the cloud. In addition, DOD has not fully assessed key risks or uncertainties, including the extent to which DOD can adopt an Agile software development approach or manage the system itself. Finally, because it has not answered key questions about the future of the system, such as the extent to which the re-design will incorporate current ALIS software, DOD has not been able to develop accurate cost estimates for the ALIS re-design. In the past, DOD has faced challenges estimating and tracking ALIS costs. For example, in 2016 we reported that while DOD had estimated that ALIS would cost approximately $17 billion, the estimate was not fully credible because DOD had not performed uncertainty and sensitivity analyses as part of the cost-estimating process. Further, for this review, the F-35 program office was not able to provide us with historic costs showing how much the department has spent on ALIS over the years. DOD officials stated that historically, the department has faced challenges allocating scarce resources across competing priorities, and that the F-35 air vehicle has generally been prioritized over ALIS. With the completion of F-35 developmental testing in April 2018, program officials said they are now in a better position to focus on ALIS and address long-standing issues with the system. However, efforts to correct ALIS are relatively new and have not been fully developed. Without a strategy to guide the re-design of ALIS, DOD will not be able to effectively plan for the transition from the current system to a future one. For example, according to F-35 program office officials, DOD recently procured additional hardware for the current system, which officials said may not be required if DOD is able to develop and field a re-designed ALIS in the near term. Officials from the Office of the Director of Operational Test and Evaluation stressed that effectively transitioning from the current system to a future one will be particularly challenging for DOD given the need to continue sustaining the more than 400 aircraft that have already been fielded with current ALIS. Further, as discussed above, there are divergent views in terms of how DOD should approach key technical and programmatic aspects of the re-design, and integrating the different efforts that are underway to fix ALIS—led by the F-35 program office, Air Force, and prime contractor—will therefore require significant direction and leadership. Without a strategy, DOD may not be able to effectively coordinate and leverage the different ALIS initiatives that are underway, potentially leading to inefficiencies. DOD also risks repeating history by failing to clearly articulate what it expects from ALIS and how it will play a more active role in the management of the system going forward. The F-35 aircraft, with its advanced warfighting capabilities, provides critical tactical aviation for the Department of Defense. However, DOD will need to overcome substantial challenges related to ALIS if it wants to find successes in both sustainment and operations of the aircraft. Current ALIS users continue to report significant challenges with the system that are affecting day-to-day operations of the aircraft, adding additional flight line-related responsibilities, and, in some instances, causing squadron leadership to assume the risk of flying aircraft when ALIS tells them to stay on the ground. Although ALIS is not currently performing well, over 5 years after we recommended it, DOD has yet to establish a performance- measurement process that would define how ALIS should perform. In the absence of such a process, DOD will be challenged to address current and future ALIS-performance issues because it cannot measure ALIS functionality compared to intended system performance. Furthermore, ALIS users collectively agree that the issues with ALIS are affecting the readiness of the aircraft; however, the degree to which this is true remains unknown. Fleet-wide mission capability rates for the F-35 are still below the warfighter’s minimum targets, but DOD does not have a process for measuring, collecting, and tracking information on how ALIS is affecting these rates. Without such a process, DOD may not understand all of the factors behind the reduced aircraft performance, thus limiting its ability to target appropriate solutions. DOD officials have acknowledged the ongoing challenges with ALIS and know that the system, as it stands today, cannot be sustained into the future; therefore, it is positive that the department has embarked on efforts to re-design and fix ALIS, as well as take on a more active role in the management of the system. However, DOD faces a significant challenge as there are several complex technical and programmatic uncertainties that will need to be resolved before any future ALIS solution can be realized. Additionally, there are divergent views among ALIS stakeholders about how to go about addressing these complex issues. The future of ALIS remains unclear because the department has not developed a strategy for the re-design of the system that would identify, among other things, what the system should look like, how will it be developed and managed, how it will address key risks, and how much it will ultimately cost. Without such a strategy, DOD will not be able to effectively plan for the transition from the current ALIS system, which is already embedded in over 400 aircraft across the global F-35 fleet, to whatever solution is determined. Furthermore, a strategy would help align what is currently a chorus of divergent views within the department on how to address the future of ALIS. With the worldwide fleet expected to grow to over 1,000 aircraft over the next four years, and with the U.S. services becoming increasingly reliant on the F-35’s capabilities to support their operational strategies, it will be imperative for DOD to address the ongoing issues related to the F-35’s logistics system. Congress should consider legislation requiring the Department of Defense to establish a performance-measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements. (Matter for Consideration 1) We are making the following two recommendations to DOD: The Secretary of Defense should ensure the Under Secretary of Defense for Acquisition and Sustainment, in consultation with the F-35 Program Executive Officer, develops a program-wide process for measuring, collecting, and tracking information on how ALIS is affecting the performance of the F-35 fleet to include, but not be limited to, its effects on mission capability rates. (Recommendation 1) The Secretary of Defense should ensure the Under Secretary of Defense for Acquisition and Sustainment, in consultation with the F-35 Program Executive Officer, develops and implements a strategy for the re-design of ALIS. The strategy should be detailed enough to clearly identify and assess the goals, key risks or uncertainties, and costs of re-designing the system. (Recommendation 2) We provided a draft of this report to DOD for review and comment. In its written comments, reproduced in appendix II, DOD concurred with our recommendations and identified actions that it was taking or planned in response. We agree that DOD is taking positive steps in addressing issues with ALIS, including the decision to replace ALIS with a future system that it has named the F-35 Operational Data Integrated Network (ODIN). According to DOD, the department is currently developing a strategy that will guide ODIN’s development. As DOD proceeds with replacing ALIS with ODIN, it will be imperative for the department to carefully consider and assess the key technical and programmatic uncertainties discussed in this report. These issues—including how much of ALIS will be incorporated in ODIN and the extent to which DOD has access to the data it needs to play a more active role in the management of the system—are complex, and will require significant direction and leadership to resolve. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 7 days from the report date. At that time, we will send copies of this report to congressional requesters; the Secretary of Defense; the Under Secretary of Defense for Acquisition and Sustainment; the F-35 Program Executive Officer; the Secretaries of the Air Force and Navy; and the Commandant of the Marine Corps. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix III. For each of our objectives, we reviewed relevant F-35 sustainment and the Autonomic Logistics Information System (ALIS)-related data, plans, program briefs, guidance, and other documentation and collected information by interviewing officials from the Office of the Under Secretary of Defense for Acquisition and Sustainment, the F-35 Joint Program Office, the Director, Operational Test and Evaluation, the Defense Contract Management Agency, the U.S. Air Force, the U.S. Navy, the U.S. Marine Corps, the Air Force Digital Service, and the prime contractor, Lockheed Martin. To interview officials and observe ALIS- related operations, we conducted site visits to five F-35 locations—Luke Air Force Base, Arizona; Edwards Air Force Base, California; Nellis Air Force Base, Nevada; Marine Corps Air Station Yuma, Arizona; and Naval Air Station Lemoore, California. We selected these locations to obtain perspectives from ALIS-users (i.e. maintainers, pilots, supply personnel, contractors) from all U.S. services participating in the F-35 program, including from operational, training, and testing locations. Additionally, we developed a data collection instrument to collect ALIS-related inputs and data from ALIS-users (i.e. maintainers, pilots, supply personnel, contractors) at all 10 U.S. F-35 locations—Luke Air Force Base, Arizona; Edwards Air Force Base, California; Nellis Air Force Base, Nevada; Marine Corps Air Station Yuma, Arizona; Naval Air Station Lemoore, California; Hill Air Force Base, Utah; Naval Air Station Patuxent, Maryland; Eglin Air Force Base, Florida; Marine Corps Air Station Beaufort, South Carolina; and Marine Corps Air Station Iwakuni, Japan. Finally, we met with officials from the F-35 Joint Program Office, Massachusetts Institute of Technology (MIT) Lincoln Labs, Lockheed Martin Rotary and Mission Systems, Air Force Digital Service, Kessel Run (Air Force), and others to discuss ALIS-related improvement efforts. In support of our objectives, we gathered data from fiscal year 2019 (the most recent full fiscal year of data available at the time of our review) from the prime contractor on the performance of the F-35 fleet such as the full and mission capability rates. We also collected the most recent available information on ALIS software deficiencies. To determine the reliability of these data, we collected information on how the data were collected, managed, and used through a questionnaire and interviews. Although we identified some limitations in the way that certain data are being collected and reported—such as data related to aircraft performance like mission capability rates—we determined that they are sufficiently reliable for the way in which we reported them and our purposes of providing information on the progress and challenges within the program. All the performance data presented in our report are sufficiently reliable to provide a general comparison of capabilities to minimum targets. To assess the extent to which there have been improvements as well as key challenges with ALIS over the last 5 years, we interviewed officials and examined guidance and briefing documents from the Office of the Under Secretary of Defense for Acquisition and Sustainment, the U.S. Services, the F-35 Joint Program Office, the Defense Contract Management Agency and Lockheed Martin Rotary and Mission Systems officials to discuss the current status of the system and plans for mitigating risks. To determine user views on risks to (or issues with) ALIS, we interviewed officials at our 5 selected bases, conducted a short data collection instrument of the other 5 bases, interviewed officials at Air Force headquarters and the contractor, and reviewed relevant documents. At the 5 bases, we interviewed groups of pilots, maintainers and supply personnel about ALIS performance, challenges, and possible improvements. In addition, we posed several targeted questions based on risks found in our last report. In total, we received input from more than 160 users at the 5 bases we visited through group discussions or interviews. We analyzed the responses provided in these group interviews, and identified the issues/risks that at least one set of users reported at each of the 5 bases. We also considered any improvements that were described as having occurred during the last few years. We also compared the responses from the interviews at the 5 bases with our data collection responses, and the other testimonial and documentary evidence we obtained. The list of issues/risks we identified contains some that were reported in our 2016 report as well as some new ones. While this list summarizes the types of issues/risks described at the 5 bases, and also in other interviews and document review, individual user views and experiences could vary by base and user group. We also interviewed officials and reviewed reports from the Air Force Audit Agency, the Director, Operational Test and Evaluation, and the Department of Defense Inspector General to identify improvements as well as any functionality issues with ALIS. We interviewed and gathered information from DOD officials on testing for ALIS, metrics on ALIS’s performance, and the operations of the system. As discussed previously, we collected and analyzed data for fiscal year 2019 that we obtained from the prime contractor on the overall aircraft performance such as the full mission capability and mission capability rates. We analyzed and compared information obtained from interviews, site visits, data collection instruments, and documents with guidance such as DOD’s System Engineering Guide for System of Systems to determine the extent to which DOD has an effective procedure for addressing and mitigating specific risks and challenges that may be associated with a major weapon system. We also compared this information with previous GAO reports from 2014, 2016, and 2018 to determine the extent to which DOD has addressed our prior recommendations on ALIS-related issues. To assess the extent to which the F-35 program has addressed issues with ALIS, we gathered and analyzed data from the prime contractor on open and closed ALIS deficiencies identified from November 2017 through October 2018. We selected this timeframe because it included the most recent data on ALIS deficiencies at the time of our review and also allowed us to observe trends in ALIS deficiencies over a two-year period. The data we received included summary information on the total number of open deficiencies, the total number of closed deficiencies, the number of newly closed deficiencies, the number of newly identified deficiencies, and the total number of open category 1 through category 3 deficiencies (considered critical or adverse) for each month during the two-year period. To determine the reliability of these data, we conducted electronic tests to identify any internal inconsistencies with the data. We also reviewed documentation from the prime contractor on the management of ALIS deficiency data and collected information on how the data were collected, managed, and used through a questionnaire. Specifically, we asked questions about inconsistencies we identified through electronic testing of the data, the extent to which the prime contractor’s system for collecting deficiency information includes edit checks or controls to help ensure the data are entered accurately, and limitations related to the accuracy or completeness of the data. As a result, we determined the data to be sufficiently reliable for the purpose of reporting trends in the number of open and closed ALIS deficiencies over time. To determine the extent to which DOD is taking actions to enhance the long-term viability of the system, we interviewed officials and reviewed guidance and/or planning documents from the Office of the Under Secretary of Defense for Acquisition and Sustainment, the F-35 Joint Program Office, and the Office of the Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics. We interviewed officials from the prime contractor to determine their role in helping DOD mitigate risks regarding the long-term viability for ALIS. Additionally, we examined briefing documents from the MIT-Lincoln Labs, a federally-funded research and development center assisting the F-35 Joint Program Office, on plans, timelines, and risks for modernizing the hardware and software. We interviewed officials from the Air Force’s Kessel Run team to discuss their Mad Hatter initiative (intended to improve ALIS functionality), the viability of current ALIS software, and any risks associated with the future of ALIS. We conducted a site visit to Nellis Air Force Base to observe the Mad Hatter initiative and discuss its results and the future of ALIS software. Further, as discussed previously, we analyzed data from November 2017 through October 2019 on ALIS deficiencies. We reviewed reports and interviewed officials from the Air Force Digital Service and the Director, Operational Test and Evaluation on the future viability of these long-term initiatives for ALIS. Finally, we analyzed and compared information obtained from interviews, site visits, and documents with applicable guidance to determine the extent to which DOD has an effective long-term plan for ALIS that addresses operational and financial risks. In support of our work, we interviewed officials from the following DOD organizations and other organizations during our review. We selected these organizations based on their oversight, planning, and/or execution roles related to F-35 ALIS operations. Office of the Under Secretary of Defense for Acquisition and Sustainment, Arlington, Virginia Office of the Director for Operational Test and Evaluation, Arlington, Defense Contract Management Agency Lockheed Martin, Orlando, F-35 Joint Program Office, Arlington, Virginia Office of the Assistant Secretary of the Air Force for Acquisition, Air Force F-35 Integration Office, Arlington, Virginia Kessel Run Team, Hanscom Air Force Base, Massachusetts Luke Air Force Base, Arizona 56th Maintenance Group 61st Aircraft Maintenance Unit 62nd Aircraft Maintenance Unit Edwards Air Force Base, California Nellis Air Force Base, Nevada 57th Aircraft Maintenance Squadron Navy F-35 Integration Office, Arlington, Virginia Naval Air Station Lemoore, California Strike Fighter Wing Pacific Strike Fighter Squadron 125 Strike Fighter Squadron 147 Marine Corps F-35 Integration Office Marine Corps Air Station Yuma, Arizona Marine Aircraft Group 13 Marine Aviation Logistics Squadron 13 Marine Fighter Attack Squadron 211 Marine Fighter Attack Squadron 122 Air Force Digital Service, Arlington, Virginia Lockheed Martin Rotary and Mission Systems, Orlando, Florida MIT Lincoln Laboratory, Lexington, Massachusetts We conducted this performance audit from August 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides and reasonable basis for our findings and conclusions based on our audit objectives. Diana Maurer, (202) 512-9627, maurerd@gao.gov In addition to the contact named above, Alissa Czyz (Assistant Director), Matthew Bader, Vincent Buquicchio, Tracy Burney, Juana Collymore, Martin De Alteriis, Michael Holland, Jeff Hubbard, Clarice Ransom, and Elisa Yoshiara made key contributions to this report. F-35 Aircraft Sustainment: DOD Faces Challenges in Sustaining a Growing Fleet. GAO-20-234T. Washington, D.C.: November 13, 2019. Space Command and Control: Comprehensive Planning and Oversight Could Help DOD Acquire Critical Capabilities and Address Challenges. GAO-20-146. Washington, D.C.: October 30, 2019. F-35 Joint Strike Fighter: Action Needed to Improve Reliability and Prepare for Modernization Efforts. GAO-19-341, Washington, D.C.: April 29, 2019. F-35 Aircraft Sustainment: DOD Needs to Address Substantial Supply Chain Challenges.GAO-19-321. Washington, D.C.: April 25, 2019. Cloud Computing: Agencies Have Increased Usage and Realized Benefits, but Cost and Savings Data Need to Be Better Tracked. GAO-19-58, Washington, D.C.: April 4, 2019. DOD Space Acquisitions: Including Users Early and Often in Software Development Could Benefit Programs. GAO-19-136. Washington, D.C.: March 18, 2019. F-35 Joint Strike Fighter: Development Is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved. GAO-18-321. Washington D.C.: June 5, 2018. Warfighter Support: DOD Needs to Share F-35 Operational Lessons Across the Military Services. GAO-18-464R, Washington D.C.: April 25, 2018. Military Aircraft: F-35 Brings Increased Capabilities, but the Marine Corps Needs to Assess Challenges Associated with Operating in the Pacific. GAO-18-79C. Washington, D.C.: March 28, 2018. Information Technology Reform: Agencies Need to Improve Certification of Incremental Development. GAO-18-148. Washington, D.C.: November 7, 2017. F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency. GAO-18-75. Washington D.C.: October 26, 2017. F-35 Joint Strike Fighter: DOD’s Proposed Follow-on Modernization Acquisition Strategy Reflects an Incremental Approach Although Plans Are Not Yet Finalized. GAO-17-690R. Washington, D.C. August 8, 2017. F-35 Joint Strike Fighter: DOD Needs to Complete Developmental Testing Before Making Significant New Investments. GAO-17-351. Washington, D.C.: April 24, 2017. F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities. GAO-16-390. Washington, D.C.: April 14, 2016. F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016. F-35 Joint Strike Fighter: Preliminary Observations on Program Progress. GAO-16-489T. Washington, D.C.: March 23, 2016. F-35 Joint Strike Fighter: Assessment Needed to Address Affordability Challenges. GAO-15-364. Washington, D.C.: April 14, 2015. F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.: September 23, 2014. F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26, 2014. F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014. F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013. F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but Affordability Challenges and Other Risks Remain. GAO-13-500T. Washington, D.C.: April 17, 2013. F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013. Software Development: Effective Practices and Federal Challenges in Applying Agile Methods. GAO-12-681. Washington, D.C.: July 27, 2012. Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks. GAO-12-437. Washington, D.C.: June 14, 2012. Joint Strike Fighter: Restructuring Added Resources and Reduced Risk, but Concurrency Is Still a Major Concern. GAO-12-525T. Washington, D.C.: March 20, 2012. Joint Strike Fighter: Implications of Program Restructuring and Other Recent Developments on Key Aspects of DOD’s Prior Alternate Engine Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19, 2011. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011. Joint Strike Fighter: Restructuring Should Improve Outcomes, but Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.: March 15, 2011.", "summary": "The F-35 is DOD's most ambitious and costly weapon system in history, with U.S. sustainment costs estimated at about $1.2 trillion over a 66-year life cycle. Central to the F-35 is ALIS—a complex system that supports operations, mission planning, supply-chain management, maintenance, and other processes. A fully functional ALIS is critical to the F-35's operational success. However, over the past 5 years GAO has reported on key risks associated with the system, such as challenges deploying the F-35 with ALIS, inaccurate data that reside in ALIS, and ineffective training for personnel who need to use ALIS. GAO was asked to review DOD's efforts to improve ALIS. This report assesses the extent to which (1) improvements have been made over the past 5 years and challenges remain for ALIS users, and (2) DOD is taking actions to enhance the long-term viability of the system. GAO reviewed F-35 and ALIS program documentation and data, interviewed DOD officials and contractor employees, and visited five U.S. F-35 sites. The Autonomic Logistics Information System (ALIS) is integral to supporting the F-35 fighter jet's operations and maintenance. F-35 personnel at 5 locations GAO visited agreed that ALIS is performing better in some aspects, such as faster processing speeds for some tasks. However, problems with ALIS continue to pose significant challenges for F-35 personnel (see figure). The Department of Defense (DOD) has not (1) developed a performance measurement process for ALIS, which GAO recommended in 2014, or (2) determined how ALIS issues affect F-35 fleet readiness. Without efforts in these areas, DOD will be hindered in addressing ALIS challenges and improving aircraft readiness. DOD and the prime contractor have a variety of initiatives underway for re-designing ALIS. However, these initiatives involve differing approaches and technical and programmatic uncertainties are hindering the re-design effort (see figure). DOD has not developed a strategy for the future of ALIS that includes goals of the re-design, an assessment of key risks, or costs. Without this, DOD may not be able to coordinate various ALIS design-improvement initiatives that are under way or meaningfully enhance the system over the long term. GAO is recommending that DOD track how ALIS is affecting readiness of the F-35 fleet and develop a strategy for the ALIS re-design. In addition, GAO believes that Congress should consider requiring DOD to develop a performance measurement process for ALIS. DOD concurred with both of GAO's recommendations.", "document_type": "gao"}
{"report": "Following Hurricane Katrina in 2005, the Post-Katrina Act required FEMA to develop a national preparedness system and assess preparedness capabilities to determine the nation’s disaster preparedness. In September 2011, DHS issued the National Preparedness Goal: a secure and resilient nation with the capabilities required across the whole community to prevent, protect against, mitigate, respond to, and recover from the threats and hazards that pose the greatest risk. The National Preparedness Goal also defined the “whole community” as individuals and communities, the private and nonprofit sectors, faith-based organizations, and all governments (local, regional/metropolitan, state, tribal, territorial, insular area, and federal). The National Preparedness Goal identifies and defines 32 core capabilities across five broad mission areas. These capabilities form the foundation for measuring overall national preparedness and assisting the nation in allocating resources to fill identified preparedness gaps. Three of the 32 core capabilities affect all mission areas and are considered to be “crosscutting” (see fig. 1). The five broad mission areas are: Prevention. Preventing an imminent threat, or actual act of terrorism. Protection. Protecting citizens, residents, visitors, and assets in a manner that allows interests, aspirations, and way of life to thrive. Mitigation. Mitigating the loss of life and property by lessening the impact of future disasters. Response. Responding quickly to save lives, protect property and the environment, and meet basic human needs in the immediate aftermath of an incident. Recovery. Recovering through a focus on the timely restoration, strengthening, and revitalization of infrastructure, housing, and a sustainable economy, as well as the health, social, cultural, historic, and environment fabric of communities affected by an incident. Since 2012, DHS has produced a National Preparedness Report annually, which assesses progress toward the National Preparedness Goal of achieving a secure and resilient nation. A key element of the National Preparedness Report is that it evaluates and measures the extent to which jurisdictions have strengthened their 32 core capabilities. From 2012 to 2017, all 50 states, District of Columbia, and 5 territories were required to assess the preparedness levels of their 32 capabilities by providing a rating of 1 to 5, with 5 being the highest preparedness rating. Emergency management capabilities with a rating of 1 or 2 are considered to have the largest capability gaps. FEMA used this assessment process to inform the National Preparedness Report by illustrating which threats and hazards occurred in the past and which capabilities have the largest gaps. FEMA’s National Preparedness Directorate, which includes the National Preparedness Assessment Division, is responsible for assisting communities in becoming more resilient by developing the capabilities needed to prevent, protect against, respond to, recover from, and mitigate against all threats and hazards. The Directorate provides guidance, programs, and processes to assist communities in completing the requirements associated with the National Preparedness System. To help jurisdictions more comprehensively assess their gaps, FEMA required they complete the Threat and Hazard Identification and Risk Assessment (THIRA) and Stakeholders Preparedness Review (SPR). The THIRA is conducted by jurisdictions every 3 years to, in part, identify threats and hazards that are both reasonably likely to affect the community and would most challenge the community’s ability to deliver one or more of its capabilities; and estimate and describe the potential impacts of those threats and hazards. The types of threats and hazards are defined as (1) natural hazards or acts of nature; (2) technological hazards that are accidents or failures of systems and structures; and (3) human-caused incidents resulting from intentional actions. Jurisdictions are to conduct the SPR annually to, among other things, identify capability gaps by assessing the capabilities against the types of threats and hazards identified in the THIRA. In 2012-2013, FEMA issued its initial guidance to jurisdictions to help them understand how to identify the threats and hazards through the THIRA, and assess their core capabilities. In 2018, FEMA issued new guidance for the THIRA and SPR requiring jurisdictions to change the methodology, moving away from proficiency-based ratings to a process that relies more on quantitative data to measure gaps across the core capabilities. In 2018, FEMA required jurisdictions to begin using the new methodology to assess the core capabilities within the response and recovery mission areas. Beginning in 2019, FEMA required jurisdictions to begin using the new methodology to assess the core capabilities across all mission areas. FEMA’s 10 regions provide technical assistance and training to help jurisdictions become more proficient in completing these capability assessments. FEMA also sponsors exercises with states, territories, tribes, and localities to help them assess their emergency management capabilities. In addition to the jurisdictions’ THIRAs and SPRs, in 2019, FEMA initiated an effort to assess the federal government’s emergency management capacity. According to FEMA, the effort is intended to provide a national THIRA and SPR that assesses the federal government’s capabilities against the nation’s threats and hazards. The Disaster Recovery Reform Act of 2018 (DRRA) requires FEMA, among other things, to provide congressional committees updates every 6 months on its progress in completing a national preparedness assessment until the assessment is complete. In July 2019, FEMA issued its 2019 National Threat and Hazard Identification and Risk Assessment (National THIRA): Overview and Methodology, describing its approach to completing a national-level risk assessment (i.e., a National THIRA). According to FEMA, the National THIRA was completed in 2020, and will be included in the 2020 National Preparedness Report. DHS, through FEMA, provides jurisdictions preparedness grants, which are used, in part, to strengthen the 32 core capabilities across the five mission areas. FEMA has traditionally provided three primary preparedness grants that jurisdictions can use to strengthen their emergency management core capabilities. Two of the three grants, the State Homeland Security Grant Program and the Urban Area Security Initiative, were established after the terrorist attacks of September 11, 2001. As established by federal law, these grants are intended to help states and localities prevent, prepare for, protect against, and respond to acts of terrorism. State Homeland Security Grant Program. Provides funding to assist state, local, and tribal governments in preventing, preparing for, protecting against, and responding to acts of terrorism. Helps support states’ implementation of homeland security strategies to address the identified planning, organization, equipment, training, and exercise needs at the state and local levels. In fiscal year 2019, the total funding available to all 50 states, District of Columbia and 5 territories was $415 million. Urban Area Security Initiative. Provides federal assistance to address the unique needs of high-threat, high-density urban areas, and assists the areas in building a capacity to prevent, prepare for, protect against, and respond to acts of terrorism. In fiscal year 2019, the total funding available to the 31 urban areas was $590 million. Emergency Management Performance Grant. Provides federal assistance to states to assist state, local, and tribal governments in preparing for all hazards. The program plays a valuable role in strengthening and sustaining the 32 core capabilities across the five mission areas. In fiscal year 2019, the total funding available to states, local governments, and tribes was $315 million. The National Preparedness System and associated preparedness grants have helped jurisdictions strengthen and sustain their emergency management capabilities. More specifically, according to National Preparedness Reports since calendar year 2012, states and territories generally have rated their capabilities within the prevention and response mission areas, as well as their crosscutting capabilities—which involve all five mission areas, as having the highest preparedness levels. By contrast, states and territories generally have rated their capabilities in the recovery and protection mission areas as having lower preparedness levels, and these ratings showed little to no improvement from 2013 to 2017. Additionally, since 2013, jurisdictions have directed nearly 87 percent of their FEMA preparedness grants toward sustaining or strengthening capabilities in the crosscutting, prevention, and response mission areas, and around 13 percent on enhancing or sustaining capabilities in the protection, mitigation, and recovery mission areas. FEMA has encouraged jurisdictions to invest future preparedness grants to strengthen their capabilities that have lower preparedness ratings and to address emerging threats, such as cybersecurity. However, FEMA officials told us their efforts to help jurisdictions enhance their capabilities, including the distribution of existing preparedness grants, will likely not be sufficient to address the capability gaps that have been identified by jurisdictions. States and territories’ 2017 preparedness data showed that eight core capabilities in the response and crosscutting mission areas had the highest level of preparedness (a rating of 4 or 5 on a 5-point scale). For example, as shown in figure 2 below, over 50 percent of the assessment ratings by the states and territories identified crosscutting capabilities, such as public information and warning and operational coordination, in the highest category of preparedness. Similarly, 57 percent of the assessment ratings by states and territories identified on-scene security, protection, and law enforcement capabilities within the response mission area in the highest preparedness categories. In our discussions with local officials who were impacted by the 2017 and 2018 hurricanes, they told us that the operational coordination and public information and warning capabilities were effective during their response efforts. For example, Craven County, NC, officials told us that in response to the flooding from Hurricane Florence, their emergency operations center was instrumental in communicating with first responders. In doing so, they were able to keep county wells running while working with utility companies to prioritize areas that needed electrical power, such as hospitals and grocery stores. Additionally, Onslow County, NC officials said their emergency operations center was instrumental in communicating and coordinating the rescue operations of around 700 residents through the use of the county’s swift water rescue teams, with assistance from the U.S. Marine Corps, North Carolina’s National Guard, and the local fire and police departments. In addition, Brazoria County, TX, officials told us that in response to Hurricane Harvey they used videos and social media to get warning messages out to the residents and businesses about evacuation assistance as well as information on hurricane preparedness. Supply chain integrity and security Hurricane Florence caused significant flooding in and around New Hanover County, North Carolina. County officials told us the state’s National Guard high wheel clearance trucks had to be used to transport food, water, fuel, and generators throughout the flooded areas to isolated communities because the county did not have the capability to deliver these commodities. According to the North Carolina National Guard, the high-water vehicles were also used to evacuate citizens to shelters and transport essential civilian personnel such as nurses, doctors, and first responders. Preparedness data from 2017 show that almost 40 percent of jurisdictions’ ratings identified five capabilities in the recovery and protection mission areas in the lowest category of preparedness (a rating of a 1-2 on a 5-point scale). For example, within the recovery mission area, 51 percent of the ratings identified disaster housing in the lowest category of preparedness. Similarly, within the protection mission area, 46 percent of the assessment ratings identified cybersecurity in the lowest category. Additionally, these capabilities have been consistently rated in the lowest preparedness categories from 2013 through 2017 and have shown little-to-no change. For example, under the recovery mission area, 56 percent of the assessment ratings by states and territories identified disaster housing in the lowest category in 2013, with minimal changes through 2017. Some of the capabilities that had the lowest preparedness ratings in 2013 were: Economic recovery. The ability to return economic and business activities (including agricultural) to a state of health and develop new economic opportunities that result in a sustainable and economically viable community. Natural and cultural resources. The ability to preserve, conserve, rehabilitate, and restore historic property consistent with post-disaster community priorities and best practices and in compliance with environmental and historic preservation laws and executive orders. Disaster housing. The ability to address pre- and post-disaster housing issues and coordinate the delivery of federal resources and activities to assist local, state, tribal, territorial, and insular area governments as they rehabilitate and reconstruct destroyed and damaged housing. Supply chain integrity and security. The ability to secure and make resilient key nodes, methods of transport between nodes, and materials in transit between a supplier and consumer. Table 1 shows the percentages of the lowest-rated capabilities from 2013 through 2017. While the National Preparedness System may help jurisdictions assess their preparedness using emergency preparedness capability assessments, jurisdictional officials we spoke with told us that real-life disasters sometimes show jurisdictions to be less prepared than their capability assessments previously indicated. As a result, some states have lowered their preparedness ratings in subsequent capability assessments following a disaster. For example, after the 2017 and 2018 hurricanes, some states told us they lowered their preparedness rating in their 2019 assessments for disaster housing because they realized the capability gap was larger than they previously believed. Officials from the North Carolina Division of Emergency Management said that in 2018, they lowered their preparedness rating for housing because their housing capacity was not able to meet the needs of disaster victims who needed immediate housing assistance. From fiscal years 2013 through 2018, jurisdictions received approximately $8.3 billion in preparedness grants funds primarily from the State Homeland Security Program, Urban Area Security Initiative, and the Emergency Management Performance Grant. Of this amount, jurisdictions directed about $7.3 billion—or about 87 percent of the funds—to capabilities in the crosscutting, prevention, and response mission areas, which constitute the highest-rated mission areas. For example, in California, $1.9 million in Urban Area Security Initiative grants were used to strengthen crosscutting and prevention capabilities by providing situational awareness to first responders and emergency managers working on active threats to infrastructure. Additionally, in Florida, up to $2.8 million of the State Homeland Security grant was used to create a system intended to strengthen crosscutting and prevention capabilities by enabling the state’s law enforcement agencies to more easily share information. Of the $8.3 billion in preparedness grant funding from fiscal years 2013 through 2018, about $1.1 billion—or about 13 percent—was directed to capabilities in the mitigation, protection, and recovery mission areas, which constitute the lowest-rated mission areas. During this time, jurisdictions directed the least amount of preparedness grant funds on the recovery mission area—$78 million, or about 1 percent (see fig. 3). Jurisdictions also directed about 5 percent of the $1.1 billion to capabilities within the mitigation mission area, though preparedness ratings in the mitigation mission area generally showed improvements each year. In 2017, 43 percent of the assessment ratings by states and territories rated three of the four mitigation-related capabilities in the highest category. Improvements in the mitigation mission area could be, in part, attributable to FEMA providing jurisdictions with grant funds other than preparedness grants, such as post-disaster grants, which include Hazard Mitigation Grant Program funds. Further, state and local decisions on how to prioritize preparedness grant awards resulted in about 1 percent—$78 million—being directed to capabilities within the recovery mission area between 2013 and 2018. As shown in figure 4, jurisdictions directed approximately 79 percent of the $78 million (about $62 million) in the recovery mission area to the infrastructure systems capability, which is intended to, in part, allow jurisdictions to re-establish critical infrastructure in disaster-impacted areas to support life sustainment activities, ongoing emergency response operations, and to help facilitate recovery efforts. Additionally, about 3 percent of the $78 million—about $2.4 million—was directed to disaster housing capabilities, such as implementing housing solutions that effectively support the temporary housing needs of an impacted jurisdiction. State officials from New York and North Carolina, as well as officials from five localities, said they often prioritize and use preparedness grants to maintain existing capabilities within the crosscutting, prevention, and response capabilities, rather than enhancing capabilities where gaps are known to exist, such as those in the recovery and protection areas. In addition, state officials from Texas, as well as officials from two localities told us that they need to use portions of their limited grant funds— especially from Emergency Management Performance Grant funds—to hire and retain local emergency management personnel, which leaves fewer funds for them to devote to enhancing lower-rated emergency management capabilities. For example, some county governments may not have the resources necessary to fund a single emergency manager, which requires them to use Emergency Management Performance Grant funds to hire and retain necessary staff. Another reason why jurisdictions do not use more of the grants toward lower-rated mission areas is because some view certain capabilities in the recovery and mitigation mission areas to be the responsibility of the federal government. Both FEMA and state officials told us that sometimes jurisdictions do not use these grants to strengthen capabilities such as housing because they consider the federal government responsible for filling the gaps. For example, preparedness data from 2013 to 2017 showed the percent of jurisdictions identifying the federal government as responsible for providing housing solutions to disaster survivors increased from 46 to 53 percent. According to state officials from North Carolina, it would not be a prudent use of grant funds for the state to purchase and store temporary housing units that may not be needed inside the borders of the state for several years. Following a major disaster declaration, FEMA coordinates with jurisdictions to provide disaster housing assistance to people displaced from their homes. For example, following Hurricane Florence, FEMA provided financial rental assistance and grants under its Individuals and Households program to help make repairs to damaged homes. In addition, FEMA, in coordination with the state of North Carolina, delivered travel trailers and manufactured housing units (i.e., mobile homes) to displaced disaster victims through FEMA’s Direct Temporary Housing Assistance program. FEMA has encouraged jurisdictions to make investments in core capabilities that have the largest preparedness gaps (i.e., the lowest preparedness scores). From 2013 to 2018, DHS identified investment priorities in its annual announcements of preparedness grant funding opportunities. The priorities focused on select capabilities where jurisdictions had reported lower preparedness scores, such as cybersecurity, disaster housing, economic recovery, natural and cultural resources, and supply chain integrity and security. Specifically, FEMA officials told us cybersecurity remains a high priority for all jurisdictions for 2020 and has identified areas from lessons learned where cybersecurity could be strengthened. Jurisdictions are considering investments in cybersecurity such as adding more information technology equipment and hiring personnel with cybersecurity expertise. However, according to state officials from New York and Texas, jurisdictions often lack the resources necessary to hire and retain personnel skilled enough to prepare for, respond to, and recover from cyberattacks. Preparedness grants, in general, are designed to allow jurisdictions discretion to spend the funds as they see fit on projects that meet eligibility requirements. While FEMA encourages jurisdictions to invest grant funds to address their capability gaps, it does not require or direct jurisdictions to spend grant funding in a certain area. In light of these challenges, FEMA has taken a number of other steps to try to address these capability gaps. FEMA proposed creating a new National Priorities Security Grant in the President’s 2019 and 2020 budget proposals, which could be used to address new and emerging threats and gaps, such as those in cybersecurity. FEMA proposed that the program’s priorities be assessed frequently and shift as needed to address emerging threats and capability gaps. In fiscal years 2019 and 2020, the President’s budget proposed $522 and $430 million respectively. The proposed grant program was not approved by Congress. In 2019, FEMA established the Regional Catastrophic Preparedness Grant Program to help jurisdictions address known capability gaps in disaster housing as well as logistics and supply chain management. In fiscal year 2019, FEMA awarded $10 million in these grants to eight local governments. In 2019, FEMA began implementing the Building Resilient Infrastructure and Communities (BRIC) program to provide jurisdictions with funding to make their infrastructure more resilient in future disasters. According to FEMA, grant recipients could use future funding to strengthen capability gaps in the recovery and mitigation mission areas. FEMA plans to issue a Notice of Funding Opportunity in the summer of 2020, followed by an application period. Based on historical disaster expenditures, FEMA anticipates BRIC will be funded between $300 million and $500 million per year on average. It is too early to assess the extent to which this program will help address capability gaps. While FEMA is taking steps to encourage jurisdictions to enhance their lower-rated capabilities, FEMA officials told us their efforts combined with existing preparedness grants, will likely not be sufficient to fully address jurisdictions’ capability gaps. Specifically, FEMA officials told us that the current suite of preparedness grants lacks the flexibility needed to address some of the long-standing capability gaps, in part, because the grants are required to be spent on capabilities that have a nexus to terrorism. In addition, as described earlier, one state official, and two local officials, suggested that the level of funding for the Emergency Management Performance Grant will likely not allow states and localities to hire and retain local emergency management personnel while also making the investments needed to address the capability gaps identified through the National Preparedness System. For example, one emergency management official from a county explained that without using the Emergency Management Performance Grant to offset his own salary, the county would not have an emergency management department with the capability to complete many of the FEMA requirements associated with receiving disaster assistance. In addition to the steps FEMA has already taken to attempt to address the capability gaps, FEMA has developed a new methodology for assessing national preparedness capabilities that uses more quantitative methods. According to FEMA, such methods could enable jurisdictions to more tangibly define what resources are needed to fill identified gaps. We describe this methodology in more detail below. FEMA has taken steps to enhance its methodology for assessing jurisdictions’ emergency management capabilities by requiring jurisdictions to collect more quantitative preparedness data to support their capability ratings. We reported in March 2011 that FEMA needed to improve its oversight of preparedness grants by establishing a framework with measurable performance objectives for assessing urban area, state, territory, and tribal capabilities to identify gaps and prioritize investments. Specifically, we recommended that FEMA complete a national preparedness assessment of capability gaps at each level of government based on tiered, capability-specific performance objectives to enable prioritization of grant funding. We also reported in March 2013 that FEMA has made some progress in assessing its preparedness capabilities, but continued to face challenges developing a national preparedness system that could assist FEMA in prioritizing preparedness grant funding. FEMA’s issuance of the 2020 National Preparedness Report could provide an assessment of capability gaps at each level of government—including an assessment of the federal government’s capabilities for the first time—and help FEMA address the intent of the 2011 recommendation. However, as discussed before, prioritizing jurisdictions’ preparedness grant funding alone may not effectively address the nation’s emergency management capability gaps. An assessment that also considers the federal government’s emergency management capabilities could help determine what capabilities federal agencies could provide to assist in the wake of disasters when jurisdictions’ capabilities become overwhelmed or are not otherwise available. Once the assessment is completed, FEMA and its federal budgeting stakeholders (i.e., Congress and the Office of Management and Budget) could use such an assessment to identify the potential costs of establishing and maintaining capabilities, not only at the jurisdictional level, but also at the federal level. FEMA has continued to take steps to implement the 2011 recommendation, but has not yet fully addressed it as of January 2020. For example, FEMA published new guidance in May 2018 to update the methodology for how jurisdictions are to evaluate their preparedness levels when completing THIRAs and SPRs. The intent was to allow communities to collect more specific, quantitative data to compare their capability targets to current capabilities, thereby more accurately defining their capability gaps. Beginning in 2018, jurisdictions used the new methodology to assess their capabilities in the crosscutting, response, and recovery mission areas. Beginning in 2019, jurisdictions were required to use the new methodology to assess the capabilities across all five mission areas: prevention, protection, mitigation, response, and recovery. According to FEMA, this new methodology improves on the prior one because the new methodology will allow jurisdictions to more accurately determine what amount of resources are needed to address specific threats and hazards. Specifically, as a result of using more quantitative data, such as the specific number of disaster victims able to be sheltered following a disaster, jurisdictions may be able to better define their capability gaps when compiling their SPRs. For example, if jurisdictions are able to understand that their current capability is less than their needed capability target, they will be able to define their capability gaps in quantitative terms. According to FEMA officials, the new methodology, if implemented successfully, will allow jurisdictions to know what additional resources and capabilities—beyond their own current capabilities—may be needed during future disasters. Table 2 shows an example of how FEMA’s updated methodology provides a more quantitative assessment to more accurately define their capabilities. In 2019, FEMA began working on its first National Threat and Hazard Identification and Risk Assessment (National THIRA) to identify what federal government capabilities will be needed to address the greatest threats to the nation. According to FEMA, the results of this effort are expected to be included in FEMA’s annual National Preparedness Report in 2020, which is expected to be published late in calendar year 2020. FEMA’s effort is intended to provide a quantitative assessment of federal capabilities, which when combined with state, territory, urban area, and tribal THIRAs and SPRs, could provide a more meaningful assessment of the nation’s overall preparedness. Figure 5 below shows how national and jurisdiction risk assessments are intended to work together to provide a collective picture of overall capability gaps. As subsequent iterations of the National THIRA and National SPR are produced, FEMA intends to consolidate them with the THIRA and SPR assessments submitted by jurisdictions to provide a comprehensive overview of national preparedness. FEMA officials told us that they have begun to assess and measure the federal government’s capabilities in the crosscutting, response, and recovery mission areas. In conducting the 2019 National THIRA—that FEMA officials told us will be included in the 2020 National Preparedness Report—FEMA coordinated with over a dozen federal departments and agencies, as well as selected national laboratories and the White House to solicit feedback on the most challenging threats and hazards facing the nation. The 2019 National THIRA consists of nine catastrophic incident scenarios and 22 capability targets across the crosscutting, response, and recovery mission areas. For example, FEMA used catastrophic scenarios, such as a pandemic (see sidebar) or New Madrid Earthquake, to assess the nation’s emergency management capacity. Pandemic Scenario In early October, the Centers for Disease Control and Prevention (CDC) reports a new strain of influenza virus in the National Capital Region. Less than 2 weeks after the first confirmed case is identified at a local hospital, the illness causes hundreds of fatalities and thousands of people seeking medical attention. As the virus spreads, approximately 30 percent of the population across the United States and other countries becomes severely ill. Conventional flu vaccines are ineffective against the current strain, and the CDC estimates that a new vaccine could be months away from mass production. Because of the pandemic, social distancing is in widespread effect. Utilities, police, fire, government, and other essential services are disrupted due to social distancing and employee absenteeism. Businesses close, resulting in a large-scale loss of services across the region (e.g. banking, food stores, gas stations). There is a shortage of medical supplies, equipment, beds, and healthcare workers as hospitals are quickly overwhelmed, with up to millions of individuals seeking outpatient medical care and millions more requiring hospitalization. Civil disorder contributes to the high rate of absenteeism and the overcrowding of hospitals and medical centers. how quickly power service can be restored to customers; how quickly life-sustaining commodities can be delivered to people; how quickly emergency sheltering, food, and water can be provided to how quickly affected healthcare facilities can restore function. In the aftermath of the sequential 2017 disasters, FEMA’s 2017 Hurricane Season FEMA After-Action Report recognized the need to more effectively scale response efforts for concurrent, complex incidents. As a result, in addition to the nine catastrophic scenarios, the National THIRA considered the challenges associated with managing concurrent incidents. To examine the potential impacts of managing concurrent incidents, FEMA developed a set of “plausible concurrent operations.” FEMA acknowledged that the agency and its federal partners “will almost certainly be engaged in ongoing disaster operations at the time of any catastrophic-level incident” and gathered data from historical incidents from recent years, including the sequential disasters that took place in 2017; three large hurricanes and wildfires in California, among others. FEMA found that combining the impacts of a National THIRA scenario with the set of plausible concurrent operations more accurately reflects the challenges the nation would need to address should one of the threat scenarios identified in the National THIRA occur. However, given that FEMA has yet to finalize inclusion of the National THIRA into the 2020 National Preparedness Report, it is too early to determine the extent to which it helps FEMA more accurately define the nation’s emergency management capability gaps and results in the nation being better prepared for future catastrophic disasters. As discussed above, the National Preparedness System has identified gaps in jurisdictions’ emergency management capabilities since 2012. While jurisdictions have used preparedness grants to strengthen select capabilities, preparedness data shows that they have not used the grants to address capability gaps across all the mission areas. Furthermore, while FEMA has encouraged jurisdictions to use grant funding to address capabilities that have the largest capability gaps, such as those in the recovery and cybersecurity areas, they do not require that jurisdictions do this. However, if FEMA were to require jurisdictions to use their grant funds to address lower-rated capabilities, it could affect jurisdictions’ ability to sustain other core capabilities—or to fund emergency management personnel in select jurisdictions, some of which only have one full-time employee. As FEMA implements its new methodology and begins to more fully assess both federal and jurisdictional capabilities, the agency is expected to have better and more quantitative information on capability gaps in order to better prioritize grant funds and resources. According to FEMA, the agency and its partners will better understand the extent of the nation’s emergency management capability gaps when they issue the National Preparedness Report by December 2020. While these actions may allow FEMA to address our 2011 recommendation and better measure the nation’s overall preparedness, the agency has yet to determine what additional actions may be needed to close the remaining gaps once the 2020 National Preparedness Report is issued. Further, while FEMA has taken some steps to close the gaps jurisdictions have identified since 2012, such as proposing the National Priorities Security Grant, this program has not been approved by Congress, and thus, will not help to address the gaps. According to FEMA officials, preparedness grants alone are unlikely to address the gaps in an effective manner. In addition, the National Preparedness Goal states that analyzing current performance against intended capabilities allows the emergency management community the opportunity to determine necessary resource levels, inform resource allocation, and help guide federal investments in preparedness. Such information could help inform budget decisions across the preparedness enterprise and help prioritize limited resources. For example, determining what steps need to be taken, following the issuance of the 2020 National Preparedness Report, could help FEMA inform key decision makers, such as Congress and the Office of Management and Budget, about the necessary level of resources—including the allocation of resources—that are needed to address the nation’s capability gaps. Such an effort could be a significant step toward enhancing the capability gaps that have been identified since 2012 and help determine the nation’s overall preparedness levels, as called for in the Post-Katrina Act. The Post-Katrina Act requires FEMA to analyze real-world events to identify and disseminate lessons learned and best practices, and to generate and disseminate, as appropriate, after-action reports to participants after real-world events. After major disasters occur, FEMA’s policy is to conduct an after-action review that identifies strengths, areas for improvement, and potential best practices identified during response and recovery efforts. Lessons learned from past disasters are to provide collective knowledge and diverse experiences for improving disaster response and recovery. Further, FEMA’s 2018-2022 Strategic Plan calls for sharing lessons learned from disasters and exercises with the whole community to help prioritize investments and anticipate known challenges during future disasters. In July 2018, FEMA published its 2017 Hurricane Season After-Action Report, which discussed findings and recommendations based on a review of the agency’s preparation for, immediate response to, and initial recovery operations for Hurricanes Harvey, Irma, and Maria. According to FEMA, the agency is implementing recommendations to address the challenges outlined in the after-action report, which include the following focus areas: scaling and staffing for concurrent complex incidents; improving logistics capabilities during response; improving response to long-term infrastructure outages; and, improving mass care to initial disaster housing operations based on innovations developed during the 2017 hurricane season. According to FEMA, the agency has taken a number of actions in response to this after-action report. For example, it increased its incident management workforce strength by 19 percent since Hurricane Harvey and updated hurricane plans, annexes, and procedures for the continental United States and for states and territories outside the continental United States, among other things. FEMA’s Continuous Improvement Program is responsible for collecting observations and conducting after-action reviews after disasters. The program is intended to consolidate feedback and information from regional, headquarters, and field operations staff and provide information to FEMA leadership and program offices to improve the efficiency and effectiveness of the agency’s disaster operations. The regional role in the Continuous Improvement Program is to identify lessons learned and best practices from disaster events in their regions, conduct after-action reviews, and track corrective actions and improvement plans applicable to the region through Continuous Improvement Working Groups. FEMA officials told us that after-action report findings that cannot be resolved at the regional level are elevated to headquarters for resolution. According to FEMA officials, FEMA headquarters reviews completed after-action reports to identify any areas for improvement that may need to be addressed through changes in policies and procedures. Although FEMA’s policy requires after-action reviews be conducted after every presidentially-declared major disaster, we found that the agency does not consistently conduct after-action reviews after all major disasters and has not instituted time frames for following up on incomplete after- action reviews. As of January 2020 FEMA had completed after-action reviews for 29 percent of disasters since January 2017, with 43 percent pending or in the process of being completed, and 27 percent having been deferred (i.e., not completed or status unknown), as shown in figure 6. Our review of relevant policy indicates that FEMA does not specify time frames for when after-action reviews are to be completed. This is consistent with what we heard from FEMA officials who explained they do not have any time frames for when a certain region is to complete after- action reviews. FEMA has recently updated its Continuous Improvement Program. For example, in 2019, FEMA updated the Continuous Improvement Directive to formalize an annual Summary of Findings that consolidates the field, regional, and headquarters’ observations from the year’s incidents in order to identify the strengths, best practices, and lessons learned that should be addressed the following year. However, FEMA officials noted that this had only been done once in 2019, and would be completed in future years. Officials from FEMA’s Continuous Improvement Program in one region cited challenges with capacity, staffing, and the number of on-going after- action reviews as reasons for not being able to complete all of their after- action reports. According to FEMA officials, in 2017 each region was assigned one to two continuous improvement advisors who are responsible for developing the region’s after-action reviews. However, FEMA officials in one region said that in 2019, they faced challenges in having the staff resources necessary to operate the Continuous Improvement Program due to competing priorities, such as responding to active disasters. In addition, FEMA officials stated that due to limited staff, the regions have to prioritize which after-action reviews they can complete based on the severity and impact of the disaster. For example, in 2017, FEMA focused resources on reviewing the agency’s response and recovery for Hurricanes Harvey, Irma, and Maria. According to FEMA regional and headquarters officials, competing priorities, such as responding to active disasters, often result in staff being unavailable to conduct after-action reviews. While we acknowledge staffing is limited and that FEMA may need to prioritize completing some after-action reviews over others, FEMA officials have not established a process or framework by which regional offices are to prioritize after-action reviews. Based on our analysis of the after-action reviews since 2017 and discussions with FEMA headquarters and regional staff, we found that FEMA does not have a formal process to prioritize after-action reviews and has not established general time frames for how long following a disaster an after-action review should be completed, or followed-up on. FEMA officials agreed that having a formal process to prioritize after-action reviews, including establishing time frames for following up on incomplete after-action reviews, could provide the agency additional opportunities to improve response and recovery operations for future disasters. According to FEMA Regional officials, timely after-action reviews are useful. For example, as a result of the 2017 Hurricanes Season After-Action Report, Region II was able to update response plans for Puerto Rico, which could prove to be beneficial for future disasters. According to The Standards for Program Management, agencies should collect, measure, and disseminate performance information, analyze program trends, and point to areas in need of adjustment. In addition, leading practices for program management indicate that project schedules should be developed to define project milestones and identify and sequence activities in order to determine start and end dates for each activity. Additionally, in other branches of FEMA, the agency provides time frames for completing after-action reports. For example, states and territories are expected to submit after-action reports within 90 days of exercises that are funded by the Homeland Security Grant Program. . Similarly, FEMA policy requires Urban Search and Rescue teams to submit after-action reports 30 days after returning from deployment. Developing a process by which regional offices are to prioritize after- action reviews could help FEMA ensure that regions have a common framework to work from when determining what disasters should be prioritized for review and could help FEMA prioritize staff resources more effectively across the Continuous Improvement Program. Furthermore, establishing time frames for following up on incomplete after-action reviews could provide FEMA with greater assurance that the reviews will be conducted in a timely fashion, so that other FEMA Regions and key stakeholders can benefit from the lessons learned. As described earlier, FEMA regional offices Continuous Improvement Working Groups are responsible for developing and tracking, to the extent possible, corrective actions and best practices identified through after- action reviews. These working groups are to elevate to FEMA headquarters any issues that cannot be resolved at the regional-level. However, FEMA does not have a formal mechanism at the headquarters level for documenting and tracking best practices, lessons learned, and corrective actions that have been elevated from the regional working groups. According to FEMA, it has taken steps to track best practices and lessons learned through a serious of Microsoft Excel files, but it is not a long term or ideal operating solution due to its lack of accessibility, ease of use, and ability to be queried. In February 2016, we recommended that FEMA implement a process to document, track, and analyze recommendations and implement lessons learned after disaster deployments. FEMA concurred with this recommendation and implemented the recommendation by using the Department of Defense’s Joint Lessons Learned Information System as its primary system to capture and manage lessons learned data. However, according to FEMA officials, as of July 2019, it no longer uses the system to capture lessons learned data. FEMA officials also said the Joint Lessons Learned Information System was not user-friendly. FEMA officials stated that they hold a quarterly meeting, as required by FEMA Directive 107-1, with FEMA’s Associate Administrators to review national priorities and issues that have been elevated to headquarters for resolution. According to FEMA officials, this group performs the function that a Continuous Improvement Working Group does at the regional level by monitoring issues that need adjudication by senior management officials. While the quarterly meeting may be helpful, it does not serve as a mechanism, such as a data system, for documenting and tracking best practices, lessons learned, and corrective actions identified after a major disaster. Additionally, continuous improvement coordinators from the regions we interviewed stated that once a finding is elevated to FEMA’s headquarters, in general the region does not have visibility into what steps, if any, FEMA headquarters is taking or plans to take to address the issue. Having a mechanism, such as a database, to record after-action report findings, such as corrective actions or best practices, could help FEMA facilitate awareness across the agency about the status of FEMA’s efforts to address them. According to the Post-Katrina Act, FEMA should conduct remedial action tracking and long-term trend analysis. Furthermore, the National Response Framework specifies that evaluation and continual process improvement are cornerstones of effective preparedness. The framework notes that effective practices with continuity planning ensures the capabilities contained in the framework can continue to be executed regardless of the threat or hazard. Without a mechanism to document and track best practices, lessons learned, and corrective actions identified through after-action reviews across the regions and headquarters, FEMA may not be able to provide assurance that it is effectively leveraging best practices and lessons learned or taking corrective actions to improve its response and recovery programs. As described earlier, the Post-Katrina Act requires FEMA to generate and disseminate, as appropriate, after-action reports to participants in exercises and real-world events. In addition, FEMA’s stated policy on knowledge sharing after disasters is to collaborate with public and private sector partners to share insights on critical issues facing emergency management, promote best practices, and discuss ways in which FEMA itself can improve. However, based on a query of FEMA’s website for after-action reports on disasters, since January 1, 2017, FEMA has placed on-line one after-action report on the 2017 hurricane season. In addition, state officials from Florida, as well as officials from ten localities told us that there has been no communication from FEMA specifically in regards to its 2017 Hurricane Season After-Action Report to ask jurisdictions to provide feedback on the final product or its findings. In addition to FEMA not communicating with jurisdictions about its final product or its findings, state and local officials we spoke with said that FEMA does not consistently share after-action reports with affected jurisdictions. For example, officials from the state of Florida and four localities told us that FEMA does not consistently share its reports after each disaster, while officials from the state of California stated that FEMA has regularly shared after-action reports from disasters. One FEMA regional official noted that it would be helpful to know who, when, and to what extent lessons learned should be shared with external partners. Further, according to FEMA, knowledge sharing allows communities impacted by disasters to prioritize investments and anticipate known challenges during disasters. According to The Standards for Program Management, agencies should collect, measure, and disseminate performance information and analyze program trends, and point to areas in need of adjustment. FEMA has guidance for sharing after-action reports internally within the agency, but according to FEMA officials has not developed guidance for when after-action reports, or findings from after-action reports, should be shared with external stakeholders. According to some state and local officials we spoke with, having access to disaster after-action reports could be useful to FEMA’s external stakeholders. For example, because FEMA’s 2017 Hurricanes Season After-Action Report was accessible, New York City officials said they were able to be proactive in areas that needed to be strengthened in the event of delayed federal assistance, such as providing disaster housing services. Lessons learned can be produced through after-action reports and are relevant to key stakeholders, such as state and local governments, which are instrumental in disaster preparedness, response, and recovery, and would play a key role in any future disasters. However, without guidance to help officials determine when it is appropriate to share after-action reports, FEMA may miss opportunities to share lessons learned. Further, FEMA’s Strategic Plan states that building a culture of preparedness requires continued learning, improvement, innovative ideas, and engagement of the whole community. As such, all sectors of society, including governments, nonprofit organizations, and the private sector, will need to be involved in preparedness for future disasters. The plan further states that insights can be gained through observations from after- action reports and through feedback from stakeholders. A FEMA official from one of the region’s Continuous Improvement Program agreed that developing guidance to determine when it is appropriate to share after- action reports, could help stakeholders better prepare for future disasters. By developing guidance for sharing after-action reports or their relevant findings—when appropriate—with key external stakeholders, FEMA could help communities better prepare for future disasters through knowledge sharing. FEMA has taken numerous steps to continue to strengthen national preparedness, such as distributing grant funds. However, FEMA has not fully defined the capability gaps and determined what steps are needed to enhance capabilities across all levels of government. Informing key stakeholders, such as the Office of Management and Budget and Congress, about what resources will be necessary to address the gaps— across all levels of government—will be critical in addressing the nation’s emergency management capability gaps. In addition, opportunities exist to enhance FEMA’s after-action review process. More specifically, until FEMA prioritizes when—and for what disasters—after-action reviews should be completed and establishes time frames for following up on incomplete after-action reports, the agency will not be able to guarantee that FEMA and its stakeholders can leverage lessons learned from recent disasters and apply corrective actions before future disasters occur. Further, without a mechanism to document and track best practices, lessons learned, and corrective actions throughout the agency, FEMA may not be able to effectively leverage best practices and lessons learned or implement corrective actions to improve its response and recovery operations. By addressing areas needing improvement (i.e., corrective actions) once after-action reviews are completed, FEMA could improve response and recovery operations in the wake of future disasters. In addition, FEMA could help communities better prepare for future disasters by developing guidance to share its after- action reports or findings from its after-action reports—when appropriate—with key stakeholders, allowing them to provide feedback on the findings or adjust their own operational plans to be better prepared to work with FEMA during future disasters. We are making the following four recommendations to the FEMA Administrator: Following the completion of the 2021 National Preparedness Report, determine what steps are needed to address the nation’s emergency management capability gaps across all levels of government and inform key stakeholders, such as the Office of Management and Budget and Congress, about what level of resources will be necessary to address the known gaps. (Recommendation 1) Develop guidance to help determine which after-action reviews should be prioritized based on factors, such as the severity of disasters and availability of staff and resources to conduct the review, and implement time frames for following up on incomplete after-action reports. (Recommendation 2) Develop a mechanism to consistently track best practices, lessons learned, and corrective actions that have been elevated to headquarters for resolution. (Recommendation 3) Develop guidance on sharing after-action reports and their relevant findings with external stakeholders, when appropriate. (Recommendation 4) We provided a draft of this report to the Department of Homeland Security (DHS) for their review and comment. DHS provided written comments, which are reproduced in appendix I. In its comments, DHS concurred with the four recommendations and described actions under way or planned to address them by March 31, 2022. DHS provided technical comments, which we incorporated as appropriate. DHS concurred with our first recommendation to determine what steps are needed to address the nation's emergency management capability gaps across all levels of government and inform key stakeholders about what level of resources will be necessary to address the known gaps. According to DHS, this recommendation is consistent with the requirements outlined in the Disaster Recovery Reform Act of 2018 (DRRA) noting that FEMA complete a national preparedness assessment of capability gaps at each level based on tiered, capability-specific performance objectives to enable prioritization of grant funding; and identify the potential costs for establishing and maintaining those capabilities at each level and determine what capabilities federal agencies should provide. DHS also stated that while the 2020 National Preparedness Report will include a nation-wide assessment of community capability against national capability targets to help understand gaps and inform grant investments, it will not include data on federal capabilities. The collection of that information, through the National Stakeholder Preparedness Report, was scheduled to begin in 2020 but was delayed due to response operations for the COVID-19 pandemic. According to DHS, this information will be incorporated into the 2021 National Preparedness Report, helping to form a more complete picture of national capabilities. FEMA stated that the costs to address the nation’s resource gaps cannot be estimated without first accounting for existing federal capabilities. According to DHS, the anticipated date for the 2020 National Preparedness Report, pending response operations to the COVID-19 pandemic, is October 30, 2020, and the 2021 National Preparedness Report is planned to be released in October 2021. DHS stated that once the 2021 National Preparedness Report is released, FEMA will develop and socialize a plan to work with the federal interagency to identify resources needed to address the national gaps identified in the 2021 National Preparedness Report. If implemented effectively, these actions combined with the steps taken to inform key stakeholders could meet the intent of our recommendation. Due to the impacts of the COVID-19 pandemic and the need to finalize the 2021 National Preparedness Report prior to being able to account for the federal government’s existing capabilities, we are adjusting the wording of this recommendation to follow the issuance of the 2021 National Preparedness Report. DHS estimates the expected completion date to be March 2022. DHS concurred with our second recommendation to develop guidance to help determine which after-action reviews should be prioritized and implement timeframes for following up on incomplete after-action reports. According to DHS, FEMA will address the prioritization of disaster after- action reports as the Continuous Improvement Program’s first priority for 2020. Additionally, FEMA plans to identify and develop timeframes for following up on after-action reports as part of a broader program evaluation effort in 2020. These actions, if implemented effectively, could meet the intent of our recommendation. While FEMA originally anticipated completing this guidance during 2020, the COVID-19 response extended this timeline. DHS estimates the expected completion date to be March 31, 2021. DHS concurred with our third recommendation to develop a formal mechanism to consistently track best practices, lessons learned, and corrective actions. DHS stated that FEMA, in December 2019, implemented an issue elevation and resolution system for tracking best practices, lessons learned, and corrective actions that are elevated to FEMA headquarters level for resolution, as appropriate. However, according to FEMA in April 2020, the agency has taken steps to track best practices and lessons learned through a serious of Microsoft Excel files, but this is not considered to be a long term or ideal operating solution due to its lack of accessibility, ease of use, and ability to be queried. Further, in April 2020, FEMA stated that it is working to identify resources to build an actual application that will be used for this purpose. These actions, if implemented effectively, could meet the intent of our recommendation. DHS concurred with our fourth recommendation that FEMA develop guidance on sharing after-action reports and their relevant findings with external stakeholders, when appropriate. According to DHS, FEMA is drafting program guidance for the Continuous Improvement Program to address the sharing of after action reports and their relevant findings with external stakeholders. These actions, if implemented effectively, could meet the intent of the recommendation. Due to the ongoing COIVD-19 pandemic, FEMA estimates its completion date to be March 31, 2021. We are sending copies of this report to the Secretary of Homeland Security, the FEMA Administrator, and the appropriate congressional committees. If you or your staff have any questions about this report, please contact me at (404) 679-1875 or curriec@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. In addition to the contact named above, Aditi Archer (Assistant Director), Robert Denton Herring (Analyst-in-Charge), Erin Guinn-Villareal, James Lawson, Ben Ayres, Eric Hauswirth, Tracey King, Amanda Miller, Kevin Reeves, and Minette Richardson made significant contributions to this report.", "summary": "FEMA uses the National Preparedness System to help assess the nation's emergency management capabilities in preparing for disasters and, in part, to help prioritize federal preparedness grants it provides to state and local jurisdictions. Since 2002, FEMA has provided over $52 billion in such grants intended to enhance preparedness capabilities. GAO was asked to examine national preparedness. This report examines the extent to which: (1) FEMA's National Preparedness System and associated preparedness grants have assisted jurisdictions in preparing for disasters; (2) FEMA has strengthened the National Preparedness System and what steps remain; and (3) FEMA is using after-action reports to identify lessons learned and strengthen future preparedness. GAO evaluated agency guidance, analyzed 2013 to 2017 capability data—the most current available; conducted site visits to five states; and interviewed FEMA, state, and local emergency management officials. The Federal Emergency Management Agency's (FEMA) National Preparedness System and associated grants have helped build some emergency management capabilities, but gaps remain. Capabilities fall in five mission areas: (1) prevention—preventing imminent acts of terrorism, (2) protection—protecting citizens and assets, (3) mitigation—mitigating the loss of life and property, (4) response—responding quickly to save lives, and (5) recovery—timely restoration of infrastructure and housing, among other things. From fiscal years 2013 through 2018, jurisdictions directed almost 90 percent of FEMA preparedness grants ($7.3 of $8.3 billion) to capabilities in the crosscutting (i.e., benefit all five mission areas), response, and prevention areas (figure below). Jurisdictions reported a higher level of preparedness in these areas compared to capabilities in the other mission areas—recovery, mitigation, and protection. Jurisdictions have consistently rated select capabilities in these three mission areas—such as disaster housing and cybersecurity—in the lowest category since 2013. FEMA does not limit jurisdictions' use of preparedness grants for select capabilities, but it has encouraged jurisdictions to address the known gaps. FEMA is taking steps to strengthen the national preparedness system, but has yet to determine what steps are needed to address the nation's capability gaps across all levels of government. Specifically, FEMA is implementing a new methodology to collect more quantitative data on capabilities at the state, territory, and local levels—as GAO recommended in 2011—and also plans to begin assessing the federal government's capabilities. Including the federal government in such an assessment would enable FEMA and jurisdictions to assess national preparedness capabilities collectively. While these are positive steps that could meet the intent of the 2011 recommendation, FEMA has yet to determine what steps are needed to address the capability gaps once they are identified, including jurisdictions' capability gaps that have been known since 2012. By determining these steps and informing key stakeholders, such as Congress, about what resources will be needed across all levels of government, FEMA will be better positioned to address the nation's capability gaps. FEMA after-action reports have identified areas for improvement and lessons learned following disasters, but has completed after-action reviews for only 29 percent of disasters from 2017 through 2019. FEMA lacks a formal mechanism to track corrective actions and does not have guidance on sharing after-action reports with key external stakeholders, as appropriate. GAO is making four recommendations that FEMA (1) determine what steps are needed to address emergency management capability gaps, and communicate it to key stakeholders (2) prioritize completion of after-action reviews, 3) track corrective actions, and (4) develop guidance on sharing findings externally. The Department of Homeland Security concurred and FEMA is taking actions in response.", "document_type": "gao"}
{"report": "DOE has issued PILT orders and policies to articulate DOE’s procedures for carrying out the PILT provision of the Atomic Energy Act. DOE has changed its PILT procedures over time, which is reflected in multiple PILT orders and policies. These changes modified eligibility requirements for PILT, as well as how PILT payments were to be calculated. In 1958, a predecessor agency to DOE issued the first order on PILT. Under the order, payments were to be based on the property value when the land was acquired and the tax rate of the year for which the payment was made; however, it allowed for exceptions to this rule. The 1958 order also allowed DOE to pay sites retroactively for years prior to their initial PILT application. In 1987, DOE issued a new PILT order with changes to address budget constraints. The new order introduced more stringent requirements for new PILT applicants; prior PILT recipients were not subject to the new restrictions. The 1987 order included an eligibility requirement called a “gross benefits test.” Under this requirement, payments were only allowed if the tax loss that was incurred exceeded the total value of all benefits derived from DOE’s activities in the community. The 1987 order also included a provision that required payments to be reduced by the amount of tax benefits a community received from DOE’s activities and eliminated retroactive payments to communities for the years prior to their application for PILT. In 1993, DOE revised its policy in response to concerns about inequities arising from the application of the 1987 order. Specifically, the 1993 policy eliminated the gross benefits test and modified the provision that required payments to be reduced to account for tax benefits from DOE activities. In addition, it allowed payments to all communities to be based on the current tax rates and value of the property in the condition in which it was acquired. In 2003, DOE issued its most recent PILT order. This order updated responsibilities outlined in the 1993 policy and shifted some details to a separate policy document. It also eliminated a detail of the 1993 policy regarding special burdens payments. In order for a community to be eligible for PILT payments, it must submit to DOE an initial PILT application. DOE uses the one-time initial application to establish the eligibility of land at a certain community. Officials from the relevant DOE site and program offices, along with officials from DOE’s Office of the Chief Financial Officer (CFO), Office of Management, and General Counsel at DOE headquarters, evaluate the application based on several criteria, such as: (1) the property must have been subject to taxation by local or state authorities immediately prior to being acquired by the federal government, (2) payments must not be retroactive, (3) payments should not be in excess of the taxes that would have been collected if the property had remained on the local tax rolls in the condition in which it was acquired, and (4) property values will be based on the highest and best use of the property based on the classification of the property when it was acquired. The CFO makes the final determination of whether to approve or reject the application. Once an application is approved, DOE and the community enter into an intergovernmental assistance agreement, which emphasizes that payments are subject to the availability of funds and to legislative or administrative reductions and states that PILT is not an entitlement to the community. After establishing eligibility through the application process, each community submits to DOE an annual PILT invoice reflecting its requested PILT amount. These annual PILT invoices specify how much a community estimates its PILT payments should be based on the community’s calculations for a specific tax year. DOE site offices—offices at various DOE sites across the United States that report to DOE program offices— review each PILT invoice and determine whether enough funding is available to pay the amount requested in the PILT invoice. If a community’s PILT invoice reflects a reclassification of the property to a new tax classification or category, a change in the amount of eligible land, or another significant change in the method of calculating the requested PILT payment by the community, the community must submit a new PILT application. PILT processes involve multiple organizations, including several parts of DOE as well as local governments (see fig. 1). DOE headquarters— including the CFO, Office of Management, and General Counsel, and program offices—is responsible for reviewing and approving initial or revised PILT applications. The CFO and program offices are responsible for ensuring that funding needed for PILT payments is included in budget requests. As of fiscal year 2019, the program offices involved with PILT include: the Office of Environmental Management, which has the mission to clean up sites contaminated by nuclear weapons development and nuclear energy research; the National Nuclear Security Administration, which is responsible for maintaining and enhancing the safety, reliability, and performance of the U.S. nuclear weapons stockpile; the Office of Science, which manages national laboratories and supports research of physics, materials science, and chemistry; the Office of Nuclear Energy, which focuses on research, development, and demonstration of nuclear reactors; and the Office of Legacy Management, which is responsible for providing long-term surveillance and maintenance of DOE sites that have closed. Under the current PILT order, DOE site offices are responsible for providing recommendations for any initial and revised PILT applications and for administering payments. These DOE site offices operate in their PILT recipient communities. DOE site offices are overseen by DOE program offices. For example, cleanup activities related to nuclear weapons production at the Hanford and Savannah River sites are overseen by the Office of Environmental Management, while the Argonne and Brookhaven National Laboratories are overseen by the Office of Science. The site of the now closed Fernald Plant is overseen by the Office of Legacy Management. At some sites, multiple communities at the site receive PILT payments. For example, three communities at the Oak Ridge site receive PILT payments: the City of Oak Ridge, Anderson County, and Roane County. Property taxes in the United States are levied by a number of different taxing authorities, including state and local governments, but mostly by local governments. Local governments, such as counties, can levy and collect taxes on behalf of smaller jurisdictions within their boundaries. Broadly speaking, property taxes are based on the assessed value of the property times the tax rate. Assessed value. The assessed value of the property is generally a function of the market value and the assessment ratio. The market value depends on the characteristics of the property and can vary across locations as a result of local conditions, including the supply and demand for the type of property. The assessment ratio is a percentage modifier applied in certain circumstances to alter the market value of the property. Some states and counties apply a lower assessment ratio to certain classifications of property, such as agricultural property. Tax rate. The tax rate is a figure—typically in the form of a percentage—that is applied to the assessed value of the property to determine the total property tax amount. Tax rates vary across locations, depending on local and state tax laws and policies. In addition, for a given property tax bill, local governments may apply a wide variety of tax rates, with different rates applied for different government-supported functions, such as education, emergency services, and roads. The classification of the property can thus influence the tax rates. PILT payments vary considerably across DOE sites, with the communities at two sites with the most eligible land receiving the majority of payments. Total PILT payments made to communities at the 12 DOE sites that receive PILT payments have increased from approximately $9.5 million in 1994 to approximately $23 million in 2017 in fiscal year 2017 dollars. Payments to communities at the Hanford and Savannah River sites account for the majority of that growth. According to DOE, communities at the majority of DOE sites do not receive PILT payments because they are ineligible for PILT or have not applied to receive payments. Specifically, of the 74 DOE sites, communities at 44 sites are ineligible for PILT. Of the 30 sites where communities are eligible or potentially eligible, 18 have communities that have not applied for PILT or currently do not receive PILT, while communities at 12 sites currently or recently received PILT as of 2017, according to DOE documents. Of the over 2 million acres covered by DOE sites, approximately 70 percent—approximately 1.5-million acres—is ineligible for PILT, according to documents provided by DOE. According to DOE, communities at most of the 44 ineligible sites are not eligible under the provisions of the Atomic Energy Act because they are on property that either: was not on local tax rolls prior to acquisition, is private land, is land controlled by another federal agency, or is university-owned. Some examples of property that is ineligible include: the Waste Isolation Pilot Plant, New Mexico, which is situated on federal land and thus not subject to prior state or local taxation; Hazelwood Interim Storage Site, Missouri, which is on land DOE leases from a private owner; Sandia Lab, Kauai, Hawaii, which is on land controlled by another federal agency; and the Radiobiological Laboratory of Utah, Utah, which is on university-owned land. In addition, in some cases, sites include a mix of eligible and ineligible acreage. Of the approximately 680,000 acres of property at the 30 sites that are eligible or potentially eligible for PILT, about 25 percent is located at the 18 sites where the communities did not receive PILT payments, according to fiscal year 2017 data provided by DOE. Examples of those sites with eligible property that have not received payments include the Weldon Spring Quarry in St. Charles County, Missouri, and the Atlas Complex in Clark County, Nevada. DOE headquarters officials that we spoke with stated that they are unsure why some communities with eligible property have not applied for PILT. Of the property that is eligible for PILT, approximately 75 percent is located at the 12 sites where the community has applied for and receives PILT payments. These sites began receiving payments at least as early as the 1950s and as late as 2012. Some sites are located in communities that previously, but no longer, receive PILT payments. For example, the community at the Mound Site, which is under the Office of Legacy Management, received its last payment in 2006. Figure 2 shows PILT eligibility and receipt by site and by acreage. In fiscal year 2017, communities at 12 DOE sites received or had pending PILT payments. These sites are located in 10 states. The sites vary in size and the amount of land at the site that is eligible under DOE’s PILT order. The two largest sites in terms of eligible acreage—Hanford and Savannah River—are the only sites that have more than 100,000 PILT- eligible acres, at nearly 180,000 and 200,000 respectively. Although the Idaho site includes about 570,000 acres, according to DOE officials, only 5 percent of those are eligible for PILT because they were previously on local tax rolls when DOE acquired the land, while the rest of the land was not on the tax rolls. Five sites—Brookhaven National Laboratory, Argonne National Laboratory, the Fernald Plant, Los Alamos National Laboratory, and Bettis Atomic Power Laboratory—have total PILT-eligible acreage of less than 2,000 acres, with the smallest, Bettis Atomic Power Laboratory, having around 200 PILT-eligible acres. Figure 3, below, shows the name, location, and PILT-associated acreage of DOE sites where local communities received PILT payments in 2017 or had pending PILT payments. Payments to communities at the 11 DOE sites that received PILT payments in fiscal year 2017 varied considerably, from less than $65,000 to more than $9 million, totaling over $23 million. Communities at the Hanford and Savannah River sites, representing over 75 percent of all PILT-eligible acreage, received approximately 70 percent of total PILT payments—approximately $9.7 million and $6.5 million, respectively. Of the communities at the remaining 9 sites, communities at 2 received more than $1 million, and communities at 2 received less than $100,000. Figure 4 shows payment amounts for the communities at the 11 sites that received payments in fiscal year 2017. See appendix III for detailed information on PILT payments from 1994 to 2017. Growth in PILT payments since 1994 is primarily a result of increases in payments to communities at two sites—Hanford and Savannah River—in addition to new PILT recipient communities at DOE sites. Since 1994, total annual PILT payments have grown from $8,582,446 to $23,170,049 in fiscal year 2017 constant dollars, as figure 5 shows. Since 1994, increases in payments to the communities at the Hanford and Savannah River sites are responsible for the nearly 60 percent of remaining total growth in PILT payments. PILT payments have increased from a total of over $19 million in 2012 to over $23 million by 2017 in real terms. Nearly all of that growth in total payments during that time is a result of higher payments to communities at the Hanford site, which community and DOE site officials attributed to increases in local land value resulting from the growth in agriculture in the region. PILT payments to the three communities at the Hanford site increased by 43 percent, or nearly $3 million, in that time frame. Communities at the Hanford site were not the only ones to experience a large payment growth rate. PILT payments to communities at two other sites, Pantex and Idaho National Laboratory, increased by approximately 90 percent and 55 percent respectively over the same time period; however, this growth was approximately $100,000 and $85,000 respectively for those communities and therefore did not account for much of the overall growth in PILT payments. The majority of communities that currently receive PILT payments began receiving them beginning in or after 1994. DOE’s 1993 policy eliminated the gross benefits test and modified a provision that required payments to be reduced by the amount of tax benefits a community received from DOE’s activities. These changes allowed for additional sites to enter into PILT agreements with DOE and allowed other sites to obtain higher payment amounts. Since 1994, communities at seven additional sites were approved for and have begun receiving PILT payments. The addition of these new PILT recipient communities after the 1993 policy change, primarily Brookhaven National Laboratory, is responsible for approximately 15 percent of the growth of total annual payments. Variations in PILT payments across sites are largely due to differences among the sites, including the different histories and market conditions at each site. However, the PILT order’s lack of requirements about PILT documentation, review of PILT invoices, and payment determinations has limited DOE’s ability to provide adequate assurance that payments fully reflect the terms of their original agreements and consistently meet PILT goals. The goal of PILT, as stated in the Atomic Energy Act and reflected in DOE’s order implementing the act, is to render financial assistance to communities, while generally not making payments in excess of the taxes that would have been payable for the property in the condition in which it was acquired. DOE officials stated that an additional PILT goal is to compensate communities for the revenues they would have received under those conditions. Although the order does not require payments to reflect the revenues communities would have received, it states that, on a case-by-case basis, PILT payments will be based on the same assessment values and tax rates that the communities apply to comparable properties with the same use and/or tax classification. Since these values and rates differ between sites, payments may also differ under the order. Consistent with DOE’s PILT order, PILT payments to communities vary given the characteristics of the property, market conditions, and tax policies applied at each site, in order to reflect the revenue the communities would have received had the property remained on their tax rolls. DOE generally bases PILT payments on the recipient communities’ estimates of the property taxes they would have received. The communities calculate their estimated payments and then communicate their requested payment amounts in annual invoices to DOE. DOE does not prescribe the use of a particular formula by communities seeking payments. However, DOE officials noted that communities usually base the calculations they use to develop their annual PILT invoices on property taxes and that they generally calculate these using a relatively standard formula. Key information in this calculation includes the amount of land, its estimated value, assessment ratio, and the property tax rate (see figure 6). Differences in PILT payments to different sites are generally not a function of variations in the payment formula, but rather of variations among the inputs into the formula, although DOE has sometimes altered payments in other ways. Based on our analysis of PILT payments in fiscal year 2017, we found that values of property, assessment ratios, and property tax rates vary across DOE sites and communities. The assessed value of the property is partially determined by characteristics, or history, of the property and market conditions. State and local tax policies may determine both the assessment ratio and the property tax rate. Characteristics of the property. The amount of PILT eligible property and its classification are factors that partially determine payment amounts. DOE provides the highest payments to communities at sites with the greatest amount of eligible acreage—the Hanford, Savannah River, and Oak Ridge sites. Similarly, lower acreage at some sites usually results in lower payments. For example, Los Alamos National Laboratory and the Fernald Plant are among the smallest sites and payments to these communities are among the smallest. In addition, the land use classification of the property, such as whether it was used for agricultural or commercial purposes when it was acquired, influences its value. Some classifications of land tend to have higher market values than others; for example, commercial land generally has a higher value than agricultural land. The land at the Bettis Atomic Power Laboratory site, located in western Pennsylvania, is classified as commercial property and was valued in 2017 for PILT purposes at an average of $64,476 per acre. As a result, although Bettis Atomic Power Laboratory has among the smallest acreage of any site—at approximately 200 acres—its payments are the fifth highest. In contrast, the land at the Pantex site, located in the Texas Panhandle, is classified as agricultural and homestead property and was valued in 2017 for PILT purposes at an average of $976 per acre. Market conditions. The market value of property varies across PILT sites as a result of local market conditions. Greater demand for land contributes to higher per-acre values than when there is less demand for land. This contributes to variations among land values, even within a given classification, for the communities’ annual PILT invoices to DOE. For example, irrigable agricultural land at Benton County—one of the communities that hosts the Hanford site—was valued at about $6,500 per acre in 2017, which DOE and county officials attributed primarily to high demand for agricultural property in Washington State’s Columbia Valley River Basin. In contrast, Carson County— which hosts the Pantex Plant and is in a region with lower farm real estate values and is not near a major city—valued its land at $976 per acre in 2017, as previously noted. State and local tax policies. Some states and counties reduce assessment ratios for certain types of property, such as agricultural property. For example, the assessed value of the property is reduced to a fraction of its market value. Some communities have reflected these assessment ratios in their calculations for their annual PILT invoices to DOE. Because assessment ratios can vary widely across locations—from 6 percent to 100 percent among communities that received PILT payments in fiscal year 2017—they can create large variations in PILT payments. For example, the communities at the Oak Ridge site assess agricultural property at 25 percent of the full market value, which they reflect in their annual PILT invoices to DOE. On the other hand, the Town of Brookhaven, which hosts Brookhaven National Laboratory, applied a 90 percent assessment ratio to its PILT-eligible property, which is categorized as residential. In addition, tax rates vary across communities. For example, in fiscal year 2017, the City of Oak Ridge applied a 2.5 percent tax rate to determine its payments; whereas, Carson County applied a 0.6 percent tax rate. DOE’s PILT order requires DOE to deduct from PILT payments an amount equal to any payments by the federal government that will be used by the community for the same, identifiable, discrete purpose. In practice, when communities calculate their annual PILT requests, they subtract this amount from their total payment requests. According to DOE and some community officials, communities have made these deductions to offset payments they received through the Department of Education’s Impact Aid program. DOE’s PILT order calls for communities to document key determinants of PILT payments in PILT applications, but it does not include requirements or procedures for DOE or communities to document key determinants of PILT payments after the initial PILT application. In addition, although the order lists evaluation criteria on which PILT payments should be based, it does not establish a process or requirements for DOE offices to review PILT invoices to ensure payments are consistent with those criteria. The order also does not require regular, independent—such as headquarters- level—involvement in such a review process. Lastly, the PILT order lacks specificity on how payments should be determined in certain scenarios. The PILT order’s lack of sufficient internal controls may have contributed to some cases in which payments may not reflect PILT goals. DOE’s PILT order lists application and evaluation criteria that it says will serve as the basis of PILT payments. Those criteria include factors, which we refer to as “key determinants,” such as: description of the property; tax rates and assessment values for comparable property; use and zoning classification of the property; and payments from the federal government that will be used for the same identifiable, discrete purpose. These key determinants are fundamental to determining how much revenue a community would have received if the property had remained on its tax rolls and to ensure that the communities’ PILT payments are not higher than that amount. The order calls for these key determinants to be documented in PILT applications. However, DOE’s PILT order does not require communities or DOE to document such key determinants of PILT payments at any later stage. Specifically, the order does not require DOE or communities to include this information in PILT intergovernmental agreements, which are agreements between DOE and each community and serve as a basis for obligating funding under PILT. The order also does not require communities to include such information in their annual PILT invoices that they submit to request PILT payments. GAO, Standards for Internal Control in the Federal Government, GAO-14-704G (Washington, D.C.: September 2014). key determinants of PILT payments for each community, DOE does not have adequate assurance that its payments are consistent with the agreed upon bases of PILT payments, and DOE is more likely to make payments that do not meet PILT goals. DOE’s PILT order states that “DOE plans to evaluate applications for PILT, and to calculate” PILT payments using specific guidelines based on key determinants, such as the description of the property, tax rates and assessment values for comparable property, use and zoning classification of the property, and deductions equivalent to certain federal payments; however, it does not call for a review process to determine whether calculations used for PILT invoices follow those guidelines. DOE’s PILT order calls for site, program office, and headquarters review of original and revised PILT applications. However, most original applications were developed decades ago and revised PILT applications are only required if the community would like to reclassify property, change the amount of property, or make other significant changes. DOE’s PILT order does not require independent, headquarters-level review at any later stage. The PILT order states that site offices will manage the administration of PILT payments. However, it does not specifically call for DOE organizations to review communities’ annual PILT invoices to determine whether PILT invoices follow payment calculation guidelines and do not exceed the amount communities would have received had the property remained on the tax rolls. DOE headquarters officials said that headquarters officials do not review annual PILT invoices. Some DOE CFO officials and officials at some sites stated that DOE sites treat the annual payments as bills to be paid, without applying much scrutiny. To the extent that PILT invoices are reviewed, they are reviewed at the site level by officials who may live in the same communities that receive PILT payments. DOE CFO officials stated that site offices are more knowledgeable of local tax authorities and local conditions than DOE headquarters and that they have expertise—in the form of local realty, legal, budget, and supervisory staff—that DOE headquarters staff rely on for the execution of PILT payments. Nevertheless, there may be an appearance of bias if the only review of PILT invoices is conducted at the site level by individuals who may benefit indirectly from payments to their communities. Because DOE’s PILT order lacks a requirement for review and validation of annual PILT invoices, DOE is not well positioned to determine whether communities’ payment requests in PILT invoices are consistent with DOE goals. Under federal standards for internal control, management should design control activities to achieve objectives and respond to risks, such as by comparing actual performance to planned or expected performance and analyzing significant differences. By requiring site office and headquarters review of key payment determinants in PILT invoices, DOE may realize benefits, including the ability to (1) evaluate whether PILT invoices are consistent with agreed-upon bases of PILT payments and PILT goals, and (2) ensure greater independence in the review process to avoid the appearance of bias on the part of site officials, who may live in the communities receiving PILT payments and may indirectly benefit from the payments. Without requirements for DOE site offices to review key PILT payment determinants in communities’ invoices for accuracy and consistency with the agreed-upon bases of PILT payments and PILT goals and for headquarters-level review and validation of annual PILT invoices, DOE is more likely to have payments that do not meet PILT goals. DOE’s PILT order lacks specificity about how it will determine PILT payment amounts in some scenarios. The PILT order includes information about some key determinants of PILT payments, such as tax rates, assessment values, and property classification, but the order does not provide guidance on other factors that may affect PILT payments, such as tax relief programs. In addition, the order states that the property value will exclude the value of improvements made after the federal government acquired the real property, but it does not state whether property values should include the value of resources such as timber. Last, the order states that payments will be reduced by an amount equal to any payments to the state or local jurisdiction for the same identifiable, discrete purpose. However, the order does not define the phrase “same identifiable, discrete purpose.” Under federal standards for internal control, management should design control activities to achieve objectives and respond to risks, such as by documenting internal control in management directives, administrative policies, or operating manuals. While DOE has documented some key determinants of PILT payments in its order, it does not clearly document how DOE should address tax relief programs in payment determinations. Without additional guidance in the PILT order on how communities should calculate payment requests for their PILT invoices, DOE is more likely to make payments that do not meet PILT goals, as is described in the following section. DOE does not have adequate assurance that payments are meeting PILT goals. This limited assurance that payments meet PILT goals may be in part a result of deficiencies in DOE’s internal controls for PILT. Based on our reviews of PILT documentation and interviews with DOE officials, we identified cases in which payments did not appear to meet the stated PILT goal of compensating communities for the revenue they would have received if the property had remained on the tax rolls. Specifically, we identified five examples of payments potentially not meeting goals as a result of issues with: property classification, determination of land value, application of state tax adjustments, payment deductions, and payment adjustments. Property classification. We identified a case in which payments appear to be higher than the amount communities would have received had the property remained on the tax rolls in the condition in which it was acquired. In the case of Benton County, the property classification that forms the basis of its requested PILT payments does not appear to be based on the classification of the property when it was acquired. Benton County’s original PILT agreement from 1996 shows that, when acquired, Hanford property in the county was classified as 11 percent farmland and 88 percent rangeland. However, the agreement also states that, considering uses of the land at the time of the agreement, 72 percent of the land would be treated for the purpose of PILT as farmland in the category of “irrigable land” and only 27 percent as rangeland. In 2017, irrigable land in Benton County was valued at $6,495 per acre whereas rangeland was valued at $410 per acre—higher percentages of irrigable land compared to rangeland therefore result in higher payments. Using these land classifications is inconsistent with the PILT goal that payments will not exceed the taxes that would have been payable for the property in the condition in which it was acquired. DOE headquarters officials we spoke with were not aware of this discrepancy in Benton County’s property classification. In addition, DOE did not have documentation to explain DOE’s decision, but an Office of the General Counsel official noted that DOE agreed to these terms as part of a settlement agreement at a time when a number of issues, beyond just PILT issues, were in dispute between Benton County and DOE. Because of this inconsistency in land classifications, it appears that Benton County’s payments may not have reflected the revenues the county would have received had the property remained on the tax rolls in the condition in which it was acquired. Had DOE maintained more thorough documentation and had there been independent review of PILT invoices, these higher payments might have been avoided. Determination of land value. We identified one case in which payments were not clearly linked to the revenue communities would have received if the property had remained on the tax rolls. Specifically, DOE negotiated with Savannah River Site counties to apply a dollar amount per acre that is not directly tied to assessed property values. DOE and the counties originally negotiated values in 1988 of $1,000 per acre for Aiken and Barnwell and $426 for Allendale counties. Those amounts remained flat until 2007, when DOE agreed to adjust them with a “time value of money” factor to $1,641 and $712 respectively. According to county officials, the counties and DOE agreed to use a negotiated rate rather than a rate based on current assessment values partly because of the difficulty of conducting appraisals because of the large amount of land, lack of comparable properties, and the high expense of an appraisal. Because of this reliance on a negotiated, rather than assessed value, it is unclear whether these payments reflect the revenues the counties would have received had the property remained on the tax rolls in the condition in which it was acquired. Had DOE required independent review of key determinants of PILT payments, this deviation from using assessed values might have been avoided. Application of tax relief programs. We identified a third case in which payments may have been higher than the revenue communities would have received if the property had remained on the tax rolls. With regard to the Hanford Site, the Open Space Taxation Act of Washington State is a tax relief program that community officials said allows assessment ratios of about 40 percent to be applied for land that is being used for agriculture or as rangeland. In the past, none of the three counties that receive PILT at the Hanford site applied special assessment ratios under this tax relief program in calculating PILT payments. Hanford site officials informed us that they were aware of this tax law and requested that the three counties at the Hanford site apply it. The DOE officials explained that the counties refused because DOE was not using those lands for agriculture or rangeland. The officials stated that the counties at Hanford decided that DOE did not meet the purpose and the terms of the program. However, if the land had remained on the tax rolls in the condition in which it was acquired, it could also be assumed that it might have been farmed or used as rangeland, in which case the counties may have applied the special assessment ratios. Although DOE’s order does not state whether PILT payments should take into account such tax relief programs, failure to take such programs into account may have resulted in DOE paying the counties at Hanford more than they would have received had the property remained on the tax rolls in the condition in which it was acquired, contrary to the order. If DOE’s PILT order had included more specificity about how tax relief programs should be addressed, DOE might have had greater assurance that these payments were not higher than the revenue the communities would have received had the property remained on the tax rolls in the condition in which it was acquired. Payment deductions. We identified a case in which it was unclear whether payments aligned with PILT goals. DOE has provided non- PILT funding to Los Alamos public schools and the Los Alamos fire department. According to DOE officials, DOE has annually provided $8 million to the county’s schools; DOE provided over $20 million for the county’s fiscal year 2020 firefighting services. DOE also provides PILT funding to Los Alamos County, which was $244,183 in fiscal year 2017. About a decade ago, DOE considered whether it should stop making PILT payments to Los Alamos County because of its other support for the community and the provision in the PILT order requiring deductions from PILT for other payments by the federal government that will be used for the same identifiable, discrete purpose. However, DOE has decided to continue paying Los Alamos County PILT. The county’s position is that the schools are a separate entity from the county government and that its payments should not be reduced to account for amounts received directly by the schools, but in 2017 the county nonetheless reduced its PILT request by the amount it would have provided to Los Alamos schools. It is unclear how the PILT order should be applied in situations like this where payments, including PILT payments, are made to multiple entities. Making continued payments in such a situation, however, may exacerbate perceptions of inequities across sites. If DOE’s PILT order had included more specificity about the reduction of payments to account for other federal payments for the same identifiable, discrete purpose, DOE might have had greater assurance that these payments meet PILT goals. Payment adjustments. We identified a case in which the PILT order’s lack of specificity led to uncertainty for PILT payment recipients when DOE’s payments did not align with the communities’ calculations of what the communities determined they would have received if the property had remained on the tax rolls. When the PILT invoices from the three counties at the Hanford Site increased by about 73 percent in real terms from a total of about $6 million in 2010 to about $10.7 million in 2017, DOE began providing payments that were lower than what the counties requested in their PILT invoices. Specifically, in 2017, DOE provided 91 percent of what the counties requested, and in 2018 DOE provided 65 percent of what they requested, which DOE officials said was because payment requests exceeded the amounts set aside for PILT purposes. DOE did not cite problems in the counties’ PILT invoices or document problems with the counties’ PILT invoices. Payment adjustments are allowable under the PILT order—both the Atomic Energy Act and DOE’s PILT order give DOE discretion as to payment amounts. However, because the order also lists key determinants for PILT payments that are based on the taxes communities would have received had they remained on the tax rolls and because DOE has typically provided what communities have requested, communities we spoke with said they began to rely on PILT in their budget formulations. The communities had developed their budgets based on the assumption that payments would align with the amounts they determined they would have received had their property remained on the tax rolls, but it is now difficult for them to plan ahead with the new uncertainty. In response to this uncertainty in the payment amount, in 2019, one of the counties at Hanford—Benton County—provided DOE with a PILT invoice that was about $5 million lower than the previous year. According to the county officials we spoke with, the goal of providing a lower PILT payment invoice was to increase the likelihood that they would receive the full amount. DOE’s order does not include any information about under what conditions DOE will adjust payments— such as if payments calculations are not consistent with PILT payment determinants—to guide DOE’s oversight. The order also does not require DOE to document or communicate such information ahead of time. Had DOE’s PILT order included more specificity on these topics, communities might have had more clarity regarding whether their payment calculations were consistent with PILT goals and whether they were likely to receive the amounts they requested. PILT payments help replace tax revenue that communities are no longer receiving because of DOE’s acquisition of property in their communities. Our past work reported that DOE allowed different standards for PILT invoices at different sites, depending on when the community applied for PILT payments, raising concerns about inequitable treatment of communities. In 1993, DOE updated its PILT order to address one of these concerns by eliminating the gross benefits test that had been applied to new communities. However, some concerns remained. DOE intentionally allows payments to communities to vary across locations because property characteristics, market conditions, and tax policies differ; this variance enables payments to reflect the taxes the communities would have received if the property had remained on local tax rolls. However, DOE’s PILT order lacks: (1) requirements for documenting key determinants of PILT payments in intergovernmental agreements and invoices, (2) requirements for independent review of PILT invoices for consistency with agreed-upon bases of payments, and (3) specificity about payment determinations in certain scenarios. This has resulted in a relatively hands-off approach to management and oversight of communities’ annual PILT invoices as well as some uncertainty about how to determine PILT payments. This is inconsistent with federal internal-control standards and has limited DOE’s ability to provide adequate assurance that DOE is meeting PILT goals. Until DOE strengthens its internal-control activities, communities may continue to perceive that there are inequities in PILT, and DOE will not be able to provide adequate assurance that it is meeting PILT goals. We are making the following three recommendations to DOE: The Secretary of Energy should direct DOE’s Office of the Chief Financial Officer to revise DOE’s PILT order to require DOE to maintain documentation of key determinants of PILT payments for each community to help ensure that payments are consistent with the agreed-upon bases of PILT payments and PILT goals. (Recommendation 1) The Secretary of Energy should direct DOE’s Office of the Chief Financial Officer to revise DOE’s PILT order to require DOE site offices to review key determinants of PILT payments in communities’ PILT invoices for accuracy and consistency with the agreed-upon bases of PILT payments and PILT goals and for DOE headquarters to document its review and validation of site office determinations. (Recommendation 2) The Secretary of Energy should direct DOE’s Office of the Chief Financial Officer to revise DOE’s PILT order to provide additional guidance on how communities should calculate their payment requests for their PILT invoices. (Recommendation 3) We provided a draft of this product to DOE for review and comment. In its comments, reproduced in appendix IV, DOE neither agreed nor disagreed with our recommendations but did describe actions that it intends to take in response to our recommendations. DOE stated that it will undertake a comprehensive assessment of the PILT program, its objectives, and the manner in which DOE accomplishes PILT’s objectives. DOE also stated that it will convene a working group to identify high-level options for PILT and recommend appropriate changes, if necessary, to DOE leadership. Although further analysis of PILT could be worthwhile, we believe our review sufficiently demonstrated that DOE’s PILT order lacks sufficient internal controls. As a result, we continue to believe that implementing our recommendations for revising the PILT order could provide better assurance that payments meet PILT goals. DOE also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Energy, and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have questions about this report, please contact David C. Trimble at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. The objectives of our review were to assess: (1) how, if at all, PILT payments vary across sites and how they have varied over time, and (2) reasons for variations in payments and the extent to which the Department of Energy (DOE) is providing assurance that payments meet PILT goals. To assess how, if at all, PILT payments vary across sites and how they have changed over time, we obtained and analyzed documentation from DOE regarding total number of DOE sites, their eligibility for PILT, and reasons for lack of eligibility, when applicable. We analyzed DOE documentation of eligible acreage at sites that are affiliated with communities that receive PILT payments and compared this with acreage of DOE property that is not eligible for PILT. Idaho National Laboratory, Idaho: Bingham County, Butte County, Clark County, Jefferson County Office of Legacy Management: Fernald Plant, Ohio: Hamilton County We took several steps to assess the reliability of PILT payment data. We collected data in two phases. The first used PILT datasets that DOE had collected prior to our review. These covered years 1989–2009 and 2012– 2017. We used those data to develop a preliminary understanding of how PILT payments varied across sites and over time. We asked DOE to collect a second, complete, data set for the purpose of our review. That data set covered years 1994-2017. Using these data, we identified possible outliers and missing data and interviewed relevant agency officials at the headquarters, field office, and site office level to determine the extent to which the data were reliable. In addition, we interviewed relevant agency officials at the headquarters, field office, and site offices regarding their internal data reliability and data control measures. A number of written questions regarding their annual PILT invoices, PILT payments, federal offsets, and other related topics that were responded to by all 12 site offices. We also requested DOE payment information that would allow spot checking of the data that DOE provided. We requested that each of the 12 sites provide documentation of their payments for one in every 5 years between 1994 and 2017. We compared this documentation with data DOE submitted for those years to spot check the data for accuracy. We reviewed past GAO reports on PILT and past GAO and DOE reports on DOE financial management systems. We determined the data to be sufficiently reliable for our purposes. For both objectives, we conducted interviews with or obtained written responses from the following DOE offices, which included representatives of all of the sites that received recent PILT payments: DOE headquarters: Office of the Chief Financial Officer and General Counsel. DOE program offices that manage sites hosted by PILT-recipient communities: National Nuclear Security Administration, Office of Environmental Management, Office of Legacy Management, Office of Nuclear Energy, and Office of Science. DOE site offices hosted by PILT-recipient communities: Argonne National Laboratory, Bettis Atomic Power Laboratory, Brookhaven National Laboratory, Fernald Plant, Knolls Atomic Power Laboratory, Hanford site, Idaho National Laboratory, Los Alamos National Laboratory, Oak Ridge site, Pantex Plant, Portsmouth site, and Savannah River site. To assess reasons for variations in payments, we identified how DOE communities calculate their requested PILT payment amounts and how DOE officials determine how much DOE will pay. We reviewed DOE’s PILT order, DOE Order 143.1, to determine how DOE specifies payments are to be calculated. We also interviewed DOE site office officials about how they expect communities to determine their requested payment amounts. We compared DOE expectations regarding annual payment request calculations with PILT invoices that communities submit to request payments. Because communities appeared to generally calculate payments to align with expected property tax revenue they would have received had the DOE-acquired property remained on the tax rolls in the condition in which it was acquired, we compared this information with information on how local and state governments determine property taxes. When we needed further clarification about how communities had determined their requested payment amounts, we sent follow up questions to DOE site officials regarding the PILT invoices they had reviewed. Once we identified how communities calculate PILT invoices, we analyzed communities’ fiscal year 2017 payment request documentation to determine how factors—such as characteristics of the property, market conditions, and state and local tax policies—influence payment amounts. We interviewed DOE site officials and some community officials, at the communities that received some of the largest payments, about instances when payments varied from what communities requested. We analyzed PILT invoices, agreements, and payment data to identify how communities and sites had determined and documented key determinants and decisions, such as property classification, deductions because of other federal payments, land values, and assessment rates. We analyzed DOE’s PILT order to identify PILT goals and requirements related to: PILT payment determinations, DOE review of communities’ PILT invoices, and PILT documentation. We compared this with federal standards for internal control. We interviewed officials from selected communities that received some of the largest payments to determine how they used PILT payments, how they assess land value, and challenges they have faced with PILT. These communities included all communities at the two sites with the largest aggregate PILT payments in fiscal year 2017: Benton, Franklin, and Grant counties at the Hanford site and Aiken, Allendale, and Barnwell counties at the Savannah River site. Regarding these same topics, we also interviewed staff at community organizations that represent communities that host DOE sites, including: the Energy Communities Alliance and the National Association of Counties. Findings from these communities at two sites and two community organizations cannot be generalized to those we did not interview as part of our review. We conducted this performance audit from October 2018 to October 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on audit objectives. The Argonne National Laboratory covers 1,363 acres in DuPage County outside of Chicago, Illinois. Established in 1946 to conduct “cooperative research in nucleonics” as part of the Atomic Energy Commission’s development of nuclear reactors, Argonne National Laboratory now has over 3,200 employees in addition to nearly 800 scientists who visit the site yearly. Additionally, Argonne has over 7,900 facility users who participate in research at five major user facilities located on site. The Bettis Atomic Power Laboratory, covering approximately 200 acres in West Mifflin outside of Pittsburgh, Pennsylvania, is a part of the Naval Nuclear Propulsion Program in the Department of Energy. The Laboratory began operations in 1948 in order to support the engineering, design, and construction of the prototypes of the first nuclear powered submarine, and by 1955 the USS Nautilus was successfully launched. Since then, the Laboratory led development on other nuclear powered crafts including the first nuclear powered ship and aircraft carrier, the USS Long Beach and USS Enterprise, respectively. Today, the Laboratory focuses on design and engineering support for nuclear-powered submarines and aircraft carriers, in addition to development for the nuclear power elements of next generation aircraft carriers. The Brookhaven National Laboratory was established in 1947 by the Atomic Energy Commission. Formerly Camp Upton, a U.S. Army installation site, Brookhaven is located on a 5,263-acre site on Long Island in Upton, New York, approximately 60 miles east of New York City. Historically, Brookhaven was involved in the construction of accelerators and research reactors such as the Cosmotron, the High Flux Beam Reactor, and the Brookhaven Graphite Research Reactor. These research facilities led the way in high-energy physics experiments and subsequent discoveries but also resulted in creation of hazardous wastes. As a result, Brookhaven was listed as a Superfund Site in 1989 and a subsequent agreement with state and federal regulators led to the building and operation of groundwater remediation facilities, and the decontamination and decommissioning of the High Flux Beam Reactor and the Brookhaven Graphite Research Reactor including offsite waste disposal. The Fernald Plant covers 839 acres in southwestern Ohio near Cincinnati, Ohio. The Fernald Plant’s production mission took place from 1951–1989 as it housed the Feed Materials Production Center, which processed uranium as the first step in the nuclear weapons production cycle. In 2006, the remediation and restoration of the site was completed and at the time was one of the largest environmental cleanup operations ever undertaken in the United States. Currently, monitoring of the site and a groundwater extraction and treatment remediation under the Office of Legacy Management is the remaining remediation activity. The site includes restored native plants and grasses and the largest manmade wetlands in Ohio. DOE is responsible for one of the world’s largest environmental cleanup projects: the treatment and disposal of millions of gallons of radioactive and hazardous waste at its 586 square mile Hanford Site in southeastern Washington State. Hanford facilities produced more than 20 million pieces of uranium metal fuel for nine nuclear reactors along the Columbia River. Five plants in the center of the Hanford Site processed 110,000 tons of fuel from the reactors, discharging an estimated 450 billion gallons of liquids to soil disposal sites and 53 million gallons of radioactive waste to 177 large underground tanks. Plutonium production ended in the late 1980s. Hanford cleanup began in 1989 and now involves (1) groundwater monitoring and treatment, (2) deactivation and decommissioning of contaminated facilities, and (3) the construction of the waste treatment and immobilization plant intended, when complete, to treat the waste in the underground tanks. DOE’s Idaho Site is an 890-square-mile federal reserve, only some of which is eligible for PILT, situated in the Arco Desert over the Snake River Plain Aquifer in central Idaho. The site is home to both the Idaho National Laboratory (INL) and the Idaho Cleanup Project. Work at the INL focuses on research and development of nuclear energy technologies, critical infrastructure protection research, and support of national defense and homeland security. The environmental cleanup mission includes remediation of contaminated legacy wastes generated from World War II- era conventional weapons testing, government-owned research and defense reactors, spent nuclear fuel reprocessing, laboratory research, and defense missions at other DOE sites. The Knolls Atomic Power Laboratory, located on 173 acres in Niskayuna, near Schenectady, NY, was established in May 1946. The original mission of the Knolls laboratory was to provide technical support for the chemical separation of plutonium and uranium from irradiated fuel. In the 1950s, Knolls changed focus to Navy submarine propulsion development. Knolls developed a series of nuclear reactor and propulsion plant designs for the U.S. Navy. Knolls is the lead design laboratory for the newest Virginia Class fast attack submarines and is leading the design effort on the next generation ballistic missile submarine. The laboratory, founded in 1943 during World War II, served as a secret facility for research and development of the first nuclear weapon. The site was chosen because the area provided controlled access, steep canyons for testing high explosives, and existing infrastructure. The Manhattan Project’s research and development efforts that were previously spread throughout the nation became centralized at Los Alamos and left a legacy of contamination. Today, the Los Alamos National Laboratory Cleanup Project is responsible for the treatment, storage, and disposition of a variety of radioactive and hazardous waste streams; removal and disposition of buried waste; protection of the regional aquifer; and removal or deactivation of unneeded facilities. DOE’s Oak Ridge Reservation is located on approximately 33,500 acres in East Tennessee. The reservation was established in the early 1940s by the Manhattan Engineer District of the United States Army Corps of Engineers and played a role in the production of enriched uranium during the Manhattan Project and the Cold War. DOE is now working to address excess and contaminated facilities, remove soil and groundwater contamination, and enable modernization that allows the National Nuclear Security Administration to continue its national security and nuclear nonproliferation responsibilities and the Oak Ridge National Laboratory to continue its mission for advancing technology and science. The Pantex Plant covers 2,000 acres and is located northeast of Amarillo, Texas. One of six production facilities in the National Nuclear Security Administration’s Nuclear Security Enterprise, since 1975 the Pantex Plant has operated as the nation’s primary facility for the assembly, dismantlement, and maintenance of nuclear weapons. The last new nuclear weapon was completed in 1991, and since then, the Pantex Plant has dismantled, retired, or stored thousands of nuclear weapons. The Portsmouth Gaseous Diffusion Plant is located in Pike County, Ohio, in southern central Ohio, approximately 20 miles north of the city of Portsmouth, Ohio. This facility was initially constructed to produce enriched uranium to support the nation’s nuclear weapons program and, later, commercial nuclear reactors. Decades of uranium enrichment and support activities required the use of a number of typical and special industrial chemicals and materials. Plant operations generated hazardous, radioactive, mixed (both hazardous and radioactive), and nonchemical (sanitary) wastes. Past operations also resulted in soil, groundwater, and surface water contamination at several sites located within plant boundaries. The Savannah River Site complex covers 198,344 acres, or 310 square miles, encompassing parts of Aiken, Barnwell, and Allendale counties in South Carolina, bordering the Savannah River. The site is a key DOE industrial complex responsible for environmental stewardship, environmental cleanup, waste management, and disposition of nuclear materials. During the early 1950s, the site began to produce materials used in nuclear weapons, primarily tritium and plutonium-239. Five reactors were built to produce nuclear materials and resulted in unusable by-products, such as radioactive waste. About 35 million gallons of radioactive liquid waste are stored in 43 underground tanks. The Defense Waste Processing Facility is processing the high-activity waste, encapsulating radioactive elements in borosilicate glass, a stable storage form. Since the facility began operations in March 1996, it has produced more than 4,000 canisters (more than 16 million pounds) of radioactive glass. In addition to the individual named above, Amanda Kolling, Assistant Director; Antoinette Capaccio; Ellen Fried; Laura Holliday; Skip McClinton; and Sara Sullivan made key contributions to this report. Also contributing to this report were Jeff Arkin, Cindy Gilbert, Michael Kendix, Richard Johnson, and Oliver Richard.", "summary": "The Atomic Energy Act, as amended, authorizes DOE to make PILT payments to communities that host DOE sites that meet specific criteria. PILT is discretionary financial assistance that provides payments to communities based on the property taxes they would have received had the property remained on their tax rolls. House Report 115-230 accompanying a bill for the Energy and Water Development and Related Agencies Appropriations Act of 2018 included a provision for GAO to review DOE PILT. This report assesses (1) how PILT payments vary, if at all, by site and over time, and (2) reasons for variations in payments and the extent to which DOE is providing assurance that payments meet PILT goals. GAO analyzed data on DOE payments to communities that DOE reported as having received PILT payments between 2008 and 2017. GAO compared 2017 data across sites and identified changes in payments to those communities between 1994 and 2017. GAO reviewed PILT's authorizing statute, DOE's PILT order, and PILT documentation. GAO interviewed officials from DOE, communities, and community organizations. The Department of Energy's (DOE) payments in lieu of taxes (PILT)—payments made to some local communities that host DOE sites—vary considerably across the sites and have generally increased over time. Communities at 11 DOE sites received PILT payments in fiscal year 2017 (the most recent fiscal year for which complete data were available), totaling approximately $23 million (see figure). Payments to communities at the Hanford and Savannah River sites accounted for approximately 70 percent of that total, while payments to six sites combined accounted for less than 5 percent. Total PILT payments have more than doubled since 1994, primarily because of growth in payments to communities at the Hanford and Savannah River sites and because communities at other sites began receiving payments since 1994. DOE intentionally allows for variations of payments across sites so that payments may reflect the revenues communities would have received had the property remained on the tax rolls in the condition in which it was acquired, which DOE officials stated is a goal of PILT. However, DOE's PILT order's lack of requirements has limited DOE's ability to provide adequate assurance that payments consistently meet this and other PILT goals. The PILT order does not require documentation of the key determinants that went into the calculation of payments, or an independent review process to determine whether payment calculations are consistent with PILT goals. The PILT order also lacks specificity about payment determinations in certain scenarios. Without updates to the PILT order to strengthen DOE's internal controls, DOE will continue to lack adequate assurance that payments meet PILT goals. GAO is making three recommendations that DOE update its PILT order to: improve collection and documentation of key determinants of PILT payments, implement a review process, and clarify how communities should calculate payment requests. DOE neither agreed nor disagreed and plans instead to further study PILT. We believe our report supports implementation of these recommendations.", "document_type": "gao"}
{"report": "State had 22,806 full-time, permanent, career employees at the end of fiscal year 2018—an increase of more than 38 percent from fiscal year 2002. Over this period, the number of full-time, permanent, career employees in State’s Civil Service rose by nearly 40 percent, from 6,831 in fiscal year 2002 to 9,546 in fiscal year 2018. Over the same period, the number of full-time, permanent, career employees in State’s Foreign Service increased by 36 percent, from 9,739 to 13,260. To increase diversity in its workforce, State carries out a variety of efforts focused on recruiting and retention. For example, the Thomas R. Pickering Foreign Affairs Fellowship Program and Charles B. Rangel International Affairs Program recruit diverse candidates for the Foreign Service by providing graduate fellowships to college seniors and college graduates. Additionally, according to State officials, recruiters for the department participate in career fairs and discussion panels and host information sessions at conferences with a focus on diversity and inclusion, such as those held by the Hispanic Association of Colleges and Universities and the Congressional Black Caucus Foundation. Some regional and functional bureaus also undertake efforts to increase diversity. According to State’s Senior Advisor for Diversity, Inclusion, and Outreach, bureau leaders set the tone, and provide support for bureau- level initiatives. The Equal Employment Opportunity Commission’s (EEOC) Management Directive 715 (MD-715) provides policy guidance and standards for establishing and maintaining effective affirmative programs of equal employment opportunity. Through MD-715, EEOC directs federal agencies to regularly evaluate their employment practices to identify barriers to equal opportunity in the workplace, take measures to eliminate identified barriers, and report annually on these efforts to EEOC. Among State’s full-time, permanent, career employees, the proportion of racial or ethnic minorities grew from 28 percent in fiscal year 2002 to 32 percent in fiscal year 2018. During this period, as figure 1 shows, the proportion of racial or ethnic minorities in the Civil Service decreased slightly, from 44 to 43 percent, and the proportion of racial or ethnic minorities in the Foreign Service increased from 17 to 24 percent. Although the overall proportion of racial or ethnic minorities at State increased from fiscal year 2002 to fiscal year 2018, the direction of change for specific racial or ethnic minority groups varied, as shown in figure 1. The proportion of African Americans at State overall declined from 17 percent in fiscal year 2002 to 15 percent in fiscal year 2018. The proportion of African Americans in State’s Civil Service decreased from 34 to 26 percent, while the proportion of African Americans in State’s Foreign Service increased from 6 to 7 percent. The proportions of Hispanics, Asians, and other racial or ethnic minorities at State overall and in both the Civil and Foreign Services increased by varying percentages from fiscal year 2002 to fiscal year 2018. As figure 2 shows, the proportions of racial or ethnic minorities in the Civil and Foreign Services were generally much smaller in higher ranks in fiscal year 2018. The proportion of racial or ethnic minorities in fiscal year 2018 was lower than the proportion of whites at GS-11, GS-13, and higher ranks in the Civil Service and at all ranks in the Foreign Service. The proportion of racial or ethnic minorities in fiscal year 2018 was progressively lower in each rank above GS-12 in the Civil Service and above Class 5 in the Foreign Service. Among State’s full-time, permanent, career employees, the overall proportion of women at State decreased slightly, from 44 percent in fiscal year 2002 to 43 percent in fiscal year 2018. During this period, as figure 3 shows, the proportion of women in State’s Civil Service decreased from 61 to 54 percent and the proportion of women in State’s Foreign Service increased from 33 to 35 percent. In addition, the proportion of women at State was generally lower than that of men in the higher ranks of both the Civil and Foreign Services in fiscal year 2018, as figure 4 shows. The proportion of women was lower than the proportion of men at GS- 14 and higher ranks in the Civil Service and at Class 4 and higher ranks in the Foreign Service in fiscal year 2018. For example, the proportion of women at Class 4 was 36 percent, while the proportion of men was 64 percent. The proportion of women in the Civil and Foreign Services in fiscal year 2018 was generally progressively smaller from the lower to the higher ranks. Our analyses of State data for fiscal years 2002 through 2018 found differences between promotion outcomes for racial or ethnic minorities relative to whites and for women relative to men. We found these differences when conducting descriptive analyses, which calculated simple averages, as well as adjusted analyses, which controlled for certain individual and occupational factors other than racial or ethnic minority status and gender that could influence promotion. Our analyses do not completely explain the reasons for differences in promotion outcomes, which may result from various unobservable factors. Thus, our analyses do not establish a causal relationship between demographic characteristics and promotion outcomes. The following are some highlights of our analysis. Promotion outcomes in State’s Civil Service were generally lower for racial or ethnic minorities than for whites. Our descriptive analysis of State data for fiscal years 2002 through 2018 found that rates of promotion from GS-11 through the executive rank were 16.1 to 42.0 percent lower for racial or ethnic minorities in the Civil Service than for their white counterparts, depending on the GS level. Our adjusted analysis, controlling for factors other than race or ethnicity that could influence promotion, found that racial or ethnic minorities in the Civil Service were 4.3 to 29.3 percent less likely to be promoted from GS-11 through the executive rank than their white counterparts. Promotion rates in State’s Foreign Service were generally lower for racial or ethnic minorities than for whites, but the differences in promotion odds were generally not statistically significant. Our descriptive analysis of State data for fiscal years 2002 through 2018 found that, relative to whites, the rate of promotion for racial or ethnic minorities in the Foreign Service was 5.0 to 15.8 percent lower for promotions from Class 4 through Class 1. Controlling for factors other than race or ethnicity that could influence promotion, our adjusted analysis found that differences in the odds of promotion for racial or ethnic minorities and whites were generally not statistically significant. However, the odds of promotion from Class 4 to Class 3 were statistically significantly lower for racial or ethnic minorities than for their white counterparts. Promotion rates were generally lower for women than men in State’s Civil Service, but differences in the odds of promotion were not statistically significant. Our descriptive analysis of State data for fiscal years 2002 through 2018 found that the rate of promotion in the Civil Service was generally lower for women than for men. Specifically, for promotions from GS-11 through the executive rank, promotion rates for women were generally 0.7 to 11.6 percent lower than the promotion rates for men, depending on the GS level. However, our adjusted analysis, controlling for factors other than gender that could influence promotion, did not find any statistically significant differences in the odds of promotion for women and men in the Civil Service. Our adjusted analysis found that the odds of promotion were generally higher for women than men in State’s Foreign Service. Our descriptive analysis of State data for fiscal years 2002 through 2018 found that women in the Foreign Service experienced a higher rate of promotion than men from Class 3 to Class 2 and from Class 2 to Class 1. Our adjusted analysis, controlling for factors other than gender that could influence promotion, found that women in the Foreign Service had higher odds of promotion than men in early to mid career. For example, the odds of promotion from Class 4 to Class 3 were 9.4 percent higher for women than for men. State has identified some diversity issues in its reports to EEOC. As table 1 shows, in fiscal years 2009 through 2018, State’s annual MD-715 reports identified and analyzed a total of 11 diversity issues related to participation of racial or ethnic minorities and women. State identified most of these issues in multiple years. However, State employee groups and our analysis have identified additional diversity issues, such as differences in promotion outcomes for racial or ethnic minorities relative to whites in early to mid career. For example, during our structured interviews with 11 employee groups, representatives of the groups discussed a variety of issues related to diversity at State. Examples include the following: Employee group representatives expressed concern about representation of minorities in the higher ranks of both the Civil and Foreign Services. For example, representatives told us that for some minority groups, it is difficult to be promoted above the GS-13 level. Employee group representatives voiced perceptions that it takes longer for women and racial or ethnic minorities to be promoted. For example, representatives of one group told us that it takes longer for employees with diverse backgrounds to reach GS-13 in the Civil Service and Class 2 in the Foreign Service and that very few of these employees are promoted beyond those levels. We recommended that the Secretary of State take additional steps to identify diversity issues that could indicate potential barriers to equal opportunity in its workforce. For example, State could conduct additional analyses of workforce data and of employee groups’ feedback. State concurred with the recommendation and noted that the agency will continue to work on initiatives to recruit, retain, develop, and empower a diverse, capable workforce. In conclusion, although State has implemented several plans, activities, and initiatives to improve diversity and representation throughout the ranks of its workforce, longstanding diversity issues—for example, underrepresentation of racial or ethnic minorities and women in the senior ranks—persist at the agency. Until State takes steps to explore such issues, it could be missing opportunities to investigate, identify, and remove barriers that impede members of some demographic groups from realizing their full potential. Chairman Castro, Ranking Member Zeldin, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions that you may have at this time. If you or your staff have any questions about this testimony, please contact Jason Bair, Director, International Affairs and Trade, at (202) 512- 6881 or bairj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. GAO staff who made key contributions to this testimony are Emil Friberg (Assistant Director), Julia Jebo Grant (Analyst-in-Charge), Nisha Rai, Moon Parks, Justin Fisher, Melinda Cordero, Courtney Lafountain, Kathleen McQueeney, Dae Park, K. Nicole Willems, Reid Lowe, and Christopher Keblitis. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.", "summary": "State has expressed a commitment to maintaining a diverse workforce and has undertaken efforts to increase diversity in its Civil and Foreign Services. EEOC directs federal agencies to regularly evaluate their employment practices to identify barriers to equal opportunity, take measures to eliminate any barriers, and report annually on these efforts. This testimony examines (1) the demographic composition of State's workforce in fiscal years 2002 through 2018; (2) any differences in promotion outcomes for various demographic groups in State's workforce; and (3) the extent to which State has identified any barriers to diversity in its workforce. For the January 2020 report on which this testimony is based (GAO-20-237), GAO analyzed State's data for its full-time, permanent, career workforce in fiscal years 2002 through 2018. GAO also analyzed the number of years until promotion from early career ranks to the executive rank in both the Civil and Foreign Services. (GAO's analyses do not completely explain the reasons for differences in promotion outcomes, which may result from various unobservable factors. Thus, GAO's analyses do not establish a causal relationship between demographic characteristics and promotion outcomes.) In addition, GAO reviewed State documents and interviewed State officials and employee group representatives. The overall proportion of racial or ethnic minorities in the Department of State's (State) full-time, permanent, career workforce grew from 28 to 32 percent from fiscal year 2002 to fiscal year 2018. The direction of change for specific groups varied. For instance, the proportion of African Americans fell from 17 to 15 percent, while the proportions of Hispanics, Asians, and other racial or ethnic minorities rose by varying percentages. The proportion of racial or ethnic minorities and women was lowest in the higher ranks of State's workforce. GAO's analyses of State data for fiscal years 2002 through 2018 found differences in promotion outcomes for racial or ethnic minorities and whites and for men and women. GAO found these differences in both descriptive analyses (calculating simple averages) and adjusted analyses (controlling for certain individual and occupational factors that could influence promotion). For example, GAO's descriptive analysis of data for State's Civil Service found that rates of promotion for racial or ethnic minorities were 16 to 42 percent lower, depending on the rank, than for whites. Similarly, after controling for certain additional factors, GAO's adjusted analysis of these data found that promotion for racial or ethnic minorites was 4 to 29 percent less likely than for whites. Also, both types of analysis generally found that promotion outcomes for women relative to men were lower in the Civil Service and higher in the Foreign Service. For example, women in the Foreign Service were more likely than men to be promoted in early to mid career. State has identified some diversity issues, but it should consider other issues that could indicate potential barriers to diversity in its workforce. State's annual reports to the Equal Employment Opportunity Commission (EEOC) for fiscal years 2009 through 2018 identified issues such as underrepresentation of Hispanic employees and underrepresentation of minorities in the senior ranks. However, GAO's analysis and GAO's interviews with State employee groups highlighted additional issues that could indicate barriers to diversity. For example, State's reports have not identified lower promotion outcomes for racial or ethnic minorities relative to whites, which GAO found in its analysis. Until State takes steps to explore such issues, it could be missing opportunities to investigate and remove barriers that impede members of some demographic groups from realizing their full potential. In its January 2020 report, GAO recommended that State take additional steps to identify diversity issues that could indicate potential barriers to equal opportunity in its workforce. State concurred with this recommendation.", "document_type": "gao"}
{"report": "The Office of Entrepreneurial Development, Office of Field Operations, and Office of Strategic Alliances are key SBA offices that administer entrepreneurial programs and manage outreach efforts that could foster entrepreneurship (see fig. 1). Office of Entrepreneurial Development. The Office of Entrepreneurial Development oversees several programs, primarily through a nationwide network of public and private resource partners that offer small business counseling and technical assistance. These resource partners include SBDCs, Women’s Business Centers, and SCORE chapters. The SBDC program receives the majority of entrepreneurial development program funding to provide technical assistance (business counseling and training) to small businesses and aspiring entrepreneurs. SBDC services include assisting small businesses access capital, develop and exchange new technologies, and improve business planning, strategy, and financial management. The entities eligible to receive SBDC funding are primarily institutions of higher education. By statute, the amount eligible entities receive is determined by a state population-based funding formula subject to the amount of an appropriation in any given fiscal year. As a condition of receiving the grant, the recipient or host institution is required to match the funding. The host institution (funding recipient) is responsible for establishing a lead center and network of service centers for a designated service area. The SBDC program has 63 lead centers (generally hosted by institutions of higher education) and more than 900 service centers, including satellite locations. SBA has identified certain special emphasis groups to be targeted for assistance by SBDCs, such as certain populations of business owners. The groups do not include institutions; thus, HBCUs are not included as special emphasis groups. According to SBA officials, SBDCs target underrepresented groups in the population of business owners near HBCUs. Office of Field Operations. SBA also provides services through a network of 10 regional offices and 68 district offices that are led by the Office of Field Operations. SBA district offices serve as the point of delivery for most SBA programs and services. Some district office staff (including business opportunity, lender relations, and economic development specialists) work directly with SBA clients. SBA’s district offices also can initiate and oversee outreach activities to foster entrepreneurship. For example, SBA district offices can implement counseling or training events on their own, participate in such events organized by third parties, or co-sponsor such activities with a third party (for-profit, nonprofit, or government entity) through a co-sponsorship agreement. Moreover, district offices can enter into a 2-year agreement with a nonprofit or government party, known as a strategic alliance memorandum, to foster a working relationship designed to strengthen small business development in a local area. Office of Strategic Alliances. The Office of Strategic Alliances, housed in SBA’s Office of Communication and Public Liaison, reviews co- sponsorship agreements and strategic alliance memorandums drafted by district or program offices. The co-sponsorship agreements and memorandums are based on an internal SBA template provided by the Office of Strategic Alliances, which also maintains records for both strategic alliance memorandums and co-sponsorship agreements. Figure 2, an interactive map, illustrates locations of SBDC lead centers and SBA district office in states with HBCUs. See appendix II for additional information on figure 2. As of December 2018, there were 101 HBCUs, located across 19 states, the District of Columbia and the U.S. Virgin Islands. As previously discussed, HBCUs educated more than 226,000 African-American students in 2017. HBCUs also have played a critical role in supporting underserved students and communities. We previously reported that a higher proportion of students at private HBCUs (77 percent) received Pell Grants in the 2015–16 school year than students at similar private colleges or universities (43 percent). Pell Grants provide low-income undergraduates who demonstrate financial need with financial assistance to help meet education expenses. Executive Order 12232 (1980) established the White House Initiative on Historically Black Colleges and Universities to strengthen the capacity of HBCUs to provide quality education. Subsequent administrations issued executive orders to continue the initiative. Most recently, as expressed in Executive Order 13779 (2017), federal priorities for working with HBCUs encompass two missions: (1) increasing the role of private-sector entities in helping to improve capacity of HBCUs, and (2) enhancing HBCUs’ capabilities for helping young adults. The initiative has been housed in the Executive Office of the President since 2017, according to representatives from the initiative. The more recent executive orders (from 2002, 2010, and 2017) direct each department and agency designated by the Secretary of Education to prepare an annual plan on efforts to strengthen HBCU capacity. Annual plans are to describe how the department or agency intends to increase the capacity of HBCUs, including by identifying federal programs and initiatives in which HBCUs are underserved or that HBCUs may have underutilized. SBA is one of the agencies designated to prepare an annual agency plan. The more recent executive orders also state that a Board of Advisors on HBCUs (in the Department of Education) shall report annually to the President on the Board’s progress in carrying out its duties, which include advising the President on matters pertaining to strengthening the educational capacity of HBCUs. The current Board was chartered in May 2019. SBA has used SBDCs, strategic alliance memorandums, and co- sponsored activities to foster entrepreneurship with HBCUs in recent years; stakeholders’ experiences collaborating with SBA varied. Small Business Development Centers. Two HBCUs—Howard University in Washington, D.C., and the University of the Virgin Islands in St. Thomas, U.S. Virgin Islands—have been longstanding host institutions for SBDCs. More specifically, they have been the only host institutions for two lead SBDCs, the District of Columbia SBDC and the Virgin Islands SBDC, for more than 30 years (and remained so as of September 2019). Colleges and universities predominately have been the institutional hosts of lead SBDCs since the 1980s according to SBA officials. According to SBA officials, there is little turnover among institutions hosting lead SBDCs because SBDC program announcements for host institutions are not full and open competitions and existing host institutions often renew their cooperative agreements to continue operating lead SBDCs. Based on the statutorily defined and population- based allocation formula, the District of Columbia SBDC and the Virgin Islands SBDC together received about 1.3–1.4 percent of the total SBDC funding awarded to institutions of higher education from fiscal year 2008 through 2018. The District of Columbia SBDC and the Virgin Islands SBDC have engaged with HBCU students, alumni, or faculty. As we previously reported, District of Columbia SBDC representatives told us that as of November 2018 they were working with 10–15 Howard University student clients. They also stated they work with all students who come to their center seeking help and do not have a cap on the number of student clients. Similarly, the Virgin Islands SBDC representatives told us as of February 2019 they had made presentations to upper-level business classes and freshmen development seminars at the University of the Virgin Islands. They also counseled students who participated in an annual entrepreneurial competition. They noted that many of the SBDC clients they serve have some affiliation with the university, such as being an alumnus or having attended classes there. In addition to establishing the lead SBDC, the host institution establishes a network of service centers to deliver services, such as counseling and training, within its service area, including at HBCUs. As of September 2018, at least 16 HBCUs hosted SBDC service centers across 11 states. Three SBDC service centers we reviewed that were HBCU- hosted had engaged with HBCU students, alumni, or faculty. For example, the Alabama SBDC service center representatives (housed at Alabama State University in Montgomery, Alabama) said the center works with several faculty members who provided training at SBDC workshops and assisted the service center on specialized topics, such as marketing. Through its relationship with faculty members, the Alabama SBDC service center also conducts outreach to students. Similarly, representatives of two service centers for the North Carolina Small Business Technology and Development Center (housed at North Carolina Central University in Durham, North Carolina, and North Carolina A&T State University in Greensboro, North Carolina) said they have worked with students on the respective campuses. For example, the service center at North Carolina Central University has engaged graduate business students on marketing projects. While the number of HBCU-hosted lead SBDCs has remained unchanged in recent years, it is unclear how many HBCUs have hosted service centers. SBA officials told us that the number fluctuates but were unable to provide the list of all service centers in existence prior to 2018. We discuss SBA’s data collection efforts later in this report. Strategic alliance memorandums. From 2013 through 2018, SBA signed at least 35 strategic alliance memorandums with HBCUs (see table 1). SBA signed at least 51 such memorandums with institutions of higher education in states with HBCUs in that period. As we previously reported, strategic alliance memorandums are mechanisms to initiate and formalize a relationship with nonprofit and governmental agencies, but they are not necessary to initiate a relationship. SBA officials told us the memorandums do not authorize or fund events or activities and are largely symbolic. In August 2019, SBA officials said that numbers of strategic alliance memorandums can fluctuate due to their 2-year duration and changes in SBA administration. Representatives of six HBCUs with whom we met that signed strategic alliance memorandums varied in their assessment of the usefulness of the memorandums. Three of the six HBCUs said they had positive experiences as a result of the memorandums: Representatives of an HBCU in North Carolina said a May 2013 memorandum established a relationship with SBA and provided access to information and resources not otherwise available. Representatives of another HBCU in North Carolina said a 2013 memorandum helped recruit speakers for two entrepreneurship classes. A representative from an HBCU in Tennessee told us that a 2013 memorandum enabled the college to connect students, alumni, and faculty with the resources of SBA’s Tennessee District Office and its resource partners. The representative said a subsequent 2018 memorandum resulted in collaboration with SBA to host a 1-day small business conference on campus. In contrast, representatives of the three other HBCUs either were unaware of the memorandum or said it produced no results: Representatives of two HBCUs (one in Alabama and one in Georgia) told us they were unaware of the signed strategic alliance memorandums (March and April 2013, respectively) due to staffing changes in senior administrative positions. A representative from another HBCU in Georgia told us the school had little involvement with the Georgia SBA district office after signing a memorandum in April 2013. Officials from the district office with whom we spoke agreed with this statement but noted the college had not asked them to participate in any events. Co-sponsored activities. As shown in table 2, from fiscal years 2013 through 2018, SBA signed at least 16 co-sponsorship agreements with HBCUs to jointly conduct activities or events. Twelve of the 16 co- sponsored activities were training or counseling events related to entrepreneurship. SBA signed at least 78 co-sponsorship agreements with institutions of higher education in states with HBCUs in that period. SBA developed a fiscal year 2018 plan for the White House Initiative on HBCUs, in accordance with Executive Order 13779 (2017). SBA’s 2018 plan included two primary goals. The first goal was to raise awareness and provide information to increase the capacity of HBCUs to participate in federally funded programs. More specifically, the plan stated that SBA would engage with HBCUs and provide them with information needed to access and compete for federal grants and contracts. The second goal was to promote collaboration among HBCUs, SBA resource partners, and SBA district offices. For example, the plan stated that SBA would encourage the formation of strategic alliance memorandums between SBA district offices and HBCUs to promote and support entrepreneurship in underserved markets. The plan also stated that SBA would explore and consider partnerships with the National Association for Equal Opportunity in Higher Education, among other organizations, to raise awareness, disseminate information, and share resources among and with HBCUs. The 2018 plan also described five measures to monitor SBA’s efforts to engage, share information, and increase the capacity of HBCUs. The measures are (1) number of outreach events, (2) number of outreach attendees, (3) number of partnerships established, (4) percentage of engaged HBCUs that pursued federal funding, and (5) percentage of HBCUs engaged that found the information useful. The two previous Executive Orders (from 2002 and 2010) on the White House Initiative on HBCUs also directed designated agencies to prepare annual plans on their efforts to support HBCUs. For years prior to 2018, SBA only could provide documentation of plans for 2011 and 2012. Officials of the Office of Entrepreneurial Development told us they were not aware of records for plans developed for the other years in the period we reviewed (2008–2018). SBA was unclear about the responsibilities of the offices involved in the agency’s efforts for addressing the White House Initiative on HBCUs. In March 2019, SBA officials told us the SBA Administrator had designated the Office of Entrepreneurial Development as the lead office for addressing the initiative in 2018. However, the responsibilities for other offices involved in efforts that include HBCUs remain unclear. SBA could not provide documentation of roles, responsibilities, or reporting lines among offices involved in addressing the White House Initiative on HBCUs. For example, the Associate Administrator for the Office of Entrepreneurial Development stated the agency’s interaction with HBCUs occurs through SBA district offices. However, there is no documentation describing how the Office of Entrepreneurial Development and Office of Field Operations, which is responsible for SBA’s district offices, should work together to address the White House Initiative on HBCUs. Moreover, because SBA has not documented specific roles and responsibilities (to include reporting lines), it is unclear how plans prepared for the White House Initiative on HBCUs would be implemented among headquarters, field offices, and resource partners. Additionally, the role of the director of the Office of Faith Based and Community Initiatives in efforts to address the HBCU initiative is unclear. SBA officials told us the Office of Faith Based and Community Initiatives is not involved in administering the initiative. However, the director of that office serves as SBA’s representative to the Interagency Working Group of the White House Initiative on HBCUs. According to SBA officials, the director’s role is to support efforts by the Office of Entrepreneurial Development on the initiative due to staffing shortages. However, officials were unable to tell us in greater detail how the director would provide such support. Federal internal control standards state that management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. For example, management assigns responsibilities to discrete units to enable the organization to operate in an efficient and effective manner and to delegate authority to key roles throughout the entity. Additionally, management establishes defined reporting lines within an organizational structure so that units can communicate (up, down, and across the organization) the quality information necessary for each unit to fulfill its overall responsibilities. SBA’s uncertainty about the responsibilities of the offices involved in the White House Initiative on HBCUs may be a result of changes over the years as to which program office was chiefly responsible for the effort. According to the Associate Administrator of the Office of Entrepreneurial Development, the responsibilities for the White House initiative on HBCUs have resided in various SBA program offices over the years. Moreover, the Associate Administrator told us the Office of Entrepreneurial Development was designated the lead office for the initiative late in the planning process; therefore, it took time to transfer responsibilities for addressing the initiative to the Office of Entrepreneurial Development. As a result, the Office of Entrepreneurial Development had not yet defined the responsibilities for other offices involved in efforts related to the White House Initiative on HBCUs. In September 2019, SBA officials told us they intended to establish an intra-agency working group focused on HBCUs, which would define the roles and responsibilities of headquarters offices related to the initiative. While the Office of Entrepreneurial Development has been designated as the lead office, without clearly assigned roles, responsibilities, and reporting lines for the other offices involved in the White House Initiative on HBCUs, SBA may not be able to effectively implement future plans for the initiative. Additionally, the lack of clearly assigned roles, responsibilities, and reporting lines has resulted in and may result in future loss of institutional knowledge on efforts to implement the initiative. SBA’s 2018 plan to support HBCUs had the goal of promoting collaboration among HBCUs, SBA resource partners, and SBA district offices. However, SBA headquarters did not communicate its plan for supporting HBCUs to SBDCs and district offices with HBCUs in their service areas. Specifically, SBA officials told us the Office of Entrepreneurial Development, which oversees the SBDC program, did not communicate the 2018 plan to support HBCUs to SBDCs, including the goal to collaborate with HBCUs. None of the SBDC representatives with whom we spoke (for six lead centers and three service centers), reported that SBA communicated information related to the 2018 plan, including goals, measures, or other HBCU-related expectations. Furthermore, none said they received guidance from SBA headquarters related to fostering entrepreneurship with HBCUs, although SBDCs deliver counseling and training to potential and existing business owners. Similarly, the Office of Field Operations, which oversees district offices, did not communicate the 2018 plan to support HBCUs to district offices, including the goal to collaborate with HBCUs, according to SBA officials. While SBA’s district offices deliver most of SBA’s programs and services, none of the representatives of the eight district offices with whom we spoke answered questions related to SBA’s planned efforts to support HBCUs because they stated they were not involved with agency plans for the White House Initiative on HBCUs or were otherwise unable to provide a response. Federal internal control standards state that management should internally communicate the necessary quality information to achieve the entity’s objectives. For example, management assigns the internal control responsibilities for key roles and communicates quality information up, down, and across reporting lines. This enables personnel to perform key roles in achieving objectives, addressing risks, and supporting the internal control system. According to SBA officials, SBA headquarters did not communicate its plan for supporting HBCUs to SBDCs and district offices due to the timing of the plan’s issuance—the 2018 plan was not finalized until near the end of the fiscal year. SBA officials told us that instead of communicating the 2018 plan at the end of the 2018 fiscal year, officials chose to focus on the upcoming fiscal year and future efforts to support HBCUs. Additionally, SBA officials stated the Office of Field Operations was not involved in addressing the White House Initiative on HBCUs, although the office is responsible for providing policy guidance and oversight to district offices in implementing agency goals and objectives. Because SBA headquarters did not communicate its plan for supporting HBCUs, SBDCs and district offices with HBCUs in their service areas were not aware of the goal to collaborate with HBCUs. Therefore, the agency may have missed opportunities to collaborate with HBCUs and work toward 2018 plan goals, even if for a brief period. As of September 2019, SBA officials told us the agency’s fiscal year 2019 plan (or update) for the White House Initiative on HBCUs had not been finalized. As a result, it was unclear when this plan would be communicated to SBDCs and district offices. If the 2019 and subsequent plans for supporting HBCUs are not communicated to SBDCs and district offices, SBA risks repeating a scenario in which SBDCs and district offices with HBCUs in their service areas are unaware of goals to support HBCUs, and therefore may miss opportunities to engage with HBCUs. The extent to which SBA collected information about its programs and activities with HBCUs is limited. More specifically, SBA did not collect relevant information to establish a baseline for performance measures developed in its 2018 plan for the White House Initiative on HBCUs. SBA officials told us that they wanted to use the measures to establish a baseline to better assess progress towards meeting the plan’s goals to support HBCUs in fiscal year 2019. As noted earlier, the 2018 plan’s five measures are (1) number of outreach events, (2) number of outreach attendees, (3) number of partnerships established, (4) percentage of engaged HBCUs that pursued federal funding, and (5) percentage of HBCUs engaged that found the information useful. Number of outreach events and attendees. SBA collects information on the number of outreach events and the number of outreach attendees, but this information is incomplete and not specific to HBCUs. According to SBA officials, SBA district offices are required to collect and report the number of outreach events and attendees to the Office of Field Operations. However, information for outreach activities is reported on an aggregate basis to headquarters and does not specifically identify which institutions hosted or participated in the events. As such, the information reported also does not specifically identify attendees affiliated with an HBCU, such as students, faculty, or alumni. Therefore, while representatives of all eight district offices we contacted said they have conducted outreach activities with HBCUs, these activities would not be readily identifiable in the information reported to headquarters. Until July 2019, SBA district offices reported outreach events through the activity contact report. District offices were able to include optional information, such as the event location and organization name for their outreach events, as shown in figure 3. SBA officials told us they can perform manual searches for specific text (such as the specific name of an institution or “HBCU”) included in information reported by district offices that may identify HBCU-related activities. However, they said manual searches are not easy or effective or routinely performed. Therefore, manually searching for specific text that may be included in information reported by district offices does not lend itself to efficient monitoring of HBCU-related outreach. SBA officials told us a temporary reporting tool (used since late July 2019 in place of the activity contact report) includes an optional data field for district offices to identify whether their activity was HBCU-related. While this additional field may enable users to conduct manual searches for HBCU-related outreach more easily, SBA officials told us the data from the field are still reported in the aggregate to SBA headquarters and therefore continue to be not readily identifiable as HBCU-related. For more information about SBA’s systems for reporting (including district offices), see appendix IV. Additionally, SBA officials told us headquarters does not have policies or guidance for district offices for systematically collecting or reporting data on their HBCU-related outreach. At least one district office, West Virginia, voluntarily tracks its activities with the HBCUs in its region, using a spreadsheet it developed. Unlike district offices, SBDCs are not required to collect and report information related to outreach (such as the number of outreach events and attendees) to the Office of Entrepreneurial Development. As a result, although all nine SBDCs we contacted conduct outreach to HBCUs, SBA lacks data about these activities. The 2020 funding opportunity for SBDCs requires SBDCs with HBCUs in their states to report outreach events with HBCUs in their semi-annual and final year-end reports. Number of partnerships established. SBA collects information on the number of partnerships established, but this information is incomplete and not specific to HBCUs. According to SBA officials, there is no written definition defining partnerships for this measure, but it would include both informal and formal partnerships. SBA collects information related to formal partnerships: SBDCs, strategic alliance memorandums, and co- sponsorship agreements. However, these records do not allow for the ready identification of HBCU partnerships because there are no data fields to identify whether the partner is an HBCU. SBDCs. Information on the number of SBDCs hosted by HBCUs is incomplete. SBA’s records do not allow for ready identification of HBCUs as host institutions because there is no field to identify whether a host institution is an HBCU. While SBA provided information on the number of SBDC lead centers hosted by HBCUs over time, information was not available on the number of SBDC service centers hosted by HBCUs during the time frame of our review (2008–2018) because according to SBA officials, host institutions for service centers can change over time. Strategic alliance memorandums. Information on the number of strategic alliance memorandums signed with HBCUs is incomplete. In September 2018, SBA provided us a list of HBCUs that signed strategic alliance memorandums from 2008 through 2018, developed by cross-referencing records of memorandums with a list of HBCUs. SBA identified 24 such HBCUs, but we identified an additional three HBCUs that had signed strategic alliance memorandums during this period. In June 2019, SBA provided us a list of all strategic alliance memorandums signed from 2015 through 2018, but we found that a 2016 memorandum with Alabama A&M University (Huntsville, Alabama) was not included. Co-sponsorship agreements. Information on the number of co- sponsorship agreements signed with HBCUs is incomplete. In November 2018, SBA provided us copies of co-sponsorship agreements signed with HBCUs from fiscal years 2013 through 2018 by manually cross-referencing its records with a list of HBCUs. SBA identified 14 such agreements, but we identified an additional two co- sponsorship agreements signed with HBCUs. Usefulness. SBA does not collect information on the percentage of HBCUs engaged in activities that found the resources and information SBA provided to be useful. According to SBA officials, district offices are not required to collect written feedback related to the usefulness of information presented during their activities, such as counseling and training. If district offices solicit feedback, it cannot be distinguished as feedback from HBCUs. For example, SBA district offices may solicit written feedback for co- sponsored activities using a headquarters-developed form. The form does not include fields for participants to identify their affiliation with an HBCU and therefore, feedback received would not be HBCU-specific. Unlike district offices, SBDCs are required to issue evaluation forms for SBDC clients who receive continuous counseling or attend an SBDC training event. For example, representatives from the Alabama SBDC lead center told us they conduct quarterly counseling surveys, which include questions related to the timeliness of the counseling and knowledge of the business advisor. SBDCs report data on client satisfaction rates to SBA headquarters. However, SBA officials told us the feedback-related information SBDCs collect and report to headquarters is not specific to HBCUs, despite the agency’s identification of this measure as relevant in its 2018 plan. Federal internal controls standards state that management should use quality information to achieve the entity’s objectives. For example, management obtains relevant data based on the identified information requirements and relevant data have a logical connection with the identified information requirements. SBA lacks information related to programs and activities with HBCUs because district offices and SBDCs with HBCUs in their service areas have not been required by relevant program offices to collect or report information specific to HBCUs, including information relevant for measures developed in SBA’s 2018 plan. Without collection of relevant information for its HBCU-related efforts, particularly for measures developed for annual plans, SBA will not be able to establish a baseline of its efforts to support HBCUs. Moreover, without this baseline SBA cannot determine the extent or effectiveness of its efforts to support and engage HBCUs. SBA’s priority goals include reaching emerging markets that are socially and economically disadvantaged. The agency’s efforts related to HBCUs, which educate many low-income students and help support their local communities, can assist the agency in advancing that goal. But while SBA has long participated in the White House Initiative on HBCUs, it has not clearly assigned responsibilities among relevant offices for addressing its plan for the initiative; communicated its plan to support HBCUs to SBA district offices and SBDCs (with HBCUs in their service areas), which deliver training and counseling; and collected relevant information to establish a baseline and track ongoing efforts to support HBCUs. Addressing these issues would better position SBA to assess the extent to which it is reaching its goals specific to supporting HBCUs, as well as agency-wide priority goals to more broadly reach socially and economically disadvantaged communities. We are making the following three recommendations to the Small Business Administration: The SBA Administrator should assign and document clear roles, responsibilities, and reporting lines for headquarters offices’ implementation of SBA’s plan for addressing the White House Initiative on HBCUs in a timely manner. (Recommendation 1) The Associate Administrators of the Office of Entrepreneurial Development and Office of Field Operations should communicate planned efforts to support HBCUs, including expectations, goals, and related measures, to the district offices and Small Business Development Centers with HBCUs in their service areas. (Recommendation 2) The Associate Administrator of the Office of Entrepreneurial Development should take and document steps to ensure that the office’s reporting mechanisms collect the information needed to establish a baseline for, and also inform future monitoring and assessment of, efforts to support HBCUs. (Recommendation 3) We provided a draft of this report to the Small Business Administration for review and comment. In comments reproduced in appendix V, the Small Business Administration agreed with our three recommendations. The Small Business Administration also provided additional examples of recent accomplishments and plans in their comments. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Acting Administrator of the Small Business Administration and other interested parties. In addition, the report will be made available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or OrtizA@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. You asked us to review SBA’s entrepreneurship-related efforts with Historically Black Colleges and Universities (HBCU). This report examines, as did two related products, (1) the Small Business Administration’s (SBA) efforts to foster entrepreneurship through key programs and activities with HBCUs in recent years, (2) SBA’s agency plans for the White House Initiative on HBCUs, and (3) the extent to which SBA collected and recorded information specific to HBCUs. Our review of efforts to foster entrepreneurship focused on counseling and training. To address the first objective, we analyzed SBA programs and activities that in previous work we identified as key for fostering entrepreneurship with HBCUs. Key programs and activities are the Small Business Development Center (SBDC) program, strategic alliance memorandums, and co-sponsored activities. We obtained data from SBA’s Office of Entrepreneurial Development and Office of Strategic Alliances for these key programs and activities, and identified the participation of institutions of higher education, including HBCUs. We reviewed and analyzed data provided to us by SBA of the host institutions and the total amount of funding obligated to administer the SBDC program in fiscal years 2008– 2018 and signed agreements (strategic alliance memorandums and co- sponsorship agreements) with institutions of higher education (HBCUs and non-HBCUs) in fiscal years 2013–2018. In addition, we conducted an on-site file review to record strategic alliance memorandums signed in fiscal years 2013–2015 that were not readily available electronically. To assess the reliability of these data, we reviewed available data, cross- walked them with publicly available information, if applicable, and requested written responses from SBA officials about the data and their limitations, if any. We determined the data were sufficiently reliable for describing the general scale of SBA’s efforts to engage with HBCUs and non-HBCUs. To address the second objective, we reviewed SBA’s 2018 plan for the White House Initiative on HBCUs and documentation of plans for 2011 and 2012, which were the only years in the period of our review (2008– 2018) for which SBA could provide documentation of such plans. We also analyzed the three most recent executive orders related to HBCUs to understand the responsibilities expected of federal agencies and identify changes over time. We also reviewed additional documents that SBA provided related to its agency plans, such as efforts to promote small business research programs, and one available annual agency submission (fiscal year 2010) to the White House Initiative on HBCUs on SBA’s efforts to support HBCUs. We interviewed SBA officials from the Office of Entrepreneurial Development, Office of Field Operations, and the Office of Strategic Alliances. We also interviewed representatives of six SBDC lead centers and three associated service centers, and eight SBA district offices (with 47 HBCUs in their areas). We selected these SBDC networks and district offices based on a combination of factors, including (1) HBCU participation in an SBDC network (hosting a lead or service center), (2) high number of HBCUs located in the state, and (3) high number of agreements (strategic alliance memorandums or co-sponsorship agreements) SBA signed with HBCUs. Based on data SBA provided of signed strategic alliance memorandums with HBCUs, we selected and contacted 12 HBCUs that had signed a strategic alliance memorandum with SBA between 2013 and 2018 or were located close to SBA offices or resource partners such as SBDCs. We interviewed staff at eight of these HBCUs and the remaining four HBCUs did not respond. We visited the District of Columbia, Maryland, North Carolina, and the U.S. Virgin Islands and met with SBDC representatives, SBA district officials, and HBCU representatives, as applicable. We also interviewed America’s SBDCs, an association for SBDCs, and representatives of the following advocacy groups: the Thurgood Marshall College Fund, the United Negro College Fund, and the National Association for Equal Opportunity. To address the third objective, we reviewed two sets of SBA standard operating procedures to understand information collected and reported for (1) the SBDC program, and (2) outreach activities that include co- sponsored activities and strategic alliance memorandums. We reviewed available program announcements or funding opportunities, and cooperative agreements for recipients of the SBDC program to identify their reporting requirements. We reviewed guidance related to the Office of Field Operations’ goals and measures to identity SBA district offices’ reporting requirements. In addition, we reviewed user manuals, data entry form templates, and data dictionaries for SBA information systems used by SBDCs and SBA district offices, such as the Entrepreneurial Development Management Information System and the Activity Contact Report, to identify the extent to which data collected and reported included HBCU-related activities. We reviewed SBA’s 2018 plan for the White House Initiative on HBCUs to identify performance measures developed to monitor SBA’s HBCU- related efforts. We then analyzed whether the information that SBDCs and SBA district offices are required to report included information for monitoring the performance measures developed in the 2018 plan. We assessed SBA’s plans and related efforts against federal internal control standards. Additionally, we interviewed SBA officials from the Office of Entrepreneurial Development, Office of Field Operations, and Office of Strategic Alliances; SBA district office officials from eight offices; and SBDC representatives from nine SBDC networks to better understand the extent to which SBA collects and records information related to their engagement with HBCUs. We conducted this performance audit, from June 2018 to November 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Small Business Administration’s (SBA) Small Business Development Centers (SBDC) and district offices provide services that could foster entrepreneurship. The SBDC program provides technical assistance (business counseling and training) to small businesses and aspiring entrepreneurs. SBDCs assist small businesses access capital, develop and exchange new technologies, and improve business planning, strategy, and financial management, among other services. The recipient or host institution of the SBDC is responsible for establishing a lead center and a network of service centers for a designated service area. SBA district offices serve as the point of delivery for most SBA programs and services. Some district office staff (including business opportunity, lender relations, and economic development specialists) work directly with SBA clients. SBA’s district offices also can initiate and oversee outreach activities to foster entrepreneurship. As of December 2018, there were 101 Historically Black Colleges and Universities (HBCU), located across 19 states, the District of Columbia, and the U.S. Virgin Islands. Table 3 lists those states (in addition to the District of Columbia and the U.S. Virgin Islands), the locations of SBDC lead centers and district offices, and the HBCUs. The Small Business Administration’s (SBA) 2018 plan for the White House Initiative on Historically Black Colleges and Universities (HBCU) identifies two programs as available resources that are underutilized by HBCUs. More specifically, the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs provide research and development funding to small businesses to develop and commercialize innovative technologies. The programs are authorized by the Small Business Act, and SBA’s Office of Investment and Innovation is responsible for their oversight, including coordinating the participating agencies’ efforts for the programs. The SBIR program began in 1982 and has four main purposes: (1) use small businesses to meet federal research and development needs, (2) stimulate technological innovation, (3) increase private-sector commercialization of innovations derived from federal research and development efforts, and (4) foster and encourage technological innovation by small businesses owned by women and disadvantaged individuals. The STTR program began in 1992 and has three main purposes: (1) stimulate technological innovation, (2) foster technological transfer through cooperative research and development between small businesses and research institutions, and (3) increase private-sector commercialization of innovations derived from federal research and development. Both programs are similar in that participating agencies identify topics for research and development projects and support small businesses, but the STTR program requires the small business to partner with a nonprofit research institution, such as a college or university or federally funded research and development center. SBA has made some recent efforts to increase awareness among HBCUs about opportunities to access these programs. For example: SBA participated in the HBCU and Minority-Serving Institution Technology Infusion Road Tour, which was organized by the National Aeronautics and Space Administration. As a part of this effort, SBA participated in presentations on the SBIR and STTR programs at three HBCUs: Tennessee State University (Nashville, Tennessee) in April 2017, Johnson C. Smith University (Charlotte, North Carolina) in February 2018, and Clark Atlanta University (Atlanta, Georgia) in March 2018. In 2018, SBA conducted an SBIR Road Tour to raise awareness of available research and development funding. As part of the tour, the agency conducted workshops and presentations at two HBCUs: Alabama A&M University (Huntsville, Alabama) and Jackson State University (Jackson, Mississippi). SBA participated in the 2018 National HBCU Week Conference hosted by the White House Initiative on HBCUs. SBA staff served as speakers and panelists in sessions related to access to federally funded programs (such as SBIR and STTR), and science, technology, engineering, and mathematics. The North Carolina Small Business and Technology Development Center, an SBA resource partner, hosted a workshop in April 2018 at an HBCU—North Carolina Central University (Durham, North Carolina)—focused on preparing proposals for the SBIR and STTR programs. The Small Business Administration’s (SBA) information systems collect a variety of information about SBA’s Small Business Development Centers (SBDC) and district office activities, including counseling and training. Partner Identification Management System. SBDC lead centers are required to maintain their lead center and service center information in SBA’s Partner Identification Management System. This information includes each SBDC service location by name, host institution, and physical address. Additionally, SBDC locations are identified as the lead center, service center, or satellite location. There is no data field to identify the type of host institution (such as institution of higher education) or whether the host institution is a Historically Black College and University (HBCU). Entrepreneurial Development Management Information System. SBDCs are required to report their program data, including counseling and training activities, through SBA’s data collection system, known as the Entrepreneurial Development Management Information System. According to the user manual, the system is designed around two primary forms: SBA’s counseling information form and SBA’s management training report. These forms include data fields for users to enter demographic information on clients and training participants, such as race, gender, and veteran status. Figure 4 shows the data fields related to demographic information included in SBA’s counseling information form. Figure 5 shows data fields related to demographic information included in SBA’s management training report. There are no data fields for users to enter information related to whether a client or training participant is associated with an institution of higher education, including an HBCU. The Entrepreneurial Development Management Information System enables SBA management to generate reports based on demographic information, such as the number of minority participants trained by SBDCs, but not on the number of HBCU- affiliated clients and training participants. Activity contact report. Until July 2019, district offices were required to report activities (including training, presentations, and interactions with stakeholders) that aligned with their goals and measures to the Office of Field Operations through SBA’s activity contact report. District office staff reported their activities in categories that included general inquiries, training, presentations, counseling and technical assistance, outreach, meetings, and special initiatives. Activity contact report forms did not include data entry fields specific to the type of institution (such as institutions of higher education, including HBCUs) that hosted or participated in the district office’s activity. Additionally, the activity contact report forms did not include data entry fields to identify whether participants were affiliated with an HBCU (students, faculty, or alumni). For some activity contact report categories, the forms included additional data entry fields for the event location and name of the organization involved. For example, the activity contact report form for meetings included optional data fields for the event location and organization name, as shown in figure 6. According to SBA officials, the temporary reporting tool (used since late July 2019 in place of the activity contact report) includes an optional data field for district offices to identity whether their activity was HBCU-related. In addition to the contact named above, Lisa Moore (Assistant Director), Chir-Jen Huang (Analyst in Charge), Rachel Beers, John Karikari, Ben Licht, John Mingus, Sulayman Njie, Maria Psara, Barbara Roesmann, Jessica Sandler, Jena Sinkfield, and Andrew Stavisky made key contributions to this report.", "summary": "The 101 HBCUs play an important role in higher education and in their local and regional economies. Among African Americans who obtained a doctorate in science, technology, engineering, or mathematics in 2005–2010, more than one-third earned their undergraduate degrees from an HBCU. SBA is part of a long-standing White House initiative to strengthen the capacity of HBCUs, including their ability to access and participate in federal programs. SBA's mission includes business development, and SBA also works with colleges and universities to provide entrepreneurial training and counseling. GAO was asked to review SBA's entrepreneurship-related efforts with HBCUs. This report examines (1) SBA efforts to foster entrepreneurship with HBCUs in recent years, (2) SBA's plans for the White House Initiative on HBCUs, and (3) the extent to which SBA collected information specific to HBCUs. GAO analyzed SBA information on HBCU participation in programs and activities for fostering entrepreneurship and reviewed related standard operating procedures. GAO also interviewed officials at SBA headquarters and eight SBA district offices, and representatives of nine Small Business Development Centers (selected for a high number of agreements with HBCUs and other factors). The Small Business Administration (SBA) worked with Historically Black Colleges and Universities (HBCU) to foster entrepreneurship, primarily through its Small Business Development Center program (which provides counseling and training), strategic alliance memorandums, and co-sponsorship agreements. Two HBCUs—Howard University and the University of the Virgin Islands—have hosted SBDC “lead centers” since the 1980s. SBA also signed at least 35 strategic alliance memorandums with HBCUs and at least 16 co-sponsorship agreements in 2013–2018. In 2018, SBA developed a plan to support HBCUs (including goals and measures) for the White House Initiative on HBCUs. However, SBA headquarters did not communicate this plan or its goals to key Small Business Development Centers or SBA district offices (those with HBCUs in their service areas). As a result, SBA may have missed opportunities to collaborate with HBCUs and help achieve the goals of its plan. SBA has collected limited information about its programs and activities with HBCUs. SBA could not establish a baseline for performance measures developed in its 2018 plan because SBA district offices and the Small Business Development Centers are not required to collect or report information about their HBCU-related outreach and other activities. For example, while representatives from the nine Small Business Development Centers with whom GAO spoke said they conducted outreach to HBCUs, this information was not reported to SBA headquarters. Without collecting relevant information about its HBCU-related efforts, including data for performance measures, SBA cannot assess the extent or effectiveness of its efforts to support HBCUs. GAO is making three recommendations, including that SBA communicate planned efforts to support HBCUs to key Small Business Development Centers or district offices, and collect additional information on its efforts to support HBCUs. SBA agreed with GAO's recommendations.", "document_type": "gao"}
{"report": "Depots are government-owned, government-operated industrial installations that maintain, overhaul, and repair a multitude of complex military weapons systems and equipment for the Department of Defense. These depots are essential to maintaining readiness for DOD, and they have a key role in sustaining weapon systems and equipment in both peacetime and during mobilization, contingency, or other emergency. There are 21 depots operated by the military services that are subject to the 6 percent minimum investment requirement (the “6 percent rule”)—four are Naval Shipyards, three are Navy Fleet Readiness Centers, two are Marine Corps Production Plants, three are Air Force Air Logistics Complexes, and nine are Army Depots and Arsenals. Figure 1 shows the location of these 21 depots across the United States. The depots are part of a larger, DOD-wide logistics enterprise that involves a number of different organizations. The Office of the Under Secretary of Defense for Acquisition and Sustainment is responsible for establishing policies for access to, and maintenance of, the defense industrial base, including depots. Specifically, the office is tasked with establishing policies and procedures for the management of DOD installations and environment to support military readiness with regard to facility construction, sustainment, and modernization. The Assistant Secretary of Defense for Sustainment serves as the principal assistant and advisor to the Under Secretary of Defense for Acquisition and Sustainment on material readiness. Among other responsibilities, the Assistant Secretary of Defense for Sustainment prescribes policies and procedures on maintenance, materiel readiness and sustainment support. DOD officials report that the Office of the Deputy Assistant Secretary of Defense for Materiel Readiness is responsible for maintenance policy along with the development of a strategic vision for DOD’s organic depot base. Finally, each service has its own logistics or materiel command component, which provides day